From webhook-mailer at python.org  Tue Oct  1 04:56:42 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Tue, 01 Oct 2019 08:56:42 -0000
Subject: [Python-checkins] bpo-38304: Remove PyConfig.struct_size (GH-16500)
 (GH-16508)
Message-ID: <mailman.168.1569920203.32504.python-checkins@python.org>

https://github.com/python/cpython/commit/3c30a76f3d3c0dcc1fb4de097fa4a3a4c92c0b0b
commit: 3c30a76f3d3c0dcc1fb4de097fa4a3a4c92c0b0b
branch: master
author: Victor Stinner <vstinner at redhat.com>
committer: GitHub <noreply at github.com>
date: 2019-10-01T10:56:37+02:00
summary:

bpo-38304: Remove PyConfig.struct_size (GH-16500) (GH-16508)

For now, we'll rely on the fact that the config structures aren't covered by the stable ABI.

We may revisit this in the future if we further explore the idea of offering a stable embedding API.

(cherry picked from commit bdace21b769998396d0ccc8da99a8ca9b507bfdf)

files:
D Misc/NEWS.d/next/C API/2019-09-28-03-43-27.bpo-38304.RqHAwd.rst
M Doc/c-api/init_config.rst
M Include/cpython/initconfig.h
M Include/internal/pycore_initconfig.h
M Modules/main.c
M PC/python_uwp.cpp
M Programs/_freeze_importlib.c
M Programs/_testembed.c
M Python/frozenmain.c
M Python/initconfig.c
M Python/pathconfig.c
M Python/preconfig.c
M Python/pylifecycle.c
M Python/pystate.c

diff --git a/Doc/c-api/init_config.rst b/Doc/c-api/init_config.rst
index 4c77d5d1e0e8..15b8b94d5a74 100644
--- a/Doc/c-api/init_config.rst
+++ b/Doc/c-api/init_config.rst
@@ -194,25 +194,18 @@ PyPreConfig
    * Configure the LC_CTYPE locale
    * Set the UTF-8 mode
 
-   The :c:member:`struct_size` field must be explicitly initialized to
-   ``sizeof(PyPreConfig)``.
-
    Function to initialize a preconfiguration:
 
-   .. c:function:: PyStatus PyPreConfig_InitIsolatedConfig(PyPreConfig *preconfig)
+   .. c:function:: void PyPreConfig_InitIsolatedConfig(PyPreConfig *preconfig)
 
       Initialize the preconfiguration with :ref:`Python Configuration
       <init-python-config>`.
 
-   .. c:function:: PyStatus PyPreConfig_InitPythonConfig(PyPreConfig *preconfig)
+   .. c:function:: void PyPreConfig_InitPythonConfig(PyPreConfig *preconfig)
 
       Initialize the preconfiguration with :ref:`Isolated Configuration
       <init-isolated-conf>`.
 
-   The caller of these functions is responsible to handle exceptions (error or
-   exit) using :c:func:`PyStatus_Exception` and
-   :c:func:`Py_ExitStatusException`.
-
    Structure fields:
 
    .. c:member:: int allocator
@@ -274,13 +267,6 @@ PyPreConfig
       same way the regular Python parses command line arguments: see
       :ref:`Command Line Arguments <using-on-cmdline>`.
 
-   .. c:member:: size_t struct_size
-
-      Size of the structure in bytes: must be initialized to
-      ``sizeof(PyPreConfig)``.
-
-      Field used for API and ABI compatibility.
-
    .. c:member:: int use_environment
 
       See :c:member:`PyConfig.use_environment`.
@@ -332,12 +318,7 @@ Example using the preinitialization to enable the UTF-8 Mode::
 
     PyStatus status;
     PyPreConfig preconfig;
-    preconfig.struct_size = sizeof(PyPreConfig);
-
-    status = PyPreConfig_InitPythonConfig(&preconfig);
-    if (PyStatus_Exception(status)) {
-        Py_ExitStatusException(status);
-    }
+    PyPreConfig_InitPythonConfig(&preconfig);
 
     preconfig.utf8_mode = 1;
 
@@ -360,9 +341,6 @@ PyConfig
 
    Structure containing most parameters to configure Python.
 
-   The :c:member:`struct_size` field must be explicitly initialized to
-   ``sizeof(PyConfig)``.
-
    Structure methods:
 
    .. c:function:: PyStatus PyConfig_InitPythonConfig(PyConfig *config)
@@ -679,13 +657,6 @@ PyConfig
       Encoding and encoding errors of :data:`sys.stdin`, :data:`sys.stdout` and
       :data:`sys.stderr`.
 
-   .. c:member:: size_t struct_size
-
-      Size of the structure in bytes: must be initialized to
-      ``sizeof(PyConfig)``.
-
-      Field used for API and ABI compatibility.
-
    .. c:member:: int tracemalloc
 
       If non-zero, call :func:`tracemalloc.start` at startup.
@@ -754,7 +725,6 @@ Example setting the program name::
     {
         PyStatus status;
         PyConfig config;
-        config.struct_size = sizeof(PyConfig);
 
         status = PyConfig_InitPythonConfig(&config);
         if (PyStatus_Exception(status)) {
@@ -787,7 +757,6 @@ configuration, and then override some parameters::
     {
         PyStatus status;
         PyConfig config;
-        config.struct_size = sizeof(PyConfig);
 
         status = PyConfig_InitPythonConfig(&config);
         if (PyStatus_Exception(status)) {
@@ -875,7 +844,6 @@ Example of customized Python always running in isolated mode::
     {
         PyStatus status;
         PyConfig config;
-        config.struct_size = sizeof(PyConfig);
 
         status = PyConfig_InitPythonConfig(&config);
         if (PyStatus_Exception(status)) {
@@ -1067,7 +1035,6 @@ phases::
     {
         PyStatus status;
         PyConfig config;
-        config.struct_size = sizeof(PyConfig);
 
         status = PyConfig_InitPythonConfig(&config);
         if (PyStatus_Exception(status)) {
diff --git a/Include/cpython/initconfig.h b/Include/cpython/initconfig.h
index 8ce5622c5c40..25e02cea10dc 100644
--- a/Include/cpython/initconfig.h
+++ b/Include/cpython/initconfig.h
@@ -45,10 +45,6 @@ PyAPI_FUNC(PyStatus) PyWideStringList_Insert(PyWideStringList *list,
 /* --- PyPreConfig ----------------------------------------------- */
 
 typedef struct {
-    /* Size of the structure in bytes: must be initialized to
-       sizeof(PyPreConfig). Field used for API and ABI compatibility. */
-    size_t struct_size;
-
     int _config_init;     /* _PyConfigInitEnum value */
 
     /* Parse Py_PreInitializeFromBytesArgs() arguments?
@@ -124,17 +120,13 @@ typedef struct {
     int allocator;
 } PyPreConfig;
 
-PyAPI_FUNC(PyStatus) PyPreConfig_InitPythonConfig(PyPreConfig *config);
-PyAPI_FUNC(PyStatus) PyPreConfig_InitIsolatedConfig(PyPreConfig *config);
+PyAPI_FUNC(void) PyPreConfig_InitPythonConfig(PyPreConfig *config);
+PyAPI_FUNC(void) PyPreConfig_InitIsolatedConfig(PyPreConfig *config);
 
 
 /* --- PyConfig ---------------------------------------------- */
 
 typedef struct {
-    /* Size of the structure in bytes: must be initialized to
-       sizeof(PyConfig). Field used for API and ABI compatibility. */
-    size_t struct_size;
-
     int _config_init;     /* _PyConfigInitEnum value */
 
     int isolated;         /* Isolated mode? see PyPreConfig.isolated */
diff --git a/Include/internal/pycore_initconfig.h b/Include/internal/pycore_initconfig.h
index eb6490f5a87f..e9810bcf90a8 100644
--- a/Include/internal/pycore_initconfig.h
+++ b/Include/internal/pycore_initconfig.h
@@ -120,8 +120,8 @@ extern PyStatus _PyPreCmdline_Read(_PyPreCmdline *cmdline,
 
 /* --- PyPreConfig ----------------------------------------------- */
 
-PyAPI_FUNC(PyStatus) _PyPreConfig_InitCompatConfig(PyPreConfig *preconfig);
-extern PyStatus _PyPreConfig_InitFromConfig(
+PyAPI_FUNC(void) _PyPreConfig_InitCompatConfig(PyPreConfig *preconfig);
+extern void _PyPreConfig_InitFromConfig(
     PyPreConfig *preconfig,
     const PyConfig *config);
 extern PyStatus _PyPreConfig_InitFromPreConfig(
diff --git a/Misc/NEWS.d/next/C API/2019-09-28-03-43-27.bpo-38304.RqHAwd.rst b/Misc/NEWS.d/next/C API/2019-09-28-03-43-27.bpo-38304.RqHAwd.rst
deleted file mode 100644
index b797933d05d6..000000000000
--- a/Misc/NEWS.d/next/C API/2019-09-28-03-43-27.bpo-38304.RqHAwd.rst	
+++ /dev/null
@@ -1,3 +0,0 @@
-Add a new ``struct_size`` field to :c:type:`PyPreConfig` and :c:type:`PyConfig`
-structures to allow to modify these structures in the future without breaking
-the backward compatibility.
diff --git a/Modules/main.c b/Modules/main.c
index ef6c66a8dbe5..9cbf3ac7060b 100644
--- a/Modules/main.c
+++ b/Modules/main.c
@@ -53,12 +53,7 @@ pymain_init(const _PyArgv *args)
 #endif
 
     PyPreConfig preconfig;
-    preconfig.struct_size = sizeof(PyPreConfig);
-
-    status = PyPreConfig_InitPythonConfig(&preconfig);
-    if (_PyStatus_EXCEPTION(status)) {
-        return status;
-    }
+    PyPreConfig_InitPythonConfig(&preconfig);
 
     status = _Py_PreInitializeFromPyArgv(&preconfig, args);
     if (_PyStatus_EXCEPTION(status)) {
@@ -66,7 +61,7 @@ pymain_init(const _PyArgv *args)
     }
 
     PyConfig config;
-    config.struct_size = sizeof(PyConfig);
+
     status = PyConfig_InitPythonConfig(&config);
     if (_PyStatus_EXCEPTION(status)) {
         goto done;
diff --git a/PC/python_uwp.cpp b/PC/python_uwp.cpp
index 2656d188c250..758c8f73725e 100644
--- a/PC/python_uwp.cpp
+++ b/PC/python_uwp.cpp
@@ -165,12 +165,8 @@ int
 wmain(int argc, wchar_t **argv)
 {
     PyStatus status;
-
     PyPreConfig preconfig;
-    preconfig.struct_size = sizeof(PyPreConfig);
-
     PyConfig config;
-    config.struct_size = sizeof(PyConfig);
 
     const wchar_t *moduleName = NULL;
     const wchar_t *p = wcsrchr(argv[0], L'\\');
@@ -189,10 +185,7 @@ wmain(int argc, wchar_t **argv)
         }
     }
 
-    status = PyPreConfig_InitPythonConfig(&preconfig);
-    if (PyStatus_Exception(status)) {
-        goto fail_without_config;
-    }
+    PyPreConfig_InitPythonConfig(&preconfig);
     if (!moduleName) {
         status = Py_PreInitializeFromArgs(&preconfig, argc, argv);
         if (PyStatus_Exception(status)) {
diff --git a/Programs/_freeze_importlib.c b/Programs/_freeze_importlib.c
index 7c494c2786c0..74735f279c58 100644
--- a/Programs/_freeze_importlib.c
+++ b/Programs/_freeze_importlib.c
@@ -78,7 +78,6 @@ main(int argc, char *argv[])
 
     PyStatus status;
     PyConfig config;
-    config.struct_size = sizeof(PyConfig);
 
     status = PyConfig_InitIsolatedConfig(&config);
     if (PyStatus_Exception(status)) {
diff --git a/Programs/_testembed.c b/Programs/_testembed.c
index c8600d58f0b4..d2d4c7f8c651 100644
--- a/Programs/_testembed.c
+++ b/Programs/_testembed.c
@@ -385,12 +385,7 @@ static int check_init_compat_config(int preinit)
 
     if (preinit) {
         PyPreConfig preconfig;
-        preconfig.struct_size = sizeof(PyPreConfig);
-
-        status = _PyPreConfig_InitCompatConfig(&preconfig);
-        if (PyStatus_Exception(status)) {
-            Py_ExitStatusException(status);
-        }
+        _PyPreConfig_InitCompatConfig(&preconfig);
 
         status = Py_PreInitialize(&preconfig);
         if (PyStatus_Exception(status)) {
@@ -399,7 +394,6 @@ static int check_init_compat_config(int preinit)
     }
 
     PyConfig config;
-    config.struct_size = sizeof(PyConfig);
 
     status = _PyConfig_InitCompatConfig(&config);
     if (PyStatus_Exception(status)) {
@@ -478,15 +472,8 @@ static int test_init_global_config(void)
 
 static int test_init_from_config(void)
 {
-    PyStatus status;
-
     PyPreConfig preconfig;
-    preconfig.struct_size = sizeof(PyPreConfig);
-
-    status = _PyPreConfig_InitCompatConfig(&preconfig);
-    if (PyStatus_Exception(status)) {
-        Py_ExitStatusException(status);
-    }
+    _PyPreConfig_InitCompatConfig(&preconfig);
 
     putenv("PYTHONMALLOC=malloc_debug");
     preconfig.allocator = PYMEM_ALLOCATOR_MALLOC;
@@ -495,13 +482,12 @@ static int test_init_from_config(void)
     Py_UTF8Mode = 0;
     preconfig.utf8_mode = 1;
 
-    status = Py_PreInitialize(&preconfig);
+    PyStatus status = Py_PreInitialize(&preconfig);
     if (PyStatus_Exception(status)) {
         Py_ExitStatusException(status);
     }
 
     PyConfig config;
-    config.struct_size = sizeof(PyConfig);
 
     status = _PyConfig_InitCompatConfig(&config);
     if (PyStatus_Exception(status)) {
@@ -638,7 +624,6 @@ static int check_init_parse_argv(int parse_argv)
     PyStatus status;
 
     PyConfig config;
-    config.struct_size = sizeof(PyConfig);
 
     status = PyConfig_InitPythonConfig(&config);
     if (PyStatus_Exception(status)) {
@@ -725,7 +710,6 @@ static int test_init_python_env(void)
     set_all_env_vars();
 
     PyConfig config;
-    config.struct_size = sizeof(PyConfig);
 
     status = PyConfig_InitPythonConfig(&config);
     if (PyStatus_Exception(status)) {
@@ -780,7 +764,6 @@ static int test_init_isolated_flag(void)
 
     /* Test PyConfig.isolated=1 */
     PyConfig config;
-    config.struct_size = sizeof(PyConfig);
 
     status = PyConfig_InitPythonConfig(&config);
     if (PyStatus_Exception(status)) {
@@ -803,25 +786,17 @@ static int test_init_isolated_flag(void)
 /* PyPreConfig.isolated=1, PyConfig.isolated=0 */
 static int test_preinit_isolated1(void)
 {
-    PyStatus status;
-
     PyPreConfig preconfig;
-    preconfig.struct_size = sizeof(PyPreConfig);
-
-    status = _PyPreConfig_InitCompatConfig(&preconfig);
-    if (PyStatus_Exception(status)) {
-        Py_ExitStatusException(status);
-    }
+    _PyPreConfig_InitCompatConfig(&preconfig);
 
     preconfig.isolated = 1;
 
-    status = Py_PreInitialize(&preconfig);
+    PyStatus status = Py_PreInitialize(&preconfig);
     if (PyStatus_Exception(status)) {
         Py_ExitStatusException(status);
     }
 
     PyConfig config;
-    config.struct_size = sizeof(PyConfig);
 
     status = _PyConfig_InitCompatConfig(&config);
     if (PyStatus_Exception(status)) {
@@ -840,26 +815,19 @@ static int test_preinit_isolated1(void)
 /* PyPreConfig.isolated=0, PyConfig.isolated=1 */
 static int test_preinit_isolated2(void)
 {
-    PyStatus status;
-
     PyPreConfig preconfig;
-    preconfig.struct_size = sizeof(PyPreConfig);
-
-    status = _PyPreConfig_InitCompatConfig(&preconfig);
-    if (PyStatus_Exception(status)) {
-        Py_ExitStatusException(status);
-    }
+    _PyPreConfig_InitCompatConfig(&preconfig);
 
     preconfig.isolated = 0;
 
-    status = Py_PreInitialize(&preconfig);
+    PyStatus status = Py_PreInitialize(&preconfig);
     if (PyStatus_Exception(status)) {
         Py_ExitStatusException(status);
     }
 
     /* Test PyConfig.isolated=1 */
     PyConfig config;
-    config.struct_size = sizeof(PyConfig);
+
     status = _PyConfig_InitCompatConfig(&config);
     if (PyStatus_Exception(status)) {
         Py_ExitStatusException(status);
@@ -880,15 +848,8 @@ static int test_preinit_isolated2(void)
 
 static int test_preinit_dont_parse_argv(void)
 {
-    PyStatus status;
-
     PyPreConfig preconfig;
-    preconfig.struct_size = sizeof(PyPreConfig);
-
-    status = PyPreConfig_InitIsolatedConfig(&preconfig);
-    if (PyStatus_Exception(status)) {
-        Py_ExitStatusException(status);
-    }
+    PyPreConfig_InitIsolatedConfig(&preconfig);
 
     preconfig.isolated = 0;
 
@@ -899,13 +860,13 @@ static int test_preinit_dont_parse_argv(void)
                        L"-X", L"dev",
                        L"-X", L"utf8",
                        L"script.py"};
-    status = Py_PreInitializeFromArgs(&preconfig, Py_ARRAY_LENGTH(argv), argv);
+    PyStatus status = Py_PreInitializeFromArgs(&preconfig,
+                                               Py_ARRAY_LENGTH(argv), argv);
     if (PyStatus_Exception(status)) {
         Py_ExitStatusException(status);
     }
 
     PyConfig config;
-    config.struct_size = sizeof(PyConfig);
 
     status = PyConfig_InitIsolatedConfig(&config);
     if (PyStatus_Exception(status)) {
@@ -931,7 +892,6 @@ static int test_preinit_parse_argv(void)
 {
     PyStatus status;
     PyConfig config;
-    config.struct_size = sizeof(PyConfig);
 
     status = PyConfig_InitPythonConfig(&config);
     if (PyStatus_Exception(status)) {
@@ -989,12 +949,7 @@ static int check_preinit_isolated_config(int preinit)
 
     if (preinit) {
         PyPreConfig preconfig;
-        preconfig.struct_size = sizeof(PyPreConfig);
-
-        status = PyPreConfig_InitIsolatedConfig(&preconfig);
-        if (PyStatus_Exception(status)) {
-            Py_ExitStatusException(status);
-        }
+        PyPreConfig_InitIsolatedConfig(&preconfig);
 
         status = Py_PreInitialize(&preconfig);
         if (PyStatus_Exception(status)) {
@@ -1007,7 +962,6 @@ static int check_preinit_isolated_config(int preinit)
     }
 
     PyConfig config;
-    config.struct_size = sizeof(PyConfig);
 
     status = PyConfig_InitIsolatedConfig(&config);
     if (PyStatus_Exception(status)) {
@@ -1058,12 +1012,7 @@ static int check_init_python_config(int preinit)
 
     if (preinit) {
         PyPreConfig preconfig;
-        preconfig.struct_size = sizeof(PyPreConfig);
-
-        status = PyPreConfig_InitPythonConfig(&preconfig);
-        if (PyStatus_Exception(status)) {
-            Py_ExitStatusException(status);
-        }
+        PyPreConfig_InitPythonConfig(&preconfig);
 
         status = Py_PreInitialize(&preconfig);
         if (PyStatus_Exception(status)) {
@@ -1072,7 +1021,6 @@ static int check_init_python_config(int preinit)
     }
 
     PyConfig config;
-    config.struct_size = sizeof(PyConfig);
 
     status = PyConfig_InitPythonConfig(&config);
     if (PyStatus_Exception(status)) {
@@ -1101,27 +1049,19 @@ static int test_init_python_config(void)
 
 static int test_init_dont_configure_locale(void)
 {
-    PyStatus status;
-
     PyPreConfig preconfig;
-    preconfig.struct_size = sizeof(PyPreConfig);
-
-    status = PyPreConfig_InitPythonConfig(&preconfig);
-    if (PyStatus_Exception(status)) {
-        Py_ExitStatusException(status);
-    }
+    PyPreConfig_InitPythonConfig(&preconfig);
 
     preconfig.configure_locale = 0;
     preconfig.coerce_c_locale = 1;
     preconfig.coerce_c_locale_warn = 1;
 
-    status = Py_PreInitialize(&preconfig);
+    PyStatus status = Py_PreInitialize(&preconfig);
     if (PyStatus_Exception(status)) {
         Py_ExitStatusException(status);
     }
 
     PyConfig config;
-    config.struct_size = sizeof(PyConfig);
 
     status = PyConfig_InitPythonConfig(&config);
     if (PyStatus_Exception(status)) {
@@ -1140,7 +1080,6 @@ static int test_init_dev_mode(void)
 {
     PyStatus status;
     PyConfig config;
-    config.struct_size = sizeof(PyConfig);
 
     status = PyConfig_InitPythonConfig(&config);
     if (PyStatus_Exception(status)) {
@@ -1365,7 +1304,6 @@ static int run_audit_run_test(int argc, wchar_t **argv, void *test)
 {
     PyStatus status;
     PyConfig config;
-    config.struct_size = sizeof(PyConfig);
 
     status = PyConfig_InitPythonConfig(&config);
     if (PyStatus_Exception(status)) {
@@ -1415,7 +1353,6 @@ static int test_init_read_set(void)
 {
     PyStatus status;
     PyConfig config;
-    config.struct_size = sizeof(PyConfig);
 
     status = PyConfig_InitPythonConfig(&config);
     if (PyStatus_Exception(status)) {
@@ -1465,7 +1402,6 @@ static int test_init_sys_add(void)
     PySys_AddWarnOption(L"ignore:::sysadd_warnoption");
 
     PyConfig config;
-    config.struct_size = sizeof(PyConfig);
 
     PyStatus status;
     status = PyConfig_InitPythonConfig(&config);
@@ -1533,18 +1469,12 @@ static int test_init_setpath(void)
 
 static int test_init_setpath_config(void)
 {
-    PyStatus status;
     PyPreConfig preconfig;
-    preconfig.struct_size = sizeof(PyPreConfig);
-
-    status = PyPreConfig_InitPythonConfig(&preconfig);
-    if (PyStatus_Exception(status)) {
-        Py_ExitStatusException(status);
-    }
+    PyPreConfig_InitPythonConfig(&preconfig);
 
     /* Explicitly preinitializes with Python preconfiguration to avoid
       Py_SetPath() implicit preinitialization with compat preconfiguration. */
-    status = Py_PreInitialize(&preconfig);
+    PyStatus status = Py_PreInitialize(&preconfig);
     if (PyStatus_Exception(status)) {
         Py_ExitStatusException(status);
     }
@@ -1564,7 +1494,6 @@ static int test_init_setpath_config(void)
     putenv("TESTPATH=");
 
     PyConfig config;
-    config.struct_size = sizeof(PyConfig);
 
     status = PyConfig_InitPythonConfig(&config);
     if (PyStatus_Exception(status)) {
@@ -1612,7 +1541,6 @@ static int test_init_warnoptions(void)
     PySys_AddWarnOption(L"ignore:::PySys_AddWarnOption2");
 
     PyConfig config;
-    config.struct_size = sizeof(PyConfig);
 
     status = PyConfig_InitPythonConfig(&config);
     if (PyStatus_Exception(status)) {
@@ -1680,7 +1608,6 @@ static int test_init_run_main(void)
 {
     PyStatus status;
     PyConfig config;
-    config.struct_size = sizeof(PyConfig);
 
     status = PyConfig_InitPythonConfig(&config);
     if (PyStatus_Exception(status)) {
@@ -1697,7 +1624,6 @@ static int test_init_main(void)
 {
     PyStatus status;
     PyConfig config;
-    config.struct_size = sizeof(PyConfig);
 
     status = PyConfig_InitPythonConfig(&config);
     if (PyStatus_Exception(status)) {
@@ -1729,7 +1655,6 @@ static int test_run_main(void)
 {
     PyStatus status;
     PyConfig config;
-    config.struct_size = sizeof(PyConfig);
 
     status = PyConfig_InitPythonConfig(&config);
     if (PyStatus_Exception(status)) {
diff --git a/Python/frozenmain.c b/Python/frozenmain.c
index 76309e9e5da2..c56938ab4899 100644
--- a/Python/frozenmain.c
+++ b/Python/frozenmain.c
@@ -40,7 +40,6 @@ Py_FrozenMain(int argc, char **argv)
     }
 
     PyConfig config;
-    config.struct_size = sizeof(PyConfig);
     status = PyConfig_InitPythonConfig(&config);
     if (PyStatus_Exception(status)) {
         PyConfig_Clear(&config);
diff --git a/Python/initconfig.c b/Python/initconfig.c
index dec4bf2e4d57..107429c7ec37 100644
--- a/Python/initconfig.c
+++ b/Python/initconfig.c
@@ -529,17 +529,6 @@ Py_GetArgcArgv(int *argc, wchar_t ***argv)
      : _PyStatus_NO_MEMORY())
 
 
-static PyStatus
-config_check_struct_size(const PyConfig *config)
-{
-    if (config->struct_size != sizeof(PyConfig)) {
-        return _PyStatus_ERR("unsupported PyConfig structure size "
-                             "(Python version mismatch?)");
-    }
-    return _PyStatus_OK();
-}
-
-
 /* Free memory allocated in config, but don't clear all attributes */
 void
 PyConfig_Clear(PyConfig *config)
@@ -583,15 +572,7 @@ PyConfig_Clear(PyConfig *config)
 PyStatus
 _PyConfig_InitCompatConfig(PyConfig *config)
 {
-    size_t struct_size = config->struct_size;
     memset(config, 0, sizeof(*config));
-    config->struct_size = struct_size;
-
-    PyStatus status = config_check_struct_size(config);
-    if (_PyStatus_EXCEPTION(status)) {
-        _PyStatus_UPDATE_FUNC(status);
-        return status;
-    }
 
     config->_config_init = (int)_PyConfig_INIT_COMPAT;
     config->isolated = -1;
@@ -775,18 +756,6 @@ _PyConfig_Copy(PyConfig *config, const PyConfig *config2)
 {
     PyStatus status;
 
-    status = config_check_struct_size(config);
-    if (_PyStatus_EXCEPTION(status)) {
-        _PyStatus_UPDATE_FUNC(status);
-        return status;
-    }
-
-    status = config_check_struct_size(config2);
-    if (_PyStatus_EXCEPTION(status)) {
-        _PyStatus_UPDATE_FUNC(status);
-        return status;
-    }
-
     PyConfig_Clear(config);
 
 #define COPY_ATTR(ATTR) config->ATTR = config2->ATTR
@@ -2280,7 +2249,6 @@ core_read_precmdline(PyConfig *config, _PyPreCmdline *precmdline)
     }
 
     PyPreConfig preconfig;
-    preconfig.struct_size = sizeof(PyPreConfig);
 
     status = _PyPreConfig_InitFromPreConfig(&preconfig, &_PyRuntime.preconfig);
     if (_PyStatus_EXCEPTION(status)) {
@@ -2475,12 +2443,6 @@ PyConfig_Read(PyConfig *config)
     PyStatus status;
     PyWideStringList orig_argv = _PyWideStringList_INIT;
 
-    status = config_check_struct_size(config);
-    if (_PyStatus_EXCEPTION(status)) {
-        _PyStatus_UPDATE_FUNC(status);
-        return status;
-    }
-
     status = _Py_PreInitializeFromConfig(config, NULL);
     if (_PyStatus_EXCEPTION(status)) {
         return status;
diff --git a/Python/pathconfig.c b/Python/pathconfig.c
index 6886ab7c42d6..5550194a5d56 100644
--- a/Python/pathconfig.c
+++ b/Python/pathconfig.c
@@ -434,7 +434,6 @@ pathconfig_global_read(_PyPathConfig *pathconfig)
 {
     PyStatus status;
     PyConfig config;
-    config.struct_size = sizeof(PyConfig);
 
     status = _PyConfig_InitCompatConfig(&config);
     if (_PyStatus_EXCEPTION(status)) {
diff --git a/Python/preconfig.c b/Python/preconfig.c
index 01c72f5d6bad..89a6227fa672 100644
--- a/Python/preconfig.c
+++ b/Python/preconfig.c
@@ -269,29 +269,10 @@ _PyPreCmdline_Read(_PyPreCmdline *cmdline, const PyPreConfig *preconfig)
 /* --- PyPreConfig ----------------------------------------------- */
 
 
-static PyStatus
-preconfig_check_struct_size(PyPreConfig *config)
-{
-    if (config->struct_size != sizeof(PyPreConfig)) {
-        return _PyStatus_ERR("unsupported PyPreConfig structure size "
-                             "(Python version mismatch?)");
-    }
-    return _PyStatus_OK();
-}
-
-
-PyStatus
+void
 _PyPreConfig_InitCompatConfig(PyPreConfig *config)
 {
-    size_t struct_size = config->struct_size;
     memset(config, 0, sizeof(*config));
-    config->struct_size = struct_size;
-
-    PyStatus status = preconfig_check_struct_size(config);
-    if (_PyStatus_EXCEPTION(status)) {
-        _PyStatus_UPDATE_FUNC(status);
-        return status;
-    }
 
     config->_config_init = (int)_PyConfig_INIT_COMPAT;
     config->parse_argv = 0;
@@ -313,18 +294,13 @@ _PyPreConfig_InitCompatConfig(PyPreConfig *config)
 #ifdef MS_WINDOWS
     config->legacy_windows_fs_encoding = -1;
 #endif
-    return _PyStatus_OK();
 }
 
 
-PyStatus
+void
 PyPreConfig_InitPythonConfig(PyPreConfig *config)
 {
-    PyStatus status = _PyPreConfig_InitCompatConfig(config);
-    if (_PyStatus_EXCEPTION(status)) {
-        _PyStatus_UPDATE_FUNC(status);
-        return status;
-    }
+    _PyPreConfig_InitCompatConfig(config);
 
     config->_config_init = (int)_PyConfig_INIT_PYTHON;
     config->isolated = 0;
@@ -339,18 +315,13 @@ PyPreConfig_InitPythonConfig(PyPreConfig *config)
 #ifdef MS_WINDOWS
     config->legacy_windows_fs_encoding = 0;
 #endif
-    return _PyStatus_OK();
 }
 
 
-PyStatus
+void
 PyPreConfig_InitIsolatedConfig(PyPreConfig *config)
 {
-    PyStatus status = _PyPreConfig_InitCompatConfig(config);
-    if (_PyStatus_EXCEPTION(status)) {
-        _PyStatus_UPDATE_FUNC(status);
-        return status;
-    }
+    _PyPreConfig_InitCompatConfig(config);
 
     config->_config_init = (int)_PyConfig_INIT_ISOLATED;
     config->configure_locale = 0;
@@ -361,7 +332,6 @@ PyPreConfig_InitIsolatedConfig(PyPreConfig *config)
 #ifdef MS_WINDOWS
     config->legacy_windows_fs_encoding = 0;
 #endif
-    return _PyStatus_OK();
 }
 
 
@@ -369,47 +339,35 @@ PyStatus
 _PyPreConfig_InitFromPreConfig(PyPreConfig *config,
                                const PyPreConfig *config2)
 {
-    PyStatus status = PyPreConfig_InitPythonConfig(config);
-    if (_PyStatus_EXCEPTION(status)) {
-        return status;
-    }
-
+    PyPreConfig_InitPythonConfig(config);
     preconfig_copy(config, config2);
     return _PyStatus_OK();
 }
 
 
-PyStatus
+void
 _PyPreConfig_InitFromConfig(PyPreConfig *preconfig, const PyConfig *config)
 {
-    PyStatus status;
     _PyConfigInitEnum config_init = (_PyConfigInitEnum)config->_config_init;
     switch (config_init) {
     case _PyConfig_INIT_PYTHON:
-        status = PyPreConfig_InitPythonConfig(preconfig);
+        PyPreConfig_InitPythonConfig(preconfig);
         break;
     case _PyConfig_INIT_ISOLATED:
-        status = PyPreConfig_InitIsolatedConfig(preconfig);
+        PyPreConfig_InitIsolatedConfig(preconfig);
         break;
     case _PyConfig_INIT_COMPAT:
     default:
-        status = _PyPreConfig_InitCompatConfig(preconfig);
-    }
-
-    if (_PyStatus_EXCEPTION(status)) {
-        return status;
+        _PyPreConfig_InitCompatConfig(preconfig);
     }
 
     _PyPreConfig_GetConfig(preconfig, config);
-    return _PyStatus_OK();
 }
 
 
 static void
 preconfig_copy(PyPreConfig *config, const PyPreConfig *config2)
 {
-    assert(config->struct_size == sizeof(PyPreConfig));
-
 #define COPY_ATTR(ATTR) config->ATTR = config2->ATTR
 
     COPY_ATTR(_config_init);
@@ -829,12 +787,6 @@ _PyPreConfig_Read(PyPreConfig *config, const _PyArgv *args)
         return status;
     }
 
-    status = preconfig_check_struct_size(config);
-    if (_PyStatus_EXCEPTION(status)) {
-        _PyStatus_UPDATE_FUNC(status);
-        return status;
-    }
-
     preconfig_get_global_vars(config);
 
     /* Copy LC_CTYPE locale, since it's modified later */
@@ -849,7 +801,6 @@ _PyPreConfig_Read(PyPreConfig *config, const _PyArgv *args)
 
     /* Save the config to be able to restore it if encodings change */
     PyPreConfig save_config;
-    save_config.struct_size = sizeof(PyPreConfig);
 
     status = _PyPreConfig_InitFromPreConfig(&save_config, config);
     if (_PyStatus_EXCEPTION(status)) {
@@ -976,7 +927,6 @@ PyStatus
 _PyPreConfig_Write(const PyPreConfig *src_config)
 {
     PyPreConfig config;
-    config.struct_size = sizeof(PyPreConfig);
 
     PyStatus status = _PyPreConfig_InitFromPreConfig(&config, src_config);
     if (_PyStatus_EXCEPTION(status)) {
diff --git a/Python/pylifecycle.c b/Python/pylifecycle.c
index ea0d7a5ee2b9..019c56c3f8b0 100644
--- a/Python/pylifecycle.c
+++ b/Python/pylifecycle.c
@@ -735,7 +735,6 @@ _Py_PreInitializeFromPyArgv(const PyPreConfig *src_config, const _PyArgv *args)
     runtime->preinitializing = 1;
 
     PyPreConfig config;
-    config.struct_size = sizeof(PyPreConfig);
 
     status = _PyPreConfig_InitFromPreConfig(&config, src_config);
     if (_PyStatus_EXCEPTION(status)) {
@@ -799,12 +798,8 @@ _Py_PreInitializeFromConfig(const PyConfig *config,
     }
 
     PyPreConfig preconfig;
-    preconfig.struct_size = sizeof(PyPreConfig);
 
-    status = _PyPreConfig_InitFromConfig(&preconfig, config);
-    if (_PyStatus_EXCEPTION(status)) {
-        return status;
-    }
+    _PyPreConfig_InitFromConfig(&preconfig, config);
 
     if (!config->parse_argv) {
         return Py_PreInitialize(&preconfig);
@@ -852,7 +847,6 @@ pyinit_core(_PyRuntimeState *runtime,
     }
 
     PyConfig config;
-    config.struct_size = sizeof(PyConfig);
 
     status = _PyConfig_InitCompatConfig(&config);
     if (_PyStatus_EXCEPTION(status)) {
@@ -1079,7 +1073,6 @@ Py_InitializeEx(int install_sigs)
     }
 
     PyConfig config;
-    config.struct_size = sizeof(PyConfig);
 
     status = _PyConfig_InitCompatConfig(&config);
     if (_PyStatus_EXCEPTION(status)) {
diff --git a/Python/pystate.c b/Python/pystate.c
index 0f0cb2299557..69b86ce4f7a3 100644
--- a/Python/pystate.c
+++ b/Python/pystate.c
@@ -61,11 +61,7 @@ _PyRuntimeState_Init_impl(_PyRuntimeState *runtime)
     _PyGC_Initialize(&runtime->gc);
     _PyEval_Initialize(&runtime->ceval);
 
-    runtime->preconfig.struct_size = sizeof(PyPreConfig);
-    PyStatus status = PyPreConfig_InitPythonConfig(&runtime->preconfig);
-    if (_PyStatus_EXCEPTION(status)) {
-        return status;
-    }
+    PyPreConfig_InitPythonConfig(&runtime->preconfig);
 
     runtime->gilstate.check_enabled = 1;
 
@@ -209,7 +205,6 @@ PyInterpreterState_New(void)
     memset(interp, 0, sizeof(*interp));
     interp->id_refcount = -1;
 
-    interp->config.struct_size = sizeof(PyConfig);
     PyStatus status = PyConfig_InitPythonConfig(&interp->config);
     if (_PyStatus_EXCEPTION(status)) {
         /* Don't report status to caller: PyConfig_InitPythonConfig()


From webhook-mailer at python.org  Tue Oct  1 06:06:20 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Tue, 01 Oct 2019 10:06:20 -0000
Subject: [Python-checkins] bpo-38304: PyConfig_InitPythonConfig() cannot
 fail anymore (GH-16509)
Message-ID: <mailman.169.1569924381.32504.python-checkins@python.org>

https://github.com/python/cpython/commit/8462a4936b3a551dc546a6adea04a70b0a07ca67
commit: 8462a4936b3a551dc546a6adea04a70b0a07ca67
branch: master
author: Victor Stinner <vstinner at redhat.com>
committer: GitHub <noreply at github.com>
date: 2019-10-01T12:06:16+02:00
summary:

bpo-38304: PyConfig_InitPythonConfig() cannot fail anymore (GH-16509)

PyConfig_InitPythonConfig() and PyConfig_InitIsolatedConfig() no
longer return PyStatus: they cannot fail anymore.

files:
M Doc/c-api/init_config.rst
M Include/cpython/initconfig.h
M Include/internal/pycore_initconfig.h
M Modules/main.c
M PC/python_uwp.cpp
M Programs/_freeze_importlib.c
M Programs/_testembed.c
M Python/frozenmain.c
M Python/initconfig.c
M Python/pathconfig.c
M Python/pylifecycle.c
M Python/pystate.c

diff --git a/Doc/c-api/init_config.rst b/Doc/c-api/init_config.rst
index 15b8b94d5a74..5e18300471d4 100644
--- a/Doc/c-api/init_config.rst
+++ b/Doc/c-api/init_config.rst
@@ -343,12 +343,12 @@ PyConfig
 
    Structure methods:
 
-   .. c:function:: PyStatus PyConfig_InitPythonConfig(PyConfig *config)
+   .. c:function:: void PyConfig_InitPythonConfig(PyConfig *config)
 
       Initialize configuration with :ref:`Python Configuration
       <init-python-config>`.
 
-   .. c:function:: PyStatus PyConfig_InitIsolatedConfig(PyConfig *config)
+   .. c:function:: void PyConfig_InitIsolatedConfig(PyConfig *config)
 
       Initialize configuration with :ref:`Isolated Configuration
       <init-isolated-conf>`.
@@ -724,12 +724,9 @@ Example setting the program name::
     void init_python(void)
     {
         PyStatus status;
-        PyConfig config;
 
-        status = PyConfig_InitPythonConfig(&config);
-        if (PyStatus_Exception(status)) {
-            goto fail;
-        }
+        PyConfig config;
+        PyConfig_InitPythonConfig(&config);
 
         /* Set the program name. Implicitly preinitialize Python. */
         status = PyConfig_SetString(&config, &config.program_name,
@@ -756,12 +753,9 @@ configuration, and then override some parameters::
     PyStatus init_python(const char *program_name)
     {
         PyStatus status;
-        PyConfig config;
 
-        status = PyConfig_InitPythonConfig(&config);
-        if (PyStatus_Exception(status)) {
-            goto done;
-        }
+        PyConfig config;
+        PyConfig_InitPythonConfig(&config);
 
         /* Set the program name before reading the configuraton
            (decode byte string from the locale encoding).
@@ -843,13 +837,9 @@ Example of customized Python always running in isolated mode::
     int main(int argc, char **argv)
     {
         PyStatus status;
-        PyConfig config;
-
-        status = PyConfig_InitPythonConfig(&config);
-        if (PyStatus_Exception(status)) {
-            goto fail;
-        }
 
+        PyConfig config;
+        PyConfig_InitPythonConfig(&config);
         config.isolated = 1;
 
         /* Decode command line arguments.
@@ -1034,14 +1024,9 @@ phases::
     void init_python(void)
     {
         PyStatus status;
-        PyConfig config;
-
-        status = PyConfig_InitPythonConfig(&config);
-        if (PyStatus_Exception(status)) {
-            PyConfig_Clear(&config);
-            Py_ExitStatusException(status);
-        }
 
+        PyConfig config;
+        PyConfig_InitPythonConfig(&config);
         config._init_main = 0;
 
         /* ... customize 'config' configuration ... */
diff --git a/Include/cpython/initconfig.h b/Include/cpython/initconfig.h
index 25e02cea10dc..4b5ceafe02db 100644
--- a/Include/cpython/initconfig.h
+++ b/Include/cpython/initconfig.h
@@ -404,8 +404,8 @@ typedef struct {
     int _init_main;
 } PyConfig;
 
-PyAPI_FUNC(PyStatus) PyConfig_InitPythonConfig(PyConfig *config);
-PyAPI_FUNC(PyStatus) PyConfig_InitIsolatedConfig(PyConfig *config);
+PyAPI_FUNC(void) PyConfig_InitPythonConfig(PyConfig *config);
+PyAPI_FUNC(void) PyConfig_InitIsolatedConfig(PyConfig *config);
 PyAPI_FUNC(void) PyConfig_Clear(PyConfig *);
 PyAPI_FUNC(PyStatus) PyConfig_SetString(
     PyConfig *config,
diff --git a/Include/internal/pycore_initconfig.h b/Include/internal/pycore_initconfig.h
index e9810bcf90a8..40831c44b2fa 100644
--- a/Include/internal/pycore_initconfig.h
+++ b/Include/internal/pycore_initconfig.h
@@ -144,7 +144,7 @@ typedef enum {
     _PyConfig_INIT_ISOLATED = 3
 } _PyConfigInitEnum;
 
-PyAPI_FUNC(PyStatus) _PyConfig_InitCompatConfig(PyConfig *config);
+PyAPI_FUNC(void) _PyConfig_InitCompatConfig(PyConfig *config);
 extern PyStatus _PyConfig_Copy(
     PyConfig *config,
     const PyConfig *config2);
diff --git a/Modules/main.c b/Modules/main.c
index 9cbf3ac7060b..526b0ade84a4 100644
--- a/Modules/main.c
+++ b/Modules/main.c
@@ -61,11 +61,7 @@ pymain_init(const _PyArgv *args)
     }
 
     PyConfig config;
-
-    status = PyConfig_InitPythonConfig(&config);
-    if (_PyStatus_EXCEPTION(status)) {
-        goto done;
-    }
+    PyConfig_InitPythonConfig(&config);
 
     /* pass NULL as the config: config is read from command line arguments,
        environment variables, configuration files */
diff --git a/PC/python_uwp.cpp b/PC/python_uwp.cpp
index 758c8f73725e..88369e8fbfeb 100644
--- a/PC/python_uwp.cpp
+++ b/PC/python_uwp.cpp
@@ -193,10 +193,7 @@ wmain(int argc, wchar_t **argv)
         }
     }
 
-    status = PyConfig_InitPythonConfig(&config);
-    if (PyStatus_Exception(status)) {
-        goto fail_without_config;
-    }
+    PyConfig_InitPythonConfig(&config);
 
     status = PyConfig_SetArgv(&config, argc, argv);
     if (PyStatus_Exception(status)) {
diff --git a/Programs/_freeze_importlib.c b/Programs/_freeze_importlib.c
index 74735f279c58..2e4ccbb154a4 100644
--- a/Programs/_freeze_importlib.c
+++ b/Programs/_freeze_importlib.c
@@ -76,17 +76,12 @@ main(int argc, char *argv[])
     }
     text[text_size] = '\0';
 
-    PyStatus status;
     PyConfig config;
-
-    status = PyConfig_InitIsolatedConfig(&config);
-    if (PyStatus_Exception(status)) {
-        PyConfig_Clear(&config);
-        Py_ExitStatusException(status);
-    }
+    PyConfig_InitIsolatedConfig(&config);
 
     config.site_import = 0;
 
+    PyStatus status;
     status = PyConfig_SetString(&config, &config.program_name,
                                 L"./_freeze_importlib");
     if (PyStatus_Exception(status)) {
diff --git a/Programs/_testembed.c b/Programs/_testembed.c
index d2d4c7f8c651..b98a38a1ba67 100644
--- a/Programs/_testembed.c
+++ b/Programs/_testembed.c
@@ -394,11 +394,7 @@ static int check_init_compat_config(int preinit)
     }
 
     PyConfig config;
-
-    status = _PyConfig_InitCompatConfig(&config);
-    if (PyStatus_Exception(status)) {
-        Py_ExitStatusException(status);
-    }
+    _PyConfig_InitCompatConfig(&config);
 
     config_set_program_name(&config);
     init_from_config_clear(&config);
@@ -488,11 +484,8 @@ static int test_init_from_config(void)
     }
 
     PyConfig config;
+    _PyConfig_InitCompatConfig(&config);
 
-    status = _PyConfig_InitCompatConfig(&config);
-    if (PyStatus_Exception(status)) {
-        Py_ExitStatusException(status);
-    }
     config.install_signal_handlers = 0;
 
     /* FIXME: test use_environment */
@@ -621,14 +614,8 @@ static int test_init_from_config(void)
 
 static int check_init_parse_argv(int parse_argv)
 {
-    PyStatus status;
-
     PyConfig config;
-
-    status = PyConfig_InitPythonConfig(&config);
-    if (PyStatus_Exception(status)) {
-        Py_ExitStatusException(status);
-    }
+    PyConfig_InitPythonConfig(&config);
 
     config.parse_argv = parse_argv;
 
@@ -705,16 +692,10 @@ static int test_init_compat_env(void)
 
 static int test_init_python_env(void)
 {
-    PyStatus status;
-
     set_all_env_vars();
 
     PyConfig config;
-
-    status = PyConfig_InitPythonConfig(&config);
-    if (PyStatus_Exception(status)) {
-        Py_ExitStatusException(status);
-    }
+    PyConfig_InitPythonConfig(&config);
 
     config_set_program_name(&config);
     init_from_config_clear(&config);
@@ -760,15 +741,9 @@ static int test_init_env_dev_mode_alloc(void)
 
 static int test_init_isolated_flag(void)
 {
-    PyStatus status;
-
     /* Test PyConfig.isolated=1 */
     PyConfig config;
-
-    status = PyConfig_InitPythonConfig(&config);
-    if (PyStatus_Exception(status)) {
-        Py_ExitStatusException(status);
-    }
+    PyConfig_InitPythonConfig(&config);
 
     Py_IsolatedFlag = 0;
     config.isolated = 1;
@@ -797,11 +772,8 @@ static int test_preinit_isolated1(void)
     }
 
     PyConfig config;
+    _PyConfig_InitCompatConfig(&config);
 
-    status = _PyConfig_InitCompatConfig(&config);
-    if (PyStatus_Exception(status)) {
-        Py_ExitStatusException(status);
-    }
     config_set_program_name(&config);
     set_all_env_vars();
     init_from_config_clear(&config);
@@ -827,11 +799,7 @@ static int test_preinit_isolated2(void)
 
     /* Test PyConfig.isolated=1 */
     PyConfig config;
-
-    status = _PyConfig_InitCompatConfig(&config);
-    if (PyStatus_Exception(status)) {
-        Py_ExitStatusException(status);
-    }
+    _PyConfig_InitCompatConfig(&config);
 
     Py_IsolatedFlag = 0;
     config.isolated = 1;
@@ -867,12 +835,7 @@ static int test_preinit_dont_parse_argv(void)
     }
 
     PyConfig config;
-
-    status = PyConfig_InitIsolatedConfig(&config);
-    if (PyStatus_Exception(status)) {
-        PyConfig_Clear(&config);
-        Py_ExitStatusException(status);
-    }
+    PyConfig_InitIsolatedConfig(&config);
 
     config.isolated = 0;
 
@@ -890,14 +853,8 @@ static int test_preinit_dont_parse_argv(void)
 
 static int test_preinit_parse_argv(void)
 {
-    PyStatus status;
     PyConfig config;
-
-    status = PyConfig_InitPythonConfig(&config);
-    if (PyStatus_Exception(status)) {
-        PyConfig_Clear(&config);
-        Py_ExitStatusException(status);
-    }
+    PyConfig_InitPythonConfig(&config);
 
     /* Pre-initialize implicitly using argv: make sure that -X dev
        is used to configure the allocation in preinitialization */
@@ -962,12 +919,8 @@ static int check_preinit_isolated_config(int preinit)
     }
 
     PyConfig config;
+    PyConfig_InitIsolatedConfig(&config);
 
-    status = PyConfig_InitIsolatedConfig(&config);
-    if (PyStatus_Exception(status)) {
-        PyConfig_Clear(&config);
-        Py_ExitStatusException(status);
-    }
     config_set_program_name(&config);
     init_from_config_clear(&config);
 
@@ -995,8 +948,6 @@ static int test_init_isolated_config(void)
 
 static int check_init_python_config(int preinit)
 {
-    PyStatus status;
-
     /* global configuration variables must be ignored */
     set_all_global_config_variables();
     Py_IsolatedFlag = 1;
@@ -1014,18 +965,15 @@ static int check_init_python_config(int preinit)
         PyPreConfig preconfig;
         PyPreConfig_InitPythonConfig(&preconfig);
 
-        status = Py_PreInitialize(&preconfig);
+        PyStatus status = Py_PreInitialize(&preconfig);
         if (PyStatus_Exception(status)) {
             Py_ExitStatusException(status);
         }
     }
 
     PyConfig config;
+    PyConfig_InitPythonConfig(&config);
 
-    status = PyConfig_InitPythonConfig(&config);
-    if (PyStatus_Exception(status)) {
-        Py_ExitStatusException(status);
-    }
     config_set_program_name(&config);
     init_from_config_clear(&config);
 
@@ -1062,11 +1010,8 @@ static int test_init_dont_configure_locale(void)
     }
 
     PyConfig config;
+    PyConfig_InitPythonConfig(&config);
 
-    status = PyConfig_InitPythonConfig(&config);
-    if (PyStatus_Exception(status)) {
-        Py_ExitStatusException(status);
-    }
     config_set_program_name(&config);
     init_from_config_clear(&config);
 
@@ -1078,13 +1023,9 @@ static int test_init_dont_configure_locale(void)
 
 static int test_init_dev_mode(void)
 {
-    PyStatus status;
     PyConfig config;
+    PyConfig_InitPythonConfig(&config);
 
-    status = PyConfig_InitPythonConfig(&config);
-    if (PyStatus_Exception(status)) {
-        Py_ExitStatusException(status);
-    }
     putenv("PYTHONFAULTHANDLER=");
     putenv("PYTHONMALLOC=");
     config.dev_mode = 1;
@@ -1302,13 +1243,9 @@ static int test_audit_run_file(void)
 
 static int run_audit_run_test(int argc, wchar_t **argv, void *test)
 {
-    PyStatus status;
     PyConfig config;
+    PyConfig_InitPythonConfig(&config);
 
-    status = PyConfig_InitPythonConfig(&config);
-    if (PyStatus_Exception(status)) {
-        Py_ExitStatusException(status);
-    }
     config.argv.length = argc;
     config.argv.items = argv;
     config.parse_argv = 1;
@@ -1320,7 +1257,7 @@ static int run_audit_run_test(int argc, wchar_t **argv, void *test)
 
     PySys_AddAuditHook(_audit_hook_run, test);
 
-    status = Py_InitializeFromConfig(&config);
+    PyStatus status = Py_InitializeFromConfig(&config);
     if (PyStatus_Exception(status)) {
         Py_ExitStatusException(status);
     }
@@ -1353,11 +1290,7 @@ static int test_init_read_set(void)
 {
     PyStatus status;
     PyConfig config;
-
-    status = PyConfig_InitPythonConfig(&config);
-    if (PyStatus_Exception(status)) {
-        Py_ExitStatusException(status);
-    }
+    PyConfig_InitPythonConfig(&config);
 
     status = PyConfig_SetBytesString(&config, &config.program_name,
                                      "./init_read_set");
@@ -1402,12 +1335,7 @@ static int test_init_sys_add(void)
     PySys_AddWarnOption(L"ignore:::sysadd_warnoption");
 
     PyConfig config;
-
-    PyStatus status;
-    status = PyConfig_InitPythonConfig(&config);
-    if (PyStatus_Exception(status)) {
-        goto fail;
-    }
+    PyConfig_InitPythonConfig(&config);
 
     wchar_t* argv[] = {
         L"python3",
@@ -1419,6 +1347,7 @@ static int test_init_sys_add(void)
     config_set_argv(&config, Py_ARRAY_LENGTH(argv), argv);
     config.parse_argv = 1;
 
+    PyStatus status;
     status = PyWideStringList_Append(&config.xoptions,
                                      L"config_xoption");
     if (PyStatus_Exception(status)) {
@@ -1494,11 +1423,8 @@ static int test_init_setpath_config(void)
     putenv("TESTPATH=");
 
     PyConfig config;
+    PyConfig_InitPythonConfig(&config);
 
-    status = PyConfig_InitPythonConfig(&config);
-    if (PyStatus_Exception(status)) {
-        Py_ExitStatusException(status);
-    }
     config_set_string(&config, &config.program_name, L"conf_program_name");
     config_set_string(&config, &config.executable, L"conf_executable");
     init_from_config_clear(&config);
@@ -1534,24 +1460,20 @@ static int test_init_setpythonhome(void)
 
 static int test_init_warnoptions(void)
 {
-    PyStatus status;
     putenv("PYTHONWARNINGS=ignore:::env1,ignore:::env2");
 
     PySys_AddWarnOption(L"ignore:::PySys_AddWarnOption1");
     PySys_AddWarnOption(L"ignore:::PySys_AddWarnOption2");
 
     PyConfig config;
-
-    status = PyConfig_InitPythonConfig(&config);
-    if (PyStatus_Exception(status)) {
-        Py_ExitStatusException(status);
-    }
+    PyConfig_InitPythonConfig(&config);
 
     config.dev_mode = 1;
     config.bytes_warning = 1;
 
     config_set_program_name(&config);
 
+    PyStatus status;
     status = PyWideStringList_Append(&config.warnoptions,
                                      L"ignore:::PyConfig_BeforeRead");
     if (PyStatus_Exception(status)) {
@@ -1606,13 +1528,9 @@ static void configure_init_main(PyConfig *config)
 
 static int test_init_run_main(void)
 {
-    PyStatus status;
     PyConfig config;
+    PyConfig_InitPythonConfig(&config);
 
-    status = PyConfig_InitPythonConfig(&config);
-    if (PyStatus_Exception(status)) {
-        Py_ExitStatusException(status);
-    }
     configure_init_main(&config);
     init_from_config_clear(&config);
 
@@ -1622,13 +1540,9 @@ static int test_init_run_main(void)
 
 static int test_init_main(void)
 {
-    PyStatus status;
     PyConfig config;
+    PyConfig_InitPythonConfig(&config);
 
-    status = PyConfig_InitPythonConfig(&config);
-    if (PyStatus_Exception(status)) {
-        Py_ExitStatusException(status);
-    }
     configure_init_main(&config);
     config._init_main = 0;
     init_from_config_clear(&config);
@@ -1642,7 +1556,7 @@ static int test_init_main(void)
         exit(1);
     }
 
-    status = _Py_InitializeMain();
+    PyStatus status = _Py_InitializeMain();
     if (PyStatus_Exception(status)) {
         Py_ExitStatusException(status);
     }
@@ -1653,14 +1567,8 @@ static int test_init_main(void)
 
 static int test_run_main(void)
 {
-    PyStatus status;
     PyConfig config;
-
-    status = PyConfig_InitPythonConfig(&config);
-    if (PyStatus_Exception(status)) {
-        PyConfig_Clear(&config);
-        Py_ExitStatusException(status);
-    }
+    PyConfig_InitPythonConfig(&config);
 
     wchar_t *argv[] = {L"python3", L"-c",
                        (L"import sys; "
diff --git a/Python/frozenmain.c b/Python/frozenmain.c
index c56938ab4899..7f9cc1931736 100644
--- a/Python/frozenmain.c
+++ b/Python/frozenmain.c
@@ -40,11 +40,7 @@ Py_FrozenMain(int argc, char **argv)
     }
 
     PyConfig config;
-    status = PyConfig_InitPythonConfig(&config);
-    if (PyStatus_Exception(status)) {
-        PyConfig_Clear(&config);
-        Py_ExitStatusException(status);
-    }
+    PyConfig_InitPythonConfig(&config);
     config.pathconfig_warnings = 0;   /* Suppress errors from getpath.c */
 
     if ((p = Py_GETENV("PYTHONINSPECT")) && *p != '\0')
diff --git a/Python/initconfig.c b/Python/initconfig.c
index 107429c7ec37..caa9bf5f5689 100644
--- a/Python/initconfig.c
+++ b/Python/initconfig.c
@@ -569,7 +569,7 @@ PyConfig_Clear(PyConfig *config)
 }
 
 
-PyStatus
+void
 _PyConfig_InitCompatConfig(PyConfig *config)
 {
     memset(config, 0, sizeof(*config));
@@ -603,17 +603,13 @@ _PyConfig_InitCompatConfig(PyConfig *config)
 #ifdef MS_WINDOWS
     config->legacy_windows_stdio = -1;
 #endif
-    return _PyStatus_OK();
 }
 
 
-static PyStatus
+static void
 config_init_defaults(PyConfig *config)
 {
-    PyStatus status = _PyConfig_InitCompatConfig(config);
-    if (_PyStatus_EXCEPTION(status)) {
-        return status;
-    }
+    _PyConfig_InitCompatConfig(config);
 
     config->isolated = 0;
     config->use_environment = 1;
@@ -632,35 +628,24 @@ config_init_defaults(PyConfig *config)
 #ifdef MS_WINDOWS
     config->legacy_windows_stdio = 0;
 #endif
-    return _PyStatus_OK();
 }
 
 
-PyStatus
+void
 PyConfig_InitPythonConfig(PyConfig *config)
 {
-    PyStatus status = config_init_defaults(config);
-    if (_PyStatus_EXCEPTION(status)) {
-        _PyStatus_UPDATE_FUNC(status);
-        return status;
-    }
+    config_init_defaults(config);
 
     config->_config_init = (int)_PyConfig_INIT_PYTHON;
     config->configure_c_stdio = 1;
     config->parse_argv = 1;
-
-    return _PyStatus_OK();
 }
 
 
-PyStatus
+void
 PyConfig_InitIsolatedConfig(PyConfig *config)
 {
-    PyStatus status = config_init_defaults(config);
-    if (_PyStatus_EXCEPTION(status)) {
-        _PyStatus_UPDATE_FUNC(status);
-        return status;
-    }
+    config_init_defaults(config);
 
     config->_config_init = (int)_PyConfig_INIT_ISOLATED;
     config->isolated = 1;
@@ -675,8 +660,6 @@ PyConfig_InitIsolatedConfig(PyConfig *config)
 #ifdef MS_WINDOWS
     config->legacy_windows_stdio = 0;
 #endif
-
-    return _PyStatus_OK();
 }
 
 
diff --git a/Python/pathconfig.c b/Python/pathconfig.c
index 5550194a5d56..61408e19e3a7 100644
--- a/Python/pathconfig.c
+++ b/Python/pathconfig.c
@@ -432,16 +432,11 @@ _PyConfig_InitPathConfig(PyConfig *config)
 static PyStatus
 pathconfig_global_read(_PyPathConfig *pathconfig)
 {
-    PyStatus status;
     PyConfig config;
-
-    status = _PyConfig_InitCompatConfig(&config);
-    if (_PyStatus_EXCEPTION(status)) {
-        goto done;
-    }
+    _PyConfig_InitCompatConfig(&config);
 
     /* Call _PyConfig_InitPathConfig() */
-    status = PyConfig_Read(&config);
+    PyStatus status = PyConfig_Read(&config);
     if (_PyStatus_EXCEPTION(status)) {
         goto done;
     }
diff --git a/Python/pylifecycle.c b/Python/pylifecycle.c
index 019c56c3f8b0..add7519ed0f7 100644
--- a/Python/pylifecycle.c
+++ b/Python/pylifecycle.c
@@ -847,11 +847,7 @@ pyinit_core(_PyRuntimeState *runtime,
     }
 
     PyConfig config;
-
-    status = _PyConfig_InitCompatConfig(&config);
-    if (_PyStatus_EXCEPTION(status)) {
-        goto done;
-    }
+    _PyConfig_InitCompatConfig(&config);
 
     status = _PyConfig_Copy(&config, src_config);
     if (_PyStatus_EXCEPTION(status)) {
@@ -1073,11 +1069,7 @@ Py_InitializeEx(int install_sigs)
     }
 
     PyConfig config;
-
-    status = _PyConfig_InitCompatConfig(&config);
-    if (_PyStatus_EXCEPTION(status)) {
-        Py_ExitStatusException(status);
-    }
+    _PyConfig_InitCompatConfig(&config);
 
     config.install_signal_handlers = install_sigs;
 
diff --git a/Python/pystate.c b/Python/pystate.c
index 69b86ce4f7a3..f3d89e7d2b25 100644
--- a/Python/pystate.c
+++ b/Python/pystate.c
@@ -205,14 +205,7 @@ PyInterpreterState_New(void)
     memset(interp, 0, sizeof(*interp));
     interp->id_refcount = -1;
 
-    PyStatus status = PyConfig_InitPythonConfig(&interp->config);
-    if (_PyStatus_EXCEPTION(status)) {
-        /* Don't report status to caller: PyConfig_InitPythonConfig()
-           can only fail with a memory allocation error. */
-        PyConfig_Clear(&interp->config);
-        PyMem_RawFree(interp);
-        return NULL;
-    }
+    PyConfig_InitPythonConfig(&interp->config);
 
     interp->eval_frame = _PyEval_EvalFrameDefault;
 #ifdef HAVE_DLOPEN


From webhook-mailer at python.org  Tue Oct  1 06:26:08 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Tue, 01 Oct 2019 10:26:08 -0000
Subject: [Python-checkins] bpo-38304: PyConfig_InitPythonConfig() cannot
 fail anymore (GH-16509)
Message-ID: <mailman.170.1569925569.32504.python-checkins@python.org>

https://github.com/python/cpython/commit/d49f096cc41f57155efe71cd089c29b38c218488
commit: d49f096cc41f57155efe71cd089c29b38c218488
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-01T03:26:04-07:00
summary:

bpo-38304: PyConfig_InitPythonConfig() cannot fail anymore (GH-16509)


PyConfig_InitPythonConfig() and PyConfig_InitIsolatedConfig() no
longer return PyStatus: they cannot fail anymore.
(cherry picked from commit 8462a4936b3a551dc546a6adea04a70b0a07ca67)

Co-authored-by: Victor Stinner <vstinner at redhat.com>

files:
M Doc/c-api/init_config.rst
M Include/cpython/initconfig.h
M Include/internal/pycore_initconfig.h
M Modules/main.c
M PC/python_uwp.cpp
M Programs/_freeze_importlib.c
M Programs/_testembed.c
M Python/frozenmain.c
M Python/initconfig.c
M Python/pathconfig.c
M Python/pylifecycle.c
M Python/pystate.c

diff --git a/Doc/c-api/init_config.rst b/Doc/c-api/init_config.rst
index 15b8b94d5a749..5e18300471d46 100644
--- a/Doc/c-api/init_config.rst
+++ b/Doc/c-api/init_config.rst
@@ -343,12 +343,12 @@ PyConfig
 
    Structure methods:
 
-   .. c:function:: PyStatus PyConfig_InitPythonConfig(PyConfig *config)
+   .. c:function:: void PyConfig_InitPythonConfig(PyConfig *config)
 
       Initialize configuration with :ref:`Python Configuration
       <init-python-config>`.
 
-   .. c:function:: PyStatus PyConfig_InitIsolatedConfig(PyConfig *config)
+   .. c:function:: void PyConfig_InitIsolatedConfig(PyConfig *config)
 
       Initialize configuration with :ref:`Isolated Configuration
       <init-isolated-conf>`.
@@ -724,12 +724,9 @@ Example setting the program name::
     void init_python(void)
     {
         PyStatus status;
-        PyConfig config;
 
-        status = PyConfig_InitPythonConfig(&config);
-        if (PyStatus_Exception(status)) {
-            goto fail;
-        }
+        PyConfig config;
+        PyConfig_InitPythonConfig(&config);
 
         /* Set the program name. Implicitly preinitialize Python. */
         status = PyConfig_SetString(&config, &config.program_name,
@@ -756,12 +753,9 @@ configuration, and then override some parameters::
     PyStatus init_python(const char *program_name)
     {
         PyStatus status;
-        PyConfig config;
 
-        status = PyConfig_InitPythonConfig(&config);
-        if (PyStatus_Exception(status)) {
-            goto done;
-        }
+        PyConfig config;
+        PyConfig_InitPythonConfig(&config);
 
         /* Set the program name before reading the configuraton
            (decode byte string from the locale encoding).
@@ -843,13 +837,9 @@ Example of customized Python always running in isolated mode::
     int main(int argc, char **argv)
     {
         PyStatus status;
-        PyConfig config;
-
-        status = PyConfig_InitPythonConfig(&config);
-        if (PyStatus_Exception(status)) {
-            goto fail;
-        }
 
+        PyConfig config;
+        PyConfig_InitPythonConfig(&config);
         config.isolated = 1;
 
         /* Decode command line arguments.
@@ -1034,14 +1024,9 @@ phases::
     void init_python(void)
     {
         PyStatus status;
-        PyConfig config;
-
-        status = PyConfig_InitPythonConfig(&config);
-        if (PyStatus_Exception(status)) {
-            PyConfig_Clear(&config);
-            Py_ExitStatusException(status);
-        }
 
+        PyConfig config;
+        PyConfig_InitPythonConfig(&config);
         config._init_main = 0;
 
         /* ... customize 'config' configuration ... */
diff --git a/Include/cpython/initconfig.h b/Include/cpython/initconfig.h
index 25e02cea10dc3..4b5ceafe02dbc 100644
--- a/Include/cpython/initconfig.h
+++ b/Include/cpython/initconfig.h
@@ -404,8 +404,8 @@ typedef struct {
     int _init_main;
 } PyConfig;
 
-PyAPI_FUNC(PyStatus) PyConfig_InitPythonConfig(PyConfig *config);
-PyAPI_FUNC(PyStatus) PyConfig_InitIsolatedConfig(PyConfig *config);
+PyAPI_FUNC(void) PyConfig_InitPythonConfig(PyConfig *config);
+PyAPI_FUNC(void) PyConfig_InitIsolatedConfig(PyConfig *config);
 PyAPI_FUNC(void) PyConfig_Clear(PyConfig *);
 PyAPI_FUNC(PyStatus) PyConfig_SetString(
     PyConfig *config,
diff --git a/Include/internal/pycore_initconfig.h b/Include/internal/pycore_initconfig.h
index e9810bcf90a85..40831c44b2fa6 100644
--- a/Include/internal/pycore_initconfig.h
+++ b/Include/internal/pycore_initconfig.h
@@ -144,7 +144,7 @@ typedef enum {
     _PyConfig_INIT_ISOLATED = 3
 } _PyConfigInitEnum;
 
-PyAPI_FUNC(PyStatus) _PyConfig_InitCompatConfig(PyConfig *config);
+PyAPI_FUNC(void) _PyConfig_InitCompatConfig(PyConfig *config);
 extern PyStatus _PyConfig_Copy(
     PyConfig *config,
     const PyConfig *config2);
diff --git a/Modules/main.c b/Modules/main.c
index 66af89f561221..2a360b58efa83 100644
--- a/Modules/main.c
+++ b/Modules/main.c
@@ -61,11 +61,7 @@ pymain_init(const _PyArgv *args)
     }
 
     PyConfig config;
-
-    status = PyConfig_InitPythonConfig(&config);
-    if (_PyStatus_EXCEPTION(status)) {
-        goto done;
-    }
+    PyConfig_InitPythonConfig(&config);
 
     /* pass NULL as the config: config is read from command line arguments,
        environment variables, configuration files */
diff --git a/PC/python_uwp.cpp b/PC/python_uwp.cpp
index 758c8f73725e7..88369e8fbfeb3 100644
--- a/PC/python_uwp.cpp
+++ b/PC/python_uwp.cpp
@@ -193,10 +193,7 @@ wmain(int argc, wchar_t **argv)
         }
     }
 
-    status = PyConfig_InitPythonConfig(&config);
-    if (PyStatus_Exception(status)) {
-        goto fail_without_config;
-    }
+    PyConfig_InitPythonConfig(&config);
 
     status = PyConfig_SetArgv(&config, argc, argv);
     if (PyStatus_Exception(status)) {
diff --git a/Programs/_freeze_importlib.c b/Programs/_freeze_importlib.c
index 74735f279c585..2e4ccbb154a41 100644
--- a/Programs/_freeze_importlib.c
+++ b/Programs/_freeze_importlib.c
@@ -76,17 +76,12 @@ main(int argc, char *argv[])
     }
     text[text_size] = '\0';
 
-    PyStatus status;
     PyConfig config;
-
-    status = PyConfig_InitIsolatedConfig(&config);
-    if (PyStatus_Exception(status)) {
-        PyConfig_Clear(&config);
-        Py_ExitStatusException(status);
-    }
+    PyConfig_InitIsolatedConfig(&config);
 
     config.site_import = 0;
 
+    PyStatus status;
     status = PyConfig_SetString(&config, &config.program_name,
                                 L"./_freeze_importlib");
     if (PyStatus_Exception(status)) {
diff --git a/Programs/_testembed.c b/Programs/_testembed.c
index d2d4c7f8c651c..b98a38a1ba678 100644
--- a/Programs/_testembed.c
+++ b/Programs/_testembed.c
@@ -394,11 +394,7 @@ static int check_init_compat_config(int preinit)
     }
 
     PyConfig config;
-
-    status = _PyConfig_InitCompatConfig(&config);
-    if (PyStatus_Exception(status)) {
-        Py_ExitStatusException(status);
-    }
+    _PyConfig_InitCompatConfig(&config);
 
     config_set_program_name(&config);
     init_from_config_clear(&config);
@@ -488,11 +484,8 @@ static int test_init_from_config(void)
     }
 
     PyConfig config;
+    _PyConfig_InitCompatConfig(&config);
 
-    status = _PyConfig_InitCompatConfig(&config);
-    if (PyStatus_Exception(status)) {
-        Py_ExitStatusException(status);
-    }
     config.install_signal_handlers = 0;
 
     /* FIXME: test use_environment */
@@ -621,14 +614,8 @@ static int test_init_from_config(void)
 
 static int check_init_parse_argv(int parse_argv)
 {
-    PyStatus status;
-
     PyConfig config;
-
-    status = PyConfig_InitPythonConfig(&config);
-    if (PyStatus_Exception(status)) {
-        Py_ExitStatusException(status);
-    }
+    PyConfig_InitPythonConfig(&config);
 
     config.parse_argv = parse_argv;
 
@@ -705,16 +692,10 @@ static int test_init_compat_env(void)
 
 static int test_init_python_env(void)
 {
-    PyStatus status;
-
     set_all_env_vars();
 
     PyConfig config;
-
-    status = PyConfig_InitPythonConfig(&config);
-    if (PyStatus_Exception(status)) {
-        Py_ExitStatusException(status);
-    }
+    PyConfig_InitPythonConfig(&config);
 
     config_set_program_name(&config);
     init_from_config_clear(&config);
@@ -760,15 +741,9 @@ static int test_init_env_dev_mode_alloc(void)
 
 static int test_init_isolated_flag(void)
 {
-    PyStatus status;
-
     /* Test PyConfig.isolated=1 */
     PyConfig config;
-
-    status = PyConfig_InitPythonConfig(&config);
-    if (PyStatus_Exception(status)) {
-        Py_ExitStatusException(status);
-    }
+    PyConfig_InitPythonConfig(&config);
 
     Py_IsolatedFlag = 0;
     config.isolated = 1;
@@ -797,11 +772,8 @@ static int test_preinit_isolated1(void)
     }
 
     PyConfig config;
+    _PyConfig_InitCompatConfig(&config);
 
-    status = _PyConfig_InitCompatConfig(&config);
-    if (PyStatus_Exception(status)) {
-        Py_ExitStatusException(status);
-    }
     config_set_program_name(&config);
     set_all_env_vars();
     init_from_config_clear(&config);
@@ -827,11 +799,7 @@ static int test_preinit_isolated2(void)
 
     /* Test PyConfig.isolated=1 */
     PyConfig config;
-
-    status = _PyConfig_InitCompatConfig(&config);
-    if (PyStatus_Exception(status)) {
-        Py_ExitStatusException(status);
-    }
+    _PyConfig_InitCompatConfig(&config);
 
     Py_IsolatedFlag = 0;
     config.isolated = 1;
@@ -867,12 +835,7 @@ static int test_preinit_dont_parse_argv(void)
     }
 
     PyConfig config;
-
-    status = PyConfig_InitIsolatedConfig(&config);
-    if (PyStatus_Exception(status)) {
-        PyConfig_Clear(&config);
-        Py_ExitStatusException(status);
-    }
+    PyConfig_InitIsolatedConfig(&config);
 
     config.isolated = 0;
 
@@ -890,14 +853,8 @@ static int test_preinit_dont_parse_argv(void)
 
 static int test_preinit_parse_argv(void)
 {
-    PyStatus status;
     PyConfig config;
-
-    status = PyConfig_InitPythonConfig(&config);
-    if (PyStatus_Exception(status)) {
-        PyConfig_Clear(&config);
-        Py_ExitStatusException(status);
-    }
+    PyConfig_InitPythonConfig(&config);
 
     /* Pre-initialize implicitly using argv: make sure that -X dev
        is used to configure the allocation in preinitialization */
@@ -962,12 +919,8 @@ static int check_preinit_isolated_config(int preinit)
     }
 
     PyConfig config;
+    PyConfig_InitIsolatedConfig(&config);
 
-    status = PyConfig_InitIsolatedConfig(&config);
-    if (PyStatus_Exception(status)) {
-        PyConfig_Clear(&config);
-        Py_ExitStatusException(status);
-    }
     config_set_program_name(&config);
     init_from_config_clear(&config);
 
@@ -995,8 +948,6 @@ static int test_init_isolated_config(void)
 
 static int check_init_python_config(int preinit)
 {
-    PyStatus status;
-
     /* global configuration variables must be ignored */
     set_all_global_config_variables();
     Py_IsolatedFlag = 1;
@@ -1014,18 +965,15 @@ static int check_init_python_config(int preinit)
         PyPreConfig preconfig;
         PyPreConfig_InitPythonConfig(&preconfig);
 
-        status = Py_PreInitialize(&preconfig);
+        PyStatus status = Py_PreInitialize(&preconfig);
         if (PyStatus_Exception(status)) {
             Py_ExitStatusException(status);
         }
     }
 
     PyConfig config;
+    PyConfig_InitPythonConfig(&config);
 
-    status = PyConfig_InitPythonConfig(&config);
-    if (PyStatus_Exception(status)) {
-        Py_ExitStatusException(status);
-    }
     config_set_program_name(&config);
     init_from_config_clear(&config);
 
@@ -1062,11 +1010,8 @@ static int test_init_dont_configure_locale(void)
     }
 
     PyConfig config;
+    PyConfig_InitPythonConfig(&config);
 
-    status = PyConfig_InitPythonConfig(&config);
-    if (PyStatus_Exception(status)) {
-        Py_ExitStatusException(status);
-    }
     config_set_program_name(&config);
     init_from_config_clear(&config);
 
@@ -1078,13 +1023,9 @@ static int test_init_dont_configure_locale(void)
 
 static int test_init_dev_mode(void)
 {
-    PyStatus status;
     PyConfig config;
+    PyConfig_InitPythonConfig(&config);
 
-    status = PyConfig_InitPythonConfig(&config);
-    if (PyStatus_Exception(status)) {
-        Py_ExitStatusException(status);
-    }
     putenv("PYTHONFAULTHANDLER=");
     putenv("PYTHONMALLOC=");
     config.dev_mode = 1;
@@ -1302,13 +1243,9 @@ static int test_audit_run_file(void)
 
 static int run_audit_run_test(int argc, wchar_t **argv, void *test)
 {
-    PyStatus status;
     PyConfig config;
+    PyConfig_InitPythonConfig(&config);
 
-    status = PyConfig_InitPythonConfig(&config);
-    if (PyStatus_Exception(status)) {
-        Py_ExitStatusException(status);
-    }
     config.argv.length = argc;
     config.argv.items = argv;
     config.parse_argv = 1;
@@ -1320,7 +1257,7 @@ static int run_audit_run_test(int argc, wchar_t **argv, void *test)
 
     PySys_AddAuditHook(_audit_hook_run, test);
 
-    status = Py_InitializeFromConfig(&config);
+    PyStatus status = Py_InitializeFromConfig(&config);
     if (PyStatus_Exception(status)) {
         Py_ExitStatusException(status);
     }
@@ -1353,11 +1290,7 @@ static int test_init_read_set(void)
 {
     PyStatus status;
     PyConfig config;
-
-    status = PyConfig_InitPythonConfig(&config);
-    if (PyStatus_Exception(status)) {
-        Py_ExitStatusException(status);
-    }
+    PyConfig_InitPythonConfig(&config);
 
     status = PyConfig_SetBytesString(&config, &config.program_name,
                                      "./init_read_set");
@@ -1402,12 +1335,7 @@ static int test_init_sys_add(void)
     PySys_AddWarnOption(L"ignore:::sysadd_warnoption");
 
     PyConfig config;
-
-    PyStatus status;
-    status = PyConfig_InitPythonConfig(&config);
-    if (PyStatus_Exception(status)) {
-        goto fail;
-    }
+    PyConfig_InitPythonConfig(&config);
 
     wchar_t* argv[] = {
         L"python3",
@@ -1419,6 +1347,7 @@ static int test_init_sys_add(void)
     config_set_argv(&config, Py_ARRAY_LENGTH(argv), argv);
     config.parse_argv = 1;
 
+    PyStatus status;
     status = PyWideStringList_Append(&config.xoptions,
                                      L"config_xoption");
     if (PyStatus_Exception(status)) {
@@ -1494,11 +1423,8 @@ static int test_init_setpath_config(void)
     putenv("TESTPATH=");
 
     PyConfig config;
+    PyConfig_InitPythonConfig(&config);
 
-    status = PyConfig_InitPythonConfig(&config);
-    if (PyStatus_Exception(status)) {
-        Py_ExitStatusException(status);
-    }
     config_set_string(&config, &config.program_name, L"conf_program_name");
     config_set_string(&config, &config.executable, L"conf_executable");
     init_from_config_clear(&config);
@@ -1534,24 +1460,20 @@ static int test_init_setpythonhome(void)
 
 static int test_init_warnoptions(void)
 {
-    PyStatus status;
     putenv("PYTHONWARNINGS=ignore:::env1,ignore:::env2");
 
     PySys_AddWarnOption(L"ignore:::PySys_AddWarnOption1");
     PySys_AddWarnOption(L"ignore:::PySys_AddWarnOption2");
 
     PyConfig config;
-
-    status = PyConfig_InitPythonConfig(&config);
-    if (PyStatus_Exception(status)) {
-        Py_ExitStatusException(status);
-    }
+    PyConfig_InitPythonConfig(&config);
 
     config.dev_mode = 1;
     config.bytes_warning = 1;
 
     config_set_program_name(&config);
 
+    PyStatus status;
     status = PyWideStringList_Append(&config.warnoptions,
                                      L"ignore:::PyConfig_BeforeRead");
     if (PyStatus_Exception(status)) {
@@ -1606,13 +1528,9 @@ static void configure_init_main(PyConfig *config)
 
 static int test_init_run_main(void)
 {
-    PyStatus status;
     PyConfig config;
+    PyConfig_InitPythonConfig(&config);
 
-    status = PyConfig_InitPythonConfig(&config);
-    if (PyStatus_Exception(status)) {
-        Py_ExitStatusException(status);
-    }
     configure_init_main(&config);
     init_from_config_clear(&config);
 
@@ -1622,13 +1540,9 @@ static int test_init_run_main(void)
 
 static int test_init_main(void)
 {
-    PyStatus status;
     PyConfig config;
+    PyConfig_InitPythonConfig(&config);
 
-    status = PyConfig_InitPythonConfig(&config);
-    if (PyStatus_Exception(status)) {
-        Py_ExitStatusException(status);
-    }
     configure_init_main(&config);
     config._init_main = 0;
     init_from_config_clear(&config);
@@ -1642,7 +1556,7 @@ static int test_init_main(void)
         exit(1);
     }
 
-    status = _Py_InitializeMain();
+    PyStatus status = _Py_InitializeMain();
     if (PyStatus_Exception(status)) {
         Py_ExitStatusException(status);
     }
@@ -1653,14 +1567,8 @@ static int test_init_main(void)
 
 static int test_run_main(void)
 {
-    PyStatus status;
     PyConfig config;
-
-    status = PyConfig_InitPythonConfig(&config);
-    if (PyStatus_Exception(status)) {
-        PyConfig_Clear(&config);
-        Py_ExitStatusException(status);
-    }
+    PyConfig_InitPythonConfig(&config);
 
     wchar_t *argv[] = {L"python3", L"-c",
                        (L"import sys; "
diff --git a/Python/frozenmain.c b/Python/frozenmain.c
index c56938ab48994..7f9cc19317360 100644
--- a/Python/frozenmain.c
+++ b/Python/frozenmain.c
@@ -40,11 +40,7 @@ Py_FrozenMain(int argc, char **argv)
     }
 
     PyConfig config;
-    status = PyConfig_InitPythonConfig(&config);
-    if (PyStatus_Exception(status)) {
-        PyConfig_Clear(&config);
-        Py_ExitStatusException(status);
-    }
+    PyConfig_InitPythonConfig(&config);
     config.pathconfig_warnings = 0;   /* Suppress errors from getpath.c */
 
     if ((p = Py_GETENV("PYTHONINSPECT")) && *p != '\0')
diff --git a/Python/initconfig.c b/Python/initconfig.c
index 9164c7cfc8e85..a41a3292d3e2c 100644
--- a/Python/initconfig.c
+++ b/Python/initconfig.c
@@ -569,7 +569,7 @@ PyConfig_Clear(PyConfig *config)
 }
 
 
-PyStatus
+void
 _PyConfig_InitCompatConfig(PyConfig *config)
 {
     memset(config, 0, sizeof(*config));
@@ -603,17 +603,13 @@ _PyConfig_InitCompatConfig(PyConfig *config)
 #ifdef MS_WINDOWS
     config->legacy_windows_stdio = -1;
 #endif
-    return _PyStatus_OK();
 }
 
 
-static PyStatus
+static void
 config_init_defaults(PyConfig *config)
 {
-    PyStatus status = _PyConfig_InitCompatConfig(config);
-    if (_PyStatus_EXCEPTION(status)) {
-        return status;
-    }
+    _PyConfig_InitCompatConfig(config);
 
     config->isolated = 0;
     config->use_environment = 1;
@@ -632,35 +628,24 @@ config_init_defaults(PyConfig *config)
 #ifdef MS_WINDOWS
     config->legacy_windows_stdio = 0;
 #endif
-    return _PyStatus_OK();
 }
 
 
-PyStatus
+void
 PyConfig_InitPythonConfig(PyConfig *config)
 {
-    PyStatus status = config_init_defaults(config);
-    if (_PyStatus_EXCEPTION(status)) {
-        _PyStatus_UPDATE_FUNC(status);
-        return status;
-    }
+    config_init_defaults(config);
 
     config->_config_init = (int)_PyConfig_INIT_PYTHON;
     config->configure_c_stdio = 1;
     config->parse_argv = 1;
-
-    return _PyStatus_OK();
 }
 
 
-PyStatus
+void
 PyConfig_InitIsolatedConfig(PyConfig *config)
 {
-    PyStatus status = config_init_defaults(config);
-    if (_PyStatus_EXCEPTION(status)) {
-        _PyStatus_UPDATE_FUNC(status);
-        return status;
-    }
+    config_init_defaults(config);
 
     config->_config_init = (int)_PyConfig_INIT_ISOLATED;
     config->isolated = 1;
@@ -675,8 +660,6 @@ PyConfig_InitIsolatedConfig(PyConfig *config)
 #ifdef MS_WINDOWS
     config->legacy_windows_stdio = 0;
 #endif
-
-    return _PyStatus_OK();
 }
 
 
diff --git a/Python/pathconfig.c b/Python/pathconfig.c
index ff4bbf2a97ebe..7f3fdcc103f7b 100644
--- a/Python/pathconfig.c
+++ b/Python/pathconfig.c
@@ -432,16 +432,11 @@ _PyConfig_InitPathConfig(PyConfig *config)
 static PyStatus
 pathconfig_global_read(_PyPathConfig *pathconfig)
 {
-    PyStatus status;
     PyConfig config;
-
-    status = _PyConfig_InitCompatConfig(&config);
-    if (_PyStatus_EXCEPTION(status)) {
-        goto done;
-    }
+    _PyConfig_InitCompatConfig(&config);
 
     /* Call _PyConfig_InitPathConfig() */
-    status = PyConfig_Read(&config);
+    PyStatus status = PyConfig_Read(&config);
     if (_PyStatus_EXCEPTION(status)) {
         goto done;
     }
diff --git a/Python/pylifecycle.c b/Python/pylifecycle.c
index 5174f749c68d6..feb9285239240 100644
--- a/Python/pylifecycle.c
+++ b/Python/pylifecycle.c
@@ -843,11 +843,7 @@ pyinit_core(_PyRuntimeState *runtime,
     }
 
     PyConfig config;
-
-    status = _PyConfig_InitCompatConfig(&config);
-    if (_PyStatus_EXCEPTION(status)) {
-        goto done;
-    }
+    _PyConfig_InitCompatConfig(&config);
 
     status = _PyConfig_Copy(&config, src_config);
     if (_PyStatus_EXCEPTION(status)) {
@@ -1070,11 +1066,7 @@ Py_InitializeEx(int install_sigs)
     }
 
     PyConfig config;
-
-    status = _PyConfig_InitCompatConfig(&config);
-    if (_PyStatus_EXCEPTION(status)) {
-        Py_ExitStatusException(status);
-    }
+    _PyConfig_InitCompatConfig(&config);
 
     config.install_signal_handlers = install_sigs;
 
diff --git a/Python/pystate.c b/Python/pystate.c
index 098280d76dfc0..aba673c00a432 100644
--- a/Python/pystate.c
+++ b/Python/pystate.c
@@ -206,14 +206,7 @@ PyInterpreterState_New(void)
     interp->id_refcount = -1;
     interp->check_interval = 100;
 
-    PyStatus status = PyConfig_InitPythonConfig(&interp->config);
-    if (_PyStatus_EXCEPTION(status)) {
-        /* Don't report status to caller: PyConfig_InitPythonConfig()
-           can only fail with a memory allocation error. */
-        PyConfig_Clear(&interp->config);
-        PyMem_RawFree(interp);
-        return NULL;
-    }
+    PyConfig_InitPythonConfig(&interp->config);
 
     interp->eval_frame = _PyEval_EvalFrameDefault;
 #ifdef HAVE_DLOPEN


From webhook-mailer at python.org  Tue Oct  1 06:29:40 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Tue, 01 Oct 2019 10:29:40 -0000
Subject: [Python-checkins] bpo-36670: Multiple regrtest bugfixes (GH-16511)
Message-ID: <mailman.171.1569925780.32504.python-checkins@python.org>

https://github.com/python/cpython/commit/982bfa4da07b2e5749a0f4e68f99e972bcc3a549
commit: 982bfa4da07b2e5749a0f4e68f99e972bcc3a549
branch: master
author: Victor Stinner <vstinner at redhat.com>
committer: GitHub <noreply at github.com>
date: 2019-10-01T12:29:36+02:00
summary:

bpo-36670: Multiple regrtest bugfixes (GH-16511)

* Windows: Fix counter name in WindowsLoadTracker. Counter names are
  localized: use the registry to get the counter name. Original
  change written by Lorenz Mende.
* Regrtest.main() now ensures that the Windows load tracker is also
  killed if an exception is raised
* TestWorkerProcess now ensures that worker processes are no longer
  running before exiting: kill also worker processes when an
  exception is raised.
* Enhance regrtest messages and warnings: include test name,
  duration, add a worker identifier, etc.
* Rename MultiprocessRunner to TestWorkerProcess
* Use print_warning() to display warnings.

Co-Authored-By: Lorenz Mende <Lorenz.mende at gmail.com>

files:
M Lib/test/libregrtest/main.py
M Lib/test/libregrtest/runtest_mp.py
M Lib/test/libregrtest/win_utils.py

diff --git a/Lib/test/libregrtest/main.py b/Lib/test/libregrtest/main.py
index 299700686928f..fd701c452ceb5 100644
--- a/Lib/test/libregrtest/main.py
+++ b/Lib/test/libregrtest/main.py
@@ -508,10 +508,6 @@ def run_tests(self):
             self.run_tests_sequential()
 
     def finalize(self):
-        if self.win_load_tracker is not None:
-            self.win_load_tracker.close()
-            self.win_load_tracker = None
-
         if self.next_single_filename:
             if self.next_single_test:
                 with open(self.next_single_filename, 'w') as fp:
@@ -680,11 +676,16 @@ def _main(self, tests, kwargs):
                 # typeperf.exe for x64, x86 or ARM
                 print(f'Failed to create WindowsLoadTracker: {error}')
 
-        self.run_tests()
-        self.display_result()
-
-        if self.ns.verbose2 and self.bad:
-            self.rerun_failed_tests()
+        try:
+            self.run_tests()
+            self.display_result()
+
+            if self.ns.verbose2 and self.bad:
+                self.rerun_failed_tests()
+        finally:
+            if self.win_load_tracker is not None:
+                self.win_load_tracker.close()
+                self.win_load_tracker = None
 
         self.finalize()
 
diff --git a/Lib/test/libregrtest/runtest_mp.py b/Lib/test/libregrtest/runtest_mp.py
index 9cb5be6bb8ae8..38b05781de5fc 100644
--- a/Lib/test/libregrtest/runtest_mp.py
+++ b/Lib/test/libregrtest/runtest_mp.py
@@ -15,7 +15,7 @@
     runtest, INTERRUPTED, CHILD_ERROR, PROGRESS_MIN_TIME,
     format_test_result, TestResult, is_failed, TIMEOUT)
 from test.libregrtest.setup import setup_tests
-from test.libregrtest.utils import format_duration
+from test.libregrtest.utils import format_duration, print_warning
 
 
 # Display the running tests if nothing happened last N seconds
@@ -103,9 +103,10 @@ class ExitThread(Exception):
     pass
 
 
-class MultiprocessThread(threading.Thread):
-    def __init__(self, pending, output, ns, timeout):
+class TestWorkerProcess(threading.Thread):
+    def __init__(self, worker_id, pending, output, ns, timeout):
         super().__init__()
+        self.worker_id = worker_id
         self.pending = pending
         self.output = output
         self.ns = ns
@@ -114,12 +115,16 @@ def __init__(self, pending, output, ns, timeout):
         self.start_time = None
         self._popen = None
         self._killed = False
+        self._stopped = False
 
     def __repr__(self):
-        info = ['MultiprocessThread']
-        test = self.current_test_name
+        info = [f'TestWorkerProcess #{self.worker_id}']
         if self.is_alive():
-            info.append('alive')
+            dt = time.monotonic() - self.start_time
+            info.append("running for %s" % format_duration(dt))
+        else:
+            info.append('stopped')
+        test = self.current_test_name
         if test:
             info.append(f'test={test}')
         popen = self._popen
@@ -128,53 +133,24 @@ def __repr__(self):
         return '<%s>' % ' '.join(info)
 
     def _kill(self):
-        dt = time.monotonic() - self.start_time
+        if self._killed:
+            return
+        self._killed = True
 
         popen = self._popen
-        pid = popen.pid
-        print("Kill worker process %s running for %.1f sec" % (pid, dt),
-              file=sys.stderr, flush=True)
+        if popen is None:
+            return
 
+        print(f"Kill {self}", file=sys.stderr, flush=True)
         try:
             popen.kill()
-            return True
         except OSError as exc:
-            print("WARNING: Failed to kill worker process %s: %r" % (pid, exc),
-                  file=sys.stderr, flush=True)
-            return False
-
-    def _close_wait(self):
-        popen = self._popen
-
-        # stdout and stderr must be closed to ensure that communicate()
-        # does not hang
-        popen.stdout.close()
-        popen.stderr.close()
-
-        try:
-            popen.wait(JOIN_TIMEOUT)
-        except (subprocess.TimeoutExpired, OSError) as exc:
-            print("WARNING: Failed to wait for worker process %s "
-                  "completion (timeout=%.1f sec): %r"
-                  % (popen.pid, JOIN_TIMEOUT, exc),
-                  file=sys.stderr, flush=True)
-
-    def kill(self):
-        """
-        Kill the current process (if any).
-
-        This method can be called by the thread running the process,
-        or by another thread.
-        """
-        self._killed = True
-
-        if self._popen is None:
-            return
-
-        if not self._kill():
-            return
+            print_warning(f"Failed to kill {self}: {exc!r}")
 
-        self._close_wait()
+    def stop(self):
+        # Method called from a different thread to stop this thread
+        self._stopped = True
+        self._kill()
 
     def mp_result_error(self, test_name, error_type, stdout='', stderr='',
                         err_msg=None):
@@ -190,59 +166,69 @@ def _timedout(self, test_name):
         try:
             stdout, stderr = popen.communicate(timeout=JOIN_TIMEOUT)
         except (subprocess.TimeoutExpired, OSError) as exc:
-            print("WARNING: Failed to read worker process %s output "
-                  "(timeout=%.1f sec): %r"
-                  % (popen.pid, JOIN_TIMEOUT, exc),
-                  file=sys.stderr, flush=True)
-
-        self._close_wait()
+            print_warning(f"Failed to read {self} output "
+                          f"(timeout={format_duration(JOIN_TIMEOUT)}): "
+                          f"{exc!r}")
 
         return self.mp_result_error(test_name, TIMEOUT, stdout, stderr)
 
-    def _runtest(self, test_name):
-        try:
-            self.start_time = time.monotonic()
-            self.current_test_name = test_name
+    def _run_process(self, test_name):
+        self.start_time = time.monotonic()
 
+        self.current_test_name = test_name
+        try:
+            self._killed = False
             self._popen = run_test_in_subprocess(test_name, self.ns)
             popen = self._popen
+        except:
+            self.current_test_name = None
+            raise
+
+        try:
+            if self._stopped:
+                # If kill() has been called before self._popen is set,
+                # self._popen is still running. Call again kill()
+                # to ensure that the process is killed.
+                self._kill()
+                raise ExitThread
+
             try:
-                try:
-                    if self._killed:
-                        # If kill() has been called before self._popen is set,
-                        # self._popen is still running. Call again kill()
-                        # to ensure that the process is killed.
-                        self.kill()
-                        raise ExitThread
-
-                    try:
-                        stdout, stderr = popen.communicate(timeout=self.timeout)
-                    except subprocess.TimeoutExpired:
-                        if self._killed:
-                            # kill() has been called: communicate() fails
-                            # on reading closed stdout/stderr
-                            raise ExitThread
-
-                        return self._timedout(test_name)
-                    except OSError:
-                        if self._killed:
-                            # kill() has been called: communicate() fails
-                            # on reading closed stdout/stderr
-                            raise ExitThread
-                        raise
-                except:
-                    self.kill()
-                    raise
-            finally:
-                self._close_wait()
+                stdout, stderr = popen.communicate(timeout=self.timeout)
+            except subprocess.TimeoutExpired:
+                if self._stopped:
+                    # kill() has been called: communicate() fails
+                    # on reading closed stdout/stderr
+                    raise ExitThread
+
+                return self._timedout(test_name)
+            except OSError:
+                if self._stopped:
+                    # kill() has been called: communicate() fails
+                    # on reading closed stdout/stderr
+                    raise ExitThread
+                raise
 
             retcode = popen.returncode
+            stdout = stdout.strip()
+            stderr = stderr.rstrip()
+
+            return (retcode, stdout, stderr)
+        except:
+            self._kill()
+            raise
         finally:
-            self.current_test_name = None
+            self._wait_completed()
             self._popen = None
+            self.current_test_name = None
+
+    def _runtest(self, test_name):
+        result = self._run_process(test_name)
 
-        stdout = stdout.strip()
-        stderr = stderr.rstrip()
+        if isinstance(result, MultiprocessResult):
+            # _timedout() case
+            return result
+
+        retcode, stdout, stderr = result
 
         err_msg = None
         if retcode != 0:
@@ -266,7 +252,7 @@ def _runtest(self, test_name):
         return MultiprocessResult(result, stdout, stderr, err_msg)
 
     def run(self):
-        while not self._killed:
+        while not self._stopped:
             try:
                 try:
                     test_name = next(self.pending)
@@ -284,6 +270,33 @@ def run(self):
                 self.output.put((True, traceback.format_exc()))
                 break
 
+    def _wait_completed(self):
+        popen = self._popen
+
+        # stdout and stderr must be closed to ensure that communicate()
+        # does not hang
+        popen.stdout.close()
+        popen.stderr.close()
+
+        try:
+            popen.wait(JOIN_TIMEOUT)
+        except (subprocess.TimeoutExpired, OSError) as exc:
+            print_warning(f"Failed to wait for {self} completion "
+                          f"(timeout={format_duration(JOIN_TIMEOUT)}): "
+                          f"{exc!r}")
+
+    def wait_stopped(self, start_time):
+        while True:
+            # Write a message every second
+            self.join(1.0)
+            if not self.is_alive():
+                break
+            dt = time.monotonic() - start_time
+            print(f"Waiting for {self} thread for {format_duration(dt)}", flush=True)
+            if dt > JOIN_TIMEOUT:
+                print_warning(f"Failed to join {self} in {format_duration(dt)}")
+                break
+
 
 def get_running(workers):
     running = []
@@ -298,7 +311,7 @@ def get_running(workers):
     return running
 
 
-class MultiprocessRunner:
+class MultiprocessTestRunner:
     def __init__(self, regrtest):
         self.regrtest = regrtest
         self.ns = regrtest.ns
@@ -311,30 +324,20 @@ def __init__(self, regrtest):
         self.workers = None
 
     def start_workers(self):
-        self.workers = [MultiprocessThread(self.pending, self.output,
-                                           self.ns, self.worker_timeout)
-                        for _ in range(self.ns.use_mp)]
+        self.workers = [TestWorkerProcess(index, self.pending, self.output,
+                                          self.ns, self.worker_timeout)
+                        for index in range(1, self.ns.use_mp + 1)]
         print("Run tests in parallel using %s child processes"
               % len(self.workers))
         for worker in self.workers:
             worker.start()
 
-    def wait_workers(self):
+    def stop_workers(self):
         start_time = time.monotonic()
         for worker in self.workers:
-            worker.kill()
+            worker.stop()
         for worker in self.workers:
-            while True:
-                worker.join(1.0)
-                if not worker.is_alive():
-                    break
-                dt = time.monotonic() - start_time
-                print("Wait for regrtest worker %r for %.1f sec" % (worker, dt),
-                      flush=True)
-                if dt > JOIN_TIMEOUT:
-                    print("Warning -- failed to join a regrtest worker %s"
-                          % worker, flush=True)
-                    break
+            worker.wait_stopped(start_time)
 
     def _get_result(self):
         if not any(worker.is_alive() for worker in self.workers):
@@ -418,10 +421,11 @@ def run_tests(self):
             if self.ns.timeout is not None:
                 faulthandler.cancel_dump_traceback_later()
 
-        # a test failed (and --failfast is set) or all tests completed
-        self.pending.stop()
-        self.wait_workers()
+            # Always ensure that all worker processes are no longer
+            # worker when we exit this function
+            self.pending.stop()
+            self.stop_workers()
 
 
 def run_tests_multiprocess(regrtest):
-    MultiprocessRunner(regrtest).run_tests()
+    MultiprocessTestRunner(regrtest).run_tests()
diff --git a/Lib/test/libregrtest/win_utils.py b/Lib/test/libregrtest/win_utils.py
index ec2d6c663e83f..f0c17b906f519 100644
--- a/Lib/test/libregrtest/win_utils.py
+++ b/Lib/test/libregrtest/win_utils.py
@@ -3,16 +3,22 @@
 import os
 import subprocess
 import uuid
+import winreg
 from test import support
+from test.libregrtest.utils import print_warning
 
 
 # Max size of asynchronous reads
 BUFSIZE = 8192
 # Exponential damping factor (see below)
 LOAD_FACTOR_1 = 0.9200444146293232478931553241
+
 # Seconds per measurement
 SAMPLING_INTERVAL = 5
-COUNTER_NAME = r'\System\Processor Queue Length'
+# Windows registry subkey of HKEY_LOCAL_MACHINE where the counter names
+# of typeperf are registered
+COUNTER_REGISTRY_KEY = (r"SOFTWARE\Microsoft\Windows NT\CurrentVersion"
+                        r"\Perflib\CurrentLanguage")
 
 
 class WindowsLoadTracker():
@@ -25,7 +31,8 @@ class WindowsLoadTracker():
 
     def __init__(self):
         self.load = 0.0
-        self.p = None
+        self.counter_name = ''
+        self.popen = None
         self.start()
 
     def start(self):
@@ -55,31 +62,46 @@ def start(self):
         overlap.GetOverlappedResult(True)
 
         # Spawn off the load monitor
-        command = ['typeperf', COUNTER_NAME, '-si', str(SAMPLING_INTERVAL)]
-        self.p = subprocess.Popen(command, stdout=command_stdout, cwd=support.SAVEDCWD)
+        counter_name = self._get_counter_name()
+        command = ['typeperf', counter_name, '-si', str(SAMPLING_INTERVAL)]
+        self.popen = subprocess.Popen(' '.join(command), stdout=command_stdout, cwd=support.SAVEDCWD)
 
         # Close our copy of the write end of the pipe
         os.close(command_stdout)
 
+    def _get_counter_name(self):
+        # accessing the registry to get the counter localization name
+        with winreg.OpenKey(winreg.HKEY_LOCAL_MACHINE, COUNTER_REGISTRY_KEY) as perfkey:
+            counters = winreg.QueryValueEx(perfkey, 'Counter')[0]
+
+        # Convert [key1, value1, key2, value2, ...] list
+        # to {key1: value1, key2: value2, ...} dict
+        counters = iter(counters)
+        counters_dict = dict(zip(counters, counters))
+
+        # System counter has key '2' and Processor Queue Length has key '44'
+        system = counters_dict['2']
+        process_queue_length = counters_dict['44']
+        return f'"\\{system}\\{process_queue_length}"'
+
     def close(self):
-        if self.p is None:
+        if self.popen is None:
             return
-        self.p.kill()
-        self.p.wait()
-        self.p = None
+        self.popen.kill()
+        self.popen.wait()
+        self.popen = None
 
     def __del__(self):
         self.close()
 
     def read_output(self):
-        import _winapi
-
         overlapped, _ = _winapi.ReadFile(self.pipe, BUFSIZE, True)
         bytes_read, res = overlapped.GetOverlappedResult(False)
         if res != 0:
             return
 
-        return overlapped.getbuffer().decode()
+        output = overlapped.getbuffer()
+        return output.decode('oem', 'replace')
 
     def getloadavg(self):
         typeperf_output = self.read_output()
@@ -89,14 +111,29 @@ def getloadavg(self):
 
         # Process the backlog of load values
         for line in typeperf_output.splitlines():
+            # Ignore the initial header:
+            # "(PDH-CSV 4.0)","\\\\WIN\\System\\Processor Queue Length"
+            if '\\\\' in line:
+                continue
+
+            # Ignore blank lines
+            if not line.strip():
+                continue
+
             # typeperf outputs in a CSV format like this:
             # "07/19/2018 01:32:26.605","3.000000"
-            toks = line.split(',')
-            # Ignore blank lines and the initial header
-            if line.strip() == '' or (COUNTER_NAME in line) or len(toks) != 2:
+            # (date, process queue length)
+            try:
+                tokens = line.split(',')
+                if len(tokens) != 2:
+                    raise ValueError
+
+                value = tokens[1].replace('"', '')
+                load = float(value)
+            except ValueError:
+                print_warning("Failed to parse typeperf output: %a" % line)
                 continue
 
-            load = float(toks[1].replace('"', ''))
             # We use an exponentially weighted moving average, imitating the
             # load calculation on Unix systems.
             # https://en.wikipedia.org/wiki/Load_(computing)#Unix-style_load_calculation


From webhook-mailer at python.org  Tue Oct  1 06:45:56 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Tue, 01 Oct 2019 10:45:56 -0000
Subject: [Python-checkins] bpo-38321: Fix compiler warning in
 _randommodule.c (GH-16512)
Message-ID: <mailman.172.1569926757.32504.python-checkins@python.org>

https://github.com/python/cpython/commit/2f90261280e36a179831d72ce794115be31c88bb
commit: 2f90261280e36a179831d72ce794115be31c88bb
branch: master
author: Victor Stinner <vstinner at redhat.com>
committer: GitHub <noreply at github.com>
date: 2019-10-01T12:45:52+02:00
summary:

bpo-38321: Fix compiler warning in _randommodule.c (GH-16512)

Fix the GCC warning: "initialization discards ?const? qualifier from
pointer target type".

files:
M Modules/_randommodule.c

diff --git a/Modules/_randommodule.c b/Modules/_randommodule.c
index 1ea2bf28abcca..1f4bf74fc7a1e 100644
--- a/Modules/_randommodule.c
+++ b/Modules/_randommodule.c
@@ -543,7 +543,7 @@ PyDoc_STRVAR(random_doc,
 "Random() -> create a random number generator with its own internal state.");
 
 static PyType_Slot Random_Type_slots[] = {
-    {Py_tp_doc, random_doc},
+    {Py_tp_doc, (void *)random_doc},
     {Py_tp_methods, random_methods},
     {Py_tp_new, random_new},
     {Py_tp_free, PyObject_Free},


From webhook-mailer at python.org  Tue Oct  1 06:47:56 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Tue, 01 Oct 2019 10:47:56 -0000
Subject: [Python-checkins] bpo-36670: Multiple regrtest bugfixes (GH-16511)
Message-ID: <mailman.173.1569926878.32504.python-checkins@python.org>

https://github.com/python/cpython/commit/6b4c70abcaa348c84a2c1bb6f5a46596cf358e36
commit: 6b4c70abcaa348c84a2c1bb6f5a46596cf358e36
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-01T03:47:52-07:00
summary:

bpo-36670: Multiple regrtest bugfixes (GH-16511)


* Windows: Fix counter name in WindowsLoadTracker. Counter names are
  localized: use the registry to get the counter name. Original
  change written by Lorenz Mende.
* Regrtest.main() now ensures that the Windows load tracker is also
  killed if an exception is raised
* TestWorkerProcess now ensures that worker processes are no longer
  running before exiting: kill also worker processes when an
  exception is raised.
* Enhance regrtest messages and warnings: include test name,
  duration, add a worker identifier, etc.
* Rename MultiprocessRunner to TestWorkerProcess
* Use print_warning() to display warnings.

Co-Authored-By: Lorenz Mende <Lorenz.mende at gmail.com>
(cherry picked from commit 982bfa4da07b2e5749a0f4e68f99e972bcc3a549)

Co-authored-by: Victor Stinner <vstinner at redhat.com>

files:
M Lib/test/libregrtest/main.py
M Lib/test/libregrtest/runtest_mp.py
M Lib/test/libregrtest/win_utils.py

diff --git a/Lib/test/libregrtest/main.py b/Lib/test/libregrtest/main.py
index 299700686928f..fd701c452ceb5 100644
--- a/Lib/test/libregrtest/main.py
+++ b/Lib/test/libregrtest/main.py
@@ -508,10 +508,6 @@ def run_tests(self):
             self.run_tests_sequential()
 
     def finalize(self):
-        if self.win_load_tracker is not None:
-            self.win_load_tracker.close()
-            self.win_load_tracker = None
-
         if self.next_single_filename:
             if self.next_single_test:
                 with open(self.next_single_filename, 'w') as fp:
@@ -680,11 +676,16 @@ def _main(self, tests, kwargs):
                 # typeperf.exe for x64, x86 or ARM
                 print(f'Failed to create WindowsLoadTracker: {error}')
 
-        self.run_tests()
-        self.display_result()
-
-        if self.ns.verbose2 and self.bad:
-            self.rerun_failed_tests()
+        try:
+            self.run_tests()
+            self.display_result()
+
+            if self.ns.verbose2 and self.bad:
+                self.rerun_failed_tests()
+        finally:
+            if self.win_load_tracker is not None:
+                self.win_load_tracker.close()
+                self.win_load_tracker = None
 
         self.finalize()
 
diff --git a/Lib/test/libregrtest/runtest_mp.py b/Lib/test/libregrtest/runtest_mp.py
index 9cb5be6bb8ae8..38b05781de5fc 100644
--- a/Lib/test/libregrtest/runtest_mp.py
+++ b/Lib/test/libregrtest/runtest_mp.py
@@ -15,7 +15,7 @@
     runtest, INTERRUPTED, CHILD_ERROR, PROGRESS_MIN_TIME,
     format_test_result, TestResult, is_failed, TIMEOUT)
 from test.libregrtest.setup import setup_tests
-from test.libregrtest.utils import format_duration
+from test.libregrtest.utils import format_duration, print_warning
 
 
 # Display the running tests if nothing happened last N seconds
@@ -103,9 +103,10 @@ class ExitThread(Exception):
     pass
 
 
-class MultiprocessThread(threading.Thread):
-    def __init__(self, pending, output, ns, timeout):
+class TestWorkerProcess(threading.Thread):
+    def __init__(self, worker_id, pending, output, ns, timeout):
         super().__init__()
+        self.worker_id = worker_id
         self.pending = pending
         self.output = output
         self.ns = ns
@@ -114,12 +115,16 @@ def __init__(self, pending, output, ns, timeout):
         self.start_time = None
         self._popen = None
         self._killed = False
+        self._stopped = False
 
     def __repr__(self):
-        info = ['MultiprocessThread']
-        test = self.current_test_name
+        info = [f'TestWorkerProcess #{self.worker_id}']
         if self.is_alive():
-            info.append('alive')
+            dt = time.monotonic() - self.start_time
+            info.append("running for %s" % format_duration(dt))
+        else:
+            info.append('stopped')
+        test = self.current_test_name
         if test:
             info.append(f'test={test}')
         popen = self._popen
@@ -128,53 +133,24 @@ def __repr__(self):
         return '<%s>' % ' '.join(info)
 
     def _kill(self):
-        dt = time.monotonic() - self.start_time
+        if self._killed:
+            return
+        self._killed = True
 
         popen = self._popen
-        pid = popen.pid
-        print("Kill worker process %s running for %.1f sec" % (pid, dt),
-              file=sys.stderr, flush=True)
+        if popen is None:
+            return
 
+        print(f"Kill {self}", file=sys.stderr, flush=True)
         try:
             popen.kill()
-            return True
         except OSError as exc:
-            print("WARNING: Failed to kill worker process %s: %r" % (pid, exc),
-                  file=sys.stderr, flush=True)
-            return False
-
-    def _close_wait(self):
-        popen = self._popen
-
-        # stdout and stderr must be closed to ensure that communicate()
-        # does not hang
-        popen.stdout.close()
-        popen.stderr.close()
-
-        try:
-            popen.wait(JOIN_TIMEOUT)
-        except (subprocess.TimeoutExpired, OSError) as exc:
-            print("WARNING: Failed to wait for worker process %s "
-                  "completion (timeout=%.1f sec): %r"
-                  % (popen.pid, JOIN_TIMEOUT, exc),
-                  file=sys.stderr, flush=True)
-
-    def kill(self):
-        """
-        Kill the current process (if any).
-
-        This method can be called by the thread running the process,
-        or by another thread.
-        """
-        self._killed = True
-
-        if self._popen is None:
-            return
-
-        if not self._kill():
-            return
+            print_warning(f"Failed to kill {self}: {exc!r}")
 
-        self._close_wait()
+    def stop(self):
+        # Method called from a different thread to stop this thread
+        self._stopped = True
+        self._kill()
 
     def mp_result_error(self, test_name, error_type, stdout='', stderr='',
                         err_msg=None):
@@ -190,59 +166,69 @@ def _timedout(self, test_name):
         try:
             stdout, stderr = popen.communicate(timeout=JOIN_TIMEOUT)
         except (subprocess.TimeoutExpired, OSError) as exc:
-            print("WARNING: Failed to read worker process %s output "
-                  "(timeout=%.1f sec): %r"
-                  % (popen.pid, JOIN_TIMEOUT, exc),
-                  file=sys.stderr, flush=True)
-
-        self._close_wait()
+            print_warning(f"Failed to read {self} output "
+                          f"(timeout={format_duration(JOIN_TIMEOUT)}): "
+                          f"{exc!r}")
 
         return self.mp_result_error(test_name, TIMEOUT, stdout, stderr)
 
-    def _runtest(self, test_name):
-        try:
-            self.start_time = time.monotonic()
-            self.current_test_name = test_name
+    def _run_process(self, test_name):
+        self.start_time = time.monotonic()
 
+        self.current_test_name = test_name
+        try:
+            self._killed = False
             self._popen = run_test_in_subprocess(test_name, self.ns)
             popen = self._popen
+        except:
+            self.current_test_name = None
+            raise
+
+        try:
+            if self._stopped:
+                # If kill() has been called before self._popen is set,
+                # self._popen is still running. Call again kill()
+                # to ensure that the process is killed.
+                self._kill()
+                raise ExitThread
+
             try:
-                try:
-                    if self._killed:
-                        # If kill() has been called before self._popen is set,
-                        # self._popen is still running. Call again kill()
-                        # to ensure that the process is killed.
-                        self.kill()
-                        raise ExitThread
-
-                    try:
-                        stdout, stderr = popen.communicate(timeout=self.timeout)
-                    except subprocess.TimeoutExpired:
-                        if self._killed:
-                            # kill() has been called: communicate() fails
-                            # on reading closed stdout/stderr
-                            raise ExitThread
-
-                        return self._timedout(test_name)
-                    except OSError:
-                        if self._killed:
-                            # kill() has been called: communicate() fails
-                            # on reading closed stdout/stderr
-                            raise ExitThread
-                        raise
-                except:
-                    self.kill()
-                    raise
-            finally:
-                self._close_wait()
+                stdout, stderr = popen.communicate(timeout=self.timeout)
+            except subprocess.TimeoutExpired:
+                if self._stopped:
+                    # kill() has been called: communicate() fails
+                    # on reading closed stdout/stderr
+                    raise ExitThread
+
+                return self._timedout(test_name)
+            except OSError:
+                if self._stopped:
+                    # kill() has been called: communicate() fails
+                    # on reading closed stdout/stderr
+                    raise ExitThread
+                raise
 
             retcode = popen.returncode
+            stdout = stdout.strip()
+            stderr = stderr.rstrip()
+
+            return (retcode, stdout, stderr)
+        except:
+            self._kill()
+            raise
         finally:
-            self.current_test_name = None
+            self._wait_completed()
             self._popen = None
+            self.current_test_name = None
+
+    def _runtest(self, test_name):
+        result = self._run_process(test_name)
 
-        stdout = stdout.strip()
-        stderr = stderr.rstrip()
+        if isinstance(result, MultiprocessResult):
+            # _timedout() case
+            return result
+
+        retcode, stdout, stderr = result
 
         err_msg = None
         if retcode != 0:
@@ -266,7 +252,7 @@ def _runtest(self, test_name):
         return MultiprocessResult(result, stdout, stderr, err_msg)
 
     def run(self):
-        while not self._killed:
+        while not self._stopped:
             try:
                 try:
                     test_name = next(self.pending)
@@ -284,6 +270,33 @@ def run(self):
                 self.output.put((True, traceback.format_exc()))
                 break
 
+    def _wait_completed(self):
+        popen = self._popen
+
+        # stdout and stderr must be closed to ensure that communicate()
+        # does not hang
+        popen.stdout.close()
+        popen.stderr.close()
+
+        try:
+            popen.wait(JOIN_TIMEOUT)
+        except (subprocess.TimeoutExpired, OSError) as exc:
+            print_warning(f"Failed to wait for {self} completion "
+                          f"(timeout={format_duration(JOIN_TIMEOUT)}): "
+                          f"{exc!r}")
+
+    def wait_stopped(self, start_time):
+        while True:
+            # Write a message every second
+            self.join(1.0)
+            if not self.is_alive():
+                break
+            dt = time.monotonic() - start_time
+            print(f"Waiting for {self} thread for {format_duration(dt)}", flush=True)
+            if dt > JOIN_TIMEOUT:
+                print_warning(f"Failed to join {self} in {format_duration(dt)}")
+                break
+
 
 def get_running(workers):
     running = []
@@ -298,7 +311,7 @@ def get_running(workers):
     return running
 
 
-class MultiprocessRunner:
+class MultiprocessTestRunner:
     def __init__(self, regrtest):
         self.regrtest = regrtest
         self.ns = regrtest.ns
@@ -311,30 +324,20 @@ def __init__(self, regrtest):
         self.workers = None
 
     def start_workers(self):
-        self.workers = [MultiprocessThread(self.pending, self.output,
-                                           self.ns, self.worker_timeout)
-                        for _ in range(self.ns.use_mp)]
+        self.workers = [TestWorkerProcess(index, self.pending, self.output,
+                                          self.ns, self.worker_timeout)
+                        for index in range(1, self.ns.use_mp + 1)]
         print("Run tests in parallel using %s child processes"
               % len(self.workers))
         for worker in self.workers:
             worker.start()
 
-    def wait_workers(self):
+    def stop_workers(self):
         start_time = time.monotonic()
         for worker in self.workers:
-            worker.kill()
+            worker.stop()
         for worker in self.workers:
-            while True:
-                worker.join(1.0)
-                if not worker.is_alive():
-                    break
-                dt = time.monotonic() - start_time
-                print("Wait for regrtest worker %r for %.1f sec" % (worker, dt),
-                      flush=True)
-                if dt > JOIN_TIMEOUT:
-                    print("Warning -- failed to join a regrtest worker %s"
-                          % worker, flush=True)
-                    break
+            worker.wait_stopped(start_time)
 
     def _get_result(self):
         if not any(worker.is_alive() for worker in self.workers):
@@ -418,10 +421,11 @@ def run_tests(self):
             if self.ns.timeout is not None:
                 faulthandler.cancel_dump_traceback_later()
 
-        # a test failed (and --failfast is set) or all tests completed
-        self.pending.stop()
-        self.wait_workers()
+            # Always ensure that all worker processes are no longer
+            # worker when we exit this function
+            self.pending.stop()
+            self.stop_workers()
 
 
 def run_tests_multiprocess(regrtest):
-    MultiprocessRunner(regrtest).run_tests()
+    MultiprocessTestRunner(regrtest).run_tests()
diff --git a/Lib/test/libregrtest/win_utils.py b/Lib/test/libregrtest/win_utils.py
index ec2d6c663e83f..f0c17b906f519 100644
--- a/Lib/test/libregrtest/win_utils.py
+++ b/Lib/test/libregrtest/win_utils.py
@@ -3,16 +3,22 @@
 import os
 import subprocess
 import uuid
+import winreg
 from test import support
+from test.libregrtest.utils import print_warning
 
 
 # Max size of asynchronous reads
 BUFSIZE = 8192
 # Exponential damping factor (see below)
 LOAD_FACTOR_1 = 0.9200444146293232478931553241
+
 # Seconds per measurement
 SAMPLING_INTERVAL = 5
-COUNTER_NAME = r'\System\Processor Queue Length'
+# Windows registry subkey of HKEY_LOCAL_MACHINE where the counter names
+# of typeperf are registered
+COUNTER_REGISTRY_KEY = (r"SOFTWARE\Microsoft\Windows NT\CurrentVersion"
+                        r"\Perflib\CurrentLanguage")
 
 
 class WindowsLoadTracker():
@@ -25,7 +31,8 @@ class WindowsLoadTracker():
 
     def __init__(self):
         self.load = 0.0
-        self.p = None
+        self.counter_name = ''
+        self.popen = None
         self.start()
 
     def start(self):
@@ -55,31 +62,46 @@ def start(self):
         overlap.GetOverlappedResult(True)
 
         # Spawn off the load monitor
-        command = ['typeperf', COUNTER_NAME, '-si', str(SAMPLING_INTERVAL)]
-        self.p = subprocess.Popen(command, stdout=command_stdout, cwd=support.SAVEDCWD)
+        counter_name = self._get_counter_name()
+        command = ['typeperf', counter_name, '-si', str(SAMPLING_INTERVAL)]
+        self.popen = subprocess.Popen(' '.join(command), stdout=command_stdout, cwd=support.SAVEDCWD)
 
         # Close our copy of the write end of the pipe
         os.close(command_stdout)
 
+    def _get_counter_name(self):
+        # accessing the registry to get the counter localization name
+        with winreg.OpenKey(winreg.HKEY_LOCAL_MACHINE, COUNTER_REGISTRY_KEY) as perfkey:
+            counters = winreg.QueryValueEx(perfkey, 'Counter')[0]
+
+        # Convert [key1, value1, key2, value2, ...] list
+        # to {key1: value1, key2: value2, ...} dict
+        counters = iter(counters)
+        counters_dict = dict(zip(counters, counters))
+
+        # System counter has key '2' and Processor Queue Length has key '44'
+        system = counters_dict['2']
+        process_queue_length = counters_dict['44']
+        return f'"\\{system}\\{process_queue_length}"'
+
     def close(self):
-        if self.p is None:
+        if self.popen is None:
             return
-        self.p.kill()
-        self.p.wait()
-        self.p = None
+        self.popen.kill()
+        self.popen.wait()
+        self.popen = None
 
     def __del__(self):
         self.close()
 
     def read_output(self):
-        import _winapi
-
         overlapped, _ = _winapi.ReadFile(self.pipe, BUFSIZE, True)
         bytes_read, res = overlapped.GetOverlappedResult(False)
         if res != 0:
             return
 
-        return overlapped.getbuffer().decode()
+        output = overlapped.getbuffer()
+        return output.decode('oem', 'replace')
 
     def getloadavg(self):
         typeperf_output = self.read_output()
@@ -89,14 +111,29 @@ def getloadavg(self):
 
         # Process the backlog of load values
         for line in typeperf_output.splitlines():
+            # Ignore the initial header:
+            # "(PDH-CSV 4.0)","\\\\WIN\\System\\Processor Queue Length"
+            if '\\\\' in line:
+                continue
+
+            # Ignore blank lines
+            if not line.strip():
+                continue
+
             # typeperf outputs in a CSV format like this:
             # "07/19/2018 01:32:26.605","3.000000"
-            toks = line.split(',')
-            # Ignore blank lines and the initial header
-            if line.strip() == '' or (COUNTER_NAME in line) or len(toks) != 2:
+            # (date, process queue length)
+            try:
+                tokens = line.split(',')
+                if len(tokens) != 2:
+                    raise ValueError
+
+                value = tokens[1].replace('"', '')
+                load = float(value)
+            except ValueError:
+                print_warning("Failed to parse typeperf output: %a" % line)
                 continue
 
-            load = float(toks[1].replace('"', ''))
             # We use an exponentially weighted moving average, imitating the
             # load calculation on Unix systems.
             # https://en.wikipedia.org/wiki/Load_(computing)#Unix-style_load_calculation


From webhook-mailer at python.org  Tue Oct  1 06:58:04 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Tue, 01 Oct 2019 10:58:04 -0000
Subject: [Python-checkins] [2.7] bpo-38243: Escape the server title of
 DocXMLRPCServer (GH-16447)
Message-ID: <mailman.174.1569927486.32504.python-checkins@python.org>

https://github.com/python/cpython/commit/8eb64155ff26823542ccf0225b3d57b6ae36ea89
commit: 8eb64155ff26823542ccf0225b3d57b6ae36ea89
branch: 2.7
author: Dong-hee Na <donghee.na92 at gmail.com>
committer: Victor Stinner <vstinner at python.org>
date: 2019-10-01T12:58:00+02:00
summary:

[2.7] bpo-38243: Escape the server title of DocXMLRPCServer (GH-16447)

Escape the server title of DocXMLRPCServer.DocXMLRPCServer
when rendering the document page as HTML.

files:
A Misc/NEWS.d/next/Security/2019-09-25-13-21-09.bpo-38243.1pfz24.rst
M Lib/DocXMLRPCServer.py
M Lib/test/test_docxmlrpc.py

diff --git a/Lib/DocXMLRPCServer.py b/Lib/DocXMLRPCServer.py
index 4064ec2e48d4d..90b037dd35d6b 100644
--- a/Lib/DocXMLRPCServer.py
+++ b/Lib/DocXMLRPCServer.py
@@ -20,6 +20,16 @@
             CGIXMLRPCRequestHandler,
             resolve_dotted_attribute)
 
+
+def _html_escape_quote(s):
+    s = s.replace("&", "&amp;") # Must be done first!
+    s = s.replace("<", "&lt;")
+    s = s.replace(">", "&gt;")
+    s = s.replace('"', "&quot;")
+    s = s.replace('\'', "&#x27;")
+    return s
+
+
 class ServerHTMLDoc(pydoc.HTMLDoc):
     """Class used to generate pydoc HTML document for a server"""
 
@@ -210,7 +220,8 @@ def generate_html_documentation(self):
                                 methods
                             )
 
-        return documenter.page(self.server_title, documentation)
+        title = _html_escape_quote(self.server_title)
+        return documenter.page(title, documentation)
 
 class DocXMLRPCRequestHandler(SimpleXMLRPCRequestHandler):
     """XML-RPC and documentation request handler class.
diff --git a/Lib/test/test_docxmlrpc.py b/Lib/test/test_docxmlrpc.py
index 4dff4159e2466..c45b892b8b3e7 100644
--- a/Lib/test/test_docxmlrpc.py
+++ b/Lib/test/test_docxmlrpc.py
@@ -1,5 +1,6 @@
 from DocXMLRPCServer import DocXMLRPCServer
 import httplib
+import re
 import sys
 from test import test_support
 threading = test_support.import_module('threading')
@@ -176,6 +177,25 @@ def test_autolink_dotted_methods(self):
         self.assertIn("""Try&nbsp;self.<strong>add</strong>,&nbsp;too.""",
                       response.read())
 
+    def test_server_title_escape(self):
+        """Test that the server title and documentation
+        are escaped for HTML.
+        """
+        self.serv.set_server_title('test_title<script>')
+        self.serv.set_server_documentation('test_documentation<script>')
+        self.assertEqual('test_title<script>', self.serv.server_title)
+        self.assertEqual('test_documentation<script>',
+                self.serv.server_documentation)
+
+        generated = self.serv.generate_html_documentation()
+        title = re.search(r'<title>(.+?)</title>', generated).group()
+        documentation = re.search(r'<p><tt>(.+?)</tt></p>', generated).group()
+        self.assertEqual('<title>Python: test_title&lt;script&gt;</title>',
+                title)
+        self.assertEqual('<p><tt>test_documentation&lt;script&gt;</tt></p>',
+                documentation)
+
+
 def test_main():
     test_support.run_unittest(DocXMLRPCHTTPGETServer)
 
diff --git a/Misc/NEWS.d/next/Security/2019-09-25-13-21-09.bpo-38243.1pfz24.rst b/Misc/NEWS.d/next/Security/2019-09-25-13-21-09.bpo-38243.1pfz24.rst
new file mode 100644
index 0000000000000..8f02baed9ebe5
--- /dev/null
+++ b/Misc/NEWS.d/next/Security/2019-09-25-13-21-09.bpo-38243.1pfz24.rst
@@ -0,0 +1,3 @@
+Escape the server title of :class:`DocXMLRPCServer.DocXMLRPCServer`
+when rendering the document page as HTML.
+(Contributed by Dong-hee Na in :issue:`38243`.)


From webhook-mailer at python.org  Tue Oct  1 06:58:57 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Tue, 01 Oct 2019 10:58:57 -0000
Subject: [Python-checkins] bpo-36670: Multiple regrtest bugfixes (GH-16511)
Message-ID: <mailman.175.1569927537.32504.python-checkins@python.org>

https://github.com/python/cpython/commit/d6a92b55944bf1ef4992e4375f02a7132717bf53
commit: d6a92b55944bf1ef4992e4375f02a7132717bf53
branch: 3.7
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-01T03:58:53-07:00
summary:

bpo-36670: Multiple regrtest bugfixes (GH-16511)


* Windows: Fix counter name in WindowsLoadTracker. Counter names are
  localized: use the registry to get the counter name. Original
  change written by Lorenz Mende.
* Regrtest.main() now ensures that the Windows load tracker is also
  killed if an exception is raised
* TestWorkerProcess now ensures that worker processes are no longer
  running before exiting: kill also worker processes when an
  exception is raised.
* Enhance regrtest messages and warnings: include test name,
  duration, add a worker identifier, etc.
* Rename MultiprocessRunner to TestWorkerProcess
* Use print_warning() to display warnings.

Co-Authored-By: Lorenz Mende <Lorenz.mende at gmail.com>
(cherry picked from commit 982bfa4da07b2e5749a0f4e68f99e972bcc3a549)

Co-authored-by: Victor Stinner <vstinner at redhat.com>

files:
M Lib/test/libregrtest/main.py
M Lib/test/libregrtest/runtest_mp.py
M Lib/test/libregrtest/win_utils.py

diff --git a/Lib/test/libregrtest/main.py b/Lib/test/libregrtest/main.py
index a59a03d4f2512..5ae8878638a5c 100644
--- a/Lib/test/libregrtest/main.py
+++ b/Lib/test/libregrtest/main.py
@@ -503,10 +503,6 @@ def run_tests(self):
             self.run_tests_sequential()
 
     def finalize(self):
-        if self.win_load_tracker is not None:
-            self.win_load_tracker.close()
-            self.win_load_tracker = None
-
         if self.next_single_filename:
             if self.next_single_test:
                 with open(self.next_single_filename, 'w') as fp:
@@ -674,11 +670,16 @@ def _main(self, tests, kwargs):
                 # typeperf.exe for x64, x86 or ARM
                 print(f'Failed to create WindowsLoadTracker: {error}')
 
-        self.run_tests()
-        self.display_result()
-
-        if self.ns.verbose2 and self.bad:
-            self.rerun_failed_tests()
+        try:
+            self.run_tests()
+            self.display_result()
+
+            if self.ns.verbose2 and self.bad:
+                self.rerun_failed_tests()
+        finally:
+            if self.win_load_tracker is not None:
+                self.win_load_tracker.close()
+                self.win_load_tracker = None
 
         self.finalize()
 
diff --git a/Lib/test/libregrtest/runtest_mp.py b/Lib/test/libregrtest/runtest_mp.py
index 9cb5be6bb8ae8..38b05781de5fc 100644
--- a/Lib/test/libregrtest/runtest_mp.py
+++ b/Lib/test/libregrtest/runtest_mp.py
@@ -15,7 +15,7 @@
     runtest, INTERRUPTED, CHILD_ERROR, PROGRESS_MIN_TIME,
     format_test_result, TestResult, is_failed, TIMEOUT)
 from test.libregrtest.setup import setup_tests
-from test.libregrtest.utils import format_duration
+from test.libregrtest.utils import format_duration, print_warning
 
 
 # Display the running tests if nothing happened last N seconds
@@ -103,9 +103,10 @@ class ExitThread(Exception):
     pass
 
 
-class MultiprocessThread(threading.Thread):
-    def __init__(self, pending, output, ns, timeout):
+class TestWorkerProcess(threading.Thread):
+    def __init__(self, worker_id, pending, output, ns, timeout):
         super().__init__()
+        self.worker_id = worker_id
         self.pending = pending
         self.output = output
         self.ns = ns
@@ -114,12 +115,16 @@ def __init__(self, pending, output, ns, timeout):
         self.start_time = None
         self._popen = None
         self._killed = False
+        self._stopped = False
 
     def __repr__(self):
-        info = ['MultiprocessThread']
-        test = self.current_test_name
+        info = [f'TestWorkerProcess #{self.worker_id}']
         if self.is_alive():
-            info.append('alive')
+            dt = time.monotonic() - self.start_time
+            info.append("running for %s" % format_duration(dt))
+        else:
+            info.append('stopped')
+        test = self.current_test_name
         if test:
             info.append(f'test={test}')
         popen = self._popen
@@ -128,53 +133,24 @@ def __repr__(self):
         return '<%s>' % ' '.join(info)
 
     def _kill(self):
-        dt = time.monotonic() - self.start_time
+        if self._killed:
+            return
+        self._killed = True
 
         popen = self._popen
-        pid = popen.pid
-        print("Kill worker process %s running for %.1f sec" % (pid, dt),
-              file=sys.stderr, flush=True)
+        if popen is None:
+            return
 
+        print(f"Kill {self}", file=sys.stderr, flush=True)
         try:
             popen.kill()
-            return True
         except OSError as exc:
-            print("WARNING: Failed to kill worker process %s: %r" % (pid, exc),
-                  file=sys.stderr, flush=True)
-            return False
-
-    def _close_wait(self):
-        popen = self._popen
-
-        # stdout and stderr must be closed to ensure that communicate()
-        # does not hang
-        popen.stdout.close()
-        popen.stderr.close()
-
-        try:
-            popen.wait(JOIN_TIMEOUT)
-        except (subprocess.TimeoutExpired, OSError) as exc:
-            print("WARNING: Failed to wait for worker process %s "
-                  "completion (timeout=%.1f sec): %r"
-                  % (popen.pid, JOIN_TIMEOUT, exc),
-                  file=sys.stderr, flush=True)
-
-    def kill(self):
-        """
-        Kill the current process (if any).
-
-        This method can be called by the thread running the process,
-        or by another thread.
-        """
-        self._killed = True
-
-        if self._popen is None:
-            return
-
-        if not self._kill():
-            return
+            print_warning(f"Failed to kill {self}: {exc!r}")
 
-        self._close_wait()
+    def stop(self):
+        # Method called from a different thread to stop this thread
+        self._stopped = True
+        self._kill()
 
     def mp_result_error(self, test_name, error_type, stdout='', stderr='',
                         err_msg=None):
@@ -190,59 +166,69 @@ def _timedout(self, test_name):
         try:
             stdout, stderr = popen.communicate(timeout=JOIN_TIMEOUT)
         except (subprocess.TimeoutExpired, OSError) as exc:
-            print("WARNING: Failed to read worker process %s output "
-                  "(timeout=%.1f sec): %r"
-                  % (popen.pid, JOIN_TIMEOUT, exc),
-                  file=sys.stderr, flush=True)
-
-        self._close_wait()
+            print_warning(f"Failed to read {self} output "
+                          f"(timeout={format_duration(JOIN_TIMEOUT)}): "
+                          f"{exc!r}")
 
         return self.mp_result_error(test_name, TIMEOUT, stdout, stderr)
 
-    def _runtest(self, test_name):
-        try:
-            self.start_time = time.monotonic()
-            self.current_test_name = test_name
+    def _run_process(self, test_name):
+        self.start_time = time.monotonic()
 
+        self.current_test_name = test_name
+        try:
+            self._killed = False
             self._popen = run_test_in_subprocess(test_name, self.ns)
             popen = self._popen
+        except:
+            self.current_test_name = None
+            raise
+
+        try:
+            if self._stopped:
+                # If kill() has been called before self._popen is set,
+                # self._popen is still running. Call again kill()
+                # to ensure that the process is killed.
+                self._kill()
+                raise ExitThread
+
             try:
-                try:
-                    if self._killed:
-                        # If kill() has been called before self._popen is set,
-                        # self._popen is still running. Call again kill()
-                        # to ensure that the process is killed.
-                        self.kill()
-                        raise ExitThread
-
-                    try:
-                        stdout, stderr = popen.communicate(timeout=self.timeout)
-                    except subprocess.TimeoutExpired:
-                        if self._killed:
-                            # kill() has been called: communicate() fails
-                            # on reading closed stdout/stderr
-                            raise ExitThread
-
-                        return self._timedout(test_name)
-                    except OSError:
-                        if self._killed:
-                            # kill() has been called: communicate() fails
-                            # on reading closed stdout/stderr
-                            raise ExitThread
-                        raise
-                except:
-                    self.kill()
-                    raise
-            finally:
-                self._close_wait()
+                stdout, stderr = popen.communicate(timeout=self.timeout)
+            except subprocess.TimeoutExpired:
+                if self._stopped:
+                    # kill() has been called: communicate() fails
+                    # on reading closed stdout/stderr
+                    raise ExitThread
+
+                return self._timedout(test_name)
+            except OSError:
+                if self._stopped:
+                    # kill() has been called: communicate() fails
+                    # on reading closed stdout/stderr
+                    raise ExitThread
+                raise
 
             retcode = popen.returncode
+            stdout = stdout.strip()
+            stderr = stderr.rstrip()
+
+            return (retcode, stdout, stderr)
+        except:
+            self._kill()
+            raise
         finally:
-            self.current_test_name = None
+            self._wait_completed()
             self._popen = None
+            self.current_test_name = None
+
+    def _runtest(self, test_name):
+        result = self._run_process(test_name)
 
-        stdout = stdout.strip()
-        stderr = stderr.rstrip()
+        if isinstance(result, MultiprocessResult):
+            # _timedout() case
+            return result
+
+        retcode, stdout, stderr = result
 
         err_msg = None
         if retcode != 0:
@@ -266,7 +252,7 @@ def _runtest(self, test_name):
         return MultiprocessResult(result, stdout, stderr, err_msg)
 
     def run(self):
-        while not self._killed:
+        while not self._stopped:
             try:
                 try:
                     test_name = next(self.pending)
@@ -284,6 +270,33 @@ def run(self):
                 self.output.put((True, traceback.format_exc()))
                 break
 
+    def _wait_completed(self):
+        popen = self._popen
+
+        # stdout and stderr must be closed to ensure that communicate()
+        # does not hang
+        popen.stdout.close()
+        popen.stderr.close()
+
+        try:
+            popen.wait(JOIN_TIMEOUT)
+        except (subprocess.TimeoutExpired, OSError) as exc:
+            print_warning(f"Failed to wait for {self} completion "
+                          f"(timeout={format_duration(JOIN_TIMEOUT)}): "
+                          f"{exc!r}")
+
+    def wait_stopped(self, start_time):
+        while True:
+            # Write a message every second
+            self.join(1.0)
+            if not self.is_alive():
+                break
+            dt = time.monotonic() - start_time
+            print(f"Waiting for {self} thread for {format_duration(dt)}", flush=True)
+            if dt > JOIN_TIMEOUT:
+                print_warning(f"Failed to join {self} in {format_duration(dt)}")
+                break
+
 
 def get_running(workers):
     running = []
@@ -298,7 +311,7 @@ def get_running(workers):
     return running
 
 
-class MultiprocessRunner:
+class MultiprocessTestRunner:
     def __init__(self, regrtest):
         self.regrtest = regrtest
         self.ns = regrtest.ns
@@ -311,30 +324,20 @@ def __init__(self, regrtest):
         self.workers = None
 
     def start_workers(self):
-        self.workers = [MultiprocessThread(self.pending, self.output,
-                                           self.ns, self.worker_timeout)
-                        for _ in range(self.ns.use_mp)]
+        self.workers = [TestWorkerProcess(index, self.pending, self.output,
+                                          self.ns, self.worker_timeout)
+                        for index in range(1, self.ns.use_mp + 1)]
         print("Run tests in parallel using %s child processes"
               % len(self.workers))
         for worker in self.workers:
             worker.start()
 
-    def wait_workers(self):
+    def stop_workers(self):
         start_time = time.monotonic()
         for worker in self.workers:
-            worker.kill()
+            worker.stop()
         for worker in self.workers:
-            while True:
-                worker.join(1.0)
-                if not worker.is_alive():
-                    break
-                dt = time.monotonic() - start_time
-                print("Wait for regrtest worker %r for %.1f sec" % (worker, dt),
-                      flush=True)
-                if dt > JOIN_TIMEOUT:
-                    print("Warning -- failed to join a regrtest worker %s"
-                          % worker, flush=True)
-                    break
+            worker.wait_stopped(start_time)
 
     def _get_result(self):
         if not any(worker.is_alive() for worker in self.workers):
@@ -418,10 +421,11 @@ def run_tests(self):
             if self.ns.timeout is not None:
                 faulthandler.cancel_dump_traceback_later()
 
-        # a test failed (and --failfast is set) or all tests completed
-        self.pending.stop()
-        self.wait_workers()
+            # Always ensure that all worker processes are no longer
+            # worker when we exit this function
+            self.pending.stop()
+            self.stop_workers()
 
 
 def run_tests_multiprocess(regrtest):
-    MultiprocessRunner(regrtest).run_tests()
+    MultiprocessTestRunner(regrtest).run_tests()
diff --git a/Lib/test/libregrtest/win_utils.py b/Lib/test/libregrtest/win_utils.py
index ec2d6c663e83f..f0c17b906f519 100644
--- a/Lib/test/libregrtest/win_utils.py
+++ b/Lib/test/libregrtest/win_utils.py
@@ -3,16 +3,22 @@
 import os
 import subprocess
 import uuid
+import winreg
 from test import support
+from test.libregrtest.utils import print_warning
 
 
 # Max size of asynchronous reads
 BUFSIZE = 8192
 # Exponential damping factor (see below)
 LOAD_FACTOR_1 = 0.9200444146293232478931553241
+
 # Seconds per measurement
 SAMPLING_INTERVAL = 5
-COUNTER_NAME = r'\System\Processor Queue Length'
+# Windows registry subkey of HKEY_LOCAL_MACHINE where the counter names
+# of typeperf are registered
+COUNTER_REGISTRY_KEY = (r"SOFTWARE\Microsoft\Windows NT\CurrentVersion"
+                        r"\Perflib\CurrentLanguage")
 
 
 class WindowsLoadTracker():
@@ -25,7 +31,8 @@ class WindowsLoadTracker():
 
     def __init__(self):
         self.load = 0.0
-        self.p = None
+        self.counter_name = ''
+        self.popen = None
         self.start()
 
     def start(self):
@@ -55,31 +62,46 @@ def start(self):
         overlap.GetOverlappedResult(True)
 
         # Spawn off the load monitor
-        command = ['typeperf', COUNTER_NAME, '-si', str(SAMPLING_INTERVAL)]
-        self.p = subprocess.Popen(command, stdout=command_stdout, cwd=support.SAVEDCWD)
+        counter_name = self._get_counter_name()
+        command = ['typeperf', counter_name, '-si', str(SAMPLING_INTERVAL)]
+        self.popen = subprocess.Popen(' '.join(command), stdout=command_stdout, cwd=support.SAVEDCWD)
 
         # Close our copy of the write end of the pipe
         os.close(command_stdout)
 
+    def _get_counter_name(self):
+        # accessing the registry to get the counter localization name
+        with winreg.OpenKey(winreg.HKEY_LOCAL_MACHINE, COUNTER_REGISTRY_KEY) as perfkey:
+            counters = winreg.QueryValueEx(perfkey, 'Counter')[0]
+
+        # Convert [key1, value1, key2, value2, ...] list
+        # to {key1: value1, key2: value2, ...} dict
+        counters = iter(counters)
+        counters_dict = dict(zip(counters, counters))
+
+        # System counter has key '2' and Processor Queue Length has key '44'
+        system = counters_dict['2']
+        process_queue_length = counters_dict['44']
+        return f'"\\{system}\\{process_queue_length}"'
+
     def close(self):
-        if self.p is None:
+        if self.popen is None:
             return
-        self.p.kill()
-        self.p.wait()
-        self.p = None
+        self.popen.kill()
+        self.popen.wait()
+        self.popen = None
 
     def __del__(self):
         self.close()
 
     def read_output(self):
-        import _winapi
-
         overlapped, _ = _winapi.ReadFile(self.pipe, BUFSIZE, True)
         bytes_read, res = overlapped.GetOverlappedResult(False)
         if res != 0:
             return
 
-        return overlapped.getbuffer().decode()
+        output = overlapped.getbuffer()
+        return output.decode('oem', 'replace')
 
     def getloadavg(self):
         typeperf_output = self.read_output()
@@ -89,14 +111,29 @@ def getloadavg(self):
 
         # Process the backlog of load values
         for line in typeperf_output.splitlines():
+            # Ignore the initial header:
+            # "(PDH-CSV 4.0)","\\\\WIN\\System\\Processor Queue Length"
+            if '\\\\' in line:
+                continue
+
+            # Ignore blank lines
+            if not line.strip():
+                continue
+
             # typeperf outputs in a CSV format like this:
             # "07/19/2018 01:32:26.605","3.000000"
-            toks = line.split(',')
-            # Ignore blank lines and the initial header
-            if line.strip() == '' or (COUNTER_NAME in line) or len(toks) != 2:
+            # (date, process queue length)
+            try:
+                tokens = line.split(',')
+                if len(tokens) != 2:
+                    raise ValueError
+
+                value = tokens[1].replace('"', '')
+                load = float(value)
+            except ValueError:
+                print_warning("Failed to parse typeperf output: %a" % line)
                 continue
 
-            load = float(toks[1].replace('"', ''))
             # We use an exponentially weighted moving average, imitating the
             # load calculation on Unix systems.
             # https://en.wikipedia.org/wiki/Load_(computing)#Unix-style_load_calculation


From webhook-mailer at python.org  Tue Oct  1 07:12:33 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Tue, 01 Oct 2019 11:12:33 -0000
Subject: [Python-checkins] bpo-37474: Don't call fedisableexcept() on
 FreeBSD (GH-16515)
Message-ID: <mailman.176.1569928354.32504.python-checkins@python.org>

https://github.com/python/cpython/commit/5e0ea7540f577c9684e272000fdfc80d29bb78a2
commit: 5e0ea7540f577c9684e272000fdfc80d29bb78a2
branch: master
author: Victor Stinner <vstinner at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-01T13:12:29+02:00
summary:

bpo-37474: Don't call fedisableexcept() on FreeBSD (GH-16515)

On FreeBSD, Python no longer calls fedisableexcept() at startup to
control the floating point control mode. The call became useless
since FreeBSD 6: it became the default mode.

files:
A Misc/NEWS.d/next/Core and Builtins/2019-10-01-12-46-30.bpo-37474.cB3se1.rst
M Modules/main.c

diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-10-01-12-46-30.bpo-37474.cB3se1.rst b/Misc/NEWS.d/next/Core and Builtins/2019-10-01-12-46-30.bpo-37474.cB3se1.rst
new file mode 100644
index 0000000000000..c259f81a3d7d4
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2019-10-01-12-46-30.bpo-37474.cB3se1.rst	
@@ -0,0 +1,3 @@
+On FreeBSD, Python no longer calls ``fedisableexcept()`` at startup to
+control the floating point control mode. The call became useless since
+FreeBSD 6: it became the default mode.
diff --git a/Modules/main.c b/Modules/main.c
index 526b0ade84a4b..929417fe775f2 100644
--- a/Modules/main.c
+++ b/Modules/main.c
@@ -6,10 +6,6 @@
 #include "pycore_pymem.h"
 #include "pycore_pystate.h"
 
-#ifdef __FreeBSD__
-#  include <fenv.h>     /* fedisableexcept() */
-#endif
-
 /* Includes for exit_sigint() */
 #include <stdio.h>      /* perror() */
 #ifdef HAVE_SIGNAL_H
@@ -43,15 +39,6 @@ pymain_init(const _PyArgv *args)
         return status;
     }
 
-    /* 754 requires that FP exceptions run in "no stop" mode by default,
-     * and until C vendors implement C99's ways to control FP exceptions,
-     * Python requires non-stop mode.  Alas, some platforms enable FP
-     * exceptions by default.  Here we disable them.
-     */
-#ifdef __FreeBSD__
-    fedisableexcept(FE_OVERFLOW);
-#endif
-
     PyPreConfig preconfig;
     PyPreConfig_InitPythonConfig(&preconfig);
 


From webhook-mailer at python.org  Tue Oct  1 07:29:57 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Tue, 01 Oct 2019 11:29:57 -0000
Subject: [Python-checkins] bpo-37802: Fix a compiler warning in longobject.c
 (GH-16517)
Message-ID: <mailman.177.1569929398.32504.python-checkins@python.org>

https://github.com/python/cpython/commit/6314abcc08f5d0f3d3a915dc9455ea223fa65517
commit: 6314abcc08f5d0f3d3a915dc9455ea223fa65517
branch: master
author: Victor Stinner <vstinner at redhat.com>
committer: GitHub <noreply at github.com>
date: 2019-10-01T13:29:53+02:00
summary:

bpo-37802: Fix a compiler warning in longobject.c (GH-16517)

bpo-37802, bpo-38321: Fix the following warnings:

    longobject.c(420): warning C4244: 'function': conversion from
    'unsigned __int64' to 'sdigit', possible loss of data

    longobject.c(428): warning C4267: 'function': conversion from
    'size_t' to 'sdigit', possible loss of data

files:
M Objects/longobject.c

diff --git a/Objects/longobject.c b/Objects/longobject.c
index f2f6f9540892e..2b57ea18666ee 100644
--- a/Objects/longobject.c
+++ b/Objects/longobject.c
@@ -383,7 +383,7 @@ PyLong_FromLong(long ival)
 #define PYLONG_FROM_UINT(INT_TYPE, ival) \
     do { \
         if (IS_SMALL_UINT(ival)) { \
-            return get_small_int((ival)); \
+            return get_small_int((sdigit)(ival)); \
         } \
         /* Count the number of Python digits. */ \
         Py_ssize_t ndigits = 0; \


From webhook-mailer at python.org  Tue Oct  1 07:52:04 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Tue, 01 Oct 2019 11:52:04 -0000
Subject: [Python-checkins] [3.8] bpo-3832: Fix compiler warnings (GH-16518)
Message-ID: <mailman.178.1569930725.32504.python-checkins@python.org>

https://github.com/python/cpython/commit/bfe1f74e39d0049a829962050e86a6a2d2a2781e
commit: bfe1f74e39d0049a829962050e86a6a2d2a2781e
branch: 3.8
author: Victor Stinner <vstinner at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-01T13:52:00+02:00
summary:

[3.8] bpo-3832: Fix compiler warnings (GH-16518)

* bpo-38321: Fix _asynciomodule.c compiler warning (GH-16493)

bpo-38248, bpo-38321: Fix warning:

    modules\_asynciomodule.c(2667):
    warning C4102: 'set_exception': unreferenced label

The related goto has been removed by
commit edad4d89e357c92f70c0324b937845d652b20afd.

(cherry picked from commit efe74b6369a8d08f27c69703fcc1686966e51068)

* bpo-38321: Fix PyCStructUnionType_update_stgdict() warning (GH-16492)

bpo-22273, bpo-38321: Fix following warning:

    modules\_ctypes\stgdict.c(704):
    warning C4244: 'initializing': conversion from 'Py_ssize_t' to 'int', possible loss of data

(cherry picked from commit c9a413ede47171a224c72dd34122005170caaad4)

files:
M Modules/_asynciomodule.c
M Modules/_ctypes/stgdict.c

diff --git a/Modules/_asynciomodule.c b/Modules/_asynciomodule.c
index cea3affe99ec3..8eb480fb77eaa 100644
--- a/Modules/_asynciomodule.c
+++ b/Modules/_asynciomodule.c
@@ -2664,7 +2664,6 @@ task_step_impl(TaskObj *task, PyObject *exc)
         /* Some other exception; pop it and call Task.set_exception() */
         PyErr_Fetch(&et, &ev, &tb);
 
-set_exception:
         assert(et);
         if (!ev || !PyObject_TypeCheck(ev, (PyTypeObject *) et)) {
             PyErr_NormalizeException(&et, &ev, &tb);
diff --git a/Modules/_ctypes/stgdict.c b/Modules/_ctypes/stgdict.c
index a27d1341dc3ec..fa664bd57a029 100644
--- a/Modules/_ctypes/stgdict.c
+++ b/Modules/_ctypes/stgdict.c
@@ -701,7 +701,7 @@ PyCStructUnionType_update_stgdict(PyObject *type, PyObject *fields, int isStruct
                 assert(actual_type_index <= MAX_ELEMENTS);
             }
             else {
-                int length = dict->length;
+                Py_ssize_t length = dict->length;
                 StgDictObject *edict;
 
                 edict = PyType_stgdict(dict->proto);


From webhook-mailer at python.org  Tue Oct  1 16:56:44 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Tue, 01 Oct 2019 20:56:44 -0000
Subject: [Python-checkins] Correct typos in the codecs module documentation
 (GH-15135)
Message-ID: <mailman.179.1569963405.32504.python-checkins@python.org>

https://github.com/python/cpython/commit/874dae48fe9e7d66d19d942033563e42aadc8060
commit: 874dae48fe9e7d66d19d942033563e42aadc8060
branch: 3.7
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-01T13:56:40-07:00
summary:

Correct typos in the codecs module documentation (GH-15135)

(cherry picked from commit 891e9e3b44c99d643dc309a4e63082451271b136)

Co-authored-by: G?ry Ogam <gery.ogam at gmail.com>

files:
M Doc/library/codecs.rst

diff --git a/Doc/library/codecs.rst b/Doc/library/codecs.rst
index ff4f4933b37d6..5c41828249c31 100644
--- a/Doc/library/codecs.rst
+++ b/Doc/library/codecs.rst
@@ -292,7 +292,7 @@ Error Handlers
 
 To simplify and standardize error handling,
 codecs may implement different error handling schemes by
-accepting the *errors* string argument.  The following string values are
+accepting the *errors* string argument. The following string values are
 defined and implemented by all standard Python codecs:
 
 .. tabularcolumns:: |l|L|
@@ -301,11 +301,11 @@ defined and implemented by all standard Python codecs:
 | Value                   | Meaning                                       |
 +=========================+===============================================+
 | ``'strict'``            | Raise :exc:`UnicodeError` (or a subclass);    |
-|                         | this is the default.  Implemented in          |
+|                         | this is the default. Implemented in           |
 |                         | :func:`strict_errors`.                        |
 +-------------------------+-----------------------------------------------+
 | ``'ignore'``            | Ignore the malformed data and continue        |
-|                         | without further notice.  Implemented in       |
+|                         | without further notice. Implemented in        |
 |                         | :func:`ignore_errors`.                        |
 +-------------------------+-----------------------------------------------+
 
@@ -327,11 +327,11 @@ The following error handlers are only applicable to
 |                         | marker; Python will use the official          |
 |                         | ``U+FFFD`` REPLACEMENT CHARACTER for the      |
 |                         | built-in codecs on decoding, and '?' on       |
-|                         | encoding.  Implemented in                     |
+|                         | encoding. Implemented in                      |
 |                         | :func:`replace_errors`.                       |
 +-------------------------+-----------------------------------------------+
 | ``'xmlcharrefreplace'`` | Replace with the appropriate XML character    |
-|                         | reference (only for encoding).  Implemented   |
+|                         | reference (only for encoding). Implemented    |
 |                         | in :func:`xmlcharrefreplace_errors`.          |
 +-------------------------+-----------------------------------------------+
 | ``'backslashreplace'``  | Replace with backslashed escape sequences.    |
@@ -339,15 +339,15 @@ The following error handlers are only applicable to
 |                         | :func:`backslashreplace_errors`.              |
 +-------------------------+-----------------------------------------------+
 | ``'namereplace'``       | Replace with ``\N{...}`` escape sequences     |
-|                         | (only for encoding).  Implemented in          |
+|                         | (only for encoding). Implemented in           |
 |                         | :func:`namereplace_errors`.                   |
 +-------------------------+-----------------------------------------------+
 | ``'surrogateescape'``   | On decoding, replace byte with individual     |
 |                         | surrogate code ranging from ``U+DC80`` to     |
-|                         | ``U+DCFF``.  This code will then be turned    |
+|                         | ``U+DCFF``. This code will then be turned     |
 |                         | back into the same byte when the              |
 |                         | ``'surrogateescape'`` error handler is used   |
-|                         | when encoding the data.  (See :pep:`383` for  |
+|                         | when encoding the data. (See :pep:`383` for   |
 |                         | more.)                                        |
 +-------------------------+-----------------------------------------------+
 
@@ -357,7 +357,7 @@ In addition, the following error handler is specific to the given codecs:
 | Value             | Codecs                 | Meaning                                   |
 +===================+========================+===========================================+
 |``'surrogatepass'``| utf-8, utf-16, utf-32, | Allow encoding and decoding of surrogate  |
-|                   | utf-16-be, utf-16-le,  | codes.  These codecs normally treat the   |
+|                   | utf-16-be, utf-16-le,  | codes. These codecs normally treat the    |
 |                   | utf-32-be, utf-32-le   | presence of surrogates as an error.       |
 +-------------------+------------------------+-------------------------------------------+
 
@@ -388,9 +388,9 @@ handler:
    error handler must either raise this or a different exception, or return a
    tuple with a replacement for the unencodable part of the input and a position
    where encoding should continue. The replacement may be either :class:`str` or
-   :class:`bytes`.  If the replacement is bytes, the encoder will simply copy
+   :class:`bytes`. If the replacement is bytes, the encoder will simply copy
    them into the output buffer. If the replacement is a string, the encoder will
-   encode the replacement.  Encoding continues on original input at the
+   encode the replacement. Encoding continues on original input at the
    specified position. Negative position values will be treated as being
    relative to the end of the input string. If the resulting position is out of
    bound an :exc:`IndexError` will be raised.
@@ -484,7 +484,7 @@ function interfaces of the stateless encoder and decoder:
 .. method:: Codec.decode(input[, errors])
 
    Decodes the object *input* and returns a tuple (output object, length
-   consumed).  For instance, for a :term:`text encoding`, decoding converts
+   consumed). For instance, for a :term:`text encoding`, decoding converts
    a bytes object encoded using a particular
    character set encoding to a string object.
 
@@ -568,7 +568,7 @@ define in order to be compatible with the Python codec registry.
       implementation should make sure that ``0`` is the most common
       state. (States that are more complicated than integers can be converted
       into an integer by marshaling/pickling the state and encoding the bytes
-      of the resulting string into an integer).
+      of the resulting string into an integer.)
 
 
    .. method:: setstate(state)
@@ -751,7 +751,7 @@ compatible with the Python codec registry.
       number of encoded bytes or code points to read
       for decoding. The decoder can modify this setting as
       appropriate. The default value -1 indicates to read and decode as much as
-      possible.  This parameter is intended to
+      possible. This parameter is intended to
       prevent having to decode huge files in one step.
 
       The *firstline* flag indicates that
@@ -780,8 +780,8 @@ compatible with the Python codec registry.
       Read all lines available on the input stream and return them as a list of
       lines.
 
-      Line-endings are implemented using the codec's decoder method and are
-      included in the list entries if *keepends* is true.
+      Line-endings are implemented using the codec's :meth:`decode` method and
+      are included in the list entries if *keepends* is true.
 
       *sizehint*, if given, is passed as the *size* argument to the stream's
       :meth:`read` method.
@@ -791,7 +791,7 @@ compatible with the Python codec registry.
 
       Resets the codec buffers used for keeping state.
 
-      Note that no stream repositioning should take place.  This method is
+      Note that no stream repositioning should take place. This method is
       primarily intended to be able to recover from decoding errors.
 
 
@@ -841,7 +841,7 @@ The design is such that one can use the factory functions returned by the
    code calling :meth:`read` and :meth:`write`, while *Reader* and *Writer*
    work on the backend?? the data in *stream*.
 
-   You can use these objects to do transparent transcodings from e.g. Latin-1
+   You can use these objects to do transparent transcodings, e.g., from Latin-1
    to UTF-8 and back.
 
    The *stream* argument must be a file-like object.
@@ -866,10 +866,10 @@ Encodings and Unicode
 ---------------------
 
 Strings are stored internally as sequences of code points in
-range ``0x0``--``0x10FFFF``.  (See :pep:`393` for
+range ``0x0``--``0x10FFFF``. (See :pep:`393` for
 more details about the implementation.)
 Once a string object is used outside of CPU and memory, endianness
-and how these arrays are stored as bytes become an issue.  As with other
+and how these arrays are stored as bytes become an issue. As with other
 codecs, serialising a string into a sequence of bytes is known as *encoding*,
 and recreating the string from the sequence of bytes is known as *decoding*.
 There are a variety of different text serialisation codecs, which are
@@ -964,7 +964,7 @@ to determine the byte order used for generating the byte sequence, but as a
 signature that helps in guessing the encoding. On encoding the utf-8-sig codec
 will write ``0xef``, ``0xbb``, ``0xbf`` as the first three bytes to the file. On
 decoding ``utf-8-sig`` will skip those three bytes if they appear as the first
-three bytes in the file.  In UTF-8, the use of the BOM is discouraged and
+three bytes in the file. In UTF-8, the use of the BOM is discouraged and
 should generally be avoided.
 
 
@@ -984,7 +984,7 @@ e.g. ``'utf-8'`` is a valid alias for the ``'utf_8'`` codec.
 .. impl-detail::
 
    Some common encodings can bypass the codecs lookup machinery to
-   improve performance.  These optimization opportunities are only
+   improve performance. These optimization opportunities are only
    recognized by CPython for a limited set of (case insensitive)
    aliases: utf-8, utf8, latin-1, latin1, iso-8859-1, iso8859-1, mbcs
    (Windows only), ascii, us-ascii, utf-16, utf16, utf-32, utf32, and
@@ -1150,7 +1150,7 @@ particular, the following variants typically exist:
 | iso2022_kr      | csiso2022kr, iso2022kr,        | Korean                         |
 |                 | iso-2022-kr                    |                                |
 +-----------------+--------------------------------+--------------------------------+
-| latin_1         | iso-8859-1, iso8859-1, 8859,   | West Europe                    |
+| latin_1         | iso-8859-1, iso8859-1, 8859,   | Western Europe                 |
 |                 | cp819, latin, latin1, L1       |                                |
 +-----------------+--------------------------------+--------------------------------+
 | iso8859_2       | iso-8859-2, latin2, L2         | Central and Eastern Europe     |
@@ -1251,11 +1251,11 @@ Python Specific Encodings
 -------------------------
 
 A number of predefined codecs are specific to Python, so their codec names have
-no meaning outside Python.  These are listed in the tables below based on the
+no meaning outside Python. These are listed in the tables below based on the
 expected input and output types (note that while text encodings are the most
 common use case for codecs, the underlying codec infrastructure supports
-arbitrary data transforms rather than just text encodings).  For asymmetric
-codecs, the stated purpose describes the encoding direction.
+arbitrary data transforms rather than just text encodings). For asymmetric
+codecs, the stated meaning describes the encoding direction.
 
 Text Encodings
 ^^^^^^^^^^^^^^
@@ -1267,27 +1267,27 @@ encodings.
 .. tabularcolumns:: |l|p{0.3\linewidth}|p{0.3\linewidth}|
 
 +--------------------+---------+---------------------------+
-| Codec              | Aliases | Purpose                   |
+| Codec              | Aliases | Meaning                   |
 +====================+=========+===========================+
-| idna               |         | Implements :rfc:`3490`,   |
+| idna               |         | Implement :rfc:`3490`,    |
 |                    |         | see also                  |
 |                    |         | :mod:`encodings.idna`.    |
 |                    |         | Only ``errors='strict'``  |
 |                    |         | is supported.             |
 +--------------------+---------+---------------------------+
-| mbcs               | ansi,   | Windows only: Encode      |
+| mbcs               | ansi,   | Windows only: Encode the  |
 |                    | dbcs    | operand according to the  |
-|                    |         | ANSI codepage (CP_ACP)    |
+|                    |         | ANSI codepage (CP_ACP).   |
 +--------------------+---------+---------------------------+
-| oem                |         | Windows only: Encode      |
+| oem                |         | Windows only: Encode the  |
 |                    |         | operand according to the  |
-|                    |         | OEM codepage (CP_OEMCP)   |
+|                    |         | OEM codepage (CP_OEMCP).  |
 |                    |         |                           |
 |                    |         | .. versionadded:: 3.6     |
 +--------------------+---------+---------------------------+
-| palmos             |         | Encoding of PalmOS 3.5    |
+| palmos             |         | Encoding of PalmOS 3.5.   |
 +--------------------+---------+---------------------------+
-| punycode           |         | Implements :rfc:`3492`.   |
+| punycode           |         | Implement :rfc:`3492`.    |
 |                    |         | Stateful codecs are not   |
 |                    |         | supported.                |
 +--------------------+---------+---------------------------+
@@ -1310,8 +1310,8 @@ encodings.
 |                    |         | literal in ASCII-encoded  |
 |                    |         | Python source code,       |
 |                    |         | except that quotes are    |
-|                    |         | not escaped. Decodes from |
-|                    |         | Latin-1 source code.      |
+|                    |         | not escaped. Decode       |
+|                    |         | from Latin-1 source code. |
 |                    |         | Beware that Python source |
 |                    |         | code actually uses UTF-8  |
 |                    |         | by default.               |
@@ -1333,19 +1333,19 @@ Binary Transforms
 ^^^^^^^^^^^^^^^^^
 
 The following codecs provide binary transforms: :term:`bytes-like object`
-to :class:`bytes` mappings.  They are not supported by :meth:`bytes.decode`
+to :class:`bytes` mappings. They are not supported by :meth:`bytes.decode`
 (which only produces :class:`str` output).
 
 
 .. tabularcolumns:: |l|L|L|L|
 
 +----------------------+------------------+------------------------------+------------------------------+
-| Codec                | Aliases          | Purpose                      | Encoder / decoder            |
+| Codec                | Aliases          | Meaning                      | Encoder / decoder            |
 +======================+==================+==============================+==============================+
-| base64_codec [#b64]_ | base64, base_64  | Convert operand to multiline | :meth:`base64.encodebytes` / |
-|                      |                  | MIME base64 (the result      | :meth:`base64.decodebytes`   |
-|                      |                  | always includes a trailing   |                              |
-|                      |                  | ``'\n'``)                    |                              |
+| base64_codec [#b64]_ | base64, base_64  | Convert the operand to       | :meth:`base64.encodebytes` / |
+|                      |                  | multiline MIME base64 (the   | :meth:`base64.decodebytes`   |
+|                      |                  | result always includes a     |                              |
+|                      |                  | trailing ``'\n'``).          |                              |
 |                      |                  |                              |                              |
 |                      |                  | .. versionchanged:: 3.4      |                              |
 |                      |                  |    accepts any               |                              |
@@ -1353,23 +1353,23 @@ to :class:`bytes` mappings.  They are not supported by :meth:`bytes.decode`
 |                      |                  |    as input for encoding and |                              |
 |                      |                  |    decoding                  |                              |
 +----------------------+------------------+------------------------------+------------------------------+
-| bz2_codec            | bz2              | Compress the operand         | :meth:`bz2.compress` /       |
-|                      |                  | using bz2                    | :meth:`bz2.decompress`       |
+| bz2_codec            | bz2              | Compress the operand using   | :meth:`bz2.compress` /       |
+|                      |                  | bz2.                         | :meth:`bz2.decompress`       |
 +----------------------+------------------+------------------------------+------------------------------+
-| hex_codec            | hex              | Convert operand to           | :meth:`binascii.b2a_hex` /   |
+| hex_codec            | hex              | Convert the operand to       | :meth:`binascii.b2a_hex` /   |
 |                      |                  | hexadecimal                  | :meth:`binascii.a2b_hex`     |
 |                      |                  | representation, with two     |                              |
-|                      |                  | digits per byte              |                              |
+|                      |                  | digits per byte.             |                              |
 +----------------------+------------------+------------------------------+------------------------------+
-| quopri_codec         | quopri,          | Convert operand to MIME      | :meth:`quopri.encode` with   |
-|                      | quotedprintable, | quoted printable             | ``quotetabs=True`` /         |
+| quopri_codec         | quopri,          | Convert the operand to MIME  | :meth:`quopri.encode` with   |
+|                      | quotedprintable, | quoted printable.            | ``quotetabs=True`` /         |
 |                      | quoted_printable |                              | :meth:`quopri.decode`        |
 +----------------------+------------------+------------------------------+------------------------------+
 | uu_codec             | uu               | Convert the operand using    | :meth:`uu.encode` /          |
-|                      |                  | uuencode                     | :meth:`uu.decode`            |
+|                      |                  | uuencode.                    | :meth:`uu.decode`            |
 +----------------------+------------------+------------------------------+------------------------------+
-| zlib_codec           | zip, zlib        | Compress the operand         | :meth:`zlib.compress` /      |
-|                      |                  | using gzip                   | :meth:`zlib.decompress`      |
+| zlib_codec           | zip, zlib        | Compress the operand using   | :meth:`zlib.compress` /      |
+|                      |                  | gzip.                        | :meth:`zlib.decompress`      |
 +----------------------+------------------+------------------------------+------------------------------+
 
 .. [#b64] In addition to :term:`bytes-like objects <bytes-like object>`,
@@ -1389,16 +1389,17 @@ Text Transforms
 ^^^^^^^^^^^^^^^
 
 The following codec provides a text transform: a :class:`str` to :class:`str`
-mapping.  It is not supported by :meth:`str.encode` (which only produces
+mapping. It is not supported by :meth:`str.encode` (which only produces
 :class:`bytes` output).
 
 .. tabularcolumns:: |l|l|L|
 
 +--------------------+---------+---------------------------+
-| Codec              | Aliases | Purpose                   |
+| Codec              | Aliases | Meaning                   |
 +====================+=========+===========================+
-| rot_13             | rot13   | Returns the Caesar-cypher |
-|                    |         | encryption of the operand |
+| rot_13             | rot13   | Return the Caesar-cypher  |
+|                    |         | encryption of the         |
+|                    |         | operand.                  |
 +--------------------+---------+---------------------------+
 
 .. versionadded:: 3.2
@@ -1436,7 +1437,7 @@ conversion between Unicode and ACE, separating an input string into labels
 based on the separator characters defined in :rfc:`section 3.1 of RFC 3490 <3490#section-3.1>`
 and converting each label to ACE as required, and conversely separating an input
 byte string into labels based on the ``.`` separator and converting any ACE
-labels found into unicode.  Furthermore, the :mod:`socket` module
+labels found into unicode. Furthermore, the :mod:`socket` module
 transparently converts Unicode host names to ACE, so that applications need not
 be concerned about converting host names themselves when they pass them to the
 socket module. On top of that, modules that have host names as function
@@ -1445,7 +1446,7 @@ names (:mod:`http.client` then also transparently sends an IDNA hostname in the
 :mailheader:`Host` field if it sends that field at all).
 
 When receiving host names from the wire (such as in reverse name lookup), no
-automatic conversion to Unicode is performed: Applications wishing to present
+automatic conversion to Unicode is performed: applications wishing to present
 such host names to the user should decode them to Unicode.
 
 The module :mod:`encodings.idna` also implements the nameprep procedure, which
@@ -1477,7 +1478,7 @@ functions can be used directly if desired.
 .. module:: encodings.mbcs
    :synopsis: Windows ANSI codepage
 
-Encode operand according to the ANSI codepage (CP_ACP).
+This module implements the ANSI codepage (CP_ACP).
 
 .. availability:: Windows only.
 
@@ -1496,7 +1497,7 @@ Encode operand according to the ANSI codepage (CP_ACP).
    :synopsis: UTF-8 codec with BOM signature
 .. moduleauthor:: Walter D?rwald
 
-This module implements a variant of the UTF-8 codec: On encoding a UTF-8 encoded
+This module implements a variant of the UTF-8 codec. On encoding, a UTF-8 encoded
 BOM will be prepended to the UTF-8 encoded bytes. For the stateful encoder this
-is only done once (on the first write to the byte stream).  For decoding an
+is only done once (on the first write to the byte stream). On decoding, an
 optional UTF-8 encoded BOM at the start of the data will be skipped.


From webhook-mailer at python.org  Tue Oct  1 17:02:33 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Tue, 01 Oct 2019 21:02:33 -0000
Subject: [Python-checkins] Correct typos in the codecs module documentation
 (GH-15135)
Message-ID: <mailman.180.1569963755.32504.python-checkins@python.org>

https://github.com/python/cpython/commit/c4976a60956627a5b256b7929e9ad04e1ea101ec
commit: c4976a60956627a5b256b7929e9ad04e1ea101ec
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-01T14:02:29-07:00
summary:

Correct typos in the codecs module documentation (GH-15135)

(cherry picked from commit 891e9e3b44c99d643dc309a4e63082451271b136)

Co-authored-by: G?ry Ogam <gery.ogam at gmail.com>

files:
M Doc/library/codecs.rst

diff --git a/Doc/library/codecs.rst b/Doc/library/codecs.rst
index 2e9314e0fab79..992672e806f33 100644
--- a/Doc/library/codecs.rst
+++ b/Doc/library/codecs.rst
@@ -292,7 +292,7 @@ Error Handlers
 
 To simplify and standardize error handling,
 codecs may implement different error handling schemes by
-accepting the *errors* string argument.  The following string values are
+accepting the *errors* string argument. The following string values are
 defined and implemented by all standard Python codecs:
 
 .. tabularcolumns:: |l|L|
@@ -301,11 +301,11 @@ defined and implemented by all standard Python codecs:
 | Value                   | Meaning                                       |
 +=========================+===============================================+
 | ``'strict'``            | Raise :exc:`UnicodeError` (or a subclass);    |
-|                         | this is the default.  Implemented in          |
+|                         | this is the default. Implemented in           |
 |                         | :func:`strict_errors`.                        |
 +-------------------------+-----------------------------------------------+
 | ``'ignore'``            | Ignore the malformed data and continue        |
-|                         | without further notice.  Implemented in       |
+|                         | without further notice. Implemented in        |
 |                         | :func:`ignore_errors`.                        |
 +-------------------------+-----------------------------------------------+
 
@@ -327,11 +327,11 @@ The following error handlers are only applicable to
 |                         | marker; Python will use the official          |
 |                         | ``U+FFFD`` REPLACEMENT CHARACTER for the      |
 |                         | built-in codecs on decoding, and '?' on       |
-|                         | encoding.  Implemented in                     |
+|                         | encoding. Implemented in                      |
 |                         | :func:`replace_errors`.                       |
 +-------------------------+-----------------------------------------------+
 | ``'xmlcharrefreplace'`` | Replace with the appropriate XML character    |
-|                         | reference (only for encoding).  Implemented   |
+|                         | reference (only for encoding). Implemented    |
 |                         | in :func:`xmlcharrefreplace_errors`.          |
 +-------------------------+-----------------------------------------------+
 | ``'backslashreplace'``  | Replace with backslashed escape sequences.    |
@@ -339,15 +339,15 @@ The following error handlers are only applicable to
 |                         | :func:`backslashreplace_errors`.              |
 +-------------------------+-----------------------------------------------+
 | ``'namereplace'``       | Replace with ``\N{...}`` escape sequences     |
-|                         | (only for encoding).  Implemented in          |
+|                         | (only for encoding). Implemented in           |
 |                         | :func:`namereplace_errors`.                   |
 +-------------------------+-----------------------------------------------+
 | ``'surrogateescape'``   | On decoding, replace byte with individual     |
 |                         | surrogate code ranging from ``U+DC80`` to     |
-|                         | ``U+DCFF``.  This code will then be turned    |
+|                         | ``U+DCFF``. This code will then be turned     |
 |                         | back into the same byte when the              |
 |                         | ``'surrogateescape'`` error handler is used   |
-|                         | when encoding the data.  (See :pep:`383` for  |
+|                         | when encoding the data. (See :pep:`383` for   |
 |                         | more.)                                        |
 +-------------------------+-----------------------------------------------+
 
@@ -357,7 +357,7 @@ In addition, the following error handler is specific to the given codecs:
 | Value             | Codecs                 | Meaning                                   |
 +===================+========================+===========================================+
 |``'surrogatepass'``| utf-8, utf-16, utf-32, | Allow encoding and decoding of surrogate  |
-|                   | utf-16-be, utf-16-le,  | codes.  These codecs normally treat the   |
+|                   | utf-16-be, utf-16-le,  | codes. These codecs normally treat the    |
 |                   | utf-32-be, utf-32-le   | presence of surrogates as an error.       |
 +-------------------+------------------------+-------------------------------------------+
 
@@ -388,9 +388,9 @@ handler:
    error handler must either raise this or a different exception, or return a
    tuple with a replacement for the unencodable part of the input and a position
    where encoding should continue. The replacement may be either :class:`str` or
-   :class:`bytes`.  If the replacement is bytes, the encoder will simply copy
+   :class:`bytes`. If the replacement is bytes, the encoder will simply copy
    them into the output buffer. If the replacement is a string, the encoder will
-   encode the replacement.  Encoding continues on original input at the
+   encode the replacement. Encoding continues on original input at the
    specified position. Negative position values will be treated as being
    relative to the end of the input string. If the resulting position is out of
    bound an :exc:`IndexError` will be raised.
@@ -484,7 +484,7 @@ function interfaces of the stateless encoder and decoder:
 .. method:: Codec.decode(input[, errors])
 
    Decodes the object *input* and returns a tuple (output object, length
-   consumed).  For instance, for a :term:`text encoding`, decoding converts
+   consumed). For instance, for a :term:`text encoding`, decoding converts
    a bytes object encoded using a particular
    character set encoding to a string object.
 
@@ -568,7 +568,7 @@ define in order to be compatible with the Python codec registry.
       implementation should make sure that ``0`` is the most common
       state. (States that are more complicated than integers can be converted
       into an integer by marshaling/pickling the state and encoding the bytes
-      of the resulting string into an integer).
+      of the resulting string into an integer.)
 
 
    .. method:: setstate(state)
@@ -751,7 +751,7 @@ compatible with the Python codec registry.
       number of encoded bytes or code points to read
       for decoding. The decoder can modify this setting as
       appropriate. The default value -1 indicates to read and decode as much as
-      possible.  This parameter is intended to
+      possible. This parameter is intended to
       prevent having to decode huge files in one step.
 
       The *firstline* flag indicates that
@@ -780,8 +780,8 @@ compatible with the Python codec registry.
       Read all lines available on the input stream and return them as a list of
       lines.
 
-      Line-endings are implemented using the codec's decoder method and are
-      included in the list entries if *keepends* is true.
+      Line-endings are implemented using the codec's :meth:`decode` method and
+      are included in the list entries if *keepends* is true.
 
       *sizehint*, if given, is passed as the *size* argument to the stream's
       :meth:`read` method.
@@ -791,7 +791,7 @@ compatible with the Python codec registry.
 
       Resets the codec buffers used for keeping state.
 
-      Note that no stream repositioning should take place.  This method is
+      Note that no stream repositioning should take place. This method is
       primarily intended to be able to recover from decoding errors.
 
 
@@ -841,7 +841,7 @@ The design is such that one can use the factory functions returned by the
    code calling :meth:`read` and :meth:`write`, while *Reader* and *Writer*
    work on the backend?? the data in *stream*.
 
-   You can use these objects to do transparent transcodings from e.g. Latin-1
+   You can use these objects to do transparent transcodings, e.g., from Latin-1
    to UTF-8 and back.
 
    The *stream* argument must be a file-like object.
@@ -866,10 +866,10 @@ Encodings and Unicode
 ---------------------
 
 Strings are stored internally as sequences of code points in
-range ``0x0``--``0x10FFFF``.  (See :pep:`393` for
+range ``0x0``--``0x10FFFF``. (See :pep:`393` for
 more details about the implementation.)
 Once a string object is used outside of CPU and memory, endianness
-and how these arrays are stored as bytes become an issue.  As with other
+and how these arrays are stored as bytes become an issue. As with other
 codecs, serialising a string into a sequence of bytes is known as *encoding*,
 and recreating the string from the sequence of bytes is known as *decoding*.
 There are a variety of different text serialisation codecs, which are
@@ -964,7 +964,7 @@ to determine the byte order used for generating the byte sequence, but as a
 signature that helps in guessing the encoding. On encoding the utf-8-sig codec
 will write ``0xef``, ``0xbb``, ``0xbf`` as the first three bytes to the file. On
 decoding ``utf-8-sig`` will skip those three bytes if they appear as the first
-three bytes in the file.  In UTF-8, the use of the BOM is discouraged and
+three bytes in the file. In UTF-8, the use of the BOM is discouraged and
 should generally be avoided.
 
 
@@ -984,7 +984,7 @@ e.g. ``'utf-8'`` is a valid alias for the ``'utf_8'`` codec.
 .. impl-detail::
 
    Some common encodings can bypass the codecs lookup machinery to
-   improve performance.  These optimization opportunities are only
+   improve performance. These optimization opportunities are only
    recognized by CPython for a limited set of (case insensitive)
    aliases: utf-8, utf8, latin-1, latin1, iso-8859-1, iso8859-1, mbcs
    (Windows only), ascii, us-ascii, utf-16, utf16, utf-32, utf32, and
@@ -1145,7 +1145,7 @@ particular, the following variants typically exist:
 | iso2022_kr      | csiso2022kr, iso2022kr,        | Korean                         |
 |                 | iso-2022-kr                    |                                |
 +-----------------+--------------------------------+--------------------------------+
-| latin_1         | iso-8859-1, iso8859-1, 8859,   | West Europe                    |
+| latin_1         | iso-8859-1, iso8859-1, 8859,   | Western Europe                 |
 |                 | cp819, latin, latin1, L1       |                                |
 +-----------------+--------------------------------+--------------------------------+
 | iso8859_2       | iso-8859-2, latin2, L2         | Central and Eastern Europe     |
@@ -1249,11 +1249,11 @@ Python Specific Encodings
 -------------------------
 
 A number of predefined codecs are specific to Python, so their codec names have
-no meaning outside Python.  These are listed in the tables below based on the
+no meaning outside Python. These are listed in the tables below based on the
 expected input and output types (note that while text encodings are the most
 common use case for codecs, the underlying codec infrastructure supports
-arbitrary data transforms rather than just text encodings).  For asymmetric
-codecs, the stated purpose describes the encoding direction.
+arbitrary data transforms rather than just text encodings). For asymmetric
+codecs, the stated meaning describes the encoding direction.
 
 Text Encodings
 ^^^^^^^^^^^^^^
@@ -1265,27 +1265,27 @@ encodings.
 .. tabularcolumns:: |l|p{0.3\linewidth}|p{0.3\linewidth}|
 
 +--------------------+---------+---------------------------+
-| Codec              | Aliases | Purpose                   |
+| Codec              | Aliases | Meaning                   |
 +====================+=========+===========================+
-| idna               |         | Implements :rfc:`3490`,   |
+| idna               |         | Implement :rfc:`3490`,    |
 |                    |         | see also                  |
 |                    |         | :mod:`encodings.idna`.    |
 |                    |         | Only ``errors='strict'``  |
 |                    |         | is supported.             |
 +--------------------+---------+---------------------------+
-| mbcs               | ansi,   | Windows only: Encode      |
+| mbcs               | ansi,   | Windows only: Encode the  |
 |                    | dbcs    | operand according to the  |
-|                    |         | ANSI codepage (CP_ACP)    |
+|                    |         | ANSI codepage (CP_ACP).   |
 +--------------------+---------+---------------------------+
-| oem                |         | Windows only: Encode      |
+| oem                |         | Windows only: Encode the  |
 |                    |         | operand according to the  |
-|                    |         | OEM codepage (CP_OEMCP)   |
+|                    |         | OEM codepage (CP_OEMCP).  |
 |                    |         |                           |
 |                    |         | .. versionadded:: 3.6     |
 +--------------------+---------+---------------------------+
-| palmos             |         | Encoding of PalmOS 3.5    |
+| palmos             |         | Encoding of PalmOS 3.5.   |
 +--------------------+---------+---------------------------+
-| punycode           |         | Implements :rfc:`3492`.   |
+| punycode           |         | Implement :rfc:`3492`.    |
 |                    |         | Stateful codecs are not   |
 |                    |         | supported.                |
 +--------------------+---------+---------------------------+
@@ -1308,8 +1308,8 @@ encodings.
 |                    |         | literal in ASCII-encoded  |
 |                    |         | Python source code,       |
 |                    |         | except that quotes are    |
-|                    |         | not escaped. Decodes from |
-|                    |         | Latin-1 source code.      |
+|                    |         | not escaped. Decode       |
+|                    |         | from Latin-1 source code. |
 |                    |         | Beware that Python source |
 |                    |         | code actually uses UTF-8  |
 |                    |         | by default.               |
@@ -1325,19 +1325,19 @@ Binary Transforms
 ^^^^^^^^^^^^^^^^^
 
 The following codecs provide binary transforms: :term:`bytes-like object`
-to :class:`bytes` mappings.  They are not supported by :meth:`bytes.decode`
+to :class:`bytes` mappings. They are not supported by :meth:`bytes.decode`
 (which only produces :class:`str` output).
 
 
 .. tabularcolumns:: |l|L|L|L|
 
 +----------------------+------------------+------------------------------+------------------------------+
-| Codec                | Aliases          | Purpose                      | Encoder / decoder            |
+| Codec                | Aliases          | Meaning                      | Encoder / decoder            |
 +======================+==================+==============================+==============================+
-| base64_codec [#b64]_ | base64, base_64  | Convert operand to multiline | :meth:`base64.encodebytes` / |
-|                      |                  | MIME base64 (the result      | :meth:`base64.decodebytes`   |
-|                      |                  | always includes a trailing   |                              |
-|                      |                  | ``'\n'``)                    |                              |
+| base64_codec [#b64]_ | base64, base_64  | Convert the operand to       | :meth:`base64.encodebytes` / |
+|                      |                  | multiline MIME base64 (the   | :meth:`base64.decodebytes`   |
+|                      |                  | result always includes a     |                              |
+|                      |                  | trailing ``'\n'``).          |                              |
 |                      |                  |                              |                              |
 |                      |                  | .. versionchanged:: 3.4      |                              |
 |                      |                  |    accepts any               |                              |
@@ -1345,23 +1345,23 @@ to :class:`bytes` mappings.  They are not supported by :meth:`bytes.decode`
 |                      |                  |    as input for encoding and |                              |
 |                      |                  |    decoding                  |                              |
 +----------------------+------------------+------------------------------+------------------------------+
-| bz2_codec            | bz2              | Compress the operand         | :meth:`bz2.compress` /       |
-|                      |                  | using bz2                    | :meth:`bz2.decompress`       |
+| bz2_codec            | bz2              | Compress the operand using   | :meth:`bz2.compress` /       |
+|                      |                  | bz2.                         | :meth:`bz2.decompress`       |
 +----------------------+------------------+------------------------------+------------------------------+
-| hex_codec            | hex              | Convert operand to           | :meth:`binascii.b2a_hex` /   |
+| hex_codec            | hex              | Convert the operand to       | :meth:`binascii.b2a_hex` /   |
 |                      |                  | hexadecimal                  | :meth:`binascii.a2b_hex`     |
 |                      |                  | representation, with two     |                              |
-|                      |                  | digits per byte              |                              |
+|                      |                  | digits per byte.             |                              |
 +----------------------+------------------+------------------------------+------------------------------+
-| quopri_codec         | quopri,          | Convert operand to MIME      | :meth:`quopri.encode` with   |
-|                      | quotedprintable, | quoted printable             | ``quotetabs=True`` /         |
+| quopri_codec         | quopri,          | Convert the operand to MIME  | :meth:`quopri.encode` with   |
+|                      | quotedprintable, | quoted printable.            | ``quotetabs=True`` /         |
 |                      | quoted_printable |                              | :meth:`quopri.decode`        |
 +----------------------+------------------+------------------------------+------------------------------+
 | uu_codec             | uu               | Convert the operand using    | :meth:`uu.encode` /          |
-|                      |                  | uuencode                     | :meth:`uu.decode`            |
+|                      |                  | uuencode.                    | :meth:`uu.decode`            |
 +----------------------+------------------+------------------------------+------------------------------+
-| zlib_codec           | zip, zlib        | Compress the operand         | :meth:`zlib.compress` /      |
-|                      |                  | using gzip                   | :meth:`zlib.decompress`      |
+| zlib_codec           | zip, zlib        | Compress the operand using   | :meth:`zlib.compress` /      |
+|                      |                  | gzip.                        | :meth:`zlib.decompress`      |
 +----------------------+------------------+------------------------------+------------------------------+
 
 .. [#b64] In addition to :term:`bytes-like objects <bytes-like object>`,
@@ -1381,16 +1381,17 @@ Text Transforms
 ^^^^^^^^^^^^^^^
 
 The following codec provides a text transform: a :class:`str` to :class:`str`
-mapping.  It is not supported by :meth:`str.encode` (which only produces
+mapping. It is not supported by :meth:`str.encode` (which only produces
 :class:`bytes` output).
 
 .. tabularcolumns:: |l|l|L|
 
 +--------------------+---------+---------------------------+
-| Codec              | Aliases | Purpose                   |
+| Codec              | Aliases | Meaning                   |
 +====================+=========+===========================+
-| rot_13             | rot13   | Returns the Caesar-cypher |
-|                    |         | encryption of the operand |
+| rot_13             | rot13   | Return the Caesar-cypher  |
+|                    |         | encryption of the         |
+|                    |         | operand.                  |
 +--------------------+---------+---------------------------+
 
 .. versionadded:: 3.2
@@ -1428,7 +1429,7 @@ conversion between Unicode and ACE, separating an input string into labels
 based on the separator characters defined in :rfc:`section 3.1 of RFC 3490 <3490#section-3.1>`
 and converting each label to ACE as required, and conversely separating an input
 byte string into labels based on the ``.`` separator and converting any ACE
-labels found into unicode.  Furthermore, the :mod:`socket` module
+labels found into unicode. Furthermore, the :mod:`socket` module
 transparently converts Unicode host names to ACE, so that applications need not
 be concerned about converting host names themselves when they pass them to the
 socket module. On top of that, modules that have host names as function
@@ -1437,7 +1438,7 @@ names (:mod:`http.client` then also transparently sends an IDNA hostname in the
 :mailheader:`Host` field if it sends that field at all).
 
 When receiving host names from the wire (such as in reverse name lookup), no
-automatic conversion to Unicode is performed: Applications wishing to present
+automatic conversion to Unicode is performed: applications wishing to present
 such host names to the user should decode them to Unicode.
 
 The module :mod:`encodings.idna` also implements the nameprep procedure, which
@@ -1469,7 +1470,7 @@ functions can be used directly if desired.
 .. module:: encodings.mbcs
    :synopsis: Windows ANSI codepage
 
-Encode operand according to the ANSI codepage (CP_ACP).
+This module implements the ANSI codepage (CP_ACP).
 
 .. availability:: Windows only.
 
@@ -1488,7 +1489,7 @@ Encode operand according to the ANSI codepage (CP_ACP).
    :synopsis: UTF-8 codec with BOM signature
 .. moduleauthor:: Walter D?rwald
 
-This module implements a variant of the UTF-8 codec: On encoding a UTF-8 encoded
+This module implements a variant of the UTF-8 codec. On encoding, a UTF-8 encoded
 BOM will be prepended to the UTF-8 encoded bytes. For the stateful encoder this
-is only done once (on the first write to the byte stream).  For decoding an
+is only done once (on the first write to the byte stream). On decoding, an
 optional UTF-8 encoded BOM at the start of the data will be skipped.


From webhook-mailer at python.org  Tue Oct  1 19:42:32 2019
From: webhook-mailer at python.org (Steve Dower)
Date: Tue, 01 Oct 2019 23:42:32 -0000
Subject: [Python-checkins] bpo-38343: Fixes version handling for nuget
 packages (GH-16527)
Message-ID: <mailman.181.1569973354.32504.python-checkins@python.org>

https://github.com/python/cpython/commit/b9a8b8296cd7be22f8b5bf92af686a788c47c7bf
commit: b9a8b8296cd7be22f8b5bf92af686a788c47c7bf
branch: master
author: Steve Dower <steve.dower at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-01T16:42:24-07:00
summary:

bpo-38343: Fixes version handling for nuget packages (GH-16527)

files:
M PC/layout/support/constants.py

diff --git a/PC/layout/support/constants.py b/PC/layout/support/constants.py
index 88ea410b340ea..d76fa3bbf3b47 100644
--- a/PC/layout/support/constants.py
+++ b/PC/layout/support/constants.py
@@ -10,7 +10,7 @@
 
 VER_MAJOR, VER_MINOR, VER_MICRO, VER_FIELD4 = struct.pack(">i", sys.hexversion)
 VER_FIELD3 = VER_MICRO << 8 | VER_FIELD4
-VER_NAME = {"alpha": "a", "beta": "b", "rc": "rc"}.get(
+VER_NAME = {"alpha": "a", "beta": "b", "candidate": "rc"}.get(
     sys.version_info.releaselevel, ""
 )
 VER_SERIAL = sys.version_info.serial if VER_NAME else ""


From webhook-mailer at python.org  Tue Oct  1 20:00:53 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Wed, 02 Oct 2019 00:00:53 -0000
Subject: [Python-checkins] bpo-38343: Fixes version handling for nuget
 packages (GH-16527)
Message-ID: <mailman.182.1569974454.32504.python-checkins@python.org>

https://github.com/python/cpython/commit/a11df75269b5d47248be617ed02e96c1d1938fd1
commit: a11df75269b5d47248be617ed02e96c1d1938fd1
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-01T17:00:49-07:00
summary:

bpo-38343: Fixes version handling for nuget packages (GH-16527)

(cherry picked from commit b9a8b8296cd7be22f8b5bf92af686a788c47c7bf)

Co-authored-by: Steve Dower <steve.dower at python.org>

files:
M PC/layout/support/constants.py

diff --git a/PC/layout/support/constants.py b/PC/layout/support/constants.py
index 88ea410b340ea..d76fa3bbf3b47 100644
--- a/PC/layout/support/constants.py
+++ b/PC/layout/support/constants.py
@@ -10,7 +10,7 @@
 
 VER_MAJOR, VER_MINOR, VER_MICRO, VER_FIELD4 = struct.pack(">i", sys.hexversion)
 VER_FIELD3 = VER_MICRO << 8 | VER_FIELD4
-VER_NAME = {"alpha": "a", "beta": "b", "rc": "rc"}.get(
+VER_NAME = {"alpha": "a", "beta": "b", "candidate": "rc"}.get(
     sys.version_info.releaselevel, ""
 )
 VER_SERIAL = sys.version_info.serial if VER_NAME else ""


From webhook-mailer at python.org  Tue Oct  1 23:58:39 2019
From: webhook-mailer at python.org (Ned Deily)
Date: Wed, 02 Oct 2019 03:58:39 -0000
Subject: [Python-checkins] Update macOS installer displays for 3.7.5rc1
Message-ID: <mailman.183.1569988721.32504.python-checkins@python.org>

https://github.com/python/cpython/commit/960068b5672abba766ae09059b53fa2013cabacc
commit: 960068b5672abba766ae09059b53fa2013cabacc
branch: 3.7
author: Ned Deily <nad at python.org>
committer: Ned Deily <nad at python.org>
date: 2019-10-01T12:59:45-04:00
summary:

Update macOS installer displays for 3.7.5rc1

files:
M Mac/BuildScript/resources/ReadMe.rtf
M Mac/BuildScript/resources/Welcome.rtf

diff --git a/Mac/BuildScript/resources/ReadMe.rtf b/Mac/BuildScript/resources/ReadMe.rtf
index 476d531167592..f933d74bda14d 100644
--- a/Mac/BuildScript/resources/ReadMe.rtf
+++ b/Mac/BuildScript/resources/ReadMe.rtf
@@ -1,4 +1,4 @@
-{\rtf1\ansi\ansicpg1252\cocoartf1671\cocoasubrtf500
+{\rtf1\ansi\ansicpg1252\cocoartf1671\cocoasubrtf600
 {\fonttbl\f0\fswiss\fcharset0 Helvetica;\f1\fswiss\fcharset0 Helvetica-Bold;\f2\fswiss\fcharset0 Helvetica-Oblique;
 \f3\fmodern\fcharset0 CourierNewPSMT;}
 {\colortbl;\red255\green255\blue255;}
@@ -10,6 +10,16 @@
 \
 \pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural\partightenfactor0
 
+\f1\b \cf0 \ul \ulc0 macOS 10.15 Catalina Support\
+\
+\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\partightenfactor0
+
+\f0\b0 \cf0 \ulnone At the time of the release of this release candidate, the next release of macOS, 
+\f1\b 10.15 Catalina
+\f0\b0 , was still in public beta.  Any updates needed to support 10.15 will be provided after it releases.\
+\
+\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural\partightenfactor0
+
 \f1\b \cf0 \ul \ulc0 Which installer variant should I use?
 \f0\b0 \ulnone \
 \
@@ -81,7 +91,7 @@ Other changes\
 \pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\partightenfactor0
 \cf0 \ulnone CHANGED in 3.7.4: 
 \f0\b0  SQLite has been updated from 3.22.0 to 3.28.0
-\f1\b \ul \ulc0 \
+\f1\b \ul \
 \pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural\partightenfactor0
 
 \f0\b0 \cf0 \ulnone \
diff --git a/Mac/BuildScript/resources/Welcome.rtf b/Mac/BuildScript/resources/Welcome.rtf
index fd0170d276430..67361e7c176af 100644
--- a/Mac/BuildScript/resources/Welcome.rtf
+++ b/Mac/BuildScript/resources/Welcome.rtf
@@ -1,4 +1,4 @@
-{\rtf1\ansi\ansicpg1252\cocoartf1671\cocoasubrtf500
+{\rtf1\ansi\ansicpg1252\cocoartf1671\cocoasubrtf600
 \cocoascreenfonts1{\fonttbl\f0\fswiss\fcharset0 Helvetica;\f1\fswiss\fcharset0 Helvetica-Bold;\f2\fmodern\fcharset0 CourierNewPSMT;
 }
 {\colortbl;\red255\green255\blue255;}
@@ -29,4 +29,8 @@ At the end of this install, click on
 \f1\b \
 CHANGED in 3.7.4: 
 \f0\b0 OpenSSL 1.1.1, SQLite 3.28.0, 10.6+ 64-/32-bit installer variant deprecated\
+\
+At the time of the release of this release candidate, the next release of macOS, 
+\f1\b 10.15 Catalina
+\f0\b0 , was still in public beta.  Any updates needed to support 10.15 will be provided after it releases.\
 }
\ No newline at end of file


From webhook-mailer at python.org  Wed Oct  2 07:35:28 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Wed, 02 Oct 2019 11:35:28 -0000
Subject: [Python-checkins] bpo-36670: regrtest bug fixes (GH-16537)
Message-ID: <mailman.184.1570016128.32504.python-checkins@python.org>

https://github.com/python/cpython/commit/2ea71a07d0a720707094ee55f78fd232c40724bc
commit: 2ea71a07d0a720707094ee55f78fd232c40724bc
branch: master
author: Victor Stinner <vstinner at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-02T13:35:11+02:00
summary:

bpo-36670: regrtest bug fixes (GH-16537)

* Fix TestWorkerProcess.__repr__(): start_time is only valid
  if _popen is not None.
* Fix _kill(): don't set _killed to True if _popen is None.
* _run_process(): only set _killed to False after calling
  run_test_in_subprocess().

files:
M Lib/test/libregrtest/runtest_mp.py

diff --git a/Lib/test/libregrtest/runtest_mp.py b/Lib/test/libregrtest/runtest_mp.py
index 38b05781de5fc..a46c78248de39 100644
--- a/Lib/test/libregrtest/runtest_mp.py
+++ b/Lib/test/libregrtest/runtest_mp.py
@@ -120,27 +120,28 @@ def __init__(self, worker_id, pending, output, ns, timeout):
     def __repr__(self):
         info = [f'TestWorkerProcess #{self.worker_id}']
         if self.is_alive():
-            dt = time.monotonic() - self.start_time
-            info.append("running for %s" % format_duration(dt))
+            info.append("running")
         else:
             info.append('stopped')
         test = self.current_test_name
         if test:
             info.append(f'test={test}')
         popen = self._popen
-        if popen:
-            info.append(f'pid={popen.pid}')
+        if popen is not None:
+            dt = time.monotonic() - self.start_time
+            info.extend((f'pid={self._popen.pid}',
+                         f'time={format_duration(dt)}'))
         return '<%s>' % ' '.join(info)
 
     def _kill(self):
-        if self._killed:
-            return
-        self._killed = True
-
         popen = self._popen
         if popen is None:
             return
 
+        if self._killed:
+            return
+        self._killed = True
+
         print(f"Kill {self}", file=sys.stderr, flush=True)
         try:
             popen.kill()
@@ -177,9 +178,10 @@ def _run_process(self, test_name):
 
         self.current_test_name = test_name
         try:
+            popen = run_test_in_subprocess(test_name, self.ns)
+
             self._killed = False
-            self._popen = run_test_in_subprocess(test_name, self.ns)
-            popen = self._popen
+            self._popen = popen
         except:
             self.current_test_name = None
             raise


From webhook-mailer at python.org  Wed Oct  2 07:54:23 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Wed, 02 Oct 2019 11:54:23 -0000
Subject: [Python-checkins] bpo-36670: regrtest bug fixes (GH-16537)
Message-ID: <mailman.185.1570017264.32504.python-checkins@python.org>

https://github.com/python/cpython/commit/a72de9338882b8013a4bb8adb930fe3308682e28
commit: a72de9338882b8013a4bb8adb930fe3308682e28
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-02T04:54:18-07:00
summary:

bpo-36670: regrtest bug fixes (GH-16537)


* Fix TestWorkerProcess.__repr__(): start_time is only valid
  if _popen is not None.
* Fix _kill(): don't set _killed to True if _popen is None.
* _run_process(): only set _killed to False after calling
  run_test_in_subprocess().
(cherry picked from commit 2ea71a07d0a720707094ee55f78fd232c40724bc)

Co-authored-by: Victor Stinner <vstinner at python.org>

files:
M Lib/test/libregrtest/runtest_mp.py

diff --git a/Lib/test/libregrtest/runtest_mp.py b/Lib/test/libregrtest/runtest_mp.py
index 38b05781de5fc..a46c78248de39 100644
--- a/Lib/test/libregrtest/runtest_mp.py
+++ b/Lib/test/libregrtest/runtest_mp.py
@@ -120,27 +120,28 @@ def __init__(self, worker_id, pending, output, ns, timeout):
     def __repr__(self):
         info = [f'TestWorkerProcess #{self.worker_id}']
         if self.is_alive():
-            dt = time.monotonic() - self.start_time
-            info.append("running for %s" % format_duration(dt))
+            info.append("running")
         else:
             info.append('stopped')
         test = self.current_test_name
         if test:
             info.append(f'test={test}')
         popen = self._popen
-        if popen:
-            info.append(f'pid={popen.pid}')
+        if popen is not None:
+            dt = time.monotonic() - self.start_time
+            info.extend((f'pid={self._popen.pid}',
+                         f'time={format_duration(dt)}'))
         return '<%s>' % ' '.join(info)
 
     def _kill(self):
-        if self._killed:
-            return
-        self._killed = True
-
         popen = self._popen
         if popen is None:
             return
 
+        if self._killed:
+            return
+        self._killed = True
+
         print(f"Kill {self}", file=sys.stderr, flush=True)
         try:
             popen.kill()
@@ -177,9 +178,10 @@ def _run_process(self, test_name):
 
         self.current_test_name = test_name
         try:
+            popen = run_test_in_subprocess(test_name, self.ns)
+
             self._killed = False
-            self._popen = run_test_in_subprocess(test_name, self.ns)
-            popen = self._popen
+            self._popen = popen
         except:
             self.current_test_name = None
             raise


From webhook-mailer at python.org  Wed Oct  2 11:52:51 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Wed, 02 Oct 2019 15:52:51 -0000
Subject: [Python-checkins] bpo-38338,
 test.pythoninfo: add more ssl infos (GH-16539)
Message-ID: <mailman.186.1570031572.32504.python-checkins@python.org>

https://github.com/python/cpython/commit/b3e7045f8314e7b62cd95861d207fe2f97e47198
commit: b3e7045f8314e7b62cd95861d207fe2f97e47198
branch: master
author: Victor Stinner <vstinner at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-02T17:52:35+02:00
summary:

bpo-38338, test.pythoninfo: add more ssl infos (GH-16539)

test.pythoninfo now logs environment variables used by OpenSSL and
Python ssl modules, and logs attributes of 3 SSL contexts
(SSLContext, default HTTPS context, stdlib context).

files:
M Lib/test/pythoninfo.py

diff --git a/Lib/test/pythoninfo.py b/Lib/test/pythoninfo.py
index d2fa6c5936596..797b3af7d3085 100644
--- a/Lib/test/pythoninfo.py
+++ b/Lib/test/pythoninfo.py
@@ -470,10 +470,15 @@ def collect_sysconfig(info_add):
 
 
 def collect_ssl(info_add):
+    import os
     try:
         import ssl
     except ImportError:
         return
+    try:
+        import _ssl
+    except ImportError:
+        _ssl = None
 
     def format_attr(attr, value):
         if attr.startswith('OP_'):
@@ -490,6 +495,32 @@ def format_attr(attr, value):
     )
     copy_attributes(info_add, ssl, 'ssl.%s', attributes, formatter=format_attr)
 
+    for name, ctx in (
+        ('SSLContext', ssl.SSLContext()),
+        ('default_https_context', ssl._create_default_https_context()),
+        ('stdlib_context', ssl._create_stdlib_context()),
+    ):
+        attributes = (
+            'minimum_version',
+            'maximum_version',
+            'protocol',
+            'options',
+            'verify_mode',
+        )
+        copy_attributes(info_add, ctx, f'ssl.{name}.%s', attributes)
+
+    env_names = ["OPENSSL_CONF", "SSLKEYLOGFILE"]
+    if _ssl is not None and hasattr(_ssl, 'get_default_verify_paths'):
+        parts = _ssl.get_default_verify_paths()
+        env_names.extend((parts[0], parts[2]))
+
+    for name in env_names:
+        try:
+            value = os.environ[name]
+        except KeyError:
+            continue
+        info_add('ssl.environ[%s]' % name, value)
+
 
 def collect_socket(info_add):
     import socket


From webhook-mailer at python.org  Wed Oct  2 12:15:01 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Wed, 02 Oct 2019 16:15:01 -0000
Subject: [Python-checkins] bpo-38338,
 test.pythoninfo: add more ssl infos (GH-16539)
Message-ID: <mailman.187.1570032902.32504.python-checkins@python.org>

https://github.com/python/cpython/commit/ab98cd8aee5a5a7222b82ff13d61f0d33e72a889
commit: ab98cd8aee5a5a7222b82ff13d61f0d33e72a889
branch: 3.7
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-02T09:14:57-07:00
summary:

bpo-38338, test.pythoninfo: add more ssl infos (GH-16539)


test.pythoninfo now logs environment variables used by OpenSSL and
Python ssl modules, and logs attributes of 3 SSL contexts
(SSLContext, default HTTPS context, stdlib context).
(cherry picked from commit b3e7045f8314e7b62cd95861d207fe2f97e47198)

Co-authored-by: Victor Stinner <vstinner at python.org>

files:
M Lib/test/pythoninfo.py

diff --git a/Lib/test/pythoninfo.py b/Lib/test/pythoninfo.py
index 580956633f4d4..9a9d26dca490c 100644
--- a/Lib/test/pythoninfo.py
+++ b/Lib/test/pythoninfo.py
@@ -439,10 +439,15 @@ def collect_sysconfig(info_add):
 
 
 def collect_ssl(info_add):
+    import os
     try:
         import ssl
     except ImportError:
         return
+    try:
+        import _ssl
+    except ImportError:
+        _ssl = None
 
     def format_attr(attr, value):
         if attr.startswith('OP_'):
@@ -459,6 +464,32 @@ def format_attr(attr, value):
     )
     copy_attributes(info_add, ssl, 'ssl.%s', attributes, formatter=format_attr)
 
+    for name, ctx in (
+        ('SSLContext', ssl.SSLContext()),
+        ('default_https_context', ssl._create_default_https_context()),
+        ('stdlib_context', ssl._create_stdlib_context()),
+    ):
+        attributes = (
+            'minimum_version',
+            'maximum_version',
+            'protocol',
+            'options',
+            'verify_mode',
+        )
+        copy_attributes(info_add, ctx, f'ssl.{name}.%s', attributes)
+
+    env_names = ["OPENSSL_CONF", "SSLKEYLOGFILE"]
+    if _ssl is not None and hasattr(_ssl, 'get_default_verify_paths'):
+        parts = _ssl.get_default_verify_paths()
+        env_names.extend((parts[0], parts[2]))
+
+    for name in env_names:
+        try:
+            value = os.environ[name]
+        except KeyError:
+            continue
+        info_add('ssl.environ[%s]' % name, value)
+
 
 def collect_socket(info_add):
     import socket


From webhook-mailer at python.org  Wed Oct  2 12:36:40 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Wed, 02 Oct 2019 16:36:40 -0000
Subject: [Python-checkins] [2.7] bpo-38338,
 test.pythoninfo: add more ssl infos (GH-16543)
Message-ID: <mailman.188.1570034202.32504.python-checkins@python.org>

https://github.com/python/cpython/commit/403ca7ea70232e520af18511fbfb89b58ef2a046
commit: 403ca7ea70232e520af18511fbfb89b58ef2a046
branch: 2.7
author: Victor Stinner <vstinner at redhat.com>
committer: GitHub <noreply at github.com>
date: 2019-10-02T18:36:32+02:00
summary:

[2.7] bpo-38338, test.pythoninfo: add more ssl infos (GH-16543)

test.pythoninfo now logs environment variables used by OpenSSL and
Python ssl modules, and logs attributes of 3 SSL contexts
(SSLContext, default HTTPS context, stdlib context).

(cherry picked from commit 1df1c2f8df53d005ff47af81aa02c58752b84e20)

files:
M Lib/test/pythoninfo.py

diff --git a/Lib/test/pythoninfo.py b/Lib/test/pythoninfo.py
index a6983ba021c67..12de99b09d5e2 100644
--- a/Lib/test/pythoninfo.py
+++ b/Lib/test/pythoninfo.py
@@ -439,10 +439,15 @@ def collect_sysconfig(info_add):
 
 
 def collect_ssl(info_add):
+    import os
     try:
         import ssl
     except ImportError:
         return
+    try:
+        import _ssl
+    except ImportError:
+        _ssl = None
 
     def format_attr(attr, value):
         if attr.startswith('OP_'):
@@ -459,6 +464,61 @@ def format_attr(attr, value):
     )
     copy_attributes(info_add, ssl, 'ssl.%s', attributes, formatter=format_attr)
 
+    options_names = []
+    protocol_names = {}
+    verify_modes = {}
+    for name in dir(ssl):
+        if name.startswith('OP_'):
+            options_names.append((name, getattr(ssl, name)))
+        elif name.startswith('PROTOCOL_'):
+            protocol_names[getattr(ssl, name)] = name
+        elif name.startswith('CERT_'):
+            verify_modes[getattr(ssl, name)] = name
+    options_names.sort(key=lambda item: item[1], reverse=True)
+
+    def formatter(attr_name, value):
+        if attr_name == 'options':
+            options_text = []
+            for opt_name, opt_value in options_names:
+                if value & opt_value:
+                    options_text.append(opt_name)
+                    value &= ~opt_value
+            if value:
+                options_text.append(str(value))
+            return '|' .join(options_text)
+        elif attr_name == 'verify_mode':
+            return verify_modes.get(value, value)
+        elif attr_name == 'protocol':
+            return protocol_names.get(value, value)
+        else:
+            return value
+
+    for name, ctx in (
+        ('SSLContext(PROTOCOL_TLS)', ssl.SSLContext(ssl.PROTOCOL_TLS)),
+        ('default_https_context', ssl._create_default_https_context()),
+        ('stdlib_context', ssl._create_stdlib_context()),
+    ):
+        attributes = (
+            'minimum_version',
+            'maximum_version',
+            'protocol',
+            'options',
+            'verify_mode',
+        )
+        copy_attributes(info_add, ctx, 'ssl.%s.%%s' % name, attributes, formatter=formatter)
+
+    env_names = ["OPENSSL_CONF", "SSLKEYLOGFILE"]
+    if _ssl is not None and hasattr(_ssl, 'get_default_verify_paths'):
+        parts = _ssl.get_default_verify_paths()
+        env_names.extend((parts[0], parts[2]))
+
+    for name in env_names:
+        try:
+            value = os.environ[name]
+        except KeyError:
+            continue
+        info_add('ssl.environ[%s]' % name, value)
+
 
 def collect_socket(info_add):
     import socket


From webhook-mailer at python.org  Wed Oct  2 12:52:55 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Wed, 02 Oct 2019 16:52:55 -0000
Subject: [Python-checkins] bpo-38338,
 test.pythoninfo: add more ssl infos (GH-16539)
Message-ID: <mailman.189.1570035176.32504.python-checkins@python.org>

https://github.com/python/cpython/commit/183733dfb6b4779d1a5e47f41a2fb86c6be08dda
commit: 183733dfb6b4779d1a5e47f41a2fb86c6be08dda
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-02T09:52:50-07:00
summary:

bpo-38338, test.pythoninfo: add more ssl infos (GH-16539)


test.pythoninfo now logs environment variables used by OpenSSL and
Python ssl modules, and logs attributes of 3 SSL contexts
(SSLContext, default HTTPS context, stdlib context).
(cherry picked from commit b3e7045f8314e7b62cd95861d207fe2f97e47198)

Co-authored-by: Victor Stinner <vstinner at python.org>

files:
M Lib/test/pythoninfo.py

diff --git a/Lib/test/pythoninfo.py b/Lib/test/pythoninfo.py
index d2fa6c5936596..797b3af7d3085 100644
--- a/Lib/test/pythoninfo.py
+++ b/Lib/test/pythoninfo.py
@@ -470,10 +470,15 @@ def collect_sysconfig(info_add):
 
 
 def collect_ssl(info_add):
+    import os
     try:
         import ssl
     except ImportError:
         return
+    try:
+        import _ssl
+    except ImportError:
+        _ssl = None
 
     def format_attr(attr, value):
         if attr.startswith('OP_'):
@@ -490,6 +495,32 @@ def format_attr(attr, value):
     )
     copy_attributes(info_add, ssl, 'ssl.%s', attributes, formatter=format_attr)
 
+    for name, ctx in (
+        ('SSLContext', ssl.SSLContext()),
+        ('default_https_context', ssl._create_default_https_context()),
+        ('stdlib_context', ssl._create_stdlib_context()),
+    ):
+        attributes = (
+            'minimum_version',
+            'maximum_version',
+            'protocol',
+            'options',
+            'verify_mode',
+        )
+        copy_attributes(info_add, ctx, f'ssl.{name}.%s', attributes)
+
+    env_names = ["OPENSSL_CONF", "SSLKEYLOGFILE"]
+    if _ssl is not None and hasattr(_ssl, 'get_default_verify_paths'):
+        parts = _ssl.get_default_verify_paths()
+        env_names.extend((parts[0], parts[2]))
+
+    for name in env_names:
+        try:
+            value = os.environ[name]
+        except KeyError:
+            continue
+        info_add('ssl.environ[%s]' % name, value)
+
 
 def collect_socket(info_add):
     import socket


From webhook-mailer at python.org  Wed Oct  2 17:51:30 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Wed, 02 Oct 2019 21:51:30 -0000
Subject: [Python-checkins] bpo-38353: Cleanup includes in the internal C API
 (GH-16548)
Message-ID: <mailman.190.1570053091.32504.python-checkins@python.org>

https://github.com/python/cpython/commit/61691d833631fed42b86605b09e1535e3e8d40e5
commit: 61691d833631fed42b86605b09e1535e3e8d40e5
branch: master
author: Victor Stinner <vstinner at redhat.com>
committer: GitHub <noreply at github.com>
date: 2019-10-02T23:51:20+02:00
summary:

bpo-38353: Cleanup includes in the internal C API (GH-16548)

Use forward declaration of types to avoid includes in the internal C
API. Add also comment to justify other includes.

files:
M Include/internal/pycore_atomic.h
M Include/internal/pycore_ceval.h
M Include/internal/pycore_context.h
M Include/internal/pycore_gil.h
M Include/internal/pycore_initconfig.h
M Include/internal/pycore_object.h
M Include/internal/pycore_pylifecycle.h
M Include/internal/pycore_pymem.h
M Include/internal/pycore_pystate.h
M Include/internal/pycore_traceback.h
M Include/internal/pycore_tupleobject.h
M Include/internal/pycore_warnings.h
M Modules/main.c
M Objects/unicodeobject.c
M PC/getpathp.c
M Python/import.c

diff --git a/Include/internal/pycore_atomic.h b/Include/internal/pycore_atomic.h
index 336bc3fec27e5..1d5c5621677eb 100644
--- a/Include/internal/pycore_atomic.h
+++ b/Include/internal/pycore_atomic.h
@@ -8,8 +8,7 @@ extern "C" {
 #  error "this header requires Py_BUILD_CORE define"
 #endif
 
-#include "dynamic_annotations.h"
-
+#include "dynamic_annotations.h"   /* _Py_ANNOTATE_MEMORY_ORDER */
 #include "pyconfig.h"
 
 #if defined(HAVE_STD_ATOMIC)
diff --git a/Include/internal/pycore_ceval.h b/Include/internal/pycore_ceval.h
index 30cd6c9e09e44..65537b1bae4a1 100644
--- a/Include/internal/pycore_ceval.h
+++ b/Include/internal/pycore_ceval.h
@@ -8,9 +8,9 @@ extern "C" {
 #  error "this header requires Py_BUILD_CORE define"
 #endif
 
-#include "pycore_atomic.h"
-#include "pycore_pystate.h"
-#include "pythread.h"
+/* Forward declarations */
+typedef struct pyruntimestate _PyRuntimeState;
+struct _ceval_runtime_state;
 
 PyAPI_FUNC(void) _Py_FinishPendingCalls(_PyRuntimeState *runtime);
 PyAPI_FUNC(void) _PyEval_Initialize(struct _ceval_runtime_state *);
diff --git a/Include/internal/pycore_context.h b/Include/internal/pycore_context.h
index 5e1ba0d0393f4..f665ad5c115b0 100644
--- a/Include/internal/pycore_context.h
+++ b/Include/internal/pycore_context.h
@@ -5,7 +5,7 @@
 #  error "this header requires Py_BUILD_CORE define"
 #endif
 
-#include "pycore_hamt.h"
+#include "pycore_hamt.h"   /* PyHamtObject */
 
 struct _pycontextobject {
     PyObject_HEAD
diff --git a/Include/internal/pycore_gil.h b/Include/internal/pycore_gil.h
index 7de316397b15e..8ebad37b686cd 100644
--- a/Include/internal/pycore_gil.h
+++ b/Include/internal/pycore_gil.h
@@ -8,8 +8,8 @@ extern "C" {
 #  error "this header requires Py_BUILD_CORE define"
 #endif
 
-#include "pycore_condvar.h"
-#include "pycore_atomic.h"
+#include "pycore_atomic.h"    /* _Py_atomic_address */
+#include "pycore_condvar.h"   /* PyCOND_T */
 
 #ifndef Py_HAVE_CONDVAR
 #  error You need either a POSIX-compatible or a Windows system!
diff --git a/Include/internal/pycore_initconfig.h b/Include/internal/pycore_initconfig.h
index 40831c44b2fa6..4a1a111af6834 100644
--- a/Include/internal/pycore_initconfig.h
+++ b/Include/internal/pycore_initconfig.h
@@ -8,7 +8,8 @@ extern "C" {
 #  error "this header requires Py_BUILD_CORE define"
 #endif
 
-#include "pycore_pystate.h"   /* _PyRuntimeState */
+/* Forward declaration */
+typedef struct pyruntimestate _PyRuntimeState;
 
 /* --- PyStatus ----------------------------------------------- */
 
@@ -60,7 +61,7 @@ PyAPI_FUNC(PyObject*) _PyWideStringList_AsList(const PyWideStringList *list);
 
 /* --- _PyArgv ---------------------------------------------------- */
 
-typedef struct {
+typedef struct _PyArgv {
     Py_ssize_t argc;
     int use_bytes_argv;
     char * const *bytes_argv;
diff --git a/Include/internal/pycore_object.h b/Include/internal/pycore_object.h
index 81548f819198e..500078d4a1dab 100644
--- a/Include/internal/pycore_object.h
+++ b/Include/internal/pycore_object.h
@@ -8,7 +8,7 @@ extern "C" {
 #  error "this header requires Py_BUILD_CORE define"
 #endif
 
-#include "pycore_pystate.h"   /* _PyRuntime */
+#include "pycore_pystate.h"   /* _PyRuntime.gc */
 
 PyAPI_FUNC(int) _PyType_CheckConsistency(PyTypeObject *type);
 PyAPI_FUNC(int) _PyUnicode_CheckConsistency(PyObject *op, int check_content);
diff --git a/Include/internal/pycore_pylifecycle.h b/Include/internal/pycore_pylifecycle.h
index b0a98f5d25971..d6f9ecc8b4872 100644
--- a/Include/internal/pycore_pylifecycle.h
+++ b/Include/internal/pycore_pylifecycle.h
@@ -8,8 +8,9 @@ extern "C" {
 #  error "this header requires Py_BUILD_CORE define"
 #endif
 
-#include "pycore_initconfig.h"   /* _PyArgv */
-#include "pycore_pystate.h"      /* _PyRuntimeState */
+/* Forward declarations */
+typedef struct _PyArgv _PyArgv;
+typedef struct pyruntimestate _PyRuntimeState;
 
 /* True if the main interpreter thread exited due to an unhandled
  * KeyboardInterrupt exception, suggesting the user pressed ^C. */
diff --git a/Include/internal/pycore_pymem.h b/Include/internal/pycore_pymem.h
index 22677d373855c..cba4b235c3aaa 100644
--- a/Include/internal/pycore_pymem.h
+++ b/Include/internal/pycore_pymem.h
@@ -8,8 +8,7 @@ extern "C" {
 #  error "this header requires Py_BUILD_CORE define"
 #endif
 
-#include "objimpl.h"
-#include "pymem.h"
+#include "pymem.h"   /* PyMemAllocatorName */
 
 
 /* GC runtime state */
diff --git a/Include/internal/pycore_pystate.h b/Include/internal/pycore_pystate.h
index 8e5a022b4a09a..91003dbead054 100644
--- a/Include/internal/pycore_pystate.h
+++ b/Include/internal/pycore_pystate.h
@@ -8,16 +8,9 @@ extern "C" {
 #  error "this header requires Py_BUILD_CORE define"
 #endif
 
-#include "cpython/initconfig.h"
-#include "fileobject.h"
-#include "pystate.h"
-#include "pythread.h"
-#include "sysmodule.h"
-
-#include "pycore_gil.h"   /* _gil_runtime_state  */
-#include "pycore_pathconfig.h"
-#include "pycore_pymem.h"
-#include "pycore_warnings.h"
+#include "pycore_gil.h"       /* struct _gil_runtime_state  */
+#include "pycore_pymem.h"     /* struct _gc_runtime_state */
+#include "pycore_warnings.h"  /* struct _warnings_runtime_state */
 
 
 /* ceval state */
diff --git a/Include/internal/pycore_traceback.h b/Include/internal/pycore_traceback.h
index bf4d7fe5851f9..8e58184c87ec9 100644
--- a/Include/internal/pycore_traceback.h
+++ b/Include/internal/pycore_traceback.h
@@ -8,7 +8,8 @@ extern "C" {
 #  error "this header requires Py_BUILD_CORE define"
 #endif
 
-#include "pystate.h"   /* PyInterpreterState */
+/* Forward declaration */
+typedef struct _is PyInterpreterState;
 
 /* Write the Python traceback into the file 'fd'. For example:
 
diff --git a/Include/internal/pycore_tupleobject.h b/Include/internal/pycore_tupleobject.h
index 9fcfc5c6ec719..f95f16c0ed02f 100644
--- a/Include/internal/pycore_tupleobject.h
+++ b/Include/internal/pycore_tupleobject.h
@@ -8,7 +8,7 @@ extern "C" {
 #  error "this header requires Py_BUILD_CORE define"
 #endif
 
-#include "tupleobject.h"
+#include "tupleobject.h"   /* _PyTuple_CAST() */
 
 #define _PyTuple_ITEMS(op) (_PyTuple_CAST(op)->ob_item)
 PyAPI_FUNC(PyObject *) _PyTuple_FromArray(PyObject *const *, Py_ssize_t);
diff --git a/Include/internal/pycore_warnings.h b/Include/internal/pycore_warnings.h
index 73e5350aff145..9a72526279b87 100644
--- a/Include/internal/pycore_warnings.h
+++ b/Include/internal/pycore_warnings.h
@@ -8,8 +8,6 @@ extern "C" {
 #  error "this header requires Py_BUILD_CORE define"
 #endif
 
-#include "object.h"
-
 struct _warnings_runtime_state {
     /* Both 'filters' and 'onceregistry' can be set in warnings.py;
        get_warnings_attr() will reset these variables accordingly. */
diff --git a/Modules/main.c b/Modules/main.c
index 929417fe775f2..0288f17324d80 100644
--- a/Modules/main.c
+++ b/Modules/main.c
@@ -2,6 +2,7 @@
 
 #include "Python.h"
 #include "pycore_initconfig.h"
+#include "pycore_pathconfig.h"
 #include "pycore_pylifecycle.h"
 #include "pycore_pymem.h"
 #include "pycore_pystate.h"
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
index 55b2519181736..0526225ca3e44 100644
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -40,9 +40,10 @@ OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
 
 #define PY_SSIZE_T_CLEAN
 #include "Python.h"
-#include "pycore_initconfig.h"
 #include "pycore_fileutils.h"
+#include "pycore_initconfig.h"
 #include "pycore_object.h"
+#include "pycore_pathconfig.h"
 #include "pycore_pylifecycle.h"
 #include "pycore_pystate.h"
 #include "ucnhash.h"
diff --git a/PC/getpathp.c b/PC/getpathp.c
index 04f24d986f667..b17c002f25c8e 100644
--- a/PC/getpathp.c
+++ b/PC/getpathp.c
@@ -80,7 +80,8 @@
 
 
 #include "Python.h"
-#include "pycore_initconfig.h"
+#include "pycore_initconfig.h"   /* PyStatus */
+#include "pycore_pathconfig.h"   /* _PyPathConfig */
 #include "pycore_pystate.h"
 #include "osdefs.h"
 #include <wchar.h>
diff --git a/Python/import.c b/Python/import.c
index 6eb079d2a4715..cbd0fa86fb58f 100644
--- a/Python/import.c
+++ b/Python/import.c
@@ -4,6 +4,7 @@
 
 #include "Python-ast.h"
 #undef Yield   /* undefine macro conflicting with <winbase.h> */
+#include "pycore_initconfig.h"
 #include "pycore_pyerrors.h"
 #include "pycore_pyhash.h"
 #include "pycore_pylifecycle.h"


From webhook-mailer at python.org  Wed Oct  2 19:04:13 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Wed, 02 Oct 2019 23:04:13 -0000
Subject: [Python-checkins] bpo-36670,
 regrtest: Fix WindowsLoadTracker() for partial line (GH-16550)
Message-ID: <mailman.191.1570057454.32504.python-checkins@python.org>

https://github.com/python/cpython/commit/3e04cd268ee9a57f95dc78d8974b21a6fac3f666
commit: 3e04cd268ee9a57f95dc78d8974b21a6fac3f666
branch: master
author: Victor Stinner <vstinner at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-03T01:04:09+02:00
summary:

bpo-36670, regrtest: Fix WindowsLoadTracker() for partial line (GH-16550)

WindowsLoadTracker.read_output() now uses a short buffer for
incomplete line.

files:
M Lib/test/libregrtest/win_utils.py

diff --git a/Lib/test/libregrtest/win_utils.py b/Lib/test/libregrtest/win_utils.py
index f0c17b906f519..f802980c20b5c 100644
--- a/Lib/test/libregrtest/win_utils.py
+++ b/Lib/test/libregrtest/win_utils.py
@@ -32,6 +32,7 @@ class WindowsLoadTracker():
     def __init__(self):
         self.load = 0.0
         self.counter_name = ''
+        self._buffer = b''
         self.popen = None
         self.start()
 
@@ -100,7 +101,9 @@ def read_output(self):
         if res != 0:
             return
 
-        output = overlapped.getbuffer()
+        # self._buffer stores an incomplete line
+        output = self._buffer + overlapped.getbuffer()
+        output, _, self._buffer = output.rpartition(b'\n')
         return output.decode('oem', 'replace')
 
     def getloadavg(self):


From webhook-mailer at python.org  Thu Oct  3 03:06:58 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Thu, 03 Oct 2019 07:06:58 -0000
Subject: [Python-checkins] bpo-38106: Fix race in pthread
 PyThread_release_lock() (GH-16047)
Message-ID: <mailman.192.1570086419.32504.python-checkins@python.org>

https://github.com/python/cpython/commit/c5abd63e94fcea10bdfa7a20842c6af3248bbd74
commit: c5abd63e94fcea10bdfa7a20842c6af3248bbd74
branch: 2.7
author: Kirill Smelkov <kirr at nexedi.com>
committer: Victor Stinner <vstinner at python.org>
date: 2019-10-03T09:06:52+02:00
summary:

bpo-38106: Fix race in pthread PyThread_release_lock() (GH-16047)

Fix race in PyThread_release_lock that was leading to memory corruption and
deadlocks. The fix applies to POSIX systems where Python locks are implemented
with mutex and condition variable because POSIX semaphores are either not
provided, or are known to be broken. One particular example of such system is
macOS.

On Darwin, even though this is considered as POSIX, Python uses
mutex+condition variable to implement its lock, and, as of 2019-08-28, Py2.7
implementation, even though similar issue was fixed for Py3 in 2012, contains
synchronization bug: the condition is signalled after mutex unlock while the
correct protocol is to signal condition from under mutex:

  https://github.com/python/cpython/blob/v2.7.16-127-g0229b56d8c0/Python/thread_pthread.h#L486-L506
  https://github.com/python/cpython/commit/187aa545165d (py3 fix)

PyPy has the same bug for both pypy2 and pypy3:

  https://bitbucket.org/pypy/pypy/src/578667b3fef9/rpython/translator/c/src/thread_pthread.c#lines-443:465
  https://bitbucket.org/pypy/pypy/src/5b42890d48c3/rpython/translator/c/src/thread_pthread.c#lines-443:465

Signalling condition outside of corresponding mutex is considered OK by
POSIX, but in Python context it can lead to at least memory corruption if we
consider the whole lifetime of python level lock. For example the following
logical scenario:

      T1                                          T2

  sema = Lock()
  sema.acquire()

                                              sema.release()

  sema.acquire()
  free(sema)

  ...

can translate to the next C-level calls:

      T1                                          T2

  # sema = Lock()
  sema = malloc(...)
  sema.locked = 0
  pthread_mutex_init(&sema.mut)
  pthread_cond_init (&sema.lock_released)

  # sema.acquire()
  pthread_mutex_lock(&sema.mut)
  # sees sema.locked == 0
  sema.locked = 1
  pthread_mutex_unlock(&sema.mut)

                                              # sema.release()
                                              pthread_mutex_lock(&sema.mut)
                                              sema.locked = 0
                                              pthread_mutex_unlock(&sema.mut)

                      # OS scheduler gets in and relinquishes control from T2
                      # to another process
                                              ...

  # second sema.acquire()
  pthread_mutex_lock(&sema.mut)
  # sees sema.locked == 0
  sema.locked = 1
  pthread_mutex_unlock(&sema.mut)

  # free(sema)
  pthread_mutex_destroy(&sema.mut)
  pthread_cond_destroy (&sema.lock_released)
  free(sema)

  # ...
  e.g. malloc() which returns memory where sema was

                                              ...
                      # OS scheduler returns control to T2
                      # sema.release() continues
                      #
                      # BUT sema was already freed and writing to anywhere
                      # inside sema block CORRUPTS MEMORY. In particular if
                      # _another_ python-level lock was allocated where sema
                      # block was, writing into the memory can have effect on
                      # further synchronization correctness and in particular
                      # lead to deadlock on lock that was next allocated.
                                              pthread_cond_signal(&sema.lock_released)

Note that T2.pthread_cond_signal(&sema.lock_released) CORRUPTS MEMORY as it
is called when sema memory was already freed and is potentially
reallocated for another object.

The fix is to move pthread_cond_signal to be done under corresponding mutex:

  # sema.release()
  pthread_mutex_lock(&sema.mut)
  sema.locked = 0
  pthread_cond_signal(&sema.lock_released)
  pthread_mutex_unlock(&sema.mut)

To do so this patch cherry-picks thread_pthread.h part of the following 3.2 commit:

commit 187aa545165d8d5eac222ecce29c8a77e0282dd4
Author: Kristj?n Valur J?nsson <kristjan at ccpgames.com>
Date:   Tue Jun 5 22:17:42 2012 +0000

    Signal condition variables with the mutex held.  Destroy condition variables
    before their mutexes.

 Python/ceval_gil.h      |  9 +++++----
 Python/thread_pthread.h | 15 +++++++++------
 2 files changed, 14 insertions(+), 10 deletions(-)

(ceval_gil.h is Python3 specific and does not apply to Python2.7)

The bug was there since 1994 - since at least [1]. It was discussed in 2001
with original code author[2], but the code was still considered to be
race-free. In 2010 the place where pthread_cond_signal should be - before or
after pthread_mutex_unlock - was discussed with the rationale to avoid
threads bouncing[3,4,5], and in 2012 pthread_cond_signal was moved to be
called from under mutex, but only for CPython3[6,7].

In 2019 the bug was (re-)discovered while testing Pygolang[8] on macOS with
CPython2 and PyPy2 and PyPy3.

[1] https://github.com/python/cpython/commit/2c8cb9f3d240
[2] https://bugs.python.org/issue433625
[3] https://bugs.python.org/issue8299#msg103224
[4] https://bugs.python.org/issue8410#msg103313
[5] https://bugs.python.org/issue8411#msg113301
[6] https://bugs.python.org/issue15038#msg163187
[7] https://github.com/python/cpython/commit/187aa545165d
[8] https://pypi.org/project/pygolang

(cherry picked from commit 187aa545165d8d5eac222ecce29c8a77e0282dd4)

Co-Authored-By: Kristj?n Valur J?nsson <kristjan at ccpgames.com>

files:
A Misc/NEWS.d/next/Core and Builtins/2019-09-12-16-36-16.bpo-38106.4pApn7.rst
M Python/thread_pthread.h

diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-09-12-16-36-16.bpo-38106.4pApn7.rst b/Misc/NEWS.d/next/Core and Builtins/2019-09-12-16-36-16.bpo-38106.4pApn7.rst
new file mode 100644
index 0000000000000..b2b80ceece8f2
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2019-09-12-16-36-16.bpo-38106.4pApn7.rst	
@@ -0,0 +1,5 @@
+Fix race in PyThread_release_lock that was leading to memory corruption and
+deadlocks. The fix applies to POSIX systems where Python locks are implemented
+with mutex and condition variable because POSIX semaphores are either not
+provided, or are known to be broken. One particular example of such system is
+macOS.
diff --git a/Python/thread_pthread.h b/Python/thread_pthread.h
index 6d4b3b389f82d..4d792a123a705 100644
--- a/Python/thread_pthread.h
+++ b/Python/thread_pthread.h
@@ -430,12 +430,15 @@ PyThread_free_lock(PyThread_type_lock lock)
     (void) error; /* silence unused-but-set-variable warning */
     dprintf(("PyThread_free_lock(%p) called\n", lock));
 
-    status = pthread_mutex_destroy( &thelock->mut );
-    CHECK_STATUS("pthread_mutex_destroy");
-
+    /* some pthread-like implementations tie the mutex to the cond
+     * and must have the cond destroyed first.
+     */
     status = pthread_cond_destroy( &thelock->lock_released );
     CHECK_STATUS("pthread_cond_destroy");
 
+    status = pthread_mutex_destroy( &thelock->mut );
+    CHECK_STATUS("pthread_mutex_destroy");
+
     free((void *)thelock);
 }
 
@@ -497,12 +500,12 @@ PyThread_release_lock(PyThread_type_lock lock)
 
     thelock->locked = 0;
 
-    status = pthread_mutex_unlock( &thelock->mut );
-    CHECK_STATUS("pthread_mutex_unlock[3]");
-
     /* wake up someone (anyone, if any) waiting on the lock */
     status = pthread_cond_signal( &thelock->lock_released );
     CHECK_STATUS("pthread_cond_signal");
+
+    status = pthread_mutex_unlock( &thelock->mut );
+    CHECK_STATUS("pthread_mutex_unlock[3]");
 }
 
 #endif /* USE_SEMAPHORES */


From webhook-mailer at python.org  Thu Oct  3 04:53:39 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Thu, 03 Oct 2019 08:53:39 -0000
Subject: [Python-checkins] bpo-36670: Enhance regrtest WindowsLoadTracker
 (GH-16553)
Message-ID: <mailman.193.1570092819.32504.python-checkins@python.org>

https://github.com/python/cpython/commit/c65119d5bfded03f80a9805889391b66fa7bf551
commit: c65119d5bfded03f80a9805889391b66fa7bf551
branch: master
author: Victor Stinner <vstinner at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-03T10:53:17+02:00
summary:

bpo-36670: Enhance regrtest WindowsLoadTracker (GH-16553)

The last line is now passed to the parser even if it does not end
with a newline, but only if it's a valid value.

files:
M Lib/test/libregrtest/win_utils.py

diff --git a/Lib/test/libregrtest/win_utils.py b/Lib/test/libregrtest/win_utils.py
index f802980c20b5c..0b73a946307ee 100644
--- a/Lib/test/libregrtest/win_utils.py
+++ b/Lib/test/libregrtest/win_utils.py
@@ -14,7 +14,7 @@
 LOAD_FACTOR_1 = 0.9200444146293232478931553241
 
 # Seconds per measurement
-SAMPLING_INTERVAL = 5
+SAMPLING_INTERVAL = 1
 # Windows registry subkey of HKEY_LOCAL_MACHINE where the counter names
 # of typeperf are registered
 COUNTER_REGISTRY_KEY = (r"SOFTWARE\Microsoft\Windows NT\CurrentVersion"
@@ -32,7 +32,7 @@ class WindowsLoadTracker():
     def __init__(self):
         self.load = 0.0
         self.counter_name = ''
-        self._buffer = b''
+        self._buffer = ''
         self.popen = None
         self.start()
 
@@ -95,44 +95,60 @@ def close(self):
     def __del__(self):
         self.close()
 
-    def read_output(self):
+    def _parse_line(self, line):
+        # typeperf outputs in a CSV format like this:
+        # "07/19/2018 01:32:26.605","3.000000"
+        # (date, process queue length)
+        tokens = line.split(',')
+        if len(tokens) != 2:
+            raise ValueError
+
+        value = tokens[1]
+        if not value.startswith('"') or not value.endswith('"'):
+            raise ValueError
+        value = value[1:-1]
+        return float(value)
+
+    def read_lines(self):
         overlapped, _ = _winapi.ReadFile(self.pipe, BUFSIZE, True)
         bytes_read, res = overlapped.GetOverlappedResult(False)
         if res != 0:
-            return
-
-        # self._buffer stores an incomplete line
-        output = self._buffer + overlapped.getbuffer()
-        output, _, self._buffer = output.rpartition(b'\n')
-        return output.decode('oem', 'replace')
+            return ()
+
+        output = overlapped.getbuffer()
+        output = output.decode('oem', 'replace')
+        output = self._buffer + output
+        lines = output.splitlines(True)
+
+        # bpo-36670: typeperf only writes a newline *before* writing a value,
+        # not after. Sometimes, the written line in incomplete (ex: only
+        # timestamp, without the process queue length). Only pass the last line
+        # to the parser if it's a valid value, otherwise store it in
+        # self._buffer.
+        try:
+            self._parse_line(lines[-1])
+        except ValueError:
+            self._buffer = lines.pop(-1)
+        else:
+            self._buffer = ''
+
+        return lines
 
     def getloadavg(self):
-        typeperf_output = self.read_output()
-        # Nothing to update, just return the current load
-        if not typeperf_output:
-            return self.load
+        for line in self.read_lines():
+            line = line.rstrip()
 
-        # Process the backlog of load values
-        for line in typeperf_output.splitlines():
             # Ignore the initial header:
             # "(PDH-CSV 4.0)","\\\\WIN\\System\\Processor Queue Length"
-            if '\\\\' in line:
+            if 'PDH-CSV' in line:
                 continue
 
             # Ignore blank lines
-            if not line.strip():
+            if not line:
                 continue
 
-            # typeperf outputs in a CSV format like this:
-            # "07/19/2018 01:32:26.605","3.000000"
-            # (date, process queue length)
             try:
-                tokens = line.split(',')
-                if len(tokens) != 2:
-                    raise ValueError
-
-                value = tokens[1].replace('"', '')
-                load = float(value)
+                load = self._parse_line(line)
             except ValueError:
                 print_warning("Failed to parse typeperf output: %a" % line)
                 continue


From webhook-mailer at python.org  Thu Oct  3 10:15:35 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Thu, 03 Oct 2019 14:15:35 -0000
Subject: [Python-checkins] bpo-36670: Enhance regrtest (GH-16556)
Message-ID: <mailman.194.1570112137.32504.python-checkins@python.org>

https://github.com/python/cpython/commit/098e25672f1c3578855d5ded4f5147795c9ed956
commit: 098e25672f1c3578855d5ded4f5147795c9ed956
branch: master
author: Victor Stinner <vstinner at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-03T16:15:16+02:00
summary:

bpo-36670: Enhance regrtest (GH-16556)

* Add log() method: add timestamp and load average prefixes
  to main messages.
* WindowsLoadTracker:

  * LOAD_FACTOR_1 is now computed using SAMPLING_INTERVAL
  * Initialize the load to the arithmetic mean of the first 5 values
    of the Processor Queue Length value (so over 5 seconds), rather
    than 0.0.
  * Handle BrokenPipeError and when typeperf exit.

* format_duration(1.5) now returns '1.5 sec', rather than
  '1 sec 500 ms'

files:
M Lib/test/libregrtest/main.py
M Lib/test/libregrtest/runtest_mp.py
M Lib/test/libregrtest/utils.py
M Lib/test/libregrtest/win_utils.py
M Lib/test/test_regrtest.py

diff --git a/Lib/test/libregrtest/main.py b/Lib/test/libregrtest/main.py
index fd701c452ceb5..76ad3359f2d7a 100644
--- a/Lib/test/libregrtest/main.py
+++ b/Lib/test/libregrtest/main.py
@@ -139,16 +139,8 @@ def accumulate_result(self, result, rerun=False):
                     print(xml_data, file=sys.__stderr__)
                     raise
 
-    def display_progress(self, test_index, text):
-        if self.ns.quiet:
-            return
-
-        # "[ 51/405/1] test_tcl passed"
-        line = f"{test_index:{self.test_count_width}}{self.test_count}"
-        fails = len(self.bad) + len(self.environment_changed)
-        if fails and not self.ns.pgo:
-            line = f"{line}/{fails}"
-        line = f"[{line}] {text}"
+    def log(self, line=''):
+        empty = not line
 
         # add the system load prefix: "load avg: 1.80 "
         load_avg = self.getloadavg()
@@ -159,8 +151,23 @@ def display_progress(self, test_index, text):
         test_time = time.monotonic() - self.start_time
         test_time = datetime.timedelta(seconds=int(test_time))
         line = f"{test_time} {line}"
+
+        if empty:
+            line = line[:-1]
+
         print(line, flush=True)
 
+    def display_progress(self, test_index, text):
+        if self.ns.quiet:
+            return
+
+        # "[ 51/405/1] test_tcl passed"
+        line = f"{test_index:{self.test_count_width}}{self.test_count}"
+        fails = len(self.bad) + len(self.environment_changed)
+        if fails and not self.ns.pgo:
+            line = f"{line}/{fails}"
+        self.log(f"[{line}] {text}")
+
     def parse_args(self, kwargs):
         ns = _parse_args(sys.argv[1:], **kwargs)
 
@@ -302,11 +309,11 @@ def rerun_failed_tests(self):
 
         self.first_result = self.get_tests_result()
 
-        print()
-        print("Re-running failed tests in verbose mode")
+        self.log()
+        self.log("Re-running failed tests in verbose mode")
         self.rerun = self.bad[:]
         for test_name in self.rerun:
-            print(f"Re-running {test_name} in verbose mode", flush=True)
+            self.log(f"Re-running {test_name} in verbose mode")
             self.ns.verbose = True
             result = runtest(self.ns, test_name)
 
@@ -387,7 +394,7 @@ def run_tests_sequential(self):
 
         save_modules = sys.modules.keys()
 
-        print("Run tests sequentially")
+        self.log("Run tests sequentially")
 
         previous_test = None
         for test_index, test_name in enumerate(self.tests, 1):
diff --git a/Lib/test/libregrtest/runtest_mp.py b/Lib/test/libregrtest/runtest_mp.py
index a46c78248de39..79afa29fa05bf 100644
--- a/Lib/test/libregrtest/runtest_mp.py
+++ b/Lib/test/libregrtest/runtest_mp.py
@@ -104,13 +104,14 @@ class ExitThread(Exception):
 
 
 class TestWorkerProcess(threading.Thread):
-    def __init__(self, worker_id, pending, output, ns, timeout):
+    def __init__(self, worker_id, runner):
         super().__init__()
         self.worker_id = worker_id
-        self.pending = pending
-        self.output = output
-        self.ns = ns
-        self.timeout = timeout
+        self.pending = runner.pending
+        self.output = runner.output
+        self.ns = runner.ns
+        self.timeout = runner.worker_timeout
+        self.regrtest = runner.regrtest
         self.current_test_name = None
         self.start_time = None
         self._popen = None
@@ -294,7 +295,8 @@ def wait_stopped(self, start_time):
             if not self.is_alive():
                 break
             dt = time.monotonic() - start_time
-            print(f"Waiting for {self} thread for {format_duration(dt)}", flush=True)
+            self.regrtest.log(f"Waiting for {self} thread "
+                              f"for {format_duration(dt)}")
             if dt > JOIN_TIMEOUT:
                 print_warning(f"Failed to join {self} in {format_duration(dt)}")
                 break
@@ -316,6 +318,7 @@ def get_running(workers):
 class MultiprocessTestRunner:
     def __init__(self, regrtest):
         self.regrtest = regrtest
+        self.log = self.regrtest.log
         self.ns = regrtest.ns
         self.output = queue.Queue()
         self.pending = MultiprocessIterator(self.regrtest.tests)
@@ -326,11 +329,10 @@ def __init__(self, regrtest):
         self.workers = None
 
     def start_workers(self):
-        self.workers = [TestWorkerProcess(index, self.pending, self.output,
-                                          self.ns, self.worker_timeout)
+        self.workers = [TestWorkerProcess(index, self)
                         for index in range(1, self.ns.use_mp + 1)]
-        print("Run tests in parallel using %s child processes"
-              % len(self.workers))
+        self.log("Run tests in parallel using %s child processes"
+                 % len(self.workers))
         for worker in self.workers:
             worker.start()
 
@@ -364,7 +366,7 @@ def _get_result(self):
             # display progress
             running = get_running(self.workers)
             if running and not self.ns.pgo:
-                print('running: %s' % ', '.join(running), flush=True)
+                self.log('running: %s' % ', '.join(running))
 
     def display_result(self, mp_result):
         result = mp_result.result
@@ -384,8 +386,7 @@ def _process_result(self, item):
         if item[0]:
             # Thread got an exception
             format_exc = item[1]
-            print(f"regrtest worker thread failed: {format_exc}",
-                  file=sys.stderr, flush=True)
+            print_warning(f"regrtest worker thread failed: {format_exc}")
             return True
 
         self.test_index += 1
diff --git a/Lib/test/libregrtest/utils.py b/Lib/test/libregrtest/utils.py
index 2691a2c30ce80..40faed832c11b 100644
--- a/Lib/test/libregrtest/utils.py
+++ b/Lib/test/libregrtest/utils.py
@@ -17,11 +17,14 @@ def format_duration(seconds):
     if minutes:
         parts.append('%s min' % minutes)
     if seconds:
-        parts.append('%s sec' % seconds)
-    if ms:
-        parts.append('%s ms' % ms)
+        if parts:
+            # 2 min 1 sec
+            parts.append('%s sec' % seconds)
+        else:
+            # 1.0 sec
+            parts.append('%.1f sec' % (seconds + ms / 1000))
     if not parts:
-        return '0 ms'
+        return '%s ms' % ms
 
     parts = parts[:2]
     return ' '.join(parts)
diff --git a/Lib/test/libregrtest/win_utils.py b/Lib/test/libregrtest/win_utils.py
index 0b73a946307ee..028c01106dee0 100644
--- a/Lib/test/libregrtest/win_utils.py
+++ b/Lib/test/libregrtest/win_utils.py
@@ -1,4 +1,5 @@
 import _winapi
+import math
 import msvcrt
 import os
 import subprocess
@@ -10,11 +11,14 @@
 
 # Max size of asynchronous reads
 BUFSIZE = 8192
-# Exponential damping factor (see below)
-LOAD_FACTOR_1 = 0.9200444146293232478931553241
-
 # Seconds per measurement
 SAMPLING_INTERVAL = 1
+# Exponential damping factor to compute exponentially weighted moving average
+# on 1 minute (60 seconds)
+LOAD_FACTOR_1 = 1 / math.exp(SAMPLING_INTERVAL / 60)
+# Initialize the load using the arithmetic mean of the first NVALUE values
+# of the Processor Queue Length
+NVALUE = 5
 # Windows registry subkey of HKEY_LOCAL_MACHINE where the counter names
 # of typeperf are registered
 COUNTER_REGISTRY_KEY = (r"SOFTWARE\Microsoft\Windows NT\CurrentVersion"
@@ -30,10 +34,10 @@ class WindowsLoadTracker():
     """
 
     def __init__(self):
-        self.load = 0.0
-        self.counter_name = ''
+        self._values = []
+        self._load = None
         self._buffer = ''
-        self.popen = None
+        self._popen = None
         self.start()
 
     def start(self):
@@ -65,7 +69,7 @@ def start(self):
         # Spawn off the load monitor
         counter_name = self._get_counter_name()
         command = ['typeperf', counter_name, '-si', str(SAMPLING_INTERVAL)]
-        self.popen = subprocess.Popen(' '.join(command), stdout=command_stdout, cwd=support.SAVEDCWD)
+        self._popen = subprocess.Popen(' '.join(command), stdout=command_stdout, cwd=support.SAVEDCWD)
 
         # Close our copy of the write end of the pipe
         os.close(command_stdout)
@@ -85,12 +89,16 @@ def _get_counter_name(self):
         process_queue_length = counters_dict['44']
         return f'"\\{system}\\{process_queue_length}"'
 
-    def close(self):
-        if self.popen is None:
+    def close(self, kill=True):
+        if self._popen is None:
             return
-        self.popen.kill()
-        self.popen.wait()
-        self.popen = None
+
+        self._load = None
+
+        if kill:
+            self._popen.kill()
+        self._popen.wait()
+        self._popen = None
 
     def __del__(self):
         self.close()
@@ -109,7 +117,7 @@ def _parse_line(self, line):
         value = value[1:-1]
         return float(value)
 
-    def read_lines(self):
+    def _read_lines(self):
         overlapped, _ = _winapi.ReadFile(self.pipe, BUFSIZE, True)
         bytes_read, res = overlapped.GetOverlappedResult(False)
         if res != 0:
@@ -135,7 +143,21 @@ def read_lines(self):
         return lines
 
     def getloadavg(self):
-        for line in self.read_lines():
+        if self._popen is None:
+            return None
+
+        returncode = self._popen.poll()
+        if returncode is not None:
+            self.close(kill=False)
+            return None
+
+        try:
+            lines = self._read_lines()
+        except BrokenPipeError:
+            self.close()
+            return None
+
+        for line in lines:
             line = line.rstrip()
 
             # Ignore the initial header:
@@ -148,7 +170,7 @@ def getloadavg(self):
                 continue
 
             try:
-                load = self._parse_line(line)
+                processor_queue_length = self._parse_line(line)
             except ValueError:
                 print_warning("Failed to parse typeperf output: %a" % line)
                 continue
@@ -156,7 +178,13 @@ def getloadavg(self):
             # We use an exponentially weighted moving average, imitating the
             # load calculation on Unix systems.
             # https://en.wikipedia.org/wiki/Load_(computing)#Unix-style_load_calculation
-            new_load = self.load * LOAD_FACTOR_1 + load * (1.0 - LOAD_FACTOR_1)
-            self.load = new_load
-
-        return self.load
+            # https://en.wikipedia.org/wiki/Moving_average#Exponential_moving_average
+            if self._load is not None:
+                self._load = (self._load * LOAD_FACTOR_1
+                              + processor_queue_length  * (1.0 - LOAD_FACTOR_1))
+            elif len(self._values) < NVALUE:
+                self._values.append(processor_queue_length)
+            else:
+                self._load = sum(self._values) / len(self._values)
+
+        return self._load
diff --git a/Lib/test/test_regrtest.py b/Lib/test/test_regrtest.py
index fa37ecde37623..aaa97e040804a 100644
--- a/Lib/test/test_regrtest.py
+++ b/Lib/test/test_regrtest.py
@@ -25,6 +25,7 @@
 Py_DEBUG = hasattr(sys, 'gettotalrefcount')
 ROOT_DIR = os.path.join(os.path.dirname(__file__), '..', '..')
 ROOT_DIR = os.path.abspath(os.path.normpath(ROOT_DIR))
+LOG_PREFIX = r'[0-9]+:[0-9]+:[0-9]+ (?:load avg: [0-9]+\.[0-9]{2} )?'
 
 TEST_INTERRUPTED = textwrap.dedent("""
     from signal import SIGINT, raise_signal
@@ -388,8 +389,8 @@ def check_line(self, output, regex):
         self.assertRegex(output, regex)
 
     def parse_executed_tests(self, output):
-        regex = (r'^[0-9]+:[0-9]+:[0-9]+ (?:load avg: [0-9]+\.[0-9]{2} )?\[ *[0-9]+(?:/ *[0-9]+)*\] (%s)'
-                 % self.TESTNAME_REGEX)
+        regex = (r'^%s\[ *[0-9]+(?:/ *[0-9]+)*\] (%s)'
+                 % (LOG_PREFIX, self.TESTNAME_REGEX))
         parser = re.finditer(regex, output, re.MULTILINE)
         return list(match.group(1) for match in parser)
 
@@ -449,9 +450,10 @@ def list_regex(line_format, tests):
         if rerun:
             regex = list_regex('%s re-run test%s', rerun)
             self.check_line(output, regex)
-            self.check_line(output, "Re-running failed tests in verbose mode")
+            regex = LOG_PREFIX + r"Re-running failed tests in verbose mode"
+            self.check_line(output, regex)
             for test_name in rerun:
-                regex = f"Re-running {test_name} in verbose mode"
+                regex = LOG_PREFIX + f"Re-running {test_name} in verbose mode"
                 self.check_line(output, regex)
 
         if no_test_ran:
@@ -1232,9 +1234,9 @@ def test_format_duration(self):
         self.assertEqual(utils.format_duration(10e-3),
                          '10 ms')
         self.assertEqual(utils.format_duration(1.5),
-                         '1 sec 500 ms')
+                         '1.5 sec')
         self.assertEqual(utils.format_duration(1),
-                         '1 sec')
+                         '1.0 sec')
         self.assertEqual(utils.format_duration(2 * 60),
                          '2 min')
         self.assertEqual(utils.format_duration(2 * 60 + 1),


From webhook-mailer at python.org  Thu Oct  3 11:26:41 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Thu, 03 Oct 2019 15:26:41 -0000
Subject: [Python-checkins] [3.8] bpo-36670,
 regrtest: Fix WindowsLoadTracker() for partial line (GH-16550)
 (GH-16560)
Message-ID: <mailman.195.1570116403.32504.python-checkins@python.org>

https://github.com/python/cpython/commit/de3195c937d5fca0d79cc93dbafa76c0f89ca5b8
commit: de3195c937d5fca0d79cc93dbafa76c0f89ca5b8
branch: 3.8
author: Victor Stinner <vstinner at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-03T17:26:25+02:00
summary:

[3.8] bpo-36670, regrtest: Fix WindowsLoadTracker() for partial line (GH-16550) (GH-16560)

* bpo-36670, regrtest: Fix WindowsLoadTracker() for partial line (GH-16550)

WindowsLoadTracker.read_output() now uses a short buffer for
incomplete line.

(cherry picked from commit 3e04cd268ee9a57f95dc78d8974b21a6fac3f666)

* bpo-36670: Enhance regrtest WindowsLoadTracker (GH-16553)

The last line is now passed to the parser even if it does not end
with a newline, but only if it's a valid value.

(cherry picked from commit c65119d5bfded03f80a9805889391b66fa7bf551)

* bpo-36670: Enhance regrtest (GH-16556)

* Add log() method: add timestamp and load average prefixes
  to main messages.
* WindowsLoadTracker:

  * LOAD_FACTOR_1 is now computed using SAMPLING_INTERVAL
  * Initialize the load to the arithmetic mean of the first 5 values
    of the Processor Queue Length value (so over 5 seconds), rather
    than 0.0.
  * Handle BrokenPipeError and when typeperf exit.

* format_duration(1.5) now returns '1.5 sec', rather than
  '1 sec 500 ms'

(cherry picked from commit 098e25672f1c3578855d5ded4f5147795c9ed956)

files:
M Lib/test/libregrtest/main.py
M Lib/test/libregrtest/runtest_mp.py
M Lib/test/libregrtest/utils.py
M Lib/test/libregrtest/win_utils.py
M Lib/test/test_regrtest.py

diff --git a/Lib/test/libregrtest/main.py b/Lib/test/libregrtest/main.py
index fd701c452ceb5..76ad3359f2d7a 100644
--- a/Lib/test/libregrtest/main.py
+++ b/Lib/test/libregrtest/main.py
@@ -139,16 +139,8 @@ def accumulate_result(self, result, rerun=False):
                     print(xml_data, file=sys.__stderr__)
                     raise
 
-    def display_progress(self, test_index, text):
-        if self.ns.quiet:
-            return
-
-        # "[ 51/405/1] test_tcl passed"
-        line = f"{test_index:{self.test_count_width}}{self.test_count}"
-        fails = len(self.bad) + len(self.environment_changed)
-        if fails and not self.ns.pgo:
-            line = f"{line}/{fails}"
-        line = f"[{line}] {text}"
+    def log(self, line=''):
+        empty = not line
 
         # add the system load prefix: "load avg: 1.80 "
         load_avg = self.getloadavg()
@@ -159,8 +151,23 @@ def display_progress(self, test_index, text):
         test_time = time.monotonic() - self.start_time
         test_time = datetime.timedelta(seconds=int(test_time))
         line = f"{test_time} {line}"
+
+        if empty:
+            line = line[:-1]
+
         print(line, flush=True)
 
+    def display_progress(self, test_index, text):
+        if self.ns.quiet:
+            return
+
+        # "[ 51/405/1] test_tcl passed"
+        line = f"{test_index:{self.test_count_width}}{self.test_count}"
+        fails = len(self.bad) + len(self.environment_changed)
+        if fails and not self.ns.pgo:
+            line = f"{line}/{fails}"
+        self.log(f"[{line}] {text}")
+
     def parse_args(self, kwargs):
         ns = _parse_args(sys.argv[1:], **kwargs)
 
@@ -302,11 +309,11 @@ def rerun_failed_tests(self):
 
         self.first_result = self.get_tests_result()
 
-        print()
-        print("Re-running failed tests in verbose mode")
+        self.log()
+        self.log("Re-running failed tests in verbose mode")
         self.rerun = self.bad[:]
         for test_name in self.rerun:
-            print(f"Re-running {test_name} in verbose mode", flush=True)
+            self.log(f"Re-running {test_name} in verbose mode")
             self.ns.verbose = True
             result = runtest(self.ns, test_name)
 
@@ -387,7 +394,7 @@ def run_tests_sequential(self):
 
         save_modules = sys.modules.keys()
 
-        print("Run tests sequentially")
+        self.log("Run tests sequentially")
 
         previous_test = None
         for test_index, test_name in enumerate(self.tests, 1):
diff --git a/Lib/test/libregrtest/runtest_mp.py b/Lib/test/libregrtest/runtest_mp.py
index a46c78248de39..79afa29fa05bf 100644
--- a/Lib/test/libregrtest/runtest_mp.py
+++ b/Lib/test/libregrtest/runtest_mp.py
@@ -104,13 +104,14 @@ class ExitThread(Exception):
 
 
 class TestWorkerProcess(threading.Thread):
-    def __init__(self, worker_id, pending, output, ns, timeout):
+    def __init__(self, worker_id, runner):
         super().__init__()
         self.worker_id = worker_id
-        self.pending = pending
-        self.output = output
-        self.ns = ns
-        self.timeout = timeout
+        self.pending = runner.pending
+        self.output = runner.output
+        self.ns = runner.ns
+        self.timeout = runner.worker_timeout
+        self.regrtest = runner.regrtest
         self.current_test_name = None
         self.start_time = None
         self._popen = None
@@ -294,7 +295,8 @@ def wait_stopped(self, start_time):
             if not self.is_alive():
                 break
             dt = time.monotonic() - start_time
-            print(f"Waiting for {self} thread for {format_duration(dt)}", flush=True)
+            self.regrtest.log(f"Waiting for {self} thread "
+                              f"for {format_duration(dt)}")
             if dt > JOIN_TIMEOUT:
                 print_warning(f"Failed to join {self} in {format_duration(dt)}")
                 break
@@ -316,6 +318,7 @@ def get_running(workers):
 class MultiprocessTestRunner:
     def __init__(self, regrtest):
         self.regrtest = regrtest
+        self.log = self.regrtest.log
         self.ns = regrtest.ns
         self.output = queue.Queue()
         self.pending = MultiprocessIterator(self.regrtest.tests)
@@ -326,11 +329,10 @@ def __init__(self, regrtest):
         self.workers = None
 
     def start_workers(self):
-        self.workers = [TestWorkerProcess(index, self.pending, self.output,
-                                          self.ns, self.worker_timeout)
+        self.workers = [TestWorkerProcess(index, self)
                         for index in range(1, self.ns.use_mp + 1)]
-        print("Run tests in parallel using %s child processes"
-              % len(self.workers))
+        self.log("Run tests in parallel using %s child processes"
+                 % len(self.workers))
         for worker in self.workers:
             worker.start()
 
@@ -364,7 +366,7 @@ def _get_result(self):
             # display progress
             running = get_running(self.workers)
             if running and not self.ns.pgo:
-                print('running: %s' % ', '.join(running), flush=True)
+                self.log('running: %s' % ', '.join(running))
 
     def display_result(self, mp_result):
         result = mp_result.result
@@ -384,8 +386,7 @@ def _process_result(self, item):
         if item[0]:
             # Thread got an exception
             format_exc = item[1]
-            print(f"regrtest worker thread failed: {format_exc}",
-                  file=sys.stderr, flush=True)
+            print_warning(f"regrtest worker thread failed: {format_exc}")
             return True
 
         self.test_index += 1
diff --git a/Lib/test/libregrtest/utils.py b/Lib/test/libregrtest/utils.py
index fb9971a64f66c..98a60f7a747d9 100644
--- a/Lib/test/libregrtest/utils.py
+++ b/Lib/test/libregrtest/utils.py
@@ -16,11 +16,14 @@ def format_duration(seconds):
     if minutes:
         parts.append('%s min' % minutes)
     if seconds:
-        parts.append('%s sec' % seconds)
-    if ms:
-        parts.append('%s ms' % ms)
+        if parts:
+            # 2 min 1 sec
+            parts.append('%s sec' % seconds)
+        else:
+            # 1.0 sec
+            parts.append('%.1f sec' % (seconds + ms / 1000))
     if not parts:
-        return '0 ms'
+        return '%s ms' % ms
 
     parts = parts[:2]
     return ' '.join(parts)
diff --git a/Lib/test/libregrtest/win_utils.py b/Lib/test/libregrtest/win_utils.py
index f0c17b906f519..028c01106dee0 100644
--- a/Lib/test/libregrtest/win_utils.py
+++ b/Lib/test/libregrtest/win_utils.py
@@ -1,4 +1,5 @@
 import _winapi
+import math
 import msvcrt
 import os
 import subprocess
@@ -10,11 +11,14 @@
 
 # Max size of asynchronous reads
 BUFSIZE = 8192
-# Exponential damping factor (see below)
-LOAD_FACTOR_1 = 0.9200444146293232478931553241
-
 # Seconds per measurement
-SAMPLING_INTERVAL = 5
+SAMPLING_INTERVAL = 1
+# Exponential damping factor to compute exponentially weighted moving average
+# on 1 minute (60 seconds)
+LOAD_FACTOR_1 = 1 / math.exp(SAMPLING_INTERVAL / 60)
+# Initialize the load using the arithmetic mean of the first NVALUE values
+# of the Processor Queue Length
+NVALUE = 5
 # Windows registry subkey of HKEY_LOCAL_MACHINE where the counter names
 # of typeperf are registered
 COUNTER_REGISTRY_KEY = (r"SOFTWARE\Microsoft\Windows NT\CurrentVersion"
@@ -30,9 +34,10 @@ class WindowsLoadTracker():
     """
 
     def __init__(self):
-        self.load = 0.0
-        self.counter_name = ''
-        self.popen = None
+        self._values = []
+        self._load = None
+        self._buffer = ''
+        self._popen = None
         self.start()
 
     def start(self):
@@ -64,7 +69,7 @@ def start(self):
         # Spawn off the load monitor
         counter_name = self._get_counter_name()
         command = ['typeperf', counter_name, '-si', str(SAMPLING_INTERVAL)]
-        self.popen = subprocess.Popen(' '.join(command), stdout=command_stdout, cwd=support.SAVEDCWD)
+        self._popen = subprocess.Popen(' '.join(command), stdout=command_stdout, cwd=support.SAVEDCWD)
 
         # Close our copy of the write end of the pipe
         os.close(command_stdout)
@@ -84,52 +89,88 @@ def _get_counter_name(self):
         process_queue_length = counters_dict['44']
         return f'"\\{system}\\{process_queue_length}"'
 
-    def close(self):
-        if self.popen is None:
+    def close(self, kill=True):
+        if self._popen is None:
             return
-        self.popen.kill()
-        self.popen.wait()
-        self.popen = None
+
+        self._load = None
+
+        if kill:
+            self._popen.kill()
+        self._popen.wait()
+        self._popen = None
 
     def __del__(self):
         self.close()
 
-    def read_output(self):
+    def _parse_line(self, line):
+        # typeperf outputs in a CSV format like this:
+        # "07/19/2018 01:32:26.605","3.000000"
+        # (date, process queue length)
+        tokens = line.split(',')
+        if len(tokens) != 2:
+            raise ValueError
+
+        value = tokens[1]
+        if not value.startswith('"') or not value.endswith('"'):
+            raise ValueError
+        value = value[1:-1]
+        return float(value)
+
+    def _read_lines(self):
         overlapped, _ = _winapi.ReadFile(self.pipe, BUFSIZE, True)
         bytes_read, res = overlapped.GetOverlappedResult(False)
         if res != 0:
-            return
+            return ()
 
         output = overlapped.getbuffer()
-        return output.decode('oem', 'replace')
+        output = output.decode('oem', 'replace')
+        output = self._buffer + output
+        lines = output.splitlines(True)
+
+        # bpo-36670: typeperf only writes a newline *before* writing a value,
+        # not after. Sometimes, the written line in incomplete (ex: only
+        # timestamp, without the process queue length). Only pass the last line
+        # to the parser if it's a valid value, otherwise store it in
+        # self._buffer.
+        try:
+            self._parse_line(lines[-1])
+        except ValueError:
+            self._buffer = lines.pop(-1)
+        else:
+            self._buffer = ''
+
+        return lines
 
     def getloadavg(self):
-        typeperf_output = self.read_output()
-        # Nothing to update, just return the current load
-        if not typeperf_output:
-            return self.load
+        if self._popen is None:
+            return None
+
+        returncode = self._popen.poll()
+        if returncode is not None:
+            self.close(kill=False)
+            return None
+
+        try:
+            lines = self._read_lines()
+        except BrokenPipeError:
+            self.close()
+            return None
+
+        for line in lines:
+            line = line.rstrip()
 
-        # Process the backlog of load values
-        for line in typeperf_output.splitlines():
             # Ignore the initial header:
             # "(PDH-CSV 4.0)","\\\\WIN\\System\\Processor Queue Length"
-            if '\\\\' in line:
+            if 'PDH-CSV' in line:
                 continue
 
             # Ignore blank lines
-            if not line.strip():
+            if not line:
                 continue
 
-            # typeperf outputs in a CSV format like this:
-            # "07/19/2018 01:32:26.605","3.000000"
-            # (date, process queue length)
             try:
-                tokens = line.split(',')
-                if len(tokens) != 2:
-                    raise ValueError
-
-                value = tokens[1].replace('"', '')
-                load = float(value)
+                processor_queue_length = self._parse_line(line)
             except ValueError:
                 print_warning("Failed to parse typeperf output: %a" % line)
                 continue
@@ -137,7 +178,13 @@ def getloadavg(self):
             # We use an exponentially weighted moving average, imitating the
             # load calculation on Unix systems.
             # https://en.wikipedia.org/wiki/Load_(computing)#Unix-style_load_calculation
-            new_load = self.load * LOAD_FACTOR_1 + load * (1.0 - LOAD_FACTOR_1)
-            self.load = new_load
-
-        return self.load
+            # https://en.wikipedia.org/wiki/Moving_average#Exponential_moving_average
+            if self._load is not None:
+                self._load = (self._load * LOAD_FACTOR_1
+                              + processor_queue_length  * (1.0 - LOAD_FACTOR_1))
+            elif len(self._values) < NVALUE:
+                self._values.append(processor_queue_length)
+            else:
+                self._load = sum(self._values) / len(self._values)
+
+        return self._load
diff --git a/Lib/test/test_regrtest.py b/Lib/test/test_regrtest.py
index 0f44c72069753..d2221b3448fa3 100644
--- a/Lib/test/test_regrtest.py
+++ b/Lib/test/test_regrtest.py
@@ -25,6 +25,7 @@
 Py_DEBUG = hasattr(sys, 'gettotalrefcount')
 ROOT_DIR = os.path.join(os.path.dirname(__file__), '..', '..')
 ROOT_DIR = os.path.abspath(os.path.normpath(ROOT_DIR))
+LOG_PREFIX = r'[0-9]+:[0-9]+:[0-9]+ (?:load avg: [0-9]+\.[0-9]{2} )?'
 
 TEST_INTERRUPTED = textwrap.dedent("""
     from signal import SIGINT, raise_signal
@@ -390,8 +391,8 @@ def check_line(self, output, regex):
         self.assertRegex(output, regex)
 
     def parse_executed_tests(self, output):
-        regex = (r'^[0-9]+:[0-9]+:[0-9]+ (?:load avg: [0-9]+\.[0-9]{2} )?\[ *[0-9]+(?:/ *[0-9]+)*\] (%s)'
-                 % self.TESTNAME_REGEX)
+        regex = (r'^%s\[ *[0-9]+(?:/ *[0-9]+)*\] (%s)'
+                 % (LOG_PREFIX, self.TESTNAME_REGEX))
         parser = re.finditer(regex, output, re.MULTILINE)
         return list(match.group(1) for match in parser)
 
@@ -451,9 +452,10 @@ def list_regex(line_format, tests):
         if rerun:
             regex = list_regex('%s re-run test%s', rerun)
             self.check_line(output, regex)
-            self.check_line(output, "Re-running failed tests in verbose mode")
+            regex = LOG_PREFIX + r"Re-running failed tests in verbose mode"
+            self.check_line(output, regex)
             for test_name in rerun:
-                regex = f"Re-running {test_name} in verbose mode"
+                regex = LOG_PREFIX + f"Re-running {test_name} in verbose mode"
                 self.check_line(output, regex)
 
         if no_test_ran:
@@ -1202,9 +1204,9 @@ def test_format_duration(self):
         self.assertEqual(utils.format_duration(10e-3),
                          '10 ms')
         self.assertEqual(utils.format_duration(1.5),
-                         '1 sec 500 ms')
+                         '1.5 sec')
         self.assertEqual(utils.format_duration(1),
-                         '1 sec')
+                         '1.0 sec')
         self.assertEqual(utils.format_duration(2 * 60),
                          '2 min')
         self.assertEqual(utils.format_duration(2 * 60 + 1),


From webhook-mailer at python.org  Thu Oct  3 11:31:10 2019
From: webhook-mailer at python.org (Steve Dower)
Date: Thu, 03 Oct 2019 15:31:10 -0000
Subject: [Python-checkins] bpo-38355: Fix ntpath.realpath failing on
 sys.executable (GH-16551)
Message-ID: <mailman.196.1570116671.32504.python-checkins@python.org>

https://github.com/python/cpython/commit/a0e3d27e4e3cb5b67e325df080fb18b70c2910cf
commit: a0e3d27e4e3cb5b67e325df080fb18b70c2910cf
branch: master
author: Steve Dower <steve.dower at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-03T08:31:03-07:00
summary:

bpo-38355: Fix ntpath.realpath failing on sys.executable (GH-16551)

files:
A Misc/NEWS.d/next/Windows/2019-10-02-15-38-49.bpo-38355.n3AWX6.rst
M Lib/ntpath.py
M Lib/test/test_ntpath.py

diff --git a/Lib/ntpath.py b/Lib/ntpath.py
index 01f1f423c906c..d4ecff97c9528 100644
--- a/Lib/ntpath.py
+++ b/Lib/ntpath.py
@@ -560,13 +560,6 @@ def _readlink_deep(path, seen=None):
         return path
 
     def _getfinalpathname_nonstrict(path):
-        # Fast path to get the final path name. If this succeeds, there
-        # is no need to go any further.
-        try:
-            return _getfinalpathname(path)
-        except OSError:
-            pass
-
         # These error codes indicate that we should stop resolving the path
         # and return the value we currently have.
         # 1: ERROR_INVALID_FUNCTION
@@ -579,8 +572,9 @@ def _getfinalpathname_nonstrict(path):
         # 67: ERROR_BAD_NET_NAME (implies remote server unavailable)
         # 87: ERROR_INVALID_PARAMETER
         # 123: ERROR_INVALID_NAME
+        # 1920: ERROR_CANT_ACCESS_FILE
         # 1921: ERROR_CANT_RESOLVE_FILENAME (implies unfollowable symlink)
-        allowed_winerror = 1, 2, 3, 5, 21, 32, 50, 67, 87, 123, 1921
+        allowed_winerror = 1, 2, 3, 5, 21, 32, 50, 67, 87, 123, 1920, 1921
 
         # Non-strict algorithm is to find as much of the target directory
         # as we can and join the rest.
@@ -615,9 +609,13 @@ def realpath(path):
             unc_prefix = '\\\\?\\UNC\\'
             new_unc_prefix = '\\\\'
             cwd = os.getcwd()
-        did_not_exist = not exists(path)
         had_prefix = path.startswith(prefix)
-        path = _getfinalpathname_nonstrict(path)
+        try:
+            path = _getfinalpathname(path)
+            initial_winerror = 0
+        except OSError as ex:
+            initial_winerror = ex.winerror
+            path = _getfinalpathname_nonstrict(path)
         # The path returned by _getfinalpathname will always start with \\?\ -
         # strip off that prefix unless it was already provided on the original
         # path.
@@ -635,7 +633,7 @@ def realpath(path):
             except OSError as ex:
                 # If the path does not exist and originally did not exist, then
                 # strip the prefix anyway.
-                if ex.winerror in {2, 3} and did_not_exist:
+                if ex.winerror == initial_winerror:
                     path = spath
         return path
 
diff --git a/Lib/test/test_ntpath.py b/Lib/test/test_ntpath.py
index 2f0faf94726fd..e0ec441985230 100644
--- a/Lib/test/test_ntpath.py
+++ b/Lib/test/test_ntpath.py
@@ -329,16 +329,14 @@ def test_realpath_symlink_loops(self):
         self.addCleanup(support.unlink, ABSTFN + "c")
         self.addCleanup(support.unlink, ABSTFN + "a")
 
-        P = "\\\\?\\"
-
         os.symlink(ABSTFN, ABSTFN)
-        self.assertPathEqual(ntpath.realpath(ABSTFN), P + ABSTFN)
+        self.assertPathEqual(ntpath.realpath(ABSTFN), ABSTFN)
 
         # cycles are non-deterministic as to which path is returned, but
         # it will always be the fully resolved path of one member of the cycle
         os.symlink(ABSTFN + "1", ABSTFN + "2")
         os.symlink(ABSTFN + "2", ABSTFN + "1")
-        expected = (P + ABSTFN + "1", P + ABSTFN + "2")
+        expected = (ABSTFN + "1", ABSTFN + "2")
         self.assertPathIn(ntpath.realpath(ABSTFN + "1"), expected)
         self.assertPathIn(ntpath.realpath(ABSTFN + "2"), expected)
 
@@ -357,14 +355,14 @@ def test_realpath_symlink_loops(self):
                           expected)
 
         os.symlink(ntpath.basename(ABSTFN) + "a\\b", ABSTFN + "a")
-        self.assertPathEqual(ntpath.realpath(ABSTFN + "a"), P + ABSTFN + "a")
+        self.assertPathEqual(ntpath.realpath(ABSTFN + "a"), ABSTFN + "a")
 
         os.symlink("..\\" + ntpath.basename(ntpath.dirname(ABSTFN))
                    + "\\" + ntpath.basename(ABSTFN) + "c", ABSTFN + "c")
-        self.assertPathEqual(ntpath.realpath(ABSTFN + "c"), P + ABSTFN + "c")
+        self.assertPathEqual(ntpath.realpath(ABSTFN + "c"), ABSTFN + "c")
 
         # Test using relative path as well.
-        self.assertPathEqual(ntpath.realpath(ntpath.basename(ABSTFN)), P + ABSTFN)
+        self.assertPathEqual(ntpath.realpath(ntpath.basename(ABSTFN)), ABSTFN)
 
     @support.skip_unless_symlink
     @unittest.skipUnless(HAVE_GETFINALPATHNAME, 'need _getfinalpathname')
diff --git a/Misc/NEWS.d/next/Windows/2019-10-02-15-38-49.bpo-38355.n3AWX6.rst b/Misc/NEWS.d/next/Windows/2019-10-02-15-38-49.bpo-38355.n3AWX6.rst
new file mode 100644
index 0000000000000..56e0f56505522
--- /dev/null
+++ b/Misc/NEWS.d/next/Windows/2019-10-02-15-38-49.bpo-38355.n3AWX6.rst
@@ -0,0 +1 @@
+Fixes ``ntpath.realpath`` failing on ``sys.executable``.


From webhook-mailer at python.org  Thu Oct  3 11:31:25 2019
From: webhook-mailer at python.org (Steve Dower)
Date: Thu, 03 Oct 2019 15:31:25 -0000
Subject: [Python-checkins] bpo-38359: Ensures pyw.exe launcher reads correct
 registry key (GH-16561)
Message-ID: <mailman.197.1570116686.32504.python-checkins@python.org>

https://github.com/python/cpython/commit/353fb1ecbfd58752dabae115c4964095e1e35e5f
commit: 353fb1ecbfd58752dabae115c4964095e1e35e5f
branch: master
author: Steve Dower <steve.dower at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-03T08:31:21-07:00
summary:

bpo-38359: Ensures pyw.exe launcher reads correct registry key (GH-16561)

files:
A Misc/NEWS.d/next/Windows/2019-10-03-08-04-14.bpo-38359.wzwsl_.rst
M PC/launcher.c

diff --git a/Misc/NEWS.d/next/Windows/2019-10-03-08-04-14.bpo-38359.wzwsl_.rst b/Misc/NEWS.d/next/Windows/2019-10-03-08-04-14.bpo-38359.wzwsl_.rst
new file mode 100644
index 0000000000000..ab6f29baf5358
--- /dev/null
+++ b/Misc/NEWS.d/next/Windows/2019-10-03-08-04-14.bpo-38359.wzwsl_.rst
@@ -0,0 +1 @@
+Ensures ``pyw.exe`` launcher reads correct registry key.
diff --git a/PC/launcher.c b/PC/launcher.c
index a260860f3c27a..2749a4e705467 100644
--- a/PC/launcher.c
+++ b/PC/launcher.c
@@ -141,6 +141,8 @@ static wchar_t * get_env(wchar_t * key)
 }
 
 #if defined(_DEBUG)
+/* Do not define EXECUTABLEPATH_VALUE in debug builds as it'll
+   never point to the debug build. */
 #if defined(_WINDOWS)
 
 #define PYTHON_EXECUTABLE L"pythonw_d.exe"
@@ -154,10 +156,12 @@ static wchar_t * get_env(wchar_t * key)
 #if defined(_WINDOWS)
 
 #define PYTHON_EXECUTABLE L"pythonw.exe"
+#define EXECUTABLEPATH_VALUE L"WindowedExecutablePath"
 
 #else
 
 #define PYTHON_EXECUTABLE L"python.exe"
+#define EXECUTABLEPATH_VALUE L"ExecutablePath"
 
 #endif
 #endif
@@ -297,8 +301,12 @@ _locate_pythons_for_key(HKEY root, LPCWSTR subkey, REGSAM flags, int bits,
                 }
                 data_size = sizeof(ip->executable) - 1;
                 append_name = FALSE;
-                status = RegQueryValueExW(ip_key, L"ExecutablePath", NULL, &type,
+#ifdef EXECUTABLEPATH_VALUE
+                status = RegQueryValueExW(ip_key, EXECUTABLEPATH_VALUE, NULL, &type,
                                           (LPBYTE)ip->executable, &data_size);
+#else
+                status = ERROR_FILE_NOT_FOUND; /* actual error doesn't matter */
+#endif
                 if (status != ERROR_SUCCESS || type != REG_SZ || !data_size) {
                     append_name = TRUE;
                     data_size = sizeof(ip->executable) - 1;


From webhook-mailer at python.org  Thu Oct  3 11:44:57 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Thu, 03 Oct 2019 15:44:57 -0000
Subject: [Python-checkins] [3.8] bpo-36670,
 regrtest: Fix WindowsLoadTracker() for partial line (GH-16550)
 (GH-16560)
Message-ID: <mailman.198.1570117499.32504.python-checkins@python.org>

https://github.com/python/cpython/commit/f9016e5fc9424d41c8e53de84f8699e78769a14a
commit: f9016e5fc9424d41c8e53de84f8699e78769a14a
branch: 3.7
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-03T08:44:52-07:00
summary:

[3.8] bpo-36670, regrtest: Fix WindowsLoadTracker() for partial line (GH-16550) (GH-16560)


* bpo-36670, regrtest: Fix WindowsLoadTracker() for partial line (GH-16550)

WindowsLoadTracker.read_output() now uses a short buffer for
incomplete line.

(cherry picked from commit 3e04cd268ee9a57f95dc78d8974b21a6fac3f666)

* bpo-36670: Enhance regrtest WindowsLoadTracker (GH-16553)

The last line is now passed to the parser even if it does not end
with a newline, but only if it's a valid value.

(cherry picked from commit c65119d5bfded03f80a9805889391b66fa7bf551)

* bpo-36670: Enhance regrtest (GH-16556)

* Add log() method: add timestamp and load average prefixes
  to main messages.
* WindowsLoadTracker:

  * LOAD_FACTOR_1 is now computed using SAMPLING_INTERVAL
  * Initialize the load to the arithmetic mean of the first 5 values
    of the Processor Queue Length value (so over 5 seconds), rather
    than 0.0.
  * Handle BrokenPipeError and when typeperf exit.

* format_duration(1.5) now returns '1.5 sec', rather than
  '1 sec 500 ms'

(cherry picked from commit 098e25672f1c3578855d5ded4f5147795c9ed956)
(cherry picked from commit de3195c937d5fca0d79cc93dbafa76c0f89ca5b8)

Co-authored-by: Victor Stinner <vstinner at python.org>

files:
M Lib/test/libregrtest/main.py
M Lib/test/libregrtest/runtest_mp.py
M Lib/test/libregrtest/utils.py
M Lib/test/libregrtest/win_utils.py
M Lib/test/test_regrtest.py

diff --git a/Lib/test/libregrtest/main.py b/Lib/test/libregrtest/main.py
index 5ae8878638a5c..05878a8af4c32 100644
--- a/Lib/test/libregrtest/main.py
+++ b/Lib/test/libregrtest/main.py
@@ -138,16 +138,8 @@ def accumulate_result(self, result, rerun=False):
                     print(xml_data, file=sys.__stderr__)
                     raise
 
-    def display_progress(self, test_index, text):
-        if self.ns.quiet:
-            return
-
-        # "[ 51/405/1] test_tcl passed"
-        line = f"{test_index:{self.test_count_width}}{self.test_count}"
-        fails = len(self.bad) + len(self.environment_changed)
-        if fails and not self.ns.pgo:
-            line = f"{line}/{fails}"
-        line = f"[{line}] {text}"
+    def log(self, line=''):
+        empty = not line
 
         # add the system load prefix: "load avg: 1.80 "
         load_avg = self.getloadavg()
@@ -158,8 +150,23 @@ def display_progress(self, test_index, text):
         test_time = time.monotonic() - self.start_time
         test_time = datetime.timedelta(seconds=int(test_time))
         line = f"{test_time} {line}"
+
+        if empty:
+            line = line[:-1]
+
         print(line, flush=True)
 
+    def display_progress(self, test_index, text):
+        if self.ns.quiet:
+            return
+
+        # "[ 51/405/1] test_tcl passed"
+        line = f"{test_index:{self.test_count_width}}{self.test_count}"
+        fails = len(self.bad) + len(self.environment_changed)
+        if fails and not self.ns.pgo:
+            line = f"{line}/{fails}"
+        self.log(f"[{line}] {text}")
+
     def parse_args(self, kwargs):
         ns = _parse_args(sys.argv[1:], **kwargs)
 
@@ -297,11 +304,11 @@ def rerun_failed_tests(self):
 
         self.first_result = self.get_tests_result()
 
-        print()
-        print("Re-running failed tests in verbose mode")
+        self.log()
+        self.log("Re-running failed tests in verbose mode")
         self.rerun = self.bad[:]
         for test_name in self.rerun:
-            print(f"Re-running {test_name} in verbose mode", flush=True)
+            self.log(f"Re-running {test_name} in verbose mode")
             self.ns.verbose = True
             result = runtest(self.ns, test_name)
 
@@ -382,7 +389,7 @@ def run_tests_sequential(self):
 
         save_modules = sys.modules.keys()
 
-        print("Run tests sequentially")
+        self.log("Run tests sequentially")
 
         previous_test = None
         for test_index, test_name in enumerate(self.tests, 1):
diff --git a/Lib/test/libregrtest/runtest_mp.py b/Lib/test/libregrtest/runtest_mp.py
index 38b05781de5fc..9d5a699852b9c 100644
--- a/Lib/test/libregrtest/runtest_mp.py
+++ b/Lib/test/libregrtest/runtest_mp.py
@@ -104,13 +104,14 @@ class ExitThread(Exception):
 
 
 class TestWorkerProcess(threading.Thread):
-    def __init__(self, worker_id, pending, output, ns, timeout):
+    def __init__(self, worker_id, runner):
         super().__init__()
         self.worker_id = worker_id
-        self.pending = pending
-        self.output = output
-        self.ns = ns
-        self.timeout = timeout
+        self.pending = runner.pending
+        self.output = runner.output
+        self.ns = runner.ns
+        self.timeout = runner.worker_timeout
+        self.regrtest = runner.regrtest
         self.current_test_name = None
         self.start_time = None
         self._popen = None
@@ -292,7 +293,8 @@ def wait_stopped(self, start_time):
             if not self.is_alive():
                 break
             dt = time.monotonic() - start_time
-            print(f"Waiting for {self} thread for {format_duration(dt)}", flush=True)
+            self.regrtest.log(f"Waiting for {self} thread "
+                              f"for {format_duration(dt)}")
             if dt > JOIN_TIMEOUT:
                 print_warning(f"Failed to join {self} in {format_duration(dt)}")
                 break
@@ -314,6 +316,7 @@ def get_running(workers):
 class MultiprocessTestRunner:
     def __init__(self, regrtest):
         self.regrtest = regrtest
+        self.log = self.regrtest.log
         self.ns = regrtest.ns
         self.output = queue.Queue()
         self.pending = MultiprocessIterator(self.regrtest.tests)
@@ -324,11 +327,10 @@ def __init__(self, regrtest):
         self.workers = None
 
     def start_workers(self):
-        self.workers = [TestWorkerProcess(index, self.pending, self.output,
-                                          self.ns, self.worker_timeout)
+        self.workers = [TestWorkerProcess(index, self)
                         for index in range(1, self.ns.use_mp + 1)]
-        print("Run tests in parallel using %s child processes"
-              % len(self.workers))
+        self.log("Run tests in parallel using %s child processes"
+                 % len(self.workers))
         for worker in self.workers:
             worker.start()
 
@@ -362,7 +364,7 @@ def _get_result(self):
             # display progress
             running = get_running(self.workers)
             if running and not self.ns.pgo:
-                print('running: %s' % ', '.join(running), flush=True)
+                self.log('running: %s' % ', '.join(running))
 
     def display_result(self, mp_result):
         result = mp_result.result
@@ -382,8 +384,7 @@ def _process_result(self, item):
         if item[0]:
             # Thread got an exception
             format_exc = item[1]
-            print(f"regrtest worker thread failed: {format_exc}",
-                  file=sys.stderr, flush=True)
+            print_warning(f"regrtest worker thread failed: {format_exc}")
             return True
 
         self.test_index += 1
diff --git a/Lib/test/libregrtest/utils.py b/Lib/test/libregrtest/utils.py
index fb9971a64f66c..98a60f7a747d9 100644
--- a/Lib/test/libregrtest/utils.py
+++ b/Lib/test/libregrtest/utils.py
@@ -16,11 +16,14 @@ def format_duration(seconds):
     if minutes:
         parts.append('%s min' % minutes)
     if seconds:
-        parts.append('%s sec' % seconds)
-    if ms:
-        parts.append('%s ms' % ms)
+        if parts:
+            # 2 min 1 sec
+            parts.append('%s sec' % seconds)
+        else:
+            # 1.0 sec
+            parts.append('%.1f sec' % (seconds + ms / 1000))
     if not parts:
-        return '0 ms'
+        return '%s ms' % ms
 
     parts = parts[:2]
     return ' '.join(parts)
diff --git a/Lib/test/libregrtest/win_utils.py b/Lib/test/libregrtest/win_utils.py
index f0c17b906f519..028c01106dee0 100644
--- a/Lib/test/libregrtest/win_utils.py
+++ b/Lib/test/libregrtest/win_utils.py
@@ -1,4 +1,5 @@
 import _winapi
+import math
 import msvcrt
 import os
 import subprocess
@@ -10,11 +11,14 @@
 
 # Max size of asynchronous reads
 BUFSIZE = 8192
-# Exponential damping factor (see below)
-LOAD_FACTOR_1 = 0.9200444146293232478931553241
-
 # Seconds per measurement
-SAMPLING_INTERVAL = 5
+SAMPLING_INTERVAL = 1
+# Exponential damping factor to compute exponentially weighted moving average
+# on 1 minute (60 seconds)
+LOAD_FACTOR_1 = 1 / math.exp(SAMPLING_INTERVAL / 60)
+# Initialize the load using the arithmetic mean of the first NVALUE values
+# of the Processor Queue Length
+NVALUE = 5
 # Windows registry subkey of HKEY_LOCAL_MACHINE where the counter names
 # of typeperf are registered
 COUNTER_REGISTRY_KEY = (r"SOFTWARE\Microsoft\Windows NT\CurrentVersion"
@@ -30,9 +34,10 @@ class WindowsLoadTracker():
     """
 
     def __init__(self):
-        self.load = 0.0
-        self.counter_name = ''
-        self.popen = None
+        self._values = []
+        self._load = None
+        self._buffer = ''
+        self._popen = None
         self.start()
 
     def start(self):
@@ -64,7 +69,7 @@ def start(self):
         # Spawn off the load monitor
         counter_name = self._get_counter_name()
         command = ['typeperf', counter_name, '-si', str(SAMPLING_INTERVAL)]
-        self.popen = subprocess.Popen(' '.join(command), stdout=command_stdout, cwd=support.SAVEDCWD)
+        self._popen = subprocess.Popen(' '.join(command), stdout=command_stdout, cwd=support.SAVEDCWD)
 
         # Close our copy of the write end of the pipe
         os.close(command_stdout)
@@ -84,52 +89,88 @@ def _get_counter_name(self):
         process_queue_length = counters_dict['44']
         return f'"\\{system}\\{process_queue_length}"'
 
-    def close(self):
-        if self.popen is None:
+    def close(self, kill=True):
+        if self._popen is None:
             return
-        self.popen.kill()
-        self.popen.wait()
-        self.popen = None
+
+        self._load = None
+
+        if kill:
+            self._popen.kill()
+        self._popen.wait()
+        self._popen = None
 
     def __del__(self):
         self.close()
 
-    def read_output(self):
+    def _parse_line(self, line):
+        # typeperf outputs in a CSV format like this:
+        # "07/19/2018 01:32:26.605","3.000000"
+        # (date, process queue length)
+        tokens = line.split(',')
+        if len(tokens) != 2:
+            raise ValueError
+
+        value = tokens[1]
+        if not value.startswith('"') or not value.endswith('"'):
+            raise ValueError
+        value = value[1:-1]
+        return float(value)
+
+    def _read_lines(self):
         overlapped, _ = _winapi.ReadFile(self.pipe, BUFSIZE, True)
         bytes_read, res = overlapped.GetOverlappedResult(False)
         if res != 0:
-            return
+            return ()
 
         output = overlapped.getbuffer()
-        return output.decode('oem', 'replace')
+        output = output.decode('oem', 'replace')
+        output = self._buffer + output
+        lines = output.splitlines(True)
+
+        # bpo-36670: typeperf only writes a newline *before* writing a value,
+        # not after. Sometimes, the written line in incomplete (ex: only
+        # timestamp, without the process queue length). Only pass the last line
+        # to the parser if it's a valid value, otherwise store it in
+        # self._buffer.
+        try:
+            self._parse_line(lines[-1])
+        except ValueError:
+            self._buffer = lines.pop(-1)
+        else:
+            self._buffer = ''
+
+        return lines
 
     def getloadavg(self):
-        typeperf_output = self.read_output()
-        # Nothing to update, just return the current load
-        if not typeperf_output:
-            return self.load
+        if self._popen is None:
+            return None
+
+        returncode = self._popen.poll()
+        if returncode is not None:
+            self.close(kill=False)
+            return None
+
+        try:
+            lines = self._read_lines()
+        except BrokenPipeError:
+            self.close()
+            return None
+
+        for line in lines:
+            line = line.rstrip()
 
-        # Process the backlog of load values
-        for line in typeperf_output.splitlines():
             # Ignore the initial header:
             # "(PDH-CSV 4.0)","\\\\WIN\\System\\Processor Queue Length"
-            if '\\\\' in line:
+            if 'PDH-CSV' in line:
                 continue
 
             # Ignore blank lines
-            if not line.strip():
+            if not line:
                 continue
 
-            # typeperf outputs in a CSV format like this:
-            # "07/19/2018 01:32:26.605","3.000000"
-            # (date, process queue length)
             try:
-                tokens = line.split(',')
-                if len(tokens) != 2:
-                    raise ValueError
-
-                value = tokens[1].replace('"', '')
-                load = float(value)
+                processor_queue_length = self._parse_line(line)
             except ValueError:
                 print_warning("Failed to parse typeperf output: %a" % line)
                 continue
@@ -137,7 +178,13 @@ def getloadavg(self):
             # We use an exponentially weighted moving average, imitating the
             # load calculation on Unix systems.
             # https://en.wikipedia.org/wiki/Load_(computing)#Unix-style_load_calculation
-            new_load = self.load * LOAD_FACTOR_1 + load * (1.0 - LOAD_FACTOR_1)
-            self.load = new_load
-
-        return self.load
+            # https://en.wikipedia.org/wiki/Moving_average#Exponential_moving_average
+            if self._load is not None:
+                self._load = (self._load * LOAD_FACTOR_1
+                              + processor_queue_length  * (1.0 - LOAD_FACTOR_1))
+            elif len(self._values) < NVALUE:
+                self._values.append(processor_queue_length)
+            else:
+                self._load = sum(self._values) / len(self._values)
+
+        return self._load
diff --git a/Lib/test/test_regrtest.py b/Lib/test/test_regrtest.py
index 7336adc4f1b8b..26c9b16293c2b 100644
--- a/Lib/test/test_regrtest.py
+++ b/Lib/test/test_regrtest.py
@@ -24,6 +24,7 @@
 Py_DEBUG = hasattr(sys, 'gettotalrefcount')
 ROOT_DIR = os.path.join(os.path.dirname(__file__), '..', '..')
 ROOT_DIR = os.path.abspath(os.path.normpath(ROOT_DIR))
+LOG_PREFIX = r'[0-9]+:[0-9]+:[0-9]+ (?:load avg: [0-9]+\.[0-9]{2} )?'
 
 TEST_INTERRUPTED = textwrap.dedent("""
     from signal import SIGINT
@@ -390,8 +391,8 @@ def check_line(self, output, regex):
         self.assertRegex(output, regex)
 
     def parse_executed_tests(self, output):
-        regex = (r'^[0-9]+:[0-9]+:[0-9]+ (?:load avg: [0-9]+\.[0-9]{2} )?\[ *[0-9]+(?:/ *[0-9]+)*\] (%s)'
-                 % self.TESTNAME_REGEX)
+        regex = (r'^%s\[ *[0-9]+(?:/ *[0-9]+)*\] (%s)'
+                 % (LOG_PREFIX, self.TESTNAME_REGEX))
         parser = re.finditer(regex, output, re.MULTILINE)
         return list(match.group(1) for match in parser)
 
@@ -451,9 +452,10 @@ def list_regex(line_format, tests):
         if rerun:
             regex = list_regex('%s re-run test%s', rerun)
             self.check_line(output, regex)
-            self.check_line(output, "Re-running failed tests in verbose mode")
+            regex = LOG_PREFIX + r"Re-running failed tests in verbose mode"
+            self.check_line(output, regex)
             for test_name in rerun:
-                regex = f"Re-running {test_name} in verbose mode"
+                regex = LOG_PREFIX + f"Re-running {test_name} in verbose mode"
                 self.check_line(output, regex)
 
         if no_test_ran:
@@ -1173,9 +1175,9 @@ def test_format_duration(self):
         self.assertEqual(utils.format_duration(10e-3),
                          '10 ms')
         self.assertEqual(utils.format_duration(1.5),
-                         '1 sec 500 ms')
+                         '1.5 sec')
         self.assertEqual(utils.format_duration(1),
-                         '1 sec')
+                         '1.0 sec')
         self.assertEqual(utils.format_duration(2 * 60),
                          '2 min')
         self.assertEqual(utils.format_duration(2 * 60 + 1),


From webhook-mailer at python.org  Thu Oct  3 11:50:00 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Thu, 03 Oct 2019 15:50:00 -0000
Subject: [Python-checkins] bpo-38355: Fix ntpath.realpath failing on
 sys.executable (GH-16551)
Message-ID: <mailman.199.1570117801.32504.python-checkins@python.org>

https://github.com/python/cpython/commit/6067e1d2bebccc2e73dd729d25c98df7bc9e2d59
commit: 6067e1d2bebccc2e73dd729d25c98df7bc9e2d59
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-03T08:49:56-07:00
summary:

bpo-38355: Fix ntpath.realpath failing on sys.executable (GH-16551)

(cherry picked from commit a0e3d27e4e3cb5b67e325df080fb18b70c2910cf)

Co-authored-by: Steve Dower <steve.dower at python.org>

files:
A Misc/NEWS.d/next/Windows/2019-10-02-15-38-49.bpo-38355.n3AWX6.rst
M Lib/ntpath.py
M Lib/test/test_ntpath.py

diff --git a/Lib/ntpath.py b/Lib/ntpath.py
index 01f1f423c906c..d4ecff97c9528 100644
--- a/Lib/ntpath.py
+++ b/Lib/ntpath.py
@@ -560,13 +560,6 @@ def _readlink_deep(path, seen=None):
         return path
 
     def _getfinalpathname_nonstrict(path):
-        # Fast path to get the final path name. If this succeeds, there
-        # is no need to go any further.
-        try:
-            return _getfinalpathname(path)
-        except OSError:
-            pass
-
         # These error codes indicate that we should stop resolving the path
         # and return the value we currently have.
         # 1: ERROR_INVALID_FUNCTION
@@ -579,8 +572,9 @@ def _getfinalpathname_nonstrict(path):
         # 67: ERROR_BAD_NET_NAME (implies remote server unavailable)
         # 87: ERROR_INVALID_PARAMETER
         # 123: ERROR_INVALID_NAME
+        # 1920: ERROR_CANT_ACCESS_FILE
         # 1921: ERROR_CANT_RESOLVE_FILENAME (implies unfollowable symlink)
-        allowed_winerror = 1, 2, 3, 5, 21, 32, 50, 67, 87, 123, 1921
+        allowed_winerror = 1, 2, 3, 5, 21, 32, 50, 67, 87, 123, 1920, 1921
 
         # Non-strict algorithm is to find as much of the target directory
         # as we can and join the rest.
@@ -615,9 +609,13 @@ def realpath(path):
             unc_prefix = '\\\\?\\UNC\\'
             new_unc_prefix = '\\\\'
             cwd = os.getcwd()
-        did_not_exist = not exists(path)
         had_prefix = path.startswith(prefix)
-        path = _getfinalpathname_nonstrict(path)
+        try:
+            path = _getfinalpathname(path)
+            initial_winerror = 0
+        except OSError as ex:
+            initial_winerror = ex.winerror
+            path = _getfinalpathname_nonstrict(path)
         # The path returned by _getfinalpathname will always start with \\?\ -
         # strip off that prefix unless it was already provided on the original
         # path.
@@ -635,7 +633,7 @@ def realpath(path):
             except OSError as ex:
                 # If the path does not exist and originally did not exist, then
                 # strip the prefix anyway.
-                if ex.winerror in {2, 3} and did_not_exist:
+                if ex.winerror == initial_winerror:
                     path = spath
         return path
 
diff --git a/Lib/test/test_ntpath.py b/Lib/test/test_ntpath.py
index 2f0faf94726fd..e0ec441985230 100644
--- a/Lib/test/test_ntpath.py
+++ b/Lib/test/test_ntpath.py
@@ -329,16 +329,14 @@ def test_realpath_symlink_loops(self):
         self.addCleanup(support.unlink, ABSTFN + "c")
         self.addCleanup(support.unlink, ABSTFN + "a")
 
-        P = "\\\\?\\"
-
         os.symlink(ABSTFN, ABSTFN)
-        self.assertPathEqual(ntpath.realpath(ABSTFN), P + ABSTFN)
+        self.assertPathEqual(ntpath.realpath(ABSTFN), ABSTFN)
 
         # cycles are non-deterministic as to which path is returned, but
         # it will always be the fully resolved path of one member of the cycle
         os.symlink(ABSTFN + "1", ABSTFN + "2")
         os.symlink(ABSTFN + "2", ABSTFN + "1")
-        expected = (P + ABSTFN + "1", P + ABSTFN + "2")
+        expected = (ABSTFN + "1", ABSTFN + "2")
         self.assertPathIn(ntpath.realpath(ABSTFN + "1"), expected)
         self.assertPathIn(ntpath.realpath(ABSTFN + "2"), expected)
 
@@ -357,14 +355,14 @@ def test_realpath_symlink_loops(self):
                           expected)
 
         os.symlink(ntpath.basename(ABSTFN) + "a\\b", ABSTFN + "a")
-        self.assertPathEqual(ntpath.realpath(ABSTFN + "a"), P + ABSTFN + "a")
+        self.assertPathEqual(ntpath.realpath(ABSTFN + "a"), ABSTFN + "a")
 
         os.symlink("..\\" + ntpath.basename(ntpath.dirname(ABSTFN))
                    + "\\" + ntpath.basename(ABSTFN) + "c", ABSTFN + "c")
-        self.assertPathEqual(ntpath.realpath(ABSTFN + "c"), P + ABSTFN + "c")
+        self.assertPathEqual(ntpath.realpath(ABSTFN + "c"), ABSTFN + "c")
 
         # Test using relative path as well.
-        self.assertPathEqual(ntpath.realpath(ntpath.basename(ABSTFN)), P + ABSTFN)
+        self.assertPathEqual(ntpath.realpath(ntpath.basename(ABSTFN)), ABSTFN)
 
     @support.skip_unless_symlink
     @unittest.skipUnless(HAVE_GETFINALPATHNAME, 'need _getfinalpathname')
diff --git a/Misc/NEWS.d/next/Windows/2019-10-02-15-38-49.bpo-38355.n3AWX6.rst b/Misc/NEWS.d/next/Windows/2019-10-02-15-38-49.bpo-38355.n3AWX6.rst
new file mode 100644
index 0000000000000..56e0f56505522
--- /dev/null
+++ b/Misc/NEWS.d/next/Windows/2019-10-02-15-38-49.bpo-38355.n3AWX6.rst
@@ -0,0 +1 @@
+Fixes ``ntpath.realpath`` failing on ``sys.executable``.


From webhook-mailer at python.org  Thu Oct  3 11:51:17 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Thu, 03 Oct 2019 15:51:17 -0000
Subject: [Python-checkins] bpo-38359: Ensures pyw.exe launcher reads correct
 registry key (GH-16561)
Message-ID: <mailman.200.1570117878.32504.python-checkins@python.org>

https://github.com/python/cpython/commit/e74fa70bcc379de1ce956273c5386fc8a7b9c5e6
commit: e74fa70bcc379de1ce956273c5386fc8a7b9c5e6
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-03T08:51:12-07:00
summary:

bpo-38359: Ensures pyw.exe launcher reads correct registry key (GH-16561)

(cherry picked from commit 353fb1ecbfd58752dabae115c4964095e1e35e5f)

Co-authored-by: Steve Dower <steve.dower at python.org>

files:
A Misc/NEWS.d/next/Windows/2019-10-03-08-04-14.bpo-38359.wzwsl_.rst
M PC/launcher.c

diff --git a/Misc/NEWS.d/next/Windows/2019-10-03-08-04-14.bpo-38359.wzwsl_.rst b/Misc/NEWS.d/next/Windows/2019-10-03-08-04-14.bpo-38359.wzwsl_.rst
new file mode 100644
index 0000000000000..ab6f29baf5358
--- /dev/null
+++ b/Misc/NEWS.d/next/Windows/2019-10-03-08-04-14.bpo-38359.wzwsl_.rst
@@ -0,0 +1 @@
+Ensures ``pyw.exe`` launcher reads correct registry key.
diff --git a/PC/launcher.c b/PC/launcher.c
index a260860f3c27a..2749a4e705467 100644
--- a/PC/launcher.c
+++ b/PC/launcher.c
@@ -141,6 +141,8 @@ static wchar_t * get_env(wchar_t * key)
 }
 
 #if defined(_DEBUG)
+/* Do not define EXECUTABLEPATH_VALUE in debug builds as it'll
+   never point to the debug build. */
 #if defined(_WINDOWS)
 
 #define PYTHON_EXECUTABLE L"pythonw_d.exe"
@@ -154,10 +156,12 @@ static wchar_t * get_env(wchar_t * key)
 #if defined(_WINDOWS)
 
 #define PYTHON_EXECUTABLE L"pythonw.exe"
+#define EXECUTABLEPATH_VALUE L"WindowedExecutablePath"
 
 #else
 
 #define PYTHON_EXECUTABLE L"python.exe"
+#define EXECUTABLEPATH_VALUE L"ExecutablePath"
 
 #endif
 #endif
@@ -297,8 +301,12 @@ _locate_pythons_for_key(HKEY root, LPCWSTR subkey, REGSAM flags, int bits,
                 }
                 data_size = sizeof(ip->executable) - 1;
                 append_name = FALSE;
-                status = RegQueryValueExW(ip_key, L"ExecutablePath", NULL, &type,
+#ifdef EXECUTABLEPATH_VALUE
+                status = RegQueryValueExW(ip_key, EXECUTABLEPATH_VALUE, NULL, &type,
                                           (LPBYTE)ip->executable, &data_size);
+#else
+                status = ERROR_FILE_NOT_FOUND; /* actual error doesn't matter */
+#endif
                 if (status != ERROR_SUCCESS || type != REG_SZ || !data_size) {
                     append_name = TRUE;
                     data_size = sizeof(ip->executable) - 1;


From webhook-mailer at python.org  Thu Oct  3 17:08:35 2019
From: webhook-mailer at python.org (Yury Selivanov)
Date: Thu, 03 Oct 2019 21:08:35 -0000
Subject: [Python-checkins] bpo-34344 Fix
 AbstractEventLoopPolicy.get_event_loop docstring (GH-16463)
Message-ID: <mailman.201.1570136917.32504.python-checkins@python.org>

https://github.com/python/cpython/commit/b23a8423a923077e4f83d3f328bb7542b4c940ed
commit: b23a8423a923077e4f83d3f328bb7542b4c940ed
branch: master
author: idomic <michael.ido at gmail.com>
committer: Yury Selivanov <yury at magic.io>
date: 2019-10-03T17:08:29-04:00
summary:

bpo-34344 Fix AbstractEventLoopPolicy.get_event_loop docstring (GH-16463)

files:
M Lib/asyncio/events.py

diff --git a/Lib/asyncio/events.py b/Lib/asyncio/events.py
index 2f06c4ae795d2..36b7ea307e03b 100644
--- a/Lib/asyncio/events.py
+++ b/Lib/asyncio/events.py
@@ -630,9 +630,9 @@ def __init__(self):
         self._local = self._Local()
 
     def get_event_loop(self):
-        """Get the event loop.
+        """Get the event loop for the current context.
 
-        This may be None or an instance of EventLoop.
+        Returns an instance of EventLoop or raises an exception.
         """
         if (self._local._loop is None and
                 not self._local._set_called and


From webhook-mailer at python.org  Thu Oct  3 17:17:04 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Thu, 03 Oct 2019 21:17:04 -0000
Subject: [Python-checkins] bpo-38353: Fix calculate_argv0_path() for
 symlinks (GH-16549)
Message-ID: <mailman.202.1570137424.32504.python-checkins@python.org>

https://github.com/python/cpython/commit/86ec5c65fe04ece16f3aada5292cc62726062ad0
commit: 86ec5c65fe04ece16f3aada5292cc62726062ad0
branch: master
author: Victor Stinner <vstinner at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-03T23:17:00+02:00
summary:

bpo-38353: Fix calculate_argv0_path() for symlinks (GH-16549)

calculate_argv0_path() must read argv0_path link, not read
program_full_path link.

files:
M Modules/getpath.c

diff --git a/Modules/getpath.c b/Modules/getpath.c
index 4246d4ca5ac0d..fca87c7b0ee93 100644
--- a/Modules/getpath.c
+++ b/Modules/getpath.c
@@ -959,7 +959,7 @@ calculate_argv0_path(PyCalculatePath *calculate, const wchar_t *program_full_pat
 #if HAVE_READLINK
     wchar_t tmpbuffer[MAXPATHLEN + 1];
     const size_t buflen = Py_ARRAY_LENGTH(tmpbuffer);
-    int linklen = _Py_wreadlink(program_full_path, tmpbuffer, buflen);
+    int linklen = _Py_wreadlink(argv0_path, tmpbuffer, buflen);
     while (linklen != -1) {
         if (_Py_isabs(tmpbuffer)) {
             /* tmpbuffer should never be longer than MAXPATHLEN,


From webhook-mailer at python.org  Thu Oct  3 17:27:01 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Thu, 03 Oct 2019 21:27:01 -0000
Subject: [Python-checkins] bpo-34344 Fix
 AbstractEventLoopPolicy.get_event_loop docstring (GH-16463)
Message-ID: <mailman.203.1570138022.32504.python-checkins@python.org>

https://github.com/python/cpython/commit/8edde5caabc611c6320fd104244abe3a409cf0b6
commit: 8edde5caabc611c6320fd104244abe3a409cf0b6
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-03T14:26:57-07:00
summary:

bpo-34344 Fix AbstractEventLoopPolicy.get_event_loop docstring (GH-16463)

(cherry picked from commit b23a8423a923077e4f83d3f328bb7542b4c940ed)

Co-authored-by: idomic <michael.ido at gmail.com>

files:
M Lib/asyncio/events.py

diff --git a/Lib/asyncio/events.py b/Lib/asyncio/events.py
index d381b1c596239..ca08663a5b358 100644
--- a/Lib/asyncio/events.py
+++ b/Lib/asyncio/events.py
@@ -626,9 +626,9 @@ def __init__(self):
         self._local = self._Local()
 
     def get_event_loop(self):
-        """Get the event loop.
+        """Get the event loop for the current context.
 
-        This may be None or an instance of EventLoop.
+        Returns an instance of EventLoop or raises an exception.
         """
         if (self._local._loop is None and
                 not self._local._set_called and


From webhook-mailer at python.org  Thu Oct  3 17:27:39 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Thu, 03 Oct 2019 21:27:39 -0000
Subject: [Python-checkins] bpo-34344 Fix
 AbstractEventLoopPolicy.get_event_loop docstring (GH-16463)
Message-ID: <mailman.204.1570138060.32504.python-checkins@python.org>

https://github.com/python/cpython/commit/4de3fbe27f672dcd9c24d3e99258f39b12c365a5
commit: 4de3fbe27f672dcd9c24d3e99258f39b12c365a5
branch: 3.7
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-03T14:27:35-07:00
summary:

bpo-34344 Fix AbstractEventLoopPolicy.get_event_loop docstring (GH-16463)

(cherry picked from commit b23a8423a923077e4f83d3f328bb7542b4c940ed)

Co-authored-by: idomic <michael.ido at gmail.com>

files:
M Lib/asyncio/events.py

diff --git a/Lib/asyncio/events.py b/Lib/asyncio/events.py
index e4e632206af1b..e8a6a1beeab9b 100644
--- a/Lib/asyncio/events.py
+++ b/Lib/asyncio/events.py
@@ -630,9 +630,9 @@ def __init__(self):
         self._local = self._Local()
 
     def get_event_loop(self):
-        """Get the event loop.
+        """Get the event loop for the current context.
 
-        This may be None or an instance of EventLoop.
+        Returns an instance of EventLoop or raises an exception.
         """
         if (self._local._loop is None and
                 not self._local._set_called and


From webhook-mailer at python.org  Thu Oct  3 18:43:15 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Thu, 03 Oct 2019 22:43:15 -0000
Subject: [Python-checkins] bpo-38353: Fix compiler warning in
 pycore_initconfig.h (GH-16570)
Message-ID: <mailman.205.1570142596.32504.python-checkins@python.org>

https://github.com/python/cpython/commit/c515b573af1189fc46ddcd3323c53ed073b84848
commit: c515b573af1189fc46ddcd3323c53ed073b84848
branch: master
author: Victor Stinner <vstinner at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-04T00:43:10+02:00
summary:

bpo-38353: Fix compiler warning in pycore_initconfig.h (GH-16570)

Replace "_PyRuntimeState" with "struct pyruntimestate" to avoid
a warning on typedef re-definition.

files:
M Include/internal/pycore_initconfig.h

diff --git a/Include/internal/pycore_initconfig.h b/Include/internal/pycore_initconfig.h
index 4a1a111af6834..8c6706c95cbd1 100644
--- a/Include/internal/pycore_initconfig.h
+++ b/Include/internal/pycore_initconfig.h
@@ -9,7 +9,7 @@ extern "C" {
 #endif
 
 /* Forward declaration */
-typedef struct pyruntimestate _PyRuntimeState;
+struct pyruntimestate;
 
 /* --- PyStatus ----------------------------------------------- */
 
@@ -151,7 +151,7 @@ extern PyStatus _PyConfig_Copy(
     const PyConfig *config2);
 extern PyStatus _PyConfig_InitPathConfig(PyConfig *config);
 extern void _PyConfig_Write(const PyConfig *config,
-    _PyRuntimeState *runtime);
+    struct pyruntimestate *runtime);
 extern PyStatus _PyConfig_SetPyArgv(
     PyConfig *config,
     const _PyArgv *args);


From webhook-mailer at python.org  Thu Oct  3 20:21:11 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Fri, 04 Oct 2019 00:21:11 -0000
Subject: [Python-checkins] bpo-38353: Fix compiler warning in internal
 headers (GH-16573)
Message-ID: <mailman.206.1570148473.32504.python-checkins@python.org>

https://github.com/python/cpython/commit/e982d8b64f5d358c578bfca5cdfe4524dbc74000
commit: e982d8b64f5d358c578bfca5cdfe4524dbc74000
branch: master
author: Victor Stinner <vstinner at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-04T02:21:05+02:00
summary:

bpo-38353: Fix compiler warning in internal headers (GH-16573)

Replace "_PyRuntimeState" with "struct pyruntimestate" to avoid
a warning on typedef re-definition.

files:
M Include/internal/pycore_ceval.h
M Include/internal/pycore_pylifecycle.h
M Include/internal/pycore_traceback.h

diff --git a/Include/internal/pycore_ceval.h b/Include/internal/pycore_ceval.h
index 65537b1bae4a1..f20e4ebb7b18c 100644
--- a/Include/internal/pycore_ceval.h
+++ b/Include/internal/pycore_ceval.h
@@ -9,10 +9,10 @@ extern "C" {
 #endif
 
 /* Forward declarations */
-typedef struct pyruntimestate _PyRuntimeState;
+struct pyruntimestate;
 struct _ceval_runtime_state;
 
-PyAPI_FUNC(void) _Py_FinishPendingCalls(_PyRuntimeState *runtime);
+PyAPI_FUNC(void) _Py_FinishPendingCalls(struct pyruntimestate *runtime);
 PyAPI_FUNC(void) _PyEval_Initialize(struct _ceval_runtime_state *);
 PyAPI_FUNC(void) _PyEval_FiniThreads(
     struct _ceval_runtime_state *ceval);
@@ -26,7 +26,7 @@ PyAPI_FUNC(int) _PyEval_AddPendingCall(
 PyAPI_FUNC(void) _PyEval_SignalAsyncExc(
     struct _ceval_runtime_state *ceval);
 PyAPI_FUNC(void) _PyEval_ReInitThreads(
-    _PyRuntimeState *runtime);
+    struct pyruntimestate *runtime);
 PyAPI_FUNC(void) _PyEval_SetCoroutineOriginTrackingDepth(
     PyThreadState *tstate,
     int new_depth);
diff --git a/Include/internal/pycore_pylifecycle.h b/Include/internal/pycore_pylifecycle.h
index d6f9ecc8b4872..d89dbeacb240b 100644
--- a/Include/internal/pycore_pylifecycle.h
+++ b/Include/internal/pycore_pylifecycle.h
@@ -9,8 +9,8 @@ extern "C" {
 #endif
 
 /* Forward declarations */
-typedef struct _PyArgv _PyArgv;
-typedef struct pyruntimestate _PyRuntimeState;
+struct _PyArgv;
+struct pyruntimestate;
 
 /* True if the main interpreter thread exited due to an unhandled
  * KeyboardInterrupt exception, suggesting the user pressed ^C. */
@@ -38,14 +38,14 @@ extern PyStatus _PyFaulthandler_Init(int enable);
 extern int _PyTraceMalloc_Init(int enable);
 extern PyObject * _PyBuiltin_Init(PyThreadState *tstate);
 extern PyStatus _PySys_Create(
-    _PyRuntimeState *runtime,
+    struct pyruntimestate *runtime,
     PyThreadState *tstate,
     PyObject **sysmod_p);
 extern PyStatus _PySys_SetPreliminaryStderr(PyObject *sysdict);
 extern PyStatus _PySys_ReadPreinitWarnOptions(PyWideStringList *options);
 extern PyStatus _PySys_ReadPreinitXOptions(PyConfig *config);
 extern int _PySys_InitMain(
-    _PyRuntimeState *runtime,
+    struct pyruntimestate *runtime,
     PyThreadState *tstate);
 extern PyStatus _PyImport_Init(PyThreadState *tstate);
 extern PyStatus _PyExc_Init(void);
@@ -78,7 +78,7 @@ extern void PyOS_FiniInterrupts(void);
 extern void _PyExc_Fini(void);
 extern void _PyImport_Fini(void);
 extern void _PyImport_Fini2(void);
-extern void _PyGC_Fini(_PyRuntimeState *runtime);
+extern void _PyGC_Fini(struct pyruntimestate *runtime);
 extern void _PyType_Fini(void);
 extern void _Py_HashRandomization_Fini(void);
 extern void _PyUnicode_Fini(void);
@@ -89,18 +89,18 @@ extern void _PyTraceMalloc_Fini(void);
 extern void _PyWarnings_Fini(PyInterpreterState *interp);
 
 extern void _PyGILState_Init(
-    _PyRuntimeState *runtime,
+    struct pyruntimestate *runtime,
     PyThreadState *tstate);
-extern void _PyGILState_Fini(_PyRuntimeState *runtime);
+extern void _PyGILState_Fini(struct pyruntimestate *runtime);
 
-PyAPI_FUNC(void) _PyGC_DumpShutdownStats(_PyRuntimeState *runtime);
+PyAPI_FUNC(void) _PyGC_DumpShutdownStats(struct pyruntimestate *runtime);
 
 PyAPI_FUNC(PyStatus) _Py_PreInitializeFromPyArgv(
     const PyPreConfig *src_config,
-    const _PyArgv *args);
+    const struct _PyArgv *args);
 PyAPI_FUNC(PyStatus) _Py_PreInitializeFromConfig(
     const PyConfig *config,
-    const _PyArgv *args);
+    const struct _PyArgv *args);
 
 
 PyAPI_FUNC(int) _Py_HandleSystemExit(int *exitcode_p);
@@ -111,7 +111,7 @@ PyAPI_FUNC(void) _PyErr_Print(PyThreadState *tstate);
 PyAPI_FUNC(void) _PyErr_Display(PyObject *file, PyObject *exception,
                                 PyObject *value, PyObject *tb);
 
-PyAPI_FUNC(void) _PyThreadState_DeleteCurrent(_PyRuntimeState *runtime);
+PyAPI_FUNC(void) _PyThreadState_DeleteCurrent(struct pyruntimestate *runtime);
 
 #ifdef __cplusplus
 }
diff --git a/Include/internal/pycore_traceback.h b/Include/internal/pycore_traceback.h
index 8e58184c87ec9..99443d7ba2706 100644
--- a/Include/internal/pycore_traceback.h
+++ b/Include/internal/pycore_traceback.h
@@ -9,7 +9,7 @@ extern "C" {
 #endif
 
 /* Forward declaration */
-typedef struct _is PyInterpreterState;
+struct _is;
 
 /* Write the Python traceback into the file 'fd'. For example:
 
@@ -57,7 +57,7 @@ PyAPI_FUNC(void) _Py_DumpTraceback(
 
 PyAPI_FUNC(const char*) _Py_DumpTracebackThreads(
     int fd,
-    PyInterpreterState *interp,
+    struct _is *interp,
     PyThreadState *current_tstate);
 
 /* Write a Unicode object into the file descriptor fd. Encode the string to


From webhook-mailer at python.org  Thu Oct  3 20:22:43 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Fri, 04 Oct 2019 00:22:43 -0000
Subject: [Python-checkins] bpo-38353: Add subfunctions to getpath.c
 (GH-16572)
Message-ID: <mailman.207.1570148564.32504.python-checkins@python.org>

https://github.com/python/cpython/commit/03a8a56faca0c1851051269e3517d70cbce830b7
commit: 03a8a56faca0c1851051269e3517d70cbce830b7
branch: master
author: Victor Stinner <vstinner at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-04T02:22:39+02:00
summary:

bpo-38353: Add subfunctions to getpath.c (GH-16572)

Following symbolic links is now limited to 40 attempts, just to
prevent loops.

Add subfunctions:

* Add resolve_symlinks()
* Add calculate_argv0_path_framework()
* Add calculate_which()
* Add calculate_program_macos()

Fix also _Py_wreadlink(): readlink() result type is Py_ssize_t, not
int.

files:
M Modules/getpath.c
M Python/fileutils.c

diff --git a/Modules/getpath.c b/Modules/getpath.c
index fca87c7b0ee93..7a66eee881361 100644
--- a/Modules/getpath.c
+++ b/Modules/getpath.c
@@ -768,21 +768,98 @@ calculate_set_exec_prefix(PyCalculatePath *calculate,
 
 
 static PyStatus
-calculate_program_full_path(PyCalculatePath *calculate, _PyPathConfig *pathconfig)
+calculate_which(const wchar_t *path_env, wchar_t *program_name,
+                wchar_t *fullpath, size_t fullpath_len, int *found)
 {
-    PyStatus status;
-    wchar_t program_full_path[MAXPATHLEN + 1];
-    const size_t program_full_path_len = Py_ARRAY_LENGTH(program_full_path);
-    memset(program_full_path, 0, sizeof(program_full_path));
+    while (1) {
+        wchar_t *delim = wcschr(path_env, DELIM);
+
+        if (delim) {
+            size_t len = delim - path_env;
+            if (len >= fullpath_len) {
+                return PATHLEN_ERR();
+            }
+            wcsncpy(fullpath, path_env, len);
+            fullpath[len] = '\0';
+        }
+        else {
+            if (safe_wcscpy(fullpath, path_env,
+                        fullpath_len) < 0) {
+                return PATHLEN_ERR();
+            }
+        }
+
+        PyStatus status = joinpath(fullpath, program_name, fullpath_len);
+        if (_PyStatus_EXCEPTION(status)) {
+            return status;
+        }
+
+        if (isxfile(fullpath)) {
+            *found = 1;
+            return _PyStatus_OK();
+        }
+
+        if (!delim) {
+            break;
+        }
+        path_env = delim + 1;
+    }
+
+    /* not found */
+    return _PyStatus_OK();
+}
+
 
 #ifdef __APPLE__
+static PyStatus
+calculate_program_macos(wchar_t *fullpath, size_t fullpath_len, int *found)
+{
     char execpath[MAXPATHLEN + 1];
 #if MAC_OS_X_VERSION_MAX_ALLOWED >= MAC_OS_X_VERSION_10_4
     uint32_t nsexeclength = Py_ARRAY_LENGTH(execpath) - 1;
 #else
     unsigned long nsexeclength = Py_ARRAY_LENGTH(execpath) - 1;
 #endif
-#endif
+
+    /* On Mac OS X, if a script uses an interpreter of the form
+       "#!/opt/python2.3/bin/python", the kernel only passes "python"
+       as argv[0], which falls through to the $PATH search below.
+       If /opt/python2.3/bin isn't in your path, or is near the end,
+       this algorithm may incorrectly find /usr/bin/python. To work
+       around this, we can use _NSGetExecutablePath to get a better
+       hint of what the intended interpreter was, although this
+       will fail if a relative path was used. but in that case,
+       absolutize() should help us out below
+     */
+    if (_NSGetExecutablePath(execpath, &nsexeclength) != 0
+        || (wchar_t)execpath[0] != SEP)
+    {
+        /* _NSGetExecutablePath() failed or the path is relative */
+        return _PyStatus_OK();
+    }
+
+    size_t len;
+    wchar_t *path = Py_DecodeLocale(execpath, &len);
+    if (path == NULL) {
+        return DECODE_LOCALE_ERR("executable path", len);
+    }
+    if (safe_wcscpy(fullpath, path, fullpath_len) < 0) {
+        PyMem_RawFree(path);
+        return PATHLEN_ERR();
+    }
+    PyMem_RawFree(path);
+
+    *found = 1;
+    return _PyStatus_OK();
+}
+#endif  /* __APPLE__ */
+
+
+static PyStatus
+calculate_program_impl(PyCalculatePath *calculate, _PyPathConfig *pathconfig,
+                       wchar_t *fullpath, size_t fullpath_len)
+{
+    PyStatus status;
 
     /* If there is no slash in the argv0 path, then we have to
      * assume python is on the user's $PATH, since there's no
@@ -790,96 +867,82 @@ calculate_program_full_path(PyCalculatePath *calculate, _PyPathConfig *pathconfi
      * $PATH isn't exported, you lose.
      */
     if (wcschr(pathconfig->program_name, SEP)) {
-        if (safe_wcscpy(program_full_path, pathconfig->program_name,
-                        program_full_path_len) < 0) {
+        if (safe_wcscpy(fullpath, pathconfig->program_name,
+                        fullpath_len) < 0) {
             return PATHLEN_ERR();
         }
+        return _PyStatus_OK();
     }
+
 #ifdef __APPLE__
-     /* On Mac OS X, if a script uses an interpreter of the form
-      * "#!/opt/python2.3/bin/python", the kernel only passes "python"
-      * as argv[0], which falls through to the $PATH search below.
-      * If /opt/python2.3/bin isn't in your path, or is near the end,
-      * this algorithm may incorrectly find /usr/bin/python. To work
-      * around this, we can use _NSGetExecutablePath to get a better
-      * hint of what the intended interpreter was, although this
-      * will fail if a relative path was used. but in that case,
-      * absolutize() should help us out below
-      */
-    else if(0 == _NSGetExecutablePath(execpath, &nsexeclength) &&
-            (wchar_t)execpath[0] == SEP)
-    {
-        size_t len;
-        wchar_t *path = Py_DecodeLocale(execpath, &len);
-        if (path == NULL) {
-            return DECODE_LOCALE_ERR("executable path", len);
+    int found = 0;
+    status = calculate_program_macos(fullpath, fullpath_len, &found);
+    if (_PyStatus_EXCEPTION(status)) {
+        return status;
+    }
+    if (found) {
+        return _PyStatus_OK();
+    }
+#endif /* __APPLE__ */
+
+    if (calculate->path_env) {
+        int found = 0;
+        status = calculate_which(calculate->path_env, pathconfig->program_name,
+                                 fullpath, fullpath_len,
+                                 &found);
+        if (_PyStatus_EXCEPTION(status)) {
+            return status;
         }
-        if (safe_wcscpy(program_full_path, path, program_full_path_len) < 0) {
-            PyMem_RawFree(path);
-            return PATHLEN_ERR();
+        if (found) {
+            return _PyStatus_OK();
         }
-        PyMem_RawFree(path);
     }
-#endif /* __APPLE__ */
-    else if (calculate->path_env) {
-        wchar_t *path = calculate->path_env;
-        while (1) {
-            wchar_t *delim = wcschr(path, DELIM);
-
-            if (delim) {
-                size_t len = delim - path;
-                if (len >= program_full_path_len) {
-                    return PATHLEN_ERR();
-                }
-                wcsncpy(program_full_path, path, len);
-                program_full_path[len] = '\0';
-            }
-            else {
-                if (safe_wcscpy(program_full_path, path,
-                                program_full_path_len) < 0) {
-                    return PATHLEN_ERR();
-                }
-            }
 
-            status = joinpath(program_full_path, pathconfig->program_name,
-                              program_full_path_len);
-            if (_PyStatus_EXCEPTION(status)) {
-                return status;
-            }
+    /* In the last resort, use an empty string */
+    fullpath[0] = '\0';
+    return _PyStatus_OK();
+}
 
-            if (isxfile(program_full_path)) {
-                break;
-            }
 
-            if (!delim) {
-                program_full_path[0] = L'\0';
-                break;
-            }
-            path = delim + 1;
-        }
-    }
-    else {
-        program_full_path[0] = '\0';
+/* Calculate pathconfig->program_full_path */
+static PyStatus
+calculate_program(PyCalculatePath *calculate, _PyPathConfig *pathconfig)
+{
+    PyStatus status;
+    wchar_t program_full_path[MAXPATHLEN + 1];
+    const size_t program_full_path_len = Py_ARRAY_LENGTH(program_full_path);
+    memset(program_full_path, 0, sizeof(program_full_path));
+
+    status = calculate_program_impl(calculate, pathconfig,
+                                    program_full_path, program_full_path_len);
+    if (_PyStatus_EXCEPTION(status)) {
+        return status;
     }
-    if (!_Py_isabs(program_full_path) && program_full_path[0] != '\0') {
-        status = absolutize(program_full_path, program_full_path_len);
-        if (_PyStatus_EXCEPTION(status)) {
-            return status;
+
+    if (program_full_path[0] != '\0') {
+        /* program_full_path is not empty */
+
+        /* Make sure that program_full_path is an absolute path
+           (or an empty string) */
+        if (!_Py_isabs(program_full_path)) {
+            status = absolutize(program_full_path, program_full_path_len);
+            if (_PyStatus_EXCEPTION(status)) {
+                return status;
+            }
         }
-    }
+
 #if defined(__CYGWIN__) || defined(__MINGW32__)
-    /* For these platforms it is necessary to ensure that the .exe suffix
-     * is appended to the filename, otherwise there is potential for
-     * sys.executable to return the name of a directory under the same
-     * path (bpo-28441).
-     */
-    if (program_full_path[0] != '\0') {
+        /* For these platforms it is necessary to ensure that the .exe suffix
+         * is appended to the filename, otherwise there is potential for
+         * sys.executable to return the name of a directory under the same
+         * path (bpo-28441).
+         */
         status = add_exe_suffix(program_full_path, program_full_path_len);
         if (_PyStatus_EXCEPTION(status)) {
             return status;
         }
-    }
 #endif
+    }
 
     pathconfig->program_full_path = _PyMem_RawWcsdup(program_full_path);
     if (pathconfig->program_full_path == NULL) {
@@ -889,97 +952,140 @@ calculate_program_full_path(PyCalculatePath *calculate, _PyPathConfig *pathconfi
 }
 
 
+#if HAVE_READLINK
 static PyStatus
-calculate_argv0_path(PyCalculatePath *calculate, const wchar_t *program_full_path,
-                     wchar_t *argv0_path, size_t argv0_path_len)
+resolve_symlinks(wchar_t *path, size_t path_len)
 {
-    if (safe_wcscpy(argv0_path, program_full_path, argv0_path_len) < 0) {
-        return PATHLEN_ERR();
+    wchar_t new_path[MAXPATHLEN + 1];
+    const size_t new_path_len = Py_ARRAY_LENGTH(new_path);
+    unsigned int links = 0;
+
+    while (1) {
+        int linklen = _Py_wreadlink(path, new_path, new_path_len);
+        if (linklen == -1) {
+            break;
+        }
+
+        if (_Py_isabs(new_path)) {
+            /* new_path should never be longer than MAXPATHLEN,
+               but extra check does not hurt */
+            if (safe_wcscpy(path, new_path, path_len) < 0) {
+                return PATHLEN_ERR();
+            }
+        }
+        else {
+            /* new_path is relative to path */
+            reduce(path);
+            PyStatus status = joinpath(path, new_path, path_len);
+            if (_PyStatus_EXCEPTION(status)) {
+                return status;
+            }
+        }
+
+        links++;
+        /* 40 is the Linux kernel 4.2 limit */
+        if (links >= 40) {
+            return _PyStatus_ERR("maximum number of symbolic links reached");
+        }
     }
+    return _PyStatus_OK();
+}
+#endif /* HAVE_READLINK */
+
 
 #ifdef WITH_NEXT_FRAMEWORK
+static PyStatus
+calculate_argv0_path_framework(PyCalculatePath *calculate,
+                               const wchar_t *program_full_path,
+                               wchar_t *argv0_path, size_t argv0_path_len)
+{
     NSModule pythonModule;
 
     /* On Mac OS X we have a special case if we're running from a framework.
-    ** This is because the python home should be set relative to the library,
-    ** which is in the framework, not relative to the executable, which may
-    ** be outside of the framework. Except when we're in the build directory...
-    */
+       This is because the python home should be set relative to the library,
+       which is in the framework, not relative to the executable, which may
+       be outside of the framework. Except when we're in the build
+       directory... */
     pythonModule = NSModuleForSymbol(NSLookupAndBindSymbol("_Py_Initialize"));
+
     /* Use dylib functions to find out where the framework was loaded from */
     const char* modPath = NSLibraryNameForModule(pythonModule);
-    if (modPath != NULL) {
-        /* We're in a framework. */
-        /* See if we might be in the build directory. The framework in the
-        ** build directory is incomplete, it only has the .dylib and a few
-        ** needed symlinks, it doesn't have the Lib directories and such.
-        ** If we're running with the framework from the build directory we must
-        ** be running the interpreter in the build directory, so we use the
-        ** build-directory-specific logic to find Lib and such.
-        */
-        PyStatus status;
-        size_t len;
-        wchar_t* wbuf = Py_DecodeLocale(modPath, &len);
-        if (wbuf == NULL) {
-            return DECODE_LOCALE_ERR("framework location", len);
-        }
+    if (modPath == NULL) {
+        return _PyStatus_OK();
+    }
+
+    /* We're in a framework.
+       See if we might be in the build directory. The framework in the
+       build directory is incomplete, it only has the .dylib and a few
+       needed symlinks, it doesn't have the Lib directories and such.
+       If we're running with the framework from the build directory we must
+       be running the interpreter in the build directory, so we use the
+       build-directory-specific logic to find Lib and such. */
+    size_t len;
+    wchar_t* wbuf = Py_DecodeLocale(modPath, &len);
+    if (wbuf == NULL) {
+        return DECODE_LOCALE_ERR("framework location", len);
+    }
+
+    /* Path: reduce(modPath) / lib_python / LANDMARK */
+    PyStatus status;
+    if (safe_wcscpy(argv0_path, wbuf, argv0_path_len) < 0) {
+        status = PATHLEN_ERR();
+        goto done;
+    }
+    reduce(argv0_path);
+    status = joinpath(argv0_path, calculate->lib_python, argv0_path_len);
+    if (_PyStatus_EXCEPTION(status)) {
+        goto done;
+    }
+    status = joinpath(argv0_path, LANDMARK, argv0_path_len);
+    if (_PyStatus_EXCEPTION(status)) {
+        goto done;
+    }
 
+    if (ismodule(argv0_path, Py_ARRAY_LENGTH(argv0_path))) {
+        /* Use the location of the library as argv0_path */
         if (safe_wcscpy(argv0_path, wbuf, argv0_path_len) < 0) {
-            return PATHLEN_ERR();
-        }
-        reduce(argv0_path);
-        status = joinpath(argv0_path, calculate->lib_python, argv0_path_len);
-        if (_PyStatus_EXCEPTION(status)) {
-            PyMem_RawFree(wbuf);
-            return status;
-        }
-        status = joinpath(argv0_path, LANDMARK, argv0_path_len);
-        if (_PyStatus_EXCEPTION(status)) {
-            PyMem_RawFree(wbuf);
-            return status;
-        }
-        if (!ismodule(argv0_path, Py_ARRAY_LENGTH(argv0_path))) {
-            /* We are in the build directory so use the name of the
-               executable - we know that the absolute path is passed */
-            if (safe_wcscpy(argv0_path, program_full_path,
-                            argv0_path_len) < 0) {
-                return PATHLEN_ERR();
-            }
+            status = PATHLEN_ERR();
+            goto done;
         }
-        else {
-            /* Use the location of the library as the program_full_path */
-            if (safe_wcscpy(argv0_path, wbuf, argv0_path_len) < 0) {
-                return PATHLEN_ERR();
-            }
+    }
+    else {
+        /* We are in the build directory so use the name of the
+           executable - we know that the absolute path is passed */
+        if (safe_wcscpy(argv0_path, program_full_path, argv0_path_len) < 0) {
+            status = PATHLEN_ERR();
+            goto done;
         }
-        PyMem_RawFree(wbuf);
     }
+    status = _PyStatus_OK();
+
+done:
+    PyMem_RawFree(wbuf);
+    return status;
+}
 #endif
 
-#if HAVE_READLINK
-    wchar_t tmpbuffer[MAXPATHLEN + 1];
-    const size_t buflen = Py_ARRAY_LENGTH(tmpbuffer);
-    int linklen = _Py_wreadlink(argv0_path, tmpbuffer, buflen);
-    while (linklen != -1) {
-        if (_Py_isabs(tmpbuffer)) {
-            /* tmpbuffer should never be longer than MAXPATHLEN,
-               but extra check does not hurt */
-            if (safe_wcscpy(argv0_path, tmpbuffer, argv0_path_len) < 0) {
-                return PATHLEN_ERR();
-            }
-        }
-        else {
-            /* Interpret relative to program_full_path */
-            PyStatus status;
-            reduce(argv0_path);
-            status = joinpath(argv0_path, tmpbuffer, argv0_path_len);
-            if (_PyStatus_EXCEPTION(status)) {
-                return status;
-            }
-        }
-        linklen = _Py_wreadlink(argv0_path, tmpbuffer, buflen);
+
+static PyStatus
+calculate_argv0_path(PyCalculatePath *calculate,
+                     const wchar_t *program_full_path,
+                     wchar_t *argv0_path, size_t argv0_path_len)
+{
+    if (safe_wcscpy(argv0_path, program_full_path, argv0_path_len) < 0) {
+        return PATHLEN_ERR();
     }
-#endif /* HAVE_READLINK */
+
+#ifdef WITH_NEXT_FRAMEWORK
+    PyStatus status;
+    status = calculate_argv0_path_framework(calculate, program_full_path,
+                                            argv0_path, argv0_path_len);
+    if (_PyStatus_EXCEPTION(status)) {
+        return status;
+    }
+#endif
+
+    resolve_symlinks(argv0_path, argv0_path_len);
 
     reduce(argv0_path);
     /* At this point, argv0_path is guaranteed to be less than
@@ -1226,7 +1332,7 @@ calculate_path(PyCalculatePath *calculate, _PyPathConfig *pathconfig)
     PyStatus status;
 
     if (pathconfig->program_full_path == NULL) {
-        status = calculate_program_full_path(calculate, pathconfig);
+        status = calculate_program(calculate, pathconfig);
         if (_PyStatus_EXCEPTION(status)) {
             return status;
         }
@@ -1275,8 +1381,8 @@ calculate_path(PyCalculatePath *calculate, _PyPathConfig *pathconfig)
         return status;
     }
 
-    if ((!calculate->prefix_found || !calculate->exec_prefix_found) &&
-        calculate->warnings)
+    if ((!calculate->prefix_found || !calculate->exec_prefix_found)
+        && calculate->warnings)
     {
         fprintf(stderr,
                 "Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]\n");
diff --git a/Python/fileutils.c b/Python/fileutils.c
index 0c05424143afb..6345553f484c7 100644
--- a/Python/fileutils.c
+++ b/Python/fileutils.c
@@ -1668,8 +1668,9 @@ _Py_wreadlink(const wchar_t *path, wchar_t *buf, size_t buflen)
 {
     char *cpath;
     char cbuf[MAXPATHLEN];
+    size_t cbuf_len = Py_ARRAY_LENGTH(cbuf);
     wchar_t *wbuf;
-    int res;
+    Py_ssize_t res;
     size_t r1;
 
     cpath = _Py_EncodeLocaleRaw(path, NULL);
@@ -1677,11 +1678,12 @@ _Py_wreadlink(const wchar_t *path, wchar_t *buf, size_t buflen)
         errno = EINVAL;
         return -1;
     }
-    res = (int)readlink(cpath, cbuf, Py_ARRAY_LENGTH(cbuf));
+    res = readlink(cpath, cbuf, cbuf_len);
     PyMem_RawFree(cpath);
-    if (res == -1)
+    if (res == -1) {
         return -1;
-    if (res == Py_ARRAY_LENGTH(cbuf)) {
+    }
+    if ((size_t)res == cbuf_len) {
         errno = EINVAL;
         return -1;
     }


From webhook-mailer at python.org  Thu Oct  3 21:53:33 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Fri, 04 Oct 2019 01:53:33 -0000
Subject: [Python-checkins] bpo-38353: Rework ismodule() in getpath.c
 (GH-16574)
Message-ID: <mailman.208.1570154013.32504.python-checkins@python.org>

https://github.com/python/cpython/commit/f0c85793d2a087974969f7eea6ca24bbd1178d53
commit: f0c85793d2a087974969f7eea6ca24bbd1178d53
branch: master
author: Victor Stinner <vstinner at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-04T03:53:26+02:00
summary:

bpo-38353: Rework ismodule() in getpath.c (GH-16574)

* ismodule() no longer modify the input string.
* copy_absolute(), joinpath(): rename parameters and local variables.

files:
M Modules/getpath.c

diff --git a/Modules/getpath.c b/Modules/getpath.c
index 7a66eee881361..1349d5c3f251b 100644
--- a/Modules/getpath.c
+++ b/Modules/getpath.c
@@ -183,25 +183,6 @@ isfile(const wchar_t *filename)
 }
 
 
-/* Is module -- check for .pyc too */
-static int
-ismodule(wchar_t *filename, size_t filename_len)
-{
-    if (isfile(filename)) {
-        return 1;
-    }
-
-    /* Check for the compiled version of prefix. */
-    if (wcslen(filename) + 2 <= filename_len) {
-        wcscat(filename, L"c");
-        if (isfile(filename)) {
-            return 1;
-        }
-    }
-    return 0;
-}
-
-
 /* Is executable file */
 static int
 isxfile(const wchar_t *filename)
@@ -222,7 +203,7 @@ isxfile(const wchar_t *filename)
 
 /* Is directory */
 static int
-isdir(wchar_t *filename)
+isdir(const wchar_t *filename)
 {
     struct stat buf;
     if (_Py_wstat(filename, &buf) != 0) {
@@ -238,29 +219,29 @@ isdir(wchar_t *filename)
 /* Add a path component, by appending stuff to buffer.
    buflen: 'buffer' length in characters including trailing NUL. */
 static PyStatus
-joinpath(wchar_t *buffer, const wchar_t *stuff, size_t buflen)
+joinpath(wchar_t *path, const wchar_t *path2, size_t path_len)
 {
-    size_t n, k;
-    if (!_Py_isabs(stuff)) {
-        n = wcslen(buffer);
-        if (n >= buflen) {
+    size_t n;
+    if (!_Py_isabs(path2)) {
+        n = wcslen(path);
+        if (n >= path_len) {
             return PATHLEN_ERR();
         }
 
-        if (n > 0 && buffer[n-1] != SEP) {
-            buffer[n++] = SEP;
+        if (n > 0 && path[n-1] != SEP) {
+            path[n++] = SEP;
         }
     }
     else {
         n = 0;
     }
 
-    k = wcslen(stuff);
-    if (n + k >= buflen) {
+    size_t k = wcslen(path2);
+    if (n + k >= path_len) {
         return PATHLEN_ERR();
     }
-    wcsncpy(buffer+n, stuff, k);
-    buffer[n+k] = '\0';
+    wcsncpy(path + n, path2, k);
+    path[n + k] = '\0';
 
     return _PyStatus_OK();
 }
@@ -280,27 +261,27 @@ safe_wcscpy(wchar_t *dst, const wchar_t *src, size_t n)
 
 
 /* copy_absolute requires that path be allocated at least
-   'pathlen' characters (including trailing NUL). */
+   'abs_path_len' characters (including trailing NUL). */
 static PyStatus
-copy_absolute(wchar_t *path, const wchar_t *p, size_t pathlen)
+copy_absolute(wchar_t *abs_path, const wchar_t *path, size_t abs_path_len)
 {
-    if (_Py_isabs(p)) {
-        if (safe_wcscpy(path, p, pathlen) < 0) {
+    if (_Py_isabs(path)) {
+        if (safe_wcscpy(abs_path, path, abs_path_len) < 0) {
             return PATHLEN_ERR();
         }
     }
     else {
-        if (!_Py_wgetcwd(path, pathlen)) {
+        if (!_Py_wgetcwd(abs_path, abs_path_len)) {
             /* unable to get the current directory */
-            if (safe_wcscpy(path, p, pathlen) < 0) {
+            if (safe_wcscpy(abs_path, path, abs_path_len) < 0) {
                 return PATHLEN_ERR();
             }
             return _PyStatus_OK();
         }
-        if (p[0] == '.' && p[1] == SEP) {
-            p += 2;
+        if (path[0] == '.' && path[1] == SEP) {
+            path += 2;
         }
-        PyStatus status = joinpath(path, p, pathlen);
+        PyStatus status = joinpath(abs_path, path, abs_path_len);
         if (_PyStatus_EXCEPTION(status)) {
             return status;
         }
@@ -330,6 +311,41 @@ absolutize(wchar_t *path, size_t path_len)
 }
 
 
+/* Is module -- check for .pyc too */
+static PyStatus
+ismodule(const wchar_t *path, int *result)
+{
+    wchar_t filename[MAXPATHLEN+1];
+    size_t filename_len = Py_ARRAY_LENGTH(filename);
+
+    if (safe_wcscpy(filename, path, filename_len) < 0) {
+        return PATHLEN_ERR();
+    }
+
+    PyStatus status = joinpath(filename, LANDMARK, filename_len);
+    if (_PyStatus_EXCEPTION(status)) {
+        return status;
+    }
+
+    if (isfile(filename)) {
+        *result = 1;
+        return _PyStatus_OK();
+    }
+
+    /* Check for the compiled version of prefix. */
+    if (wcslen(filename) + 2 <= filename_len) {
+        wcscat(filename, L"c");
+        if (isfile(filename)) {
+            *result = 1;
+            return _PyStatus_OK();
+        }
+    }
+
+    *result = 0;
+    return _PyStatus_OK();
+}
+
+
 #if defined(__CYGWIN__) || defined(__MINGW32__)
 #ifndef EXE_SUFFIX
 #define EXE_SUFFIX L".exe"
@@ -337,26 +353,25 @@ absolutize(wchar_t *path, size_t path_len)
 
 /* pathlen: 'path' length in characters including trailing NUL */
 static PyStatus
-add_exe_suffix(wchar_t *progpath, size_t progpathlen)
+add_exe_suffix(wchar_t *path, size_t pathlen)
 {
     /* Check for already have an executable suffix */
-    size_t n = wcslen(progpath);
+    size_t n = wcslen(path);
     size_t s = wcslen(EXE_SUFFIX);
-    if (wcsncasecmp(EXE_SUFFIX, progpath + n - s, s) == 0) {
+    if (wcsncasecmp(EXE_SUFFIX, path + n - s, s) == 0) {
         return _PyStatus_OK();
     }
 
-    if (n + s >= progpathlen) {
+    if (n + s >= pathlen) {
         return PATHLEN_ERR();
     }
-    wcsncpy(progpath + n, EXE_SUFFIX, s);
-    progpath[n+s] = '\0';
+    wcsncpy(path + n, EXE_SUFFIX, s);
+    path[n + s] = '\0';
 
-    if (!isxfile(progpath)) {
+    if (!isxfile(path)) {
         /* Path that added suffix is invalid: truncate (remove suffix) */
-        progpath[n] = '\0';
+        path[n] = '\0';
     }
-
     return _PyStatus_OK();
 }
 #endif
@@ -422,14 +437,14 @@ search_for_prefix(PyCalculatePath *calculate, _PyPathConfig *pathconfig,
             if (_PyStatus_EXCEPTION(status)) {
                 return status;
             }
-            status = joinpath(prefix, LANDMARK, prefix_len);
+
+            int module;
+            status = ismodule(prefix, &module);
             if (_PyStatus_EXCEPTION(status)) {
                 return status;
             }
-
-            if (ismodule(prefix, prefix_len)) {
+            if (module) {
                 *found = -1;
-                reduce(prefix);
                 return _PyStatus_OK();
             }
         }
@@ -448,14 +463,14 @@ search_for_prefix(PyCalculatePath *calculate, _PyPathConfig *pathconfig,
         if (_PyStatus_EXCEPTION(status)) {
             return status;
         }
-        status = joinpath(prefix, LANDMARK, prefix_len);
+
+        int module;
+        status = ismodule(prefix, &module);
         if (_PyStatus_EXCEPTION(status)) {
             return status;
         }
-
-        if (ismodule(prefix, prefix_len)) {
+        if (module) {
             *found = 1;
-            reduce(prefix);
             return _PyStatus_OK();
         }
         prefix[n] = L'\0';
@@ -471,14 +486,14 @@ search_for_prefix(PyCalculatePath *calculate, _PyPathConfig *pathconfig,
     if (_PyStatus_EXCEPTION(status)) {
         return status;
     }
-    status = joinpath(prefix, LANDMARK, prefix_len);
+
+    int module;
+    status = ismodule(prefix, &module);
     if (_PyStatus_EXCEPTION(status)) {
         return status;
     }
-
-    if (ismodule(prefix, prefix_len)) {
+    if (module) {
         *found = 1;
-        reduce(prefix);
         return _PyStatus_OK();
     }
 
@@ -1038,12 +1053,13 @@ calculate_argv0_path_framework(PyCalculatePath *calculate,
     if (_PyStatus_EXCEPTION(status)) {
         goto done;
     }
-    status = joinpath(argv0_path, LANDMARK, argv0_path_len);
+
+    int module;
+    status = ismodule(argv0_path, &module);
     if (_PyStatus_EXCEPTION(status)) {
         goto done;
     }
-
-    if (ismodule(argv0_path, Py_ARRAY_LENGTH(argv0_path))) {
+    if (module) {
         /* Use the location of the library as argv0_path */
         if (safe_wcscpy(argv0_path, wbuf, argv0_path_len) < 0) {
             status = PATHLEN_ERR();


From webhook-mailer at python.org  Fri Oct  4 03:55:20 2019
From: webhook-mailer at python.org (Vinay Sajip)
Date: Fri, 04 Oct 2019 07:55:20 -0000
Subject: [Python-checkins] bpo-38235: Correct some arguments names in
 logging documentation (GH-16571)
Message-ID: <mailman.209.1570175721.32504.python-checkins@python.org>

https://github.com/python/cpython/commit/3142c667b50254daaa28f22c79bdda177136bd03
commit: 3142c667b50254daaa28f22c79bdda177136bd03
branch: master
author: Ashley Whetter <AWhetter at users.noreply.github.com>
committer: Vinay Sajip <vinay_sajip at yahoo.co.uk>
date: 2019-10-04T08:55:14+01:00
summary:

bpo-38235: Correct some arguments names in logging documentation (GH-16571)

files:
M Doc/library/logging.rst

diff --git a/Doc/library/logging.rst b/Doc/library/logging.rst
index dec27bff564e6..3ed24605db6fd 100644
--- a/Doc/library/logging.rst
+++ b/Doc/library/logging.rst
@@ -124,11 +124,11 @@ is the module's name in the Python package namespace.
          :meth:`isEnabledFor` will return/expect to be passed integers.
 
 
-   .. method:: Logger.isEnabledFor(lvl)
+   .. method:: Logger.isEnabledFor(level)
 
-      Indicates if a message of severity *lvl* would be processed by this logger.
+      Indicates if a message of severity *level* would be processed by this logger.
       This method checks first the module-level level set by
-      ``logging.disable(lvl)`` and then the logger's effective level as determined
+      ``logging.disable(level)`` and then the logger's effective level as determined
       by :meth:`getEffectiveLevel`.
 
 
@@ -269,9 +269,9 @@ is the module's name in the Python package namespace.
       interpreted as for :meth:`debug`.
 
 
-   .. method:: Logger.log(lvl, msg, *args, **kwargs)
+   .. method:: Logger.log(level, msg, *args, **kwargs)
 
-      Logs a message with integer level *lvl* on this logger. The other arguments are
+      Logs a message with integer level *level* on this logger. The other arguments are
       interpreted as for :meth:`debug`.
 
 
@@ -333,7 +333,7 @@ is the module's name in the Python package namespace.
       Logger-level filtering is applied using :meth:`~Logger.filter`.
 
 
-   .. method:: Logger.makeRecord(name, lvl, fn, lno, msg, args, exc_info, func=None, extra=None, sinfo=None)
+   .. method:: Logger.makeRecord(name, level, fn, lno, msg, args, exc_info, func=None, extra=None, sinfo=None)
 
       This is a factory method which can be overridden in subclasses to create
       specialized :class:`LogRecord` instances.
@@ -1066,12 +1066,12 @@ functions.
       handlers being added multiple times to the root logger, which can in turn
       lead to multiple messages for the same event.
 
-.. function:: disable(lvl=CRITICAL)
+.. function:: disable(level=CRITICAL)
 
-   Provides an overriding level *lvl* for all loggers which takes precedence over
+   Provides an overriding level *level* for all loggers which takes precedence over
    the logger's own level. When the need arises to temporarily throttle logging
    output down across the whole application, this function can be useful. Its
-   effect is to disable all logging calls of severity *lvl* and below, so that
+   effect is to disable all logging calls of severity *level* and below, so that
    if you call it with a value of INFO, then all INFO and DEBUG events would be
    discarded, whereas those of severity WARNING and above would be processed
    according to the logger's effective level. If
@@ -1081,16 +1081,16 @@ functions.
 
    Note that if you have defined any custom logging level higher than
    ``CRITICAL`` (this is not recommended), you won't be able to rely on the
-   default value for the *lvl* parameter, but will have to explicitly supply a
+   default value for the *level* parameter, but will have to explicitly supply a
    suitable value.
 
    .. versionchanged:: 3.7
-      The *lvl* parameter was defaulted to level ``CRITICAL``. See Issue
+      The *level* parameter was defaulted to level ``CRITICAL``. See Issue
       #28524 for more information about this change.
 
-.. function:: addLevelName(lvl, levelName)
+.. function:: addLevelName(level, levelName)
 
-   Associates level *lvl* with text *levelName* in an internal dictionary, which is
+   Associates level *level* with text *levelName* in an internal dictionary, which is
    used to map numeric levels to a textual representation, for example when a
    :class:`Formatter` formats a message. This function can also be used to define
    your own levels. The only constraints are that all levels used must be
@@ -1100,15 +1100,15 @@ functions.
    .. note:: If you are thinking of defining your own levels, please see the
       section on :ref:`custom-levels`.
 
-.. function:: getLevelName(lvl)
+.. function:: getLevelName(level)
 
-   Returns the textual representation of logging level *lvl*. If the level is one
+   Returns the textual representation of logging level *level*. If the level is one
    of the predefined levels :const:`CRITICAL`, :const:`ERROR`, :const:`WARNING`,
    :const:`INFO` or :const:`DEBUG` then you get the corresponding string. If you
    have associated levels with names using :func:`addLevelName` then the name you
-   have associated with *lvl* is returned. If a numeric value corresponding to one
+   have associated with *level* is returned. If a numeric value corresponding to one
    of the defined levels is passed in, the corresponding string representation is
-   returned. Otherwise, the string 'Level %s' % lvl is returned.
+   returned. Otherwise, the string 'Level %s' % level is returned.
 
    .. note:: Levels are internally integers (as they need to be compared in the
       logging logic). This function is used to convert between an integer level


From webhook-mailer at python.org  Fri Oct  4 04:14:05 2019
From: webhook-mailer at python.org (Vinay Sajip)
Date: Fri, 04 Oct 2019 08:14:05 -0000
Subject: [Python-checkins] bpo-38235: Correct some arguments names in
 logging documentation (GH-16571) (GH-16576)
Message-ID: <mailman.210.1570176846.32504.python-checkins@python.org>

https://github.com/python/cpython/commit/3af2733a8265a2685b2c0466a58a66e544a81c64
commit: 3af2733a8265a2685b2c0466a58a66e544a81c64
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Vinay Sajip <vinay_sajip at yahoo.co.uk>
date: 2019-10-04T09:14:01+01:00
summary:

bpo-38235: Correct some arguments names in logging documentation (GH-16571) (GH-16576)

(cherry picked from commit 3142c667b50254daaa28f22c79bdda177136bd03)

Co-authored-by: Ashley Whetter <AWhetter at users.noreply.github.com>

files:
M Doc/library/logging.rst

diff --git a/Doc/library/logging.rst b/Doc/library/logging.rst
index 8ea00a92fb734..0fe2158ef06fa 100644
--- a/Doc/library/logging.rst
+++ b/Doc/library/logging.rst
@@ -124,11 +124,11 @@ is the module's name in the Python package namespace.
          :meth:`isEnabledFor` will return/expect to be passed integers.
 
 
-   .. method:: Logger.isEnabledFor(lvl)
+   .. method:: Logger.isEnabledFor(level)
 
-      Indicates if a message of severity *lvl* would be processed by this logger.
+      Indicates if a message of severity *level* would be processed by this logger.
       This method checks first the module-level level set by
-      ``logging.disable(lvl)`` and then the logger's effective level as determined
+      ``logging.disable(level)`` and then the logger's effective level as determined
       by :meth:`getEffectiveLevel`.
 
 
@@ -269,9 +269,9 @@ is the module's name in the Python package namespace.
       interpreted as for :meth:`debug`.
 
 
-   .. method:: Logger.log(lvl, msg, *args, **kwargs)
+   .. method:: Logger.log(level, msg, *args, **kwargs)
 
-      Logs a message with integer level *lvl* on this logger. The other arguments are
+      Logs a message with integer level *level* on this logger. The other arguments are
       interpreted as for :meth:`debug`.
 
 
@@ -333,7 +333,7 @@ is the module's name in the Python package namespace.
       Logger-level filtering is applied using :meth:`~Logger.filter`.
 
 
-   .. method:: Logger.makeRecord(name, lvl, fn, lno, msg, args, exc_info, func=None, extra=None, sinfo=None)
+   .. method:: Logger.makeRecord(name, level, fn, lno, msg, args, exc_info, func=None, extra=None, sinfo=None)
 
       This is a factory method which can be overridden in subclasses to create
       specialized :class:`LogRecord` instances.
@@ -1063,12 +1063,12 @@ functions.
       handlers being added multiple times to the root logger, which can in turn
       lead to multiple messages for the same event.
 
-.. function:: disable(lvl=CRITICAL)
+.. function:: disable(level=CRITICAL)
 
-   Provides an overriding level *lvl* for all loggers which takes precedence over
+   Provides an overriding level *level* for all loggers which takes precedence over
    the logger's own level. When the need arises to temporarily throttle logging
    output down across the whole application, this function can be useful. Its
-   effect is to disable all logging calls of severity *lvl* and below, so that
+   effect is to disable all logging calls of severity *level* and below, so that
    if you call it with a value of INFO, then all INFO and DEBUG events would be
    discarded, whereas those of severity WARNING and above would be processed
    according to the logger's effective level. If
@@ -1078,16 +1078,16 @@ functions.
 
    Note that if you have defined any custom logging level higher than
    ``CRITICAL`` (this is not recommended), you won't be able to rely on the
-   default value for the *lvl* parameter, but will have to explicitly supply a
+   default value for the *level* parameter, but will have to explicitly supply a
    suitable value.
 
    .. versionchanged:: 3.7
-      The *lvl* parameter was defaulted to level ``CRITICAL``. See Issue
+      The *level* parameter was defaulted to level ``CRITICAL``. See Issue
       #28524 for more information about this change.
 
-.. function:: addLevelName(lvl, levelName)
+.. function:: addLevelName(level, levelName)
 
-   Associates level *lvl* with text *levelName* in an internal dictionary, which is
+   Associates level *level* with text *levelName* in an internal dictionary, which is
    used to map numeric levels to a textual representation, for example when a
    :class:`Formatter` formats a message. This function can also be used to define
    your own levels. The only constraints are that all levels used must be
@@ -1097,15 +1097,15 @@ functions.
    .. note:: If you are thinking of defining your own levels, please see the
       section on :ref:`custom-levels`.
 
-.. function:: getLevelName(lvl)
+.. function:: getLevelName(level)
 
-   Returns the textual representation of logging level *lvl*. If the level is one
+   Returns the textual representation of logging level *level*. If the level is one
    of the predefined levels :const:`CRITICAL`, :const:`ERROR`, :const:`WARNING`,
    :const:`INFO` or :const:`DEBUG` then you get the corresponding string. If you
    have associated levels with names using :func:`addLevelName` then the name you
-   have associated with *lvl* is returned. If a numeric value corresponding to one
+   have associated with *level* is returned. If a numeric value corresponding to one
    of the defined levels is passed in, the corresponding string representation is
-   returned. Otherwise, the string 'Level %s' % lvl is returned.
+   returned. Otherwise, the string 'Level %s' % level is returned.
 
    .. note:: Levels are internally integers (as they need to be compared in the
       logging logic). This function is used to convert between an integer level


From webhook-mailer at python.org  Fri Oct  4 04:14:32 2019
From: webhook-mailer at python.org (Vinay Sajip)
Date: Fri, 04 Oct 2019 08:14:32 -0000
Subject: [Python-checkins] bpo-38235: Correct some arguments names in
 logging documentation (GH-16571) (GH-16577)
Message-ID: <mailman.211.1570176873.32504.python-checkins@python.org>

https://github.com/python/cpython/commit/4f82a53c5d34df00bf2d563c2417f5e2638d1004
commit: 4f82a53c5d34df00bf2d563c2417f5e2638d1004
branch: 3.7
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Vinay Sajip <vinay_sajip at yahoo.co.uk>
date: 2019-10-04T09:14:28+01:00
summary:

bpo-38235: Correct some arguments names in logging documentation (GH-16571) (GH-16577)

(cherry picked from commit 3142c667b50254daaa28f22c79bdda177136bd03)

Co-authored-by: Ashley Whetter <AWhetter at users.noreply.github.com>

files:
M Doc/library/logging.rst

diff --git a/Doc/library/logging.rst b/Doc/library/logging.rst
index 73ce9ad1deecc..cec2f142dfc11 100644
--- a/Doc/library/logging.rst
+++ b/Doc/library/logging.rst
@@ -124,11 +124,11 @@ is the module's name in the Python package namespace.
          :meth:`isEnabledFor` will return/expect to be passed integers.
 
 
-   .. method:: Logger.isEnabledFor(lvl)
+   .. method:: Logger.isEnabledFor(level)
 
-      Indicates if a message of severity *lvl* would be processed by this logger.
+      Indicates if a message of severity *level* would be processed by this logger.
       This method checks first the module-level level set by
-      ``logging.disable(lvl)`` and then the logger's effective level as determined
+      ``logging.disable(level)`` and then the logger's effective level as determined
       by :meth:`getEffectiveLevel`.
 
 
@@ -258,9 +258,9 @@ is the module's name in the Python package namespace.
       interpreted as for :meth:`debug`.
 
 
-   .. method:: Logger.log(lvl, msg, *args, **kwargs)
+   .. method:: Logger.log(level, msg, *args, **kwargs)
 
-      Logs a message with integer level *lvl* on this logger. The other arguments are
+      Logs a message with integer level *level* on this logger. The other arguments are
       interpreted as for :meth:`debug`.
 
 
@@ -315,7 +315,7 @@ is the module's name in the Python package namespace.
       Logger-level filtering is applied using :meth:`~Logger.filter`.
 
 
-   .. method:: Logger.makeRecord(name, lvl, fn, lno, msg, args, exc_info, func=None, extra=None, sinfo=None)
+   .. method:: Logger.makeRecord(name, level, fn, lno, msg, args, exc_info, func=None, extra=None, sinfo=None)
 
       This is a factory method which can be overridden in subclasses to create
       specialized :class:`LogRecord` instances.
@@ -1041,12 +1041,12 @@ functions.
       handlers being added multiple times to the root logger, which can in turn
       lead to multiple messages for the same event.
 
-.. function:: disable(lvl=CRITICAL)
+.. function:: disable(level=CRITICAL)
 
-   Provides an overriding level *lvl* for all loggers which takes precedence over
+   Provides an overriding level *level* for all loggers which takes precedence over
    the logger's own level. When the need arises to temporarily throttle logging
    output down across the whole application, this function can be useful. Its
-   effect is to disable all logging calls of severity *lvl* and below, so that
+   effect is to disable all logging calls of severity *level* and below, so that
    if you call it with a value of INFO, then all INFO and DEBUG events would be
    discarded, whereas those of severity WARNING and above would be processed
    according to the logger's effective level. If
@@ -1056,16 +1056,16 @@ functions.
 
    Note that if you have defined any custom logging level higher than
    ``CRITICAL`` (this is not recommended), you won't be able to rely on the
-   default value for the *lvl* parameter, but will have to explicitly supply a
+   default value for the *level* parameter, but will have to explicitly supply a
    suitable value.
 
    .. versionchanged:: 3.7
-      The *lvl* parameter was defaulted to level ``CRITICAL``. See Issue
+      The *level* parameter was defaulted to level ``CRITICAL``. See Issue
       #28524 for more information about this change.
 
-.. function:: addLevelName(lvl, levelName)
+.. function:: addLevelName(level, levelName)
 
-   Associates level *lvl* with text *levelName* in an internal dictionary, which is
+   Associates level *level* with text *levelName* in an internal dictionary, which is
    used to map numeric levels to a textual representation, for example when a
    :class:`Formatter` formats a message. This function can also be used to define
    your own levels. The only constraints are that all levels used must be
@@ -1075,15 +1075,15 @@ functions.
    .. note:: If you are thinking of defining your own levels, please see the
       section on :ref:`custom-levels`.
 
-.. function:: getLevelName(lvl)
+.. function:: getLevelName(level)
 
-   Returns the textual representation of logging level *lvl*. If the level is one
+   Returns the textual representation of logging level *level*. If the level is one
    of the predefined levels :const:`CRITICAL`, :const:`ERROR`, :const:`WARNING`,
    :const:`INFO` or :const:`DEBUG` then you get the corresponding string. If you
    have associated levels with names using :func:`addLevelName` then the name you
-   have associated with *lvl* is returned. If a numeric value corresponding to one
+   have associated with *level* is returned. If a numeric value corresponding to one
    of the defined levels is passed in, the corresponding string representation is
-   returned. Otherwise, the string 'Level %s' % lvl is returned.
+   returned. Otherwise, the string 'Level %s' % level is returned.
 
    .. note:: Levels are internally integers (as they need to be compared in the
       logging logic). This function is used to convert between an integer level


From webhook-mailer at python.org  Fri Oct  4 04:48:42 2019
From: webhook-mailer at python.org (Vinay Sajip)
Date: Fri, 04 Oct 2019 08:48:42 -0000
Subject: [Python-checkins] Updated CODEOWNERS to indicate ownership of some
 modules. (GH-16578)
Message-ID: <mailman.212.1570178924.32504.python-checkins@python.org>

https://github.com/python/cpython/commit/2290b23dfc9cce71ffc49bfcb6a5a16c570cefae
commit: 2290b23dfc9cce71ffc49bfcb6a5a16c570cefae
branch: master
author: Vinay Sajip <vinay_sajip at yahoo.co.uk>
committer: GitHub <noreply at github.com>
date: 2019-10-04T09:48:24+01:00
summary:

Updated CODEOWNERS to indicate ownership of some modules. (GH-16578)

files:
M .github/CODEOWNERS

diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS
index 440a2b93f9126..af5deea722625 100644
--- a/.github/CODEOWNERS
+++ b/.github/CODEOWNERS
@@ -22,6 +22,14 @@ Objects/dict*                 @methane
 /Modules/_blake2/**           @python/crypto-team @tiran
 /Modules/_sha3/**             @python/crypto-team @tiran
 
+# logging
+**/*logging*                  @vsajip
+
+# venv
+**/*venv*                     @vsajip
+
+# Launcher
+/PC/launcher.c                @vsajip
 
 # HTML
 /Lib/html/                    @ezio-melotti


From webhook-mailer at python.org  Fri Oct  4 06:09:56 2019
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Fri, 04 Oct 2019 10:09:56 -0000
Subject: [Python-checkins] bpo-13153: Use OS native encoding for converting
 between Python and Tcl. (GH-16545)
Message-ID: <mailman.213.1570183797.32504.python-checkins@python.org>

https://github.com/python/cpython/commit/06cb94bc8419b9a24df6b0d724fcd8e40c6971d6
commit: 06cb94bc8419b9a24df6b0d724fcd8e40c6971d6
branch: master
author: Serhiy Storchaka <storchaka at gmail.com>
committer: GitHub <noreply at github.com>
date: 2019-10-04T13:09:52+03:00
summary:

bpo-13153: Use OS native encoding for converting between Python and Tcl. (GH-16545)

On Windows use UTF-16 (or UTF-32 for 32-bit Tcl_UniChar) with the
"surrogatepass" error handler for converting to/from Tcl Unicode objects.

On Linux use UTF-8 with the "surrogateescape" error handler for converting
to/from Tcl String objects.

Converting strings from Tcl to Python and back now never fails
(except MemoryError).

files:
A Misc/NEWS.d/next/Library/2019-09-29-22-47-37.bpo-13153.0mO9qR.rst
M Lib/idlelib/editor.py
M Lib/idlelib/idle_test/test_editor.py
M Lib/idlelib/pyshell.py
M Lib/idlelib/runscript.py
M Lib/test/test_tcl.py
M Lib/tkinter/test/test_tkinter/test_misc.py
M Lib/tkinter/test/test_ttk/test_widgets.py
M Modules/_tkinter.c

diff --git a/Lib/idlelib/editor.py b/Lib/idlelib/editor.py
index b969f8c493b27..adeed74059f84 100644
--- a/Lib/idlelib/editor.py
+++ b/Lib/idlelib/editor.py
@@ -358,21 +358,6 @@ def set_width(self):
             Font(text, font=text.cget('font')).measure('0')
         self.width = pixel_width // zero_char_width
 
-    def _filename_to_unicode(self, filename):
-        """Return filename as BMP unicode so displayable in Tk."""
-        # Decode bytes to unicode.
-        if isinstance(filename, bytes):
-            try:
-                filename = filename.decode(self.filesystemencoding)
-            except UnicodeDecodeError:
-                try:
-                    filename = filename.decode(self.encoding)
-                except UnicodeDecodeError:
-                    # byte-to-byte conversion
-                    filename = filename.decode('iso8859-1')
-        # Replace non-BMP char with diamond questionmark.
-        return re.sub('[\U00010000-\U0010FFFF]', '\ufffd', filename)
-
     def new_callback(self, event):
         dirname, basename = self.io.defaultfilename()
         self.flist.new(dirname)
@@ -963,10 +948,8 @@ def update_recent_files_list(self, new_file=None):
             menu.delete(0, END)  # clear, and rebuild:
             for i, file_name in enumerate(rf_list):
                 file_name = file_name.rstrip()  # zap \n
-                # make unicode string to display non-ASCII chars correctly
-                ufile_name = self._filename_to_unicode(file_name)
                 callback = instance.__recent_file_callback(file_name)
-                menu.add_command(label=ulchars[i] + " " + ufile_name,
+                menu.add_command(label=ulchars[i] + " " + file_name,
                                  command=callback,
                                  underline=0)
 
@@ -1004,16 +987,10 @@ def reset_undo(self):
 
     def short_title(self):
         filename = self.io.filename
-        if filename:
-            filename = os.path.basename(filename)
-        else:
-            filename = "untitled"
-        # return unicode string to display non-ASCII chars correctly
-        return self._filename_to_unicode(filename)
+        return os.path.basename(filename) if filename else "untitled"
 
     def long_title(self):
-        # return unicode string to display non-ASCII chars correctly
-        return self._filename_to_unicode(self.io.filename or "")
+        return self.io.filename or ""
 
     def center_insert_event(self, event):
         self.center()
diff --git a/Lib/idlelib/idle_test/test_editor.py b/Lib/idlelib/idle_test/test_editor.py
index 4af4ff0242d74..240db71747a28 100644
--- a/Lib/idlelib/idle_test/test_editor.py
+++ b/Lib/idlelib/idle_test/test_editor.py
@@ -30,18 +30,6 @@ def test_init(self):
         e._close()
 
 
-class EditorFunctionTest(unittest.TestCase):
-
-    def test_filename_to_unicode(self):
-        func = Editor._filename_to_unicode
-        class dummy():
-            filesystemencoding = 'utf-8'
-        pairs = (('abc', 'abc'), ('a\U00011111c', 'a\ufffdc'),
-                 (b'abc', 'abc'), (b'a\xf0\x91\x84\x91c', 'a\ufffdc'))
-        for inp, out in pairs:
-            self.assertEqual(func(dummy, inp), out)
-
-
 class TestGetLineIndent(unittest.TestCase):
     def test_empty_lines(self):
         for tabwidth in [1, 2, 4, 6, 8]:
diff --git a/Lib/idlelib/pyshell.py b/Lib/idlelib/pyshell.py
index 2e4dfad401221..bc87d24eaabf8 100755
--- a/Lib/idlelib/pyshell.py
+++ b/Lib/idlelib/pyshell.py
@@ -679,14 +679,6 @@ def runsource(self, source):
         self.more = 0
         # at the moment, InteractiveInterpreter expects str
         assert isinstance(source, str)
-        #if isinstance(source, str):
-        #    from idlelib import iomenu
-        #    try:
-        #        source = source.encode(iomenu.encoding)
-        #    except UnicodeError:
-        #        self.tkconsole.resetoutput()
-        #        self.write("Unsupported characters in input\n")
-        #        return
         # InteractiveInterpreter.runsource() calls its runcode() method,
         # which is overridden (see below)
         return InteractiveInterpreter.runsource(self, source, filename)
@@ -1298,16 +1290,6 @@ def resetoutput(self):
         self.set_line_and_column()
 
     def write(self, s, tags=()):
-        if isinstance(s, str) and len(s) and max(s) > '\uffff':
-            # Tk doesn't support outputting non-BMP characters
-            # Let's assume what printed string is not very long,
-            # find first non-BMP character and construct informative
-            # UnicodeEncodeError exception.
-            for start, char in enumerate(s):
-                if char > '\uffff':
-                    break
-            raise UnicodeEncodeError("UCS-2", char, start, start+1,
-                                     'Non-BMP character not supported in Tk')
         try:
             self.text.mark_gravity("iomark", "right")
             count = OutputWindow.write(self, s, tags, "iomark")
diff --git a/Lib/idlelib/runscript.py b/Lib/idlelib/runscript.py
index de73bf8458151..e99d0d2e027fc 100644
--- a/Lib/idlelib/runscript.py
+++ b/Lib/idlelib/runscript.py
@@ -147,8 +147,7 @@ def _run_module_event(self, event, *, customize=False):
         interp = self.shell.interp
         if pyshell.use_subprocess and restart:
             interp.restart_subprocess(
-                    with_cwd=False, filename=
-                    self.editwin._filename_to_unicode(filename))
+                    with_cwd=False, filename=filename)
         dirname = os.path.dirname(filename)
         argv = [filename]
         if self.cli_args:
diff --git a/Lib/test/test_tcl.py b/Lib/test/test_tcl.py
index 80f1668bcecab..3183ea850f73b 100644
--- a/Lib/test/test_tcl.py
+++ b/Lib/test/test_tcl.py
@@ -429,9 +429,12 @@ def passValue(value):
         self.assertEqual(passValue(False), False if self.wantobjects else '0')
         self.assertEqual(passValue('string'), 'string')
         self.assertEqual(passValue('string\u20ac'), 'string\u20ac')
+        self.assertEqual(passValue('string\U0001f4bb'), 'string\U0001f4bb')
         self.assertEqual(passValue('str\x00ing'), 'str\x00ing')
         self.assertEqual(passValue('str\x00ing\xbd'), 'str\x00ing\xbd')
         self.assertEqual(passValue('str\x00ing\u20ac'), 'str\x00ing\u20ac')
+        self.assertEqual(passValue('str\x00ing\U0001f4bb'),
+                         'str\x00ing\U0001f4bb')
         self.assertEqual(passValue(b'str\x00ing'),
                          b'str\x00ing' if self.wantobjects else 'str\x00ing')
         self.assertEqual(passValue(b'str\xc0\x80ing'),
@@ -490,6 +493,7 @@ def float_eq(actual, expected):
         check('string')
         check('string\xbd')
         check('string\u20ac')
+        check('string\U0001f4bb')
         check('')
         check(b'string', 'string')
         check(b'string\xe2\x82\xac', 'string\xe2\x82\xac')
@@ -531,6 +535,7 @@ def test_splitlist(self):
             ('a\n b\t\r c\n ', ('a', 'b', 'c')),
             (b'a\n b\t\r c\n ', ('a', 'b', 'c')),
             ('a \u20ac', ('a', '\u20ac')),
+            ('a \U0001f4bb', ('a', '\U0001f4bb')),
             (b'a \xe2\x82\xac', ('a', '\u20ac')),
             (b'a\xc0\x80b c\xc0\x80d', ('a\x00b', 'c\x00d')),
             ('a {b c}', ('a', 'b c')),
diff --git a/Lib/tkinter/test/test_tkinter/test_misc.py b/Lib/tkinter/test/test_tkinter/test_misc.py
index 9d5a93ef6fb34..1e089747a91ee 100644
--- a/Lib/tkinter/test/test_tkinter/test_misc.py
+++ b/Lib/tkinter/test/test_tkinter/test_misc.py
@@ -170,6 +170,28 @@ def callback():
         with self.assertRaises(tkinter.TclError):
             root.tk.call('after', 'info', idle1)
 
+    def test_clipboard(self):
+        root = self.root
+        root.clipboard_clear()
+        root.clipboard_append('???')
+        self.assertEqual(root.clipboard_get(), '???')
+        root.clipboard_append('????')
+        self.assertEqual(root.clipboard_get(), '???????')
+        root.clipboard_clear()
+        with self.assertRaises(tkinter.TclError):
+            root.clipboard_get()
+
+    def test_clipboard_astral(self):
+        root = self.root
+        root.clipboard_clear()
+        root.clipboard_append('???')
+        self.assertEqual(root.clipboard_get(), '???')
+        root.clipboard_append('????')
+        self.assertEqual(root.clipboard_get(), '???????')
+        root.clipboard_clear()
+        with self.assertRaises(tkinter.TclError):
+            root.clipboard_get()
+
 
 tests_gui = (MiscTest, )
 
diff --git a/Lib/tkinter/test/test_ttk/test_widgets.py b/Lib/tkinter/test/test_ttk/test_widgets.py
index 69e4b3ffa17fe..2598bc6765207 100644
--- a/Lib/tkinter/test/test_ttk/test_widgets.py
+++ b/Lib/tkinter/test/test_ttk/test_widgets.py
@@ -489,8 +489,7 @@ def check_get_current(getval, currval):
                         expected=('mon', 'tue', 'wed', 'thur'))
         self.checkParam(self.combo, 'values', ('mon', 'tue', 'wed', 'thur'))
         self.checkParam(self.combo, 'values', (42, 3.14, '', 'any string'))
-        self.checkParam(self.combo, 'values', '',
-                        expected='' if get_tk_patchlevel() < (8, 5, 10) else ())
+        self.checkParam(self.combo, 'values', '')
 
         self.combo['values'] = ['a', 1, 'c']
 
@@ -1245,12 +1244,7 @@ def test_values(self):
                         expected=('mon', 'tue', 'wed', 'thur'))
         self.checkParam(self.spin, 'values', ('mon', 'tue', 'wed', 'thur'))
         self.checkParam(self.spin, 'values', (42, 3.14, '', 'any string'))
-        self.checkParam(
-            self.spin,
-            'values',
-            '',
-            expected='' if get_tk_patchlevel() < (8, 5, 10) else ()
-        )
+        self.checkParam(self.spin, 'values', '')
 
         self.spin['values'] = ['a', 1, 'c']
 
@@ -1308,8 +1302,7 @@ def test_columns(self):
         self.checkParam(widget, 'columns', 'a b c',
                         expected=('a', 'b', 'c'))
         self.checkParam(widget, 'columns', ('a', 'b', 'c'))
-        self.checkParam(widget, 'columns', (),
-                        expected='' if get_tk_patchlevel() < (8, 5, 10) else ())
+        self.checkParam(widget, 'columns', '')
 
     def test_displaycolumns(self):
         widget = self.create()
diff --git a/Misc/NEWS.d/next/Library/2019-09-29-22-47-37.bpo-13153.0mO9qR.rst b/Misc/NEWS.d/next/Library/2019-09-29-22-47-37.bpo-13153.0mO9qR.rst
new file mode 100644
index 0000000000000..6f9561539d8d3
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-09-29-22-47-37.bpo-13153.0mO9qR.rst
@@ -0,0 +1,4 @@
+OS native encoding is now used for converting between Python strings and
+Tcl objects.  This allows to display, copy and paste to clipboard emoji and
+other non-BMP characters.  Converting strings from Tcl to Python and back
+now never fails (except MemoryError).
diff --git a/Modules/_tkinter.c b/Modules/_tkinter.c
index 94d5b649e0420..c431d611d4d0c 100644
--- a/Modules/_tkinter.c
+++ b/Modules/_tkinter.c
@@ -96,6 +96,24 @@ Copyright (C) 1994 Steen Lumholt.
 
 #endif /* HAVE_CREATEFILEHANDLER */
 
+/* Use OS native encoding for converting between Python strings and
+   Tcl objects.
+   On Windows use UTF-16 (or UTF-32 for 32-bit Tcl_UniChar) with the
+   "surrogatepass" error handler for converting to/from Tcl Unicode objects.
+   On Linux use UTF-8 with the "surrogateescape" error handler for converting
+   to/from Tcl String objects. */
+#ifdef MS_WINDOWS
+#define USE_TCL_UNICODE 1
+#else
+#define USE_TCL_UNICODE 0
+#endif
+
+#if PY_LITTLE_ENDIAN
+#define NATIVE_BYTEORDER -1
+#else
+#define NATIVE_BYTEORDER 1
+#endif
+
 #ifdef MS_WINDOWS
 #include <conio.h>
 #define WAIT_FOR_STDIN
@@ -290,7 +308,6 @@ typedef struct {
 } TkappObject;
 
 #define Tkapp_Interp(v) (((TkappObject *) (v))->interp)
-#define Tkapp_Result(v) Tcl_GetStringResult(Tkapp_Interp(v))
 
 #define DEBUG_REFCNT(v) (printf("DEBUG: id=%p, refcnt=%i\n", \
 (void *) v, Py_REFCNT(v)))
@@ -311,10 +328,16 @@ static int tk_load_failed = 0;
 #endif
 
 
+static PyObject *Tkapp_UnicodeResult(TkappObject *);
+
 static PyObject *
-Tkinter_Error(PyObject *v)
+Tkinter_Error(TkappObject *self)
 {
-    PyErr_SetString(Tkinter_TclError, Tkapp_Result(v));
+    PyObject *res = Tkapp_UnicodeResult(self);
+    if (res != NULL) {
+        PyErr_SetObject(Tkinter_TclError, res);
+        Py_DECREF(res);
+    }
     return NULL;
 }
 
@@ -368,30 +391,35 @@ static PyObject *
 unicodeFromTclStringAndSize(const char *s, Py_ssize_t size)
 {
     PyObject *r = PyUnicode_DecodeUTF8(s, size, NULL);
-    if (!r && PyErr_ExceptionMatches(PyExc_UnicodeDecodeError)) {
-        /* Tcl encodes null character as \xc0\x80 */
-        if (memchr(s, '\xc0', size)) {
-            char *buf, *q;
-            const char *e = s + size;
-            PyErr_Clear();
-            q = buf = (char *)PyMem_Malloc(size);
-            if (buf == NULL) {
-                PyErr_NoMemory();
-                return NULL;
-            }
-            while (s != e) {
-                if (s + 1 != e && s[0] == '\xc0' && s[1] == '\x80') {
-                    *q++ = '\0';
-                    s += 2;
-                }
-                else
-                    *q++ = *s++;
+    if (r != NULL || !PyErr_ExceptionMatches(PyExc_UnicodeDecodeError)) {
+        return r;
+    }
+
+    char *buf = NULL;
+    PyErr_Clear();
+    /* Tcl encodes null character as \xc0\x80 */
+    if (memchr(s, '\xc0', size)) {
+        char *q;
+        const char *e = s + size;
+        q = buf = (char *)PyMem_Malloc(size);
+        if (buf == NULL) {
+            PyErr_NoMemory();
+            return NULL;
+        }
+        while (s != e) {
+            if (s + 1 != e && s[0] == '\xc0' && s[1] == '\x80') {
+                *q++ = '\0';
+                s += 2;
             }
-            s = buf;
-            size = q - s;
-            r = PyUnicode_DecodeUTF8(s, size, NULL);
-            PyMem_Free(buf);
+            else
+                *q++ = *s++;
         }
+        s = buf;
+        size = q - s;
+    }
+    r = PyUnicode_DecodeUTF8(s, size, "surrogateescape");
+    if (buf != NULL) {
+        PyMem_Free(buf);
     }
     return r;
 }
@@ -406,8 +434,21 @@ static PyObject *
 unicodeFromTclObj(Tcl_Obj *value)
 {
     int len;
-    char *s = Tcl_GetStringFromObj(value, &len);
+#if USE_TCL_UNICODE
+    int byteorder = NATIVE_BYTEORDER;
+    const Tcl_UniChar *u = Tcl_GetUnicodeFromObj(value, &len);
+    if (sizeof(Tcl_UniChar) == 2)
+        return PyUnicode_DecodeUTF16((const char *)u, len * 2,
+                                     "surrogatepass", &byteorder);
+    else if (sizeof(Tcl_UniChar) == 4)
+        return PyUnicode_DecodeUTF32((const char *)u, len * 4,
+                                     "surrogatepass", &byteorder);
+    else
+        Py_UNREACHABLE();
+#else
+    const char *s = Tcl_GetStringFromObj(value, &len);
     return unicodeFromTclStringAndSize(s, len);
+#endif
 }
 
 
@@ -746,7 +787,7 @@ Tkapp_New(const char *screenName, const char *className,
 #endif
 
     if (Tcl_AppInit(v->interp) != TCL_OK) {
-        PyObject *result = Tkinter_Error((PyObject *)v);
+        PyObject *result = Tkinter_Error(v);
 #ifdef TKINTER_PROTECT_LOADTK
         if (wantTk) {
             const char *_tkinter_tk_failed;
@@ -817,12 +858,6 @@ PyTclObject_dealloc(PyTclObject *self)
     Py_DECREF(tp);
 }
 
-static const char *
-PyTclObject_TclString(PyObject *self)
-{
-    return Tcl_GetString(((PyTclObject*)self)->value);
-}
-
 /* Like _str, but create Unicode if necessary. */
 PyDoc_STRVAR(PyTclObject_string__doc__,
 "the string representation of this object, either as str or bytes");
@@ -1048,53 +1083,51 @@ AsObj(PyObject *value)
     }
 
     if (PyUnicode_Check(value)) {
-        void *inbuf;
-        Py_ssize_t size;
-        int kind;
-        Tcl_UniChar *outbuf = NULL;
-        Py_ssize_t i;
-        size_t allocsize;
-
         if (PyUnicode_READY(value) == -1)
             return NULL;
 
-        inbuf = PyUnicode_DATA(value);
-        size = PyUnicode_GET_LENGTH(value);
-        if (size == 0)
-            return Tcl_NewUnicodeObj((const void *)"", 0);
+        Py_ssize_t size = PyUnicode_GET_LENGTH(value);
+        if (size == 0) {
+            return Tcl_NewStringObj("", 0);
+        }
         if (!CHECK_SIZE(size, sizeof(Tcl_UniChar))) {
             PyErr_SetString(PyExc_OverflowError, "string is too long");
             return NULL;
         }
-        kind = PyUnicode_KIND(value);
-        if (kind == sizeof(Tcl_UniChar))
-            return Tcl_NewUnicodeObj(inbuf, (int)size);
-        allocsize = ((size_t)size) * sizeof(Tcl_UniChar);
-        outbuf = (Tcl_UniChar*)PyMem_Malloc(allocsize);
-        /* Else overflow occurred, and we take the next exit */
-        if (!outbuf) {
-            PyErr_NoMemory();
-            return NULL;
+        if (PyUnicode_IS_ASCII(value)) {
+            return Tcl_NewStringObj((const char *)PyUnicode_DATA(value),
+                                    (int)size);
         }
-        for (i = 0; i < size; i++) {
-            Py_UCS4 ch = PyUnicode_READ(kind, inbuf, i);
-            /* We cannot test for sizeof(Tcl_UniChar) directly,
-               so we test for UTF-8 size instead. */
-#if TCL_UTF_MAX == 3
-            if (ch >= 0x10000) {
-                /* Tcl doesn't do UTF-16, yet. */
-                PyErr_Format(Tkinter_TclError,
-                             "character U+%x is above the range "
-                             "(U+0000-U+FFFF) allowed by Tcl",
-                             ch);
-                PyMem_Free(outbuf);
-                return NULL;
-            }
+
+        PyObject *encoded;
+#if USE_TCL_UNICODE
+        if (sizeof(Tcl_UniChar) == 2)
+            encoded = _PyUnicode_EncodeUTF16(value,
+                    "surrogatepass", NATIVE_BYTEORDER);
+        else if (sizeof(Tcl_UniChar) == 4)
+            encoded = _PyUnicode_EncodeUTF32(value,
+                    "surrogatepass", NATIVE_BYTEORDER);
+        else
+            Py_UNREACHABLE();
+#else
+        encoded = _PyUnicode_AsUTF8String(value, "surrogateescape");
 #endif
-            outbuf[i] = ch;
+        if (!encoded) {
+            return NULL;
         }
-        result = Tcl_NewUnicodeObj(outbuf, (int)size);
-        PyMem_Free(outbuf);
+        size = PyBytes_GET_SIZE(encoded);
+        if (size > INT_MAX) {
+            Py_DECREF(encoded);
+            PyErr_SetString(PyExc_OverflowError, "string is too long");
+            return NULL;
+        }
+#if USE_TCL_UNICODE
+        result = Tcl_NewUnicodeObj((const Tcl_UniChar *)PyBytes_AS_STRING(encoded),
+                                   (int)(size / sizeof(Tcl_UniChar)));
+#else
+        result = Tcl_NewStringObj(PyBytes_AS_STRING(encoded), (int)size);
+#endif
+        Py_DECREF(encoded);
         return result;
     }
 
@@ -1113,7 +1146,7 @@ AsObj(PyObject *value)
 }
 
 static PyObject *
-fromBoolean(PyObject* tkapp, Tcl_Obj *value)
+fromBoolean(TkappObject *tkapp, Tcl_Obj *value)
 {
     int boolValue;
     if (Tcl_GetBooleanFromObj(Tkapp_Interp(tkapp), value, &boolValue) == TCL_ERROR)
@@ -1122,7 +1155,7 @@ fromBoolean(PyObject* tkapp, Tcl_Obj *value)
 }
 
 static PyObject*
-fromWideIntObj(PyObject* tkapp, Tcl_Obj *value)
+fromWideIntObj(TkappObject *tkapp, Tcl_Obj *value)
 {
         Tcl_WideInt wideValue;
         if (Tcl_GetWideIntFromObj(Tkapp_Interp(tkapp), value, &wideValue) == TCL_OK) {
@@ -1138,7 +1171,7 @@ fromWideIntObj(PyObject* tkapp, Tcl_Obj *value)
 
 #ifdef HAVE_LIBTOMMAMTH
 static PyObject*
-fromBignumObj(PyObject* tkapp, Tcl_Obj *value)
+fromBignumObj(TkappObject *tkapp, Tcl_Obj *value)
 {
     mp_int bigValue;
     unsigned long numBytes;
@@ -1174,32 +1207,31 @@ fromBignumObj(PyObject* tkapp, Tcl_Obj *value)
 #endif
 
 static PyObject*
-FromObj(PyObject* tkapp, Tcl_Obj *value)
+FromObj(TkappObject *tkapp, Tcl_Obj *value)
 {
     PyObject *result = NULL;
-    TkappObject *app = (TkappObject*)tkapp;
     Tcl_Interp *interp = Tkapp_Interp(tkapp);
 
     if (value->typePtr == NULL) {
-        return unicodeFromTclStringAndSize(value->bytes, value->length);
+        return unicodeFromTclObj(value);
     }
 
-    if (value->typePtr == app->BooleanType ||
-        value->typePtr == app->OldBooleanType) {
+    if (value->typePtr == tkapp->BooleanType ||
+        value->typePtr == tkapp->OldBooleanType) {
         return fromBoolean(tkapp, value);
     }
 
-    if (value->typePtr == app->ByteArrayType) {
+    if (value->typePtr == tkapp->ByteArrayType) {
         int size;
         char *data = (char*)Tcl_GetByteArrayFromObj(value, &size);
         return PyBytes_FromStringAndSize(data, size);
     }
 
-    if (value->typePtr == app->DoubleType) {
+    if (value->typePtr == tkapp->DoubleType) {
         return PyFloat_FromDouble(value->internalRep.doubleValue);
     }
 
-    if (value->typePtr == app->IntType) {
+    if (value->typePtr == tkapp->IntType) {
         long longValue;
         if (Tcl_GetLongFromObj(interp, value, &longValue) == TCL_OK)
             return PyLong_FromLong(longValue);
@@ -1207,8 +1239,8 @@ FromObj(PyObject* tkapp, Tcl_Obj *value)
            fall through to wideInt handling. */
     }
 
-    if (value->typePtr == app->IntType ||
-        value->typePtr == app->WideIntType) {
+    if (value->typePtr == tkapp->IntType ||
+        value->typePtr == tkapp->WideIntType) {
         result = fromWideIntObj(tkapp, value);
         if (result != NULL || PyErr_Occurred())
             return result;
@@ -1218,14 +1250,14 @@ FromObj(PyObject* tkapp, Tcl_Obj *value)
     }
 
 #ifdef HAVE_LIBTOMMAMTH
-    if (value->typePtr == app->IntType ||
-        value->typePtr == app->WideIntType ||
-        value->typePtr == app->BignumType) {
+    if (value->typePtr == tkapp->IntType ||
+        value->typePtr == tkapp->WideIntType ||
+        value->typePtr == tkapp->BignumType) {
         return fromBignumObj(tkapp, value);
     }
 #endif
 
-    if (value->typePtr == app->ListType) {
+    if (value->typePtr == tkapp->ListType) {
         int size;
         int i, status;
         PyObject *elem;
@@ -1253,30 +1285,28 @@ FromObj(PyObject* tkapp, Tcl_Obj *value)
         return result;
     }
 
-    if (value->typePtr == app->ProcBodyType) {
+    if (value->typePtr == tkapp->ProcBodyType) {
       /* fall through: return tcl object. */
     }
 
-    if (value->typePtr == app->StringType) {
-        return PyUnicode_FromKindAndData(
-            sizeof(Tcl_UniChar), Tcl_GetUnicode(value),
-            Tcl_GetCharLength(value));
+    if (value->typePtr == tkapp->StringType) {
+        return unicodeFromTclObj(value);
     }
 
 #if TK_HEX_VERSION >= 0x08050000
-    if (app->BooleanType == NULL &&
+    if (tkapp->BooleanType == NULL &&
         strcmp(value->typePtr->name, "booleanString") == 0) {
         /* booleanString type is not registered in Tcl */
-        app->BooleanType = value->typePtr;
+        tkapp->BooleanType = value->typePtr;
         return fromBoolean(tkapp, value);
     }
 #endif
 
 #ifdef HAVE_LIBTOMMAMTH
-    if (app->BignumType == NULL &&
+    if (tkapp->BignumType == NULL &&
         strcmp(value->typePtr->name, "bignum") == 0) {
         /* bignum type is not registered in Tcl */
-        app->BignumType = value->typePtr;
+        tkapp->BignumType = value->typePtr;
         return fromBignumObj(tkapp, value);
     }
 #endif
@@ -1366,19 +1396,28 @@ Tkapp_CallArgs(PyObject *args, Tcl_Obj** objStore, int *pobjc)
     return NULL;
 }
 
+/* Convert the results of a command call into a Python string. */
+
+static PyObject *
+Tkapp_UnicodeResult(TkappObject *self)
+{
+    return unicodeFromTclObj(Tcl_GetObjResult(self->interp));
+}
+
+
 /* Convert the results of a command call into a Python objects. */
 
-static PyObject*
-Tkapp_CallResult(TkappObject *self)
+static PyObject *
+Tkapp_ObjectResult(TkappObject *self)
 {
     PyObject *res = NULL;
     Tcl_Obj *value = Tcl_GetObjResult(self->interp);
-    if(self->wantobjects) {
+    if (self->wantobjects) {
         /* Not sure whether the IncrRef is necessary, but something
            may overwrite the interpreter result while we are
            converting it. */
         Tcl_IncrRefCount(value);
-        res = FromObj((PyObject*)self, value);
+        res = FromObj(self, value);
         Tcl_DecrRefCount(value);
     } else {
         res = unicodeFromTclObj(value);
@@ -1410,15 +1449,13 @@ Tkapp_CallProc(Tkapp_CallEvent *e, int flags)
     i = Tcl_EvalObjv(e->self->interp, objc, objv, e->flags);
     ENTER_PYTHON
     if (i == TCL_ERROR) {
-        *(e->res) = NULL;
-        *(e->exc_type) = NULL;
-        *(e->exc_tb) = NULL;
-        *(e->exc_value) = PyObject_CallFunction(
-            Tkinter_TclError, "s",
-            Tcl_GetStringResult(e->self->interp));
+        *(e->res) = Tkinter_Error(e->self);
     }
     else {
-        *(e->res) = Tkapp_CallResult(e->self);
+        *(e->res) = Tkapp_ObjectResult(e->self);
+    }
+    if (*(e->res) == NULL) {
+        PyErr_Fetch(e->exc_type, e->exc_value, e->exc_tb);
     }
     LEAVE_PYTHON
 
@@ -1506,9 +1543,9 @@ Tkapp_Call(PyObject *selfptr, PyObject *args)
         ENTER_OVERLAP
 
         if (i == TCL_ERROR)
-            Tkinter_Error(selfptr);
+            Tkinter_Error(self);
         else
-            res = Tkapp_CallResult(self);
+            res = Tkapp_ObjectResult(self);
 
         LEAVE_OVERLAP_TCL
 
@@ -1540,9 +1577,9 @@ _tkinter_tkapp_eval_impl(TkappObject *self, const char *script)
     err = Tcl_Eval(Tkapp_Interp(self), script);
     ENTER_OVERLAP
     if (err == TCL_ERROR)
-        res = Tkinter_Error((PyObject *)self);
+        res = Tkinter_Error(self);
     else
-        res = unicodeFromTclString(Tkapp_Result(self));
+        res = Tkapp_UnicodeResult(self);
     LEAVE_OVERLAP_TCL
     return res;
 }
@@ -1569,9 +1606,9 @@ _tkinter_tkapp_evalfile_impl(TkappObject *self, const char *fileName)
     err = Tcl_EvalFile(Tkapp_Interp(self), fileName);
     ENTER_OVERLAP
     if (err == TCL_ERROR)
-        res = Tkinter_Error((PyObject *)self);
+        res = Tkinter_Error(self);
     else
-        res = unicodeFromTclString(Tkapp_Result(self));
+        res = Tkapp_UnicodeResult(self);
     LEAVE_OVERLAP_TCL
     return res;
 }
@@ -1598,9 +1635,9 @@ _tkinter_tkapp_record_impl(TkappObject *self, const char *script)
     err = Tcl_RecordAndEval(Tkapp_Interp(self), script, TCL_NO_EVAL);
     ENTER_OVERLAP
     if (err == TCL_ERROR)
-        res = Tkinter_Error((PyObject *)self);
+        res = Tkinter_Error(self);
     else
-        res = unicodeFromTclString(Tkapp_Result(self));
+        res = Tkapp_UnicodeResult(self);
     LEAVE_OVERLAP_TCL
     return res;
 }
@@ -1631,13 +1668,13 @@ _tkinter_tkapp_adderrorinfo_impl(TkappObject *self, const char *msg)
 
 /** Tcl Variable **/
 
-typedef PyObject* (*EventFunc)(PyObject*, PyObject *args, int flags);
+typedef PyObject* (*EventFunc)(TkappObject *, PyObject *, int);
 
 TCL_DECLARE_MUTEX(var_mutex)
 
 typedef struct VarEvent {
     Tcl_Event ev; /* must be first */
-    PyObject *self;
+    TkappObject *self;
     PyObject *args;
     int flags;
     EventFunc func;
@@ -1692,7 +1729,7 @@ varname_converter(PyObject *in, void *_out)
         return 1;
     }
     if (PyTclObject_Check(in)) {
-        *out = PyTclObject_TclString(in);
+        *out = Tcl_GetString(((PyTclObject *)in)->value);
         return 1;
     }
     PyErr_Format(PyExc_TypeError,
@@ -1750,7 +1787,7 @@ var_invoke(EventFunc func, PyObject *selfptr, PyObject *args, int flags)
             PyErr_NoMemory();
             return NULL;
         }
-        ev->self = selfptr;
+        ev->self = self;
         ev->args = args;
         ev->flags = flags;
         ev->func = func;
@@ -1770,11 +1807,11 @@ var_invoke(EventFunc func, PyObject *selfptr, PyObject *args, int flags)
         return res;
     }
     /* Tcl is not threaded, or this is the interpreter thread. */
-    return func(selfptr, args, flags);
+    return func(self, args, flags);
 }
 
 static PyObject *
-SetVar(PyObject *self, PyObject *args, int flags)
+SetVar(TkappObject *self, PyObject *args, int flags)
 {
     const char *name1, *name2;
     PyObject *newValue;
@@ -1843,7 +1880,7 @@ Tkapp_GlobalSetVar(PyObject *self, PyObject *args)
 
 
 static PyObject *
-GetVar(PyObject *self, PyObject *args, int flags)
+GetVar(TkappObject *self, PyObject *args, int flags)
 {
     const char *name1, *name2=NULL;
     PyObject *res = NULL;
@@ -1858,10 +1895,9 @@ GetVar(PyObject *self, PyObject *args, int flags)
     tres = Tcl_GetVar2Ex(Tkapp_Interp(self), name1, name2, flags);
     ENTER_OVERLAP
     if (tres == NULL) {
-        PyErr_SetString(Tkinter_TclError,
-                        Tcl_GetStringResult(Tkapp_Interp(self)));
+        Tkinter_Error(self);
     } else {
-        if (((TkappObject*)self)->wantobjects) {
+        if (self->wantobjects) {
             res = FromObj(self, tres);
         }
         else {
@@ -1887,7 +1923,7 @@ Tkapp_GlobalGetVar(PyObject *self, PyObject *args)
 
 
 static PyObject *
-UnsetVar(PyObject *self, PyObject *args, int flags)
+UnsetVar(TkappObject *self, PyObject *args, int flags)
 {
     char *name1, *name2=NULL;
     int code;
@@ -1959,7 +1995,7 @@ _tkinter_tkapp_getint(TkappObject *self, PyObject *arg)
         CHECK_STRING_LENGTH(s);
         value = Tcl_NewStringObj(s, -1);
         if (value == NULL)
-            return Tkinter_Error((PyObject *)self);
+            return Tkinter_Error(self);
     }
     /* Don't use Tcl_GetInt() because it returns ambiguous result for value
        in ranges -2**32..-2**31-1 and 2**31..2**32-1 (on 32-bit platform).
@@ -1968,14 +2004,14 @@ _tkinter_tkapp_getint(TkappObject *self, PyObject *arg)
        value in ranges -2**64..-2**63-1 and 2**63..2**64-1 (on 32-bit platform).
      */
 #ifdef HAVE_LIBTOMMAMTH
-    result = fromBignumObj((PyObject *)self, value);
+    result = fromBignumObj(self, value);
 #else
-    result = fromWideIntObj((PyObject *)self, value);
+    result = fromWideIntObj(self, value);
 #endif
     Tcl_DecrRefCount(value);
     if (result != NULL || PyErr_Occurred())
         return result;
-    return Tkinter_Error((PyObject *)self);
+    return Tkinter_Error(self);
 }
 
 /*[clinic input]
@@ -2006,7 +2042,7 @@ _tkinter_tkapp_getdouble(TkappObject *self, PyObject *arg)
         if (Tcl_GetDoubleFromObj(Tkapp_Interp(self),
                                  ((PyTclObject*)arg)->value,
                                  &v) == TCL_ERROR)
-            return Tkinter_Error((PyObject *)self);
+            return Tkinter_Error(self);
         return PyFloat_FromDouble(v);
     }
 
@@ -2014,7 +2050,7 @@ _tkinter_tkapp_getdouble(TkappObject *self, PyObject *arg)
         return NULL;
     CHECK_STRING_LENGTH(s);
     if (Tcl_GetDouble(Tkapp_Interp(self), s, &v) == TCL_ERROR)
-        return Tkinter_Error((PyObject *)self);
+        return Tkinter_Error(self);
     return PyFloat_FromDouble(v);
 }
 
@@ -2041,7 +2077,7 @@ _tkinter_tkapp_getboolean(TkappObject *self, PyObject *arg)
         if (Tcl_GetBooleanFromObj(Tkapp_Interp(self),
                                   ((PyTclObject*)arg)->value,
                                   &v) == TCL_ERROR)
-            return Tkinter_Error((PyObject *)self);
+            return Tkinter_Error(self);
         return PyBool_FromLong(v);
     }
 
@@ -2049,7 +2085,7 @@ _tkinter_tkapp_getboolean(TkappObject *self, PyObject *arg)
         return NULL;
     CHECK_STRING_LENGTH(s);
     if (Tcl_GetBoolean(Tkapp_Interp(self), s, &v) == TCL_ERROR)
-        return Tkinter_Error((PyObject *)self);
+        return Tkinter_Error(self);
     return PyBool_FromLong(v);
 }
 
@@ -2075,9 +2111,9 @@ _tkinter_tkapp_exprstring_impl(TkappObject *self, const char *s)
     retval = Tcl_ExprString(Tkapp_Interp(self), s);
     ENTER_OVERLAP
     if (retval == TCL_ERROR)
-        res = Tkinter_Error((PyObject *)self);
+        res = Tkinter_Error(self);
     else
-        res = unicodeFromTclString(Tkapp_Result(self));
+        res = Tkapp_UnicodeResult(self);
     LEAVE_OVERLAP_TCL
     return res;
 }
@@ -2105,7 +2141,7 @@ _tkinter_tkapp_exprlong_impl(TkappObject *self, const char *s)
     retval = Tcl_ExprLong(Tkapp_Interp(self), s, &v);
     ENTER_OVERLAP
     if (retval == TCL_ERROR)
-        res = Tkinter_Error((PyObject *)self);
+        res = Tkinter_Error(self);
     else
         res = PyLong_FromLong(v);
     LEAVE_OVERLAP_TCL
@@ -2136,7 +2172,7 @@ _tkinter_tkapp_exprdouble_impl(TkappObject *self, const char *s)
     ENTER_OVERLAP
     PyFPE_END_PROTECT(retval)
     if (retval == TCL_ERROR)
-        res = Tkinter_Error((PyObject *)self);
+        res = Tkinter_Error(self);
     else
         res = PyFloat_FromDouble(v);
     LEAVE_OVERLAP_TCL
@@ -2165,7 +2201,7 @@ _tkinter_tkapp_exprboolean_impl(TkappObject *self, const char *s)
     retval = Tcl_ExprBoolean(Tkapp_Interp(self), s, &v);
     ENTER_OVERLAP
     if (retval == TCL_ERROR)
-        res = Tkinter_Error((PyObject *)self);
+        res = Tkinter_Error(self);
     else
         res = PyLong_FromLong(v);
     LEAVE_OVERLAP_TCL
@@ -2198,12 +2234,12 @@ _tkinter_tkapp_splitlist(TkappObject *self, PyObject *arg)
         if (Tcl_ListObjGetElements(Tkapp_Interp(self),
                                    ((PyTclObject*)arg)->value,
                                    &objc, &objv) == TCL_ERROR) {
-            return Tkinter_Error((PyObject *)self);
+            return Tkinter_Error(self);
         }
         if (!(v = PyTuple_New(objc)))
             return NULL;
         for (i = 0; i < objc; i++) {
-            PyObject *s = FromObj((PyObject*)self, objv[i]);
+            PyObject *s = FromObj(self, objv[i]);
             if (!s) {
                 Py_DECREF(v);
                 return NULL;
@@ -2231,7 +2267,7 @@ _tkinter_tkapp_splitlist(TkappObject *self, PyObject *arg)
     if (Tcl_SplitList(Tkapp_Interp(self), list,
                       &argc, &argv) == TCL_ERROR)  {
         PyMem_Free(list);
-        return Tkinter_Error((PyObject *)self);
+        return Tkinter_Error(self);
     }
 
     if (!(v = PyTuple_New(argc)))
@@ -2275,16 +2311,16 @@ _tkinter_tkapp_split(TkappObject *self, PyObject *arg)
         int i;
         if (Tcl_ListObjGetElements(Tkapp_Interp(self), value,
                                    &objc, &objv) == TCL_ERROR) {
-            return FromObj((PyObject*)self, value);
+            return FromObj(self, value);
         }
         if (objc == 0)
             return PyUnicode_FromString("");
         if (objc == 1)
-            return FromObj((PyObject*)self, objv[0]);
+            return FromObj(self, objv[0]);
         if (!(v = PyTuple_New(objc)))
             return NULL;
         for (i = 0; i < objc; i++) {
-            PyObject *s = FromObj((PyObject*)self, objv[i]);
+            PyObject *s = FromObj(self, objv[i]);
             if (!s) {
                 Py_DECREF(v);
                 return NULL;
@@ -2331,34 +2367,31 @@ PythonCmd_Error(Tcl_Interp *interp)
  * function or method.
  */
 static int
-PythonCmd(ClientData clientData, Tcl_Interp *interp, int argc, const char *argv[])
+PythonCmd(ClientData clientData, Tcl_Interp *interp,
+          int objc, Tcl_Obj *const objv[])
 {
     PythonCmd_ClientData *data = (PythonCmd_ClientData *)clientData;
-    PyObject *func, *arg, *res;
-    int i, rv;
+    PyObject *args, *res;
+    int i;
     Tcl_Obj *obj_res;
 
     ENTER_PYTHON
 
-    /* TBD: no error checking here since we know, via the
-     * Tkapp_CreateCommand() that the client data is a two-tuple
-     */
-    func = data->func;
-
-    /* Create argument list (argv1, ..., argvN) */
-    if (!(arg = PyTuple_New(argc - 1)))
+    /* Create argument tuple (objv1, ..., objvN) */
+    if (!(args = PyTuple_New(objc - 1)))
         return PythonCmd_Error(interp);
 
-    for (i = 0; i < (argc - 1); i++) {
-        PyObject *s = unicodeFromTclString(argv[i + 1]);
+    for (i = 0; i < (objc - 1); i++) {
+        PyObject *s = unicodeFromTclObj(objv[i + 1]);
         if (!s) {
-            Py_DECREF(arg);
+            Py_DECREF(args);
             return PythonCmd_Error(interp);
         }
-        PyTuple_SET_ITEM(arg, i, s);
+        PyTuple_SET_ITEM(args, i, s);
     }
-    res = PyObject_Call(func, arg, NULL);
-    Py_DECREF(arg);
+
+    res = PyObject_Call(data->func, args, NULL);
+    Py_DECREF(args);
 
     if (res == NULL)
         return PythonCmd_Error(interp);
@@ -2368,18 +2401,15 @@ PythonCmd(ClientData clientData, Tcl_Interp *interp, int argc, const char *argv[
         Py_DECREF(res);
         return PythonCmd_Error(interp);
     }
-    else {
-        Tcl_SetObjResult(interp, obj_res);
-        rv = TCL_OK;
-    }
-
+    Tcl_SetObjResult(interp, obj_res);
     Py_DECREF(res);
 
     LEAVE_PYTHON
 
-    return rv;
+    return TCL_OK;
 }
 
+
 static void
 PythonCmdDelete(ClientData clientData)
 {
@@ -2411,7 +2441,7 @@ static int
 Tkapp_CommandProc(CommandEvent *ev, int flags)
 {
     if (ev->create)
-        *ev->status = Tcl_CreateCommand(
+        *ev->status = Tcl_CreateObjCommand(
             ev->interp, ev->name, PythonCmd,
             ev->data, PythonCmdDelete) == NULL;
     else
@@ -2477,7 +2507,7 @@ _tkinter_tkapp_createcommand_impl(TkappObject *self, const char *name,
     else
     {
         ENTER_TCL
-        err = Tcl_CreateCommand(
+        err = Tcl_CreateObjCommand(
             Tkapp_Interp(self), name, PythonCmd,
             (ClientData)data, PythonCmdDelete) == NULL;
         LEAVE_TCL
@@ -2953,9 +2983,9 @@ _tkinter_tkapp_loadtk_impl(TkappObject *self)
     if (err == TCL_ERROR) {
         /* This sets an exception, but we cannot return right
            away because we need to exit the overlap first. */
-        Tkinter_Error((PyObject *)self);
+        Tkinter_Error(self);
     } else {
-        _tk_exists = Tkapp_Result(self);
+        _tk_exists = Tcl_GetStringResult(Tkapp_Interp(self));
     }
     LEAVE_OVERLAP_TCL
     if (err == TCL_ERROR) {
@@ -2963,8 +2993,7 @@ _tkinter_tkapp_loadtk_impl(TkappObject *self)
     }
     if (_tk_exists == NULL || strcmp(_tk_exists, "1") != 0)     {
         if (Tk_Init(interp)             == TCL_ERROR) {
-            PyErr_SetString(Tkinter_TclError,
-                            Tcl_GetStringResult(Tkapp_Interp(self)));
+            Tkinter_Error(self);
 #ifdef TKINTER_PROTECT_LOADTK
             tk_load_failed = 1;
 #endif


From webhook-mailer at python.org  Fri Oct  4 06:28:57 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Fri, 04 Oct 2019 10:28:57 -0000
Subject: [Python-checkins] bpo-13153: Use OS native encoding for converting
 between Python and Tcl. (GH-16545)
Message-ID: <mailman.214.1570184938.32504.python-checkins@python.org>

https://github.com/python/cpython/commit/6c3fbbc177f5e1867ab09a315dbf58554a80accd
commit: 6c3fbbc177f5e1867ab09a315dbf58554a80accd
branch: 3.7
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-04T03:28:53-07:00
summary:

bpo-13153: Use OS native encoding for converting between Python and Tcl. (GH-16545)


On Windows use UTF-16 (or UTF-32 for 32-bit Tcl_UniChar) with the
"surrogatepass" error handler for converting to/from Tcl Unicode objects.

On Linux use UTF-8 with the "surrogateescape" error handler for converting
to/from Tcl String objects.

Converting strings from Tcl to Python and back now never fails
(except MemoryError).
(cherry picked from commit 06cb94bc8419b9a24df6b0d724fcd8e40c6971d6)

Co-authored-by: Serhiy Storchaka <storchaka at gmail.com>

files:
A Misc/NEWS.d/next/Library/2019-09-29-22-47-37.bpo-13153.0mO9qR.rst
M Lib/idlelib/editor.py
M Lib/idlelib/idle_test/test_editor.py
M Lib/idlelib/pyshell.py
M Lib/idlelib/runscript.py
M Lib/test/test_tcl.py
M Lib/tkinter/test/test_tkinter/test_misc.py
M Lib/tkinter/test/test_ttk/test_widgets.py
M Modules/_tkinter.c

diff --git a/Lib/idlelib/editor.py b/Lib/idlelib/editor.py
index b969f8c493b27..adeed74059f84 100644
--- a/Lib/idlelib/editor.py
+++ b/Lib/idlelib/editor.py
@@ -358,21 +358,6 @@ def set_width(self):
             Font(text, font=text.cget('font')).measure('0')
         self.width = pixel_width // zero_char_width
 
-    def _filename_to_unicode(self, filename):
-        """Return filename as BMP unicode so displayable in Tk."""
-        # Decode bytes to unicode.
-        if isinstance(filename, bytes):
-            try:
-                filename = filename.decode(self.filesystemencoding)
-            except UnicodeDecodeError:
-                try:
-                    filename = filename.decode(self.encoding)
-                except UnicodeDecodeError:
-                    # byte-to-byte conversion
-                    filename = filename.decode('iso8859-1')
-        # Replace non-BMP char with diamond questionmark.
-        return re.sub('[\U00010000-\U0010FFFF]', '\ufffd', filename)
-
     def new_callback(self, event):
         dirname, basename = self.io.defaultfilename()
         self.flist.new(dirname)
@@ -963,10 +948,8 @@ def update_recent_files_list(self, new_file=None):
             menu.delete(0, END)  # clear, and rebuild:
             for i, file_name in enumerate(rf_list):
                 file_name = file_name.rstrip()  # zap \n
-                # make unicode string to display non-ASCII chars correctly
-                ufile_name = self._filename_to_unicode(file_name)
                 callback = instance.__recent_file_callback(file_name)
-                menu.add_command(label=ulchars[i] + " " + ufile_name,
+                menu.add_command(label=ulchars[i] + " " + file_name,
                                  command=callback,
                                  underline=0)
 
@@ -1004,16 +987,10 @@ def reset_undo(self):
 
     def short_title(self):
         filename = self.io.filename
-        if filename:
-            filename = os.path.basename(filename)
-        else:
-            filename = "untitled"
-        # return unicode string to display non-ASCII chars correctly
-        return self._filename_to_unicode(filename)
+        return os.path.basename(filename) if filename else "untitled"
 
     def long_title(self):
-        # return unicode string to display non-ASCII chars correctly
-        return self._filename_to_unicode(self.io.filename or "")
+        return self.io.filename or ""
 
     def center_insert_event(self, event):
         self.center()
diff --git a/Lib/idlelib/idle_test/test_editor.py b/Lib/idlelib/idle_test/test_editor.py
index 4af4ff0242d74..240db71747a28 100644
--- a/Lib/idlelib/idle_test/test_editor.py
+++ b/Lib/idlelib/idle_test/test_editor.py
@@ -30,18 +30,6 @@ def test_init(self):
         e._close()
 
 
-class EditorFunctionTest(unittest.TestCase):
-
-    def test_filename_to_unicode(self):
-        func = Editor._filename_to_unicode
-        class dummy():
-            filesystemencoding = 'utf-8'
-        pairs = (('abc', 'abc'), ('a\U00011111c', 'a\ufffdc'),
-                 (b'abc', 'abc'), (b'a\xf0\x91\x84\x91c', 'a\ufffdc'))
-        for inp, out in pairs:
-            self.assertEqual(func(dummy, inp), out)
-
-
 class TestGetLineIndent(unittest.TestCase):
     def test_empty_lines(self):
         for tabwidth in [1, 2, 4, 6, 8]:
diff --git a/Lib/idlelib/pyshell.py b/Lib/idlelib/pyshell.py
index 2e4dfad401221..bc87d24eaabf8 100755
--- a/Lib/idlelib/pyshell.py
+++ b/Lib/idlelib/pyshell.py
@@ -679,14 +679,6 @@ def runsource(self, source):
         self.more = 0
         # at the moment, InteractiveInterpreter expects str
         assert isinstance(source, str)
-        #if isinstance(source, str):
-        #    from idlelib import iomenu
-        #    try:
-        #        source = source.encode(iomenu.encoding)
-        #    except UnicodeError:
-        #        self.tkconsole.resetoutput()
-        #        self.write("Unsupported characters in input\n")
-        #        return
         # InteractiveInterpreter.runsource() calls its runcode() method,
         # which is overridden (see below)
         return InteractiveInterpreter.runsource(self, source, filename)
@@ -1298,16 +1290,6 @@ def resetoutput(self):
         self.set_line_and_column()
 
     def write(self, s, tags=()):
-        if isinstance(s, str) and len(s) and max(s) > '\uffff':
-            # Tk doesn't support outputting non-BMP characters
-            # Let's assume what printed string is not very long,
-            # find first non-BMP character and construct informative
-            # UnicodeEncodeError exception.
-            for start, char in enumerate(s):
-                if char > '\uffff':
-                    break
-            raise UnicodeEncodeError("UCS-2", char, start, start+1,
-                                     'Non-BMP character not supported in Tk')
         try:
             self.text.mark_gravity("iomark", "right")
             count = OutputWindow.write(self, s, tags, "iomark")
diff --git a/Lib/idlelib/runscript.py b/Lib/idlelib/runscript.py
index de73bf8458151..e99d0d2e027fc 100644
--- a/Lib/idlelib/runscript.py
+++ b/Lib/idlelib/runscript.py
@@ -147,8 +147,7 @@ def _run_module_event(self, event, *, customize=False):
         interp = self.shell.interp
         if pyshell.use_subprocess and restart:
             interp.restart_subprocess(
-                    with_cwd=False, filename=
-                    self.editwin._filename_to_unicode(filename))
+                    with_cwd=False, filename=filename)
         dirname = os.path.dirname(filename)
         argv = [filename]
         if self.cli_args:
diff --git a/Lib/test/test_tcl.py b/Lib/test/test_tcl.py
index 80f1668bcecab..3183ea850f73b 100644
--- a/Lib/test/test_tcl.py
+++ b/Lib/test/test_tcl.py
@@ -429,9 +429,12 @@ def passValue(value):
         self.assertEqual(passValue(False), False if self.wantobjects else '0')
         self.assertEqual(passValue('string'), 'string')
         self.assertEqual(passValue('string\u20ac'), 'string\u20ac')
+        self.assertEqual(passValue('string\U0001f4bb'), 'string\U0001f4bb')
         self.assertEqual(passValue('str\x00ing'), 'str\x00ing')
         self.assertEqual(passValue('str\x00ing\xbd'), 'str\x00ing\xbd')
         self.assertEqual(passValue('str\x00ing\u20ac'), 'str\x00ing\u20ac')
+        self.assertEqual(passValue('str\x00ing\U0001f4bb'),
+                         'str\x00ing\U0001f4bb')
         self.assertEqual(passValue(b'str\x00ing'),
                          b'str\x00ing' if self.wantobjects else 'str\x00ing')
         self.assertEqual(passValue(b'str\xc0\x80ing'),
@@ -490,6 +493,7 @@ def float_eq(actual, expected):
         check('string')
         check('string\xbd')
         check('string\u20ac')
+        check('string\U0001f4bb')
         check('')
         check(b'string', 'string')
         check(b'string\xe2\x82\xac', 'string\xe2\x82\xac')
@@ -531,6 +535,7 @@ def test_splitlist(self):
             ('a\n b\t\r c\n ', ('a', 'b', 'c')),
             (b'a\n b\t\r c\n ', ('a', 'b', 'c')),
             ('a \u20ac', ('a', '\u20ac')),
+            ('a \U0001f4bb', ('a', '\U0001f4bb')),
             (b'a \xe2\x82\xac', ('a', '\u20ac')),
             (b'a\xc0\x80b c\xc0\x80d', ('a\x00b', 'c\x00d')),
             ('a {b c}', ('a', 'b c')),
diff --git a/Lib/tkinter/test/test_tkinter/test_misc.py b/Lib/tkinter/test/test_tkinter/test_misc.py
index 1d1a3c29f6bc4..236cae0e9fac4 100644
--- a/Lib/tkinter/test/test_tkinter/test_misc.py
+++ b/Lib/tkinter/test/test_tkinter/test_misc.py
@@ -156,6 +156,28 @@ def callback():
         with self.assertRaises(tkinter.TclError):
             root.tk.call('after', 'info', idle1)
 
+    def test_clipboard(self):
+        root = self.root
+        root.clipboard_clear()
+        root.clipboard_append('???')
+        self.assertEqual(root.clipboard_get(), '???')
+        root.clipboard_append('????')
+        self.assertEqual(root.clipboard_get(), '???????')
+        root.clipboard_clear()
+        with self.assertRaises(tkinter.TclError):
+            root.clipboard_get()
+
+    def test_clipboard_astral(self):
+        root = self.root
+        root.clipboard_clear()
+        root.clipboard_append('???')
+        self.assertEqual(root.clipboard_get(), '???')
+        root.clipboard_append('????')
+        self.assertEqual(root.clipboard_get(), '???????')
+        root.clipboard_clear()
+        with self.assertRaises(tkinter.TclError):
+            root.clipboard_get()
+
 
 tests_gui = (MiscTest, )
 
diff --git a/Lib/tkinter/test/test_ttk/test_widgets.py b/Lib/tkinter/test/test_ttk/test_widgets.py
index ba9e3b54f78e7..9667e05cab12c 100644
--- a/Lib/tkinter/test/test_ttk/test_widgets.py
+++ b/Lib/tkinter/test/test_ttk/test_widgets.py
@@ -489,8 +489,7 @@ def check_get_current(getval, currval):
                         expected=('mon', 'tue', 'wed', 'thur'))
         self.checkParam(self.combo, 'values', ('mon', 'tue', 'wed', 'thur'))
         self.checkParam(self.combo, 'values', (42, 3.14, '', 'any string'))
-        self.checkParam(self.combo, 'values', '',
-                        expected='' if get_tk_patchlevel() < (8, 5, 10) else ())
+        self.checkParam(self.combo, 'values', '')
 
         self.combo['values'] = ['a', 1, 'c']
 
@@ -1245,12 +1244,7 @@ def test_values(self):
                         expected=('mon', 'tue', 'wed', 'thur'))
         self.checkParam(self.spin, 'values', ('mon', 'tue', 'wed', 'thur'))
         self.checkParam(self.spin, 'values', (42, 3.14, '', 'any string'))
-        self.checkParam(
-            self.spin,
-            'values',
-            '',
-            expected='' if get_tk_patchlevel() < (8, 5, 10) else ()
-        )
+        self.checkParam(self.spin, 'values', '')
 
         self.spin['values'] = ['a', 1, 'c']
 
@@ -1308,8 +1302,7 @@ def test_columns(self):
         self.checkParam(widget, 'columns', 'a b c',
                         expected=('a', 'b', 'c'))
         self.checkParam(widget, 'columns', ('a', 'b', 'c'))
-        self.checkParam(widget, 'columns', (),
-                        expected='' if get_tk_patchlevel() < (8, 5, 10) else ())
+        self.checkParam(widget, 'columns', '')
 
     def test_displaycolumns(self):
         widget = self.create()
diff --git a/Misc/NEWS.d/next/Library/2019-09-29-22-47-37.bpo-13153.0mO9qR.rst b/Misc/NEWS.d/next/Library/2019-09-29-22-47-37.bpo-13153.0mO9qR.rst
new file mode 100644
index 0000000000000..6f9561539d8d3
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-09-29-22-47-37.bpo-13153.0mO9qR.rst
@@ -0,0 +1,4 @@
+OS native encoding is now used for converting between Python strings and
+Tcl objects.  This allows to display, copy and paste to clipboard emoji and
+other non-BMP characters.  Converting strings from Tcl to Python and back
+now never fails (except MemoryError).
diff --git a/Modules/_tkinter.c b/Modules/_tkinter.c
index a96924c9c6e71..8ea302ea639f9 100644
--- a/Modules/_tkinter.c
+++ b/Modules/_tkinter.c
@@ -96,6 +96,24 @@ Copyright (C) 1994 Steen Lumholt.
 
 #endif /* HAVE_CREATEFILEHANDLER */
 
+/* Use OS native encoding for converting between Python strings and
+   Tcl objects.
+   On Windows use UTF-16 (or UTF-32 for 32-bit Tcl_UniChar) with the
+   "surrogatepass" error handler for converting to/from Tcl Unicode objects.
+   On Linux use UTF-8 with the "surrogateescape" error handler for converting
+   to/from Tcl String objects. */
+#ifdef MS_WINDOWS
+#define USE_TCL_UNICODE 1
+#else
+#define USE_TCL_UNICODE 0
+#endif
+
+#if PY_LITTLE_ENDIAN
+#define NATIVE_BYTEORDER -1
+#else
+#define NATIVE_BYTEORDER 1
+#endif
+
 #ifdef MS_WINDOWS
 #include <conio.h>
 #define WAIT_FOR_STDIN
@@ -290,7 +308,6 @@ typedef struct {
 } TkappObject;
 
 #define Tkapp_Interp(v) (((TkappObject *) (v))->interp)
-#define Tkapp_Result(v) Tcl_GetStringResult(Tkapp_Interp(v))
 
 #define DEBUG_REFCNT(v) (printf("DEBUG: id=%p, refcnt=%i\n", \
 (void *) v, Py_REFCNT(v)))
@@ -311,10 +328,16 @@ static int tk_load_failed = 0;
 #endif
 
 
+static PyObject *Tkapp_UnicodeResult(TkappObject *);
+
 static PyObject *
-Tkinter_Error(PyObject *v)
+Tkinter_Error(TkappObject *self)
 {
-    PyErr_SetString(Tkinter_TclError, Tkapp_Result(v));
+    PyObject *res = Tkapp_UnicodeResult(self);
+    if (res != NULL) {
+        PyErr_SetObject(Tkinter_TclError, res);
+        Py_DECREF(res);
+    }
     return NULL;
 }
 
@@ -368,30 +391,35 @@ static PyObject *
 unicodeFromTclStringAndSize(const char *s, Py_ssize_t size)
 {
     PyObject *r = PyUnicode_DecodeUTF8(s, size, NULL);
-    if (!r && PyErr_ExceptionMatches(PyExc_UnicodeDecodeError)) {
-        /* Tcl encodes null character as \xc0\x80 */
-        if (memchr(s, '\xc0', size)) {
-            char *buf, *q;
-            const char *e = s + size;
-            PyErr_Clear();
-            q = buf = (char *)PyMem_Malloc(size);
-            if (buf == NULL) {
-                PyErr_NoMemory();
-                return NULL;
-            }
-            while (s != e) {
-                if (s + 1 != e && s[0] == '\xc0' && s[1] == '\x80') {
-                    *q++ = '\0';
-                    s += 2;
-                }
-                else
-                    *q++ = *s++;
+    if (r != NULL || !PyErr_ExceptionMatches(PyExc_UnicodeDecodeError)) {
+        return r;
+    }
+
+    char *buf = NULL;
+    PyErr_Clear();
+    /* Tcl encodes null character as \xc0\x80 */
+    if (memchr(s, '\xc0', size)) {
+        char *q;
+        const char *e = s + size;
+        q = buf = (char *)PyMem_Malloc(size);
+        if (buf == NULL) {
+            PyErr_NoMemory();
+            return NULL;
+        }
+        while (s != e) {
+            if (s + 1 != e && s[0] == '\xc0' && s[1] == '\x80') {
+                *q++ = '\0';
+                s += 2;
             }
-            s = buf;
-            size = q - s;
-            r = PyUnicode_DecodeUTF8(s, size, NULL);
-            PyMem_Free(buf);
+            else
+                *q++ = *s++;
         }
+        s = buf;
+        size = q - s;
+    }
+    r = PyUnicode_DecodeUTF8(s, size, "surrogateescape");
+    if (buf != NULL) {
+        PyMem_Free(buf);
     }
     return r;
 }
@@ -406,8 +434,21 @@ static PyObject *
 unicodeFromTclObj(Tcl_Obj *value)
 {
     int len;
-    char *s = Tcl_GetStringFromObj(value, &len);
+#if USE_TCL_UNICODE
+    int byteorder = NATIVE_BYTEORDER;
+    const Tcl_UniChar *u = Tcl_GetUnicodeFromObj(value, &len);
+    if (sizeof(Tcl_UniChar) == 2)
+        return PyUnicode_DecodeUTF16((const char *)u, len * 2,
+                                     "surrogatepass", &byteorder);
+    else if (sizeof(Tcl_UniChar) == 4)
+        return PyUnicode_DecodeUTF32((const char *)u, len * 4,
+                                     "surrogatepass", &byteorder);
+    else
+        Py_UNREACHABLE();
+#else
+    const char *s = Tcl_GetStringFromObj(value, &len);
     return unicodeFromTclStringAndSize(s, len);
+#endif
 }
 
 
@@ -747,7 +788,7 @@ Tkapp_New(const char *screenName, const char *className,
 #endif
 
     if (Tcl_AppInit(v->interp) != TCL_OK) {
-        PyObject *result = Tkinter_Error((PyObject *)v);
+        PyObject *result = Tkinter_Error(v);
 #ifdef TKINTER_PROTECT_LOADTK
         if (wantTk) {
             const char *_tkinter_tk_failed;
@@ -819,12 +860,6 @@ PyTclObject_dealloc(PyTclObject *self)
     Py_DECREF(tp);
 }
 
-static const char *
-PyTclObject_TclString(PyObject *self)
-{
-    return Tcl_GetString(((PyTclObject*)self)->value);
-}
-
 /* Like _str, but create Unicode if necessary. */
 PyDoc_STRVAR(PyTclObject_string__doc__,
 "the string representation of this object, either as str or bytes");
@@ -1050,53 +1085,51 @@ AsObj(PyObject *value)
     }
 
     if (PyUnicode_Check(value)) {
-        void *inbuf;
-        Py_ssize_t size;
-        int kind;
-        Tcl_UniChar *outbuf = NULL;
-        Py_ssize_t i;
-        size_t allocsize;
-
         if (PyUnicode_READY(value) == -1)
             return NULL;
 
-        inbuf = PyUnicode_DATA(value);
-        size = PyUnicode_GET_LENGTH(value);
-        if (size == 0)
-            return Tcl_NewUnicodeObj((const void *)"", 0);
+        Py_ssize_t size = PyUnicode_GET_LENGTH(value);
+        if (size == 0) {
+            return Tcl_NewStringObj("", 0);
+        }
         if (!CHECK_SIZE(size, sizeof(Tcl_UniChar))) {
             PyErr_SetString(PyExc_OverflowError, "string is too long");
             return NULL;
         }
-        kind = PyUnicode_KIND(value);
-        if (kind == sizeof(Tcl_UniChar))
-            return Tcl_NewUnicodeObj(inbuf, (int)size);
-        allocsize = ((size_t)size) * sizeof(Tcl_UniChar);
-        outbuf = (Tcl_UniChar*)PyMem_Malloc(allocsize);
-        /* Else overflow occurred, and we take the next exit */
-        if (!outbuf) {
-            PyErr_NoMemory();
-            return NULL;
+        if (PyUnicode_IS_ASCII(value)) {
+            return Tcl_NewStringObj((const char *)PyUnicode_DATA(value),
+                                    (int)size);
         }
-        for (i = 0; i < size; i++) {
-            Py_UCS4 ch = PyUnicode_READ(kind, inbuf, i);
-            /* We cannot test for sizeof(Tcl_UniChar) directly,
-               so we test for UTF-8 size instead. */
-#if TCL_UTF_MAX == 3
-            if (ch >= 0x10000) {
-                /* Tcl doesn't do UTF-16, yet. */
-                PyErr_Format(Tkinter_TclError,
-                             "character U+%x is above the range "
-                             "(U+0000-U+FFFF) allowed by Tcl",
-                             ch);
-                PyMem_Free(outbuf);
-                return NULL;
-            }
+
+        PyObject *encoded;
+#if USE_TCL_UNICODE
+        if (sizeof(Tcl_UniChar) == 2)
+            encoded = _PyUnicode_EncodeUTF16(value,
+                    "surrogatepass", NATIVE_BYTEORDER);
+        else if (sizeof(Tcl_UniChar) == 4)
+            encoded = _PyUnicode_EncodeUTF32(value,
+                    "surrogatepass", NATIVE_BYTEORDER);
+        else
+            Py_UNREACHABLE();
+#else
+        encoded = _PyUnicode_AsUTF8String(value, "surrogateescape");
 #endif
-            outbuf[i] = ch;
+        if (!encoded) {
+            return NULL;
         }
-        result = Tcl_NewUnicodeObj(outbuf, (int)size);
-        PyMem_Free(outbuf);
+        size = PyBytes_GET_SIZE(encoded);
+        if (size > INT_MAX) {
+            Py_DECREF(encoded);
+            PyErr_SetString(PyExc_OverflowError, "string is too long");
+            return NULL;
+        }
+#if USE_TCL_UNICODE
+        result = Tcl_NewUnicodeObj((const Tcl_UniChar *)PyBytes_AS_STRING(encoded),
+                                   (int)(size / sizeof(Tcl_UniChar)));
+#else
+        result = Tcl_NewStringObj(PyBytes_AS_STRING(encoded), (int)size);
+#endif
+        Py_DECREF(encoded);
         return result;
     }
 
@@ -1115,7 +1148,7 @@ AsObj(PyObject *value)
 }
 
 static PyObject *
-fromBoolean(PyObject* tkapp, Tcl_Obj *value)
+fromBoolean(TkappObject *tkapp, Tcl_Obj *value)
 {
     int boolValue;
     if (Tcl_GetBooleanFromObj(Tkapp_Interp(tkapp), value, &boolValue) == TCL_ERROR)
@@ -1124,7 +1157,7 @@ fromBoolean(PyObject* tkapp, Tcl_Obj *value)
 }
 
 static PyObject*
-fromWideIntObj(PyObject* tkapp, Tcl_Obj *value)
+fromWideIntObj(TkappObject *tkapp, Tcl_Obj *value)
 {
         Tcl_WideInt wideValue;
         if (Tcl_GetWideIntFromObj(Tkapp_Interp(tkapp), value, &wideValue) == TCL_OK) {
@@ -1140,7 +1173,7 @@ fromWideIntObj(PyObject* tkapp, Tcl_Obj *value)
 
 #ifdef HAVE_LIBTOMMAMTH
 static PyObject*
-fromBignumObj(PyObject* tkapp, Tcl_Obj *value)
+fromBignumObj(TkappObject *tkapp, Tcl_Obj *value)
 {
     mp_int bigValue;
     unsigned long numBytes;
@@ -1176,32 +1209,31 @@ fromBignumObj(PyObject* tkapp, Tcl_Obj *value)
 #endif
 
 static PyObject*
-FromObj(PyObject* tkapp, Tcl_Obj *value)
+FromObj(TkappObject *tkapp, Tcl_Obj *value)
 {
     PyObject *result = NULL;
-    TkappObject *app = (TkappObject*)tkapp;
     Tcl_Interp *interp = Tkapp_Interp(tkapp);
 
     if (value->typePtr == NULL) {
-        return unicodeFromTclStringAndSize(value->bytes, value->length);
+        return unicodeFromTclObj(value);
     }
 
-    if (value->typePtr == app->BooleanType ||
-        value->typePtr == app->OldBooleanType) {
+    if (value->typePtr == tkapp->BooleanType ||
+        value->typePtr == tkapp->OldBooleanType) {
         return fromBoolean(tkapp, value);
     }
 
-    if (value->typePtr == app->ByteArrayType) {
+    if (value->typePtr == tkapp->ByteArrayType) {
         int size;
         char *data = (char*)Tcl_GetByteArrayFromObj(value, &size);
         return PyBytes_FromStringAndSize(data, size);
     }
 
-    if (value->typePtr == app->DoubleType) {
+    if (value->typePtr == tkapp->DoubleType) {
         return PyFloat_FromDouble(value->internalRep.doubleValue);
     }
 
-    if (value->typePtr == app->IntType) {
+    if (value->typePtr == tkapp->IntType) {
         long longValue;
         if (Tcl_GetLongFromObj(interp, value, &longValue) == TCL_OK)
             return PyLong_FromLong(longValue);
@@ -1209,8 +1241,8 @@ FromObj(PyObject* tkapp, Tcl_Obj *value)
            fall through to wideInt handling. */
     }
 
-    if (value->typePtr == app->IntType ||
-        value->typePtr == app->WideIntType) {
+    if (value->typePtr == tkapp->IntType ||
+        value->typePtr == tkapp->WideIntType) {
         result = fromWideIntObj(tkapp, value);
         if (result != NULL || PyErr_Occurred())
             return result;
@@ -1220,14 +1252,14 @@ FromObj(PyObject* tkapp, Tcl_Obj *value)
     }
 
 #ifdef HAVE_LIBTOMMAMTH
-    if (value->typePtr == app->IntType ||
-        value->typePtr == app->WideIntType ||
-        value->typePtr == app->BignumType) {
+    if (value->typePtr == tkapp->IntType ||
+        value->typePtr == tkapp->WideIntType ||
+        value->typePtr == tkapp->BignumType) {
         return fromBignumObj(tkapp, value);
     }
 #endif
 
-    if (value->typePtr == app->ListType) {
+    if (value->typePtr == tkapp->ListType) {
         int size;
         int i, status;
         PyObject *elem;
@@ -1255,30 +1287,28 @@ FromObj(PyObject* tkapp, Tcl_Obj *value)
         return result;
     }
 
-    if (value->typePtr == app->ProcBodyType) {
+    if (value->typePtr == tkapp->ProcBodyType) {
       /* fall through: return tcl object. */
     }
 
-    if (value->typePtr == app->StringType) {
-        return PyUnicode_FromKindAndData(
-            sizeof(Tcl_UniChar), Tcl_GetUnicode(value),
-            Tcl_GetCharLength(value));
+    if (value->typePtr == tkapp->StringType) {
+        return unicodeFromTclObj(value);
     }
 
 #if TK_HEX_VERSION >= 0x08050000
-    if (app->BooleanType == NULL &&
+    if (tkapp->BooleanType == NULL &&
         strcmp(value->typePtr->name, "booleanString") == 0) {
         /* booleanString type is not registered in Tcl */
-        app->BooleanType = value->typePtr;
+        tkapp->BooleanType = value->typePtr;
         return fromBoolean(tkapp, value);
     }
 #endif
 
 #ifdef HAVE_LIBTOMMAMTH
-    if (app->BignumType == NULL &&
+    if (tkapp->BignumType == NULL &&
         strcmp(value->typePtr->name, "bignum") == 0) {
         /* bignum type is not registered in Tcl */
-        app->BignumType = value->typePtr;
+        tkapp->BignumType = value->typePtr;
         return fromBignumObj(tkapp, value);
     }
 #endif
@@ -1368,19 +1398,28 @@ Tkapp_CallArgs(PyObject *args, Tcl_Obj** objStore, int *pobjc)
     return NULL;
 }
 
+/* Convert the results of a command call into a Python string. */
+
+static PyObject *
+Tkapp_UnicodeResult(TkappObject *self)
+{
+    return unicodeFromTclObj(Tcl_GetObjResult(self->interp));
+}
+
+
 /* Convert the results of a command call into a Python objects. */
 
-static PyObject*
-Tkapp_CallResult(TkappObject *self)
+static PyObject *
+Tkapp_ObjectResult(TkappObject *self)
 {
     PyObject *res = NULL;
     Tcl_Obj *value = Tcl_GetObjResult(self->interp);
-    if(self->wantobjects) {
+    if (self->wantobjects) {
         /* Not sure whether the IncrRef is necessary, but something
            may overwrite the interpreter result while we are
            converting it. */
         Tcl_IncrRefCount(value);
-        res = FromObj((PyObject*)self, value);
+        res = FromObj(self, value);
         Tcl_DecrRefCount(value);
     } else {
         res = unicodeFromTclObj(value);
@@ -1412,15 +1451,13 @@ Tkapp_CallProc(Tkapp_CallEvent *e, int flags)
     i = Tcl_EvalObjv(e->self->interp, objc, objv, e->flags);
     ENTER_PYTHON
     if (i == TCL_ERROR) {
-        *(e->res) = NULL;
-        *(e->exc_type) = NULL;
-        *(e->exc_tb) = NULL;
-        *(e->exc_value) = PyObject_CallFunction(
-            Tkinter_TclError, "s",
-            Tcl_GetStringResult(e->self->interp));
+        *(e->res) = Tkinter_Error(e->self);
     }
     else {
-        *(e->res) = Tkapp_CallResult(e->self);
+        *(e->res) = Tkapp_ObjectResult(e->self);
+    }
+    if (*(e->res) == NULL) {
+        PyErr_Fetch(e->exc_type, e->exc_value, e->exc_tb);
     }
     LEAVE_PYTHON
 
@@ -1508,9 +1545,9 @@ Tkapp_Call(PyObject *selfptr, PyObject *args)
         ENTER_OVERLAP
 
         if (i == TCL_ERROR)
-            Tkinter_Error(selfptr);
+            Tkinter_Error(self);
         else
-            res = Tkapp_CallResult(self);
+            res = Tkapp_ObjectResult(self);
 
         LEAVE_OVERLAP_TCL
 
@@ -1542,9 +1579,9 @@ _tkinter_tkapp_eval_impl(TkappObject *self, const char *script)
     err = Tcl_Eval(Tkapp_Interp(self), script);
     ENTER_OVERLAP
     if (err == TCL_ERROR)
-        res = Tkinter_Error((PyObject *)self);
+        res = Tkinter_Error(self);
     else
-        res = unicodeFromTclString(Tkapp_Result(self));
+        res = Tkapp_UnicodeResult(self);
     LEAVE_OVERLAP_TCL
     return res;
 }
@@ -1571,9 +1608,9 @@ _tkinter_tkapp_evalfile_impl(TkappObject *self, const char *fileName)
     err = Tcl_EvalFile(Tkapp_Interp(self), fileName);
     ENTER_OVERLAP
     if (err == TCL_ERROR)
-        res = Tkinter_Error((PyObject *)self);
+        res = Tkinter_Error(self);
     else
-        res = unicodeFromTclString(Tkapp_Result(self));
+        res = Tkapp_UnicodeResult(self);
     LEAVE_OVERLAP_TCL
     return res;
 }
@@ -1600,9 +1637,9 @@ _tkinter_tkapp_record_impl(TkappObject *self, const char *script)
     err = Tcl_RecordAndEval(Tkapp_Interp(self), script, TCL_NO_EVAL);
     ENTER_OVERLAP
     if (err == TCL_ERROR)
-        res = Tkinter_Error((PyObject *)self);
+        res = Tkinter_Error(self);
     else
-        res = unicodeFromTclString(Tkapp_Result(self));
+        res = Tkapp_UnicodeResult(self);
     LEAVE_OVERLAP_TCL
     return res;
 }
@@ -1633,13 +1670,13 @@ _tkinter_tkapp_adderrorinfo_impl(TkappObject *self, const char *msg)
 
 /** Tcl Variable **/
 
-typedef PyObject* (*EventFunc)(PyObject*, PyObject *args, int flags);
+typedef PyObject* (*EventFunc)(TkappObject *, PyObject *, int);
 
 TCL_DECLARE_MUTEX(var_mutex)
 
 typedef struct VarEvent {
     Tcl_Event ev; /* must be first */
-    PyObject *self;
+    TkappObject *self;
     PyObject *args;
     int flags;
     EventFunc func;
@@ -1694,7 +1731,7 @@ varname_converter(PyObject *in, void *_out)
         return 1;
     }
     if (PyTclObject_Check(in)) {
-        *out = PyTclObject_TclString(in);
+        *out = Tcl_GetString(((PyTclObject *)in)->value);
         return 1;
     }
     PyErr_Format(PyExc_TypeError,
@@ -1752,7 +1789,7 @@ var_invoke(EventFunc func, PyObject *selfptr, PyObject *args, int flags)
             PyErr_NoMemory();
             return NULL;
         }
-        ev->self = selfptr;
+        ev->self = self;
         ev->args = args;
         ev->flags = flags;
         ev->func = func;
@@ -1772,11 +1809,11 @@ var_invoke(EventFunc func, PyObject *selfptr, PyObject *args, int flags)
         return res;
     }
     /* Tcl is not threaded, or this is the interpreter thread. */
-    return func(selfptr, args, flags);
+    return func(self, args, flags);
 }
 
 static PyObject *
-SetVar(PyObject *self, PyObject *args, int flags)
+SetVar(TkappObject *self, PyObject *args, int flags)
 {
     const char *name1, *name2;
     PyObject *newValue;
@@ -1845,7 +1882,7 @@ Tkapp_GlobalSetVar(PyObject *self, PyObject *args)
 
 
 static PyObject *
-GetVar(PyObject *self, PyObject *args, int flags)
+GetVar(TkappObject *self, PyObject *args, int flags)
 {
     const char *name1, *name2=NULL;
     PyObject *res = NULL;
@@ -1860,10 +1897,9 @@ GetVar(PyObject *self, PyObject *args, int flags)
     tres = Tcl_GetVar2Ex(Tkapp_Interp(self), name1, name2, flags);
     ENTER_OVERLAP
     if (tres == NULL) {
-        PyErr_SetString(Tkinter_TclError,
-                        Tcl_GetStringResult(Tkapp_Interp(self)));
+        Tkinter_Error(self);
     } else {
-        if (((TkappObject*)self)->wantobjects) {
+        if (self->wantobjects) {
             res = FromObj(self, tres);
         }
         else {
@@ -1889,7 +1925,7 @@ Tkapp_GlobalGetVar(PyObject *self, PyObject *args)
 
 
 static PyObject *
-UnsetVar(PyObject *self, PyObject *args, int flags)
+UnsetVar(TkappObject *self, PyObject *args, int flags)
 {
     char *name1, *name2=NULL;
     int code;
@@ -1961,7 +1997,7 @@ _tkinter_tkapp_getint(TkappObject *self, PyObject *arg)
         CHECK_STRING_LENGTH(s);
         value = Tcl_NewStringObj(s, -1);
         if (value == NULL)
-            return Tkinter_Error((PyObject *)self);
+            return Tkinter_Error(self);
     }
     /* Don't use Tcl_GetInt() because it returns ambiguous result for value
        in ranges -2**32..-2**31-1 and 2**31..2**32-1 (on 32-bit platform).
@@ -1970,14 +2006,14 @@ _tkinter_tkapp_getint(TkappObject *self, PyObject *arg)
        value in ranges -2**64..-2**63-1 and 2**63..2**64-1 (on 32-bit platform).
      */
 #ifdef HAVE_LIBTOMMAMTH
-    result = fromBignumObj((PyObject *)self, value);
+    result = fromBignumObj(self, value);
 #else
-    result = fromWideIntObj((PyObject *)self, value);
+    result = fromWideIntObj(self, value);
 #endif
     Tcl_DecrRefCount(value);
     if (result != NULL || PyErr_Occurred())
         return result;
-    return Tkinter_Error((PyObject *)self);
+    return Tkinter_Error(self);
 }
 
 /*[clinic input]
@@ -2008,7 +2044,7 @@ _tkinter_tkapp_getdouble(TkappObject *self, PyObject *arg)
         if (Tcl_GetDoubleFromObj(Tkapp_Interp(self),
                                  ((PyTclObject*)arg)->value,
                                  &v) == TCL_ERROR)
-            return Tkinter_Error((PyObject *)self);
+            return Tkinter_Error(self);
         return PyFloat_FromDouble(v);
     }
 
@@ -2016,7 +2052,7 @@ _tkinter_tkapp_getdouble(TkappObject *self, PyObject *arg)
         return NULL;
     CHECK_STRING_LENGTH(s);
     if (Tcl_GetDouble(Tkapp_Interp(self), s, &v) == TCL_ERROR)
-        return Tkinter_Error((PyObject *)self);
+        return Tkinter_Error(self);
     return PyFloat_FromDouble(v);
 }
 
@@ -2043,7 +2079,7 @@ _tkinter_tkapp_getboolean(TkappObject *self, PyObject *arg)
         if (Tcl_GetBooleanFromObj(Tkapp_Interp(self),
                                   ((PyTclObject*)arg)->value,
                                   &v) == TCL_ERROR)
-            return Tkinter_Error((PyObject *)self);
+            return Tkinter_Error(self);
         return PyBool_FromLong(v);
     }
 
@@ -2051,7 +2087,7 @@ _tkinter_tkapp_getboolean(TkappObject *self, PyObject *arg)
         return NULL;
     CHECK_STRING_LENGTH(s);
     if (Tcl_GetBoolean(Tkapp_Interp(self), s, &v) == TCL_ERROR)
-        return Tkinter_Error((PyObject *)self);
+        return Tkinter_Error(self);
     return PyBool_FromLong(v);
 }
 
@@ -2077,9 +2113,9 @@ _tkinter_tkapp_exprstring_impl(TkappObject *self, const char *s)
     retval = Tcl_ExprString(Tkapp_Interp(self), s);
     ENTER_OVERLAP
     if (retval == TCL_ERROR)
-        res = Tkinter_Error((PyObject *)self);
+        res = Tkinter_Error(self);
     else
-        res = unicodeFromTclString(Tkapp_Result(self));
+        res = Tkapp_UnicodeResult(self);
     LEAVE_OVERLAP_TCL
     return res;
 }
@@ -2107,7 +2143,7 @@ _tkinter_tkapp_exprlong_impl(TkappObject *self, const char *s)
     retval = Tcl_ExprLong(Tkapp_Interp(self), s, &v);
     ENTER_OVERLAP
     if (retval == TCL_ERROR)
-        res = Tkinter_Error((PyObject *)self);
+        res = Tkinter_Error(self);
     else
         res = PyLong_FromLong(v);
     LEAVE_OVERLAP_TCL
@@ -2138,7 +2174,7 @@ _tkinter_tkapp_exprdouble_impl(TkappObject *self, const char *s)
     ENTER_OVERLAP
     PyFPE_END_PROTECT(retval)
     if (retval == TCL_ERROR)
-        res = Tkinter_Error((PyObject *)self);
+        res = Tkinter_Error(self);
     else
         res = PyFloat_FromDouble(v);
     LEAVE_OVERLAP_TCL
@@ -2167,7 +2203,7 @@ _tkinter_tkapp_exprboolean_impl(TkappObject *self, const char *s)
     retval = Tcl_ExprBoolean(Tkapp_Interp(self), s, &v);
     ENTER_OVERLAP
     if (retval == TCL_ERROR)
-        res = Tkinter_Error((PyObject *)self);
+        res = Tkinter_Error(self);
     else
         res = PyLong_FromLong(v);
     LEAVE_OVERLAP_TCL
@@ -2200,12 +2236,12 @@ _tkinter_tkapp_splitlist(TkappObject *self, PyObject *arg)
         if (Tcl_ListObjGetElements(Tkapp_Interp(self),
                                    ((PyTclObject*)arg)->value,
                                    &objc, &objv) == TCL_ERROR) {
-            return Tkinter_Error((PyObject *)self);
+            return Tkinter_Error(self);
         }
         if (!(v = PyTuple_New(objc)))
             return NULL;
         for (i = 0; i < objc; i++) {
-            PyObject *s = FromObj((PyObject*)self, objv[i]);
+            PyObject *s = FromObj(self, objv[i]);
             if (!s) {
                 Py_DECREF(v);
                 return NULL;
@@ -2233,7 +2269,7 @@ _tkinter_tkapp_splitlist(TkappObject *self, PyObject *arg)
     if (Tcl_SplitList(Tkapp_Interp(self), list,
                       &argc, &argv) == TCL_ERROR)  {
         PyMem_Free(list);
-        return Tkinter_Error((PyObject *)self);
+        return Tkinter_Error(self);
     }
 
     if (!(v = PyTuple_New(argc)))
@@ -2277,16 +2313,16 @@ _tkinter_tkapp_split(TkappObject *self, PyObject *arg)
         int i;
         if (Tcl_ListObjGetElements(Tkapp_Interp(self), value,
                                    &objc, &objv) == TCL_ERROR) {
-            return FromObj((PyObject*)self, value);
+            return FromObj(self, value);
         }
         if (objc == 0)
             return PyUnicode_FromString("");
         if (objc == 1)
-            return FromObj((PyObject*)self, objv[0]);
+            return FromObj(self, objv[0]);
         if (!(v = PyTuple_New(objc)))
             return NULL;
         for (i = 0; i < objc; i++) {
-            PyObject *s = FromObj((PyObject*)self, objv[i]);
+            PyObject *s = FromObj(self, objv[i]);
             if (!s) {
                 Py_DECREF(v);
                 return NULL;
@@ -2333,34 +2369,31 @@ PythonCmd_Error(Tcl_Interp *interp)
  * function or method.
  */
 static int
-PythonCmd(ClientData clientData, Tcl_Interp *interp, int argc, const char *argv[])
+PythonCmd(ClientData clientData, Tcl_Interp *interp,
+          int objc, Tcl_Obj *const objv[])
 {
     PythonCmd_ClientData *data = (PythonCmd_ClientData *)clientData;
-    PyObject *func, *arg, *res;
-    int i, rv;
+    PyObject *args, *res;
+    int i;
     Tcl_Obj *obj_res;
 
     ENTER_PYTHON
 
-    /* TBD: no error checking here since we know, via the
-     * Tkapp_CreateCommand() that the client data is a two-tuple
-     */
-    func = data->func;
-
-    /* Create argument list (argv1, ..., argvN) */
-    if (!(arg = PyTuple_New(argc - 1)))
+    /* Create argument tuple (objv1, ..., objvN) */
+    if (!(args = PyTuple_New(objc - 1)))
         return PythonCmd_Error(interp);
 
-    for (i = 0; i < (argc - 1); i++) {
-        PyObject *s = unicodeFromTclString(argv[i + 1]);
+    for (i = 0; i < (objc - 1); i++) {
+        PyObject *s = unicodeFromTclObj(objv[i + 1]);
         if (!s) {
-            Py_DECREF(arg);
+            Py_DECREF(args);
             return PythonCmd_Error(interp);
         }
-        PyTuple_SET_ITEM(arg, i, s);
+        PyTuple_SET_ITEM(args, i, s);
     }
-    res = PyObject_Call(func, arg, NULL);
-    Py_DECREF(arg);
+
+    res = PyObject_Call(data->func, args, NULL);
+    Py_DECREF(args);
 
     if (res == NULL)
         return PythonCmd_Error(interp);
@@ -2370,18 +2403,15 @@ PythonCmd(ClientData clientData, Tcl_Interp *interp, int argc, const char *argv[
         Py_DECREF(res);
         return PythonCmd_Error(interp);
     }
-    else {
-        Tcl_SetObjResult(interp, obj_res);
-        rv = TCL_OK;
-    }
-
+    Tcl_SetObjResult(interp, obj_res);
     Py_DECREF(res);
 
     LEAVE_PYTHON
 
-    return rv;
+    return TCL_OK;
 }
 
+
 static void
 PythonCmdDelete(ClientData clientData)
 {
@@ -2413,7 +2443,7 @@ static int
 Tkapp_CommandProc(CommandEvent *ev, int flags)
 {
     if (ev->create)
-        *ev->status = Tcl_CreateCommand(
+        *ev->status = Tcl_CreateObjCommand(
             ev->interp, ev->name, PythonCmd,
             ev->data, PythonCmdDelete) == NULL;
     else
@@ -2479,7 +2509,7 @@ _tkinter_tkapp_createcommand_impl(TkappObject *self, const char *name,
     else
     {
         ENTER_TCL
-        err = Tcl_CreateCommand(
+        err = Tcl_CreateObjCommand(
             Tkapp_Interp(self), name, PythonCmd,
             (ClientData)data, PythonCmdDelete) == NULL;
         LEAVE_TCL
@@ -2956,9 +2986,9 @@ _tkinter_tkapp_loadtk_impl(TkappObject *self)
     if (err == TCL_ERROR) {
         /* This sets an exception, but we cannot return right
            away because we need to exit the overlap first. */
-        Tkinter_Error((PyObject *)self);
+        Tkinter_Error(self);
     } else {
-        _tk_exists = Tkapp_Result(self);
+        _tk_exists = Tcl_GetStringResult(Tkapp_Interp(self));
     }
     LEAVE_OVERLAP_TCL
     if (err == TCL_ERROR) {
@@ -2966,8 +2996,7 @@ _tkinter_tkapp_loadtk_impl(TkappObject *self)
     }
     if (_tk_exists == NULL || strcmp(_tk_exists, "1") != 0)     {
         if (Tk_Init(interp)             == TCL_ERROR) {
-            PyErr_SetString(Tkinter_TclError,
-                            Tcl_GetStringResult(Tkapp_Interp(self)));
+            Tkinter_Error(self);
 #ifdef TKINTER_PROTECT_LOADTK
             tk_load_failed = 1;
 #endif


From webhook-mailer at python.org  Fri Oct  4 06:28:58 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Fri, 04 Oct 2019 10:28:58 -0000
Subject: [Python-checkins] bpo-13153: Use OS native encoding for converting
 between Python and Tcl. (GH-16545)
Message-ID: <mailman.215.1570184940.32504.python-checkins@python.org>

https://github.com/python/cpython/commit/dc191245d8f63f5ab41afff0468b7463a07e7b00
commit: dc191245d8f63f5ab41afff0468b7463a07e7b00
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-04T03:28:54-07:00
summary:

bpo-13153: Use OS native encoding for converting between Python and Tcl. (GH-16545)


On Windows use UTF-16 (or UTF-32 for 32-bit Tcl_UniChar) with the
"surrogatepass" error handler for converting to/from Tcl Unicode objects.

On Linux use UTF-8 with the "surrogateescape" error handler for converting
to/from Tcl String objects.

Converting strings from Tcl to Python and back now never fails
(except MemoryError).
(cherry picked from commit 06cb94bc8419b9a24df6b0d724fcd8e40c6971d6)

Co-authored-by: Serhiy Storchaka <storchaka at gmail.com>

files:
A Misc/NEWS.d/next/Library/2019-09-29-22-47-37.bpo-13153.0mO9qR.rst
M Lib/idlelib/editor.py
M Lib/idlelib/idle_test/test_editor.py
M Lib/idlelib/pyshell.py
M Lib/idlelib/runscript.py
M Lib/test/test_tcl.py
M Lib/tkinter/test/test_tkinter/test_misc.py
M Lib/tkinter/test/test_ttk/test_widgets.py
M Modules/_tkinter.c

diff --git a/Lib/idlelib/editor.py b/Lib/idlelib/editor.py
index b969f8c493b27..adeed74059f84 100644
--- a/Lib/idlelib/editor.py
+++ b/Lib/idlelib/editor.py
@@ -358,21 +358,6 @@ def set_width(self):
             Font(text, font=text.cget('font')).measure('0')
         self.width = pixel_width // zero_char_width
 
-    def _filename_to_unicode(self, filename):
-        """Return filename as BMP unicode so displayable in Tk."""
-        # Decode bytes to unicode.
-        if isinstance(filename, bytes):
-            try:
-                filename = filename.decode(self.filesystemencoding)
-            except UnicodeDecodeError:
-                try:
-                    filename = filename.decode(self.encoding)
-                except UnicodeDecodeError:
-                    # byte-to-byte conversion
-                    filename = filename.decode('iso8859-1')
-        # Replace non-BMP char with diamond questionmark.
-        return re.sub('[\U00010000-\U0010FFFF]', '\ufffd', filename)
-
     def new_callback(self, event):
         dirname, basename = self.io.defaultfilename()
         self.flist.new(dirname)
@@ -963,10 +948,8 @@ def update_recent_files_list(self, new_file=None):
             menu.delete(0, END)  # clear, and rebuild:
             for i, file_name in enumerate(rf_list):
                 file_name = file_name.rstrip()  # zap \n
-                # make unicode string to display non-ASCII chars correctly
-                ufile_name = self._filename_to_unicode(file_name)
                 callback = instance.__recent_file_callback(file_name)
-                menu.add_command(label=ulchars[i] + " " + ufile_name,
+                menu.add_command(label=ulchars[i] + " " + file_name,
                                  command=callback,
                                  underline=0)
 
@@ -1004,16 +987,10 @@ def reset_undo(self):
 
     def short_title(self):
         filename = self.io.filename
-        if filename:
-            filename = os.path.basename(filename)
-        else:
-            filename = "untitled"
-        # return unicode string to display non-ASCII chars correctly
-        return self._filename_to_unicode(filename)
+        return os.path.basename(filename) if filename else "untitled"
 
     def long_title(self):
-        # return unicode string to display non-ASCII chars correctly
-        return self._filename_to_unicode(self.io.filename or "")
+        return self.io.filename or ""
 
     def center_insert_event(self, event):
         self.center()
diff --git a/Lib/idlelib/idle_test/test_editor.py b/Lib/idlelib/idle_test/test_editor.py
index 4af4ff0242d74..240db71747a28 100644
--- a/Lib/idlelib/idle_test/test_editor.py
+++ b/Lib/idlelib/idle_test/test_editor.py
@@ -30,18 +30,6 @@ def test_init(self):
         e._close()
 
 
-class EditorFunctionTest(unittest.TestCase):
-
-    def test_filename_to_unicode(self):
-        func = Editor._filename_to_unicode
-        class dummy():
-            filesystemencoding = 'utf-8'
-        pairs = (('abc', 'abc'), ('a\U00011111c', 'a\ufffdc'),
-                 (b'abc', 'abc'), (b'a\xf0\x91\x84\x91c', 'a\ufffdc'))
-        for inp, out in pairs:
-            self.assertEqual(func(dummy, inp), out)
-
-
 class TestGetLineIndent(unittest.TestCase):
     def test_empty_lines(self):
         for tabwidth in [1, 2, 4, 6, 8]:
diff --git a/Lib/idlelib/pyshell.py b/Lib/idlelib/pyshell.py
index 2e4dfad401221..bc87d24eaabf8 100755
--- a/Lib/idlelib/pyshell.py
+++ b/Lib/idlelib/pyshell.py
@@ -679,14 +679,6 @@ def runsource(self, source):
         self.more = 0
         # at the moment, InteractiveInterpreter expects str
         assert isinstance(source, str)
-        #if isinstance(source, str):
-        #    from idlelib import iomenu
-        #    try:
-        #        source = source.encode(iomenu.encoding)
-        #    except UnicodeError:
-        #        self.tkconsole.resetoutput()
-        #        self.write("Unsupported characters in input\n")
-        #        return
         # InteractiveInterpreter.runsource() calls its runcode() method,
         # which is overridden (see below)
         return InteractiveInterpreter.runsource(self, source, filename)
@@ -1298,16 +1290,6 @@ def resetoutput(self):
         self.set_line_and_column()
 
     def write(self, s, tags=()):
-        if isinstance(s, str) and len(s) and max(s) > '\uffff':
-            # Tk doesn't support outputting non-BMP characters
-            # Let's assume what printed string is not very long,
-            # find first non-BMP character and construct informative
-            # UnicodeEncodeError exception.
-            for start, char in enumerate(s):
-                if char > '\uffff':
-                    break
-            raise UnicodeEncodeError("UCS-2", char, start, start+1,
-                                     'Non-BMP character not supported in Tk')
         try:
             self.text.mark_gravity("iomark", "right")
             count = OutputWindow.write(self, s, tags, "iomark")
diff --git a/Lib/idlelib/runscript.py b/Lib/idlelib/runscript.py
index de73bf8458151..e99d0d2e027fc 100644
--- a/Lib/idlelib/runscript.py
+++ b/Lib/idlelib/runscript.py
@@ -147,8 +147,7 @@ def _run_module_event(self, event, *, customize=False):
         interp = self.shell.interp
         if pyshell.use_subprocess and restart:
             interp.restart_subprocess(
-                    with_cwd=False, filename=
-                    self.editwin._filename_to_unicode(filename))
+                    with_cwd=False, filename=filename)
         dirname = os.path.dirname(filename)
         argv = [filename]
         if self.cli_args:
diff --git a/Lib/test/test_tcl.py b/Lib/test/test_tcl.py
index 80f1668bcecab..3183ea850f73b 100644
--- a/Lib/test/test_tcl.py
+++ b/Lib/test/test_tcl.py
@@ -429,9 +429,12 @@ def passValue(value):
         self.assertEqual(passValue(False), False if self.wantobjects else '0')
         self.assertEqual(passValue('string'), 'string')
         self.assertEqual(passValue('string\u20ac'), 'string\u20ac')
+        self.assertEqual(passValue('string\U0001f4bb'), 'string\U0001f4bb')
         self.assertEqual(passValue('str\x00ing'), 'str\x00ing')
         self.assertEqual(passValue('str\x00ing\xbd'), 'str\x00ing\xbd')
         self.assertEqual(passValue('str\x00ing\u20ac'), 'str\x00ing\u20ac')
+        self.assertEqual(passValue('str\x00ing\U0001f4bb'),
+                         'str\x00ing\U0001f4bb')
         self.assertEqual(passValue(b'str\x00ing'),
                          b'str\x00ing' if self.wantobjects else 'str\x00ing')
         self.assertEqual(passValue(b'str\xc0\x80ing'),
@@ -490,6 +493,7 @@ def float_eq(actual, expected):
         check('string')
         check('string\xbd')
         check('string\u20ac')
+        check('string\U0001f4bb')
         check('')
         check(b'string', 'string')
         check(b'string\xe2\x82\xac', 'string\xe2\x82\xac')
@@ -531,6 +535,7 @@ def test_splitlist(self):
             ('a\n b\t\r c\n ', ('a', 'b', 'c')),
             (b'a\n b\t\r c\n ', ('a', 'b', 'c')),
             ('a \u20ac', ('a', '\u20ac')),
+            ('a \U0001f4bb', ('a', '\U0001f4bb')),
             (b'a \xe2\x82\xac', ('a', '\u20ac')),
             (b'a\xc0\x80b c\xc0\x80d', ('a\x00b', 'c\x00d')),
             ('a {b c}', ('a', 'b c')),
diff --git a/Lib/tkinter/test/test_tkinter/test_misc.py b/Lib/tkinter/test/test_tkinter/test_misc.py
index 1d1a3c29f6bc4..236cae0e9fac4 100644
--- a/Lib/tkinter/test/test_tkinter/test_misc.py
+++ b/Lib/tkinter/test/test_tkinter/test_misc.py
@@ -156,6 +156,28 @@ def callback():
         with self.assertRaises(tkinter.TclError):
             root.tk.call('after', 'info', idle1)
 
+    def test_clipboard(self):
+        root = self.root
+        root.clipboard_clear()
+        root.clipboard_append('???')
+        self.assertEqual(root.clipboard_get(), '???')
+        root.clipboard_append('????')
+        self.assertEqual(root.clipboard_get(), '???????')
+        root.clipboard_clear()
+        with self.assertRaises(tkinter.TclError):
+            root.clipboard_get()
+
+    def test_clipboard_astral(self):
+        root = self.root
+        root.clipboard_clear()
+        root.clipboard_append('???')
+        self.assertEqual(root.clipboard_get(), '???')
+        root.clipboard_append('????')
+        self.assertEqual(root.clipboard_get(), '???????')
+        root.clipboard_clear()
+        with self.assertRaises(tkinter.TclError):
+            root.clipboard_get()
+
 
 tests_gui = (MiscTest, )
 
diff --git a/Lib/tkinter/test/test_ttk/test_widgets.py b/Lib/tkinter/test/test_ttk/test_widgets.py
index 69e4b3ffa17fe..2598bc6765207 100644
--- a/Lib/tkinter/test/test_ttk/test_widgets.py
+++ b/Lib/tkinter/test/test_ttk/test_widgets.py
@@ -489,8 +489,7 @@ def check_get_current(getval, currval):
                         expected=('mon', 'tue', 'wed', 'thur'))
         self.checkParam(self.combo, 'values', ('mon', 'tue', 'wed', 'thur'))
         self.checkParam(self.combo, 'values', (42, 3.14, '', 'any string'))
-        self.checkParam(self.combo, 'values', '',
-                        expected='' if get_tk_patchlevel() < (8, 5, 10) else ())
+        self.checkParam(self.combo, 'values', '')
 
         self.combo['values'] = ['a', 1, 'c']
 
@@ -1245,12 +1244,7 @@ def test_values(self):
                         expected=('mon', 'tue', 'wed', 'thur'))
         self.checkParam(self.spin, 'values', ('mon', 'tue', 'wed', 'thur'))
         self.checkParam(self.spin, 'values', (42, 3.14, '', 'any string'))
-        self.checkParam(
-            self.spin,
-            'values',
-            '',
-            expected='' if get_tk_patchlevel() < (8, 5, 10) else ()
-        )
+        self.checkParam(self.spin, 'values', '')
 
         self.spin['values'] = ['a', 1, 'c']
 
@@ -1308,8 +1302,7 @@ def test_columns(self):
         self.checkParam(widget, 'columns', 'a b c',
                         expected=('a', 'b', 'c'))
         self.checkParam(widget, 'columns', ('a', 'b', 'c'))
-        self.checkParam(widget, 'columns', (),
-                        expected='' if get_tk_patchlevel() < (8, 5, 10) else ())
+        self.checkParam(widget, 'columns', '')
 
     def test_displaycolumns(self):
         widget = self.create()
diff --git a/Misc/NEWS.d/next/Library/2019-09-29-22-47-37.bpo-13153.0mO9qR.rst b/Misc/NEWS.d/next/Library/2019-09-29-22-47-37.bpo-13153.0mO9qR.rst
new file mode 100644
index 0000000000000..6f9561539d8d3
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-09-29-22-47-37.bpo-13153.0mO9qR.rst
@@ -0,0 +1,4 @@
+OS native encoding is now used for converting between Python strings and
+Tcl objects.  This allows to display, copy and paste to clipboard emoji and
+other non-BMP characters.  Converting strings from Tcl to Python and back
+now never fails (except MemoryError).
diff --git a/Modules/_tkinter.c b/Modules/_tkinter.c
index 02924d480f25e..b622c158a48fa 100644
--- a/Modules/_tkinter.c
+++ b/Modules/_tkinter.c
@@ -96,6 +96,24 @@ Copyright (C) 1994 Steen Lumholt.
 
 #endif /* HAVE_CREATEFILEHANDLER */
 
+/* Use OS native encoding for converting between Python strings and
+   Tcl objects.
+   On Windows use UTF-16 (or UTF-32 for 32-bit Tcl_UniChar) with the
+   "surrogatepass" error handler for converting to/from Tcl Unicode objects.
+   On Linux use UTF-8 with the "surrogateescape" error handler for converting
+   to/from Tcl String objects. */
+#ifdef MS_WINDOWS
+#define USE_TCL_UNICODE 1
+#else
+#define USE_TCL_UNICODE 0
+#endif
+
+#if PY_LITTLE_ENDIAN
+#define NATIVE_BYTEORDER -1
+#else
+#define NATIVE_BYTEORDER 1
+#endif
+
 #ifdef MS_WINDOWS
 #include <conio.h>
 #define WAIT_FOR_STDIN
@@ -290,7 +308,6 @@ typedef struct {
 } TkappObject;
 
 #define Tkapp_Interp(v) (((TkappObject *) (v))->interp)
-#define Tkapp_Result(v) Tcl_GetStringResult(Tkapp_Interp(v))
 
 #define DEBUG_REFCNT(v) (printf("DEBUG: id=%p, refcnt=%i\n", \
 (void *) v, Py_REFCNT(v)))
@@ -311,10 +328,16 @@ static int tk_load_failed = 0;
 #endif
 
 
+static PyObject *Tkapp_UnicodeResult(TkappObject *);
+
 static PyObject *
-Tkinter_Error(PyObject *v)
+Tkinter_Error(TkappObject *self)
 {
-    PyErr_SetString(Tkinter_TclError, Tkapp_Result(v));
+    PyObject *res = Tkapp_UnicodeResult(self);
+    if (res != NULL) {
+        PyErr_SetObject(Tkinter_TclError, res);
+        Py_DECREF(res);
+    }
     return NULL;
 }
 
@@ -368,30 +391,35 @@ static PyObject *
 unicodeFromTclStringAndSize(const char *s, Py_ssize_t size)
 {
     PyObject *r = PyUnicode_DecodeUTF8(s, size, NULL);
-    if (!r && PyErr_ExceptionMatches(PyExc_UnicodeDecodeError)) {
-        /* Tcl encodes null character as \xc0\x80 */
-        if (memchr(s, '\xc0', size)) {
-            char *buf, *q;
-            const char *e = s + size;
-            PyErr_Clear();
-            q = buf = (char *)PyMem_Malloc(size);
-            if (buf == NULL) {
-                PyErr_NoMemory();
-                return NULL;
-            }
-            while (s != e) {
-                if (s + 1 != e && s[0] == '\xc0' && s[1] == '\x80') {
-                    *q++ = '\0';
-                    s += 2;
-                }
-                else
-                    *q++ = *s++;
+    if (r != NULL || !PyErr_ExceptionMatches(PyExc_UnicodeDecodeError)) {
+        return r;
+    }
+
+    char *buf = NULL;
+    PyErr_Clear();
+    /* Tcl encodes null character as \xc0\x80 */
+    if (memchr(s, '\xc0', size)) {
+        char *q;
+        const char *e = s + size;
+        q = buf = (char *)PyMem_Malloc(size);
+        if (buf == NULL) {
+            PyErr_NoMemory();
+            return NULL;
+        }
+        while (s != e) {
+            if (s + 1 != e && s[0] == '\xc0' && s[1] == '\x80') {
+                *q++ = '\0';
+                s += 2;
             }
-            s = buf;
-            size = q - s;
-            r = PyUnicode_DecodeUTF8(s, size, NULL);
-            PyMem_Free(buf);
+            else
+                *q++ = *s++;
         }
+        s = buf;
+        size = q - s;
+    }
+    r = PyUnicode_DecodeUTF8(s, size, "surrogateescape");
+    if (buf != NULL) {
+        PyMem_Free(buf);
     }
     return r;
 }
@@ -406,8 +434,21 @@ static PyObject *
 unicodeFromTclObj(Tcl_Obj *value)
 {
     int len;
-    char *s = Tcl_GetStringFromObj(value, &len);
+#if USE_TCL_UNICODE
+    int byteorder = NATIVE_BYTEORDER;
+    const Tcl_UniChar *u = Tcl_GetUnicodeFromObj(value, &len);
+    if (sizeof(Tcl_UniChar) == 2)
+        return PyUnicode_DecodeUTF16((const char *)u, len * 2,
+                                     "surrogatepass", &byteorder);
+    else if (sizeof(Tcl_UniChar) == 4)
+        return PyUnicode_DecodeUTF32((const char *)u, len * 4,
+                                     "surrogatepass", &byteorder);
+    else
+        Py_UNREACHABLE();
+#else
+    const char *s = Tcl_GetStringFromObj(value, &len);
     return unicodeFromTclStringAndSize(s, len);
+#endif
 }
 
 
@@ -746,7 +787,7 @@ Tkapp_New(const char *screenName, const char *className,
 #endif
 
     if (Tcl_AppInit(v->interp) != TCL_OK) {
-        PyObject *result = Tkinter_Error((PyObject *)v);
+        PyObject *result = Tkinter_Error(v);
 #ifdef TKINTER_PROTECT_LOADTK
         if (wantTk) {
             const char *_tkinter_tk_failed;
@@ -817,12 +858,6 @@ PyTclObject_dealloc(PyTclObject *self)
     Py_DECREF(tp);
 }
 
-static const char *
-PyTclObject_TclString(PyObject *self)
-{
-    return Tcl_GetString(((PyTclObject*)self)->value);
-}
-
 /* Like _str, but create Unicode if necessary. */
 PyDoc_STRVAR(PyTclObject_string__doc__,
 "the string representation of this object, either as str or bytes");
@@ -1048,53 +1083,51 @@ AsObj(PyObject *value)
     }
 
     if (PyUnicode_Check(value)) {
-        void *inbuf;
-        Py_ssize_t size;
-        int kind;
-        Tcl_UniChar *outbuf = NULL;
-        Py_ssize_t i;
-        size_t allocsize;
-
         if (PyUnicode_READY(value) == -1)
             return NULL;
 
-        inbuf = PyUnicode_DATA(value);
-        size = PyUnicode_GET_LENGTH(value);
-        if (size == 0)
-            return Tcl_NewUnicodeObj((const void *)"", 0);
+        Py_ssize_t size = PyUnicode_GET_LENGTH(value);
+        if (size == 0) {
+            return Tcl_NewStringObj("", 0);
+        }
         if (!CHECK_SIZE(size, sizeof(Tcl_UniChar))) {
             PyErr_SetString(PyExc_OverflowError, "string is too long");
             return NULL;
         }
-        kind = PyUnicode_KIND(value);
-        if (kind == sizeof(Tcl_UniChar))
-            return Tcl_NewUnicodeObj(inbuf, (int)size);
-        allocsize = ((size_t)size) * sizeof(Tcl_UniChar);
-        outbuf = (Tcl_UniChar*)PyMem_Malloc(allocsize);
-        /* Else overflow occurred, and we take the next exit */
-        if (!outbuf) {
-            PyErr_NoMemory();
-            return NULL;
+        if (PyUnicode_IS_ASCII(value)) {
+            return Tcl_NewStringObj((const char *)PyUnicode_DATA(value),
+                                    (int)size);
         }
-        for (i = 0; i < size; i++) {
-            Py_UCS4 ch = PyUnicode_READ(kind, inbuf, i);
-            /* We cannot test for sizeof(Tcl_UniChar) directly,
-               so we test for UTF-8 size instead. */
-#if TCL_UTF_MAX == 3
-            if (ch >= 0x10000) {
-                /* Tcl doesn't do UTF-16, yet. */
-                PyErr_Format(Tkinter_TclError,
-                             "character U+%x is above the range "
-                             "(U+0000-U+FFFF) allowed by Tcl",
-                             ch);
-                PyMem_Free(outbuf);
-                return NULL;
-            }
+
+        PyObject *encoded;
+#if USE_TCL_UNICODE
+        if (sizeof(Tcl_UniChar) == 2)
+            encoded = _PyUnicode_EncodeUTF16(value,
+                    "surrogatepass", NATIVE_BYTEORDER);
+        else if (sizeof(Tcl_UniChar) == 4)
+            encoded = _PyUnicode_EncodeUTF32(value,
+                    "surrogatepass", NATIVE_BYTEORDER);
+        else
+            Py_UNREACHABLE();
+#else
+        encoded = _PyUnicode_AsUTF8String(value, "surrogateescape");
 #endif
-            outbuf[i] = ch;
+        if (!encoded) {
+            return NULL;
         }
-        result = Tcl_NewUnicodeObj(outbuf, (int)size);
-        PyMem_Free(outbuf);
+        size = PyBytes_GET_SIZE(encoded);
+        if (size > INT_MAX) {
+            Py_DECREF(encoded);
+            PyErr_SetString(PyExc_OverflowError, "string is too long");
+            return NULL;
+        }
+#if USE_TCL_UNICODE
+        result = Tcl_NewUnicodeObj((const Tcl_UniChar *)PyBytes_AS_STRING(encoded),
+                                   (int)(size / sizeof(Tcl_UniChar)));
+#else
+        result = Tcl_NewStringObj(PyBytes_AS_STRING(encoded), (int)size);
+#endif
+        Py_DECREF(encoded);
         return result;
     }
 
@@ -1113,7 +1146,7 @@ AsObj(PyObject *value)
 }
 
 static PyObject *
-fromBoolean(PyObject* tkapp, Tcl_Obj *value)
+fromBoolean(TkappObject *tkapp, Tcl_Obj *value)
 {
     int boolValue;
     if (Tcl_GetBooleanFromObj(Tkapp_Interp(tkapp), value, &boolValue) == TCL_ERROR)
@@ -1122,7 +1155,7 @@ fromBoolean(PyObject* tkapp, Tcl_Obj *value)
 }
 
 static PyObject*
-fromWideIntObj(PyObject* tkapp, Tcl_Obj *value)
+fromWideIntObj(TkappObject *tkapp, Tcl_Obj *value)
 {
         Tcl_WideInt wideValue;
         if (Tcl_GetWideIntFromObj(Tkapp_Interp(tkapp), value, &wideValue) == TCL_OK) {
@@ -1138,7 +1171,7 @@ fromWideIntObj(PyObject* tkapp, Tcl_Obj *value)
 
 #ifdef HAVE_LIBTOMMAMTH
 static PyObject*
-fromBignumObj(PyObject* tkapp, Tcl_Obj *value)
+fromBignumObj(TkappObject *tkapp, Tcl_Obj *value)
 {
     mp_int bigValue;
     unsigned long numBytes;
@@ -1174,32 +1207,31 @@ fromBignumObj(PyObject* tkapp, Tcl_Obj *value)
 #endif
 
 static PyObject*
-FromObj(PyObject* tkapp, Tcl_Obj *value)
+FromObj(TkappObject *tkapp, Tcl_Obj *value)
 {
     PyObject *result = NULL;
-    TkappObject *app = (TkappObject*)tkapp;
     Tcl_Interp *interp = Tkapp_Interp(tkapp);
 
     if (value->typePtr == NULL) {
-        return unicodeFromTclStringAndSize(value->bytes, value->length);
+        return unicodeFromTclObj(value);
     }
 
-    if (value->typePtr == app->BooleanType ||
-        value->typePtr == app->OldBooleanType) {
+    if (value->typePtr == tkapp->BooleanType ||
+        value->typePtr == tkapp->OldBooleanType) {
         return fromBoolean(tkapp, value);
     }
 
-    if (value->typePtr == app->ByteArrayType) {
+    if (value->typePtr == tkapp->ByteArrayType) {
         int size;
         char *data = (char*)Tcl_GetByteArrayFromObj(value, &size);
         return PyBytes_FromStringAndSize(data, size);
     }
 
-    if (value->typePtr == app->DoubleType) {
+    if (value->typePtr == tkapp->DoubleType) {
         return PyFloat_FromDouble(value->internalRep.doubleValue);
     }
 
-    if (value->typePtr == app->IntType) {
+    if (value->typePtr == tkapp->IntType) {
         long longValue;
         if (Tcl_GetLongFromObj(interp, value, &longValue) == TCL_OK)
             return PyLong_FromLong(longValue);
@@ -1207,8 +1239,8 @@ FromObj(PyObject* tkapp, Tcl_Obj *value)
            fall through to wideInt handling. */
     }
 
-    if (value->typePtr == app->IntType ||
-        value->typePtr == app->WideIntType) {
+    if (value->typePtr == tkapp->IntType ||
+        value->typePtr == tkapp->WideIntType) {
         result = fromWideIntObj(tkapp, value);
         if (result != NULL || PyErr_Occurred())
             return result;
@@ -1218,14 +1250,14 @@ FromObj(PyObject* tkapp, Tcl_Obj *value)
     }
 
 #ifdef HAVE_LIBTOMMAMTH
-    if (value->typePtr == app->IntType ||
-        value->typePtr == app->WideIntType ||
-        value->typePtr == app->BignumType) {
+    if (value->typePtr == tkapp->IntType ||
+        value->typePtr == tkapp->WideIntType ||
+        value->typePtr == tkapp->BignumType) {
         return fromBignumObj(tkapp, value);
     }
 #endif
 
-    if (value->typePtr == app->ListType) {
+    if (value->typePtr == tkapp->ListType) {
         int size;
         int i, status;
         PyObject *elem;
@@ -1253,30 +1285,28 @@ FromObj(PyObject* tkapp, Tcl_Obj *value)
         return result;
     }
 
-    if (value->typePtr == app->ProcBodyType) {
+    if (value->typePtr == tkapp->ProcBodyType) {
       /* fall through: return tcl object. */
     }
 
-    if (value->typePtr == app->StringType) {
-        return PyUnicode_FromKindAndData(
-            sizeof(Tcl_UniChar), Tcl_GetUnicode(value),
-            Tcl_GetCharLength(value));
+    if (value->typePtr == tkapp->StringType) {
+        return unicodeFromTclObj(value);
     }
 
 #if TK_HEX_VERSION >= 0x08050000
-    if (app->BooleanType == NULL &&
+    if (tkapp->BooleanType == NULL &&
         strcmp(value->typePtr->name, "booleanString") == 0) {
         /* booleanString type is not registered in Tcl */
-        app->BooleanType = value->typePtr;
+        tkapp->BooleanType = value->typePtr;
         return fromBoolean(tkapp, value);
     }
 #endif
 
 #ifdef HAVE_LIBTOMMAMTH
-    if (app->BignumType == NULL &&
+    if (tkapp->BignumType == NULL &&
         strcmp(value->typePtr->name, "bignum") == 0) {
         /* bignum type is not registered in Tcl */
-        app->BignumType = value->typePtr;
+        tkapp->BignumType = value->typePtr;
         return fromBignumObj(tkapp, value);
     }
 #endif
@@ -1366,19 +1396,28 @@ Tkapp_CallArgs(PyObject *args, Tcl_Obj** objStore, int *pobjc)
     return NULL;
 }
 
+/* Convert the results of a command call into a Python string. */
+
+static PyObject *
+Tkapp_UnicodeResult(TkappObject *self)
+{
+    return unicodeFromTclObj(Tcl_GetObjResult(self->interp));
+}
+
+
 /* Convert the results of a command call into a Python objects. */
 
-static PyObject*
-Tkapp_CallResult(TkappObject *self)
+static PyObject *
+Tkapp_ObjectResult(TkappObject *self)
 {
     PyObject *res = NULL;
     Tcl_Obj *value = Tcl_GetObjResult(self->interp);
-    if(self->wantobjects) {
+    if (self->wantobjects) {
         /* Not sure whether the IncrRef is necessary, but something
            may overwrite the interpreter result while we are
            converting it. */
         Tcl_IncrRefCount(value);
-        res = FromObj((PyObject*)self, value);
+        res = FromObj(self, value);
         Tcl_DecrRefCount(value);
     } else {
         res = unicodeFromTclObj(value);
@@ -1410,15 +1449,13 @@ Tkapp_CallProc(Tkapp_CallEvent *e, int flags)
     i = Tcl_EvalObjv(e->self->interp, objc, objv, e->flags);
     ENTER_PYTHON
     if (i == TCL_ERROR) {
-        *(e->res) = NULL;
-        *(e->exc_type) = NULL;
-        *(e->exc_tb) = NULL;
-        *(e->exc_value) = PyObject_CallFunction(
-            Tkinter_TclError, "s",
-            Tcl_GetStringResult(e->self->interp));
+        *(e->res) = Tkinter_Error(e->self);
     }
     else {
-        *(e->res) = Tkapp_CallResult(e->self);
+        *(e->res) = Tkapp_ObjectResult(e->self);
+    }
+    if (*(e->res) == NULL) {
+        PyErr_Fetch(e->exc_type, e->exc_value, e->exc_tb);
     }
     LEAVE_PYTHON
 
@@ -1506,9 +1543,9 @@ Tkapp_Call(PyObject *selfptr, PyObject *args)
         ENTER_OVERLAP
 
         if (i == TCL_ERROR)
-            Tkinter_Error(selfptr);
+            Tkinter_Error(self);
         else
-            res = Tkapp_CallResult(self);
+            res = Tkapp_ObjectResult(self);
 
         LEAVE_OVERLAP_TCL
 
@@ -1540,9 +1577,9 @@ _tkinter_tkapp_eval_impl(TkappObject *self, const char *script)
     err = Tcl_Eval(Tkapp_Interp(self), script);
     ENTER_OVERLAP
     if (err == TCL_ERROR)
-        res = Tkinter_Error((PyObject *)self);
+        res = Tkinter_Error(self);
     else
-        res = unicodeFromTclString(Tkapp_Result(self));
+        res = Tkapp_UnicodeResult(self);
     LEAVE_OVERLAP_TCL
     return res;
 }
@@ -1569,9 +1606,9 @@ _tkinter_tkapp_evalfile_impl(TkappObject *self, const char *fileName)
     err = Tcl_EvalFile(Tkapp_Interp(self), fileName);
     ENTER_OVERLAP
     if (err == TCL_ERROR)
-        res = Tkinter_Error((PyObject *)self);
+        res = Tkinter_Error(self);
     else
-        res = unicodeFromTclString(Tkapp_Result(self));
+        res = Tkapp_UnicodeResult(self);
     LEAVE_OVERLAP_TCL
     return res;
 }
@@ -1598,9 +1635,9 @@ _tkinter_tkapp_record_impl(TkappObject *self, const char *script)
     err = Tcl_RecordAndEval(Tkapp_Interp(self), script, TCL_NO_EVAL);
     ENTER_OVERLAP
     if (err == TCL_ERROR)
-        res = Tkinter_Error((PyObject *)self);
+        res = Tkinter_Error(self);
     else
-        res = unicodeFromTclString(Tkapp_Result(self));
+        res = Tkapp_UnicodeResult(self);
     LEAVE_OVERLAP_TCL
     return res;
 }
@@ -1631,13 +1668,13 @@ _tkinter_tkapp_adderrorinfo_impl(TkappObject *self, const char *msg)
 
 /** Tcl Variable **/
 
-typedef PyObject* (*EventFunc)(PyObject*, PyObject *args, int flags);
+typedef PyObject* (*EventFunc)(TkappObject *, PyObject *, int);
 
 TCL_DECLARE_MUTEX(var_mutex)
 
 typedef struct VarEvent {
     Tcl_Event ev; /* must be first */
-    PyObject *self;
+    TkappObject *self;
     PyObject *args;
     int flags;
     EventFunc func;
@@ -1692,7 +1729,7 @@ varname_converter(PyObject *in, void *_out)
         return 1;
     }
     if (PyTclObject_Check(in)) {
-        *out = PyTclObject_TclString(in);
+        *out = Tcl_GetString(((PyTclObject *)in)->value);
         return 1;
     }
     PyErr_Format(PyExc_TypeError,
@@ -1750,7 +1787,7 @@ var_invoke(EventFunc func, PyObject *selfptr, PyObject *args, int flags)
             PyErr_NoMemory();
             return NULL;
         }
-        ev->self = selfptr;
+        ev->self = self;
         ev->args = args;
         ev->flags = flags;
         ev->func = func;
@@ -1770,11 +1807,11 @@ var_invoke(EventFunc func, PyObject *selfptr, PyObject *args, int flags)
         return res;
     }
     /* Tcl is not threaded, or this is the interpreter thread. */
-    return func(selfptr, args, flags);
+    return func(self, args, flags);
 }
 
 static PyObject *
-SetVar(PyObject *self, PyObject *args, int flags)
+SetVar(TkappObject *self, PyObject *args, int flags)
 {
     const char *name1, *name2;
     PyObject *newValue;
@@ -1843,7 +1880,7 @@ Tkapp_GlobalSetVar(PyObject *self, PyObject *args)
 
 
 static PyObject *
-GetVar(PyObject *self, PyObject *args, int flags)
+GetVar(TkappObject *self, PyObject *args, int flags)
 {
     const char *name1, *name2=NULL;
     PyObject *res = NULL;
@@ -1858,10 +1895,9 @@ GetVar(PyObject *self, PyObject *args, int flags)
     tres = Tcl_GetVar2Ex(Tkapp_Interp(self), name1, name2, flags);
     ENTER_OVERLAP
     if (tres == NULL) {
-        PyErr_SetString(Tkinter_TclError,
-                        Tcl_GetStringResult(Tkapp_Interp(self)));
+        Tkinter_Error(self);
     } else {
-        if (((TkappObject*)self)->wantobjects) {
+        if (self->wantobjects) {
             res = FromObj(self, tres);
         }
         else {
@@ -1887,7 +1923,7 @@ Tkapp_GlobalGetVar(PyObject *self, PyObject *args)
 
 
 static PyObject *
-UnsetVar(PyObject *self, PyObject *args, int flags)
+UnsetVar(TkappObject *self, PyObject *args, int flags)
 {
     char *name1, *name2=NULL;
     int code;
@@ -1959,7 +1995,7 @@ _tkinter_tkapp_getint(TkappObject *self, PyObject *arg)
         CHECK_STRING_LENGTH(s);
         value = Tcl_NewStringObj(s, -1);
         if (value == NULL)
-            return Tkinter_Error((PyObject *)self);
+            return Tkinter_Error(self);
     }
     /* Don't use Tcl_GetInt() because it returns ambiguous result for value
        in ranges -2**32..-2**31-1 and 2**31..2**32-1 (on 32-bit platform).
@@ -1968,14 +2004,14 @@ _tkinter_tkapp_getint(TkappObject *self, PyObject *arg)
        value in ranges -2**64..-2**63-1 and 2**63..2**64-1 (on 32-bit platform).
      */
 #ifdef HAVE_LIBTOMMAMTH
-    result = fromBignumObj((PyObject *)self, value);
+    result = fromBignumObj(self, value);
 #else
-    result = fromWideIntObj((PyObject *)self, value);
+    result = fromWideIntObj(self, value);
 #endif
     Tcl_DecrRefCount(value);
     if (result != NULL || PyErr_Occurred())
         return result;
-    return Tkinter_Error((PyObject *)self);
+    return Tkinter_Error(self);
 }
 
 /*[clinic input]
@@ -2006,7 +2042,7 @@ _tkinter_tkapp_getdouble(TkappObject *self, PyObject *arg)
         if (Tcl_GetDoubleFromObj(Tkapp_Interp(self),
                                  ((PyTclObject*)arg)->value,
                                  &v) == TCL_ERROR)
-            return Tkinter_Error((PyObject *)self);
+            return Tkinter_Error(self);
         return PyFloat_FromDouble(v);
     }
 
@@ -2014,7 +2050,7 @@ _tkinter_tkapp_getdouble(TkappObject *self, PyObject *arg)
         return NULL;
     CHECK_STRING_LENGTH(s);
     if (Tcl_GetDouble(Tkapp_Interp(self), s, &v) == TCL_ERROR)
-        return Tkinter_Error((PyObject *)self);
+        return Tkinter_Error(self);
     return PyFloat_FromDouble(v);
 }
 
@@ -2041,7 +2077,7 @@ _tkinter_tkapp_getboolean(TkappObject *self, PyObject *arg)
         if (Tcl_GetBooleanFromObj(Tkapp_Interp(self),
                                   ((PyTclObject*)arg)->value,
                                   &v) == TCL_ERROR)
-            return Tkinter_Error((PyObject *)self);
+            return Tkinter_Error(self);
         return PyBool_FromLong(v);
     }
 
@@ -2049,7 +2085,7 @@ _tkinter_tkapp_getboolean(TkappObject *self, PyObject *arg)
         return NULL;
     CHECK_STRING_LENGTH(s);
     if (Tcl_GetBoolean(Tkapp_Interp(self), s, &v) == TCL_ERROR)
-        return Tkinter_Error((PyObject *)self);
+        return Tkinter_Error(self);
     return PyBool_FromLong(v);
 }
 
@@ -2075,9 +2111,9 @@ _tkinter_tkapp_exprstring_impl(TkappObject *self, const char *s)
     retval = Tcl_ExprString(Tkapp_Interp(self), s);
     ENTER_OVERLAP
     if (retval == TCL_ERROR)
-        res = Tkinter_Error((PyObject *)self);
+        res = Tkinter_Error(self);
     else
-        res = unicodeFromTclString(Tkapp_Result(self));
+        res = Tkapp_UnicodeResult(self);
     LEAVE_OVERLAP_TCL
     return res;
 }
@@ -2105,7 +2141,7 @@ _tkinter_tkapp_exprlong_impl(TkappObject *self, const char *s)
     retval = Tcl_ExprLong(Tkapp_Interp(self), s, &v);
     ENTER_OVERLAP
     if (retval == TCL_ERROR)
-        res = Tkinter_Error((PyObject *)self);
+        res = Tkinter_Error(self);
     else
         res = PyLong_FromLong(v);
     LEAVE_OVERLAP_TCL
@@ -2136,7 +2172,7 @@ _tkinter_tkapp_exprdouble_impl(TkappObject *self, const char *s)
     ENTER_OVERLAP
     PyFPE_END_PROTECT(retval)
     if (retval == TCL_ERROR)
-        res = Tkinter_Error((PyObject *)self);
+        res = Tkinter_Error(self);
     else
         res = PyFloat_FromDouble(v);
     LEAVE_OVERLAP_TCL
@@ -2165,7 +2201,7 @@ _tkinter_tkapp_exprboolean_impl(TkappObject *self, const char *s)
     retval = Tcl_ExprBoolean(Tkapp_Interp(self), s, &v);
     ENTER_OVERLAP
     if (retval == TCL_ERROR)
-        res = Tkinter_Error((PyObject *)self);
+        res = Tkinter_Error(self);
     else
         res = PyLong_FromLong(v);
     LEAVE_OVERLAP_TCL
@@ -2198,12 +2234,12 @@ _tkinter_tkapp_splitlist(TkappObject *self, PyObject *arg)
         if (Tcl_ListObjGetElements(Tkapp_Interp(self),
                                    ((PyTclObject*)arg)->value,
                                    &objc, &objv) == TCL_ERROR) {
-            return Tkinter_Error((PyObject *)self);
+            return Tkinter_Error(self);
         }
         if (!(v = PyTuple_New(objc)))
             return NULL;
         for (i = 0; i < objc; i++) {
-            PyObject *s = FromObj((PyObject*)self, objv[i]);
+            PyObject *s = FromObj(self, objv[i]);
             if (!s) {
                 Py_DECREF(v);
                 return NULL;
@@ -2231,7 +2267,7 @@ _tkinter_tkapp_splitlist(TkappObject *self, PyObject *arg)
     if (Tcl_SplitList(Tkapp_Interp(self), list,
                       &argc, &argv) == TCL_ERROR)  {
         PyMem_Free(list);
-        return Tkinter_Error((PyObject *)self);
+        return Tkinter_Error(self);
     }
 
     if (!(v = PyTuple_New(argc)))
@@ -2275,16 +2311,16 @@ _tkinter_tkapp_split(TkappObject *self, PyObject *arg)
         int i;
         if (Tcl_ListObjGetElements(Tkapp_Interp(self), value,
                                    &objc, &objv) == TCL_ERROR) {
-            return FromObj((PyObject*)self, value);
+            return FromObj(self, value);
         }
         if (objc == 0)
             return PyUnicode_FromString("");
         if (objc == 1)
-            return FromObj((PyObject*)self, objv[0]);
+            return FromObj(self, objv[0]);
         if (!(v = PyTuple_New(objc)))
             return NULL;
         for (i = 0; i < objc; i++) {
-            PyObject *s = FromObj((PyObject*)self, objv[i]);
+            PyObject *s = FromObj(self, objv[i]);
             if (!s) {
                 Py_DECREF(v);
                 return NULL;
@@ -2331,34 +2367,31 @@ PythonCmd_Error(Tcl_Interp *interp)
  * function or method.
  */
 static int
-PythonCmd(ClientData clientData, Tcl_Interp *interp, int argc, const char *argv[])
+PythonCmd(ClientData clientData, Tcl_Interp *interp,
+          int objc, Tcl_Obj *const objv[])
 {
     PythonCmd_ClientData *data = (PythonCmd_ClientData *)clientData;
-    PyObject *func, *arg, *res;
-    int i, rv;
+    PyObject *args, *res;
+    int i;
     Tcl_Obj *obj_res;
 
     ENTER_PYTHON
 
-    /* TBD: no error checking here since we know, via the
-     * Tkapp_CreateCommand() that the client data is a two-tuple
-     */
-    func = data->func;
-
-    /* Create argument list (argv1, ..., argvN) */
-    if (!(arg = PyTuple_New(argc - 1)))
+    /* Create argument tuple (objv1, ..., objvN) */
+    if (!(args = PyTuple_New(objc - 1)))
         return PythonCmd_Error(interp);
 
-    for (i = 0; i < (argc - 1); i++) {
-        PyObject *s = unicodeFromTclString(argv[i + 1]);
+    for (i = 0; i < (objc - 1); i++) {
+        PyObject *s = unicodeFromTclObj(objv[i + 1]);
         if (!s) {
-            Py_DECREF(arg);
+            Py_DECREF(args);
             return PythonCmd_Error(interp);
         }
-        PyTuple_SET_ITEM(arg, i, s);
+        PyTuple_SET_ITEM(args, i, s);
     }
-    res = PyObject_Call(func, arg, NULL);
-    Py_DECREF(arg);
+
+    res = PyObject_Call(data->func, args, NULL);
+    Py_DECREF(args);
 
     if (res == NULL)
         return PythonCmd_Error(interp);
@@ -2368,18 +2401,15 @@ PythonCmd(ClientData clientData, Tcl_Interp *interp, int argc, const char *argv[
         Py_DECREF(res);
         return PythonCmd_Error(interp);
     }
-    else {
-        Tcl_SetObjResult(interp, obj_res);
-        rv = TCL_OK;
-    }
-
+    Tcl_SetObjResult(interp, obj_res);
     Py_DECREF(res);
 
     LEAVE_PYTHON
 
-    return rv;
+    return TCL_OK;
 }
 
+
 static void
 PythonCmdDelete(ClientData clientData)
 {
@@ -2411,7 +2441,7 @@ static int
 Tkapp_CommandProc(CommandEvent *ev, int flags)
 {
     if (ev->create)
-        *ev->status = Tcl_CreateCommand(
+        *ev->status = Tcl_CreateObjCommand(
             ev->interp, ev->name, PythonCmd,
             ev->data, PythonCmdDelete) == NULL;
     else
@@ -2477,7 +2507,7 @@ _tkinter_tkapp_createcommand_impl(TkappObject *self, const char *name,
     else
     {
         ENTER_TCL
-        err = Tcl_CreateCommand(
+        err = Tcl_CreateObjCommand(
             Tkapp_Interp(self), name, PythonCmd,
             (ClientData)data, PythonCmdDelete) == NULL;
         LEAVE_TCL
@@ -2953,9 +2983,9 @@ _tkinter_tkapp_loadtk_impl(TkappObject *self)
     if (err == TCL_ERROR) {
         /* This sets an exception, but we cannot return right
            away because we need to exit the overlap first. */
-        Tkinter_Error((PyObject *)self);
+        Tkinter_Error(self);
     } else {
-        _tk_exists = Tkapp_Result(self);
+        _tk_exists = Tcl_GetStringResult(Tkapp_Interp(self));
     }
     LEAVE_OVERLAP_TCL
     if (err == TCL_ERROR) {
@@ -2963,8 +2993,7 @@ _tkinter_tkapp_loadtk_impl(TkappObject *self)
     }
     if (_tk_exists == NULL || strcmp(_tk_exists, "1") != 0)     {
         if (Tk_Init(interp)             == TCL_ERROR) {
-            PyErr_SetString(Tkinter_TclError,
-                            Tcl_GetStringResult(Tkapp_Interp(self)));
+            Tkinter_Error(self);
 #ifdef TKINTER_PROTECT_LOADTK
             tk_load_failed = 1;
 #endif


From webhook-mailer at python.org  Fri Oct  4 07:35:54 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Fri, 04 Oct 2019 11:35:54 -0000
Subject: [Python-checkins] bpo-38266: Revert bpo-37878: Make
 PyThreadState_DeleteCurrent() Internal (GH-16558)
Message-ID: <mailman.216.1570188956.32504.python-checkins@python.org>

https://github.com/python/cpython/commit/8855e47d09d4db1206c65b24efc8ad0df585ac46
commit: 8855e47d09d4db1206c65b24efc8ad0df585ac46
branch: master
author: Joannah Nanjekye <33177550+nanjekyejoannah at users.noreply.github.com>
committer: Victor Stinner <vstinner at python.org>
date: 2019-10-04T13:35:42+02:00
summary:

bpo-38266: Revert bpo-37878: Make PyThreadState_DeleteCurrent() Internal (GH-16558)

Revert the removal of PyThreadState_DeleteCurrent() with documentation.

files:
A Misc/NEWS.d/next/C API/2019-10-03-12-53-53.bpo-38266.0FIC1q.rst
M Doc/c-api/init.rst
M Include/cpython/pystate.h
M Python/pystate.c

diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst
index 0b7a84d031532..5456ad446cf23 100644
--- a/Doc/c-api/init.rst
+++ b/Doc/c-api/init.rst
@@ -1035,6 +1035,14 @@ All of the following functions must be called after :c:func:`Py_Initialize`.
    :c:func:`PyThreadState_Clear`.
 
 
+   .. c:function:: void PyThreadState_DeleteCurrent()
+
+    Destroy the current thread state and release the global interpreter lock.
+    Like :c:func:`PyThreadState_Delete`, the global interpreter lock need not
+    be held. The thread state must have been reset with a previous call
+    to :c:func:`PyThreadState_Clear`.
+
+
 .. c:function:: PY_INT64_T PyInterpreterState_GetID(PyInterpreterState *interp)
 
    Return the interpreter's unique ID.  If there was any error in doing
diff --git a/Include/cpython/pystate.h b/Include/cpython/pystate.h
index f872351780367..6c8d2ae041ea5 100644
--- a/Include/cpython/pystate.h
+++ b/Include/cpython/pystate.h
@@ -183,6 +183,7 @@ PyAPI_FUNC(PyInterpreterState *) PyInterpreterState_Head(void);
 PyAPI_FUNC(PyInterpreterState *) PyInterpreterState_Next(PyInterpreterState *);
 PyAPI_FUNC(PyThreadState *) PyInterpreterState_ThreadHead(PyInterpreterState *);
 PyAPI_FUNC(PyThreadState *) PyThreadState_Next(PyThreadState *);
+PyAPI_FUNC(void) PyThreadState_DeleteCurrent(void);
 
 typedef struct _frame *(*PyThreadFrameGetter)(PyThreadState *self_);
 
diff --git a/Misc/NEWS.d/next/C API/2019-10-03-12-53-53.bpo-38266.0FIC1q.rst b/Misc/NEWS.d/next/C API/2019-10-03-12-53-53.bpo-38266.0FIC1q.rst
new file mode 100644
index 0000000000000..3b4c4232eb897
--- /dev/null
+++ b/Misc/NEWS.d/next/C API/2019-10-03-12-53-53.bpo-38266.0FIC1q.rst	
@@ -0,0 +1 @@
+Revert the removal of PyThreadState_DeleteCurrent() with documentation.
\ No newline at end of file
diff --git a/Python/pystate.c b/Python/pystate.c
index f3d89e7d2b256..b4b12472286a3 100644
--- a/Python/pystate.c
+++ b/Python/pystate.c
@@ -801,7 +801,7 @@ PyThreadState_Clear(PyThreadState *tstate)
 }
 
 
-/* Common code for PyThreadState_Delete() and _PyThreadState_DeleteCurrent() */
+/* Common code for PyThreadState_Delete() and PyThreadState_DeleteCurrent() */
 static void
 tstate_delete_common(_PyRuntimeState *runtime, PyThreadState *tstate)
 {
@@ -857,7 +857,7 @@ _PyThreadState_DeleteCurrent(_PyRuntimeState *runtime)
     PyThreadState *tstate = _PyRuntimeGILState_GetThreadState(gilstate);
     if (tstate == NULL)
         Py_FatalError(
-            "_PyThreadState_DeleteCurrent: no current tstate");
+            "PyThreadState_DeleteCurrent: no current tstate");
     tstate_delete_common(runtime, tstate);
     if (gilstate->autoInterpreterState &&
         PyThread_tss_get(&gilstate->autoTSSkey) == tstate)
@@ -868,6 +868,12 @@ _PyThreadState_DeleteCurrent(_PyRuntimeState *runtime)
     PyEval_ReleaseLock();
 }
 
+void
+PyThreadState_DeleteCurrent(void)
+{
+    _PyThreadState_DeleteCurrent(&_PyRuntime);
+}
+
 
 /*
  * Delete all thread states except the one passed as argument.


From webhook-mailer at python.org  Fri Oct  4 09:26:03 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Fri, 04 Oct 2019 13:26:03 -0000
Subject: [Python-checkins] bpo-38353: getpath.c uses dynamically allocated
 strings (GH-16582)
Message-ID: <mailman.217.1570195565.32504.python-checkins@python.org>

https://github.com/python/cpython/commit/abd7cd856ba326bd7574135c7d034e98492ab695
commit: abd7cd856ba326bd7574135c7d034e98492ab695
branch: master
author: Victor Stinner <vstinner at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-04T15:25:50+02:00
summary:

bpo-38353: getpath.c uses dynamically allocated strings (GH-16582)

* PyCalculatePath: add "_macro" to the 3 fields storing macro values
* Add dynamically allocated strings to PyCalculatePath

files:
M Modules/getpath.c

diff --git a/Modules/getpath.c b/Modules/getpath.c
index 1349d5c3f251b..5f35ca6e319d8 100644
--- a/Modules/getpath.c
+++ b/Modules/getpath.c
@@ -123,9 +123,9 @@ extern "C" {
 typedef struct {
     wchar_t *path_env;                 /* PATH environment variable */
 
-    wchar_t *pythonpath;               /* PYTHONPATH macro */
-    wchar_t *prefix;                   /* PREFIX macro */
-    wchar_t *exec_prefix;              /* EXEC_PREFIX macro */
+    wchar_t *pythonpath_macro;         /* PYTHONPATH macro */
+    wchar_t *prefix_macro;             /* PREFIX macro */
+    wchar_t *exec_prefix_macro;        /* EXEC_PREFIX macro */
 
     wchar_t *lib_python;               /* "lib/pythonX.Y" */
 
@@ -134,6 +134,11 @@ typedef struct {
 
     int warnings;
     const wchar_t *pythonpath_env;
+
+    wchar_t *argv0_path;
+    wchar_t *zip_path;
+    wchar_t *prefix;
+    wchar_t *exec_prefix;
 } PyCalculatePath;
 
 static const wchar_t delimiter[2] = {DELIM, '\0'};
@@ -382,7 +387,6 @@ add_exe_suffix(wchar_t *path, size_t pathlen)
 */
 static PyStatus
 search_for_prefix(PyCalculatePath *calculate, _PyPathConfig *pathconfig,
-                  const wchar_t *argv0_path,
                   wchar_t *prefix, size_t prefix_len, int *found)
 {
     wchar_t path[MAXPATHLEN+1];
@@ -410,7 +414,7 @@ search_for_prefix(PyCalculatePath *calculate, _PyPathConfig *pathconfig,
     }
 
     /* Check to see if argv[0] is in the build directory */
-    if (safe_wcscpy(path, argv0_path, path_len) < 0) {
+    if (safe_wcscpy(path, calculate->argv0_path, path_len) < 0) {
         return PATHLEN_ERR();
     }
     status = joinpath(path, L"Modules/Setup.local", path_len);
@@ -424,7 +428,7 @@ search_for_prefix(PyCalculatePath *calculate, _PyPathConfig *pathconfig,
         wchar_t *vpath = Py_DecodeLocale(VPATH, NULL);
         if (vpath != NULL) {
             /* Path: <argv0_path> / <vpath> / Lib / LANDMARK */
-            if (safe_wcscpy(prefix, argv0_path, prefix_len) < 0) {
+            if (safe_wcscpy(prefix, calculate->argv0_path, prefix_len) < 0) {
                 return PATHLEN_ERR();
             }
             status = joinpath(prefix, vpath, prefix_len);
@@ -451,7 +455,7 @@ search_for_prefix(PyCalculatePath *calculate, _PyPathConfig *pathconfig,
     }
 
     /* Search from argv0_path, until root is found */
-    status = copy_absolute(prefix, argv0_path, prefix_len);
+    status = copy_absolute(prefix, calculate->argv0_path, prefix_len);
     if (_PyStatus_EXCEPTION(status)) {
         return status;
     }
@@ -479,7 +483,7 @@ search_for_prefix(PyCalculatePath *calculate, _PyPathConfig *pathconfig,
 
     /* Look at configure's PREFIX.
        Path: <PREFIX macro> / <lib_python> / LANDMARK */
-    if (safe_wcscpy(prefix, calculate->prefix, prefix_len) < 0) {
+    if (safe_wcscpy(prefix, calculate->prefix_macro, prefix_len) < 0) {
         return PATHLEN_ERR();
     }
     status = joinpath(prefix, calculate->lib_python, prefix_len);
@@ -504,13 +508,14 @@ search_for_prefix(PyCalculatePath *calculate, _PyPathConfig *pathconfig,
 
 
 static PyStatus
-calculate_prefix(PyCalculatePath *calculate, _PyPathConfig *pathconfig,
-                 const wchar_t *argv0_path,
-                 wchar_t *prefix, size_t prefix_len)
+calculate_prefix(PyCalculatePath *calculate, _PyPathConfig *pathconfig)
 {
-    PyStatus status;
+    wchar_t prefix[MAXPATHLEN+1];
+    memset(prefix, 0, sizeof(prefix));
+    size_t prefix_len = Py_ARRAY_LENGTH(prefix);
 
-    status = search_for_prefix(calculate, pathconfig, argv0_path,
+    PyStatus status;
+    status = search_for_prefix(calculate, pathconfig,
                                prefix, prefix_len,
                                &calculate->prefix_found);
     if (_PyStatus_EXCEPTION(status)) {
@@ -522,7 +527,7 @@ calculate_prefix(PyCalculatePath *calculate, _PyPathConfig *pathconfig,
             fprintf(stderr,
                 "Could not find platform independent libraries <prefix>\n");
         }
-        if (safe_wcscpy(prefix, calculate->prefix, prefix_len) < 0) {
+        if (safe_wcscpy(prefix, calculate->prefix_macro, prefix_len) < 0) {
             return PATHLEN_ERR();
         }
         status = joinpath(prefix, calculate->lib_python, prefix_len);
@@ -530,13 +535,17 @@ calculate_prefix(PyCalculatePath *calculate, _PyPathConfig *pathconfig,
             return status;
         }
     }
+
+    calculate->prefix = _PyMem_RawWcsdup(prefix);
+    if (calculate->prefix == NULL) {
+        return _PyStatus_NO_MEMORY();
+    }
     return _PyStatus_OK();
 }
 
 
 static PyStatus
-calculate_set_prefix(PyCalculatePath *calculate, _PyPathConfig *pathconfig,
-                     wchar_t *prefix)
+calculate_set_prefix(PyCalculatePath *calculate, _PyPathConfig *pathconfig)
 {
     /* Reduce prefix and exec_prefix to their essence,
      * e.g. /usr/local/lib/python1.5 is reduced to /usr/local.
@@ -544,6 +553,11 @@ calculate_set_prefix(PyCalculatePath *calculate, _PyPathConfig *pathconfig,
      * return the compiled-in defaults instead.
      */
     if (calculate->prefix_found > 0) {
+        wchar_t prefix[MAXPATHLEN+1];
+        memset(prefix, 0, sizeof(prefix));
+
+        wcscpy(prefix, calculate->prefix);
+
         reduce(prefix);
         reduce(prefix);
         /* The prefix is the root directory, but reduce() chopped
@@ -554,7 +568,7 @@ calculate_set_prefix(PyCalculatePath *calculate, _PyPathConfig *pathconfig,
         pathconfig->prefix = _PyMem_RawWcsdup(prefix);
     }
     else {
-        pathconfig->prefix = _PyMem_RawWcsdup(calculate->prefix);
+        pathconfig->prefix = _PyMem_RawWcsdup(calculate->prefix_macro);
     }
 
     if (pathconfig->prefix == NULL) {
@@ -630,7 +644,6 @@ calculate_pybuilddir(const wchar_t *argv0_path,
 */
 static PyStatus
 search_for_exec_prefix(PyCalculatePath *calculate, _PyPathConfig *pathconfig,
-                       const wchar_t *argv0_path,
                        wchar_t *exec_prefix, size_t exec_prefix_len,
                        int *found)
 {
@@ -664,8 +677,8 @@ search_for_exec_prefix(PyCalculatePath *calculate, _PyPathConfig *pathconfig,
 
     /* Check for pybuilddir.txt */
     assert(*found == 0);
-    status = calculate_pybuilddir(argv0_path, exec_prefix, exec_prefix_len,
-                                  found);
+    status = calculate_pybuilddir(calculate->argv0_path,
+                                  exec_prefix, exec_prefix_len, found);
     if (_PyStatus_EXCEPTION(status)) {
         return status;
     }
@@ -674,7 +687,7 @@ search_for_exec_prefix(PyCalculatePath *calculate, _PyPathConfig *pathconfig,
     }
 
     /* Search from argv0_path, until root is found */
-    status = copy_absolute(exec_prefix, argv0_path, exec_prefix_len);
+    status = copy_absolute(exec_prefix, calculate->argv0_path, exec_prefix_len);
     if (_PyStatus_EXCEPTION(status)) {
         return status;
     }
@@ -701,7 +714,7 @@ search_for_exec_prefix(PyCalculatePath *calculate, _PyPathConfig *pathconfig,
     /* Look at configure's EXEC_PREFIX.
 
        Path: <EXEC_PREFIX macro> / <lib_python> / "lib-dynload" */
-    if (safe_wcscpy(exec_prefix, calculate->exec_prefix, exec_prefix_len) < 0) {
+    if (safe_wcscpy(exec_prefix, calculate->exec_prefix_macro, exec_prefix_len) < 0) {
         return PATHLEN_ERR();
     }
     status = joinpath(exec_prefix, calculate->lib_python, exec_prefix_len);
@@ -724,13 +737,14 @@ search_for_exec_prefix(PyCalculatePath *calculate, _PyPathConfig *pathconfig,
 
 
 static PyStatus
-calculate_exec_prefix(PyCalculatePath *calculate, _PyPathConfig *pathconfig,
-                      const wchar_t *argv0_path,
-                      wchar_t *exec_prefix, size_t exec_prefix_len)
+calculate_exec_prefix(PyCalculatePath *calculate, _PyPathConfig *pathconfig)
 {
     PyStatus status;
+    wchar_t exec_prefix[MAXPATHLEN+1];
+    memset(exec_prefix, 0, sizeof(exec_prefix));
+    size_t exec_prefix_len = Py_ARRAY_LENGTH(exec_prefix);
 
-    status = search_for_exec_prefix(calculate, pathconfig, argv0_path,
+    status = search_for_exec_prefix(calculate, pathconfig,
                                     exec_prefix, exec_prefix_len,
                                     &calculate->exec_prefix_found);
     if (_PyStatus_EXCEPTION(status)) {
@@ -742,7 +756,7 @@ calculate_exec_prefix(PyCalculatePath *calculate, _PyPathConfig *pathconfig,
             fprintf(stderr,
                 "Could not find platform dependent libraries <exec_prefix>\n");
         }
-        if (safe_wcscpy(exec_prefix, calculate->exec_prefix, exec_prefix_len) < 0) {
+        if (safe_wcscpy(exec_prefix, calculate->exec_prefix_macro, exec_prefix_len) < 0) {
             return PATHLEN_ERR();
         }
         status = joinpath(exec_prefix, L"lib/lib-dynload", exec_prefix_len);
@@ -750,17 +764,26 @@ calculate_exec_prefix(PyCalculatePath *calculate, _PyPathConfig *pathconfig,
             return status;
         }
     }
+
     /* If we found EXEC_PREFIX do *not* reduce it!  (Yet.) */
+    calculate->exec_prefix = _PyMem_RawWcsdup(exec_prefix);
+    if (calculate->exec_prefix == NULL) {
+        return _PyStatus_NO_MEMORY();
+    }
     return _PyStatus_OK();
 }
 
 
 static PyStatus
 calculate_set_exec_prefix(PyCalculatePath *calculate,
-                          _PyPathConfig *pathconfig,
-                          wchar_t *exec_prefix)
+                          _PyPathConfig *pathconfig)
 {
     if (calculate->exec_prefix_found > 0) {
+        wchar_t exec_prefix[MAXPATHLEN+1];
+        memset(exec_prefix, 0, sizeof(exec_prefix));
+
+        wcscpy(exec_prefix, calculate->exec_prefix);
+
         reduce(exec_prefix);
         reduce(exec_prefix);
         reduce(exec_prefix);
@@ -771,7 +794,7 @@ calculate_set_exec_prefix(PyCalculatePath *calculate,
         pathconfig->exec_prefix = _PyMem_RawWcsdup(exec_prefix);
     }
     else {
-        pathconfig->exec_prefix = _PyMem_RawWcsdup(calculate->exec_prefix);
+        pathconfig->exec_prefix = _PyMem_RawWcsdup(calculate->exec_prefix_macro);
     }
 
     if (pathconfig->exec_prefix == NULL) {
@@ -1085,9 +1108,12 @@ calculate_argv0_path_framework(PyCalculatePath *calculate,
 
 static PyStatus
 calculate_argv0_path(PyCalculatePath *calculate,
-                     const wchar_t *program_full_path,
-                     wchar_t *argv0_path, size_t argv0_path_len)
+                     const wchar_t *program_full_path)
 {
+    wchar_t argv0_path[MAXPATHLEN+1];
+    memset(argv0_path, 0, sizeof(argv0_path));
+    size_t argv0_path_len = Py_ARRAY_LENGTH(argv0_path);
+
     if (safe_wcscpy(argv0_path, program_full_path, argv0_path_len) < 0) {
         return PATHLEN_ERR();
     }
@@ -1106,6 +1132,11 @@ calculate_argv0_path(PyCalculatePath *calculate,
     reduce(argv0_path);
     /* At this point, argv0_path is guaranteed to be less than
        MAXPATHLEN bytes long. */
+
+    calculate->argv0_path = _PyMem_RawWcsdup(argv0_path);
+    if (calculate->argv0_path == NULL) {
+        return _PyStatus_NO_MEMORY();
+    }
     return _PyStatus_OK();
 }
 
@@ -1113,10 +1144,10 @@ calculate_argv0_path(PyCalculatePath *calculate,
 /* Search for an "pyvenv.cfg" environment configuration file, first in the
    executable's directory and then in the parent directory.
    If found, open it for use when searching for prefixes.
-*/
+
+   Write the 'home' variable of pyvenv.cfg into calculate->argv0_path. */
 static PyStatus
-calculate_read_pyenv(PyCalculatePath *calculate,
-                     wchar_t *argv0_path, size_t argv0_path_len)
+calculate_read_pyenv(PyCalculatePath *calculate)
 {
     PyStatus status;
     const wchar_t *env_cfg = L"pyvenv.cfg";
@@ -1127,7 +1158,7 @@ calculate_read_pyenv(PyCalculatePath *calculate,
     memset(filename, 0, sizeof(filename));
 
     /* Filename: <argv0_path_len> / "pyvenv.cfg" */
-    if (safe_wcscpy(filename, argv0_path, filename_len) < 0) {
+    if (safe_wcscpy(filename, calculate->argv0_path, filename_len) < 0) {
         return PATHLEN_ERR();
     }
 
@@ -1160,10 +1191,12 @@ calculate_read_pyenv(PyCalculatePath *calculate,
 
     if (_Py_FindEnvConfigValue(env_file, L"home",
                                home, Py_ARRAY_LENGTH(home))) {
-        if (safe_wcscpy(argv0_path, home, argv0_path_len) < 0) {
+        PyMem_RawFree(calculate->argv0_path);
+        calculate->argv0_path = _PyMem_RawWcsdup(home);
+        if (calculate->argv0_path == NULL) {
             fclose(env_file);
-            return PATHLEN_ERR();
-        }
+            return _PyStatus_NO_MEMORY();
+         }
     }
     fclose(env_file);
     return _PyStatus_OK();
@@ -1171,21 +1204,24 @@ calculate_read_pyenv(PyCalculatePath *calculate,
 
 
 static PyStatus
-calculate_zip_path(PyCalculatePath *calculate, const wchar_t *prefix,
-                   wchar_t *zip_path, size_t zip_path_len)
+calculate_zip_path(PyCalculatePath *calculate)
 {
     PyStatus status;
 
+    wchar_t zip_path[MAXPATHLEN+1];    /* ".../lib/pythonXY.zip" */
+    memset(zip_path, 0, sizeof(zip_path));
+    size_t zip_path_len = Py_ARRAY_LENGTH(zip_path);
+
     if (calculate->prefix_found > 0) {
         /* Use the reduced prefix returned by Py_GetPrefix() */
-        if (safe_wcscpy(zip_path, prefix, zip_path_len) < 0) {
+        if (safe_wcscpy(zip_path, calculate->prefix, zip_path_len) < 0) {
             return PATHLEN_ERR();
         }
         reduce(zip_path);
         reduce(zip_path);
     }
     else {
-        if (safe_wcscpy(zip_path, calculate->prefix, zip_path_len) < 0) {
+        if (safe_wcscpy(zip_path, calculate->prefix_macro, zip_path_len) < 0) {
             return PATHLEN_ERR();
         }
     }
@@ -1198,16 +1234,18 @@ calculate_zip_path(PyCalculatePath *calculate, const wchar_t *prefix,
     size_t bufsz = wcslen(zip_path);
     zip_path[bufsz - 6] = VERSION[0];
     zip_path[bufsz - 5] = VERSION[2];
+
+    calculate->zip_path = _PyMem_RawWcsdup(zip_path);
+    if (calculate->zip_path == NULL) {
+        return _PyStatus_NO_MEMORY();
+    }
     return _PyStatus_OK();
 }
 
 
 static PyStatus
 calculate_module_search_path(PyCalculatePath *calculate,
-                             _PyPathConfig *pathconfig,
-                             const wchar_t *prefix,
-                             const wchar_t *exec_prefix,
-                             const wchar_t *zip_path)
+                             _PyPathConfig *pathconfig)
 {
     /* Calculate size of return buffer */
     size_t bufsz = 0;
@@ -1215,8 +1253,8 @@ calculate_module_search_path(PyCalculatePath *calculate,
         bufsz += wcslen(calculate->pythonpath_env) + 1;
     }
 
-    wchar_t *defpath = calculate->pythonpath;
-    size_t prefixsz = wcslen(prefix) + 1;
+    wchar_t *defpath = calculate->pythonpath_macro;
+    size_t prefixsz = wcslen(calculate->prefix) + 1;
     while (1) {
         wchar_t *delim = wcschr(defpath, DELIM);
 
@@ -1235,8 +1273,8 @@ calculate_module_search_path(PyCalculatePath *calculate,
         defpath = delim + 1;
     }
 
-    bufsz += wcslen(zip_path) + 1;
-    bufsz += wcslen(exec_prefix) + 1;
+    bufsz += wcslen(calculate->zip_path) + 1;
+    bufsz += wcslen(calculate->exec_prefix) + 1;
 
     /* Allocate the buffer */
     wchar_t *buf = PyMem_RawMalloc(bufsz * sizeof(wchar_t));
@@ -1252,19 +1290,19 @@ calculate_module_search_path(PyCalculatePath *calculate,
     }
 
     /* Next is the default zip path */
-    wcscat(buf, zip_path);
+    wcscat(buf, calculate->zip_path);
     wcscat(buf, delimiter);
 
     /* Next goes merge of compile-time $PYTHONPATH with
      * dynamically located prefix.
      */
-    defpath = calculate->pythonpath;
+    defpath = calculate->pythonpath_macro;
     while (1) {
         wchar_t *delim = wcschr(defpath, DELIM);
 
         if (!_Py_isabs(defpath)) {
-            wcscat(buf, prefix);
-            if (prefixsz >= 2 && prefix[prefixsz - 2] != SEP &&
+            wcscat(buf, calculate->prefix);
+            if (prefixsz >= 2 && calculate->prefix[prefixsz - 2] != SEP &&
                 defpath[0] != (delim ? DELIM : L'\0'))
             {
                 /* not empty */
@@ -1287,7 +1325,7 @@ calculate_module_search_path(PyCalculatePath *calculate,
     wcscat(buf, delimiter);
 
     /* Finally, on goes the directory for dynamic-load modules */
-    wcscat(buf, exec_prefix);
+    wcscat(buf, calculate->exec_prefix);
 
     pathconfig->module_search_path = buf;
     return _PyStatus_OK();
@@ -1306,17 +1344,17 @@ calculate_init(PyCalculatePath *calculate, const PyConfig *config)
         }
     }
 
-    calculate->pythonpath = Py_DecodeLocale(PYTHONPATH, &len);
-    if (!calculate->pythonpath) {
+    calculate->pythonpath_macro = Py_DecodeLocale(PYTHONPATH, &len);
+    if (!calculate->pythonpath_macro) {
         return DECODE_LOCALE_ERR("PYTHONPATH define", len);
     }
 
-    calculate->prefix = Py_DecodeLocale(PREFIX, &len);
-    if (!calculate->prefix) {
+    calculate->prefix_macro = Py_DecodeLocale(PREFIX, &len);
+    if (!calculate->prefix_macro) {
         return DECODE_LOCALE_ERR("PREFIX define", len);
     }
-    calculate->exec_prefix = Py_DecodeLocale(EXEC_PREFIX, &len);
-    if (!calculate->exec_prefix) {
+    calculate->exec_prefix_macro = Py_DecodeLocale(EXEC_PREFIX, &len);
+    if (!calculate->exec_prefix_macro) {
         return DECODE_LOCALE_ERR("EXEC_PREFIX define", len);
     }
     calculate->lib_python = Py_DecodeLocale("lib/python" VERSION, &len);
@@ -1334,11 +1372,15 @@ calculate_init(PyCalculatePath *calculate, const PyConfig *config)
 static void
 calculate_free(PyCalculatePath *calculate)
 {
-    PyMem_RawFree(calculate->pythonpath);
-    PyMem_RawFree(calculate->prefix);
-    PyMem_RawFree(calculate->exec_prefix);
+    PyMem_RawFree(calculate->pythonpath_macro);
+    PyMem_RawFree(calculate->prefix_macro);
+    PyMem_RawFree(calculate->exec_prefix_macro);
     PyMem_RawFree(calculate->lib_python);
     PyMem_RawFree(calculate->path_env);
+    PyMem_RawFree(calculate->zip_path);
+    PyMem_RawFree(calculate->argv0_path);
+    PyMem_RawFree(calculate->prefix);
+    PyMem_RawFree(calculate->exec_prefix);
 }
 
 
@@ -1354,45 +1396,29 @@ calculate_path(PyCalculatePath *calculate, _PyPathConfig *pathconfig)
         }
     }
 
-    wchar_t argv0_path[MAXPATHLEN+1];
-    memset(argv0_path, 0, sizeof(argv0_path));
-
-    status = calculate_argv0_path(calculate, pathconfig->program_full_path,
-                                  argv0_path, Py_ARRAY_LENGTH(argv0_path));
+    status = calculate_argv0_path(calculate, pathconfig->program_full_path);
     if (_PyStatus_EXCEPTION(status)) {
         return status;
     }
 
     /* If a pyvenv.cfg configure file is found,
        argv0_path is overriden with its 'home' variable. */
-    status = calculate_read_pyenv(calculate,
-                                  argv0_path, Py_ARRAY_LENGTH(argv0_path));
+    status = calculate_read_pyenv(calculate);
     if (_PyStatus_EXCEPTION(status)) {
         return status;
     }
 
-    wchar_t prefix[MAXPATHLEN+1];
-    memset(prefix, 0, sizeof(prefix));
-    status = calculate_prefix(calculate, pathconfig,
-                              argv0_path,
-                              prefix, Py_ARRAY_LENGTH(prefix));
+    status = calculate_prefix(calculate, pathconfig);
     if (_PyStatus_EXCEPTION(status)) {
         return status;
     }
 
-    wchar_t zip_path[MAXPATHLEN+1];    /* ".../lib/pythonXY.zip" */
-    memset(zip_path, 0, sizeof(zip_path));
-
-    status = calculate_zip_path(calculate, prefix,
-                                zip_path, Py_ARRAY_LENGTH(zip_path));
+    status = calculate_zip_path(calculate);
     if (_PyStatus_EXCEPTION(status)) {
         return status;
     }
 
-    wchar_t exec_prefix[MAXPATHLEN+1];
-    memset(exec_prefix, 0, sizeof(exec_prefix));
-    status = calculate_exec_prefix(calculate, pathconfig, argv0_path,
-                                   exec_prefix, Py_ARRAY_LENGTH(exec_prefix));
+    status = calculate_exec_prefix(calculate, pathconfig);
     if (_PyStatus_EXCEPTION(status)) {
         return status;
     }
@@ -1405,27 +1431,25 @@ calculate_path(PyCalculatePath *calculate, _PyPathConfig *pathconfig)
     }
 
     if (pathconfig->module_search_path == NULL) {
-        status = calculate_module_search_path(calculate, pathconfig,
-                                              prefix, exec_prefix, zip_path);
+        status = calculate_module_search_path(calculate, pathconfig);
         if (_PyStatus_EXCEPTION(status)) {
             return status;
         }
     }
 
     if (pathconfig->prefix == NULL) {
-        status = calculate_set_prefix(calculate, pathconfig, prefix);
+        status = calculate_set_prefix(calculate, pathconfig);
         if (_PyStatus_EXCEPTION(status)) {
             return status;
         }
     }
 
     if (pathconfig->exec_prefix == NULL) {
-        status = calculate_set_exec_prefix(calculate, pathconfig, exec_prefix);
+        status = calculate_set_exec_prefix(calculate, pathconfig);
         if (_PyStatus_EXCEPTION(status)) {
             return status;
         }
     }
-
     return _PyStatus_OK();
 }
 


From webhook-mailer at python.org  Fri Oct  4 13:54:02 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Fri, 04 Oct 2019 17:54:02 -0000
Subject: [Python-checkins] bpo-38353: getpath.c: allocates strings on the
 heap (GH-16585)
Message-ID: <mailman.218.1570211643.32504.python-checkins@python.org>

https://github.com/python/cpython/commit/c02b41b1fb115c87693530ea6a480b2e15460424
commit: c02b41b1fb115c87693530ea6a480b2e15460424
branch: master
author: Victor Stinner <vstinner at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-04T19:53:43+02:00
summary:

bpo-38353: getpath.c: allocates strings on the heap (GH-16585)

* _Py_FindEnvConfigValue() now returns a string allocated
  by PyMem_RawMalloc().
* calculate_init() now decodes VPATH macro.
* Add calculate_open_pyenv() function.
* Add substring() and joinpath2() functions.

* Fix add_exe_suffix()

And a few cleanup changes.

files:
M Include/internal/pycore_pathconfig.h
M Modules/getpath.c
M PC/getpathp.c
M Python/pathconfig.c

diff --git a/Include/internal/pycore_pathconfig.h b/Include/internal/pycore_pathconfig.h
index ce75ccee835a2..257c056a77d0d 100644
--- a/Include/internal/pycore_pathconfig.h
+++ b/Include/internal/pycore_pathconfig.h
@@ -56,11 +56,10 @@ extern PyStatus _PyPathConfig_Calculate(
 extern int _PyPathConfig_ComputeSysPath0(
     const PyWideStringList *argv,
     PyObject **path0);
-extern int _Py_FindEnvConfigValue(
+extern PyStatus _Py_FindEnvConfigValue(
     FILE *env_file,
     const wchar_t *key,
-    wchar_t *value,
-    size_t value_size);
+    wchar_t **value_p);
 
 #ifdef MS_WINDOWS
 extern wchar_t* _Py_GetDLLPath(void);
diff --git a/Modules/getpath.c b/Modules/getpath.c
index 5f35ca6e319d8..993276fdc27d0 100644
--- a/Modules/getpath.c
+++ b/Modules/getpath.c
@@ -113,6 +113,8 @@ extern "C" {
 #define LANDMARK L"os.py"
 #endif
 
+#define BUILD_LANDMARK L"Modules/Setup.local"
+
 #define DECODE_LOCALE_ERR(NAME, LEN) \
     ((LEN) == (size_t)-2) \
      ? _PyStatus_ERR("cannot decode " NAME) \
@@ -126,6 +128,7 @@ typedef struct {
     wchar_t *pythonpath_macro;         /* PYTHONPATH macro */
     wchar_t *prefix_macro;             /* PREFIX macro */
     wchar_t *exec_prefix_macro;        /* EXEC_PREFIX macro */
+    wchar_t *vpath_macro;              /* VPATH macro */
 
     wchar_t *lib_python;               /* "lib/pythonX.Y" */
 
@@ -222,7 +225,12 @@ isdir(const wchar_t *filename)
 
 
 /* Add a path component, by appending stuff to buffer.
-   buflen: 'buffer' length in characters including trailing NUL. */
+   buflen: 'buffer' length in characters including trailing NUL.
+
+   If path2 is empty:
+
+   - if path doesn't end with SEP and is not empty, add SEP to path
+   - otherwise, do nothing. */
 static PyStatus
 joinpath(wchar_t *path, const wchar_t *path2, size_t path_len)
 {
@@ -252,6 +260,48 @@ joinpath(wchar_t *path, const wchar_t *path2, size_t path_len)
 }
 
 
+static wchar_t*
+substring(const wchar_t *str, size_t len)
+{
+    wchar_t *substr = PyMem_RawMalloc((len + 1) * sizeof(wchar_t));
+    if (substr == NULL) {
+        return NULL;
+    }
+
+    if (len) {
+        memcpy(substr, str, len * sizeof(wchar_t));
+    }
+    substr[len] = L'\0';
+    return substr;
+}
+
+
+static wchar_t*
+joinpath2(const wchar_t *path, const wchar_t *path2)
+{
+    if (_Py_isabs(path2)) {
+        return _PyMem_RawWcsdup(path2);
+    }
+
+    size_t len = wcslen(path);
+    int add_sep = (len > 0 && path[len - 1] != SEP);
+    len += add_sep;
+    len += wcslen(path2);
+
+    wchar_t *new_path = PyMem_RawMalloc((len + 1) * sizeof(wchar_t));
+    if (new_path == NULL) {
+        return NULL;
+    }
+
+    wcscpy(new_path, path);
+    if (add_sep) {
+        wcscat(new_path, separator);
+    }
+    wcscat(new_path, path2);
+    return new_path;
+}
+
+
 static inline int
 safe_wcscpy(wchar_t *dst, const wchar_t *src, size_t n)
 {
@@ -297,20 +347,22 @@ copy_absolute(wchar_t *abs_path, const wchar_t *path, size_t abs_path_len)
 
 /* path_len: path length in characters including trailing NUL */
 static PyStatus
-absolutize(wchar_t *path, size_t path_len)
+absolutize(wchar_t **path_p)
 {
-    if (_Py_isabs(path)) {
-        return _PyStatus_OK();
-    }
+    assert(!_Py_isabs(*path_p));
 
     wchar_t abs_path[MAXPATHLEN+1];
+    wchar_t *path = *path_p;
+
     PyStatus status = copy_absolute(abs_path, path, Py_ARRAY_LENGTH(abs_path));
     if (_PyStatus_EXCEPTION(status)) {
         return status;
     }
 
-    if (safe_wcscpy(path, abs_path, path_len) < 0) {
-        return PATHLEN_ERR();
+    PyMem_RawFree(*path_p);
+    *path_p = _PyMem_RawWcsdup(abs_path);
+    if (*path_p == NULL) {
+        return _PyStatus_NO_MEMORY();
     }
     return _PyStatus_OK();
 }
@@ -320,33 +372,33 @@ absolutize(wchar_t *path, size_t path_len)
 static PyStatus
 ismodule(const wchar_t *path, int *result)
 {
-    wchar_t filename[MAXPATHLEN+1];
-    size_t filename_len = Py_ARRAY_LENGTH(filename);
-
-    if (safe_wcscpy(filename, path, filename_len) < 0) {
-        return PATHLEN_ERR();
-    }
-
-    PyStatus status = joinpath(filename, LANDMARK, filename_len);
-    if (_PyStatus_EXCEPTION(status)) {
-        return status;
+    wchar_t *filename = joinpath2(path, LANDMARK);
+    if (filename == NULL) {
+        return _PyStatus_NO_MEMORY();
     }
 
     if (isfile(filename)) {
+        PyMem_RawFree(filename);
         *result = 1;
         return _PyStatus_OK();
     }
 
     /* Check for the compiled version of prefix. */
-    if (wcslen(filename) + 2 <= filename_len) {
-        wcscat(filename, L"c");
-        if (isfile(filename)) {
-            *result = 1;
-            return _PyStatus_OK();
-        }
+    size_t len = wcslen(filename);
+    wchar_t *pyc = PyMem_RawMalloc((len + 2) * sizeof(wchar_t));
+    if (pyc == NULL) {
+        PyMem_RawFree(filename);
+        return _PyStatus_NO_MEMORY();
     }
 
-    *result = 0;
+    memcpy(pyc, filename, len * sizeof(wchar_t));
+    pyc[len] = L'c';
+    pyc[len + 1] = L'\0';
+    *result = isfile(pyc);
+
+    PyMem_RawFree(filename);
+    PyMem_RawFree(pyc);
+
     return _PyStatus_OK();
 }
 
@@ -358,24 +410,32 @@ ismodule(const wchar_t *path, int *result)
 
 /* pathlen: 'path' length in characters including trailing NUL */
 static PyStatus
-add_exe_suffix(wchar_t *path, size_t pathlen)
+add_exe_suffix(wchar_t **progpath_p)
 {
+    wchar_t *progpath = *progpath_p;
+
     /* Check for already have an executable suffix */
-    size_t n = wcslen(path);
+    size_t n = wcslen(progpath);
     size_t s = wcslen(EXE_SUFFIX);
-    if (wcsncasecmp(EXE_SUFFIX, path + n - s, s) == 0) {
+    if (wcsncasecmp(EXE_SUFFIX, progpath + n - s, s) == 0) {
         return _PyStatus_OK();
     }
 
-    if (n + s >= pathlen) {
-        return PATHLEN_ERR();
+    wchar_t *progpath2 = PyMem_RawMalloc((n + s + 1) * sizeof(wchar_t));
+    if (progpath2 == NULL) {
+        return _PyStatus_NO_MEMORY();
     }
-    wcsncpy(path + n, EXE_SUFFIX, s);
-    path[n + s] = '\0';
 
-    if (!isxfile(path)) {
-        /* Path that added suffix is invalid: truncate (remove suffix) */
-        path[n] = '\0';
+    memcpy(progpath2, progpath, n * sizeof(wchar_t));
+    memcpy(progpath2 + n, EXE_SUFFIX, s * sizeof(wchar_t));
+    progpath2[n+s] = L'\0';
+
+    if (isxfile(progpath2)) {
+        PyMem_RawFree(*progpath_p);
+        *progpath_p = progpath2;
+    }
+    else {
+        PyMem_RawFree(progpath2);
     }
     return _PyStatus_OK();
 }
@@ -389,10 +449,6 @@ static PyStatus
 search_for_prefix(PyCalculatePath *calculate, _PyPathConfig *pathconfig,
                   wchar_t *prefix, size_t prefix_len, int *found)
 {
-    wchar_t path[MAXPATHLEN+1];
-    memset(path, 0, sizeof(path));
-    size_t path_len = Py_ARRAY_LENGTH(path);
-
     PyStatus status;
 
     /* If PYTHONHOME is set, we believe it unconditionally */
@@ -413,44 +469,46 @@ search_for_prefix(PyCalculatePath *calculate, _PyPathConfig *pathconfig,
         return _PyStatus_OK();
     }
 
-    /* Check to see if argv[0] is in the build directory */
-    if (safe_wcscpy(path, calculate->argv0_path, path_len) < 0) {
-        return PATHLEN_ERR();
-    }
-    status = joinpath(path, L"Modules/Setup.local", path_len);
-    if (_PyStatus_EXCEPTION(status)) {
-        return status;
+    /* Check to see if argv0_path is in the build directory
+
+       Path: <argv0_path> / <BUILD_LANDMARK define> */
+    wchar_t *path = joinpath2(calculate->argv0_path, BUILD_LANDMARK);
+    if (path == NULL) {
+        return _PyStatus_NO_MEMORY();
     }
 
-    if (isfile(path)) {
-        /* Check VPATH to see if argv0_path is in the build directory.
-           VPATH can be empty. */
-        wchar_t *vpath = Py_DecodeLocale(VPATH, NULL);
-        if (vpath != NULL) {
-            /* Path: <argv0_path> / <vpath> / Lib / LANDMARK */
-            if (safe_wcscpy(prefix, calculate->argv0_path, prefix_len) < 0) {
-                return PATHLEN_ERR();
-            }
-            status = joinpath(prefix, vpath, prefix_len);
-            PyMem_RawFree(vpath);
-            if (_PyStatus_EXCEPTION(status)) {
-                return status;
-            }
+    int is_build_dir = isfile(path);
+    PyMem_RawFree(path);
 
-            status = joinpath(prefix, L"Lib", prefix_len);
-            if (_PyStatus_EXCEPTION(status)) {
-                return status;
-            }
+    if (is_build_dir) {
+        /* argv0_path is the build directory (BUILD_LANDMARK exists),
+           now also check LANDMARK using ismodule(). */
 
-            int module;
-            status = ismodule(prefix, &module);
-            if (_PyStatus_EXCEPTION(status)) {
-                return status;
-            }
-            if (module) {
-                *found = -1;
-                return _PyStatus_OK();
-            }
+        /* Path: <argv0_path> / <VPATH macro> / Lib */
+        /* or if VPATH is empty: <argv0_path> / Lib */
+        if (safe_wcscpy(prefix, calculate->argv0_path, prefix_len) < 0) {
+            return PATHLEN_ERR();
+        }
+
+        status = joinpath(prefix, calculate->vpath_macro, prefix_len);
+        if (_PyStatus_EXCEPTION(status)) {
+            return status;
+        }
+
+        status = joinpath(prefix, L"Lib", prefix_len);
+        if (_PyStatus_EXCEPTION(status)) {
+            return status;
+        }
+
+        int module;
+        status = ismodule(prefix, &module);
+        if (_PyStatus_EXCEPTION(status)) {
+            return status;
+        }
+        if (module) {
+            /* BUILD_LANDMARK and LANDMARK found */
+            *found = -1;
+            return _PyStatus_OK();
         }
     }
 
@@ -527,16 +585,14 @@ calculate_prefix(PyCalculatePath *calculate, _PyPathConfig *pathconfig)
             fprintf(stderr,
                 "Could not find platform independent libraries <prefix>\n");
         }
-        if (safe_wcscpy(prefix, calculate->prefix_macro, prefix_len) < 0) {
-            return PATHLEN_ERR();
-        }
-        status = joinpath(prefix, calculate->lib_python, prefix_len);
-        if (_PyStatus_EXCEPTION(status)) {
-            return status;
-        }
+
+        calculate->prefix = joinpath2(calculate->prefix_macro,
+                                      calculate->lib_python);
+    }
+    else {
+        calculate->prefix = _PyMem_RawWcsdup(prefix);
     }
 
-    calculate->prefix = _PyMem_RawWcsdup(prefix);
     if (calculate->prefix == NULL) {
         return _PyStatus_NO_MEMORY();
     }
@@ -553,26 +609,32 @@ calculate_set_prefix(PyCalculatePath *calculate, _PyPathConfig *pathconfig)
      * return the compiled-in defaults instead.
      */
     if (calculate->prefix_found > 0) {
-        wchar_t prefix[MAXPATHLEN+1];
-        memset(prefix, 0, sizeof(prefix));
-
-        wcscpy(prefix, calculate->prefix);
+        wchar_t *prefix = _PyMem_RawWcsdup(calculate->prefix);
+        if (prefix == NULL) {
+            return _PyStatus_NO_MEMORY();
+        }
 
         reduce(prefix);
         reduce(prefix);
-        /* The prefix is the root directory, but reduce() chopped
-         * off the "/". */
-        if (!prefix[0]) {
-            wcscpy(prefix, separator);
+        if (prefix[0]) {
+            pathconfig->prefix = prefix;
+        }
+        else {
+            PyMem_RawFree(prefix);
+
+            /* The prefix is the root directory, but reduce() chopped
+               off the "/". */
+            pathconfig->prefix = _PyMem_RawWcsdup(separator);
+            if (pathconfig->prefix == NULL) {
+                return _PyStatus_NO_MEMORY();
+            }
         }
-        pathconfig->prefix = _PyMem_RawWcsdup(prefix);
     }
     else {
         pathconfig->prefix = _PyMem_RawWcsdup(calculate->prefix_macro);
-    }
-
-    if (pathconfig->prefix == NULL) {
-        return _PyStatus_NO_MEMORY();
+        if (pathconfig->prefix == NULL) {
+            return _PyStatus_NO_MEMORY();
+        }
     }
     return _PyStatus_OK();
 }
@@ -585,28 +647,23 @@ calculate_pybuilddir(const wchar_t *argv0_path,
 {
     PyStatus status;
 
-    wchar_t filename[MAXPATHLEN+1];
-    memset(filename, 0, sizeof(filename));
-    size_t filename_len = Py_ARRAY_LENGTH(filename);
-
     /* Check to see if argv[0] is in the build directory. "pybuilddir.txt"
        is written by setup.py and contains the relative path to the location
        of shared library modules.
 
        Filename: <argv0_path> / "pybuilddir.txt" */
-    if (safe_wcscpy(filename, argv0_path, filename_len) < 0) {
-        return PATHLEN_ERR();
-    }
-    status = joinpath(filename, L"pybuilddir.txt", filename_len);
-    if (_PyStatus_EXCEPTION(status)) {
-        return status;
+    wchar_t *filename = joinpath2(argv0_path, L"pybuilddir.txt");
+    if (filename == NULL) {
+        return _PyStatus_NO_MEMORY();
     }
 
     if (!isfile(filename)) {
+        PyMem_RawFree(filename);
         return _PyStatus_OK();
     }
 
     FILE *fp = _Py_wfopen(filename, L"rb");
+    PyMem_RawFree(filename);
     if (fp == NULL) {
         errno = 0;
         return _PyStatus_OK();
@@ -756,19 +813,19 @@ calculate_exec_prefix(PyCalculatePath *calculate, _PyPathConfig *pathconfig)
             fprintf(stderr,
                 "Could not find platform dependent libraries <exec_prefix>\n");
         }
-        if (safe_wcscpy(exec_prefix, calculate->exec_prefix_macro, exec_prefix_len) < 0) {
-            return PATHLEN_ERR();
-        }
-        status = joinpath(exec_prefix, L"lib/lib-dynload", exec_prefix_len);
-        if (_PyStatus_EXCEPTION(status)) {
-            return status;
+
+        calculate->exec_prefix = joinpath2(calculate->exec_prefix_macro,
+                                           L"lib/lib-dynload");
+        if (calculate->exec_prefix == NULL) {
+            return _PyStatus_NO_MEMORY();
         }
     }
-
-    /* If we found EXEC_PREFIX do *not* reduce it!  (Yet.) */
-    calculate->exec_prefix = _PyMem_RawWcsdup(exec_prefix);
-    if (calculate->exec_prefix == NULL) {
-        return _PyStatus_NO_MEMORY();
+    else {
+        /* If we found EXEC_PREFIX do *not* reduce it!  (Yet.) */
+        calculate->exec_prefix = _PyMem_RawWcsdup(exec_prefix);
+        if (calculate->exec_prefix == NULL) {
+            return _PyStatus_NO_MEMORY();
+        }
     }
     return _PyStatus_OK();
 }
@@ -779,63 +836,71 @@ calculate_set_exec_prefix(PyCalculatePath *calculate,
                           _PyPathConfig *pathconfig)
 {
     if (calculate->exec_prefix_found > 0) {
-        wchar_t exec_prefix[MAXPATHLEN+1];
-        memset(exec_prefix, 0, sizeof(exec_prefix));
-
-        wcscpy(exec_prefix, calculate->exec_prefix);
+        wchar_t *exec_prefix = _PyMem_RawWcsdup(calculate->exec_prefix);
+        if (exec_prefix == NULL) {
+            return _PyStatus_NO_MEMORY();
+        }
 
         reduce(exec_prefix);
         reduce(exec_prefix);
         reduce(exec_prefix);
-        if (!exec_prefix[0]) {
-            wcscpy(exec_prefix, separator);
-        }
 
-        pathconfig->exec_prefix = _PyMem_RawWcsdup(exec_prefix);
+        if (exec_prefix[0]) {
+            pathconfig->exec_prefix = exec_prefix;
+        }
+        else {
+            /* empty string: use SEP instead */
+            PyMem_RawFree(exec_prefix);
+
+            /* The exec_prefix is the root directory, but reduce() chopped
+               off the "/". */
+            pathconfig->exec_prefix = _PyMem_RawWcsdup(separator);
+            if (pathconfig->exec_prefix == NULL) {
+                return _PyStatus_NO_MEMORY();
+            }
+        }
     }
     else {
         pathconfig->exec_prefix = _PyMem_RawWcsdup(calculate->exec_prefix_macro);
+        if (pathconfig->exec_prefix == NULL) {
+            return _PyStatus_NO_MEMORY();
+        }
     }
-
-    if (pathconfig->exec_prefix == NULL) {
-        return _PyStatus_NO_MEMORY();
-    }
-
     return _PyStatus_OK();
 }
 
 
+/* Similar to shutil.which().
+   If found, write the path into *abs_path_p. */
 static PyStatus
 calculate_which(const wchar_t *path_env, wchar_t *program_name,
-                wchar_t *fullpath, size_t fullpath_len, int *found)
+                wchar_t **abs_path_p)
 {
     while (1) {
         wchar_t *delim = wcschr(path_env, DELIM);
+        wchar_t *abs_path;
 
         if (delim) {
-            size_t len = delim - path_env;
-            if (len >= fullpath_len) {
-                return PATHLEN_ERR();
+            wchar_t *path = substring(path_env, delim - path_env);
+            if (path == NULL) {
+                return _PyStatus_NO_MEMORY();
             }
-            wcsncpy(fullpath, path_env, len);
-            fullpath[len] = '\0';
+            abs_path = joinpath2(path, program_name);
+            PyMem_RawFree(path);
         }
         else {
-            if (safe_wcscpy(fullpath, path_env,
-                        fullpath_len) < 0) {
-                return PATHLEN_ERR();
-            }
+            abs_path = joinpath2(path_env, program_name);
         }
 
-        PyStatus status = joinpath(fullpath, program_name, fullpath_len);
-        if (_PyStatus_EXCEPTION(status)) {
-            return status;
+        if (abs_path == NULL) {
+            return _PyStatus_NO_MEMORY();
         }
 
-        if (isxfile(fullpath)) {
-            *found = 1;
+        if (isxfile(abs_path)) {
+            *abs_path_p = abs_path;
             return _PyStatus_OK();
         }
+        PyMem_RawFree(abs_path);
 
         if (!delim) {
             break;
@@ -850,7 +915,7 @@ calculate_which(const wchar_t *path_env, wchar_t *program_name,
 
 #ifdef __APPLE__
 static PyStatus
-calculate_program_macos(wchar_t *fullpath, size_t fullpath_len, int *found)
+calculate_program_macos(wchar_t **abs_path_p)
 {
     char execpath[MAXPATHLEN + 1];
 #if MAC_OS_X_VERSION_MAX_ALLOWED >= MAC_OS_X_VERSION_10_4
@@ -877,26 +942,20 @@ calculate_program_macos(wchar_t *fullpath, size_t fullpath_len, int *found)
     }
 
     size_t len;
-    wchar_t *path = Py_DecodeLocale(execpath, &len);
-    if (path == NULL) {
+    *abs_path_p = Py_DecodeLocale(execpath, &len);
+    if (*abs_path_p == NULL) {
         return DECODE_LOCALE_ERR("executable path", len);
     }
-    if (safe_wcscpy(fullpath, path, fullpath_len) < 0) {
-        PyMem_RawFree(path);
-        return PATHLEN_ERR();
-    }
-    PyMem_RawFree(path);
-
-    *found = 1;
     return _PyStatus_OK();
 }
 #endif  /* __APPLE__ */
 
 
 static PyStatus
-calculate_program_impl(PyCalculatePath *calculate, _PyPathConfig *pathconfig,
-                       wchar_t *fullpath, size_t fullpath_len)
+calculate_program_impl(PyCalculatePath *calculate, _PyPathConfig *pathconfig)
 {
+    assert(pathconfig->program_full_path == NULL);
+
     PyStatus status;
 
     /* If there is no slash in the argv0 path, then we have to
@@ -905,39 +964,43 @@ calculate_program_impl(PyCalculatePath *calculate, _PyPathConfig *pathconfig,
      * $PATH isn't exported, you lose.
      */
     if (wcschr(pathconfig->program_name, SEP)) {
-        if (safe_wcscpy(fullpath, pathconfig->program_name,
-                        fullpath_len) < 0) {
-            return PATHLEN_ERR();
+        pathconfig->program_full_path = _PyMem_RawWcsdup(pathconfig->program_name);
+        if (pathconfig->program_full_path == NULL) {
+            return _PyStatus_NO_MEMORY();
         }
         return _PyStatus_OK();
     }
 
 #ifdef __APPLE__
-    int found = 0;
-    status = calculate_program_macos(fullpath, fullpath_len, &found);
+    wchar_t *abs_path = NULL;
+    status = calculate_program_macos(&abs_path);
     if (_PyStatus_EXCEPTION(status)) {
         return status;
     }
-    if (found) {
+    if (abs_path) {
+        pathconfig->program_full_path = abs_path;
         return _PyStatus_OK();
     }
 #endif /* __APPLE__ */
 
     if (calculate->path_env) {
-        int found = 0;
+        wchar_t *abs_path = NULL;
         status = calculate_which(calculate->path_env, pathconfig->program_name,
-                                 fullpath, fullpath_len,
-                                 &found);
+                                 &abs_path);
         if (_PyStatus_EXCEPTION(status)) {
             return status;
         }
-        if (found) {
+        if (abs_path) {
+            pathconfig->program_full_path = abs_path;
             return _PyStatus_OK();
         }
     }
 
     /* In the last resort, use an empty string */
-    fullpath[0] = '\0';
+    pathconfig->program_full_path = _PyMem_RawWcsdup(L"");
+    if (pathconfig->program_full_path == NULL) {
+        return _PyStatus_NO_MEMORY();
+    }
     return _PyStatus_OK();
 }
 
@@ -947,23 +1010,18 @@ static PyStatus
 calculate_program(PyCalculatePath *calculate, _PyPathConfig *pathconfig)
 {
     PyStatus status;
-    wchar_t program_full_path[MAXPATHLEN + 1];
-    const size_t program_full_path_len = Py_ARRAY_LENGTH(program_full_path);
-    memset(program_full_path, 0, sizeof(program_full_path));
 
-    status = calculate_program_impl(calculate, pathconfig,
-                                    program_full_path, program_full_path_len);
+    status = calculate_program_impl(calculate, pathconfig);
     if (_PyStatus_EXCEPTION(status)) {
         return status;
     }
 
-    if (program_full_path[0] != '\0') {
+    if (pathconfig->program_full_path[0] != '\0') {
         /* program_full_path is not empty */
 
-        /* Make sure that program_full_path is an absolute path
-           (or an empty string) */
-        if (!_Py_isabs(program_full_path)) {
-            status = absolutize(program_full_path, program_full_path_len);
+        /* Make sure that program_full_path is an absolute path */
+        if (!_Py_isabs(pathconfig->program_full_path)) {
+            status = absolutize(&pathconfig->program_full_path);
             if (_PyStatus_EXCEPTION(status)) {
                 return status;
             }
@@ -975,54 +1033,54 @@ calculate_program(PyCalculatePath *calculate, _PyPathConfig *pathconfig)
          * sys.executable to return the name of a directory under the same
          * path (bpo-28441).
          */
-        status = add_exe_suffix(program_full_path, program_full_path_len);
+        status = add_exe_suffix(&pathconfig->program_full_path);
         if (_PyStatus_EXCEPTION(status)) {
             return status;
         }
 #endif
     }
-
-    pathconfig->program_full_path = _PyMem_RawWcsdup(program_full_path);
-    if (pathconfig->program_full_path == NULL) {
-        return _PyStatus_NO_MEMORY();
-    }
     return _PyStatus_OK();
 }
 
 
 #if HAVE_READLINK
 static PyStatus
-resolve_symlinks(wchar_t *path, size_t path_len)
+resolve_symlinks(wchar_t **path_p)
 {
     wchar_t new_path[MAXPATHLEN + 1];
     const size_t new_path_len = Py_ARRAY_LENGTH(new_path);
-    unsigned int links = 0;
+    unsigned int nlink = 0;
 
     while (1) {
-        int linklen = _Py_wreadlink(path, new_path, new_path_len);
+        int linklen = _Py_wreadlink(*path_p, new_path, new_path_len);
         if (linklen == -1) {
+            /* not a symbolic link: we are done */
             break;
         }
 
         if (_Py_isabs(new_path)) {
-            /* new_path should never be longer than MAXPATHLEN,
-               but extra check does not hurt */
-            if (safe_wcscpy(path, new_path, path_len) < 0) {
-                return PATHLEN_ERR();
+            PyMem_RawFree(*path_p);
+            *path_p = _PyMem_RawWcsdup(new_path);
+            if (*path_p == NULL) {
+                return _PyStatus_NO_MEMORY();
             }
         }
         else {
             /* new_path is relative to path */
-            reduce(path);
-            PyStatus status = joinpath(path, new_path, path_len);
-            if (_PyStatus_EXCEPTION(status)) {
-                return status;
+            reduce(*path_p);
+
+            wchar_t *abs_path = joinpath2(*path_p, new_path);
+            if (abs_path == NULL) {
+                return _PyStatus_NO_MEMORY();
             }
+
+            PyMem_RawFree(*path_p);
+            *path_p = abs_path;
         }
 
-        links++;
+        nlink++;
         /* 40 is the Linux kernel 4.2 limit */
-        if (links >= 40) {
+        if (nlink >= 40) {
             return _PyStatus_ERR("maximum number of symbolic links reached");
         }
     }
@@ -1033,9 +1091,7 @@ resolve_symlinks(wchar_t *path, size_t path_len)
 
 #ifdef WITH_NEXT_FRAMEWORK
 static PyStatus
-calculate_argv0_path_framework(PyCalculatePath *calculate,
-                               const wchar_t *program_full_path,
-                               wchar_t *argv0_path, size_t argv0_path_len)
+calculate_argv0_path_framework(PyCalculatePath *calculate, _PyPathConfig *pathconfig)
 {
     NSModule pythonModule;
 
@@ -1067,37 +1123,47 @@ calculate_argv0_path_framework(PyCalculatePath *calculate,
 
     /* Path: reduce(modPath) / lib_python / LANDMARK */
     PyStatus status;
-    if (safe_wcscpy(argv0_path, wbuf, argv0_path_len) < 0) {
-        status = PATHLEN_ERR();
+
+    wchar_t *parent = _PyMem_RawWcsdup(wbuf);
+    if (parent == NULL) {
+        status = _PyStatus_NO_MEMORY();
         goto done;
     }
-    reduce(argv0_path);
-    status = joinpath(argv0_path, calculate->lib_python, argv0_path_len);
-    if (_PyStatus_EXCEPTION(status)) {
+
+    reduce(parent);
+    wchar_t *lib_python = joinpath2(path, calculate->lib_python);
+    PyMem_RawFree(parent);
+
+    if (lib_python == NULL) {
+        status = _PyStatus_NO_MEMORY();
         goto done;
     }
 
     int module;
-    status = ismodule(argv0_path, &module);
+    status = ismodule(lib_python, &module);
+    PyMem_RawFree(lib_python);
+
     if (_PyStatus_EXCEPTION(status)) {
         goto done;
     }
-    if (module) {
-        /* Use the location of the library as argv0_path */
-        if (safe_wcscpy(argv0_path, wbuf, argv0_path_len) < 0) {
-            status = PATHLEN_ERR();
-            goto done;
-        }
-    }
-    else {
+    if (!module) {
         /* We are in the build directory so use the name of the
            executable - we know that the absolute path is passed */
-        if (safe_wcscpy(argv0_path, program_full_path, argv0_path_len) < 0) {
-            status = PATHLEN_ERR();
+        PyMem_RawFree(*calculate->argv0_path);
+        calculate->argv0_path = _PyMem_RawWcsdup(pathconfig->program_full_path);
+        if (calculate->argv0_path == NULL) {
+            status = _PyStatus_NO_MEMORY();
             goto done;
         }
+
+        status = _PyStatus_OK();
+        goto done;
     }
-    status = _PyStatus_OK();
+
+    /* Use the location of the library as argv0_path */
+    PyMem_RawFree(*calculate->argv0_path);
+    calculate->argv0_path = wbuf
+    return _PyStatus_OK();
 
 done:
     PyMem_RawFree(wbuf);
@@ -1108,35 +1174,77 @@ calculate_argv0_path_framework(PyCalculatePath *calculate,
 
 static PyStatus
 calculate_argv0_path(PyCalculatePath *calculate,
-                     const wchar_t *program_full_path)
+                     _PyPathConfig *pathconfig)
 {
-    wchar_t argv0_path[MAXPATHLEN+1];
-    memset(argv0_path, 0, sizeof(argv0_path));
-    size_t argv0_path_len = Py_ARRAY_LENGTH(argv0_path);
+    PyStatus status;
 
-    if (safe_wcscpy(argv0_path, program_full_path, argv0_path_len) < 0) {
-        return PATHLEN_ERR();
+    calculate->argv0_path = _PyMem_RawWcsdup(pathconfig->program_full_path);
+    if (calculate->argv0_path == NULL) {
+        return _PyStatus_NO_MEMORY();
     }
 
 #ifdef WITH_NEXT_FRAMEWORK
-    PyStatus status;
-    status = calculate_argv0_path_framework(calculate, program_full_path,
-                                            argv0_path, argv0_path_len);
+    status = calculate_argv0_path_framework(calculate, pathconfig);
     if (_PyStatus_EXCEPTION(status)) {
         return status;
     }
 #endif
 
-    resolve_symlinks(argv0_path, argv0_path_len);
+    status = resolve_symlinks(&calculate->argv0_path);
+    if (_PyStatus_EXCEPTION(status)) {
+        return status;
+    }
 
-    reduce(argv0_path);
-    /* At this point, argv0_path is guaranteed to be less than
-       MAXPATHLEN bytes long. */
+    reduce(calculate->argv0_path);
 
-    calculate->argv0_path = _PyMem_RawWcsdup(argv0_path);
-    if (calculate->argv0_path == NULL) {
+    return _PyStatus_OK();
+}
+
+
+static PyStatus
+calculate_open_pyenv(PyCalculatePath *calculate, FILE **env_file_p)
+{
+    *env_file_p = NULL;
+
+    const wchar_t *env_cfg = L"pyvenv.cfg";
+
+    /* Filename: <argv0_path> / "pyvenv.cfg" */
+    wchar_t *filename = joinpath2(calculate->argv0_path, env_cfg);
+    if (filename == NULL) {
+        return _PyStatus_NO_MEMORY();
+    }
+
+    *env_file_p = _Py_wfopen(filename, L"r");
+    PyMem_RawFree(filename);
+
+    if (*env_file_p != NULL) {
+        return _PyStatus_OK();
+
+    }
+
+    /* fopen() failed: reset errno */
+    errno = 0;
+
+    /* Path: <basename(argv0_path)> / "pyvenv.cfg" */
+    wchar_t *parent = _PyMem_RawWcsdup(calculate->argv0_path);
+    if (parent == NULL) {
+        return _PyStatus_NO_MEMORY();
+    }
+    reduce(parent);
+
+    filename = joinpath2(parent, env_cfg);
+    PyMem_RawFree(parent);
+    if (filename == NULL) {
         return _PyStatus_NO_MEMORY();
     }
+
+    *env_file_p = _Py_wfopen(filename, L"r");
+    PyMem_RawFree(filename);
+
+    if (*env_file_p == NULL) {
+        /* fopen() failed: reset errno */
+        errno = 0;
+    }
     return _PyStatus_OK();
 }
 
@@ -1150,53 +1258,28 @@ static PyStatus
 calculate_read_pyenv(PyCalculatePath *calculate)
 {
     PyStatus status;
-    const wchar_t *env_cfg = L"pyvenv.cfg";
-    FILE *env_file;
-
-    wchar_t filename[MAXPATHLEN+1];
-    const size_t filename_len = Py_ARRAY_LENGTH(filename);
-    memset(filename, 0, sizeof(filename));
-
-    /* Filename: <argv0_path_len> / "pyvenv.cfg" */
-    if (safe_wcscpy(filename, calculate->argv0_path, filename_len) < 0) {
-        return PATHLEN_ERR();
-    }
+    FILE *env_file = NULL;
 
-    status = joinpath(filename, env_cfg, filename_len);
+    status = calculate_open_pyenv(calculate, &env_file);
     if (_PyStatus_EXCEPTION(status)) {
         return status;
     }
-    env_file = _Py_wfopen(filename, L"r");
     if (env_file == NULL) {
-        errno = 0;
-
-        /* Filename: <basename(basename(argv0_path_len))> / "pyvenv.cfg" */
-        reduce(filename);
-        reduce(filename);
-        status = joinpath(filename, env_cfg, filename_len);
-        if (_PyStatus_EXCEPTION(status)) {
-            return status;
-        }
-
-        env_file = _Py_wfopen(filename, L"r");
-        if (env_file == NULL) {
-            errno = 0;
-            return _PyStatus_OK();
-        }
+        /* pyvenv.cfg not found */
+        return _PyStatus_OK();
     }
 
     /* Look for a 'home' variable and set argv0_path to it, if found */
-    wchar_t home[MAXPATHLEN+1];
-    memset(home, 0, sizeof(home));
+    wchar_t *home = NULL;
+    status = _Py_FindEnvConfigValue(env_file, L"home", &home);
+    if (_PyStatus_EXCEPTION(status)) {
+        fclose(env_file);
+        return status;
+    }
 
-    if (_Py_FindEnvConfigValue(env_file, L"home",
-                               home, Py_ARRAY_LENGTH(home))) {
+    if (home) {
         PyMem_RawFree(calculate->argv0_path);
-        calculate->argv0_path = _PyMem_RawWcsdup(home);
-        if (calculate->argv0_path == NULL) {
-            fclose(env_file);
-            return _PyStatus_NO_MEMORY();
-         }
+        calculate->argv0_path = home;
     }
     fclose(env_file);
     return _PyStatus_OK();
@@ -1206,39 +1289,34 @@ calculate_read_pyenv(PyCalculatePath *calculate)
 static PyStatus
 calculate_zip_path(PyCalculatePath *calculate)
 {
-    PyStatus status;
-
-    wchar_t zip_path[MAXPATHLEN+1];    /* ".../lib/pythonXY.zip" */
-    memset(zip_path, 0, sizeof(zip_path));
-    size_t zip_path_len = Py_ARRAY_LENGTH(zip_path);
+    const wchar_t *lib_python = L"lib/python00.zip";
 
     if (calculate->prefix_found > 0) {
-        /* Use the reduced prefix returned by Py_GetPrefix() */
-        if (safe_wcscpy(zip_path, calculate->prefix, zip_path_len) < 0) {
-            return PATHLEN_ERR();
+        /* Use the reduced prefix returned by Py_GetPrefix()
+
+           Path: <basename(basename(prefix))> / <lib_python> */
+        wchar_t *parent = _PyMem_RawWcsdup(calculate->prefix);
+        if (parent == NULL) {
+            return _PyStatus_NO_MEMORY();
         }
-        reduce(zip_path);
-        reduce(zip_path);
+        reduce(parent);
+        reduce(parent);
+        calculate->zip_path = joinpath2(parent, lib_python);
+        PyMem_RawFree(parent);
     }
     else {
-        if (safe_wcscpy(zip_path, calculate->prefix_macro, zip_path_len) < 0) {
-            return PATHLEN_ERR();
-        }
-    }
-    status = joinpath(zip_path, L"lib/python00.zip", zip_path_len);
-    if (_PyStatus_EXCEPTION(status)) {
-        return status;
+        calculate->zip_path = joinpath2(calculate->prefix_macro, lib_python);
     }
 
-    /* Replace "00" with version */
-    size_t bufsz = wcslen(zip_path);
-    zip_path[bufsz - 6] = VERSION[0];
-    zip_path[bufsz - 5] = VERSION[2];
-
-    calculate->zip_path = _PyMem_RawWcsdup(zip_path);
     if (calculate->zip_path == NULL) {
         return _PyStatus_NO_MEMORY();
     }
+
+    /* Replace "00" with version */
+    size_t len = wcslen(calculate->zip_path);
+    calculate->zip_path[len - 6] = VERSION[0];
+    calculate->zip_path[len - 5] = VERSION[2];
+
     return _PyStatus_OK();
 }
 
@@ -1344,22 +1422,27 @@ calculate_init(PyCalculatePath *calculate, const PyConfig *config)
         }
     }
 
+    /* Decode macros */
     calculate->pythonpath_macro = Py_DecodeLocale(PYTHONPATH, &len);
     if (!calculate->pythonpath_macro) {
-        return DECODE_LOCALE_ERR("PYTHONPATH define", len);
+        return DECODE_LOCALE_ERR("PYTHONPATH macro", len);
     }
-
     calculate->prefix_macro = Py_DecodeLocale(PREFIX, &len);
     if (!calculate->prefix_macro) {
-        return DECODE_LOCALE_ERR("PREFIX define", len);
+        return DECODE_LOCALE_ERR("PREFIX macro", len);
     }
     calculate->exec_prefix_macro = Py_DecodeLocale(EXEC_PREFIX, &len);
     if (!calculate->exec_prefix_macro) {
-        return DECODE_LOCALE_ERR("EXEC_PREFIX define", len);
+        return DECODE_LOCALE_ERR("EXEC_PREFIX macro", len);
     }
+    calculate->vpath_macro = Py_DecodeLocale(VPATH, &len);
+    if (!calculate->vpath_macro) {
+        return DECODE_LOCALE_ERR("VPATH macro", len);
+    }
+
     calculate->lib_python = Py_DecodeLocale("lib/python" VERSION, &len);
     if (!calculate->lib_python) {
-        return DECODE_LOCALE_ERR("EXEC_PREFIX define", len);
+        return DECODE_LOCALE_ERR("EXEC_PREFIX macro", len);
     }
 
     calculate->warnings = config->pathconfig_warnings;
@@ -1375,6 +1458,7 @@ calculate_free(PyCalculatePath *calculate)
     PyMem_RawFree(calculate->pythonpath_macro);
     PyMem_RawFree(calculate->prefix_macro);
     PyMem_RawFree(calculate->exec_prefix_macro);
+    PyMem_RawFree(calculate->vpath_macro);
     PyMem_RawFree(calculate->lib_python);
     PyMem_RawFree(calculate->path_env);
     PyMem_RawFree(calculate->zip_path);
@@ -1396,7 +1480,7 @@ calculate_path(PyCalculatePath *calculate, _PyPathConfig *pathconfig)
         }
     }
 
-    status = calculate_argv0_path(calculate, pathconfig->program_full_path);
+    status = calculate_argv0_path(calculate, pathconfig);
     if (_PyStatus_EXCEPTION(status)) {
         return status;
     }
@@ -1501,6 +1585,10 @@ _PyPathConfig_Calculate(_PyPathConfig *pathconfig, const PyConfig *config)
         goto done;
     }
 
+    /* program_full_path must an either an empty string or an absolute path */
+    assert(wcslen(pathconfig->program_full_path) == 0
+           || _Py_isabs(pathconfig->program_full_path));
+
     status = _PyStatus_OK();
 
 done:
diff --git a/PC/getpathp.c b/PC/getpathp.c
index b17c002f25c8e..085caf195a992 100644
--- a/PC/getpathp.c
+++ b/PC/getpathp.c
@@ -752,7 +752,7 @@ calculate_pth_file(PyCalculatePath *calculate, _PyPathConfig *pathconfig,
    executable's directory and then in the parent directory.
    If found, open it for use when searching for prefixes.
 */
-static void
+static PyStatus
 calculate_pyvenv_file(PyCalculatePath *calculate,
                       wchar_t *argv0_path, size_t argv0_path_len)
 {
@@ -775,17 +775,23 @@ calculate_pyvenv_file(PyCalculatePath *calculate,
         env_file = _Py_wfopen(filename, L"r");
         if (env_file == NULL) {
             errno = 0;
-            return;
+            return _PyStatus_OK();
         }
     }
 
     /* Look for a 'home' variable and set argv0_path to it, if found */
-    wchar_t home[MAXPATHLEN+1];
-    if (_Py_FindEnvConfigValue(env_file, L"home",
-                               home, Py_ARRAY_LENGTH(home))) {
+    wchar_t *home = NULL;
+    PyStatus status = _Py_FindEnvConfigValue(env_file, L"home", &home);
+    if (_PyStatus_EXCEPTION(status)) {
+        fclose(env_file);
+        return status;
+    }
+    if (home) {
         wcscpy_s(argv0_path, argv0_path_len, home);
+        PyMem_RawFree(home);
     }
     fclose(env_file);
+    return _PyStatus_OK();
 }
 
 
@@ -1022,7 +1028,11 @@ calculate_path(PyCalculatePath *calculate, _PyPathConfig *pathconfig)
         goto done;
     }
 
-    calculate_pyvenv_file(calculate, argv0_path, Py_ARRAY_LENGTH(argv0_path));
+    status = calculate_pyvenv_file(calculate,
+                                   argv0_path, Py_ARRAY_LENGTH(argv0_path));
+    if (_PyStatus_EXCEPTION(status)) {
+        return status;
+    }
 
     /* Calculate zip archive path from DLL or exe path */
     wchar_t zip_path[MAXPATHLEN+1];
diff --git a/Python/pathconfig.c b/Python/pathconfig.c
index 61408e19e3a79..363b7686bc451 100644
--- a/Python/pathconfig.c
+++ b/Python/pathconfig.c
@@ -777,12 +777,17 @@ _PyPathConfig_ComputeSysPath0(const PyWideStringList *argv, PyObject **path0_p)
 #endif
 
 /* Search for a prefix value in an environment file (pyvenv.cfg).
-   If found, copy it into the provided buffer. */
-int
+
+   - If found, copy it into *value_p: string which must be freed by
+     PyMem_RawFree().
+   - If not found, *value_p is set to NULL.
+*/
+PyStatus
 _Py_FindEnvConfigValue(FILE *env_file, const wchar_t *key,
-                       wchar_t *value, size_t value_size)
+                       wchar_t **value_p)
 {
-    int result = 0; /* meaning not found */
+    *value_p = NULL;
+
     char buffer[MAXPATHLEN * 2 + 1];  /* allow extra for key, '=', etc. */
     buffer[Py_ARRAY_LENGTH(buffer)-1] = '\0';
 
@@ -812,18 +817,24 @@ _Py_FindEnvConfigValue(FILE *env_file, const wchar_t *key,
                 if ((tok != NULL) && !wcscmp(tok, L"=")) {
                     tok = WCSTOK(NULL, L"\r\n", &state);
                     if (tok != NULL) {
-                        wcsncpy(value, tok, value_size - 1);
-                        value[value_size - 1] = L'\0';
-                        result = 1;
+                        *value_p = _PyMem_RawWcsdup(tok);
                         PyMem_RawFree(tmpbuffer);
-                        break;
+
+                        if (*value_p == NULL) {
+                            return _PyStatus_NO_MEMORY();
+                        }
+
+                        /* found */
+                        return _PyStatus_OK();
                     }
                 }
             }
             PyMem_RawFree(tmpbuffer);
         }
     }
-    return result;
+
+    /* not found */
+    return _PyStatus_OK();
 }
 
 #ifdef __cplusplus


From webhook-mailer at python.org  Fri Oct  4 20:31:02 2019
From: webhook-mailer at python.org (Abhilash Raj)
Date: Sat, 05 Oct 2019 00:31:02 -0000
Subject: [Python-checkins] bpo-38341: Add SMTPNotSupportedError in the
 exports of smtplib (#16525)
Message-ID: <mailman.219.1570235463.32504.python-checkins@python.org>

https://github.com/python/cpython/commit/3faf826e5879536d2272f1a51c58965a16827f81
commit: 3faf826e5879536d2272f1a51c58965a16827f81
branch: master
author: nde <denayer.norman at gmail.com>
committer: Abhilash Raj <maxking at users.noreply.github.com>
date: 2019-10-04T17:30:58-07:00
summary:

bpo-38341: Add SMTPNotSupportedError in the exports of smtplib (#16525)

Add SMTPNotSupportedError in the exports of smtplib

Co-Authored-By: Brandt Bucher <brandtbucher at gmail.com>

files:
A Misc/NEWS.d/next/Library/2019-10-01-21-06-18.bpo-38341.uqwgU_.rst
M Lib/smtplib.py

diff --git a/Lib/smtplib.py b/Lib/smtplib.py
index 43a00d9a4e3c4..6513842eb6c61 100755
--- a/Lib/smtplib.py
+++ b/Lib/smtplib.py
@@ -54,7 +54,7 @@
 import sys
 from email.base64mime import body_encode as encode_base64
 
-__all__ = ["SMTPException", "SMTPServerDisconnected", "SMTPResponseException",
+__all__ = ["SMTPException", "SMTPNotSupportedError", "SMTPServerDisconnected", "SMTPResponseException",
            "SMTPSenderRefused", "SMTPRecipientsRefused", "SMTPDataError",
            "SMTPConnectError", "SMTPHeloError", "SMTPAuthenticationError",
            "quoteaddr", "quotedata", "SMTP"]
diff --git a/Misc/NEWS.d/next/Library/2019-10-01-21-06-18.bpo-38341.uqwgU_.rst b/Misc/NEWS.d/next/Library/2019-10-01-21-06-18.bpo-38341.uqwgU_.rst
new file mode 100644
index 0000000000000..dd1a6b080b627
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-10-01-21-06-18.bpo-38341.uqwgU_.rst
@@ -0,0 +1 @@
+Add :exc:`smtplib.SMTPNotSupportedError` to the :mod:`smtplib` exported names.


From webhook-mailer at python.org  Sat Oct  5 12:19:21 2019
From: webhook-mailer at python.org (Abhilash Raj)
Date: Sat, 05 Oct 2019 16:19:21 -0000
Subject: [Python-checkins] bpo-38332: Catch KeyError from unknown cte in
 encoded-word. (GH-16503)
Message-ID: <mailman.220.1570292362.32504.python-checkins@python.org>

https://github.com/python/cpython/commit/65dcc8a8dc41d3453fd6b987073a5f1b30c5c0fd
commit: 65dcc8a8dc41d3453fd6b987073a5f1b30c5c0fd
branch: master
author: Andrei Troie <andreitroie90 at gmail.com>
committer: Abhilash Raj <maxking at users.noreply.github.com>
date: 2019-10-05T09:19:15-07:00
summary:

bpo-38332: Catch KeyError from unknown cte in encoded-word. (GH-16503)

KeyError should cause a failure in parsing the encoded word and should be caught and raised as a _InvalidEWError instead.

files:
A Misc/NEWS.d/next/Library/2019-10-05-02-07-52.bpo-38332.hwrPN7.rst
M Lib/email/_header_value_parser.py
M Lib/test/test_email/test__encoded_words.py
M Lib/test/test_email/test__header_value_parser.py

diff --git a/Lib/email/_header_value_parser.py b/Lib/email/_header_value_parser.py
index 16c19907d68d5..1668b4a14e9b9 100644
--- a/Lib/email/_header_value_parser.py
+++ b/Lib/email/_header_value_parser.py
@@ -1057,7 +1057,7 @@ def get_encoded_word(value):
     value = ''.join(remainder)
     try:
         text, charset, lang, defects = _ew.decode('=?' + tok + '?=')
-    except ValueError:
+    except (ValueError, KeyError):
         raise _InvalidEwError(
             "encoded word format invalid: '{}'".format(ew.cte))
     ew.charset = charset
diff --git a/Lib/test/test_email/test__encoded_words.py b/Lib/test/test_email/test__encoded_words.py
index 5a59aebba89be..0b8b1de3359aa 100644
--- a/Lib/test/test_email/test__encoded_words.py
+++ b/Lib/test/test_email/test__encoded_words.py
@@ -58,6 +58,8 @@ def test_wrong_format_input_raises(self):
             _ew.decode('=?')
         with self.assertRaises(ValueError):
             _ew.decode('')
+        with self.assertRaises(KeyError):
+            _ew.decode('=?utf-8?X?somevalue?=')
 
     def _test(self, source, result, charset='us-ascii', lang='', defects=[]):
         res, char, l, d = _ew.decode(source)
diff --git a/Lib/test/test_email/test__header_value_parser.py b/Lib/test/test_email/test__header_value_parser.py
index 058d902459b60..46d90b38e91e1 100644
--- a/Lib/test/test_email/test__header_value_parser.py
+++ b/Lib/test/test_email/test__header_value_parser.py
@@ -89,6 +89,10 @@ def test_get_encoded_word_missing_middle_raises(self):
         with self.assertRaises(errors.HeaderParseError):
             parser.get_encoded_word('=?abc?=')
 
+    def test_get_encoded_word_invalid_cte(self):
+        with self.assertRaises(errors.HeaderParseError):
+            parser.get_encoded_word('=?utf-8?X?somevalue?=')
+
     def test_get_encoded_word_valid_ew(self):
         self._test_get_x(parser.get_encoded_word,
                          '=?us-ascii?q?this_is_a_test?=  bird',
@@ -399,6 +403,14 @@ def test_get_unstructured_invalid_ew(self):
             [],
             '')
 
+    def test_get_unstructured_invalid_ew_cte(self):
+        self._test_get_x(self._get_unst,
+            '=?utf-8?X?=somevalue?=',
+            '=?utf-8?X?=somevalue?=',
+            '=?utf-8?X?=somevalue?=',
+            [],
+            '')
+
     # get_qp_ctext
 
     def test_get_qp_ctext_only(self):
diff --git a/Misc/NEWS.d/next/Library/2019-10-05-02-07-52.bpo-38332.hwrPN7.rst b/Misc/NEWS.d/next/Library/2019-10-05-02-07-52.bpo-38332.hwrPN7.rst
new file mode 100644
index 0000000000000..600c702cf3bbd
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-10-05-02-07-52.bpo-38332.hwrPN7.rst
@@ -0,0 +1,3 @@
+Prevent :exc:`KeyError` thrown by :func:`_encoded_words.decode` when given
+an encoded-word with invalid content-type encoding from propagating all the
+way to :func:`email.message.get`.


From webhook-mailer at python.org  Sun Oct  6 07:28:40 2019
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Sun, 06 Oct 2019 11:28:40 -0000
Subject: [Python-checkins] bpo-38210: Fix intersection operation with dict
 view and iterator. (GH-16602)
Message-ID: <mailman.0.1570361321.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/c38e725d17537b20ff090b1b5ec7db1820ff9b63
commit: c38e725d17537b20ff090b1b5ec7db1820ff9b63
branch: master
author: Dong-hee Na <donghee.na92 at gmail.com>
committer: Serhiy Storchaka <storchaka at gmail.com>
date: 2019-10-06T14:28:33+03:00
summary:

bpo-38210: Fix intersection operation with dict view and iterator. (GH-16602)

files:
A Misc/NEWS.d/next/Core and Builtins/2019-10-06-15-01-57.bpo-38210.Xgc6F_.rst
M Lib/test/test_dictviews.py
M Objects/dictobject.c

diff --git a/Lib/test/test_dictviews.py b/Lib/test/test_dictviews.py
index b15cfebc98912..7cf019250244b 100644
--- a/Lib/test/test_dictviews.py
+++ b/Lib/test/test_dictviews.py
@@ -214,6 +214,17 @@ def test_items_set_operations(self):
         self.assertTrue(de.items().isdisjoint(de.items()))
         self.assertTrue(de.items().isdisjoint([1]))
 
+    def test_set_operations_with_iterator(self):
+        origin = {1: 2, 3: 4}
+        self.assertEqual(origin.keys() & iter([1, 2]), {1})
+        self.assertEqual(origin.keys() | iter([1, 2]), {1, 2, 3})
+        self.assertEqual(origin.keys() ^ iter([1, 2]), {2, 3})
+
+        items = origin.items()
+        self.assertEqual(items & iter([(1, 2)]), {(1, 2)})
+        self.assertEqual(items ^ iter([(1, 2)]), {(3, 4)})
+        self.assertEqual(items | iter([(1, 2)]), {(1, 2), (3, 4)})
+
     def test_recursive_repr(self):
         d = {}
         d[42] = d.values()
diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-10-06-15-01-57.bpo-38210.Xgc6F_.rst b/Misc/NEWS.d/next/Core and Builtins/2019-10-06-15-01-57.bpo-38210.Xgc6F_.rst
new file mode 100644
index 0000000000000..768c6d4e644a5
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2019-10-06-15-01-57.bpo-38210.Xgc6F_.rst	
@@ -0,0 +1,2 @@
+Remove unecessary intersection and update set operation in dictview with
+empty set. (Contributed by Dong-hee Na in :issue:`38210`.)
diff --git a/Objects/dictobject.c b/Objects/dictobject.c
index 2f86946b985f5..5c3f1fb3c1a25 100644
--- a/Objects/dictobject.c
+++ b/Objects/dictobject.c
@@ -4225,14 +4225,6 @@ _PyDictView_Intersect(PyObject* self, PyObject *other)
 
     it = PyObject_GetIter(other);
 
-    _Py_IDENTIFIER(intersection_update);
-    tmp = _PyObject_CallMethodIdOneArg(result, &PyId_intersection_update, other);
-    if (tmp == NULL) {
-        Py_DECREF(result);
-        return NULL;
-    }
-    Py_DECREF(tmp);
-
     if (PyDictKeys_Check(self)) {
         dict_contains = dictkeys_contains;
     }


From webhook-mailer at python.org  Sun Oct  6 08:17:23 2019
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Sun, 06 Oct 2019 12:17:23 -0000
Subject: [Python-checkins] bpo-38383: Fix possible integer overflow in
 startswith() of bytes and bytearray. (GH-16603)
Message-ID: <mailman.1.1570364243.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/24ddd9c2d6ab61cbce7e68d6de36d4df9bd2c3fb
commit: 24ddd9c2d6ab61cbce7e68d6de36d4df9bd2c3fb
branch: master
author: Hai Shi <shihai1992 at gmail.com>
committer: Serhiy Storchaka <storchaka at gmail.com>
date: 2019-10-06T15:17:18+03:00
summary:

bpo-38383: Fix possible integer overflow in startswith() of bytes and bytearray. (GH-16603)

files:
M Objects/bytes_methods.c

diff --git a/Objects/bytes_methods.c b/Objects/bytes_methods.c
index 37c5f7dbc8040..7d13184205922 100644
--- a/Objects/bytes_methods.c
+++ b/Objects/bytes_methods.c
@@ -743,7 +743,7 @@ tailmatch(const char *str, Py_ssize_t len, PyObject *substr,
 
     if (direction < 0) {
         /* startswith */
-        if (start + slen > len)
+        if (start > len - slen)
             goto notfound;
     } else {
         /* endswith */


From webhook-mailer at python.org  Sun Oct  6 08:36:48 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Sun, 06 Oct 2019 12:36:48 -0000
Subject: [Python-checkins] bpo-38383: Fix possible integer overflow in
 startswith() of bytes and bytearray. (GH-16603)
Message-ID: <mailman.2.1570365409.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/b40442ba88fec5e64ab1802ce37df0afec59fa53
commit: b40442ba88fec5e64ab1802ce37df0afec59fa53
branch: 3.7
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-06T05:36:44-07:00
summary:

bpo-38383: Fix possible integer overflow in startswith() of bytes and bytearray. (GH-16603)

(cherry picked from commit 24ddd9c2d6ab61cbce7e68d6de36d4df9bd2c3fb)

Co-authored-by: Hai Shi <shihai1992 at gmail.com>

files:
M Objects/bytes_methods.c

diff --git a/Objects/bytes_methods.c b/Objects/bytes_methods.c
index 07842f7469101..7c8ea815494e9 100644
--- a/Objects/bytes_methods.c
+++ b/Objects/bytes_methods.c
@@ -757,7 +757,7 @@ tailmatch(const char *str, Py_ssize_t len, PyObject *substr,
 
     if (direction < 0) {
         /* startswith */
-        if (start + slen > len)
+        if (start > len - slen)
             goto notfound;
     } else {
         /* endswith */


From webhook-mailer at python.org  Sun Oct  6 08:37:23 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Sun, 06 Oct 2019 12:37:23 -0000
Subject: [Python-checkins] bpo-38383: Fix possible integer overflow in
 startswith() of bytes and bytearray. (GH-16603)
Message-ID: <mailman.3.1570365444.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/ce3c913909746aa1d782ed8bca5076a656422931
commit: ce3c913909746aa1d782ed8bca5076a656422931
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-06T05:37:20-07:00
summary:

bpo-38383: Fix possible integer overflow in startswith() of bytes and bytearray. (GH-16603)

(cherry picked from commit 24ddd9c2d6ab61cbce7e68d6de36d4df9bd2c3fb)

Co-authored-by: Hai Shi <shihai1992 at gmail.com>

files:
M Objects/bytes_methods.c

diff --git a/Objects/bytes_methods.c b/Objects/bytes_methods.c
index 37c5f7dbc8040..7d13184205922 100644
--- a/Objects/bytes_methods.c
+++ b/Objects/bytes_methods.c
@@ -743,7 +743,7 @@ tailmatch(const char *str, Py_ssize_t len, PyObject *substr,
 
     if (direction < 0) {
         /* startswith */
-        if (start + slen > len)
+        if (start > len - slen)
             goto notfound;
     } else {
         /* endswith */


From webhook-mailer at python.org  Sun Oct  6 22:08:55 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Mon, 07 Oct 2019 02:08:55 -0000
Subject: [Python-checkins] bpo-26510: Add versionchanged for required arg of
 add_subparsers (GH-16588)
Message-ID: <mailman.4.1570414136.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/9e71917e0290972f65711f75510078f799cf0b59
commit: 9e71917e0290972f65711f75510078f799cf0b59
branch: master
author: Adam J. Stewart <ajstewart426 at gmail.com>
committer: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
date: 2019-10-06T19:08:48-07:00
summary:

 bpo-26510: Add versionchanged for required arg of add_subparsers (GH-16588)



The `required` argument to `argparse.add_subparsers` was added in #3027. This PR specifies the earliest version of Python where it is available.


https://bugs.python.org/issue26510



Automerge-Triggered-By: @merwok

files:
M Doc/library/argparse.rst
M Doc/whatsnew/3.7.rst

diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst
index a8aeca41a70a6..00115d5cd1e9b 100644
--- a/Doc/library/argparse.rst
+++ b/Doc/library/argparse.rst
@@ -1636,7 +1636,7 @@ Sub-commands
      stored; by default ``None`` and no value is stored
 
    * required_ - Whether or not a subcommand must be provided, by default
-     ``False``.
+     ``False`` (added in 3.7)
 
    * help_ - help for sub-parser group in help output, by default ``None``
 
@@ -1792,6 +1792,9 @@ Sub-commands
      >>> parser.parse_args(['2', 'frobble'])
      Namespace(subparser_name='2', y='frobble')
 
+   .. versionchanged:: 3.7
+      New *required* keyword argument.
+
 
 FileType objects
 ^^^^^^^^^^^^^^^^
diff --git a/Doc/whatsnew/3.7.rst b/Doc/whatsnew/3.7.rst
index af7e22d9faa9e..1ec8e0c04c76b 100644
--- a/Doc/whatsnew/3.7.rst
+++ b/Doc/whatsnew/3.7.rst
@@ -2401,6 +2401,10 @@ Changes in the Python API
   instead of a :class:`bytes` instance.
   (Contributed by Victor Stinner in :issue:`21071`.)
 
+* :mod:`argparse` subparsers can now be made mandatory by passing ``required=True``
+  to :meth:`ArgumentParser.add_subparsers() <argparse.ArgumentParser.add_subparsers>`.
+  (Contributed by Anthony Sottile in :issue:`26510`.)
+
 * :meth:`ast.literal_eval()` is now stricter.  Addition and subtraction of
   arbitrary numbers are no longer allowed.
   (Contributed by Serhiy Storchaka in :issue:`31778`.)


From webhook-mailer at python.org  Sun Oct  6 22:14:54 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Mon, 07 Oct 2019 02:14:54 -0000
Subject: [Python-checkins] bpo-26510: Add versionchanged for required arg of
 add_subparsers (GH-16588)
Message-ID: <mailman.5.1570414495.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/50b8d579421e621130806e62d6108f5f83d4be7f
commit: 50b8d579421e621130806e62d6108f5f83d4be7f
branch: 3.7
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-06T19:14:49-07:00
summary:

bpo-26510: Add versionchanged for required arg of add_subparsers (GH-16588)


The `required` argument to `argparse.add_subparsers` was added in GH-3027. This PR specifies the earliest version of Python where it is available.

https://bugs.python.org/issue26510

Automerge-Triggered-By: @merwok
(cherry picked from commit 9e71917e0290972f65711f75510078f799cf0b59)

Co-authored-by: Adam J. Stewart <ajstewart426 at gmail.com>

files:
M Doc/library/argparse.rst
M Doc/whatsnew/3.7.rst

diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst
index cef197f305558..a2baa077cd890 100644
--- a/Doc/library/argparse.rst
+++ b/Doc/library/argparse.rst
@@ -1583,7 +1583,7 @@ Sub-commands
      stored; by default ``None`` and no value is stored
 
    * required_ - Whether or not a subcommand must be provided, by default
-     ``False``.
+     ``False`` (added in 3.7)
 
    * help_ - help for sub-parser group in help output, by default ``None``
 
@@ -1739,6 +1739,9 @@ Sub-commands
      >>> parser.parse_args(['2', 'frobble'])
      Namespace(subparser_name='2', y='frobble')
 
+   .. versionchanged:: 3.7
+      New *required* keyword argument.
+
 
 FileType objects
 ^^^^^^^^^^^^^^^^
diff --git a/Doc/whatsnew/3.7.rst b/Doc/whatsnew/3.7.rst
index 54c7b8f37da90..f6476a30f889f 100644
--- a/Doc/whatsnew/3.7.rst
+++ b/Doc/whatsnew/3.7.rst
@@ -2414,6 +2414,10 @@ Changes in the Python API
   instead of a :class:`bytes` instance.
   (Contributed by Victor Stinner in :issue:`21071`.)
 
+* :mod:`argparse` subparsers can now be made mandatory by passing ``required=True``
+  to :meth:`ArgumentParser.add_subparsers() <argparse.ArgumentParser.add_subparsers>`.
+  (Contributed by Anthony Sottile in :issue:`26510`.)
+
 * :meth:`ast.literal_eval()` is now stricter.  Addition and subtraction of
   arbitrary numbers are no longer allowed.
   (Contributed by Serhiy Storchaka in :issue:`31778`.)


From webhook-mailer at python.org  Sun Oct  6 22:15:48 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Mon, 07 Oct 2019 02:15:48 -0000
Subject: [Python-checkins] bpo-26510: Add versionchanged for required arg of
 add_subparsers (GH-16588)
Message-ID: <mailman.6.1570414549.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/f19b4d7474973236e63d3d8260bfb8072acfff9d
commit: f19b4d7474973236e63d3d8260bfb8072acfff9d
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-06T19:15:44-07:00
summary:

bpo-26510: Add versionchanged for required arg of add_subparsers (GH-16588)


The `required` argument to `argparse.add_subparsers` was added in GH-3027. This PR specifies the earliest version of Python where it is available.

https://bugs.python.org/issue26510

Automerge-Triggered-By: @merwok
(cherry picked from commit 9e71917e0290972f65711f75510078f799cf0b59)

Co-authored-by: Adam J. Stewart <ajstewart426 at gmail.com>

files:
M Doc/library/argparse.rst
M Doc/whatsnew/3.7.rst

diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst
index 56bd64172f801..ee00559485c1d 100644
--- a/Doc/library/argparse.rst
+++ b/Doc/library/argparse.rst
@@ -1595,7 +1595,7 @@ Sub-commands
      stored; by default ``None`` and no value is stored
 
    * required_ - Whether or not a subcommand must be provided, by default
-     ``False``.
+     ``False`` (added in 3.7)
 
    * help_ - help for sub-parser group in help output, by default ``None``
 
@@ -1751,6 +1751,9 @@ Sub-commands
      >>> parser.parse_args(['2', 'frobble'])
      Namespace(subparser_name='2', y='frobble')
 
+   .. versionchanged:: 3.7
+      New *required* keyword argument.
+
 
 FileType objects
 ^^^^^^^^^^^^^^^^
diff --git a/Doc/whatsnew/3.7.rst b/Doc/whatsnew/3.7.rst
index af7e22d9faa9e..1ec8e0c04c76b 100644
--- a/Doc/whatsnew/3.7.rst
+++ b/Doc/whatsnew/3.7.rst
@@ -2401,6 +2401,10 @@ Changes in the Python API
   instead of a :class:`bytes` instance.
   (Contributed by Victor Stinner in :issue:`21071`.)
 
+* :mod:`argparse` subparsers can now be made mandatory by passing ``required=True``
+  to :meth:`ArgumentParser.add_subparsers() <argparse.ArgumentParser.add_subparsers>`.
+  (Contributed by Anthony Sottile in :issue:`26510`.)
+
 * :meth:`ast.literal_eval()` is now stricter.  Addition and subtraction of
   arbitrary numbers are no longer allowed.
   (Contributed by Serhiy Storchaka in :issue:`31778`.)


From webhook-mailer at python.org  Mon Oct  7 05:10:22 2019
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Mon, 07 Oct 2019 09:10:22 -0000
Subject: [Python-checkins] bpo-25988: Do not expose abstract collection
 classes in the collections module. (GH-10596)
Message-ID: <mailman.7.1570439424.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/ef092fe9905f61ca27889092ca1248a11aa74498
commit: ef092fe9905f61ca27889092ca1248a11aa74498
branch: master
author: Serhiy Storchaka <storchaka at gmail.com>
committer: GitHub <noreply at github.com>
date: 2019-10-07T12:10:15+03:00
summary:

bpo-25988: Do not expose abstract collection classes in the collections module. (GH-10596)

files:
A Misc/NEWS.d/next/Library/2018-11-19-10-23-58.bpo-25988.6o7gGK.rst
M Doc/whatsnew/3.9.rst
M Lib/collections/__init__.py

diff --git a/Doc/whatsnew/3.9.rst b/Doc/whatsnew/3.9.rst
index 690b536fd881f..9f2a659a9dca7 100644
--- a/Doc/whatsnew/3.9.rst
+++ b/Doc/whatsnew/3.9.rst
@@ -195,6 +195,11 @@ Deprecated
 Removed
 =======
 
+* The abstract base classes in :mod:`collections.abc` no longer are
+  exposed in the regular :mod:`collections` module.  This will help
+  create a clearer distinction between the concrete classes and the abstract
+  base classes.
+
 * The undocumented ``sys.callstats()`` function has been removed. Since Python
   3.7, it was deprecated and always returned :const:`None`. It required a special
   build option ``CALL_PROFILE`` which was already removed in Python 3.7.
diff --git a/Lib/collections/__init__.py b/Lib/collections/__init__.py
index 859b8460e48d5..2fc71e2991ab6 100644
--- a/Lib/collections/__init__.py
+++ b/Lib/collections/__init__.py
@@ -39,21 +39,6 @@
     pass
 
 
-def __getattr__(name):
-    # For backwards compatibility, continue to make the collections ABCs
-    # through Python 3.6 available through the collections module.
-    # Note, no new collections ABCs were added in Python 3.7
-    if name in _collections_abc.__all__:
-        obj = getattr(_collections_abc, name)
-        import warnings
-        warnings.warn("Using or importing the ABCs from 'collections' instead "
-                      "of from 'collections.abc' is deprecated since Python 3.3, "
-                      "and in 3.9 it will stop working",
-                      DeprecationWarning, stacklevel=2)
-        globals()[name] = obj
-        return obj
-    raise AttributeError(f'module {__name__!r} has no attribute {name!r}')
-
 ################################################################################
 ### OrderedDict
 ################################################################################
diff --git a/Misc/NEWS.d/next/Library/2018-11-19-10-23-58.bpo-25988.6o7gGK.rst b/Misc/NEWS.d/next/Library/2018-11-19-10-23-58.bpo-25988.6o7gGK.rst
new file mode 100644
index 0000000000000..21d25e8d74ced
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2018-11-19-10-23-58.bpo-25988.6o7gGK.rst
@@ -0,0 +1,2 @@
+The abstract base classes in :mod:`collections.abc` no longer are exposed in
+the regular :mod:`collections` module.


From webhook-mailer at python.org  Mon Oct  7 06:20:10 2019
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Mon, 07 Oct 2019 10:20:10 -0000
Subject: [Python-checkins] Fix a compile warning in dictobject.c (GH-16610)
Message-ID: <mailman.8.1570443610.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/ed8efd8e2c993e9d0528ffb875f73e2aefada068
commit: ed8efd8e2c993e9d0528ffb875f73e2aefada068
branch: master
author: Hai Shi <shihai1992 at gmail.com>
committer: Serhiy Storchaka <storchaka at gmail.com>
date: 2019-10-07T13:20:05+03:00
summary:

Fix a compile warning in dictobject.c (GH-16610)

files:
M Objects/dictobject.c

diff --git a/Objects/dictobject.c b/Objects/dictobject.c
index 5c3f1fb3c1a25..164fe2a273b1d 100644
--- a/Objects/dictobject.c
+++ b/Objects/dictobject.c
@@ -4186,7 +4186,6 @@ _PyDictView_Intersect(PyObject* self, PyObject *other)
     Py_ssize_t len_self;
     int rv;
     int (*dict_contains)(_PyDictViewObject *, PyObject *);
-    PyObject *tmp;
 
     /* Python interpreter swaps parameters when dict view
        is on right side of & */


From webhook-mailer at python.org  Mon Oct  7 06:28:20 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Mon, 07 Oct 2019 10:28:20 -0000
Subject: [Python-checkins] bpo-38210: Fix compiler warning in dictobject.c
 (GH-16611)
Message-ID: <mailman.9.1570444102.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/d97f1ce6dba6c4aa5614adc06ad2e0948709845c
commit: d97f1ce6dba6c4aa5614adc06ad2e0948709845c
branch: master
author: Victor Stinner <vstinner at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-07T12:28:16+02:00
summary:

bpo-38210: Fix compiler warning in dictobject.c (GH-16611)

The "tmp" variable is no longer used.

files:




From webhook-mailer at python.org  Mon Oct  7 06:57:03 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Mon, 07 Oct 2019 10:57:03 -0000
Subject: [Python-checkins] bpo-38353: Simplify calculate_pybuilddir()
 (GH-16614)
Message-ID: <mailman.10.1570445823.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/b96145a6b5d89599ebccd2111d93f5670ddae840
commit: b96145a6b5d89599ebccd2111d93f5670ddae840
branch: master
author: Victor Stinner <vstinner at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-07T12:56:59+02:00
summary:

bpo-38353: Simplify calculate_pybuilddir() (GH-16614)

Calling _Py_wfopen() is enough to check if filename is an existing
file or not. There is no need to check first isfile().

files:
M Modules/getpath.c

diff --git a/Modules/getpath.c b/Modules/getpath.c
index 993276fdc27d0..057a929019755 100644
--- a/Modules/getpath.c
+++ b/Modules/getpath.c
@@ -657,11 +657,6 @@ calculate_pybuilddir(const wchar_t *argv0_path,
         return _PyStatus_NO_MEMORY();
     }
 
-    if (!isfile(filename)) {
-        PyMem_RawFree(filename);
-        return _PyStatus_OK();
-    }
-
     FILE *fp = _Py_wfopen(filename, L"rb");
     PyMem_RawFree(filename);
     if (fp == NULL) {


From webhook-mailer at python.org  Mon Oct  7 07:57:54 2019
From: webhook-mailer at python.org (Xiang Zhang)
Date: Mon, 07 Oct 2019 11:57:54 -0000
Subject: [Python-checkins] bpo-38391: Fixing a typo for Py_DECREF (GH-16616)
Message-ID: <mailman.11.1570449476.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/038503e08ac5b10601b95d5adc2c2cab7be10163
commit: 038503e08ac5b10601b95d5adc2c2cab7be10163
branch: master
author: Krishna Oza <krishoza15sep at gmail.com>
committer: Xiang Zhang <angwerzx at 126.com>
date: 2019-10-07T19:57:45+08:00
summary:

bpo-38391: Fixing a typo for Py_DECREF (GH-16616)

files:
M Doc/extending/newtypes_tutorial.rst

diff --git a/Doc/extending/newtypes_tutorial.rst b/Doc/extending/newtypes_tutorial.rst
index 94ca747c7aeb8..b2c819c858237 100644
--- a/Doc/extending/newtypes_tutorial.rst
+++ b/Doc/extending/newtypes_tutorial.rst
@@ -182,7 +182,7 @@ set to *NULL*. ::
    Py_INCREF(&CustomType);
    if (PyModule_AddObject(m, "Custom", (PyObject *) &CustomType) < 0) {
        Py_DECREF(&CustomType);
-       PY_DECREF(m);
+       Py_DECREF(m);
        return NULL;
    }
 


From webhook-mailer at python.org  Mon Oct  7 08:01:58 2019
From: webhook-mailer at python.org (Xiang Zhang)
Date: Mon, 07 Oct 2019 12:01:58 -0000
Subject: [Python-checkins] Fix a compile warning in selectmodule.c.
 (GH-16617)
Message-ID: <mailman.12.1570449719.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/303475e87372aacb17990802538e440735a69525
commit: 303475e87372aacb17990802538e440735a69525
branch: master
author: Xiang Zhang <angwerzx at 126.com>
committer: GitHub <noreply at github.com>
date: 2019-10-07T20:01:54+08:00
summary:

Fix a compile warning in selectmodule.c. (GH-16617)

files:
M Modules/selectmodule.c

diff --git a/Modules/selectmodule.c b/Modules/selectmodule.c
index 98da7d5027f4a..79cc1b265559b 100644
--- a/Modules/selectmodule.c
+++ b/Modules/selectmodule.c
@@ -2362,7 +2362,7 @@ static PyMethodDef kqueue_queue_methods[] = {
 
 static PyType_Slot kqueue_queue_Type_slots[] = {
     {Py_tp_dealloc, kqueue_queue_dealloc},
-    {Py_tp_doc, select_kqueue__doc__},
+    {Py_tp_doc, (void*)select_kqueue__doc__},
     {Py_tp_getset, kqueue_queue_getsetlist},
     {Py_tp_methods, kqueue_queue_methods},
     {Py_tp_new, select_kqueue},


From webhook-mailer at python.org  Mon Oct  7 08:04:13 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Mon, 07 Oct 2019 12:04:13 -0000
Subject: [Python-checkins] bpo-38391: Fixing a typo for Py_DECREF (GH-16616)
Message-ID: <mailman.13.1570449854.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/77b4a659b60536caf11c547332d30677f3e8f02e
commit: 77b4a659b60536caf11c547332d30677f3e8f02e
branch: 3.7
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-07T05:04:09-07:00
summary:

bpo-38391: Fixing a typo for Py_DECREF (GH-16616)

(cherry picked from commit 038503e08ac5b10601b95d5adc2c2cab7be10163)

Co-authored-by: Krishna Oza <krishoza15sep at gmail.com>

files:
M Doc/extending/newtypes_tutorial.rst

diff --git a/Doc/extending/newtypes_tutorial.rst b/Doc/extending/newtypes_tutorial.rst
index e6448e46411bf..50be28fbcba28 100644
--- a/Doc/extending/newtypes_tutorial.rst
+++ b/Doc/extending/newtypes_tutorial.rst
@@ -182,7 +182,7 @@ set to *NULL*. ::
    Py_INCREF(&CustomType);
    if (PyModule_AddObject(m, "Custom", (PyObject *) &CustomType) < 0) {
        Py_DECREF(&CustomType);
-       PY_DECREF(m);
+       Py_DECREF(m);
        return NULL;
    }
 


From webhook-mailer at python.org  Mon Oct  7 08:05:13 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Mon, 07 Oct 2019 12:05:13 -0000
Subject: [Python-checkins] bpo-38391: Fixing a typo for Py_DECREF (GH-16616)
Message-ID: <mailman.14.1570449914.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/985ec986e864b670687b900a4d6ca977b4e9694d
commit: 985ec986e864b670687b900a4d6ca977b4e9694d
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-07T05:05:09-07:00
summary:

bpo-38391: Fixing a typo for Py_DECREF (GH-16616)

(cherry picked from commit 038503e08ac5b10601b95d5adc2c2cab7be10163)

Co-authored-by: Krishna Oza <krishoza15sep at gmail.com>

files:
M Doc/extending/newtypes_tutorial.rst

diff --git a/Doc/extending/newtypes_tutorial.rst b/Doc/extending/newtypes_tutorial.rst
index 94ca747c7aeb8..b2c819c858237 100644
--- a/Doc/extending/newtypes_tutorial.rst
+++ b/Doc/extending/newtypes_tutorial.rst
@@ -182,7 +182,7 @@ set to *NULL*. ::
    Py_INCREF(&CustomType);
    if (PyModule_AddObject(m, "Custom", (PyObject *) &CustomType) < 0) {
        Py_DECREF(&CustomType);
-       PY_DECREF(m);
+       Py_DECREF(m);
        return NULL;
    }
 


From webhook-mailer at python.org  Mon Oct  7 12:20:02 2019
From: webhook-mailer at python.org (Yury Selivanov)
Date: Mon, 07 Oct 2019 16:20:02 -0000
Subject: [Python-checkins] bpo-36356: Fix memory leak in _asynciomodule.c
 (GH-16598)
Message-ID: <mailman.15.1570465203.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/321def805abc5b7c92c7e90ca90cb2434fdab855
commit: 321def805abc5b7c92c7e90ca90cb2434fdab855
branch: master
author: Ben Harper <btharper1221 at gmail.com>
committer: Yury Selivanov <yury at magic.io>
date: 2019-10-07T12:19:58-04:00
summary:

bpo-36356: Fix memory leak in _asynciomodule.c (GH-16598)

files:
M Modules/_asynciomodule.c

diff --git a/Modules/_asynciomodule.c b/Modules/_asynciomodule.c
index b67afd41768fe..89b2fdea0f632 100644
--- a/Modules/_asynciomodule.c
+++ b/Modules/_asynciomodule.c
@@ -33,6 +33,7 @@ static PyObject *asyncio_task_repr_info_func;
 static PyObject *asyncio_InvalidStateError;
 static PyObject *asyncio_CancelledError;
 static PyObject *context_kwname;
+static int module_initialized;
 
 static PyObject *cached_running_holder;
 static volatile uint64_t cached_running_holder_tsid;
@@ -3247,6 +3248,12 @@ module_init(void)
     if (asyncio_mod == NULL) {
         goto fail;
     }
+    if (module_initialized != 0) {
+        return 0;
+    } 
+    else {
+        module_initialized = 1;
+    }
 
     current_tasks = PyDict_New();
     if (current_tasks == NULL) {


From webhook-mailer at python.org  Mon Oct  7 12:38:04 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Mon, 07 Oct 2019 16:38:04 -0000
Subject: [Python-checkins] bpo-36356: Fix memory leak in _asynciomodule.c
 (GH-16598)
Message-ID: <mailman.16.1570466285.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/13915a3100608f011b29da2f3716c990f523b631
commit: 13915a3100608f011b29da2f3716c990f523b631
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-07T09:38:00-07:00
summary:

bpo-36356: Fix memory leak in _asynciomodule.c (GH-16598)

(cherry picked from commit 321def805abc5b7c92c7e90ca90cb2434fdab855)

Co-authored-by: Ben Harper <btharper1221 at gmail.com>

files:
M Modules/_asynciomodule.c

diff --git a/Modules/_asynciomodule.c b/Modules/_asynciomodule.c
index 8eb480fb77eaa..5261ed3d4c3d0 100644
--- a/Modules/_asynciomodule.c
+++ b/Modules/_asynciomodule.c
@@ -33,6 +33,7 @@ static PyObject *asyncio_task_repr_info_func;
 static PyObject *asyncio_InvalidStateError;
 static PyObject *asyncio_CancelledError;
 static PyObject *context_kwname;
+static int module_initialized;
 
 static PyObject *cached_running_holder;
 static volatile uint64_t cached_running_holder_tsid;
@@ -3254,6 +3255,12 @@ module_init(void)
     if (asyncio_mod == NULL) {
         goto fail;
     }
+    if (module_initialized != 0) {
+        return 0;
+    } 
+    else {
+        module_initialized = 1;
+    }
 
     current_tasks = PyDict_New();
     if (current_tasks == NULL) {


From webhook-mailer at python.org  Mon Oct  7 12:42:05 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Mon, 07 Oct 2019 16:42:05 -0000
Subject: [Python-checkins] bpo-36389: _PyObject_CheckConsistency() available
 in release mode (GH-16612)
Message-ID: <mailman.17.1570466525.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/6876257eaabdb30f27ebcbd7d2557278ce2e5705
commit: 6876257eaabdb30f27ebcbd7d2557278ce2e5705
branch: master
author: Victor Stinner <vstinner at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-07T18:42:01+02:00
summary:

bpo-36389: _PyObject_CheckConsistency() available in release mode (GH-16612)

bpo-36389, bpo-38376: The _PyObject_CheckConsistency() function is
now also available in release mode. For example, it can be used to
debug a crash in the visit_decref() function of the GC.

Modify the following functions to also work in release mode:

* _PyDict_CheckConsistency()
* _PyObject_CheckConsistency()
* _PyType_CheckConsistency()
* _PyUnicode_CheckConsistency()

Other changes:

* _PyMem_IsPtrFreed(ptr) now also returns 1 if ptr is NULL
  (equals to 0).
* _PyBytesWriter_CheckConsistency() now returns 1 and is only used
  with assert().
* Reorder _PyObject_Dump() to write safe fields first, and only
  attempt to render repr() at the end.

files:
A Misc/NEWS.d/next/C API/2019-10-07-17-15-09.bpo-36389.hFX_jD.rst
M Include/cpython/unicodeobject.h
M Include/internal/pycore_object.h
M Include/internal/pycore_pymem.h
M Include/unicodeobject.h
M Lib/test/test_capi.py
M Lib/test/test_gc.py
M Modules/_testcapimodule.c
M Modules/gcmodule.c
M Objects/bytesobject.c
M Objects/dictobject.c
M Objects/object.c
M Objects/typeobject.c
M Objects/unicodeobject.c

diff --git a/Include/cpython/unicodeobject.h b/Include/cpython/unicodeobject.h
index c11503d339992..54a13e32ba22b 100644
--- a/Include/cpython/unicodeobject.h
+++ b/Include/cpython/unicodeobject.h
@@ -246,6 +246,10 @@ typedef struct {
     } data;                     /* Canonical, smallest-form Unicode buffer */
 } PyUnicodeObject;
 
+PyAPI_FUNC(int) _PyUnicode_CheckConsistency(
+    PyObject *op,
+    int check_content);
+
 /* Fast access macros */
 #define PyUnicode_WSTR_LENGTH(op) \
     (PyUnicode_IS_COMPACT_ASCII(op) ?                  \
diff --git a/Include/internal/pycore_object.h b/Include/internal/pycore_object.h
index 500078d4a1dab..0ae5995848626 100644
--- a/Include/internal/pycore_object.h
+++ b/Include/internal/pycore_object.h
@@ -11,7 +11,6 @@ extern "C" {
 #include "pycore_pystate.h"   /* _PyRuntime.gc */
 
 PyAPI_FUNC(int) _PyType_CheckConsistency(PyTypeObject *type);
-PyAPI_FUNC(int) _PyUnicode_CheckConsistency(PyObject *op, int check_content);
 PyAPI_FUNC(int) _PyDict_CheckConsistency(PyObject *mp, int check_content);
 
 /* Tell the GC to track this object.
diff --git a/Include/internal/pycore_pymem.h b/Include/internal/pycore_pymem.h
index cba4b235c3aaa..5be2d3352b1a8 100644
--- a/Include/internal/pycore_pymem.h
+++ b/Include/internal/pycore_pymem.h
@@ -155,8 +155,9 @@ PyAPI_FUNC(int) _PyMem_SetDefaultAllocator(
     PyMemAllocatorEx *old_alloc);
 
 /* Heuristic checking if a pointer value is newly allocated
-   (uninitialized) or newly freed. The pointer is not dereferenced, only the
-   pointer value is checked.
+   (uninitialized), newly freed or NULL (is equal to zero).
+
+   The pointer is not dereferenced, only the pointer value is checked.
 
    The heuristic relies on the debug hooks on Python memory allocators which
    fills newly allocated memory with CLEANBYTE (0xCD) and newly freed memory
@@ -166,11 +167,13 @@ static inline int _PyMem_IsPtrFreed(void *ptr)
 {
     uintptr_t value = (uintptr_t)ptr;
 #if SIZEOF_VOID_P == 8
-    return (value == (uintptr_t)0xCDCDCDCDCDCDCDCD
+    return (value == 0
+            || value == (uintptr_t)0xCDCDCDCDCDCDCDCD
             || value == (uintptr_t)0xDDDDDDDDDDDDDDDD
             || value == (uintptr_t)0xFDFDFDFDFDFDFDFD);
 #elif SIZEOF_VOID_P == 4
-    return (value == (uintptr_t)0xCDCDCDCD
+    return (value == 0
+            || value == (uintptr_t)0xCDCDCDCD
             || value == (uintptr_t)0xDDDDDDDD
             || value == (uintptr_t)0xFDFDFDFD);
 #else
diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h
index 6d141b37bf890..97d8cd12f6de5 100644
--- a/Include/unicodeobject.h
+++ b/Include/unicodeobject.h
@@ -1032,16 +1032,6 @@ PyAPI_FUNC(int) PyUnicode_IsIdentifier(PyObject *s);
 
 /* === Characters Type APIs =============================================== */
 
-#if defined(Py_DEBUG) && !defined(Py_LIMITED_API)
-PyAPI_FUNC(int) _PyUnicode_CheckConsistency(
-    PyObject *op,
-    int check_content);
-#elif !defined(NDEBUG)
-/* For asserts that call _PyUnicode_CheckConsistency(), which would
- * otherwise be a problem when building with asserts but without Py_DEBUG. */
-#define _PyUnicode_CheckConsistency(op, check_content) PyUnicode_Check(op)
-#endif
-
 #ifndef Py_LIMITED_API
 #  define Py_CPYTHON_UNICODEOBJECT_H
 #  include  "cpython/unicodeobject.h"
diff --git a/Lib/test/test_capi.py b/Lib/test/test_capi.py
index 79059bc7c9641..6731447af4a7e 100644
--- a/Lib/test/test_capi.py
+++ b/Lib/test/test_capi.py
@@ -719,6 +719,9 @@ def check_pyobject_is_freed(self, func_name):
         ''')
         assert_python_ok('-c', code, PYTHONMALLOC=self.PYTHONMALLOC)
 
+    def test_pyobject_null_is_freed(self):
+        self.check_pyobject_is_freed('check_pyobject_null_is_freed')
+
     def test_pyobject_uninitialized_is_freed(self):
         self.check_pyobject_is_freed('check_pyobject_uninitialized_is_freed')
 
diff --git a/Lib/test/test_gc.py b/Lib/test/test_gc.py
index 311143da91d07..8215390cb7089 100644
--- a/Lib/test/test_gc.py
+++ b/Lib/test/test_gc.py
@@ -985,16 +985,19 @@ def test_refcount_errors(self):
             br'gcmodule\.c:[0-9]+: gc_decref: Assertion "gc_get_refs\(g\) > 0" failed.')
         self.assertRegex(stderr,
             br'refcount is too small')
+        # "address : 0x7fb5062efc18"
+        # "address : 7FB5062EFC18"
+        address_regex = br'[0-9a-fA-Fx]+'
         self.assertRegex(stderr,
-            br'object  : \[1, 2, 3\]')
+            br'object address  : ' + address_regex)
         self.assertRegex(stderr,
-            br'type    : list')
+            br'object refcount : 1')
         self.assertRegex(stderr,
-            br'refcount: 1')
-        # "address : 0x7fb5062efc18"
-        # "address : 7FB5062EFC18"
+            br'object type     : ' + address_regex)
+        self.assertRegex(stderr,
+            br'object type name: list')
         self.assertRegex(stderr,
-            br'address : [0-9a-fA-Fx]+')
+            br'object repr     : \[1, 2, 3\]')
 
 
 class GCTogglingTests(unittest.TestCase):
diff --git a/Misc/NEWS.d/next/C API/2019-10-07-17-15-09.bpo-36389.hFX_jD.rst b/Misc/NEWS.d/next/C API/2019-10-07-17-15-09.bpo-36389.hFX_jD.rst
new file mode 100644
index 0000000000000..6c42882cdbaa6
--- /dev/null
+++ b/Misc/NEWS.d/next/C API/2019-10-07-17-15-09.bpo-36389.hFX_jD.rst	
@@ -0,0 +1,3 @@
+The ``_PyObject_CheckConsistency()`` function is now also available in release
+mode. For example, it can be used to debug a crash in the ``visit_decref()``
+function of the GC.
diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c
index 3f3ed664880af..c009d254da8f1 100644
--- a/Modules/_testcapimodule.c
+++ b/Modules/_testcapimodule.c
@@ -4611,6 +4611,14 @@ test_pyobject_is_freed(const char *test_name, PyObject *op)
 }
 
 
+static PyObject*
+check_pyobject_null_is_freed(PyObject *self, PyObject *Py_UNUSED(args))
+{
+    PyObject *op = NULL;
+    return test_pyobject_is_freed("check_pyobject_null_is_freed", op);
+}
+
+
 static PyObject*
 check_pyobject_uninitialized_is_freed(PyObject *self, PyObject *Py_UNUSED(args))
 {
@@ -5475,6 +5483,7 @@ static PyMethodDef TestMethods[] = {
     {"pymem_api_misuse", pymem_api_misuse, METH_NOARGS},
     {"pymem_malloc_without_gil", pymem_malloc_without_gil, METH_NOARGS},
     {"pymem_getallocatorsname", test_pymem_getallocatorsname, METH_NOARGS},
+    {"check_pyobject_null_is_freed", check_pyobject_null_is_freed, METH_NOARGS},
     {"check_pyobject_uninitialized_is_freed", check_pyobject_uninitialized_is_freed, METH_NOARGS},
     {"check_pyobject_forbidden_bytes_is_freed", check_pyobject_forbidden_bytes_is_freed, METH_NOARGS},
     {"check_pyobject_freed_is_freed", check_pyobject_freed_is_freed, METH_NOARGS},
diff --git a/Modules/gcmodule.c b/Modules/gcmodule.c
index b95f676663a39..107ce39967e20 100644
--- a/Modules/gcmodule.c
+++ b/Modules/gcmodule.c
@@ -375,7 +375,6 @@ update_refs(PyGC_Head *containers)
 static int
 visit_decref(PyObject *op, void *data)
 {
-    assert(op != NULL);
     _PyObject_ASSERT(op, !_PyObject_IsFreed(op));
 
     if (PyObject_IS_GC(op)) {
diff --git a/Objects/bytesobject.c b/Objects/bytesobject.c
index 7df93440a14c6..4223dc9977ed9 100644
--- a/Objects/bytesobject.c
+++ b/Objects/bytesobject.c
@@ -3206,10 +3206,10 @@ _PyBytesWriter_GetSize(_PyBytesWriter *writer, char *str)
     return str - start;
 }
 
-Py_LOCAL_INLINE(void)
+#ifndef NDEBUG
+Py_LOCAL_INLINE(int)
 _PyBytesWriter_CheckConsistency(_PyBytesWriter *writer, char *str)
 {
-#ifdef Py_DEBUG
     char *start, *end;
 
     if (writer->use_small_buffer) {
@@ -3239,15 +3239,16 @@ _PyBytesWriter_CheckConsistency(_PyBytesWriter *writer, char *str)
     end = start + writer->allocated;
     assert(str != NULL);
     assert(start <= str && str <= end);
-#endif
+    return 1;
 }
+#endif
 
 void*
 _PyBytesWriter_Resize(_PyBytesWriter *writer, void *str, Py_ssize_t size)
 {
     Py_ssize_t allocated, pos;
 
-    _PyBytesWriter_CheckConsistency(writer, str);
+    assert(_PyBytesWriter_CheckConsistency(writer, str));
     assert(writer->allocated < size);
 
     allocated = size;
@@ -3303,7 +3304,7 @@ _PyBytesWriter_Resize(_PyBytesWriter *writer, void *str, Py_ssize_t size)
     writer->allocated = allocated;
 
     str = _PyBytesWriter_AsString(writer) + pos;
-    _PyBytesWriter_CheckConsistency(writer, str);
+    assert(_PyBytesWriter_CheckConsistency(writer, str));
     return str;
 
 error:
@@ -3316,7 +3317,7 @@ _PyBytesWriter_Prepare(_PyBytesWriter *writer, void *str, Py_ssize_t size)
 {
     Py_ssize_t new_min_size;
 
-    _PyBytesWriter_CheckConsistency(writer, str);
+    assert(_PyBytesWriter_CheckConsistency(writer, str));
     assert(size >= 0);
 
     if (size == 0) {
@@ -3377,7 +3378,7 @@ _PyBytesWriter_Finish(_PyBytesWriter *writer, void *str)
     Py_ssize_t size;
     PyObject *result;
 
-    _PyBytesWriter_CheckConsistency(writer, str);
+    assert(_PyBytesWriter_CheckConsistency(writer, str));
 
     size = _PyBytesWriter_GetSize(writer, str);
     if (size == 0 && !writer->use_bytearray) {
diff --git a/Objects/dictobject.c b/Objects/dictobject.c
index 164fe2a273b1d..99908a8a375d2 100644
--- a/Objects/dictobject.c
+++ b/Objects/dictobject.c
@@ -459,23 +459,26 @@ static PyObject *empty_values[1] = { NULL };
 int
 _PyDict_CheckConsistency(PyObject *op, int check_content)
 {
-#ifndef NDEBUG
-    _PyObject_ASSERT(op, PyDict_Check(op));
+#define CHECK(expr) \
+    do { if (!(expr)) { _PyObject_ASSERT_FAILED_MSG(op, Py_STRINGIFY(expr)); } } while (0)
+
+    assert(op != NULL);
+    CHECK(PyDict_Check(op));
     PyDictObject *mp = (PyDictObject *)op;
 
     PyDictKeysObject *keys = mp->ma_keys;
     int splitted = _PyDict_HasSplitTable(mp);
     Py_ssize_t usable = USABLE_FRACTION(keys->dk_size);
 
-    _PyObject_ASSERT(op, 0 <= mp->ma_used && mp->ma_used <= usable);
-    _PyObject_ASSERT(op, IS_POWER_OF_2(keys->dk_size));
-    _PyObject_ASSERT(op, 0 <= keys->dk_usable && keys->dk_usable <= usable);
-    _PyObject_ASSERT(op, 0 <= keys->dk_nentries && keys->dk_nentries <= usable);
-    _PyObject_ASSERT(op, keys->dk_usable + keys->dk_nentries <= usable);
+    CHECK(0 <= mp->ma_used && mp->ma_used <= usable);
+    CHECK(IS_POWER_OF_2(keys->dk_size));
+    CHECK(0 <= keys->dk_usable && keys->dk_usable <= usable);
+    CHECK(0 <= keys->dk_nentries && keys->dk_nentries <= usable);
+    CHECK(keys->dk_usable + keys->dk_nentries <= usable);
 
     if (!splitted) {
         /* combined table */
-        _PyObject_ASSERT(op, keys->dk_refcnt == 1);
+        CHECK(keys->dk_refcnt == 1);
     }
 
     if (check_content) {
@@ -484,7 +487,7 @@ _PyDict_CheckConsistency(PyObject *op, int check_content)
 
         for (i=0; i < keys->dk_size; i++) {
             Py_ssize_t ix = dictkeys_get_index(keys, i);
-            _PyObject_ASSERT(op, DKIX_DUMMY <= ix && ix <= usable);
+            CHECK(DKIX_DUMMY <= ix && ix <= usable);
         }
 
         for (i=0; i < usable; i++) {
@@ -494,32 +497,33 @@ _PyDict_CheckConsistency(PyObject *op, int check_content)
             if (key != NULL) {
                 if (PyUnicode_CheckExact(key)) {
                     Py_hash_t hash = ((PyASCIIObject *)key)->hash;
-                    _PyObject_ASSERT(op, hash != -1);
-                    _PyObject_ASSERT(op, entry->me_hash == hash);
+                    CHECK(hash != -1);
+                    CHECK(entry->me_hash == hash);
                 }
                 else {
                     /* test_dict fails if PyObject_Hash() is called again */
-                    _PyObject_ASSERT(op, entry->me_hash != -1);
+                    CHECK(entry->me_hash != -1);
                 }
                 if (!splitted) {
-                    _PyObject_ASSERT(op, entry->me_value != NULL);
+                    CHECK(entry->me_value != NULL);
                 }
             }
 
             if (splitted) {
-                _PyObject_ASSERT(op, entry->me_value == NULL);
+                CHECK(entry->me_value == NULL);
             }
         }
 
         if (splitted) {
             /* splitted table */
             for (i=0; i < mp->ma_used; i++) {
-                _PyObject_ASSERT(op, mp->ma_values[i] != NULL);
+                CHECK(mp->ma_values[i] != NULL);
             }
         }
     }
-#endif
     return 1;
+
+#undef CHECK
 }
 
 
diff --git a/Objects/object.c b/Objects/object.c
index 43ed4c59a71ba..e94480f7e998f 100644
--- a/Objects/object.c
+++ b/Objects/object.c
@@ -25,13 +25,14 @@ _Py_IDENTIFIER(__isabstractmethod__);
 int
 _PyObject_CheckConsistency(PyObject *op, int check_content)
 {
-    _PyObject_ASSERT(op, op != NULL);
-    _PyObject_ASSERT(op, !_PyObject_IsFreed(op));
-    _PyObject_ASSERT(op, Py_REFCNT(op) >= 1);
+#define CHECK(expr) \
+    do { if (!(expr)) { _PyObject_ASSERT_FAILED_MSG(op, Py_STRINGIFY(expr)); } } while (0)
 
-    PyTypeObject *type = op->ob_type;
-    _PyObject_ASSERT(op, type != NULL);
-    _PyType_CheckConsistency(type);
+    CHECK(!_PyObject_IsFreed(op));
+    CHECK(Py_REFCNT(op) >= 1);
+
+    CHECK(op->ob_type != NULL);
+    _PyType_CheckConsistency(op->ob_type);
 
     if (PyUnicode_Check(op)) {
         _PyUnicode_CheckConsistency(op, check_content);
@@ -40,6 +41,8 @@ _PyObject_CheckConsistency(PyObject *op, int check_content)
         _PyDict_CheckConsistency(op, check_content);
     }
     return 1;
+
+#undef CHECK
 }
 
 
@@ -463,41 +466,41 @@ _PyObject_IsFreed(PyObject *op)
 void
 _PyObject_Dump(PyObject* op)
 {
-    if (op == NULL) {
-        fprintf(stderr, "<object at NULL>\n");
-        fflush(stderr);
-        return;
-    }
-
     if (_PyObject_IsFreed(op)) {
         /* It seems like the object memory has been freed:
            don't access it to prevent a segmentation fault. */
         fprintf(stderr, "<object at %p is freed>\n", op);
+        fflush(stderr);
         return;
     }
 
-    PyGILState_STATE gil;
-    PyObject *error_type, *error_value, *error_traceback;
+    /* first, write fields which are the least likely to crash */
+    fprintf(stderr, "object address  : %p\n", (void *)op);
+    /* XXX(twouters) cast refcount to long until %zd is
+       universally available */
+    fprintf(stderr, "object refcount : %ld\n", (long)op->ob_refcnt);
+    fflush(stderr);
+
+    PyTypeObject *type = Py_TYPE(op);
+    fprintf(stderr, "object type     : %p\n", type);
+    fprintf(stderr, "object type name: %s\n",
+            type==NULL ? "NULL" : type->tp_name);
 
-    fprintf(stderr, "object  : ");
+    /* the most dangerous part */
+    fprintf(stderr, "object repr     : ");
     fflush(stderr);
-    gil = PyGILState_Ensure();
 
+    PyGILState_STATE gil = PyGILState_Ensure();
+    PyObject *error_type, *error_value, *error_traceback;
     PyErr_Fetch(&error_type, &error_value, &error_traceback);
+
     (void)PyObject_Print(op, stderr, 0);
     fflush(stderr);
-    PyErr_Restore(error_type, error_value, error_traceback);
 
+    PyErr_Restore(error_type, error_value, error_traceback);
     PyGILState_Release(gil);
-    /* XXX(twouters) cast refcount to long until %zd is
-       universally available */
-    fprintf(stderr, "\n"
-        "type    : %s\n"
-        "refcount: %ld\n"
-        "address : %p\n",
-        Py_TYPE(op)==NULL ? "NULL" : Py_TYPE(op)->tp_name,
-        (long)op->ob_refcnt,
-        (void *)op);
+
+    fprintf(stderr, "\n");
     fflush(stderr);
 }
 
@@ -2146,6 +2149,7 @@ _PyObject_AssertFailed(PyObject *obj, const char *expr, const char *msg,
         fprintf(stderr, "%s: ", function);
     }
     fflush(stderr);
+
     if (expr) {
         fprintf(stderr, "Assertion \"%s\" failed", expr);
     }
@@ -2153,26 +2157,18 @@ _PyObject_AssertFailed(PyObject *obj, const char *expr, const char *msg,
         fprintf(stderr, "Assertion failed");
     }
     fflush(stderr);
+
     if (msg) {
         fprintf(stderr, ": %s", msg);
     }
     fprintf(stderr, "\n");
     fflush(stderr);
 
-    if (obj == NULL) {
-        fprintf(stderr, "<object at NULL>\n");
-    }
-    else if (_PyObject_IsFreed(obj)) {
+    if (_PyObject_IsFreed(obj)) {
         /* It seems like the object memory has been freed:
            don't access it to prevent a segmentation fault. */
         fprintf(stderr, "<object at %p is freed>\n", obj);
-    }
-    else if (Py_TYPE(obj) == NULL) {
-        fprintf(stderr, "<object at %p: ob_type=NULL>\n", obj);
-    }
-    else if (_PyObject_IsFreed((PyObject *)Py_TYPE(obj))) {
-        fprintf(stderr, "<object at %p: type at %p is freed>\n",
-                obj, (void *)Py_TYPE(obj));
+        fflush(stderr);
     }
     else {
         /* Display the traceback where the object has been allocated.
@@ -2192,7 +2188,6 @@ _PyObject_AssertFailed(PyObject *obj, const char *expr, const char *msg,
            try to provide any extra info we can: */
         _PyObject_Dump(obj);
     }
-    fflush(stderr);
 
     Py_FatalError("_PyObject_AssertFailed");
 }
diff --git a/Objects/typeobject.c b/Objects/typeobject.c
index 94a4da22d6793..890246e14a695 100644
--- a/Objects/typeobject.c
+++ b/Objects/typeobject.c
@@ -137,22 +137,24 @@ skip_signature(const char *doc)
 int
 _PyType_CheckConsistency(PyTypeObject *type)
 {
-#define ASSERT(expr) _PyObject_ASSERT((PyObject *)type, (expr))
+#define CHECK(expr) \
+    do { if (!(expr)) { _PyObject_ASSERT_FAILED_MSG((PyObject *)type, Py_STRINGIFY(expr)); } } while (0)
+
+    CHECK(!_PyObject_IsFreed((PyObject *)type));
 
     if (!(type->tp_flags & Py_TPFLAGS_READY)) {
-        /* don't check types before PyType_Ready() */
+        /* don't check static types before PyType_Ready() */
         return 1;
     }
 
-    ASSERT(!_PyObject_IsFreed((PyObject *)type));
-    ASSERT(Py_REFCNT(type) >= 1);
-    ASSERT(PyType_Check(type));
+    CHECK(Py_REFCNT(type) >= 1);
+    CHECK(PyType_Check(type));
 
-    ASSERT(!(type->tp_flags & Py_TPFLAGS_READYING));
-    ASSERT(type->tp_dict != NULL);
+    CHECK(!(type->tp_flags & Py_TPFLAGS_READYING));
+    CHECK(type->tp_dict != NULL);
 
     return 1;
-#undef ASSERT
+#undef CHECK
 }
 
 static const char *
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
index 0526225ca3e44..8d442fb6d82b8 100644
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -487,65 +487,63 @@ PyUnicode_GetMax(void)
 int
 _PyUnicode_CheckConsistency(PyObject *op, int check_content)
 {
+#define CHECK(expr) \
+    do { if (!(expr)) { _PyObject_ASSERT_FAILED_MSG(op, Py_STRINGIFY(expr)); } } while (0)
+
     PyASCIIObject *ascii;
     unsigned int kind;
 
-    _PyObject_ASSERT(op, PyUnicode_Check(op));
+    assert(op != NULL);
+    CHECK(PyUnicode_Check(op));
 
     ascii = (PyASCIIObject *)op;
     kind = ascii->state.kind;
 
     if (ascii->state.ascii == 1 && ascii->state.compact == 1) {
-        _PyObject_ASSERT(op, kind == PyUnicode_1BYTE_KIND);
-        _PyObject_ASSERT(op, ascii->state.ready == 1);
+        CHECK(kind == PyUnicode_1BYTE_KIND);
+        CHECK(ascii->state.ready == 1);
     }
     else {
         PyCompactUnicodeObject *compact = (PyCompactUnicodeObject *)op;
-#ifndef NDEBUG
         void *data;
-#endif
 
         if (ascii->state.compact == 1) {
-#ifndef NDEBUG
             data = compact + 1;
-#endif
-            _PyObject_ASSERT(op, kind == PyUnicode_1BYTE_KIND
+            CHECK(kind == PyUnicode_1BYTE_KIND
                                  || kind == PyUnicode_2BYTE_KIND
                                  || kind == PyUnicode_4BYTE_KIND);
-            _PyObject_ASSERT(op, ascii->state.ascii == 0);
-            _PyObject_ASSERT(op, ascii->state.ready == 1);
-            _PyObject_ASSERT(op, compact->utf8 != data);
+            CHECK(ascii->state.ascii == 0);
+            CHECK(ascii->state.ready == 1);
+            CHECK(compact->utf8 != data);
         }
         else {
-#ifndef NDEBUG
             PyUnicodeObject *unicode = (PyUnicodeObject *)op;
 
             data = unicode->data.any;
-#endif
             if (kind == PyUnicode_WCHAR_KIND) {
-                _PyObject_ASSERT(op, ascii->length == 0);
-                _PyObject_ASSERT(op, ascii->hash == -1);
-                _PyObject_ASSERT(op, ascii->state.compact == 0);
-                _PyObject_ASSERT(op, ascii->state.ascii == 0);
-                _PyObject_ASSERT(op, ascii->state.ready == 0);
-                _PyObject_ASSERT(op, ascii->state.interned == SSTATE_NOT_INTERNED);
-                _PyObject_ASSERT(op, ascii->wstr != NULL);
-                _PyObject_ASSERT(op, data == NULL);
-                _PyObject_ASSERT(op, compact->utf8 == NULL);
+                CHECK(ascii->length == 0);
+                CHECK(ascii->hash == -1);
+                CHECK(ascii->state.compact == 0);
+                CHECK(ascii->state.ascii == 0);
+                CHECK(ascii->state.ready == 0);
+                CHECK(ascii->state.interned == SSTATE_NOT_INTERNED);
+                CHECK(ascii->wstr != NULL);
+                CHECK(data == NULL);
+                CHECK(compact->utf8 == NULL);
             }
             else {
-                _PyObject_ASSERT(op, kind == PyUnicode_1BYTE_KIND
+                CHECK(kind == PyUnicode_1BYTE_KIND
                                      || kind == PyUnicode_2BYTE_KIND
                                      || kind == PyUnicode_4BYTE_KIND);
-                _PyObject_ASSERT(op, ascii->state.compact == 0);
-                _PyObject_ASSERT(op, ascii->state.ready == 1);
-                _PyObject_ASSERT(op, data != NULL);
+                CHECK(ascii->state.compact == 0);
+                CHECK(ascii->state.ready == 1);
+                CHECK(data != NULL);
                 if (ascii->state.ascii) {
-                    _PyObject_ASSERT(op, compact->utf8 == data);
-                    _PyObject_ASSERT(op, compact->utf8_length == ascii->length);
+                    CHECK(compact->utf8 == data);
+                    CHECK(compact->utf8_length == ascii->length);
                 }
                 else
-                    _PyObject_ASSERT(op, compact->utf8 != data);
+                    CHECK(compact->utf8 != data);
             }
         }
         if (kind != PyUnicode_WCHAR_KIND) {
@@ -557,16 +555,16 @@ _PyUnicode_CheckConsistency(PyObject *op, int check_content)
 #endif
                )
             {
-                _PyObject_ASSERT(op, ascii->wstr == data);
-                _PyObject_ASSERT(op, compact->wstr_length == ascii->length);
+                CHECK(ascii->wstr == data);
+                CHECK(compact->wstr_length == ascii->length);
             } else
-                _PyObject_ASSERT(op, ascii->wstr != data);
+                CHECK(ascii->wstr != data);
         }
 
         if (compact->utf8 == NULL)
-            _PyObject_ASSERT(op, compact->utf8_length == 0);
+            CHECK(compact->utf8_length == 0);
         if (ascii->wstr == NULL)
-            _PyObject_ASSERT(op, compact->wstr_length == 0);
+            CHECK(compact->wstr_length == 0);
     }
 
     /* check that the best kind is used: O(n) operation */
@@ -585,23 +583,25 @@ _PyUnicode_CheckConsistency(PyObject *op, int check_content)
         }
         if (kind == PyUnicode_1BYTE_KIND) {
             if (ascii->state.ascii == 0) {
-                _PyObject_ASSERT(op, maxchar >= 128);
-                _PyObject_ASSERT(op, maxchar <= 255);
+                CHECK(maxchar >= 128);
+                CHECK(maxchar <= 255);
             }
             else
-                _PyObject_ASSERT(op, maxchar < 128);
+                CHECK(maxchar < 128);
         }
         else if (kind == PyUnicode_2BYTE_KIND) {
-            _PyObject_ASSERT(op, maxchar >= 0x100);
-            _PyObject_ASSERT(op, maxchar <= 0xFFFF);
+            CHECK(maxchar >= 0x100);
+            CHECK(maxchar <= 0xFFFF);
         }
         else {
-            _PyObject_ASSERT(op, maxchar >= 0x10000);
-            _PyObject_ASSERT(op, maxchar <= MAX_UNICODE);
+            CHECK(maxchar >= 0x10000);
+            CHECK(maxchar <= MAX_UNICODE);
         }
-        _PyObject_ASSERT(op, PyUnicode_READ(kind, data, ascii->length) == 0);
+        CHECK(PyUnicode_READ(kind, data, ascii->length) == 0);
     }
     return 1;
+
+#undef CHECK
 }
 
 


From webhook-mailer at python.org  Mon Oct  7 15:12:20 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Mon, 07 Oct 2019 19:12:20 -0000
Subject: [Python-checkins] bpo-38376: Fix _PyUnicode_CheckConsistency()
 definition (GH-16623)
Message-ID: <mailman.18.1570475540.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/435b2eeb7bc5b8ec264e900b38bcb3d76552d777
commit: 435b2eeb7bc5b8ec264e900b38bcb3d76552d777
branch: 3.8
author: Victor Stinner <vstinner at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-07T21:12:05+02:00
summary:

bpo-38376: Fix _PyUnicode_CheckConsistency() definition (GH-16623)

Always define _PyUnicode_CheckConsistency() in the CPython C API.

files:
M Include/cpython/unicodeobject.h
M Include/internal/pycore_object.h
M Include/unicodeobject.h

diff --git a/Include/cpython/unicodeobject.h b/Include/cpython/unicodeobject.h
index c11503d339992..54a13e32ba22b 100644
--- a/Include/cpython/unicodeobject.h
+++ b/Include/cpython/unicodeobject.h
@@ -246,6 +246,10 @@ typedef struct {
     } data;                     /* Canonical, smallest-form Unicode buffer */
 } PyUnicodeObject;
 
+PyAPI_FUNC(int) _PyUnicode_CheckConsistency(
+    PyObject *op,
+    int check_content);
+
 /* Fast access macros */
 #define PyUnicode_WSTR_LENGTH(op) \
     (PyUnicode_IS_COMPACT_ASCII(op) ?                  \
diff --git a/Include/internal/pycore_object.h b/Include/internal/pycore_object.h
index 81548f819198e..7418c6936b8f0 100644
--- a/Include/internal/pycore_object.h
+++ b/Include/internal/pycore_object.h
@@ -11,7 +11,6 @@ extern "C" {
 #include "pycore_pystate.h"   /* _PyRuntime */
 
 PyAPI_FUNC(int) _PyType_CheckConsistency(PyTypeObject *type);
-PyAPI_FUNC(int) _PyUnicode_CheckConsistency(PyObject *op, int check_content);
 PyAPI_FUNC(int) _PyDict_CheckConsistency(PyObject *mp, int check_content);
 
 /* Tell the GC to track this object.
diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h
index 6d141b37bf890..97d8cd12f6de5 100644
--- a/Include/unicodeobject.h
+++ b/Include/unicodeobject.h
@@ -1032,16 +1032,6 @@ PyAPI_FUNC(int) PyUnicode_IsIdentifier(PyObject *s);
 
 /* === Characters Type APIs =============================================== */
 
-#if defined(Py_DEBUG) && !defined(Py_LIMITED_API)
-PyAPI_FUNC(int) _PyUnicode_CheckConsistency(
-    PyObject *op,
-    int check_content);
-#elif !defined(NDEBUG)
-/* For asserts that call _PyUnicode_CheckConsistency(), which would
- * otherwise be a problem when building with asserts but without Py_DEBUG. */
-#define _PyUnicode_CheckConsistency(op, check_content) PyUnicode_Check(op)
-#endif
-
 #ifndef Py_LIMITED_API
 #  define Py_CPYTHON_UNICODEOBJECT_H
 #  include  "cpython/unicodeobject.h"


From webhook-mailer at python.org  Mon Oct  7 16:31:47 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Mon, 07 Oct 2019 20:31:47 -0000
Subject: [Python-checkins] bpo-36389: Fix _PyBytesWriter in release mode
 (GH-16624)
Message-ID: <mailman.19.1570480308.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/60ec6efd96d95476fe5e38c491491add04f026e5
commit: 60ec6efd96d95476fe5e38c491491add04f026e5
branch: master
author: Victor Stinner <vstinner at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-07T22:31:42+02:00
summary:

bpo-36389: Fix _PyBytesWriter in release mode (GH-16624)

Fix _PyBytesWriter API when Python is built in release mode with
assertions.

files:
M Include/internal/pycore_pymem.h
M Objects/bytesobject.c
M Objects/obmalloc.c

diff --git a/Include/internal/pycore_pymem.h b/Include/internal/pycore_pymem.h
index 5be2d3352b1a8..d61ff6ee4b77d 100644
--- a/Include/internal/pycore_pymem.h
+++ b/Include/internal/pycore_pymem.h
@@ -154,6 +154,21 @@ PyAPI_FUNC(int) _PyMem_SetDefaultAllocator(
     PyMemAllocatorDomain domain,
     PyMemAllocatorEx *old_alloc);
 
+/* Special bytes broadcast into debug memory blocks at appropriate times.
+   Strings of these are unlikely to be valid addresses, floats, ints or
+   7-bit ASCII.
+
+   - PYMEM_CLEANBYTE: clean (newly allocated) memory
+   - PYMEM_DEADBYTE dead (newly freed) memory
+   - PYMEM_FORBIDDENBYTE: untouchable bytes at each end of a block
+
+   Byte patterns 0xCB, 0xBB and 0xFB have been replaced with 0xCD, 0xDD and
+   0xFD to use the same values than Windows CRT debug malloc() and free().
+   If modified, _PyMem_IsPtrFreed() should be updated as well. */
+#define PYMEM_CLEANBYTE      0xCD
+#define PYMEM_DEADBYTE       0xDD
+#define PYMEM_FORBIDDENBYTE  0xFD
+
 /* Heuristic checking if a pointer value is newly allocated
    (uninitialized), newly freed or NULL (is equal to zero).
 
diff --git a/Objects/bytesobject.c b/Objects/bytesobject.c
index 4223dc9977ed9..f9823f18e8699 100644
--- a/Objects/bytesobject.c
+++ b/Objects/bytesobject.c
@@ -667,9 +667,6 @@ _PyBytes_FormatEx(const char *format, Py_ssize_t format_len,
             Py_ssize_t len = 0;
             char onechar; /* For byte_converter() */
             Py_ssize_t alloc;
-#ifdef Py_DEBUG
-            char *before;
-#endif
 
             fmt++;
             if (*fmt == '%') {
@@ -981,8 +978,8 @@ _PyBytes_FormatEx(const char *format, Py_ssize_t format_len,
                 if (res == NULL)
                     goto error;
             }
-#ifdef Py_DEBUG
-            before = res;
+#ifndef NDEBUG
+            char *before = res;
 #endif
 
             /* Write the sign if needed */
@@ -1047,7 +1044,7 @@ _PyBytes_FormatEx(const char *format, Py_ssize_t format_len,
             }
             Py_XDECREF(temp);
 
-#ifdef Py_DEBUG
+#ifndef NDEBUG
             /* check that we computed the exact size for this write */
             assert((res - before) == alloc);
 #endif
@@ -3168,8 +3165,9 @@ _PyBytesWriter_Init(_PyBytesWriter *writer)
 {
     /* Set all attributes before small_buffer to 0 */
     memset(writer, 0, offsetof(_PyBytesWriter, small_buffer));
-#ifdef Py_DEBUG
-    memset(writer->small_buffer, 0xCB, sizeof(writer->small_buffer));
+#ifndef NDEBUG
+    memset(writer->small_buffer, PYMEM_CLEANBYTE,
+           sizeof(writer->small_buffer));
 #endif
 }
 
@@ -3297,8 +3295,9 @@ _PyBytesWriter_Resize(_PyBytesWriter *writer, void *str, Py_ssize_t size)
         }
 
         writer->use_small_buffer = 0;
-#ifdef Py_DEBUG
-        memset(writer->small_buffer, 0xDB, sizeof(writer->small_buffer));
+#ifndef NDEBUG
+        memset(writer->small_buffer, PYMEM_CLEANBYTE,
+               sizeof(writer->small_buffer));
 #endif
     }
     writer->allocated = allocated;
@@ -3350,7 +3349,7 @@ _PyBytesWriter_Alloc(_PyBytesWriter *writer, Py_ssize_t size)
     assert(size >= 0);
 
     writer->use_small_buffer = 1;
-#ifdef Py_DEBUG
+#ifndef NDEBUG
     writer->allocated = sizeof(writer->small_buffer) - 1;
     /* In debug mode, don't use the full small buffer because it is less
        efficient than bytes and bytearray objects to detect buffer underflow
diff --git a/Objects/obmalloc.c b/Objects/obmalloc.c
index 40c098ddc30ae..50701dbd38454 100644
--- a/Objects/obmalloc.c
+++ b/Objects/obmalloc.c
@@ -2031,20 +2031,6 @@ _Py_GetAllocatedBlocks(void)
  * it wraps a real allocator, adding extra debugging info to the memory blocks.
  */
 
-/* Special bytes broadcast into debug memory blocks at appropriate times.
- * Strings of these are unlikely to be valid addresses, floats, ints or
- * 7-bit ASCII. If modified, _PyMem_IsPtrFreed() should be updated as well.
- *
- * Byte patterns 0xCB, 0xBB and 0xFB have been replaced with 0xCD, 0xDD and
- * 0xFD to use the same values than Windows CRT debug malloc() and free().
- */
-#undef CLEANBYTE
-#undef DEADBYTE
-#undef FORBIDDENBYTE
-#define CLEANBYTE      0xCD    /* clean (newly allocated) memory */
-#define DEADBYTE       0xDD    /* dead (newly freed) memory */
-#define FORBIDDENBYTE  0xFD    /* untouchable bytes at each end of a block */
-
 /* Uncomment this define to add the "serialno" field */
 /* #define PYMEM_DEBUG_SERIALNO */
 
@@ -2106,14 +2092,14 @@ p[0: S]
 p[S]
     API ID.  See PEP 445.  This is a character, but seems undocumented.
 p[S+1: 2*S]
-    Copies of FORBIDDENBYTE.  Used to catch under- writes and reads.
+    Copies of PYMEM_FORBIDDENBYTE.  Used to catch under- writes and reads.
 p[2*S: 2*S+n]
-    The requested memory, filled with copies of CLEANBYTE.
+    The requested memory, filled with copies of PYMEM_CLEANBYTE.
     Used to catch reference to uninitialized memory.
     &p[2*S] is returned.  Note that this is 8-byte aligned if pymalloc
     handled the request itself.
 p[2*S+n: 2*S+n+S]
-    Copies of FORBIDDENBYTE.  Used to catch over- writes and reads.
+    Copies of PYMEM_FORBIDDENBYTE.  Used to catch over- writes and reads.
 p[2*S+n+S: 2*S+n+2*S]
     A serial number, incremented by 1 on each call to _PyMem_DebugMalloc
     and _PyMem_DebugRealloc.
@@ -2170,15 +2156,15 @@ _PyMem_DebugRawAlloc(int use_calloc, void *ctx, size_t nbytes)
     /* at p, write size (SST bytes), id (1 byte), pad (SST-1 bytes) */
     write_size_t(p, nbytes);
     p[SST] = (uint8_t)api->api_id;
-    memset(p + SST + 1, FORBIDDENBYTE, SST-1);
+    memset(p + SST + 1, PYMEM_FORBIDDENBYTE, SST-1);
 
     if (nbytes > 0 && !use_calloc) {
-        memset(data, CLEANBYTE, nbytes);
+        memset(data, PYMEM_CLEANBYTE, nbytes);
     }
 
     /* at tail, write pad (SST bytes) and serialno (SST bytes) */
     tail = data + nbytes;
-    memset(tail, FORBIDDENBYTE, SST);
+    memset(tail, PYMEM_FORBIDDENBYTE, SST);
 #ifdef PYMEM_DEBUG_SERIALNO
     write_size_t(tail + SST, serialno);
 #endif
@@ -2204,7 +2190,7 @@ _PyMem_DebugRawCalloc(void *ctx, size_t nelem, size_t elsize)
 
 /* The debug free first checks the 2*SST bytes on each end for sanity (in
    particular, that the FORBIDDENBYTEs with the api ID are still intact).
-   Then fills the original bytes with DEADBYTE.
+   Then fills the original bytes with PYMEM_DEADBYTE.
    Then calls the underlying free.
 */
 static void
@@ -2222,7 +2208,7 @@ _PyMem_DebugRawFree(void *ctx, void *p)
     _PyMem_DebugCheckAddress(api->api_id, p);
     nbytes = read_size_t(q);
     nbytes += PYMEM_DEBUG_EXTRA_BYTES;
-    memset(q, DEADBYTE, nbytes);
+    memset(q, PYMEM_DEADBYTE, nbytes);
     api->alloc.free(api->alloc.ctx, q);
 }
 
@@ -2264,14 +2250,14 @@ _PyMem_DebugRawRealloc(void *ctx, void *p, size_t nbytes)
      */
     if (original_nbytes <= sizeof(save)) {
         memcpy(save, data, original_nbytes);
-        memset(data - 2 * SST, DEADBYTE,
+        memset(data - 2 * SST, PYMEM_DEADBYTE,
                original_nbytes + PYMEM_DEBUG_EXTRA_BYTES);
     }
     else {
         memcpy(save, data, ERASED_SIZE);
-        memset(head, DEADBYTE, ERASED_SIZE + 2 * SST);
+        memset(head, PYMEM_DEADBYTE, ERASED_SIZE + 2 * SST);
         memcpy(&save[ERASED_SIZE], tail - ERASED_SIZE, ERASED_SIZE);
-        memset(tail - ERASED_SIZE, DEADBYTE,
+        memset(tail - ERASED_SIZE, PYMEM_DEADBYTE,
                ERASED_SIZE + PYMEM_DEBUG_EXTRA_BYTES - 2 * SST);
     }
 
@@ -2293,10 +2279,10 @@ _PyMem_DebugRawRealloc(void *ctx, void *p, size_t nbytes)
 
     write_size_t(head, nbytes);
     head[SST] = (uint8_t)api->api_id;
-    memset(head + SST + 1, FORBIDDENBYTE, SST-1);
+    memset(head + SST + 1, PYMEM_FORBIDDENBYTE, SST-1);
 
     tail = data + nbytes;
-    memset(tail, FORBIDDENBYTE, SST);
+    memset(tail, PYMEM_FORBIDDENBYTE, SST);
 #ifdef PYMEM_DEBUG_SERIALNO
     write_size_t(tail + SST, block_serialno);
 #endif
@@ -2320,7 +2306,8 @@ _PyMem_DebugRawRealloc(void *ctx, void *p, size_t nbytes)
 
     if (nbytes > original_nbytes) {
         /* growing: mark new extra memory clean */
-        memset(data + original_nbytes, CLEANBYTE, nbytes - original_nbytes);
+        memset(data + original_nbytes, PYMEM_CLEANBYTE,
+               nbytes - original_nbytes);
     }
 
     return data;
@@ -2399,7 +2386,7 @@ _PyMem_DebugCheckAddress(char api, const void *p)
      * the tail could lead to a segfault then.
      */
     for (i = SST-1; i >= 1; --i) {
-        if (*(q-i) != FORBIDDENBYTE) {
+        if (*(q-i) != PYMEM_FORBIDDENBYTE) {
             msg = "bad leading pad byte";
             goto error;
         }
@@ -2408,7 +2395,7 @@ _PyMem_DebugCheckAddress(char api, const void *p)
     nbytes = read_size_t(q - 2*SST);
     tail = q + nbytes;
     for (i = 0; i < SST; ++i) {
-        if (tail[i] != FORBIDDENBYTE) {
+        if (tail[i] != PYMEM_FORBIDDENBYTE) {
             msg = "bad trailing pad byte";
             goto error;
         }
@@ -2448,7 +2435,7 @@ _PyObject_DebugDumpAddress(const void *p)
     fprintf(stderr, "    The %d pad bytes at p-%d are ", SST-1, SST-1);
     ok = 1;
     for (i = 1; i <= SST-1; ++i) {
-        if (*(q-i) != FORBIDDENBYTE) {
+        if (*(q-i) != PYMEM_FORBIDDENBYTE) {
             ok = 0;
             break;
         }
@@ -2457,11 +2444,11 @@ _PyObject_DebugDumpAddress(const void *p)
         fputs("FORBIDDENBYTE, as expected.\n", stderr);
     else {
         fprintf(stderr, "not all FORBIDDENBYTE (0x%02x):\n",
-            FORBIDDENBYTE);
+            PYMEM_FORBIDDENBYTE);
         for (i = SST-1; i >= 1; --i) {
             const uint8_t byte = *(q-i);
             fprintf(stderr, "        at p-%d: 0x%02x", i, byte);
-            if (byte != FORBIDDENBYTE)
+            if (byte != PYMEM_FORBIDDENBYTE)
                 fputs(" *** OUCH", stderr);
             fputc('\n', stderr);
         }
@@ -2476,7 +2463,7 @@ _PyObject_DebugDumpAddress(const void *p)
     fprintf(stderr, "    The %d pad bytes at tail=%p are ", SST, (void *)tail);
     ok = 1;
     for (i = 0; i < SST; ++i) {
-        if (tail[i] != FORBIDDENBYTE) {
+        if (tail[i] != PYMEM_FORBIDDENBYTE) {
             ok = 0;
             break;
         }
@@ -2485,12 +2472,12 @@ _PyObject_DebugDumpAddress(const void *p)
         fputs("FORBIDDENBYTE, as expected.\n", stderr);
     else {
         fprintf(stderr, "not all FORBIDDENBYTE (0x%02x):\n",
-                FORBIDDENBYTE);
+                PYMEM_FORBIDDENBYTE);
         for (i = 0; i < SST; ++i) {
             const uint8_t byte = tail[i];
             fprintf(stderr, "        at tail+%d: 0x%02x",
                     i, byte);
-            if (byte != FORBIDDENBYTE)
+            if (byte != PYMEM_FORBIDDENBYTE)
                 fputs(" *** OUCH", stderr);
             fputc('\n', stderr);
         }


From webhook-mailer at python.org  Mon Oct  7 16:54:42 2019
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Mon, 07 Oct 2019 20:54:42 -0000
Subject: [Python-checkins] bpo-38294: Add list of no-longer-escaped chars to
 re.escape documentation. (GH-16442)
Message-ID: <mailman.20.1570481682.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/15ae75d660befe643ed42eb2707a557cea97256c
commit: 15ae75d660befe643ed42eb2707a557cea97256c
branch: master
author: Ricardo B?nffy <rbanffy at gmail.com>
committer: Serhiy Storchaka <storchaka at gmail.com>
date: 2019-10-07T23:54:35+03:00
summary:

bpo-38294: Add list of no-longer-escaped chars to re.escape documentation. (GH-16442)

Prior to 3.7, re.escape escaped many characters that don't have
special meaning in Python, but that use to require escaping in other
tools and languages. This commit aims to make it clear which characters
were, but are no longer escaped.

files:
A Misc/NEWS.d/next/Documentation/2019-09-27-23-37-41.bpo-38294.go_jFf.rst
M Doc/library/re.rst
M Doc/tools/susp-ignored.csv

diff --git a/Doc/library/re.rst b/Doc/library/re.rst
index df29c4d390908..7c950bfd5b1fd 100644
--- a/Doc/library/re.rst
+++ b/Doc/library/re.rst
@@ -931,8 +931,8 @@ form.
    This is useful if you want to match an arbitrary literal string that may
    have regular expression metacharacters in it.  For example::
 
-      >>> print(re.escape('python.exe'))
-      python\.exe
+      >>> print(re.escape('http://www.python.org'))
+      http://www\.python\.org
 
       >>> legal_chars = string.ascii_lowercase + string.digits + "!#$%&'*+-.^_`|~:"
       >>> print('[%s]+' % re.escape(legal_chars))
@@ -955,7 +955,9 @@ form.
 
    .. versionchanged:: 3.7
       Only characters that can have special meaning in a regular expression
-      are escaped.
+      are escaped. As a result, ``'!'``, ``'"'``, ``'%'``, ``"'"``, ``','``,
+      ``'/'``, ``':'``, ``';'``, ``'<'``, ``'='``, ``'>'``, ``'@'``, and
+      ``"`"`` are no longer escaped.
 
 
 .. function:: purge()
diff --git a/Doc/tools/susp-ignored.csv b/Doc/tools/susp-ignored.csv
index a7408620aeb48..740850cae847c 100644
--- a/Doc/tools/susp-ignored.csv
+++ b/Doc/tools/susp-ignored.csv
@@ -356,3 +356,4 @@ whatsnew/changelog,,::,default::BytesWarning
 whatsnew/changelog,,::,default::DeprecationWarning
 library/importlib.metadata,,:main,"EntryPoint(name='wheel', value='wheel.cli:main', group='console_scripts')"
 library/importlib.metadata,,`,loading the metadata for packages for the indicated ``context``.
+library/re,,`,"`"
diff --git a/Misc/NEWS.d/next/Documentation/2019-09-27-23-37-41.bpo-38294.go_jFf.rst b/Misc/NEWS.d/next/Documentation/2019-09-27-23-37-41.bpo-38294.go_jFf.rst
new file mode 100644
index 0000000000000..0bde8c868c9ec
--- /dev/null
+++ b/Misc/NEWS.d/next/Documentation/2019-09-27-23-37-41.bpo-38294.go_jFf.rst
@@ -0,0 +1 @@
+Add list of no-longer-escaped chars to re.escape documentation
\ No newline at end of file


From webhook-mailer at python.org  Mon Oct  7 17:07:24 2019
From: webhook-mailer at python.org (Steve Dower)
Date: Mon, 07 Oct 2019 21:07:24 -0000
Subject: [Python-checkins] bpo-38344: Fix syntax in activate.bat (GH-16533)
Message-ID: <mailman.21.1570482445.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/e310af9e2941c2fbb7370e003276cc37eb230f16
commit: e310af9e2941c2fbb7370e003276cc37eb230f16
branch: master
author: James Abel <j at abel.co>
committer: Steve Dower <steve.dower at python.org>
date: 2019-10-07T14:07:19-07:00
summary:

bpo-38344: Fix syntax in activate.bat (GH-16533)

files:
A Misc/NEWS.d/next/Windows/2019-10-05-05-50-58.bpo-38344.scr2LO.rst
M Lib/venv/scripts/nt/activate.bat

diff --git a/Lib/venv/scripts/nt/activate.bat b/Lib/venv/scripts/nt/activate.bat
index 8eb5c1ff664fb..af4c7e0abacb1 100644
--- a/Lib/venv/scripts/nt/activate.bat
+++ b/Lib/venv/scripts/nt/activate.bat
@@ -22,7 +22,7 @@ if defined PYTHONHOME set _OLD_VIRTUAL_PYTHONHOME=%PYTHONHOME%
 set PYTHONHOME=
 
 if defined _OLD_VIRTUAL_PATH set PATH=%_OLD_VIRTUAL_PATH%
-else set _OLD_VIRTUAL_PATH=%PATH%
+if not defined _OLD_VIRTUAL_PATH set _OLD_VIRTUAL_PATH=%PATH%
 
 set PATH=%VIRTUAL_ENV%\__VENV_BIN_NAME__;%PATH%
 
diff --git a/Misc/NEWS.d/next/Windows/2019-10-05-05-50-58.bpo-38344.scr2LO.rst b/Misc/NEWS.d/next/Windows/2019-10-05-05-50-58.bpo-38344.scr2LO.rst
new file mode 100644
index 0000000000000..b5854d1ecb0d3
--- /dev/null
+++ b/Misc/NEWS.d/next/Windows/2019-10-05-05-50-58.bpo-38344.scr2LO.rst
@@ -0,0 +1 @@
+Fix error message in activate.bat


From webhook-mailer at python.org  Mon Oct  7 17:07:50 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Mon, 07 Oct 2019 21:07:50 -0000
Subject: [Python-checkins] bpo-38294: Add list of no-longer-escaped chars to
 re.escape documentation. (GH-16442)
Message-ID: <mailman.22.1570482471.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/b731fc521cf78e53268e35777d836ca80e7ab305
commit: b731fc521cf78e53268e35777d836ca80e7ab305
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-07T14:07:46-07:00
summary:

bpo-38294: Add list of no-longer-escaped chars to re.escape documentation. (GH-16442)


Prior to 3.7, re.escape escaped many characters that don't have
special meaning in Python, but that use to require escaping in other
tools and languages. This commit aims to make it clear which characters
were, but are no longer escaped.
(cherry picked from commit 15ae75d660befe643ed42eb2707a557cea97256c)

Co-authored-by: Ricardo B?nffy <rbanffy at gmail.com>

files:
A Misc/NEWS.d/next/Documentation/2019-09-27-23-37-41.bpo-38294.go_jFf.rst
M Doc/library/re.rst
M Doc/tools/susp-ignored.csv

diff --git a/Doc/library/re.rst b/Doc/library/re.rst
index df29c4d390908..7c950bfd5b1fd 100644
--- a/Doc/library/re.rst
+++ b/Doc/library/re.rst
@@ -931,8 +931,8 @@ form.
    This is useful if you want to match an arbitrary literal string that may
    have regular expression metacharacters in it.  For example::
 
-      >>> print(re.escape('python.exe'))
-      python\.exe
+      >>> print(re.escape('http://www.python.org'))
+      http://www\.python\.org
 
       >>> legal_chars = string.ascii_lowercase + string.digits + "!#$%&'*+-.^_`|~:"
       >>> print('[%s]+' % re.escape(legal_chars))
@@ -955,7 +955,9 @@ form.
 
    .. versionchanged:: 3.7
       Only characters that can have special meaning in a regular expression
-      are escaped.
+      are escaped. As a result, ``'!'``, ``'"'``, ``'%'``, ``"'"``, ``','``,
+      ``'/'``, ``':'``, ``';'``, ``'<'``, ``'='``, ``'>'``, ``'@'``, and
+      ``"`"`` are no longer escaped.
 
 
 .. function:: purge()
diff --git a/Doc/tools/susp-ignored.csv b/Doc/tools/susp-ignored.csv
index 1f0cf3fd6ed44..a0fc874829f5f 100644
--- a/Doc/tools/susp-ignored.csv
+++ b/Doc/tools/susp-ignored.csv
@@ -354,3 +354,4 @@ whatsnew/changelog,,::,default::BytesWarning
 whatsnew/changelog,,::,default::DeprecationWarning
 library/importlib.metadata,,:main,"EntryPoint(name='wheel', value='wheel.cli:main', group='console_scripts')"
 library/importlib.metadata,,`,loading the metadata for packages for the indicated ``context``.
+library/re,,`,"`"
diff --git a/Misc/NEWS.d/next/Documentation/2019-09-27-23-37-41.bpo-38294.go_jFf.rst b/Misc/NEWS.d/next/Documentation/2019-09-27-23-37-41.bpo-38294.go_jFf.rst
new file mode 100644
index 0000000000000..0bde8c868c9ec
--- /dev/null
+++ b/Misc/NEWS.d/next/Documentation/2019-09-27-23-37-41.bpo-38294.go_jFf.rst
@@ -0,0 +1 @@
+Add list of no-longer-escaped chars to re.escape documentation
\ No newline at end of file


From webhook-mailer at python.org  Mon Oct  7 17:25:10 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Mon, 07 Oct 2019 21:25:10 -0000
Subject: [Python-checkins] bpo-38344: Fix syntax in activate.bat (GH-16533)
Message-ID: <mailman.23.1570483511.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/bfb5f9a66cc348c9cf25c857ad9910479e587517
commit: bfb5f9a66cc348c9cf25c857ad9910479e587517
branch: 3.7
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-07T14:25:06-07:00
summary:

bpo-38344: Fix syntax in activate.bat (GH-16533)

(cherry picked from commit e310af9e2941c2fbb7370e003276cc37eb230f16)

Co-authored-by: James Abel <j at abel.co>

files:
A Misc/NEWS.d/next/Windows/2019-10-05-05-50-58.bpo-38344.scr2LO.rst
M Lib/venv/scripts/nt/activate.bat

diff --git a/Lib/venv/scripts/nt/activate.bat b/Lib/venv/scripts/nt/activate.bat
index 8eb5c1ff664fb..af4c7e0abacb1 100644
--- a/Lib/venv/scripts/nt/activate.bat
+++ b/Lib/venv/scripts/nt/activate.bat
@@ -22,7 +22,7 @@ if defined PYTHONHOME set _OLD_VIRTUAL_PYTHONHOME=%PYTHONHOME%
 set PYTHONHOME=
 
 if defined _OLD_VIRTUAL_PATH set PATH=%_OLD_VIRTUAL_PATH%
-else set _OLD_VIRTUAL_PATH=%PATH%
+if not defined _OLD_VIRTUAL_PATH set _OLD_VIRTUAL_PATH=%PATH%
 
 set PATH=%VIRTUAL_ENV%\__VENV_BIN_NAME__;%PATH%
 
diff --git a/Misc/NEWS.d/next/Windows/2019-10-05-05-50-58.bpo-38344.scr2LO.rst b/Misc/NEWS.d/next/Windows/2019-10-05-05-50-58.bpo-38344.scr2LO.rst
new file mode 100644
index 0000000000000..b5854d1ecb0d3
--- /dev/null
+++ b/Misc/NEWS.d/next/Windows/2019-10-05-05-50-58.bpo-38344.scr2LO.rst
@@ -0,0 +1 @@
+Fix error message in activate.bat


From webhook-mailer at python.org  Mon Oct  7 17:27:01 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Mon, 07 Oct 2019 21:27:01 -0000
Subject: [Python-checkins] bpo-38344: Fix syntax in activate.bat (GH-16533)
Message-ID: <mailman.24.1570483622.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/4880e5a1b66d9175fcc402cb0288bc1898356831
commit: 4880e5a1b66d9175fcc402cb0288bc1898356831
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-07T14:26:57-07:00
summary:

bpo-38344: Fix syntax in activate.bat (GH-16533)

(cherry picked from commit e310af9e2941c2fbb7370e003276cc37eb230f16)

Co-authored-by: James Abel <j at abel.co>

files:
A Misc/NEWS.d/next/Windows/2019-10-05-05-50-58.bpo-38344.scr2LO.rst
M Lib/venv/scripts/nt/activate.bat

diff --git a/Lib/venv/scripts/nt/activate.bat b/Lib/venv/scripts/nt/activate.bat
index 8eb5c1ff664fb..af4c7e0abacb1 100644
--- a/Lib/venv/scripts/nt/activate.bat
+++ b/Lib/venv/scripts/nt/activate.bat
@@ -22,7 +22,7 @@ if defined PYTHONHOME set _OLD_VIRTUAL_PYTHONHOME=%PYTHONHOME%
 set PYTHONHOME=
 
 if defined _OLD_VIRTUAL_PATH set PATH=%_OLD_VIRTUAL_PATH%
-else set _OLD_VIRTUAL_PATH=%PATH%
+if not defined _OLD_VIRTUAL_PATH set _OLD_VIRTUAL_PATH=%PATH%
 
 set PATH=%VIRTUAL_ENV%\__VENV_BIN_NAME__;%PATH%
 
diff --git a/Misc/NEWS.d/next/Windows/2019-10-05-05-50-58.bpo-38344.scr2LO.rst b/Misc/NEWS.d/next/Windows/2019-10-05-05-50-58.bpo-38344.scr2LO.rst
new file mode 100644
index 0000000000000..b5854d1ecb0d3
--- /dev/null
+++ b/Misc/NEWS.d/next/Windows/2019-10-05-05-50-58.bpo-38344.scr2LO.rst
@@ -0,0 +1 @@
+Fix error message in activate.bat


From webhook-mailer at python.org  Mon Oct  7 17:44:09 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Mon, 07 Oct 2019 21:44:09 -0000
Subject: [Python-checkins] bpo-36389: Add newline to
 _PyObject_AssertFailed() (GH-16629)
Message-ID: <mailman.25.1570484650.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/7775349895088a7ae68cecf0c74cf817f15e2c74
commit: 7775349895088a7ae68cecf0c74cf817f15e2c74
branch: master
author: Victor Stinner <vstinner at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-07T23:44:05+02:00
summary:

bpo-36389: Add newline to _PyObject_AssertFailed() (GH-16629)

Add a newline between the verbose object dump and the Py_FatalError()
logs for readability.

files:
M Objects/object.c

diff --git a/Objects/object.c b/Objects/object.c
index e94480f7e998f..ae76e33e1f4f1 100644
--- a/Objects/object.c
+++ b/Objects/object.c
@@ -2187,6 +2187,9 @@ _PyObject_AssertFailed(PyObject *obj, const char *expr, const char *msg,
         /* This might succeed or fail, but we're about to abort, so at least
            try to provide any extra info we can: */
         _PyObject_Dump(obj);
+
+        fprintf(stderr, "\n");
+        fflush(stderr);
     }
 
     Py_FatalError("_PyObject_AssertFailed");


From webhook-mailer at python.org  Mon Oct  7 18:09:35 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Mon, 07 Oct 2019 22:09:35 -0000
Subject: [Python-checkins] bpo-38392: PyObject_GC_Track() validates object
 in debug mode (GH-16615)
Message-ID: <mailman.26.1570486176.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/1b1845569539db5c1a6948a5d32daea381f1e35f
commit: 1b1845569539db5c1a6948a5d32daea381f1e35f
branch: master
author: Victor Stinner <vstinner at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-08T00:09:31+02:00
summary:

bpo-38392: PyObject_GC_Track() validates object in debug mode (GH-16615)

In debug mode, PyObject_GC_Track() now calls tp_traverse() of the
object type to ensure that the object is valid: test that objects
visited by tp_traverse() are valid.

Fix pyexpat.c: only track the parser in the GC once the parser is
fully initialized.

files:
A Misc/NEWS.d/next/Core and Builtins/2019-10-07-22-51-39.bpo-38392.KaXXps.rst
M Modules/_elementtree.c
M Modules/gcmodule.c
M Modules/pyexpat.c

diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-10-07-22-51-39.bpo-38392.KaXXps.rst b/Misc/NEWS.d/next/Core and Builtins/2019-10-07-22-51-39.bpo-38392.KaXXps.rst
new file mode 100644
index 0000000000000..07358ece753f5
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2019-10-07-22-51-39.bpo-38392.KaXXps.rst	
@@ -0,0 +1,3 @@
+In debug mode, :c:func:`PyObject_GC_Track` now calls ``tp_traverse()`` of
+the object type to ensure that the object is valid: test that objects
+visited by ``tp_traverse()`` are valid.
diff --git a/Modules/_elementtree.c b/Modules/_elementtree.c
index 9050144b70f32..c096eab43ea6a 100644
--- a/Modules/_elementtree.c
+++ b/Modules/_elementtree.c
@@ -2317,8 +2317,6 @@ create_elementiter(ElementObject *self, PyObject *tag, int gettext)
     Py_INCREF(self);
     it->root_element = self;
 
-    PyObject_GC_Track(it);
-
     it->parent_stack = PyMem_New(ParentLocator, INIT_PARENT_STACK_SIZE);
     if (it->parent_stack == NULL) {
         Py_DECREF(it);
@@ -2328,6 +2326,8 @@ create_elementiter(ElementObject *self, PyObject *tag, int gettext)
     it->parent_stack_used = 0;
     it->parent_stack_size = INIT_PARENT_STACK_SIZE;
 
+    PyObject_GC_Track(it);
+
     return (PyObject *)it;
 }
 
diff --git a/Modules/gcmodule.c b/Modules/gcmodule.c
index 107ce39967e20..2cbf573cf46ec 100644
--- a/Modules/gcmodule.c
+++ b/Modules/gcmodule.c
@@ -1922,6 +1922,18 @@ _PyGC_Dump(PyGC_Head *g)
     _PyObject_Dump(FROM_GC(g));
 }
 
+static int
+visit_validate(PyObject *op, void *parent_raw)
+{
+    PyObject *parent = _PyObject_CAST(parent_raw);
+    if (_PyObject_IsFreed(op)) {
+        _PyObject_ASSERT_FAILED_MSG(parent,
+                                    "PyObject_GC_Track() object is not valid");
+    }
+    return 0;
+}
+
+
 /* extension modules might be compiled with GC support so these
    functions must always be available */
 
@@ -1935,6 +1947,13 @@ PyObject_GC_Track(void *op_raw)
                                     "by the garbage collector");
     }
     _PyObject_GC_TRACK(op);
+
+#ifdef Py_DEBUG
+    /* Check that the object is valid: validate objects traversed
+       by tp_traverse() */
+    traverseproc traverse = Py_TYPE(op)->tp_traverse;
+    (void)traverse(op, visit_validate, op);
+#endif
 }
 
 void
diff --git a/Modules/pyexpat.c b/Modules/pyexpat.c
index d9123354bbed2..90167342fee50 100644
--- a/Modules/pyexpat.c
+++ b/Modules/pyexpat.c
@@ -938,7 +938,6 @@ pyexpat_xmlparser_ExternalEntityParserCreate_impl(xmlparseobject *self,
     new_parser->handlers = 0;
     new_parser->intern = self->intern;
     Py_XINCREF(new_parser->intern);
-    PyObject_GC_Track(new_parser);
 
     if (self->buffer != NULL) {
         new_parser->buffer = PyMem_Malloc(new_parser->buffer_size);
@@ -975,6 +974,8 @@ pyexpat_xmlparser_ExternalEntityParserCreate_impl(xmlparseobject *self,
                                    handler_info[i].handler);
         }
     }
+
+    PyObject_GC_Track(new_parser);
     return (PyObject *)new_parser;
 }
 
@@ -1122,7 +1123,6 @@ newxmlparseobject(const char *encoding, const char *namespace_separator, PyObjec
     self->handlers = NULL;
     self->intern = intern;
     Py_XINCREF(self->intern);
-    PyObject_GC_Track(self);
 
     /* namespace_separator is either NULL or contains one char + \0 */
     self->itself = XML_ParserCreate_MM(encoding, &ExpatMemoryHandler,
@@ -1152,6 +1152,7 @@ newxmlparseobject(const char *encoding, const char *namespace_separator, PyObjec
     }
     clear_handlers(self, 1);
 
+    PyObject_GC_Track(self);
     return (PyObject*)self;
 }
 


From webhook-mailer at python.org  Mon Oct  7 19:43:18 2019
From: webhook-mailer at python.org (Pablo Galindo)
Date: Mon, 07 Oct 2019 23:43:18 -0000
Subject: [Python-checkins] bpo-38400 Don't check for NULL linked list
 pointers in _PyObject_IsFreed (GH-16630)
Message-ID: <mailman.27.1570491799.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/36e33c360ed7716a2b5ab2b53210da81f8ce1295
commit: 36e33c360ed7716a2b5ab2b53210da81f8ce1295
branch: master
author: Pablo Galindo <Pablogsal at gmail.com>
committer: GitHub <noreply at github.com>
date: 2019-10-08T00:43:14+01:00
summary:

bpo-38400 Don't check for NULL linked list pointers in _PyObject_IsFreed (GH-16630)

Some objects like Py_None are not initialized with conventional means
that prepare the circular linked list pointers, leaving them unlinked
from the rest of the objects. For those objects, NULL pointers does
not mean that they are freed, so we need to skip the check in those
cases.

files:
M Objects/object.c

diff --git a/Objects/object.c b/Objects/object.c
index ae76e33e1f4f1..2c8e823f05ee9 100644
--- a/Objects/object.c
+++ b/Objects/object.c
@@ -454,9 +454,12 @@ _PyObject_IsFreed(PyObject *op)
     /* ignore op->ob_ref: its value can have be modified
        by Py_INCREF() and Py_DECREF(). */
 #ifdef Py_TRACE_REFS
-    if (_PyMem_IsPtrFreed(op->_ob_next) || _PyMem_IsPtrFreed(op->_ob_prev)) {
+    if (op->_ob_next != NULL && _PyMem_IsPtrFreed(op->_ob_next)) {
         return 1;
     }
+    if (op->_ob_prev != NULL && _PyMem_IsPtrFreed(op->_ob_prev)) {
+         return 1;
+     }
 #endif
     return 0;
 }


From webhook-mailer at python.org  Mon Oct  7 20:37:45 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Tue, 08 Oct 2019 00:37:45 -0000
Subject: [Python-checkins] bpo-38070: Enhance visit_decref() debug trace
 (GH-16631)
Message-ID: <mailman.28.1570495066.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/4d5f94b8cd20f804c7868c5395a15aa6032f874c
commit: 4d5f94b8cd20f804c7868c5395a15aa6032f874c
branch: master
author: Victor Stinner <vstinner at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-08T02:37:38+02:00
summary:

bpo-38070: Enhance visit_decref() debug trace (GH-16631)

subtract_refs() now pass the parent object to visit_decref() which
pass it to _PyObject_ASSERT(). So if the "is freed" assertion fails,
the parent is used in debug trace, rather than the freed object. The
parent object is more likely to contain useful information. Freed
objects cannot be inspected are are displayed as "<object at xxx is
freed>" with no other detail.

files:
M Modules/gcmodule.c

diff --git a/Modules/gcmodule.c b/Modules/gcmodule.c
index 2cbf573cf46ec..2b47abae1ae4d 100644
--- a/Modules/gcmodule.c
+++ b/Modules/gcmodule.c
@@ -373,9 +373,9 @@ update_refs(PyGC_Head *containers)
 
 /* A traversal callback for subtract_refs. */
 static int
-visit_decref(PyObject *op, void *data)
+visit_decref(PyObject *op, void *parent)
 {
-    _PyObject_ASSERT(op, !_PyObject_IsFreed(op));
+    _PyObject_ASSERT(_PyObject_CAST(parent), !_PyObject_IsFreed(op));
 
     if (PyObject_IS_GC(op)) {
         PyGC_Head *gc = AS_GC(op);
@@ -401,10 +401,11 @@ subtract_refs(PyGC_Head *containers)
     traverseproc traverse;
     PyGC_Head *gc = GC_NEXT(containers);
     for (; gc != containers; gc = GC_NEXT(gc)) {
-        traverse = Py_TYPE(FROM_GC(gc))->tp_traverse;
+        PyObject *op = FROM_GC(gc);
+        traverse = Py_TYPE(op)->tp_traverse;
         (void) traverse(FROM_GC(gc),
                        (visitproc)visit_decref,
-                       NULL);
+                       op);
     }
 }
 


From webhook-mailer at python.org  Mon Oct  7 21:54:09 2019
From: webhook-mailer at python.org (Benjamin Peterson)
Date: Tue, 08 Oct 2019 01:54:09 -0000
Subject: [Python-checkins] [2.7] bpo-37664: Update ensurepip bundled wheels,
 again (GH-16633)
Message-ID: <mailman.29.1570499650.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/e7e58fe03175adc660c192e724b91ae7ccba9cb6
commit: e7e58fe03175adc660c192e724b91ae7ccba9cb6
branch: 2.7
author: Benjamin Peterson <benjamin at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-07T18:54:05-07:00
summary:

[2.7] bpo-37664: Update ensurepip bundled wheels, again (GH-16633)

(cherry picked from commit 10c452b894d95fed06056fe11e8fe8e1a2a60040)

Co-authored-by: Pradyun Gedam <pradyunsg at gmail.com>

files:
A Lib/ensurepip/_bundled/pip-19.2.3-py2.py3-none-any.whl
A Lib/ensurepip/_bundled/setuptools-41.2.0-py2.py3-none-any.whl
A Misc/NEWS.d/next/Library/2019-07-24-18-27-44.bpo-37664.o-GYZC.rst
D Lib/ensurepip/_bundled/pip-19.0.3-py2.py3-none-any.whl
D Lib/ensurepip/_bundled/setuptools-40.8.0-py2.py3-none-any.whl
M Lib/ensurepip/__init__.py

diff --git a/Lib/ensurepip/__init__.py b/Lib/ensurepip/__init__.py
index cbadfaa41a263..0955141184782 100644
--- a/Lib/ensurepip/__init__.py
+++ b/Lib/ensurepip/__init__.py
@@ -12,9 +12,9 @@
 __all__ = ["version", "bootstrap"]
 
 
-_SETUPTOOLS_VERSION = "40.8.0"
+_SETUPTOOLS_VERSION = "41.2.0"
 
-_PIP_VERSION = "19.0.3"
+_PIP_VERSION = "19.2.3"
 
 _PROJECTS = [
     ("setuptools", _SETUPTOOLS_VERSION),
diff --git a/Lib/ensurepip/_bundled/pip-19.0.3-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/pip-19.2.3-py2.py3-none-any.whl
similarity index 51%
rename from Lib/ensurepip/_bundled/pip-19.0.3-py2.py3-none-any.whl
rename to Lib/ensurepip/_bundled/pip-19.2.3-py2.py3-none-any.whl
index 24f247caa0ab4..8118df8ac1940 100644
Binary files a/Lib/ensurepip/_bundled/pip-19.0.3-py2.py3-none-any.whl and b/Lib/ensurepip/_bundled/pip-19.2.3-py2.py3-none-any.whl differ
diff --git a/Lib/ensurepip/_bundled/setuptools-40.8.0-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/setuptools-41.2.0-py2.py3-none-any.whl
similarity index 84%
rename from Lib/ensurepip/_bundled/setuptools-40.8.0-py2.py3-none-any.whl
rename to Lib/ensurepip/_bundled/setuptools-41.2.0-py2.py3-none-any.whl
index fdc66e9330e07..82df6f63f4ee9 100644
Binary files a/Lib/ensurepip/_bundled/setuptools-40.8.0-py2.py3-none-any.whl and b/Lib/ensurepip/_bundled/setuptools-41.2.0-py2.py3-none-any.whl differ
diff --git a/Misc/NEWS.d/next/Library/2019-07-24-18-27-44.bpo-37664.o-GYZC.rst b/Misc/NEWS.d/next/Library/2019-07-24-18-27-44.bpo-37664.o-GYZC.rst
new file mode 100644
index 0000000000000..f12590540eb76
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-07-24-18-27-44.bpo-37664.o-GYZC.rst
@@ -0,0 +1 @@
+Update wheels bundled with ensurepip (pip 19.2.3 and setuptools 41.2.0)


From webhook-mailer at python.org  Mon Oct  7 22:00:05 2019
From: webhook-mailer at python.org (Benjamin Peterson)
Date: Tue, 08 Oct 2019 02:00:05 -0000
Subject: [Python-checkins] [2.7] bpo-38216,
 bpo-36274: Allow subclasses to separately override validation and
 encoding behavior (GH-16476)
Message-ID: <mailman.30.1570500006.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/f5b1abbb3b0083381925dcd5898ae6d019224826
commit: f5b1abbb3b0083381925dcd5898ae6d019224826
branch: 2.7
author: Jason R. Coombs <jaraco at jaraco.com>
committer: Benjamin Peterson <benjamin at python.org>
date: 2019-10-07T19:00:01-07:00
summary:

[2.7] bpo-38216, bpo-36274: Allow subclasses to separately override validation and encoding behavior (GH-16476)

Backporting this change, I observe a couple of things:

1. The _encode_request call is no longer meaningful because the request construction will implicitly encode the request using the default encoding when the format string is used (request = '%s %s %s'...). In order to keep the code as consistent as possible, I decided to include the call as a pass-through. I'd be just as happy to remove it entirely, but I'll leave that up to the reviewer to decide. It's okay that this functionality is disabled on Python 2 because this functionality was mainly around bpo-36274, which was mainly a concern with the transition to Python 3.
2. Because _encode_request is no longer meaningful, neither is the test for it, so I've removed that test. Therefore, the meaningful part of this test is that for bpo-38216, adding a (underscore-protected) hook to customize/disable validation.

(cherry picked from commit 7774d7831e8809795c64ce27f7df52674581d298)

Co-authored-by: Jason R. Coombs <jaraco at jaraco.com>

files:
A Misc/NEWS.d/next/Library/2019-09-27-15-24-45.bpo-38216.-7yvZR.rst
M Lib/httplib.py
M Lib/test/test_httplib.py

diff --git a/Lib/httplib.py b/Lib/httplib.py
index 1b41c346e090a..79532b91149b1 100644
--- a/Lib/httplib.py
+++ b/Lib/httplib.py
@@ -933,19 +933,15 @@ def putrequest(self, method, url, skip_host=0, skip_accept_encoding=0):
         else:
             raise CannotSendRequest()
 
-        # Save the method we use, we need it later in the response phase
+        # Save the method for use later in the response phase
         self._method = method
-        if not url:
-            url = '/'
-        # Prevent CVE-2019-9740.
-        match = _contains_disallowed_url_pchar_re.search(url)
-        if match:
-            raise InvalidURL("URL can't contain control characters. %r "
-                             "(found at least %r)"
-                             % (url, match.group()))
-        hdr = '%s %s %s' % (method, url, self._http_vsn_str)
 
-        self._output(hdr)
+        url = url or '/'
+        self._validate_path(url)
+
+        request = '%s %s %s' % (method, url, self._http_vsn_str)
+
+        self._output(self._encode_request(request))
 
         if self._http_vsn == 11:
             # Issue some standard headers for better HTTP/1.1 compliance
@@ -1018,6 +1014,21 @@ def putrequest(self, method, url, skip_host=0, skip_accept_encoding=0):
             # For HTTP/1.0, the server will assume "not chunked"
             pass
 
+    def _encode_request(self, request):
+        # On Python 2, request is already encoded (default)
+        return request
+
+    def _validate_path(self, url):
+        """Validate a url for putrequest."""
+        # Prevent CVE-2019-9740.
+        match = _contains_disallowed_url_pchar_re.search(url)
+        if match:
+            msg = (
+                "URL can't contain control characters. {url!r} "
+                "(found at least {matched!r})"
+            ).format(matched=match.group(), url=url)
+            raise InvalidURL(msg)
+
     def putheader(self, header, *values):
         """Send a request header line to the server.
 
diff --git a/Lib/test/test_httplib.py b/Lib/test/test_httplib.py
index 44ffac7036886..5462fdd503c83 100644
--- a/Lib/test/test_httplib.py
+++ b/Lib/test/test_httplib.py
@@ -702,6 +702,20 @@ def test_proxy_tunnel_without_status_line(self):
         with self.assertRaisesRegexp(socket.error, "Invalid response"):
             conn._tunnel()
 
+    def test_putrequest_override_validation(self):
+        """
+        It should be possible to override the default validation
+        behavior in putrequest (bpo-38216).
+        """
+        class UnsafeHTTPConnection(httplib.HTTPConnection):
+            def _validate_path(self, url):
+                pass
+
+        conn = UnsafeHTTPConnection('example.com')
+        conn.sock = FakeSocket('')
+        conn.putrequest('GET', '/\x00')
+
+
 class OfflineTest(TestCase):
     def test_responses(self):
         self.assertEqual(httplib.responses[httplib.NOT_FOUND], "Not Found")
diff --git a/Misc/NEWS.d/next/Library/2019-09-27-15-24-45.bpo-38216.-7yvZR.rst b/Misc/NEWS.d/next/Library/2019-09-27-15-24-45.bpo-38216.-7yvZR.rst
new file mode 100644
index 0000000000000..ac8e2b042d929
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-09-27-15-24-45.bpo-38216.-7yvZR.rst
@@ -0,0 +1,4 @@
+Allow the rare code that wants to send invalid http requests from the
+`http.client` library a way to do so.  The fixes for bpo-30458 led to
+breakage for some projects that were relying on this ability to test their
+own behavior in the face of bad requests.


From webhook-mailer at python.org  Mon Oct  7 22:03:12 2019
From: webhook-mailer at python.org (Benjamin Peterson)
Date: Tue, 08 Oct 2019 02:03:12 -0000
Subject: [Python-checkins] Roll up news for 2.7.17rc1.
Message-ID: <mailman.31.1570500193.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/89dea4664215ffba384dea79fac166822e2dbdda
commit: 89dea4664215ffba384dea79fac166822e2dbdda
branch: 2.7
author: Benjamin Peterson <benjamin at python.org>
committer: Benjamin Peterson <benjamin at python.org>
date: 2019-10-07T19:01:18-07:00
summary:

Roll up news for 2.7.17rc1.

files:
A Misc/NEWS.d/2.7.17rc1.rst
D Misc/NEWS.d/next/Build/2018-11-16-15-19-09.bpo-35264.h5GxH3.rst
D Misc/NEWS.d/next/Build/2019-04-11-18-50-58.bpo-36605.gk5czf.rst
D Misc/NEWS.d/next/Build/2019-05-12-10-52-37.bpo-14353.LK1qWM.rst
D Misc/NEWS.d/next/Build/2019-09-28-02-37-11.bpo-38301.123456.rst
D Misc/NEWS.d/next/C API/2019-06-06-08-47-04.bpo-37170.hO_fpM.rst
D Misc/NEWS.d/next/Core and Builtins/2018-03-07-09-10-42.bpo-33006.Bzx3LA.rst
D Misc/NEWS.d/next/Core and Builtins/2019-02-28-13-52-18.bpo-36149.GJdnh4.rst
D Misc/NEWS.d/next/Core and Builtins/2019-03-11-15-37-33.bpo-36262.v3N6Fz.rst
D Misc/NEWS.d/next/Core and Builtins/2019-03-14-17-30-46.bpo-18368.WXaHAo.rst
D Misc/NEWS.d/next/Core and Builtins/2019-03-25-23-37-26.bpo-36430.sd9xxQ.rst
D Misc/NEWS.d/next/Core and Builtins/2019-03-27-22-35-16.bpo-36459.UAvkKp.rst
D Misc/NEWS.d/next/Core and Builtins/2019-04-02-04-10-32.bpo-36504.k_V8Bm.rst
D Misc/NEWS.d/next/Core and Builtins/2019-04-16-11-52-21.bpo-27987.n2_DcQ.rst
D Misc/NEWS.d/next/Core and Builtins/2019-05-27-18-00-19.bpo-26423.RgUOE8.rst
D Misc/NEWS.d/next/Core and Builtins/2019-06-18-15-08-32.bpo-37329.7XVDRv.rst
D Misc/NEWS.d/next/Core and Builtins/2019-09-12-16-36-16.bpo-38106.4pApn7.rst
D Misc/NEWS.d/next/Documentation/2018-06-15-15-57-37.bpo-33832.xBFhKw.rst
D Misc/NEWS.d/next/Documentation/2018-12-22-22-52-05.bpo-35564.TuEU_D.rst
D Misc/NEWS.d/next/Documentation/2018-12-30-09-56-13.bpo-35605.gAWt32.rst
D Misc/NEWS.d/next/Documentation/2019-02-18-10-01-07.bpo-35126.LWwl8X.rst
D Misc/NEWS.d/next/Documentation/2019-07-06-02-19-02.bpo-37149.NumHn3.rst
D Misc/NEWS.d/next/Documentation/2019-07-06-17-19-26.bpo-37487.QagfZ5.rst
D Misc/NEWS.d/next/IDLE/2019-05-05-16-27-53.bpo-13102.AGNWYJ.rst
D Misc/NEWS.d/next/IDLE/2019-06-07-00-17-41.bpo-37177.voU6pQ.rst
D Misc/NEWS.d/next/Library/2017-08-30-20-27-00.bpo-31292.dKIaZb.rst
D Misc/NEWS.d/next/Library/2019-02-25-13-21-43.bpo-36106.VuhEiQ.rst
D Misc/NEWS.d/next/Library/2019-03-04-10-42-46.bpo-36179.jEyuI-.rst
D Misc/NEWS.d/next/Library/2019-03-04-16-13-01.bpo-13096.SGPt_n.rst
D Misc/NEWS.d/next/Library/2019-03-04-16-39-16.bpo-36186.Hqw1A_.rst
D Misc/NEWS.d/next/Library/2019-03-06-13-21-33.bpo-35807.W7mmu3.rst
D Misc/NEWS.d/next/Library/2019-03-08-13-32-21.bpo-36235._M72wU.rst
D Misc/NEWS.d/next/Library/2019-03-14-14-40-22.bpo-36212.IEgRI8.rst
D Misc/NEWS.d/next/Library/2019-03-14-15-42-48.bpo-36289.wYKS47.rst
D Misc/NEWS.d/next/Library/2019-03-14-15-54-46.bpo-36291.UalHXP.rst
D Misc/NEWS.d/next/Library/2019-03-18-10-08-30.bpo-36337.QhJnXy.rst
D Misc/NEWS.d/next/Library/2019-04-18-16-10-29.bpo-28552.MW1TLt.rst
D Misc/NEWS.d/next/Library/2019-05-23-15-57-36.bpo-36713.sjPhnf.rst
D Misc/NEWS.d/next/Library/2019-05-30-16-16-47.bpo-12639.TQFOR4.rst
D Misc/NEWS.d/next/Library/2019-06-10-12-02-45.bpo-36742.UEdHXJ.rst
D Misc/NEWS.d/next/Library/2019-06-27-20-33-50.bpo-37437.du39_A.rst
D Misc/NEWS.d/next/Library/2019-07-24-18-27-44.bpo-37664.o-GYZC.rst
D Misc/NEWS.d/next/Library/2019-08-23-14-47-09.bpo-34521.Y2BYu5.rst
D Misc/NEWS.d/next/Library/2019-08-28-14-04-18.bpo-37965.7xGE-C.rst
D Misc/NEWS.d/next/Library/2019-08-31-01-52-59.bpo-34410.7KbWZQ.rst
D Misc/NEWS.d/next/Library/2019-09-14-10-34-00.bpo-33936.8wCI_n.rst
D Misc/NEWS.d/next/Library/2019-09-15-10-30-33.bpo-38175.61XlUv.rst
D Misc/NEWS.d/next/Library/2019-09-27-15-24-45.bpo-38216.-7yvZR.rst
D Misc/NEWS.d/next/Security/2019-02-13-17-21-10.bpo-35907.ckk2zg.rst
D Misc/NEWS.d/next/Security/2019-03-06-09-38-40.bpo-36216.6q1m4a.rst
D Misc/NEWS.d/next/Security/2019-04-10-08-53-30.bpo-30458.51E-DA.rst
D Misc/NEWS.d/next/Security/2019-04-29-15-34-59.bpo-36742.QCUY0i.rst
D Misc/NEWS.d/next/Security/2019-05-04-13-33-37.bpo-34155.MJll68.rst
D Misc/NEWS.d/next/Security/2019-05-20-00-35-12.bpo-35121.RRi-HU.rst
D Misc/NEWS.d/next/Security/2019-05-20-00-49-29.bpo-35647.oWmiGU.rst
D Misc/NEWS.d/next/Security/2019-06-17-08-43-19.bpo-34631.pJ8CGR.rst
D Misc/NEWS.d/next/Security/2019-09-23-21-02-46.bpo-38174.MeWuJd.rst
D Misc/NEWS.d/next/Security/2019-09-25-13-21-09.bpo-38243.1pfz24.rst
D Misc/NEWS.d/next/Tests/2019-01-11-14-01-19.bpo-34836.7fat9-.rst
D Misc/NEWS.d/next/Tests/2019-02-24-02-44-52.bpo-26386.YZylPP.rst
D Misc/NEWS.d/next/Tests/2019-02-24-03-19-43.bpo-27313.Sj9veH.rst
D Misc/NEWS.d/next/Tests/2019-03-05-13-27-36.bpo-36019.ebUjCm.rst
D Misc/NEWS.d/next/Tests/2019-03-08-12-53-37.bpo-36234.NRVK6W.rst
D Misc/NEWS.d/next/Tests/2019-04-09-14-08-02.bpo-36560._ejeOr.rst
D Misc/NEWS.d/next/Tests/2019-05-06-18-29-54.bpo-35925.gwQPuC.rst
D Misc/NEWS.d/next/Tests/2019-05-08-15-55-46.bpo-36816.WBKRGZ.rst
D Misc/NEWS.d/next/Tests/2019-06-21-15-47-33.bpo-37362.D3xppx.rst
D Misc/NEWS.d/next/Tests/2019-06-24-10-47-07.bpo-37359.CkdtyO.rst
D Misc/NEWS.d/next/Tests/2019-06-26-15-28-45.bpo-37411.5lGNhM.rst
D Misc/NEWS.d/next/Tools-Demos/2019-04-30-14-30-29.bpo-14546.r38Y-6.rst
D Misc/NEWS.d/next/Tools-Demos/2019-07-24-16-20-54.bpo-37675.951Cvf.rst
D Misc/NEWS.d/next/Windows/2017-11-24-12-53-54.bpo-1104.1CWSZp.rst
D Misc/NEWS.d/next/Windows/2019-06-18-09-53-15.bpo-35360.tdqSmo.rst
D Misc/NEWS.d/next/Windows/2019-06-28-18-10-29.bpo-37445.LsdYO6.rst
D Misc/NEWS.d/next/Windows/2019-09-16-12-47-12.bpo-38117.JaiB0X.rst
D Misc/NEWS.d/next/macOS/2019-06-03-05-49-49.bpo-36231.RfmW_p.rst
D Misc/NEWS.d/next/macOS/2019-06-18-05-39-01.bpo-34631.StdZhE.rst
D Misc/NEWS.d/next/macOS/2019-06-18-10-02-11.bpo-35360.-CWbfy.rst
D Misc/NEWS.d/next/macOS/2019-06-20-01-16-16.bpo-19960.LpLUPF.rst
D Misc/NEWS.d/next/macOS/2019-09-15-21-45-15.bpo-38117._U9F2r.rst

diff --git a/Misc/NEWS.d/2.7.17rc1.rst b/Misc/NEWS.d/2.7.17rc1.rst
new file mode 100644
index 0000000000000..06ce6eb70ffa5
--- /dev/null
+++ b/Misc/NEWS.d/2.7.17rc1.rst
@@ -0,0 +1,785 @@
+.. bpo: 38243
+.. date: 2019-09-25-13-21-09
+.. nonce: 1pfz24
+.. release date: 2019-10-07
+.. section: Security
+
+Escape the server title of :class:`DocXMLRPCServer.DocXMLRPCServer` when
+rendering the document page as HTML. (Contributed by Dong-hee Na in
+:issue:`38243`.)
+
+..
+
+.. bpo: 38174
+.. date: 2019-09-23-21-02-46
+.. nonce: MeWuJd
+.. section: Security
+
+Update vendorized expat library version to 2.2.8, which resolves
+CVE-2019-15903.
+
+..
+
+.. bpo: 34631
+.. date: 2019-06-17-08-43-19
+.. nonce: pJ8CGR
+.. section: Security
+
+Updated OpenSSL to 1.0.2s in Windows installer
+
+..
+
+.. bpo: 35647
+.. date: 2019-05-20-00-49-29
+.. nonce: oWmiGU
+.. section: Security
+
+Don't set cookie for a request when the request path is a prefix match of
+the cookie's path attribute but doesn't end with "/". Patch by Karthikeyan
+Singaravelan.
+
+..
+
+.. bpo: 35121
+.. date: 2019-05-20-00-35-12
+.. nonce: RRi-HU
+.. section: Security
+
+Don't send cookies of domain A without Domain attribute to domain B when
+domain A is a suffix match of domain B while using a cookiejar with
+:class:`cookielib.DefaultCookiePolicy` policy. Patch by Karthikeyan
+Singaravelan.
+
+..
+
+.. bpo: 34155
+.. date: 2019-05-04-13-33-37
+.. nonce: MJll68
+.. section: Security
+
+Fix parsing of invalid email addresses with more than one ``@`` (e.g.
+a at b@c.com.) to not return the part before 2nd ``@`` as valid email address.
+Patch by maxking & jpic.
+
+..
+
+.. bpo: 36742
+.. date: 2019-04-29-15-34-59
+.. nonce: QCUY0i
+.. section: Security
+
+Fixes mishandling of pre-normalization characters in urlsplit().
+
+..
+
+.. bpo: 30458
+.. date: 2019-04-10-08-53-30
+.. nonce: 51E-DA
+.. section: Security
+
+Address CVE-2019-9740 by disallowing URL paths with embedded whitespace or
+control characters through into the underlying http client request.  Such
+potentially malicious header injection URLs now cause an httplib.InvalidURL
+exception to be raised.
+
+..
+
+.. bpo: 36216
+.. date: 2019-03-06-09-38-40
+.. nonce: 6q1m4a
+.. section: Security
+
+Changes urlsplit() to raise ValueError when the URL contains characters that
+decompose under IDNA encoding (NFKC-normalization) into characters that
+affect how the URL is parsed.
+
+..
+
+.. bpo: 35907
+.. date: 2019-02-13-17-21-10
+.. nonce: ckk2zg
+.. section: Security
+
+CVE-2019-9948: Avoid file reading by disallowing ``local-file://`` and
+``local_file://`` URL schemes in :func:`urllib.urlopen`,
+:meth:`urllib.URLopener.open` and :meth:`urllib.URLopener.retrieve`.
+
+..
+
+.. bpo: 38106
+.. date: 2019-09-12-16-36-16
+.. nonce: 4pApn7
+.. section: Core and Builtins
+
+Fix race in PyThread_release_lock that was leading to memory corruption and
+deadlocks. The fix applies to POSIX systems where Python locks are
+implemented with mutex and condition variable because POSIX semaphores are
+either not provided, or are known to be broken. One particular example of
+such system is macOS.
+
+..
+
+.. bpo: 37329
+.. date: 2019-06-18-15-08-32
+.. nonce: 7XVDRv
+.. section: Core and Builtins
+
+valgrind: suppress a false alarm in memory leak checks. _PyWarnings_Init()
+only allocates memory once at startup but it is not released at exit. Ignore
+this issue to be able to catch other bugs more easily.
+
+..
+
+.. bpo: 26423
+.. date: 2019-05-27-18-00-19
+.. nonce: RgUOE8
+.. section: Core and Builtins
+
+Fix possible overflow in ``wrap_lenfunc()`` when ``sizeof(long) <
+sizeof(Py_ssize_t)`` (e.g., 64-bit Windows).
+
+..
+
+.. bpo: 27987
+.. date: 2019-04-16-11-52-21
+.. nonce: n2_DcQ
+.. section: Core and Builtins
+
+pymalloc returns memory blocks aligned by 16 bytes, instead of 8 bytes, on
+64-bit platforms to conform x86-64 ABI. Recent compilers assume this
+alignment more often. Patch by Inada Naoki.
+
+..
+
+.. bpo: 36504
+.. date: 2019-04-02-04-10-32
+.. nonce: k_V8Bm
+.. section: Core and Builtins
+
+Fix signed integer overflow in _ctypes.c's ``PyCArrayType_new()``.
+
+..
+
+.. bpo: 36459
+.. date: 2019-03-27-22-35-16
+.. nonce: UAvkKp
+.. section: Core and Builtins
+
+Fix a possible double ``PyMem_FREE()`` due to tokenizer.c's ``tok_nextc()``.
+
+..
+
+.. bpo: 36430
+.. date: 2019-03-25-23-37-26
+.. nonce: sd9xxQ
+.. section: Core and Builtins
+
+Fix a possible reference leak in :func:`itertools.count`.
+
+..
+
+.. bpo: 18368
+.. date: 2019-03-14-17-30-46
+.. nonce: WXaHAo
+.. section: Core and Builtins
+
+PyOS_StdioReadline() no longer leaks memory when realloc() fails.
+
+..
+
+.. bpo: 36262
+.. date: 2019-03-11-15-37-33
+.. nonce: v3N6Fz
+.. section: Core and Builtins
+
+Fix an unlikely memory leak on conversion from string to float in the
+function ``_Py_dg_strtod()`` used by ``float(str)``, ``complex(str)``,
+:func:`pickle.load`, :func:`marshal.load`, etc.
+
+..
+
+.. bpo: 36149
+.. date: 2019-02-28-13-52-18
+.. nonce: GJdnh4
+.. section: Core and Builtins
+
+Fix use of uninitialized memory in cPickle when reading a truncated pickle
+from a file object.
+
+..
+
+.. bpo: 33006
+.. date: 2018-03-07-09-10-42
+.. nonce: Bzx3LA
+.. section: Core and Builtins
+
+Clarified Doc string for builtin filter function. 2nd Argument can be any
+iterable. Patch by Tony Flury
+
+..
+
+.. bpo: 38216
+.. date: 2019-09-27-15-24-45
+.. nonce: -7yvZR
+.. section: Library
+
+Allow the rare code that wants to send invalid http requests from the
+`http.client` library a way to do so.  The fixes for bpo-30458 led to
+breakage for some projects that were relying on this ability to test their
+own behavior in the face of bad requests.
+
+..
+
+.. bpo: 38175
+.. date: 2019-09-15-10-30-33
+.. nonce: 61XlUv
+.. section: Library
+
+Fix a memory leak in comparison of :class:`sqlite3.Row` objects.
+
+..
+
+.. bpo: 33936
+.. date: 2019-09-14-10-34-00
+.. nonce: 8wCI_n
+.. section: Library
+
+_hashlib no longer calls obsolete OpenSSL initialization function with
+OpenSSL 1.1.0+.
+
+..
+
+.. bpo: 34410
+.. date: 2019-08-31-01-52-59
+.. nonce: 7KbWZQ
+.. section: Library
+
+Fixed a crash in the :func:`tee` iterator when re-enter it. RuntimeError is
+now raised in this case.
+
+..
+
+.. bpo: 37965
+.. date: 2019-08-28-14-04-18
+.. nonce: 7xGE-C
+.. section: Library
+
+Fix C compiler warning caused by distutils.ccompiler.CCompiler.has_function.
+
+..
+
+.. bpo: 34521
+.. date: 2019-08-23-14-47-09
+.. nonce: Y2BYu5
+.. section: Library
+
+Fix file descriptors transfer in multiprocessing on FreeBSD: use
+``CMSG_SPACE()`` rather than ``CMSG_LEN()``; see :rfc:`3542`.
+
+..
+
+.. bpo: 37664
+.. date: 2019-07-24-18-27-44
+.. nonce: o-GYZC
+.. section: Library
+
+Update wheels bundled with ensurepip (pip 19.2.3 and setuptools 41.2.0)
+
+..
+
+.. bpo: 37437
+.. date: 2019-06-27-20-33-50
+.. nonce: du39_A
+.. section: Library
+
+Update vendorized expat version to 2.2.7.
+
+..
+
+.. bpo: 36742
+.. date: 2019-06-10-12-02-45
+.. nonce: UEdHXJ
+.. section: Library
+
+:func:`urlparse.urlsplit` error message for invalid ``netloc`` according to
+NFKC normalization is now a :class:`str` string, rather than a
+:class:`unicode` string, to prevent error when displaying the error.
+
+..
+
+.. bpo: 12639
+.. date: 2019-05-30-16-16-47
+.. nonce: TQFOR4
+.. section: Library
+
+:meth:`msilib.Directory.start_component()` no longer fails if *keyfile* is
+not ``None``.
+
+..
+
+.. bpo: 36713
+.. date: 2019-05-23-15-57-36
+.. nonce: sjPhnf
+.. section: Library
+
+Rename the :meth:`test_ascii_replace` to :meth:`test_ascii_strict`.
+
+..
+
+.. bpo: 28552
+.. date: 2019-04-18-16-10-29
+.. nonce: MW1TLt
+.. section: Library
+
+Fix :mod:`distutils.sysconfig` if :data:`sys.executable` is ``None`` or an
+empty string: use :func:`os.getcwd` to initialize ``project_base``.  Fix
+also the distutils build command: don't use :data:`sys.executable` if it is
+``None`` or an empty string.
+
+..
+
+.. bpo: 36337
+.. date: 2019-03-18-10-08-30
+.. nonce: QhJnXy
+.. section: Library
+
+Fix buffer overflow in :meth:`~socket.socket.send` and
+:meth:`~socket.socket.sendall` methods of :func:`socket.socket` for data
+larger than 2 GiB.
+
+..
+
+.. bpo: 36291
+.. date: 2019-03-14-15-54-46
+.. nonce: UalHXP
+.. section: Library
+
+Fix a possible reference leak in the json module.
+
+..
+
+.. bpo: 36289
+.. date: 2019-03-14-15-42-48
+.. nonce: wYKS47
+.. section: Library
+
+Fix a possible reference leak in the io module.
+
+..
+
+.. bpo: 36212
+.. date: 2019-03-14-14-40-22
+.. nonce: IEgRI8
+.. section: Library
+
+Fix two possible reference leaks in the hotshot module.
+
+..
+
+.. bpo: 36235
+.. date: 2019-03-08-13-32-21
+.. nonce: _M72wU
+.. section: Library
+
+Fix ``CFLAGS`` in ``customize_compiler()`` of ``distutils.sysconfig``: when
+the ``CFLAGS`` environment variable is defined, don't override ``CFLAGS``
+variable with the ``OPT`` variable anymore. Initial patch written by David
+Malcolm.
+
+..
+
+.. bpo: 35807
+.. date: 2019-03-06-13-21-33
+.. nonce: W7mmu3
+.. section: Library
+
+Update ensurepip to install pip 19.0.3 and setuptools 40.8.0.
+
+..
+
+.. bpo: 36186
+.. date: 2019-03-04-16-39-16
+.. nonce: Hqw1A_
+.. section: Library
+
+Fix linuxaudiodev.linux_audio_device() error handling: close the internal
+file descriptor if it fails to open the device.
+
+..
+
+.. bpo: 13096
+.. date: 2019-03-04-16-13-01
+.. nonce: SGPt_n
+.. section: Library
+
+Fix memory leak in ctypes POINTER handling of large values.
+
+..
+
+.. bpo: 36179
+.. date: 2019-03-04-10-42-46
+.. nonce: jEyuI-
+.. section: Library
+
+Fix two unlikely reference leaks in _hashopenssl. The leaks only occur in
+out-of-memory cases.
+
+..
+
+.. bpo: 36106
+.. date: 2019-02-25-13-21-43
+.. nonce: VuhEiQ
+.. section: Library
+
+Resolve potential name clash with libm's sinpi(). Patch by Dmitrii
+Pasechnik.
+
+..
+
+.. bpo: 31292
+.. date: 2017-08-30-20-27-00
+.. nonce: dKIaZb
+.. section: Library
+
+Fix ``setup.py check --restructuredtext`` for files containing ``include``
+directives.
+
+..
+
+.. bpo: 37487
+.. date: 2019-07-06-17-19-26
+.. nonce: QagfZ5
+.. section: Documentation
+
+Fix PyList_GetItem index description to include 0.
+
+..
+
+.. bpo: 37149
+.. date: 2019-07-06-02-19-02
+.. nonce: NumHn3
+.. section: Documentation
+
+Replace the dead link to the Tkinter 8.5 reference by John Shipman, New
+Mexico Tech, with a link to the archive.org copy.
+
+..
+
+.. bpo: 35126
+.. date: 2019-02-18-10-01-07
+.. nonce: LWwl8X
+.. section: Documentation
+
+Improve the examples in the "How do I convert a number to string?" question
+of the "Programming" section of the FAQ.  Contributed by St?phane Wirtel.
+
+..
+
+.. bpo: 35605
+.. date: 2018-12-30-09-56-13
+.. nonce: gAWt32
+.. section: Documentation
+
+Fix documentation build for sphinx<1.6.  Patch by Anthony Sottile.
+
+..
+
+.. bpo: 35564
+.. date: 2018-12-22-22-52-05
+.. nonce: TuEU_D
+.. section: Documentation
+
+Explicitly set master_doc variable in conf.py for compliance with Sphinx 2.0
+
+..
+
+.. bpo: 33832
+.. date: 2018-06-15-15-57-37
+.. nonce: xBFhKw
+.. section: Documentation
+
+Add glossary entry for 'magic method'.
+
+..
+
+.. bpo: 37411
+.. date: 2019-06-26-15-28-45
+.. nonce: 5lGNhM
+.. section: Tests
+
+Fix test_wsgiref.testEnviron() to no longer depend on the environment
+variables (don't fail if "X" variable is set).
+
+..
+
+.. bpo: 37359
+.. date: 2019-06-24-10-47-07
+.. nonce: CkdtyO
+.. section: Tests
+
+Add --cleanup option to python3 -m test to remove ``test_python_*``
+directories of previous failed jobs. Add "make cleantest" to run ``python3
+-m test --cleanup``.
+
+..
+
+.. bpo: 37362
+.. date: 2019-06-21-15-47-33
+.. nonce: D3xppx
+.. section: Tests
+
+test_gdb no longer fails if it gets an "unexpected" message on stderr: it
+now ignores stderr. The purpose of test_gdb is to test that python-gdb.py
+commands work as expected, not to test gdb.
+
+..
+
+.. bpo: 36816
+.. date: 2019-05-08-15-55-46
+.. nonce: WBKRGZ
+.. section: Tests
+
+Update Lib/test/selfsigned_pythontestdotnet.pem to match
+self-signed.pythontest.net's new TLS certificate.
+
+..
+
+.. bpo: 35925
+.. date: 2019-05-06-18-29-54
+.. nonce: gwQPuC
+.. section: Tests
+
+Skip specific nntplib and ssl networking tests when they would otherwise
+fail due to a modern OS or distro with a default OpenSSL policy of rejecting
+connections to servers with weak certificates or disabling TLS below
+TLSv1.2.
+
+..
+
+.. bpo: 36560
+.. date: 2019-04-09-14-08-02
+.. nonce: _ejeOr
+.. section: Tests
+
+Fix reference leak hunting in regrtest: compute also deltas (of reference
+count and file descriptor count) during warmup, to ensure that everything is
+initialized before starting to hunt reference leaks.
+
+..
+
+.. bpo: 36234
+.. date: 2019-03-08-12-53-37
+.. nonce: NRVK6W
+.. section: Tests
+
+test_posix.PosixUidGidTests: add tests for invalid uid/gid type (str). Patch
+written by David Malcolm.
+
+..
+
+.. bpo: 36019
+.. date: 2019-03-05-13-27-36
+.. nonce: ebUjCm
+.. section: Tests
+
+Add test.support.TEST_HTTP_URL and replace references of
+http://www.example.com by this new constant. Contributed by St?phane Wirtel.
+
+..
+
+.. bpo: 27313
+.. date: 2019-02-24-03-19-43
+.. nonce: Sj9veH
+.. section: Tests
+
+Avoid test_ttk_guionly ComboboxTest failure with macOS Cocoa Tk.
+
+..
+
+.. bpo: 26386
+.. date: 2019-02-24-02-44-52
+.. nonce: YZylPP
+.. section: Tests
+
+Re-enable missing widget testcases in test_ttk_guionly.
+
+..
+
+.. bpo: 34836
+.. date: 2019-01-11-14-01-19
+.. nonce: 7fat9-
+.. section: Tests
+
+Fix ``test_default_ecdh_curve`` when TLSv1.3 is enabled by default.
+
+..
+
+.. bpo: 38301
+.. date: 2019-09-28-02-37-11
+.. nonce: 123456
+.. section: Build
+
+In Solaris family, we must be sure to use ``-D_REENTRANT``. Patch by Jes?s
+Cea Avi?n.
+
+..
+
+.. bpo: 14353
+.. date: 2019-05-12-10-52-37
+.. nonce: LK1qWM
+.. section: Build
+
+Fix detection of the bind_textdomain_codeset function for building gettext
+support into the locale module.
+
+..
+
+.. bpo: 36605
+.. date: 2019-04-11-18-50-58
+.. nonce: gk5czf
+.. section: Build
+
+``make tags`` and ``make TAGS`` now also parse ``Modules/_io/*.c`` and
+``Modules/_io/*.h``.
+
+..
+
+.. bpo: 35264
+.. date: 2018-11-16-15-19-09
+.. nonce: h5GxH3
+.. section: Build
+
+Fix SSL module build with OpenSSL 1.1.0
+
+..
+
+.. bpo: 38117
+.. date: 2019-09-16-12-47-12
+.. nonce: JaiB0X
+.. section: Windows
+
+Updates bundled OpenSSL to 1.0.2t
+
+..
+
+.. bpo: 37445
+.. date: 2019-06-28-18-10-29
+.. nonce: LsdYO6
+.. section: Windows
+
+Include the ``FORMAT_MESSAGE_IGNORE_INSERTS`` flag in ``FormatMessageW()``
+calls.
+
+..
+
+.. bpo: 35360
+.. date: 2019-06-18-09-53-15
+.. nonce: tdqSmo
+.. section: Windows
+
+Update Windows builds to use SQLite 3.28.0.
+
+..
+
+.. bpo: 1104
+.. date: 2017-11-24-12-53-54
+.. nonce: 1CWSZp
+.. section: Windows
+
+Correctly handle string length in ``msilib.SummaryInfo.GetProperty()`` to
+prevent it from truncating the last character.
+
+..
+
+.. bpo: 38117
+.. date: 2019-09-15-21-45-15
+.. nonce: _U9F2r
+.. section: macOS
+
+Updated OpenSSL to 1.0.2t in macOS installer for 2.7.x.
+
+..
+
+.. bpo: 19960
+.. date: 2019-06-20-01-16-16
+.. nonce: LpLUPF
+.. section: macOS
+
+When building 2.7 on macOS without system header files installed in
+``/usr/include``, a few extension modules dependent on system-supplied
+third-party libraries were not being built, most notably zlib.
+
+..
+
+.. bpo: 35360
+.. date: 2019-06-18-10-02-11
+.. nonce: -CWbfy
+.. section: macOS
+
+Update macOS installer to use SQLite 3.28.0.
+
+..
+
+.. bpo: 34631
+.. date: 2019-06-18-05-39-01
+.. nonce: StdZhE
+.. section: macOS
+
+Updated OpenSSL to 1.0.2s in macOS installer.
+
+..
+
+.. bpo: 36231
+.. date: 2019-06-03-05-49-49
+.. nonce: RfmW_p
+.. section: macOS
+
+Support building Python on macOS without /usr/include installed. As of macOS
+10.14, system header files are only available within an SDK provided by
+either the Command Line Tools or the Xcode app.
+
+..
+
+.. bpo: 37177
+.. date: 2019-06-07-00-17-41
+.. nonce: voU6pQ
+.. section: IDLE
+
+Properly 'attach' search dialogs to their main window so that they behave
+like other dialogs and do not get hidden behind their main window.
+
+..
+
+.. bpo: 13102
+.. date: 2019-05-05-16-27-53
+.. nonce: AGNWYJ
+.. section: IDLE
+
+When saving a file, call os.fsync() so bits are flushed to e.g. USB drive.
+
+..
+
+.. bpo: 37675
+.. date: 2019-07-24-16-20-54
+.. nonce: 951Cvf
+.. section: Tools/Demos
+
+2to3 now works when run from a zipped standard library.
+
+..
+
+.. bpo: 14546
+.. date: 2019-04-30-14-30-29
+.. nonce: r38Y-6
+.. section: Tools/Demos
+
+Fix the argument handling in Tools/scripts/lll.py.
+
+..
+
+.. bpo: 37170
+.. date: 2019-06-06-08-47-04
+.. nonce: hO_fpM
+.. section: C API
+
+Fix the cast on error in :c:func:`PyLong_AsUnsignedLongLongMask()`.
diff --git a/Misc/NEWS.d/next/Build/2018-11-16-15-19-09.bpo-35264.h5GxH3.rst b/Misc/NEWS.d/next/Build/2018-11-16-15-19-09.bpo-35264.h5GxH3.rst
deleted file mode 100644
index d17968a52debf..0000000000000
--- a/Misc/NEWS.d/next/Build/2018-11-16-15-19-09.bpo-35264.h5GxH3.rst
+++ /dev/null
@@ -1 +0,0 @@
-Fix SSL module build with OpenSSL 1.1.0
diff --git a/Misc/NEWS.d/next/Build/2019-04-11-18-50-58.bpo-36605.gk5czf.rst b/Misc/NEWS.d/next/Build/2019-04-11-18-50-58.bpo-36605.gk5czf.rst
deleted file mode 100644
index 4a558fa94d6f7..0000000000000
--- a/Misc/NEWS.d/next/Build/2019-04-11-18-50-58.bpo-36605.gk5czf.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-``make tags`` and ``make TAGS`` now also parse ``Modules/_io/*.c`` and
-``Modules/_io/*.h``.
diff --git a/Misc/NEWS.d/next/Build/2019-05-12-10-52-37.bpo-14353.LK1qWM.rst b/Misc/NEWS.d/next/Build/2019-05-12-10-52-37.bpo-14353.LK1qWM.rst
deleted file mode 100644
index 00ff22893812f..0000000000000
--- a/Misc/NEWS.d/next/Build/2019-05-12-10-52-37.bpo-14353.LK1qWM.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Fix detection of the bind_textdomain_codeset function for building gettext
-support into the locale module.
diff --git a/Misc/NEWS.d/next/Build/2019-09-28-02-37-11.bpo-38301.123456.rst b/Misc/NEWS.d/next/Build/2019-09-28-02-37-11.bpo-38301.123456.rst
deleted file mode 100644
index 59c9a76385e2b..0000000000000
--- a/Misc/NEWS.d/next/Build/2019-09-28-02-37-11.bpo-38301.123456.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-In Solaris family, we must be sure to use ``-D_REENTRANT``.
-Patch by Jes?s Cea Avi?n.
diff --git a/Misc/NEWS.d/next/C API/2019-06-06-08-47-04.bpo-37170.hO_fpM.rst b/Misc/NEWS.d/next/C API/2019-06-06-08-47-04.bpo-37170.hO_fpM.rst
deleted file mode 100644
index 7a35c9583d691..0000000000000
--- a/Misc/NEWS.d/next/C API/2019-06-06-08-47-04.bpo-37170.hO_fpM.rst	
+++ /dev/null
@@ -1 +0,0 @@
-Fix the cast on error in :c:func:`PyLong_AsUnsignedLongLongMask()`.
diff --git a/Misc/NEWS.d/next/Core and Builtins/2018-03-07-09-10-42.bpo-33006.Bzx3LA.rst b/Misc/NEWS.d/next/Core and Builtins/2018-03-07-09-10-42.bpo-33006.Bzx3LA.rst
deleted file mode 100644
index 0bcc0a4b987dc..0000000000000
--- a/Misc/NEWS.d/next/Core and Builtins/2018-03-07-09-10-42.bpo-33006.Bzx3LA.rst	
+++ /dev/null
@@ -1,2 +0,0 @@
-Clarified Doc string for builtin filter function. 2nd Argument can be any
-iterable. Patch by Tony Flury
diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-02-28-13-52-18.bpo-36149.GJdnh4.rst b/Misc/NEWS.d/next/Core and Builtins/2019-02-28-13-52-18.bpo-36149.GJdnh4.rst
deleted file mode 100644
index 672db6c1fc07c..0000000000000
--- a/Misc/NEWS.d/next/Core and Builtins/2019-02-28-13-52-18.bpo-36149.GJdnh4.rst	
+++ /dev/null
@@ -1,2 +0,0 @@
-Fix use of uninitialized memory in cPickle when reading a truncated pickle
-from a file object.
diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-03-11-15-37-33.bpo-36262.v3N6Fz.rst b/Misc/NEWS.d/next/Core and Builtins/2019-03-11-15-37-33.bpo-36262.v3N6Fz.rst
deleted file mode 100644
index b5ccc95fd705b..0000000000000
--- a/Misc/NEWS.d/next/Core and Builtins/2019-03-11-15-37-33.bpo-36262.v3N6Fz.rst	
+++ /dev/null
@@ -1,3 +0,0 @@
-Fix an unlikely memory leak on conversion from string to float in the function
-``_Py_dg_strtod()`` used by ``float(str)``, ``complex(str)``,
-:func:`pickle.load`, :func:`marshal.load`, etc.
diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-03-14-17-30-46.bpo-18368.WXaHAo.rst b/Misc/NEWS.d/next/Core and Builtins/2019-03-14-17-30-46.bpo-18368.WXaHAo.rst
deleted file mode 100644
index 7f2fb898fbc6e..0000000000000
--- a/Misc/NEWS.d/next/Core and Builtins/2019-03-14-17-30-46.bpo-18368.WXaHAo.rst	
+++ /dev/null
@@ -1 +0,0 @@
-PyOS_StdioReadline() no longer leaks memory when realloc() fails.
diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-03-25-23-37-26.bpo-36430.sd9xxQ.rst b/Misc/NEWS.d/next/Core and Builtins/2019-03-25-23-37-26.bpo-36430.sd9xxQ.rst
deleted file mode 100644
index a65ee096efc4a..0000000000000
--- a/Misc/NEWS.d/next/Core and Builtins/2019-03-25-23-37-26.bpo-36430.sd9xxQ.rst	
+++ /dev/null
@@ -1 +0,0 @@
-Fix a possible reference leak in :func:`itertools.count`.
diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-03-27-22-35-16.bpo-36459.UAvkKp.rst b/Misc/NEWS.d/next/Core and Builtins/2019-03-27-22-35-16.bpo-36459.UAvkKp.rst
deleted file mode 100644
index 6c234a6a76d9a..0000000000000
--- a/Misc/NEWS.d/next/Core and Builtins/2019-03-27-22-35-16.bpo-36459.UAvkKp.rst	
+++ /dev/null
@@ -1 +0,0 @@
-Fix a possible double ``PyMem_FREE()`` due to tokenizer.c's ``tok_nextc()``.
diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-04-02-04-10-32.bpo-36504.k_V8Bm.rst b/Misc/NEWS.d/next/Core and Builtins/2019-04-02-04-10-32.bpo-36504.k_V8Bm.rst
deleted file mode 100644
index 8ac209d4a789b..0000000000000
--- a/Misc/NEWS.d/next/Core and Builtins/2019-04-02-04-10-32.bpo-36504.k_V8Bm.rst	
+++ /dev/null
@@ -1 +0,0 @@
-Fix signed integer overflow in _ctypes.c's ``PyCArrayType_new()``.
diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-04-16-11-52-21.bpo-27987.n2_DcQ.rst b/Misc/NEWS.d/next/Core and Builtins/2019-04-16-11-52-21.bpo-27987.n2_DcQ.rst
deleted file mode 100644
index b0f32a5c6c3f4..0000000000000
--- a/Misc/NEWS.d/next/Core and Builtins/2019-04-16-11-52-21.bpo-27987.n2_DcQ.rst	
+++ /dev/null
@@ -1,3 +0,0 @@
-pymalloc returns memory blocks aligned by 16 bytes, instead of 8 bytes, on
-64-bit platforms to conform x86-64 ABI. Recent compilers assume this alignment
-more often. Patch by Inada Naoki.
diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-05-27-18-00-19.bpo-26423.RgUOE8.rst b/Misc/NEWS.d/next/Core and Builtins/2019-05-27-18-00-19.bpo-26423.RgUOE8.rst
deleted file mode 100644
index 6bf2031a33848..0000000000000
--- a/Misc/NEWS.d/next/Core and Builtins/2019-05-27-18-00-19.bpo-26423.RgUOE8.rst	
+++ /dev/null
@@ -1,2 +0,0 @@
-Fix possible overflow in ``wrap_lenfunc()`` when
-``sizeof(long) < sizeof(Py_ssize_t)`` (e.g., 64-bit Windows).
diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-06-18-15-08-32.bpo-37329.7XVDRv.rst b/Misc/NEWS.d/next/Core and Builtins/2019-06-18-15-08-32.bpo-37329.7XVDRv.rst
deleted file mode 100644
index faae4f633a061..0000000000000
--- a/Misc/NEWS.d/next/Core and Builtins/2019-06-18-15-08-32.bpo-37329.7XVDRv.rst	
+++ /dev/null
@@ -1,3 +0,0 @@
-valgrind: suppress a false alarm in memory leak checks. _PyWarnings_Init()
-only allocates memory once at startup but it is not released at exit. Ignore
-this issue to be able to catch other bugs more easily.
diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-09-12-16-36-16.bpo-38106.4pApn7.rst b/Misc/NEWS.d/next/Core and Builtins/2019-09-12-16-36-16.bpo-38106.4pApn7.rst
deleted file mode 100644
index b2b80ceece8f2..0000000000000
--- a/Misc/NEWS.d/next/Core and Builtins/2019-09-12-16-36-16.bpo-38106.4pApn7.rst	
+++ /dev/null
@@ -1,5 +0,0 @@
-Fix race in PyThread_release_lock that was leading to memory corruption and
-deadlocks. The fix applies to POSIX systems where Python locks are implemented
-with mutex and condition variable because POSIX semaphores are either not
-provided, or are known to be broken. One particular example of such system is
-macOS.
diff --git a/Misc/NEWS.d/next/Documentation/2018-06-15-15-57-37.bpo-33832.xBFhKw.rst b/Misc/NEWS.d/next/Documentation/2018-06-15-15-57-37.bpo-33832.xBFhKw.rst
deleted file mode 100644
index 3d1c63acca3bb..0000000000000
--- a/Misc/NEWS.d/next/Documentation/2018-06-15-15-57-37.bpo-33832.xBFhKw.rst
+++ /dev/null
@@ -1 +0,0 @@
-Add glossary entry for 'magic method'.
diff --git a/Misc/NEWS.d/next/Documentation/2018-12-22-22-52-05.bpo-35564.TuEU_D.rst b/Misc/NEWS.d/next/Documentation/2018-12-22-22-52-05.bpo-35564.TuEU_D.rst
deleted file mode 100644
index 8ca95eed4c56e..0000000000000
--- a/Misc/NEWS.d/next/Documentation/2018-12-22-22-52-05.bpo-35564.TuEU_D.rst
+++ /dev/null
@@ -1 +0,0 @@
-Explicitly set master_doc variable in conf.py for compliance with Sphinx 2.0
diff --git a/Misc/NEWS.d/next/Documentation/2018-12-30-09-56-13.bpo-35605.gAWt32.rst b/Misc/NEWS.d/next/Documentation/2018-12-30-09-56-13.bpo-35605.gAWt32.rst
deleted file mode 100644
index cbc0f1e07f312..0000000000000
--- a/Misc/NEWS.d/next/Documentation/2018-12-30-09-56-13.bpo-35605.gAWt32.rst
+++ /dev/null
@@ -1 +0,0 @@
-Fix documentation build for sphinx<1.6.  Patch by Anthony Sottile.
diff --git a/Misc/NEWS.d/next/Documentation/2019-02-18-10-01-07.bpo-35126.LWwl8X.rst b/Misc/NEWS.d/next/Documentation/2019-02-18-10-01-07.bpo-35126.LWwl8X.rst
deleted file mode 100644
index 8ad392ee3c97e..0000000000000
--- a/Misc/NEWS.d/next/Documentation/2019-02-18-10-01-07.bpo-35126.LWwl8X.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Improve the examples in the "How do I convert a number to string?" question
-of the "Programming" section of the FAQ.  Contributed by St?phane Wirtel.
diff --git a/Misc/NEWS.d/next/Documentation/2019-07-06-02-19-02.bpo-37149.NumHn3.rst b/Misc/NEWS.d/next/Documentation/2019-07-06-02-19-02.bpo-37149.NumHn3.rst
deleted file mode 100644
index f9b88dc0bfe0d..0000000000000
--- a/Misc/NEWS.d/next/Documentation/2019-07-06-02-19-02.bpo-37149.NumHn3.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Replace the dead link to the Tkinter 8.5 reference by John Shipman, New
-Mexico Tech, with a link to the archive.org copy.
diff --git a/Misc/NEWS.d/next/Documentation/2019-07-06-17-19-26.bpo-37487.QagfZ5.rst b/Misc/NEWS.d/next/Documentation/2019-07-06-17-19-26.bpo-37487.QagfZ5.rst
deleted file mode 100644
index 605d08c3c0406..0000000000000
--- a/Misc/NEWS.d/next/Documentation/2019-07-06-17-19-26.bpo-37487.QagfZ5.rst
+++ /dev/null
@@ -1 +0,0 @@
-Fix PyList_GetItem index description to include 0.
diff --git a/Misc/NEWS.d/next/IDLE/2019-05-05-16-27-53.bpo-13102.AGNWYJ.rst b/Misc/NEWS.d/next/IDLE/2019-05-05-16-27-53.bpo-13102.AGNWYJ.rst
deleted file mode 100644
index 2a905bf0eecfd..0000000000000
--- a/Misc/NEWS.d/next/IDLE/2019-05-05-16-27-53.bpo-13102.AGNWYJ.rst
+++ /dev/null
@@ -1 +0,0 @@
-When saving a file, call os.fsync() so bits are flushed to e.g. USB drive.
diff --git a/Misc/NEWS.d/next/IDLE/2019-06-07-00-17-41.bpo-37177.voU6pQ.rst b/Misc/NEWS.d/next/IDLE/2019-06-07-00-17-41.bpo-37177.voU6pQ.rst
deleted file mode 100644
index 74e48eef89a62..0000000000000
--- a/Misc/NEWS.d/next/IDLE/2019-06-07-00-17-41.bpo-37177.voU6pQ.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Properly 'attach' search dialogs to their main window so that they behave
-like other dialogs and do not get hidden behind their main window.
diff --git a/Misc/NEWS.d/next/Library/2017-08-30-20-27-00.bpo-31292.dKIaZb.rst b/Misc/NEWS.d/next/Library/2017-08-30-20-27-00.bpo-31292.dKIaZb.rst
deleted file mode 100644
index b62eee3c5ee49..0000000000000
--- a/Misc/NEWS.d/next/Library/2017-08-30-20-27-00.bpo-31292.dKIaZb.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Fix ``setup.py check --restructuredtext`` for
-files containing ``include`` directives.
diff --git a/Misc/NEWS.d/next/Library/2019-02-25-13-21-43.bpo-36106.VuhEiQ.rst b/Misc/NEWS.d/next/Library/2019-02-25-13-21-43.bpo-36106.VuhEiQ.rst
deleted file mode 100644
index 36e17508cd4a0..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-02-25-13-21-43.bpo-36106.VuhEiQ.rst
+++ /dev/null
@@ -1 +0,0 @@
-Resolve potential name clash with libm's sinpi(). Patch by Dmitrii Pasechnik.
diff --git a/Misc/NEWS.d/next/Library/2019-03-04-10-42-46.bpo-36179.jEyuI-.rst b/Misc/NEWS.d/next/Library/2019-03-04-10-42-46.bpo-36179.jEyuI-.rst
deleted file mode 100644
index 61a98778b78e9..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-03-04-10-42-46.bpo-36179.jEyuI-.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Fix two unlikely reference leaks in _hashopenssl. The leaks only occur in
-out-of-memory cases.
diff --git a/Misc/NEWS.d/next/Library/2019-03-04-16-13-01.bpo-13096.SGPt_n.rst b/Misc/NEWS.d/next/Library/2019-03-04-16-13-01.bpo-13096.SGPt_n.rst
deleted file mode 100644
index 2bf49c855f60c..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-03-04-16-13-01.bpo-13096.SGPt_n.rst
+++ /dev/null
@@ -1 +0,0 @@
-Fix memory leak in ctypes POINTER handling of large values.
diff --git a/Misc/NEWS.d/next/Library/2019-03-04-16-39-16.bpo-36186.Hqw1A_.rst b/Misc/NEWS.d/next/Library/2019-03-04-16-39-16.bpo-36186.Hqw1A_.rst
deleted file mode 100644
index a14d155ae9734..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-03-04-16-39-16.bpo-36186.Hqw1A_.rst
+++ /dev/null
@@ -1 +0,0 @@
-Fix linuxaudiodev.linux_audio_device() error handling: close the internal file descriptor if it fails to open the device.
diff --git a/Misc/NEWS.d/next/Library/2019-03-06-13-21-33.bpo-35807.W7mmu3.rst b/Misc/NEWS.d/next/Library/2019-03-06-13-21-33.bpo-35807.W7mmu3.rst
deleted file mode 100644
index 1109fbed3f9cc..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-03-06-13-21-33.bpo-35807.W7mmu3.rst
+++ /dev/null
@@ -1 +0,0 @@
-Update ensurepip to install pip 19.0.3 and setuptools 40.8.0.
diff --git a/Misc/NEWS.d/next/Library/2019-03-08-13-32-21.bpo-36235._M72wU.rst b/Misc/NEWS.d/next/Library/2019-03-08-13-32-21.bpo-36235._M72wU.rst
deleted file mode 100644
index 59df98c101afc..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-03-08-13-32-21.bpo-36235._M72wU.rst
+++ /dev/null
@@ -1,4 +0,0 @@
-Fix ``CFLAGS`` in ``customize_compiler()`` of ``distutils.sysconfig``: when
-the ``CFLAGS`` environment variable is defined, don't override ``CFLAGS``
-variable with the ``OPT`` variable anymore. Initial patch written by David
-Malcolm.
diff --git a/Misc/NEWS.d/next/Library/2019-03-14-14-40-22.bpo-36212.IEgRI8.rst b/Misc/NEWS.d/next/Library/2019-03-14-14-40-22.bpo-36212.IEgRI8.rst
deleted file mode 100644
index b2d264234e4d8..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-03-14-14-40-22.bpo-36212.IEgRI8.rst
+++ /dev/null
@@ -1 +0,0 @@
-Fix two possible reference leaks in the hotshot module.
diff --git a/Misc/NEWS.d/next/Library/2019-03-14-15-42-48.bpo-36289.wYKS47.rst b/Misc/NEWS.d/next/Library/2019-03-14-15-42-48.bpo-36289.wYKS47.rst
deleted file mode 100644
index 8917bee727bf0..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-03-14-15-42-48.bpo-36289.wYKS47.rst
+++ /dev/null
@@ -1 +0,0 @@
-Fix a possible reference leak in the io module.
diff --git a/Misc/NEWS.d/next/Library/2019-03-14-15-54-46.bpo-36291.UalHXP.rst b/Misc/NEWS.d/next/Library/2019-03-14-15-54-46.bpo-36291.UalHXP.rst
deleted file mode 100644
index 07d780cd7fd6e..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-03-14-15-54-46.bpo-36291.UalHXP.rst
+++ /dev/null
@@ -1 +0,0 @@
-Fix a possible reference leak in the json module.
diff --git a/Misc/NEWS.d/next/Library/2019-03-18-10-08-30.bpo-36337.QhJnXy.rst b/Misc/NEWS.d/next/Library/2019-03-18-10-08-30.bpo-36337.QhJnXy.rst
deleted file mode 100644
index 07cd8e2bd6118..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-03-18-10-08-30.bpo-36337.QhJnXy.rst
+++ /dev/null
@@ -1,3 +0,0 @@
-Fix buffer overflow in :meth:`~socket.socket.send` and
-:meth:`~socket.socket.sendall` methods of :func:`socket.socket` for data larger
-than 2 GiB.
diff --git a/Misc/NEWS.d/next/Library/2019-04-18-16-10-29.bpo-28552.MW1TLt.rst b/Misc/NEWS.d/next/Library/2019-04-18-16-10-29.bpo-28552.MW1TLt.rst
deleted file mode 100644
index 2aa30c98c4524..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-04-18-16-10-29.bpo-28552.MW1TLt.rst
+++ /dev/null
@@ -1,4 +0,0 @@
-Fix :mod:`distutils.sysconfig` if :data:`sys.executable` is ``None`` or an
-empty string: use :func:`os.getcwd` to initialize ``project_base``.  Fix
-also the distutils build command: don't use :data:`sys.executable` if it is
-``None`` or an empty string.
diff --git a/Misc/NEWS.d/next/Library/2019-05-23-15-57-36.bpo-36713.sjPhnf.rst b/Misc/NEWS.d/next/Library/2019-05-23-15-57-36.bpo-36713.sjPhnf.rst
deleted file mode 100644
index cc52bff701e61..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-05-23-15-57-36.bpo-36713.sjPhnf.rst
+++ /dev/null
@@ -1 +0,0 @@
-Rename the :meth:`test_ascii_replace` to :meth:`test_ascii_strict`.
diff --git a/Misc/NEWS.d/next/Library/2019-05-30-16-16-47.bpo-12639.TQFOR4.rst b/Misc/NEWS.d/next/Library/2019-05-30-16-16-47.bpo-12639.TQFOR4.rst
deleted file mode 100644
index aade9121b4bbb..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-05-30-16-16-47.bpo-12639.TQFOR4.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-:meth:`msilib.Directory.start_component()` no longer fails if *keyfile* is
-not ``None``.
diff --git a/Misc/NEWS.d/next/Library/2019-06-10-12-02-45.bpo-36742.UEdHXJ.rst b/Misc/NEWS.d/next/Library/2019-06-10-12-02-45.bpo-36742.UEdHXJ.rst
deleted file mode 100644
index 3ba774056f15f..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-06-10-12-02-45.bpo-36742.UEdHXJ.rst
+++ /dev/null
@@ -1,3 +0,0 @@
-:func:`urlparse.urlsplit` error message for invalid ``netloc`` according to
-NFKC normalization is now a :class:`str` string, rather than a
-:class:`unicode` string, to prevent error when displaying the error.
diff --git a/Misc/NEWS.d/next/Library/2019-06-27-20-33-50.bpo-37437.du39_A.rst b/Misc/NEWS.d/next/Library/2019-06-27-20-33-50.bpo-37437.du39_A.rst
deleted file mode 100644
index 80719ee152d7a..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-06-27-20-33-50.bpo-37437.du39_A.rst
+++ /dev/null
@@ -1 +0,0 @@
-Update vendorized expat version to 2.2.7.
diff --git a/Misc/NEWS.d/next/Library/2019-07-24-18-27-44.bpo-37664.o-GYZC.rst b/Misc/NEWS.d/next/Library/2019-07-24-18-27-44.bpo-37664.o-GYZC.rst
deleted file mode 100644
index f12590540eb76..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-07-24-18-27-44.bpo-37664.o-GYZC.rst
+++ /dev/null
@@ -1 +0,0 @@
-Update wheels bundled with ensurepip (pip 19.2.3 and setuptools 41.2.0)
diff --git a/Misc/NEWS.d/next/Library/2019-08-23-14-47-09.bpo-34521.Y2BYu5.rst b/Misc/NEWS.d/next/Library/2019-08-23-14-47-09.bpo-34521.Y2BYu5.rst
deleted file mode 100644
index 06f42a841ea1c..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-08-23-14-47-09.bpo-34521.Y2BYu5.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Fix file descriptors transfer in multiprocessing on FreeBSD: use
-``CMSG_SPACE()`` rather than ``CMSG_LEN()``; see :rfc:`3542`.
diff --git a/Misc/NEWS.d/next/Library/2019-08-28-14-04-18.bpo-37965.7xGE-C.rst b/Misc/NEWS.d/next/Library/2019-08-28-14-04-18.bpo-37965.7xGE-C.rst
deleted file mode 100644
index 116a9e49dccc1..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-08-28-14-04-18.bpo-37965.7xGE-C.rst
+++ /dev/null
@@ -1 +0,0 @@
-Fix C compiler warning caused by distutils.ccompiler.CCompiler.has_function.
diff --git a/Misc/NEWS.d/next/Library/2019-08-31-01-52-59.bpo-34410.7KbWZQ.rst b/Misc/NEWS.d/next/Library/2019-08-31-01-52-59.bpo-34410.7KbWZQ.rst
deleted file mode 100644
index 64e778ee0913c..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-08-31-01-52-59.bpo-34410.7KbWZQ.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Fixed a crash in the :func:`tee` iterator when re-enter it. RuntimeError is
-now raised in this case.
diff --git a/Misc/NEWS.d/next/Library/2019-09-14-10-34-00.bpo-33936.8wCI_n.rst b/Misc/NEWS.d/next/Library/2019-09-14-10-34-00.bpo-33936.8wCI_n.rst
deleted file mode 100644
index 7bc7fed483a2e..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-09-14-10-34-00.bpo-33936.8wCI_n.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-_hashlib no longer calls obsolete OpenSSL initialization function with
-OpenSSL 1.1.0+.
diff --git a/Misc/NEWS.d/next/Library/2019-09-15-10-30-33.bpo-38175.61XlUv.rst b/Misc/NEWS.d/next/Library/2019-09-15-10-30-33.bpo-38175.61XlUv.rst
deleted file mode 100644
index 6d9f280bbd050..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-09-15-10-30-33.bpo-38175.61XlUv.rst
+++ /dev/null
@@ -1 +0,0 @@
-Fix a memory leak in comparison of :class:`sqlite3.Row` objects.
diff --git a/Misc/NEWS.d/next/Library/2019-09-27-15-24-45.bpo-38216.-7yvZR.rst b/Misc/NEWS.d/next/Library/2019-09-27-15-24-45.bpo-38216.-7yvZR.rst
deleted file mode 100644
index ac8e2b042d929..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-09-27-15-24-45.bpo-38216.-7yvZR.rst
+++ /dev/null
@@ -1,4 +0,0 @@
-Allow the rare code that wants to send invalid http requests from the
-`http.client` library a way to do so.  The fixes for bpo-30458 led to
-breakage for some projects that were relying on this ability to test their
-own behavior in the face of bad requests.
diff --git a/Misc/NEWS.d/next/Security/2019-02-13-17-21-10.bpo-35907.ckk2zg.rst b/Misc/NEWS.d/next/Security/2019-02-13-17-21-10.bpo-35907.ckk2zg.rst
deleted file mode 100644
index a42a386022fa1..0000000000000
--- a/Misc/NEWS.d/next/Security/2019-02-13-17-21-10.bpo-35907.ckk2zg.rst
+++ /dev/null
@@ -1,3 +0,0 @@
-CVE-2019-9948: Avoid file reading by disallowing ``local-file://`` and
-``local_file://`` URL schemes in :func:`urllib.urlopen`,
-:meth:`urllib.URLopener.open` and :meth:`urllib.URLopener.retrieve`.
diff --git a/Misc/NEWS.d/next/Security/2019-03-06-09-38-40.bpo-36216.6q1m4a.rst b/Misc/NEWS.d/next/Security/2019-03-06-09-38-40.bpo-36216.6q1m4a.rst
deleted file mode 100644
index 1e1ad92c6feb4..0000000000000
--- a/Misc/NEWS.d/next/Security/2019-03-06-09-38-40.bpo-36216.6q1m4a.rst
+++ /dev/null
@@ -1,3 +0,0 @@
-Changes urlsplit() to raise ValueError when the URL contains characters that
-decompose under IDNA encoding (NFKC-normalization) into characters that
-affect how the URL is parsed.
\ No newline at end of file
diff --git a/Misc/NEWS.d/next/Security/2019-04-10-08-53-30.bpo-30458.51E-DA.rst b/Misc/NEWS.d/next/Security/2019-04-10-08-53-30.bpo-30458.51E-DA.rst
deleted file mode 100644
index 47cb899df1af5..0000000000000
--- a/Misc/NEWS.d/next/Security/2019-04-10-08-53-30.bpo-30458.51E-DA.rst
+++ /dev/null
@@ -1 +0,0 @@
-Address CVE-2019-9740 by disallowing URL paths with embedded whitespace or control characters through into the underlying http client request.  Such potentially malicious header injection URLs now cause an httplib.InvalidURL exception to be raised.
diff --git a/Misc/NEWS.d/next/Security/2019-04-29-15-34-59.bpo-36742.QCUY0i.rst b/Misc/NEWS.d/next/Security/2019-04-29-15-34-59.bpo-36742.QCUY0i.rst
deleted file mode 100644
index d729ed2f3cd50..0000000000000
--- a/Misc/NEWS.d/next/Security/2019-04-29-15-34-59.bpo-36742.QCUY0i.rst
+++ /dev/null
@@ -1 +0,0 @@
-Fixes mishandling of pre-normalization characters in urlsplit().
diff --git a/Misc/NEWS.d/next/Security/2019-05-04-13-33-37.bpo-34155.MJll68.rst b/Misc/NEWS.d/next/Security/2019-05-04-13-33-37.bpo-34155.MJll68.rst
deleted file mode 100644
index 50292e29ed1d2..0000000000000
--- a/Misc/NEWS.d/next/Security/2019-05-04-13-33-37.bpo-34155.MJll68.rst
+++ /dev/null
@@ -1 +0,0 @@
-Fix parsing of invalid email addresses with more than one ``@`` (e.g. a at b@c.com.) to not return the part before 2nd ``@`` as valid email address. Patch by maxking & jpic.
diff --git a/Misc/NEWS.d/next/Security/2019-05-20-00-35-12.bpo-35121.RRi-HU.rst b/Misc/NEWS.d/next/Security/2019-05-20-00-35-12.bpo-35121.RRi-HU.rst
deleted file mode 100644
index 77251806163b3..0000000000000
--- a/Misc/NEWS.d/next/Security/2019-05-20-00-35-12.bpo-35121.RRi-HU.rst
+++ /dev/null
@@ -1,4 +0,0 @@
-Don't send cookies of domain A without Domain attribute to domain B when
-domain A is a suffix match of domain B while using a cookiejar with
-:class:`cookielib.DefaultCookiePolicy` policy. Patch by Karthikeyan
-Singaravelan.
diff --git a/Misc/NEWS.d/next/Security/2019-05-20-00-49-29.bpo-35647.oWmiGU.rst b/Misc/NEWS.d/next/Security/2019-05-20-00-49-29.bpo-35647.oWmiGU.rst
deleted file mode 100644
index 032e1e2c00bc5..0000000000000
--- a/Misc/NEWS.d/next/Security/2019-05-20-00-49-29.bpo-35647.oWmiGU.rst
+++ /dev/null
@@ -1,3 +0,0 @@
-Don't set cookie for a request when the request path is a prefix match of
-the cookie's path attribute but doesn't end with "/". Patch by Karthikeyan
-Singaravelan.
diff --git a/Misc/NEWS.d/next/Security/2019-06-17-08-43-19.bpo-34631.pJ8CGR.rst b/Misc/NEWS.d/next/Security/2019-06-17-08-43-19.bpo-34631.pJ8CGR.rst
deleted file mode 100644
index 41a972e9665aa..0000000000000
--- a/Misc/NEWS.d/next/Security/2019-06-17-08-43-19.bpo-34631.pJ8CGR.rst
+++ /dev/null
@@ -1 +0,0 @@
-Updated OpenSSL to 1.0.2s in Windows installer
diff --git a/Misc/NEWS.d/next/Security/2019-09-23-21-02-46.bpo-38174.MeWuJd.rst b/Misc/NEWS.d/next/Security/2019-09-23-21-02-46.bpo-38174.MeWuJd.rst
deleted file mode 100644
index e3da6fcdd139f..0000000000000
--- a/Misc/NEWS.d/next/Security/2019-09-23-21-02-46.bpo-38174.MeWuJd.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Update vendorized expat library version to 2.2.8, which resolves
-CVE-2019-15903.
diff --git a/Misc/NEWS.d/next/Security/2019-09-25-13-21-09.bpo-38243.1pfz24.rst b/Misc/NEWS.d/next/Security/2019-09-25-13-21-09.bpo-38243.1pfz24.rst
deleted file mode 100644
index 8f02baed9ebe5..0000000000000
--- a/Misc/NEWS.d/next/Security/2019-09-25-13-21-09.bpo-38243.1pfz24.rst
+++ /dev/null
@@ -1,3 +0,0 @@
-Escape the server title of :class:`DocXMLRPCServer.DocXMLRPCServer`
-when rendering the document page as HTML.
-(Contributed by Dong-hee Na in :issue:`38243`.)
diff --git a/Misc/NEWS.d/next/Tests/2019-01-11-14-01-19.bpo-34836.7fat9-.rst b/Misc/NEWS.d/next/Tests/2019-01-11-14-01-19.bpo-34836.7fat9-.rst
deleted file mode 100644
index aed4d3f75c93d..0000000000000
--- a/Misc/NEWS.d/next/Tests/2019-01-11-14-01-19.bpo-34836.7fat9-.rst
+++ /dev/null
@@ -1 +0,0 @@
-Fix ``test_default_ecdh_curve`` when TLSv1.3 is enabled by default.
diff --git a/Misc/NEWS.d/next/Tests/2019-02-24-02-44-52.bpo-26386.YZylPP.rst b/Misc/NEWS.d/next/Tests/2019-02-24-02-44-52.bpo-26386.YZylPP.rst
deleted file mode 100644
index b047fc8a7eba7..0000000000000
--- a/Misc/NEWS.d/next/Tests/2019-02-24-02-44-52.bpo-26386.YZylPP.rst
+++ /dev/null
@@ -1 +0,0 @@
-Re-enable missing widget testcases in test_ttk_guionly.
diff --git a/Misc/NEWS.d/next/Tests/2019-02-24-03-19-43.bpo-27313.Sj9veH.rst b/Misc/NEWS.d/next/Tests/2019-02-24-03-19-43.bpo-27313.Sj9veH.rst
deleted file mode 100644
index 189b9cf69f073..0000000000000
--- a/Misc/NEWS.d/next/Tests/2019-02-24-03-19-43.bpo-27313.Sj9veH.rst
+++ /dev/null
@@ -1 +0,0 @@
-Avoid test_ttk_guionly ComboboxTest failure with macOS Cocoa Tk.
diff --git a/Misc/NEWS.d/next/Tests/2019-03-05-13-27-36.bpo-36019.ebUjCm.rst b/Misc/NEWS.d/next/Tests/2019-03-05-13-27-36.bpo-36019.ebUjCm.rst
deleted file mode 100644
index c4abaf1fb4769..0000000000000
--- a/Misc/NEWS.d/next/Tests/2019-03-05-13-27-36.bpo-36019.ebUjCm.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Add test.support.TEST_HTTP_URL and replace references of
-http://www.example.com by this new constant. Contributed by St?phane Wirtel.
diff --git a/Misc/NEWS.d/next/Tests/2019-03-08-12-53-37.bpo-36234.NRVK6W.rst b/Misc/NEWS.d/next/Tests/2019-03-08-12-53-37.bpo-36234.NRVK6W.rst
deleted file mode 100644
index ef32dc7170dfc..0000000000000
--- a/Misc/NEWS.d/next/Tests/2019-03-08-12-53-37.bpo-36234.NRVK6W.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-test_posix.PosixUidGidTests: add tests for invalid uid/gid type (str).
-Patch written by David Malcolm.
diff --git a/Misc/NEWS.d/next/Tests/2019-04-09-14-08-02.bpo-36560._ejeOr.rst b/Misc/NEWS.d/next/Tests/2019-04-09-14-08-02.bpo-36560._ejeOr.rst
deleted file mode 100644
index a8de72c9ab13e..0000000000000
--- a/Misc/NEWS.d/next/Tests/2019-04-09-14-08-02.bpo-36560._ejeOr.rst
+++ /dev/null
@@ -1,3 +0,0 @@
-Fix reference leak hunting in regrtest: compute also deltas (of reference count
-and file descriptor count) during warmup, to ensure that everything is
-initialized before starting to hunt reference leaks.
diff --git a/Misc/NEWS.d/next/Tests/2019-05-06-18-29-54.bpo-35925.gwQPuC.rst b/Misc/NEWS.d/next/Tests/2019-05-06-18-29-54.bpo-35925.gwQPuC.rst
deleted file mode 100644
index 428326cdfe5bb..0000000000000
--- a/Misc/NEWS.d/next/Tests/2019-05-06-18-29-54.bpo-35925.gwQPuC.rst
+++ /dev/null
@@ -1 +0,0 @@
-Skip specific nntplib and ssl networking tests when they would otherwise fail due to a modern OS or distro with a default OpenSSL policy of rejecting connections to servers with weak certificates or disabling TLS below TLSv1.2.
diff --git a/Misc/NEWS.d/next/Tests/2019-05-08-15-55-46.bpo-36816.WBKRGZ.rst b/Misc/NEWS.d/next/Tests/2019-05-08-15-55-46.bpo-36816.WBKRGZ.rst
deleted file mode 100644
index 420dfe8323663..0000000000000
--- a/Misc/NEWS.d/next/Tests/2019-05-08-15-55-46.bpo-36816.WBKRGZ.rst
+++ /dev/null
@@ -1 +0,0 @@
-Update Lib/test/selfsigned_pythontestdotnet.pem to match self-signed.pythontest.net's new TLS certificate.
\ No newline at end of file
diff --git a/Misc/NEWS.d/next/Tests/2019-06-21-15-47-33.bpo-37362.D3xppx.rst b/Misc/NEWS.d/next/Tests/2019-06-21-15-47-33.bpo-37362.D3xppx.rst
deleted file mode 100644
index 43fdc1030c575..0000000000000
--- a/Misc/NEWS.d/next/Tests/2019-06-21-15-47-33.bpo-37362.D3xppx.rst
+++ /dev/null
@@ -1,3 +0,0 @@
-test_gdb no longer fails if it gets an "unexpected" message on stderr: it now
-ignores stderr. The purpose of test_gdb is to test that python-gdb.py commands
-work as expected, not to test gdb.
diff --git a/Misc/NEWS.d/next/Tests/2019-06-24-10-47-07.bpo-37359.CkdtyO.rst b/Misc/NEWS.d/next/Tests/2019-06-24-10-47-07.bpo-37359.CkdtyO.rst
deleted file mode 100644
index 3d5350de4f43f..0000000000000
--- a/Misc/NEWS.d/next/Tests/2019-06-24-10-47-07.bpo-37359.CkdtyO.rst
+++ /dev/null
@@ -1,4 +0,0 @@
-Add --cleanup option to python3 -m test to remove ``test_python_*``
-directories of previous failed jobs. Add "make cleantest" to run
-``python3 -m test --cleanup``.
-
diff --git a/Misc/NEWS.d/next/Tests/2019-06-26-15-28-45.bpo-37411.5lGNhM.rst b/Misc/NEWS.d/next/Tests/2019-06-26-15-28-45.bpo-37411.5lGNhM.rst
deleted file mode 100644
index 20e52d3c74f72..0000000000000
--- a/Misc/NEWS.d/next/Tests/2019-06-26-15-28-45.bpo-37411.5lGNhM.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Fix test_wsgiref.testEnviron() to no longer depend on the environment
-variables (don't fail if "X" variable is set).
diff --git a/Misc/NEWS.d/next/Tools-Demos/2019-04-30-14-30-29.bpo-14546.r38Y-6.rst b/Misc/NEWS.d/next/Tools-Demos/2019-04-30-14-30-29.bpo-14546.r38Y-6.rst
deleted file mode 100644
index b8659b886af81..0000000000000
--- a/Misc/NEWS.d/next/Tools-Demos/2019-04-30-14-30-29.bpo-14546.r38Y-6.rst
+++ /dev/null
@@ -1 +0,0 @@
-Fix the argument handling in Tools/scripts/lll.py.
diff --git a/Misc/NEWS.d/next/Tools-Demos/2019-07-24-16-20-54.bpo-37675.951Cvf.rst b/Misc/NEWS.d/next/Tools-Demos/2019-07-24-16-20-54.bpo-37675.951Cvf.rst
deleted file mode 100644
index e28fa207f918a..0000000000000
--- a/Misc/NEWS.d/next/Tools-Demos/2019-07-24-16-20-54.bpo-37675.951Cvf.rst
+++ /dev/null
@@ -1 +0,0 @@
-2to3 now works when run from a zipped standard library.
diff --git a/Misc/NEWS.d/next/Windows/2017-11-24-12-53-54.bpo-1104.1CWSZp.rst b/Misc/NEWS.d/next/Windows/2017-11-24-12-53-54.bpo-1104.1CWSZp.rst
deleted file mode 100644
index a4043496bc246..0000000000000
--- a/Misc/NEWS.d/next/Windows/2017-11-24-12-53-54.bpo-1104.1CWSZp.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Correctly handle string length in ``msilib.SummaryInfo.GetProperty()`` to
-prevent it from truncating the last character.
diff --git a/Misc/NEWS.d/next/Windows/2019-06-18-09-53-15.bpo-35360.tdqSmo.rst b/Misc/NEWS.d/next/Windows/2019-06-18-09-53-15.bpo-35360.tdqSmo.rst
deleted file mode 100644
index 8fea56a00fdc0..0000000000000
--- a/Misc/NEWS.d/next/Windows/2019-06-18-09-53-15.bpo-35360.tdqSmo.rst
+++ /dev/null
@@ -1 +0,0 @@
-Update Windows builds to use SQLite 3.28.0.
diff --git a/Misc/NEWS.d/next/Windows/2019-06-28-18-10-29.bpo-37445.LsdYO6.rst b/Misc/NEWS.d/next/Windows/2019-06-28-18-10-29.bpo-37445.LsdYO6.rst
deleted file mode 100644
index e4805b4e02fb1..0000000000000
--- a/Misc/NEWS.d/next/Windows/2019-06-28-18-10-29.bpo-37445.LsdYO6.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Include the ``FORMAT_MESSAGE_IGNORE_INSERTS`` flag in ``FormatMessageW()``
-calls.
diff --git a/Misc/NEWS.d/next/Windows/2019-09-16-12-47-12.bpo-38117.JaiB0X.rst b/Misc/NEWS.d/next/Windows/2019-09-16-12-47-12.bpo-38117.JaiB0X.rst
deleted file mode 100644
index 35e1cabb9b4a6..0000000000000
--- a/Misc/NEWS.d/next/Windows/2019-09-16-12-47-12.bpo-38117.JaiB0X.rst
+++ /dev/null
@@ -1 +0,0 @@
-Updates bundled OpenSSL to 1.0.2t
diff --git a/Misc/NEWS.d/next/macOS/2019-06-03-05-49-49.bpo-36231.RfmW_p.rst b/Misc/NEWS.d/next/macOS/2019-06-03-05-49-49.bpo-36231.RfmW_p.rst
deleted file mode 100644
index c82e54c12c89f..0000000000000
--- a/Misc/NEWS.d/next/macOS/2019-06-03-05-49-49.bpo-36231.RfmW_p.rst
+++ /dev/null
@@ -1,3 +0,0 @@
-Support building Python on macOS without /usr/include installed. As of macOS
-10.14, system header files are only available within an SDK provided by
-either the Command Line Tools or the Xcode app.
diff --git a/Misc/NEWS.d/next/macOS/2019-06-18-05-39-01.bpo-34631.StdZhE.rst b/Misc/NEWS.d/next/macOS/2019-06-18-05-39-01.bpo-34631.StdZhE.rst
deleted file mode 100644
index 2b62a469a6c0e..0000000000000
--- a/Misc/NEWS.d/next/macOS/2019-06-18-05-39-01.bpo-34631.StdZhE.rst
+++ /dev/null
@@ -1 +0,0 @@
-Updated OpenSSL to 1.0.2s in macOS installer.
diff --git a/Misc/NEWS.d/next/macOS/2019-06-18-10-02-11.bpo-35360.-CWbfy.rst b/Misc/NEWS.d/next/macOS/2019-06-18-10-02-11.bpo-35360.-CWbfy.rst
deleted file mode 100644
index 5e05ac56b0956..0000000000000
--- a/Misc/NEWS.d/next/macOS/2019-06-18-10-02-11.bpo-35360.-CWbfy.rst
+++ /dev/null
@@ -1 +0,0 @@
-Update macOS installer to use SQLite 3.28.0.
diff --git a/Misc/NEWS.d/next/macOS/2019-06-20-01-16-16.bpo-19960.LpLUPF.rst b/Misc/NEWS.d/next/macOS/2019-06-20-01-16-16.bpo-19960.LpLUPF.rst
deleted file mode 100644
index b15c6d61fa0ac..0000000000000
--- a/Misc/NEWS.d/next/macOS/2019-06-20-01-16-16.bpo-19960.LpLUPF.rst
+++ /dev/null
@@ -1,4 +0,0 @@
-When building 2.7 on macOS without system header files installed in
-``/usr/include``, a few extension modules dependent on system-supplied
-third-party libraries were not being built, most notably zlib.
-
diff --git a/Misc/NEWS.d/next/macOS/2019-09-15-21-45-15.bpo-38117._U9F2r.rst b/Misc/NEWS.d/next/macOS/2019-09-15-21-45-15.bpo-38117._U9F2r.rst
deleted file mode 100644
index dd9840da084df..0000000000000
--- a/Misc/NEWS.d/next/macOS/2019-09-15-21-45-15.bpo-38117._U9F2r.rst
+++ /dev/null
@@ -1 +0,0 @@
-Updated OpenSSL to 1.0.2t in macOS installer for 2.7.x.


From webhook-mailer at python.org  Mon Oct  7 22:03:42 2019
From: webhook-mailer at python.org (Benjamin Peterson)
Date: Tue, 08 Oct 2019 02:03:42 -0000
Subject: [Python-checkins] Bump version to 2.7.17rc1.
Message-ID: <mailman.32.1570500222.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/a6df224570eee0d30f720b621a441ecf2a1e5149
commit: a6df224570eee0d30f720b621a441ecf2a1e5149
branch: 2.7
author: Benjamin Peterson <benjamin at python.org>
committer: Benjamin Peterson <benjamin at python.org>
date: 2019-10-07T19:03:32-07:00
summary:

Bump version to 2.7.17rc1.

files:
M Include/patchlevel.h
M README

diff --git a/Include/patchlevel.h b/Include/patchlevel.h
index a123df8da053b..de01a84abbdbe 100644
--- a/Include/patchlevel.h
+++ b/Include/patchlevel.h
@@ -22,12 +22,12 @@
 /*--start constants--*/
 #define PY_MAJOR_VERSION	2
 #define PY_MINOR_VERSION	7
-#define PY_MICRO_VERSION	16
-#define PY_RELEASE_LEVEL	PY_RELEASE_LEVEL_FINAL
-#define PY_RELEASE_SERIAL	0
+#define PY_MICRO_VERSION	17
+#define PY_RELEASE_LEVEL	PY_RELEASE_LEVEL_GAMMA
+#define PY_RELEASE_SERIAL	1
 
 /* Version as a string */
-#define PY_VERSION      	"2.7.16+"
+#define PY_VERSION      	"2.7.17rc1"
 /*--end constants--*/
 
 /* Subversion Revision number of this file (not of the repository). Empty
diff --git a/README b/README
index 689a3e38dc476..9f03976cba5ec 100644
--- a/README
+++ b/README
@@ -1,4 +1,4 @@
-This is Python version 2.7.16
+This is Python version 2.7.17
 =============================
 
 Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011,


From webhook-mailer at python.org  Mon Oct  7 22:13:08 2019
From: webhook-mailer at python.org (Ned Deily)
Date: Tue, 08 Oct 2019 02:13:08 -0000
Subject: [Python-checkins] Update macOS installer displays for 2.7.17rc1
 (#16634)
Message-ID: <mailman.33.1570500788.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/1c7b14197b10924e2efc1e6c99c720958be1f681
commit: 1c7b14197b10924e2efc1e6c99c720958be1f681
branch: 2.7
author: Ned Deily <nad at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-07T22:13:04-04:00
summary:

Update macOS installer displays for 2.7.17rc1 (#16634)

files:
M Mac/BuildScript/resources/ReadMe.rtf
M Mac/BuildScript/resources/Welcome.rtf

diff --git a/Mac/BuildScript/resources/ReadMe.rtf b/Mac/BuildScript/resources/ReadMe.rtf
index 8133c35b2169d..b0c5d70396714 100644
--- a/Mac/BuildScript/resources/ReadMe.rtf
+++ b/Mac/BuildScript/resources/ReadMe.rtf
@@ -1,4 +1,4 @@
-{\rtf1\ansi\ansicpg1252\cocoartf1671\cocoasubrtf100
+{\rtf1\ansi\ansicpg1252\cocoartf1671\cocoasubrtf600
 {\fonttbl\f0\fswiss\fcharset0 Helvetica;\f1\fswiss\fcharset0 Helvetica-Bold;\f2\fswiss\fcharset0 Helvetica-Oblique;
 \f3\fmodern\fcharset0 CourierNewPSMT;}
 {\colortbl;\red255\green255\blue255;}
@@ -8,6 +8,13 @@
 
 \f0\fs24 \cf0 This package will install Python $FULL_VERSION for macOS $MACOSX_DEPLOYMENT_TARGET for the following architecture(s): $ARCHITECTURES.\
 \
+\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\partightenfactor0
+
+\f1\b \cf0 NOTE: 
+\f0\b0 This is the release candidate for Python 2.7.17.  Any updates needed to support the latest release of macOS, 
+\f1\b 10.15 Catalina,
+\f0\b0  will be provided after in 2.7.17 final.\
+\
 \pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural\partightenfactor0
 
 \f1\b \cf0 \ul \ulc0 Which installer variant should I use? [CHANGED in 2.7.15]
@@ -63,7 +70,7 @@ As of 2.7.15, the 10.9+ installer variant comes with its own private version of
 \f0\b0 The 10.6+ variant now also uses a private version of Tcl/Tk 8.6.\
 \pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural\partightenfactor0
 
-\f1\b \cf0 \ul \ulc0 \
+\f1\b \cf0 \ul \
 Binary installer support for Mac OS X 10.5 and earlier discontinued [CHANGED for Python 2.7.15]
 \f0\b0 \ulnone \
 \
diff --git a/Mac/BuildScript/resources/Welcome.rtf b/Mac/BuildScript/resources/Welcome.rtf
index 4810c334ebf1f..289f6a1198d99 100644
--- a/Mac/BuildScript/resources/Welcome.rtf
+++ b/Mac/BuildScript/resources/Welcome.rtf
@@ -1,4 +1,4 @@
-{\rtf1\ansi\ansicpg1252\cocoartf1671\cocoasubrtf100
+{\rtf1\ansi\ansicpg1252\cocoartf1671\cocoasubrtf600
 \cocoascreenfonts1{\fonttbl\f0\fswiss\fcharset0 Helvetica;\f1\fswiss\fcharset0 Helvetica-Bold;\f2\fmodern\fcharset0 CourierNewPSMT;
 }
 {\colortbl;\red255\green255\blue255;}
@@ -30,4 +30,10 @@
 
 \f1\b CHANGED in 2.7.16:
 \f0\b0   the 10.6+ variant now also uses a built-in Tcl/Tk 8.6\
+\
+
+\f1\b NOTE: 
+\f0\b0 This is the release candidate for Python 2.7.17.  Any updates needed to support the latest release of macOS, 
+\f1\b 10.15 Catalina,
+\f0\b0  will be provided after in 2.7.17 final.\
 }
\ No newline at end of file


From webhook-mailer at python.org  Mon Oct  7 23:37:49 2019
From: webhook-mailer at python.org (Benjamin Peterson)
Date: Tue, 08 Oct 2019 03:37:49 -0000
Subject: [Python-checkins] [2.7] Stop using deprecated logging API in Sphinx
 suspicious checker (GH-16635)
Message-ID: <mailman.34.1570505870.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/c9a195ec0b830d719ce5ebf8abe82c6e4a3247b8
commit: c9a195ec0b830d719ce5ebf8abe82c6e4a3247b8
branch: 2.7
author: Benjamin Peterson <benjamin at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-07T20:37:45-07:00
summary:

[2.7] Stop using deprecated logging API in Sphinx suspicious checker (GH-16635)

(cherry picked from commit ee171a26c1169abfae534b08acc0d95c6e45a22a)

Co-authored-by: Pablo Galindo <Pablogsal at gmail.com>

files:
M .travis.yml
M Doc/Makefile
M Doc/tools/extensions/suspicious.py

diff --git a/.travis.yml b/.travis.yml
index 80e9be7c1962b..cb9c059359380 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -36,13 +36,14 @@ matrix:
             - xvfb
     - os: linux
       language: python
-      python: 2.7
+      # Build the docs against a stable version of Python so code bugs don't hold up doc-related PRs.
+      python: 3.6
       env: TESTING=docs
       before_script:
         - cd Doc
         # Sphinx is pinned so that new versions that introduce new warnings won't suddenly cause build failures.
         # (Updating the version is fine as long as no warnings are raised by doing so.)
-        - python -m pip install sphinx~=1.6.1
+        - python3 -m pip install sphinx~=2.0.1
       script:
         - make check suspicious html SPHINXOPTS="-q -W -j4"
 
diff --git a/Doc/Makefile b/Doc/Makefile
index ebabc0206bfd4..7ca4c7a2dbf7c 100644
--- a/Doc/Makefile
+++ b/Doc/Makefile
@@ -148,7 +148,7 @@ dist:
 	cp -pPR build/epub/Python.epub dist/python-$(DISTVERSION)-docs.epub
 
 check:
-	$(PYTHON) tools/rstlint.py -i tools
+	$(PYTHON)2 tools/rstlint.py -i tools
 
 serve:
 	../Tools/scripts/serve.py build/html
diff --git a/Doc/tools/extensions/suspicious.py b/Doc/tools/extensions/suspicious.py
index 0a70e57d2b044..8d80f6759bffd 100644
--- a/Doc/tools/extensions/suspicious.py
+++ b/Doc/tools/extensions/suspicious.py
@@ -48,6 +48,7 @@
 
 from docutils import nodes
 from sphinx.builders import Builder
+import sphinx.util
 
 detect_all = re.compile(r'''
     ::(?=[^=])|            # two :: (but NOT ::=)
@@ -85,6 +86,7 @@ class CheckSuspiciousMarkupBuilder(Builder):
     Checks for possibly invalid markup that may leak into the output.
     """
     name = 'suspicious'
+    logger = sphinx.util.logging.getLogger("CheckSuspiciousMarkupBuilder")
 
     def init(self):
         # create output file
@@ -116,7 +118,7 @@ def finish(self):
             self.warn('Found %s/%s unused rules:' %
                       (len(unused_rules), len(self.rules)))
             for rule in unused_rules:
-                self.info(repr(rule))
+                self.logger.info(repr(rule))
         return
 
     def check_issue(self, line, lineno, issue):
@@ -146,7 +148,7 @@ def is_ignored(self, line, lineno, issue):
         return False
 
     def report_issue(self, text, lineno, issue):
-        if not self.any_issue: self.info()
+        if not self.any_issue: self.logger.info()
         self.any_issue = True
         self.write_log_entry(lineno, issue, text)
         if py3:
@@ -181,7 +183,7 @@ def load_rules(self, filename):
         A csv file, with exactly the same format as suspicious.csv
         Fields: document name (normalized), line number, issue, surrounding text
         """
-        self.info("loading ignore rules... ", nonl=1)
+        self.logger.info("loading ignore rules... ", nonl=1)
         self.rules = rules = []
         try:
             if py3:
@@ -206,7 +208,7 @@ def load_rules(self, filename):
             rule = Rule(docname, lineno, issue, text)
             rules.append(rule)
         f.close()
-        self.info('done, %d rules loaded' % len(self.rules))
+        self.logger.info('done, %d rules loaded' % len(self.rules))
 
 
 def get_lineno(node):


From webhook-mailer at python.org  Mon Oct  7 23:42:55 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Tue, 08 Oct 2019 03:42:55 -0000
Subject: [Python-checkins] bpo-31589 : Build PDF using xelatex for better
 UTF8 support. (GH-3940)
Message-ID: <mailman.35.1570506177.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/d9b321692b6770c0fe7cb7ba05f9bd5c6082a880
commit: d9b321692b6770c0fe7cb7ba05f9bd5c6082a880
branch: 2.7
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-07T20:42:51-07:00
summary:

bpo-31589 : Build PDF using xelatex for better UTF8 support. (GH-3940)


Also addresses doc build failures documented in bpo-32200.
(cherry picked from commit 7324b5ce8e7c031a0a3832a6a8d7c639111ae0ff)

Co-authored-by: Julien Palard <julien at palard.fr>

files:
M Doc/conf.py

diff --git a/Doc/conf.py b/Doc/conf.py
index d82514f605197..393018d08f51d 100644
--- a/Doc/conf.py
+++ b/Doc/conf.py
@@ -84,11 +84,10 @@
 # Options for LaTeX output
 # ------------------------
 
+latex_engine = 'xelatex'
+
 # Get LaTeX to handle Unicode correctly
 latex_elements = {
-    'inputenc': r'\usepackage[utf8x]{inputenc}',
-    'utf8extra': '',
-    'fontenc': r'\usepackage[T1,T2A]{fontenc}',
 }
 
 # Additional stuff for the LaTeX preamble.


From webhook-mailer at python.org  Mon Oct  7 23:43:57 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Tue, 08 Oct 2019 03:43:57 -0000
Subject: [Python-checkins] bpo-35036: Remove empty log line in the
 suspicious.py tool (GH-10024)
Message-ID: <mailman.36.1570506237.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/e78d79c15cc53bdf8161d41c627cbfb551d8ee96
commit: e78d79c15cc53bdf8161d41c627cbfb551d8ee96
branch: 2.7
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-07T20:43:53-07:00
summary:

bpo-35036: Remove empty log line in the suspicious.py tool (GH-10024)


Previous to commit ee171a2 the logline was working because of self.info() (now
deprecated) defaults to an empty message.
(cherry picked from commit c3f52a59ce8406d9e59253ad4621e4749abdaeef)

Co-authored-by: Xtreak <tirkarthi at users.noreply.github.com>

files:
M Doc/tools/extensions/suspicious.py

diff --git a/Doc/tools/extensions/suspicious.py b/Doc/tools/extensions/suspicious.py
index 8d80f6759bffd..494efabc46234 100644
--- a/Doc/tools/extensions/suspicious.py
+++ b/Doc/tools/extensions/suspicious.py
@@ -148,7 +148,6 @@ def is_ignored(self, line, lineno, issue):
         return False
 
     def report_issue(self, text, lineno, issue):
-        if not self.any_issue: self.logger.info()
         self.any_issue = True
         self.write_log_entry(lineno, issue, text)
         if py3:


From webhook-mailer at python.org  Mon Oct  7 23:57:09 2019
From: webhook-mailer at python.org (Benjamin Peterson)
Date: Tue, 08 Oct 2019 03:57:09 -0000
Subject: [Python-checkins] [2.7] bpo-31036: Allow sphinx and blurb to be
 found automatically (GH-16638)
Message-ID: <mailman.37.1570507031.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/0bd59d655280ad4c4aff2d327789815b89caee18
commit: 0bd59d655280ad4c4aff2d327789815b89caee18
branch: 2.7
author: Benjamin Peterson <benjamin at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-07T20:57:05-07:00
summary:

[2.7] bpo-31036: Allow sphinx and blurb to be found automatically (GH-16638)

Rather than requiring the path to blurb and/or sphinx-build to be specified to the make rule, enhance the Doc/Makefile to look for each first in a virtual environment created by make venv and, if not found, look on the normal process PATH. This allows the Doc/Makefile to take advantage of an installed spinx-build or blurb and, thus, do the right thing most of the time. Also, make the directory for the venv be configurable and document the `make venv` target..
(cherry picked from commit 590665c399fc4aa3c4a9f8e7104d43a02e9f3a0c)

Co-authored-by: Ned Deily <nad at python.org>

files:
M .gitignore
M Doc/Makefile

diff --git a/.gitignore b/.gitignore
index de25253df82ed..8f3a2d133b5e3 100644
--- a/.gitignore
+++ b/.gitignore
@@ -16,10 +16,10 @@
 *.profraw
 *.dyn
 Doc/build/
-Doc/tools/docutils/
-Doc/tools/jinja2/
-Doc/tools/pygments/
-Doc/tools/sphinx/
+Doc/venv/
+Doc/.venv/
+Doc/env/
+Doc/.env/
 Lib/lib2to3/*.pickle
 Lib/test/data/*
 Makefile
diff --git a/Doc/Makefile b/Doc/Makefile
index 7ca4c7a2dbf7c..3d7e4ac2bedce 100644
--- a/Doc/Makefile
+++ b/Doc/Makefile
@@ -5,7 +5,9 @@
 
 # You can set these variables from the command line.
 PYTHON       = python
-SPHINXBUILD  = sphinx-build
+VENVDIR      = ./venv
+SPHINXBUILD  = PATH=$(VENVDIR)/bin:$$PATH sphinx-build
+BLURB        = PATH=$(VENVDIR)/bin:$$PATH blurb
 PAPER        =
 SOURCES      =
 DISTVERSION  = $(shell $(PYTHON) tools/extensions/patchlevel.py)
@@ -102,7 +104,12 @@ htmlview: html
 	 $(PYTHON) -c "import webbrowser; webbrowser.open('build/html/index.html')"
 
 clean:
-	-rm -rf build/*
+	-rm -rf build/* $(VENVDIR)/*
+
+venv:
+	$(PYTHON) -m venv $(VENVDIR)
+	$(VENVDIR)/bin/python3 -m pip install -U Sphinx blurb
+	@echo "The venv has been created in the $(VENVDIR) directory"
 
 dist:
 	rm -rf dist
@@ -148,7 +155,7 @@ dist:
 	cp -pPR build/epub/Python.epub dist/python-$(DISTVERSION)-docs.epub
 
 check:
-	$(PYTHON)2 tools/rstlint.py -i tools
+	$(PYTHON)2 tools/rstlint.py -i tools -i $(VENVDIR)
 
 serve:
 	../Tools/scripts/serve.py build/html


From webhook-mailer at python.org  Tue Oct  8 00:22:22 2019
From: webhook-mailer at python.org (Benjamin Peterson)
Date: Tue, 08 Oct 2019 04:22:22 -0000
Subject: [Python-checkins] closes bpo-38402: Check error of primitive
 crypt/crypt_r. (GH-16599)
Message-ID: <mailman.38.1570508544.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/0d3fe8ae4961bf551e7d5e42559e2ede1a08fd7c
commit: 0d3fe8ae4961bf551e7d5e42559e2ede1a08fd7c
branch: master
author: Antonio Gutierrez <chibby0ne at gmail.com>
committer: Benjamin Peterson <benjamin at python.org>
date: 2019-10-07T21:22:17-07:00
summary:

closes bpo-38402: Check error of primitive crypt/crypt_r. (GH-16599)

Checks also for encryption algorithms methods not supported in different
OSs.

Signed-off-by: Antonio Gutierrez <chibby0ne at gmail.com>

files:
A Misc/NEWS.d/next/Core and Builtins/2019-10-05-19-36-16.bpo-38402.EZuzgK.rst
M Lib/crypt.py
M Modules/_cryptmodule.c

diff --git a/Lib/crypt.py b/Lib/crypt.py
index 8846602d7613e..33dbc46bb3e96 100644
--- a/Lib/crypt.py
+++ b/Lib/crypt.py
@@ -10,6 +10,7 @@
     else:
         raise ImportError("The required _crypt module was not built as part of CPython")
 
+import errno
 import string as _string
 from random import SystemRandom as _SystemRandom
 from collections import namedtuple as _namedtuple
@@ -88,7 +89,14 @@ def _add_method(name, *args, rounds=None):
     method = _Method(name, *args)
     globals()['METHOD_' + name] = method
     salt = mksalt(method, rounds=rounds)
-    result = crypt('', salt)
+    result = None
+    try:
+        result = crypt('', salt)
+    except OSError as e:
+        # Not all libc libraries support all encryption methods.
+        if e.errno == errno.EINVAL:
+            return False
+        raise
     if result and len(result) == method.total_size:
         methods.append(method)
         return True
diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-10-05-19-36-16.bpo-38402.EZuzgK.rst b/Misc/NEWS.d/next/Core and Builtins/2019-10-05-19-36-16.bpo-38402.EZuzgK.rst
new file mode 100644
index 0000000000000..8331500bf1a3e
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2019-10-05-19-36-16.bpo-38402.EZuzgK.rst	
@@ -0,0 +1 @@
+Check the error from the system's underlying ``crypt`` or ``crypt_r``.
diff --git a/Modules/_cryptmodule.c b/Modules/_cryptmodule.c
index 5d03f45f64361..00c1f4f69841b 100644
--- a/Modules/_cryptmodule.c
+++ b/Modules/_cryptmodule.c
@@ -42,6 +42,9 @@ crypt_crypt_impl(PyObject *module, const char *word, const char *salt)
 #else
     crypt_result = crypt(word, salt);
 #endif
+    if (crypt_result == NULL) {
+        return PyErr_SetFromErrno(PyExc_OSError);
+    }
     return Py_BuildValue("s", crypt_result);
 }
 


From webhook-mailer at python.org  Tue Oct  8 04:56:24 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Tue, 08 Oct 2019 08:56:24 -0000
Subject: [Python-checkins] Fix typo in _warnings.warn_explicit() docstring
 (GH-16625)
Message-ID: <mailman.39.1570524985.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/5dfbb4d50333e7a91fc0cd8c03a2f2f2cf56dbd9
commit: 5dfbb4d50333e7a91fc0cd8c03a2f2f2cf56dbd9
branch: master
author: Hansraj Das <raj.das.136 at gmail.com>
committer: Victor Stinner <vstinner at python.org>
date: 2019-10-08T10:56:07+02:00
summary:

Fix typo in _warnings.warn_explicit() docstring (GH-16625)

files:
M Python/_warnings.c

diff --git a/Python/_warnings.c b/Python/_warnings.c
index ecee399db6e8d..dd2b072d41685 100644
--- a/Python/_warnings.c
+++ b/Python/_warnings.c
@@ -1305,7 +1305,7 @@ _PyErr_WarnUnawaitedCoroutine(PyObject *coro)
 }
 
 PyDoc_STRVAR(warn_explicit_doc,
-"Low-level inferface to warnings functionality.");
+"Low-level interface to warnings functionality.");
 
 static PyMethodDef warnings_functions[] = {
     WARNINGS_WARN_METHODDEF


From webhook-mailer at python.org  Tue Oct  8 05:16:10 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Tue, 08 Oct 2019 09:16:10 -0000
Subject: [Python-checkins] Fix typo in _warnings.warn_explicit() docstring
 (GH-16625)
Message-ID: <mailman.40.1570526172.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/aa9d5b8ec33fd647ed9a0fe113e60ff826bea49a
commit: aa9d5b8ec33fd647ed9a0fe113e60ff826bea49a
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-08T02:16:06-07:00
summary:

Fix typo in _warnings.warn_explicit() docstring (GH-16625)

(cherry picked from commit 5dfbb4d50333e7a91fc0cd8c03a2f2f2cf56dbd9)

Co-authored-by: Hansraj Das <raj.das.136 at gmail.com>

files:
M Python/_warnings.c

diff --git a/Python/_warnings.c b/Python/_warnings.c
index 0b192580e1073..e02d28305126d 100644
--- a/Python/_warnings.c
+++ b/Python/_warnings.c
@@ -1305,7 +1305,7 @@ _PyErr_WarnUnawaitedCoroutine(PyObject *coro)
 }
 
 PyDoc_STRVAR(warn_explicit_doc,
-"Low-level inferface to warnings functionality.");
+"Low-level interface to warnings functionality.");
 
 static PyMethodDef warnings_functions[] = {
     WARNINGS_WARN_METHODDEF


From webhook-mailer at python.org  Tue Oct  8 05:26:05 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Tue, 08 Oct 2019 09:26:05 -0000
Subject: [Python-checkins] Fix typo in _warnings.warn_explicit() docstring
 (GH-16625)
Message-ID: <mailman.41.1570526765.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/13bc1523a7423e643d070b147596373578e2ed7e
commit: 13bc1523a7423e643d070b147596373578e2ed7e
branch: 3.7
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-08T02:25:56-07:00
summary:

Fix typo in _warnings.warn_explicit() docstring (GH-16625)

(cherry picked from commit 5dfbb4d50333e7a91fc0cd8c03a2f2f2cf56dbd9)

Co-authored-by: Hansraj Das <raj.das.136 at gmail.com>

files:
M Python/_warnings.c

diff --git a/Python/_warnings.c b/Python/_warnings.c
index 80dca65fd353f..e86a582a01cf4 100644
--- a/Python/_warnings.c
+++ b/Python/_warnings.c
@@ -1213,7 +1213,7 @@ _PyErr_WarnUnawaitedCoroutine(PyObject *coro)
 }
 
 PyDoc_STRVAR(warn_explicit_doc,
-"Low-level inferface to warnings functionality.");
+"Low-level interface to warnings functionality.");
 
 static PyMethodDef warnings_functions[] = {
     WARNINGS_WARN_METHODDEF


From webhook-mailer at python.org  Tue Oct  8 06:46:21 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Tue, 08 Oct 2019 10:46:21 -0000
Subject: [Python-checkins] bpo-33714: Output an exception raised in module's
 m_clear(). (GH-16592)
Message-ID: <mailman.42.1570531582.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/d7c387384a27f37e4e3fa7890c859212c56b45b2
commit: d7c387384a27f37e4e3fa7890c859212c56b45b2
branch: master
author: Serhiy Storchaka <storchaka at gmail.com>
committer: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
date: 2019-10-08T03:46:17-07:00
summary:

bpo-33714: Output an exception raised in module's m_clear(). (GH-16592)



It is similar to the more general code in the gc module, but
here we know the name of the module.



https://bugs.python.org/issue33714



Automerge-Triggered-By: @encukou

files:
M Objects/moduleobject.c

diff --git a/Objects/moduleobject.c b/Objects/moduleobject.c
index 92f97e6dd4850..03c7381311aa1 100644
--- a/Objects/moduleobject.c
+++ b/Objects/moduleobject.c
@@ -681,7 +681,7 @@ module_dealloc(PyModuleObject *m)
 
     PyObject_GC_UnTrack(m);
     if (verbose && m->md_name) {
-        PySys_FormatStderr("# destroy %S\n", m->md_name);
+        PySys_FormatStderr("# destroy %U\n", m->md_name);
     }
     if (m->md_weaklist != NULL)
         PyObject_ClearWeakRefs((PyObject *) m);
@@ -784,6 +784,12 @@ module_clear(PyModuleObject *m)
 {
     if (m->md_def && m->md_def->m_clear) {
         int res = m->md_def->m_clear((PyObject*)m);
+        if (PyErr_Occurred()) {
+            PySys_FormatStderr("Exception ignored in m_clear of module%s%V\n",
+                               m->md_name ? " " : "",
+                               m->md_name, "");
+            PyErr_WriteUnraisable(NULL);
+        }
         if (res)
             return res;
     }


From webhook-mailer at python.org  Tue Oct  8 07:31:45 2019
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Tue, 08 Oct 2019 11:31:45 -0000
Subject: [Python-checkins] bpo-38371: Tkinter: deprecate the split() method.
 (GH-16584)
Message-ID: <mailman.43.1570534306.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/d05b000c6bcd39dba4f4b2656e45954802649562
commit: d05b000c6bcd39dba4f4b2656e45954802649562
branch: master
author: Serhiy Storchaka <storchaka at gmail.com>
committer: GitHub <noreply at github.com>
date: 2019-10-08T14:31:35+03:00
summary:

bpo-38371: Tkinter: deprecate the split() method. (GH-16584)

files:
A Misc/NEWS.d/next/Library/2019-10-04-18-39-59.bpo-38371.S6Klvm.rst
M Doc/whatsnew/3.9.rst
M Lib/test/test_tcl.py
M Modules/_tkinter.c

diff --git a/Doc/whatsnew/3.9.rst b/Doc/whatsnew/3.9.rst
index 9f2a659a9dca7..6b4abc0567c30 100644
--- a/Doc/whatsnew/3.9.rst
+++ b/Doc/whatsnew/3.9.rst
@@ -191,6 +191,11 @@ Deprecated
   the module will restrict its seeds to :const:`None`, :class:`int`,
   :class:`float`, :class:`str`, :class:`bytes`, and :class:`bytearray`.
 
+* Deprecated the ``split()`` method of :class:`_tkinter.TkappType` in
+  favour of the ``splitlist()`` method which has more consistent and
+  predicable behavior.
+  (Contributed by Serhiy Storchaka in :issue:`38371`.)
+
 
 Removed
 =======
diff --git a/Lib/test/test_tcl.py b/Lib/test/test_tcl.py
index 3183ea850f73b..1c5b9cf2bd2a8 100644
--- a/Lib/test/test_tcl.py
+++ b/Lib/test/test_tcl.py
@@ -3,6 +3,7 @@
 import subprocess
 import sys
 import os
+import warnings
 from test import support
 
 # Skip this test if the _tkinter module wasn't built.
@@ -573,9 +574,12 @@ def test_splitlist(self):
     def test_split(self):
         split = self.interp.tk.split
         call = self.interp.tk.call
-        self.assertRaises(TypeError, split)
-        self.assertRaises(TypeError, split, 'a', 'b')
-        self.assertRaises(TypeError, split, 2)
+        with warnings.catch_warnings():
+            warnings.filterwarnings('ignore', r'\bsplit\b.*\bsplitlist\b',
+                                    DeprecationWarning)
+            self.assertRaises(TypeError, split)
+            self.assertRaises(TypeError, split, 'a', 'b')
+            self.assertRaises(TypeError, split, 2)
         testcases = [
             ('2', '2'),
             ('', ''),
@@ -617,7 +621,8 @@ def test_split(self):
                     expected),
             ]
         for arg, res in testcases:
-            self.assertEqual(split(arg), res, msg=arg)
+            with self.assertWarns(DeprecationWarning):
+                self.assertEqual(split(arg), res, msg=arg)
 
     def test_splitdict(self):
         splitdict = tkinter._splitdict
diff --git a/Misc/NEWS.d/next/Library/2019-10-04-18-39-59.bpo-38371.S6Klvm.rst b/Misc/NEWS.d/next/Library/2019-10-04-18-39-59.bpo-38371.S6Klvm.rst
new file mode 100644
index 0000000000000..583399531acd1
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-10-04-18-39-59.bpo-38371.S6Klvm.rst
@@ -0,0 +1,3 @@
+Deprecated the ``split()`` method in :class:`_tkinter.TkappType` in favour
+of the ``splitlist()`` method which has more consistent and predicable
+behavior.
diff --git a/Modules/_tkinter.c b/Modules/_tkinter.c
index c431d611d4d0c..235cb6bc28dd5 100644
--- a/Modules/_tkinter.c
+++ b/Modules/_tkinter.c
@@ -2304,6 +2304,12 @@ _tkinter_tkapp_split(TkappObject *self, PyObject *arg)
     PyObject *v;
     char *list;
 
+    if (PyErr_WarnEx(PyExc_DeprecationWarning,
+            "split() is deprecated; consider using splitlist() instead", 1))
+    {
+        return NULL;
+    }
+
     if (PyTclObject_Check(arg)) {
         Tcl_Obj *value = ((PyTclObject*)arg)->value;
         int objc;


From webhook-mailer at python.org  Tue Oct  8 07:32:29 2019
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Tue, 08 Oct 2019 11:32:29 -0000
Subject: [Python-checkins] bpo-36698: IDLE no longer fails when write
 non-encodable characters to stderr. (GH-16583)
Message-ID: <mailman.44.1570534350.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/b690a2759e62d9ee0b6ea1b20e8f7e4b2cdbf8bb
commit: b690a2759e62d9ee0b6ea1b20e8f7e4b2cdbf8bb
branch: master
author: Serhiy Storchaka <storchaka at gmail.com>
committer: GitHub <noreply at github.com>
date: 2019-10-08T14:32:25+03:00
summary:

bpo-36698: IDLE no longer fails when write non-encodable characters to stderr. (GH-16583)

It now escapes them with a backslash, as the regular Python interpreter.
Added the "errors" field to the standard streams.

files:
A Misc/NEWS.d/next/IDLE/2019-10-04-18-03-09.bpo-36698.BKcmom.rst
M Lib/idlelib/idle_test/test_run.py
M Lib/idlelib/iomenu.py
M Lib/idlelib/pyshell.py
M Lib/idlelib/run.py

diff --git a/Lib/idlelib/idle_test/test_run.py b/Lib/idlelib/idle_test/test_run.py
index cad0b4d98f8e0..9995dbe2eca50 100644
--- a/Lib/idlelib/idle_test/test_run.py
+++ b/Lib/idlelib/idle_test/test_run.py
@@ -36,7 +36,7 @@ def __eq__(self, other):
         self.assertIn('UnhashableException: ex1', tb[10])
 
 
-# PseudoFile tests.
+# StdioFile tests.
 
 class S(str):
     def __str__(self):
@@ -68,14 +68,14 @@ def push(self, lines):
         self.lines = list(lines)[::-1]
 
 
-class PseudeInputFilesTest(unittest.TestCase):
+class StdInputFilesTest(unittest.TestCase):
 
     def test_misc(self):
         shell = MockShell()
-        f = run.PseudoInputFile(shell, 'stdin', 'utf-8')
+        f = run.StdInputFile(shell, 'stdin')
         self.assertIsInstance(f, io.TextIOBase)
         self.assertEqual(f.encoding, 'utf-8')
-        self.assertIsNone(f.errors)
+        self.assertEqual(f.errors, 'strict')
         self.assertIsNone(f.newlines)
         self.assertEqual(f.name, '<stdin>')
         self.assertFalse(f.closed)
@@ -86,7 +86,7 @@ def test_misc(self):
 
     def test_unsupported(self):
         shell = MockShell()
-        f = run.PseudoInputFile(shell, 'stdin', 'utf-8')
+        f = run.StdInputFile(shell, 'stdin')
         self.assertRaises(OSError, f.fileno)
         self.assertRaises(OSError, f.tell)
         self.assertRaises(OSError, f.seek, 0)
@@ -95,7 +95,7 @@ def test_unsupported(self):
 
     def test_read(self):
         shell = MockShell()
-        f = run.PseudoInputFile(shell, 'stdin', 'utf-8')
+        f = run.StdInputFile(shell, 'stdin')
         shell.push(['one\n', 'two\n', ''])
         self.assertEqual(f.read(), 'one\ntwo\n')
         shell.push(['one\n', 'two\n', ''])
@@ -115,7 +115,7 @@ def test_read(self):
 
     def test_readline(self):
         shell = MockShell()
-        f = run.PseudoInputFile(shell, 'stdin', 'utf-8')
+        f = run.StdInputFile(shell, 'stdin')
         shell.push(['one\n', 'two\n', 'three\n', 'four\n'])
         self.assertEqual(f.readline(), 'one\n')
         self.assertEqual(f.readline(-1), 'two\n')
@@ -140,7 +140,7 @@ def test_readline(self):
 
     def test_readlines(self):
         shell = MockShell()
-        f = run.PseudoInputFile(shell, 'stdin', 'utf-8')
+        f = run.StdInputFile(shell, 'stdin')
         shell.push(['one\n', 'two\n', ''])
         self.assertEqual(f.readlines(), ['one\n', 'two\n'])
         shell.push(['one\n', 'two\n', ''])
@@ -161,7 +161,7 @@ def test_readlines(self):
 
     def test_close(self):
         shell = MockShell()
-        f = run.PseudoInputFile(shell, 'stdin', 'utf-8')
+        f = run.StdInputFile(shell, 'stdin')
         shell.push(['one\n', 'two\n', ''])
         self.assertFalse(f.closed)
         self.assertEqual(f.readline(), 'one\n')
@@ -171,14 +171,14 @@ def test_close(self):
         self.assertRaises(TypeError, f.close, 1)
 
 
-class PseudeOutputFilesTest(unittest.TestCase):
+class StdOutputFilesTest(unittest.TestCase):
 
     def test_misc(self):
         shell = MockShell()
-        f = run.PseudoOutputFile(shell, 'stdout', 'utf-8')
+        f = run.StdOutputFile(shell, 'stdout')
         self.assertIsInstance(f, io.TextIOBase)
         self.assertEqual(f.encoding, 'utf-8')
-        self.assertIsNone(f.errors)
+        self.assertEqual(f.errors, 'strict')
         self.assertIsNone(f.newlines)
         self.assertEqual(f.name, '<stdout>')
         self.assertFalse(f.closed)
@@ -189,7 +189,7 @@ def test_misc(self):
 
     def test_unsupported(self):
         shell = MockShell()
-        f = run.PseudoOutputFile(shell, 'stdout', 'utf-8')
+        f = run.StdOutputFile(shell, 'stdout')
         self.assertRaises(OSError, f.fileno)
         self.assertRaises(OSError, f.tell)
         self.assertRaises(OSError, f.seek, 0)
@@ -198,16 +198,36 @@ def test_unsupported(self):
 
     def test_write(self):
         shell = MockShell()
-        f = run.PseudoOutputFile(shell, 'stdout', 'utf-8')
+        f = run.StdOutputFile(shell, 'stdout')
         f.write('test')
         self.assertEqual(shell.written, [('test', 'stdout')])
         shell.reset()
-        f.write('t\xe8st')
-        self.assertEqual(shell.written, [('t\xe8st', 'stdout')])
+        f.write('t\xe8\u015b\U0001d599')
+        self.assertEqual(shell.written, [('t\xe8\u015b\U0001d599', 'stdout')])
         shell.reset()
 
-        f.write(S('t\xe8st'))
-        self.assertEqual(shell.written, [('t\xe8st', 'stdout')])
+        f.write(S('t\xe8\u015b\U0001d599'))
+        self.assertEqual(shell.written, [('t\xe8\u015b\U0001d599', 'stdout')])
+        self.assertEqual(type(shell.written[0][0]), str)
+        shell.reset()
+
+        self.assertRaises(TypeError, f.write)
+        self.assertEqual(shell.written, [])
+        self.assertRaises(TypeError, f.write, b'test')
+        self.assertRaises(TypeError, f.write, 123)
+        self.assertEqual(shell.written, [])
+        self.assertRaises(TypeError, f.write, 'test', 'spam')
+        self.assertEqual(shell.written, [])
+
+    def test_write_stderr_nonencodable(self):
+        shell = MockShell()
+        f = run.StdOutputFile(shell, 'stderr', 'iso-8859-15', 'backslashreplace')
+        f.write('t\xe8\u015b\U0001d599\xa4')
+        self.assertEqual(shell.written, [('t\xe8\\u015b\\U0001d599\\xa4', 'stderr')])
+        shell.reset()
+
+        f.write(S('t\xe8\u015b\U0001d599\xa4'))
+        self.assertEqual(shell.written, [('t\xe8\\u015b\\U0001d599\\xa4', 'stderr')])
         self.assertEqual(type(shell.written[0][0]), str)
         shell.reset()
 
@@ -221,7 +241,7 @@ def test_write(self):
 
     def test_writelines(self):
         shell = MockShell()
-        f = run.PseudoOutputFile(shell, 'stdout', 'utf-8')
+        f = run.StdOutputFile(shell, 'stdout')
         f.writelines([])
         self.assertEqual(shell.written, [])
         shell.reset()
@@ -251,7 +271,7 @@ def test_writelines(self):
 
     def test_close(self):
         shell = MockShell()
-        f = run.PseudoOutputFile(shell, 'stdout', 'utf-8')
+        f = run.StdOutputFile(shell, 'stdout')
         self.assertFalse(f.closed)
         f.write('test')
         f.close()
diff --git a/Lib/idlelib/iomenu.py b/Lib/idlelib/iomenu.py
index b9e813be0630e..b5533be79f996 100644
--- a/Lib/idlelib/iomenu.py
+++ b/Lib/idlelib/iomenu.py
@@ -15,6 +15,7 @@
 
 if idlelib.testing:  # Set True by test.test_idle to avoid setlocale.
     encoding = 'utf-8'
+    errors = 'surrogateescape'
 else:
     # Try setting the locale, so that we can find out
     # what encoding to use
@@ -24,15 +25,9 @@
     except (ImportError, locale.Error):
         pass
 
-    locale_decode = 'ascii'
     if sys.platform == 'win32':
-        # On Windows, we could use "mbcs". However, to give the user
-        # a portable encoding name, we need to find the code page
-        try:
-            locale_encoding = locale.getdefaultlocale()[1]
-            codecs.lookup(locale_encoding)
-        except LookupError:
-            pass
+        encoding = 'utf-8'
+        errors = 'surrogateescape'
     else:
         try:
             # Different things can fail here: the locale module may not be
@@ -40,30 +35,30 @@
             # resulting codeset may be unknown to Python. We ignore all
             # these problems, falling back to ASCII
             locale_encoding = locale.nl_langinfo(locale.CODESET)
-            if locale_encoding is None or locale_encoding == '':
-                # situation occurs on macOS
-                locale_encoding = 'ascii'
-            codecs.lookup(locale_encoding)
+            if locale_encoding:
+                codecs.lookup(locale_encoding)
         except (NameError, AttributeError, LookupError):
             # Try getdefaultlocale: it parses environment variables,
             # which may give a clue. Unfortunately, getdefaultlocale has
             # bugs that can cause ValueError.
             try:
                 locale_encoding = locale.getdefaultlocale()[1]
-                if locale_encoding is None or locale_encoding == '':
-                    # situation occurs on macOS
-                    locale_encoding = 'ascii'
-                codecs.lookup(locale_encoding)
+                if locale_encoding:
+                    codecs.lookup(locale_encoding)
             except (ValueError, LookupError):
                 pass
 
-    locale_encoding = locale_encoding.lower()
-
-    encoding = locale_encoding
-    # Encoding is used in multiple files; locale_encoding nowhere.
-    # The only use of 'encoding' below is in _decode as initial value
-    # of deprecated block asking user for encoding.
-    # Perhaps use elsewhere should be reviewed.
+        if locale_encoding:
+            encoding = locale_encoding.lower()
+            errors = 'strict'
+        else:
+            # POSIX locale or macOS
+            encoding = 'ascii'
+            errors = 'surrogateescape'
+        # Encoding is used in multiple files; locale_encoding nowhere.
+        # The only use of 'encoding' below is in _decode as initial value
+        # of deprecated block asking user for encoding.
+        # Perhaps use elsewhere should be reviewed.
 
 coding_re = re.compile(r'^[ \t\f]*#.*?coding[:=][ \t]*([-\w.]+)', re.ASCII)
 blank_re = re.compile(r'^[ \t\f]*(?:[#\r\n]|$)', re.ASCII)
diff --git a/Lib/idlelib/pyshell.py b/Lib/idlelib/pyshell.py
index bc87d24eaabf8..065122dec7a73 100755
--- a/Lib/idlelib/pyshell.py
+++ b/Lib/idlelib/pyshell.py
@@ -54,7 +54,7 @@
 from idlelib.filelist import FileList
 from idlelib.outwin import OutputWindow
 from idlelib import rpc
-from idlelib.run import idle_formatwarning, PseudoInputFile, PseudoOutputFile
+from idlelib.run import idle_formatwarning, StdInputFile, StdOutputFile
 from idlelib.undo import UndoDelegator
 
 HOST = '127.0.0.1' # python execution server on localhost loopback
@@ -902,10 +902,14 @@ def __init__(self, flist=None):
         self.save_stderr = sys.stderr
         self.save_stdin = sys.stdin
         from idlelib import iomenu
-        self.stdin = PseudoInputFile(self, "stdin", iomenu.encoding)
-        self.stdout = PseudoOutputFile(self, "stdout", iomenu.encoding)
-        self.stderr = PseudoOutputFile(self, "stderr", iomenu.encoding)
-        self.console = PseudoOutputFile(self, "console", iomenu.encoding)
+        self.stdin = StdInputFile(self, "stdin",
+                                  iomenu.encoding, iomenu.errors)
+        self.stdout = StdOutputFile(self, "stdout",
+                                    iomenu.encoding, iomenu.errors)
+        self.stderr = StdOutputFile(self, "stderr",
+                                    iomenu.encoding, "backslashreplace")
+        self.console = StdOutputFile(self, "console",
+                                     iomenu.encoding, iomenu.errors)
         if not use_subprocess:
             sys.stdout = self.stdout
             sys.stderr = self.stderr
diff --git a/Lib/idlelib/run.py b/Lib/idlelib/run.py
index 41e0ded440293..5bd84aadcd801 100644
--- a/Lib/idlelib/run.py
+++ b/Lib/idlelib/run.py
@@ -401,17 +401,22 @@ def handle_error(self, request, client_address):
 
 # Pseudofiles for shell-remote communication (also used in pyshell)
 
-class PseudoFile(io.TextIOBase):
+class StdioFile(io.TextIOBase):
 
-    def __init__(self, shell, tags, encoding=None):
+    def __init__(self, shell, tags, encoding='utf-8', errors='strict'):
         self.shell = shell
         self.tags = tags
         self._encoding = encoding
+        self._errors = errors
 
     @property
     def encoding(self):
         return self._encoding
 
+    @property
+    def errors(self):
+        return self._errors
+
     @property
     def name(self):
         return '<%s>' % self.tags
@@ -420,7 +425,7 @@ def isatty(self):
         return True
 
 
-class PseudoOutputFile(PseudoFile):
+class StdOutputFile(StdioFile):
 
     def writable(self):
         return True
@@ -428,19 +433,12 @@ def writable(self):
     def write(self, s):
         if self.closed:
             raise ValueError("write to closed file")
-        if type(s) is not str:
-            if not isinstance(s, str):
-                raise TypeError('must be str, not ' + type(s).__name__)
-            # See issue #19481
-            s = str.__str__(s)
+        s = str.encode(s, self.encoding, self.errors).decode(self.encoding, self.errors)
         return self.shell.write(s, self.tags)
 
 
-class PseudoInputFile(PseudoFile):
-
-    def __init__(self, shell, tags, encoding=None):
-        PseudoFile.__init__(self, shell, tags, encoding)
-        self._line_buffer = ''
+class StdInputFile(StdioFile):
+    _line_buffer = ''
 
     def readable(self):
         return True
@@ -495,12 +493,12 @@ def handle(self):
         executive = Executive(self)
         self.register("exec", executive)
         self.console = self.get_remote_proxy("console")
-        sys.stdin = PseudoInputFile(self.console, "stdin",
-                iomenu.encoding)
-        sys.stdout = PseudoOutputFile(self.console, "stdout",
-                iomenu.encoding)
-        sys.stderr = PseudoOutputFile(self.console, "stderr",
-                iomenu.encoding)
+        sys.stdin = StdInputFile(self.console, "stdin",
+                                 iomenu.encoding, iomenu.errors)
+        sys.stdout = StdOutputFile(self.console, "stdout",
+                                   iomenu.encoding, iomenu.errors)
+        sys.stderr = StdOutputFile(self.console, "stderr",
+                                   iomenu.encoding, "backslashreplace")
 
         sys.displayhook = rpc.displayhook
         # page help() text to shell.
diff --git a/Misc/NEWS.d/next/IDLE/2019-10-04-18-03-09.bpo-36698.BKcmom.rst b/Misc/NEWS.d/next/IDLE/2019-10-04-18-03-09.bpo-36698.BKcmom.rst
new file mode 100644
index 0000000000000..5aaa3c92646a1
--- /dev/null
+++ b/Misc/NEWS.d/next/IDLE/2019-10-04-18-03-09.bpo-36698.BKcmom.rst
@@ -0,0 +1,3 @@
+IDLE no longer fails when write non-encodable characters to stderr.  It now
+escapes them with a backslash, as the regular Python interpreter. Added the
+``errors`` field to the standard streams.


From webhook-mailer at python.org  Tue Oct  8 07:51:20 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Tue, 08 Oct 2019 11:51:20 -0000
Subject: [Python-checkins] bpo-36698: IDLE no longer fails when write
 non-encodable characters to stderr. (GH-16583)
Message-ID: <mailman.45.1570535481.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/a1f45008f190b0d340ec0eac623f05ef98e6b4c9
commit: a1f45008f190b0d340ec0eac623f05ef98e6b4c9
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-08T04:51:16-07:00
summary:

bpo-36698: IDLE no longer fails when write non-encodable characters to stderr. (GH-16583)


It now escapes them with a backslash, as the regular Python interpreter.
Added the "errors" field to the standard streams.
(cherry picked from commit b690a2759e62d9ee0b6ea1b20e8f7e4b2cdbf8bb)

Co-authored-by: Serhiy Storchaka <storchaka at gmail.com>

files:
A Misc/NEWS.d/next/IDLE/2019-10-04-18-03-09.bpo-36698.BKcmom.rst
M Lib/idlelib/idle_test/test_run.py
M Lib/idlelib/iomenu.py
M Lib/idlelib/pyshell.py
M Lib/idlelib/run.py

diff --git a/Lib/idlelib/idle_test/test_run.py b/Lib/idlelib/idle_test/test_run.py
index cad0b4d98f8e0..9995dbe2eca50 100644
--- a/Lib/idlelib/idle_test/test_run.py
+++ b/Lib/idlelib/idle_test/test_run.py
@@ -36,7 +36,7 @@ def __eq__(self, other):
         self.assertIn('UnhashableException: ex1', tb[10])
 
 
-# PseudoFile tests.
+# StdioFile tests.
 
 class S(str):
     def __str__(self):
@@ -68,14 +68,14 @@ def push(self, lines):
         self.lines = list(lines)[::-1]
 
 
-class PseudeInputFilesTest(unittest.TestCase):
+class StdInputFilesTest(unittest.TestCase):
 
     def test_misc(self):
         shell = MockShell()
-        f = run.PseudoInputFile(shell, 'stdin', 'utf-8')
+        f = run.StdInputFile(shell, 'stdin')
         self.assertIsInstance(f, io.TextIOBase)
         self.assertEqual(f.encoding, 'utf-8')
-        self.assertIsNone(f.errors)
+        self.assertEqual(f.errors, 'strict')
         self.assertIsNone(f.newlines)
         self.assertEqual(f.name, '<stdin>')
         self.assertFalse(f.closed)
@@ -86,7 +86,7 @@ def test_misc(self):
 
     def test_unsupported(self):
         shell = MockShell()
-        f = run.PseudoInputFile(shell, 'stdin', 'utf-8')
+        f = run.StdInputFile(shell, 'stdin')
         self.assertRaises(OSError, f.fileno)
         self.assertRaises(OSError, f.tell)
         self.assertRaises(OSError, f.seek, 0)
@@ -95,7 +95,7 @@ def test_unsupported(self):
 
     def test_read(self):
         shell = MockShell()
-        f = run.PseudoInputFile(shell, 'stdin', 'utf-8')
+        f = run.StdInputFile(shell, 'stdin')
         shell.push(['one\n', 'two\n', ''])
         self.assertEqual(f.read(), 'one\ntwo\n')
         shell.push(['one\n', 'two\n', ''])
@@ -115,7 +115,7 @@ def test_read(self):
 
     def test_readline(self):
         shell = MockShell()
-        f = run.PseudoInputFile(shell, 'stdin', 'utf-8')
+        f = run.StdInputFile(shell, 'stdin')
         shell.push(['one\n', 'two\n', 'three\n', 'four\n'])
         self.assertEqual(f.readline(), 'one\n')
         self.assertEqual(f.readline(-1), 'two\n')
@@ -140,7 +140,7 @@ def test_readline(self):
 
     def test_readlines(self):
         shell = MockShell()
-        f = run.PseudoInputFile(shell, 'stdin', 'utf-8')
+        f = run.StdInputFile(shell, 'stdin')
         shell.push(['one\n', 'two\n', ''])
         self.assertEqual(f.readlines(), ['one\n', 'two\n'])
         shell.push(['one\n', 'two\n', ''])
@@ -161,7 +161,7 @@ def test_readlines(self):
 
     def test_close(self):
         shell = MockShell()
-        f = run.PseudoInputFile(shell, 'stdin', 'utf-8')
+        f = run.StdInputFile(shell, 'stdin')
         shell.push(['one\n', 'two\n', ''])
         self.assertFalse(f.closed)
         self.assertEqual(f.readline(), 'one\n')
@@ -171,14 +171,14 @@ def test_close(self):
         self.assertRaises(TypeError, f.close, 1)
 
 
-class PseudeOutputFilesTest(unittest.TestCase):
+class StdOutputFilesTest(unittest.TestCase):
 
     def test_misc(self):
         shell = MockShell()
-        f = run.PseudoOutputFile(shell, 'stdout', 'utf-8')
+        f = run.StdOutputFile(shell, 'stdout')
         self.assertIsInstance(f, io.TextIOBase)
         self.assertEqual(f.encoding, 'utf-8')
-        self.assertIsNone(f.errors)
+        self.assertEqual(f.errors, 'strict')
         self.assertIsNone(f.newlines)
         self.assertEqual(f.name, '<stdout>')
         self.assertFalse(f.closed)
@@ -189,7 +189,7 @@ def test_misc(self):
 
     def test_unsupported(self):
         shell = MockShell()
-        f = run.PseudoOutputFile(shell, 'stdout', 'utf-8')
+        f = run.StdOutputFile(shell, 'stdout')
         self.assertRaises(OSError, f.fileno)
         self.assertRaises(OSError, f.tell)
         self.assertRaises(OSError, f.seek, 0)
@@ -198,16 +198,36 @@ def test_unsupported(self):
 
     def test_write(self):
         shell = MockShell()
-        f = run.PseudoOutputFile(shell, 'stdout', 'utf-8')
+        f = run.StdOutputFile(shell, 'stdout')
         f.write('test')
         self.assertEqual(shell.written, [('test', 'stdout')])
         shell.reset()
-        f.write('t\xe8st')
-        self.assertEqual(shell.written, [('t\xe8st', 'stdout')])
+        f.write('t\xe8\u015b\U0001d599')
+        self.assertEqual(shell.written, [('t\xe8\u015b\U0001d599', 'stdout')])
         shell.reset()
 
-        f.write(S('t\xe8st'))
-        self.assertEqual(shell.written, [('t\xe8st', 'stdout')])
+        f.write(S('t\xe8\u015b\U0001d599'))
+        self.assertEqual(shell.written, [('t\xe8\u015b\U0001d599', 'stdout')])
+        self.assertEqual(type(shell.written[0][0]), str)
+        shell.reset()
+
+        self.assertRaises(TypeError, f.write)
+        self.assertEqual(shell.written, [])
+        self.assertRaises(TypeError, f.write, b'test')
+        self.assertRaises(TypeError, f.write, 123)
+        self.assertEqual(shell.written, [])
+        self.assertRaises(TypeError, f.write, 'test', 'spam')
+        self.assertEqual(shell.written, [])
+
+    def test_write_stderr_nonencodable(self):
+        shell = MockShell()
+        f = run.StdOutputFile(shell, 'stderr', 'iso-8859-15', 'backslashreplace')
+        f.write('t\xe8\u015b\U0001d599\xa4')
+        self.assertEqual(shell.written, [('t\xe8\\u015b\\U0001d599\\xa4', 'stderr')])
+        shell.reset()
+
+        f.write(S('t\xe8\u015b\U0001d599\xa4'))
+        self.assertEqual(shell.written, [('t\xe8\\u015b\\U0001d599\\xa4', 'stderr')])
         self.assertEqual(type(shell.written[0][0]), str)
         shell.reset()
 
@@ -221,7 +241,7 @@ def test_write(self):
 
     def test_writelines(self):
         shell = MockShell()
-        f = run.PseudoOutputFile(shell, 'stdout', 'utf-8')
+        f = run.StdOutputFile(shell, 'stdout')
         f.writelines([])
         self.assertEqual(shell.written, [])
         shell.reset()
@@ -251,7 +271,7 @@ def test_writelines(self):
 
     def test_close(self):
         shell = MockShell()
-        f = run.PseudoOutputFile(shell, 'stdout', 'utf-8')
+        f = run.StdOutputFile(shell, 'stdout')
         self.assertFalse(f.closed)
         f.write('test')
         f.close()
diff --git a/Lib/idlelib/iomenu.py b/Lib/idlelib/iomenu.py
index b9e813be0630e..b5533be79f996 100644
--- a/Lib/idlelib/iomenu.py
+++ b/Lib/idlelib/iomenu.py
@@ -15,6 +15,7 @@
 
 if idlelib.testing:  # Set True by test.test_idle to avoid setlocale.
     encoding = 'utf-8'
+    errors = 'surrogateescape'
 else:
     # Try setting the locale, so that we can find out
     # what encoding to use
@@ -24,15 +25,9 @@
     except (ImportError, locale.Error):
         pass
 
-    locale_decode = 'ascii'
     if sys.platform == 'win32':
-        # On Windows, we could use "mbcs". However, to give the user
-        # a portable encoding name, we need to find the code page
-        try:
-            locale_encoding = locale.getdefaultlocale()[1]
-            codecs.lookup(locale_encoding)
-        except LookupError:
-            pass
+        encoding = 'utf-8'
+        errors = 'surrogateescape'
     else:
         try:
             # Different things can fail here: the locale module may not be
@@ -40,30 +35,30 @@
             # resulting codeset may be unknown to Python. We ignore all
             # these problems, falling back to ASCII
             locale_encoding = locale.nl_langinfo(locale.CODESET)
-            if locale_encoding is None or locale_encoding == '':
-                # situation occurs on macOS
-                locale_encoding = 'ascii'
-            codecs.lookup(locale_encoding)
+            if locale_encoding:
+                codecs.lookup(locale_encoding)
         except (NameError, AttributeError, LookupError):
             # Try getdefaultlocale: it parses environment variables,
             # which may give a clue. Unfortunately, getdefaultlocale has
             # bugs that can cause ValueError.
             try:
                 locale_encoding = locale.getdefaultlocale()[1]
-                if locale_encoding is None or locale_encoding == '':
-                    # situation occurs on macOS
-                    locale_encoding = 'ascii'
-                codecs.lookup(locale_encoding)
+                if locale_encoding:
+                    codecs.lookup(locale_encoding)
             except (ValueError, LookupError):
                 pass
 
-    locale_encoding = locale_encoding.lower()
-
-    encoding = locale_encoding
-    # Encoding is used in multiple files; locale_encoding nowhere.
-    # The only use of 'encoding' below is in _decode as initial value
-    # of deprecated block asking user for encoding.
-    # Perhaps use elsewhere should be reviewed.
+        if locale_encoding:
+            encoding = locale_encoding.lower()
+            errors = 'strict'
+        else:
+            # POSIX locale or macOS
+            encoding = 'ascii'
+            errors = 'surrogateescape'
+        # Encoding is used in multiple files; locale_encoding nowhere.
+        # The only use of 'encoding' below is in _decode as initial value
+        # of deprecated block asking user for encoding.
+        # Perhaps use elsewhere should be reviewed.
 
 coding_re = re.compile(r'^[ \t\f]*#.*?coding[:=][ \t]*([-\w.]+)', re.ASCII)
 blank_re = re.compile(r'^[ \t\f]*(?:[#\r\n]|$)', re.ASCII)
diff --git a/Lib/idlelib/pyshell.py b/Lib/idlelib/pyshell.py
index bc87d24eaabf8..065122dec7a73 100755
--- a/Lib/idlelib/pyshell.py
+++ b/Lib/idlelib/pyshell.py
@@ -54,7 +54,7 @@
 from idlelib.filelist import FileList
 from idlelib.outwin import OutputWindow
 from idlelib import rpc
-from idlelib.run import idle_formatwarning, PseudoInputFile, PseudoOutputFile
+from idlelib.run import idle_formatwarning, StdInputFile, StdOutputFile
 from idlelib.undo import UndoDelegator
 
 HOST = '127.0.0.1' # python execution server on localhost loopback
@@ -902,10 +902,14 @@ def __init__(self, flist=None):
         self.save_stderr = sys.stderr
         self.save_stdin = sys.stdin
         from idlelib import iomenu
-        self.stdin = PseudoInputFile(self, "stdin", iomenu.encoding)
-        self.stdout = PseudoOutputFile(self, "stdout", iomenu.encoding)
-        self.stderr = PseudoOutputFile(self, "stderr", iomenu.encoding)
-        self.console = PseudoOutputFile(self, "console", iomenu.encoding)
+        self.stdin = StdInputFile(self, "stdin",
+                                  iomenu.encoding, iomenu.errors)
+        self.stdout = StdOutputFile(self, "stdout",
+                                    iomenu.encoding, iomenu.errors)
+        self.stderr = StdOutputFile(self, "stderr",
+                                    iomenu.encoding, "backslashreplace")
+        self.console = StdOutputFile(self, "console",
+                                     iomenu.encoding, iomenu.errors)
         if not use_subprocess:
             sys.stdout = self.stdout
             sys.stderr = self.stderr
diff --git a/Lib/idlelib/run.py b/Lib/idlelib/run.py
index 41e0ded440293..5bd84aadcd801 100644
--- a/Lib/idlelib/run.py
+++ b/Lib/idlelib/run.py
@@ -401,17 +401,22 @@ def handle_error(self, request, client_address):
 
 # Pseudofiles for shell-remote communication (also used in pyshell)
 
-class PseudoFile(io.TextIOBase):
+class StdioFile(io.TextIOBase):
 
-    def __init__(self, shell, tags, encoding=None):
+    def __init__(self, shell, tags, encoding='utf-8', errors='strict'):
         self.shell = shell
         self.tags = tags
         self._encoding = encoding
+        self._errors = errors
 
     @property
     def encoding(self):
         return self._encoding
 
+    @property
+    def errors(self):
+        return self._errors
+
     @property
     def name(self):
         return '<%s>' % self.tags
@@ -420,7 +425,7 @@ def isatty(self):
         return True
 
 
-class PseudoOutputFile(PseudoFile):
+class StdOutputFile(StdioFile):
 
     def writable(self):
         return True
@@ -428,19 +433,12 @@ def writable(self):
     def write(self, s):
         if self.closed:
             raise ValueError("write to closed file")
-        if type(s) is not str:
-            if not isinstance(s, str):
-                raise TypeError('must be str, not ' + type(s).__name__)
-            # See issue #19481
-            s = str.__str__(s)
+        s = str.encode(s, self.encoding, self.errors).decode(self.encoding, self.errors)
         return self.shell.write(s, self.tags)
 
 
-class PseudoInputFile(PseudoFile):
-
-    def __init__(self, shell, tags, encoding=None):
-        PseudoFile.__init__(self, shell, tags, encoding)
-        self._line_buffer = ''
+class StdInputFile(StdioFile):
+    _line_buffer = ''
 
     def readable(self):
         return True
@@ -495,12 +493,12 @@ def handle(self):
         executive = Executive(self)
         self.register("exec", executive)
         self.console = self.get_remote_proxy("console")
-        sys.stdin = PseudoInputFile(self.console, "stdin",
-                iomenu.encoding)
-        sys.stdout = PseudoOutputFile(self.console, "stdout",
-                iomenu.encoding)
-        sys.stderr = PseudoOutputFile(self.console, "stderr",
-                iomenu.encoding)
+        sys.stdin = StdInputFile(self.console, "stdin",
+                                 iomenu.encoding, iomenu.errors)
+        sys.stdout = StdOutputFile(self.console, "stdout",
+                                   iomenu.encoding, iomenu.errors)
+        sys.stderr = StdOutputFile(self.console, "stderr",
+                                   iomenu.encoding, "backslashreplace")
 
         sys.displayhook = rpc.displayhook
         # page help() text to shell.
diff --git a/Misc/NEWS.d/next/IDLE/2019-10-04-18-03-09.bpo-36698.BKcmom.rst b/Misc/NEWS.d/next/IDLE/2019-10-04-18-03-09.bpo-36698.BKcmom.rst
new file mode 100644
index 0000000000000..5aaa3c92646a1
--- /dev/null
+++ b/Misc/NEWS.d/next/IDLE/2019-10-04-18-03-09.bpo-36698.BKcmom.rst
@@ -0,0 +1,3 @@
+IDLE no longer fails when write non-encodable characters to stderr.  It now
+escapes them with a backslash, as the regular Python interpreter. Added the
+``errors`` field to the standard streams.


From webhook-mailer at python.org  Tue Oct  8 07:52:52 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Tue, 08 Oct 2019 11:52:52 -0000
Subject: [Python-checkins] bpo-36698: IDLE no longer fails when write
 non-encodable characters to stderr. (GH-16583)
Message-ID: <mailman.46.1570535573.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/4f962ecfa1aadc45facc250841208f6dd2ce690f
commit: 4f962ecfa1aadc45facc250841208f6dd2ce690f
branch: 3.7
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-08T04:52:48-07:00
summary:

bpo-36698: IDLE no longer fails when write non-encodable characters to stderr. (GH-16583)


It now escapes them with a backslash, as the regular Python interpreter.
Added the "errors" field to the standard streams.
(cherry picked from commit b690a2759e62d9ee0b6ea1b20e8f7e4b2cdbf8bb)

Co-authored-by: Serhiy Storchaka <storchaka at gmail.com>

files:
A Misc/NEWS.d/next/IDLE/2019-10-04-18-03-09.bpo-36698.BKcmom.rst
M Lib/idlelib/idle_test/test_run.py
M Lib/idlelib/iomenu.py
M Lib/idlelib/pyshell.py
M Lib/idlelib/run.py

diff --git a/Lib/idlelib/idle_test/test_run.py b/Lib/idlelib/idle_test/test_run.py
index cad0b4d98f8e0..9995dbe2eca50 100644
--- a/Lib/idlelib/idle_test/test_run.py
+++ b/Lib/idlelib/idle_test/test_run.py
@@ -36,7 +36,7 @@ def __eq__(self, other):
         self.assertIn('UnhashableException: ex1', tb[10])
 
 
-# PseudoFile tests.
+# StdioFile tests.
 
 class S(str):
     def __str__(self):
@@ -68,14 +68,14 @@ def push(self, lines):
         self.lines = list(lines)[::-1]
 
 
-class PseudeInputFilesTest(unittest.TestCase):
+class StdInputFilesTest(unittest.TestCase):
 
     def test_misc(self):
         shell = MockShell()
-        f = run.PseudoInputFile(shell, 'stdin', 'utf-8')
+        f = run.StdInputFile(shell, 'stdin')
         self.assertIsInstance(f, io.TextIOBase)
         self.assertEqual(f.encoding, 'utf-8')
-        self.assertIsNone(f.errors)
+        self.assertEqual(f.errors, 'strict')
         self.assertIsNone(f.newlines)
         self.assertEqual(f.name, '<stdin>')
         self.assertFalse(f.closed)
@@ -86,7 +86,7 @@ def test_misc(self):
 
     def test_unsupported(self):
         shell = MockShell()
-        f = run.PseudoInputFile(shell, 'stdin', 'utf-8')
+        f = run.StdInputFile(shell, 'stdin')
         self.assertRaises(OSError, f.fileno)
         self.assertRaises(OSError, f.tell)
         self.assertRaises(OSError, f.seek, 0)
@@ -95,7 +95,7 @@ def test_unsupported(self):
 
     def test_read(self):
         shell = MockShell()
-        f = run.PseudoInputFile(shell, 'stdin', 'utf-8')
+        f = run.StdInputFile(shell, 'stdin')
         shell.push(['one\n', 'two\n', ''])
         self.assertEqual(f.read(), 'one\ntwo\n')
         shell.push(['one\n', 'two\n', ''])
@@ -115,7 +115,7 @@ def test_read(self):
 
     def test_readline(self):
         shell = MockShell()
-        f = run.PseudoInputFile(shell, 'stdin', 'utf-8')
+        f = run.StdInputFile(shell, 'stdin')
         shell.push(['one\n', 'two\n', 'three\n', 'four\n'])
         self.assertEqual(f.readline(), 'one\n')
         self.assertEqual(f.readline(-1), 'two\n')
@@ -140,7 +140,7 @@ def test_readline(self):
 
     def test_readlines(self):
         shell = MockShell()
-        f = run.PseudoInputFile(shell, 'stdin', 'utf-8')
+        f = run.StdInputFile(shell, 'stdin')
         shell.push(['one\n', 'two\n', ''])
         self.assertEqual(f.readlines(), ['one\n', 'two\n'])
         shell.push(['one\n', 'two\n', ''])
@@ -161,7 +161,7 @@ def test_readlines(self):
 
     def test_close(self):
         shell = MockShell()
-        f = run.PseudoInputFile(shell, 'stdin', 'utf-8')
+        f = run.StdInputFile(shell, 'stdin')
         shell.push(['one\n', 'two\n', ''])
         self.assertFalse(f.closed)
         self.assertEqual(f.readline(), 'one\n')
@@ -171,14 +171,14 @@ def test_close(self):
         self.assertRaises(TypeError, f.close, 1)
 
 
-class PseudeOutputFilesTest(unittest.TestCase):
+class StdOutputFilesTest(unittest.TestCase):
 
     def test_misc(self):
         shell = MockShell()
-        f = run.PseudoOutputFile(shell, 'stdout', 'utf-8')
+        f = run.StdOutputFile(shell, 'stdout')
         self.assertIsInstance(f, io.TextIOBase)
         self.assertEqual(f.encoding, 'utf-8')
-        self.assertIsNone(f.errors)
+        self.assertEqual(f.errors, 'strict')
         self.assertIsNone(f.newlines)
         self.assertEqual(f.name, '<stdout>')
         self.assertFalse(f.closed)
@@ -189,7 +189,7 @@ def test_misc(self):
 
     def test_unsupported(self):
         shell = MockShell()
-        f = run.PseudoOutputFile(shell, 'stdout', 'utf-8')
+        f = run.StdOutputFile(shell, 'stdout')
         self.assertRaises(OSError, f.fileno)
         self.assertRaises(OSError, f.tell)
         self.assertRaises(OSError, f.seek, 0)
@@ -198,16 +198,36 @@ def test_unsupported(self):
 
     def test_write(self):
         shell = MockShell()
-        f = run.PseudoOutputFile(shell, 'stdout', 'utf-8')
+        f = run.StdOutputFile(shell, 'stdout')
         f.write('test')
         self.assertEqual(shell.written, [('test', 'stdout')])
         shell.reset()
-        f.write('t\xe8st')
-        self.assertEqual(shell.written, [('t\xe8st', 'stdout')])
+        f.write('t\xe8\u015b\U0001d599')
+        self.assertEqual(shell.written, [('t\xe8\u015b\U0001d599', 'stdout')])
         shell.reset()
 
-        f.write(S('t\xe8st'))
-        self.assertEqual(shell.written, [('t\xe8st', 'stdout')])
+        f.write(S('t\xe8\u015b\U0001d599'))
+        self.assertEqual(shell.written, [('t\xe8\u015b\U0001d599', 'stdout')])
+        self.assertEqual(type(shell.written[0][0]), str)
+        shell.reset()
+
+        self.assertRaises(TypeError, f.write)
+        self.assertEqual(shell.written, [])
+        self.assertRaises(TypeError, f.write, b'test')
+        self.assertRaises(TypeError, f.write, 123)
+        self.assertEqual(shell.written, [])
+        self.assertRaises(TypeError, f.write, 'test', 'spam')
+        self.assertEqual(shell.written, [])
+
+    def test_write_stderr_nonencodable(self):
+        shell = MockShell()
+        f = run.StdOutputFile(shell, 'stderr', 'iso-8859-15', 'backslashreplace')
+        f.write('t\xe8\u015b\U0001d599\xa4')
+        self.assertEqual(shell.written, [('t\xe8\\u015b\\U0001d599\\xa4', 'stderr')])
+        shell.reset()
+
+        f.write(S('t\xe8\u015b\U0001d599\xa4'))
+        self.assertEqual(shell.written, [('t\xe8\\u015b\\U0001d599\\xa4', 'stderr')])
         self.assertEqual(type(shell.written[0][0]), str)
         shell.reset()
 
@@ -221,7 +241,7 @@ def test_write(self):
 
     def test_writelines(self):
         shell = MockShell()
-        f = run.PseudoOutputFile(shell, 'stdout', 'utf-8')
+        f = run.StdOutputFile(shell, 'stdout')
         f.writelines([])
         self.assertEqual(shell.written, [])
         shell.reset()
@@ -251,7 +271,7 @@ def test_writelines(self):
 
     def test_close(self):
         shell = MockShell()
-        f = run.PseudoOutputFile(shell, 'stdout', 'utf-8')
+        f = run.StdOutputFile(shell, 'stdout')
         self.assertFalse(f.closed)
         f.write('test')
         f.close()
diff --git a/Lib/idlelib/iomenu.py b/Lib/idlelib/iomenu.py
index b9e813be0630e..b5533be79f996 100644
--- a/Lib/idlelib/iomenu.py
+++ b/Lib/idlelib/iomenu.py
@@ -15,6 +15,7 @@
 
 if idlelib.testing:  # Set True by test.test_idle to avoid setlocale.
     encoding = 'utf-8'
+    errors = 'surrogateescape'
 else:
     # Try setting the locale, so that we can find out
     # what encoding to use
@@ -24,15 +25,9 @@
     except (ImportError, locale.Error):
         pass
 
-    locale_decode = 'ascii'
     if sys.platform == 'win32':
-        # On Windows, we could use "mbcs". However, to give the user
-        # a portable encoding name, we need to find the code page
-        try:
-            locale_encoding = locale.getdefaultlocale()[1]
-            codecs.lookup(locale_encoding)
-        except LookupError:
-            pass
+        encoding = 'utf-8'
+        errors = 'surrogateescape'
     else:
         try:
             # Different things can fail here: the locale module may not be
@@ -40,30 +35,30 @@
             # resulting codeset may be unknown to Python. We ignore all
             # these problems, falling back to ASCII
             locale_encoding = locale.nl_langinfo(locale.CODESET)
-            if locale_encoding is None or locale_encoding == '':
-                # situation occurs on macOS
-                locale_encoding = 'ascii'
-            codecs.lookup(locale_encoding)
+            if locale_encoding:
+                codecs.lookup(locale_encoding)
         except (NameError, AttributeError, LookupError):
             # Try getdefaultlocale: it parses environment variables,
             # which may give a clue. Unfortunately, getdefaultlocale has
             # bugs that can cause ValueError.
             try:
                 locale_encoding = locale.getdefaultlocale()[1]
-                if locale_encoding is None or locale_encoding == '':
-                    # situation occurs on macOS
-                    locale_encoding = 'ascii'
-                codecs.lookup(locale_encoding)
+                if locale_encoding:
+                    codecs.lookup(locale_encoding)
             except (ValueError, LookupError):
                 pass
 
-    locale_encoding = locale_encoding.lower()
-
-    encoding = locale_encoding
-    # Encoding is used in multiple files; locale_encoding nowhere.
-    # The only use of 'encoding' below is in _decode as initial value
-    # of deprecated block asking user for encoding.
-    # Perhaps use elsewhere should be reviewed.
+        if locale_encoding:
+            encoding = locale_encoding.lower()
+            errors = 'strict'
+        else:
+            # POSIX locale or macOS
+            encoding = 'ascii'
+            errors = 'surrogateescape'
+        # Encoding is used in multiple files; locale_encoding nowhere.
+        # The only use of 'encoding' below is in _decode as initial value
+        # of deprecated block asking user for encoding.
+        # Perhaps use elsewhere should be reviewed.
 
 coding_re = re.compile(r'^[ \t\f]*#.*?coding[:=][ \t]*([-\w.]+)', re.ASCII)
 blank_re = re.compile(r'^[ \t\f]*(?:[#\r\n]|$)', re.ASCII)
diff --git a/Lib/idlelib/pyshell.py b/Lib/idlelib/pyshell.py
index bc87d24eaabf8..065122dec7a73 100755
--- a/Lib/idlelib/pyshell.py
+++ b/Lib/idlelib/pyshell.py
@@ -54,7 +54,7 @@
 from idlelib.filelist import FileList
 from idlelib.outwin import OutputWindow
 from idlelib import rpc
-from idlelib.run import idle_formatwarning, PseudoInputFile, PseudoOutputFile
+from idlelib.run import idle_formatwarning, StdInputFile, StdOutputFile
 from idlelib.undo import UndoDelegator
 
 HOST = '127.0.0.1' # python execution server on localhost loopback
@@ -902,10 +902,14 @@ def __init__(self, flist=None):
         self.save_stderr = sys.stderr
         self.save_stdin = sys.stdin
         from idlelib import iomenu
-        self.stdin = PseudoInputFile(self, "stdin", iomenu.encoding)
-        self.stdout = PseudoOutputFile(self, "stdout", iomenu.encoding)
-        self.stderr = PseudoOutputFile(self, "stderr", iomenu.encoding)
-        self.console = PseudoOutputFile(self, "console", iomenu.encoding)
+        self.stdin = StdInputFile(self, "stdin",
+                                  iomenu.encoding, iomenu.errors)
+        self.stdout = StdOutputFile(self, "stdout",
+                                    iomenu.encoding, iomenu.errors)
+        self.stderr = StdOutputFile(self, "stderr",
+                                    iomenu.encoding, "backslashreplace")
+        self.console = StdOutputFile(self, "console",
+                                     iomenu.encoding, iomenu.errors)
         if not use_subprocess:
             sys.stdout = self.stdout
             sys.stderr = self.stderr
diff --git a/Lib/idlelib/run.py b/Lib/idlelib/run.py
index 41e0ded440293..5bd84aadcd801 100644
--- a/Lib/idlelib/run.py
+++ b/Lib/idlelib/run.py
@@ -401,17 +401,22 @@ def handle_error(self, request, client_address):
 
 # Pseudofiles for shell-remote communication (also used in pyshell)
 
-class PseudoFile(io.TextIOBase):
+class StdioFile(io.TextIOBase):
 
-    def __init__(self, shell, tags, encoding=None):
+    def __init__(self, shell, tags, encoding='utf-8', errors='strict'):
         self.shell = shell
         self.tags = tags
         self._encoding = encoding
+        self._errors = errors
 
     @property
     def encoding(self):
         return self._encoding
 
+    @property
+    def errors(self):
+        return self._errors
+
     @property
     def name(self):
         return '<%s>' % self.tags
@@ -420,7 +425,7 @@ def isatty(self):
         return True
 
 
-class PseudoOutputFile(PseudoFile):
+class StdOutputFile(StdioFile):
 
     def writable(self):
         return True
@@ -428,19 +433,12 @@ def writable(self):
     def write(self, s):
         if self.closed:
             raise ValueError("write to closed file")
-        if type(s) is not str:
-            if not isinstance(s, str):
-                raise TypeError('must be str, not ' + type(s).__name__)
-            # See issue #19481
-            s = str.__str__(s)
+        s = str.encode(s, self.encoding, self.errors).decode(self.encoding, self.errors)
         return self.shell.write(s, self.tags)
 
 
-class PseudoInputFile(PseudoFile):
-
-    def __init__(self, shell, tags, encoding=None):
-        PseudoFile.__init__(self, shell, tags, encoding)
-        self._line_buffer = ''
+class StdInputFile(StdioFile):
+    _line_buffer = ''
 
     def readable(self):
         return True
@@ -495,12 +493,12 @@ def handle(self):
         executive = Executive(self)
         self.register("exec", executive)
         self.console = self.get_remote_proxy("console")
-        sys.stdin = PseudoInputFile(self.console, "stdin",
-                iomenu.encoding)
-        sys.stdout = PseudoOutputFile(self.console, "stdout",
-                iomenu.encoding)
-        sys.stderr = PseudoOutputFile(self.console, "stderr",
-                iomenu.encoding)
+        sys.stdin = StdInputFile(self.console, "stdin",
+                                 iomenu.encoding, iomenu.errors)
+        sys.stdout = StdOutputFile(self.console, "stdout",
+                                   iomenu.encoding, iomenu.errors)
+        sys.stderr = StdOutputFile(self.console, "stderr",
+                                   iomenu.encoding, "backslashreplace")
 
         sys.displayhook = rpc.displayhook
         # page help() text to shell.
diff --git a/Misc/NEWS.d/next/IDLE/2019-10-04-18-03-09.bpo-36698.BKcmom.rst b/Misc/NEWS.d/next/IDLE/2019-10-04-18-03-09.bpo-36698.BKcmom.rst
new file mode 100644
index 0000000000000..5aaa3c92646a1
--- /dev/null
+++ b/Misc/NEWS.d/next/IDLE/2019-10-04-18-03-09.bpo-36698.BKcmom.rst
@@ -0,0 +1,3 @@
+IDLE no longer fails when write non-encodable characters to stderr.  It now
+escapes them with a backslash, as the regular Python interpreter. Added the
+``errors`` field to the standard streams.


From webhook-mailer at python.org  Tue Oct  8 09:29:58 2019
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Tue, 08 Oct 2019 13:29:58 -0000
Subject: [Python-checkins] bpo-38405: Make nested subclasses of
 typing.NamedTuple pickleable. (GH-16641)
Message-ID: <mailman.47.1570541398.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/13abda41003daf599587991d8291f0dacf6e9519
commit: 13abda41003daf599587991d8291f0dacf6e9519
branch: master
author: Serhiy Storchaka <storchaka at gmail.com>
committer: GitHub <noreply at github.com>
date: 2019-10-08T16:29:52+03:00
summary:

bpo-38405: Make nested subclasses of typing.NamedTuple pickleable. (GH-16641)

files:
A Misc/NEWS.d/next/Library/2019-10-08-11-18-40.bpo-38405.0-7e7s.rst
M Lib/test/test_typing.py
M Lib/typing.py

diff --git a/Lib/test/test_typing.py b/Lib/test/test_typing.py
index 97e6d88ebf70c..49417efe5111b 100644
--- a/Lib/test/test_typing.py
+++ b/Lib/test/test_typing.py
@@ -3468,6 +3468,9 @@ class D(UserName):
 
 
 class NamedTupleTests(BaseTestCase):
+    class NestedEmployee(NamedTuple):
+        name: str
+        cool: int
 
     def test_basics(self):
         Emp = NamedTuple('Emp', [('name', str), ('id', int)])
@@ -3587,14 +3590,25 @@ def test_namedtuple_errors(self):
         with self.assertRaises(TypeError):
             NamedTuple('Emp', fields=[('name', str), ('id', int)])
 
-    def test_pickle(self):
+    def test_copy_and_pickle(self):
         global Emp  # pickle wants to reference the class by name
-        Emp = NamedTuple('Emp', [('name', str), ('id', int)])
-        jane = Emp('jane', 37)
-        for proto in range(pickle.HIGHEST_PROTOCOL + 1):
-            z = pickle.dumps(jane, proto)
-            jane2 = pickle.loads(z)
-            self.assertEqual(jane2, jane)
+        Emp = NamedTuple('Emp', [('name', str), ('cool', int)])
+        for cls in Emp, CoolEmployee, self.NestedEmployee:
+            with self.subTest(cls=cls):
+                jane = cls('jane', 37)
+                for proto in range(pickle.HIGHEST_PROTOCOL + 1):
+                    z = pickle.dumps(jane, proto)
+                    jane2 = pickle.loads(z)
+                    self.assertEqual(jane2, jane)
+                    self.assertIsInstance(jane2, cls)
+
+                jane2 = copy(jane)
+                self.assertEqual(jane2, jane)
+                self.assertIsInstance(jane2, cls)
+
+                jane2 = deepcopy(jane)
+                self.assertEqual(jane2, jane)
+                self.assertIsInstance(jane2, cls)
 
 
 class TypedDictTests(BaseTestCase):
diff --git a/Lib/typing.py b/Lib/typing.py
index 2c75a76964873..0e842ff9f2048 100644
--- a/Lib/typing.py
+++ b/Lib/typing.py
@@ -1593,7 +1593,7 @@ def _make_nmtuple(name, types):
                '_fields', '_field_defaults', '_field_types',
                '_make', '_replace', '_asdict', '_source')
 
-_special = ('__module__', '__name__', '__qualname__', '__annotations__')
+_special = ('__module__', '__name__', '__annotations__')
 
 
 class NamedTupleMeta(type):
diff --git a/Misc/NEWS.d/next/Library/2019-10-08-11-18-40.bpo-38405.0-7e7s.rst b/Misc/NEWS.d/next/Library/2019-10-08-11-18-40.bpo-38405.0-7e7s.rst
new file mode 100644
index 0000000000000..ee346a30ec416
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-10-08-11-18-40.bpo-38405.0-7e7s.rst
@@ -0,0 +1 @@
+Nested subclasses of :class:`typing.NamedTuple` are now pickleable.


From webhook-mailer at python.org  Tue Oct  8 09:30:20 2019
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Tue, 08 Oct 2019 13:30:20 -0000
Subject: [Python-checkins] bpo-38407: Add docstrings for typing.SupportsXXX
 classes. (GH-16644)
Message-ID: <mailman.48.1570541422.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/8252c52e57283515ace5d4251584255dc5c60eb5
commit: 8252c52e57283515ace5d4251584255dc5c60eb5
branch: master
author: Serhiy Storchaka <storchaka at gmail.com>
committer: GitHub <noreply at github.com>
date: 2019-10-08T16:30:17+03:00
summary:

bpo-38407: Add docstrings for typing.SupportsXXX classes. (GH-16644)

files:
M Lib/typing.py

diff --git a/Lib/typing.py b/Lib/typing.py
index 0e842ff9f2048..27be37a369a92 100644
--- a/Lib/typing.py
+++ b/Lib/typing.py
@@ -1513,6 +1513,7 @@ def new_user(user_class: Type[U]) -> U:
 
 @runtime_checkable
 class SupportsInt(Protocol):
+    """An ABC with one abstract method __int__."""
     __slots__ = ()
 
     @abstractmethod
@@ -1522,6 +1523,7 @@ def __int__(self) -> int:
 
 @runtime_checkable
 class SupportsFloat(Protocol):
+    """An ABC with one abstract method __float__."""
     __slots__ = ()
 
     @abstractmethod
@@ -1531,6 +1533,7 @@ def __float__(self) -> float:
 
 @runtime_checkable
 class SupportsComplex(Protocol):
+    """An ABC with one abstract method __complex__."""
     __slots__ = ()
 
     @abstractmethod
@@ -1540,6 +1543,7 @@ def __complex__(self) -> complex:
 
 @runtime_checkable
 class SupportsBytes(Protocol):
+    """An ABC with one abstract method __bytes__."""
     __slots__ = ()
 
     @abstractmethod
@@ -1549,6 +1553,7 @@ def __bytes__(self) -> bytes:
 
 @runtime_checkable
 class SupportsIndex(Protocol):
+    """An ABC with one abstract method __index__."""
     __slots__ = ()
 
     @abstractmethod
@@ -1558,6 +1563,7 @@ def __index__(self) -> int:
 
 @runtime_checkable
 class SupportsAbs(Protocol[T_co]):
+    """An ABC with one abstract method __abs__ that is covariant in its return type."""
     __slots__ = ()
 
     @abstractmethod
@@ -1567,6 +1573,7 @@ def __abs__(self) -> T_co:
 
 @runtime_checkable
 class SupportsRound(Protocol[T_co]):
+    """An ABC with one abstract method __round__ that is covariant in its return type."""
     __slots__ = ()
 
     @abstractmethod


From webhook-mailer at python.org  Tue Oct  8 09:42:25 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Tue, 08 Oct 2019 13:42:25 -0000
Subject: [Python-checkins] bpo-38118: Ignore Valgrind false alarm in
 PyUnicode_Decode() (GH-16651)
Message-ID: <mailman.49.1570542146.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/03ab6b4fc6f59a4452756e7a3a46310ce30ec4b2
commit: 03ab6b4fc6f59a4452756e7a3a46310ce30ec4b2
branch: master
author: Victor Stinner <vstinner at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-08T15:42:17+02:00
summary:

bpo-38118: Ignore Valgrind false alarm in PyUnicode_Decode() (GH-16651)

Valgrind emits "Conditional jump or move depends on uninitialised
value(s)" false alarms on GCC builtin strcmp() function. The GCC code
is correct.

Valgrind bug: https://bugs.kde.org/show_bug.cgi?id=264936

files:
A Misc/NEWS.d/next/Tools-Demos/2019-10-08-15-07-52.bpo-38118.pIZD6H.rst
M Misc/valgrind-python.supp

diff --git a/Misc/NEWS.d/next/Tools-Demos/2019-10-08-15-07-52.bpo-38118.pIZD6H.rst b/Misc/NEWS.d/next/Tools-Demos/2019-10-08-15-07-52.bpo-38118.pIZD6H.rst
new file mode 100644
index 0000000000000..13adadb0b8f7a
--- /dev/null
+++ b/Misc/NEWS.d/next/Tools-Demos/2019-10-08-15-07-52.bpo-38118.pIZD6H.rst
@@ -0,0 +1,2 @@
+Update Valgrind suppression file to ignore a false alarm in
+:c:func:`PyUnicode_Decode` when using GCC builtin strcmp().
diff --git a/Misc/valgrind-python.supp b/Misc/valgrind-python.supp
index bc8f77f261a80..38a5ea3cd2b82 100644
--- a/Misc/valgrind-python.supp
+++ b/Misc/valgrind-python.supp
@@ -283,6 +283,17 @@
    fun:rl_initialize
 }
 
+# Valgrind emits "Conditional jump or move depends on uninitialised value(s)"
+# false alarms on GCC builtin strcmp() function. The GCC code is correct.
+#
+# Valgrind bug: https://bugs.kde.org/show_bug.cgi?id=264936
+{
+   bpo-38118: Valgrind emits false alarm on GCC builtin strcmp()
+   Memcheck:Cond
+   fun:PyUnicode_Decode
+}
+
+
 ###
 ### These occur from somewhere within the SSL, when running
 ###  test_socket_sll.  They are too general to leave on by default.


From webhook-mailer at python.org  Tue Oct  8 09:49:16 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Tue, 08 Oct 2019 13:49:16 -0000
Subject: [Python-checkins] bpo-38118: Ignore Valgrind false alarm in
 PyUnicode_Decode() (GH-16651)
Message-ID: <mailman.50.1570542557.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/364532fb4962991d278da6470c3aec4e443e533d
commit: 364532fb4962991d278da6470c3aec4e443e533d
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-08T06:49:12-07:00
summary:

bpo-38118: Ignore Valgrind false alarm in PyUnicode_Decode() (GH-16651)


Valgrind emits "Conditional jump or move depends on uninitialised
value(s)" false alarms on GCC builtin strcmp() function. The GCC code
is correct.

Valgrind bug: https://bugs.kde.org/show_bug.cgi?id=264936
(cherry picked from commit 03ab6b4fc6f59a4452756e7a3a46310ce30ec4b2)

Co-authored-by: Victor Stinner <vstinner at python.org>

files:
A Misc/NEWS.d/next/Tools-Demos/2019-10-08-15-07-52.bpo-38118.pIZD6H.rst
M Misc/valgrind-python.supp

diff --git a/Misc/NEWS.d/next/Tools-Demos/2019-10-08-15-07-52.bpo-38118.pIZD6H.rst b/Misc/NEWS.d/next/Tools-Demos/2019-10-08-15-07-52.bpo-38118.pIZD6H.rst
new file mode 100644
index 0000000000000..13adadb0b8f7a
--- /dev/null
+++ b/Misc/NEWS.d/next/Tools-Demos/2019-10-08-15-07-52.bpo-38118.pIZD6H.rst
@@ -0,0 +1,2 @@
+Update Valgrind suppression file to ignore a false alarm in
+:c:func:`PyUnicode_Decode` when using GCC builtin strcmp().
diff --git a/Misc/valgrind-python.supp b/Misc/valgrind-python.supp
index bc8f77f261a80..38a5ea3cd2b82 100644
--- a/Misc/valgrind-python.supp
+++ b/Misc/valgrind-python.supp
@@ -283,6 +283,17 @@
    fun:rl_initialize
 }
 
+# Valgrind emits "Conditional jump or move depends on uninitialised value(s)"
+# false alarms on GCC builtin strcmp() function. The GCC code is correct.
+#
+# Valgrind bug: https://bugs.kde.org/show_bug.cgi?id=264936
+{
+   bpo-38118: Valgrind emits false alarm on GCC builtin strcmp()
+   Memcheck:Cond
+   fun:PyUnicode_Decode
+}
+
+
 ###
 ### These occur from somewhere within the SSL, when running
 ###  test_socket_sll.  They are too general to leave on by default.


From webhook-mailer at python.org  Tue Oct  8 09:56:12 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Tue, 08 Oct 2019 13:56:12 -0000
Subject: [Python-checkins] bpo-38118: Ignore Valgrind false alarm in
 PyUnicode_Decode() (GH-16651)
Message-ID: <mailman.51.1570542972.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/98043b4fcd80bdfa8cc2be1730fa2a4cf4950958
commit: 98043b4fcd80bdfa8cc2be1730fa2a4cf4950958
branch: 3.7
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-08T06:56:08-07:00
summary:

bpo-38118: Ignore Valgrind false alarm in PyUnicode_Decode() (GH-16651)


Valgrind emits "Conditional jump or move depends on uninitialised
value(s)" false alarms on GCC builtin strcmp() function. The GCC code
is correct.

Valgrind bug: https://bugs.kde.org/show_bug.cgi?id=264936
(cherry picked from commit 03ab6b4fc6f59a4452756e7a3a46310ce30ec4b2)

Co-authored-by: Victor Stinner <vstinner at python.org>

files:
A Misc/NEWS.d/next/Tools-Demos/2019-10-08-15-07-52.bpo-38118.pIZD6H.rst
M Misc/valgrind-python.supp

diff --git a/Misc/NEWS.d/next/Tools-Demos/2019-10-08-15-07-52.bpo-38118.pIZD6H.rst b/Misc/NEWS.d/next/Tools-Demos/2019-10-08-15-07-52.bpo-38118.pIZD6H.rst
new file mode 100644
index 0000000000000..13adadb0b8f7a
--- /dev/null
+++ b/Misc/NEWS.d/next/Tools-Demos/2019-10-08-15-07-52.bpo-38118.pIZD6H.rst
@@ -0,0 +1,2 @@
+Update Valgrind suppression file to ignore a false alarm in
+:c:func:`PyUnicode_Decode` when using GCC builtin strcmp().
diff --git a/Misc/valgrind-python.supp b/Misc/valgrind-python.supp
index bc8f77f261a80..38a5ea3cd2b82 100644
--- a/Misc/valgrind-python.supp
+++ b/Misc/valgrind-python.supp
@@ -283,6 +283,17 @@
    fun:rl_initialize
 }
 
+# Valgrind emits "Conditional jump or move depends on uninitialised value(s)"
+# false alarms on GCC builtin strcmp() function. The GCC code is correct.
+#
+# Valgrind bug: https://bugs.kde.org/show_bug.cgi?id=264936
+{
+   bpo-38118: Valgrind emits false alarm on GCC builtin strcmp()
+   Memcheck:Cond
+   fun:PyUnicode_Decode
+}
+
+
 ###
 ### These occur from somewhere within the SSL, when running
 ###  test_socket_sll.  They are too general to leave on by default.


From webhook-mailer at python.org  Tue Oct  8 10:12:46 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Tue, 08 Oct 2019 14:12:46 -0000
Subject: [Python-checkins] bpo-38405: Make nested subclasses of
 typing.NamedTuple pickleable. (GH-16641)
Message-ID: <mailman.52.1570543967.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/10b475a151dc35c8ca4047331d591130973e914d
commit: 10b475a151dc35c8ca4047331d591130973e914d
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-08T07:12:38-07:00
summary:

bpo-38405: Make nested subclasses of typing.NamedTuple pickleable. (GH-16641)

(cherry picked from commit 13abda41003daf599587991d8291f0dacf6e9519)

Co-authored-by: Serhiy Storchaka <storchaka at gmail.com>

files:
A Misc/NEWS.d/next/Library/2019-10-08-11-18-40.bpo-38405.0-7e7s.rst
M Lib/test/test_typing.py
M Lib/typing.py

diff --git a/Lib/test/test_typing.py b/Lib/test/test_typing.py
index 3013f8ec41e4c..104b7c035ca6e 100644
--- a/Lib/test/test_typing.py
+++ b/Lib/test/test_typing.py
@@ -3468,6 +3468,9 @@ class D(UserName):
 
 
 class NamedTupleTests(BaseTestCase):
+    class NestedEmployee(NamedTuple):
+        name: str
+        cool: int
 
     def test_basics(self):
         Emp = NamedTuple('Emp', [('name', str), ('id', int)])
@@ -3593,14 +3596,25 @@ def test_namedtuple_errors(self):
         self.assertEqual(Emp.__name__, 'Emp')
         self.assertEqual(Emp._fields, ('name', 'id'))
 
-    def test_pickle(self):
+    def test_copy_and_pickle(self):
         global Emp  # pickle wants to reference the class by name
-        Emp = NamedTuple('Emp', [('name', str), ('id', int)])
-        jane = Emp('jane', 37)
-        for proto in range(pickle.HIGHEST_PROTOCOL + 1):
-            z = pickle.dumps(jane, proto)
-            jane2 = pickle.loads(z)
-            self.assertEqual(jane2, jane)
+        Emp = NamedTuple('Emp', [('name', str), ('cool', int)])
+        for cls in Emp, CoolEmployee, self.NestedEmployee:
+            with self.subTest(cls=cls):
+                jane = cls('jane', 37)
+                for proto in range(pickle.HIGHEST_PROTOCOL + 1):
+                    z = pickle.dumps(jane, proto)
+                    jane2 = pickle.loads(z)
+                    self.assertEqual(jane2, jane)
+                    self.assertIsInstance(jane2, cls)
+
+                jane2 = copy(jane)
+                self.assertEqual(jane2, jane)
+                self.assertIsInstance(jane2, cls)
+
+                jane2 = deepcopy(jane)
+                self.assertEqual(jane2, jane)
+                self.assertIsInstance(jane2, cls)
 
 
 class TypedDictTests(BaseTestCase):
diff --git a/Lib/typing.py b/Lib/typing.py
index 825be10812dee..eeebd85772788 100644
--- a/Lib/typing.py
+++ b/Lib/typing.py
@@ -1593,7 +1593,7 @@ def _make_nmtuple(name, types):
                '_fields', '_field_defaults', '_field_types',
                '_make', '_replace', '_asdict', '_source')
 
-_special = ('__module__', '__name__', '__qualname__', '__annotations__')
+_special = ('__module__', '__name__', '__annotations__')
 
 
 class NamedTupleMeta(type):
diff --git a/Misc/NEWS.d/next/Library/2019-10-08-11-18-40.bpo-38405.0-7e7s.rst b/Misc/NEWS.d/next/Library/2019-10-08-11-18-40.bpo-38405.0-7e7s.rst
new file mode 100644
index 0000000000000..ee346a30ec416
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-10-08-11-18-40.bpo-38405.0-7e7s.rst
@@ -0,0 +1 @@
+Nested subclasses of :class:`typing.NamedTuple` are now pickleable.


From webhook-mailer at python.org  Tue Oct  8 10:15:02 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Tue, 08 Oct 2019 14:15:02 -0000
Subject: [Python-checkins] bpo-38407: Add docstrings for typing.SupportsXXX
 classes. (GH-16644)
Message-ID: <mailman.53.1570544103.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/7162440a7247f77d22cd937143f358fd5ac85af4
commit: 7162440a7247f77d22cd937143f358fd5ac85af4
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-08T07:14:57-07:00
summary:

bpo-38407: Add docstrings for typing.SupportsXXX classes. (GH-16644)

(cherry picked from commit 8252c52e57283515ace5d4251584255dc5c60eb5)

Co-authored-by: Serhiy Storchaka <storchaka at gmail.com>

files:
M Lib/typing.py

diff --git a/Lib/typing.py b/Lib/typing.py
index eeebd85772788..bd30da90ad796 100644
--- a/Lib/typing.py
+++ b/Lib/typing.py
@@ -1513,6 +1513,7 @@ def new_user(user_class: Type[U]) -> U:
 
 @runtime_checkable
 class SupportsInt(Protocol):
+    """An ABC with one abstract method __int__."""
     __slots__ = ()
 
     @abstractmethod
@@ -1522,6 +1523,7 @@ def __int__(self) -> int:
 
 @runtime_checkable
 class SupportsFloat(Protocol):
+    """An ABC with one abstract method __float__."""
     __slots__ = ()
 
     @abstractmethod
@@ -1531,6 +1533,7 @@ def __float__(self) -> float:
 
 @runtime_checkable
 class SupportsComplex(Protocol):
+    """An ABC with one abstract method __complex__."""
     __slots__ = ()
 
     @abstractmethod
@@ -1540,6 +1543,7 @@ def __complex__(self) -> complex:
 
 @runtime_checkable
 class SupportsBytes(Protocol):
+    """An ABC with one abstract method __bytes__."""
     __slots__ = ()
 
     @abstractmethod
@@ -1549,6 +1553,7 @@ def __bytes__(self) -> bytes:
 
 @runtime_checkable
 class SupportsIndex(Protocol):
+    """An ABC with one abstract method __index__."""
     __slots__ = ()
 
     @abstractmethod
@@ -1558,6 +1563,7 @@ def __index__(self) -> int:
 
 @runtime_checkable
 class SupportsAbs(Protocol[T_co]):
+    """An ABC with one abstract method __abs__ that is covariant in its return type."""
     __slots__ = ()
 
     @abstractmethod
@@ -1567,6 +1573,7 @@ def __abs__(self) -> T_co:
 
 @runtime_checkable
 class SupportsRound(Protocol[T_co]):
+    """An ABC with one abstract method __round__ that is covariant in its return type."""
     __slots__ = ()
 
     @abstractmethod


From webhook-mailer at python.org  Tue Oct  8 11:30:57 2019
From: webhook-mailer at python.org (Pablo Galindo)
Date: Tue, 08 Oct 2019 15:30:57 -0000
Subject: [Python-checkins] bpo-38395: Fix ownership in weakref.proxy methods
 (GH-16632)
Message-ID: <mailman.54.1570548658.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/10cd00a9e3c22af37c748ea5a417f6fb66601e21
commit: 10cd00a9e3c22af37c748ea5a417f6fb66601e21
branch: master
author: Pablo Galindo <Pablogsal at gmail.com>
committer: GitHub <noreply at github.com>
date: 2019-10-08T16:30:50+01:00
summary:

bpo-38395: Fix ownership in weakref.proxy methods (GH-16632)

The implementation of weakref.proxy's methods call back into the Python
API using a borrowed references of the weakly referenced object
(acquired via PyWeakref_GET_OBJECT). This API call may delete the last
reference to the object (either directly or via GC), leaving a dangling
pointer, which can be subsequently dereferenced.

To fix this, claim a temporary ownership of the referenced object when
calling the appropriate method. Some functions because at the moment they
do not need to access the borrowed referent, but to protect against
future changes to these functions, ownership need to be fixed in
all potentially affected methods.

files:
A Misc/NEWS.d/next/C API/2019-10-08-01-23-24.bpo-38395.MJ6Ey9.rst
M Lib/test/test_weakref.py
M Objects/weakrefobject.c

diff --git a/Lib/test/test_weakref.py b/Lib/test/test_weakref.py
index e14df9ea08942..228bc17b23c59 100644
--- a/Lib/test/test_weakref.py
+++ b/Lib/test/test_weakref.py
@@ -391,6 +391,26 @@ class List(list): pass
         lyst = List()
         self.assertEqual(bool(weakref.proxy(lyst)), bool(lyst))
 
+    def test_proxy_iter(self):
+        # Test fails with a debug build of the interpreter
+        # (see bpo-38395).
+
+        obj = None
+
+        class MyObj:
+            def __iter__(self):
+                nonlocal obj
+                del obj
+                return NotImplemented
+
+        obj = MyObj()
+        p = weakref.proxy(obj)
+        with self.assertRaises(TypeError):
+            # "blech" in p calls MyObj.__iter__ through the proxy,
+            # without keeping a reference to the real object, so it
+            # can be killed in the middle of the call
+            "blech" in p
+
     def test_getweakrefcount(self):
         o = C()
         ref1 = weakref.ref(o)
diff --git a/Misc/NEWS.d/next/C API/2019-10-08-01-23-24.bpo-38395.MJ6Ey9.rst b/Misc/NEWS.d/next/C API/2019-10-08-01-23-24.bpo-38395.MJ6Ey9.rst
new file mode 100644
index 0000000000000..4bc30600ca758
--- /dev/null
+++ b/Misc/NEWS.d/next/C API/2019-10-08-01-23-24.bpo-38395.MJ6Ey9.rst	
@@ -0,0 +1,3 @@
+Fix a crash in :class:`weakref.proxy` objects due to incorrect lifetime
+management when calling some associated methods that may delete the last
+reference to object being referenced by the proxy. Patch by Pablo Galindo.
diff --git a/Objects/weakrefobject.c b/Objects/weakrefobject.c
index daee476444a4f..bf79e0c7ecbcc 100644
--- a/Objects/weakrefobject.c
+++ b/Objects/weakrefobject.c
@@ -145,11 +145,14 @@ weakref_hash(PyWeakReference *self)
 {
     if (self->hash != -1)
         return self->hash;
-    if (PyWeakref_GET_OBJECT(self) == Py_None) {
+    PyObject* obj = PyWeakref_GET_OBJECT(self);
+    if (obj == Py_None) {
         PyErr_SetString(PyExc_TypeError, "weak object has gone away");
         return -1;
     }
-    self->hash = PyObject_Hash(PyWeakref_GET_OBJECT(self));
+    Py_INCREF(obj);
+    self->hash = PyObject_Hash(obj);
+    Py_DECREF(obj);
     return self->hash;
 }
 
@@ -159,11 +162,15 @@ weakref_repr(PyWeakReference *self)
 {
     PyObject *name, *repr;
     _Py_IDENTIFIER(__name__);
+    PyObject* obj = PyWeakref_GET_OBJECT(self);
 
-    if (PyWeakref_GET_OBJECT(self) == Py_None)
+    if (obj == Py_None) {
         return PyUnicode_FromFormat("<weakref at %p; dead>", self);
+    }
 
-    if (_PyObject_LookupAttrId(PyWeakref_GET_OBJECT(self), &PyId___name__, &name) < 0) {
+    Py_INCREF(obj);
+    if (_PyObject_LookupAttrId(obj, &PyId___name__, &name) < 0) {
+        Py_DECREF(obj);
         return NULL;
     }
     if (name == NULL || !PyUnicode_Check(name)) {
@@ -171,16 +178,17 @@ weakref_repr(PyWeakReference *self)
             "<weakref at %p; to '%s' at %p>",
             self,
             Py_TYPE(PyWeakref_GET_OBJECT(self))->tp_name,
-            PyWeakref_GET_OBJECT(self));
+            obj);
     }
     else {
         repr = PyUnicode_FromFormat(
             "<weakref at %p; to '%s' at %p (%U)>",
             self,
             Py_TYPE(PyWeakref_GET_OBJECT(self))->tp_name,
-            PyWeakref_GET_OBJECT(self),
+            obj,
             name);
     }
+    Py_DECREF(obj);
     Py_XDECREF(name);
     return repr;
 }
@@ -207,8 +215,14 @@ weakref_richcompare(PyWeakReference* self, PyWeakReference* other, int op)
         else
             Py_RETURN_FALSE;
     }
-    return PyObject_RichCompare(PyWeakref_GET_OBJECT(self),
-                                PyWeakref_GET_OBJECT(other), op);
+    PyObject* obj = PyWeakref_GET_OBJECT(self);
+    PyObject* other_obj = PyWeakref_GET_OBJECT(other);
+    Py_INCREF(obj);
+    Py_INCREF(other_obj);
+    PyObject* res = PyObject_RichCompare(obj, other_obj, op);
+    Py_DECREF(obj);
+    Py_DECREF(other_obj);
+    return res;
 }
 
 /* Given the head of an object's list of weak references, extract the
@@ -415,18 +429,14 @@ proxy_checkref(PyWeakReference *proxy)
             o = PyWeakref_GET_OBJECT(o); \
         }
 
-#define UNWRAP_I(o) \
-        if (PyWeakref_CheckProxy(o)) { \
-            if (!proxy_checkref((PyWeakReference *)o)) \
-                return -1; \
-            o = PyWeakref_GET_OBJECT(o); \
-        }
-
 #define WRAP_UNARY(method, generic) \
     static PyObject * \
     method(PyObject *proxy) { \
         UNWRAP(proxy); \
-        return generic(proxy); \
+        Py_INCREF(proxy); \
+        PyObject* res = generic(proxy); \
+        Py_DECREF(proxy); \
+        return res; \
     }
 
 #define WRAP_BINARY(method, generic) \
@@ -434,7 +444,12 @@ proxy_checkref(PyWeakReference *proxy)
     method(PyObject *x, PyObject *y) { \
         UNWRAP(x); \
         UNWRAP(y); \
-        return generic(x, y); \
+        Py_INCREF(x); \
+        Py_INCREF(y); \
+        PyObject* res = generic(x, y); \
+        Py_DECREF(x); \
+        Py_DECREF(y); \
+        return res; \
     }
 
 /* Note that the third arg needs to be checked for NULL since the tp_call
@@ -447,7 +462,14 @@ proxy_checkref(PyWeakReference *proxy)
         UNWRAP(v); \
         if (w != NULL) \
             UNWRAP(w); \
-        return generic(proxy, v, w); \
+        Py_INCREF(proxy); \
+        Py_INCREF(v); \
+        Py_XINCREF(w); \
+        PyObject* res = generic(proxy, v, w); \
+        Py_DECREF(proxy); \
+        Py_DECREF(v); \
+        Py_XDECREF(w); \
+        return res; \
     }
 
 #define WRAP_METHOD(method, special) \
@@ -455,7 +477,10 @@ proxy_checkref(PyWeakReference *proxy)
     method(PyObject *proxy, PyObject *Py_UNUSED(ignored)) { \
             _Py_IDENTIFIER(special); \
             UNWRAP(proxy); \
-                return _PyObject_CallMethodIdNoArgs(proxy, &PyId_##special); \
+            Py_INCREF(proxy); \
+            PyObject* res = _PyObject_CallMethodIdNoArgs(proxy, &PyId_##special); \
+            Py_DECREF(proxy); \
+            return res; \
         }
 
 
@@ -481,7 +506,11 @@ proxy_setattr(PyWeakReference *proxy, PyObject *name, PyObject *value)
 {
     if (!proxy_checkref(proxy))
         return -1;
-    return PyObject_SetAttr(PyWeakref_GET_OBJECT(proxy), name, value);
+    PyObject *obj = PyWeakref_GET_OBJECT(proxy);
+    Py_INCREF(obj);
+    int res = PyObject_SetAttr(obj, name, value);
+    Py_DECREF(obj);
+    return res;
 }
 
 static PyObject *
@@ -532,9 +561,13 @@ static int
 proxy_bool(PyWeakReference *proxy)
 {
     PyObject *o = PyWeakref_GET_OBJECT(proxy);
-    if (!proxy_checkref(proxy))
+    if (!proxy_checkref(proxy)) {
         return -1;
-    return PyObject_IsTrue(o);
+    }
+    Py_INCREF(o);
+    int res = PyObject_IsTrue(o);
+    Py_DECREF(o);
+    return res;
 }
 
 static void
@@ -553,9 +586,13 @@ proxy_contains(PyWeakReference *proxy, PyObject *value)
 {
     if (!proxy_checkref(proxy))
         return -1;
-    return PySequence_Contains(PyWeakref_GET_OBJECT(proxy), value);
-}
 
+    PyObject *obj = PyWeakref_GET_OBJECT(proxy);
+    Py_INCREF(obj);
+    int res = PySequence_Contains(obj, value);
+    Py_DECREF(obj);
+    return res;
+}
 
 /* mapping slots */
 
@@ -564,7 +601,12 @@ proxy_length(PyWeakReference *proxy)
 {
     if (!proxy_checkref(proxy))
         return -1;
-    return PyObject_Length(PyWeakref_GET_OBJECT(proxy));
+
+    PyObject *obj = PyWeakref_GET_OBJECT(proxy);
+    Py_INCREF(obj);
+    Py_ssize_t res = PyObject_Length(obj);
+    Py_DECREF(obj);
+    return res;
 }
 
 WRAP_BINARY(proxy_getitem, PyObject_GetItem)
@@ -575,10 +617,16 @@ proxy_setitem(PyWeakReference *proxy, PyObject *key, PyObject *value)
     if (!proxy_checkref(proxy))
         return -1;
 
-    if (value == NULL)
-        return PyObject_DelItem(PyWeakref_GET_OBJECT(proxy), key);
-    else
-        return PyObject_SetItem(PyWeakref_GET_OBJECT(proxy), key, value);
+    PyObject *obj = PyWeakref_GET_OBJECT(proxy);
+    Py_INCREF(obj);
+    int res;
+    if (value == NULL) {
+        res = PyObject_DelItem(obj, key);
+    } else {
+        res = PyObject_SetItem(obj, key, value);
+    }
+    Py_DECREF(obj);
+    return res;
 }
 
 /* iterator slots */
@@ -588,7 +636,11 @@ proxy_iter(PyWeakReference *proxy)
 {
     if (!proxy_checkref(proxy))
         return NULL;
-    return PyObject_GetIter(PyWeakref_GET_OBJECT(proxy));
+    PyObject *obj = PyWeakref_GET_OBJECT(proxy);
+    Py_INCREF(obj);
+    PyObject* res = PyObject_GetIter(obj);
+    Py_DECREF(obj);
+    return res;
 }
 
 static PyObject *
@@ -596,7 +648,12 @@ proxy_iternext(PyWeakReference *proxy)
 {
     if (!proxy_checkref(proxy))
         return NULL;
-    return PyIter_Next(PyWeakref_GET_OBJECT(proxy));
+
+    PyObject *obj = PyWeakref_GET_OBJECT(proxy);
+    Py_INCREF(obj);
+    PyObject* res = PyIter_Next(obj);
+    Py_DECREF(obj);
+    return res;
 }
 
 


From webhook-mailer at python.org  Tue Oct  8 11:59:14 2019
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Tue, 08 Oct 2019 15:59:14 -0000
Subject: [Python-checkins] test_dictviews: Add testcase for dictviews_sub
 (GH-16660)
Message-ID: <mailman.55.1570550355.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/e53c5800df3d4a07c9596ed933d431f7f77a2980
commit: e53c5800df3d4a07c9596ed933d431f7f77a2980
branch: master
author: Dong-hee Na <donghee.na92 at gmail.com>
committer: Serhiy Storchaka <storchaka at gmail.com>
date: 2019-10-08T18:59:10+03:00
summary:

test_dictviews: Add testcase for dictviews_sub (GH-16660)

files:
M Lib/test/test_dictviews.py

diff --git a/Lib/test/test_dictviews.py b/Lib/test/test_dictviews.py
index 7cf019250244b..8410e8b5c44d4 100644
--- a/Lib/test/test_dictviews.py
+++ b/Lib/test/test_dictviews.py
@@ -219,11 +219,13 @@ def test_set_operations_with_iterator(self):
         self.assertEqual(origin.keys() & iter([1, 2]), {1})
         self.assertEqual(origin.keys() | iter([1, 2]), {1, 2, 3})
         self.assertEqual(origin.keys() ^ iter([1, 2]), {2, 3})
+        self.assertEqual(origin.keys() - iter([1, 2]), {3})
 
         items = origin.items()
         self.assertEqual(items & iter([(1, 2)]), {(1, 2)})
         self.assertEqual(items ^ iter([(1, 2)]), {(3, 4)})
         self.assertEqual(items | iter([(1, 2)]), {(1, 2), (3, 4)})
+        self.assertEqual(items - iter([(1, 2)]), {(3, 4)})
 
     def test_recursive_repr(self):
         d = {}


From webhook-mailer at python.org  Tue Oct  8 12:41:05 2019
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Tue, 08 Oct 2019 16:41:05 -0000
Subject: [Python-checkins] [3.7] bpo-38294: Add list of no-longer-escaped
 chars to re.escape documentation. (GH-16442) (GH-16647)
Message-ID: <mailman.56.1570552866.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/43f5c0c4d0a608c04c3978adb45117f3943203b7
commit: 43f5c0c4d0a608c04c3978adb45117f3943203b7
branch: 3.7
author: Ricardo B?nffy <rbanffy at gmail.com>
committer: Serhiy Storchaka <storchaka at gmail.com>
date: 2019-10-08T19:41:00+03:00
summary:

[3.7] bpo-38294: Add list of no-longer-escaped chars to re.escape documentation. (GH-16442) (GH-16647)

Prior to 3.7, re.escape escaped many characters that don't have
special meaning in Python, but that use to require escaping in other
tools and languages. This commit aims to make it clear which characters
were, but are no longer escaped.
(cherry picked from commit 15ae75d660befe643ed42eb2707a557cea97256c)

files:
A Misc/NEWS.d/next/Documentation/2019-09-27-23-37-41.bpo-38294.go_jFf.rst
M Doc/library/re.rst
M Doc/tools/susp-ignored.csv

diff --git a/Doc/library/re.rst b/Doc/library/re.rst
index a23e45f2eb52f..4245c2417fee6 100644
--- a/Doc/library/re.rst
+++ b/Doc/library/re.rst
@@ -925,8 +925,8 @@ form.
    This is useful if you want to match an arbitrary literal string that may
    have regular expression metacharacters in it.  For example::
 
-      >>> print(re.escape('python.exe'))
-      python\.exe
+      >>> print(re.escape('http://www.python.org'))
+      http://www\.python\.org
 
       >>> legal_chars = string.ascii_lowercase + string.digits + "!#$%&'*+-.^_`|~:"
       >>> print('[%s]+' % re.escape(legal_chars))
@@ -949,7 +949,9 @@ form.
 
    .. versionchanged:: 3.7
       Only characters that can have special meaning in a regular expression
-      are escaped.
+      are escaped. As a result, ``'!'``, ``'"'``, ``'%'``, ``"'"``, ``','``,
+      ``'/'``, ``':'``, ``';'``, ``'<'``, ``'='``, ``'>'``, ``'@'``, and
+      ``"`"`` are no longer escaped.
 
 
 .. function:: purge()
diff --git a/Doc/tools/susp-ignored.csv b/Doc/tools/susp-ignored.csv
index 35dfce2c5c80e..36d93a45b0209 100644
--- a/Doc/tools/susp-ignored.csv
+++ b/Doc/tools/susp-ignored.csv
@@ -354,3 +354,4 @@ whatsnew/3.7,,::,error::BytesWarning
 whatsnew/changelog,,::,error::BytesWarning
 whatsnew/changelog,,::,default::BytesWarning
 whatsnew/changelog,,::,default::DeprecationWarning
+library/re,,`,"`"
diff --git a/Misc/NEWS.d/next/Documentation/2019-09-27-23-37-41.bpo-38294.go_jFf.rst b/Misc/NEWS.d/next/Documentation/2019-09-27-23-37-41.bpo-38294.go_jFf.rst
new file mode 100644
index 0000000000000..0bde8c868c9ec
--- /dev/null
+++ b/Misc/NEWS.d/next/Documentation/2019-09-27-23-37-41.bpo-38294.go_jFf.rst
@@ -0,0 +1 @@
+Add list of no-longer-escaped chars to re.escape documentation
\ No newline at end of file


From webhook-mailer at python.org  Tue Oct  8 12:45:54 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Tue, 08 Oct 2019 16:45:54 -0000
Subject: [Python-checkins] bpo-37531: regrtest ignores output on timeout
 (GH-16659)
Message-ID: <mailman.57.1570553156.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/0ec618af98ac250a91ee9c91f8569e6df6772758
commit: 0ec618af98ac250a91ee9c91f8569e6df6772758
branch: master
author: Victor Stinner <vstinner at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-08T18:45:43+02:00
summary:

bpo-37531: regrtest ignores output on timeout (GH-16659)

bpo-37531, bpo-38207: On timeout, regrtest no longer attempts to call
`popen.communicate() again: it can hang until all child processes
using stdout and stderr pipes completes. Kill the worker process and
ignores its output.

Reenable test_regrtest.test_multiprocessing_timeout().

bpo-37531: Change also the faulthandler timeout of the main process
from 1 minute to 5 minutes, for Python slowest buildbots.

files:
A Misc/NEWS.d/next/Tests/2019-10-08-16-42-05.bpo-37531.7v-_Ca.rst
M Lib/test/libregrtest/runtest_mp.py
M Lib/test/test_regrtest.py

diff --git a/Lib/test/libregrtest/runtest_mp.py b/Lib/test/libregrtest/runtest_mp.py
index 79afa29fa05bf..71db8d6966859 100644
--- a/Lib/test/libregrtest/runtest_mp.py
+++ b/Lib/test/libregrtest/runtest_mp.py
@@ -22,6 +22,12 @@
 PROGRESS_UPDATE = 30.0   # seconds
 assert PROGRESS_UPDATE >= PROGRESS_MIN_TIME
 
+# Kill the main process after 5 minutes. It is supposed to write an update
+# every PROGRESS_UPDATE seconds. Tolerate 5 minutes for Python slowest
+# buildbot workers.
+MAIN_PROCESS_TIMEOUT = 5 * 60.0
+assert MAIN_PROCESS_TIMEOUT >= PROGRESS_UPDATE
+
 # Time to wait until a worker completes: should be immediate
 JOIN_TIMEOUT = 30.0   # seconds
 
@@ -160,20 +166,6 @@ def mp_result_error(self, test_name, error_type, stdout='', stderr='',
         result = TestResult(test_name, error_type, test_time, None)
         return MultiprocessResult(result, stdout, stderr, err_msg)
 
-    def _timedout(self, test_name):
-        self._kill()
-
-        stdout = stderr = ''
-        popen = self._popen
-        try:
-            stdout, stderr = popen.communicate(timeout=JOIN_TIMEOUT)
-        except (subprocess.TimeoutExpired, OSError) as exc:
-            print_warning(f"Failed to read {self} output "
-                          f"(timeout={format_duration(JOIN_TIMEOUT)}): "
-                          f"{exc!r}")
-
-        return self.mp_result_error(test_name, TIMEOUT, stdout, stderr)
-
     def _run_process(self, test_name):
         self.start_time = time.monotonic()
 
@@ -197,23 +189,32 @@ def _run_process(self, test_name):
 
             try:
                 stdout, stderr = popen.communicate(timeout=self.timeout)
+                retcode = popen.returncode
+                assert retcode is not None
             except subprocess.TimeoutExpired:
                 if self._stopped:
                     # kill() has been called: communicate() fails
                     # on reading closed stdout/stderr
                     raise ExitThread
 
-                return self._timedout(test_name)
+                # On timeout, kill the process
+                self._kill()
+
+                # None means TIMEOUT for the caller
+                retcode = None
+                # bpo-38207: Don't attempt to call communicate() again: on it
+                # can hang until all child processes using stdout and stderr
+                # pipes completes.
+                stdout = stderr = ''
             except OSError:
                 if self._stopped:
                     # kill() has been called: communicate() fails
                     # on reading closed stdout/stderr
                     raise ExitThread
                 raise
-
-            retcode = popen.returncode
-            stdout = stdout.strip()
-            stderr = stderr.rstrip()
+            else:
+                stdout = stdout.strip()
+                stderr = stderr.rstrip()
 
             return (retcode, stdout, stderr)
         except:
@@ -225,13 +226,10 @@ def _run_process(self, test_name):
             self.current_test_name = None
 
     def _runtest(self, test_name):
-        result = self._run_process(test_name)
-
-        if isinstance(result, MultiprocessResult):
-            # _timedout() case
-            return result
+        retcode, stdout, stderr = self._run_process(test_name)
 
-        retcode, stdout, stderr = result
+        if retcode is None:
+            return self.mp_result_error(test_name, TIMEOUT, stdout, stderr)
 
         err_msg = None
         if retcode != 0:
@@ -250,7 +248,8 @@ def _runtest(self, test_name):
                     err_msg = "Failed to parse worker JSON: %s" % exc
 
         if err_msg is not None:
-            return self.mp_result_error(test_name, CHILD_ERROR, stdout, stderr, err_msg)
+            return self.mp_result_error(test_name, CHILD_ERROR,
+                                        stdout, stderr, err_msg)
 
         return MultiprocessResult(result, stdout, stderr, err_msg)
 
@@ -289,6 +288,16 @@ def _wait_completed(self):
                           f"{exc!r}")
 
     def wait_stopped(self, start_time):
+        # bpo-38207: MultiprocessTestRunner.stop_workers() called self.stop()
+        # which killed the process. Sometimes, killing the process from the
+        # main thread does not interrupt popen.communicate() in
+        # TestWorkerProcess thread. This loop with a timeout is a workaround
+        # for that.
+        #
+        # Moreover, if this method fails to join the thread, it is likely
+        # that Python will hang at exit while calling threading._shutdown()
+        # which tries again to join the blocked thread. Regrtest.main()
+        # uses EXIT_TIMEOUT to workaround this second bug.
         while True:
             # Write a message every second
             self.join(1.0)
@@ -355,7 +364,8 @@ def _get_result(self):
         timeout = PROGRESS_UPDATE
         while True:
             if use_faulthandler:
-                faulthandler.dump_traceback_later(timeout * 2.0, exit=True)
+                faulthandler.dump_traceback_later(MAIN_PROCESS_TIMEOUT,
+                                                  exit=True)
 
             # wait for a thread
             try:
diff --git a/Lib/test/test_regrtest.py b/Lib/test/test_regrtest.py
index aaa97e040804a..5df78861db9ce 100644
--- a/Lib/test/test_regrtest.py
+++ b/Lib/test/test_regrtest.py
@@ -1153,7 +1153,6 @@ def test_garbage(self):
                                   env_changed=[testname],
                                   fail_env_changed=True)
 
-    @unittest.skipIf(True, 'bpo-37531, bpo-38207: test hangs randomly')
     def test_multiprocessing_timeout(self):
         code = textwrap.dedent(r"""
             import time
diff --git a/Misc/NEWS.d/next/Tests/2019-10-08-16-42-05.bpo-37531.7v-_Ca.rst b/Misc/NEWS.d/next/Tests/2019-10-08-16-42-05.bpo-37531.7v-_Ca.rst
new file mode 100644
index 0000000000000..78ab636516514
--- /dev/null
+++ b/Misc/NEWS.d/next/Tests/2019-10-08-16-42-05.bpo-37531.7v-_Ca.rst
@@ -0,0 +1,5 @@
+On timeout, regrtest no longer attempts to call ``popen.communicate()``
+again: it can hang until all child processes using stdout and stderr pipes
+completes. Kill the worker process and ignores its output. Change also the
+faulthandler timeout of the main process from 1 minute to 5 minutes, for Python
+slowest buildbots.


From webhook-mailer at python.org  Tue Oct  8 13:12:26 2019
From: webhook-mailer at python.org (larryhastings)
Date: Tue, 08 Oct 2019 17:12:26 -0000
Subject: [Python-checkins] [3.7] Doc: Bump sphinx. (GH-10676) (GH-10803)
 (#16522)
Message-ID: <mailman.58.1570554747.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/edd9bc9cc99c919bdc87e5955ec8ad93c295fbe2
commit: edd9bc9cc99c919bdc87e5955ec8ad93c295fbe2
branch: 3.5
author: Victor Stinner <vstinner at redhat.com>
committer: larryhastings <larry at hastings.org>
date: 2019-10-08T18:12:21+01:00
summary:

[3.7] Doc: Bump sphinx. (GH-10676) (GH-10803) (#16522)

(cherry picked from commit 2db96ae7444880d66d4ef65abab8a5e6ff328711)

Co-authored-by: Julien Palard <julien at palard.fr>
(cherry picked from commit 23a98abd4256f931ed89b65ec6babd4f06dbff97)

files:
M .travis.yml

diff --git a/.travis.yml b/.travis.yml
index eb98a657b2706..20912d5440be8 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -40,7 +40,7 @@ matrix:
         - cd Doc
         # Sphinx is pinned so that new versions that introduce new warnings won't suddenly cause build failures.
         # (Updating the version is fine as long as no warnings are raised by doing so.)
-        - python -m pip install sphinx~=1.6.1 blurb
+        - python -m pip install sphinx==1.8.2 blurb
       script:
         - make check suspicious html SPHINXOPTS="-q -W -j4"
     - os: linux


From webhook-mailer at python.org  Tue Oct  8 13:36:57 2019
From: webhook-mailer at python.org (larryhastings)
Date: Tue, 08 Oct 2019 17:36:57 -0000
Subject: [Python-checkins] [3.5] bpo-38216,
 bpo-36274: Allow subclasses to separately override validation and
 encoding behavior (GH-16448) (#16475)
Message-ID: <mailman.59.1570556219.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/2784e78dc3445c6dd59e915d86c336374c1fa09a
commit: 2784e78dc3445c6dd59e915d86c336374c1fa09a
branch: 3.5
author: Jason R. Coombs <jaraco at jaraco.com>
committer: larryhastings <larry at hastings.org>
date: 2019-10-08T18:36:44+01:00
summary:

[3.5] bpo-38216, bpo-36274: Allow subclasses to separately override validation and encoding behavior (GH-16448) (#16475)

* [3.5] bpo-38216, bpo-36274: Allow subclasses to separately override validation and encoding behavior (GH-16448)

files:
A Misc/NEWS.d/next/Library/2019-09-27-15-24-45.bpo-38216.-7yvZR.rst
M Lib/http/client.py
M Lib/test/test_httplib.py

diff --git a/Lib/http/client.py b/Lib/http/client.py
index 76b9be69a3748..85dc8028ef57f 100644
--- a/Lib/http/client.py
+++ b/Lib/http/client.py
@@ -984,20 +984,15 @@ def putrequest(self, method, url, skip_host=False,
         else:
             raise CannotSendRequest(self.__state)
 
-        # Save the method we use, we need it later in the response phase
+        # Save the method for use later in the response phase
         self._method = method
-        if not url:
-            url = '/'
-        # Prevent CVE-2019-9740.
-        match = _contains_disallowed_url_pchar_re.search(url)
-        if match:
-            raise InvalidURL("URL can't contain control characters. {!r} "
-                             "(found at least {!r})".format(url,
-                                                            match.group()))
+
+        url = url or '/'
+        self._validate_path(url)
+
         request = '%s %s %s' % (method, url, self._http_vsn_str)
 
-        # Non-ASCII characters should have been eliminated earlier
-        self._output(request.encode('ascii'))
+        self._output(self._encode_request(request))
 
         if self._http_vsn == 11:
             # Issue some standard headers for better HTTP/1.1 compliance
@@ -1075,6 +1070,21 @@ def putrequest(self, method, url, skip_host=False,
             # For HTTP/1.0, the server will assume "not chunked"
             pass
 
+    def _encode_request(self, request):
+        # ASCII also helps prevent CVE-2019-9740.
+        return request.encode('ascii')
+
+    def _validate_path(self, url):
+        """Validate a url for putrequest."""
+        # Prevent CVE-2019-9740.
+        match = _contains_disallowed_url_pchar_re.search(url)
+        if match:
+            msg = (
+                "URL can't contain control characters. {url!r} "
+                "(found at least {matched!r})"
+            ).format(matched=match.group(), **locals())
+            raise InvalidURL(msg)
+
     def putheader(self, header, *values):
         """Send a request header line to the server.
 
diff --git a/Lib/test/test_httplib.py b/Lib/test/test_httplib.py
index 61ed6bbbd7cf3..c12a4298bb04e 100644
--- a/Lib/test/test_httplib.py
+++ b/Lib/test/test_httplib.py
@@ -986,6 +986,34 @@ def run_server():
             thread.join()
         self.assertEqual(result, b"proxied data\n")
 
+    def test_putrequest_override_validation(self):
+        """
+        It should be possible to override the default validation
+        behavior in putrequest (bpo-38216).
+        """
+        class UnsafeHTTPConnection(client.HTTPConnection):
+            def _validate_path(self, url):
+                pass
+
+        conn = UnsafeHTTPConnection('example.com')
+        conn.sock = FakeSocket('')
+        conn.putrequest('GET', '/\x00')
+
+    def test_putrequest_override_encoding(self):
+        """
+        It should be possible to override the default encoding
+        to transmit bytes in another encoding even if invalid
+        (bpo-36274).
+        """
+        class UnsafeHTTPConnection(client.HTTPConnection):
+            def _encode_request(self, str_url):
+                return str_url.encode('utf-8')
+
+        conn = UnsafeHTTPConnection('example.com')
+        conn.sock = FakeSocket('')
+        conn.putrequest('GET', '/?')
+
+
 class ExtendedReadTest(TestCase):
     """
     Test peek(), read1(), readline()
@@ -1110,6 +1138,7 @@ def test_peek_0(self):
         p = self.resp.peek(0)
         self.assertLessEqual(0, len(p))
 
+
 class ExtendedReadTestChunked(ExtendedReadTest):
     """
     Test peek(), read1(), readline() in chunked mode
diff --git a/Misc/NEWS.d/next/Library/2019-09-27-15-24-45.bpo-38216.-7yvZR.rst b/Misc/NEWS.d/next/Library/2019-09-27-15-24-45.bpo-38216.-7yvZR.rst
new file mode 100644
index 0000000000000..ac8e2b042d929
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-09-27-15-24-45.bpo-38216.-7yvZR.rst
@@ -0,0 +1,4 @@
+Allow the rare code that wants to send invalid http requests from the
+`http.client` library a way to do so.  The fixes for bpo-30458 led to
+breakage for some projects that were relying on this ability to test their
+own behavior in the face of bad requests.


From webhook-mailer at python.org  Tue Oct  8 16:38:15 2019
From: webhook-mailer at python.org (Pablo Galindo)
Date: Tue, 08 Oct 2019 20:38:15 -0000
Subject: [Python-checkins] [3.8] bpo-38395: Fix ownership in weakref.proxy
 methods (GH-16632) (GH-16662)
Message-ID: <mailman.60.1570567096.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/526ef856dd598fd3cefdfadeb18ede7a8e57aa41
commit: 526ef856dd598fd3cefdfadeb18ede7a8e57aa41
branch: 3.8
author: Pablo Galindo <Pablogsal at gmail.com>
committer: GitHub <noreply at github.com>
date: 2019-10-08T21:38:11+01:00
summary:

[3.8] bpo-38395: Fix ownership in weakref.proxy methods (GH-16632) (GH-16662)

The implementation of weakref.proxy's methods call back into the Python
API using a borrowed references of the weakly referenced object
(acquired via PyWeakref_GET_OBJECT). This API call may delete the last
reference to the object (either directly or via GC), leaving a dangling
pointer, which can be subsequently dereferenced.

To fix this, claim a temporary ownership of the referenced object when
calling the appropriate method. Some functions because at the moment they
do not need to access the borrowed referent, but to protect against
future changes to these functions, ownership need to be fixed in
all potentially affected methods..
(cherry picked from commit 10cd00a9e3c22af37c748ea5a417f6fb66601e21)

Co-authored-by: Pablo Galindo <Pablogsal at gmail.com>

files:
A Misc/NEWS.d/next/C API/2019-10-08-01-23-24.bpo-38395.MJ6Ey9.rst
M Lib/test/test_weakref.py
M Objects/weakrefobject.c

diff --git a/Lib/test/test_weakref.py b/Lib/test/test_weakref.py
index d9e1b201f6567..2dbf3f452c10e 100644
--- a/Lib/test/test_weakref.py
+++ b/Lib/test/test_weakref.py
@@ -391,6 +391,26 @@ class List(list): pass
         lyst = List()
         self.assertEqual(bool(weakref.proxy(lyst)), bool(lyst))
 
+    def test_proxy_iter(self):
+        # Test fails with a debug build of the interpreter
+        # (see bpo-38395).
+
+        obj = None
+
+        class MyObj:
+            def __iter__(self):
+                nonlocal obj
+                del obj
+                return NotImplemented
+
+        obj = MyObj()
+        p = weakref.proxy(obj)
+        with self.assertRaises(TypeError):
+            # "blech" in p calls MyObj.__iter__ through the proxy,
+            # without keeping a reference to the real object, so it
+            # can be killed in the middle of the call
+            "blech" in p
+
     def test_getweakrefcount(self):
         o = C()
         ref1 = weakref.ref(o)
diff --git a/Misc/NEWS.d/next/C API/2019-10-08-01-23-24.bpo-38395.MJ6Ey9.rst b/Misc/NEWS.d/next/C API/2019-10-08-01-23-24.bpo-38395.MJ6Ey9.rst
new file mode 100644
index 0000000000000..4bc30600ca758
--- /dev/null
+++ b/Misc/NEWS.d/next/C API/2019-10-08-01-23-24.bpo-38395.MJ6Ey9.rst	
@@ -0,0 +1,3 @@
+Fix a crash in :class:`weakref.proxy` objects due to incorrect lifetime
+management when calling some associated methods that may delete the last
+reference to object being referenced by the proxy. Patch by Pablo Galindo.
diff --git a/Objects/weakrefobject.c b/Objects/weakrefobject.c
index 4b81450de83f1..58fe09fa3b51b 100644
--- a/Objects/weakrefobject.c
+++ b/Objects/weakrefobject.c
@@ -145,11 +145,14 @@ weakref_hash(PyWeakReference *self)
 {
     if (self->hash != -1)
         return self->hash;
-    if (PyWeakref_GET_OBJECT(self) == Py_None) {
+    PyObject* obj = PyWeakref_GET_OBJECT(self);
+    if (obj == Py_None) {
         PyErr_SetString(PyExc_TypeError, "weak object has gone away");
         return -1;
     }
-    self->hash = PyObject_Hash(PyWeakref_GET_OBJECT(self));
+    Py_INCREF(obj);
+    self->hash = PyObject_Hash(obj);
+    Py_DECREF(obj);
     return self->hash;
 }
 
@@ -159,11 +162,15 @@ weakref_repr(PyWeakReference *self)
 {
     PyObject *name, *repr;
     _Py_IDENTIFIER(__name__);
+    PyObject* obj = PyWeakref_GET_OBJECT(self);
 
-    if (PyWeakref_GET_OBJECT(self) == Py_None)
+    if (obj == Py_None) {
         return PyUnicode_FromFormat("<weakref at %p; dead>", self);
+    }
 
-    if (_PyObject_LookupAttrId(PyWeakref_GET_OBJECT(self), &PyId___name__, &name) < 0) {
+    Py_INCREF(obj);
+    if (_PyObject_LookupAttrId(obj, &PyId___name__, &name) < 0) {
+        Py_DECREF(obj);
         return NULL;
     }
     if (name == NULL || !PyUnicode_Check(name)) {
@@ -171,16 +178,17 @@ weakref_repr(PyWeakReference *self)
             "<weakref at %p; to '%s' at %p>",
             self,
             Py_TYPE(PyWeakref_GET_OBJECT(self))->tp_name,
-            PyWeakref_GET_OBJECT(self));
+            obj);
     }
     else {
         repr = PyUnicode_FromFormat(
             "<weakref at %p; to '%s' at %p (%U)>",
             self,
             Py_TYPE(PyWeakref_GET_OBJECT(self))->tp_name,
-            PyWeakref_GET_OBJECT(self),
+            obj,
             name);
     }
+    Py_DECREF(obj);
     Py_XDECREF(name);
     return repr;
 }
@@ -207,8 +215,14 @@ weakref_richcompare(PyWeakReference* self, PyWeakReference* other, int op)
         else
             Py_RETURN_FALSE;
     }
-    return PyObject_RichCompare(PyWeakref_GET_OBJECT(self),
-                                PyWeakref_GET_OBJECT(other), op);
+    PyObject* obj = PyWeakref_GET_OBJECT(self);
+    PyObject* other_obj = PyWeakref_GET_OBJECT(other);
+    Py_INCREF(obj);
+    Py_INCREF(other_obj);
+    PyObject* res = PyObject_RichCompare(obj, other_obj, op);
+    Py_DECREF(obj);
+    Py_DECREF(other_obj);
+    return res;
 }
 
 /* Given the head of an object's list of weak references, extract the
@@ -415,18 +429,14 @@ proxy_checkref(PyWeakReference *proxy)
             o = PyWeakref_GET_OBJECT(o); \
         }
 
-#define UNWRAP_I(o) \
-        if (PyWeakref_CheckProxy(o)) { \
-            if (!proxy_checkref((PyWeakReference *)o)) \
-                return -1; \
-            o = PyWeakref_GET_OBJECT(o); \
-        }
-
 #define WRAP_UNARY(method, generic) \
     static PyObject * \
     method(PyObject *proxy) { \
         UNWRAP(proxy); \
-        return generic(proxy); \
+        Py_INCREF(proxy); \
+        PyObject* res = generic(proxy); \
+        Py_DECREF(proxy); \
+        return res; \
     }
 
 #define WRAP_BINARY(method, generic) \
@@ -434,7 +444,12 @@ proxy_checkref(PyWeakReference *proxy)
     method(PyObject *x, PyObject *y) { \
         UNWRAP(x); \
         UNWRAP(y); \
-        return generic(x, y); \
+        Py_INCREF(x); \
+        Py_INCREF(y); \
+        PyObject* res = generic(x, y); \
+        Py_DECREF(x); \
+        Py_DECREF(y); \
+        return res; \
     }
 
 /* Note that the third arg needs to be checked for NULL since the tp_call
@@ -447,7 +462,14 @@ proxy_checkref(PyWeakReference *proxy)
         UNWRAP(v); \
         if (w != NULL) \
             UNWRAP(w); \
-        return generic(proxy, v, w); \
+        Py_INCREF(proxy); \
+        Py_INCREF(v); \
+        Py_XINCREF(w); \
+        PyObject* res = generic(proxy, v, w); \
+        Py_DECREF(proxy); \
+        Py_DECREF(v); \
+        Py_XDECREF(w); \
+        return res; \
     }
 
 #define WRAP_METHOD(method, special) \
@@ -455,7 +477,10 @@ proxy_checkref(PyWeakReference *proxy)
     method(PyObject *proxy, PyObject *Py_UNUSED(ignored)) { \
             _Py_IDENTIFIER(special); \
             UNWRAP(proxy); \
-                return _PyObject_CallMethodId(proxy, &PyId_##special, NULL); \
+            Py_INCREF(proxy); \
+            PyObject* res = _PyObject_CallMethodId(proxy, &PyId_##special, NULL); \
+            Py_DECREF(proxy); \
+            return res; \
         }
 
 
@@ -481,7 +506,11 @@ proxy_setattr(PyWeakReference *proxy, PyObject *name, PyObject *value)
 {
     if (!proxy_checkref(proxy))
         return -1;
-    return PyObject_SetAttr(PyWeakref_GET_OBJECT(proxy), name, value);
+    PyObject *obj = PyWeakref_GET_OBJECT(proxy);
+    Py_INCREF(obj);
+    int res = PyObject_SetAttr(obj, name, value);
+    Py_DECREF(obj);
+    return res;
 }
 
 static PyObject *
@@ -532,9 +561,13 @@ static int
 proxy_bool(PyWeakReference *proxy)
 {
     PyObject *o = PyWeakref_GET_OBJECT(proxy);
-    if (!proxy_checkref(proxy))
+    if (!proxy_checkref(proxy)) {
         return -1;
-    return PyObject_IsTrue(o);
+    }
+    Py_INCREF(o);
+    int res = PyObject_IsTrue(o);
+    Py_DECREF(o);
+    return res;
 }
 
 static void
@@ -553,9 +586,13 @@ proxy_contains(PyWeakReference *proxy, PyObject *value)
 {
     if (!proxy_checkref(proxy))
         return -1;
-    return PySequence_Contains(PyWeakref_GET_OBJECT(proxy), value);
-}
 
+    PyObject *obj = PyWeakref_GET_OBJECT(proxy);
+    Py_INCREF(obj);
+    int res = PySequence_Contains(obj, value);
+    Py_DECREF(obj);
+    return res;
+}
 
 /* mapping slots */
 
@@ -564,7 +601,12 @@ proxy_length(PyWeakReference *proxy)
 {
     if (!proxy_checkref(proxy))
         return -1;
-    return PyObject_Length(PyWeakref_GET_OBJECT(proxy));
+
+    PyObject *obj = PyWeakref_GET_OBJECT(proxy);
+    Py_INCREF(obj);
+    Py_ssize_t res = PyObject_Length(obj);
+    Py_DECREF(obj);
+    return res;
 }
 
 WRAP_BINARY(proxy_getitem, PyObject_GetItem)
@@ -575,10 +617,16 @@ proxy_setitem(PyWeakReference *proxy, PyObject *key, PyObject *value)
     if (!proxy_checkref(proxy))
         return -1;
 
-    if (value == NULL)
-        return PyObject_DelItem(PyWeakref_GET_OBJECT(proxy), key);
-    else
-        return PyObject_SetItem(PyWeakref_GET_OBJECT(proxy), key, value);
+    PyObject *obj = PyWeakref_GET_OBJECT(proxy);
+    Py_INCREF(obj);
+    int res;
+    if (value == NULL) {
+        res = PyObject_DelItem(obj, key);
+    } else {
+        res = PyObject_SetItem(obj, key, value);
+    }
+    Py_DECREF(obj);
+    return res;
 }
 
 /* iterator slots */
@@ -588,7 +636,11 @@ proxy_iter(PyWeakReference *proxy)
 {
     if (!proxy_checkref(proxy))
         return NULL;
-    return PyObject_GetIter(PyWeakref_GET_OBJECT(proxy));
+    PyObject *obj = PyWeakref_GET_OBJECT(proxy);
+    Py_INCREF(obj);
+    PyObject* res = PyObject_GetIter(obj);
+    Py_DECREF(obj);
+    return res;
 }
 
 static PyObject *
@@ -596,7 +648,12 @@ proxy_iternext(PyWeakReference *proxy)
 {
     if (!proxy_checkref(proxy))
         return NULL;
-    return PyIter_Next(PyWeakref_GET_OBJECT(proxy));
+
+    PyObject *obj = PyWeakref_GET_OBJECT(proxy);
+    Py_INCREF(obj);
+    PyObject* res = PyIter_Next(obj);
+    Py_DECREF(obj);
+    return res;
 }
 
 


From webhook-mailer at python.org  Tue Oct  8 16:59:09 2019
From: webhook-mailer at python.org (Vinay Sajip)
Date: Tue, 08 Oct 2019 20:59:09 -0000
Subject: [Python-checkins] =?utf-8?q?bpo-38368=3A_Added_fix_for_ctypes_cr?=
 =?utf-8?q?ash_when_handling_arrays_in_structs=E2=80=A6_=28GH-16589=29?=
Message-ID: <mailman.61.1570568350.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/e8bedbddadaa86be6bd86dc32dbdbd53933a4988
commit: e8bedbddadaa86be6bd86dc32dbdbd53933a4988
branch: master
author: Vinay Sajip <vinay_sajip at yahoo.co.uk>
committer: GitHub <noreply at github.com>
date: 2019-10-08T21:59:06+01:00
summary:

bpo-38368: Added fix for ctypes crash when handling arrays in structs? (GH-16589)

files:
M Lib/ctypes/test/test_structures.py
M Modules/_ctypes/_ctypes_test.c
M Modules/_ctypes/stgdict.c

diff --git a/Lib/ctypes/test/test_structures.py b/Lib/ctypes/test/test_structures.py
index 0c01c799042cc..cdf4a9182439f 100644
--- a/Lib/ctypes/test/test_structures.py
+++ b/Lib/ctypes/test/test_structures.py
@@ -1,4 +1,5 @@
 import platform
+import sys
 import unittest
 from ctypes import *
 from ctypes.test import need_symbol
@@ -497,6 +498,16 @@ class Test3(Structure):
                 ('data', c_double * 2),
             ]
 
+        class Test3A(Structure):
+            _fields_ = [
+                ('data', c_float * 2),
+            ]
+
+        class Test3B(Test3A):
+            _fields_ = [
+                ('more_data', c_float * 2),
+            ]
+
         s = Test2()
         expected = 0
         for i in range(16):
@@ -525,6 +536,46 @@ class Test3(Structure):
         self.assertEqual(s.data[0], 3.14159)
         self.assertEqual(s.data[1], 2.71828)
 
+        s = Test3B()
+        s.data[0] = 3.14159
+        s.data[1] = 2.71828
+        s.more_data[0] = -3.0
+        s.more_data[1] = -2.0
+
+        expected = 3.14159 + 2.71828 - 5.0
+        func = dll._testfunc_array_in_struct2a
+        func.restype = c_double
+        func.argtypes = (Test3B,)
+        result = func(s)
+        self.assertAlmostEqual(result, expected, places=6)
+        # check the passed-in struct hasn't changed
+        self.assertAlmostEqual(s.data[0], 3.14159, places=6)
+        self.assertAlmostEqual(s.data[1], 2.71828, places=6)
+        self.assertAlmostEqual(s.more_data[0], -3.0, places=6)
+        self.assertAlmostEqual(s.more_data[1], -2.0, places=6)
+
+    def test_38368(self):
+        class U(Union):
+            _fields_ = [
+                ('f1', c_uint8 * 16),
+                ('f2', c_uint16 * 8),
+                ('f3', c_uint32 * 4),
+            ]
+        u = U()
+        u.f3[0] = 0x01234567
+        u.f3[1] = 0x89ABCDEF
+        u.f3[2] = 0x76543210
+        u.f3[3] = 0xFEDCBA98
+        f1 = [u.f1[i] for i in range(16)]
+        f2 = [u.f2[i] for i in range(8)]
+        if sys.byteorder == 'little':
+            self.assertEqual(f1, [0x67, 0x45, 0x23, 0x01,
+                                  0xef, 0xcd, 0xab, 0x89,
+                                  0x10, 0x32, 0x54, 0x76,
+                                  0x98, 0xba, 0xdc, 0xfe])
+            self.assertEqual(f2, [0x4567, 0x0123, 0xcdef, 0x89ab,
+                                  0x3210, 0x7654, 0xba98, 0xfedc])
+
 class PointerMemberTestCase(unittest.TestCase):
 
     def test(self):
diff --git a/Modules/_ctypes/_ctypes_test.c b/Modules/_ctypes/_ctypes_test.c
index 10f6591d24cd7..8a0e5e9195f7b 100644
--- a/Modules/_ctypes/_ctypes_test.c
+++ b/Modules/_ctypes/_ctypes_test.c
@@ -100,6 +100,11 @@ typedef struct {
     double data[2];
 } Test3;
 
+typedef struct {
+    float data[2];
+    float more_data[2];
+} Test3B;
+
 EXPORT(double)
 _testfunc_array_in_struct2(Test3 in)
 {
@@ -114,6 +119,22 @@ _testfunc_array_in_struct2(Test3 in)
     return result;
 }
 
+EXPORT(double)
+_testfunc_array_in_struct2a(Test3B in)
+{
+    double result = 0;
+
+    for (unsigned i = 0; i < 2; i++)
+        result += in.data[i];
+    for (unsigned i = 0; i < 2; i++)
+        result += in.more_data[i];
+    /* As the structure is passed by value, changes to it shouldn't be
+     * reflected in the caller.
+     */
+    memset(in.data, 0, sizeof(in.data));
+    return result;
+}
+
 EXPORT(void)testfunc_array(int values[4])
 {
     printf("testfunc_array %d %d %d %d\n",
diff --git a/Modules/_ctypes/stgdict.c b/Modules/_ctypes/stgdict.c
index b794f10a2cb71..97bcf5539aeea 100644
--- a/Modules/_ctypes/stgdict.c
+++ b/Modules/_ctypes/stgdict.c
@@ -644,9 +644,9 @@ PyCStructUnionType_update_stgdict(PyObject *type, PyObject *fields, int isStruct
     stgdict->align = total_align;
     stgdict->length = len;      /* ADD ffi_ofs? */
 
-#define MAX_ELEMENTS 16
+#define MAX_STRUCT_SIZE 16
 
-    if (arrays_seen && (size <= MAX_ELEMENTS)) {
+    if (arrays_seen && (size <= MAX_STRUCT_SIZE)) {
         /*
          * See bpo-22273. Arrays are normally treated as pointers, which is
          * fine when an array name is being passed as parameter, but not when
@@ -667,11 +667,138 @@ PyCStructUnionType_update_stgdict(PyObject *type, PyObject *fields, int isStruct
          * Although the passing in registers is specific to 64-bit Linux, the
          * array-in-struct vs. pointer problem is general. But we restrict the
          * type transformation to small structs nonetheless.
+         *
+         * Note that although a union may be small in terms of memory usage, it
+         * could contain many overlapping declarations of arrays, e.g.
+         *
+         * union {
+         *     unsigned int_8 foo [16];
+         *     unsigned uint_8 bar [16];
+         *     unsigned int_16 baz[8];
+         *     unsigned uint_16 bozz[8];
+         *     unsigned int_32 fizz[4];
+         *     unsigned uint_32 buzz[4];
+         * }
+         *
+         * which is still only 16 bytes in size. We need to convert this into
+         * the following equivalent for libffi:
+         *
+         * union {
+         *     struct { int_8 e1; int_8 e2; ... int_8 e_16; } f1;
+         *     struct { uint_8 e1; uint_8 e2; ... uint_8 e_16; } f2;
+         *     struct { int_16 e1; int_16 e2; ... int_16 e_8; } f3;
+         *     struct { uint_16 e1; uint_16 e2; ... uint_16 e_8; } f4;
+         *     struct { int_32 e1; int_32 e2; ... int_32 e_4; } f5;
+         *     struct { uint_32 e1; uint_32 e2; ... uint_32 e_4; } f6;
+         * }
+         *
+         * So the struct/union needs setting up as follows: all non-array
+         * elements copied across as is, and all array elements replaced with
+         * an equivalent struct which has as many fields as the array has
+         * elements, plus one NULL pointer.
+         */
+
+        Py_ssize_t num_ffi_type_pointers = 0;  /* for the dummy fields */
+        Py_ssize_t num_ffi_types = 0;  /* for the dummy structures */
+        size_t alloc_size;  /* total bytes to allocate */
+        void *type_block;  /* to hold all the type information needed */
+        ffi_type **element_types;  /* of this struct/union */
+        ffi_type **dummy_types;  /* of the dummy struct elements */
+        ffi_type *structs;  /* point to struct aliases of arrays */
+        Py_ssize_t element_index;  /* index into element_types for this */
+        Py_ssize_t dummy_index = 0; /* index into dummy field pointers */
+        Py_ssize_t struct_index = 0; /* index into dummy structs */
+
+        /* first pass to see how much memory to allocate */
+        for (i = 0; i < len; ++i) {
+            PyObject *name, *desc;
+            PyObject *pair = PySequence_GetItem(fields, i);
+            StgDictObject *dict;
+            int bitsize = 0;
+
+            if (pair == NULL) {
+                return -1;
+            }
+            if (!PyArg_ParseTuple(pair, "UO|i", &name, &desc, &bitsize)) {
+                PyErr_SetString(PyExc_TypeError,
+                    "'_fields_' must be a sequence of (name, C type) pairs");
+                Py_DECREF(pair);
+                return -1;
+            }
+            dict = PyType_stgdict(desc);
+            if (dict == NULL) {
+                Py_DECREF(pair);
+                PyErr_Format(PyExc_TypeError,
+                    "second item in _fields_ tuple (index %zd) must be a C type",
+                    i);
+                return -1;
+            }
+            if (!PyCArrayTypeObject_Check(desc)) {
+                /* Not an array. Just need an ffi_type pointer. */
+                num_ffi_type_pointers++;
+            }
+            else {
+                /* It's an array. */
+                Py_ssize_t length = dict->length;
+                StgDictObject *edict;
+
+                edict = PyType_stgdict(dict->proto);
+                if (edict == NULL) {
+                    Py_DECREF(pair);
+                    PyErr_Format(PyExc_TypeError,
+                        "second item in _fields_ tuple (index %zd) must be a C type",
+                        i);
+                    return -1;
+                }
+                /*
+                 * We need one extra ffi_type to hold the struct, and one
+                 * ffi_type pointer per array element + one for a NULL to
+                 * mark the end.
+                 */
+                num_ffi_types++;
+                num_ffi_type_pointers += length + 1;
+            }
+            Py_DECREF(pair);
+        }
+
+        /*
+         * At this point, we know we need storage for some ffi_types and some
+         * ffi_type pointers. We'll allocate these in one block.
+         * There are three sub-blocks of information: the ffi_type pointers to
+         * this structure/union's elements, the ffi_type_pointers to the
+         * dummy fields standing in for array elements, and the
+         * ffi_types representing the dummy structures.
          */
-        ffi_type *actual_types[MAX_ELEMENTS + 1];
-        int actual_type_index = 0;
+        alloc_size = (ffi_ofs + 1 + len + num_ffi_type_pointers) * sizeof(ffi_type *) +
+                        num_ffi_types * sizeof(ffi_type);
+        type_block = PyMem_Malloc(alloc_size);
 
-        memset(actual_types, 0, sizeof(actual_types));
+        if (type_block == NULL) {
+            PyErr_NoMemory();
+            return -1;
+        }
+        /*
+         * the first block takes up ffi_ofs + len + 1 which is the pointers *
+         * for this struct/union. The second block takes up
+         * num_ffi_type_pointers, so the sum of these is ffi_ofs + len + 1 +
+         * num_ffi_type_pointers as allocated above. The last bit is the
+         * num_ffi_types structs.
+         */
+        element_types = (ffi_type **) type_block;
+        dummy_types = &element_types[ffi_ofs + len + 1];
+        structs = (ffi_type *) &dummy_types[num_ffi_type_pointers];
+
+        if (num_ffi_types > 0) {
+            memset(structs, 0, num_ffi_types * sizeof(ffi_type));
+        }
+        if (ffi_ofs && (basedict != NULL)) {
+            memcpy(element_types,
+                basedict->ffi_type_pointer.elements,
+                ffi_ofs * sizeof(ffi_type *));
+        }
+        element_index = ffi_ofs;
+
+        /* second pass to actually set the type pointers */
         for (i = 0; i < len; ++i) {
             PyObject *name, *desc;
             PyObject *pair = PySequence_GetItem(fields, i);
@@ -679,26 +806,36 @@ PyCStructUnionType_update_stgdict(PyObject *type, PyObject *fields, int isStruct
             int bitsize = 0;
 
             if (pair == NULL) {
+                PyMem_Free(type_block);
                 return -1;
             }
+            /* In theory, we made this call in the first pass, so it *shouldn't*
+             * fail. However, you never know, and the code above might change
+             * later - keeping the check in here is a tad defensive but it
+             * will affect program size only slightly and performance hardly at
+             * all.
+             */
             if (!PyArg_ParseTuple(pair, "UO|i", &name, &desc, &bitsize)) {
                 PyErr_SetString(PyExc_TypeError,
                                 "'_fields_' must be a sequence of (name, C type) pairs");
-                Py_XDECREF(pair);
+                Py_DECREF(pair);
+                PyMem_Free(type_block);
                 return -1;
             }
             dict = PyType_stgdict(desc);
+            /* Possibly this check could be avoided, but see above comment. */
             if (dict == NULL) {
                 Py_DECREF(pair);
+                PyMem_Free(type_block);
                 PyErr_Format(PyExc_TypeError,
                              "second item in _fields_ tuple (index %zd) must be a C type",
                              i);
                 return -1;
             }
+            assert(element_index < (ffi_ofs + len)); /* will be used below */
             if (!PyCArrayTypeObject_Check(desc)) {
                 /* Not an array. Just copy over the element ffi_type. */
-                actual_types[actual_type_index++] = &dict->ffi_type_pointer;
-                assert(actual_type_index <= MAX_ELEMENTS);
+                element_types[element_index++] = &dict->ffi_type_pointer;
             }
             else {
                 Py_ssize_t length = dict->length;
@@ -707,42 +844,38 @@ PyCStructUnionType_update_stgdict(PyObject *type, PyObject *fields, int isStruct
                 edict = PyType_stgdict(dict->proto);
                 if (edict == NULL) {
                     Py_DECREF(pair);
+                    PyMem_Free(type_block);
                     PyErr_Format(PyExc_TypeError,
                                  "second item in _fields_ tuple (index %zd) must be a C type",
                                  i);
                     return -1;
                 }
+                element_types[element_index++] = &structs[struct_index];
+                structs[struct_index].size = length * edict->ffi_type_pointer.size;
+                structs[struct_index].alignment = edict->ffi_type_pointer.alignment;
+                structs[struct_index].type = FFI_TYPE_STRUCT;
+                structs[struct_index].elements = &dummy_types[dummy_index];
+                ++struct_index;
                 /* Copy over the element's type, length times. */
                 while (length > 0) {
-                    actual_types[actual_type_index++] = &edict->ffi_type_pointer;
-                    assert(actual_type_index <= MAX_ELEMENTS);
+                    assert(dummy_index < (num_ffi_type_pointers));
+                    dummy_types[dummy_index++] = &edict->ffi_type_pointer;
                     length--;
                 }
+                assert(dummy_index < (num_ffi_type_pointers));
+                dummy_types[dummy_index++] = NULL;
             }
             Py_DECREF(pair);
         }
 
-        actual_types[actual_type_index++] = NULL;
+        element_types[element_index] = NULL;
         /*
          * Replace the old elements with the new, taking into account
          * base class elements where necessary.
          */
         assert(stgdict->ffi_type_pointer.elements);
         PyMem_Free(stgdict->ffi_type_pointer.elements);
-        stgdict->ffi_type_pointer.elements = PyMem_New(ffi_type *,
-                                                       ffi_ofs + actual_type_index);
-        if (stgdict->ffi_type_pointer.elements == NULL) {
-            PyErr_NoMemory();
-            return -1;
-        }
-        if (ffi_ofs) {
-            memcpy(stgdict->ffi_type_pointer.elements,
-                   basedict->ffi_type_pointer.elements,
-                   ffi_ofs * sizeof(ffi_type *));
-
-        }
-        memcpy(&stgdict->ffi_type_pointer.elements[ffi_ofs], actual_types,
-               actual_type_index * sizeof(ffi_type *));
+        stgdict->ffi_type_pointer.elements = element_types;
     }
 
     /* We did check that this flag was NOT set above, it must not


From webhook-mailer at python.org  Tue Oct  8 22:19:57 2019
From: webhook-mailer at python.org (Benjamin Peterson)
Date: Wed, 09 Oct 2019 02:19:57 -0000
Subject: [Python-checkins] closes bpo-36161: Use thread-safe ttyname_r
 instead of ttyname. (GH-14868)
Message-ID: <mailman.62.1570587598.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/594e2edfb5e0d24e03469d035d8f39ff29a64d99
commit: 594e2edfb5e0d24e03469d035d8f39ff29a64d99
branch: master
author: Antonio Gutierrez <chibby0ne at gmail.com>
committer: Benjamin Peterson <benjamin at python.org>
date: 2019-10-08T19:19:48-07:00
summary:

closes bpo-36161: Use thread-safe ttyname_r instead of ttyname. (GH-14868)

Signed-off-by: Antonio Gutierrez <chibby0ne at gmail.com>

files:
A Misc/NEWS.d/next/Library/2019-07-20-01-17-43.bpo-36161.Fzf-f9.rst
M Modules/posixmodule.c

diff --git a/Misc/NEWS.d/next/Library/2019-07-20-01-17-43.bpo-36161.Fzf-f9.rst b/Misc/NEWS.d/next/Library/2019-07-20-01-17-43.bpo-36161.Fzf-f9.rst
new file mode 100644
index 0000000000000..b31496c701744
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-07-20-01-17-43.bpo-36161.Fzf-f9.rst
@@ -0,0 +1 @@
+In :mod:`posix`, use ``ttyname_r`` instead of ``ttyname`` for thread safety.
diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
index 5664027b2d3a3..aeb0e9ddb1953 100644
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -2774,13 +2774,24 @@ static PyObject *
 os_ttyname_impl(PyObject *module, int fd)
 /*[clinic end generated code: output=c424d2e9d1cd636a input=9ff5a58b08115c55]*/
 {
-    char *ret;
 
-    ret = ttyname(fd);
-    if (ret == NULL) {
+    long size = sysconf(_SC_TTY_NAME_MAX);
+    if (size == -1) {
+        return posix_error();
+    }
+    char *buffer = (char *)PyMem_RawMalloc(size);
+    if (buffer == NULL) {
+        return PyErr_NoMemory();
+    }
+    int ret = ttyname_r(fd, buffer, size);
+    if (ret != 0) {
+        PyMem_RawFree(buffer);
+        errno = ret;
         return posix_error();
     }
-    return PyUnicode_DecodeFSDefault(ret);
+    PyObject *res = PyUnicode_DecodeFSDefault(buffer);
+    PyMem_RawFree(buffer);
+    return res;
 }
 #endif
 


From webhook-mailer at python.org  Tue Oct  8 22:24:06 2019
From: webhook-mailer at python.org (Benjamin Peterson)
Date: Wed, 09 Oct 2019 02:24:06 -0000
Subject: [Python-checkins] Typo fix: "empy" should be "empty". (GH-16666)
Message-ID: <mailman.63.1570587848.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/01171ebd966b0cd6352057799ad876dd1e07942e
commit: 01171ebd966b0cd6352057799ad876dd1e07942e
branch: master
author: Hansraj Das <raj.das.136 at gmail.com>
committer: Benjamin Peterson <benjamin at python.org>
date: 2019-10-08T19:24:02-07:00
summary:

Typo fix: "empy" should be "empty". (GH-16666)

files:
M Python/compile.c

diff --git a/Python/compile.c b/Python/compile.c
index edd8625cba91c..3b2188e6e7438 100644
--- a/Python/compile.c
+++ b/Python/compile.c
@@ -1751,7 +1751,7 @@ compiler_body(struct compiler *c, asdl_seq *stmts)
     /* Set current line number to the line number of first statement.
        This way line number for SETUP_ANNOTATIONS will always
        coincide with the line number of first "real" statement in module.
-       If body is empy, then lineno will be set later in assemble. */
+       If body is empty, then lineno will be set later in assemble. */
     if (c->u->u_scope_type == COMPILER_SCOPE_MODULE &&
         !c->u->u_lineno && asdl_seq_LEN(stmts)) {
         st = (stmt_ty)asdl_seq_GET(stmts, 0);


From webhook-mailer at python.org  Tue Oct  8 22:42:19 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Wed, 09 Oct 2019 02:42:19 -0000
Subject: [Python-checkins] Typo fix: "empy" should be "empty". (GH-16666)
Message-ID: <mailman.64.1570588940.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/443370d8acd107da235d2e9758e06ab3583be4ea
commit: 443370d8acd107da235d2e9758e06ab3583be4ea
branch: 3.7
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-08T19:42:15-07:00
summary:

Typo fix: "empy" should be "empty". (GH-16666)

(cherry picked from commit 01171ebd966b0cd6352057799ad876dd1e07942e)

Co-authored-by: Hansraj Das <raj.das.136 at gmail.com>

files:
M Python/compile.c

diff --git a/Python/compile.c b/Python/compile.c
index 5688ef8479c0d..32884be34de3c 100644
--- a/Python/compile.c
+++ b/Python/compile.c
@@ -1482,7 +1482,7 @@ compiler_body(struct compiler *c, asdl_seq *stmts)
     /* Set current line number to the line number of first statement.
        This way line number for SETUP_ANNOTATIONS will always
        coincide with the line number of first "real" statement in module.
-       If body is empy, then lineno will be set later in assemble. */
+       If body is empty, then lineno will be set later in assemble. */
     if (c->u->u_scope_type == COMPILER_SCOPE_MODULE &&
         !c->u->u_lineno && asdl_seq_LEN(stmts)) {
         st = (stmt_ty)asdl_seq_GET(stmts, 0);


From webhook-mailer at python.org  Tue Oct  8 22:42:26 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Wed, 09 Oct 2019 02:42:26 -0000
Subject: [Python-checkins] Typo fix: "empy" should be "empty". (GH-16666)
Message-ID: <mailman.65.1570588946.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/d8071cbf5bd0c276bae41b8505da22ad29d1be51
commit: d8071cbf5bd0c276bae41b8505da22ad29d1be51
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-08T19:42:22-07:00
summary:

Typo fix: "empy" should be "empty". (GH-16666)

(cherry picked from commit 01171ebd966b0cd6352057799ad876dd1e07942e)

Co-authored-by: Hansraj Das <raj.das.136 at gmail.com>

files:
M Python/compile.c

diff --git a/Python/compile.c b/Python/compile.c
index a6c5ab6324f0d..a512e9cbd13a1 100644
--- a/Python/compile.c
+++ b/Python/compile.c
@@ -1749,7 +1749,7 @@ compiler_body(struct compiler *c, asdl_seq *stmts)
     /* Set current line number to the line number of first statement.
        This way line number for SETUP_ANNOTATIONS will always
        coincide with the line number of first "real" statement in module.
-       If body is empy, then lineno will be set later in assemble. */
+       If body is empty, then lineno will be set later in assemble. */
     if (c->u->u_scope_type == COMPILER_SCOPE_MODULE &&
         !c->u->u_lineno && asdl_seq_LEN(stmts)) {
         st = (stmt_ty)asdl_seq_GET(stmts, 0);


From webhook-mailer at python.org  Wed Oct  9 01:34:32 2019
From: webhook-mailer at python.org (larryhastings)
Date: Wed, 09 Oct 2019 05:34:32 -0000
Subject: [Python-checkins] closes bpo-38174: Update vendored expat library
 to 2.2.8. (GH-16346) (#16434)
Message-ID: <mailman.66.1570599274.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/c386c8b06c6e92786f083ef6aba27b37087fdd20
commit: c386c8b06c6e92786f083ef6aba27b37087fdd20
branch: 3.5
author: Victor Stinner <vstinner at redhat.com>
committer: larryhastings <larry at hastings.org>
date: 2019-10-09T06:34:26+01:00
summary:

closes bpo-38174: Update vendored expat library to 2.2.8. (GH-16346) (#16434)

Fixes CVE-2019-15903. See full changelog at https://github.com/libexpat/libexpat/blob/R_2_2_8/expat/Changes.

files:
A Misc/NEWS.d/next/Security/2019-09-23-21-02-46.bpo-38174.MeWuJd.rst
D Modules/expat/loadlibrary.c
M Modules/expat/asciitab.h
M Modules/expat/expat.h
M Modules/expat/expat_external.h
M Modules/expat/iasciitab.h
M Modules/expat/internal.h
M Modules/expat/latin1tab.h
M Modules/expat/nametab.h
M Modules/expat/siphash.h
M Modules/expat/utf8tab.h
M Modules/expat/winconfig.h
M Modules/expat/xmlparse.c
M Modules/expat/xmlrole.c
M Modules/expat/xmlrole.h
M Modules/expat/xmltok.c
M Modules/expat/xmltok.h
M Modules/expat/xmltok_impl.c
M Modules/expat/xmltok_impl.h
M Modules/expat/xmltok_ns.c
M PCbuild/_elementtree.vcxproj
M PCbuild/pyexpat.vcxproj

diff --git a/Misc/NEWS.d/next/Security/2019-09-23-21-02-46.bpo-38174.MeWuJd.rst b/Misc/NEWS.d/next/Security/2019-09-23-21-02-46.bpo-38174.MeWuJd.rst
new file mode 100644
index 0000000000000..e3da6fcdd139f
--- /dev/null
+++ b/Misc/NEWS.d/next/Security/2019-09-23-21-02-46.bpo-38174.MeWuJd.rst
@@ -0,0 +1,2 @@
+Update vendorized expat library version to 2.2.8, which resolves
+CVE-2019-15903.
diff --git a/Modules/expat/asciitab.h b/Modules/expat/asciitab.h
index 2f59fd9290653..63b1d1b4482ef 100644
--- a/Modules/expat/asciitab.h
+++ b/Modules/expat/asciitab.h
@@ -31,34 +31,34 @@
 */
 
 /* 0x00 */ BT_NONXML, BT_NONXML, BT_NONXML, BT_NONXML,
-/* 0x04 */ BT_NONXML, BT_NONXML, BT_NONXML, BT_NONXML,
-/* 0x08 */ BT_NONXML, BT_S, BT_LF, BT_NONXML,
-/* 0x0C */ BT_NONXML, BT_CR, BT_NONXML, BT_NONXML,
-/* 0x10 */ BT_NONXML, BT_NONXML, BT_NONXML, BT_NONXML,
-/* 0x14 */ BT_NONXML, BT_NONXML, BT_NONXML, BT_NONXML,
-/* 0x18 */ BT_NONXML, BT_NONXML, BT_NONXML, BT_NONXML,
-/* 0x1C */ BT_NONXML, BT_NONXML, BT_NONXML, BT_NONXML,
-/* 0x20 */ BT_S, BT_EXCL, BT_QUOT, BT_NUM,
-/* 0x24 */ BT_OTHER, BT_PERCNT, BT_AMP, BT_APOS,
-/* 0x28 */ BT_LPAR, BT_RPAR, BT_AST, BT_PLUS,
-/* 0x2C */ BT_COMMA, BT_MINUS, BT_NAME, BT_SOL,
-/* 0x30 */ BT_DIGIT, BT_DIGIT, BT_DIGIT, BT_DIGIT,
-/* 0x34 */ BT_DIGIT, BT_DIGIT, BT_DIGIT, BT_DIGIT,
-/* 0x38 */ BT_DIGIT, BT_DIGIT, BT_COLON, BT_SEMI,
-/* 0x3C */ BT_LT, BT_EQUALS, BT_GT, BT_QUEST,
-/* 0x40 */ BT_OTHER, BT_HEX, BT_HEX, BT_HEX,
-/* 0x44 */ BT_HEX, BT_HEX, BT_HEX, BT_NMSTRT,
-/* 0x48 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
-/* 0x4C */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
-/* 0x50 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
-/* 0x54 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
-/* 0x58 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_LSQB,
-/* 0x5C */ BT_OTHER, BT_RSQB, BT_OTHER, BT_NMSTRT,
-/* 0x60 */ BT_OTHER, BT_HEX, BT_HEX, BT_HEX,
-/* 0x64 */ BT_HEX, BT_HEX, BT_HEX, BT_NMSTRT,
-/* 0x68 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
-/* 0x6C */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
-/* 0x70 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
-/* 0x74 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
-/* 0x78 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_OTHER,
-/* 0x7C */ BT_VERBAR, BT_OTHER, BT_OTHER, BT_OTHER,
+    /* 0x04 */ BT_NONXML, BT_NONXML, BT_NONXML, BT_NONXML,
+    /* 0x08 */ BT_NONXML, BT_S, BT_LF, BT_NONXML,
+    /* 0x0C */ BT_NONXML, BT_CR, BT_NONXML, BT_NONXML,
+    /* 0x10 */ BT_NONXML, BT_NONXML, BT_NONXML, BT_NONXML,
+    /* 0x14 */ BT_NONXML, BT_NONXML, BT_NONXML, BT_NONXML,
+    /* 0x18 */ BT_NONXML, BT_NONXML, BT_NONXML, BT_NONXML,
+    /* 0x1C */ BT_NONXML, BT_NONXML, BT_NONXML, BT_NONXML,
+    /* 0x20 */ BT_S, BT_EXCL, BT_QUOT, BT_NUM,
+    /* 0x24 */ BT_OTHER, BT_PERCNT, BT_AMP, BT_APOS,
+    /* 0x28 */ BT_LPAR, BT_RPAR, BT_AST, BT_PLUS,
+    /* 0x2C */ BT_COMMA, BT_MINUS, BT_NAME, BT_SOL,
+    /* 0x30 */ BT_DIGIT, BT_DIGIT, BT_DIGIT, BT_DIGIT,
+    /* 0x34 */ BT_DIGIT, BT_DIGIT, BT_DIGIT, BT_DIGIT,
+    /* 0x38 */ BT_DIGIT, BT_DIGIT, BT_COLON, BT_SEMI,
+    /* 0x3C */ BT_LT, BT_EQUALS, BT_GT, BT_QUEST,
+    /* 0x40 */ BT_OTHER, BT_HEX, BT_HEX, BT_HEX,
+    /* 0x44 */ BT_HEX, BT_HEX, BT_HEX, BT_NMSTRT,
+    /* 0x48 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
+    /* 0x4C */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
+    /* 0x50 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
+    /* 0x54 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
+    /* 0x58 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_LSQB,
+    /* 0x5C */ BT_OTHER, BT_RSQB, BT_OTHER, BT_NMSTRT,
+    /* 0x60 */ BT_OTHER, BT_HEX, BT_HEX, BT_HEX,
+    /* 0x64 */ BT_HEX, BT_HEX, BT_HEX, BT_NMSTRT,
+    /* 0x68 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
+    /* 0x6C */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
+    /* 0x70 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
+    /* 0x74 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
+    /* 0x78 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_OTHER,
+    /* 0x7C */ BT_VERBAR, BT_OTHER, BT_OTHER, BT_OTHER,
diff --git a/Modules/expat/expat.h b/Modules/expat/expat.h
index d0735bb5c6167..6c8eb1fda4a30 100644
--- a/Modules/expat/expat.h
+++ b/Modules/expat/expat.h
@@ -33,15 +33,6 @@
 #ifndef Expat_INCLUDED
 #define Expat_INCLUDED 1
 
-#ifdef __VMS
-/*      0        1         2         3      0        1         2         3
-        1234567890123456789012345678901     1234567890123456789012345678901 */
-#define XML_SetProcessingInstructionHandler XML_SetProcessingInstrHandler
-#define XML_SetUnparsedEntityDeclHandler    XML_SetUnparsedEntDeclHandler
-#define XML_SetStartNamespaceDeclHandler    XML_SetStartNamespcDeclHandler
-#define XML_SetExternalEntityRefHandlerArg  XML_SetExternalEntRefHandlerArg
-#endif
-
 #include <stdlib.h>
 #include "expat_external.h"
 
@@ -53,8 +44,8 @@ struct XML_ParserStruct;
 typedef struct XML_ParserStruct *XML_Parser;
 
 typedef unsigned char XML_Bool;
-#define XML_TRUE   ((XML_Bool) 1)
-#define XML_FALSE  ((XML_Bool) 0)
+#define XML_TRUE ((XML_Bool)1)
+#define XML_FALSE ((XML_Bool)0)
 
 /* The XML_Status enum gives the possible return values for several
    API functions.  The preprocessor #defines are included so this
@@ -164,25 +155,23 @@ enum XML_Content_Quant {
 typedef struct XML_cp XML_Content;
 
 struct XML_cp {
-  enum XML_Content_Type         type;
-  enum XML_Content_Quant        quant;
-  XML_Char *                    name;
-  unsigned int                  numchildren;
-  XML_Content *                 children;
+  enum XML_Content_Type type;
+  enum XML_Content_Quant quant;
+  XML_Char *name;
+  unsigned int numchildren;
+  XML_Content *children;
 };
 
-
 /* This is called for an element declaration. See above for
    description of the model argument. It's the caller's responsibility
    to free model when finished with it.
 */
-typedef void (XMLCALL *XML_ElementDeclHandler) (void *userData,
-                                                const XML_Char *name,
-                                                XML_Content *model);
+typedef void(XMLCALL *XML_ElementDeclHandler)(void *userData,
+                                              const XML_Char *name,
+                                              XML_Content *model);
 
 XMLPARSEAPI(void)
-XML_SetElementDeclHandler(XML_Parser parser,
-                          XML_ElementDeclHandler eldecl);
+XML_SetElementDeclHandler(XML_Parser parser, XML_ElementDeclHandler eldecl);
 
 /* The Attlist declaration handler is called for *each* attribute. So
    a single Attlist declaration with multiple attributes declared will
@@ -192,17 +181,12 @@ XML_SetElementDeclHandler(XML_Parser parser,
    value will be NULL in the case of "#REQUIRED". If "isrequired" is
    true and default is non-NULL, then this is a "#FIXED" default.
 */
-typedef void (XMLCALL *XML_AttlistDeclHandler) (
-                                    void            *userData,
-                                    const XML_Char  *elname,
-                                    const XML_Char  *attname,
-                                    const XML_Char  *att_type,
-                                    const XML_Char  *dflt,
-                                    int              isrequired);
+typedef void(XMLCALL *XML_AttlistDeclHandler)(
+    void *userData, const XML_Char *elname, const XML_Char *attname,
+    const XML_Char *att_type, const XML_Char *dflt, int isrequired);
 
 XMLPARSEAPI(void)
-XML_SetAttlistDeclHandler(XML_Parser parser,
-                          XML_AttlistDeclHandler attdecl);
+XML_SetAttlistDeclHandler(XML_Parser parser, XML_AttlistDeclHandler attdecl);
 
 /* The XML declaration handler is called for *both* XML declarations
    and text declarations. The way to distinguish is that the version
@@ -212,15 +196,13 @@ XML_SetAttlistDeclHandler(XML_Parser parser,
    was no standalone parameter in the declaration, that it was given
    as no, or that it was given as yes.
 */
-typedef void (XMLCALL *XML_XmlDeclHandler) (void           *userData,
-                                            const XML_Char *version,
-                                            const XML_Char *encoding,
-                                            int             standalone);
+typedef void(XMLCALL *XML_XmlDeclHandler)(void *userData,
+                                          const XML_Char *version,
+                                          const XML_Char *encoding,
+                                          int standalone);
 
 XMLPARSEAPI(void)
-XML_SetXmlDeclHandler(XML_Parser parser,
-                      XML_XmlDeclHandler xmldecl);
-
+XML_SetXmlDeclHandler(XML_Parser parser, XML_XmlDeclHandler xmldecl);
 
 typedef struct {
   void *(*malloc_fcn)(size_t size);
@@ -248,7 +230,6 @@ XML_ParserCreate(const XML_Char *encoding);
 XMLPARSEAPI(XML_Parser)
 XML_ParserCreateNS(const XML_Char *encoding, XML_Char namespaceSeparator);
 
-
 /* Constructs a new parser using the memory management suite referred to
    by memsuite. If memsuite is NULL, then use the standard library memory
    suite. If namespaceSeparator is non-NULL it creates a parser with
@@ -264,7 +245,7 @@ XML_ParserCreate_MM(const XML_Char *encoding,
                     const XML_Char *namespaceSeparator);
 
 /* Prepare a parser object to be re-used.  This is particularly
-   valuable when memory allocation overhead is disproportionatly high,
+   valuable when memory allocation overhead is disproportionately high,
    such as when a large number of small documnents need to be parsed.
    All handlers are cleared from the parser, except for the
    unknownEncodingHandler. The parser's external state is re-initialized
@@ -278,31 +259,27 @@ XML_ParserReset(XML_Parser parser, const XML_Char *encoding);
 /* atts is array of name/value pairs, terminated by 0;
    names and values are 0 terminated.
 */
-typedef void (XMLCALL *XML_StartElementHandler) (void *userData,
-                                                 const XML_Char *name,
-                                                 const XML_Char **atts);
-
-typedef void (XMLCALL *XML_EndElementHandler) (void *userData,
-                                               const XML_Char *name);
+typedef void(XMLCALL *XML_StartElementHandler)(void *userData,
+                                               const XML_Char *name,
+                                               const XML_Char **atts);
 
+typedef void(XMLCALL *XML_EndElementHandler)(void *userData,
+                                             const XML_Char *name);
 
 /* s is not 0 terminated. */
-typedef void (XMLCALL *XML_CharacterDataHandler) (void *userData,
-                                                  const XML_Char *s,
-                                                  int len);
+typedef void(XMLCALL *XML_CharacterDataHandler)(void *userData,
+                                                const XML_Char *s, int len);
 
 /* target and data are 0 terminated */
-typedef void (XMLCALL *XML_ProcessingInstructionHandler) (
-                                                void *userData,
-                                                const XML_Char *target,
-                                                const XML_Char *data);
+typedef void(XMLCALL *XML_ProcessingInstructionHandler)(void *userData,
+                                                        const XML_Char *target,
+                                                        const XML_Char *data);
 
 /* data is 0 terminated */
-typedef void (XMLCALL *XML_CommentHandler) (void *userData,
-                                            const XML_Char *data);
+typedef void(XMLCALL *XML_CommentHandler)(void *userData, const XML_Char *data);
 
-typedef void (XMLCALL *XML_StartCdataSectionHandler) (void *userData);
-typedef void (XMLCALL *XML_EndCdataSectionHandler) (void *userData);
+typedef void(XMLCALL *XML_StartCdataSectionHandler)(void *userData);
+typedef void(XMLCALL *XML_EndCdataSectionHandler)(void *userData);
 
 /* This is called for any characters in the XML document for which
    there is no applicable handler.  This includes both characters that
@@ -317,25 +294,23 @@ typedef void (XMLCALL *XML_EndCdataSectionHandler) (void *userData);
    default handler: for example, a comment might be split between
    multiple calls.
 */
-typedef void (XMLCALL *XML_DefaultHandler) (void *userData,
-                                            const XML_Char *s,
-                                            int len);
+typedef void(XMLCALL *XML_DefaultHandler)(void *userData, const XML_Char *s,
+                                          int len);
 
 /* This is called for the start of the DOCTYPE declaration, before
    any DTD or internal subset is parsed.
 */
-typedef void (XMLCALL *XML_StartDoctypeDeclHandler) (
-                                            void *userData,
-                                            const XML_Char *doctypeName,
-                                            const XML_Char *sysid,
-                                            const XML_Char *pubid,
-                                            int has_internal_subset);
+typedef void(XMLCALL *XML_StartDoctypeDeclHandler)(void *userData,
+                                                   const XML_Char *doctypeName,
+                                                   const XML_Char *sysid,
+                                                   const XML_Char *pubid,
+                                                   int has_internal_subset);
 
 /* This is called for the start of the DOCTYPE declaration when the
    closing > is encountered, but after processing any external
    subset.
 */
-typedef void (XMLCALL *XML_EndDoctypeDeclHandler)(void *userData);
+typedef void(XMLCALL *XML_EndDoctypeDeclHandler)(void *userData);
 
 /* This is called for entity declarations. The is_parameter_entity
    argument will be non-zero if the entity is a parameter entity, zero
@@ -355,20 +330,14 @@ typedef void (XMLCALL *XML_EndDoctypeDeclHandler)(void *userData);
    Note that is_parameter_entity can't be changed to XML_Bool, since
    that would break binary compatibility.
 */
-typedef void (XMLCALL *XML_EntityDeclHandler) (
-                              void *userData,
-                              const XML_Char *entityName,
-                              int is_parameter_entity,
-                              const XML_Char *value,
-                              int value_length,
-                              const XML_Char *base,
-                              const XML_Char *systemId,
-                              const XML_Char *publicId,
-                              const XML_Char *notationName);
+typedef void(XMLCALL *XML_EntityDeclHandler)(
+    void *userData, const XML_Char *entityName, int is_parameter_entity,
+    const XML_Char *value, int value_length, const XML_Char *base,
+    const XML_Char *systemId, const XML_Char *publicId,
+    const XML_Char *notationName);
 
 XMLPARSEAPI(void)
-XML_SetEntityDeclHandler(XML_Parser parser,
-                         XML_EntityDeclHandler handler);
+XML_SetEntityDeclHandler(XML_Parser parser, XML_EntityDeclHandler handler);
 
 /* OBSOLETE -- OBSOLETE -- OBSOLETE
    This handler has been superseded by the EntityDeclHandler above.
@@ -379,24 +348,20 @@ XML_SetEntityDeclHandler(XML_Parser parser,
    entityName, systemId and notationName arguments will never be
    NULL. The other arguments may be.
 */
-typedef void (XMLCALL *XML_UnparsedEntityDeclHandler) (
-                                    void *userData,
-                                    const XML_Char *entityName,
-                                    const XML_Char *base,
-                                    const XML_Char *systemId,
-                                    const XML_Char *publicId,
-                                    const XML_Char *notationName);
+typedef void(XMLCALL *XML_UnparsedEntityDeclHandler)(
+    void *userData, const XML_Char *entityName, const XML_Char *base,
+    const XML_Char *systemId, const XML_Char *publicId,
+    const XML_Char *notationName);
 
 /* This is called for a declaration of notation.  The base argument is
    whatever was set by XML_SetBase. The notationName will never be
    NULL.  The other arguments can be.
 */
-typedef void (XMLCALL *XML_NotationDeclHandler) (
-                                    void *userData,
-                                    const XML_Char *notationName,
-                                    const XML_Char *base,
-                                    const XML_Char *systemId,
-                                    const XML_Char *publicId);
+typedef void(XMLCALL *XML_NotationDeclHandler)(void *userData,
+                                               const XML_Char *notationName,
+                                               const XML_Char *base,
+                                               const XML_Char *systemId,
+                                               const XML_Char *publicId);
 
 /* When namespace processing is enabled, these are called once for
    each namespace declaration. The call to the start and end element
@@ -404,14 +369,12 @@ typedef void (XMLCALL *XML_NotationDeclHandler) (
    declaration handlers. For an xmlns attribute, prefix will be
    NULL.  For an xmlns="" attribute, uri will be NULL.
 */
-typedef void (XMLCALL *XML_StartNamespaceDeclHandler) (
-                                    void *userData,
-                                    const XML_Char *prefix,
-                                    const XML_Char *uri);
+typedef void(XMLCALL *XML_StartNamespaceDeclHandler)(void *userData,
+                                                     const XML_Char *prefix,
+                                                     const XML_Char *uri);
 
-typedef void (XMLCALL *XML_EndNamespaceDeclHandler) (
-                                    void *userData,
-                                    const XML_Char *prefix);
+typedef void(XMLCALL *XML_EndNamespaceDeclHandler)(void *userData,
+                                                   const XML_Char *prefix);
 
 /* This is called if the document is not standalone, that is, it has an
    external subset or a reference to a parameter entity, but does not
@@ -422,7 +385,7 @@ typedef void (XMLCALL *XML_EndNamespaceDeclHandler) (
    conditions above this handler will only be called if the referenced
    entity was actually read.
 */
-typedef int (XMLCALL *XML_NotStandaloneHandler) (void *userData);
+typedef int(XMLCALL *XML_NotStandaloneHandler)(void *userData);
 
 /* This is called for a reference to an external parsed general
    entity.  The referenced entity is not automatically parsed.  The
@@ -458,12 +421,11 @@ typedef int (XMLCALL *XML_NotStandaloneHandler) (void *userData);
    Note that unlike other handlers the first argument is the parser,
    not userData.
 */
-typedef int (XMLCALL *XML_ExternalEntityRefHandler) (
-                                    XML_Parser parser,
-                                    const XML_Char *context,
-                                    const XML_Char *base,
-                                    const XML_Char *systemId,
-                                    const XML_Char *publicId);
+typedef int(XMLCALL *XML_ExternalEntityRefHandler)(XML_Parser parser,
+                                                   const XML_Char *context,
+                                                   const XML_Char *base,
+                                                   const XML_Char *systemId,
+                                                   const XML_Char *publicId);
 
 /* This is called in two situations:
    1) An entity reference is encountered for which no declaration
@@ -475,10 +437,9 @@ typedef int (XMLCALL *XML_ExternalEntityRefHandler) (
          the event would be out of sync with the reporting of the
          declarations or attribute values
 */
-typedef void (XMLCALL *XML_SkippedEntityHandler) (
-                                    void *userData,
-                                    const XML_Char *entityName,
-                                    int is_parameter_entity);
+typedef void(XMLCALL *XML_SkippedEntityHandler)(void *userData,
+                                                const XML_Char *entityName,
+                                                int is_parameter_entity);
 
 /* This structure is filled in by the XML_UnknownEncodingHandler to
    provide information to the parser about encodings that are unknown
@@ -535,8 +496,8 @@ typedef void (XMLCALL *XML_SkippedEntityHandler) (
 typedef struct {
   int map[256];
   void *data;
-  int (XMLCALL *convert)(void *data, const char *s);
-  void (XMLCALL *release)(void *data);
+  int(XMLCALL *convert)(void *data, const char *s);
+  void(XMLCALL *release)(void *data);
 } XML_Encoding;
 
 /* This is called for an encoding that is unknown to the parser.
@@ -554,23 +515,19 @@ typedef struct {
    If info does not describe a suitable encoding, then the parser will
    return an XML_UNKNOWN_ENCODING error.
 */
-typedef int (XMLCALL *XML_UnknownEncodingHandler) (
-                                    void *encodingHandlerData,
-                                    const XML_Char *name,
-                                    XML_Encoding *info);
+typedef int(XMLCALL *XML_UnknownEncodingHandler)(void *encodingHandlerData,
+                                                 const XML_Char *name,
+                                                 XML_Encoding *info);
 
 XMLPARSEAPI(void)
-XML_SetElementHandler(XML_Parser parser,
-                      XML_StartElementHandler start,
+XML_SetElementHandler(XML_Parser parser, XML_StartElementHandler start,
                       XML_EndElementHandler end);
 
 XMLPARSEAPI(void)
-XML_SetStartElementHandler(XML_Parser parser,
-                           XML_StartElementHandler handler);
+XML_SetStartElementHandler(XML_Parser parser, XML_StartElementHandler handler);
 
 XMLPARSEAPI(void)
-XML_SetEndElementHandler(XML_Parser parser,
-                         XML_EndElementHandler handler);
+XML_SetEndElementHandler(XML_Parser parser, XML_EndElementHandler handler);
 
 XMLPARSEAPI(void)
 XML_SetCharacterDataHandler(XML_Parser parser,
@@ -580,8 +537,7 @@ XMLPARSEAPI(void)
 XML_SetProcessingInstructionHandler(XML_Parser parser,
                                     XML_ProcessingInstructionHandler handler);
 XMLPARSEAPI(void)
-XML_SetCommentHandler(XML_Parser parser,
-                      XML_CommentHandler handler);
+XML_SetCommentHandler(XML_Parser parser, XML_CommentHandler handler);
 
 XMLPARSEAPI(void)
 XML_SetCdataSectionHandler(XML_Parser parser,
@@ -601,20 +557,17 @@ XML_SetEndCdataSectionHandler(XML_Parser parser,
    default handler, or to the skipped entity handler, if one is set.
 */
 XMLPARSEAPI(void)
-XML_SetDefaultHandler(XML_Parser parser,
-                      XML_DefaultHandler handler);
+XML_SetDefaultHandler(XML_Parser parser, XML_DefaultHandler handler);
 
 /* This sets the default handler but does not inhibit expansion of
    internal entities.  The entity reference will not be passed to the
    default handler.
 */
 XMLPARSEAPI(void)
-XML_SetDefaultHandlerExpand(XML_Parser parser,
-                            XML_DefaultHandler handler);
+XML_SetDefaultHandlerExpand(XML_Parser parser, XML_DefaultHandler handler);
 
 XMLPARSEAPI(void)
-XML_SetDoctypeDeclHandler(XML_Parser parser,
-                          XML_StartDoctypeDeclHandler start,
+XML_SetDoctypeDeclHandler(XML_Parser parser, XML_StartDoctypeDeclHandler start,
                           XML_EndDoctypeDeclHandler end);
 
 XMLPARSEAPI(void)
@@ -622,16 +575,14 @@ XML_SetStartDoctypeDeclHandler(XML_Parser parser,
                                XML_StartDoctypeDeclHandler start);
 
 XMLPARSEAPI(void)
-XML_SetEndDoctypeDeclHandler(XML_Parser parser,
-                             XML_EndDoctypeDeclHandler end);
+XML_SetEndDoctypeDeclHandler(XML_Parser parser, XML_EndDoctypeDeclHandler end);
 
 XMLPARSEAPI(void)
 XML_SetUnparsedEntityDeclHandler(XML_Parser parser,
                                  XML_UnparsedEntityDeclHandler handler);
 
 XMLPARSEAPI(void)
-XML_SetNotationDeclHandler(XML_Parser parser,
-                           XML_NotationDeclHandler handler);
+XML_SetNotationDeclHandler(XML_Parser parser, XML_NotationDeclHandler handler);
 
 XMLPARSEAPI(void)
 XML_SetNamespaceDeclHandler(XML_Parser parser,
@@ -659,8 +610,7 @@ XML_SetExternalEntityRefHandler(XML_Parser parser,
    instead of the parser object.
 */
 XMLPARSEAPI(void)
-XML_SetExternalEntityRefHandlerArg(XML_Parser parser,
-                                   void *arg);
+XML_SetExternalEntityRefHandlerArg(XML_Parser parser, void *arg);
 
 XMLPARSEAPI(void)
 XML_SetSkippedEntityHandler(XML_Parser parser,
@@ -740,7 +690,6 @@ XML_UseParserAsHandlerArg(XML_Parser parser);
 XMLPARSEAPI(enum XML_Error)
 XML_UseForeignDTD(XML_Parser parser, XML_Bool useDTD);
 
-
 /* Sets the base to be used for resolving relative URIs in system
    identifiers in declarations.  Resolving relative identifiers is
    left to the application: this value will be passed through as the
@@ -780,10 +729,10 @@ XML_GetIdAttributeIndex(XML_Parser parser);
    info->valueEnd - info->valueStart = 4 bytes.
 */
 typedef struct {
-  XML_Index  nameStart;  /* Offset to beginning of the attribute name. */
-  XML_Index  nameEnd;    /* Offset after the attribute name's last byte. */
-  XML_Index  valueStart; /* Offset to beginning of the attribute value. */
-  XML_Index  valueEnd;   /* Offset after the attribute value's last byte. */
+  XML_Index nameStart;  /* Offset to beginning of the attribute name. */
+  XML_Index nameEnd;    /* Offset after the attribute name's last byte. */
+  XML_Index valueStart; /* Offset to beginning of the attribute value. */
+  XML_Index valueEnd;   /* Offset after the attribute value's last byte. */
 } XML_AttrInfo;
 
 /* Returns an array of XML_AttrInfo structures for the attribute/value pairs
@@ -819,20 +768,20 @@ XML_ParseBuffer(XML_Parser parser, int len, int isFinal);
    (resumable = 0) an already suspended parser. Some call-backs may
    still follow because they would otherwise get lost. Examples:
    - endElementHandler() for empty elements when stopped in
-     startElementHandler(), 
-   - endNameSpaceDeclHandler() when stopped in endElementHandler(), 
+     startElementHandler(),
+   - endNameSpaceDeclHandler() when stopped in endElementHandler(),
    and possibly others.
 
    Can be called from most handlers, including DTD related call-backs,
    except when parsing an external parameter entity and resumable != 0.
    Returns XML_STATUS_OK when successful, XML_STATUS_ERROR otherwise.
-   Possible error codes: 
+   Possible error codes:
    - XML_ERROR_SUSPENDED: when suspending an already suspended parser.
    - XML_ERROR_FINISHED: when the parser has already finished.
    - XML_ERROR_SUSPEND_PE: when suspending while parsing an external PE.
 
-   When resumable != 0 (true) then parsing is suspended, that is, 
-   XML_Parse() and XML_ParseBuffer() return XML_STATUS_SUSPENDED. 
+   When resumable != 0 (true) then parsing is suspended, that is,
+   XML_Parse() and XML_ParseBuffer() return XML_STATUS_SUSPENDED.
    Otherwise, parsing is aborted, that is, XML_Parse() and XML_ParseBuffer()
    return XML_STATUS_ERROR with error code XML_ERROR_ABORTED.
 
@@ -843,7 +792,7 @@ XML_ParseBuffer(XML_Parser parser, int len, int isFinal);
    the externalEntityRefHandler() to call XML_StopParser() on the parent
    parser (recursively), if one wants to stop parsing altogether.
 
-   When suspended, parsing can be resumed by calling XML_ResumeParser(). 
+   When suspended, parsing can be resumed by calling XML_ResumeParser().
 */
 XMLPARSEAPI(enum XML_Status)
 XML_StopParser(XML_Parser parser, XML_Bool resumable);
@@ -851,7 +800,7 @@ XML_StopParser(XML_Parser parser, XML_Bool resumable);
 /* Resumes parsing after it has been suspended with XML_StopParser().
    Must not be called from within a handler call-back. Returns same
    status codes as XML_Parse() or XML_ParseBuffer().
-   Additional error code XML_ERROR_NOT_SUSPENDED possible.   
+   Additional error code XML_ERROR_NOT_SUSPENDED possible.
 
    *Note*:
    This must be called on the most deeply nested child parser instance
@@ -863,12 +812,7 @@ XML_StopParser(XML_Parser parser, XML_Bool resumable);
 XMLPARSEAPI(enum XML_Status)
 XML_ResumeParser(XML_Parser parser);
 
-enum XML_Parsing {
-  XML_INITIALIZED,
-  XML_PARSING,
-  XML_FINISHED,
-  XML_SUSPENDED
-};
+enum XML_Parsing { XML_INITIALIZED, XML_PARSING, XML_FINISHED, XML_SUSPENDED };
 
 typedef struct {
   enum XML_Parsing parsing;
@@ -900,8 +844,7 @@ XML_GetParsingStatus(XML_Parser parser, XML_ParsingStatus *status);
    Otherwise returns a new XML_Parser object.
 */
 XMLPARSEAPI(XML_Parser)
-XML_ExternalEntityParserCreate(XML_Parser parser,
-                               const XML_Char *context,
+XML_ExternalEntityParserCreate(XML_Parser parser, const XML_Char *context,
                                const XML_Char *encoding);
 
 enum XML_ParamEntityParsing {
@@ -945,8 +888,7 @@ XML_SetParamEntityParsing(XML_Parser parser,
    Note: If parser == NULL, the function will do nothing and return 0.
 */
 XMLPARSEAPI(int)
-XML_SetHashSalt(XML_Parser parser,
-                unsigned long hash_salt);
+XML_SetHashSalt(XML_Parser parser, unsigned long hash_salt);
 
 /* If XML_Parse or XML_ParseBuffer have returned XML_STATUS_ERROR, then
    XML_GetErrorCode returns information about the error.
@@ -963,7 +905,7 @@ XML_GetErrorCode(XML_Parser parser);
    be within the relevant markup.  When called outside of the callback
    functions, the position indicated will be just past the last parse
    event (regardless of whether there was an associated callback).
-   
+
    They may also be called after returning from a call to XML_Parse
    or XML_ParseBuffer.  If the return value is XML_STATUS_ERROR then
    the location is the location of the character at which the error
@@ -995,14 +937,12 @@ XML_GetCurrentByteCount(XML_Parser parser);
    the handler that makes the call.
 */
 XMLPARSEAPI(const char *)
-XML_GetInputContext(XML_Parser parser,
-                    int *offset,
-                    int *size);
+XML_GetInputContext(XML_Parser parser, int *offset, int *size);
 
 /* For backwards compatibility with previous versions. */
-#define XML_GetErrorLineNumber   XML_GetCurrentLineNumber
+#define XML_GetErrorLineNumber XML_GetCurrentLineNumber
 #define XML_GetErrorColumnNumber XML_GetCurrentColumnNumber
-#define XML_GetErrorByteIndex    XML_GetCurrentByteIndex
+#define XML_GetErrorByteIndex XML_GetCurrentByteIndex
 
 /* Frees the content model passed to the element declaration handler */
 XMLPARSEAPI(void)
@@ -1062,21 +1002,20 @@ enum XML_FeatureEnum {
 };
 
 typedef struct {
-  enum XML_FeatureEnum  feature;
-  const XML_LChar       *name;
-  long int              value;
+  enum XML_FeatureEnum feature;
+  const XML_LChar *name;
+  long int value;
 } XML_Feature;
 
 XMLPARSEAPI(const XML_Feature *)
 XML_GetFeatureList(void);
 
-
 /* Expat follows the semantic versioning convention.
    See http://semver.org.
 */
 #define XML_MAJOR_VERSION 2
 #define XML_MINOR_VERSION 2
-#define XML_MICRO_VERSION 4
+#define XML_MICRO_VERSION 8
 
 #ifdef __cplusplus
 }
diff --git a/Modules/expat/expat_external.h b/Modules/expat/expat_external.h
index 8110285649696..f2b75dda8e279 100644
--- a/Modules/expat/expat_external.h
+++ b/Modules/expat/expat_external.h
@@ -35,14 +35,6 @@
 
 /* External API definitions */
 
-/* Namespace external symbols to allow multiple libexpat version to
-   co-exist. */
-#include "pyexpatns.h"
-
-#if defined(_MSC_EXTENSIONS) && !defined(__BEOS__) && !defined(__CYGWIN__)
-#define XML_USE_MSC_EXTENSIONS 1
-#endif
-
 /* Expat tries very hard to make the API boundary very specifically
    defined.  There are two macros defined to control this boundary;
    each of these can be defined before including this header to
@@ -65,12 +57,17 @@
    compiled with the cdecl calling convention as the default since
    system headers may assume the cdecl convention.
 */
+
+/* Namespace external symbols to allow multiple libexpat version to
+   co-exist. */
+#include "pyexpatns.h"
+
 #ifndef XMLCALL
-#if defined(_MSC_VER)
-#define XMLCALL __cdecl
-#elif defined(__GNUC__) && defined(__i386) && !defined(__INTEL_COMPILER)
-#define XMLCALL __attribute__((cdecl))
-#else
+#  if defined(_MSC_VER)
+#    define XMLCALL __cdecl
+#  elif defined(__GNUC__) && defined(__i386) && ! defined(__INTEL_COMPILER)
+#    define XMLCALL __attribute__((cdecl))
+#  else
 /* For any platform which uses this definition and supports more than
    one calling convention, we need to extend this definition to
    declare the convention used on that platform, if it's possible to
@@ -81,41 +78,46 @@
    pre-processor and how to specify the same calling convention as the
    platform's malloc() implementation.
 */
-#define XMLCALL
-#endif
-#endif  /* not defined XMLCALL */
+#    define XMLCALL
+#  endif
+#endif /* not defined XMLCALL */
 
-
-#if !defined(XML_STATIC) && !defined(XMLIMPORT)
-#ifndef XML_BUILDING_EXPAT
+#if ! defined(XML_STATIC) && ! defined(XMLIMPORT)
+#  ifndef XML_BUILDING_EXPAT
 /* using Expat from an application */
 
-#ifdef XML_USE_MSC_EXTENSIONS
-#define XMLIMPORT __declspec(dllimport)
-#endif
+#    if defined(_MSC_EXTENSIONS) && ! defined(__BEOS__) && ! defined(__CYGWIN__)
+#      define XMLIMPORT __declspec(dllimport)
+#    endif
 
+#  endif
+#endif /* not defined XML_STATIC */
+
+#ifndef XML_ENABLE_VISIBILITY
+#  define XML_ENABLE_VISIBILITY 0
 #endif
-#endif  /* not defined XML_STATIC */
 
-#if !defined(XMLIMPORT) && defined(__GNUC__) && (__GNUC__ >= 4)
-#define XMLIMPORT __attribute__ ((visibility ("default")))
+#if ! defined(XMLIMPORT) && XML_ENABLE_VISIBILITY
+#  define XMLIMPORT __attribute__((visibility("default")))
 #endif
 
 /* If we didn't define it above, define it away: */
 #ifndef XMLIMPORT
-#define XMLIMPORT
+#  define XMLIMPORT
 #endif
 
-#if defined(__GNUC__) && (__GNUC__ > 2 || (__GNUC__ == 2 && __GNUC_MINOR__ >= 96))
-#define XML_ATTR_MALLOC __attribute__((__malloc__))
+#if defined(__GNUC__)                                                          \
+    && (__GNUC__ > 2 || (__GNUC__ == 2 && __GNUC_MINOR__ >= 96))
+#  define XML_ATTR_MALLOC __attribute__((__malloc__))
 #else
-#define XML_ATTR_MALLOC
+#  define XML_ATTR_MALLOC
 #endif
 
-#if defined(__GNUC__) && ((__GNUC__ > 4) || (__GNUC__ == 4 && __GNUC_MINOR__ >= 3))
-#define XML_ATTR_ALLOC_SIZE(x)  __attribute__((__alloc_size__(x)))
+#if defined(__GNUC__)                                                          \
+    && ((__GNUC__ > 4) || (__GNUC__ == 4 && __GNUC_MINOR__ >= 3))
+#  define XML_ATTR_ALLOC_SIZE(x) __attribute__((__alloc_size__(x)))
 #else
-#define XML_ATTR_ALLOC_SIZE(x)
+#  define XML_ATTR_ALLOC_SIZE(x)
 #endif
 
 #define XMLPARSEAPI(type) XMLIMPORT type XMLCALL
@@ -125,33 +127,30 @@ extern "C" {
 #endif
 
 #ifdef XML_UNICODE_WCHAR_T
-# define XML_UNICODE
-# if defined(__SIZEOF_WCHAR_T__) && (__SIZEOF_WCHAR_T__ != 2)
-#  error "sizeof(wchar_t) != 2; Need -fshort-wchar for both Expat and libc"
-# endif
+#  ifndef XML_UNICODE
+#    define XML_UNICODE
+#  endif
+#  if defined(__SIZEOF_WCHAR_T__) && (__SIZEOF_WCHAR_T__ != 2)
+#    error "sizeof(wchar_t) != 2; Need -fshort-wchar for both Expat and libc"
+#  endif
 #endif
 
-#ifdef XML_UNICODE     /* Information is UTF-16 encoded. */
-#ifdef XML_UNICODE_WCHAR_T
+#ifdef XML_UNICODE /* Information is UTF-16 encoded. */
+#  ifdef XML_UNICODE_WCHAR_T
 typedef wchar_t XML_Char;
 typedef wchar_t XML_LChar;
-#else
+#  else
 typedef unsigned short XML_Char;
 typedef char XML_LChar;
-#endif /* XML_UNICODE_WCHAR_T */
-#else                  /* Information is UTF-8 encoded. */
+#  endif /* XML_UNICODE_WCHAR_T */
+#else    /* Information is UTF-8 encoded. */
 typedef char XML_Char;
 typedef char XML_LChar;
-#endif /* XML_UNICODE */
+#endif   /* XML_UNICODE */
 
-#ifdef XML_LARGE_SIZE  /* Use large integers for file/stream positions. */
-#if defined(XML_USE_MSC_EXTENSIONS) && _MSC_VER < 1400
-typedef __int64 XML_Index; 
-typedef unsigned __int64 XML_Size;
-#else
+#ifdef XML_LARGE_SIZE /* Use large integers for file/stream positions. */
 typedef long long XML_Index;
 typedef unsigned long long XML_Size;
-#endif
 #else
 typedef long XML_Index;
 typedef unsigned long XML_Size;
diff --git a/Modules/expat/iasciitab.h b/Modules/expat/iasciitab.h
index ce4a4bf7edef5..ea97cfcf678e0 100644
--- a/Modules/expat/iasciitab.h
+++ b/Modules/expat/iasciitab.h
@@ -32,34 +32,34 @@
 
 /* Like asciitab.h, except that 0xD has code BT_S rather than BT_CR */
 /* 0x00 */ BT_NONXML, BT_NONXML, BT_NONXML, BT_NONXML,
-/* 0x04 */ BT_NONXML, BT_NONXML, BT_NONXML, BT_NONXML,
-/* 0x08 */ BT_NONXML, BT_S, BT_LF, BT_NONXML,
-/* 0x0C */ BT_NONXML, BT_S, BT_NONXML, BT_NONXML,
-/* 0x10 */ BT_NONXML, BT_NONXML, BT_NONXML, BT_NONXML,
-/* 0x14 */ BT_NONXML, BT_NONXML, BT_NONXML, BT_NONXML,
-/* 0x18 */ BT_NONXML, BT_NONXML, BT_NONXML, BT_NONXML,
-/* 0x1C */ BT_NONXML, BT_NONXML, BT_NONXML, BT_NONXML,
-/* 0x20 */ BT_S, BT_EXCL, BT_QUOT, BT_NUM,
-/* 0x24 */ BT_OTHER, BT_PERCNT, BT_AMP, BT_APOS,
-/* 0x28 */ BT_LPAR, BT_RPAR, BT_AST, BT_PLUS,
-/* 0x2C */ BT_COMMA, BT_MINUS, BT_NAME, BT_SOL,
-/* 0x30 */ BT_DIGIT, BT_DIGIT, BT_DIGIT, BT_DIGIT,
-/* 0x34 */ BT_DIGIT, BT_DIGIT, BT_DIGIT, BT_DIGIT,
-/* 0x38 */ BT_DIGIT, BT_DIGIT, BT_COLON, BT_SEMI,
-/* 0x3C */ BT_LT, BT_EQUALS, BT_GT, BT_QUEST,
-/* 0x40 */ BT_OTHER, BT_HEX, BT_HEX, BT_HEX,
-/* 0x44 */ BT_HEX, BT_HEX, BT_HEX, BT_NMSTRT,
-/* 0x48 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
-/* 0x4C */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
-/* 0x50 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
-/* 0x54 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
-/* 0x58 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_LSQB,
-/* 0x5C */ BT_OTHER, BT_RSQB, BT_OTHER, BT_NMSTRT,
-/* 0x60 */ BT_OTHER, BT_HEX, BT_HEX, BT_HEX,
-/* 0x64 */ BT_HEX, BT_HEX, BT_HEX, BT_NMSTRT,
-/* 0x68 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
-/* 0x6C */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
-/* 0x70 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
-/* 0x74 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
-/* 0x78 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_OTHER,
-/* 0x7C */ BT_VERBAR, BT_OTHER, BT_OTHER, BT_OTHER,
+    /* 0x04 */ BT_NONXML, BT_NONXML, BT_NONXML, BT_NONXML,
+    /* 0x08 */ BT_NONXML, BT_S, BT_LF, BT_NONXML,
+    /* 0x0C */ BT_NONXML, BT_S, BT_NONXML, BT_NONXML,
+    /* 0x10 */ BT_NONXML, BT_NONXML, BT_NONXML, BT_NONXML,
+    /* 0x14 */ BT_NONXML, BT_NONXML, BT_NONXML, BT_NONXML,
+    /* 0x18 */ BT_NONXML, BT_NONXML, BT_NONXML, BT_NONXML,
+    /* 0x1C */ BT_NONXML, BT_NONXML, BT_NONXML, BT_NONXML,
+    /* 0x20 */ BT_S, BT_EXCL, BT_QUOT, BT_NUM,
+    /* 0x24 */ BT_OTHER, BT_PERCNT, BT_AMP, BT_APOS,
+    /* 0x28 */ BT_LPAR, BT_RPAR, BT_AST, BT_PLUS,
+    /* 0x2C */ BT_COMMA, BT_MINUS, BT_NAME, BT_SOL,
+    /* 0x30 */ BT_DIGIT, BT_DIGIT, BT_DIGIT, BT_DIGIT,
+    /* 0x34 */ BT_DIGIT, BT_DIGIT, BT_DIGIT, BT_DIGIT,
+    /* 0x38 */ BT_DIGIT, BT_DIGIT, BT_COLON, BT_SEMI,
+    /* 0x3C */ BT_LT, BT_EQUALS, BT_GT, BT_QUEST,
+    /* 0x40 */ BT_OTHER, BT_HEX, BT_HEX, BT_HEX,
+    /* 0x44 */ BT_HEX, BT_HEX, BT_HEX, BT_NMSTRT,
+    /* 0x48 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
+    /* 0x4C */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
+    /* 0x50 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
+    /* 0x54 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
+    /* 0x58 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_LSQB,
+    /* 0x5C */ BT_OTHER, BT_RSQB, BT_OTHER, BT_NMSTRT,
+    /* 0x60 */ BT_OTHER, BT_HEX, BT_HEX, BT_HEX,
+    /* 0x64 */ BT_HEX, BT_HEX, BT_HEX, BT_NMSTRT,
+    /* 0x68 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
+    /* 0x6C */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
+    /* 0x70 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
+    /* 0x74 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
+    /* 0x78 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_OTHER,
+    /* 0x7C */ BT_VERBAR, BT_OTHER, BT_OTHER, BT_OTHER,
diff --git a/Modules/expat/internal.h b/Modules/expat/internal.h
index 3c5d6e913d6db..60913dab762f8 100644
--- a/Modules/expat/internal.h
+++ b/Modules/expat/internal.h
@@ -49,7 +49,7 @@
    USE OR OTHER DEALINGS IN THE SOFTWARE.
 */
 
-#if defined(__GNUC__) && defined(__i386__) && !defined(__MINGW32__)
+#if defined(__GNUC__) && defined(__i386__) && ! defined(__MINGW32__)
 /* We'll use this version by default only where we know it helps.
 
    regparm() generates warnings on Solaris boxes.   See SF bug #692878.
@@ -59,8 +59,8 @@
    #define FASTCALL __attribute__((stdcall, regparm(3)))
    and let's try this:
 */
-#define FASTCALL __attribute__((regparm(3)))
-#define PTRFASTCALL __attribute__((regparm(3)))
+#  define FASTCALL __attribute__((regparm(3)))
+#  define PTRFASTCALL __attribute__((regparm(3)))
 #endif
 
 /* Using __fastcall seems to have an unexpected negative effect under
@@ -74,50 +74,49 @@
 /* Make sure all of these are defined if they aren't already. */
 
 #ifndef FASTCALL
-#define FASTCALL
+#  define FASTCALL
 #endif
 
 #ifndef PTRCALL
-#define PTRCALL
+#  define PTRCALL
 #endif
 
 #ifndef PTRFASTCALL
-#define PTRFASTCALL
+#  define PTRFASTCALL
 #endif
 
 #ifndef XML_MIN_SIZE
-#if !defined(__cplusplus) && !defined(inline)
-#ifdef __GNUC__
-#define inline __inline
-#endif /* __GNUC__ */
-#endif
+#  if ! defined(__cplusplus) && ! defined(inline)
+#    ifdef __GNUC__
+#      define inline __inline
+#    endif /* __GNUC__ */
+#  endif
 #endif /* XML_MIN_SIZE */
 
 #ifdef __cplusplus
-#define inline inline
+#  define inline inline
 #else
-#ifndef inline
-#define inline
-#endif
+#  ifndef inline
+#    define inline
+#  endif
 #endif
 
 #ifndef UNUSED_P
-# ifdef __GNUC__
-#  define UNUSED_P(p) UNUSED_ ## p __attribute__((__unused__))
-# else
-#  define UNUSED_P(p) UNUSED_ ## p
-# endif
+#  define UNUSED_P(p) (void)p
 #endif
 
-
 #ifdef __cplusplus
 extern "C" {
 #endif
 
-
+#ifdef XML_ENABLE_VISIBILITY
+#  if XML_ENABLE_VISIBILITY
+__attribute__((visibility("default")))
+#  endif
+#endif
 void
-align_limit_to_full_utf8_characters(const char * from, const char ** fromLimRef);
-
+_INTERNAL_trim_to_complete_utf8_characters(const char *from,
+                                           const char **fromLimRef);
 
 #ifdef __cplusplus
 }
diff --git a/Modules/expat/latin1tab.h b/Modules/expat/latin1tab.h
index 95dfa52b1fbeb..6f916041355a8 100644
--- a/Modules/expat/latin1tab.h
+++ b/Modules/expat/latin1tab.h
@@ -31,34 +31,34 @@
 */
 
 /* 0x80 */ BT_OTHER, BT_OTHER, BT_OTHER, BT_OTHER,
-/* 0x84 */ BT_OTHER, BT_OTHER, BT_OTHER, BT_OTHER,
-/* 0x88 */ BT_OTHER, BT_OTHER, BT_OTHER, BT_OTHER,
-/* 0x8C */ BT_OTHER, BT_OTHER, BT_OTHER, BT_OTHER,
-/* 0x90 */ BT_OTHER, BT_OTHER, BT_OTHER, BT_OTHER,
-/* 0x94 */ BT_OTHER, BT_OTHER, BT_OTHER, BT_OTHER,
-/* 0x98 */ BT_OTHER, BT_OTHER, BT_OTHER, BT_OTHER,
-/* 0x9C */ BT_OTHER, BT_OTHER, BT_OTHER, BT_OTHER,
-/* 0xA0 */ BT_OTHER, BT_OTHER, BT_OTHER, BT_OTHER,
-/* 0xA4 */ BT_OTHER, BT_OTHER, BT_OTHER, BT_OTHER,
-/* 0xA8 */ BT_OTHER, BT_OTHER, BT_NMSTRT, BT_OTHER,
-/* 0xAC */ BT_OTHER, BT_OTHER, BT_OTHER, BT_OTHER,
-/* 0xB0 */ BT_OTHER, BT_OTHER, BT_OTHER, BT_OTHER,
-/* 0xB4 */ BT_OTHER, BT_NMSTRT, BT_OTHER, BT_NAME,
-/* 0xB8 */ BT_OTHER, BT_OTHER, BT_NMSTRT, BT_OTHER,
-/* 0xBC */ BT_OTHER, BT_OTHER, BT_OTHER, BT_OTHER,
-/* 0xC0 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
-/* 0xC4 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
-/* 0xC8 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
-/* 0xCC */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
-/* 0xD0 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
-/* 0xD4 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_OTHER,
-/* 0xD8 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
-/* 0xDC */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
-/* 0xE0 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
-/* 0xE4 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
-/* 0xE8 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
-/* 0xEC */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
-/* 0xF0 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
-/* 0xF4 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_OTHER,
-/* 0xF8 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
-/* 0xFC */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
+    /* 0x84 */ BT_OTHER, BT_OTHER, BT_OTHER, BT_OTHER,
+    /* 0x88 */ BT_OTHER, BT_OTHER, BT_OTHER, BT_OTHER,
+    /* 0x8C */ BT_OTHER, BT_OTHER, BT_OTHER, BT_OTHER,
+    /* 0x90 */ BT_OTHER, BT_OTHER, BT_OTHER, BT_OTHER,
+    /* 0x94 */ BT_OTHER, BT_OTHER, BT_OTHER, BT_OTHER,
+    /* 0x98 */ BT_OTHER, BT_OTHER, BT_OTHER, BT_OTHER,
+    /* 0x9C */ BT_OTHER, BT_OTHER, BT_OTHER, BT_OTHER,
+    /* 0xA0 */ BT_OTHER, BT_OTHER, BT_OTHER, BT_OTHER,
+    /* 0xA4 */ BT_OTHER, BT_OTHER, BT_OTHER, BT_OTHER,
+    /* 0xA8 */ BT_OTHER, BT_OTHER, BT_NMSTRT, BT_OTHER,
+    /* 0xAC */ BT_OTHER, BT_OTHER, BT_OTHER, BT_OTHER,
+    /* 0xB0 */ BT_OTHER, BT_OTHER, BT_OTHER, BT_OTHER,
+    /* 0xB4 */ BT_OTHER, BT_NMSTRT, BT_OTHER, BT_NAME,
+    /* 0xB8 */ BT_OTHER, BT_OTHER, BT_NMSTRT, BT_OTHER,
+    /* 0xBC */ BT_OTHER, BT_OTHER, BT_OTHER, BT_OTHER,
+    /* 0xC0 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
+    /* 0xC4 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
+    /* 0xC8 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
+    /* 0xCC */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
+    /* 0xD0 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
+    /* 0xD4 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_OTHER,
+    /* 0xD8 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
+    /* 0xDC */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
+    /* 0xE0 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
+    /* 0xE4 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
+    /* 0xE8 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
+    /* 0xEC */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
+    /* 0xF0 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
+    /* 0xF4 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_OTHER,
+    /* 0xF8 */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
+    /* 0xFC */ BT_NMSTRT, BT_NMSTRT, BT_NMSTRT, BT_NMSTRT,
diff --git a/Modules/expat/loadlibrary.c b/Modules/expat/loadlibrary.c
deleted file mode 100644
index 452ae92db2673..0000000000000
--- a/Modules/expat/loadlibrary.c
+++ /dev/null
@@ -1,143 +0,0 @@
-/***************************************************************************
- *                                  _   _ ____  _
- *  Project                     ___| | | |  _ \| |
- *                             / __| | | | |_) | |
- *                            | (__| |_| |  _ <| |___
- *                             \___|\___/|_| \_\_____|
- *
- * Copyright (C) 2016 - 2017, Steve Holme, <steve_holme at hotmail.com>.
- * Copyright (C) 2017, Expat development team
- *
- * All rights reserved.
- * Licensed under the MIT license:
- *
- * Permission to  use, copy,  modify, and distribute  this software  for any
- * purpose with  or without fee is  hereby granted, provided that  the above
- * copyright notice and this permission notice appear in all copies.
- *
- * THE  SOFTWARE  IS  PROVIDED  "AS  IS",  WITHOUT  WARRANTY  OF  ANY  KIND,
- * EXPRESS  OR IMPLIED,  INCLUDING  BUT  NOT LIMITED  TO  THE WARRANTIES  OF
- * MERCHANTABILITY, FITNESS FOR A  PARTICULAR PURPOSE AND NONINFRINGEMENT OF
- * THIRD PARTY RIGHTS. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
- * LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF
- * CONTRACT, TORT OR  OTHERWISE, ARISING FROM, OUT OF OR  IN CONNECTION WITH
- * THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
- *
- * Except as contained in this notice,  the name of a copyright holder shall
- * not be used in advertising or otherwise to promote the sale, use or other
- * dealings  in this  Software without  prior written  authorization of  the
- * copyright holder.
- *
- ***************************************************************************/
-
-#if defined(_WIN32)
-
-#include <windows.h>
-#include <tchar.h>
-
-
-HMODULE _Expat_LoadLibrary(LPCTSTR filename);
-
-
-#if !defined(LOAD_WITH_ALTERED_SEARCH_PATH)
-#define LOAD_WITH_ALTERED_SEARCH_PATH  0x00000008
-#endif
-
-#if !defined(LOAD_LIBRARY_SEARCH_SYSTEM32)
-#define LOAD_LIBRARY_SEARCH_SYSTEM32   0x00000800
-#endif
-
-/* We use our own typedef here since some headers might lack these */
-typedef HMODULE (APIENTRY *LOADLIBRARYEX_FN)(LPCTSTR, HANDLE, DWORD);
-
-/* See function definitions in winbase.h */
-#ifdef UNICODE
-#  ifdef _WIN32_WCE
-#    define LOADLIBARYEX  L"LoadLibraryExW"
-#  else
-#    define LOADLIBARYEX  "LoadLibraryExW"
-#  endif
-#else
-#  define LOADLIBARYEX    "LoadLibraryExA"
-#endif
-
-
-/*
- * _Expat_LoadLibrary()
- *
- * This is used to dynamically load DLLs using the most secure method available
- * for the version of Windows that we are running on.
- *
- * Parameters:
- *
- * filename  [in] - The filename or full path of the DLL to load. If only the
- *                  filename is passed then the DLL will be loaded from the
- *                  Windows system directory.
- *
- * Returns the handle of the module on success; otherwise NULL.
- */
-HMODULE _Expat_LoadLibrary(LPCTSTR filename)
-{
-  HMODULE hModule = NULL;
-  LOADLIBRARYEX_FN pLoadLibraryEx = NULL;
-
-  /* Get a handle to kernel32 so we can access it's functions at runtime */
-  HMODULE hKernel32 = GetModuleHandle(TEXT("kernel32"));
-  if(!hKernel32)
-    return NULL;
-
-  /* Attempt to find LoadLibraryEx() which is only available on Windows 2000
-     and above */
-  pLoadLibraryEx = (LOADLIBRARYEX_FN) GetProcAddress(hKernel32, LOADLIBARYEX);
-
-  /* Detect if there's already a path in the filename and load the library if
-     there is. Note: Both back slashes and forward slashes have been supported
-     since the earlier days of DOS at an API level although they are not
-     supported by command prompt */
-  if(_tcspbrk(filename, TEXT("\\/"))) {
-    /** !checksrc! disable BANNEDFUNC 1 **/
-    hModule = pLoadLibraryEx ?
-      pLoadLibraryEx(filename, NULL, LOAD_WITH_ALTERED_SEARCH_PATH) :
-      LoadLibrary(filename);
-  }
-  /* Detect if KB2533623 is installed, as LOAD_LIBARY_SEARCH_SYSTEM32 is only
-     supported on Windows Vista, Windows Server 2008, Windows 7 and Windows
-     Server 2008 R2 with this patch or natively on Windows 8 and above */
-  else if(pLoadLibraryEx && GetProcAddress(hKernel32, "AddDllDirectory")) {
-    /* Load the DLL from the Windows system directory */
-    hModule = pLoadLibraryEx(filename, NULL, LOAD_LIBRARY_SEARCH_SYSTEM32);
-  }
-  else {
-    /* Attempt to get the Windows system path */
-    UINT systemdirlen = GetSystemDirectory(NULL, 0);
-    if(systemdirlen) {
-      /* Allocate space for the full DLL path (Room for the null terminator
-         is included in systemdirlen) */
-      size_t filenamelen = _tcslen(filename);
-      TCHAR *path = malloc(sizeof(TCHAR) * (systemdirlen + 1 + filenamelen));
-      if(path && GetSystemDirectory(path, systemdirlen)) {
-        /* Calculate the full DLL path */
-        _tcscpy(path + _tcslen(path), TEXT("\\"));
-        _tcscpy(path + _tcslen(path), filename);
-
-        /* Load the DLL from the Windows system directory */
-        /** !checksrc! disable BANNEDFUNC 1 **/
-        hModule = pLoadLibraryEx ?
-          pLoadLibraryEx(path, NULL, LOAD_WITH_ALTERED_SEARCH_PATH) :
-          LoadLibrary(path);
-
-      }
-      free(path);
-    }
-  }
-
-  return hModule;
-}
-
-#else /* defined(_WIN32) */
-
-/* ISO C requires a translation unit to contain at least one declaration
-   [-Wempty-translation-unit] */
-typedef int _TRANSLATION_UNIT_LOAD_LIBRARY_C_NOT_EMTPY;
-
-#endif /* defined(_WIN32) */
diff --git a/Modules/expat/nametab.h b/Modules/expat/nametab.h
index bfa2bd38cd95c..3681df348eebd 100644
--- a/Modules/expat/nametab.h
+++ b/Modules/expat/nametab.h
@@ -31,152 +31,106 @@
 */
 
 static const unsigned namingBitmap[] = {
-0x00000000, 0x00000000, 0x00000000, 0x00000000,
-0x00000000, 0x00000000, 0x00000000, 0x00000000,
-0xFFFFFFFF, 0xFFFFFFFF, 0xFFFFFFFF, 0xFFFFFFFF,
-0xFFFFFFFF, 0xFFFFFFFF, 0xFFFFFFFF, 0xFFFFFFFF,
-0x00000000, 0x04000000, 0x87FFFFFE, 0x07FFFFFE,
-0x00000000, 0x00000000, 0xFF7FFFFF, 0xFF7FFFFF,
-0xFFFFFFFF, 0x7FF3FFFF, 0xFFFFFDFE, 0x7FFFFFFF,
-0xFFFFFFFF, 0xFFFFFFFF, 0xFFFFE00F, 0xFC31FFFF,
-0x00FFFFFF, 0x00000000, 0xFFFF0000, 0xFFFFFFFF,
-0xFFFFFFFF, 0xF80001FF, 0x00000003, 0x00000000,
-0x00000000, 0x00000000, 0x00000000, 0x00000000,
-0xFFFFD740, 0xFFFFFFFB, 0x547F7FFF, 0x000FFFFD,
-0xFFFFDFFE, 0xFFFFFFFF, 0xDFFEFFFF, 0xFFFFFFFF,
-0xFFFF0003, 0xFFFFFFFF, 0xFFFF199F, 0x033FCFFF,
-0x00000000, 0xFFFE0000, 0x027FFFFF, 0xFFFFFFFE,
-0x0000007F, 0x00000000, 0xFFFF0000, 0x000707FF,
-0x00000000, 0x07FFFFFE, 0x000007FE, 0xFFFE0000,
-0xFFFFFFFF, 0x7CFFFFFF, 0x002F7FFF, 0x00000060,
-0xFFFFFFE0, 0x23FFFFFF, 0xFF000000, 0x00000003,
-0xFFF99FE0, 0x03C5FDFF, 0xB0000000, 0x00030003,
-0xFFF987E0, 0x036DFDFF, 0x5E000000, 0x001C0000,
-0xFFFBAFE0, 0x23EDFDFF, 0x00000000, 0x00000001,
-0xFFF99FE0, 0x23CDFDFF, 0xB0000000, 0x00000003,
-0xD63DC7E0, 0x03BFC718, 0x00000000, 0x00000000,
-0xFFFDDFE0, 0x03EFFDFF, 0x00000000, 0x00000003,
-0xFFFDDFE0, 0x03EFFDFF, 0x40000000, 0x00000003,
-0xFFFDDFE0, 0x03FFFDFF, 0x00000000, 0x00000003,
-0x00000000, 0x00000000, 0x00000000, 0x00000000,
-0xFFFFFFFE, 0x000D7FFF, 0x0000003F, 0x00000000,
-0xFEF02596, 0x200D6CAE, 0x0000001F, 0x00000000,
-0x00000000, 0x00000000, 0xFFFFFEFF, 0x000003FF,
-0x00000000, 0x00000000, 0x00000000, 0x00000000,
-0x00000000, 0x00000000, 0x00000000, 0x00000000,
-0x00000000, 0xFFFFFFFF, 0xFFFF003F, 0x007FFFFF,
-0x0007DAED, 0x50000000, 0x82315001, 0x002C62AB,
-0x40000000, 0xF580C900, 0x00000007, 0x02010800,
-0xFFFFFFFF, 0xFFFFFFFF, 0xFFFFFFFF, 0xFFFFFFFF,
-0x0FFFFFFF, 0xFFFFFFFF, 0xFFFFFFFF, 0x03FFFFFF,
-0x3F3FFFFF, 0xFFFFFFFF, 0xAAFF3F3F, 0x3FFFFFFF,
-0xFFFFFFFF, 0x5FDFFFFF, 0x0FCF1FDC, 0x1FDC1FFF,
-0x00000000, 0x00004C40, 0x00000000, 0x00000000,
-0x00000007, 0x00000000, 0x00000000, 0x00000000,
-0x00000080, 0x000003FE, 0xFFFFFFFE, 0xFFFFFFFF,
-0x001FFFFF, 0xFFFFFFFE, 0xFFFFFFFF, 0x07FFFFFF,
-0xFFFFFFE0, 0x00001FFF, 0x00000000, 0x00000000,
-0x00000000, 0x00000000, 0x00000000, 0x00000000,
-0xFFFFFFFF, 0xFFFFFFFF, 0xFFFFFFFF, 0xFFFFFFFF,
-0xFFFFFFFF, 0x0000003F, 0x00000000, 0x00000000,
-0xFFFFFFFF, 0xFFFFFFFF, 0xFFFFFFFF, 0xFFFFFFFF,
-0xFFFFFFFF, 0x0000000F, 0x00000000, 0x00000000,
-0x00000000, 0x07FF6000, 0x87FFFFFE, 0x07FFFFFE,
-0x00000000, 0x00800000, 0xFF7FFFFF, 0xFF7FFFFF,
-0x00FFFFFF, 0x00000000, 0xFFFF0000, 0xFFFFFFFF,
-0xFFFFFFFF, 0xF80001FF, 0x00030003, 0x00000000,
-0xFFFFFFFF, 0xFFFFFFFF, 0x0000003F, 0x00000003,
-0xFFFFD7C0, 0xFFFFFFFB, 0x547F7FFF, 0x000FFFFD,
-0xFFFFDFFE, 0xFFFFFFFF, 0xDFFEFFFF, 0xFFFFFFFF,
-0xFFFF007B, 0xFFFFFFFF, 0xFFFF199F, 0x033FCFFF,
-0x00000000, 0xFFFE0000, 0x027FFFFF, 0xFFFFFFFE,
-0xFFFE007F, 0xBBFFFFFB, 0xFFFF0016, 0x000707FF,
-0x00000000, 0x07FFFFFE, 0x0007FFFF, 0xFFFF03FF,
-0xFFFFFFFF, 0x7CFFFFFF, 0xFFEF7FFF, 0x03FF3DFF,
-0xFFFFFFEE, 0xF3FFFFFF, 0xFF1E3FFF, 0x0000FFCF,
-0xFFF99FEE, 0xD3C5FDFF, 0xB080399F, 0x0003FFCF,
-0xFFF987E4, 0xD36DFDFF, 0x5E003987, 0x001FFFC0,
-0xFFFBAFEE, 0xF3EDFDFF, 0x00003BBF, 0x0000FFC1,
-0xFFF99FEE, 0xF3CDFDFF, 0xB0C0398F, 0x0000FFC3,
-0xD63DC7EC, 0xC3BFC718, 0x00803DC7, 0x0000FF80,
-0xFFFDDFEE, 0xC3EFFDFF, 0x00603DDF, 0x0000FFC3,
-0xFFFDDFEC, 0xC3EFFDFF, 0x40603DDF, 0x0000FFC3,
-0xFFFDDFEC, 0xC3FFFDFF, 0x00803DCF, 0x0000FFC3,
-0x00000000, 0x00000000, 0x00000000, 0x00000000,
-0xFFFFFFFE, 0x07FF7FFF, 0x03FF7FFF, 0x00000000,
-0xFEF02596, 0x3BFF6CAE, 0x03FF3F5F, 0x00000000,
-0x03000000, 0xC2A003FF, 0xFFFFFEFF, 0xFFFE03FF,
-0xFEBF0FDF, 0x02FE3FFF, 0x00000000, 0x00000000,
-0x00000000, 0x00000000, 0x00000000, 0x00000000,
-0x00000000, 0x00000000, 0x1FFF0000, 0x00000002,
-0x000000A0, 0x003EFFFE, 0xFFFFFFFE, 0xFFFFFFFF,
-0x661FFFFF, 0xFFFFFFFE, 0xFFFFFFFF, 0x77FFFFFF,
+    0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000,
+    0x00000000, 0x00000000, 0xFFFFFFFF, 0xFFFFFFFF, 0xFFFFFFFF, 0xFFFFFFFF,
+    0xFFFFFFFF, 0xFFFFFFFF, 0xFFFFFFFF, 0xFFFFFFFF, 0x00000000, 0x04000000,
+    0x87FFFFFE, 0x07FFFFFE, 0x00000000, 0x00000000, 0xFF7FFFFF, 0xFF7FFFFF,
+    0xFFFFFFFF, 0x7FF3FFFF, 0xFFFFFDFE, 0x7FFFFFFF, 0xFFFFFFFF, 0xFFFFFFFF,
+    0xFFFFE00F, 0xFC31FFFF, 0x00FFFFFF, 0x00000000, 0xFFFF0000, 0xFFFFFFFF,
+    0xFFFFFFFF, 0xF80001FF, 0x00000003, 0x00000000, 0x00000000, 0x00000000,
+    0x00000000, 0x00000000, 0xFFFFD740, 0xFFFFFFFB, 0x547F7FFF, 0x000FFFFD,
+    0xFFFFDFFE, 0xFFFFFFFF, 0xDFFEFFFF, 0xFFFFFFFF, 0xFFFF0003, 0xFFFFFFFF,
+    0xFFFF199F, 0x033FCFFF, 0x00000000, 0xFFFE0000, 0x027FFFFF, 0xFFFFFFFE,
+    0x0000007F, 0x00000000, 0xFFFF0000, 0x000707FF, 0x00000000, 0x07FFFFFE,
+    0x000007FE, 0xFFFE0000, 0xFFFFFFFF, 0x7CFFFFFF, 0x002F7FFF, 0x00000060,
+    0xFFFFFFE0, 0x23FFFFFF, 0xFF000000, 0x00000003, 0xFFF99FE0, 0x03C5FDFF,
+    0xB0000000, 0x00030003, 0xFFF987E0, 0x036DFDFF, 0x5E000000, 0x001C0000,
+    0xFFFBAFE0, 0x23EDFDFF, 0x00000000, 0x00000001, 0xFFF99FE0, 0x23CDFDFF,
+    0xB0000000, 0x00000003, 0xD63DC7E0, 0x03BFC718, 0x00000000, 0x00000000,
+    0xFFFDDFE0, 0x03EFFDFF, 0x00000000, 0x00000003, 0xFFFDDFE0, 0x03EFFDFF,
+    0x40000000, 0x00000003, 0xFFFDDFE0, 0x03FFFDFF, 0x00000000, 0x00000003,
+    0x00000000, 0x00000000, 0x00000000, 0x00000000, 0xFFFFFFFE, 0x000D7FFF,
+    0x0000003F, 0x00000000, 0xFEF02596, 0x200D6CAE, 0x0000001F, 0x00000000,
+    0x00000000, 0x00000000, 0xFFFFFEFF, 0x000003FF, 0x00000000, 0x00000000,
+    0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000,
+    0x00000000, 0xFFFFFFFF, 0xFFFF003F, 0x007FFFFF, 0x0007DAED, 0x50000000,
+    0x82315001, 0x002C62AB, 0x40000000, 0xF580C900, 0x00000007, 0x02010800,
+    0xFFFFFFFF, 0xFFFFFFFF, 0xFFFFFFFF, 0xFFFFFFFF, 0x0FFFFFFF, 0xFFFFFFFF,
+    0xFFFFFFFF, 0x03FFFFFF, 0x3F3FFFFF, 0xFFFFFFFF, 0xAAFF3F3F, 0x3FFFFFFF,
+    0xFFFFFFFF, 0x5FDFFFFF, 0x0FCF1FDC, 0x1FDC1FFF, 0x00000000, 0x00004C40,
+    0x00000000, 0x00000000, 0x00000007, 0x00000000, 0x00000000, 0x00000000,
+    0x00000080, 0x000003FE, 0xFFFFFFFE, 0xFFFFFFFF, 0x001FFFFF, 0xFFFFFFFE,
+    0xFFFFFFFF, 0x07FFFFFF, 0xFFFFFFE0, 0x00001FFF, 0x00000000, 0x00000000,
+    0x00000000, 0x00000000, 0x00000000, 0x00000000, 0xFFFFFFFF, 0xFFFFFFFF,
+    0xFFFFFFFF, 0xFFFFFFFF, 0xFFFFFFFF, 0x0000003F, 0x00000000, 0x00000000,
+    0xFFFFFFFF, 0xFFFFFFFF, 0xFFFFFFFF, 0xFFFFFFFF, 0xFFFFFFFF, 0x0000000F,
+    0x00000000, 0x00000000, 0x00000000, 0x07FF6000, 0x87FFFFFE, 0x07FFFFFE,
+    0x00000000, 0x00800000, 0xFF7FFFFF, 0xFF7FFFFF, 0x00FFFFFF, 0x00000000,
+    0xFFFF0000, 0xFFFFFFFF, 0xFFFFFFFF, 0xF80001FF, 0x00030003, 0x00000000,
+    0xFFFFFFFF, 0xFFFFFFFF, 0x0000003F, 0x00000003, 0xFFFFD7C0, 0xFFFFFFFB,
+    0x547F7FFF, 0x000FFFFD, 0xFFFFDFFE, 0xFFFFFFFF, 0xDFFEFFFF, 0xFFFFFFFF,
+    0xFFFF007B, 0xFFFFFFFF, 0xFFFF199F, 0x033FCFFF, 0x00000000, 0xFFFE0000,
+    0x027FFFFF, 0xFFFFFFFE, 0xFFFE007F, 0xBBFFFFFB, 0xFFFF0016, 0x000707FF,
+    0x00000000, 0x07FFFFFE, 0x0007FFFF, 0xFFFF03FF, 0xFFFFFFFF, 0x7CFFFFFF,
+    0xFFEF7FFF, 0x03FF3DFF, 0xFFFFFFEE, 0xF3FFFFFF, 0xFF1E3FFF, 0x0000FFCF,
+    0xFFF99FEE, 0xD3C5FDFF, 0xB080399F, 0x0003FFCF, 0xFFF987E4, 0xD36DFDFF,
+    0x5E003987, 0x001FFFC0, 0xFFFBAFEE, 0xF3EDFDFF, 0x00003BBF, 0x0000FFC1,
+    0xFFF99FEE, 0xF3CDFDFF, 0xB0C0398F, 0x0000FFC3, 0xD63DC7EC, 0xC3BFC718,
+    0x00803DC7, 0x0000FF80, 0xFFFDDFEE, 0xC3EFFDFF, 0x00603DDF, 0x0000FFC3,
+    0xFFFDDFEC, 0xC3EFFDFF, 0x40603DDF, 0x0000FFC3, 0xFFFDDFEC, 0xC3FFFDFF,
+    0x00803DCF, 0x0000FFC3, 0x00000000, 0x00000000, 0x00000000, 0x00000000,
+    0xFFFFFFFE, 0x07FF7FFF, 0x03FF7FFF, 0x00000000, 0xFEF02596, 0x3BFF6CAE,
+    0x03FF3F5F, 0x00000000, 0x03000000, 0xC2A003FF, 0xFFFFFEFF, 0xFFFE03FF,
+    0xFEBF0FDF, 0x02FE3FFF, 0x00000000, 0x00000000, 0x00000000, 0x00000000,
+    0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x1FFF0000, 0x00000002,
+    0x000000A0, 0x003EFFFE, 0xFFFFFFFE, 0xFFFFFFFF, 0x661FFFFF, 0xFFFFFFFE,
+    0xFFFFFFFF, 0x77FFFFFF,
 };
 static const unsigned char nmstrtPages[] = {
-0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x00,
-0x00, 0x09, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F,
-0x10, 0x11, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
-0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x12, 0x13,
-0x00, 0x14, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
-0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
-0x15, 0x16, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
-0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
-0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
-0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x01,
-0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
-0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
-0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
-0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
-0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
-0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
-0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
-0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
-0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
-0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x17,
-0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
-0x00, 0x00, 0x00, 0x00, 0x01, 0x01, 0x01, 0x01,
-0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
-0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
-0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
-0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
-0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x18,
-0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
-0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
-0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
-0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
-0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+    0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x00, 0x00, 0x09, 0x0A, 0x0B,
+    0x0C, 0x0D, 0x0E, 0x0F, 0x10, 0x11, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+    0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x12, 0x13, 0x00, 0x14, 0x00, 0x00,
+    0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+    0x15, 0x16, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+    0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+    0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
+    0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
+    0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
+    0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
+    0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
+    0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
+    0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
+    0x01, 0x01, 0x01, 0x17, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+    0x00, 0x00, 0x00, 0x00, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
+    0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
+    0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
+    0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x18,
+    0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+    0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+    0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+    0x00, 0x00, 0x00, 0x00,
 };
 static const unsigned char namePages[] = {
-0x19, 0x03, 0x1A, 0x1B, 0x1C, 0x1D, 0x1E, 0x00,
-0x00, 0x1F, 0x20, 0x21, 0x22, 0x23, 0x24, 0x25,
-0x10, 0x11, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
-0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x12, 0x13,
-0x26, 0x14, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
-0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
-0x27, 0x16, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
-0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
-0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
-0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x01,
-0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
-0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
-0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
-0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
-0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
-0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
-0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
-0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
-0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
-0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x17,
-0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
-0x00, 0x00, 0x00, 0x00, 0x01, 0x01, 0x01, 0x01,
-0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
-0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
-0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
-0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
-0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x18,
-0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
-0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
-0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
-0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
-0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+    0x19, 0x03, 0x1A, 0x1B, 0x1C, 0x1D, 0x1E, 0x00, 0x00, 0x1F, 0x20, 0x21,
+    0x22, 0x23, 0x24, 0x25, 0x10, 0x11, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+    0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x12, 0x13, 0x26, 0x14, 0x00, 0x00,
+    0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+    0x27, 0x16, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+    0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+    0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
+    0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
+    0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
+    0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
+    0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
+    0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
+    0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
+    0x01, 0x01, 0x01, 0x17, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+    0x00, 0x00, 0x00, 0x00, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
+    0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
+    0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01,
+    0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x01, 0x18,
+    0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+    0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+    0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+    0x00, 0x00, 0x00, 0x00,
 };
diff --git a/Modules/expat/siphash.h b/Modules/expat/siphash.h
index 581872df7b499..bfee65a332f1b 100644
--- a/Modules/expat/siphash.h
+++ b/Modules/expat/siphash.h
@@ -11,6 +11,16 @@
  * --------------------------------------------------------------------------
  * HISTORY:
  *
+ * 2019-08-03  (Sebastian Pipping)
+ *   - Mark part of sip24_valid as to be excluded from clang-format
+ *   - Re-format code using clang-format 9
+ *
+ * 2018-07-08  (Anton Maklakov)
+ *   - Add "fall through" markers for GCC's -Wimplicit-fallthrough
+ *
+ * 2017-11-03  (Sebastian Pipping)
+ *   - Hide sip_tobin and sip_binof unless SIPHASH_TOBIN macro is defined
+ *
  * 2017-07-25  (Vadim Zeitlin)
  *   - Fix use of SIPHASH_MAIN macro
  *
@@ -88,175 +98,186 @@
 #include <stddef.h> /* size_t */
 
 #if defined(_WIN32) && defined(_MSC_VER) && (_MSC_VER < 1600)
-  /* For vs2003/7.1 up to vs2008/9.0; _MSC_VER 1600 is vs2010/10.0 */
-  typedef unsigned __int8   uint8_t;
-  typedef unsigned __int32 uint32_t;
-  typedef unsigned __int64 uint64_t;
+/* For vs2003/7.1 up to vs2008/9.0; _MSC_VER 1600 is vs2010/10.0 */
+typedef unsigned __int8 uint8_t;
+typedef unsigned __int32 uint32_t;
+typedef unsigned __int64 uint64_t;
 #else
- #include <stdint.h> /* uint64_t uint32_t uint8_t */
+#  include <stdint.h> /* uint64_t uint32_t uint8_t */
 #endif
 
-
 /*
  * Workaround to not require a C++11 compiler for using ULL suffix
  * if this code is included and compiled as C++; related GCC warning is:
  * warning: use of C++11 long long integer constant [-Wlong-long]
  */
-#define _SIP_ULL(high, low)  (((uint64_t)high << 32) | low)
+#define _SIP_ULL(high, low) (((uint64_t)high << 32) | low)
 
+#define SIP_ROTL(x, b) (uint64_t)(((x) << (b)) | ((x) >> (64 - (b))))
 
-#define SIP_ROTL(x, b) (uint64_t)(((x) << (b)) | ( (x) >> (64 - (b))))
+#define SIP_U32TO8_LE(p, v)                                                    \
+  (p)[0] = (uint8_t)((v) >> 0);                                                \
+  (p)[1] = (uint8_t)((v) >> 8);                                                \
+  (p)[2] = (uint8_t)((v) >> 16);                                               \
+  (p)[3] = (uint8_t)((v) >> 24);
 
-#define SIP_U32TO8_LE(p, v) \
-	(p)[0] = (uint8_t)((v) >>  0); (p)[1] = (uint8_t)((v) >>  8); \
-	(p)[2] = (uint8_t)((v) >> 16); (p)[3] = (uint8_t)((v) >> 24);
+#define SIP_U64TO8_LE(p, v)                                                    \
+  SIP_U32TO8_LE((p) + 0, (uint32_t)((v) >> 0));                                \
+  SIP_U32TO8_LE((p) + 4, (uint32_t)((v) >> 32));
 
-#define SIP_U64TO8_LE(p, v) \
-	SIP_U32TO8_LE((p) + 0, (uint32_t)((v) >>  0)); \
-	SIP_U32TO8_LE((p) + 4, (uint32_t)((v) >> 32));
+#define SIP_U8TO64_LE(p)                                                       \
+  (((uint64_t)((p)[0]) << 0) | ((uint64_t)((p)[1]) << 8)                       \
+   | ((uint64_t)((p)[2]) << 16) | ((uint64_t)((p)[3]) << 24)                   \
+   | ((uint64_t)((p)[4]) << 32) | ((uint64_t)((p)[5]) << 40)                   \
+   | ((uint64_t)((p)[6]) << 48) | ((uint64_t)((p)[7]) << 56))
 
-#define SIP_U8TO64_LE(p) \
-	(((uint64_t)((p)[0]) <<  0) | \
-	 ((uint64_t)((p)[1]) <<  8) | \
-	 ((uint64_t)((p)[2]) << 16) | \
-	 ((uint64_t)((p)[3]) << 24) | \
-	 ((uint64_t)((p)[4]) << 32) | \
-	 ((uint64_t)((p)[5]) << 40) | \
-	 ((uint64_t)((p)[6]) << 48) | \
-	 ((uint64_t)((p)[7]) << 56))
-
-
-#define SIPHASH_INITIALIZER { 0, 0, 0, 0, { 0 }, 0, 0 }
+#define SIPHASH_INITIALIZER                                                    \
+  { 0, 0, 0, 0, {0}, 0, 0 }
 
 struct siphash {
-	uint64_t v0, v1, v2, v3;
+  uint64_t v0, v1, v2, v3;
 
-	unsigned char buf[8], *p;
-	uint64_t c;
+  unsigned char buf[8], *p;
+  uint64_t c;
 }; /* struct siphash */
 
-
 #define SIP_KEYLEN 16
 
 struct sipkey {
-	uint64_t k[2];
+  uint64_t k[2];
 }; /* struct sipkey */
 
-#define sip_keyof(k) sip_tokey(&(struct sipkey){ { 0 } }, (k))
+#define sip_keyof(k) sip_tokey(&(struct sipkey){{0}}, (k))
 
-static struct sipkey *sip_tokey(struct sipkey *key, const void *src) {
-	key->k[0] = SIP_U8TO64_LE((const unsigned char *)src);
-	key->k[1] = SIP_U8TO64_LE((const unsigned char *)src + 8);
-	return key;
+static struct sipkey *
+sip_tokey(struct sipkey *key, const void *src) {
+  key->k[0] = SIP_U8TO64_LE((const unsigned char *)src);
+  key->k[1] = SIP_U8TO64_LE((const unsigned char *)src + 8);
+  return key;
 } /* sip_tokey() */
 
+#ifdef SIPHASH_TOBIN
 
-#define sip_binof(v) sip_tobin((unsigned char[8]){ 0 }, (v))
+#  define sip_binof(v) sip_tobin((unsigned char[8]){0}, (v))
 
-static void *sip_tobin(void *dst, uint64_t u64) {
-	SIP_U64TO8_LE((unsigned char *)dst, u64);
-	return dst;
+static void *
+sip_tobin(void *dst, uint64_t u64) {
+  SIP_U64TO8_LE((unsigned char *)dst, u64);
+  return dst;
 } /* sip_tobin() */
 
+#endif /* SIPHASH_TOBIN */
 
-static void sip_round(struct siphash *H, const int rounds) {
-	int i;
+static void
+sip_round(struct siphash *H, const int rounds) {
+  int i;
 
-	for (i = 0; i < rounds; i++) {
-		H->v0 += H->v1;
-		H->v1 = SIP_ROTL(H->v1, 13);
-		H->v1 ^= H->v0;
-		H->v0 = SIP_ROTL(H->v0, 32);
+  for (i = 0; i < rounds; i++) {
+    H->v0 += H->v1;
+    H->v1 = SIP_ROTL(H->v1, 13);
+    H->v1 ^= H->v0;
+    H->v0 = SIP_ROTL(H->v0, 32);
 
-		H->v2 += H->v3;
-		H->v3 = SIP_ROTL(H->v3, 16);
-		H->v3 ^= H->v2;
+    H->v2 += H->v3;
+    H->v3 = SIP_ROTL(H->v3, 16);
+    H->v3 ^= H->v2;
 
-		H->v0 += H->v3;
-		H->v3 = SIP_ROTL(H->v3, 21);
-		H->v3 ^= H->v0;
+    H->v0 += H->v3;
+    H->v3 = SIP_ROTL(H->v3, 21);
+    H->v3 ^= H->v0;
 
-		H->v2 += H->v1;
-		H->v1 = SIP_ROTL(H->v1, 17);
-		H->v1 ^= H->v2;
-		H->v2 = SIP_ROTL(H->v2, 32);
-	}
+    H->v2 += H->v1;
+    H->v1 = SIP_ROTL(H->v1, 17);
+    H->v1 ^= H->v2;
+    H->v2 = SIP_ROTL(H->v2, 32);
+  }
 } /* sip_round() */
 
+static struct siphash *
+sip24_init(struct siphash *H, const struct sipkey *key) {
+  H->v0 = _SIP_ULL(0x736f6d65U, 0x70736575U) ^ key->k[0];
+  H->v1 = _SIP_ULL(0x646f7261U, 0x6e646f6dU) ^ key->k[1];
+  H->v2 = _SIP_ULL(0x6c796765U, 0x6e657261U) ^ key->k[0];
+  H->v3 = _SIP_ULL(0x74656462U, 0x79746573U) ^ key->k[1];
 
-static struct siphash *sip24_init(struct siphash *H,
-		const struct sipkey *key) {
-	H->v0 = _SIP_ULL(0x736f6d65U, 0x70736575U) ^ key->k[0];
-	H->v1 = _SIP_ULL(0x646f7261U, 0x6e646f6dU) ^ key->k[1];
-	H->v2 = _SIP_ULL(0x6c796765U, 0x6e657261U) ^ key->k[0];
-	H->v3 = _SIP_ULL(0x74656462U, 0x79746573U) ^ key->k[1];
+  H->p = H->buf;
+  H->c = 0;
 
-	H->p = H->buf;
-	H->c = 0;
-
-	return H;
+  return H;
 } /* sip24_init() */
 
+#define sip_endof(a) (&(a)[sizeof(a) / sizeof *(a)])
 
-#define sip_endof(a) (&(a)[sizeof (a) / sizeof *(a)])
-
-static struct siphash *sip24_update(struct siphash *H, const void *src,
-		size_t len) {
-	const unsigned char *p = (const unsigned char *)src, *pe = p + len;
-	uint64_t m;
+static struct siphash *
+sip24_update(struct siphash *H, const void *src, size_t len) {
+  const unsigned char *p = (const unsigned char *)src, *pe = p + len;
+  uint64_t m;
 
-	do {
-		while (p < pe && H->p < sip_endof(H->buf))
-			*H->p++ = *p++;
+  do {
+    while (p < pe && H->p < sip_endof(H->buf))
+      *H->p++ = *p++;
 
-		if (H->p < sip_endof(H->buf))
-			break;
+    if (H->p < sip_endof(H->buf))
+      break;
 
-		m = SIP_U8TO64_LE(H->buf);
-		H->v3 ^= m;
-		sip_round(H, 2);
-		H->v0 ^= m;
+    m = SIP_U8TO64_LE(H->buf);
+    H->v3 ^= m;
+    sip_round(H, 2);
+    H->v0 ^= m;
 
-		H->p = H->buf;
-		H->c += 8;
-	} while (p < pe);
+    H->p = H->buf;
+    H->c += 8;
+  } while (p < pe);
 
-	return H;
+  return H;
 } /* sip24_update() */
 
-
-static uint64_t sip24_final(struct siphash *H) {
-	const char left = (char)(H->p - H->buf);
-	uint64_t b = (H->c + left) << 56;
-
-	switch (left) {
-	case 7: b |= (uint64_t)H->buf[6] << 48;
-	case 6: b |= (uint64_t)H->buf[5] << 40;
-	case 5: b |= (uint64_t)H->buf[4] << 32;
-	case 4: b |= (uint64_t)H->buf[3] << 24;
-	case 3: b |= (uint64_t)H->buf[2] << 16;
-	case 2: b |= (uint64_t)H->buf[1] << 8;
-	case 1: b |= (uint64_t)H->buf[0] << 0;
-	case 0: break;
-	}
-
-	H->v3 ^= b;
-	sip_round(H, 2);
-	H->v0 ^= b;
-	H->v2 ^= 0xff;
-	sip_round(H, 4);
-
-	return H->v0 ^ H->v1 ^ H->v2  ^ H->v3;
+static uint64_t
+sip24_final(struct siphash *H) {
+  const char left = (char)(H->p - H->buf);
+  uint64_t b = (H->c + left) << 56;
+
+  switch (left) {
+  case 7:
+    b |= (uint64_t)H->buf[6] << 48;
+    /* fall through */
+  case 6:
+    b |= (uint64_t)H->buf[5] << 40;
+    /* fall through */
+  case 5:
+    b |= (uint64_t)H->buf[4] << 32;
+    /* fall through */
+  case 4:
+    b |= (uint64_t)H->buf[3] << 24;
+    /* fall through */
+  case 3:
+    b |= (uint64_t)H->buf[2] << 16;
+    /* fall through */
+  case 2:
+    b |= (uint64_t)H->buf[1] << 8;
+    /* fall through */
+  case 1:
+    b |= (uint64_t)H->buf[0] << 0;
+    /* fall through */
+  case 0:
+    break;
+  }
+
+  H->v3 ^= b;
+  sip_round(H, 2);
+  H->v0 ^= b;
+  H->v2 ^= 0xff;
+  sip_round(H, 4);
+
+  return H->v0 ^ H->v1 ^ H->v2 ^ H->v3;
 } /* sip24_final() */
 
-
-static uint64_t siphash24(const void *src, size_t len,
-		const struct sipkey *key) {
-	struct siphash state = SIPHASH_INITIALIZER;
-	return sip24_final(sip24_update(sip24_init(&state, key), src, len));
+static uint64_t
+siphash24(const void *src, size_t len, const struct sipkey *key) {
+  struct siphash state = SIPHASH_INITIALIZER;
+  return sip24_final(sip24_update(sip24_init(&state, key), src, len));
 } /* siphash24() */
 
-
 /*
  * SipHash-2-4 output with
  * k = 00 01 02 ...
@@ -268,107 +289,110 @@ static uint64_t siphash24(const void *src, size_t len,
  * ...
  * in = 00 01 02 ... 3e (63 bytes)
  */
-static int sip24_valid(void) {
-	static const unsigned char vectors[64][8] = {
-		{ 0x31, 0x0e, 0x0e, 0xdd, 0x47, 0xdb, 0x6f, 0x72, },
-		{ 0xfd, 0x67, 0xdc, 0x93, 0xc5, 0x39, 0xf8, 0x74, },
-		{ 0x5a, 0x4f, 0xa9, 0xd9, 0x09, 0x80, 0x6c, 0x0d, },
-		{ 0x2d, 0x7e, 0xfb, 0xd7, 0x96, 0x66, 0x67, 0x85, },
-		{ 0xb7, 0x87, 0x71, 0x27, 0xe0, 0x94, 0x27, 0xcf, },
-		{ 0x8d, 0xa6, 0x99, 0xcd, 0x64, 0x55, 0x76, 0x18, },
-		{ 0xce, 0xe3, 0xfe, 0x58, 0x6e, 0x46, 0xc9, 0xcb, },
-		{ 0x37, 0xd1, 0x01, 0x8b, 0xf5, 0x00, 0x02, 0xab, },
-		{ 0x62, 0x24, 0x93, 0x9a, 0x79, 0xf5, 0xf5, 0x93, },
-		{ 0xb0, 0xe4, 0xa9, 0x0b, 0xdf, 0x82, 0x00, 0x9e, },
-		{ 0xf3, 0xb9, 0xdd, 0x94, 0xc5, 0xbb, 0x5d, 0x7a, },
-		{ 0xa7, 0xad, 0x6b, 0x22, 0x46, 0x2f, 0xb3, 0xf4, },
-		{ 0xfb, 0xe5, 0x0e, 0x86, 0xbc, 0x8f, 0x1e, 0x75, },
-		{ 0x90, 0x3d, 0x84, 0xc0, 0x27, 0x56, 0xea, 0x14, },
-		{ 0xee, 0xf2, 0x7a, 0x8e, 0x90, 0xca, 0x23, 0xf7, },
-		{ 0xe5, 0x45, 0xbe, 0x49, 0x61, 0xca, 0x29, 0xa1, },
-		{ 0xdb, 0x9b, 0xc2, 0x57, 0x7f, 0xcc, 0x2a, 0x3f, },
-		{ 0x94, 0x47, 0xbe, 0x2c, 0xf5, 0xe9, 0x9a, 0x69, },
-		{ 0x9c, 0xd3, 0x8d, 0x96, 0xf0, 0xb3, 0xc1, 0x4b, },
-		{ 0xbd, 0x61, 0x79, 0xa7, 0x1d, 0xc9, 0x6d, 0xbb, },
-		{ 0x98, 0xee, 0xa2, 0x1a, 0xf2, 0x5c, 0xd6, 0xbe, },
-		{ 0xc7, 0x67, 0x3b, 0x2e, 0xb0, 0xcb, 0xf2, 0xd0, },
-		{ 0x88, 0x3e, 0xa3, 0xe3, 0x95, 0x67, 0x53, 0x93, },
-		{ 0xc8, 0xce, 0x5c, 0xcd, 0x8c, 0x03, 0x0c, 0xa8, },
-		{ 0x94, 0xaf, 0x49, 0xf6, 0xc6, 0x50, 0xad, 0xb8, },
-		{ 0xea, 0xb8, 0x85, 0x8a, 0xde, 0x92, 0xe1, 0xbc, },
-		{ 0xf3, 0x15, 0xbb, 0x5b, 0xb8, 0x35, 0xd8, 0x17, },
-		{ 0xad, 0xcf, 0x6b, 0x07, 0x63, 0x61, 0x2e, 0x2f, },
-		{ 0xa5, 0xc9, 0x1d, 0xa7, 0xac, 0xaa, 0x4d, 0xde, },
-		{ 0x71, 0x65, 0x95, 0x87, 0x66, 0x50, 0xa2, 0xa6, },
-		{ 0x28, 0xef, 0x49, 0x5c, 0x53, 0xa3, 0x87, 0xad, },
-		{ 0x42, 0xc3, 0x41, 0xd8, 0xfa, 0x92, 0xd8, 0x32, },
-		{ 0xce, 0x7c, 0xf2, 0x72, 0x2f, 0x51, 0x27, 0x71, },
-		{ 0xe3, 0x78, 0x59, 0xf9, 0x46, 0x23, 0xf3, 0xa7, },
-		{ 0x38, 0x12, 0x05, 0xbb, 0x1a, 0xb0, 0xe0, 0x12, },
-		{ 0xae, 0x97, 0xa1, 0x0f, 0xd4, 0x34, 0xe0, 0x15, },
-		{ 0xb4, 0xa3, 0x15, 0x08, 0xbe, 0xff, 0x4d, 0x31, },
-		{ 0x81, 0x39, 0x62, 0x29, 0xf0, 0x90, 0x79, 0x02, },
-		{ 0x4d, 0x0c, 0xf4, 0x9e, 0xe5, 0xd4, 0xdc, 0xca, },
-		{ 0x5c, 0x73, 0x33, 0x6a, 0x76, 0xd8, 0xbf, 0x9a, },
-		{ 0xd0, 0xa7, 0x04, 0x53, 0x6b, 0xa9, 0x3e, 0x0e, },
-		{ 0x92, 0x59, 0x58, 0xfc, 0xd6, 0x42, 0x0c, 0xad, },
-		{ 0xa9, 0x15, 0xc2, 0x9b, 0xc8, 0x06, 0x73, 0x18, },
-		{ 0x95, 0x2b, 0x79, 0xf3, 0xbc, 0x0a, 0xa6, 0xd4, },
-		{ 0xf2, 0x1d, 0xf2, 0xe4, 0x1d, 0x45, 0x35, 0xf9, },
-		{ 0x87, 0x57, 0x75, 0x19, 0x04, 0x8f, 0x53, 0xa9, },
-		{ 0x10, 0xa5, 0x6c, 0xf5, 0xdf, 0xcd, 0x9a, 0xdb, },
-		{ 0xeb, 0x75, 0x09, 0x5c, 0xcd, 0x98, 0x6c, 0xd0, },
-		{ 0x51, 0xa9, 0xcb, 0x9e, 0xcb, 0xa3, 0x12, 0xe6, },
-		{ 0x96, 0xaf, 0xad, 0xfc, 0x2c, 0xe6, 0x66, 0xc7, },
-		{ 0x72, 0xfe, 0x52, 0x97, 0x5a, 0x43, 0x64, 0xee, },
-		{ 0x5a, 0x16, 0x45, 0xb2, 0x76, 0xd5, 0x92, 0xa1, },
-		{ 0xb2, 0x74, 0xcb, 0x8e, 0xbf, 0x87, 0x87, 0x0a, },
-		{ 0x6f, 0x9b, 0xb4, 0x20, 0x3d, 0xe7, 0xb3, 0x81, },
-		{ 0xea, 0xec, 0xb2, 0xa3, 0x0b, 0x22, 0xa8, 0x7f, },
-		{ 0x99, 0x24, 0xa4, 0x3c, 0xc1, 0x31, 0x57, 0x24, },
-		{ 0xbd, 0x83, 0x8d, 0x3a, 0xaf, 0xbf, 0x8d, 0xb7, },
-		{ 0x0b, 0x1a, 0x2a, 0x32, 0x65, 0xd5, 0x1a, 0xea, },
-		{ 0x13, 0x50, 0x79, 0xa3, 0x23, 0x1c, 0xe6, 0x60, },
-		{ 0x93, 0x2b, 0x28, 0x46, 0xe4, 0xd7, 0x06, 0x66, },
-		{ 0xe1, 0x91, 0x5f, 0x5c, 0xb1, 0xec, 0xa4, 0x6c, },
-		{ 0xf3, 0x25, 0x96, 0x5c, 0xa1, 0x6d, 0x62, 0x9f, },
-		{ 0x57, 0x5f, 0xf2, 0x8e, 0x60, 0x38, 0x1b, 0xe5, },
-		{ 0x72, 0x45, 0x06, 0xeb, 0x4c, 0x32, 0x8a, 0x95, }
-	};
-	unsigned char in[64];
-	struct sipkey k;
-	size_t i;
-
-	sip_tokey(&k, "\000\001\002\003\004\005\006\007\010\011"
-			"\012\013\014\015\016\017");
-
-	for (i = 0; i < sizeof in; ++i) {
-		in[i] = (unsigned char)i;
-
-		if (siphash24(in, i, &k) != SIP_U8TO64_LE(vectors[i]))
-			return 0;
-	}
-
-	return 1;
+static int
+sip24_valid(void) {
+  /* clang-format off */
+  static const unsigned char vectors[64][8] = {
+    { 0x31, 0x0e, 0x0e, 0xdd, 0x47, 0xdb, 0x6f, 0x72, },
+    { 0xfd, 0x67, 0xdc, 0x93, 0xc5, 0x39, 0xf8, 0x74, },
+    { 0x5a, 0x4f, 0xa9, 0xd9, 0x09, 0x80, 0x6c, 0x0d, },
+    { 0x2d, 0x7e, 0xfb, 0xd7, 0x96, 0x66, 0x67, 0x85, },
+    { 0xb7, 0x87, 0x71, 0x27, 0xe0, 0x94, 0x27, 0xcf, },
+    { 0x8d, 0xa6, 0x99, 0xcd, 0x64, 0x55, 0x76, 0x18, },
+    { 0xce, 0xe3, 0xfe, 0x58, 0x6e, 0x46, 0xc9, 0xcb, },
+    { 0x37, 0xd1, 0x01, 0x8b, 0xf5, 0x00, 0x02, 0xab, },
+    { 0x62, 0x24, 0x93, 0x9a, 0x79, 0xf5, 0xf5, 0x93, },
+    { 0xb0, 0xe4, 0xa9, 0x0b, 0xdf, 0x82, 0x00, 0x9e, },
+    { 0xf3, 0xb9, 0xdd, 0x94, 0xc5, 0xbb, 0x5d, 0x7a, },
+    { 0xa7, 0xad, 0x6b, 0x22, 0x46, 0x2f, 0xb3, 0xf4, },
+    { 0xfb, 0xe5, 0x0e, 0x86, 0xbc, 0x8f, 0x1e, 0x75, },
+    { 0x90, 0x3d, 0x84, 0xc0, 0x27, 0x56, 0xea, 0x14, },
+    { 0xee, 0xf2, 0x7a, 0x8e, 0x90, 0xca, 0x23, 0xf7, },
+    { 0xe5, 0x45, 0xbe, 0x49, 0x61, 0xca, 0x29, 0xa1, },
+    { 0xdb, 0x9b, 0xc2, 0x57, 0x7f, 0xcc, 0x2a, 0x3f, },
+    { 0x94, 0x47, 0xbe, 0x2c, 0xf5, 0xe9, 0x9a, 0x69, },
+    { 0x9c, 0xd3, 0x8d, 0x96, 0xf0, 0xb3, 0xc1, 0x4b, },
+    { 0xbd, 0x61, 0x79, 0xa7, 0x1d, 0xc9, 0x6d, 0xbb, },
+    { 0x98, 0xee, 0xa2, 0x1a, 0xf2, 0x5c, 0xd6, 0xbe, },
+    { 0xc7, 0x67, 0x3b, 0x2e, 0xb0, 0xcb, 0xf2, 0xd0, },
+    { 0x88, 0x3e, 0xa3, 0xe3, 0x95, 0x67, 0x53, 0x93, },
+    { 0xc8, 0xce, 0x5c, 0xcd, 0x8c, 0x03, 0x0c, 0xa8, },
+    { 0x94, 0xaf, 0x49, 0xf6, 0xc6, 0x50, 0xad, 0xb8, },
+    { 0xea, 0xb8, 0x85, 0x8a, 0xde, 0x92, 0xe1, 0xbc, },
+    { 0xf3, 0x15, 0xbb, 0x5b, 0xb8, 0x35, 0xd8, 0x17, },
+    { 0xad, 0xcf, 0x6b, 0x07, 0x63, 0x61, 0x2e, 0x2f, },
+    { 0xa5, 0xc9, 0x1d, 0xa7, 0xac, 0xaa, 0x4d, 0xde, },
+    { 0x71, 0x65, 0x95, 0x87, 0x66, 0x50, 0xa2, 0xa6, },
+    { 0x28, 0xef, 0x49, 0x5c, 0x53, 0xa3, 0x87, 0xad, },
+    { 0x42, 0xc3, 0x41, 0xd8, 0xfa, 0x92, 0xd8, 0x32, },
+    { 0xce, 0x7c, 0xf2, 0x72, 0x2f, 0x51, 0x27, 0x71, },
+    { 0xe3, 0x78, 0x59, 0xf9, 0x46, 0x23, 0xf3, 0xa7, },
+    { 0x38, 0x12, 0x05, 0xbb, 0x1a, 0xb0, 0xe0, 0x12, },
+    { 0xae, 0x97, 0xa1, 0x0f, 0xd4, 0x34, 0xe0, 0x15, },
+    { 0xb4, 0xa3, 0x15, 0x08, 0xbe, 0xff, 0x4d, 0x31, },
+    { 0x81, 0x39, 0x62, 0x29, 0xf0, 0x90, 0x79, 0x02, },
+    { 0x4d, 0x0c, 0xf4, 0x9e, 0xe5, 0xd4, 0xdc, 0xca, },
+    { 0x5c, 0x73, 0x33, 0x6a, 0x76, 0xd8, 0xbf, 0x9a, },
+    { 0xd0, 0xa7, 0x04, 0x53, 0x6b, 0xa9, 0x3e, 0x0e, },
+    { 0x92, 0x59, 0x58, 0xfc, 0xd6, 0x42, 0x0c, 0xad, },
+    { 0xa9, 0x15, 0xc2, 0x9b, 0xc8, 0x06, 0x73, 0x18, },
+    { 0x95, 0x2b, 0x79, 0xf3, 0xbc, 0x0a, 0xa6, 0xd4, },
+    { 0xf2, 0x1d, 0xf2, 0xe4, 0x1d, 0x45, 0x35, 0xf9, },
+    { 0x87, 0x57, 0x75, 0x19, 0x04, 0x8f, 0x53, 0xa9, },
+    { 0x10, 0xa5, 0x6c, 0xf5, 0xdf, 0xcd, 0x9a, 0xdb, },
+    { 0xeb, 0x75, 0x09, 0x5c, 0xcd, 0x98, 0x6c, 0xd0, },
+    { 0x51, 0xa9, 0xcb, 0x9e, 0xcb, 0xa3, 0x12, 0xe6, },
+    { 0x96, 0xaf, 0xad, 0xfc, 0x2c, 0xe6, 0x66, 0xc7, },
+    { 0x72, 0xfe, 0x52, 0x97, 0x5a, 0x43, 0x64, 0xee, },
+    { 0x5a, 0x16, 0x45, 0xb2, 0x76, 0xd5, 0x92, 0xa1, },
+    { 0xb2, 0x74, 0xcb, 0x8e, 0xbf, 0x87, 0x87, 0x0a, },
+    { 0x6f, 0x9b, 0xb4, 0x20, 0x3d, 0xe7, 0xb3, 0x81, },
+    { 0xea, 0xec, 0xb2, 0xa3, 0x0b, 0x22, 0xa8, 0x7f, },
+    { 0x99, 0x24, 0xa4, 0x3c, 0xc1, 0x31, 0x57, 0x24, },
+    { 0xbd, 0x83, 0x8d, 0x3a, 0xaf, 0xbf, 0x8d, 0xb7, },
+    { 0x0b, 0x1a, 0x2a, 0x32, 0x65, 0xd5, 0x1a, 0xea, },
+    { 0x13, 0x50, 0x79, 0xa3, 0x23, 0x1c, 0xe6, 0x60, },
+    { 0x93, 0x2b, 0x28, 0x46, 0xe4, 0xd7, 0x06, 0x66, },
+    { 0xe1, 0x91, 0x5f, 0x5c, 0xb1, 0xec, 0xa4, 0x6c, },
+    { 0xf3, 0x25, 0x96, 0x5c, 0xa1, 0x6d, 0x62, 0x9f, },
+    { 0x57, 0x5f, 0xf2, 0x8e, 0x60, 0x38, 0x1b, 0xe5, },
+    { 0x72, 0x45, 0x06, 0xeb, 0x4c, 0x32, 0x8a, 0x95, }
+  };
+  /* clang-format on */
+
+  unsigned char in[64];
+  struct sipkey k;
+  size_t i;
+
+  sip_tokey(&k, "\000\001\002\003\004\005\006\007\010\011"
+                "\012\013\014\015\016\017");
+
+  for (i = 0; i < sizeof in; ++i) {
+    in[i] = (unsigned char)i;
+
+    if (siphash24(in, i, &k) != SIP_U8TO64_LE(vectors[i]))
+      return 0;
+  }
+
+  return 1;
 } /* sip24_valid() */
 
-
 #ifdef SIPHASH_MAIN
 
-#include <stdio.h>
+#  include <stdio.h>
 
-int main(void) {
-	const int ok = sip24_valid();
+int
+main(void) {
+  const int ok = sip24_valid();
 
-	if (ok)
-		puts("OK");
-	else
-		puts("FAIL");
+  if (ok)
+    puts("OK");
+  else
+    puts("FAIL");
 
-	return !ok;
+  return ! ok;
 } /* main() */
 
 #endif /* SIPHASH_MAIN */
 
-
 #endif /* SIPHASH_H */
diff --git a/Modules/expat/utf8tab.h b/Modules/expat/utf8tab.h
index fa0bed6f5d751..a22986acbb952 100644
--- a/Modules/expat/utf8tab.h
+++ b/Modules/expat/utf8tab.h
@@ -31,34 +31,34 @@
 */
 
 /* 0x80 */ BT_TRAIL, BT_TRAIL, BT_TRAIL, BT_TRAIL,
-/* 0x84 */ BT_TRAIL, BT_TRAIL, BT_TRAIL, BT_TRAIL,
-/* 0x88 */ BT_TRAIL, BT_TRAIL, BT_TRAIL, BT_TRAIL,
-/* 0x8C */ BT_TRAIL, BT_TRAIL, BT_TRAIL, BT_TRAIL,
-/* 0x90 */ BT_TRAIL, BT_TRAIL, BT_TRAIL, BT_TRAIL,
-/* 0x94 */ BT_TRAIL, BT_TRAIL, BT_TRAIL, BT_TRAIL,
-/* 0x98 */ BT_TRAIL, BT_TRAIL, BT_TRAIL, BT_TRAIL,
-/* 0x9C */ BT_TRAIL, BT_TRAIL, BT_TRAIL, BT_TRAIL,
-/* 0xA0 */ BT_TRAIL, BT_TRAIL, BT_TRAIL, BT_TRAIL,
-/* 0xA4 */ BT_TRAIL, BT_TRAIL, BT_TRAIL, BT_TRAIL,
-/* 0xA8 */ BT_TRAIL, BT_TRAIL, BT_TRAIL, BT_TRAIL,
-/* 0xAC */ BT_TRAIL, BT_TRAIL, BT_TRAIL, BT_TRAIL,
-/* 0xB0 */ BT_TRAIL, BT_TRAIL, BT_TRAIL, BT_TRAIL,
-/* 0xB4 */ BT_TRAIL, BT_TRAIL, BT_TRAIL, BT_TRAIL,
-/* 0xB8 */ BT_TRAIL, BT_TRAIL, BT_TRAIL, BT_TRAIL,
-/* 0xBC */ BT_TRAIL, BT_TRAIL, BT_TRAIL, BT_TRAIL,
-/* 0xC0 */ BT_LEAD2, BT_LEAD2, BT_LEAD2, BT_LEAD2,
-/* 0xC4 */ BT_LEAD2, BT_LEAD2, BT_LEAD2, BT_LEAD2,
-/* 0xC8 */ BT_LEAD2, BT_LEAD2, BT_LEAD2, BT_LEAD2,
-/* 0xCC */ BT_LEAD2, BT_LEAD2, BT_LEAD2, BT_LEAD2,
-/* 0xD0 */ BT_LEAD2, BT_LEAD2, BT_LEAD2, BT_LEAD2,
-/* 0xD4 */ BT_LEAD2, BT_LEAD2, BT_LEAD2, BT_LEAD2,
-/* 0xD8 */ BT_LEAD2, BT_LEAD2, BT_LEAD2, BT_LEAD2,
-/* 0xDC */ BT_LEAD2, BT_LEAD2, BT_LEAD2, BT_LEAD2,
-/* 0xE0 */ BT_LEAD3, BT_LEAD3, BT_LEAD3, BT_LEAD3,
-/* 0xE4 */ BT_LEAD3, BT_LEAD3, BT_LEAD3, BT_LEAD3,
-/* 0xE8 */ BT_LEAD3, BT_LEAD3, BT_LEAD3, BT_LEAD3,
-/* 0xEC */ BT_LEAD3, BT_LEAD3, BT_LEAD3, BT_LEAD3,
-/* 0xF0 */ BT_LEAD4, BT_LEAD4, BT_LEAD4, BT_LEAD4,
-/* 0xF4 */ BT_LEAD4, BT_NONXML, BT_NONXML, BT_NONXML,
-/* 0xF8 */ BT_NONXML, BT_NONXML, BT_NONXML, BT_NONXML,
-/* 0xFC */ BT_NONXML, BT_NONXML, BT_MALFORM, BT_MALFORM,
+    /* 0x84 */ BT_TRAIL, BT_TRAIL, BT_TRAIL, BT_TRAIL,
+    /* 0x88 */ BT_TRAIL, BT_TRAIL, BT_TRAIL, BT_TRAIL,
+    /* 0x8C */ BT_TRAIL, BT_TRAIL, BT_TRAIL, BT_TRAIL,
+    /* 0x90 */ BT_TRAIL, BT_TRAIL, BT_TRAIL, BT_TRAIL,
+    /* 0x94 */ BT_TRAIL, BT_TRAIL, BT_TRAIL, BT_TRAIL,
+    /* 0x98 */ BT_TRAIL, BT_TRAIL, BT_TRAIL, BT_TRAIL,
+    /* 0x9C */ BT_TRAIL, BT_TRAIL, BT_TRAIL, BT_TRAIL,
+    /* 0xA0 */ BT_TRAIL, BT_TRAIL, BT_TRAIL, BT_TRAIL,
+    /* 0xA4 */ BT_TRAIL, BT_TRAIL, BT_TRAIL, BT_TRAIL,
+    /* 0xA8 */ BT_TRAIL, BT_TRAIL, BT_TRAIL, BT_TRAIL,
+    /* 0xAC */ BT_TRAIL, BT_TRAIL, BT_TRAIL, BT_TRAIL,
+    /* 0xB0 */ BT_TRAIL, BT_TRAIL, BT_TRAIL, BT_TRAIL,
+    /* 0xB4 */ BT_TRAIL, BT_TRAIL, BT_TRAIL, BT_TRAIL,
+    /* 0xB8 */ BT_TRAIL, BT_TRAIL, BT_TRAIL, BT_TRAIL,
+    /* 0xBC */ BT_TRAIL, BT_TRAIL, BT_TRAIL, BT_TRAIL,
+    /* 0xC0 */ BT_LEAD2, BT_LEAD2, BT_LEAD2, BT_LEAD2,
+    /* 0xC4 */ BT_LEAD2, BT_LEAD2, BT_LEAD2, BT_LEAD2,
+    /* 0xC8 */ BT_LEAD2, BT_LEAD2, BT_LEAD2, BT_LEAD2,
+    /* 0xCC */ BT_LEAD2, BT_LEAD2, BT_LEAD2, BT_LEAD2,
+    /* 0xD0 */ BT_LEAD2, BT_LEAD2, BT_LEAD2, BT_LEAD2,
+    /* 0xD4 */ BT_LEAD2, BT_LEAD2, BT_LEAD2, BT_LEAD2,
+    /* 0xD8 */ BT_LEAD2, BT_LEAD2, BT_LEAD2, BT_LEAD2,
+    /* 0xDC */ BT_LEAD2, BT_LEAD2, BT_LEAD2, BT_LEAD2,
+    /* 0xE0 */ BT_LEAD3, BT_LEAD3, BT_LEAD3, BT_LEAD3,
+    /* 0xE4 */ BT_LEAD3, BT_LEAD3, BT_LEAD3, BT_LEAD3,
+    /* 0xE8 */ BT_LEAD3, BT_LEAD3, BT_LEAD3, BT_LEAD3,
+    /* 0xEC */ BT_LEAD3, BT_LEAD3, BT_LEAD3, BT_LEAD3,
+    /* 0xF0 */ BT_LEAD4, BT_LEAD4, BT_LEAD4, BT_LEAD4,
+    /* 0xF4 */ BT_LEAD4, BT_NONXML, BT_NONXML, BT_NONXML,
+    /* 0xF8 */ BT_NONXML, BT_NONXML, BT_NONXML, BT_NONXML,
+    /* 0xFC */ BT_NONXML, BT_NONXML, BT_MALFORM, BT_MALFORM,
diff --git a/Modules/expat/winconfig.h b/Modules/expat/winconfig.h
index 17fea4689001f..562a4a82dc1d6 100644
--- a/Modules/expat/winconfig.h
+++ b/Modules/expat/winconfig.h
@@ -40,24 +40,17 @@
 #include <memory.h>
 #include <string.h>
 
+#if defined(HAVE_EXPAT_CONFIG_H) /* e.g. MinGW */
+#  include <expat_config.h>
+#else /* !defined(HAVE_EXPAT_CONFIG_H) */
 
-#if defined(HAVE_EXPAT_CONFIG_H)  /* e.g. MinGW */
-# include <expat_config.h>
-#else  /* !defined(HAVE_EXPAT_CONFIG_H) */
-
-
-#define XML_NS 1
-#define XML_DTD 1
-#define XML_CONTEXT_BYTES 1024
+#  define XML_NS 1
+#  define XML_DTD 1
+#  define XML_CONTEXT_BYTES 1024
 
 /* we will assume all Windows platforms are little endian */
-#define BYTEORDER 1234
-
-/* Windows has memmove() available. */
-#define HAVE_MEMMOVE
-
+#  define BYTEORDER 1234
 
 #endif /* !defined(HAVE_EXPAT_CONFIG_H) */
 
-
 #endif /* ndef WINCONFIG_H */
diff --git a/Modules/expat/xmlparse.c b/Modules/expat/xmlparse.c
index 0df68830f05e3..e740f0e19c7d4 100644
--- a/Modules/expat/xmlparse.c
+++ b/Modules/expat/xmlparse.c
@@ -1,4 +1,4 @@
-/* 8c6b2be7c6281da65ce05218fc15c339f02a811706340824ab596aa86e1fd51a (2.2.4+)
+/* f2d0ab6d1d4422a08cf1cf3bbdfba96b49dea42fb5ff4615e03a2a25c306e769 (2.2.8+)
                             __  __            _
                          ___\ \/ /_ __   __ _| |_
                         / _ \\  /| '_ \ / _` | __|
@@ -30,33 +30,38 @@
    USE OR OTHER DEALINGS IN THE SOFTWARE.
 */
 
-#if !defined(_GNU_SOURCE)
-# define _GNU_SOURCE 1                  /* syscall prototype */
+#if ! defined(_GNU_SOURCE)
+#  define _GNU_SOURCE 1 /* syscall prototype */
+#endif
+
+#ifdef _WIN32
+/* force stdlib to define rand_s() */
+#  define _CRT_RAND_S
 #endif
 
 #include <stddef.h>
-#include <string.h>                     /* memset(), memcpy() */
+#include <string.h> /* memset(), memcpy() */
 #include <assert.h>
-#include <limits.h>                     /* UINT_MAX */
-#include <stdio.h>                      /* fprintf */
-#include <stdlib.h>                     /* getenv */
+#include <limits.h> /* UINT_MAX */
+#include <stdio.h>  /* fprintf */
+#include <stdlib.h> /* getenv, rand_s */
 
 #ifdef _WIN32
-#define getpid GetCurrentProcessId
+#  define getpid GetCurrentProcessId
 #else
-#include <sys/time.h>                   /* gettimeofday() */
-#include <sys/types.h>                  /* getpid() */
-#include <unistd.h>                     /* getpid() */
-#include <fcntl.h>                      /* O_RDONLY */
-#include <errno.h>
+#  include <sys/time.h>  /* gettimeofday() */
+#  include <sys/types.h> /* getpid() */
+#  include <unistd.h>    /* getpid() */
+#  include <fcntl.h>     /* O_RDONLY */
+#  include <errno.h>
 #endif
 
 #define XML_BUILDING_EXPAT 1
 
 #ifdef _WIN32
-#include "winconfig.h"
+#  include "winconfig.h"
 #elif defined(HAVE_EXPAT_CONFIG_H)
-#include <expat_config.h>
+#  include <expat_config.h>
 #endif /* ndef _WIN32 */
 
 #include "ascii.h"
@@ -64,33 +69,31 @@
 #include "siphash.h"
 
 #if defined(HAVE_GETRANDOM) || defined(HAVE_SYSCALL_GETRANDOM)
-# if defined(HAVE_GETRANDOM)
-#  include <sys/random.h>    /* getrandom */
-# else
-#  include <unistd.h>        /* syscall */
-#  include <sys/syscall.h>   /* SYS_getrandom */
-# endif
-# if ! defined(GRND_NONBLOCK)
-#  define GRND_NONBLOCK  0x0001
-# endif  /* defined(GRND_NONBLOCK) */
-#endif  /* defined(HAVE_GETRANDOM) || defined(HAVE_SYSCALL_GETRANDOM) */
-
-#if defined(HAVE_LIBBSD) \
+#  if defined(HAVE_GETRANDOM)
+#    include <sys/random.h> /* getrandom */
+#  else
+#    include <unistd.h>      /* syscall */
+#    include <sys/syscall.h> /* SYS_getrandom */
+#  endif
+#  if ! defined(GRND_NONBLOCK)
+#    define GRND_NONBLOCK 0x0001
+#  endif /* defined(GRND_NONBLOCK) */
+#endif   /* defined(HAVE_GETRANDOM) || defined(HAVE_SYSCALL_GETRANDOM) */
+
+#if defined(HAVE_LIBBSD)                                                       \
     && (defined(HAVE_ARC4RANDOM_BUF) || defined(HAVE_ARC4RANDOM))
-# include <bsd/stdlib.h>
+#  include <bsd/stdlib.h>
 #endif
 
-#if defined(_WIN32) && !defined(LOAD_LIBRARY_SEARCH_SYSTEM32)
-# define LOAD_LIBRARY_SEARCH_SYSTEM32  0x00000800
+#if defined(_WIN32) && ! defined(LOAD_LIBRARY_SEARCH_SYSTEM32)
+#  define LOAD_LIBRARY_SEARCH_SYSTEM32 0x00000800
 #endif
 
-#if !defined(HAVE_GETRANDOM) && !defined(HAVE_SYSCALL_GETRANDOM) \
-    && !defined(HAVE_ARC4RANDOM_BUF) && !defined(HAVE_ARC4RANDOM) \
-    && !defined(XML_DEV_URANDOM) \
-    && !defined(_WIN32) \
-    && !defined(XML_POOR_ENTROPY)
-# error  \
-    You do not have support for any sources of high quality entropy \
+#if ! defined(HAVE_GETRANDOM) && ! defined(HAVE_SYSCALL_GETRANDOM)             \
+    && ! defined(HAVE_ARC4RANDOM_BUF) && ! defined(HAVE_ARC4RANDOM)            \
+    && ! defined(XML_DEV_URANDOM) && ! defined(_WIN32)                         \
+    && ! defined(XML_POOR_ENTROPY)
+#  error You do not have support for any sources of high quality entropy \
     enabled.  For end user security, that is probably not what you want. \
     \
     Your options include: \
@@ -101,7 +104,7 @@
       * libbsd (arc4random_buf): HAVE_ARC4RANDOM_BUF + HAVE_LIBBSD, \
       * libbsd (arc4random): HAVE_ARC4RANDOM + HAVE_LIBBSD, \
       * Linux / BSD / macOS (/dev/urandom): XML_DEV_URANDOM \
-      * Windows (RtlGenRandom): _WIN32. \
+      * Windows (rand_s): _WIN32. \
     \
     If insist on not using any of these, bypass this error by defining \
     XML_POOR_ENTROPY; you have been warned. \
@@ -110,65 +113,58 @@
     to the build system, please open a bug.  Thank you!
 #endif
 
-
 #ifdef XML_UNICODE
-#define XML_ENCODE_MAX XML_UTF16_ENCODE_MAX
-#define XmlConvert XmlUtf16Convert
-#define XmlGetInternalEncoding XmlGetUtf16InternalEncoding
-#define XmlGetInternalEncodingNS XmlGetUtf16InternalEncodingNS
-#define XmlEncode XmlUtf16Encode
+#  define XML_ENCODE_MAX XML_UTF16_ENCODE_MAX
+#  define XmlConvert XmlUtf16Convert
+#  define XmlGetInternalEncoding XmlGetUtf16InternalEncoding
+#  define XmlGetInternalEncodingNS XmlGetUtf16InternalEncodingNS
+#  define XmlEncode XmlUtf16Encode
 /* Using pointer subtraction to convert to integer type. */
-#define MUST_CONVERT(enc, s) (!(enc)->isUtf16 || (((char *)(s) - (char *)NULL) & 1))
+#  define MUST_CONVERT(enc, s)                                                 \
+    (! (enc)->isUtf16 || (((char *)(s) - (char *)NULL) & 1))
 typedef unsigned short ICHAR;
 #else
-#define XML_ENCODE_MAX XML_UTF8_ENCODE_MAX
-#define XmlConvert XmlUtf8Convert
-#define XmlGetInternalEncoding XmlGetUtf8InternalEncoding
-#define XmlGetInternalEncodingNS XmlGetUtf8InternalEncodingNS
-#define XmlEncode XmlUtf8Encode
-#define MUST_CONVERT(enc, s) (!(enc)->isUtf8)
+#  define XML_ENCODE_MAX XML_UTF8_ENCODE_MAX
+#  define XmlConvert XmlUtf8Convert
+#  define XmlGetInternalEncoding XmlGetUtf8InternalEncoding
+#  define XmlGetInternalEncodingNS XmlGetUtf8InternalEncodingNS
+#  define XmlEncode XmlUtf8Encode
+#  define MUST_CONVERT(enc, s) (! (enc)->isUtf8)
 typedef char ICHAR;
 #endif
 
-
 #ifndef XML_NS
 
-#define XmlInitEncodingNS XmlInitEncoding
-#define XmlInitUnknownEncodingNS XmlInitUnknownEncoding
-#undef XmlGetInternalEncodingNS
-#define XmlGetInternalEncodingNS XmlGetInternalEncoding
-#define XmlParseXmlDeclNS XmlParseXmlDecl
+#  define XmlInitEncodingNS XmlInitEncoding
+#  define XmlInitUnknownEncodingNS XmlInitUnknownEncoding
+#  undef XmlGetInternalEncodingNS
+#  define XmlGetInternalEncodingNS XmlGetInternalEncoding
+#  define XmlParseXmlDeclNS XmlParseXmlDecl
 
 #endif
 
 #ifdef XML_UNICODE
 
-#ifdef XML_UNICODE_WCHAR_T
-#define XML_T(x) (const wchar_t)x
-#define XML_L(x) L ## x
-#else
-#define XML_T(x) (const unsigned short)x
-#define XML_L(x) x
-#endif
+#  ifdef XML_UNICODE_WCHAR_T
+#    define XML_T(x) (const wchar_t) x
+#    define XML_L(x) L##x
+#  else
+#    define XML_T(x) (const unsigned short)x
+#    define XML_L(x) x
+#  endif
 
 #else
 
-#define XML_T(x) x
-#define XML_L(x) x
+#  define XML_T(x) x
+#  define XML_L(x) x
 
 #endif
 
 /* Round up n to be a multiple of sz, where sz is a power of 2. */
-#define ROUND_UP(n, sz) (((n) + ((sz) - 1)) & ~((sz) - 1))
+#define ROUND_UP(n, sz) (((n) + ((sz)-1)) & ~((sz)-1))
 
-/* Handle the case where memmove() doesn't exist. */
-#ifndef HAVE_MEMMOVE
-#ifdef HAVE_BCOPY
-#define memmove(d,s,l) bcopy((s),(d),(l))
-#else
-#error memmove does not exist on this platform, nor is a substitute available
-#endif /* HAVE_BCOPY */
-#endif /* HAVE_MEMMOVE */
+/* Do safe (NULL-aware) pointer arithmetic */
+#define EXPAT_SAFE_PTR_DIFF(p, q) (((p) && (q)) ? ((p) - (q)) : 0)
 
 #include "internal.h"
 #include "xmltok.h"
@@ -188,11 +184,9 @@ typedef struct {
   const XML_Memory_Handling_Suite *mem;
 } HASH_TABLE;
 
-static size_t
-keylen(KEY s);
+static size_t keylen(KEY s);
 
-static void
-copy_salt_to_sipkey(XML_Parser parser, struct sipkey * key);
+static void copy_salt_to_sipkey(XML_Parser parser, struct sipkey *key);
 
 /* For probing (after a collision) we need a step size relative prime
    to the hash table size, which is a power of 2. We use double-hashing,
@@ -202,9 +196,9 @@ copy_salt_to_sipkey(XML_Parser parser, struct sipkey * key);
    We limit the maximum step size to table->size / 4 (mask >> 2) and make
    it odd, since odd numbers are always relative prime to a power of 2.
 */
-#define SECOND_HASH(hash, mask, power) \
-  ((((hash) & ~(mask)) >> ((power) - 1)) & ((mask) >> 2))
-#define PROBE_STEP(hash, mask, power) \
+#define SECOND_HASH(hash, mask, power)                                         \
+  ((((hash) & ~(mask)) >> ((power)-1)) & ((mask) >> 2))
+#define PROBE_STEP(hash, mask, power)                                          \
   ((unsigned char)((SECOND_HASH(hash, mask, power)) | 1))
 
 typedef struct {
@@ -212,7 +206,7 @@ typedef struct {
   NAMED **end;
 } HASH_TABLE_ITER;
 
-#define INIT_TAG_BUF_SIZE 32  /* must be a multiple of sizeof(XML_Char) */
+#define INIT_TAG_BUF_SIZE 32 /* must be a multiple of sizeof(XML_Char) */
 #define INIT_DATA_BUF_SIZE 1024
 #define INIT_ATTS_SIZE 16
 #define INIT_ATTS_VERSION 0xFFFFFFFF
@@ -259,20 +253,20 @@ typedef struct {
    TAG objects in a free list.
 */
 typedef struct tag {
-  struct tag *parent;           /* parent of this element */
-  const char *rawName;          /* tagName in the original encoding */
+  struct tag *parent;  /* parent of this element */
+  const char *rawName; /* tagName in the original encoding */
   int rawNameLength;
-  TAG_NAME name;                /* tagName in the API encoding */
-  char *buf;                    /* buffer for name components */
-  char *bufEnd;                 /* end of the buffer */
+  TAG_NAME name; /* tagName in the API encoding */
+  char *buf;     /* buffer for name components */
+  char *bufEnd;  /* end of the buffer */
   BINDING *bindings;
 } TAG;
 
 typedef struct {
   const XML_Char *name;
   const XML_Char *textPtr;
-  int textLen;                  /* length in XML_Chars */
-  int processed;                /* # of processed bytes - when suspended */
+  int textLen;   /* length in XML_Chars */
+  int processed; /* # of processed bytes - when suspended */
   const XML_Char *systemId;
   const XML_Char *base;
   const XML_Char *publicId;
@@ -283,13 +277,13 @@ typedef struct {
 } ENTITY;
 
 typedef struct {
-  enum XML_Content_Type         type;
-  enum XML_Content_Quant        quant;
-  const XML_Char *              name;
-  int                           firstchild;
-  int                           lastchild;
-  int                           childcnt;
-  int                           nextsib;
+  enum XML_Content_Type type;
+  enum XML_Content_Quant quant;
+  const XML_Char *name;
+  int firstchild;
+  int lastchild;
+  int childcnt;
+  int nextsib;
 } CONTENT_SCAFFOLD;
 
 #define INIT_SCAFFOLD_ELEMENTS 32
@@ -377,10 +371,8 @@ typedef struct open_internal_entity {
   XML_Bool betweenDecl; /* WFC: PE Between Declarations */
 } OPEN_INTERNAL_ENTITY;
 
-typedef enum XML_Error PTRCALL Processor(XML_Parser parser,
-                                         const char *start,
-                                         const char *end,
-                                         const char **endPtr);
+typedef enum XML_Error PTRCALL Processor(XML_Parser parser, const char *start,
+                                         const char *end, const char **endPtr);
 
 static Processor prologProcessor;
 static Processor prologInitProcessor;
@@ -401,118 +393,101 @@ static Processor externalEntityInitProcessor3;
 static Processor externalEntityContentProcessor;
 static Processor internalEntityProcessor;
 
-static enum XML_Error
-handleUnknownEncoding(XML_Parser parser, const XML_Char *encodingName);
-static enum XML_Error
-processXmlDecl(XML_Parser parser, int isGeneralTextEntity,
-               const char *s, const char *next);
-static enum XML_Error
-initializeEncoding(XML_Parser parser);
-static enum XML_Error
-doProlog(XML_Parser parser, const ENCODING *enc, const char *s,
-         const char *end, int tok, const char *next, const char **nextPtr,
-         XML_Bool haveMore);
-static enum XML_Error
-processInternalEntity(XML_Parser parser, ENTITY *entity,
-                      XML_Bool betweenDecl);
-static enum XML_Error
-doContent(XML_Parser parser, int startTagLevel, const ENCODING *enc,
-          const char *start, const char *end, const char **endPtr,
-          XML_Bool haveMore);
-static enum XML_Error
-doCdataSection(XML_Parser parser, const ENCODING *, const char **startPtr,
-               const char *end, const char **nextPtr, XML_Bool haveMore);
+static enum XML_Error handleUnknownEncoding(XML_Parser parser,
+                                            const XML_Char *encodingName);
+static enum XML_Error processXmlDecl(XML_Parser parser, int isGeneralTextEntity,
+                                     const char *s, const char *next);
+static enum XML_Error initializeEncoding(XML_Parser parser);
+static enum XML_Error doProlog(XML_Parser parser, const ENCODING *enc,
+                               const char *s, const char *end, int tok,
+                               const char *next, const char **nextPtr,
+                               XML_Bool haveMore, XML_Bool allowClosingDoctype);
+static enum XML_Error processInternalEntity(XML_Parser parser, ENTITY *entity,
+                                            XML_Bool betweenDecl);
+static enum XML_Error doContent(XML_Parser parser, int startTagLevel,
+                                const ENCODING *enc, const char *start,
+                                const char *end, const char **endPtr,
+                                XML_Bool haveMore);
+static enum XML_Error doCdataSection(XML_Parser parser, const ENCODING *,
+                                     const char **startPtr, const char *end,
+                                     const char **nextPtr, XML_Bool haveMore);
 #ifdef XML_DTD
-static enum XML_Error
-doIgnoreSection(XML_Parser parser, const ENCODING *, const char **startPtr,
-                const char *end, const char **nextPtr, XML_Bool haveMore);
+static enum XML_Error doIgnoreSection(XML_Parser parser, const ENCODING *,
+                                      const char **startPtr, const char *end,
+                                      const char **nextPtr, XML_Bool haveMore);
 #endif /* XML_DTD */
 
-static void
-freeBindings(XML_Parser parser, BINDING *bindings);
-static enum XML_Error
-storeAtts(XML_Parser parser, const ENCODING *, const char *s,
-          TAG_NAME *tagNamePtr, BINDING **bindingsPtr);
-static enum XML_Error
-addBinding(XML_Parser parser, PREFIX *prefix, const ATTRIBUTE_ID *attId,
-           const XML_Char *uri, BINDING **bindingsPtr);
-static int
-defineAttribute(ELEMENT_TYPE *type, ATTRIBUTE_ID *, XML_Bool isCdata,
-                XML_Bool isId, const XML_Char *dfltValue, XML_Parser parser);
-static enum XML_Error
-storeAttributeValue(XML_Parser parser, const ENCODING *, XML_Bool isCdata,
-                    const char *, const char *, STRING_POOL *);
-static enum XML_Error
-appendAttributeValue(XML_Parser parser, const ENCODING *, XML_Bool isCdata,
-                     const char *, const char *, STRING_POOL *);
-static ATTRIBUTE_ID *
-getAttributeId(XML_Parser parser, const ENCODING *enc, const char *start,
-               const char *end);
-static int
-setElementTypePrefix(XML_Parser parser, ELEMENT_TYPE *);
-static enum XML_Error
-storeEntityValue(XML_Parser parser, const ENCODING *enc, const char *start,
-                 const char *end);
-static int
-reportProcessingInstruction(XML_Parser parser, const ENCODING *enc,
-                            const char *start, const char *end);
-static int
-reportComment(XML_Parser parser, const ENCODING *enc, const char *start,
-              const char *end);
-static void
-reportDefault(XML_Parser parser, const ENCODING *enc, const char *start,
-              const char *end);
-
-static const XML_Char * getContext(XML_Parser parser);
-static XML_Bool
-setContext(XML_Parser parser, const XML_Char *context);
+static void freeBindings(XML_Parser parser, BINDING *bindings);
+static enum XML_Error storeAtts(XML_Parser parser, const ENCODING *,
+                                const char *s, TAG_NAME *tagNamePtr,
+                                BINDING **bindingsPtr);
+static enum XML_Error addBinding(XML_Parser parser, PREFIX *prefix,
+                                 const ATTRIBUTE_ID *attId, const XML_Char *uri,
+                                 BINDING **bindingsPtr);
+static int defineAttribute(ELEMENT_TYPE *type, ATTRIBUTE_ID *, XML_Bool isCdata,
+                           XML_Bool isId, const XML_Char *dfltValue,
+                           XML_Parser parser);
+static enum XML_Error storeAttributeValue(XML_Parser parser, const ENCODING *,
+                                          XML_Bool isCdata, const char *,
+                                          const char *, STRING_POOL *);
+static enum XML_Error appendAttributeValue(XML_Parser parser, const ENCODING *,
+                                           XML_Bool isCdata, const char *,
+                                           const char *, STRING_POOL *);
+static ATTRIBUTE_ID *getAttributeId(XML_Parser parser, const ENCODING *enc,
+                                    const char *start, const char *end);
+static int setElementTypePrefix(XML_Parser parser, ELEMENT_TYPE *);
+static enum XML_Error storeEntityValue(XML_Parser parser, const ENCODING *enc,
+                                       const char *start, const char *end);
+static int reportProcessingInstruction(XML_Parser parser, const ENCODING *enc,
+                                       const char *start, const char *end);
+static int reportComment(XML_Parser parser, const ENCODING *enc,
+                         const char *start, const char *end);
+static void reportDefault(XML_Parser parser, const ENCODING *enc,
+                          const char *start, const char *end);
+
+static const XML_Char *getContext(XML_Parser parser);
+static XML_Bool setContext(XML_Parser parser, const XML_Char *context);
 
 static void FASTCALL normalizePublicId(XML_Char *s);
 
-static DTD * dtdCreate(const XML_Memory_Handling_Suite *ms);
-/* do not call if parentParser != NULL */
+static DTD *dtdCreate(const XML_Memory_Handling_Suite *ms);
+/* do not call if m_parentParser != NULL */
 static void dtdReset(DTD *p, const XML_Memory_Handling_Suite *ms);
-static void
-dtdDestroy(DTD *p, XML_Bool isDocEntity, const XML_Memory_Handling_Suite *ms);
-static int
-dtdCopy(XML_Parser oldParser,
-        DTD *newDtd, const DTD *oldDtd, const XML_Memory_Handling_Suite *ms);
-static int
-copyEntityTable(XML_Parser oldParser,
-                HASH_TABLE *, STRING_POOL *, const HASH_TABLE *);
-static NAMED *
-lookup(XML_Parser parser, HASH_TABLE *table, KEY name, size_t createSize);
-static void FASTCALL
-hashTableInit(HASH_TABLE *, const XML_Memory_Handling_Suite *ms);
+static void dtdDestroy(DTD *p, XML_Bool isDocEntity,
+                       const XML_Memory_Handling_Suite *ms);
+static int dtdCopy(XML_Parser oldParser, DTD *newDtd, const DTD *oldDtd,
+                   const XML_Memory_Handling_Suite *ms);
+static int copyEntityTable(XML_Parser oldParser, HASH_TABLE *, STRING_POOL *,
+                           const HASH_TABLE *);
+static NAMED *lookup(XML_Parser parser, HASH_TABLE *table, KEY name,
+                     size_t createSize);
+static void FASTCALL hashTableInit(HASH_TABLE *,
+                                   const XML_Memory_Handling_Suite *ms);
 static void FASTCALL hashTableClear(HASH_TABLE *);
 static void FASTCALL hashTableDestroy(HASH_TABLE *);
-static void FASTCALL
-hashTableIterInit(HASH_TABLE_ITER *, const HASH_TABLE *);
-static NAMED * FASTCALL hashTableIterNext(HASH_TABLE_ITER *);
+static void FASTCALL hashTableIterInit(HASH_TABLE_ITER *, const HASH_TABLE *);
+static NAMED *FASTCALL hashTableIterNext(HASH_TABLE_ITER *);
 
-static void FASTCALL
-poolInit(STRING_POOL *, const XML_Memory_Handling_Suite *ms);
+static void FASTCALL poolInit(STRING_POOL *,
+                              const XML_Memory_Handling_Suite *ms);
 static void FASTCALL poolClear(STRING_POOL *);
 static void FASTCALL poolDestroy(STRING_POOL *);
-static XML_Char *
-poolAppend(STRING_POOL *pool, const ENCODING *enc,
-           const char *ptr, const char *end);
-static XML_Char *
-poolStoreString(STRING_POOL *pool, const ENCODING *enc,
-                const char *ptr, const char *end);
+static XML_Char *poolAppend(STRING_POOL *pool, const ENCODING *enc,
+                            const char *ptr, const char *end);
+static XML_Char *poolStoreString(STRING_POOL *pool, const ENCODING *enc,
+                                 const char *ptr, const char *end);
 static XML_Bool FASTCALL poolGrow(STRING_POOL *pool);
-static const XML_Char * FASTCALL
-poolCopyString(STRING_POOL *pool, const XML_Char *s);
-static const XML_Char *
-poolCopyStringN(STRING_POOL *pool, const XML_Char *s, int n);
-static const XML_Char * FASTCALL
-poolAppendString(STRING_POOL *pool, const XML_Char *s);
+static const XML_Char *FASTCALL poolCopyString(STRING_POOL *pool,
+                                               const XML_Char *s);
+static const XML_Char *poolCopyStringN(STRING_POOL *pool, const XML_Char *s,
+                                       int n);
+static const XML_Char *FASTCALL poolAppendString(STRING_POOL *pool,
+                                                 const XML_Char *s);
 
 static int FASTCALL nextScaffoldPart(XML_Parser parser);
-static XML_Content * build_model(XML_Parser parser);
-static ELEMENT_TYPE *
-getElementType(XML_Parser parser, const ENCODING *enc,
-               const char *ptr, const char *end);
+static XML_Content *build_model(XML_Parser parser);
+static ELEMENT_TYPE *getElementType(XML_Parser parser, const ENCODING *enc,
+                                    const char *ptr, const char *end);
 
 static XML_Char *copyString(const XML_Char *s,
                             const XML_Memory_Handling_Suite *memsuite);
@@ -520,14 +495,11 @@ static XML_Char *copyString(const XML_Char *s,
 static unsigned long generate_hash_secret_salt(XML_Parser parser);
 static XML_Bool startParsing(XML_Parser parser);
 
-static XML_Parser
-parserCreate(const XML_Char *encodingName,
-             const XML_Memory_Handling_Suite *memsuite,
-             const XML_Char *nameSep,
-             DTD *dtd);
+static XML_Parser parserCreate(const XML_Char *encodingName,
+                               const XML_Memory_Handling_Suite *memsuite,
+                               const XML_Char *nameSep, DTD *dtd);
 
-static void
-parserInit(XML_Parser parser, const XML_Char *encodingName);
+static void parserInit(XML_Parser parser, const XML_Char *encodingName);
 
 #define poolStart(pool) ((pool)->start)
 #define poolEnd(pool) ((pool)->ptr)
@@ -536,13 +508,13 @@ parserInit(XML_Parser parser, const XML_Char *encodingName);
 #define poolLastChar(pool) (((pool)->ptr)[-1])
 #define poolDiscard(pool) ((pool)->ptr = (pool)->start)
 #define poolFinish(pool) ((pool)->start = (pool)->ptr)
-#define poolAppendChar(pool, c) \
-  (((pool)->ptr == (pool)->end && !poolGrow(pool)) \
-   ? 0 \
-   : ((*((pool)->ptr)++ = c), 1))
+#define poolAppendChar(pool, c)                                                \
+  (((pool)->ptr == (pool)->end && ! poolGrow(pool))                            \
+       ? 0                                                                     \
+       : ((*((pool)->ptr)++ = c), 1))
 
 struct XML_ParserStruct {
-  /* The first member must be userData so that the XML_GetUserData
+  /* The first member must be m_userData so that the XML_GetUserData
      macro works. */
   void *m_userData;
   void *m_handlerArg;
@@ -552,7 +524,7 @@ struct XML_ParserStruct {
   const char *m_bufferPtr;
   /* past last character to be parsed */
   char *m_bufferEnd;
-  /* allocated end of buffer */
+  /* allocated end of m_buffer */
   const char *m_bufferLim;
   XML_Index m_parseEndByteIndex;
   const char *m_parseEndPtr;
@@ -590,7 +562,7 @@ struct XML_ParserStruct {
   void *m_unknownEncodingMem;
   void *m_unknownEncodingData;
   void *m_unknownEncodingHandlerData;
-  void (XMLCALL *m_unknownEncodingRelease)(void *);
+  void(XMLCALL *m_unknownEncodingRelease)(void *);
   PROLOG_STATE m_prologState;
   Processor *m_processor;
   enum XML_Error m_errorCode;
@@ -644,157 +616,55 @@ struct XML_ParserStruct {
   unsigned long m_hash_secret_salt;
 };
 
-#define MALLOC(s) (parser->m_mem.malloc_fcn((s)))
-#define REALLOC(p,s) (parser->m_mem.realloc_fcn((p),(s)))
-#define FREE(p) (parser->m_mem.free_fcn((p)))
-
-#define userData (parser->m_userData)
-#define handlerArg (parser->m_handlerArg)
-#define startElementHandler (parser->m_startElementHandler)
-#define endElementHandler (parser->m_endElementHandler)
-#define characterDataHandler (parser->m_characterDataHandler)
-#define processingInstructionHandler \
-        (parser->m_processingInstructionHandler)
-#define commentHandler (parser->m_commentHandler)
-#define startCdataSectionHandler \
-        (parser->m_startCdataSectionHandler)
-#define endCdataSectionHandler (parser->m_endCdataSectionHandler)
-#define defaultHandler (parser->m_defaultHandler)
-#define startDoctypeDeclHandler (parser->m_startDoctypeDeclHandler)
-#define endDoctypeDeclHandler (parser->m_endDoctypeDeclHandler)
-#define unparsedEntityDeclHandler \
-        (parser->m_unparsedEntityDeclHandler)
-#define notationDeclHandler (parser->m_notationDeclHandler)
-#define startNamespaceDeclHandler \
-        (parser->m_startNamespaceDeclHandler)
-#define endNamespaceDeclHandler (parser->m_endNamespaceDeclHandler)
-#define notStandaloneHandler (parser->m_notStandaloneHandler)
-#define externalEntityRefHandler \
-        (parser->m_externalEntityRefHandler)
-#define externalEntityRefHandlerArg \
-        (parser->m_externalEntityRefHandlerArg)
-#define internalEntityRefHandler \
-        (parser->m_internalEntityRefHandler)
-#define skippedEntityHandler (parser->m_skippedEntityHandler)
-#define unknownEncodingHandler (parser->m_unknownEncodingHandler)
-#define elementDeclHandler (parser->m_elementDeclHandler)
-#define attlistDeclHandler (parser->m_attlistDeclHandler)
-#define entityDeclHandler (parser->m_entityDeclHandler)
-#define xmlDeclHandler (parser->m_xmlDeclHandler)
-#define encoding (parser->m_encoding)
-#define initEncoding (parser->m_initEncoding)
-#define internalEncoding (parser->m_internalEncoding)
-#define unknownEncodingMem (parser->m_unknownEncodingMem)
-#define unknownEncodingData (parser->m_unknownEncodingData)
-#define unknownEncodingHandlerData \
-  (parser->m_unknownEncodingHandlerData)
-#define unknownEncodingRelease (parser->m_unknownEncodingRelease)
-#define protocolEncodingName (parser->m_protocolEncodingName)
-#define ns (parser->m_ns)
-#define ns_triplets (parser->m_ns_triplets)
-#define prologState (parser->m_prologState)
-#define processor (parser->m_processor)
-#define errorCode (parser->m_errorCode)
-#define eventPtr (parser->m_eventPtr)
-#define eventEndPtr (parser->m_eventEndPtr)
-#define positionPtr (parser->m_positionPtr)
-#define position (parser->m_position)
-#define openInternalEntities (parser->m_openInternalEntities)
-#define freeInternalEntities (parser->m_freeInternalEntities)
-#define defaultExpandInternalEntities \
-        (parser->m_defaultExpandInternalEntities)
-#define tagLevel (parser->m_tagLevel)
-#define buffer (parser->m_buffer)
-#define bufferPtr (parser->m_bufferPtr)
-#define bufferEnd (parser->m_bufferEnd)
-#define parseEndByteIndex (parser->m_parseEndByteIndex)
-#define parseEndPtr (parser->m_parseEndPtr)
-#define bufferLim (parser->m_bufferLim)
-#define dataBuf (parser->m_dataBuf)
-#define dataBufEnd (parser->m_dataBufEnd)
-#define _dtd (parser->m_dtd)
-#define curBase (parser->m_curBase)
-#define declEntity (parser->m_declEntity)
-#define doctypeName (parser->m_doctypeName)
-#define doctypeSysid (parser->m_doctypeSysid)
-#define doctypePubid (parser->m_doctypePubid)
-#define declAttributeType (parser->m_declAttributeType)
-#define declNotationName (parser->m_declNotationName)
-#define declNotationPublicId (parser->m_declNotationPublicId)
-#define declElementType (parser->m_declElementType)
-#define declAttributeId (parser->m_declAttributeId)
-#define declAttributeIsCdata (parser->m_declAttributeIsCdata)
-#define declAttributeIsId (parser->m_declAttributeIsId)
-#define freeTagList (parser->m_freeTagList)
-#define freeBindingList (parser->m_freeBindingList)
-#define inheritedBindings (parser->m_inheritedBindings)
-#define tagStack (parser->m_tagStack)
-#define atts (parser->m_atts)
-#define attsSize (parser->m_attsSize)
-#define nSpecifiedAtts (parser->m_nSpecifiedAtts)
-#define idAttIndex (parser->m_idAttIndex)
-#define nsAtts (parser->m_nsAtts)
-#define nsAttsVersion (parser->m_nsAttsVersion)
-#define nsAttsPower (parser->m_nsAttsPower)
-#define attInfo (parser->m_attInfo)
-#define tempPool (parser->m_tempPool)
-#define temp2Pool (parser->m_temp2Pool)
-#define groupConnector (parser->m_groupConnector)
-#define groupSize (parser->m_groupSize)
-#define namespaceSeparator (parser->m_namespaceSeparator)
-#define parentParser (parser->m_parentParser)
-#define ps_parsing (parser->m_parsingStatus.parsing)
-#define ps_finalBuffer (parser->m_parsingStatus.finalBuffer)
-#ifdef XML_DTD
-#define isParamEntity (parser->m_isParamEntity)
-#define useForeignDTD (parser->m_useForeignDTD)
-#define paramEntityParsing (parser->m_paramEntityParsing)
-#endif /* XML_DTD */
-#define hash_secret_salt (parser->m_hash_secret_salt)
+#define MALLOC(parser, s) (parser->m_mem.malloc_fcn((s)))
+#define REALLOC(parser, p, s) (parser->m_mem.realloc_fcn((p), (s)))
+#define FREE(parser, p) (parser->m_mem.free_fcn((p)))
 
 XML_Parser XMLCALL
-XML_ParserCreate(const XML_Char *encodingName)
-{
+XML_ParserCreate(const XML_Char *encodingName) {
   return XML_ParserCreate_MM(encodingName, NULL, NULL);
 }
 
 XML_Parser XMLCALL
-XML_ParserCreateNS(const XML_Char *encodingName, XML_Char nsSep)
-{
+XML_ParserCreateNS(const XML_Char *encodingName, XML_Char nsSep) {
   XML_Char tmp[2];
   *tmp = nsSep;
   return XML_ParserCreate_MM(encodingName, NULL, tmp);
 }
 
-static const XML_Char implicitContext[] = {
-  ASCII_x, ASCII_m, ASCII_l, ASCII_EQUALS, ASCII_h, ASCII_t, ASCII_t, ASCII_p,
-  ASCII_COLON, ASCII_SLASH, ASCII_SLASH, ASCII_w, ASCII_w, ASCII_w,
-  ASCII_PERIOD, ASCII_w, ASCII_3, ASCII_PERIOD, ASCII_o, ASCII_r, ASCII_g,
-  ASCII_SLASH, ASCII_X, ASCII_M, ASCII_L, ASCII_SLASH, ASCII_1, ASCII_9,
-  ASCII_9, ASCII_8, ASCII_SLASH, ASCII_n, ASCII_a, ASCII_m, ASCII_e,
-  ASCII_s, ASCII_p, ASCII_a, ASCII_c, ASCII_e, '\0'
-};
+static const XML_Char implicitContext[]
+    = {ASCII_x,     ASCII_m,     ASCII_l,      ASCII_EQUALS, ASCII_h,
+       ASCII_t,     ASCII_t,     ASCII_p,      ASCII_COLON,  ASCII_SLASH,
+       ASCII_SLASH, ASCII_w,     ASCII_w,      ASCII_w,      ASCII_PERIOD,
+       ASCII_w,     ASCII_3,     ASCII_PERIOD, ASCII_o,      ASCII_r,
+       ASCII_g,     ASCII_SLASH, ASCII_X,      ASCII_M,      ASCII_L,
+       ASCII_SLASH, ASCII_1,     ASCII_9,      ASCII_9,      ASCII_8,
+       ASCII_SLASH, ASCII_n,     ASCII_a,      ASCII_m,      ASCII_e,
+       ASCII_s,     ASCII_p,     ASCII_a,      ASCII_c,      ASCII_e,
+       '\0'};
 
+/* To avoid warnings about unused functions: */
+#if ! defined(HAVE_ARC4RANDOM_BUF) && ! defined(HAVE_ARC4RANDOM)
 
-#if defined(HAVE_GETRANDOM) || defined(HAVE_SYSCALL_GETRANDOM)
+#  if defined(HAVE_GETRANDOM) || defined(HAVE_SYSCALL_GETRANDOM)
 
 /* Obtain entropy on Linux 3.17+ */
 static int
-writeRandomBytes_getrandom_nonblock(void * target, size_t count) {
-  int success = 0;  /* full count bytes written? */
+writeRandomBytes_getrandom_nonblock(void *target, size_t count) {
+  int success = 0; /* full count bytes written? */
   size_t bytesWrittenTotal = 0;
   const unsigned int getrandomFlags = GRND_NONBLOCK;
 
   do {
-    void * const currentTarget = (void*)((char*)target + bytesWrittenTotal);
+    void *const currentTarget = (void *)((char *)target + bytesWrittenTotal);
     const size_t bytesToWrite = count - bytesWrittenTotal;
 
     const int bytesWrittenMore =
-#if defined(HAVE_GETRANDOM)
+#    if defined(HAVE_GETRANDOM)
         getrandom(currentTarget, bytesToWrite, getrandomFlags);
-#else
+#    else
         syscall(SYS_getrandom, currentTarget, bytesToWrite, getrandomFlags);
-#endif
+#    endif
 
     if (bytesWrittenMore > 0) {
       bytesWrittenTotal += bytesWrittenMore;
@@ -806,15 +676,14 @@ writeRandomBytes_getrandom_nonblock(void * target, size_t count) {
   return success;
 }
 
-#endif  /* defined(HAVE_GETRANDOM) || defined(HAVE_SYSCALL_GETRANDOM) */
-
+#  endif /* defined(HAVE_GETRANDOM) || defined(HAVE_SYSCALL_GETRANDOM) */
 
-#if ! defined(_WIN32) && defined(XML_DEV_URANDOM)
+#  if ! defined(_WIN32) && defined(XML_DEV_URANDOM)
 
 /* Extract entropy from /dev/urandom */
 static int
-writeRandomBytes_dev_urandom(void * target, size_t count) {
-  int success = 0;  /* full count bytes written? */
+writeRandomBytes_dev_urandom(void *target, size_t count) {
+  int success = 0; /* full count bytes written? */
   size_t bytesWrittenTotal = 0;
 
   const int fd = open("/dev/urandom", O_RDONLY);
@@ -823,7 +692,7 @@ writeRandomBytes_dev_urandom(void * target, size_t count) {
   }
 
   do {
-    void * const currentTarget = (void*)((char*)target + bytesWrittenTotal);
+    void *const currentTarget = (void *)((char *)target + bytesWrittenTotal);
     const size_t bytesToWrite = count - bytesWrittenTotal;
 
     const ssize_t bytesWrittenMore = read(fd, currentTarget, bytesToWrite);
@@ -839,13 +708,14 @@ writeRandomBytes_dev_urandom(void * target, size_t count) {
   return success;
 }
 
-#endif  /* ! defined(_WIN32) && defined(XML_DEV_URANDOM) */
+#  endif /* ! defined(_WIN32) && defined(XML_DEV_URANDOM) */
 
+#endif /* ! defined(HAVE_ARC4RANDOM_BUF) && ! defined(HAVE_ARC4RANDOM) */
 
-#if defined(HAVE_ARC4RANDOM)
+#if defined(HAVE_ARC4RANDOM) && ! defined(HAVE_ARC4RANDOM_BUF)
 
 static void
-writeRandomBytes_arc4random(void * target, size_t count) {
+writeRandomBytes_arc4random(void *target, size_t count) {
   size_t bytesWrittenTotal = 0;
 
   while (bytesWrittenTotal < count) {
@@ -853,95 +723,86 @@ writeRandomBytes_arc4random(void * target, size_t count) {
     size_t i = 0;
 
     for (; (i < sizeof(random32)) && (bytesWrittenTotal < count);
-        i++, bytesWrittenTotal++) {
+         i++, bytesWrittenTotal++) {
       const uint8_t random8 = (uint8_t)(random32 >> (i * 8));
       ((uint8_t *)target)[bytesWrittenTotal] = random8;
     }
   }
 }
 
-#endif  /* defined(HAVE_ARC4RANDOM) */
-
+#endif /* defined(HAVE_ARC4RANDOM) && ! defined(HAVE_ARC4RANDOM_BUF) */
 
 #ifdef _WIN32
 
-typedef BOOLEAN (APIENTRY *RTLGENRANDOM_FUNC)(PVOID, ULONG);
-HMODULE _Expat_LoadLibrary(LPCTSTR filename);  /* see loadlibrary.c */
-
-/* Obtain entropy on Windows XP / Windows Server 2003 and later.
- * Hint on RtlGenRandom and the following article from libsodium.
- *
- * Michael Howard: Cryptographically Secure Random number on Windows without using CryptoAPI
- * https://blogs.msdn.microsoft.com/michael_howard/2005/01/14/cryptographically-secure-random-number-on-windows-without-using-cryptoapi/
+/* Obtain entropy on Windows using the rand_s() function which
+ * generates cryptographically secure random numbers.  Internally it
+ * uses RtlGenRandom API which is present in Windows XP and later.
  */
 static int
-writeRandomBytes_RtlGenRandom(void * target, size_t count) {
-  int success = 0;  /* full count bytes written? */
-  const HMODULE advapi32 = _Expat_LoadLibrary(TEXT("ADVAPI32.DLL"));
-
-  if (advapi32) {
-    const RTLGENRANDOM_FUNC RtlGenRandom
-        = (RTLGENRANDOM_FUNC)GetProcAddress(advapi32, "SystemFunction036");
-    if (RtlGenRandom) {
-      if (RtlGenRandom((PVOID)target, (ULONG)count) == TRUE) {
-        success = 1;
-      }
+writeRandomBytes_rand_s(void *target, size_t count) {
+  size_t bytesWrittenTotal = 0;
+
+  while (bytesWrittenTotal < count) {
+    unsigned int random32 = 0;
+    size_t i = 0;
+
+    if (rand_s(&random32))
+      return 0; /* failure */
+
+    for (; (i < sizeof(random32)) && (bytesWrittenTotal < count);
+         i++, bytesWrittenTotal++) {
+      const uint8_t random8 = (uint8_t)(random32 >> (i * 8));
+      ((uint8_t *)target)[bytesWrittenTotal] = random8;
     }
-    FreeLibrary(advapi32);
   }
-
-  return success;
+  return 1; /* success */
 }
 
 #endif /* _WIN32 */
 
-
 #if ! defined(HAVE_ARC4RANDOM_BUF) && ! defined(HAVE_ARC4RANDOM)
 
 static unsigned long
-gather_time_entropy(void)
-{
-#ifdef _WIN32
+gather_time_entropy(void) {
+#  ifdef _WIN32
   FILETIME ft;
   GetSystemTimeAsFileTime(&ft); /* never fails */
   return ft.dwHighDateTime ^ ft.dwLowDateTime;
-#else
+#  else
   struct timeval tv;
   int gettimeofday_res;
 
   gettimeofday_res = gettimeofday(&tv, NULL);
 
-#if defined(NDEBUG)
+#    if defined(NDEBUG)
   (void)gettimeofday_res;
-#else
-  assert (gettimeofday_res == 0);
-#endif  /* defined(NDEBUG) */
+#    else
+  assert(gettimeofday_res == 0);
+#    endif /* defined(NDEBUG) */
 
   /* Microseconds time is <20 bits entropy */
   return tv.tv_usec;
-#endif
+#  endif
 }
 
-#endif  /* ! defined(HAVE_ARC4RANDOM_BUF) && ! defined(HAVE_ARC4RANDOM) */
-
+#endif /* ! defined(HAVE_ARC4RANDOM_BUF) && ! defined(HAVE_ARC4RANDOM) */
 
 static unsigned long
-ENTROPY_DEBUG(const char * label, unsigned long entropy) {
-  const char * const EXPAT_ENTROPY_DEBUG = getenv("EXPAT_ENTROPY_DEBUG");
+ENTROPY_DEBUG(const char *label, unsigned long entropy) {
+  const char *const EXPAT_ENTROPY_DEBUG = getenv("EXPAT_ENTROPY_DEBUG");
   if (EXPAT_ENTROPY_DEBUG && ! strcmp(EXPAT_ENTROPY_DEBUG, "1")) {
-    fprintf(stderr, "Entropy: %s --> 0x%0*lx (%lu bytes)\n",
-        label,
-        (int)sizeof(entropy) * 2, entropy,
-        (unsigned long)sizeof(entropy));
+    fprintf(stderr, "Entropy: %s --> 0x%0*lx (%lu bytes)\n", label,
+            (int)sizeof(entropy) * 2, entropy, (unsigned long)sizeof(entropy));
   }
   return entropy;
 }
 
 static unsigned long
-generate_hash_secret_salt(XML_Parser parser)
-{
+generate_hash_secret_salt(XML_Parser parser) {
   unsigned long entropy;
   (void)parser;
+
+  /* "Failproof" high quality providers: */
 #if defined(HAVE_ARC4RANDOM_BUF)
   arc4random_buf(&entropy, sizeof(entropy));
   return ENTROPY_DEBUG("arc4random_buf", entropy);
@@ -950,20 +811,20 @@ generate_hash_secret_salt(XML_Parser parser)
   return ENTROPY_DEBUG("arc4random", entropy);
 #else
   /* Try high quality providers first .. */
-#ifdef _WIN32
-  if (writeRandomBytes_RtlGenRandom((void *)&entropy, sizeof(entropy))) {
-    return ENTROPY_DEBUG("RtlGenRandom", entropy);
+#  ifdef _WIN32
+  if (writeRandomBytes_rand_s((void *)&entropy, sizeof(entropy))) {
+    return ENTROPY_DEBUG("rand_s", entropy);
   }
-#elif defined(HAVE_GETRANDOM) || defined(HAVE_SYSCALL_GETRANDOM)
+#  elif defined(HAVE_GETRANDOM) || defined(HAVE_SYSCALL_GETRANDOM)
   if (writeRandomBytes_getrandom_nonblock((void *)&entropy, sizeof(entropy))) {
     return ENTROPY_DEBUG("getrandom", entropy);
   }
-#endif
-#if ! defined(_WIN32) && defined(XML_DEV_URANDOM)
+#  endif
+#  if ! defined(_WIN32) && defined(XML_DEV_URANDOM)
   if (writeRandomBytes_dev_urandom((void *)&entropy, sizeof(entropy))) {
     return ENTROPY_DEBUG("/dev/urandom", entropy);
   }
-#endif  /* ! defined(_WIN32) && defined(XML_DEV_URANDOM) */
+#  endif /* ! defined(_WIN32) && defined(XML_DEV_URANDOM) */
   /* .. and self-made low quality for backup: */
 
   /* Process ID is 0 bits entropy if attacker has local access */
@@ -974,7 +835,7 @@ generate_hash_secret_salt(XML_Parser parser)
     return ENTROPY_DEBUG("fallback(4)", entropy * 2147483647);
   } else {
     return ENTROPY_DEBUG("fallback(8)",
-        entropy * (unsigned long)2305843009213693951ULL);
+                         entropy * (unsigned long)2305843009213693951ULL);
   }
 #endif
 }
@@ -986,49 +847,43 @@ get_hash_secret_salt(XML_Parser parser) {
   return parser->m_hash_secret_salt;
 }
 
-static XML_Bool  /* only valid for root parser */
-startParsing(XML_Parser parser)
-{
-    /* hash functions must be initialized before setContext() is called */
-    if (hash_secret_salt == 0)
-      hash_secret_salt = generate_hash_secret_salt(parser);
-    if (ns) {
-      /* implicit context only set for root parser, since child
-         parsers (i.e. external entity parsers) will inherit it
-      */
-      return setContext(parser, implicitContext);
-    }
-    return XML_TRUE;
+static XML_Bool /* only valid for root parser */
+startParsing(XML_Parser parser) {
+  /* hash functions must be initialized before setContext() is called */
+  if (parser->m_hash_secret_salt == 0)
+    parser->m_hash_secret_salt = generate_hash_secret_salt(parser);
+  if (parser->m_ns) {
+    /* implicit context only set for root parser, since child
+       parsers (i.e. external entity parsers) will inherit it
+    */
+    return setContext(parser, implicitContext);
+  }
+  return XML_TRUE;
 }
 
 XML_Parser XMLCALL
 XML_ParserCreate_MM(const XML_Char *encodingName,
                     const XML_Memory_Handling_Suite *memsuite,
-                    const XML_Char *nameSep)
-{
+                    const XML_Char *nameSep) {
   return parserCreate(encodingName, memsuite, nameSep, NULL);
 }
 
 static XML_Parser
 parserCreate(const XML_Char *encodingName,
-             const XML_Memory_Handling_Suite *memsuite,
-             const XML_Char *nameSep,
-             DTD *dtd)
-{
+             const XML_Memory_Handling_Suite *memsuite, const XML_Char *nameSep,
+             DTD *dtd) {
   XML_Parser parser;
 
   if (memsuite) {
     XML_Memory_Handling_Suite *mtemp;
-    parser = (XML_Parser)
-      memsuite->malloc_fcn(sizeof(struct XML_ParserStruct));
+    parser = (XML_Parser)memsuite->malloc_fcn(sizeof(struct XML_ParserStruct));
     if (parser != NULL) {
       mtemp = (XML_Memory_Handling_Suite *)&(parser->m_mem);
       mtemp->malloc_fcn = memsuite->malloc_fcn;
       mtemp->realloc_fcn = memsuite->realloc_fcn;
       mtemp->free_fcn = memsuite->free_fcn;
     }
-  }
-  else {
+  } else {
     XML_Memory_Handling_Suite *mtemp;
     parser = (XML_Parser)malloc(sizeof(struct XML_ParserStruct));
     if (parser != NULL) {
@@ -1039,252 +894,249 @@ parserCreate(const XML_Char *encodingName,
     }
   }
 
-  if (!parser)
+  if (! parser)
     return parser;
 
-  buffer = NULL;
-  bufferLim = NULL;
+  parser->m_buffer = NULL;
+  parser->m_bufferLim = NULL;
 
-  attsSize = INIT_ATTS_SIZE;
-  atts = (ATTRIBUTE *)MALLOC(attsSize * sizeof(ATTRIBUTE));
-  if (atts == NULL) {
-    FREE(parser);
+  parser->m_attsSize = INIT_ATTS_SIZE;
+  parser->m_atts
+      = (ATTRIBUTE *)MALLOC(parser, parser->m_attsSize * sizeof(ATTRIBUTE));
+  if (parser->m_atts == NULL) {
+    FREE(parser, parser);
     return NULL;
   }
 #ifdef XML_ATTR_INFO
-  attInfo = (XML_AttrInfo*)MALLOC(attsSize * sizeof(XML_AttrInfo));
-  if (attInfo == NULL) {
-    FREE(atts);
-    FREE(parser);
+  parser->m_attInfo = (XML_AttrInfo *)MALLOC(
+      parser, parser->m_attsSize * sizeof(XML_AttrInfo));
+  if (parser->m_attInfo == NULL) {
+    FREE(parser, parser->m_atts);
+    FREE(parser, parser);
     return NULL;
   }
 #endif
-  dataBuf = (XML_Char *)MALLOC(INIT_DATA_BUF_SIZE * sizeof(XML_Char));
-  if (dataBuf == NULL) {
-    FREE(atts);
+  parser->m_dataBuf
+      = (XML_Char *)MALLOC(parser, INIT_DATA_BUF_SIZE * sizeof(XML_Char));
+  if (parser->m_dataBuf == NULL) {
+    FREE(parser, parser->m_atts);
 #ifdef XML_ATTR_INFO
-    FREE(attInfo);
+    FREE(parser, parser->m_attInfo);
 #endif
-    FREE(parser);
+    FREE(parser, parser);
     return NULL;
   }
-  dataBufEnd = dataBuf + INIT_DATA_BUF_SIZE;
+  parser->m_dataBufEnd = parser->m_dataBuf + INIT_DATA_BUF_SIZE;
 
   if (dtd)
-    _dtd = dtd;
+    parser->m_dtd = dtd;
   else {
-    _dtd = dtdCreate(&parser->m_mem);
-    if (_dtd == NULL) {
-      FREE(dataBuf);
-      FREE(atts);
+    parser->m_dtd = dtdCreate(&parser->m_mem);
+    if (parser->m_dtd == NULL) {
+      FREE(parser, parser->m_dataBuf);
+      FREE(parser, parser->m_atts);
 #ifdef XML_ATTR_INFO
-      FREE(attInfo);
+      FREE(parser, parser->m_attInfo);
 #endif
-      FREE(parser);
+      FREE(parser, parser);
       return NULL;
     }
   }
 
-  freeBindingList = NULL;
-  freeTagList = NULL;
-  freeInternalEntities = NULL;
+  parser->m_freeBindingList = NULL;
+  parser->m_freeTagList = NULL;
+  parser->m_freeInternalEntities = NULL;
 
-  groupSize = 0;
-  groupConnector = NULL;
+  parser->m_groupSize = 0;
+  parser->m_groupConnector = NULL;
 
-  unknownEncodingHandler = NULL;
-  unknownEncodingHandlerData = NULL;
+  parser->m_unknownEncodingHandler = NULL;
+  parser->m_unknownEncodingHandlerData = NULL;
 
-  namespaceSeparator = ASCII_EXCL;
-  ns = XML_FALSE;
-  ns_triplets = XML_FALSE;
+  parser->m_namespaceSeparator = ASCII_EXCL;
+  parser->m_ns = XML_FALSE;
+  parser->m_ns_triplets = XML_FALSE;
 
-  nsAtts = NULL;
-  nsAttsVersion = 0;
-  nsAttsPower = 0;
+  parser->m_nsAtts = NULL;
+  parser->m_nsAttsVersion = 0;
+  parser->m_nsAttsPower = 0;
 
-  protocolEncodingName = NULL;
+  parser->m_protocolEncodingName = NULL;
 
-  poolInit(&tempPool, &(parser->m_mem));
-  poolInit(&temp2Pool, &(parser->m_mem));
+  poolInit(&parser->m_tempPool, &(parser->m_mem));
+  poolInit(&parser->m_temp2Pool, &(parser->m_mem));
   parserInit(parser, encodingName);
 
-  if (encodingName && !protocolEncodingName) {
+  if (encodingName && ! parser->m_protocolEncodingName) {
     XML_ParserFree(parser);
     return NULL;
   }
 
   if (nameSep) {
-    ns = XML_TRUE;
-    internalEncoding = XmlGetInternalEncodingNS();
-    namespaceSeparator = *nameSep;
-  }
-  else {
-    internalEncoding = XmlGetInternalEncoding();
+    parser->m_ns = XML_TRUE;
+    parser->m_internalEncoding = XmlGetInternalEncodingNS();
+    parser->m_namespaceSeparator = *nameSep;
+  } else {
+    parser->m_internalEncoding = XmlGetInternalEncoding();
   }
 
   return parser;
 }
 
 static void
-parserInit(XML_Parser parser, const XML_Char *encodingName)
-{
-  processor = prologInitProcessor;
-  XmlPrologStateInit(&prologState);
+parserInit(XML_Parser parser, const XML_Char *encodingName) {
+  parser->m_processor = prologInitProcessor;
+  XmlPrologStateInit(&parser->m_prologState);
   if (encodingName != NULL) {
-    protocolEncodingName = copyString(encodingName, &(parser->m_mem));
-  }
-  curBase = NULL;
-  XmlInitEncoding(&initEncoding, &encoding, 0);
-  userData = NULL;
-  handlerArg = NULL;
-  startElementHandler = NULL;
-  endElementHandler = NULL;
-  characterDataHandler = NULL;
-  processingInstructionHandler = NULL;
-  commentHandler = NULL;
-  startCdataSectionHandler = NULL;
-  endCdataSectionHandler = NULL;
-  defaultHandler = NULL;
-  startDoctypeDeclHandler = NULL;
-  endDoctypeDeclHandler = NULL;
-  unparsedEntityDeclHandler = NULL;
-  notationDeclHandler = NULL;
-  startNamespaceDeclHandler = NULL;
-  endNamespaceDeclHandler = NULL;
-  notStandaloneHandler = NULL;
-  externalEntityRefHandler = NULL;
-  externalEntityRefHandlerArg = parser;
-  skippedEntityHandler = NULL;
-  elementDeclHandler = NULL;
-  attlistDeclHandler = NULL;
-  entityDeclHandler = NULL;
-  xmlDeclHandler = NULL;
-  bufferPtr = buffer;
-  bufferEnd = buffer;
-  parseEndByteIndex = 0;
-  parseEndPtr = NULL;
-  declElementType = NULL;
-  declAttributeId = NULL;
-  declEntity = NULL;
-  doctypeName = NULL;
-  doctypeSysid = NULL;
-  doctypePubid = NULL;
-  declAttributeType = NULL;
-  declNotationName = NULL;
-  declNotationPublicId = NULL;
-  declAttributeIsCdata = XML_FALSE;
-  declAttributeIsId = XML_FALSE;
-  memset(&position, 0, sizeof(POSITION));
-  errorCode = XML_ERROR_NONE;
-  eventPtr = NULL;
-  eventEndPtr = NULL;
-  positionPtr = NULL;
-  openInternalEntities = NULL;
-  defaultExpandInternalEntities = XML_TRUE;
-  tagLevel = 0;
-  tagStack = NULL;
-  inheritedBindings = NULL;
-  nSpecifiedAtts = 0;
-  unknownEncodingMem = NULL;
-  unknownEncodingRelease = NULL;
-  unknownEncodingData = NULL;
-  parentParser = NULL;
-  ps_parsing = XML_INITIALIZED;
+    parser->m_protocolEncodingName = copyString(encodingName, &(parser->m_mem));
+  }
+  parser->m_curBase = NULL;
+  XmlInitEncoding(&parser->m_initEncoding, &parser->m_encoding, 0);
+  parser->m_userData = NULL;
+  parser->m_handlerArg = NULL;
+  parser->m_startElementHandler = NULL;
+  parser->m_endElementHandler = NULL;
+  parser->m_characterDataHandler = NULL;
+  parser->m_processingInstructionHandler = NULL;
+  parser->m_commentHandler = NULL;
+  parser->m_startCdataSectionHandler = NULL;
+  parser->m_endCdataSectionHandler = NULL;
+  parser->m_defaultHandler = NULL;
+  parser->m_startDoctypeDeclHandler = NULL;
+  parser->m_endDoctypeDeclHandler = NULL;
+  parser->m_unparsedEntityDeclHandler = NULL;
+  parser->m_notationDeclHandler = NULL;
+  parser->m_startNamespaceDeclHandler = NULL;
+  parser->m_endNamespaceDeclHandler = NULL;
+  parser->m_notStandaloneHandler = NULL;
+  parser->m_externalEntityRefHandler = NULL;
+  parser->m_externalEntityRefHandlerArg = parser;
+  parser->m_skippedEntityHandler = NULL;
+  parser->m_elementDeclHandler = NULL;
+  parser->m_attlistDeclHandler = NULL;
+  parser->m_entityDeclHandler = NULL;
+  parser->m_xmlDeclHandler = NULL;
+  parser->m_bufferPtr = parser->m_buffer;
+  parser->m_bufferEnd = parser->m_buffer;
+  parser->m_parseEndByteIndex = 0;
+  parser->m_parseEndPtr = NULL;
+  parser->m_declElementType = NULL;
+  parser->m_declAttributeId = NULL;
+  parser->m_declEntity = NULL;
+  parser->m_doctypeName = NULL;
+  parser->m_doctypeSysid = NULL;
+  parser->m_doctypePubid = NULL;
+  parser->m_declAttributeType = NULL;
+  parser->m_declNotationName = NULL;
+  parser->m_declNotationPublicId = NULL;
+  parser->m_declAttributeIsCdata = XML_FALSE;
+  parser->m_declAttributeIsId = XML_FALSE;
+  memset(&parser->m_position, 0, sizeof(POSITION));
+  parser->m_errorCode = XML_ERROR_NONE;
+  parser->m_eventPtr = NULL;
+  parser->m_eventEndPtr = NULL;
+  parser->m_positionPtr = NULL;
+  parser->m_openInternalEntities = NULL;
+  parser->m_defaultExpandInternalEntities = XML_TRUE;
+  parser->m_tagLevel = 0;
+  parser->m_tagStack = NULL;
+  parser->m_inheritedBindings = NULL;
+  parser->m_nSpecifiedAtts = 0;
+  parser->m_unknownEncodingMem = NULL;
+  parser->m_unknownEncodingRelease = NULL;
+  parser->m_unknownEncodingData = NULL;
+  parser->m_parentParser = NULL;
+  parser->m_parsingStatus.parsing = XML_INITIALIZED;
 #ifdef XML_DTD
-  isParamEntity = XML_FALSE;
-  useForeignDTD = XML_FALSE;
-  paramEntityParsing = XML_PARAM_ENTITY_PARSING_NEVER;
+  parser->m_isParamEntity = XML_FALSE;
+  parser->m_useForeignDTD = XML_FALSE;
+  parser->m_paramEntityParsing = XML_PARAM_ENTITY_PARSING_NEVER;
 #endif
-  hash_secret_salt = 0;
+  parser->m_hash_secret_salt = 0;
 }
 
-/* moves list of bindings to freeBindingList */
+/* moves list of bindings to m_freeBindingList */
 static void FASTCALL
-moveToFreeBindingList(XML_Parser parser, BINDING *bindings)
-{
+moveToFreeBindingList(XML_Parser parser, BINDING *bindings) {
   while (bindings) {
     BINDING *b = bindings;
     bindings = bindings->nextTagBinding;
-    b->nextTagBinding = freeBindingList;
-    freeBindingList = b;
+    b->nextTagBinding = parser->m_freeBindingList;
+    parser->m_freeBindingList = b;
   }
 }
 
 XML_Bool XMLCALL
-XML_ParserReset(XML_Parser parser, const XML_Char *encodingName)
-{
+XML_ParserReset(XML_Parser parser, const XML_Char *encodingName) {
   TAG *tStk;
   OPEN_INTERNAL_ENTITY *openEntityList;
 
   if (parser == NULL)
-      return XML_FALSE;
+    return XML_FALSE;
 
-  if (parentParser)
+  if (parser->m_parentParser)
     return XML_FALSE;
-  /* move tagStack to freeTagList */
-  tStk = tagStack;
+  /* move m_tagStack to m_freeTagList */
+  tStk = parser->m_tagStack;
   while (tStk) {
     TAG *tag = tStk;
     tStk = tStk->parent;
-    tag->parent = freeTagList;
+    tag->parent = parser->m_freeTagList;
     moveToFreeBindingList(parser, tag->bindings);
     tag->bindings = NULL;
-    freeTagList = tag;
+    parser->m_freeTagList = tag;
   }
-  /* move openInternalEntities to freeInternalEntities */
-  openEntityList = openInternalEntities;
+  /* move m_openInternalEntities to m_freeInternalEntities */
+  openEntityList = parser->m_openInternalEntities;
   while (openEntityList) {
     OPEN_INTERNAL_ENTITY *openEntity = openEntityList;
     openEntityList = openEntity->next;
-    openEntity->next = freeInternalEntities;
-    freeInternalEntities = openEntity;
-  }
-  moveToFreeBindingList(parser, inheritedBindings);
-  FREE(unknownEncodingMem);
-  if (unknownEncodingRelease)
-    unknownEncodingRelease(unknownEncodingData);
-  poolClear(&tempPool);
-  poolClear(&temp2Pool);
-  FREE((void *)protocolEncodingName);
-  protocolEncodingName = NULL;
+    openEntity->next = parser->m_freeInternalEntities;
+    parser->m_freeInternalEntities = openEntity;
+  }
+  moveToFreeBindingList(parser, parser->m_inheritedBindings);
+  FREE(parser, parser->m_unknownEncodingMem);
+  if (parser->m_unknownEncodingRelease)
+    parser->m_unknownEncodingRelease(parser->m_unknownEncodingData);
+  poolClear(&parser->m_tempPool);
+  poolClear(&parser->m_temp2Pool);
+  FREE(parser, (void *)parser->m_protocolEncodingName);
+  parser->m_protocolEncodingName = NULL;
   parserInit(parser, encodingName);
-  dtdReset(_dtd, &parser->m_mem);
+  dtdReset(parser->m_dtd, &parser->m_mem);
   return XML_TRUE;
 }
 
 enum XML_Status XMLCALL
-XML_SetEncoding(XML_Parser parser, const XML_Char *encodingName)
-{
+XML_SetEncoding(XML_Parser parser, const XML_Char *encodingName) {
   if (parser == NULL)
-      return XML_STATUS_ERROR;
+    return XML_STATUS_ERROR;
   /* Block after XML_Parse()/XML_ParseBuffer() has been called.
      XXX There's no way for the caller to determine which of the
      XXX possible error cases caused the XML_STATUS_ERROR return.
   */
-  if (ps_parsing == XML_PARSING || ps_parsing == XML_SUSPENDED)
+  if (parser->m_parsingStatus.parsing == XML_PARSING
+      || parser->m_parsingStatus.parsing == XML_SUSPENDED)
     return XML_STATUS_ERROR;
 
   /* Get rid of any previous encoding name */
-  FREE((void *)protocolEncodingName);
+  FREE(parser, (void *)parser->m_protocolEncodingName);
 
   if (encodingName == NULL)
     /* No new encoding name */
-    protocolEncodingName = NULL;
+    parser->m_protocolEncodingName = NULL;
   else {
     /* Copy the new encoding name into allocated memory */
-    protocolEncodingName = copyString(encodingName, &(parser->m_mem));
-    if (!protocolEncodingName)
+    parser->m_protocolEncodingName = copyString(encodingName, &(parser->m_mem));
+    if (! parser->m_protocolEncodingName)
       return XML_STATUS_ERROR;
   }
   return XML_STATUS_OK;
 }
 
 XML_Parser XMLCALL
-XML_ExternalEntityParserCreate(XML_Parser oldParser,
-                               const XML_Char *context,
-                               const XML_Char *encodingName)
-{
+XML_ExternalEntityParserCreate(XML_Parser oldParser, const XML_Char *context,
+                               const XML_Char *encodingName) {
   XML_Parser parser = oldParser;
   DTD *newDtd = NULL;
   DTD *oldDtd;
@@ -1308,7 +1160,7 @@ XML_ExternalEntityParserCreate(XML_Parser oldParser,
   XML_AttlistDeclHandler oldAttlistDeclHandler;
   XML_EntityDeclHandler oldEntityDeclHandler;
   XML_XmlDeclHandler oldXmlDeclHandler;
-  ELEMENT_TYPE * oldDeclElementType;
+  ELEMENT_TYPE *oldDeclElementType;
 
   void *oldUserData;
   void *oldHandlerArg;
@@ -1331,47 +1183,47 @@ XML_ExternalEntityParserCreate(XML_Parser oldParser,
     return NULL;
 
   /* Stash the original parser contents on the stack */
-  oldDtd = _dtd;
-  oldStartElementHandler = startElementHandler;
-  oldEndElementHandler = endElementHandler;
-  oldCharacterDataHandler = characterDataHandler;
-  oldProcessingInstructionHandler = processingInstructionHandler;
-  oldCommentHandler = commentHandler;
-  oldStartCdataSectionHandler = startCdataSectionHandler;
-  oldEndCdataSectionHandler = endCdataSectionHandler;
-  oldDefaultHandler = defaultHandler;
-  oldUnparsedEntityDeclHandler = unparsedEntityDeclHandler;
-  oldNotationDeclHandler = notationDeclHandler;
-  oldStartNamespaceDeclHandler = startNamespaceDeclHandler;
-  oldEndNamespaceDeclHandler = endNamespaceDeclHandler;
-  oldNotStandaloneHandler = notStandaloneHandler;
-  oldExternalEntityRefHandler = externalEntityRefHandler;
-  oldSkippedEntityHandler = skippedEntityHandler;
-  oldUnknownEncodingHandler = unknownEncodingHandler;
-  oldElementDeclHandler = elementDeclHandler;
-  oldAttlistDeclHandler = attlistDeclHandler;
-  oldEntityDeclHandler = entityDeclHandler;
-  oldXmlDeclHandler = xmlDeclHandler;
-  oldDeclElementType = declElementType;
-
-  oldUserData = userData;
-  oldHandlerArg = handlerArg;
-  oldDefaultExpandInternalEntities = defaultExpandInternalEntities;
-  oldExternalEntityRefHandlerArg = externalEntityRefHandlerArg;
+  oldDtd = parser->m_dtd;
+  oldStartElementHandler = parser->m_startElementHandler;
+  oldEndElementHandler = parser->m_endElementHandler;
+  oldCharacterDataHandler = parser->m_characterDataHandler;
+  oldProcessingInstructionHandler = parser->m_processingInstructionHandler;
+  oldCommentHandler = parser->m_commentHandler;
+  oldStartCdataSectionHandler = parser->m_startCdataSectionHandler;
+  oldEndCdataSectionHandler = parser->m_endCdataSectionHandler;
+  oldDefaultHandler = parser->m_defaultHandler;
+  oldUnparsedEntityDeclHandler = parser->m_unparsedEntityDeclHandler;
+  oldNotationDeclHandler = parser->m_notationDeclHandler;
+  oldStartNamespaceDeclHandler = parser->m_startNamespaceDeclHandler;
+  oldEndNamespaceDeclHandler = parser->m_endNamespaceDeclHandler;
+  oldNotStandaloneHandler = parser->m_notStandaloneHandler;
+  oldExternalEntityRefHandler = parser->m_externalEntityRefHandler;
+  oldSkippedEntityHandler = parser->m_skippedEntityHandler;
+  oldUnknownEncodingHandler = parser->m_unknownEncodingHandler;
+  oldElementDeclHandler = parser->m_elementDeclHandler;
+  oldAttlistDeclHandler = parser->m_attlistDeclHandler;
+  oldEntityDeclHandler = parser->m_entityDeclHandler;
+  oldXmlDeclHandler = parser->m_xmlDeclHandler;
+  oldDeclElementType = parser->m_declElementType;
+
+  oldUserData = parser->m_userData;
+  oldHandlerArg = parser->m_handlerArg;
+  oldDefaultExpandInternalEntities = parser->m_defaultExpandInternalEntities;
+  oldExternalEntityRefHandlerArg = parser->m_externalEntityRefHandlerArg;
 #ifdef XML_DTD
-  oldParamEntityParsing = paramEntityParsing;
-  oldInEntityValue = prologState.inEntityValue;
+  oldParamEntityParsing = parser->m_paramEntityParsing;
+  oldInEntityValue = parser->m_prologState.inEntityValue;
 #endif
-  oldns_triplets = ns_triplets;
+  oldns_triplets = parser->m_ns_triplets;
   /* Note that the new parser shares the same hash secret as the old
      parser, so that dtdCopy and copyEntityTable can lookup values
      from hash tables associated with either parser without us having
      to worry which hash secrets each table has.
   */
-  oldhash_secret_salt = hash_secret_salt;
+  oldhash_secret_salt = parser->m_hash_secret_salt;
 
 #ifdef XML_DTD
-  if (!context)
+  if (! context)
     newDtd = oldDtd;
 #endif /* XML_DTD */
 
@@ -1380,175 +1232,171 @@ XML_ExternalEntityParserCreate(XML_Parser oldParser,
      here.  This makes this function more painful to follow than it
      would be otherwise.
   */
-  if (ns) {
+  if (parser->m_ns) {
     XML_Char tmp[2];
-    *tmp = namespaceSeparator;
+    *tmp = parser->m_namespaceSeparator;
     parser = parserCreate(encodingName, &parser->m_mem, tmp, newDtd);
-  }
-  else {
+  } else {
     parser = parserCreate(encodingName, &parser->m_mem, NULL, newDtd);
   }
 
-  if (!parser)
+  if (! parser)
     return NULL;
 
-  startElementHandler = oldStartElementHandler;
-  endElementHandler = oldEndElementHandler;
-  characterDataHandler = oldCharacterDataHandler;
-  processingInstructionHandler = oldProcessingInstructionHandler;
-  commentHandler = oldCommentHandler;
-  startCdataSectionHandler = oldStartCdataSectionHandler;
-  endCdataSectionHandler = oldEndCdataSectionHandler;
-  defaultHandler = oldDefaultHandler;
-  unparsedEntityDeclHandler = oldUnparsedEntityDeclHandler;
-  notationDeclHandler = oldNotationDeclHandler;
-  startNamespaceDeclHandler = oldStartNamespaceDeclHandler;
-  endNamespaceDeclHandler = oldEndNamespaceDeclHandler;
-  notStandaloneHandler = oldNotStandaloneHandler;
-  externalEntityRefHandler = oldExternalEntityRefHandler;
-  skippedEntityHandler = oldSkippedEntityHandler;
-  unknownEncodingHandler = oldUnknownEncodingHandler;
-  elementDeclHandler = oldElementDeclHandler;
-  attlistDeclHandler = oldAttlistDeclHandler;
-  entityDeclHandler = oldEntityDeclHandler;
-  xmlDeclHandler = oldXmlDeclHandler;
-  declElementType = oldDeclElementType;
-  userData = oldUserData;
+  parser->m_startElementHandler = oldStartElementHandler;
+  parser->m_endElementHandler = oldEndElementHandler;
+  parser->m_characterDataHandler = oldCharacterDataHandler;
+  parser->m_processingInstructionHandler = oldProcessingInstructionHandler;
+  parser->m_commentHandler = oldCommentHandler;
+  parser->m_startCdataSectionHandler = oldStartCdataSectionHandler;
+  parser->m_endCdataSectionHandler = oldEndCdataSectionHandler;
+  parser->m_defaultHandler = oldDefaultHandler;
+  parser->m_unparsedEntityDeclHandler = oldUnparsedEntityDeclHandler;
+  parser->m_notationDeclHandler = oldNotationDeclHandler;
+  parser->m_startNamespaceDeclHandler = oldStartNamespaceDeclHandler;
+  parser->m_endNamespaceDeclHandler = oldEndNamespaceDeclHandler;
+  parser->m_notStandaloneHandler = oldNotStandaloneHandler;
+  parser->m_externalEntityRefHandler = oldExternalEntityRefHandler;
+  parser->m_skippedEntityHandler = oldSkippedEntityHandler;
+  parser->m_unknownEncodingHandler = oldUnknownEncodingHandler;
+  parser->m_elementDeclHandler = oldElementDeclHandler;
+  parser->m_attlistDeclHandler = oldAttlistDeclHandler;
+  parser->m_entityDeclHandler = oldEntityDeclHandler;
+  parser->m_xmlDeclHandler = oldXmlDeclHandler;
+  parser->m_declElementType = oldDeclElementType;
+  parser->m_userData = oldUserData;
   if (oldUserData == oldHandlerArg)
-    handlerArg = userData;
+    parser->m_handlerArg = parser->m_userData;
   else
-    handlerArg = parser;
+    parser->m_handlerArg = parser;
   if (oldExternalEntityRefHandlerArg != oldParser)
-    externalEntityRefHandlerArg = oldExternalEntityRefHandlerArg;
-  defaultExpandInternalEntities = oldDefaultExpandInternalEntities;
-  ns_triplets = oldns_triplets;
-  hash_secret_salt = oldhash_secret_salt;
-  parentParser = oldParser;
+    parser->m_externalEntityRefHandlerArg = oldExternalEntityRefHandlerArg;
+  parser->m_defaultExpandInternalEntities = oldDefaultExpandInternalEntities;
+  parser->m_ns_triplets = oldns_triplets;
+  parser->m_hash_secret_salt = oldhash_secret_salt;
+  parser->m_parentParser = oldParser;
 #ifdef XML_DTD
-  paramEntityParsing = oldParamEntityParsing;
-  prologState.inEntityValue = oldInEntityValue;
+  parser->m_paramEntityParsing = oldParamEntityParsing;
+  parser->m_prologState.inEntityValue = oldInEntityValue;
   if (context) {
 #endif /* XML_DTD */
-    if (!dtdCopy(oldParser, _dtd, oldDtd, &parser->m_mem)
-      || !setContext(parser, context)) {
+    if (! dtdCopy(oldParser, parser->m_dtd, oldDtd, &parser->m_mem)
+        || ! setContext(parser, context)) {
       XML_ParserFree(parser);
       return NULL;
     }
-    processor = externalEntityInitProcessor;
+    parser->m_processor = externalEntityInitProcessor;
 #ifdef XML_DTD
-  }
-  else {
-    /* The DTD instance referenced by _dtd is shared between the document's
-       root parser and external PE parsers, therefore one does not need to
-       call setContext. In addition, one also *must* not call setContext,
-       because this would overwrite existing prefix->binding pointers in
-       _dtd with ones that get destroyed with the external PE parser.
-       This would leave those prefixes with dangling pointers.
+  } else {
+    /* The DTD instance referenced by parser->m_dtd is shared between the
+       document's root parser and external PE parsers, therefore one does not
+       need to call setContext. In addition, one also *must* not call
+       setContext, because this would overwrite existing prefix->binding
+       pointers in parser->m_dtd with ones that get destroyed with the external
+       PE parser. This would leave those prefixes with dangling pointers.
     */
-    isParamEntity = XML_TRUE;
-    XmlPrologStateInitExternalEntity(&prologState);
-    processor = externalParEntInitProcessor;
+    parser->m_isParamEntity = XML_TRUE;
+    XmlPrologStateInitExternalEntity(&parser->m_prologState);
+    parser->m_processor = externalParEntInitProcessor;
   }
 #endif /* XML_DTD */
   return parser;
 }
 
 static void FASTCALL
-destroyBindings(BINDING *bindings, XML_Parser parser)
-{
+destroyBindings(BINDING *bindings, XML_Parser parser) {
   for (;;) {
     BINDING *b = bindings;
-    if (!b)
+    if (! b)
       break;
     bindings = b->nextTagBinding;
-    FREE(b->uri);
-    FREE(b);
+    FREE(parser, b->uri);
+    FREE(parser, b);
   }
 }
 
 void XMLCALL
-XML_ParserFree(XML_Parser parser)
-{
+XML_ParserFree(XML_Parser parser) {
   TAG *tagList;
   OPEN_INTERNAL_ENTITY *entityList;
   if (parser == NULL)
     return;
-  /* free tagStack and freeTagList */
-  tagList = tagStack;
+  /* free m_tagStack and m_freeTagList */
+  tagList = parser->m_tagStack;
   for (;;) {
     TAG *p;
     if (tagList == NULL) {
-      if (freeTagList == NULL)
+      if (parser->m_freeTagList == NULL)
         break;
-      tagList = freeTagList;
-      freeTagList = NULL;
+      tagList = parser->m_freeTagList;
+      parser->m_freeTagList = NULL;
     }
     p = tagList;
     tagList = tagList->parent;
-    FREE(p->buf);
+    FREE(parser, p->buf);
     destroyBindings(p->bindings, parser);
-    FREE(p);
+    FREE(parser, p);
   }
-  /* free openInternalEntities and freeInternalEntities */
-  entityList = openInternalEntities;
+  /* free m_openInternalEntities and m_freeInternalEntities */
+  entityList = parser->m_openInternalEntities;
   for (;;) {
     OPEN_INTERNAL_ENTITY *openEntity;
     if (entityList == NULL) {
-      if (freeInternalEntities == NULL)
+      if (parser->m_freeInternalEntities == NULL)
         break;
-      entityList = freeInternalEntities;
-      freeInternalEntities = NULL;
+      entityList = parser->m_freeInternalEntities;
+      parser->m_freeInternalEntities = NULL;
     }
     openEntity = entityList;
     entityList = entityList->next;
-    FREE(openEntity);
+    FREE(parser, openEntity);
   }
 
-  destroyBindings(freeBindingList, parser);
-  destroyBindings(inheritedBindings, parser);
-  poolDestroy(&tempPool);
-  poolDestroy(&temp2Pool);
-  FREE((void *)protocolEncodingName);
+  destroyBindings(parser->m_freeBindingList, parser);
+  destroyBindings(parser->m_inheritedBindings, parser);
+  poolDestroy(&parser->m_tempPool);
+  poolDestroy(&parser->m_temp2Pool);
+  FREE(parser, (void *)parser->m_protocolEncodingName);
 #ifdef XML_DTD
   /* external parameter entity parsers share the DTD structure
      parser->m_dtd with the root parser, so we must not destroy it
   */
-  if (!isParamEntity && _dtd)
+  if (! parser->m_isParamEntity && parser->m_dtd)
 #else
-  if (_dtd)
+  if (parser->m_dtd)
 #endif /* XML_DTD */
-    dtdDestroy(_dtd, (XML_Bool)!parentParser, &parser->m_mem);
-  FREE((void *)atts);
+    dtdDestroy(parser->m_dtd, (XML_Bool)! parser->m_parentParser,
+               &parser->m_mem);
+  FREE(parser, (void *)parser->m_atts);
 #ifdef XML_ATTR_INFO
-  FREE((void *)attInfo);
+  FREE(parser, (void *)parser->m_attInfo);
 #endif
-  FREE(groupConnector);
-  FREE(buffer);
-  FREE(dataBuf);
-  FREE(nsAtts);
-  FREE(unknownEncodingMem);
-  if (unknownEncodingRelease)
-    unknownEncodingRelease(unknownEncodingData);
-  FREE(parser);
+  FREE(parser, parser->m_groupConnector);
+  FREE(parser, parser->m_buffer);
+  FREE(parser, parser->m_dataBuf);
+  FREE(parser, parser->m_nsAtts);
+  FREE(parser, parser->m_unknownEncodingMem);
+  if (parser->m_unknownEncodingRelease)
+    parser->m_unknownEncodingRelease(parser->m_unknownEncodingData);
+  FREE(parser, parser);
 }
 
 void XMLCALL
-XML_UseParserAsHandlerArg(XML_Parser parser)
-{
+XML_UseParserAsHandlerArg(XML_Parser parser) {
   if (parser != NULL)
-    handlerArg = parser;
+    parser->m_handlerArg = parser;
 }
 
 enum XML_Error XMLCALL
-XML_UseForeignDTD(XML_Parser parser, XML_Bool useDTD)
-{
+XML_UseForeignDTD(XML_Parser parser, XML_Bool useDTD) {
   if (parser == NULL)
     return XML_ERROR_INVALID_ARGUMENT;
 #ifdef XML_DTD
   /* block after XML_Parse()/XML_ParseBuffer() has been called */
-  if (ps_parsing == XML_PARSING || ps_parsing == XML_SUSPENDED)
+  if (parser->m_parsingStatus.parsing == XML_PARSING
+      || parser->m_parsingStatus.parsing == XML_SUSPENDED)
     return XML_ERROR_CANT_CHANGE_FEATURE_ONCE_PARSING;
-  useForeignDTD = useDTD;
+  parser->m_useForeignDTD = useDTD;
   return XML_ERROR_NONE;
 #else
   return XML_ERROR_FEATURE_REQUIRES_XML_DTD;
@@ -1556,325 +1404,285 @@ XML_UseForeignDTD(XML_Parser parser, XML_Bool useDTD)
 }
 
 void XMLCALL
-XML_SetReturnNSTriplet(XML_Parser parser, int do_nst)
-{
+XML_SetReturnNSTriplet(XML_Parser parser, int do_nst) {
   if (parser == NULL)
     return;
   /* block after XML_Parse()/XML_ParseBuffer() has been called */
-  if (ps_parsing == XML_PARSING || ps_parsing == XML_SUSPENDED)
+  if (parser->m_parsingStatus.parsing == XML_PARSING
+      || parser->m_parsingStatus.parsing == XML_SUSPENDED)
     return;
-  ns_triplets = do_nst ? XML_TRUE : XML_FALSE;
+  parser->m_ns_triplets = do_nst ? XML_TRUE : XML_FALSE;
 }
 
 void XMLCALL
-XML_SetUserData(XML_Parser parser, void *p)
-{
+XML_SetUserData(XML_Parser parser, void *p) {
   if (parser == NULL)
     return;
-  if (handlerArg == userData)
-    handlerArg = userData = p;
+  if (parser->m_handlerArg == parser->m_userData)
+    parser->m_handlerArg = parser->m_userData = p;
   else
-    userData = p;
+    parser->m_userData = p;
 }
 
 enum XML_Status XMLCALL
-XML_SetBase(XML_Parser parser, const XML_Char *p)
-{
+XML_SetBase(XML_Parser parser, const XML_Char *p) {
   if (parser == NULL)
     return XML_STATUS_ERROR;
   if (p) {
-    p = poolCopyString(&_dtd->pool, p);
-    if (!p)
+    p = poolCopyString(&parser->m_dtd->pool, p);
+    if (! p)
       return XML_STATUS_ERROR;
-    curBase = p;
-  }
-  else
-    curBase = NULL;
+    parser->m_curBase = p;
+  } else
+    parser->m_curBase = NULL;
   return XML_STATUS_OK;
 }
 
-const XML_Char * XMLCALL
-XML_GetBase(XML_Parser parser)
-{
+const XML_Char *XMLCALL
+XML_GetBase(XML_Parser parser) {
   if (parser == NULL)
     return NULL;
-  return curBase;
+  return parser->m_curBase;
 }
 
 int XMLCALL
-XML_GetSpecifiedAttributeCount(XML_Parser parser)
-{
+XML_GetSpecifiedAttributeCount(XML_Parser parser) {
   if (parser == NULL)
     return -1;
-  return nSpecifiedAtts;
+  return parser->m_nSpecifiedAtts;
 }
 
 int XMLCALL
-XML_GetIdAttributeIndex(XML_Parser parser)
-{
+XML_GetIdAttributeIndex(XML_Parser parser) {
   if (parser == NULL)
     return -1;
-  return idAttIndex;
+  return parser->m_idAttIndex;
 }
 
 #ifdef XML_ATTR_INFO
-const XML_AttrInfo * XMLCALL
-XML_GetAttributeInfo(XML_Parser parser)
-{
+const XML_AttrInfo *XMLCALL
+XML_GetAttributeInfo(XML_Parser parser) {
   if (parser == NULL)
     return NULL;
-  return attInfo;
+  return parser->m_attInfo;
 }
 #endif
 
 void XMLCALL
-XML_SetElementHandler(XML_Parser parser,
-                      XML_StartElementHandler start,
-                      XML_EndElementHandler end)
-{
+XML_SetElementHandler(XML_Parser parser, XML_StartElementHandler start,
+                      XML_EndElementHandler end) {
   if (parser == NULL)
     return;
-  startElementHandler = start;
-  endElementHandler = end;
+  parser->m_startElementHandler = start;
+  parser->m_endElementHandler = end;
 }
 
 void XMLCALL
-XML_SetStartElementHandler(XML_Parser parser,
-                           XML_StartElementHandler start) {
+XML_SetStartElementHandler(XML_Parser parser, XML_StartElementHandler start) {
   if (parser != NULL)
-    startElementHandler = start;
+    parser->m_startElementHandler = start;
 }
 
 void XMLCALL
-XML_SetEndElementHandler(XML_Parser parser,
-                         XML_EndElementHandler end) {
+XML_SetEndElementHandler(XML_Parser parser, XML_EndElementHandler end) {
   if (parser != NULL)
-    endElementHandler = end;
+    parser->m_endElementHandler = end;
 }
 
 void XMLCALL
 XML_SetCharacterDataHandler(XML_Parser parser,
-                            XML_CharacterDataHandler handler)
-{
+                            XML_CharacterDataHandler handler) {
   if (parser != NULL)
-    characterDataHandler = handler;
+    parser->m_characterDataHandler = handler;
 }
 
 void XMLCALL
 XML_SetProcessingInstructionHandler(XML_Parser parser,
-                                    XML_ProcessingInstructionHandler handler)
-{
+                                    XML_ProcessingInstructionHandler handler) {
   if (parser != NULL)
-    processingInstructionHandler = handler;
+    parser->m_processingInstructionHandler = handler;
 }
 
 void XMLCALL
-XML_SetCommentHandler(XML_Parser parser,
-                      XML_CommentHandler handler)
-{
+XML_SetCommentHandler(XML_Parser parser, XML_CommentHandler handler) {
   if (parser != NULL)
-    commentHandler = handler;
+    parser->m_commentHandler = handler;
 }
 
 void XMLCALL
 XML_SetCdataSectionHandler(XML_Parser parser,
                            XML_StartCdataSectionHandler start,
-                           XML_EndCdataSectionHandler end)
-{
+                           XML_EndCdataSectionHandler end) {
   if (parser == NULL)
     return;
-  startCdataSectionHandler = start;
-  endCdataSectionHandler = end;
+  parser->m_startCdataSectionHandler = start;
+  parser->m_endCdataSectionHandler = end;
 }
 
 void XMLCALL
 XML_SetStartCdataSectionHandler(XML_Parser parser,
                                 XML_StartCdataSectionHandler start) {
   if (parser != NULL)
-    startCdataSectionHandler = start;
+    parser->m_startCdataSectionHandler = start;
 }
 
 void XMLCALL
 XML_SetEndCdataSectionHandler(XML_Parser parser,
                               XML_EndCdataSectionHandler end) {
   if (parser != NULL)
-    endCdataSectionHandler = end;
+    parser->m_endCdataSectionHandler = end;
 }
 
 void XMLCALL
-XML_SetDefaultHandler(XML_Parser parser,
-                      XML_DefaultHandler handler)
-{
+XML_SetDefaultHandler(XML_Parser parser, XML_DefaultHandler handler) {
   if (parser == NULL)
     return;
-  defaultHandler = handler;
-  defaultExpandInternalEntities = XML_FALSE;
+  parser->m_defaultHandler = handler;
+  parser->m_defaultExpandInternalEntities = XML_FALSE;
 }
 
 void XMLCALL
-XML_SetDefaultHandlerExpand(XML_Parser parser,
-                            XML_DefaultHandler handler)
-{
+XML_SetDefaultHandlerExpand(XML_Parser parser, XML_DefaultHandler handler) {
   if (parser == NULL)
     return;
-  defaultHandler = handler;
-  defaultExpandInternalEntities = XML_TRUE;
+  parser->m_defaultHandler = handler;
+  parser->m_defaultExpandInternalEntities = XML_TRUE;
 }
 
 void XMLCALL
-XML_SetDoctypeDeclHandler(XML_Parser parser,
-                          XML_StartDoctypeDeclHandler start,
-                          XML_EndDoctypeDeclHandler end)
-{
+XML_SetDoctypeDeclHandler(XML_Parser parser, XML_StartDoctypeDeclHandler start,
+                          XML_EndDoctypeDeclHandler end) {
   if (parser == NULL)
     return;
-  startDoctypeDeclHandler = start;
-  endDoctypeDeclHandler = end;
+  parser->m_startDoctypeDeclHandler = start;
+  parser->m_endDoctypeDeclHandler = end;
 }
 
 void XMLCALL
 XML_SetStartDoctypeDeclHandler(XML_Parser parser,
                                XML_StartDoctypeDeclHandler start) {
   if (parser != NULL)
-    startDoctypeDeclHandler = start;
+    parser->m_startDoctypeDeclHandler = start;
 }
 
 void XMLCALL
-XML_SetEndDoctypeDeclHandler(XML_Parser parser,
-                             XML_EndDoctypeDeclHandler end) {
+XML_SetEndDoctypeDeclHandler(XML_Parser parser, XML_EndDoctypeDeclHandler end) {
   if (parser != NULL)
-    endDoctypeDeclHandler = end;
+    parser->m_endDoctypeDeclHandler = end;
 }
 
 void XMLCALL
 XML_SetUnparsedEntityDeclHandler(XML_Parser parser,
-                                 XML_UnparsedEntityDeclHandler handler)
-{
+                                 XML_UnparsedEntityDeclHandler handler) {
   if (parser != NULL)
-    unparsedEntityDeclHandler = handler;
+    parser->m_unparsedEntityDeclHandler = handler;
 }
 
 void XMLCALL
-XML_SetNotationDeclHandler(XML_Parser parser,
-                           XML_NotationDeclHandler handler)
-{
+XML_SetNotationDeclHandler(XML_Parser parser, XML_NotationDeclHandler handler) {
   if (parser != NULL)
-    notationDeclHandler = handler;
+    parser->m_notationDeclHandler = handler;
 }
 
 void XMLCALL
 XML_SetNamespaceDeclHandler(XML_Parser parser,
                             XML_StartNamespaceDeclHandler start,
-                            XML_EndNamespaceDeclHandler end)
-{
+                            XML_EndNamespaceDeclHandler end) {
   if (parser == NULL)
     return;
-  startNamespaceDeclHandler = start;
-  endNamespaceDeclHandler = end;
+  parser->m_startNamespaceDeclHandler = start;
+  parser->m_endNamespaceDeclHandler = end;
 }
 
 void XMLCALL
 XML_SetStartNamespaceDeclHandler(XML_Parser parser,
                                  XML_StartNamespaceDeclHandler start) {
   if (parser != NULL)
-    startNamespaceDeclHandler = start;
+    parser->m_startNamespaceDeclHandler = start;
 }
 
 void XMLCALL
 XML_SetEndNamespaceDeclHandler(XML_Parser parser,
                                XML_EndNamespaceDeclHandler end) {
   if (parser != NULL)
-    endNamespaceDeclHandler = end;
+    parser->m_endNamespaceDeclHandler = end;
 }
 
 void XMLCALL
 XML_SetNotStandaloneHandler(XML_Parser parser,
-                            XML_NotStandaloneHandler handler)
-{
+                            XML_NotStandaloneHandler handler) {
   if (parser != NULL)
-    notStandaloneHandler = handler;
+    parser->m_notStandaloneHandler = handler;
 }
 
 void XMLCALL
 XML_SetExternalEntityRefHandler(XML_Parser parser,
-                                XML_ExternalEntityRefHandler handler)
-{
+                                XML_ExternalEntityRefHandler handler) {
   if (parser != NULL)
-    externalEntityRefHandler = handler;
+    parser->m_externalEntityRefHandler = handler;
 }
 
 void XMLCALL
-XML_SetExternalEntityRefHandlerArg(XML_Parser parser, void *arg)
-{
+XML_SetExternalEntityRefHandlerArg(XML_Parser parser, void *arg) {
   if (parser == NULL)
     return;
   if (arg)
-    externalEntityRefHandlerArg = (XML_Parser)arg;
+    parser->m_externalEntityRefHandlerArg = (XML_Parser)arg;
   else
-    externalEntityRefHandlerArg = parser;
+    parser->m_externalEntityRefHandlerArg = parser;
 }
 
 void XMLCALL
 XML_SetSkippedEntityHandler(XML_Parser parser,
-                            XML_SkippedEntityHandler handler)
-{
+                            XML_SkippedEntityHandler handler) {
   if (parser != NULL)
-    skippedEntityHandler = handler;
+    parser->m_skippedEntityHandler = handler;
 }
 
 void XMLCALL
 XML_SetUnknownEncodingHandler(XML_Parser parser,
-                              XML_UnknownEncodingHandler handler,
-                              void *data)
-{
+                              XML_UnknownEncodingHandler handler, void *data) {
   if (parser == NULL)
     return;
-  unknownEncodingHandler = handler;
-  unknownEncodingHandlerData = data;
+  parser->m_unknownEncodingHandler = handler;
+  parser->m_unknownEncodingHandlerData = data;
 }
 
 void XMLCALL
-XML_SetElementDeclHandler(XML_Parser parser,
-                          XML_ElementDeclHandler eldecl)
-{
+XML_SetElementDeclHandler(XML_Parser parser, XML_ElementDeclHandler eldecl) {
   if (parser != NULL)
-    elementDeclHandler = eldecl;
+    parser->m_elementDeclHandler = eldecl;
 }
 
 void XMLCALL
-XML_SetAttlistDeclHandler(XML_Parser parser,
-                          XML_AttlistDeclHandler attdecl)
-{
+XML_SetAttlistDeclHandler(XML_Parser parser, XML_AttlistDeclHandler attdecl) {
   if (parser != NULL)
-    attlistDeclHandler = attdecl;
+    parser->m_attlistDeclHandler = attdecl;
 }
 
 void XMLCALL
-XML_SetEntityDeclHandler(XML_Parser parser,
-                         XML_EntityDeclHandler handler)
-{
+XML_SetEntityDeclHandler(XML_Parser parser, XML_EntityDeclHandler handler) {
   if (parser != NULL)
-    entityDeclHandler = handler;
+    parser->m_entityDeclHandler = handler;
 }
 
 void XMLCALL
-XML_SetXmlDeclHandler(XML_Parser parser,
-                      XML_XmlDeclHandler handler) {
+XML_SetXmlDeclHandler(XML_Parser parser, XML_XmlDeclHandler handler) {
   if (parser != NULL)
-    xmlDeclHandler = handler;
+    parser->m_xmlDeclHandler = handler;
 }
 
 int XMLCALL
 XML_SetParamEntityParsing(XML_Parser parser,
-                          enum XML_ParamEntityParsing peParsing)
-{
+                          enum XML_ParamEntityParsing peParsing) {
   if (parser == NULL)
     return 0;
   /* block after XML_Parse()/XML_ParseBuffer() has been called */
-  if (ps_parsing == XML_PARSING || ps_parsing == XML_SUSPENDED)
+  if (parser->m_parsingStatus.parsing == XML_PARSING
+      || parser->m_parsingStatus.parsing == XML_SUSPENDED)
     return 0;
 #ifdef XML_DTD
-  paramEntityParsing = peParsing;
+  parser->m_paramEntityParsing = peParsing;
   return 1;
 #else
   return peParsing == XML_PARAM_ENTITY_PARSING_NEVER;
@@ -1882,59 +1690,60 @@ XML_SetParamEntityParsing(XML_Parser parser,
 }
 
 int XMLCALL
-XML_SetHashSalt(XML_Parser parser,
-                unsigned long hash_salt)
-{
+XML_SetHashSalt(XML_Parser parser, unsigned long hash_salt) {
   if (parser == NULL)
     return 0;
   if (parser->m_parentParser)
     return XML_SetHashSalt(parser->m_parentParser, hash_salt);
   /* block after XML_Parse()/XML_ParseBuffer() has been called */
-  if (ps_parsing == XML_PARSING || ps_parsing == XML_SUSPENDED)
+  if (parser->m_parsingStatus.parsing == XML_PARSING
+      || parser->m_parsingStatus.parsing == XML_SUSPENDED)
     return 0;
-  hash_secret_salt = hash_salt;
+  parser->m_hash_secret_salt = hash_salt;
   return 1;
 }
 
 enum XML_Status XMLCALL
-XML_Parse(XML_Parser parser, const char *s, int len, int isFinal)
-{
+XML_Parse(XML_Parser parser, const char *s, int len, int isFinal) {
   if ((parser == NULL) || (len < 0) || ((s == NULL) && (len != 0))) {
     if (parser != NULL)
       parser->m_errorCode = XML_ERROR_INVALID_ARGUMENT;
     return XML_STATUS_ERROR;
   }
-  switch (ps_parsing) {
+  switch (parser->m_parsingStatus.parsing) {
   case XML_SUSPENDED:
-    errorCode = XML_ERROR_SUSPENDED;
+    parser->m_errorCode = XML_ERROR_SUSPENDED;
     return XML_STATUS_ERROR;
   case XML_FINISHED:
-    errorCode = XML_ERROR_FINISHED;
+    parser->m_errorCode = XML_ERROR_FINISHED;
     return XML_STATUS_ERROR;
   case XML_INITIALIZED:
-    if (parentParser == NULL && !startParsing(parser)) {
-      errorCode = XML_ERROR_NO_MEMORY;
+    if (parser->m_parentParser == NULL && ! startParsing(parser)) {
+      parser->m_errorCode = XML_ERROR_NO_MEMORY;
       return XML_STATUS_ERROR;
     }
+    /* fall through */
   default:
-    ps_parsing = XML_PARSING;
+    parser->m_parsingStatus.parsing = XML_PARSING;
   }
 
   if (len == 0) {
-    ps_finalBuffer = (XML_Bool)isFinal;
-    if (!isFinal)
+    parser->m_parsingStatus.finalBuffer = (XML_Bool)isFinal;
+    if (! isFinal)
       return XML_STATUS_OK;
-    positionPtr = bufferPtr;
-    parseEndPtr = bufferEnd;
+    parser->m_positionPtr = parser->m_bufferPtr;
+    parser->m_parseEndPtr = parser->m_bufferEnd;
 
     /* If data are left over from last buffer, and we now know that these
        data are the final chunk of input, then we have to check them again
        to detect errors based on that fact.
     */
-    errorCode = processor(parser, bufferPtr, parseEndPtr, &bufferPtr);
+    parser->m_errorCode
+        = parser->m_processor(parser, parser->m_bufferPtr,
+                              parser->m_parseEndPtr, &parser->m_bufferPtr);
 
-    if (errorCode == XML_ERROR_NONE) {
-      switch (ps_parsing) {
+    if (parser->m_errorCode == XML_ERROR_NONE) {
+      switch (parser->m_parsingStatus.parsing) {
       case XML_SUSPENDED:
         /* It is hard to be certain, but it seems that this case
          * cannot occur.  This code is cleaning up a previous parse
@@ -1948,54 +1757,55 @@ XML_Parse(XML_Parser parser, const char *s, int len, int isFinal)
          *
          * LCOV_EXCL_START
          */
-        XmlUpdatePosition(encoding, positionPtr, bufferPtr, &position);
-        positionPtr = bufferPtr;
+        XmlUpdatePosition(parser->m_encoding, parser->m_positionPtr,
+                          parser->m_bufferPtr, &parser->m_position);
+        parser->m_positionPtr = parser->m_bufferPtr;
         return XML_STATUS_SUSPENDED;
         /* LCOV_EXCL_STOP */
       case XML_INITIALIZED:
       case XML_PARSING:
-        ps_parsing = XML_FINISHED;
+        parser->m_parsingStatus.parsing = XML_FINISHED;
         /* fall through */
       default:
         return XML_STATUS_OK;
       }
     }
-    eventEndPtr = eventPtr;
-    processor = errorProcessor;
+    parser->m_eventEndPtr = parser->m_eventPtr;
+    parser->m_processor = errorProcessor;
     return XML_STATUS_ERROR;
   }
 #ifndef XML_CONTEXT_BYTES
-  else if (bufferPtr == bufferEnd) {
+  else if (parser->m_bufferPtr == parser->m_bufferEnd) {
     const char *end;
     int nLeftOver;
     enum XML_Status result;
     /* Detect overflow (a+b > MAX <==> b > MAX-a) */
-    if (len > ((XML_Size)-1) / 2 - parseEndByteIndex) {
-       errorCode = XML_ERROR_NO_MEMORY;
-       eventPtr = eventEndPtr = NULL;
-       processor = errorProcessor;
-       return XML_STATUS_ERROR;
+    if (len > ((XML_Size)-1) / 2 - parser->m_parseEndByteIndex) {
+      parser->m_errorCode = XML_ERROR_NO_MEMORY;
+      parser->m_eventPtr = parser->m_eventEndPtr = NULL;
+      parser->m_processor = errorProcessor;
+      return XML_STATUS_ERROR;
     }
-    parseEndByteIndex += len;
-    positionPtr = s;
-    ps_finalBuffer = (XML_Bool)isFinal;
+    parser->m_parseEndByteIndex += len;
+    parser->m_positionPtr = s;
+    parser->m_parsingStatus.finalBuffer = (XML_Bool)isFinal;
 
-    errorCode = processor(parser, s, parseEndPtr = s + len, &end);
+    parser->m_errorCode
+        = parser->m_processor(parser, s, parser->m_parseEndPtr = s + len, &end);
 
-    if (errorCode != XML_ERROR_NONE) {
-      eventEndPtr = eventPtr;
-      processor = errorProcessor;
+    if (parser->m_errorCode != XML_ERROR_NONE) {
+      parser->m_eventEndPtr = parser->m_eventPtr;
+      parser->m_processor = errorProcessor;
       return XML_STATUS_ERROR;
-    }
-    else {
-      switch (ps_parsing) {
+    } else {
+      switch (parser->m_parsingStatus.parsing) {
       case XML_SUSPENDED:
         result = XML_STATUS_SUSPENDED;
         break;
       case XML_INITIALIZED:
       case XML_PARSING:
         if (isFinal) {
-          ps_parsing = XML_FINISHED;
+          parser->m_parsingStatus.parsing = XML_FINISHED;
           return XML_STATUS_OK;
         }
       /* fall through */
@@ -2004,38 +1814,38 @@ XML_Parse(XML_Parser parser, const char *s, int len, int isFinal)
       }
     }
 
-    XmlUpdatePosition(encoding, positionPtr, end, &position);
+    XmlUpdatePosition(parser->m_encoding, parser->m_positionPtr, end,
+                      &parser->m_position);
     nLeftOver = s + len - end;
     if (nLeftOver) {
-      if (buffer == NULL || nLeftOver > bufferLim - buffer) {
+      if (parser->m_buffer == NULL
+          || nLeftOver > parser->m_bufferLim - parser->m_buffer) {
         /* avoid _signed_ integer overflow */
         char *temp = NULL;
         const int bytesToAllocate = (int)((unsigned)len * 2U);
         if (bytesToAllocate > 0) {
-          temp = (buffer == NULL
-                ? (char *)MALLOC(bytesToAllocate)
-                : (char *)REALLOC(buffer, bytesToAllocate));
+          temp = (char *)REALLOC(parser, parser->m_buffer, bytesToAllocate);
         }
         if (temp == NULL) {
-          errorCode = XML_ERROR_NO_MEMORY;
-          eventPtr = eventEndPtr = NULL;
-          processor = errorProcessor;
+          parser->m_errorCode = XML_ERROR_NO_MEMORY;
+          parser->m_eventPtr = parser->m_eventEndPtr = NULL;
+          parser->m_processor = errorProcessor;
           return XML_STATUS_ERROR;
         }
-        buffer = temp;
-        bufferLim = buffer + bytesToAllocate;
+        parser->m_buffer = temp;
+        parser->m_bufferLim = parser->m_buffer + bytesToAllocate;
       }
-      memcpy(buffer, end, nLeftOver);
+      memcpy(parser->m_buffer, end, nLeftOver);
     }
-    bufferPtr = buffer;
-    bufferEnd = buffer + nLeftOver;
-    positionPtr = bufferPtr;
-    parseEndPtr = bufferEnd;
-    eventPtr = bufferPtr;
-    eventEndPtr = bufferPtr;
+    parser->m_bufferPtr = parser->m_buffer;
+    parser->m_bufferEnd = parser->m_buffer + nLeftOver;
+    parser->m_positionPtr = parser->m_bufferPtr;
+    parser->m_parseEndPtr = parser->m_bufferEnd;
+    parser->m_eventPtr = parser->m_bufferPtr;
+    parser->m_eventEndPtr = parser->m_bufferPtr;
     return result;
   }
-#endif  /* not defined XML_CONTEXT_BYTES */
+#endif /* not defined XML_CONTEXT_BYTES */
   else {
     void *buff = XML_GetBuffer(parser, len);
     if (buff == NULL)
@@ -2048,236 +1858,254 @@ XML_Parse(XML_Parser parser, const char *s, int len, int isFinal)
 }
 
 enum XML_Status XMLCALL
-XML_ParseBuffer(XML_Parser parser, int len, int isFinal)
-{
+XML_ParseBuffer(XML_Parser parser, int len, int isFinal) {
   const char *start;
   enum XML_Status result = XML_STATUS_OK;
 
   if (parser == NULL)
     return XML_STATUS_ERROR;
-  switch (ps_parsing) {
+  switch (parser->m_parsingStatus.parsing) {
   case XML_SUSPENDED:
-    errorCode = XML_ERROR_SUSPENDED;
+    parser->m_errorCode = XML_ERROR_SUSPENDED;
     return XML_STATUS_ERROR;
   case XML_FINISHED:
-    errorCode = XML_ERROR_FINISHED;
+    parser->m_errorCode = XML_ERROR_FINISHED;
     return XML_STATUS_ERROR;
   case XML_INITIALIZED:
-    if (parentParser == NULL && !startParsing(parser)) {
-      errorCode = XML_ERROR_NO_MEMORY;
+    if (parser->m_parentParser == NULL && ! startParsing(parser)) {
+      parser->m_errorCode = XML_ERROR_NO_MEMORY;
       return XML_STATUS_ERROR;
     }
+    /* fall through */
   default:
-    ps_parsing = XML_PARSING;
+    parser->m_parsingStatus.parsing = XML_PARSING;
   }
 
-  start = bufferPtr;
-  positionPtr = start;
-  bufferEnd += len;
-  parseEndPtr = bufferEnd;
-  parseEndByteIndex += len;
-  ps_finalBuffer = (XML_Bool)isFinal;
+  start = parser->m_bufferPtr;
+  parser->m_positionPtr = start;
+  parser->m_bufferEnd += len;
+  parser->m_parseEndPtr = parser->m_bufferEnd;
+  parser->m_parseEndByteIndex += len;
+  parser->m_parsingStatus.finalBuffer = (XML_Bool)isFinal;
 
-  errorCode = processor(parser, start, parseEndPtr, &bufferPtr);
+  parser->m_errorCode = parser->m_processor(
+      parser, start, parser->m_parseEndPtr, &parser->m_bufferPtr);
 
-  if (errorCode != XML_ERROR_NONE) {
-    eventEndPtr = eventPtr;
-    processor = errorProcessor;
+  if (parser->m_errorCode != XML_ERROR_NONE) {
+    parser->m_eventEndPtr = parser->m_eventPtr;
+    parser->m_processor = errorProcessor;
     return XML_STATUS_ERROR;
-  }
-  else {
-    switch (ps_parsing) {
+  } else {
+    switch (parser->m_parsingStatus.parsing) {
     case XML_SUSPENDED:
       result = XML_STATUS_SUSPENDED;
       break;
     case XML_INITIALIZED:
     case XML_PARSING:
       if (isFinal) {
-        ps_parsing = XML_FINISHED;
+        parser->m_parsingStatus.parsing = XML_FINISHED;
         return result;
       }
-    default: ;  /* should not happen */
+    default:; /* should not happen */
     }
   }
 
-  XmlUpdatePosition(encoding, positionPtr, bufferPtr, &position);
-  positionPtr = bufferPtr;
+  XmlUpdatePosition(parser->m_encoding, parser->m_positionPtr,
+                    parser->m_bufferPtr, &parser->m_position);
+  parser->m_positionPtr = parser->m_bufferPtr;
   return result;
 }
 
-void * XMLCALL
-XML_GetBuffer(XML_Parser parser, int len)
-{
+void *XMLCALL
+XML_GetBuffer(XML_Parser parser, int len) {
   if (parser == NULL)
     return NULL;
   if (len < 0) {
-    errorCode = XML_ERROR_NO_MEMORY;
+    parser->m_errorCode = XML_ERROR_NO_MEMORY;
     return NULL;
   }
-  switch (ps_parsing) {
+  switch (parser->m_parsingStatus.parsing) {
   case XML_SUSPENDED:
-    errorCode = XML_ERROR_SUSPENDED;
+    parser->m_errorCode = XML_ERROR_SUSPENDED;
     return NULL;
   case XML_FINISHED:
-    errorCode = XML_ERROR_FINISHED;
+    parser->m_errorCode = XML_ERROR_FINISHED;
     return NULL;
-  default: ;
+  default:;
   }
 
-  if (len > bufferLim - bufferEnd) {
+  if (len > EXPAT_SAFE_PTR_DIFF(parser->m_bufferLim, parser->m_bufferEnd)) {
 #ifdef XML_CONTEXT_BYTES
     int keep;
-#endif  /* defined XML_CONTEXT_BYTES */
+#endif /* defined XML_CONTEXT_BYTES */
     /* Do not invoke signed arithmetic overflow: */
-    int neededSize = (int) ((unsigned)len + (unsigned)(bufferEnd - bufferPtr));
+    int neededSize = (int)((unsigned)len
+                           + (unsigned)EXPAT_SAFE_PTR_DIFF(
+                               parser->m_bufferEnd, parser->m_bufferPtr));
     if (neededSize < 0) {
-      errorCode = XML_ERROR_NO_MEMORY;
+      parser->m_errorCode = XML_ERROR_NO_MEMORY;
       return NULL;
     }
 #ifdef XML_CONTEXT_BYTES
-    keep = (int)(bufferPtr - buffer);
+    keep = (int)EXPAT_SAFE_PTR_DIFF(parser->m_bufferPtr, parser->m_buffer);
     if (keep > XML_CONTEXT_BYTES)
       keep = XML_CONTEXT_BYTES;
     neededSize += keep;
-#endif  /* defined XML_CONTEXT_BYTES */
-    if (neededSize  <= bufferLim - buffer) {
+#endif /* defined XML_CONTEXT_BYTES */
+    if (neededSize
+        <= EXPAT_SAFE_PTR_DIFF(parser->m_bufferLim, parser->m_buffer)) {
 #ifdef XML_CONTEXT_BYTES
-      if (keep < bufferPtr - buffer) {
-        int offset = (int)(bufferPtr - buffer) - keep;
-        memmove(buffer, &buffer[offset], bufferEnd - bufferPtr + keep);
-        bufferEnd -= offset;
-        bufferPtr -= offset;
+      if (keep < EXPAT_SAFE_PTR_DIFF(parser->m_bufferPtr, parser->m_buffer)) {
+        int offset
+            = (int)EXPAT_SAFE_PTR_DIFF(parser->m_bufferPtr, parser->m_buffer)
+              - keep;
+        /* The buffer pointers cannot be NULL here; we have at least some bytes
+         * in the buffer */
+        memmove(parser->m_buffer, &parser->m_buffer[offset],
+                parser->m_bufferEnd - parser->m_bufferPtr + keep);
+        parser->m_bufferEnd -= offset;
+        parser->m_bufferPtr -= offset;
       }
 #else
-      memmove(buffer, bufferPtr, bufferEnd - bufferPtr);
-      bufferEnd = buffer + (bufferEnd - bufferPtr);
-      bufferPtr = buffer;
-#endif  /* not defined XML_CONTEXT_BYTES */
-    }
-    else {
+      if (parser->m_buffer && parser->m_bufferPtr) {
+        memmove(parser->m_buffer, parser->m_bufferPtr,
+                EXPAT_SAFE_PTR_DIFF(parser->m_bufferEnd, parser->m_bufferPtr));
+        parser->m_bufferEnd
+            = parser->m_buffer
+              + EXPAT_SAFE_PTR_DIFF(parser->m_bufferEnd, parser->m_bufferPtr);
+        parser->m_bufferPtr = parser->m_buffer;
+      }
+#endif /* not defined XML_CONTEXT_BYTES */
+    } else {
       char *newBuf;
-      int bufferSize = (int)(bufferLim - bufferPtr);
+      int bufferSize
+          = (int)EXPAT_SAFE_PTR_DIFF(parser->m_bufferLim, parser->m_bufferPtr);
       if (bufferSize == 0)
         bufferSize = INIT_BUFFER_SIZE;
       do {
         /* Do not invoke signed arithmetic overflow: */
-        bufferSize = (int) (2U * (unsigned) bufferSize);
+        bufferSize = (int)(2U * (unsigned)bufferSize);
       } while (bufferSize < neededSize && bufferSize > 0);
       if (bufferSize <= 0) {
-        errorCode = XML_ERROR_NO_MEMORY;
+        parser->m_errorCode = XML_ERROR_NO_MEMORY;
         return NULL;
       }
-      newBuf = (char *)MALLOC(bufferSize);
+      newBuf = (char *)MALLOC(parser, bufferSize);
       if (newBuf == 0) {
-        errorCode = XML_ERROR_NO_MEMORY;
+        parser->m_errorCode = XML_ERROR_NO_MEMORY;
         return NULL;
       }
-      bufferLim = newBuf + bufferSize;
+      parser->m_bufferLim = newBuf + bufferSize;
 #ifdef XML_CONTEXT_BYTES
-      if (bufferPtr) {
-        int keep = (int)(bufferPtr - buffer);
-        if (keep > XML_CONTEXT_BYTES)
-          keep = XML_CONTEXT_BYTES;
-        memcpy(newBuf, &bufferPtr[-keep], bufferEnd - bufferPtr + keep);
-        FREE(buffer);
-        buffer = newBuf;
-        bufferEnd = buffer + (bufferEnd - bufferPtr) + keep;
-        bufferPtr = buffer + keep;
-      }
-      else {
-        bufferEnd = newBuf + (bufferEnd - bufferPtr);
-        bufferPtr = buffer = newBuf;
+      if (parser->m_bufferPtr) {
+        memcpy(newBuf, &parser->m_bufferPtr[-keep],
+               EXPAT_SAFE_PTR_DIFF(parser->m_bufferEnd, parser->m_bufferPtr)
+                   + keep);
+        FREE(parser, parser->m_buffer);
+        parser->m_buffer = newBuf;
+        parser->m_bufferEnd
+            = parser->m_buffer
+              + EXPAT_SAFE_PTR_DIFF(parser->m_bufferEnd, parser->m_bufferPtr)
+              + keep;
+        parser->m_bufferPtr = parser->m_buffer + keep;
+      } else {
+        /* This must be a brand new buffer with no data in it yet */
+        parser->m_bufferEnd = newBuf;
+        parser->m_bufferPtr = parser->m_buffer = newBuf;
       }
 #else
-      if (bufferPtr) {
-        memcpy(newBuf, bufferPtr, bufferEnd - bufferPtr);
-        FREE(buffer);
+      if (parser->m_bufferPtr) {
+        memcpy(newBuf, parser->m_bufferPtr,
+               EXPAT_SAFE_PTR_DIFF(parser->m_bufferEnd, parser->m_bufferPtr));
+        FREE(parser, parser->m_buffer);
+        parser->m_bufferEnd
+            = newBuf
+              + EXPAT_SAFE_PTR_DIFF(parser->m_bufferEnd, parser->m_bufferPtr);
+      } else {
+        /* This must be a brand new buffer with no data in it yet */
+        parser->m_bufferEnd = newBuf;
       }
-      bufferEnd = newBuf + (bufferEnd - bufferPtr);
-      bufferPtr = buffer = newBuf;
-#endif  /* not defined XML_CONTEXT_BYTES */
+      parser->m_bufferPtr = parser->m_buffer = newBuf;
+#endif /* not defined XML_CONTEXT_BYTES */
     }
-    eventPtr = eventEndPtr = NULL;
-    positionPtr = NULL;
+    parser->m_eventPtr = parser->m_eventEndPtr = NULL;
+    parser->m_positionPtr = NULL;
   }
-  return bufferEnd;
+  return parser->m_bufferEnd;
 }
 
 enum XML_Status XMLCALL
-XML_StopParser(XML_Parser parser, XML_Bool resumable)
-{
+XML_StopParser(XML_Parser parser, XML_Bool resumable) {
   if (parser == NULL)
     return XML_STATUS_ERROR;
-  switch (ps_parsing) {
+  switch (parser->m_parsingStatus.parsing) {
   case XML_SUSPENDED:
     if (resumable) {
-      errorCode = XML_ERROR_SUSPENDED;
+      parser->m_errorCode = XML_ERROR_SUSPENDED;
       return XML_STATUS_ERROR;
     }
-    ps_parsing = XML_FINISHED;
+    parser->m_parsingStatus.parsing = XML_FINISHED;
     break;
   case XML_FINISHED:
-    errorCode = XML_ERROR_FINISHED;
+    parser->m_errorCode = XML_ERROR_FINISHED;
     return XML_STATUS_ERROR;
   default:
     if (resumable) {
 #ifdef XML_DTD
-      if (isParamEntity) {
-        errorCode = XML_ERROR_SUSPEND_PE;
+      if (parser->m_isParamEntity) {
+        parser->m_errorCode = XML_ERROR_SUSPEND_PE;
         return XML_STATUS_ERROR;
       }
 #endif
-      ps_parsing = XML_SUSPENDED;
-    }
-    else
-      ps_parsing = XML_FINISHED;
+      parser->m_parsingStatus.parsing = XML_SUSPENDED;
+    } else
+      parser->m_parsingStatus.parsing = XML_FINISHED;
   }
   return XML_STATUS_OK;
 }
 
 enum XML_Status XMLCALL
-XML_ResumeParser(XML_Parser parser)
-{
+XML_ResumeParser(XML_Parser parser) {
   enum XML_Status result = XML_STATUS_OK;
 
   if (parser == NULL)
     return XML_STATUS_ERROR;
-  if (ps_parsing != XML_SUSPENDED) {
-    errorCode = XML_ERROR_NOT_SUSPENDED;
+  if (parser->m_parsingStatus.parsing != XML_SUSPENDED) {
+    parser->m_errorCode = XML_ERROR_NOT_SUSPENDED;
     return XML_STATUS_ERROR;
   }
-  ps_parsing = XML_PARSING;
+  parser->m_parsingStatus.parsing = XML_PARSING;
 
-  errorCode = processor(parser, bufferPtr, parseEndPtr, &bufferPtr);
+  parser->m_errorCode = parser->m_processor(
+      parser, parser->m_bufferPtr, parser->m_parseEndPtr, &parser->m_bufferPtr);
 
-  if (errorCode != XML_ERROR_NONE) {
-    eventEndPtr = eventPtr;
-    processor = errorProcessor;
+  if (parser->m_errorCode != XML_ERROR_NONE) {
+    parser->m_eventEndPtr = parser->m_eventPtr;
+    parser->m_processor = errorProcessor;
     return XML_STATUS_ERROR;
-  }
-  else {
-    switch (ps_parsing) {
+  } else {
+    switch (parser->m_parsingStatus.parsing) {
     case XML_SUSPENDED:
       result = XML_STATUS_SUSPENDED;
       break;
     case XML_INITIALIZED:
     case XML_PARSING:
-      if (ps_finalBuffer) {
-        ps_parsing = XML_FINISHED;
+      if (parser->m_parsingStatus.finalBuffer) {
+        parser->m_parsingStatus.parsing = XML_FINISHED;
         return result;
       }
-    default: ;
+    default:;
     }
   }
 
-  XmlUpdatePosition(encoding, positionPtr, bufferPtr, &position);
-  positionPtr = bufferPtr;
+  XmlUpdatePosition(parser->m_encoding, parser->m_positionPtr,
+                    parser->m_bufferPtr, &parser->m_position);
+  parser->m_positionPtr = parser->m_bufferPtr;
   return result;
 }
 
 void XMLCALL
-XML_GetParsingStatus(XML_Parser parser, XML_ParsingStatus *status)
-{
+XML_GetParsingStatus(XML_Parser parser, XML_ParsingStatus *status) {
   if (parser == NULL)
     return;
   assert(status != NULL);
@@ -2285,178 +2113,215 @@ XML_GetParsingStatus(XML_Parser parser, XML_ParsingStatus *status)
 }
 
 enum XML_Error XMLCALL
-XML_GetErrorCode(XML_Parser parser)
-{
+XML_GetErrorCode(XML_Parser parser) {
   if (parser == NULL)
     return XML_ERROR_INVALID_ARGUMENT;
-  return errorCode;
+  return parser->m_errorCode;
 }
 
 XML_Index XMLCALL
-XML_GetCurrentByteIndex(XML_Parser parser)
-{
+XML_GetCurrentByteIndex(XML_Parser parser) {
   if (parser == NULL)
     return -1;
-  if (eventPtr)
-    return (XML_Index)(parseEndByteIndex - (parseEndPtr - eventPtr));
+  if (parser->m_eventPtr)
+    return (XML_Index)(parser->m_parseEndByteIndex
+                       - (parser->m_parseEndPtr - parser->m_eventPtr));
   return -1;
 }
 
 int XMLCALL
-XML_GetCurrentByteCount(XML_Parser parser)
-{
+XML_GetCurrentByteCount(XML_Parser parser) {
   if (parser == NULL)
     return 0;
-  if (eventEndPtr && eventPtr)
-    return (int)(eventEndPtr - eventPtr);
+  if (parser->m_eventEndPtr && parser->m_eventPtr)
+    return (int)(parser->m_eventEndPtr - parser->m_eventPtr);
   return 0;
 }
 
-const char * XMLCALL
-XML_GetInputContext(XML_Parser parser, int *offset, int *size)
-{
+const char *XMLCALL
+XML_GetInputContext(XML_Parser parser, int *offset, int *size) {
 #ifdef XML_CONTEXT_BYTES
   if (parser == NULL)
     return NULL;
-  if (eventPtr && buffer) {
+  if (parser->m_eventPtr && parser->m_buffer) {
     if (offset != NULL)
-      *offset = (int)(eventPtr - buffer);
+      *offset = (int)(parser->m_eventPtr - parser->m_buffer);
     if (size != NULL)
-      *size   = (int)(bufferEnd - buffer);
-    return buffer;
+      *size = (int)(parser->m_bufferEnd - parser->m_buffer);
+    return parser->m_buffer;
   }
 #else
   (void)parser;
   (void)offset;
   (void)size;
 #endif /* defined XML_CONTEXT_BYTES */
-  return (char *) 0;
+  return (char *)0;
 }
 
 XML_Size XMLCALL
-XML_GetCurrentLineNumber(XML_Parser parser)
-{
+XML_GetCurrentLineNumber(XML_Parser parser) {
   if (parser == NULL)
     return 0;
-  if (eventPtr && eventPtr >= positionPtr) {
-    XmlUpdatePosition(encoding, positionPtr, eventPtr, &position);
-    positionPtr = eventPtr;
+  if (parser->m_eventPtr && parser->m_eventPtr >= parser->m_positionPtr) {
+    XmlUpdatePosition(parser->m_encoding, parser->m_positionPtr,
+                      parser->m_eventPtr, &parser->m_position);
+    parser->m_positionPtr = parser->m_eventPtr;
   }
-  return position.lineNumber + 1;
+  return parser->m_position.lineNumber + 1;
 }
 
 XML_Size XMLCALL
-XML_GetCurrentColumnNumber(XML_Parser parser)
-{
+XML_GetCurrentColumnNumber(XML_Parser parser) {
   if (parser == NULL)
     return 0;
-  if (eventPtr && eventPtr >= positionPtr) {
-    XmlUpdatePosition(encoding, positionPtr, eventPtr, &position);
-    positionPtr = eventPtr;
+  if (parser->m_eventPtr && parser->m_eventPtr >= parser->m_positionPtr) {
+    XmlUpdatePosition(parser->m_encoding, parser->m_positionPtr,
+                      parser->m_eventPtr, &parser->m_position);
+    parser->m_positionPtr = parser->m_eventPtr;
   }
-  return position.columnNumber;
+  return parser->m_position.columnNumber;
 }
 
 void XMLCALL
-XML_FreeContentModel(XML_Parser parser, XML_Content *model)
-{
+XML_FreeContentModel(XML_Parser parser, XML_Content *model) {
   if (parser != NULL)
-    FREE(model);
+    FREE(parser, model);
 }
 
-void * XMLCALL
-XML_MemMalloc(XML_Parser parser, size_t size)
-{
+void *XMLCALL
+XML_MemMalloc(XML_Parser parser, size_t size) {
   if (parser == NULL)
     return NULL;
-  return MALLOC(size);
+  return MALLOC(parser, size);
 }
 
-void * XMLCALL
-XML_MemRealloc(XML_Parser parser, void *ptr, size_t size)
-{
+void *XMLCALL
+XML_MemRealloc(XML_Parser parser, void *ptr, size_t size) {
   if (parser == NULL)
     return NULL;
-  return REALLOC(ptr, size);
+  return REALLOC(parser, ptr, size);
 }
 
 void XMLCALL
-XML_MemFree(XML_Parser parser, void *ptr)
-{
+XML_MemFree(XML_Parser parser, void *ptr) {
   if (parser != NULL)
-    FREE(ptr);
+    FREE(parser, ptr);
 }
 
 void XMLCALL
-XML_DefaultCurrent(XML_Parser parser)
-{
+XML_DefaultCurrent(XML_Parser parser) {
   if (parser == NULL)
     return;
-  if (defaultHandler) {
-    if (openInternalEntities)
-      reportDefault(parser,
-                    internalEncoding,
-                    openInternalEntities->internalEventPtr,
-                    openInternalEntities->internalEventEndPtr);
+  if (parser->m_defaultHandler) {
+    if (parser->m_openInternalEntities)
+      reportDefault(parser, parser->m_internalEncoding,
+                    parser->m_openInternalEntities->internalEventPtr,
+                    parser->m_openInternalEntities->internalEventEndPtr);
     else
-      reportDefault(parser, encoding, eventPtr, eventEndPtr);
+      reportDefault(parser, parser->m_encoding, parser->m_eventPtr,
+                    parser->m_eventEndPtr);
   }
 }
 
-const XML_LChar * XMLCALL
-XML_ErrorString(enum XML_Error code)
-{
-  static const XML_LChar* const message[] = {
-    0,
-    XML_L("out of memory"),
-    XML_L("syntax error"),
-    XML_L("no element found"),
-    XML_L("not well-formed (invalid token)"),
-    XML_L("unclosed token"),
-    XML_L("partial character"),
-    XML_L("mismatched tag"),
-    XML_L("duplicate attribute"),
-    XML_L("junk after document element"),
-    XML_L("illegal parameter entity reference"),
-    XML_L("undefined entity"),
-    XML_L("recursive entity reference"),
-    XML_L("asynchronous entity"),
-    XML_L("reference to invalid character number"),
-    XML_L("reference to binary entity"),
-    XML_L("reference to external entity in attribute"),
-    XML_L("XML or text declaration not at start of entity"),
-    XML_L("unknown encoding"),
-    XML_L("encoding specified in XML declaration is incorrect"),
-    XML_L("unclosed CDATA section"),
-    XML_L("error in processing external entity reference"),
-    XML_L("document is not standalone"),
-    XML_L("unexpected parser state - please send a bug report"),
-    XML_L("entity declared in parameter entity"),
-    XML_L("requested feature requires XML_DTD support in Expat"),
-    XML_L("cannot change setting once parsing has begun"),
-    XML_L("unbound prefix"),
-    XML_L("must not undeclare prefix"),
-    XML_L("incomplete markup in parameter entity"),
-    XML_L("XML declaration not well-formed"),
-    XML_L("text declaration not well-formed"),
-    XML_L("illegal character(s) in public id"),
-    XML_L("parser suspended"),
-    XML_L("parser not suspended"),
-    XML_L("parsing aborted"),
-    XML_L("parsing finished"),
-    XML_L("cannot suspend in external parameter entity"),
-    XML_L("reserved prefix (xml) must not be undeclared or bound to another namespace name"),
-    XML_L("reserved prefix (xmlns) must not be declared or undeclared"),
-    XML_L("prefix must not be bound to one of the reserved namespace names")
-  };
-  if (code > 0 && code < sizeof(message)/sizeof(message[0]))
-    return message[code];
+const XML_LChar *XMLCALL
+XML_ErrorString(enum XML_Error code) {
+  switch (code) {
+  case XML_ERROR_NONE:
+    return NULL;
+  case XML_ERROR_NO_MEMORY:
+    return XML_L("out of memory");
+  case XML_ERROR_SYNTAX:
+    return XML_L("syntax error");
+  case XML_ERROR_NO_ELEMENTS:
+    return XML_L("no element found");
+  case XML_ERROR_INVALID_TOKEN:
+    return XML_L("not well-formed (invalid token)");
+  case XML_ERROR_UNCLOSED_TOKEN:
+    return XML_L("unclosed token");
+  case XML_ERROR_PARTIAL_CHAR:
+    return XML_L("partial character");
+  case XML_ERROR_TAG_MISMATCH:
+    return XML_L("mismatched tag");
+  case XML_ERROR_DUPLICATE_ATTRIBUTE:
+    return XML_L("duplicate attribute");
+  case XML_ERROR_JUNK_AFTER_DOC_ELEMENT:
+    return XML_L("junk after document element");
+  case XML_ERROR_PARAM_ENTITY_REF:
+    return XML_L("illegal parameter entity reference");
+  case XML_ERROR_UNDEFINED_ENTITY:
+    return XML_L("undefined entity");
+  case XML_ERROR_RECURSIVE_ENTITY_REF:
+    return XML_L("recursive entity reference");
+  case XML_ERROR_ASYNC_ENTITY:
+    return XML_L("asynchronous entity");
+  case XML_ERROR_BAD_CHAR_REF:
+    return XML_L("reference to invalid character number");
+  case XML_ERROR_BINARY_ENTITY_REF:
+    return XML_L("reference to binary entity");
+  case XML_ERROR_ATTRIBUTE_EXTERNAL_ENTITY_REF:
+    return XML_L("reference to external entity in attribute");
+  case XML_ERROR_MISPLACED_XML_PI:
+    return XML_L("XML or text declaration not at start of entity");
+  case XML_ERROR_UNKNOWN_ENCODING:
+    return XML_L("unknown encoding");
+  case XML_ERROR_INCORRECT_ENCODING:
+    return XML_L("encoding specified in XML declaration is incorrect");
+  case XML_ERROR_UNCLOSED_CDATA_SECTION:
+    return XML_L("unclosed CDATA section");
+  case XML_ERROR_EXTERNAL_ENTITY_HANDLING:
+    return XML_L("error in processing external entity reference");
+  case XML_ERROR_NOT_STANDALONE:
+    return XML_L("document is not standalone");
+  case XML_ERROR_UNEXPECTED_STATE:
+    return XML_L("unexpected parser state - please send a bug report");
+  case XML_ERROR_ENTITY_DECLARED_IN_PE:
+    return XML_L("entity declared in parameter entity");
+  case XML_ERROR_FEATURE_REQUIRES_XML_DTD:
+    return XML_L("requested feature requires XML_DTD support in Expat");
+  case XML_ERROR_CANT_CHANGE_FEATURE_ONCE_PARSING:
+    return XML_L("cannot change setting once parsing has begun");
+  /* Added in 1.95.7. */
+  case XML_ERROR_UNBOUND_PREFIX:
+    return XML_L("unbound prefix");
+  /* Added in 1.95.8. */
+  case XML_ERROR_UNDECLARING_PREFIX:
+    return XML_L("must not undeclare prefix");
+  case XML_ERROR_INCOMPLETE_PE:
+    return XML_L("incomplete markup in parameter entity");
+  case XML_ERROR_XML_DECL:
+    return XML_L("XML declaration not well-formed");
+  case XML_ERROR_TEXT_DECL:
+    return XML_L("text declaration not well-formed");
+  case XML_ERROR_PUBLICID:
+    return XML_L("illegal character(s) in public id");
+  case XML_ERROR_SUSPENDED:
+    return XML_L("parser suspended");
+  case XML_ERROR_NOT_SUSPENDED:
+    return XML_L("parser not suspended");
+  case XML_ERROR_ABORTED:
+    return XML_L("parsing aborted");
+  case XML_ERROR_FINISHED:
+    return XML_L("parsing finished");
+  case XML_ERROR_SUSPEND_PE:
+    return XML_L("cannot suspend in external parameter entity");
+  /* Added in 2.0.0. */
+  case XML_ERROR_RESERVED_PREFIX_XML:
+    return XML_L(
+        "reserved prefix (xml) must not be undeclared or bound to another namespace name");
+  case XML_ERROR_RESERVED_PREFIX_XMLNS:
+    return XML_L("reserved prefix (xmlns) must not be declared or undeclared");
+  case XML_ERROR_RESERVED_NAMESPACE_URI:
+    return XML_L(
+        "prefix must not be bound to one of the reserved namespace names");
+  /* Added in 2.2.5. */
+  case XML_ERROR_INVALID_ARGUMENT: /* Constant added in 2.2.1, already */
+    return XML_L("invalid argument");
+  }
   return NULL;
 }
 
-const XML_LChar * XMLCALL
+const XML_LChar *XMLCALL
 XML_ExpatVersion(void) {
-
   /* V1 is used to string-ize the version number. However, it would
      string-ize the actual version macro *names* unless we get them
      substituted before being passed to V1. CPP is defined to expand
@@ -2465,8 +2330,8 @@ XML_ExpatVersion(void) {
      with the correct numerals. */
   /* ### I'm assuming cpp is portable in this respect... */
 
-#define V1(a,b,c) XML_L(#a)XML_L(".")XML_L(#b)XML_L(".")XML_L(#c)
-#define V2(a,b,c) XML_L("expat_")V1(a,b,c)
+#define V1(a, b, c) XML_L(#a) XML_L(".") XML_L(#b) XML_L(".") XML_L(#c)
+#define V2(a, b, c) XML_L("expat_") V1(a, b, c)
 
   return V2(XML_MAJOR_VERSION, XML_MINOR_VERSION, XML_MICRO_VERSION);
 
@@ -2475,8 +2340,7 @@ XML_ExpatVersion(void) {
 }
 
 XML_Expat_Version XMLCALL
-XML_ExpatVersionInfo(void)
-{
+XML_ExpatVersionInfo(void) {
   XML_Expat_Version version;
 
   version.major = XML_MAJOR_VERSION;
@@ -2486,41 +2350,39 @@ XML_ExpatVersionInfo(void)
   return version;
 }
 
-const XML_Feature * XMLCALL
-XML_GetFeatureList(void)
-{
-  static const XML_Feature features[] = {
-    {XML_FEATURE_SIZEOF_XML_CHAR,  XML_L("sizeof(XML_Char)"),
-     sizeof(XML_Char)},
-    {XML_FEATURE_SIZEOF_XML_LCHAR, XML_L("sizeof(XML_LChar)"),
-     sizeof(XML_LChar)},
+const XML_Feature *XMLCALL
+XML_GetFeatureList(void) {
+  static const XML_Feature features[]
+      = {{XML_FEATURE_SIZEOF_XML_CHAR, XML_L("sizeof(XML_Char)"),
+          sizeof(XML_Char)},
+         {XML_FEATURE_SIZEOF_XML_LCHAR, XML_L("sizeof(XML_LChar)"),
+          sizeof(XML_LChar)},
 #ifdef XML_UNICODE
-    {XML_FEATURE_UNICODE,          XML_L("XML_UNICODE"), 0},
+         {XML_FEATURE_UNICODE, XML_L("XML_UNICODE"), 0},
 #endif
 #ifdef XML_UNICODE_WCHAR_T
-    {XML_FEATURE_UNICODE_WCHAR_T,  XML_L("XML_UNICODE_WCHAR_T"), 0},
+         {XML_FEATURE_UNICODE_WCHAR_T, XML_L("XML_UNICODE_WCHAR_T"), 0},
 #endif
 #ifdef XML_DTD
-    {XML_FEATURE_DTD,              XML_L("XML_DTD"), 0},
+         {XML_FEATURE_DTD, XML_L("XML_DTD"), 0},
 #endif
 #ifdef XML_CONTEXT_BYTES
-    {XML_FEATURE_CONTEXT_BYTES,    XML_L("XML_CONTEXT_BYTES"),
-     XML_CONTEXT_BYTES},
+         {XML_FEATURE_CONTEXT_BYTES, XML_L("XML_CONTEXT_BYTES"),
+          XML_CONTEXT_BYTES},
 #endif
 #ifdef XML_MIN_SIZE
-    {XML_FEATURE_MIN_SIZE,         XML_L("XML_MIN_SIZE"), 0},
+         {XML_FEATURE_MIN_SIZE, XML_L("XML_MIN_SIZE"), 0},
 #endif
 #ifdef XML_NS
-    {XML_FEATURE_NS,               XML_L("XML_NS"), 0},
+         {XML_FEATURE_NS, XML_L("XML_NS"), 0},
 #endif
 #ifdef XML_LARGE_SIZE
-    {XML_FEATURE_LARGE_SIZE,       XML_L("XML_LARGE_SIZE"), 0},
+         {XML_FEATURE_LARGE_SIZE, XML_L("XML_LARGE_SIZE"), 0},
 #endif
 #ifdef XML_ATTR_INFO
-    {XML_FEATURE_ATTR_INFO,        XML_L("XML_ATTR_INFO"), 0},
+         {XML_FEATURE_ATTR_INFO, XML_L("XML_ATTR_INFO"), 0},
 #endif
-    {XML_FEATURE_END,              NULL, 0}
-  };
+         {XML_FEATURE_END, NULL, 0}};
 
   return features;
 }
@@ -2531,14 +2393,13 @@ XML_GetFeatureList(void)
    permanent location, since the parse buffer is about to be discarded.
 */
 static XML_Bool
-storeRawNames(XML_Parser parser)
-{
-  TAG *tag = tagStack;
+storeRawNames(XML_Parser parser) {
+  TAG *tag = parser->m_tagStack;
   while (tag) {
     int bufSize;
     int nameLen = sizeof(XML_Char) * (tag->name.strLen + 1);
     char *rawNameBuf = tag->buf + nameLen;
-    /* Stop if already stored.  Since tagStack is a stack, we can stop
+    /* Stop if already stored.  Since m_tagStack is a stack, we can stop
        at the first entry that has already been copied; everything
        below it in the stack is already been accounted for in a
        previous call to this function.
@@ -2550,7 +2411,7 @@ storeRawNames(XML_Parser parser)
     */
     bufSize = nameLen + ROUND_UP(tag->rawNameLength, sizeof(XML_Char));
     if (bufSize > tag->bufEnd - tag->buf) {
-      char *temp = (char *)REALLOC(tag->buf, bufSize);
+      char *temp = (char *)REALLOC(parser, tag->buf, bufSize);
       if (temp == NULL)
         return XML_FALSE;
       /* if tag->name.str points to tag->buf (only when namespace
@@ -2562,8 +2423,8 @@ storeRawNames(XML_Parser parser)
          then update it as well, since it will always point into tag->buf
       */
       if (tag->name.localPart)
-        tag->name.localPart = (XML_Char *)temp + (tag->name.localPart -
-                                                  (XML_Char *)tag->buf);
+        tag->name.localPart
+            = (XML_Char *)temp + (tag->name.localPart - (XML_Char *)tag->buf);
       tag->buf = temp;
       tag->bufEnd = temp + bufSize;
       rawNameBuf = temp + nameLen;
@@ -2576,41 +2437,33 @@ storeRawNames(XML_Parser parser)
 }
 
 static enum XML_Error PTRCALL
-contentProcessor(XML_Parser parser,
-                 const char *start,
-                 const char *end,
-                 const char **endPtr)
-{
-  enum XML_Error result = doContent(parser, 0, encoding, start, end,
-                                    endPtr, (XML_Bool)!ps_finalBuffer);
+contentProcessor(XML_Parser parser, const char *start, const char *end,
+                 const char **endPtr) {
+  enum XML_Error result
+      = doContent(parser, 0, parser->m_encoding, start, end, endPtr,
+                  (XML_Bool)! parser->m_parsingStatus.finalBuffer);
   if (result == XML_ERROR_NONE) {
-    if (!storeRawNames(parser))
+    if (! storeRawNames(parser))
       return XML_ERROR_NO_MEMORY;
   }
   return result;
 }
 
 static enum XML_Error PTRCALL
-externalEntityInitProcessor(XML_Parser parser,
-                            const char *start,
-                            const char *end,
-                            const char **endPtr)
-{
+externalEntityInitProcessor(XML_Parser parser, const char *start,
+                            const char *end, const char **endPtr) {
   enum XML_Error result = initializeEncoding(parser);
   if (result != XML_ERROR_NONE)
     return result;
-  processor = externalEntityInitProcessor2;
+  parser->m_processor = externalEntityInitProcessor2;
   return externalEntityInitProcessor2(parser, start, end, endPtr);
 }
 
 static enum XML_Error PTRCALL
-externalEntityInitProcessor2(XML_Parser parser,
-                             const char *start,
-                             const char *end,
-                             const char **endPtr)
-{
+externalEntityInitProcessor2(XML_Parser parser, const char *start,
+                             const char *end, const char **endPtr) {
   const char *next = start; /* XmlContentTok doesn't always set the last arg */
-  int tok = XmlContentTok(encoding, start, end, &next);
+  int tok = XmlContentTok(parser->m_encoding, start, end, &next);
   switch (tok) {
   case XML_TOK_BOM:
     /* If we are at the end of the buffer, this would cause the next stage,
@@ -2618,115 +2471,102 @@ externalEntityInitProcessor2(XML_Parser parser,
        doContent (by detecting XML_TOK_NONE) without processing any xml text
        declaration - causing the error XML_ERROR_MISPLACED_XML_PI in doContent.
     */
-    if (next == end && !ps_finalBuffer) {
+    if (next == end && ! parser->m_parsingStatus.finalBuffer) {
       *endPtr = next;
       return XML_ERROR_NONE;
     }
     start = next;
     break;
   case XML_TOK_PARTIAL:
-    if (!ps_finalBuffer) {
+    if (! parser->m_parsingStatus.finalBuffer) {
       *endPtr = start;
       return XML_ERROR_NONE;
     }
-    eventPtr = start;
+    parser->m_eventPtr = start;
     return XML_ERROR_UNCLOSED_TOKEN;
   case XML_TOK_PARTIAL_CHAR:
-    if (!ps_finalBuffer) {
+    if (! parser->m_parsingStatus.finalBuffer) {
       *endPtr = start;
       return XML_ERROR_NONE;
     }
-    eventPtr = start;
+    parser->m_eventPtr = start;
     return XML_ERROR_PARTIAL_CHAR;
   }
-  processor = externalEntityInitProcessor3;
+  parser->m_processor = externalEntityInitProcessor3;
   return externalEntityInitProcessor3(parser, start, end, endPtr);
 }
 
 static enum XML_Error PTRCALL
-externalEntityInitProcessor3(XML_Parser parser,
-                             const char *start,
-                             const char *end,
-                             const char **endPtr)
-{
+externalEntityInitProcessor3(XML_Parser parser, const char *start,
+                             const char *end, const char **endPtr) {
   int tok;
   const char *next = start; /* XmlContentTok doesn't always set the last arg */
-  eventPtr = start;
-  tok = XmlContentTok(encoding, start, end, &next);
-  eventEndPtr = next;
+  parser->m_eventPtr = start;
+  tok = XmlContentTok(parser->m_encoding, start, end, &next);
+  parser->m_eventEndPtr = next;
 
   switch (tok) {
-  case XML_TOK_XML_DECL:
-    {
-      enum XML_Error result;
-      result = processXmlDecl(parser, 1, start, next);
-      if (result != XML_ERROR_NONE)
-        return result;
-      switch (ps_parsing) {
-      case XML_SUSPENDED:
-        *endPtr = next;
-        return XML_ERROR_NONE;
-      case XML_FINISHED:
-        return XML_ERROR_ABORTED;
-      default:
-        start = next;
-      }
+  case XML_TOK_XML_DECL: {
+    enum XML_Error result;
+    result = processXmlDecl(parser, 1, start, next);
+    if (result != XML_ERROR_NONE)
+      return result;
+    switch (parser->m_parsingStatus.parsing) {
+    case XML_SUSPENDED:
+      *endPtr = next;
+      return XML_ERROR_NONE;
+    case XML_FINISHED:
+      return XML_ERROR_ABORTED;
+    default:
+      start = next;
     }
-    break;
+  } break;
   case XML_TOK_PARTIAL:
-    if (!ps_finalBuffer) {
+    if (! parser->m_parsingStatus.finalBuffer) {
       *endPtr = start;
       return XML_ERROR_NONE;
     }
     return XML_ERROR_UNCLOSED_TOKEN;
   case XML_TOK_PARTIAL_CHAR:
-    if (!ps_finalBuffer) {
+    if (! parser->m_parsingStatus.finalBuffer) {
       *endPtr = start;
       return XML_ERROR_NONE;
     }
     return XML_ERROR_PARTIAL_CHAR;
   }
-  processor = externalEntityContentProcessor;
-  tagLevel = 1;
+  parser->m_processor = externalEntityContentProcessor;
+  parser->m_tagLevel = 1;
   return externalEntityContentProcessor(parser, start, end, endPtr);
 }
 
 static enum XML_Error PTRCALL
-externalEntityContentProcessor(XML_Parser parser,
-                               const char *start,
-                               const char *end,
-                               const char **endPtr)
-{
-  enum XML_Error result = doContent(parser, 1, encoding, start, end,
-                                    endPtr, (XML_Bool)!ps_finalBuffer);
+externalEntityContentProcessor(XML_Parser parser, const char *start,
+                               const char *end, const char **endPtr) {
+  enum XML_Error result
+      = doContent(parser, 1, parser->m_encoding, start, end, endPtr,
+                  (XML_Bool)! parser->m_parsingStatus.finalBuffer);
   if (result == XML_ERROR_NONE) {
-    if (!storeRawNames(parser))
+    if (! storeRawNames(parser))
       return XML_ERROR_NO_MEMORY;
   }
   return result;
 }
 
 static enum XML_Error
-doContent(XML_Parser parser,
-          int startTagLevel,
-          const ENCODING *enc,
-          const char *s,
-          const char *end,
-          const char **nextPtr,
-          XML_Bool haveMore)
-{
+doContent(XML_Parser parser, int startTagLevel, const ENCODING *enc,
+          const char *s, const char *end, const char **nextPtr,
+          XML_Bool haveMore) {
   /* save one level of indirection */
-  DTD * const dtd = _dtd;
+  DTD *const dtd = parser->m_dtd;
 
   const char **eventPP;
   const char **eventEndPP;
-  if (enc == encoding) {
-    eventPP = &eventPtr;
-    eventEndPP = &eventEndPtr;
-  }
-  else {
-    eventPP = &(openInternalEntities->internalEventPtr);
-    eventEndPP = &(openInternalEntities->internalEventEndPtr);
+  if (enc == parser->m_encoding) {
+    eventPP = &parser->m_eventPtr;
+    eventEndPP = &parser->m_eventEndPtr;
+  } else {
+    eventPP = &(parser->m_openInternalEntities->internalEventPtr);
+    eventEndPP = &(parser->m_openInternalEntities->internalEventEndPtr);
   }
   *eventPP = s;
 
@@ -2741,18 +2581,17 @@ doContent(XML_Parser parser,
         return XML_ERROR_NONE;
       }
       *eventEndPP = end;
-      if (characterDataHandler) {
+      if (parser->m_characterDataHandler) {
         XML_Char c = 0xA;
-        characterDataHandler(handlerArg, &c, 1);
-      }
-      else if (defaultHandler)
+        parser->m_characterDataHandler(parser->m_handlerArg, &c, 1);
+      } else if (parser->m_defaultHandler)
         reportDefault(parser, enc, s, end);
       /* We are at the end of the final buffer, should we check for
          XML_SUSPENDED, XML_FINISHED?
       */
       if (startTagLevel == 0)
         return XML_ERROR_NO_ELEMENTS;
-      if (tagLevel != startTagLevel)
+      if (parser->m_tagLevel != startTagLevel)
         return XML_ERROR_ASYNC_ENTITY;
       *nextPtr = end;
       return XML_ERROR_NONE;
@@ -2762,7 +2601,7 @@ doContent(XML_Parser parser,
         return XML_ERROR_NONE;
       }
       if (startTagLevel > 0) {
-        if (tagLevel != startTagLevel)
+        if (parser->m_tagLevel != startTagLevel)
           return XML_ERROR_ASYNC_ENTITY;
         *nextPtr = s;
         return XML_ERROR_NONE;
@@ -2783,315 +2622,314 @@ doContent(XML_Parser parser,
         return XML_ERROR_NONE;
       }
       return XML_ERROR_PARTIAL_CHAR;
-    case XML_TOK_ENTITY_REF:
-      {
-        const XML_Char *name;
-        ENTITY *entity;
-        XML_Char ch = (XML_Char) XmlPredefinedEntityName(enc,
-                                              s + enc->minBytesPerChar,
-                                              next - enc->minBytesPerChar);
-        if (ch) {
-          if (characterDataHandler)
-            characterDataHandler(handlerArg, &ch, 1);
-          else if (defaultHandler)
+    case XML_TOK_ENTITY_REF: {
+      const XML_Char *name;
+      ENTITY *entity;
+      XML_Char ch = (XML_Char)XmlPredefinedEntityName(
+          enc, s + enc->minBytesPerChar, next - enc->minBytesPerChar);
+      if (ch) {
+        if (parser->m_characterDataHandler)
+          parser->m_characterDataHandler(parser->m_handlerArg, &ch, 1);
+        else if (parser->m_defaultHandler)
+          reportDefault(parser, enc, s, next);
+        break;
+      }
+      name = poolStoreString(&dtd->pool, enc, s + enc->minBytesPerChar,
+                             next - enc->minBytesPerChar);
+      if (! name)
+        return XML_ERROR_NO_MEMORY;
+      entity = (ENTITY *)lookup(parser, &dtd->generalEntities, name, 0);
+      poolDiscard(&dtd->pool);
+      /* First, determine if a check for an existing declaration is needed;
+         if yes, check that the entity exists, and that it is internal,
+         otherwise call the skipped entity or default handler.
+      */
+      if (! dtd->hasParamEntityRefs || dtd->standalone) {
+        if (! entity)
+          return XML_ERROR_UNDEFINED_ENTITY;
+        else if (! entity->is_internal)
+          return XML_ERROR_ENTITY_DECLARED_IN_PE;
+      } else if (! entity) {
+        if (parser->m_skippedEntityHandler)
+          parser->m_skippedEntityHandler(parser->m_handlerArg, name, 0);
+        else if (parser->m_defaultHandler)
+          reportDefault(parser, enc, s, next);
+        break;
+      }
+      if (entity->open)
+        return XML_ERROR_RECURSIVE_ENTITY_REF;
+      if (entity->notation)
+        return XML_ERROR_BINARY_ENTITY_REF;
+      if (entity->textPtr) {
+        enum XML_Error result;
+        if (! parser->m_defaultExpandInternalEntities) {
+          if (parser->m_skippedEntityHandler)
+            parser->m_skippedEntityHandler(parser->m_handlerArg, entity->name,
+                                           0);
+          else if (parser->m_defaultHandler)
             reportDefault(parser, enc, s, next);
           break;
         }
-        name = poolStoreString(&dtd->pool, enc,
-                                s + enc->minBytesPerChar,
-                                next - enc->minBytesPerChar);
-        if (!name)
+        result = processInternalEntity(parser, entity, XML_FALSE);
+        if (result != XML_ERROR_NONE)
+          return result;
+      } else if (parser->m_externalEntityRefHandler) {
+        const XML_Char *context;
+        entity->open = XML_TRUE;
+        context = getContext(parser);
+        entity->open = XML_FALSE;
+        if (! context)
           return XML_ERROR_NO_MEMORY;
-        entity = (ENTITY *)lookup(parser, &dtd->generalEntities, name, 0);
-        poolDiscard(&dtd->pool);
-        /* First, determine if a check for an existing declaration is needed;
-           if yes, check that the entity exists, and that it is internal,
-           otherwise call the skipped entity or default handler.
-        */
-        if (!dtd->hasParamEntityRefs || dtd->standalone) {
-          if (!entity)
-            return XML_ERROR_UNDEFINED_ENTITY;
-          else if (!entity->is_internal)
-            return XML_ERROR_ENTITY_DECLARED_IN_PE;
-        }
-        else if (!entity) {
-          if (skippedEntityHandler)
-            skippedEntityHandler(handlerArg, name, 0);
-          else if (defaultHandler)
-            reportDefault(parser, enc, s, next);
-          break;
-        }
-        if (entity->open)
-          return XML_ERROR_RECURSIVE_ENTITY_REF;
-        if (entity->notation)
-          return XML_ERROR_BINARY_ENTITY_REF;
-        if (entity->textPtr) {
-          enum XML_Error result;
-          if (!defaultExpandInternalEntities) {
-            if (skippedEntityHandler)
-              skippedEntityHandler(handlerArg, entity->name, 0);
-            else if (defaultHandler)
-              reportDefault(parser, enc, s, next);
-            break;
-          }
-          result = processInternalEntity(parser, entity, XML_FALSE);
-          if (result != XML_ERROR_NONE)
-            return result;
-        }
-        else if (externalEntityRefHandler) {
-          const XML_Char *context;
-          entity->open = XML_TRUE;
-          context = getContext(parser);
-          entity->open = XML_FALSE;
-          if (!context)
-            return XML_ERROR_NO_MEMORY;
-          if (!externalEntityRefHandler(externalEntityRefHandlerArg,
-                                        context,
-                                        entity->base,
-                                        entity->systemId,
-                                        entity->publicId))
-            return XML_ERROR_EXTERNAL_ENTITY_HANDLING;
-          poolDiscard(&tempPool);
-        }
-        else if (defaultHandler)
-          reportDefault(parser, enc, s, next);
-        break;
-      }
+        if (! parser->m_externalEntityRefHandler(
+                parser->m_externalEntityRefHandlerArg, context, entity->base,
+                entity->systemId, entity->publicId))
+          return XML_ERROR_EXTERNAL_ENTITY_HANDLING;
+        poolDiscard(&parser->m_tempPool);
+      } else if (parser->m_defaultHandler)
+        reportDefault(parser, enc, s, next);
+      break;
+    }
     case XML_TOK_START_TAG_NO_ATTS:
       /* fall through */
-    case XML_TOK_START_TAG_WITH_ATTS:
-      {
-        TAG *tag;
-        enum XML_Error result;
-        XML_Char *toPtr;
-        if (freeTagList) {
-          tag = freeTagList;
-          freeTagList = freeTagList->parent;
+    case XML_TOK_START_TAG_WITH_ATTS: {
+      TAG *tag;
+      enum XML_Error result;
+      XML_Char *toPtr;
+      if (parser->m_freeTagList) {
+        tag = parser->m_freeTagList;
+        parser->m_freeTagList = parser->m_freeTagList->parent;
+      } else {
+        tag = (TAG *)MALLOC(parser, sizeof(TAG));
+        if (! tag)
+          return XML_ERROR_NO_MEMORY;
+        tag->buf = (char *)MALLOC(parser, INIT_TAG_BUF_SIZE);
+        if (! tag->buf) {
+          FREE(parser, tag);
+          return XML_ERROR_NO_MEMORY;
         }
-        else {
-          tag = (TAG *)MALLOC(sizeof(TAG));
-          if (!tag)
-            return XML_ERROR_NO_MEMORY;
-          tag->buf = (char *)MALLOC(INIT_TAG_BUF_SIZE);
-          if (!tag->buf) {
-            FREE(tag);
-            return XML_ERROR_NO_MEMORY;
+        tag->bufEnd = tag->buf + INIT_TAG_BUF_SIZE;
+      }
+      tag->bindings = NULL;
+      tag->parent = parser->m_tagStack;
+      parser->m_tagStack = tag;
+      tag->name.localPart = NULL;
+      tag->name.prefix = NULL;
+      tag->rawName = s + enc->minBytesPerChar;
+      tag->rawNameLength = XmlNameLength(enc, tag->rawName);
+      ++parser->m_tagLevel;
+      {
+        const char *rawNameEnd = tag->rawName + tag->rawNameLength;
+        const char *fromPtr = tag->rawName;
+        toPtr = (XML_Char *)tag->buf;
+        for (;;) {
+          int bufSize;
+          int convLen;
+          const enum XML_Convert_Result convert_res
+              = XmlConvert(enc, &fromPtr, rawNameEnd, (ICHAR **)&toPtr,
+                           (ICHAR *)tag->bufEnd - 1);
+          convLen = (int)(toPtr - (XML_Char *)tag->buf);
+          if ((fromPtr >= rawNameEnd)
+              || (convert_res == XML_CONVERT_INPUT_INCOMPLETE)) {
+            tag->name.strLen = convLen;
+            break;
           }
-          tag->bufEnd = tag->buf + INIT_TAG_BUF_SIZE;
-        }
-        tag->bindings = NULL;
-        tag->parent = tagStack;
-        tagStack = tag;
-        tag->name.localPart = NULL;
-        tag->name.prefix = NULL;
-        tag->rawName = s + enc->minBytesPerChar;
-        tag->rawNameLength = XmlNameLength(enc, tag->rawName);
-        ++tagLevel;
-        {
-          const char *rawNameEnd = tag->rawName + tag->rawNameLength;
-          const char *fromPtr = tag->rawName;
-          toPtr = (XML_Char *)tag->buf;
-          for (;;) {
-            int bufSize;
-            int convLen;
-            const enum XML_Convert_Result convert_res = XmlConvert(enc,
-                       &fromPtr, rawNameEnd,
-                       (ICHAR **)&toPtr, (ICHAR *)tag->bufEnd - 1);
-            convLen = (int)(toPtr - (XML_Char *)tag->buf);
-            if ((fromPtr >= rawNameEnd) || (convert_res == XML_CONVERT_INPUT_INCOMPLETE)) {
-              tag->name.strLen = convLen;
-              break;
-            }
-            bufSize = (int)(tag->bufEnd - tag->buf) << 1;
-            {
-              char *temp = (char *)REALLOC(tag->buf, bufSize);
-              if (temp == NULL)
-                return XML_ERROR_NO_MEMORY;
-              tag->buf = temp;
-              tag->bufEnd = temp + bufSize;
-              toPtr = (XML_Char *)temp + convLen;
-            }
+          bufSize = (int)(tag->bufEnd - tag->buf) << 1;
+          {
+            char *temp = (char *)REALLOC(parser, tag->buf, bufSize);
+            if (temp == NULL)
+              return XML_ERROR_NO_MEMORY;
+            tag->buf = temp;
+            tag->bufEnd = temp + bufSize;
+            toPtr = (XML_Char *)temp + convLen;
           }
         }
-        tag->name.str = (XML_Char *)tag->buf;
-        *toPtr = XML_T('\0');
-        result = storeAtts(parser, enc, s, &(tag->name), &(tag->bindings));
-        if (result)
-          return result;
-        if (startElementHandler)
-          startElementHandler(handlerArg, tag->name.str,
-                              (const XML_Char **)atts);
-        else if (defaultHandler)
-          reportDefault(parser, enc, s, next);
-        poolClear(&tempPool);
-        break;
       }
+      tag->name.str = (XML_Char *)tag->buf;
+      *toPtr = XML_T('\0');
+      result = storeAtts(parser, enc, s, &(tag->name), &(tag->bindings));
+      if (result)
+        return result;
+      if (parser->m_startElementHandler)
+        parser->m_startElementHandler(parser->m_handlerArg, tag->name.str,
+                                      (const XML_Char **)parser->m_atts);
+      else if (parser->m_defaultHandler)
+        reportDefault(parser, enc, s, next);
+      poolClear(&parser->m_tempPool);
+      break;
+    }
     case XML_TOK_EMPTY_ELEMENT_NO_ATTS:
       /* fall through */
-    case XML_TOK_EMPTY_ELEMENT_WITH_ATTS:
-      {
-        const char *rawName = s + enc->minBytesPerChar;
-        enum XML_Error result;
-        BINDING *bindings = NULL;
-        XML_Bool noElmHandlers = XML_TRUE;
-        TAG_NAME name;
-        name.str = poolStoreString(&tempPool, enc, rawName,
-                                   rawName + XmlNameLength(enc, rawName));
-        if (!name.str)
-          return XML_ERROR_NO_MEMORY;
-        poolFinish(&tempPool);
-        result = storeAtts(parser, enc, s, &name, &bindings);
-        if (result != XML_ERROR_NONE) {
-          freeBindings(parser, bindings);
-          return result;
-        }
-        poolFinish(&tempPool);
-        if (startElementHandler) {
-          startElementHandler(handlerArg, name.str, (const XML_Char **)atts);
-          noElmHandlers = XML_FALSE;
-        }
-        if (endElementHandler) {
-          if (startElementHandler)
-            *eventPP = *eventEndPP;
-          endElementHandler(handlerArg, name.str);
-          noElmHandlers = XML_FALSE;
-        }
-        if (noElmHandlers && defaultHandler)
-          reportDefault(parser, enc, s, next);
-        poolClear(&tempPool);
+    case XML_TOK_EMPTY_ELEMENT_WITH_ATTS: {
+      const char *rawName = s + enc->minBytesPerChar;
+      enum XML_Error result;
+      BINDING *bindings = NULL;
+      XML_Bool noElmHandlers = XML_TRUE;
+      TAG_NAME name;
+      name.str = poolStoreString(&parser->m_tempPool, enc, rawName,
+                                 rawName + XmlNameLength(enc, rawName));
+      if (! name.str)
+        return XML_ERROR_NO_MEMORY;
+      poolFinish(&parser->m_tempPool);
+      result = storeAtts(parser, enc, s, &name, &bindings);
+      if (result != XML_ERROR_NONE) {
         freeBindings(parser, bindings);
+        return result;
+      }
+      poolFinish(&parser->m_tempPool);
+      if (parser->m_startElementHandler) {
+        parser->m_startElementHandler(parser->m_handlerArg, name.str,
+                                      (const XML_Char **)parser->m_atts);
+        noElmHandlers = XML_FALSE;
+      }
+      if (parser->m_endElementHandler) {
+        if (parser->m_startElementHandler)
+          *eventPP = *eventEndPP;
+        parser->m_endElementHandler(parser->m_handlerArg, name.str);
+        noElmHandlers = XML_FALSE;
+      }
+      if (noElmHandlers && parser->m_defaultHandler)
+        reportDefault(parser, enc, s, next);
+      poolClear(&parser->m_tempPool);
+      freeBindings(parser, bindings);
+    }
+      if ((parser->m_tagLevel == 0)
+          && (parser->m_parsingStatus.parsing != XML_FINISHED)) {
+        if (parser->m_parsingStatus.parsing == XML_SUSPENDED)
+          parser->m_processor = epilogProcessor;
+        else
+          return epilogProcessor(parser, next, end, nextPtr);
       }
-      if (tagLevel == 0)
-        return epilogProcessor(parser, next, end, nextPtr);
       break;
     case XML_TOK_END_TAG:
-      if (tagLevel == startTagLevel)
+      if (parser->m_tagLevel == startTagLevel)
         return XML_ERROR_ASYNC_ENTITY;
       else {
         int len;
         const char *rawName;
-        TAG *tag = tagStack;
-        tagStack = tag->parent;
-        tag->parent = freeTagList;
-        freeTagList = tag;
-        rawName = s + enc->minBytesPerChar*2;
+        TAG *tag = parser->m_tagStack;
+        parser->m_tagStack = tag->parent;
+        tag->parent = parser->m_freeTagList;
+        parser->m_freeTagList = tag;
+        rawName = s + enc->minBytesPerChar * 2;
         len = XmlNameLength(enc, rawName);
         if (len != tag->rawNameLength
             || memcmp(tag->rawName, rawName, len) != 0) {
           *eventPP = rawName;
           return XML_ERROR_TAG_MISMATCH;
         }
-        --tagLevel;
-        if (endElementHandler) {
+        --parser->m_tagLevel;
+        if (parser->m_endElementHandler) {
           const XML_Char *localPart;
           const XML_Char *prefix;
           XML_Char *uri;
           localPart = tag->name.localPart;
-          if (ns && localPart) {
+          if (parser->m_ns && localPart) {
             /* localPart and prefix may have been overwritten in
                tag->name.str, since this points to the binding->uri
                buffer which gets re-used; so we have to add them again
             */
             uri = (XML_Char *)tag->name.str + tag->name.uriLen;
             /* don't need to check for space - already done in storeAtts() */
-            while (*localPart) *uri++ = *localPart++;
+            while (*localPart)
+              *uri++ = *localPart++;
             prefix = (XML_Char *)tag->name.prefix;
-            if (ns_triplets && prefix) {
-              *uri++ = namespaceSeparator;
-              while (*prefix) *uri++ = *prefix++;
-             }
+            if (parser->m_ns_triplets && prefix) {
+              *uri++ = parser->m_namespaceSeparator;
+              while (*prefix)
+                *uri++ = *prefix++;
+            }
             *uri = XML_T('\0');
           }
-          endElementHandler(handlerArg, tag->name.str);
-        }
-        else if (defaultHandler)
+          parser->m_endElementHandler(parser->m_handlerArg, tag->name.str);
+        } else if (parser->m_defaultHandler)
           reportDefault(parser, enc, s, next);
         while (tag->bindings) {
           BINDING *b = tag->bindings;
-          if (endNamespaceDeclHandler)
-            endNamespaceDeclHandler(handlerArg, b->prefix->name);
+          if (parser->m_endNamespaceDeclHandler)
+            parser->m_endNamespaceDeclHandler(parser->m_handlerArg,
+                                              b->prefix->name);
           tag->bindings = tag->bindings->nextTagBinding;
-          b->nextTagBinding = freeBindingList;
-          freeBindingList = b;
+          b->nextTagBinding = parser->m_freeBindingList;
+          parser->m_freeBindingList = b;
           b->prefix->binding = b->prevPrefixBinding;
         }
-        if (tagLevel == 0)
-          return epilogProcessor(parser, next, end, nextPtr);
-      }
-      break;
-    case XML_TOK_CHAR_REF:
-      {
-        int n = XmlCharRefNumber(enc, s);
-        if (n < 0)
-          return XML_ERROR_BAD_CHAR_REF;
-        if (characterDataHandler) {
-          XML_Char buf[XML_ENCODE_MAX];
-          characterDataHandler(handlerArg, buf, XmlEncode(n, (ICHAR *)buf));
+        if ((parser->m_tagLevel == 0)
+            && (parser->m_parsingStatus.parsing != XML_FINISHED)) {
+          if (parser->m_parsingStatus.parsing == XML_SUSPENDED)
+            parser->m_processor = epilogProcessor;
+          else
+            return epilogProcessor(parser, next, end, nextPtr);
         }
-        else if (defaultHandler)
-          reportDefault(parser, enc, s, next);
       }
       break;
+    case XML_TOK_CHAR_REF: {
+      int n = XmlCharRefNumber(enc, s);
+      if (n < 0)
+        return XML_ERROR_BAD_CHAR_REF;
+      if (parser->m_characterDataHandler) {
+        XML_Char buf[XML_ENCODE_MAX];
+        parser->m_characterDataHandler(parser->m_handlerArg, buf,
+                                       XmlEncode(n, (ICHAR *)buf));
+      } else if (parser->m_defaultHandler)
+        reportDefault(parser, enc, s, next);
+    } break;
     case XML_TOK_XML_DECL:
       return XML_ERROR_MISPLACED_XML_PI;
     case XML_TOK_DATA_NEWLINE:
-      if (characterDataHandler) {
+      if (parser->m_characterDataHandler) {
         XML_Char c = 0xA;
-        characterDataHandler(handlerArg, &c, 1);
-      }
-      else if (defaultHandler)
+        parser->m_characterDataHandler(parser->m_handlerArg, &c, 1);
+      } else if (parser->m_defaultHandler)
         reportDefault(parser, enc, s, next);
       break;
-    case XML_TOK_CDATA_SECT_OPEN:
-      {
-        enum XML_Error result;
-        if (startCdataSectionHandler)
-          startCdataSectionHandler(handlerArg);
-#if 0
-        /* Suppose you doing a transformation on a document that involves
-           changing only the character data.  You set up a defaultHandler
-           and a characterDataHandler.  The defaultHandler simply copies
-           characters through.  The characterDataHandler does the
-           transformation and writes the characters out escaping them as
-           necessary.  This case will fail to work if we leave out the
-           following two lines (because & and < inside CDATA sections will
-           be incorrectly escaped).
-
-           However, now we have a start/endCdataSectionHandler, so it seems
-           easier to let the user deal with this.
-        */
-        else if (characterDataHandler)
-          characterDataHandler(handlerArg, dataBuf, 0);
-#endif
-        else if (defaultHandler)
-          reportDefault(parser, enc, s, next);
-        result = doCdataSection(parser, enc, &next, end, nextPtr, haveMore);
-        if (result != XML_ERROR_NONE)
-          return result;
-        else if (!next) {
-          processor = cdataSectionProcessor;
-          return result;
-        }
+    case XML_TOK_CDATA_SECT_OPEN: {
+      enum XML_Error result;
+      if (parser->m_startCdataSectionHandler)
+        parser->m_startCdataSectionHandler(parser->m_handlerArg);
+      /* BEGIN disabled code */
+      /* Suppose you doing a transformation on a document that involves
+         changing only the character data.  You set up a defaultHandler
+         and a characterDataHandler.  The defaultHandler simply copies
+         characters through.  The characterDataHandler does the
+         transformation and writes the characters out escaping them as
+         necessary.  This case will fail to work if we leave out the
+         following two lines (because & and < inside CDATA sections will
+         be incorrectly escaped).
+
+         However, now we have a start/endCdataSectionHandler, so it seems
+         easier to let the user deal with this.
+      */
+      else if (0 && parser->m_characterDataHandler)
+        parser->m_characterDataHandler(parser->m_handlerArg, parser->m_dataBuf,
+                                       0);
+      /* END disabled code */
+      else if (parser->m_defaultHandler)
+        reportDefault(parser, enc, s, next);
+      result = doCdataSection(parser, enc, &next, end, nextPtr, haveMore);
+      if (result != XML_ERROR_NONE)
+        return result;
+      else if (! next) {
+        parser->m_processor = cdataSectionProcessor;
+        return result;
       }
-      break;
+    } break;
     case XML_TOK_TRAILING_RSQB:
       if (haveMore) {
         *nextPtr = s;
         return XML_ERROR_NONE;
       }
-      if (characterDataHandler) {
+      if (parser->m_characterDataHandler) {
         if (MUST_CONVERT(enc, s)) {
-          ICHAR *dataPtr = (ICHAR *)dataBuf;
-          XmlConvert(enc, &s, end, &dataPtr, (ICHAR *)dataBufEnd);
-          characterDataHandler(handlerArg, dataBuf,
-                               (int)(dataPtr - (ICHAR *)dataBuf));
-        }
-        else
-          characterDataHandler(handlerArg,
-                               (XML_Char *)s,
-                               (int)((XML_Char *)end - (XML_Char *)s));
-      }
-      else if (defaultHandler)
+          ICHAR *dataPtr = (ICHAR *)parser->m_dataBuf;
+          XmlConvert(enc, &s, end, &dataPtr, (ICHAR *)parser->m_dataBufEnd);
+          parser->m_characterDataHandler(
+              parser->m_handlerArg, parser->m_dataBuf,
+              (int)(dataPtr - (ICHAR *)parser->m_dataBuf));
+        } else
+          parser->m_characterDataHandler(
+              parser->m_handlerArg, (XML_Char *)s,
+              (int)((XML_Char *)end - (XML_Char *)s));
+      } else if (parser->m_defaultHandler)
         reportDefault(parser, enc, s, end);
       /* We are at the end of the final buffer, should we check for
          XML_SUSPENDED, XML_FINISHED?
@@ -3100,43 +2938,40 @@ doContent(XML_Parser parser,
         *eventPP = end;
         return XML_ERROR_NO_ELEMENTS;
       }
-      if (tagLevel != startTagLevel) {
+      if (parser->m_tagLevel != startTagLevel) {
         *eventPP = end;
         return XML_ERROR_ASYNC_ENTITY;
       }
       *nextPtr = end;
       return XML_ERROR_NONE;
-    case XML_TOK_DATA_CHARS:
-      {
-        XML_CharacterDataHandler charDataHandler = characterDataHandler;
-        if (charDataHandler) {
-          if (MUST_CONVERT(enc, s)) {
-            for (;;) {
-              ICHAR *dataPtr = (ICHAR *)dataBuf;
-              const enum XML_Convert_Result convert_res = XmlConvert(enc, &s, next, &dataPtr, (ICHAR *)dataBufEnd);
-              *eventEndPP = s;
-              charDataHandler(handlerArg, dataBuf,
-                              (int)(dataPtr - (ICHAR *)dataBuf));
-              if ((convert_res == XML_CONVERT_COMPLETED) || (convert_res == XML_CONVERT_INPUT_INCOMPLETE))
-                break;
-              *eventPP = s;
-            }
+    case XML_TOK_DATA_CHARS: {
+      XML_CharacterDataHandler charDataHandler = parser->m_characterDataHandler;
+      if (charDataHandler) {
+        if (MUST_CONVERT(enc, s)) {
+          for (;;) {
+            ICHAR *dataPtr = (ICHAR *)parser->m_dataBuf;
+            const enum XML_Convert_Result convert_res = XmlConvert(
+                enc, &s, next, &dataPtr, (ICHAR *)parser->m_dataBufEnd);
+            *eventEndPP = s;
+            charDataHandler(parser->m_handlerArg, parser->m_dataBuf,
+                            (int)(dataPtr - (ICHAR *)parser->m_dataBuf));
+            if ((convert_res == XML_CONVERT_COMPLETED)
+                || (convert_res == XML_CONVERT_INPUT_INCOMPLETE))
+              break;
+            *eventPP = s;
           }
-          else
-            charDataHandler(handlerArg,
-                            (XML_Char *)s,
-                            (int)((XML_Char *)next - (XML_Char *)s));
-        }
-        else if (defaultHandler)
-          reportDefault(parser, enc, s, next);
-      }
-      break;
+        } else
+          charDataHandler(parser->m_handlerArg, (XML_Char *)s,
+                          (int)((XML_Char *)next - (XML_Char *)s));
+      } else if (parser->m_defaultHandler)
+        reportDefault(parser, enc, s, next);
+    } break;
     case XML_TOK_PI:
-      if (!reportProcessingInstruction(parser, enc, s, next))
+      if (! reportProcessingInstruction(parser, enc, s, next))
         return XML_ERROR_NO_MEMORY;
       break;
     case XML_TOK_COMMENT:
-      if (!reportComment(parser, enc, s, next))
+      if (! reportComment(parser, enc, s, next))
         return XML_ERROR_NO_MEMORY;
       break;
     default:
@@ -3147,43 +2982,42 @@ doContent(XML_Parser parser,
        *
        * LCOV_EXCL_START
        */
-      if (defaultHandler)
+      if (parser->m_defaultHandler)
         reportDefault(parser, enc, s, next);
       break;
       /* LCOV_EXCL_STOP */
     }
     *eventPP = s = next;
-    switch (ps_parsing) {
+    switch (parser->m_parsingStatus.parsing) {
     case XML_SUSPENDED:
       *nextPtr = next;
       return XML_ERROR_NONE;
     case XML_FINISHED:
       return XML_ERROR_ABORTED;
-    default: ;
+    default:;
     }
   }
   /* not reached */
 }
 
 /* This function does not call free() on the allocated memory, merely
- * moving it to the parser's freeBindingList where it can be freed or
+ * moving it to the parser's m_freeBindingList where it can be freed or
  * reused as appropriate.
  */
 static void
-freeBindings(XML_Parser parser, BINDING *bindings)
-{
+freeBindings(XML_Parser parser, BINDING *bindings) {
   while (bindings) {
     BINDING *b = bindings;
 
-    /* startNamespaceDeclHandler will have been called for this
+    /* m_startNamespaceDeclHandler will have been called for this
      * binding in addBindings(), so call the end handler now.
      */
-    if (endNamespaceDeclHandler)
-        endNamespaceDeclHandler(handlerArg, b->prefix->name);
+    if (parser->m_endNamespaceDeclHandler)
+      parser->m_endNamespaceDeclHandler(parser->m_handlerArg, b->prefix->name);
 
     bindings = bindings->nextTagBinding;
-    b->nextTagBinding = freeBindingList;
-    freeBindingList = b;
+    b->nextTagBinding = parser->m_freeBindingList;
+    parser->m_freeBindingList = b;
     b->prefix->binding = b->prevPrefixBinding;
   }
 }
@@ -3199,14 +3033,12 @@ freeBindings(XML_Parser parser, BINDING *bindings)
    - generate namespace aware element name (URI, prefix)
 */
 static enum XML_Error
-storeAtts(XML_Parser parser, const ENCODING *enc,
-          const char *attStr, TAG_NAME *tagNamePtr,
-          BINDING **bindingsPtr)
-{
-  DTD * const dtd = _dtd;  /* save one level of indirection */
+storeAtts(XML_Parser parser, const ENCODING *enc, const char *attStr,
+          TAG_NAME *tagNamePtr, BINDING **bindingsPtr) {
+  DTD *const dtd = parser->m_dtd; /* save one level of indirection */
   ELEMENT_TYPE *elementType;
   int nDefaultAtts;
-  const XML_Char **appAtts;   /* the attribute list for the application */
+  const XML_Char **appAtts; /* the attribute list for the application */
   int attIndex = 0;
   int prefixLen;
   int i;
@@ -3217,79 +3049,84 @@ storeAtts(XML_Parser parser, const ENCODING *enc,
   const XML_Char *localPart;
 
   /* lookup the element type name */
-  elementType = (ELEMENT_TYPE *)lookup(parser, &dtd->elementTypes, tagNamePtr->str,0);
-  if (!elementType) {
+  elementType
+      = (ELEMENT_TYPE *)lookup(parser, &dtd->elementTypes, tagNamePtr->str, 0);
+  if (! elementType) {
     const XML_Char *name = poolCopyString(&dtd->pool, tagNamePtr->str);
-    if (!name)
+    if (! name)
       return XML_ERROR_NO_MEMORY;
     elementType = (ELEMENT_TYPE *)lookup(parser, &dtd->elementTypes, name,
                                          sizeof(ELEMENT_TYPE));
-    if (!elementType)
+    if (! elementType)
       return XML_ERROR_NO_MEMORY;
-    if (ns && !setElementTypePrefix(parser, elementType))
+    if (parser->m_ns && ! setElementTypePrefix(parser, elementType))
       return XML_ERROR_NO_MEMORY;
   }
   nDefaultAtts = elementType->nDefaultAtts;
 
   /* get the attributes from the tokenizer */
-  n = XmlGetAttributes(enc, attStr, attsSize, atts);
-  if (n + nDefaultAtts > attsSize) {
-    int oldAttsSize = attsSize;
+  n = XmlGetAttributes(enc, attStr, parser->m_attsSize, parser->m_atts);
+  if (n + nDefaultAtts > parser->m_attsSize) {
+    int oldAttsSize = parser->m_attsSize;
     ATTRIBUTE *temp;
 #ifdef XML_ATTR_INFO
     XML_AttrInfo *temp2;
 #endif
-    attsSize = n + nDefaultAtts + INIT_ATTS_SIZE;
-    temp = (ATTRIBUTE *)REALLOC((void *)atts, attsSize * sizeof(ATTRIBUTE));
+    parser->m_attsSize = n + nDefaultAtts + INIT_ATTS_SIZE;
+    temp = (ATTRIBUTE *)REALLOC(parser, (void *)parser->m_atts,
+                                parser->m_attsSize * sizeof(ATTRIBUTE));
     if (temp == NULL) {
-      attsSize = oldAttsSize;
+      parser->m_attsSize = oldAttsSize;
       return XML_ERROR_NO_MEMORY;
     }
-    atts = temp;
+    parser->m_atts = temp;
 #ifdef XML_ATTR_INFO
-    temp2 = (XML_AttrInfo *)REALLOC((void *)attInfo, attsSize * sizeof(XML_AttrInfo));
+    temp2 = (XML_AttrInfo *)REALLOC(parser, (void *)parser->m_attInfo,
+                                    parser->m_attsSize * sizeof(XML_AttrInfo));
     if (temp2 == NULL) {
-      attsSize = oldAttsSize;
+      parser->m_attsSize = oldAttsSize;
       return XML_ERROR_NO_MEMORY;
     }
-    attInfo = temp2;
+    parser->m_attInfo = temp2;
 #endif
     if (n > oldAttsSize)
-      XmlGetAttributes(enc, attStr, n, atts);
+      XmlGetAttributes(enc, attStr, n, parser->m_atts);
   }
 
-  appAtts = (const XML_Char **)atts;
+  appAtts = (const XML_Char **)parser->m_atts;
   for (i = 0; i < n; i++) {
-    ATTRIBUTE *currAtt = &atts[i];
+    ATTRIBUTE *currAtt = &parser->m_atts[i];
 #ifdef XML_ATTR_INFO
-    XML_AttrInfo *currAttInfo = &attInfo[i];
+    XML_AttrInfo *currAttInfo = &parser->m_attInfo[i];
 #endif
     /* add the name and value to the attribute list */
-    ATTRIBUTE_ID *attId = getAttributeId(parser, enc, currAtt->name,
-                                         currAtt->name
-                                         + XmlNameLength(enc, currAtt->name));
-    if (!attId)
+    ATTRIBUTE_ID *attId
+        = getAttributeId(parser, enc, currAtt->name,
+                         currAtt->name + XmlNameLength(enc, currAtt->name));
+    if (! attId)
       return XML_ERROR_NO_MEMORY;
 #ifdef XML_ATTR_INFO
-    currAttInfo->nameStart = parseEndByteIndex - (parseEndPtr - currAtt->name);
-    currAttInfo->nameEnd = currAttInfo->nameStart +
-                           XmlNameLength(enc, currAtt->name);
-    currAttInfo->valueStart = parseEndByteIndex -
-                            (parseEndPtr - currAtt->valuePtr);
-    currAttInfo->valueEnd = parseEndByteIndex - (parseEndPtr - currAtt->valueEnd);
+    currAttInfo->nameStart
+        = parser->m_parseEndByteIndex - (parser->m_parseEndPtr - currAtt->name);
+    currAttInfo->nameEnd
+        = currAttInfo->nameStart + XmlNameLength(enc, currAtt->name);
+    currAttInfo->valueStart = parser->m_parseEndByteIndex
+                              - (parser->m_parseEndPtr - currAtt->valuePtr);
+    currAttInfo->valueEnd = parser->m_parseEndByteIndex
+                            - (parser->m_parseEndPtr - currAtt->valueEnd);
 #endif
     /* Detect duplicate attributes by their QNames. This does not work when
        namespace processing is turned on and different prefixes for the same
        namespace are used. For this case we have a check further down.
     */
     if ((attId->name)[-1]) {
-      if (enc == encoding)
-        eventPtr = atts[i].name;
+      if (enc == parser->m_encoding)
+        parser->m_eventPtr = parser->m_atts[i].name;
       return XML_ERROR_DUPLICATE_ATTRIBUTE;
     }
     (attId->name)[-1] = 1;
     appAtts[attIndex++] = attId->name;
-    if (!atts[i].normalized) {
+    if (! parser->m_atts[i].normalized) {
       enum XML_Error result;
       XML_Bool isCdata = XML_TRUE;
 
@@ -3305,21 +3142,21 @@ storeAtts(XML_Parser parser, const ENCODING *enc,
       }
 
       /* normalize the attribute value */
-      result = storeAttributeValue(parser, enc, isCdata,
-                                   atts[i].valuePtr, atts[i].valueEnd,
-                                   &tempPool);
+      result = storeAttributeValue(
+          parser, enc, isCdata, parser->m_atts[i].valuePtr,
+          parser->m_atts[i].valueEnd, &parser->m_tempPool);
       if (result)
         return result;
-      appAtts[attIndex] = poolStart(&tempPool);
-      poolFinish(&tempPool);
-    }
-    else {
+      appAtts[attIndex] = poolStart(&parser->m_tempPool);
+      poolFinish(&parser->m_tempPool);
+    } else {
       /* the value did not need normalizing */
-      appAtts[attIndex] = poolStoreString(&tempPool, enc, atts[i].valuePtr,
-                                          atts[i].valueEnd);
+      appAtts[attIndex] = poolStoreString(&parser->m_tempPool, enc,
+                                          parser->m_atts[i].valuePtr,
+                                          parser->m_atts[i].valueEnd);
       if (appAtts[attIndex] == 0)
         return XML_ERROR_NO_MEMORY;
-      poolFinish(&tempPool);
+      poolFinish(&parser->m_tempPool);
     }
     /* handle prefixed attribute names */
     if (attId->prefix) {
@@ -3330,49 +3167,44 @@ storeAtts(XML_Parser parser, const ENCODING *enc,
         if (result)
           return result;
         --attIndex;
-      }
-      else {
+      } else {
         /* deal with other prefixed names later */
         attIndex++;
         nPrefixes++;
         (attId->name)[-1] = 2;
       }
-    }
-    else
+    } else
       attIndex++;
   }
 
   /* set-up for XML_GetSpecifiedAttributeCount and XML_GetIdAttributeIndex */
-  nSpecifiedAtts = attIndex;
+  parser->m_nSpecifiedAtts = attIndex;
   if (elementType->idAtt && (elementType->idAtt->name)[-1]) {
     for (i = 0; i < attIndex; i += 2)
       if (appAtts[i] == elementType->idAtt->name) {
-        idAttIndex = i;
+        parser->m_idAttIndex = i;
         break;
       }
-  }
-  else
-    idAttIndex = -1;
+  } else
+    parser->m_idAttIndex = -1;
 
   /* do attribute defaulting */
   for (i = 0; i < nDefaultAtts; i++) {
     const DEFAULT_ATTRIBUTE *da = elementType->defaultAtts + i;
-    if (!(da->id->name)[-1] && da->value) {
+    if (! (da->id->name)[-1] && da->value) {
       if (da->id->prefix) {
         if (da->id->xmlns) {
           enum XML_Error result = addBinding(parser, da->id->prefix, da->id,
                                              da->value, bindingsPtr);
           if (result)
             return result;
-        }
-        else {
+        } else {
           (da->id->name)[-1] = 2;
           nPrefixes++;
           appAtts[attIndex++] = da->id->name;
           appAtts[attIndex++] = da->value;
         }
-      }
-      else {
+      } else {
         (da->id->name)[-1] = 1;
         appAtts[attIndex++] = da->id->name;
         appAtts[attIndex++] = da->value;
@@ -3385,39 +3217,42 @@ storeAtts(XML_Parser parser, const ENCODING *enc,
      and clear flags that say whether attributes were specified */
   i = 0;
   if (nPrefixes) {
-    int j;  /* hash table index */
-    unsigned long version = nsAttsVersion;
-    int nsAttsSize = (int)1 << nsAttsPower;
-    unsigned char oldNsAttsPower = nsAttsPower;
+    int j; /* hash table index */
+    unsigned long version = parser->m_nsAttsVersion;
+    int nsAttsSize = (int)1 << parser->m_nsAttsPower;
+    unsigned char oldNsAttsPower = parser->m_nsAttsPower;
     /* size of hash table must be at least 2 * (# of prefixed attributes) */
-    if ((nPrefixes << 1) >> nsAttsPower) {  /* true for nsAttsPower = 0 */
+    if ((nPrefixes << 1)
+        >> parser->m_nsAttsPower) { /* true for m_nsAttsPower = 0 */
       NS_ATT *temp;
       /* hash table size must also be a power of 2 and >= 8 */
-      while (nPrefixes >> nsAttsPower++);
-      if (nsAttsPower < 3)
-        nsAttsPower = 3;
-      nsAttsSize = (int)1 << nsAttsPower;
-      temp = (NS_ATT *)REALLOC(nsAtts, nsAttsSize * sizeof(NS_ATT));
-      if (!temp) {
-        /* Restore actual size of memory in nsAtts */
-        nsAttsPower = oldNsAttsPower;
+      while (nPrefixes >> parser->m_nsAttsPower++)
+        ;
+      if (parser->m_nsAttsPower < 3)
+        parser->m_nsAttsPower = 3;
+      nsAttsSize = (int)1 << parser->m_nsAttsPower;
+      temp = (NS_ATT *)REALLOC(parser, parser->m_nsAtts,
+                               nsAttsSize * sizeof(NS_ATT));
+      if (! temp) {
+        /* Restore actual size of memory in m_nsAtts */
+        parser->m_nsAttsPower = oldNsAttsPower;
         return XML_ERROR_NO_MEMORY;
       }
-      nsAtts = temp;
-      version = 0;  /* force re-initialization of nsAtts hash table */
+      parser->m_nsAtts = temp;
+      version = 0; /* force re-initialization of m_nsAtts hash table */
     }
-    /* using a version flag saves us from initializing nsAtts every time */
-    if (!version) {  /* initialize version flags when version wraps around */
+    /* using a version flag saves us from initializing m_nsAtts every time */
+    if (! version) { /* initialize version flags when version wraps around */
       version = INIT_ATTS_VERSION;
-      for (j = nsAttsSize; j != 0; )
-        nsAtts[--j].version = version;
+      for (j = nsAttsSize; j != 0;)
+        parser->m_nsAtts[--j].version = version;
     }
-    nsAttsVersion = --version;
+    parser->m_nsAttsVersion = --version;
 
     /* expand prefixed names and check for duplicates */
     for (; i < attIndex; i += 2) {
       const XML_Char *s = appAtts[i];
-      if (s[-1] == 2) {  /* prefixed */
+      if (s[-1] == 2) { /* prefixed */
         ATTRIBUTE_ID *id;
         const BINDING *b;
         unsigned long uriHash;
@@ -3427,9 +3262,9 @@ storeAtts(XML_Parser parser, const ENCODING *enc,
         copy_salt_to_sipkey(parser, &sip_key);
         sip24_init(&sip_state, &sip_key);
 
-        ((XML_Char *)s)[-1] = 0;  /* clear flag */
+        ((XML_Char *)s)[-1] = 0; /* clear flag */
         id = (ATTRIBUTE_ID *)lookup(parser, &dtd->attributeIds, s, 0);
-        if (!id || !id->prefix) {
+        if (! id || ! id->prefix) {
           /* This code is walking through the appAtts array, dealing
            * with (in this case) a prefixed attribute name.  To be in
            * the array, the attribute must have already been bound, so
@@ -3447,12 +3282,12 @@ storeAtts(XML_Parser parser, const ENCODING *enc,
           return XML_ERROR_NO_MEMORY; /* LCOV_EXCL_LINE */
         }
         b = id->prefix->binding;
-        if (!b)
+        if (! b)
           return XML_ERROR_UNBOUND_PREFIX;
 
         for (j = 0; j < b->uriLen; j++) {
           const XML_Char c = b->uri[j];
-          if (!poolAppendChar(&tempPool, c))
+          if (! poolAppendChar(&parser->m_tempPool, c))
             return XML_ERROR_NO_MEMORY;
         }
 
@@ -3463,8 +3298,8 @@ storeAtts(XML_Parser parser, const ENCODING *enc,
 
         sip24_update(&sip_state, s, keylen(s) * sizeof(XML_Char));
 
-        do {  /* copies null terminator */
-          if (!poolAppendChar(&tempPool, *s))
+        do { /* copies null terminator */
+          if (! poolAppendChar(&parser->m_tempPool, *s))
             return XML_ERROR_NO_MEMORY;
         } while (*s++);
 
@@ -3475,49 +3310,49 @@ storeAtts(XML_Parser parser, const ENCODING *enc,
           */
           unsigned char step = 0;
           unsigned long mask = nsAttsSize - 1;
-          j = uriHash & mask;  /* index into hash table */
-          while (nsAtts[j].version == version) {
+          j = uriHash & mask; /* index into hash table */
+          while (parser->m_nsAtts[j].version == version) {
             /* for speed we compare stored hash values first */
-            if (uriHash == nsAtts[j].hash) {
-              const XML_Char *s1 = poolStart(&tempPool);
-              const XML_Char *s2 = nsAtts[j].uriName;
+            if (uriHash == parser->m_nsAtts[j].hash) {
+              const XML_Char *s1 = poolStart(&parser->m_tempPool);
+              const XML_Char *s2 = parser->m_nsAtts[j].uriName;
               /* s1 is null terminated, but not s2 */
-              for (; *s1 == *s2 && *s1 != 0; s1++, s2++);
+              for (; *s1 == *s2 && *s1 != 0; s1++, s2++)
+                ;
               if (*s1 == 0)
                 return XML_ERROR_DUPLICATE_ATTRIBUTE;
             }
-            if (!step)
-              step = PROBE_STEP(uriHash, mask, nsAttsPower);
+            if (! step)
+              step = PROBE_STEP(uriHash, mask, parser->m_nsAttsPower);
             j < step ? (j += nsAttsSize - step) : (j -= step);
           }
         }
 
-        if (ns_triplets) {  /* append namespace separator and prefix */
-          tempPool.ptr[-1] = namespaceSeparator;
+        if (parser->m_ns_triplets) { /* append namespace separator and prefix */
+          parser->m_tempPool.ptr[-1] = parser->m_namespaceSeparator;
           s = b->prefix->name;
           do {
-            if (!poolAppendChar(&tempPool, *s))
+            if (! poolAppendChar(&parser->m_tempPool, *s))
               return XML_ERROR_NO_MEMORY;
           } while (*s++);
         }
 
         /* store expanded name in attribute list */
-        s = poolStart(&tempPool);
-        poolFinish(&tempPool);
+        s = poolStart(&parser->m_tempPool);
+        poolFinish(&parser->m_tempPool);
         appAtts[i] = s;
 
         /* fill empty slot with new version, uriName and hash value */
-        nsAtts[j].version = version;
-        nsAtts[j].hash = uriHash;
-        nsAtts[j].uriName = s;
+        parser->m_nsAtts[j].version = version;
+        parser->m_nsAtts[j].hash = uriHash;
+        parser->m_nsAtts[j].uriName = s;
 
-        if (!--nPrefixes) {
+        if (! --nPrefixes) {
           i += 2;
           break;
         }
-      }
-      else  /* not prefixed */
-        ((XML_Char *)s)[-1] = 0;  /* clear flag */
+      } else                     /* not prefixed */
+        ((XML_Char *)s)[-1] = 0; /* clear flag */
     }
   }
   /* clear flags for the remaining attributes */
@@ -3526,56 +3361,54 @@ storeAtts(XML_Parser parser, const ENCODING *enc,
   for (binding = *bindingsPtr; binding; binding = binding->nextTagBinding)
     binding->attId->name[-1] = 0;
 
-  if (!ns)
+  if (! parser->m_ns)
     return XML_ERROR_NONE;
 
   /* expand the element type name */
   if (elementType->prefix) {
     binding = elementType->prefix->binding;
-    if (!binding)
+    if (! binding)
       return XML_ERROR_UNBOUND_PREFIX;
     localPart = tagNamePtr->str;
     while (*localPart++ != XML_T(ASCII_COLON))
       ;
-  }
-  else if (dtd->defaultPrefix.binding) {
+  } else if (dtd->defaultPrefix.binding) {
     binding = dtd->defaultPrefix.binding;
     localPart = tagNamePtr->str;
-  }
-  else
+  } else
     return XML_ERROR_NONE;
   prefixLen = 0;
-  if (ns_triplets && binding->prefix->name) {
+  if (parser->m_ns_triplets && binding->prefix->name) {
     for (; binding->prefix->name[prefixLen++];)
-      ;  /* prefixLen includes null terminator */
+      ; /* prefixLen includes null terminator */
   }
   tagNamePtr->localPart = localPart;
   tagNamePtr->uriLen = binding->uriLen;
   tagNamePtr->prefix = binding->prefix->name;
   tagNamePtr->prefixLen = prefixLen;
   for (i = 0; localPart[i++];)
-    ;  /* i includes null terminator */
+    ; /* i includes null terminator */
   n = i + binding->uriLen + prefixLen;
   if (n > binding->uriAlloc) {
     TAG *p;
-    uri = (XML_Char *)MALLOC((n + EXPAND_SPARE) * sizeof(XML_Char));
-    if (!uri)
+    uri = (XML_Char *)MALLOC(parser, (n + EXPAND_SPARE) * sizeof(XML_Char));
+    if (! uri)
       return XML_ERROR_NO_MEMORY;
     binding->uriAlloc = n + EXPAND_SPARE;
     memcpy(uri, binding->uri, binding->uriLen * sizeof(XML_Char));
-    for (p = tagStack; p; p = p->parent)
+    for (p = parser->m_tagStack; p; p = p->parent)
       if (p->name.str == binding->uri)
         p->name.str = uri;
-    FREE(binding->uri);
+    FREE(parser, binding->uri);
     binding->uri = uri;
   }
-  /* if namespaceSeparator != '\0' then uri includes it already */
+  /* if m_namespaceSeparator != '\0' then uri includes it already */
   uri = binding->uri + binding->uriLen;
   memcpy(uri, localPart, i * sizeof(XML_Char));
   /* we always have a namespace separator between localPart and prefix */
   if (prefixLen) {
     uri += i - 1;
-    *uri = namespaceSeparator;  /* replace null terminator */
+    *uri = parser->m_namespaceSeparator; /* replace null terminator */
     memcpy(uri + 1, binding->prefix->name, prefixLen * sizeof(XML_Char));
   }
   tagNamePtr->str = binding->uri;
@@ -3587,27 +3420,25 @@ storeAtts(XML_Parser parser, const ENCODING *enc,
 */
 static enum XML_Error
 addBinding(XML_Parser parser, PREFIX *prefix, const ATTRIBUTE_ID *attId,
-           const XML_Char *uri, BINDING **bindingsPtr)
-{
-  static const XML_Char xmlNamespace[] = {
-    ASCII_h, ASCII_t, ASCII_t, ASCII_p, ASCII_COLON, ASCII_SLASH, ASCII_SLASH,
-    ASCII_w, ASCII_w, ASCII_w, ASCII_PERIOD, ASCII_w, ASCII_3, ASCII_PERIOD,
-    ASCII_o, ASCII_r, ASCII_g, ASCII_SLASH, ASCII_X, ASCII_M, ASCII_L,
-    ASCII_SLASH, ASCII_1, ASCII_9, ASCII_9, ASCII_8, ASCII_SLASH,
-    ASCII_n, ASCII_a, ASCII_m, ASCII_e, ASCII_s, ASCII_p, ASCII_a, ASCII_c,
-    ASCII_e, '\0'
-  };
-  static const int xmlLen =
-    (int)sizeof(xmlNamespace)/sizeof(XML_Char) - 1;
-  static const XML_Char xmlnsNamespace[] = {
-    ASCII_h, ASCII_t, ASCII_t, ASCII_p, ASCII_COLON, ASCII_SLASH, ASCII_SLASH,
-    ASCII_w, ASCII_w, ASCII_w, ASCII_PERIOD, ASCII_w, ASCII_3, ASCII_PERIOD,
-    ASCII_o, ASCII_r, ASCII_g, ASCII_SLASH, ASCII_2, ASCII_0, ASCII_0,
-    ASCII_0, ASCII_SLASH, ASCII_x, ASCII_m, ASCII_l, ASCII_n, ASCII_s,
-    ASCII_SLASH, '\0'
-  };
-  static const int xmlnsLen =
-    (int)sizeof(xmlnsNamespace)/sizeof(XML_Char) - 1;
+           const XML_Char *uri, BINDING **bindingsPtr) {
+  static const XML_Char xmlNamespace[]
+      = {ASCII_h,      ASCII_t,     ASCII_t,     ASCII_p,      ASCII_COLON,
+         ASCII_SLASH,  ASCII_SLASH, ASCII_w,     ASCII_w,      ASCII_w,
+         ASCII_PERIOD, ASCII_w,     ASCII_3,     ASCII_PERIOD, ASCII_o,
+         ASCII_r,      ASCII_g,     ASCII_SLASH, ASCII_X,      ASCII_M,
+         ASCII_L,      ASCII_SLASH, ASCII_1,     ASCII_9,      ASCII_9,
+         ASCII_8,      ASCII_SLASH, ASCII_n,     ASCII_a,      ASCII_m,
+         ASCII_e,      ASCII_s,     ASCII_p,     ASCII_a,      ASCII_c,
+         ASCII_e,      '\0'};
+  static const int xmlLen = (int)sizeof(xmlNamespace) / sizeof(XML_Char) - 1;
+  static const XML_Char xmlnsNamespace[]
+      = {ASCII_h,     ASCII_t,      ASCII_t, ASCII_p, ASCII_COLON,  ASCII_SLASH,
+         ASCII_SLASH, ASCII_w,      ASCII_w, ASCII_w, ASCII_PERIOD, ASCII_w,
+         ASCII_3,     ASCII_PERIOD, ASCII_o, ASCII_r, ASCII_g,      ASCII_SLASH,
+         ASCII_2,     ASCII_0,      ASCII_0, ASCII_0, ASCII_SLASH,  ASCII_x,
+         ASCII_m,     ASCII_l,      ASCII_n, ASCII_s, ASCII_SLASH,  '\0'};
+  static const int xmlnsLen
+      = (int)sizeof(xmlnsNamespace) / sizeof(XML_Char) - 1;
 
   XML_Bool mustBeXML = XML_FALSE;
   XML_Bool isXML = XML_TRUE;
@@ -3620,14 +3451,11 @@ addBinding(XML_Parser parser, PREFIX *prefix, const ATTRIBUTE_ID *attId,
   if (*uri == XML_T('\0') && prefix->name)
     return XML_ERROR_UNDECLARING_PREFIX;
 
-  if (prefix->name
-      && prefix->name[0] == XML_T(ASCII_x)
+  if (prefix->name && prefix->name[0] == XML_T(ASCII_x)
       && prefix->name[1] == XML_T(ASCII_m)
       && prefix->name[2] == XML_T(ASCII_l)) {
-
     /* Not allowed to bind xmlns */
-    if (prefix->name[3] == XML_T(ASCII_n)
-        && prefix->name[4] == XML_T(ASCII_s)
+    if (prefix->name[3] == XML_T(ASCII_n) && prefix->name[4] == XML_T(ASCII_s)
         && prefix->name[5] == XML_T('\0'))
       return XML_ERROR_RESERVED_PREFIX_XMLNS;
 
@@ -3639,7 +3467,7 @@ addBinding(XML_Parser parser, PREFIX *prefix, const ATTRIBUTE_ID *attId,
     if (isXML && (len > xmlLen || uri[len] != xmlNamespace[len]))
       isXML = XML_FALSE;
 
-    if (!mustBeXML && isXMLNS
+    if (! mustBeXML && isXMLNS
         && (len > xmlnsLen || uri[len] != xmlnsNamespace[len]))
       isXMLNS = XML_FALSE;
   }
@@ -3653,49 +3481,49 @@ addBinding(XML_Parser parser, PREFIX *prefix, const ATTRIBUTE_ID *attId,
   if (isXMLNS)
     return XML_ERROR_RESERVED_NAMESPACE_URI;
 
-  if (namespaceSeparator)
+  if (parser->m_namespaceSeparator)
     len++;
-  if (freeBindingList) {
-    b = freeBindingList;
+  if (parser->m_freeBindingList) {
+    b = parser->m_freeBindingList;
     if (len > b->uriAlloc) {
-      XML_Char *temp = (XML_Char *)REALLOC(b->uri,
-                          sizeof(XML_Char) * (len + EXPAND_SPARE));
+      XML_Char *temp = (XML_Char *)REALLOC(
+          parser, b->uri, sizeof(XML_Char) * (len + EXPAND_SPARE));
       if (temp == NULL)
         return XML_ERROR_NO_MEMORY;
       b->uri = temp;
       b->uriAlloc = len + EXPAND_SPARE;
     }
-    freeBindingList = b->nextTagBinding;
-  }
-  else {
-    b = (BINDING *)MALLOC(sizeof(BINDING));
-    if (!b)
+    parser->m_freeBindingList = b->nextTagBinding;
+  } else {
+    b = (BINDING *)MALLOC(parser, sizeof(BINDING));
+    if (! b)
       return XML_ERROR_NO_MEMORY;
-    b->uri = (XML_Char *)MALLOC(sizeof(XML_Char) * (len + EXPAND_SPARE));
-    if (!b->uri) {
-      FREE(b);
+    b->uri
+        = (XML_Char *)MALLOC(parser, sizeof(XML_Char) * (len + EXPAND_SPARE));
+    if (! b->uri) {
+      FREE(parser, b);
       return XML_ERROR_NO_MEMORY;
     }
     b->uriAlloc = len + EXPAND_SPARE;
   }
   b->uriLen = len;
   memcpy(b->uri, uri, len * sizeof(XML_Char));
-  if (namespaceSeparator)
-    b->uri[len - 1] = namespaceSeparator;
+  if (parser->m_namespaceSeparator)
+    b->uri[len - 1] = parser->m_namespaceSeparator;
   b->prefix = prefix;
   b->attId = attId;
   b->prevPrefixBinding = prefix->binding;
   /* NULL binding when default namespace undeclared */
-  if (*uri == XML_T('\0') && prefix == &_dtd->defaultPrefix)
+  if (*uri == XML_T('\0') && prefix == &parser->m_dtd->defaultPrefix)
     prefix->binding = NULL;
   else
     prefix->binding = b;
   b->nextTagBinding = *bindingsPtr;
   *bindingsPtr = b;
   /* if attId == NULL then we are not starting a namespace scope */
-  if (attId && startNamespaceDeclHandler)
-    startNamespaceDeclHandler(handlerArg, prefix->name,
-                              prefix->binding ? uri : 0);
+  if (attId && parser->m_startNamespaceDeclHandler)
+    parser->m_startNamespaceDeclHandler(parser->m_handlerArg, prefix->name,
+                                        prefix->binding ? uri : 0);
   return XML_ERROR_NONE;
 }
 
@@ -3703,22 +3531,19 @@ addBinding(XML_Parser parser, PREFIX *prefix, const ATTRIBUTE_ID *attId,
    the whole file is parsed with one call.
 */
 static enum XML_Error PTRCALL
-cdataSectionProcessor(XML_Parser parser,
-                      const char *start,
-                      const char *end,
-                      const char **endPtr)
-{
-  enum XML_Error result = doCdataSection(parser, encoding, &start, end,
-                                         endPtr, (XML_Bool)!ps_finalBuffer);
+cdataSectionProcessor(XML_Parser parser, const char *start, const char *end,
+                      const char **endPtr) {
+  enum XML_Error result
+      = doCdataSection(parser, parser->m_encoding, &start, end, endPtr,
+                       (XML_Bool)! parser->m_parsingStatus.finalBuffer);
   if (result != XML_ERROR_NONE)
     return result;
   if (start) {
-    if (parentParser) {  /* we are parsing an external entity */
-      processor = externalEntityContentProcessor;
+    if (parser->m_parentParser) { /* we are parsing an external entity */
+      parser->m_processor = externalEntityContentProcessor;
       return externalEntityContentProcessor(parser, start, end, endPtr);
-    }
-    else {
-      processor = contentProcessor;
+    } else {
+      parser->m_processor = contentProcessor;
       return contentProcessor(parser, start, end, endPtr);
     }
   }
@@ -3729,24 +3554,18 @@ cdataSectionProcessor(XML_Parser parser,
    the section is not yet closed.
 */
 static enum XML_Error
-doCdataSection(XML_Parser parser,
-               const ENCODING *enc,
-               const char **startPtr,
-               const char *end,
-               const char **nextPtr,
-               XML_Bool haveMore)
-{
+doCdataSection(XML_Parser parser, const ENCODING *enc, const char **startPtr,
+               const char *end, const char **nextPtr, XML_Bool haveMore) {
   const char *s = *startPtr;
   const char **eventPP;
   const char **eventEndPP;
-  if (enc == encoding) {
-    eventPP = &eventPtr;
+  if (enc == parser->m_encoding) {
+    eventPP = &parser->m_eventPtr;
     *eventPP = s;
-    eventEndPP = &eventEndPtr;
-  }
-  else {
-    eventPP = &(openInternalEntities->internalEventPtr);
-    eventEndPP = &(openInternalEntities->internalEventEndPtr);
+    eventEndPP = &parser->m_eventEndPtr;
+  } else {
+    eventPP = &(parser->m_openInternalEntities->internalEventPtr);
+    eventEndPP = &(parser->m_openInternalEntities->internalEventEndPtr);
   }
   *eventPP = s;
   *startPtr = NULL;
@@ -3757,54 +3576,51 @@ doCdataSection(XML_Parser parser,
     *eventEndPP = next;
     switch (tok) {
     case XML_TOK_CDATA_SECT_CLOSE:
-      if (endCdataSectionHandler)
-        endCdataSectionHandler(handlerArg);
-#if 0
+      if (parser->m_endCdataSectionHandler)
+        parser->m_endCdataSectionHandler(parser->m_handlerArg);
+      /* BEGIN disabled code */
       /* see comment under XML_TOK_CDATA_SECT_OPEN */
-      else if (characterDataHandler)
-        characterDataHandler(handlerArg, dataBuf, 0);
-#endif
-      else if (defaultHandler)
+      else if (0 && parser->m_characterDataHandler)
+        parser->m_characterDataHandler(parser->m_handlerArg, parser->m_dataBuf,
+                                       0);
+      /* END disabled code */
+      else if (parser->m_defaultHandler)
         reportDefault(parser, enc, s, next);
       *startPtr = next;
       *nextPtr = next;
-      if (ps_parsing == XML_FINISHED)
+      if (parser->m_parsingStatus.parsing == XML_FINISHED)
         return XML_ERROR_ABORTED;
-      else
-        return XML_ERROR_NONE;
-    case XML_TOK_DATA_NEWLINE:
-      if (characterDataHandler) {
-        XML_Char c = 0xA;
-        characterDataHandler(handlerArg, &c, 1);
-      }
-      else if (defaultHandler)
-        reportDefault(parser, enc, s, next);
-      break;
-    case XML_TOK_DATA_CHARS:
-      {
-        XML_CharacterDataHandler charDataHandler = characterDataHandler;
-        if (charDataHandler) {
-          if (MUST_CONVERT(enc, s)) {
-            for (;;) {
-              ICHAR *dataPtr = (ICHAR *)dataBuf;
-              const enum XML_Convert_Result convert_res = XmlConvert(enc, &s, next, &dataPtr, (ICHAR *)dataBufEnd);
-              *eventEndPP = next;
-              charDataHandler(handlerArg, dataBuf,
-                              (int)(dataPtr - (ICHAR *)dataBuf));
-              if ((convert_res == XML_CONVERT_COMPLETED) || (convert_res == XML_CONVERT_INPUT_INCOMPLETE))
-                break;
-              *eventPP = s;
-            }
-          }
-          else
-            charDataHandler(handlerArg,
-                            (XML_Char *)s,
-                            (int)((XML_Char *)next - (XML_Char *)s));
-        }
-        else if (defaultHandler)
-          reportDefault(parser, enc, s, next);
-      }
+      else
+        return XML_ERROR_NONE;
+    case XML_TOK_DATA_NEWLINE:
+      if (parser->m_characterDataHandler) {
+        XML_Char c = 0xA;
+        parser->m_characterDataHandler(parser->m_handlerArg, &c, 1);
+      } else if (parser->m_defaultHandler)
+        reportDefault(parser, enc, s, next);
       break;
+    case XML_TOK_DATA_CHARS: {
+      XML_CharacterDataHandler charDataHandler = parser->m_characterDataHandler;
+      if (charDataHandler) {
+        if (MUST_CONVERT(enc, s)) {
+          for (;;) {
+            ICHAR *dataPtr = (ICHAR *)parser->m_dataBuf;
+            const enum XML_Convert_Result convert_res = XmlConvert(
+                enc, &s, next, &dataPtr, (ICHAR *)parser->m_dataBufEnd);
+            *eventEndPP = next;
+            charDataHandler(parser->m_handlerArg, parser->m_dataBuf,
+                            (int)(dataPtr - (ICHAR *)parser->m_dataBuf));
+            if ((convert_res == XML_CONVERT_COMPLETED)
+                || (convert_res == XML_CONVERT_INPUT_INCOMPLETE))
+              break;
+            *eventPP = s;
+          }
+        } else
+          charDataHandler(parser->m_handlerArg, (XML_Char *)s,
+                          (int)((XML_Char *)next - (XML_Char *)s));
+      } else if (parser->m_defaultHandler)
+        reportDefault(parser, enc, s, next);
+    } break;
     case XML_TOK_INVALID:
       *eventPP = next;
       return XML_ERROR_INVALID_TOKEN;
@@ -3828,20 +3644,20 @@ doCdataSection(XML_Parser parser,
        * statistics.
        *
        * LCOV_EXCL_START
-      */
+       */
       *eventPP = next;
       return XML_ERROR_UNEXPECTED_STATE;
       /* LCOV_EXCL_STOP */
     }
 
     *eventPP = s = next;
-    switch (ps_parsing) {
+    switch (parser->m_parsingStatus.parsing) {
     case XML_SUSPENDED:
       *nextPtr = next;
       return XML_ERROR_NONE;
     case XML_FINISHED:
       return XML_ERROR_ABORTED;
-    default: ;
+    default:;
     }
   }
   /* not reached */
@@ -3853,17 +3669,15 @@ doCdataSection(XML_Parser parser,
    the whole file is parsed with one call.
 */
 static enum XML_Error PTRCALL
-ignoreSectionProcessor(XML_Parser parser,
-                       const char *start,
-                       const char *end,
-                       const char **endPtr)
-{
-  enum XML_Error result = doIgnoreSection(parser, encoding, &start, end,
-                                          endPtr, (XML_Bool)!ps_finalBuffer);
+ignoreSectionProcessor(XML_Parser parser, const char *start, const char *end,
+                       const char **endPtr) {
+  enum XML_Error result
+      = doIgnoreSection(parser, parser->m_encoding, &start, end, endPtr,
+                        (XML_Bool)! parser->m_parsingStatus.finalBuffer);
   if (result != XML_ERROR_NONE)
     return result;
   if (start) {
-    processor = prologProcessor;
+    parser->m_processor = prologProcessor;
     return prologProcessor(parser, start, end, endPtr);
   }
   return result;
@@ -3873,27 +3687,21 @@ ignoreSectionProcessor(XML_Parser parser,
    if the section is not yet closed.
 */
 static enum XML_Error
-doIgnoreSection(XML_Parser parser,
-                const ENCODING *enc,
-                const char **startPtr,
-                const char *end,
-                const char **nextPtr,
-                XML_Bool haveMore)
-{
+doIgnoreSection(XML_Parser parser, const ENCODING *enc, const char **startPtr,
+                const char *end, const char **nextPtr, XML_Bool haveMore) {
   const char *next;
   int tok;
   const char *s = *startPtr;
   const char **eventPP;
   const char **eventEndPP;
-  if (enc == encoding) {
-    eventPP = &eventPtr;
+  if (enc == parser->m_encoding) {
+    eventPP = &parser->m_eventPtr;
     *eventPP = s;
-    eventEndPP = &eventEndPtr;
-  }
-  else {
+    eventEndPP = &parser->m_eventEndPtr;
+  } else {
     /* It's not entirely clear, but it seems the following two lines
      * of code cannot be executed.  The only occasions on which 'enc'
-     * is not 'parser->m_encoding' are when this function is called
+     * is not 'encoding' are when this function is called
      * from the internal entity processing, and IGNORE sections are an
      * error in internal entities.
      *
@@ -3902,8 +3710,8 @@ doIgnoreSection(XML_Parser parser,
      *
      * LCOV_EXCL_START
      */
-    eventPP = &(openInternalEntities->internalEventPtr);
-    eventEndPP = &(openInternalEntities->internalEventEndPtr);
+    eventPP = &(parser->m_openInternalEntities->internalEventPtr);
+    eventEndPP = &(parser->m_openInternalEntities->internalEventEndPtr);
     /* LCOV_EXCL_STOP */
   }
   *eventPP = s;
@@ -3912,11 +3720,11 @@ doIgnoreSection(XML_Parser parser,
   *eventEndPP = next;
   switch (tok) {
   case XML_TOK_IGNORE_SECT:
-    if (defaultHandler)
+    if (parser->m_defaultHandler)
       reportDefault(parser, enc, s, next);
     *startPtr = next;
     *nextPtr = next;
-    if (ps_parsing == XML_FINISHED)
+    if (parser->m_parsingStatus.parsing == XML_FINISHED)
       return XML_ERROR_ABORTED;
     else
       return XML_ERROR_NONE;
@@ -3954,39 +3762,38 @@ doIgnoreSection(XML_Parser parser,
 #endif /* XML_DTD */
 
 static enum XML_Error
-initializeEncoding(XML_Parser parser)
-{
+initializeEncoding(XML_Parser parser) {
   const char *s;
 #ifdef XML_UNICODE
   char encodingBuf[128];
   /* See comments abount `protoclEncodingName` in parserInit() */
-  if (!protocolEncodingName)
+  if (! parser->m_protocolEncodingName)
     s = NULL;
   else {
     int i;
-    for (i = 0; protocolEncodingName[i]; i++) {
+    for (i = 0; parser->m_protocolEncodingName[i]; i++) {
       if (i == sizeof(encodingBuf) - 1
-          || (protocolEncodingName[i] & ~0x7f) != 0) {
+          || (parser->m_protocolEncodingName[i] & ~0x7f) != 0) {
         encodingBuf[0] = '\0';
         break;
       }
-      encodingBuf[i] = (char)protocolEncodingName[i];
+      encodingBuf[i] = (char)parser->m_protocolEncodingName[i];
     }
     encodingBuf[i] = '\0';
     s = encodingBuf;
   }
 #else
-  s = protocolEncodingName;
+  s = parser->m_protocolEncodingName;
 #endif
-  if ((ns ? XmlInitEncodingNS : XmlInitEncoding)(&initEncoding, &encoding, s))
+  if ((parser->m_ns ? XmlInitEncodingNS : XmlInitEncoding)(
+          &parser->m_initEncoding, &parser->m_encoding, s))
     return XML_ERROR_NONE;
-  return handleUnknownEncoding(parser, protocolEncodingName);
+  return handleUnknownEncoding(parser, parser->m_protocolEncodingName);
 }
 
 static enum XML_Error
-processXmlDecl(XML_Parser parser, int isGeneralTextEntity,
-               const char *s, const char *next)
-{
+processXmlDecl(XML_Parser parser, int isGeneralTextEntity, const char *s,
+               const char *next) {
   const char *encodingName = NULL;
   const XML_Char *storedEncName = NULL;
   const ENCODING *newEncoding = NULL;
@@ -3994,95 +3801,82 @@ processXmlDecl(XML_Parser parser, int isGeneralTextEntity,
   const char *versionend;
   const XML_Char *storedversion = NULL;
   int standalone = -1;
-  if (!(ns
-        ? XmlParseXmlDeclNS
-        : XmlParseXmlDecl)(isGeneralTextEntity,
-                           encoding,
-                           s,
-                           next,
-                           &eventPtr,
-                           &version,
-                           &versionend,
-                           &encodingName,
-                           &newEncoding,
-                           &standalone)) {
+  if (! (parser->m_ns ? XmlParseXmlDeclNS : XmlParseXmlDecl)(
+          isGeneralTextEntity, parser->m_encoding, s, next, &parser->m_eventPtr,
+          &version, &versionend, &encodingName, &newEncoding, &standalone)) {
     if (isGeneralTextEntity)
       return XML_ERROR_TEXT_DECL;
     else
       return XML_ERROR_XML_DECL;
   }
-  if (!isGeneralTextEntity && standalone == 1) {
-    _dtd->standalone = XML_TRUE;
+  if (! isGeneralTextEntity && standalone == 1) {
+    parser->m_dtd->standalone = XML_TRUE;
 #ifdef XML_DTD
-    if (paramEntityParsing == XML_PARAM_ENTITY_PARSING_UNLESS_STANDALONE)
-      paramEntityParsing = XML_PARAM_ENTITY_PARSING_NEVER;
+    if (parser->m_paramEntityParsing
+        == XML_PARAM_ENTITY_PARSING_UNLESS_STANDALONE)
+      parser->m_paramEntityParsing = XML_PARAM_ENTITY_PARSING_NEVER;
 #endif /* XML_DTD */
   }
-  if (xmlDeclHandler) {
+  if (parser->m_xmlDeclHandler) {
     if (encodingName != NULL) {
-      storedEncName = poolStoreString(&temp2Pool,
-                                      encoding,
-                                      encodingName,
-                                      encodingName
-                                      + XmlNameLength(encoding, encodingName));
-      if (!storedEncName)
-              return XML_ERROR_NO_MEMORY;
-      poolFinish(&temp2Pool);
+      storedEncName = poolStoreString(
+          &parser->m_temp2Pool, parser->m_encoding, encodingName,
+          encodingName + XmlNameLength(parser->m_encoding, encodingName));
+      if (! storedEncName)
+        return XML_ERROR_NO_MEMORY;
+      poolFinish(&parser->m_temp2Pool);
     }
     if (version) {
-      storedversion = poolStoreString(&temp2Pool,
-                                      encoding,
-                                      version,
-                                      versionend - encoding->minBytesPerChar);
-      if (!storedversion)
+      storedversion
+          = poolStoreString(&parser->m_temp2Pool, parser->m_encoding, version,
+                            versionend - parser->m_encoding->minBytesPerChar);
+      if (! storedversion)
         return XML_ERROR_NO_MEMORY;
     }
-    xmlDeclHandler(handlerArg, storedversion, storedEncName, standalone);
-  }
-  else if (defaultHandler)
-    reportDefault(parser, encoding, s, next);
-  if (protocolEncodingName == NULL) {
+    parser->m_xmlDeclHandler(parser->m_handlerArg, storedversion, storedEncName,
+                             standalone);
+  } else if (parser->m_defaultHandler)
+    reportDefault(parser, parser->m_encoding, s, next);
+  if (parser->m_protocolEncodingName == NULL) {
     if (newEncoding) {
       /* Check that the specified encoding does not conflict with what
        * the parser has already deduced.  Do we have the same number
        * of bytes in the smallest representation of a character?  If
        * this is UTF-16, is it the same endianness?
        */
-      if (newEncoding->minBytesPerChar != encoding->minBytesPerChar
-          || (newEncoding->minBytesPerChar == 2 &&
-              newEncoding != encoding)) {
-        eventPtr = encodingName;
+      if (newEncoding->minBytesPerChar != parser->m_encoding->minBytesPerChar
+          || (newEncoding->minBytesPerChar == 2
+              && newEncoding != parser->m_encoding)) {
+        parser->m_eventPtr = encodingName;
         return XML_ERROR_INCORRECT_ENCODING;
       }
-      encoding = newEncoding;
-    }
-    else if (encodingName) {
+      parser->m_encoding = newEncoding;
+    } else if (encodingName) {
       enum XML_Error result;
-      if (!storedEncName) {
+      if (! storedEncName) {
         storedEncName = poolStoreString(
-          &temp2Pool, encoding, encodingName,
-          encodingName + XmlNameLength(encoding, encodingName));
-        if (!storedEncName)
+            &parser->m_temp2Pool, parser->m_encoding, encodingName,
+            encodingName + XmlNameLength(parser->m_encoding, encodingName));
+        if (! storedEncName)
           return XML_ERROR_NO_MEMORY;
       }
       result = handleUnknownEncoding(parser, storedEncName);
-      poolClear(&temp2Pool);
+      poolClear(&parser->m_temp2Pool);
       if (result == XML_ERROR_UNKNOWN_ENCODING)
-        eventPtr = encodingName;
+        parser->m_eventPtr = encodingName;
       return result;
     }
   }
 
   if (storedEncName || storedversion)
-    poolClear(&temp2Pool);
+    poolClear(&parser->m_temp2Pool);
 
   return XML_ERROR_NONE;
 }
 
 static enum XML_Error
-handleUnknownEncoding(XML_Parser parser, const XML_Char *encodingName)
-{
-  if (unknownEncodingHandler) {
+handleUnknownEncoding(XML_Parser parser, const XML_Char *encodingName) {
+  if (parser->m_unknownEncodingHandler) {
     XML_Encoding info;
     int i;
     for (i = 0; i < 256; i++)
@@ -4090,25 +3884,21 @@ handleUnknownEncoding(XML_Parser parser, const XML_Char *encodingName)
     info.convert = NULL;
     info.data = NULL;
     info.release = NULL;
-    if (unknownEncodingHandler(unknownEncodingHandlerData, encodingName,
-                               &info)) {
+    if (parser->m_unknownEncodingHandler(parser->m_unknownEncodingHandlerData,
+                                         encodingName, &info)) {
       ENCODING *enc;
-      unknownEncodingMem = MALLOC(XmlSizeOfUnknownEncoding());
-      if (!unknownEncodingMem) {
+      parser->m_unknownEncodingMem = MALLOC(parser, XmlSizeOfUnknownEncoding());
+      if (! parser->m_unknownEncodingMem) {
         if (info.release)
           info.release(info.data);
         return XML_ERROR_NO_MEMORY;
       }
-      enc = (ns
-             ? XmlInitUnknownEncodingNS
-             : XmlInitUnknownEncoding)(unknownEncodingMem,
-                                       info.map,
-                                       info.convert,
-                                       info.data);
+      enc = (parser->m_ns ? XmlInitUnknownEncodingNS : XmlInitUnknownEncoding)(
+          parser->m_unknownEncodingMem, info.map, info.convert, info.data);
       if (enc) {
-        unknownEncodingData = info.data;
-        unknownEncodingRelease = info.release;
-        encoding = enc;
+        parser->m_unknownEncodingData = info.data;
+        parser->m_unknownEncodingRelease = info.release;
+        parser->m_encoding = enc;
         return XML_ERROR_NONE;
       }
     }
@@ -4119,60 +3909,50 @@ handleUnknownEncoding(XML_Parser parser, const XML_Char *encodingName)
 }
 
 static enum XML_Error PTRCALL
-prologInitProcessor(XML_Parser parser,
-                    const char *s,
-                    const char *end,
-                    const char **nextPtr)
-{
+prologInitProcessor(XML_Parser parser, const char *s, const char *end,
+                    const char **nextPtr) {
   enum XML_Error result = initializeEncoding(parser);
   if (result != XML_ERROR_NONE)
     return result;
-  processor = prologProcessor;
+  parser->m_processor = prologProcessor;
   return prologProcessor(parser, s, end, nextPtr);
 }
 
 #ifdef XML_DTD
 
 static enum XML_Error PTRCALL
-externalParEntInitProcessor(XML_Parser parser,
-                            const char *s,
-                            const char *end,
-                            const char **nextPtr)
-{
+externalParEntInitProcessor(XML_Parser parser, const char *s, const char *end,
+                            const char **nextPtr) {
   enum XML_Error result = initializeEncoding(parser);
   if (result != XML_ERROR_NONE)
     return result;
 
   /* we know now that XML_Parse(Buffer) has been called,
      so we consider the external parameter entity read */
-  _dtd->paramEntityRead = XML_TRUE;
+  parser->m_dtd->paramEntityRead = XML_TRUE;
 
-  if (prologState.inEntityValue) {
-    processor = entityValueInitProcessor;
+  if (parser->m_prologState.inEntityValue) {
+    parser->m_processor = entityValueInitProcessor;
     return entityValueInitProcessor(parser, s, end, nextPtr);
-  }
-  else {
-    processor = externalParEntProcessor;
+  } else {
+    parser->m_processor = externalParEntProcessor;
     return externalParEntProcessor(parser, s, end, nextPtr);
   }
 }
 
 static enum XML_Error PTRCALL
-entityValueInitProcessor(XML_Parser parser,
-                         const char *s,
-                         const char *end,
-                         const char **nextPtr)
-{
+entityValueInitProcessor(XML_Parser parser, const char *s, const char *end,
+                         const char **nextPtr) {
   int tok;
   const char *start = s;
   const char *next = start;
-  eventPtr = start;
+  parser->m_eventPtr = start;
 
   for (;;) {
-    tok = XmlPrologTok(encoding, start, end, &next);
-    eventEndPtr = next;
+    tok = XmlPrologTok(parser->m_encoding, start, end, &next);
+    parser->m_eventEndPtr = next;
     if (tok <= 0) {
-      if (!ps_finalBuffer && tok != XML_TOK_INVALID) {
+      if (! parser->m_parsingStatus.finalBuffer && tok != XML_TOK_INVALID) {
         *nextPtr = s;
         return XML_ERROR_NONE;
       }
@@ -4183,28 +3963,27 @@ entityValueInitProcessor(XML_Parser parser,
         return XML_ERROR_UNCLOSED_TOKEN;
       case XML_TOK_PARTIAL_CHAR:
         return XML_ERROR_PARTIAL_CHAR;
-      case XML_TOK_NONE:   /* start == end */
+      case XML_TOK_NONE: /* start == end */
       default:
         break;
       }
       /* found end of entity value - can store it now */
-      return storeEntityValue(parser, encoding, s, end);
-    }
-    else if (tok == XML_TOK_XML_DECL) {
+      return storeEntityValue(parser, parser->m_encoding, s, end);
+    } else if (tok == XML_TOK_XML_DECL) {
       enum XML_Error result;
       result = processXmlDecl(parser, 0, start, next);
       if (result != XML_ERROR_NONE)
         return result;
-      /* At this point, ps_parsing cannot be XML_SUSPENDED.  For that
-       * to happen, a parameter entity parsing handler must have
-       * attempted to suspend the parser, which fails and raises an
-       * error.  The parser can be aborted, but can't be suspended.
+      /* At this point, m_parsingStatus.parsing cannot be XML_SUSPENDED.  For
+       * that to happen, a parameter entity parsing handler must have attempted
+       * to suspend the parser, which fails and raises an error.  The parser can
+       * be aborted, but can't be suspended.
        */
-      if (ps_parsing == XML_FINISHED)
+      if (parser->m_parsingStatus.parsing == XML_FINISHED)
         return XML_ERROR_ABORTED;
       *nextPtr = next;
       /* stop scanning for text declaration - we found one */
-      processor = entityValueProcessor;
+      parser->m_processor = entityValueProcessor;
       return entityValueProcessor(parser, next, end, nextPtr);
     }
     /* If we are at the end of the buffer, this would cause XmlPrologTok to
@@ -4214,7 +3993,8 @@ entityValueInitProcessor(XML_Parser parser,
        then, when this routine is entered the next time, XmlPrologTok will
        return XML_TOK_INVALID, since the BOM is still in the buffer
     */
-    else if (tok == XML_TOK_BOM && next == end && !ps_finalBuffer) {
+    else if (tok == XML_TOK_BOM && next == end
+             && ! parser->m_parsingStatus.finalBuffer) {
       *nextPtr = next;
       return XML_ERROR_NONE;
     }
@@ -4227,22 +4007,19 @@ entityValueInitProcessor(XML_Parser parser,
       return XML_ERROR_SYNTAX;
     }
     start = next;
-    eventPtr = start;
+    parser->m_eventPtr = start;
   }
 }
 
 static enum XML_Error PTRCALL
-externalParEntProcessor(XML_Parser parser,
-                        const char *s,
-                        const char *end,
-                        const char **nextPtr)
-{
+externalParEntProcessor(XML_Parser parser, const char *s, const char *end,
+                        const char **nextPtr) {
   const char *next = s;
   int tok;
 
-  tok = XmlPrologTok(encoding, s, end, &next);
+  tok = XmlPrologTok(parser->m_encoding, s, end, &next);
   if (tok <= 0) {
-    if (!ps_finalBuffer && tok != XML_TOK_INVALID) {
+    if (! parser->m_parsingStatus.finalBuffer && tok != XML_TOK_INVALID) {
       *nextPtr = s;
       return XML_ERROR_NONE;
     }
@@ -4253,7 +4030,7 @@ externalParEntProcessor(XML_Parser parser,
       return XML_ERROR_UNCLOSED_TOKEN;
     case XML_TOK_PARTIAL_CHAR:
       return XML_ERROR_PARTIAL_CHAR;
-    case XML_TOK_NONE:   /* start == end */
+    case XML_TOK_NONE: /* start == end */
     default:
       break;
     }
@@ -4264,29 +4041,26 @@ externalParEntProcessor(XML_Parser parser,
   */
   else if (tok == XML_TOK_BOM) {
     s = next;
-    tok = XmlPrologTok(encoding, s, end, &next);
+    tok = XmlPrologTok(parser->m_encoding, s, end, &next);
   }
 
-  processor = prologProcessor;
-  return doProlog(parser, encoding, s, end, tok, next,
-                  nextPtr, (XML_Bool)!ps_finalBuffer);
+  parser->m_processor = prologProcessor;
+  return doProlog(parser, parser->m_encoding, s, end, tok, next, nextPtr,
+                  (XML_Bool)! parser->m_parsingStatus.finalBuffer, XML_TRUE);
 }
 
 static enum XML_Error PTRCALL
-entityValueProcessor(XML_Parser parser,
-                     const char *s,
-                     const char *end,
-                     const char **nextPtr)
-{
+entityValueProcessor(XML_Parser parser, const char *s, const char *end,
+                     const char **nextPtr) {
   const char *start = s;
   const char *next = s;
-  const ENCODING *enc = encoding;
+  const ENCODING *enc = parser->m_encoding;
   int tok;
 
   for (;;) {
     tok = XmlPrologTok(enc, start, end, &next);
     if (tok <= 0) {
-      if (!ps_finalBuffer && tok != XML_TOK_INVALID) {
+      if (! parser->m_parsingStatus.finalBuffer && tok != XML_TOK_INVALID) {
         *nextPtr = s;
         return XML_ERROR_NONE;
       }
@@ -4297,7 +4071,7 @@ entityValueProcessor(XML_Parser parser,
         return XML_ERROR_UNCLOSED_TOKEN;
       case XML_TOK_PARTIAL_CHAR:
         return XML_ERROR_PARTIAL_CHAR;
-      case XML_TOK_NONE:   /* start == end */
+      case XML_TOK_NONE: /* start == end */
       default:
         break;
       }
@@ -4311,64 +4085,57 @@ entityValueProcessor(XML_Parser parser,
 #endif /* XML_DTD */
 
 static enum XML_Error PTRCALL
-prologProcessor(XML_Parser parser,
-                const char *s,
-                const char *end,
-                const char **nextPtr)
-{
+prologProcessor(XML_Parser parser, const char *s, const char *end,
+                const char **nextPtr) {
   const char *next = s;
-  int tok = XmlPrologTok(encoding, s, end, &next);
-  return doProlog(parser, encoding, s, end, tok, next,
-                  nextPtr, (XML_Bool)!ps_finalBuffer);
+  int tok = XmlPrologTok(parser->m_encoding, s, end, &next);
+  return doProlog(parser, parser->m_encoding, s, end, tok, next, nextPtr,
+                  (XML_Bool)! parser->m_parsingStatus.finalBuffer, XML_TRUE);
 }
 
 static enum XML_Error
-doProlog(XML_Parser parser,
-         const ENCODING *enc,
-         const char *s,
-         const char *end,
-         int tok,
-         const char *next,
-         const char **nextPtr,
-         XML_Bool haveMore)
-{
+doProlog(XML_Parser parser, const ENCODING *enc, const char *s, const char *end,
+         int tok, const char *next, const char **nextPtr, XML_Bool haveMore,
+         XML_Bool allowClosingDoctype) {
 #ifdef XML_DTD
-  static const XML_Char externalSubsetName[] = { ASCII_HASH , '\0' };
+  static const XML_Char externalSubsetName[] = {ASCII_HASH, '\0'};
 #endif /* XML_DTD */
-  static const XML_Char atypeCDATA[] =
-      { ASCII_C, ASCII_D, ASCII_A, ASCII_T, ASCII_A, '\0' };
-  static const XML_Char atypeID[] = { ASCII_I, ASCII_D, '\0' };
-  static const XML_Char atypeIDREF[] =
-      { ASCII_I, ASCII_D, ASCII_R, ASCII_E, ASCII_F, '\0' };
-  static const XML_Char atypeIDREFS[] =
-      { ASCII_I, ASCII_D, ASCII_R, ASCII_E, ASCII_F, ASCII_S, '\0' };
-  static const XML_Char atypeENTITY[] =
-      { ASCII_E, ASCII_N, ASCII_T, ASCII_I, ASCII_T, ASCII_Y, '\0' };
-  static const XML_Char atypeENTITIES[] = { ASCII_E, ASCII_N,
-      ASCII_T, ASCII_I, ASCII_T, ASCII_I, ASCII_E, ASCII_S, '\0' };
-  static const XML_Char atypeNMTOKEN[] = {
-      ASCII_N, ASCII_M, ASCII_T, ASCII_O, ASCII_K, ASCII_E, ASCII_N, '\0' };
-  static const XML_Char atypeNMTOKENS[] = { ASCII_N, ASCII_M, ASCII_T,
-      ASCII_O, ASCII_K, ASCII_E, ASCII_N, ASCII_S, '\0' };
-  static const XML_Char notationPrefix[] = { ASCII_N, ASCII_O, ASCII_T,
-      ASCII_A, ASCII_T, ASCII_I, ASCII_O, ASCII_N, ASCII_LPAREN, '\0' };
-  static const XML_Char enumValueSep[] = { ASCII_PIPE, '\0' };
-  static const XML_Char enumValueStart[] = { ASCII_LPAREN, '\0' };
+  static const XML_Char atypeCDATA[]
+      = {ASCII_C, ASCII_D, ASCII_A, ASCII_T, ASCII_A, '\0'};
+  static const XML_Char atypeID[] = {ASCII_I, ASCII_D, '\0'};
+  static const XML_Char atypeIDREF[]
+      = {ASCII_I, ASCII_D, ASCII_R, ASCII_E, ASCII_F, '\0'};
+  static const XML_Char atypeIDREFS[]
+      = {ASCII_I, ASCII_D, ASCII_R, ASCII_E, ASCII_F, ASCII_S, '\0'};
+  static const XML_Char atypeENTITY[]
+      = {ASCII_E, ASCII_N, ASCII_T, ASCII_I, ASCII_T, ASCII_Y, '\0'};
+  static const XML_Char atypeENTITIES[]
+      = {ASCII_E, ASCII_N, ASCII_T, ASCII_I, ASCII_T,
+         ASCII_I, ASCII_E, ASCII_S, '\0'};
+  static const XML_Char atypeNMTOKEN[]
+      = {ASCII_N, ASCII_M, ASCII_T, ASCII_O, ASCII_K, ASCII_E, ASCII_N, '\0'};
+  static const XML_Char atypeNMTOKENS[]
+      = {ASCII_N, ASCII_M, ASCII_T, ASCII_O, ASCII_K,
+         ASCII_E, ASCII_N, ASCII_S, '\0'};
+  static const XML_Char notationPrefix[]
+      = {ASCII_N, ASCII_O, ASCII_T, ASCII_A,      ASCII_T,
+         ASCII_I, ASCII_O, ASCII_N, ASCII_LPAREN, '\0'};
+  static const XML_Char enumValueSep[] = {ASCII_PIPE, '\0'};
+  static const XML_Char enumValueStart[] = {ASCII_LPAREN, '\0'};
 
   /* save one level of indirection */
-  DTD * const dtd = _dtd;
+  DTD *const dtd = parser->m_dtd;
 
   const char **eventPP;
   const char **eventEndPP;
   enum XML_Content_Quant quant;
 
-  if (enc == encoding) {
-    eventPP = &eventPtr;
-    eventEndPP = &eventEndPtr;
-  }
-  else {
-    eventPP = &(openInternalEntities->internalEventPtr);
-    eventEndPP = &(openInternalEntities->internalEventEndPtr);
+  if (enc == parser->m_encoding) {
+    eventPP = &parser->m_eventPtr;
+    eventEndPP = &parser->m_eventEndPtr;
+  } else {
+    eventPP = &(parser->m_openInternalEntities->internalEventPtr);
+    eventEndPP = &(parser->m_openInternalEntities->internalEventEndPtr);
   }
 
   for (;;) {
@@ -4395,7 +4162,8 @@ doProlog(XML_Parser parser,
       case XML_TOK_NONE:
 #ifdef XML_DTD
         /* for internal PE NOT referenced between declarations */
-        if (enc != encoding && !openInternalEntities->betweenDecl) {
+        if (enc != parser->m_encoding
+            && ! parser->m_openInternalEntities->betweenDecl) {
           *nextPtr = s;
           return XML_ERROR_NONE;
         }
@@ -4403,8 +4171,8 @@ doProlog(XML_Parser parser,
            complete markup, not only for external PEs, but also for
            internal PEs if the reference occurs between declarations.
         */
-        if (isParamEntity || enc != encoding) {
-          if (XmlTokenRole(&prologState, XML_TOK_NONE, end, end, enc)
+        if (parser->m_isParamEntity || enc != parser->m_encoding) {
+          if (XmlTokenRole(&parser->m_prologState, XML_TOK_NONE, end, end, enc)
               == XML_ROLE_ERROR)
             return XML_ERROR_INCOMPLETE_PE;
           *nextPtr = s;
@@ -4418,114 +4186,117 @@ doProlog(XML_Parser parser,
         break;
       }
     }
-    role = XmlTokenRole(&prologState, tok, s, next, enc);
+    role = XmlTokenRole(&parser->m_prologState, tok, s, next, enc);
     switch (role) {
-    case XML_ROLE_XML_DECL:
-      {
-        enum XML_Error result = processXmlDecl(parser, 0, s, next);
-        if (result != XML_ERROR_NONE)
-          return result;
-        enc = encoding;
-        handleDefault = XML_FALSE;
-      }
-      break;
+    case XML_ROLE_XML_DECL: {
+      enum XML_Error result = processXmlDecl(parser, 0, s, next);
+      if (result != XML_ERROR_NONE)
+        return result;
+      enc = parser->m_encoding;
+      handleDefault = XML_FALSE;
+    } break;
     case XML_ROLE_DOCTYPE_NAME:
-      if (startDoctypeDeclHandler) {
-        doctypeName = poolStoreString(&tempPool, enc, s, next);
-        if (!doctypeName)
+      if (parser->m_startDoctypeDeclHandler) {
+        parser->m_doctypeName
+            = poolStoreString(&parser->m_tempPool, enc, s, next);
+        if (! parser->m_doctypeName)
           return XML_ERROR_NO_MEMORY;
-        poolFinish(&tempPool);
-        doctypePubid = NULL;
+        poolFinish(&parser->m_tempPool);
+        parser->m_doctypePubid = NULL;
         handleDefault = XML_FALSE;
       }
-      doctypeSysid = NULL; /* always initialize to NULL */
+      parser->m_doctypeSysid = NULL; /* always initialize to NULL */
       break;
     case XML_ROLE_DOCTYPE_INTERNAL_SUBSET:
-      if (startDoctypeDeclHandler) {
-        startDoctypeDeclHandler(handlerArg, doctypeName, doctypeSysid,
-                                doctypePubid, 1);
-        doctypeName = NULL;
-        poolClear(&tempPool);
+      if (parser->m_startDoctypeDeclHandler) {
+        parser->m_startDoctypeDeclHandler(
+            parser->m_handlerArg, parser->m_doctypeName, parser->m_doctypeSysid,
+            parser->m_doctypePubid, 1);
+        parser->m_doctypeName = NULL;
+        poolClear(&parser->m_tempPool);
         handleDefault = XML_FALSE;
       }
       break;
 #ifdef XML_DTD
-    case XML_ROLE_TEXT_DECL:
-      {
-        enum XML_Error result = processXmlDecl(parser, 1, s, next);
-        if (result != XML_ERROR_NONE)
-          return result;
-        enc = encoding;
-        handleDefault = XML_FALSE;
-      }
-      break;
+    case XML_ROLE_TEXT_DECL: {
+      enum XML_Error result = processXmlDecl(parser, 1, s, next);
+      if (result != XML_ERROR_NONE)
+        return result;
+      enc = parser->m_encoding;
+      handleDefault = XML_FALSE;
+    } break;
 #endif /* XML_DTD */
     case XML_ROLE_DOCTYPE_PUBLIC_ID:
 #ifdef XML_DTD
-      useForeignDTD = XML_FALSE;
-      declEntity = (ENTITY *)lookup(parser,
-                                    &dtd->paramEntities,
-                                    externalSubsetName,
-                                    sizeof(ENTITY));
-      if (!declEntity)
+      parser->m_useForeignDTD = XML_FALSE;
+      parser->m_declEntity = (ENTITY *)lookup(
+          parser, &dtd->paramEntities, externalSubsetName, sizeof(ENTITY));
+      if (! parser->m_declEntity)
         return XML_ERROR_NO_MEMORY;
 #endif /* XML_DTD */
       dtd->hasParamEntityRefs = XML_TRUE;
-      if (startDoctypeDeclHandler) {
+      if (parser->m_startDoctypeDeclHandler) {
         XML_Char *pubId;
-        if (!XmlIsPublicId(enc, s, next, eventPP))
+        if (! XmlIsPublicId(enc, s, next, eventPP))
           return XML_ERROR_PUBLICID;
-        pubId = poolStoreString(&tempPool, enc,
+        pubId = poolStoreString(&parser->m_tempPool, enc,
                                 s + enc->minBytesPerChar,
                                 next - enc->minBytesPerChar);
-        if (!pubId)
+        if (! pubId)
           return XML_ERROR_NO_MEMORY;
         normalizePublicId(pubId);
-        poolFinish(&tempPool);
-        doctypePubid = pubId;
+        poolFinish(&parser->m_tempPool);
+        parser->m_doctypePubid = pubId;
         handleDefault = XML_FALSE;
         goto alreadyChecked;
       }
       /* fall through */
     case XML_ROLE_ENTITY_PUBLIC_ID:
-      if (!XmlIsPublicId(enc, s, next, eventPP))
+      if (! XmlIsPublicId(enc, s, next, eventPP))
         return XML_ERROR_PUBLICID;
     alreadyChecked:
-      if (dtd->keepProcessing && declEntity) {
-        XML_Char *tem = poolStoreString(&dtd->pool,
-                                        enc,
-                                        s + enc->minBytesPerChar,
-                                        next - enc->minBytesPerChar);
-        if (!tem)
+      if (dtd->keepProcessing && parser->m_declEntity) {
+        XML_Char *tem
+            = poolStoreString(&dtd->pool, enc, s + enc->minBytesPerChar,
+                              next - enc->minBytesPerChar);
+        if (! tem)
           return XML_ERROR_NO_MEMORY;
         normalizePublicId(tem);
-        declEntity->publicId = tem;
+        parser->m_declEntity->publicId = tem;
         poolFinish(&dtd->pool);
-        if (entityDeclHandler)
+        /* Don't suppress the default handler if we fell through from
+         * the XML_ROLE_DOCTYPE_PUBLIC_ID case.
+         */
+        if (parser->m_entityDeclHandler && role == XML_ROLE_ENTITY_PUBLIC_ID)
           handleDefault = XML_FALSE;
       }
       break;
     case XML_ROLE_DOCTYPE_CLOSE:
-      if (doctypeName) {
-        startDoctypeDeclHandler(handlerArg, doctypeName,
-                                doctypeSysid, doctypePubid, 0);
-        poolClear(&tempPool);
+      if (allowClosingDoctype != XML_TRUE) {
+        /* Must not close doctype from within expanded parameter entities */
+        return XML_ERROR_INVALID_TOKEN;
+      }
+
+      if (parser->m_doctypeName) {
+        parser->m_startDoctypeDeclHandler(
+            parser->m_handlerArg, parser->m_doctypeName, parser->m_doctypeSysid,
+            parser->m_doctypePubid, 0);
+        poolClear(&parser->m_tempPool);
         handleDefault = XML_FALSE;
       }
-      /* doctypeSysid will be non-NULL in the case of a previous
-         XML_ROLE_DOCTYPE_SYSTEM_ID, even if startDoctypeDeclHandler
+      /* parser->m_doctypeSysid will be non-NULL in the case of a previous
+         XML_ROLE_DOCTYPE_SYSTEM_ID, even if parser->m_startDoctypeDeclHandler
          was not set, indicating an external subset
       */
 #ifdef XML_DTD
-      if (doctypeSysid || useForeignDTD) {
+      if (parser->m_doctypeSysid || parser->m_useForeignDTD) {
         XML_Bool hadParamEntityRefs = dtd->hasParamEntityRefs;
         dtd->hasParamEntityRefs = XML_TRUE;
-        if (paramEntityParsing && externalEntityRefHandler) {
-          ENTITY *entity = (ENTITY *)lookup(parser,
-                                            &dtd->paramEntities,
-                                            externalSubsetName,
-                                            sizeof(ENTITY));
-          if (!entity) {
+        if (parser->m_paramEntityParsing
+            && parser->m_externalEntityRefHandler) {
+          ENTITY *entity = (ENTITY *)lookup(parser, &dtd->paramEntities,
+                                            externalSubsetName, sizeof(ENTITY));
+          if (! entity) {
             /* The external subset name "#" will have already been
              * inserted into the hash table at the start of the
              * external entity parsing, so no allocation will happen
@@ -4533,33 +4304,30 @@ doProlog(XML_Parser parser,
              */
             return XML_ERROR_NO_MEMORY; /* LCOV_EXCL_LINE */
           }
-          if (useForeignDTD)
-            entity->base = curBase;
+          if (parser->m_useForeignDTD)
+            entity->base = parser->m_curBase;
           dtd->paramEntityRead = XML_FALSE;
-          if (!externalEntityRefHandler(externalEntityRefHandlerArg,
-                                        0,
-                                        entity->base,
-                                        entity->systemId,
-                                        entity->publicId))
+          if (! parser->m_externalEntityRefHandler(
+                  parser->m_externalEntityRefHandlerArg, 0, entity->base,
+                  entity->systemId, entity->publicId))
             return XML_ERROR_EXTERNAL_ENTITY_HANDLING;
           if (dtd->paramEntityRead) {
-            if (!dtd->standalone &&
-                notStandaloneHandler &&
-                !notStandaloneHandler(handlerArg))
+            if (! dtd->standalone && parser->m_notStandaloneHandler
+                && ! parser->m_notStandaloneHandler(parser->m_handlerArg))
               return XML_ERROR_NOT_STANDALONE;
           }
           /* if we didn't read the foreign DTD then this means that there
              is no external subset and we must reset dtd->hasParamEntityRefs
           */
-          else if (!doctypeSysid)
+          else if (! parser->m_doctypeSysid)
             dtd->hasParamEntityRefs = hadParamEntityRefs;
           /* end of DTD - no need to update dtd->keepProcessing */
         }
-        useForeignDTD = XML_FALSE;
+        parser->m_useForeignDTD = XML_FALSE;
       }
 #endif /* XML_DTD */
-      if (endDoctypeDeclHandler) {
-        endDoctypeDeclHandler(handlerArg);
+      if (parser->m_endDoctypeDeclHandler) {
+        parser->m_endDoctypeDeclHandler(parser->m_handlerArg);
         handleDefault = XML_FALSE;
       }
       break;
@@ -4568,27 +4336,24 @@ doProlog(XML_Parser parser,
       /* if there is no DOCTYPE declaration then now is the
          last chance to read the foreign DTD
       */
-      if (useForeignDTD) {
+      if (parser->m_useForeignDTD) {
         XML_Bool hadParamEntityRefs = dtd->hasParamEntityRefs;
         dtd->hasParamEntityRefs = XML_TRUE;
-        if (paramEntityParsing && externalEntityRefHandler) {
+        if (parser->m_paramEntityParsing
+            && parser->m_externalEntityRefHandler) {
           ENTITY *entity = (ENTITY *)lookup(parser, &dtd->paramEntities,
-                                            externalSubsetName,
-                                            sizeof(ENTITY));
-          if (!entity)
+                                            externalSubsetName, sizeof(ENTITY));
+          if (! entity)
             return XML_ERROR_NO_MEMORY;
-          entity->base = curBase;
+          entity->base = parser->m_curBase;
           dtd->paramEntityRead = XML_FALSE;
-          if (!externalEntityRefHandler(externalEntityRefHandlerArg,
-                                        0,
-                                        entity->base,
-                                        entity->systemId,
-                                        entity->publicId))
+          if (! parser->m_externalEntityRefHandler(
+                  parser->m_externalEntityRefHandlerArg, 0, entity->base,
+                  entity->systemId, entity->publicId))
             return XML_ERROR_EXTERNAL_ENTITY_HANDLING;
           if (dtd->paramEntityRead) {
-            if (!dtd->standalone &&
-                notStandaloneHandler &&
-                !notStandaloneHandler(handlerArg))
+            if (! dtd->standalone && parser->m_notStandaloneHandler
+                && ! parser->m_notStandaloneHandler(parser->m_handlerArg))
               return XML_ERROR_NOT_STANDALONE;
           }
           /* if we didn't read the foreign DTD then this means that there
@@ -4600,93 +4365,93 @@ doProlog(XML_Parser parser,
         }
       }
 #endif /* XML_DTD */
-      processor = contentProcessor;
+      parser->m_processor = contentProcessor;
       return contentProcessor(parser, s, end, nextPtr);
     case XML_ROLE_ATTLIST_ELEMENT_NAME:
-      declElementType = getElementType(parser, enc, s, next);
-      if (!declElementType)
+      parser->m_declElementType = getElementType(parser, enc, s, next);
+      if (! parser->m_declElementType)
         return XML_ERROR_NO_MEMORY;
       goto checkAttListDeclHandler;
     case XML_ROLE_ATTRIBUTE_NAME:
-      declAttributeId = getAttributeId(parser, enc, s, next);
-      if (!declAttributeId)
+      parser->m_declAttributeId = getAttributeId(parser, enc, s, next);
+      if (! parser->m_declAttributeId)
         return XML_ERROR_NO_MEMORY;
-      declAttributeIsCdata = XML_FALSE;
-      declAttributeType = NULL;
-      declAttributeIsId = XML_FALSE;
+      parser->m_declAttributeIsCdata = XML_FALSE;
+      parser->m_declAttributeType = NULL;
+      parser->m_declAttributeIsId = XML_FALSE;
       goto checkAttListDeclHandler;
     case XML_ROLE_ATTRIBUTE_TYPE_CDATA:
-      declAttributeIsCdata = XML_TRUE;
-      declAttributeType = atypeCDATA;
+      parser->m_declAttributeIsCdata = XML_TRUE;
+      parser->m_declAttributeType = atypeCDATA;
       goto checkAttListDeclHandler;
     case XML_ROLE_ATTRIBUTE_TYPE_ID:
-      declAttributeIsId = XML_TRUE;
-      declAttributeType = atypeID;
+      parser->m_declAttributeIsId = XML_TRUE;
+      parser->m_declAttributeType = atypeID;
       goto checkAttListDeclHandler;
     case XML_ROLE_ATTRIBUTE_TYPE_IDREF:
-      declAttributeType = atypeIDREF;
+      parser->m_declAttributeType = atypeIDREF;
       goto checkAttListDeclHandler;
     case XML_ROLE_ATTRIBUTE_TYPE_IDREFS:
-      declAttributeType = atypeIDREFS;
+      parser->m_declAttributeType = atypeIDREFS;
       goto checkAttListDeclHandler;
     case XML_ROLE_ATTRIBUTE_TYPE_ENTITY:
-      declAttributeType = atypeENTITY;
+      parser->m_declAttributeType = atypeENTITY;
       goto checkAttListDeclHandler;
     case XML_ROLE_ATTRIBUTE_TYPE_ENTITIES:
-      declAttributeType = atypeENTITIES;
+      parser->m_declAttributeType = atypeENTITIES;
       goto checkAttListDeclHandler;
     case XML_ROLE_ATTRIBUTE_TYPE_NMTOKEN:
-      declAttributeType = atypeNMTOKEN;
+      parser->m_declAttributeType = atypeNMTOKEN;
       goto checkAttListDeclHandler;
     case XML_ROLE_ATTRIBUTE_TYPE_NMTOKENS:
-      declAttributeType = atypeNMTOKENS;
+      parser->m_declAttributeType = atypeNMTOKENS;
     checkAttListDeclHandler:
-      if (dtd->keepProcessing && attlistDeclHandler)
+      if (dtd->keepProcessing && parser->m_attlistDeclHandler)
         handleDefault = XML_FALSE;
       break;
     case XML_ROLE_ATTRIBUTE_ENUM_VALUE:
     case XML_ROLE_ATTRIBUTE_NOTATION_VALUE:
-      if (dtd->keepProcessing && attlistDeclHandler) {
+      if (dtd->keepProcessing && parser->m_attlistDeclHandler) {
         const XML_Char *prefix;
-        if (declAttributeType) {
+        if (parser->m_declAttributeType) {
           prefix = enumValueSep;
+        } else {
+          prefix = (role == XML_ROLE_ATTRIBUTE_NOTATION_VALUE ? notationPrefix
+                                                              : enumValueStart);
         }
-        else {
-          prefix = (role == XML_ROLE_ATTRIBUTE_NOTATION_VALUE
-                    ? notationPrefix
-                    : enumValueStart);
-        }
-        if (!poolAppendString(&tempPool, prefix))
+        if (! poolAppendString(&parser->m_tempPool, prefix))
           return XML_ERROR_NO_MEMORY;
-        if (!poolAppend(&tempPool, enc, s, next))
+        if (! poolAppend(&parser->m_tempPool, enc, s, next))
           return XML_ERROR_NO_MEMORY;
-        declAttributeType = tempPool.start;
+        parser->m_declAttributeType = parser->m_tempPool.start;
         handleDefault = XML_FALSE;
       }
       break;
     case XML_ROLE_IMPLIED_ATTRIBUTE_VALUE:
     case XML_ROLE_REQUIRED_ATTRIBUTE_VALUE:
       if (dtd->keepProcessing) {
-        if (!defineAttribute(declElementType, declAttributeId,
-                             declAttributeIsCdata, declAttributeIsId,
-                             0, parser))
+        if (! defineAttribute(parser->m_declElementType,
+                              parser->m_declAttributeId,
+                              parser->m_declAttributeIsCdata,
+                              parser->m_declAttributeIsId, 0, parser))
           return XML_ERROR_NO_MEMORY;
-        if (attlistDeclHandler && declAttributeType) {
-          if (*declAttributeType == XML_T(ASCII_LPAREN)
-              || (*declAttributeType == XML_T(ASCII_N)
-                  && declAttributeType[1] == XML_T(ASCII_O))) {
+        if (parser->m_attlistDeclHandler && parser->m_declAttributeType) {
+          if (*parser->m_declAttributeType == XML_T(ASCII_LPAREN)
+              || (*parser->m_declAttributeType == XML_T(ASCII_N)
+                  && parser->m_declAttributeType[1] == XML_T(ASCII_O))) {
             /* Enumerated or Notation type */
-            if (!poolAppendChar(&tempPool, XML_T(ASCII_RPAREN))
-                || !poolAppendChar(&tempPool, XML_T('\0')))
+            if (! poolAppendChar(&parser->m_tempPool, XML_T(ASCII_RPAREN))
+                || ! poolAppendChar(&parser->m_tempPool, XML_T('\0')))
               return XML_ERROR_NO_MEMORY;
-            declAttributeType = tempPool.start;
-            poolFinish(&tempPool);
+            parser->m_declAttributeType = parser->m_tempPool.start;
+            poolFinish(&parser->m_tempPool);
           }
           *eventEndPP = s;
-          attlistDeclHandler(handlerArg, declElementType->name,
-                             declAttributeId->name, declAttributeType,
-                             0, role == XML_ROLE_REQUIRED_ATTRIBUTE_VALUE);
-          poolClear(&tempPool);
+          parser->m_attlistDeclHandler(
+              parser->m_handlerArg, parser->m_declElementType->name,
+              parser->m_declAttributeId->name, parser->m_declAttributeType, 0,
+              role == XML_ROLE_REQUIRED_ATTRIBUTE_VALUE);
+          poolClear(&parser->m_tempPool);
           handleDefault = XML_FALSE;
         }
       }
@@ -4695,61 +4460,57 @@ doProlog(XML_Parser parser,
     case XML_ROLE_FIXED_ATTRIBUTE_VALUE:
       if (dtd->keepProcessing) {
         const XML_Char *attVal;
-        enum XML_Error result =
-          storeAttributeValue(parser, enc, declAttributeIsCdata,
-                              s + enc->minBytesPerChar,
-                              next - enc->minBytesPerChar,
-                              &dtd->pool);
+        enum XML_Error result = storeAttributeValue(
+            parser, enc, parser->m_declAttributeIsCdata,
+            s + enc->minBytesPerChar, next - enc->minBytesPerChar, &dtd->pool);
         if (result)
           return result;
         attVal = poolStart(&dtd->pool);
         poolFinish(&dtd->pool);
         /* ID attributes aren't allowed to have a default */
-        if (!defineAttribute(declElementType, declAttributeId,
-                             declAttributeIsCdata, XML_FALSE, attVal, parser))
+        if (! defineAttribute(
+                parser->m_declElementType, parser->m_declAttributeId,
+                parser->m_declAttributeIsCdata, XML_FALSE, attVal, parser))
           return XML_ERROR_NO_MEMORY;
-        if (attlistDeclHandler && declAttributeType) {
-          if (*declAttributeType == XML_T(ASCII_LPAREN)
-              || (*declAttributeType == XML_T(ASCII_N)
-                  && declAttributeType[1] == XML_T(ASCII_O))) {
+        if (parser->m_attlistDeclHandler && parser->m_declAttributeType) {
+          if (*parser->m_declAttributeType == XML_T(ASCII_LPAREN)
+              || (*parser->m_declAttributeType == XML_T(ASCII_N)
+                  && parser->m_declAttributeType[1] == XML_T(ASCII_O))) {
             /* Enumerated or Notation type */
-            if (!poolAppendChar(&tempPool, XML_T(ASCII_RPAREN))
-                || !poolAppendChar(&tempPool, XML_T('\0')))
+            if (! poolAppendChar(&parser->m_tempPool, XML_T(ASCII_RPAREN))
+                || ! poolAppendChar(&parser->m_tempPool, XML_T('\0')))
               return XML_ERROR_NO_MEMORY;
-            declAttributeType = tempPool.start;
-            poolFinish(&tempPool);
+            parser->m_declAttributeType = parser->m_tempPool.start;
+            poolFinish(&parser->m_tempPool);
           }
           *eventEndPP = s;
-          attlistDeclHandler(handlerArg, declElementType->name,
-                             declAttributeId->name, declAttributeType,
-                             attVal,
-                             role == XML_ROLE_FIXED_ATTRIBUTE_VALUE);
-          poolClear(&tempPool);
+          parser->m_attlistDeclHandler(
+              parser->m_handlerArg, parser->m_declElementType->name,
+              parser->m_declAttributeId->name, parser->m_declAttributeType,
+              attVal, role == XML_ROLE_FIXED_ATTRIBUTE_VALUE);
+          poolClear(&parser->m_tempPool);
           handleDefault = XML_FALSE;
         }
       }
       break;
     case XML_ROLE_ENTITY_VALUE:
       if (dtd->keepProcessing) {
-        enum XML_Error result = storeEntityValue(parser, enc,
-                                            s + enc->minBytesPerChar,
-                                            next - enc->minBytesPerChar);
-        if (declEntity) {
-          declEntity->textPtr = poolStart(&dtd->entityValuePool);
-          declEntity->textLen = (int)(poolLength(&dtd->entityValuePool));
+        enum XML_Error result = storeEntityValue(
+            parser, enc, s + enc->minBytesPerChar, next - enc->minBytesPerChar);
+        if (parser->m_declEntity) {
+          parser->m_declEntity->textPtr = poolStart(&dtd->entityValuePool);
+          parser->m_declEntity->textLen
+              = (int)(poolLength(&dtd->entityValuePool));
           poolFinish(&dtd->entityValuePool);
-          if (entityDeclHandler) {
+          if (parser->m_entityDeclHandler) {
             *eventEndPP = s;
-            entityDeclHandler(handlerArg,
-                              declEntity->name,
-                              declEntity->is_param,
-                              declEntity->textPtr,
-                              declEntity->textLen,
-                              curBase, 0, 0, 0);
+            parser->m_entityDeclHandler(
+                parser->m_handlerArg, parser->m_declEntity->name,
+                parser->m_declEntity->is_param, parser->m_declEntity->textPtr,
+                parser->m_declEntity->textLen, parser->m_curBase, 0, 0, 0);
             handleDefault = XML_FALSE;
           }
-        }
-        else
+        } else
           poolDiscard(&dtd->entityValuePool);
         if (result != XML_ERROR_NONE)
           return result;
@@ -4757,227 +4518,212 @@ doProlog(XML_Parser parser,
       break;
     case XML_ROLE_DOCTYPE_SYSTEM_ID:
 #ifdef XML_DTD
-      useForeignDTD = XML_FALSE;
+      parser->m_useForeignDTD = XML_FALSE;
 #endif /* XML_DTD */
       dtd->hasParamEntityRefs = XML_TRUE;
-      if (startDoctypeDeclHandler) {
-        doctypeSysid = poolStoreString(&tempPool, enc,
-                                       s + enc->minBytesPerChar,
-                                       next - enc->minBytesPerChar);
-        if (doctypeSysid == NULL)
+      if (parser->m_startDoctypeDeclHandler) {
+        parser->m_doctypeSysid = poolStoreString(&parser->m_tempPool, enc,
+                                                 s + enc->minBytesPerChar,
+                                                 next - enc->minBytesPerChar);
+        if (parser->m_doctypeSysid == NULL)
           return XML_ERROR_NO_MEMORY;
-        poolFinish(&tempPool);
+        poolFinish(&parser->m_tempPool);
         handleDefault = XML_FALSE;
       }
 #ifdef XML_DTD
       else
-        /* use externalSubsetName to make doctypeSysid non-NULL
-           for the case where no startDoctypeDeclHandler is set */
-        doctypeSysid = externalSubsetName;
+        /* use externalSubsetName to make parser->m_doctypeSysid non-NULL
+           for the case where no parser->m_startDoctypeDeclHandler is set */
+        parser->m_doctypeSysid = externalSubsetName;
 #endif /* XML_DTD */
-      if (!dtd->standalone
+      if (! dtd->standalone
 #ifdef XML_DTD
-          && !paramEntityParsing
+          && ! parser->m_paramEntityParsing
 #endif /* XML_DTD */
-          && notStandaloneHandler
-          && !notStandaloneHandler(handlerArg))
+          && parser->m_notStandaloneHandler
+          && ! parser->m_notStandaloneHandler(parser->m_handlerArg))
         return XML_ERROR_NOT_STANDALONE;
 #ifndef XML_DTD
       break;
-#else /* XML_DTD */
-      if (!declEntity) {
-        declEntity = (ENTITY *)lookup(parser,
-                                      &dtd->paramEntities,
-                                      externalSubsetName,
-                                      sizeof(ENTITY));
-        if (!declEntity)
+#else  /* XML_DTD */
+      if (! parser->m_declEntity) {
+        parser->m_declEntity = (ENTITY *)lookup(
+            parser, &dtd->paramEntities, externalSubsetName, sizeof(ENTITY));
+        if (! parser->m_declEntity)
           return XML_ERROR_NO_MEMORY;
-        declEntity->publicId = NULL;
+        parser->m_declEntity->publicId = NULL;
       }
-      /* fall through */
 #endif /* XML_DTD */
+      /* fall through */
     case XML_ROLE_ENTITY_SYSTEM_ID:
-      if (dtd->keepProcessing && declEntity) {
-        declEntity->systemId = poolStoreString(&dtd->pool, enc,
-                                               s + enc->minBytesPerChar,
-                                               next - enc->minBytesPerChar);
-        if (!declEntity->systemId)
+      if (dtd->keepProcessing && parser->m_declEntity) {
+        parser->m_declEntity->systemId
+            = poolStoreString(&dtd->pool, enc, s + enc->minBytesPerChar,
+                              next - enc->minBytesPerChar);
+        if (! parser->m_declEntity->systemId)
           return XML_ERROR_NO_MEMORY;
-        declEntity->base = curBase;
+        parser->m_declEntity->base = parser->m_curBase;
         poolFinish(&dtd->pool);
-        if (entityDeclHandler)
+        /* Don't suppress the default handler if we fell through from
+         * the XML_ROLE_DOCTYPE_SYSTEM_ID case.
+         */
+        if (parser->m_entityDeclHandler && role == XML_ROLE_ENTITY_SYSTEM_ID)
           handleDefault = XML_FALSE;
       }
       break;
     case XML_ROLE_ENTITY_COMPLETE:
-      if (dtd->keepProcessing && declEntity && entityDeclHandler) {
+      if (dtd->keepProcessing && parser->m_declEntity
+          && parser->m_entityDeclHandler) {
         *eventEndPP = s;
-        entityDeclHandler(handlerArg,
-                          declEntity->name,
-                          declEntity->is_param,
-                          0,0,
-                          declEntity->base,
-                          declEntity->systemId,
-                          declEntity->publicId,
-                          0);
+        parser->m_entityDeclHandler(
+            parser->m_handlerArg, parser->m_declEntity->name,
+            parser->m_declEntity->is_param, 0, 0, parser->m_declEntity->base,
+            parser->m_declEntity->systemId, parser->m_declEntity->publicId, 0);
         handleDefault = XML_FALSE;
       }
       break;
     case XML_ROLE_ENTITY_NOTATION_NAME:
-      if (dtd->keepProcessing && declEntity) {
-        declEntity->notation = poolStoreString(&dtd->pool, enc, s, next);
-        if (!declEntity->notation)
+      if (dtd->keepProcessing && parser->m_declEntity) {
+        parser->m_declEntity->notation
+            = poolStoreString(&dtd->pool, enc, s, next);
+        if (! parser->m_declEntity->notation)
           return XML_ERROR_NO_MEMORY;
         poolFinish(&dtd->pool);
-        if (unparsedEntityDeclHandler) {
+        if (parser->m_unparsedEntityDeclHandler) {
           *eventEndPP = s;
-          unparsedEntityDeclHandler(handlerArg,
-                                    declEntity->name,
-                                    declEntity->base,
-                                    declEntity->systemId,
-                                    declEntity->publicId,
-                                    declEntity->notation);
+          parser->m_unparsedEntityDeclHandler(
+              parser->m_handlerArg, parser->m_declEntity->name,
+              parser->m_declEntity->base, parser->m_declEntity->systemId,
+              parser->m_declEntity->publicId, parser->m_declEntity->notation);
           handleDefault = XML_FALSE;
-        }
-        else if (entityDeclHandler) {
+        } else if (parser->m_entityDeclHandler) {
           *eventEndPP = s;
-          entityDeclHandler(handlerArg,
-                            declEntity->name,
-                            0,0,0,
-                            declEntity->base,
-                            declEntity->systemId,
-                            declEntity->publicId,
-                            declEntity->notation);
+          parser->m_entityDeclHandler(
+              parser->m_handlerArg, parser->m_declEntity->name, 0, 0, 0,
+              parser->m_declEntity->base, parser->m_declEntity->systemId,
+              parser->m_declEntity->publicId, parser->m_declEntity->notation);
           handleDefault = XML_FALSE;
         }
       }
       break;
-    case XML_ROLE_GENERAL_ENTITY_NAME:
-      {
-        if (XmlPredefinedEntityName(enc, s, next)) {
-          declEntity = NULL;
-          break;
-        }
-        if (dtd->keepProcessing) {
-          const XML_Char *name = poolStoreString(&dtd->pool, enc, s, next);
-          if (!name)
-            return XML_ERROR_NO_MEMORY;
-          declEntity = (ENTITY *)lookup(parser, &dtd->generalEntities, name,
-                                        sizeof(ENTITY));
-          if (!declEntity)
-            return XML_ERROR_NO_MEMORY;
-          if (declEntity->name != name) {
-            poolDiscard(&dtd->pool);
-            declEntity = NULL;
-          }
-          else {
-            poolFinish(&dtd->pool);
-            declEntity->publicId = NULL;
-            declEntity->is_param = XML_FALSE;
-            /* if we have a parent parser or are reading an internal parameter
-               entity, then the entity declaration is not considered "internal"
-            */
-            declEntity->is_internal = !(parentParser || openInternalEntities);
-            if (entityDeclHandler)
-              handleDefault = XML_FALSE;
-          }
-        }
-        else {
+    case XML_ROLE_GENERAL_ENTITY_NAME: {
+      if (XmlPredefinedEntityName(enc, s, next)) {
+        parser->m_declEntity = NULL;
+        break;
+      }
+      if (dtd->keepProcessing) {
+        const XML_Char *name = poolStoreString(&dtd->pool, enc, s, next);
+        if (! name)
+          return XML_ERROR_NO_MEMORY;
+        parser->m_declEntity = (ENTITY *)lookup(parser, &dtd->generalEntities,
+                                                name, sizeof(ENTITY));
+        if (! parser->m_declEntity)
+          return XML_ERROR_NO_MEMORY;
+        if (parser->m_declEntity->name != name) {
           poolDiscard(&dtd->pool);
-          declEntity = NULL;
+          parser->m_declEntity = NULL;
+        } else {
+          poolFinish(&dtd->pool);
+          parser->m_declEntity->publicId = NULL;
+          parser->m_declEntity->is_param = XML_FALSE;
+          /* if we have a parent parser or are reading an internal parameter
+             entity, then the entity declaration is not considered "internal"
+          */
+          parser->m_declEntity->is_internal
+              = ! (parser->m_parentParser || parser->m_openInternalEntities);
+          if (parser->m_entityDeclHandler)
+            handleDefault = XML_FALSE;
         }
+      } else {
+        poolDiscard(&dtd->pool);
+        parser->m_declEntity = NULL;
       }
-      break;
+    } break;
     case XML_ROLE_PARAM_ENTITY_NAME:
 #ifdef XML_DTD
       if (dtd->keepProcessing) {
         const XML_Char *name = poolStoreString(&dtd->pool, enc, s, next);
-        if (!name)
+        if (! name)
           return XML_ERROR_NO_MEMORY;
-        declEntity = (ENTITY *)lookup(parser, &dtd->paramEntities,
-                                           name, sizeof(ENTITY));
-        if (!declEntity)
+        parser->m_declEntity = (ENTITY *)lookup(parser, &dtd->paramEntities,
+                                                name, sizeof(ENTITY));
+        if (! parser->m_declEntity)
           return XML_ERROR_NO_MEMORY;
-        if (declEntity->name != name) {
+        if (parser->m_declEntity->name != name) {
           poolDiscard(&dtd->pool);
-          declEntity = NULL;
-        }
-        else {
+          parser->m_declEntity = NULL;
+        } else {
           poolFinish(&dtd->pool);
-          declEntity->publicId = NULL;
-          declEntity->is_param = XML_TRUE;
+          parser->m_declEntity->publicId = NULL;
+          parser->m_declEntity->is_param = XML_TRUE;
           /* if we have a parent parser or are reading an internal parameter
              entity, then the entity declaration is not considered "internal"
           */
-          declEntity->is_internal = !(parentParser || openInternalEntities);
-          if (entityDeclHandler)
+          parser->m_declEntity->is_internal
+              = ! (parser->m_parentParser || parser->m_openInternalEntities);
+          if (parser->m_entityDeclHandler)
             handleDefault = XML_FALSE;
         }
-      }
-      else {
+      } else {
         poolDiscard(&dtd->pool);
-        declEntity = NULL;
+        parser->m_declEntity = NULL;
       }
-#else /* not XML_DTD */
-      declEntity = NULL;
+#else  /* not XML_DTD */
+      parser->m_declEntity = NULL;
 #endif /* XML_DTD */
       break;
     case XML_ROLE_NOTATION_NAME:
-      declNotationPublicId = NULL;
-      declNotationName = NULL;
-      if (notationDeclHandler) {
-        declNotationName = poolStoreString(&tempPool, enc, s, next);
-        if (!declNotationName)
+      parser->m_declNotationPublicId = NULL;
+      parser->m_declNotationName = NULL;
+      if (parser->m_notationDeclHandler) {
+        parser->m_declNotationName
+            = poolStoreString(&parser->m_tempPool, enc, s, next);
+        if (! parser->m_declNotationName)
           return XML_ERROR_NO_MEMORY;
-        poolFinish(&tempPool);
+        poolFinish(&parser->m_tempPool);
         handleDefault = XML_FALSE;
       }
       break;
     case XML_ROLE_NOTATION_PUBLIC_ID:
-      if (!XmlIsPublicId(enc, s, next, eventPP))
+      if (! XmlIsPublicId(enc, s, next, eventPP))
         return XML_ERROR_PUBLICID;
-      if (declNotationName) {  /* means notationDeclHandler != NULL */
-        XML_Char *tem = poolStoreString(&tempPool,
-                                        enc,
+      if (parser
+              ->m_declNotationName) { /* means m_notationDeclHandler != NULL */
+        XML_Char *tem = poolStoreString(&parser->m_tempPool, enc,
                                         s + enc->minBytesPerChar,
                                         next - enc->minBytesPerChar);
-        if (!tem)
+        if (! tem)
           return XML_ERROR_NO_MEMORY;
         normalizePublicId(tem);
-        declNotationPublicId = tem;
-        poolFinish(&tempPool);
+        parser->m_declNotationPublicId = tem;
+        poolFinish(&parser->m_tempPool);
         handleDefault = XML_FALSE;
       }
       break;
     case XML_ROLE_NOTATION_SYSTEM_ID:
-      if (declNotationName && notationDeclHandler) {
-        const XML_Char *systemId
-          = poolStoreString(&tempPool, enc,
-                            s + enc->minBytesPerChar,
-                            next - enc->minBytesPerChar);
-        if (!systemId)
+      if (parser->m_declNotationName && parser->m_notationDeclHandler) {
+        const XML_Char *systemId = poolStoreString(&parser->m_tempPool, enc,
+                                                   s + enc->minBytesPerChar,
+                                                   next - enc->minBytesPerChar);
+        if (! systemId)
           return XML_ERROR_NO_MEMORY;
         *eventEndPP = s;
-        notationDeclHandler(handlerArg,
-                            declNotationName,
-                            curBase,
-                            systemId,
-                            declNotationPublicId);
+        parser->m_notationDeclHandler(
+            parser->m_handlerArg, parser->m_declNotationName, parser->m_curBase,
+            systemId, parser->m_declNotationPublicId);
         handleDefault = XML_FALSE;
       }
-      poolClear(&tempPool);
+      poolClear(&parser->m_tempPool);
       break;
     case XML_ROLE_NOTATION_NO_SYSTEM_ID:
-      if (declNotationPublicId && notationDeclHandler) {
+      if (parser->m_declNotationPublicId && parser->m_notationDeclHandler) {
         *eventEndPP = s;
-        notationDeclHandler(handlerArg,
-                            declNotationName,
-                            curBase,
-                            0,
-                            declNotationPublicId);
+        parser->m_notationDeclHandler(
+            parser->m_handlerArg, parser->m_declNotationName, parser->m_curBase,
+            0, parser->m_declNotationPublicId);
         handleDefault = XML_FALSE;
       }
-      poolClear(&tempPool);
+      poolClear(&parser->m_tempPool);
       break;
     case XML_ROLE_ERROR:
       switch (tok) {
@@ -4991,94 +4737,95 @@ doProlog(XML_Parser parser,
         return XML_ERROR_SYNTAX;
       }
 #ifdef XML_DTD
-    case XML_ROLE_IGNORE_SECT:
-      {
-        enum XML_Error result;
-        if (defaultHandler)
-          reportDefault(parser, enc, s, next);
-        handleDefault = XML_FALSE;
-        result = doIgnoreSection(parser, enc, &next, end, nextPtr, haveMore);
-        if (result != XML_ERROR_NONE)
-          return result;
-        else if (!next) {
-          processor = ignoreSectionProcessor;
-          return result;
-        }
+    case XML_ROLE_IGNORE_SECT: {
+      enum XML_Error result;
+      if (parser->m_defaultHandler)
+        reportDefault(parser, enc, s, next);
+      handleDefault = XML_FALSE;
+      result = doIgnoreSection(parser, enc, &next, end, nextPtr, haveMore);
+      if (result != XML_ERROR_NONE)
+        return result;
+      else if (! next) {
+        parser->m_processor = ignoreSectionProcessor;
+        return result;
       }
-      break;
+    } break;
 #endif /* XML_DTD */
     case XML_ROLE_GROUP_OPEN:
-      if (prologState.level >= groupSize) {
-        if (groupSize) {
-          char *temp = (char *)REALLOC(groupConnector, groupSize *= 2);
-          if (temp == NULL) {
-            groupSize /= 2;
-            return XML_ERROR_NO_MEMORY;
+      if (parser->m_prologState.level >= parser->m_groupSize) {
+        if (parser->m_groupSize) {
+          {
+            char *const new_connector = (char *)REALLOC(
+                parser, parser->m_groupConnector, parser->m_groupSize *= 2);
+            if (new_connector == NULL) {
+              parser->m_groupSize /= 2;
+              return XML_ERROR_NO_MEMORY;
+            }
+            parser->m_groupConnector = new_connector;
           }
-          groupConnector = temp;
+
           if (dtd->scaffIndex) {
-            int *temp = (int *)REALLOC(dtd->scaffIndex,
-                          groupSize * sizeof(int));
-            if (temp == NULL)
+            int *const new_scaff_index = (int *)REALLOC(
+                parser, dtd->scaffIndex, parser->m_groupSize * sizeof(int));
+            if (new_scaff_index == NULL)
               return XML_ERROR_NO_MEMORY;
-            dtd->scaffIndex = temp;
+            dtd->scaffIndex = new_scaff_index;
           }
-        }
-        else {
-          groupConnector = (char *)MALLOC(groupSize = 32);
-          if (!groupConnector) {
-            groupSize = 0;
+        } else {
+          parser->m_groupConnector
+              = (char *)MALLOC(parser, parser->m_groupSize = 32);
+          if (! parser->m_groupConnector) {
+            parser->m_groupSize = 0;
             return XML_ERROR_NO_MEMORY;
           }
         }
       }
-      groupConnector[prologState.level] = 0;
+      parser->m_groupConnector[parser->m_prologState.level] = 0;
       if (dtd->in_eldecl) {
         int myindex = nextScaffoldPart(parser);
         if (myindex < 0)
           return XML_ERROR_NO_MEMORY;
+        assert(dtd->scaffIndex != NULL);
         dtd->scaffIndex[dtd->scaffLevel] = myindex;
         dtd->scaffLevel++;
         dtd->scaffold[myindex].type = XML_CTYPE_SEQ;
-        if (elementDeclHandler)
+        if (parser->m_elementDeclHandler)
           handleDefault = XML_FALSE;
       }
       break;
     case XML_ROLE_GROUP_SEQUENCE:
-      if (groupConnector[prologState.level] == ASCII_PIPE)
+      if (parser->m_groupConnector[parser->m_prologState.level] == ASCII_PIPE)
         return XML_ERROR_SYNTAX;
-      groupConnector[prologState.level] = ASCII_COMMA;
-      if (dtd->in_eldecl && elementDeclHandler)
+      parser->m_groupConnector[parser->m_prologState.level] = ASCII_COMMA;
+      if (dtd->in_eldecl && parser->m_elementDeclHandler)
         handleDefault = XML_FALSE;
       break;
     case XML_ROLE_GROUP_CHOICE:
-      if (groupConnector[prologState.level] == ASCII_COMMA)
+      if (parser->m_groupConnector[parser->m_prologState.level] == ASCII_COMMA)
         return XML_ERROR_SYNTAX;
       if (dtd->in_eldecl
-          && !groupConnector[prologState.level]
+          && ! parser->m_groupConnector[parser->m_prologState.level]
           && (dtd->scaffold[dtd->scaffIndex[dtd->scaffLevel - 1]].type
-              != XML_CTYPE_MIXED)
-          ) {
+              != XML_CTYPE_MIXED)) {
         dtd->scaffold[dtd->scaffIndex[dtd->scaffLevel - 1]].type
             = XML_CTYPE_CHOICE;
-        if (elementDeclHandler)
+        if (parser->m_elementDeclHandler)
           handleDefault = XML_FALSE;
       }
-      groupConnector[prologState.level] = ASCII_PIPE;
+      parser->m_groupConnector[parser->m_prologState.level] = ASCII_PIPE;
       break;
     case XML_ROLE_PARAM_ENTITY_REF:
 #ifdef XML_DTD
     case XML_ROLE_INNER_PARAM_ENTITY_REF:
       dtd->hasParamEntityRefs = XML_TRUE;
-      if (!paramEntityParsing)
+      if (! parser->m_paramEntityParsing)
         dtd->keepProcessing = dtd->standalone;
       else {
         const XML_Char *name;
         ENTITY *entity;
-        name = poolStoreString(&dtd->pool, enc,
-                                s + enc->minBytesPerChar,
-                                next - enc->minBytesPerChar);
-        if (!name)
+        name = poolStoreString(&dtd->pool, enc, s + enc->minBytesPerChar,
+                               next - enc->minBytesPerChar);
+        if (! name)
           return XML_ERROR_NO_MEMORY;
         entity = (ENTITY *)lookup(parser, &dtd->paramEntities, name, 0);
         poolDiscard(&dtd->pool);
@@ -5086,13 +4833,12 @@ doProlog(XML_Parser parser,
            if yes, check that the entity exists, and that it is internal,
            otherwise call the skipped entity handler
         */
-        if (prologState.documentEntity &&
-            (dtd->standalone
-             ? !openInternalEntities
-             : !dtd->hasParamEntityRefs)) {
-          if (!entity)
+        if (parser->m_prologState.documentEntity
+            && (dtd->standalone ? ! parser->m_openInternalEntities
+                                : ! dtd->hasParamEntityRefs)) {
+          if (! entity)
             return XML_ERROR_UNDEFINED_ENTITY;
-          else if (!entity->is_internal) {
+          else if (! entity->is_internal) {
             /* It's hard to exhaustively search the code to be sure,
              * but there doesn't seem to be a way of executing the
              * following line.  There are two cases:
@@ -5115,12 +4861,12 @@ doProlog(XML_Parser parser,
              */
             return XML_ERROR_ENTITY_DECLARED_IN_PE; /* LCOV_EXCL_LINE */
           }
-        }
-        else if (!entity) {
+        } else if (! entity) {
           dtd->keepProcessing = dtd->standalone;
           /* cannot report skipped entities in declarations */
-          if ((role == XML_ROLE_PARAM_ENTITY_REF) && skippedEntityHandler) {
-            skippedEntityHandler(handlerArg, name, 1);
+          if ((role == XML_ROLE_PARAM_ENTITY_REF)
+              && parser->m_skippedEntityHandler) {
+            parser->m_skippedEntityHandler(parser->m_handlerArg, name, 1);
             handleDefault = XML_FALSE;
           }
           break;
@@ -5129,50 +4875,46 @@ doProlog(XML_Parser parser,
           return XML_ERROR_RECURSIVE_ENTITY_REF;
         if (entity->textPtr) {
           enum XML_Error result;
-          XML_Bool betweenDecl =
-            (role == XML_ROLE_PARAM_ENTITY_REF ? XML_TRUE : XML_FALSE);
+          XML_Bool betweenDecl
+              = (role == XML_ROLE_PARAM_ENTITY_REF ? XML_TRUE : XML_FALSE);
           result = processInternalEntity(parser, entity, betweenDecl);
           if (result != XML_ERROR_NONE)
             return result;
           handleDefault = XML_FALSE;
           break;
         }
-        if (externalEntityRefHandler) {
+        if (parser->m_externalEntityRefHandler) {
           dtd->paramEntityRead = XML_FALSE;
           entity->open = XML_TRUE;
-          if (!externalEntityRefHandler(externalEntityRefHandlerArg,
-                                        0,
-                                        entity->base,
-                                        entity->systemId,
-                                        entity->publicId)) {
+          if (! parser->m_externalEntityRefHandler(
+                  parser->m_externalEntityRefHandlerArg, 0, entity->base,
+                  entity->systemId, entity->publicId)) {
             entity->open = XML_FALSE;
             return XML_ERROR_EXTERNAL_ENTITY_HANDLING;
           }
           entity->open = XML_FALSE;
           handleDefault = XML_FALSE;
-          if (!dtd->paramEntityRead) {
+          if (! dtd->paramEntityRead) {
             dtd->keepProcessing = dtd->standalone;
             break;
           }
-        }
-        else {
+        } else {
           dtd->keepProcessing = dtd->standalone;
           break;
         }
       }
 #endif /* XML_DTD */
-      if (!dtd->standalone &&
-          notStandaloneHandler &&
-          !notStandaloneHandler(handlerArg))
+      if (! dtd->standalone && parser->m_notStandaloneHandler
+          && ! parser->m_notStandaloneHandler(parser->m_handlerArg))
         return XML_ERROR_NOT_STANDALONE;
       break;
 
-    /* Element declaration stuff */
+      /* Element declaration stuff */
 
     case XML_ROLE_ELEMENT_NAME:
-      if (elementDeclHandler) {
-        declElementType = getElementType(parser, enc, s, next);
-        if (!declElementType)
+      if (parser->m_elementDeclHandler) {
+        parser->m_declElementType = getElementType(parser, enc, s, next);
+        if (! parser->m_declElementType)
           return XML_ERROR_NO_MEMORY;
         dtd->scaffLevel = 0;
         dtd->scaffCount = 0;
@@ -5184,19 +4926,20 @@ doProlog(XML_Parser parser,
     case XML_ROLE_CONTENT_ANY:
     case XML_ROLE_CONTENT_EMPTY:
       if (dtd->in_eldecl) {
-        if (elementDeclHandler) {
-          XML_Content * content = (XML_Content *) MALLOC(sizeof(XML_Content));
-          if (!content)
+        if (parser->m_elementDeclHandler) {
+          XML_Content *content
+              = (XML_Content *)MALLOC(parser, sizeof(XML_Content));
+          if (! content)
             return XML_ERROR_NO_MEMORY;
           content->quant = XML_CQUANT_NONE;
           content->name = NULL;
           content->numchildren = 0;
           content->children = NULL;
-          content->type = ((role == XML_ROLE_CONTENT_ANY) ?
-                           XML_CTYPE_ANY :
-                           XML_CTYPE_EMPTY);
+          content->type = ((role == XML_ROLE_CONTENT_ANY) ? XML_CTYPE_ANY
+                                                          : XML_CTYPE_EMPTY);
           *eventEndPP = s;
-          elementDeclHandler(handlerArg, declElementType->name, content);
+          parser->m_elementDeclHandler(
+              parser->m_handlerArg, parser->m_declElementType->name, content);
           handleDefault = XML_FALSE;
         }
         dtd->in_eldecl = XML_FALSE;
@@ -5207,7 +4950,7 @@ doProlog(XML_Parser parser,
       if (dtd->in_eldecl) {
         dtd->scaffold[dtd->scaffIndex[dtd->scaffLevel - 1]].type
             = XML_CTYPE_MIXED;
-        if (elementDeclHandler)
+        if (parser->m_elementDeclHandler)
           handleDefault = XML_FALSE;
       }
       break;
@@ -5228,23 +4971,23 @@ doProlog(XML_Parser parser,
         ELEMENT_TYPE *el;
         const XML_Char *name;
         int nameLen;
-        const char *nxt = (quant == XML_CQUANT_NONE
-                           ? next
-                           : next - enc->minBytesPerChar);
+        const char *nxt
+            = (quant == XML_CQUANT_NONE ? next : next - enc->minBytesPerChar);
         int myindex = nextScaffoldPart(parser);
         if (myindex < 0)
           return XML_ERROR_NO_MEMORY;
         dtd->scaffold[myindex].type = XML_CTYPE_NAME;
         dtd->scaffold[myindex].quant = quant;
         el = getElementType(parser, enc, s, nxt);
-        if (!el)
+        if (! el)
           return XML_ERROR_NO_MEMORY;
         name = el->name;
         dtd->scaffold[myindex].name = name;
         nameLen = 0;
-        for (; name[nameLen++]; );
-        dtd->contentStringLen +=  nameLen;
-        if (elementDeclHandler)
+        for (; name[nameLen++];)
+          ;
+        dtd->contentStringLen += nameLen;
+        if (parser->m_elementDeclHandler)
           handleDefault = XML_FALSE;
       }
       break;
@@ -5262,17 +5005,18 @@ doProlog(XML_Parser parser,
       quant = XML_CQUANT_PLUS;
     closeGroup:
       if (dtd->in_eldecl) {
-        if (elementDeclHandler)
+        if (parser->m_elementDeclHandler)
           handleDefault = XML_FALSE;
         dtd->scaffLevel--;
         dtd->scaffold[dtd->scaffIndex[dtd->scaffLevel]].quant = quant;
         if (dtd->scaffLevel == 0) {
-          if (!handleDefault) {
+          if (! handleDefault) {
             XML_Content *model = build_model(parser);
-            if (!model)
+            if (! model)
               return XML_ERROR_NO_MEMORY;
             *eventEndPP = s;
-            elementDeclHandler(handlerArg, declElementType->name, model);
+            parser->m_elementDeclHandler(
+                parser->m_handlerArg, parser->m_declElementType->name, model);
           }
           dtd->in_eldecl = XML_FALSE;
           dtd->contentStringLen = 0;
@@ -5282,12 +5026,12 @@ doProlog(XML_Parser parser,
       /* End element declaration stuff */
 
     case XML_ROLE_PI:
-      if (!reportProcessingInstruction(parser, enc, s, next))
+      if (! reportProcessingInstruction(parser, enc, s, next))
         return XML_ERROR_NO_MEMORY;
       handleDefault = XML_FALSE;
       break;
     case XML_ROLE_COMMENT:
-      if (!reportComment(parser, enc, s, next))
+      if (! reportComment(parser, enc, s, next))
         return XML_ERROR_NO_MEMORY;
       handleDefault = XML_FALSE;
       break;
@@ -5299,31 +5043,31 @@ doProlog(XML_Parser parser,
       }
       break;
     case XML_ROLE_DOCTYPE_NONE:
-      if (startDoctypeDeclHandler)
+      if (parser->m_startDoctypeDeclHandler)
         handleDefault = XML_FALSE;
       break;
     case XML_ROLE_ENTITY_NONE:
-      if (dtd->keepProcessing && entityDeclHandler)
+      if (dtd->keepProcessing && parser->m_entityDeclHandler)
         handleDefault = XML_FALSE;
       break;
     case XML_ROLE_NOTATION_NONE:
-      if (notationDeclHandler)
+      if (parser->m_notationDeclHandler)
         handleDefault = XML_FALSE;
       break;
     case XML_ROLE_ATTLIST_NONE:
-      if (dtd->keepProcessing && attlistDeclHandler)
+      if (dtd->keepProcessing && parser->m_attlistDeclHandler)
         handleDefault = XML_FALSE;
       break;
     case XML_ROLE_ELEMENT_NONE:
-      if (elementDeclHandler)
+      if (parser->m_elementDeclHandler)
         handleDefault = XML_FALSE;
       break;
     } /* end of big switch */
 
-    if (handleDefault && defaultHandler)
+    if (handleDefault && parser->m_defaultHandler)
       reportDefault(parser, enc, s, next);
 
-    switch (ps_parsing) {
+    switch (parser->m_parsingStatus.parsing) {
     case XML_SUSPENDED:
       *nextPtr = next;
       return XML_ERROR_NONE;
@@ -5338,23 +5082,20 @@ doProlog(XML_Parser parser,
 }
 
 static enum XML_Error PTRCALL
-epilogProcessor(XML_Parser parser,
-                const char *s,
-                const char *end,
-                const char **nextPtr)
-{
-  processor = epilogProcessor;
-  eventPtr = s;
+epilogProcessor(XML_Parser parser, const char *s, const char *end,
+                const char **nextPtr) {
+  parser->m_processor = epilogProcessor;
+  parser->m_eventPtr = s;
   for (;;) {
     const char *next = NULL;
-    int tok = XmlPrologTok(encoding, s, end, &next);
-    eventEndPtr = next;
+    int tok = XmlPrologTok(parser->m_encoding, s, end, &next);
+    parser->m_eventEndPtr = next;
     switch (tok) {
     /* report partial linebreak - it might be the last token */
     case -XML_TOK_PROLOG_S:
-      if (defaultHandler) {
-        reportDefault(parser, encoding, s, next);
-        if (ps_parsing == XML_FINISHED)
+      if (parser->m_defaultHandler) {
+        reportDefault(parser, parser->m_encoding, s, next);
+        if (parser->m_parsingStatus.parsing == XML_FINISHED)
           return XML_ERROR_ABORTED;
       }
       *nextPtr = next;
@@ -5363,28 +5104,28 @@ epilogProcessor(XML_Parser parser,
       *nextPtr = s;
       return XML_ERROR_NONE;
     case XML_TOK_PROLOG_S:
-      if (defaultHandler)
-        reportDefault(parser, encoding, s, next);
+      if (parser->m_defaultHandler)
+        reportDefault(parser, parser->m_encoding, s, next);
       break;
     case XML_TOK_PI:
-      if (!reportProcessingInstruction(parser, encoding, s, next))
+      if (! reportProcessingInstruction(parser, parser->m_encoding, s, next))
         return XML_ERROR_NO_MEMORY;
       break;
     case XML_TOK_COMMENT:
-      if (!reportComment(parser, encoding, s, next))
+      if (! reportComment(parser, parser->m_encoding, s, next))
         return XML_ERROR_NO_MEMORY;
       break;
     case XML_TOK_INVALID:
-      eventPtr = next;
+      parser->m_eventPtr = next;
       return XML_ERROR_INVALID_TOKEN;
     case XML_TOK_PARTIAL:
-      if (!ps_finalBuffer) {
+      if (! parser->m_parsingStatus.finalBuffer) {
         *nextPtr = s;
         return XML_ERROR_NONE;
       }
       return XML_ERROR_UNCLOSED_TOKEN;
     case XML_TOK_PARTIAL_CHAR:
-      if (!ps_finalBuffer) {
+      if (! parser->m_parsingStatus.finalBuffer) {
         *nextPtr = s;
         return XML_ERROR_NONE;
       }
@@ -5392,42 +5133,40 @@ epilogProcessor(XML_Parser parser,
     default:
       return XML_ERROR_JUNK_AFTER_DOC_ELEMENT;
     }
-    eventPtr = s = next;
-    switch (ps_parsing) {
+    parser->m_eventPtr = s = next;
+    switch (parser->m_parsingStatus.parsing) {
     case XML_SUSPENDED:
       *nextPtr = next;
       return XML_ERROR_NONE;
     case XML_FINISHED:
       return XML_ERROR_ABORTED;
-    default: ;
+    default:;
     }
   }
 }
 
 static enum XML_Error
-processInternalEntity(XML_Parser parser, ENTITY *entity,
-                      XML_Bool betweenDecl)
-{
+processInternalEntity(XML_Parser parser, ENTITY *entity, XML_Bool betweenDecl) {
   const char *textStart, *textEnd;
   const char *next;
   enum XML_Error result;
   OPEN_INTERNAL_ENTITY *openEntity;
 
-  if (freeInternalEntities) {
-    openEntity = freeInternalEntities;
-    freeInternalEntities = openEntity->next;
-  }
-  else {
-    openEntity = (OPEN_INTERNAL_ENTITY *)MALLOC(sizeof(OPEN_INTERNAL_ENTITY));
-    if (!openEntity)
+  if (parser->m_freeInternalEntities) {
+    openEntity = parser->m_freeInternalEntities;
+    parser->m_freeInternalEntities = openEntity->next;
+  } else {
+    openEntity
+        = (OPEN_INTERNAL_ENTITY *)MALLOC(parser, sizeof(OPEN_INTERNAL_ENTITY));
+    if (! openEntity)
       return XML_ERROR_NO_MEMORY;
   }
   entity->open = XML_TRUE;
   entity->processed = 0;
-  openEntity->next = openInternalEntities;
-  openInternalEntities = openEntity;
+  openEntity->next = parser->m_openInternalEntities;
+  parser->m_openInternalEntities = openEntity;
   openEntity->entity = entity;
-  openEntity->startTagLevel = tagLevel;
+  openEntity->startTagLevel = parser->m_tagLevel;
   openEntity->betweenDecl = betweenDecl;
   openEntity->internalEventPtr = NULL;
   openEntity->internalEventEndPtr = NULL;
@@ -5438,43 +5177,39 @@ processInternalEntity(XML_Parser parser, ENTITY *entity,
 
 #ifdef XML_DTD
   if (entity->is_param) {
-    int tok = XmlPrologTok(internalEncoding, textStart, textEnd, &next);
-    result = doProlog(parser, internalEncoding, textStart, textEnd, tok,
-                      next, &next, XML_FALSE);
-  }
-  else
+    int tok
+        = XmlPrologTok(parser->m_internalEncoding, textStart, textEnd, &next);
+    result = doProlog(parser, parser->m_internalEncoding, textStart, textEnd,
+                      tok, next, &next, XML_FALSE, XML_FALSE);
+  } else
 #endif /* XML_DTD */
-    result = doContent(parser, tagLevel, internalEncoding, textStart,
-                       textEnd, &next, XML_FALSE);
+    result = doContent(parser, parser->m_tagLevel, parser->m_internalEncoding,
+                       textStart, textEnd, &next, XML_FALSE);
 
   if (result == XML_ERROR_NONE) {
-    if (textEnd != next && ps_parsing == XML_SUSPENDED) {
+    if (textEnd != next && parser->m_parsingStatus.parsing == XML_SUSPENDED) {
       entity->processed = (int)(next - textStart);
-      processor = internalEntityProcessor;
-    }
-    else {
+      parser->m_processor = internalEntityProcessor;
+    } else {
       entity->open = XML_FALSE;
-      openInternalEntities = openEntity->next;
+      parser->m_openInternalEntities = openEntity->next;
       /* put openEntity back in list of free instances */
-      openEntity->next = freeInternalEntities;
-      freeInternalEntities = openEntity;
+      openEntity->next = parser->m_freeInternalEntities;
+      parser->m_freeInternalEntities = openEntity;
     }
   }
   return result;
 }
 
 static enum XML_Error PTRCALL
-internalEntityProcessor(XML_Parser parser,
-                        const char *s,
-                        const char *end,
-                        const char **nextPtr)
-{
+internalEntityProcessor(XML_Parser parser, const char *s, const char *end,
+                        const char **nextPtr) {
   ENTITY *entity;
   const char *textStart, *textEnd;
   const char *next;
   enum XML_Error result;
-  OPEN_INTERNAL_ENTITY *openEntity = openInternalEntities;
-  if (!openEntity)
+  OPEN_INTERNAL_ENTITY *openEntity = parser->m_openInternalEntities;
+  if (! openEntity)
     return XML_ERROR_UNEXPECTED_STATE;
 
   entity = openEntity->entity;
@@ -5485,78 +5220,75 @@ internalEntityProcessor(XML_Parser parser,
 
 #ifdef XML_DTD
   if (entity->is_param) {
-    int tok = XmlPrologTok(internalEncoding, textStart, textEnd, &next);
-    result = doProlog(parser, internalEncoding, textStart, textEnd, tok,
-                      next, &next, XML_FALSE);
-  }
-  else
+    int tok
+        = XmlPrologTok(parser->m_internalEncoding, textStart, textEnd, &next);
+    result = doProlog(parser, parser->m_internalEncoding, textStart, textEnd,
+                      tok, next, &next, XML_FALSE, XML_TRUE);
+  } else
 #endif /* XML_DTD */
-    result = doContent(parser, openEntity->startTagLevel, internalEncoding,
-                       textStart, textEnd, &next, XML_FALSE);
+    result = doContent(parser, openEntity->startTagLevel,
+                       parser->m_internalEncoding, textStart, textEnd, &next,
+                       XML_FALSE);
 
   if (result != XML_ERROR_NONE)
     return result;
-  else if (textEnd != next && ps_parsing == XML_SUSPENDED) {
+  else if (textEnd != next
+           && parser->m_parsingStatus.parsing == XML_SUSPENDED) {
     entity->processed = (int)(next - (char *)entity->textPtr);
     return result;
-  }
-  else {
+  } else {
     entity->open = XML_FALSE;
-    openInternalEntities = openEntity->next;
+    parser->m_openInternalEntities = openEntity->next;
     /* put openEntity back in list of free instances */
-    openEntity->next = freeInternalEntities;
-    freeInternalEntities = openEntity;
+    openEntity->next = parser->m_freeInternalEntities;
+    parser->m_freeInternalEntities = openEntity;
   }
 
 #ifdef XML_DTD
   if (entity->is_param) {
     int tok;
-    processor = prologProcessor;
-    tok = XmlPrologTok(encoding, s, end, &next);
-    return doProlog(parser, encoding, s, end, tok, next, nextPtr,
-                    (XML_Bool)!ps_finalBuffer);
-  }
-  else
+    parser->m_processor = prologProcessor;
+    tok = XmlPrologTok(parser->m_encoding, s, end, &next);
+    return doProlog(parser, parser->m_encoding, s, end, tok, next, nextPtr,
+                    (XML_Bool)! parser->m_parsingStatus.finalBuffer, XML_TRUE);
+  } else
 #endif /* XML_DTD */
   {
-    processor = contentProcessor;
+    parser->m_processor = contentProcessor;
     /* see externalEntityContentProcessor vs contentProcessor */
-    return doContent(parser, parentParser ? 1 : 0, encoding, s, end,
-                     nextPtr, (XML_Bool)!ps_finalBuffer);
+    return doContent(parser, parser->m_parentParser ? 1 : 0, parser->m_encoding,
+                     s, end, nextPtr,
+                     (XML_Bool)! parser->m_parsingStatus.finalBuffer);
   }
 }
 
 static enum XML_Error PTRCALL
-errorProcessor(XML_Parser parser,
-               const char *UNUSED_P(s),
-               const char *UNUSED_P(end),
-               const char **UNUSED_P(nextPtr))
-{
-  return errorCode;
+errorProcessor(XML_Parser parser, const char *s, const char *end,
+               const char **nextPtr) {
+  UNUSED_P(s);
+  UNUSED_P(end);
+  UNUSED_P(nextPtr);
+  return parser->m_errorCode;
 }
 
 static enum XML_Error
 storeAttributeValue(XML_Parser parser, const ENCODING *enc, XML_Bool isCdata,
-                    const char *ptr, const char *end,
-                    STRING_POOL *pool)
-{
-  enum XML_Error result = appendAttributeValue(parser, enc, isCdata, ptr,
-                                               end, pool);
+                    const char *ptr, const char *end, STRING_POOL *pool) {
+  enum XML_Error result
+      = appendAttributeValue(parser, enc, isCdata, ptr, end, pool);
   if (result)
     return result;
-  if (!isCdata && poolLength(pool) && poolLastChar(pool) == 0x20)
+  if (! isCdata && poolLength(pool) && poolLastChar(pool) == 0x20)
     poolChop(pool);
-  if (!poolAppendChar(pool, XML_T('\0')))
+  if (! poolAppendChar(pool, XML_T('\0')))
     return XML_ERROR_NO_MEMORY;
   return XML_ERROR_NONE;
 }
 
 static enum XML_Error
 appendAttributeValue(XML_Parser parser, const ENCODING *enc, XML_Bool isCdata,
-                     const char *ptr, const char *end,
-                     STRING_POOL *pool)
-{
-  DTD * const dtd = _dtd;  /* save one level of indirection */
+                     const char *ptr, const char *end, STRING_POOL *pool) {
+  DTD *const dtd = parser->m_dtd; /* save one level of indirection */
   for (;;) {
     const char *next;
     int tok = XmlAttributeValueTok(enc, ptr, end, &next);
@@ -5564,45 +5296,42 @@ appendAttributeValue(XML_Parser parser, const ENCODING *enc, XML_Bool isCdata,
     case XML_TOK_NONE:
       return XML_ERROR_NONE;
     case XML_TOK_INVALID:
-      if (enc == encoding)
-        eventPtr = next;
+      if (enc == parser->m_encoding)
+        parser->m_eventPtr = next;
       return XML_ERROR_INVALID_TOKEN;
     case XML_TOK_PARTIAL:
-      if (enc == encoding)
-        eventPtr = ptr;
+      if (enc == parser->m_encoding)
+        parser->m_eventPtr = ptr;
       return XML_ERROR_INVALID_TOKEN;
-    case XML_TOK_CHAR_REF:
-      {
-        XML_Char buf[XML_ENCODE_MAX];
-        int i;
-        int n = XmlCharRefNumber(enc, ptr);
-        if (n < 0) {
-          if (enc == encoding)
-            eventPtr = ptr;
-          return XML_ERROR_BAD_CHAR_REF;
-        }
-        if (!isCdata
-            && n == 0x20 /* space */
-            && (poolLength(pool) == 0 || poolLastChar(pool) == 0x20))
-          break;
-        n = XmlEncode(n, (ICHAR *)buf);
-        /* The XmlEncode() functions can never return 0 here.  That
-         * error return happens if the code point passed in is either
-         * negative or greater than or equal to 0x110000.  The
-         * XmlCharRefNumber() functions will all return a number
-         * strictly less than 0x110000 or a negative value if an error
-         * occurred.  The negative value is intercepted above, so
-         * XmlEncode() is never passed a value it might return an
-         * error for.
-         */
-        for (i = 0; i < n; i++) {
-          if (!poolAppendChar(pool, buf[i]))
-            return XML_ERROR_NO_MEMORY;
-        }
+    case XML_TOK_CHAR_REF: {
+      XML_Char buf[XML_ENCODE_MAX];
+      int i;
+      int n = XmlCharRefNumber(enc, ptr);
+      if (n < 0) {
+        if (enc == parser->m_encoding)
+          parser->m_eventPtr = ptr;
+        return XML_ERROR_BAD_CHAR_REF;
       }
-      break;
+      if (! isCdata && n == 0x20 /* space */
+          && (poolLength(pool) == 0 || poolLastChar(pool) == 0x20))
+        break;
+      n = XmlEncode(n, (ICHAR *)buf);
+      /* The XmlEncode() functions can never return 0 here.  That
+       * error return happens if the code point passed in is either
+       * negative or greater than or equal to 0x110000.  The
+       * XmlCharRefNumber() functions will all return a number
+       * strictly less than 0x110000 or a negative value if an error
+       * occurred.  The negative value is intercepted above, so
+       * XmlEncode() is never passed a value it might return an
+       * error for.
+       */
+      for (i = 0; i < n; i++) {
+        if (! poolAppendChar(pool, buf[i]))
+          return XML_ERROR_NO_MEMORY;
+      }
+    } break;
     case XML_TOK_DATA_CHARS:
-      if (!poolAppend(pool, enc, ptr, next))
+      if (! poolAppend(pool, enc, ptr, next))
         return XML_ERROR_NO_MEMORY;
       break;
     case XML_TOK_TRAILING_CR:
@@ -5610,109 +5339,103 @@ appendAttributeValue(XML_Parser parser, const ENCODING *enc, XML_Bool isCdata,
       /* fall through */
     case XML_TOK_ATTRIBUTE_VALUE_S:
     case XML_TOK_DATA_NEWLINE:
-      if (!isCdata && (poolLength(pool) == 0 || poolLastChar(pool) == 0x20))
+      if (! isCdata && (poolLength(pool) == 0 || poolLastChar(pool) == 0x20))
         break;
-      if (!poolAppendChar(pool, 0x20))
+      if (! poolAppendChar(pool, 0x20))
         return XML_ERROR_NO_MEMORY;
       break;
-    case XML_TOK_ENTITY_REF:
-      {
-        const XML_Char *name;
-        ENTITY *entity;
-        char checkEntityDecl;
-        XML_Char ch = (XML_Char) XmlPredefinedEntityName(enc,
-                                              ptr + enc->minBytesPerChar,
-                                              next - enc->minBytesPerChar);
-        if (ch) {
-          if (!poolAppendChar(pool, ch))
-                return XML_ERROR_NO_MEMORY;
-          break;
-        }
-        name = poolStoreString(&temp2Pool, enc,
-                               ptr + enc->minBytesPerChar,
-                               next - enc->minBytesPerChar);
-        if (!name)
+    case XML_TOK_ENTITY_REF: {
+      const XML_Char *name;
+      ENTITY *entity;
+      char checkEntityDecl;
+      XML_Char ch = (XML_Char)XmlPredefinedEntityName(
+          enc, ptr + enc->minBytesPerChar, next - enc->minBytesPerChar);
+      if (ch) {
+        if (! poolAppendChar(pool, ch))
           return XML_ERROR_NO_MEMORY;
-        entity = (ENTITY *)lookup(parser, &dtd->generalEntities, name, 0);
-        poolDiscard(&temp2Pool);
-        /* First, determine if a check for an existing declaration is needed;
-           if yes, check that the entity exists, and that it is internal.
-        */
-        if (pool == &dtd->pool)  /* are we called from prolog? */
-          checkEntityDecl =
+        break;
+      }
+      name = poolStoreString(&parser->m_temp2Pool, enc,
+                             ptr + enc->minBytesPerChar,
+                             next - enc->minBytesPerChar);
+      if (! name)
+        return XML_ERROR_NO_MEMORY;
+      entity = (ENTITY *)lookup(parser, &dtd->generalEntities, name, 0);
+      poolDiscard(&parser->m_temp2Pool);
+      /* First, determine if a check for an existing declaration is needed;
+         if yes, check that the entity exists, and that it is internal.
+      */
+      if (pool == &dtd->pool) /* are we called from prolog? */
+        checkEntityDecl =
 #ifdef XML_DTD
-              prologState.documentEntity &&
+            parser->m_prologState.documentEntity &&
 #endif /* XML_DTD */
-              (dtd->standalone
-               ? !openInternalEntities
-               : !dtd->hasParamEntityRefs);
-        else /* if (pool == &tempPool): we are called from content */
-          checkEntityDecl = !dtd->hasParamEntityRefs || dtd->standalone;
-        if (checkEntityDecl) {
-          if (!entity)
-            return XML_ERROR_UNDEFINED_ENTITY;
-          else if (!entity->is_internal)
-            return XML_ERROR_ENTITY_DECLARED_IN_PE;
-        }
-        else if (!entity) {
-          /* Cannot report skipped entity here - see comments on
-             skippedEntityHandler.
-          if (skippedEntityHandler)
-            skippedEntityHandler(handlerArg, name, 0);
-          */
-          /* Cannot call the default handler because this would be
-             out of sync with the call to the startElementHandler.
-          if ((pool == &tempPool) && defaultHandler)
-            reportDefault(parser, enc, ptr, next);
-          */
-          break;
-        }
-        if (entity->open) {
-          if (enc == encoding) {
-            /* It does not appear that this line can be executed.
-             *
-             * The "if (entity->open)" check catches recursive entity
-             * definitions.  In order to be called with an open
-             * entity, it must have gone through this code before and
-             * been through the recursive call to
-             * appendAttributeValue() some lines below.  That call
-             * sets the local encoding ("enc") to the parser's
-             * internal encoding (internal_utf8 or internal_utf16),
-             * which can never be the same as the principle encoding.
-             * It doesn't appear there is another code path that gets
-             * here with entity->open being TRUE.
-             *
-             * Since it is not certain that this logic is watertight,
-             * we keep the line and merely exclude it from coverage
-             * tests.
-             */
-            eventPtr = ptr; /* LCOV_EXCL_LINE */
-          }
-          return XML_ERROR_RECURSIVE_ENTITY_REF;
-        }
-        if (entity->notation) {
-          if (enc == encoding)
-            eventPtr = ptr;
-          return XML_ERROR_BINARY_ENTITY_REF;
-        }
-        if (!entity->textPtr) {
-          if (enc == encoding)
-            eventPtr = ptr;
-          return XML_ERROR_ATTRIBUTE_EXTERNAL_ENTITY_REF;
-        }
-        else {
-          enum XML_Error result;
-          const XML_Char *textEnd = entity->textPtr + entity->textLen;
-          entity->open = XML_TRUE;
-          result = appendAttributeValue(parser, internalEncoding, isCdata,
-                                        (char *)entity->textPtr,
-                                        (char *)textEnd, pool);
-          entity->open = XML_FALSE;
-          if (result)
-            return result;
+            (dtd->standalone ? ! parser->m_openInternalEntities
+                             : ! dtd->hasParamEntityRefs);
+      else /* if (pool == &parser->m_tempPool): we are called from content */
+        checkEntityDecl = ! dtd->hasParamEntityRefs || dtd->standalone;
+      if (checkEntityDecl) {
+        if (! entity)
+          return XML_ERROR_UNDEFINED_ENTITY;
+        else if (! entity->is_internal)
+          return XML_ERROR_ENTITY_DECLARED_IN_PE;
+      } else if (! entity) {
+        /* Cannot report skipped entity here - see comments on
+           parser->m_skippedEntityHandler.
+        if (parser->m_skippedEntityHandler)
+          parser->m_skippedEntityHandler(parser->m_handlerArg, name, 0);
+        */
+        /* Cannot call the default handler because this would be
+           out of sync with the call to the startElementHandler.
+        if ((pool == &parser->m_tempPool) && parser->m_defaultHandler)
+          reportDefault(parser, enc, ptr, next);
+        */
+        break;
+      }
+      if (entity->open) {
+        if (enc == parser->m_encoding) {
+          /* It does not appear that this line can be executed.
+           *
+           * The "if (entity->open)" check catches recursive entity
+           * definitions.  In order to be called with an open
+           * entity, it must have gone through this code before and
+           * been through the recursive call to
+           * appendAttributeValue() some lines below.  That call
+           * sets the local encoding ("enc") to the parser's
+           * internal encoding (internal_utf8 or internal_utf16),
+           * which can never be the same as the principle encoding.
+           * It doesn't appear there is another code path that gets
+           * here with entity->open being TRUE.
+           *
+           * Since it is not certain that this logic is watertight,
+           * we keep the line and merely exclude it from coverage
+           * tests.
+           */
+          parser->m_eventPtr = ptr; /* LCOV_EXCL_LINE */
         }
+        return XML_ERROR_RECURSIVE_ENTITY_REF;
       }
-      break;
+      if (entity->notation) {
+        if (enc == parser->m_encoding)
+          parser->m_eventPtr = ptr;
+        return XML_ERROR_BINARY_ENTITY_REF;
+      }
+      if (! entity->textPtr) {
+        if (enc == parser->m_encoding)
+          parser->m_eventPtr = ptr;
+        return XML_ERROR_ATTRIBUTE_EXTERNAL_ENTITY_REF;
+      } else {
+        enum XML_Error result;
+        const XML_Char *textEnd = entity->textPtr + entity->textLen;
+        entity->open = XML_TRUE;
+        result = appendAttributeValue(parser, parser->m_internalEncoding,
+                                      isCdata, (char *)entity->textPtr,
+                                      (char *)textEnd, pool);
+        entity->open = XML_FALSE;
+        if (result)
+          return result;
+      }
+    } break;
     default:
       /* The only token returned by XmlAttributeValueTok() that does
        * not have an explicit case here is XML_TOK_PARTIAL_CHAR.
@@ -5725,8 +5448,8 @@ appendAttributeValue(XML_Parser parser, const ENCODING *enc, XML_Bool isCdata,
        *
        * LCOV_EXCL_START
        */
-      if (enc == encoding)
-        eventPtr = ptr;
+      if (enc == parser->m_encoding)
+        parser->m_eventPtr = ptr;
       return XML_ERROR_UNEXPECTED_STATE;
       /* LCOV_EXCL_STOP */
     }
@@ -5736,23 +5459,20 @@ appendAttributeValue(XML_Parser parser, const ENCODING *enc, XML_Bool isCdata,
 }
 
 static enum XML_Error
-storeEntityValue(XML_Parser parser,
-                 const ENCODING *enc,
-                 const char *entityTextPtr,
-                 const char *entityTextEnd)
-{
-  DTD * const dtd = _dtd;  /* save one level of indirection */
+storeEntityValue(XML_Parser parser, const ENCODING *enc,
+                 const char *entityTextPtr, const char *entityTextEnd) {
+  DTD *const dtd = parser->m_dtd; /* save one level of indirection */
   STRING_POOL *pool = &(dtd->entityValuePool);
   enum XML_Error result = XML_ERROR_NONE;
 #ifdef XML_DTD
-  int oldInEntityValue = prologState.inEntityValue;
-  prologState.inEntityValue = 1;
+  int oldInEntityValue = parser->m_prologState.inEntityValue;
+  parser->m_prologState.inEntityValue = 1;
 #endif /* XML_DTD */
   /* never return Null for the value argument in EntityDeclHandler,
      since this would indicate an external entity; therefore we
      have to make sure that entityValuePool.start is not null */
-  if (!pool->blocks) {
-    if (!poolGrow(pool))
+  if (! pool->blocks) {
+    if (! poolGrow(pool))
       return XML_ERROR_NO_MEMORY;
   }
 
@@ -5762,61 +5482,55 @@ storeEntityValue(XML_Parser parser,
     switch (tok) {
     case XML_TOK_PARAM_ENTITY_REF:
 #ifdef XML_DTD
-      if (isParamEntity || enc != encoding) {
+      if (parser->m_isParamEntity || enc != parser->m_encoding) {
         const XML_Char *name;
         ENTITY *entity;
-        name = poolStoreString(&tempPool, enc,
+        name = poolStoreString(&parser->m_tempPool, enc,
                                entityTextPtr + enc->minBytesPerChar,
                                next - enc->minBytesPerChar);
-        if (!name) {
+        if (! name) {
           result = XML_ERROR_NO_MEMORY;
           goto endEntityValue;
         }
         entity = (ENTITY *)lookup(parser, &dtd->paramEntities, name, 0);
-        poolDiscard(&tempPool);
-        if (!entity) {
+        poolDiscard(&parser->m_tempPool);
+        if (! entity) {
           /* not a well-formedness error - see XML 1.0: WFC Entity Declared */
           /* cannot report skipped entity here - see comments on
-             skippedEntityHandler
-          if (skippedEntityHandler)
-            skippedEntityHandler(handlerArg, name, 0);
+             parser->m_skippedEntityHandler
+          if (parser->m_skippedEntityHandler)
+            parser->m_skippedEntityHandler(parser->m_handlerArg, name, 0);
           */
           dtd->keepProcessing = dtd->standalone;
           goto endEntityValue;
         }
         if (entity->open) {
-          if (enc == encoding)
-            eventPtr = entityTextPtr;
+          if (enc == parser->m_encoding)
+            parser->m_eventPtr = entityTextPtr;
           result = XML_ERROR_RECURSIVE_ENTITY_REF;
           goto endEntityValue;
         }
         if (entity->systemId) {
-          if (externalEntityRefHandler) {
+          if (parser->m_externalEntityRefHandler) {
             dtd->paramEntityRead = XML_FALSE;
             entity->open = XML_TRUE;
-            if (!externalEntityRefHandler(externalEntityRefHandlerArg,
-                                          0,
-                                          entity->base,
-                                          entity->systemId,
-                                          entity->publicId)) {
+            if (! parser->m_externalEntityRefHandler(
+                    parser->m_externalEntityRefHandlerArg, 0, entity->base,
+                    entity->systemId, entity->publicId)) {
               entity->open = XML_FALSE;
               result = XML_ERROR_EXTERNAL_ENTITY_HANDLING;
               goto endEntityValue;
             }
             entity->open = XML_FALSE;
-            if (!dtd->paramEntityRead)
+            if (! dtd->paramEntityRead)
               dtd->keepProcessing = dtd->standalone;
-          }
-          else
+          } else
             dtd->keepProcessing = dtd->standalone;
-        }
-        else {
+        } else {
           entity->open = XML_TRUE;
-          result = storeEntityValue(parser,
-                                    internalEncoding,
-                                    (char *)entity->textPtr,
-                                    (char *)(entity->textPtr
-                                             + entity->textLen));
+          result = storeEntityValue(
+              parser, parser->m_internalEncoding, (char *)entity->textPtr,
+              (char *)(entity->textPtr + entity->textLen));
           entity->open = XML_FALSE;
           if (result)
             goto endEntityValue;
@@ -5826,7 +5540,7 @@ storeEntityValue(XML_Parser parser,
 #endif /* XML_DTD */
       /* In the internal subset, PE references are not legal
          within markup declarations, e.g entity values in this case. */
-      eventPtr = entityTextPtr;
+      parser->m_eventPtr = entityTextPtr;
       result = XML_ERROR_PARAM_ENTITY_REF;
       goto endEntityValue;
     case XML_TOK_NONE:
@@ -5834,7 +5548,7 @@ storeEntityValue(XML_Parser parser,
       goto endEntityValue;
     case XML_TOK_ENTITY_REF:
     case XML_TOK_DATA_CHARS:
-      if (!poolAppend(pool, enc, entityTextPtr, next)) {
+      if (! poolAppend(pool, enc, entityTextPtr, next)) {
         result = XML_ERROR_NO_MEMORY;
         goto endEntityValue;
       }
@@ -5843,50 +5557,48 @@ storeEntityValue(XML_Parser parser,
       next = entityTextPtr + enc->minBytesPerChar;
       /* fall through */
     case XML_TOK_DATA_NEWLINE:
-      if (pool->end == pool->ptr && !poolGrow(pool)) {
-              result = XML_ERROR_NO_MEMORY;
+      if (pool->end == pool->ptr && ! poolGrow(pool)) {
+        result = XML_ERROR_NO_MEMORY;
         goto endEntityValue;
       }
       *(pool->ptr)++ = 0xA;
       break;
-    case XML_TOK_CHAR_REF:
-      {
-        XML_Char buf[XML_ENCODE_MAX];
-        int i;
-        int n = XmlCharRefNumber(enc, entityTextPtr);
-        if (n < 0) {
-          if (enc == encoding)
-            eventPtr = entityTextPtr;
-          result = XML_ERROR_BAD_CHAR_REF;
+    case XML_TOK_CHAR_REF: {
+      XML_Char buf[XML_ENCODE_MAX];
+      int i;
+      int n = XmlCharRefNumber(enc, entityTextPtr);
+      if (n < 0) {
+        if (enc == parser->m_encoding)
+          parser->m_eventPtr = entityTextPtr;
+        result = XML_ERROR_BAD_CHAR_REF;
+        goto endEntityValue;
+      }
+      n = XmlEncode(n, (ICHAR *)buf);
+      /* The XmlEncode() functions can never return 0 here.  That
+       * error return happens if the code point passed in is either
+       * negative or greater than or equal to 0x110000.  The
+       * XmlCharRefNumber() functions will all return a number
+       * strictly less than 0x110000 or a negative value if an error
+       * occurred.  The negative value is intercepted above, so
+       * XmlEncode() is never passed a value it might return an
+       * error for.
+       */
+      for (i = 0; i < n; i++) {
+        if (pool->end == pool->ptr && ! poolGrow(pool)) {
+          result = XML_ERROR_NO_MEMORY;
           goto endEntityValue;
         }
-        n = XmlEncode(n, (ICHAR *)buf);
-        /* The XmlEncode() functions can never return 0 here.  That
-         * error return happens if the code point passed in is either
-         * negative or greater than or equal to 0x110000.  The
-         * XmlCharRefNumber() functions will all return a number
-         * strictly less than 0x110000 or a negative value if an error
-         * occurred.  The negative value is intercepted above, so
-         * XmlEncode() is never passed a value it might return an
-         * error for.
-         */
-        for (i = 0; i < n; i++) {
-          if (pool->end == pool->ptr && !poolGrow(pool)) {
-            result = XML_ERROR_NO_MEMORY;
-            goto endEntityValue;
-          }
-          *(pool->ptr)++ = buf[i];
-        }
+        *(pool->ptr)++ = buf[i];
       }
-      break;
+    } break;
     case XML_TOK_PARTIAL:
-      if (enc == encoding)
-        eventPtr = entityTextPtr;
+      if (enc == parser->m_encoding)
+        parser->m_eventPtr = entityTextPtr;
       result = XML_ERROR_INVALID_TOKEN;
       goto endEntityValue;
     case XML_TOK_INVALID:
-      if (enc == encoding)
-        eventPtr = next;
+      if (enc == parser->m_encoding)
+        parser->m_eventPtr = next;
       result = XML_ERROR_INVALID_TOKEN;
       goto endEntityValue;
     default:
@@ -5897,8 +5609,8 @@ storeEntityValue(XML_Parser parser,
        *
        * LCOV_EXCL_START
        */
-      if (enc == encoding)
-        eventPtr = entityTextPtr;
+      if (enc == parser->m_encoding)
+        parser->m_eventPtr = entityTextPtr;
       result = XML_ERROR_UNEXPECTED_STATE;
       goto endEntityValue;
       /* LCOV_EXCL_STOP */
@@ -5907,14 +5619,13 @@ storeEntityValue(XML_Parser parser,
   }
 endEntityValue:
 #ifdef XML_DTD
-  prologState.inEntityValue = oldInEntityValue;
+  parser->m_prologState.inEntityValue = oldInEntityValue;
 #endif /* XML_DTD */
   return result;
 }
 
 static void FASTCALL
-normalizeLines(XML_Char *s)
-{
+normalizeLines(XML_Char *s) {
   XML_Char *p;
   for (;; s++) {
     if (*s == XML_T('\0'))
@@ -5928,8 +5639,7 @@ normalizeLines(XML_Char *s)
       *p++ = 0xA;
       if (*++s == 0xA)
         s++;
-    }
-    else
+    } else
       *p++ = *s++;
   } while (*s);
   *p = XML_T('\0');
@@ -5937,68 +5647,62 @@ normalizeLines(XML_Char *s)
 
 static int
 reportProcessingInstruction(XML_Parser parser, const ENCODING *enc,
-                            const char *start, const char *end)
-{
+                            const char *start, const char *end) {
   const XML_Char *target;
   XML_Char *data;
   const char *tem;
-  if (!processingInstructionHandler) {
-    if (defaultHandler)
+  if (! parser->m_processingInstructionHandler) {
+    if (parser->m_defaultHandler)
       reportDefault(parser, enc, start, end);
     return 1;
   }
   start += enc->minBytesPerChar * 2;
   tem = start + XmlNameLength(enc, start);
-  target = poolStoreString(&tempPool, enc, start, tem);
-  if (!target)
+  target = poolStoreString(&parser->m_tempPool, enc, start, tem);
+  if (! target)
     return 0;
-  poolFinish(&tempPool);
-  data = poolStoreString(&tempPool, enc,
-                        XmlSkipS(enc, tem),
-                        end - enc->minBytesPerChar*2);
-  if (!data)
+  poolFinish(&parser->m_tempPool);
+  data = poolStoreString(&parser->m_tempPool, enc, XmlSkipS(enc, tem),
+                         end - enc->minBytesPerChar * 2);
+  if (! data)
     return 0;
   normalizeLines(data);
-  processingInstructionHandler(handlerArg, target, data);
-  poolClear(&tempPool);
+  parser->m_processingInstructionHandler(parser->m_handlerArg, target, data);
+  poolClear(&parser->m_tempPool);
   return 1;
 }
 
 static int
-reportComment(XML_Parser parser, const ENCODING *enc,
-              const char *start, const char *end)
-{
+reportComment(XML_Parser parser, const ENCODING *enc, const char *start,
+              const char *end) {
   XML_Char *data;
-  if (!commentHandler) {
-    if (defaultHandler)
+  if (! parser->m_commentHandler) {
+    if (parser->m_defaultHandler)
       reportDefault(parser, enc, start, end);
     return 1;
   }
-  data = poolStoreString(&tempPool,
-                         enc,
+  data = poolStoreString(&parser->m_tempPool, enc,
                          start + enc->minBytesPerChar * 4,
                          end - enc->minBytesPerChar * 3);
-  if (!data)
+  if (! data)
     return 0;
   normalizeLines(data);
-  commentHandler(handlerArg, data);
-  poolClear(&tempPool);
+  parser->m_commentHandler(parser->m_handlerArg, data);
+  poolClear(&parser->m_tempPool);
   return 1;
 }
 
 static void
-reportDefault(XML_Parser parser, const ENCODING *enc,
-              const char *s, const char *end)
-{
+reportDefault(XML_Parser parser, const ENCODING *enc, const char *s,
+              const char *end) {
   if (MUST_CONVERT(enc, s)) {
     enum XML_Convert_Result convert_res;
     const char **eventPP;
     const char **eventEndPP;
-    if (enc == encoding) {
-      eventPP = &eventPtr;
-      eventEndPP = &eventEndPtr;
-    }
-    else {
+    if (enc == parser->m_encoding) {
+      eventPP = &parser->m_eventPtr;
+      eventEndPP = &parser->m_eventEndPtr;
+    } else {
       /* To get here, two things must be true; the parser must be
        * using a character encoding that is not the same as the
        * encoding passed in, and the encoding passed in must need
@@ -6015,27 +5719,28 @@ reportDefault(XML_Parser parser, const ENCODING *enc,
        *
        * LCOV_EXCL_START
        */
-      eventPP = &(openInternalEntities->internalEventPtr);
-      eventEndPP = &(openInternalEntities->internalEventEndPtr);
+      eventPP = &(parser->m_openInternalEntities->internalEventPtr);
+      eventEndPP = &(parser->m_openInternalEntities->internalEventEndPtr);
       /* LCOV_EXCL_STOP */
     }
     do {
-      ICHAR *dataPtr = (ICHAR *)dataBuf;
-      convert_res = XmlConvert(enc, &s, end, &dataPtr, (ICHAR *)dataBufEnd);
+      ICHAR *dataPtr = (ICHAR *)parser->m_dataBuf;
+      convert_res
+          = XmlConvert(enc, &s, end, &dataPtr, (ICHAR *)parser->m_dataBufEnd);
       *eventEndPP = s;
-      defaultHandler(handlerArg, dataBuf, (int)(dataPtr - (ICHAR *)dataBuf));
+      parser->m_defaultHandler(parser->m_handlerArg, parser->m_dataBuf,
+                               (int)(dataPtr - (ICHAR *)parser->m_dataBuf));
       *eventPP = s;
-    } while ((convert_res != XML_CONVERT_COMPLETED) && (convert_res != XML_CONVERT_INPUT_INCOMPLETE));
-  }
-  else
-    defaultHandler(handlerArg, (XML_Char *)s, (int)((XML_Char *)end - (XML_Char *)s));
+    } while ((convert_res != XML_CONVERT_COMPLETED)
+             && (convert_res != XML_CONVERT_INPUT_INCOMPLETE));
+  } else
+    parser->m_defaultHandler(parser->m_handlerArg, (XML_Char *)s,
+                             (int)((XML_Char *)end - (XML_Char *)s));
 }
 
-
 static int
 defineAttribute(ELEMENT_TYPE *type, ATTRIBUTE_ID *attId, XML_Bool isCdata,
-                XML_Bool isId, const XML_Char *value, XML_Parser parser)
-{
+                XML_Bool isId, const XML_Char *value, XML_Parser parser) {
   DEFAULT_ATTRIBUTE *att;
   if (value || isId) {
     /* The handling of default attributes gets messed up if we have
@@ -6044,22 +5749,23 @@ defineAttribute(ELEMENT_TYPE *type, ATTRIBUTE_ID *attId, XML_Bool isCdata,
     for (i = 0; i < type->nDefaultAtts; i++)
       if (attId == type->defaultAtts[i].id)
         return 1;
-    if (isId && !type->idAtt && !attId->xmlns)
+    if (isId && ! type->idAtt && ! attId->xmlns)
       type->idAtt = attId;
   }
   if (type->nDefaultAtts == type->allocDefaultAtts) {
     if (type->allocDefaultAtts == 0) {
       type->allocDefaultAtts = 8;
-      type->defaultAtts = (DEFAULT_ATTRIBUTE *)MALLOC(type->allocDefaultAtts
-                            * sizeof(DEFAULT_ATTRIBUTE));
-      if (!type->defaultAtts)
+      type->defaultAtts = (DEFAULT_ATTRIBUTE *)MALLOC(
+          parser, type->allocDefaultAtts * sizeof(DEFAULT_ATTRIBUTE));
+      if (! type->defaultAtts) {
+        type->allocDefaultAtts = 0;
         return 0;
-    }
-    else {
+      }
+    } else {
       DEFAULT_ATTRIBUTE *temp;
       int count = type->allocDefaultAtts * 2;
-      temp = (DEFAULT_ATTRIBUTE *)
-        REALLOC(type->defaultAtts, (count * sizeof(DEFAULT_ATTRIBUTE)));
+      temp = (DEFAULT_ATTRIBUTE *)REALLOC(parser, type->defaultAtts,
+                                          (count * sizeof(DEFAULT_ATTRIBUTE)));
       if (temp == NULL)
         return 0;
       type->allocDefaultAtts = count;
@@ -6070,92 +5776,89 @@ defineAttribute(ELEMENT_TYPE *type, ATTRIBUTE_ID *attId, XML_Bool isCdata,
   att->id = attId;
   att->value = value;
   att->isCdata = isCdata;
-  if (!isCdata)
+  if (! isCdata)
     attId->maybeTokenized = XML_TRUE;
   type->nDefaultAtts += 1;
   return 1;
 }
 
 static int
-setElementTypePrefix(XML_Parser parser, ELEMENT_TYPE *elementType)
-{
-  DTD * const dtd = _dtd;  /* save one level of indirection */
+setElementTypePrefix(XML_Parser parser, ELEMENT_TYPE *elementType) {
+  DTD *const dtd = parser->m_dtd; /* save one level of indirection */
   const XML_Char *name;
   for (name = elementType->name; *name; name++) {
     if (*name == XML_T(ASCII_COLON)) {
       PREFIX *prefix;
       const XML_Char *s;
       for (s = elementType->name; s != name; s++) {
-        if (!poolAppendChar(&dtd->pool, *s))
+        if (! poolAppendChar(&dtd->pool, *s))
           return 0;
       }
-      if (!poolAppendChar(&dtd->pool, XML_T('\0')))
+      if (! poolAppendChar(&dtd->pool, XML_T('\0')))
         return 0;
       prefix = (PREFIX *)lookup(parser, &dtd->prefixes, poolStart(&dtd->pool),
                                 sizeof(PREFIX));
-      if (!prefix)
+      if (! prefix)
         return 0;
       if (prefix->name == poolStart(&dtd->pool))
         poolFinish(&dtd->pool);
       else
         poolDiscard(&dtd->pool);
       elementType->prefix = prefix;
-
+      break;
     }
   }
   return 1;
 }
 
 static ATTRIBUTE_ID *
-getAttributeId(XML_Parser parser, const ENCODING *enc,
-               const char *start, const char *end)
-{
-  DTD * const dtd = _dtd;  /* save one level of indirection */
+getAttributeId(XML_Parser parser, const ENCODING *enc, const char *start,
+               const char *end) {
+  DTD *const dtd = parser->m_dtd; /* save one level of indirection */
   ATTRIBUTE_ID *id;
   const XML_Char *name;
-  if (!poolAppendChar(&dtd->pool, XML_T('\0')))
+  if (! poolAppendChar(&dtd->pool, XML_T('\0')))
     return NULL;
   name = poolStoreString(&dtd->pool, enc, start, end);
-  if (!name)
+  if (! name)
     return NULL;
   /* skip quotation mark - its storage will be re-used (like in name[-1]) */
   ++name;
-  id = (ATTRIBUTE_ID *)lookup(parser, &dtd->attributeIds, name, sizeof(ATTRIBUTE_ID));
-  if (!id)
+  id = (ATTRIBUTE_ID *)lookup(parser, &dtd->attributeIds, name,
+                              sizeof(ATTRIBUTE_ID));
+  if (! id)
     return NULL;
   if (id->name != name)
     poolDiscard(&dtd->pool);
   else {
     poolFinish(&dtd->pool);
-    if (!ns)
+    if (! parser->m_ns)
       ;
-    else if (name[0] == XML_T(ASCII_x)
-        && name[1] == XML_T(ASCII_m)
-        && name[2] == XML_T(ASCII_l)
-        && name[3] == XML_T(ASCII_n)
-        && name[4] == XML_T(ASCII_s)
-        && (name[5] == XML_T('\0') || name[5] == XML_T(ASCII_COLON))) {
+    else if (name[0] == XML_T(ASCII_x) && name[1] == XML_T(ASCII_m)
+             && name[2] == XML_T(ASCII_l) && name[3] == XML_T(ASCII_n)
+             && name[4] == XML_T(ASCII_s)
+             && (name[5] == XML_T('\0') || name[5] == XML_T(ASCII_COLON))) {
       if (name[5] == XML_T('\0'))
         id->prefix = &dtd->defaultPrefix;
       else
-        id->prefix = (PREFIX *)lookup(parser, &dtd->prefixes, name + 6, sizeof(PREFIX));
+        id->prefix = (PREFIX *)lookup(parser, &dtd->prefixes, name + 6,
+                                      sizeof(PREFIX));
       id->xmlns = XML_TRUE;
-    }
-    else {
+    } else {
       int i;
       for (i = 0; name[i]; i++) {
         /* attributes without prefix are *not* in the default namespace */
         if (name[i] == XML_T(ASCII_COLON)) {
           int j;
           for (j = 0; j < i; j++) {
-            if (!poolAppendChar(&dtd->pool, name[j]))
+            if (! poolAppendChar(&dtd->pool, name[j]))
               return NULL;
           }
-          if (!poolAppendChar(&dtd->pool, XML_T('\0')))
+          if (! poolAppendChar(&dtd->pool, XML_T('\0')))
             return NULL;
-          id->prefix = (PREFIX *)lookup(parser, &dtd->prefixes, poolStart(&dtd->pool),
-                                        sizeof(PREFIX));
-          if (!id->prefix)
+          id->prefix = (PREFIX *)lookup(parser, &dtd->prefixes,
+                                        poolStart(&dtd->pool), sizeof(PREFIX));
+          if (! id->prefix)
             return NULL;
           if (id->prefix->name == poolStart(&dtd->pool))
             poolFinish(&dtd->pool);
@@ -6172,22 +5875,22 @@ getAttributeId(XML_Parser parser, const ENCODING *enc,
 #define CONTEXT_SEP XML_T(ASCII_FF)
 
 static const XML_Char *
-getContext(XML_Parser parser)
-{
-  DTD * const dtd = _dtd;  /* save one level of indirection */
+getContext(XML_Parser parser) {
+  DTD *const dtd = parser->m_dtd; /* save one level of indirection */
   HASH_TABLE_ITER iter;
   XML_Bool needSep = XML_FALSE;
 
   if (dtd->defaultPrefix.binding) {
     int i;
     int len;
-    if (!poolAppendChar(&tempPool, XML_T(ASCII_EQUALS)))
+    if (! poolAppendChar(&parser->m_tempPool, XML_T(ASCII_EQUALS)))
       return NULL;
     len = dtd->defaultPrefix.binding->uriLen;
-    if (namespaceSeparator)
+    if (parser->m_namespaceSeparator)
       len--;
     for (i = 0; i < len; i++) {
-      if (!poolAppendChar(&tempPool, dtd->defaultPrefix.binding->uri[i])) {
+      if (! poolAppendChar(&parser->m_tempPool,
+                           dtd->defaultPrefix.binding->uri[i])) {
         /* Because of memory caching, I don't believe this line can be
          * executed.
          *
@@ -6219,9 +5922,9 @@ getContext(XML_Parser parser)
     int len;
     const XML_Char *s;
     PREFIX *prefix = (PREFIX *)hashTableIterNext(&iter);
-    if (!prefix)
+    if (! prefix)
       break;
-    if (!prefix->binding) {
+    if (! prefix->binding) {
       /* This test appears to be (justifiable) paranoia.  There does
        * not seem to be a way of injecting a prefix without a binding
        * that doesn't get errored long before this function is called.
@@ -6230,98 +5933,96 @@ getContext(XML_Parser parser)
        */
       continue; /* LCOV_EXCL_LINE */
     }
-    if (needSep && !poolAppendChar(&tempPool, CONTEXT_SEP))
+    if (needSep && ! poolAppendChar(&parser->m_tempPool, CONTEXT_SEP))
       return NULL;
     for (s = prefix->name; *s; s++)
-      if (!poolAppendChar(&tempPool, *s))
+      if (! poolAppendChar(&parser->m_tempPool, *s))
         return NULL;
-    if (!poolAppendChar(&tempPool, XML_T(ASCII_EQUALS)))
+    if (! poolAppendChar(&parser->m_tempPool, XML_T(ASCII_EQUALS)))
       return NULL;
     len = prefix->binding->uriLen;
-    if (namespaceSeparator)
+    if (parser->m_namespaceSeparator)
       len--;
     for (i = 0; i < len; i++)
-      if (!poolAppendChar(&tempPool, prefix->binding->uri[i]))
+      if (! poolAppendChar(&parser->m_tempPool, prefix->binding->uri[i]))
         return NULL;
     needSep = XML_TRUE;
   }
 
-
   hashTableIterInit(&iter, &(dtd->generalEntities));
   for (;;) {
     const XML_Char *s;
     ENTITY *e = (ENTITY *)hashTableIterNext(&iter);
-    if (!e)
+    if (! e)
       break;
-    if (!e->open)
+    if (! e->open)
       continue;
-    if (needSep && !poolAppendChar(&tempPool, CONTEXT_SEP))
+    if (needSep && ! poolAppendChar(&parser->m_tempPool, CONTEXT_SEP))
       return NULL;
     for (s = e->name; *s; s++)
-      if (!poolAppendChar(&tempPool, *s))
+      if (! poolAppendChar(&parser->m_tempPool, *s))
         return 0;
     needSep = XML_TRUE;
   }
 
-  if (!poolAppendChar(&tempPool, XML_T('\0')))
+  if (! poolAppendChar(&parser->m_tempPool, XML_T('\0')))
     return NULL;
-  return tempPool.start;
+  return parser->m_tempPool.start;
 }
 
 static XML_Bool
-setContext(XML_Parser parser, const XML_Char *context)
-{
-  DTD * const dtd = _dtd;  /* save one level of indirection */
+setContext(XML_Parser parser, const XML_Char *context) {
+  DTD *const dtd = parser->m_dtd; /* save one level of indirection */
   const XML_Char *s = context;
 
   while (*context != XML_T('\0')) {
     if (*s == CONTEXT_SEP || *s == XML_T('\0')) {
       ENTITY *e;
-      if (!poolAppendChar(&tempPool, XML_T('\0')))
+      if (! poolAppendChar(&parser->m_tempPool, XML_T('\0')))
         return XML_FALSE;
-      e = (ENTITY *)lookup(parser, &dtd->generalEntities, poolStart(&tempPool), 0);
+      e = (ENTITY *)lookup(parser, &dtd->generalEntities,
+                           poolStart(&parser->m_tempPool), 0);
       if (e)
         e->open = XML_TRUE;
       if (*s != XML_T('\0'))
         s++;
       context = s;
-      poolDiscard(&tempPool);
-    }
-    else if (*s == XML_T(ASCII_EQUALS)) {
+      poolDiscard(&parser->m_tempPool);
+    } else if (*s == XML_T(ASCII_EQUALS)) {
       PREFIX *prefix;
-      if (poolLength(&tempPool) == 0)
+      if (poolLength(&parser->m_tempPool) == 0)
         prefix = &dtd->defaultPrefix;
       else {
-        if (!poolAppendChar(&tempPool, XML_T('\0')))
+        if (! poolAppendChar(&parser->m_tempPool, XML_T('\0')))
           return XML_FALSE;
-        prefix = (PREFIX *)lookup(parser, &dtd->prefixes, poolStart(&tempPool),
-                                  sizeof(PREFIX));
-        if (!prefix)
+        prefix
+            = (PREFIX *)lookup(parser, &dtd->prefixes,
+                               poolStart(&parser->m_tempPool), sizeof(PREFIX));
+        if (! prefix)
           return XML_FALSE;
-        if (prefix->name == poolStart(&tempPool)) {
+        if (prefix->name == poolStart(&parser->m_tempPool)) {
           prefix->name = poolCopyString(&dtd->pool, prefix->name);
-          if (!prefix->name)
+          if (! prefix->name)
             return XML_FALSE;
         }
-        poolDiscard(&tempPool);
+        poolDiscard(&parser->m_tempPool);
       }
-      for (context = s + 1;
-           *context != CONTEXT_SEP && *context != XML_T('\0');
+      for (context = s + 1; *context != CONTEXT_SEP && *context != XML_T('\0');
            context++)
-        if (!poolAppendChar(&tempPool, *context))
+        if (! poolAppendChar(&parser->m_tempPool, *context))
           return XML_FALSE;
-      if (!poolAppendChar(&tempPool, XML_T('\0')))
+      if (! poolAppendChar(&parser->m_tempPool, XML_T('\0')))
         return XML_FALSE;
-      if (addBinding(parser, prefix, NULL, poolStart(&tempPool),
-                     &inheritedBindings) != XML_ERROR_NONE)
+      if (addBinding(parser, prefix, NULL, poolStart(&parser->m_tempPool),
+                     &parser->m_inheritedBindings)
+          != XML_ERROR_NONE)
         return XML_FALSE;
-      poolDiscard(&tempPool);
+      poolDiscard(&parser->m_tempPool);
       if (*context != XML_T('\0'))
         ++context;
       s = context;
-    }
-    else {
-      if (!poolAppendChar(&tempPool, *s))
+    } else {
+      if (! poolAppendChar(&parser->m_tempPool, *s))
         return XML_FALSE;
       s++;
     }
@@ -6330,8 +6031,7 @@ setContext(XML_Parser parser, const XML_Char *context)
 }
 
 static void FASTCALL
-normalizePublicId(XML_Char *publicId)
-{
+normalizePublicId(XML_Char *publicId) {
   XML_Char *p = publicId;
   XML_Char *s;
   for (s = publicId; *s; s++) {
@@ -6352,8 +6052,7 @@ normalizePublicId(XML_Char *publicId)
 }
 
 static DTD *
-dtdCreate(const XML_Memory_Handling_Suite *ms)
-{
+dtdCreate(const XML_Memory_Handling_Suite *ms) {
   DTD *p = (DTD *)ms->malloc_fcn(sizeof(DTD));
   if (p == NULL)
     return p;
@@ -6385,13 +6084,12 @@ dtdCreate(const XML_Memory_Handling_Suite *ms)
 }
 
 static void
-dtdReset(DTD *p, const XML_Memory_Handling_Suite *ms)
-{
+dtdReset(DTD *p, const XML_Memory_Handling_Suite *ms) {
   HASH_TABLE_ITER iter;
   hashTableIterInit(&iter, &(p->elementTypes));
   for (;;) {
     ELEMENT_TYPE *e = (ELEMENT_TYPE *)hashTableIterNext(&iter);
-    if (!e)
+    if (! e)
       break;
     if (e->allocDefaultAtts != 0)
       ms->free_fcn(e->defaultAtts);
@@ -6427,13 +6125,12 @@ dtdReset(DTD *p, const XML_Memory_Handling_Suite *ms)
 }
 
 static void
-dtdDestroy(DTD *p, XML_Bool isDocEntity, const XML_Memory_Handling_Suite *ms)
-{
+dtdDestroy(DTD *p, XML_Bool isDocEntity, const XML_Memory_Handling_Suite *ms) {
   HASH_TABLE_ITER iter;
   hashTableIterInit(&iter, &(p->elementTypes));
   for (;;) {
     ELEMENT_TYPE *e = (ELEMENT_TYPE *)hashTableIterNext(&iter);
-    if (!e)
+    if (! e)
       break;
     if (e->allocDefaultAtts != 0)
       ms->free_fcn(e->defaultAtts);
@@ -6458,8 +6155,8 @@ dtdDestroy(DTD *p, XML_Bool isDocEntity, const XML_Memory_Handling_Suite *ms)
    The new DTD has already been initialized.
 */
 static int
-dtdCopy(XML_Parser oldParser, DTD *newDtd, const DTD *oldDtd, const XML_Memory_Handling_Suite *ms)
-{
+dtdCopy(XML_Parser oldParser, DTD *newDtd, const DTD *oldDtd,
+        const XML_Memory_Handling_Suite *ms) {
   HASH_TABLE_ITER iter;
 
   /* Copy the prefix table. */
@@ -6468,12 +6165,12 @@ dtdCopy(XML_Parser oldParser, DTD *newDtd, const DTD *oldDtd, const XML_Memory_H
   for (;;) {
     const XML_Char *name;
     const PREFIX *oldP = (PREFIX *)hashTableIterNext(&iter);
-    if (!oldP)
+    if (! oldP)
       break;
     name = poolCopyString(&(newDtd->pool), oldP->name);
-    if (!name)
+    if (! name)
       return 0;
-    if (!lookup(oldParser, &(newDtd->prefixes), name, sizeof(PREFIX)))
+    if (! lookup(oldParser, &(newDtd->prefixes), name, sizeof(PREFIX)))
       return 0;
   }
 
@@ -6486,18 +6183,18 @@ dtdCopy(XML_Parser oldParser, DTD *newDtd, const DTD *oldDtd, const XML_Memory_H
     const XML_Char *name;
     const ATTRIBUTE_ID *oldA = (ATTRIBUTE_ID *)hashTableIterNext(&iter);
 
-    if (!oldA)
+    if (! oldA)
       break;
     /* Remember to allocate the scratch byte before the name. */
-    if (!poolAppendChar(&(newDtd->pool), XML_T('\0')))
+    if (! poolAppendChar(&(newDtd->pool), XML_T('\0')))
       return 0;
     name = poolCopyString(&(newDtd->pool), oldA->name);
-    if (!name)
+    if (! name)
       return 0;
     ++name;
     newA = (ATTRIBUTE_ID *)lookup(oldParser, &(newDtd->attributeIds), name,
                                   sizeof(ATTRIBUTE_ID));
-    if (!newA)
+    if (! newA)
       return 0;
     newA->maybeTokenized = oldA->maybeTokenized;
     if (oldA->prefix) {
@@ -6519,57 +6216,52 @@ dtdCopy(XML_Parser oldParser, DTD *newDtd, const DTD *oldDtd, const XML_Memory_H
     ELEMENT_TYPE *newE;
     const XML_Char *name;
     const ELEMENT_TYPE *oldE = (ELEMENT_TYPE *)hashTableIterNext(&iter);
-    if (!oldE)
+    if (! oldE)
       break;
     name = poolCopyString(&(newDtd->pool), oldE->name);
-    if (!name)
+    if (! name)
       return 0;
     newE = (ELEMENT_TYPE *)lookup(oldParser, &(newDtd->elementTypes), name,
                                   sizeof(ELEMENT_TYPE));
-    if (!newE)
+    if (! newE)
       return 0;
     if (oldE->nDefaultAtts) {
-      newE->defaultAtts = (DEFAULT_ATTRIBUTE *)
-          ms->malloc_fcn(oldE->nDefaultAtts * sizeof(DEFAULT_ATTRIBUTE));
-      if (!newE->defaultAtts) {
+      newE->defaultAtts = (DEFAULT_ATTRIBUTE *)ms->malloc_fcn(
+          oldE->nDefaultAtts * sizeof(DEFAULT_ATTRIBUTE));
+      if (! newE->defaultAtts) {
         return 0;
       }
     }
     if (oldE->idAtt)
-      newE->idAtt = (ATTRIBUTE_ID *)
-          lookup(oldParser, &(newDtd->attributeIds), oldE->idAtt->name, 0);
+      newE->idAtt = (ATTRIBUTE_ID *)lookup(oldParser, &(newDtd->attributeIds),
+                                           oldE->idAtt->name, 0);
     newE->allocDefaultAtts = newE->nDefaultAtts = oldE->nDefaultAtts;
     if (oldE->prefix)
       newE->prefix = (PREFIX *)lookup(oldParser, &(newDtd->prefixes),
                                       oldE->prefix->name, 0);
     for (i = 0; i < newE->nDefaultAtts; i++) {
-      newE->defaultAtts[i].id = (ATTRIBUTE_ID *)
-          lookup(oldParser, &(newDtd->attributeIds), oldE->defaultAtts[i].id->name, 0);
+      newE->defaultAtts[i].id = (ATTRIBUTE_ID *)lookup(
+          oldParser, &(newDtd->attributeIds), oldE->defaultAtts[i].id->name, 0);
       newE->defaultAtts[i].isCdata = oldE->defaultAtts[i].isCdata;
       if (oldE->defaultAtts[i].value) {
         newE->defaultAtts[i].value
             = poolCopyString(&(newDtd->pool), oldE->defaultAtts[i].value);
-        if (!newE->defaultAtts[i].value)
+        if (! newE->defaultAtts[i].value)
           return 0;
-      }
-      else
+      } else
         newE->defaultAtts[i].value = NULL;
     }
   }
 
   /* Copy the entity tables. */
-  if (!copyEntityTable(oldParser,
-                       &(newDtd->generalEntities),
-                       &(newDtd->pool),
-                       &(oldDtd->generalEntities)))
-      return 0;
+  if (! copyEntityTable(oldParser, &(newDtd->generalEntities), &(newDtd->pool),
+                        &(oldDtd->generalEntities)))
+    return 0;
 
 #ifdef XML_DTD
-  if (!copyEntityTable(oldParser,
-                       &(newDtd->paramEntities),
-                       &(newDtd->pool),
-                       &(oldDtd->paramEntities)))
-      return 0;
+  if (! copyEntityTable(oldParser, &(newDtd->paramEntities), &(newDtd->pool),
+                        &(oldDtd->paramEntities)))
+    return 0;
   newDtd->paramEntityRead = oldDtd->paramEntityRead;
 #endif /* XML_DTD */
 
@@ -6586,14 +6278,11 @@ dtdCopy(XML_Parser oldParser, DTD *newDtd, const DTD *oldDtd, const XML_Memory_H
   newDtd->scaffIndex = oldDtd->scaffIndex;
 
   return 1;
-}  /* End dtdCopy */
+} /* End dtdCopy */
 
 static int
-copyEntityTable(XML_Parser oldParser,
-                HASH_TABLE *newTable,
-                STRING_POOL *newPool,
-                const HASH_TABLE *oldTable)
-{
+copyEntityTable(XML_Parser oldParser, HASH_TABLE *newTable,
+                STRING_POOL *newPool, const HASH_TABLE *oldTable) {
   HASH_TABLE_ITER iter;
   const XML_Char *cachedOldBase = NULL;
   const XML_Char *cachedNewBase = NULL;
@@ -6604,17 +6293,17 @@ copyEntityTable(XML_Parser oldParser,
     ENTITY *newE;
     const XML_Char *name;
     const ENTITY *oldE = (ENTITY *)hashTableIterNext(&iter);
-    if (!oldE)
+    if (! oldE)
       break;
     name = poolCopyString(newPool, oldE->name);
-    if (!name)
+    if (! name)
       return 0;
     newE = (ENTITY *)lookup(oldParser, newTable, name, sizeof(ENTITY));
-    if (!newE)
+    if (! newE)
       return 0;
     if (oldE->systemId) {
       const XML_Char *tem = poolCopyString(newPool, oldE->systemId);
-      if (!tem)
+      if (! tem)
         return 0;
       newE->systemId = tem;
       if (oldE->base) {
@@ -6623,29 +6312,28 @@ copyEntityTable(XML_Parser oldParser,
         else {
           cachedOldBase = oldE->base;
           tem = poolCopyString(newPool, cachedOldBase);
-          if (!tem)
+          if (! tem)
             return 0;
           cachedNewBase = newE->base = tem;
         }
       }
       if (oldE->publicId) {
         tem = poolCopyString(newPool, oldE->publicId);
-        if (!tem)
+        if (! tem)
           return 0;
         newE->publicId = tem;
       }
-    }
-    else {
-      const XML_Char *tem = poolCopyStringN(newPool, oldE->textPtr,
-                                            oldE->textLen);
-      if (!tem)
+    } else {
+      const XML_Char *tem
+          = poolCopyStringN(newPool, oldE->textPtr, oldE->textLen);
+      if (! tem)
         return 0;
       newE->textPtr = tem;
       newE->textLen = oldE->textLen;
     }
     if (oldE->notation) {
       const XML_Char *tem = poolCopyString(newPool, oldE->notation);
-      if (!tem)
+      if (! tem)
         return 0;
       newE->notation = tem;
     }
@@ -6658,8 +6346,7 @@ copyEntityTable(XML_Parser oldParser,
 #define INIT_POWER 6
 
 static XML_Bool FASTCALL
-keyeq(KEY s1, KEY s2)
-{
+keyeq(KEY s1, KEY s2) {
   for (; *s1 == *s2; s1++, s2++)
     if (*s1 == 0)
       return XML_TRUE;
@@ -6667,26 +6354,23 @@ keyeq(KEY s1, KEY s2)
 }
 
 static size_t
-keylen(KEY s)
-{
+keylen(KEY s) {
   size_t len = 0;
-  for (; *s; s++, len++);
+  for (; *s; s++, len++)
+    ;
   return len;
 }
 
 static void
-copy_salt_to_sipkey(XML_Parser parser, struct sipkey * key)
-{
+copy_salt_to_sipkey(XML_Parser parser, struct sipkey *key) {
   key->k[0] = 0;
   key->k[1] = get_hash_secret_salt(parser);
 }
 
 static unsigned long FASTCALL
-hash(XML_Parser parser, KEY s)
-{
+hash(XML_Parser parser, KEY s) {
   struct siphash state;
   struct sipkey key;
-  (void)sip_tobin;
   (void)sip24_valid;
   copy_salt_to_sipkey(parser, &key);
   sip24_init(&state, &key);
@@ -6695,26 +6379,24 @@ hash(XML_Parser parser, KEY s)
 }
 
 static NAMED *
-lookup(XML_Parser parser, HASH_TABLE *table, KEY name, size_t createSize)
-{
+lookup(XML_Parser parser, HASH_TABLE *table, KEY name, size_t createSize) {
   size_t i;
   if (table->size == 0) {
     size_t tsize;
-    if (!createSize)
+    if (! createSize)
       return NULL;
     table->power = INIT_POWER;
     /* table->size is a power of 2 */
     table->size = (size_t)1 << INIT_POWER;
     tsize = table->size * sizeof(NAMED *);
     table->v = (NAMED **)table->mem->malloc_fcn(tsize);
-    if (!table->v) {
+    if (! table->v) {
       table->size = 0;
       return NULL;
     }
     memset(table->v, 0, tsize);
     i = hash(parser, name) & ((unsigned long)table->size - 1);
-  }
-  else {
+  } else {
     unsigned long h = hash(parser, name);
     unsigned long mask = (unsigned long)table->size - 1;
     unsigned char step = 0;
@@ -6722,11 +6404,11 @@ lookup(XML_Parser parser, HASH_TABLE *table, KEY name, size_t createSize)
     while (table->v[i]) {
       if (keyeq(name, table->v[i]->name))
         return table->v[i];
-      if (!step)
+      if (! step)
         step = PROBE_STEP(h, mask, table->power);
       i < step ? (i += table->size - step) : (i -= step);
     }
-    if (!createSize)
+    if (! createSize)
       return NULL;
 
     /* check for overflow (table is half full) */
@@ -6736,7 +6418,7 @@ lookup(XML_Parser parser, HASH_TABLE *table, KEY name, size_t createSize)
       unsigned long newMask = (unsigned long)newSize - 1;
       size_t tsize = newSize * sizeof(NAMED *);
       NAMED **newV = (NAMED **)table->mem->malloc_fcn(tsize);
-      if (!newV)
+      if (! newV)
         return NULL;
       memset(newV, 0, tsize);
       for (i = 0; i < table->size; i++)
@@ -6745,7 +6427,7 @@ lookup(XML_Parser parser, HASH_TABLE *table, KEY name, size_t createSize)
           size_t j = newHash & newMask;
           step = 0;
           while (newV[j]) {
-            if (!step)
+            if (! step)
               step = PROBE_STEP(newHash, newMask, newPower);
             j < step ? (j += newSize - step) : (j -= step);
           }
@@ -6758,14 +6440,14 @@ lookup(XML_Parser parser, HASH_TABLE *table, KEY name, size_t createSize)
       i = h & newMask;
       step = 0;
       while (table->v[i]) {
-        if (!step)
+        if (! step)
           step = PROBE_STEP(h, newMask, newPower);
         i < step ? (i += newSize - step) : (i -= step);
       }
     }
   }
   table->v[i] = (NAMED *)table->mem->malloc_fcn(createSize);
-  if (!table->v[i])
+  if (! table->v[i])
     return NULL;
   memset(table->v[i], 0, createSize);
   table->v[i]->name = name;
@@ -6774,8 +6456,7 @@ lookup(XML_Parser parser, HASH_TABLE *table, KEY name, size_t createSize)
 }
 
 static void FASTCALL
-hashTableClear(HASH_TABLE *table)
-{
+hashTableClear(HASH_TABLE *table) {
   size_t i;
   for (i = 0; i < table->size; i++) {
     table->mem->free_fcn(table->v[i]);
@@ -6785,8 +6466,7 @@ hashTableClear(HASH_TABLE *table)
 }
 
 static void FASTCALL
-hashTableDestroy(HASH_TABLE *table)
-{
+hashTableDestroy(HASH_TABLE *table) {
   size_t i;
   for (i = 0; i < table->size; i++)
     table->mem->free_fcn(table->v[i]);
@@ -6794,8 +6474,7 @@ hashTableDestroy(HASH_TABLE *table)
 }
 
 static void FASTCALL
-hashTableInit(HASH_TABLE *p, const XML_Memory_Handling_Suite *ms)
-{
+hashTableInit(HASH_TABLE *p, const XML_Memory_Handling_Suite *ms) {
   p->power = 0;
   p->size = 0;
   p->used = 0;
@@ -6804,15 +6483,13 @@ hashTableInit(HASH_TABLE *p, const XML_Memory_Handling_Suite *ms)
 }
 
 static void FASTCALL
-hashTableIterInit(HASH_TABLE_ITER *iter, const HASH_TABLE *table)
-{
+hashTableIterInit(HASH_TABLE_ITER *iter, const HASH_TABLE *table) {
   iter->p = table->v;
   iter->end = iter->p + table->size;
 }
 
-static NAMED * FASTCALL
-hashTableIterNext(HASH_TABLE_ITER *iter)
-{
+static NAMED *FASTCALL
+hashTableIterNext(HASH_TABLE_ITER *iter) {
   while (iter->p != iter->end) {
     NAMED *tem = *(iter->p)++;
     if (tem)
@@ -6822,8 +6499,7 @@ hashTableIterNext(HASH_TABLE_ITER *iter)
 }
 
 static void FASTCALL
-poolInit(STRING_POOL *pool, const XML_Memory_Handling_Suite *ms)
-{
+poolInit(STRING_POOL *pool, const XML_Memory_Handling_Suite *ms) {
   pool->blocks = NULL;
   pool->freeBlocks = NULL;
   pool->start = NULL;
@@ -6833,9 +6509,8 @@ poolInit(STRING_POOL *pool, const XML_Memory_Handling_Suite *ms)
 }
 
 static void FASTCALL
-poolClear(STRING_POOL *pool)
-{
-  if (!pool->freeBlocks)
+poolClear(STRING_POOL *pool) {
+  if (! pool->freeBlocks)
     pool->freeBlocks = pool->blocks;
   else {
     BLOCK *p = pool->blocks;
@@ -6853,8 +6528,7 @@ poolClear(STRING_POOL *pool)
 }
 
 static void FASTCALL
-poolDestroy(STRING_POOL *pool)
-{
+poolDestroy(STRING_POOL *pool) {
   BLOCK *p = pool->blocks;
   while (p) {
     BLOCK *tem = p->next;
@@ -6870,26 +6544,26 @@ poolDestroy(STRING_POOL *pool)
 }
 
 static XML_Char *
-poolAppend(STRING_POOL *pool, const ENCODING *enc,
-           const char *ptr, const char *end)
-{
-  if (!pool->ptr && !poolGrow(pool))
+poolAppend(STRING_POOL *pool, const ENCODING *enc, const char *ptr,
+           const char *end) {
+  if (! pool->ptr && ! poolGrow(pool))
     return NULL;
   for (;;) {
-    const enum XML_Convert_Result convert_res = XmlConvert(enc, &ptr, end, (ICHAR **)&(pool->ptr), (ICHAR *)pool->end);
-    if ((convert_res == XML_CONVERT_COMPLETED) || (convert_res == XML_CONVERT_INPUT_INCOMPLETE))
+    const enum XML_Convert_Result convert_res = XmlConvert(
+        enc, &ptr, end, (ICHAR **)&(pool->ptr), (ICHAR *)pool->end);
+    if ((convert_res == XML_CONVERT_COMPLETED)
+        || (convert_res == XML_CONVERT_INPUT_INCOMPLETE))
       break;
-    if (!poolGrow(pool))
+    if (! poolGrow(pool))
       return NULL;
   }
   return pool->start;
 }
 
-static const XML_Char * FASTCALL
-poolCopyString(STRING_POOL *pool, const XML_Char *s)
-{
+static const XML_Char *FASTCALL
+poolCopyString(STRING_POOL *pool, const XML_Char *s) {
   do {
-    if (!poolAppendChar(pool, *s))
+    if (! poolAppendChar(pool, *s))
       return NULL;
   } while (*s++);
   s = pool->start;
@@ -6898,9 +6572,8 @@ poolCopyString(STRING_POOL *pool, const XML_Char *s)
 }
 
 static const XML_Char *
-poolCopyStringN(STRING_POOL *pool, const XML_Char *s, int n)
-{
-  if (!pool->ptr && !poolGrow(pool)) {
+poolCopyStringN(STRING_POOL *pool, const XML_Char *s, int n) {
+  if (! pool->ptr && ! poolGrow(pool)) {
     /* The following line is unreachable given the current usage of
      * poolCopyStringN().  Currently it is called from exactly one
      * place to copy the text of a simple general entity.  By that
@@ -6915,7 +6588,7 @@ poolCopyStringN(STRING_POOL *pool, const XML_Char *s, int n)
     return NULL; /* LCOV_EXCL_LINE */
   }
   for (; n > 0; --n, s++) {
-    if (!poolAppendChar(pool, *s))
+    if (! poolAppendChar(pool, *s))
       return NULL;
   }
   s = pool->start;
@@ -6923,11 +6596,10 @@ poolCopyStringN(STRING_POOL *pool, const XML_Char *s, int n)
   return s;
 }
 
-static const XML_Char * FASTCALL
-poolAppendString(STRING_POOL *pool, const XML_Char *s)
-{
+static const XML_Char *FASTCALL
+poolAppendString(STRING_POOL *pool, const XML_Char *s) {
   while (*s) {
-    if (!poolAppendChar(pool, *s))
+    if (! poolAppendChar(pool, *s))
       return NULL;
     s++;
   }
@@ -6935,20 +6607,18 @@ poolAppendString(STRING_POOL *pool, const XML_Char *s)
 }
 
 static XML_Char *
-poolStoreString(STRING_POOL *pool, const ENCODING *enc,
-                const char *ptr, const char *end)
-{
-  if (!poolAppend(pool, enc, ptr, end))
+poolStoreString(STRING_POOL *pool, const ENCODING *enc, const char *ptr,
+                const char *end) {
+  if (! poolAppend(pool, enc, ptr, end))
     return NULL;
-  if (pool->ptr == pool->end && !poolGrow(pool))
+  if (pool->ptr == pool->end && ! poolGrow(pool))
     return NULL;
   *(pool->ptr)++ = 0;
   return pool->start;
 }
 
 static size_t
-poolBytesToAllocateFor(int blockSize)
-{
+poolBytesToAllocateFor(int blockSize) {
   /* Unprotected math would be:
   ** return offsetof(BLOCK, s) + blockSize * sizeof(XML_Char);
   **
@@ -6956,7 +6626,7 @@ poolBytesToAllocateFor(int blockSize)
   ** For a + b * c we check b * c in isolation first, so that addition of a
   ** on top has no chance of making us accept a small non-negative number
   */
-  const size_t stretch = sizeof(XML_Char);  /* can be 4 bytes */
+  const size_t stretch = sizeof(XML_Char); /* can be 4 bytes */
 
   if (blockSize <= 0)
     return 0;
@@ -6966,8 +6636,8 @@ poolBytesToAllocateFor(int blockSize)
 
   {
     const int stretchedBlockSize = blockSize * (int)stretch;
-    const int bytesToAllocate = (int)(
-        offsetof(BLOCK, s) + (unsigned)stretchedBlockSize);
+    const int bytesToAllocate
+        = (int)(offsetof(BLOCK, s) + (unsigned)stretchedBlockSize);
     if (bytesToAllocate < 0)
       return 0;
 
@@ -6976,8 +6646,7 @@ poolBytesToAllocateFor(int blockSize)
 }
 
 static XML_Bool FASTCALL
-poolGrow(STRING_POOL *pool)
-{
+poolGrow(STRING_POOL *pool) {
   if (pool->freeBlocks) {
     if (pool->start == 0) {
       pool->blocks = pool->freeBlocks;
@@ -7003,11 +6672,11 @@ poolGrow(STRING_POOL *pool)
   }
   if (pool->blocks && pool->start == pool->blocks->s) {
     BLOCK *temp;
-    int blockSize = (int)((unsigned)(pool->end - pool->start)*2U);
+    int blockSize = (int)((unsigned)(pool->end - pool->start) * 2U);
     size_t bytesToAllocate;
 
-    // NOTE: Needs to be calculated prior to calling `realloc`
-    //       to avoid dangling pointers:
+    /* NOTE: Needs to be calculated prior to calling `realloc`
+             to avoid dangling pointers: */
     const ptrdiff_t offsetInsideBlock = pool->ptr - pool->start;
 
     if (blockSize < 0) {
@@ -7024,8 +6693,8 @@ poolGrow(STRING_POOL *pool)
     if (bytesToAllocate == 0)
       return XML_FALSE;
 
-    temp = (BLOCK *)
-      pool->mem->realloc_fcn(pool->blocks, (unsigned)bytesToAllocate);
+    temp = (BLOCK *)pool->mem->realloc_fcn(pool->blocks,
+                                           (unsigned)bytesToAllocate);
     if (temp == NULL)
       return XML_FALSE;
     pool->blocks = temp;
@@ -7033,8 +6702,7 @@ poolGrow(STRING_POOL *pool)
     pool->ptr = pool->blocks->s + offsetInsideBlock;
     pool->start = pool->blocks->s;
     pool->end = pool->start + blockSize;
-  }
-  else {
+  } else {
     BLOCK *tem;
     int blockSize = (int)(pool->end - pool->start);
     size_t bytesToAllocate;
@@ -7049,7 +6717,7 @@ poolGrow(STRING_POOL *pool)
        * function).  Either way it isn't readily testable, so we
        * exclude it from the coverage statistics.
        */
-      return XML_FALSE;  /* LCOV_EXCL_LINE */
+      return XML_FALSE; /* LCOV_EXCL_LINE */
     }
 
     if (blockSize < INIT_BLOCK_SIZE)
@@ -7067,14 +6735,13 @@ poolGrow(STRING_POOL *pool)
       return XML_FALSE;
 
     tem = (BLOCK *)pool->mem->malloc_fcn(bytesToAllocate);
-    if (!tem)
+    if (! tem)
       return XML_FALSE;
     tem->size = blockSize;
     tem->next = pool->blocks;
     pool->blocks = tem;
     if (pool->ptr != pool->start)
-      memcpy(tem->s, pool->start,
-             (pool->ptr - pool->start) * sizeof(XML_Char));
+      memcpy(tem->s, pool->start, (pool->ptr - pool->start) * sizeof(XML_Char));
     pool->ptr = tem->s + (pool->ptr - pool->start);
     pool->start = tem->s;
     pool->end = tem->s + blockSize;
@@ -7083,15 +6750,14 @@ poolGrow(STRING_POOL *pool)
 }
 
 static int FASTCALL
-nextScaffoldPart(XML_Parser parser)
-{
-  DTD * const dtd = _dtd;  /* save one level of indirection */
-  CONTENT_SCAFFOLD * me;
+nextScaffoldPart(XML_Parser parser) {
+  DTD *const dtd = parser->m_dtd; /* save one level of indirection */
+  CONTENT_SCAFFOLD *me;
   int next;
 
-  if (!dtd->scaffIndex) {
-    dtd->scaffIndex = (int *)MALLOC(groupSize * sizeof(int));
-    if (!dtd->scaffIndex)
+  if (! dtd->scaffIndex) {
+    dtd->scaffIndex = (int *)MALLOC(parser, parser->m_groupSize * sizeof(int));
+    if (! dtd->scaffIndex)
       return -1;
     dtd->scaffIndex[0] = 0;
   }
@@ -7099,15 +6765,14 @@ nextScaffoldPart(XML_Parser parser)
   if (dtd->scaffCount >= dtd->scaffSize) {
     CONTENT_SCAFFOLD *temp;
     if (dtd->scaffold) {
-      temp = (CONTENT_SCAFFOLD *)
-        REALLOC(dtd->scaffold, dtd->scaffSize * 2 * sizeof(CONTENT_SCAFFOLD));
+      temp = (CONTENT_SCAFFOLD *)REALLOC(
+          parser, dtd->scaffold, dtd->scaffSize * 2 * sizeof(CONTENT_SCAFFOLD));
       if (temp == NULL)
         return -1;
       dtd->scaffSize *= 2;
-    }
-    else {
-      temp = (CONTENT_SCAFFOLD *)MALLOC(INIT_SCAFFOLD_ELEMENTS
-                                        * sizeof(CONTENT_SCAFFOLD));
+    } else {
+      temp = (CONTENT_SCAFFOLD *)MALLOC(parser, INIT_SCAFFOLD_ELEMENTS
+                                                    * sizeof(CONTENT_SCAFFOLD));
       if (temp == NULL)
         return -1;
       dtd->scaffSize = INIT_SCAFFOLD_ELEMENTS;
@@ -7117,11 +6782,12 @@ nextScaffoldPart(XML_Parser parser)
   next = dtd->scaffCount++;
   me = &dtd->scaffold[next];
   if (dtd->scaffLevel) {
-    CONTENT_SCAFFOLD *parent = &dtd->scaffold[dtd->scaffIndex[dtd->scaffLevel-1]];
+    CONTENT_SCAFFOLD *parent
+        = &dtd->scaffold[dtd->scaffIndex[dtd->scaffLevel - 1]];
     if (parent->lastchild) {
       dtd->scaffold[parent->lastchild].nextsib = next;
     }
-    if (!parent->childcnt)
+    if (! parent->childcnt)
       parent->firstchild = next;
     parent->lastchild = next;
     parent->childcnt++;
@@ -7131,13 +6797,9 @@ nextScaffoldPart(XML_Parser parser)
 }
 
 static void
-build_node(XML_Parser parser,
-           int src_node,
-           XML_Content *dest,
-           XML_Content **contpos,
-           XML_Char **strpos)
-{
-  DTD * const dtd = _dtd;  /* save one level of indirection */
+build_node(XML_Parser parser, int src_node, XML_Content *dest,
+           XML_Content **contpos, XML_Char **strpos) {
+  DTD *const dtd = parser->m_dtd; /* save one level of indirection */
   dest->type = dtd->scaffold[src_node].type;
   dest->quant = dtd->scaffold[src_node].quant;
   if (dest->type == XML_CTYPE_NAME) {
@@ -7146,21 +6808,19 @@ build_node(XML_Parser parser,
     src = dtd->scaffold[src_node].name;
     for (;;) {
       *(*strpos)++ = *src;
-      if (!*src)
+      if (! *src)
         break;
       src++;
     }
     dest->numchildren = 0;
     dest->children = NULL;
-  }
-  else {
+  } else {
     unsigned int i;
     int cn;
     dest->numchildren = dtd->scaffold[src_node].childcnt;
     dest->children = *contpos;
     *contpos += dest->numchildren;
-    for (i = 0, cn = dtd->scaffold[src_node].firstchild;
-         i < dest->numchildren;
+    for (i = 0, cn = dtd->scaffold[src_node].firstchild; i < dest->numchildren;
          i++, cn = dtd->scaffold[cn].nextsib) {
       build_node(parser, cn, &(dest->children[i]), contpos, strpos);
     }
@@ -7169,20 +6829,19 @@ build_node(XML_Parser parser,
 }
 
 static XML_Content *
-build_model (XML_Parser parser)
-{
-  DTD * const dtd = _dtd;  /* save one level of indirection */
+build_model(XML_Parser parser) {
+  DTD *const dtd = parser->m_dtd; /* save one level of indirection */
   XML_Content *ret;
   XML_Content *cpos;
-  XML_Char * str;
+  XML_Char *str;
   int allocsize = (dtd->scaffCount * sizeof(XML_Content)
                    + (dtd->contentStringLen * sizeof(XML_Char)));
 
-  ret = (XML_Content *)MALLOC(allocsize);
-  if (!ret)
+  ret = (XML_Content *)MALLOC(parser, allocsize);
+  if (! ret)
     return NULL;
 
-  str =  (XML_Char *) (&ret[dtd->scaffCount]);
+  str = (XML_Char *)(&ret[dtd->scaffCount]);
   cpos = &ret[1];
 
   build_node(parser, 0, ret, &cpos, &str);
@@ -7190,49 +6849,45 @@ build_model (XML_Parser parser)
 }
 
 static ELEMENT_TYPE *
-getElementType(XML_Parser parser,
-               const ENCODING *enc,
-               const char *ptr,
-               const char *end)
-{
-  DTD * const dtd = _dtd;  /* save one level of indirection */
+getElementType(XML_Parser parser, const ENCODING *enc, const char *ptr,
+               const char *end) {
+  DTD *const dtd = parser->m_dtd; /* save one level of indirection */
   const XML_Char *name = poolStoreString(&dtd->pool, enc, ptr, end);
   ELEMENT_TYPE *ret;
 
-  if (!name)
+  if (! name)
     return NULL;
-  ret = (ELEMENT_TYPE *) lookup(parser, &dtd->elementTypes, name, sizeof(ELEMENT_TYPE));
-  if (!ret)
+  ret = (ELEMENT_TYPE *)lookup(parser, &dtd->elementTypes, name,
+                               sizeof(ELEMENT_TYPE));
+  if (! ret)
     return NULL;
   if (ret->name != name)
     poolDiscard(&dtd->pool);
   else {
     poolFinish(&dtd->pool);
-    if (!setElementTypePrefix(parser, ret))
+    if (! setElementTypePrefix(parser, ret))
       return NULL;
   }
   return ret;
 }
 
 static XML_Char *
-copyString(const XML_Char *s,
-           const XML_Memory_Handling_Suite *memsuite)
-{
-    int charsRequired = 0;
-    XML_Char *result;
-
-    /* First determine how long the string is */
-    while (s[charsRequired] != 0) {
-      charsRequired++;
-    }
-    /* Include the terminator */
+copyString(const XML_Char *s, const XML_Memory_Handling_Suite *memsuite) {
+  int charsRequired = 0;
+  XML_Char *result;
+
+  /* First determine how long the string is */
+  while (s[charsRequired] != 0) {
     charsRequired++;
+  }
+  /* Include the terminator */
+  charsRequired++;
 
-    /* Now allocate space for the copy */
-    result = memsuite->malloc_fcn(charsRequired * sizeof(XML_Char));
-    if (result == NULL)
-        return NULL;
-    /* Copy the original into place */
-    memcpy(result, s, charsRequired * sizeof(XML_Char));
-    return result;
+  /* Now allocate space for the copy */
+  result = memsuite->malloc_fcn(charsRequired * sizeof(XML_Char));
+  if (result == NULL)
+    return NULL;
+  /* Copy the original into place */
+  memcpy(result, s, charsRequired * sizeof(XML_Char));
+  return result;
 }
diff --git a/Modules/expat/xmlrole.c b/Modules/expat/xmlrole.c
index 708507d575be8..4d3e3e86e9e86 100644
--- a/Modules/expat/xmlrole.c
+++ b/Modules/expat/xmlrole.c
@@ -33,11 +33,11 @@
 #include <stddef.h>
 
 #ifdef _WIN32
-#include "winconfig.h"
+#  include "winconfig.h"
 #else
-#ifdef HAVE_EXPAT_CONFIG_H
-#include <expat_config.h>
-#endif
+#  ifdef HAVE_EXPAT_CONFIG_H
+#    include <expat_config.h>
+#  endif
 #endif /* ndef _WIN32 */
 
 #include "expat_external.h"
@@ -52,107 +52,88 @@
 
 */
 
-static const char KW_ANY[] = {
-    ASCII_A, ASCII_N, ASCII_Y, '\0' };
-static const char KW_ATTLIST[] = {
-    ASCII_A, ASCII_T, ASCII_T, ASCII_L, ASCII_I, ASCII_S, ASCII_T, '\0' };
-static const char KW_CDATA[] = {
-    ASCII_C, ASCII_D, ASCII_A, ASCII_T, ASCII_A, '\0' };
-static const char KW_DOCTYPE[] = {
-    ASCII_D, ASCII_O, ASCII_C, ASCII_T, ASCII_Y, ASCII_P, ASCII_E, '\0' };
-static const char KW_ELEMENT[] = {
-    ASCII_E, ASCII_L, ASCII_E, ASCII_M, ASCII_E, ASCII_N, ASCII_T, '\0' };
-static const char KW_EMPTY[] = {
-    ASCII_E, ASCII_M, ASCII_P, ASCII_T, ASCII_Y, '\0' };
-static const char KW_ENTITIES[] = {
-    ASCII_E, ASCII_N, ASCII_T, ASCII_I, ASCII_T, ASCII_I, ASCII_E, ASCII_S,
-    '\0' };
-static const char KW_ENTITY[] = {
-    ASCII_E, ASCII_N, ASCII_T, ASCII_I, ASCII_T, ASCII_Y, '\0' };
-static const char KW_FIXED[] = {
-    ASCII_F, ASCII_I, ASCII_X, ASCII_E, ASCII_D, '\0' };
-static const char KW_ID[] = {
-    ASCII_I, ASCII_D, '\0' };
-static const char KW_IDREF[] = {
-    ASCII_I, ASCII_D, ASCII_R, ASCII_E, ASCII_F, '\0' };
-static const char KW_IDREFS[] = {
-    ASCII_I, ASCII_D, ASCII_R, ASCII_E, ASCII_F, ASCII_S, '\0' };
+static const char KW_ANY[] = {ASCII_A, ASCII_N, ASCII_Y, '\0'};
+static const char KW_ATTLIST[]
+    = {ASCII_A, ASCII_T, ASCII_T, ASCII_L, ASCII_I, ASCII_S, ASCII_T, '\0'};
+static const char KW_CDATA[]
+    = {ASCII_C, ASCII_D, ASCII_A, ASCII_T, ASCII_A, '\0'};
+static const char KW_DOCTYPE[]
+    = {ASCII_D, ASCII_O, ASCII_C, ASCII_T, ASCII_Y, ASCII_P, ASCII_E, '\0'};
+static const char KW_ELEMENT[]
+    = {ASCII_E, ASCII_L, ASCII_E, ASCII_M, ASCII_E, ASCII_N, ASCII_T, '\0'};
+static const char KW_EMPTY[]
+    = {ASCII_E, ASCII_M, ASCII_P, ASCII_T, ASCII_Y, '\0'};
+static const char KW_ENTITIES[] = {ASCII_E, ASCII_N, ASCII_T, ASCII_I, ASCII_T,
+                                   ASCII_I, ASCII_E, ASCII_S, '\0'};
+static const char KW_ENTITY[]
+    = {ASCII_E, ASCII_N, ASCII_T, ASCII_I, ASCII_T, ASCII_Y, '\0'};
+static const char KW_FIXED[]
+    = {ASCII_F, ASCII_I, ASCII_X, ASCII_E, ASCII_D, '\0'};
+static const char KW_ID[] = {ASCII_I, ASCII_D, '\0'};
+static const char KW_IDREF[]
+    = {ASCII_I, ASCII_D, ASCII_R, ASCII_E, ASCII_F, '\0'};
+static const char KW_IDREFS[]
+    = {ASCII_I, ASCII_D, ASCII_R, ASCII_E, ASCII_F, ASCII_S, '\0'};
 #ifdef XML_DTD
-static const char KW_IGNORE[] = {
-    ASCII_I, ASCII_G, ASCII_N, ASCII_O, ASCII_R, ASCII_E, '\0' };
+static const char KW_IGNORE[]
+    = {ASCII_I, ASCII_G, ASCII_N, ASCII_O, ASCII_R, ASCII_E, '\0'};
 #endif
-static const char KW_IMPLIED[] = {
-    ASCII_I, ASCII_M, ASCII_P, ASCII_L, ASCII_I, ASCII_E, ASCII_D, '\0' };
+static const char KW_IMPLIED[]
+    = {ASCII_I, ASCII_M, ASCII_P, ASCII_L, ASCII_I, ASCII_E, ASCII_D, '\0'};
 #ifdef XML_DTD
-static const char KW_INCLUDE[] = {
-    ASCII_I, ASCII_N, ASCII_C, ASCII_L, ASCII_U, ASCII_D, ASCII_E, '\0' };
+static const char KW_INCLUDE[]
+    = {ASCII_I, ASCII_N, ASCII_C, ASCII_L, ASCII_U, ASCII_D, ASCII_E, '\0'};
 #endif
-static const char KW_NDATA[] = {
-    ASCII_N, ASCII_D, ASCII_A, ASCII_T, ASCII_A, '\0' };
-static const char KW_NMTOKEN[] = {
-    ASCII_N, ASCII_M, ASCII_T, ASCII_O, ASCII_K, ASCII_E, ASCII_N, '\0' };
-static const char KW_NMTOKENS[] = {
-    ASCII_N, ASCII_M, ASCII_T, ASCII_O, ASCII_K, ASCII_E, ASCII_N, ASCII_S,
-    '\0' };
-static const char KW_NOTATION[] =
-    { ASCII_N, ASCII_O, ASCII_T, ASCII_A, ASCII_T, ASCII_I, ASCII_O, ASCII_N,
-      '\0' };
-static const char KW_PCDATA[] = {
-    ASCII_P, ASCII_C, ASCII_D, ASCII_A, ASCII_T, ASCII_A, '\0' };
-static const char KW_PUBLIC[] = {
-    ASCII_P, ASCII_U, ASCII_B, ASCII_L, ASCII_I, ASCII_C, '\0' };
-static const char KW_REQUIRED[] = {
-    ASCII_R, ASCII_E, ASCII_Q, ASCII_U, ASCII_I, ASCII_R, ASCII_E, ASCII_D,
-    '\0' };
-static const char KW_SYSTEM[] = {
-    ASCII_S, ASCII_Y, ASCII_S, ASCII_T, ASCII_E, ASCII_M, '\0' };
+static const char KW_NDATA[]
+    = {ASCII_N, ASCII_D, ASCII_A, ASCII_T, ASCII_A, '\0'};
+static const char KW_NMTOKEN[]
+    = {ASCII_N, ASCII_M, ASCII_T, ASCII_O, ASCII_K, ASCII_E, ASCII_N, '\0'};
+static const char KW_NMTOKENS[] = {ASCII_N, ASCII_M, ASCII_T, ASCII_O, ASCII_K,
+                                   ASCII_E, ASCII_N, ASCII_S, '\0'};
+static const char KW_NOTATION[] = {ASCII_N, ASCII_O, ASCII_T, ASCII_A, ASCII_T,
+                                   ASCII_I, ASCII_O, ASCII_N, '\0'};
+static const char KW_PCDATA[]
+    = {ASCII_P, ASCII_C, ASCII_D, ASCII_A, ASCII_T, ASCII_A, '\0'};
+static const char KW_PUBLIC[]
+    = {ASCII_P, ASCII_U, ASCII_B, ASCII_L, ASCII_I, ASCII_C, '\0'};
+static const char KW_REQUIRED[] = {ASCII_R, ASCII_E, ASCII_Q, ASCII_U, ASCII_I,
+                                   ASCII_R, ASCII_E, ASCII_D, '\0'};
+static const char KW_SYSTEM[]
+    = {ASCII_S, ASCII_Y, ASCII_S, ASCII_T, ASCII_E, ASCII_M, '\0'};
 
 #ifndef MIN_BYTES_PER_CHAR
-#define MIN_BYTES_PER_CHAR(enc) ((enc)->minBytesPerChar)
+#  define MIN_BYTES_PER_CHAR(enc) ((enc)->minBytesPerChar)
 #endif
 
 #ifdef XML_DTD
-#define setTopLevel(state) \
-  ((state)->handler = ((state)->documentEntity \
-                       ? internalSubset \
-                       : externalSubset1))
+#  define setTopLevel(state)                                                   \
+    ((state)->handler                                                          \
+     = ((state)->documentEntity ? internalSubset : externalSubset1))
 #else /* not XML_DTD */
-#define setTopLevel(state) ((state)->handler = internalSubset)
+#  define setTopLevel(state) ((state)->handler = internalSubset)
 #endif /* not XML_DTD */
 
-typedef int PTRCALL PROLOG_HANDLER(PROLOG_STATE *state,
-                                   int tok,
-                                   const char *ptr,
-                                   const char *end,
+typedef int PTRCALL PROLOG_HANDLER(PROLOG_STATE *state, int tok,
+                                   const char *ptr, const char *end,
                                    const ENCODING *enc);
 
-static PROLOG_HANDLER
-  prolog0, prolog1, prolog2,
-  doctype0, doctype1, doctype2, doctype3, doctype4, doctype5,
-  internalSubset,
-  entity0, entity1, entity2, entity3, entity4, entity5, entity6,
-  entity7, entity8, entity9, entity10,
-  notation0, notation1, notation2, notation3, notation4,
-  attlist0, attlist1, attlist2, attlist3, attlist4, attlist5, attlist6,
-  attlist7, attlist8, attlist9,
-  element0, element1, element2, element3, element4, element5, element6,
-  element7,
+static PROLOG_HANDLER prolog0, prolog1, prolog2, doctype0, doctype1, doctype2,
+    doctype3, doctype4, doctype5, internalSubset, entity0, entity1, entity2,
+    entity3, entity4, entity5, entity6, entity7, entity8, entity9, entity10,
+    notation0, notation1, notation2, notation3, notation4, attlist0, attlist1,
+    attlist2, attlist3, attlist4, attlist5, attlist6, attlist7, attlist8,
+    attlist9, element0, element1, element2, element3, element4, element5,
+    element6, element7,
 #ifdef XML_DTD
-  externalSubset0, externalSubset1,
-  condSect0, condSect1, condSect2,
+    externalSubset0, externalSubset1, condSect0, condSect1, condSect2,
 #endif /* XML_DTD */
-  declClose,
-  error;
+    declClose, error;
 
 static int FASTCALL common(PROLOG_STATE *state, int tok);
 
 static int PTRCALL
-prolog0(PROLOG_STATE *state,
-        int tok,
-        const char *ptr,
-        const char *end,
-        const ENCODING *enc)
-{
+prolog0(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+        const ENCODING *enc) {
   switch (tok) {
   case XML_TOK_PROLOG_S:
     state->handler = prolog1;
@@ -169,10 +150,8 @@ prolog0(PROLOG_STATE *state,
   case XML_TOK_BOM:
     return XML_ROLE_NONE;
   case XML_TOK_DECL_OPEN:
-    if (!XmlNameMatchesAscii(enc,
-                             ptr + 2 * MIN_BYTES_PER_CHAR(enc),
-                             end,
-                             KW_DOCTYPE))
+    if (! XmlNameMatchesAscii(enc, ptr + 2 * MIN_BYTES_PER_CHAR(enc), end,
+                              KW_DOCTYPE))
       break;
     state->handler = doctype0;
     return XML_ROLE_DOCTYPE_NONE;
@@ -184,12 +163,8 @@ prolog0(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-prolog1(PROLOG_STATE *state,
-        int tok,
-        const char *ptr,
-        const char *end,
-        const ENCODING *enc)
-{
+prolog1(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+        const ENCODING *enc) {
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_NONE;
@@ -207,10 +182,8 @@ prolog1(PROLOG_STATE *state,
      */
     return XML_ROLE_NONE; /* LCOV_EXCL_LINE */
   case XML_TOK_DECL_OPEN:
-    if (!XmlNameMatchesAscii(enc,
-                             ptr + 2 * MIN_BYTES_PER_CHAR(enc),
-                             end,
-                             KW_DOCTYPE))
+    if (! XmlNameMatchesAscii(enc, ptr + 2 * MIN_BYTES_PER_CHAR(enc), end,
+                              KW_DOCTYPE))
       break;
     state->handler = doctype0;
     return XML_ROLE_DOCTYPE_NONE;
@@ -222,12 +195,11 @@ prolog1(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-prolog2(PROLOG_STATE *state,
-        int tok,
-        const char *UNUSED_P(ptr),
-        const char *UNUSED_P(end),
-        const ENCODING *UNUSED_P(enc))
-{
+prolog2(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+        const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_NONE;
@@ -243,12 +215,11 @@ prolog2(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-doctype0(PROLOG_STATE *state,
-         int tok,
-         const char *UNUSED_P(ptr),
-         const char *UNUSED_P(end),
-         const ENCODING *UNUSED_P(enc))
-{
+doctype0(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+         const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_DOCTYPE_NONE;
@@ -261,12 +232,8 @@ doctype0(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-doctype1(PROLOG_STATE *state,
-         int tok,
-         const char *ptr,
-         const char *end,
-         const ENCODING *enc)
-{
+doctype1(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+         const ENCODING *enc) {
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_DOCTYPE_NONE;
@@ -291,12 +258,11 @@ doctype1(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-doctype2(PROLOG_STATE *state,
-         int tok,
-         const char *UNUSED_P(ptr),
-         const char *UNUSED_P(end),
-         const ENCODING *UNUSED_P(enc))
-{
+doctype2(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+         const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_DOCTYPE_NONE;
@@ -308,12 +274,11 @@ doctype2(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-doctype3(PROLOG_STATE *state,
-         int tok,
-         const char *UNUSED_P(ptr),
-         const char *UNUSED_P(end),
-         const ENCODING *UNUSED_P(enc))
-{
+doctype3(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+         const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_DOCTYPE_NONE;
@@ -325,12 +290,11 @@ doctype3(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-doctype4(PROLOG_STATE *state,
-         int tok,
-         const char *UNUSED_P(ptr),
-         const char *UNUSED_P(end),
-         const ENCODING *UNUSED_P(enc))
-{
+doctype4(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+         const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_DOCTYPE_NONE;
@@ -345,12 +309,11 @@ doctype4(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-doctype5(PROLOG_STATE *state,
-         int tok,
-         const char *UNUSED_P(ptr),
-         const char *UNUSED_P(end),
-         const ENCODING *UNUSED_P(enc))
-{
+doctype5(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+         const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_DOCTYPE_NONE;
@@ -362,40 +325,28 @@ doctype5(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-internalSubset(PROLOG_STATE *state,
-               int tok,
-               const char *ptr,
-               const char *end,
-               const ENCODING *enc)
-{
+internalSubset(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+               const ENCODING *enc) {
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_NONE;
   case XML_TOK_DECL_OPEN:
-    if (XmlNameMatchesAscii(enc,
-                            ptr + 2 * MIN_BYTES_PER_CHAR(enc),
-                            end,
+    if (XmlNameMatchesAscii(enc, ptr + 2 * MIN_BYTES_PER_CHAR(enc), end,
                             KW_ENTITY)) {
       state->handler = entity0;
       return XML_ROLE_ENTITY_NONE;
     }
-    if (XmlNameMatchesAscii(enc,
-                            ptr + 2 * MIN_BYTES_PER_CHAR(enc),
-                            end,
+    if (XmlNameMatchesAscii(enc, ptr + 2 * MIN_BYTES_PER_CHAR(enc), end,
                             KW_ATTLIST)) {
       state->handler = attlist0;
       return XML_ROLE_ATTLIST_NONE;
     }
-    if (XmlNameMatchesAscii(enc,
-                            ptr + 2 * MIN_BYTES_PER_CHAR(enc),
-                            end,
+    if (XmlNameMatchesAscii(enc, ptr + 2 * MIN_BYTES_PER_CHAR(enc), end,
                             KW_ELEMENT)) {
       state->handler = element0;
       return XML_ROLE_ELEMENT_NONE;
     }
-    if (XmlNameMatchesAscii(enc,
-                            ptr + 2 * MIN_BYTES_PER_CHAR(enc),
-                            end,
+    if (XmlNameMatchesAscii(enc, ptr + 2 * MIN_BYTES_PER_CHAR(enc), end,
                             KW_NOTATION)) {
       state->handler = notation0;
       return XML_ROLE_NOTATION_NONE;
@@ -419,12 +370,8 @@ internalSubset(PROLOG_STATE *state,
 #ifdef XML_DTD
 
 static int PTRCALL
-externalSubset0(PROLOG_STATE *state,
-                int tok,
-                const char *ptr,
-                const char *end,
-                const ENCODING *enc)
-{
+externalSubset0(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+                const ENCODING *enc) {
   state->handler = externalSubset1;
   if (tok == XML_TOK_XML_DECL)
     return XML_ROLE_TEXT_DECL;
@@ -432,12 +379,8 @@ externalSubset0(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-externalSubset1(PROLOG_STATE *state,
-                int tok,
-                const char *ptr,
-                const char *end,
-                const ENCODING *enc)
-{
+externalSubset1(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+                const ENCODING *enc) {
   switch (tok) {
   case XML_TOK_COND_SECT_OPEN:
     state->handler = condSect0;
@@ -464,12 +407,11 @@ externalSubset1(PROLOG_STATE *state,
 #endif /* XML_DTD */
 
 static int PTRCALL
-entity0(PROLOG_STATE *state,
-        int tok,
-        const char *UNUSED_P(ptr),
-        const char *UNUSED_P(end),
-        const ENCODING *UNUSED_P(enc))
-{
+entity0(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+        const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ENTITY_NONE;
@@ -484,12 +426,11 @@ entity0(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-entity1(PROLOG_STATE *state,
-        int tok,
-        const char *UNUSED_P(ptr),
-        const char *UNUSED_P(end),
-        const ENCODING *UNUSED_P(enc))
-{
+entity1(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+        const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ENTITY_NONE;
@@ -501,12 +442,8 @@ entity1(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-entity2(PROLOG_STATE *state,
-        int tok,
-        const char *ptr,
-        const char *end,
-        const ENCODING *enc)
-{
+entity2(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+        const ENCODING *enc) {
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ENTITY_NONE;
@@ -529,12 +466,11 @@ entity2(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-entity3(PROLOG_STATE *state,
-        int tok,
-        const char *UNUSED_P(ptr),
-        const char *UNUSED_P(end),
-        const ENCODING *UNUSED_P(enc))
-{
+entity3(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+        const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ENTITY_NONE;
@@ -546,12 +482,11 @@ entity3(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-entity4(PROLOG_STATE *state,
-        int tok,
-        const char *UNUSED_P(ptr),
-        const char *UNUSED_P(end),
-        const ENCODING *UNUSED_P(enc))
-{
+entity4(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+        const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ENTITY_NONE;
@@ -563,12 +498,8 @@ entity4(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-entity5(PROLOG_STATE *state,
-        int tok,
-        const char *ptr,
-        const char *end,
-        const ENCODING *enc)
-{
+entity5(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+        const ENCODING *enc) {
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ENTITY_NONE;
@@ -586,12 +517,11 @@ entity5(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-entity6(PROLOG_STATE *state,
-        int tok,
-        const char *UNUSED_P(ptr),
-        const char *UNUSED_P(end),
-        const ENCODING *UNUSED_P(enc))
-{
+entity6(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+        const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ENTITY_NONE;
@@ -604,12 +534,8 @@ entity6(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-entity7(PROLOG_STATE *state,
-        int tok,
-        const char *ptr,
-        const char *end,
-        const ENCODING *enc)
-{
+entity7(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+        const ENCODING *enc) {
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ENTITY_NONE;
@@ -632,12 +558,11 @@ entity7(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-entity8(PROLOG_STATE *state,
-        int tok,
-        const char *UNUSED_P(ptr),
-        const char *UNUSED_P(end),
-        const ENCODING *UNUSED_P(enc))
-{
+entity8(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+        const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ENTITY_NONE;
@@ -649,12 +574,11 @@ entity8(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-entity9(PROLOG_STATE *state,
-        int tok,
-        const char *UNUSED_P(ptr),
-        const char *UNUSED_P(end),
-        const ENCODING *UNUSED_P(enc))
-{
+entity9(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+        const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ENTITY_NONE;
@@ -666,12 +590,11 @@ entity9(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-entity10(PROLOG_STATE *state,
-         int tok,
-         const char *UNUSED_P(ptr),
-         const char *UNUSED_P(end),
-         const ENCODING *UNUSED_P(enc))
-{
+entity10(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+         const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ENTITY_NONE;
@@ -683,12 +606,11 @@ entity10(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-notation0(PROLOG_STATE *state,
-          int tok,
-          const char *UNUSED_P(ptr),
-          const char *UNUSED_P(end),
-          const ENCODING *UNUSED_P(enc))
-{
+notation0(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+          const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_NOTATION_NONE;
@@ -700,12 +622,8 @@ notation0(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-notation1(PROLOG_STATE *state,
-          int tok,
-          const char *ptr,
-          const char *end,
-          const ENCODING *enc)
-{
+notation1(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+          const ENCODING *enc) {
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_NOTATION_NONE;
@@ -724,12 +642,11 @@ notation1(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-notation2(PROLOG_STATE *state,
-          int tok,
-          const char *UNUSED_P(ptr),
-          const char *UNUSED_P(end),
-          const ENCODING *UNUSED_P(enc))
-{
+notation2(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+          const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_NOTATION_NONE;
@@ -741,12 +658,11 @@ notation2(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-notation3(PROLOG_STATE *state,
-          int tok,
-          const char *UNUSED_P(ptr),
-          const char *UNUSED_P(end),
-          const ENCODING *UNUSED_P(enc))
-{
+notation3(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+          const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_NOTATION_NONE;
@@ -759,12 +675,11 @@ notation3(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-notation4(PROLOG_STATE *state,
-          int tok,
-          const char *UNUSED_P(ptr),
-          const char *UNUSED_P(end),
-          const ENCODING *UNUSED_P(enc))
-{
+notation4(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+          const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_NOTATION_NONE;
@@ -780,12 +695,11 @@ notation4(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-attlist0(PROLOG_STATE *state,
-         int tok,
-         const char *UNUSED_P(ptr),
-         const char *UNUSED_P(end),
-         const ENCODING *UNUSED_P(enc))
-{
+attlist0(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+         const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ATTLIST_NONE;
@@ -798,12 +712,11 @@ attlist0(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-attlist1(PROLOG_STATE *state,
-         int tok,
-         const char *UNUSED_P(ptr),
-         const char *UNUSED_P(end),
-         const ENCODING *UNUSED_P(enc))
-{
+attlist1(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+         const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ATTLIST_NONE;
@@ -819,34 +732,23 @@ attlist1(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-attlist2(PROLOG_STATE *state,
-         int tok,
-         const char *ptr,
-         const char *end,
-         const ENCODING *enc)
-{
+attlist2(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+         const ENCODING *enc) {
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ATTLIST_NONE;
-  case XML_TOK_NAME:
-    {
-      static const char * const types[] = {
-        KW_CDATA,
-        KW_ID,
-        KW_IDREF,
-        KW_IDREFS,
-        KW_ENTITY,
-        KW_ENTITIES,
-        KW_NMTOKEN,
-        KW_NMTOKENS,
-      };
-      int i;
-      for (i = 0; i < (int)(sizeof(types)/sizeof(types[0])); i++)
-        if (XmlNameMatchesAscii(enc, ptr, end, types[i])) {
-          state->handler = attlist8;
-          return XML_ROLE_ATTRIBUTE_TYPE_CDATA + i;
-        }
-    }
+  case XML_TOK_NAME: {
+    static const char *const types[] = {
+        KW_CDATA,  KW_ID,       KW_IDREF,   KW_IDREFS,
+        KW_ENTITY, KW_ENTITIES, KW_NMTOKEN, KW_NMTOKENS,
+    };
+    int i;
+    for (i = 0; i < (int)(sizeof(types) / sizeof(types[0])); i++)
+      if (XmlNameMatchesAscii(enc, ptr, end, types[i])) {
+        state->handler = attlist8;
+        return XML_ROLE_ATTRIBUTE_TYPE_CDATA + i;
+      }
+  }
     if (XmlNameMatchesAscii(enc, ptr, end, KW_NOTATION)) {
       state->handler = attlist5;
       return XML_ROLE_ATTLIST_NONE;
@@ -860,12 +762,11 @@ attlist2(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-attlist3(PROLOG_STATE *state,
-         int tok,
-         const char *UNUSED_P(ptr),
-         const char *UNUSED_P(end),
-         const ENCODING *UNUSED_P(enc))
-{
+attlist3(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+         const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ATTLIST_NONE;
@@ -879,12 +780,11 @@ attlist3(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-attlist4(PROLOG_STATE *state,
-         int tok,
-         const char *UNUSED_P(ptr),
-         const char *UNUSED_P(end),
-         const ENCODING *UNUSED_P(enc))
-{
+attlist4(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+         const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ATTLIST_NONE;
@@ -899,12 +799,11 @@ attlist4(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-attlist5(PROLOG_STATE *state,
-         int tok,
-         const char *UNUSED_P(ptr),
-         const char *UNUSED_P(end),
-         const ENCODING *UNUSED_P(enc))
-{
+attlist5(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+         const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ATTLIST_NONE;
@@ -916,12 +815,11 @@ attlist5(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-attlist6(PROLOG_STATE *state,
-         int tok,
-         const char *UNUSED_P(ptr),
-         const char *UNUSED_P(end),
-         const ENCODING *UNUSED_P(enc))
-{
+attlist6(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+         const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ATTLIST_NONE;
@@ -933,12 +831,11 @@ attlist6(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-attlist7(PROLOG_STATE *state,
-         int tok,
-         const char *UNUSED_P(ptr),
-         const char *UNUSED_P(end),
-         const ENCODING *UNUSED_P(enc))
-{
+attlist7(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+         const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ATTLIST_NONE;
@@ -954,33 +851,23 @@ attlist7(PROLOG_STATE *state,
 
 /* default value */
 static int PTRCALL
-attlist8(PROLOG_STATE *state,
-         int tok,
-         const char *ptr,
-         const char *end,
-         const ENCODING *enc)
-{
+attlist8(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+         const ENCODING *enc) {
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ATTLIST_NONE;
   case XML_TOK_POUND_NAME:
-    if (XmlNameMatchesAscii(enc,
-                            ptr + MIN_BYTES_PER_CHAR(enc),
-                            end,
+    if (XmlNameMatchesAscii(enc, ptr + MIN_BYTES_PER_CHAR(enc), end,
                             KW_IMPLIED)) {
       state->handler = attlist1;
       return XML_ROLE_IMPLIED_ATTRIBUTE_VALUE;
     }
-    if (XmlNameMatchesAscii(enc,
-                            ptr + MIN_BYTES_PER_CHAR(enc),
-                            end,
+    if (XmlNameMatchesAscii(enc, ptr + MIN_BYTES_PER_CHAR(enc), end,
                             KW_REQUIRED)) {
       state->handler = attlist1;
       return XML_ROLE_REQUIRED_ATTRIBUTE_VALUE;
     }
-    if (XmlNameMatchesAscii(enc,
-                            ptr + MIN_BYTES_PER_CHAR(enc),
-                            end,
+    if (XmlNameMatchesAscii(enc, ptr + MIN_BYTES_PER_CHAR(enc), end,
                             KW_FIXED)) {
       state->handler = attlist9;
       return XML_ROLE_ATTLIST_NONE;
@@ -994,12 +881,11 @@ attlist8(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-attlist9(PROLOG_STATE *state,
-         int tok,
-         const char *UNUSED_P(ptr),
-         const char *UNUSED_P(end),
-         const ENCODING *UNUSED_P(enc))
-{
+attlist9(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+         const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ATTLIST_NONE;
@@ -1011,12 +897,11 @@ attlist9(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-element0(PROLOG_STATE *state,
-         int tok,
-         const char *UNUSED_P(ptr),
-         const char *UNUSED_P(end),
-         const ENCODING *UNUSED_P(enc))
-{
+element0(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+         const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ELEMENT_NONE;
@@ -1029,12 +914,8 @@ element0(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-element1(PROLOG_STATE *state,
-         int tok,
-         const char *ptr,
-         const char *end,
-         const ENCODING *enc)
-{
+element1(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+         const ENCODING *enc) {
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ELEMENT_NONE;
@@ -1059,19 +940,13 @@ element1(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-element2(PROLOG_STATE *state,
-         int tok,
-         const char *ptr,
-         const char *end,
-         const ENCODING *enc)
-{
+element2(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+         const ENCODING *enc) {
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ELEMENT_NONE;
   case XML_TOK_POUND_NAME:
-    if (XmlNameMatchesAscii(enc,
-                            ptr + MIN_BYTES_PER_CHAR(enc),
-                            end,
+    if (XmlNameMatchesAscii(enc, ptr + MIN_BYTES_PER_CHAR(enc), end,
                             KW_PCDATA)) {
       state->handler = element3;
       return XML_ROLE_CONTENT_PCDATA;
@@ -1099,12 +974,11 @@ element2(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-element3(PROLOG_STATE *state,
-         int tok,
-         const char *UNUSED_P(ptr),
-         const char *UNUSED_P(end),
-         const ENCODING *UNUSED_P(enc))
-{
+element3(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+         const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ELEMENT_NONE;
@@ -1124,12 +998,11 @@ element3(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-element4(PROLOG_STATE *state,
-         int tok,
-         const char *UNUSED_P(ptr),
-         const char *UNUSED_P(end),
-         const ENCODING *UNUSED_P(enc))
-{
+element4(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+         const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ELEMENT_NONE;
@@ -1142,12 +1015,11 @@ element4(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-element5(PROLOG_STATE *state,
-         int tok,
-         const char *UNUSED_P(ptr),
-         const char *UNUSED_P(end),
-         const ENCODING *UNUSED_P(enc))
-{
+element5(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+         const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ELEMENT_NONE;
@@ -1163,12 +1035,11 @@ element5(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-element6(PROLOG_STATE *state,
-         int tok,
-         const char *UNUSED_P(ptr),
-         const char *UNUSED_P(end),
-         const ENCODING *UNUSED_P(enc))
-{
+element6(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+         const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ELEMENT_NONE;
@@ -1193,12 +1064,11 @@ element6(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-element7(PROLOG_STATE *state,
-         int tok,
-         const char *UNUSED_P(ptr),
-         const char *UNUSED_P(end),
-         const ENCODING *UNUSED_P(enc))
-{
+element7(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+         const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_ELEMENT_NONE;
@@ -1243,12 +1113,8 @@ element7(PROLOG_STATE *state,
 #ifdef XML_DTD
 
 static int PTRCALL
-condSect0(PROLOG_STATE *state,
-          int tok,
-          const char *ptr,
-          const char *end,
-          const ENCODING *enc)
-{
+condSect0(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+          const ENCODING *enc) {
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_NONE;
@@ -1267,12 +1133,11 @@ condSect0(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-condSect1(PROLOG_STATE *state,
-          int tok,
-          const char *UNUSED_P(ptr),
-          const char *UNUSED_P(end),
-          const ENCODING *UNUSED_P(enc))
-{
+condSect1(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+          const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_NONE;
@@ -1285,12 +1150,11 @@ condSect1(PROLOG_STATE *state,
 }
 
 static int PTRCALL
-condSect2(PROLOG_STATE *state,
-          int tok,
-          const char *UNUSED_P(ptr),
-          const char *UNUSED_P(end),
-          const ENCODING *UNUSED_P(enc))
-{
+condSect2(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+          const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return XML_ROLE_NONE;
@@ -1304,12 +1168,11 @@ condSect2(PROLOG_STATE *state,
 #endif /* XML_DTD */
 
 static int PTRCALL
-declClose(PROLOG_STATE *state,
-          int tok,
-          const char *UNUSED_P(ptr),
-          const char *UNUSED_P(end),
-          const ENCODING *UNUSED_P(enc))
-{
+declClose(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+          const ENCODING *enc) {
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   switch (tok) {
   case XML_TOK_PROLOG_S:
     return state->role_none;
@@ -1341,21 +1204,21 @@ declClose(PROLOG_STATE *state,
  * LCOV_EXCL_START
  */
 static int PTRCALL
-error(PROLOG_STATE *UNUSED_P(state),
-      int UNUSED_P(tok),
-      const char *UNUSED_P(ptr),
-      const char *UNUSED_P(end),
-      const ENCODING *UNUSED_P(enc))
-{
+error(PROLOG_STATE *state, int tok, const char *ptr, const char *end,
+      const ENCODING *enc) {
+  UNUSED_P(state);
+  UNUSED_P(tok);
+  UNUSED_P(ptr);
+  UNUSED_P(end);
+  UNUSED_P(enc);
   return XML_ROLE_NONE;
 }
 /* LCOV_EXCL_STOP */
 
 static int FASTCALL
-common(PROLOG_STATE *state, int tok)
-{
+common(PROLOG_STATE *state, int tok) {
 #ifdef XML_DTD
-  if (!state->documentEntity && tok == XML_TOK_PARAM_ENTITY_REF)
+  if (! state->documentEntity && tok == XML_TOK_PARAM_ENTITY_REF)
     return XML_ROLE_INNER_PARAM_ENTITY_REF;
 #endif
   state->handler = error;
@@ -1363,8 +1226,7 @@ common(PROLOG_STATE *state, int tok)
 }
 
 void
-XmlPrologStateInit(PROLOG_STATE *state)
-{
+XmlPrologStateInit(PROLOG_STATE *state) {
   state->handler = prolog0;
 #ifdef XML_DTD
   state->documentEntity = 1;
@@ -1376,8 +1238,7 @@ XmlPrologStateInit(PROLOG_STATE *state)
 #ifdef XML_DTD
 
 void
-XmlPrologStateInitExternalEntity(PROLOG_STATE *state)
-{
+XmlPrologStateInitExternalEntity(PROLOG_STATE *state) {
   state->handler = externalSubset0;
   state->documentEntity = 0;
   state->includeLevel = 0;
diff --git a/Modules/expat/xmlrole.h b/Modules/expat/xmlrole.h
index e5f048eab55c6..036aba64fd29c 100644
--- a/Modules/expat/xmlrole.h
+++ b/Modules/expat/xmlrole.h
@@ -36,7 +36,7 @@
 #ifdef __VMS
 /*      0        1         2         3      0        1         2         3
         1234567890123456789012345678901     1234567890123456789012345678901 */
-#define XmlPrologStateInitExternalEntity    XmlPrologStateInitExternalEnt
+#  define XmlPrologStateInitExternalEntity XmlPrologStateInitExternalEnt
 #endif
 
 #include "xmltok.h"
@@ -113,11 +113,8 @@ enum {
 };
 
 typedef struct prolog_state {
-  int (PTRCALL *handler) (struct prolog_state *state,
-                          int tok,
-                          const char *ptr,
-                          const char *end,
-                          const ENCODING *enc);
+  int(PTRCALL *handler)(struct prolog_state *state, int tok, const char *ptr,
+                        const char *end, const ENCODING *enc);
   unsigned level;
   int role_none;
 #ifdef XML_DTD
@@ -132,8 +129,8 @@ void XmlPrologStateInit(PROLOG_STATE *);
 void XmlPrologStateInitExternalEntity(PROLOG_STATE *);
 #endif /* XML_DTD */
 
-#define XmlTokenRole(state, tok, ptr, end, enc) \
- (((state)->handler)(state, tok, ptr, end, enc))
+#define XmlTokenRole(state, tok, ptr, end, enc)                                \
+  (((state)->handler)(state, tok, ptr, end, enc))
 
 #ifdef __cplusplus
 }
diff --git a/Modules/expat/xmltok.c b/Modules/expat/xmltok.c
index 70663ce9d4b8b..11e9d1ccdad42 100644
--- a/Modules/expat/xmltok.c
+++ b/Modules/expat/xmltok.c
@@ -31,15 +31,23 @@
 */
 
 #include <stddef.h>
-#include <stdbool.h>
-#include <string.h>  // memcpy
+#include <string.h> /* memcpy */
 
-#ifdef _WIN32
-#include "winconfig.h"
+#if defined(_MSC_VER) && (_MSC_VER <= 1700)
+/* for vs2012/11.0/1700 and earlier Visual Studio compilers */
+#  define bool int
+#  define false 0
+#  define true 1
 #else
-#ifdef HAVE_EXPAT_CONFIG_H
-#include <expat_config.h>
+#  include <stdbool.h>
 #endif
+
+#ifdef _WIN32
+#  include "winconfig.h"
+#else
+#  ifdef HAVE_EXPAT_CONFIG_H
+#    include <expat_config.h>
+#  endif
 #endif /* ndef _WIN32 */
 
 #include "expat_external.h"
@@ -48,59 +56,49 @@
 #include "nametab.h"
 
 #ifdef XML_DTD
-#define IGNORE_SECTION_TOK_VTABLE , PREFIX(ignoreSectionTok)
+#  define IGNORE_SECTION_TOK_VTABLE , PREFIX(ignoreSectionTok)
 #else
-#define IGNORE_SECTION_TOK_VTABLE /* as nothing */
+#  define IGNORE_SECTION_TOK_VTABLE /* as nothing */
 #endif
 
-#define VTABLE1 \
-  { PREFIX(prologTok), PREFIX(contentTok), \
-    PREFIX(cdataSectionTok) IGNORE_SECTION_TOK_VTABLE }, \
-  { PREFIX(attributeValueTok), PREFIX(entityValueTok) }, \
-  PREFIX(sameName), \
-  PREFIX(nameMatchesAscii), \
-  PREFIX(nameLength), \
-  PREFIX(skipS), \
-  PREFIX(getAtts), \
-  PREFIX(charRefNumber), \
-  PREFIX(predefinedEntityName), \
-  PREFIX(updatePosition), \
-  PREFIX(isPublicId)
+#define VTABLE1                                                                \
+  {PREFIX(prologTok), PREFIX(contentTok),                                      \
+   PREFIX(cdataSectionTok) IGNORE_SECTION_TOK_VTABLE},                         \
+      {PREFIX(attributeValueTok), PREFIX(entityValueTok)},                     \
+      PREFIX(nameMatchesAscii), PREFIX(nameLength), PREFIX(skipS),             \
+      PREFIX(getAtts), PREFIX(charRefNumber), PREFIX(predefinedEntityName),    \
+      PREFIX(updatePosition), PREFIX(isPublicId)
 
 #define VTABLE VTABLE1, PREFIX(toUtf8), PREFIX(toUtf16)
 
-#define UCS2_GET_NAMING(pages, hi, lo) \
-   (namingBitmap[(pages[hi] << 3) + ((lo) >> 5)] & (1u << ((lo) & 0x1F)))
+#define UCS2_GET_NAMING(pages, hi, lo)                                         \
+  (namingBitmap[(pages[hi] << 3) + ((lo) >> 5)] & (1u << ((lo)&0x1F)))
 
 /* A 2 byte UTF-8 representation splits the characters 11 bits between
    the bottom 5 and 6 bits of the bytes.  We need 8 bits to index into
    pages, 3 bits to add to that index and 5 bits to generate the mask.
 */
-#define UTF8_GET_NAMING2(pages, byte) \
-    (namingBitmap[((pages)[(((byte)[0]) >> 2) & 7] << 3) \
-                      + ((((byte)[0]) & 3) << 1) \
-                      + ((((byte)[1]) >> 5) & 1)] \
-         & (1u << (((byte)[1]) & 0x1F)))
+#define UTF8_GET_NAMING2(pages, byte)                                          \
+  (namingBitmap[((pages)[(((byte)[0]) >> 2) & 7] << 3)                         \
+                + ((((byte)[0]) & 3) << 1) + ((((byte)[1]) >> 5) & 1)]         \
+   & (1u << (((byte)[1]) & 0x1F)))
 
 /* A 3 byte UTF-8 representation splits the characters 16 bits between
    the bottom 4, 6 and 6 bits of the bytes.  We need 8 bits to index
    into pages, 3 bits to add to that index and 5 bits to generate the
    mask.
 */
-#define UTF8_GET_NAMING3(pages, byte) \
-  (namingBitmap[((pages)[((((byte)[0]) & 0xF) << 4) \
-                             + ((((byte)[1]) >> 2) & 0xF)] \
-                       << 3) \
-                      + ((((byte)[1]) & 3) << 1) \
-                      + ((((byte)[2]) >> 5) & 1)] \
-         & (1u << (((byte)[2]) & 0x1F)))
-
-#define UTF8_GET_NAMING(pages, p, n) \
-  ((n) == 2 \
-  ? UTF8_GET_NAMING2(pages, (const unsigned char *)(p)) \
-  : ((n) == 3 \
-     ? UTF8_GET_NAMING3(pages, (const unsigned char *)(p)) \
-     : 0))
+#define UTF8_GET_NAMING3(pages, byte)                                          \
+  (namingBitmap                                                                \
+       [((pages)[((((byte)[0]) & 0xF) << 4) + ((((byte)[1]) >> 2) & 0xF)]      \
+         << 3)                                                                 \
+        + ((((byte)[1]) & 3) << 1) + ((((byte)[2]) >> 5) & 1)]                 \
+   & (1u << (((byte)[2]) & 0x1F)))
+
+#define UTF8_GET_NAMING(pages, p, n)                                           \
+  ((n) == 2                                                                    \
+       ? UTF8_GET_NAMING2(pages, (const unsigned char *)(p))                   \
+       : ((n) == 3 ? UTF8_GET_NAMING3(pages, (const unsigned char *)(p)) : 0))
 
 /* Detection of invalid UTF-8 sequences is based on Table 3.1B
    of Unicode 3.2: http://www.unicode.org/unicode/reports/tr28/
@@ -112,88 +110,76 @@
      (A & 0xC0) == 0xC0  means A > 0xBF
 */
 
-#define UTF8_INVALID2(p) \
+#define UTF8_INVALID2(p)                                                       \
   ((*p) < 0xC2 || ((p)[1] & 0x80) == 0 || ((p)[1] & 0xC0) == 0xC0)
 
-#define UTF8_INVALID3(p) \
-  (((p)[2] & 0x80) == 0 \
-  || \
-  ((*p) == 0xEF && (p)[1] == 0xBF \
-    ? \
-    (p)[2] > 0xBD \
-    : \
-    ((p)[2] & 0xC0) == 0xC0) \
-  || \
-  ((*p) == 0xE0 \
-    ? \
-    (p)[1] < 0xA0 || ((p)[1] & 0xC0) == 0xC0 \
-    : \
-    ((p)[1] & 0x80) == 0 \
-    || \
-    ((*p) == 0xED ? (p)[1] > 0x9F : ((p)[1] & 0xC0) == 0xC0)))
-
-#define UTF8_INVALID4(p) \
-  (((p)[3] & 0x80) == 0 || ((p)[3] & 0xC0) == 0xC0 \
-  || \
-  ((p)[2] & 0x80) == 0 || ((p)[2] & 0xC0) == 0xC0 \
-  || \
-  ((*p) == 0xF0 \
-    ? \
-    (p)[1] < 0x90 || ((p)[1] & 0xC0) == 0xC0 \
-    : \
-    ((p)[1] & 0x80) == 0 \
-    || \
-    ((*p) == 0xF4 ? (p)[1] > 0x8F : ((p)[1] & 0xC0) == 0xC0)))
+#define UTF8_INVALID3(p)                                                       \
+  (((p)[2] & 0x80) == 0                                                        \
+   || ((*p) == 0xEF && (p)[1] == 0xBF ? (p)[2] > 0xBD                          \
+                                      : ((p)[2] & 0xC0) == 0xC0)               \
+   || ((*p) == 0xE0                                                            \
+           ? (p)[1] < 0xA0 || ((p)[1] & 0xC0) == 0xC0                          \
+           : ((p)[1] & 0x80) == 0                                              \
+                 || ((*p) == 0xED ? (p)[1] > 0x9F : ((p)[1] & 0xC0) == 0xC0)))
+
+#define UTF8_INVALID4(p)                                                       \
+  (((p)[3] & 0x80) == 0 || ((p)[3] & 0xC0) == 0xC0 || ((p)[2] & 0x80) == 0     \
+   || ((p)[2] & 0xC0) == 0xC0                                                  \
+   || ((*p) == 0xF0                                                            \
+           ? (p)[1] < 0x90 || ((p)[1] & 0xC0) == 0xC0                          \
+           : ((p)[1] & 0x80) == 0                                              \
+                 || ((*p) == 0xF4 ? (p)[1] > 0x8F : ((p)[1] & 0xC0) == 0xC0)))
 
 static int PTRFASTCALL
-isNever(const ENCODING *UNUSED_P(enc), const char *UNUSED_P(p))
-{
+isNever(const ENCODING *enc, const char *p) {
+  UNUSED_P(enc);
+  UNUSED_P(p);
   return 0;
 }
 
 static int PTRFASTCALL
-utf8_isName2(const ENCODING *UNUSED_P(enc), const char *p)
-{
+utf8_isName2(const ENCODING *enc, const char *p) {
+  UNUSED_P(enc);
   return UTF8_GET_NAMING2(namePages, (const unsigned char *)p);
 }
 
 static int PTRFASTCALL
-utf8_isName3(const ENCODING *UNUSED_P(enc), const char *p)
-{
+utf8_isName3(const ENCODING *enc, const char *p) {
+  UNUSED_P(enc);
   return UTF8_GET_NAMING3(namePages, (const unsigned char *)p);
 }
 
 #define utf8_isName4 isNever
 
 static int PTRFASTCALL
-utf8_isNmstrt2(const ENCODING *UNUSED_P(enc), const char *p)
-{
+utf8_isNmstrt2(const ENCODING *enc, const char *p) {
+  UNUSED_P(enc);
   return UTF8_GET_NAMING2(nmstrtPages, (const unsigned char *)p);
 }
 
 static int PTRFASTCALL
-utf8_isNmstrt3(const ENCODING *UNUSED_P(enc), const char *p)
-{
+utf8_isNmstrt3(const ENCODING *enc, const char *p) {
+  UNUSED_P(enc);
   return UTF8_GET_NAMING3(nmstrtPages, (const unsigned char *)p);
 }
 
 #define utf8_isNmstrt4 isNever
 
 static int PTRFASTCALL
-utf8_isInvalid2(const ENCODING *UNUSED_P(enc), const char *p)
-{
+utf8_isInvalid2(const ENCODING *enc, const char *p) {
+  UNUSED_P(enc);
   return UTF8_INVALID2((const unsigned char *)p);
 }
 
 static int PTRFASTCALL
-utf8_isInvalid3(const ENCODING *UNUSED_P(enc), const char *p)
-{
+utf8_isInvalid3(const ENCODING *enc, const char *p) {
+  UNUSED_P(enc);
   return UTF8_INVALID3((const unsigned char *)p);
 }
 
 static int PTRFASTCALL
-utf8_isInvalid4(const ENCODING *UNUSED_P(enc), const char *p)
-{
+utf8_isInvalid4(const ENCODING *enc, const char *p) {
+  UNUSED_P(enc);
   return UTF8_INVALID4((const unsigned char *)p);
 }
 
@@ -201,61 +187,44 @@ struct normal_encoding {
   ENCODING enc;
   unsigned char type[256];
 #ifdef XML_MIN_SIZE
-  int (PTRFASTCALL *byteType)(const ENCODING *, const char *);
-  int (PTRFASTCALL *isNameMin)(const ENCODING *, const char *);
-  int (PTRFASTCALL *isNmstrtMin)(const ENCODING *, const char *);
-  int (PTRFASTCALL *byteToAscii)(const ENCODING *, const char *);
-  int (PTRCALL *charMatches)(const ENCODING *, const char *, int);
+  int(PTRFASTCALL *byteType)(const ENCODING *, const char *);
+  int(PTRFASTCALL *isNameMin)(const ENCODING *, const char *);
+  int(PTRFASTCALL *isNmstrtMin)(const ENCODING *, const char *);
+  int(PTRFASTCALL *byteToAscii)(const ENCODING *, const char *);
+  int(PTRCALL *charMatches)(const ENCODING *, const char *, int);
 #endif /* XML_MIN_SIZE */
-  int (PTRFASTCALL *isName2)(const ENCODING *, const char *);
-  int (PTRFASTCALL *isName3)(const ENCODING *, const char *);
-  int (PTRFASTCALL *isName4)(const ENCODING *, const char *);
-  int (PTRFASTCALL *isNmstrt2)(const ENCODING *, const char *);
-  int (PTRFASTCALL *isNmstrt3)(const ENCODING *, const char *);
-  int (PTRFASTCALL *isNmstrt4)(const ENCODING *, const char *);
-  int (PTRFASTCALL *isInvalid2)(const ENCODING *, const char *);
-  int (PTRFASTCALL *isInvalid3)(const ENCODING *, const char *);
-  int (PTRFASTCALL *isInvalid4)(const ENCODING *, const char *);
+  int(PTRFASTCALL *isName2)(const ENCODING *, const char *);
+  int(PTRFASTCALL *isName3)(const ENCODING *, const char *);
+  int(PTRFASTCALL *isName4)(const ENCODING *, const char *);
+  int(PTRFASTCALL *isNmstrt2)(const ENCODING *, const char *);
+  int(PTRFASTCALL *isNmstrt3)(const ENCODING *, const char *);
+  int(PTRFASTCALL *isNmstrt4)(const ENCODING *, const char *);
+  int(PTRFASTCALL *isInvalid2)(const ENCODING *, const char *);
+  int(PTRFASTCALL *isInvalid3)(const ENCODING *, const char *);
+  int(PTRFASTCALL *isInvalid4)(const ENCODING *, const char *);
 };
 
-#define AS_NORMAL_ENCODING(enc)   ((const struct normal_encoding *) (enc))
+#define AS_NORMAL_ENCODING(enc) ((const struct normal_encoding *)(enc))
 
 #ifdef XML_MIN_SIZE
 
-#define STANDARD_VTABLE(E) \
- E ## byteType, \
- E ## isNameMin, \
- E ## isNmstrtMin, \
- E ## byteToAscii, \
- E ## charMatches,
+#  define STANDARD_VTABLE(E)                                                   \
+    E##byteType, E##isNameMin, E##isNmstrtMin, E##byteToAscii, E##charMatches,
 
 #else
 
-#define STANDARD_VTABLE(E) /* as nothing */
+#  define STANDARD_VTABLE(E) /* as nothing */
 
 #endif
 
-#define NORMAL_VTABLE(E) \
- E ## isName2, \
- E ## isName3, \
- E ## isName4, \
- E ## isNmstrt2, \
- E ## isNmstrt3, \
- E ## isNmstrt4, \
- E ## isInvalid2, \
- E ## isInvalid3, \
- E ## isInvalid4
-
-#define NULL_VTABLE \
- /* isName2 */ NULL, \
- /* isName3 */ NULL, \
- /* isName4 */ NULL, \
- /* isNmstrt2 */ NULL, \
- /* isNmstrt3 */ NULL, \
- /* isNmstrt4 */ NULL, \
- /* isInvalid2 */ NULL, \
- /* isInvalid3 */ NULL, \
- /* isInvalid4 */ NULL
+#define NORMAL_VTABLE(E)                                                       \
+  E##isName2, E##isName3, E##isName4, E##isNmstrt2, E##isNmstrt3,              \
+      E##isNmstrt4, E##isInvalid2, E##isInvalid3, E##isInvalid4
+
+#define NULL_VTABLE                                                            \
+  /* isName2 */ NULL, /* isName3 */ NULL, /* isName4 */ NULL,                  \
+      /* isNmstrt2 */ NULL, /* isNmstrt3 */ NULL, /* isNmstrt4 */ NULL,        \
+      /* isInvalid2 */ NULL, /* isInvalid3 */ NULL, /* isInvalid4 */ NULL
 
 static int FASTCALL checkCharRefNumber(int);
 
@@ -263,75 +232,70 @@ static int FASTCALL checkCharRefNumber(int);
 #include "ascii.h"
 
 #ifdef XML_MIN_SIZE
-#define sb_isNameMin isNever
-#define sb_isNmstrtMin isNever
+#  define sb_isNameMin isNever
+#  define sb_isNmstrtMin isNever
 #endif
 
 #ifdef XML_MIN_SIZE
-#define MINBPC(enc) ((enc)->minBytesPerChar)
+#  define MINBPC(enc) ((enc)->minBytesPerChar)
 #else
 /* minimum bytes per character */
-#define MINBPC(enc) 1
+#  define MINBPC(enc) 1
 #endif
 
-#define SB_BYTE_TYPE(enc, p) \
+#define SB_BYTE_TYPE(enc, p)                                                   \
   (((struct normal_encoding *)(enc))->type[(unsigned char)*(p)])
 
 #ifdef XML_MIN_SIZE
 static int PTRFASTCALL
-sb_byteType(const ENCODING *enc, const char *p)
-{
+sb_byteType(const ENCODING *enc, const char *p) {
   return SB_BYTE_TYPE(enc, p);
 }
-#define BYTE_TYPE(enc, p) \
- (AS_NORMAL_ENCODING(enc)->byteType(enc, p))
+#  define BYTE_TYPE(enc, p) (AS_NORMAL_ENCODING(enc)->byteType(enc, p))
 #else
-#define BYTE_TYPE(enc, p) SB_BYTE_TYPE(enc, p)
+#  define BYTE_TYPE(enc, p) SB_BYTE_TYPE(enc, p)
 #endif
 
 #ifdef XML_MIN_SIZE
-#define BYTE_TO_ASCII(enc, p) \
- (AS_NORMAL_ENCODING(enc)->byteToAscii(enc, p))
+#  define BYTE_TO_ASCII(enc, p) (AS_NORMAL_ENCODING(enc)->byteToAscii(enc, p))
 static int PTRFASTCALL
-sb_byteToAscii(const ENCODING *enc, const char *p)
-{
+sb_byteToAscii(const ENCODING *enc, const char *p) {
+  UNUSED_P(enc);
   return *p;
 }
 #else
-#define BYTE_TO_ASCII(enc, p) (*(p))
+#  define BYTE_TO_ASCII(enc, p) (*(p))
 #endif
 
-#define IS_NAME_CHAR(enc, p, n) \
- (AS_NORMAL_ENCODING(enc)->isName ## n(enc, p))
-#define IS_NMSTRT_CHAR(enc, p, n) \
- (AS_NORMAL_ENCODING(enc)->isNmstrt ## n(enc, p))
-#define IS_INVALID_CHAR(enc, p, n) \
- (AS_NORMAL_ENCODING(enc)->isInvalid ## n(enc, p))
+#define IS_NAME_CHAR(enc, p, n) (AS_NORMAL_ENCODING(enc)->isName##n(enc, p))
+#define IS_NMSTRT_CHAR(enc, p, n) (AS_NORMAL_ENCODING(enc)->isNmstrt##n(enc, p))
+#define IS_INVALID_CHAR(enc, p, n)                                             \
+  (AS_NORMAL_ENCODING(enc)->isInvalid##n(enc, p))
 
 #ifdef XML_MIN_SIZE
-#define IS_NAME_CHAR_MINBPC(enc, p) \
- (AS_NORMAL_ENCODING(enc)->isNameMin(enc, p))
-#define IS_NMSTRT_CHAR_MINBPC(enc, p) \
- (AS_NORMAL_ENCODING(enc)->isNmstrtMin(enc, p))
+#  define IS_NAME_CHAR_MINBPC(enc, p)                                          \
+    (AS_NORMAL_ENCODING(enc)->isNameMin(enc, p))
+#  define IS_NMSTRT_CHAR_MINBPC(enc, p)                                        \
+    (AS_NORMAL_ENCODING(enc)->isNmstrtMin(enc, p))
 #else
-#define IS_NAME_CHAR_MINBPC(enc, p) (0)
-#define IS_NMSTRT_CHAR_MINBPC(enc, p) (0)
+#  define IS_NAME_CHAR_MINBPC(enc, p) (0)
+#  define IS_NMSTRT_CHAR_MINBPC(enc, p) (0)
 #endif
 
 #ifdef XML_MIN_SIZE
-#define CHAR_MATCHES(enc, p, c) \
- (AS_NORMAL_ENCODING(enc)->charMatches(enc, p, c))
+#  define CHAR_MATCHES(enc, p, c)                                              \
+    (AS_NORMAL_ENCODING(enc)->charMatches(enc, p, c))
 static int PTRCALL
-sb_charMatches(const ENCODING *enc, const char *p, int c)
-{
+sb_charMatches(const ENCODING *enc, const char *p, int c) {
+  UNUSED_P(enc);
   return *p == c;
 }
 #else
 /* c is an ASCII character */
-#define CHAR_MATCHES(enc, p, c) (*(p) == c)
+#  define CHAR_MATCHES(enc, p, c) (*(p) == c)
 #endif
 
-#define PREFIX(ident) normal_ ## ident
+#define PREFIX(ident) normal_##ident
 #define XML_TOK_IMPL_C
 #include "xmltok_impl.c"
 #undef XML_TOK_IMPL_C
@@ -346,42 +310,46 @@ sb_charMatches(const ENCODING *enc, const char *p, int c)
 #undef IS_NMSTRT_CHAR_MINBPC
 #undef IS_INVALID_CHAR
 
-enum {  /* UTF8_cvalN is value of masked first byte of N byte sequence */
-  UTF8_cval1 = 0x00,
-  UTF8_cval2 = 0xc0,
-  UTF8_cval3 = 0xe0,
-  UTF8_cval4 = 0xf0
+enum { /* UTF8_cvalN is value of masked first byte of N byte sequence */
+       UTF8_cval1 = 0x00,
+       UTF8_cval2 = 0xc0,
+       UTF8_cval3 = 0xe0,
+       UTF8_cval4 = 0xf0
 };
 
 void
-align_limit_to_full_utf8_characters(const char * from, const char ** fromLimRef)
-{
-  const char * fromLim = *fromLimRef;
+_INTERNAL_trim_to_complete_utf8_characters(const char *from,
+                                           const char **fromLimRef) {
+  const char *fromLim = *fromLimRef;
   size_t walked = 0;
   for (; fromLim > from; fromLim--, walked++) {
     const unsigned char prev = (unsigned char)fromLim[-1];
-    if ((prev & 0xf8u) == 0xf0u) { /* 4-byte character, lead by 0b11110xxx byte */
+    if ((prev & 0xf8u)
+        == 0xf0u) { /* 4-byte character, lead by 0b11110xxx byte */
       if (walked + 1 >= 4) {
         fromLim += 4 - 1;
         break;
       } else {
         walked = 0;
       }
-    } else if ((prev & 0xf0u) == 0xe0u) { /* 3-byte character, lead by 0b1110xxxx byte */
+    } else if ((prev & 0xf0u)
+               == 0xe0u) { /* 3-byte character, lead by 0b1110xxxx byte */
       if (walked + 1 >= 3) {
         fromLim += 3 - 1;
         break;
       } else {
         walked = 0;
       }
-    } else if ((prev & 0xe0u) == 0xc0u) { /* 2-byte character, lead by 0b110xxxxx byte */
+    } else if ((prev & 0xe0u)
+               == 0xc0u) { /* 2-byte character, lead by 0b110xxxxx byte */
       if (walked + 1 >= 2) {
         fromLim += 2 - 1;
         break;
       } else {
         walked = 0;
       }
-    } else if ((prev & 0x80u) == 0x00u) { /* 1-byte character, matching 0b0xxxxxxx */
+    } else if ((prev & 0x80u)
+               == 0x00u) { /* 1-byte character, matching 0b0xxxxxxx */
       break;
     }
   }
@@ -389,16 +357,15 @@ align_limit_to_full_utf8_characters(const char * from, const char ** fromLimRef)
 }
 
 static enum XML_Convert_Result PTRCALL
-utf8_toUtf8(const ENCODING *UNUSED_P(enc),
-            const char **fromP, const char *fromLim,
-            char **toP, const char *toLim)
-{
+utf8_toUtf8(const ENCODING *enc, const char **fromP, const char *fromLim,
+            char **toP, const char *toLim) {
   bool input_incomplete = false;
   bool output_exhausted = false;
 
   /* Avoid copying partial characters (due to limited space). */
   const ptrdiff_t bytesAvailable = fromLim - *fromP;
   const ptrdiff_t bytesStorable = toLim - *toP;
+  UNUSED_P(enc);
   if (bytesAvailable > bytesStorable) {
     fromLim = *fromP + bytesStorable;
     output_exhausted = true;
@@ -406,8 +373,8 @@ utf8_toUtf8(const ENCODING *UNUSED_P(enc),
 
   /* Avoid copying partial characters (from incomplete input). */
   {
-    const char * const fromLimBefore = fromLim;
-    align_limit_to_full_utf8_characters(*fromP, &fromLim);
+    const char *const fromLimBefore = fromLim;
+    _INTERNAL_trim_to_complete_utf8_characters(*fromP, &fromLim);
     if (fromLim < fromLimBefore) {
       input_incomplete = true;
     }
@@ -415,12 +382,12 @@ utf8_toUtf8(const ENCODING *UNUSED_P(enc),
 
   {
     const ptrdiff_t bytesToCopy = fromLim - *fromP;
-    memcpy((void *)*toP, (const void *)*fromP, (size_t)bytesToCopy);
+    memcpy(*toP, *fromP, bytesToCopy);
     *fromP += bytesToCopy;
     *toP += bytesToCopy;
   }
 
-  if (output_exhausted)  // needs to go first
+  if (output_exhausted) /* needs to go first */
     return XML_CONVERT_OUTPUT_EXHAUSTED;
   else if (input_incomplete)
     return XML_CONVERT_INPUT_INCOMPLETE;
@@ -429,10 +396,8 @@ utf8_toUtf8(const ENCODING *UNUSED_P(enc),
 }
 
 static enum XML_Convert_Result PTRCALL
-utf8_toUtf16(const ENCODING *enc,
-             const char **fromP, const char *fromLim,
-             unsigned short **toP, const unsigned short *toLim)
-{
+utf8_toUtf16(const ENCODING *enc, const char **fromP, const char *fromLim,
+             unsigned short **toP, const unsigned short *toLim) {
   enum XML_Convert_Result res = XML_CONVERT_COMPLETED;
   unsigned short *to = *toP;
   const char *from = *fromP;
@@ -451,30 +416,28 @@ utf8_toUtf16(const ENCODING *enc,
         res = XML_CONVERT_INPUT_INCOMPLETE;
         goto after;
       }
-      *to++ = (unsigned short)(((from[0] & 0xf) << 12)
-                               | ((from[1] & 0x3f) << 6) | (from[2] & 0x3f));
+      *to++ = (unsigned short)(((from[0] & 0xf) << 12) | ((from[1] & 0x3f) << 6)
+                               | (from[2] & 0x3f));
       from += 3;
       break;
-    case BT_LEAD4:
-      {
-        unsigned long n;
-        if (toLim - to < 2) {
-          res = XML_CONVERT_OUTPUT_EXHAUSTED;
-          goto after;
-        }
-        if (fromLim - from < 4) {
-          res = XML_CONVERT_INPUT_INCOMPLETE;
-          goto after;
-        }
-        n = ((from[0] & 0x7) << 18) | ((from[1] & 0x3f) << 12)
-            | ((from[2] & 0x3f) << 6) | (from[3] & 0x3f);
-        n -= 0x10000;
-        to[0] = (unsigned short)((n >> 10) | 0xD800);
-        to[1] = (unsigned short)((n & 0x3FF) | 0xDC00);
-        to += 2;
-        from += 4;
+    case BT_LEAD4: {
+      unsigned long n;
+      if (toLim - to < 2) {
+        res = XML_CONVERT_OUTPUT_EXHAUSTED;
+        goto after;
       }
-      break;
+      if (fromLim - from < 4) {
+        res = XML_CONVERT_INPUT_INCOMPLETE;
+        goto after;
+      }
+      n = ((from[0] & 0x7) << 18) | ((from[1] & 0x3f) << 12)
+          | ((from[2] & 0x3f) << 6) | (from[3] & 0x3f);
+      n -= 0x10000;
+      to[0] = (unsigned short)((n >> 10) | 0xD800);
+      to[1] = (unsigned short)((n & 0x3FF) | 0xDC00);
+      to += 2;
+      from += 4;
+    } break;
     default:
       *to++ = *from++;
       break;
@@ -489,56 +452,51 @@ utf8_toUtf16(const ENCODING *enc,
 }
 
 #ifdef XML_NS
-static const struct normal_encoding utf8_encoding_ns = {
-  { VTABLE1, utf8_toUtf8, utf8_toUtf16, 1, 1, 0 },
-  {
-#include "asciitab.h"
-#include "utf8tab.h"
-  },
-  STANDARD_VTABLE(sb_) NORMAL_VTABLE(utf8_)
-};
+static const struct normal_encoding utf8_encoding_ns
+    = {{VTABLE1, utf8_toUtf8, utf8_toUtf16, 1, 1, 0},
+       {
+#  include "asciitab.h"
+#  include "utf8tab.h"
+       },
+       STANDARD_VTABLE(sb_) NORMAL_VTABLE(utf8_)};
 #endif
 
-static const struct normal_encoding utf8_encoding = {
-  { VTABLE1, utf8_toUtf8, utf8_toUtf16, 1, 1, 0 },
-  {
+static const struct normal_encoding utf8_encoding
+    = {{VTABLE1, utf8_toUtf8, utf8_toUtf16, 1, 1, 0},
+       {
 #define BT_COLON BT_NMSTRT
 #include "asciitab.h"
 #undef BT_COLON
 #include "utf8tab.h"
-  },
-  STANDARD_VTABLE(sb_) NORMAL_VTABLE(utf8_)
-};
+       },
+       STANDARD_VTABLE(sb_) NORMAL_VTABLE(utf8_)};
 
 #ifdef XML_NS
 
-static const struct normal_encoding internal_utf8_encoding_ns = {
-  { VTABLE1, utf8_toUtf8, utf8_toUtf16, 1, 1, 0 },
-  {
-#include "iasciitab.h"
-#include "utf8tab.h"
-  },
-  STANDARD_VTABLE(sb_) NORMAL_VTABLE(utf8_)
-};
+static const struct normal_encoding internal_utf8_encoding_ns
+    = {{VTABLE1, utf8_toUtf8, utf8_toUtf16, 1, 1, 0},
+       {
+#  include "iasciitab.h"
+#  include "utf8tab.h"
+       },
+       STANDARD_VTABLE(sb_) NORMAL_VTABLE(utf8_)};
 
 #endif
 
-static const struct normal_encoding internal_utf8_encoding = {
-  { VTABLE1, utf8_toUtf8, utf8_toUtf16, 1, 1, 0 },
-  {
+static const struct normal_encoding internal_utf8_encoding
+    = {{VTABLE1, utf8_toUtf8, utf8_toUtf16, 1, 1, 0},
+       {
 #define BT_COLON BT_NMSTRT
 #include "iasciitab.h"
 #undef BT_COLON
 #include "utf8tab.h"
-  },
-  STANDARD_VTABLE(sb_) NORMAL_VTABLE(utf8_)
-};
+       },
+       STANDARD_VTABLE(sb_) NORMAL_VTABLE(utf8_)};
 
 static enum XML_Convert_Result PTRCALL
-latin1_toUtf8(const ENCODING *UNUSED_P(enc),
-              const char **fromP, const char *fromLim,
-              char **toP, const char *toLim)
-{
+latin1_toUtf8(const ENCODING *enc, const char **fromP, const char *fromLim,
+              char **toP, const char *toLim) {
+  UNUSED_P(enc);
   for (;;) {
     unsigned char c;
     if (*fromP == fromLim)
@@ -550,8 +508,7 @@ latin1_toUtf8(const ENCODING *UNUSED_P(enc),
       *(*toP)++ = (char)((c >> 6) | UTF8_cval2);
       *(*toP)++ = (char)((c & 0x3f) | 0x80);
       (*fromP)++;
-    }
-    else {
+    } else {
       if (*toP == toLim)
         return XML_CONVERT_OUTPUT_EXHAUSTED;
       *(*toP)++ = *(*fromP)++;
@@ -560,10 +517,9 @@ latin1_toUtf8(const ENCODING *UNUSED_P(enc),
 }
 
 static enum XML_Convert_Result PTRCALL
-latin1_toUtf16(const ENCODING *UNUSED_P(enc),
-               const char **fromP, const char *fromLim,
-               unsigned short **toP, const unsigned short *toLim)
-{
+latin1_toUtf16(const ENCODING *enc, const char **fromP, const char *fromLim,
+               unsigned short **toP, const unsigned short *toLim) {
+  UNUSED_P(enc);
   while (*fromP < fromLim && *toP < toLim)
     *(*toP)++ = (unsigned char)*(*fromP)++;
 
@@ -575,33 +531,30 @@ latin1_toUtf16(const ENCODING *UNUSED_P(enc),
 
 #ifdef XML_NS
 
-static const struct normal_encoding latin1_encoding_ns = {
-  { VTABLE1, latin1_toUtf8, latin1_toUtf16, 1, 0, 0 },
-  {
-#include "asciitab.h"
-#include "latin1tab.h"
-  },
-  STANDARD_VTABLE(sb_) NULL_VTABLE
-};
+static const struct normal_encoding latin1_encoding_ns
+    = {{VTABLE1, latin1_toUtf8, latin1_toUtf16, 1, 0, 0},
+       {
+#  include "asciitab.h"
+#  include "latin1tab.h"
+       },
+       STANDARD_VTABLE(sb_) NULL_VTABLE};
 
 #endif
 
-static const struct normal_encoding latin1_encoding = {
-  { VTABLE1, latin1_toUtf8, latin1_toUtf16, 1, 0, 0 },
-  {
+static const struct normal_encoding latin1_encoding
+    = {{VTABLE1, latin1_toUtf8, latin1_toUtf16, 1, 0, 0},
+       {
 #define BT_COLON BT_NMSTRT
 #include "asciitab.h"
 #undef BT_COLON
 #include "latin1tab.h"
-  },
-  STANDARD_VTABLE(sb_) NULL_VTABLE
-};
+       },
+       STANDARD_VTABLE(sb_) NULL_VTABLE};
 
 static enum XML_Convert_Result PTRCALL
-ascii_toUtf8(const ENCODING *UNUSED_P(enc),
-             const char **fromP, const char *fromLim,
-             char **toP, const char *toLim)
-{
+ascii_toUtf8(const ENCODING *enc, const char **fromP, const char *fromLim,
+             char **toP, const char *toLim) {
+  UNUSED_P(enc);
   while (*fromP < fromLim && *toP < toLim)
     *(*toP)++ = *(*fromP)++;
 
@@ -613,40 +566,45 @@ ascii_toUtf8(const ENCODING *UNUSED_P(enc),
 
 #ifdef XML_NS
 
-static const struct normal_encoding ascii_encoding_ns = {
-  { VTABLE1, ascii_toUtf8, latin1_toUtf16, 1, 1, 0 },
-  {
-#include "asciitab.h"
-/* BT_NONXML == 0 */
-  },
-  STANDARD_VTABLE(sb_) NULL_VTABLE
-};
+static const struct normal_encoding ascii_encoding_ns
+    = {{VTABLE1, ascii_toUtf8, latin1_toUtf16, 1, 1, 0},
+       {
+#  include "asciitab.h"
+           /* BT_NONXML == 0 */
+       },
+       STANDARD_VTABLE(sb_) NULL_VTABLE};
 
 #endif
 
-static const struct normal_encoding ascii_encoding = {
-  { VTABLE1, ascii_toUtf8, latin1_toUtf16, 1, 1, 0 },
-  {
+static const struct normal_encoding ascii_encoding
+    = {{VTABLE1, ascii_toUtf8, latin1_toUtf16, 1, 1, 0},
+       {
 #define BT_COLON BT_NMSTRT
 #include "asciitab.h"
 #undef BT_COLON
-/* BT_NONXML == 0 */
-  },
-  STANDARD_VTABLE(sb_) NULL_VTABLE
-};
+           /* BT_NONXML == 0 */
+       },
+       STANDARD_VTABLE(sb_) NULL_VTABLE};
 
 static int PTRFASTCALL
-unicode_byte_type(char hi, char lo)
-{
+unicode_byte_type(char hi, char lo) {
   switch ((unsigned char)hi) {
-  case 0xD8: case 0xD9: case 0xDA: case 0xDB:
+  /* 0xD800?0xDBFF first 16-bit code unit or high surrogate (W1) */
+  case 0xD8:
+  case 0xD9:
+  case 0xDA:
+  case 0xDB:
     return BT_LEAD4;
-  case 0xDC: case 0xDD: case 0xDE: case 0xDF:
+  /* 0xDC00?0xDFFF second 16-bit code unit or low surrogate (W2) */
+  case 0xDC:
+  case 0xDD:
+  case 0xDE:
+  case 0xDF:
     return BT_TRAIL;
   case 0xFF:
     switch ((unsigned char)lo) {
-    case 0xFF:
-    case 0xFE:
+    case 0xFF: /* noncharacter-FFFF */
+    case 0xFE: /* noncharacter-FFFE */
       return BT_NONXML;
     }
     break;
@@ -654,102 +612,105 @@ unicode_byte_type(char hi, char lo)
   return BT_NONASCII;
 }
 
-#define DEFINE_UTF16_TO_UTF8(E) \
-static enum XML_Convert_Result  PTRCALL \
-E ## toUtf8(const ENCODING *UNUSED_P(enc), \
-            const char **fromP, const char *fromLim, \
-            char **toP, const char *toLim) \
-{ \
-  const char *from = *fromP; \
-  fromLim = from + (((fromLim - from) >> 1) << 1);  /* shrink to even */ \
-  for (; from < fromLim; from += 2) { \
-    int plane; \
-    unsigned char lo2; \
-    unsigned char lo = GET_LO(from); \
-    unsigned char hi = GET_HI(from); \
-    switch (hi) { \
-    case 0: \
-      if (lo < 0x80) { \
-        if (*toP == toLim) { \
-          *fromP = from; \
-          return XML_CONVERT_OUTPUT_EXHAUSTED; \
-        } \
-        *(*toP)++ = lo; \
-        break; \
-      } \
-      /* fall through */ \
-    case 0x1: case 0x2: case 0x3: \
-    case 0x4: case 0x5: case 0x6: case 0x7: \
-      if (toLim -  *toP < 2) { \
-        *fromP = from; \
-        return XML_CONVERT_OUTPUT_EXHAUSTED; \
-      } \
-      *(*toP)++ = ((lo >> 6) | (hi << 2) |  UTF8_cval2); \
-      *(*toP)++ = ((lo & 0x3f) | 0x80); \
-      break; \
-    default: \
-      if (toLim -  *toP < 3)  { \
-        *fromP = from; \
-        return XML_CONVERT_OUTPUT_EXHAUSTED; \
-      } \
-      /* 16 bits divided 4, 6, 6 amongst 3 bytes */ \
-      *(*toP)++ = ((hi >> 4) | UTF8_cval3); \
-      *(*toP)++ = (((hi & 0xf) << 2) | (lo >> 6) | 0x80); \
-      *(*toP)++ = ((lo & 0x3f) | 0x80); \
-      break; \
-    case 0xD8: case 0xD9: case 0xDA: case 0xDB: \
-      if (toLim -  *toP < 4) { \
-        *fromP = from; \
-        return XML_CONVERT_OUTPUT_EXHAUSTED; \
-      } \
-      if (fromLim - from < 4) { \
-        *fromP = from; \
-        return XML_CONVERT_INPUT_INCOMPLETE; \
-      } \
-      plane = (((hi & 0x3) << 2) | ((lo >> 6) & 0x3)) + 1; \
-      *(*toP)++ = ((plane >> 2) | UTF8_cval4); \
-      *(*toP)++ = (((lo >> 2) & 0xF) | ((plane & 0x3) << 4) | 0x80); \
-      from += 2; \
-      lo2 = GET_LO(from); \
-      *(*toP)++ = (((lo & 0x3) << 4) \
-                   | ((GET_HI(from) & 0x3) << 2) \
-                   | (lo2 >> 6) \
-                   | 0x80); \
-      *(*toP)++ = ((lo2 & 0x3f) | 0x80); \
-      break; \
-    } \
-  } \
-  *fromP = from; \
-  if (from < fromLim) \
-    return XML_CONVERT_INPUT_INCOMPLETE; \
-  else \
-    return XML_CONVERT_COMPLETED; \
-}
+#define DEFINE_UTF16_TO_UTF8(E)                                                \
+  static enum XML_Convert_Result PTRCALL E##toUtf8(                            \
+      const ENCODING *enc, const char **fromP, const char *fromLim,            \
+      char **toP, const char *toLim) {                                         \
+    const char *from = *fromP;                                                 \
+    UNUSED_P(enc);                                                             \
+    fromLim = from + (((fromLim - from) >> 1) << 1); /* shrink to even */      \
+    for (; from < fromLim; from += 2) {                                        \
+      int plane;                                                               \
+      unsigned char lo2;                                                       \
+      unsigned char lo = GET_LO(from);                                         \
+      unsigned char hi = GET_HI(from);                                         \
+      switch (hi) {                                                            \
+      case 0:                                                                  \
+        if (lo < 0x80) {                                                       \
+          if (*toP == toLim) {                                                 \
+            *fromP = from;                                                     \
+            return XML_CONVERT_OUTPUT_EXHAUSTED;                               \
+          }                                                                    \
+          *(*toP)++ = lo;                                                      \
+          break;                                                               \
+        }                                                                      \
+        /* fall through */                                                     \
+      case 0x1:                                                                \
+      case 0x2:                                                                \
+      case 0x3:                                                                \
+      case 0x4:                                                                \
+      case 0x5:                                                                \
+      case 0x6:                                                                \
+      case 0x7:                                                                \
+        if (toLim - *toP < 2) {                                                \
+          *fromP = from;                                                       \
+          return XML_CONVERT_OUTPUT_EXHAUSTED;                                 \
+        }                                                                      \
+        *(*toP)++ = ((lo >> 6) | (hi << 2) | UTF8_cval2);                      \
+        *(*toP)++ = ((lo & 0x3f) | 0x80);                                      \
+        break;                                                                 \
+      default:                                                                 \
+        if (toLim - *toP < 3) {                                                \
+          *fromP = from;                                                       \
+          return XML_CONVERT_OUTPUT_EXHAUSTED;                                 \
+        }                                                                      \
+        /* 16 bits divided 4, 6, 6 amongst 3 bytes */                          \
+        *(*toP)++ = ((hi >> 4) | UTF8_cval3);                                  \
+        *(*toP)++ = (((hi & 0xf) << 2) | (lo >> 6) | 0x80);                    \
+        *(*toP)++ = ((lo & 0x3f) | 0x80);                                      \
+        break;                                                                 \
+      case 0xD8:                                                               \
+      case 0xD9:                                                               \
+      case 0xDA:                                                               \
+      case 0xDB:                                                               \
+        if (toLim - *toP < 4) {                                                \
+          *fromP = from;                                                       \
+          return XML_CONVERT_OUTPUT_EXHAUSTED;                                 \
+        }                                                                      \
+        if (fromLim - from < 4) {                                              \
+          *fromP = from;                                                       \
+          return XML_CONVERT_INPUT_INCOMPLETE;                                 \
+        }                                                                      \
+        plane = (((hi & 0x3) << 2) | ((lo >> 6) & 0x3)) + 1;                   \
+        *(*toP)++ = (char)((plane >> 2) | UTF8_cval4);                         \
+        *(*toP)++ = (((lo >> 2) & 0xF) | ((plane & 0x3) << 4) | 0x80);         \
+        from += 2;                                                             \
+        lo2 = GET_LO(from);                                                    \
+        *(*toP)++ = (((lo & 0x3) << 4) | ((GET_HI(from) & 0x3) << 2)           \
+                     | (lo2 >> 6) | 0x80);                                     \
+        *(*toP)++ = ((lo2 & 0x3f) | 0x80);                                     \
+        break;                                                                 \
+      }                                                                        \
+    }                                                                          \
+    *fromP = from;                                                             \
+    if (from < fromLim)                                                        \
+      return XML_CONVERT_INPUT_INCOMPLETE;                                     \
+    else                                                                       \
+      return XML_CONVERT_COMPLETED;                                            \
+  }
 
-#define DEFINE_UTF16_TO_UTF16(E) \
-static enum XML_Convert_Result  PTRCALL \
-E ## toUtf16(const ENCODING *UNUSED_P(enc), \
-             const char **fromP, const char *fromLim, \
-             unsigned short **toP, const unsigned short *toLim) \
-{ \
-  enum XML_Convert_Result res = XML_CONVERT_COMPLETED; \
-  fromLim = *fromP + (((fromLim - *fromP) >> 1) << 1);  /* shrink to even */ \
-  /* Avoid copying first half only of surrogate */ \
-  if (fromLim - *fromP > ((toLim - *toP) << 1) \
-      && (GET_HI(fromLim - 2) & 0xF8) == 0xD8) { \
-    fromLim -= 2; \
-    res = XML_CONVERT_INPUT_INCOMPLETE; \
-  } \
-  for (; *fromP < fromLim && *toP < toLim; *fromP += 2) \
-    *(*toP)++ = (GET_HI(*fromP) << 8) | GET_LO(*fromP); \
-  if ((*toP == toLim) && (*fromP < fromLim)) \
-    return XML_CONVERT_OUTPUT_EXHAUSTED; \
-  else \
-    return res; \
-}
+#define DEFINE_UTF16_TO_UTF16(E)                                               \
+  static enum XML_Convert_Result PTRCALL E##toUtf16(                           \
+      const ENCODING *enc, const char **fromP, const char *fromLim,            \
+      unsigned short **toP, const unsigned short *toLim) {                     \
+    enum XML_Convert_Result res = XML_CONVERT_COMPLETED;                       \
+    UNUSED_P(enc);                                                             \
+    fromLim = *fromP + (((fromLim - *fromP) >> 1) << 1); /* shrink to even */  \
+    /* Avoid copying first half only of surrogate */                           \
+    if (fromLim - *fromP > ((toLim - *toP) << 1)                               \
+        && (GET_HI(fromLim - 2) & 0xF8) == 0xD8) {                             \
+      fromLim -= 2;                                                            \
+      res = XML_CONVERT_INPUT_INCOMPLETE;                                      \
+    }                                                                          \
+    for (; *fromP < fromLim && *toP < toLim; *fromP += 2)                      \
+      *(*toP)++ = (GET_HI(*fromP) << 8) | GET_LO(*fromP);                      \
+    if ((*toP == toLim) && (*fromP < fromLim))                                 \
+      return XML_CONVERT_OUTPUT_EXHAUSTED;                                     \
+    else                                                                       \
+      return res;                                                              \
+  }
 
-#define SET2(ptr, ch) \
-  (((ptr)[0] = ((ch) & 0xff)), ((ptr)[1] = ((ch) >> 8)))
+#define SET2(ptr, ch) (((ptr)[0] = ((ch)&0xff)), ((ptr)[1] = ((ch) >> 8)))
 #define GET_LO(ptr) ((unsigned char)(ptr)[0])
 #define GET_HI(ptr) ((unsigned char)(ptr)[1])
 
@@ -760,8 +721,7 @@ DEFINE_UTF16_TO_UTF16(little2_)
 #undef GET_LO
 #undef GET_HI
 
-#define SET2(ptr, ch) \
-  (((ptr)[0] = ((ch) >> 8)), ((ptr)[1] = ((ch) & 0xFF)))
+#define SET2(ptr, ch) (((ptr)[0] = ((ch) >> 8)), ((ptr)[1] = ((ch)&0xFF)))
 #define GET_LO(ptr) ((unsigned char)(ptr)[1])
 #define GET_HI(ptr) ((unsigned char)(ptr)[0])
 
@@ -772,292 +732,279 @@ DEFINE_UTF16_TO_UTF16(big2_)
 #undef GET_LO
 #undef GET_HI
 
-#define LITTLE2_BYTE_TYPE(enc, p) \
- ((p)[1] == 0 \
-  ? ((struct normal_encoding *)(enc))->type[(unsigned char)*(p)] \
-  : unicode_byte_type((p)[1], (p)[0]))
-#define LITTLE2_BYTE_TO_ASCII(enc, p) ((p)[1] == 0 ? (p)[0] : -1)
-#define LITTLE2_CHAR_MATCHES(enc, p, c) ((p)[1] == 0 && (p)[0] == c)
-#define LITTLE2_IS_NAME_CHAR_MINBPC(enc, p) \
+#define LITTLE2_BYTE_TYPE(enc, p)                                              \
+  ((p)[1] == 0 ? ((struct normal_encoding *)(enc))->type[(unsigned char)*(p)]  \
+               : unicode_byte_type((p)[1], (p)[0]))
+#define LITTLE2_BYTE_TO_ASCII(p) ((p)[1] == 0 ? (p)[0] : -1)
+#define LITTLE2_CHAR_MATCHES(p, c) ((p)[1] == 0 && (p)[0] == c)
+#define LITTLE2_IS_NAME_CHAR_MINBPC(p)                                         \
   UCS2_GET_NAMING(namePages, (unsigned char)p[1], (unsigned char)p[0])
-#define LITTLE2_IS_NMSTRT_CHAR_MINBPC(enc, p) \
+#define LITTLE2_IS_NMSTRT_CHAR_MINBPC(p)                                       \
   UCS2_GET_NAMING(nmstrtPages, (unsigned char)p[1], (unsigned char)p[0])
 
 #ifdef XML_MIN_SIZE
 
 static int PTRFASTCALL
-little2_byteType(const ENCODING *enc, const char *p)
-{
+little2_byteType(const ENCODING *enc, const char *p) {
   return LITTLE2_BYTE_TYPE(enc, p);
 }
 
 static int PTRFASTCALL
-little2_byteToAscii(const ENCODING *enc, const char *p)
-{
-  return LITTLE2_BYTE_TO_ASCII(enc, p);
+little2_byteToAscii(const ENCODING *enc, const char *p) {
+  UNUSED_P(enc);
+  return LITTLE2_BYTE_TO_ASCII(p);
 }
 
 static int PTRCALL
-little2_charMatches(const ENCODING *enc, const char *p, int c)
-{
-  return LITTLE2_CHAR_MATCHES(enc, p, c);
+little2_charMatches(const ENCODING *enc, const char *p, int c) {
+  UNUSED_P(enc);
+  return LITTLE2_CHAR_MATCHES(p, c);
 }
 
 static int PTRFASTCALL
-little2_isNameMin(const ENCODING *enc, const char *p)
-{
-  return LITTLE2_IS_NAME_CHAR_MINBPC(enc, p);
+little2_isNameMin(const ENCODING *enc, const char *p) {
+  UNUSED_P(enc);
+  return LITTLE2_IS_NAME_CHAR_MINBPC(p);
 }
 
 static int PTRFASTCALL
-little2_isNmstrtMin(const ENCODING *enc, const char *p)
-{
-  return LITTLE2_IS_NMSTRT_CHAR_MINBPC(enc, p);
+little2_isNmstrtMin(const ENCODING *enc, const char *p) {
+  UNUSED_P(enc);
+  return LITTLE2_IS_NMSTRT_CHAR_MINBPC(p);
 }
 
-#undef VTABLE
-#define VTABLE VTABLE1, little2_toUtf8, little2_toUtf16
+#  undef VTABLE
+#  define VTABLE VTABLE1, little2_toUtf8, little2_toUtf16
 
 #else /* not XML_MIN_SIZE */
 
-#undef PREFIX
-#define PREFIX(ident) little2_ ## ident
-#define MINBPC(enc) 2
+#  undef PREFIX
+#  define PREFIX(ident) little2_##ident
+#  define MINBPC(enc) 2
 /* CHAR_MATCHES is guaranteed to have MINBPC bytes available. */
-#define BYTE_TYPE(enc, p) LITTLE2_BYTE_TYPE(enc, p)
-#define BYTE_TO_ASCII(enc, p) LITTLE2_BYTE_TO_ASCII(enc, p)
-#define CHAR_MATCHES(enc, p, c) LITTLE2_CHAR_MATCHES(enc, p, c)
-#define IS_NAME_CHAR(enc, p, n) 0
-#define IS_NAME_CHAR_MINBPC(enc, p) LITTLE2_IS_NAME_CHAR_MINBPC(enc, p)
-#define IS_NMSTRT_CHAR(enc, p, n) (0)
-#define IS_NMSTRT_CHAR_MINBPC(enc, p) LITTLE2_IS_NMSTRT_CHAR_MINBPC(enc, p)
-
-#define XML_TOK_IMPL_C
-#include "xmltok_impl.c"
-#undef XML_TOK_IMPL_C
-
-#undef MINBPC
-#undef BYTE_TYPE
-#undef BYTE_TO_ASCII
-#undef CHAR_MATCHES
-#undef IS_NAME_CHAR
-#undef IS_NAME_CHAR_MINBPC
-#undef IS_NMSTRT_CHAR
-#undef IS_NMSTRT_CHAR_MINBPC
-#undef IS_INVALID_CHAR
+#  define BYTE_TYPE(enc, p) LITTLE2_BYTE_TYPE(enc, p)
+#  define BYTE_TO_ASCII(enc, p) LITTLE2_BYTE_TO_ASCII(p)
+#  define CHAR_MATCHES(enc, p, c) LITTLE2_CHAR_MATCHES(p, c)
+#  define IS_NAME_CHAR(enc, p, n) 0
+#  define IS_NAME_CHAR_MINBPC(enc, p) LITTLE2_IS_NAME_CHAR_MINBPC(p)
+#  define IS_NMSTRT_CHAR(enc, p, n) (0)
+#  define IS_NMSTRT_CHAR_MINBPC(enc, p) LITTLE2_IS_NMSTRT_CHAR_MINBPC(p)
+
+#  define XML_TOK_IMPL_C
+#  include "xmltok_impl.c"
+#  undef XML_TOK_IMPL_C
+
+#  undef MINBPC
+#  undef BYTE_TYPE
+#  undef BYTE_TO_ASCII
+#  undef CHAR_MATCHES
+#  undef IS_NAME_CHAR
+#  undef IS_NAME_CHAR_MINBPC
+#  undef IS_NMSTRT_CHAR
+#  undef IS_NMSTRT_CHAR_MINBPC
+#  undef IS_INVALID_CHAR
 
 #endif /* not XML_MIN_SIZE */
 
 #ifdef XML_NS
 
-static const struct normal_encoding little2_encoding_ns = {
-  { VTABLE, 2, 0,
-#if BYTEORDER == 1234
-    1
-#else
-    0
-#endif
-  },
-  {
-#include "asciitab.h"
-#include "latin1tab.h"
-  },
-  STANDARD_VTABLE(little2_) NULL_VTABLE
-};
+static const struct normal_encoding little2_encoding_ns
+    = {{VTABLE, 2, 0,
+#  if BYTEORDER == 1234
+        1
+#  else
+        0
+#  endif
+       },
+       {
+#  include "asciitab.h"
+#  include "latin1tab.h"
+       },
+       STANDARD_VTABLE(little2_) NULL_VTABLE};
 
 #endif
 
-static const struct normal_encoding little2_encoding = {
-  { VTABLE, 2, 0,
+static const struct normal_encoding little2_encoding
+    = {{VTABLE, 2, 0,
 #if BYTEORDER == 1234
-    1
+        1
 #else
-    0
+        0
 #endif
-  },
-  {
+       },
+       {
 #define BT_COLON BT_NMSTRT
 #include "asciitab.h"
 #undef BT_COLON
 #include "latin1tab.h"
-  },
-  STANDARD_VTABLE(little2_) NULL_VTABLE
-};
+       },
+       STANDARD_VTABLE(little2_) NULL_VTABLE};
 
 #if BYTEORDER != 4321
 
-#ifdef XML_NS
+#  ifdef XML_NS
 
-static const struct normal_encoding internal_little2_encoding_ns = {
-  { VTABLE, 2, 0, 1 },
-  {
-#include "iasciitab.h"
-#include "latin1tab.h"
-  },
-  STANDARD_VTABLE(little2_) NULL_VTABLE
-};
+static const struct normal_encoding internal_little2_encoding_ns
+    = {{VTABLE, 2, 0, 1},
+       {
+#    include "iasciitab.h"
+#    include "latin1tab.h"
+       },
+       STANDARD_VTABLE(little2_) NULL_VTABLE};
 
-#endif
+#  endif
 
-static const struct normal_encoding internal_little2_encoding = {
-  { VTABLE, 2, 0, 1 },
-  {
-#define BT_COLON BT_NMSTRT
-#include "iasciitab.h"
-#undef BT_COLON
-#include "latin1tab.h"
-  },
-  STANDARD_VTABLE(little2_) NULL_VTABLE
-};
+static const struct normal_encoding internal_little2_encoding
+    = {{VTABLE, 2, 0, 1},
+       {
+#  define BT_COLON BT_NMSTRT
+#  include "iasciitab.h"
+#  undef BT_COLON
+#  include "latin1tab.h"
+       },
+       STANDARD_VTABLE(little2_) NULL_VTABLE};
 
 #endif
 
-
-#define BIG2_BYTE_TYPE(enc, p) \
- ((p)[0] == 0 \
-  ? ((struct normal_encoding *)(enc))->type[(unsigned char)(p)[1]] \
-  : unicode_byte_type((p)[0], (p)[1]))
-#define BIG2_BYTE_TO_ASCII(enc, p) ((p)[0] == 0 ? (p)[1] : -1)
-#define BIG2_CHAR_MATCHES(enc, p, c) ((p)[0] == 0 && (p)[1] == c)
-#define BIG2_IS_NAME_CHAR_MINBPC(enc, p) \
+#define BIG2_BYTE_TYPE(enc, p)                                                 \
+  ((p)[0] == 0                                                                 \
+       ? ((struct normal_encoding *)(enc))->type[(unsigned char)(p)[1]]        \
+       : unicode_byte_type((p)[0], (p)[1]))
+#define BIG2_BYTE_TO_ASCII(p) ((p)[0] == 0 ? (p)[1] : -1)
+#define BIG2_CHAR_MATCHES(p, c) ((p)[0] == 0 && (p)[1] == c)
+#define BIG2_IS_NAME_CHAR_MINBPC(p)                                            \
   UCS2_GET_NAMING(namePages, (unsigned char)p[0], (unsigned char)p[1])
-#define BIG2_IS_NMSTRT_CHAR_MINBPC(enc, p) \
+#define BIG2_IS_NMSTRT_CHAR_MINBPC(p)                                          \
   UCS2_GET_NAMING(nmstrtPages, (unsigned char)p[0], (unsigned char)p[1])
 
 #ifdef XML_MIN_SIZE
 
 static int PTRFASTCALL
-big2_byteType(const ENCODING *enc, const char *p)
-{
+big2_byteType(const ENCODING *enc, const char *p) {
   return BIG2_BYTE_TYPE(enc, p);
 }
 
 static int PTRFASTCALL
-big2_byteToAscii(const ENCODING *enc, const char *p)
-{
-  return BIG2_BYTE_TO_ASCII(enc, p);
+big2_byteToAscii(const ENCODING *enc, const char *p) {
+  UNUSED_P(enc);
+  return BIG2_BYTE_TO_ASCII(p);
 }
 
 static int PTRCALL
-big2_charMatches(const ENCODING *enc, const char *p, int c)
-{
-  return BIG2_CHAR_MATCHES(enc, p, c);
+big2_charMatches(const ENCODING *enc, const char *p, int c) {
+  UNUSED_P(enc);
+  return BIG2_CHAR_MATCHES(p, c);
 }
 
 static int PTRFASTCALL
-big2_isNameMin(const ENCODING *enc, const char *p)
-{
-  return BIG2_IS_NAME_CHAR_MINBPC(enc, p);
+big2_isNameMin(const ENCODING *enc, const char *p) {
+  UNUSED_P(enc);
+  return BIG2_IS_NAME_CHAR_MINBPC(p);
 }
 
 static int PTRFASTCALL
-big2_isNmstrtMin(const ENCODING *enc, const char *p)
-{
-  return BIG2_IS_NMSTRT_CHAR_MINBPC(enc, p);
+big2_isNmstrtMin(const ENCODING *enc, const char *p) {
+  UNUSED_P(enc);
+  return BIG2_IS_NMSTRT_CHAR_MINBPC(p);
 }
 
-#undef VTABLE
-#define VTABLE VTABLE1, big2_toUtf8, big2_toUtf16
+#  undef VTABLE
+#  define VTABLE VTABLE1, big2_toUtf8, big2_toUtf16
 
 #else /* not XML_MIN_SIZE */
 
-#undef PREFIX
-#define PREFIX(ident) big2_ ## ident
-#define MINBPC(enc) 2
+#  undef PREFIX
+#  define PREFIX(ident) big2_##ident
+#  define MINBPC(enc) 2
 /* CHAR_MATCHES is guaranteed to have MINBPC bytes available. */
-#define BYTE_TYPE(enc, p) BIG2_BYTE_TYPE(enc, p)
-#define BYTE_TO_ASCII(enc, p) BIG2_BYTE_TO_ASCII(enc, p)
-#define CHAR_MATCHES(enc, p, c) BIG2_CHAR_MATCHES(enc, p, c)
-#define IS_NAME_CHAR(enc, p, n) 0
-#define IS_NAME_CHAR_MINBPC(enc, p) BIG2_IS_NAME_CHAR_MINBPC(enc, p)
-#define IS_NMSTRT_CHAR(enc, p, n) (0)
-#define IS_NMSTRT_CHAR_MINBPC(enc, p) BIG2_IS_NMSTRT_CHAR_MINBPC(enc, p)
-
-#define XML_TOK_IMPL_C
-#include "xmltok_impl.c"
-#undef XML_TOK_IMPL_C
-
-#undef MINBPC
-#undef BYTE_TYPE
-#undef BYTE_TO_ASCII
-#undef CHAR_MATCHES
-#undef IS_NAME_CHAR
-#undef IS_NAME_CHAR_MINBPC
-#undef IS_NMSTRT_CHAR
-#undef IS_NMSTRT_CHAR_MINBPC
-#undef IS_INVALID_CHAR
+#  define BYTE_TYPE(enc, p) BIG2_BYTE_TYPE(enc, p)
+#  define BYTE_TO_ASCII(enc, p) BIG2_BYTE_TO_ASCII(p)
+#  define CHAR_MATCHES(enc, p, c) BIG2_CHAR_MATCHES(p, c)
+#  define IS_NAME_CHAR(enc, p, n) 0
+#  define IS_NAME_CHAR_MINBPC(enc, p) BIG2_IS_NAME_CHAR_MINBPC(p)
+#  define IS_NMSTRT_CHAR(enc, p, n) (0)
+#  define IS_NMSTRT_CHAR_MINBPC(enc, p) BIG2_IS_NMSTRT_CHAR_MINBPC(p)
+
+#  define XML_TOK_IMPL_C
+#  include "xmltok_impl.c"
+#  undef XML_TOK_IMPL_C
+
+#  undef MINBPC
+#  undef BYTE_TYPE
+#  undef BYTE_TO_ASCII
+#  undef CHAR_MATCHES
+#  undef IS_NAME_CHAR
+#  undef IS_NAME_CHAR_MINBPC
+#  undef IS_NMSTRT_CHAR
+#  undef IS_NMSTRT_CHAR_MINBPC
+#  undef IS_INVALID_CHAR
 
 #endif /* not XML_MIN_SIZE */
 
 #ifdef XML_NS
 
-static const struct normal_encoding big2_encoding_ns = {
-  { VTABLE, 2, 0,
-#if BYTEORDER == 4321
-  1
-#else
-  0
-#endif
-  },
-  {
-#include "asciitab.h"
-#include "latin1tab.h"
-  },
-  STANDARD_VTABLE(big2_) NULL_VTABLE
-};
+static const struct normal_encoding big2_encoding_ns
+    = {{VTABLE, 2, 0,
+#  if BYTEORDER == 4321
+        1
+#  else
+        0
+#  endif
+       },
+       {
+#  include "asciitab.h"
+#  include "latin1tab.h"
+       },
+       STANDARD_VTABLE(big2_) NULL_VTABLE};
 
 #endif
 
-static const struct normal_encoding big2_encoding = {
-  { VTABLE, 2, 0,
+static const struct normal_encoding big2_encoding
+    = {{VTABLE, 2, 0,
 #if BYTEORDER == 4321
-  1
+        1
 #else
-  0
+        0
 #endif
-  },
-  {
+       },
+       {
 #define BT_COLON BT_NMSTRT
 #include "asciitab.h"
 #undef BT_COLON
 #include "latin1tab.h"
-  },
-  STANDARD_VTABLE(big2_) NULL_VTABLE
-};
+       },
+       STANDARD_VTABLE(big2_) NULL_VTABLE};
 
 #if BYTEORDER != 1234
 
-#ifdef XML_NS
+#  ifdef XML_NS
 
-static const struct normal_encoding internal_big2_encoding_ns = {
-  { VTABLE, 2, 0, 1 },
-  {
-#include "iasciitab.h"
-#include "latin1tab.h"
-  },
-  STANDARD_VTABLE(big2_) NULL_VTABLE
-};
+static const struct normal_encoding internal_big2_encoding_ns
+    = {{VTABLE, 2, 0, 1},
+       {
+#    include "iasciitab.h"
+#    include "latin1tab.h"
+       },
+       STANDARD_VTABLE(big2_) NULL_VTABLE};
 
-#endif
+#  endif
 
-static const struct normal_encoding internal_big2_encoding = {
-  { VTABLE, 2, 0, 1 },
-  {
-#define BT_COLON BT_NMSTRT
-#include "iasciitab.h"
-#undef BT_COLON
-#include "latin1tab.h"
-  },
-  STANDARD_VTABLE(big2_) NULL_VTABLE
-};
+static const struct normal_encoding internal_big2_encoding
+    = {{VTABLE, 2, 0, 1},
+       {
+#  define BT_COLON BT_NMSTRT
+#  include "iasciitab.h"
+#  undef BT_COLON
+#  include "latin1tab.h"
+       },
+       STANDARD_VTABLE(big2_) NULL_VTABLE};
 
 #endif
 
 #undef PREFIX
 
 static int FASTCALL
-streqci(const char *s1, const char *s2)
-{
+streqci(const char *s1, const char *s2) {
   for (;;) {
     char c1 = *s1++;
     char c2 = *s2++;
@@ -1071,22 +1018,21 @@ streqci(const char *s1, const char *s2)
       c2 += ASCII_A - ASCII_a; /* LCOV_EXCL_LINE */
     if (c1 != c2)
       return 0;
-    if (!c1)
+    if (! c1)
       break;
   }
   return 1;
 }
 
 static void PTRCALL
-initUpdatePosition(const ENCODING *UNUSED_P(enc), const char *ptr,
-                   const char *end, POSITION *pos)
-{
+initUpdatePosition(const ENCODING *enc, const char *ptr, const char *end,
+                   POSITION *pos) {
+  UNUSED_P(enc);
   normal_updatePosition(&utf8_encoding.enc, ptr, end, pos);
 }
 
 static int
-toAscii(const ENCODING *enc, const char *ptr, const char *end)
-{
+toAscii(const ENCODING *enc, const char *ptr, const char *end) {
   char buf[1];
   char *p = buf;
   XmlUtf8Convert(enc, &ptr, end, &p, p + 1);
@@ -1097,8 +1043,7 @@ toAscii(const ENCODING *enc, const char *ptr, const char *end)
 }
 
 static int FASTCALL
-isSpace(int c)
-{
+isSpace(int c) {
   switch (c) {
   case 0x20:
   case 0xD:
@@ -1113,21 +1058,16 @@ isSpace(int c)
    followed by name=val.
 */
 static int
-parsePseudoAttribute(const ENCODING *enc,
-                     const char *ptr,
-                     const char *end,
-                     const char **namePtr,
-                     const char **nameEndPtr,
-                     const char **valPtr,
-                     const char **nextTokPtr)
-{
+parsePseudoAttribute(const ENCODING *enc, const char *ptr, const char *end,
+                     const char **namePtr, const char **nameEndPtr,
+                     const char **valPtr, const char **nextTokPtr) {
   int c;
   char open;
   if (ptr == end) {
     *namePtr = NULL;
     return 1;
   }
-  if (!isSpace(toAscii(enc, ptr, end))) {
+  if (! isSpace(toAscii(enc, ptr, end))) {
     *nextTokPtr = ptr;
     return 0;
   }
@@ -1183,12 +1123,9 @@ parsePseudoAttribute(const ENCODING *enc,
     c = toAscii(enc, ptr, end);
     if (c == open)
       break;
-    if (!(ASCII_a <= c && c <= ASCII_z)
-        && !(ASCII_A <= c && c <= ASCII_Z)
-        && !(ASCII_0 <= c && c <= ASCII_9)
-        && c != ASCII_PERIOD
-        && c != ASCII_MINUS
-        && c != ASCII_UNDERSCORE) {
+    if (! (ASCII_a <= c && c <= ASCII_z) && ! (ASCII_A <= c && c <= ASCII_Z)
+        && ! (ASCII_0 <= c && c <= ASCII_9) && c != ASCII_PERIOD
+        && c != ASCII_MINUS && c != ASCII_UNDERSCORE) {
       *nextTokPtr = ptr;
       return 0;
     }
@@ -1197,68 +1134,52 @@ parsePseudoAttribute(const ENCODING *enc,
   return 1;
 }
 
-static const char KW_version[] = {
-  ASCII_v, ASCII_e, ASCII_r, ASCII_s, ASCII_i, ASCII_o, ASCII_n, '\0'
-};
+static const char KW_version[]
+    = {ASCII_v, ASCII_e, ASCII_r, ASCII_s, ASCII_i, ASCII_o, ASCII_n, '\0'};
 
-static const char KW_encoding[] = {
-  ASCII_e, ASCII_n, ASCII_c, ASCII_o, ASCII_d, ASCII_i, ASCII_n, ASCII_g, '\0'
-};
+static const char KW_encoding[] = {ASCII_e, ASCII_n, ASCII_c, ASCII_o, ASCII_d,
+                                   ASCII_i, ASCII_n, ASCII_g, '\0'};
 
-static const char KW_standalone[] = {
-  ASCII_s, ASCII_t, ASCII_a, ASCII_n, ASCII_d, ASCII_a, ASCII_l, ASCII_o,
-  ASCII_n, ASCII_e, '\0'
-};
+static const char KW_standalone[]
+    = {ASCII_s, ASCII_t, ASCII_a, ASCII_n, ASCII_d, ASCII_a,
+       ASCII_l, ASCII_o, ASCII_n, ASCII_e, '\0'};
 
-static const char KW_yes[] = {
-  ASCII_y, ASCII_e, ASCII_s,  '\0'
-};
+static const char KW_yes[] = {ASCII_y, ASCII_e, ASCII_s, '\0'};
 
-static const char KW_no[] = {
-  ASCII_n, ASCII_o,  '\0'
-};
+static const char KW_no[] = {ASCII_n, ASCII_o, '\0'};
 
 static int
-doParseXmlDecl(const ENCODING *(*encodingFinder)(const ENCODING *,
-                                                 const char *,
+doParseXmlDecl(const ENCODING *(*encodingFinder)(const ENCODING *, const char *,
                                                  const char *),
-               int isGeneralTextEntity,
-               const ENCODING *enc,
-               const char *ptr,
-               const char *end,
-               const char **badPtr,
-               const char **versionPtr,
-               const char **versionEndPtr,
-               const char **encodingName,
-               const ENCODING **encoding,
-               int *standalone)
-{
+               int isGeneralTextEntity, const ENCODING *enc, const char *ptr,
+               const char *end, const char **badPtr, const char **versionPtr,
+               const char **versionEndPtr, const char **encodingName,
+               const ENCODING **encoding, int *standalone) {
   const char *val = NULL;
   const char *name = NULL;
   const char *nameEnd = NULL;
   ptr += 5 * enc->minBytesPerChar;
   end -= 2 * enc->minBytesPerChar;
-  if (!parsePseudoAttribute(enc, ptr, end, &name, &nameEnd, &val, &ptr)
-      || !name) {
+  if (! parsePseudoAttribute(enc, ptr, end, &name, &nameEnd, &val, &ptr)
+      || ! name) {
     *badPtr = ptr;
     return 0;
   }
-  if (!XmlNameMatchesAscii(enc, name, nameEnd, KW_version)) {
-    if (!isGeneralTextEntity) {
+  if (! XmlNameMatchesAscii(enc, name, nameEnd, KW_version)) {
+    if (! isGeneralTextEntity) {
       *badPtr = name;
       return 0;
     }
-  }
-  else {
+  } else {
     if (versionPtr)
       *versionPtr = val;
     if (versionEndPtr)
       *versionEndPtr = ptr;
-    if (!parsePseudoAttribute(enc, ptr, end, &name, &nameEnd, &val, &ptr)) {
+    if (! parsePseudoAttribute(enc, ptr, end, &name, &nameEnd, &val, &ptr)) {
       *badPtr = ptr;
       return 0;
     }
-    if (!name) {
+    if (! name) {
       if (isGeneralTextEntity) {
         /* a TextDecl must have an EncodingDecl */
         *badPtr = ptr;
@@ -1269,7 +1190,7 @@ doParseXmlDecl(const ENCODING *(*encodingFinder)(const ENCODING *,
   }
   if (XmlNameMatchesAscii(enc, name, nameEnd, KW_encoding)) {
     int c = toAscii(enc, val, end);
-    if (!(ASCII_a <= c && c <= ASCII_z) && !(ASCII_A <= c && c <= ASCII_Z)) {
+    if (! (ASCII_a <= c && c <= ASCII_z) && ! (ASCII_A <= c && c <= ASCII_Z)) {
       *badPtr = val;
       return 0;
     }
@@ -1277,14 +1198,14 @@ doParseXmlDecl(const ENCODING *(*encodingFinder)(const ENCODING *,
       *encodingName = val;
     if (encoding)
       *encoding = encodingFinder(enc, val, ptr - enc->minBytesPerChar);
-    if (!parsePseudoAttribute(enc, ptr, end, &name, &nameEnd, &val, &ptr)) {
+    if (! parsePseudoAttribute(enc, ptr, end, &name, &nameEnd, &val, &ptr)) {
       *badPtr = ptr;
       return 0;
     }
-    if (!name)
+    if (! name)
       return 1;
   }
-  if (!XmlNameMatchesAscii(enc, name, nameEnd, KW_standalone)
+  if (! XmlNameMatchesAscii(enc, name, nameEnd, KW_standalone)
       || isGeneralTextEntity) {
     *badPtr = name;
     return 0;
@@ -1292,12 +1213,10 @@ doParseXmlDecl(const ENCODING *(*encodingFinder)(const ENCODING *,
   if (XmlNameMatchesAscii(enc, val, ptr - enc->minBytesPerChar, KW_yes)) {
     if (standalone)
       *standalone = 1;
-  }
-  else if (XmlNameMatchesAscii(enc, val, ptr - enc->minBytesPerChar, KW_no)) {
+  } else if (XmlNameMatchesAscii(enc, val, ptr - enc->minBytesPerChar, KW_no)) {
     if (standalone)
       *standalone = 0;
-  }
-  else {
+  } else {
     *badPtr = val;
     return 0;
   }
@@ -1311,11 +1230,16 @@ doParseXmlDecl(const ENCODING *(*encodingFinder)(const ENCODING *,
 }
 
 static int FASTCALL
-checkCharRefNumber(int result)
-{
+checkCharRefNumber(int result) {
   switch (result >> 8) {
-  case 0xD8: case 0xD9: case 0xDA: case 0xDB:
-  case 0xDC: case 0xDD: case 0xDE: case 0xDF:
+  case 0xD8:
+  case 0xD9:
+  case 0xDA:
+  case 0xDB:
+  case 0xDC:
+  case 0xDD:
+  case 0xDE:
+  case 0xDF:
     return -1;
   case 0:
     if (latin1_encoding.type[result] == BT_NONXML)
@@ -1330,8 +1254,7 @@ checkCharRefNumber(int result)
 }
 
 int FASTCALL
-XmlUtf8Encode(int c, char *buf)
-{
+XmlUtf8Encode(int c, char *buf) {
   enum {
     /* minN is minimum legal resulting value for N byte sequence */
     min2 = 0x80,
@@ -1367,8 +1290,7 @@ XmlUtf8Encode(int c, char *buf)
 }
 
 int FASTCALL
-XmlUtf16Encode(int charNum, unsigned short *buf)
-{
+XmlUtf16Encode(int charNum, unsigned short *buf) {
   if (charNum < 0)
     return 0;
   if (charNum < 0x10000) {
@@ -1392,17 +1314,15 @@ struct unknown_encoding {
   char utf8[256][4];
 };
 
-#define AS_UNKNOWN_ENCODING(enc)  ((const struct unknown_encoding *) (enc))
+#define AS_UNKNOWN_ENCODING(enc) ((const struct unknown_encoding *)(enc))
 
 int
-XmlSizeOfUnknownEncoding(void)
-{
+XmlSizeOfUnknownEncoding(void) {
   return sizeof(struct unknown_encoding);
 }
 
 static int PTRFASTCALL
-unknown_isName(const ENCODING *enc, const char *p)
-{
+unknown_isName(const ENCODING *enc, const char *p) {
   const struct unknown_encoding *uenc = AS_UNKNOWN_ENCODING(enc);
   int c = uenc->convert(uenc->userData, p);
   if (c & ~0xFFFF)
@@ -1411,8 +1331,7 @@ unknown_isName(const ENCODING *enc, const char *p)
 }
 
 static int PTRFASTCALL
-unknown_isNmstrt(const ENCODING *enc, const char *p)
-{
+unknown_isNmstrt(const ENCODING *enc, const char *p) {
   const struct unknown_encoding *uenc = AS_UNKNOWN_ENCODING(enc);
   int c = uenc->convert(uenc->userData, p);
   if (c & ~0xFFFF)
@@ -1421,18 +1340,15 @@ unknown_isNmstrt(const ENCODING *enc, const char *p)
 }
 
 static int PTRFASTCALL
-unknown_isInvalid(const ENCODING *enc, const char *p)
-{
+unknown_isInvalid(const ENCODING *enc, const char *p) {
   const struct unknown_encoding *uenc = AS_UNKNOWN_ENCODING(enc);
   int c = uenc->convert(uenc->userData, p);
   return (c & ~0xFFFF) || checkCharRefNumber(c) < 0;
 }
 
 static enum XML_Convert_Result PTRCALL
-unknown_toUtf8(const ENCODING *enc,
-               const char **fromP, const char *fromLim,
-               char **toP, const char *toLim)
-{
+unknown_toUtf8(const ENCODING *enc, const char **fromP, const char *fromLim,
+               char **toP, const char *toLim) {
   const struct unknown_encoding *uenc = AS_UNKNOWN_ENCODING(enc);
   char buf[XML_UTF8_ENCODE_MAX];
   for (;;) {
@@ -1450,33 +1366,27 @@ unknown_toUtf8(const ENCODING *enc,
       utf8 = buf;
       *fromP += (AS_NORMAL_ENCODING(enc)->type[(unsigned char)**fromP]
                  - (BT_LEAD2 - 2));
-    }
-    else {
+    } else {
       if (n > toLim - *toP)
         return XML_CONVERT_OUTPUT_EXHAUSTED;
       (*fromP)++;
     }
-    do {
-      *(*toP)++ = *utf8++;
-    } while (--n != 0);
+    memcpy(*toP, utf8, n);
+    *toP += n;
   }
 }
 
 static enum XML_Convert_Result PTRCALL
-unknown_toUtf16(const ENCODING *enc,
-                const char **fromP, const char *fromLim,
-                unsigned short **toP, const unsigned short *toLim)
-{
+unknown_toUtf16(const ENCODING *enc, const char **fromP, const char *fromLim,
+                unsigned short **toP, const unsigned short *toLim) {
   const struct unknown_encoding *uenc = AS_UNKNOWN_ENCODING(enc);
   while (*fromP < fromLim && *toP < toLim) {
     unsigned short c = uenc->utf16[(unsigned char)**fromP];
     if (c == 0) {
-      c = (unsigned short)
-          uenc->convert(uenc->userData, *fromP);
+      c = (unsigned short)uenc->convert(uenc->userData, *fromP);
       *fromP += (AS_NORMAL_ENCODING(enc)->type[(unsigned char)**fromP]
                  - (BT_LEAD2 - 2));
-    }
-    else
+    } else
       (*fromP)++;
     *(*toP)++ = c;
   }
@@ -1488,19 +1398,14 @@ unknown_toUtf16(const ENCODING *enc,
 }
 
 ENCODING *
-XmlInitUnknownEncoding(void *mem,
-                       int *table,
-                       CONVERTER convert,
-                       void *userData)
-{
+XmlInitUnknownEncoding(void *mem, int *table, CONVERTER convert,
+                       void *userData) {
   int i;
   struct unknown_encoding *e = (struct unknown_encoding *)mem;
-  for (i = 0; i < (int)sizeof(struct normal_encoding); i++)
-    ((char *)mem)[i] = ((char *)&latin1_encoding)[i];
+  memcpy(mem, &latin1_encoding, sizeof(struct normal_encoding));
   for (i = 0; i < 128; i++)
     if (latin1_encoding.type[i] != BT_OTHER
-        && latin1_encoding.type[i] != BT_NONXML
-        && table[i] != i)
+        && latin1_encoding.type[i] != BT_NONXML && table[i] != i)
       return 0;
   for (i = 0; i < 256; i++) {
     int c = table[i];
@@ -1510,35 +1415,30 @@ XmlInitUnknownEncoding(void *mem,
       e->utf16[i] = 0xFFFF;
       e->utf8[i][0] = 1;
       e->utf8[i][1] = 0;
-    }
-    else if (c < 0) {
+    } else if (c < 0) {
       if (c < -4)
         return 0;
       /* Multi-byte sequences need a converter function */
-      if (!convert)
+      if (! convert)
         return 0;
       e->normal.type[i] = (unsigned char)(BT_LEAD2 - (c + 2));
       e->utf8[i][0] = 0;
       e->utf16[i] = 0;
-    }
-    else if (c < 0x80) {
+    } else if (c < 0x80) {
       if (latin1_encoding.type[c] != BT_OTHER
-          && latin1_encoding.type[c] != BT_NONXML
-          && c != i)
+          && latin1_encoding.type[c] != BT_NONXML && c != i)
         return 0;
       e->normal.type[i] = latin1_encoding.type[c];
       e->utf8[i][0] = 1;
       e->utf8[i][1] = (char)c;
       e->utf16[i] = (unsigned short)(c == 0 ? 0xFFFF : c);
-    }
-    else if (checkCharRefNumber(c) < 0) {
+    } else if (checkCharRefNumber(c) < 0) {
       e->normal.type[i] = BT_NONXML;
       /* This shouldn't really get used. */
       e->utf16[i] = 0xFFFF;
       e->utf8[i][0] = 1;
       e->utf8[i][1] = 0;
-    }
-    else {
+    } else {
       if (c > 0xFFFF)
         return 0;
       if (UCS2_GET_NAMING(nmstrtPages, c >> 8, c & 0xff))
@@ -1583,44 +1483,32 @@ enum {
   NO_ENC
 };
 
-static const char KW_ISO_8859_1[] = {
-  ASCII_I, ASCII_S, ASCII_O, ASCII_MINUS, ASCII_8, ASCII_8, ASCII_5, ASCII_9,
-  ASCII_MINUS, ASCII_1, '\0'
-};
-static const char KW_US_ASCII[] = {
-  ASCII_U, ASCII_S, ASCII_MINUS, ASCII_A, ASCII_S, ASCII_C, ASCII_I, ASCII_I,
-  '\0'
-};
-static const char KW_UTF_8[] =  {
-  ASCII_U, ASCII_T, ASCII_F, ASCII_MINUS, ASCII_8, '\0'
-};
-static const char KW_UTF_16[] = {
-  ASCII_U, ASCII_T, ASCII_F, ASCII_MINUS, ASCII_1, ASCII_6, '\0'
-};
-static const char KW_UTF_16BE[] = {
-  ASCII_U, ASCII_T, ASCII_F, ASCII_MINUS, ASCII_1, ASCII_6, ASCII_B, ASCII_E,
-  '\0'
-};
-static const char KW_UTF_16LE[] = {
-  ASCII_U, ASCII_T, ASCII_F, ASCII_MINUS, ASCII_1, ASCII_6, ASCII_L, ASCII_E,
-  '\0'
-};
+static const char KW_ISO_8859_1[]
+    = {ASCII_I, ASCII_S, ASCII_O,     ASCII_MINUS, ASCII_8, ASCII_8,
+       ASCII_5, ASCII_9, ASCII_MINUS, ASCII_1,     '\0'};
+static const char KW_US_ASCII[]
+    = {ASCII_U, ASCII_S, ASCII_MINUS, ASCII_A, ASCII_S,
+       ASCII_C, ASCII_I, ASCII_I,     '\0'};
+static const char KW_UTF_8[]
+    = {ASCII_U, ASCII_T, ASCII_F, ASCII_MINUS, ASCII_8, '\0'};
+static const char KW_UTF_16[]
+    = {ASCII_U, ASCII_T, ASCII_F, ASCII_MINUS, ASCII_1, ASCII_6, '\0'};
+static const char KW_UTF_16BE[]
+    = {ASCII_U, ASCII_T, ASCII_F, ASCII_MINUS, ASCII_1,
+       ASCII_6, ASCII_B, ASCII_E, '\0'};
+static const char KW_UTF_16LE[]
+    = {ASCII_U, ASCII_T, ASCII_F, ASCII_MINUS, ASCII_1,
+       ASCII_6, ASCII_L, ASCII_E, '\0'};
 
 static int FASTCALL
-getEncodingIndex(const char *name)
-{
-  static const char * const encodingNames[] = {
-    KW_ISO_8859_1,
-    KW_US_ASCII,
-    KW_UTF_8,
-    KW_UTF_16,
-    KW_UTF_16BE,
-    KW_UTF_16LE,
+getEncodingIndex(const char *name) {
+  static const char *const encodingNames[] = {
+      KW_ISO_8859_1, KW_US_ASCII, KW_UTF_8, KW_UTF_16, KW_UTF_16BE, KW_UTF_16LE,
   };
   int i;
   if (name == NULL)
     return NO_ENC;
-  for (i = 0; i < (int)(sizeof(encodingNames)/sizeof(encodingNames[0])); i++)
+  for (i = 0; i < (int)(sizeof(encodingNames) / sizeof(encodingNames[0])); i++)
     if (streqci(name, encodingNames[i]))
       return i;
   return UNKNOWN_ENC;
@@ -1640,15 +1528,9 @@ getEncodingIndex(const char *name)
    XML_PROLOG_STATE otherwise.
 */
 
-
 static int
-initScan(const ENCODING * const *encodingTable,
-         const INIT_ENCODING *enc,
-         int state,
-         const char *ptr,
-         const char *end,
-         const char **nextTokPtr)
-{
+initScan(const ENCODING *const *encodingTable, const INIT_ENCODING *enc,
+         int state, const char *ptr, const char *end, const char **nextTokPtr) {
   const ENCODING **encPtr;
 
   if (ptr >= end)
@@ -1673,20 +1555,17 @@ initScan(const ENCODING * const *encodingTable,
     case 0xFE:
     case 0xFF:
     case 0xEF: /* possibly first byte of UTF-8 BOM */
-      if (INIT_ENC_INDEX(enc) == ISO_8859_1_ENC
-          && state == XML_CONTENT_STATE)
+      if (INIT_ENC_INDEX(enc) == ISO_8859_1_ENC && state == XML_CONTENT_STATE)
         break;
       /* fall through */
     case 0x00:
     case 0x3C:
       return XML_TOK_PARTIAL;
     }
-  }
-  else {
+  } else {
     switch (((unsigned char)ptr[0] << 8) | (unsigned char)ptr[1]) {
     case 0xFEFF:
-      if (INIT_ENC_INDEX(enc) == ISO_8859_1_ENC
-          && state == XML_CONTENT_STATE)
+      if (INIT_ENC_INDEX(enc) == ISO_8859_1_ENC && state == XML_CONTENT_STATE)
         break;
       *nextTokPtr = ptr + 2;
       *encPtr = encodingTable[UTF_16BE_ENC];
@@ -1700,8 +1579,7 @@ initScan(const ENCODING * const *encodingTable,
       *encPtr = encodingTable[UTF_16LE_ENC];
       return XmlTok(*encPtr, state, ptr, end, nextTokPtr);
     case 0xFFFE:
-      if (INIT_ENC_INDEX(enc) == ISO_8859_1_ENC
-          && state == XML_CONTENT_STATE)
+      if (INIT_ENC_INDEX(enc) == ISO_8859_1_ENC && state == XML_CONTENT_STATE)
         break;
       *nextTokPtr = ptr + 2;
       *encPtr = encodingTable[UTF_16LE_ENC];
@@ -1716,8 +1594,8 @@ initScan(const ENCODING * const *encodingTable,
       */
       if (state == XML_CONTENT_STATE) {
         int e = INIT_ENC_INDEX(enc);
-        if (e == ISO_8859_1_ENC || e == UTF_16BE_ENC
-            || e == UTF_16LE_ENC || e == UTF_16_ENC)
+        if (e == ISO_8859_1_ENC || e == UTF_16BE_ENC || e == UTF_16LE_ENC
+            || e == UTF_16_ENC)
           break;
       }
       if (ptr + 2 == end)
@@ -1740,8 +1618,7 @@ initScan(const ENCODING * const *encodingTable,
           break;
         *encPtr = encodingTable[UTF_16BE_ENC];
         return XmlTok(*encPtr, state, ptr, end, nextTokPtr);
-      }
-      else if (ptr[1] == '\0') {
+      } else if (ptr[1] == '\0') {
         /* We could recover here in the case:
             - parsing an external entity
             - second byte is 0
@@ -1763,7 +1640,6 @@ initScan(const ENCODING * const *encodingTable,
   return XmlTok(*encPtr, state, ptr, end, nextTokPtr);
 }
 
-
 #define NS(x) x
 #define ns(x) x
 #define XML_TOK_NS_C
@@ -1774,22 +1650,19 @@ initScan(const ENCODING * const *encodingTable,
 
 #ifdef XML_NS
 
-#define NS(x) x ## NS
-#define ns(x) x ## _ns
+#  define NS(x) x##NS
+#  define ns(x) x##_ns
 
-#define XML_TOK_NS_C
-#include "xmltok_ns.c"
-#undef XML_TOK_NS_C
+#  define XML_TOK_NS_C
+#  include "xmltok_ns.c"
+#  undef XML_TOK_NS_C
 
-#undef NS
-#undef ns
+#  undef NS
+#  undef ns
 
 ENCODING *
-XmlInitUnknownEncodingNS(void *mem,
-                         int *table,
-                         CONVERTER convert,
-                         void *userData)
-{
+XmlInitUnknownEncodingNS(void *mem, int *table, CONVERTER convert,
+                         void *userData) {
   ENCODING *enc = XmlInitUnknownEncoding(mem, table, convert, userData);
   if (enc)
     ((struct normal_encoding *)enc)->type[ASCII_COLON] = BT_COLON;
diff --git a/Modules/expat/xmltok.h b/Modules/expat/xmltok.h
index 6d31879b331d8..2adbf5307befa 100644
--- a/Modules/expat/xmltok.h
+++ b/Modules/expat/xmltok.h
@@ -38,16 +38,18 @@ extern "C" {
 #endif
 
 /* The following token may be returned by XmlContentTok */
-#define XML_TOK_TRAILING_RSQB -5 /* ] or ]] at the end of the scan; might be
-                                    start of illegal ]]> sequence */
+#define XML_TOK_TRAILING_RSQB                                                  \
+  -5 /* ] or ]] at the end of the scan; might be                               \
+        start of illegal ]]> sequence */
 /* The following tokens may be returned by both XmlPrologTok and
    XmlContentTok.
 */
-#define XML_TOK_NONE -4          /* The string to be scanned is empty */
-#define XML_TOK_TRAILING_CR -3   /* A CR at the end of the scan;
-                                    might be part of CRLF sequence */
-#define XML_TOK_PARTIAL_CHAR -2  /* only part of a multibyte sequence */
-#define XML_TOK_PARTIAL -1       /* only part of a token */
+#define XML_TOK_NONE -4 /* The string to be scanned is empty */
+#define XML_TOK_TRAILING_CR                                                    \
+  -3                            /* A CR at the end of the scan;                \
+                                   might be part of CRLF sequence */
+#define XML_TOK_PARTIAL_CHAR -2 /* only part of a multibyte sequence */
+#define XML_TOK_PARTIAL -1      /* only part of a token */
 #define XML_TOK_INVALID 0
 
 /* The following tokens are returned by XmlContentTok; some are also
@@ -62,24 +64,24 @@ extern "C" {
 #define XML_TOK_DATA_NEWLINE 7
 #define XML_TOK_CDATA_SECT_OPEN 8
 #define XML_TOK_ENTITY_REF 9
-#define XML_TOK_CHAR_REF 10               /* numeric character reference */
+#define XML_TOK_CHAR_REF 10 /* numeric character reference */
 
 /* The following tokens may be returned by both XmlPrologTok and
    XmlContentTok.
 */
-#define XML_TOK_PI 11                     /* processing instruction */
-#define XML_TOK_XML_DECL 12               /* XML decl or text decl */
+#define XML_TOK_PI 11       /* processing instruction */
+#define XML_TOK_XML_DECL 12 /* XML decl or text decl */
 #define XML_TOK_COMMENT 13
-#define XML_TOK_BOM 14                    /* Byte order mark */
+#define XML_TOK_BOM 14 /* Byte order mark */
 
 /* The following tokens are returned only by XmlPrologTok */
 #define XML_TOK_PROLOG_S 15
-#define XML_TOK_DECL_OPEN 16              /* <!foo */
-#define XML_TOK_DECL_CLOSE 17             /* > */
+#define XML_TOK_DECL_OPEN 16  /* <!foo */
+#define XML_TOK_DECL_CLOSE 17 /* > */
 #define XML_TOK_NAME 18
 #define XML_TOK_NMTOKEN 19
-#define XML_TOK_POUND_NAME 20             /* #name */
-#define XML_TOK_OR 21                     /* | */
+#define XML_TOK_POUND_NAME 20 /* #name */
+#define XML_TOK_OR 21         /* | */
 #define XML_TOK_PERCENT 22
 #define XML_TOK_OPEN_PAREN 23
 #define XML_TOK_CLOSE_PAREN 24
@@ -90,14 +92,14 @@ extern "C" {
 #define XML_TOK_INSTANCE_START 29
 
 /* The following occur only in element type declarations */
-#define XML_TOK_NAME_QUESTION 30          /* name? */
-#define XML_TOK_NAME_ASTERISK 31          /* name* */
-#define XML_TOK_NAME_PLUS 32              /* name+ */
-#define XML_TOK_COND_SECT_OPEN 33         /* <![ */
-#define XML_TOK_COND_SECT_CLOSE 34        /* ]]> */
-#define XML_TOK_CLOSE_PAREN_QUESTION 35   /* )? */
-#define XML_TOK_CLOSE_PAREN_ASTERISK 36   /* )* */
-#define XML_TOK_CLOSE_PAREN_PLUS 37       /* )+ */
+#define XML_TOK_NAME_QUESTION 30        /* name? */
+#define XML_TOK_NAME_ASTERISK 31        /* name* */
+#define XML_TOK_NAME_PLUS 32            /* name+ */
+#define XML_TOK_COND_SECT_OPEN 33       /* <![ */
+#define XML_TOK_COND_SECT_CLOSE 34      /* ]]> */
+#define XML_TOK_CLOSE_PAREN_QUESTION 35 /* )? */
+#define XML_TOK_CLOSE_PAREN_ASTERISK 36 /* )* */
+#define XML_TOK_CLOSE_PAREN_PLUS 37     /* )+ */
 #define XML_TOK_COMMA 38
 
 /* The following token is returned only by XmlAttributeValueTok */
@@ -112,20 +114,20 @@ extern "C" {
 #define XML_TOK_PREFIXED_NAME 41
 
 #ifdef XML_DTD
-#define XML_TOK_IGNORE_SECT 42
+#  define XML_TOK_IGNORE_SECT 42
 #endif /* XML_DTD */
 
 #ifdef XML_DTD
-#define XML_N_STATES 4
+#  define XML_N_STATES 4
 #else /* not XML_DTD */
-#define XML_N_STATES 3
+#  define XML_N_STATES 3
 #endif /* not XML_DTD */
 
 #define XML_PROLOG_STATE 0
 #define XML_CONTENT_STATE 1
 #define XML_CDATA_SECTION_STATE 2
 #ifdef XML_DTD
-#define XML_IGNORE_SECTION_STATE 3
+#  define XML_IGNORE_SECTION_STATE 3
 #endif /* XML_DTD */
 
 #define XML_N_LITERAL_TYPES 2
@@ -153,55 +155,41 @@ typedef struct {
 struct encoding;
 typedef struct encoding ENCODING;
 
-typedef int (PTRCALL *SCANNER)(const ENCODING *,
-                               const char *,
-                               const char *,
-                               const char **);
+typedef int(PTRCALL *SCANNER)(const ENCODING *, const char *, const char *,
+                              const char **);
 
 enum XML_Convert_Result {
   XML_CONVERT_COMPLETED = 0,
   XML_CONVERT_INPUT_INCOMPLETE = 1,
-  XML_CONVERT_OUTPUT_EXHAUSTED = 2  /* and therefore potentially input remaining as well */
+  XML_CONVERT_OUTPUT_EXHAUSTED
+  = 2 /* and therefore potentially input remaining as well */
 };
 
 struct encoding {
   SCANNER scanners[XML_N_STATES];
   SCANNER literalScanners[XML_N_LITERAL_TYPES];
-  int (PTRCALL *sameName)(const ENCODING *,
-                          const char *,
-                          const char *);
-  int (PTRCALL *nameMatchesAscii)(const ENCODING *,
-                                  const char *,
-                                  const char *,
-                                  const char *);
-  int (PTRFASTCALL *nameLength)(const ENCODING *, const char *);
+  int(PTRCALL *nameMatchesAscii)(const ENCODING *, const char *, const char *,
+                                 const char *);
+  int(PTRFASTCALL *nameLength)(const ENCODING *, const char *);
   const char *(PTRFASTCALL *skipS)(const ENCODING *, const char *);
-  int (PTRCALL *getAtts)(const ENCODING *enc,
-                         const char *ptr,
-                         int attsMax,
-                         ATTRIBUTE *atts);
-  int (PTRFASTCALL *charRefNumber)(const ENCODING *enc, const char *ptr);
-  int (PTRCALL *predefinedEntityName)(const ENCODING *,
-                                      const char *,
-                                      const char *);
-  void (PTRCALL *updatePosition)(const ENCODING *,
-                                 const char *ptr,
-                                 const char *end,
-                                 POSITION *);
-  int (PTRCALL *isPublicId)(const ENCODING *enc,
-                            const char *ptr,
-                            const char *end,
-                            const char **badPtr);
-  enum XML_Convert_Result (PTRCALL *utf8Convert)(const ENCODING *enc,
-                              const char **fromP,
-                              const char *fromLim,
-                              char **toP,
-                              const char *toLim);
-  enum XML_Convert_Result (PTRCALL *utf16Convert)(const ENCODING *enc,
-                               const char **fromP,
-                               const char *fromLim,
-                               unsigned short **toP,
-                               const unsigned short *toLim);
+  int(PTRCALL *getAtts)(const ENCODING *enc, const char *ptr, int attsMax,
+                        ATTRIBUTE *atts);
+  int(PTRFASTCALL *charRefNumber)(const ENCODING *enc, const char *ptr);
+  int(PTRCALL *predefinedEntityName)(const ENCODING *, const char *,
+                                     const char *);
+  void(PTRCALL *updatePosition)(const ENCODING *, const char *ptr,
+                                const char *end, POSITION *);
+  int(PTRCALL *isPublicId)(const ENCODING *enc, const char *ptr,
+                           const char *end, const char **badPtr);
+  enum XML_Convert_Result(PTRCALL *utf8Convert)(const ENCODING *enc,
+                                                const char **fromP,
+                                                const char *fromLim, char **toP,
+                                                const char *toLim);
+  enum XML_Convert_Result(PTRCALL *utf16Convert)(const ENCODING *enc,
+                                                 const char **fromP,
+                                                 const char *fromLim,
+                                                 unsigned short **toP,
+                                                 const unsigned short *toLim);
   int minBytesPerChar;
   char isUtf8;
   char isUtf16;
@@ -228,68 +216,62 @@ struct encoding {
    the prolog outside literals, comments and processing instructions.
 */
 
-
-#define XmlTok(enc, state, ptr, end, nextTokPtr) \
+#define XmlTok(enc, state, ptr, end, nextTokPtr)                               \
   (((enc)->scanners[state])(enc, ptr, end, nextTokPtr))
 
-#define XmlPrologTok(enc, ptr, end, nextTokPtr) \
-   XmlTok(enc, XML_PROLOG_STATE, ptr, end, nextTokPtr)
+#define XmlPrologTok(enc, ptr, end, nextTokPtr)                                \
+  XmlTok(enc, XML_PROLOG_STATE, ptr, end, nextTokPtr)
 
-#define XmlContentTok(enc, ptr, end, nextTokPtr) \
-   XmlTok(enc, XML_CONTENT_STATE, ptr, end, nextTokPtr)
+#define XmlContentTok(enc, ptr, end, nextTokPtr)                               \
+  XmlTok(enc, XML_CONTENT_STATE, ptr, end, nextTokPtr)
 
-#define XmlCdataSectionTok(enc, ptr, end, nextTokPtr) \
-   XmlTok(enc, XML_CDATA_SECTION_STATE, ptr, end, nextTokPtr)
+#define XmlCdataSectionTok(enc, ptr, end, nextTokPtr)                          \
+  XmlTok(enc, XML_CDATA_SECTION_STATE, ptr, end, nextTokPtr)
 
 #ifdef XML_DTD
 
-#define XmlIgnoreSectionTok(enc, ptr, end, nextTokPtr) \
-   XmlTok(enc, XML_IGNORE_SECTION_STATE, ptr, end, nextTokPtr)
+#  define XmlIgnoreSectionTok(enc, ptr, end, nextTokPtr)                       \
+    XmlTok(enc, XML_IGNORE_SECTION_STATE, ptr, end, nextTokPtr)
 
 #endif /* XML_DTD */
 
 /* This is used for performing a 2nd-level tokenization on the content
    of a literal that has already been returned by XmlTok.
 */
-#define XmlLiteralTok(enc, literalType, ptr, end, nextTokPtr) \
+#define XmlLiteralTok(enc, literalType, ptr, end, nextTokPtr)                  \
   (((enc)->literalScanners[literalType])(enc, ptr, end, nextTokPtr))
 
-#define XmlAttributeValueTok(enc, ptr, end, nextTokPtr) \
-   XmlLiteralTok(enc, XML_ATTRIBUTE_VALUE_LITERAL, ptr, end, nextTokPtr)
-
-#define XmlEntityValueTok(enc, ptr, end, nextTokPtr) \
-   XmlLiteralTok(enc, XML_ENTITY_VALUE_LITERAL, ptr, end, nextTokPtr)
+#define XmlAttributeValueTok(enc, ptr, end, nextTokPtr)                        \
+  XmlLiteralTok(enc, XML_ATTRIBUTE_VALUE_LITERAL, ptr, end, nextTokPtr)
 
-#define XmlSameName(enc, ptr1, ptr2) (((enc)->sameName)(enc, ptr1, ptr2))
+#define XmlEntityValueTok(enc, ptr, end, nextTokPtr)                           \
+  XmlLiteralTok(enc, XML_ENTITY_VALUE_LITERAL, ptr, end, nextTokPtr)
 
-#define XmlNameMatchesAscii(enc, ptr1, end1, ptr2) \
+#define XmlNameMatchesAscii(enc, ptr1, end1, ptr2)                             \
   (((enc)->nameMatchesAscii)(enc, ptr1, end1, ptr2))
 
-#define XmlNameLength(enc, ptr) \
-  (((enc)->nameLength)(enc, ptr))
+#define XmlNameLength(enc, ptr) (((enc)->nameLength)(enc, ptr))
 
-#define XmlSkipS(enc, ptr) \
-  (((enc)->skipS)(enc, ptr))
+#define XmlSkipS(enc, ptr) (((enc)->skipS)(enc, ptr))
 
-#define XmlGetAttributes(enc, ptr, attsMax, atts) \
+#define XmlGetAttributes(enc, ptr, attsMax, atts)                              \
   (((enc)->getAtts)(enc, ptr, attsMax, atts))
 
-#define XmlCharRefNumber(enc, ptr) \
-  (((enc)->charRefNumber)(enc, ptr))
+#define XmlCharRefNumber(enc, ptr) (((enc)->charRefNumber)(enc, ptr))
 
-#define XmlPredefinedEntityName(enc, ptr, end) \
+#define XmlPredefinedEntityName(enc, ptr, end)                                 \
   (((enc)->predefinedEntityName)(enc, ptr, end))
 
-#define XmlUpdatePosition(enc, ptr, end, pos) \
+#define XmlUpdatePosition(enc, ptr, end, pos)                                  \
   (((enc)->updatePosition)(enc, ptr, end, pos))
 
-#define XmlIsPublicId(enc, ptr, end, badPtr) \
+#define XmlIsPublicId(enc, ptr, end, badPtr)                                   \
   (((enc)->isPublicId)(enc, ptr, end, badPtr))
 
-#define XmlUtf8Convert(enc, fromP, fromLim, toP, toLim) \
+#define XmlUtf8Convert(enc, fromP, fromLim, toP, toLim)                        \
   (((enc)->utf8Convert)(enc, fromP, fromLim, toP, toLim))
 
-#define XmlUtf16Convert(enc, fromP, fromLim, toP, toLim) \
+#define XmlUtf16Convert(enc, fromP, fromLim, toP, toLim)                       \
   (((enc)->utf16Convert)(enc, fromP, fromLim, toP, toLim))
 
 typedef struct {
@@ -297,16 +279,11 @@ typedef struct {
   const ENCODING **encPtr;
 } INIT_ENCODING;
 
-int XmlParseXmlDecl(int isGeneralTextEntity,
-                    const ENCODING *enc,
-                    const char *ptr,
-                    const char *end,
-                    const char **badPtr,
-                    const char **versionPtr,
-                    const char **versionEndPtr,
+int XmlParseXmlDecl(int isGeneralTextEntity, const ENCODING *enc,
+                    const char *ptr, const char *end, const char **badPtr,
+                    const char **versionPtr, const char **versionEndPtr,
                     const char **encodingNamePtr,
-                    const ENCODING **namedEncodingPtr,
-                    int *standalonePtr);
+                    const ENCODING **namedEncodingPtr, int *standalonePtr);
 
 int XmlInitEncoding(INIT_ENCODING *, const ENCODING **, const char *name);
 const ENCODING *XmlGetUtf8InternalEncoding(void);
@@ -315,34 +292,22 @@ int FASTCALL XmlUtf8Encode(int charNumber, char *buf);
 int FASTCALL XmlUtf16Encode(int charNumber, unsigned short *buf);
 int XmlSizeOfUnknownEncoding(void);
 
+typedef int(XMLCALL *CONVERTER)(void *userData, const char *p);
 
-typedef int (XMLCALL *CONVERTER) (void *userData, const char *p);
-
-ENCODING *
-XmlInitUnknownEncoding(void *mem,
-                       int *table,
-                       CONVERTER convert,
-                       void *userData);
+ENCODING *XmlInitUnknownEncoding(void *mem, int *table, CONVERTER convert,
+                                 void *userData);
 
-int XmlParseXmlDeclNS(int isGeneralTextEntity,
-                      const ENCODING *enc,
-                      const char *ptr,
-                      const char *end,
-                      const char **badPtr,
-                      const char **versionPtr,
-                      const char **versionEndPtr,
+int XmlParseXmlDeclNS(int isGeneralTextEntity, const ENCODING *enc,
+                      const char *ptr, const char *end, const char **badPtr,
+                      const char **versionPtr, const char **versionEndPtr,
                       const char **encodingNamePtr,
-                      const ENCODING **namedEncodingPtr,
-                      int *standalonePtr);
+                      const ENCODING **namedEncodingPtr, int *standalonePtr);
 
 int XmlInitEncodingNS(INIT_ENCODING *, const ENCODING **, const char *name);
 const ENCODING *XmlGetUtf8InternalEncodingNS(void);
 const ENCODING *XmlGetUtf16InternalEncodingNS(void);
-ENCODING *
-XmlInitUnknownEncodingNS(void *mem,
-                         int *table,
-                         CONVERTER convert,
-                         void *userData);
+ENCODING *XmlInitUnknownEncodingNS(void *mem, int *table, CONVERTER convert,
+                                   void *userData);
 #ifdef __cplusplus
 }
 #endif
diff --git a/Modules/expat/xmltok_impl.c b/Modules/expat/xmltok_impl.c
index 93328b841a168..c209221cd79d1 100644
--- a/Modules/expat/xmltok_impl.c
+++ b/Modules/expat/xmltok_impl.c
@@ -32,128 +32,124 @@
 
 #ifdef XML_TOK_IMPL_C
 
-#ifndef IS_INVALID_CHAR
-#define IS_INVALID_CHAR(enc, ptr, n) (0)
-#endif
-
-#define INVALID_LEAD_CASE(n, ptr, nextTokPtr) \
-    case BT_LEAD ## n: \
-      if (end - ptr < n) \
-        return XML_TOK_PARTIAL_CHAR; \
-      if (IS_INVALID_CHAR(enc, ptr, n)) { \
-        *(nextTokPtr) = (ptr); \
-        return XML_TOK_INVALID; \
-      } \
-      ptr += n; \
-      break;
+#  ifndef IS_INVALID_CHAR
+#    define IS_INVALID_CHAR(enc, ptr, n) (0)
+#  endif
+
+#  define INVALID_LEAD_CASE(n, ptr, nextTokPtr)                                \
+  case BT_LEAD##n:                                                             \
+    if (end - ptr < n)                                                         \
+      return XML_TOK_PARTIAL_CHAR;                                             \
+    if (IS_INVALID_CHAR(enc, ptr, n)) {                                        \
+      *(nextTokPtr) = (ptr);                                                   \
+      return XML_TOK_INVALID;                                                  \
+    }                                                                          \
+    ptr += n;                                                                  \
+    break;
 
-#define INVALID_CASES(ptr, nextTokPtr) \
-  INVALID_LEAD_CASE(2, ptr, nextTokPtr) \
-  INVALID_LEAD_CASE(3, ptr, nextTokPtr) \
-  INVALID_LEAD_CASE(4, ptr, nextTokPtr) \
-  case BT_NONXML: \
-  case BT_MALFORM: \
-  case BT_TRAIL: \
-    *(nextTokPtr) = (ptr); \
+#  define INVALID_CASES(ptr, nextTokPtr)                                       \
+    INVALID_LEAD_CASE(2, ptr, nextTokPtr)                                      \
+    INVALID_LEAD_CASE(3, ptr, nextTokPtr)                                      \
+    INVALID_LEAD_CASE(4, ptr, nextTokPtr)                                      \
+  case BT_NONXML:                                                              \
+  case BT_MALFORM:                                                             \
+  case BT_TRAIL:                                                               \
+    *(nextTokPtr) = (ptr);                                                     \
     return XML_TOK_INVALID;
 
-#define CHECK_NAME_CASE(n, enc, ptr, end, nextTokPtr) \
-   case BT_LEAD ## n: \
-     if (end - ptr < n) \
-       return XML_TOK_PARTIAL_CHAR; \
-     if (!IS_NAME_CHAR(enc, ptr, n)) { \
-       *nextTokPtr = ptr; \
-       return XML_TOK_INVALID; \
-     } \
-     ptr += n; \
-     break;
-
-#define CHECK_NAME_CASES(enc, ptr, end, nextTokPtr) \
-  case BT_NONASCII: \
-    if (!IS_NAME_CHAR_MINBPC(enc, ptr)) { \
-      *nextTokPtr = ptr; \
-      return XML_TOK_INVALID; \
-    } \
-  case BT_NMSTRT: \
-  case BT_HEX: \
-  case BT_DIGIT: \
-  case BT_NAME: \
-  case BT_MINUS: \
-    ptr += MINBPC(enc); \
-    break; \
-  CHECK_NAME_CASE(2, enc, ptr, end, nextTokPtr) \
-  CHECK_NAME_CASE(3, enc, ptr, end, nextTokPtr) \
-  CHECK_NAME_CASE(4, enc, ptr, end, nextTokPtr)
-
-#define CHECK_NMSTRT_CASE(n, enc, ptr, end, nextTokPtr) \
-   case BT_LEAD ## n: \
-     if (end - ptr < n) \
-       return XML_TOK_PARTIAL_CHAR; \
-     if (!IS_NMSTRT_CHAR(enc, ptr, n)) { \
-       *nextTokPtr = ptr; \
-       return XML_TOK_INVALID; \
-     } \
-     ptr += n; \
-     break;
-
-#define CHECK_NMSTRT_CASES(enc, ptr, end, nextTokPtr) \
-  case BT_NONASCII: \
-    if (!IS_NMSTRT_CHAR_MINBPC(enc, ptr)) { \
-      *nextTokPtr = ptr; \
-      return XML_TOK_INVALID; \
-    } \
-  case BT_NMSTRT: \
-  case BT_HEX: \
-    ptr += MINBPC(enc); \
-    break; \
-  CHECK_NMSTRT_CASE(2, enc, ptr, end, nextTokPtr) \
-  CHECK_NMSTRT_CASE(3, enc, ptr, end, nextTokPtr) \
-  CHECK_NMSTRT_CASE(4, enc, ptr, end, nextTokPtr)
-
-#ifndef PREFIX
-#define PREFIX(ident) ident
-#endif
-
-
-#define HAS_CHARS(enc, ptr, end, count) \
-    (end - ptr >= count * MINBPC(enc))
+#  define CHECK_NAME_CASE(n, enc, ptr, end, nextTokPtr)                        \
+  case BT_LEAD##n:                                                             \
+    if (end - ptr < n)                                                         \
+      return XML_TOK_PARTIAL_CHAR;                                             \
+    if (! IS_NAME_CHAR(enc, ptr, n)) {                                         \
+      *nextTokPtr = ptr;                                                       \
+      return XML_TOK_INVALID;                                                  \
+    }                                                                          \
+    ptr += n;                                                                  \
+    break;
 
-#define HAS_CHAR(enc, ptr, end) \
-    HAS_CHARS(enc, ptr, end, 1)
+#  define CHECK_NAME_CASES(enc, ptr, end, nextTokPtr)                          \
+  case BT_NONASCII:                                                            \
+    if (! IS_NAME_CHAR_MINBPC(enc, ptr)) {                                     \
+      *nextTokPtr = ptr;                                                       \
+      return XML_TOK_INVALID;                                                  \
+    }                                                                          \
+    /* fall through */                                                         \
+  case BT_NMSTRT:                                                              \
+  case BT_HEX:                                                                 \
+  case BT_DIGIT:                                                               \
+  case BT_NAME:                                                                \
+  case BT_MINUS:                                                               \
+    ptr += MINBPC(enc);                                                        \
+    break;                                                                     \
+    CHECK_NAME_CASE(2, enc, ptr, end, nextTokPtr)                              \
+    CHECK_NAME_CASE(3, enc, ptr, end, nextTokPtr)                              \
+    CHECK_NAME_CASE(4, enc, ptr, end, nextTokPtr)
+
+#  define CHECK_NMSTRT_CASE(n, enc, ptr, end, nextTokPtr)                      \
+  case BT_LEAD##n:                                                             \
+    if (end - ptr < n)                                                         \
+      return XML_TOK_PARTIAL_CHAR;                                             \
+    if (! IS_NMSTRT_CHAR(enc, ptr, n)) {                                       \
+      *nextTokPtr = ptr;                                                       \
+      return XML_TOK_INVALID;                                                  \
+    }                                                                          \
+    ptr += n;                                                                  \
+    break;
 
-#define REQUIRE_CHARS(enc, ptr, end, count) \
-    { \
-      if (! HAS_CHARS(enc, ptr, end, count)) { \
-        return XML_TOK_PARTIAL; \
-      } \
+#  define CHECK_NMSTRT_CASES(enc, ptr, end, nextTokPtr)                        \
+  case BT_NONASCII:                                                            \
+    if (! IS_NMSTRT_CHAR_MINBPC(enc, ptr)) {                                   \
+      *nextTokPtr = ptr;                                                       \
+      return XML_TOK_INVALID;                                                  \
+    }                                                                          \
+    /* fall through */                                                         \
+  case BT_NMSTRT:                                                              \
+  case BT_HEX:                                                                 \
+    ptr += MINBPC(enc);                                                        \
+    break;                                                                     \
+    CHECK_NMSTRT_CASE(2, enc, ptr, end, nextTokPtr)                            \
+    CHECK_NMSTRT_CASE(3, enc, ptr, end, nextTokPtr)                            \
+    CHECK_NMSTRT_CASE(4, enc, ptr, end, nextTokPtr)
+
+#  ifndef PREFIX
+#    define PREFIX(ident) ident
+#  endif
+
+#  define HAS_CHARS(enc, ptr, end, count) (end - ptr >= count * MINBPC(enc))
+
+#  define HAS_CHAR(enc, ptr, end) HAS_CHARS(enc, ptr, end, 1)
+
+#  define REQUIRE_CHARS(enc, ptr, end, count)                                  \
+    {                                                                          \
+      if (! HAS_CHARS(enc, ptr, end, count)) {                                 \
+        return XML_TOK_PARTIAL;                                                \
+      }                                                                        \
     }
 
-#define REQUIRE_CHAR(enc, ptr, end) \
-    REQUIRE_CHARS(enc, ptr, end, 1)
-
+#  define REQUIRE_CHAR(enc, ptr, end) REQUIRE_CHARS(enc, ptr, end, 1)
 
 /* ptr points to character following "<!-" */
 
 static int PTRCALL
-PREFIX(scanComment)(const ENCODING *enc, const char *ptr,
-                    const char *end, const char **nextTokPtr)
-{
+PREFIX(scanComment)(const ENCODING *enc, const char *ptr, const char *end,
+                    const char **nextTokPtr) {
   if (HAS_CHAR(enc, ptr, end)) {
-    if (!CHAR_MATCHES(enc, ptr, ASCII_MINUS)) {
+    if (! CHAR_MATCHES(enc, ptr, ASCII_MINUS)) {
       *nextTokPtr = ptr;
       return XML_TOK_INVALID;
     }
     ptr += MINBPC(enc);
     while (HAS_CHAR(enc, ptr, end)) {
       switch (BYTE_TYPE(enc, ptr)) {
-      INVALID_CASES(ptr, nextTokPtr)
+        INVALID_CASES(ptr, nextTokPtr)
       case BT_MINUS:
         ptr += MINBPC(enc);
         REQUIRE_CHAR(enc, ptr, end);
         if (CHAR_MATCHES(enc, ptr, ASCII_MINUS)) {
           ptr += MINBPC(enc);
           REQUIRE_CHAR(enc, ptr, end);
-          if (!CHAR_MATCHES(enc, ptr, ASCII_GT)) {
+          if (! CHAR_MATCHES(enc, ptr, ASCII_GT)) {
             *nextTokPtr = ptr;
             return XML_TOK_INVALID;
           }
@@ -173,9 +169,8 @@ PREFIX(scanComment)(const ENCODING *enc, const char *ptr,
 /* ptr points to character following "<!" */
 
 static int PTRCALL
-PREFIX(scanDecl)(const ENCODING *enc, const char *ptr,
-                 const char *end, const char **nextTokPtr)
-{
+PREFIX(scanDecl)(const ENCODING *enc, const char *ptr, const char *end,
+                 const char **nextTokPtr) {
   REQUIRE_CHAR(enc, ptr, end);
   switch (BYTE_TYPE(enc, ptr)) {
   case BT_MINUS:
@@ -197,12 +192,17 @@ PREFIX(scanDecl)(const ENCODING *enc, const char *ptr,
       REQUIRE_CHARS(enc, ptr, end, 2);
       /* don't allow <!ENTITY% foo "whatever"> */
       switch (BYTE_TYPE(enc, ptr + MINBPC(enc))) {
-      case BT_S: case BT_CR: case BT_LF: case BT_PERCNT:
+      case BT_S:
+      case BT_CR:
+      case BT_LF:
+      case BT_PERCNT:
         *nextTokPtr = ptr;
         return XML_TOK_INVALID;
       }
       /* fall through */
-    case BT_S: case BT_CR: case BT_LF:
+    case BT_S:
+    case BT_CR:
+    case BT_LF:
       *nextTokPtr = ptr;
       return XML_TOK_DECL_OPEN;
     case BT_NMSTRT:
@@ -218,12 +218,12 @@ PREFIX(scanDecl)(const ENCODING *enc, const char *ptr,
 }
 
 static int PTRCALL
-PREFIX(checkPiTarget)(const ENCODING *UNUSED_P(enc), const char *ptr,
-                      const char *end, int *tokPtr)
-{
+PREFIX(checkPiTarget)(const ENCODING *enc, const char *ptr, const char *end,
+                      int *tokPtr) {
   int upper = 0;
+  UNUSED_P(enc);
   *tokPtr = XML_TOK_PI;
-  if (end - ptr != MINBPC(enc)*3)
+  if (end - ptr != MINBPC(enc) * 3)
     return 1;
   switch (BYTE_TO_ASCII(enc, ptr)) {
   case ASCII_x:
@@ -263,30 +263,31 @@ PREFIX(checkPiTarget)(const ENCODING *UNUSED_P(enc), const char *ptr,
 /* ptr points to character following "<?" */
 
 static int PTRCALL
-PREFIX(scanPi)(const ENCODING *enc, const char *ptr,
-               const char *end, const char **nextTokPtr)
-{
+PREFIX(scanPi)(const ENCODING *enc, const char *ptr, const char *end,
+               const char **nextTokPtr) {
   int tok;
   const char *target = ptr;
   REQUIRE_CHAR(enc, ptr, end);
   switch (BYTE_TYPE(enc, ptr)) {
-  CHECK_NMSTRT_CASES(enc, ptr, end, nextTokPtr)
+    CHECK_NMSTRT_CASES(enc, ptr, end, nextTokPtr)
   default:
     *nextTokPtr = ptr;
     return XML_TOK_INVALID;
   }
   while (HAS_CHAR(enc, ptr, end)) {
     switch (BYTE_TYPE(enc, ptr)) {
-    CHECK_NAME_CASES(enc, ptr, end, nextTokPtr)
-    case BT_S: case BT_CR: case BT_LF:
-      if (!PREFIX(checkPiTarget)(enc, target, ptr, &tok)) {
+      CHECK_NAME_CASES(enc, ptr, end, nextTokPtr)
+    case BT_S:
+    case BT_CR:
+    case BT_LF:
+      if (! PREFIX(checkPiTarget)(enc, target, ptr, &tok)) {
         *nextTokPtr = ptr;
         return XML_TOK_INVALID;
       }
       ptr += MINBPC(enc);
       while (HAS_CHAR(enc, ptr, end)) {
         switch (BYTE_TYPE(enc, ptr)) {
-        INVALID_CASES(ptr, nextTokPtr)
+          INVALID_CASES(ptr, nextTokPtr)
         case BT_QUEST:
           ptr += MINBPC(enc);
           REQUIRE_CHAR(enc, ptr, end);
@@ -302,7 +303,7 @@ PREFIX(scanPi)(const ENCODING *enc, const char *ptr,
       }
       return XML_TOK_PARTIAL;
     case BT_QUEST:
-      if (!PREFIX(checkPiTarget)(enc, target, ptr, &tok)) {
+      if (! PREFIX(checkPiTarget)(enc, target, ptr, &tok)) {
         *nextTokPtr = ptr;
         return XML_TOK_INVALID;
       }
@@ -322,16 +323,16 @@ PREFIX(scanPi)(const ENCODING *enc, const char *ptr,
 }
 
 static int PTRCALL
-PREFIX(scanCdataSection)(const ENCODING *UNUSED_P(enc), const char *ptr,
-                         const char *end, const char **nextTokPtr)
-{
-  static const char CDATA_LSQB[] = { ASCII_C, ASCII_D, ASCII_A,
-                                     ASCII_T, ASCII_A, ASCII_LSQB };
+PREFIX(scanCdataSection)(const ENCODING *enc, const char *ptr, const char *end,
+                         const char **nextTokPtr) {
+  static const char CDATA_LSQB[]
+      = {ASCII_C, ASCII_D, ASCII_A, ASCII_T, ASCII_A, ASCII_LSQB};
   int i;
+  UNUSED_P(enc);
   /* CDATA[ */
   REQUIRE_CHARS(enc, ptr, end, 6);
   for (i = 0; i < 6; i++, ptr += MINBPC(enc)) {
-    if (!CHAR_MATCHES(enc, ptr, CDATA_LSQB[i])) {
+    if (! CHAR_MATCHES(enc, ptr, CDATA_LSQB[i])) {
       *nextTokPtr = ptr;
       return XML_TOK_INVALID;
     }
@@ -341,9 +342,8 @@ PREFIX(scanCdataSection)(const ENCODING *UNUSED_P(enc), const char *ptr,
 }
 
 static int PTRCALL
-PREFIX(cdataSectionTok)(const ENCODING *enc, const char *ptr,
-                        const char *end, const char **nextTokPtr)
-{
+PREFIX(cdataSectionTok)(const ENCODING *enc, const char *ptr, const char *end,
+                        const char **nextTokPtr) {
   if (ptr >= end)
     return XML_TOK_NONE;
   if (MINBPC(enc) > 1) {
@@ -359,11 +359,11 @@ PREFIX(cdataSectionTok)(const ENCODING *enc, const char *ptr,
   case BT_RSQB:
     ptr += MINBPC(enc);
     REQUIRE_CHAR(enc, ptr, end);
-    if (!CHAR_MATCHES(enc, ptr, ASCII_RSQB))
+    if (! CHAR_MATCHES(enc, ptr, ASCII_RSQB))
       break;
     ptr += MINBPC(enc);
     REQUIRE_CHAR(enc, ptr, end);
-    if (!CHAR_MATCHES(enc, ptr, ASCII_GT)) {
+    if (! CHAR_MATCHES(enc, ptr, ASCII_GT)) {
       ptr -= MINBPC(enc);
       break;
     }
@@ -379,23 +379,25 @@ PREFIX(cdataSectionTok)(const ENCODING *enc, const char *ptr,
   case BT_LF:
     *nextTokPtr = ptr + MINBPC(enc);
     return XML_TOK_DATA_NEWLINE;
-  INVALID_CASES(ptr, nextTokPtr)
+    INVALID_CASES(ptr, nextTokPtr)
   default:
     ptr += MINBPC(enc);
     break;
   }
   while (HAS_CHAR(enc, ptr, end)) {
     switch (BYTE_TYPE(enc, ptr)) {
-#define LEAD_CASE(n) \
-    case BT_LEAD ## n: \
-      if (end - ptr < n || IS_INVALID_CHAR(enc, ptr, n)) { \
-        *nextTokPtr = ptr; \
-        return XML_TOK_DATA_CHARS; \
-      } \
-      ptr += n; \
-      break;
-    LEAD_CASE(2) LEAD_CASE(3) LEAD_CASE(4)
-#undef LEAD_CASE
+#  define LEAD_CASE(n)                                                         \
+  case BT_LEAD##n:                                                             \
+    if (end - ptr < n || IS_INVALID_CHAR(enc, ptr, n)) {                       \
+      *nextTokPtr = ptr;                                                       \
+      return XML_TOK_DATA_CHARS;                                               \
+    }                                                                          \
+    ptr += n;                                                                  \
+    break;
+      LEAD_CASE(2)
+      LEAD_CASE(3)
+      LEAD_CASE(4)
+#  undef LEAD_CASE
     case BT_NONXML:
     case BT_MALFORM:
     case BT_TRAIL:
@@ -416,23 +418,26 @@ PREFIX(cdataSectionTok)(const ENCODING *enc, const char *ptr,
 /* ptr points to character following "</" */
 
 static int PTRCALL
-PREFIX(scanEndTag)(const ENCODING *enc, const char *ptr,
-                   const char *end, const char **nextTokPtr)
-{
+PREFIX(scanEndTag)(const ENCODING *enc, const char *ptr, const char *end,
+                   const char **nextTokPtr) {
   REQUIRE_CHAR(enc, ptr, end);
   switch (BYTE_TYPE(enc, ptr)) {
-  CHECK_NMSTRT_CASES(enc, ptr, end, nextTokPtr)
+    CHECK_NMSTRT_CASES(enc, ptr, end, nextTokPtr)
   default:
     *nextTokPtr = ptr;
     return XML_TOK_INVALID;
   }
   while (HAS_CHAR(enc, ptr, end)) {
     switch (BYTE_TYPE(enc, ptr)) {
-    CHECK_NAME_CASES(enc, ptr, end, nextTokPtr)
-    case BT_S: case BT_CR: case BT_LF:
+      CHECK_NAME_CASES(enc, ptr, end, nextTokPtr)
+    case BT_S:
+    case BT_CR:
+    case BT_LF:
       for (ptr += MINBPC(enc); HAS_CHAR(enc, ptr, end); ptr += MINBPC(enc)) {
         switch (BYTE_TYPE(enc, ptr)) {
-        case BT_S: case BT_CR: case BT_LF:
+        case BT_S:
+        case BT_CR:
+        case BT_LF:
           break;
         case BT_GT:
           *nextTokPtr = ptr + MINBPC(enc);
@@ -443,13 +448,13 @@ PREFIX(scanEndTag)(const ENCODING *enc, const char *ptr,
         }
       }
       return XML_TOK_PARTIAL;
-#ifdef XML_NS
+#  ifdef XML_NS
     case BT_COLON:
       /* no need to check qname syntax here,
          since end-tag must match exactly */
       ptr += MINBPC(enc);
       break;
-#endif
+#  endif
     case BT_GT:
       *nextTokPtr = ptr + MINBPC(enc);
       return XML_TOK_END_TAG;
@@ -464,9 +469,8 @@ PREFIX(scanEndTag)(const ENCODING *enc, const char *ptr,
 /* ptr points to character following "&#X" */
 
 static int PTRCALL
-PREFIX(scanHexCharRef)(const ENCODING *enc, const char *ptr,
-                       const char *end, const char **nextTokPtr)
-{
+PREFIX(scanHexCharRef)(const ENCODING *enc, const char *ptr, const char *end,
+                       const char **nextTokPtr) {
   if (HAS_CHAR(enc, ptr, end)) {
     switch (BYTE_TYPE(enc, ptr)) {
     case BT_DIGIT:
@@ -496,9 +500,8 @@ PREFIX(scanHexCharRef)(const ENCODING *enc, const char *ptr,
 /* ptr points to character following "&#" */
 
 static int PTRCALL
-PREFIX(scanCharRef)(const ENCODING *enc, const char *ptr,
-                    const char *end, const char **nextTokPtr)
-{
+PREFIX(scanCharRef)(const ENCODING *enc, const char *ptr, const char *end,
+                    const char **nextTokPtr) {
   if (HAS_CHAR(enc, ptr, end)) {
     if (CHAR_MATCHES(enc, ptr, ASCII_x))
       return PREFIX(scanHexCharRef)(enc, ptr + MINBPC(enc), end, nextTokPtr);
@@ -529,11 +532,10 @@ PREFIX(scanCharRef)(const ENCODING *enc, const char *ptr,
 
 static int PTRCALL
 PREFIX(scanRef)(const ENCODING *enc, const char *ptr, const char *end,
-                const char **nextTokPtr)
-{
+                const char **nextTokPtr) {
   REQUIRE_CHAR(enc, ptr, end);
   switch (BYTE_TYPE(enc, ptr)) {
-  CHECK_NMSTRT_CASES(enc, ptr, end, nextTokPtr)
+    CHECK_NMSTRT_CASES(enc, ptr, end, nextTokPtr)
   case BT_NUM:
     return PREFIX(scanCharRef)(enc, ptr + MINBPC(enc), end, nextTokPtr);
   default:
@@ -542,7 +544,7 @@ PREFIX(scanRef)(const ENCODING *enc, const char *ptr, const char *end,
   }
   while (HAS_CHAR(enc, ptr, end)) {
     switch (BYTE_TYPE(enc, ptr)) {
-    CHECK_NAME_CASES(enc, ptr, end, nextTokPtr)
+      CHECK_NAME_CASES(enc, ptr, end, nextTokPtr)
     case BT_SEMI:
       *nextTokPtr = ptr + MINBPC(enc);
       return XML_TOK_ENTITY_REF;
@@ -558,15 +560,14 @@ PREFIX(scanRef)(const ENCODING *enc, const char *ptr, const char *end,
 
 static int PTRCALL
 PREFIX(scanAtts)(const ENCODING *enc, const char *ptr, const char *end,
-                 const char **nextTokPtr)
-{
-#ifdef XML_NS
+                 const char **nextTokPtr) {
+#  ifdef XML_NS
   int hadColon = 0;
-#endif
+#  endif
   while (HAS_CHAR(enc, ptr, end)) {
     switch (BYTE_TYPE(enc, ptr)) {
-    CHECK_NAME_CASES(enc, ptr, end, nextTokPtr)
-#ifdef XML_NS
+      CHECK_NAME_CASES(enc, ptr, end, nextTokPtr)
+#  ifdef XML_NS
     case BT_COLON:
       if (hadColon) {
         *nextTokPtr = ptr;
@@ -576,14 +577,16 @@ PREFIX(scanAtts)(const ENCODING *enc, const char *ptr, const char *end,
       ptr += MINBPC(enc);
       REQUIRE_CHAR(enc, ptr, end);
       switch (BYTE_TYPE(enc, ptr)) {
-      CHECK_NMSTRT_CASES(enc, ptr, end, nextTokPtr)
+        CHECK_NMSTRT_CASES(enc, ptr, end, nextTokPtr)
       default:
         *nextTokPtr = ptr;
         return XML_TOK_INVALID;
       }
       break;
-#endif
-    case BT_S: case BT_CR: case BT_LF:
+#  endif
+    case BT_S:
+    case BT_CR:
+    case BT_LF:
       for (;;) {
         int t;
 
@@ -602,102 +605,102 @@ PREFIX(scanAtts)(const ENCODING *enc, const char *ptr, const char *end,
           return XML_TOK_INVALID;
         }
       }
-    /* fall through */
-    case BT_EQUALS:
-      {
-        int open;
-#ifdef XML_NS
-        hadColon = 0;
-#endif
-        for (;;) {
-          ptr += MINBPC(enc);
-          REQUIRE_CHAR(enc, ptr, end);
-          open = BYTE_TYPE(enc, ptr);
-          if (open == BT_QUOT || open == BT_APOS)
-            break;
-          switch (open) {
-          case BT_S:
-          case BT_LF:
-          case BT_CR:
-            break;
-          default:
-            *nextTokPtr = ptr;
-            return XML_TOK_INVALID;
-          }
-        }
+      /* fall through */
+    case BT_EQUALS: {
+      int open;
+#  ifdef XML_NS
+      hadColon = 0;
+#  endif
+      for (;;) {
         ptr += MINBPC(enc);
-        /* in attribute value */
-        for (;;) {
-          int t;
-          REQUIRE_CHAR(enc, ptr, end);
-          t = BYTE_TYPE(enc, ptr);
-          if (t == open)
-            break;
-          switch (t) {
+        REQUIRE_CHAR(enc, ptr, end);
+        open = BYTE_TYPE(enc, ptr);
+        if (open == BT_QUOT || open == BT_APOS)
+          break;
+        switch (open) {
+        case BT_S:
+        case BT_LF:
+        case BT_CR:
+          break;
+        default:
+          *nextTokPtr = ptr;
+          return XML_TOK_INVALID;
+        }
+      }
+      ptr += MINBPC(enc);
+      /* in attribute value */
+      for (;;) {
+        int t;
+        REQUIRE_CHAR(enc, ptr, end);
+        t = BYTE_TYPE(enc, ptr);
+        if (t == open)
+          break;
+        switch (t) {
           INVALID_CASES(ptr, nextTokPtr)
-          case BT_AMP:
-            {
-              int tok = PREFIX(scanRef)(enc, ptr + MINBPC(enc), end, &ptr);
-              if (tok <= 0) {
-                if (tok == XML_TOK_INVALID)
-                  *nextTokPtr = ptr;
-                return tok;
-              }
-              break;
-            }
-          case BT_LT:
-            *nextTokPtr = ptr;
-            return XML_TOK_INVALID;
-          default:
-            ptr += MINBPC(enc);
-            break;
+        case BT_AMP: {
+          int tok = PREFIX(scanRef)(enc, ptr + MINBPC(enc), end, &ptr);
+          if (tok <= 0) {
+            if (tok == XML_TOK_INVALID)
+              *nextTokPtr = ptr;
+            return tok;
           }
+          break;
+        }
+        case BT_LT:
+          *nextTokPtr = ptr;
+          return XML_TOK_INVALID;
+        default:
+          ptr += MINBPC(enc);
+          break;
         }
+      }
+      ptr += MINBPC(enc);
+      REQUIRE_CHAR(enc, ptr, end);
+      switch (BYTE_TYPE(enc, ptr)) {
+      case BT_S:
+      case BT_CR:
+      case BT_LF:
+        break;
+      case BT_SOL:
+        goto sol;
+      case BT_GT:
+        goto gt;
+      default:
+        *nextTokPtr = ptr;
+        return XML_TOK_INVALID;
+      }
+      /* ptr points to closing quote */
+      for (;;) {
         ptr += MINBPC(enc);
         REQUIRE_CHAR(enc, ptr, end);
         switch (BYTE_TYPE(enc, ptr)) {
+          CHECK_NMSTRT_CASES(enc, ptr, end, nextTokPtr)
         case BT_S:
         case BT_CR:
         case BT_LF:
-          break;
-        case BT_SOL:
-          goto sol;
+          continue;
         case BT_GT:
-          goto gt;
-        default:
-          *nextTokPtr = ptr;
-          return XML_TOK_INVALID;
-        }
-        /* ptr points to closing quote */
-        for (;;) {
+        gt:
+          *nextTokPtr = ptr + MINBPC(enc);
+          return XML_TOK_START_TAG_WITH_ATTS;
+        case BT_SOL:
+        sol:
           ptr += MINBPC(enc);
           REQUIRE_CHAR(enc, ptr, end);
-          switch (BYTE_TYPE(enc, ptr)) {
-          CHECK_NMSTRT_CASES(enc, ptr, end, nextTokPtr)
-          case BT_S: case BT_CR: case BT_LF:
-            continue;
-          case BT_GT:
-          gt:
-            *nextTokPtr = ptr + MINBPC(enc);
-            return XML_TOK_START_TAG_WITH_ATTS;
-          case BT_SOL:
-          sol:
-            ptr += MINBPC(enc);
-            REQUIRE_CHAR(enc, ptr, end);
-            if (!CHAR_MATCHES(enc, ptr, ASCII_GT)) {
-              *nextTokPtr = ptr;
-              return XML_TOK_INVALID;
-            }
-            *nextTokPtr = ptr + MINBPC(enc);
-            return XML_TOK_EMPTY_ELEMENT_WITH_ATTS;
-          default:
+          if (! CHAR_MATCHES(enc, ptr, ASCII_GT)) {
             *nextTokPtr = ptr;
             return XML_TOK_INVALID;
           }
-          break;
+          *nextTokPtr = ptr + MINBPC(enc);
+          return XML_TOK_EMPTY_ELEMENT_WITH_ATTS;
+        default:
+          *nextTokPtr = ptr;
+          return XML_TOK_INVALID;
         }
         break;
       }
+      break;
+    }
     default:
       *nextTokPtr = ptr;
       return XML_TOK_INVALID;
@@ -710,14 +713,13 @@ PREFIX(scanAtts)(const ENCODING *enc, const char *ptr, const char *end,
 
 static int PTRCALL
 PREFIX(scanLt)(const ENCODING *enc, const char *ptr, const char *end,
-               const char **nextTokPtr)
-{
-#ifdef XML_NS
+               const char **nextTokPtr) {
+#  ifdef XML_NS
   int hadColon;
-#endif
+#  endif
   REQUIRE_CHAR(enc, ptr, end);
   switch (BYTE_TYPE(enc, ptr)) {
-  CHECK_NMSTRT_CASES(enc, ptr, end, nextTokPtr)
+    CHECK_NMSTRT_CASES(enc, ptr, end, nextTokPtr)
   case BT_EXCL:
     ptr += MINBPC(enc);
     REQUIRE_CHAR(enc, ptr, end);
@@ -725,8 +727,7 @@ PREFIX(scanLt)(const ENCODING *enc, const char *ptr, const char *end,
     case BT_MINUS:
       return PREFIX(scanComment)(enc, ptr + MINBPC(enc), end, nextTokPtr);
     case BT_LSQB:
-      return PREFIX(scanCdataSection)(enc, ptr + MINBPC(enc),
-                                      end, nextTokPtr);
+      return PREFIX(scanCdataSection)(enc, ptr + MINBPC(enc), end, nextTokPtr);
     }
     *nextTokPtr = ptr;
     return XML_TOK_INVALID;
@@ -738,14 +739,14 @@ PREFIX(scanLt)(const ENCODING *enc, const char *ptr, const char *end,
     *nextTokPtr = ptr;
     return XML_TOK_INVALID;
   }
-#ifdef XML_NS
+#  ifdef XML_NS
   hadColon = 0;
-#endif
+#  endif
   /* we have a start-tag */
   while (HAS_CHAR(enc, ptr, end)) {
     switch (BYTE_TYPE(enc, ptr)) {
-    CHECK_NAME_CASES(enc, ptr, end, nextTokPtr)
-#ifdef XML_NS
+      CHECK_NAME_CASES(enc, ptr, end, nextTokPtr)
+#  ifdef XML_NS
     case BT_COLON:
       if (hadColon) {
         *nextTokPtr = ptr;
@@ -755,34 +756,37 @@ PREFIX(scanLt)(const ENCODING *enc, const char *ptr, const char *end,
       ptr += MINBPC(enc);
       REQUIRE_CHAR(enc, ptr, end);
       switch (BYTE_TYPE(enc, ptr)) {
-      CHECK_NMSTRT_CASES(enc, ptr, end, nextTokPtr)
+        CHECK_NMSTRT_CASES(enc, ptr, end, nextTokPtr)
       default:
         *nextTokPtr = ptr;
         return XML_TOK_INVALID;
       }
       break;
-#endif
-    case BT_S: case BT_CR: case BT_LF:
-      {
-        ptr += MINBPC(enc);
-        while (HAS_CHAR(enc, ptr, end)) {
-          switch (BYTE_TYPE(enc, ptr)) {
+#  endif
+    case BT_S:
+    case BT_CR:
+    case BT_LF: {
+      ptr += MINBPC(enc);
+      while (HAS_CHAR(enc, ptr, end)) {
+        switch (BYTE_TYPE(enc, ptr)) {
           CHECK_NMSTRT_CASES(enc, ptr, end, nextTokPtr)
-          case BT_GT:
-            goto gt;
-          case BT_SOL:
-            goto sol;
-          case BT_S: case BT_CR: case BT_LF:
-            ptr += MINBPC(enc);
-            continue;
-          default:
-            *nextTokPtr = ptr;
-            return XML_TOK_INVALID;
-          }
-          return PREFIX(scanAtts)(enc, ptr, end, nextTokPtr);
+        case BT_GT:
+          goto gt;
+        case BT_SOL:
+          goto sol;
+        case BT_S:
+        case BT_CR:
+        case BT_LF:
+          ptr += MINBPC(enc);
+          continue;
+        default:
+          *nextTokPtr = ptr;
+          return XML_TOK_INVALID;
         }
-        return XML_TOK_PARTIAL;
+        return PREFIX(scanAtts)(enc, ptr, end, nextTokPtr);
       }
+      return XML_TOK_PARTIAL;
+    }
     case BT_GT:
     gt:
       *nextTokPtr = ptr + MINBPC(enc);
@@ -791,7 +795,7 @@ PREFIX(scanLt)(const ENCODING *enc, const char *ptr, const char *end,
     sol:
       ptr += MINBPC(enc);
       REQUIRE_CHAR(enc, ptr, end);
-      if (!CHAR_MATCHES(enc, ptr, ASCII_GT)) {
+      if (! CHAR_MATCHES(enc, ptr, ASCII_GT)) {
         *nextTokPtr = ptr;
         return XML_TOK_INVALID;
       }
@@ -807,8 +811,7 @@ PREFIX(scanLt)(const ENCODING *enc, const char *ptr, const char *end,
 
 static int PTRCALL
 PREFIX(contentTok)(const ENCODING *enc, const char *ptr, const char *end,
-                   const char **nextTokPtr)
-{
+                   const char **nextTokPtr) {
   if (ptr >= end)
     return XML_TOK_NONE;
   if (MINBPC(enc) > 1) {
@@ -840,48 +843,50 @@ PREFIX(contentTok)(const ENCODING *enc, const char *ptr, const char *end,
     ptr += MINBPC(enc);
     if (! HAS_CHAR(enc, ptr, end))
       return XML_TOK_TRAILING_RSQB;
-    if (!CHAR_MATCHES(enc, ptr, ASCII_RSQB))
+    if (! CHAR_MATCHES(enc, ptr, ASCII_RSQB))
       break;
     ptr += MINBPC(enc);
     if (! HAS_CHAR(enc, ptr, end))
       return XML_TOK_TRAILING_RSQB;
-    if (!CHAR_MATCHES(enc, ptr, ASCII_GT)) {
+    if (! CHAR_MATCHES(enc, ptr, ASCII_GT)) {
       ptr -= MINBPC(enc);
       break;
     }
     *nextTokPtr = ptr;
     return XML_TOK_INVALID;
-  INVALID_CASES(ptr, nextTokPtr)
+    INVALID_CASES(ptr, nextTokPtr)
   default:
     ptr += MINBPC(enc);
     break;
   }
   while (HAS_CHAR(enc, ptr, end)) {
     switch (BYTE_TYPE(enc, ptr)) {
-#define LEAD_CASE(n) \
-    case BT_LEAD ## n: \
-      if (end - ptr < n || IS_INVALID_CHAR(enc, ptr, n)) { \
-        *nextTokPtr = ptr; \
-        return XML_TOK_DATA_CHARS; \
-      } \
-      ptr += n; \
-      break;
-    LEAD_CASE(2) LEAD_CASE(3) LEAD_CASE(4)
-#undef LEAD_CASE
+#  define LEAD_CASE(n)                                                         \
+  case BT_LEAD##n:                                                             \
+    if (end - ptr < n || IS_INVALID_CHAR(enc, ptr, n)) {                       \
+      *nextTokPtr = ptr;                                                       \
+      return XML_TOK_DATA_CHARS;                                               \
+    }                                                                          \
+    ptr += n;                                                                  \
+    break;
+      LEAD_CASE(2)
+      LEAD_CASE(3)
+      LEAD_CASE(4)
+#  undef LEAD_CASE
     case BT_RSQB:
       if (HAS_CHARS(enc, ptr, end, 2)) {
-         if (!CHAR_MATCHES(enc, ptr + MINBPC(enc), ASCII_RSQB)) {
-           ptr += MINBPC(enc);
-           break;
-         }
-         if (HAS_CHARS(enc, ptr, end, 3)) {
-           if (!CHAR_MATCHES(enc, ptr + 2*MINBPC(enc), ASCII_GT)) {
-             ptr += MINBPC(enc);
-             break;
-           }
-           *nextTokPtr = ptr + 2*MINBPC(enc);
-           return XML_TOK_INVALID;
-         }
+        if (! CHAR_MATCHES(enc, ptr + MINBPC(enc), ASCII_RSQB)) {
+          ptr += MINBPC(enc);
+          break;
+        }
+        if (HAS_CHARS(enc, ptr, end, 3)) {
+          if (! CHAR_MATCHES(enc, ptr + 2 * MINBPC(enc), ASCII_GT)) {
+            ptr += MINBPC(enc);
+            break;
+          }
+          *nextTokPtr = ptr + 2 * MINBPC(enc);
+          return XML_TOK_INVALID;
+        }
       }
       /* fall through */
     case BT_AMP:
@@ -906,12 +911,14 @@ PREFIX(contentTok)(const ENCODING *enc, const char *ptr, const char *end,
 
 static int PTRCALL
 PREFIX(scanPercent)(const ENCODING *enc, const char *ptr, const char *end,
-                    const char **nextTokPtr)
-{
+                    const char **nextTokPtr) {
   REQUIRE_CHAR(enc, ptr, end);
   switch (BYTE_TYPE(enc, ptr)) {
-  CHECK_NMSTRT_CASES(enc, ptr, end, nextTokPtr)
-  case BT_S: case BT_LF: case BT_CR: case BT_PERCNT:
+    CHECK_NMSTRT_CASES(enc, ptr, end, nextTokPtr)
+  case BT_S:
+  case BT_LF:
+  case BT_CR:
+  case BT_PERCNT:
     *nextTokPtr = ptr;
     return XML_TOK_PERCENT;
   default:
@@ -920,7 +927,7 @@ PREFIX(scanPercent)(const ENCODING *enc, const char *ptr, const char *end,
   }
   while (HAS_CHAR(enc, ptr, end)) {
     switch (BYTE_TYPE(enc, ptr)) {
-    CHECK_NAME_CASES(enc, ptr, end, nextTokPtr)
+      CHECK_NAME_CASES(enc, ptr, end, nextTokPtr)
     case BT_SEMI:
       *nextTokPtr = ptr + MINBPC(enc);
       return XML_TOK_PARAM_ENTITY_REF;
@@ -934,20 +941,24 @@ PREFIX(scanPercent)(const ENCODING *enc, const char *ptr, const char *end,
 
 static int PTRCALL
 PREFIX(scanPoundName)(const ENCODING *enc, const char *ptr, const char *end,
-                      const char **nextTokPtr)
-{
+                      const char **nextTokPtr) {
   REQUIRE_CHAR(enc, ptr, end);
   switch (BYTE_TYPE(enc, ptr)) {
-  CHECK_NMSTRT_CASES(enc, ptr, end, nextTokPtr)
+    CHECK_NMSTRT_CASES(enc, ptr, end, nextTokPtr)
   default:
     *nextTokPtr = ptr;
     return XML_TOK_INVALID;
   }
   while (HAS_CHAR(enc, ptr, end)) {
     switch (BYTE_TYPE(enc, ptr)) {
-    CHECK_NAME_CASES(enc, ptr, end, nextTokPtr)
-    case BT_CR: case BT_LF: case BT_S:
-    case BT_RPAR: case BT_GT: case BT_PERCNT: case BT_VERBAR:
+      CHECK_NAME_CASES(enc, ptr, end, nextTokPtr)
+    case BT_CR:
+    case BT_LF:
+    case BT_S:
+    case BT_RPAR:
+    case BT_GT:
+    case BT_PERCNT:
+    case BT_VERBAR:
       *nextTokPtr = ptr;
       return XML_TOK_POUND_NAME;
     default:
@@ -959,14 +970,12 @@ PREFIX(scanPoundName)(const ENCODING *enc, const char *ptr, const char *end,
 }
 
 static int PTRCALL
-PREFIX(scanLit)(int open, const ENCODING *enc,
-                const char *ptr, const char *end,
-                const char **nextTokPtr)
-{
+PREFIX(scanLit)(int open, const ENCODING *enc, const char *ptr, const char *end,
+                const char **nextTokPtr) {
   while (HAS_CHAR(enc, ptr, end)) {
     int t = BYTE_TYPE(enc, ptr);
     switch (t) {
-    INVALID_CASES(ptr, nextTokPtr)
+      INVALID_CASES(ptr, nextTokPtr)
     case BT_QUOT:
     case BT_APOS:
       ptr += MINBPC(enc);
@@ -976,8 +985,12 @@ PREFIX(scanLit)(int open, const ENCODING *enc,
         return -XML_TOK_LITERAL;
       *nextTokPtr = ptr;
       switch (BYTE_TYPE(enc, ptr)) {
-      case BT_S: case BT_CR: case BT_LF:
-      case BT_GT: case BT_PERCNT: case BT_LSQB:
+      case BT_S:
+      case BT_CR:
+      case BT_LF:
+      case BT_GT:
+      case BT_PERCNT:
+      case BT_LSQB:
         return XML_TOK_LITERAL;
       default:
         return XML_TOK_INVALID;
@@ -992,8 +1005,7 @@ PREFIX(scanLit)(int open, const ENCODING *enc,
 
 static int PTRCALL
 PREFIX(prologTok)(const ENCODING *enc, const char *ptr, const char *end,
-                  const char **nextTokPtr)
-{
+                  const char **nextTokPtr) {
   int tok;
   if (ptr >= end)
     return XML_TOK_NONE;
@@ -1011,27 +1023,26 @@ PREFIX(prologTok)(const ENCODING *enc, const char *ptr, const char *end,
     return PREFIX(scanLit)(BT_QUOT, enc, ptr + MINBPC(enc), end, nextTokPtr);
   case BT_APOS:
     return PREFIX(scanLit)(BT_APOS, enc, ptr + MINBPC(enc), end, nextTokPtr);
-  case BT_LT:
-    {
-      ptr += MINBPC(enc);
-      REQUIRE_CHAR(enc, ptr, end);
-      switch (BYTE_TYPE(enc, ptr)) {
-      case BT_EXCL:
-        return PREFIX(scanDecl)(enc, ptr + MINBPC(enc), end, nextTokPtr);
-      case BT_QUEST:
-        return PREFIX(scanPi)(enc, ptr + MINBPC(enc), end, nextTokPtr);
-      case BT_NMSTRT:
-      case BT_HEX:
-      case BT_NONASCII:
-      case BT_LEAD2:
-      case BT_LEAD3:
-      case BT_LEAD4:
-        *nextTokPtr = ptr - MINBPC(enc);
-        return XML_TOK_INSTANCE_START;
-      }
-      *nextTokPtr = ptr;
-      return XML_TOK_INVALID;
+  case BT_LT: {
+    ptr += MINBPC(enc);
+    REQUIRE_CHAR(enc, ptr, end);
+    switch (BYTE_TYPE(enc, ptr)) {
+    case BT_EXCL:
+      return PREFIX(scanDecl)(enc, ptr + MINBPC(enc), end, nextTokPtr);
+    case BT_QUEST:
+      return PREFIX(scanPi)(enc, ptr + MINBPC(enc), end, nextTokPtr);
+    case BT_NMSTRT:
+    case BT_HEX:
+    case BT_NONASCII:
+    case BT_LEAD2:
+    case BT_LEAD3:
+    case BT_LEAD4:
+      *nextTokPtr = ptr - MINBPC(enc);
+      return XML_TOK_INSTANCE_START;
     }
+    *nextTokPtr = ptr;
+    return XML_TOK_INVALID;
+  }
   case BT_CR:
     if (ptr + MINBPC(enc) == end) {
       *nextTokPtr = end;
@@ -1039,13 +1050,15 @@ PREFIX(prologTok)(const ENCODING *enc, const char *ptr, const char *end,
       return -XML_TOK_PROLOG_S;
     }
     /* fall through */
-  case BT_S: case BT_LF:
+  case BT_S:
+  case BT_LF:
     for (;;) {
       ptr += MINBPC(enc);
       if (! HAS_CHAR(enc, ptr, end))
         break;
       switch (BYTE_TYPE(enc, ptr)) {
-      case BT_S: case BT_LF:
+      case BT_S:
+      case BT_LF:
         break;
       case BT_CR:
         /* don't split CR/LF pair */
@@ -1074,7 +1087,7 @@ PREFIX(prologTok)(const ENCODING *enc, const char *ptr, const char *end,
     if (CHAR_MATCHES(enc, ptr, ASCII_RSQB)) {
       REQUIRE_CHARS(enc, ptr, end, 2);
       if (CHAR_MATCHES(enc, ptr + MINBPC(enc), ASCII_GT)) {
-        *nextTokPtr = ptr + 2*MINBPC(enc);
+        *nextTokPtr = ptr + 2 * MINBPC(enc);
         return XML_TOK_COND_SECT_CLOSE;
       }
     }
@@ -1097,8 +1110,12 @@ PREFIX(prologTok)(const ENCODING *enc, const char *ptr, const char *end,
     case BT_PLUS:
       *nextTokPtr = ptr + MINBPC(enc);
       return XML_TOK_CLOSE_PAREN_PLUS;
-    case BT_CR: case BT_LF: case BT_S:
-    case BT_GT: case BT_COMMA: case BT_VERBAR:
+    case BT_CR:
+    case BT_LF:
+    case BT_S:
+    case BT_GT:
+    case BT_COMMA:
+    case BT_VERBAR:
     case BT_RPAR:
       *nextTokPtr = ptr;
       return XML_TOK_CLOSE_PAREN;
@@ -1113,24 +1130,26 @@ PREFIX(prologTok)(const ENCODING *enc, const char *ptr, const char *end,
     return XML_TOK_DECL_CLOSE;
   case BT_NUM:
     return PREFIX(scanPoundName)(enc, ptr + MINBPC(enc), end, nextTokPtr);
-#define LEAD_CASE(n) \
-  case BT_LEAD ## n: \
-    if (end - ptr < n) \
-      return XML_TOK_PARTIAL_CHAR; \
-    if (IS_NMSTRT_CHAR(enc, ptr, n)) { \
-      ptr += n; \
-      tok = XML_TOK_NAME; \
-      break; \
-    } \
-    if (IS_NAME_CHAR(enc, ptr, n)) { \
-      ptr += n; \
-      tok = XML_TOK_NMTOKEN; \
-      break; \
-    } \
-    *nextTokPtr = ptr; \
+#  define LEAD_CASE(n)                                                         \
+  case BT_LEAD##n:                                                             \
+    if (end - ptr < n)                                                         \
+      return XML_TOK_PARTIAL_CHAR;                                             \
+    if (IS_NMSTRT_CHAR(enc, ptr, n)) {                                         \
+      ptr += n;                                                                \
+      tok = XML_TOK_NAME;                                                      \
+      break;                                                                   \
+    }                                                                          \
+    if (IS_NAME_CHAR(enc, ptr, n)) {                                           \
+      ptr += n;                                                                \
+      tok = XML_TOK_NMTOKEN;                                                   \
+      break;                                                                   \
+    }                                                                          \
+    *nextTokPtr = ptr;                                                         \
     return XML_TOK_INVALID;
-    LEAD_CASE(2) LEAD_CASE(3) LEAD_CASE(4)
-#undef LEAD_CASE
+    LEAD_CASE(2)
+    LEAD_CASE(3)
+    LEAD_CASE(4)
+#  undef LEAD_CASE
   case BT_NMSTRT:
   case BT_HEX:
     tok = XML_TOK_NAME;
@@ -1139,9 +1158,9 @@ PREFIX(prologTok)(const ENCODING *enc, const char *ptr, const char *end,
   case BT_DIGIT:
   case BT_NAME:
   case BT_MINUS:
-#ifdef XML_NS
+#  ifdef XML_NS
   case BT_COLON:
-#endif
+#  endif
     tok = XML_TOK_NMTOKEN;
     ptr += MINBPC(enc);
     break;
@@ -1163,13 +1182,19 @@ PREFIX(prologTok)(const ENCODING *enc, const char *ptr, const char *end,
   }
   while (HAS_CHAR(enc, ptr, end)) {
     switch (BYTE_TYPE(enc, ptr)) {
-    CHECK_NAME_CASES(enc, ptr, end, nextTokPtr)
-    case BT_GT: case BT_RPAR: case BT_COMMA:
-    case BT_VERBAR: case BT_LSQB: case BT_PERCNT:
-    case BT_S: case BT_CR: case BT_LF:
+      CHECK_NAME_CASES(enc, ptr, end, nextTokPtr)
+    case BT_GT:
+    case BT_RPAR:
+    case BT_COMMA:
+    case BT_VERBAR:
+    case BT_LSQB:
+    case BT_PERCNT:
+    case BT_S:
+    case BT_CR:
+    case BT_LF:
       *nextTokPtr = ptr;
       return tok;
-#ifdef XML_NS
+#  ifdef XML_NS
     case BT_COLON:
       ptr += MINBPC(enc);
       switch (tok) {
@@ -1177,7 +1202,7 @@ PREFIX(prologTok)(const ENCODING *enc, const char *ptr, const char *end,
         REQUIRE_CHAR(enc, ptr, end);
         tok = XML_TOK_PREFIXED_NAME;
         switch (BYTE_TYPE(enc, ptr)) {
-        CHECK_NAME_CASES(enc, ptr, end, nextTokPtr)
+          CHECK_NAME_CASES(enc, ptr, end, nextTokPtr)
         default:
           tok = XML_TOK_NMTOKEN;
           break;
@@ -1188,23 +1213,23 @@ PREFIX(prologTok)(const ENCODING *enc, const char *ptr, const char *end,
         break;
       }
       break;
-#endif
+#  endif
     case BT_PLUS:
-      if (tok == XML_TOK_NMTOKEN)  {
+      if (tok == XML_TOK_NMTOKEN) {
         *nextTokPtr = ptr;
         return XML_TOK_INVALID;
       }
       *nextTokPtr = ptr + MINBPC(enc);
       return XML_TOK_NAME_PLUS;
     case BT_AST:
-      if (tok == XML_TOK_NMTOKEN)  {
+      if (tok == XML_TOK_NMTOKEN) {
         *nextTokPtr = ptr;
         return XML_TOK_INVALID;
       }
       *nextTokPtr = ptr + MINBPC(enc);
       return XML_TOK_NAME_ASTERISK;
     case BT_QUEST:
-      if (tok == XML_TOK_NMTOKEN)  {
+      if (tok == XML_TOK_NMTOKEN) {
         *nextTokPtr = ptr;
         return XML_TOK_INVALID;
       }
@@ -1219,9 +1244,8 @@ PREFIX(prologTok)(const ENCODING *enc, const char *ptr, const char *end,
 }
 
 static int PTRCALL
-PREFIX(attributeValueTok)(const ENCODING *enc, const char *ptr,
-                          const char *end, const char **nextTokPtr)
-{
+PREFIX(attributeValueTok)(const ENCODING *enc, const char *ptr, const char *end,
+                          const char **nextTokPtr) {
   const char *start;
   if (ptr >= end)
     return XML_TOK_NONE;
@@ -1236,10 +1260,14 @@ PREFIX(attributeValueTok)(const ENCODING *enc, const char *ptr,
   start = ptr;
   while (HAS_CHAR(enc, ptr, end)) {
     switch (BYTE_TYPE(enc, ptr)) {
-#define LEAD_CASE(n) \
-    case BT_LEAD ## n: ptr += n; break;
-    LEAD_CASE(2) LEAD_CASE(3) LEAD_CASE(4)
-#undef LEAD_CASE
+#  define LEAD_CASE(n)                                                         \
+  case BT_LEAD##n:                                                             \
+    ptr += n;                                                                  \
+    break;
+      LEAD_CASE(2)
+      LEAD_CASE(3)
+      LEAD_CASE(4)
+#  undef LEAD_CASE
     case BT_AMP:
       if (ptr == start)
         return PREFIX(scanRef)(enc, ptr + MINBPC(enc), end, nextTokPtr);
@@ -1285,9 +1313,8 @@ PREFIX(attributeValueTok)(const ENCODING *enc, const char *ptr,
 }
 
 static int PTRCALL
-PREFIX(entityValueTok)(const ENCODING *enc, const char *ptr,
-                       const char *end, const char **nextTokPtr)
-{
+PREFIX(entityValueTok)(const ENCODING *enc, const char *ptr, const char *end,
+                       const char **nextTokPtr) {
   const char *start;
   if (ptr >= end)
     return XML_TOK_NONE;
@@ -1302,10 +1329,14 @@ PREFIX(entityValueTok)(const ENCODING *enc, const char *ptr,
   start = ptr;
   while (HAS_CHAR(enc, ptr, end)) {
     switch (BYTE_TYPE(enc, ptr)) {
-#define LEAD_CASE(n) \
-    case BT_LEAD ## n: ptr += n; break;
-    LEAD_CASE(2) LEAD_CASE(3) LEAD_CASE(4)
-#undef LEAD_CASE
+#  define LEAD_CASE(n)                                                         \
+  case BT_LEAD##n:                                                             \
+    ptr += n;                                                                  \
+    break;
+      LEAD_CASE(2)
+      LEAD_CASE(3)
+      LEAD_CASE(4)
+#  undef LEAD_CASE
     case BT_AMP:
       if (ptr == start)
         return PREFIX(scanRef)(enc, ptr + MINBPC(enc), end, nextTokPtr);
@@ -1313,8 +1344,7 @@ PREFIX(entityValueTok)(const ENCODING *enc, const char *ptr,
       return XML_TOK_DATA_CHARS;
     case BT_PERCNT:
       if (ptr == start) {
-        int tok =  PREFIX(scanPercent)(enc, ptr + MINBPC(enc),
-                                       end, nextTokPtr);
+        int tok = PREFIX(scanPercent)(enc, ptr + MINBPC(enc), end, nextTokPtr);
         return (tok == XML_TOK_PERCENT) ? XML_TOK_INVALID : tok;
       }
       *nextTokPtr = ptr;
@@ -1347,12 +1377,11 @@ PREFIX(entityValueTok)(const ENCODING *enc, const char *ptr,
   return XML_TOK_DATA_CHARS;
 }
 
-#ifdef XML_DTD
+#  ifdef XML_DTD
 
 static int PTRCALL
-PREFIX(ignoreSectionTok)(const ENCODING *enc, const char *ptr,
-                         const char *end, const char **nextTokPtr)
-{
+PREFIX(ignoreSectionTok)(const ENCODING *enc, const char *ptr, const char *end,
+                         const char **nextTokPtr) {
   int level = 0;
   if (MINBPC(enc) > 1) {
     size_t n = end - ptr;
@@ -1363,7 +1392,7 @@ PREFIX(ignoreSectionTok)(const ENCODING *enc, const char *ptr,
   }
   while (HAS_CHAR(enc, ptr, end)) {
     switch (BYTE_TYPE(enc, ptr)) {
-    INVALID_CASES(ptr, nextTokPtr)
+      INVALID_CASES(ptr, nextTokPtr)
     case BT_LT:
       ptr += MINBPC(enc);
       REQUIRE_CHAR(enc, ptr, end);
@@ -1400,12 +1429,11 @@ PREFIX(ignoreSectionTok)(const ENCODING *enc, const char *ptr,
   return XML_TOK_PARTIAL;
 }
 
-#endif /* XML_DTD */
+#  endif /* XML_DTD */
 
 static int PTRCALL
 PREFIX(isPublicId)(const ENCODING *enc, const char *ptr, const char *end,
-                   const char **badPtr)
-{
+                   const char **badPtr) {
   ptr += MINBPC(enc);
   end -= MINBPC(enc);
   for (; HAS_CHAR(enc, ptr, end); ptr += MINBPC(enc)) {
@@ -1428,9 +1456,9 @@ PREFIX(isPublicId)(const ENCODING *enc, const char *ptr, const char *end,
     case BT_AST:
     case BT_PERCNT:
     case BT_NUM:
-#ifdef XML_NS
+#  ifdef XML_NS
     case BT_COLON:
-#endif
+#  endif
       break;
     case BT_S:
       if (CHAR_MATCHES(enc, ptr, ASCII_TAB)) {
@@ -1440,8 +1468,9 @@ PREFIX(isPublicId)(const ENCODING *enc, const char *ptr, const char *end,
       break;
     case BT_NAME:
     case BT_NMSTRT:
-      if (!(BYTE_TO_ASCII(enc, ptr) & ~0x7f))
+      if (! (BYTE_TO_ASCII(enc, ptr) & ~0x7f))
         break;
+      /* fall through */
     default:
       switch (BYTE_TO_ASCII(enc, ptr)) {
       case 0x24: /* $ */
@@ -1463,9 +1492,8 @@ PREFIX(isPublicId)(const ENCODING *enc, const char *ptr, const char *end,
 */
 
 static int PTRCALL
-PREFIX(getAtts)(const ENCODING *enc, const char *ptr,
-                int attsMax, ATTRIBUTE *atts)
-{
+PREFIX(getAtts)(const ENCODING *enc, const char *ptr, int attsMax,
+                ATTRIBUTE *atts) {
   enum { other, inName, inValue } state = inName;
   int nAtts = 0;
   int open = 0; /* defined when state == inValue;
@@ -1473,32 +1501,35 @@ PREFIX(getAtts)(const ENCODING *enc, const char *ptr,
 
   for (ptr += MINBPC(enc);; ptr += MINBPC(enc)) {
     switch (BYTE_TYPE(enc, ptr)) {
-#define START_NAME \
-      if (state == other) { \
-        if (nAtts < attsMax) { \
-          atts[nAtts].name = ptr; \
-          atts[nAtts].normalized = 1; \
-        } \
-        state = inName; \
-      }
-#define LEAD_CASE(n) \
-    case BT_LEAD ## n: START_NAME ptr += (n - MINBPC(enc)); break;
-    LEAD_CASE(2) LEAD_CASE(3) LEAD_CASE(4)
-#undef LEAD_CASE
+#  define START_NAME                                                           \
+    if (state == other) {                                                      \
+      if (nAtts < attsMax) {                                                   \
+        atts[nAtts].name = ptr;                                                \
+        atts[nAtts].normalized = 1;                                            \
+      }                                                                        \
+      state = inName;                                                          \
+    }
+#  define LEAD_CASE(n)                                                         \
+  case BT_LEAD##n:                                                             \
+    START_NAME ptr += (n - MINBPC(enc));                                       \
+    break;
+      LEAD_CASE(2)
+      LEAD_CASE(3)
+      LEAD_CASE(4)
+#  undef LEAD_CASE
     case BT_NONASCII:
     case BT_NMSTRT:
     case BT_HEX:
       START_NAME
       break;
-#undef START_NAME
+#  undef START_NAME
     case BT_QUOT:
       if (state != inValue) {
         if (nAtts < attsMax)
           atts[nAtts].valuePtr = ptr + MINBPC(enc);
         state = inValue;
         open = BT_QUOT;
-      }
-      else if (open == BT_QUOT) {
+      } else if (open == BT_QUOT) {
         state = other;
         if (nAtts < attsMax)
           atts[nAtts].valueEnd = ptr;
@@ -1511,8 +1542,7 @@ PREFIX(getAtts)(const ENCODING *enc, const char *ptr,
           atts[nAtts].valuePtr = ptr + MINBPC(enc);
         state = inValue;
         open = BT_APOS;
-      }
-      else if (open == BT_APOS) {
+      } else if (open == BT_APOS) {
         state = other;
         if (nAtts < attsMax)
           atts[nAtts].valueEnd = ptr;
@@ -1526,16 +1556,15 @@ PREFIX(getAtts)(const ENCODING *enc, const char *ptr,
     case BT_S:
       if (state == inName)
         state = other;
-      else if (state == inValue
-               && nAtts < attsMax
-               && atts[nAtts].normalized
+      else if (state == inValue && nAtts < attsMax && atts[nAtts].normalized
                && (ptr == atts[nAtts].valuePtr
                    || BYTE_TO_ASCII(enc, ptr) != ASCII_SPACE
                    || BYTE_TO_ASCII(enc, ptr + MINBPC(enc)) == ASCII_SPACE
                    || BYTE_TYPE(enc, ptr + MINBPC(enc)) == open))
         atts[nAtts].normalized = 0;
       break;
-    case BT_CR: case BT_LF:
+    case BT_CR:
+    case BT_LF:
       /* This case ensures that the first attribute name is counted
          Apart from that we could just change state on the quote. */
       if (state == inName)
@@ -1556,29 +1585,44 @@ PREFIX(getAtts)(const ENCODING *enc, const char *ptr,
 }
 
 static int PTRFASTCALL
-PREFIX(charRefNumber)(const ENCODING *UNUSED_P(enc), const char *ptr)
-{
+PREFIX(charRefNumber)(const ENCODING *enc, const char *ptr) {
   int result = 0;
   /* skip &# */
-  ptr += 2*MINBPC(enc);
+  UNUSED_P(enc);
+  ptr += 2 * MINBPC(enc);
   if (CHAR_MATCHES(enc, ptr, ASCII_x)) {
-    for (ptr += MINBPC(enc);
-         !CHAR_MATCHES(enc, ptr, ASCII_SEMI);
+    for (ptr += MINBPC(enc); ! CHAR_MATCHES(enc, ptr, ASCII_SEMI);
          ptr += MINBPC(enc)) {
       int c = BYTE_TO_ASCII(enc, ptr);
       switch (c) {
-      case ASCII_0: case ASCII_1: case ASCII_2: case ASCII_3: case ASCII_4:
-      case ASCII_5: case ASCII_6: case ASCII_7: case ASCII_8: case ASCII_9:
+      case ASCII_0:
+      case ASCII_1:
+      case ASCII_2:
+      case ASCII_3:
+      case ASCII_4:
+      case ASCII_5:
+      case ASCII_6:
+      case ASCII_7:
+      case ASCII_8:
+      case ASCII_9:
         result <<= 4;
         result |= (c - ASCII_0);
         break;
-      case ASCII_A: case ASCII_B: case ASCII_C:
-      case ASCII_D: case ASCII_E: case ASCII_F:
+      case ASCII_A:
+      case ASCII_B:
+      case ASCII_C:
+      case ASCII_D:
+      case ASCII_E:
+      case ASCII_F:
         result <<= 4;
         result += 10 + (c - ASCII_A);
         break;
-      case ASCII_a: case ASCII_b: case ASCII_c:
-      case ASCII_d: case ASCII_e: case ASCII_f:
+      case ASCII_a:
+      case ASCII_b:
+      case ASCII_c:
+      case ASCII_d:
+      case ASCII_e:
+      case ASCII_f:
         result <<= 4;
         result += 10 + (c - ASCII_a);
         break;
@@ -1586,9 +1630,8 @@ PREFIX(charRefNumber)(const ENCODING *UNUSED_P(enc), const char *ptr)
       if (result >= 0x110000)
         return -1;
     }
-  }
-  else {
-    for (; !CHAR_MATCHES(enc, ptr, ASCII_SEMI); ptr += MINBPC(enc)) {
+  } else {
+    for (; ! CHAR_MATCHES(enc, ptr, ASCII_SEMI); ptr += MINBPC(enc)) {
       int c = BYTE_TO_ASCII(enc, ptr);
       result *= 10;
       result += (c - ASCII_0);
@@ -1600,10 +1643,10 @@ PREFIX(charRefNumber)(const ENCODING *UNUSED_P(enc), const char *ptr)
 }
 
 static int PTRCALL
-PREFIX(predefinedEntityName)(const ENCODING *UNUSED_P(enc), const char *ptr,
-                             const char *end)
-{
-  switch ((end - ptr)/MINBPC(enc)) {
+PREFIX(predefinedEntityName)(const ENCODING *enc, const char *ptr,
+                             const char *end) {
+  UNUSED_P(enc);
+  switch ((end - ptr) / MINBPC(enc)) {
   case 2:
     if (CHAR_MATCHES(enc, ptr + MINBPC(enc), ASCII_t)) {
       switch (BYTE_TO_ASCII(enc, ptr)) {
@@ -1653,113 +1696,43 @@ PREFIX(predefinedEntityName)(const ENCODING *UNUSED_P(enc), const char *ptr,
   return 0;
 }
 
-/* This function does not appear to be called from anywhere within the
- * library code.  It is used via the macro XmlSameName(), which is
- * defined but never used.  Since it appears in the encoding function
- * table, removing it is not a thing to be undertaken lightly.  For
- * the moment, we simply exclude it from coverage tests.
- *
- * LCOV_EXCL_START
- */
-static int PTRCALL
-PREFIX(sameName)(const ENCODING *enc, const char *ptr1, const char *ptr2)
-{
-  for (;;) {
-    switch (BYTE_TYPE(enc, ptr1)) {
-#define LEAD_CASE(n) \
-    case BT_LEAD ## n: \
-      if (*ptr1++ != *ptr2++) \
-        return 0;
-    LEAD_CASE(4) LEAD_CASE(3) LEAD_CASE(2)
-#undef LEAD_CASE
-      /* fall through */
-      if (*ptr1++ != *ptr2++)
-        return 0;
-      break;
-    case BT_NONASCII:
-    case BT_NMSTRT:
-#ifdef XML_NS
-    case BT_COLON:
-#endif
-    case BT_HEX:
-    case BT_DIGIT:
-    case BT_NAME:
-    case BT_MINUS:
-      if (*ptr2++ != *ptr1++)
-        return 0;
-      if (MINBPC(enc) > 1) {
-        if (*ptr2++ != *ptr1++)
-          return 0;
-        if (MINBPC(enc) > 2) {
-          if (*ptr2++ != *ptr1++)
-            return 0;
-          if (MINBPC(enc) > 3) {
-            if (*ptr2++ != *ptr1++)
-              return 0;
-          }
-        }
-      }
-      break;
-    default:
-      if (MINBPC(enc) == 1 && *ptr1 == *ptr2)
-        return 1;
-      switch (BYTE_TYPE(enc, ptr2)) {
-      case BT_LEAD2:
-      case BT_LEAD3:
-      case BT_LEAD4:
-      case BT_NONASCII:
-      case BT_NMSTRT:
-#ifdef XML_NS
-      case BT_COLON:
-#endif
-      case BT_HEX:
-      case BT_DIGIT:
-      case BT_NAME:
-      case BT_MINUS:
-        return 0;
-      default:
-        return 1;
-      }
-    }
-  }
-  /* not reached */
-}
-/* LCOV_EXCL_STOP */
-
 static int PTRCALL
-PREFIX(nameMatchesAscii)(const ENCODING *UNUSED_P(enc), const char *ptr1,
-                         const char *end1, const char *ptr2)
-{
+PREFIX(nameMatchesAscii)(const ENCODING *enc, const char *ptr1,
+                         const char *end1, const char *ptr2) {
+  UNUSED_P(enc);
   for (; *ptr2; ptr1 += MINBPC(enc), ptr2++) {
     if (end1 - ptr1 < MINBPC(enc)) {
-      /* This line cannot be executed.  THe incoming data has already
-       * been tokenized once, so imcomplete characters like this have
+      /* This line cannot be executed.  The incoming data has already
+       * been tokenized once, so incomplete characters like this have
        * already been eliminated from the input.  Retaining the
        * paranoia check is still valuable, however.
        */
       return 0; /* LCOV_EXCL_LINE */
     }
-    if (!CHAR_MATCHES(enc, ptr1, *ptr2))
+    if (! CHAR_MATCHES(enc, ptr1, *ptr2))
       return 0;
   }
   return ptr1 == end1;
 }
 
 static int PTRFASTCALL
-PREFIX(nameLength)(const ENCODING *enc, const char *ptr)
-{
+PREFIX(nameLength)(const ENCODING *enc, const char *ptr) {
   const char *start = ptr;
   for (;;) {
     switch (BYTE_TYPE(enc, ptr)) {
-#define LEAD_CASE(n) \
-    case BT_LEAD ## n: ptr += n; break;
-    LEAD_CASE(2) LEAD_CASE(3) LEAD_CASE(4)
-#undef LEAD_CASE
+#  define LEAD_CASE(n)                                                         \
+  case BT_LEAD##n:                                                             \
+    ptr += n;                                                                  \
+    break;
+      LEAD_CASE(2)
+      LEAD_CASE(3)
+      LEAD_CASE(4)
+#  undef LEAD_CASE
     case BT_NONASCII:
     case BT_NMSTRT:
-#ifdef XML_NS
+#  ifdef XML_NS
     case BT_COLON:
-#endif
+#  endif
     case BT_HEX:
     case BT_DIGIT:
     case BT_NAME:
@@ -1772,9 +1745,8 @@ PREFIX(nameLength)(const ENCODING *enc, const char *ptr)
   }
 }
 
-static const char * PTRFASTCALL
-PREFIX(skipS)(const ENCODING *enc, const char *ptr)
-{
+static const char *PTRFASTCALL
+PREFIX(skipS)(const ENCODING *enc, const char *ptr) {
   for (;;) {
     switch (BYTE_TYPE(enc, ptr)) {
     case BT_LF:
@@ -1789,19 +1761,18 @@ PREFIX(skipS)(const ENCODING *enc, const char *ptr)
 }
 
 static void PTRCALL
-PREFIX(updatePosition)(const ENCODING *enc,
-                       const char *ptr,
-                       const char *end,
-                       POSITION *pos)
-{
+PREFIX(updatePosition)(const ENCODING *enc, const char *ptr, const char *end,
+                       POSITION *pos) {
   while (HAS_CHAR(enc, ptr, end)) {
     switch (BYTE_TYPE(enc, ptr)) {
-#define LEAD_CASE(n) \
-    case BT_LEAD ## n: \
-      ptr += n; \
-      break;
-    LEAD_CASE(2) LEAD_CASE(3) LEAD_CASE(4)
-#undef LEAD_CASE
+#  define LEAD_CASE(n)                                                         \
+  case BT_LEAD##n:                                                             \
+    ptr += n;                                                                  \
+    break;
+      LEAD_CASE(2)
+      LEAD_CASE(3)
+      LEAD_CASE(4)
+#  undef LEAD_CASE
     case BT_LF:
       pos->columnNumber = (XML_Size)-1;
       pos->lineNumber++;
@@ -1822,12 +1793,12 @@ PREFIX(updatePosition)(const ENCODING *enc,
   }
 }
 
-#undef DO_LEAD_CASE
-#undef MULTIBYTE_CASES
-#undef INVALID_CASES
-#undef CHECK_NAME_CASE
-#undef CHECK_NAME_CASES
-#undef CHECK_NMSTRT_CASE
-#undef CHECK_NMSTRT_CASES
+#  undef DO_LEAD_CASE
+#  undef MULTIBYTE_CASES
+#  undef INVALID_CASES
+#  undef CHECK_NAME_CASE
+#  undef CHECK_NAME_CASES
+#  undef CHECK_NMSTRT_CASE
+#  undef CHECK_NMSTRT_CASES
 
 #endif /* XML_TOK_IMPL_C */
diff --git a/Modules/expat/xmltok_impl.h b/Modules/expat/xmltok_impl.h
index a6420f48eedc0..e925dbc7e2c83 100644
--- a/Modules/expat/xmltok_impl.h
+++ b/Modules/expat/xmltok_impl.h
@@ -31,43 +31,43 @@
 */
 
 enum {
-  BT_NONXML,
-  BT_MALFORM,
-  BT_LT,
-  BT_AMP,
-  BT_RSQB,
-  BT_LEAD2,
-  BT_LEAD3,
-  BT_LEAD4,
-  BT_TRAIL,
-  BT_CR,
-  BT_LF,
-  BT_GT,
-  BT_QUOT,
-  BT_APOS,
-  BT_EQUALS,
-  BT_QUEST,
-  BT_EXCL,
-  BT_SOL,
-  BT_SEMI,
-  BT_NUM,
-  BT_LSQB,
-  BT_S,
-  BT_NMSTRT,
-  BT_COLON,
-  BT_HEX,
-  BT_DIGIT,
-  BT_NAME,
-  BT_MINUS,
-  BT_OTHER, /* known not to be a name or name start character */
+  BT_NONXML,   /* e.g. noncharacter-FFFF */
+  BT_MALFORM,  /* illegal, with regard to encoding */
+  BT_LT,       /* less than = "<" */
+  BT_AMP,      /* ampersand = "&" */
+  BT_RSQB,     /* right square bracket = "[" */
+  BT_LEAD2,    /* lead byte of a 2-byte UTF-8 character */
+  BT_LEAD3,    /* lead byte of a 3-byte UTF-8 character */
+  BT_LEAD4,    /* lead byte of a 4-byte UTF-8 character */
+  BT_TRAIL,    /* trailing unit, e.g. second 16-bit unit of a 4-byte char. */
+  BT_CR,       /* carriage return = "\r" */
+  BT_LF,       /* line feed = "\n" */
+  BT_GT,       /* greater than = ">" */
+  BT_QUOT,     /* quotation character = "\"" */
+  BT_APOS,     /* aposthrophe = "'" */
+  BT_EQUALS,   /* equal sign = "=" */
+  BT_QUEST,    /* question mark = "?" */
+  BT_EXCL,     /* exclamation mark = "!" */
+  BT_SOL,      /* solidus, slash = "/" */
+  BT_SEMI,     /* semicolon = ";" */
+  BT_NUM,      /* number sign = "#" */
+  BT_LSQB,     /* left square bracket = "[" */
+  BT_S,        /* white space, e.g. "\t", " "[, "\r"] */
+  BT_NMSTRT,   /* non-hex name start letter = "G".."Z" + "g".."z" + "_" */
+  BT_COLON,    /* colon = ":" */
+  BT_HEX,      /* hex letter = "A".."F" + "a".."f" */
+  BT_DIGIT,    /* digit = "0".."9" */
+  BT_NAME,     /* dot and middle dot = "." + chr(0xb7) */
+  BT_MINUS,    /* minus = "-" */
+  BT_OTHER,    /* known not to be a name or name start character */
   BT_NONASCII, /* might be a name or name start character */
-  BT_PERCNT,
-  BT_LPAR,
-  BT_RPAR,
-  BT_AST,
-  BT_PLUS,
-  BT_COMMA,
-  BT_VERBAR
+  BT_PERCNT,   /* percent sign = "%" */
+  BT_LPAR,     /* left parenthesis = "(" */
+  BT_RPAR,     /* right parenthesis = "(" */
+  BT_AST,      /* asterisk = "*" */
+  BT_PLUS,     /* plus sign = "+" */
+  BT_COMMA,    /* comma = "," */
+  BT_VERBAR    /* vertical bar = "|" */
 };
 
 #include <stddef.h>
diff --git a/Modules/expat/xmltok_ns.c b/Modules/expat/xmltok_ns.c
index 23d31e8e42491..919c74e9f97fe 100644
--- a/Modules/expat/xmltok_ns.c
+++ b/Modules/expat/xmltok_ns.c
@@ -33,56 +33,47 @@
 #ifdef XML_TOK_NS_C
 
 const ENCODING *
-NS(XmlGetUtf8InternalEncoding)(void)
-{
+NS(XmlGetUtf8InternalEncoding)(void) {
   return &ns(internal_utf8_encoding).enc;
 }
 
 const ENCODING *
-NS(XmlGetUtf16InternalEncoding)(void)
-{
-#if BYTEORDER == 1234
+NS(XmlGetUtf16InternalEncoding)(void) {
+#  if BYTEORDER == 1234
   return &ns(internal_little2_encoding).enc;
-#elif BYTEORDER == 4321
+#  elif BYTEORDER == 4321
   return &ns(internal_big2_encoding).enc;
-#else
+#  else
   const short n = 1;
-  return (*(const char *)&n
-          ? &ns(internal_little2_encoding).enc
-          : &ns(internal_big2_encoding).enc);
-#endif
+  return (*(const char *)&n ? &ns(internal_little2_encoding).enc
+                            : &ns(internal_big2_encoding).enc);
+#  endif
 }
 
-static const ENCODING * const NS(encodings)[] = {
-  &ns(latin1_encoding).enc,
-  &ns(ascii_encoding).enc,
-  &ns(utf8_encoding).enc,
-  &ns(big2_encoding).enc,
-  &ns(big2_encoding).enc,
-  &ns(little2_encoding).enc,
-  &ns(utf8_encoding).enc /* NO_ENC */
+static const ENCODING *const NS(encodings)[] = {
+    &ns(latin1_encoding).enc, &ns(ascii_encoding).enc,
+    &ns(utf8_encoding).enc,   &ns(big2_encoding).enc,
+    &ns(big2_encoding).enc,   &ns(little2_encoding).enc,
+    &ns(utf8_encoding).enc /* NO_ENC */
 };
 
 static int PTRCALL
 NS(initScanProlog)(const ENCODING *enc, const char *ptr, const char *end,
-                   const char **nextTokPtr)
-{
-  return initScan(NS(encodings), (const INIT_ENCODING *)enc,
-                  XML_PROLOG_STATE, ptr, end, nextTokPtr);
+                   const char **nextTokPtr) {
+  return initScan(NS(encodings), (const INIT_ENCODING *)enc, XML_PROLOG_STATE,
+                  ptr, end, nextTokPtr);
 }
 
 static int PTRCALL
 NS(initScanContent)(const ENCODING *enc, const char *ptr, const char *end,
-                    const char **nextTokPtr)
-{
-  return initScan(NS(encodings), (const INIT_ENCODING *)enc,
-                  XML_CONTENT_STATE, ptr, end, nextTokPtr);
+                    const char **nextTokPtr) {
+  return initScan(NS(encodings), (const INIT_ENCODING *)enc, XML_CONTENT_STATE,
+                  ptr, end, nextTokPtr);
 }
 
 int
 NS(XmlInitEncoding)(INIT_ENCODING *p, const ENCODING **encPtr,
-                    const char *name)
-{
+                    const char *name) {
   int i = getEncodingIndex(name);
   if (i == UNKNOWN_ENC)
     return 0;
@@ -96,9 +87,8 @@ NS(XmlInitEncoding)(INIT_ENCODING *p, const ENCODING **encPtr,
 }
 
 static const ENCODING *
-NS(findEncoding)(const ENCODING *enc, const char *ptr, const char *end)
-{
-#define ENCODING_MAX 128
+NS(findEncoding)(const ENCODING *enc, const char *ptr, const char *end) {
+#  define ENCODING_MAX 128
   char buf[ENCODING_MAX];
   char *p = buf;
   int i;
@@ -115,28 +105,14 @@ NS(findEncoding)(const ENCODING *enc, const char *ptr, const char *end)
 }
 
 int
-NS(XmlParseXmlDecl)(int isGeneralTextEntity,
-                    const ENCODING *enc,
-                    const char *ptr,
-                    const char *end,
-                    const char **badPtr,
-                    const char **versionPtr,
-                    const char **versionEndPtr,
-                    const char **encodingName,
-                    const ENCODING **encoding,
-                    int *standalone)
-{
-  return doParseXmlDecl(NS(findEncoding),
-                        isGeneralTextEntity,
-                        enc,
-                        ptr,
-                        end,
-                        badPtr,
-                        versionPtr,
-                        versionEndPtr,
-                        encodingName,
-                        encoding,
-                        standalone);
+NS(XmlParseXmlDecl)(int isGeneralTextEntity, const ENCODING *enc,
+                    const char *ptr, const char *end, const char **badPtr,
+                    const char **versionPtr, const char **versionEndPtr,
+                    const char **encodingName, const ENCODING **encoding,
+                    int *standalone) {
+  return doParseXmlDecl(NS(findEncoding), isGeneralTextEntity, enc, ptr, end,
+                        badPtr, versionPtr, versionEndPtr, encodingName,
+                        encoding, standalone);
 }
 
 #endif /* XML_TOK_NS_C */
diff --git a/PCbuild/_elementtree.vcxproj b/PCbuild/_elementtree.vcxproj
index 12f3bd2ac3820..639ba479db578 100644
--- a/PCbuild/_elementtree.vcxproj
+++ b/PCbuild/_elementtree.vcxproj
@@ -86,7 +86,6 @@
     <ClInclude Include="..\Modules\expat\xmltok.h" />
   </ItemGroup>
   <ItemGroup>
-    <ClCompile Include="..\Modules\expat\loadlibrary.c" />
     <ClCompile Include="..\Modules\_elementtree.c" />
     <ClCompile Include="..\Modules\expat\xmlparse.c" />
     <ClCompile Include="..\Modules\expat\xmlrole.c" />
@@ -104,4 +103,4 @@
   <Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
   <ImportGroup Label="ExtensionTargets">
   </ImportGroup>
-</Project>
\ No newline at end of file
+</Project>
diff --git a/PCbuild/pyexpat.vcxproj b/PCbuild/pyexpat.vcxproj
index 0bed878dfc1af..9a765ff07bde8 100644
--- a/PCbuild/pyexpat.vcxproj
+++ b/PCbuild/pyexpat.vcxproj
@@ -67,7 +67,6 @@
     <ClInclude Include="..\Modules\expat\xmltok.h" />
   </ItemGroup>
   <ItemGroup>
-    <ClCompile Include="..\Modules\expat\loadlibrary.c" />
     <ClCompile Include="..\Modules\pyexpat.c" />
     <ClCompile Include="..\Modules\expat\xmlparse.c" />
     <ClCompile Include="..\Modules\expat\xmlrole.c" />
@@ -85,4 +84,4 @@
   <Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
   <ImportGroup Label="ExtensionTargets">
   </ImportGroup>
-</Project>
\ No newline at end of file
+</Project>


From webhook-mailer at python.org  Wed Oct  9 01:48:02 2019
From: webhook-mailer at python.org (Vinay Sajip)
Date: Wed, 09 Oct 2019 05:48:02 -0000
Subject: [Python-checkins] bpo-38368: Added fix for ctypes crash when
 handling arrays in structs/unions. (GH-16589) (GH-16671)
Message-ID: <mailman.67.1570600083.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/d004a5b082d0770682e6efbf03105a67469d4820
commit: d004a5b082d0770682e6efbf03105a67469d4820
branch: 3.8
author: Vinay Sajip <vinay_sajip at yahoo.co.uk>
committer: GitHub <noreply at github.com>
date: 2019-10-09T06:47:57+01:00
summary:

bpo-38368: Added fix for ctypes crash when handling arrays in structs/unions. (GH-16589) (GH-16671)

(cherry picked from commit e8bedbddadaa86be6bd86dc32dbdbd53933a4988)

files:
M Lib/ctypes/test/test_structures.py
M Modules/_ctypes/_ctypes_test.c
M Modules/_ctypes/stgdict.c

diff --git a/Lib/ctypes/test/test_structures.py b/Lib/ctypes/test/test_structures.py
index 11c194b0b73e2..1efd36dd62f49 100644
--- a/Lib/ctypes/test/test_structures.py
+++ b/Lib/ctypes/test/test_structures.py
@@ -1,3 +1,5 @@
+import platform
+import sys
 import unittest
 from ctypes import *
 from ctypes.test import need_symbol
@@ -491,6 +493,16 @@ class Test3(Structure):
                 ('data', c_double * 2),
             ]
 
+        class Test3A(Structure):
+            _fields_ = [
+                ('data', c_float * 2),
+            ]
+
+        class Test3B(Test3A):
+            _fields_ = [
+                ('more_data', c_float * 2),
+            ]
+
         s = Test2()
         expected = 0
         for i in range(16):
@@ -519,6 +531,46 @@ class Test3(Structure):
         self.assertEqual(s.data[0], 3.14159)
         self.assertEqual(s.data[1], 2.71828)
 
+        s = Test3B()
+        s.data[0] = 3.14159
+        s.data[1] = 2.71828
+        s.more_data[0] = -3.0
+        s.more_data[1] = -2.0
+
+        expected = 3.14159 + 2.71828 - 5.0
+        func = dll._testfunc_array_in_struct2a
+        func.restype = c_double
+        func.argtypes = (Test3B,)
+        result = func(s)
+        self.assertAlmostEqual(result, expected, places=6)
+        # check the passed-in struct hasn't changed
+        self.assertAlmostEqual(s.data[0], 3.14159, places=6)
+        self.assertAlmostEqual(s.data[1], 2.71828, places=6)
+        self.assertAlmostEqual(s.more_data[0], -3.0, places=6)
+        self.assertAlmostEqual(s.more_data[1], -2.0, places=6)
+
+    def test_38368(self):
+        class U(Union):
+            _fields_ = [
+                ('f1', c_uint8 * 16),
+                ('f2', c_uint16 * 8),
+                ('f3', c_uint32 * 4),
+            ]
+        u = U()
+        u.f3[0] = 0x01234567
+        u.f3[1] = 0x89ABCDEF
+        u.f3[2] = 0x76543210
+        u.f3[3] = 0xFEDCBA98
+        f1 = [u.f1[i] for i in range(16)]
+        f2 = [u.f2[i] for i in range(8)]
+        if sys.byteorder == 'little':
+            self.assertEqual(f1, [0x67, 0x45, 0x23, 0x01,
+                                  0xef, 0xcd, 0xab, 0x89,
+                                  0x10, 0x32, 0x54, 0x76,
+                                  0x98, 0xba, 0xdc, 0xfe])
+            self.assertEqual(f2, [0x4567, 0x0123, 0xcdef, 0x89ab,
+                                  0x3210, 0x7654, 0xba98, 0xfedc])
+
 class PointerMemberTestCase(unittest.TestCase):
 
     def test(self):
diff --git a/Modules/_ctypes/_ctypes_test.c b/Modules/_ctypes/_ctypes_test.c
index a7a8976d2d88a..19066b31c1c87 100644
--- a/Modules/_ctypes/_ctypes_test.c
+++ b/Modules/_ctypes/_ctypes_test.c
@@ -100,6 +100,11 @@ typedef struct {
     double data[2];
 } Test3;
 
+typedef struct {
+    float data[2];
+    float more_data[2];
+} Test3B;
+
 EXPORT(double)
 _testfunc_array_in_struct2(Test3 in)
 {
@@ -114,6 +119,22 @@ _testfunc_array_in_struct2(Test3 in)
     return result;
 }
 
+EXPORT(double)
+_testfunc_array_in_struct2a(Test3B in)
+{
+    double result = 0;
+
+    for (unsigned i = 0; i < 2; i++)
+        result += in.data[i];
+    for (unsigned i = 0; i < 2; i++)
+        result += in.more_data[i];
+    /* As the structure is passed by value, changes to it shouldn't be
+     * reflected in the caller.
+     */
+    memset(in.data, 0, sizeof(in.data));
+    return result;
+}
+
 EXPORT(void)testfunc_array(int values[4])
 {
     printf("testfunc_array %d %d %d %d\n",
diff --git a/Modules/_ctypes/stgdict.c b/Modules/_ctypes/stgdict.c
index fa664bd57a029..97bcf5539aeea 100644
--- a/Modules/_ctypes/stgdict.c
+++ b/Modules/_ctypes/stgdict.c
@@ -644,9 +644,9 @@ PyCStructUnionType_update_stgdict(PyObject *type, PyObject *fields, int isStruct
     stgdict->align = total_align;
     stgdict->length = len;      /* ADD ffi_ofs? */
 
-#define MAX_ELEMENTS 16
+#define MAX_STRUCT_SIZE 16
 
-    if (arrays_seen && (size <= 16)) {
+    if (arrays_seen && (size <= MAX_STRUCT_SIZE)) {
         /*
          * See bpo-22273. Arrays are normally treated as pointers, which is
          * fine when an array name is being passed as parameter, but not when
@@ -667,11 +667,138 @@ PyCStructUnionType_update_stgdict(PyObject *type, PyObject *fields, int isStruct
          * Although the passing in registers is specific to 64-bit Linux, the
          * array-in-struct vs. pointer problem is general. But we restrict the
          * type transformation to small structs nonetheless.
+         *
+         * Note that although a union may be small in terms of memory usage, it
+         * could contain many overlapping declarations of arrays, e.g.
+         *
+         * union {
+         *     unsigned int_8 foo [16];
+         *     unsigned uint_8 bar [16];
+         *     unsigned int_16 baz[8];
+         *     unsigned uint_16 bozz[8];
+         *     unsigned int_32 fizz[4];
+         *     unsigned uint_32 buzz[4];
+         * }
+         *
+         * which is still only 16 bytes in size. We need to convert this into
+         * the following equivalent for libffi:
+         *
+         * union {
+         *     struct { int_8 e1; int_8 e2; ... int_8 e_16; } f1;
+         *     struct { uint_8 e1; uint_8 e2; ... uint_8 e_16; } f2;
+         *     struct { int_16 e1; int_16 e2; ... int_16 e_8; } f3;
+         *     struct { uint_16 e1; uint_16 e2; ... uint_16 e_8; } f4;
+         *     struct { int_32 e1; int_32 e2; ... int_32 e_4; } f5;
+         *     struct { uint_32 e1; uint_32 e2; ... uint_32 e_4; } f6;
+         * }
+         *
+         * So the struct/union needs setting up as follows: all non-array
+         * elements copied across as is, and all array elements replaced with
+         * an equivalent struct which has as many fields as the array has
+         * elements, plus one NULL pointer.
+         */
+
+        Py_ssize_t num_ffi_type_pointers = 0;  /* for the dummy fields */
+        Py_ssize_t num_ffi_types = 0;  /* for the dummy structures */
+        size_t alloc_size;  /* total bytes to allocate */
+        void *type_block;  /* to hold all the type information needed */
+        ffi_type **element_types;  /* of this struct/union */
+        ffi_type **dummy_types;  /* of the dummy struct elements */
+        ffi_type *structs;  /* point to struct aliases of arrays */
+        Py_ssize_t element_index;  /* index into element_types for this */
+        Py_ssize_t dummy_index = 0; /* index into dummy field pointers */
+        Py_ssize_t struct_index = 0; /* index into dummy structs */
+
+        /* first pass to see how much memory to allocate */
+        for (i = 0; i < len; ++i) {
+            PyObject *name, *desc;
+            PyObject *pair = PySequence_GetItem(fields, i);
+            StgDictObject *dict;
+            int bitsize = 0;
+
+            if (pair == NULL) {
+                return -1;
+            }
+            if (!PyArg_ParseTuple(pair, "UO|i", &name, &desc, &bitsize)) {
+                PyErr_SetString(PyExc_TypeError,
+                    "'_fields_' must be a sequence of (name, C type) pairs");
+                Py_DECREF(pair);
+                return -1;
+            }
+            dict = PyType_stgdict(desc);
+            if (dict == NULL) {
+                Py_DECREF(pair);
+                PyErr_Format(PyExc_TypeError,
+                    "second item in _fields_ tuple (index %zd) must be a C type",
+                    i);
+                return -1;
+            }
+            if (!PyCArrayTypeObject_Check(desc)) {
+                /* Not an array. Just need an ffi_type pointer. */
+                num_ffi_type_pointers++;
+            }
+            else {
+                /* It's an array. */
+                Py_ssize_t length = dict->length;
+                StgDictObject *edict;
+
+                edict = PyType_stgdict(dict->proto);
+                if (edict == NULL) {
+                    Py_DECREF(pair);
+                    PyErr_Format(PyExc_TypeError,
+                        "second item in _fields_ tuple (index %zd) must be a C type",
+                        i);
+                    return -1;
+                }
+                /*
+                 * We need one extra ffi_type to hold the struct, and one
+                 * ffi_type pointer per array element + one for a NULL to
+                 * mark the end.
+                 */
+                num_ffi_types++;
+                num_ffi_type_pointers += length + 1;
+            }
+            Py_DECREF(pair);
+        }
+
+        /*
+         * At this point, we know we need storage for some ffi_types and some
+         * ffi_type pointers. We'll allocate these in one block.
+         * There are three sub-blocks of information: the ffi_type pointers to
+         * this structure/union's elements, the ffi_type_pointers to the
+         * dummy fields standing in for array elements, and the
+         * ffi_types representing the dummy structures.
          */
-        ffi_type *actual_types[MAX_ELEMENTS + 1];
-        int actual_type_index = 0;
+        alloc_size = (ffi_ofs + 1 + len + num_ffi_type_pointers) * sizeof(ffi_type *) +
+                        num_ffi_types * sizeof(ffi_type);
+        type_block = PyMem_Malloc(alloc_size);
 
-        memset(actual_types, 0, sizeof(actual_types));
+        if (type_block == NULL) {
+            PyErr_NoMemory();
+            return -1;
+        }
+        /*
+         * the first block takes up ffi_ofs + len + 1 which is the pointers *
+         * for this struct/union. The second block takes up
+         * num_ffi_type_pointers, so the sum of these is ffi_ofs + len + 1 +
+         * num_ffi_type_pointers as allocated above. The last bit is the
+         * num_ffi_types structs.
+         */
+        element_types = (ffi_type **) type_block;
+        dummy_types = &element_types[ffi_ofs + len + 1];
+        structs = (ffi_type *) &dummy_types[num_ffi_type_pointers];
+
+        if (num_ffi_types > 0) {
+            memset(structs, 0, num_ffi_types * sizeof(ffi_type));
+        }
+        if (ffi_ofs && (basedict != NULL)) {
+            memcpy(element_types,
+                basedict->ffi_type_pointer.elements,
+                ffi_ofs * sizeof(ffi_type *));
+        }
+        element_index = ffi_ofs;
+
+        /* second pass to actually set the type pointers */
         for (i = 0; i < len; ++i) {
             PyObject *name, *desc;
             PyObject *pair = PySequence_GetItem(fields, i);
@@ -679,26 +806,36 @@ PyCStructUnionType_update_stgdict(PyObject *type, PyObject *fields, int isStruct
             int bitsize = 0;
 
             if (pair == NULL) {
+                PyMem_Free(type_block);
                 return -1;
             }
+            /* In theory, we made this call in the first pass, so it *shouldn't*
+             * fail. However, you never know, and the code above might change
+             * later - keeping the check in here is a tad defensive but it
+             * will affect program size only slightly and performance hardly at
+             * all.
+             */
             if (!PyArg_ParseTuple(pair, "UO|i", &name, &desc, &bitsize)) {
                 PyErr_SetString(PyExc_TypeError,
                                 "'_fields_' must be a sequence of (name, C type) pairs");
-                Py_XDECREF(pair);
+                Py_DECREF(pair);
+                PyMem_Free(type_block);
                 return -1;
             }
             dict = PyType_stgdict(desc);
+            /* Possibly this check could be avoided, but see above comment. */
             if (dict == NULL) {
                 Py_DECREF(pair);
+                PyMem_Free(type_block);
                 PyErr_Format(PyExc_TypeError,
                              "second item in _fields_ tuple (index %zd) must be a C type",
                              i);
                 return -1;
             }
+            assert(element_index < (ffi_ofs + len)); /* will be used below */
             if (!PyCArrayTypeObject_Check(desc)) {
                 /* Not an array. Just copy over the element ffi_type. */
-                actual_types[actual_type_index++] = &dict->ffi_type_pointer;
-                assert(actual_type_index <= MAX_ELEMENTS);
+                element_types[element_index++] = &dict->ffi_type_pointer;
             }
             else {
                 Py_ssize_t length = dict->length;
@@ -707,42 +844,38 @@ PyCStructUnionType_update_stgdict(PyObject *type, PyObject *fields, int isStruct
                 edict = PyType_stgdict(dict->proto);
                 if (edict == NULL) {
                     Py_DECREF(pair);
+                    PyMem_Free(type_block);
                     PyErr_Format(PyExc_TypeError,
                                  "second item in _fields_ tuple (index %zd) must be a C type",
                                  i);
                     return -1;
                 }
+                element_types[element_index++] = &structs[struct_index];
+                structs[struct_index].size = length * edict->ffi_type_pointer.size;
+                structs[struct_index].alignment = edict->ffi_type_pointer.alignment;
+                structs[struct_index].type = FFI_TYPE_STRUCT;
+                structs[struct_index].elements = &dummy_types[dummy_index];
+                ++struct_index;
                 /* Copy over the element's type, length times. */
                 while (length > 0) {
-                    actual_types[actual_type_index++] = &edict->ffi_type_pointer;
-                    assert(actual_type_index <= MAX_ELEMENTS);
+                    assert(dummy_index < (num_ffi_type_pointers));
+                    dummy_types[dummy_index++] = &edict->ffi_type_pointer;
                     length--;
                 }
+                assert(dummy_index < (num_ffi_type_pointers));
+                dummy_types[dummy_index++] = NULL;
             }
             Py_DECREF(pair);
         }
 
-        actual_types[actual_type_index++] = NULL;
+        element_types[element_index] = NULL;
         /*
          * Replace the old elements with the new, taking into account
          * base class elements where necessary.
          */
         assert(stgdict->ffi_type_pointer.elements);
         PyMem_Free(stgdict->ffi_type_pointer.elements);
-        stgdict->ffi_type_pointer.elements = PyMem_New(ffi_type *,
-                                                       ffi_ofs + actual_type_index);
-        if (stgdict->ffi_type_pointer.elements == NULL) {
-            PyErr_NoMemory();
-            return -1;
-        }
-        if (ffi_ofs) {
-            memcpy(stgdict->ffi_type_pointer.elements,
-                   basedict->ffi_type_pointer.elements,
-                   ffi_ofs * sizeof(ffi_type *));
-
-        }
-        memcpy(&stgdict->ffi_type_pointer.elements[ffi_ofs], actual_types,
-               actual_type_index * sizeof(ffi_type *));
+        stgdict->ffi_type_pointer.elements = element_types;
     }
 
     /* We did check that this flag was NOT set above, it must not


From webhook-mailer at python.org  Wed Oct  9 01:48:51 2019
From: webhook-mailer at python.org (Vinay Sajip)
Date: Wed, 09 Oct 2019 05:48:51 -0000
Subject: [Python-checkins] bpo-38368: Added fix for ctypes crash when
 handling arrays in structs/unions. (GH-16589) (GH-16672)
Message-ID: <mailman.68.1570600132.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/129c2b3d44a962fc5964ef8fd4e6b22dbb691a6c
commit: 129c2b3d44a962fc5964ef8fd4e6b22dbb691a6c
branch: 3.7
author: Vinay Sajip <vinay_sajip at yahoo.co.uk>
committer: GitHub <noreply at github.com>
date: 2019-10-09T06:48:48+01:00
summary:

bpo-38368: Added fix for ctypes crash when handling arrays in structs/unions. (GH-16589) (GH-16672)

(cherry picked from commit e8bedbddadaa86be6bd86dc32dbdbd53933a4988)

files:
M Lib/ctypes/test/test_structures.py
M Modules/_ctypes/_ctypes_test.c
M Modules/_ctypes/stgdict.c

diff --git a/Lib/ctypes/test/test_structures.py b/Lib/ctypes/test/test_structures.py
index 20d0fa72ce432..d8d6c657928bf 100644
--- a/Lib/ctypes/test/test_structures.py
+++ b/Lib/ctypes/test/test_structures.py
@@ -1,3 +1,5 @@
+import platform
+import sys
 import unittest
 from ctypes import *
 from ctypes.test import need_symbol
@@ -452,6 +454,16 @@ class Test3(Structure):
                 ('data', c_double * 2),
             ]
 
+        class Test3A(Structure):
+            _fields_ = [
+                ('data', c_float * 2),
+            ]
+
+        class Test3B(Test3A):
+            _fields_ = [
+                ('more_data', c_float * 2),
+            ]
+
         s = Test2()
         expected = 0
         for i in range(16):
@@ -480,6 +492,46 @@ class Test3(Structure):
         self.assertEqual(s.data[0], 3.14159)
         self.assertEqual(s.data[1], 2.71828)
 
+        s = Test3B()
+        s.data[0] = 3.14159
+        s.data[1] = 2.71828
+        s.more_data[0] = -3.0
+        s.more_data[1] = -2.0
+
+        expected = 3.14159 + 2.71828 - 5.0
+        func = dll._testfunc_array_in_struct2a
+        func.restype = c_double
+        func.argtypes = (Test3B,)
+        result = func(s)
+        self.assertAlmostEqual(result, expected, places=6)
+        # check the passed-in struct hasn't changed
+        self.assertAlmostEqual(s.data[0], 3.14159, places=6)
+        self.assertAlmostEqual(s.data[1], 2.71828, places=6)
+        self.assertAlmostEqual(s.more_data[0], -3.0, places=6)
+        self.assertAlmostEqual(s.more_data[1], -2.0, places=6)
+
+    def test_38368(self):
+        class U(Union):
+            _fields_ = [
+                ('f1', c_uint8 * 16),
+                ('f2', c_uint16 * 8),
+                ('f3', c_uint32 * 4),
+            ]
+        u = U()
+        u.f3[0] = 0x01234567
+        u.f3[1] = 0x89ABCDEF
+        u.f3[2] = 0x76543210
+        u.f3[3] = 0xFEDCBA98
+        f1 = [u.f1[i] for i in range(16)]
+        f2 = [u.f2[i] for i in range(8)]
+        if sys.byteorder == 'little':
+            self.assertEqual(f1, [0x67, 0x45, 0x23, 0x01,
+                                  0xef, 0xcd, 0xab, 0x89,
+                                  0x10, 0x32, 0x54, 0x76,
+                                  0x98, 0xba, 0xdc, 0xfe])
+            self.assertEqual(f2, [0x4567, 0x0123, 0xcdef, 0x89ab,
+                                  0x3210, 0x7654, 0xba98, 0xfedc])
+
 class PointerMemberTestCase(unittest.TestCase):
 
     def test(self):
diff --git a/Modules/_ctypes/_ctypes_test.c b/Modules/_ctypes/_ctypes_test.c
index 0b3348b7fe006..f59abcc451591 100644
--- a/Modules/_ctypes/_ctypes_test.c
+++ b/Modules/_ctypes/_ctypes_test.c
@@ -100,6 +100,11 @@ typedef struct {
     double data[2];
 } Test3;
 
+typedef struct {
+    float data[2];
+    float more_data[2];
+} Test3B;
+
 EXPORT(double)
 _testfunc_array_in_struct2(Test3 in)
 {
@@ -114,6 +119,22 @@ _testfunc_array_in_struct2(Test3 in)
     return result;
 }
 
+EXPORT(double)
+_testfunc_array_in_struct2a(Test3B in)
+{
+    double result = 0;
+
+    for (unsigned i = 0; i < 2; i++)
+        result += in.data[i];
+    for (unsigned i = 0; i < 2; i++)
+        result += in.more_data[i];
+    /* As the structure is passed by value, changes to it shouldn't be
+     * reflected in the caller.
+     */
+    memset(in.data, 0, sizeof(in.data));
+    return result;
+}
+
 EXPORT(void)testfunc_array(int values[4])
 {
     printf("testfunc_array %d %d %d %d\n",
diff --git a/Modules/_ctypes/stgdict.c b/Modules/_ctypes/stgdict.c
index 5388a3154b504..af12d314ab98f 100644
--- a/Modules/_ctypes/stgdict.c
+++ b/Modules/_ctypes/stgdict.c
@@ -611,9 +611,9 @@ PyCStructUnionType_update_stgdict(PyObject *type, PyObject *fields, int isStruct
     stgdict->align = total_align;
     stgdict->length = len;      /* ADD ffi_ofs? */
 
-#define MAX_ELEMENTS 16
+#define MAX_STRUCT_SIZE 16
 
-    if (arrays_seen && (size <= 16)) {
+    if (arrays_seen && (size <= MAX_STRUCT_SIZE)) {
         /*
          * See bpo-22273. Arrays are normally treated as pointers, which is
          * fine when an array name is being passed as parameter, but not when
@@ -634,11 +634,49 @@ PyCStructUnionType_update_stgdict(PyObject *type, PyObject *fields, int isStruct
          * Although the passing in registers is specific to 64-bit Linux, the
          * array-in-struct vs. pointer problem is general. But we restrict the
          * type transformation to small structs nonetheless.
+         *
+         * Note that although a union may be small in terms of memory usage, it
+         * could contain many overlapping declarations of arrays, e.g.
+         *
+         * union {
+         *     unsigned int_8 foo [16];
+         *     unsigned uint_8 bar [16];
+         *     unsigned int_16 baz[8];
+         *     unsigned uint_16 bozz[8];
+         *     unsigned int_32 fizz[4];
+         *     unsigned uint_32 buzz[4];
+         * }
+         *
+         * which is still only 16 bytes in size. We need to convert this into
+         * the following equivalent for libffi:
+         *
+         * union {
+         *     struct { int_8 e1; int_8 e2; ... int_8 e_16; } f1;
+         *     struct { uint_8 e1; uint_8 e2; ... uint_8 e_16; } f2;
+         *     struct { int_16 e1; int_16 e2; ... int_16 e_8; } f3;
+         *     struct { uint_16 e1; uint_16 e2; ... uint_16 e_8; } f4;
+         *     struct { int_32 e1; int_32 e2; ... int_32 e_4; } f5;
+         *     struct { uint_32 e1; uint_32 e2; ... uint_32 e_4; } f6;
+         * }
+         *
+         * So the struct/union needs setting up as follows: all non-array
+         * elements copied across as is, and all array elements replaced with
+         * an equivalent struct which has as many fields as the array has
+         * elements, plus one NULL pointer.
          */
-        ffi_type *actual_types[MAX_ELEMENTS + 1];
-        int actual_type_index = 0;
 
-        memset(actual_types, 0, sizeof(actual_types));
+        Py_ssize_t num_ffi_type_pointers = 0;  /* for the dummy fields */
+        Py_ssize_t num_ffi_types = 0;  /* for the dummy structures */
+        size_t alloc_size;  /* total bytes to allocate */
+        void *type_block;  /* to hold all the type information needed */
+        ffi_type **element_types;  /* of this struct/union */
+        ffi_type **dummy_types;  /* of the dummy struct elements */
+        ffi_type *structs;  /* point to struct aliases of arrays */
+        Py_ssize_t element_index;  /* index into element_types for this */
+        Py_ssize_t dummy_index = 0; /* index into dummy field pointers */
+        Py_ssize_t struct_index = 0; /* index into dummy structs */
+
+        /* first pass to see how much memory to allocate */
         for (i = 0; i < len; ++i) {
             PyObject *name, *desc;
             PyObject *pair = PySequence_GetItem(fields, i);
@@ -648,24 +686,123 @@ PyCStructUnionType_update_stgdict(PyObject *type, PyObject *fields, int isStruct
             if (pair == NULL) {
                 return -1;
             }
+            if (!PyArg_ParseTuple(pair, "UO|i", &name, &desc, &bitsize)) {
+                PyErr_SetString(PyExc_TypeError,
+                    "'_fields_' must be a sequence of (name, C type) pairs");
+                Py_DECREF(pair);
+                return -1;
+            }
+            dict = PyType_stgdict(desc);
+            if (dict == NULL) {
+                Py_DECREF(pair);
+                PyErr_Format(PyExc_TypeError,
+                    "second item in _fields_ tuple (index %zd) must be a C type",
+                    i);
+                return -1;
+            }
+            if (!PyCArrayTypeObject_Check(desc)) {
+                /* Not an array. Just need an ffi_type pointer. */
+                num_ffi_type_pointers++;
+            }
+            else {
+                /* It's an array. */
+                Py_ssize_t length = dict->length;
+                StgDictObject *edict;
+
+                edict = PyType_stgdict(dict->proto);
+                if (edict == NULL) {
+                    Py_DECREF(pair);
+                    PyErr_Format(PyExc_TypeError,
+                        "second item in _fields_ tuple (index %zd) must be a C type",
+                        i);
+                    return -1;
+                }
+                /*
+                 * We need one extra ffi_type to hold the struct, and one
+                 * ffi_type pointer per array element + one for a NULL to
+                 * mark the end.
+                 */
+                num_ffi_types++;
+                num_ffi_type_pointers += length + 1;
+            }
+            Py_DECREF(pair);
+        }
+
+        /*
+         * At this point, we know we need storage for some ffi_types and some
+         * ffi_type pointers. We'll allocate these in one block.
+         * There are three sub-blocks of information: the ffi_type pointers to
+         * this structure/union's elements, the ffi_type_pointers to the
+         * dummy fields standing in for array elements, and the
+         * ffi_types representing the dummy structures.
+         */
+        alloc_size = (ffi_ofs + 1 + len + num_ffi_type_pointers) * sizeof(ffi_type *) +
+                        num_ffi_types * sizeof(ffi_type);
+        type_block = PyMem_Malloc(alloc_size);
+
+        if (type_block == NULL) {
+            PyErr_NoMemory();
+            return -1;
+        }
+        /*
+         * the first block takes up ffi_ofs + len + 1 which is the pointers *
+         * for this struct/union. The second block takes up
+         * num_ffi_type_pointers, so the sum of these is ffi_ofs + len + 1 +
+         * num_ffi_type_pointers as allocated above. The last bit is the
+         * num_ffi_types structs.
+         */
+        element_types = (ffi_type **) type_block;
+        dummy_types = &element_types[ffi_ofs + len + 1];
+        structs = (ffi_type *) &dummy_types[num_ffi_type_pointers];
+
+        if (num_ffi_types > 0) {
+            memset(structs, 0, num_ffi_types * sizeof(ffi_type));
+        }
+        if (ffi_ofs && (basedict != NULL)) {
+            memcpy(element_types,
+                basedict->ffi_type_pointer.elements,
+                ffi_ofs * sizeof(ffi_type *));
+        }
+        element_index = ffi_ofs;
+
+        /* second pass to actually set the type pointers */
+        for (i = 0; i < len; ++i) {
+            PyObject *name, *desc;
+            PyObject *pair = PySequence_GetItem(fields, i);
+            StgDictObject *dict;
+            int bitsize = 0;
+
+            if (pair == NULL) {
+                PyMem_Free(type_block);
+                return -1;
+            }
+            /* In theory, we made this call in the first pass, so it *shouldn't*
+             * fail. However, you never know, and the code above might change
+             * later - keeping the check in here is a tad defensive but it
+             * will affect program size only slightly and performance hardly at
+             * all.
+             */
             if (!PyArg_ParseTuple(pair, "UO|i", &name, &desc, &bitsize)) {
                 PyErr_SetString(PyExc_TypeError,
                                 "'_fields_' must be a sequence of (name, C type) pairs");
-                Py_XDECREF(pair);
+                Py_DECREF(pair);
+                PyMem_Free(type_block);
                 return -1;
             }
             dict = PyType_stgdict(desc);
+            /* Possibly this check could be avoided, but see above comment. */
             if (dict == NULL) {
                 Py_DECREF(pair);
+                PyMem_Free(type_block);
                 PyErr_Format(PyExc_TypeError,
                              "second item in _fields_ tuple (index %zd) must be a C type",
                              i);
                 return -1;
             }
+            assert(element_index < (ffi_ofs + len)); /* will be used below */
             if (!PyCArrayTypeObject_Check(desc)) {
                 /* Not an array. Just copy over the element ffi_type. */
-                actual_types[actual_type_index++] = &dict->ffi_type_pointer;
-                assert(actual_type_index <= MAX_ELEMENTS);
+                element_types[element_index++] = &dict->ffi_type_pointer;
             }
             else {
                 int length = dict->length;
@@ -674,42 +811,38 @@ PyCStructUnionType_update_stgdict(PyObject *type, PyObject *fields, int isStruct
                 edict = PyType_stgdict(dict->proto);
                 if (edict == NULL) {
                     Py_DECREF(pair);
+                    PyMem_Free(type_block);
                     PyErr_Format(PyExc_TypeError,
                                  "second item in _fields_ tuple (index %zd) must be a C type",
                                  i);
                     return -1;
                 }
+                element_types[element_index++] = &structs[struct_index];
+                structs[struct_index].size = length * edict->ffi_type_pointer.size;
+                structs[struct_index].alignment = edict->ffi_type_pointer.alignment;
+                structs[struct_index].type = FFI_TYPE_STRUCT;
+                structs[struct_index].elements = &dummy_types[dummy_index];
+                ++struct_index;
                 /* Copy over the element's type, length times. */
                 while (length > 0) {
-                    actual_types[actual_type_index++] = &edict->ffi_type_pointer;
-                    assert(actual_type_index <= MAX_ELEMENTS);
+                    assert(dummy_index < (num_ffi_type_pointers));
+                    dummy_types[dummy_index++] = &edict->ffi_type_pointer;
                     length--;
                 }
+                assert(dummy_index < (num_ffi_type_pointers));
+                dummy_types[dummy_index++] = NULL;
             }
             Py_DECREF(pair);
         }
 
-        actual_types[actual_type_index++] = NULL;
+        element_types[element_index] = NULL;
         /*
          * Replace the old elements with the new, taking into account
          * base class elements where necessary.
          */
         assert(stgdict->ffi_type_pointer.elements);
         PyMem_Free(stgdict->ffi_type_pointer.elements);
-        stgdict->ffi_type_pointer.elements = PyMem_New(ffi_type *,
-                                                       ffi_ofs + actual_type_index);
-        if (stgdict->ffi_type_pointer.elements == NULL) {
-            PyErr_NoMemory();
-            return -1;
-        }
-        if (ffi_ofs) {
-            memcpy(stgdict->ffi_type_pointer.elements,
-                   basedict->ffi_type_pointer.elements,
-                   ffi_ofs * sizeof(ffi_type *));
-
-        }
-        memcpy(&stgdict->ffi_type_pointer.elements[ffi_ofs], actual_types,
-               actual_type_index * sizeof(ffi_type *));
+        stgdict->ffi_type_pointer.elements = element_types;
     }
 
     /* We did check that this flag was NOT set above, it must not


From webhook-mailer at python.org  Wed Oct  9 05:29:38 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Wed, 09 Oct 2019 09:29:38 -0000
Subject: [Python-checkins] bpo-36953: Delay removal of ABCs from
 collections. (GH-13409)
Message-ID: <mailman.69.1570613379.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/056fa7f52a49a758b1ed5f5a81d8b7ce5c71cec4
commit: 056fa7f52a49a758b1ed5f5a81d8b7ce5c71cec4
branch: 3.7
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-09T02:29:31-07:00
summary:

bpo-36953: Delay removal of ABCs from collections. (GH-13409)


Bump the removal to 3.9, indicate collections.abc available since 3.3,
replace version-changed directive to deprecated-removed.

https://bugs.python.org/issue36953
(cherry picked from commit eea47e09394dfb64d3a59a601d947d25bb1bdc96)

Co-authored-by: Matthias Bussonnier <mbussonnier at ucmerced.edu>

files:
A Misc/NEWS.d/next/Library/2019-05-20-08-54-41.bpo-36952.I_glok.rst
M Doc/library/collections.rst
M Lib/collections/__init__.py

diff --git a/Doc/library/collections.rst b/Doc/library/collections.rst
index 9f47e8978da35..e9dbd094e0a96 100644
--- a/Doc/library/collections.rst
+++ b/Doc/library/collections.rst
@@ -33,10 +33,10 @@ Python's general purpose built-in containers, :class:`dict`, :class:`list`,
 :class:`UserString`     wrapper around string objects for easier string subclassing
 =====================   ====================================================================
 
-.. versionchanged:: 3.3
+.. deprecated-removed:: 3.3 3.9
     Moved :ref:`collections-abstract-base-classes` to the :mod:`collections.abc` module.
     For backwards compatibility, they continue to be visible in this module through
-    Python 3.7.  Subsequently, they will be removed entirely.
+    Python 3.8.
 
 
 :class:`ChainMap` objects
diff --git a/Lib/collections/__init__.py b/Lib/collections/__init__.py
index 7ff3c80a06a7e..d353ff2b6f127 100644
--- a/Lib/collections/__init__.py
+++ b/Lib/collections/__init__.py
@@ -47,8 +47,8 @@ def __getattr__(name):
         obj = getattr(_collections_abc, name)
         import warnings
         warnings.warn("Using or importing the ABCs from 'collections' instead "
-                      "of from 'collections.abc' is deprecated, "
-                      "and in 3.8 it will stop working",
+                      "of from 'collections.abc' is deprecated since Python 3.3,"
+                      "and in 3.9 it will stop working",
                       DeprecationWarning, stacklevel=2)
         globals()[name] = obj
         return obj
diff --git a/Misc/NEWS.d/next/Library/2019-05-20-08-54-41.bpo-36952.I_glok.rst b/Misc/NEWS.d/next/Library/2019-05-20-08-54-41.bpo-36952.I_glok.rst
new file mode 100644
index 0000000000000..eeb4fd71e67bf
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-05-20-08-54-41.bpo-36952.I_glok.rst
@@ -0,0 +1,5 @@
+Starting with Python 3.3, importing ABCs from :mod:`collections` is
+deprecated, and import should be done from :mod:`collections.abc`. Still
+being able to import from :mod:`collections` was marked for removal in 3.8,
+but has been delayed to 3.9; documentation and ``DeprecationWarning``
+clarified.


From webhook-mailer at python.org  Wed Oct  9 05:56:53 2019
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Wed, 09 Oct 2019 09:56:53 -0000
Subject: [Python-checkins] [3.7] bpo-38405: Make nested subclasses of
 typing.NamedTuple pickleable. (GH-16641). (GH-16674)
Message-ID: <mailman.70.1570615013.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/87db4d343c9609137f6e62aa3595db610a33842c
commit: 87db4d343c9609137f6e62aa3595db610a33842c
branch: 3.7
author: Serhiy Storchaka <storchaka at gmail.com>
committer: GitHub <noreply at github.com>
date: 2019-10-09T12:56:32+03:00
summary:

[3.7] bpo-38405: Make nested subclasses of typing.NamedTuple pickleable. (GH-16641). (GH-16674)

(cherry picked from commit 13abda41003daf599587991d8291f0dacf6e9519)

files:
A Misc/NEWS.d/next/Library/2019-10-08-11-18-40.bpo-38405.0-7e7s.rst
M Lib/test/test_typing.py
M Lib/typing.py

diff --git a/Lib/test/test_typing.py b/Lib/test/test_typing.py
index 4871fb7c4ddc3..d8abc0ac96a10 100644
--- a/Lib/test/test_typing.py
+++ b/Lib/test/test_typing.py
@@ -2441,6 +2441,9 @@ class D(UserName):
 
 
 class NamedTupleTests(BaseTestCase):
+    class NestedEmployee(NamedTuple):
+        name: str
+        cool: int
 
     def test_basics(self):
         Emp = NamedTuple('Emp', [('name', str), ('id', int)])
@@ -2564,14 +2567,25 @@ def test_namedtuple_errors(self):
         self.assertEqual(Emp.__name__, 'Emp')
         self.assertEqual(Emp._fields, ('name', 'id'))
 
-    def test_pickle(self):
+    def test_copy_and_pickle(self):
         global Emp  # pickle wants to reference the class by name
-        Emp = NamedTuple('Emp', [('name', str), ('id', int)])
-        jane = Emp('jane', 37)
-        for proto in range(pickle.HIGHEST_PROTOCOL + 1):
-            z = pickle.dumps(jane, proto)
-            jane2 = pickle.loads(z)
-            self.assertEqual(jane2, jane)
+        Emp = NamedTuple('Emp', [('name', str), ('cool', int)])
+        for cls in Emp, CoolEmployee, self.NestedEmployee:
+            with self.subTest(cls=cls):
+                jane = cls('jane', 37)
+                for proto in range(pickle.HIGHEST_PROTOCOL + 1):
+                    z = pickle.dumps(jane, proto)
+                    jane2 = pickle.loads(z)
+                    self.assertEqual(jane2, jane)
+                    self.assertIsInstance(jane2, cls)
+
+                jane2 = copy(jane)
+                self.assertEqual(jane2, jane)
+                self.assertIsInstance(jane2, cls)
+
+                jane2 = deepcopy(jane)
+                self.assertEqual(jane2, jane)
+                self.assertIsInstance(jane2, cls)
 
 
 class IOTests(BaseTestCase):
diff --git a/Lib/typing.py b/Lib/typing.py
index 021ef9552e652..dea7babaf792f 100644
--- a/Lib/typing.py
+++ b/Lib/typing.py
@@ -1343,7 +1343,7 @@ def _make_nmtuple(name, types):
                '_fields', '_field_defaults', '_field_types',
                '_make', '_replace', '_asdict', '_source')
 
-_special = ('__module__', '__name__', '__qualname__', '__annotations__')
+_special = ('__module__', '__name__', '__annotations__')
 
 
 class NamedTupleMeta(type):
diff --git a/Misc/NEWS.d/next/Library/2019-10-08-11-18-40.bpo-38405.0-7e7s.rst b/Misc/NEWS.d/next/Library/2019-10-08-11-18-40.bpo-38405.0-7e7s.rst
new file mode 100644
index 0000000000000..ee346a30ec416
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-10-08-11-18-40.bpo-38405.0-7e7s.rst
@@ -0,0 +1 @@
+Nested subclasses of :class:`typing.NamedTuple` are now pickleable.


From webhook-mailer at python.org  Wed Oct  9 05:57:19 2019
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Wed, 09 Oct 2019 09:57:19 -0000
Subject: [Python-checkins] [3.7] bpo-38407: Add docstrings for
 typing.SupportsXXX classes. (GH-16644). (GH-16673)
Message-ID: <mailman.71.1570615040.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/0b354fc2bfe1a4624551907df396ed36ea8bf4ca
commit: 0b354fc2bfe1a4624551907df396ed36ea8bf4ca
branch: 3.7
author: Serhiy Storchaka <storchaka at gmail.com>
committer: GitHub <noreply at github.com>
date: 2019-10-09T12:57:14+03:00
summary:

[3.7] bpo-38407: Add docstrings for typing.SupportsXXX classes. (GH-16644). (GH-16673)

(cherry picked from commit 8252c52e57283515ace5d4251584255dc5c60eb5)

files:
M Lib/typing.py

diff --git a/Lib/typing.py b/Lib/typing.py
index dea7babaf792f..ea8dea54a5a3d 100644
--- a/Lib/typing.py
+++ b/Lib/typing.py
@@ -1277,6 +1277,7 @@ def new_user(user_class: Type[U]) -> U:
 
 
 class SupportsInt(_Protocol):
+    """An ABC with one abstract method __int__."""
     __slots__ = ()
 
     @abstractmethod
@@ -1285,6 +1286,7 @@ def __int__(self) -> int:
 
 
 class SupportsFloat(_Protocol):
+    """An ABC with one abstract method __float__."""
     __slots__ = ()
 
     @abstractmethod
@@ -1293,6 +1295,7 @@ def __float__(self) -> float:
 
 
 class SupportsComplex(_Protocol):
+    """An ABC with one abstract method __complex__."""
     __slots__ = ()
 
     @abstractmethod
@@ -1301,6 +1304,7 @@ def __complex__(self) -> complex:
 
 
 class SupportsBytes(_Protocol):
+    """An ABC with one abstract method __bytes__."""
     __slots__ = ()
 
     @abstractmethod
@@ -1309,6 +1313,7 @@ def __bytes__(self) -> bytes:
 
 
 class SupportsAbs(_Protocol[T_co]):
+    """An ABC with one abstract method __abs__ that is covariant in its return type."""
     __slots__ = ()
 
     @abstractmethod
@@ -1317,6 +1322,7 @@ def __abs__(self) -> T_co:
 
 
 class SupportsRound(_Protocol[T_co]):
+    """An ABC with one abstract method __round__ that is covariant in its return type."""
     __slots__ = ()
 
     @abstractmethod


From webhook-mailer at python.org  Wed Oct  9 09:52:38 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Wed, 09 Oct 2019 13:52:38 -0000
Subject: [Python-checkins] bpo-37531: regrtest ignores output on timeout
 (GH-16659)
Message-ID: <mailman.72.1570629160.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/598bfa4d457d61431142ad99ecbb9bd10cf314e6
commit: 598bfa4d457d61431142ad99ecbb9bd10cf314e6
branch: 3.7
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-09T06:52:34-07:00
summary:

bpo-37531: regrtest ignores output on timeout (GH-16659)


bpo-37531, bpo-38207: On timeout, regrtest no longer attempts to call
`popen.communicate() again: it can hang until all child processes
using stdout and stderr pipes completes. Kill the worker process and
ignores its output.

Reenable test_regrtest.test_multiprocessing_timeout().

bpo-37531: Change also the faulthandler timeout of the main process
from 1 minute to 5 minutes, for Python slowest buildbots.
(cherry picked from commit 0ec618af98ac250a91ee9c91f8569e6df6772758)

Co-authored-by: Victor Stinner <vstinner at python.org>

files:
A Misc/NEWS.d/next/Tests/2019-10-08-16-42-05.bpo-37531.7v-_Ca.rst
M Lib/test/libregrtest/runtest_mp.py

diff --git a/Lib/test/libregrtest/runtest_mp.py b/Lib/test/libregrtest/runtest_mp.py
index 9d5a699852b9c..2770cf93bd7d8 100644
--- a/Lib/test/libregrtest/runtest_mp.py
+++ b/Lib/test/libregrtest/runtest_mp.py
@@ -22,6 +22,12 @@
 PROGRESS_UPDATE = 30.0   # seconds
 assert PROGRESS_UPDATE >= PROGRESS_MIN_TIME
 
+# Kill the main process after 5 minutes. It is supposed to write an update
+# every PROGRESS_UPDATE seconds. Tolerate 5 minutes for Python slowest
+# buildbot workers.
+MAIN_PROCESS_TIMEOUT = 5 * 60.0
+assert MAIN_PROCESS_TIMEOUT >= PROGRESS_UPDATE
+
 # Time to wait until a worker completes: should be immediate
 JOIN_TIMEOUT = 30.0   # seconds
 
@@ -159,20 +165,6 @@ def mp_result_error(self, test_name, error_type, stdout='', stderr='',
         result = TestResult(test_name, error_type, test_time, None)
         return MultiprocessResult(result, stdout, stderr, err_msg)
 
-    def _timedout(self, test_name):
-        self._kill()
-
-        stdout = stderr = ''
-        popen = self._popen
-        try:
-            stdout, stderr = popen.communicate(timeout=JOIN_TIMEOUT)
-        except (subprocess.TimeoutExpired, OSError) as exc:
-            print_warning(f"Failed to read {self} output "
-                          f"(timeout={format_duration(JOIN_TIMEOUT)}): "
-                          f"{exc!r}")
-
-        return self.mp_result_error(test_name, TIMEOUT, stdout, stderr)
-
     def _run_process(self, test_name):
         self.start_time = time.monotonic()
 
@@ -195,23 +187,32 @@ def _run_process(self, test_name):
 
             try:
                 stdout, stderr = popen.communicate(timeout=self.timeout)
+                retcode = popen.returncode
+                assert retcode is not None
             except subprocess.TimeoutExpired:
                 if self._stopped:
                     # kill() has been called: communicate() fails
                     # on reading closed stdout/stderr
                     raise ExitThread
 
-                return self._timedout(test_name)
+                # On timeout, kill the process
+                self._kill()
+
+                # None means TIMEOUT for the caller
+                retcode = None
+                # bpo-38207: Don't attempt to call communicate() again: on it
+                # can hang until all child processes using stdout and stderr
+                # pipes completes.
+                stdout = stderr = ''
             except OSError:
                 if self._stopped:
                     # kill() has been called: communicate() fails
                     # on reading closed stdout/stderr
                     raise ExitThread
                 raise
-
-            retcode = popen.returncode
-            stdout = stdout.strip()
-            stderr = stderr.rstrip()
+            else:
+                stdout = stdout.strip()
+                stderr = stderr.rstrip()
 
             return (retcode, stdout, stderr)
         except:
@@ -223,13 +224,10 @@ def _run_process(self, test_name):
             self.current_test_name = None
 
     def _runtest(self, test_name):
-        result = self._run_process(test_name)
-
-        if isinstance(result, MultiprocessResult):
-            # _timedout() case
-            return result
+        retcode, stdout, stderr = self._run_process(test_name)
 
-        retcode, stdout, stderr = result
+        if retcode is None:
+            return self.mp_result_error(test_name, TIMEOUT, stdout, stderr)
 
         err_msg = None
         if retcode != 0:
@@ -248,7 +246,8 @@ def _runtest(self, test_name):
                     err_msg = "Failed to parse worker JSON: %s" % exc
 
         if err_msg is not None:
-            return self.mp_result_error(test_name, CHILD_ERROR, stdout, stderr, err_msg)
+            return self.mp_result_error(test_name, CHILD_ERROR,
+                                        stdout, stderr, err_msg)
 
         return MultiprocessResult(result, stdout, stderr, err_msg)
 
@@ -287,6 +286,16 @@ def _wait_completed(self):
                           f"{exc!r}")
 
     def wait_stopped(self, start_time):
+        # bpo-38207: MultiprocessTestRunner.stop_workers() called self.stop()
+        # which killed the process. Sometimes, killing the process from the
+        # main thread does not interrupt popen.communicate() in
+        # TestWorkerProcess thread. This loop with a timeout is a workaround
+        # for that.
+        #
+        # Moreover, if this method fails to join the thread, it is likely
+        # that Python will hang at exit while calling threading._shutdown()
+        # which tries again to join the blocked thread. Regrtest.main()
+        # uses EXIT_TIMEOUT to workaround this second bug.
         while True:
             # Write a message every second
             self.join(1.0)
@@ -353,7 +362,8 @@ def _get_result(self):
         timeout = PROGRESS_UPDATE
         while True:
             if use_faulthandler:
-                faulthandler.dump_traceback_later(timeout * 2.0, exit=True)
+                faulthandler.dump_traceback_later(MAIN_PROCESS_TIMEOUT,
+                                                  exit=True)
 
             # wait for a thread
             try:
diff --git a/Misc/NEWS.d/next/Tests/2019-10-08-16-42-05.bpo-37531.7v-_Ca.rst b/Misc/NEWS.d/next/Tests/2019-10-08-16-42-05.bpo-37531.7v-_Ca.rst
new file mode 100644
index 0000000000000..78ab636516514
--- /dev/null
+++ b/Misc/NEWS.d/next/Tests/2019-10-08-16-42-05.bpo-37531.7v-_Ca.rst
@@ -0,0 +1,5 @@
+On timeout, regrtest no longer attempts to call ``popen.communicate()``
+again: it can hang until all child processes using stdout and stderr pipes
+completes. Kill the worker process and ignores its output. Change also the
+faulthandler timeout of the main process from 1 minute to 5 minutes, for Python
+slowest buildbots.


From webhook-mailer at python.org  Wed Oct  9 10:02:19 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Wed, 09 Oct 2019 14:02:19 -0000
Subject: [Python-checkins] bpo-37531: regrtest ignores output on timeout
 (GH-16659)
Message-ID: <mailman.73.1570629740.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/359a1975cbca488ccd5ea13bd7268d7e88664078
commit: 359a1975cbca488ccd5ea13bd7268d7e88664078
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-09T07:02:15-07:00
summary:

bpo-37531: regrtest ignores output on timeout (GH-16659)


bpo-37531, bpo-38207: On timeout, regrtest no longer attempts to call
`popen.communicate() again: it can hang until all child processes
using stdout and stderr pipes completes. Kill the worker process and
ignores its output.

Reenable test_regrtest.test_multiprocessing_timeout().

bpo-37531: Change also the faulthandler timeout of the main process
from 1 minute to 5 minutes, for Python slowest buildbots.
(cherry picked from commit 0ec618af98ac250a91ee9c91f8569e6df6772758)

Co-authored-by: Victor Stinner <vstinner at python.org>

files:
A Misc/NEWS.d/next/Tests/2019-10-08-16-42-05.bpo-37531.7v-_Ca.rst
M Lib/test/libregrtest/runtest_mp.py

diff --git a/Lib/test/libregrtest/runtest_mp.py b/Lib/test/libregrtest/runtest_mp.py
index 79afa29fa05bf..71db8d6966859 100644
--- a/Lib/test/libregrtest/runtest_mp.py
+++ b/Lib/test/libregrtest/runtest_mp.py
@@ -22,6 +22,12 @@
 PROGRESS_UPDATE = 30.0   # seconds
 assert PROGRESS_UPDATE >= PROGRESS_MIN_TIME
 
+# Kill the main process after 5 minutes. It is supposed to write an update
+# every PROGRESS_UPDATE seconds. Tolerate 5 minutes for Python slowest
+# buildbot workers.
+MAIN_PROCESS_TIMEOUT = 5 * 60.0
+assert MAIN_PROCESS_TIMEOUT >= PROGRESS_UPDATE
+
 # Time to wait until a worker completes: should be immediate
 JOIN_TIMEOUT = 30.0   # seconds
 
@@ -160,20 +166,6 @@ def mp_result_error(self, test_name, error_type, stdout='', stderr='',
         result = TestResult(test_name, error_type, test_time, None)
         return MultiprocessResult(result, stdout, stderr, err_msg)
 
-    def _timedout(self, test_name):
-        self._kill()
-
-        stdout = stderr = ''
-        popen = self._popen
-        try:
-            stdout, stderr = popen.communicate(timeout=JOIN_TIMEOUT)
-        except (subprocess.TimeoutExpired, OSError) as exc:
-            print_warning(f"Failed to read {self} output "
-                          f"(timeout={format_duration(JOIN_TIMEOUT)}): "
-                          f"{exc!r}")
-
-        return self.mp_result_error(test_name, TIMEOUT, stdout, stderr)
-
     def _run_process(self, test_name):
         self.start_time = time.monotonic()
 
@@ -197,23 +189,32 @@ def _run_process(self, test_name):
 
             try:
                 stdout, stderr = popen.communicate(timeout=self.timeout)
+                retcode = popen.returncode
+                assert retcode is not None
             except subprocess.TimeoutExpired:
                 if self._stopped:
                     # kill() has been called: communicate() fails
                     # on reading closed stdout/stderr
                     raise ExitThread
 
-                return self._timedout(test_name)
+                # On timeout, kill the process
+                self._kill()
+
+                # None means TIMEOUT for the caller
+                retcode = None
+                # bpo-38207: Don't attempt to call communicate() again: on it
+                # can hang until all child processes using stdout and stderr
+                # pipes completes.
+                stdout = stderr = ''
             except OSError:
                 if self._stopped:
                     # kill() has been called: communicate() fails
                     # on reading closed stdout/stderr
                     raise ExitThread
                 raise
-
-            retcode = popen.returncode
-            stdout = stdout.strip()
-            stderr = stderr.rstrip()
+            else:
+                stdout = stdout.strip()
+                stderr = stderr.rstrip()
 
             return (retcode, stdout, stderr)
         except:
@@ -225,13 +226,10 @@ def _run_process(self, test_name):
             self.current_test_name = None
 
     def _runtest(self, test_name):
-        result = self._run_process(test_name)
-
-        if isinstance(result, MultiprocessResult):
-            # _timedout() case
-            return result
+        retcode, stdout, stderr = self._run_process(test_name)
 
-        retcode, stdout, stderr = result
+        if retcode is None:
+            return self.mp_result_error(test_name, TIMEOUT, stdout, stderr)
 
         err_msg = None
         if retcode != 0:
@@ -250,7 +248,8 @@ def _runtest(self, test_name):
                     err_msg = "Failed to parse worker JSON: %s" % exc
 
         if err_msg is not None:
-            return self.mp_result_error(test_name, CHILD_ERROR, stdout, stderr, err_msg)
+            return self.mp_result_error(test_name, CHILD_ERROR,
+                                        stdout, stderr, err_msg)
 
         return MultiprocessResult(result, stdout, stderr, err_msg)
 
@@ -289,6 +288,16 @@ def _wait_completed(self):
                           f"{exc!r}")
 
     def wait_stopped(self, start_time):
+        # bpo-38207: MultiprocessTestRunner.stop_workers() called self.stop()
+        # which killed the process. Sometimes, killing the process from the
+        # main thread does not interrupt popen.communicate() in
+        # TestWorkerProcess thread. This loop with a timeout is a workaround
+        # for that.
+        #
+        # Moreover, if this method fails to join the thread, it is likely
+        # that Python will hang at exit while calling threading._shutdown()
+        # which tries again to join the blocked thread. Regrtest.main()
+        # uses EXIT_TIMEOUT to workaround this second bug.
         while True:
             # Write a message every second
             self.join(1.0)
@@ -355,7 +364,8 @@ def _get_result(self):
         timeout = PROGRESS_UPDATE
         while True:
             if use_faulthandler:
-                faulthandler.dump_traceback_later(timeout * 2.0, exit=True)
+                faulthandler.dump_traceback_later(MAIN_PROCESS_TIMEOUT,
+                                                  exit=True)
 
             # wait for a thread
             try:
diff --git a/Misc/NEWS.d/next/Tests/2019-10-08-16-42-05.bpo-37531.7v-_Ca.rst b/Misc/NEWS.d/next/Tests/2019-10-08-16-42-05.bpo-37531.7v-_Ca.rst
new file mode 100644
index 0000000000000..78ab636516514
--- /dev/null
+++ b/Misc/NEWS.d/next/Tests/2019-10-08-16-42-05.bpo-37531.7v-_Ca.rst
@@ -0,0 +1,5 @@
+On timeout, regrtest no longer attempts to call ``popen.communicate()``
+again: it can hang until all child processes using stdout and stderr pipes
+completes. Kill the worker process and ignores its output. Change also the
+faulthandler timeout of the main process from 1 minute to 5 minutes, for Python
+slowest buildbots.


From webhook-mailer at python.org  Wed Oct  9 13:37:42 2019
From: webhook-mailer at python.org (Tim Peters)
Date: Wed, 09 Oct 2019 17:37:42 -0000
Subject: [Python-checkins] bpo-38379: don't claim objects are collected when
 they aren't (#16658)
Message-ID: <mailman.74.1570642663.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/ecbf35f9335b0420cb8adfda6f299d6747a16515
commit: ecbf35f9335b0420cb8adfda6f299d6747a16515
branch: master
author: Tim Peters <tim.peters at gmail.com>
committer: GitHub <noreply at github.com>
date: 2019-10-09T12:37:30-05:00
summary:

bpo-38379:  don't claim objects are collected when they aren't (#16658)

* bpo-38379:  when a finalizer resurrects an object,
nothing is actually collected in this run of gc.
Change the stats to relect that truth.

files:
A Misc/NEWS.d/next/Core and Builtins/2019-10-09-16-50-52.bpo-38379.oz5qZx.rst
M Lib/test/test_gc.py
M Modules/gcmodule.c

diff --git a/Lib/test/test_gc.py b/Lib/test/test_gc.py
index 8215390cb7089..f52db1eab169c 100644
--- a/Lib/test/test_gc.py
+++ b/Lib/test/test_gc.py
@@ -822,6 +822,76 @@ def test_get_objects_arguments(self):
         self.assertRaises(TypeError, gc.get_objects, "1")
         self.assertRaises(TypeError, gc.get_objects, 1.234)
 
+    def test_38379(self):
+        # When a finalizer resurrects objects, stats were reporting them as
+        # having been collected.  This affected both collect()'s return
+        # value and the dicts returned by get_stats().
+        N = 100
+
+        class A:  # simple self-loop
+            def __init__(self):
+                self.me = self
+
+        class Z(A):  # resurrecting __del__
+            def __del__(self):
+                zs.append(self)
+
+        zs = []
+
+        def getstats():
+            d = gc.get_stats()[-1]
+            return d['collected'], d['uncollectable']
+
+        gc.collect()
+        gc.disable()
+
+        # No problems if just collecting A() instances.
+        oldc, oldnc = getstats()
+        for i in range(N):
+            A()
+        t = gc.collect()
+        c, nc = getstats()
+        self.assertEqual(t, 2*N) # instance object & its dict
+        self.assertEqual(c - oldc, 2*N)
+        self.assertEqual(nc - oldnc, 0)
+
+        # But Z() is not actually collected.
+        oldc, oldnc = c, nc
+        Z()
+        # Nothing is collected - Z() is merely resurrected.
+        t = gc.collect()
+        c, nc = getstats()
+        #self.assertEqual(t, 2)  # before
+        self.assertEqual(t, 0)  # after
+        #self.assertEqual(c - oldc, 2)   # before
+        self.assertEqual(c - oldc, 0)   # after
+        self.assertEqual(nc - oldnc, 0)
+
+        # Unfortunately, a Z() prevents _anything_ from being collected.
+        # It should be possible to collect the A instances anyway, but
+        # that will require non-trivial code changes.
+        oldc, oldnc = c, nc
+        for i in range(N):
+            A()
+        Z()
+        # Z() prevents anything from being collected.
+        t = gc.collect()
+        c, nc = getstats()
+        #self.assertEqual(t, 2*N + 2)  # before
+        self.assertEqual(t, 0)  # after
+        #self.assertEqual(c - oldc, 2*N + 2)   # before
+        self.assertEqual(c - oldc, 0)   # after
+        self.assertEqual(nc - oldnc, 0)
+
+        # But the A() trash is reclaimed on the next run.
+        oldc, oldnc = c, nc
+        t = gc.collect()
+        c, nc = getstats()
+        self.assertEqual(t, 2*N)
+        self.assertEqual(c - oldc, 2*N)
+        self.assertEqual(nc - oldnc, 0)
+
+        gc.enable()
 
 class GCCallbackTests(unittest.TestCase):
     def setUp(self):
diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-10-09-16-50-52.bpo-38379.oz5qZx.rst b/Misc/NEWS.d/next/Core and Builtins/2019-10-09-16-50-52.bpo-38379.oz5qZx.rst
new file mode 100644
index 0000000000000..82dcb525dd49d
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2019-10-09-16-50-52.bpo-38379.oz5qZx.rst	
@@ -0,0 +1 @@
+When cyclic garbage collection (gc) runs finalizers that resurrect unreachable objects, the current gc run ends, without collecting any cyclic trash.  However, the statistics reported by ``collect()`` and ``get_stats()`` claimed that all cyclic trash found was collected, and that the resurrected objects were collected.   Changed the stats to report that none were collected.
diff --git a/Modules/gcmodule.c b/Modules/gcmodule.c
index 2b47abae1ae4d..766f8e0c67fa8 100644
--- a/Modules/gcmodule.c
+++ b/Modules/gcmodule.c
@@ -1095,12 +1095,9 @@ collect(struct _gc_runtime_state *state, int generation,
     validate_list(&finalizers, 0);
     validate_list(&unreachable, PREV_MASK_COLLECTING);
 
-    /* Collect statistics on collectable objects found and print
-     * debugging information.
-     */
-    for (gc = GC_NEXT(&unreachable); gc != &unreachable; gc = GC_NEXT(gc)) {
-        m++;
-        if (state->debug & DEBUG_COLLECTABLE) {
+    /* Print debugging information. */
+    if (state->debug & DEBUG_COLLECTABLE) {
+        for (gc = GC_NEXT(&unreachable); gc != &unreachable; gc = GC_NEXT(gc)) {
             debug_cycle("collectable", FROM_GC(gc));
         }
     }
@@ -1122,6 +1119,7 @@ collect(struct _gc_runtime_state *state, int generation,
          * the reference cycles to be broken.  It may also cause some objects
          * in finalizers to be freed.
          */
+        m += gc_list_size(&unreachable);
         delete_garbage(state, &unreachable, old);
     }
 


From webhook-mailer at python.org  Wed Oct  9 17:09:04 2019
From: webhook-mailer at python.org (Zachary Ware)
Date: Wed, 09 Oct 2019 21:09:04 -0000
Subject: [Python-checkins] bpo-38409: Fix grammar in str.strip() docstring
 (GH-16682)
Message-ID: <mailman.75.1570655345.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/09895c27cd8ff60563a794016e8c099bc897cc74
commit: 09895c27cd8ff60563a794016e8c099bc897cc74
branch: master
author: Zachary Ware <zachary.ware at gmail.com>
committer: GitHub <noreply at github.com>
date: 2019-10-09T16:09:00-05:00
summary:

bpo-38409: Fix grammar in str.strip() docstring (GH-16682)

files:
M Objects/clinic/unicodeobject.c.h
M Objects/unicodeobject.c

diff --git a/Objects/clinic/unicodeobject.c.h b/Objects/clinic/unicodeobject.c.h
index c7d6edb5fd488..0d134064bab09 100644
--- a/Objects/clinic/unicodeobject.c.h
+++ b/Objects/clinic/unicodeobject.c.h
@@ -582,7 +582,7 @@ PyDoc_STRVAR(unicode_strip__doc__,
 "strip($self, chars=None, /)\n"
 "--\n"
 "\n"
-"Return a copy of the string with leading and trailing whitespace remove.\n"
+"Return a copy of the string with leading and trailing whitespace removed.\n"
 "\n"
 "If chars is given and not None, remove characters in chars instead.");
 
@@ -1232,4 +1232,4 @@ unicode_sizeof(PyObject *self, PyObject *Py_UNUSED(ignored))
 {
     return unicode_sizeof_impl(self);
 }
-/*[clinic end generated code: output=5e15747f78f18329 input=a9049054013a1b77]*/
+/*[clinic end generated code: output=e4ed33400979c7e8 input=a9049054013a1b77]*/
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
index 8d442fb6d82b8..2d60627b19193 100644
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -12544,14 +12544,14 @@ str.strip as unicode_strip
     chars: object = None
     /
 
-Return a copy of the string with leading and trailing whitespace remove.
+Return a copy of the string with leading and trailing whitespace removed.
 
 If chars is given and not None, remove characters in chars instead.
 [clinic start generated code]*/
 
 static PyObject *
 unicode_strip_impl(PyObject *self, PyObject *chars)
-/*[clinic end generated code: output=ca19018454345d57 input=eefe24a1059c352b]*/
+/*[clinic end generated code: output=ca19018454345d57 input=385289c6f423b954]*/
 {
     return do_argstrip(self, BOTHSTRIP, chars);
 }


From webhook-mailer at python.org  Wed Oct  9 17:25:10 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Wed, 09 Oct 2019 21:25:10 -0000
Subject: [Python-checkins] [3.8] bpo-38379: don't claim objects are
 collected when they aren't (GH-16658) (GH-16683)
Message-ID: <mailman.76.1570656310.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/0bd9fac7a866ec886ae8f93f9c24dcda6d436929
commit: 0bd9fac7a866ec886ae8f93f9c24dcda6d436929
branch: 3.8
author: Pablo Galindo <Pablogsal at gmail.com>
committer: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
date: 2019-10-09T14:25:06-07:00
summary:

[3.8] bpo-38379:  don't claim objects are collected when they aren't  (GH-16658) (GH-16683)



* [bpo-38379](https://bugs.python.org/issue38379):  when a finalizer resurrects an object,
nothing is actually collected in this run of gc.
Change the stats to relect that truth.
(cherry picked from commit ecbf35f9335b0420cb8adfda6f299d6747a16515)

Co-authored-by: Tim Peters <tim.peters at gmail.com>





https://bugs.python.org/issue38379



Automerge-Triggered-By: @pablogsal

files:
A Misc/NEWS.d/next/Core and Builtins/2019-10-09-16-50-52.bpo-38379.oz5qZx.rst
M Lib/test/test_gc.py
M Modules/gcmodule.c

diff --git a/Lib/test/test_gc.py b/Lib/test/test_gc.py
index 311143da91d07..6c29e96c30fbd 100644
--- a/Lib/test/test_gc.py
+++ b/Lib/test/test_gc.py
@@ -822,6 +822,76 @@ def test_get_objects_arguments(self):
         self.assertRaises(TypeError, gc.get_objects, "1")
         self.assertRaises(TypeError, gc.get_objects, 1.234)
 
+    def test_38379(self):
+        # When a finalizer resurrects objects, stats were reporting them as
+        # having been collected.  This affected both collect()'s return
+        # value and the dicts returned by get_stats().
+        N = 100
+
+        class A:  # simple self-loop
+            def __init__(self):
+                self.me = self
+
+        class Z(A):  # resurrecting __del__
+            def __del__(self):
+                zs.append(self)
+
+        zs = []
+
+        def getstats():
+            d = gc.get_stats()[-1]
+            return d['collected'], d['uncollectable']
+
+        gc.collect()
+        gc.disable()
+
+        # No problems if just collecting A() instances.
+        oldc, oldnc = getstats()
+        for i in range(N):
+            A()
+        t = gc.collect()
+        c, nc = getstats()
+        self.assertEqual(t, 2*N) # instance object & its dict
+        self.assertEqual(c - oldc, 2*N)
+        self.assertEqual(nc - oldnc, 0)
+
+        # But Z() is not actually collected.
+        oldc, oldnc = c, nc
+        Z()
+        # Nothing is collected - Z() is merely resurrected.
+        t = gc.collect()
+        c, nc = getstats()
+        #self.assertEqual(t, 2)  # before
+        self.assertEqual(t, 0)  # after
+        #self.assertEqual(c - oldc, 2)   # before
+        self.assertEqual(c - oldc, 0)   # after
+        self.assertEqual(nc - oldnc, 0)
+
+        # Unfortunately, a Z() prevents _anything_ from being collected.
+        # It should be possible to collect the A instances anyway, but
+        # that will require non-trivial code changes.
+        oldc, oldnc = c, nc
+        for i in range(N):
+            A()
+        Z()
+        # Z() prevents anything from being collected.
+        t = gc.collect()
+        c, nc = getstats()
+        #self.assertEqual(t, 2*N + 2)  # before
+        self.assertEqual(t, 0)  # after
+        #self.assertEqual(c - oldc, 2*N + 2)   # before
+        self.assertEqual(c - oldc, 0)   # after
+        self.assertEqual(nc - oldnc, 0)
+
+        # But the A() trash is reclaimed on the next run.
+        oldc, oldnc = c, nc
+        t = gc.collect()
+        c, nc = getstats()
+        self.assertEqual(t, 2*N)
+        self.assertEqual(c - oldc, 2*N)
+        self.assertEqual(nc - oldnc, 0)
+
+        gc.enable()
 
 class GCCallbackTests(unittest.TestCase):
     def setUp(self):
diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-10-09-16-50-52.bpo-38379.oz5qZx.rst b/Misc/NEWS.d/next/Core and Builtins/2019-10-09-16-50-52.bpo-38379.oz5qZx.rst
new file mode 100644
index 0000000000000..82dcb525dd49d
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2019-10-09-16-50-52.bpo-38379.oz5qZx.rst	
@@ -0,0 +1 @@
+When cyclic garbage collection (gc) runs finalizers that resurrect unreachable objects, the current gc run ends, without collecting any cyclic trash.  However, the statistics reported by ``collect()`` and ``get_stats()`` claimed that all cyclic trash found was collected, and that the resurrected objects were collected.   Changed the stats to report that none were collected.
diff --git a/Modules/gcmodule.c b/Modules/gcmodule.c
index 8bdbafef167e0..8d7d67cb6701d 100644
--- a/Modules/gcmodule.c
+++ b/Modules/gcmodule.c
@@ -1095,12 +1095,9 @@ collect(struct _gc_runtime_state *state, int generation,
     validate_list(&finalizers, 0);
     validate_list(&unreachable, PREV_MASK_COLLECTING);
 
-    /* Collect statistics on collectable objects found and print
-     * debugging information.
-     */
-    for (gc = GC_NEXT(&unreachable); gc != &unreachable; gc = GC_NEXT(gc)) {
-        m++;
-        if (state->debug & DEBUG_COLLECTABLE) {
+    /* Print debugging information. */
+    if (state->debug & DEBUG_COLLECTABLE) {
+        for (gc = GC_NEXT(&unreachable); gc != &unreachable; gc = GC_NEXT(gc)) {
             debug_cycle("collectable", FROM_GC(gc));
         }
     }
@@ -1122,6 +1119,7 @@ collect(struct _gc_runtime_state *state, int generation,
          * the reference cycles to be broken.  It may also cause some objects
          * in finalizers to be freed.
          */
+        m += gc_list_size(&unreachable);
         delete_garbage(state, &unreachable, old);
     }
 


From webhook-mailer at python.org  Wed Oct  9 17:42:58 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Wed, 09 Oct 2019 21:42:58 -0000
Subject: [Python-checkins] [3.7] bpo-38379: don't claim objects are
 collected when they aren't (GH-16658) (GH-16685)
Message-ID: <mailman.77.1570657379.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/a081e931505f190b5ccdff9a781e59b3f13fcc31
commit: a081e931505f190b5ccdff9a781e59b3f13fcc31
branch: 3.7
author: Pablo Galindo <Pablogsal at gmail.com>
committer: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
date: 2019-10-09T14:42:54-07:00
summary:

[3.7] bpo-38379:  don't claim objects are collected when they aren't (GH-16658) (GH-16685)



* [bpo-38379](https://bugs.python.org/issue38379):  when a finalizer resurrects an object,
nothing is actually collected in this run of gc.
Change the stats to relect that truth..
(cherry picked from commit ecbf35f9335b0420cb8adfda6f299d6747a16515)

Co-authored-by: Tim Peters <tim.peters at gmail.com>





https://bugs.python.org/issue38379



Automerge-Triggered-By: @pablogsal

files:
A Misc/NEWS.d/next/Core and Builtins/2019-10-09-16-50-52.bpo-38379.oz5qZx.rst
M Lib/test/test_gc.py
M Modules/gcmodule.c

diff --git a/Lib/test/test_gc.py b/Lib/test/test_gc.py
index 8d806db3ba57a..a2fa8bbace4c4 100644
--- a/Lib/test/test_gc.py
+++ b/Lib/test/test_gc.py
@@ -755,6 +755,77 @@ def test_freeze(self):
         gc.unfreeze()
         self.assertEqual(gc.get_freeze_count(), 0)
 
+    def test_38379(self):
+        # When a finalizer resurrects objects, stats were reporting them as
+        # having been collected.  This affected both collect()'s return
+        # value and the dicts returned by get_stats().
+        N = 100
+
+        class A:  # simple self-loop
+            def __init__(self):
+                self.me = self
+
+        class Z(A):  # resurrecting __del__
+            def __del__(self):
+                zs.append(self)
+
+        zs = []
+
+        def getstats():
+            d = gc.get_stats()[-1]
+            return d['collected'], d['uncollectable']
+
+        gc.collect()
+        gc.disable()
+
+        # No problems if just collecting A() instances.
+        oldc, oldnc = getstats()
+        for i in range(N):
+            A()
+        t = gc.collect()
+        c, nc = getstats()
+        self.assertEqual(t, 2*N) # instance object & its dict
+        self.assertEqual(c - oldc, 2*N)
+        self.assertEqual(nc - oldnc, 0)
+
+        # But Z() is not actually collected.
+        oldc, oldnc = c, nc
+        Z()
+        # Nothing is collected - Z() is merely resurrected.
+        t = gc.collect()
+        c, nc = getstats()
+        #self.assertEqual(t, 2)  # before
+        self.assertEqual(t, 0)  # after
+        #self.assertEqual(c - oldc, 2)   # before
+        self.assertEqual(c - oldc, 0)   # after
+        self.assertEqual(nc - oldnc, 0)
+
+        # Unfortunately, a Z() prevents _anything_ from being collected.
+        # It should be possible to collect the A instances anyway, but
+        # that will require non-trivial code changes.
+        oldc, oldnc = c, nc
+        for i in range(N):
+            A()
+        Z()
+        # Z() prevents anything from being collected.
+        t = gc.collect()
+        c, nc = getstats()
+        #self.assertEqual(t, 2*N + 2)  # before
+        self.assertEqual(t, 0)  # after
+        #self.assertEqual(c - oldc, 2*N + 2)   # before
+        self.assertEqual(c - oldc, 0)   # after
+        self.assertEqual(nc - oldnc, 0)
+
+        # But the A() trash is reclaimed on the next run.
+        oldc, oldnc = c, nc
+        t = gc.collect()
+        c, nc = getstats()
+        self.assertEqual(t, 2*N)
+        self.assertEqual(c - oldc, 2*N)
+        self.assertEqual(nc - oldnc, 0)
+
+        gc.enable()
+
 
 class GCCallbackTests(unittest.TestCase):
     def setUp(self):
diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-10-09-16-50-52.bpo-38379.oz5qZx.rst b/Misc/NEWS.d/next/Core and Builtins/2019-10-09-16-50-52.bpo-38379.oz5qZx.rst
new file mode 100644
index 0000000000000..82dcb525dd49d
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2019-10-09-16-50-52.bpo-38379.oz5qZx.rst	
@@ -0,0 +1 @@
+When cyclic garbage collection (gc) runs finalizers that resurrect unreachable objects, the current gc run ends, without collecting any cyclic trash.  However, the statistics reported by ``collect()`` and ``get_stats()`` claimed that all cyclic trash found was collected, and that the resurrected objects were collected.   Changed the stats to report that none were collected.
diff --git a/Modules/gcmodule.c b/Modules/gcmodule.c
index 4d701cb72e8c5..32586031c039c 100644
--- a/Modules/gcmodule.c
+++ b/Modules/gcmodule.c
@@ -887,13 +887,9 @@ collect(int generation, Py_ssize_t *n_collected, Py_ssize_t *n_uncollectable,
      */
     move_legacy_finalizer_reachable(&finalizers);
 
-    /* Collect statistics on collectable objects found and print
-     * debugging information.
-     */
-    for (gc = unreachable.gc.gc_next; gc != &unreachable;
-                    gc = gc->gc.gc_next) {
-        m++;
-        if (_PyRuntime.gc.debug & DEBUG_COLLECTABLE) {
+    /* Print debugging information. */
+    if (_PyRuntime.gc.debug & DEBUG_COLLECTABLE) {
+        for (gc = unreachable.gc.gc_next; gc != &unreachable; gc = gc->gc.gc_next) {
             debug_cycle("collectable", FROM_GC(gc));
         }
     }
@@ -913,6 +909,7 @@ collect(int generation, Py_ssize_t *n_collected, Py_ssize_t *n_uncollectable,
          * the reference cycles to be broken.  It may also cause some objects
          * in finalizers to be freed.
          */
+        m += gc_list_size(&unreachable);
         delete_garbage(&unreachable, old);
     }
 


From webhook-mailer at python.org  Wed Oct  9 17:55:43 2019
From: webhook-mailer at python.org (Zachary Ware)
Date: Wed, 09 Oct 2019 21:55:43 -0000
Subject: [Python-checkins] bpo-38409: Fix grammar in str.strip() docstring
 (GH-16682) (GH-16684)
Message-ID: <mailman.78.1570658144.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/0baa6b3c7d9ef1d96f10939c40f8ff95aba9155b
commit: 0baa6b3c7d9ef1d96f10939c40f8ff95aba9155b
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Zachary Ware <zachary.ware at gmail.com>
date: 2019-10-09T16:55:39-05:00
summary:

bpo-38409: Fix grammar in str.strip() docstring (GH-16682) (GH-16684)

(cherry picked from commit 09895c27cd8ff60563a794016e8c099bc897cc74)

files:
M Objects/clinic/unicodeobject.c.h
M Objects/unicodeobject.c

diff --git a/Objects/clinic/unicodeobject.c.h b/Objects/clinic/unicodeobject.c.h
index c7d6edb5fd488..0d134064bab09 100644
--- a/Objects/clinic/unicodeobject.c.h
+++ b/Objects/clinic/unicodeobject.c.h
@@ -582,7 +582,7 @@ PyDoc_STRVAR(unicode_strip__doc__,
 "strip($self, chars=None, /)\n"
 "--\n"
 "\n"
-"Return a copy of the string with leading and trailing whitespace remove.\n"
+"Return a copy of the string with leading and trailing whitespace removed.\n"
 "\n"
 "If chars is given and not None, remove characters in chars instead.");
 
@@ -1232,4 +1232,4 @@ unicode_sizeof(PyObject *self, PyObject *Py_UNUSED(ignored))
 {
     return unicode_sizeof_impl(self);
 }
-/*[clinic end generated code: output=5e15747f78f18329 input=a9049054013a1b77]*/
+/*[clinic end generated code: output=e4ed33400979c7e8 input=a9049054013a1b77]*/
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
index 1466e8a6e72b7..9eef05a621676 100644
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -12476,14 +12476,14 @@ str.strip as unicode_strip
     chars: object = None
     /
 
-Return a copy of the string with leading and trailing whitespace remove.
+Return a copy of the string with leading and trailing whitespace removed.
 
 If chars is given and not None, remove characters in chars instead.
 [clinic start generated code]*/
 
 static PyObject *
 unicode_strip_impl(PyObject *self, PyObject *chars)
-/*[clinic end generated code: output=ca19018454345d57 input=eefe24a1059c352b]*/
+/*[clinic end generated code: output=ca19018454345d57 input=385289c6f423b954]*/
 {
     return do_argstrip(self, BOTHSTRIP, chars);
 }


From webhook-mailer at python.org  Wed Oct  9 17:56:11 2019
From: webhook-mailer at python.org (Zachary Ware)
Date: Wed, 09 Oct 2019 21:56:11 -0000
Subject: [Python-checkins] [3.7] bpo-38409: Fix grammar in str.strip()
 docstring (GH-16682) (GH-16686)
Message-ID: <mailman.79.1570658172.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/a774ac6637e8f155a28b7733a0a93d4a680584d4
commit: a774ac6637e8f155a28b7733a0a93d4a680584d4
branch: 3.7
author: Zachary Ware <zachary.ware at gmail.com>
committer: GitHub <noreply at github.com>
date: 2019-10-09T16:56:07-05:00
summary:

[3.7] bpo-38409: Fix grammar in str.strip() docstring (GH-16682) (GH-16686)

(cherry picked from commit 09895c27cd8ff60563a794016e8c099bc897cc74)

files:
M Objects/clinic/unicodeobject.c.h
M Objects/unicodeobject.c

diff --git a/Objects/clinic/unicodeobject.c.h b/Objects/clinic/unicodeobject.c.h
index 8072516a8a36d..2b744b622c85a 100644
--- a/Objects/clinic/unicodeobject.c.h
+++ b/Objects/clinic/unicodeobject.c.h
@@ -484,7 +484,7 @@ PyDoc_STRVAR(unicode_strip__doc__,
 "strip($self, chars=None, /)\n"
 "--\n"
 "\n"
-"Return a copy of the string with leading and trailing whitespace remove.\n"
+"Return a copy of the string with leading and trailing whitespace removed.\n"
 "\n"
 "If chars is given and not None, remove characters in chars instead.");
 
@@ -951,4 +951,4 @@ unicode_sizeof(PyObject *self, PyObject *Py_UNUSED(ignored))
 {
     return unicode_sizeof_impl(self);
 }
-/*[clinic end generated code: output=561c88c912b8fe3b input=a9049054013a1b77]*/
+/*[clinic end generated code: output=49bf50c732b9b53b input=a9049054013a1b77]*/
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
index 9e0731bf2ba2f..d7e5a9d9cc98f 100644
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -12474,14 +12474,14 @@ str.strip as unicode_strip
     chars: object = None
     /
 
-Return a copy of the string with leading and trailing whitespace remove.
+Return a copy of the string with leading and trailing whitespace removed.
 
 If chars is given and not None, remove characters in chars instead.
 [clinic start generated code]*/
 
 static PyObject *
 unicode_strip_impl(PyObject *self, PyObject *chars)
-/*[clinic end generated code: output=ca19018454345d57 input=eefe24a1059c352b]*/
+/*[clinic end generated code: output=ca19018454345d57 input=385289c6f423b954]*/
 {
     return do_argstrip(self, BOTHSTRIP, chars);
 }


From python-checkins at python.org  Wed Oct  9 22:03:27 2019
From: python-checkins at python.org (python-checkins at python.org)
Date: Thu, 10 Oct 2019 02:03:27 +0000
Subject: [Python-checkins] =?utf-8?q?Mauvaise_nouvelle_=28=C3=A0_lire=29?=
Message-ID: <bf214f351e2ba4edd9584d65d88a4727@138.68.21.12>

 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: Mauvaise-nouvelle.pdf
Type: application/octet-stream
Size: 40643 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-checkins/attachments/20191010/1db52eea/attachment-0001.obj>

From webhook-mailer at python.org  Thu Oct 10 03:32:20 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Thu, 10 Oct 2019 07:32:20 -0000
Subject: [Python-checkins] bpo-38392: Only declare visit_validate() if
 Py_DEBUG is defined (GH-16689)
Message-ID: <mailman.80.1570692741.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/a5447735c334a041ee2ffdeb5c7e13d7d4502ea2
commit: a5447735c334a041ee2ffdeb5c7e13d7d4502ea2
branch: master
author: Victor Stinner <vstinner at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-10T09:32:13+02:00
summary:

bpo-38392: Only declare visit_validate() if Py_DEBUG is defined (GH-16689)

bpo-38392, bpo-38426: Fix a compiler warning in gcmodule.c.

Fix also a typo in PYMEM_DEADBYTE macro comment.

files:
M Include/internal/pycore_pymem.h
M Modules/gcmodule.c

diff --git a/Include/internal/pycore_pymem.h b/Include/internal/pycore_pymem.h
index d61ff6ee4b77d..535f99e8cc52e 100644
--- a/Include/internal/pycore_pymem.h
+++ b/Include/internal/pycore_pymem.h
@@ -162,7 +162,7 @@ PyAPI_FUNC(int) _PyMem_SetDefaultAllocator(
    - PYMEM_DEADBYTE dead (newly freed) memory
    - PYMEM_FORBIDDENBYTE: untouchable bytes at each end of a block
 
-   Byte patterns 0xCB, 0xBB and 0xFB have been replaced with 0xCD, 0xDD and
+   Byte patterns 0xCB, 0xDB and 0xFB have been replaced with 0xCD, 0xDD and
    0xFD to use the same values than Windows CRT debug malloc() and free().
    If modified, _PyMem_IsPtrFreed() should be updated as well. */
 #define PYMEM_CLEANBYTE      0xCD
diff --git a/Modules/gcmodule.c b/Modules/gcmodule.c
index 766f8e0c67fa8..b1e802e33e498 100644
--- a/Modules/gcmodule.c
+++ b/Modules/gcmodule.c
@@ -1921,6 +1921,8 @@ _PyGC_Dump(PyGC_Head *g)
     _PyObject_Dump(FROM_GC(g));
 }
 
+
+#ifdef Py_DEBUG
 static int
 visit_validate(PyObject *op, void *parent_raw)
 {
@@ -1931,6 +1933,7 @@ visit_validate(PyObject *op, void *parent_raw)
     }
     return 0;
 }
+#endif
 
 
 /* extension modules might be compiled with GC support so these


From webhook-mailer at python.org  Thu Oct 10 03:34:50 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Thu, 10 Oct 2019 07:34:50 -0000
Subject: [Python-checkins] bpo-38109: Add missing constants to Lib/stat.py
 (GH-16665)
Message-ID: <mailman.81.1570692890.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/7bb14316b8ceddb813f31040a299af94a57ab339
commit: 7bb14316b8ceddb813f31040a299af94a57ab339
branch: master
author: Ronan Lamy <ronan.lamy at gmail.com>
committer: Victor Stinner <vstinner at python.org>
date: 2019-10-10T09:34:46+02:00
summary:

bpo-38109: Add missing constants to Lib/stat.py (GH-16665)

Add missing stat.S_IFDOOR, stat.S_IFPORT, stat.S_IFWHT,
stat.S_ISDOOR, stat.S_ISPORT, and stat.S_ISWHT values to
the Python implementation of the stat module.

files:
A Misc/NEWS.d/next/Library/2019-10-10-00-25-28.bpo-38109.9w-IGF.rst
M Lib/stat.py
M Lib/test/test_stat.py

diff --git a/Lib/stat.py b/Lib/stat.py
index a9c678ec03c9c..fc024db3f4fbe 100644
--- a/Lib/stat.py
+++ b/Lib/stat.py
@@ -40,6 +40,10 @@ def S_IFMT(mode):
 S_IFIFO  = 0o010000  # fifo (named pipe)
 S_IFLNK  = 0o120000  # symbolic link
 S_IFSOCK = 0o140000  # socket file
+# Fallbacks for uncommon platform-specific constants
+S_IFDOOR = 0
+S_IFPORT = 0
+S_IFWHT = 0
 
 # Functions to test for each file type
 
@@ -71,6 +75,18 @@ def S_ISSOCK(mode):
     """Return True if mode is from a socket."""
     return S_IFMT(mode) == S_IFSOCK
 
+def S_ISDOOR(mode):
+    """Return True if mode is from a door."""
+    return False
+
+def S_ISPORT(mode):
+    """Return True if mode is from an event port."""
+    return False
+
+def S_ISWHT(mode):
+    """Return True if mode is from a whiteout."""
+    return False
+
 # Names for permission bits
 
 S_ISUID = 0o4000  # set UID bit
diff --git a/Lib/test/test_stat.py b/Lib/test/test_stat.py
index 17443bed0738b..be01db2ed5574 100644
--- a/Lib/test/test_stat.py
+++ b/Lib/test/test_stat.py
@@ -16,10 +16,10 @@ class TestFilemode:
                   'UF_IMMUTABLE', 'UF_NODUMP', 'UF_NOUNLINK', 'UF_OPAQUE'}
 
     formats = {'S_IFBLK', 'S_IFCHR', 'S_IFDIR', 'S_IFIFO', 'S_IFLNK',
-               'S_IFREG', 'S_IFSOCK'}
+               'S_IFREG', 'S_IFSOCK', 'S_IFDOOR', 'S_IFPORT', 'S_IFWHT'}
 
     format_funcs = {'S_ISBLK', 'S_ISCHR', 'S_ISDIR', 'S_ISFIFO', 'S_ISLNK',
-                    'S_ISREG', 'S_ISSOCK'}
+                    'S_ISREG', 'S_ISSOCK', 'S_ISDOOR', 'S_ISPORT', 'S_ISWHT'}
 
     stat_struct = {
         'ST_MODE': 0,
@@ -231,10 +231,6 @@ def test_file_attribute_constants(self):
 class TestFilemodeCStat(TestFilemode, unittest.TestCase):
     statmod = c_stat
 
-    formats = TestFilemode.formats | {'S_IFDOOR', 'S_IFPORT', 'S_IFWHT'}
-    format_funcs = TestFilemode.format_funcs | {'S_ISDOOR', 'S_ISPORT',
-                                                'S_ISWHT'}
-
 
 class TestFilemodePyStat(TestFilemode, unittest.TestCase):
     statmod = py_stat
diff --git a/Misc/NEWS.d/next/Library/2019-10-10-00-25-28.bpo-38109.9w-IGF.rst b/Misc/NEWS.d/next/Library/2019-10-10-00-25-28.bpo-38109.9w-IGF.rst
new file mode 100644
index 0000000000000..3f4484dc644fa
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-10-10-00-25-28.bpo-38109.9w-IGF.rst
@@ -0,0 +1,3 @@
+Add missing :data:`stat.S_IFDOOR`, :data:`stat.S_IFPORT`, :data:`stat.S_IFWHT`,
+:func:`stat.S_ISDOOR`, :func:`stat.S_ISPORT`, and :func:`stat.S_ISWHT` values to
+the Python implementation of :mod:`stat`.
\ No newline at end of file


From webhook-mailer at python.org  Thu Oct 10 03:41:30 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Thu, 10 Oct 2019 07:41:30 -0000
Subject: [Python-checkins] =?utf-8?b?YnBvLTM4NDI1OiBGaXgg4oCYcmVz4oCZIG1h?=
 =?utf-8?q?y_be_used_uninitialized_warning_=28GH-16688=29?=
Message-ID: <mailman.82.1570693291.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/a05fcd3c7adf6e3a0944da8cf80a3346882e9b3b
commit: a05fcd3c7adf6e3a0944da8cf80a3346882e9b3b
branch: master
author: Dong-hee Na <donghee.na92 at gmail.com>
committer: Victor Stinner <vstinner at python.org>
date: 2019-10-10T09:41:26+02:00
summary:

bpo-38425: Fix ?res? may be used uninitialized warning (GH-16688)

files:
M Parser/asdl_c.py
M Python/Python-ast.c

diff --git a/Parser/asdl_c.py b/Parser/asdl_c.py
index 3a9d4e6b4b923..3a821cc9265c9 100755
--- a/Parser/asdl_c.py
+++ b/Parser/asdl_c.py
@@ -1197,7 +1197,6 @@ class PartingShots(StaticVisitor):
 /* mode is 0 for "exec", 1 for "eval" and 2 for "single" input */
 mod_ty PyAST_obj2mod(PyObject* ast, PyArena* arena, int mode)
 {
-    mod_ty res;
     PyObject *req_type[3];
     const char * const req_name[] = {"Module", "Expression", "Interactive"};
     int isinstance;
@@ -1223,6 +1222,8 @@ class PartingShots(StaticVisitor):
                      req_name[mode], _PyType_Name(Py_TYPE(ast)));
         return NULL;
     }
+
+    mod_ty res = NULL;
     if (obj2ast_mod(ast, &res, arena) != 0)
         return NULL;
     else
diff --git a/Python/Python-ast.c b/Python/Python-ast.c
index 558d9ebea5c72..e2c703d815552 100644
--- a/Python/Python-ast.c
+++ b/Python/Python-ast.c
@@ -10250,7 +10250,6 @@ PyObject* PyAST_mod2obj(mod_ty t)
 /* mode is 0 for "exec", 1 for "eval" and 2 for "single" input */
 mod_ty PyAST_obj2mod(PyObject* ast, PyArena* arena, int mode)
 {
-    mod_ty res;
     PyObject *req_type[3];
     const char * const req_name[] = {"Module", "Expression", "Interactive"};
     int isinstance;
@@ -10276,6 +10275,8 @@ mod_ty PyAST_obj2mod(PyObject* ast, PyArena* arena, int mode)
                      req_name[mode], _PyType_Name(Py_TYPE(ast)));
         return NULL;
     }
+
+    mod_ty res = NULL;
     if (obj2ast_mod(ast, &res, arena) != 0)
         return NULL;
     else


From webhook-mailer at python.org  Thu Oct 10 04:00:30 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Thu, 10 Oct 2019 08:00:30 -0000
Subject: [Python-checkins] =?utf-8?b?YnBvLTM4NDI1OiBGaXgg4oCYcmVz4oCZIG1h?=
 =?utf-8?q?y_be_used_uninitialized_warning_=28GH-16688=29?=
Message-ID: <mailman.83.1570694432.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/6b6935e563562c427d5bb1b2864d6a2fed0e74fa
commit: 6b6935e563562c427d5bb1b2864d6a2fed0e74fa
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-10T01:00:19-07:00
summary:

bpo-38425: Fix ?res? may be used uninitialized warning (GH-16688)

(cherry picked from commit a05fcd3c7adf6e3a0944da8cf80a3346882e9b3b)

Co-authored-by: Dong-hee Na <donghee.na92 at gmail.com>

files:
M Parser/asdl_c.py
M Python/Python-ast.c

diff --git a/Parser/asdl_c.py b/Parser/asdl_c.py
index f4fa271b6595b..574fcb0e2faaf 100644
--- a/Parser/asdl_c.py
+++ b/Parser/asdl_c.py
@@ -1191,7 +1191,6 @@ class PartingShots(StaticVisitor):
 /* mode is 0 for "exec", 1 for "eval" and 2 for "single" input */
 mod_ty PyAST_obj2mod(PyObject* ast, PyArena* arena, int mode)
 {
-    mod_ty res;
     PyObject *req_type[3];
     char *req_name[] = {"Module", "Expression", "Interactive"};
     int isinstance;
@@ -1217,6 +1216,8 @@ class PartingShots(StaticVisitor):
                      req_name[mode], Py_TYPE(ast)->tp_name);
         return NULL;
     }
+
+    mod_ty res = NULL;
     if (obj2ast_mod(ast, &res, arena) != 0)
         return NULL;
     else
diff --git a/Python/Python-ast.c b/Python/Python-ast.c
index fd96964a1e77e..f73f035845f8c 100644
--- a/Python/Python-ast.c
+++ b/Python/Python-ast.c
@@ -8991,7 +8991,6 @@ PyObject* PyAST_mod2obj(mod_ty t)
 /* mode is 0 for "exec", 1 for "eval" and 2 for "single" input */
 mod_ty PyAST_obj2mod(PyObject* ast, PyArena* arena, int mode)
 {
-    mod_ty res;
     PyObject *req_type[3];
     char *req_name[] = {"Module", "Expression", "Interactive"};
     int isinstance;
@@ -9017,6 +9016,8 @@ mod_ty PyAST_obj2mod(PyObject* ast, PyArena* arena, int mode)
                      req_name[mode], Py_TYPE(ast)->tp_name);
         return NULL;
     }
+
+    mod_ty res = NULL;
     if (obj2ast_mod(ast, &res, arena) != 0)
         return NULL;
     else


From webhook-mailer at python.org  Thu Oct 10 04:00:45 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Thu, 10 Oct 2019 08:00:45 -0000
Subject: [Python-checkins] =?utf-8?b?YnBvLTM4NDI1OiBGaXgg4oCYcmVz4oCZIG1h?=
 =?utf-8?q?y_be_used_uninitialized_warning_=28GH-16688=29?=
Message-ID: <mailman.84.1570694446.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/d27a9b1695a1b46c81b3a83a7a35034791265d89
commit: d27a9b1695a1b46c81b3a83a7a35034791265d89
branch: 3.7
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-10T01:00:41-07:00
summary:

bpo-38425: Fix ?res? may be used uninitialized warning (GH-16688)

(cherry picked from commit a05fcd3c7adf6e3a0944da8cf80a3346882e9b3b)

Co-authored-by: Dong-hee Na <donghee.na92 at gmail.com>

files:
M Parser/asdl_c.py
M Python/Python-ast.c

diff --git a/Parser/asdl_c.py b/Parser/asdl_c.py
index 44e3d40c61553..20a71866e3577 100644
--- a/Parser/asdl_c.py
+++ b/Parser/asdl_c.py
@@ -1209,7 +1209,6 @@ class PartingShots(StaticVisitor):
 /* mode is 0 for "exec", 1 for "eval" and 2 for "single" input */
 mod_ty PyAST_obj2mod(PyObject* ast, PyArena* arena, int mode)
 {
-    mod_ty res;
     PyObject *req_type[3];
     char *req_name[] = {"Module", "Expression", "Interactive"};
     int isinstance;
@@ -1231,6 +1230,8 @@ class PartingShots(StaticVisitor):
                      req_name[mode], Py_TYPE(ast)->tp_name);
         return NULL;
     }
+
+    mod_ty res = NULL;
     if (obj2ast_mod(ast, &res, arena) != 0)
         return NULL;
     else
diff --git a/Python/Python-ast.c b/Python/Python-ast.c
index 6a2f28e0e712d..74fae33e5f4e6 100644
--- a/Python/Python-ast.c
+++ b/Python/Python-ast.c
@@ -8365,7 +8365,6 @@ PyObject* PyAST_mod2obj(mod_ty t)
 /* mode is 0 for "exec", 1 for "eval" and 2 for "single" input */
 mod_ty PyAST_obj2mod(PyObject* ast, PyArena* arena, int mode)
 {
-    mod_ty res;
     PyObject *req_type[3];
     char *req_name[] = {"Module", "Expression", "Interactive"};
     int isinstance;
@@ -8387,6 +8386,8 @@ mod_ty PyAST_obj2mod(PyObject* ast, PyArena* arena, int mode)
                      req_name[mode], Py_TYPE(ast)->tp_name);
         return NULL;
     }
+
+    mod_ty res = NULL;
     if (obj2ast_mod(ast, &res, arena) != 0)
         return NULL;
     else


From webhook-mailer at python.org  Thu Oct 10 07:11:42 2019
From: webhook-mailer at python.org (Ivan Levkivskyi)
Date: Thu, 10 Oct 2019 11:11:42 -0000
Subject: [Python-checkins] bpo-32996: Documentation fix-up. (GH-16646)
Message-ID: <mailman.85.1570705903.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/d47f0dd2e85ce032aebfedbde18cdb2e728fa79f
commit: d47f0dd2e85ce032aebfedbde18cdb2e728fa79f
branch: master
author: M. Eric Irrgang <mei2n at virginia.edu>
committer: Ivan Levkivskyi <levkivskyi at gmail.com>
date: 2019-10-10T12:11:33+01:00
summary:

bpo-32996: Documentation fix-up. (GH-16646)

PR #4906 changed the typing.Generic class hierarchy, leaving an
outdated comment in the library reference. User-defined Generic ABCs now
must get a abc.ABCMeta metaclass from something other than typing.Generic
inheritance.

files:
M Doc/library/typing.rst

diff --git a/Doc/library/typing.rst b/Doc/library/typing.rst
index f4d1b2404328d..642f8c653eeb5 100644
--- a/Doc/library/typing.rst
+++ b/Doc/library/typing.rst
@@ -307,9 +307,10 @@ User defined generic type aliases are also supported. Examples::
    def inproduct(v: Vec[T]) -> T: # Same as Iterable[Tuple[T, T]]
        return sum(x*y for x, y in v)
 
-The metaclass used by :class:`Generic` is a subclass of :class:`abc.ABCMeta`.
-A generic class can be an ABC by including abstract methods or properties,
-and generic classes can also have ABCs as base classes without a metaclass
+.. versionchanged:: 3.7
+    :class:`Generic` no longer has a custom metaclass.
+
+A user-defined generic class can have ABCs as base classes without a metaclass
 conflict. Generic metaclasses are not supported. The outcome of parameterizing
 generics is cached, and most types in the typing module are hashable and
 comparable for equality.


From webhook-mailer at python.org  Thu Oct 10 07:17:08 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Thu, 10 Oct 2019 11:17:08 -0000
Subject: [Python-checkins] bpo-32996: Documentation fix-up. (GH-16646)
Message-ID: <mailman.86.1570706230.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/ca012360b103f2ab1acfa506aed032b0ed9b105c
commit: ca012360b103f2ab1acfa506aed032b0ed9b105c
branch: 3.7
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-10T04:17:05-07:00
summary:

bpo-32996: Documentation fix-up. (GH-16646)


PR GH-4906 changed the typing.Generic class hierarchy, leaving an
outdated comment in the library reference. User-defined Generic ABCs now
must get a abc.ABCMeta metaclass from something other than typing.Generic
inheritance.
(cherry picked from commit d47f0dd2e85ce032aebfedbde18cdb2e728fa79f)

Co-authored-by: M. Eric Irrgang <mei2n at virginia.edu>

files:
M Doc/library/typing.rst

diff --git a/Doc/library/typing.rst b/Doc/library/typing.rst
index 627c1f77c157d..b42f33ccc0d35 100644
--- a/Doc/library/typing.rst
+++ b/Doc/library/typing.rst
@@ -306,9 +306,10 @@ User defined generic type aliases are also supported. Examples::
    def inproduct(v: Vec[T]) -> T: # Same as Iterable[Tuple[T, T]]
        return sum(x*y for x, y in v)
 
-The metaclass used by :class:`Generic` is a subclass of :class:`abc.ABCMeta`.
-A generic class can be an ABC by including abstract methods or properties,
-and generic classes can also have ABCs as base classes without a metaclass
+.. versionchanged:: 3.7
+    :class:`Generic` no longer has a custom metaclass.
+
+A user-defined generic class can have ABCs as base classes without a metaclass
 conflict. Generic metaclasses are not supported. The outcome of parameterizing
 generics is cached, and most types in the typing module are hashable and
 comparable for equality.


From webhook-mailer at python.org  Thu Oct 10 07:19:07 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Thu, 10 Oct 2019 11:19:07 -0000
Subject: [Python-checkins] bpo-32996: Documentation fix-up. (GH-16646)
Message-ID: <mailman.87.1570706347.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/baf8a9b03a56ce1ee2284763b267ff56ab4507fe
commit: baf8a9b03a56ce1ee2284763b267ff56ab4507fe
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-10T04:19:03-07:00
summary:

bpo-32996: Documentation fix-up. (GH-16646)


PR GH-4906 changed the typing.Generic class hierarchy, leaving an
outdated comment in the library reference. User-defined Generic ABCs now
must get a abc.ABCMeta metaclass from something other than typing.Generic
inheritance.
(cherry picked from commit d47f0dd2e85ce032aebfedbde18cdb2e728fa79f)

Co-authored-by: M. Eric Irrgang <mei2n at virginia.edu>

files:
M Doc/library/typing.rst

diff --git a/Doc/library/typing.rst b/Doc/library/typing.rst
index f4d1b2404328d..642f8c653eeb5 100644
--- a/Doc/library/typing.rst
+++ b/Doc/library/typing.rst
@@ -307,9 +307,10 @@ User defined generic type aliases are also supported. Examples::
    def inproduct(v: Vec[T]) -> T: # Same as Iterable[Tuple[T, T]]
        return sum(x*y for x, y in v)
 
-The metaclass used by :class:`Generic` is a subclass of :class:`abc.ABCMeta`.
-A generic class can be an ABC by including abstract methods or properties,
-and generic classes can also have ABCs as base classes without a metaclass
+.. versionchanged:: 3.7
+    :class:`Generic` no longer has a custom metaclass.
+
+A user-defined generic class can have ABCs as base classes without a metaclass
 conflict. Generic metaclasses are not supported. The outcome of parameterizing
 generics is cached, and most types in the typing module are hashable and
 comparable for equality.


From webhook-mailer at python.org  Thu Oct 10 09:42:36 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Thu, 10 Oct 2019 13:42:36 -0000
Subject: [Python-checkins] bpo-38353: Fix typos in
 calculate_argv0_path_framework() (GH-16695)
Message-ID: <mailman.88.1570714958.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/b6e0fc74265db96fdd0e0b8151c768c583ad80f0
commit: b6e0fc74265db96fdd0e0b8151c768c583ad80f0
branch: master
author: Victor Stinner <vstinner at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-10T15:42:30+02:00
summary:

bpo-38353: Fix typos in calculate_argv0_path_framework() (GH-16695)

bpo-38353, bpo-38429: Fix typos introduced by commit
c02b41b1fb115c87693530ea6a480b2e15460424 in
calculate_argv0_path_framework() of getpath.c.

files:
M Modules/getpath.c

diff --git a/Modules/getpath.c b/Modules/getpath.c
index 057a929019755..8632d1fe6d80f 100644
--- a/Modules/getpath.c
+++ b/Modules/getpath.c
@@ -1126,7 +1126,7 @@ calculate_argv0_path_framework(PyCalculatePath *calculate, _PyPathConfig *pathco
     }
 
     reduce(parent);
-    wchar_t *lib_python = joinpath2(path, calculate->lib_python);
+    wchar_t *lib_python = joinpath2(parent, calculate->lib_python);
     PyMem_RawFree(parent);
 
     if (lib_python == NULL) {
@@ -1144,7 +1144,7 @@ calculate_argv0_path_framework(PyCalculatePath *calculate, _PyPathConfig *pathco
     if (!module) {
         /* We are in the build directory so use the name of the
            executable - we know that the absolute path is passed */
-        PyMem_RawFree(*calculate->argv0_path);
+        PyMem_RawFree(calculate->argv0_path);
         calculate->argv0_path = _PyMem_RawWcsdup(pathconfig->program_full_path);
         if (calculate->argv0_path == NULL) {
             status = _PyStatus_NO_MEMORY();
@@ -1156,8 +1156,8 @@ calculate_argv0_path_framework(PyCalculatePath *calculate, _PyPathConfig *pathco
     }
 
     /* Use the location of the library as argv0_path */
-    PyMem_RawFree(*calculate->argv0_path);
-    calculate->argv0_path = wbuf
+    PyMem_RawFree(calculate->argv0_path);
+    calculate->argv0_path = wbuf;
     return _PyStatus_OK();
 
 done:


From webhook-mailer at python.org  Thu Oct 10 09:50:37 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Thu, 10 Oct 2019 13:50:37 -0000
Subject: [Python-checkins] bpo-38109: Add missing constants to Lib/stat.py
 (GH-16665) (GH-16690)
Message-ID: <mailman.89.1570715438.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/8ab11c433a81b6c7ef84a66e36fbe199915e2ff6
commit: 8ab11c433a81b6c7ef84a66e36fbe199915e2ff6
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Victor Stinner <vstinner at python.org>
date: 2019-10-10T15:50:32+02:00
summary:

bpo-38109: Add missing constants to Lib/stat.py (GH-16665) (GH-16690)

Add missing stat.S_IFDOOR, stat.S_IFPORT, stat.S_IFWHT,
stat.S_ISDOOR, stat.S_ISPORT, and stat.S_ISWHT values to
the Python implementation of the stat module.
(cherry picked from commit 7bb14316b8ceddb813f31040a299af94a57ab339)

Co-authored-by: Ronan Lamy <ronan.lamy at gmail.com>

files:
A Misc/NEWS.d/next/Library/2019-10-10-00-25-28.bpo-38109.9w-IGF.rst
M Lib/stat.py
M Lib/test/test_stat.py

diff --git a/Lib/stat.py b/Lib/stat.py
index a9c678ec03c9c..fc024db3f4fbe 100644
--- a/Lib/stat.py
+++ b/Lib/stat.py
@@ -40,6 +40,10 @@ def S_IFMT(mode):
 S_IFIFO  = 0o010000  # fifo (named pipe)
 S_IFLNK  = 0o120000  # symbolic link
 S_IFSOCK = 0o140000  # socket file
+# Fallbacks for uncommon platform-specific constants
+S_IFDOOR = 0
+S_IFPORT = 0
+S_IFWHT = 0
 
 # Functions to test for each file type
 
@@ -71,6 +75,18 @@ def S_ISSOCK(mode):
     """Return True if mode is from a socket."""
     return S_IFMT(mode) == S_IFSOCK
 
+def S_ISDOOR(mode):
+    """Return True if mode is from a door."""
+    return False
+
+def S_ISPORT(mode):
+    """Return True if mode is from an event port."""
+    return False
+
+def S_ISWHT(mode):
+    """Return True if mode is from a whiteout."""
+    return False
+
 # Names for permission bits
 
 S_ISUID = 0o4000  # set UID bit
diff --git a/Lib/test/test_stat.py b/Lib/test/test_stat.py
index 17443bed0738b..be01db2ed5574 100644
--- a/Lib/test/test_stat.py
+++ b/Lib/test/test_stat.py
@@ -16,10 +16,10 @@ class TestFilemode:
                   'UF_IMMUTABLE', 'UF_NODUMP', 'UF_NOUNLINK', 'UF_OPAQUE'}
 
     formats = {'S_IFBLK', 'S_IFCHR', 'S_IFDIR', 'S_IFIFO', 'S_IFLNK',
-               'S_IFREG', 'S_IFSOCK'}
+               'S_IFREG', 'S_IFSOCK', 'S_IFDOOR', 'S_IFPORT', 'S_IFWHT'}
 
     format_funcs = {'S_ISBLK', 'S_ISCHR', 'S_ISDIR', 'S_ISFIFO', 'S_ISLNK',
-                    'S_ISREG', 'S_ISSOCK'}
+                    'S_ISREG', 'S_ISSOCK', 'S_ISDOOR', 'S_ISPORT', 'S_ISWHT'}
 
     stat_struct = {
         'ST_MODE': 0,
@@ -231,10 +231,6 @@ def test_file_attribute_constants(self):
 class TestFilemodeCStat(TestFilemode, unittest.TestCase):
     statmod = c_stat
 
-    formats = TestFilemode.formats | {'S_IFDOOR', 'S_IFPORT', 'S_IFWHT'}
-    format_funcs = TestFilemode.format_funcs | {'S_ISDOOR', 'S_ISPORT',
-                                                'S_ISWHT'}
-
 
 class TestFilemodePyStat(TestFilemode, unittest.TestCase):
     statmod = py_stat
diff --git a/Misc/NEWS.d/next/Library/2019-10-10-00-25-28.bpo-38109.9w-IGF.rst b/Misc/NEWS.d/next/Library/2019-10-10-00-25-28.bpo-38109.9w-IGF.rst
new file mode 100644
index 0000000000000..3f4484dc644fa
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-10-10-00-25-28.bpo-38109.9w-IGF.rst
@@ -0,0 +1,3 @@
+Add missing :data:`stat.S_IFDOOR`, :data:`stat.S_IFPORT`, :data:`stat.S_IFWHT`,
+:func:`stat.S_ISDOOR`, :func:`stat.S_ISPORT`, and :func:`stat.S_ISWHT` values to
+the Python implementation of :mod:`stat`.
\ No newline at end of file


From webhook-mailer at python.org  Thu Oct 10 09:50:42 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Thu, 10 Oct 2019 13:50:42 -0000
Subject: [Python-checkins] bpo-38109: Add missing constants to Lib/stat.py
 (GH-16665) (GH-16691)
Message-ID: <mailman.90.1570715444.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/f5ed41c1ae9a575e965d55c6a5e86fb59181eee8
commit: f5ed41c1ae9a575e965d55c6a5e86fb59181eee8
branch: 3.7
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Victor Stinner <vstinner at python.org>
date: 2019-10-10T15:50:38+02:00
summary:

bpo-38109: Add missing constants to Lib/stat.py (GH-16665) (GH-16691)

Add missing stat.S_IFDOOR, stat.S_IFPORT, stat.S_IFWHT,
stat.S_ISDOOR, stat.S_ISPORT, and stat.S_ISWHT values to
the Python implementation of the stat module.
(cherry picked from commit 7bb14316b8ceddb813f31040a299af94a57ab339)

Co-authored-by: Ronan Lamy <ronan.lamy at gmail.com>

files:
A Misc/NEWS.d/next/Library/2019-10-10-00-25-28.bpo-38109.9w-IGF.rst
M Lib/stat.py
M Lib/test/test_stat.py

diff --git a/Lib/stat.py b/Lib/stat.py
index 46837c06dacfb..165e057576615 100644
--- a/Lib/stat.py
+++ b/Lib/stat.py
@@ -40,6 +40,10 @@ def S_IFMT(mode):
 S_IFIFO  = 0o010000  # fifo (named pipe)
 S_IFLNK  = 0o120000  # symbolic link
 S_IFSOCK = 0o140000  # socket file
+# Fallbacks for uncommon platform-specific constants
+S_IFDOOR = 0
+S_IFPORT = 0
+S_IFWHT = 0
 
 # Functions to test for each file type
 
@@ -71,6 +75,18 @@ def S_ISSOCK(mode):
     """Return True if mode is from a socket."""
     return S_IFMT(mode) == S_IFSOCK
 
+def S_ISDOOR(mode):
+    """Return True if mode is from a door."""
+    return False
+
+def S_ISPORT(mode):
+    """Return True if mode is from an event port."""
+    return False
+
+def S_ISWHT(mode):
+    """Return True if mode is from a whiteout."""
+    return False
+
 # Names for permission bits
 
 S_ISUID = 0o4000  # set UID bit
diff --git a/Lib/test/test_stat.py b/Lib/test/test_stat.py
index 73cd901bdbf5c..4ee0dc2ea35fc 100644
--- a/Lib/test/test_stat.py
+++ b/Lib/test/test_stat.py
@@ -14,10 +14,10 @@ class TestFilemode:
                   'UF_IMMUTABLE', 'UF_NODUMP', 'UF_NOUNLINK', 'UF_OPAQUE'}
 
     formats = {'S_IFBLK', 'S_IFCHR', 'S_IFDIR', 'S_IFIFO', 'S_IFLNK',
-               'S_IFREG', 'S_IFSOCK'}
+               'S_IFREG', 'S_IFSOCK', 'S_IFDOOR', 'S_IFPORT', 'S_IFWHT'}
 
     format_funcs = {'S_ISBLK', 'S_ISCHR', 'S_ISDIR', 'S_ISFIFO', 'S_ISLNK',
-                    'S_ISREG', 'S_ISSOCK'}
+                    'S_ISREG', 'S_ISSOCK', 'S_ISDOOR', 'S_ISPORT', 'S_ISWHT'}
 
     stat_struct = {
         'ST_MODE': 0,
@@ -221,10 +221,6 @@ def test_file_attribute_constants(self):
 class TestFilemodeCStat(TestFilemode, unittest.TestCase):
     statmod = c_stat
 
-    formats = TestFilemode.formats | {'S_IFDOOR', 'S_IFPORT', 'S_IFWHT'}
-    format_funcs = TestFilemode.format_funcs | {'S_ISDOOR', 'S_ISPORT',
-                                                'S_ISWHT'}
-
 
 class TestFilemodePyStat(TestFilemode, unittest.TestCase):
     statmod = py_stat
diff --git a/Misc/NEWS.d/next/Library/2019-10-10-00-25-28.bpo-38109.9w-IGF.rst b/Misc/NEWS.d/next/Library/2019-10-10-00-25-28.bpo-38109.9w-IGF.rst
new file mode 100644
index 0000000000000..3f4484dc644fa
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-10-10-00-25-28.bpo-38109.9w-IGF.rst
@@ -0,0 +1,3 @@
+Add missing :data:`stat.S_IFDOOR`, :data:`stat.S_IFPORT`, :data:`stat.S_IFWHT`,
+:func:`stat.S_ISDOOR`, :func:`stat.S_ISPORT`, and :func:`stat.S_ISWHT` values to
+the Python implementation of :mod:`stat`.
\ No newline at end of file


From webhook-mailer at python.org  Thu Oct 10 12:23:48 2019
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Thu, 10 Oct 2019 16:23:48 -0000
Subject: [Python-checkins] Re-enable the OverflowError test for test_truediv
 on test_complex (GH-16591)
Message-ID: <mailman.91.1570724629.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/1dbe5373851acb85ba91f0be7b83c69563acd68d
commit: 1dbe5373851acb85ba91f0be7b83c69563acd68d
branch: master
author: Dong-hee Na <donghee.na92 at gmail.com>
committer: Serhiy Storchaka <storchaka at gmail.com>
date: 2019-10-10T19:23:36+03:00
summary:

Re-enable the OverflowError test for test_truediv on test_complex (GH-16591)

files:
M Lib/test/test_complex.py

diff --git a/Lib/test/test_complex.py b/Lib/test/test_complex.py
index fe1e566562de4..dee5c7fa308bd 100644
--- a/Lib/test/test_complex.py
+++ b/Lib/test/test_complex.py
@@ -100,8 +100,7 @@ def test_truediv(self):
                            complex(random(), random()))
 
         self.assertRaises(ZeroDivisionError, complex.__truediv__, 1+1j, 0+0j)
-        # FIXME: The following currently crashes on Alpha
-        # self.assertRaises(OverflowError, pow, 1e200+1j, 1e200+1j)
+        self.assertRaises(OverflowError, pow, 1e200+1j, 1e200+1j)
 
         self.assertAlmostEqual(complex.__truediv__(2+0j, 1+1j), 1-1j)
         self.assertRaises(ZeroDivisionError, complex.__truediv__, 1+1j, 0+0j)


From webhook-mailer at python.org  Thu Oct 10 13:11:12 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Thu, 10 Oct 2019 17:11:12 -0000
Subject: [Python-checkins] Re-enable the OverflowError test for test_truediv
 on test_complex (GH-16591)
Message-ID: <mailman.92.1570727472.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/398d84747d9299f1d4e8385fa8f6904ce961545e
commit: 398d84747d9299f1d4e8385fa8f6904ce961545e
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-10T10:11:07-07:00
summary:

Re-enable the OverflowError test for test_truediv on test_complex (GH-16591)

(cherry picked from commit 1dbe5373851acb85ba91f0be7b83c69563acd68d)

Co-authored-by: Dong-hee Na <donghee.na92 at gmail.com>

files:
M Lib/test/test_complex.py

diff --git a/Lib/test/test_complex.py b/Lib/test/test_complex.py
index fe1e566562de4..dee5c7fa308bd 100644
--- a/Lib/test/test_complex.py
+++ b/Lib/test/test_complex.py
@@ -100,8 +100,7 @@ def test_truediv(self):
                            complex(random(), random()))
 
         self.assertRaises(ZeroDivisionError, complex.__truediv__, 1+1j, 0+0j)
-        # FIXME: The following currently crashes on Alpha
-        # self.assertRaises(OverflowError, pow, 1e200+1j, 1e200+1j)
+        self.assertRaises(OverflowError, pow, 1e200+1j, 1e200+1j)
 
         self.assertAlmostEqual(complex.__truediv__(2+0j, 1+1j), 1-1j)
         self.assertRaises(ZeroDivisionError, complex.__truediv__, 1+1j, 0+0j)


From webhook-mailer at python.org  Thu Oct 10 15:30:24 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Thu, 10 Oct 2019 19:30:24 -0000
Subject: [Python-checkins] bpo-38282: Rewrite getsockaddrarg() helper
 function (GH-16698)
Message-ID: <mailman.93.1570735825.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/d565fb9828ee9c494bb7a80057a08e4738273e30
commit: d565fb9828ee9c494bb7a80057a08e4738273e30
branch: master
author: Victor Stinner <vstinner at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-10T21:30:20+02:00
summary:

bpo-38282: Rewrite getsockaddrarg() helper function (GH-16698)

Rewrite getsockaddrarg() helper function of socketmodule.c (_socket
module) to prevent a false alarm when compiling codde using GCC with
_FORTIFY_SOURCE=2. Pass a pointer of the sock_addr_t union, rather
than passing a pointer to a sockaddr structure.

Add "struct sockaddr_tipc tipc;" to the sock_addr_t union.

files:
M Modules/socketmodule.c
M Modules/socketmodule.h

diff --git a/Modules/socketmodule.c b/Modules/socketmodule.c
index c649fa3c982f9..d42fa7ce18293 100644
--- a/Modules/socketmodule.c
+++ b/Modules/socketmodule.c
@@ -1688,14 +1688,13 @@ idna_converter(PyObject *obj, struct maybe_idna *data)
 
 static int
 getsockaddrarg(PySocketSockObject *s, PyObject *args,
-               struct sockaddr *addr_ret, int *len_ret, const char *caller)
+               sock_addr_t *addrbuf, int *len_ret, const char *caller)
 {
     switch (s->sock_family) {
 
 #if defined(AF_UNIX)
     case AF_UNIX:
     {
-        struct sockaddr_un* addr;
         Py_buffer path;
         int retval = 0;
 
@@ -1713,7 +1712,7 @@ getsockaddrarg(PySocketSockObject *s, PyObject *args,
         }
         assert(path.len >= 0);
 
-        addr = (struct sockaddr_un*)addr_ret;
+        struct sockaddr_un* addr = &addrbuf->un;
 #ifdef __linux__
         if (path.len > 0 && *(const char *)path.buf == 0) {
             /* Linux abstract namespace extension */
@@ -1748,9 +1747,8 @@ getsockaddrarg(PySocketSockObject *s, PyObject *args,
 #if defined(AF_NETLINK)
     case AF_NETLINK:
     {
-        struct sockaddr_nl* addr;
         int pid, groups;
-        addr = (struct sockaddr_nl *)addr_ret;
+        struct sockaddr_nl* addr = &addrbuf->nl;
         if (!PyTuple_Check(args)) {
             PyErr_Format(
                 PyExc_TypeError,
@@ -1776,9 +1774,8 @@ getsockaddrarg(PySocketSockObject *s, PyObject *args,
 #if defined(AF_QIPCRTR)
     case AF_QIPCRTR:
     {
-        struct sockaddr_qrtr* addr;
         unsigned int node, port;
-        addr = (struct sockaddr_qrtr *)addr_ret;
+        struct sockaddr_qrtr* addr = &addrbuf->sq;
         if (!PyTuple_Check(args)) {
             PyErr_Format(
                 PyExc_TypeError,
@@ -1800,9 +1797,8 @@ getsockaddrarg(PySocketSockObject *s, PyObject *args,
 #if defined(AF_VSOCK)
     case AF_VSOCK:
     {
-        struct sockaddr_vm* addr;
+        struct sockaddr_vm* addr = &addrbuf->vm;
         int port, cid;
-        addr = (struct sockaddr_vm *)addr_ret;
         memset(addr, 0, sizeof(struct sockaddr_vm));
         if (!PyTuple_Check(args)) {
             PyErr_Format(
@@ -1830,7 +1826,6 @@ getsockaddrarg(PySocketSockObject *s, PyObject *args,
 
     case AF_INET:
     {
-        struct sockaddr_in* addr;
         struct maybe_idna host = {NULL, NULL};
         int port, result;
         if (!PyTuple_Check(args)) {
@@ -1852,7 +1847,7 @@ getsockaddrarg(PySocketSockObject *s, PyObject *args,
             }
             return 0;
         }
-        addr=(struct sockaddr_in*)addr_ret;
+        struct sockaddr_in* addr = &addrbuf->in;
         result = setipaddr(host.buf, (struct sockaddr *)addr,
                            sizeof(*addr),  AF_INET);
         idna_cleanup(&host);
@@ -1873,7 +1868,6 @@ getsockaddrarg(PySocketSockObject *s, PyObject *args,
 #ifdef ENABLE_IPV6
     case AF_INET6:
     {
-        struct sockaddr_in6* addr;
         struct maybe_idna host = {NULL, NULL};
         int port, result;
         unsigned int flowinfo, scope_id;
@@ -1898,7 +1892,7 @@ getsockaddrarg(PySocketSockObject *s, PyObject *args,
             }
             return 0;
         }
-        addr = (struct sockaddr_in6*)addr_ret;
+        struct sockaddr_in6* addr = &addrbuf->in6;
         result = setipaddr(host.buf, (struct sockaddr *)addr,
                            sizeof(*addr), AF_INET6);
         idna_cleanup(&host);
@@ -1932,10 +1926,9 @@ getsockaddrarg(PySocketSockObject *s, PyObject *args,
 #ifdef BTPROTO_L2CAP
         case BTPROTO_L2CAP:
         {
-            struct sockaddr_l2 *addr;
             const char *straddr;
 
-            addr = (struct sockaddr_l2 *)addr_ret;
+            struct sockaddr_l2 *addr = &addrbuf->bt_l2;
             memset(addr, 0, sizeof(struct sockaddr_l2));
             _BT_L2_MEMB(addr, family) = AF_BLUETOOTH;
             if (!PyArg_ParseTuple(args, "si", &straddr,
@@ -1953,10 +1946,8 @@ getsockaddrarg(PySocketSockObject *s, PyObject *args,
 #endif /* BTPROTO_L2CAP */
         case BTPROTO_RFCOMM:
         {
-            struct sockaddr_rc *addr;
             const char *straddr;
-
-            addr = (struct sockaddr_rc *)addr_ret;
+            struct sockaddr_rc *addr = &addrbuf->bt_rc;
             _BT_RC_MEMB(addr, family) = AF_BLUETOOTH;
             if (!PyArg_ParseTuple(args, "si", &straddr,
                                   &_BT_RC_MEMB(addr, channel))) {
@@ -1973,7 +1964,7 @@ getsockaddrarg(PySocketSockObject *s, PyObject *args,
 #ifdef BTPROTO_HCI
         case BTPROTO_HCI:
         {
-            struct sockaddr_hci *addr = (struct sockaddr_hci *)addr_ret;
+            struct sockaddr_hci *addr = &addrbuf->bt_hci;
 #if defined(__NetBSD__) || defined(__DragonFly__)
             const char *straddr;
             _BT_HCI_MEMB(addr, family) = AF_BLUETOOTH;
@@ -1999,10 +1990,9 @@ getsockaddrarg(PySocketSockObject *s, PyObject *args,
 #if !defined(__FreeBSD__)
         case BTPROTO_SCO:
         {
-            struct sockaddr_sco *addr;
             const char *straddr;
 
-            addr = (struct sockaddr_sco *)addr_ret;
+            struct sockaddr_sco *addr = &addrbuf->bt_sco;
             _BT_SCO_MEMB(addr, family) = AF_BLUETOOTH;
             if (!PyBytes_Check(args)) {
                 PyErr_Format(PyExc_OSError,
@@ -2029,7 +2019,6 @@ getsockaddrarg(PySocketSockObject *s, PyObject *args,
 #if defined(HAVE_NETPACKET_PACKET_H) && defined(SIOCGIFINDEX)
     case AF_PACKET:
     {
-        struct sockaddr_ll* addr;
         struct ifreq ifr;
         const char *interfaceName;
         int protoNumber;
@@ -2080,7 +2069,7 @@ getsockaddrarg(PySocketSockObject *s, PyObject *args,
             PyBuffer_Release(&haddr);
             return 0;
         }
-        addr = (struct sockaddr_ll*)addr_ret;
+        struct sockaddr_ll* addr = &addrbuf->ll;
         addr->sll_family = AF_PACKET;
         addr->sll_protocol = htons((short)protoNumber);
         addr->sll_ifindex = ifr.ifr_ifindex;
@@ -2103,7 +2092,6 @@ getsockaddrarg(PySocketSockObject *s, PyObject *args,
     {
         unsigned int atype, v1, v2, v3;
         unsigned int scope = TIPC_CLUSTER_SCOPE;
-        struct sockaddr_tipc *addr;
 
         if (!PyTuple_Check(args)) {
             PyErr_Format(
@@ -2121,7 +2109,7 @@ getsockaddrarg(PySocketSockObject *s, PyObject *args,
             return 0;
         }
 
-        addr = (struct sockaddr_tipc *) addr_ret;
+        struct sockaddr_tipc *addr = &addrbuf->tipc;
         memset(addr, 0, sizeof(struct sockaddr_tipc));
 
         addr->family = AF_TIPC;
@@ -2162,11 +2150,10 @@ getsockaddrarg(PySocketSockObject *s, PyObject *args,
 #endif
 #if defined(CAN_RAW) || defined(CAN_BCM)
         {
-            struct sockaddr_can *addr;
             PyObject *interfaceName;
             struct ifreq ifr;
             Py_ssize_t len;
-            addr = (struct sockaddr_can *)addr_ret;
+            struct sockaddr_can *addr = &addrbuf->can;
 
             if (!PyTuple_Check(args)) {
                 PyErr_Format(PyExc_TypeError,
@@ -2213,13 +2200,12 @@ getsockaddrarg(PySocketSockObject *s, PyObject *args,
 #ifdef CAN_ISOTP
         case CAN_ISOTP:
         {
-            struct sockaddr_can *addr;
             PyObject *interfaceName;
             struct ifreq ifr;
             Py_ssize_t len;
             unsigned long int rx_id, tx_id;
 
-            addr = (struct sockaddr_can *)addr_ret;
+            struct sockaddr_can *addr = &addrbuf->can;
 
             if (!PyArg_ParseTuple(args, "O&kk", PyUnicode_FSConverter,
                                               &interfaceName,
@@ -2269,9 +2255,7 @@ getsockaddrarg(PySocketSockObject *s, PyObject *args,
 #ifdef SYSPROTO_CONTROL
         case SYSPROTO_CONTROL:
         {
-            struct sockaddr_ctl *addr;
-
-            addr = (struct sockaddr_ctl *)addr_ret;
+            struct sockaddr_ctl *addr = &addrbuf->ctl;
             addr->sc_family = AF_SYSTEM;
             addr->ss_sysaddr = AF_SYS_CONTROL;
 
@@ -2323,10 +2307,9 @@ getsockaddrarg(PySocketSockObject *s, PyObject *args,
 #ifdef HAVE_SOCKADDR_ALG
     case AF_ALG:
     {
-        struct sockaddr_alg *sa;
         const char *type;
         const char *name;
-        sa = (struct sockaddr_alg *)addr_ret;
+        struct sockaddr_alg *sa = &addrbuf->alg;
 
         memset(sa, 0, sizeof(*sa));
         sa->salg_family = AF_ALG;
@@ -3110,7 +3093,7 @@ sock_bind(PySocketSockObject *s, PyObject *addro)
     int addrlen;
     int res;
 
-    if (!getsockaddrarg(s, addro, SAS2SA(&addrbuf), &addrlen, "bind")) {
+    if (!getsockaddrarg(s, addro, &addrbuf, &addrlen, "bind")) {
         return NULL;
     }
 
@@ -3280,7 +3263,7 @@ sock_connect(PySocketSockObject *s, PyObject *addro)
     int addrlen;
     int res;
 
-    if (!getsockaddrarg(s, addro, SAS2SA(&addrbuf), &addrlen, "connect")) {
+    if (!getsockaddrarg(s, addro, &addrbuf, &addrlen, "connect")) {
         return NULL;
     }
 
@@ -3311,7 +3294,7 @@ sock_connect_ex(PySocketSockObject *s, PyObject *addro)
     int addrlen;
     int res;
 
-    if (!getsockaddrarg(s, addro, SAS2SA(&addrbuf), &addrlen, "connect_ex")) {
+    if (!getsockaddrarg(s, addro, &addrbuf, &addrlen, "connect_ex")) {
         return NULL;
     }
 
@@ -4316,7 +4299,7 @@ sock_sendto(PySocketSockObject *s, PyObject *args)
         return select_error();
     }
 
-    if (!getsockaddrarg(s, addro, SAS2SA(&addrbuf), &addrlen, "sendto")) {
+    if (!getsockaddrarg(s, addro, &addrbuf, &addrlen, "sendto")) {
         PyBuffer_Release(&pbuf);
         return NULL;
     }
@@ -4451,7 +4434,7 @@ sock_sendmsg(PySocketSockObject *s, PyObject *args)
 
     /* Parse destination address. */
     if (addr_arg != NULL && addr_arg != Py_None) {
-        if (!getsockaddrarg(s, addr_arg, SAS2SA(&addrbuf), &addrlen,
+        if (!getsockaddrarg(s, addr_arg, &addrbuf, &addrlen,
                             "sendmsg"))
         {
             goto finally;
diff --git a/Modules/socketmodule.h b/Modules/socketmodule.h
index 3d95fe709ac78..e06e4c388827e 100644
--- a/Modules/socketmodule.h
+++ b/Modules/socketmodule.h
@@ -261,6 +261,9 @@ typedef union sock_addr {
 #ifdef AF_VSOCK
     struct sockaddr_vm vm;
 #endif
+#ifdef HAVE_LINUX_TIPC_H
+    struct sockaddr_tipc tipc;
+#endif
 } sock_addr_t;
 
 /* The object holding a socket.  It holds some extra information,


From webhook-mailer at python.org  Thu Oct 10 17:39:19 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Thu, 10 Oct 2019 21:39:19 -0000
Subject: [Python-checkins] Remove AppVeyor badge now that we don't use it in
 the CI anymore (GH-16708)
Message-ID: <mailman.94.1570743560.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/72bbd2a2e34a9e58d4bc9eae4c84c15a656215be
commit: 72bbd2a2e34a9e58d4bc9eae4c84c15a656215be
branch: master
author: Pablo Galindo <Pablogsal at gmail.com>
committer: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
date: 2019-10-10T14:39:15-07:00
summary:

Remove AppVeyor badge now that we don't use it in the CI anymore (GH-16708)

files:
M README.rst

diff --git a/README.rst b/README.rst
index 0f658531cbac2..ca397b896e2dc 100644
--- a/README.rst
+++ b/README.rst
@@ -5,10 +5,6 @@ This is Python version 3.9.0 alpha 0
    :alt: CPython build status on Travis CI
    :target: https://travis-ci.org/python/cpython
 
-.. image:: https://ci.appveyor.com/api/projects/status/4mew1a93xdkbf5ua/branch/master?svg=true
-   :alt: CPython build status on Appveyor
-   :target: https://ci.appveyor.com/project/python/cpython/branch/master
-
 .. image:: https://dev.azure.com/python/cpython/_apis/build/status/Azure%20Pipelines%20CI?branchName=master
    :alt: CPython build status on Azure DevOps
    :target: https://dev.azure.com/python/cpython/_build/latest?definitionId=4&branchName=master


From webhook-mailer at python.org  Thu Oct 10 17:45:21 2019
From: webhook-mailer at python.org (Pablo Galindo)
Date: Thu, 10 Oct 2019 21:45:21 -0000
Subject: [Python-checkins] bpo-38437: Activate GC_DEBUG when PY_DEBUG is set
 (GH-16707)
Message-ID: <mailman.95.1570743922.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/320dd504ddf65efe946e6d1e89053ed2d7ebe1e9
commit: 320dd504ddf65efe946e6d1e89053ed2d7ebe1e9
branch: master
author: Pablo Galindo <Pablogsal at gmail.com>
committer: GitHub <noreply at github.com>
date: 2019-10-10T22:45:17+01:00
summary:

bpo-38437: Activate GC_DEBUG when PY_DEBUG is set (GH-16707)

files:
A Misc/NEWS.d/next/Core and Builtins/2019-10-10-20-42-09.bpo-38437.z_0mZp.rst
M Modules/gcmodule.c

diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-10-10-20-42-09.bpo-38437.z_0mZp.rst b/Misc/NEWS.d/next/Core and Builtins/2019-10-10-20-42-09.bpo-38437.z_0mZp.rst
new file mode 100644
index 0000000000000..69a8843ce0f69
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2019-10-10-20-42-09.bpo-38437.z_0mZp.rst	
@@ -0,0 +1,2 @@
+Activate the ``GC_DEBUG`` macro for debug builds of the interpreter (when
+``Py_DEBUG`` is set). Patch by Pablo Galindo.
diff --git a/Modules/gcmodule.c b/Modules/gcmodule.c
index b1e802e33e498..a1cb323bd24cb 100644
--- a/Modules/gcmodule.c
+++ b/Modules/gcmodule.c
@@ -37,7 +37,10 @@ module gc
 [clinic start generated code]*/
 /*[clinic end generated code: output=da39a3ee5e6b4b0d input=b5c9690ecc842d79]*/
 
-#define GC_DEBUG (0)  /* Enable more asserts */
+
+#ifdef Py_DEBUG
+#  define GC_DEBUG
+#endif
 
 #define GC_NEXT _PyGCHead_NEXT
 #define GC_PREV _PyGCHead_PREV
@@ -316,7 +319,7 @@ append_objects(PyObject *py_list, PyGC_Head *gc_list)
     return 0;
 }
 
-#if GC_DEBUG
+#ifdef GC_DEBUG
 // validate_list checks list consistency.  And it works as document
 // describing when expected_mask is set / unset.
 static void


From webhook-mailer at python.org  Thu Oct 10 19:18:52 2019
From: webhook-mailer at python.org (Yury Selivanov)
Date: Thu, 10 Oct 2019 23:18:52 -0000
Subject: [Python-checkins] docs: Add asyncio source code links (GH-16640)
Message-ID: <mailman.96.1570749533.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/f900064ac4b35226caad7502abc8a7e64f1c0e9d
commit: f900064ac4b35226caad7502abc8a7e64f1c0e9d
branch: master
author: Kyle Stanley <aeros167 at gmail.com>
committer: Yury Selivanov <yury at magic.io>
date: 2019-10-10T19:18:46-04:00
summary:

docs: Add asyncio source code links (GH-16640)

files:
M Doc/library/asyncio-eventloop.rst
M Doc/library/asyncio-exceptions.rst
M Doc/library/asyncio-future.rst
M Doc/library/asyncio-platforms.rst
M Doc/library/asyncio-protocol.rst
M Doc/library/asyncio-queue.rst
M Doc/library/asyncio-stream.rst
M Doc/library/asyncio-subprocess.rst
M Doc/library/asyncio-sync.rst
M Doc/library/asyncio-task.rst
M Doc/library/asyncio.rst

diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst
index 045787e0ce787..240d814f13dea 100644
--- a/Doc/library/asyncio-eventloop.rst
+++ b/Doc/library/asyncio-eventloop.rst
@@ -5,6 +5,10 @@
 Event Loop
 ==========
 
+**Source code:** :source:`Lib/asyncio/events.py`,
+:source:`Lib/asyncio/base_events.py`
+
+------------------------------------
 
 .. rubric:: Preface
 
diff --git a/Doc/library/asyncio-exceptions.rst b/Doc/library/asyncio-exceptions.rst
index b1e6ecfda15ea..7166d5c4bd88f 100644
--- a/Doc/library/asyncio-exceptions.rst
+++ b/Doc/library/asyncio-exceptions.rst
@@ -7,6 +7,9 @@
 Exceptions
 ==========
 
+**Source code:** :source:`Lib/asyncio/exceptions.py`
+
+----------------------------------------------------
 
 .. exception:: TimeoutError
 
diff --git a/Doc/library/asyncio-future.rst b/Doc/library/asyncio-future.rst
index c069471b519be..832d58119b7b0 100644
--- a/Doc/library/asyncio-future.rst
+++ b/Doc/library/asyncio-future.rst
@@ -7,6 +7,11 @@
 Futures
 =======
 
+**Source code:** :source:`Lib/asyncio/futures.py`,
+:source:`Lib/asyncio/base_futures.py`
+
+-------------------------------------
+
 *Future* objects are used to bridge **low-level callback-based code**
 with high-level async/await code.
 
diff --git a/Doc/library/asyncio-platforms.rst b/Doc/library/asyncio-platforms.rst
index 7e4a70f91c6ed..390ee1969d096 100644
--- a/Doc/library/asyncio-platforms.rst
+++ b/Doc/library/asyncio-platforms.rst
@@ -23,6 +23,12 @@ All Platforms
 Windows
 =======
 
+**Source code:** :source:`Lib/asyncio/proactor_events.py`,
+:source:`Lib/asyncio/windows_events.py`,
+:source:`Lib/asyncio/windows_utils.py`
+
+--------------------------------------
+
 .. versionchanged:: 3.8
 
    On Windows, :class:`ProactorEventLoop` is now the default event loop.
diff --git a/Doc/library/asyncio-protocol.rst b/Doc/library/asyncio-protocol.rst
index 67ca121081684..ffac9018127c6 100644
--- a/Doc/library/asyncio-protocol.rst
+++ b/Doc/library/asyncio-protocol.rst
@@ -69,6 +69,10 @@ This documentation page contains the following sections:
 Transports
 ==========
 
+**Source code:** :source:`Lib/asyncio/transports.py`
+
+----------------------------------------------------
+
 Transports are classes provided by :mod:`asyncio` in order to abstract
 various kinds of communication channels.
 
@@ -431,6 +435,10 @@ Subprocess Transports
 Protocols
 =========
 
+**Source code:** :source:`Lib/asyncio/protocols.py`
+
+---------------------------------------------------
+
 asyncio provides a set of abstract base classes that should be used
 to implement network protocols.  Those classes are meant to be used
 together with :ref:`transports <asyncio-transport>`.
diff --git a/Doc/library/asyncio-queue.rst b/Doc/library/asyncio-queue.rst
index e7775da54f589..524560b691d72 100644
--- a/Doc/library/asyncio-queue.rst
+++ b/Doc/library/asyncio-queue.rst
@@ -6,6 +6,10 @@
 Queues
 ======
 
+**Source code:** :source:`Lib/asyncio/queues.py`
+
+------------------------------------------------
+
 asyncio queues are designed to be similar to classes of the
 :mod:`queue` module.  Although asyncio queues are not thread-safe,
 they are designed to be used specifically in async/await code.
diff --git a/Doc/library/asyncio-stream.rst b/Doc/library/asyncio-stream.rst
index 471e6e9099c2c..bb899714a0ec0 100644
--- a/Doc/library/asyncio-stream.rst
+++ b/Doc/library/asyncio-stream.rst
@@ -6,6 +6,10 @@
 Streams
 =======
 
+**Source code:** :source:`Lib/asyncio/streams.py`
+
+-------------------------------------------------
+
 Streams are high-level async/await-ready primitives to work with
 network connections.  Streams allow sending and receiving data without
 using callbacks or low-level protocols and transports.
diff --git a/Doc/library/asyncio-subprocess.rst b/Doc/library/asyncio-subprocess.rst
index bd92257c82efa..1d87d2f8005ec 100644
--- a/Doc/library/asyncio-subprocess.rst
+++ b/Doc/library/asyncio-subprocess.rst
@@ -6,6 +6,11 @@
 Subprocesses
 ============
 
+**Source code:** :source:`Lib/asyncio/subprocess.py`,
+:source:`Lib/asyncio/base_subprocess.py`
+
+----------------------------------------
+
 This section describes high-level async/await asyncio APIs to
 create and manage subprocesses.
 
diff --git a/Doc/library/asyncio-sync.rst b/Doc/library/asyncio-sync.rst
index cc8c29dc3b6bc..f080b03bc7c51 100644
--- a/Doc/library/asyncio-sync.rst
+++ b/Doc/library/asyncio-sync.rst
@@ -6,6 +6,10 @@
 Synchronization Primitives
 ==========================
 
+**Source code:** :source:`Lib/asyncio/locks.py`
+
+-----------------------------------------------
+
 asyncio synchronization primitives are designed to be similar to
 those of the :mod:`threading` module with two important caveats:
 
diff --git a/Doc/library/asyncio-task.rst b/Doc/library/asyncio-task.rst
index 155887a3ab3af..d992b0011dc66 100644
--- a/Doc/library/asyncio-task.rst
+++ b/Doc/library/asyncio-task.rst
@@ -18,9 +18,9 @@ and Tasks.
 Coroutines
 ==========
 
-Coroutines declared with async/await syntax is the preferred way of
-writing asyncio applications.  For example, the following snippet
-of code (requires Python 3.7+) prints "hello", waits 1 second,
+:term:`Coroutines <coroutine>` declared with the async/await syntax is the
+preferred way of writing asyncio applications.  For example, the following
+snippet of code (requires Python 3.7+) prints "hello", waits 1 second,
 and then prints "world"::
 
     >>> import asyncio
@@ -240,6 +240,10 @@ Running an asyncio Program
     .. versionchanged:: 3.9
        Updated to use :meth:`loop.shutdown_default_executor`.
 
+    .. note::
+       The source code for ``asyncio.run()`` can be found in
+       :source:`Lib/asyncio/runners.py`.
+
 Creating Tasks
 ==============
 
diff --git a/Doc/library/asyncio.rst b/Doc/library/asyncio.rst
index 6990adb21e360..94a853259d348 100644
--- a/Doc/library/asyncio.rst
+++ b/Doc/library/asyncio.rst
@@ -4,7 +4,7 @@
 .. module:: asyncio
    :synopsis: Asynchronous I/O.
 
---------------
+-------------------------------
 
 .. sidebar:: Hello World!
 
@@ -91,3 +91,6 @@ Additionally, there are **low-level** APIs for
    asyncio-api-index.rst
    asyncio-llapi-index.rst
    asyncio-dev.rst
+
+.. note::
+   The source code for asyncio can be found in :source:`Lib/asyncio/`.


From webhook-mailer at python.org  Thu Oct 10 19:25:32 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Thu, 10 Oct 2019 23:25:32 -0000
Subject: [Python-checkins] docs: Add asyncio source code links (GH-16640)
Message-ID: <mailman.97.1570749933.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/6bfeb819e5255035563afd7c6224f364d44a6213
commit: 6bfeb819e5255035563afd7c6224f364d44a6213
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-10T16:25:28-07:00
summary:

docs: Add asyncio source code links (GH-16640)

(cherry picked from commit f900064ac4b35226caad7502abc8a7e64f1c0e9d)

Co-authored-by: Kyle Stanley <aeros167 at gmail.com>

files:
M Doc/library/asyncio-eventloop.rst
M Doc/library/asyncio-exceptions.rst
M Doc/library/asyncio-future.rst
M Doc/library/asyncio-platforms.rst
M Doc/library/asyncio-protocol.rst
M Doc/library/asyncio-queue.rst
M Doc/library/asyncio-stream.rst
M Doc/library/asyncio-subprocess.rst
M Doc/library/asyncio-sync.rst
M Doc/library/asyncio-task.rst
M Doc/library/asyncio.rst

diff --git a/Doc/library/asyncio-eventloop.rst b/Doc/library/asyncio-eventloop.rst
index f763fd5f036d6..1f4f0b6e75b9e 100644
--- a/Doc/library/asyncio-eventloop.rst
+++ b/Doc/library/asyncio-eventloop.rst
@@ -5,6 +5,10 @@
 Event Loop
 ==========
 
+**Source code:** :source:`Lib/asyncio/events.py`,
+:source:`Lib/asyncio/base_events.py`
+
+------------------------------------
 
 .. rubric:: Preface
 
diff --git a/Doc/library/asyncio-exceptions.rst b/Doc/library/asyncio-exceptions.rst
index b1e6ecfda15ea..7166d5c4bd88f 100644
--- a/Doc/library/asyncio-exceptions.rst
+++ b/Doc/library/asyncio-exceptions.rst
@@ -7,6 +7,9 @@
 Exceptions
 ==========
 
+**Source code:** :source:`Lib/asyncio/exceptions.py`
+
+----------------------------------------------------
 
 .. exception:: TimeoutError
 
diff --git a/Doc/library/asyncio-future.rst b/Doc/library/asyncio-future.rst
index c069471b519be..832d58119b7b0 100644
--- a/Doc/library/asyncio-future.rst
+++ b/Doc/library/asyncio-future.rst
@@ -7,6 +7,11 @@
 Futures
 =======
 
+**Source code:** :source:`Lib/asyncio/futures.py`,
+:source:`Lib/asyncio/base_futures.py`
+
+-------------------------------------
+
 *Future* objects are used to bridge **low-level callback-based code**
 with high-level async/await code.
 
diff --git a/Doc/library/asyncio-platforms.rst b/Doc/library/asyncio-platforms.rst
index 7e4a70f91c6ed..390ee1969d096 100644
--- a/Doc/library/asyncio-platforms.rst
+++ b/Doc/library/asyncio-platforms.rst
@@ -23,6 +23,12 @@ All Platforms
 Windows
 =======
 
+**Source code:** :source:`Lib/asyncio/proactor_events.py`,
+:source:`Lib/asyncio/windows_events.py`,
+:source:`Lib/asyncio/windows_utils.py`
+
+--------------------------------------
+
 .. versionchanged:: 3.8
 
    On Windows, :class:`ProactorEventLoop` is now the default event loop.
diff --git a/Doc/library/asyncio-protocol.rst b/Doc/library/asyncio-protocol.rst
index 67ca121081684..ffac9018127c6 100644
--- a/Doc/library/asyncio-protocol.rst
+++ b/Doc/library/asyncio-protocol.rst
@@ -69,6 +69,10 @@ This documentation page contains the following sections:
 Transports
 ==========
 
+**Source code:** :source:`Lib/asyncio/transports.py`
+
+----------------------------------------------------
+
 Transports are classes provided by :mod:`asyncio` in order to abstract
 various kinds of communication channels.
 
@@ -431,6 +435,10 @@ Subprocess Transports
 Protocols
 =========
 
+**Source code:** :source:`Lib/asyncio/protocols.py`
+
+---------------------------------------------------
+
 asyncio provides a set of abstract base classes that should be used
 to implement network protocols.  Those classes are meant to be used
 together with :ref:`transports <asyncio-transport>`.
diff --git a/Doc/library/asyncio-queue.rst b/Doc/library/asyncio-queue.rst
index e7775da54f589..524560b691d72 100644
--- a/Doc/library/asyncio-queue.rst
+++ b/Doc/library/asyncio-queue.rst
@@ -6,6 +6,10 @@
 Queues
 ======
 
+**Source code:** :source:`Lib/asyncio/queues.py`
+
+------------------------------------------------
+
 asyncio queues are designed to be similar to classes of the
 :mod:`queue` module.  Although asyncio queues are not thread-safe,
 they are designed to be used specifically in async/await code.
diff --git a/Doc/library/asyncio-stream.rst b/Doc/library/asyncio-stream.rst
index 471e6e9099c2c..bb899714a0ec0 100644
--- a/Doc/library/asyncio-stream.rst
+++ b/Doc/library/asyncio-stream.rst
@@ -6,6 +6,10 @@
 Streams
 =======
 
+**Source code:** :source:`Lib/asyncio/streams.py`
+
+-------------------------------------------------
+
 Streams are high-level async/await-ready primitives to work with
 network connections.  Streams allow sending and receiving data without
 using callbacks or low-level protocols and transports.
diff --git a/Doc/library/asyncio-subprocess.rst b/Doc/library/asyncio-subprocess.rst
index bd92257c82efa..1d87d2f8005ec 100644
--- a/Doc/library/asyncio-subprocess.rst
+++ b/Doc/library/asyncio-subprocess.rst
@@ -6,6 +6,11 @@
 Subprocesses
 ============
 
+**Source code:** :source:`Lib/asyncio/subprocess.py`,
+:source:`Lib/asyncio/base_subprocess.py`
+
+----------------------------------------
+
 This section describes high-level async/await asyncio APIs to
 create and manage subprocesses.
 
diff --git a/Doc/library/asyncio-sync.rst b/Doc/library/asyncio-sync.rst
index cc8c29dc3b6bc..f080b03bc7c51 100644
--- a/Doc/library/asyncio-sync.rst
+++ b/Doc/library/asyncio-sync.rst
@@ -6,6 +6,10 @@
 Synchronization Primitives
 ==========================
 
+**Source code:** :source:`Lib/asyncio/locks.py`
+
+-----------------------------------------------
+
 asyncio synchronization primitives are designed to be similar to
 those of the :mod:`threading` module with two important caveats:
 
diff --git a/Doc/library/asyncio-task.rst b/Doc/library/asyncio-task.rst
index 6829233793916..1a23661fc772e 100644
--- a/Doc/library/asyncio-task.rst
+++ b/Doc/library/asyncio-task.rst
@@ -18,9 +18,9 @@ and Tasks.
 Coroutines
 ==========
 
-Coroutines declared with async/await syntax is the preferred way of
-writing asyncio applications.  For example, the following snippet
-of code (requires Python 3.7+) prints "hello", waits 1 second,
+:term:`Coroutines <coroutine>` declared with the async/await syntax is the
+preferred way of writing asyncio applications.  For example, the following
+snippet of code (requires Python 3.7+) prints "hello", waits 1 second,
 and then prints "world"::
 
     >>> import asyncio
@@ -238,6 +238,10 @@ Running an asyncio Program
     .. versionadded:: 3.7
 
 
+    .. note::
+       The source code for ``asyncio.run()`` can be found in
+       :source:`Lib/asyncio/runners.py`.
+
 Creating Tasks
 ==============
 
diff --git a/Doc/library/asyncio.rst b/Doc/library/asyncio.rst
index 6990adb21e360..94a853259d348 100644
--- a/Doc/library/asyncio.rst
+++ b/Doc/library/asyncio.rst
@@ -4,7 +4,7 @@
 .. module:: asyncio
    :synopsis: Asynchronous I/O.
 
---------------
+-------------------------------
 
 .. sidebar:: Hello World!
 
@@ -91,3 +91,6 @@ Additionally, there are **low-level** APIs for
    asyncio-api-index.rst
    asyncio-llapi-index.rst
    asyncio-dev.rst
+
+.. note::
+   The source code for asyncio can be found in :source:`Lib/asyncio/`.


From webhook-mailer at python.org  Thu Oct 10 23:43:25 2019
From: webhook-mailer at python.org (Benjamin Peterson)
Date: Fri, 11 Oct 2019 03:43:25 -0000
Subject: [Python-checkins] Typo fix: "throuhgh" should be "through".
 (GH-16704)
Message-ID: <mailman.98.1570765406.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/a8e0d3141e271b3c0fbb7399a911f0c3aa567e30
commit: a8e0d3141e271b3c0fbb7399a911f0c3aa567e30
branch: master
author: Hansraj Das <raj.das.136 at gmail.com>
committer: Benjamin Peterson <benjamin at python.org>
date: 2019-10-10T20:43:13-07:00
summary:

Typo fix: "throuhgh" should be "through". (GH-16704)

files:
M Python/condvar.h

diff --git a/Python/condvar.h b/Python/condvar.h
index f54adc5ea8e82..8cba19b84612d 100644
--- a/Python/condvar.h
+++ b/Python/condvar.h
@@ -178,7 +178,7 @@ _PyCOND_WAIT_MS(PyCOND_T *cv, PyMUTEX_T *cs, DWORD ms)
          * just means an extra spurious wakeup for a waiting thread.
          * ('waiting' corresponds to the semaphore's "negative" count and
          * we may end up with e.g. (waiting == -1 && sem.count == 1).  When
-         * a new thread comes along, it will pass right throuhgh, having
+         * a new thread comes along, it will pass right through, having
          * adjusted it to (waiting == 0 && sem.count == 0).
          */
 


From webhook-mailer at python.org  Fri Oct 11 00:01:39 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Fri, 11 Oct 2019 04:01:39 -0000
Subject: [Python-checkins] Typo fix: "throuhgh" should be "through".
 (GH-16704)
Message-ID: <mailman.99.1570766500.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/08afed76051c1ae67a47b456f6b737d39bde0082
commit: 08afed76051c1ae67a47b456f6b737d39bde0082
branch: 3.7
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-10T21:01:33-07:00
summary:

Typo fix: "throuhgh" should be "through". (GH-16704)

(cherry picked from commit a8e0d3141e271b3c0fbb7399a911f0c3aa567e30)

Co-authored-by: Hansraj Das <raj.das.136 at gmail.com>

files:
M Python/condvar.h

diff --git a/Python/condvar.h b/Python/condvar.h
index 951b8ad47f07b..802d53fbe69da 100644
--- a/Python/condvar.h
+++ b/Python/condvar.h
@@ -194,7 +194,7 @@ _PyCOND_WAIT_MS(PyCOND_T *cv, PyMUTEX_T *cs, DWORD ms)
          * just means an extra spurious wakeup for a waiting thread.
          * ('waiting' corresponds to the semaphore's "negative" count and
          * we may end up with e.g. (waiting == -1 && sem.count == 1).  When
-         * a new thread comes along, it will pass right throuhgh, having
+         * a new thread comes along, it will pass right through, having
          * adjusted it to (waiting == 0 && sem.count == 0).
          */
 


From webhook-mailer at python.org  Fri Oct 11 00:02:41 2019
From: webhook-mailer at python.org (Benjamin Peterson)
Date: Fri, 11 Oct 2019 04:02:41 -0000
Subject: [Python-checkins] [3.8] Typo fix: "throuhgh" should be "through".
 (GH-16713)
Message-ID: <mailman.100.1570766561.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/380eb427ddd067042539d5d5f2b7f88a7183801c
commit: 380eb427ddd067042539d5d5f2b7f88a7183801c
branch: 3.8
author: Benjamin Peterson <benjamin at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-10T21:02:38-07:00
summary:

[3.8] Typo fix: "throuhgh" should be "through". (GH-16713)

(cherry picked from commit a8e0d3141e271b3c0fbb7399a911f0c3aa567e30)

Co-authored-by: Hansraj Das <raj.das.136 at gmail.com>

files:
M Python/condvar.h

diff --git a/Python/condvar.h b/Python/condvar.h
index f54adc5ea8e82..8cba19b84612d 100644
--- a/Python/condvar.h
+++ b/Python/condvar.h
@@ -178,7 +178,7 @@ _PyCOND_WAIT_MS(PyCOND_T *cv, PyMUTEX_T *cs, DWORD ms)
          * just means an extra spurious wakeup for a waiting thread.
          * ('waiting' corresponds to the semaphore's "negative" count and
          * we may end up with e.g. (waiting == -1 && sem.count == 1).  When
-         * a new thread comes along, it will pass right throuhgh, having
+         * a new thread comes along, it will pass right through, having
          * adjusted it to (waiting == 0 && sem.count == 0).
          */
 


From webhook-mailer at python.org  Fri Oct 11 04:43:19 2019
From: webhook-mailer at python.org (Inada Naoki)
Date: Fri, 11 Oct 2019 08:43:19 -0000
Subject: [Python-checkins] Fix strict-aliasing rules errors on gcc 4.8.5.
 (GH-16714)
Message-ID: <mailman.101.1570783400.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/c39d1ddc012987e2159a997e27665d2d579c0ce0
commit: c39d1ddc012987e2159a997e27665d2d579c0ce0
branch: master
author: Dong-hee Na <donghee.na92 at gmail.com>
committer: Inada Naoki <songofacandy at gmail.com>
date: 2019-10-11T17:43:11+09:00
summary:

Fix strict-aliasing rules errors on gcc 4.8.5. (GH-16714)

files:
M Objects/dictobject.c

diff --git a/Objects/dictobject.c b/Objects/dictobject.c
index 99908a8a375d2..b496350d47093 100644
--- a/Objects/dictobject.c
+++ b/Objects/dictobject.c
@@ -1133,7 +1133,7 @@ insert_to_emptydict(PyDictObject *mp, PyObject *key, Py_hash_t hash,
     MAINTAIN_TRACKING(mp, key, value);
 
     size_t hashpos = (size_t)hash & (PyDict_MINSIZE-1);
-    PyDictKeyEntry *ep = &DK_ENTRIES(mp->ma_keys)[0];
+    PyDictKeyEntry *ep = DK_ENTRIES(mp->ma_keys);
     dictkeys_set_index(mp->ma_keys, hashpos, 0);
     ep->me_key = key;
     ep->me_hash = hash;


From webhook-mailer at python.org  Fri Oct 11 04:44:48 2019
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Fri, 11 Oct 2019 08:44:48 -0000
Subject: [Python-checkins] bpo-38442: Remove an execution bit from
 Doc/whatsnew/3.8.rst. (GH-16715)
Message-ID: <mailman.102.1570783489.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/cbb548130caf24275fa16100bccc537570cb46df
commit: cbb548130caf24275fa16100bccc537570cb46df
branch: master
author: Serhiy Storchaka <storchaka at gmail.com>
committer: GitHub <noreply at github.com>
date: 2019-10-11T11:44:42+03:00
summary:

bpo-38442: Remove an execution bit from Doc/whatsnew/3.8.rst. (GH-16715)

files:
M Doc/whatsnew/3.8.rst

diff --git a/Doc/whatsnew/3.8.rst b/Doc/whatsnew/3.8.rst
old mode 100755
new mode 100644


From webhook-mailer at python.org  Fri Oct 11 09:36:56 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Fri, 11 Oct 2019 13:36:56 -0000
Subject: [Python-checkins] bpo-38347: find pathfix for Python scripts whose
 name contain a '-' (GH-16536)
Message-ID: <mailman.103.1570801018.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/2b7dc40b2af6578181808ba73c1533fc114e55df
commit: 2b7dc40b2af6578181808ba73c1533fc114e55df
branch: master
author: Ruediger Pluem <r.pluem at gmx.de>
committer: Victor Stinner <vstinner at python.org>
date: 2019-10-11T15:36:50+02:00
summary:

bpo-38347: find pathfix for Python scripts whose name contain a '-' (GH-16536)

 pathfix.py: Assume all files that end on '.py' are Python scripts when working recursively.

files:
A Misc/NEWS.d/next/Tools-Demos/2019-10-02-09-48-42.bpo-38347.2Tq5D1.rst
M Lib/test/test_tools/test_pathfix.py
M Tools/scripts/pathfix.py

diff --git a/Lib/test/test_tools/test_pathfix.py b/Lib/test/test_tools/test_pathfix.py
index a879924913445..ec361178e6d81 100644
--- a/Lib/test/test_tools/test_pathfix.py
+++ b/Lib/test/test_tools/test_pathfix.py
@@ -14,25 +14,34 @@ class TestPathfixFunctional(unittest.TestCase):
     script = os.path.join(scriptsdir, 'pathfix.py')
 
     def setUp(self):
-        self.temp_file = support.TESTFN
         self.addCleanup(support.unlink, support.TESTFN)
 
-    def pathfix(self, shebang, pathfix_flags, exitcode=0, stdout='', stderr=''):
-        with open(self.temp_file, 'w', encoding='utf8') as f:
+    def pathfix(self, shebang, pathfix_flags, exitcode=0, stdout='', stderr='',
+                directory=''):
+        if directory:
+            # bpo-38347: Test filename should contain lowercase, uppercase,
+            # "-", "_" and digits.
+            filename = os.path.join(directory, 'script-A_1.py')
+            pathfix_arg = directory
+        else:
+            filename = support.TESTFN
+            pathfix_arg = filename
+
+        with open(filename, 'w', encoding='utf8') as f:
             f.write(f'{shebang}\n' + 'print("Hello world")\n')
 
         proc = subprocess.run(
             [sys.executable, self.script,
-             *pathfix_flags, '-n', self.temp_file],
+             *pathfix_flags, '-n', pathfix_arg],
             capture_output=True, text=1)
 
         if stdout == '' and proc.returncode == 0:
-            stdout = f'{self.temp_file}: updating\n'
+            stdout = f'{filename}: updating\n'
         self.assertEqual(proc.returncode, exitcode, proc)
         self.assertEqual(proc.stdout, stdout, proc)
         self.assertEqual(proc.stderr, stderr, proc)
 
-        with open(self.temp_file, 'r', encoding='utf8') as f:
+        with open(filename, 'r', encoding='utf8') as f:
             output = f.read()
 
         lines = output.split('\n')
@@ -44,6 +53,19 @@ def pathfix(self, shebang, pathfix_flags, exitcode=0, stdout='', stderr=''):
 
         return new_shebang
 
+    def test_recursive(self):
+        tmpdir = support.TESTFN + '.d'
+        self.addCleanup(support.rmtree, tmpdir)
+        os.mkdir(tmpdir)
+        expected_stderr = f"recursedown('{os.path.basename(tmpdir)}')\n"
+        self.assertEqual(
+            self.pathfix(
+                '#! /usr/bin/env python',
+                ['-i', '/usr/bin/python3'],
+                directory=tmpdir,
+                stderr=expected_stderr),
+            '#! /usr/bin/python3')
+
     def test_pathfix(self):
         self.assertEqual(
             self.pathfix(
diff --git a/Misc/NEWS.d/next/Tools-Demos/2019-10-02-09-48-42.bpo-38347.2Tq5D1.rst b/Misc/NEWS.d/next/Tools-Demos/2019-10-02-09-48-42.bpo-38347.2Tq5D1.rst
new file mode 100644
index 0000000000000..ae64a319b5756
--- /dev/null
+++ b/Misc/NEWS.d/next/Tools-Demos/2019-10-02-09-48-42.bpo-38347.2Tq5D1.rst
@@ -0,0 +1 @@
+pathfix.py: Assume all files that end on '.py' are Python scripts when working recursively.
diff --git a/Tools/scripts/pathfix.py b/Tools/scripts/pathfix.py
index 237a3d96e9fab..d252321a21a17 100755
--- a/Tools/scripts/pathfix.py
+++ b/Tools/scripts/pathfix.py
@@ -89,11 +89,8 @@ def main():
     sys.exit(bad)
 
 
-ispythonprog = re.compile(r'^[a-zA-Z0-9_]+\.py$')
-
-
 def ispython(name):
-    return bool(ispythonprog.match(name))
+    return name.endswith('.py')
 
 
 def recursedown(dirname):


From webhook-mailer at python.org  Fri Oct 11 09:56:52 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Fri, 11 Oct 2019 13:56:52 -0000
Subject: [Python-checkins] bpo-38347: find pathfix for Python scripts whose
 name contain a '-' (GH-16536)
Message-ID: <mailman.104.1570802214.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/73665b41470532f873315d5ae39906160192320d
commit: 73665b41470532f873315d5ae39906160192320d
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-11T06:56:49-07:00
summary:

bpo-38347: find pathfix for Python scripts whose name contain a '-' (GH-16536)


 pathfix.py: Assume all files that end on '.py' are Python scripts when working recursively.
(cherry picked from commit 2b7dc40b2af6578181808ba73c1533fc114e55df)

Co-authored-by: Ruediger Pluem <r.pluem at gmx.de>

files:
A Misc/NEWS.d/next/Tools-Demos/2019-10-02-09-48-42.bpo-38347.2Tq5D1.rst
M Lib/test/test_tools/test_pathfix.py
M Tools/scripts/pathfix.py

diff --git a/Lib/test/test_tools/test_pathfix.py b/Lib/test/test_tools/test_pathfix.py
index a879924913445..ec361178e6d81 100644
--- a/Lib/test/test_tools/test_pathfix.py
+++ b/Lib/test/test_tools/test_pathfix.py
@@ -14,25 +14,34 @@ class TestPathfixFunctional(unittest.TestCase):
     script = os.path.join(scriptsdir, 'pathfix.py')
 
     def setUp(self):
-        self.temp_file = support.TESTFN
         self.addCleanup(support.unlink, support.TESTFN)
 
-    def pathfix(self, shebang, pathfix_flags, exitcode=0, stdout='', stderr=''):
-        with open(self.temp_file, 'w', encoding='utf8') as f:
+    def pathfix(self, shebang, pathfix_flags, exitcode=0, stdout='', stderr='',
+                directory=''):
+        if directory:
+            # bpo-38347: Test filename should contain lowercase, uppercase,
+            # "-", "_" and digits.
+            filename = os.path.join(directory, 'script-A_1.py')
+            pathfix_arg = directory
+        else:
+            filename = support.TESTFN
+            pathfix_arg = filename
+
+        with open(filename, 'w', encoding='utf8') as f:
             f.write(f'{shebang}\n' + 'print("Hello world")\n')
 
         proc = subprocess.run(
             [sys.executable, self.script,
-             *pathfix_flags, '-n', self.temp_file],
+             *pathfix_flags, '-n', pathfix_arg],
             capture_output=True, text=1)
 
         if stdout == '' and proc.returncode == 0:
-            stdout = f'{self.temp_file}: updating\n'
+            stdout = f'{filename}: updating\n'
         self.assertEqual(proc.returncode, exitcode, proc)
         self.assertEqual(proc.stdout, stdout, proc)
         self.assertEqual(proc.stderr, stderr, proc)
 
-        with open(self.temp_file, 'r', encoding='utf8') as f:
+        with open(filename, 'r', encoding='utf8') as f:
             output = f.read()
 
         lines = output.split('\n')
@@ -44,6 +53,19 @@ def pathfix(self, shebang, pathfix_flags, exitcode=0, stdout='', stderr=''):
 
         return new_shebang
 
+    def test_recursive(self):
+        tmpdir = support.TESTFN + '.d'
+        self.addCleanup(support.rmtree, tmpdir)
+        os.mkdir(tmpdir)
+        expected_stderr = f"recursedown('{os.path.basename(tmpdir)}')\n"
+        self.assertEqual(
+            self.pathfix(
+                '#! /usr/bin/env python',
+                ['-i', '/usr/bin/python3'],
+                directory=tmpdir,
+                stderr=expected_stderr),
+            '#! /usr/bin/python3')
+
     def test_pathfix(self):
         self.assertEqual(
             self.pathfix(
diff --git a/Misc/NEWS.d/next/Tools-Demos/2019-10-02-09-48-42.bpo-38347.2Tq5D1.rst b/Misc/NEWS.d/next/Tools-Demos/2019-10-02-09-48-42.bpo-38347.2Tq5D1.rst
new file mode 100644
index 0000000000000..ae64a319b5756
--- /dev/null
+++ b/Misc/NEWS.d/next/Tools-Demos/2019-10-02-09-48-42.bpo-38347.2Tq5D1.rst
@@ -0,0 +1 @@
+pathfix.py: Assume all files that end on '.py' are Python scripts when working recursively.
diff --git a/Tools/scripts/pathfix.py b/Tools/scripts/pathfix.py
index 237a3d96e9fab..d252321a21a17 100755
--- a/Tools/scripts/pathfix.py
+++ b/Tools/scripts/pathfix.py
@@ -89,11 +89,8 @@ def main():
     sys.exit(bad)
 
 
-ispythonprog = re.compile(r'^[a-zA-Z0-9_]+\.py$')
-
-
 def ispython(name):
-    return bool(ispythonprog.match(name))
+    return name.endswith('.py')
 
 
 def recursedown(dirname):


From webhook-mailer at python.org  Fri Oct 11 11:37:10 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Fri, 11 Oct 2019 15:37:10 -0000
Subject: [Python-checkins] bpo-38347: find pathfix for Python scripts whose
 name contain a '-' (GH-16536) (GH-16719)
Message-ID: <mailman.105.1570808231.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/ed189ce49423f15ba8774ff174c15d03d12bc807
commit: ed189ce49423f15ba8774ff174c15d03d12bc807
branch: 3.7
author: Victor Stinner <vstinner at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-11T17:36:56+02:00
summary:

bpo-38347: find pathfix for Python scripts whose name contain a '-' (GH-16536) (GH-16719)

pathfix.py: Assume all files that end on '.py' are Python scripts when working recursively.

(cherry picked from commit 2b7dc40b2af6578181808ba73c1533fc114e55df)

files:
A Misc/NEWS.d/next/Tools-Demos/2019-10-02-09-48-42.bpo-38347.2Tq5D1.rst
M Tools/scripts/pathfix.py

diff --git a/Misc/NEWS.d/next/Tools-Demos/2019-10-02-09-48-42.bpo-38347.2Tq5D1.rst b/Misc/NEWS.d/next/Tools-Demos/2019-10-02-09-48-42.bpo-38347.2Tq5D1.rst
new file mode 100644
index 0000000000000..ae64a319b5756
--- /dev/null
+++ b/Misc/NEWS.d/next/Tools-Demos/2019-10-02-09-48-42.bpo-38347.2Tq5D1.rst
@@ -0,0 +1 @@
+pathfix.py: Assume all files that end on '.py' are Python scripts when working recursively.
diff --git a/Tools/scripts/pathfix.py b/Tools/scripts/pathfix.py
index c5bf984306a35..28ee428a3a1d8 100755
--- a/Tools/scripts/pathfix.py
+++ b/Tools/scripts/pathfix.py
@@ -70,9 +70,9 @@ def main():
             if fix(arg): bad = 1
     sys.exit(bad)
 
-ispythonprog = re.compile(r'^[a-zA-Z0-9_]+\.py$')
+
 def ispython(name):
-    return bool(ispythonprog.match(name))
+    return name.endswith('.py')
 
 def recursedown(dirname):
     dbg('recursedown(%r)\n' % (dirname,))


From webhook-mailer at python.org  Fri Oct 11 12:30:14 2019
From: webhook-mailer at python.org (Zachary Ware)
Date: Fri, 11 Oct 2019 16:30:14 -0000
Subject: [Python-checkins] Update badges in README.rst (GH-16709)
Message-ID: <mailman.106.1570811416.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/50290e3973d8e573863860cac77a9b5b5781a536
commit: 50290e3973d8e573863860cac77a9b5b5781a536
branch: 3.8
author: Zachary Ware <zachary.ware at gmail.com>
committer: GitHub <noreply at github.com>
date: 2019-10-11T11:30:03-05:00
summary:

Update badges in README.rst (GH-16709)

We no longer use AppVeyor on 3.x, and others were still pointed at master.

files:
M README.rst

diff --git a/README.rst b/README.rst
index 6482de800f230..48376362ecfa1 100644
--- a/README.rst
+++ b/README.rst
@@ -1,21 +1,17 @@
 This is Python version 3.8.0 release candidate 1
 ================================================
 
-.. image:: https://travis-ci.org/python/cpython.svg?branch=master
+.. image:: https://travis-ci.org/python/cpython.svg?branch=3.8
    :alt: CPython build status on Travis CI
-   :target: https://travis-ci.org/python/cpython
+   :target: https://travis-ci.org/python/cpython/branches
 
-.. image:: https://ci.appveyor.com/api/projects/status/4mew1a93xdkbf5ua/branch/master?svg=true
-   :alt: CPython build status on Appveyor
-   :target: https://ci.appveyor.com/project/python/cpython/branch/master
-
-.. image:: https://dev.azure.com/python/cpython/_apis/build/status/Azure%20Pipelines%20CI?branchName=master
+.. image:: https://dev.azure.com/python/cpython/_apis/build/status/Azure%20Pipelines%20CI?branchName=3.8
    :alt: CPython build status on Azure DevOps
-   :target: https://dev.azure.com/python/cpython/_build/latest?definitionId=4&branchName=master
+   :target: https://dev.azure.com/python/cpython/_build/latest?definitionId=4&branchName=3.8
 
-.. image:: https://codecov.io/gh/python/cpython/branch/master/graph/badge.svg
+.. image:: https://codecov.io/gh/python/cpython/branch/3.8/graph/badge.svg
    :alt: CPython code coverage on Codecov
-   :target: https://codecov.io/gh/python/cpython
+   :target: https://codecov.io/gh/python/cpython/branch/3.8
 
 .. image:: https://img.shields.io/badge/zulip-join_chat-brightgreen.svg
    :alt: Python Zulip chat


From webhook-mailer at python.org  Fri Oct 11 12:30:42 2019
From: webhook-mailer at python.org (Zachary Ware)
Date: Fri, 11 Oct 2019 16:30:42 -0000
Subject: [Python-checkins] Update badges in README.rst (GH-16710)
Message-ID: <mailman.107.1570811443.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/dd63ee26a8284051b916a4ffd3a59e3d26aaec00
commit: dd63ee26a8284051b916a4ffd3a59e3d26aaec00
branch: 3.7
author: Zachary Ware <zachary.ware at gmail.com>
committer: GitHub <noreply at github.com>
date: 2019-10-11T11:30:37-05:00
summary:

Update badges in README.rst (GH-16710)

We no longer use AppVeyor in 3.x, and others were still pointed at master.

files:
M README.rst

diff --git a/README.rst b/README.rst
index 74417e148e835..aaacaa0c52a0d 100644
--- a/README.rst
+++ b/README.rst
@@ -1,21 +1,17 @@
 This is Python version 3.7.5 release candidate+
 ===============================================
 
-.. image:: https://travis-ci.org/python/cpython.svg?branch=master
+.. image:: https://travis-ci.org/python/cpython.svg?branch=3.7
    :alt: CPython build status on Travis CI
-   :target: https://travis-ci.org/python/cpython
-
-.. image:: https://ci.appveyor.com/api/projects/status/4mew1a93xdkbf5ua/branch/master?svg=true
-   :alt: CPython build status on Appveyor
-   :target: https://ci.appveyor.com/project/python/cpython/branch/master
+   :target: https://travis-ci.org/python/cpython/branches
 
 .. image:: https://dev.azure.com/python/cpython/_apis/build/status/Azure%20Pipelines%20CI?branchName=3.7
    :alt: CPython build status on Azure Pipelines
    :target: https://dev.azure.com/python/cpython/_build/latest?definitionId=4&branchName=3.7
 
-.. image:: https://codecov.io/gh/python/cpython/branch/master/graph/badge.svg
+.. image:: https://codecov.io/gh/python/cpython/branch/3.7/graph/badge.svg
    :alt: CPython code coverage on Codecov
-   :target: https://codecov.io/gh/python/cpython
+   :target: https://codecov.io/gh/python/cpython/branch/3.7
 
 Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011,
 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019 Python Software Foundation.  All


From webhook-mailer at python.org  Fri Oct 11 13:27:23 2019
From: webhook-mailer at python.org (Lisa Roach)
Date: Fri, 11 Oct 2019 17:27:23 -0000
Subject: [Python-checkins] Fix typo: equivalent code of `async with cond`
 (GH-11681) (GH-16720)
Message-ID: <mailman.108.1570814844.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/1845997dd9328a79da6395a0c23077c42102e1b0
commit: 1845997dd9328a79da6395a0c23077c42102e1b0
branch: 3.7
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Lisa Roach <lisaroach14 at gmail.com>
date: 2019-10-11T10:27:16-07:00
summary:

Fix typo: equivalent code of `async with cond` (GH-11681) (GH-16720)

files:
M Doc/library/asyncio-sync.rst

diff --git a/Doc/library/asyncio-sync.rst b/Doc/library/asyncio-sync.rst
index 993bd13fb0301..79f6b02d85e2e 100644
--- a/Doc/library/asyncio-sync.rst
+++ b/Doc/library/asyncio-sync.rst
@@ -187,11 +187,11 @@ Condition
        cond = asyncio.Condition()
 
        # ... later
-       await lock.acquire()
+       await cond.acquire()
        try:
            await cond.wait()
        finally:
-           lock.release()
+           cond.release()
 
    .. coroutinemethod:: acquire()
 


From webhook-mailer at python.org  Fri Oct 11 16:29:09 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Fri, 11 Oct 2019 20:29:09 -0000
Subject: [Python-checkins] [3.7] bpo-38395: Fix ownership in weakref.proxy
 methods (GH-16632) (GH-16663)
Message-ID: <mailman.109.1570825750.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/193366e25c4f84a58d2f6c6c577fd9f0143dc6e1
commit: 193366e25c4f84a58d2f6c6c577fd9f0143dc6e1
branch: 3.7
author: Pablo Galindo <Pablogsal at gmail.com>
committer: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
date: 2019-10-11T13:28:49-07:00
summary:

[3.7] bpo-38395: Fix ownership in weakref.proxy methods (GH-16632) (GH-16663)



The implementation of weakref.proxy's methods call back into the Python
API using a borrowed references of the weakly referenced object
(acquired via PyWeakref_GET_OBJECT). This API call may delete the last
reference to the object (either directly or via GC), leaving a dangling
pointer, which can be subsequently dereferenced.

To fix this, claim a temporary ownership of the referenced object when
calling the appropriate method. Some functions because at the moment they
do not need to access the borrowed referent, but to protect against
future changes to these functions, ownership need to be fixed in
all potentially affected methods..
(cherry picked from commit 10cd00a9e3c22af37c748ea5a417f6fb66601e21)

Co-authored-by: Pablo Galindo <Pablogsal at gmail.com>





https://bugs.python.org/issue38395

files:
A Misc/NEWS.d/next/C API/2019-10-08-01-23-24.bpo-38395.MJ6Ey9.rst
M Lib/test/test_weakref.py
M Objects/weakrefobject.c

diff --git a/Lib/test/test_weakref.py b/Lib/test/test_weakref.py
index d3396fc9cf90c..14ec8efdc638e 100644
--- a/Lib/test/test_weakref.py
+++ b/Lib/test/test_weakref.py
@@ -376,6 +376,26 @@ class List(list): pass
         lyst = List()
         self.assertEqual(bool(weakref.proxy(lyst)), bool(lyst))
 
+    def test_proxy_iter(self):
+        # Test fails with a debug build of the interpreter
+        # (see bpo-38395).
+
+        obj = None
+
+        class MyObj:
+            def __iter__(self):
+                nonlocal obj
+                del obj
+                return NotImplemented
+
+        obj = MyObj()
+        p = weakref.proxy(obj)
+        with self.assertRaises(TypeError):
+            # "blech" in p calls MyObj.__iter__ through the proxy,
+            # without keeping a reference to the real object, so it
+            # can be killed in the middle of the call
+            "blech" in p
+
     def test_getweakrefcount(self):
         o = C()
         ref1 = weakref.ref(o)
diff --git a/Misc/NEWS.d/next/C API/2019-10-08-01-23-24.bpo-38395.MJ6Ey9.rst b/Misc/NEWS.d/next/C API/2019-10-08-01-23-24.bpo-38395.MJ6Ey9.rst
new file mode 100644
index 0000000000000..4bc30600ca758
--- /dev/null
+++ b/Misc/NEWS.d/next/C API/2019-10-08-01-23-24.bpo-38395.MJ6Ey9.rst	
@@ -0,0 +1,3 @@
+Fix a crash in :class:`weakref.proxy` objects due to incorrect lifetime
+management when calling some associated methods that may delete the last
+reference to object being referenced by the proxy. Patch by Pablo Galindo.
diff --git a/Objects/weakrefobject.c b/Objects/weakrefobject.c
index f600179d0e9ba..4bf662ced1de1 100644
--- a/Objects/weakrefobject.c
+++ b/Objects/weakrefobject.c
@@ -145,11 +145,14 @@ weakref_hash(PyWeakReference *self)
 {
     if (self->hash != -1)
         return self->hash;
-    if (PyWeakref_GET_OBJECT(self) == Py_None) {
+    PyObject* obj = PyWeakref_GET_OBJECT(self);
+    if (obj == Py_None) {
         PyErr_SetString(PyExc_TypeError, "weak object has gone away");
         return -1;
     }
-    self->hash = PyObject_Hash(PyWeakref_GET_OBJECT(self));
+    Py_INCREF(obj);
+    self->hash = PyObject_Hash(obj);
+    Py_DECREF(obj);
     return self->hash;
 }
 
@@ -159,11 +162,18 @@ weakref_repr(PyWeakReference *self)
 {
     PyObject *name, *repr;
     _Py_IDENTIFIER(__name__);
+    PyObject* obj = PyWeakref_GET_OBJECT(self);
 
-    if (PyWeakref_GET_OBJECT(self) == Py_None)
+    if (obj == Py_None) {
         return PyUnicode_FromFormat("<weakref at %p; dead>", self);
+    }
+
+    Py_INCREF(obj);
+    if (_PyObject_LookupAttrId(obj, &PyId___name__, &name) < 0) {
+        Py_DECREF(obj);
+        return NULL;
+    }
 
-    name = _PyObject_GetAttrId(PyWeakref_GET_OBJECT(self), &PyId___name__);
     if (name == NULL || !PyUnicode_Check(name)) {
         if (name == NULL)
             PyErr_Clear();
@@ -171,16 +181,17 @@ weakref_repr(PyWeakReference *self)
             "<weakref at %p; to '%s' at %p>",
             self,
             Py_TYPE(PyWeakref_GET_OBJECT(self))->tp_name,
-            PyWeakref_GET_OBJECT(self));
+            obj);
     }
     else {
         repr = PyUnicode_FromFormat(
             "<weakref at %p; to '%s' at %p (%U)>",
             self,
             Py_TYPE(PyWeakref_GET_OBJECT(self))->tp_name,
-            PyWeakref_GET_OBJECT(self),
+            obj,
             name);
     }
+    Py_DECREF(obj);
     Py_XDECREF(name);
     return repr;
 }
@@ -207,8 +218,14 @@ weakref_richcompare(PyWeakReference* self, PyWeakReference* other, int op)
         else
             Py_RETURN_FALSE;
     }
-    return PyObject_RichCompare(PyWeakref_GET_OBJECT(self),
-                                PyWeakref_GET_OBJECT(other), op);
+    PyObject* obj = PyWeakref_GET_OBJECT(self);
+    PyObject* other_obj = PyWeakref_GET_OBJECT(other);
+    Py_INCREF(obj);
+    Py_INCREF(other_obj);
+    PyObject* res = PyObject_RichCompare(obj, other_obj, op);
+    Py_DECREF(obj);
+    Py_DECREF(other_obj);
+    return res;
 }
 
 /* Given the head of an object's list of weak references, extract the
@@ -415,18 +432,14 @@ proxy_checkref(PyWeakReference *proxy)
             o = PyWeakref_GET_OBJECT(o); \
         }
 
-#define UNWRAP_I(o) \
-        if (PyWeakref_CheckProxy(o)) { \
-            if (!proxy_checkref((PyWeakReference *)o)) \
-                return -1; \
-            o = PyWeakref_GET_OBJECT(o); \
-        }
-
 #define WRAP_UNARY(method, generic) \
     static PyObject * \
     method(PyObject *proxy) { \
         UNWRAP(proxy); \
-        return generic(proxy); \
+        Py_INCREF(proxy); \
+        PyObject* res = generic(proxy); \
+        Py_DECREF(proxy); \
+        return res; \
     }
 
 #define WRAP_BINARY(method, generic) \
@@ -434,7 +447,12 @@ proxy_checkref(PyWeakReference *proxy)
     method(PyObject *x, PyObject *y) { \
         UNWRAP(x); \
         UNWRAP(y); \
-        return generic(x, y); \
+        Py_INCREF(x); \
+        Py_INCREF(y); \
+        PyObject* res = generic(x, y); \
+        Py_DECREF(x); \
+        Py_DECREF(y); \
+        return res; \
     }
 
 /* Note that the third arg needs to be checked for NULL since the tp_call
@@ -447,7 +465,14 @@ proxy_checkref(PyWeakReference *proxy)
         UNWRAP(v); \
         if (w != NULL) \
             UNWRAP(w); \
-        return generic(proxy, v, w); \
+        Py_INCREF(proxy); \
+        Py_INCREF(v); \
+        Py_XINCREF(w); \
+        PyObject* res = generic(proxy, v, w); \
+        Py_DECREF(proxy); \
+        Py_DECREF(v); \
+        Py_XDECREF(w); \
+        return res; \
     }
 
 #define WRAP_METHOD(method, special) \
@@ -455,7 +480,10 @@ proxy_checkref(PyWeakReference *proxy)
     method(PyObject *proxy) { \
             _Py_IDENTIFIER(special); \
             UNWRAP(proxy); \
-                return _PyObject_CallMethodId(proxy, &PyId_##special, NULL); \
+            Py_INCREF(proxy); \
+            PyObject* res = _PyObject_CallMethodId(proxy, &PyId_##special, NULL); \
+            Py_DECREF(proxy); \
+            return res; \
         }
 
 
@@ -481,7 +509,11 @@ proxy_setattr(PyWeakReference *proxy, PyObject *name, PyObject *value)
 {
     if (!proxy_checkref(proxy))
         return -1;
-    return PyObject_SetAttr(PyWeakref_GET_OBJECT(proxy), name, value);
+    PyObject *obj = PyWeakref_GET_OBJECT(proxy);
+    Py_INCREF(obj);
+    int res = PyObject_SetAttr(obj, name, value);
+    Py_DECREF(obj);
+    return res;
 }
 
 static PyObject *
@@ -530,9 +562,13 @@ static int
 proxy_bool(PyWeakReference *proxy)
 {
     PyObject *o = PyWeakref_GET_OBJECT(proxy);
-    if (!proxy_checkref(proxy))
+    if (!proxy_checkref(proxy)) {
         return -1;
-    return PyObject_IsTrue(o);
+    }
+    Py_INCREF(o);
+    int res = PyObject_IsTrue(o);
+    Py_DECREF(o);
+    return res;
 }
 
 static void
@@ -551,9 +587,13 @@ proxy_contains(PyWeakReference *proxy, PyObject *value)
 {
     if (!proxy_checkref(proxy))
         return -1;
-    return PySequence_Contains(PyWeakref_GET_OBJECT(proxy), value);
-}
 
+    PyObject *obj = PyWeakref_GET_OBJECT(proxy);
+    Py_INCREF(obj);
+    int res = PySequence_Contains(obj, value);
+    Py_DECREF(obj);
+    return res;
+}
 
 /* mapping slots */
 
@@ -562,7 +602,12 @@ proxy_length(PyWeakReference *proxy)
 {
     if (!proxy_checkref(proxy))
         return -1;
-    return PyObject_Length(PyWeakref_GET_OBJECT(proxy));
+
+    PyObject *obj = PyWeakref_GET_OBJECT(proxy);
+    Py_INCREF(obj);
+    Py_ssize_t res = PyObject_Length(obj);
+    Py_DECREF(obj);
+    return res;
 }
 
 WRAP_BINARY(proxy_getitem, PyObject_GetItem)
@@ -573,10 +618,16 @@ proxy_setitem(PyWeakReference *proxy, PyObject *key, PyObject *value)
     if (!proxy_checkref(proxy))
         return -1;
 
-    if (value == NULL)
-        return PyObject_DelItem(PyWeakref_GET_OBJECT(proxy), key);
-    else
-        return PyObject_SetItem(PyWeakref_GET_OBJECT(proxy), key, value);
+    PyObject *obj = PyWeakref_GET_OBJECT(proxy);
+    Py_INCREF(obj);
+    int res;
+    if (value == NULL) {
+        res = PyObject_DelItem(obj, key);
+    } else {
+        res = PyObject_SetItem(obj, key, value);
+    }
+    Py_DECREF(obj);
+    return res;
 }
 
 /* iterator slots */
@@ -586,7 +637,11 @@ proxy_iter(PyWeakReference *proxy)
 {
     if (!proxy_checkref(proxy))
         return NULL;
-    return PyObject_GetIter(PyWeakref_GET_OBJECT(proxy));
+    PyObject *obj = PyWeakref_GET_OBJECT(proxy);
+    Py_INCREF(obj);
+    PyObject* res = PyObject_GetIter(obj);
+    Py_DECREF(obj);
+    return res;
 }
 
 static PyObject *
@@ -594,7 +649,12 @@ proxy_iternext(PyWeakReference *proxy)
 {
     if (!proxy_checkref(proxy))
         return NULL;
-    return PyIter_Next(PyWeakref_GET_OBJECT(proxy));
+
+    PyObject *obj = PyWeakref_GET_OBJECT(proxy);
+    Py_INCREF(obj);
+    PyObject* res = PyIter_Next(obj);
+    Py_DECREF(obj);
+    return res;
 }
 
 


From webhook-mailer at python.org  Fri Oct 11 17:09:52 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Fri, 11 Oct 2019 21:09:52 -0000
Subject: [Python-checkins] Fix typo in the "Porting to Python 3.8" section.
 (GH-16435)
Message-ID: <mailman.110.1570828192.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/294c5229b12c3c6150719910846bdcdba8ed4f3e
commit: 294c5229b12c3c6150719910846bdcdba8ed4f3e
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-11T14:09:44-07:00
summary:

Fix typo in the "Porting to Python 3.8" section. (GH-16435)

(cherry picked from commit dd6117c6d7859fee57751593cd56f0862131de8b)

Co-authored-by: bariod <35639254+bariod at users.noreply.github.com>

files:
M Doc/whatsnew/3.8.rst

diff --git a/Doc/whatsnew/3.8.rst b/Doc/whatsnew/3.8.rst
index 0995cb3b91196..4cab2f7af118d 100644
--- a/Doc/whatsnew/3.8.rst
+++ b/Doc/whatsnew/3.8.rst
@@ -1531,7 +1531,7 @@ Changes in Python behavior
   :class:`int`, :class:`float`, :class:`complex` and few classes from
   the standard library.  They now inherit ``__str__()`` from :class:`object`.
   As result, defining the ``__repr__()`` method in the subclass of these
-  classes will affect they string representation.
+  classes will affect their string representation.
   (Contributed by Serhiy Storchaka in :issue:`36793`.)
 
 * On AIX, :attr:`sys.platform` doesn't contain the major version anymore.


From webhook-mailer at python.org  Sat Oct 12 01:41:51 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Sat, 12 Oct 2019 05:41:51 -0000
Subject: [Python-checkins] bpo-38449: Revert "bpo-22347: Update
 mimetypes.guess_type to allow oper parsing of URLs (GH-15522)" (GH-16724)
Message-ID: <mailman.111.1570858912.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/19a3d873005e5730eeabdc394c961e93f2ec02f0
commit: 19a3d873005e5730eeabdc394c961e93f2ec02f0
branch: master
author: Abhilash Raj <maxking at users.noreply.github.com>
committer: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
date: 2019-10-11T22:41:35-07:00
summary:

bpo-38449: Revert "bpo-22347: Update mimetypes.guess_type to allow oper parsing of URLs (GH-15522)" (GH-16724)



This reverts commit 87bd2071c756188b6cd577889fb1682831142ceb.



https://bugs.python.org/issue38449

files:
A Misc/NEWS.d/next/Library/2019-10-11-18-49-00.bpo-38449.9TWMlz.rst
D Misc/NEWS.d/next/Library/2019-08-27-01-03-26.bpo-22347._TRpYr.rst
M Lib/mimetypes.py
M Lib/test/test_mimetypes.py
M Lib/test/test_urllib2.py

diff --git a/Lib/mimetypes.py b/Lib/mimetypes.py
index 9fdd6ebd6348f..9b42bf6dd2ca7 100644
--- a/Lib/mimetypes.py
+++ b/Lib/mimetypes.py
@@ -114,8 +114,7 @@ def guess_type(self, url, strict=True):
         but non-standard types.
         """
         url = os.fspath(url)
-        p = urllib.parse.urlparse(url)
-        scheme, url = p.scheme, p.path
+        scheme, url = urllib.parse._splittype(url)
         if scheme == 'data':
             # syntax of data URLs:
             # dataurl   := "data:" [ mediatype ] [ ";base64" ] "," data
diff --git a/Lib/test/test_mimetypes.py b/Lib/test/test_mimetypes.py
index 7761c3fe867a7..bfd5eeedaa77b 100644
--- a/Lib/test/test_mimetypes.py
+++ b/Lib/test/test_mimetypes.py
@@ -51,14 +51,6 @@ def test_non_standard_types(self):
         eq(self.db.guess_type('foo.xul', strict=False), ('text/xul', None))
         eq(self.db.guess_extension('image/jpg', strict=False), '.jpg')
 
-    def test_url(self):
-        result = self.db.guess_type('http://host.html')
-        msg = 'URL only has a host name, not a file'
-        self.assertSequenceEqual(result, (None, None), msg)
-        result = self.db.guess_type('http://example.com/host.html')
-        msg = 'Should be text/html'
-        self.assertSequenceEqual(result, ('text/html', None), msg)
-
     def test_guess_all_types(self):
         eq = self.assertEqual
         unless = self.assertTrue
diff --git a/Lib/test/test_urllib2.py b/Lib/test/test_urllib2.py
index 186a96765971a..8abedaac9850a 100644
--- a/Lib/test/test_urllib2.py
+++ b/Lib/test/test_urllib2.py
@@ -742,7 +742,7 @@ def connect_ftp(self, user, passwd, host, port, dirs,
              ["foo", "bar"], "", None),
             ("ftp://localhost/baz.gif;type=a",
              "localhost", ftplib.FTP_PORT, "", "", "A",
-             [], "baz.gif", "image/gif"),
+             [], "baz.gif", None),  # XXX really this should guess image/gif
             ]:
             req = Request(url)
             req.timeout = None
diff --git a/Misc/NEWS.d/next/Library/2019-08-27-01-03-26.bpo-22347._TRpYr.rst b/Misc/NEWS.d/next/Library/2019-08-27-01-03-26.bpo-22347._TRpYr.rst
deleted file mode 100644
index 1a3c19938217c..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-08-27-01-03-26.bpo-22347._TRpYr.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Update mimetypes.guess_type to allow proper parsing of URLs with only a host name.
-Patch by Dong-hee Na.
diff --git a/Misc/NEWS.d/next/Library/2019-10-11-18-49-00.bpo-38449.9TWMlz.rst b/Misc/NEWS.d/next/Library/2019-10-11-18-49-00.bpo-38449.9TWMlz.rst
new file mode 100644
index 0000000000000..f7b1dbf8d8951
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-10-11-18-49-00.bpo-38449.9TWMlz.rst
@@ -0,0 +1,2 @@
+Revert GH-15522, which introduces a regression in
+:meth:`mimetypes.guess_type` due to improper handling of filenames as urls.


From webhook-mailer at python.org  Sat Oct 12 08:09:04 2019
From: webhook-mailer at python.org (Larry Hastings)
Date: Sat, 12 Oct 2019 12:09:04 -0000
Subject: [Python-checkins] Fix docs, blurb release,
 pydoc-topics for 3.5.8rc2.
Message-ID: <mailman.112.1570882146.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/280a0f539a1db5bc548b031dca04af30b180f7c4
commit: 280a0f539a1db5bc548b031dca04af30b180f7c4
branch: 3.5
author: Larry Hastings <larry at hastings.org>
committer: Larry Hastings <larry at hastings.org>
date: 2019-10-12T12:07:42+01:00
summary:

Fix docs, blurb release, pydoc-topics for 3.5.8rc2.

files:
A Misc/NEWS.d/3.5.8rc2.rst
D Misc/NEWS.d/next/Library/2019-09-27-15-24-45.bpo-38216.-7yvZR.rst
D Misc/NEWS.d/next/Security/2019-09-23-21-02-46.bpo-38174.MeWuJd.rst
M Doc/tools/extensions/pyspecific.py
M Lib/pydoc_data/topics.py

diff --git a/Doc/tools/extensions/pyspecific.py b/Doc/tools/extensions/pyspecific.py
index f25c4bcd5c98e..88cbf2aab66e7 100644
--- a/Doc/tools/extensions/pyspecific.py
+++ b/Doc/tools/extensions/pyspecific.py
@@ -24,7 +24,7 @@
 from sphinx.builders import Builder
 from sphinx.util.nodes import split_explicit_title
 from sphinx.writers.html import HTMLTranslator
-from sphinx.writers.text import TextWriter
+from sphinx.writers.text import TextWriter, TextTranslator
 from sphinx.writers.latex import LaTeXTranslator
 from sphinx.domains.python import PyModulelevel, PyClassmember
 
@@ -286,8 +286,11 @@ def run(self):
 class PydocTopicsBuilder(Builder):
     name = 'pydoc-topics'
 
+    default_translator_class = TextTranslator
+
     def init(self):
         self.topics = {}
+        self.secnumbers = {}
 
     def get_outdated_docs(self):
         return 'all pydoc topics'
@@ -296,8 +299,12 @@ def get_target_uri(self, docname, typ=None):
         return ''  # no URIs
 
     def write(self, *ignored):
+        try:  # sphinx>=1.6
+            from sphinx.util import status_iterator
+        except ImportError:  # sphinx<1.6
+            status_iterator = self.status_iterator
         writer = TextWriter(self)
-        for label in self.status_iterator(pydoc_topic_labels,
+        for label in status_iterator(pydoc_topic_labels,
                                           'building topics... ',
                                           length=len(pydoc_topic_labels)):
             if label not in self.env.domaindata['std']['labels']:
diff --git a/Lib/pydoc_data/topics.py b/Lib/pydoc_data/topics.py
index 172b4f86ada36..31a8bd0876978 100644
--- a/Lib/pydoc_data/topics.py
+++ b/Lib/pydoc_data/topics.py
@@ -1,7 +1,6 @@
 # -*- coding: utf-8 -*-
-# Autogenerated by Sphinx on Mon Sep  9 12:15:17 2019
-topics = {'assert': '\n'
-           'The "assert" statement\n'
+# Autogenerated by Sphinx on Sat Oct 12 12:05:57 2019
+topics = {'assert': 'The "assert" statement\n'
            '**********************\n'
            '\n'
            'Assert statements are a convenient way to insert debugging '
@@ -39,8 +38,7 @@
            'Assignments to "__debug__" are illegal.  The value for the '
            'built-in\n'
            'variable is determined when the interpreter starts.\n',
- 'assignment': '\n'
-               'Assignment statements\n'
+ 'assignment': 'Assignment statements\n'
                '*********************\n'
                '\n'
                'Assignment statements are used to (re)bind names to values and '
@@ -58,8 +56,9 @@
                '              | slicing\n'
                '              | "*" target\n'
                '\n'
-               '(See section *Primaries* for the syntax definitions for\n'
-               '*attributeref*, *subscription*, and *slicing*.)\n'
+               '(See section Primaries for the syntax definitions for '
+               '*attributeref*,\n'
+               '*subscription*, and *slicing*.)\n'
                '\n'
                'An assignment statement evaluates the expression list '
                '(remember that\n'
@@ -80,9 +79,8 @@
                'rules\n'
                'observed by various types and the exceptions raised are given '
                'with the\n'
-               'definition of the object types (see section *The standard '
-               'type\n'
-               'hierarchy*).\n'
+               'definition of the object types (see section The standard type\n'
+               'hierarchy).\n'
                '\n'
                'Assignment of an object to a target list, optionally enclosed '
                'in\n'
@@ -108,7 +106,7 @@
                '  corresponding targets.\n'
                '\n'
                '  * If the target list contains one target prefixed with an\n'
-               '    asterisk, called a "starred" target: The object must be '
+               '    asterisk, called a ?starred? target: The object must be '
                'an\n'
                '    iterable with at least as many items as there are targets '
                'in the\n'
@@ -205,10 +203,10 @@
                '  If the primary is a mutable sequence object (such as a '
                'list), the\n'
                '  subscript must yield an integer.  If it is negative, the '
-               "sequence's\n"
+               'sequence?s\n'
                '  length is added to it.  The resulting value must be a '
                'nonnegative\n'
-               "  integer less than the sequence's length, and the sequence is "
+               '  integer less than the sequence?s length, and the sequence is '
                'asked\n'
                '  to assign the assigned object to its item with that index.  '
                'If the\n'
@@ -218,7 +216,7 @@
                '\n'
                '  If the primary is a mapping object (such as a dictionary), '
                'the\n'
-               "  subscript must have a type compatible with the mapping's key "
+               '  subscript must have a type compatible with the mapping?s key '
                'type,\n'
                '  and the mapping is then asked to create a key/datum pair '
                'which maps\n'
@@ -241,12 +239,12 @@
                'expressions are\n'
                '  evaluated, insofar they are present; defaults are zero and '
                'the\n'
-               "  sequence's length.  The bounds should evaluate to integers. "
+               '  sequence?s length.  The bounds should evaluate to integers. '
                'If\n'
-               "  either bound is negative, the sequence's length is added to "
+               '  either bound is negative, the sequence?s length is added to '
                'it.  The\n'
                '  resulting bounds are clipped to lie between zero and the '
-               "sequence's\n"
+               'sequence?s\n'
                '  length, inclusive.  Finally, the sequence object is asked to '
                'replace\n'
                '  the slice with the items of the assigned sequence.  The '
@@ -267,7 +265,7 @@
                '\n'
                'Although the definition of assignment implies that overlaps '
                'between\n'
-               "the left-hand side and the right-hand side are 'simultaneous' "
+               'the left-hand side and the right-hand side are ?simultaneous? '
                '(for\n'
                'example "a, b = b, a" swaps two variables), overlaps *within* '
                'the\n'
@@ -283,8 +281,9 @@
                'updated\n'
                '   print(x)\n'
                '\n'
-               'See also: **PEP 3132** - Extended Iterable Unpacking\n'
+               'See also:\n'
                '\n'
+               '  **PEP 3132** - Extended Iterable Unpacking\n'
                '     The specification for the "*target" feature.\n'
                '\n'
                '\n'
@@ -303,8 +302,8 @@
                '"/=" | "//=" | "%=" | "**="\n'
                '             | ">>=" | "<<=" | "&=" | "^=" | "|="\n'
                '\n'
-               '(See section *Primaries* for the syntax definitions of the '
-               'last three\n'
+               '(See section Primaries for the syntax definitions of the last '
+               'three\n'
                'symbols.)\n'
                '\n'
                'An augmented assignment evaluates the target (which, unlike '
@@ -349,19 +348,18 @@
                'normal\n'
                'binary operations.\n'
                '\n'
-               'For targets which are attribute references, the same *caveat '
+               'For targets which are attribute references, the same caveat '
                'about\n'
-               'class and instance attributes* applies as for regular '
+               'class and instance attributes applies as for regular '
                'assignments.\n',
- 'atom-identifiers': '\n'
-                     'Identifiers (Names)\n'
+ 'atom-identifiers': 'Identifiers (Names)\n'
                      '*******************\n'
                      '\n'
                      'An identifier occurring as an atom is a name.  See '
-                     'section\n'
-                     '*Identifiers and keywords* for lexical definition and '
-                     'section *Naming\n'
-                     'and binding* for documentation of naming and binding.\n'
+                     'section Identifiers\n'
+                     'and keywords for lexical definition and section Naming '
+                     'and binding for\n'
+                     'documentation of naming and binding.\n'
                      '\n'
                      'When the name is bound to an object, evaluation of the '
                      'atom yields\n'
@@ -394,8 +392,7 @@
                      'happen. If the class name consists only of underscores, '
                      'no\n'
                      'transformation is done.\n',
- 'atom-literals': '\n'
-                  'Literals\n'
+ 'atom-literals': 'Literals\n'
                   '********\n'
                   '\n'
                   'Python supports string and bytes literals and various '
@@ -411,12 +408,12 @@
                   'the given\n'
                   'value.  The value may be approximated in the case of '
                   'floating point\n'
-                  'and imaginary (complex) literals.  See section *Literals* '
-                  'for details.\n'
+                  'and imaginary (complex) literals.  See section Literals for '
+                  'details.\n'
                   '\n'
                   'All literals correspond to immutable data types, and hence '
                   'the\n'
-                  "object's identity is less important than its value.  "
+                  'object?s identity is less important than its value.  '
                   'Multiple\n'
                   'evaluations of literals with the same value (either the '
                   'same\n'
@@ -424,8 +421,7 @@
                   'may obtain\n'
                   'the same object or a different object with the same '
                   'value.\n',
- 'attribute-access': '\n'
-                     'Customizing attribute access\n'
+ 'attribute-access': 'Customizing attribute access\n'
                      '****************************\n'
                      '\n'
                      'The following methods can be defined to customize the '
@@ -489,8 +485,7 @@
                      'up special\n'
                      '     methods as the result of implicit invocation via '
                      'language syntax\n'
-                     '     or built-in functions. See *Special method '
-                     'lookup*.\n'
+                     '     or built-in functions. See Special method lookup.\n'
                      '\n'
                      'object.__setattr__(self, name, value)\n'
                      '\n'
@@ -533,13 +528,12 @@
                      'containing the method (a so-called *descriptor* class) '
                      'appears in an\n'
                      '*owner* class (the descriptor must be in either the '
-                     "owner's class\n"
+                     'owner?s class\n'
                      'dictionary or in the class dictionary for one of its '
                      'parents).  In the\n'
-                     'examples below, "the attribute" refers to the attribute '
+                     'examples below, ?the attribute? refers to the attribute '
                      'whose name is\n'
-                     "the key of the property in the owner class' "
-                     '"__dict__".\n'
+                     'the key of the property in the owner class? "__dict__".\n'
                      '\n'
                      'object.__get__(self, instance, owner)\n'
                      '\n'
@@ -588,8 +582,8 @@
                      '====================\n'
                      '\n'
                      'In general, a descriptor is an object attribute with '
-                     '"binding\n'
-                     'behavior", one whose attribute access has been '
+                     '?binding\n'
+                     'behavior?, one whose attribute access has been '
                      'overridden by methods\n'
                      'in the descriptor protocol:  "__get__()", "__set__()", '
                      'and\n'
@@ -599,7 +593,7 @@
                      '\n'
                      'The default behavior for attribute access is to get, '
                      'set, or delete\n'
-                     "the attribute from an object's dictionary. For instance, "
+                     'the attribute from an object?s dictionary. For instance, '
                      '"a.x" has a\n'
                      'lookup chain starting with "a.__dict__[\'x\']", then\n'
                      '"type(a).__dict__[\'x\']", and continuing through the '
@@ -654,7 +648,7 @@
                      'does not define "__get__()", then accessing the '
                      'attribute will return\n'
                      'the descriptor object itself unless there is a value in '
-                     "the object's\n"
+                     'the object?s\n'
                      'instance dictionary.  If the descriptor defines '
                      '"__set__()" and/or\n'
                      '"__delete__()", it is a data descriptor; if it defines '
@@ -752,10 +746,10 @@
                      '\n'
                      '* *__slots__* are implemented at the class level by '
                      'creating\n'
-                     '  descriptors (*Implementing Descriptors*) for each '
-                     'variable name.  As\n'
-                     '  a result, class attributes cannot be used to set '
-                     'default values for\n'
+                     '  descriptors (Implementing Descriptors) for each '
+                     'variable name.  As a\n'
+                     '  result, class attributes cannot be used to set default '
+                     'values for\n'
                      '  instance variables defined by *__slots__*; otherwise, '
                      'the class\n'
                      '  attribute would overwrite the descriptor assignment.\n'
@@ -780,7 +774,7 @@
                      '\n'
                      '* Nonempty *__slots__* does not work for classes derived '
                      'from\n'
-                     '  "variable-length" built-in types such as "int", '
+                     '  ?variable-length? built-in types such as "int", '
                      '"bytes" and "tuple".\n'
                      '\n'
                      '* Any non-string iterable may be assigned to '
@@ -792,8 +786,7 @@
                      '* *__class__* assignment works only if both classes have '
                      'the same\n'
                      '  *__slots__*.\n',
- 'attribute-references': '\n'
-                         'Attribute references\n'
+ 'attribute-references': 'Attribute references\n'
                          '********************\n'
                          '\n'
                          'An attribute reference is a primary followed by a '
@@ -816,8 +809,7 @@
                          'determined by the object.  Multiple evaluations of '
                          'the same attribute\n'
                          'reference may yield different objects.\n',
- 'augassign': '\n'
-              'Augmented assignment statements\n'
+ 'augassign': 'Augmented assignment statements\n'
               '*******************************\n'
               '\n'
               'Augmented assignment is the combination, in a single statement, '
@@ -832,7 +824,7 @@
               '"/=" | "//=" | "%=" | "**="\n'
               '             | ">>=" | "<<=" | "&=" | "^=" | "|="\n'
               '\n'
-              '(See section *Primaries* for the syntax definitions of the last '
+              '(See section Primaries for the syntax definitions of the last '
               'three\n'
               'symbols.)\n'
               '\n'
@@ -877,12 +869,11 @@
               'normal\n'
               'binary operations.\n'
               '\n'
-              'For targets which are attribute references, the same *caveat '
+              'For targets which are attribute references, the same caveat '
               'about\n'
-              'class and instance attributes* applies as for regular '
+              'class and instance attributes applies as for regular '
               'assignments.\n',
- 'binary': '\n'
-           'Binary arithmetic operations\n'
+ 'binary': 'Binary arithmetic operations\n'
            '****************************\n'
            '\n'
            'The binary arithmetic operations have the conventional priority\n'
@@ -918,7 +909,7 @@
            'while\n'
            'floor division of integers results in an integer; the result is '
            'that\n'
-           "of mathematical division with the 'floor' function applied to the\n"
+           'of mathematical division with the ?floor? function applied to the\n'
            'result.  Division by zero raises the "ZeroDivisionError" '
            'exception.\n'
            '\n'
@@ -951,7 +942,7 @@
            'old-style\n'
            'string formatting (also known as interpolation).  The syntax for\n'
            'string formatting is described in the Python Library Reference,\n'
-           'section *printf-style String Formatting*.\n'
+           'section printf-style String Formatting.\n'
            '\n'
            'The floor division operator, the modulo operator, and the '
            '"divmod()"\n'
@@ -970,8 +961,7 @@
            'The "-" (subtraction) operator yields the difference of its '
            'arguments.\n'
            'The numeric arguments are first converted to a common type.\n',
- 'bitwise': '\n'
-            'Binary bitwise operations\n'
+ 'bitwise': 'Binary bitwise operations\n'
             '*************************\n'
             '\n'
             'Each of the three bitwise operations has a different priority '
@@ -991,15 +981,14 @@
             'The "|" operator yields the bitwise (inclusive) OR of its '
             'arguments,\n'
             'which must be integers.\n',
- 'bltin-code-objects': '\n'
-                       'Code Objects\n'
+ 'bltin-code-objects': 'Code Objects\n'
                        '************\n'
                        '\n'
                        'Code objects are used by the implementation to '
-                       'represent "pseudo-\n'
-                       'compiled" executable Python code such as a function '
+                       'represent ?pseudo-\n'
+                       'compiled? executable Python code such as a function '
                        'body. They differ\n'
-                       "from function objects because they don't contain a "
+                       'from function objects because they don?t contain a '
                        'reference to their\n'
                        'global execution environment.  Code objects are '
                        'returned by the built-\n'
@@ -1013,14 +1002,13 @@
                        'source string) to the "exec()" or "eval()"  built-in '
                        'functions.\n'
                        '\n'
-                       'See *The standard type hierarchy* for more '
+                       'See The standard type hierarchy for more '
                        'information.\n',
- 'bltin-ellipsis-object': '\n'
-                          'The Ellipsis Object\n'
+ 'bltin-ellipsis-object': 'The Ellipsis Object\n'
                           '*******************\n'
                           '\n'
                           'This object is commonly used by slicing (see '
-                          '*Slicings*).  It supports\n'
+                          'Slicings).  It supports\n'
                           'no special operations.  There is exactly one '
                           'ellipsis object, named\n'
                           '"Ellipsis" (a built-in name).  "type(Ellipsis)()" '
@@ -1028,11 +1016,10 @@
                           '"Ellipsis" singleton.\n'
                           '\n'
                           'It is written as "Ellipsis" or "...".\n',
- 'bltin-null-object': '\n'
-                      'The Null Object\n'
+ 'bltin-null-object': 'The Null Object\n'
                       '***************\n'
                       '\n'
-                      "This object is returned by functions that don't "
+                      'This object is returned by functions that don?t '
                       'explicitly return a\n'
                       'value.  It supports no special operations.  There is '
                       'exactly one null\n'
@@ -1041,12 +1028,11 @@
                       'same singleton.\n'
                       '\n'
                       'It is written as "None".\n',
- 'bltin-type-objects': '\n'
-                       'Type Objects\n'
+ 'bltin-type-objects': 'Type Objects\n'
                        '************\n'
                        '\n'
                        'Type objects represent the various object types.  An '
-                       "object's type is\n"
+                       'object?s type is\n'
                        'accessed by the built-in function "type()".  There are '
                        'no special\n'
                        'operations on types.  The standard module "types" '
@@ -1054,8 +1040,7 @@
                        'all standard built-in types.\n'
                        '\n'
                        'Types are written like this: "<class \'int\'>".\n',
- 'booleans': '\n'
-             'Boolean operations\n'
+ 'booleans': 'Boolean operations\n'
              '******************\n'
              '\n'
              '   or_test  ::= and_test | or_test "or" and_test\n'
@@ -1104,8 +1089,7 @@
              'its\n'
              'argument (for example, "not \'foo\'" produces "False" rather '
              'than "\'\'".)\n',
- 'break': '\n'
-          'The "break" statement\n'
+ 'break': 'The "break" statement\n'
           '*********************\n'
           '\n'
           '   break_stmt ::= "break"\n'
@@ -1126,18 +1110,16 @@
           'clause, that "finally" clause is executed before really leaving '
           'the\n'
           'loop.\n',
- 'callable-types': '\n'
-                   'Emulating callable objects\n'
+ 'callable-types': 'Emulating callable objects\n'
                    '**************************\n'
                    '\n'
                    'object.__call__(self[, args...])\n'
                    '\n'
-                   '   Called when the instance is "called" as a function; if '
+                   '   Called when the instance is ?called? as a function; if '
                    'this method\n'
                    '   is defined, "x(arg1, arg2, ...)" is a shorthand for\n'
                    '   "x.__call__(arg1, arg2, ...)".\n',
- 'calls': '\n'
-          'Calls\n'
+ 'calls': 'Calls\n'
           '*****\n'
           '\n'
           'A call calls a callable object (e.g., a *function*) with a '
@@ -1170,7 +1152,7 @@
           'objects, methods of class instances, and all objects having a\n'
           '"__call__()" method are callable).  All argument expressions are\n'
           'evaluated before the call is attempted.  Please refer to section\n'
-          '*Function definitions* for the syntax of formal *parameter* lists.\n'
+          'Function definitions for the syntax of formal *parameter* lists.\n'
           '\n'
           'If keyword arguments are present, they are first converted to\n'
           'positional arguments, as follows.  First, a list of unfilled slots '
@@ -1193,7 +1175,7 @@
           'values are calculated, once, when the function is defined; thus, a\n'
           'mutable object such as a list or dictionary used as default value '
           'will\n'
-          "be shared by all calls that don't specify an argument value for "
+          'be shared by all calls that don?t specify an argument value for '
           'the\n'
           'corresponding slot; this should usually be avoided.)  If there are '
           'any\n'
@@ -1206,7 +1188,7 @@
           '**CPython implementation detail:** An implementation may provide\n'
           'built-in functions whose positional parameters do not have names, '
           'even\n'
-          "if they are 'named' for the purpose of documentation, and which\n"
+          'if they are ?named? for the purpose of documentation, and which\n'
           'therefore cannot be supplied by keyword.  In CPython, this is the '
           'case\n'
           'for functions implemented in C that use "PyArg_ParseTuple()" to '
@@ -1238,16 +1220,17 @@
           'must evaluate to an *iterable*.  Elements from these iterables are\n'
           'treated as if they were additional positional arguments.  For the '
           'call\n'
-          '"f(x1, x2, *y, x3, x4)", if *y* evaluates to a sequence *y1*, ...,\n'
-          '*yM*, this is equivalent to a call with M+4 positional arguments '
-          '*x1*,\n'
-          '*x2*, *y1*, ..., *yM*, *x3*, *x4*.\n'
+          '"f(x1, x2, *y, x3, x4)", if *y* evaluates to a sequence *y1*, ?, '
+          '*yM*,\n'
+          'this is equivalent to a call with M+4 positional arguments *x1*, '
+          '*x2*,\n'
+          '*y1*, ?, *yM*, *x3*, *x4*.\n'
           '\n'
           'A consequence of this is that although the "*expression" syntax '
           'may\n'
           'appear *after* explicit keyword arguments, it is processed '
           '*before*\n'
-          'the keyword arguments (and any "**expression" arguments -- see '
+          'the keyword arguments (and any "**expression" arguments ? see '
           'below).\n'
           'So:\n'
           '\n'
@@ -1294,7 +1277,7 @@
           'exception.  How this value is computed depends on the type of the\n'
           'callable object.\n'
           '\n'
-          'If it is---\n'
+          'If it is?\n'
           '\n'
           'a user-defined function:\n'
           '   The code block for the function is executed, passing it the\n'
@@ -1302,15 +1285,14 @@
           'the\n'
           '   formal parameters to the arguments; this is described in '
           'section\n'
-          '   *Function definitions*.  When the code block executes a '
-          '"return"\n'
+          '   Function definitions.  When the code block executes a "return"\n'
           '   statement, this specifies the return value of the function '
           'call.\n'
           '\n'
           'a built-in function or method:\n'
-          '   The result is up to the interpreter; see *Built-in Functions* '
-          'for\n'
-          '   the descriptions of built-in functions and methods.\n'
+          '   The result is up to the interpreter; see Built-in Functions for '
+          'the\n'
+          '   descriptions of built-in functions and methods.\n'
           '\n'
           'a class object:\n'
           '   A new instance of that class is returned.\n'
@@ -1325,13 +1307,12 @@
           '   The class must define a "__call__()" method; the effect is then '
           'the\n'
           '   same as if that method was called.\n',
- 'class': '\n'
-          'Class definitions\n'
+ 'class': 'Class definitions\n'
           '*****************\n'
           '\n'
-          'A class definition defines a class object (see section *The '
+          'A class definition defines a class object (see section The '
           'standard\n'
-          'type hierarchy*):\n'
+          'type hierarchy):\n'
           '\n'
           '   classdef    ::= [decorators] "class" classname [inheritance] ":" '
           'suite\n'
@@ -1340,8 +1321,8 @@
           '\n'
           'A class definition is an executable statement.  The inheritance '
           'list\n'
-          'usually gives a list of base classes (see *Customizing class '
-          'creation*\n'
+          'usually gives a list of base classes (see Customizing class '
+          'creation\n'
           'for more advanced uses), so each item in the list should evaluate '
           'to a\n'
           'class object which allows subclassing.  Classes without an '
@@ -1356,11 +1337,10 @@
           '   class Foo(object):\n'
           '       pass\n'
           '\n'
-          "The class's suite is then executed in a new execution frame (see\n"
-          '*Naming and binding*), using a newly created local namespace and '
-          'the\n'
+          'The class?s suite is then executed in a new execution frame (see\n'
+          'Naming and binding), using a newly created local namespace and the\n'
           'original global namespace. (Usually, the suite contains mostly\n'
-          "function definitions.)  When the class's suite finishes execution, "
+          'function definitions.)  When the class?s suite finishes execution, '
           'its\n'
           'execution frame is discarded but its local namespace is saved. [4] '
           'A\n'
@@ -1371,7 +1351,7 @@
           'The class name is bound to this class object in the original local\n'
           'namespace.\n'
           '\n'
-          'Class creation can be customized heavily using *metaclasses*.\n'
+          'Class creation can be customized heavily using metaclasses.\n'
           '\n'
           'Classes can also be decorated: just like when decorating '
           'functions,\n'
@@ -1389,26 +1369,25 @@
           'for\n'
           'function decorators.  The result is then bound to the class name.\n'
           '\n'
-          "**Programmer's note:** Variables defined in the class definition "
+          '**Programmer?s note:** Variables defined in the class definition '
           'are\n'
           'class attributes; they are shared by instances.  Instance '
           'attributes\n'
           'can be set in a method with "self.name = value".  Both class and\n'
           'instance attributes are accessible through the notation '
-          '""self.name"",\n'
+          '?"self.name"?,\n'
           'and an instance attribute hides a class attribute with the same '
           'name\n'
           'when accessed in this way.  Class attributes can be used as '
           'defaults\n'
           'for instance attributes, but using mutable values there can lead '
           'to\n'
-          'unexpected results.  *Descriptors* can be used to create instance\n'
+          'unexpected results.  Descriptors can be used to create instance\n'
           'variables with different implementation details.\n'
           '\n'
           'See also: **PEP 3115** - Metaclasses in Python 3 **PEP 3129** -\n'
           '  Class Decorators\n',
- 'comparisons': '\n'
-                'Comparisons\n'
+ 'comparisons': 'Comparisons\n'
                 '***********\n'
                 '\n'
                 'Unlike C, all comparison operations in Python have the same '
@@ -1433,15 +1412,15 @@
                 'y" is\n'
                 'found to be false).\n'
                 '\n'
-                'Formally, if *a*, *b*, *c*, ..., *y*, *z* are expressions and '
+                'Formally, if *a*, *b*, *c*, ?, *y*, *z* are expressions and '
                 '*op1*,\n'
-                '*op2*, ..., *opN* are comparison operators, then "a op1 b op2 '
-                'c ... y\n'
+                '*op2*, ?, *opN* are comparison operators, then "a op1 b op2 c '
+                '... y\n'
                 'opN z" is equivalent to "a op1 b and b op2 c and ... y opN '
                 'z", except\n'
                 'that each expression is evaluated at most once.\n'
                 '\n'
-                'Note that "a op1 b op2 c" doesn\'t imply any kind of '
+                'Note that "a op1 b op2 c" doesn?t imply any kind of '
                 'comparison between\n'
                 '*a* and *c*, so that, e.g., "x < y > z" is perfectly legal '
                 '(though\n'
@@ -1456,23 +1435,23 @@
                 'of two objects.  The objects do not need to have the same '
                 'type.\n'
                 '\n'
-                'Chapter *Objects, values and types* states that objects have '
-                'a value\n'
-                '(in addition to type and identity).  The value of an object '
-                'is a\n'
-                'rather abstract notion in Python: For example, there is no '
-                'canonical\n'
-                "access method for an object's value.  Also, there is no "
-                'requirement\n'
-                'that the value of an object should be constructed in a '
-                'particular way,\n'
-                'e.g. comprised of all its data attributes. Comparison '
-                'operators\n'
-                'implement a particular notion of what the value of an object '
-                'is.  One\n'
-                'can think of them as defining the value of an object '
-                'indirectly, by\n'
-                'means of their comparison implementation.\n'
+                'Chapter Objects, values and types states that objects have a '
+                'value (in\n'
+                'addition to type and identity).  The value of an object is a '
+                'rather\n'
+                'abstract notion in Python: For example, there is no canonical '
+                'access\n'
+                'method for an object?s value.  Also, there is no requirement '
+                'that the\n'
+                'value of an object should be constructed in a particular way, '
+                'e.g.\n'
+                'comprised of all its data attributes. Comparison operators '
+                'implement a\n'
+                'particular notion of what the value of an object is.  One can '
+                'think of\n'
+                'them as defining the value of an object indirectly, by means '
+                'of their\n'
+                'comparison implementation.\n'
                 '\n'
                 'Because all types are (direct or indirect) subtypes of '
                 '"object", they\n'
@@ -1480,8 +1459,7 @@
                 'can\n'
                 'customize their comparison behavior by implementing *rich '
                 'comparison\n'
-                'methods* like "__lt__()", described in *Basic '
-                'customization*.\n'
+                'methods* like "__lt__()", described in Basic customization.\n'
                 '\n'
                 'The default behavior for equality comparison ("==" and "!=") '
                 'is based\n'
@@ -1517,9 +1495,9 @@
                 'most\n'
                 'important built-in types.\n'
                 '\n'
-                '* Numbers of built-in numeric types (*Numeric Types --- int, '
+                '* Numbers of built-in numeric types (Numeric Types ? int, '
                 'float,\n'
-                '  complex*) and of the standard library types '
+                '  complex) and of the standard library types '
                 '"fractions.Fraction" and\n'
                 '  "decimal.Decimal" can be compared within and across their '
                 'types,\n'
@@ -1780,8 +1758,7 @@
                 'is determined using the "id()" function.  "x is not y" yields '
                 'the\n'
                 'inverse truth value. [4]\n',
- 'compound': '\n'
-             'Compound statements\n'
+ 'compound': 'Compound statements\n'
              '*******************\n'
              '\n'
              'Compound statements contain (groups of) other statements; they '
@@ -1805,9 +1782,9 @@
              'compound\n'
              'statements.\n'
              '\n'
-             "A compound statement consists of one or more 'clauses.'  A "
+             'A compound statement consists of one or more ?clauses.?  A '
              'clause\n'
-             "consists of a header and a 'suite.'  The clause headers of a\n"
+             'consists of a header and a ?suite.?  The clause headers of a\n'
              'particular compound statement are all at the same indentation '
              'level.\n'
              'Each clause header begins with a uniquely identifying keyword '
@@ -1815,12 +1792,12 @@
              'with a colon.  A suite is a group of statements controlled by a\n'
              'clause.  A suite can be one or more semicolon-separated simple\n'
              'statements on the same line as the header, following the '
-             "header's\n"
+             'header?s\n'
              'colon, or it can be one or more indented statements on '
              'subsequent\n'
              'lines.  Only the latter form of a suite can contain nested '
              'compound\n'
-             "statements; the following is illegal, mostly because it wouldn't "
+             'statements; the following is illegal, mostly because it wouldn?t '
              'be\n'
              'clear to which "if" clause a following "else" clause would '
              'belong:\n'
@@ -1857,7 +1834,7 @@
              '"DEDENT".  Also note that optional continuation clauses always '
              'begin\n'
              'with a keyword that cannot start a statement, thus there are no\n'
-             'ambiguities (the \'dangling "else"\' problem is solved in Python '
+             'ambiguities (the ?dangling "else"? problem is solved in Python '
              'by\n'
              'requiring nested "if" statements to be indented).\n'
              '\n'
@@ -1877,8 +1854,8 @@
              '\n'
              'It selects exactly one of the suites by evaluating the '
              'expressions one\n'
-             'by one until one is found to be true (see section *Boolean '
-             'operations*\n'
+             'by one until one is found to be true (see section Boolean '
+             'operations\n'
              'for the definition of true and false); then that suite is '
              'executed\n'
              '(and no other part of the "if" statement is executed or '
@@ -1908,7 +1885,7 @@
              '\n'
              'A "break" statement executed in the first suite terminates the '
              'loop\n'
-             'without executing the "else" clause\'s suite.  A "continue" '
+             'without executing the "else" clause?s suite.  A "continue" '
              'statement\n'
              'executed in the first suite skips the rest of the suite and goes '
              'back\n'
@@ -1935,8 +1912,8 @@
              'iterator.  Each\n'
              'item in turn is assigned to the target list using the standard '
              'rules\n'
-             'for assignments (see *Assignment statements*), and then the '
-             'suite is\n'
+             'for assignments (see Assignment statements), and then the suite '
+             'is\n'
              'executed.  When the items are exhausted (which is immediately '
              'when the\n'
              'sequence is empty or an iterator raises a "StopIteration" '
@@ -1947,7 +1924,7 @@
              '\n'
              'A "break" statement executed in the first suite terminates the '
              'loop\n'
-             'without executing the "else" clause\'s suite.  A "continue" '
+             'without executing the "else" clause?s suite.  A "continue" '
              'statement\n'
              'executed in the first suite skips the rest of the suite and '
              'continues\n'
@@ -1974,7 +1951,7 @@
              'to at\n'
              'all by the loop.  Hint: the built-in function "range()" returns '
              'an\n'
-             "iterator of integers suitable to emulate the effect of Pascal's "
+             'iterator of integers suitable to emulate the effect of Pascal?s '
              '"for i\n'
              ':= a to b do"; e.g., "list(range(3))" returns the list "[0, 1, '
              '2]".\n'
@@ -2036,7 +2013,7 @@
              'expression\n'
              'is evaluated, and the clause matches the exception if the '
              'resulting\n'
-             'object is "compatible" with the exception.  An object is '
+             'object is ?compatible? with the exception.  An object is '
              'compatible\n'
              'with an exception if it is the class or a base class of the '
              'exception\n'
@@ -2063,7 +2040,7 @@
              'assigned to\n'
              'the target specified after the "as" keyword in that except '
              'clause, if\n'
-             "present, and the except clause's suite is executed.  All except\n"
+             'present, and the except clause?s suite is executed.  All except\n'
              'clauses must have an executable block.  When the end of this '
              'block is\n'
              'reached, execution continues normally after the entire try '
@@ -2099,14 +2076,14 @@
              'alive\n'
              'until the next garbage collection occurs.\n'
              '\n'
-             "Before an except clause's suite is executed, details about the\n"
+             'Before an except clause?s suite is executed, details about the\n'
              'exception are stored in the "sys" module and can be accessed '
              'via\n'
              '"sys.exc_info()". "sys.exc_info()" returns a 3-tuple consisting '
              'of the\n'
              'exception class, the exception instance and a traceback object '
              '(see\n'
-             'section *The standard type hierarchy*) identifying the point in '
+             'section The standard type hierarchy) identifying the point in '
              'the\n'
              'program where the exception occurred.  "sys.exc_info()" values '
              'are\n'
@@ -2120,8 +2097,8 @@
              'are\n'
              'not handled by the preceding "except" clauses.\n'
              '\n'
-             'If "finally" is present, it specifies a \'cleanup\' handler.  '
-             'The "try"\n'
+             'If "finally" is present, it specifies a ?cleanup? handler.  The '
+             '"try"\n'
              'clause is executed, including any "except" and "else" clauses.  '
              'If an\n'
              'exception occurs in any of the clauses and is not handled, the\n'
@@ -2152,13 +2129,12 @@
              '\n'
              'When a "return", "break" or "continue" statement is executed in '
              'the\n'
-             '"try" suite of a "try"..."finally" statement, the "finally" '
-             'clause is\n'
-             'also executed \'on the way out.\' A "continue" statement is '
+             '"try" suite of a "try"?"finally" statement, the "finally" clause '
+             'is\n'
+             'also executed ?on the way out.? A "continue" statement is '
              'illegal in\n'
              'the "finally" clause. (The reason is a problem with the current\n'
-             'implementation --- this restriction may be lifted in the '
-             'future).\n'
+             'implementation ? this restriction may be lifted in the future).\n'
              '\n'
              'The return value of a function is determined by the last '
              '"return"\n'
@@ -2178,9 +2154,9 @@
              "   'finally'\n"
              '\n'
              'Additional information on exceptions can be found in section\n'
-             '*Exceptions*, and information on using the "raise" statement to\n'
-             'generate exceptions may be found in section *The raise '
-             'statement*.\n'
+             'Exceptions, and information on using the "raise" statement to '
+             'generate\n'
+             'exceptions may be found in section The raise statement.\n'
              '\n'
              '\n'
              'The "with" statement\n'
@@ -2188,16 +2164,16 @@
              '\n'
              'The "with" statement is used to wrap the execution of a block '
              'with\n'
-             'methods defined by a context manager (see section *With '
+             'methods defined by a context manager (see section With '
              'Statement\n'
-             'Context Managers*). This allows common '
-             '"try"..."except"..."finally"\n'
-             'usage patterns to be encapsulated for convenient reuse.\n'
+             'Context Managers). This allows common "try"?"except"?"finally" '
+             'usage\n'
+             'patterns to be encapsulated for convenient reuse.\n'
              '\n'
              '   with_stmt ::= "with" with_item ("," with_item)* ":" suite\n'
              '   with_item ::= expression ["as" target]\n'
              '\n'
-             'The execution of the "with" statement with one "item" proceeds '
+             'The execution of the "with" statement with one ?item? proceeds '
              'as\n'
              'follows:\n'
              '\n'
@@ -2205,9 +2181,9 @@
              '"with_item")\n'
              '   is evaluated to obtain a context manager.\n'
              '\n'
-             '2. The context manager\'s "__exit__()" is loaded for later use.\n'
+             '2. The context manager?s "__exit__()" is loaded for later use.\n'
              '\n'
-             '3. The context manager\'s "__enter__()" method is invoked.\n'
+             '3. The context manager?s "__enter__()" method is invoked.\n'
              '\n'
              '4. If a target was included in the "with" statement, the return\n'
              '   value from "__enter__()" is assigned to it.\n'
@@ -2224,8 +2200,7 @@
              '\n'
              '5. The suite is executed.\n'
              '\n'
-             '6. The context manager\'s "__exit__()" method is invoked.  If '
-             'an\n'
+             '6. The context manager?s "__exit__()" method is invoked.  If an\n'
              '   exception caused the suite to be exited, its type, value, '
              'and\n'
              '   traceback are passed as arguments to "__exit__()". Otherwise, '
@@ -2263,8 +2238,9 @@
              'Changed in version 3.1: Support for multiple context '
              'expressions.\n'
              '\n'
-             'See also: **PEP 343** - The "with" statement\n'
+             'See also:\n'
              '\n'
+             '  **PEP 343** - The ?with? statement\n'
              '     The specification, background, and examples for the Python '
              '"with"\n'
              '     statement.\n'
@@ -2275,7 +2251,7 @@
              '\n'
              'A function definition defines a user-defined function object '
              '(see\n'
-             'section *The standard type hierarchy*):\n'
+             'section The standard type hierarchy):\n'
              '\n'
              '   funcdef        ::= [decorators] "def" funcname "(" '
              '[parameter_list] ")" ["->" expression] ":" suite\n'
@@ -2332,25 +2308,24 @@
              '"func".\n'
              '\n'
              'When one or more *parameters* have the form *parameter* "="\n'
-             '*expression*, the function is said to have "default parameter '
-             'values."\n'
+             '*expression*, the function is said to have ?default parameter '
+             'values.?\n'
              'For a parameter with a default value, the corresponding '
              '*argument* may\n'
-             "be omitted from a call, in which case the parameter's default "
+             'be omitted from a call, in which case the parameter?s default '
              'value is\n'
              'substituted.  If a parameter has a default value, all following\n'
-             'parameters up until the ""*"" must also have a default value --- '
-             'this\n'
-             'is a syntactic restriction that is not expressed by the '
-             'grammar.\n'
+             'parameters up until the ?"*"? must also have a default value ? '
+             'this is\n'
+             'a syntactic restriction that is not expressed by the grammar.\n'
              '\n'
              '**Default parameter values are evaluated from left to right when '
              'the\n'
              'function definition is executed.** This means that the '
              'expression is\n'
              'evaluated once, when the function is defined, and that the same '
-             '"pre-\n'
-             'computed" value is used for each call.  This is especially '
+             '?pre-\n'
+             'computed? value is used for each call.  This is especially '
              'important\n'
              'to understand when a default parameter is a mutable object, such '
              'as a\n'
@@ -2370,30 +2345,32 @@
              '       penguin.append("property of the zoo")\n'
              '       return penguin\n'
              '\n'
-             'Function call semantics are described in more detail in section\n'
-             '*Calls*. A function call always assigns values to all '
-             'parameters\n'
-             'mentioned in the parameter list, either from position arguments, '
-             'from\n'
-             'keyword arguments, or from default values.  If the form\n'
-             '""*identifier"" is present, it is initialized to a tuple '
-             'receiving any\n'
-             'excess positional parameters, defaulting to the empty tuple.  If '
-             'the\n'
-             'form ""**identifier"" is present, it is initialized to a new\n'
-             'dictionary receiving any excess keyword arguments, defaulting to '
-             'a new\n'
-             'empty dictionary. Parameters after ""*"" or ""*identifier"" are\n'
-             'keyword-only parameters and may only be passed used keyword '
-             'arguments.\n'
-             '\n'
-             'Parameters may have annotations of the form "": expression"" '
+             'Function call semantics are described in more detail in section '
+             'Calls.\n'
+             'A function call always assigns values to all parameters '
+             'mentioned in\n'
+             'the parameter list, either from position arguments, from '
+             'keyword\n'
+             'arguments, or from default values.  If the form ?"*identifier"? '
+             'is\n'
+             'present, it is initialized to a tuple receiving any excess '
+             'positional\n'
+             'parameters, defaulting to the empty tuple.  If the form\n'
+             '?"**identifier"? is present, it is initialized to a new '
+             'dictionary\n'
+             'receiving any excess keyword arguments, defaulting to a new '
+             'empty\n'
+             'dictionary. Parameters after ?"*"? or ?"*identifier"? are '
+             'keyword-only\n'
+             'parameters and may only be passed used keyword arguments.\n'
+             '\n'
+             'Parameters may have annotations of the form ?": expression"? '
              'following\n'
              'the parameter name.  Any parameter may have an annotation even '
              'those\n'
              'of the form "*identifier" or "**identifier".  Functions may '
              'have\n'
-             '"return" annotation of the form ""-> expression"" after the '
+             '?return? annotation of the form ?"-> expression"? after the '
              'parameter\n'
              'list.  These annotations can be any valid Python expression and '
              'are\n'
@@ -2403,46 +2380,47 @@
              'code.\n'
              'The presence of annotations does not change the semantics of a\n'
              'function.  The annotation values are available as values of a\n'
-             "dictionary keyed by the parameters' names in the "
+             'dictionary keyed by the parameters? names in the '
              '"__annotations__"\n'
              'attribute of the function object.\n'
              '\n'
              'It is also possible to create anonymous functions (functions not '
              'bound\n'
              'to a name), for immediate use in expressions.  This uses lambda\n'
-             'expressions, described in section *Lambdas*.  Note that the '
+             'expressions, described in section Lambdas.  Note that the '
              'lambda\n'
              'expression is merely a shorthand for a simplified function '
              'definition;\n'
-             'a function defined in a ""def"" statement can be passed around '
+             'a function defined in a ?"def"? statement can be passed around '
              'or\n'
              'assigned to another name just like a function defined by a '
              'lambda\n'
-             'expression.  The ""def"" form is actually more powerful since '
+             'expression.  The ?"def"? form is actually more powerful since '
              'it\n'
              'allows the execution of multiple statements and annotations.\n'
              '\n'
-             "**Programmer's note:** Functions are first-class objects.  A "
-             '""def""\n'
+             '**Programmer?s note:** Functions are first-class objects.  A '
+             '?"def"?\n'
              'statement executed inside a function definition defines a local\n'
              'function that can be returned or passed around.  Free variables '
              'used\n'
              'in the nested function can access the local variables of the '
              'function\n'
-             'containing the def.  See section *Naming and binding* for '
+             'containing the def.  See section Naming and binding for '
              'details.\n'
              '\n'
-             'See also: **PEP 3107** - Function Annotations\n'
+             'See also:\n'
              '\n'
+             '  **PEP 3107** - Function Annotations\n'
              '     The original specification for function annotations.\n'
              '\n'
              '\n'
              'Class definitions\n'
              '=================\n'
              '\n'
-             'A class definition defines a class object (see section *The '
+             'A class definition defines a class object (see section The '
              'standard\n'
-             'type hierarchy*):\n'
+             'type hierarchy):\n'
              '\n'
              '   classdef    ::= [decorators] "class" classname [inheritance] '
              '":" suite\n'
@@ -2451,8 +2429,8 @@
              '\n'
              'A class definition is an executable statement.  The inheritance '
              'list\n'
-             'usually gives a list of base classes (see *Customizing class '
-             'creation*\n'
+             'usually gives a list of base classes (see Customizing class '
+             'creation\n'
              'for more advanced uses), so each item in the list should '
              'evaluate to a\n'
              'class object which allows subclassing.  Classes without an '
@@ -2467,12 +2445,12 @@
              '   class Foo(object):\n'
              '       pass\n'
              '\n'
-             "The class's suite is then executed in a new execution frame "
+             'The class?s suite is then executed in a new execution frame '
              '(see\n'
-             '*Naming and binding*), using a newly created local namespace and '
+             'Naming and binding), using a newly created local namespace and '
              'the\n'
              'original global namespace. (Usually, the suite contains mostly\n'
-             "function definitions.)  When the class's suite finishes "
+             'function definitions.)  When the class?s suite finishes '
              'execution, its\n'
              'execution frame is discarded but its local namespace is saved. '
              '[4] A\n'
@@ -2484,7 +2462,7 @@
              'local\n'
              'namespace.\n'
              '\n'
-             'Class creation can be customized heavily using *metaclasses*.\n'
+             'Class creation can be customized heavily using metaclasses.\n'
              '\n'
              'Classes can also be decorated: just like when decorating '
              'functions,\n'
@@ -2503,22 +2481,21 @@
              'function decorators.  The result is then bound to the class '
              'name.\n'
              '\n'
-             "**Programmer's note:** Variables defined in the class definition "
+             '**Programmer?s note:** Variables defined in the class definition '
              'are\n'
              'class attributes; they are shared by instances.  Instance '
              'attributes\n'
              'can be set in a method with "self.name = value".  Both class '
              'and\n'
              'instance attributes are accessible through the notation '
-             '""self.name"",\n'
+             '?"self.name"?,\n'
              'and an instance attribute hides a class attribute with the same '
              'name\n'
              'when accessed in this way.  Class attributes can be used as '
              'defaults\n'
              'for instance attributes, but using mutable values there can lead '
              'to\n'
-             'unexpected results.  *Descriptors* can be used to create '
-             'instance\n'
+             'unexpected results.  Descriptors can be used to create instance\n'
              'variables with different implementation details.\n'
              '\n'
              'See also: **PEP 3115** - Metaclasses in Python 3 **PEP 3129** -\n'
@@ -2647,22 +2624,20 @@
              '    exception. That new exception causes the old one to be '
              'lost.\n'
              '\n'
-             '[2] Currently, control "flows off the end" except in the case '
+             '[2] Currently, control ?flows off the end? except in the case '
              'of\n'
              '    an exception or the execution of a "return", "continue", or\n'
              '    "break" statement.\n'
              '\n'
              '[3] A string literal appearing as the first statement in the\n'
-             '    function body is transformed into the function\'s "__doc__"\n'
-             "    attribute and therefore the function's *docstring*.\n"
+             '    function body is transformed into the function?s "__doc__"\n'
+             '    attribute and therefore the function?s *docstring*.\n'
              '\n'
              '[4] A string literal appearing as the first statement in the '
              'class\n'
-             '    body is transformed into the namespace\'s "__doc__" item '
-             'and\n'
-             "    therefore the class's *docstring*.\n",
- 'context-managers': '\n'
-                     'With Statement Context Managers\n'
+             '    body is transformed into the namespace?s "__doc__" item and\n'
+             '    therefore the class?s *docstring*.\n',
+ 'context-managers': 'With Statement Context Managers\n'
                      '*******************************\n'
                      '\n'
                      'A *context manager* is an object that defines the '
@@ -2674,8 +2649,8 @@
                      'for the execution of the block of code.  Context '
                      'managers are normally\n'
                      'invoked using the "with" statement (described in section '
-                     '*The with\n'
-                     'statement*), but can also be used by directly invoking '
+                     'The with\n'
+                     'statement), but can also be used by directly invoking '
                      'their methods.\n'
                      '\n'
                      'Typical uses of context managers include saving and '
@@ -2684,14 +2659,14 @@
                      'closing opened\n'
                      'files, etc.\n'
                      '\n'
-                     'For more information on context managers, see *Context '
-                     'Manager Types*.\n'
+                     'For more information on context managers, see Context '
+                     'Manager Types.\n'
                      '\n'
                      'object.__enter__(self)\n'
                      '\n'
                      '   Enter the runtime context related to this object. The '
                      '"with"\n'
-                     "   statement will bind this method's return value to the "
+                     '   statement will bind this method?s return value to the '
                      'target(s)\n'
                      '   specified in the "as" clause of the statement, if '
                      'any.\n'
@@ -2716,15 +2691,15 @@
                      '\n'
                      '   Note that "__exit__()" methods should not reraise the '
                      'passed-in\n'
-                     "   exception; this is the caller's responsibility.\n"
+                     '   exception; this is the caller?s responsibility.\n'
                      '\n'
-                     'See also: **PEP 343** - The "with" statement\n'
+                     'See also:\n'
                      '\n'
+                     '  **PEP 343** - The ?with? statement\n'
                      '     The specification, background, and examples for the '
                      'Python "with"\n'
                      '     statement.\n',
- 'continue': '\n'
-             'The "continue" statement\n'
+ 'continue': 'The "continue" statement\n'
              '************************\n'
              '\n'
              '   continue_stmt ::= "continue"\n'
@@ -2741,13 +2716,12 @@
              '"finally" clause, that "finally" clause is executed before '
              'really\n'
              'starting the next loop cycle.\n',
- 'conversions': '\n'
-                'Arithmetic conversions\n'
+ 'conversions': 'Arithmetic conversions\n'
                 '**********************\n'
                 '\n'
                 'When a description of an arithmetic operator below uses the '
                 'phrase\n'
-                '"the numeric arguments are converted to a common type," this '
+                '?the numeric arguments are converted to a common type,? this '
                 'means\n'
                 'that the operator implementation for built-in types works as '
                 'follows:\n'
@@ -2765,11 +2739,10 @@
                 '\n'
                 'Some additional rules apply for certain operators (e.g., a '
                 'string as a\n'
-                "left argument to the '%' operator).  Extensions must define "
+                'left argument to the ?%? operator).  Extensions must define '
                 'their own\n'
                 'conversion behavior.\n',
- 'customization': '\n'
-                  'Basic customization\n'
+ 'customization': 'Basic customization\n'
                   '*******************\n'
                   '\n'
                   'object.__new__(cls[, ...])\n'
@@ -2790,7 +2763,7 @@
                   '\n'
                   '   Typical implementations create a new instance of the '
                   'class by\n'
-                  '   invoking the superclass\'s "__new__()" method using\n'
+                  '   invoking the superclass?s "__new__()" method using\n'
                   '   "super().__new__(cls[, ...])" with appropriate arguments '
                   'and then\n'
                   '   modifying the newly-created instance as necessary before '
@@ -2799,7 +2772,7 @@
                   '\n'
                   '   If "__new__()" returns an instance of *cls*, then the '
                   'new\n'
-                  '   instance\'s "__init__()" method will be invoked like\n'
+                  '   instance?s "__init__()" method will be invoked like\n'
                   '   "__init__(self[, ...])", where *self* is the new '
                   'instance and the\n'
                   '   remaining arguments are the same as were passed to '
@@ -2807,7 +2780,7 @@
                   '\n'
                   '   If "__new__()" does not return an instance of *cls*, '
                   'then the new\n'
-                  '   instance\'s "__init__()" method will not be invoked.\n'
+                  '   instance?s "__init__()" method will not be invoked.\n'
                   '\n'
                   '   "__new__()" is intended mainly to allow subclasses of '
                   'immutable\n'
@@ -2825,7 +2798,7 @@
                   'those\n'
                   '   passed to the class constructor expression.  If a base '
                   'class has an\n'
-                  '   "__init__()" method, the derived class\'s "__init__()" '
+                  '   "__init__()" method, the derived class?s "__init__()" '
                   'method, if\n'
                   '   any, must explicitly call it to ensure proper '
                   'initialization of the\n'
@@ -2846,7 +2819,7 @@
                   'is also\n'
                   '   called a destructor.  If a base class has a "__del__()" '
                   'method, the\n'
-                  '   derived class\'s "__del__()" method, if any, must '
+                  '   derived class?s "__del__()" method, if any, must '
                   'explicitly call it\n'
                   '   to ensure proper deletion of the base class part of the '
                   'instance.\n'
@@ -2862,11 +2835,11 @@
                   'exist when\n'
                   '   the interpreter exits.\n'
                   '\n'
-                  '   Note: "del x" doesn\'t directly call "x.__del__()" --- '
-                  'the former\n'
+                  '   Note: "del x" doesn?t directly call "x.__del__()" ? the '
+                  'former\n'
                   '     decrements the reference count for "x" by one, and the '
                   'latter is\n'
-                  '     only called when "x"\'s reference count reaches zero.  '
+                  '     only called when "x"?s reference count reaches zero.  '
                   'Some common\n'
                   '     situations that may prevent the reference count of an '
                   'object from\n'
@@ -2896,7 +2869,7 @@
                   'Circular\n'
                   '     references which are garbage are detected and cleaned '
                   'up when the\n'
-                  "     cyclic garbage collector is enabled (it's on by "
+                  '     cyclic garbage collector is enabled (it?s on by '
                   'default). Refer\n'
                   '     to the documentation for the "gc" module for more '
                   'information\n'
@@ -2935,7 +2908,7 @@
                   'object.__repr__(self)\n'
                   '\n'
                   '   Called by the "repr()" built-in function to compute the '
-                  '"official"\n'
+                  '?official?\n'
                   '   string representation of an object.  If at all possible, '
                   'this\n'
                   '   should look like a valid Python expression that could be '
@@ -2949,7 +2922,7 @@
                   '   value must be a string object. If a class defines '
                   '"__repr__()" but\n'
                   '   not "__str__()", then "__repr__()" is also used when an '
-                  '"informal"\n'
+                  '?informal?\n'
                   '   string representation of instances of that class is '
                   'required.\n'
                   '\n'
@@ -2961,10 +2934,10 @@
                   '\n'
                   '   Called by "str(object)" and the built-in functions '
                   '"format()" and\n'
-                  '   "print()" to compute the "informal" or nicely printable '
+                  '   "print()" to compute the ?informal? or nicely printable '
                   'string\n'
                   '   representation of an object.  The return value must be a '
-                  '*string*\n'
+                  'string\n'
                   '   object.\n'
                   '\n'
                   '   This method differs from "object.__repr__()" in that '
@@ -2988,7 +2961,7 @@
                   '   Called by the "format()" built-in function (and by '
                   'extension, the\n'
                   '   "str.format()" method of class "str") to produce a '
-                  '"formatted"\n'
+                  '?formatted?\n'
                   '   string representation of an object. The "format_spec" '
                   'argument is a\n'
                   '   string that contains a description of the formatting '
@@ -3001,8 +2974,8 @@
                   'or use a\n'
                   '   similar formatting option syntax.\n'
                   '\n'
-                  '   See *Format Specification Mini-Language* for a '
-                  'description of the\n'
+                  '   See Format Specification Mini-Language for a description '
+                  'of the\n'
                   '   standard formatting syntax.\n'
                   '\n'
                   '   The return value must be a string object.\n'
@@ -3018,7 +2991,7 @@
                   'object.__gt__(self, other)\n'
                   'object.__ge__(self, other)\n'
                   '\n'
-                  '   These are the so-called "rich comparison" methods. The\n'
+                  '   These are the so-called ?rich comparison? methods. The\n'
                   '   correspondence between operator symbols and method names '
                   'is as\n'
                   '   follows: "x<y" calls "x.__lt__(y)", "x<=y" calls '
@@ -3065,18 +3038,18 @@
                   '   when the left argument does not support the operation '
                   'but the right\n'
                   '   argument does); rather, "__lt__()" and "__gt__()" are '
-                  "each other's\n"
-                  '   reflection, "__le__()" and "__ge__()" are each other\'s '
+                  'each other?s\n'
+                  '   reflection, "__le__()" and "__ge__()" are each other?s '
                   'reflection,\n'
                   '   and "__eq__()" and "__ne__()" are their own reflection. '
                   'If the\n'
-                  "   operands are of different types, and right operand's "
+                  '   operands are of different types, and right operand?s '
                   'type is a\n'
-                  "   direct or indirect subclass of the left operand's type, "
+                  '   direct or indirect subclass of the left operand?s type, '
                   'the\n'
                   '   reflected method of the right operand has priority, '
                   'otherwise the\n'
-                  "   left operand's method has priority.  Virtual subclassing "
+                  '   left operand?s method has priority.  Virtual subclassing '
                   'is not\n'
                   '   considered.\n'
                   '\n'
@@ -3100,12 +3073,12 @@
                   '          return hash((self.name, self.nick, self.color))\n'
                   '\n'
                   '   Note: "hash()" truncates the value returned from an '
-                  "object's\n"
+                  'object?s\n'
                   '     custom "__hash__()" method to the size of a '
                   '"Py_ssize_t".  This\n'
                   '     is typically 8 bytes on 64-bit builds and 4 bytes on '
                   '32-bit\n'
-                  '     builds. If an object\'s   "__hash__()" must '
+                  '     builds. If an object?s   "__hash__()" must '
                   'interoperate on builds\n'
                   '     of different bit sizes, be sure to check the width on '
                   'all\n'
@@ -3125,8 +3098,8 @@
                   'implement\n'
                   '   "__hash__()", since the implementation of hashable '
                   'collections\n'
-                  "   requires that a key's hash value is immutable (if the "
-                  "object's hash\n"
+                  '   requires that a key?s hash value is immutable (if the '
+                  'object?s hash\n'
                   '   value changes, it will be in the wrong hash bucket).\n'
                   '\n'
                   '   User-defined classes have "__eq__()" and "__hash__()" '
@@ -3171,7 +3144,7 @@
                   '\n'
                   '   Note: By default, the "__hash__()" values of str, bytes '
                   'and\n'
-                  '     datetime objects are "salted" with an unpredictable '
+                  '     datetime objects are ?salted? with an unpredictable '
                   'random value.\n'
                   '     Although they remain constant within an individual '
                   'Python\n'
@@ -3209,9 +3182,8 @@
                   '   neither "__len__()" nor "__bool__()", all its instances '
                   'are\n'
                   '   considered true.\n',
- 'debugger': '\n'
-             '"pdb" --- The Python Debugger\n'
-             '*****************************\n'
+ 'debugger': '"pdb" ? The Python Debugger\n'
+             '***************************\n'
              '\n'
              '**Source code:** Lib/pdb.py\n'
              '\n'
@@ -3229,15 +3201,15 @@
              'debugging\n'
              'and can be called under program control.\n'
              '\n'
-             'The debugger is extensible -- it is actually defined as the '
+             'The debugger is extensible ? it is actually defined as the '
              'class\n'
              '"Pdb". This is currently undocumented but easily understood by '
              'reading\n'
              'the source.  The extension interface uses the modules "bdb" and '
              '"cmd".\n'
              '\n'
-             'The debugger\'s prompt is "(Pdb)". Typical usage to run a '
-             'program under\n'
+             'The debugger?s prompt is "(Pdb)". Typical usage to run a program '
+             'under\n'
              'control of the debugger is:\n'
              '\n'
              '   >>> import pdb\n'
@@ -3269,16 +3241,15 @@
              'post-\n'
              'mortem debugging (or after normal exit of the program), pdb '
              'will\n'
-             "restart the program.  Automatic restarting preserves pdb's state "
+             'restart the program.  Automatic restarting preserves pdb?s state '
              '(such\n'
              'as breakpoints) and in most cases is more useful than quitting '
              'the\n'
-             "debugger upon program's exit.\n"
+             'debugger upon program?s exit.\n'
              '\n'
              'New in version 3.2: "pdb.py" now accepts a "-c" option that '
              'executes\n'
-             'commands as if given in a ".pdbrc" file, see *Debugger '
-             'Commands*.\n'
+             'commands as if given in a ".pdbrc" file, see Debugger Commands.\n'
              '\n'
              'The typical usage to break into the debugger from a running '
              'program is\n'
@@ -3380,7 +3351,7 @@
              'want\n'
              'to access further features, you have to do this yourself:\n'
              '\n'
-             "class class pdb.Pdb(completekey='tab', stdin=None, stdout=None, "
+             "class pdb.Pdb(completekey='tab', stdin=None, stdout=None, "
              'skip=None, nosigint=False)\n'
              '\n'
              '   "Pdb" is the debugger class.\n'
@@ -3447,7 +3418,7 @@
              'the last command was a "list" command, the next 11 lines are '
              'listed.\n'
              '\n'
-             "Commands that the debugger doesn't recognize are assumed to be "
+             'Commands that the debugger doesn?t recognize are assumed to be '
              'Python\n'
              'statements and are executed in the context of the program being\n'
              'debugged.  Python statements can also be prefixed with an '
@@ -3458,9 +3429,9 @@
              'function.\n'
              'When an exception occurs in such a statement, the exception name '
              'is\n'
-             "printed but the debugger's state is not changed.\n"
+             'printed but the debugger?s state is not changed.\n'
              '\n'
-             'The debugger supports *aliases*.  Aliases can have parameters '
+             'The debugger supports aliases.  Aliases can have parameters '
              'which\n'
              'allows one a certain level of adaptability to the context under\n'
              'examination.\n'
@@ -3475,7 +3446,7 @@
              'first\n'
              '";;" pair, even if it is in the middle of a quoted string.\n'
              '\n'
-             'If a file ".pdbrc" exists in the user\'s home directory or in '
+             'If a file ".pdbrc" exists in the user?s home directory or in '
              'the\n'
              'current directory, it is read in and executed as if it had been '
              'typed\n'
@@ -3531,7 +3502,7 @@
              'prefixed\n'
              '   with a filename and a colon, to specify a breakpoint in '
              'another\n'
-             "   file (probably one that hasn't been loaded yet).  The file "
+             '   file (probably one that hasn?t been loaded yet).  The file '
              'is\n'
              '   searched on "sys.path".  Note that each breakpoint is '
              'assigned a\n'
@@ -3639,7 +3610,7 @@
              'which\n'
              '   list to execute.\n'
              '\n'
-             "   If you use the 'silent' command in the command list, the "
+             '   If you use the ?silent? command in the command list, the '
              'usual\n'
              '   message about stopping at a breakpoint is not printed.  This '
              'may be\n'
@@ -3698,13 +3669,13 @@
              'the\n'
              '   bottom-most frame.  This lets you jump back and execute code '
              'again,\n'
-             "   or jump forward to skip code that you don't want to run.\n"
+             '   or jump forward to skip code that you don?t want to run.\n'
              '\n'
-             '   It should be noted that not all jumps are allowed -- for '
-             'instance\n'
-             '   it is not possible to jump into the middle of a "for" loop or '
-             'out\n'
-             '   of a "finally" clause.\n'
+             '   It should be noted that not all jumps are allowed ? for '
+             'instance it\n'
+             '   is not possible to jump into the middle of a "for" loop or '
+             'out of a\n'
+             '   "finally" clause.\n'
              '\n'
              'l(ist) [first[, last]]\n'
              '\n'
@@ -3747,8 +3718,8 @@
              '   value.\n'
              '\n'
              '   Note: "print()" can also be used, but is not a debugger '
-             'command\n'
-             '     --- this executes the Python "print()" function.\n'
+             'command ?\n'
+             '     this executes the Python "print()" function.\n'
              '\n'
              'pp expression\n'
              '\n'
@@ -3870,8 +3841,7 @@
              '[1] Whether a frame is considered to originate in a certain '
              'module\n'
              '    is determined by the "__name__" in the frame globals.\n',
- 'del': '\n'
-        'The "del" statement\n'
+ 'del': 'The "del" statement\n'
         '*******************\n'
         '\n'
         '   del_stmt ::= "del" target_list\n'
@@ -3900,8 +3870,7 @@
         'Changed in version 3.2: Previously it was illegal to delete a name\n'
         'from the local namespace if it occurs as a free variable in a nested\n'
         'block.\n',
- 'dict': '\n'
-         'Dictionary displays\n'
+ 'dict': 'Dictionary displays\n'
          '*******************\n'
          '\n'
          'A dictionary display is a possibly empty series of key/datum pairs\n'
@@ -3920,7 +3889,7 @@
          'dictionary:\n'
          'each key object is used as a key into the dictionary to store the\n'
          'corresponding datum.  This means that you can specify the same key\n'
-         "multiple times in the key/datum list, and the final dictionary's "
+         'multiple times in the key/datum list, and the final dictionary?s '
          'value\n'
          'for that key will be the last one given.\n'
          '\n'
@@ -3934,19 +3903,18 @@
          '\n'
          'A dict comprehension, in contrast to list and set comprehensions,\n'
          'needs two expressions separated with a colon followed by the usual\n'
-         '"for" and "if" clauses. When the comprehension is run, the '
+         '?for? and ?if? clauses. When the comprehension is run, the '
          'resulting\n'
          'key and value elements are inserted in the new dictionary in the '
          'order\n'
          'they are produced.\n'
          '\n'
          'Restrictions on the types of the key values are listed earlier in\n'
-         'section *The standard type hierarchy*.  (To summarize, the key type\n'
+         'section The standard type hierarchy.  (To summarize, the key type\n'
          'should be *hashable*, which excludes all mutable objects.)  Clashes\n'
          'between duplicate keys are not detected; the last datum (textually\n'
          'rightmost in the display) stored for a given key value prevails.\n',
- 'dynamic-features': '\n'
-                     'Interaction with dynamic features\n'
+ 'dynamic-features': 'Interaction with dynamic features\n'
                      '*********************************\n'
                      '\n'
                      'Name resolution of free variables occurs at runtime, not '
@@ -3972,8 +3940,7 @@
                      'override the global and local namespace.  If only one '
                      'namespace is\n'
                      'specified, it is used for both.\n',
- 'else': '\n'
-         'The "if" statement\n'
+ 'else': 'The "if" statement\n'
          '******************\n'
          '\n'
          'The "if" statement is used for conditional execution:\n'
@@ -3984,14 +3951,13 @@
          '\n'
          'It selects exactly one of the suites by evaluating the expressions '
          'one\n'
-         'by one until one is found to be true (see section *Boolean '
-         'operations*\n'
+         'by one until one is found to be true (see section Boolean '
+         'operations\n'
          'for the definition of true and false); then that suite is executed\n'
          '(and no other part of the "if" statement is executed or evaluated).\n'
          'If all expressions are false, the suite of the "else" clause, if\n'
          'present, is executed.\n',
- 'exceptions': '\n'
-               'Exceptions\n'
+ 'exceptions': 'Exceptions\n'
                '**********\n'
                '\n'
                'Exceptions are a means of breaking out of the normal flow of '
@@ -4011,7 +3977,7 @@
                'error (such as division by zero).  A Python program can also\n'
                'explicitly raise an exception with the "raise" statement. '
                'Exception\n'
-               'handlers are specified with the "try" ... "except" statement.  '
+               'handlers are specified with the "try" ? "except" statement.  '
                'The\n'
                '"finally" clause of such a statement can be used to specify '
                'cleanup\n'
@@ -4019,7 +3985,7 @@
                'whether an\n'
                'exception occurred or not in the preceding code.\n'
                '\n'
-               'Python uses the "termination" model of error handling: an '
+               'Python uses the ?termination? model of error handling: an '
                'exception\n'
                'handler can find out what happened and continue execution at '
                'an outer\n'
@@ -4055,10 +4021,10 @@
                'under\n'
                '  multiple versions of the interpreter.\n'
                '\n'
-               'See also the description of the "try" statement in section '
-               '*The try\n'
-               'statement* and "raise" statement in section *The raise '
-               'statement*.\n'
+               'See also the description of the "try" statement in section The '
+               'try\n'
+               'statement and "raise" statement in section The raise '
+               'statement.\n'
                '\n'
                '-[ Footnotes ]-\n'
                '\n'
@@ -4067,8 +4033,7 @@
                '    these operations is not available at the time the module '
                'is\n'
                '    compiled.\n',
- 'execmodel': '\n'
-              'Execution model\n'
+ 'execmodel': 'Execution model\n'
               '***************\n'
               '\n'
               '\n'
@@ -4088,7 +4053,7 @@
               'argument to the interpreter) is a code block.  A script command '
               '(a\n'
               'command specified on the interpreter command line with the '
-              "'**-c**'\n"
+              '?**-c**?\n'
               'option) is a code block.  The string argument passed to the '
               'built-in\n'
               'functions "eval()" and "exec()" is a code block.\n'
@@ -4097,7 +4062,7 @@
               'contains\n'
               'some administrative information (used for debugging) and '
               'determines\n'
-              "where and how execution continues after the code block's "
+              'where and how execution continues after the code block?s '
               'execution has\n'
               'completed.\n'
               '\n'
@@ -4172,7 +4137,7 @@
               'nearest\n'
               'enclosing scope.  The set of all such scopes visible to a code '
               'block\n'
-              "is called the block's *environment*.\n"
+              'is called the block?s *environment*.\n'
               '\n'
               'When a name is not found at all, a "NameError" exception is '
               'raised. If\n'
@@ -4250,7 +4215,7 @@
               'the class. The scope of names defined in a class block is '
               'limited to\n'
               'the class block; it does not extend to the code blocks of '
-              'methods --\n'
+              'methods ?\n'
               'this includes comprehensions and generator expressions since '
               'they are\n'
               'implemented using a function scope.  This means that the '
@@ -4271,7 +4236,7 @@
               'global\n'
               'namespace; this should be a dictionary or a module (in the '
               'latter case\n'
-              "the module's dictionary is used).  By default, when in the "
+              'the module?s dictionary is used).  By default, when in the '
               '"__main__"\n'
               'module, "__builtins__" is the built-in module "builtins"; when '
               'in any\n'
@@ -4335,7 +4300,7 @@
               'error (such as division by zero).  A Python program can also\n'
               'explicitly raise an exception with the "raise" statement. '
               'Exception\n'
-              'handlers are specified with the "try" ... "except" statement.  '
+              'handlers are specified with the "try" ? "except" statement.  '
               'The\n'
               '"finally" clause of such a statement can be used to specify '
               'cleanup\n'
@@ -4343,7 +4308,7 @@
               'whether an\n'
               'exception occurred or not in the preceding code.\n'
               '\n'
-              'Python uses the "termination" model of error handling: an '
+              'Python uses the ?termination? model of error handling: an '
               'exception\n'
               'handler can find out what happened and continue execution at an '
               'outer\n'
@@ -4379,10 +4344,10 @@
               'under\n'
               '  multiple versions of the interpreter.\n'
               '\n'
-              'See also the description of the "try" statement in section *The '
+              'See also the description of the "try" statement in section The '
               'try\n'
-              'statement* and "raise" statement in section *The raise '
-              'statement*.\n'
+              'statement and "raise" statement in section The raise '
+              'statement.\n'
               '\n'
               '-[ Footnotes ]-\n'
               '\n'
@@ -4391,8 +4356,7 @@
               '    these operations is not available at the time the module '
               'is\n'
               '    compiled.\n',
- 'exprlists': '\n'
-              'Expression lists\n'
+ 'exprlists': 'Expression lists\n'
               '****************\n'
               '\n'
               '   expression_list    ::= expression ( "," expression )* [","]\n'
@@ -4424,13 +4388,12 @@
               '(a.k.a. a\n'
               '*singleton*); it is optional in all other cases.  A single '
               'expression\n'
-              "without a trailing comma doesn't create a tuple, but rather "
+              'without a trailing comma doesn?t create a tuple, but rather '
               'yields the\n'
               'value of that expression. (To create an empty tuple, use an '
               'empty pair\n'
               'of parentheses: "()".)\n',
- 'floating': '\n'
-             'Floating point literals\n'
+ 'floating': 'Floating point literals\n'
              '***********************\n'
              '\n'
              'Floating point literals are described by the following lexical\n'
@@ -4458,8 +4421,7 @@
              'is actually an expression composed of the unary operator "-" and '
              'the\n'
              'literal "1".\n',
- 'for': '\n'
-        'The "for" statement\n'
+ 'for': 'The "for" statement\n'
         '*******************\n'
         '\n'
         'The "for" statement is used to iterate over the elements of a '
@@ -4475,7 +4437,7 @@
         'provided by the iterator, in the order returned by the iterator.  '
         'Each\n'
         'item in turn is assigned to the target list using the standard rules\n'
-        'for assignments (see *Assignment statements*), and then the suite is\n'
+        'for assignments (see Assignment statements), and then the suite is\n'
         'executed.  When the items are exhausted (which is immediately when '
         'the\n'
         'sequence is empty or an iterator raises a "StopIteration" '
@@ -4485,8 +4447,7 @@
         'terminates.\n'
         '\n'
         'A "break" statement executed in the first suite terminates the loop\n'
-        'without executing the "else" clause\'s suite.  A "continue" '
-        'statement\n'
+        'without executing the "else" clause?s suite.  A "continue" statement\n'
         'executed in the first suite skips the rest of the suite and '
         'continues\n'
         'with the next item, or with the "else" clause if there is no next\n'
@@ -4508,7 +4469,7 @@
         'Names in the target list are not deleted when the loop is finished,\n'
         'but if the sequence is empty, they will not have been assigned to at\n'
         'all by the loop.  Hint: the built-in function "range()" returns an\n'
-        'iterator of integers suitable to emulate the effect of Pascal\'s "for '
+        'iterator of integers suitable to emulate the effect of Pascal?s "for '
         'i\n'
         ':= a to b do"; e.g., "list(range(3))" returns the list "[0, 1, 2]".\n'
         '\n'
@@ -4531,8 +4492,7 @@
         '\n'
         '     for x in a[:]:\n'
         '         if x < 0: a.remove(x)\n',
- 'formatstrings': '\n'
-                  'Format String Syntax\n'
+ 'formatstrings': 'Format String Syntax\n'
                   '********************\n'
                   '\n'
                   'The "str.format()" method and the "Formatter" class share '
@@ -4541,7 +4501,7 @@
                   '"Formatter",\n'
                   'subclasses can define their own format string syntax).\n'
                   '\n'
-                  'Format strings contain "replacement fields" surrounded by '
+                  'Format strings contain ?replacement fields? surrounded by '
                   'curly braces\n'
                   '"{}". Anything that is not contained in braces is '
                   'considered literal\n'
@@ -4580,33 +4540,31 @@
                   'format for the\n'
                   'replacement value.\n'
                   '\n'
-                  'See also the *Format Specification Mini-Language* section.\n'
+                  'See also the Format Specification Mini-Language section.\n'
                   '\n'
                   'The *field_name* itself begins with an *arg_name* that is '
                   'either a\n'
-                  "number or a keyword.  If it's a number, it refers to a "
+                  'number or a keyword.  If it?s a number, it refers to a '
                   'positional\n'
-                  "argument, and if it's a keyword, it refers to a named "
+                  'argument, and if it?s a keyword, it refers to a named '
                   'keyword\n'
                   'argument.  If the numerical arg_names in a format string '
                   'are 0, 1, 2,\n'
-                  '... in sequence, they can all be omitted (not just some) '
-                  'and the\n'
-                  'numbers 0, 1, 2, ... will be automatically inserted in that '
-                  'order.\n'
-                  'Because *arg_name* is not quote-delimited, it is not '
-                  'possible to\n'
-                  'specify arbitrary dictionary keys (e.g., the strings '
-                  '"\'10\'" or\n'
-                  '"\':-]\'") within a format string. The *arg_name* can be '
-                  'followed by any\n'
-                  'number of index or attribute expressions. An expression of '
-                  'the form\n'
-                  '"\'.name\'" selects the named attribute using "getattr()", '
-                  'while an\n'
-                  'expression of the form "\'[index]\'" does an index lookup '
-                  'using\n'
-                  '"__getitem__()".\n'
+                  '? in sequence, they can all be omitted (not just some) and '
+                  'the numbers\n'
+                  '0, 1, 2, ? will be automatically inserted in that order. '
+                  'Because\n'
+                  '*arg_name* is not quote-delimited, it is not possible to '
+                  'specify\n'
+                  'arbitrary dictionary keys (e.g., the strings "\'10\'" or '
+                  '"\':-]\'") within\n'
+                  'a format string. The *arg_name* can be followed by any '
+                  'number of index\n'
+                  'or attribute expressions. An expression of the form '
+                  '"\'.name\'" selects\n'
+                  'the named attribute using "getattr()", while an expression '
+                  'of the form\n'
+                  '"\'[index]\'" does an index lookup using "__getitem__()".\n'
                   '\n'
                   'Changed in version 3.1: The positional argument specifiers '
                   'can be\n'
@@ -4661,7 +4619,7 @@
                   'alignment,\n'
                   'padding, decimal precision and so on.  Each value type can '
                   'define its\n'
-                  'own "formatting mini-language" or interpretation of the '
+                  'own ?formatting mini-language? or interpretation of the '
                   '*format_spec*.\n'
                   '\n'
                   'Most built-in types support a common formatting '
@@ -4681,17 +4639,17 @@
                   'allows the formatting of a value to be dynamically '
                   'specified.\n'
                   '\n'
-                  'See the *Format examples* section for some examples.\n'
+                  'See the Format examples section for some examples.\n'
                   '\n'
                   '\n'
                   'Format Specification Mini-Language\n'
                   '==================================\n'
                   '\n'
-                  '"Format specifications" are used within replacement fields '
+                  '?Format specifications? are used within replacement fields '
                   'contained\n'
                   'within a format string to define how individual values are '
                   'presented\n'
-                  '(see *Format String Syntax*).  They can also be passed '
+                  '(see Format String Syntax).  They can also be passed '
                   'directly to the\n'
                   'built-in "format()" function.  Each formattable type may '
                   'define how\n'
@@ -4726,12 +4684,12 @@
                   'character that can be any character and defaults to a space '
                   'if\n'
                   'omitted. It is not possible to use a literal curly brace '
-                  '(""{"" or\n'
-                  '""}"") as the *fill* character when using the '
+                  '(?"{"? or\n'
+                  '?"}"?) as the *fill* character when using the '
                   '"str.format()" method.\n'
                   'However, it is possible to insert a curly brace with a '
                   'nested\n'
-                  "replacement field.  This limitation doesn't affect the "
+                  'replacement field.  This limitation doesn?t affect the '
                   '"format()"\n'
                   'function.\n'
                   '\n'
@@ -4747,18 +4705,33 @@
                   '+===========+============================================================+\n'
                   '   | "\'<\'"     | Forces the field to be left-aligned '
                   'within the available   |\n'
+                  '   |           | space (this is the default for most '
+                  'objects).              |\n'
                   '   '
                   '+-----------+------------------------------------------------------------+\n'
                   '   | "\'>\'"     | Forces the field to be right-aligned '
                   'within the available  |\n'
+                  '   |           | space (this is the default for '
+                  'numbers).                   |\n'
                   '   '
                   '+-----------+------------------------------------------------------------+\n'
                   '   | "\'=\'"     | Forces the padding to be placed after '
                   'the sign (if any)    |\n'
+                  '   |           | but before the digits.  This is used for '
+                  'printing fields   |\n'
+                  '   |           | in the form ?+000000120?. This alignment '
+                  'option is only    |\n'
+                  '   |           | valid for numeric types.  It becomes the '
+                  'default when ?0?  |\n'
+                  '   |           | immediately precedes the field '
+                  'width.                      |\n'
                   '   '
                   '+-----------+------------------------------------------------------------+\n'
                   '   | "\'^\'"     | Forces the field to be centered within '
                   'the available       |\n'
+                  '   |           | '
+                  'space.                                                     '
+                  '|\n'
                   '   '
                   '+-----------+------------------------------------------------------------+\n'
                   '\n'
@@ -4781,18 +4754,24 @@
                   '+===========+============================================================+\n'
                   '   | "\'+\'"     | indicates that a sign should be used for '
                   'both positive as  |\n'
+                  '   |           | well as negative '
+                  'numbers.                                  |\n'
                   '   '
                   '+-----------+------------------------------------------------------------+\n'
                   '   | "\'-\'"     | indicates that a sign should be used '
                   'only for negative     |\n'
+                  '   |           | numbers (this is the default '
+                  'behavior).                    |\n'
                   '   '
                   '+-----------+------------------------------------------------------------+\n'
                   '   | space     | indicates that a leading space should be '
                   'used on positive  |\n'
+                  '   |           | numbers, and a minus sign on negative '
+                  'numbers.             |\n'
                   '   '
                   '+-----------+------------------------------------------------------------+\n'
                   '\n'
-                  'The "\'#\'" option causes the "alternate form" to be used '
+                  'The "\'#\'" option causes the ?alternate form? to be used '
                   'for the\n'
                   'conversion.  The alternate form is defined differently for '
                   'different\n'
@@ -4865,6 +4844,8 @@
                   '+===========+============================================================+\n'
                   '   | "\'s\'"     | String format. This is the default type '
                   'for strings and    |\n'
+                  '   |           | may be '
+                  'omitted.                                            |\n'
                   '   '
                   '+-----------+------------------------------------------------------------+\n'
                   '   | None      | The same as '
@@ -4887,6 +4868,8 @@
                   '+-----------+------------------------------------------------------------+\n'
                   '   | "\'c\'"     | Character. Converts the integer to the '
                   'corresponding       |\n'
+                  '   |           | unicode character before '
+                  'printing.                         |\n'
                   '   '
                   '+-----------+------------------------------------------------------------+\n'
                   '   | "\'d\'"     | Decimal Integer. Outputs the number in '
@@ -4899,14 +4882,22 @@
                   '+-----------+------------------------------------------------------------+\n'
                   '   | "\'x\'"     | Hex format. Outputs the number in base '
                   '16, using lower-    |\n'
+                  '   |           | case letters for the digits above '
+                  '9.                       |\n'
                   '   '
                   '+-----------+------------------------------------------------------------+\n'
                   '   | "\'X\'"     | Hex format. Outputs the number in base '
                   '16, using upper-    |\n'
+                  '   |           | case letters for the digits above '
+                  '9.                       |\n'
                   '   '
                   '+-----------+------------------------------------------------------------+\n'
                   '   | "\'n\'"     | Number. This is the same as "\'d\'", '
                   'except that it uses the |\n'
+                  '   |           | current locale setting to insert the '
+                  'appropriate number    |\n'
+                  '   |           | separator '
+                  'characters.                                      |\n'
                   '   '
                   '+-----------+------------------------------------------------------------+\n'
                   '   | None      | The same as '
@@ -4935,38 +4926,103 @@
                   '+===========+============================================================+\n'
                   '   | "\'e\'"     | Exponent notation. Prints the number in '
                   'scientific         |\n'
+                  '   |           | notation using the letter ?e? to indicate '
+                  'the exponent.    |\n'
+                  '   |           | The default precision is '
+                  '"6".                              |\n'
                   '   '
                   '+-----------+------------------------------------------------------------+\n'
                   '   | "\'E\'"     | Exponent notation. Same as "\'e\'" '
                   'except it uses an upper   |\n'
+                  '   |           | case ?E? as the separator '
+                  'character.                       |\n'
                   '   '
                   '+-----------+------------------------------------------------------------+\n'
                   '   | "\'f\'"     | Fixed point. Displays the number as a '
                   'fixed-point number.  |\n'
+                  '   |           | The default precision is '
+                  '"6".                              |\n'
                   '   '
                   '+-----------+------------------------------------------------------------+\n'
                   '   | "\'F\'"     | Fixed point. Same as "\'f\'", but '
                   'converts "nan" to "NAN"    |\n'
+                  '   |           | and "inf" to '
+                  '"INF".                                        |\n'
                   '   '
                   '+-----------+------------------------------------------------------------+\n'
                   '   | "\'g\'"     | General format.  For a given precision '
                   '"p >= 1", this      |\n'
+                  '   |           | rounds the number to "p" significant '
+                  'digits and then       |\n'
+                  '   |           | formats the result in either fixed-point '
+                  'format or in      |\n'
+                  '   |           | scientific notation, depending on its '
+                  'magnitude.  The      |\n'
+                  '   |           | precise rules are as follows: suppose that '
+                  'the result      |\n'
+                  '   |           | formatted with presentation type "\'e\'" '
+                  'and precision "p-1" |\n'
+                  '   |           | would have exponent "exp".  Then if "-4 <= '
+                  'exp < p", the   |\n'
+                  '   |           | number is formatted with presentation type '
+                  '"\'f\'" and       |\n'
+                  '   |           | precision "p-1-exp".  Otherwise, the '
+                  'number is formatted   |\n'
+                  '   |           | with presentation type "\'e\'" and '
+                  'precision "p-1". In both  |\n'
+                  '   |           | cases insignificant trailing zeros are '
+                  'removed from the    |\n'
+                  '   |           | significand, and the decimal point is also '
+                  'removed if      |\n'
+                  '   |           | there are no remaining digits following '
+                  'it.  Positive and  |\n'
+                  '   |           | negative infinity, positive and negative '
+                  'zero, and nans,   |\n'
+                  '   |           | are formatted as "inf", "-inf", "0", "-0" '
+                  'and "nan"        |\n'
+                  '   |           | respectively, regardless of the '
+                  'precision.  A precision of |\n'
+                  '   |           | "0" is treated as equivalent to a '
+                  'precision of "1". The    |\n'
+                  '   |           | default precision is '
+                  '"6".                                  |\n'
                   '   '
                   '+-----------+------------------------------------------------------------+\n'
                   '   | "\'G\'"     | General format. Same as "\'g\'" except '
                   'switches to "\'E\'" if  |\n'
+                  '   |           | the number gets too large. The '
+                  'representations of infinity |\n'
+                  '   |           | and NaN are uppercased, '
+                  'too.                               |\n'
                   '   '
                   '+-----------+------------------------------------------------------------+\n'
                   '   | "\'n\'"     | Number. This is the same as "\'g\'", '
                   'except that it uses the |\n'
+                  '   |           | current locale setting to insert the '
+                  'appropriate number    |\n'
+                  '   |           | separator '
+                  'characters.                                      |\n'
                   '   '
                   '+-----------+------------------------------------------------------------+\n'
                   '   | "\'%\'"     | Percentage. Multiplies the number by 100 '
                   'and displays in   |\n'
+                  '   |           | fixed ("\'f\'") format, followed by a '
+                  'percent sign.          |\n'
                   '   '
                   '+-----------+------------------------------------------------------------+\n'
                   '   | None      | Similar to "\'g\'", except that '
                   'fixed-point notation, when   |\n'
+                  '   |           | used, has at least one digit past the '
+                  'decimal point. The   |\n'
+                  '   |           | default precision is as high as needed to '
+                  'represent the    |\n'
+                  '   |           | particular value. The overall effect is to '
+                  'match the       |\n'
+                  '   |           | output of "str()" as altered by the other '
+                  'format           |\n'
+                  '   |           | '
+                  'modifiers.                                                 '
+                  '|\n'
                   '   '
                   '+-----------+------------------------------------------------------------+\n'
                   '\n'
@@ -5015,7 +5071,7 @@
                   "{longitude}'.format(**coord)\n"
                   "   'Coordinates: 37.24N, -115.81W'\n"
                   '\n'
-                  "Accessing arguments' attributes:\n"
+                  'Accessing arguments? attributes:\n'
                   '\n'
                   '   >>> c = 3-5j\n'
                   "   >>> ('The complex number {0} is formed from the real "
@@ -5033,7 +5089,7 @@
                   '   >>> str(Point(4, 2))\n'
                   "   'Point(4, 2)'\n"
                   '\n'
-                  "Accessing arguments' items:\n"
+                  'Accessing arguments? items:\n'
                   '\n'
                   '   >>> coord = (3, 5)\n'
                   "   >>> 'X: {0[0]};  Y: {0[1]}'.format(coord)\n"
@@ -5118,8 +5174,7 @@
                   '   3232235521\n'
                   '   >>>\n'
                   '   >>> width = 5\n'
-                  '   >>> for num in range(5,12): #doctest: '
-                  '+NORMALIZE_WHITESPACE\n'
+                  '   >>> for num in range(5,12): \n'
                   "   ...     for base in 'dXob':\n"
                   "   ...         print('{0:{width}{base}}'.format(num, "
                   "base=base, width=width), end=' ')\n"
@@ -5132,13 +5187,12 @@
                   '       9     9    11  1001\n'
                   '      10     A    12  1010\n'
                   '      11     B    13  1011\n',
- 'function': '\n'
-             'Function definitions\n'
+ 'function': 'Function definitions\n'
              '********************\n'
              '\n'
              'A function definition defines a user-defined function object '
              '(see\n'
-             'section *The standard type hierarchy*):\n'
+             'section The standard type hierarchy):\n'
              '\n'
              '   funcdef        ::= [decorators] "def" funcname "(" '
              '[parameter_list] ")" ["->" expression] ":" suite\n'
@@ -5195,25 +5249,24 @@
              '"func".\n'
              '\n'
              'When one or more *parameters* have the form *parameter* "="\n'
-             '*expression*, the function is said to have "default parameter '
-             'values."\n'
+             '*expression*, the function is said to have ?default parameter '
+             'values.?\n'
              'For a parameter with a default value, the corresponding '
              '*argument* may\n'
-             "be omitted from a call, in which case the parameter's default "
+             'be omitted from a call, in which case the parameter?s default '
              'value is\n'
              'substituted.  If a parameter has a default value, all following\n'
-             'parameters up until the ""*"" must also have a default value --- '
-             'this\n'
-             'is a syntactic restriction that is not expressed by the '
-             'grammar.\n'
+             'parameters up until the ?"*"? must also have a default value ? '
+             'this is\n'
+             'a syntactic restriction that is not expressed by the grammar.\n'
              '\n'
              '**Default parameter values are evaluated from left to right when '
              'the\n'
              'function definition is executed.** This means that the '
              'expression is\n'
              'evaluated once, when the function is defined, and that the same '
-             '"pre-\n'
-             'computed" value is used for each call.  This is especially '
+             '?pre-\n'
+             'computed? value is used for each call.  This is especially '
              'important\n'
              'to understand when a default parameter is a mutable object, such '
              'as a\n'
@@ -5233,30 +5286,32 @@
              '       penguin.append("property of the zoo")\n'
              '       return penguin\n'
              '\n'
-             'Function call semantics are described in more detail in section\n'
-             '*Calls*. A function call always assigns values to all '
-             'parameters\n'
-             'mentioned in the parameter list, either from position arguments, '
-             'from\n'
-             'keyword arguments, or from default values.  If the form\n'
-             '""*identifier"" is present, it is initialized to a tuple '
-             'receiving any\n'
-             'excess positional parameters, defaulting to the empty tuple.  If '
-             'the\n'
-             'form ""**identifier"" is present, it is initialized to a new\n'
-             'dictionary receiving any excess keyword arguments, defaulting to '
-             'a new\n'
-             'empty dictionary. Parameters after ""*"" or ""*identifier"" are\n'
-             'keyword-only parameters and may only be passed used keyword '
-             'arguments.\n'
-             '\n'
-             'Parameters may have annotations of the form "": expression"" '
+             'Function call semantics are described in more detail in section '
+             'Calls.\n'
+             'A function call always assigns values to all parameters '
+             'mentioned in\n'
+             'the parameter list, either from position arguments, from '
+             'keyword\n'
+             'arguments, or from default values.  If the form ?"*identifier"? '
+             'is\n'
+             'present, it is initialized to a tuple receiving any excess '
+             'positional\n'
+             'parameters, defaulting to the empty tuple.  If the form\n'
+             '?"**identifier"? is present, it is initialized to a new '
+             'dictionary\n'
+             'receiving any excess keyword arguments, defaulting to a new '
+             'empty\n'
+             'dictionary. Parameters after ?"*"? or ?"*identifier"? are '
+             'keyword-only\n'
+             'parameters and may only be passed used keyword arguments.\n'
+             '\n'
+             'Parameters may have annotations of the form ?": expression"? '
              'following\n'
              'the parameter name.  Any parameter may have an annotation even '
              'those\n'
              'of the form "*identifier" or "**identifier".  Functions may '
              'have\n'
-             '"return" annotation of the form ""-> expression"" after the '
+             '?return? annotation of the form ?"-> expression"? after the '
              'parameter\n'
              'list.  These annotations can be any valid Python expression and '
              'are\n'
@@ -5266,40 +5321,40 @@
              'code.\n'
              'The presence of annotations does not change the semantics of a\n'
              'function.  The annotation values are available as values of a\n'
-             "dictionary keyed by the parameters' names in the "
+             'dictionary keyed by the parameters? names in the '
              '"__annotations__"\n'
              'attribute of the function object.\n'
              '\n'
              'It is also possible to create anonymous functions (functions not '
              'bound\n'
              'to a name), for immediate use in expressions.  This uses lambda\n'
-             'expressions, described in section *Lambdas*.  Note that the '
+             'expressions, described in section Lambdas.  Note that the '
              'lambda\n'
              'expression is merely a shorthand for a simplified function '
              'definition;\n'
-             'a function defined in a ""def"" statement can be passed around '
+             'a function defined in a ?"def"? statement can be passed around '
              'or\n'
              'assigned to another name just like a function defined by a '
              'lambda\n'
-             'expression.  The ""def"" form is actually more powerful since '
+             'expression.  The ?"def"? form is actually more powerful since '
              'it\n'
              'allows the execution of multiple statements and annotations.\n'
              '\n'
-             "**Programmer's note:** Functions are first-class objects.  A "
-             '""def""\n'
+             '**Programmer?s note:** Functions are first-class objects.  A '
+             '?"def"?\n'
              'statement executed inside a function definition defines a local\n'
              'function that can be returned or passed around.  Free variables '
              'used\n'
              'in the nested function can access the local variables of the '
              'function\n'
-             'containing the def.  See section *Naming and binding* for '
+             'containing the def.  See section Naming and binding for '
              'details.\n'
              '\n'
-             'See also: **PEP 3107** - Function Annotations\n'
+             'See also:\n'
              '\n'
+             '  **PEP 3107** - Function Annotations\n'
              '     The original specification for function annotations.\n',
- 'global': '\n'
-           'The "global" statement\n'
+ 'global': 'The "global" statement\n'
            '**********************\n'
            '\n'
            '   global_stmt ::= "global" identifier ("," identifier)*\n'
@@ -5329,8 +5384,7 @@
            'change\n'
            'the meaning of the program.\n'
            '\n'
-           '**Programmer\'s note:** "global" is a directive to the parser.  '
-           'It\n'
+           '**Programmer?s note:** "global" is a directive to the parser.  It\n'
            'applies only to code parsed at the same time as the "global"\n'
            'statement. In particular, a "global" statement contained in a '
            'string\n'
@@ -5342,8 +5396,7 @@
            'code containing the function call.  The same applies to the '
            '"eval()"\n'
            'and "compile()" functions.\n',
- 'id-classes': '\n'
-               'Reserved classes of identifiers\n'
+ 'id-classes': 'Reserved classes of identifiers\n'
                '*******************************\n'
                '\n'
                'Certain classes of identifiers (besides keywords) have '
@@ -5361,7 +5414,7 @@
                'When not\n'
                '   in interactive mode, "_" has no special meaning and is not '
                'defined.\n'
-               '   See section *The import statement*.\n'
+               '   See section The import statement.\n'
                '\n'
                '   Note: The name "_" is often used in conjunction with\n'
                '     internationalization; refer to the documentation for the\n'
@@ -5373,10 +5426,10 @@
                'interpreter\n'
                '   and its implementation (including the standard library).  '
                'Current\n'
-               '   system names are discussed in the *Special method names* '
-               'section\n'
-               '   and elsewhere.  More will likely be defined in future '
-               'versions of\n'
+               '   system names are discussed in the Special method names '
+               'section and\n'
+               '   elsewhere.  More will likely be defined in future versions '
+               'of\n'
                '   Python.  *Any* use of "__*__" names, in any context, that '
                'does not\n'
                '   follow explicitly documented use, is subject to breakage '
@@ -5388,11 +5441,10 @@
                'within the\n'
                '   context of a class definition, are re-written to use a '
                'mangled form\n'
-               '   to help avoid name clashes between "private" attributes of '
+               '   to help avoid name clashes between ?private? attributes of '
                'base and\n'
-               '   derived classes. See section *Identifiers (Names)*.\n',
- 'identifiers': '\n'
-                'Identifiers and keywords\n'
+               '   derived classes. See section Identifiers (Names).\n',
+ 'identifiers': 'Identifiers and keywords\n'
                 '************************\n'
                 '\n'
                 'Identifiers (also referred to as *names*) are described by '
@@ -5509,7 +5561,7 @@
                 'When not\n'
                 '   in interactive mode, "_" has no special meaning and is not '
                 'defined.\n'
-                '   See section *The import statement*.\n'
+                '   See section The import statement.\n'
                 '\n'
                 '   Note: The name "_" is often used in conjunction with\n'
                 '     internationalization; refer to the documentation for '
@@ -5522,10 +5574,10 @@
                 'interpreter\n'
                 '   and its implementation (including the standard library).  '
                 'Current\n'
-                '   system names are discussed in the *Special method names* '
-                'section\n'
-                '   and elsewhere.  More will likely be defined in future '
-                'versions of\n'
+                '   system names are discussed in the Special method names '
+                'section and\n'
+                '   elsewhere.  More will likely be defined in future versions '
+                'of\n'
                 '   Python.  *Any* use of "__*__" names, in any context, that '
                 'does not\n'
                 '   follow explicitly documented use, is subject to breakage '
@@ -5537,11 +5589,10 @@
                 'within the\n'
                 '   context of a class definition, are re-written to use a '
                 'mangled form\n'
-                '   to help avoid name clashes between "private" attributes of '
+                '   to help avoid name clashes between ?private? attributes of '
                 'base and\n'
-                '   derived classes. See section *Identifiers (Names)*.\n',
- 'if': '\n'
-       'The "if" statement\n'
+                '   derived classes. See section Identifiers (Names).\n',
+ 'if': 'The "if" statement\n'
        '******************\n'
        '\n'
        'The "if" statement is used for conditional execution:\n'
@@ -5552,14 +5603,12 @@
        '\n'
        'It selects exactly one of the suites by evaluating the expressions '
        'one\n'
-       'by one until one is found to be true (see section *Boolean '
-       'operations*\n'
+       'by one until one is found to be true (see section Boolean operations\n'
        'for the definition of true and false); then that suite is executed\n'
        '(and no other part of the "if" statement is executed or evaluated).\n'
        'If all expressions are false, the suite of the "else" clause, if\n'
        'present, is executed.\n',
- 'imaginary': '\n'
-              'Imaginary literals\n'
+ 'imaginary': 'Imaginary literals\n'
               '******************\n'
               '\n'
               'Imaginary literals are described by the following lexical '
@@ -5578,8 +5627,7 @@
               'e.g., "(3+4j)".  Some examples of imaginary literals:\n'
               '\n'
               '   3.14j   10.j    10j     .001j   1e100j  3.14e-10j\n',
- 'import': '\n'
-           'The "import" statement\n'
+ 'import': 'The "import" statement\n'
            '**********************\n'
            '\n'
            '   import_stmt     ::= "import" module ["as" name] ( "," module '
@@ -5611,17 +5659,17 @@
            'statements.\n'
            '\n'
            'The details of the first step, finding and loading modules are\n'
-           'described in greater detail in the section on the *import '
-           'system*,\n'
-           'which also describes the various types of packages and modules '
-           'that\n'
-           'can be imported, as well as all the hooks that can be used to\n'
-           'customize the import system. Note that failures in this step may\n'
-           'indicate either that the module could not be located, *or* that '
-           'an\n'
-           'error occurred while initializing the module, which includes '
-           'execution\n'
-           "of the module's code.\n"
+           'described in greater detail in the section on the import system, '
+           'which\n'
+           'also describes the various types of packages and modules that can '
+           'be\n'
+           'imported, as well as all the hooks that can be used to customize '
+           'the\n'
+           'import system. Note that failures in this step may indicate '
+           'either\n'
+           'that the module could not be located, *or* that an error occurred\n'
+           'while initializing the module, which includes execution of the\n'
+           'module?s code.\n'
            '\n'
            'If the requested module is retrieved successfully, it will be '
            'made\n'
@@ -5632,7 +5680,7 @@
            '\n'
            '* If no other name is specified, and the module being imported is '
            'a\n'
-           "  top level module, the module's name is bound in the local "
+           '  top level module, the module?s name is bound in the local '
            'namespace\n'
            '  as a reference to the imported module\n'
            '\n'
@@ -5685,7 +5733,7 @@
            '\n'
            'The *public names* defined by a module are determined by checking '
            'the\n'
-           'module\'s namespace for a variable named "__all__"; if defined, it '
+           'module?s namespace for a variable named "__all__"; if defined, it '
            'must\n'
            'be a sequence of strings which are names defined or imported by '
            'that\n'
@@ -5693,7 +5741,7 @@
            'and\n'
            'are required to exist.  If "__all__" is not defined, the set of '
            'public\n'
-           "names includes all names found in the module's namespace which do "
+           'names includes all names found in the module?s namespace which do '
            'not\n'
            'begin with an underscore character ("\'_\'").  "__all__" should '
            'contain\n'
@@ -5703,8 +5751,7 @@
            'were\n'
            'imported and used within the module).\n'
            '\n'
-           'The wild card form of import --- "from module import *" --- is '
-           'only\n'
+           'The wild card form of import ? "from module import *" ? is only\n'
            'allowed at the module level.  Attempting to use it in class or\n'
            'function definitions will raise a "SyntaxError".\n'
            '\n'
@@ -5812,7 +5859,7 @@
            '\n'
            '   import __future__ [as name]\n'
            '\n'
-           "That is not a future statement; it's an ordinary import statement "
+           'That is not a future statement; it?s an ordinary import statement '
            'with\n'
            'no special semantics or syntax restrictions.\n'
            '\n'
@@ -5823,24 +5870,23 @@
            'the\n'
            'future statement.  This can be controlled by optional arguments '
            'to\n'
-           '"compile()" --- see the documentation of that function for '
-           'details.\n'
+           '"compile()" ? see the documentation of that function for details.\n'
            '\n'
            'A future statement typed at an interactive interpreter prompt '
            'will\n'
            'take effect for the rest of the interpreter session.  If an\n'
-           'interpreter is started with the *-i* option, is passed a script '
+           'interpreter is started with the "-i" option, is passed a script '
            'name\n'
            'to execute, and the script includes a future statement, it will be '
            'in\n'
            'effect in the interactive session started after the script is\n'
            'executed.\n'
            '\n'
-           'See also: **PEP 236** - Back to the __future__\n'
+           'See also:\n'
            '\n'
+           '  **PEP 236** - Back to the __future__\n'
            '     The original proposal for the __future__ mechanism.\n',
- 'in': '\n'
-       'Membership test operations\n'
+ 'in': 'Membership test operations\n'
        '**************************\n'
        '\n'
        'The operators "in" and "not in" test for membership.  "x in s"\n'
@@ -5876,8 +5922,7 @@
        '\n'
        'The operator "not in" is defined to have the inverse true value of\n'
        '"in".\n',
- 'integers': '\n'
-             'Integer literals\n'
+ 'integers': 'Integer literals\n'
              '****************\n'
              '\n'
              'Integer literals are described by the following lexical '
@@ -5909,8 +5954,7 @@
              '\n'
              '   7     2147483647                        0o177    0b100110111\n'
              '   3     79228162514264337593543950336     0o377    0xdeadbeef\n',
- 'lambda': '\n'
-           'Lambdas\n'
+ 'lambda': 'Lambdas\n'
            '*******\n'
            '\n'
            '   lambda_expr        ::= "lambda" [parameter_list]: expression\n'
@@ -5928,13 +5972,12 @@
            '   def <lambda>(arguments):\n'
            '       return expression\n'
            '\n'
-           'See section *Function definitions* for the syntax of parameter '
+           'See section Function definitions for the syntax of parameter '
            'lists.\n'
            'Note that functions created with lambda expressions cannot '
            'contain\n'
            'statements or annotations.\n',
- 'lists': '\n'
-          'List displays\n'
+ 'lists': 'List displays\n'
           '*************\n'
           '\n'
           'A list display is a possibly empty series of expressions enclosed '
@@ -5951,8 +5994,7 @@
           'from left to right and placed into the list object in that order.\n'
           'When a comprehension is supplied, the list is constructed from the\n'
           'elements resulting from the comprehension.\n',
- 'naming': '\n'
-           'Naming and binding\n'
+ 'naming': 'Naming and binding\n'
            '******************\n'
            '\n'
            '\n'
@@ -6020,7 +6062,7 @@
            'nearest\n'
            'enclosing scope.  The set of all such scopes visible to a code '
            'block\n'
-           "is called the block's *environment*.\n"
+           'is called the block?s *environment*.\n'
            '\n'
            'When a name is not found at all, a "NameError" exception is '
            'raised. If\n'
@@ -6094,7 +6136,7 @@
            'the class. The scope of names defined in a class block is limited '
            'to\n'
            'the class block; it does not extend to the code blocks of methods '
-           '--\n'
+           '?\n'
            'this includes comprehensions and generator expressions since they '
            'are\n'
            'implemented using a function scope.  This means that the '
@@ -6115,7 +6157,7 @@
            'global\n'
            'namespace; this should be a dictionary or a module (in the latter '
            'case\n'
-           "the module's dictionary is used).  By default, when in the "
+           'the module?s dictionary is used).  By default, when in the '
            '"__main__"\n'
            'module, "__builtins__" is the built-in module "builtins"; when in '
            'any\n'
@@ -6157,8 +6199,7 @@
            'override the global and local namespace.  If only one namespace '
            'is\n'
            'specified, it is used for both.\n',
- 'nonlocal': '\n'
-             'The "nonlocal" statement\n'
+ 'nonlocal': 'The "nonlocal" statement\n'
              '************************\n'
              '\n'
              '   nonlocal_stmt ::= "nonlocal" identifier ("," identifier)*\n'
@@ -6185,11 +6226,11 @@
              'pre-\n'
              'existing bindings in the local scope.\n'
              '\n'
-             'See also: **PEP 3104** - Access to Names in Outer Scopes\n'
+             'See also:\n'
              '\n'
+             '  **PEP 3104** - Access to Names in Outer Scopes\n'
              '     The specification for the "nonlocal" statement.\n',
- 'numbers': '\n'
-            'Numeric literals\n'
+ 'numbers': 'Numeric literals\n'
             '****************\n'
             '\n'
             'There are three types of numeric literals: integers, floating '
@@ -6200,11 +6241,10 @@
             '\n'
             'Note that numeric literals do not include a sign; a phrase like '
             '"-1"\n'
-            'is actually an expression composed of the unary operator \'"-"\' '
+            'is actually an expression composed of the unary operator ?"-"? '
             'and the\n'
             'literal "1".\n',
- 'numeric-types': '\n'
-                  'Emulating numeric types\n'
+ 'numeric-types': 'Emulating numeric types\n'
                   '***********************\n'
                   '\n'
                   'The following methods can be defined to emulate numeric '
@@ -6291,15 +6331,15 @@
                   '"__rpow__()" (the\n'
                   '   coercion rules would become too complicated).\n'
                   '\n'
-                  "   Note: If the right operand's type is a subclass of the "
+                  '   Note: If the right operand?s type is a subclass of the '
                   'left\n'
-                  "     operand's type and that subclass provides the "
+                  '     operand?s type and that subclass provides the '
                   'reflected method\n'
                   '     for the operation, this method will be called before '
                   'the left\n'
-                  "     operand's non-reflected method.  This behavior allows "
+                  '     operand?s non-reflected method.  This behavior allows '
                   'subclasses\n'
-                  "     to override their ancestors' operations.\n"
+                  '     to override their ancestors? operations.\n'
                   '\n'
                   'object.__iadd__(self, other)\n'
                   'object.__isub__(self, other)\n'
@@ -6337,9 +6377,9 @@
                   'certain\n'
                   '   situations, augmented assignment can result in '
                   'unexpected errors\n'
-                  "   (see *Why does a_tuple[i] += ['item'] raise an exception "
+                  '   (see Why does a_tuple[i] += [?item?] raise an exception '
                   'when the\n'
-                  '   addition works?*), but this behavior is in fact part of '
+                  '   addition works?), but this behavior is in fact part of '
                   'the data\n'
                   '   model.\n'
                   '\n'
@@ -6380,22 +6420,21 @@
                   '     "__index__()" is defined "__int__()" should also be '
                   'defined, and\n'
                   '     both should return the same value.\n',
- 'objects': '\n'
-            'Objects, values and types\n'
+ 'objects': 'Objects, values and types\n'
             '*************************\n'
             '\n'
-            "*Objects* are Python's abstraction for data.  All data in a "
+            '*Objects* are Python?s abstraction for data.  All data in a '
             'Python\n'
             'program is represented by objects or by relations between '
             'objects. (In\n'
-            'a sense, and in conformance to Von Neumann\'s model of a "stored\n'
-            'program computer," code is also represented by objects.)\n'
+            'a sense, and in conformance to Von Neumann?s model of a ?stored\n'
+            'program computer,? code is also represented by objects.)\n'
             '\n'
-            "Every object has an identity, a type and a value.  An object's\n"
+            'Every object has an identity, a type and a value.  An object?s\n'
             '*identity* never changes once it has been created; you may think '
             'of it\n'
-            'as the object\'s address in memory.  The \'"is"\' operator '
-            'compares the\n'
+            'as the object?s address in memory.  The ?"is"? operator compares '
+            'the\n'
             'identity of two objects; the "id()" function returns an integer\n'
             'representing its identity.\n'
             '\n'
@@ -6403,14 +6442,14 @@
             'memory\n'
             'address where "x" is stored.\n'
             '\n'
-            "An object's type determines the operations that the object "
+            'An object?s type determines the operations that the object '
             'supports\n'
-            '(e.g., "does it have a length?") and also defines the possible '
+            '(e.g., ?does it have a length??) and also defines the possible '
             'values\n'
             'for objects of that type.  The "type()" function returns an '
-            "object's\n"
+            'object?s\n'
             'type (which is an object itself).  Like its identity, an '
-            "object's\n"
+            'object?s\n'
             '*type* is also unchangeable. [1]\n'
             '\n'
             'The *value* of some objects can change.  Objects whose value can\n'
@@ -6419,14 +6458,14 @@
             'once they are created are called *immutable*. (The value of an\n'
             'immutable container object that contains a reference to a '
             'mutable\n'
-            "object can change when the latter's value is changed; however "
+            'object can change when the latter?s value is changed; however '
             'the\n'
             'container is still considered immutable, because the collection '
             'of\n'
             'objects it contains cannot be changed.  So, immutability is not\n'
             'strictly the same as having an unchangeable value, it is more '
             'subtle.)\n'
-            "An object's mutability is determined by its type; for instance,\n"
+            'An object?s mutability is determined by its type; for instance,\n'
             'numbers, strings and tuples are immutable, while dictionaries '
             'and\n'
             'lists are mutable.\n'
@@ -6434,9 +6473,9 @@
             'Objects are never explicitly destroyed; however, when they '
             'become\n'
             'unreachable they may be garbage-collected.  An implementation is\n'
-            'allowed to postpone garbage collection or omit it altogether --- '
-            'it is\n'
-            'a matter of implementation quality how garbage collection is\n'
+            'allowed to postpone garbage collection or omit it altogether ? it '
+            'is a\n'
+            'matter of implementation quality how garbage collection is\n'
             'implemented, as long as no objects are collected that are still\n'
             'reachable.\n'
             '\n'
@@ -6456,13 +6495,14 @@
             '(so\n'
             'you should always close files explicitly).\n'
             '\n'
-            "Note that the use of the implementation's tracing or debugging\n"
+            'Note that the use of the implementation?s tracing or debugging\n'
             'facilities may keep objects alive that would normally be '
             'collectable.\n'
-            'Also note that catching an exception with a \'"try"..."except"\'\n'
-            'statement may keep objects alive.\n'
+            'Also note that catching an exception with a ?"try"?"except"? '
+            'statement\n'
+            'may keep objects alive.\n'
             '\n'
-            'Some objects contain references to "external" resources such as '
+            'Some objects contain references to ?external? resources such as '
             'open\n'
             'files or windows.  It is understood that these resources are '
             'freed\n'
@@ -6473,14 +6513,13 @@
             'release the external resource, usually a "close()" method. '
             'Programs\n'
             'are strongly recommended to explicitly close such objects.  The\n'
-            '\'"try"..."finally"\' statement and the \'"with"\' statement '
-            'provide\n'
+            '?"try"?"finally"? statement and the ?"with"? statement provide\n'
             'convenient ways to do this.\n'
             '\n'
             'Some objects contain references to other objects; these are '
             'called\n'
             '*containers*. Examples of containers are tuples, lists and\n'
-            "dictionaries.  The references are part of a container's value.  "
+            'dictionaries.  The references are part of a container?s value.  '
             'In\n'
             'most cases, when we talk about the value of a container, we imply '
             'the\n'
@@ -6509,8 +6548,7 @@
             'created empty lists. (Note that "c = d = []" assigns the same '
             'object\n'
             'to both "c" and "d".)\n',
- 'operator-summary': '\n'
-                     'Operator precedence\n'
+ 'operator-summary': 'Operator precedence\n'
                      '*******************\n'
                      '\n'
                      'The following table summarizes the operator precedence '
@@ -6529,7 +6567,7 @@
                      'tests, all have\n'
                      'the same precedence and have a left-to-right chaining '
                      'feature as\n'
-                     'described in the *Comparisons* section.\n'
+                     'described in the Comparisons section.\n'
                      '\n'
                      '+-------------------------------------------------+---------------------------------------+\n'
                      '| Operator                                        | '
@@ -6538,7 +6576,7 @@
                      '| "lambda"                                        | '
                      'Lambda expression                     |\n'
                      '+-------------------------------------------------+---------------------------------------+\n'
-                     '| "if" -- "else"                                  | '
+                     '| "if" ? "else"                                   | '
                      'Conditional expression                |\n'
                      '+-------------------------------------------------+---------------------------------------+\n'
                      '| "or"                                            | '
@@ -6572,6 +6610,8 @@
                      '+-------------------------------------------------+---------------------------------------+\n'
                      '| "*", "@", "/", "//", "%"                        | '
                      'Multiplication, matrix multiplication |\n'
+                     '|                                                 | '
+                     'division, remainder [5]               |\n'
                      '+-------------------------------------------------+---------------------------------------+\n'
                      '| "+x", "-x", "~x"                                | '
                      'Positive, negative, bitwise NOT       |\n'
@@ -6591,6 +6631,8 @@
                      'Binding or tuple display, list        |\n'
                      '| value...}", "{expressions...}"                  | '
                      'display, dictionary display, set      |\n'
+                     '|                                                 | '
+                     'display                               |\n'
                      '+-------------------------------------------------+---------------------------------------+\n'
                      '\n'
                      '-[ Footnotes ]-\n'
@@ -6615,7 +6657,7 @@
                      'application.\n'
                      '\n'
                      '[2] If x is very close to an exact integer multiple of '
-                     "y, it's\n"
+                     'y, it?s\n'
                      '    possible for "x//y" to be one larger than '
                      '"(x-x%y)//y" due to\n'
                      '    rounding.  In such cases, Python returns the latter '
@@ -6626,8 +6668,8 @@
                      '\n'
                      '[3] The Unicode standard distinguishes between *code '
                      'points* (e.g.\n'
-                     '    U+0041) and *abstract characters* (e.g. "LATIN '
-                     'CAPITAL LETTER A").\n'
+                     '    U+0041) and *abstract characters* (e.g. ?LATIN '
+                     'CAPITAL LETTER A?).\n'
                      '    While most abstract characters in Unicode are only '
                      'represented\n'
                      '    using one code point, there is a number of abstract '
@@ -6635,8 +6677,8 @@
                      '    that can in addition be represented using a sequence '
                      'of more than\n'
                      '    one code point.  For example, the abstract character '
-                     '"LATIN\n'
-                     '    CAPITAL LETTER C WITH CEDILLA" can be represented as '
+                     '?LATIN\n'
+                     '    CAPITAL LETTER C WITH CEDILLA? can be represented as '
                      'a single\n'
                      '    *precomposed character* at code position U+00C7, or '
                      'as a sequence\n'
@@ -6652,9 +6694,9 @@
                      'to humans.  For\n'
                      '    example, ""\\u00C7" == "\\u0043\\u0327"" is "False", '
                      'even though both\n'
-                     '    strings represent the same abstract character "LATIN '
+                     '    strings represent the same abstract character ?LATIN '
                      'CAPITAL\n'
-                     '    LETTER C WITH CEDILLA".\n'
+                     '    LETTER C WITH CEDILLA?.\n'
                      '\n'
                      '    To compare strings at the level of abstract '
                      'characters (that is,\n'
@@ -6679,22 +6721,21 @@
                      'arithmetic\n'
                      '    or bitwise unary operator on its right, that is, '
                      '"2**-1" is "0.5".\n',
- 'pass': '\n'
-         'The "pass" statement\n'
+ 'pass': 'The "pass" statement\n'
          '********************\n'
          '\n'
          '   pass_stmt ::= "pass"\n'
          '\n'
-         '"pass" is a null operation --- when it is executed, nothing '
-         'happens.\n'
-         'It is useful as a placeholder when a statement is required\n'
-         'syntactically, but no code needs to be executed, for example:\n'
+         '"pass" is a null operation ? when it is executed, nothing happens. '
+         'It\n'
+         'is useful as a placeholder when a statement is required '
+         'syntactically,\n'
+         'but no code needs to be executed, for example:\n'
          '\n'
          '   def f(arg): pass    # a function that does nothing (yet)\n'
          '\n'
          '   class C: pass       # a class with no methods (yet)\n',
- 'power': '\n'
-          'The power operator\n'
+ 'power': 'The power operator\n'
           '******************\n'
           '\n'
           'The power operator binds more tightly than unary operators on its\n'
@@ -6728,8 +6769,7 @@
           'Raising a negative number to a fractional power results in a '
           '"complex"\n'
           'number. (In earlier versions it raised a "ValueError".)\n',
- 'raise': '\n'
-          'The "raise" statement\n'
+ 'raise': 'The "raise" statement\n'
           '*********************\n'
           '\n'
           '   raise_stmt ::= "raise" [expression ["from" expression]]\n'
@@ -6746,7 +6786,7 @@
           '"BaseException". If it is a class, the exception instance will be\n'
           'obtained when needed by instantiating the class with no arguments.\n'
           '\n'
-          "The *type* of the exception is the exception instance's class, the\n"
+          'The *type* of the exception is the exception instance?s class, the\n'
           '*value* is the instance itself.\n'
           '\n'
           'A traceback object is normally created automatically when an '
@@ -6792,7 +6832,7 @@
           'inside\n'
           'an exception handler or a "finally" clause: the previous exception '
           'is\n'
-          'then attached as the new exception\'s "__context__" attribute:\n'
+          'then attached as the new exception?s "__context__" attribute:\n'
           '\n'
           '   >>> try:\n'
           '   ...     print(1 / 0)\n'
@@ -6811,11 +6851,10 @@
           '   RuntimeError: Something bad happened\n'
           '\n'
           'Additional information on exceptions can be found in section\n'
-          '*Exceptions*, and information about handling exceptions is in '
+          'Exceptions, and information about handling exceptions is in '
           'section\n'
-          '*The try statement*.\n',
- 'return': '\n'
-           'The "return" statement\n'
+          'The try statement.\n',
+ 'return': 'The "return" statement\n'
            '**********************\n'
            '\n'
            '   return_stmt ::= "return" [expression_list]\n'
@@ -6843,8 +6882,7 @@
            'The\n'
            'returned value (if any) is used as an argument to construct\n'
            '"StopIteration" and becomes the "StopIteration.value" attribute.\n',
- 'sequence-types': '\n'
-                   'Emulating container types\n'
+ 'sequence-types': 'Emulating container types\n'
                    '*************************\n'
                    '\n'
                    'The following methods can be defined to implement '
@@ -6867,7 +6905,7 @@
                    '"clear()",\n'
                    '"setdefault()", "pop()", "popitem()", "copy()", and '
                    '"update()"\n'
-                   "behaving similar to those for Python's standard dictionary "
+                   'behaving similar to those for Python?s standard dictionary '
                    'objects.\n'
                    'The "collections" module provides a "MutableMapping" '
                    'abstract base\n'
@@ -6893,7 +6931,7 @@
                    'the\n'
                    '"__contains__()" method to allow efficient use of the "in" '
                    'operator;\n'
-                   'for mappings, "in" should search the mapping\'s keys; for '
+                   'for mappings, "in" should search the mapping?s keys; for '
                    'sequences, it\n'
                    'should search through the values.  It is further '
                    'recommended that both\n'
@@ -6911,7 +6949,7 @@
                    'Should return\n'
                    '   the length of the object, an integer ">=" 0.  Also, an '
                    'object that\n'
-                   '   doesn\'t define a "__bool__()" method and whose '
+                   '   doesn?t define a "__bool__()" method and whose '
                    '"__len__()" method\n'
                    '   returns zero is considered to be false in a Boolean '
                    'context.\n'
@@ -7026,7 +7064,7 @@
                    'they are\n'
                    '   required to return themselves.  For more information on '
                    'iterator\n'
-                   '   objects, see *Iterator Types*.\n'
+                   '   objects, see Iterator Types.\n'
                    '\n'
                    'object.__reversed__(self)\n'
                    '\n'
@@ -7068,15 +7106,14 @@
                    'values or\n'
                    '   the key-item pairs.\n'
                    '\n'
-                   '   For objects that don\'t define "__contains__()", the '
+                   '   For objects that don?t define "__contains__()", the '
                    'membership test\n'
                    '   first tries iteration via "__iter__()", then the old '
                    'sequence\n'
-                   '   iteration protocol via "__getitem__()", see *this '
+                   '   iteration protocol via "__getitem__()", see this '
                    'section in the\n'
-                   '   language reference*.\n',
- 'shifting': '\n'
-             'Shifting operations\n'
+                   '   language reference.\n',
+ 'shifting': 'Shifting operations\n'
              '*******************\n'
              '\n'
              'The shifting operations have lower priority than the arithmetic\n'
@@ -7100,8 +7137,7 @@
              'operand is\n'
              '  larger than "sys.maxsize" an "OverflowError" exception is '
              'raised.\n',
- 'slicings': '\n'
-             'Slicings\n'
+ 'slicings': 'Slicings\n'
              '********\n'
              '\n'
              'A slicing selects a range of items in a sequence object (e.g., '
@@ -7145,12 +7181,14 @@
              'an\n'
              'expression is that expression.  The conversion of a proper slice '
              'is a\n'
-             'slice object (see section *The standard type hierarchy*) whose\n'
-             '"start", "stop" and "step" attributes are the values of the\n'
-             'expressions given as lower bound, upper bound and stride,\n'
-             'respectively, substituting "None" for missing expressions.\n',
- 'specialattrs': '\n'
-                 'Special Attributes\n'
+             'slice object (see section The standard type hierarchy) whose '
+             '"start",\n'
+             '"stop" and "step" attributes are the values of the expressions '
+             'given\n'
+             'as lower bound, upper bound and stride, respectively, '
+             'substituting\n'
+             '"None" for missing expressions.\n',
+ 'specialattrs': 'Special Attributes\n'
                  '******************\n'
                  '\n'
                  'The implementation adds a few special read-only attributes '
@@ -7162,7 +7200,7 @@
                  'object.__dict__\n'
                  '\n'
                  '   A dictionary or other mapping object used to store an '
-                 "object's\n"
+                 'object?s\n'
                  '   (writable) attributes.\n'
                  '\n'
                  'instance.__class__\n'
@@ -7216,34 +7254,33 @@
                  '\n'
                  '[1] Additional information on these special methods may be '
                  'found\n'
-                 '    in the Python Reference Manual (*Basic customization*).\n'
+                 '    in the Python Reference Manual (Basic customization).\n'
                  '\n'
                  '[2] As a consequence, the list "[1, 2]" is considered equal '
                  'to\n'
                  '    "[1.0, 2.0]", and similarly for tuples.\n'
                  '\n'
-                 "[3] They must have since the parser can't tell the type of "
+                 '[3] They must have since the parser can?t tell the type of '
                  'the\n'
                  '    operands.\n'
                  '\n'
                  '[4] Cased characters are those with general category '
                  'property\n'
-                 '    being one of "Lu" (Letter, uppercase), "Ll" (Letter, '
+                 '    being one of ?Lu? (Letter, uppercase), ?Ll? (Letter, '
                  'lowercase),\n'
-                 '    or "Lt" (Letter, titlecase).\n'
+                 '    or ?Lt? (Letter, titlecase).\n'
                  '\n'
                  '[5] To format only a tuple you should therefore provide a\n'
                  '    singleton tuple whose only element is the tuple to be '
                  'formatted.\n',
- 'specialnames': '\n'
-                 'Special method names\n'
+ 'specialnames': 'Special method names\n'
                  '********************\n'
                  '\n'
                  'A class can implement certain operations that are invoked by '
                  'special\n'
                  'syntax (such as arithmetic operations or subscripting and '
                  'slicing) by\n'
-                 "defining methods with special names. This is Python's "
+                 'defining methods with special names. This is Python?s '
                  'approach to\n'
                  '*operator overloading*, allowing classes to define their own '
                  'behavior\n'
@@ -7268,7 +7305,7 @@
                  'elements, but\n'
                  'extracting a slice may not make sense.  (One example of this '
                  'is the\n'
-                 '"NodeList" interface in the W3C\'s Document Object Model.)\n'
+                 '"NodeList" interface in the W3C?s Document Object Model.)\n'
                  '\n'
                  '\n'
                  'Basic customization\n'
@@ -7292,7 +7329,7 @@
                  '\n'
                  '   Typical implementations create a new instance of the '
                  'class by\n'
-                 '   invoking the superclass\'s "__new__()" method using\n'
+                 '   invoking the superclass?s "__new__()" method using\n'
                  '   "super().__new__(cls[, ...])" with appropriate arguments '
                  'and then\n'
                  '   modifying the newly-created instance as necessary before '
@@ -7301,7 +7338,7 @@
                  '\n'
                  '   If "__new__()" returns an instance of *cls*, then the '
                  'new\n'
-                 '   instance\'s "__init__()" method will be invoked like\n'
+                 '   instance?s "__init__()" method will be invoked like\n'
                  '   "__init__(self[, ...])", where *self* is the new instance '
                  'and the\n'
                  '   remaining arguments are the same as were passed to '
@@ -7309,7 +7346,7 @@
                  '\n'
                  '   If "__new__()" does not return an instance of *cls*, then '
                  'the new\n'
-                 '   instance\'s "__init__()" method will not be invoked.\n'
+                 '   instance?s "__init__()" method will not be invoked.\n'
                  '\n'
                  '   "__new__()" is intended mainly to allow subclasses of '
                  'immutable\n'
@@ -7327,7 +7364,7 @@
                  'those\n'
                  '   passed to the class constructor expression.  If a base '
                  'class has an\n'
-                 '   "__init__()" method, the derived class\'s "__init__()" '
+                 '   "__init__()" method, the derived class?s "__init__()" '
                  'method, if\n'
                  '   any, must explicitly call it to ensure proper '
                  'initialization of the\n'
@@ -7348,7 +7385,7 @@
                  'is also\n'
                  '   called a destructor.  If a base class has a "__del__()" '
                  'method, the\n'
-                 '   derived class\'s "__del__()" method, if any, must '
+                 '   derived class?s "__del__()" method, if any, must '
                  'explicitly call it\n'
                  '   to ensure proper deletion of the base class part of the '
                  'instance.\n'
@@ -7364,11 +7401,11 @@
                  'exist when\n'
                  '   the interpreter exits.\n'
                  '\n'
-                 '   Note: "del x" doesn\'t directly call "x.__del__()" --- '
-                 'the former\n'
+                 '   Note: "del x" doesn?t directly call "x.__del__()" ? the '
+                 'former\n'
                  '     decrements the reference count for "x" by one, and the '
                  'latter is\n'
-                 '     only called when "x"\'s reference count reaches zero.  '
+                 '     only called when "x"?s reference count reaches zero.  '
                  'Some common\n'
                  '     situations that may prevent the reference count of an '
                  'object from\n'
@@ -7397,7 +7434,7 @@
                  'Circular\n'
                  '     references which are garbage are detected and cleaned '
                  'up when the\n'
-                 "     cyclic garbage collector is enabled (it's on by "
+                 '     cyclic garbage collector is enabled (it?s on by '
                  'default). Refer\n'
                  '     to the documentation for the "gc" module for more '
                  'information\n'
@@ -7435,7 +7472,7 @@
                  'object.__repr__(self)\n'
                  '\n'
                  '   Called by the "repr()" built-in function to compute the '
-                 '"official"\n'
+                 '?official?\n'
                  '   string representation of an object.  If at all possible, '
                  'this\n'
                  '   should look like a valid Python expression that could be '
@@ -7449,7 +7486,7 @@
                  '   value must be a string object. If a class defines '
                  '"__repr__()" but\n'
                  '   not "__str__()", then "__repr__()" is also used when an '
-                 '"informal"\n'
+                 '?informal?\n'
                  '   string representation of instances of that class is '
                  'required.\n'
                  '\n'
@@ -7461,10 +7498,10 @@
                  '\n'
                  '   Called by "str(object)" and the built-in functions '
                  '"format()" and\n'
-                 '   "print()" to compute the "informal" or nicely printable '
+                 '   "print()" to compute the ?informal? or nicely printable '
                  'string\n'
                  '   representation of an object.  The return value must be a '
-                 '*string*\n'
+                 'string\n'
                  '   object.\n'
                  '\n'
                  '   This method differs from "object.__repr__()" in that '
@@ -7488,7 +7525,7 @@
                  '   Called by the "format()" built-in function (and by '
                  'extension, the\n'
                  '   "str.format()" method of class "str") to produce a '
-                 '"formatted"\n'
+                 '?formatted?\n'
                  '   string representation of an object. The "format_spec" '
                  'argument is a\n'
                  '   string that contains a description of the formatting '
@@ -7501,8 +7538,8 @@
                  'or use a\n'
                  '   similar formatting option syntax.\n'
                  '\n'
-                 '   See *Format Specification Mini-Language* for a '
-                 'description of the\n'
+                 '   See Format Specification Mini-Language for a description '
+                 'of the\n'
                  '   standard formatting syntax.\n'
                  '\n'
                  '   The return value must be a string object.\n'
@@ -7518,7 +7555,7 @@
                  'object.__gt__(self, other)\n'
                  'object.__ge__(self, other)\n'
                  '\n'
-                 '   These are the so-called "rich comparison" methods. The\n'
+                 '   These are the so-called ?rich comparison? methods. The\n'
                  '   correspondence between operator symbols and method names '
                  'is as\n'
                  '   follows: "x<y" calls "x.__lt__(y)", "x<=y" calls '
@@ -7565,18 +7602,18 @@
                  '   when the left argument does not support the operation but '
                  'the right\n'
                  '   argument does); rather, "__lt__()" and "__gt__()" are '
-                 "each other's\n"
-                 '   reflection, "__le__()" and "__ge__()" are each other\'s '
+                 'each other?s\n'
+                 '   reflection, "__le__()" and "__ge__()" are each other?s '
                  'reflection,\n'
                  '   and "__eq__()" and "__ne__()" are their own reflection. '
                  'If the\n'
-                 "   operands are of different types, and right operand's type "
+                 '   operands are of different types, and right operand?s type '
                  'is a\n'
-                 "   direct or indirect subclass of the left operand's type, "
+                 '   direct or indirect subclass of the left operand?s type, '
                  'the\n'
                  '   reflected method of the right operand has priority, '
                  'otherwise the\n'
-                 "   left operand's method has priority.  Virtual subclassing "
+                 '   left operand?s method has priority.  Virtual subclassing '
                  'is not\n'
                  '   considered.\n'
                  '\n'
@@ -7600,13 +7637,13 @@
                  '          return hash((self.name, self.nick, self.color))\n'
                  '\n'
                  '   Note: "hash()" truncates the value returned from an '
-                 "object's\n"
+                 'object?s\n'
                  '     custom "__hash__()" method to the size of a '
                  '"Py_ssize_t".  This\n'
                  '     is typically 8 bytes on 64-bit builds and 4 bytes on '
                  '32-bit\n'
-                 '     builds. If an object\'s   "__hash__()" must '
-                 'interoperate on builds\n'
+                 '     builds. If an object?s   "__hash__()" must interoperate '
+                 'on builds\n'
                  '     of different bit sizes, be sure to check the width on '
                  'all\n'
                  '     supported builds.  An easy way to do this is with '
@@ -7624,8 +7661,8 @@
                  '   implements an "__eq__()" method, it should not implement\n'
                  '   "__hash__()", since the implementation of hashable '
                  'collections\n'
-                 "   requires that a key's hash value is immutable (if the "
-                 "object's hash\n"
+                 '   requires that a key?s hash value is immutable (if the '
+                 'object?s hash\n'
                  '   value changes, it will be in the wrong hash bucket).\n'
                  '\n'
                  '   User-defined classes have "__eq__()" and "__hash__()" '
@@ -7669,7 +7706,7 @@
                  '\n'
                  '   Note: By default, the "__hash__()" values of str, bytes '
                  'and\n'
-                 '     datetime objects are "salted" with an unpredictable '
+                 '     datetime objects are ?salted? with an unpredictable '
                  'random value.\n'
                  '     Although they remain constant within an individual '
                  'Python\n'
@@ -7772,7 +7809,7 @@
                  'special\n'
                  '     methods as the result of implicit invocation via '
                  'language syntax\n'
-                 '     or built-in functions. See *Special method lookup*.\n'
+                 '     or built-in functions. See Special method lookup.\n'
                  '\n'
                  'object.__setattr__(self, name, value)\n'
                  '\n'
@@ -7814,13 +7851,13 @@
                  'class\n'
                  'containing the method (a so-called *descriptor* class) '
                  'appears in an\n'
-                 "*owner* class (the descriptor must be in either the owner's "
+                 '*owner* class (the descriptor must be in either the owner?s '
                  'class\n'
                  'dictionary or in the class dictionary for one of its '
                  'parents).  In the\n'
-                 'examples below, "the attribute" refers to the attribute '
+                 'examples below, ?the attribute? refers to the attribute '
                  'whose name is\n'
-                 'the key of the property in the owner class\' "__dict__".\n'
+                 'the key of the property in the owner class? "__dict__".\n'
                  '\n'
                  'object.__get__(self, instance, owner)\n'
                  '\n'
@@ -7869,8 +7906,8 @@
                  '--------------------\n'
                  '\n'
                  'In general, a descriptor is an object attribute with '
-                 '"binding\n'
-                 'behavior", one whose attribute access has been overridden by '
+                 '?binding\n'
+                 'behavior?, one whose attribute access has been overridden by '
                  'methods\n'
                  'in the descriptor protocol:  "__get__()", "__set__()", and\n'
                  '"__delete__()". If any of those methods are defined for an '
@@ -7879,7 +7916,7 @@
                  '\n'
                  'The default behavior for attribute access is to get, set, or '
                  'delete\n'
-                 "the attribute from an object's dictionary. For instance, "
+                 'the attribute from an object?s dictionary. For instance, '
                  '"a.x" has a\n'
                  'lookup chain starting with "a.__dict__[\'x\']", then\n'
                  '"type(a).__dict__[\'x\']", and continuing through the base '
@@ -7933,7 +7970,7 @@
                  'does not define "__get__()", then accessing the attribute '
                  'will return\n'
                  'the descriptor object itself unless there is a value in the '
-                 "object's\n"
+                 'object?s\n'
                  'instance dictionary.  If the descriptor defines "__set__()" '
                  'and/or\n'
                  '"__delete__()", it is a data descriptor; if it defines '
@@ -8029,9 +8066,9 @@
                  '\n'
                  '* *__slots__* are implemented at the class level by '
                  'creating\n'
-                 '  descriptors (*Implementing Descriptors*) for each variable '
-                 'name.  As\n'
-                 '  a result, class attributes cannot be used to set default '
+                 '  descriptors (Implementing Descriptors) for each variable '
+                 'name.  As a\n'
+                 '  result, class attributes cannot be used to set default '
                  'values for\n'
                  '  instance variables defined by *__slots__*; otherwise, the '
                  'class\n'
@@ -8057,7 +8094,7 @@
                  '\n'
                  '* Nonempty *__slots__* does not work for classes derived '
                  'from\n'
-                 '  "variable-length" built-in types such as "int", "bytes" '
+                 '  ?variable-length? built-in types such as "int", "bytes" '
                  'and "tuple".\n'
                  '\n'
                  '* Any non-string iterable may be assigned to *__slots__*. '
@@ -8165,8 +8202,9 @@
                  'class\n'
                  'namespace is initialised as an empty "dict()" instance.\n'
                  '\n'
-                 'See also: **PEP 3115** - Metaclasses in Python 3000\n'
+                 'See also:\n'
                  '\n'
+                 '  **PEP 3115** - Metaclasses in Python 3000\n'
                  '     Introduced the "__prepare__" namespace hook\n'
                  '\n'
                  '\n'
@@ -8236,8 +8274,9 @@
                  '"__dict__"\n'
                  'attribute of the class object.\n'
                  '\n'
-                 'See also: **PEP 3135** - New super\n'
+                 'See also:\n'
                  '\n'
+                 '  **PEP 3135** - New super\n'
                  '     Describes the implicit "__class__" closure reference\n'
                  '\n'
                  '\n'
@@ -8280,7 +8319,7 @@
                  '\n'
                  'When the class definition for *A* gets executed, the process '
                  'begins\n'
-                 'with calling the metaclass\'s "__prepare__()" method which '
+                 'with calling the metaclass?s "__prepare__()" method which '
                  'returns an\n'
                  'empty "collections.OrderedDict".  That mapping records the '
                  'methods and\n'
@@ -8288,7 +8327,7 @@
                  'class\n'
                  'statement. Once those definitions are executed, the ordered '
                  'dictionary\n'
-                 'is fully populated and the metaclass\'s "__new__()" method '
+                 'is fully populated and the metaclass?s "__new__()" method '
                  'gets\n'
                  'invoked.  That method builds the new type and it saves the '
                  'ordered\n'
@@ -8306,7 +8345,7 @@
                  'methods in\n'
                  'order to allow the addition of Abstract Base Classes (ABCs) '
                  'as\n'
-                 '"virtual base classes" to any class or type (including '
+                 '?virtual base classes? to any class or type (including '
                  'built-in\n'
                  'types), including other ABCs.\n'
                  '\n'
@@ -8335,8 +8374,9 @@
                  'on instances, only in this case the instance is itself a '
                  'class.\n'
                  '\n'
-                 'See also: **PEP 3119** - Introducing Abstract Base Classes\n'
+                 'See also:\n'
                  '\n'
+                 '  **PEP 3119** - Introducing Abstract Base Classes\n'
                  '     Includes the specification for customizing '
                  '"isinstance()" and\n'
                  '     "issubclass()" behavior through "__instancecheck__()" '
@@ -8353,7 +8393,7 @@
                  '\n'
                  'object.__call__(self[, args...])\n'
                  '\n'
-                 '   Called when the instance is "called" as a function; if '
+                 '   Called when the instance is ?called? as a function; if '
                  'this method\n'
                  '   is defined, "x(arg1, arg2, ...)" is a shorthand for\n'
                  '   "x.__call__(arg1, arg2, ...)".\n'
@@ -8382,7 +8422,7 @@
                  '"clear()",\n'
                  '"setdefault()", "pop()", "popitem()", "copy()", and '
                  '"update()"\n'
-                 "behaving similar to those for Python's standard dictionary "
+                 'behaving similar to those for Python?s standard dictionary '
                  'objects.\n'
                  'The "collections" module provides a "MutableMapping" '
                  'abstract base\n'
@@ -8407,7 +8447,7 @@
                  'recommended that both mappings and sequences implement the\n'
                  '"__contains__()" method to allow efficient use of the "in" '
                  'operator;\n'
-                 'for mappings, "in" should search the mapping\'s keys; for '
+                 'for mappings, "in" should search the mapping?s keys; for '
                  'sequences, it\n'
                  'should search through the values.  It is further recommended '
                  'that both\n'
@@ -8425,7 +8465,7 @@
                  'Should return\n'
                  '   the length of the object, an integer ">=" 0.  Also, an '
                  'object that\n'
-                 '   doesn\'t define a "__bool__()" method and whose '
+                 '   doesn?t define a "__bool__()" method and whose '
                  '"__len__()" method\n'
                  '   returns zero is considered to be false in a Boolean '
                  'context.\n'
@@ -8539,7 +8579,7 @@
                  'are\n'
                  '   required to return themselves.  For more information on '
                  'iterator\n'
-                 '   objects, see *Iterator Types*.\n'
+                 '   objects, see Iterator Types.\n'
                  '\n'
                  'object.__reversed__(self)\n'
                  '\n'
@@ -8581,13 +8621,13 @@
                  'values or\n'
                  '   the key-item pairs.\n'
                  '\n'
-                 '   For objects that don\'t define "__contains__()", the '
+                 '   For objects that don?t define "__contains__()", the '
                  'membership test\n'
                  '   first tries iteration via "__iter__()", then the old '
                  'sequence\n'
-                 '   iteration protocol via "__getitem__()", see *this section '
+                 '   iteration protocol via "__getitem__()", see this section '
                  'in the\n'
-                 '   language reference*.\n'
+                 '   language reference.\n'
                  '\n'
                  '\n'
                  'Emulating numeric types\n'
@@ -8677,15 +8717,15 @@
                  '"__rpow__()" (the\n'
                  '   coercion rules would become too complicated).\n'
                  '\n'
-                 "   Note: If the right operand's type is a subclass of the "
+                 '   Note: If the right operand?s type is a subclass of the '
                  'left\n'
-                 "     operand's type and that subclass provides the reflected "
+                 '     operand?s type and that subclass provides the reflected '
                  'method\n'
                  '     for the operation, this method will be called before '
                  'the left\n'
-                 "     operand's non-reflected method.  This behavior allows "
+                 '     operand?s non-reflected method.  This behavior allows '
                  'subclasses\n'
-                 "     to override their ancestors' operations.\n"
+                 '     to override their ancestors? operations.\n'
                  '\n'
                  'object.__iadd__(self, other)\n'
                  'object.__isub__(self, other)\n'
@@ -8723,9 +8763,9 @@
                  'certain\n'
                  '   situations, augmented assignment can result in unexpected '
                  'errors\n'
-                 "   (see *Why does a_tuple[i] += ['item'] raise an exception "
+                 '   (see Why does a_tuple[i] += [?item?] raise an exception '
                  'when the\n'
-                 '   addition works?*), but this behavior is in fact part of '
+                 '   addition works?), but this behavior is in fact part of '
                  'the data\n'
                  '   model.\n'
                  '\n'
@@ -8779,9 +8819,9 @@
                  'runtime context\n'
                  'for the execution of the block of code.  Context managers '
                  'are normally\n'
-                 'invoked using the "with" statement (described in section '
-                 '*The with\n'
-                 'statement*), but can also be used by directly invoking their '
+                 'invoked using the "with" statement (described in section The '
+                 'with\n'
+                 'statement), but can also be used by directly invoking their '
                  'methods.\n'
                  '\n'
                  'Typical uses of context managers include saving and '
@@ -8790,14 +8830,14 @@
                  'closing opened\n'
                  'files, etc.\n'
                  '\n'
-                 'For more information on context managers, see *Context '
-                 'Manager Types*.\n'
+                 'For more information on context managers, see Context '
+                 'Manager Types.\n'
                  '\n'
                  'object.__enter__(self)\n'
                  '\n'
                  '   Enter the runtime context related to this object. The '
                  '"with"\n'
-                 "   statement will bind this method's return value to the "
+                 '   statement will bind this method?s return value to the '
                  'target(s)\n'
                  '   specified in the "as" clause of the statement, if any.\n'
                  '\n'
@@ -8821,10 +8861,11 @@
                  '\n'
                  '   Note that "__exit__()" methods should not reraise the '
                  'passed-in\n'
-                 "   exception; this is the caller's responsibility.\n"
+                 '   exception; this is the caller?s responsibility.\n'
                  '\n'
-                 'See also: **PEP 343** - The "with" statement\n'
+                 'See also:\n'
                  '\n'
+                 '  **PEP 343** - The ?with? statement\n'
                  '     The specification, background, and examples for the '
                  'Python "with"\n'
                  '     statement.\n'
@@ -8835,9 +8876,9 @@
                  '\n'
                  'For custom classes, implicit invocations of special methods '
                  'are only\n'
-                 "guaranteed to work correctly if defined on an object's type, "
+                 'guaranteed to work correctly if defined on an object?s type, '
                  'not in\n'
-                 "the object's instance dictionary.  That behaviour is the "
+                 'the object?s instance dictionary.  That behaviour is the '
                  'reason why\n'
                  'the following code raises an exception:\n'
                  '\n'
@@ -8871,7 +8912,7 @@
                  '\n'
                  'Incorrectly attempting to invoke an unbound method of a '
                  'class in this\n'
-                 "way is sometimes referred to as 'metaclass confusion', and "
+                 'way is sometimes referred to as ?metaclass confusion?, and '
                  'is avoided\n'
                  'by bypassing the instance when looking up special methods:\n'
                  '\n'
@@ -8884,7 +8925,7 @@
                  'interest of\n'
                  'correctness, implicit special method lookup generally also '
                  'bypasses\n'
-                 'the "__getattribute__()" method even of the object\'s '
+                 'the "__getattribute__()" method even of the object?s '
                  'metaclass:\n'
                  '\n'
                  '   >>> class Meta(type):\n'
@@ -8920,11 +8961,10 @@
                  'special method *must* be set on the class object itself in '
                  'order to be\n'
                  'consistently invoked by the interpreter).\n',
- 'string-methods': '\n'
-                   'String Methods\n'
+ 'string-methods': 'String Methods\n'
                    '**************\n'
                    '\n'
-                   'Strings implement all of the *common* sequence operations, '
+                   'Strings implement all of the common sequence operations, '
                    'along with\n'
                    'the additional methods described below.\n'
                    '\n'
@@ -8932,19 +8972,18 @@
                    'providing a\n'
                    'large degree of flexibility and customization (see '
                    '"str.format()",\n'
-                   '*Format String Syntax* and *Custom String Formatting*) and '
-                   'the other\n'
-                   'based on C "printf" style formatting that handles a '
-                   'narrower range of\n'
-                   'types and is slightly harder to use correctly, but is '
-                   'often faster for\n'
-                   'the cases it can handle (*printf-style String '
-                   'Formatting*).\n'
-                   '\n'
-                   'The *Text Processing Services* section of the standard '
-                   'library covers\n'
-                   'a number of other modules that provide various text '
-                   'related utilities\n'
+                   'Format String Syntax and Custom String Formatting) and the '
+                   'other based\n'
+                   'on C "printf" style formatting that handles a narrower '
+                   'range of types\n'
+                   'and is slightly harder to use correctly, but is often '
+                   'faster for the\n'
+                   'cases it can handle (printf-style String Formatting).\n'
+                   '\n'
+                   'The Text Processing Services section of the standard '
+                   'library covers a\n'
+                   'number of other modules that provide various text related '
+                   'utilities\n'
                    '(including regular expression support in the "re" '
                    'module).\n'
                    '\n'
@@ -9008,10 +9047,9 @@
                    '"\'xmlcharrefreplace\'",\n'
                    '   "\'backslashreplace\'" and any other name registered '
                    'via\n'
-                   '   "codecs.register_error()", see section *Error '
-                   'Handlers*. For a list\n'
-                   '   of possible encodings, see section *Standard '
-                   'Encodings*.\n'
+                   '   "codecs.register_error()", see section Error Handlers. '
+                   'For a list\n'
+                   '   of possible encodings, see section Standard Encodings.\n'
                    '\n'
                    '   Changed in version 3.1: Support for keyword arguments '
                    'added.\n'
@@ -9098,7 +9136,7 @@
                    '   >>> "The sum of 1 + 2 is {0}".format(1+2)\n'
                    "   'The sum of 1 + 2 is 3'\n"
                    '\n'
-                   '   See *Format String Syntax* for a description of the '
+                   '   See Format String Syntax for a description of the '
                    'various\n'
                    '   formatting options that can be specified in format '
                    'strings.\n'
@@ -9146,11 +9184,11 @@
                    'Alphabetic\n'
                    '   characters are those characters defined in the Unicode '
                    'character\n'
-                   '   database as "Letter", i.e., those with general category '
+                   '   database as ?Letter?, i.e., those with general category '
                    'property\n'
-                   '   being one of "Lm", "Lt", "Lu", "Ll", or "Lo".  Note '
+                   '   being one of ?Lm?, ?Lt?, ?Lu?, ?Ll?, or ?Lo?.  Note '
                    'that this is\n'
-                   '   different from the "Alphabetic" property defined in the '
+                   '   different from the ?Alphabetic? property defined in the '
                    'Unicode\n'
                    '   Standard.\n'
                    '\n'
@@ -9164,7 +9202,7 @@
                    'in base 10,\n'
                    '   e.g. U+0660, ARABIC-INDIC DIGIT ZERO.  Formally a '
                    'decimal character\n'
-                   '   is a character in the Unicode General Category "Nd".\n'
+                   '   is a character in the Unicode General Category ?Nd?.\n'
                    '\n'
                    'str.isdigit()\n'
                    '\n'
@@ -9186,8 +9224,7 @@
                    '\n'
                    '   Return true if the string is a valid identifier '
                    'according to the\n'
-                   '   language definition, section *Identifiers and '
-                   'keywords*.\n'
+                   '   language definition, section Identifiers and keywords.\n'
                    '\n'
                    '   Use "keyword.iskeyword()" to test for reserved '
                    'identifiers such as\n'
@@ -9224,7 +9261,7 @@
                    'characters are\n'
                    '   those characters defined in the Unicode character '
                    'database as\n'
-                   '   "Other" or "Separator", excepting the ASCII space '
+                   '   ?Other? or ?Separator?, excepting the ASCII space '
                    '(0x20) which is\n'
                    '   considered printable.  (Note that printable characters '
                    'in this\n'
@@ -9242,9 +9279,9 @@
                    'Whitespace\n'
                    '   characters  are those characters defined in the Unicode '
                    'character\n'
-                   '   database as "Other" or "Separator" and those with '
+                   '   database as ?Other? or ?Separator? and those with '
                    'bidirectional\n'
-                   '   property being one of "WS", "B", or "S".\n'
+                   '   property being one of ?WS?, ?B?, or ?S?.\n'
                    '\n'
                    'str.istitle()\n'
                    '\n'
@@ -9702,9 +9739,9 @@
                    '"str.upper().isupper()" might be\n'
                    '   "False" if "s" contains uncased characters or if the '
                    'Unicode\n'
-                   '   category of the resulting character(s) is not "Lu" '
+                   '   category of the resulting character(s) is not ?Lu? '
                    '(Letter,\n'
-                   '   uppercase), but e.g. "Lt" (Letter, titlecase).\n'
+                   '   uppercase), but e.g. ?Lt? (Letter, titlecase).\n'
                    '\n'
                    '   The uppercasing algorithm used is described in section '
                    '3.13 of the\n'
@@ -9727,8 +9764,7 @@
                    "      '00042'\n"
                    '      >>> "-42".zfill(5)\n'
                    "      '-0042'\n",
- 'strings': '\n'
-            'String and Bytes literals\n'
+ 'strings': 'String and Bytes literals\n'
             '*************************\n'
             '\n'
             'String literals are described by the following lexical '
@@ -9769,7 +9805,7 @@
             'by\n'
             'the encoding declaration; it is UTF-8 if no encoding declaration '
             'is\n'
-            'given in the source file; see section *Encoding declarations*.\n'
+            'given in the source file; see section Encoding declarations.\n'
             '\n'
             'In plain English: Both types of literals can be enclosed in '
             'matching\n'
@@ -9803,9 +9839,9 @@
             'literals,\n'
             '"\'\\U\'" and "\'\\u\'" escapes in raw strings are not treated '
             'specially.\n'
-            "Given that Python 2.x's raw unicode literals behave differently "
+            'Given that Python 2.x?s raw unicode literals behave differently '
             'than\n'
-            'Python 3.x\'s the "\'ur\'" syntax is not supported.\n'
+            'Python 3.x?s the "\'ur\'" syntax is not supported.\n'
             '\n'
             'New in version 3.3: The "\'rb\'" prefix of raw bytes literals has '
             'been\n'
@@ -9820,7 +9856,7 @@
             'In triple-quoted literals, unescaped newlines and quotes are '
             'allowed\n'
             '(and are retained), except that three unescaped quotes in a row\n'
-            'terminate the literal.  (A "quote" is the character used to open '
+            'terminate the literal.  (A ?quote? is the character used to open '
             'the\n'
             'literal, i.e. either "\'" or """.)\n'
             '\n'
@@ -9834,16 +9870,38 @@
             '| Escape Sequence   | Meaning                           | Notes   '
             '|\n'
             '+===================+===================================+=========+\n'
+            '| "\\newline"        | Backslash and newline ignored     '
+            '|         |\n'
             '+-------------------+-----------------------------------+---------+\n'
+            '| "\\\\"              | Backslash ("\\")                   '
+            '|         |\n'
             '+-------------------+-----------------------------------+---------+\n'
+            '| "\\\'"              | Single quote ("\'")                '
+            '|         |\n'
             '+-------------------+-----------------------------------+---------+\n'
+            '| "\\""              | Double quote (""")                '
+            '|         |\n'
             '+-------------------+-----------------------------------+---------+\n'
+            '| "\\a"              | ASCII Bell (BEL)                  '
+            '|         |\n'
             '+-------------------+-----------------------------------+---------+\n'
+            '| "\\b"              | ASCII Backspace (BS)              '
+            '|         |\n'
             '+-------------------+-----------------------------------+---------+\n'
+            '| "\\f"              | ASCII Formfeed (FF)               '
+            '|         |\n'
             '+-------------------+-----------------------------------+---------+\n'
+            '| "\\n"              | ASCII Linefeed (LF)               '
+            '|         |\n'
             '+-------------------+-----------------------------------+---------+\n'
+            '| "\\r"              | ASCII Carriage Return (CR)        '
+            '|         |\n'
             '+-------------------+-----------------------------------+---------+\n'
+            '| "\\t"              | ASCII Horizontal Tab (TAB)        '
+            '|         |\n'
             '+-------------------+-----------------------------------+---------+\n'
+            '| "\\v"              | ASCII Vertical Tab (VT)           '
+            '|         |\n'
             '+-------------------+-----------------------------------+---------+\n'
             '| "\\ooo"            | Character with octal value *ooo*  | '
             '(1,3)   |\n'
@@ -9860,12 +9918,18 @@
             '+===================+===================================+=========+\n'
             '| "\\N{name}"        | Character named *name* in the     | '
             '(4)     |\n'
+            '|                   | Unicode database                  |         '
+            '|\n'
             '+-------------------+-----------------------------------+---------+\n'
             '| "\\uxxxx"          | Character with 16-bit hex value   | '
             '(5)     |\n'
+            '|                   | *xxxx*                            |         '
+            '|\n'
             '+-------------------+-----------------------------------+---------+\n'
             '| "\\Uxxxxxxxx"      | Character with 32-bit hex value   | '
             '(6)     |\n'
+            '|                   | *xxxxxxxx*                        |         '
+            '|\n'
             '+-------------------+-----------------------------------+---------+\n'
             '\n'
             'Notes:\n'
@@ -9917,8 +9981,7 @@
             'followed by a newline is interpreted as those two characters as '
             'part\n'
             'of the literal, *not* as a line continuation.\n',
- 'subscriptions': '\n'
-                  'Subscriptions\n'
+ 'subscriptions': 'Subscriptions\n'
                   '*************\n'
                   '\n'
                   'A subscription selects an item of a sequence (string, tuple '
@@ -9966,17 +10029,16 @@
                   'item whose\n'
                   'index is that value (counting from zero). Since the support '
                   'for\n'
-                  "negative indices and slicing occurs in the object's "
+                  'negative indices and slicing occurs in the object?s '
                   '"__getitem__()"\n'
                   'method, subclasses overriding this method will need to '
                   'explicitly add\n'
                   'that support.\n'
                   '\n'
-                  "A string's items are characters.  A character is not a "
+                  'A string?s items are characters.  A character is not a '
                   'separate data\n'
                   'type but a string of exactly one character.\n',
- 'truth': '\n'
-          'Truth Value Testing\n'
+ 'truth': 'Truth Value Testing\n'
           '*******************\n'
           '\n'
           'Any object can be tested for truth value, for use in an "if" or\n'
@@ -9998,7 +10060,7 @@
           '  "__bool__()" or "__len__()" method, when that method returns the\n'
           '  integer zero or "bool" value "False". [1]\n'
           '\n'
-          'All other values are considered true --- so objects of many types '
+          'All other values are considered true ? so objects of many types '
           'are\n'
           'always true.\n'
           '\n'
@@ -10008,8 +10070,7 @@
           'otherwise stated. (Important exception: the Boolean operations '
           '"or"\n'
           'and "and" always return one of their operands.)\n',
- 'try': '\n'
-        'The "try" statement\n'
+ 'try': 'The "try" statement\n'
         '*******************\n'
         '\n'
         'The "try" statement specifies exception handlers and/or cleanup code\n'
@@ -10036,7 +10097,7 @@
         'less except clause, if present, must be last; it matches any\n'
         'exception.  For an except clause with an expression, that expression\n'
         'is evaluated, and the clause matches the exception if the resulting\n'
-        'object is "compatible" with the exception.  An object is compatible\n'
+        'object is ?compatible? with the exception.  An object is compatible\n'
         'with an exception if it is the class or a base class of the '
         'exception\n'
         'object or a tuple containing an item compatible with the exception.\n'
@@ -10058,7 +10119,7 @@
         'When a matching except clause is found, the exception is assigned to\n'
         'the target specified after the "as" keyword in that except clause, '
         'if\n'
-        "present, and the except clause's suite is executed.  All except\n"
+        'present, and the except clause?s suite is executed.  All except\n'
         'clauses must have an executable block.  When the end of this block '
         'is\n'
         'reached, execution continues normally after the entire try '
@@ -10088,12 +10149,12 @@
         'cycle with the stack frame, keeping all locals in that frame alive\n'
         'until the next garbage collection occurs.\n'
         '\n'
-        "Before an except clause's suite is executed, details about the\n"
+        'Before an except clause?s suite is executed, details about the\n'
         'exception are stored in the "sys" module and can be accessed via\n'
         '"sys.exc_info()". "sys.exc_info()" returns a 3-tuple consisting of '
         'the\n'
         'exception class, the exception instance and a traceback object (see\n'
-        'section *The standard type hierarchy*) identifying the point in the\n'
+        'section The standard type hierarchy) identifying the point in the\n'
         'program where the exception occurred.  "sys.exc_info()" values are\n'
         'restored to their previous values (before the call) when returning\n'
         'from a function that handled an exception.\n'
@@ -10102,7 +10163,7 @@
         'the end of the "try" clause. [2] Exceptions in the "else" clause are\n'
         'not handled by the preceding "except" clauses.\n'
         '\n'
-        'If "finally" is present, it specifies a \'cleanup\' handler.  The '
+        'If "finally" is present, it specifies a ?cleanup? handler.  The '
         '"try"\n'
         'clause is executed, including any "except" and "else" clauses.  If '
         'an\n'
@@ -10130,12 +10191,10 @@
         'execution of the "finally" clause.\n'
         '\n'
         'When a "return", "break" or "continue" statement is executed in the\n'
-        '"try" suite of a "try"..."finally" statement, the "finally" clause '
-        'is\n'
-        'also executed \'on the way out.\' A "continue" statement is illegal '
-        'in\n'
+        '"try" suite of a "try"?"finally" statement, the "finally" clause is\n'
+        'also executed ?on the way out.? A "continue" statement is illegal in\n'
         'the "finally" clause. (The reason is a problem with the current\n'
-        'implementation --- this restriction may be lifted in the future).\n'
+        'implementation ? this restriction may be lifted in the future).\n'
         '\n'
         'The return value of a function is determined by the last "return"\n'
         'statement executed.  Since the "finally" clause always executes, a\n'
@@ -10153,10 +10212,10 @@
         "   'finally'\n"
         '\n'
         'Additional information on exceptions can be found in section\n'
-        '*Exceptions*, and information on using the "raise" statement to\n'
-        'generate exceptions may be found in section *The raise statement*.\n',
- 'types': '\n'
-          'The standard type hierarchy\n'
+        'Exceptions, and information on using the "raise" statement to '
+        'generate\n'
+        'exceptions may be found in section The raise statement.\n',
+ 'types': 'The standard type hierarchy\n'
           '***************************\n'
           '\n'
           'Below is a list of the types that are built into Python.  '
@@ -10170,7 +10229,7 @@
           'will often be provided via the standard library instead.\n'
           '\n'
           'Some of the type descriptions below contain a paragraph listing\n'
-          "'special attributes.'  These are attributes that provide access to "
+          '?special attributes.?  These are attributes that provide access to '
           'the\n'
           'implementation and are not intended for general use.  Their '
           'definition\n'
@@ -10183,7 +10242,7 @@
           'It\n'
           '   is used to signify the absence of a value in many situations, '
           'e.g.,\n'
-          "   it is returned from functions that don't explicitly return\n"
+          '   it is returned from functions that don?t explicitly return\n'
           '   anything. Its truth value is false.\n'
           '\n'
           'NotImplemented\n'
@@ -10197,7 +10256,7 @@
           '   reflected operation, or some other fallback, depending on the\n'
           '   operator.)  Its truth value is true.\n'
           '\n'
-          '   See *Implementing the arithmetic operations* for more details.\n'
+          '   See Implementing the arithmetic operations for more details.\n'
           '\n'
           'Ellipsis\n'
           '   This type has a single value.  There is a single object with '
@@ -10234,7 +10293,7 @@
           'shift\n'
           '         and mask operations, a binary representation is assumed, '
           'and\n'
-          "         negative numbers are represented in a variant of 2's\n"
+          '         negative numbers are represented in a variant of 2?s\n'
           '         complement which gives the illusion of an infinite string '
           'of\n'
           '         sign bits extending to the left.\n'
@@ -10288,7 +10347,7 @@
           'items\n'
           '   of a sequence. When the length of a sequence is *n*, the index '
           'set\n'
-          '   contains the numbers 0, 1, ..., *n*-1.  Item *i* of sequence *a* '
+          '   contains the numbers 0, 1, ?, *n*-1.  Item *i* of sequence *a* '
           'is\n'
           '   selected by "a[i]".\n'
           '\n'
@@ -10298,8 +10357,8 @@
           'implies\n'
           '   that the index set is renumbered so that it starts at 0.\n'
           '\n'
-          '   Some sequences also support "extended slicing" with a third '
-          '"step"\n'
+          '   Some sequences also support ?extended slicing? with a third '
+          '?step?\n'
           '   parameter: "a[i:j:k]" selects all items of *a* with index *x* '
           'where\n'
           '   "x = i + n*k", *n* ">=" "0" and *i* "<=" *x* "<" *j*.\n'
@@ -10324,7 +10383,7 @@
           'code\n'
           '         points. All the code points in the range "U+0000 - '
           'U+10FFFF"\n'
-          "         can be represented in a string.  Python doesn't have a "
+          '         can be represented in a string.  Python doesn?t have a '
           '"char"\n'
           '         type; instead, every code point in the string is '
           'represented\n'
@@ -10343,7 +10402,7 @@
           '         The items of a tuple are arbitrary Python objects. Tuples '
           'of\n'
           '         two or more items are formed by comma-separated lists of\n'
-          "         expressions.  A tuple of one item (a 'singleton') can be\n"
+          '         expressions.  A tuple of one item (a ?singleton?) can be\n'
           '         formed by affixing a comma to an expression (an expression '
           'by\n'
           '         itself does not create a tuple, since parentheses must be\n'
@@ -10449,7 +10508,7 @@
           'object\n'
           '      identity, the reason being that the efficient implementation '
           'of\n'
-          "      dictionaries requires a key's hash value to remain constant.\n"
+          '      dictionaries requires a key?s hash value to remain constant.\n'
           '      Numeric types used for keys obey the normal rules for '
           'numeric\n'
           '      comparison: if two numbers compare equal (e.g., "1" and '
@@ -10458,7 +10517,7 @@
           '      dictionary entry.\n'
           '\n'
           '      Dictionaries are mutable; they can be created by the "{...}"\n'
-          '      notation (see section *Dictionary displays*).\n'
+          '      notation (see section Dictionary displays).\n'
           '\n'
           '      The extension modules "dbm.ndbm" and "dbm.gnu" provide\n'
           '      additional examples of mapping types, as does the '
@@ -10467,68 +10526,115 @@
           '\n'
           'Callable types\n'
           '   These are the types to which the function call operation (see\n'
-          '   section *Calls*) can be applied:\n'
+          '   section Calls) can be applied:\n'
           '\n'
           '   User-defined functions\n'
           '      A user-defined function object is created by a function\n'
-          '      definition (see section *Function definitions*).  It should '
-          'be\n'
+          '      definition (see section Function definitions).  It should be\n'
           '      called with an argument list containing the same number of '
           'items\n'
-          "      as the function's formal parameter list.\n"
+          '      as the function?s formal parameter list.\n'
           '\n'
           '      Special attributes:\n'
           '\n'
           '      '
           '+---------------------------+---------------------------------+-------------+\n'
+          '      | Attribute                 | Meaning                         '
+          '|             |\n'
           '      '
           '+===========================+=================================+=============+\n'
-          '      | "__doc__"                 | The function\'s '
-          'documentation    | Writable    |\n'
+          '      | "__doc__"                 | The function?s documentation    '
+          '| Writable    |\n'
+          '      |                           | string, or "None" if            '
+          '|             |\n'
+          '      |                           | unavailable; not inherited by   '
+          '|             |\n'
+          '      |                           | subclasses                      '
+          '|             |\n'
           '      '
           '+---------------------------+---------------------------------+-------------+\n'
-          '      | "__name__"                | The function\'s '
-          'name             | Writable    |\n'
+          '      | "__name__"                | The function?s name             '
+          '| Writable    |\n'
           '      '
           '+---------------------------+---------------------------------+-------------+\n'
-          '      | "__qualname__"            | The function\'s *qualified '
-          'name* | Writable    |\n'
+          '      | "__qualname__"            | The function?s *qualified name* '
+          '| Writable    |\n'
+          '      |                           | New in version 3.3.             '
+          '|             |\n'
           '      '
           '+---------------------------+---------------------------------+-------------+\n'
           '      | "__module__"              | The name of the module the      '
           '| Writable    |\n'
+          '      |                           | function was defined in, or     '
+          '|             |\n'
+          '      |                           | "None" if unavailable.          '
+          '|             |\n'
           '      '
           '+---------------------------+---------------------------------+-------------+\n'
           '      | "__defaults__"            | A tuple containing default      '
           '| Writable    |\n'
+          '      |                           | argument values for those       '
+          '|             |\n'
+          '      |                           | arguments that have defaults,   '
+          '|             |\n'
+          '      |                           | or "None" if no arguments have  '
+          '|             |\n'
+          '      |                           | a default value                 '
+          '|             |\n'
           '      '
           '+---------------------------+---------------------------------+-------------+\n'
           '      | "__code__"                | The code object representing    '
           '| Writable    |\n'
+          '      |                           | the compiled function body.     '
+          '|             |\n'
           '      '
           '+---------------------------+---------------------------------+-------------+\n'
           '      | "__globals__"             | A reference to the dictionary   '
           '| Read-only   |\n'
+          '      |                           | that holds the function?s       '
+          '|             |\n'
+          '      |                           | global variables ? the global   '
+          '|             |\n'
+          '      |                           | namespace of the module in      '
+          '|             |\n'
+          '      |                           | which the function was defined. '
+          '|             |\n'
           '      '
           '+---------------------------+---------------------------------+-------------+\n'
           '      | "__dict__"                | The namespace supporting        '
           '| Writable    |\n'
+          '      |                           | arbitrary function attributes.  '
+          '|             |\n'
           '      '
           '+---------------------------+---------------------------------+-------------+\n'
           '      | "__closure__"             | "None" or a tuple of cells that '
           '| Read-only   |\n'
+          '      |                           | contain bindings for the        '
+          '|             |\n'
+          '      |                           | function?s free variables.      '
+          '|             |\n'
           '      '
           '+---------------------------+---------------------------------+-------------+\n'
           '      | "__annotations__"         | A dict containing annotations   '
           '| Writable    |\n'
+          '      |                           | of parameters.  The keys of the '
+          '|             |\n'
+          '      |                           | dict are the parameter names,   '
+          '|             |\n'
+          '      |                           | and "\'return\'" for the '
+          'return   |             |\n'
+          '      |                           | annotation, if provided.        '
+          '|             |\n'
           '      '
           '+---------------------------+---------------------------------+-------------+\n'
           '      | "__kwdefaults__"          | A dict containing defaults for  '
           '| Writable    |\n'
+          '      |                           | keyword-only parameters.        '
+          '|             |\n'
           '      '
           '+---------------------------+---------------------------------+-------------+\n'
           '\n'
-          '      Most of the attributes labelled "Writable" check the type of '
+          '      Most of the attributes labelled ?Writable? check the type of '
           'the\n'
           '      assigned value.\n'
           '\n'
@@ -10544,7 +10650,7 @@
           '      attributes on built-in functions may be supported in the\n'
           '      future.*\n'
           '\n'
-          "      Additional information about a function's definition can be\n"
+          '      Additional information about a function?s definition can be\n'
           '      retrieved from its code object; see the description of '
           'internal\n'
           '      types below.\n'
@@ -10557,7 +10663,7 @@
           '      Special read-only attributes: "__self__" is the class '
           'instance\n'
           '      object, "__func__" is the function object; "__doc__" is the\n'
-          '      method\'s documentation (same as "__func__.__doc__"); '
+          '      method?s documentation (same as "__func__.__doc__"); '
           '"__name__"\n'
           '      is the method name (same as "__func__.__name__"); '
           '"__module__"\n'
@@ -10581,7 +10687,7 @@
           'instances,\n'
           '      its "__self__" attribute is the instance, and the method '
           'object\n'
-          '      is said to be bound.  The new method\'s "__func__" attribute '
+          '      is said to be bound.  The new method?s "__func__" attribute '
           'is\n'
           '      the original function object.\n'
           '\n'
@@ -10614,7 +10720,7 @@
           '\n'
           '      When an instance method object is derived from a class '
           'method\n'
-          '      object, the "class instance" stored in "__self__" will '
+          '      object, the ?class instance? stored in "__self__" will '
           'actually\n'
           '      be the class itself, so that calling either "x.f(1)" or '
           '"C.f(1)"\n'
@@ -10641,13 +10747,13 @@
           '\n'
           '   Generator functions\n'
           '      A function or method which uses the "yield" statement (see\n'
-          '      section *The yield statement*) is called a *generator '
+          '      section The yield statement) is called a *generator '
           'function*.\n'
           '      Such a function, when called, always returns an iterator '
           'object\n'
           '      which can be used to execute the body of the function:  '
           'calling\n'
-          '      the iterator\'s "iterator.__next__()" method will cause the\n'
+          '      the iterator?s "iterator.__next__()" method will cause the\n'
           '      function to execute until it provides a value using the '
           '"yield"\n'
           '      statement.  When the function executes a "return" statement '
@@ -10665,7 +10771,7 @@
           '      returns a *coroutine* object.  It may contain "await"\n'
           '      expressions, as well as "async with" and "async for" '
           'statements.\n'
-          '      See also the *Coroutine Objects* section.\n'
+          '      See also the Coroutine Objects section.\n'
           '\n'
           '   Built-in functions\n'
           '      A built-in function object is a wrapper around a C function.\n'
@@ -10674,9 +10780,9 @@
           'of\n'
           '      the arguments are determined by the C function. Special '
           'read-\n'
-          '      only attributes: "__doc__" is the function\'s documentation\n'
+          '      only attributes: "__doc__" is the function?s documentation\n'
           '      string, or "None" if unavailable; "__name__" is the '
-          "function's\n"
+          'function?s\n'
           '      name; "__self__" is set to "None" (but see the next item);\n'
           '      "__module__" is the name of the module the function was '
           'defined\n'
@@ -10708,8 +10814,7 @@
           '\n'
           'Modules\n'
           '   Modules are a basic organizational unit of Python code, and are\n'
-          '   created by the *import system* as invoked either by the '
-          '"import"\n'
+          '   created by the import system as invoked either by the "import"\n'
           '   statement (see "import"), or by calling functions such as\n'
           '   "importlib.import_module()" and built-in "__import__()".  A '
           'module\n'
@@ -10720,14 +10825,14 @@
           '   translated to lookups in this dictionary, e.g., "m.x" is '
           'equivalent\n'
           '   to "m.__dict__["x"]". A module object does not contain the code\n'
-          "   object used to initialize the module (since it isn't needed "
+          '   object used to initialize the module (since it isn?t needed '
           'once\n'
           '   the initialization is done).\n'
           '\n'
-          "   Attribute assignment updates the module's namespace dictionary,\n"
+          '   Attribute assignment updates the module?s namespace dictionary,\n'
           '   e.g., "m.x = 1" is equivalent to "m.__dict__["x"] = 1".\n'
           '\n'
-          '   Special read-only attribute: "__dict__" is the module\'s '
+          '   Special read-only attribute: "__dict__" is the module?s '
           'namespace\n'
           '   as a dictionary object.\n'
           '\n'
@@ -10740,9 +10845,9 @@
           'the\n'
           '   module around while using its dictionary directly.\n'
           '\n'
-          '   Predefined (writable) attributes: "__name__" is the module\'s '
+          '   Predefined (writable) attributes: "__name__" is the module?s '
           'name;\n'
-          '   "__doc__" is the module\'s documentation string, or "None" if\n'
+          '   "__doc__" is the module?s documentation string, or "None" if\n'
           '   unavailable; "__file__" is the pathname of the file from which '
           'the\n'
           '   module was loaded, if it was loaded from a file. The "__file__"\n'
@@ -10756,11 +10861,11 @@
           'Custom classes\n'
           '   Custom class types are typically created by class definitions '
           '(see\n'
-          '   section *Class definitions*).  A class has a namespace '
-          'implemented\n'
-          '   by a dictionary object. Class attribute references are '
-          'translated\n'
-          '   to lookups in this dictionary, e.g., "C.x" is translated to\n'
+          '   section Class definitions).  A class has a namespace implemented '
+          'by\n'
+          '   a dictionary object. Class attribute references are translated '
+          'to\n'
+          '   lookups in this dictionary, e.g., "C.x" is translated to\n'
           '   "C.__dict__["x"]" (although there are a number of hooks which '
           'allow\n'
           '   for other means of locating attributes). When the attribute name '
@@ -10769,7 +10874,7 @@
           '   classes. This search of the base classes uses the C3 method\n'
           '   resolution order which behaves correctly even in the presence '
           'of\n'
-          "   'diamond' inheritance structures where there are multiple\n"
+          '   ?diamond? inheritance structures where there are multiple\n'
           '   inheritance paths leading back to a common ancestor. Additional\n'
           '   details on the C3 MRO used by Python can be found in the\n'
           '   documentation accompanying the 2.3 release at\n'
@@ -10782,12 +10887,13 @@
           'a\n'
           '   static method object, it is transformed into the object wrapped '
           'by\n'
-          '   the static method object. See section *Implementing '
-          'Descriptors*\n'
-          '   for another way in which attributes retrieved from a class may\n'
-          '   differ from those actually contained in its "__dict__".\n'
+          '   the static method object. See section Implementing Descriptors '
+          'for\n'
+          '   another way in which attributes retrieved from a class may '
+          'differ\n'
+          '   from those actually contained in its "__dict__".\n'
           '\n'
-          "   Class attribute assignments update the class's dictionary, "
+          '   Class attribute assignments update the class?s dictionary, '
           'never\n'
           '   the dictionary of a base class.\n'
           '\n'
@@ -10799,11 +10905,11 @@
           'is\n'
           '   the module name in which the class was defined; "__dict__" is '
           'the\n'
-          '   dictionary containing the class\'s namespace; "__bases__" is a '
+          '   dictionary containing the class?s namespace; "__bases__" is a '
           'tuple\n'
           '   containing the base classes, in the order of their occurrence '
           'in\n'
-          '   the base class list; "__doc__" is the class\'s documentation '
+          '   the base class list; "__doc__" is the class?s documentation '
           'string,\n'
           '   or "None" if undefined.\n'
           '\n'
@@ -10813,7 +10919,7 @@
           '   A class instance has a namespace implemented as a dictionary '
           'which\n'
           '   is the first place in which attribute references are searched.\n'
-          "   When an attribute is not found there, and the instance's class "
+          '   When an attribute is not found there, and the instance?s class '
           'has\n'
           '   an attribute by that name, the search continues with the class\n'
           '   attributes.  If a class attribute is found that is a '
@@ -10822,16 +10928,17 @@
           'object\n'
           '   whose "__self__" attribute is the instance.  Static method and\n'
           '   class method objects are also transformed; see above under\n'
-          '   "Classes".  See section *Implementing Descriptors* for another '
-          'way\n'
-          '   in which attributes of a class retrieved via its instances may\n'
-          "   differ from the objects actually stored in the class's "
-          '"__dict__".\n'
-          "   If no class attribute is found, and the object's class has a\n"
+          '   ?Classes?.  See section Implementing Descriptors for another way '
+          'in\n'
+          '   which attributes of a class retrieved via its instances may '
+          'differ\n'
+          '   from the objects actually stored in the class?s "__dict__".  If '
+          'no\n'
+          '   class attribute is found, and the object?s class has a\n'
           '   "__getattr__()" method, that is called to satisfy the lookup.\n'
           '\n'
-          "   Attribute assignments and deletions update the instance's\n"
-          "   dictionary, never a class's dictionary.  If the class has a\n"
+          '   Attribute assignments and deletions update the instance?s\n'
+          '   dictionary, never a class?s dictionary.  If the class has a\n'
           '   "__setattr__()" or "__delattr__()" method, this is called '
           'instead\n'
           '   of updating the instance dictionary directly.\n'
@@ -10839,10 +10946,10 @@
           '   Class instances can pretend to be numbers, sequences, or '
           'mappings\n'
           '   if they have methods with certain special names.  See section\n'
-          '   *Special method names*.\n'
+          '   Special method names.\n'
           '\n'
           '   Special attributes: "__dict__" is the attribute dictionary;\n'
-          '   "__class__" is the instance\'s class.\n'
+          '   "__class__" is the instance?s class.\n'
           '\n'
           'I/O objects (also known as file objects)\n'
           '   A *file object* represents an open file.  Various shortcuts are\n'
@@ -10854,7 +10961,7 @@
           '   provided by extension modules).\n'
           '\n'
           '   The objects "sys.stdin", "sys.stdout" and "sys.stderr" are\n'
-          "   initialized to file objects corresponding to the interpreter's\n"
+          '   initialized to file objects corresponding to the interpreter?s\n'
           '   standard input, output and error streams; they are all open in '
           'text\n'
           '   mode and therefore follow the interface defined by the\n'
@@ -10872,7 +10979,7 @@
           '      or *bytecode*. The difference between a code object and a\n'
           '      function object is that the function object contains an '
           'explicit\n'
-          "      reference to the function's globals (the module in which it "
+          '      reference to the function?s globals (the module in which it '
           'was\n'
           '      defined), while a code object contains no context; also the\n'
           '      default argument values are stored in the function object, '
@@ -10964,7 +11071,7 @@
           'is\n'
           '      used by the debugger); "f_lineno" is the current line number '
           'of\n'
-          '      the frame --- writing to this from within a trace function '
+          '      the frame ? writing to this from within a trace function '
           'jumps\n'
           '      to the given line (only for the bottom-most frame).  A '
           'debugger\n'
@@ -11003,7 +11110,7 @@
           'entered,\n'
           '      the stack trace is made available to the program. (See '
           'section\n'
-          '      *The try statement*.) It is accessible as the third item of '
+          '      The try statement.) It is accessible as the third item of '
           'the\n'
           '      tuple returned by "sys.exc_info()". When the program contains '
           'no\n'
@@ -11081,11 +11188,10 @@
           '      is retrieved from classes and class instances. The behaviour '
           'of\n'
           '      class method objects upon such retrieval is described above,\n'
-          '      under "User-defined methods". Class method objects are '
+          '      under ?User-defined methods?. Class method objects are '
           'created\n'
           '      by the built-in "classmethod()" constructor.\n',
- 'typesfunctions': '\n'
-                   'Functions\n'
+ 'typesfunctions': 'Functions\n'
                    '*********\n'
                    '\n'
                    'Function objects are created by function definitions.  The '
@@ -11101,10 +11207,9 @@
                    'the\n'
                    'different object types.\n'
                    '\n'
-                   'See *Function definitions* for more information.\n',
- 'typesmapping': '\n'
-                 'Mapping Types --- "dict"\n'
-                 '************************\n'
+                   'See Function definitions for more information.\n',
+ 'typesmapping': 'Mapping Types ? "dict"\n'
+                 '**********************\n'
                  '\n'
                  'A *mapping* object maps *hashable* values to arbitrary '
                  'objects.\n'
@@ -11115,7 +11220,7 @@
                  'in "list", "set", and "tuple" classes, and the "collections" '
                  'module.)\n'
                  '\n'
-                 "A dictionary's keys are *almost* arbitrary values.  Values "
+                 'A dictionary?s keys are *almost* arbitrary values.  Values '
                  'that are\n'
                  'not *hashable*, that is, values containing lists, '
                  'dictionaries or\n'
@@ -11141,9 +11246,9 @@
                  '"dict"\n'
                  'constructor.\n'
                  '\n'
-                 'class class dict(**kwarg)\n'
-                 'class class dict(mapping, **kwarg)\n'
-                 'class class dict(iterable, **kwarg)\n'
+                 'class dict(**kwarg)\n'
+                 'class dict(mapping, **kwarg)\n'
+                 'class dict(iterable, **kwarg)\n'
                  '\n'
                  '   Return a new dictionary initialized from an optional '
                  'positional\n'
@@ -11291,14 +11396,14 @@
                  '\n'
                  '   items()\n'
                  '\n'
-                 '      Return a new view of the dictionary\'s items ("(key, '
+                 '      Return a new view of the dictionary?s items ("(key, '
                  'value)"\n'
-                 '      pairs). See the *documentation of view objects*.\n'
+                 '      pairs). See the documentation of view objects.\n'
                  '\n'
                  '   keys()\n'
                  '\n'
-                 "      Return a new view of the dictionary's keys.  See the\n"
-                 '      *documentation of view objects*.\n'
+                 '      Return a new view of the dictionary?s keys.  See the\n'
+                 '      documentation of view objects.\n'
                  '\n'
                  '   pop(key[, default])\n'
                  '\n'
@@ -11345,14 +11450,14 @@
                  '\n'
                  '   values()\n'
                  '\n'
-                 "      Return a new view of the dictionary's values.  See "
+                 '      Return a new view of the dictionary?s values.  See '
                  'the\n'
-                 '      *documentation of view objects*.\n'
+                 '      documentation of view objects.\n'
                  '\n'
                  '   Dictionaries compare equal if and only if they have the '
                  'same "(key,\n'
-                 '   value)" pairs. Order comparisons (\'<\', \'<=\', \'>=\', '
-                 "'>') raise\n"
+                 '   value)" pairs. Order comparisons (?<?, ?<=?, ?>=?, ?>?) '
+                 'raise\n'
                  '   "TypeError".\n'
                  '\n'
                  'See also: "types.MappingProxyType" can be used to create a '
@@ -11366,7 +11471,7 @@
                  'The objects returned by "dict.keys()", "dict.values()" and\n'
                  '"dict.items()" are *view objects*.  They provide a dynamic '
                  'view on the\n'
-                 "dictionary's entries, which means that when the dictionary "
+                 'dictionary?s entries, which means that when the dictionary '
                  'changes,\n'
                  'the view reflects these changes.\n'
                  '\n'
@@ -11388,7 +11493,7 @@
                  'which is\n'
                  '   non-random, varies across Python implementations, and '
                  'depends on\n'
-                 "   the dictionary's history of insertions and deletions. If "
+                 '   the dictionary?s history of insertions and deletions. If '
                  'keys,\n'
                  '   values and items views are iterated over with no '
                  'intervening\n'
@@ -11408,7 +11513,7 @@
                  '\n'
                  'x in dictview\n'
                  '\n'
-                 '   Return "True" if *x* is in the underlying dictionary\'s '
+                 '   Return "True" if *x* is in the underlying dictionary?s '
                  'keys, values\n'
                  '   or items (in the latter case, *x* should be a "(key, '
                  'value)"\n'
@@ -11460,8 +11565,7 @@
                  "   {'bacon'}\n"
                  "   >>> keys ^ {'sausage', 'juice'}\n"
                  "   {'juice', 'sausage', 'bacon', 'spam'}\n",
- 'typesmethods': '\n'
-                 'Methods\n'
+ 'typesmethods': 'Methods\n'
                  '*******\n'
                  '\n'
                  'Methods are functions that are called using the attribute '
@@ -11517,15 +11621,14 @@
                  '   >>> c.method.whoami\n'
                  "   'my name is method'\n"
                  '\n'
-                 'See *The standard type hierarchy* for more information.\n',
- 'typesmodules': '\n'
-                 'Modules\n'
+                 'See The standard type hierarchy for more information.\n',
+ 'typesmodules': 'Modules\n'
                  '*******\n'
                  '\n'
                  'The only special operation on a module is attribute access: '
                  '"m.name",\n'
                  'where *m* is a module and *name* accesses a name defined in '
-                 "*m*'s\n"
+                 '*m*?s\n'
                  'symbol table. Module attributes can be assigned to.  (Note '
                  'that the\n'
                  '"import" statement is not, strictly speaking, an operation '
@@ -11538,14 +11641,14 @@
                  '\n'
                  'A special attribute of every module is "__dict__". This is '
                  'the\n'
-                 "dictionary containing the module's symbol table. Modifying "
+                 'dictionary containing the module?s symbol table. Modifying '
                  'this\n'
-                 "dictionary will actually change the module's symbol table, "
+                 'dictionary will actually change the module?s symbol table, '
                  'but direct\n'
                  'assignment to the "__dict__" attribute is not possible (you '
                  'can write\n'
                  '"m.__dict__[\'a\'] = 1", which defines "m.a" to be "1", but '
-                 "you can't\n"
+                 'you can?t\n'
                  'write "m.__dict__ = {}").  Modifying "__dict__" directly is '
                  'not\n'
                  'recommended.\n'
@@ -11556,14 +11659,13 @@
                  'written as\n'
                  '"<module \'os\' from '
                  '\'/usr/local/lib/pythonX.Y/os.pyc\'>".\n',
- 'typesseq': '\n'
-             'Sequence Types --- "list", "tuple", "range"\n'
-             '*******************************************\n'
+ 'typesseq': 'Sequence Types ? "list", "tuple", "range"\n'
+             '*****************************************\n'
              '\n'
              'There are three basic sequence types: lists, tuples, and range\n'
              'objects. Additional sequence types tailored for processing of '
-             '*binary\n'
-             'data* and *text strings* are described in dedicated sections.\n'
+             'binary\n'
+             'data and text strings are described in dedicated sections.\n'
              '\n'
              '\n'
              'Common Sequence Operations\n'
@@ -11598,15 +11700,21 @@
              '+============================+==================================+============+\n'
              '| "x in s"                   | "True" if an item of *s* is      '
              '| (1)        |\n'
+             '|                            | equal to *x*, else "False"       '
+             '|            |\n'
              '+----------------------------+----------------------------------+------------+\n'
              '| "x not in s"               | "False" if an item of *s* is     '
              '| (1)        |\n'
+             '|                            | equal to *x*, else "True"        '
+             '|            |\n'
              '+----------------------------+----------------------------------+------------+\n'
              '| "s + t"                    | the concatenation of *s* and *t* '
              '| (6)(7)     |\n'
              '+----------------------------+----------------------------------+------------+\n'
              '| "s * n" or "n * s"         | equivalent to adding *s* to      '
              '| (2)(7)     |\n'
+             '|                            | itself *n* times                 '
+             '|            |\n'
              '+----------------------------+----------------------------------+------------+\n'
              '| "s[i]"                     | *i*th item of *s*, origin 0      '
              '| (3)        |\n'
@@ -11616,13 +11724,29 @@
              '+----------------------------+----------------------------------+------------+\n'
              '| "s[i:j:k]"                 | slice of *s* from *i* to *j*     '
              '| (3)(5)     |\n'
+             '|                            | with step *k*                    '
+             '|            |\n'
              '+----------------------------+----------------------------------+------------+\n'
+             '| "len(s)"                   | length of *s*                    '
+             '|            |\n'
              '+----------------------------+----------------------------------+------------+\n'
+             '| "min(s)"                   | smallest item of *s*             '
+             '|            |\n'
              '+----------------------------+----------------------------------+------------+\n'
+             '| "max(s)"                   | largest item of *s*              '
+             '|            |\n'
              '+----------------------------+----------------------------------+------------+\n'
              '| "s.index(x[, i[, j]])"     | index of the first occurrence of '
              '| (8)        |\n'
+             '|                            | *x* in *s* (at or after index    '
+             '|            |\n'
+             '|                            | *i* and before index *j*)        '
+             '|            |\n'
              '+----------------------------+----------------------------------+------------+\n'
+             '| "s.count(x)"               | total number of occurrences of   '
+             '|            |\n'
+             '|                            | *x* in *s*                       '
+             '|            |\n'
              '+----------------------------+----------------------------------+------------+\n'
              '\n'
              'Sequences of the same type also support comparisons.  In '
@@ -11632,7 +11756,7 @@
              'element must compare equal and the two sequences must be of the '
              'same\n'
              'type and have the same length.  (For full details see '
-             '*Comparisons* in\n'
+             'Comparisons in\n'
              'the language reference.)\n'
              '\n'
              'Notes:\n'
@@ -11677,9 +11801,9 @@
              '      >>> lists\n'
              '      [[3], [5], [7]]\n'
              '\n'
-             '   Further explanation is available in the FAQ entry *How do I '
-             'create\n'
-             '   a multidimensional list?*.\n'
+             '   Further explanation is available in the FAQ entry How do I '
+             'create a\n'
+             '   multidimensional list?.\n'
              '\n'
              '3. If *i* or *j* is negative, the index is relative to the end '
              'of\n'
@@ -11710,7 +11834,7 @@
              '*j* are\n'
              '   reduced to "len(s) - 1" if they are greater.  If *i* or *j* '
              'are\n'
-             '   omitted or "None", they become "end" values (which end '
+             '   omitted or "None", they become ?end? values (which end '
              'depends on\n'
              '   the sign of *k*).  Note, *k* cannot be zero. If *k* is '
              '"None", it\n'
@@ -11741,7 +11865,7 @@
              'documentation\n'
              '\n'
              '7. Some sequence types (such as "range") only support item\n'
-             "   sequences that follow specific patterns, and hence don't "
+             '   sequences that follow specific patterns, and hence don?t '
              'support\n'
              '   sequence concatenation or repetition.\n'
              '\n'
@@ -11803,33 +11927,81 @@
              '| Operation                      | '
              'Result                           | Notes                 |\n'
              '+================================+==================================+=======================+\n'
+             '| "s[i] = x"                     | item *i* of *s* is replaced '
+             'by   |                       |\n'
+             '|                                | '
+             '*x*                              |                       |\n'
              '+--------------------------------+----------------------------------+-----------------------+\n'
+             '| "s[i:j] = t"                   | slice of *s* from *i* to *j* '
+             'is  |                       |\n'
+             '|                                | replaced by the contents of '
+             'the  |                       |\n'
+             '|                                | iterable '
+             '*t*                     |                       |\n'
              '+--------------------------------+----------------------------------+-----------------------+\n'
+             '| "del s[i:j]"                   | same as "s[i:j] = '
+             '[]"            |                       |\n'
              '+--------------------------------+----------------------------------+-----------------------+\n'
              '| "s[i:j:k] = t"                 | the elements of "s[i:j:k]" '
              'are   | (1)                   |\n'
+             '|                                | replaced by those of '
+             '*t*         |                       |\n'
              '+--------------------------------+----------------------------------+-----------------------+\n'
+             '| "del s[i:j:k]"                 | removes the elements '
+             'of          |                       |\n'
+             '|                                | "s[i:j:k]" from the '
+             'list         |                       |\n'
              '+--------------------------------+----------------------------------+-----------------------+\n'
+             '| "s.append(x)"                  | appends *x* to the end of '
+             'the    |                       |\n'
+             '|                                | sequence (same '
+             'as                |                       |\n'
+             '|                                | "s[len(s):len(s)] = '
+             '[x]")        |                       |\n'
              '+--------------------------------+----------------------------------+-----------------------+\n'
              '| "s.clear()"                    | removes all items from "s" '
              '(same | (5)                   |\n'
+             '|                                | as "del '
+             's[:]")                   |                       |\n'
              '+--------------------------------+----------------------------------+-----------------------+\n'
              '| "s.copy()"                     | creates a shallow copy of '
              '"s"    | (5)                   |\n'
+             '|                                | (same as '
+             '"s[:]")                 |                       |\n'
              '+--------------------------------+----------------------------------+-----------------------+\n'
+             '| "s.extend(t)" or "s += t"      | extends *s* with the contents '
+             'of |                       |\n'
+             '|                                | *t* (for the most part the '
+             'same  |                       |\n'
+             '|                                | as "s[len(s):len(s)] = '
+             't")       |                       |\n'
              '+--------------------------------+----------------------------------+-----------------------+\n'
              '| "s *= n"                       | updates *s* with its '
              'contents    | (6)                   |\n'
+             '|                                | repeated *n* '
+             'times               |                       |\n'
              '+--------------------------------+----------------------------------+-----------------------+\n'
+             '| "s.insert(i, x)"               | inserts *x* into *s* at '
+             'the      |                       |\n'
+             '|                                | index given by *i* (same '
+             'as      |                       |\n'
+             '|                                | "s[i:i] = '
+             '[x]")                  |                       |\n'
              '+--------------------------------+----------------------------------+-----------------------+\n'
              '| "s.pop([i])"                   | retrieves the item at *i* '
              'and    | (2)                   |\n'
+             '|                                | also removes it from '
+             '*s*         |                       |\n'
              '+--------------------------------+----------------------------------+-----------------------+\n'
              '| "s.remove(x)"                  | remove the first item from '
              '*s*   | (3)                   |\n'
+             '|                                | where "s[i] == '
+             'x"                |                       |\n'
              '+--------------------------------+----------------------------------+-----------------------+\n'
              '| "s.reverse()"                  | reverses the items of *s* '
              'in     | (4)                   |\n'
+             '|                                | '
+             'place                            |                       |\n'
              '+--------------------------------+----------------------------------+-----------------------+\n'
              '\n'
              'Notes:\n'
@@ -11850,7 +12022,7 @@
              '   sequence.\n'
              '\n'
              '5. "clear()" and "copy()" are included for consistency with the\n'
-             "   interfaces of mutable containers that don't support slicing\n"
+             '   interfaces of mutable containers that don?t support slicing\n'
              '   operations (such as "dict" and "set")\n'
              '\n'
              '   New in version 3.3: "clear()" and "copy()" methods.\n'
@@ -11860,8 +12032,8 @@
              'sequence.\n'
              '   Items in the sequence are not copied; they are referenced '
              'multiple\n'
-             '   times, as explained for "s * n" under *Common Sequence '
-             'Operations*.\n'
+             '   times, as explained for "s * n" under Common Sequence '
+             'Operations.\n'
              '\n'
              '\n'
              'Lists\n'
@@ -11873,7 +12045,7 @@
              'vary by\n'
              'application).\n'
              '\n'
-             'class class list([iterable])\n'
+             'class list([iterable])\n'
              '\n'
              '   Lists may be constructed in several ways:\n'
              '\n'
@@ -11890,7 +12062,7 @@
              '\n'
              '   The constructor builds a list whose items are the same and in '
              'the\n'
-             "   same order as *iterable*'s items.  *iterable* may be either "
+             '   same order as *iterable*?s items.  *iterable* may be either '
              'a\n'
              '   sequence, a container that supports iteration, or an '
              'iterator\n'
@@ -11907,9 +12079,9 @@
              '"sorted()"\n'
              '   built-in.\n'
              '\n'
-             '   Lists implement all of the *common* and *mutable* sequence\n'
-             '   operations. Lists also provide the following additional '
-             'method:\n'
+             '   Lists implement all of the common and mutable sequence '
+             'operations.\n'
+             '   Lists also provide the following additional method:\n'
              '\n'
              '   sort(*, key=None, reverse=None)\n'
              '\n'
@@ -11922,7 +12094,7 @@
              '      list will likely be left in a partially modified state).\n'
              '\n'
              '      "sort()" accepts two arguments that can only be passed by\n'
-             '      keyword (*keyword-only arguments*):\n'
+             '      keyword (keyword-only arguments):\n'
              '\n'
              '      *key* specifies a function of one argument that is used '
              'to\n'
@@ -11957,8 +12129,8 @@
              'is\n'
              '      stable if it guarantees not to change the relative order '
              'of\n'
-             '      elements that compare equal --- this is helpful for '
-             'sorting in\n'
+             '      elements that compare equal ? this is helpful for sorting '
+             'in\n'
              '      multiple passes (for example, sort by department, then by '
              'salary\n'
              '      grade).\n'
@@ -11987,7 +12159,7 @@
              '"set" or\n'
              '"dict" instance).\n'
              '\n'
-             'class class tuple([iterable])\n'
+             'class tuple([iterable])\n'
              '\n'
              '   Tuples may be constructed in a number of ways:\n'
              '\n'
@@ -12004,7 +12176,7 @@
              '\n'
              '   The constructor builds a tuple whose items are the same and '
              'in the\n'
-             "   same order as *iterable*'s items.  *iterable* may be either "
+             '   same order as *iterable*?s items.  *iterable* may be either '
              'a\n'
              '   sequence, a container that supports iteration, or an '
              'iterator\n'
@@ -12027,7 +12199,7 @@
              'sole\n'
              '   argument.\n'
              '\n'
-             '   Tuples implement all of the *common* sequence operations.\n'
+             '   Tuples implement all of the common sequence operations.\n'
              '\n'
              'For heterogeneous collections of data where access by name is '
              'clearer\n'
@@ -12043,8 +12215,8 @@
              'commonly used for looping a specific number of times in "for" '
              'loops.\n'
              '\n'
-             'class class range(stop)\n'
-             'class class range(start, stop[, step])\n'
+             'class range(stop)\n'
+             'class range(start, stop[, step])\n'
              '\n'
              '   The arguments to the range constructor must be integers '
              '(either\n'
@@ -12095,7 +12267,7 @@
              '      >>> list(range(1, 0))\n'
              '      []\n'
              '\n'
-             '   Ranges implement all of the *common* sequence operations '
+             '   Ranges implement all of the common sequence operations '
              'except\n'
              '   concatenation and repetition (due to the fact that range '
              'objects\n'
@@ -12133,9 +12305,9 @@
              'Range objects implement the "collections.abc.Sequence" ABC, and\n'
              'provide features such as containment tests, element index '
              'lookup,\n'
-             'slicing and support for negative indices (see *Sequence Types '
-             '---\n'
-             'list, tuple, range*):\n'
+             'slicing and support for negative indices (see Sequence Types ? '
+             'list,\n'
+             'tuple, range):\n'
              '\n'
              '>>> r = range(0, 20, 2)\n'
              '>>> r\n'
@@ -12171,15 +12343,14 @@
              'constant\n'
              'time instead of iterating through all items.\n'
              '\n'
-             "Changed in version 3.3: Define '==' and '!=' to compare range "
+             'Changed in version 3.3: Define ?==? and ?!=? to compare range '
              'objects\n'
              'based on the sequence of values they define (instead of '
              'comparing\n'
              'based on object identity).\n'
              '\n'
              'New in version 3.3: The "start", "stop" and "step" attributes.\n',
- 'typesseq-mutable': '\n'
-                     'Mutable Sequence Types\n'
+ 'typesseq-mutable': 'Mutable Sequence Types\n'
                      '**********************\n'
                      '\n'
                      'The operations in the following table are defined on '
@@ -12204,33 +12375,83 @@
                      'Result                           | Notes                 '
                      '|\n'
                      '+================================+==================================+=======================+\n'
+                     '| "s[i] = x"                     | item *i* of *s* is '
+                     'replaced by   |                       |\n'
+                     '|                                | '
+                     '*x*                              |                       '
+                     '|\n'
                      '+--------------------------------+----------------------------------+-----------------------+\n'
+                     '| "s[i:j] = t"                   | slice of *s* from *i* '
+                     'to *j* is  |                       |\n'
+                     '|                                | replaced by the '
+                     'contents of the  |                       |\n'
+                     '|                                | iterable '
+                     '*t*                     |                       |\n'
                      '+--------------------------------+----------------------------------+-----------------------+\n'
+                     '| "del s[i:j]"                   | same as "s[i:j] = '
+                     '[]"            |                       |\n'
                      '+--------------------------------+----------------------------------+-----------------------+\n'
                      '| "s[i:j:k] = t"                 | the elements of '
                      '"s[i:j:k]" are   | (1)                   |\n'
+                     '|                                | replaced by those of '
+                     '*t*         |                       |\n'
                      '+--------------------------------+----------------------------------+-----------------------+\n'
+                     '| "del s[i:j:k]"                 | removes the elements '
+                     'of          |                       |\n'
+                     '|                                | "s[i:j:k]" from the '
+                     'list         |                       |\n'
                      '+--------------------------------+----------------------------------+-----------------------+\n'
+                     '| "s.append(x)"                  | appends *x* to the '
+                     'end of the    |                       |\n'
+                     '|                                | sequence (same '
+                     'as                |                       |\n'
+                     '|                                | "s[len(s):len(s)] = '
+                     '[x]")        |                       |\n'
                      '+--------------------------------+----------------------------------+-----------------------+\n'
                      '| "s.clear()"                    | removes all items '
                      'from "s" (same | (5)                   |\n'
+                     '|                                | as "del '
+                     's[:]")                   |                       |\n'
                      '+--------------------------------+----------------------------------+-----------------------+\n'
                      '| "s.copy()"                     | creates a shallow '
                      'copy of "s"    | (5)                   |\n'
+                     '|                                | (same as '
+                     '"s[:]")                 |                       |\n'
                      '+--------------------------------+----------------------------------+-----------------------+\n'
+                     '| "s.extend(t)" or "s += t"      | extends *s* with the '
+                     'contents of |                       |\n'
+                     '|                                | *t* (for the most '
+                     'part the same  |                       |\n'
+                     '|                                | as "s[len(s):len(s)] '
+                     '= t")       |                       |\n'
                      '+--------------------------------+----------------------------------+-----------------------+\n'
                      '| "s *= n"                       | updates *s* with its '
                      'contents    | (6)                   |\n'
+                     '|                                | repeated *n* '
+                     'times               |                       |\n'
                      '+--------------------------------+----------------------------------+-----------------------+\n'
+                     '| "s.insert(i, x)"               | inserts *x* into *s* '
+                     'at the      |                       |\n'
+                     '|                                | index given by *i* '
+                     '(same as      |                       |\n'
+                     '|                                | "s[i:i] = '
+                     '[x]")                  |                       |\n'
                      '+--------------------------------+----------------------------------+-----------------------+\n'
                      '| "s.pop([i])"                   | retrieves the item at '
                      '*i* and    | (2)                   |\n'
+                     '|                                | also removes it from '
+                     '*s*         |                       |\n'
                      '+--------------------------------+----------------------------------+-----------------------+\n'
                      '| "s.remove(x)"                  | remove the first item '
                      'from *s*   | (3)                   |\n'
+                     '|                                | where "s[i] == '
+                     'x"                |                       |\n'
                      '+--------------------------------+----------------------------------+-----------------------+\n'
                      '| "s.reverse()"                  | reverses the items of '
                      '*s* in     | (4)                   |\n'
+                     '|                                | '
+                     'place                            |                       '
+                     '|\n'
                      '+--------------------------------+----------------------------------+-----------------------+\n'
                      '\n'
                      'Notes:\n'
@@ -12255,7 +12476,7 @@
                      '\n'
                      '5. "clear()" and "copy()" are included for consistency '
                      'with the\n'
-                     "   interfaces of mutable containers that don't support "
+                     '   interfaces of mutable containers that don?t support '
                      'slicing\n'
                      '   operations (such as "dict" and "set")\n'
                      '\n'
@@ -12267,10 +12488,9 @@
                      'the sequence.\n'
                      '   Items in the sequence are not copied; they are '
                      'referenced multiple\n'
-                     '   times, as explained for "s * n" under *Common '
-                     'Sequence Operations*.\n',
- 'unary': '\n'
-          'Unary arithmetic and bitwise operations\n'
+                     '   times, as explained for "s * n" under Common Sequence '
+                     'Operations.\n',
+ 'unary': 'Unary arithmetic and bitwise operations\n'
           '***************************************\n'
           '\n'
           'All unary arithmetic and bitwise operations have the same '
@@ -12292,8 +12512,7 @@
           'In all three cases, if the argument does not have the proper type, '
           'a\n'
           '"TypeError" exception is raised.\n',
- 'while': '\n'
-          'The "while" statement\n'
+ 'while': 'The "while" statement\n'
           '*********************\n'
           '\n'
           'The "while" statement is used for repeated execution as long as an\n'
@@ -12312,32 +12531,32 @@
           '\n'
           'A "break" statement executed in the first suite terminates the '
           'loop\n'
-          'without executing the "else" clause\'s suite.  A "continue" '
+          'without executing the "else" clause?s suite.  A "continue" '
           'statement\n'
           'executed in the first suite skips the rest of the suite and goes '
           'back\n'
           'to testing the expression.\n',
- 'with': '\n'
-         'The "with" statement\n'
+ 'with': 'The "with" statement\n'
          '********************\n'
          '\n'
          'The "with" statement is used to wrap the execution of a block with\n'
-         'methods defined by a context manager (see section *With Statement\n'
-         'Context Managers*). This allows common "try"..."except"..."finally"\n'
-         'usage patterns to be encapsulated for convenient reuse.\n'
+         'methods defined by a context manager (see section With Statement\n'
+         'Context Managers). This allows common "try"?"except"?"finally" '
+         'usage\n'
+         'patterns to be encapsulated for convenient reuse.\n'
          '\n'
          '   with_stmt ::= "with" with_item ("," with_item)* ":" suite\n'
          '   with_item ::= expression ["as" target]\n'
          '\n'
-         'The execution of the "with" statement with one "item" proceeds as\n'
+         'The execution of the "with" statement with one ?item? proceeds as\n'
          'follows:\n'
          '\n'
          '1. The context expression (the expression given in the "with_item")\n'
          '   is evaluated to obtain a context manager.\n'
          '\n'
-         '2. The context manager\'s "__exit__()" is loaded for later use.\n'
+         '2. The context manager?s "__exit__()" is loaded for later use.\n'
          '\n'
-         '3. The context manager\'s "__enter__()" method is invoked.\n'
+         '3. The context manager?s "__enter__()" method is invoked.\n'
          '\n'
          '4. If a target was included in the "with" statement, the return\n'
          '   value from "__enter__()" is assigned to it.\n'
@@ -12351,7 +12570,7 @@
          '\n'
          '5. The suite is executed.\n'
          '\n'
-         '6. The context manager\'s "__exit__()" method is invoked.  If an\n'
+         '6. The context manager?s "__exit__()" method is invoked.  If an\n'
          '   exception caused the suite to be exited, its type, value, and\n'
          '   traceback are passed as arguments to "__exit__()". Otherwise, '
          'three\n'
@@ -12385,23 +12604,23 @@
          '\n'
          'Changed in version 3.1: Support for multiple context expressions.\n'
          '\n'
-         'See also: **PEP 343** - The "with" statement\n'
+         'See also:\n'
          '\n'
+         '  **PEP 343** - The ?with? statement\n'
          '     The specification, background, and examples for the Python '
          '"with"\n'
          '     statement.\n',
- 'yield': '\n'
-          'The "yield" statement\n'
+ 'yield': 'The "yield" statement\n'
           '*********************\n'
           '\n'
           '   yield_stmt ::= yield_expression\n'
           '\n'
-          'A "yield" statement is semantically equivalent to a *yield\n'
-          'expression*. The yield statement can be used to omit the '
-          'parentheses\n'
-          'that would otherwise be required in the equivalent yield '
-          'expression\n'
-          'statement. For example, the yield statements\n'
+          'A "yield" statement is semantically equivalent to a yield '
+          'expression.\n'
+          'The yield statement can be used to omit the parentheses that would\n'
+          'otherwise be required in the equivalent yield expression '
+          'statement.\n'
+          'For example, the yield statements\n'
           '\n'
           '   yield <expr>\n'
           '   yield from <expr>\n'
@@ -12419,5 +12638,6 @@
           'that definition to create a generator function instead of a normal\n'
           'function.\n'
           '\n'
-          'For full details of "yield" semantics, refer to the *Yield\n'
-          'expressions* section.\n'}
+          'For full details of "yield" semantics, refer to the Yield '
+          'expressions\n'
+          'section.\n'}
diff --git a/Misc/NEWS.d/3.5.8rc2.rst b/Misc/NEWS.d/3.5.8rc2.rst
new file mode 100644
index 0000000000000..e4910d6adb1d7
--- /dev/null
+++ b/Misc/NEWS.d/3.5.8rc2.rst
@@ -0,0 +1,20 @@
+.. bpo: 38174
+.. date: 2019-09-23-21-02-46
+.. nonce: MeWuJd
+.. release date: 2019-10-12
+.. section: Security
+
+Update vendorized expat library version to 2.2.8, which resolves
+CVE-2019-15903.
+
+..
+
+.. bpo: 38216
+.. date: 2019-09-27-15-24-45
+.. nonce: -7yvZR
+.. section: Library
+
+Allow the rare code that wants to send invalid http requests from the
+`http.client` library a way to do so.  The fixes for bpo-30458 led to
+breakage for some projects that were relying on this ability to test their
+own behavior in the face of bad requests.
diff --git a/Misc/NEWS.d/next/Library/2019-09-27-15-24-45.bpo-38216.-7yvZR.rst b/Misc/NEWS.d/next/Library/2019-09-27-15-24-45.bpo-38216.-7yvZR.rst
deleted file mode 100644
index ac8e2b042d929..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-09-27-15-24-45.bpo-38216.-7yvZR.rst
+++ /dev/null
@@ -1,4 +0,0 @@
-Allow the rare code that wants to send invalid http requests from the
-`http.client` library a way to do so.  The fixes for bpo-30458 led to
-breakage for some projects that were relying on this ability to test their
-own behavior in the face of bad requests.
diff --git a/Misc/NEWS.d/next/Security/2019-09-23-21-02-46.bpo-38174.MeWuJd.rst b/Misc/NEWS.d/next/Security/2019-09-23-21-02-46.bpo-38174.MeWuJd.rst
deleted file mode 100644
index e3da6fcdd139f..0000000000000
--- a/Misc/NEWS.d/next/Security/2019-09-23-21-02-46.bpo-38174.MeWuJd.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Update vendorized expat library version to 2.2.8, which resolves
-CVE-2019-15903.


From webhook-mailer at python.org  Sat Oct 12 12:08:27 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Sat, 12 Oct 2019 16:08:27 -0000
Subject: [Python-checkins] bpo-38341: Add SMTPNotSupportedError in the
 exports of smtplib (GH-16525)
Message-ID: <mailman.113.1570896508.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/1da648aceb2496c672aff82ba37ee071ac6054ac
commit: 1da648aceb2496c672aff82ba37ee071ac6054ac
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-12T09:08:24-07:00
summary:

bpo-38341: Add SMTPNotSupportedError in the exports of smtplib (GH-16525)


Add SMTPNotSupportedError in the exports of smtplib

Co-Authored-By: Brandt Bucher <brandtbucher at gmail.com>
(cherry picked from commit 3faf826e5879536d2272f1a51c58965a16827f81)

Co-authored-by: nde <denayer.norman at gmail.com>

files:
A Misc/NEWS.d/next/Library/2019-10-01-21-06-18.bpo-38341.uqwgU_.rst
M Lib/smtplib.py

diff --git a/Lib/smtplib.py b/Lib/smtplib.py
index a634f7ae7336d..8e3d4bf0ccf3a 100755
--- a/Lib/smtplib.py
+++ b/Lib/smtplib.py
@@ -54,7 +54,7 @@
 import sys
 from email.base64mime import body_encode as encode_base64
 
-__all__ = ["SMTPException", "SMTPServerDisconnected", "SMTPResponseException",
+__all__ = ["SMTPException", "SMTPNotSupportedError", "SMTPServerDisconnected", "SMTPResponseException",
            "SMTPSenderRefused", "SMTPRecipientsRefused", "SMTPDataError",
            "SMTPConnectError", "SMTPHeloError", "SMTPAuthenticationError",
            "quoteaddr", "quotedata", "SMTP"]
diff --git a/Misc/NEWS.d/next/Library/2019-10-01-21-06-18.bpo-38341.uqwgU_.rst b/Misc/NEWS.d/next/Library/2019-10-01-21-06-18.bpo-38341.uqwgU_.rst
new file mode 100644
index 0000000000000..dd1a6b080b627
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-10-01-21-06-18.bpo-38341.uqwgU_.rst
@@ -0,0 +1 @@
+Add :exc:`smtplib.SMTPNotSupportedError` to the :mod:`smtplib` exported names.


From webhook-mailer at python.org  Sat Oct 12 12:26:09 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Sat, 12 Oct 2019 16:26:09 -0000
Subject: [Python-checkins] bpo-38341: Add SMTPNotSupportedError in the
 exports of smtplib (GH-16525)
Message-ID: <mailman.114.1570897570.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/cb580d6fa8349acff723a7a044181d896d076871
commit: cb580d6fa8349acff723a7a044181d896d076871
branch: 3.7
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-12T09:26:04-07:00
summary:

bpo-38341: Add SMTPNotSupportedError in the exports of smtplib (GH-16525)


Add SMTPNotSupportedError in the exports of smtplib

Co-Authored-By: Brandt Bucher <brandtbucher at gmail.com>
(cherry picked from commit 3faf826e5879536d2272f1a51c58965a16827f81)

Co-authored-by: nde <denayer.norman at gmail.com>

files:
A Misc/NEWS.d/next/Library/2019-10-01-21-06-18.bpo-38341.uqwgU_.rst
M Lib/smtplib.py

diff --git a/Lib/smtplib.py b/Lib/smtplib.py
index 6091c7fb7acad..22d5097c697fc 100755
--- a/Lib/smtplib.py
+++ b/Lib/smtplib.py
@@ -54,7 +54,7 @@
 import sys
 from email.base64mime import body_encode as encode_base64
 
-__all__ = ["SMTPException", "SMTPServerDisconnected", "SMTPResponseException",
+__all__ = ["SMTPException", "SMTPNotSupportedError", "SMTPServerDisconnected", "SMTPResponseException",
            "SMTPSenderRefused", "SMTPRecipientsRefused", "SMTPDataError",
            "SMTPConnectError", "SMTPHeloError", "SMTPAuthenticationError",
            "quoteaddr", "quotedata", "SMTP"]
diff --git a/Misc/NEWS.d/next/Library/2019-10-01-21-06-18.bpo-38341.uqwgU_.rst b/Misc/NEWS.d/next/Library/2019-10-01-21-06-18.bpo-38341.uqwgU_.rst
new file mode 100644
index 0000000000000..dd1a6b080b627
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-10-01-21-06-18.bpo-38341.uqwgU_.rst
@@ -0,0 +1 @@
+Add :exc:`smtplib.SMTPNotSupportedError` to the :mod:`smtplib` exported names.


From webhook-mailer at python.org  Sat Oct 12 12:58:16 2019
From: webhook-mailer at python.org (Abhilash Raj)
Date: Sat, 12 Oct 2019 16:58:16 -0000
Subject: [Python-checkins] [3.8] bpo-38449: Revert "bpo-22347: Update
 mimetypes.guess_type to allow oper parsing of URLs" (GH-16724) (GH-16728)
Message-ID: <mailman.115.1570899496.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/5a638a805503131f4a9cc2bbc5944611295c1500
commit: 5a638a805503131f4a9cc2bbc5944611295c1500
branch: 3.8
author: Abhilash Raj <maxking at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-12T09:58:11-07:00
summary:

[3.8] bpo-38449: Revert "bpo-22347: Update mimetypes.guess_type to allow oper parsing of URLs" (GH-16724) (GH-16728)

This reverts commit 87bd2071c756188b6cd577889fb1682831142ceb.

The reason for revert is a regression caused by the change in 3.8.0rc1, see bpo-38449 for more details.

https://bugs.python.org/issue38449
(cherry picked from commit 19a3d873005e5730eeabdc394c961e93f2ec02f0)

Co-authored-by: Abhilash Raj <maxking at users.noreply.github.com>

files:
A Misc/NEWS.d/next/Library/2019-10-11-18-49-00.bpo-38449.9TWMlz.rst
M Lib/mimetypes.py
M Lib/test/test_mimetypes.py
M Lib/test/test_urllib2.py

diff --git a/Lib/mimetypes.py b/Lib/mimetypes.py
index 9fdd6ebd6348f..9b42bf6dd2ca7 100644
--- a/Lib/mimetypes.py
+++ b/Lib/mimetypes.py
@@ -114,8 +114,7 @@ def guess_type(self, url, strict=True):
         but non-standard types.
         """
         url = os.fspath(url)
-        p = urllib.parse.urlparse(url)
-        scheme, url = p.scheme, p.path
+        scheme, url = urllib.parse._splittype(url)
         if scheme == 'data':
             # syntax of data URLs:
             # dataurl   := "data:" [ mediatype ] [ ";base64" ] "," data
diff --git a/Lib/test/test_mimetypes.py b/Lib/test/test_mimetypes.py
index 7761c3fe867a7..bfd5eeedaa77b 100644
--- a/Lib/test/test_mimetypes.py
+++ b/Lib/test/test_mimetypes.py
@@ -51,14 +51,6 @@ def test_non_standard_types(self):
         eq(self.db.guess_type('foo.xul', strict=False), ('text/xul', None))
         eq(self.db.guess_extension('image/jpg', strict=False), '.jpg')
 
-    def test_url(self):
-        result = self.db.guess_type('http://host.html')
-        msg = 'URL only has a host name, not a file'
-        self.assertSequenceEqual(result, (None, None), msg)
-        result = self.db.guess_type('http://example.com/host.html')
-        msg = 'Should be text/html'
-        self.assertSequenceEqual(result, ('text/html', None), msg)
-
     def test_guess_all_types(self):
         eq = self.assertEqual
         unless = self.assertTrue
diff --git a/Lib/test/test_urllib2.py b/Lib/test/test_urllib2.py
index e7e80bc5dabfe..61e3ecc612d6c 100644
--- a/Lib/test/test_urllib2.py
+++ b/Lib/test/test_urllib2.py
@@ -744,7 +744,7 @@ def connect_ftp(self, user, passwd, host, port, dirs,
              ["foo", "bar"], "", None),
             ("ftp://localhost/baz.gif;type=a",
              "localhost", ftplib.FTP_PORT, "", "", "A",
-             [], "baz.gif", "image/gif"),
+             [], "baz.gif", None),  # XXX really this should guess image/gif
             ]:
             req = Request(url)
             req.timeout = None
diff --git a/Misc/NEWS.d/next/Library/2019-10-11-18-49-00.bpo-38449.9TWMlz.rst b/Misc/NEWS.d/next/Library/2019-10-11-18-49-00.bpo-38449.9TWMlz.rst
new file mode 100644
index 0000000000000..f7b1dbf8d8951
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-10-11-18-49-00.bpo-38449.9TWMlz.rst
@@ -0,0 +1,2 @@
+Revert GH-15522, which introduces a regression in
+:meth:`mimetypes.guess_type` due to improper handling of filenames as urls.


From webhook-mailer at python.org  Sat Oct 12 13:02:29 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Sat, 12 Oct 2019 17:02:29 -0000
Subject: [Python-checkins] bpo-38332: Catch KeyError from unknown cte in
 encoded-word. (GH-16503)
Message-ID: <mailman.116.1570899750.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/febe359559781019c0c8432a2f768809d00af6af
commit: febe359559781019c0c8432a2f768809d00af6af
branch: 3.7
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-12T10:02:23-07:00
summary:

bpo-38332: Catch KeyError from unknown cte in encoded-word. (GH-16503)


KeyError should cause a failure in parsing the encoded word and should be caught and raised as a _InvalidEWError instead.
(cherry picked from commit 65dcc8a8dc41d3453fd6b987073a5f1b30c5c0fd)

Co-authored-by: Andrei Troie <andreitroie90 at gmail.com>

files:
A Misc/NEWS.d/next/Library/2019-10-05-02-07-52.bpo-38332.hwrPN7.rst
M Lib/email/_header_value_parser.py
M Lib/test/test_email/test__encoded_words.py
M Lib/test/test_email/test__header_value_parser.py

diff --git a/Lib/email/_header_value_parser.py b/Lib/email/_header_value_parser.py
index 7cc9a4672601f..3197d49762e9a 100644
--- a/Lib/email/_header_value_parser.py
+++ b/Lib/email/_header_value_parser.py
@@ -1055,7 +1055,7 @@ def get_encoded_word(value):
     value = ''.join(remainder)
     try:
         text, charset, lang, defects = _ew.decode('=?' + tok + '?=')
-    except ValueError:
+    except (ValueError, KeyError):
         raise _InvalidEwError(
             "encoded word format invalid: '{}'".format(ew.cte))
     ew.charset = charset
diff --git a/Lib/test/test_email/test__encoded_words.py b/Lib/test/test_email/test__encoded_words.py
index 5a59aebba89be..0b8b1de3359aa 100644
--- a/Lib/test/test_email/test__encoded_words.py
+++ b/Lib/test/test_email/test__encoded_words.py
@@ -58,6 +58,8 @@ def test_wrong_format_input_raises(self):
             _ew.decode('=?')
         with self.assertRaises(ValueError):
             _ew.decode('')
+        with self.assertRaises(KeyError):
+            _ew.decode('=?utf-8?X?somevalue?=')
 
     def _test(self, source, result, charset='us-ascii', lang='', defects=[]):
         res, char, l, d = _ew.decode(source)
diff --git a/Lib/test/test_email/test__header_value_parser.py b/Lib/test/test_email/test__header_value_parser.py
index 8bb0365739f7e..4894d8f22e124 100644
--- a/Lib/test/test_email/test__header_value_parser.py
+++ b/Lib/test/test_email/test__header_value_parser.py
@@ -89,6 +89,10 @@ def test_get_encoded_word_missing_middle_raises(self):
         with self.assertRaises(errors.HeaderParseError):
             parser.get_encoded_word('=?abc?=')
 
+    def test_get_encoded_word_invalid_cte(self):
+        with self.assertRaises(errors.HeaderParseError):
+            parser.get_encoded_word('=?utf-8?X?somevalue?=')
+
     def test_get_encoded_word_valid_ew(self):
         self._test_get_x(parser.get_encoded_word,
                          '=?us-ascii?q?this_is_a_test?=  bird',
@@ -399,6 +403,14 @@ def test_get_unstructured_invalid_ew(self):
             [],
             '')
 
+    def test_get_unstructured_invalid_ew_cte(self):
+        self._test_get_x(self._get_unst,
+            '=?utf-8?X?=somevalue?=',
+            '=?utf-8?X?=somevalue?=',
+            '=?utf-8?X?=somevalue?=',
+            [],
+            '')
+
     # get_qp_ctext
 
     def test_get_qp_ctext_only(self):
diff --git a/Misc/NEWS.d/next/Library/2019-10-05-02-07-52.bpo-38332.hwrPN7.rst b/Misc/NEWS.d/next/Library/2019-10-05-02-07-52.bpo-38332.hwrPN7.rst
new file mode 100644
index 0000000000000..600c702cf3bbd
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-10-05-02-07-52.bpo-38332.hwrPN7.rst
@@ -0,0 +1,3 @@
+Prevent :exc:`KeyError` thrown by :func:`_encoded_words.decode` when given
+an encoded-word with invalid content-type encoding from propagating all the
+way to :func:`email.message.get`.


From webhook-mailer at python.org  Sat Oct 12 13:03:28 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Sat, 12 Oct 2019 17:03:28 -0000
Subject: [Python-checkins] bpo-38332: Catch KeyError from unknown cte in
 encoded-word. (GH-16503)
Message-ID: <mailman.117.1570899809.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/e540bb546163f108c7c304f2e6865efaa78cd4c2
commit: e540bb546163f108c7c304f2e6865efaa78cd4c2
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-12T10:03:24-07:00
summary:

bpo-38332: Catch KeyError from unknown cte in encoded-word. (GH-16503)


KeyError should cause a failure in parsing the encoded word and should be caught and raised as a _InvalidEWError instead.
(cherry picked from commit 65dcc8a8dc41d3453fd6b987073a5f1b30c5c0fd)

Co-authored-by: Andrei Troie <andreitroie90 at gmail.com>

files:
A Misc/NEWS.d/next/Library/2019-10-05-02-07-52.bpo-38332.hwrPN7.rst
M Lib/email/_header_value_parser.py
M Lib/test/test_email/test__encoded_words.py
M Lib/test/test_email/test__header_value_parser.py

diff --git a/Lib/email/_header_value_parser.py b/Lib/email/_header_value_parser.py
index 16c19907d68d5..1668b4a14e9b9 100644
--- a/Lib/email/_header_value_parser.py
+++ b/Lib/email/_header_value_parser.py
@@ -1057,7 +1057,7 @@ def get_encoded_word(value):
     value = ''.join(remainder)
     try:
         text, charset, lang, defects = _ew.decode('=?' + tok + '?=')
-    except ValueError:
+    except (ValueError, KeyError):
         raise _InvalidEwError(
             "encoded word format invalid: '{}'".format(ew.cte))
     ew.charset = charset
diff --git a/Lib/test/test_email/test__encoded_words.py b/Lib/test/test_email/test__encoded_words.py
index 5a59aebba89be..0b8b1de3359aa 100644
--- a/Lib/test/test_email/test__encoded_words.py
+++ b/Lib/test/test_email/test__encoded_words.py
@@ -58,6 +58,8 @@ def test_wrong_format_input_raises(self):
             _ew.decode('=?')
         with self.assertRaises(ValueError):
             _ew.decode('')
+        with self.assertRaises(KeyError):
+            _ew.decode('=?utf-8?X?somevalue?=')
 
     def _test(self, source, result, charset='us-ascii', lang='', defects=[]):
         res, char, l, d = _ew.decode(source)
diff --git a/Lib/test/test_email/test__header_value_parser.py b/Lib/test/test_email/test__header_value_parser.py
index dd33b065c804b..e442c44a2a74d 100644
--- a/Lib/test/test_email/test__header_value_parser.py
+++ b/Lib/test/test_email/test__header_value_parser.py
@@ -89,6 +89,10 @@ def test_get_encoded_word_missing_middle_raises(self):
         with self.assertRaises(errors.HeaderParseError):
             parser.get_encoded_word('=?abc?=')
 
+    def test_get_encoded_word_invalid_cte(self):
+        with self.assertRaises(errors.HeaderParseError):
+            parser.get_encoded_word('=?utf-8?X?somevalue?=')
+
     def test_get_encoded_word_valid_ew(self):
         self._test_get_x(parser.get_encoded_word,
                          '=?us-ascii?q?this_is_a_test?=  bird',
@@ -399,6 +403,14 @@ def test_get_unstructured_invalid_ew(self):
             [],
             '')
 
+    def test_get_unstructured_invalid_ew_cte(self):
+        self._test_get_x(self._get_unst,
+            '=?utf-8?X?=somevalue?=',
+            '=?utf-8?X?=somevalue?=',
+            '=?utf-8?X?=somevalue?=',
+            [],
+            '')
+
     # get_qp_ctext
 
     def test_get_qp_ctext_only(self):
diff --git a/Misc/NEWS.d/next/Library/2019-10-05-02-07-52.bpo-38332.hwrPN7.rst b/Misc/NEWS.d/next/Library/2019-10-05-02-07-52.bpo-38332.hwrPN7.rst
new file mode 100644
index 0000000000000..600c702cf3bbd
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-10-05-02-07-52.bpo-38332.hwrPN7.rst
@@ -0,0 +1,3 @@
+Prevent :exc:`KeyError` thrown by :func:`_encoded_words.decode` when given
+an encoded-word with invalid content-type encoding from propagating all the
+way to :func:`email.message.get`.


From webhook-mailer at python.org  Sat Oct 12 13:24:30 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Sat, 12 Oct 2019 17:24:30 -0000
Subject: [Python-checkins] bpo-35800: Deprecate smtpd.MailmanProxy (GH-11675)
Message-ID: <mailman.118.1570901071.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/822922af90f389b20166f0cd6f5b7c76fd6d42d9
commit: 822922af90f389b20166f0cd6f5b7c76fd6d42d9
branch: master
author: Samuel Colvin <samcolvin at gmail.com>
committer: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
date: 2019-10-12T10:24:26-07:00
summary:

bpo-35800: Deprecate smtpd.MailmanProxy (GH-11675)



Since `smtpd.MailmanProxy` is already broken, it is not formally deprecated in 3.9. It will be removed in 3.10.


https://bugs.python.org/issue35800

files:
A Misc/NEWS.d/next/Library/2019-01-25-17-12-17.bpo-35800.MCGJdQ.rst
M Doc/library/smtpd.rst
M Lib/smtpd.py
M Misc/ACKS

diff --git a/Doc/library/smtpd.rst b/Doc/library/smtpd.rst
index 85ee8a75cf77a..d84e74a8ceaaf 100644
--- a/Doc/library/smtpd.rst
+++ b/Doc/library/smtpd.rst
@@ -148,6 +148,12 @@ MailmanProxy Objects
 
 .. class:: MailmanProxy(localaddr, remoteaddr)
 
+   .. deprecated-removed:: 3.9 3.11
+
+      :class:`MailmanProxy` is deprecated, it depends on a ``Mailman``
+      module which no longer exists and therefore is already broken.
+
+
    Create a new pure proxy server. Arguments are as per :class:`SMTPServer`.
    Everything will be relayed to *remoteaddr*, unless local mailman configurations
    knows about an address, in which case it will be handled via mailman.  Note that
diff --git a/Lib/smtpd.py b/Lib/smtpd.py
index 8103ca9af0d7b..8f1a22e937871 100755
--- a/Lib/smtpd.py
+++ b/Lib/smtpd.py
@@ -779,6 +779,8 @@ def _deliver(self, mailfrom, rcpttos, data):
 
 class MailmanProxy(PureProxy):
     def __init__(self, *args, **kwargs):
+        warn('MailmanProxy is deprecated and will be removed '
+             'in future', DeprecationWarning, 2)
         if 'enable_SMTPUTF8' in kwargs and kwargs['enable_SMTPUTF8']:
             raise ValueError("MailmanProxy does not support SMTPUTF8.")
         super(PureProxy, self).__init__(*args, **kwargs)
diff --git a/Misc/ACKS b/Misc/ACKS
index 71e61c3db386c..d8e2630814a86 100644
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -321,6 +321,7 @@ Benjamin Collar
 Jeffery Collins
 Robert Collins
 Paul Colomiets
+Samuel Colvin
 Christophe Combelles
 Geremy Condra
 Denver Coneybeare
diff --git a/Misc/NEWS.d/next/Library/2019-01-25-17-12-17.bpo-35800.MCGJdQ.rst b/Misc/NEWS.d/next/Library/2019-01-25-17-12-17.bpo-35800.MCGJdQ.rst
new file mode 100644
index 0000000000000..27e8d0bb3a097
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-01-25-17-12-17.bpo-35800.MCGJdQ.rst
@@ -0,0 +1 @@
+Deprecate ``smtpd.MailmanProxy`` ready for future removal.


From webhook-mailer at python.org  Sat Oct 12 13:46:20 2019
From: webhook-mailer at python.org (Yury Selivanov)
Date: Sat, 12 Oct 2019 17:46:20 -0000
Subject: [Python-checkins] Announce the change in the CancelledError
 inheritance (GH-16730)
Message-ID: <mailman.119.1570902381.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/e634da27471a76f4abe685e86e6e302e0ed6b553
commit: e634da27471a76f4abe685e86e6e302e0ed6b553
branch: master
author: Phil Jones <philip.graham.jones at googlemail.com>
committer: Yury Selivanov <yury at magic.io>
date: 2019-10-12T13:46:13-04:00
summary:

Announce the change in the CancelledError inheritance (GH-16730)

This is a fairly noticeable change that requires adjustments in
existing asyncio code. It should therefore be announced.

files:
M Doc/whatsnew/3.8.rst

diff --git a/Doc/whatsnew/3.8.rst b/Doc/whatsnew/3.8.rst
index 4cab2f7af118d..9bf028deaa9de 100644
--- a/Doc/whatsnew/3.8.rst
+++ b/Doc/whatsnew/3.8.rst
@@ -1659,6 +1659,10 @@ Changes in the Python API
   environment variable and does not use :envvar:`HOME`, which is not normally
   set for regular user accounts.
 
+* The Exception :class:`asyncio.CancelledError` now inherits from
+  :class:`BaseException` rather than a :class:`Exception`.
+  (Contributed by Yury Selivanov in :issue:`13528`.)
+
 .. _bpo-36085-whatsnew:
 
 * DLL dependencies for extension modules and DLLs loaded with :mod:`ctypes` on


From webhook-mailer at python.org  Sat Oct 12 13:53:40 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Sat, 12 Oct 2019 17:53:40 -0000
Subject: [Python-checkins] Announce the change in the CancelledError
 inheritance (GH-16730)
Message-ID: <mailman.120.1570902820.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/fdc7cbeedc20f96733788a0aa728f185465b7954
commit: fdc7cbeedc20f96733788a0aa728f185465b7954
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-12T10:53:32-07:00
summary:

Announce the change in the CancelledError inheritance (GH-16730)


This is a fairly noticeable change that requires adjustments in
existing asyncio code. It should therefore be announced.
(cherry picked from commit e634da27471a76f4abe685e86e6e302e0ed6b553)

Co-authored-by: Phil Jones <philip.graham.jones at googlemail.com>

files:
M Doc/whatsnew/3.8.rst

diff --git a/Doc/whatsnew/3.8.rst b/Doc/whatsnew/3.8.rst
index 4cab2f7af118d..9bf028deaa9de 100644
--- a/Doc/whatsnew/3.8.rst
+++ b/Doc/whatsnew/3.8.rst
@@ -1659,6 +1659,10 @@ Changes in the Python API
   environment variable and does not use :envvar:`HOME`, which is not normally
   set for regular user accounts.
 
+* The Exception :class:`asyncio.CancelledError` now inherits from
+  :class:`BaseException` rather than a :class:`Exception`.
+  (Contributed by Yury Selivanov in :issue:`13528`.)
+
 .. _bpo-36085-whatsnew:
 
 * DLL dependencies for extension modules and DLLs loaded with :mod:`ctypes` on


From webhook-mailer at python.org  Sat Oct 12 13:53:40 2019
From: webhook-mailer at python.org (Raymond Hettinger)
Date: Sat, 12 Oct 2019 17:53:40 -0000
Subject: [Python-checkins] Fix minor typos in Whatsnew
Message-ID: <mailman.121.1570902821.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/547c60c96e0119e559fb7312beea5fb36be66fdc
commit: 547c60c96e0119e559fb7312beea5fb36be66fdc
branch: master
author: Hugo van Kemenade <hugovk at users.noreply.github.com>
committer: Raymond Hettinger <rhettinger at users.noreply.github.com>
date: 2019-10-12T10:53:36-07:00
summary:

Fix minor typos in Whatsnew

files:
M Doc/whatsnew/3.8.rst

diff --git a/Doc/whatsnew/3.8.rst b/Doc/whatsnew/3.8.rst
index 9bf028deaa9de..a609d19776e3d 100644
--- a/Doc/whatsnew/3.8.rst
+++ b/Doc/whatsnew/3.8.rst
@@ -183,7 +183,7 @@ keywords, the parameters names remain available for use in ``**kwargs``::
 
 This greatly simplifies the implementation of functions and methods
 that need to accept arbitrary keyword arguments.  For example, here
-is an except from code in the :mod:`collections` module::
+is an excerpt from code in the :mod:`collections` module::
 
   class Counter(dict):
 
@@ -215,7 +215,7 @@ subdirectories).
 Debug build uses the same ABI as release build
 -----------------------------------------------
 
-Python now uses the same ABI whether it built in release or debug mode. On
+Python now uses the same ABI whether it's built in release or debug mode. On
 Unix, when Python is built in debug mode, it is now possible to load C
 extensions built in release mode and C extensions built using the stable ABI.
 
@@ -567,7 +567,7 @@ The :func:`ast.parse` function has some new flags:
   comments" (returned for function definition AST nodes);
 
 * ``feature_version=(3, N)`` allows specifying an earlier Python 3
-  version.  (For example, ``feature_version=(3, 4)`` will treat
+  version. (For example, ``feature_version=(3, 4)`` will treat
   ``async`` and ``await`` as non-reserved words.)
 
 New function :func:`ast.get_source_segment` returns the source code
@@ -1284,7 +1284,7 @@ Build and C API Changes
     without calling functions. This API is now installed by ``make install``.
 
   (Contributed by Victor Stinner in :issue:`35134` and :issue:`35081`,
-  work initiated by Eric Snow in Python 3.7)
+  work initiated by Eric Snow in Python 3.7.)
 
 * Some macros have been converted to static inline functions: parameter types
   and return type are well defined, they don't have issues specific to macros,
@@ -1378,12 +1378,11 @@ Deprecated
 
   Implementations of these methods have been ignoring their *index* parameter,
   and returning the next item instead.
-
   (Contributed by Berker Peksag in :issue:`9372`.)
 
 * The :class:`typing.NamedTuple` class has deprecated the ``_field_types``
   attribute in favor of the ``__annotations__`` attribute which has the same
-  information.  (Contributed by Raymond Hettinger in :issue:`36320`.)
+  information. (Contributed by Raymond Hettinger in :issue:`36320`.)
 
 * :mod:`ast` classes ``Num``, ``Str``, ``Bytes``, ``NameConstant`` and
   ``Ellipsis`` are considered deprecated and will be removed in future Python
@@ -1411,7 +1410,6 @@ Deprecated
   parameter of functions :func:`~gettext.translation` and
   :func:`~gettext.install` are also deprecated, since they are only used for
   for the ``l*gettext()`` functions.
-
   (Contributed by Serhiy Storchaka in :issue:`33710`.)
 
 * The :meth:`~threading.Thread.isAlive()` method of :class:`threading.Thread` has been deprecated.
@@ -1493,11 +1491,11 @@ The following features and APIs have been removed from Python 3.8:
 
 * The ``bufsize`` keyword argument of :func:`fileinput.input` and
   :func:`fileinput.FileInput` which was ignored and deprecated since Python 3.6
-  has been removed. :issue:`36952` (Contributed by Matthias Bussonnier)
+  has been removed. :issue:`36952` (Contributed by Matthias Bussonnier.)
 
 * The functions :func:`sys.set_coroutine_wrapper` and
   :func:`sys.get_coroutine_wrapper` deprecated in Python 3.7 have been removed;
-  :issue:`36933` (Contributed by Matthias Bussonnier)
+  :issue:`36933` (Contributed by Matthias Bussonnier.)
 
 
 Porting to Python 3.8
@@ -1523,9 +1521,9 @@ Changes in Python behavior
   (Contributed by Serhiy Storchaka in :issue:`34850`.)
 
 * The CPython interpreter can swallow exceptions in some circumstances.
-  In Python 3.8 this happens in less cases.  In particular, exceptions
+  In Python 3.8 this happens in fewer cases.  In particular, exceptions
   raised when getting the attribute from the type dictionary are no longer
-  ignored.  (Contributed by Serhiy Storchaka in :issue:`35459`.)
+  ignored. (Contributed by Serhiy Storchaka in :issue:`35459`.)
 
 * Removed ``__str__`` implementations from builtin types :class:`bool`,
   :class:`int`, :class:`float`, :class:`complex` and few classes from
@@ -1620,7 +1618,7 @@ Changes in the Python API
   16 KiB to 1 MiB.
 
 * The ``PyGC_Head`` struct has changed completely.  All code that touched the
-  struct member should be rewritten.  (See :issue:`33597`)
+  struct member should be rewritten.  (See :issue:`33597`.)
 
 * The ``PyInterpreterState`` struct has been moved into the "internal"
   header files (specifically Include/internal/pycore_pystate.h).  An


From webhook-mailer at python.org  Sat Oct 12 14:50:07 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Sat, 12 Oct 2019 18:50:07 -0000
Subject: [Python-checkins] [3.7] bpo-38449: Revert "bpo-22347: Update
 mimetypes.guess_type to allow oper parsing of URLs (GH-15685)" (GH-16724)
 (GH-16727)
Message-ID: <mailman.122.1570906208.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/164bee296ab1f87cc05566b39ee8fb9fb64b3e5a
commit: 164bee296ab1f87cc05566b39ee8fb9fb64b3e5a
branch: 3.7
author: Abhilash Raj <maxking at users.noreply.github.com>
committer: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
date: 2019-10-12T11:50:03-07:00
summary:

[3.7] bpo-38449: Revert "bpo-22347: Update mimetypes.guess_type to allow oper parsing of URLs (GH-15685)" (GH-16724) (GH-16727)



Reverts GH-15687 which caused the issue.


https://bugs.python.org/issue22347

https://bugs.python.org/issue38449

files:
A Misc/NEWS.d/next/Library/2019-10-12-08-57-34.bpo-38449.9TWMlz.rst
M Lib/mimetypes.py
M Lib/test/test_mimetypes.py
M Lib/test/test_urllib2.py

diff --git a/Lib/mimetypes.py b/Lib/mimetypes.py
index b5e1ce1ca1041..bcf522835f4c9 100644
--- a/Lib/mimetypes.py
+++ b/Lib/mimetypes.py
@@ -114,8 +114,7 @@ def guess_type(self, url, strict=True):
         but non-standard types.
         """
         url = os.fspath(url)
-        p = urllib.parse.urlparse(url)
-        scheme, url = p.scheme, p.path
+        scheme, url = urllib.parse.splittype(url)
         if scheme == 'data':
             # syntax of data URLs:
             # dataurl   := "data:" [ mediatype ] [ ";base64" ] "," data
diff --git a/Lib/test/test_mimetypes.py b/Lib/test/test_mimetypes.py
index 2f26edeb99037..adbec8dab4266 100644
--- a/Lib/test/test_mimetypes.py
+++ b/Lib/test/test_mimetypes.py
@@ -50,14 +50,6 @@ def test_non_standard_types(self):
         eq(self.db.guess_type('foo.xul', strict=False), ('text/xul', None))
         eq(self.db.guess_extension('image/jpg', strict=False), '.jpg')
 
-    def test_url(self):
-        result = self.db.guess_type('http://host.html')
-        msg = 'URL only has a host name, not a file'
-        self.assertSequenceEqual(result, (None, None), msg)
-        result = self.db.guess_type('http://example.com/host.html')
-        msg = 'Should be text/html'
-        self.assertSequenceEqual(result, ('text/html', None), msg)
-
     def test_guess_all_types(self):
         eq = self.assertEqual
         unless = self.assertTrue
diff --git a/Lib/test/test_urllib2.py b/Lib/test/test_urllib2.py
index 0677390c2b0b2..876fcd4199fb9 100644
--- a/Lib/test/test_urllib2.py
+++ b/Lib/test/test_urllib2.py
@@ -746,7 +746,7 @@ def connect_ftp(self, user, passwd, host, port, dirs,
              ["foo", "bar"], "", None),
             ("ftp://localhost/baz.gif;type=a",
              "localhost", ftplib.FTP_PORT, "", "", "A",
-             [], "baz.gif", "image/gif"),
+             [], "baz.gif", None),  # XXX really this should guess image/gif
             ]:
             req = Request(url)
             req.timeout = None
diff --git a/Misc/NEWS.d/next/Library/2019-10-12-08-57-34.bpo-38449.9TWMlz.rst b/Misc/NEWS.d/next/Library/2019-10-12-08-57-34.bpo-38449.9TWMlz.rst
new file mode 100644
index 0000000000000..f7b1dbf8d8951
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-10-12-08-57-34.bpo-38449.9TWMlz.rst
@@ -0,0 +1,2 @@
+Revert GH-15522, which introduces a regression in
+:meth:`mimetypes.guess_type` due to improper handling of filenames as urls.


From webhook-mailer at python.org  Sat Oct 12 15:14:14 2019
From: webhook-mailer at python.org (Pablo Galindo)
Date: Sat, 12 Oct 2019 19:14:14 -0000
Subject: [Python-checkins] bpo-37731: Reorder includes in xmltok.c to avoid
 redefinition of _POSIX_C_SOURCE (GH-16733)
Message-ID: <mailman.123.1570907656.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/8177404d520e81f16324a900f093adf3856d33f8
commit: 8177404d520e81f16324a900f093adf3856d33f8
branch: master
author: Pablo Galindo <Pablogsal at gmail.com>
committer: GitHub <noreply at github.com>
date: 2019-10-12T20:14:11+01:00
summary:

bpo-37731: Reorder includes in xmltok.c to avoid redefinition of _POSIX_C_SOURCE (GH-16733)

files:
M Modules/expat/xmltok.c

diff --git a/Modules/expat/xmltok.c b/Modules/expat/xmltok.c
index 11e9d1ccdad42..54cfedb85c28c 100644
--- a/Modules/expat/xmltok.c
+++ b/Modules/expat/xmltok.c
@@ -30,6 +30,14 @@
    USE OR OTHER DEALINGS IN THE SOFTWARE.
 */
 
+#ifdef _WIN32
+#  include "winconfig.h"
+#else
+#  ifdef HAVE_EXPAT_CONFIG_H
+#    include <expat_config.h>
+#  endif
+#endif /* ndef _WIN32 */
+
 #include <stddef.h>
 #include <string.h> /* memcpy */
 
@@ -42,14 +50,6 @@
 #  include <stdbool.h>
 #endif
 
-#ifdef _WIN32
-#  include "winconfig.h"
-#else
-#  ifdef HAVE_EXPAT_CONFIG_H
-#    include <expat_config.h>
-#  endif
-#endif /* ndef _WIN32 */
-
 #include "expat_external.h"
 #include "internal.h"
 #include "xmltok.h"


From webhook-mailer at python.org  Sat Oct 12 15:32:53 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Sat, 12 Oct 2019 19:32:53 -0000
Subject: [Python-checkins] bpo-37731: Reorder includes in xmltok.c to avoid
 redefinition of _POSIX_C_SOURCE (GH-16733)
Message-ID: <mailman.124.1570908775.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/ba44ea6ff8be2b7874c4358f89d5b6f4a558f711
commit: ba44ea6ff8be2b7874c4358f89d5b6f4a558f711
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-12T12:32:48-07:00
summary:

bpo-37731: Reorder includes in xmltok.c to avoid redefinition of _POSIX_C_SOURCE (GH-16733)

(cherry picked from commit 8177404d520e81f16324a900f093adf3856d33f8)

Co-authored-by: Pablo Galindo <Pablogsal at gmail.com>

files:
M Modules/expat/xmltok.c

diff --git a/Modules/expat/xmltok.c b/Modules/expat/xmltok.c
index 11e9d1ccdad42..54cfedb85c28c 100644
--- a/Modules/expat/xmltok.c
+++ b/Modules/expat/xmltok.c
@@ -30,6 +30,14 @@
    USE OR OTHER DEALINGS IN THE SOFTWARE.
 */
 
+#ifdef _WIN32
+#  include "winconfig.h"
+#else
+#  ifdef HAVE_EXPAT_CONFIG_H
+#    include <expat_config.h>
+#  endif
+#endif /* ndef _WIN32 */
+
 #include <stddef.h>
 #include <string.h> /* memcpy */
 
@@ -42,14 +50,6 @@
 #  include <stdbool.h>
 #endif
 
-#ifdef _WIN32
-#  include "winconfig.h"
-#else
-#  ifdef HAVE_EXPAT_CONFIG_H
-#    include <expat_config.h>
-#  endif
-#endif /* ndef _WIN32 */
-
 #include "expat_external.h"
 #include "internal.h"
 #include "xmltok.h"


From webhook-mailer at python.org  Sat Oct 12 16:25:00 2019
From: webhook-mailer at python.org (Gregory P. Smith)
Date: Sat, 12 Oct 2019 20:25:00 -0000
Subject: [Python-checkins] bpo-38417: Add umask support to subprocess
 (GH-16726)
Message-ID: <mailman.125.1570911901.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/f3751efb5c8b53b37efbbf75d9422c1d11c01646
commit: f3751efb5c8b53b37efbbf75d9422c1d11c01646
branch: master
author: Gregory P. Smith <greg at krypto.org>
committer: GitHub <noreply at github.com>
date: 2019-10-12T13:24:56-07:00
summary:

bpo-38417: Add umask support to subprocess (GH-16726)

On POSIX systems, allow the umask to be set in the child process before we exec.

files:
A Misc/NEWS.d/next/Library/2019-10-12-00-13-47.bpo-38417.W7x_aS.rst
M Doc/library/subprocess.rst
M Lib/multiprocessing/util.py
M Lib/subprocess.py
M Lib/test/test_capi.py
M Lib/test/test_subprocess.py
M Modules/_posixsubprocess.c

diff --git a/Doc/library/subprocess.rst b/Doc/library/subprocess.rst
index 1a98bb33d0c1c..19290bfc34c7d 100644
--- a/Doc/library/subprocess.rst
+++ b/Doc/library/subprocess.rst
@@ -339,9 +339,9 @@ functions.
                  stderr=None, preexec_fn=None, close_fds=True, shell=False, \
                  cwd=None, env=None, universal_newlines=None, \
                  startupinfo=None, creationflags=0, restore_signals=True, \
-                 start_new_session=False, pass_fds=(), *, group=None, \
-                 extra_groups=None, user=None, encoding=None, errors=None, \
-                 text=None)
+                 start_new_session=False, pass_fds=(), \*, group=None, \
+                 extra_groups=None, user=None, umask=-1, \
+                 encoding=None, errors=None, text=None)
 
    Execute a child program in a new process.  On POSIX, the class uses
    :meth:`os.execvp`-like behavior to execute the child program.  On Windows,
@@ -572,6 +572,12 @@ functions.
    .. availability:: POSIX
    .. versionadded:: 3.9
 
+   If *umask* is not negative, the umask() system call will be made in the
+   child process prior to the execution of the subprocess.
+
+   .. availability:: POSIX
+   .. versionadded:: 3.9
+
    If *env* is not ``None``, it must be a mapping that defines the environment
    variables for the new process; these are used instead of the default
    behavior of inheriting the current process' environment.
diff --git a/Lib/multiprocessing/util.py b/Lib/multiprocessing/util.py
index 207a2f70b2547..3e640b944e958 100644
--- a/Lib/multiprocessing/util.py
+++ b/Lib/multiprocessing/util.py
@@ -429,7 +429,7 @@ def spawnv_passfds(path, args, passfds):
         return _posixsubprocess.fork_exec(
             args, [os.fsencode(path)], True, passfds, None, None,
             -1, -1, -1, -1, -1, -1, errpipe_read, errpipe_write,
-            False, False, None, None, None, None)
+            False, False, None, None, None, -1, None)
     finally:
         os.close(errpipe_read)
         os.close(errpipe_write)
diff --git a/Lib/subprocess.py b/Lib/subprocess.py
index 1016874c4a7f4..b5a45d9fea23c 100644
--- a/Lib/subprocess.py
+++ b/Lib/subprocess.py
@@ -733,6 +733,8 @@ class Popen(object):
 
       user (POSIX only)
 
+      umask (POSIX only)
+
       pass_fds (POSIX only)
 
       encoding and errors: Text mode encoding and error handling to use for
@@ -750,7 +752,7 @@ def __init__(self, args, bufsize=-1, executable=None,
                  startupinfo=None, creationflags=0,
                  restore_signals=True, start_new_session=False,
                  pass_fds=(), *, user=None, group=None, extra_groups=None,
-                 encoding=None, errors=None, text=None):
+                 encoding=None, errors=None, text=None, umask=-1):
         """Create new Popen instance."""
         _cleanup()
         # Held while anything is calling waitpid before returncode has been
@@ -945,7 +947,7 @@ def __init__(self, args, bufsize=-1, executable=None,
                                 c2pread, c2pwrite,
                                 errread, errwrite,
                                 restore_signals,
-                                gid, gids, uid,
+                                gid, gids, uid, umask,
                                 start_new_session)
         except:
             # Cleanup if the child failed starting.
@@ -1318,6 +1320,7 @@ def _execute_child(self, args, executable, preexec_fn, close_fds,
                            errread, errwrite,
                            unused_restore_signals,
                            unused_gid, unused_gids, unused_uid,
+                           unused_umask,
                            unused_start_new_session):
             """Execute program (MS Windows version)"""
 
@@ -1645,7 +1648,7 @@ def _execute_child(self, args, executable, preexec_fn, close_fds,
                            c2pread, c2pwrite,
                            errread, errwrite,
                            restore_signals,
-                           gid, gids, uid,
+                           gid, gids, uid, umask,
                            start_new_session):
             """Execute program (POSIX version)"""
 
@@ -1684,7 +1687,8 @@ def _execute_child(self, args, executable, preexec_fn, close_fds,
                     and not start_new_session
                     and gid is None
                     and gids is None
-                    and uid is None):
+                    and uid is None
+                    and umask < 0):
                 self._posix_spawn(args, executable, env, restore_signals,
                                   p2cread, p2cwrite,
                                   c2pread, c2pwrite,
@@ -1738,7 +1742,7 @@ def _execute_child(self, args, executable, preexec_fn, close_fds,
                             errread, errwrite,
                             errpipe_read, errpipe_write,
                             restore_signals, start_new_session,
-                            gid, gids, uid,
+                            gid, gids, uid, umask,
                             preexec_fn)
                     self._child_created = True
                 finally:
diff --git a/Lib/test/test_capi.py b/Lib/test/test_capi.py
index 6731447af4a7e..2024435109793 100644
--- a/Lib/test/test_capi.py
+++ b/Lib/test/test_capi.py
@@ -97,7 +97,7 @@ class Z(object):
             def __len__(self):
                 return 1
         self.assertRaises(TypeError, _posixsubprocess.fork_exec,
-                          1,Z(),3,(1, 2),5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20)
+                          1,Z(),3,(1, 2),5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21)
         # Issue #15736: overflow in _PySequence_BytesToCharpArray()
         class Z(object):
             def __len__(self):
@@ -105,7 +105,7 @@ def __len__(self):
             def __getitem__(self, i):
                 return b'x'
         self.assertRaises(MemoryError, _posixsubprocess.fork_exec,
-                          1,Z(),3,(1, 2),5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20)
+                          1,Z(),3,(1, 2),5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21)
 
     @unittest.skipUnless(_posixsubprocess, '_posixsubprocess required for this test.')
     def test_subprocess_fork_exec(self):
@@ -115,7 +115,7 @@ def __len__(self):
 
         # Issue #15738: crash in subprocess_fork_exec()
         self.assertRaises(TypeError, _posixsubprocess.fork_exec,
-                          Z(),[b'1'],3,(1, 2),5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20)
+                          Z(),[b'1'],3,(1, 2),5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21)
 
     @unittest.skipIf(MISSING_C_DOCSTRINGS,
                      "Signature information for builtins requires docstrings")
diff --git a/Lib/test/test_subprocess.py b/Lib/test/test_subprocess.py
index 22516402da0e8..5cc324b87894c 100644
--- a/Lib/test/test_subprocess.py
+++ b/Lib/test/test_subprocess.py
@@ -1894,6 +1894,28 @@ def test_extra_groups_error(self):
         with self.assertRaises(ValueError):
             subprocess.check_call([sys.executable, "-c", "pass"], extra_groups=[])
 
+    @unittest.skipIf(mswindows or not hasattr(os, 'umask'),
+                     'POSIX umask() is not available.')
+    def test_umask(self):
+        tmpdir = None
+        try:
+            tmpdir = tempfile.mkdtemp()
+            name = os.path.join(tmpdir, "beans")
+            # We set an unusual umask in the child so as a unique mode
+            # for us to test the child's touched file for.
+            subprocess.check_call(
+                    [sys.executable, "-c", f"open({name!r}, 'w')"],  # touch
+                    umask=0o053)
+            # Ignore execute permissions entirely in our test,
+            # filesystems could be mounted to ignore or force that.
+            st_mode = os.stat(name).st_mode & 0o666
+            expected_mode = 0o624
+            self.assertEqual(expected_mode, st_mode,
+                             msg=f'{oct(expected_mode)} != {oct(st_mode)}')
+        finally:
+            if tmpdir is not None:
+                shutil.rmtree(tmpdir)
+
     def test_run_abort(self):
         # returncode handles signal termination
         with support.SuppressCrashReport():
@@ -2950,7 +2972,7 @@ def test_fork_exec(self):
                         -1, -1, -1, -1,
                         1, 2, 3, 4,
                         True, True,
-                        False, [], 0,
+                        False, [], 0, -1,
                         func)
                 # Attempt to prevent
                 # "TypeError: fork_exec() takes exactly N arguments (M given)"
@@ -2999,7 +3021,7 @@ def __int__(self):
                         -1, -1, -1, -1,
                         1, 2, 3, 4,
                         True, True,
-                        None, None, None,
+                        None, None, None, -1,
                         None)
                 self.assertIn('fds_to_keep', str(c.exception))
         finally:
diff --git a/Misc/NEWS.d/next/Library/2019-10-12-00-13-47.bpo-38417.W7x_aS.rst b/Misc/NEWS.d/next/Library/2019-10-12-00-13-47.bpo-38417.W7x_aS.rst
new file mode 100644
index 0000000000000..c2356ddbf140f
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-10-12-00-13-47.bpo-38417.W7x_aS.rst
@@ -0,0 +1,2 @@
+Added support for setting the umask in the child process to the subprocess
+module on POSIX systems.
diff --git a/Modules/_posixsubprocess.c b/Modules/_posixsubprocess.c
index 66db93e4500d0..80bb44dce9092 100644
--- a/Modules/_posixsubprocess.c
+++ b/Modules/_posixsubprocess.c
@@ -8,7 +8,7 @@
 #ifdef HAVE_SYS_TYPES_H
 #include <sys/types.h>
 #endif
-#if defined(HAVE_SYS_STAT_H) && defined(__FreeBSD__)
+#if defined(HAVE_SYS_STAT_H)
 #include <sys/stat.h>
 #endif
 #ifdef HAVE_SYS_SYSCALL_H
@@ -428,7 +428,7 @@ child_exec(char *const exec_array[],
            int call_setsid,
            int call_setgid, gid_t gid,
            int call_setgroups, size_t groups_size, const gid_t *groups,
-           int call_setuid, uid_t uid,
+           int call_setuid, uid_t uid, int child_umask,
            PyObject *py_fds_to_keep,
            PyObject *preexec_fn,
            PyObject *preexec_fn_args_tuple)
@@ -498,6 +498,9 @@ child_exec(char *const exec_array[],
     if (cwd)
         POSIX_CALL(chdir(cwd));
 
+    if (child_umask >= 0)
+        umask(child_umask);  /* umask() always succeeds. */
+
     if (restore_signals)
         _Py_RestoreSignals();
 
@@ -609,6 +612,7 @@ subprocess_fork_exec(PyObject* self, PyObject *args)
     int call_setgid = 0, call_setgroups = 0, call_setuid = 0;
     uid_t uid;
     gid_t gid, *groups = NULL;
+    int child_umask;
     PyObject *cwd_obj, *cwd_obj2;
     const char *cwd;
     pid_t pid;
@@ -619,14 +623,14 @@ subprocess_fork_exec(PyObject* self, PyObject *args)
     int saved_errno = 0;
 
     if (!PyArg_ParseTuple(
-            args, "OOpO!OOiiiiiiiiiiOOOO:fork_exec",
+            args, "OOpO!OOiiiiiiiiiiOOOiO:fork_exec",
             &process_args, &executable_list,
             &close_fds, &PyTuple_Type, &py_fds_to_keep,
             &cwd_obj, &env_list,
             &p2cread, &p2cwrite, &c2pread, &c2pwrite,
             &errread, &errwrite, &errpipe_read, &errpipe_write,
             &restore_signals, &call_setsid,
-            &gid_object, &groups_list, &uid_object,
+            &gid_object, &groups_list, &uid_object, &child_umask,
             &preexec_fn))
         return NULL;
 
@@ -843,7 +847,7 @@ subprocess_fork_exec(PyObject* self, PyObject *args)
                    errread, errwrite, errpipe_read, errpipe_write,
                    close_fds, restore_signals, call_setsid,
                    call_setgid, gid, call_setgroups, num_groups, groups,
-                   call_setuid, uid,
+                   call_setuid, uid, child_umask,
                    py_fds_to_keep, preexec_fn, preexec_fn_args_tuple);
         _exit(255);
         return NULL;  /* Dead code to avoid a potential compiler warning. */


From webhook-mailer at python.org  Sat Oct 12 19:35:57 2019
From: webhook-mailer at python.org (Gregory P. Smith)
Date: Sat, 12 Oct 2019 23:35:57 -0000
Subject: [Python-checkins] bpo-38456: Use /bin/true in test_subprocess
 (GH-16736)
Message-ID: <mailman.126.1570923358.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/67b93f80c764bca01c81c989d74a99df208bea4d
commit: 67b93f80c764bca01c81c989d74a99df208bea4d
branch: master
author: Gregory P. Smith <greg at krypto.org>
committer: GitHub <noreply at github.com>
date: 2019-10-12T16:35:53-07:00
summary:

bpo-38456: Use /bin/true in test_subprocess (GH-16736)

* bpo-38456: Use /bin/true in test_subprocess.

Instead of sys.executable, "-c", "pass" or "import sys; sys.exit(0)"
use /bin/true when it is available.  On a reasonable machine this
shaves up to two seconds wall time off the otherwise ~40sec execution
on a --with-pydebug build.  It should be more notable on many
buildbots or overloaded slower I/O systems (CI, etc).

files:
M Lib/test/test_subprocess.py

diff --git a/Lib/test/test_subprocess.py b/Lib/test/test_subprocess.py
index 5cc324b87894c..2231ff49235c8 100644
--- a/Lib/test/test_subprocess.py
+++ b/Lib/test/test_subprocess.py
@@ -54,6 +54,16 @@
 # Ignore errors that indicate the command was not found
 NONEXISTING_ERRORS = (FileNotFoundError, NotADirectoryError, PermissionError)
 
+ZERO_RETURN_CMD = (sys.executable, '-c', 'pass')
+
+
+def setUpModule():
+    shell_true = shutil.which('true')
+    if (os.access(shell_true, os.X_OK) and
+        subprocess.run([shell_true]).returncode == 0):
+        global ZERO_RETURN_CMD
+        ZERO_RETURN_CMD = (shell_true,)  # Faster than Python startup.
+
 
 class BaseTestCase(unittest.TestCase):
     def setUp(self):
@@ -98,7 +108,7 @@ def _execute_child(self, *args, **kwargs):
 class ProcessTestCase(BaseTestCase):
 
     def test_io_buffered_by_default(self):
-        p = subprocess.Popen([sys.executable, "-c", "import sys; sys.exit(0)"],
+        p = subprocess.Popen(ZERO_RETURN_CMD,
                              stdin=subprocess.PIPE, stdout=subprocess.PIPE,
                              stderr=subprocess.PIPE)
         try:
@@ -112,7 +122,7 @@ def test_io_buffered_by_default(self):
             p.wait()
 
     def test_io_unbuffered_works(self):
-        p = subprocess.Popen([sys.executable, "-c", "import sys; sys.exit(0)"],
+        p = subprocess.Popen(ZERO_RETURN_CMD,
                              stdin=subprocess.PIPE, stdout=subprocess.PIPE,
                              stderr=subprocess.PIPE, bufsize=0)
         try:
@@ -142,8 +152,7 @@ def test_call_timeout(self):
 
     def test_check_call_zero(self):
         # check_call() function with zero return code
-        rc = subprocess.check_call([sys.executable, "-c",
-                                    "import sys; sys.exit(0)"])
+        rc = subprocess.check_call(ZERO_RETURN_CMD)
         self.assertEqual(rc, 0)
 
     def test_check_call_nonzero(self):
@@ -709,19 +718,19 @@ def test_invalid_env(self):
         newenv = os.environ.copy()
         newenv["FRUIT\0VEGETABLE"] = "cabbage"
         with self.assertRaises(ValueError):
-            subprocess.Popen([sys.executable, "-c", "pass"], env=newenv)
+            subprocess.Popen(ZERO_RETURN_CMD, env=newenv)
 
         # null character in the environment variable value
         newenv = os.environ.copy()
         newenv["FRUIT"] = "orange\0VEGETABLE=cabbage"
         with self.assertRaises(ValueError):
-            subprocess.Popen([sys.executable, "-c", "pass"], env=newenv)
+            subprocess.Popen(ZERO_RETURN_CMD, env=newenv)
 
         # equal character in the environment variable name
         newenv = os.environ.copy()
         newenv["FRUIT=ORANGE"] = "lemon"
         with self.assertRaises(ValueError):
-            subprocess.Popen([sys.executable, "-c", "pass"], env=newenv)
+            subprocess.Popen(ZERO_RETURN_CMD, env=newenv)
 
         # equal character in the environment variable value
         newenv = os.environ.copy()
@@ -822,7 +831,7 @@ def test_communicate_pipe_fd_leak(self):
                         options['stderr'] = subprocess.PIPE
                     if not options:
                         continue
-                    p = subprocess.Popen((sys.executable, "-c", "pass"), **options)
+                    p = subprocess.Popen(ZERO_RETURN_CMD, **options)
                     p.communicate()
                     if p.stdin is not None:
                         self.assertTrue(p.stdin.closed)
@@ -961,7 +970,7 @@ def test_universal_newlines_communicate_input_none(self):
         #
         # We set stdout to PIPE because, as of this writing, a different
         # code path is tested when the number of pipes is zero or one.
-        p = subprocess.Popen([sys.executable, "-c", "pass"],
+        p = subprocess.Popen(ZERO_RETURN_CMD,
                              stdin=subprocess.PIPE,
                              stdout=subprocess.PIPE,
                              universal_newlines=True)
@@ -1109,7 +1118,7 @@ def test_poll(self):
         self.assertEqual(p.poll(), 0)
 
     def test_wait(self):
-        p = subprocess.Popen([sys.executable, "-c", "pass"])
+        p = subprocess.Popen(ZERO_RETURN_CMD)
         self.assertEqual(p.wait(), 0)
         # Subsequent invocations should just return the returncode
         self.assertEqual(p.wait(), 0)
@@ -1128,14 +1137,14 @@ def test_invalid_bufsize(self):
         # an invalid type of the bufsize argument should raise
         # TypeError.
         with self.assertRaises(TypeError):
-            subprocess.Popen([sys.executable, "-c", "pass"], "orange")
+            subprocess.Popen(ZERO_RETURN_CMD, "orange")
 
     def test_bufsize_is_none(self):
         # bufsize=None should be the same as bufsize=0.
-        p = subprocess.Popen([sys.executable, "-c", "pass"], None)
+        p = subprocess.Popen(ZERO_RETURN_CMD, None)
         self.assertEqual(p.wait(), 0)
         # Again with keyword arg
-        p = subprocess.Popen([sys.executable, "-c", "pass"], bufsize=None)
+        p = subprocess.Popen(ZERO_RETURN_CMD, bufsize=None)
         self.assertEqual(p.wait(), 0)
 
     def _test_bufsize_equal_one(self, line, expected, universal_newlines):
@@ -1340,7 +1349,7 @@ def test_handles_closed_on_exception(self):
 
     def test_communicate_epipe(self):
         # Issue 10963: communicate() should hide EPIPE
-        p = subprocess.Popen([sys.executable, "-c", 'pass'],
+        p = subprocess.Popen(ZERO_RETURN_CMD,
                              stdin=subprocess.PIPE,
                              stdout=subprocess.PIPE,
                              stderr=subprocess.PIPE)
@@ -1351,7 +1360,7 @@ def test_communicate_epipe(self):
 
     def test_communicate_epipe_only_stdin(self):
         # Issue 10963: communicate() should hide EPIPE
-        p = subprocess.Popen([sys.executable, "-c", 'pass'],
+        p = subprocess.Popen(ZERO_RETURN_CMD,
                              stdin=subprocess.PIPE)
         self.addCleanup(p.stdin.close)
         p.wait()
@@ -1390,7 +1399,7 @@ def test_failed_child_execute_fd_leak(self):
         fds_before_popen = os.listdir(fd_directory)
         with self.assertRaises(PopenTestException):
             PopenExecuteChildRaises(
-                    [sys.executable, '-c', 'pass'], stdin=subprocess.PIPE,
+                    ZERO_RETURN_CMD, stdin=subprocess.PIPE,
                     stdout=subprocess.PIPE, stderr=subprocess.PIPE)
 
         # NOTE: This test doesn't verify that the real _execute_child
@@ -1433,7 +1442,7 @@ def test_check(self):
 
     def test_check_zero(self):
         # check_returncode shouldn't raise when returncode is zero
-        cp = self.run_python("import sys; sys.exit(0)", check=True)
+        cp = subprocess.run(ZERO_RETURN_CMD, check=True)
         self.assertEqual(cp.returncode, 0)
 
     def test_timeout(self):
@@ -1791,16 +1800,16 @@ def test_user(self):
                         self.assertEqual(child_user, user_uid)
 
         with self.assertRaises(ValueError):
-            subprocess.check_call([sys.executable, "-c", "pass"], user=-1)
+            subprocess.check_call(ZERO_RETURN_CMD, user=-1)
 
         if pwd is None:
             with self.assertRaises(ValueError):
-                subprocess.check_call([sys.executable, "-c", "pass"], user=name_uid)
+                subprocess.check_call(ZERO_RETURN_CMD, user=name_uid)
 
     @unittest.skipIf(hasattr(os, 'setreuid'), 'setreuid() available on platform')
     def test_user_error(self):
         with self.assertRaises(ValueError):
-            subprocess.check_call([sys.executable, "-c", "pass"], user=65535)
+            subprocess.check_call(ZERO_RETURN_CMD, user=65535)
 
     @unittest.skipUnless(hasattr(os, 'setregid'), 'no setregid() on platform')
     def test_group(self):
@@ -1834,16 +1843,16 @@ def test_group(self):
 
         # make sure we bomb on negative values
         with self.assertRaises(ValueError):
-            subprocess.check_call([sys.executable, "-c", "pass"], group=-1)
+            subprocess.check_call(ZERO_RETURN_CMD, group=-1)
 
         if grp is None:
             with self.assertRaises(ValueError):
-                subprocess.check_call([sys.executable, "-c", "pass"], group=name_group)
+                subprocess.check_call(ZERO_RETURN_CMD, group=name_group)
 
     @unittest.skipIf(hasattr(os, 'setregid'), 'setregid() available on platform')
     def test_group_error(self):
         with self.assertRaises(ValueError):
-            subprocess.check_call([sys.executable, "-c", "pass"], group=65535)
+            subprocess.check_call(ZERO_RETURN_CMD, group=65535)
 
     @unittest.skipUnless(hasattr(os, 'setgroups'), 'no setgroups() on platform')
     def test_extra_groups(self):
@@ -1882,17 +1891,17 @@ def test_extra_groups(self):
 
         # make sure we bomb on negative values
         with self.assertRaises(ValueError):
-            subprocess.check_call([sys.executable, "-c", "pass"], extra_groups=[-1])
+            subprocess.check_call(ZERO_RETURN_CMD, extra_groups=[-1])
 
         if grp is None:
             with self.assertRaises(ValueError):
-                subprocess.check_call([sys.executable, "-c", "pass"],
+                subprocess.check_call(ZERO_RETURN_CMD,
                                       extra_groups=[name_group])
 
     @unittest.skipIf(hasattr(os, 'setgroups'), 'setgroups() available on platform')
     def test_extra_groups_error(self):
         with self.assertRaises(ValueError):
-            subprocess.check_call([sys.executable, "-c", "pass"], extra_groups=[])
+            subprocess.check_call(ZERO_RETURN_CMD, extra_groups=[])
 
     @unittest.skipIf(mswindows or not hasattr(os, 'umask'),
                      'POSIX umask() is not available.')
@@ -1904,7 +1913,7 @@ def test_umask(self):
             # We set an unusual umask in the child so as a unique mode
             # for us to test the child's touched file for.
             subprocess.check_call(
-                    [sys.executable, "-c", f"open({name!r}, 'w')"],  # touch
+                    [sys.executable, "-c", f"open({name!r}, 'w').close()"],
                     umask=0o053)
             # Ignore execute permissions entirely in our test,
             # filesystems could be mounted to ignore or force that.
@@ -2007,7 +2016,7 @@ def raise_it():
 
         with self.assertRaises(subprocess.SubprocessError):
             self._TestExecuteChildPopen(
-                        self, [sys.executable, "-c", "pass"],
+                        self, ZERO_RETURN_CMD,
                         stdin=subprocess.PIPE, stdout=subprocess.PIPE,
                         stderr=subprocess.PIPE, preexec_fn=raise_it)
 
@@ -2464,7 +2473,7 @@ def prepare():
 
         try:
             subprocess.call(
-                [sys.executable, "-c", "pass"],
+                ZERO_RETURN_CMD,
                 preexec_fn=prepare)
         except ValueError as err:
             # Pure Python implementations keeps the message
@@ -2507,29 +2516,30 @@ def test_undecodable_env(self):
             self.assertEqual(stdout.decode('ascii'), ascii(encoded_value))
 
     def test_bytes_program(self):
-        abs_program = os.fsencode(sys.executable)
-        path, program = os.path.split(sys.executable)
+        abs_program = os.fsencode(ZERO_RETURN_CMD[0])
+        args = list(ZERO_RETURN_CMD[1:])
+        path, program = os.path.split(ZERO_RETURN_CMD[0])
         program = os.fsencode(program)
 
         # absolute bytes path
-        exitcode = subprocess.call([abs_program, "-c", "pass"])
+        exitcode = subprocess.call([abs_program]+args)
         self.assertEqual(exitcode, 0)
 
         # absolute bytes path as a string
-        cmd = b"'" + abs_program + b"' -c pass"
+        cmd = b"'%s' %s" % (abs_program, " ".join(args).encode("utf-8"))
         exitcode = subprocess.call(cmd, shell=True)
         self.assertEqual(exitcode, 0)
 
         # bytes program, unicode PATH
         env = os.environ.copy()
         env["PATH"] = path
-        exitcode = subprocess.call([program, "-c", "pass"], env=env)
+        exitcode = subprocess.call([program]+args, env=env)
         self.assertEqual(exitcode, 0)
 
         # bytes program, bytes PATH
         envb = os.environb.copy()
         envb[b"PATH"] = os.fsencode(path)
-        exitcode = subprocess.call([program, "-c", "pass"], env=envb)
+        exitcode = subprocess.call([program]+args, env=envb)
         self.assertEqual(exitcode, 0)
 
     def test_pipe_cloexec(self):
@@ -2757,7 +2767,7 @@ def test_pass_fds(self):
             # pass_fds overrides close_fds with a warning.
             with self.assertWarns(RuntimeWarning) as context:
                 self.assertFalse(subprocess.call(
-                        [sys.executable, "-c", "import sys; sys.exit(0)"],
+                        ZERO_RETURN_CMD,
                         close_fds=False, pass_fds=(fd, )))
             self.assertIn('overriding close_fds', str(context.warning))
 
@@ -2819,19 +2829,19 @@ def test_pass_fds_redirected(self):
 
     def test_stdout_stdin_are_single_inout_fd(self):
         with io.open(os.devnull, "r+") as inout:
-            p = subprocess.Popen([sys.executable, "-c", "import sys; sys.exit(0)"],
+            p = subprocess.Popen(ZERO_RETURN_CMD,
                                  stdout=inout, stdin=inout)
             p.wait()
 
     def test_stdout_stderr_are_single_inout_fd(self):
         with io.open(os.devnull, "r+") as inout:
-            p = subprocess.Popen([sys.executable, "-c", "import sys; sys.exit(0)"],
+            p = subprocess.Popen(ZERO_RETURN_CMD,
                                  stdout=inout, stderr=inout)
             p.wait()
 
     def test_stderr_stdin_are_single_inout_fd(self):
         with io.open(os.devnull, "r+") as inout:
-            p = subprocess.Popen([sys.executable, "-c", "import sys; sys.exit(0)"],
+            p = subprocess.Popen(ZERO_RETURN_CMD,
                                  stderr=inout, stdin=inout)
             p.wait()
 
@@ -3031,7 +3041,7 @@ def __int__(self):
     def test_communicate_BrokenPipeError_stdin_close(self):
         # By not setting stdout or stderr or a timeout we force the fast path
         # that just calls _stdin_write() internally due to our mock.
-        proc = subprocess.Popen([sys.executable, '-c', 'pass'])
+        proc = subprocess.Popen(ZERO_RETURN_CMD)
         with proc, mock.patch.object(proc, 'stdin') as mock_proc_stdin:
             mock_proc_stdin.close.side_effect = BrokenPipeError
             proc.communicate()  # Should swallow BrokenPipeError from close.
@@ -3040,7 +3050,7 @@ def test_communicate_BrokenPipeError_stdin_close(self):
     def test_communicate_BrokenPipeError_stdin_write(self):
         # By not setting stdout or stderr or a timeout we force the fast path
         # that just calls _stdin_write() internally due to our mock.
-        proc = subprocess.Popen([sys.executable, '-c', 'pass'])
+        proc = subprocess.Popen(ZERO_RETURN_CMD)
         with proc, mock.patch.object(proc, 'stdin') as mock_proc_stdin:
             mock_proc_stdin.write.side_effect = BrokenPipeError
             proc.communicate(b'stuff')  # Should swallow the BrokenPipeError.
@@ -3079,7 +3089,7 @@ def test_communicate_BrokenPipeError_stdin_close_with_timeout(self):
                          'need _testcapi.W_STOPCODE')
     def test_stopped(self):
         """Test wait() behavior when waitpid returns WIFSTOPPED; issue29335."""
-        args = [sys.executable, '-c', 'pass']
+        args = ZERO_RETURN_CMD
         proc = subprocess.Popen(args)
 
         # Wait until the real process completes to avoid zombie process
@@ -3109,7 +3119,7 @@ def test_startupinfo(self):
         # Since Python is a console process, it won't be affected
         # by wShowWindow, but the argument should be silently
         # ignored
-        subprocess.call([sys.executable, "-c", "import sys; sys.exit(0)"],
+        subprocess.call(ZERO_RETURN_CMD,
                         startupinfo=startupinfo)
 
     def test_startupinfo_keywords(self):
@@ -3125,7 +3135,7 @@ def test_startupinfo_keywords(self):
         # Since Python is a console process, it won't be affected
         # by wShowWindow, but the argument should be silently
         # ignored
-        subprocess.call([sys.executable, "-c", "import sys; sys.exit(0)"],
+        subprocess.call(ZERO_RETURN_CMD,
                         startupinfo=startupinfo)
 
     def test_startupinfo_copy(self):
@@ -3137,7 +3147,7 @@ def test_startupinfo_copy(self):
         # Call Popen() twice with the same startupinfo object to make sure
         # that it's not modified
         for _ in range(2):
-            cmd = [sys.executable, "-c", "pass"]
+            cmd = ZERO_RETURN_CMD
             with open(os.devnull, 'w') as null:
                 proc = subprocess.Popen(cmd,
                                         stdout=null,
@@ -3177,7 +3187,7 @@ def test_issue31471(self):
         class BadEnv(dict):
             keys = None
         with self.assertRaises(TypeError):
-            subprocess.Popen([sys.executable, "-c", "pass"], env=BadEnv())
+            subprocess.Popen(ZERO_RETURN_CMD, env=BadEnv())
 
     def test_close_fds(self):
         # close file descriptors
@@ -3238,13 +3248,13 @@ def test_close_fds_with_stdio(self):
     def test_empty_attribute_list(self):
         startupinfo = subprocess.STARTUPINFO()
         startupinfo.lpAttributeList = {}
-        subprocess.call([sys.executable, "-c", "import sys; sys.exit(0)"],
+        subprocess.call(ZERO_RETURN_CMD,
                         startupinfo=startupinfo)
 
     def test_empty_handle_list(self):
         startupinfo = subprocess.STARTUPINFO()
         startupinfo.lpAttributeList = {"handle_list": []}
-        subprocess.call([sys.executable, "-c", "import sys; sys.exit(0)"],
+        subprocess.call(ZERO_RETURN_CMD,
                         startupinfo=startupinfo)
 
     def test_shell_sequence(self):
@@ -3543,7 +3553,7 @@ def test_invalid_args(self):
 
     def test_broken_pipe_cleanup(self):
         """Broken pipe error should not prevent wait() (Issue 21619)"""
-        proc = subprocess.Popen([sys.executable, '-c', 'pass'],
+        proc = subprocess.Popen(ZERO_RETURN_CMD,
                                 stdin=subprocess.PIPE,
                                 bufsize=support.PIPE_MAX_SIZE*2)
         proc = proc.__enter__()


From webhook-mailer at python.org  Sat Oct 12 21:03:58 2019
From: webhook-mailer at python.org (Pablo Galindo)
Date: Sun, 13 Oct 2019 01:03:58 -0000
Subject: [Python-checkins] bpo-38282: Correctly manage the Bluetooth L2CAP
 socket structure in FreeBSD (GH-16738)
Message-ID: <mailman.127.1570928639.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/27b33fb41a7c64a6211d73d14804aa0cd6defccb
commit: 27b33fb41a7c64a6211d73d14804aa0cd6defccb
branch: master
author: Pablo Galindo <Pablogsal at gmail.com>
committer: GitHub <noreply at github.com>
date: 2019-10-13T02:03:54+01:00
summary:

bpo-38282: Correctly manage the Bluetooth L2CAP socket structure in FreeBSD (GH-16738)

files:
M Modules/socketmodule.h

diff --git a/Modules/socketmodule.h b/Modules/socketmodule.h
index e06e4c388827e..7684e59cd4546 100644
--- a/Modules/socketmodule.h
+++ b/Modules/socketmodule.h
@@ -235,7 +235,12 @@ typedef union sock_addr {
     struct sockaddr_in6 in6;
     struct sockaddr_storage storage;
 #endif
-#ifdef HAVE_BLUETOOTH_BLUETOOTH_H
+#if defined(HAVE_BLUETOOTH_H) && defined(__FreeBSD__)
+    struct sockaddr_l2cap bt_l2;
+    struct sockaddr_rfcomm bt_rc;
+    struct sockaddr_sco bt_sco;
+    struct sockaddr_hci bt_hci;
+#elif defined(HAVE_BLUETOOTH_BLUETOOTH_H)
     struct sockaddr_l2 bt_l2;
     struct sockaddr_rc bt_rc;
     struct sockaddr_sco bt_sco;


From webhook-mailer at python.org  Sat Oct 12 21:40:28 2019
From: webhook-mailer at python.org (Pablo Galindo)
Date: Sun, 13 Oct 2019 01:40:28 -0000
Subject: [Python-checkins] bpo-38456: Handle the case when there is no
 'true' command (GH-16739)
Message-ID: <mailman.128.1570930829.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/46113e0cf32748f66cf64cd633984d143b433cd1
commit: 46113e0cf32748f66cf64cd633984d143b433cd1
branch: master
author: Pablo Galindo <Pablogsal at gmail.com>
committer: GitHub <noreply at github.com>
date: 2019-10-13T02:40:24+01:00
summary:

bpo-38456: Handle the case when there is no 'true' command (GH-16739)

files:
M Lib/test/test_subprocess.py

diff --git a/Lib/test/test_subprocess.py b/Lib/test/test_subprocess.py
index 2231ff49235c8..9e96a6d9a8747 100644
--- a/Lib/test/test_subprocess.py
+++ b/Lib/test/test_subprocess.py
@@ -59,6 +59,8 @@
 
 def setUpModule():
     shell_true = shutil.which('true')
+    if shell_true is None:
+        return
     if (os.access(shell_true, os.X_OK) and
         subprocess.run([shell_true]).returncode == 0):
         global ZERO_RETURN_CMD


From webhook-mailer at python.org  Sun Oct 13 04:59:37 2019
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Sun, 13 Oct 2019 08:59:37 -0000
Subject: [Python-checkins] bpo-38378: Rename parameters "out" and "in" of
 os.sendfile(). (GH-16742)
Message-ID: <mailman.129.1570957178.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/140a7d1f3579e778656a6b6bfad72489e9870a4d
commit: 140a7d1f3579e778656a6b6bfad72489e9870a4d
branch: master
author: Serhiy Storchaka <storchaka at gmail.com>
committer: GitHub <noreply at github.com>
date: 2019-10-13T11:59:31+03:00
summary:

bpo-38378: Rename parameters "out" and "in" of os.sendfile(). (GH-16742)

They conflicted with keyword "in".

Also rename positional-only parameters of private os._fcopyfile()
for consistency.

files:
A Misc/NEWS.d/next/Library/2019-10-13-11-00-03.bpo-38378.yYNpSm.rst
M Doc/library/os.rst
M Lib/test/test_os.py
M Modules/clinic/posixmodule.c.h
M Modules/posixmodule.c

diff --git a/Doc/library/os.rst b/Doc/library/os.rst
index 67fe36b54e2f8..fe9531b170177 100644
--- a/Doc/library/os.rst
+++ b/Doc/library/os.rst
@@ -1251,27 +1251,27 @@ or `the MSDN <https://msdn.microsoft.com/en-us/library/z0kc8e3z.aspx>`_ on Windo
       :exc:`InterruptedError` exception (see :pep:`475` for the rationale).
 
 
-.. function:: sendfile(out, in, offset, count)
-              sendfile(out, in, offset, count, [headers], [trailers], flags=0)
+.. function:: sendfile(out_fd, in_fd, offset, count)
+              sendfile(out_fd, in_fd, offset, count, [headers], [trailers], flags=0)
 
-   Copy *count* bytes from file descriptor *in* to file descriptor *out*
+   Copy *count* bytes from file descriptor *in_fd* to file descriptor *out_fd*
    starting at *offset*.
-   Return the number of bytes sent. When EOF is reached return 0.
+   Return the number of bytes sent. When EOF is reached return ``0``.
 
    The first function notation is supported by all platforms that define
    :func:`sendfile`.
 
    On Linux, if *offset* is given as ``None``, the bytes are read from the
-   current position of *in* and the position of *in* is updated.
+   current position of *in_fd* and the position of *in_fd* is updated.
 
    The second case may be used on Mac OS X and FreeBSD where *headers* and
    *trailers* are arbitrary sequences of buffers that are written before and
-   after the data from *in* is written. It returns the same as the first case.
+   after the data from *in_fd* is written. It returns the same as the first case.
 
-   On Mac OS X and FreeBSD, a value of 0 for *count* specifies to send until
-   the end of *in* is reached.
+   On Mac OS X and FreeBSD, a value of ``0`` for *count* specifies to send until
+   the end of *in_fd* is reached.
 
-   All platforms support sockets as *out* file descriptor, and some platforms
+   All platforms support sockets as *out_fd* file descriptor, and some platforms
    allow other types (e.g. regular file, pipe) as well.
 
    Cross-platform applications should not use *headers*, *trailers* and *flags*
@@ -1286,6 +1286,9 @@ or `the MSDN <https://msdn.microsoft.com/en-us/library/z0kc8e3z.aspx>`_ on Windo
 
    .. versionadded:: 3.3
 
+   .. versionchanged:: 3.9
+      Parameters *out* and *in* was renamed to *out_fd* and *in_fd*.
+
 
 .. function:: set_blocking(fd, blocking)
 
diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py
index 4a076e3bbf542..d2bd9c2f232f1 100644
--- a/Lib/test/test_os.py
+++ b/Lib/test/test_os.py
@@ -3110,11 +3110,12 @@ def test_invalid_offset(self):
 
     def test_keywords(self):
         # Keyword arguments should be supported
-        os.sendfile(out=self.sockno, offset=0, count=4096,
-            **{'in': self.fileno})
+        os.sendfile(out_fd=self.sockno, in_fd=self.fileno,
+                    offset=0, count=4096)
         if self.SUPPORT_HEADERS_TRAILERS:
-            os.sendfile(self.sockno, self.fileno, offset=0, count=4096,
-                headers=(), trailers=(), flags=0)
+            os.sendfile(out_fd=self.sockno, in_fd=self.fileno,
+                        offset=0, count=4096,
+                        headers=(), trailers=(), flags=0)
 
     # --- headers / trailers tests
 
diff --git a/Misc/NEWS.d/next/Library/2019-10-13-11-00-03.bpo-38378.yYNpSm.rst b/Misc/NEWS.d/next/Library/2019-10-13-11-00-03.bpo-38378.yYNpSm.rst
new file mode 100644
index 0000000000000..1bc52825ee9b5
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-10-13-11-00-03.bpo-38378.yYNpSm.rst
@@ -0,0 +1,2 @@
+Parameters *out* and *in* of :func:`os.sendfile` was renamed to *out_fd* and
+*in_fd*.
diff --git a/Modules/clinic/posixmodule.c.h b/Modules/clinic/posixmodule.c.h
index 00b05f7989a94..561cbb0ca82ab 100644
--- a/Modules/clinic/posixmodule.c.h
+++ b/Modules/clinic/posixmodule.c.h
@@ -4986,7 +4986,7 @@ os_write(PyObject *module, PyObject *const *args, Py_ssize_t nargs)
 #if defined(__APPLE__)
 
 PyDoc_STRVAR(os__fcopyfile__doc__,
-"_fcopyfile($module, infd, outfd, flags, /)\n"
+"_fcopyfile($module, in_fd, out_fd, flags, /)\n"
 "--\n"
 "\n"
 "Efficiently copy content or metadata of 2 regular file descriptors (macOS).");
@@ -4995,14 +4995,14 @@ PyDoc_STRVAR(os__fcopyfile__doc__,
     {"_fcopyfile", (PyCFunction)(void(*)(void))os__fcopyfile, METH_FASTCALL, os__fcopyfile__doc__},
 
 static PyObject *
-os__fcopyfile_impl(PyObject *module, int infd, int outfd, int flags);
+os__fcopyfile_impl(PyObject *module, int in_fd, int out_fd, int flags);
 
 static PyObject *
 os__fcopyfile(PyObject *module, PyObject *const *args, Py_ssize_t nargs)
 {
     PyObject *return_value = NULL;
-    int infd;
-    int outfd;
+    int in_fd;
+    int out_fd;
     int flags;
 
     if (!_PyArg_CheckPositional("_fcopyfile", nargs, 3, 3)) {
@@ -5013,8 +5013,8 @@ os__fcopyfile(PyObject *module, PyObject *const *args, Py_ssize_t nargs)
                         "integer argument expected, got float" );
         goto exit;
     }
-    infd = _PyLong_AsInt(args[0]);
-    if (infd == -1 && PyErr_Occurred()) {
+    in_fd = _PyLong_AsInt(args[0]);
+    if (in_fd == -1 && PyErr_Occurred()) {
         goto exit;
     }
     if (PyFloat_Check(args[1])) {
@@ -5022,8 +5022,8 @@ os__fcopyfile(PyObject *module, PyObject *const *args, Py_ssize_t nargs)
                         "integer argument expected, got float" );
         goto exit;
     }
-    outfd = _PyLong_AsInt(args[1]);
-    if (outfd == -1 && PyErr_Occurred()) {
+    out_fd = _PyLong_AsInt(args[1]);
+    if (out_fd == -1 && PyErr_Occurred()) {
         goto exit;
     }
     if (PyFloat_Check(args[2])) {
@@ -5035,7 +5035,7 @@ os__fcopyfile(PyObject *module, PyObject *const *args, Py_ssize_t nargs)
     if (flags == -1 && PyErr_Occurred()) {
         goto exit;
     }
-    return_value = os__fcopyfile_impl(module, infd, outfd, flags);
+    return_value = os__fcopyfile_impl(module, in_fd, out_fd, flags);
 
 exit:
     return return_value;
@@ -8731,4 +8731,4 @@ os__remove_dll_directory(PyObject *module, PyObject *const *args, Py_ssize_t nar
 #ifndef OS__REMOVE_DLL_DIRECTORY_METHODDEF
     #define OS__REMOVE_DLL_DIRECTORY_METHODDEF
 #endif /* !defined(OS__REMOVE_DLL_DIRECTORY_METHODDEF) */
-/*[clinic end generated code: output=799c75140d84ace5 input=a9049054013a1b77]*/
+/*[clinic end generated code: output=fe7897441fed5402 input=a9049054013a1b77]*/
diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
index aeb0e9ddb1953..3c4e254abb56d 100644
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -8993,10 +8993,10 @@ os_write_impl(PyObject *module, int fd, Py_buffer *data)
 
 #ifdef HAVE_SENDFILE
 PyDoc_STRVAR(posix_sendfile__doc__,
-"sendfile(out, in, offset, count) -> byteswritten\n\
-sendfile(out, in, offset, count[, headers][, trailers], flags=0)\n\
+"sendfile(out_fd, in_fd, offset, count) -> byteswritten\n\
+sendfile(out_fd, in_fd, offset, count[, headers][, trailers], flags=0)\n\
             -> byteswritten\n\
-Copy count bytes from file descriptor in to file descriptor out.");
+Copy count bytes from file descriptor in_fd to file descriptor out_fd.");
 
 /* AC 3.5: don't bother converting, has optional group*/
 static PyObject *
@@ -9016,8 +9016,7 @@ posix_sendfile(PyObject *self, PyObject *args, PyObject *kwdict)
     off_t sbytes;
     struct sf_hdtr sf;
     int flags = 0;
-    /* Beware that "in" clashes with Python's own "in" operator keyword */
-    static char *keywords[] = {"out", "in",
+    static char *keywords[] = {"out_fd", "in_fd",
                                 "offset", "count",
                                 "headers", "trailers", "flags", NULL};
 
@@ -9133,7 +9132,7 @@ posix_sendfile(PyObject *self, PyObject *args, PyObject *kwdict)
 #else
     Py_ssize_t count;
     PyObject *offobj;
-    static char *keywords[] = {"out", "in",
+    static char *keywords[] = {"out_fd", "in_fd",
                                 "offset", "count", NULL};
     if (!PyArg_ParseTupleAndKeywords(args, kwdict, "iiOn:sendfile",
             keywords, &out, &in, &offobj, &count))
@@ -9170,8 +9169,8 @@ posix_sendfile(PyObject *self, PyObject *args, PyObject *kwdict)
 /*[clinic input]
 os._fcopyfile
 
-    infd: int
-    outfd: int
+    in_fd: int
+    out_fd: int
     flags: int
     /
 
@@ -9179,13 +9178,13 @@ Efficiently copy content or metadata of 2 regular file descriptors (macOS).
 [clinic start generated code]*/
 
 static PyObject *
-os__fcopyfile_impl(PyObject *module, int infd, int outfd, int flags)
-/*[clinic end generated code: output=8e8885c721ec38e3 input=69e0770e600cb44f]*/
+os__fcopyfile_impl(PyObject *module, int in_fd, int out_fd, int flags)
+/*[clinic end generated code: output=c9d1a35a992e401b input=1e34638a86948795]*/
 {
     int ret;
 
     Py_BEGIN_ALLOW_THREADS
-    ret = fcopyfile(infd, outfd, NULL, flags);
+    ret = fcopyfile(in_fd, out_fd, NULL, flags);
     Py_END_ALLOW_THREADS
     if (ret < 0)
         return posix_error();


From webhook-mailer at python.org  Sun Oct 13 07:45:40 2019
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Sun, 13 Oct 2019 11:45:40 -0000
Subject: [Python-checkins] bpo-38431: Fix __repr__ method of InitVar to work
 with typing objects. (GH-16702)
Message-ID: <mailman.130.1570967140.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/793cb85437299a3da3d74fe65480d720af330cbb
commit: 793cb85437299a3da3d74fe65480d720af330cbb
branch: master
author: Samuel Colvin <samcolvin at gmail.com>
committer: Serhiy Storchaka <storchaka at gmail.com>
date: 2019-10-13T14:45:36+03:00
summary:

bpo-38431: Fix __repr__ method of InitVar to work with typing objects. (GH-16702)

files:
A Misc/NEWS.d/next/Library/2019-10-10-16-53-00.bpo-38431.d5wzNp.rst
M Lib/dataclasses.py
M Lib/test/test_dataclasses.py

diff --git a/Lib/dataclasses.py b/Lib/dataclasses.py
index 9135b07c9f259..91c1f6f80fc68 100644
--- a/Lib/dataclasses.py
+++ b/Lib/dataclasses.py
@@ -206,7 +206,12 @@ def __init__(self, type):
         self.type = type
 
     def __repr__(self):
-        return f'dataclasses.InitVar[{self.type.__name__}]'
+        if isinstance(self.type, type):
+            type_name = self.type.__name__
+        else:
+            # typing objects, e.g. List[int]
+            type_name = repr(self.type)
+        return f'dataclasses.InitVar[{type_name}]'
 
     def __class_getitem__(cls, type):
         return InitVar(type)
diff --git a/Lib/test/test_dataclasses.py b/Lib/test/test_dataclasses.py
index 037bf4c221427..238335e7d9eda 100644
--- a/Lib/test/test_dataclasses.py
+++ b/Lib/test/test_dataclasses.py
@@ -1102,6 +1102,8 @@ def test_init_var_preserve_type(self):
 
         # Make sure the repr is correct.
         self.assertEqual(repr(InitVar[int]), 'dataclasses.InitVar[int]')
+        self.assertEqual(repr(InitVar[List[int]]),
+                         'dataclasses.InitVar[typing.List[int]]')
 
     def test_init_var_inheritance(self):
         # Note that this deliberately tests that a dataclass need not
diff --git a/Misc/NEWS.d/next/Library/2019-10-10-16-53-00.bpo-38431.d5wzNp.rst b/Misc/NEWS.d/next/Library/2019-10-10-16-53-00.bpo-38431.d5wzNp.rst
new file mode 100644
index 0000000000000..c2f860d804c13
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-10-10-16-53-00.bpo-38431.d5wzNp.rst
@@ -0,0 +1 @@
+Fix ``__repr__`` method for :class:`dataclasses.InitVar` to support typing objects, patch by Samuel Colvin.


From webhook-mailer at python.org  Sun Oct 13 07:49:09 2019
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Sun, 13 Oct 2019 11:49:09 -0000
Subject: [Python-checkins] bpo-38202: Fix a crash in dict_view &
 non-itearble. (GH-16241)
Message-ID: <mailman.131.1570967350.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/b16e382c446d76ede22780b15c75f43c5f132e25
commit: b16e382c446d76ede22780b15c75f43c5f132e25
branch: master
author: Zackery Spytz <zspytz at gmail.com>
committer: Serhiy Storchaka <storchaka at gmail.com>
date: 2019-10-13T14:49:05+03:00
summary:

bpo-38202: Fix a crash in dict_view & non-itearble. (GH-16241)

files:
M Lib/test/test_dictviews.py
M Objects/dictobject.c

diff --git a/Lib/test/test_dictviews.py b/Lib/test/test_dictviews.py
index 8410e8b5c44d4..be271bebaaf1f 100644
--- a/Lib/test/test_dictviews.py
+++ b/Lib/test/test_dictviews.py
@@ -227,6 +227,25 @@ def test_set_operations_with_iterator(self):
         self.assertEqual(items | iter([(1, 2)]), {(1, 2), (3, 4)})
         self.assertEqual(items - iter([(1, 2)]), {(3, 4)})
 
+    def test_set_operations_with_noniterable(self):
+        with self.assertRaises(TypeError):
+            {}.keys() & 1
+        with self.assertRaises(TypeError):
+            {}.keys() | 1
+        with self.assertRaises(TypeError):
+            {}.keys() ^ 1
+        with self.assertRaises(TypeError):
+            {}.keys() - 1
+
+        with self.assertRaises(TypeError):
+            {}.items() & 1
+        with self.assertRaises(TypeError):
+            {}.items() | 1
+        with self.assertRaises(TypeError):
+            {}.items() ^ 1
+        with self.assertRaises(TypeError):
+            {}.items() - 1
+
     def test_recursive_repr(self):
         d = {}
         d[42] = d.values()
diff --git a/Objects/dictobject.c b/Objects/dictobject.c
index b496350d47093..64876e05191e2 100644
--- a/Objects/dictobject.c
+++ b/Objects/dictobject.c
@@ -4227,6 +4227,10 @@ _PyDictView_Intersect(PyObject* self, PyObject *other)
         return NULL;
 
     it = PyObject_GetIter(other);
+    if (it == NULL) {
+        Py_DECREF(result);
+        return NULL;
+    }
 
     if (PyDictKeys_Check(self)) {
         dict_contains = dictkeys_contains;


From webhook-mailer at python.org  Sun Oct 13 08:04:08 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Sun, 13 Oct 2019 12:04:08 -0000
Subject: [Python-checkins] bpo-38431: Fix __repr__ method of InitVar to work
 with typing objects. (GH-16702)
Message-ID: <mailman.132.1570968249.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/6da52ac411947d1a7958bbad831fcf8dfc8c95fe
commit: 6da52ac411947d1a7958bbad831fcf8dfc8c95fe
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-13T05:04:05-07:00
summary:

bpo-38431: Fix __repr__ method of InitVar to work with typing objects. (GH-16702)

(cherry picked from commit 793cb85437299a3da3d74fe65480d720af330cbb)

Co-authored-by: Samuel Colvin <samcolvin at gmail.com>

files:
A Misc/NEWS.d/next/Library/2019-10-10-16-53-00.bpo-38431.d5wzNp.rst
M Lib/dataclasses.py
M Lib/test/test_dataclasses.py

diff --git a/Lib/dataclasses.py b/Lib/dataclasses.py
index 2f0e5ff0b6e03..391f32e11d52a 100644
--- a/Lib/dataclasses.py
+++ b/Lib/dataclasses.py
@@ -210,7 +210,12 @@ def __init__(self, type):
         self.type = type
 
     def __repr__(self):
-        return f'dataclasses.InitVar[{self.type.__name__}]'
+        if isinstance(self.type, type):
+            type_name = self.type.__name__
+        else:
+            # typing objects, e.g. List[int]
+            type_name = repr(self.type)
+        return f'dataclasses.InitVar[{type_name}]'
 
 
 # Instances of Field are only ever created from within this module,
diff --git a/Lib/test/test_dataclasses.py b/Lib/test/test_dataclasses.py
index 0885c9798fe68..b018133cf2302 100755
--- a/Lib/test/test_dataclasses.py
+++ b/Lib/test/test_dataclasses.py
@@ -1102,6 +1102,8 @@ def test_init_var_preserve_type(self):
 
         # Make sure the repr is correct.
         self.assertEqual(repr(InitVar[int]), 'dataclasses.InitVar[int]')
+        self.assertEqual(repr(InitVar[List[int]]),
+                         'dataclasses.InitVar[typing.List[int]]')
 
     def test_init_var_inheritance(self):
         # Note that this deliberately tests that a dataclass need not
diff --git a/Misc/NEWS.d/next/Library/2019-10-10-16-53-00.bpo-38431.d5wzNp.rst b/Misc/NEWS.d/next/Library/2019-10-10-16-53-00.bpo-38431.d5wzNp.rst
new file mode 100644
index 0000000000000..c2f860d804c13
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-10-10-16-53-00.bpo-38431.d5wzNp.rst
@@ -0,0 +1 @@
+Fix ``__repr__`` method for :class:`dataclasses.InitVar` to support typing objects, patch by Samuel Colvin.


From webhook-mailer at python.org  Sun Oct 13 11:35:52 2019
From: webhook-mailer at python.org (Pablo Galindo)
Date: Sun, 13 Oct 2019 15:35:52 -0000
Subject: [Python-checkins] Correct signature of __build_class__ (GH-16735)
Message-ID: <mailman.133.1570980952.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/e3babbd03cd2bcb3c85deabae3bc6976aa95a3c3
commit: e3babbd03cd2bcb3c85deabae3bc6976aa95a3c3
branch: master
author: Pablo Galindo <Pablogsal at gmail.com>
committer: GitHub <noreply at github.com>
date: 2019-10-13T16:35:41+01:00
summary:

Correct signature of __build_class__ (GH-16735)

files:
M Python/bltinmodule.c

diff --git a/Python/bltinmodule.c b/Python/bltinmodule.c
index 728ba5bdc99bd..d79e254cc68ca 100644
--- a/Python/bltinmodule.c
+++ b/Python/bltinmodule.c
@@ -260,7 +260,7 @@ builtin___build_class__(PyObject *self, PyObject *const *args, Py_ssize_t nargs,
 }
 
 PyDoc_STRVAR(build_class_doc,
-"__build_class__(func, name, *bases, metaclass=None, **kwds) -> class\n\
+"__build_class__(func, name, /, *bases, [metaclass], **kwds) -> class\n\
 \n\
 Internal helper function used by the class statement.");
 


From webhook-mailer at python.org  Sun Oct 13 11:49:03 2019
From: webhook-mailer at python.org (Pablo Galindo)
Date: Sun, 13 Oct 2019 15:49:03 -0000
Subject: [Python-checkins] bpo-38379: Don't block collection of unreachable
 objects when some objects resurrect (GH-16687)
Message-ID: <mailman.134.1570981744.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/466326dcdf038b948d94302c315be407c73e60d1
commit: 466326dcdf038b948d94302c315be407c73e60d1
branch: master
author: Pablo Galindo <Pablogsal at gmail.com>
committer: GitHub <noreply at github.com>
date: 2019-10-13T16:48:59+01:00
summary:

bpo-38379: Don't block collection of unreachable objects when some objects resurrect (GH-16687)

Currently if any finalizer invoked during garbage collection resurrects any object, the gc gives up and aborts the collection. Although finalizers are assured to only run once per object, this behaviour of the gc can lead to an ever-increasing memory situation if new resurrecting objects are allocated in every new gc collection.

To avoid this, recompute what objects among the unreachable set need to be resurrected and what objects can be safely collected. In this way, resurrecting objects will not block the collection of other objects in the unreachable set.

files:
A Misc/NEWS.d/next/Core and Builtins/2019-10-10-01-41-02.bpo-38379._q4dhn.rst
M Lib/test/test_gc.py
M Modules/gcmodule.c

diff --git a/Lib/test/test_gc.py b/Lib/test/test_gc.py
index f52db1eab169c..fe9d6c2f766cc 100644
--- a/Lib/test/test_gc.py
+++ b/Lib/test/test_gc.py
@@ -822,7 +822,78 @@ def test_get_objects_arguments(self):
         self.assertRaises(TypeError, gc.get_objects, "1")
         self.assertRaises(TypeError, gc.get_objects, 1.234)
 
-    def test_38379(self):
+    def test_resurrection_only_happens_once_per_object(self):
+        class A:  # simple self-loop
+            def __init__(self):
+                self.me = self
+
+        class Lazarus(A):
+            resurrected = 0
+            resurrected_instances = []
+
+            def __del__(self):
+                Lazarus.resurrected += 1
+                Lazarus.resurrected_instances.append(self)
+
+        gc.collect()
+        gc.disable()
+
+        # We start with 0 resurrections
+        laz = Lazarus()
+        self.assertEqual(Lazarus.resurrected, 0)
+
+        # Deleting the instance and triggering a collection
+        # resurrects the object
+        del laz
+        gc.collect()
+        self.assertEqual(Lazarus.resurrected, 1)
+        self.assertEqual(len(Lazarus.resurrected_instances), 1)
+
+        # Clearing the references and forcing a collection
+        # should not resurrect the object again.
+        Lazarus.resurrected_instances.clear()
+        self.assertEqual(Lazarus.resurrected, 1)
+        gc.collect()
+        self.assertEqual(Lazarus.resurrected, 1)
+
+        gc.enable()
+
+    def test_resurrection_is_transitive(self):
+        class Cargo:
+            def __init__(self):
+                self.me = self
+
+        class Lazarus:
+            resurrected_instances = []
+
+            def __del__(self):
+                Lazarus.resurrected_instances.append(self)
+
+        gc.collect()
+        gc.disable()
+
+        laz = Lazarus()
+        cargo = Cargo()
+        cargo_id = id(cargo)
+
+        # Create a cycle between cargo and laz
+        laz.cargo = cargo
+        cargo.laz = laz
+
+        # Drop the references, force a collection and check that
+        # everything was resurrected.
+        del laz, cargo
+        gc.collect()
+        self.assertEqual(len(Lazarus.resurrected_instances), 1)
+        instance = Lazarus.resurrected_instances.pop()
+        self.assertTrue(hasattr(instance, "cargo"))
+        self.assertEqual(id(instance.cargo), cargo_id)
+
+        gc.collect()
+        gc.enable()
+
+    def test_resurrection_does_not_block_cleanup_of_other_objects(self):
+
         # When a finalizer resurrects objects, stats were reporting them as
         # having been collected.  This affected both collect()'s return
         # value and the dicts returned by get_stats().
@@ -861,34 +932,29 @@ def getstats():
         # Nothing is collected - Z() is merely resurrected.
         t = gc.collect()
         c, nc = getstats()
-        #self.assertEqual(t, 2)  # before
-        self.assertEqual(t, 0)  # after
-        #self.assertEqual(c - oldc, 2)   # before
-        self.assertEqual(c - oldc, 0)   # after
+        self.assertEqual(t, 0)
+        self.assertEqual(c - oldc, 0)
         self.assertEqual(nc - oldnc, 0)
 
-        # Unfortunately, a Z() prevents _anything_ from being collected.
-        # It should be possible to collect the A instances anyway, but
-        # that will require non-trivial code changes.
+        # Z() should not prevent anything else from being collected.
         oldc, oldnc = c, nc
         for i in range(N):
             A()
         Z()
-        # Z() prevents anything from being collected.
         t = gc.collect()
         c, nc = getstats()
-        #self.assertEqual(t, 2*N + 2)  # before
-        self.assertEqual(t, 0)  # after
-        #self.assertEqual(c - oldc, 2*N + 2)   # before
-        self.assertEqual(c - oldc, 0)   # after
+        self.assertEqual(t, 2*N)
+        self.assertEqual(c - oldc, 2*N)
         self.assertEqual(nc - oldnc, 0)
 
-        # But the A() trash is reclaimed on the next run.
+        # The A() trash should have been reclaimed already but the
+        # 2 copies of Z are still in zs (and the associated dicts).
         oldc, oldnc = c, nc
+        zs.clear()
         t = gc.collect()
         c, nc = getstats()
-        self.assertEqual(t, 2*N)
-        self.assertEqual(c - oldc, 2*N)
+        self.assertEqual(t, 4)
+        self.assertEqual(c - oldc, 4)
         self.assertEqual(nc - oldnc, 0)
 
         gc.enable()
diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-10-10-01-41-02.bpo-38379._q4dhn.rst b/Misc/NEWS.d/next/Core and Builtins/2019-10-10-01-41-02.bpo-38379._q4dhn.rst
new file mode 100644
index 0000000000000..86f908b67740b
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2019-10-10-01-41-02.bpo-38379._q4dhn.rst	
@@ -0,0 +1,4 @@
+When the garbage collector makes a collection in which some objects
+resurrect (they are reachable from outside the isolated cycles after the
+finalizers have been executed), do not block the collection of all objects
+that are still unreachable. Patch by Pablo Galindo and Tim Peters.
diff --git a/Modules/gcmodule.c b/Modules/gcmodule.c
index a1cb323bd24cb..0fb9ac2ac1ff5 100644
--- a/Modules/gcmodule.c
+++ b/Modules/gcmodule.c
@@ -301,8 +301,18 @@ gc_list_size(PyGC_Head *list)
     return n;
 }
 
+/* Walk the list and mark all objects as non-collecting */
+static inline void
+gc_list_clear_collecting(PyGC_Head *collectable)
+{
+    PyGC_Head *gc;
+    for (gc = GC_NEXT(collectable); gc != collectable; gc = GC_NEXT(gc)) {
+        gc_clear_collecting(gc);
+    }
+}
+
 /* Append objects in a GC list to a Python list.
- * Return 0 if all OK, < 0 if error (out of memory for list).
+ * Return 0 if all OK, < 0 if error (out of memory for list)
  */
 static int
 append_objects(PyObject *py_list, PyGC_Head *gc_list)
@@ -613,6 +623,22 @@ move_legacy_finalizers(PyGC_Head *unreachable, PyGC_Head *finalizers)
     }
 }
 
+static inline void
+clear_unreachable_mask(PyGC_Head *unreachable)
+{
+    /* Check that the list head does not have the unreachable bit set */
+    assert(((uintptr_t)unreachable & NEXT_MASK_UNREACHABLE) == 0);
+
+    PyGC_Head *gc, *next;
+    unreachable->_gc_next &= ~NEXT_MASK_UNREACHABLE;
+    for (gc = GC_NEXT(unreachable); gc != unreachable; gc = next) {
+        _PyObject_ASSERT((PyObject*)FROM_GC(gc), gc->_gc_next & NEXT_MASK_UNREACHABLE);
+        gc->_gc_next &= ~NEXT_MASK_UNREACHABLE;
+        next = (PyGC_Head*)gc->_gc_next;
+    }
+    validate_list(unreachable, PREV_MASK_COLLECTING);
+}
+
 /* A traversal callback for move_legacy_finalizer_reachable. */
 static int
 visit_move(PyObject *op, PyGC_Head *tolist)
@@ -891,36 +917,6 @@ finalize_garbage(PyGC_Head *collectable)
     gc_list_merge(&seen, collectable);
 }
 
-/* Walk the collectable list and check that they are really unreachable
-   from the outside (some objects could have been resurrected by a
-   finalizer). */
-static int
-check_garbage(PyGC_Head *collectable)
-{
-    int ret = 0;
-    PyGC_Head *gc;
-    for (gc = GC_NEXT(collectable); gc != collectable; gc = GC_NEXT(gc)) {
-        // Use gc_refs and break gc_prev again.
-        gc_set_refs(gc, Py_REFCNT(FROM_GC(gc)));
-        _PyObject_ASSERT(FROM_GC(gc), gc_get_refs(gc) != 0);
-    }
-    subtract_refs(collectable);
-    PyGC_Head *prev = collectable;
-    for (gc = GC_NEXT(collectable); gc != collectable; gc = GC_NEXT(gc)) {
-        _PyObject_ASSERT_WITH_MSG(FROM_GC(gc),
-                                  gc_get_refs(gc) >= 0,
-                                  "refcount is too small");
-        if (gc_get_refs(gc) != 0) {
-            ret = -1;
-        }
-        // Restore gc_prev here.
-        _PyGCHead_SET_PREV(gc, prev);
-        gc_clear_collecting(gc);
-        prev = gc;
-    }
-    return ret;
-}
-
 /* Break reference cycles by clearing the containers involved.  This is
  * tricky business as the lists can be changing and we don't know which
  * objects may be freed.  It is possible I screwed something up here.
@@ -958,6 +954,7 @@ delete_garbage(struct _gc_runtime_state *state,
         }
         if (GC_NEXT(collectable) == gc) {
             /* object is still alive, move it, it may die later */
+            gc_clear_collecting(gc);
             gc_list_move(gc, old);
         }
     }
@@ -1003,6 +1000,87 @@ show_stats_each_generations(struct _gc_runtime_state *state)
         buf, gc_list_size(&state->permanent_generation.head));
 }
 
+/* Deduce wich objects among "base" are unreachable from outside the list
+   and move them to 'unreachable'. The process consist in the following steps:
+
+1. Copy all reference counts to a different field (gc_prev is used to hold
+   this copy to save memory).
+2. Traverse all objects in "base" and visit all referred objects using
+   "tp_traverse" and for every visited object, substract 1 to the reference
+   count (the one that we copied in the previous step). After this step, all
+   objects that can be reached directly from outside must have strictly positive
+   reference count, while all unreachable objects must have a count of exactly 0.
+3. Indentify all unreachable objects (the ones with 0 reference count) and move
+   them to the "unreachable" list. This step also needs to move back to "base" all
+   objects that were initially marked as unreachable but are referred transitively
+   by the reachable objects (the ones with strictly positive reference count).
+
+Contracts:
+
+    * The "base" has to be a valid list with no mask set.
+
+    * The "unreachable" list must be uninitialized (this function calls
+      gc_list_init over 'unreachable').
+    
+IMPORTANT: This function leaves 'unreachable' with the NEXT_MASK_UNREACHABLE
+flag set but it does not clear it to skip unnecessary iteration. Before the
+flag is cleared (for example, by using 'clear_unreachable_mask' function or
+by a call to 'move_legacy_finalizers'), the 'unreachable' list is not a normal
+list and we can not use most gc_list_* functions for it. */
+static inline void
+deduce_unreachable(PyGC_Head *base, PyGC_Head *unreachable) {
+    validate_list(base, 0);
+    /* Using ob_refcnt and gc_refs, calculate which objects in the
+     * container set are reachable from outside the set (i.e., have a
+     * refcount greater than 0 when all the references within the
+     * set are taken into account).
+     */
+    update_refs(base);  // gc_prev is used for gc_refs
+    subtract_refs(base);
+
+    /* Leave everything reachable from outside base in base, and move
+     * everything else (in base) to unreachable.
+     * NOTE:  This used to move the reachable objects into a reachable
+     * set instead.  But most things usually turn out to be reachable,
+     * so it's more efficient to move the unreachable things.
+     */
+    gc_list_init(unreachable);
+    move_unreachable(base, unreachable);  // gc_prev is pointer again
+    validate_list(base, 0);
+}
+
+/* Handle objects that may have resurrected after a call to 'finalize_garbage', moving
+   them to 'old_generation' and placing the rest on 'still_unreachable'.
+
+   Contracts:
+       * After this function 'unreachable' must not be used anymore and 'still_unreachable'
+         will contain the objects that did not resurrect.
+
+       * The "still_unreachable" list must be uninitialized (this function calls
+         gc_list_init over 'still_unreachable').
+
+IMPORTANT: After a call to this function, the 'still_unreachable' set will have the
+PREV_MARK_COLLECTING set, but the objects in this set are going to be removed so
+we can skip the expense of clearing the flag to avoid extra iteration. */
+static inline void
+handle_resurrected_objects(PyGC_Head *unreachable, PyGC_Head* still_unreachable,
+                           PyGC_Head *old_generation)
+{
+    // Remove the PREV_MASK_COLLECTING from unreachable
+    // to prepare it for a new call to 'deduce_unreachable'
+    gc_list_clear_collecting(unreachable);
+
+    // After the call to deduce_unreachable, the 'still_unreachable' set will
+    // have the PREV_MARK_COLLECTING set, but the objects are going to be
+    // removed so we can skip the expense of clearing the flag.
+    PyGC_Head* resurrected = unreachable;
+    deduce_unreachable(resurrected, still_unreachable);
+    clear_unreachable_mask(still_unreachable);
+
+    // Move the resurrected objects to the old generation for future collection.
+    gc_list_merge(resurrected, old_generation);
+}
+
 /* This is the main function.  Read this to understand how the
  * collection process works. */
 static Py_ssize_t
@@ -1045,26 +1123,9 @@ collect(struct _gc_runtime_state *state, int generation,
         old = GEN_HEAD(state, generation+1);
     else
         old = young;
-
-    validate_list(young, 0);
     validate_list(old, 0);
-    /* Using ob_refcnt and gc_refs, calculate which objects in the
-     * container set are reachable from outside the set (i.e., have a
-     * refcount greater than 0 when all the references within the
-     * set are taken into account).
-     */
-    update_refs(young);  // gc_prev is used for gc_refs
-    subtract_refs(young);
 
-    /* Leave everything reachable from outside young in young, and move
-     * everything else (in young) to unreachable.
-     * NOTE:  This used to move the reachable objects into a reachable
-     * set instead.  But most things usually turn out to be reachable,
-     * so it's more efficient to move the unreachable things.
-     */
-    gc_list_init(&unreachable);
-    move_unreachable(young, &unreachable);  // gc_prev is pointer again
-    validate_list(young, 0);
+    deduce_unreachable(young, &unreachable);
 
     untrack_tuples(young);
     /* Move reachable objects to next generation. */
@@ -1114,17 +1175,18 @@ collect(struct _gc_runtime_state *state, int generation,
     /* Call tp_finalize on objects which have one. */
     finalize_garbage(&unreachable);
 
-    if (check_garbage(&unreachable)) { // clear PREV_MASK_COLLECTING here
-        gc_list_merge(&unreachable, old);
-    }
-    else {
-        /* Call tp_clear on objects in the unreachable set.  This will cause
-         * the reference cycles to be broken.  It may also cause some objects
-         * in finalizers to be freed.
-         */
-        m += gc_list_size(&unreachable);
-        delete_garbage(state, &unreachable, old);
-    }
+    /* Handle any objects that may have resurrected after the call
+     * to 'finalize_garbage' and continue the collection with the
+     * objects that are still unreachable */
+    PyGC_Head final_unreachable;
+    handle_resurrected_objects(&unreachable, &final_unreachable, old);
+
+    /* Call tp_clear on objects in the final_unreachable set.  This will cause
+    * the reference cycles to be broken.  It may also cause some objects
+    * in finalizers to be freed.
+    */
+    m += gc_list_size(&final_unreachable);
+    delete_garbage(state, &final_unreachable, old);
 
     /* Collect statistics on uncollectable objects found and print
      * debugging information. */


From webhook-mailer at python.org  Sun Oct 13 11:51:40 2019
From: webhook-mailer at python.org (Andrew Kuchling)
Date: Sun, 13 Oct 2019 15:51:40 -0000
Subject: [Python-checkins] Rebased version of what's new PR (#16745)
Message-ID: <mailman.135.1570981901.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/bb78f6cfa6f2b84fa4611d39c35baf1c7dce7f8d
commit: bb78f6cfa6f2b84fa4611d39c35baf1c7dce7f8d
branch: master
author: Andrew Kuchling <amk at amk.ca>
committer: GitHub <noreply at github.com>
date: 2019-10-13T11:51:36-04:00
summary:

Rebased version of what's new PR (#16745)

* Use Unicode character for accent
* Various grammar fixes
* Sort library modules alphabetically; remove duplicated idlelib/IDLE section

files:
M Doc/whatsnew/3.8.rst

diff --git a/Doc/whatsnew/3.8.rst b/Doc/whatsnew/3.8.rst
index a609d19776e3d..422a6497e0a2e 100644
--- a/Doc/whatsnew/3.8.rst
+++ b/Doc/whatsnew/3.8.rst
@@ -82,7 +82,7 @@ Assignment expressions
 ----------------------
 
 There is new syntax ``:=`` that assigns values to variables as part of a larger
-expression. It is affectionately known as "walrus operator" due to
+expression. It is affectionately known as "the walrus operator" due to
 its resemblance to `the eyes and tusks of a walrus
 <https://en.wikipedia.org/wiki/Walrus#/media/File:Pacific_Walrus_-_Bull_(8247646168).jpg>`_.
 
@@ -620,6 +620,16 @@ where the DLL is stored (if a full or partial path is used to load the initial
 DLL) and paths added by :func:`~os.add_dll_directory`.
 
 
+datetime
+--------
+
+Added new alternate constructors :meth:`datetime.date.fromisocalendar` and
+:meth:`datetime.datetime.fromisocalendar`, which construct :class:`date` and
+:class:`datetime` objects respectively from ISO year, week number, and weekday;
+these are the inverse of each class's ``isocalendar`` method.
+(Contributed by Paul Ganssle in :issue:`36004`.)
+
+
 functools
 ---------
 
@@ -637,14 +647,12 @@ than as a function returning a decorator.  So both of these are now supported::
 (Contributed by Raymond Hettinger in :issue:`36772`.)
 
 
-datetime
---------
+gc
+--
 
-Added new alternate constructors :meth:`datetime.date.fromisocalendar` and
-:meth:`datetime.datetime.fromisocalendar`, which construct :class:`date` and
-:class:`datetime` objects respectively from ISO year, week number and weekday;
-these are the inverse of each class's ``isocalendar`` method.
-(Contributed by Paul Ganssle in :issue:`36004`.)
+:func:`~gc.get_objects` can now receive an optional *generation* parameter
+indicating a generation to get objects from. Contributed in
+:issue:`36016` by Pablo Galindo.
 
 
 gettext
@@ -654,6 +662,18 @@ Added :func:`~gettext.pgettext` and its variants.
 (Contributed by Franz Glasner, ?ric Araujo, and Cheryl Sabella in :issue:`2504`.)
 
 
+gzip
+----
+
+Added the *mtime* parameter to :func:`gzip.compress` for reproducible output.
+(Contributed by Guo Ci Teo in :issue:`34898`.)
+
+A :exc:`~gzip.BadGzipFile` exception is now raised instead of :exc:`OSError`
+for certain types of invalid or corrupt gzip files.
+(Contributed by Filip Gruszczy?ski, Michele Orr?, and Zackery Spytz in
+:issue:`6584`.)
+
+
 idlelib and IDLE
 ----------------
 
@@ -704,44 +724,6 @@ fails. The exception is ignored silently by default in release build.
 (Contributed by Victor Stinner in :issue:`18748`.)
 
 
-gc
---
-
-:func:`~gc.get_objects` can now receive an optional *generation* parameter
-indicating a generation to get objects from. Contributed in
-:issue:`36016` by Pablo Galindo.
-
-
-gzip
-----
-
-Added the *mtime* parameter to :func:`gzip.compress` for reproducible output.
-(Contributed by Guo Ci Teo in :issue:`34898`.)
-
-A :exc:`~gzip.BadGzipFile` exception is now raised instead of :exc:`OSError`
-for certain types of invalid or corrupt gzip files.
-(Contributed by Filip Gruszczy?ski, Michele Orr?, and Zackery Spytz in
-:issue:`6584`.)
-
-
-idlelib and IDLE
-----------------
-
-Add optional line numbers for IDLE editor windows. Windows
-open without line numbers unless set otherwise in the General
-tab of the configuration dialog.
-(Contributed by Tal Einat and Saimadhav Heblikar in :issue:`17535`.)
-
-Output over N lines (50 by default) is squeezed down to a button.
-N can be changed in the PyShell section of the General page of the
-Settings dialog.  Fewer, but possibly extra long, lines can be squeezed by
-right clicking on the output.  Squeezed output can be expanded in place
-by double-clicking the button or into the clipboard or a separate window
-by right-clicking the button.  (Contributed by Tal Einat in :issue:`1529353`.)
-
-The changes above have been backported to 3.7 maintenance releases.
-
-
 json.tool
 ---------
 
@@ -899,18 +881,6 @@ py_compile
 (Contributed by Joannah Nanjekye in :issue:`22640`.)
 
 
-socket
-------
-
-Added :meth:`~socket.create_server()` and :meth:`~socket.has_dualstack_ipv6()`
-convenience functions to automate the necessary tasks usually involved when
-creating a server socket, including accepting both IPv4 and IPv6 connections
-on the same socket.  (Contributed by Giampaolo Rodola in :issue:`17561`.)
-
-The :func:`socket.if_nameindex()`, :func:`socket.if_nametoindex()`, and
-:func:`socket.if_indextoname()` functions have been implemented on Windows.
-(Contributed by Zackery Spytz in :issue:`37007`.)
-
 shlex
 ----------
 
@@ -932,6 +902,19 @@ inherited from the corresponding change to the :mod:`tarfile` module.
 recursively removing their contents first.
 
 
+socket
+------
+
+Added :meth:`~socket.create_server()` and :meth:`~socket.has_dualstack_ipv6()`
+convenience functions to automate the necessary tasks usually involved when
+creating a server socket, including accepting both IPv4 and IPv6 connections
+on the same socket.  (Contributed by Giampaolo Rodol? in :issue:`17561`.)
+
+The :func:`socket.if_nameindex()`, :func:`socket.if_nametoindex()`, and
+:func:`socket.if_indextoname()` functions have been implemented on Windows.
+(Contributed by Zackery Spytz in :issue:`37007`.)
+
+
 ssl
 ---
 
@@ -1202,14 +1185,14 @@ Optimizations
   +26% on Linux, +50% on macOS and +40% on Windows. Also, much less CPU cycles
   are consumed.
   See :ref:`shutil-platform-dependent-efficient-copy-operations` section.
-  (Contributed by Giampaolo Rodola' in :issue:`33671`.)
+  (Contributed by Giampaolo Rodol? in :issue:`33671`.)
 
 * :func:`shutil.copytree` uses :func:`os.scandir` function and all copy
   functions depending from it use cached :func:`os.stat` values. The speedup
   for copying a directory with 8000 files is around +9% on Linux, +20% on
   Windows and +30% on a Windows SMB share. Also the number of :func:`os.stat`
   syscalls is reduced by 38% making :func:`shutil.copytree` especially faster
-  on network filesystems. (Contributed by Giampaolo Rodola' in :issue:`33695`.)
+  on network filesystems. (Contributed by Giampaolo Rodol? in :issue:`33695`.)
 
 * The default protocol in the :mod:`pickle` module is now Protocol 4,
   first introduced in Python 3.4.  It offers better performance and smaller
@@ -1361,7 +1344,7 @@ Deprecated
   (Contributed by Victor Stinner in :issue:`37481`.)
 
 * Deprecated methods ``getchildren()`` and ``getiterator()`` in
-  the :mod:`~xml.etree.ElementTree` module emit now a
+  the :mod:`~xml.etree.ElementTree` module now emit a
   :exc:`DeprecationWarning` instead of :exc:`PendingDeprecationWarning`.
   They will be removed in Python 3.9.
   (Contributed by Serhiy Storchaka in :issue:`29209`.)
@@ -1454,28 +1437,29 @@ The following features and APIs have been removed from Python 3.8:
 * The :mod:`macpath` module, deprecated in Python 3.7, has been removed.
   (Contributed by Victor Stinner in :issue:`35471`.)
 
-* The function :func:`platform.popen` has been removed, it was deprecated since
-  Python 3.3: use :func:`os.popen` instead.
+* The function :func:`platform.popen` has been removed, after having been
+  deprecated since Python 3.3: use :func:`os.popen` instead.
   (Contributed by Victor Stinner in :issue:`35345`.)
 
-* The function :func:`time.clock` has been removed, it was deprecated since Python
-  3.3: use :func:`time.perf_counter` or :func:`time.process_time` instead, depending
-  on your requirements, to have a well defined behavior.
+* The function :func:`time.clock` has been removed, after having been
+  deprecated since Python 3.3: use :func:`time.perf_counter` or
+  :func:`time.process_time` instead, depending
+  on your requirements, to have well-defined behavior.
   (Contributed by Matthias Bussonnier in :issue:`36895`.)
 
 * The ``pyvenv`` script has been removed in favor of ``python3.8 -m venv``
   to help eliminate confusion as to what Python interpreter the ``pyvenv``
   script is tied to. (Contributed by Brett Cannon in :issue:`25427`.)
 
-* ``parse_qs``, ``parse_qsl``, and ``escape`` are removed from :mod:`cgi`
-  module.  They are deprecated from Python 3.2 or older. They should be imported
+* ``parse_qs``, ``parse_qsl``, and ``escape`` are removed from the :mod:`cgi`
+  module.  They are deprecated in Python 3.2 or older. They should be imported
   from the ``urllib.parse`` and ``html`` modules instead.
 
-* ``filemode`` function is removed from :mod:`tarfile` module.
+* ``filemode`` function is removed from the :mod:`tarfile` module.
   It is not documented and deprecated since Python 3.3.
 
 * The :class:`~xml.etree.ElementTree.XMLParser` constructor no longer accepts
-  the *html* argument.  It never had effect and was deprecated in Python 3.4.
+  the *html* argument.  It never had an effect and was deprecated in Python 3.4.
   All other parameters are now :ref:`keyword-only <keyword-only_parameter>`.
   (Contributed by Serhiy Storchaka in :issue:`29209`.)
 
@@ -1534,8 +1518,8 @@ Changes in Python behavior
 
 * On AIX, :attr:`sys.platform` doesn't contain the major version anymore.
   It is always ``'aix'``, instead of ``'aix3'`` .. ``'aix7'``.  Since
-  older Python versions include the version number, it is recommended to
-  always use the ``sys.platform.startswith('aix')``.
+  older Python versions include the version number, so it is recommended to
+  always use ``sys.platform.startswith('aix')``.
   (Contributed by M. Felt in :issue:`36588`.)
 
 * :c:func:`PyEval_AcquireLock` and :c:func:`PyEval_AcquireThread` now
@@ -1555,8 +1539,8 @@ Changes in the Python API
 
 * :class:`subprocess.Popen` can now use :func:`os.posix_spawn` in some cases
   for better performance. On Windows Subsystem for Linux and QEMU User
-  Emulation, Popen constructor using :func:`os.posix_spawn` no longer raise an
-  exception on errors like missing program, but the child process fails with a
+  Emulation, the :class:`Popen` constructor using :func:`os.posix_spawn` no longer raises an
+  exception on errors like "missing program".  Instead the child process fails with a
   non-zero :attr:`~Popen.returncode`.
   (Contributed by Joannah Nanjekye and Victor Stinner in :issue:`35537`.)
 
@@ -1569,7 +1553,7 @@ Changes in the Python API
 * The :meth:`imap.IMAP4.logout` method no longer ignores silently arbitrary
   exceptions.
 
-* The function :func:`platform.popen` has been removed, it was deprecated since
+* The function :func:`platform.popen` has been removed, after having been deprecated since
   Python 3.3: use :func:`os.popen` instead.
   (Contributed by Victor Stinner in :issue:`35345`.)
 
@@ -1584,9 +1568,9 @@ Changes in the Python API
   specialized methods like :meth:`~tkinter.ttk.Treeview.selection_set` for
   changing the selection.  (Contributed by Serhiy Storchaka in :issue:`31508`.)
 
-* The :meth:`writexml`, :meth:`toxml` and :meth:`toprettyxml` methods of the
-  :mod:`xml.dom.minidom` module, and :mod:`xml.etree` now preserve the attribute
-  order specified by the user.
+* The :meth:`writexml`, :meth:`toxml` and :meth:`toprettyxml` methods of
+  :mod:`xml.dom.minidom`, and the :meth:`write` method of :mod:`xml.etree`,
+  now preserve the attribute order specified by the user.
   (Contributed by Diego Rojas and Raymond Hettinger in :issue:`34160`.)
 
 * A :mod:`dbm.dumb` database opened with flags ``'r'`` is now read-only.
@@ -1596,7 +1580,7 @@ Changes in the Python API
 
 * The ``doctype()`` method defined in a subclass of
   :class:`~xml.etree.ElementTree.XMLParser` will no longer be called and will
-  cause emitting a :exc:`RuntimeWarning` instead of a :exc:`DeprecationWarning`.
+  emit a :exc:`RuntimeWarning` instead of a :exc:`DeprecationWarning`.
   Define the :meth:`doctype() <xml.etree.ElementTree.TreeBuilder.doctype>`
   method on a target for handling an XML doctype declaration.
   (Contributed by Serhiy Storchaka in :issue:`29209`.)
@@ -1639,7 +1623,7 @@ Changes in the Python API
 
 * The :meth:`mmap.flush() <mmap.mmap.flush>` method now returns ``None`` on
   success and raises an exception on error under all platforms.  Previously,
-  its behavior was platform-depended: a nonzero value was returned on success;
+  its behavior was platform-dependent: a nonzero value was returned on success;
   zero was returned on error under Windows.  A zero value was returned on
   success; an exception was raised on error under Unix.
   (Contributed by Berker Peksag in :issue:`2122`.)
@@ -1683,7 +1667,7 @@ Changes in the Python API
 * :class:`types.CodeType` has a new parameter in the second position of the
   constructor (*posonlyargcount*) to support positional-only arguments defined
   in :pep:`570`. The first argument (*argcount*) now represents the total
-  number of positional arguments (including positional-only arguments). A new
+  number of positional arguments (including positional-only arguments). The new
   ``replace()`` method of :class:`types.CodeType` can be used to make the code
   future-proof.
 
@@ -1691,9 +1675,9 @@ Changes in the Python API
 Changes in the C API
 --------------------
 
-* The :c:type:`PyCompilerFlags` structure gets a new *cf_feature_version*
+* The :c:type:`PyCompilerFlags` structure got a new *cf_feature_version*
   field. It should be initialized to ``PY_MINOR_VERSION``. The field is ignored
-  by default, it is used if and only if ``PyCF_ONLY_AST`` flag is set in
+  by default, and is used if and only if ``PyCF_ONLY_AST`` flag is set in
   *cf_flags*.
 
 * The :c:func:`PyEval_ReInitThreads` function has been removed from the C API.


From webhook-mailer at python.org  Sun Oct 13 11:53:58 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Sun, 13 Oct 2019 15:53:58 -0000
Subject: [Python-checkins] Correct signature of __build_class__ (GH-16735)
Message-ID: <mailman.136.1570982039.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/a04c7ebcb97fefe9b2fea0c805112afd927ac66b
commit: a04c7ebcb97fefe9b2fea0c805112afd927ac66b
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-13T08:53:54-07:00
summary:

Correct signature of __build_class__ (GH-16735)

(cherry picked from commit e3babbd03cd2bcb3c85deabae3bc6976aa95a3c3)

Co-authored-by: Pablo Galindo <Pablogsal at gmail.com>

files:
M Python/bltinmodule.c

diff --git a/Python/bltinmodule.c b/Python/bltinmodule.c
index 90a6b9fc6e246..33f969094e7d1 100644
--- a/Python/bltinmodule.c
+++ b/Python/bltinmodule.c
@@ -261,7 +261,7 @@ builtin___build_class__(PyObject *self, PyObject *const *args, Py_ssize_t nargs,
 }
 
 PyDoc_STRVAR(build_class_doc,
-"__build_class__(func, name, *bases, metaclass=None, **kwds) -> class\n\
+"__build_class__(func, name, /, *bases, [metaclass], **kwds) -> class\n\
 \n\
 Internal helper function used by the class statement.");
 


From webhook-mailer at python.org  Sun Oct 13 12:53:12 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Sun, 13 Oct 2019 16:53:12 -0000
Subject: [Python-checkins] bpo-28556: Remove another mention of metaclass of
 Generic in typing docs (GH-16743)
Message-ID: <mailman.137.1570985593.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/8144095707f87bdee6f3e1bbb15283ea61381be6
commit: 8144095707f87bdee6f3e1bbb15283ea61381be6
branch: master
author: Ivan Levkivskyi <levkivskyi at gmail.com>
committer: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
date: 2019-10-13T09:53:06-07:00
summary:

bpo-28556: Remove another mention of metaclass of Generic in typing docs (GH-16743)



Metaclass was removed in Python 3.7 (there is already a `versionchanged` item about this).


https://bugs.python.org/issue28556

files:
M Doc/library/typing.rst

diff --git a/Doc/library/typing.rst b/Doc/library/typing.rst
index 642f8c653eeb5..94d60b4602526 100644
--- a/Doc/library/typing.rst
+++ b/Doc/library/typing.rst
@@ -231,8 +231,8 @@ A user-defined class can be defined as a generic class.
 single type parameter ``T`` . This also makes ``T`` valid as a type within the
 class body.
 
-The :class:`Generic` base class uses a metaclass that defines
-:meth:`__getitem__` so that ``LoggedVar[t]`` is valid as a type::
+The :class:`Generic` base class defines :meth:`__class_getitem__` so that
+``LoggedVar[t]`` is valid as a type::
 
    from typing import Iterable
 


From webhook-mailer at python.org  Sun Oct 13 12:59:03 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Sun, 13 Oct 2019 16:59:03 -0000
Subject: [Python-checkins] bpo-28556: Remove another mention of metaclass of
 Generic in typing docs (GH-16743)
Message-ID: <mailman.138.1570985944.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/20d1f49a0008e7c306cbe34dcc1b7d6e62dc3d24
commit: 20d1f49a0008e7c306cbe34dcc1b7d6e62dc3d24
branch: 3.7
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-13T09:58:59-07:00
summary:

bpo-28556: Remove another mention of metaclass of Generic in typing docs (GH-16743)


Metaclass was removed in Python 3.7 (there is already a `versionchanged` item about this).

https://bugs.python.org/issue28556
(cherry picked from commit 8144095707f87bdee6f3e1bbb15283ea61381be6)

Co-authored-by: Ivan Levkivskyi <levkivskyi at gmail.com>

files:
M Doc/library/typing.rst

diff --git a/Doc/library/typing.rst b/Doc/library/typing.rst
index b42f33ccc0d35..2c84551e4eae0 100644
--- a/Doc/library/typing.rst
+++ b/Doc/library/typing.rst
@@ -230,8 +230,8 @@ A user-defined class can be defined as a generic class.
 single type parameter ``T`` . This also makes ``T`` valid as a type within the
 class body.
 
-The :class:`Generic` base class uses a metaclass that defines
-:meth:`__getitem__` so that ``LoggedVar[t]`` is valid as a type::
+The :class:`Generic` base class defines :meth:`__class_getitem__` so that
+``LoggedVar[t]`` is valid as a type::
 
    from typing import Iterable
 


From webhook-mailer at python.org  Sun Oct 13 13:00:23 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Sun, 13 Oct 2019 17:00:23 -0000
Subject: [Python-checkins] bpo-28556: Remove another mention of metaclass of
 Generic in typing docs (GH-16743)
Message-ID: <mailman.139.1570986025.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/e742826531d4369cb1eff65337fa95fd67de8d42
commit: e742826531d4369cb1eff65337fa95fd67de8d42
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-13T10:00:19-07:00
summary:

bpo-28556: Remove another mention of metaclass of Generic in typing docs (GH-16743)


Metaclass was removed in Python 3.7 (there is already a `versionchanged` item about this).

https://bugs.python.org/issue28556
(cherry picked from commit 8144095707f87bdee6f3e1bbb15283ea61381be6)

Co-authored-by: Ivan Levkivskyi <levkivskyi at gmail.com>

files:
M Doc/library/typing.rst

diff --git a/Doc/library/typing.rst b/Doc/library/typing.rst
index 642f8c653eeb5..94d60b4602526 100644
--- a/Doc/library/typing.rst
+++ b/Doc/library/typing.rst
@@ -231,8 +231,8 @@ A user-defined class can be defined as a generic class.
 single type parameter ``T`` . This also makes ``T`` valid as a type within the
 class body.
 
-The :class:`Generic` base class uses a metaclass that defines
-:meth:`__getitem__` so that ``LoggedVar[t]`` is valid as a type::
+The :class:`Generic` base class defines :meth:`__class_getitem__` so that
+``LoggedVar[t]`` is valid as a type::
 
    from typing import Iterable
 


From webhook-mailer at python.org  Sun Oct 13 14:31:39 2019
From: webhook-mailer at python.org (Ivan Levkivskyi)
Date: Sun, 13 Oct 2019 18:31:39 -0000
Subject: [Python-checkins] bpo-38467: Fix argument name of typing functions
 (GH-16753)
Message-ID: <mailman.140.1570991501.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/fdfe2833ace93021278fe4c41c40e1d08d70abf9
commit: fdfe2833ace93021278fe4c41c40e1d08d70abf9
branch: master
author: Sebastian Rittau <srittau at rittau.biz>
committer: Ivan Levkivskyi <levkivskyi at gmail.com>
date: 2019-10-13T19:31:35+01:00
summary:

bpo-38467: Fix argument name of typing functions (GH-16753)

files:
M Doc/library/typing.rst

diff --git a/Doc/library/typing.rst b/Doc/library/typing.rst
index 94d60b4602526..323dac2082201 100644
--- a/Doc/library/typing.rst
+++ b/Doc/library/typing.rst
@@ -1041,8 +1041,8 @@ The module defines the following classes, functions and decorators:
    a dictionary constructed by merging all the ``__annotations__`` along
    ``C.__mro__`` in reverse order.
 
-.. function:: get_origin(typ)
-.. function:: get_args(typ)
+.. function:: get_origin(tp)
+.. function:: get_args(tp)
 
    Provide basic introspection for generic types and special typing forms.
 


From webhook-mailer at python.org  Sun Oct 13 14:38:49 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Sun, 13 Oct 2019 18:38:49 -0000
Subject: [Python-checkins] bpo-38467: Fix argument name of typing functions
 (GH-16753)
Message-ID: <mailman.141.1570991930.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/c732660b7b18352ba8524ac78d39975afe4392ef
commit: c732660b7b18352ba8524ac78d39975afe4392ef
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-13T11:38:45-07:00
summary:

bpo-38467: Fix argument name of typing functions (GH-16753)

(cherry picked from commit fdfe2833ace93021278fe4c41c40e1d08d70abf9)

Co-authored-by: Sebastian Rittau <srittau at rittau.biz>

files:
M Doc/library/typing.rst

diff --git a/Doc/library/typing.rst b/Doc/library/typing.rst
index 94d60b4602526..323dac2082201 100644
--- a/Doc/library/typing.rst
+++ b/Doc/library/typing.rst
@@ -1041,8 +1041,8 @@ The module defines the following classes, functions and decorators:
    a dictionary constructed by merging all the ``__annotations__`` along
    ``C.__mro__`` in reverse order.
 
-.. function:: get_origin(typ)
-.. function:: get_args(typ)
+.. function:: get_origin(tp)
+.. function:: get_args(tp)
 
    Provide basic introspection for generic types and special typing forms.
 


From webhook-mailer at python.org  Sun Oct 13 17:47:08 2019
From: webhook-mailer at python.org (Tim Peters)
Date: Sun, 13 Oct 2019 21:47:08 -0000
Subject: [Python-checkins] Misc gc code & comment cleanups. (GH-16752)
Message-ID: <mailman.142.1571003229.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/95bfc8a11a0ef09912b288ed3415405d928d0dee
commit: 95bfc8a11a0ef09912b288ed3415405d928d0dee
branch: master
author: Tim Peters <tim.peters at gmail.com>
committer: GitHub <noreply at github.com>
date: 2019-10-13T16:47:04-05:00
summary:

Misc gc code & comment cleanups. (GH-16752)

* Misc gc code & comment cleanups.

validate_list:  there are two temp flags polluting pointers, but this checked only one.  Now it checks both, and verifies that the list head's pointers are not polluted.

move_unreachable: repaired incoherent comments.  Added new comments.  Cleared the pollution of the unreachable list head's 'next' pointer (it was expedient while the function was running, but there's no excuse for letting this damage survive the function's end).

* Update Modules/gcmodule.c

Co-Authored-By: Pablo Galindo <Pablogsal at gmail.com>

files:
M Modules/gcmodule.c

diff --git a/Modules/gcmodule.c b/Modules/gcmodule.c
index 0fb9ac2ac1ff5..aca3f2260b6e1 100644
--- a/Modules/gcmodule.c
+++ b/Modules/gcmodule.c
@@ -331,23 +331,37 @@ append_objects(PyObject *py_list, PyGC_Head *gc_list)
 
 #ifdef GC_DEBUG
 // validate_list checks list consistency.  And it works as document
-// describing when expected_mask is set / unset.
+// describing when flags are expected to be set / unset.
+// `head` must be a doubly-linked gc list, although it's fine (expected!) if
+// the prev and next pointers are "polluted" with flags.
+// What's checked:
+// - The `head` pointers are not polluted.
+// - The objects' prev pointers' PREV_MASK_COLLECTING flags are all
+//   `prev_value` (PREV_MASK_COLLECTING for set, 0 for clear).
+// - The objects' next pointers' NEXT_MASK_UNREACHABLE flags are all
+//   `next_value` (NEXT_MASK_UNREACHABLE for set, 0 for clear).
+// - The prev and next pointers are mutually consistent.
 static void
-validate_list(PyGC_Head *head, uintptr_t expected_mask)
+validate_list(PyGC_Head *head, uintptr_t prev_value, uintptr_t next_value)
 {
+    assert((head->_gc_prev & PREV_MASK_COLLECTING) == 0);
+    assert((head->_gc_next & NEXT_MASK_UNREACHABLE) == 0);
     PyGC_Head *prev = head;
     PyGC_Head *gc = GC_NEXT(head);
     while (gc != head) {
-        assert(GC_NEXT(gc) != NULL);
-        assert(GC_PREV(gc) == prev);
-        assert((gc->_gc_prev & PREV_MASK_COLLECTING) == expected_mask);
+        PyGC_Head *trueprev = GC_PREV(gc);
+        PyGC_Head *truenext = (PyGC_Head *)(gc->_gc_next  & ~NEXT_MASK_UNREACHABLE);
+        assert(truenext != NULL);
+        assert(trueprev == prev);
+        assert((gc->_gc_prev & PREV_MASK_COLLECTING) == prev_value);
+        assert((gc->_gc_next & NEXT_MASK_UNREACHABLE) == next_value);
         prev = gc;
-        gc = GC_NEXT(gc);
+        gc = truenext;
     }
     assert(prev == GC_PREV(head));
 }
 #else
-#define validate_list(x,y) do{}while(0)
+#define validate_list(x, y, z) do{}while(0)
 #endif
 
 /*** end of list stuff ***/
@@ -434,6 +448,9 @@ visit_reachable(PyObject *op, PyGC_Head *reachable)
     const Py_ssize_t gc_refs = gc_get_refs(gc);
 
     // Ignore untracked objects and objects in other generation.
+    // This also skips objects "to the left" of the current position in
+    // move_unreachable's scan of the 'young' list - they've already been
+    // traversed, and no longer have the PREV_MASK_COLLECTING flag.
     if (gc->_gc_next == 0 || !gc_is_collecting(gc)) {
         return 0;
     }
@@ -445,7 +462,8 @@ visit_reachable(PyObject *op, PyGC_Head *reachable)
          * and move_unreachable will eventually get to it
          * again.
          */
-        // Manually unlink gc from unreachable list because
+        // Manually unlink gc from unreachable list because the list functions
+        // don't work right in the presence of NEXT_MASK_UNREACHABLE flags.
         PyGC_Head *prev = GC_PREV(gc);
         PyGC_Head *next = (PyGC_Head*)(gc->_gc_next & ~NEXT_MASK_UNREACHABLE);
         _PyObject_ASSERT(FROM_GC(prev),
@@ -546,8 +564,9 @@ move_unreachable(PyGC_Head *young, PyGC_Head *unreachable)
             PyGC_Head *last = GC_PREV(unreachable);
             // NOTE: Since all objects in unreachable set has
             // NEXT_MASK_UNREACHABLE flag, we set it unconditionally.
-            // But this may set the flat to unreachable too.
-            // move_legacy_finalizers() should care about it.
+            // But this may pollute the unreachable list head's 'next' pointer
+            // too. That's semantically senseless but expedient here - the
+            // damage is repaired when this fumction ends.
             last->_gc_next = (NEXT_MASK_UNREACHABLE | (uintptr_t)gc);
             _PyGCHead_SET_PREV(gc, last);
             gc->_gc_next = (NEXT_MASK_UNREACHABLE | (uintptr_t)unreachable);
@@ -557,6 +576,8 @@ move_unreachable(PyGC_Head *young, PyGC_Head *unreachable)
     }
     // young->_gc_prev must be last element remained in the list.
     young->_gc_prev = (uintptr_t)prev;
+    // don't let the pollution of the list head's next pointer leak
+    unreachable->_gc_next &= ~NEXT_MASK_UNREACHABLE;
 }
 
 static void
@@ -604,7 +625,7 @@ static void
 move_legacy_finalizers(PyGC_Head *unreachable, PyGC_Head *finalizers)
 {
     PyGC_Head *gc, *next;
-    unreachable->_gc_next &= ~NEXT_MASK_UNREACHABLE;
+    assert((unreachable->_gc_next & NEXT_MASK_UNREACHABLE) == 0);
 
     /* March over unreachable.  Move objects with finalizers into
      * `finalizers`.
@@ -630,13 +651,13 @@ clear_unreachable_mask(PyGC_Head *unreachable)
     assert(((uintptr_t)unreachable & NEXT_MASK_UNREACHABLE) == 0);
 
     PyGC_Head *gc, *next;
-    unreachable->_gc_next &= ~NEXT_MASK_UNREACHABLE;
+    assert((unreachable->_gc_next & NEXT_MASK_UNREACHABLE) == 0);
     for (gc = GC_NEXT(unreachable); gc != unreachable; gc = next) {
         _PyObject_ASSERT((PyObject*)FROM_GC(gc), gc->_gc_next & NEXT_MASK_UNREACHABLE);
         gc->_gc_next &= ~NEXT_MASK_UNREACHABLE;
         next = (PyGC_Head*)gc->_gc_next;
     }
-    validate_list(unreachable, PREV_MASK_COLLECTING);
+    validate_list(unreachable, PREV_MASK_COLLECTING, 0);
 }
 
 /* A traversal callback for move_legacy_finalizer_reachable. */
@@ -1021,7 +1042,7 @@ show_stats_each_generations(struct _gc_runtime_state *state)
 
     * The "unreachable" list must be uninitialized (this function calls
       gc_list_init over 'unreachable').
-    
+
 IMPORTANT: This function leaves 'unreachable' with the NEXT_MASK_UNREACHABLE
 flag set but it does not clear it to skip unnecessary iteration. Before the
 flag is cleared (for example, by using 'clear_unreachable_mask' function or
@@ -1029,7 +1050,7 @@ by a call to 'move_legacy_finalizers'), the 'unreachable' list is not a normal
 list and we can not use most gc_list_* functions for it. */
 static inline void
 deduce_unreachable(PyGC_Head *base, PyGC_Head *unreachable) {
-    validate_list(base, 0);
+    validate_list(base, 0, 0);
     /* Using ob_refcnt and gc_refs, calculate which objects in the
      * container set are reachable from outside the set (i.e., have a
      * refcount greater than 0 when all the references within the
@@ -1046,7 +1067,8 @@ deduce_unreachable(PyGC_Head *base, PyGC_Head *unreachable) {
      */
     gc_list_init(unreachable);
     move_unreachable(base, unreachable);  // gc_prev is pointer again
-    validate_list(base, 0);
+    validate_list(base, 0, 0);
+    validate_list(unreachable, PREV_MASK_COLLECTING, NEXT_MASK_UNREACHABLE);
 }
 
 /* Handle objects that may have resurrected after a call to 'finalize_garbage', moving
@@ -1123,7 +1145,7 @@ collect(struct _gc_runtime_state *state, int generation,
         old = GEN_HEAD(state, generation+1);
     else
         old = young;
-    validate_list(old, 0);
+    validate_list(old, 0, 0);
 
     deduce_unreachable(young, &unreachable);
 
@@ -1156,8 +1178,8 @@ collect(struct _gc_runtime_state *state, int generation,
      */
     move_legacy_finalizer_reachable(&finalizers);
 
-    validate_list(&finalizers, 0);
-    validate_list(&unreachable, PREV_MASK_COLLECTING);
+    validate_list(&finalizers, 0, 0);
+    validate_list(&unreachable, PREV_MASK_COLLECTING, 0);
 
     /* Print debugging information. */
     if (state->debug & DEBUG_COLLECTABLE) {
@@ -1169,8 +1191,8 @@ collect(struct _gc_runtime_state *state, int generation,
     /* Clear weakrefs and invoke callbacks as necessary. */
     m += handle_weakrefs(&unreachable, old);
 
-    validate_list(old, 0);
-    validate_list(&unreachable, PREV_MASK_COLLECTING);
+    validate_list(old, 0, 0);
+    validate_list(&unreachable, PREV_MASK_COLLECTING, 0);
 
     /* Call tp_finalize on objects which have one. */
     finalize_garbage(&unreachable);
@@ -1208,7 +1230,7 @@ collect(struct _gc_runtime_state *state, int generation,
      * this if they insist on creating this type of structure.
      */
     handle_legacy_finalizers(state, &finalizers, old);
-    validate_list(old, 0);
+    validate_list(old, 0, 0);
 
     /* Clear free list only during the collection of the highest
      * generation */


From webhook-mailer at python.org  Sun Oct 13 22:14:34 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Mon, 14 Oct 2019 02:14:34 -0000
Subject: [Python-checkins] Fix strict-aliasing rules errors on gcc 4.8.5.
 (GH-16714)
Message-ID: <mailman.143.1571019275.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/9a7c43bafd07d7b8db5f2329b264c9f83652c682
commit: 9a7c43bafd07d7b8db5f2329b264c9f83652c682
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-13T19:14:27-07:00
summary:

Fix strict-aliasing rules errors on gcc 4.8.5. (GH-16714)

(cherry picked from commit c39d1ddc012987e2159a997e27665d2d579c0ce0)

Co-authored-by: Dong-hee Na <donghee.na92 at gmail.com>

files:
M Objects/dictobject.c

diff --git a/Objects/dictobject.c b/Objects/dictobject.c
index 31f962d16b25e..a6ddf70cd8fa6 100644
--- a/Objects/dictobject.c
+++ b/Objects/dictobject.c
@@ -1129,7 +1129,7 @@ insert_to_emptydict(PyDictObject *mp, PyObject *key, Py_hash_t hash,
     MAINTAIN_TRACKING(mp, key, value);
 
     size_t hashpos = (size_t)hash & (PyDict_MINSIZE-1);
-    PyDictKeyEntry *ep = &DK_ENTRIES(mp->ma_keys)[0];
+    PyDictKeyEntry *ep = DK_ENTRIES(mp->ma_keys);
     dictkeys_set_index(mp->ma_keys, hashpos, 0);
     ep->me_key = key;
     ep->me_hash = hash;


From webhook-mailer at python.org  Sun Oct 13 22:53:34 2019
From: webhook-mailer at python.org (Raymond Hettinger)
Date: Mon, 14 Oct 2019 02:53:34 -0000
Subject: [Python-checkins] bpo-38464: Document parameter for
 NormalDist.quantiles() (GH-16757)
Message-ID: <mailman.144.1571021615.30028.python-checkins@python.org>

https://github.com/python/cpython/commit/8a6cbf8adb68aa2c43a48769af92e50d5ca1539d
commit: 8a6cbf8adb68aa2c43a48769af92e50d5ca1539d
branch: master
author: Raymond Hettinger <rhettinger at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-13T19:53:30-07:00
summary:

bpo-38464: Document parameter for NormalDist.quantiles() (GH-16757)

files:
M Doc/library/statistics.rst

diff --git a/Doc/library/statistics.rst b/Doc/library/statistics.rst
index bdd706d0a93e3..a702b2463c39b 100644
--- a/Doc/library/statistics.rst
+++ b/Doc/library/statistics.rst
@@ -681,7 +681,7 @@ of applications in statistics.
        the two probability density functions
        <https://www.rasch.org/rmt/rmt101r.htm>`_.
 
-    .. method:: NormalDist.quantiles()
+    .. method:: NormalDist.quantiles(n=4)
 
         Divide the normal distribution into *n* continuous intervals with
         equal probability.  Returns a list of (n - 1) cut points separating


From webhook-mailer at python.org  Sun Oct 13 23:04:33 2019
From: webhook-mailer at python.org (Raymond Hettinger)
Date: Mon, 14 Oct 2019 03:04:33 -0000
Subject: [Python-checkins] bpo-38464: Document parameter for
 NormalDist.quantiles() (GH-16757) (GH-16758)
Message-ID: <mailman.0.1571022274.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/36f62860f631acefd7988b76901863339ffa26c9
commit: 36f62860f631acefd7988b76901863339ffa26c9
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Raymond Hettinger <rhettinger at users.noreply.github.com>
date: 2019-10-13T20:04:29-07:00
summary:

bpo-38464: Document parameter for NormalDist.quantiles() (GH-16757) (GH-16758)

(cherry picked from commit 8a6cbf8adb68aa2c43a48769af92e50d5ca1539d)

Co-authored-by: Raymond Hettinger <rhettinger at users.noreply.github.com>

files:
M Doc/library/statistics.rst

diff --git a/Doc/library/statistics.rst b/Doc/library/statistics.rst
index bdd706d0a93e3..a702b2463c39b 100644
--- a/Doc/library/statistics.rst
+++ b/Doc/library/statistics.rst
@@ -681,7 +681,7 @@ of applications in statistics.
        the two probability density functions
        <https://www.rasch.org/rmt/rmt101r.htm>`_.
 
-    .. method:: NormalDist.quantiles()
+    .. method:: NormalDist.quantiles(n=4)
 
         Divide the normal distribution into *n* continuous intervals with
         equal probability.  Returns a list of (n - 1) cut points separating


From webhook-mailer at python.org  Sun Oct 13 23:35:17 2019
From: webhook-mailer at python.org (Raymond Hettinger)
Date: Mon, 14 Oct 2019 03:35:17 -0000
Subject: [Python-checkins] Fix minor typos in Whatsnew (GH-16759)
Message-ID: <mailman.1.1571024117.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/75d8a7529574f9997c6b13f9c45f7e79c868b33d
commit: 75d8a7529574f9997c6b13f9c45f7e79c868b33d
branch: 3.8
author: Raymond Hettinger <rhettinger at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-13T20:35:11-07:00
summary:

Fix minor typos in Whatsnew (GH-16759)

files:
M Doc/whatsnew/3.8.rst

diff --git a/Doc/whatsnew/3.8.rst b/Doc/whatsnew/3.8.rst
index c53bffb08edea..422a6497e0a2e 100644
--- a/Doc/whatsnew/3.8.rst
+++ b/Doc/whatsnew/3.8.rst
@@ -183,7 +183,7 @@ keywords, the parameters names remain available for use in ``**kwargs``::
 
 This greatly simplifies the implementation of functions and methods
 that need to accept arbitrary keyword arguments.  For example, here
-is an except from code in the :mod:`collections` module::
+is an excerpt from code in the :mod:`collections` module::
 
   class Counter(dict):
 
@@ -215,7 +215,7 @@ subdirectories).
 Debug build uses the same ABI as release build
 -----------------------------------------------
 
-Python now uses the same ABI whether it built in release or debug mode. On
+Python now uses the same ABI whether it's built in release or debug mode. On
 Unix, when Python is built in debug mode, it is now possible to load C
 extensions built in release mode and C extensions built using the stable ABI.
 
@@ -567,7 +567,7 @@ The :func:`ast.parse` function has some new flags:
   comments" (returned for function definition AST nodes);
 
 * ``feature_version=(3, N)`` allows specifying an earlier Python 3
-  version.  (For example, ``feature_version=(3, 4)`` will treat
+  version. (For example, ``feature_version=(3, 4)`` will treat
   ``async`` and ``await`` as non-reserved words.)
 
 New function :func:`ast.get_source_segment` returns the source code
@@ -1267,7 +1267,7 @@ Build and C API Changes
     without calling functions. This API is now installed by ``make install``.
 
   (Contributed by Victor Stinner in :issue:`35134` and :issue:`35081`,
-  work initiated by Eric Snow in Python 3.7)
+  work initiated by Eric Snow in Python 3.7.)
 
 * Some macros have been converted to static inline functions: parameter types
   and return type are well defined, they don't have issues specific to macros,
@@ -1361,12 +1361,11 @@ Deprecated
 
   Implementations of these methods have been ignoring their *index* parameter,
   and returning the next item instead.
-
   (Contributed by Berker Peksag in :issue:`9372`.)
 
 * The :class:`typing.NamedTuple` class has deprecated the ``_field_types``
   attribute in favor of the ``__annotations__`` attribute which has the same
-  information.  (Contributed by Raymond Hettinger in :issue:`36320`.)
+  information. (Contributed by Raymond Hettinger in :issue:`36320`.)
 
 * :mod:`ast` classes ``Num``, ``Str``, ``Bytes``, ``NameConstant`` and
   ``Ellipsis`` are considered deprecated and will be removed in future Python
@@ -1394,7 +1393,6 @@ Deprecated
   parameter of functions :func:`~gettext.translation` and
   :func:`~gettext.install` are also deprecated, since they are only used for
   for the ``l*gettext()`` functions.
-
   (Contributed by Serhiy Storchaka in :issue:`33710`.)
 
 * The :meth:`~threading.Thread.isAlive()` method of :class:`threading.Thread` has been deprecated.
@@ -1477,11 +1475,11 @@ The following features and APIs have been removed from Python 3.8:
 
 * The ``bufsize`` keyword argument of :func:`fileinput.input` and
   :func:`fileinput.FileInput` which was ignored and deprecated since Python 3.6
-  has been removed. :issue:`36952` (Contributed by Matthias Bussonnier)
+  has been removed. :issue:`36952` (Contributed by Matthias Bussonnier.)
 
 * The functions :func:`sys.set_coroutine_wrapper` and
   :func:`sys.get_coroutine_wrapper` deprecated in Python 3.7 have been removed;
-  :issue:`36933` (Contributed by Matthias Bussonnier)
+  :issue:`36933` (Contributed by Matthias Bussonnier.)
 
 
 Porting to Python 3.8
@@ -1507,9 +1505,9 @@ Changes in Python behavior
   (Contributed by Serhiy Storchaka in :issue:`34850`.)
 
 * The CPython interpreter can swallow exceptions in some circumstances.
-  In Python 3.8 this happens in less cases.  In particular, exceptions
+  In Python 3.8 this happens in fewer cases.  In particular, exceptions
   raised when getting the attribute from the type dictionary are no longer
-  ignored.  (Contributed by Serhiy Storchaka in :issue:`35459`.)
+  ignored. (Contributed by Serhiy Storchaka in :issue:`35459`.)
 
 * Removed ``__str__`` implementations from builtin types :class:`bool`,
   :class:`int`, :class:`float`, :class:`complex` and few classes from
@@ -1604,7 +1602,7 @@ Changes in the Python API
   16 KiB to 1 MiB.
 
 * The ``PyGC_Head`` struct has changed completely.  All code that touched the
-  struct member should be rewritten.  (See :issue:`33597`)
+  struct member should be rewritten.  (See :issue:`33597`.)
 
 * The ``PyInterpreterState`` struct has been moved into the "internal"
   header files (specifically Include/internal/pycore_pystate.h).  An


From webhook-mailer at python.org  Mon Oct 14 00:18:08 2019
From: webhook-mailer at python.org (Pablo Galindo)
Date: Mon, 14 Oct 2019 04:18:08 -0000
Subject: [Python-checkins] bpo-38469: Handle named expression scope with
 global/nonlocal keywords (GH-16755)
Message-ID: <mailman.2.1571026690.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/fd5c414880b2e05720b9cf14ab0b0d7ae2b7d925
commit: fd5c414880b2e05720b9cf14ab0b0d7ae2b7d925
branch: master
author: Pablo Galindo <Pablogsal at gmail.com>
committer: GitHub <noreply at github.com>
date: 2019-10-14T05:18:05+01:00
summary:

bpo-38469: Handle named expression scope with global/nonlocal keywords (GH-16755)

The symbol table handing of PEP572's assignment expressions is not resolving correctly the scope of some variables in presence of global/nonlocal keywords in conjunction with comprehensions.

files:
A Misc/NEWS.d/next/Core and Builtins/2019-10-13-23-41-38.bpo-38469.9kmuQj.rst
M Lib/test/test_named_expressions.py
M Python/symtable.c

diff --git a/Lib/test/test_named_expressions.py b/Lib/test/test_named_expressions.py
index b1027ce78006d..01e26c8dfaf25 100644
--- a/Lib/test/test_named_expressions.py
+++ b/Lib/test/test_named_expressions.py
@@ -1,6 +1,7 @@
 import os
 import unittest
 
+GLOBAL_VAR = None
 
 class NamedExpressionInvalidTest(unittest.TestCase):
 
@@ -470,5 +471,49 @@ def test_named_expression_variable_reuse_in_comprehensions(self):
                 self.assertEqual(ns["x"], 2)
                 self.assertEqual(ns["result"], [0, 1, 2])
 
+    def test_named_expression_global_scope(self):
+        sentinel = object()
+        global GLOBAL_VAR
+        def f():
+            global GLOBAL_VAR
+            [GLOBAL_VAR := sentinel for _ in range(1)]
+            self.assertEqual(GLOBAL_VAR, sentinel)
+        try:
+            f()
+            self.assertEqual(GLOBAL_VAR, sentinel)
+        finally:
+            GLOBAL_VAR = None
+
+    def test_named_expression_global_scope_no_global_keyword(self):
+        sentinel = object()
+        def f():
+            GLOBAL_VAR = None
+            [GLOBAL_VAR := sentinel for _ in range(1)]
+            self.assertEqual(GLOBAL_VAR, sentinel)
+        f()
+        self.assertEqual(GLOBAL_VAR, None)
+
+    def test_named_expression_nonlocal_scope(self):
+        sentinel = object()
+        def f():
+            nonlocal_var = None
+            def g():
+                nonlocal nonlocal_var
+                [nonlocal_var := sentinel for _ in range(1)]
+            g()
+            self.assertEqual(nonlocal_var, sentinel)
+        f()
+
+    def test_named_expression_nonlocal_scope_no_nonlocal_keyword(self):
+        sentinel = object()
+        def f():
+            nonlocal_var = None
+            def g():
+                [nonlocal_var := sentinel for _ in range(1)]
+            g()
+            self.assertEqual(nonlocal_var, None)
+        f()
+
+
 if __name__ == "__main__":
     unittest.main()
diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-10-13-23-41-38.bpo-38469.9kmuQj.rst b/Misc/NEWS.d/next/Core and Builtins/2019-10-13-23-41-38.bpo-38469.9kmuQj.rst
new file mode 100644
index 0000000000000..328a1b70825af
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2019-10-13-23-41-38.bpo-38469.9kmuQj.rst	
@@ -0,0 +1,2 @@
+Fixed a bug where the scope of named expressions was not being resolved
+correctly in the presence of the *global* keyword. Patch by Pablo Galindo.
diff --git a/Python/symtable.c b/Python/symtable.c
index f2453db69dd7d..b8713588b9a91 100644
--- a/Python/symtable.c
+++ b/Python/symtable.c
@@ -1467,10 +1467,16 @@ symtable_extend_namedexpr_scope(struct symtable *st, expr_ty e)
             continue;
         }
 
-        /* If we find a FunctionBlock entry, add as NONLOCAL/LOCAL */
+        /* If we find a FunctionBlock entry, add as GLOBAL/LOCAL or NONLOCAL/LOCAL */
         if (ste->ste_type == FunctionBlock) {
-            if (!symtable_add_def(st, target_name, DEF_NONLOCAL))
-                VISIT_QUIT(st, 0);
+            long target_in_scope = _PyST_GetSymbol(ste, target_name);
+            if (target_in_scope & DEF_GLOBAL) {
+                if (!symtable_add_def(st, target_name, DEF_GLOBAL))
+                    VISIT_QUIT(st, 0);
+            } else {
+                if (!symtable_add_def(st, target_name, DEF_NONLOCAL))
+                    VISIT_QUIT(st, 0);
+            }
             if (!symtable_record_directive(st, target_name, e->lineno, e->col_offset))
                 VISIT_QUIT(st, 0);
 


From webhook-mailer at python.org  Mon Oct 14 00:31:16 2019
From: webhook-mailer at python.org (Raymond Hettinger)
Date: Mon, 14 Oct 2019 04:31:16 -0000
Subject: [Python-checkins] bpo-38461 and bpo-38463: Minor fixes to Whatsnew
 3.8 (GH-16761)
Message-ID: <mailman.3.1571027477.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/61a6db5e79921b89b9e2a154990f01f5f3150213
commit: 61a6db5e79921b89b9e2a154990f01f5f3150213
branch: master
author: Raymond Hettinger <rhettinger at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-13T21:31:12-07:00
summary:

bpo-38461 and bpo-38463: Minor fixes to Whatsnew 3.8 (GH-16761)

* bpo-38461: ncurses misspelled as curses

* bpo-38463: Fix broken link

files:
M Doc/whatsnew/3.8.rst

diff --git a/Doc/whatsnew/3.8.rst b/Doc/whatsnew/3.8.rst
index 422a6497e0a2e..a3adf96188243 100644
--- a/Doc/whatsnew/3.8.rst
+++ b/Doc/whatsnew/3.8.rst
@@ -599,6 +599,7 @@ marked with the ``CO_COROUTINE`` flag may then be returned.
 
 (Contributed by Matthias Bussonnier in :issue:`34616`)
 
+
 collections
 -----------
 
@@ -610,6 +611,14 @@ to cast the result to the desired type: ``OrderedDict(nt._asdict())``.
 (Contributed by Raymond Hettinger in :issue:`35864`.)
 
 
+curses
+-------
+
+Added a new variable holding structured version information for the
+underlying ncurses library: :data:`~curses.ncurses_version`.
+(Contributed by Serhiy Storchaka in :issue:`31680`.)
+
+
 ctypes
 ------
 
@@ -826,14 +835,6 @@ non-existent directory.
 symlinks and directory junctions.
 
 
-ncurses
--------
-
-Added a new variable holding structured version information for the
-underlying ncurses library: :data:`~curses.ncurses_version`.
-(Contributed by Serhiy Storchaka in :issue:`31680`.)
-
-
 pathlib
 -------
 
@@ -918,7 +919,7 @@ The :func:`socket.if_nameindex()`, :func:`socket.if_nametoindex()`, and
 ssl
 ---
 
-Added :attr:`SSLContext.post_handshake_auth` to enable and
+Added :attr:`ssl.SSLContext.post_handshake_auth` to enable and
 :meth:`ssl.SSLSocket.verify_client_post_handshake` to initiate TLS 1.3
 post-handshake authentication.
 (Contributed by Christian Heimes in :issue:`34670`.)


From webhook-mailer at python.org  Mon Oct 14 00:54:13 2019
From: webhook-mailer at python.org (Raymond Hettinger)
Date: Mon, 14 Oct 2019 04:54:13 -0000
Subject: [Python-checkins] bpo-38461 and bpo-38463: Minor fixes to Whatsnew
 3.8 (GH-16761) (GH-16762)
Message-ID: <mailman.4.1571028854.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/c09379bd6ebdd896547008c641e8899a1e36aa42
commit: c09379bd6ebdd896547008c641e8899a1e36aa42
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Raymond Hettinger <rhettinger at users.noreply.github.com>
date: 2019-10-13T21:54:05-07:00
summary:

bpo-38461 and bpo-38463: Minor fixes to Whatsnew 3.8 (GH-16761) (GH-16762)

* bpo-38461: ncurses misspelled as curses

* bpo-38463: Fix broken link
(cherry picked from commit 61a6db5e79921b89b9e2a154990f01f5f3150213)

Co-authored-by: Raymond Hettinger <rhettinger at users.noreply.github.com>

files:
M Doc/whatsnew/3.8.rst

diff --git a/Doc/whatsnew/3.8.rst b/Doc/whatsnew/3.8.rst
index 422a6497e0a2e..a3adf96188243 100644
--- a/Doc/whatsnew/3.8.rst
+++ b/Doc/whatsnew/3.8.rst
@@ -599,6 +599,7 @@ marked with the ``CO_COROUTINE`` flag may then be returned.
 
 (Contributed by Matthias Bussonnier in :issue:`34616`)
 
+
 collections
 -----------
 
@@ -610,6 +611,14 @@ to cast the result to the desired type: ``OrderedDict(nt._asdict())``.
 (Contributed by Raymond Hettinger in :issue:`35864`.)
 
 
+curses
+-------
+
+Added a new variable holding structured version information for the
+underlying ncurses library: :data:`~curses.ncurses_version`.
+(Contributed by Serhiy Storchaka in :issue:`31680`.)
+
+
 ctypes
 ------
 
@@ -826,14 +835,6 @@ non-existent directory.
 symlinks and directory junctions.
 
 
-ncurses
--------
-
-Added a new variable holding structured version information for the
-underlying ncurses library: :data:`~curses.ncurses_version`.
-(Contributed by Serhiy Storchaka in :issue:`31680`.)
-
-
 pathlib
 -------
 
@@ -918,7 +919,7 @@ The :func:`socket.if_nameindex()`, :func:`socket.if_nametoindex()`, and
 ssl
 ---
 
-Added :attr:`SSLContext.post_handshake_auth` to enable and
+Added :attr:`ssl.SSLContext.post_handshake_auth` to enable and
 :meth:`ssl.SSLSocket.verify_client_post_handshake` to initiate TLS 1.3
 post-handshake authentication.
 (Contributed by Christian Heimes in :issue:`34670`.)


From webhook-mailer at python.org  Mon Oct 14 02:32:09 2019
From: webhook-mailer at python.org (Raymond Hettinger)
Date: Mon, 14 Oct 2019 06:32:09 -0000
Subject: [Python-checkins] bpo-37759: Add examples for the new typing
 features (GH-16763)
Message-ID: <mailman.5.1571034730.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/a3291531ea51455cfa5989119db496267425323c
commit: a3291531ea51455cfa5989119db496267425323c
branch: master
author: Raymond Hettinger <rhettinger at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-13T23:32:03-07:00
summary:

bpo-37759: Add examples for the new typing features (GH-16763)

files:
M Doc/whatsnew/3.8.rst

diff --git a/Doc/whatsnew/3.8.rst b/Doc/whatsnew/3.8.rst
index a3adf96188243..0c59ef8015a58 100644
--- a/Doc/whatsnew/3.8.rst
+++ b/Doc/whatsnew/3.8.rst
@@ -1048,17 +1048,33 @@ typing
 
 The :mod:`typing` module incorporates several new features:
 
-* Protocol definitions.  See :pep:`544`, :class:`typing.Protocol` and
-  :func:`typing.runtime_checkable`.  Simple ABCs like
-  :class:`typing.SupportsInt` are now ``Protocol`` subclasses.
-
 * A dictionary type with per-key types.  See :pep:`589` and
   :class:`typing.TypedDict`.
+  TypedDict uses only string keys.  By default, every key is required
+  to be present. Specify "total=False" to allow keys to be optional::
+
+      class Location(TypedDict, total=False):
+          lat_long: tuple
+          grid_square: str
+          xy_coordinate: tuple
 
 * Literal types.  See :pep:`586` and :class:`typing.Literal`.
+  Literal types indicate that a parameter or return value
+  is constrained to one or more specific literal values::
+
+      def get_status(port: int) -> Literal['connected', 'disconnected']:
+          ...
 
 * "Final" variables, functions, methods and classes.  See :pep:`591`,
   :class:`typing.Final` and :func:`typing.final`.
+  The final qualifier instructs a static type checker to restrict
+  subclassing, overriding, or reassignment::
+
+      pi: Final[float] = 3.1415926536
+
+* Protocol definitions.  See :pep:`544`, :class:`typing.Protocol` and
+  :func:`typing.runtime_checkable`.  Simple ABCs like
+  :class:`typing.SupportsInt` are now ``Protocol`` subclasses.
 
 * New protocol class :class:`typing.SupportsIndex`.
 
@@ -1527,7 +1543,7 @@ Changes in Python behavior
   terminate the current thread if called while the interpreter is
   finalizing, making them consistent with :c:func:`PyEval_RestoreThread`,
   :c:func:`Py_END_ALLOW_THREADS`, and :c:func:`PyGILState_Ensure`. If this
-  behaviour is not desired, guard the call by checking :c:func:`_Py_IsFinalizing`
+  behavior is not desired, guard the call by checking :c:func:`_Py_IsFinalizing`
   or :c:func:`sys.is_finalizing`.
 
 Changes in the Python API


From webhook-mailer at python.org  Mon Oct 14 02:45:51 2019
From: webhook-mailer at python.org (Raymond Hettinger)
Date: Mon, 14 Oct 2019 06:45:51 -0000
Subject: [Python-checkins] bpo-37759: Add examples for the new typing
 features (GH-16763) (GH-16764)
Message-ID: <mailman.6.1571035552.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/3ac152948d63f52267d312be33306067d3847e2a
commit: 3ac152948d63f52267d312be33306067d3847e2a
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Raymond Hettinger <rhettinger at users.noreply.github.com>
date: 2019-10-13T23:45:45-07:00
summary:

bpo-37759: Add examples for the new typing features (GH-16763) (GH-16764)

(cherry picked from commit a3291531ea51455cfa5989119db496267425323c)

Co-authored-by: Raymond Hettinger <rhettinger at users.noreply.github.com>

files:
M Doc/whatsnew/3.8.rst

diff --git a/Doc/whatsnew/3.8.rst b/Doc/whatsnew/3.8.rst
index a3adf96188243..0c59ef8015a58 100644
--- a/Doc/whatsnew/3.8.rst
+++ b/Doc/whatsnew/3.8.rst
@@ -1048,17 +1048,33 @@ typing
 
 The :mod:`typing` module incorporates several new features:
 
-* Protocol definitions.  See :pep:`544`, :class:`typing.Protocol` and
-  :func:`typing.runtime_checkable`.  Simple ABCs like
-  :class:`typing.SupportsInt` are now ``Protocol`` subclasses.
-
 * A dictionary type with per-key types.  See :pep:`589` and
   :class:`typing.TypedDict`.
+  TypedDict uses only string keys.  By default, every key is required
+  to be present. Specify "total=False" to allow keys to be optional::
+
+      class Location(TypedDict, total=False):
+          lat_long: tuple
+          grid_square: str
+          xy_coordinate: tuple
 
 * Literal types.  See :pep:`586` and :class:`typing.Literal`.
+  Literal types indicate that a parameter or return value
+  is constrained to one or more specific literal values::
+
+      def get_status(port: int) -> Literal['connected', 'disconnected']:
+          ...
 
 * "Final" variables, functions, methods and classes.  See :pep:`591`,
   :class:`typing.Final` and :func:`typing.final`.
+  The final qualifier instructs a static type checker to restrict
+  subclassing, overriding, or reassignment::
+
+      pi: Final[float] = 3.1415926536
+
+* Protocol definitions.  See :pep:`544`, :class:`typing.Protocol` and
+  :func:`typing.runtime_checkable`.  Simple ABCs like
+  :class:`typing.SupportsInt` are now ``Protocol`` subclasses.
 
 * New protocol class :class:`typing.SupportsIndex`.
 
@@ -1527,7 +1543,7 @@ Changes in Python behavior
   terminate the current thread if called while the interpreter is
   finalizing, making them consistent with :c:func:`PyEval_RestoreThread`,
   :c:func:`Py_END_ALLOW_THREADS`, and :c:func:`PyGILState_Ensure`. If this
-  behaviour is not desired, guard the call by checking :c:func:`_Py_IsFinalizing`
+  behavior is not desired, guard the call by checking :c:func:`_Py_IsFinalizing`
   or :c:func:`sys.is_finalizing`.
 
 Changes in the Python API


From webhook-mailer at python.org  Mon Oct 14 04:26:51 2019
From: webhook-mailer at python.org (Ned Deily)
Date: Mon, 14 Oct 2019 08:26:51 -0000
Subject: [Python-checkins] Update macOS installer display files for 3.9.0a1
 (GH-16765)
Message-ID: <mailman.7.1571041612.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/9cb51f4e20033f5fd4fed46036e347f263bb6d5b
commit: 9cb51f4e20033f5fd4fed46036e347f263bb6d5b
branch: master
author: Ned Deily <nad at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-14T04:26:45-04:00
summary:

Update macOS installer display files for 3.9.0a1 (GH-16765)

files:
M Mac/BuildScript/resources/License.rtf
M Mac/BuildScript/resources/ReadMe.rtf
M Mac/BuildScript/resources/Welcome.rtf

diff --git a/Mac/BuildScript/resources/License.rtf b/Mac/BuildScript/resources/License.rtf
index 20982a46cc36a..86e54f6662a20 100644
--- a/Mac/BuildScript/resources/License.rtf
+++ b/Mac/BuildScript/resources/License.rtf
@@ -1,16 +1,17 @@
-{\rtf1\ansi\ansicpg1252\cocoartf1343\cocoasubrtf160
-{\fonttbl\f0\fswiss\fcharset0 Helvetica;\f1\fmodern\fcharset0 CourierNewPS-BoldMT;\f2\fmodern\fcharset0 CourierNewPSMT;
-}
+{\rtf1\ansi\ansicpg1252\cocoartf1671\cocoasubrtf600
+{\fonttbl\f0\fswiss\fcharset0 Helvetica-Bold;\f1\fswiss\fcharset0 Helvetica;\f2\fmodern\fcharset0 CourierNewPS-BoldMT;
+\f3\fmodern\fcharset0 CourierNewPSMT;}
 {\colortbl;\red255\green255\blue255;}
+{\*\expandedcolortbl;;}
 \margl1440\margr1440\vieww14620\viewh13380\viewkind0
-\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural
+\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural\partightenfactor0
 
 \f0\b\fs36 \cf0 \ul \ulc0 HISTORY AND LICENSE\
 
 \fs24 \
 HISTORY OF THE SOFTWARE\
 
-\b0 \ulnone \
+\f1\b0 \ulnone \
 Python was created in the early 1990s by Guido van Rossum at Stichting Mathematisch Centrum (CWI, see http://www.cwi.nl) in the Netherlands as a successor of a language called ABC. Guido remains Python's principal author, although it includes many contributions from others.\
 \
 In 1995, Guido continued his work on Python at the Corporation for National Research Initiatives (CNRI, see http://www.cnri.reston.va.us) in Reston, Virginia where he released several versions of the software.\
@@ -20,10 +21,10 @@ In May 2000, Guido and the Python core development team moved to BeOpen.com to f
 All Python releases are Open Source (see http://www.opensource.org for the Open Source Definition). Historically, most, but not all, Python releases have also been GPL-compatible; the table below summarizes the various releases.\
 \
 
-\f1\b Release         Derived     Year        Owner       GPL-\
+\f2\b Release         Derived     Year        Owner       GPL-\
                 from                                compatible?\
 
-\f2\b0 \
+\f3\b0 \
 0.9.0 thru 1.2  n/a         1991-1995   CWI         yes\
 1.3 thru 1.5.2  1.2         1995-1999   CNRI        yes\
 1.6             1.5.2       2000        CNRI        no\
@@ -36,25 +37,25 @@ All Python releases are Open Source (see http://www.opensource.org for the Open
 2.1.3           2.1.2       2002        PSF         yes\
 2.2 and above   2.1.1       2001-now    PSF         yes\
 
-\f0 \
+\f1 \
 
-\b Note:  
-\b0 GPL-compatible doesn't mean that we're distributing Python under the GPL. All Python licenses, unlike the GPL, let you distribute a modified version without making your changes open source. The GPL-compatible licenses make it possible to combine Python with other software that is released under the GPL; the others don't.\
+\f0\b Note:  
+\f1\b0 GPL-compatible doesn't mean that we're distributing Python under the GPL. All Python licenses, unlike the GPL, let you distribute a modified version without making your changes open source. The GPL-compatible licenses make it possible to combine Python with other software that is released under the GPL; the others don't.\
 \
 Thanks to the many outside volunteers who have worked under Guido's direction to make these releases possible.\
 \
 \
 
-\b \ul TERMS AND CONDITIONS FOR ACCESSING OR OTHERWISE USING PYTHON\
+\f0\b \ul TERMS AND CONDITIONS FOR ACCESSING OR OTHERWISE USING PYTHON\
 
-\b0 \ulnone \
+\f1\b0 \ulnone \
 
-\b PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2\
+\f0\b PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2\
 
-\b0 \
+\f1\b0 \
 1. This LICENSE AGREEMENT is between the Python Software Foundation ("PSF"), and the Individual or Organization ("Licensee") accessing and otherwise using this software ("Python") in source or binary form and its associated documentation.\
 \
-2. Subject to the terms and conditions of this License Agreement, PSF hereby grants Licensee a nonexclusive, royalty-free, world-wide license to reproduce, analyze, test, perform and/or display publicly, prepare derivative works, distribute, and otherwise use Python alone or in any derivative version, provided, however, that PSF's License Agreement and PSF's notice of copyright, i.e., "Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016 Python Software Foundation; All Rights Reserved" are retained in Python alone or in any derivative version prepared by Licensee.\
+2. Subject to the terms and conditions of this License Agreement, PSF hereby grants Licensee a nonexclusive, royalty-free, world-wide license to reproduce, analyze, test, perform and/or display publicly, prepare derivative works, distribute, and otherwise use Python alone or in any derivative version, provided, however, that PSF's License Agreement and PSF's notice of copyright, i.e., "Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019 Python Software Foundation; All Rights Reserved" are retained in Python alone or in any derivative version prepared by Licensee.\
 \
 3. In the event Licensee prepares a derivative work that is based on or incorporates Python or any part thereof, and wants to make the derivative work available to others as provided herein, then Licensee hereby agrees to include in any such work a brief summary of the changes made to Python.\
 \
@@ -70,9 +71,9 @@ Thanks to the many outside volunteers who have worked under Guido's direction to
 \
 \
 
-\b BEOPEN.COM LICENSE AGREEMENT FOR PYTHON 2.0\
+\f0\b BEOPEN.COM LICENSE AGREEMENT FOR PYTHON 2.0\
 
-\b0 \
+\f1\b0 \
 BEOPEN PYTHON OPEN SOURCE LICENSE AGREEMENT VERSION 1\
 \
 1. This LICENSE AGREEMENT is between BeOpen.com ("BeOpen"), having an office at 160 Saratoga Avenue, Santa Clara, CA 95051, and the Individual or Organization ("Licensee") accessing and otherwise using this software in source or binary form and its associated documentation ("the Software").\
@@ -91,9 +92,9 @@ BEOPEN PYTHON OPEN SOURCE LICENSE AGREEMENT VERSION 1\
 \
 \
 
-\b CNRI LICENSE AGREEMENT FOR PYTHON 1.6.1\
+\f0\b CNRI LICENSE AGREEMENT FOR PYTHON 1.6.1\
 
-\b0 \
+\f1\b0 \
 1. This LICENSE AGREEMENT is between the Corporation for National Research Initiatives, having an office at 1895 Preston White Drive, Reston, VA 20191 ("CNRI"), and the Individual or Organization ("Licensee") accessing and otherwise using Python 1.6.1 software in source or binary form and its associated documentation.\
 \
 2. Subject to the terms and conditions of this License Agreement, CNRI hereby grants Licensee a nonexclusive, royalty-free, world-wide license to reproduce, analyze, test, perform and/or display publicly, prepare derivative works, distribute, and otherwise use Python 1.6.1 alone or in any derivative version, provided, however, that CNRI's License Agreement and CNRI's notice of copyright, i.e., "Copyright (c) 1995-2001 Corporation for National Research Initiatives; All Rights Reserved" are retained in Python 1.6.1 alone or in any derivative version prepared by Licensee. Alternately, in lieu of CNRI's License Agreement, Licensee may substitute the following text (omitting the quotes): "Python 1.6.1 is made available subject to the terms and conditions in CNRI's License Agreement. This Agreement together with Python 1.6.1 may be located on the Internet using the following unique, persistent identifier (known as a handle): 1895.22/1013. This Agreement may also be obtained from a proxy server on the Internet using the following URL: http://hdl.handle.net/1895.22/1013".\
@@ -114,9 +115,9 @@ BEOPEN PYTHON OPEN SOURCE LICENSE AGREEMENT VERSION 1\
 \
 \
 
-\b CWI LICENSE AGREEMENT FOR PYTHON 0.9.0 THROUGH 1.2\
+\f0\b CWI LICENSE AGREEMENT FOR PYTHON 0.9.0 THROUGH 1.2\
 
-\b0 \
+\f1\b0 \
 Copyright (c) 1991 - 1995, Stichting Mathematisch Centrum Amsterdam, The Netherlands. All rights reserved.\
 \
 Permission to use, copy, modify, and distribute this software and its documentation for any purpose and without fee is hereby granted, provided that the above copyright notice appear in all copies and that both that copyright notice and this permission notice appear in supporting documentation, and that the name of Stichting Mathematisch Centrum or CWI not be used in advertising or publicity pertaining to distribution of the software without specific, written prior permission.\
@@ -125,16 +126,16 @@ STICHTING MATHEMATISCH CENTRUM DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFT
 \
 \
 
-\b \ul LICENSES AND ACKNOWLEDGEMENTS FOR INCORPORATED SOFTWARE\
+\f0\b \ul LICENSES AND ACKNOWLEDGEMENTS FOR INCORPORATED SOFTWARE\
 
-\b0 \ulnone \
+\f1\b0 \ulnone \
 This installer incorporates portions of the following third-party software:\
 \
 
-\f2 $THIRD_PARTY_LIBS\
+\f3 $THIRD_PARTY_LIBS\
 \
 
-\f0 For licenses and acknowledgements for these and other third-party software incorporated in this Python distribution, please refer to the on-line documentation {\field{\*\fldinst{HYPERLINK "https://docs.python.org/$VERSION/license.html#licenses-and-acknowledgements-for-incorporated-software"}}{\fldrslt here}}.\
+\f1 For licenses and acknowledgements for these and other third-party software incorporated in this Python distribution, please refer to the on-line documentation {\field{\*\fldinst{HYPERLINK "https://docs.python.org/$VERSION/license.html#licenses-and-acknowledgements-for-incorporated-software"}}{\fldrslt here}}.\
 \
 \
 \
diff --git a/Mac/BuildScript/resources/ReadMe.rtf b/Mac/BuildScript/resources/ReadMe.rtf
index ec83750c6e691..fc5739aee540a 100644
--- a/Mac/BuildScript/resources/ReadMe.rtf
+++ b/Mac/BuildScript/resources/ReadMe.rtf
@@ -1,4 +1,4 @@
-{\rtf1\ansi\ansicpg1252\cocoartf1671\cocoasubrtf500
+{\rtf1\ansi\ansicpg1252\cocoartf1671\cocoasubrtf600
 {\fonttbl\f0\fswiss\fcharset0 Helvetica;\f1\fswiss\fcharset0 Helvetica-Bold;\f2\fswiss\fcharset0 Helvetica-Oblique;
 \f3\fmodern\fcharset0 CourierNewPSMT;}
 {\colortbl;\red255\green255\blue255;}
@@ -11,7 +11,7 @@
 \pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\partightenfactor0
 
 \f1\b \cf0 NOTE: 
-\f0\b0 This is a beta test preview of Python 3.8.0, the next feature release of Python 3.8.  It is not intended for production use.\
+\f0\b0 This is an alpha test preview of Python 3.9.0, the next feature release of Python 3.  It is not intended for production use.\
 \pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural\partightenfactor0
 \cf0 \
 \pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural\partightenfactor0
@@ -26,25 +26,37 @@ This package includes its own private copy of OpenSSL 1.1.1.   The trust certifi
 \f0\i0  command line utility are not used as defaults by the Python 
 \f3 ssl
 \f0  module.  A sample command script is included in 
-\f3 /Applications/Python 3.8
+\f3 /Applications/Python 3.9
 \f0  to install a curated bundle of default root certificates from the third-party 
 \f3 certifi
-\f0  package ({\field{\*\fldinst{HYPERLINK "https://pypi.org/project/certifi/"}}{\fldrslt https://pypi.org/project/certifi/}}).  If you choose to use 
-\f3 certifi
-\f0 , you should consider subscribing to the{\field{\*\fldinst{HYPERLINK "https://certifi.io/en/latest/"}}{\fldrslt  project's email update service}} to be notified when the certificate bundle is updated.\
+\f0  package ({\field{\*\fldinst{HYPERLINK "https://pypi.org/project/certifi/"}}{\fldrslt https://pypi.org/project/certifi/}}).  Double-click on 
+\f3 Install Certificates
+\f0  to run it.\
 \
 The bundled 
 \f3 pip
-\f0  included with this installer has its own default certificate store for verifying download connections.\
+\f0  has its own default certificate store for verifying download connections.\
 \
 
 \f1\b \ul Using IDLE or other Tk applications 
 \f0\b0 \ulnone \
 \
 This package includes its own private version of Tcl/Tk 8.6. It does not use any system-supplied or third-party supplied versions of Tcl/Tk.\
+\
+Due to new security checks on macOS 10.15 Catalina, when launching IDLE macOS may open a window with a message 
+\f1\b "Python" would like to access files in your Documents folder
+\f0\b0 .  This is normal as IDLE uses your 
+\f1\b Documents
+\f0\b0  folder as its default when opening and saving files; you can still choose other locations in the 
+\f1\b Open
+\f0\b0  and 
+\f1\b Save
+\f0\b0  file dialog windows.  Click on the 
+\f1\b OK
+\f0\b0  button to proceed.\
+\
 
-\f1\b \ul \
-Other changes\
+\f1\b \ul Other changes\
 
 \f0\b0 \ulnone \
 For other changes in this release, see the 
@@ -52,34 +64,4 @@ For other changes in this release, see the
 \f0\i0  section in the {\field{\*\fldinst{HYPERLINK "https://www.python.org/doc/"}}{\fldrslt Documentation Set}} for this release and its 
 \f2\i Release Notes
 \f0\i0  link at {\field{\*\fldinst{HYPERLINK "https://www.python.org/downloads/"}}{\fldrslt https://www.python.org/downloads/}}.\
-
-\f1\b \ul \
-Python 3 and Python 2 Co-existence\
-
-\f0\b0 \ulnone \
-Python.org Python $VERSION and 2.7.x versions can both be installed on your system and will not conflict. Command names for Python 3 contain a 3 in them, 
-\f3 python3
-\f0  (or
-\f3  python$VERSION
-\f0 ), 
-\f3 idle3
-\f0  (or i
-\f3 dle$VERSION
-\f0 ), 
-\f3 pip3
-\f0  (or 
-\f3 pip$VERSION
-\f0 ), etc.  Python 2.7 command names contain a 2 or no digit: 
-\f3 python2
-\f0  (or 
-\f3 python2.7
-\f0  or 
-\f3 python
-\f0 ), 
-\f3 idle2
-\f0  (or 
-\f3 idle2.7
-\f0  or 
-\f3 idle
-\f0 ), etc.\
 }
\ No newline at end of file
diff --git a/Mac/BuildScript/resources/Welcome.rtf b/Mac/BuildScript/resources/Welcome.rtf
index 053084d0425b3..bf6aebe92483f 100644
--- a/Mac/BuildScript/resources/Welcome.rtf
+++ b/Mac/BuildScript/resources/Welcome.rtf
@@ -1,4 +1,4 @@
-{\rtf1\ansi\ansicpg1252\cocoartf1671\cocoasubrtf500
+{\rtf1\ansi\ansicpg1252\cocoartf1671\cocoasubrtf600
 \cocoascreenfonts1{\fonttbl\f0\fswiss\fcharset0 Helvetica;\f1\fswiss\fcharset0 Helvetica-Bold;\f2\fmodern\fcharset0 CourierNewPSMT;
 }
 {\colortbl;\red255\green255\blue255;}
@@ -14,15 +14,17 @@
 \
 
 \f1\b Python for macOS
-\f0\b0  consists of the Python programming language interpreter, plus a set of programs to allow easy access to it for macOS users including an integrated development environment 
+\f0\b0  consists of the {\field{\*\fldinst{HYPERLINK "https://www.python.org"}}{\fldrslt Python}} programming language interpreter and its batteries-included standard library to allow easy access to macOS features.  It also includes the Python integrated development environment, 
 \f1\b IDLE
-\f0\b0 .\
+\f0\b0 .  You can also use the included 
+\f1\b pip
+\f0\b0  to download and install third-party packages from the {\field{\*\fldinst{HYPERLINK "https://pypi.org"}}{\fldrslt Python Package Index}}. \
 \
 At the end of this install, click on 
 \f2 Install Certificates
-\f0  for SSL root certificates\
+\f0  to install a set of current SSL root certificates.\
 \
 
 \f1\b NOTE: 
-\f0\b0 This is a beta test preview of Python 3.8.0, the next feature release of Python 3.8.  It is not intended for production use.\
+\f0\b0 This is an alpha test preview of Python 3.9.0, the next feature release of Python 3.  It is not intended for production use.\
 }
\ No newline at end of file


From webhook-mailer at python.org  Mon Oct 14 04:36:40 2019
From: webhook-mailer at python.org (Ned Deily)
Date: Mon, 14 Oct 2019 08:36:40 -0000
Subject: [Python-checkins] [3.8] Update macOS installer display files for
 3.8.0 (GH-16766)
Message-ID: <mailman.8.1571042201.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/ac53ba6c7c0c43100bab8bc427d54563d9b65a0e
commit: ac53ba6c7c0c43100bab8bc427d54563d9b65a0e
branch: 3.8
author: Ned Deily <nad at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-14T04:36:34-04:00
summary:

[3.8] Update macOS installer display files for 3.8.0 (GH-16766)

files:
M Mac/BuildScript/resources/License.rtf
M Mac/BuildScript/resources/ReadMe.rtf
M Mac/BuildScript/resources/Welcome.rtf

diff --git a/Mac/BuildScript/resources/License.rtf b/Mac/BuildScript/resources/License.rtf
index 20982a46cc36a..86e54f6662a20 100644
--- a/Mac/BuildScript/resources/License.rtf
+++ b/Mac/BuildScript/resources/License.rtf
@@ -1,16 +1,17 @@
-{\rtf1\ansi\ansicpg1252\cocoartf1343\cocoasubrtf160
-{\fonttbl\f0\fswiss\fcharset0 Helvetica;\f1\fmodern\fcharset0 CourierNewPS-BoldMT;\f2\fmodern\fcharset0 CourierNewPSMT;
-}
+{\rtf1\ansi\ansicpg1252\cocoartf1671\cocoasubrtf600
+{\fonttbl\f0\fswiss\fcharset0 Helvetica-Bold;\f1\fswiss\fcharset0 Helvetica;\f2\fmodern\fcharset0 CourierNewPS-BoldMT;
+\f3\fmodern\fcharset0 CourierNewPSMT;}
 {\colortbl;\red255\green255\blue255;}
+{\*\expandedcolortbl;;}
 \margl1440\margr1440\vieww14620\viewh13380\viewkind0
-\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural
+\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural\partightenfactor0
 
 \f0\b\fs36 \cf0 \ul \ulc0 HISTORY AND LICENSE\
 
 \fs24 \
 HISTORY OF THE SOFTWARE\
 
-\b0 \ulnone \
+\f1\b0 \ulnone \
 Python was created in the early 1990s by Guido van Rossum at Stichting Mathematisch Centrum (CWI, see http://www.cwi.nl) in the Netherlands as a successor of a language called ABC. Guido remains Python's principal author, although it includes many contributions from others.\
 \
 In 1995, Guido continued his work on Python at the Corporation for National Research Initiatives (CNRI, see http://www.cnri.reston.va.us) in Reston, Virginia where he released several versions of the software.\
@@ -20,10 +21,10 @@ In May 2000, Guido and the Python core development team moved to BeOpen.com to f
 All Python releases are Open Source (see http://www.opensource.org for the Open Source Definition). Historically, most, but not all, Python releases have also been GPL-compatible; the table below summarizes the various releases.\
 \
 
-\f1\b Release         Derived     Year        Owner       GPL-\
+\f2\b Release         Derived     Year        Owner       GPL-\
                 from                                compatible?\
 
-\f2\b0 \
+\f3\b0 \
 0.9.0 thru 1.2  n/a         1991-1995   CWI         yes\
 1.3 thru 1.5.2  1.2         1995-1999   CNRI        yes\
 1.6             1.5.2       2000        CNRI        no\
@@ -36,25 +37,25 @@ All Python releases are Open Source (see http://www.opensource.org for the Open
 2.1.3           2.1.2       2002        PSF         yes\
 2.2 and above   2.1.1       2001-now    PSF         yes\
 
-\f0 \
+\f1 \
 
-\b Note:  
-\b0 GPL-compatible doesn't mean that we're distributing Python under the GPL. All Python licenses, unlike the GPL, let you distribute a modified version without making your changes open source. The GPL-compatible licenses make it possible to combine Python with other software that is released under the GPL; the others don't.\
+\f0\b Note:  
+\f1\b0 GPL-compatible doesn't mean that we're distributing Python under the GPL. All Python licenses, unlike the GPL, let you distribute a modified version without making your changes open source. The GPL-compatible licenses make it possible to combine Python with other software that is released under the GPL; the others don't.\
 \
 Thanks to the many outside volunteers who have worked under Guido's direction to make these releases possible.\
 \
 \
 
-\b \ul TERMS AND CONDITIONS FOR ACCESSING OR OTHERWISE USING PYTHON\
+\f0\b \ul TERMS AND CONDITIONS FOR ACCESSING OR OTHERWISE USING PYTHON\
 
-\b0 \ulnone \
+\f1\b0 \ulnone \
 
-\b PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2\
+\f0\b PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2\
 
-\b0 \
+\f1\b0 \
 1. This LICENSE AGREEMENT is between the Python Software Foundation ("PSF"), and the Individual or Organization ("Licensee") accessing and otherwise using this software ("Python") in source or binary form and its associated documentation.\
 \
-2. Subject to the terms and conditions of this License Agreement, PSF hereby grants Licensee a nonexclusive, royalty-free, world-wide license to reproduce, analyze, test, perform and/or display publicly, prepare derivative works, distribute, and otherwise use Python alone or in any derivative version, provided, however, that PSF's License Agreement and PSF's notice of copyright, i.e., "Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016 Python Software Foundation; All Rights Reserved" are retained in Python alone or in any derivative version prepared by Licensee.\
+2. Subject to the terms and conditions of this License Agreement, PSF hereby grants Licensee a nonexclusive, royalty-free, world-wide license to reproduce, analyze, test, perform and/or display publicly, prepare derivative works, distribute, and otherwise use Python alone or in any derivative version, provided, however, that PSF's License Agreement and PSF's notice of copyright, i.e., "Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019 Python Software Foundation; All Rights Reserved" are retained in Python alone or in any derivative version prepared by Licensee.\
 \
 3. In the event Licensee prepares a derivative work that is based on or incorporates Python or any part thereof, and wants to make the derivative work available to others as provided herein, then Licensee hereby agrees to include in any such work a brief summary of the changes made to Python.\
 \
@@ -70,9 +71,9 @@ Thanks to the many outside volunteers who have worked under Guido's direction to
 \
 \
 
-\b BEOPEN.COM LICENSE AGREEMENT FOR PYTHON 2.0\
+\f0\b BEOPEN.COM LICENSE AGREEMENT FOR PYTHON 2.0\
 
-\b0 \
+\f1\b0 \
 BEOPEN PYTHON OPEN SOURCE LICENSE AGREEMENT VERSION 1\
 \
 1. This LICENSE AGREEMENT is between BeOpen.com ("BeOpen"), having an office at 160 Saratoga Avenue, Santa Clara, CA 95051, and the Individual or Organization ("Licensee") accessing and otherwise using this software in source or binary form and its associated documentation ("the Software").\
@@ -91,9 +92,9 @@ BEOPEN PYTHON OPEN SOURCE LICENSE AGREEMENT VERSION 1\
 \
 \
 
-\b CNRI LICENSE AGREEMENT FOR PYTHON 1.6.1\
+\f0\b CNRI LICENSE AGREEMENT FOR PYTHON 1.6.1\
 
-\b0 \
+\f1\b0 \
 1. This LICENSE AGREEMENT is between the Corporation for National Research Initiatives, having an office at 1895 Preston White Drive, Reston, VA 20191 ("CNRI"), and the Individual or Organization ("Licensee") accessing and otherwise using Python 1.6.1 software in source or binary form and its associated documentation.\
 \
 2. Subject to the terms and conditions of this License Agreement, CNRI hereby grants Licensee a nonexclusive, royalty-free, world-wide license to reproduce, analyze, test, perform and/or display publicly, prepare derivative works, distribute, and otherwise use Python 1.6.1 alone or in any derivative version, provided, however, that CNRI's License Agreement and CNRI's notice of copyright, i.e., "Copyright (c) 1995-2001 Corporation for National Research Initiatives; All Rights Reserved" are retained in Python 1.6.1 alone or in any derivative version prepared by Licensee. Alternately, in lieu of CNRI's License Agreement, Licensee may substitute the following text (omitting the quotes): "Python 1.6.1 is made available subject to the terms and conditions in CNRI's License Agreement. This Agreement together with Python 1.6.1 may be located on the Internet using the following unique, persistent identifier (known as a handle): 1895.22/1013. This Agreement may also be obtained from a proxy server on the Internet using the following URL: http://hdl.handle.net/1895.22/1013".\
@@ -114,9 +115,9 @@ BEOPEN PYTHON OPEN SOURCE LICENSE AGREEMENT VERSION 1\
 \
 \
 
-\b CWI LICENSE AGREEMENT FOR PYTHON 0.9.0 THROUGH 1.2\
+\f0\b CWI LICENSE AGREEMENT FOR PYTHON 0.9.0 THROUGH 1.2\
 
-\b0 \
+\f1\b0 \
 Copyright (c) 1991 - 1995, Stichting Mathematisch Centrum Amsterdam, The Netherlands. All rights reserved.\
 \
 Permission to use, copy, modify, and distribute this software and its documentation for any purpose and without fee is hereby granted, provided that the above copyright notice appear in all copies and that both that copyright notice and this permission notice appear in supporting documentation, and that the name of Stichting Mathematisch Centrum or CWI not be used in advertising or publicity pertaining to distribution of the software without specific, written prior permission.\
@@ -125,16 +126,16 @@ STICHTING MATHEMATISCH CENTRUM DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFT
 \
 \
 
-\b \ul LICENSES AND ACKNOWLEDGEMENTS FOR INCORPORATED SOFTWARE\
+\f0\b \ul LICENSES AND ACKNOWLEDGEMENTS FOR INCORPORATED SOFTWARE\
 
-\b0 \ulnone \
+\f1\b0 \ulnone \
 This installer incorporates portions of the following third-party software:\
 \
 
-\f2 $THIRD_PARTY_LIBS\
+\f3 $THIRD_PARTY_LIBS\
 \
 
-\f0 For licenses and acknowledgements for these and other third-party software incorporated in this Python distribution, please refer to the on-line documentation {\field{\*\fldinst{HYPERLINK "https://docs.python.org/$VERSION/license.html#licenses-and-acknowledgements-for-incorporated-software"}}{\fldrslt here}}.\
+\f1 For licenses and acknowledgements for these and other third-party software incorporated in this Python distribution, please refer to the on-line documentation {\field{\*\fldinst{HYPERLINK "https://docs.python.org/$VERSION/license.html#licenses-and-acknowledgements-for-incorporated-software"}}{\fldrslt here}}.\
 \
 \
 \
diff --git a/Mac/BuildScript/resources/ReadMe.rtf b/Mac/BuildScript/resources/ReadMe.rtf
index ec83750c6e691..d321fcd80c2eb 100644
--- a/Mac/BuildScript/resources/ReadMe.rtf
+++ b/Mac/BuildScript/resources/ReadMe.rtf
@@ -1,4 +1,4 @@
-{\rtf1\ansi\ansicpg1252\cocoartf1671\cocoasubrtf500
+{\rtf1\ansi\ansicpg1252\cocoartf1671\cocoasubrtf600
 {\fonttbl\f0\fswiss\fcharset0 Helvetica;\f1\fswiss\fcharset0 Helvetica-Bold;\f2\fswiss\fcharset0 Helvetica-Oblique;
 \f3\fmodern\fcharset0 CourierNewPSMT;}
 {\colortbl;\red255\green255\blue255;}
@@ -8,12 +8,6 @@
 
 \f0\fs24 \cf0 This package will install Python $FULL_VERSION for macOS $MACOSX_DEPLOYMENT_TARGET for the following architecture(s): $ARCHITECTURES.\
 \
-\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\partightenfactor0
-
-\f1\b \cf0 NOTE: 
-\f0\b0 This is a beta test preview of Python 3.8.0, the next feature release of Python 3.8.  It is not intended for production use.\
-\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural\partightenfactor0
-\cf0 \
 \pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural\partightenfactor0
 
 \f1\b \cf0 \ul \ulc0 Certificate verification and OpenSSL\
@@ -29,21 +23,35 @@ This package includes its own private copy of OpenSSL 1.1.1.   The trust certifi
 \f3 /Applications/Python 3.8
 \f0  to install a curated bundle of default root certificates from the third-party 
 \f3 certifi
-\f0  package ({\field{\*\fldinst{HYPERLINK "https://pypi.org/project/certifi/"}}{\fldrslt https://pypi.org/project/certifi/}}).  If you choose to use 
-\f3 certifi
-\f0 , you should consider subscribing to the{\field{\*\fldinst{HYPERLINK "https://certifi.io/en/latest/"}}{\fldrslt  project's email update service}} to be notified when the certificate bundle is updated.\
+\f0  package ({\field{\*\fldinst{HYPERLINK "https://pypi.org/project/certifi/"}}{\fldrslt https://pypi.org/project/certifi/}}).  Double-click on 
+\f3 Install Certificates
+\f0  to run it.\
 \
 The bundled 
 \f3 pip
-\f0  included with this installer has its own default certificate store for verifying download connections.\
+\f0  has its own default certificate store for verifying download connections.\
 \
 
 \f1\b \ul Using IDLE or other Tk applications 
 \f0\b0 \ulnone \
 \
 This package includes its own private version of Tcl/Tk 8.6. It does not use any system-supplied or third-party supplied versions of Tcl/Tk.\
+\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural\partightenfactor0
+\cf0 \
+Due to new security checks on macOS 10.15 Catalina, when launching IDLE macOS may open a window with a message 
+\f1\b "Python" would like to access files in your Documents folder
+\f0\b0 .  This is normal as IDLE uses your 
+\f1\b Documents
+\f0\b0  folder as its default when opening and saving files; you can still choose other locations in the 
+\f1\b Open
+\f0\b0  and 
+\f1\b Save
+\f0\b0  file dialog windows.  Click on the 
+\f1\b OK
+\f0\b0  button to proceed.\
+\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural\partightenfactor0
 
-\f1\b \ul \
+\f1\b \cf0 \ul \ulc0 \
 Other changes\
 
 \f0\b0 \ulnone \
@@ -52,34 +60,4 @@ For other changes in this release, see the
 \f0\i0  section in the {\field{\*\fldinst{HYPERLINK "https://www.python.org/doc/"}}{\fldrslt Documentation Set}} for this release and its 
 \f2\i Release Notes
 \f0\i0  link at {\field{\*\fldinst{HYPERLINK "https://www.python.org/downloads/"}}{\fldrslt https://www.python.org/downloads/}}.\
-
-\f1\b \ul \
-Python 3 and Python 2 Co-existence\
-
-\f0\b0 \ulnone \
-Python.org Python $VERSION and 2.7.x versions can both be installed on your system and will not conflict. Command names for Python 3 contain a 3 in them, 
-\f3 python3
-\f0  (or
-\f3  python$VERSION
-\f0 ), 
-\f3 idle3
-\f0  (or i
-\f3 dle$VERSION
-\f0 ), 
-\f3 pip3
-\f0  (or 
-\f3 pip$VERSION
-\f0 ), etc.  Python 2.7 command names contain a 2 or no digit: 
-\f3 python2
-\f0  (or 
-\f3 python2.7
-\f0  or 
-\f3 python
-\f0 ), 
-\f3 idle2
-\f0  (or 
-\f3 idle2.7
-\f0  or 
-\f3 idle
-\f0 ), etc.\
 }
\ No newline at end of file
diff --git a/Mac/BuildScript/resources/Welcome.rtf b/Mac/BuildScript/resources/Welcome.rtf
index 053084d0425b3..ce53bd78bbd46 100644
--- a/Mac/BuildScript/resources/Welcome.rtf
+++ b/Mac/BuildScript/resources/Welcome.rtf
@@ -1,4 +1,4 @@
-{\rtf1\ansi\ansicpg1252\cocoartf1671\cocoasubrtf500
+{\rtf1\ansi\ansicpg1252\cocoartf1671\cocoasubrtf600
 \cocoascreenfonts1{\fonttbl\f0\fswiss\fcharset0 Helvetica;\f1\fswiss\fcharset0 Helvetica-Bold;\f2\fmodern\fcharset0 CourierNewPSMT;
 }
 {\colortbl;\red255\green255\blue255;}
@@ -14,15 +14,13 @@
 \
 
 \f1\b Python for macOS
-\f0\b0  consists of the Python programming language interpreter, plus a set of programs to allow easy access to it for macOS users including an integrated development environment 
+\f0\b0  consists of the {\field{\*\fldinst{HYPERLINK "https://www.python.org"}}{\fldrslt Python}} programming language interpreter and its batteries-included standard library to allow easy access to macOS features.  It also includes the Python integrated development environment, 
 \f1\b IDLE
-\f0\b0 .\
+\f0\b0 .  You can also use the included 
+\f1\b pip
+\f0\b0  to download and install third-party packages from the {\field{\*\fldinst{HYPERLINK "https://pypi.org"}}{\fldrslt Python Package Index}}. \
 \
 At the end of this install, click on 
 \f2 Install Certificates
-\f0  for SSL root certificates\
-\
-
-\f1\b NOTE: 
-\f0\b0 This is a beta test preview of Python 3.8.0, the next feature release of Python 3.8.  It is not intended for production use.\
+\f0  to install a set of current SSL root certificates.\
 }
\ No newline at end of file


From webhook-mailer at python.org  Mon Oct 14 04:38:17 2019
From: webhook-mailer at python.org (Ned Deily)
Date: Mon, 14 Oct 2019 08:38:17 -0000
Subject: [Python-checkins] [3.7] Update macOS installer display files for
 3.7.5 (GH-16767)
Message-ID: <mailman.9.1571042298.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/c60bed32587c78c0a9ee01fd6f3dcb9f427a2afc
commit: c60bed32587c78c0a9ee01fd6f3dcb9f427a2afc
branch: 3.7
author: Ned Deily <nad at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-14T04:38:13-04:00
summary:

[3.7] Update macOS installer display files for 3.7.5 (GH-16767)

files:
M Mac/BuildScript/resources/License.rtf
M Mac/BuildScript/resources/ReadMe.rtf
M Mac/BuildScript/resources/Welcome.rtf

diff --git a/Mac/BuildScript/resources/License.rtf b/Mac/BuildScript/resources/License.rtf
index 20982a46cc36a..86e54f6662a20 100644
--- a/Mac/BuildScript/resources/License.rtf
+++ b/Mac/BuildScript/resources/License.rtf
@@ -1,16 +1,17 @@
-{\rtf1\ansi\ansicpg1252\cocoartf1343\cocoasubrtf160
-{\fonttbl\f0\fswiss\fcharset0 Helvetica;\f1\fmodern\fcharset0 CourierNewPS-BoldMT;\f2\fmodern\fcharset0 CourierNewPSMT;
-}
+{\rtf1\ansi\ansicpg1252\cocoartf1671\cocoasubrtf600
+{\fonttbl\f0\fswiss\fcharset0 Helvetica-Bold;\f1\fswiss\fcharset0 Helvetica;\f2\fmodern\fcharset0 CourierNewPS-BoldMT;
+\f3\fmodern\fcharset0 CourierNewPSMT;}
 {\colortbl;\red255\green255\blue255;}
+{\*\expandedcolortbl;;}
 \margl1440\margr1440\vieww14620\viewh13380\viewkind0
-\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural
+\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural\partightenfactor0
 
 \f0\b\fs36 \cf0 \ul \ulc0 HISTORY AND LICENSE\
 
 \fs24 \
 HISTORY OF THE SOFTWARE\
 
-\b0 \ulnone \
+\f1\b0 \ulnone \
 Python was created in the early 1990s by Guido van Rossum at Stichting Mathematisch Centrum (CWI, see http://www.cwi.nl) in the Netherlands as a successor of a language called ABC. Guido remains Python's principal author, although it includes many contributions from others.\
 \
 In 1995, Guido continued his work on Python at the Corporation for National Research Initiatives (CNRI, see http://www.cnri.reston.va.us) in Reston, Virginia where he released several versions of the software.\
@@ -20,10 +21,10 @@ In May 2000, Guido and the Python core development team moved to BeOpen.com to f
 All Python releases are Open Source (see http://www.opensource.org for the Open Source Definition). Historically, most, but not all, Python releases have also been GPL-compatible; the table below summarizes the various releases.\
 \
 
-\f1\b Release         Derived     Year        Owner       GPL-\
+\f2\b Release         Derived     Year        Owner       GPL-\
                 from                                compatible?\
 
-\f2\b0 \
+\f3\b0 \
 0.9.0 thru 1.2  n/a         1991-1995   CWI         yes\
 1.3 thru 1.5.2  1.2         1995-1999   CNRI        yes\
 1.6             1.5.2       2000        CNRI        no\
@@ -36,25 +37,25 @@ All Python releases are Open Source (see http://www.opensource.org for the Open
 2.1.3           2.1.2       2002        PSF         yes\
 2.2 and above   2.1.1       2001-now    PSF         yes\
 
-\f0 \
+\f1 \
 
-\b Note:  
-\b0 GPL-compatible doesn't mean that we're distributing Python under the GPL. All Python licenses, unlike the GPL, let you distribute a modified version without making your changes open source. The GPL-compatible licenses make it possible to combine Python with other software that is released under the GPL; the others don't.\
+\f0\b Note:  
+\f1\b0 GPL-compatible doesn't mean that we're distributing Python under the GPL. All Python licenses, unlike the GPL, let you distribute a modified version without making your changes open source. The GPL-compatible licenses make it possible to combine Python with other software that is released under the GPL; the others don't.\
 \
 Thanks to the many outside volunteers who have worked under Guido's direction to make these releases possible.\
 \
 \
 
-\b \ul TERMS AND CONDITIONS FOR ACCESSING OR OTHERWISE USING PYTHON\
+\f0\b \ul TERMS AND CONDITIONS FOR ACCESSING OR OTHERWISE USING PYTHON\
 
-\b0 \ulnone \
+\f1\b0 \ulnone \
 
-\b PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2\
+\f0\b PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2\
 
-\b0 \
+\f1\b0 \
 1. This LICENSE AGREEMENT is between the Python Software Foundation ("PSF"), and the Individual or Organization ("Licensee") accessing and otherwise using this software ("Python") in source or binary form and its associated documentation.\
 \
-2. Subject to the terms and conditions of this License Agreement, PSF hereby grants Licensee a nonexclusive, royalty-free, world-wide license to reproduce, analyze, test, perform and/or display publicly, prepare derivative works, distribute, and otherwise use Python alone or in any derivative version, provided, however, that PSF's License Agreement and PSF's notice of copyright, i.e., "Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016 Python Software Foundation; All Rights Reserved" are retained in Python alone or in any derivative version prepared by Licensee.\
+2. Subject to the terms and conditions of this License Agreement, PSF hereby grants Licensee a nonexclusive, royalty-free, world-wide license to reproduce, analyze, test, perform and/or display publicly, prepare derivative works, distribute, and otherwise use Python alone or in any derivative version, provided, however, that PSF's License Agreement and PSF's notice of copyright, i.e., "Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019 Python Software Foundation; All Rights Reserved" are retained in Python alone or in any derivative version prepared by Licensee.\
 \
 3. In the event Licensee prepares a derivative work that is based on or incorporates Python or any part thereof, and wants to make the derivative work available to others as provided herein, then Licensee hereby agrees to include in any such work a brief summary of the changes made to Python.\
 \
@@ -70,9 +71,9 @@ Thanks to the many outside volunteers who have worked under Guido's direction to
 \
 \
 
-\b BEOPEN.COM LICENSE AGREEMENT FOR PYTHON 2.0\
+\f0\b BEOPEN.COM LICENSE AGREEMENT FOR PYTHON 2.0\
 
-\b0 \
+\f1\b0 \
 BEOPEN PYTHON OPEN SOURCE LICENSE AGREEMENT VERSION 1\
 \
 1. This LICENSE AGREEMENT is between BeOpen.com ("BeOpen"), having an office at 160 Saratoga Avenue, Santa Clara, CA 95051, and the Individual or Organization ("Licensee") accessing and otherwise using this software in source or binary form and its associated documentation ("the Software").\
@@ -91,9 +92,9 @@ BEOPEN PYTHON OPEN SOURCE LICENSE AGREEMENT VERSION 1\
 \
 \
 
-\b CNRI LICENSE AGREEMENT FOR PYTHON 1.6.1\
+\f0\b CNRI LICENSE AGREEMENT FOR PYTHON 1.6.1\
 
-\b0 \
+\f1\b0 \
 1. This LICENSE AGREEMENT is between the Corporation for National Research Initiatives, having an office at 1895 Preston White Drive, Reston, VA 20191 ("CNRI"), and the Individual or Organization ("Licensee") accessing and otherwise using Python 1.6.1 software in source or binary form and its associated documentation.\
 \
 2. Subject to the terms and conditions of this License Agreement, CNRI hereby grants Licensee a nonexclusive, royalty-free, world-wide license to reproduce, analyze, test, perform and/or display publicly, prepare derivative works, distribute, and otherwise use Python 1.6.1 alone or in any derivative version, provided, however, that CNRI's License Agreement and CNRI's notice of copyright, i.e., "Copyright (c) 1995-2001 Corporation for National Research Initiatives; All Rights Reserved" are retained in Python 1.6.1 alone or in any derivative version prepared by Licensee. Alternately, in lieu of CNRI's License Agreement, Licensee may substitute the following text (omitting the quotes): "Python 1.6.1 is made available subject to the terms and conditions in CNRI's License Agreement. This Agreement together with Python 1.6.1 may be located on the Internet using the following unique, persistent identifier (known as a handle): 1895.22/1013. This Agreement may also be obtained from a proxy server on the Internet using the following URL: http://hdl.handle.net/1895.22/1013".\
@@ -114,9 +115,9 @@ BEOPEN PYTHON OPEN SOURCE LICENSE AGREEMENT VERSION 1\
 \
 \
 
-\b CWI LICENSE AGREEMENT FOR PYTHON 0.9.0 THROUGH 1.2\
+\f0\b CWI LICENSE AGREEMENT FOR PYTHON 0.9.0 THROUGH 1.2\
 
-\b0 \
+\f1\b0 \
 Copyright (c) 1991 - 1995, Stichting Mathematisch Centrum Amsterdam, The Netherlands. All rights reserved.\
 \
 Permission to use, copy, modify, and distribute this software and its documentation for any purpose and without fee is hereby granted, provided that the above copyright notice appear in all copies and that both that copyright notice and this permission notice appear in supporting documentation, and that the name of Stichting Mathematisch Centrum or CWI not be used in advertising or publicity pertaining to distribution of the software without specific, written prior permission.\
@@ -125,16 +126,16 @@ STICHTING MATHEMATISCH CENTRUM DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFT
 \
 \
 
-\b \ul LICENSES AND ACKNOWLEDGEMENTS FOR INCORPORATED SOFTWARE\
+\f0\b \ul LICENSES AND ACKNOWLEDGEMENTS FOR INCORPORATED SOFTWARE\
 
-\b0 \ulnone \
+\f1\b0 \ulnone \
 This installer incorporates portions of the following third-party software:\
 \
 
-\f2 $THIRD_PARTY_LIBS\
+\f3 $THIRD_PARTY_LIBS\
 \
 
-\f0 For licenses and acknowledgements for these and other third-party software incorporated in this Python distribution, please refer to the on-line documentation {\field{\*\fldinst{HYPERLINK "https://docs.python.org/$VERSION/license.html#licenses-and-acknowledgements-for-incorporated-software"}}{\fldrslt here}}.\
+\f1 For licenses and acknowledgements for these and other third-party software incorporated in this Python distribution, please refer to the on-line documentation {\field{\*\fldinst{HYPERLINK "https://docs.python.org/$VERSION/license.html#licenses-and-acknowledgements-for-incorporated-software"}}{\fldrslt here}}.\
 \
 \
 \
diff --git a/Mac/BuildScript/resources/ReadMe.rtf b/Mac/BuildScript/resources/ReadMe.rtf
index f933d74bda14d..b71bd642902a2 100644
--- a/Mac/BuildScript/resources/ReadMe.rtf
+++ b/Mac/BuildScript/resources/ReadMe.rtf
@@ -7,127 +7,80 @@
 \pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural\partightenfactor0
 
 \f0\fs24 \cf0 This package will install Python $FULL_VERSION for macOS $MACOSX_DEPLOYMENT_TARGET for the following architecture(s): $ARCHITECTURES.\
-\
-\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural\partightenfactor0
-
-\f1\b \cf0 \ul \ulc0 macOS 10.15 Catalina Support\
-\
 \pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\partightenfactor0
-
-\f0\b0 \cf0 \ulnone At the time of the release of this release candidate, the next release of macOS, 
-\f1\b 10.15 Catalina
-\f0\b0 , was still in public beta.  Any updates needed to support 10.15 will be provided after it releases.\
-\
+\cf0 \
 \pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural\partightenfactor0
 
-\f1\b \cf0 \ul \ulc0 Which installer variant should I use?
-\f0\b0 \ulnone \
-\
-\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\partightenfactor0
-
-\f1\b \cf0 CHANGED in 3.7.4: 
-\f0\b0  The 
-\f1\b 10.6+ 64-/32-bit installer variant  is being deprecated
-\f0\b0 .  
-\f1\b Python 3.8.0
-\f0\b0  will 
-\f1\b not
-\f0\b0  include a binary installer for 10.6+ and 
-\f1\b future bugfix releases of 3.7.x
-\f0\b0  may not, either.  Mac OS X 10.6 Snow Leopard was released in 2009 and has not been supported by Apple for many years including lack of security updates.  It is becoming increasingly difficult to ensure new Python features and bug fixes are compatible with such old systems especially with Apple's deprecation and removal of 32-bit support in recent and upcoming macOS releases.  We believe that there is now very little usage of this installer variant and so we would like to focus our resources on supporting newer systems.  We do not plan to intentionally break Python support on 10.6 and we will consider bug fixes for problems found when building from source on 10.6. \
-\
-\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural\partightenfactor0
-\cf0 For Python.3.7, python.org currently provides two installer variants for download: one that installs a 
-\f2\i 64-bit-only 
-\f0\i0 Python capable of running on 
-\f2\i macOS 10.9 (Mavericks)
-\f0\i0  or later; and one that installs a 
-\f2\i 64-bit/32-bit Intel
-\f0\i0  Python capable of running on 
-\f2\i macOS 10.6 (Snow Leopard)
-\f0\i0  or later.  (This ReadMe was installed with the 
-\f2\i $MACOSX_DEPLOYMENT_TARGET
-\f0\i0  variant.)  If you are running on macOS 10.9 or later and if you have no need for compatibility with older systems, use the 10.9 variant.  Use the 10.6 variant if you are running on macOS 10.6 through 10.8 or if you want to produce standalone applications that can run on systems from 10.6.  The Pythons installed by these installers are built with private copies of some third-party libraries not included with or newer than those in macOS itself.  The list of these libraries varies by installer variant and is included at the end of the License.rtf file.
-\f1\b \ul \
-\
-Certificate verification and OpenSSL\
+\f1\b \cf0 \ul Certificate verification and OpenSSL\
 
 \f0\b0 \ulnone \
-\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\partightenfactor0
-\cf0 This variant of Python 3.7 includes its own private copy of OpenSSL 1.1.1.  The deprecated Apple-supplied OpenSSL libraries are no longer used.  This means that the trust certificates in system and user keychains managed by the 
+This package includes its own private copy of OpenSSL 1.1.1.   The trust certificates in system and user keychains managed by the 
 \f2\i Keychain Access 
 \f0\i0 application and the 
 \f2\i security
-\f0\i0  command line utility are no longer used as defaults by the Python 
+\f0\i0  command line utility are not used as defaults by the Python 
 \f3 ssl
 \f0  module.  A sample command script is included in 
 \f3 /Applications/Python 3.7
 \f0  to install a curated bundle of default root certificates from the third-party 
 \f3 certifi
-\f0  package ({\field{\*\fldinst{HYPERLINK "https://pypi.org/project/certifi/"}}{\fldrslt https://pypi.org/project/certifi/}}).  If you choose to use 
-\f3 certifi
-\f0 , you should consider subscribing to the{\field{\*\fldinst{HYPERLINK "https://certifi.io/en/latest/"}}{\fldrslt  project's email update service}} to be notified when the certificate bundle is updated.\
-\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural\partightenfactor0
-\cf0 \
+\f0  package ({\field{\*\fldinst{HYPERLINK "https://pypi.org/project/certifi/"}}{\fldrslt https://pypi.org/project/certifi/}}).  Double-click on 
+\f3 Install Certificates
+\f0  to run it.\
+\
 The bundled 
 \f3 pip
-\f0  included with this installer has its own default certificate store for verifying download connections.\
-\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\partightenfactor0
+\f0  has its own default certificate store for verifying download connections.\
+\
 
-\f1\b \cf0 \
-CHANGED in 3.7.4: 
-\f0\b0  OpenSSL has been updated from 1.1.0 to 1.1.1.\
+\f1\b \ul Which installer variant should I use?
+\f0\b0 \ulnone \
+\
+\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\partightenfactor0
+\cf0 In almost all cases, you should use the 
+\f1\b macOS 64-bit installer for OS X 10.9 and later
+\f0\b0 .\
+\
+The legacy 
+\f1\b macOS 64-bit/32-bit installer for Mac OS X 10.6 and later 
+\f0\b0 variant  is now deprecated.  
+\f1\b Python 3.8.0
+\f0\b0  will 
+\f1\b not
+\f0\b0  include a binary installer for the 10.6+ variant and 
+\f1\b future bugfix releases of 3.7.x
+\f0\b0  may not, either.  macOS 10.6 Snow Leopard was released in 2009 and has not been supported by Apple for many years including lack of security updates.  It is becoming increasingly difficult to ensure new Python features and bug fixes are compatible with such old systems.  Note that, due to recent Apple installer packaging changes, the 10.6+ installer pkg we provide can no longer be opened by the Apple system installer application on 10.6;  10.7 and 10.8 are not affected.  We believe that there is now very little usage of this installer variant and so we would like to focus our resources on supporting newer systems.  We do not plan to intentionally break Python support on 10.6 through 10.8 and we will consider bug fixes for problems found when building from source on those systems.  
+\f1\b macOS 10.15 Catalina
+\f0\b0  removes support for running 32-bit architecture programs; we do not recommend trying to use the 10.6+ variant on it and it may not install on 10.15 systems without intervention. \
 \pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural\partightenfactor0
-\cf0 \
 
-\f1\b \ul Using IDLE or other Tk applications 
+\f1\b \cf0 \ul \
+Using IDLE or other Tk applications 
 \f0\b0 \ulnone \
 \
-Both installer variants now come with their own private version of Tcl/Tk 8.6. They no longer use system-supplied or third-party supplied versions of Tcl/Tk as in previous releases.\
+This package includes its own private version of Tcl/Tk 8.6. It does not use any system-supplied or third-party supplied versions of Tcl/Tk.\
+\
+\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural\partightenfactor0
+\cf0 Due to new security checks on macOS 10.15 Catalina, when launching IDLE macOS may open a window with a message 
+\f1\b "Python" would like to access files in your Documents folder
+\f0\b0 .  This is normal as IDLE uses your 
+\f1\b Documents
+\f0\b0  folder as its default when opening and saving files; you can still choose other locations in the 
+\f1\b Open
+\f0\b0  and 
+\f1\b Save
+\f0\b0  file dialog windows.  Click on the 
+\f1\b OK
+\f0\b0  button to proceed.\
+\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural\partightenfactor0
 
-\f1\b \ul \
+\f1\b \cf0 \ul \ulc0 \
 Other changes\
 \
-\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\partightenfactor0
-\cf0 \ulnone CHANGED in 3.7.4: 
-\f0\b0  SQLite has been updated from 3.22.0 to 3.28.0
-\f1\b \ul \
-\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural\partightenfactor0
 
-\f0\b0 \cf0 \ulnone \
-For other changes in this release, see the 
+\f0\b0 \ulnone For other changes in this release, see the 
 \f2\i What's new
 \f0\i0  section in the {\field{\*\fldinst{HYPERLINK "https://www.python.org/doc/"}}{\fldrslt Documentation Set}} for this release and its 
 \f2\i Release Notes
 \f0\i0  link at {\field{\*\fldinst{HYPERLINK "https://www.python.org/downloads/"}}{\fldrslt https://www.python.org/downloads/}}.\
-
-\f1\b \ul \
-Python 3 and Python 2 Co-existence\
-
-\f0\b0 \ulnone \
-Python.org Python $VERSION and 2.7.x versions can both be installed on your system and will not conflict. Command names for Python 3 contain a 3 in them, 
-\f3 python3
-\f0  (or
-\f3  python$VERSION
-\f0 ), 
-\f3 idle3
-\f0  (or i
-\f3 dle$VERSION
-\f0 ), 
-\f3 pip3
-\f0  (or 
-\f3 pip$VERSION
-\f0 ), etc.  Python 2.7 command names contain a 2 or no digit: 
-\f3 python2
-\f0  (or 
-\f3 python2.7
-\f0  or 
-\f3 python
-\f0 ), 
-\f3 idle2
-\f0  (or 
-\f3 idle2.7
-\f0  or 
-\f3 idle
-\f0 ), etc.\
 }
\ No newline at end of file
diff --git a/Mac/BuildScript/resources/Welcome.rtf b/Mac/BuildScript/resources/Welcome.rtf
index 67361e7c176af..410ad425516f1 100644
--- a/Mac/BuildScript/resources/Welcome.rtf
+++ b/Mac/BuildScript/resources/Welcome.rtf
@@ -3,7 +3,7 @@
 }
 {\colortbl;\red255\green255\blue255;}
 {\*\expandedcolortbl;;}
-\paperw11905\paperh16837\margl1440\margr1440\vieww12200\viewh10880\viewkind0
+\paperw11905\paperh16837\margl1440\margr1440\vieww12380\viewh10900\viewkind0
 \pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\partightenfactor0
 
 \f0\fs24 \cf0 This package will install 
@@ -14,23 +14,13 @@
 \
 
 \f1\b Python for macOS
-\f0\b0  consists of the Python programming language interpreter, plus a set of programs to allow easy access to it for macOS users including an integrated development environment 
+\f0\b0  consists of the {\field{\*\fldinst{HYPERLINK "https://www.python.org"}}{\fldrslt Python}} programming language interpreter and its batteries-included standard library to allow easy access to macOS features.  It also includes the Python integrated development environment, 
 \f1\b IDLE
-\f0\b0 .\
+\f0\b0 .  You can also use the included 
+\f1\b pip
+\f0\b0  to download and install third-party packages from the {\field{\*\fldinst{HYPERLINK "https://pypi.org"}}{\fldrslt Python Package Index}}. \
 \
 At the end of this install, click on 
 \f2 Install Certificates
-\f0  for SSL root certificates\
-\
-
-\f1\b NEW in 3.7.0: 
-\f0\b0 two installer variants (10.9+ 64-bit-only, 10.6+ 64-/32-bit), built-in Tcl/Tk 8.6 support (no additional third-party downloads!), OpenSSL 1.1, and more!\
-
-\f1\b \
-CHANGED in 3.7.4: 
-\f0\b0 OpenSSL 1.1.1, SQLite 3.28.0, 10.6+ 64-/32-bit installer variant deprecated\
-\
-At the time of the release of this release candidate, the next release of macOS, 
-\f1\b 10.15 Catalina
-\f0\b0 , was still in public beta.  Any updates needed to support 10.15 will be provided after it releases.\
+\f0  to install a set of current SSL root certificates.\
 }
\ No newline at end of file


From webhook-mailer at python.org  Mon Oct 14 04:39:04 2019
From: webhook-mailer at python.org (Ned Deily)
Date: Mon, 14 Oct 2019 08:39:04 -0000
Subject: [Python-checkins] [2.7] Update macOS installer display files for
 2.7.17 (GH-16768)
Message-ID: <mailman.10.1571042345.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/2c9d70a229943ba628614c0bd57d13441777b1a0
commit: 2c9d70a229943ba628614c0bd57d13441777b1a0
branch: 2.7
author: Ned Deily <nad at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-14T04:39:00-04:00
summary:

[2.7] Update macOS installer display files for 2.7.17 (GH-16768)

files:
M Mac/BuildScript/resources/License.rtf
M Mac/BuildScript/resources/ReadMe.rtf
M Mac/BuildScript/resources/Welcome.rtf
M Mac/BuildScript/resources/install_certificates.command

diff --git a/Mac/BuildScript/resources/License.rtf b/Mac/BuildScript/resources/License.rtf
index c6c30288db6e4..86e54f6662a20 100644
--- a/Mac/BuildScript/resources/License.rtf
+++ b/Mac/BuildScript/resources/License.rtf
@@ -1,16 +1,17 @@
-{\rtf1\ansi\ansicpg1252\cocoartf1343\cocoasubrtf160
-{\fonttbl\f0\fswiss\fcharset0 Helvetica;\f1\fmodern\fcharset0 CourierNewPS-BoldMT;\f2\fmodern\fcharset0 CourierNewPSMT;
-}
+{\rtf1\ansi\ansicpg1252\cocoartf1671\cocoasubrtf600
+{\fonttbl\f0\fswiss\fcharset0 Helvetica-Bold;\f1\fswiss\fcharset0 Helvetica;\f2\fmodern\fcharset0 CourierNewPS-BoldMT;
+\f3\fmodern\fcharset0 CourierNewPSMT;}
 {\colortbl;\red255\green255\blue255;}
+{\*\expandedcolortbl;;}
 \margl1440\margr1440\vieww14620\viewh13380\viewkind0
-\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural
+\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural\partightenfactor0
 
 \f0\b\fs36 \cf0 \ul \ulc0 HISTORY AND LICENSE\
 
 \fs24 \
 HISTORY OF THE SOFTWARE\
 
-\b0 \ulnone \
+\f1\b0 \ulnone \
 Python was created in the early 1990s by Guido van Rossum at Stichting Mathematisch Centrum (CWI, see http://www.cwi.nl) in the Netherlands as a successor of a language called ABC. Guido remains Python's principal author, although it includes many contributions from others.\
 \
 In 1995, Guido continued his work on Python at the Corporation for National Research Initiatives (CNRI, see http://www.cnri.reston.va.us) in Reston, Virginia where he released several versions of the software.\
@@ -20,10 +21,10 @@ In May 2000, Guido and the Python core development team moved to BeOpen.com to f
 All Python releases are Open Source (see http://www.opensource.org for the Open Source Definition). Historically, most, but not all, Python releases have also been GPL-compatible; the table below summarizes the various releases.\
 \
 
-\f1\b Release         Derived     Year        Owner       GPL-\
+\f2\b Release         Derived     Year        Owner       GPL-\
                 from                                compatible?\
 
-\f2\b0 \
+\f3\b0 \
 0.9.0 thru 1.2  n/a         1991-1995   CWI         yes\
 1.3 thru 1.5.2  1.2         1995-1999   CNRI        yes\
 1.6             1.5.2       2000        CNRI        no\
@@ -36,25 +37,25 @@ All Python releases are Open Source (see http://www.opensource.org for the Open
 2.1.3           2.1.2       2002        PSF         yes\
 2.2 and above   2.1.1       2001-now    PSF         yes\
 
-\f0 \
+\f1 \
 
-\b Note:  
-\b0 GPL-compatible doesn't mean that we're distributing Python under the GPL. All Python licenses, unlike the GPL, let you distribute a modified version without making your changes open source. The GPL-compatible licenses make it possible to combine Python with other software that is released under the GPL; the others don't.\
+\f0\b Note:  
+\f1\b0 GPL-compatible doesn't mean that we're distributing Python under the GPL. All Python licenses, unlike the GPL, let you distribute a modified version without making your changes open source. The GPL-compatible licenses make it possible to combine Python with other software that is released under the GPL; the others don't.\
 \
 Thanks to the many outside volunteers who have worked under Guido's direction to make these releases possible.\
 \
 \
 
-\b \ul TERMS AND CONDITIONS FOR ACCESSING OR OTHERWISE USING PYTHON\
+\f0\b \ul TERMS AND CONDITIONS FOR ACCESSING OR OTHERWISE USING PYTHON\
 
-\b0 \ulnone \
+\f1\b0 \ulnone \
 
-\b PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2\
+\f0\b PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2\
 
-\b0 \
+\f1\b0 \
 1. This LICENSE AGREEMENT is between the Python Software Foundation ("PSF"), and the Individual or Organization ("Licensee") accessing and otherwise using this software ("Python") in source or binary form and its associated documentation.\
 \
-2. Subject to the terms and conditions of this License Agreement, PSF hereby grants Licensee a nonexclusive, royalty-free, world-wide license to reproduce, analyze, test, perform and/or display publicly, prepare derivative works, distribute, and otherwise use Python alone or in any derivative version, provided, however, that PSF's License Agreement and PSF's notice of copyright, i.e., "Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017 Python Software Foundation; All Rights Reserved" are retained in Python alone or in any derivative version prepared by Licensee.\
+2. Subject to the terms and conditions of this License Agreement, PSF hereby grants Licensee a nonexclusive, royalty-free, world-wide license to reproduce, analyze, test, perform and/or display publicly, prepare derivative works, distribute, and otherwise use Python alone or in any derivative version, provided, however, that PSF's License Agreement and PSF's notice of copyright, i.e., "Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019 Python Software Foundation; All Rights Reserved" are retained in Python alone or in any derivative version prepared by Licensee.\
 \
 3. In the event Licensee prepares a derivative work that is based on or incorporates Python or any part thereof, and wants to make the derivative work available to others as provided herein, then Licensee hereby agrees to include in any such work a brief summary of the changes made to Python.\
 \
@@ -70,9 +71,9 @@ Thanks to the many outside volunteers who have worked under Guido's direction to
 \
 \
 
-\b BEOPEN.COM LICENSE AGREEMENT FOR PYTHON 2.0\
+\f0\b BEOPEN.COM LICENSE AGREEMENT FOR PYTHON 2.0\
 
-\b0 \
+\f1\b0 \
 BEOPEN PYTHON OPEN SOURCE LICENSE AGREEMENT VERSION 1\
 \
 1. This LICENSE AGREEMENT is between BeOpen.com ("BeOpen"), having an office at 160 Saratoga Avenue, Santa Clara, CA 95051, and the Individual or Organization ("Licensee") accessing and otherwise using this software in source or binary form and its associated documentation ("the Software").\
@@ -91,9 +92,9 @@ BEOPEN PYTHON OPEN SOURCE LICENSE AGREEMENT VERSION 1\
 \
 \
 
-\b CNRI LICENSE AGREEMENT FOR PYTHON 1.6.1\
+\f0\b CNRI LICENSE AGREEMENT FOR PYTHON 1.6.1\
 
-\b0 \
+\f1\b0 \
 1. This LICENSE AGREEMENT is between the Corporation for National Research Initiatives, having an office at 1895 Preston White Drive, Reston, VA 20191 ("CNRI"), and the Individual or Organization ("Licensee") accessing and otherwise using Python 1.6.1 software in source or binary form and its associated documentation.\
 \
 2. Subject to the terms and conditions of this License Agreement, CNRI hereby grants Licensee a nonexclusive, royalty-free, world-wide license to reproduce, analyze, test, perform and/or display publicly, prepare derivative works, distribute, and otherwise use Python 1.6.1 alone or in any derivative version, provided, however, that CNRI's License Agreement and CNRI's notice of copyright, i.e., "Copyright (c) 1995-2001 Corporation for National Research Initiatives; All Rights Reserved" are retained in Python 1.6.1 alone or in any derivative version prepared by Licensee. Alternately, in lieu of CNRI's License Agreement, Licensee may substitute the following text (omitting the quotes): "Python 1.6.1 is made available subject to the terms and conditions in CNRI's License Agreement. This Agreement together with Python 1.6.1 may be located on the Internet using the following unique, persistent identifier (known as a handle): 1895.22/1013. This Agreement may also be obtained from a proxy server on the Internet using the following URL: http://hdl.handle.net/1895.22/1013".\
@@ -114,9 +115,9 @@ BEOPEN PYTHON OPEN SOURCE LICENSE AGREEMENT VERSION 1\
 \
 \
 
-\b CWI LICENSE AGREEMENT FOR PYTHON 0.9.0 THROUGH 1.2\
+\f0\b CWI LICENSE AGREEMENT FOR PYTHON 0.9.0 THROUGH 1.2\
 
-\b0 \
+\f1\b0 \
 Copyright (c) 1991 - 1995, Stichting Mathematisch Centrum Amsterdam, The Netherlands. All rights reserved.\
 \
 Permission to use, copy, modify, and distribute this software and its documentation for any purpose and without fee is hereby granted, provided that the above copyright notice appear in all copies and that both that copyright notice and this permission notice appear in supporting documentation, and that the name of Stichting Mathematisch Centrum or CWI not be used in advertising or publicity pertaining to distribution of the software without specific, written prior permission.\
@@ -125,18 +126,18 @@ STICHTING MATHEMATISCH CENTRUM DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFT
 \
 \
 
-\b \ul LICENSES AND ACKNOWLEDGEMENTS FOR INCORPORATED SOFTWARE\
+\f0\b \ul LICENSES AND ACKNOWLEDGEMENTS FOR INCORPORATED SOFTWARE\
 
-\b0 \ulnone \
+\f1\b0 \ulnone \
 This installer incorporates portions of the following third-party software:\
 \
 
-\f2 $THIRD_PARTY_LIBS\
+\f3 $THIRD_PARTY_LIBS\
 \
 
-\f0 For licenses and acknowledgements for these and other third-party software incorporated in this Python distribution, please refer to the on-line documentation {\field{\*\fldinst{HYPERLINK "https://docs.python.org/$VERSION/license.html#licenses-and-acknowledgements-for-incorporated-software"}}{\fldrslt here}}.\
+\f1 For licenses and acknowledgements for these and other third-party software incorporated in this Python distribution, please refer to the on-line documentation {\field{\*\fldinst{HYPERLINK "https://docs.python.org/$VERSION/license.html#licenses-and-acknowledgements-for-incorporated-software"}}{\fldrslt here}}.\
 \
 \
 \
 \
-}
+}
\ No newline at end of file
diff --git a/Mac/BuildScript/resources/ReadMe.rtf b/Mac/BuildScript/resources/ReadMe.rtf
index b0c5d70396714..a4c351a7ff78a 100644
--- a/Mac/BuildScript/resources/ReadMe.rtf
+++ b/Mac/BuildScript/resources/ReadMe.rtf
@@ -3,158 +3,83 @@
 \f3\fmodern\fcharset0 CourierNewPSMT;}
 {\colortbl;\red255\green255\blue255;}
 {\*\expandedcolortbl;;}
-\margl1440\margr1440\vieww15240\viewh15540\viewkind0
+\margl1440\margr1440\vieww14900\viewh15540\viewkind0
 \pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural\partightenfactor0
 
 \f0\fs24 \cf0 This package will install Python $FULL_VERSION for macOS $MACOSX_DEPLOYMENT_TARGET for the following architecture(s): $ARCHITECTURES.\
 \
 \pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\partightenfactor0
-
-\f1\b \cf0 NOTE: 
-\f0\b0 This is the release candidate for Python 2.7.17.  Any updates needed to support the latest release of macOS, 
-\f1\b 10.15 Catalina,
-\f0\b0  will be provided after in 2.7.17 final.\
-\
+\cf0 NOTE: 
+\f1\b \ul Python 2 reaches end-of-life in 2020 and will no longer be supported or updated thereafter\ulnone .
+\f0\b0   You should 
+\f1\b upgrade to Python 3 
+\f0\b0 as soon as you can.  {\field{\*\fldinst{HYPERLINK "https://www.python.org/doc/sunset-python-2/"}}{\fldrslt Read more here}}.\
+\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural\partightenfactor0
+\cf0 \
 \pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural\partightenfactor0
 
-\f1\b \cf0 \ul \ulc0 Which installer variant should I use? [CHANGED in 2.7.15]
-\f0\b0 \ulnone \
-\
-With Python 2.7.15, the python.org website now provides two installer variants for download: one that installs a 
-\f2\i 64-bit-only 
-\f0\i0 Python capable of running on 
-\f2\i macOS 10.9 (Mavericks)
-\f0\i0  or later; and one that installs a 
-\f2\i 64-bit/32-bit Intel
-\f0\i0  Python capable of running on 
-\f2\i macOS 10.6 (Snow Leopard)
-\f0\i0  or later.  (This ReadMe was installed with the 
-\f2\i $MACOSX_DEPLOYMENT_TARGET
-\f0\i0  variant.)  Previous Python 2.7.x releases provided the 10.6 or later installer and a 10.5 or later 32-bit-only variant. If you are running on macOS 10.9 or later and if you have no need for compatibility with older systems, use the 10.9 variant.  Use the 10.6 variant if you are running on macOS 10.6 through 10.8, if you need to maintain compatibility with previous 2.7.x releases, or if you want to produce standalone applications that can run on systems from 10.6.  The Pythons installed by these installers are built with private copies of some third-party libraries not included with or newer than those in macOS itself.  The list of these libraries varies by installer variant and is included at the end of the 
-\f3 License.rtf
-\f0  file.\
-
-\f1\b \ul \
-Certificate verification and OpenSSL_[CHANGED in 2.7.15]\
+\f1\b \cf0 \ul \ulc0 Certificate verification and OpenSSL\
 
 \f0\b0 \ulnone \
-This variant of Python 2.7 now includes its own private copy of OpenSSL 1.0.2.  Unlike previous releases, the deprecated Apple-supplied OpenSSL libraries are no longer used.  This also means that the trust certificates in system and user keychains managed by the 
+This package includes its own private copy of OpenSSL 1.0.2.   The trust certificates in system and user keychains managed by the 
 \f2\i Keychain Access 
 \f0\i0 application and the 
 \f2\i security
-\f0\i0  command line utility are no longer used as defaults by the Python 
+\f0\i0  command line utility are not used as defaults by the Python 
 \f3 ssl
 \f0  module.  A sample command script is included in 
 \f3 /Applications/Python 2.7
 \f0  to install a curated bundle of default root certificates from the third-party 
 \f3 certifi
-\f0  package ({\field{\*\fldinst{HYPERLINK "https://pypi.python.org/pypi/certifi"}}{\fldrslt https://pypi.python.org/pypi/certifi}}).  Click on 
+\f0  package ({\field{\*\fldinst{HYPERLINK "https://pypi.org/project/certifi/"}}{\fldrslt https://pypi.org/project/certifi/}}).  Double-click on 
 \f3 Install Certificates
-\f0  to run it.  If you choose to use 
-\f3 certifi
-\f0 , you should consider subscribing to the{\field{\*\fldinst{HYPERLINK "https://certifi.io/en/latest/"}}{\fldrslt  project's email update service}} to be notified when the certificate bundle is updated.\
+\f0  to run it.\
 \
 The bundled 
 \f3 pip
-\f0  included with the Python 2.7 installer has its own default certificate store for verifying download connections.\
+\f0  has its own default certificate store for verifying download connections.\
 \
 
-\f1\b \ul Using IDLE or other Tk applications [NEW/CHANGED in 2.7.15] 
+\f1\b \ul Which installer variant should I use?
 \f0\b0 \ulnone \
 \
-As of 2.7.15, the 10.9+ installer variant comes with its own private version of Tcl/Tk 8.6. It does not use system-supplied or third-party supplied versions of Tcl/Tk.\
-\
-\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural\partightenfactor0
-
-\f1\b \cf0 CHANGED in 2.7.16: 
-\f0\b0 The 10.6+ variant now also uses a private version of Tcl/Tk 8.6.\
+\pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\partightenfactor0
+\cf0 In almost all cases, you should use the 
+\f1\b macOS 64-bit installer for OS X 10.9 and later
+\f0\b0 .\
+\
+The legacy 
+\f1\b macOS 64-bit/32-bit installer for Mac OS X 10.6 and later 
+\f0\b0 variant  is now deprecated.  macOS 10.6 Snow Leopard was released in 2009 and has not been supported by Apple for many years including lack of security updates.  It is becoming increasingly difficult to ensure new Python features and bug fixes are compatible with such old systems.  Note that, due to recent Apple installer packaging changes, the 10.6+ installer pkg we provide can no longer be opened by the Apple system installer application on 10.6;  10.7 and 10.8 are not affected.  We believe that there is now very little usage of this installer variant and so we would like to focus our resources on supporting newer systems.  We do not plan to intentionally break Python support on 10.6 through 10.8 and we will consider bug fixes for problems found when building from source on those systems through the support window of Python 2.7.  
+\f1\b macOS 10.15 Catalina
+\f0\b0  removes support for running 32-bit architecture programs; we do not recommend trying to use the 10.6+ variant on it and it may not install on 10.15 systems without intervention. \
 \pard\tx720\tx1440\tx2160\tx2880\tx3600\tx4320\tx5040\tx5760\tx6480\tx7200\tx7920\tx8640\pardirnatural\partightenfactor0
 
 \f1\b \cf0 \ul \
-Binary installer support for Mac OS X 10.5 and earlier discontinued [CHANGED for Python 2.7.15]
+Using IDLE or other Tk applications 
 \f0\b0 \ulnone \
 \
-As of Python 2.7.15, binary installers from python.org no longer support Mac OS X 10.5 (Leopard) systems.   Binary installer support for Mac OS X 10.3.9 (Panther) and 10.4.x (Tiger) systems was previously dropped in Python 2.7.9.  Mac OS X 10.5 was originally released by Apple in 2007 and last updated in 2009 and was the last OS X release for PPC machines (G4 and G5).  If needed, it is still possible to build Python from source for 10.3.9, 10.4, or 10.5.\
-\
-
-\f1\b \ul Packages installed with the system Python 2.7 are no longer searched for [CHANGED for Python 2.7.13]
-\f0\b0 \ulnone \
-\
-As of Python 2.7.0, user-installed Python 2.7 versions from python.org installers added the system-wide site-packages directory for the Apple-supplied Python 2.7 to the end of their search path.  This meant that packages installed with the system Python 2.7 could also be used by the user-installed Python 2.7.  While sometimes convenient, this also often caused confusion with the implicit coupling between the two Python instances.  Separately, as of macOS 10.12, Apple changed the layout of the system site-packages directory, 
-\f3 /Library/Python/2.7/site-packages
-\f0 , in a way that can now cause installation of 
-\f3 pip
-\f0  components to fail.  To avoid the confusion and the installation failures, as of 2.7.13 user-installed Pythons no longer add 
-\f3 /Library/Python/2.7/site-packages
-\f0  to 
-\f3 sys.path
-\f0 .  If you are using a package with both a user-installed Python 2.7 and the system Python 2.7, you will now need to ensure that separate copies of the package are installed for each instance.\
-\
+This package includes its own private version of Tcl/Tk 8.6. It does not use any system-supplied or third-party supplied versions of Tcl/Tk.\
+\
+Due to new security checks on macOS 10.15 Catalina, when launching IDLE macOS may open a window with a message 
+\f1\b "Python" would like to access files in your Documents folder
+\f0\b0 .  This is normal as IDLE uses your 
+\f1\b Documents
+\f0\b0  folder as its default when opening and saving files; you can still choose other locations in the 
+\f1\b Open
+\f0\b0  and 
+\f1\b Save
+\f0\b0  file dialog windows.  Click on the 
+\f1\b OK
+\f0\b0  button to proceed.\
 
-\f1\b \ul Installing on OS X 10.8 (Mountain Lion) or later systems [CHANGED for Python 2.7.9]
-\f0\b0 \ulnone \
-\
-As of Python 2.7.9, installer packages from python.org are now compatible with the Gatekeeper security feature introduced in OS X 10.8.   Downloaded packages can now be directly installed by double-clicking with the default system security settings.  Python.org installer packages for macOS are signed with the Developer ID of the builder, as identified on {\field{\*\fldinst{HYPERLINK "https://www.python.org/downloads/"}}{\fldrslt the download page}} for this release.  To inspect the digital signature of the package, click on the lock icon in the upper right corner of the 
-\f2\i Install Python
-\f0\i0  installer window.  Refer to Apple\'92s support pages for {\field{\*\fldinst{HYPERLINK "http://support.apple.com/kb/ht5290"}}{\fldrslt more information on Gatekeeper}}.\
-\
-
-\f1\b \ul Simplified web-based installs [NEW for Python 2.7.9]
-\f0\b0 \ulnone \
-\
-With the change to the newer flat format installer package, the download file now has a 
-\f3 .pkg
-\f0  extension as it is no longer necessary to embed the installer within a disk image (
-\f3 .dmg
-\f0 ) container.   If you download the Python installer through a web browser, the macOS installer application may open automatically to allow you to perform the install.  If your browser settings do not allow automatic open, double click on the downloaded installer file.\
+\f1\b \ul \
+Other changes\
 \
 
-\f1\b \ul New Installation Options and Defaults [NEW for Python 2.7.9]
-\f0\b0 \ulnone \
-\
-The Python installer now includes an option to automatically install or upgrade 
-\f3 pip
-\f0 , a tool for installing and managing Python packages.  This option is enabled by default and no Internet access is required.  If you do not want the installer to do this, select the 
-\f2\i Customize
-\f0\i0  option at the 
-\f2\i Installation Type
-\f0\i0  step and uncheck the 
-\f2\i Install or upgrade pip
-\f0\i0  option.  For other changes in this release, see the 
+\f0\b0 \ulnone For other changes in this release, see the 
+\f2\i What's new
+\f0\i0  section in the {\field{\*\fldinst{HYPERLINK "https://www.python.org/doc/"}}{\fldrslt Documentation Set}} for this release and its 
 \f2\i Release Notes
-\f0\i0  link for this release at {\field{\*\fldinst{HYPERLINK "https://www.python.org/downloads/"}}{\fldrslt https://www.python.org/downloads/}}.\
-\
-
-\f1\b \ul Python 3 and Python 2 Co-existence\
-
-\f0\b0 \ulnone \
-Python.org Python 2.7 and 3.x versions can both be installed on your system and will not conflict.  Python 2.7 command names contain a 2 or no digit: 
-\f3 python2
-\f0  (or 
-\f3 python2.7
-\f0  or 
-\f3 python
-\f0 ), 
-\f3 idle2
-\f0  (or 
-\f3 idle2.7
-\f0  or 
-\f3 idle
-\f0 ), 
-\f3 pip2
-\f0  (or 
-\f3 pip2.7
-\f0  or 
-\f3 pip
-\f0 ), etc.  Command names for Python 3 contain a 3 in them, 
-\f3 python3
-\f0 , 
-\f3 idle3
-\f0 , 
-\f3 pip3
-\f0 , etc.  Also, installing a python.org Python 2.7 does not alter or remove any Apple-supplied system Pythons, found in 
-\f3 /usr/bin
-\f0 .\
-\
-\
+\f0\i0  link at {\field{\*\fldinst{HYPERLINK "https://www.python.org/downloads/"}}{\fldrslt https://www.python.org/downloads/}}.\
 }
\ No newline at end of file
diff --git a/Mac/BuildScript/resources/Welcome.rtf b/Mac/BuildScript/resources/Welcome.rtf
index 289f6a1198d99..2786a74985488 100644
--- a/Mac/BuildScript/resources/Welcome.rtf
+++ b/Mac/BuildScript/resources/Welcome.rtf
@@ -15,25 +15,19 @@
 \cf0 \
 
 \f1\b Python for macOS
-\f0\b0  consists of the Python programming language interpreter, plus a set of programs to allow easy access to it for macOS users including an integrated development environment 
+\f0\b0  consists of the {\field{\*\fldinst{HYPERLINK "https://www.python.org"}}{\fldrslt Python}} programming language interpreter and its batteries-included standard library to allow easy access to macOS features.  It also includes the Python integrated development environment, 
 \f1\b IDLE
-\f0\b0 .\
+\f0\b0 .  You can also use the included 
+\f1\b pip
+\f0\b0  to download and install third-party packages from the {\field{\*\fldinst{HYPERLINK "https://pypi.org"}}{\fldrslt Python Package Index}}. \
 \
-
-\f1\b NEW in 2.7.15: 
-\f0\b0 two installer variants (10.9+ 64-bit-only, 10.6+ 64-/32-bit), built-in Tcl/Tk 8.6 support in the 10.9+ variant (no additional third-party downloads!), updated  
-\f2 pip,
-\f0  built-in OpenSSL 1.0.2 (click on 
+At the end of this install, click on 
 \f2 Install Certificates
-\f0  for root certificates)\
-\
-
-\f1\b CHANGED in 2.7.16:
-\f0\b0   the 10.6+ variant now also uses a built-in Tcl/Tk 8.6\
+\f0  to install a set of current SSL root certificates.\
 \
 
-\f1\b NOTE: 
-\f0\b0 This is the release candidate for Python 2.7.17.  Any updates needed to support the latest release of macOS, 
-\f1\b 10.15 Catalina,
-\f0\b0  will be provided after in 2.7.17 final.\
+\f1\b NOTE: \ul Python 2 reaches end-of-life in 2020 and will no longer be supported or updated thereafter\ulnone .
+\f0\b0   You should 
+\f1\b upgrade to Python 3 
+\f0\b0 as soon as you can.  {\field{\*\fldinst{HYPERLINK "https://www.python.org/doc/sunset-python-2/"}}{\fldrslt Read more here}}.\
 }
\ No newline at end of file
diff --git a/Mac/BuildScript/resources/install_certificates.command b/Mac/BuildScript/resources/install_certificates.command
index 6cae65c44dc89..70da41455e781 100755
--- a/Mac/BuildScript/resources/install_certificates.command
+++ b/Mac/BuildScript/resources/install_certificates.command
@@ -6,7 +6,7 @@
 #
 # sample script to install or update a set of default Root Certificates
 # for the ssl module.  Uses the certificates provided by the certifi package:
-#       https://pypi.python.org/pypi/certifi
+#       https://pypi.org/project/certifi/
 
 import os
 import os.path


From webhook-mailer at python.org  Mon Oct 14 06:36:33 2019
From: webhook-mailer at python.org (Tal Einat)
Date: Mon, 14 Oct 2019 10:36:33 -0000
Subject: [Python-checkins] bpo-32498: urllib.parse.unquote also accepts
 bytes (GH-7768)
Message-ID: <mailman.11.1571049394.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/aad2ee01561f260c69af1951c0d6fcaf75c4d41b
commit: aad2ee01561f260c69af1951c0d6fcaf75c4d41b
branch: master
author: Stein Karlsen <karlsen.stein at gmail.com>
committer: Tal Einat <taleinat+github at gmail.com>
date: 2019-10-14T13:36:29+03:00
summary:

bpo-32498: urllib.parse.unquote also accepts bytes (GH-7768)

files:
A Misc/NEWS.d/next/Library/2018-06-17-21-02-25.bpo-32498.La3TZz.rst
M Doc/library/urllib.parse.rst
M Lib/test/test_urllib.py
M Lib/urllib/parse.py

diff --git a/Doc/library/urllib.parse.rst b/Doc/library/urllib.parse.rst
index 49276daa7ff43..84d289bc4415c 100644
--- a/Doc/library/urllib.parse.rst
+++ b/Doc/library/urllib.parse.rst
@@ -571,7 +571,7 @@ task isn't already covered by the URL parsing functions above.
    percent-encoded sequences into Unicode characters, as accepted by the
    :meth:`bytes.decode` method.
 
-   *string* must be a :class:`str`.
+   *string* may be either a :class:`str` or a :class:`bytes`.
 
    *encoding* defaults to ``'utf-8'``.
    *errors* defaults to ``'replace'``, meaning invalid sequences are replaced
@@ -579,6 +579,11 @@ task isn't already covered by the URL parsing functions above.
 
    Example: ``unquote('/El%20Ni%C3%B1o/')`` yields ``'/El Ni?o/'``.
 
+   .. versionchanged:: 3.9
+      *string* parameter supports bytes and str objects (previously only str).
+
+
+
 
 .. function:: unquote_plus(string, encoding='utf-8', errors='replace')
 
diff --git a/Lib/test/test_urllib.py b/Lib/test/test_urllib.py
index 9a6b5f66b7a13..3f59c66084593 100644
--- a/Lib/test/test_urllib.py
+++ b/Lib/test/test_urllib.py
@@ -1049,8 +1049,6 @@ def test_unquoting(self):
                          "%s" % result)
         self.assertRaises((TypeError, AttributeError), urllib.parse.unquote, None)
         self.assertRaises((TypeError, AttributeError), urllib.parse.unquote, ())
-        with support.check_warnings(('', BytesWarning), quiet=True):
-            self.assertRaises((TypeError, AttributeError), urllib.parse.unquote, b'')
 
     def test_unquoting_badpercent(self):
         # Test unquoting on bad percent-escapes
@@ -1210,6 +1208,29 @@ def test_unquote_with_unicode(self):
         self.assertEqual(expect, result,
                          "using unquote(): %r != %r" % (expect, result))
 
+    def test_unquoting_with_bytes_input(self):
+        # ASCII characters decoded to a string
+        given = b'blueberryjam'
+        expect = 'blueberryjam'
+        result = urllib.parse.unquote(given)
+        self.assertEqual(expect, result,
+                         "using unquote(): %r != %r" % (expect, result))
+
+        # A mix of non-ASCII hex-encoded characters and ASCII characters
+        given = b'bl\xc3\xa5b\xc3\xa6rsyltet\xc3\xb8y'
+        expect = 'bl\u00e5b\u00e6rsyltet\u00f8y'
+        result = urllib.parse.unquote(given)
+        self.assertEqual(expect, result,
+                         "using unquote(): %r != %r" % (expect, result))
+
+        # A mix of non-ASCII percent-encoded characters and ASCII characters
+        given = b'bl%c3%a5b%c3%a6rsyltet%c3%b8j'
+        expect = 'bl\u00e5b\u00e6rsyltet\u00f8j'
+        result = urllib.parse.unquote(given)
+        self.assertEqual(expect, result,
+                         "using unquote(): %r != %r" % (expect, result))
+
+
 class urlencode_Tests(unittest.TestCase):
     """Tests for urlencode()"""
 
diff --git a/Lib/urllib/parse.py b/Lib/urllib/parse.py
index b6608783a8947..3a38dc14c9047 100644
--- a/Lib/urllib/parse.py
+++ b/Lib/urllib/parse.py
@@ -631,6 +631,8 @@ def unquote(string, encoding='utf-8', errors='replace'):
 
     unquote('abc%20def') -> 'abc def'.
     """
+    if isinstance(string, bytes):
+        return unquote_to_bytes(string).decode(encoding, errors)
     if '%' not in string:
         string.split
         return string
diff --git a/Misc/NEWS.d/next/Library/2018-06-17-21-02-25.bpo-32498.La3TZz.rst b/Misc/NEWS.d/next/Library/2018-06-17-21-02-25.bpo-32498.La3TZz.rst
new file mode 100644
index 0000000000000..9df9e65e55b3e
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2018-06-17-21-02-25.bpo-32498.La3TZz.rst
@@ -0,0 +1,2 @@
+Made :func:`urllib.parse.unquote()` accept bytes in addition to strings.
+Patch by Stein Karlsen.


From webhook-mailer at python.org  Mon Oct 14 08:16:25 2019
From: webhook-mailer at python.org (Ned Deily)
Date: Mon, 14 Oct 2019 12:16:25 -0000
Subject: [Python-checkins] Update build docs for macOS (GH-16771)
Message-ID: <mailman.12.1571055385.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/19d6842c15c145f65178f806048ac1a8ae7d05f6
commit: 19d6842c15c145f65178f806048ac1a8ae7d05f6
branch: master
author: Ned Deily <nad at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-14T08:16:10-04:00
summary:

Update build docs for macOS (GH-16771)

files:
A Mac/BuildScript/README.rst
A Mac/README.rst
D Mac/BuildScript/README.txt
D Mac/README
M README.rst

diff --git a/Mac/BuildScript/README.txt b/Mac/BuildScript/README.rst
similarity index 100%
rename from Mac/BuildScript/README.txt
rename to Mac/BuildScript/README.rst
diff --git a/Mac/README b/Mac/README.rst
similarity index 74%
rename from Mac/README
rename to Mac/README.rst
index d4194747ebb05..4f2e2ce6623df 100644
--- a/Mac/README
+++ b/Mac/README.rst
@@ -1,25 +1,23 @@
-=========================
-Python on Mac OS X README
-=========================
+======================
+Python on macOS README
+======================
 
 :Authors:
     Jack Jansen (2004-07),
     Ronald Oussoren (2010-04),
     Ned Deily (2012-06)
 
-:Version: 3.4.0
-
-This document provides a quick overview of some Mac OS X specific features in
+This document provides a quick overview of some macOS specific features in
 the Python distribution.
 
-OS X specific arguments to configure
-====================================
+macOS specific arguments to configure
+=====================================
 
 * ``--enable-framework[=DIR]``
 
   If this argument is specified the build will create a Python.framework rather
   than a traditional Unix install. See the section
-  _`Building and using a framework-based Python on Mac OS X` for more
+  _`Building and using a framework-based Python on macOS` for more
   information on frameworks.
 
   If the optional directory argument is specified the framework is installed
@@ -43,41 +41,51 @@ OS X specific arguments to configure
   Create a universal binary build of Python. This can be used with both
   regular and framework builds.
 
-  The optional argument specifies which OS X SDK should be used to perform the
-  build.  If xcodebuild is available and configured, this defaults to
-  the Xcode default MacOS X SDK, otherwise ``/Developer/SDKs/MacOSX.10.4u.sdk``
-  if available or ``/`` if not.  When building on OS X 10.5 or later, you can
-  specify ``/`` to use the installed system headers rather than an SDK.  As of
-  OS X 10.9, you should install the optional system headers from the Command
-  Line Tools component using ``xcode-select``::
-
-     $ sudo xcode-select --install
-
-  See the section _`Building and using a universal binary of Python on Mac OS X`
-  for more information.
+  The optional argument specifies which macOS SDK should be used to perform the
+  build.  In most cases on current systems, you do not need to specify PATH or
+  you can just use ``/``; the default MacOSX SDK for the active Xcode or Command
+  Line Tools developer directory will be used.  See the macOS ``xcrun`` man page
+  for more information.  Current versions of macOS and Xcode no longer install
+  system header files in their traditional locations, like ``/usr/include`` and
+  ``/System/Library/Frameworks``; instead they are found within a MacOSX SDK.
+  The Apple-supplied build tools handle this transparently and current
+  versiona of Python now handle this as well.  So it is no longer necessary,
+  and since macOS 10.14, no longer possible to force the installation of system
+  headers with ``xcode-select``.
 
 * ``--with-universal-archs=VALUE``
 
   Specify the kind of universal binary that should be created. This option is
   only valid when ``--enable-universalsdk`` is specified.  The default is
-  ``32-bit`` if a building with a SDK that supports PPC, otherwise defaults
-  to ``intel``.
+  ``32-bit`` if building with a SDK that supports PPC, otherwise defaults
+  to ``intel``.  Note that ``intel`` means a universal build of both 32-bit
+  and 64-bit binaries and that may not be what you want; for example,
+  as of macOS 10.15 Catalina, 32-bit execution is no longer supported by
+  the operating system.  Thus it is best to either explicitly specify
+  values for ``--with-universal-archs``:
 
+      ``--enable-universalsdk --with-universal-archs=intel-64``
 
-Building and using a universal binary of Python on Mac OS X
-===========================================================
+  or avoid using either.
+
+
+Building and using a universal binary of Python on macOS
+========================================================
 
 1. What is a universal binary
 -----------------------------
 
 A universal binary build of Python contains object code for more than one
-CPU architecture.  A universal OS X executable file or library combines the
+CPU architecture.  A universal macOS executable file or library combines the
 architecture-specific code into one file and can therefore run at native
 speed on all supported architectures.  Universal files were introduced in
-OS X 10.4 to add support for Intel-based Macs to the existing PowerPC (PPC)
-machines.  In OS X 10.5 support was extended to 64-bit Intel and 64-bit PPC
+macOS 10.4 to add support for Intel-based Macs to the existing PowerPC (PPC)
+machines.  In macOS 10.5 support was extended to 64-bit Intel and 64-bit PPC
 architectures.  It is possible to build Python with various combinations
-of architectures depending on the build tools and OS X version in use.
+of architectures depending on the build tools and macOS version in use.
+Note that PPC support was removed in macOS 10.7 and 32-bit Intel support
+was removed in macOS 10.15.  So currently as of macOS 10.15, macOS only
+supports one execution architecture, 64-bit Intel (``x86_64``).
 
 2. How do I build a universal binary
 ------------------------------------
@@ -90,14 +98,14 @@ flag to configure::
   $ make install
 
 This flag can be used with a framework build of python, but also with a classic
-unix build. Universal builds were first supported with OS X 10.4 with Xcode 2.1
-and the 10.4u SDK.  Starting with Xcode 3 and OS X 10.5, more configurations are
+unix build. Universal builds were first supported with macOS 10.4 with Xcode 2.1
+and the 10.4u SDK.  Starting with Xcode 3 and macOS 10.5, more configurations are
 available.
 
 In general, universal builds depend on specific features provided by the
 Apple-supplied compilers and other build tools included in Apple's Xcode
-development tools.  You should install Xcode and the command line tools
-component appropriate for the OS X release you are running on.  See the
+development tools.  You should install Xcode or the command line tools
+component appropriate for the macOS release you are running on.  See the
 Python Developer's Guide (https://devguide.python.org/setup/)
 for more information.
 
@@ -116,6 +124,8 @@ values are available:
 
   * ``intel-32``: ``i386``
 
+  * ``intel-64``: ``x86_64``
+
   * ``32-bit``:   ``ppc``, ``i386``
 
   * ``3-way``:	  ``i386``, ``x86_64``, ``ppc``
@@ -125,28 +135,29 @@ values are available:
   * ``all``:      ``ppc``, ``ppc64``, ``i386``, ``x86_64``
 
 To build a universal binary that includes a 64-bit architecture, you must build
-on a system running OS X 10.5 or later.  The ``all`` and ``64-bit`` flavors can
+on a system running macOS 10.5 or later.  The ``all`` and ``64-bit`` flavors can
 only be built with a 10.5 SDK because ``ppc64`` support was only included with
-OS X 10.5.  Although legacy ``ppc`` support was included with Xcode 3 on OS X
-10.6, it was removed in Xcode 4, versions of which were released on OS X 10.6
-and which is the standard for OS X 10.7.  To summarize, the
+macOS 10.5.  Although legacy ``ppc`` support was included with Xcode 3 on macOS
+10.6, it was removed in Xcode 4, versions of which were released on macOS 10.6
+and which is the standard for macOS 10.7.  To summarize, the
 following combinations of SDKs and universal-archs flavors are available:
 
   * 10.4u SDK with Xcode 2 supports ``32-bit`` only
 
   * 10.5 SDK with Xcode 3.1.x supports all flavors
 
-  * 10.6 SDK with Xcode 3.2.x supports ``intel``, ``3-way``, and ``32-bit``
+  * 10.6 SDK with Xcode 3.2.x supports ``intel``, ``intel-32``,
+    ``intel-64``, ``3-way``, and ``32-bit``
 
-  * 10.6 SDK with Xcode 4 supports ``intel`` only
+  * 10.6 SDK with Xcode 4 supports ``intel``, ``intel-32``, and ``intel-64``
 
-  * 10.7 and 10.8 SDKs with Xcode 4 support ``intel`` only
+  * 10.7 through 10.14 SDKs support ``intel``, ``intel-32``, and ``intel-64``
 
-  * 10.8 and 10.9 SDKs with Xcode 5 support ``intel`` only
+  * 10.15 and later SDKs support ``intel-64`` only
 
-The makefile for a framework build will also install ``python3.4-32``
+The makefile for a framework build will also install ``python3.x-32``
 binaries when the universal architecture includes at least one 32-bit
-architecture (that is, for all flavors but ``64-bit``).
+architecture (that is, for all flavors but ``64-bit`` and ``intel-64``).
 
 Running a specific architecture
 ...............................
@@ -159,20 +170,15 @@ Or to explicitly run in 32-bit mode, regardless of the machine hardware::
 
    $ arch -i386 -ppc python
 
-NOTE: When you're using a framework install of Python this requires at least
-Python 2.7 or 3.2, in earlier versions the python (and pythonw) commands are
-wrapper tools that execute the real interpreter without ensuring that the
-real interpreter runs with the same architecture.
-
 Using ``arch`` is not a perfect solution as the selected architecture will
 not automatically carry through to subprocesses launched by programs and tests
 under that Python.  If you want to ensure that Python interpreters launched in
 subprocesses also run in 32-bit-mode if the main interpreter does, use
-a ``python3.4-32`` binary and use the value of ``sys.executable`` as the
+a ``python3.x-32`` binary and use the value of ``sys.executable`` as the
 ``subprocess`` ``Popen`` executable value.
 
-Building and using a framework-based Python on Mac OS X.
-========================================================
+Building and using a framework-based Python on macOS
+====================================================
 
 
 1. Why would I want a framework Python instead of a normal static Python?
@@ -180,18 +186,18 @@ Building and using a framework-based Python on Mac OS X.
 
 The main reason is because you want to create GUI programs in Python. With the
 exception of X11/XDarwin-based GUI toolkits all GUI programs need to be run
-from a Mac OS X application bundle (".app").
+from a macOS application bundle (".app").
 
 While it is technically possible to create a .app without using frameworks you
 will have to do the work yourself if you really want this.
 
 A second reason for using frameworks is that they put Python-related items in
 only two places: "/Library/Framework/Python.framework" and
-"/Applications/Python <VERSION>" where ``<VERSION>`` can be e.g. "3.4",
+"/Applications/Python <VERSION>" where ``<VERSION>`` can be e.g. "3.8",
 "2.7", etc.  This simplifies matters for users installing
 Python from a binary distribution if they want to get rid of it again. Moreover,
-due to the way frameworks work, a user without admin privileges can install a
-binary distribution in his or her home directory without recompilation.
+due to the way frameworks work, usera without admin privileges can install a
+binary distribution in their home directory without recompilation.
 
 2. How does a framework Python differ from a normal static Python?
 ------------------------------------------------------------------
@@ -205,12 +211,12 @@ Versions/Current and you will see the familiar bin and lib directories.
 3. Do I need extra packages?
 ----------------------------
 
-Yes, probably.  If you want Tkinter support you need to get the OS X AquaTk
-distribution, this is installed by default on Mac OS X 10.4 or later.  Be
-aware, though, that the Cocoa-based AquaTk's supplied starting with OS X
+Yes, probably.  If you want Tkinter support you need to get the macOS AquaTk
+distribution, this is installed by default on macOS 10.4 or later.  Be
+aware, though, that the Cocoa-based AquaTk's supplied starting with macOS
 10.6 have proven to be unstable.  If possible, you should consider
-installing a newer version before building on OS X 10.6 or later, such as
-the ActiveTcl 8.5.  See http://www.python.org/download/mac/tcltk/.  If you
+installing a newer version before building on macOS 10.6 or later, such as
+the ActiveTcl 8.6.  See http://www.python.org/download/mac/tcltk/.  If you
 are building with an SDK, ensure that the newer Tcl and Tk frameworks are
 seen in the SDK's ``Library/Frameworks`` directory; you may need to
 manually create symlinks to their installed location, ``/Library/Frameworks``.
@@ -221,7 +227,7 @@ If you want Cocoa you need to get PyObjC.
 -------------------------------------
 
 This directory contains a Makefile that will create a couple of python-related
-applications (full-blown OS X .app applications, that is) in
+applications (full-blown macOS .app applications, that is) in
 "/Applications/Python <VERSION>", and a hidden helper application Python.app
 inside the Python.framework, and unix tools including "python" into
 /usr/local/bin.  In addition it has a target "installmacsubtree" that installs
@@ -271,7 +277,7 @@ through Python Launcher's preferences dialog.
 
 The program ``pythonx.x`` runs python scripts from the command line.
 Previously, various compatibility aliases were also installed, including
-``pythonwx.x`` which in early releases of Python on OS X was required to run
+``pythonwx.x`` which in early releases of Python on macOS was required to run
 GUI programs.  As of 3.4.0, the ``pythonwx.x`` aliases are no longer installed.
 
 How do I create a binary distribution?
@@ -288,18 +294,17 @@ installer package will create links to the documentation for use by IDLE,
 pydoc, shell users, and Finder user.
 
 The script will build a universal binary so you'll therefore have to run this
-script on Mac OS X 10.4 or later and with Xcode 2.1 or later installed.
+script on macOS 10.4 or later and with Xcode 2.1 or later installed.
 However, the Python build process itself has several build dependencies not
-available out of the box with OS X 10.4 so you may have to install
-additional software beyond what is provided with Xcode 2.  OS X 10.5
-provides a recent enough system Python (in ``/usr/bin``) to build
-the Python documentation set.  It should be possible to use SDKs and/or older
+available out of the box with macOS 10.4 so you may have to install
+additional software beyond what is provided with Xcode 2.
+It should be possible to use SDKs and/or older
 versions of Xcode to build installers that are compatible with older systems
 on a newer system but this may not be completely foolproof so the resulting
 executables, shared libraries, and ``.so`` bundles should be carefully
 examined and tested on all supported systems for proper dynamic linking
 dependencies.  It is safest to build the distribution on a system running the
-minimum OS X version supported.
+minimum macOS version supported.
 
 All of this is normally done completely isolated in /tmp/_py, so it does not
 use your normal build directory nor does it install into /.
@@ -333,7 +338,7 @@ Uninstalling a framework install, including the binary installer
 
 Uninstalling a framework can be done by manually removing all bits that got installed.
 That's true for both installations from source and installations using the binary installer.
-OS X does not provide a central uninstaller.
+macOS does not provide a central uninstaller.
 
 The main bit of a framework install is the framework itself, installed in
 ``/Library/Frameworks/Python.framework``. This can contain multiple versions
diff --git a/README.rst b/README.rst
index ca397b896e2dc..dfc8566d5ec40 100644
--- a/README.rst
+++ b/README.rst
@@ -62,18 +62,20 @@ On Unix, Linux, BSD, macOS, and Cygwin::
 This will install Python as ``python3``.
 
 You can pass many options to the configure script; run ``./configure --help``
-to find out more.  On macOS and Cygwin, the executable is called ``python.exe``;
-elsewhere it's just ``python``.
+to find out more.  On macOS case-insensitive file systems and on Cygwin,
+the executable is called ``python.exe``; elsewhere it's just ``python``.
 
-If you are running on macOS with the latest updates installed, make sure to install
-OpenSSL or some other SSL software along with Homebrew or another package manager.
-If issues persist, see https://devguide.python.org/setup/#macos-and-os-x for more
-information.
+Building a complete Python installation requires the use of various
+additional third-party libraries, depending on your build platform and
+configure options.  Not all standard library modules are buildable or
+useable on all platforms.  Refer to the
+`Install dependencies <https://devguide.python.org/setup/#install-dependencies>`_
+section of the `Developer Guide`_ for current detailed information on
+dependencies for various Linux distributions and macOS.
 
-On macOS, if you have configured Python with ``--enable-framework``, you
-should use ``make frameworkinstall`` to do the installation.  Note that this
-installs the Python executable in a place that is not normally on your PATH,
-you may want to set up a symlink in ``/usr/local/bin``.
+On macOS, there are additional configure and build options related
+to macOS framework and universal builds.  Refer to `Mac/README.rst
+<https://github.com/python/cpython/blob/master/Mac/README.rst>`_.
 
 On Windows, see `PCbuild/readme.txt
 <https://github.com/python/cpython/blob/master/PCbuild/readme.txt>`_.
@@ -96,7 +98,6 @@ Profile Guided Optimization (PGO) and may be used to auto-enable Link Time
 Optimization (LTO) on some platforms.  For more details, see the sections
 below.
 
-
 Profile Guided Optimization
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 


From webhook-mailer at python.org  Mon Oct 14 08:35:32 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Mon, 14 Oct 2019 12:35:32 -0000
Subject: [Python-checkins] Update build docs for macOS (GH-16771)
Message-ID: <mailman.13.1571056534.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/80f8532f0ff22d3473cb6cdb11d5f37ae357f493
commit: 80f8532f0ff22d3473cb6cdb11d5f37ae357f493
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-14T05:35:28-07:00
summary:

Update build docs for macOS (GH-16771)

(cherry picked from commit 19d6842c15c145f65178f806048ac1a8ae7d05f6)

Co-authored-by: Ned Deily <nad at python.org>

files:
A Mac/BuildScript/README.rst
A Mac/README.rst
D Mac/BuildScript/README.txt
D Mac/README
M README.rst

diff --git a/Mac/BuildScript/README.txt b/Mac/BuildScript/README.rst
similarity index 100%
rename from Mac/BuildScript/README.txt
rename to Mac/BuildScript/README.rst
diff --git a/Mac/README b/Mac/README.rst
similarity index 74%
rename from Mac/README
rename to Mac/README.rst
index d4194747ebb05..4f2e2ce6623df 100644
--- a/Mac/README
+++ b/Mac/README.rst
@@ -1,25 +1,23 @@
-=========================
-Python on Mac OS X README
-=========================
+======================
+Python on macOS README
+======================
 
 :Authors:
     Jack Jansen (2004-07),
     Ronald Oussoren (2010-04),
     Ned Deily (2012-06)
 
-:Version: 3.4.0
-
-This document provides a quick overview of some Mac OS X specific features in
+This document provides a quick overview of some macOS specific features in
 the Python distribution.
 
-OS X specific arguments to configure
-====================================
+macOS specific arguments to configure
+=====================================
 
 * ``--enable-framework[=DIR]``
 
   If this argument is specified the build will create a Python.framework rather
   than a traditional Unix install. See the section
-  _`Building and using a framework-based Python on Mac OS X` for more
+  _`Building and using a framework-based Python on macOS` for more
   information on frameworks.
 
   If the optional directory argument is specified the framework is installed
@@ -43,41 +41,51 @@ OS X specific arguments to configure
   Create a universal binary build of Python. This can be used with both
   regular and framework builds.
 
-  The optional argument specifies which OS X SDK should be used to perform the
-  build.  If xcodebuild is available and configured, this defaults to
-  the Xcode default MacOS X SDK, otherwise ``/Developer/SDKs/MacOSX.10.4u.sdk``
-  if available or ``/`` if not.  When building on OS X 10.5 or later, you can
-  specify ``/`` to use the installed system headers rather than an SDK.  As of
-  OS X 10.9, you should install the optional system headers from the Command
-  Line Tools component using ``xcode-select``::
-
-     $ sudo xcode-select --install
-
-  See the section _`Building and using a universal binary of Python on Mac OS X`
-  for more information.
+  The optional argument specifies which macOS SDK should be used to perform the
+  build.  In most cases on current systems, you do not need to specify PATH or
+  you can just use ``/``; the default MacOSX SDK for the active Xcode or Command
+  Line Tools developer directory will be used.  See the macOS ``xcrun`` man page
+  for more information.  Current versions of macOS and Xcode no longer install
+  system header files in their traditional locations, like ``/usr/include`` and
+  ``/System/Library/Frameworks``; instead they are found within a MacOSX SDK.
+  The Apple-supplied build tools handle this transparently and current
+  versiona of Python now handle this as well.  So it is no longer necessary,
+  and since macOS 10.14, no longer possible to force the installation of system
+  headers with ``xcode-select``.
 
 * ``--with-universal-archs=VALUE``
 
   Specify the kind of universal binary that should be created. This option is
   only valid when ``--enable-universalsdk`` is specified.  The default is
-  ``32-bit`` if a building with a SDK that supports PPC, otherwise defaults
-  to ``intel``.
+  ``32-bit`` if building with a SDK that supports PPC, otherwise defaults
+  to ``intel``.  Note that ``intel`` means a universal build of both 32-bit
+  and 64-bit binaries and that may not be what you want; for example,
+  as of macOS 10.15 Catalina, 32-bit execution is no longer supported by
+  the operating system.  Thus it is best to either explicitly specify
+  values for ``--with-universal-archs``:
 
+      ``--enable-universalsdk --with-universal-archs=intel-64``
 
-Building and using a universal binary of Python on Mac OS X
-===========================================================
+  or avoid using either.
+
+
+Building and using a universal binary of Python on macOS
+========================================================
 
 1. What is a universal binary
 -----------------------------
 
 A universal binary build of Python contains object code for more than one
-CPU architecture.  A universal OS X executable file or library combines the
+CPU architecture.  A universal macOS executable file or library combines the
 architecture-specific code into one file and can therefore run at native
 speed on all supported architectures.  Universal files were introduced in
-OS X 10.4 to add support for Intel-based Macs to the existing PowerPC (PPC)
-machines.  In OS X 10.5 support was extended to 64-bit Intel and 64-bit PPC
+macOS 10.4 to add support for Intel-based Macs to the existing PowerPC (PPC)
+machines.  In macOS 10.5 support was extended to 64-bit Intel and 64-bit PPC
 architectures.  It is possible to build Python with various combinations
-of architectures depending on the build tools and OS X version in use.
+of architectures depending on the build tools and macOS version in use.
+Note that PPC support was removed in macOS 10.7 and 32-bit Intel support
+was removed in macOS 10.15.  So currently as of macOS 10.15, macOS only
+supports one execution architecture, 64-bit Intel (``x86_64``).
 
 2. How do I build a universal binary
 ------------------------------------
@@ -90,14 +98,14 @@ flag to configure::
   $ make install
 
 This flag can be used with a framework build of python, but also with a classic
-unix build. Universal builds were first supported with OS X 10.4 with Xcode 2.1
-and the 10.4u SDK.  Starting with Xcode 3 and OS X 10.5, more configurations are
+unix build. Universal builds were first supported with macOS 10.4 with Xcode 2.1
+and the 10.4u SDK.  Starting with Xcode 3 and macOS 10.5, more configurations are
 available.
 
 In general, universal builds depend on specific features provided by the
 Apple-supplied compilers and other build tools included in Apple's Xcode
-development tools.  You should install Xcode and the command line tools
-component appropriate for the OS X release you are running on.  See the
+development tools.  You should install Xcode or the command line tools
+component appropriate for the macOS release you are running on.  See the
 Python Developer's Guide (https://devguide.python.org/setup/)
 for more information.
 
@@ -116,6 +124,8 @@ values are available:
 
   * ``intel-32``: ``i386``
 
+  * ``intel-64``: ``x86_64``
+
   * ``32-bit``:   ``ppc``, ``i386``
 
   * ``3-way``:	  ``i386``, ``x86_64``, ``ppc``
@@ -125,28 +135,29 @@ values are available:
   * ``all``:      ``ppc``, ``ppc64``, ``i386``, ``x86_64``
 
 To build a universal binary that includes a 64-bit architecture, you must build
-on a system running OS X 10.5 or later.  The ``all`` and ``64-bit`` flavors can
+on a system running macOS 10.5 or later.  The ``all`` and ``64-bit`` flavors can
 only be built with a 10.5 SDK because ``ppc64`` support was only included with
-OS X 10.5.  Although legacy ``ppc`` support was included with Xcode 3 on OS X
-10.6, it was removed in Xcode 4, versions of which were released on OS X 10.6
-and which is the standard for OS X 10.7.  To summarize, the
+macOS 10.5.  Although legacy ``ppc`` support was included with Xcode 3 on macOS
+10.6, it was removed in Xcode 4, versions of which were released on macOS 10.6
+and which is the standard for macOS 10.7.  To summarize, the
 following combinations of SDKs and universal-archs flavors are available:
 
   * 10.4u SDK with Xcode 2 supports ``32-bit`` only
 
   * 10.5 SDK with Xcode 3.1.x supports all flavors
 
-  * 10.6 SDK with Xcode 3.2.x supports ``intel``, ``3-way``, and ``32-bit``
+  * 10.6 SDK with Xcode 3.2.x supports ``intel``, ``intel-32``,
+    ``intel-64``, ``3-way``, and ``32-bit``
 
-  * 10.6 SDK with Xcode 4 supports ``intel`` only
+  * 10.6 SDK with Xcode 4 supports ``intel``, ``intel-32``, and ``intel-64``
 
-  * 10.7 and 10.8 SDKs with Xcode 4 support ``intel`` only
+  * 10.7 through 10.14 SDKs support ``intel``, ``intel-32``, and ``intel-64``
 
-  * 10.8 and 10.9 SDKs with Xcode 5 support ``intel`` only
+  * 10.15 and later SDKs support ``intel-64`` only
 
-The makefile for a framework build will also install ``python3.4-32``
+The makefile for a framework build will also install ``python3.x-32``
 binaries when the universal architecture includes at least one 32-bit
-architecture (that is, for all flavors but ``64-bit``).
+architecture (that is, for all flavors but ``64-bit`` and ``intel-64``).
 
 Running a specific architecture
 ...............................
@@ -159,20 +170,15 @@ Or to explicitly run in 32-bit mode, regardless of the machine hardware::
 
    $ arch -i386 -ppc python
 
-NOTE: When you're using a framework install of Python this requires at least
-Python 2.7 or 3.2, in earlier versions the python (and pythonw) commands are
-wrapper tools that execute the real interpreter without ensuring that the
-real interpreter runs with the same architecture.
-
 Using ``arch`` is not a perfect solution as the selected architecture will
 not automatically carry through to subprocesses launched by programs and tests
 under that Python.  If you want to ensure that Python interpreters launched in
 subprocesses also run in 32-bit-mode if the main interpreter does, use
-a ``python3.4-32`` binary and use the value of ``sys.executable`` as the
+a ``python3.x-32`` binary and use the value of ``sys.executable`` as the
 ``subprocess`` ``Popen`` executable value.
 
-Building and using a framework-based Python on Mac OS X.
-========================================================
+Building and using a framework-based Python on macOS
+====================================================
 
 
 1. Why would I want a framework Python instead of a normal static Python?
@@ -180,18 +186,18 @@ Building and using a framework-based Python on Mac OS X.
 
 The main reason is because you want to create GUI programs in Python. With the
 exception of X11/XDarwin-based GUI toolkits all GUI programs need to be run
-from a Mac OS X application bundle (".app").
+from a macOS application bundle (".app").
 
 While it is technically possible to create a .app without using frameworks you
 will have to do the work yourself if you really want this.
 
 A second reason for using frameworks is that they put Python-related items in
 only two places: "/Library/Framework/Python.framework" and
-"/Applications/Python <VERSION>" where ``<VERSION>`` can be e.g. "3.4",
+"/Applications/Python <VERSION>" where ``<VERSION>`` can be e.g. "3.8",
 "2.7", etc.  This simplifies matters for users installing
 Python from a binary distribution if they want to get rid of it again. Moreover,
-due to the way frameworks work, a user without admin privileges can install a
-binary distribution in his or her home directory without recompilation.
+due to the way frameworks work, usera without admin privileges can install a
+binary distribution in their home directory without recompilation.
 
 2. How does a framework Python differ from a normal static Python?
 ------------------------------------------------------------------
@@ -205,12 +211,12 @@ Versions/Current and you will see the familiar bin and lib directories.
 3. Do I need extra packages?
 ----------------------------
 
-Yes, probably.  If you want Tkinter support you need to get the OS X AquaTk
-distribution, this is installed by default on Mac OS X 10.4 or later.  Be
-aware, though, that the Cocoa-based AquaTk's supplied starting with OS X
+Yes, probably.  If you want Tkinter support you need to get the macOS AquaTk
+distribution, this is installed by default on macOS 10.4 or later.  Be
+aware, though, that the Cocoa-based AquaTk's supplied starting with macOS
 10.6 have proven to be unstable.  If possible, you should consider
-installing a newer version before building on OS X 10.6 or later, such as
-the ActiveTcl 8.5.  See http://www.python.org/download/mac/tcltk/.  If you
+installing a newer version before building on macOS 10.6 or later, such as
+the ActiveTcl 8.6.  See http://www.python.org/download/mac/tcltk/.  If you
 are building with an SDK, ensure that the newer Tcl and Tk frameworks are
 seen in the SDK's ``Library/Frameworks`` directory; you may need to
 manually create symlinks to their installed location, ``/Library/Frameworks``.
@@ -221,7 +227,7 @@ If you want Cocoa you need to get PyObjC.
 -------------------------------------
 
 This directory contains a Makefile that will create a couple of python-related
-applications (full-blown OS X .app applications, that is) in
+applications (full-blown macOS .app applications, that is) in
 "/Applications/Python <VERSION>", and a hidden helper application Python.app
 inside the Python.framework, and unix tools including "python" into
 /usr/local/bin.  In addition it has a target "installmacsubtree" that installs
@@ -271,7 +277,7 @@ through Python Launcher's preferences dialog.
 
 The program ``pythonx.x`` runs python scripts from the command line.
 Previously, various compatibility aliases were also installed, including
-``pythonwx.x`` which in early releases of Python on OS X was required to run
+``pythonwx.x`` which in early releases of Python on macOS was required to run
 GUI programs.  As of 3.4.0, the ``pythonwx.x`` aliases are no longer installed.
 
 How do I create a binary distribution?
@@ -288,18 +294,17 @@ installer package will create links to the documentation for use by IDLE,
 pydoc, shell users, and Finder user.
 
 The script will build a universal binary so you'll therefore have to run this
-script on Mac OS X 10.4 or later and with Xcode 2.1 or later installed.
+script on macOS 10.4 or later and with Xcode 2.1 or later installed.
 However, the Python build process itself has several build dependencies not
-available out of the box with OS X 10.4 so you may have to install
-additional software beyond what is provided with Xcode 2.  OS X 10.5
-provides a recent enough system Python (in ``/usr/bin``) to build
-the Python documentation set.  It should be possible to use SDKs and/or older
+available out of the box with macOS 10.4 so you may have to install
+additional software beyond what is provided with Xcode 2.
+It should be possible to use SDKs and/or older
 versions of Xcode to build installers that are compatible with older systems
 on a newer system but this may not be completely foolproof so the resulting
 executables, shared libraries, and ``.so`` bundles should be carefully
 examined and tested on all supported systems for proper dynamic linking
 dependencies.  It is safest to build the distribution on a system running the
-minimum OS X version supported.
+minimum macOS version supported.
 
 All of this is normally done completely isolated in /tmp/_py, so it does not
 use your normal build directory nor does it install into /.
@@ -333,7 +338,7 @@ Uninstalling a framework install, including the binary installer
 
 Uninstalling a framework can be done by manually removing all bits that got installed.
 That's true for both installations from source and installations using the binary installer.
-OS X does not provide a central uninstaller.
+macOS does not provide a central uninstaller.
 
 The main bit of a framework install is the framework itself, installed in
 ``/Library/Frameworks/Python.framework``. This can contain multiple versions
diff --git a/README.rst b/README.rst
index 48376362ecfa1..74336a4dae47e 100644
--- a/README.rst
+++ b/README.rst
@@ -62,18 +62,20 @@ On Unix, Linux, BSD, macOS, and Cygwin::
 This will install Python as ``python3``.
 
 You can pass many options to the configure script; run ``./configure --help``
-to find out more.  On macOS and Cygwin, the executable is called ``python.exe``;
-elsewhere it's just ``python``.
+to find out more.  On macOS case-insensitive file systems and on Cygwin,
+the executable is called ``python.exe``; elsewhere it's just ``python``.
 
-If you are running on macOS with the latest updates installed, make sure to install
-OpenSSL or some other SSL software along with Homebrew or another package manager.
-If issues persist, see https://devguide.python.org/setup/#macos-and-os-x for more
-information.
+Building a complete Python installation requires the use of various
+additional third-party libraries, depending on your build platform and
+configure options.  Not all standard library modules are buildable or
+useable on all platforms.  Refer to the
+`Install dependencies <https://devguide.python.org/setup/#install-dependencies>`_
+section of the `Developer Guide`_ for current detailed information on
+dependencies for various Linux distributions and macOS.
 
-On macOS, if you have configured Python with ``--enable-framework``, you
-should use ``make frameworkinstall`` to do the installation.  Note that this
-installs the Python executable in a place that is not normally on your PATH,
-you may want to set up a symlink in ``/usr/local/bin``.
+On macOS, there are additional configure and build options related
+to macOS framework and universal builds.  Refer to `Mac/README.rst
+<https://github.com/python/cpython/blob/master/Mac/README.rst>`_.
 
 On Windows, see `PCbuild/readme.txt
 <https://github.com/python/cpython/blob/master/PCbuild/readme.txt>`_.
@@ -96,7 +98,6 @@ Profile Guided Optimization (PGO) and may be used to auto-enable Link Time
 Optimization (LTO) on some platforms.  For more details, see the sections
 below.
 
-
 Profile Guided Optimization
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 


From webhook-mailer at python.org  Mon Oct 14 09:14:25 2019
From: webhook-mailer at python.org (Ned Deily)
Date: Mon, 14 Oct 2019 13:14:25 -0000
Subject: [Python-checkins] Make 3.8 README link to 3.8 doc pages,
 not master (GH-16776)
Message-ID: <mailman.14.1571058866.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/c0c6ae91f214f763e426e969b986773fe0817730
commit: c0c6ae91f214f763e426e969b986773fe0817730
branch: 3.8
author: Ned Deily <nad at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-14T09:14:21-04:00
summary:

Make 3.8 README link to 3.8 doc pages, not master (GH-16776)

files:
M README.rst

diff --git a/README.rst b/README.rst
index 74336a4dae47e..4bc5214480347 100644
--- a/README.rst
+++ b/README.rst
@@ -75,10 +75,10 @@ dependencies for various Linux distributions and macOS.
 
 On macOS, there are additional configure and build options related
 to macOS framework and universal builds.  Refer to `Mac/README.rst
-<https://github.com/python/cpython/blob/master/Mac/README.rst>`_.
+<https://github.com/python/cpython/blob/3.8/Mac/README.rst>`_.
 
 On Windows, see `PCbuild/readme.txt
-<https://github.com/python/cpython/blob/master/PCbuild/readme.txt>`_.
+<https://github.com/python/cpython/blob/3.8/PCbuild/readme.txt>`_.
 
 If you wish, you can create a subdirectory and invoke configure from there.
 For example::
@@ -139,9 +139,9 @@ What's New
 We have a comprehensive overview of the changes in the `What's New in Python
 3.8 <https://docs.python.org/3.8/whatsnew/3.8.html>`_ document.  For a more
 detailed change log, read `Misc/NEWS
-<https://github.com/python/cpython/blob/master/Misc/NEWS.d>`_, but a full
+<https://github.com/python/cpython/blob/3.8/Misc/NEWS.d>`_, but a full
 accounting of changes can only be gleaned from the `commit history
-<https://github.com/python/cpython/commits/master>`_.
+<https://github.com/python/cpython/commits/3.8>`_.
 
 If you want to install multiple versions of Python, see the section below
 entitled "Installing multiple versions".
@@ -159,7 +159,7 @@ is primarily for documentation authors, translators, and people with special
 formatting requirements.
 
 For information about building Python's documentation, refer to `Doc/README.rst
-<https://github.com/python/cpython/blob/master/Doc/README.rst>`_.
+<https://github.com/python/cpython/blob/3.8/Doc/README.rst>`_.
 
 
 Converting From Python 2.x to 3.x


From webhook-mailer at python.org  Mon Oct 14 09:27:07 2019
From: webhook-mailer at python.org (Ned Deily)
Date: Mon, 14 Oct 2019 13:27:07 -0000
Subject: [Python-checkins] [3.7] Update build docs for macOS (GH-16771)
 (GH-16774)
Message-ID: <mailman.15.1571059627.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/d1e05689a96bdc03add1e132ed9cb01b30220e3d
commit: d1e05689a96bdc03add1e132ed9cb01b30220e3d
branch: 3.7
author: Ned Deily <nad at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-14T09:27:03-04:00
summary:

[3.7] Update build docs for macOS (GH-16771) (GH-16774)

files:
A Mac/BuildScript/README.rst
A Mac/README.rst
D Mac/BuildScript/README.txt
D Mac/README
M README.rst

diff --git a/Mac/BuildScript/README.txt b/Mac/BuildScript/README.rst
similarity index 100%
rename from Mac/BuildScript/README.txt
rename to Mac/BuildScript/README.rst
diff --git a/Mac/README b/Mac/README.rst
similarity index 74%
rename from Mac/README
rename to Mac/README.rst
index d4194747ebb05..4f2e2ce6623df 100644
--- a/Mac/README
+++ b/Mac/README.rst
@@ -1,25 +1,23 @@
-=========================
-Python on Mac OS X README
-=========================
+======================
+Python on macOS README
+======================
 
 :Authors:
     Jack Jansen (2004-07),
     Ronald Oussoren (2010-04),
     Ned Deily (2012-06)
 
-:Version: 3.4.0
-
-This document provides a quick overview of some Mac OS X specific features in
+This document provides a quick overview of some macOS specific features in
 the Python distribution.
 
-OS X specific arguments to configure
-====================================
+macOS specific arguments to configure
+=====================================
 
 * ``--enable-framework[=DIR]``
 
   If this argument is specified the build will create a Python.framework rather
   than a traditional Unix install. See the section
-  _`Building and using a framework-based Python on Mac OS X` for more
+  _`Building and using a framework-based Python on macOS` for more
   information on frameworks.
 
   If the optional directory argument is specified the framework is installed
@@ -43,41 +41,51 @@ OS X specific arguments to configure
   Create a universal binary build of Python. This can be used with both
   regular and framework builds.
 
-  The optional argument specifies which OS X SDK should be used to perform the
-  build.  If xcodebuild is available and configured, this defaults to
-  the Xcode default MacOS X SDK, otherwise ``/Developer/SDKs/MacOSX.10.4u.sdk``
-  if available or ``/`` if not.  When building on OS X 10.5 or later, you can
-  specify ``/`` to use the installed system headers rather than an SDK.  As of
-  OS X 10.9, you should install the optional system headers from the Command
-  Line Tools component using ``xcode-select``::
-
-     $ sudo xcode-select --install
-
-  See the section _`Building and using a universal binary of Python on Mac OS X`
-  for more information.
+  The optional argument specifies which macOS SDK should be used to perform the
+  build.  In most cases on current systems, you do not need to specify PATH or
+  you can just use ``/``; the default MacOSX SDK for the active Xcode or Command
+  Line Tools developer directory will be used.  See the macOS ``xcrun`` man page
+  for more information.  Current versions of macOS and Xcode no longer install
+  system header files in their traditional locations, like ``/usr/include`` and
+  ``/System/Library/Frameworks``; instead they are found within a MacOSX SDK.
+  The Apple-supplied build tools handle this transparently and current
+  versiona of Python now handle this as well.  So it is no longer necessary,
+  and since macOS 10.14, no longer possible to force the installation of system
+  headers with ``xcode-select``.
 
 * ``--with-universal-archs=VALUE``
 
   Specify the kind of universal binary that should be created. This option is
   only valid when ``--enable-universalsdk`` is specified.  The default is
-  ``32-bit`` if a building with a SDK that supports PPC, otherwise defaults
-  to ``intel``.
+  ``32-bit`` if building with a SDK that supports PPC, otherwise defaults
+  to ``intel``.  Note that ``intel`` means a universal build of both 32-bit
+  and 64-bit binaries and that may not be what you want; for example,
+  as of macOS 10.15 Catalina, 32-bit execution is no longer supported by
+  the operating system.  Thus it is best to either explicitly specify
+  values for ``--with-universal-archs``:
 
+      ``--enable-universalsdk --with-universal-archs=intel-64``
 
-Building and using a universal binary of Python on Mac OS X
-===========================================================
+  or avoid using either.
+
+
+Building and using a universal binary of Python on macOS
+========================================================
 
 1. What is a universal binary
 -----------------------------
 
 A universal binary build of Python contains object code for more than one
-CPU architecture.  A universal OS X executable file or library combines the
+CPU architecture.  A universal macOS executable file or library combines the
 architecture-specific code into one file and can therefore run at native
 speed on all supported architectures.  Universal files were introduced in
-OS X 10.4 to add support for Intel-based Macs to the existing PowerPC (PPC)
-machines.  In OS X 10.5 support was extended to 64-bit Intel and 64-bit PPC
+macOS 10.4 to add support for Intel-based Macs to the existing PowerPC (PPC)
+machines.  In macOS 10.5 support was extended to 64-bit Intel and 64-bit PPC
 architectures.  It is possible to build Python with various combinations
-of architectures depending on the build tools and OS X version in use.
+of architectures depending on the build tools and macOS version in use.
+Note that PPC support was removed in macOS 10.7 and 32-bit Intel support
+was removed in macOS 10.15.  So currently as of macOS 10.15, macOS only
+supports one execution architecture, 64-bit Intel (``x86_64``).
 
 2. How do I build a universal binary
 ------------------------------------
@@ -90,14 +98,14 @@ flag to configure::
   $ make install
 
 This flag can be used with a framework build of python, but also with a classic
-unix build. Universal builds were first supported with OS X 10.4 with Xcode 2.1
-and the 10.4u SDK.  Starting with Xcode 3 and OS X 10.5, more configurations are
+unix build. Universal builds were first supported with macOS 10.4 with Xcode 2.1
+and the 10.4u SDK.  Starting with Xcode 3 and macOS 10.5, more configurations are
 available.
 
 In general, universal builds depend on specific features provided by the
 Apple-supplied compilers and other build tools included in Apple's Xcode
-development tools.  You should install Xcode and the command line tools
-component appropriate for the OS X release you are running on.  See the
+development tools.  You should install Xcode or the command line tools
+component appropriate for the macOS release you are running on.  See the
 Python Developer's Guide (https://devguide.python.org/setup/)
 for more information.
 
@@ -116,6 +124,8 @@ values are available:
 
   * ``intel-32``: ``i386``
 
+  * ``intel-64``: ``x86_64``
+
   * ``32-bit``:   ``ppc``, ``i386``
 
   * ``3-way``:	  ``i386``, ``x86_64``, ``ppc``
@@ -125,28 +135,29 @@ values are available:
   * ``all``:      ``ppc``, ``ppc64``, ``i386``, ``x86_64``
 
 To build a universal binary that includes a 64-bit architecture, you must build
-on a system running OS X 10.5 or later.  The ``all`` and ``64-bit`` flavors can
+on a system running macOS 10.5 or later.  The ``all`` and ``64-bit`` flavors can
 only be built with a 10.5 SDK because ``ppc64`` support was only included with
-OS X 10.5.  Although legacy ``ppc`` support was included with Xcode 3 on OS X
-10.6, it was removed in Xcode 4, versions of which were released on OS X 10.6
-and which is the standard for OS X 10.7.  To summarize, the
+macOS 10.5.  Although legacy ``ppc`` support was included with Xcode 3 on macOS
+10.6, it was removed in Xcode 4, versions of which were released on macOS 10.6
+and which is the standard for macOS 10.7.  To summarize, the
 following combinations of SDKs and universal-archs flavors are available:
 
   * 10.4u SDK with Xcode 2 supports ``32-bit`` only
 
   * 10.5 SDK with Xcode 3.1.x supports all flavors
 
-  * 10.6 SDK with Xcode 3.2.x supports ``intel``, ``3-way``, and ``32-bit``
+  * 10.6 SDK with Xcode 3.2.x supports ``intel``, ``intel-32``,
+    ``intel-64``, ``3-way``, and ``32-bit``
 
-  * 10.6 SDK with Xcode 4 supports ``intel`` only
+  * 10.6 SDK with Xcode 4 supports ``intel``, ``intel-32``, and ``intel-64``
 
-  * 10.7 and 10.8 SDKs with Xcode 4 support ``intel`` only
+  * 10.7 through 10.14 SDKs support ``intel``, ``intel-32``, and ``intel-64``
 
-  * 10.8 and 10.9 SDKs with Xcode 5 support ``intel`` only
+  * 10.15 and later SDKs support ``intel-64`` only
 
-The makefile for a framework build will also install ``python3.4-32``
+The makefile for a framework build will also install ``python3.x-32``
 binaries when the universal architecture includes at least one 32-bit
-architecture (that is, for all flavors but ``64-bit``).
+architecture (that is, for all flavors but ``64-bit`` and ``intel-64``).
 
 Running a specific architecture
 ...............................
@@ -159,20 +170,15 @@ Or to explicitly run in 32-bit mode, regardless of the machine hardware::
 
    $ arch -i386 -ppc python
 
-NOTE: When you're using a framework install of Python this requires at least
-Python 2.7 or 3.2, in earlier versions the python (and pythonw) commands are
-wrapper tools that execute the real interpreter without ensuring that the
-real interpreter runs with the same architecture.
-
 Using ``arch`` is not a perfect solution as the selected architecture will
 not automatically carry through to subprocesses launched by programs and tests
 under that Python.  If you want to ensure that Python interpreters launched in
 subprocesses also run in 32-bit-mode if the main interpreter does, use
-a ``python3.4-32`` binary and use the value of ``sys.executable`` as the
+a ``python3.x-32`` binary and use the value of ``sys.executable`` as the
 ``subprocess`` ``Popen`` executable value.
 
-Building and using a framework-based Python on Mac OS X.
-========================================================
+Building and using a framework-based Python on macOS
+====================================================
 
 
 1. Why would I want a framework Python instead of a normal static Python?
@@ -180,18 +186,18 @@ Building and using a framework-based Python on Mac OS X.
 
 The main reason is because you want to create GUI programs in Python. With the
 exception of X11/XDarwin-based GUI toolkits all GUI programs need to be run
-from a Mac OS X application bundle (".app").
+from a macOS application bundle (".app").
 
 While it is technically possible to create a .app without using frameworks you
 will have to do the work yourself if you really want this.
 
 A second reason for using frameworks is that they put Python-related items in
 only two places: "/Library/Framework/Python.framework" and
-"/Applications/Python <VERSION>" where ``<VERSION>`` can be e.g. "3.4",
+"/Applications/Python <VERSION>" where ``<VERSION>`` can be e.g. "3.8",
 "2.7", etc.  This simplifies matters for users installing
 Python from a binary distribution if they want to get rid of it again. Moreover,
-due to the way frameworks work, a user without admin privileges can install a
-binary distribution in his or her home directory without recompilation.
+due to the way frameworks work, usera without admin privileges can install a
+binary distribution in their home directory without recompilation.
 
 2. How does a framework Python differ from a normal static Python?
 ------------------------------------------------------------------
@@ -205,12 +211,12 @@ Versions/Current and you will see the familiar bin and lib directories.
 3. Do I need extra packages?
 ----------------------------
 
-Yes, probably.  If you want Tkinter support you need to get the OS X AquaTk
-distribution, this is installed by default on Mac OS X 10.4 or later.  Be
-aware, though, that the Cocoa-based AquaTk's supplied starting with OS X
+Yes, probably.  If you want Tkinter support you need to get the macOS AquaTk
+distribution, this is installed by default on macOS 10.4 or later.  Be
+aware, though, that the Cocoa-based AquaTk's supplied starting with macOS
 10.6 have proven to be unstable.  If possible, you should consider
-installing a newer version before building on OS X 10.6 or later, such as
-the ActiveTcl 8.5.  See http://www.python.org/download/mac/tcltk/.  If you
+installing a newer version before building on macOS 10.6 or later, such as
+the ActiveTcl 8.6.  See http://www.python.org/download/mac/tcltk/.  If you
 are building with an SDK, ensure that the newer Tcl and Tk frameworks are
 seen in the SDK's ``Library/Frameworks`` directory; you may need to
 manually create symlinks to their installed location, ``/Library/Frameworks``.
@@ -221,7 +227,7 @@ If you want Cocoa you need to get PyObjC.
 -------------------------------------
 
 This directory contains a Makefile that will create a couple of python-related
-applications (full-blown OS X .app applications, that is) in
+applications (full-blown macOS .app applications, that is) in
 "/Applications/Python <VERSION>", and a hidden helper application Python.app
 inside the Python.framework, and unix tools including "python" into
 /usr/local/bin.  In addition it has a target "installmacsubtree" that installs
@@ -271,7 +277,7 @@ through Python Launcher's preferences dialog.
 
 The program ``pythonx.x`` runs python scripts from the command line.
 Previously, various compatibility aliases were also installed, including
-``pythonwx.x`` which in early releases of Python on OS X was required to run
+``pythonwx.x`` which in early releases of Python on macOS was required to run
 GUI programs.  As of 3.4.0, the ``pythonwx.x`` aliases are no longer installed.
 
 How do I create a binary distribution?
@@ -288,18 +294,17 @@ installer package will create links to the documentation for use by IDLE,
 pydoc, shell users, and Finder user.
 
 The script will build a universal binary so you'll therefore have to run this
-script on Mac OS X 10.4 or later and with Xcode 2.1 or later installed.
+script on macOS 10.4 or later and with Xcode 2.1 or later installed.
 However, the Python build process itself has several build dependencies not
-available out of the box with OS X 10.4 so you may have to install
-additional software beyond what is provided with Xcode 2.  OS X 10.5
-provides a recent enough system Python (in ``/usr/bin``) to build
-the Python documentation set.  It should be possible to use SDKs and/or older
+available out of the box with macOS 10.4 so you may have to install
+additional software beyond what is provided with Xcode 2.
+It should be possible to use SDKs and/or older
 versions of Xcode to build installers that are compatible with older systems
 on a newer system but this may not be completely foolproof so the resulting
 executables, shared libraries, and ``.so`` bundles should be carefully
 examined and tested on all supported systems for proper dynamic linking
 dependencies.  It is safest to build the distribution on a system running the
-minimum OS X version supported.
+minimum macOS version supported.
 
 All of this is normally done completely isolated in /tmp/_py, so it does not
 use your normal build directory nor does it install into /.
@@ -333,7 +338,7 @@ Uninstalling a framework install, including the binary installer
 
 Uninstalling a framework can be done by manually removing all bits that got installed.
 That's true for both installations from source and installations using the binary installer.
-OS X does not provide a central uninstaller.
+macOS does not provide a central uninstaller.
 
 The main bit of a framework install is the framework itself, installed in
 ``/Library/Frameworks/Python.framework``. This can contain multiple versions
diff --git a/README.rst b/README.rst
index aaacaa0c52a0d..65a5d4929d61b 100644
--- a/README.rst
+++ b/README.rst
@@ -59,21 +59,23 @@ On Unix, Linux, BSD, macOS, and Cygwin::
 This will install Python as ``python3``.
 
 You can pass many options to the configure script; run ``./configure --help``
-to find out more.  On macOS and Cygwin, the executable is called ``python.exe``;
-elsewhere it's just ``python``.
+to find out more.  On macOS case-insensitive file systems and on Cygwin,
+the executable is called ``python.exe``; elsewhere it's just ``python``.
 
-If you are running on macOS with the latest updates installed, make sure to install
-openSSL or some other SSL software along with Homebrew or another package manager.
-If issues persist, see https://devguide.python.org/setup/#macos-and-os-x for more 
-information. 
+Building a complete Python installation requires the use of various
+additional third-party libraries, depending on your build platform and
+configure options.  Not all standard library modules are buildable or
+useable on all platforms.  Refer to the
+`Install dependencies <https://devguide.python.org/setup/#install-dependencies>`_
+section of the `Developer Guide`_ for current detailed information on
+dependencies for various Linux distributions and macOS.
 
-On macOS, if you have configured Python with ``--enable-framework``, you
-should use ``make frameworkinstall`` to do the installation.  Note that this
-installs the Python executable in a place that is not normally on your PATH,
-you may want to set up a symlink in ``/usr/local/bin``.
+On macOS, there are additional configure and build options related
+to macOS framework and universal builds.  Refer to `Mac/README.rst
+<https://github.com/python/cpython/blob/3.7/Mac/README.rst>`_.
 
 On Windows, see `PCbuild/readme.txt
-<https://github.com/python/cpython/blob/master/PCbuild/readme.txt>`_.
+<https://github.com/python/cpython/blob/3.7/PCbuild/readme.txt>`_.
 
 If you wish, you can create a subdirectory and invoke configure from there.
 For example::
@@ -135,9 +137,9 @@ What's New
 We have a comprehensive overview of the changes in the `What's New in Python
 3.7 <https://docs.python.org/3.7/whatsnew/3.7.html>`_ document.  For a more
 detailed change log, read `Misc/NEWS
-<https://github.com/python/cpython/blob/master/Misc/NEWS.d>`_, but a full
+<https://github.com/python/cpython/blob/3.7/Misc/NEWS.d>`_, but a full
 accounting of changes can only be gleaned from the `commit history
-<https://github.com/python/cpython/commits/master>`_.
+<https://github.com/python/cpython/commits/3.7>`_.
 
 If you want to install multiple versions of Python see the section below
 entitled "Installing multiple versions".
@@ -155,7 +157,7 @@ is primarily for documentation authors, translators, and people with special
 formatting requirements.
 
 For information about building Python's documentation, refer to `Doc/README.rst
-<https://github.com/python/cpython/blob/master/Doc/README.rst>`_.
+<https://github.com/python/cpython/blob/3.7/Doc/README.rst>`_.
 
 
 Converting From Python 2.x to 3.x


From webhook-mailer at python.org  Mon Oct 14 11:42:26 2019
From: webhook-mailer at python.org (Steve Dower)
Date: Mon, 14 Oct 2019 15:42:26 -0000
Subject: [Python-checkins] bpo-38453: Resolve test directories before chdir
 to them (GH-16723)
Message-ID: <mailman.16.1571067748.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/d83fc2702951f56a7339aa95d62414ed6e0fb40d
commit: d83fc2702951f56a7339aa95d62414ed6e0fb40d
branch: master
author: Steve Dower <steve.dower at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-14T08:42:21-07:00
summary:

bpo-38453: Resolve test directories before chdir to them (GH-16723)

files:
M Lib/test/support/__init__.py

diff --git a/Lib/test/support/__init__.py b/Lib/test/support/__init__.py
index d593fc18d96e1..0f294c5b0f456 100644
--- a/Lib/test/support/__init__.py
+++ b/Lib/test/support/__init__.py
@@ -1068,7 +1068,7 @@ def change_cwd(path, quiet=False):
     """
     saved_dir = os.getcwd()
     try:
-        os.chdir(path)
+        os.chdir(os.path.realpath(path))
     except OSError as exc:
         if not quiet:
             raise


From webhook-mailer at python.org  Mon Oct 14 12:01:09 2019
From: webhook-mailer at python.org (Raymond Hettinger)
Date: Mon, 14 Oct 2019 16:01:09 -0000
Subject: [Python-checkins] Remove draft status. Add asyncio REPL example
 (GH-16785)
Message-ID: <mailman.17.1571068870.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/274bd013da707f9fb4a6f9e531fdc4288a9dd1ca
commit: 274bd013da707f9fb4a6f9e531fdc4288a9dd1ca
branch: master
author: Raymond Hettinger <rhettinger at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-14T09:01:05-07:00
summary:

Remove draft status.  Add asyncio REPL example (GH-16785)

files:
M Doc/whatsnew/3.8.rst

diff --git a/Doc/whatsnew/3.8.rst b/Doc/whatsnew/3.8.rst
index 8108fb7e14704..85351e10e6994 100644
--- a/Doc/whatsnew/3.8.rst
+++ b/Doc/whatsnew/3.8.rst
@@ -47,14 +47,6 @@
 This article explains the new features in Python 3.8, compared to 3.7.
 For full details, see the :ref:`changelog <changelog>`.
 
-Prerelease users should be aware that this document is currently in
-draft form. It will be updated as Python 3.8 moves towards release, so
-it's worth checking back even after reading earlier versions. Some
-notable items not yet covered are:
-
-* :pep:`578` - Runtime audit hooks for potentially sensitive operations
-* ``python -m asyncio`` runs a natively async REPL
-
 .. testsetup::
 
    from datetime import date
@@ -287,6 +279,18 @@ calculations can be shown::
 (Contributed by Eric V. Smith and Larry Hastings in :issue:`36817`.)
 
 
+PEP 578: Python Runtime Audit Hooks
+-----------------------------------
+
+The PEP adds an Audit Hook and Verified Open Hook. Both are available from
+Python and native code, allowing applications and frameworks written in pure
+Python code to take advantage of extra notifications, while also allowing
+embedders or system administrators to deploy builds of Python where auditing is
+always enabled.
+
+See :pep:`578` for full details.
+
+
 PEP 587: Python Initialization Configuration
 --------------------------------------------
 
@@ -584,6 +588,23 @@ The :func:`ast.parse` function has some new flags:
 asyncio
 -------
 
+Running ``python -m asyncio`` launches a natively async REPL.  This allows rapid
+experimentation with code that has a top-level :keyword:`await`.  There is no
+longer a need to directly call ``asyncio.run()`` which would spawn a new event
+loop on every invocation:
+
+.. code-block:: none
+
+    $ python -m asyncio
+    asyncio REPL 3.8.0
+    Use "await" directly instead of "asyncio.run()".
+    Type "help", "copyright", "credits" or "license" for more information.
+    >>> import asyncio
+    >>> await asyncio.sleep(10, result='hello')
+    hello
+
+(Contributed by Yury Selivanov in :issue:`37028`.)
+
 On Windows, the default event loop is now :class:`~asyncio.ProactorEventLoop`.
 (Contributed by Victor Stinner in :issue:`34687`.)
 


From webhook-mailer at python.org  Mon Oct 14 12:01:33 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Mon, 14 Oct 2019 16:01:33 -0000
Subject: [Python-checkins] bpo-38453: Resolve test directories before chdir
 to them (GH-16723)
Message-ID: <mailman.18.1571068894.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/aa909b6b1242b4969b20bb0250ac386f9b4120d7
commit: aa909b6b1242b4969b20bb0250ac386f9b4120d7
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-14T09:01:29-07:00
summary:

bpo-38453: Resolve test directories before chdir to them (GH-16723)

(cherry picked from commit d83fc2702951f56a7339aa95d62414ed6e0fb40d)

Co-authored-by: Steve Dower <steve.dower at python.org>

files:
M Lib/test/support/__init__.py

diff --git a/Lib/test/support/__init__.py b/Lib/test/support/__init__.py
index 2ff7640535e95..d6a7819cb0139 100644
--- a/Lib/test/support/__init__.py
+++ b/Lib/test/support/__init__.py
@@ -1066,7 +1066,7 @@ def change_cwd(path, quiet=False):
     """
     saved_dir = os.getcwd()
     try:
-        os.chdir(path)
+        os.chdir(os.path.realpath(path))
     except OSError as exc:
         if not quiet:
             raise


From webhook-mailer at python.org  Mon Oct 14 12:08:36 2019
From: webhook-mailer at python.org (Raymond Hettinger)
Date: Mon, 14 Oct 2019 16:08:36 -0000
Subject: [Python-checkins] Remove draft status. Add asyncio REPL example
 (GH-16785) (GH-16786)
Message-ID: <mailman.19.1571069317.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/deba4fac5d248e74eade5dbd3c5c50a19f427c73
commit: deba4fac5d248e74eade5dbd3c5c50a19f427c73
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Raymond Hettinger <rhettinger at users.noreply.github.com>
date: 2019-10-14T09:08:30-07:00
summary:

Remove draft status.  Add asyncio REPL example (GH-16785) (GH-16786)

(cherry picked from commit 274bd013da707f9fb4a6f9e531fdc4288a9dd1ca)

Co-authored-by: Raymond Hettinger <rhettinger at users.noreply.github.com>

files:
M Doc/whatsnew/3.8.rst

diff --git a/Doc/whatsnew/3.8.rst b/Doc/whatsnew/3.8.rst
index 8108fb7e14704..85351e10e6994 100644
--- a/Doc/whatsnew/3.8.rst
+++ b/Doc/whatsnew/3.8.rst
@@ -47,14 +47,6 @@
 This article explains the new features in Python 3.8, compared to 3.7.
 For full details, see the :ref:`changelog <changelog>`.
 
-Prerelease users should be aware that this document is currently in
-draft form. It will be updated as Python 3.8 moves towards release, so
-it's worth checking back even after reading earlier versions. Some
-notable items not yet covered are:
-
-* :pep:`578` - Runtime audit hooks for potentially sensitive operations
-* ``python -m asyncio`` runs a natively async REPL
-
 .. testsetup::
 
    from datetime import date
@@ -287,6 +279,18 @@ calculations can be shown::
 (Contributed by Eric V. Smith and Larry Hastings in :issue:`36817`.)
 
 
+PEP 578: Python Runtime Audit Hooks
+-----------------------------------
+
+The PEP adds an Audit Hook and Verified Open Hook. Both are available from
+Python and native code, allowing applications and frameworks written in pure
+Python code to take advantage of extra notifications, while also allowing
+embedders or system administrators to deploy builds of Python where auditing is
+always enabled.
+
+See :pep:`578` for full details.
+
+
 PEP 587: Python Initialization Configuration
 --------------------------------------------
 
@@ -584,6 +588,23 @@ The :func:`ast.parse` function has some new flags:
 asyncio
 -------
 
+Running ``python -m asyncio`` launches a natively async REPL.  This allows rapid
+experimentation with code that has a top-level :keyword:`await`.  There is no
+longer a need to directly call ``asyncio.run()`` which would spawn a new event
+loop on every invocation:
+
+.. code-block:: none
+
+    $ python -m asyncio
+    asyncio REPL 3.8.0
+    Use "await" directly instead of "asyncio.run()".
+    Type "help", "copyright", "credits" or "license" for more information.
+    >>> import asyncio
+    >>> await asyncio.sleep(10, result='hello')
+    hello
+
+(Contributed by Yury Selivanov in :issue:`37028`.)
+
 On Windows, the default event loop is now :class:`~asyncio.ProactorEventLoop`.
 (Contributed by Victor Stinner in :issue:`34687`.)
 


From webhook-mailer at python.org  Mon Oct 14 16:40:37 2019
From: webhook-mailer at python.org (Gregory P. Smith)
Date: Mon, 14 Oct 2019 20:40:37 -0000
Subject: [Python-checkins] [3.8] bpo-38456: Use /bin/true in test_subprocess
 (GH-16737)
Message-ID: <mailman.20.1571085638.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/4f0587f161786318cdfa22c42459676fa42aacb6
commit: 4f0587f161786318cdfa22c42459676fa42aacb6
branch: 3.8
author: Gregory P. Smith <greg at krypto.org>
committer: GitHub <noreply at github.com>
date: 2019-10-14T13:40:32-07:00
summary:

[3.8] bpo-38456: Use /bin/true in test_subprocess (GH-16737)

* [3.8] bpo-38456: Use /bin/true in test_subprocess (GH-16736)

Instead of sys.executable, "-c", "pass" or "import sys; sys.exit(0)"
use /bin/true when it is available.  On a reasonable machine this
shaves up to two seconds wall time off the otherwise ~40sec execution
on a --with-pydebug build.  It should be more notable on many
buildbots or overloaded slower I/O systems (CI, etc)..
(cherry picked from commit 67b93f80c764bca01c81c989d74a99df208bea4d)

* Handle when there is no 'true' command

backport of 46113e0cf32748f66cf64cd633984d143b433cd1 by Pablo Galindo.

files:
M Lib/test/test_subprocess.py

diff --git a/Lib/test/test_subprocess.py b/Lib/test/test_subprocess.py
index 69ac584667d3e..b2afd127dfed8 100644
--- a/Lib/test/test_subprocess.py
+++ b/Lib/test/test_subprocess.py
@@ -45,6 +45,18 @@
 # Ignore errors that indicate the command was not found
 NONEXISTING_ERRORS = (FileNotFoundError, NotADirectoryError, PermissionError)
 
+ZERO_RETURN_CMD = (sys.executable, '-c', 'pass')
+
+
+def setUpModule():
+    shell_true = shutil.which('true')
+    if shell_true is None:
+        return
+    if (os.access(shell_true, os.X_OK) and
+        subprocess.run([shell_true]).returncode == 0):
+        global ZERO_RETURN_CMD
+        ZERO_RETURN_CMD = (shell_true,)  # Faster than Python startup.
+
 
 class BaseTestCase(unittest.TestCase):
     def setUp(self):
@@ -89,7 +101,7 @@ def _execute_child(self, *args, **kwargs):
 class ProcessTestCase(BaseTestCase):
 
     def test_io_buffered_by_default(self):
-        p = subprocess.Popen([sys.executable, "-c", "import sys; sys.exit(0)"],
+        p = subprocess.Popen(ZERO_RETURN_CMD,
                              stdin=subprocess.PIPE, stdout=subprocess.PIPE,
                              stderr=subprocess.PIPE)
         try:
@@ -103,7 +115,7 @@ def test_io_buffered_by_default(self):
             p.wait()
 
     def test_io_unbuffered_works(self):
-        p = subprocess.Popen([sys.executable, "-c", "import sys; sys.exit(0)"],
+        p = subprocess.Popen(ZERO_RETURN_CMD,
                              stdin=subprocess.PIPE, stdout=subprocess.PIPE,
                              stderr=subprocess.PIPE, bufsize=0)
         try:
@@ -133,8 +145,7 @@ def test_call_timeout(self):
 
     def test_check_call_zero(self):
         # check_call() function with zero return code
-        rc = subprocess.check_call([sys.executable, "-c",
-                                    "import sys; sys.exit(0)"])
+        rc = subprocess.check_call(ZERO_RETURN_CMD)
         self.assertEqual(rc, 0)
 
     def test_check_call_nonzero(self):
@@ -700,19 +711,19 @@ def test_invalid_env(self):
         newenv = os.environ.copy()
         newenv["FRUIT\0VEGETABLE"] = "cabbage"
         with self.assertRaises(ValueError):
-            subprocess.Popen([sys.executable, "-c", "pass"], env=newenv)
+            subprocess.Popen(ZERO_RETURN_CMD, env=newenv)
 
         # null character in the environment variable value
         newenv = os.environ.copy()
         newenv["FRUIT"] = "orange\0VEGETABLE=cabbage"
         with self.assertRaises(ValueError):
-            subprocess.Popen([sys.executable, "-c", "pass"], env=newenv)
+            subprocess.Popen(ZERO_RETURN_CMD, env=newenv)
 
         # equal character in the environment variable name
         newenv = os.environ.copy()
         newenv["FRUIT=ORANGE"] = "lemon"
         with self.assertRaises(ValueError):
-            subprocess.Popen([sys.executable, "-c", "pass"], env=newenv)
+            subprocess.Popen(ZERO_RETURN_CMD, env=newenv)
 
         # equal character in the environment variable value
         newenv = os.environ.copy()
@@ -813,7 +824,7 @@ def test_communicate_pipe_fd_leak(self):
                         options['stderr'] = subprocess.PIPE
                     if not options:
                         continue
-                    p = subprocess.Popen((sys.executable, "-c", "pass"), **options)
+                    p = subprocess.Popen(ZERO_RETURN_CMD, **options)
                     p.communicate()
                     if p.stdin is not None:
                         self.assertTrue(p.stdin.closed)
@@ -952,7 +963,7 @@ def test_universal_newlines_communicate_input_none(self):
         #
         # We set stdout to PIPE because, as of this writing, a different
         # code path is tested when the number of pipes is zero or one.
-        p = subprocess.Popen([sys.executable, "-c", "pass"],
+        p = subprocess.Popen(ZERO_RETURN_CMD,
                              stdin=subprocess.PIPE,
                              stdout=subprocess.PIPE,
                              universal_newlines=True)
@@ -1100,7 +1111,7 @@ def test_poll(self):
         self.assertEqual(p.poll(), 0)
 
     def test_wait(self):
-        p = subprocess.Popen([sys.executable, "-c", "pass"])
+        p = subprocess.Popen(ZERO_RETURN_CMD)
         self.assertEqual(p.wait(), 0)
         # Subsequent invocations should just return the returncode
         self.assertEqual(p.wait(), 0)
@@ -1119,14 +1130,14 @@ def test_invalid_bufsize(self):
         # an invalid type of the bufsize argument should raise
         # TypeError.
         with self.assertRaises(TypeError):
-            subprocess.Popen([sys.executable, "-c", "pass"], "orange")
+            subprocess.Popen(ZERO_RETURN_CMD, "orange")
 
     def test_bufsize_is_none(self):
         # bufsize=None should be the same as bufsize=0.
-        p = subprocess.Popen([sys.executable, "-c", "pass"], None)
+        p = subprocess.Popen(ZERO_RETURN_CMD, None)
         self.assertEqual(p.wait(), 0)
         # Again with keyword arg
-        p = subprocess.Popen([sys.executable, "-c", "pass"], bufsize=None)
+        p = subprocess.Popen(ZERO_RETURN_CMD, bufsize=None)
         self.assertEqual(p.wait(), 0)
 
     def _test_bufsize_equal_one(self, line, expected, universal_newlines):
@@ -1331,7 +1342,7 @@ def test_handles_closed_on_exception(self):
 
     def test_communicate_epipe(self):
         # Issue 10963: communicate() should hide EPIPE
-        p = subprocess.Popen([sys.executable, "-c", 'pass'],
+        p = subprocess.Popen(ZERO_RETURN_CMD,
                              stdin=subprocess.PIPE,
                              stdout=subprocess.PIPE,
                              stderr=subprocess.PIPE)
@@ -1342,7 +1353,7 @@ def test_communicate_epipe(self):
 
     def test_communicate_epipe_only_stdin(self):
         # Issue 10963: communicate() should hide EPIPE
-        p = subprocess.Popen([sys.executable, "-c", 'pass'],
+        p = subprocess.Popen(ZERO_RETURN_CMD,
                              stdin=subprocess.PIPE)
         self.addCleanup(p.stdin.close)
         p.wait()
@@ -1381,7 +1392,7 @@ def test_failed_child_execute_fd_leak(self):
         fds_before_popen = os.listdir(fd_directory)
         with self.assertRaises(PopenTestException):
             PopenExecuteChildRaises(
-                    [sys.executable, '-c', 'pass'], stdin=subprocess.PIPE,
+                    ZERO_RETURN_CMD, stdin=subprocess.PIPE,
                     stdout=subprocess.PIPE, stderr=subprocess.PIPE)
 
         # NOTE: This test doesn't verify that the real _execute_child
@@ -1424,7 +1435,7 @@ def test_check(self):
 
     def test_check_zero(self):
         # check_returncode shouldn't raise when returncode is zero
-        cp = self.run_python("import sys; sys.exit(0)", check=True)
+        cp = subprocess.run(ZERO_RETURN_CMD, check=True)
         self.assertEqual(cp.returncode, 0)
 
     def test_timeout(self):
@@ -1824,7 +1835,7 @@ def raise_it():
 
         with self.assertRaises(subprocess.SubprocessError):
             self._TestExecuteChildPopen(
-                        self, [sys.executable, "-c", "pass"],
+                        self, ZERO_RETURN_CMD,
                         stdin=subprocess.PIPE, stdout=subprocess.PIPE,
                         stderr=subprocess.PIPE, preexec_fn=raise_it)
 
@@ -2281,7 +2292,7 @@ def prepare():
 
         try:
             subprocess.call(
-                [sys.executable, "-c", "pass"],
+                ZERO_RETURN_CMD,
                 preexec_fn=prepare)
         except ValueError as err:
             # Pure Python implementations keeps the message
@@ -2324,29 +2335,30 @@ def test_undecodable_env(self):
             self.assertEqual(stdout.decode('ascii'), ascii(encoded_value))
 
     def test_bytes_program(self):
-        abs_program = os.fsencode(sys.executable)
-        path, program = os.path.split(sys.executable)
+        abs_program = os.fsencode(ZERO_RETURN_CMD[0])
+        args = list(ZERO_RETURN_CMD[1:])
+        path, program = os.path.split(ZERO_RETURN_CMD[0])
         program = os.fsencode(program)
 
         # absolute bytes path
-        exitcode = subprocess.call([abs_program, "-c", "pass"])
+        exitcode = subprocess.call([abs_program]+args)
         self.assertEqual(exitcode, 0)
 
         # absolute bytes path as a string
-        cmd = b"'" + abs_program + b"' -c pass"
+        cmd = b"'%s' %s" % (abs_program, " ".join(args).encode("utf-8"))
         exitcode = subprocess.call(cmd, shell=True)
         self.assertEqual(exitcode, 0)
 
         # bytes program, unicode PATH
         env = os.environ.copy()
         env["PATH"] = path
-        exitcode = subprocess.call([program, "-c", "pass"], env=env)
+        exitcode = subprocess.call([program]+args, env=env)
         self.assertEqual(exitcode, 0)
 
         # bytes program, bytes PATH
         envb = os.environb.copy()
         envb[b"PATH"] = os.fsencode(path)
-        exitcode = subprocess.call([program, "-c", "pass"], env=envb)
+        exitcode = subprocess.call([program]+args, env=envb)
         self.assertEqual(exitcode, 0)
 
     def test_pipe_cloexec(self):
@@ -2574,7 +2586,7 @@ def test_pass_fds(self):
             # pass_fds overrides close_fds with a warning.
             with self.assertWarns(RuntimeWarning) as context:
                 self.assertFalse(subprocess.call(
-                        [sys.executable, "-c", "import sys; sys.exit(0)"],
+                        ZERO_RETURN_CMD,
                         close_fds=False, pass_fds=(fd, )))
             self.assertIn('overriding close_fds', str(context.warning))
 
@@ -2636,19 +2648,19 @@ def test_pass_fds_redirected(self):
 
     def test_stdout_stdin_are_single_inout_fd(self):
         with io.open(os.devnull, "r+") as inout:
-            p = subprocess.Popen([sys.executable, "-c", "import sys; sys.exit(0)"],
+            p = subprocess.Popen(ZERO_RETURN_CMD,
                                  stdout=inout, stdin=inout)
             p.wait()
 
     def test_stdout_stderr_are_single_inout_fd(self):
         with io.open(os.devnull, "r+") as inout:
-            p = subprocess.Popen([sys.executable, "-c", "import sys; sys.exit(0)"],
+            p = subprocess.Popen(ZERO_RETURN_CMD,
                                  stdout=inout, stderr=inout)
             p.wait()
 
     def test_stderr_stdin_are_single_inout_fd(self):
         with io.open(os.devnull, "r+") as inout:
-            p = subprocess.Popen([sys.executable, "-c", "import sys; sys.exit(0)"],
+            p = subprocess.Popen(ZERO_RETURN_CMD,
                                  stderr=inout, stdin=inout)
             p.wait()
 
@@ -2836,7 +2848,7 @@ def __int__(self):
     def test_communicate_BrokenPipeError_stdin_close(self):
         # By not setting stdout or stderr or a timeout we force the fast path
         # that just calls _stdin_write() internally due to our mock.
-        proc = subprocess.Popen([sys.executable, '-c', 'pass'])
+        proc = subprocess.Popen(ZERO_RETURN_CMD)
         with proc, mock.patch.object(proc, 'stdin') as mock_proc_stdin:
             mock_proc_stdin.close.side_effect = BrokenPipeError
             proc.communicate()  # Should swallow BrokenPipeError from close.
@@ -2845,7 +2857,7 @@ def test_communicate_BrokenPipeError_stdin_close(self):
     def test_communicate_BrokenPipeError_stdin_write(self):
         # By not setting stdout or stderr or a timeout we force the fast path
         # that just calls _stdin_write() internally due to our mock.
-        proc = subprocess.Popen([sys.executable, '-c', 'pass'])
+        proc = subprocess.Popen(ZERO_RETURN_CMD)
         with proc, mock.patch.object(proc, 'stdin') as mock_proc_stdin:
             mock_proc_stdin.write.side_effect = BrokenPipeError
             proc.communicate(b'stuff')  # Should swallow the BrokenPipeError.
@@ -2884,7 +2896,7 @@ def test_communicate_BrokenPipeError_stdin_close_with_timeout(self):
                          'need _testcapi.W_STOPCODE')
     def test_stopped(self):
         """Test wait() behavior when waitpid returns WIFSTOPPED; issue29335."""
-        args = [sys.executable, '-c', 'pass']
+        args = ZERO_RETURN_CMD
         proc = subprocess.Popen(args)
 
         # Wait until the real process completes to avoid zombie process
@@ -2914,7 +2926,7 @@ def test_startupinfo(self):
         # Since Python is a console process, it won't be affected
         # by wShowWindow, but the argument should be silently
         # ignored
-        subprocess.call([sys.executable, "-c", "import sys; sys.exit(0)"],
+        subprocess.call(ZERO_RETURN_CMD,
                         startupinfo=startupinfo)
 
     def test_startupinfo_keywords(self):
@@ -2930,7 +2942,7 @@ def test_startupinfo_keywords(self):
         # Since Python is a console process, it won't be affected
         # by wShowWindow, but the argument should be silently
         # ignored
-        subprocess.call([sys.executable, "-c", "import sys; sys.exit(0)"],
+        subprocess.call(ZERO_RETURN_CMD,
                         startupinfo=startupinfo)
 
     def test_startupinfo_copy(self):
@@ -2942,7 +2954,7 @@ def test_startupinfo_copy(self):
         # Call Popen() twice with the same startupinfo object to make sure
         # that it's not modified
         for _ in range(2):
-            cmd = [sys.executable, "-c", "pass"]
+            cmd = ZERO_RETURN_CMD
             with open(os.devnull, 'w') as null:
                 proc = subprocess.Popen(cmd,
                                         stdout=null,
@@ -2982,7 +2994,7 @@ def test_issue31471(self):
         class BadEnv(dict):
             keys = None
         with self.assertRaises(TypeError):
-            subprocess.Popen([sys.executable, "-c", "pass"], env=BadEnv())
+            subprocess.Popen(ZERO_RETURN_CMD, env=BadEnv())
 
     def test_close_fds(self):
         # close file descriptors
@@ -3043,13 +3055,13 @@ def test_close_fds_with_stdio(self):
     def test_empty_attribute_list(self):
         startupinfo = subprocess.STARTUPINFO()
         startupinfo.lpAttributeList = {}
-        subprocess.call([sys.executable, "-c", "import sys; sys.exit(0)"],
+        subprocess.call(ZERO_RETURN_CMD,
                         startupinfo=startupinfo)
 
     def test_empty_handle_list(self):
         startupinfo = subprocess.STARTUPINFO()
         startupinfo.lpAttributeList = {"handle_list": []}
-        subprocess.call([sys.executable, "-c", "import sys; sys.exit(0)"],
+        subprocess.call(ZERO_RETURN_CMD,
                         startupinfo=startupinfo)
 
     def test_shell_sequence(self):
@@ -3348,7 +3360,7 @@ def test_invalid_args(self):
 
     def test_broken_pipe_cleanup(self):
         """Broken pipe error should not prevent wait() (Issue 21619)"""
-        proc = subprocess.Popen([sys.executable, '-c', 'pass'],
+        proc = subprocess.Popen(ZERO_RETURN_CMD,
                                 stdin=subprocess.PIPE,
                                 bufsize=support.PIPE_MAX_SIZE*2)
         proc = proc.__enter__()


From webhook-mailer at python.org  Mon Oct 14 18:13:41 2019
From: webhook-mailer at python.org (Ned Deily)
Date: Mon, 14 Oct 2019 22:13:41 -0000
Subject: [Python-checkins] Doc: 3.8 is now stable. (GH-16790)
Message-ID: <mailman.21.1571091223.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/4504b4500d2a1a80c26b27b0bfff8b624d5ce06c
commit: 4504b4500d2a1a80c26b27b0bfff8b624d5ce06c
branch: master
author: Julien Palard <julien at palard.fr>
committer: Ned Deily <nad at python.org>
date: 2019-10-14T18:13:38-04:00
summary:

Doc: 3.8 is now stable. (GH-16790)

files:
M Doc/tools/templates/indexsidebar.html

diff --git a/Doc/tools/templates/indexsidebar.html b/Doc/tools/templates/indexsidebar.html
index 4fd7423430ca8..c51dcc7582e54 100644
--- a/Doc/tools/templates/indexsidebar.html
+++ b/Doc/tools/templates/indexsidebar.html
@@ -3,7 +3,7 @@ <h3>{% trans %}Download{% endtrans %}</h3>
 <h3>{% trans %}Docs by version{% endtrans %}</h3>
 <ul>
   <li><a href="https://docs.python.org/3.9/">{% trans %}Python 3.9 (in development){% endtrans %}</a></li>
-  <li><a href="https://docs.python.org/3.8/">{% trans %}Python 3.8 (pre-release){% endtrans %}</a></li>
+  <li><a href="https://docs.python.org/3.8/">{% trans %}Python 3.8 (stable){% endtrans %}</a></li>
   <li><a href="https://docs.python.org/3.7/">{% trans %}Python 3.7 (stable){% endtrans %}</a></li>
   <li><a href="https://docs.python.org/3.6/">{% trans %}Python 3.6 (security-fixes){% endtrans %}</a></li>
   <li><a href="https://docs.python.org/3.5/">{% trans %}Python 3.5 (security-fixes){% endtrans %}</a></li>


From webhook-mailer at python.org  Mon Oct 14 18:19:55 2019
From: webhook-mailer at python.org (Ned Deily)
Date: Mon, 14 Oct 2019 22:19:55 -0000
Subject: [Python-checkins] Doc: 3.8 is now stable. (GH-16790) (GH-16792)
Message-ID: <mailman.22.1571091596.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/796e8c83e9a5264cf14774ae16503fda96fda990
commit: 796e8c83e9a5264cf14774ae16503fda96fda990
branch: 3.7
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Ned Deily <nad at python.org>
date: 2019-10-14T18:19:51-04:00
summary:

Doc: 3.8 is now stable. (GH-16790) (GH-16792)

(cherry picked from commit 4504b4500d2a1a80c26b27b0bfff8b624d5ce06c)

Co-authored-by: Julien Palard <julien at palard.fr>

files:
M Doc/tools/templates/indexsidebar.html

diff --git a/Doc/tools/templates/indexsidebar.html b/Doc/tools/templates/indexsidebar.html
index 0adaf038d0e52..e5748f2ab0fc6 100644
--- a/Doc/tools/templates/indexsidebar.html
+++ b/Doc/tools/templates/indexsidebar.html
@@ -3,7 +3,7 @@ <h3>{% trans %}Download{% endtrans %}</h3>
 <h3>{% trans %}Docs by version{% endtrans %}</h3>
 <ul>
   <li><a href="https://docs.python.org/3.9/">{% trans %}Python 3.9 (in development){% endtrans %}</a></li>
-  <li><a href="https://docs.python.org/3.8/">{% trans %}Python 3.8 (pre-release){% endtrans %}</a></li>
+  <li><a href="https://docs.python.org/3.8/">{% trans %}Python 3.8 (stable){% endtrans %}</a></li>
   <li><a href="https://docs.python.org/3.7/">{% trans %}Python 3.7 (stable){% endtrans %}</a></li>
   <li><a href="https://docs.python.org/3.6/">{% trans %}Python 3.6 (security-fixes){% endtrans %}</a></li>
   <li><a href="https://docs.python.org/3.5/">{% trans %}Python 3.5 (security-fixes){% endtrans %}</a></li>


From webhook-mailer at python.org  Mon Oct 14 18:21:36 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Mon, 14 Oct 2019 22:21:36 -0000
Subject: [Python-checkins] Doc: 3.8 is now stable. (GH-16790)
Message-ID: <mailman.23.1571091697.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/6fa79f983d7baa48132fe1ac6ebedbdc7ce3fbcc
commit: 6fa79f983d7baa48132fe1ac6ebedbdc7ce3fbcc
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-14T15:21:32-07:00
summary:

Doc: 3.8 is now stable. (GH-16790)

(cherry picked from commit 4504b4500d2a1a80c26b27b0bfff8b624d5ce06c)

Co-authored-by: Julien Palard <julien at palard.fr>

files:
M Doc/tools/templates/indexsidebar.html

diff --git a/Doc/tools/templates/indexsidebar.html b/Doc/tools/templates/indexsidebar.html
index 4fd7423430ca8..c51dcc7582e54 100644
--- a/Doc/tools/templates/indexsidebar.html
+++ b/Doc/tools/templates/indexsidebar.html
@@ -3,7 +3,7 @@ <h3>{% trans %}Download{% endtrans %}</h3>
 <h3>{% trans %}Docs by version{% endtrans %}</h3>
 <ul>
   <li><a href="https://docs.python.org/3.9/">{% trans %}Python 3.9 (in development){% endtrans %}</a></li>
-  <li><a href="https://docs.python.org/3.8/">{% trans %}Python 3.8 (pre-release){% endtrans %}</a></li>
+  <li><a href="https://docs.python.org/3.8/">{% trans %}Python 3.8 (stable){% endtrans %}</a></li>
   <li><a href="https://docs.python.org/3.7/">{% trans %}Python 3.7 (stable){% endtrans %}</a></li>
   <li><a href="https://docs.python.org/3.6/">{% trans %}Python 3.6 (security-fixes){% endtrans %}</a></li>
   <li><a href="https://docs.python.org/3.5/">{% trans %}Python 3.5 (security-fixes){% endtrans %}</a></li>


From webhook-mailer at python.org  Mon Oct 14 18:21:47 2019
From: webhook-mailer at python.org (Ned Deily)
Date: Mon, 14 Oct 2019 22:21:47 -0000
Subject: [Python-checkins] Doc: 3.8 is now stable. (GH-16790) (GH-16793)
Message-ID: <mailman.24.1571091708.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/819ad377766d64428b1ffdc885d953eae86c6502
commit: 819ad377766d64428b1ffdc885d953eae86c6502
branch: 3.6
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Ned Deily <nad at python.org>
date: 2019-10-14T18:21:43-04:00
summary:

Doc: 3.8 is now stable. (GH-16790) (GH-16793)

(cherry picked from commit 4504b4500d2a1a80c26b27b0bfff8b624d5ce06c)

Co-authored-by: Julien Palard <julien at palard.fr>

files:
M Doc/tools/templates/indexsidebar.html

diff --git a/Doc/tools/templates/indexsidebar.html b/Doc/tools/templates/indexsidebar.html
index 0adaf038d0e52..e5748f2ab0fc6 100644
--- a/Doc/tools/templates/indexsidebar.html
+++ b/Doc/tools/templates/indexsidebar.html
@@ -3,7 +3,7 @@ <h3>{% trans %}Download{% endtrans %}</h3>
 <h3>{% trans %}Docs by version{% endtrans %}</h3>
 <ul>
   <li><a href="https://docs.python.org/3.9/">{% trans %}Python 3.9 (in development){% endtrans %}</a></li>
-  <li><a href="https://docs.python.org/3.8/">{% trans %}Python 3.8 (pre-release){% endtrans %}</a></li>
+  <li><a href="https://docs.python.org/3.8/">{% trans %}Python 3.8 (stable){% endtrans %}</a></li>
   <li><a href="https://docs.python.org/3.7/">{% trans %}Python 3.7 (stable){% endtrans %}</a></li>
   <li><a href="https://docs.python.org/3.6/">{% trans %}Python 3.6 (security-fixes){% endtrans %}</a></li>
   <li><a href="https://docs.python.org/3.5/">{% trans %}Python 3.5 (security-fixes){% endtrans %}</a></li>


From webhook-mailer at python.org  Mon Oct 14 18:22:22 2019
From: webhook-mailer at python.org (Ned Deily)
Date: Mon, 14 Oct 2019 22:22:22 -0000
Subject: [Python-checkins] Doc: 3.8 is now stable. (GH-16790) (GH-16794)
Message-ID: <mailman.25.1571091743.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/bef8d9acda7a4b9203e55371bb7a7da1c4a7fcbe
commit: bef8d9acda7a4b9203e55371bb7a7da1c4a7fcbe
branch: 2.7
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Ned Deily <nad at python.org>
date: 2019-10-14T18:22:18-04:00
summary:

Doc: 3.8 is now stable. (GH-16790) (GH-16794)

(cherry picked from commit 4504b4500d2a1a80c26b27b0bfff8b624d5ce06c)

Co-authored-by: Julien Palard <julien at palard.fr>

files:
M Doc/tools/templates/indexsidebar.html

diff --git a/Doc/tools/templates/indexsidebar.html b/Doc/tools/templates/indexsidebar.html
index 0adaf038d0e52..e5748f2ab0fc6 100644
--- a/Doc/tools/templates/indexsidebar.html
+++ b/Doc/tools/templates/indexsidebar.html
@@ -3,7 +3,7 @@ <h3>{% trans %}Download{% endtrans %}</h3>
 <h3>{% trans %}Docs by version{% endtrans %}</h3>
 <ul>
   <li><a href="https://docs.python.org/3.9/">{% trans %}Python 3.9 (in development){% endtrans %}</a></li>
-  <li><a href="https://docs.python.org/3.8/">{% trans %}Python 3.8 (pre-release){% endtrans %}</a></li>
+  <li><a href="https://docs.python.org/3.8/">{% trans %}Python 3.8 (stable){% endtrans %}</a></li>
   <li><a href="https://docs.python.org/3.7/">{% trans %}Python 3.7 (stable){% endtrans %}</a></li>
   <li><a href="https://docs.python.org/3.6/">{% trans %}Python 3.6 (security-fixes){% endtrans %}</a></li>
   <li><a href="https://docs.python.org/3.5/">{% trans %}Python 3.5 (security-fixes){% endtrans %}</a></li>


From webhook-mailer at python.org  Mon Oct 14 18:45:47 2019
From: webhook-mailer at python.org (Steve Dower)
Date: Mon, 14 Oct 2019 22:45:47 -0000
Subject: [Python-checkins] bpo-38133: Update docs to reflect fixes to py.exe
 launcher (GH-16791)
Message-ID: <mailman.26.1571093147.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/4d202281c128e2026e78fc5f4cc752b1dafbf3ad
commit: 4d202281c128e2026e78fc5f4cc752b1dafbf3ad
branch: master
author: Steve Dower <steve.dower at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-14T15:45:43-07:00
summary:

bpo-38133: Update docs to reflect fixes to py.exe launcher (GH-16791)

files:
M Doc/using/windows.rst

diff --git a/Doc/using/windows.rst b/Doc/using/windows.rst
index 0b87f6e048673..4912048a1ab55 100644
--- a/Doc/using/windows.rst
+++ b/Doc/using/windows.rst
@@ -325,6 +325,9 @@ If you have installed another version of Python and added it to your
 one from the Microsoft Store. To access the new installation, use
 ``python3.exe`` or ``python3.x.exe``.
 
+The ``py.exe`` launcher will detect this Python installation, but will prefer
+installations from the traditional installer.
+
 To remove Python, open Settings and use Apps and Features, or else find
 Python in Start and right-click to select Uninstall. Uninstalling will
 remove all packages you installed directly into this Python installation, but
@@ -333,9 +336,6 @@ will not remove any virtual environments
 Known Issues
 ------------
 
-Currently, the ``py.exe`` launcher cannot be used to start Python when it
-has been installed from the Microsoft Store.
-
 Because of restrictions on Microsoft Store apps, Python scripts may not have
 full write access to shared locations such as ``TEMP`` and the registry.
 Instead, it will write to a private copy. If your scripts must modify the


From webhook-mailer at python.org  Mon Oct 14 18:52:52 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Mon, 14 Oct 2019 22:52:52 -0000
Subject: [Python-checkins] bpo-38133: Update docs to reflect fixes to py.exe
 launcher (GH-16791)
Message-ID: <mailman.27.1571093574.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/42308e8b27c8023e3f9d037f5e4a8892f2dcbc48
commit: 42308e8b27c8023e3f9d037f5e4a8892f2dcbc48
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-14T15:52:47-07:00
summary:

bpo-38133: Update docs to reflect fixes to py.exe launcher (GH-16791)

(cherry picked from commit 4d202281c128e2026e78fc5f4cc752b1dafbf3ad)

Co-authored-by: Steve Dower <steve.dower at python.org>

files:
M Doc/using/windows.rst

diff --git a/Doc/using/windows.rst b/Doc/using/windows.rst
index 50eb9d3145d21..9dc3e796cc211 100644
--- a/Doc/using/windows.rst
+++ b/Doc/using/windows.rst
@@ -325,6 +325,9 @@ If you have installed another version of Python and added it to your
 one from the Microsoft Store. To access the new installation, use
 ``python3.exe`` or ``python3.x.exe``.
 
+The ``py.exe`` launcher will detect this Python installation, but will prefer
+installations from the traditional installer.
+
 To remove Python, open Settings and use Apps and Features, or else find
 Python in Start and right-click to select Uninstall. Uninstalling will
 remove all packages you installed directly into this Python installation, but
@@ -333,9 +336,6 @@ will not remove any virtual environments
 Known Issues
 ------------
 
-Currently, the ``py.exe`` launcher cannot be used to start Python when it
-has been installed from the Microsoft Store.
-
 Because of restrictions on Microsoft Store apps, Python scripts may not have
 full write access to shared locations such as ``TEMP`` and the registry.
 Instead, it will write to a private copy. If your scripts must modify the


From webhook-mailer at python.org  Mon Oct 14 21:06:22 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Tue, 15 Oct 2019 01:06:22 -0000
Subject: [Python-checkins] [3.8] bpo-36389: Backport debug enhancements from
 master (GH-16796)
Message-ID: <mailman.28.1571101583.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/f82ce5b1b12873b65927149a016be6a7c65e979d
commit: f82ce5b1b12873b65927149a016be6a7c65e979d
branch: 3.8
author: Victor Stinner <vstinner at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-15T03:06:16+02:00
summary:

[3.8] bpo-36389: Backport debug enhancements from master (GH-16796)

* bpo-36389: _PyObject_CheckConsistency() available in release mode (GH-16612)

bpo-36389, bpo-38376: The _PyObject_CheckConsistency() function is
now also available in release mode. For example, it can be used to
debug a crash in the visit_decref() function of the GC.

Modify the following functions to also work in release mode:

* _PyDict_CheckConsistency()
* _PyObject_CheckConsistency()
* _PyType_CheckConsistency()
* _PyUnicode_CheckConsistency()

Other changes:

* _PyMem_IsPtrFreed(ptr) now also returns 1 if ptr is NULL
  (equals to 0).
* _PyBytesWriter_CheckConsistency() now returns 1 and is only used
  with assert().
* Reorder _PyObject_Dump() to write safe fields first, and only
  attempt to render repr() at the end.

(cherry picked from commit 6876257eaabdb30f27ebcbd7d2557278ce2e5705)

* bpo-36389: Fix _PyBytesWriter in release mode (GH-16624)

Fix _PyBytesWriter API when Python is built in release mode with
assertions.

(cherry picked from commit 60ec6efd96d95476fe5e38c491491add04f026e5)

* bpo-38070: Enhance visit_decref() debug trace (GH-16631)

subtract_refs() now pass the parent object to visit_decref() which
pass it to _PyObject_ASSERT(). So if the "is freed" assertion fails,
the parent is used in debug trace, rather than the freed object. The
parent object is more likely to contain useful information. Freed
objects cannot be inspected are are displayed as "<object at xxx is
freed>" with no other detail.

(cherry picked from commit 4d5f94b8cd20f804c7868c5395a15aa6032f874c)

* Fix also a typo in PYMEM_DEADBYTE macro comment

* bpo-36389: Add newline to _PyObject_AssertFailed() (GH-16629)

Add a newline between the verbose object dump and the Py_FatalError()
logs for readability.

(cherry picked from commit 7775349895088a7ae68cecf0c74cf817f15e2c74)

files:
A Misc/NEWS.d/next/C API/2019-10-07-17-15-09.bpo-36389.hFX_jD.rst
M Include/internal/pycore_pymem.h
M Lib/test/test_capi.py
M Lib/test/test_gc.py
M Modules/_testcapimodule.c
M Modules/gcmodule.c
M Objects/bytesobject.c
M Objects/dictobject.c
M Objects/object.c
M Objects/obmalloc.c
M Objects/typeobject.c
M Objects/unicodeobject.c

diff --git a/Include/internal/pycore_pymem.h b/Include/internal/pycore_pymem.h
index 22677d373855c..47d092f3e53ff 100644
--- a/Include/internal/pycore_pymem.h
+++ b/Include/internal/pycore_pymem.h
@@ -155,9 +155,25 @@ PyAPI_FUNC(int) _PyMem_SetDefaultAllocator(
     PyMemAllocatorDomain domain,
     PyMemAllocatorEx *old_alloc);
 
+/* Special bytes broadcast into debug memory blocks at appropriate times.
+   Strings of these are unlikely to be valid addresses, floats, ints or
+   7-bit ASCII.
+
+   - PYMEM_CLEANBYTE: clean (newly allocated) memory
+   - PYMEM_DEADBYTE dead (newly freed) memory
+   - PYMEM_FORBIDDENBYTE: untouchable bytes at each end of a block
+
+   Byte patterns 0xCB, 0xDB and 0xFB have been replaced with 0xCD, 0xDD and
+   0xFD to use the same values than Windows CRT debug malloc() and free().
+   If modified, _PyMem_IsPtrFreed() should be updated as well. */
+#define PYMEM_CLEANBYTE      0xCD
+#define PYMEM_DEADBYTE       0xDD
+#define PYMEM_FORBIDDENBYTE  0xFD
+
 /* Heuristic checking if a pointer value is newly allocated
-   (uninitialized) or newly freed. The pointer is not dereferenced, only the
-   pointer value is checked.
+   (uninitialized), newly freed or NULL (is equal to zero).
+
+   The pointer is not dereferenced, only the pointer value is checked.
 
    The heuristic relies on the debug hooks on Python memory allocators which
    fills newly allocated memory with CLEANBYTE (0xCD) and newly freed memory
@@ -167,11 +183,13 @@ static inline int _PyMem_IsPtrFreed(void *ptr)
 {
     uintptr_t value = (uintptr_t)ptr;
 #if SIZEOF_VOID_P == 8
-    return (value == (uintptr_t)0xCDCDCDCDCDCDCDCD
+    return (value == 0
+            || value == (uintptr_t)0xCDCDCDCDCDCDCDCD
             || value == (uintptr_t)0xDDDDDDDDDDDDDDDD
             || value == (uintptr_t)0xFDFDFDFDFDFDFDFD);
 #elif SIZEOF_VOID_P == 4
-    return (value == (uintptr_t)0xCDCDCDCD
+    return (value == 0
+            || value == (uintptr_t)0xCDCDCDCD
             || value == (uintptr_t)0xDDDDDDDD
             || value == (uintptr_t)0xFDFDFDFD);
 #else
diff --git a/Lib/test/test_capi.py b/Lib/test/test_capi.py
index ff7acac43b61e..78c51b3cb61e8 100644
--- a/Lib/test/test_capi.py
+++ b/Lib/test/test_capi.py
@@ -692,6 +692,9 @@ def check_pyobject_is_freed(self, func_name):
         ''')
         assert_python_ok('-c', code, PYTHONMALLOC=self.PYTHONMALLOC)
 
+    def test_pyobject_null_is_freed(self):
+        self.check_pyobject_is_freed('check_pyobject_null_is_freed')
+
     def test_pyobject_uninitialized_is_freed(self):
         self.check_pyobject_is_freed('check_pyobject_uninitialized_is_freed')
 
diff --git a/Lib/test/test_gc.py b/Lib/test/test_gc.py
index 6c29e96c30fbd..f52db1eab169c 100644
--- a/Lib/test/test_gc.py
+++ b/Lib/test/test_gc.py
@@ -1055,16 +1055,19 @@ def test_refcount_errors(self):
             br'gcmodule\.c:[0-9]+: gc_decref: Assertion "gc_get_refs\(g\) > 0" failed.')
         self.assertRegex(stderr,
             br'refcount is too small')
+        # "address : 0x7fb5062efc18"
+        # "address : 7FB5062EFC18"
+        address_regex = br'[0-9a-fA-Fx]+'
         self.assertRegex(stderr,
-            br'object  : \[1, 2, 3\]')
+            br'object address  : ' + address_regex)
         self.assertRegex(stderr,
-            br'type    : list')
+            br'object refcount : 1')
         self.assertRegex(stderr,
-            br'refcount: 1')
-        # "address : 0x7fb5062efc18"
-        # "address : 7FB5062EFC18"
+            br'object type     : ' + address_regex)
+        self.assertRegex(stderr,
+            br'object type name: list')
         self.assertRegex(stderr,
-            br'address : [0-9a-fA-Fx]+')
+            br'object repr     : \[1, 2, 3\]')
 
 
 class GCTogglingTests(unittest.TestCase):
diff --git a/Misc/NEWS.d/next/C API/2019-10-07-17-15-09.bpo-36389.hFX_jD.rst b/Misc/NEWS.d/next/C API/2019-10-07-17-15-09.bpo-36389.hFX_jD.rst
new file mode 100644
index 0000000000000..6c42882cdbaa6
--- /dev/null
+++ b/Misc/NEWS.d/next/C API/2019-10-07-17-15-09.bpo-36389.hFX_jD.rst	
@@ -0,0 +1,3 @@
+The ``_PyObject_CheckConsistency()`` function is now also available in release
+mode. For example, it can be used to debug a crash in the ``visit_decref()``
+function of the GC.
diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c
index cccf9249040df..46d772cff608e 100644
--- a/Modules/_testcapimodule.c
+++ b/Modules/_testcapimodule.c
@@ -4500,6 +4500,14 @@ test_pyobject_is_freed(const char *test_name, PyObject *op)
 }
 
 
+static PyObject*
+check_pyobject_null_is_freed(PyObject *self, PyObject *Py_UNUSED(args))
+{
+    PyObject *op = NULL;
+    return test_pyobject_is_freed("check_pyobject_null_is_freed", op);
+}
+
+
 static PyObject*
 check_pyobject_uninitialized_is_freed(PyObject *self, PyObject *Py_UNUSED(args))
 {
@@ -5268,6 +5276,7 @@ static PyMethodDef TestMethods[] = {
     {"pymem_api_misuse", pymem_api_misuse, METH_NOARGS},
     {"pymem_malloc_without_gil", pymem_malloc_without_gil, METH_NOARGS},
     {"pymem_getallocatorsname", test_pymem_getallocatorsname, METH_NOARGS},
+    {"check_pyobject_null_is_freed", check_pyobject_null_is_freed, METH_NOARGS},
     {"check_pyobject_uninitialized_is_freed", check_pyobject_uninitialized_is_freed, METH_NOARGS},
     {"check_pyobject_forbidden_bytes_is_freed", check_pyobject_forbidden_bytes_is_freed, METH_NOARGS},
     {"check_pyobject_freed_is_freed", check_pyobject_freed_is_freed, METH_NOARGS},
diff --git a/Modules/gcmodule.c b/Modules/gcmodule.c
index 8d7d67cb6701d..5a6a81d81364d 100644
--- a/Modules/gcmodule.c
+++ b/Modules/gcmodule.c
@@ -373,10 +373,9 @@ update_refs(PyGC_Head *containers)
 
 /* A traversal callback for subtract_refs. */
 static int
-visit_decref(PyObject *op, void *data)
+visit_decref(PyObject *op, void *parent)
 {
-    assert(op != NULL);
-    _PyObject_ASSERT(op, !_PyObject_IsFreed(op));
+    _PyObject_ASSERT(_PyObject_CAST(parent), !_PyObject_IsFreed(op));
 
     if (PyObject_IS_GC(op)) {
         PyGC_Head *gc = AS_GC(op);
@@ -402,10 +401,11 @@ subtract_refs(PyGC_Head *containers)
     traverseproc traverse;
     PyGC_Head *gc = GC_NEXT(containers);
     for (; gc != containers; gc = GC_NEXT(gc)) {
-        traverse = Py_TYPE(FROM_GC(gc))->tp_traverse;
+        PyObject *op = FROM_GC(gc);
+        traverse = Py_TYPE(op)->tp_traverse;
         (void) traverse(FROM_GC(gc),
                        (visitproc)visit_decref,
-                       NULL);
+                       op);
     }
 }
 
diff --git a/Objects/bytesobject.c b/Objects/bytesobject.c
index 3b69fecde27b1..feeabcb8b4c7a 100644
--- a/Objects/bytesobject.c
+++ b/Objects/bytesobject.c
@@ -667,9 +667,6 @@ _PyBytes_FormatEx(const char *format, Py_ssize_t format_len,
             Py_ssize_t len = 0;
             char onechar; /* For byte_converter() */
             Py_ssize_t alloc;
-#ifdef Py_DEBUG
-            char *before;
-#endif
 
             fmt++;
             if (*fmt == '%') {
@@ -981,8 +978,8 @@ _PyBytes_FormatEx(const char *format, Py_ssize_t format_len,
                 if (res == NULL)
                     goto error;
             }
-#ifdef Py_DEBUG
-            before = res;
+#ifndef NDEBUG
+            char *before = res;
 #endif
 
             /* Write the sign if needed */
@@ -1047,7 +1044,7 @@ _PyBytes_FormatEx(const char *format, Py_ssize_t format_len,
             }
             Py_XDECREF(temp);
 
-#ifdef Py_DEBUG
+#ifndef NDEBUG
             /* check that we computed the exact size for this write */
             assert((res - before) == alloc);
 #endif
@@ -3225,8 +3222,9 @@ _PyBytesWriter_Init(_PyBytesWriter *writer)
 {
     /* Set all attributes before small_buffer to 0 */
     memset(writer, 0, offsetof(_PyBytesWriter, small_buffer));
-#ifdef Py_DEBUG
-    memset(writer->small_buffer, 0xCB, sizeof(writer->small_buffer));
+#ifndef NDEBUG
+    memset(writer->small_buffer, PYMEM_CLEANBYTE,
+           sizeof(writer->small_buffer));
 #endif
 }
 
@@ -3263,10 +3261,10 @@ _PyBytesWriter_GetSize(_PyBytesWriter *writer, char *str)
     return str - start;
 }
 
-Py_LOCAL_INLINE(void)
+#ifndef NDEBUG
+Py_LOCAL_INLINE(int)
 _PyBytesWriter_CheckConsistency(_PyBytesWriter *writer, char *str)
 {
-#ifdef Py_DEBUG
     char *start, *end;
 
     if (writer->use_small_buffer) {
@@ -3296,15 +3294,16 @@ _PyBytesWriter_CheckConsistency(_PyBytesWriter *writer, char *str)
     end = start + writer->allocated;
     assert(str != NULL);
     assert(start <= str && str <= end);
-#endif
+    return 1;
 }
+#endif
 
 void*
 _PyBytesWriter_Resize(_PyBytesWriter *writer, void *str, Py_ssize_t size)
 {
     Py_ssize_t allocated, pos;
 
-    _PyBytesWriter_CheckConsistency(writer, str);
+    assert(_PyBytesWriter_CheckConsistency(writer, str));
     assert(writer->allocated < size);
 
     allocated = size;
@@ -3353,14 +3352,15 @@ _PyBytesWriter_Resize(_PyBytesWriter *writer, void *str, Py_ssize_t size)
         }
 
         writer->use_small_buffer = 0;
-#ifdef Py_DEBUG
-        memset(writer->small_buffer, 0xDB, sizeof(writer->small_buffer));
+#ifndef NDEBUG
+        memset(writer->small_buffer, PYMEM_CLEANBYTE,
+               sizeof(writer->small_buffer));
 #endif
     }
     writer->allocated = allocated;
 
     str = _PyBytesWriter_AsString(writer) + pos;
-    _PyBytesWriter_CheckConsistency(writer, str);
+    assert(_PyBytesWriter_CheckConsistency(writer, str));
     return str;
 
 error:
@@ -3373,7 +3373,7 @@ _PyBytesWriter_Prepare(_PyBytesWriter *writer, void *str, Py_ssize_t size)
 {
     Py_ssize_t new_min_size;
 
-    _PyBytesWriter_CheckConsistency(writer, str);
+    assert(_PyBytesWriter_CheckConsistency(writer, str));
     assert(size >= 0);
 
     if (size == 0) {
@@ -3406,7 +3406,7 @@ _PyBytesWriter_Alloc(_PyBytesWriter *writer, Py_ssize_t size)
     assert(size >= 0);
 
     writer->use_small_buffer = 1;
-#ifdef Py_DEBUG
+#ifndef NDEBUG
     writer->allocated = sizeof(writer->small_buffer) - 1;
     /* In debug mode, don't use the full small buffer because it is less
        efficient than bytes and bytearray objects to detect buffer underflow
@@ -3434,7 +3434,7 @@ _PyBytesWriter_Finish(_PyBytesWriter *writer, void *str)
     Py_ssize_t size;
     PyObject *result;
 
-    _PyBytesWriter_CheckConsistency(writer, str);
+    assert(_PyBytesWriter_CheckConsistency(writer, str));
 
     size = _PyBytesWriter_GetSize(writer, str);
     if (size == 0 && !writer->use_bytearray) {
diff --git a/Objects/dictobject.c b/Objects/dictobject.c
index a6ddf70cd8fa6..5dad1b6d9208c 100644
--- a/Objects/dictobject.c
+++ b/Objects/dictobject.c
@@ -459,23 +459,26 @@ static PyObject *empty_values[1] = { NULL };
 int
 _PyDict_CheckConsistency(PyObject *op, int check_content)
 {
-#ifndef NDEBUG
-    _PyObject_ASSERT(op, PyDict_Check(op));
+#define CHECK(expr) \
+    do { if (!(expr)) { _PyObject_ASSERT_FAILED_MSG(op, Py_STRINGIFY(expr)); } } while (0)
+
+    assert(op != NULL);
+    CHECK(PyDict_Check(op));
     PyDictObject *mp = (PyDictObject *)op;
 
     PyDictKeysObject *keys = mp->ma_keys;
     int splitted = _PyDict_HasSplitTable(mp);
     Py_ssize_t usable = USABLE_FRACTION(keys->dk_size);
 
-    _PyObject_ASSERT(op, 0 <= mp->ma_used && mp->ma_used <= usable);
-    _PyObject_ASSERT(op, IS_POWER_OF_2(keys->dk_size));
-    _PyObject_ASSERT(op, 0 <= keys->dk_usable && keys->dk_usable <= usable);
-    _PyObject_ASSERT(op, 0 <= keys->dk_nentries && keys->dk_nentries <= usable);
-    _PyObject_ASSERT(op, keys->dk_usable + keys->dk_nentries <= usable);
+    CHECK(0 <= mp->ma_used && mp->ma_used <= usable);
+    CHECK(IS_POWER_OF_2(keys->dk_size));
+    CHECK(0 <= keys->dk_usable && keys->dk_usable <= usable);
+    CHECK(0 <= keys->dk_nentries && keys->dk_nentries <= usable);
+    CHECK(keys->dk_usable + keys->dk_nentries <= usable);
 
     if (!splitted) {
         /* combined table */
-        _PyObject_ASSERT(op, keys->dk_refcnt == 1);
+        CHECK(keys->dk_refcnt == 1);
     }
 
     if (check_content) {
@@ -484,7 +487,7 @@ _PyDict_CheckConsistency(PyObject *op, int check_content)
 
         for (i=0; i < keys->dk_size; i++) {
             Py_ssize_t ix = dictkeys_get_index(keys, i);
-            _PyObject_ASSERT(op, DKIX_DUMMY <= ix && ix <= usable);
+            CHECK(DKIX_DUMMY <= ix && ix <= usable);
         }
 
         for (i=0; i < usable; i++) {
@@ -494,32 +497,33 @@ _PyDict_CheckConsistency(PyObject *op, int check_content)
             if (key != NULL) {
                 if (PyUnicode_CheckExact(key)) {
                     Py_hash_t hash = ((PyASCIIObject *)key)->hash;
-                    _PyObject_ASSERT(op, hash != -1);
-                    _PyObject_ASSERT(op, entry->me_hash == hash);
+                    CHECK(hash != -1);
+                    CHECK(entry->me_hash == hash);
                 }
                 else {
                     /* test_dict fails if PyObject_Hash() is called again */
-                    _PyObject_ASSERT(op, entry->me_hash != -1);
+                    CHECK(entry->me_hash != -1);
                 }
                 if (!splitted) {
-                    _PyObject_ASSERT(op, entry->me_value != NULL);
+                    CHECK(entry->me_value != NULL);
                 }
             }
 
             if (splitted) {
-                _PyObject_ASSERT(op, entry->me_value == NULL);
+                CHECK(entry->me_value == NULL);
             }
         }
 
         if (splitted) {
             /* splitted table */
             for (i=0; i < mp->ma_used; i++) {
-                _PyObject_ASSERT(op, mp->ma_values[i] != NULL);
+                CHECK(mp->ma_values[i] != NULL);
             }
         }
     }
-#endif
     return 1;
+
+#undef CHECK
 }
 
 
diff --git a/Objects/object.c b/Objects/object.c
index df2531371fa5b..566593a9c67cc 100644
--- a/Objects/object.c
+++ b/Objects/object.c
@@ -25,13 +25,14 @@ _Py_IDENTIFIER(__isabstractmethod__);
 int
 _PyObject_CheckConsistency(PyObject *op, int check_content)
 {
-    _PyObject_ASSERT(op, op != NULL);
-    _PyObject_ASSERT(op, !_PyObject_IsFreed(op));
-    _PyObject_ASSERT(op, Py_REFCNT(op) >= 1);
+#define CHECK(expr) \
+    do { if (!(expr)) { _PyObject_ASSERT_FAILED_MSG(op, Py_STRINGIFY(expr)); } } while (0)
 
-    PyTypeObject *type = op->ob_type;
-    _PyObject_ASSERT(op, type != NULL);
-    _PyType_CheckConsistency(type);
+    CHECK(!_PyObject_IsFreed(op));
+    CHECK(Py_REFCNT(op) >= 1);
+
+    CHECK(op->ob_type != NULL);
+    _PyType_CheckConsistency(op->ob_type);
 
     if (PyUnicode_Check(op)) {
         _PyUnicode_CheckConsistency(op, check_content);
@@ -40,6 +41,8 @@ _PyObject_CheckConsistency(PyObject *op, int check_content)
         _PyDict_CheckConsistency(op, check_content);
     }
     return 1;
+
+#undef CHECK
 }
 
 
@@ -463,41 +466,41 @@ _PyObject_IsFreed(PyObject *op)
 void
 _PyObject_Dump(PyObject* op)
 {
-    if (op == NULL) {
-        fprintf(stderr, "<object at NULL>\n");
-        fflush(stderr);
-        return;
-    }
-
     if (_PyObject_IsFreed(op)) {
         /* It seems like the object memory has been freed:
            don't access it to prevent a segmentation fault. */
         fprintf(stderr, "<object at %p is freed>\n", op);
+        fflush(stderr);
         return;
     }
 
-    PyGILState_STATE gil;
-    PyObject *error_type, *error_value, *error_traceback;
+    /* first, write fields which are the least likely to crash */
+    fprintf(stderr, "object address  : %p\n", (void *)op);
+    /* XXX(twouters) cast refcount to long until %zd is
+       universally available */
+    fprintf(stderr, "object refcount : %ld\n", (long)op->ob_refcnt);
+    fflush(stderr);
 
-    fprintf(stderr, "object  : ");
+    PyTypeObject *type = Py_TYPE(op);
+    fprintf(stderr, "object type     : %p\n", type);
+    fprintf(stderr, "object type name: %s\n",
+            type==NULL ? "NULL" : type->tp_name);
+
+    /* the most dangerous part */
+    fprintf(stderr, "object repr     : ");
     fflush(stderr);
-    gil = PyGILState_Ensure();
 
+    PyGILState_STATE gil = PyGILState_Ensure();
+    PyObject *error_type, *error_value, *error_traceback;
     PyErr_Fetch(&error_type, &error_value, &error_traceback);
+
     (void)PyObject_Print(op, stderr, 0);
     fflush(stderr);
-    PyErr_Restore(error_type, error_value, error_traceback);
 
+    PyErr_Restore(error_type, error_value, error_traceback);
     PyGILState_Release(gil);
-    /* XXX(twouters) cast refcount to long until %zd is
-       universally available */
-    fprintf(stderr, "\n"
-        "type    : %s\n"
-        "refcount: %ld\n"
-        "address : %p\n",
-        Py_TYPE(op)==NULL ? "NULL" : Py_TYPE(op)->tp_name,
-        (long)op->ob_refcnt,
-        (void *)op);
+
+    fprintf(stderr, "\n");
     fflush(stderr);
 }
 
@@ -2148,6 +2151,7 @@ _PyObject_AssertFailed(PyObject *obj, const char *expr, const char *msg,
         fprintf(stderr, "%s: ", function);
     }
     fflush(stderr);
+
     if (expr) {
         fprintf(stderr, "Assertion \"%s\" failed", expr);
     }
@@ -2155,26 +2159,18 @@ _PyObject_AssertFailed(PyObject *obj, const char *expr, const char *msg,
         fprintf(stderr, "Assertion failed");
     }
     fflush(stderr);
+
     if (msg) {
         fprintf(stderr, ": %s", msg);
     }
     fprintf(stderr, "\n");
     fflush(stderr);
 
-    if (obj == NULL) {
-        fprintf(stderr, "<object at NULL>\n");
-    }
-    else if (_PyObject_IsFreed(obj)) {
+    if (_PyObject_IsFreed(obj)) {
         /* It seems like the object memory has been freed:
            don't access it to prevent a segmentation fault. */
         fprintf(stderr, "<object at %p is freed>\n", obj);
-    }
-    else if (Py_TYPE(obj) == NULL) {
-        fprintf(stderr, "<object at %p: ob_type=NULL>\n", obj);
-    }
-    else if (_PyObject_IsFreed((PyObject *)Py_TYPE(obj))) {
-        fprintf(stderr, "<object at %p: type at %p is freed>\n",
-                obj, (void *)Py_TYPE(obj));
+        fflush(stderr);
     }
     else {
         /* Display the traceback where the object has been allocated.
@@ -2193,8 +2189,10 @@ _PyObject_AssertFailed(PyObject *obj, const char *expr, const char *msg,
         /* This might succeed or fail, but we're about to abort, so at least
            try to provide any extra info we can: */
         _PyObject_Dump(obj);
+
+        fprintf(stderr, "\n");
+        fflush(stderr);
     }
-    fflush(stderr);
 
     Py_FatalError("_PyObject_AssertFailed");
 }
diff --git a/Objects/obmalloc.c b/Objects/obmalloc.c
index f420e19761741..bd6480a625e8a 100644
--- a/Objects/obmalloc.c
+++ b/Objects/obmalloc.c
@@ -2006,20 +2006,6 @@ _Py_GetAllocatedBlocks(void)
  * it wraps a real allocator, adding extra debugging info to the memory blocks.
  */
 
-/* Special bytes broadcast into debug memory blocks at appropriate times.
- * Strings of these are unlikely to be valid addresses, floats, ints or
- * 7-bit ASCII. If modified, _PyMem_IsPtrFreed() should be updated as well.
- *
- * Byte patterns 0xCB, 0xBB and 0xFB have been replaced with 0xCD, 0xDD and
- * 0xFD to use the same values than Windows CRT debug malloc() and free().
- */
-#undef CLEANBYTE
-#undef DEADBYTE
-#undef FORBIDDENBYTE
-#define CLEANBYTE      0xCD    /* clean (newly allocated) memory */
-#define DEADBYTE       0xDD    /* dead (newly freed) memory */
-#define FORBIDDENBYTE  0xFD    /* untouchable bytes at each end of a block */
-
 /* Uncomment this define to add the "serialno" field */
 /* #define PYMEM_DEBUG_SERIALNO */
 
@@ -2081,14 +2067,14 @@ p[0: S]
 p[S]
     API ID.  See PEP 445.  This is a character, but seems undocumented.
 p[S+1: 2*S]
-    Copies of FORBIDDENBYTE.  Used to catch under- writes and reads.
+    Copies of PYMEM_FORBIDDENBYTE.  Used to catch under- writes and reads.
 p[2*S: 2*S+n]
-    The requested memory, filled with copies of CLEANBYTE.
+    The requested memory, filled with copies of PYMEM_CLEANBYTE.
     Used to catch reference to uninitialized memory.
     &p[2*S] is returned.  Note that this is 8-byte aligned if pymalloc
     handled the request itself.
 p[2*S+n: 2*S+n+S]
-    Copies of FORBIDDENBYTE.  Used to catch over- writes and reads.
+    Copies of PYMEM_FORBIDDENBYTE.  Used to catch over- writes and reads.
 p[2*S+n+S: 2*S+n+2*S]
     A serial number, incremented by 1 on each call to _PyMem_DebugMalloc
     and _PyMem_DebugRealloc.
@@ -2145,15 +2131,15 @@ _PyMem_DebugRawAlloc(int use_calloc, void *ctx, size_t nbytes)
     /* at p, write size (SST bytes), id (1 byte), pad (SST-1 bytes) */
     write_size_t(p, nbytes);
     p[SST] = (uint8_t)api->api_id;
-    memset(p + SST + 1, FORBIDDENBYTE, SST-1);
+    memset(p + SST + 1, PYMEM_FORBIDDENBYTE, SST-1);
 
     if (nbytes > 0 && !use_calloc) {
-        memset(data, CLEANBYTE, nbytes);
+        memset(data, PYMEM_CLEANBYTE, nbytes);
     }
 
     /* at tail, write pad (SST bytes) and serialno (SST bytes) */
     tail = data + nbytes;
-    memset(tail, FORBIDDENBYTE, SST);
+    memset(tail, PYMEM_FORBIDDENBYTE, SST);
 #ifdef PYMEM_DEBUG_SERIALNO
     write_size_t(tail + SST, serialno);
 #endif
@@ -2179,7 +2165,7 @@ _PyMem_DebugRawCalloc(void *ctx, size_t nelem, size_t elsize)
 
 /* The debug free first checks the 2*SST bytes on each end for sanity (in
    particular, that the FORBIDDENBYTEs with the api ID are still intact).
-   Then fills the original bytes with DEADBYTE.
+   Then fills the original bytes with PYMEM_DEADBYTE.
    Then calls the underlying free.
 */
 static void
@@ -2197,7 +2183,7 @@ _PyMem_DebugRawFree(void *ctx, void *p)
     _PyMem_DebugCheckAddress(api->api_id, p);
     nbytes = read_size_t(q);
     nbytes += PYMEM_DEBUG_EXTRA_BYTES;
-    memset(q, DEADBYTE, nbytes);
+    memset(q, PYMEM_DEADBYTE, nbytes);
     api->alloc.free(api->alloc.ctx, q);
 }
 
@@ -2239,14 +2225,14 @@ _PyMem_DebugRawRealloc(void *ctx, void *p, size_t nbytes)
      */
     if (original_nbytes <= sizeof(save)) {
         memcpy(save, data, original_nbytes);
-        memset(data - 2 * SST, DEADBYTE,
+        memset(data - 2 * SST, PYMEM_DEADBYTE,
                original_nbytes + PYMEM_DEBUG_EXTRA_BYTES);
     }
     else {
         memcpy(save, data, ERASED_SIZE);
-        memset(head, DEADBYTE, ERASED_SIZE + 2 * SST);
+        memset(head, PYMEM_DEADBYTE, ERASED_SIZE + 2 * SST);
         memcpy(&save[ERASED_SIZE], tail - ERASED_SIZE, ERASED_SIZE);
-        memset(tail - ERASED_SIZE, DEADBYTE,
+        memset(tail - ERASED_SIZE, PYMEM_DEADBYTE,
                ERASED_SIZE + PYMEM_DEBUG_EXTRA_BYTES - 2 * SST);
     }
 
@@ -2268,10 +2254,10 @@ _PyMem_DebugRawRealloc(void *ctx, void *p, size_t nbytes)
 
     write_size_t(head, nbytes);
     head[SST] = (uint8_t)api->api_id;
-    memset(head + SST + 1, FORBIDDENBYTE, SST-1);
+    memset(head + SST + 1, PYMEM_FORBIDDENBYTE, SST-1);
 
     tail = data + nbytes;
-    memset(tail, FORBIDDENBYTE, SST);
+    memset(tail, PYMEM_FORBIDDENBYTE, SST);
 #ifdef PYMEM_DEBUG_SERIALNO
     write_size_t(tail + SST, block_serialno);
 #endif
@@ -2295,7 +2281,8 @@ _PyMem_DebugRawRealloc(void *ctx, void *p, size_t nbytes)
 
     if (nbytes > original_nbytes) {
         /* growing: mark new extra memory clean */
-        memset(data + original_nbytes, CLEANBYTE, nbytes - original_nbytes);
+        memset(data + original_nbytes, PYMEM_CLEANBYTE,
+               nbytes - original_nbytes);
     }
 
     return data;
@@ -2374,7 +2361,7 @@ _PyMem_DebugCheckAddress(char api, const void *p)
      * the tail could lead to a segfault then.
      */
     for (i = SST-1; i >= 1; --i) {
-        if (*(q-i) != FORBIDDENBYTE) {
+        if (*(q-i) != PYMEM_FORBIDDENBYTE) {
             msg = "bad leading pad byte";
             goto error;
         }
@@ -2383,7 +2370,7 @@ _PyMem_DebugCheckAddress(char api, const void *p)
     nbytes = read_size_t(q - 2*SST);
     tail = q + nbytes;
     for (i = 0; i < SST; ++i) {
-        if (tail[i] != FORBIDDENBYTE) {
+        if (tail[i] != PYMEM_FORBIDDENBYTE) {
             msg = "bad trailing pad byte";
             goto error;
         }
@@ -2423,7 +2410,7 @@ _PyObject_DebugDumpAddress(const void *p)
     fprintf(stderr, "    The %d pad bytes at p-%d are ", SST-1, SST-1);
     ok = 1;
     for (i = 1; i <= SST-1; ++i) {
-        if (*(q-i) != FORBIDDENBYTE) {
+        if (*(q-i) != PYMEM_FORBIDDENBYTE) {
             ok = 0;
             break;
         }
@@ -2432,11 +2419,11 @@ _PyObject_DebugDumpAddress(const void *p)
         fputs("FORBIDDENBYTE, as expected.\n", stderr);
     else {
         fprintf(stderr, "not all FORBIDDENBYTE (0x%02x):\n",
-            FORBIDDENBYTE);
+            PYMEM_FORBIDDENBYTE);
         for (i = SST-1; i >= 1; --i) {
             const uint8_t byte = *(q-i);
             fprintf(stderr, "        at p-%d: 0x%02x", i, byte);
-            if (byte != FORBIDDENBYTE)
+            if (byte != PYMEM_FORBIDDENBYTE)
                 fputs(" *** OUCH", stderr);
             fputc('\n', stderr);
         }
@@ -2451,7 +2438,7 @@ _PyObject_DebugDumpAddress(const void *p)
     fprintf(stderr, "    The %d pad bytes at tail=%p are ", SST, (void *)tail);
     ok = 1;
     for (i = 0; i < SST; ++i) {
-        if (tail[i] != FORBIDDENBYTE) {
+        if (tail[i] != PYMEM_FORBIDDENBYTE) {
             ok = 0;
             break;
         }
@@ -2460,12 +2447,12 @@ _PyObject_DebugDumpAddress(const void *p)
         fputs("FORBIDDENBYTE, as expected.\n", stderr);
     else {
         fprintf(stderr, "not all FORBIDDENBYTE (0x%02x):\n",
-                FORBIDDENBYTE);
+                PYMEM_FORBIDDENBYTE);
         for (i = 0; i < SST; ++i) {
             const uint8_t byte = tail[i];
             fprintf(stderr, "        at tail+%d: 0x%02x",
                     i, byte);
-            if (byte != FORBIDDENBYTE)
+            if (byte != PYMEM_FORBIDDENBYTE)
                 fputs(" *** OUCH", stderr);
             fputc('\n', stderr);
         }
diff --git a/Objects/typeobject.c b/Objects/typeobject.c
index e635b718d6a95..5df121775af00 100644
--- a/Objects/typeobject.c
+++ b/Objects/typeobject.c
@@ -137,22 +137,24 @@ skip_signature(const char *doc)
 int
 _PyType_CheckConsistency(PyTypeObject *type)
 {
-#define ASSERT(expr) _PyObject_ASSERT((PyObject *)type, (expr))
+#define CHECK(expr) \
+    do { if (!(expr)) { _PyObject_ASSERT_FAILED_MSG((PyObject *)type, Py_STRINGIFY(expr)); } } while (0)
+
+    CHECK(!_PyObject_IsFreed((PyObject *)type));
 
     if (!(type->tp_flags & Py_TPFLAGS_READY)) {
-        /* don't check types before PyType_Ready() */
+        /* don't check static types before PyType_Ready() */
         return 1;
     }
 
-    ASSERT(!_PyObject_IsFreed((PyObject *)type));
-    ASSERT(Py_REFCNT(type) >= 1);
-    ASSERT(PyType_Check(type));
+    CHECK(Py_REFCNT(type) >= 1);
+    CHECK(PyType_Check(type));
 
-    ASSERT(!(type->tp_flags & Py_TPFLAGS_READYING));
-    ASSERT(type->tp_dict != NULL);
+    CHECK(!(type->tp_flags & Py_TPFLAGS_READYING));
+    CHECK(type->tp_dict != NULL);
 
     return 1;
-#undef ASSERT
+#undef CHECK
 }
 
 static const char *
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
index 9eef05a621676..0556eff8d0713 100644
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -442,65 +442,63 @@ PyUnicode_GetMax(void)
 int
 _PyUnicode_CheckConsistency(PyObject *op, int check_content)
 {
+#define CHECK(expr) \
+    do { if (!(expr)) { _PyObject_ASSERT_FAILED_MSG(op, Py_STRINGIFY(expr)); } } while (0)
+
     PyASCIIObject *ascii;
     unsigned int kind;
 
-    _PyObject_ASSERT(op, PyUnicode_Check(op));
+    assert(op != NULL);
+    CHECK(PyUnicode_Check(op));
 
     ascii = (PyASCIIObject *)op;
     kind = ascii->state.kind;
 
     if (ascii->state.ascii == 1 && ascii->state.compact == 1) {
-        _PyObject_ASSERT(op, kind == PyUnicode_1BYTE_KIND);
-        _PyObject_ASSERT(op, ascii->state.ready == 1);
+        CHECK(kind == PyUnicode_1BYTE_KIND);
+        CHECK(ascii->state.ready == 1);
     }
     else {
         PyCompactUnicodeObject *compact = (PyCompactUnicodeObject *)op;
-#ifndef NDEBUG
         void *data;
-#endif
 
         if (ascii->state.compact == 1) {
-#ifndef NDEBUG
             data = compact + 1;
-#endif
-            _PyObject_ASSERT(op, kind == PyUnicode_1BYTE_KIND
+            CHECK(kind == PyUnicode_1BYTE_KIND
                                  || kind == PyUnicode_2BYTE_KIND
                                  || kind == PyUnicode_4BYTE_KIND);
-            _PyObject_ASSERT(op, ascii->state.ascii == 0);
-            _PyObject_ASSERT(op, ascii->state.ready == 1);
-            _PyObject_ASSERT(op, compact->utf8 != data);
+            CHECK(ascii->state.ascii == 0);
+            CHECK(ascii->state.ready == 1);
+            CHECK(compact->utf8 != data);
         }
         else {
-#ifndef NDEBUG
             PyUnicodeObject *unicode = (PyUnicodeObject *)op;
 
             data = unicode->data.any;
-#endif
             if (kind == PyUnicode_WCHAR_KIND) {
-                _PyObject_ASSERT(op, ascii->length == 0);
-                _PyObject_ASSERT(op, ascii->hash == -1);
-                _PyObject_ASSERT(op, ascii->state.compact == 0);
-                _PyObject_ASSERT(op, ascii->state.ascii == 0);
-                _PyObject_ASSERT(op, ascii->state.ready == 0);
-                _PyObject_ASSERT(op, ascii->state.interned == SSTATE_NOT_INTERNED);
-                _PyObject_ASSERT(op, ascii->wstr != NULL);
-                _PyObject_ASSERT(op, data == NULL);
-                _PyObject_ASSERT(op, compact->utf8 == NULL);
+                CHECK(ascii->length == 0);
+                CHECK(ascii->hash == -1);
+                CHECK(ascii->state.compact == 0);
+                CHECK(ascii->state.ascii == 0);
+                CHECK(ascii->state.ready == 0);
+                CHECK(ascii->state.interned == SSTATE_NOT_INTERNED);
+                CHECK(ascii->wstr != NULL);
+                CHECK(data == NULL);
+                CHECK(compact->utf8 == NULL);
             }
             else {
-                _PyObject_ASSERT(op, kind == PyUnicode_1BYTE_KIND
+                CHECK(kind == PyUnicode_1BYTE_KIND
                                      || kind == PyUnicode_2BYTE_KIND
                                      || kind == PyUnicode_4BYTE_KIND);
-                _PyObject_ASSERT(op, ascii->state.compact == 0);
-                _PyObject_ASSERT(op, ascii->state.ready == 1);
-                _PyObject_ASSERT(op, data != NULL);
+                CHECK(ascii->state.compact == 0);
+                CHECK(ascii->state.ready == 1);
+                CHECK(data != NULL);
                 if (ascii->state.ascii) {
-                    _PyObject_ASSERT(op, compact->utf8 == data);
-                    _PyObject_ASSERT(op, compact->utf8_length == ascii->length);
+                    CHECK(compact->utf8 == data);
+                    CHECK(compact->utf8_length == ascii->length);
                 }
                 else
-                    _PyObject_ASSERT(op, compact->utf8 != data);
+                    CHECK(compact->utf8 != data);
             }
         }
         if (kind != PyUnicode_WCHAR_KIND) {
@@ -512,16 +510,16 @@ _PyUnicode_CheckConsistency(PyObject *op, int check_content)
 #endif
                )
             {
-                _PyObject_ASSERT(op, ascii->wstr == data);
-                _PyObject_ASSERT(op, compact->wstr_length == ascii->length);
+                CHECK(ascii->wstr == data);
+                CHECK(compact->wstr_length == ascii->length);
             } else
-                _PyObject_ASSERT(op, ascii->wstr != data);
+                CHECK(ascii->wstr != data);
         }
 
         if (compact->utf8 == NULL)
-            _PyObject_ASSERT(op, compact->utf8_length == 0);
+            CHECK(compact->utf8_length == 0);
         if (ascii->wstr == NULL)
-            _PyObject_ASSERT(op, compact->wstr_length == 0);
+            CHECK(compact->wstr_length == 0);
     }
 
     /* check that the best kind is used: O(n) operation */
@@ -540,23 +538,25 @@ _PyUnicode_CheckConsistency(PyObject *op, int check_content)
         }
         if (kind == PyUnicode_1BYTE_KIND) {
             if (ascii->state.ascii == 0) {
-                _PyObject_ASSERT(op, maxchar >= 128);
-                _PyObject_ASSERT(op, maxchar <= 255);
+                CHECK(maxchar >= 128);
+                CHECK(maxchar <= 255);
             }
             else
-                _PyObject_ASSERT(op, maxchar < 128);
+                CHECK(maxchar < 128);
         }
         else if (kind == PyUnicode_2BYTE_KIND) {
-            _PyObject_ASSERT(op, maxchar >= 0x100);
-            _PyObject_ASSERT(op, maxchar <= 0xFFFF);
+            CHECK(maxchar >= 0x100);
+            CHECK(maxchar <= 0xFFFF);
         }
         else {
-            _PyObject_ASSERT(op, maxchar >= 0x10000);
-            _PyObject_ASSERT(op, maxchar <= MAX_UNICODE);
+            CHECK(maxchar >= 0x10000);
+            CHECK(maxchar <= MAX_UNICODE);
         }
-        _PyObject_ASSERT(op, PyUnicode_READ(kind, data, ascii->length) == 0);
+        CHECK(PyUnicode_READ(kind, data, ascii->length) == 0);
     }
     return 1;
+
+#undef CHECK
 }
 
 


From webhook-mailer at python.org  Tue Oct 15 03:26:21 2019
From: webhook-mailer at python.org (Vinay Sajip)
Date: Tue, 15 Oct 2019 07:26:21 -0000
Subject: [Python-checkins] bpo-11410: Standardize and use symbol visibility
 attributes across POSIX and Windows. (GH-16347)
Message-ID: <mailman.29.1571124382.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/0b60f64e4343913b4931dc27379d9808e5b78fe1
commit: 0b60f64e4343913b4931dc27379d9808e5b78fe1
branch: master
author: Vinay Sajip <vinay_sajip at yahoo.co.uk>
committer: GitHub <noreply at github.com>
date: 2019-10-15T08:26:12+01:00
summary:

bpo-11410: Standardize and use symbol visibility attributes across POSIX and Windows. (GH-16347)

files:
A Include/exports.h
A Misc/NEWS.d/next/Core and Builtins/2019-09-24-05-32-27.bpo-11410.vS182p.rst
M Include/pyport.h
M Modules/_ctypes/_ctypes_test.c
M Modules/_io/_iomodule.h
M Modules/_io/bytesio.c
M Parser/pgen/grammar.py
M Python/getargs.c
M Python/graminit.c
M Python/pythonrun.c
M configure
M configure.ac

diff --git a/Include/exports.h b/Include/exports.h
new file mode 100644
index 0000000000000..fc1a5c5ead627
--- /dev/null
+++ b/Include/exports.h
@@ -0,0 +1,30 @@
+#ifndef Py_EXPORTS_H
+#define Py_EXPORTS_H
+
+#if defined(_WIN32) || defined(__CYGWIN__)
+    #define Py_IMPORTED_SYMBOL __declspec(dllimport)
+    #define Py_EXPORTED_SYMBOL __declspec(dllexport)
+    #define Py_LOCAL_SYMBOL
+#else
+/*
+ * If we only ever used gcc >= 5, we could use __has_attribute(visibility)
+ * as a cross-platform way to determine if visibility is supported. However,
+ * we may still need to support gcc >= 4, as some Ubuntu LTS and Centos versions
+ * have 4 < gcc < 5.
+ */
+    #ifndef __has_attribute
+      #define __has_attribute(x) 0  // Compatibility with non-clang compilers.
+    #endif
+    #if (defined(__GNUC__) && (__GNUC__ >= 4)) ||\
+        (defined(__clang__) && __has_attribute(visibility))
+        #define Py_IMPORTED_SYMBOL __attribute__ ((visibility ("default")))
+        #define Py_EXPORTED_SYMBOL __attribute__ ((visibility ("default")))
+        #define Py_LOCAL_SYMBOL  __attribute__ ((visibility ("hidden")))
+    #else
+        #define Py_IMPORTED_SYMBOL
+        #define Py_EXPORTED_SYMBOL
+        #define Py_LOCAL_SYMBOL
+    #endif
+#endif
+
+#endif /* Py_EXPORTS_H */
diff --git a/Include/pyport.h b/Include/pyport.h
index 51967d186672c..64c73f012e7a7 100644
--- a/Include/pyport.h
+++ b/Include/pyport.h
@@ -638,16 +638,18 @@ extern char * _getpty(int *, int, mode_t, int);
 #       define HAVE_DECLSPEC_DLL
 #endif
 
+#include "exports.h"
+
 /* only get special linkage if built as shared or platform is Cygwin */
 #if defined(Py_ENABLE_SHARED) || defined(__CYGWIN__)
 #       if defined(HAVE_DECLSPEC_DLL)
 #               if defined(Py_BUILD_CORE) && !defined(Py_BUILD_CORE_MODULE)
-#                       define PyAPI_FUNC(RTYPE) __declspec(dllexport) RTYPE
-#                       define PyAPI_DATA(RTYPE) extern __declspec(dllexport) RTYPE
+#                       define PyAPI_FUNC(RTYPE) Py_EXPORTED_SYMBOL RTYPE
+#                       define PyAPI_DATA(RTYPE) extern Py_EXPORTED_SYMBOL RTYPE
         /* module init functions inside the core need no external linkage */
         /* except for Cygwin to handle embedding */
 #                       if defined(__CYGWIN__)
-#                               define PyMODINIT_FUNC __declspec(dllexport) PyObject*
+#                               define PyMODINIT_FUNC Py_EXPORTED_SYMBOL PyObject*
 #                       else /* __CYGWIN__ */
 #                               define PyMODINIT_FUNC PyObject*
 #                       endif /* __CYGWIN__ */
@@ -658,14 +660,14 @@ extern char * _getpty(int *, int, mode_t, int);
         /* failures similar to those described at the bottom of 4.1: */
         /* http://docs.python.org/extending/windows.html#a-cookbook-approach */
 #                       if !defined(__CYGWIN__)
-#                               define PyAPI_FUNC(RTYPE) __declspec(dllimport) RTYPE
+#                               define PyAPI_FUNC(RTYPE) Py_IMPORTED_SYMBOL RTYPE
 #                       endif /* !__CYGWIN__ */
-#                       define PyAPI_DATA(RTYPE) extern __declspec(dllimport) RTYPE
+#                       define PyAPI_DATA(RTYPE) extern Py_IMPORTED_SYMBOL RTYPE
         /* module init functions outside the core must be exported */
 #                       if defined(__cplusplus)
-#                               define PyMODINIT_FUNC extern "C" __declspec(dllexport) PyObject*
+#                               define PyMODINIT_FUNC extern "C" Py_EXPORTED_SYMBOL PyObject*
 #                       else /* __cplusplus */
-#                               define PyMODINIT_FUNC __declspec(dllexport) PyObject*
+#                               define PyMODINIT_FUNC Py_EXPORTED_SYMBOL PyObject*
 #                       endif /* __cplusplus */
 #               endif /* Py_BUILD_CORE */
 #       endif /* HAVE_DECLSPEC_DLL */
@@ -673,16 +675,16 @@ extern char * _getpty(int *, int, mode_t, int);
 
 /* If no external linkage macros defined by now, create defaults */
 #ifndef PyAPI_FUNC
-#       define PyAPI_FUNC(RTYPE) RTYPE
+#       define PyAPI_FUNC(RTYPE) Py_EXPORTED_SYMBOL RTYPE
 #endif
 #ifndef PyAPI_DATA
-#       define PyAPI_DATA(RTYPE) extern RTYPE
+#       define PyAPI_DATA(RTYPE) extern Py_EXPORTED_SYMBOL RTYPE
 #endif
 #ifndef PyMODINIT_FUNC
 #       if defined(__cplusplus)
-#               define PyMODINIT_FUNC extern "C" PyObject*
+#               define PyMODINIT_FUNC extern "C" Py_EXPORTED_SYMBOL PyObject*
 #       else /* __cplusplus */
-#               define PyMODINIT_FUNC PyObject*
+#               define PyMODINIT_FUNC Py_EXPORTED_SYMBOL PyObject*
 #       endif /* __cplusplus */
 #endif
 
diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-09-24-05-32-27.bpo-11410.vS182p.rst b/Misc/NEWS.d/next/Core and Builtins/2019-09-24-05-32-27.bpo-11410.vS182p.rst
new file mode 100644
index 0000000000000..2b572ad20f9a5
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2019-09-24-05-32-27.bpo-11410.vS182p.rst	
@@ -0,0 +1,4 @@
+Better control over symbol visibility is provided through use of the
+visibility attributes available in gcc >= 4.0, provided in a uniform way
+across POSIX and Windows. The POSIX build files have been updated to compile
+with -fvisibility=hidden, minimising exported symbols.
diff --git a/Modules/_ctypes/_ctypes_test.c b/Modules/_ctypes/_ctypes_test.c
index 8a0e5e9195f7b..40da652620271 100644
--- a/Modules/_ctypes/_ctypes_test.c
+++ b/Modules/_ctypes/_ctypes_test.c
@@ -4,11 +4,7 @@
 #include <windows.h>
 #endif
 
-#if defined(MS_WIN32) || defined(__CYGWIN__)
-#define EXPORT(x) __declspec(dllexport) x
-#else
-#define EXPORT(x) x
-#endif
+#define EXPORT(x) Py_EXPORTED_SYMBOL x
 
 /* some functions handy for testing */
 
diff --git a/Modules/_io/_iomodule.h b/Modules/_io/_iomodule.h
index 4d318acd0b3f8..a8f3951e57feb 100644
--- a/Modules/_io/_iomodule.h
+++ b/Modules/_io/_iomodule.h
@@ -2,6 +2,8 @@
  * Declarations shared between the different parts of the io module
  */
 
+#include "exports.h"
+
 /* ABCs */
 extern PyTypeObject PyIOBase_Type;
 extern PyTypeObject PyRawIOBase_Type;
@@ -183,4 +185,4 @@ extern PyObject *_PyIO_str_write;
 extern PyObject *_PyIO_empty_str;
 extern PyObject *_PyIO_empty_bytes;
 
-extern PyTypeObject _PyBytesIOBuffer_Type;
+extern Py_EXPORTED_SYMBOL PyTypeObject _PyBytesIOBuffer_Type;
diff --git a/Modules/_io/bytesio.c b/Modules/_io/bytesio.c
index a5f4c47863dfd..b5d308a8bca42 100644
--- a/Modules/_io/bytesio.c
+++ b/Modules/_io/bytesio.c
@@ -1124,7 +1124,7 @@ static PyBufferProcs bytesiobuf_as_buffer = {
     (releasebufferproc) bytesiobuf_releasebuffer,
 };
 
-PyTypeObject _PyBytesIOBuffer_Type = {
+Py_EXPORTED_SYMBOL PyTypeObject _PyBytesIOBuffer_Type = {
     PyVarObject_HEAD_INIT(NULL, 0)
     "_io._BytesIOBuffer",                      /*tp_name*/
     sizeof(bytesiobuf),                        /*tp_basicsize*/
diff --git a/Parser/pgen/grammar.py b/Parser/pgen/grammar.py
index 56188db775a80..ce40e160ca886 100644
--- a/Parser/pgen/grammar.py
+++ b/Parser/pgen/grammar.py
@@ -61,13 +61,14 @@ def produce_graminit_h(self, writer):
     def produce_graminit_c(self, writer):
         writer("/* Generated by Parser/pgen */\n\n")
 
+        writer('#include "exports.h"\n')
         writer('#include "grammar.h"\n')
-        writer("grammar _PyParser_Grammar;\n")
+        writer("Py_EXPORTED_SYMBOL grammar _PyParser_Grammar;\n")
 
         self.print_dfas(writer)
         self.print_labels(writer)
 
-        writer("grammar _PyParser_Grammar = {\n")
+        writer("Py_EXPORTED_SYMBOL grammar _PyParser_Grammar = {\n")
         writer("    {n_dfas},\n".format(n_dfas=len(self.dfas)))
         writer("    dfas,\n")
         writer("    {{{n_labels}, labels}},\n".format(n_labels=len(self.labels)))
diff --git a/Python/getargs.c b/Python/getargs.c
index 7723ae353710c..0ca0862912c87 100644
--- a/Python/getargs.c
+++ b/Python/getargs.c
@@ -106,7 +106,7 @@ PyArg_Parse(PyObject *args, const char *format, ...)
     return retval;
 }
 
-int
+PyAPI_FUNC(int)
 _PyArg_Parse_SizeT(PyObject *args, const char *format, ...)
 {
     int retval;
@@ -131,7 +131,7 @@ PyArg_ParseTuple(PyObject *args, const char *format, ...)
     return retval;
 }
 
-int
+PyAPI_FUNC(int)
 _PyArg_ParseTuple_SizeT(PyObject *args, const char *format, ...)
 {
     int retval;
@@ -156,7 +156,7 @@ _PyArg_ParseStack(PyObject *const *args, Py_ssize_t nargs, const char *format, .
     return retval;
 }
 
-int
+PyAPI_FUNC(int)
 _PyArg_ParseStack_SizeT(PyObject *const *args, Py_ssize_t nargs, const char *format, ...)
 {
     int retval;
@@ -182,7 +182,7 @@ PyArg_VaParse(PyObject *args, const char *format, va_list va)
     return retval;
 }
 
-int
+PyAPI_FUNC(int)
 _PyArg_VaParse_SizeT(PyObject *args, const char *format, va_list va)
 {
     va_list lva;
@@ -1442,7 +1442,7 @@ PyArg_ParseTupleAndKeywords(PyObject *args,
     return retval;
 }
 
-int
+PyAPI_FUNC(int)
 _PyArg_ParseTupleAndKeywords_SizeT(PyObject *args,
                                   PyObject *keywords,
                                   const char *format,
@@ -1493,7 +1493,7 @@ PyArg_VaParseTupleAndKeywords(PyObject *args,
     return retval;
 }
 
-int
+PyAPI_FUNC(int)
 _PyArg_VaParseTupleAndKeywords_SizeT(PyObject *args,
                                     PyObject *keywords,
                                     const char *format,
@@ -1519,7 +1519,7 @@ _PyArg_VaParseTupleAndKeywords_SizeT(PyObject *args,
     return retval;
 }
 
-int
+PyAPI_FUNC(int)
 _PyArg_ParseTupleAndKeywordsFast(PyObject *args, PyObject *keywords,
                             struct _PyArg_Parser *parser, ...)
 {
@@ -1532,7 +1532,7 @@ _PyArg_ParseTupleAndKeywordsFast(PyObject *args, PyObject *keywords,
     return retval;
 }
 
-int
+PyAPI_FUNC(int)
 _PyArg_ParseTupleAndKeywordsFast_SizeT(PyObject *args, PyObject *keywords,
                             struct _PyArg_Parser *parser, ...)
 {
@@ -1545,7 +1545,7 @@ _PyArg_ParseTupleAndKeywordsFast_SizeT(PyObject *args, PyObject *keywords,
     return retval;
 }
 
-int
+PyAPI_FUNC(int)
 _PyArg_ParseStackAndKeywords(PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames,
                   struct _PyArg_Parser *parser, ...)
 {
@@ -1558,7 +1558,7 @@ _PyArg_ParseStackAndKeywords(PyObject *const *args, Py_ssize_t nargs, PyObject *
     return retval;
 }
 
-int
+PyAPI_FUNC(int)
 _PyArg_ParseStackAndKeywords_SizeT(PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames,
                         struct _PyArg_Parser *parser, ...)
 {
@@ -1572,7 +1572,7 @@ _PyArg_ParseStackAndKeywords_SizeT(PyObject *const *args, Py_ssize_t nargs, PyOb
 }
 
 
-int
+PyAPI_FUNC(int)
 _PyArg_VaParseTupleAndKeywordsFast(PyObject *args, PyObject *keywords,
                             struct _PyArg_Parser *parser, va_list va)
 {
@@ -1586,7 +1586,7 @@ _PyArg_VaParseTupleAndKeywordsFast(PyObject *args, PyObject *keywords,
     return retval;
 }
 
-int
+PyAPI_FUNC(int)
 _PyArg_VaParseTupleAndKeywordsFast_SizeT(PyObject *args, PyObject *keywords,
                             struct _PyArg_Parser *parser, va_list va)
 {
diff --git a/Python/graminit.c b/Python/graminit.c
index 7c40ce933c1d6..62d9ae207f72f 100644
--- a/Python/graminit.c
+++ b/Python/graminit.c
@@ -1,7 +1,8 @@
 /* Generated by Parser/pgen */
 
+#include "exports.h"
 #include "grammar.h"
-grammar _PyParser_Grammar;
+Py_EXPORTED_SYMBOL grammar _PyParser_Grammar;
 static const arc arcs_0_0[3] = {
     {2, 1},
     {3, 2},
@@ -2693,7 +2694,7 @@ static const label labels[183] = {
     {346, 0},
     {347, 0},
 };
-grammar _PyParser_Grammar = {
+Py_EXPORTED_SYMBOL grammar _PyParser_Grammar = {
     92,
     dfas,
     {183, labels},
diff --git a/Python/pythonrun.c b/Python/pythonrun.c
index 6089086078730..702505893fb99 100644
--- a/Python/pythonrun.c
+++ b/Python/pythonrun.c
@@ -57,7 +57,7 @@ _Py_static_string(PyId_string, "<string>");
 extern "C" {
 #endif
 
-extern grammar _PyParser_Grammar; /* From graminit.c */
+extern Py_EXPORTED_SYMBOL grammar _PyParser_Grammar; /* From graminit.c */
 
 /* Forward */
 static void flush_io(void);
diff --git a/configure b/configure
index f1979c1b8124c..4f094ea650702 100755
--- a/configure
+++ b/configure
@@ -7341,6 +7341,47 @@ $as_echo "$ac_cv_enable_implicit_function_declaration_error" >&6; }
       CFLAGS_NODIST="$CFLAGS_NODIST -Werror=implicit-function-declaration"
     fi
 
+    { $as_echo "$as_me:${as_lineno-$LINENO}: checking if we can use visibility in $CC" >&5
+$as_echo_n "checking if we can use visibility in $CC... " >&6; }
+     ac_save_cc="$CC"
+     CC="$CC -fvisibility=hidden"
+     if ${ac_cv_enable_visibility+:} false; then :
+  $as_echo_n "(cached) " >&6
+else
+  cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+/* end confdefs.h.  */
+
+
+int
+main ()
+{
+
+  ;
+  return 0;
+}
+
+_ACEOF
+if ac_fn_c_try_compile "$LINENO"; then :
+
+           ac_cv_enable_visibility=yes
+
+else
+
+           ac_cv_enable_visibility=no
+
+fi
+rm -f core conftest.err conftest.$ac_objext conftest.$ac_ext
+fi
+
+     CC="$ac_save_cc"
+    { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_enable_visibility" >&5
+$as_echo "$ac_cv_enable_visibility" >&6; }
+
+    if test $ac_cv_enable_visibility = yes
+    then
+      CFLAGS_NODIST="$CFLAGS_NODIST -fvisibility=hidden"
+    fi
+
     # if using gcc on alpha, use -mieee to get (near) full IEEE 754
     # support.  Without this, treatment of subnormals doesn't follow
     # the standard.
diff --git a/configure.ac b/configure.ac
index 917a9087185e3..8de53408eea74 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1787,6 +1787,26 @@ yes)
       CFLAGS_NODIST="$CFLAGS_NODIST -Werror=implicit-function-declaration"
     fi
 
+    AC_MSG_CHECKING(if we can use visibility in $CC)
+     ac_save_cc="$CC"
+     CC="$CC -fvisibility=hidden"
+     AC_CACHE_VAL(ac_cv_enable_visibility,
+       AC_COMPILE_IFELSE(
+         [
+	   AC_LANG_PROGRAM([[]], [[]])
+	 ],[
+           ac_cv_enable_visibility=yes
+	 ],[
+           ac_cv_enable_visibility=no
+	 ]))
+     CC="$ac_save_cc"
+    AC_MSG_RESULT($ac_cv_enable_visibility)
+
+    if test $ac_cv_enable_visibility = yes
+    then
+      CFLAGS_NODIST="$CFLAGS_NODIST -fvisibility=hidden"
+    fi
+
     # if using gcc on alpha, use -mieee to get (near) full IEEE 754
     # support.  Without this, treatment of subnormals doesn't follow
     # the standard.


From webhook-mailer at python.org  Tue Oct 15 03:30:24 2019
From: webhook-mailer at python.org (Ned Deily)
Date: Tue, 15 Oct 2019 07:30:24 -0000
Subject: [Python-checkins] bpo-34344 Fix
 AbstractEventLoopPolicy.get_event_loop docstring (GH-16463)
Message-ID: <mailman.30.1571124625.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/88204a3908663a574919f9c17278bcabed0598fa
commit: 88204a3908663a574919f9c17278bcabed0598fa
branch: 3.7
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Ned Deily <nad at python.org>
date: 2019-10-14T16:48:16-04:00
summary:

bpo-34344 Fix AbstractEventLoopPolicy.get_event_loop docstring (GH-16463)

(cherry picked from commit b23a8423a923077e4f83d3f328bb7542b4c940ed)

Co-authored-by: idomic <michael.ido at gmail.com>

files:
M Lib/asyncio/events.py

diff --git a/Lib/asyncio/events.py b/Lib/asyncio/events.py
index e4e632206af1b..e8a6a1beeab9b 100644
--- a/Lib/asyncio/events.py
+++ b/Lib/asyncio/events.py
@@ -630,9 +630,9 @@ def __init__(self):
         self._local = self._Local()
 
     def get_event_loop(self):
-        """Get the event loop.
+        """Get the event loop for the current context.
 
-        This may be None or an instance of EventLoop.
+        Returns an instance of EventLoop or raises an exception.
         """
         if (self._local._loop is None and
                 not self._local._set_called and


From webhook-mailer at python.org  Tue Oct 15 05:26:23 2019
From: webhook-mailer at python.org (Petr Viktorin)
Date: Tue, 15 Oct 2019 09:26:23 -0000
Subject: [Python-checkins] bpo-38470: Fix
 test_compileall.test_compile_dir_maxlevels() (GH-16789)
Message-ID: <mailman.31.1571131583.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/eb1dda2b56f67f09352c303588c28880c471ae87
commit: eb1dda2b56f67f09352c303588c28880c471ae87
branch: master
author: Victor Stinner <vstinner at redhat.com>
committer: Petr Viktorin <encukou at gmail.com>
date: 2019-10-15T11:26:13+02:00
summary:

bpo-38470: Fix test_compileall.test_compile_dir_maxlevels() (GH-16789)

Fix test_compile_dir_maxlevels() on Windows without long path
support: only create 3 subdirectories instead of between 20 and 100
subdirectories.

Fix also compile_dir() to use the current sys.getrecursionlimit()
value as the default maxlevels value, rather than using
sys.getrecursionlimit() value read at startup.

files:
A Misc/NEWS.d/next/Tests/2019-10-14-22-46-35.bpo-38470.NHtzpy.rst
M Lib/compileall.py
M Lib/test/test_compileall.py

diff --git a/Lib/compileall.py b/Lib/compileall.py
index 26caf34baec0a..8cfde5b7b3e0a 100644
--- a/Lib/compileall.py
+++ b/Lib/compileall.py
@@ -19,11 +19,9 @@
 from functools import partial
 from pathlib import Path
 
-RECURSION_LIMIT = sys.getrecursionlimit()
-
 __all__ = ["compile_dir","compile_file","compile_path"]
 
-def _walk_dir(dir, maxlevels=RECURSION_LIMIT, quiet=0):
+def _walk_dir(dir, maxlevels, quiet=0):
     if quiet < 2 and isinstance(dir, os.PathLike):
         dir = os.fspath(dir)
     if not quiet:
@@ -46,7 +44,7 @@ def _walk_dir(dir, maxlevels=RECURSION_LIMIT, quiet=0):
             yield from _walk_dir(fullname, maxlevels=maxlevels - 1,
                                  quiet=quiet)
 
-def compile_dir(dir, maxlevels=RECURSION_LIMIT, ddir=None, force=False,
+def compile_dir(dir, maxlevels=None, ddir=None, force=False,
                 rx=None, quiet=0, legacy=False, optimize=-1, workers=1,
                 invalidation_mode=None, stripdir=None,
                 prependdir=None, limit_sl_dest=None):
@@ -83,6 +81,8 @@ def compile_dir(dir, maxlevels=RECURSION_LIMIT, ddir=None, force=False,
             from concurrent.futures import ProcessPoolExecutor
         except ImportError:
             workers = 1
+    if maxlevels is None:
+        maxlevels = sys.getrecursionlimit()
     files = _walk_dir(dir, quiet=quiet, maxlevels=maxlevels)
     success = True
     if workers != 1 and ProcessPoolExecutor is not None:
@@ -285,7 +285,7 @@ def main():
     parser = argparse.ArgumentParser(
         description='Utilities to support installing Python libraries.')
     parser.add_argument('-l', action='store_const', const=0,
-                        default=RECURSION_LIMIT, dest='maxlevels',
+                        default=None, dest='maxlevels',
                         help="don't recurse into subdirectories")
     parser.add_argument('-r', type=int, dest='recursion',
                         help=('control the maximum recursion level. '
diff --git a/Lib/test/test_compileall.py b/Lib/test/test_compileall.py
index 3bb03b3dd3846..2ebcb424095d8 100644
--- a/Lib/test/test_compileall.py
+++ b/Lib/test/test_compileall.py
@@ -46,57 +46,6 @@ def setUp(self):
     def tearDown(self):
         shutil.rmtree(self.directory)
 
-    def create_long_path(self):
-        long_path = os.path.join(self.directory, "long")
-
-        # Create a long path, 10 directories at a time.
-        # It will be 100 directories deep, or shorter if the OS limits it.
-        for i in range(10):
-            longer_path = os.path.join(
-                long_path, *(f"dir_{i}_{j}" for j in range(10))
-            )
-
-            # Check if we can open __pycache__/*.pyc.
-            # Also, put in the source file that we want to compile
-            longer_source = os.path.join(longer_path, '_test_long.py')
-            longer_cache = importlib.util.cache_from_source(longer_source)
-            try:
-                os.makedirs(longer_path)
-                shutil.copyfile(self.source_path, longer_source)
-                os.makedirs(os.path.dirname(longer_cache))
-                # Make sure we can write to the cache
-                with open(longer_cache, 'w'):
-                    pass
-            except FileNotFoundError:
-                # On Windows, a  FileNotFoundError("The filename or extension
-                # is too long") is raised for long paths
-                if sys.platform == "win32":
-                    break
-                else:
-                    raise
-            except OSError as exc:
-                if exc.errno == errno.ENAMETOOLONG:
-                    break
-                else:
-                    raise
-
-            # Remove the __pycache__
-            shutil.rmtree(os.path.dirname(longer_cache))
-
-            long_path = longer_path
-            long_source = longer_source
-            long_cache = longer_cache
-
-        # On Windows, MAX_PATH is 260 characters, our path with the 20
-        # directories is 160 characters long, leaving something for the
-        # root (self.directory) as well.
-        # Tests assume long_path contains at least 10 directories.
-        if i < 2:
-            raise ValueError(f'"Long path" is too short: {long_path}')
-
-        self.source_path_long = long_source
-        self.bc_path_long = long_cache
-
     def add_bad_source_file(self):
         self.bad_source_path = os.path.join(self.directory, '_test_bad.py')
         with open(self.bad_source_path, 'w') as file:
@@ -247,14 +196,21 @@ def test_compile_missing_multiprocessing(self, compile_file_mock):
         self.assertTrue(compile_file_mock.called)
 
     def test_compile_dir_maxlevels(self):
-        # Test the actual impact of maxlevels attr
-        self.create_long_path()
-        compileall.compile_dir(os.path.join(self.directory, "long"),
-                               maxlevels=10, quiet=True)
-        self.assertFalse(os.path.isfile(self.bc_path_long))
-        compileall.compile_dir(os.path.join(self.directory, "long"),
-                               quiet=True)
-        self.assertTrue(os.path.isfile(self.bc_path_long))
+        # Test the actual impact of maxlevels parameter
+        depth = 3
+        path = self.directory
+        for i in range(1, depth + 1):
+            path = os.path.join(path, f"dir_{i}")
+            source = os.path.join(path, 'script.py')
+            os.mkdir(path)
+            shutil.copyfile(self.source_path, source)
+        pyc_filename = importlib.util.cache_from_source(source)
+
+        compileall.compile_dir(self.directory, quiet=True, maxlevels=depth - 1)
+        self.assertFalse(os.path.isfile(pyc_filename))
+
+        compileall.compile_dir(self.directory, quiet=True, maxlevels=depth)
+        self.assertTrue(os.path.isfile(pyc_filename))
 
     def test_strip_only(self):
         fullpath = ["test", "build", "real", "path"]
diff --git a/Misc/NEWS.d/next/Tests/2019-10-14-22-46-35.bpo-38470.NHtzpy.rst b/Misc/NEWS.d/next/Tests/2019-10-14-22-46-35.bpo-38470.NHtzpy.rst
new file mode 100644
index 0000000000000..bb4710f86d4c8
--- /dev/null
+++ b/Misc/NEWS.d/next/Tests/2019-10-14-22-46-35.bpo-38470.NHtzpy.rst
@@ -0,0 +1,3 @@
+Fix ``test_compileall.test_compile_dir_maxlevels()`` on Windows without long
+path support: only create 3 subdirectories instead of between 20 and 100
+subdirectories.


From webhook-mailer at python.org  Tue Oct 15 07:40:18 2019
From: webhook-mailer at python.org (Pablo Galindo)
Date: Tue, 15 Oct 2019 11:40:18 -0000
Subject: [Python-checkins] bpo-38478: Correctly handle keyword argument with
 same name as positional-only parameter (GH-16800)
Message-ID: <mailman.32.1571139620.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/f3ef06a7cb347ab7bd3cc2b0b3dcebe4f9ff36f9
commit: f3ef06a7cb347ab7bd3cc2b0b3dcebe4f9ff36f9
branch: master
author: Pablo Galindo <Pablogsal at gmail.com>
committer: GitHub <noreply at github.com>
date: 2019-10-15T12:40:02+01:00
summary:

bpo-38478: Correctly handle keyword argument with same name as positional-only parameter (GH-16800)

files:
A Misc/NEWS.d/next/Library/2019-10-15-11-37-57.bpo-38478.A87OPO.rst
M Lib/inspect.py
M Lib/test/test_inspect.py

diff --git a/Lib/inspect.py b/Lib/inspect.py
index c2a1ed4148e74..3ff395ca33377 100644
--- a/Lib/inspect.py
+++ b/Lib/inspect.py
@@ -2960,7 +2960,7 @@ def _bind(self, args, kwargs, *, partial=False):
                         arguments[param.name] = tuple(values)
                         break
 
-                    if param.name in kwargs:
+                    if param.name in kwargs and param.kind != _POSITIONAL_ONLY:
                         raise TypeError(
                             'multiple values for argument {arg!r}'.format(
                                 arg=param.name)) from None
diff --git a/Lib/test/test_inspect.py b/Lib/test/test_inspect.py
index 9d28d5ad570d1..d95e742c8dd64 100644
--- a/Lib/test/test_inspect.py
+++ b/Lib/test/test_inspect.py
@@ -3573,6 +3573,16 @@ def make_set():
         iterator = iter(range(5))
         self.assertEqual(self.call(setcomp_func, iterator), {0, 1, 4, 9, 16})
 
+    def test_signature_bind_posonly_kwargs(self):
+        def foo(bar, /, **kwargs):
+            return bar, kwargs.get(bar)
+
+        sig = inspect.signature(foo)
+        result = sig.bind("pos-only", bar="keyword")
+
+        self.assertEqual(result.kwargs, {"bar": "keyword"})
+        self.assertIn(("bar", "pos-only"), result.arguments.items())
+
 
 class TestBoundArguments(unittest.TestCase):
     def test_signature_bound_arguments_unhashable(self):
diff --git a/Misc/NEWS.d/next/Library/2019-10-15-11-37-57.bpo-38478.A87OPO.rst b/Misc/NEWS.d/next/Library/2019-10-15-11-37-57.bpo-38478.A87OPO.rst
new file mode 100644
index 0000000000000..b19fa23639d88
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-10-15-11-37-57.bpo-38478.A87OPO.rst
@@ -0,0 +1,3 @@
+Fixed a bug in :meth:`inspect.signature.bind` that was causing it to fail
+when handling a keyword argument with same name as positional-only parameter.
+Patch by Pablo Galindo.


From webhook-mailer at python.org  Tue Oct 15 08:00:22 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Tue, 15 Oct 2019 12:00:22 -0000
Subject: [Python-checkins] bpo-37961,
 tracemalloc: add Traceback.total_nframe (GH-15545)
Message-ID: <mailman.33.1571140822.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/8d59eb1b66c51b2b918da9881c57d07d08df43b7
commit: 8d59eb1b66c51b2b918da9881c57d07d08df43b7
branch: master
author: Julien Danjou <julien at danjou.info>
committer: Victor Stinner <vstinner at python.org>
date: 2019-10-15T14:00:16+02:00
summary:

bpo-37961, tracemalloc: add Traceback.total_nframe (GH-15545)

Add a total_nframe field to the traces collected by the tracemalloc module.
This field indicates the original number of frames before it was truncated.

files:
A Misc/NEWS.d/next/Library/2019-08-27-10-30-44.bpo-37961.4nm0zZ.rst
M Doc/library/tracemalloc.rst
M Lib/test/test_tracemalloc.py
M Lib/tracemalloc.py
M Modules/_tracemalloc.c

diff --git a/Doc/library/tracemalloc.rst b/Doc/library/tracemalloc.rst
index 2d327c0254099..e423470036558 100644
--- a/Doc/library/tracemalloc.rst
+++ b/Doc/library/tracemalloc.rst
@@ -313,6 +313,9 @@ Functions
    frames. By default, a trace of a memory block only stores the most recent
    frame: the limit is ``1``. *nframe* must be greater or equal to ``1``.
 
+   You can still read the original number of total frames that composed the
+   traceback by looking at the :attr:`Traceback.total_nframe` attribute.
+
    Storing more than ``1`` frame is only useful to compute statistics grouped
    by ``'traceback'`` or to compute cumulative statistics: see the
    :meth:`Snapshot.compare_to` and :meth:`Snapshot.statistics` methods.
@@ -659,6 +662,9 @@ Traceback
 
    When a snapshot is taken, tracebacks of traces are limited to
    :func:`get_traceback_limit` frames. See the :func:`take_snapshot` function.
+   The original number of frames of the traceback is stored in the
+   :attr:`Traceback.total_nframe` attribute. That allows to know if a traceback
+   has been truncated by the traceback limit.
 
    The :attr:`Trace.traceback` attribute is an instance of :class:`Traceback`
    instance.
@@ -666,6 +672,15 @@ Traceback
    .. versionchanged:: 3.7
       Frames are now sorted from the oldest to the most recent, instead of most recent to oldest.
 
+   .. attribute:: total_nframe
+
+      Total number of frames that composed the traceback before truncation.
+      This attribute can be set to ``None`` if the information is not
+      available.
+
+   .. versionchanged:: 3.9
+      The :attr:`Traceback.total_nframe` attribute was added.
+
    .. method:: format(limit=None, most_recent_first=False)
 
       Format the traceback as a list of lines with newlines. Use the
diff --git a/Lib/test/test_tracemalloc.py b/Lib/test/test_tracemalloc.py
index 4b9bf4ed5da1d..7283d9c31db4e 100644
--- a/Lib/test/test_tracemalloc.py
+++ b/Lib/test/test_tracemalloc.py
@@ -36,7 +36,7 @@ def allocate_bytes(size):
     bytes_len = (size - EMPTY_STRING_SIZE)
     frames = get_frames(nframe, 1)
     data = b'x' * bytes_len
-    return data, tracemalloc.Traceback(frames)
+    return data, tracemalloc.Traceback(frames, min(len(frames), nframe))
 
 def create_snapshots():
     traceback_limit = 2
@@ -45,27 +45,27 @@ def create_snapshots():
     # traceback_frames) tuples. traceback_frames is a tuple of (filename,
     # line_number) tuples.
     raw_traces = [
-        (0, 10, (('a.py', 2), ('b.py', 4))),
-        (0, 10, (('a.py', 2), ('b.py', 4))),
-        (0, 10, (('a.py', 2), ('b.py', 4))),
+        (0, 10, (('a.py', 2), ('b.py', 4)), 3),
+        (0, 10, (('a.py', 2), ('b.py', 4)), 3),
+        (0, 10, (('a.py', 2), ('b.py', 4)), 3),
 
-        (1, 2, (('a.py', 5), ('b.py', 4))),
+        (1, 2, (('a.py', 5), ('b.py', 4)), 3),
 
-        (2, 66, (('b.py', 1),)),
+        (2, 66, (('b.py', 1),), 1),
 
-        (3, 7, (('<unknown>', 0),)),
+        (3, 7, (('<unknown>', 0),), 1),
     ]
     snapshot = tracemalloc.Snapshot(raw_traces, traceback_limit)
 
     raw_traces2 = [
-        (0, 10, (('a.py', 2), ('b.py', 4))),
-        (0, 10, (('a.py', 2), ('b.py', 4))),
-        (0, 10, (('a.py', 2), ('b.py', 4))),
+        (0, 10, (('a.py', 2), ('b.py', 4)), 3),
+        (0, 10, (('a.py', 2), ('b.py', 4)), 3),
+        (0, 10, (('a.py', 2), ('b.py', 4)), 3),
 
-        (2, 2, (('a.py', 5), ('b.py', 4))),
-        (2, 5000, (('a.py', 5), ('b.py', 4))),
+        (2, 2, (('a.py', 5), ('b.py', 4)), 3),
+        (2, 5000, (('a.py', 5), ('b.py', 4)), 3),
 
-        (4, 400, (('c.py', 578),)),
+        (4, 400, (('c.py', 578),), 1),
     ]
     snapshot2 = tracemalloc.Snapshot(raw_traces2, traceback_limit)
 
@@ -125,7 +125,7 @@ def test_new_reference(self):
 
         nframe = tracemalloc.get_traceback_limit()
         frames = get_frames(nframe, -3)
-        obj_traceback = tracemalloc.Traceback(frames)
+        obj_traceback = tracemalloc.Traceback(frames, min(len(frames), nframe))
 
         traceback = tracemalloc.get_object_traceback(obj)
         self.assertIsNotNone(traceback)
@@ -167,7 +167,7 @@ def test_get_traces(self):
         trace = self.find_trace(traces, obj_traceback)
 
         self.assertIsInstance(trace, tuple)
-        domain, size, traceback = trace
+        domain, size, traceback, length = trace
         self.assertEqual(size, obj_size)
         self.assertEqual(traceback, obj_traceback._frames)
 
@@ -197,8 +197,8 @@ def allocate_bytes4(size):
 
         trace1 = self.find_trace(traces, obj1_traceback)
         trace2 = self.find_trace(traces, obj2_traceback)
-        domain1, size1, traceback1 = trace1
-        domain2, size2, traceback2 = trace2
+        domain1, size1, traceback1, length1 = trace1
+        domain2, size2, traceback2, length2 = trace2
         self.assertIs(traceback2, traceback1)
 
     def test_get_traced_memory(self):
@@ -259,6 +259,9 @@ def test_snapshot(self):
         # take a snapshot
         snapshot = tracemalloc.take_snapshot()
 
+        # This can vary
+        self.assertGreater(snapshot.traces[1].traceback.total_nframe, 10)
+
         # write on disk
         snapshot.dump(support.TESTFN)
         self.addCleanup(support.unlink, support.TESTFN)
@@ -321,7 +324,7 @@ class TestSnapshot(unittest.TestCase):
     maxDiff = 4000
 
     def test_create_snapshot(self):
-        raw_traces = [(0, 5, (('a.py', 2),))]
+        raw_traces = [(0, 5, (('a.py', 2),), 10)]
 
         with contextlib.ExitStack() as stack:
             stack.enter_context(patch.object(tracemalloc, 'is_tracing',
@@ -336,6 +339,7 @@ def test_create_snapshot(self):
             self.assertEqual(len(snapshot.traces), 1)
             trace = snapshot.traces[0]
             self.assertEqual(trace.size, 5)
+            self.assertEqual(trace.traceback.total_nframe, 10)
             self.assertEqual(len(trace.traceback), 1)
             self.assertEqual(trace.traceback[0].filename, 'a.py')
             self.assertEqual(trace.traceback[0].lineno, 2)
@@ -351,11 +355,11 @@ def test_filter_traces(self):
         # exclude b.py
         snapshot3 = snapshot.filter_traces((filter1,))
         self.assertEqual(snapshot3.traces._traces, [
-            (0, 10, (('a.py', 2), ('b.py', 4))),
-            (0, 10, (('a.py', 2), ('b.py', 4))),
-            (0, 10, (('a.py', 2), ('b.py', 4))),
-            (1, 2, (('a.py', 5), ('b.py', 4))),
-            (3, 7, (('<unknown>', 0),)),
+            (0, 10, (('a.py', 2), ('b.py', 4)), 3),
+            (0, 10, (('a.py', 2), ('b.py', 4)), 3),
+            (0, 10, (('a.py', 2), ('b.py', 4)), 3),
+            (1, 2, (('a.py', 5), ('b.py', 4)), 3),
+            (3, 7, (('<unknown>', 0),), 1),
         ])
 
         # filter_traces() must not touch the original snapshot
@@ -364,10 +368,10 @@ def test_filter_traces(self):
         # only include two lines of a.py
         snapshot4 = snapshot3.filter_traces((filter2, filter3))
         self.assertEqual(snapshot4.traces._traces, [
-            (0, 10, (('a.py', 2), ('b.py', 4))),
-            (0, 10, (('a.py', 2), ('b.py', 4))),
-            (0, 10, (('a.py', 2), ('b.py', 4))),
-            (1, 2, (('a.py', 5), ('b.py', 4))),
+            (0, 10, (('a.py', 2), ('b.py', 4)), 3),
+            (0, 10, (('a.py', 2), ('b.py', 4)), 3),
+            (0, 10, (('a.py', 2), ('b.py', 4)), 3),
+            (1, 2, (('a.py', 5), ('b.py', 4)), 3),
         ])
 
         # No filter: just duplicate the snapshot
@@ -388,21 +392,21 @@ def test_filter_traces_domain(self):
         # exclude a.py of domain 1
         snapshot3 = snapshot.filter_traces((filter1,))
         self.assertEqual(snapshot3.traces._traces, [
-            (0, 10, (('a.py', 2), ('b.py', 4))),
-            (0, 10, (('a.py', 2), ('b.py', 4))),
-            (0, 10, (('a.py', 2), ('b.py', 4))),
-            (2, 66, (('b.py', 1),)),
-            (3, 7, (('<unknown>', 0),)),
+            (0, 10, (('a.py', 2), ('b.py', 4)), 3),
+            (0, 10, (('a.py', 2), ('b.py', 4)), 3),
+            (0, 10, (('a.py', 2), ('b.py', 4)), 3),
+            (2, 66, (('b.py', 1),), 1),
+            (3, 7, (('<unknown>', 0),), 1),
         ])
 
         # include domain 1
         snapshot3 = snapshot.filter_traces((filter1,))
         self.assertEqual(snapshot3.traces._traces, [
-            (0, 10, (('a.py', 2), ('b.py', 4))),
-            (0, 10, (('a.py', 2), ('b.py', 4))),
-            (0, 10, (('a.py', 2), ('b.py', 4))),
-            (2, 66, (('b.py', 1),)),
-            (3, 7, (('<unknown>', 0),)),
+            (0, 10, (('a.py', 2), ('b.py', 4)), 3),
+            (0, 10, (('a.py', 2), ('b.py', 4)), 3),
+            (0, 10, (('a.py', 2), ('b.py', 4)), 3),
+            (2, 66, (('b.py', 1),), 1),
+            (3, 7, (('<unknown>', 0),), 1),
         ])
 
     def test_filter_traces_domain_filter(self):
@@ -413,17 +417,17 @@ def test_filter_traces_domain_filter(self):
         # exclude domain 2
         snapshot3 = snapshot.filter_traces((filter1,))
         self.assertEqual(snapshot3.traces._traces, [
-            (0, 10, (('a.py', 2), ('b.py', 4))),
-            (0, 10, (('a.py', 2), ('b.py', 4))),
-            (0, 10, (('a.py', 2), ('b.py', 4))),
-            (1, 2, (('a.py', 5), ('b.py', 4))),
-            (2, 66, (('b.py', 1),)),
+            (0, 10, (('a.py', 2), ('b.py', 4)), 3),
+            (0, 10, (('a.py', 2), ('b.py', 4)), 3),
+            (0, 10, (('a.py', 2), ('b.py', 4)), 3),
+            (1, 2, (('a.py', 5), ('b.py', 4)), 3),
+            (2, 66, (('b.py', 1),), 1),
         ])
 
         # include domain 2
         snapshot3 = snapshot.filter_traces((filter2,))
         self.assertEqual(snapshot3.traces._traces, [
-            (3, 7, (('<unknown>', 0),)),
+            (3, 7, (('<unknown>', 0),), 1),
         ])
 
     def test_snapshot_group_by_line(self):
diff --git a/Lib/tracemalloc.py b/Lib/tracemalloc.py
index 80b521c2e1af9..69b4170ec8246 100644
--- a/Lib/tracemalloc.py
+++ b/Lib/tracemalloc.py
@@ -182,15 +182,20 @@ class Traceback(Sequence):
     Sequence of Frame instances sorted from the oldest frame
     to the most recent frame.
     """
-    __slots__ = ("_frames",)
+    __slots__ = ("_frames", '_total_nframe')
 
-    def __init__(self, frames):
+    def __init__(self, frames, total_nframe=None):
         Sequence.__init__(self)
         # frames is a tuple of frame tuples: see Frame constructor for the
         # format of a frame tuple; it is reversed, because _tracemalloc
         # returns frames sorted from most recent to oldest, but the
         # Python API expects oldest to most recent
         self._frames = tuple(reversed(frames))
+        self._total_nframe = total_nframe
+
+    @property
+    def total_nframe(self):
+        return self._total_nframe
 
     def __len__(self):
         return len(self._frames)
@@ -221,7 +226,12 @@ def __str__(self):
         return str(self[0])
 
     def __repr__(self):
-        return "<Traceback %r>" % (tuple(self),)
+        s = "<Traceback %r" % tuple(self)
+        if self._total_nframe is None:
+            s += ">"
+        else:
+            s += f" total_nframe={self.total_nframe}>"
+        return s
 
     def format(self, limit=None, most_recent_first=False):
         lines = []
@@ -280,7 +290,7 @@ def size(self):
 
     @property
     def traceback(self):
-        return Traceback(self._trace[2])
+        return Traceback(*self._trace[2:])
 
     def __eq__(self, other):
         if not isinstance(other, Trace):
@@ -378,7 +388,7 @@ def _match_traceback(self, traceback):
             return self._match_frame(filename, lineno)
 
     def _match(self, trace):
-        domain, size, traceback = trace
+        domain, size, traceback, total_nframe = trace
         res = self._match_traceback(traceback)
         if self.domain is not None:
             if self.inclusive:
@@ -398,7 +408,7 @@ def domain(self):
         return self._domain
 
     def _match(self, trace):
-        domain, size, traceback = trace
+        domain, size, traceback, total_nframe = trace
         return (domain == self.domain) ^ (not self.inclusive)
 
 
@@ -475,7 +485,7 @@ def _group_by(self, key_type, cumulative):
         tracebacks = {}
         if not cumulative:
             for trace in self.traces._traces:
-                domain, size, trace_traceback = trace
+                domain, size, trace_traceback, total_nframe = trace
                 try:
                     traceback = tracebacks[trace_traceback]
                 except KeyError:
@@ -496,7 +506,7 @@ def _group_by(self, key_type, cumulative):
         else:
             # cumulative statistics
             for trace in self.traces._traces:
-                domain, size, trace_traceback = trace
+                domain, size, trace_traceback, total_nframe = trace
                 for frame in trace_traceback:
                     try:
                         traceback = tracebacks[frame]
diff --git a/Misc/NEWS.d/next/Library/2019-08-27-10-30-44.bpo-37961.4nm0zZ.rst b/Misc/NEWS.d/next/Library/2019-08-27-10-30-44.bpo-37961.4nm0zZ.rst
new file mode 100644
index 0000000000000..ebfff77decff3
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-08-27-10-30-44.bpo-37961.4nm0zZ.rst
@@ -0,0 +1,2 @@
+Add a ``total_nframe`` field to the traces collected by the tracemalloc module.
+This field indicates the original number of frames before it was truncated.
diff --git a/Modules/_tracemalloc.c b/Modules/_tracemalloc.c
index 26d76004aaa52..211c6fbb8bcef 100644
--- a/Modules/_tracemalloc.c
+++ b/Modules/_tracemalloc.c
@@ -78,15 +78,20 @@ __attribute__((packed))
 
 typedef struct {
     Py_uhash_t hash;
-    int nframe;
+    /* Number of frames stored */
+    uint16_t nframe;
+    /* Total number of frames the traceback had */
+    uint16_t total_nframe;
     frame_t frames[1];
 } traceback_t;
 
 #define TRACEBACK_SIZE(NFRAME) \
         (sizeof(traceback_t) + sizeof(frame_t) * (NFRAME - 1))
 
-#define MAX_NFRAME \
-        ((INT_MAX - (int)sizeof(traceback_t)) / (int)sizeof(frame_t) + 1)
+/* The maximum number of frames is either:
+ - The maximum number of frames we can store in `traceback_t.nframe`
+ - The maximum memory size_t we can allocate */
+static const unsigned long MAX_NFRAME = Py_MIN(UINT16_MAX, ((SIZE_MAX - sizeof(traceback_t)) / sizeof(frame_t) + 1));
 
 
 static PyObject *unknown_filename = NULL;
@@ -308,6 +313,9 @@ hashtable_compare_traceback(_Py_hashtable_t *ht, const void *pkey,
     if (traceback1->nframe != traceback2->nframe)
         return 0;
 
+    if (traceback1->total_nframe != traceback2->total_nframe)
+        return 0;
+
     for (i=0; i < traceback1->nframe; i++) {
         frame1 = &traceback1->frames[i];
         frame2 = &traceback2->frames[i];
@@ -416,6 +424,7 @@ traceback_hash(traceback_t *traceback)
         /* the cast might truncate len; that doesn't change hash stability */
         mult += (Py_uhash_t)(82520UL + len + len);
     }
+    x ^= traceback->total_nframe;
     x += 97531UL;
     return x;
 }
@@ -436,11 +445,13 @@ traceback_get_frames(traceback_t *traceback)
     }
 
     for (pyframe = tstate->frame; pyframe != NULL; pyframe = pyframe->f_back) {
-        tracemalloc_get_frame(pyframe, &traceback->frames[traceback->nframe]);
-        assert(traceback->frames[traceback->nframe].filename != NULL);
-        traceback->nframe++;
-        if (traceback->nframe == _Py_tracemalloc_config.max_nframe)
-            break;
+        if (traceback->nframe < _Py_tracemalloc_config.max_nframe) {
+            tracemalloc_get_frame(pyframe, &traceback->frames[traceback->nframe]);
+            assert(traceback->frames[traceback->nframe].filename != NULL);
+            traceback->nframe++;
+        }
+        if (traceback->total_nframe < UINT16_MAX)
+            traceback->total_nframe++;
     }
 }
 
@@ -456,6 +467,7 @@ traceback_new(void)
     /* get frames */
     traceback = tracemalloc_traceback;
     traceback->nframe = 0;
+    traceback->total_nframe = 0;
     traceback_get_frames(traceback);
     if (traceback->nframe == 0)
         return &tracemalloc_empty_traceback;
@@ -1001,6 +1013,7 @@ tracemalloc_init(void)
     PyUnicode_InternInPlace(&unknown_filename);
 
     tracemalloc_empty_traceback.nframe = 1;
+    tracemalloc_empty_traceback.total_nframe = 1;
     /* borrowed reference */
     tracemalloc_empty_traceback.frames[0].filename = unknown_filename;
     tracemalloc_empty_traceback.frames[0].lineno = 0;
@@ -1046,10 +1059,10 @@ tracemalloc_start(int max_nframe)
     PyMemAllocatorEx alloc;
     size_t size;
 
-    if (max_nframe < 1 || max_nframe > MAX_NFRAME) {
+    if (max_nframe < 1 || (unsigned long) max_nframe > MAX_NFRAME) {
         PyErr_Format(PyExc_ValueError,
-                     "the number of frames must be in range [1; %i]",
-                     (int)MAX_NFRAME);
+                     "the number of frames must be in range [1; %lu]",
+                     MAX_NFRAME);
         return -1;
     }
 
@@ -1062,7 +1075,6 @@ tracemalloc_start(int max_nframe)
         return 0;
     }
 
-    assert(1 <= max_nframe && max_nframe <= MAX_NFRAME);
     _Py_tracemalloc_config.max_nframe = max_nframe;
 
     /* allocate a buffer to store a new traceback */
@@ -1234,7 +1246,7 @@ trace_to_pyobject(unsigned int domain, trace_t *trace,
     PyObject *trace_obj = NULL;
     PyObject *obj;
 
-    trace_obj = PyTuple_New(3);
+    trace_obj = PyTuple_New(4);
     if (trace_obj == NULL)
         return NULL;
 
@@ -1259,6 +1271,13 @@ trace_to_pyobject(unsigned int domain, trace_t *trace,
     }
     PyTuple_SET_ITEM(trace_obj, 2, obj);
 
+    obj = PyLong_FromUnsignedLong(trace->traceback->total_nframe);
+    if (obj == NULL) {
+        Py_DECREF(trace_obj);
+        return NULL;
+    }
+    PyTuple_SET_ITEM(trace_obj, 3, obj);
+
     return trace_obj;
 }
 


From webhook-mailer at python.org  Tue Oct 15 08:02:38 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Tue, 15 Oct 2019 12:02:38 -0000
Subject: [Python-checkins] bpo-38478: Correctly handle keyword argument with
 same name as positional-only parameter (GH-16800)
Message-ID: <mailman.34.1571140959.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/f705f8e9b58955d0d9083e98d71ba01b2e69798c
commit: f705f8e9b58955d0d9083e98d71ba01b2e69798c
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-15T05:02:34-07:00
summary:

bpo-38478: Correctly handle keyword argument with same name as positional-only parameter (GH-16800)

(cherry picked from commit f3ef06a7cb347ab7bd3cc2b0b3dcebe4f9ff36f9)

Co-authored-by: Pablo Galindo <Pablogsal at gmail.com>

files:
A Misc/NEWS.d/next/Library/2019-10-15-11-37-57.bpo-38478.A87OPO.rst
M Lib/inspect.py
M Lib/test/test_inspect.py

diff --git a/Lib/inspect.py b/Lib/inspect.py
index c2a1ed4148e74..3ff395ca33377 100644
--- a/Lib/inspect.py
+++ b/Lib/inspect.py
@@ -2960,7 +2960,7 @@ def _bind(self, args, kwargs, *, partial=False):
                         arguments[param.name] = tuple(values)
                         break
 
-                    if param.name in kwargs:
+                    if param.name in kwargs and param.kind != _POSITIONAL_ONLY:
                         raise TypeError(
                             'multiple values for argument {arg!r}'.format(
                                 arg=param.name)) from None
diff --git a/Lib/test/test_inspect.py b/Lib/test/test_inspect.py
index 1c516546f7684..ad93f304827fb 100644
--- a/Lib/test/test_inspect.py
+++ b/Lib/test/test_inspect.py
@@ -3577,6 +3577,16 @@ def make_set():
         iterator = iter(range(5))
         self.assertEqual(self.call(setcomp_func, iterator), {0, 1, 4, 9, 16})
 
+    def test_signature_bind_posonly_kwargs(self):
+        def foo(bar, /, **kwargs):
+            return bar, kwargs.get(bar)
+
+        sig = inspect.signature(foo)
+        result = sig.bind("pos-only", bar="keyword")
+
+        self.assertEqual(result.kwargs, {"bar": "keyword"})
+        self.assertIn(("bar", "pos-only"), result.arguments.items())
+
 
 class TestBoundArguments(unittest.TestCase):
     def test_signature_bound_arguments_unhashable(self):
diff --git a/Misc/NEWS.d/next/Library/2019-10-15-11-37-57.bpo-38478.A87OPO.rst b/Misc/NEWS.d/next/Library/2019-10-15-11-37-57.bpo-38478.A87OPO.rst
new file mode 100644
index 0000000000000..b19fa23639d88
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-10-15-11-37-57.bpo-38478.A87OPO.rst
@@ -0,0 +1,3 @@
+Fixed a bug in :meth:`inspect.signature.bind` that was causing it to fail
+when handling a keyword argument with same name as positional-only parameter.
+Patch by Pablo Galindo.


From webhook-mailer at python.org  Tue Oct 15 10:18:53 2019
From: webhook-mailer at python.org (Joannah Nanjekye)
Date: Tue, 15 Oct 2019 14:18:53 -0000
Subject: [Python-checkins] bpo-38468 : Refactor python-config (#16749)
Message-ID: <mailman.35.1571149134.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/3cd21aa6a1467723ccc85e6411a6cbe7fa81ef76
commit: 3cd21aa6a1467723ccc85e6411a6cbe7fa81ef76
branch: master
author: Joannah Nanjekye <33177550+nanjekyejoannah at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-15T11:18:47-03:00
summary:

bpo-38468 : Refactor python-config (#16749)

* refactor python-config

* ?? Added by blurb_it.

* Update Misc/NEWS.d/next/Build/2019-10-13-16-18-24.bpo-38468.buCO84.rst

Co-Authored-By: Pablo Galindo <Pablogsal at gmail.com>

* Use getvar for pyver

* Update news entry with affected file.

* Update Misc/NEWS.d/next/Build/2019-10-13-16-18-24.bpo-38468.buCO84.rst

Co-Authored-By: Victor Stinner <vstinner at python.org>

* Update 2019-10-13-16-18-24.bpo-38468.buCO84.rst

files:
A Misc/NEWS.d/next/Build/2019-10-13-16-18-24.bpo-38468.buCO84.rst
M Misc/python-config.in

diff --git a/Misc/NEWS.d/next/Build/2019-10-13-16-18-24.bpo-38468.buCO84.rst b/Misc/NEWS.d/next/Build/2019-10-13-16-18-24.bpo-38468.buCO84.rst
new file mode 100644
index 0000000000000..4fe4c2b25a1aa
--- /dev/null
+++ b/Misc/NEWS.d/next/Build/2019-10-13-16-18-24.bpo-38468.buCO84.rst
@@ -0,0 +1,2 @@
+Misc/python-config.in now uses `getvar()` for all still existing `sysconfig.get_config_var()` calls.
+Patch by Joannah Nanjekye.
diff --git a/Misc/python-config.in b/Misc/python-config.in
index 727c4a8682270..ebd99daa0c279 100644
--- a/Misc/python-config.in
+++ b/Misc/python-config.in
@@ -25,8 +25,8 @@ except getopt.error:
 if not opts:
     exit_with_usage()
 
-pyver = sysconfig.get_config_var('VERSION')
 getvar = sysconfig.get_config_var
+pyver = getvar('VERSION')
 
 opt_flags = [flag for (flag, val) in opts]
 
@@ -35,10 +35,10 @@ if '--help' in opt_flags:
 
 for opt in opt_flags:
     if opt == '--prefix':
-        print(sysconfig.get_config_var('prefix'))
+        print(getvar('prefix'))
 
     elif opt == '--exec-prefix':
-        print(sysconfig.get_config_var('exec_prefix'))
+        print(getvar('exec_prefix'))
 
     elif opt in ('--includes', '--cflags'):
         flags = ['-I' + sysconfig.get_path('include'),
@@ -65,10 +65,10 @@ for opt in opt_flags:
         print(' '.join(libs))
 
     elif opt == '--extension-suffix':
-        print(sysconfig.get_config_var('EXT_SUFFIX'))
+        print(getvar('EXT_SUFFIX'))
 
     elif opt == '--abiflags':
         print(sys.abiflags)
 
     elif opt == '--configdir':
-        print(sysconfig.get_config_var('LIBPL'))
+        print(getvar('LIBPL'))


From webhook-mailer at python.org  Tue Oct 15 16:49:21 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Tue, 15 Oct 2019 20:49:21 -0000
Subject: [Python-checkins] Typo fix - implemention should be implementation
 (GH-16806)
Message-ID: <mailman.36.1571172563.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/2798b60c55619d8981288b69c20ba4a09efc7b0b
commit: 2798b60c55619d8981288b69c20ba4a09efc7b0b
branch: master
author: Hansraj Das <raj.das.136 at gmail.com>
committer: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
date: 2019-10-15T13:49:13-07:00
summary:

Typo fix - implemention should be implementation (GH-16806)

files:
M Python/hamt.c

diff --git a/Python/hamt.c b/Python/hamt.c
index 28b4e1ef6cd2f..ea659c800fc46 100644
--- a/Python/hamt.c
+++ b/Python/hamt.c
@@ -6,7 +6,7 @@
 #include "structmember.h"
 
 /*
-This file provides an implemention of an immutable mapping using the
+This file provides an implementation of an immutable mapping using the
 Hash Array Mapped Trie (or HAMT) datastructure.
 
 This design allows to have:


From webhook-mailer at python.org  Tue Oct 15 17:07:45 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Tue, 15 Oct 2019 21:07:45 -0000
Subject: [Python-checkins] Typo fix - implemention should be implementation
 (GH-16806)
Message-ID: <mailman.37.1571173666.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/079a1e6307be6cd35122af7773e2bcecdb828239
commit: 079a1e6307be6cd35122af7773e2bcecdb828239
branch: 3.7
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-15T14:07:40-07:00
summary:

Typo fix - implemention should be implementation (GH-16806)

(cherry picked from commit 2798b60c55619d8981288b69c20ba4a09efc7b0b)

Co-authored-by: Hansraj Das <raj.das.136 at gmail.com>

files:
M Python/hamt.c

diff --git a/Python/hamt.c b/Python/hamt.c
index 71a5ec87c5856..6bcdfac15fd30 100644
--- a/Python/hamt.c
+++ b/Python/hamt.c
@@ -5,7 +5,7 @@
 #include "internal/hamt.h"
 
 /*
-This file provides an implemention of an immutable mapping using the
+This file provides an implementation of an immutable mapping using the
 Hash Array Mapped Trie (or HAMT) datastructure.
 
 This design allows to have:


From webhook-mailer at python.org  Tue Oct 15 17:09:23 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Tue, 15 Oct 2019 21:09:23 -0000
Subject: [Python-checkins] Typo fix - implemention should be implementation
 (GH-16806)
Message-ID: <mailman.38.1571173764.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/8ca282f12a91551368321f828a1d482df75fe920
commit: 8ca282f12a91551368321f828a1d482df75fe920
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-15T14:09:19-07:00
summary:

Typo fix - implemention should be implementation (GH-16806)

(cherry picked from commit 2798b60c55619d8981288b69c20ba4a09efc7b0b)

Co-authored-by: Hansraj Das <raj.das.136 at gmail.com>

files:
M Python/hamt.c

diff --git a/Python/hamt.c b/Python/hamt.c
index b3cbf9ac82087..38412596a37b0 100644
--- a/Python/hamt.c
+++ b/Python/hamt.c
@@ -6,7 +6,7 @@
 #include "structmember.h"
 
 /*
-This file provides an implemention of an immutable mapping using the
+This file provides an implementation of an immutable mapping using the
 Hash Array Mapped Trie (or HAMT) datastructure.
 
 This design allows to have:


From webhook-mailer at python.org  Tue Oct 15 17:26:08 2019
From: webhook-mailer at python.org (Ned Deily)
Date: Tue, 15 Oct 2019 21:26:08 -0000
Subject: [Python-checkins] Update doc switcher list for 3.8.0 (GH-16809)
Message-ID: <mailman.39.1571174769.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/3f36043db22361500f52634f2b8de49dde0e7da9
commit: 3f36043db22361500f52634f2b8de49dde0e7da9
branch: master
author: Ned Deily <nad at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-15T17:26:05-04:00
summary:

Update doc switcher list for 3.8.0 (GH-16809)

files:
M Doc/tools/static/switchers.js

diff --git a/Doc/tools/static/switchers.js b/Doc/tools/static/switchers.js
index fa298a76b0fe1..ef6f473452065 100644
--- a/Doc/tools/static/switchers.js
+++ b/Doc/tools/static/switchers.js
@@ -11,7 +11,7 @@
 
   var all_versions = {
     '3.9': 'dev (3.9)',
-    '3.8': 'pre (3.8)',
+    '3.8': '3.8',
     '3.7': '3.7',
     '3.6': '3.6',
     '3.5': '3.5',


From webhook-mailer at python.org  Tue Oct 15 17:31:55 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Tue, 15 Oct 2019 21:31:55 -0000
Subject: [Python-checkins] Update doc switcher list for 3.8.0 (GH-16809)
Message-ID: <mailman.40.1571175116.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/5ff8e2dfbe7fa528f0e7ea73cbe9767784c29f04
commit: 5ff8e2dfbe7fa528f0e7ea73cbe9767784c29f04
branch: 3.7
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-15T14:31:51-07:00
summary:

Update doc switcher list for 3.8.0 (GH-16809)

(cherry picked from commit 3f36043db22361500f52634f2b8de49dde0e7da9)

Co-authored-by: Ned Deily <nad at python.org>

files:
M Doc/tools/static/switchers.js

diff --git a/Doc/tools/static/switchers.js b/Doc/tools/static/switchers.js
index fa298a76b0fe1..ef6f473452065 100644
--- a/Doc/tools/static/switchers.js
+++ b/Doc/tools/static/switchers.js
@@ -11,7 +11,7 @@
 
   var all_versions = {
     '3.9': 'dev (3.9)',
-    '3.8': 'pre (3.8)',
+    '3.8': '3.8',
     '3.7': '3.7',
     '3.6': '3.6',
     '3.5': '3.5',


From webhook-mailer at python.org  Tue Oct 15 17:33:37 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Tue, 15 Oct 2019 21:33:37 -0000
Subject: [Python-checkins] Update doc switcher list for 3.8.0 (GH-16809)
Message-ID: <mailman.41.1571175219.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/2d647c0728d82a143bc20707258a7a1be0366558
commit: 2d647c0728d82a143bc20707258a7a1be0366558
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-15T14:33:34-07:00
summary:

Update doc switcher list for 3.8.0 (GH-16809)

(cherry picked from commit 3f36043db22361500f52634f2b8de49dde0e7da9)

Co-authored-by: Ned Deily <nad at python.org>

files:
M Doc/tools/static/switchers.js

diff --git a/Doc/tools/static/switchers.js b/Doc/tools/static/switchers.js
index fa298a76b0fe1..ef6f473452065 100644
--- a/Doc/tools/static/switchers.js
+++ b/Doc/tools/static/switchers.js
@@ -11,7 +11,7 @@
 
   var all_versions = {
     '3.9': 'dev (3.9)',
-    '3.8': 'pre (3.8)',
+    '3.8': '3.8',
     '3.7': '3.7',
     '3.6': '3.6',
     '3.5': '3.5',


From webhook-mailer at python.org  Tue Oct 15 17:46:23 2019
From: webhook-mailer at python.org (Ned Deily)
Date: Tue, 15 Oct 2019 21:46:23 -0000
Subject: [Python-checkins] Update doc switcher list for 3.8.0 (GH-16809)
 (GH-16812)
Message-ID: <mailman.42.1571175983.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/1039f215746fd2ba21741e8a11f1f8badcea2a66
commit: 1039f215746fd2ba21741e8a11f1f8badcea2a66
branch: 3.6
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Ned Deily <nad at python.org>
date: 2019-10-15T17:46:19-04:00
summary:

Update doc switcher list for 3.8.0 (GH-16809) (GH-16812)

(cherry picked from commit 3f36043db22361500f52634f2b8de49dde0e7da9)

Co-authored-by: Ned Deily <nad at python.org>

files:
M Doc/tools/static/switchers.js

diff --git a/Doc/tools/static/switchers.js b/Doc/tools/static/switchers.js
index 7a5c8bd384783..075e4774938f4 100644
--- a/Doc/tools/static/switchers.js
+++ b/Doc/tools/static/switchers.js
@@ -11,7 +11,7 @@
 
   var all_versions = {
     '3.9': 'dev (3.9)',
-    '3.8': 'pre (3.8)',
+    '3.8': '3.8',
     '3.7': '3.7',
     '3.6': '3.6',
     '3.5': '3.5',


From webhook-mailer at python.org  Tue Oct 15 20:36:48 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Wed, 16 Oct 2019 00:36:48 -0000
Subject: [Python-checkins] bpo-35998: Fix
 test_asyncio.test_start_tls_server_1() (GH-16815)
Message-ID: <mailman.43.1571186208.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/fab4ef2df0857ab0c97f3058ac5ec3280c4eb891
commit: fab4ef2df0857ab0c97f3058ac5ec3280c4eb891
branch: master
author: Victor Stinner <vstinner at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-16T02:36:42+02:00
summary:

bpo-35998: Fix test_asyncio.test_start_tls_server_1() (GH-16815)

main() is now responsible to send the ANSWER, rather than
ServerProto. main() now waits until it got the HELLO before sending
the ANSWER over the new transport.

Previously, there was a race condition between main() replacing the
protocol and the protocol sending the ANSWER once it gets the HELLO.

TLSv1.3 was disabled for the test: reenable it.

files:
A Misc/NEWS.d/next/Tests/2019-10-16-01-36-15.bpo-35998.G305Bf.rst
M Lib/test/test_asyncio/test_sslproto.py

diff --git a/Lib/test/test_asyncio/test_sslproto.py b/Lib/test/test_asyncio/test_sslproto.py
index c115e46e705be..a7c089018ab43 100644
--- a/Lib/test/test_asyncio/test_sslproto.py
+++ b/Lib/test/test_asyncio/test_sslproto.py
@@ -497,14 +497,6 @@ def test_start_tls_server_1(self):
 
         server_context = test_utils.simple_server_sslcontext()
         client_context = test_utils.simple_client_sslcontext()
-        if (sys.platform.startswith('freebsd')
-                or sys.platform.startswith('win')
-                or sys.platform.startswith('darwin')):
-            # bpo-35031: Some FreeBSD and Windows buildbots fail to run this test
-            # as the eof was not being received by the server if the payload
-            # size is not big enough. This behaviour only appears if the
-            # client is using TLS1.3.  Also seen on macOS.
-            client_context.options |= ssl.OP_NO_TLSv1_3
         answer = None
 
         def client(sock, addr):
@@ -521,9 +513,10 @@ def client(sock, addr):
             sock.close()
 
         class ServerProto(asyncio.Protocol):
-            def __init__(self, on_con, on_con_lost):
+            def __init__(self, on_con, on_con_lost, on_got_hello):
                 self.on_con = on_con
                 self.on_con_lost = on_con_lost
+                self.on_got_hello = on_got_hello
                 self.data = b''
                 self.transport = None
 
@@ -537,7 +530,7 @@ def replace_transport(self, tr):
             def data_received(self, data):
                 self.data += data
                 if len(self.data) >= len(HELLO_MSG):
-                    self.transport.write(ANSWER)
+                    self.on_got_hello.set_result(None)
 
             def connection_lost(self, exc):
                 self.transport = None
@@ -546,7 +539,7 @@ def connection_lost(self, exc):
                 else:
                     self.on_con_lost.set_exception(exc)
 
-        async def main(proto, on_con, on_con_lost):
+        async def main(proto, on_con, on_con_lost, on_got_hello):
             tr = await on_con
             tr.write(HELLO_MSG)
 
@@ -556,9 +549,11 @@ def connection_lost(self, exc):
                 tr, proto, server_context,
                 server_side=True,
                 ssl_handshake_timeout=self.TIMEOUT)
-
             proto.replace_transport(new_tr)
 
+            await on_got_hello
+            new_tr.write(ANSWER)
+
             await on_con_lost
             self.assertEqual(proto.data, HELLO_MSG)
             new_tr.close()
@@ -566,7 +561,8 @@ def connection_lost(self, exc):
         async def run_main():
             on_con = self.loop.create_future()
             on_con_lost = self.loop.create_future()
-            proto = ServerProto(on_con, on_con_lost)
+            on_got_hello = self.loop.create_future()
+            proto = ServerProto(on_con, on_con_lost, on_got_hello)
 
             server = await self.loop.create_server(
                 lambda: proto, '127.0.0.1', 0)
@@ -575,7 +571,7 @@ def connection_lost(self, exc):
             with self.tcp_client(lambda sock: client(sock, addr),
                                  timeout=self.TIMEOUT):
                 await asyncio.wait_for(
-                    main(proto, on_con, on_con_lost),
+                    main(proto, on_con, on_con_lost, on_got_hello),
                     timeout=self.TIMEOUT)
 
             server.close()
diff --git a/Misc/NEWS.d/next/Tests/2019-10-16-01-36-15.bpo-35998.G305Bf.rst b/Misc/NEWS.d/next/Tests/2019-10-16-01-36-15.bpo-35998.G305Bf.rst
new file mode 100644
index 0000000000000..43d3942fd0772
--- /dev/null
+++ b/Misc/NEWS.d/next/Tests/2019-10-16-01-36-15.bpo-35998.G305Bf.rst
@@ -0,0 +1,5 @@
+Fix a race condition in test_asyncio.test_start_tls_server_1(). Previously,
+there was a race condition between the test main() function which replaces the
+protocol and the test ServerProto protocol which sends ANSWER once it gets
+HELLO. Now, only the test main() function is responsible to send data,
+ServerProto no longer sends data.


From webhook-mailer at python.org  Tue Oct 15 23:56:54 2019
From: webhook-mailer at python.org (Neil Schemenauer)
Date: Wed, 16 Oct 2019 03:56:54 -0000
Subject: [Python-checkins] bpo-38006: Add unit test for weakref clear bug
 (GH-16788)
Message-ID: <mailman.44.1571198214.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/392a13bb9346331b087bcd8bb1b37072c126abee
commit: 392a13bb9346331b087bcd8bb1b37072c126abee
branch: master
author: Neil Schemenauer <nas-github at arctrix.com>
committer: GitHub <noreply at github.com>
date: 2019-10-15T20:56:48-07:00
summary:

bpo-38006: Add unit test for weakref clear bug (GH-16788)

files:
M Lib/test/test_gc.py
M Modules/_testcapimodule.c

diff --git a/Lib/test/test_gc.py b/Lib/test/test_gc.py
index fe9d6c2f766cc..fdb8752b66817 100644
--- a/Lib/test/test_gc.py
+++ b/Lib/test/test_gc.py
@@ -1,4 +1,5 @@
 import unittest
+import unittest.mock
 from test.support import (verbose, refcount_test, run_unittest,
                           strip_python_stderr, cpython_only, start_threads,
                           temp_dir, requires_type_collecting, TESTFN, unlink,
@@ -22,6 +23,11 @@ def __new__(cls, *args, **kwargs):
                 raise TypeError('requires _testcapi.with_tp_del')
         return C
 
+try:
+    from _testcapi import ContainerNoGC
+except ImportError:
+    ContainerNoGC = None
+
 ### Support code
 ###############################################################################
 
@@ -959,6 +965,66 @@ def getstats():
 
         gc.enable()
 
+    @unittest.skipIf(ContainerNoGC is None,
+                     'requires ContainerNoGC extension type')
+    def test_trash_weakref_clear(self):
+        # Test that trash weakrefs are properly cleared (bpo-38006).
+        #
+        # Structure we are creating:
+        #
+        #   Z <- Y <- A--+--> WZ -> C
+        #             ^  |
+        #             +--+
+        # where:
+        #   WZ is a weakref to Z with callback C
+        #   Y doesn't implement tp_traverse
+        #   A contains a reference to itself, Y and WZ
+        #
+        # A, Y, Z, WZ are all trash.  The GC doesn't know that Z is trash
+        # because Y does not implement tp_traverse.  To show the bug, WZ needs
+        # to live long enough so that Z is deallocated before it.  Then, if
+        # gcmodule is buggy, when Z is being deallocated, C will run.
+        #
+        # To ensure WZ lives long enough, we put it in a second reference
+        # cycle.  That trick only works due to the ordering of the GC prev/next
+        # linked lists.  So, this test is a bit fragile.
+        #
+        # The bug reported in bpo-38006 is caused because the GC did not
+        # clear WZ before starting the process of calling tp_clear on the
+        # trash.  Normally, handle_weakrefs() would find the weakref via Z and
+        # clear it.  However, since the GC cannot find Z, WR is not cleared and
+        # it can execute during delete_garbage().  That can lead to disaster
+        # since the callback might tinker with objects that have already had
+        # tp_clear called on them (leaving them in possibly invalid states).
+
+        callback = unittest.mock.Mock()
+
+        class A:
+            __slots__ = ['a', 'y', 'wz']
+
+        class Z:
+            pass
+
+        # setup required object graph, as described above
+        a = A()
+        a.a = a
+        a.y = ContainerNoGC(Z())
+        a.wz = weakref.ref(a.y.value, callback)
+        # create second cycle to keep WZ alive longer
+        wr_cycle = [a.wz]
+        wr_cycle.append(wr_cycle)
+        # ensure trash unrelated to this test is gone
+        gc.collect()
+        gc.disable()
+        # release references and create trash
+        del a, wr_cycle
+        gc.collect()
+        # if called, it means there is a bug in the GC.  The weakref should be
+        # cleared before Z dies.
+        callback.assert_not_called()
+        gc.enable()
+
+
 class GCCallbackTests(unittest.TestCase):
     def setUp(self):
         # Save gc state and disable it.
diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c
index c009d254da8f1..a18a8e7553ef2 100644
--- a/Modules/_testcapimodule.c
+++ b/Modules/_testcapimodule.c
@@ -6538,6 +6538,53 @@ static PyTypeObject MethStatic_Type = {
         "Class with static methods to test calling conventions"),
 };
 
+/* ContainerNoGC -- a simple container without GC methods */
+
+typedef struct {
+    PyObject_HEAD
+    PyObject *value;
+} ContainerNoGCobject;
+
+static PyObject *
+ContainerNoGC_new(PyTypeObject *type, PyObject *args, PyObject *kwargs)
+{
+    PyObject *value;
+    char *names[] = {"value", NULL};
+    if (!PyArg_ParseTupleAndKeywords(args, kwargs, "O", names, &value)) {
+        return NULL;
+    }
+    PyObject *self = type->tp_alloc(type, 0);
+    if (self == NULL) {
+        return NULL;
+    }
+    Py_INCREF(value);
+    ((ContainerNoGCobject *)self)->value = value;
+    return self;
+}
+
+static void
+ContainerNoGC_dealloc(ContainerNoGCobject *self)
+{
+    Py_DECREF(self->value);
+    Py_TYPE(self)->tp_free((PyObject *)self);
+}
+
+static PyMemberDef ContainerNoGC_members[] = {
+    {"value", T_OBJECT, offsetof(ContainerNoGCobject, value), READONLY,
+     PyDoc_STR("a container value for test purposes")},
+    {0}
+};
+
+static PyTypeObject ContainerNoGC_type = {
+    PyVarObject_HEAD_INIT(NULL, 0)
+    "_testcapi.ContainerNoGC",
+    sizeof(ContainerNoGCobject),
+    .tp_dealloc = (destructor)ContainerNoGC_dealloc,
+    .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE,
+    .tp_members = ContainerNoGC_members,
+    .tp_new = ContainerNoGC_new,
+};
+
 
 static struct PyModuleDef _testcapimodule = {
     PyModuleDef_HEAD_INIT,
@@ -6735,6 +6782,14 @@ PyInit__testcapi(void)
     Py_DECREF(subclass_with_finalizer_bases);
     PyModule_AddObject(m, "HeapCTypeSubclassWithFinalizer", HeapCTypeSubclassWithFinalizer);
 
+    if (PyType_Ready(&ContainerNoGC_type) < 0) {
+        return NULL;
+    }
+    Py_INCREF(&ContainerNoGC_type);
+    if (PyModule_AddObject(m, "ContainerNoGC",
+                           (PyObject *) &ContainerNoGC_type) < 0)
+        return NULL;
+
     PyState_AddModule(m, &_testcapimodule);
     return m;
 }


From webhook-mailer at python.org  Wed Oct 16 04:37:02 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Wed, 16 Oct 2019 08:37:02 -0000
Subject: [Python-checkins] bpo-35998: Fix
 test_asyncio.test_start_tls_server_1() (GH-16815)
Message-ID: <mailman.45.1571215023.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/afbcd9f26d1bf5390f56f18c9afbf753e48f230d
commit: afbcd9f26d1bf5390f56f18c9afbf753e48f230d
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-16T01:36:57-07:00
summary:

bpo-35998: Fix test_asyncio.test_start_tls_server_1() (GH-16815)


main() is now responsible to send the ANSWER, rather than
ServerProto. main() now waits until it got the HELLO before sending
the ANSWER over the new transport.

Previously, there was a race condition between main() replacing the
protocol and the protocol sending the ANSWER once it gets the HELLO.

TLSv1.3 was disabled for the test: reenable it.
(cherry picked from commit fab4ef2df0857ab0c97f3058ac5ec3280c4eb891)

Co-authored-by: Victor Stinner <vstinner at python.org>

files:
A Misc/NEWS.d/next/Tests/2019-10-16-01-36-15.bpo-35998.G305Bf.rst
M Lib/test/test_asyncio/test_sslproto.py

diff --git a/Lib/test/test_asyncio/test_sslproto.py b/Lib/test/test_asyncio/test_sslproto.py
index c115e46e705be..a7c089018ab43 100644
--- a/Lib/test/test_asyncio/test_sslproto.py
+++ b/Lib/test/test_asyncio/test_sslproto.py
@@ -497,14 +497,6 @@ def test_start_tls_server_1(self):
 
         server_context = test_utils.simple_server_sslcontext()
         client_context = test_utils.simple_client_sslcontext()
-        if (sys.platform.startswith('freebsd')
-                or sys.platform.startswith('win')
-                or sys.platform.startswith('darwin')):
-            # bpo-35031: Some FreeBSD and Windows buildbots fail to run this test
-            # as the eof was not being received by the server if the payload
-            # size is not big enough. This behaviour only appears if the
-            # client is using TLS1.3.  Also seen on macOS.
-            client_context.options |= ssl.OP_NO_TLSv1_3
         answer = None
 
         def client(sock, addr):
@@ -521,9 +513,10 @@ def client(sock, addr):
             sock.close()
 
         class ServerProto(asyncio.Protocol):
-            def __init__(self, on_con, on_con_lost):
+            def __init__(self, on_con, on_con_lost, on_got_hello):
                 self.on_con = on_con
                 self.on_con_lost = on_con_lost
+                self.on_got_hello = on_got_hello
                 self.data = b''
                 self.transport = None
 
@@ -537,7 +530,7 @@ def replace_transport(self, tr):
             def data_received(self, data):
                 self.data += data
                 if len(self.data) >= len(HELLO_MSG):
-                    self.transport.write(ANSWER)
+                    self.on_got_hello.set_result(None)
 
             def connection_lost(self, exc):
                 self.transport = None
@@ -546,7 +539,7 @@ def connection_lost(self, exc):
                 else:
                     self.on_con_lost.set_exception(exc)
 
-        async def main(proto, on_con, on_con_lost):
+        async def main(proto, on_con, on_con_lost, on_got_hello):
             tr = await on_con
             tr.write(HELLO_MSG)
 
@@ -556,9 +549,11 @@ def connection_lost(self, exc):
                 tr, proto, server_context,
                 server_side=True,
                 ssl_handshake_timeout=self.TIMEOUT)
-
             proto.replace_transport(new_tr)
 
+            await on_got_hello
+            new_tr.write(ANSWER)
+
             await on_con_lost
             self.assertEqual(proto.data, HELLO_MSG)
             new_tr.close()
@@ -566,7 +561,8 @@ def connection_lost(self, exc):
         async def run_main():
             on_con = self.loop.create_future()
             on_con_lost = self.loop.create_future()
-            proto = ServerProto(on_con, on_con_lost)
+            on_got_hello = self.loop.create_future()
+            proto = ServerProto(on_con, on_con_lost, on_got_hello)
 
             server = await self.loop.create_server(
                 lambda: proto, '127.0.0.1', 0)
@@ -575,7 +571,7 @@ def connection_lost(self, exc):
             with self.tcp_client(lambda sock: client(sock, addr),
                                  timeout=self.TIMEOUT):
                 await asyncio.wait_for(
-                    main(proto, on_con, on_con_lost),
+                    main(proto, on_con, on_con_lost, on_got_hello),
                     timeout=self.TIMEOUT)
 
             server.close()
diff --git a/Misc/NEWS.d/next/Tests/2019-10-16-01-36-15.bpo-35998.G305Bf.rst b/Misc/NEWS.d/next/Tests/2019-10-16-01-36-15.bpo-35998.G305Bf.rst
new file mode 100644
index 0000000000000..43d3942fd0772
--- /dev/null
+++ b/Misc/NEWS.d/next/Tests/2019-10-16-01-36-15.bpo-35998.G305Bf.rst
@@ -0,0 +1,5 @@
+Fix a race condition in test_asyncio.test_start_tls_server_1(). Previously,
+there was a race condition between the test main() function which replaces the
+protocol and the test ServerProto protocol which sends ANSWER once it gets
+HELLO. Now, only the test main() function is responsible to send data,
+ServerProto no longer sends data.


From webhook-mailer at python.org  Wed Oct 16 05:44:42 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Wed, 16 Oct 2019 09:44:42 -0000
Subject: [Python-checkins] bpo-35998: Fix
 test_asyncio.test_start_tls_server_1() (GH-16815) (GH-16818)
Message-ID: <mailman.46.1571219083.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/1bceb0e58905fd6d241d7d566e19ca78558973dd
commit: 1bceb0e58905fd6d241d7d566e19ca78558973dd
branch: 3.7
author: Victor Stinner <vstinner at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-16T11:44:30+02:00
summary:

bpo-35998: Fix test_asyncio.test_start_tls_server_1() (GH-16815) (GH-16818)

main() is now responsible to send the ANSWER, rather than
ServerProto. main() now waits until it got the HELLO before sending
the ANSWER over the new transport.

Previously, there was a race condition between main() replacing the
protocol and the protocol sending the ANSWER once it gets the HELLO.

TLSv1.3 was disabled for the test: reenable it.

(cherry picked from commit fab4ef2df0857ab0c97f3058ac5ec3280c4eb891)

files:
A Misc/NEWS.d/next/Tests/2019-10-16-01-36-15.bpo-35998.G305Bf.rst
M Lib/test/test_asyncio/test_sslproto.py

diff --git a/Lib/test/test_asyncio/test_sslproto.py b/Lib/test/test_asyncio/test_sslproto.py
index 866ef81fb2ee3..8720961442f60 100644
--- a/Lib/test/test_asyncio/test_sslproto.py
+++ b/Lib/test/test_asyncio/test_sslproto.py
@@ -495,14 +495,6 @@ def test_start_tls_server_1(self):
 
         server_context = test_utils.simple_server_sslcontext()
         client_context = test_utils.simple_client_sslcontext()
-        if (sys.platform.startswith('freebsd')
-                or sys.platform.startswith('win')
-                or sys.platform.startswith('darwin')):
-            # bpo-35031: Some FreeBSD and Windows buildbots fail to run this test
-            # as the eof was not being received by the server if the payload
-            # size is not big enough. This behaviour only appears if the
-            # client is using TLS1.3.  Also seen on macOS.
-            client_context.options |= ssl.OP_NO_TLSv1_3
         answer = None
 
         def client(sock, addr):
@@ -519,9 +511,10 @@ def client(sock, addr):
             sock.close()
 
         class ServerProto(asyncio.Protocol):
-            def __init__(self, on_con, on_con_lost):
+            def __init__(self, on_con, on_con_lost, on_got_hello):
                 self.on_con = on_con
                 self.on_con_lost = on_con_lost
+                self.on_got_hello = on_got_hello
                 self.data = b''
                 self.transport = None
 
@@ -535,7 +528,7 @@ def replace_transport(self, tr):
             def data_received(self, data):
                 self.data += data
                 if len(self.data) >= len(HELLO_MSG):
-                    self.transport.write(ANSWER)
+                    self.on_got_hello.set_result(None)
 
             def connection_lost(self, exc):
                 self.transport = None
@@ -544,7 +537,7 @@ def connection_lost(self, exc):
                 else:
                     self.on_con_lost.set_exception(exc)
 
-        async def main(proto, on_con, on_con_lost):
+        async def main(proto, on_con, on_con_lost, on_got_hello):
             tr = await on_con
             tr.write(HELLO_MSG)
 
@@ -554,9 +547,11 @@ def connection_lost(self, exc):
                 tr, proto, server_context,
                 server_side=True,
                 ssl_handshake_timeout=self.TIMEOUT)
-
             proto.replace_transport(new_tr)
 
+            await on_got_hello
+            new_tr.write(ANSWER)
+
             await on_con_lost
             self.assertEqual(proto.data, HELLO_MSG)
             new_tr.close()
@@ -564,7 +559,8 @@ def connection_lost(self, exc):
         async def run_main():
             on_con = self.loop.create_future()
             on_con_lost = self.loop.create_future()
-            proto = ServerProto(on_con, on_con_lost)
+            on_got_hello = self.loop.create_future()
+            proto = ServerProto(on_con, on_con_lost, on_got_hello)
 
             server = await self.loop.create_server(
                 lambda: proto, '127.0.0.1', 0)
@@ -573,7 +569,7 @@ def connection_lost(self, exc):
             with self.tcp_client(lambda sock: client(sock, addr),
                                  timeout=self.TIMEOUT):
                 await asyncio.wait_for(
-                    main(proto, on_con, on_con_lost),
+                    main(proto, on_con, on_con_lost, on_got_hello),
                     loop=self.loop, timeout=self.TIMEOUT)
 
             server.close()
diff --git a/Misc/NEWS.d/next/Tests/2019-10-16-01-36-15.bpo-35998.G305Bf.rst b/Misc/NEWS.d/next/Tests/2019-10-16-01-36-15.bpo-35998.G305Bf.rst
new file mode 100644
index 0000000000000..43d3942fd0772
--- /dev/null
+++ b/Misc/NEWS.d/next/Tests/2019-10-16-01-36-15.bpo-35998.G305Bf.rst
@@ -0,0 +1,5 @@
+Fix a race condition in test_asyncio.test_start_tls_server_1(). Previously,
+there was a race condition between the test main() function which replaces the
+protocol and the test ServerProto protocol which sends ANSWER once it gets
+HELLO. Now, only the test main() function is responsible to send data,
+ServerProto no longer sends data.


From webhook-mailer at python.org  Wed Oct 16 13:27:24 2019
From: webhook-mailer at python.org (Steve Dower)
Date: Wed, 16 Oct 2019 17:27:24 -0000
Subject: [Python-checkins] bpo-38492: Remove pythonw.exe dependency on the
 Microsoft C++ runtime (GH-16824)
Message-ID: <mailman.47.1571246845.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/7aebbd1182bc818324656b2fb764679faf51fdff
commit: 7aebbd1182bc818324656b2fb764679faf51fdff
branch: master
author: Steve Dower <steve.dower at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-16T10:27:17-07:00
summary:

bpo-38492: Remove pythonw.exe dependency on the Microsoft C++ runtime (GH-16824)

files:
A Misc/NEWS.d/next/Windows/2019-10-16-09-49-09.bpo-38492.Te1LxC.rst
M PCbuild/pythonw_uwp.vcxproj

diff --git a/Misc/NEWS.d/next/Windows/2019-10-16-09-49-09.bpo-38492.Te1LxC.rst b/Misc/NEWS.d/next/Windows/2019-10-16-09-49-09.bpo-38492.Te1LxC.rst
new file mode 100644
index 0000000000000..41fe695413f97
--- /dev/null
+++ b/Misc/NEWS.d/next/Windows/2019-10-16-09-49-09.bpo-38492.Te1LxC.rst
@@ -0,0 +1 @@
+Remove ``pythonw.exe`` dependency on the Microsoft C++ runtime.
diff --git a/PCbuild/pythonw_uwp.vcxproj b/PCbuild/pythonw_uwp.vcxproj
index 79e105877fbeb..e2c01710498ee 100644
--- a/PCbuild/pythonw_uwp.vcxproj
+++ b/PCbuild/pythonw_uwp.vcxproj
@@ -65,6 +65,15 @@
       <SubSystem>Windows</SubSystem>
     </Link>
   </ItemDefinitionGroup>
+  <ItemDefinitionGroup Condition="$(Configuration) != 'Debug'">
+    <ClCompile>
+      <RuntimeLibrary>Multithreaded</RuntimeLibrary>
+    </ClCompile>
+    <Link>
+      <AdditionalDependencies>ucrt.lib;%(AdditionalDependencies)</AdditionalDependencies>
+      <IgnoreSpecificDefaultLibraries>libucrt;%(IgnoreSpecificDefaultLibraries)</IgnoreSpecificDefaultLibraries>
+    </Link>
+  </ItemDefinitionGroup>
   <ItemGroup>
     <None Include="..\PC\pyconw.ico" />
   </ItemGroup>


From webhook-mailer at python.org  Wed Oct 16 13:46:39 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Wed, 16 Oct 2019 17:46:39 -0000
Subject: [Python-checkins] bpo-38492: Remove pythonw.exe dependency on the
 Microsoft C++ runtime (GH-16824)
Message-ID: <mailman.48.1571248000.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/7773d391f8261a97af502ecfa74a2a588ee05f36
commit: 7773d391f8261a97af502ecfa74a2a588ee05f36
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-16T10:46:29-07:00
summary:

bpo-38492: Remove pythonw.exe dependency on the Microsoft C++ runtime (GH-16824)

(cherry picked from commit 7aebbd1182bc818324656b2fb764679faf51fdff)

Co-authored-by: Steve Dower <steve.dower at python.org>

files:
A Misc/NEWS.d/next/Windows/2019-10-16-09-49-09.bpo-38492.Te1LxC.rst
M PCbuild/pythonw_uwp.vcxproj

diff --git a/Misc/NEWS.d/next/Windows/2019-10-16-09-49-09.bpo-38492.Te1LxC.rst b/Misc/NEWS.d/next/Windows/2019-10-16-09-49-09.bpo-38492.Te1LxC.rst
new file mode 100644
index 0000000000000..41fe695413f97
--- /dev/null
+++ b/Misc/NEWS.d/next/Windows/2019-10-16-09-49-09.bpo-38492.Te1LxC.rst
@@ -0,0 +1 @@
+Remove ``pythonw.exe`` dependency on the Microsoft C++ runtime.
diff --git a/PCbuild/pythonw_uwp.vcxproj b/PCbuild/pythonw_uwp.vcxproj
index 79e105877fbeb..e2c01710498ee 100644
--- a/PCbuild/pythonw_uwp.vcxproj
+++ b/PCbuild/pythonw_uwp.vcxproj
@@ -65,6 +65,15 @@
       <SubSystem>Windows</SubSystem>
     </Link>
   </ItemDefinitionGroup>
+  <ItemDefinitionGroup Condition="$(Configuration) != 'Debug'">
+    <ClCompile>
+      <RuntimeLibrary>Multithreaded</RuntimeLibrary>
+    </ClCompile>
+    <Link>
+      <AdditionalDependencies>ucrt.lib;%(AdditionalDependencies)</AdditionalDependencies>
+      <IgnoreSpecificDefaultLibraries>libucrt;%(IgnoreSpecificDefaultLibraries)</IgnoreSpecificDefaultLibraries>
+    </Link>
+  </ItemDefinitionGroup>
   <ItemGroup>
     <None Include="..\PC\pyconw.ico" />
   </ItemGroup>


From webhook-mailer at python.org  Wed Oct 16 18:29:19 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Wed, 16 Oct 2019 22:29:19 -0000
Subject: [Python-checkins] bpo-37531: regrtest now catchs ProcessLookupError
 (GH-16827)
Message-ID: <mailman.49.1571264960.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/a661392f8fb5ac4fc095aa1845d1eb7a25c4e9be
commit: a661392f8fb5ac4fc095aa1845d1eb7a25c4e9be
branch: master
author: Victor Stinner <vstinner at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-17T00:29:12+02:00
summary:

bpo-37531: regrtest now catchs ProcessLookupError (GH-16827)

Fix a warning on a race condition on TestWorkerProcess.kill(): ignore
silently ProcessLookupError rather than logging an useless warning.

files:
M Lib/test/libregrtest/runtest_mp.py

diff --git a/Lib/test/libregrtest/runtest_mp.py b/Lib/test/libregrtest/runtest_mp.py
index 71db8d6966859..8026de187d587 100644
--- a/Lib/test/libregrtest/runtest_mp.py
+++ b/Lib/test/libregrtest/runtest_mp.py
@@ -152,6 +152,11 @@ def _kill(self):
         print(f"Kill {self}", file=sys.stderr, flush=True)
         try:
             popen.kill()
+        except ProcessLookupError:
+            # Process completed, the TestWorkerProcess thread read its exit
+            # status, but Popen.send_signal() read the returncode just before
+            # Popen.wait() set returncode.
+            pass
         except OSError as exc:
             print_warning(f"Failed to kill {self}: {exc!r}")
 


From webhook-mailer at python.org  Thu Oct 17 03:12:47 2019
From: webhook-mailer at python.org (Inada Naoki)
Date: Thu, 17 Oct 2019 07:12:47 -0000
Subject: [Python-checkins] bpo-37587: json: Use _PyUnicodeWriter when
 scanning string. (GH-15591)
Message-ID: <mailman.50.1571296369.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/9c11029bb41caab5576f354fbf808a5e91325bb0
commit: 9c11029bb41caab5576f354fbf808a5e91325bb0
branch: master
author: Inada Naoki <songofacandy at gmail.com>
committer: GitHub <noreply at github.com>
date: 2019-10-17T16:12:41+09:00
summary:

bpo-37587: json: Use _PyUnicodeWriter when scanning string. (GH-15591)

files:
A Misc/NEWS.d/next/Library/2019-08-29-18-48-48.bpo-37587.N7TGTC.rst
M Modules/_json.c

diff --git a/Misc/NEWS.d/next/Library/2019-08-29-18-48-48.bpo-37587.N7TGTC.rst b/Misc/NEWS.d/next/Library/2019-08-29-18-48-48.bpo-37587.N7TGTC.rst
new file mode 100644
index 0000000000000..92bebeea30215
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-08-29-18-48-48.bpo-37587.N7TGTC.rst
@@ -0,0 +1,2 @@
+``_json.scanstring`` is now up to 3x faster when there are many backslash
+escaped characters in the JSON string.
diff --git a/Modules/_json.c b/Modules/_json.c
index 112903ea577a2..54ac605fd7ef4 100644
--- a/Modules/_json.c
+++ b/Modules/_json.c
@@ -73,19 +73,6 @@ static PyMemberDef encoder_members[] = {
     {NULL}
 };
 
-static PyObject *
-join_list_unicode(PyObject *lst)
-{
-    /* return u''.join(lst) */
-    static PyObject *sep = NULL;
-    if (sep == NULL) {
-        sep = PyUnicode_FromStringAndSize("", 0);
-        if (sep == NULL)
-            return NULL;
-    }
-    return PyUnicode_Join(sep, lst);
-}
-
 /* Forward decls */
 
 static PyObject *
@@ -385,21 +372,6 @@ _build_rval_index_tuple(PyObject *rval, Py_ssize_t idx) {
     return tpl;
 }
 
-#define APPEND_OLD_CHUNK \
-    if (chunk != NULL) { \
-        if (chunks == NULL) { \
-            chunks = PyList_New(0); \
-            if (chunks == NULL) { \
-                goto bail; \
-            } \
-        } \
-        if (PyList_Append(chunks, chunk)) { \
-            Py_CLEAR(chunk); \
-            goto bail; \
-        } \
-        Py_CLEAR(chunk); \
-    }
-
 static PyObject *
 scanstring_unicode(PyObject *pystr, Py_ssize_t end, int strict, Py_ssize_t *next_end_ptr)
 {
@@ -417,12 +389,14 @@ scanstring_unicode(PyObject *pystr, Py_ssize_t end, int strict, Py_ssize_t *next
     Py_ssize_t next /* = begin */;
     const void *buf;
     int kind;
-    PyObject *chunks = NULL;
-    PyObject *chunk = NULL;
 
     if (PyUnicode_READY(pystr) == -1)
         return 0;
 
+    _PyUnicodeWriter writer;
+    _PyUnicodeWriter_Init(&writer);
+    writer.overallocate = 1;
+
     len = PyUnicode_GET_LENGTH(pystr);
     buf = PyUnicode_DATA(pystr);
     kind = PyUnicode_KIND(pystr);
@@ -449,18 +423,26 @@ scanstring_unicode(PyObject *pystr, Py_ssize_t end, int strict, Py_ssize_t *next
             }
             c = d;
         }
-        if (!(c == '"' || c == '\\')) {
+
+        if (c == '"') {
+            // Fast path for simple case.
+            if (writer.buffer == NULL) {
+                PyObject *ret = PyUnicode_Substring(pystr, end, next);
+                if (ret == NULL) {
+                    goto bail;
+                }
+                *next_end_ptr = next + 1;;
+                return ret;
+            }
+        }
+        else if (c != '\\') {
             raise_errmsg("Unterminated string starting at", pystr, begin);
             goto bail;
         }
+
         /* Pick up this chunk if it's not zero length */
         if (next != end) {
-            APPEND_OLD_CHUNK
-                chunk = PyUnicode_FromKindAndData(
-                    kind,
-                    (char*)buf + kind * end,
-                    next - end);
-            if (chunk == NULL) {
+            if (_PyUnicodeWriter_WriteSubstring(&writer, pystr, end, next) < 0) {
                 goto bail;
             }
         }
@@ -551,34 +533,18 @@ scanstring_unicode(PyObject *pystr, Py_ssize_t end, int strict, Py_ssize_t *next
                     end -= 6;
             }
         }
-        APPEND_OLD_CHUNK
-        chunk = PyUnicode_FromKindAndData(PyUnicode_4BYTE_KIND, &c, 1);
-        if (chunk == NULL) {
+        if (_PyUnicodeWriter_WriteChar(&writer, c) < 0) {
             goto bail;
         }
     }
 
-    if (chunks == NULL) {
-        if (chunk != NULL)
-            rval = chunk;
-        else
-            rval = PyUnicode_FromStringAndSize("", 0);
-    }
-    else {
-        APPEND_OLD_CHUNK
-        rval = join_list_unicode(chunks);
-        if (rval == NULL) {
-            goto bail;
-        }
-        Py_CLEAR(chunks);
-    }
-
+    rval = _PyUnicodeWriter_Finish(&writer);
     *next_end_ptr = end;
     return rval;
+
 bail:
     *next_end_ptr = -1;
-    Py_XDECREF(chunks);
-    Py_XDECREF(chunk);
+    _PyUnicodeWriter_Dealloc(&writer);
     return NULL;
 }
 


From webhook-mailer at python.org  Thu Oct 17 06:41:41 2019
From: webhook-mailer at python.org (Xiang Zhang)
Date: Thu, 17 Oct 2019 10:41:41 -0000
Subject: [Python-checkins] bpo-34953: Implement `mmap.mmap.__repr__`
 (GH-9891)
Message-ID: <mailman.51.1571308902.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/d8ca2354ed30c12b9ce37c4535222b700a727b32
commit: d8ca2354ed30c12b9ce37c4535222b700a727b32
branch: master
author: Taine Zhao <twshere at outlook.com>
committer: Xiang Zhang <angwerzx at 126.com>
date: 2019-10-17T18:41:35+08:00
summary:

bpo-34953: Implement `mmap.mmap.__repr__` (GH-9891)

files:
M Lib/test/test_mmap.py
M Modules/mmapmodule.c

diff --git a/Lib/test/test_mmap.py b/Lib/test/test_mmap.py
index 88c501d81a07f..5400f25f50800 100644
--- a/Lib/test/test_mmap.py
+++ b/Lib/test/test_mmap.py
@@ -740,6 +740,42 @@ def test_flush_return_value(self):
             # See bpo-34754 for details.
             self.assertRaises(OSError, mm.flush, 1, len(b'python'))
 
+    def test_repr(self):
+        open_mmap_repr_pat = re.compile(
+            r"<mmap.mmap closed=False, "
+            r"access=(?P<access>\S+), "
+            r"length=(?P<length>\d+), "
+            r"pos=(?P<pos>\d+), "
+            r"offset=(?P<offset>\d+)>")
+        closed_mmap_repr_pat = re.compile(r"<mmap.mmap closed=True>")
+        mapsizes = (50, 100, 1_000, 1_000_000, 10_000_000)
+        offsets = tuple((mapsize // 2 // mmap.ALLOCATIONGRANULARITY)
+                        * mmap.ALLOCATIONGRANULARITY for mapsize in mapsizes)
+        for offset, mapsize in zip(offsets, mapsizes):
+            data = b'a' * mapsize
+            length = mapsize - offset
+            accesses = ('ACCESS_DEFAULT', 'ACCESS_READ',
+                        'ACCESS_COPY', 'ACCESS_WRITE')
+            positions = (0, length//10, length//5, length//4)
+            with open(TESTFN, "wb+") as fp:
+                fp.write(data)
+                fp.flush()
+                for access, pos in itertools.product(accesses, positions):
+                    accint = getattr(mmap, access)
+                    with mmap.mmap(fp.fileno(),
+                                   length,
+                                   access=accint,
+                                   offset=offset) as mm:
+                        mm.seek(pos)
+                        match = open_mmap_repr_pat.match(repr(mm))
+                        self.assertIsNotNone(match)
+                        self.assertEqual(match.group('access'), access)
+                        self.assertEqual(match.group('length'), str(length))
+                        self.assertEqual(match.group('pos'), str(pos))
+                        self.assertEqual(match.group('offset'), str(offset))
+                    match = closed_mmap_repr_pat.match(repr(mm))
+                    self.assertIsNotNone(match)
+
     @unittest.skipUnless(hasattr(mmap.mmap, 'madvise'), 'needs madvise')
     def test_madvise(self):
         size = 2 * PAGESIZE
diff --git a/Modules/mmapmodule.c b/Modules/mmapmodule.c
index 0c641636a1a83..a3b9e4df95305 100644
--- a/Modules/mmapmodule.c
+++ b/Modules/mmapmodule.c
@@ -695,6 +695,51 @@ mmap__exit__method(PyObject *self, PyObject *args)
     return _PyObject_CallMethodIdNoArgs(self, &PyId_close);
 }
 
+static PyObject *
+mmap__repr__method(PyObject *self)
+{
+    mmap_object *mobj = (mmap_object *)self;
+
+#ifdef MS_WINDOWS
+#define _Py_FORMAT_OFFSET "lld"
+    if (mobj->map_handle == NULL)
+#elif defined(UNIX)
+# ifdef HAVE_LARGEFILE_SUPPORT
+# define _Py_FORMAT_OFFSET "lld"
+# else
+# define _Py_FORMAT_OFFSET "ld"
+# endif
+    if (mobj->data == NULL)
+#endif
+    {
+        return PyUnicode_FromFormat("<%s closed=True>", Py_TYPE(self)->tp_name);
+    } else {
+        const char *access_str;
+
+        switch (mobj->access) {
+            case ACCESS_DEFAULT:
+                access_str = "ACCESS_DEFAULT";
+                break;
+            case ACCESS_READ:
+                access_str = "ACCESS_READ";
+                break;
+            case ACCESS_WRITE:
+                access_str = "ACCESS_WRITE";
+                break;
+            case ACCESS_COPY:
+                access_str = "ACCESS_COPY";
+                break;
+            default:
+                Py_UNREACHABLE();
+        }
+
+        return PyUnicode_FromFormat("<%s closed=False, access=%s, length=%zd, "
+                                    "pos=%zd, offset=%" _Py_FORMAT_OFFSET ">",
+                                    Py_TYPE(self)->tp_name, access_str,
+                                    mobj->size, mobj->pos, mobj->offset);
+    }
+}
+
 #ifdef MS_WINDOWS
 static PyObject *
 mmap__sizeof__method(mmap_object *self, void *unused)
@@ -1044,23 +1089,23 @@ static PyTypeObject mmap_object_type = {
     sizeof(mmap_object),                        /* tp_basicsize */
     0,                                          /* tp_itemsize */
     /* methods */
-    (destructor) mmap_object_dealloc,           /* tp_dealloc */
+    (destructor)mmap_object_dealloc,            /* tp_dealloc */
     0,                                          /* tp_vectorcall_offset */
     0,                                          /* tp_getattr */
     0,                                          /* tp_setattr */
     0,                                          /* tp_as_async */
-    0,                                          /* tp_repr */
+    (reprfunc)mmap__repr__method,               /* tp_repr */
     0,                                          /* tp_as_number */
-    &mmap_as_sequence,                          /*tp_as_sequence*/
-    &mmap_as_mapping,                           /*tp_as_mapping*/
-    0,                                          /*tp_hash*/
-    0,                                          /*tp_call*/
-    0,                                          /*tp_str*/
-    PyObject_GenericGetAttr,                    /*tp_getattro*/
-    0,                                          /*tp_setattro*/
-    &mmap_as_buffer,                            /*tp_as_buffer*/
-    Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE,   /*tp_flags*/
-    mmap_doc,                                   /*tp_doc*/
+    &mmap_as_sequence,                          /* tp_as_sequence */
+    &mmap_as_mapping,                           /* tp_as_mapping */
+    0,                                          /* tp_hash */
+    0,                                          /* tp_call */
+    0,                                          /* tp_str */
+    PyObject_GenericGetAttr,                    /* tp_getattro */
+    0,                                          /* tp_setattro */
+    &mmap_as_buffer,                            /* tp_as_buffer */
+    Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE,   /* tp_flags */
+    mmap_doc,                                   /* tp_doc */
     0,                                          /* tp_traverse */
     0,                                          /* tp_clear */
     0,                                          /* tp_richcompare */


From webhook-mailer at python.org  Thu Oct 17 23:30:50 2019
From: webhook-mailer at python.org (Gregory P. Smith)
Date: Fri, 18 Oct 2019 03:30:50 -0000
Subject: [Python-checkins] bpo-33604: Raise TypeError on missing hmac arg.
 (GH-16805)
Message-ID: <mailman.52.1571369452.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/f33c57d5c780da1500619f548585792bb5b750ee
commit: f33c57d5c780da1500619f548585792bb5b750ee
branch: master
author: Gregory P. Smith <greg at krypto.org>
committer: GitHub <noreply at github.com>
date: 2019-10-17T20:30:42-07:00
summary:

bpo-33604: Raise TypeError on missing hmac arg. (GH-16805)

Also updates the documentation to clarify the situation surrounding
the digestmod parameter that is required despite its position in the
argument list as of 3.8.0 as well as removing old python2 era
references to "binary strings".

We indavertently had this raise ValueError in 3.8.0 for the missing
arg.  This is not considered an API change as no reasonable code would
be catching this missing argument error in order to handle it.

files:
A Misc/NEWS.d/next/Library/2019-10-15-09-47-40.bpo-33604.J12cWT.rst
M Doc/library/hmac.rst
M Lib/hmac.py
M Lib/test/test_hmac.py

diff --git a/Doc/library/hmac.rst b/Doc/library/hmac.rst
index dc994b07c35c0..57ac8bb16120f 100644
--- a/Doc/library/hmac.rst
+++ b/Doc/library/hmac.rst
@@ -14,12 +14,13 @@
 This module implements the HMAC algorithm as described by :rfc:`2104`.
 
 
-.. function:: new(key, msg=None, digestmod=None)
+.. function:: new(key, msg=None, digestmod='')
 
    Return a new hmac object.  *key* is a bytes or bytearray object giving the
    secret key.  If *msg* is present, the method call ``update(msg)`` is made.
    *digestmod* is the digest name, digest constructor or module for the HMAC
-   object to use. It supports any name suitable to :func:`hashlib.new`.
+   object to use.  It may be any name suitable to :func:`hashlib.new`.
+   Despite its argument position, it is required.
 
    .. versionchanged:: 3.4
       Parameter *key* can be a bytes or bytearray object.
@@ -28,6 +29,8 @@ This module implements the HMAC algorithm as described by :rfc:`2104`.
 
    .. deprecated-removed:: 3.4 3.8
       MD5 as implicit default digest for *digestmod* is deprecated.
+      The digestmod parameter is now required.  Pass it as a keyword
+      argument to avoid awkwardness when you do not have an initial msg.
 
 
 .. function:: digest(key, msg, digest)
@@ -127,7 +130,6 @@ This module also provides the following helper function:
       a timing attack could theoretically reveal information about the
       types and lengths of *a* and *b*?but not their values.
 
-
    .. versionadded:: 3.3
 
 
diff --git a/Lib/hmac.py b/Lib/hmac.py
index 890eaba08e8f6..b769876e6f774 100644
--- a/Lib/hmac.py
+++ b/Lib/hmac.py
@@ -1,4 +1,4 @@
-"""HMAC (Keyed-Hashing for Message Authentication) Python module.
+"""HMAC (Keyed-Hashing for Message Authentication) module.
 
 Implements the HMAC algorithm as described by RFC 2104.
 """
@@ -30,23 +30,25 @@ class HMAC:
     """
     blocksize = 64  # 512-bit HMAC; can be changed in subclasses.
 
-    def __init__(self, key, msg = None, digestmod = None):
+    def __init__(self, key, msg=None, digestmod=''):
         """Create a new HMAC object.
 
-        key:       key for the keyed hash object.
-        msg:       Initial input for the hash, if provided.
-        digestmod: Required.  A module supporting PEP 247.  *OR*
-                   A hashlib constructor returning a new hash object.  *OR*
-                   A hash name suitable for hashlib.new().
+        key: bytes or buffer, key for the keyed hash object.
+        msg: bytes or buffer, Initial input for the hash or None.
+        digestmod: A hash name suitable for hashlib.new(). *OR*
+                   A hashlib constructor returning a new hash object. *OR*
+                   A module supporting PEP 247.
 
-        Note: key and msg must be a bytes or bytearray objects.
+                   Required as of 3.8, despite its position after the optional
+                   msg argument.  Passing it as a keyword argument is
+                   recommended, though not required for legacy API reasons.
         """
 
         if not isinstance(key, (bytes, bytearray)):
             raise TypeError("key: expected bytes or bytearray, but got %r" % type(key).__name__)
 
-        if digestmod is None:
-            raise ValueError('`digestmod` is required.')
+        if not digestmod:
+            raise TypeError("Missing required parameter 'digestmod'.")
 
         if callable(digestmod):
             self.digest_cons = digestmod
@@ -90,8 +92,7 @@ def name(self):
         return "hmac-" + self.inner.name
 
     def update(self, msg):
-        """Update this hashing object with the string msg.
-        """
+        """Feed data from msg into this hashing object."""
         self.inner.update(msg)
 
     def copy(self):
@@ -119,7 +120,7 @@ def _current(self):
     def digest(self):
         """Return the hash value of this hashing object.
 
-        This returns a string containing 8-bit data.  The object is
+        This returns the hmac value as bytes.  The object is
         not altered in any way by this function; you can continue
         updating the object after calling this function.
         """
@@ -132,30 +133,34 @@ def hexdigest(self):
         h = self._current()
         return h.hexdigest()
 
-def new(key, msg = None, digestmod = None):
+def new(key, msg=None, digestmod=''):
     """Create a new hashing object and return it.
 
-    key: The starting key for the hash.
-    msg: if available, will immediately be hashed into the object's starting
-    state.
+    key: bytes or buffer, The starting key for the hash.
+    msg: bytes or buffer, Initial input for the hash, or None.
+    digestmod: A hash name suitable for hashlib.new(). *OR*
+               A hashlib constructor returning a new hash object. *OR*
+               A module supporting PEP 247.
+
+               Required as of 3.8, despite its position after the optional
+               msg argument.  Passing it as a keyword argument is
+               recommended, though not required for legacy API reasons.
 
-    You can now feed arbitrary strings into the object using its update()
+    You can now feed arbitrary bytes into the object using its update()
     method, and can ask for the hash value at any time by calling its digest()
-    method.
+    or hexdigest() methods.
     """
     return HMAC(key, msg, digestmod)
 
 
 def digest(key, msg, digest):
-    """Fast inline implementation of HMAC
+    """Fast inline implementation of HMAC.
 
-    key:    key for the keyed hash object.
-    msg:    input message
+    key: bytes or buffer, The key for the keyed hash object.
+    msg: bytes or buffer, Input message.
     digest: A hash name suitable for hashlib.new() for best performance. *OR*
             A hashlib constructor returning a new hash object. *OR*
             A module supporting PEP 247.
-
-    Note: key and msg must be a bytes or bytearray objects.
     """
     if (_hashopenssl is not None and
             isinstance(digest, str) and digest in _openssl_md_meths):
diff --git a/Lib/test/test_hmac.py b/Lib/test/test_hmac.py
index 1bbf201727d7a..ea00367c8027a 100644
--- a/Lib/test/test_hmac.py
+++ b/Lib/test/test_hmac.py
@@ -312,10 +312,15 @@ def digest(self):
                 self.fail('Expected warning about small block_size')
 
     def test_with_digestmod_no_default(self):
-        with self.assertRaises(ValueError):
+        """The digestmod parameter is required as of Python 3.8."""
+        with self.assertRaisesRegex(TypeError, r'required.*digestmod'):
             key = b"\x0b" * 16
             data = b"Hi There"
             hmac.HMAC(key, data, digestmod=None)
+        with self.assertRaisesRegex(TypeError, r'required.*digestmod'):
+            hmac.new(key, data)
+        with self.assertRaisesRegex(TypeError, r'required.*digestmod'):
+            hmac.HMAC(key, msg=data, digestmod='')
 
 
 class ConstructorTestCase(unittest.TestCase):
diff --git a/Misc/NEWS.d/next/Library/2019-10-15-09-47-40.bpo-33604.J12cWT.rst b/Misc/NEWS.d/next/Library/2019-10-15-09-47-40.bpo-33604.J12cWT.rst
new file mode 100644
index 0000000000000..fbd73003cfcb6
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-10-15-09-47-40.bpo-33604.J12cWT.rst
@@ -0,0 +1,3 @@
+Fixed `hmac.new` and `hmac.HMAC` to raise TypeError instead of ValueError
+when the digestmod parameter, now required in 3.8, is omitted.  Also
+clarified the hmac module documentation and docstrings.


From webhook-mailer at python.org  Thu Oct 17 23:48:47 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Fri, 18 Oct 2019 03:48:47 -0000
Subject: [Python-checkins] bpo-33604: Raise TypeError on missing hmac arg.
 (GH-16805)
Message-ID: <mailman.53.1571370527.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/c615db608dbb36a5d10188f7d265dcec7bfcc3cf
commit: c615db608dbb36a5d10188f7d265dcec7bfcc3cf
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-17T20:48:42-07:00
summary:

bpo-33604: Raise TypeError on missing hmac arg. (GH-16805)


Also updates the documentation to clarify the situation surrounding
the digestmod parameter that is required despite its position in the
argument list as of 3.8.0 as well as removing old python2 era
references to "binary strings".

We indavertently had this raise ValueError in 3.8.0 for the missing
arg.  This is not considered an API change as no reasonable code would
be catching this missing argument error in order to handle it.
(cherry picked from commit f33c57d5c780da1500619f548585792bb5b750ee)

Co-authored-by: Gregory P. Smith <greg at krypto.org>

files:
A Misc/NEWS.d/next/Library/2019-10-15-09-47-40.bpo-33604.J12cWT.rst
M Doc/library/hmac.rst
M Lib/hmac.py
M Lib/test/test_hmac.py

diff --git a/Doc/library/hmac.rst b/Doc/library/hmac.rst
index dc994b07c35c0..57ac8bb16120f 100644
--- a/Doc/library/hmac.rst
+++ b/Doc/library/hmac.rst
@@ -14,12 +14,13 @@
 This module implements the HMAC algorithm as described by :rfc:`2104`.
 
 
-.. function:: new(key, msg=None, digestmod=None)
+.. function:: new(key, msg=None, digestmod='')
 
    Return a new hmac object.  *key* is a bytes or bytearray object giving the
    secret key.  If *msg* is present, the method call ``update(msg)`` is made.
    *digestmod* is the digest name, digest constructor or module for the HMAC
-   object to use. It supports any name suitable to :func:`hashlib.new`.
+   object to use.  It may be any name suitable to :func:`hashlib.new`.
+   Despite its argument position, it is required.
 
    .. versionchanged:: 3.4
       Parameter *key* can be a bytes or bytearray object.
@@ -28,6 +29,8 @@ This module implements the HMAC algorithm as described by :rfc:`2104`.
 
    .. deprecated-removed:: 3.4 3.8
       MD5 as implicit default digest for *digestmod* is deprecated.
+      The digestmod parameter is now required.  Pass it as a keyword
+      argument to avoid awkwardness when you do not have an initial msg.
 
 
 .. function:: digest(key, msg, digest)
@@ -127,7 +130,6 @@ This module also provides the following helper function:
       a timing attack could theoretically reveal information about the
       types and lengths of *a* and *b*?but not their values.
 
-
    .. versionadded:: 3.3
 
 
diff --git a/Lib/hmac.py b/Lib/hmac.py
index 890eaba08e8f6..b769876e6f774 100644
--- a/Lib/hmac.py
+++ b/Lib/hmac.py
@@ -1,4 +1,4 @@
-"""HMAC (Keyed-Hashing for Message Authentication) Python module.
+"""HMAC (Keyed-Hashing for Message Authentication) module.
 
 Implements the HMAC algorithm as described by RFC 2104.
 """
@@ -30,23 +30,25 @@ class HMAC:
     """
     blocksize = 64  # 512-bit HMAC; can be changed in subclasses.
 
-    def __init__(self, key, msg = None, digestmod = None):
+    def __init__(self, key, msg=None, digestmod=''):
         """Create a new HMAC object.
 
-        key:       key for the keyed hash object.
-        msg:       Initial input for the hash, if provided.
-        digestmod: Required.  A module supporting PEP 247.  *OR*
-                   A hashlib constructor returning a new hash object.  *OR*
-                   A hash name suitable for hashlib.new().
+        key: bytes or buffer, key for the keyed hash object.
+        msg: bytes or buffer, Initial input for the hash or None.
+        digestmod: A hash name suitable for hashlib.new(). *OR*
+                   A hashlib constructor returning a new hash object. *OR*
+                   A module supporting PEP 247.
 
-        Note: key and msg must be a bytes or bytearray objects.
+                   Required as of 3.8, despite its position after the optional
+                   msg argument.  Passing it as a keyword argument is
+                   recommended, though not required for legacy API reasons.
         """
 
         if not isinstance(key, (bytes, bytearray)):
             raise TypeError("key: expected bytes or bytearray, but got %r" % type(key).__name__)
 
-        if digestmod is None:
-            raise ValueError('`digestmod` is required.')
+        if not digestmod:
+            raise TypeError("Missing required parameter 'digestmod'.")
 
         if callable(digestmod):
             self.digest_cons = digestmod
@@ -90,8 +92,7 @@ def name(self):
         return "hmac-" + self.inner.name
 
     def update(self, msg):
-        """Update this hashing object with the string msg.
-        """
+        """Feed data from msg into this hashing object."""
         self.inner.update(msg)
 
     def copy(self):
@@ -119,7 +120,7 @@ def _current(self):
     def digest(self):
         """Return the hash value of this hashing object.
 
-        This returns a string containing 8-bit data.  The object is
+        This returns the hmac value as bytes.  The object is
         not altered in any way by this function; you can continue
         updating the object after calling this function.
         """
@@ -132,30 +133,34 @@ def hexdigest(self):
         h = self._current()
         return h.hexdigest()
 
-def new(key, msg = None, digestmod = None):
+def new(key, msg=None, digestmod=''):
     """Create a new hashing object and return it.
 
-    key: The starting key for the hash.
-    msg: if available, will immediately be hashed into the object's starting
-    state.
+    key: bytes or buffer, The starting key for the hash.
+    msg: bytes or buffer, Initial input for the hash, or None.
+    digestmod: A hash name suitable for hashlib.new(). *OR*
+               A hashlib constructor returning a new hash object. *OR*
+               A module supporting PEP 247.
+
+               Required as of 3.8, despite its position after the optional
+               msg argument.  Passing it as a keyword argument is
+               recommended, though not required for legacy API reasons.
 
-    You can now feed arbitrary strings into the object using its update()
+    You can now feed arbitrary bytes into the object using its update()
     method, and can ask for the hash value at any time by calling its digest()
-    method.
+    or hexdigest() methods.
     """
     return HMAC(key, msg, digestmod)
 
 
 def digest(key, msg, digest):
-    """Fast inline implementation of HMAC
+    """Fast inline implementation of HMAC.
 
-    key:    key for the keyed hash object.
-    msg:    input message
+    key: bytes or buffer, The key for the keyed hash object.
+    msg: bytes or buffer, Input message.
     digest: A hash name suitable for hashlib.new() for best performance. *OR*
             A hashlib constructor returning a new hash object. *OR*
             A module supporting PEP 247.
-
-    Note: key and msg must be a bytes or bytearray objects.
     """
     if (_hashopenssl is not None and
             isinstance(digest, str) and digest in _openssl_md_meths):
diff --git a/Lib/test/test_hmac.py b/Lib/test/test_hmac.py
index 1bbf201727d7a..ea00367c8027a 100644
--- a/Lib/test/test_hmac.py
+++ b/Lib/test/test_hmac.py
@@ -312,10 +312,15 @@ def digest(self):
                 self.fail('Expected warning about small block_size')
 
     def test_with_digestmod_no_default(self):
-        with self.assertRaises(ValueError):
+        """The digestmod parameter is required as of Python 3.8."""
+        with self.assertRaisesRegex(TypeError, r'required.*digestmod'):
             key = b"\x0b" * 16
             data = b"Hi There"
             hmac.HMAC(key, data, digestmod=None)
+        with self.assertRaisesRegex(TypeError, r'required.*digestmod'):
+            hmac.new(key, data)
+        with self.assertRaisesRegex(TypeError, r'required.*digestmod'):
+            hmac.HMAC(key, msg=data, digestmod='')
 
 
 class ConstructorTestCase(unittest.TestCase):
diff --git a/Misc/NEWS.d/next/Library/2019-10-15-09-47-40.bpo-33604.J12cWT.rst b/Misc/NEWS.d/next/Library/2019-10-15-09-47-40.bpo-33604.J12cWT.rst
new file mode 100644
index 0000000000000..fbd73003cfcb6
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-10-15-09-47-40.bpo-33604.J12cWT.rst
@@ -0,0 +1,3 @@
+Fixed `hmac.new` and `hmac.HMAC` to raise TypeError instead of ValueError
+when the digestmod parameter, now required in 3.8, is omitted.  Also
+clarified the hmac module documentation and docstrings.


From webhook-mailer at python.org  Fri Oct 18 03:21:11 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Fri, 18 Oct 2019 07:21:11 -0000
Subject: [Python-checkins] Doc: Add missing entry for
 functools.cached_property (GH-16803)
Message-ID: <mailman.54.1571383272.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/380d443a6ac73cb4719d86fd5781544630bcec5e
commit: 380d443a6ac73cb4719d86fd5781544630bcec5e
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-18T00:21:05-07:00
summary:

Doc: Add missing entry for functools.cached_property (GH-16803)

(cherry picked from commit 93b81e1fcbeb61c1b49ac2fa52c5a0dff929940b)

Co-authored-by: St?phane Wirtel <stephane at wirtel.be>

files:
M Doc/whatsnew/3.8.rst

diff --git a/Doc/whatsnew/3.8.rst b/Doc/whatsnew/3.8.rst
index e1388e2169f47..ef4209f9db499 100644
--- a/Doc/whatsnew/3.8.rst
+++ b/Doc/whatsnew/3.8.rst
@@ -685,6 +685,22 @@ than as a function returning a decorator.  So both of these are now supported::
 
 (Contributed by Raymond Hettinger in :issue:`36772`.)
 
+Added a new :func:`functools.cached_property` decorator, for computed properties
+cached for the life of the instance. ::
+
+   import functools
+   import statistics
+
+   class Dataset:
+      def __init__(self, sequence_of_numbers):
+         self.data = sequence_of_numbers
+
+      @functools.cached_property
+      def variance(self):
+         return statistics.variance(self.data)
+
+(Contributed by Carl Meyer in :issue:`21145`)
+
 
 gc
 --


From webhook-mailer at python.org  Fri Oct 18 03:48:54 2019
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Fri, 18 Oct 2019 07:48:54 -0000
Subject: [Python-checkins] Fix minor typos in What's New in Python 3.8.
 (GH-16770)
Message-ID: <mailman.55.1571384936.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/b1fa72a5b7e8666a40b6899d5d54778a2a42c3e8
commit: b1fa72a5b7e8666a40b6899d5d54778a2a42c3e8
branch: master
author: Jero Bado <tokidokitalkyou at gmail.com>
committer: Serhiy Storchaka <storchaka at gmail.com>
date: 2019-10-18T10:48:48+03:00
summary:

Fix minor typos in What's New in Python 3.8. (GH-16770)

Added periods at the end of the sentences.

files:
M Doc/whatsnew/3.8.rst

diff --git a/Doc/whatsnew/3.8.rst b/Doc/whatsnew/3.8.rst
index fef712d2ab296..ed450051170a6 100644
--- a/Doc/whatsnew/3.8.rst
+++ b/Doc/whatsnew/3.8.rst
@@ -805,7 +805,7 @@ numbers::
     >>> math.prod(likelihoods, start=prior)
     0.126
 
-(Contributed by Pablo Galindo in :issue:`35606`)
+(Contributed by Pablo Galindo in :issue:`35606`.)
 
 Added new function :func:`math.isqrt` for computing integer square roots.
 (Contributed by Mark Dickinson in :issue:`36887`.)
@@ -911,12 +911,12 @@ pickle
 Reduction methods can now include a 6th item in the tuple they return. This
 item should specify a custom state-setting method that's called instead of the
 regular ``__setstate__`` method.
-(Contributed by Pierre Glaser and Olivier Grisel in :issue:`35900`)
+(Contributed by Pierre Glaser and Olivier Grisel in :issue:`35900`.)
 
 :mod:`pickle` extensions subclassing the C-optimized :class:`~pickle.Pickler`
 can now override the pickling logic of functions and classes by defining the
 special :meth:`~pickle.Pickler.reducer_override` method.
-(Contributed by Pierre Glaser and Olivier Grisel in :issue:`35900`)
+(Contributed by Pierre Glaser and Olivier Grisel in :issue:`35900`.)
 
 
 plistlib
@@ -1276,7 +1276,7 @@ Optimizations
 
 * Removed one ``Py_ssize_t`` member from ``PyGC_Head``.  All GC tracked
   objects (e.g. tuple, list, dict) size is reduced 4 or 8 bytes.
-  (Contributed by Inada Naoki in :issue:`33597`)
+  (Contributed by Inada Naoki in :issue:`33597`.)
 
 * :class:`uuid.UUID` now uses ``__slots__`` to reduce its memory footprint.
   (Contributed by Wouter Bolsterlee and Tal Einat in :issue:`30977`)


From webhook-mailer at python.org  Fri Oct 18 03:52:19 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Fri, 18 Oct 2019 07:52:19 -0000
Subject: [Python-checkins] bpo-38418: Fixes audit event for os.system to be
 named 'os.system' (GH-16670)
Message-ID: <mailman.56.1571385140.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/fbe3c76c7ce1eec887d332d801d3784212cc0f73
commit: fbe3c76c7ce1eec887d332d801d3784212cc0f73
branch: master
author: Steve Dower <steve.dower at python.org>
committer: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
date: 2019-10-18T00:52:15-07:00
summary:

bpo-38418: Fixes audit event for os.system to be named 'os.system' (GH-16670)



https://bugs.python.org/issue38418



Automerge-Triggered-By: @zooba

files:
A Misc/NEWS.d/next/Security/2019-10-08-19-29-55.bpo-38418.QL7s0-.rst
M Modules/posixmodule.c

diff --git a/Misc/NEWS.d/next/Security/2019-10-08-19-29-55.bpo-38418.QL7s0-.rst b/Misc/NEWS.d/next/Security/2019-10-08-19-29-55.bpo-38418.QL7s0-.rst
new file mode 100644
index 0000000000000..b42a00047b96c
--- /dev/null
+++ b/Misc/NEWS.d/next/Security/2019-10-08-19-29-55.bpo-38418.QL7s0-.rst
@@ -0,0 +1 @@
+Fixes audit event for :func:`os.system` to be named ``os.system``.
diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
index 3c4e254abb56d..686497090f88b 100644
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -4324,7 +4324,7 @@ os_system_impl(PyObject *module, const Py_UNICODE *command)
 {
     long result;
 
-    if (PySys_Audit("system", "(u)", command) < 0) {
+    if (PySys_Audit("os.system", "(u)", command) < 0) {
         return -1;
     }
 
@@ -4351,7 +4351,7 @@ os_system_impl(PyObject *module, PyObject *command)
     long result;
     const char *bytes = PyBytes_AsString(command);
 
-    if (PySys_Audit("system", "(O)", command) < 0) {
+    if (PySys_Audit("os.system", "(O)", command) < 0) {
         return -1;
     }
 


From webhook-mailer at python.org  Fri Oct 18 03:55:26 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Fri, 18 Oct 2019 07:55:26 -0000
Subject: [Python-checkins] Fix minor typos in What's New in Python 3.8.
 (GH-16770)
Message-ID: <mailman.57.1571385327.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/de812682a674cdf2bac0f9547200f107069781ad
commit: de812682a674cdf2bac0f9547200f107069781ad
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-18T00:55:22-07:00
summary:

Fix minor typos in What's New in Python 3.8. (GH-16770)


Added periods at the end of the sentences.
(cherry picked from commit b1fa72a5b7e8666a40b6899d5d54778a2a42c3e8)

Co-authored-by: Jero Bado <tokidokitalkyou at gmail.com>

files:
M Doc/whatsnew/3.8.rst

diff --git a/Doc/whatsnew/3.8.rst b/Doc/whatsnew/3.8.rst
index ef4209f9db499..b1497880c6323 100644
--- a/Doc/whatsnew/3.8.rst
+++ b/Doc/whatsnew/3.8.rst
@@ -807,7 +807,7 @@ numbers::
     >>> math.prod(likelihoods, start=prior)
     0.126
 
-(Contributed by Pablo Galindo in :issue:`35606`)
+(Contributed by Pablo Galindo in :issue:`35606`.)
 
 Added new function :func:`math.isqrt` for computing integer square roots.
 (Contributed by Mark Dickinson in :issue:`36887`.)
@@ -913,12 +913,12 @@ pickle
 Reduction methods can now include a 6th item in the tuple they return. This
 item should specify a custom state-setting method that's called instead of the
 regular ``__setstate__`` method.
-(Contributed by Pierre Glaser and Olivier Grisel in :issue:`35900`)
+(Contributed by Pierre Glaser and Olivier Grisel in :issue:`35900`.)
 
 :mod:`pickle` extensions subclassing the C-optimized :class:`~pickle.Pickler`
 can now override the pickling logic of functions and classes by defining the
 special :meth:`~pickle.Pickler.reducer_override` method.
-(Contributed by Pierre Glaser and Olivier Grisel in :issue:`35900`)
+(Contributed by Pierre Glaser and Olivier Grisel in :issue:`35900`.)
 
 
 plistlib
@@ -1278,7 +1278,7 @@ Optimizations
 
 * Removed one ``Py_ssize_t`` member from ``PyGC_Head``.  All GC tracked
   objects (e.g. tuple, list, dict) size is reduced 4 or 8 bytes.
-  (Contributed by Inada Naoki in :issue:`33597`)
+  (Contributed by Inada Naoki in :issue:`33597`.)
 
 * :class:`uuid.UUID` now uses ``__slots__`` to reduce its memory footprint.
   (Contributed by Wouter Bolsterlee and Tal Einat in :issue:`30977`)


From webhook-mailer at python.org  Fri Oct 18 04:00:07 2019
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Fri, 18 Oct 2019 08:00:07 -0000
Subject: [Python-checkins] bpo-32758: Warn that compile() can crash when
 compiling to an AST object (GH-6043) (GH-16566)
Message-ID: <mailman.58.1571385608.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/8eb27cc35489848596d9fb4b1c91fac00ae75d21
commit: 8eb27cc35489848596d9fb4b1c91fac00ae75d21
branch: 2.7
author: Ashley Whetter <AWhetter at users.noreply.github.com>
committer: Serhiy Storchaka <storchaka at gmail.com>
date: 2019-10-18T11:00:03+03:00
summary:

bpo-32758: Warn that compile() can crash when compiling to an AST object (GH-6043) (GH-16566)

(cherry picked from commit f7a6ff6fcab32a53f262ba3f8a072c27afc330d7)

Co-authored-by: Brett Cannon <brettcannon at users.noreply.github.com>

files:
M Doc/library/functions.rst

diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst
index 8701f9d8ffb0e..23f34a3de0f05 100644
--- a/Doc/library/functions.rst
+++ b/Doc/library/functions.rst
@@ -248,6 +248,12 @@ section.
       character.  This is to facilitate detection of incomplete and complete
       statements in the :mod:`code` module.
 
+   .. warning::
+
+      It is possible to crash the Python interpreter with a
+      sufficiently large/complex string when compiling to an AST
+      object due to stack depth limitations in Python's AST compiler.
+
    .. versionchanged:: 2.3
       The *flags* and *dont_inherit* arguments were added.
 


From webhook-mailer at python.org  Fri Oct 18 04:00:27 2019
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Fri, 18 Oct 2019 08:00:27 -0000
Subject: [Python-checkins] bpo-32758: Warn that ast.parse() and
 ast.literal_eval() can segfault the interpreter (GH-5960) (GH-16565)
Message-ID: <mailman.59.1571385628.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/dedb99acdb5bbc179872235f975248133d3fb440
commit: dedb99acdb5bbc179872235f975248133d3fb440
branch: 2.7
author: Ashley Whetter <AWhetter at users.noreply.github.com>
committer: Serhiy Storchaka <storchaka at gmail.com>
date: 2019-10-18T11:00:22+03:00
summary:

bpo-32758: Warn that ast.parse() and ast.literal_eval() can segfault the interpreter (GH-5960) (GH-16565)

(cherry picked from commit 7a7f100eb352d08938ee0f5ba59c18f56dc4a7b5)

Co-authored-by: Brett Cannon <brettcannon at users.noreply.github.com>

files:
M Doc/library/ast.rst

diff --git a/Doc/library/ast.rst b/Doc/library/ast.rst
index 6d5855b52bcce..b04a92d33adc5 100644
--- a/Doc/library/ast.rst
+++ b/Doc/library/ast.rst
@@ -129,6 +129,11 @@ and classes for traversing abstract syntax trees:
    Parse the source into an AST node.  Equivalent to ``compile(source,
    filename, mode, ast.PyCF_ONLY_AST)``.
 
+   .. warning::
+      It is possible to crash the Python interpreter with a
+      sufficiently large/complex string due to stack depth limitations
+      in Python's AST compiler.
+
 
 .. function:: literal_eval(node_or_string)
 
@@ -142,6 +147,11 @@ and classes for traversing abstract syntax trees:
    capable of evaluating arbitrarily complex expressions, for example involving
    operators or indexing.
 
+   .. warning::
+      It is possible to crash the Python interpreter with a
+      sufficiently large/complex string due to stack depth limitations
+      in Python's AST compiler.
+
 
 .. function:: get_docstring(node, clean=True)
 


From webhook-mailer at python.org  Fri Oct 18 09:07:37 2019
From: webhook-mailer at python.org (Senthil Kumaran)
Date: Fri, 18 Oct 2019 13:07:37 -0000
Subject: [Python-checkins] bpo-27657: Fix urlparse() with numeric paths
 (#661)
Message-ID: <mailman.60.1571404058.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/5a88d50ff013a64fbdb25b877c87644a9034c969
commit: 5a88d50ff013a64fbdb25b877c87644a9034c969
branch: master
author: Tim Graham <timograham at gmail.com>
committer: Senthil Kumaran <skumaran at gatech.edu>
date: 2019-10-18T06:07:20-07:00
summary:

bpo-27657: Fix urlparse() with numeric paths (#661)

* bpo-27657: Fix urlparse() with numeric paths

Revert parsing decision from bpo-754016 in favor of the documented
consensus in bpo-16932 of how to treat strings without a // to
designate the netloc.

* bpo-22891: Remove urlsplit() optimization for 'http' prefixed inputs.

files:
A Misc/NEWS.d/next/Library/2017-12-26-14-32-23.bpo-27657.6BhyVK.rst
M Lib/test/test_urlparse.py
M Lib/urllib/parse.py

diff --git a/Lib/test/test_urlparse.py b/Lib/test/test_urlparse.py
index 4ae6ed33858ce..762500789f73a 100644
--- a/Lib/test/test_urlparse.py
+++ b/Lib/test/test_urlparse.py
@@ -709,15 +709,17 @@ def test_withoutscheme(self):
 
     def test_portseparator(self):
         # Issue 754016 makes changes for port separator ':' from scheme separator
-        self.assertEqual(urllib.parse.urlparse("path:80"),
-                ('','','path:80','','',''))
+        self.assertEqual(urllib.parse.urlparse("http:80"), ('http','','80','','',''))
+        self.assertEqual(urllib.parse.urlparse("https:80"), ('https','','80','','',''))
+        self.assertEqual(urllib.parse.urlparse("path:80"), ('path','','80','','',''))
         self.assertEqual(urllib.parse.urlparse("http:"),('http','','','','',''))
         self.assertEqual(urllib.parse.urlparse("https:"),('https','','','','',''))
         self.assertEqual(urllib.parse.urlparse("http://www.python.org:80"),
                 ('http','www.python.org:80','','','',''))
         # As usual, need to check bytes input as well
-        self.assertEqual(urllib.parse.urlparse(b"path:80"),
-                (b'',b'',b'path:80',b'',b'',b''))
+        self.assertEqual(urllib.parse.urlparse(b"http:80"), (b'http',b'',b'80',b'',b'',b''))
+        self.assertEqual(urllib.parse.urlparse(b"https:80"), (b'https',b'',b'80',b'',b'',b''))
+        self.assertEqual(urllib.parse.urlparse(b"path:80"), (b'path',b'',b'80',b'',b'',b''))
         self.assertEqual(urllib.parse.urlparse(b"http:"),(b'http',b'',b'',b'',b'',b''))
         self.assertEqual(urllib.parse.urlparse(b"https:"),(b'https',b'',b'',b'',b'',b''))
         self.assertEqual(urllib.parse.urlparse(b"http://www.python.org:80"),
diff --git a/Lib/urllib/parse.py b/Lib/urllib/parse.py
index 3a38dc14c9047..31fd7e16ee72c 100644
--- a/Lib/urllib/parse.py
+++ b/Lib/urllib/parse.py
@@ -431,31 +431,11 @@ def urlsplit(url, scheme='', allow_fragments=True):
     netloc = query = fragment = ''
     i = url.find(':')
     if i > 0:
-        if url[:i] == 'http': # optimize the common case
-            url = url[i+1:]
-            if url[:2] == '//':
-                netloc, url = _splitnetloc(url, 2)
-                if (('[' in netloc and ']' not in netloc) or
-                        (']' in netloc and '[' not in netloc)):
-                    raise ValueError("Invalid IPv6 URL")
-            if allow_fragments and '#' in url:
-                url, fragment = url.split('#', 1)
-            if '?' in url:
-                url, query = url.split('?', 1)
-            _checknetloc(netloc)
-            v = SplitResult('http', netloc, url, query, fragment)
-            _parse_cache[key] = v
-            return _coerce_result(v)
         for c in url[:i]:
             if c not in scheme_chars:
                 break
         else:
-            # make sure "url" is not actually a port number (in which case
-            # "scheme" is really part of the path)
-            rest = url[i+1:]
-            if not rest or any(c not in '0123456789' for c in rest):
-                # not a port number
-                scheme, url = url[:i].lower(), rest
+            scheme, url = url[:i].lower(), url[i+1:]
 
     if url[:2] == '//':
         netloc, url = _splitnetloc(url, 2)
diff --git a/Misc/NEWS.d/next/Library/2017-12-26-14-32-23.bpo-27657.6BhyVK.rst b/Misc/NEWS.d/next/Library/2017-12-26-14-32-23.bpo-27657.6BhyVK.rst
new file mode 100644
index 0000000000000..77746c0ce630f
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2017-12-26-14-32-23.bpo-27657.6BhyVK.rst
@@ -0,0 +1,2 @@
+Fix urllib.parse.urlparse() with numeric paths. A string like "path:80" is
+no longer parsed as a path but as a scheme ("path") and a path ("80").


From webhook-mailer at python.org  Fri Oct 18 09:24:32 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Fri, 18 Oct 2019 13:24:32 -0000
Subject: [Python-checkins] bpo-27657: Fix urlparse() with numeric paths
 (GH-661)
Message-ID: <mailman.61.1571405074.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/82b5f6b16e051f8a2ac6e87ba86b082fa1c4a77f
commit: 82b5f6b16e051f8a2ac6e87ba86b082fa1c4a77f
branch: 3.7
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-18T06:24:28-07:00
summary:

bpo-27657: Fix urlparse() with numeric paths (GH-661)


* bpo-27657: Fix urlparse() with numeric paths

Revert parsing decision from bpo-754016 in favor of the documented
consensus in bpo-16932 of how to treat strings without a // to
designate the netloc.

* bpo-22891: Remove urlsplit() optimization for 'http' prefixed inputs.
(cherry picked from commit 5a88d50ff013a64fbdb25b877c87644a9034c969)

Co-authored-by: Tim Graham <timograham at gmail.com>

files:
A Misc/NEWS.d/next/Library/2017-12-26-14-32-23.bpo-27657.6BhyVK.rst
M Lib/test/test_urlparse.py
M Lib/urllib/parse.py

diff --git a/Lib/test/test_urlparse.py b/Lib/test/test_urlparse.py
index 68f633ca3a7db..999272d3e3ad6 100644
--- a/Lib/test/test_urlparse.py
+++ b/Lib/test/test_urlparse.py
@@ -709,15 +709,17 @@ def test_withoutscheme(self):
 
     def test_portseparator(self):
         # Issue 754016 makes changes for port separator ':' from scheme separator
-        self.assertEqual(urllib.parse.urlparse("path:80"),
-                ('','','path:80','','',''))
+        self.assertEqual(urllib.parse.urlparse("http:80"), ('http','','80','','',''))
+        self.assertEqual(urllib.parse.urlparse("https:80"), ('https','','80','','',''))
+        self.assertEqual(urllib.parse.urlparse("path:80"), ('path','','80','','',''))
         self.assertEqual(urllib.parse.urlparse("http:"),('http','','','','',''))
         self.assertEqual(urllib.parse.urlparse("https:"),('https','','','','',''))
         self.assertEqual(urllib.parse.urlparse("http://www.python.org:80"),
                 ('http','www.python.org:80','','','',''))
         # As usual, need to check bytes input as well
-        self.assertEqual(urllib.parse.urlparse(b"path:80"),
-                (b'',b'',b'path:80',b'',b'',b''))
+        self.assertEqual(urllib.parse.urlparse(b"http:80"), (b'http',b'',b'80',b'',b'',b''))
+        self.assertEqual(urllib.parse.urlparse(b"https:80"), (b'https',b'',b'80',b'',b'',b''))
+        self.assertEqual(urllib.parse.urlparse(b"path:80"), (b'path',b'',b'80',b'',b'',b''))
         self.assertEqual(urllib.parse.urlparse(b"http:"),(b'http',b'',b'',b'',b'',b''))
         self.assertEqual(urllib.parse.urlparse(b"https:"),(b'https',b'',b'',b'',b'',b''))
         self.assertEqual(urllib.parse.urlparse(b"http://www.python.org:80"),
diff --git a/Lib/urllib/parse.py b/Lib/urllib/parse.py
index 4c8e77fe39123..1d4dc74b040ed 100644
--- a/Lib/urllib/parse.py
+++ b/Lib/urllib/parse.py
@@ -426,31 +426,11 @@ def urlsplit(url, scheme='', allow_fragments=True):
     netloc = query = fragment = ''
     i = url.find(':')
     if i > 0:
-        if url[:i] == 'http': # optimize the common case
-            url = url[i+1:]
-            if url[:2] == '//':
-                netloc, url = _splitnetloc(url, 2)
-                if (('[' in netloc and ']' not in netloc) or
-                        (']' in netloc and '[' not in netloc)):
-                    raise ValueError("Invalid IPv6 URL")
-            if allow_fragments and '#' in url:
-                url, fragment = url.split('#', 1)
-            if '?' in url:
-                url, query = url.split('?', 1)
-            _checknetloc(netloc)
-            v = SplitResult('http', netloc, url, query, fragment)
-            _parse_cache[key] = v
-            return _coerce_result(v)
         for c in url[:i]:
             if c not in scheme_chars:
                 break
         else:
-            # make sure "url" is not actually a port number (in which case
-            # "scheme" is really part of the path)
-            rest = url[i+1:]
-            if not rest or any(c not in '0123456789' for c in rest):
-                # not a port number
-                scheme, url = url[:i].lower(), rest
+            scheme, url = url[:i].lower(), url[i+1:]
 
     if url[:2] == '//':
         netloc, url = _splitnetloc(url, 2)
diff --git a/Misc/NEWS.d/next/Library/2017-12-26-14-32-23.bpo-27657.6BhyVK.rst b/Misc/NEWS.d/next/Library/2017-12-26-14-32-23.bpo-27657.6BhyVK.rst
new file mode 100644
index 0000000000000..77746c0ce630f
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2017-12-26-14-32-23.bpo-27657.6BhyVK.rst
@@ -0,0 +1,2 @@
+Fix urllib.parse.urlparse() with numeric paths. A string like "path:80" is
+no longer parsed as a path but as a scheme ("path") and a path ("80").


From webhook-mailer at python.org  Fri Oct 18 09:49:12 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Fri, 18 Oct 2019 13:49:12 -0000
Subject: [Python-checkins] bpo-38502: regrtest uses process groups if
 available (GH-16829)
Message-ID: <mailman.62.1571406553.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/ecb035cd14c11521276343397151929a94018a22
commit: ecb035cd14c11521276343397151929a94018a22
branch: master
author: Victor Stinner <vstinner at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-18T15:49:08+02:00
summary:

bpo-38502: regrtest uses process groups if available (GH-16829)

test.regrtest now uses process groups in the multiprocessing mode
(-jN command line option) if process groups are available: if
os.setsid() and os.killpg() functions are available.

files:
A Misc/NEWS.d/next/Tests/2019-10-17-00-49-38.bpo-38502.vUEic7.rst
M Lib/test/libregrtest/runtest_mp.py

diff --git a/Lib/test/libregrtest/runtest_mp.py b/Lib/test/libregrtest/runtest_mp.py
index 8026de187d587..fc12ea7c5fc14 100644
--- a/Lib/test/libregrtest/runtest_mp.py
+++ b/Lib/test/libregrtest/runtest_mp.py
@@ -3,6 +3,7 @@
 import json
 import os
 import queue
+import signal
 import subprocess
 import sys
 import threading
@@ -31,6 +32,8 @@
 # Time to wait until a worker completes: should be immediate
 JOIN_TIMEOUT = 30.0   # seconds
 
+USE_PROCESS_GROUP = (hasattr(os, "setsid") and hasattr(os, "killpg"))
+
 
 def must_stop(result, ns):
     if result.result == INTERRUPTED:
@@ -59,12 +62,16 @@ def run_test_in_subprocess(testname, ns):
     # Running the child from the same working directory as regrtest's original
     # invocation ensures that TEMPDIR for the child is the same when
     # sysconfig.is_python_build() is true. See issue 15300.
+    kw = {}
+    if USE_PROCESS_GROUP:
+        kw['start_new_session'] = True
     return subprocess.Popen(cmd,
                             stdout=subprocess.PIPE,
                             stderr=subprocess.PIPE,
                             universal_newlines=True,
                             close_fds=(os.name != 'nt'),
-                            cwd=support.SAVEDCWD)
+                            cwd=support.SAVEDCWD,
+                            **kw)
 
 
 def run_tests_worker(ns, test_name):
@@ -149,16 +156,24 @@ def _kill(self):
             return
         self._killed = True
 
-        print(f"Kill {self}", file=sys.stderr, flush=True)
+        if USE_PROCESS_GROUP:
+            what = f"{self} process group"
+        else:
+            what = f"{self}"
+
+        print(f"Kill {what}", file=sys.stderr, flush=True)
         try:
-            popen.kill()
+            if USE_PROCESS_GROUP:
+                os.killpg(popen.pid, signal.SIGKILL)
+            else:
+                popen.kill()
         except ProcessLookupError:
-            # Process completed, the TestWorkerProcess thread read its exit
-            # status, but Popen.send_signal() read the returncode just before
-            # Popen.wait() set returncode.
+            # popen.kill(): the process completed, the TestWorkerProcess thread
+            # read its exit status, but Popen.send_signal() read the returncode
+            # just before Popen.wait() set returncode.
             pass
         except OSError as exc:
-            print_warning(f"Failed to kill {self}: {exc!r}")
+            print_warning(f"Failed to kill {what}: {exc!r}")
 
     def stop(self):
         # Method called from a different thread to stop this thread
diff --git a/Misc/NEWS.d/next/Tests/2019-10-17-00-49-38.bpo-38502.vUEic7.rst b/Misc/NEWS.d/next/Tests/2019-10-17-00-49-38.bpo-38502.vUEic7.rst
new file mode 100644
index 0000000000000..1df523e90c388
--- /dev/null
+++ b/Misc/NEWS.d/next/Tests/2019-10-17-00-49-38.bpo-38502.vUEic7.rst
@@ -0,0 +1,3 @@
+test.regrtest now uses process groups in the multiprocessing mode (-jN command
+line option) if process groups are available: if :func:`os.setsid` and
+:func:`os.killpg` functions are available.


From webhook-mailer at python.org  Fri Oct 18 11:23:21 2019
From: webhook-mailer at python.org (Senthil Kumaran)
Date: Fri, 18 Oct 2019 15:23:21 -0000
Subject: [Python-checkins] [3.8] bpo-27657: Fix urlparse() with numeric
 paths (GH-661) (#16839)
Message-ID: <mailman.63.1571412202.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/0f3187c1ce3b3ace60f6c1691dfa3d4e744f0384
commit: 0f3187c1ce3b3ace60f6c1691dfa3d4e744f0384
branch: 3.8
author: Senthil Kumaran <skumaran at gatech.edu>
committer: GitHub <noreply at github.com>
date: 2019-10-18T08:23:14-07:00
summary:

[3.8] bpo-27657: Fix urlparse() with numeric paths (GH-661) (#16839)

* bpo-27657: Fix urlparse() with numeric paths

Revert parsing decision from bpo-754016 in favor of the documented
consensus in bpo-16932 of how to treat strings without a // to
designate the netloc.

* bpo-22891: Remove urlsplit() optimization for 'http' prefixed inputs.
(cherry picked from commit 5a88d50ff013a64fbdb25b877c87644a9034c969)

Co-authored-by: Tim Graham <timograham at gmail.com>

files:
A Misc/NEWS.d/next/Library/2017-12-26-14-32-23.bpo-27657.6BhyVK.rst
M Lib/test/test_urlparse.py
M Lib/urllib/parse.py

diff --git a/Lib/test/test_urlparse.py b/Lib/test/test_urlparse.py
index 4ae6ed33858ce..762500789f73a 100644
--- a/Lib/test/test_urlparse.py
+++ b/Lib/test/test_urlparse.py
@@ -709,15 +709,17 @@ def test_withoutscheme(self):
 
     def test_portseparator(self):
         # Issue 754016 makes changes for port separator ':' from scheme separator
-        self.assertEqual(urllib.parse.urlparse("path:80"),
-                ('','','path:80','','',''))
+        self.assertEqual(urllib.parse.urlparse("http:80"), ('http','','80','','',''))
+        self.assertEqual(urllib.parse.urlparse("https:80"), ('https','','80','','',''))
+        self.assertEqual(urllib.parse.urlparse("path:80"), ('path','','80','','',''))
         self.assertEqual(urllib.parse.urlparse("http:"),('http','','','','',''))
         self.assertEqual(urllib.parse.urlparse("https:"),('https','','','','',''))
         self.assertEqual(urllib.parse.urlparse("http://www.python.org:80"),
                 ('http','www.python.org:80','','','',''))
         # As usual, need to check bytes input as well
-        self.assertEqual(urllib.parse.urlparse(b"path:80"),
-                (b'',b'',b'path:80',b'',b'',b''))
+        self.assertEqual(urllib.parse.urlparse(b"http:80"), (b'http',b'',b'80',b'',b'',b''))
+        self.assertEqual(urllib.parse.urlparse(b"https:80"), (b'https',b'',b'80',b'',b'',b''))
+        self.assertEqual(urllib.parse.urlparse(b"path:80"), (b'path',b'',b'80',b'',b'',b''))
         self.assertEqual(urllib.parse.urlparse(b"http:"),(b'http',b'',b'',b'',b'',b''))
         self.assertEqual(urllib.parse.urlparse(b"https:"),(b'https',b'',b'',b'',b'',b''))
         self.assertEqual(urllib.parse.urlparse(b"http://www.python.org:80"),
diff --git a/Lib/urllib/parse.py b/Lib/urllib/parse.py
index b6608783a8947..d497925b943f4 100644
--- a/Lib/urllib/parse.py
+++ b/Lib/urllib/parse.py
@@ -431,31 +431,11 @@ def urlsplit(url, scheme='', allow_fragments=True):
     netloc = query = fragment = ''
     i = url.find(':')
     if i > 0:
-        if url[:i] == 'http': # optimize the common case
-            url = url[i+1:]
-            if url[:2] == '//':
-                netloc, url = _splitnetloc(url, 2)
-                if (('[' in netloc and ']' not in netloc) or
-                        (']' in netloc and '[' not in netloc)):
-                    raise ValueError("Invalid IPv6 URL")
-            if allow_fragments and '#' in url:
-                url, fragment = url.split('#', 1)
-            if '?' in url:
-                url, query = url.split('?', 1)
-            _checknetloc(netloc)
-            v = SplitResult('http', netloc, url, query, fragment)
-            _parse_cache[key] = v
-            return _coerce_result(v)
         for c in url[:i]:
             if c not in scheme_chars:
                 break
         else:
-            # make sure "url" is not actually a port number (in which case
-            # "scheme" is really part of the path)
-            rest = url[i+1:]
-            if not rest or any(c not in '0123456789' for c in rest):
-                # not a port number
-                scheme, url = url[:i].lower(), rest
+            scheme, url = url[:i].lower(), url[i+1:]
 
     if url[:2] == '//':
         netloc, url = _splitnetloc(url, 2)
diff --git a/Misc/NEWS.d/next/Library/2017-12-26-14-32-23.bpo-27657.6BhyVK.rst b/Misc/NEWS.d/next/Library/2017-12-26-14-32-23.bpo-27657.6BhyVK.rst
new file mode 100644
index 0000000000000..77746c0ce630f
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2017-12-26-14-32-23.bpo-27657.6BhyVK.rst
@@ -0,0 +1,2 @@
+Fix urllib.parse.urlparse() with numeric paths. A string like "path:80" is
+no longer parsed as a path but as a scheme ("path") and a path ("80").


From webhook-mailer at python.org  Fri Oct 18 12:32:39 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Fri, 18 Oct 2019 16:32:39 -0000
Subject: [Python-checkins] bpo-38418: Fixes audit event for os.system to be
 named 'os.system' (GH-16670)
Message-ID: <mailman.64.1571416360.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/5fb81420d5ad64a900940e908b546cf40f2d6807
commit: 5fb81420d5ad64a900940e908b546cf40f2d6807
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-18T09:32:14-07:00
summary:

bpo-38418: Fixes audit event for os.system to be named 'os.system' (GH-16670)


https://bugs.python.org/issue38418

Automerge-Triggered-By: @zooba
(cherry picked from commit fbe3c76c7ce1eec887d332d801d3784212cc0f73)

Co-authored-by: Steve Dower <steve.dower at python.org>

files:
A Misc/NEWS.d/next/Security/2019-10-08-19-29-55.bpo-38418.QL7s0-.rst
M Modules/posixmodule.c

diff --git a/Misc/NEWS.d/next/Security/2019-10-08-19-29-55.bpo-38418.QL7s0-.rst b/Misc/NEWS.d/next/Security/2019-10-08-19-29-55.bpo-38418.QL7s0-.rst
new file mode 100644
index 0000000000000..b42a00047b96c
--- /dev/null
+++ b/Misc/NEWS.d/next/Security/2019-10-08-19-29-55.bpo-38418.QL7s0-.rst
@@ -0,0 +1 @@
+Fixes audit event for :func:`os.system` to be named ``os.system``.
diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
index b09204d63392d..7c823434e67fe 100644
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -4316,7 +4316,7 @@ os_system_impl(PyObject *module, const Py_UNICODE *command)
 {
     long result;
 
-    if (PySys_Audit("system", "(u)", command) < 0) {
+    if (PySys_Audit("os.system", "(u)", command) < 0) {
         return -1;
     }
 
@@ -4343,7 +4343,7 @@ os_system_impl(PyObject *module, PyObject *command)
     long result;
     const char *bytes = PyBytes_AsString(command);
 
-    if (PySys_Audit("system", "(O)", command) < 0) {
+    if (PySys_Audit("os.system", "(O)", command) < 0) {
         return -1;
     }
 


From webhook-mailer at python.org  Fri Oct 18 17:20:41 2019
From: webhook-mailer at python.org (Raymond Hettinger)
Date: Fri, 18 Oct 2019 21:20:41 -0000
Subject: [Python-checkins] bpo-38521: Fix error in NormalDist.__eq__()
 (GH-16840)
Message-ID: <mailman.65.1571433642.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/5eabec022b9a10734fcf58faad02c4d233592f64
commit: 5eabec022b9a10734fcf58faad02c4d233592f64
branch: master
author: Raymond Hettinger <rhettinger at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-18T14:20:35-07:00
summary:

bpo-38521: Fix error in NormalDist.__eq__() (GH-16840)

files:
A Misc/NEWS.d/next/Library/2019-10-18-13-57-31.bpo-38521.U-7aaM.rst
M Lib/statistics.py
M Lib/test/test_statistics.py

diff --git a/Lib/statistics.py b/Lib/statistics.py
index 0d747b3d6c053..461ffae3f4914 100644
--- a/Lib/statistics.py
+++ b/Lib/statistics.py
@@ -1092,7 +1092,7 @@ def __eq__(x1, x2):
         "Two NormalDist objects are equal if their mu and sigma are both equal."
         if not isinstance(x2, NormalDist):
             return NotImplemented
-        return (x1._mu, x2._sigma) == (x2._mu, x2._sigma)
+        return x1._mu == x2._mu and x1._sigma == x2._sigma
 
     def __hash__(self):
         "NormalDist objects hash equal if their mu and sigma are both equal."
diff --git a/Lib/test/test_statistics.py b/Lib/test/test_statistics.py
index af26473e8fdfc..bebd9b5d6f502 100644
--- a/Lib/test/test_statistics.py
+++ b/Lib/test/test_statistics.py
@@ -2651,9 +2651,13 @@ def test_equality(self):
         nd2 = NormalDist(2, 4)
         nd3 = NormalDist()
         nd4 = NormalDist(2, 4)
+        nd5 = NormalDist(2, 8)
+        nd6 = NormalDist(8, 4)
         self.assertNotEqual(nd1, nd2)
         self.assertEqual(nd1, nd3)
         self.assertEqual(nd2, nd4)
+        self.assertNotEqual(nd2, nd5)
+        self.assertNotEqual(nd2, nd6)
 
         # Test NotImplemented when types are different
         class A:
diff --git a/Misc/NEWS.d/next/Library/2019-10-18-13-57-31.bpo-38521.U-7aaM.rst b/Misc/NEWS.d/next/Library/2019-10-18-13-57-31.bpo-38521.U-7aaM.rst
new file mode 100644
index 0000000000000..9335bdc9545c4
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-10-18-13-57-31.bpo-38521.U-7aaM.rst
@@ -0,0 +1 @@
+Fixed erroneous equality comparison in statistics.NormalDist().


From webhook-mailer at python.org  Fri Oct 18 17:41:23 2019
From: webhook-mailer at python.org (Raymond Hettinger)
Date: Fri, 18 Oct 2019 21:41:23 -0000
Subject: [Python-checkins] bpo-38521: Fix error in NormalDist.__eq__()
 (GH-16840) (GH-16842)
Message-ID: <mailman.66.1571434884.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/652a1cb0e11eb7cf0040acaf121492d2a99768d9
commit: 652a1cb0e11eb7cf0040acaf121492d2a99768d9
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Raymond Hettinger <rhettinger at users.noreply.github.com>
date: 2019-10-18T14:41:19-07:00
summary:

bpo-38521: Fix error in NormalDist.__eq__() (GH-16840) (GH-16842)

(cherry picked from commit 5eabec022b9a10734fcf58faad02c4d233592f64)

Co-authored-by: Raymond Hettinger <rhettinger at users.noreply.github.com>

files:
A Misc/NEWS.d/next/Library/2019-10-18-13-57-31.bpo-38521.U-7aaM.rst
M Lib/statistics.py
M Lib/test/test_statistics.py

diff --git a/Lib/statistics.py b/Lib/statistics.py
index 0d747b3d6c053..461ffae3f4914 100644
--- a/Lib/statistics.py
+++ b/Lib/statistics.py
@@ -1092,7 +1092,7 @@ def __eq__(x1, x2):
         "Two NormalDist objects are equal if their mu and sigma are both equal."
         if not isinstance(x2, NormalDist):
             return NotImplemented
-        return (x1._mu, x2._sigma) == (x2._mu, x2._sigma)
+        return x1._mu == x2._mu and x1._sigma == x2._sigma
 
     def __hash__(self):
         "NormalDist objects hash equal if their mu and sigma are both equal."
diff --git a/Lib/test/test_statistics.py b/Lib/test/test_statistics.py
index af26473e8fdfc..bebd9b5d6f502 100644
--- a/Lib/test/test_statistics.py
+++ b/Lib/test/test_statistics.py
@@ -2651,9 +2651,13 @@ def test_equality(self):
         nd2 = NormalDist(2, 4)
         nd3 = NormalDist()
         nd4 = NormalDist(2, 4)
+        nd5 = NormalDist(2, 8)
+        nd6 = NormalDist(8, 4)
         self.assertNotEqual(nd1, nd2)
         self.assertEqual(nd1, nd3)
         self.assertEqual(nd2, nd4)
+        self.assertNotEqual(nd2, nd5)
+        self.assertNotEqual(nd2, nd6)
 
         # Test NotImplemented when types are different
         class A:
diff --git a/Misc/NEWS.d/next/Library/2019-10-18-13-57-31.bpo-38521.U-7aaM.rst b/Misc/NEWS.d/next/Library/2019-10-18-13-57-31.bpo-38521.U-7aaM.rst
new file mode 100644
index 0000000000000..9335bdc9545c4
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-10-18-13-57-31.bpo-38521.U-7aaM.rst
@@ -0,0 +1 @@
+Fixed erroneous equality comparison in statistics.NormalDist().


From webhook-mailer at python.org  Fri Oct 18 21:59:22 2019
From: webhook-mailer at python.org (Tim Peters)
Date: Sat, 19 Oct 2019 01:59:22 -0000
Subject: [Python-checkins] validate_list: make flags argument impossible to
 spell wrongly. (GH-16843)
Message-ID: <mailman.67.1571450364.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/ea55c51bd937f6019c35b39b87029644e469c059
commit: ea55c51bd937f6019c35b39b87029644e469c059
branch: master
author: Tim Peters <tim.peters at gmail.com>
committer: GitHub <noreply at github.com>
date: 2019-10-18T20:59:14-05:00
summary:

validate_list: make flags argument impossible to spell wrongly. (GH-16843)

files:
M Modules/gcmodule.c

diff --git a/Modules/gcmodule.c b/Modules/gcmodule.c
index aca3f2260b6e1..9750d2a6f3fb2 100644
--- a/Modules/gcmodule.c
+++ b/Modules/gcmodule.c
@@ -329,6 +329,12 @@ append_objects(PyObject *py_list, PyGC_Head *gc_list)
     return 0;
 }
 
+// Constants for validate_list's flags argument.
+enum flagstates {collecting_clear_unreachable_clear,
+                 collecting_clear_unreachable_set,
+                 collecting_set_unreachable_clear,
+                 collecting_set_unreachable_set};
+
 #ifdef GC_DEBUG
 // validate_list checks list consistency.  And it works as document
 // describing when flags are expected to be set / unset.
@@ -336,16 +342,31 @@ append_objects(PyObject *py_list, PyGC_Head *gc_list)
 // the prev and next pointers are "polluted" with flags.
 // What's checked:
 // - The `head` pointers are not polluted.
-// - The objects' prev pointers' PREV_MASK_COLLECTING flags are all
-//   `prev_value` (PREV_MASK_COLLECTING for set, 0 for clear).
-// - The objects' next pointers' NEXT_MASK_UNREACHABLE flags are all
-//   `next_value` (NEXT_MASK_UNREACHABLE for set, 0 for clear).
+// - The objects' PREV_MASK_COLLECTING and NEXT_MASK_UNREACHABLE flags are all
+//   `set or clear, as specified by the 'flags' argument.
 // - The prev and next pointers are mutually consistent.
 static void
-validate_list(PyGC_Head *head, uintptr_t prev_value, uintptr_t next_value)
+validate_list(PyGC_Head *head, enum flagstates flags)
 {
     assert((head->_gc_prev & PREV_MASK_COLLECTING) == 0);
     assert((head->_gc_next & NEXT_MASK_UNREACHABLE) == 0);
+    uintptr_t prev_value = 0, next_value = 0;
+    switch (flags) {
+        case collecting_clear_unreachable_clear:
+            break;
+        case collecting_set_unreachable_clear:
+            prev_value = PREV_MASK_COLLECTING;
+            break;
+        case collecting_clear_unreachable_set:
+            next_value = NEXT_MASK_UNREACHABLE;
+            break;
+        case collecting_set_unreachable_set:
+            prev_value = PREV_MASK_COLLECTING;
+            next_value = NEXT_MASK_UNREACHABLE;
+            break;
+        default:
+            assert(! "bad internal flags argument");
+    }
     PyGC_Head *prev = head;
     PyGC_Head *gc = GC_NEXT(head);
     while (gc != head) {
@@ -361,7 +382,7 @@ validate_list(PyGC_Head *head, uintptr_t prev_value, uintptr_t next_value)
     assert(prev == GC_PREV(head));
 }
 #else
-#define validate_list(x, y, z) do{}while(0)
+#define validate_list(x, y) do{}while(0)
 #endif
 
 /*** end of list stuff ***/
@@ -657,7 +678,7 @@ clear_unreachable_mask(PyGC_Head *unreachable)
         gc->_gc_next &= ~NEXT_MASK_UNREACHABLE;
         next = (PyGC_Head*)gc->_gc_next;
     }
-    validate_list(unreachable, PREV_MASK_COLLECTING, 0);
+    validate_list(unreachable, collecting_set_unreachable_clear);
 }
 
 /* A traversal callback for move_legacy_finalizer_reachable. */
@@ -1050,7 +1071,7 @@ by a call to 'move_legacy_finalizers'), the 'unreachable' list is not a normal
 list and we can not use most gc_list_* functions for it. */
 static inline void
 deduce_unreachable(PyGC_Head *base, PyGC_Head *unreachable) {
-    validate_list(base, 0, 0);
+    validate_list(base, collecting_clear_unreachable_clear);
     /* Using ob_refcnt and gc_refs, calculate which objects in the
      * container set are reachable from outside the set (i.e., have a
      * refcount greater than 0 when all the references within the
@@ -1067,8 +1088,8 @@ deduce_unreachable(PyGC_Head *base, PyGC_Head *unreachable) {
      */
     gc_list_init(unreachable);
     move_unreachable(base, unreachable);  // gc_prev is pointer again
-    validate_list(base, 0, 0);
-    validate_list(unreachable, PREV_MASK_COLLECTING, NEXT_MASK_UNREACHABLE);
+    validate_list(base, collecting_clear_unreachable_clear);
+    validate_list(unreachable, collecting_set_unreachable_set);
 }
 
 /* Handle objects that may have resurrected after a call to 'finalize_garbage', moving
@@ -1145,7 +1166,7 @@ collect(struct _gc_runtime_state *state, int generation,
         old = GEN_HEAD(state, generation+1);
     else
         old = young;
-    validate_list(old, 0, 0);
+    validate_list(old, collecting_clear_unreachable_clear);
 
     deduce_unreachable(young, &unreachable);
 
@@ -1178,8 +1199,8 @@ collect(struct _gc_runtime_state *state, int generation,
      */
     move_legacy_finalizer_reachable(&finalizers);
 
-    validate_list(&finalizers, 0, 0);
-    validate_list(&unreachable, PREV_MASK_COLLECTING, 0);
+    validate_list(&finalizers, collecting_clear_unreachable_clear);
+    validate_list(&unreachable, collecting_set_unreachable_clear);
 
     /* Print debugging information. */
     if (state->debug & DEBUG_COLLECTABLE) {
@@ -1191,8 +1212,8 @@ collect(struct _gc_runtime_state *state, int generation,
     /* Clear weakrefs and invoke callbacks as necessary. */
     m += handle_weakrefs(&unreachable, old);
 
-    validate_list(old, 0, 0);
-    validate_list(&unreachable, PREV_MASK_COLLECTING, 0);
+    validate_list(old, collecting_clear_unreachable_clear);
+    validate_list(&unreachable, collecting_set_unreachable_clear);
 
     /* Call tp_finalize on objects which have one. */
     finalize_garbage(&unreachable);
@@ -1230,7 +1251,7 @@ collect(struct _gc_runtime_state *state, int generation,
      * this if they insist on creating this type of structure.
      */
     handle_legacy_finalizers(state, &finalizers, old);
-    validate_list(old, 0, 0);
+    validate_list(old, collecting_clear_unreachable_clear);
 
     /* Clear free list only during the collection of the highest
      * generation */


From webhook-mailer at python.org  Fri Oct 18 22:00:08 2019
From: webhook-mailer at python.org (Eric Snow)
Date: Sat, 19 Oct 2019 02:00:08 -0000
Subject: [Python-checkins] bpo-36876: Re-organize the c-analyzer tool code.
 (gh-16841)
Message-ID: <mailman.68.1571450409.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/e4c431ecf50def40eb93c3969c1e4eeaf7bf32f1
commit: e4c431ecf50def40eb93c3969c1e4eeaf7bf32f1
branch: master
author: Eric Snow <ericsnowcurrently at gmail.com>
committer: GitHub <noreply at github.com>
date: 2019-10-18T19:00:04-07:00
summary:

bpo-36876: Re-organize the c-analyzer tool code. (gh-16841)

This is partly a cleanup of the code. It also is preparation for getting the variables from the source (cross-platform) rather than from the symbols.

The change only touches the tool (and its tests).

files:
A Lib/test/test_tools/test_c_analyzer/test_common/__init__.py
A Lib/test/test_tools/test_c_analyzer/test_common/test_files.py
A Lib/test/test_tools/test_c_analyzer/test_common/test_info.py
A Lib/test/test_tools/test_c_analyzer/test_common/test_show.py
A Lib/test/test_tools/test_c_analyzer/test_cpython/__init__.py
A Lib/test/test_tools/test_c_analyzer/test_cpython/test___main__.py
A Lib/test/test_tools/test_c_analyzer/test_cpython/test_functional.py
A Lib/test/test_tools/test_c_analyzer/test_cpython/test_supported.py
A Lib/test/test_tools/test_c_analyzer/test_parser/__init__.py
A Lib/test/test_tools/test_c_analyzer/test_parser/test_declarations.py
A Lib/test/test_tools/test_c_analyzer/test_parser/test_preprocessor.py
A Lib/test/test_tools/test_c_analyzer/test_symbols/__init__.py
A Lib/test/test_tools/test_c_analyzer/test_symbols/test_info.py
A Lib/test/test_tools/test_c_analyzer/test_variables/__init__.py
A Lib/test/test_tools/test_c_analyzer/test_variables/test_find.py
A Lib/test/test_tools/test_c_analyzer/test_variables/test_info.py
A Lib/test/test_tools/test_c_analyzer/test_variables/test_known.py
A Tools/c-analyzer/c_analyzer/__init__.py
A Tools/c-analyzer/c_analyzer/common/__init__.py
A Tools/c-analyzer/c_analyzer/common/files.py
A Tools/c-analyzer/c_analyzer/common/info.py
A Tools/c-analyzer/c_analyzer/common/show.py
A Tools/c-analyzer/c_analyzer/common/util.py
A Tools/c-analyzer/c_analyzer/parser/__init__.py
A Tools/c-analyzer/c_analyzer/parser/declarations.py
A Tools/c-analyzer/c_analyzer/parser/find.py
A Tools/c-analyzer/c_analyzer/parser/naive.py
A Tools/c-analyzer/c_analyzer/parser/preprocessor.py
A Tools/c-analyzer/c_analyzer/parser/source.py
A Tools/c-analyzer/c_analyzer/symbols/__init__.py
A Tools/c-analyzer/c_analyzer/symbols/_nm.py
A Tools/c-analyzer/c_analyzer/symbols/find.py
A Tools/c-analyzer/c_analyzer/symbols/info.py
A Tools/c-analyzer/c_analyzer/variables/__init__.py
A Tools/c-analyzer/c_analyzer/variables/find.py
A Tools/c-analyzer/c_analyzer/variables/info.py
A Tools/c-analyzer/c_analyzer/variables/known.py
A Tools/c-analyzer/cpython/README
A Tools/c-analyzer/cpython/__init__.py
A Tools/c-analyzer/cpython/__main__.py
A Tools/c-analyzer/cpython/_generate.py
A Tools/c-analyzer/cpython/files.py
A Tools/c-analyzer/cpython/find.py
A Tools/c-analyzer/cpython/known.py
A Tools/c-analyzer/cpython/supported.py
D Lib/test/test_tools/test_c_analyzer/test_c_analyzer_common/__init__.py
D Lib/test/test_tools/test_c_analyzer/test_c_analyzer_common/test_files.py
D Lib/test/test_tools/test_c_analyzer/test_c_analyzer_common/test_info.py
D Lib/test/test_tools/test_c_analyzer/test_c_analyzer_common/test_known.py
D Lib/test/test_tools/test_c_analyzer/test_c_globals/__init__.py
D Lib/test/test_tools/test_c_analyzer/test_c_globals/test___main__.py
D Lib/test/test_tools/test_c_analyzer/test_c_globals/test_find.py
D Lib/test/test_tools/test_c_analyzer/test_c_globals/test_functional.py
D Lib/test/test_tools/test_c_analyzer/test_c_globals/test_show.py
D Lib/test/test_tools/test_c_analyzer/test_c_globals/test_supported.py
D Lib/test/test_tools/test_c_analyzer/test_c_parser/__init__.py
D Lib/test/test_tools/test_c_analyzer/test_c_parser/test_declarations.py
D Lib/test/test_tools/test_c_analyzer/test_c_parser/test_info.py
D Lib/test/test_tools/test_c_analyzer/test_c_parser/test_preprocessor.py
D Lib/test/test_tools/test_c_analyzer/test_c_symbols/__init__.py
D Lib/test/test_tools/test_c_analyzer/test_c_symbols/test_info.py
D Tools/c-analyzer/c_analyzer_common/__init__.py
D Tools/c-analyzer/c_analyzer_common/_generate.py
D Tools/c-analyzer/c_analyzer_common/files.py
D Tools/c-analyzer/c_analyzer_common/info.py
D Tools/c-analyzer/c_analyzer_common/known.py
D Tools/c-analyzer/c_analyzer_common/util.py
D Tools/c-analyzer/c_globals/README
D Tools/c-analyzer/c_globals/__init__.py
D Tools/c-analyzer/c_globals/__main__.py
D Tools/c-analyzer/c_globals/find.py
D Tools/c-analyzer/c_globals/show.py
D Tools/c-analyzer/c_globals/supported.py
D Tools/c-analyzer/c_parser/__init__.py
D Tools/c-analyzer/c_parser/declarations.py
D Tools/c-analyzer/c_parser/info.py
D Tools/c-analyzer/c_parser/naive.py
D Tools/c-analyzer/c_parser/preprocessor.py
D Tools/c-analyzer/c_parser/source.py
D Tools/c-analyzer/c_symbols/__init__.py
D Tools/c-analyzer/c_symbols/binary.py
D Tools/c-analyzer/c_symbols/info.py
D Tools/c-analyzer/c_symbols/resolve.py
D Tools/c-analyzer/c_symbols/source.py
M Lib/test/test_check_c_globals.py
M Tools/c-analyzer/c-globals.py

diff --git a/Lib/test/test_check_c_globals.py b/Lib/test/test_check_c_globals.py
index a3925f0ca887e..030debc452e40 100644
--- a/Lib/test/test_check_c_globals.py
+++ b/Lib/test/test_check_c_globals.py
@@ -3,7 +3,7 @@
 
 test.test_tools.skip_if_missing('c-analyzer')
 with test.test_tools.imports_under_tool('c-analyzer'):
-    from c_globals.__main__ import main
+    from cpython.__main__ import main
 
 
 class ActualChecks(unittest.TestCase):
diff --git a/Lib/test/test_tools/test_c_analyzer/test_c_analyzer_common/test_known.py b/Lib/test/test_tools/test_c_analyzer/test_c_analyzer_common/test_known.py
deleted file mode 100644
index 215023da577b9..0000000000000
--- a/Lib/test/test_tools/test_c_analyzer/test_c_analyzer_common/test_known.py
+++ /dev/null
@@ -1,68 +0,0 @@
-import re
-import textwrap
-import unittest
-
-from .. import tool_imports_for_tests
-with tool_imports_for_tests():
-    from c_parser.info import Variable
-    from c_analyzer_common.info import ID
-    from c_analyzer_common.known import from_file
-
-
-class FromFileTests(unittest.TestCase):
-
-    maxDiff = None
-
-    _return_read_tsv = ()
-
-    @property
-    def calls(self):
-        try:
-            return self._calls
-        except AttributeError:
-            self._calls = []
-            return self._calls
-
-    def _read_tsv(self, *args):
-        self.calls.append(('_read_tsv', args))
-        return self._return_read_tsv
-
-    def test_typical(self):
-        lines = textwrap.dedent('''
-            filename    funcname        name    kind    declaration
-            file1.c     -       var1    variable        static int
-            file1.c     func1   local1  variable        static int
-            file1.c     -       var2    variable        int
-            file1.c     func2   local2  variable        char *
-            file2.c     -       var1    variable        char *
-            ''').strip().splitlines()
-        lines = [re.sub(r'\s+', '\t', line, 4) for line in lines]
-        self._return_read_tsv = [tuple(v.strip() for v in line.split('\t'))
-                                 for line in lines[1:]]
-
-        known = from_file('spam.c', _read_tsv=self._read_tsv)
-
-        self.assertEqual(known, {
-            'variables': {v.id: v for v in [
-                Variable.from_parts('file1.c', '', 'var1', 'static int'),
-                Variable.from_parts('file1.c', 'func1', 'local1', 'static int'),
-                Variable.from_parts('file1.c', '', 'var2', 'int'),
-                Variable.from_parts('file1.c', 'func2', 'local2', 'char *'),
-                Variable.from_parts('file2.c', '', 'var1', 'char *'),
-                ]},
-            })
-        self.assertEqual(self.calls, [
-            ('_read_tsv', ('spam.c', 'filename\tfuncname\tname\tkind\tdeclaration')),
-            ])
-
-    def test_empty(self):
-        self._return_read_tsv = []
-
-        known = from_file('spam.c', _read_tsv=self._read_tsv)
-
-        self.assertEqual(known, {
-            'variables': {},
-            })
-        self.assertEqual(self.calls, [
-            ('_read_tsv', ('spam.c', 'filename\tfuncname\tname\tkind\tdeclaration')),
-            ])
diff --git a/Lib/test/test_tools/test_c_analyzer/test_c_globals/test_find.py b/Lib/test/test_tools/test_c_analyzer/test_c_globals/test_find.py
deleted file mode 100644
index 828899201b7ca..0000000000000
--- a/Lib/test/test_tools/test_c_analyzer/test_c_globals/test_find.py
+++ /dev/null
@@ -1,335 +0,0 @@
-import unittest
-
-from .. import tool_imports_for_tests
-with tool_imports_for_tests():
-    from c_parser import info
-    from c_globals.find import globals_from_binary, globals
-
-
-class _Base(unittest.TestCase):
-
-    maxDiff = None
-
-    @property
-    def calls(self):
-        try:
-            return self._calls
-        except AttributeError:
-            self._calls = []
-            return self._calls
-
-
-class StaticsFromBinaryTests(_Base):
-
-    _return_iter_symbols = ()
-    _return_resolve_symbols = ()
-    _return_get_symbol_resolver = None
-
-    def setUp(self):
-        super().setUp()
-
-        self.kwargs = dict(
-                _iter_symbols=self._iter_symbols,
-                _resolve=self._resolve_symbols,
-                _get_symbol_resolver=self._get_symbol_resolver,
-                )
-
-    def _iter_symbols(self, binfile, find_local_symbol):
-        self.calls.append(('_iter_symbols', (binfile, find_local_symbol)))
-        return self._return_iter_symbols
-
-    def _resolve_symbols(self, symbols, resolve):
-        self.calls.append(('_resolve_symbols', (symbols, resolve,)))
-        return self._return_resolve_symbols
-
-    def _get_symbol_resolver(self, knownvars, dirnames=None):
-        self.calls.append(('_get_symbol_resolver', (knownvars, dirnames)))
-        return self._return_get_symbol_resolver
-
-    def test_typical(self):
-        symbols = self._return_iter_symbols = ()
-        resolver = self._return_get_symbol_resolver = object()
-        variables = self._return_resolve_symbols = [
-            info.Variable.from_parts('dir1/spam.c', None, 'var1', 'int'),
-            info.Variable.from_parts('dir1/spam.c', None, 'var2', 'static int'),
-            info.Variable.from_parts('dir1/spam.c', None, 'var3', 'char *'),
-            info.Variable.from_parts('dir1/spam.c', 'func2', 'var4', 'const char *'),
-            info.Variable.from_parts('dir1/eggs.c', None, 'var1', 'static int'),
-            info.Variable.from_parts('dir1/eggs.c', 'func1', 'var2', 'static char *'),
-            ]
-        knownvars = object()
-
-        found = list(globals_from_binary('python',
-                                         knownvars=knownvars,
-                                         **self.kwargs))
-
-        self.assertEqual(found, [
-            info.Variable.from_parts('dir1/spam.c', None, 'var1', 'int'),
-            info.Variable.from_parts('dir1/spam.c', None, 'var2', 'static int'),
-            info.Variable.from_parts('dir1/spam.c', None, 'var3', 'char *'),
-            info.Variable.from_parts('dir1/eggs.c', None, 'var1', 'static int'),
-            info.Variable.from_parts('dir1/eggs.c', 'func1', 'var2', 'static char *'),
-            ])
-        self.assertEqual(self.calls, [
-            ('_iter_symbols', ('python', None)),
-            ('_get_symbol_resolver', (knownvars, None)),
-            ('_resolve_symbols', (symbols, resolver)),
-            ])
-
-#        self._return_iter_symbols = [
-#                s_info.Symbol(('dir1/spam.c', None, 'var1'), 'variable', False),
-#                s_info.Symbol(('dir1/spam.c', None, 'var2'), 'variable', False),
-#                s_info.Symbol(('dir1/spam.c', None, 'func1'), 'function', False),
-#                s_info.Symbol(('dir1/spam.c', None, 'func2'), 'function', True),
-#                s_info.Symbol(('dir1/spam.c', None, 'var3'), 'variable', False),
-#                s_info.Symbol(('dir1/spam.c', 'func2', 'var4'), 'variable', False),
-#                s_info.Symbol(('dir1/ham.c', None, 'var1'), 'variable', True),
-#                s_info.Symbol(('dir1/eggs.c', None, 'var1'), 'variable', False),
-#                s_info.Symbol(('dir1/eggs.c', None, 'xyz'), 'other', False),
-#                s_info.Symbol(('dir1/eggs.c', '???', 'var2'), 'variable', False),
-#                s_info.Symbol(('???', None, 'var_x'), 'variable', False),
-#                s_info.Symbol(('???', '???', 'var_y'), 'variable', False),
-#                s_info.Symbol((None, None, '???'), 'other', False),
-#                ]
-#        known = object()
-#
-#        globals_from_binary('python', knownvars=known, **this.kwargs)
-#        found = list(globals_from_symbols(['dir1'], self.iter_symbols))
-#
-#        self.assertEqual(found, [
-#            info.Variable.from_parts('dir1/spam.c', None, 'var1', '???'),
-#            info.Variable.from_parts('dir1/spam.c', None, 'var2', '???'),
-#            info.Variable.from_parts('dir1/spam.c', None, 'var3', '???'),
-#            info.Variable.from_parts('dir1/spam.c', 'func2', 'var4', '???'),
-#            info.Variable.from_parts('dir1/eggs.c', None, 'var1', '???'),
-#            ])
-#        self.assertEqual(self.calls, [
-#            ('iter_symbols', (['dir1'],)),
-#            ])
-#
-#    def test_no_symbols(self):
-#        self._return_iter_symbols = []
-#
-#        found = list(globals_from_symbols(['dir1'], self.iter_symbols))
-#
-#        self.assertEqual(found, [])
-#        self.assertEqual(self.calls, [
-#            ('iter_symbols', (['dir1'],)),
-#            ])
-
-    # XXX need functional test
-
-
-#class StaticFromDeclarationsTests(_Base):
-#
-#    _return_iter_declarations = ()
-#
-#    def iter_declarations(self, dirnames):
-#        self.calls.append(('iter_declarations', (dirnames,)))
-#        return iter(self._return_iter_declarations)
-#
-#    def test_typical(self):
-#        self._return_iter_declarations = [
-#            None,
-#            info.Variable.from_parts('dir1/spam.c', None, 'var1', '???'),
-#            object(),
-#            info.Variable.from_parts('dir1/spam.c', None, 'var2', '???'),
-#            info.Variable.from_parts('dir1/spam.c', None, 'var3', '???'),
-#            object(),
-#            info.Variable.from_parts('dir1/spam.c', 'func2', 'var4', '???'),
-#            object(),
-#            info.Variable.from_parts('dir1/eggs.c', None, 'var1', '???'),
-#            object(),
-#            ]
-#
-#        found = list(globals_from_declarations(['dir1'], self.iter_declarations))
-#
-#        self.assertEqual(found, [
-#            info.Variable.from_parts('dir1/spam.c', None, 'var1', '???'),
-#            info.Variable.from_parts('dir1/spam.c', None, 'var2', '???'),
-#            info.Variable.from_parts('dir1/spam.c', None, 'var3', '???'),
-#            info.Variable.from_parts('dir1/spam.c', 'func2', 'var4', '???'),
-#            info.Variable.from_parts('dir1/eggs.c', None, 'var1', '???'),
-#            ])
-#        self.assertEqual(self.calls, [
-#            ('iter_declarations', (['dir1'],)),
-#            ])
-#
-#    def test_no_declarations(self):
-#        self._return_iter_declarations = []
-#
-#        found = list(globals_from_declarations(['dir1'], self.iter_declarations))
-#
-#        self.assertEqual(found, [])
-#        self.assertEqual(self.calls, [
-#            ('iter_declarations', (['dir1'],)),
-#            ])
-
-
-#class IterVariablesTests(_Base):
-#
-#    _return_from_symbols = ()
-#    _return_from_declarations = ()
-#
-#    def _from_symbols(self, dirnames, iter_symbols):
-#        self.calls.append(('_from_symbols', (dirnames, iter_symbols)))
-#        return iter(self._return_from_symbols)
-#
-#    def _from_declarations(self, dirnames, iter_declarations):
-#        self.calls.append(('_from_declarations', (dirnames, iter_declarations)))
-#        return iter(self._return_from_declarations)
-#
-#    def test_typical(self):
-#        expected = [
-#            info.Variable.from_parts('dir1/spam.c', None, 'var1', '???'),
-#            info.Variable.from_parts('dir1/spam.c', None, 'var2', '???'),
-#            info.Variable.from_parts('dir1/spam.c', None, 'var3', '???'),
-#            info.Variable.from_parts('dir1/spam.c', 'func2', 'var4', '???'),
-#            info.Variable.from_parts('dir1/eggs.c', None, 'var1', '???'),
-#            ]
-#        self._return_from_symbols = expected
-#
-#        found = list(iter_variables(['dir1'],
-#                                  _from_symbols=self._from_symbols,
-#                                  _from_declarations=self._from_declarations))
-#
-#        self.assertEqual(found, expected)
-#        self.assertEqual(self.calls, [
-#            ('_from_symbols', (['dir1'], b_symbols.iter_symbols)),
-#            ])
-#
-#    def test_no_symbols(self):
-#        self._return_from_symbols = []
-#
-#        found = list(iter_variables(['dir1'],
-#                                  _from_symbols=self._from_symbols,
-#                                  _from_declarations=self._from_declarations))
-#
-#        self.assertEqual(found, [])
-#        self.assertEqual(self.calls, [
-#            ('_from_symbols', (['dir1'], b_symbols.iter_symbols)),
-#            ])
-#
-#    def test_from_binary(self):
-#        expected = [
-#            info.Variable.from_parts('dir1/spam.c', None, 'var1', '???'),
-#            info.Variable.from_parts('dir1/spam.c', None, 'var2', '???'),
-#            info.Variable.from_parts('dir1/spam.c', None, 'var3', '???'),
-#            info.Variable.from_parts('dir1/spam.c', 'func2', 'var4', '???'),
-#            info.Variable.from_parts('dir1/eggs.c', None, 'var1', '???'),
-#            ]
-#        self._return_from_symbols = expected
-#
-#        found = list(iter_variables(['dir1'], 'platform',
-#                                  _from_symbols=self._from_symbols,
-#                                  _from_declarations=self._from_declarations))
-#
-#        self.assertEqual(found, expected)
-#        self.assertEqual(self.calls, [
-#            ('_from_symbols', (['dir1'], b_symbols.iter_symbols)),
-#            ])
-#
-#    def test_from_symbols(self):
-#        expected = [
-#            info.Variable.from_parts('dir1/spam.c', None, 'var1', '???'),
-#            info.Variable.from_parts('dir1/spam.c', None, 'var2', '???'),
-#            info.Variable.from_parts('dir1/spam.c', None, 'var3', '???'),
-#            info.Variable.from_parts('dir1/spam.c', 'func2', 'var4', '???'),
-#            info.Variable.from_parts('dir1/eggs.c', None, 'var1', '???'),
-#            ]
-#        self._return_from_symbols = expected
-#
-#        found = list(iter_variables(['dir1'], 'symbols',
-#                                  _from_symbols=self._from_symbols,
-#                                  _from_declarations=self._from_declarations))
-#
-#        self.assertEqual(found, expected)
-#        self.assertEqual(self.calls, [
-#            ('_from_symbols', (['dir1'], s_symbols.iter_symbols)),
-#            ])
-#
-#    def test_from_declarations(self):
-#        expected = [
-#            info.Variable.from_parts('dir1/spam.c', None, 'var1', '???'),
-#            info.Variable.from_parts('dir1/spam.c', None, 'var2', '???'),
-#            info.Variable.from_parts('dir1/spam.c', None, 'var3', '???'),
-#            info.Variable.from_parts('dir1/spam.c', 'func2', 'var4', '???'),
-#            info.Variable.from_parts('dir1/eggs.c', None, 'var1', '???'),
-#            ]
-#        self._return_from_declarations = expected
-#
-#        found = list(iter_variables(['dir1'], 'declarations',
-#                                  _from_symbols=self._from_symbols,
-#                                  _from_declarations=self._from_declarations))
-#
-#        self.assertEqual(found, expected)
-#        self.assertEqual(self.calls, [
-#            ('_from_declarations', (['dir1'], declarations.iter_all)),
-#            ])
-#
-#    def test_from_preprocessed(self):
-#        expected = [
-#            info.Variable.from_parts('dir1/spam.c', None, 'var1', '???'),
-#            info.Variable.from_parts('dir1/spam.c', None, 'var2', '???'),
-#            info.Variable.from_parts('dir1/spam.c', None, 'var3', '???'),
-#            info.Variable.from_parts('dir1/spam.c', 'func2', 'var4', '???'),
-#            info.Variable.from_parts('dir1/eggs.c', None, 'var1', '???'),
-#            ]
-#        self._return_from_declarations = expected
-#
-#        found = list(iter_variables(['dir1'], 'preprocessed',
-#                                  _from_symbols=self._from_symbols,
-#                                  _from_declarations=self._from_declarations))
-#
-#        self.assertEqual(found, expected)
-#        self.assertEqual(self.calls, [
-#            ('_from_declarations', (['dir1'], declarations.iter_preprocessed)),
-#            ])
-
-
-class StaticsTest(_Base):
-
-    _return_iter_variables = None
-
-    def _iter_variables(self, kind, *, known, dirnames):
-        self.calls.append(
-                ('_iter_variables', (kind, known, dirnames)))
-        return iter(self._return_iter_variables or ())
-
-    def test_typical(self):
-        self._return_iter_variables = [
-            info.Variable.from_parts('src1/spam.c', None, 'var1', 'static const char *'),
-            info.Variable.from_parts('src1/spam.c', None, 'var1b', 'const char *'),
-            info.Variable.from_parts('src1/spam.c', 'ham', 'initialized', 'static int'),
-            info.Variable.from_parts('src1/spam.c', 'ham', 'result', 'int'),  # skipped
-            info.Variable.from_parts('src1/spam.c', None, 'var2', 'static PyObject *'),
-            info.Variable.from_parts('src1/eggs.c', 'tofu', 'ready', 'static int'),
-            info.Variable.from_parts('src1/spam.c', None, 'freelist', 'static (PyTupleObject *)[10]'),
-            info.Variable.from_parts('src1/sub/ham.c', None, 'var1', 'static const char const *'),
-            info.Variable.from_parts('src2/jam.c', None, 'var1', 'static int'),
-            info.Variable.from_parts('src2/jam.c', None, 'var2', 'static MyObject *'),
-            info.Variable.from_parts('Include/spam.h', None, 'data', 'static const int'),
-            ]
-        dirnames = object()
-        known = object()
-
-        found = list(globals(dirnames, known,
-                             kind='platform',
-                             _iter_variables=self._iter_variables,
-                             ))
-
-        self.assertEqual(found, [
-            info.Variable.from_parts('src1/spam.c', None, 'var1', 'static const char *'),
-            info.Variable.from_parts('src1/spam.c', None, 'var1b', 'const char *'),
-            info.Variable.from_parts('src1/spam.c', 'ham', 'initialized', 'static int'),
-            info.Variable.from_parts('src1/spam.c', None, 'var2', 'static PyObject *'),
-            info.Variable.from_parts('src1/eggs.c', 'tofu', 'ready', 'static int'),
-            info.Variable.from_parts('src1/spam.c', None, 'freelist', 'static (PyTupleObject *)[10]'),
-            info.Variable.from_parts('src1/sub/ham.c', None, 'var1', 'static const char const *'),
-            info.Variable.from_parts('src2/jam.c', None, 'var1', 'static int'),
-            info.Variable.from_parts('src2/jam.c', None, 'var2', 'static MyObject *'),
-            info.Variable.from_parts('Include/spam.h', None, 'data', 'static const int'),
-            ])
-        self.assertEqual(self.calls, [
-            ('_iter_variables', ('platform', known, dirnames)),
-            ])
diff --git a/Lib/test/test_tools/test_c_analyzer/test_c_analyzer_common/__init__.py b/Lib/test/test_tools/test_c_analyzer/test_common/__init__.py
similarity index 100%
rename from Lib/test/test_tools/test_c_analyzer/test_c_analyzer_common/__init__.py
rename to Lib/test/test_tools/test_c_analyzer/test_common/__init__.py
diff --git a/Lib/test/test_tools/test_c_analyzer/test_c_analyzer_common/test_files.py b/Lib/test/test_tools/test_c_analyzer/test_common/test_files.py
similarity index 99%
rename from Lib/test/test_tools/test_c_analyzer/test_c_analyzer_common/test_files.py
rename to Lib/test/test_tools/test_c_analyzer/test_common/test_files.py
index 6d14aea78a486..0c97d2a0bbf9a 100644
--- a/Lib/test/test_tools/test_c_analyzer/test_c_analyzer_common/test_files.py
+++ b/Lib/test/test_tools/test_c_analyzer/test_common/test_files.py
@@ -3,7 +3,7 @@
 
 from .. import tool_imports_for_tests
 with tool_imports_for_tests():
-    from c_analyzer_common.files import (
+    from c_analyzer.common.files import (
             iter_files, _walk_tree, glob_tree,
             )
 
diff --git a/Lib/test/test_tools/test_c_analyzer/test_c_analyzer_common/test_info.py b/Lib/test/test_tools/test_c_analyzer/test_common/test_info.py
similarity index 98%
rename from Lib/test/test_tools/test_c_analyzer/test_c_analyzer_common/test_info.py
rename to Lib/test/test_tools/test_c_analyzer/test_common/test_info.py
index 2d386713b9989..69dbb582c6b68 100644
--- a/Lib/test/test_tools/test_c_analyzer/test_c_analyzer_common/test_info.py
+++ b/Lib/test/test_tools/test_c_analyzer/test_common/test_info.py
@@ -4,7 +4,10 @@
 from ..util import PseudoStr, StrProxy, Object
 from .. import tool_imports_for_tests
 with tool_imports_for_tests():
-    from c_analyzer_common.info import ID
+    from c_analyzer.common.info import (
+            UNKNOWN,
+            ID,
+            )
 
 
 class IDTests(unittest.TestCase):
diff --git a/Lib/test/test_tools/test_c_analyzer/test_c_globals/test_show.py b/Lib/test/test_tools/test_c_analyzer/test_common/test_show.py
similarity index 94%
rename from Lib/test/test_tools/test_c_analyzer/test_c_globals/test_show.py
rename to Lib/test/test_tools/test_c_analyzer/test_common/test_show.py
index ce1dad85db1b8..91ca2f3b344dd 100644
--- a/Lib/test/test_tools/test_c_analyzer/test_c_globals/test_show.py
+++ b/Lib/test/test_tools/test_c_analyzer/test_common/test_show.py
@@ -2,8 +2,10 @@
 
 from .. import tool_imports_for_tests
 with tool_imports_for_tests():
-    from c_parser import info
-    from c_globals.show import basic
+    from c_analyzer.variables import info
+    from c_analyzer.common.show import (
+            basic,
+            )
 
 
 TYPICAL = [
diff --git a/Lib/test/test_tools/test_c_analyzer/test_c_globals/__init__.py b/Lib/test/test_tools/test_c_analyzer/test_cpython/__init__.py
similarity index 100%
rename from Lib/test/test_tools/test_c_analyzer/test_c_globals/__init__.py
rename to Lib/test/test_tools/test_c_analyzer/test_cpython/__init__.py
diff --git a/Lib/test/test_tools/test_c_analyzer/test_c_globals/test___main__.py b/Lib/test/test_tools/test_c_analyzer/test_cpython/test___main__.py
similarity index 79%
rename from Lib/test/test_tools/test_c_analyzer/test_c_globals/test___main__.py
rename to Lib/test/test_tools/test_c_analyzer/test_cpython/test___main__.py
index 5f52c588d7c8b..6d69ed7525b59 100644
--- a/Lib/test/test_tools/test_c_analyzer/test_c_globals/test___main__.py
+++ b/Lib/test/test_tools/test_c_analyzer/test_cpython/test___main__.py
@@ -3,12 +3,13 @@
 
 from .. import tool_imports_for_tests
 with tool_imports_for_tests():
-    from c_analyzer_common import SOURCE_DIRS
-    from c_analyzer_common.known import DATA_FILE as KNOWN_FILE
-    from c_parser import info
-    import c_globals as cg
-    from c_globals.supported import IGNORED_FILE
-    from c_globals.__main__ import cmd_check, cmd_show, parse_args, main
+    from c_analyzer.variables import info
+    from cpython import SOURCE_DIRS
+    from cpython.supported import IGNORED_FILE
+    from cpython.known import DATA_FILE as KNOWN_FILE
+    from cpython.__main__ import (
+            cmd_check, cmd_show, parse_args, main,
+            )
 
 
 TYPICAL = [
@@ -46,6 +47,8 @@ class CMDBase(unittest.TestCase):
 
     maxDiff = None
 
+#    _return_known_from_file = None
+#    _return_ignored_from_file = None
     _return_find = ()
 
     @property
@@ -56,8 +59,16 @@ def calls(self):
             self._calls = []
             return self._calls
 
-    def _find(self, *args):
-        self.calls.append(('_find', args))
+#    def _known_from_file(self, *args):
+#        self.calls.append(('_known_from_file', args))
+#        return self._return_known_from_file or {}
+#
+#    def _ignored_from_file(self, *args):
+#        self.calls.append(('_ignored_from_file', args))
+#        return self._return_ignored_from_file or {}
+
+    def _find(self, known, ignored, skip_objects=False):
+        self.calls.append(('_find', (known, ignored, skip_objects)))
         return self._return_find
 
     def _show(self, *args):
@@ -78,41 +89,35 @@ def test_defaults(self):
                   _print=self._print,
                   )
 
-        self.assertEqual(self.calls[0], (
-            '_find', (
-                SOURCE_DIRS,
-                KNOWN_FILE,
-                IGNORED_FILE,
-                ),
-            ))
+        self.assertEqual(
+                self.calls[0],
+                ('_find', (KNOWN_FILE, IGNORED_FILE, False)),
+                )
 
     def test_all_supported(self):
         self._return_find = [(v, s) for v, s in TYPICAL if s]
         dirs = ['src1', 'src2', 'Include']
 
         cmd_check('check',
-                 dirs,
-                 ignored='ignored.tsv',
-                 known='known.tsv',
-                 _find=self._find,
-                 _show=self._show,
-                 _print=self._print,
-                 )
+                  known='known.tsv',
+                  ignored='ignored.tsv',
+                  _find=self._find,
+                  _show=self._show,
+                  _print=self._print,
+                  )
 
         self.assertEqual(self.calls, [
-            ('_find', (dirs, 'known.tsv', 'ignored.tsv')),
+            ('_find', ('known.tsv', 'ignored.tsv', False)),
             #('_print', ('okay',)),
             ])
 
     def test_some_unsupported(self):
         self._return_find = TYPICAL
-        dirs = ['src1', 'src2', 'Include']
 
         with self.assertRaises(SystemExit) as cm:
             cmd_check('check',
-                      dirs,
-                      ignored='ignored.tsv',
                       known='known.tsv',
+                      ignored='ignored.tsv',
                       _find=self._find,
                       _show=self._show,
                       _print=self._print,
@@ -120,7 +125,7 @@ def test_some_unsupported(self):
 
         unsupported = [v for v, s in TYPICAL if not s]
         self.assertEqual(self.calls, [
-            ('_find', (dirs, 'known.tsv', 'ignored.tsv')),
+            ('_find', ('known.tsv', 'ignored.tsv', False)),
             ('_print', ('ERROR: found unsupported global variables',)),
             ('_print', ()),
             ('_show', (sorted(unsupported),)),
@@ -140,20 +145,15 @@ def test_defaults(self):
                  _print=self._print,
                  )
 
-        self.assertEqual(self.calls[0], (
-            '_find', (
-                SOURCE_DIRS,
-                KNOWN_FILE,
-                IGNORED_FILE,
-                ),
-            ))
+        self.assertEqual(
+                self.calls[0],
+                ('_find', (KNOWN_FILE, IGNORED_FILE, False)),
+                )
 
     def test_typical(self):
         self._return_find = TYPICAL
-        dirs = ['src1', 'src2', 'Include']
 
         cmd_show('show',
-                 dirs,
                  known='known.tsv',
                  ignored='ignored.tsv',
                  _find=self._find,
@@ -164,7 +164,7 @@ def test_typical(self):
         supported = [v for v, s in TYPICAL if s]
         unsupported = [v for v, s in TYPICAL if not s]
         self.assertEqual(self.calls, [
-            ('_find', (dirs, 'known.tsv', 'ignored.tsv')),
+            ('_find', ('known.tsv', 'ignored.tsv', False)),
             ('_print', ('supported:',)),
             ('_print', ('----------',)),
             ('_show', (sorted(supported),)),
@@ -201,7 +201,7 @@ def test_check_no_args(self):
         self.assertEqual(cmdkwargs, {
             'ignored': IGNORED_FILE,
             'known': KNOWN_FILE,
-            'dirs': SOURCE_DIRS,
+            #'dirs': SOURCE_DIRS,
             })
 
     def test_check_full_args(self):
@@ -209,16 +209,16 @@ def test_check_full_args(self):
             'check',
             '--ignored', 'spam.tsv',
             '--known', 'eggs.tsv',
-            'dir1',
-            'dir2',
-            'dir3',
+            #'dir1',
+            #'dir2',
+            #'dir3',
             ])
 
         self.assertEqual(cmd, 'check')
         self.assertEqual(cmdkwargs, {
             'ignored': 'spam.tsv',
             'known': 'eggs.tsv',
-            'dirs': ['dir1', 'dir2', 'dir3']
+            #'dirs': ['dir1', 'dir2', 'dir3']
             })
 
     def test_show_no_args(self):
@@ -230,7 +230,7 @@ def test_show_no_args(self):
         self.assertEqual(cmdkwargs, {
             'ignored': IGNORED_FILE,
             'known': KNOWN_FILE,
-            'dirs': SOURCE_DIRS,
+            #'dirs': SOURCE_DIRS,
             'skip_objects': False,
             })
 
@@ -239,16 +239,16 @@ def test_show_full_args(self):
             'show',
             '--ignored', 'spam.tsv',
             '--known', 'eggs.tsv',
-            'dir1',
-            'dir2',
-            'dir3',
+            #'dir1',
+            #'dir2',
+            #'dir3',
             ])
 
         self.assertEqual(cmd, 'show')
         self.assertEqual(cmdkwargs, {
             'ignored': 'spam.tsv',
             'known': 'eggs.tsv',
-            'dirs': ['dir1', 'dir2', 'dir3'],
+            #'dirs': ['dir1', 'dir2', 'dir3'],
             'skip_objects': False,
             })
 
diff --git a/Lib/test/test_tools/test_c_analyzer/test_c_globals/test_functional.py b/Lib/test/test_tools/test_c_analyzer/test_cpython/test_functional.py
similarity index 100%
rename from Lib/test/test_tools/test_c_analyzer/test_c_globals/test_functional.py
rename to Lib/test/test_tools/test_c_analyzer/test_cpython/test_functional.py
diff --git a/Lib/test/test_tools/test_c_analyzer/test_c_globals/test_supported.py b/Lib/test/test_tools/test_c_analyzer/test_cpython/test_supported.py
similarity index 85%
rename from Lib/test/test_tools/test_c_analyzer/test_c_globals/test_supported.py
rename to Lib/test/test_tools/test_c_analyzer/test_cpython/test_supported.py
index 1e7d40e2afcbd..a244b97e1fc7c 100644
--- a/Lib/test/test_tools/test_c_analyzer/test_c_globals/test_supported.py
+++ b/Lib/test/test_tools/test_c_analyzer/test_cpython/test_supported.py
@@ -4,9 +4,11 @@
 
 from .. import tool_imports_for_tests
 with tool_imports_for_tests():
-    from c_analyzer_common.info import ID
-    from c_parser import info
-    from c_globals.supported import is_supported, ignored_from_file
+    from c_analyzer.common.info import ID
+    from c_analyzer.variables.info import Variable
+    from cpython.supported import (
+            is_supported, ignored_from_file,
+            )
 
 
 class IsSupportedTests(unittest.TestCase):
@@ -14,8 +16,8 @@ class IsSupportedTests(unittest.TestCase):
     @unittest.expectedFailure
     def test_supported(self):
         statics = [
-                info.StaticVar('src1/spam.c', None, 'var1', 'const char *'),
-                info.StaticVar('src1/spam.c', None, 'var1', 'int'),
+                Variable('src1/spam.c', None, 'var1', 'const char *'),
+                Variable('src1/spam.c', None, 'var1', 'int'),
                 ]
         for static in statics:
             with self.subTest(static):
@@ -26,8 +28,8 @@ def test_supported(self):
     @unittest.expectedFailure
     def test_not_supported(self):
         statics = [
-                info.StaticVar('src1/spam.c', None, 'var1', 'PyObject *'),
-                info.StaticVar('src1/spam.c', None, 'var1', 'PyObject[10]'),
+                Variable('src1/spam.c', None, 'var1', 'PyObject *'),
+                Variable('src1/spam.c', None, 'var1', 'PyObject[10]'),
                 ]
         for static in statics:
             with self.subTest(static):
diff --git a/Lib/test/test_tools/test_c_analyzer/test_c_parser/__init__.py b/Lib/test/test_tools/test_c_analyzer/test_parser/__init__.py
similarity index 100%
rename from Lib/test/test_tools/test_c_analyzer/test_c_parser/__init__.py
rename to Lib/test/test_tools/test_c_analyzer/test_parser/__init__.py
diff --git a/Lib/test/test_tools/test_c_analyzer/test_c_parser/test_declarations.py b/Lib/test/test_tools/test_c_analyzer/test_parser/test_declarations.py
similarity index 99%
rename from Lib/test/test_tools/test_c_analyzer/test_c_parser/test_declarations.py
rename to Lib/test/test_tools/test_c_analyzer/test_parser/test_declarations.py
index b68744ef0aba4..674fcb1af1c7a 100644
--- a/Lib/test/test_tools/test_c_analyzer/test_c_parser/test_declarations.py
+++ b/Lib/test/test_tools/test_c_analyzer/test_parser/test_declarations.py
@@ -3,9 +3,9 @@
 
 from .. import tool_imports_for_tests
 with tool_imports_for_tests():
-    from c_parser.declarations import (
+    from c_analyzer.parser.declarations import (
         iter_global_declarations, iter_local_statements,
-        parse_func, parse_var, parse_compound,
+        parse_func, _parse_var, parse_compound,
         iter_variables,
         )
 
@@ -515,7 +515,7 @@ def test_typical(self):
             ])
         for stmt, expected in tests:
             with self.subTest(stmt):
-                name, vartype = parse_var(stmt)
+                name, vartype = _parse_var(stmt)
 
                 self.assertEqual((name, vartype), expected)
 
diff --git a/Lib/test/test_tools/test_c_analyzer/test_c_parser/test_preprocessor.py b/Lib/test/test_tools/test_c_analyzer/test_parser/test_preprocessor.py
similarity index 99%
rename from Lib/test/test_tools/test_c_analyzer/test_c_parser/test_preprocessor.py
rename to Lib/test/test_tools/test_c_analyzer/test_parser/test_preprocessor.py
index 89e15570d6530..56a1c9c612f72 100644
--- a/Lib/test/test_tools/test_c_analyzer/test_c_parser/test_preprocessor.py
+++ b/Lib/test/test_tools/test_c_analyzer/test_parser/test_preprocessor.py
@@ -6,7 +6,7 @@
 from ..util import wrapped_arg_combos, StrProxy
 from .. import tool_imports_for_tests
 with tool_imports_for_tests():
-    from c_parser.preprocessor import (
+    from c_analyzer.parser.preprocessor import (
         iter_lines,
         # directives
         parse_directive, PreprocessorDirective,
diff --git a/Lib/test/test_tools/test_c_analyzer/test_c_symbols/__init__.py b/Lib/test/test_tools/test_c_analyzer/test_symbols/__init__.py
similarity index 100%
rename from Lib/test/test_tools/test_c_analyzer/test_c_symbols/__init__.py
rename to Lib/test/test_tools/test_c_analyzer/test_symbols/__init__.py
diff --git a/Lib/test/test_tools/test_c_analyzer/test_c_symbols/test_info.py b/Lib/test/test_tools/test_c_analyzer/test_symbols/test_info.py
similarity index 98%
rename from Lib/test/test_tools/test_c_analyzer/test_c_symbols/test_info.py
rename to Lib/test/test_tools/test_c_analyzer/test_symbols/test_info.py
index e029dcf66124d..1282a89718c82 100644
--- a/Lib/test/test_tools/test_c_analyzer/test_c_symbols/test_info.py
+++ b/Lib/test/test_tools/test_c_analyzer/test_symbols/test_info.py
@@ -4,8 +4,8 @@
 from ..util import PseudoStr, StrProxy, Object
 from .. import tool_imports_for_tests
 with tool_imports_for_tests():
-    from c_analyzer_common.info import ID
-    from c_symbols.info import Symbol
+    from c_analyzer.common.info import ID
+    from c_analyzer.symbols.info import Symbol
 
 
 class SymbolTests(unittest.TestCase):
diff --git a/Lib/test/test_tools/test_c_analyzer/test_variables/__init__.py b/Lib/test/test_tools/test_c_analyzer/test_variables/__init__.py
new file mode 100644
index 0000000000000..bc502ef32d291
--- /dev/null
+++ b/Lib/test/test_tools/test_c_analyzer/test_variables/__init__.py
@@ -0,0 +1,6 @@
+import os.path
+from test.support import load_package_tests
+
+
+def load_tests(*args):
+    return load_package_tests(os.path.dirname(__file__), *args)
diff --git a/Lib/test/test_tools/test_c_analyzer/test_variables/test_find.py b/Lib/test/test_tools/test_c_analyzer/test_variables/test_find.py
new file mode 100644
index 0000000000000..7a13cf3f5bf56
--- /dev/null
+++ b/Lib/test/test_tools/test_c_analyzer/test_variables/test_find.py
@@ -0,0 +1,124 @@
+import unittest
+
+from .. import tool_imports_for_tests
+with tool_imports_for_tests():
+    from c_analyzer.variables import info
+    from c_analyzer.variables.find import (
+            vars_from_binary,
+            )
+
+
+class _Base(unittest.TestCase):
+
+    maxDiff = None
+
+    @property
+    def calls(self):
+        try:
+            return self._calls
+        except AttributeError:
+            self._calls = []
+            return self._calls
+
+
+class VarsFromBinaryTests(_Base):
+
+    _return_iter_vars = ()
+    _return_get_symbol_resolver = None
+
+    def setUp(self):
+        super().setUp()
+
+        self.kwargs = dict(
+                _iter_vars=self._iter_vars,
+                _get_symbol_resolver=self._get_symbol_resolver,
+                )
+
+    def _iter_vars(self, binfile, resolve, handle_id):
+        self.calls.append(('_iter_vars', (binfile, resolve, handle_id)))
+        return [(v, v.id) for v in self._return_iter_vars]
+
+    def _get_symbol_resolver(self, known=None, dirnames=(), *,
+                             handle_var,
+                             filenames=None,
+                             check_filename=None,
+                             perfilecache=None,
+                             ):
+        self.calls.append(('_get_symbol_resolver',
+                           (known, dirnames, handle_var, filenames,
+                            check_filename, perfilecache)))
+        return self._return_get_symbol_resolver
+
+    def test_typical(self):
+        resolver = self._return_get_symbol_resolver = object()
+        variables = self._return_iter_vars = [
+            info.Variable.from_parts('dir1/spam.c', None, 'var1', 'int'),
+            info.Variable.from_parts('dir1/spam.c', None, 'var2', 'static int'),
+            info.Variable.from_parts('dir1/spam.c', None, 'var3', 'char *'),
+            info.Variable.from_parts('dir1/spam.c', 'func2', 'var4', 'const char *'),
+            info.Variable.from_parts('dir1/eggs.c', None, 'var1', 'static int'),
+            info.Variable.from_parts('dir1/eggs.c', 'func1', 'var2', 'static char *'),
+            ]
+        known = object()
+        filenames = object()
+
+        found = list(vars_from_binary('python',
+                                      known=known,
+                                      filenames=filenames,
+                                      **self.kwargs))
+
+        self.assertEqual(found, [
+            info.Variable.from_parts('dir1/spam.c', None, 'var1', 'int'),
+            info.Variable.from_parts('dir1/spam.c', None, 'var2', 'static int'),
+            info.Variable.from_parts('dir1/spam.c', None, 'var3', 'char *'),
+            info.Variable.from_parts('dir1/spam.c', 'func2', 'var4', 'const char *'),
+            info.Variable.from_parts('dir1/eggs.c', None, 'var1', 'static int'),
+            info.Variable.from_parts('dir1/eggs.c', 'func1', 'var2', 'static char *'),
+            ])
+        self.assertEqual(self.calls, [
+            ('_get_symbol_resolver', (filenames, known, info.Variable.from_id, None, None, {})),
+            ('_iter_vars', ('python', resolver, None)),
+            ])
+
+#        self._return_iter_symbols = [
+#                s_info.Symbol(('dir1/spam.c', None, 'var1'), 'variable', False),
+#                s_info.Symbol(('dir1/spam.c', None, 'var2'), 'variable', False),
+#                s_info.Symbol(('dir1/spam.c', None, 'func1'), 'function', False),
+#                s_info.Symbol(('dir1/spam.c', None, 'func2'), 'function', True),
+#                s_info.Symbol(('dir1/spam.c', None, 'var3'), 'variable', False),
+#                s_info.Symbol(('dir1/spam.c', 'func2', 'var4'), 'variable', False),
+#                s_info.Symbol(('dir1/ham.c', None, 'var1'), 'variable', True),
+#                s_info.Symbol(('dir1/eggs.c', None, 'var1'), 'variable', False),
+#                s_info.Symbol(('dir1/eggs.c', None, 'xyz'), 'other', False),
+#                s_info.Symbol(('dir1/eggs.c', '???', 'var2'), 'variable', False),
+#                s_info.Symbol(('???', None, 'var_x'), 'variable', False),
+#                s_info.Symbol(('???', '???', 'var_y'), 'variable', False),
+#                s_info.Symbol((None, None, '???'), 'other', False),
+#                ]
+#        known = object()
+#
+#        vars_from_binary('python', knownvars=known, **this.kwargs)
+#        found = list(globals_from_symbols(['dir1'], self.iter_symbols))
+#
+#        self.assertEqual(found, [
+#            info.Variable.from_parts('dir1/spam.c', None, 'var1', '???'),
+#            info.Variable.from_parts('dir1/spam.c', None, 'var2', '???'),
+#            info.Variable.from_parts('dir1/spam.c', None, 'var3', '???'),
+#            info.Variable.from_parts('dir1/spam.c', 'func2', 'var4', '???'),
+#            info.Variable.from_parts('dir1/eggs.c', None, 'var1', '???'),
+#            ])
+#        self.assertEqual(self.calls, [
+#            ('iter_symbols', (['dir1'],)),
+#            ])
+#
+#    def test_no_symbols(self):
+#        self._return_iter_symbols = []
+#
+#        found = list(globals_from_symbols(['dir1'], self.iter_symbols))
+#
+#        self.assertEqual(found, [])
+#        self.assertEqual(self.calls, [
+#            ('iter_symbols', (['dir1'],)),
+#            ])
+
+    # XXX need functional test
diff --git a/Lib/test/test_tools/test_c_analyzer/test_c_parser/test_info.py b/Lib/test/test_tools/test_c_analyzer/test_variables/test_info.py
similarity index 98%
rename from Lib/test/test_tools/test_c_analyzer/test_c_parser/test_info.py
rename to Lib/test/test_tools/test_c_analyzer/test_variables/test_info.py
index d1a966c58904d..d424d8eebb811 100644
--- a/Lib/test/test_tools/test_c_analyzer/test_c_parser/test_info.py
+++ b/Lib/test/test_tools/test_c_analyzer/test_variables/test_info.py
@@ -4,10 +4,10 @@
 from ..util import PseudoStr, StrProxy, Object
 from .. import tool_imports_for_tests
 with tool_imports_for_tests():
-    from c_analyzer_common.info import ID, UNKNOWN
-    from c_parser.info import (
-        normalize_vartype, Variable,
-        )
+    from c_analyzer.common.info import UNKNOWN, ID
+    from c_analyzer.variables.info import (
+            normalize_vartype, Variable
+            )
 
 
 class NormalizeVartypeTests(unittest.TestCase):
diff --git a/Lib/test/test_tools/test_c_analyzer/test_variables/test_known.py b/Lib/test/test_tools/test_c_analyzer/test_variables/test_known.py
new file mode 100644
index 0000000000000..49ff45c6d1b2c
--- /dev/null
+++ b/Lib/test/test_tools/test_c_analyzer/test_variables/test_known.py
@@ -0,0 +1,139 @@
+import re
+import textwrap
+import unittest
+
+from .. import tool_imports_for_tests
+with tool_imports_for_tests():
+    from c_analyzer.common.info import ID
+    from c_analyzer.variables.info import Variable
+    from c_analyzer.variables.known import (
+            read_file,
+            from_file,
+            )
+
+class _BaseTests(unittest.TestCase):
+
+    maxDiff = None
+
+    @property
+    def calls(self):
+        try:
+            return self._calls
+        except AttributeError:
+            self._calls = []
+            return self._calls
+
+
+class ReadFileTests(_BaseTests):
+
+    _return_read_tsv = ()
+
+    def _read_tsv(self, *args):
+        self.calls.append(('_read_tsv', args))
+        return self._return_read_tsv
+
+    def test_typical(self):
+        lines = textwrap.dedent('''
+            filename    funcname        name    kind    declaration
+            file1.c     -       var1    variable        static int
+            file1.c     func1   local1  variable        static int
+            file1.c     -       var2    variable        int
+            file1.c     func2   local2  variable        char *
+            file2.c     -       var1    variable        char *
+            ''').strip().splitlines()
+        lines = [re.sub(r'\s+', '\t', line, 4) for line in lines]
+        self._return_read_tsv = [tuple(v.strip() for v in line.split('\t'))
+                                 for line in lines[1:]]
+
+        known = list(read_file('known.tsv', _read_tsv=self._read_tsv))
+
+        self.assertEqual(known, [
+            ('variable', ID('file1.c', '', 'var1'), 'static int'),
+            ('variable', ID('file1.c', 'func1', 'local1'), 'static int'),
+            ('variable', ID('file1.c', '', 'var2'), 'int'),
+            ('variable', ID('file1.c', 'func2', 'local2'), 'char *'),
+            ('variable', ID('file2.c', '', 'var1'), 'char *'),
+            ])
+        self.assertEqual(self.calls, [
+            ('_read_tsv',
+             ('known.tsv', 'filename\tfuncname\tname\tkind\tdeclaration')),
+            ])
+
+    def test_empty(self):
+        self._return_read_tsv = []
+
+        known = list(read_file('known.tsv', _read_tsv=self._read_tsv))
+
+        self.assertEqual(known, [])
+        self.assertEqual(self.calls, [
+            ('_read_tsv', ('known.tsv', 'filename\tfuncname\tname\tkind\tdeclaration')),
+            ])
+
+
+class FromFileTests(_BaseTests):
+
+    _return_read_file = ()
+    _return_handle_var = ()
+
+    def _read_file(self, infile):
+        self.calls.append(('_read_file', (infile,)))
+        return iter(self._return_read_file)
+
+    def _handle_var(self, varid, decl):
+        self.calls.append(('_handle_var', (varid, decl)))
+        var = self._return_handle_var.pop(0)
+        return var
+
+    def test_typical(self):
+        expected = [
+            Variable.from_parts('file1.c', '', 'var1', 'static int'),
+            Variable.from_parts('file1.c', 'func1', 'local1', 'static int'),
+            Variable.from_parts('file1.c', '', 'var2', 'int'),
+            Variable.from_parts('file1.c', 'func2', 'local2', 'char *'),
+            Variable.from_parts('file2.c', '', 'var1', 'char *'),
+            ]
+        self._return_read_file = [('variable', v.id, v.vartype)
+                                  for v in expected]
+#            ('variable', ID('file1.c', '', 'var1'), 'static int'),
+#            ('variable', ID('file1.c', 'func1', 'local1'), 'static int'),
+#            ('variable', ID('file1.c', '', 'var2'), 'int'),
+#            ('variable', ID('file1.c', 'func2', 'local2'), 'char *'),
+#            ('variable', ID('file2.c', '', 'var1'), 'char *'),
+#            ]
+        self._return_handle_var = list(expected)  # a copy
+
+        known = from_file('known.tsv',
+                          handle_var=self._handle_var,
+                          _read_file=self._read_file,
+                          )
+
+        self.assertEqual(known, {
+            'variables': {v.id: v for v in expected},
+            })
+#                Variable.from_parts('file1.c', '', 'var1', 'static int'),
+#                Variable.from_parts('file1.c', 'func1', 'local1', 'static int'),
+#                Variable.from_parts('file1.c', '', 'var2', 'int'),
+#                Variable.from_parts('file1.c', 'func2', 'local2', 'char *'),
+#                Variable.from_parts('file2.c', '', 'var1', 'char *'),
+#                ]},
+#            })
+        self.assertEqual(self.calls, [
+            ('_read_file', ('known.tsv',)),
+            *[('_handle_var', (v.id, v.vartype))
+              for v in expected],
+            ])
+
+    def test_empty(self):
+        self._return_read_file = []
+
+        known = from_file('known.tsv',
+                          handle_var=self._handle_var,
+                          _read_file=self._read_file,
+                          )
+
+        self.assertEqual(known, {
+            'variables': {},
+            })
+        self.assertEqual(self.calls, [
+            ('_read_file', ('known.tsv',)),
+            ])
diff --git a/Tools/c-analyzer/c-globals.py b/Tools/c-analyzer/c-globals.py
index 9afe059b28c66..b36b791241d53 100644
--- a/Tools/c-analyzer/c-globals.py
+++ b/Tools/c-analyzer/c-globals.py
@@ -1,6 +1,6 @@
 # This is a script equivalent of running "python -m test.test_c_globals.cg".
 
-from c_globals.__main__ import parse_args, main
+from cpython.__main__ import parse_args, main
 
 
 # This is effectively copied from cg/__main__.py:
diff --git a/Tools/c-analyzer/c_globals/__init__.py b/Tools/c-analyzer/c_analyzer/__init__.py
similarity index 100%
rename from Tools/c-analyzer/c_globals/__init__.py
rename to Tools/c-analyzer/c_analyzer/__init__.py
diff --git a/Tools/c-analyzer/c_parser/__init__.py b/Tools/c-analyzer/c_analyzer/common/__init__.py
similarity index 100%
rename from Tools/c-analyzer/c_parser/__init__.py
rename to Tools/c-analyzer/c_analyzer/common/__init__.py
diff --git a/Tools/c-analyzer/c_analyzer_common/files.py b/Tools/c-analyzer/c_analyzer/common/files.py
similarity index 82%
rename from Tools/c-analyzer/c_analyzer_common/files.py
rename to Tools/c-analyzer/c_analyzer/common/files.py
index b3cd16c8dc008..ab551a84bad15 100644
--- a/Tools/c-analyzer/c_analyzer_common/files.py
+++ b/Tools/c-analyzer/c_analyzer/common/files.py
@@ -2,7 +2,10 @@
 import os
 import os.path
 
-from . import SOURCE_DIRS, REPO_ROOT
+# XXX need tests:
+# * walk_tree()
+# * glob_tree()
+# * iter_files_by_suffix()
 
 
 C_SOURCE_SUFFIXES = ('.c', '.h')
@@ -115,24 +118,3 @@ def iter_files_by_suffix(root, suffixes, relparent=None, *,
     # XXX Ignore repeated suffixes?
     for suffix in suffixes:
         yield from _iter_files(root, suffix, relparent)
-
-
-def iter_cpython_files(*,
-                       walk=walk_tree,
-                       _files=iter_files_by_suffix,
-                       ):
-    """Yield each file in the tree for each of the given directory names."""
-    excludedtrees = [
-        os.path.join('Include', 'cpython', ''),
-        ]
-    def is_excluded(filename):
-        for root in excludedtrees:
-            if filename.startswith(root):
-                return True
-        return False
-    for filename in _files(SOURCE_DIRS, C_SOURCE_SUFFIXES, REPO_ROOT,
-                           walk=walk,
-                           ):
-        if is_excluded(filename):
-            continue
-        yield filename
diff --git a/Tools/c-analyzer/c_analyzer/common/info.py b/Tools/c-analyzer/c_analyzer/common/info.py
new file mode 100644
index 0000000000000..3f3f8c5b05de5
--- /dev/null
+++ b/Tools/c-analyzer/c_analyzer/common/info.py
@@ -0,0 +1,138 @@
+from collections import namedtuple
+import re
+
+from .util import classonly, _NTBase
+
+# XXX need tests:
+# * ID.match()
+
+
+UNKNOWN = '???'
+
+NAME_RE = re.compile(r'^([a-zA-Z]|_\w*[a-zA-Z]\w*|[a-zA-Z]\w*)$')
+
+
+class ID(_NTBase, namedtuple('ID', 'filename funcname name')):
+    """A unique ID for a single symbol or declaration."""
+
+    __slots__ = ()
+    # XXX Add optional conditions (tuple of strings) field.
+    #conditions = Slot()
+
+    @classonly
+    def from_raw(cls, raw):
+        if not raw:
+            return None
+        if isinstance(raw, str):
+            return cls(None, None, raw)
+        try:
+            name, = raw
+            filename = None
+        except ValueError:
+            try:
+                filename, name = raw
+            except ValueError:
+                return super().from_raw(raw)
+        return cls(filename, None, name)
+
+    def __new__(cls, filename, funcname, name):
+        self = super().__new__(
+                cls,
+                filename=str(filename) if filename else None,
+                funcname=str(funcname) if funcname else None,
+                name=str(name) if name else None,
+                )
+        #cls.conditions.set(self, tuple(str(s) if s else None
+        #                               for s in conditions or ()))
+        return self
+
+    def validate(self):
+        """Fail if the object is invalid (i.e. init with bad data)."""
+        if not self.name:
+            raise TypeError('missing name')
+        else:
+            if not NAME_RE.match(self.name):
+                raise ValueError(
+                        f'name must be an identifier, got {self.name!r}')
+
+        # Symbols from a binary might not have filename/funcname info.
+
+        if self.funcname:
+            if not self.filename:
+                raise TypeError('missing filename')
+            if not NAME_RE.match(self.funcname) and self.funcname != UNKNOWN:
+                raise ValueError(
+                        f'name must be an identifier, got {self.funcname!r}')
+
+        # XXX Require the filename (at least UNKONWN)?
+        # XXX Check the filename?
+
+    @property
+    def islocal(self):
+        return self.funcname is not None
+
+    def match(self, other, *,
+              match_files=(lambda f1, f2: f1 == f2),
+              ):
+        """Return True if the two match.
+
+        At least one of the two must be completely valid (no UNKNOWN
+        anywhere).  Otherwise False is returned.  The remaining one
+        *may* have UNKNOWN for both funcname and filename.  It must
+        have a valid name though.
+
+        The caller is responsible for knowing which of the two is valid
+        (and which to use if both are valid).
+        """
+        # First check the name.
+        if self.name is None:
+            return False
+        if other.name != self.name:
+            return False
+
+        # Then check the filename.
+        if self.filename is None:
+            return False
+        if other.filename is None:
+            return False
+        if self.filename == UNKNOWN:
+            # "other" must be the valid one.
+            if other.funcname == UNKNOWN:
+                return False
+            elif self.funcname != UNKNOWN:
+                # XXX Try matching funcname even though we don't
+                # know the filename?
+                raise NotImplementedError
+            else:
+                return True
+        elif other.filename == UNKNOWN:
+            # "self" must be the valid one.
+            if self.funcname == UNKNOWN:
+                return False
+            elif other.funcname != UNKNOWN:
+                # XXX Try matching funcname even though we don't
+                # know the filename?
+                raise NotImplementedError
+            else:
+                return True
+        elif not match_files(self.filename, other.filename):
+            return False
+
+        # Finally, check the funcname.
+        if self.funcname == UNKNOWN:
+            # "other" must be the valid one.
+            if other.funcname == UNKNOWN:
+                return False
+            else:
+                return other.funcname is not None
+        elif other.funcname == UNKNOWN:
+            # "self" must be the valid one.
+            if self.funcname == UNKNOWN:
+                return False
+            else:
+                return self.funcname is not None
+        elif self.funcname == other.funcname:
+            # Both are valid.
+            return True
+
+        return False
diff --git a/Tools/c-analyzer/c_analyzer/common/show.py b/Tools/c-analyzer/c_analyzer/common/show.py
new file mode 100644
index 0000000000000..5f3cb1c2fb0b5
--- /dev/null
+++ b/Tools/c-analyzer/c_analyzer/common/show.py
@@ -0,0 +1,11 @@
+
+def basic(variables, *,
+          _print=print):
+    """Print each row simply."""
+    for var in variables:
+        if var.funcname:
+            line = f'{var.filename}:{var.funcname}():{var.name}'
+        else:
+            line = f'{var.filename}:{var.name}'
+        line = f'{line:<64} {var.vartype}'
+        _print(line)
diff --git a/Tools/c-analyzer/c_analyzer_common/util.py b/Tools/c-analyzer/c_analyzer/common/util.py
similarity index 100%
rename from Tools/c-analyzer/c_analyzer_common/util.py
rename to Tools/c-analyzer/c_analyzer/common/util.py
diff --git a/Tools/c-analyzer/c_symbols/__init__.py b/Tools/c-analyzer/c_analyzer/parser/__init__.py
similarity index 100%
rename from Tools/c-analyzer/c_symbols/__init__.py
rename to Tools/c-analyzer/c_analyzer/parser/__init__.py
diff --git a/Tools/c-analyzer/c_parser/declarations.py b/Tools/c-analyzer/c_analyzer/parser/declarations.py
similarity index 83%
rename from Tools/c-analyzer/c_parser/declarations.py
rename to Tools/c-analyzer/c_analyzer/parser/declarations.py
index 19fa3ff4e66bb..f37072cccad86 100644
--- a/Tools/c-analyzer/c_parser/declarations.py
+++ b/Tools/c-analyzer/c_analyzer/parser/declarations.py
@@ -2,6 +2,8 @@
 import shlex
 import subprocess
 
+from ..common.info import UNKNOWN
+
 from . import source
 
 
@@ -194,7 +196,28 @@ def parse_func(stmt, body):
     return name, signature
 
 
-def parse_var(stmt):
+#TYPE_SPEC = rf'''(?:
+#        )'''
+#VAR_DECLARATOR = rf'''(?:
+#        )'''
+#VAR_DECL = rf'''(?:
+#            {TYPE_SPEC}+
+#            {VAR_DECLARATOR}
+#            \s*
+#        )'''
+#VAR_DECLARATION = rf'''(?:
+#            {VAR_DECL}
+#            (?: = [^=] [^;]* )?
+#            ;
+#        )'''
+#
+#
+#def parse_variable(decl, *, inFunc=False):
+#    """Return [(name, storage, vartype)] for the given variable declaration."""
+#    ...
+
+
+def _parse_var(stmt):
     """Return (name, vartype) for the given variable declaration."""
     stmt = stmt.rstrip(';')
     m = LOCAL_STMT_START_RE.match(stmt)
@@ -220,6 +243,27 @@ def parse_var(stmt):
     return name, vartype
 
 
+def extract_storage(decl, *, infunc=None):
+    """Return (storage, vartype) based on the given declaration.
+
+    The default storage is "implicit" (or "local" if infunc is True).
+    """
+    if decl == UNKNOWN:
+        return decl
+    if decl.startswith('static '):
+        return 'static'
+        #return 'static', decl.partition(' ')[2].strip()
+    elif decl.startswith('extern '):
+        return 'extern'
+        #return 'extern', decl.partition(' ')[2].strip()
+    elif re.match('.*\b(static|extern)\b', decl):
+        raise NotImplementedError
+    elif infunc:
+        return 'local'
+    else:
+        return 'implicit'
+
+
 def parse_compound(stmt, blocks):
     """Return (headers, bodies) for the given compound statement."""
     # XXX Identify declarations inside compound statements
@@ -228,14 +272,17 @@ def parse_compound(stmt, blocks):
 
 
 def iter_variables(filename, *,
+                   preprocessed=False,
                    _iter_source_lines=source.iter_lines,
                    _iter_global=iter_global_declarations,
                    _iter_local=iter_local_statements,
                    _parse_func=parse_func,
-                   _parse_var=parse_var,
+                   _parse_var=_parse_var,
                    _parse_compound=parse_compound,
                    ):
     """Yield (funcname, name, vartype) for every variable in the given file."""
+    if preprocessed:
+        raise NotImplementedError
     lines = _iter_source_lines(filename)
     for stmt, body in _iter_global(lines):
         # At the file top-level we only have to worry about vars & funcs.
@@ -256,7 +303,7 @@ def iter_variables(filename, *,
 
 def _iter_locals(lines, *,
                  _iter_statements=iter_local_statements,
-                 _parse_var=parse_var,
+                 _parse_var=_parse_var,
                  _parse_compound=parse_compound,
                  ):
     compound = [lines]
@@ -278,18 +325,15 @@ def _iter_locals(lines, *,
                 compound.extend(bodies)
 
 
-def iter_all(dirnames):
+def iter_all(filename, *,
+             preprocessed=False,
+             ):
     """Yield a Declaration for each one found.
 
     If there are duplicates, due to preprocessor conditionals, then
     they are checked to make sure they are the same.
     """
-    raise NotImplementedError
-
-
-def iter_preprocessed(dirnames):
-    """Yield a Declaration for each one found.
-
-    All source files are run through the preprocessor first.
-    """
-    raise NotImplementedError
+    # XXX For the moment we cheat.
+    for funcname, name, decl in iter_variables(filename,
+                                               preprocessed=preprocessed):
+        yield 'variable', funcname, name, decl
diff --git a/Tools/c-analyzer/c_analyzer/parser/find.py b/Tools/c-analyzer/c_analyzer/parser/find.py
new file mode 100644
index 0000000000000..3860d3d459b18
--- /dev/null
+++ b/Tools/c-analyzer/c_analyzer/parser/find.py
@@ -0,0 +1,107 @@
+from ..common.info import UNKNOWN, ID
+
+from . import declarations
+
+# XXX need tests:
+# * variables
+# * variable
+# * variable_from_id
+
+
+def _iter_vars(filenames, preprocessed, *,
+               handle_id=None,
+               _iter_decls=declarations.iter_all,
+               ):
+    if handle_id is None:
+        handle_id = ID
+
+    for filename in filenames or ():
+        for kind, funcname, name, decl in _iter_decls(filename,
+                                                      preprocessed=preprocessed,
+                                                      ):
+            if kind != 'variable':
+                continue
+            varid = handle_id(filename, funcname, name)
+            yield varid, decl
+
+
+# XXX Add a "handle_var" arg like we did for get_resolver()?
+
+def variables(*filenames,
+              perfilecache=None,
+              preprocessed=False,
+              known=None,  # for types
+              handle_id=None,
+              _iter_vars=_iter_vars,
+              ):
+    """Yield (varid, decl) for each variable found in the given files.
+
+    If "preprocessed" is provided (and not False/None) then it is used
+    to decide which tool to use to parse the source code after it runs
+    through the C preprocessor.  Otherwise the raw
+    """
+    if len(filenames) == 1 and not (filenames[0], str):
+        filenames, = filenames
+
+    if perfilecache is None:
+        yield from _iter_vars(filenames, preprocessed)
+    else:
+        # XXX Cache per-file variables (e.g. `{filename: [(varid, decl)]}`).
+        raise NotImplementedError
+
+
+def variable(name, filenames, *,
+             local=False,
+             perfilecache=None,
+             preprocessed=False,
+             handle_id=None,
+             _iter_vars=variables,
+             ):
+    """Return (varid, decl) for the first found variable that matches.
+
+    If "local" is True then the first matching local variable in the
+    file will always be returned.  To avoid that, pass perfilecache and
+    pop each variable from the cache after using it.
+    """
+    for varid, decl in _iter_vars(filenames,
+                                  perfilecache=perfilecache,
+                                  preprocessed=preprocessed,
+                                  ):
+        if varid.name != name:
+            continue
+        if local:
+            if varid.funcname:
+                if varid.funcname == UNKNOWN:
+                    raise NotImplementedError
+                return varid, decl
+        elif not varid.funcname:
+            return varid, decl
+    else:
+        return None, None  # No matching variable was found.
+
+
+def variable_from_id(id, filenames, *,
+                     perfilecache=None,
+                     preprocessed=False,
+                     handle_id=None,
+                     _get_var=variable,
+                     ):
+    """Return (varid, decl) for the first found variable that matches."""
+    local = False
+    if isinstance(id, str):
+        name = id
+    else:
+        if id.funcname == UNKNOWN:
+            local = True
+        elif id.funcname:
+            raise NotImplementedError
+
+        name = id.name
+        if id.filename and id.filename != UNKNOWN:
+            filenames = [id.filename]
+    return _get_var(name, filenames,
+                    local=local,
+                    perfilecache=perfilecache,
+                    preprocessed=preprocessed,
+                    handle_id=handle_id,
+                    )
diff --git a/Tools/c-analyzer/c_parser/naive.py b/Tools/c-analyzer/c_analyzer/parser/naive.py
similarity index 80%
rename from Tools/c-analyzer/c_parser/naive.py
rename to Tools/c-analyzer/c_analyzer/parser/naive.py
index 160f96c279e26..4a4822d84ff54 100644
--- a/Tools/c-analyzer/c_parser/naive.py
+++ b/Tools/c-analyzer/c_analyzer/parser/naive.py
@@ -1,8 +1,7 @@
 import re
 
-from c_analyzer_common.info import UNKNOWN
+from ..common.info import UNKNOWN, ID
 
-from .info import Variable
 from .preprocessor import _iter_clean_lines
 
 
@@ -55,7 +54,7 @@ def parse_variable_declaration(srcline):
 
 
 def parse_variable(srcline, funcname=None):
-    """Return a Variable for the variable declared on the line (or None)."""
+    """Return (varid, decl) for the variable declared on the line (or None)."""
     line = srcline.strip()
 
     # XXX Handle more than just static variables.
@@ -74,7 +73,7 @@ def iter_variables(filename, *,
                    _get_srclines=get_srclines,
                    _default_parse_variable=parse_variable,
                    ):
-    """Yield a Variable for each in the given source file."""
+    """Yield (varid, decl) for each variable in the given source file."""
     if parse_variable is None:
         parse_variable = _default_parse_variable
 
@@ -99,13 +98,13 @@ def iter_variables(filename, *,
         info = parse_variable(line, funcname)
         if isinstance(info, list):
             for name, _funcname, decl in info:
-                yield Variable.from_parts(filename, _funcname, name, decl)
+                yield ID(filename, _funcname, name), decl
             continue
         name, decl = info
 
         if name is None:
             continue
-        yield Variable.from_parts(filename, funcname, name, decl)
+        yield ID(filename, funcname, name), decl
 
 
 def _match_varid(variable, name, funcname, ignored=None):
@@ -134,12 +133,12 @@ def find_variable(filename, funcname, name, *,
 
     Return None if the variable is not found.
     """
-    for variable in _iter_variables(filename,
+    for varid, decl in _iter_variables(filename,
                                     srccache=srccache,
                                     parse_variable=parse_variable,
                                     ):
-        if _match_varid(variable, name, funcname, ignored):
-            return variable
+        if _match_varid(varid, name, funcname, ignored):
+            return varid, decl
     else:
         return None
 
@@ -149,10 +148,10 @@ def find_variables(varids, filenames=None, *,
                    parse_variable=None,
                    _find_symbol=find_variable,
                    ):
-    """Yield a Variable for each ID.
+    """Yield (varid, decl) for each ID.
 
     If the variable is not found then its decl will be UNKNOWN.  That
-    way there will be one resulting Variable per given ID.
+    way there will be one resulting variable per given ID.
     """
     if srccache is _NOT_SET:
         srccache = {}
@@ -163,18 +162,18 @@ def find_variables(varids, filenames=None, *,
             srcfiles = [varid.filename]
         else:
             if not filenames:
-                yield Variable(varid, UNKNOWN, UNKNOWN)
+                yield varid, UNKNOWN
                 continue
             srcfiles = filenames
         for filename in srcfiles:
-            found = _find_varid(filename, varid.funcname, varid.name,
-                                 ignored=used,
-                                 srccache=srccache,
-                                 parse_variable=parse_variable,
-                                 )
-            if found:
-                yield found
-                used.add(found)
+            varid, decl = _find_varid(filename, varid.funcname, varid.name,
+                                      ignored=used,
+                                      srccache=srccache,
+                                      parse_variable=parse_variable,
+                                      )
+            if varid:
+                yield varid, decl
+                used.add(varid)
                 break
         else:
-            yield Variable(varid, UNKNOWN, UNKNOWN)
+            yield varid, UNKNOWN
diff --git a/Tools/c-analyzer/c_parser/preprocessor.py b/Tools/c-analyzer/c_analyzer/parser/preprocessor.py
similarity index 99%
rename from Tools/c-analyzer/c_parser/preprocessor.py
rename to Tools/c-analyzer/c_analyzer/parser/preprocessor.py
index 0e2866e4873e6..41f306e5f8022 100644
--- a/Tools/c-analyzer/c_parser/preprocessor.py
+++ b/Tools/c-analyzer/c_analyzer/parser/preprocessor.py
@@ -3,8 +3,7 @@
 import os
 import re
 
-from c_analyzer_common import util
-from . import info
+from ..common import util, info
 
 
 CONTINUATION = '\\' + os.linesep
diff --git a/Tools/c-analyzer/c_parser/source.py b/Tools/c-analyzer/c_analyzer/parser/source.py
similarity index 100%
rename from Tools/c-analyzer/c_parser/source.py
rename to Tools/c-analyzer/c_analyzer/parser/source.py
diff --git a/Tools/c-analyzer/c_analyzer/symbols/__init__.py b/Tools/c-analyzer/c_analyzer/symbols/__init__.py
new file mode 100644
index 0000000000000..e69de29bb2d1d
diff --git a/Tools/c-analyzer/c_symbols/binary.py b/Tools/c-analyzer/c_analyzer/symbols/_nm.py
similarity index 50%
rename from Tools/c-analyzer/c_symbols/binary.py
rename to Tools/c-analyzer/c_analyzer/symbols/_nm.py
index e125dbd5b5edc..f3a75a6d4ba82 100644
--- a/Tools/c-analyzer/c_symbols/binary.py
+++ b/Tools/c-analyzer/c_analyzer/symbols/_nm.py
@@ -1,46 +1,24 @@
-import os
 import os.path
 import shutil
-import sys
-
-from c_analyzer_common import util, info
-from . import source
-from .info import Symbol
-
-
-#PYTHON = os.path.join(REPO_ROOT, 'python')
-PYTHON = sys.executable
 
+from c_analyzer.common import util, info
 
-def iter_symbols(binary=PYTHON, dirnames=None, *,
-                 # Alternately, use look_up_known_symbol()
-                 # from c_globals.supported.
-                 find_local_symbol=source.find_symbol,
-                 _file_exists=os.path.exists,
-                 _iter_symbols_nm=(lambda b, *a: _iter_symbols_nm(b, *a)),
-                 ):
-    """Yield a Symbol for each symbol found in the binary."""
-    if not _file_exists(binary):
-        raise Exception('executable missing (need to build it first?)')
-
-    if find_local_symbol:
-        cache = {}
-        def find_local_symbol(name, *, _find=find_local_symbol):
-            return _find(name, dirnames, _perfilecache=cache)
-    else:
-        find_local_symbol = None
+from .info import Symbol
 
-    if os.name == 'nt':
-        # XXX Support this.
-        raise NotImplementedError
-    else:
-        yield from _iter_symbols_nm(binary, find_local_symbol)
 
+# XXX need tests:
+# * iter_symbols
 
-#############################
-# binary format (e.g. ELF)
+NM_KINDS = {
+        'b': Symbol.KIND.VARIABLE,  # uninitialized
+        'd': Symbol.KIND.VARIABLE,  # initialized
+        #'g': Symbol.KIND.VARIABLE,  # uninitialized
+        #'s': Symbol.KIND.VARIABLE,  # initialized
+        't': Symbol.KIND.FUNCTION,
+        }
 
 SPECIAL_SYMBOLS = {
+        # binary format (e.g. ELF)
         '__bss_start',
         '__data_start',
         '__dso_handle',
@@ -63,29 +41,23 @@ def _is_special_symbol(name):
     return False
 
 
-#############################
-# "nm"
-
-NM_KINDS = {
-        'b': Symbol.KIND.VARIABLE,  # uninitialized
-        'd': Symbol.KIND.VARIABLE,  # initialized
-        #'g': Symbol.KIND.VARIABLE,  # uninitialized
-        #'s': Symbol.KIND.VARIABLE,  # initialized
-        't': Symbol.KIND.FUNCTION,
-        }
-
+def iter_symbols(binfile, *,
+                 nm=None,
+                 handle_id=None,
+                 _which=shutil.which,
+                 _run=util.run_cmd,
+                 ):
+    """Yield a Symbol for each relevant entry reported by the "nm" command."""
+    if nm is None:
+        nm = _which('nm')
+        if not nm:
+            raise NotImplementedError
+    if handle_id is None:
+        handle_id = info.ID
 
-def _iter_symbols_nm(binary, find_local_symbol=None,
-                     *,
-                     _which=shutil.which,
-                     _run=util.run_cmd,
-                     ):
-    nm = _which('nm')
-    if not nm:
-        raise NotImplementedError
     argv = [nm,
             '--line-numbers',
-            binary,
+            binfile,
             ]
     try:
         output = _run(argv)
@@ -95,23 +67,20 @@ def _iter_symbols_nm(binary, find_local_symbol=None,
             raise NotImplementedError
         raise
     for line in output.splitlines():
-        (name, kind, external, filename, funcname, vartype,
-         ) = _parse_nm_line(line,
-                            _find_local_symbol=find_local_symbol,
-                            )
+        (name, kind, external, filename, funcname,
+         ) = _parse_nm_line(line)
         if kind != Symbol.KIND.VARIABLE:
             continue
         elif _is_special_symbol(name):
             continue
-        assert vartype is None
         yield Symbol(
-                id=(filename, funcname, name),
+                id=handle_id(filename, funcname, name),
                 kind=kind,
                 external=external,
                 )
 
 
-def _parse_nm_line(line, *, _find_local_symbol=None):
+def _parse_nm_line(line):
     _origline = line
     _, _, line = line.partition(' ')  # strip off the address
     line = line.strip()
@@ -128,18 +97,9 @@ def _parse_nm_line(line, *, _find_local_symbol=None):
     else:
         filename = info.UNKNOWN
 
-    vartype = None
     name, islocal = _parse_nm_name(name, kind)
-    if islocal:
-        funcname = info.UNKNOWN
-        if _find_local_symbol is not None:
-            filename, funcname, vartype = _find_local_symbol(name)
-            filename = filename or info.UNKNOWN
-            funcname = funcname or info.UNKNOWN
-    else:
-        funcname = None
-        # XXX fine filename and vartype?
-    return name, kind, external, filename, funcname, vartype
+    funcname = info.UNKNOWN if islocal else None
+    return name, kind, external, filename, funcname
 
 
 def _parse_nm_name(name, kind):
diff --git a/Tools/c-analyzer/c_analyzer/symbols/find.py b/Tools/c-analyzer/c_analyzer/symbols/find.py
new file mode 100644
index 0000000000000..85646523f7a60
--- /dev/null
+++ b/Tools/c-analyzer/c_analyzer/symbols/find.py
@@ -0,0 +1,175 @@
+import os
+import os.path
+import shutil
+
+from ..common import files
+from ..common.info import UNKNOWN, ID
+from ..parser import find as p_find
+
+from . import _nm
+from .info import Symbol
+
+# XXX need tests:
+# * get_resolver()
+# * get_resolver_from_dirs()
+# * symbol()
+# * symbols()
+# * variables()
+
+
+def _resolve_known(symbol, knownvars):
+    for varid in knownvars:
+        if symbol.match(varid):
+            break
+    else:
+        return None
+    return knownvars.pop(varid)
+
+
+def get_resolver(filenames=None, known=None, *,
+                 handle_var,
+                 check_filename=None,
+                 perfilecache=None,
+                 preprocessed=False,
+                 _from_source=p_find.variable_from_id,
+                 ):
+    """Return a "resolver" func for the given known vars/types and filenames.
+
+    "handle_var" is a callable that takes (ID, decl) and returns a
+    Variable.  Variable.from_id is a suitable callable.
+
+    The returned func takes a single Symbol and returns a corresponding
+    Variable.  If the symbol was located then the variable will be
+    valid, populated with the corresponding information.  Otherwise None
+    is returned.
+    """
+    knownvars = (known or {}).get('variables')
+    if knownvars:
+        knownvars = dict(knownvars)  # a copy
+        if filenames:
+            if check_filename is None:
+                filenames = list(filenames)
+                def check_filename(filename):
+                    return filename in filenames
+            def resolve(symbol):
+                # XXX Check "found" instead?
+                if not check_filename(symbol.filename):
+                    return None
+                found = _resolve_known(symbol, knownvars)
+                if found is None:
+                    #return None
+                    varid, decl = _from_source(symbol, filenames,
+                                               perfilecache=perfilecache,
+                                               preprocessed=preprocessed,
+                                               )
+                    found = handle_var(varid, decl)
+                return found
+        else:
+            def resolve(symbol):
+                return _resolve_known(symbol, knownvars)
+    elif filenames:
+        def resolve(symbol):
+            varid, decl = _from_source(symbol, filenames,
+                                       perfilecache=perfilecache,
+                                       preprocessed=preprocessed,
+                                       )
+            return handle_var(varid, decl)
+    else:
+        def resolve(symbol):
+            return None
+    return resolve
+
+
+def get_resolver_from_dirs(dirnames, known=None, *,
+                           handle_var,
+                           suffixes=('.c',),
+                           perfilecache=None,
+                           preprocessed=False,
+                           _iter_files=files.iter_files_by_suffix,
+                           _get_resolver=get_resolver,
+                           ):
+    """Return a "resolver" func for the given known vars/types and filenames.
+
+    "dirnames" should be absolute paths.  If not then they will be
+    resolved relative to CWD.
+
+    See get_resolver().
+    """
+    dirnames = [d if d.endswith(os.path.sep) else d + os.path.sep
+                for d in dirnames]
+    filenames = _iter_files(dirnames, suffixes)
+    def check_filename(filename):
+        for dirname in dirnames:
+            if filename.startswith(dirname):
+                return True
+        else:
+            return False
+    return _get_resolver(filenames, known,
+                         handle_var=handle_var,
+                         check_filename=check_filename,
+                         perfilecache=perfilecache,
+                         preprocessed=preprocessed,
+                         )
+
+
+def symbol(symbol, filenames, known=None, *,
+           perfilecache=None,
+           preprocessed=False,
+           handle_id=None,
+           _get_resolver=get_resolver,
+           ):
+    """Return a Variable for the one matching the given symbol.
+
+    "symbol" can be one of several objects:
+
+    * Symbol - use the contained info
+    * name (str) - look for a global variable with that name
+    * (filename, name) - look for named global in file
+    * (filename, funcname, name) - look for named local in file
+
+    A name is always required.  If the filename is None, "", or
+    "UNKNOWN" then all files will be searched.  If the funcname is
+    "" or "UNKNOWN" then only local variables will be searched for.
+    """
+    resolve = _get_resolver(known, filenames,
+                            handle_id=handle_id,
+                            perfilecache=perfilecache,
+                            preprocessed=preprocessed,
+                            )
+    return resolve(symbol)
+
+
+def _get_platform_tool():
+    if os.name == 'nt':
+        # XXX Support this.
+        raise NotImplementedError
+    elif nm := shutil.which('nm'):
+        return lambda b, hi: _nm.iter_symbols(b, nm=nm, handle_id=hi)
+    else:
+        raise NotImplementedError
+
+
+def symbols(binfile, *,
+            handle_id=None,
+            _file_exists=os.path.exists,
+            _get_platform_tool=_get_platform_tool,
+            ):
+    """Yield a Symbol for each one found in the binary."""
+    if not _file_exists(binfile):
+        raise Exception('executable missing (need to build it first?)')
+
+    _iter_symbols = _get_platform_tool()
+    yield from _iter_symbols(binfile, handle_id)
+
+
+def variables(binfile, *,
+              resolve,
+              handle_id=None,
+              _iter_symbols=symbols,
+              ):
+    """Yield (Variable, Symbol) for each found symbol."""
+    for symbol in _iter_symbols(binfile, handle_id=handle_id):
+        if symbol.kind != Symbol.KIND.VARIABLE:
+            continue
+        var = resolve(symbol) or None
+        yield var, symbol
diff --git a/Tools/c-analyzer/c_symbols/info.py b/Tools/c-analyzer/c_analyzer/symbols/info.py
similarity index 93%
rename from Tools/c-analyzer/c_symbols/info.py
rename to Tools/c-analyzer/c_analyzer/symbols/info.py
index f6ed52c8f0714..96a251abb7c7f 100644
--- a/Tools/c-analyzer/c_symbols/info.py
+++ b/Tools/c-analyzer/c_analyzer/symbols/info.py
@@ -1,7 +1,7 @@
 from collections import namedtuple
 
-from c_analyzer_common.info import ID
-from c_analyzer_common.util import classonly, _NTBase
+from c_analyzer.common.info import ID
+from c_analyzer.common.util import classonly, _NTBase
 
 
 class Symbol(_NTBase, namedtuple('Symbol', 'id kind external')):
diff --git a/Tools/c-analyzer/c_analyzer/variables/__init__.py b/Tools/c-analyzer/c_analyzer/variables/__init__.py
new file mode 100644
index 0000000000000..e69de29bb2d1d
diff --git a/Tools/c-analyzer/c_analyzer/variables/find.py b/Tools/c-analyzer/c_analyzer/variables/find.py
new file mode 100644
index 0000000000000..3fe7284fc00a7
--- /dev/null
+++ b/Tools/c-analyzer/c_analyzer/variables/find.py
@@ -0,0 +1,75 @@
+from ..common import files
+from ..common.info import UNKNOWN
+from ..parser import (
+        find as p_find,
+        )
+from ..symbols import (
+        info as s_info,
+        find as s_find,
+        )
+from .info import Variable
+
+# XXX need tests:
+# * vars_from_source
+
+
+def _remove_cached(cache, var):
+    if not cache:
+        return
+    try:
+        cached = cache[var.filename]
+        cached.remove(var)
+    except (KeyError, IndexError):
+        pass
+
+
+def vars_from_binary(binfile, *,
+                     known=None,
+                     filenames=None,
+                     handle_id=None,
+                     check_filename=None,
+                     handle_var=Variable.from_id,
+                     _iter_vars=s_find.variables,
+                     _get_symbol_resolver=s_find.get_resolver,
+                     ):
+    """Yield a Variable for each found Symbol.
+
+    Details are filled in from the given "known" variables and types.
+    """
+    cache = {}
+    resolve = _get_symbol_resolver(filenames, known,
+                                   handle_var=handle_var,
+                                   check_filename=check_filename,
+                                   perfilecache=cache,
+                                   )
+    for var, symbol in _iter_vars(binfile,
+                                  resolve=resolve,
+                                  handle_id=handle_id,
+                                  ):
+        if var is None:
+            var = Variable(symbol.id, UNKNOWN, UNKNOWN)
+        yield var
+        _remove_cached(cache, var)
+
+
+def vars_from_source(filenames, *,
+                     preprocessed=None,
+                     known=None,
+                     handle_id=None,
+                     handle_var=Variable.from_id,
+                     iter_vars=p_find.variables,
+                     ):
+    """Yield a Variable for each declaration in the raw source code.
+
+    Details are filled in from the given "known" variables and types.
+    """
+    cache = {}
+    for varid, decl in iter_vars(filenames or (),
+                                 perfilecache=cache,
+                                 preprocessed=preprocessed,
+                                 known=known,
+                                 handle_id=handle_id,
+                                 ):
+        var = handle_var(varid, decl)
+        yield var
+        _remove_cached(cache, var)
diff --git a/Tools/c-analyzer/c_parser/info.py b/Tools/c-analyzer/c_analyzer/variables/info.py
similarity index 61%
rename from Tools/c-analyzer/c_parser/info.py
rename to Tools/c-analyzer/c_analyzer/variables/info.py
index a4e32d75eed73..336a523c7a2db 100644
--- a/Tools/c-analyzer/c_parser/info.py
+++ b/Tools/c-analyzer/c_analyzer/variables/info.py
@@ -1,8 +1,7 @@
 from collections import namedtuple
-import re
 
-from c_analyzer_common import info, util
-from c_analyzer_common.util import classonly, _NTBase
+from ..common.info import ID, UNKNOWN
+from ..common.util import classonly, _NTBase
 
 
 def normalize_vartype(vartype):
@@ -16,26 +15,7 @@ def normalize_vartype(vartype):
     return str(vartype)
 
 
-def extract_storage(decl, *, isfunc=False):
-    """Return (storage, vartype) based on the given declaration.
-
-    The default storage is "implicit" or "local".
-    """
-    if decl == info.UNKNOWN:
-        return decl, decl
-    if decl.startswith('static '):
-        return 'static', decl
-        #return 'static', decl.partition(' ')[2].strip()
-    elif decl.startswith('extern '):
-        return 'extern', decl
-        #return 'extern', decl.partition(' ')[2].strip()
-    elif re.match('.*\b(static|extern)\b', decl):
-        raise NotImplementedError
-    elif isfunc:
-        return 'local', decl
-    else:
-        return 'implicit', decl
-
+# XXX Variable.vartype -> decl (Declaration).
 
 class Variable(_NTBase,
                namedtuple('Variable', 'id storage vartype')):
@@ -52,16 +32,23 @@ class Variable(_NTBase,
 
     @classonly
     def from_parts(cls, filename, funcname, name, decl, storage=None):
+        varid = ID(filename, funcname, name)
         if storage is None:
-            storage, decl = extract_storage(decl, isfunc=funcname)
-        id = info.ID(filename, funcname, name)
-        self = cls(id, storage, decl)
+            self = cls.from_id(varid, decl)
+        else:
+            self = cls(varid, storage, decl)
         return self
 
+    @classonly
+    def from_id(cls, varid, decl):
+        from ..parser.declarations import extract_storage
+        storage = extract_storage(decl, infunc=varid.funcname)
+        return cls(varid, storage, decl)
+
     def __new__(cls, id, storage, vartype):
         self = super().__new__(
                 cls,
-                id=info.ID.from_raw(id),
+                id=ID.from_raw(id),
                 storage=str(storage) if storage else None,
                 vartype=normalize_vartype(vartype) if vartype else None,
                 )
@@ -77,10 +64,10 @@ def _validate_id(self):
         if not self.id:
             raise TypeError('missing id')
 
-        if not self.filename or self.filename == info.UNKNOWN:
+        if not self.filename or self.filename == UNKNOWN:
             raise TypeError(f'id missing filename ({self.id})')
 
-        if self.funcname and self.funcname == info.UNKNOWN:
+        if self.funcname and self.funcname == UNKNOWN:
             raise TypeError(f'id missing funcname ({self.id})')
 
         self.id.validate()
@@ -89,12 +76,12 @@ def validate(self):
         """Fail if the object is invalid (i.e. init with bad data)."""
         self._validate_id()
 
-        if self.storage is None or self.storage == info.UNKNOWN:
+        if self.storage is None or self.storage == UNKNOWN:
             raise TypeError('missing storage')
         elif self.storage not in self.STORAGE:
             raise ValueError(f'unsupported storage {self.storage:r}')
 
-        if self.vartype is None or self.vartype == info.UNKNOWN:
+        if self.vartype is None or self.vartype == UNKNOWN:
             raise TypeError('missing vartype')
 
     @property
diff --git a/Tools/c-analyzer/c_analyzer/variables/known.py b/Tools/c-analyzer/c_analyzer/variables/known.py
new file mode 100644
index 0000000000000..aa2934a069e16
--- /dev/null
+++ b/Tools/c-analyzer/c_analyzer/variables/known.py
@@ -0,0 +1,91 @@
+import csv
+
+from ..common.info import ID, UNKNOWN
+from ..common.util import read_tsv
+from .info import Variable
+
+
+# XXX need tests:
+# * read_file()
+# * look_up_variable()
+
+
+COLUMNS = ('filename', 'funcname', 'name', 'kind', 'declaration')
+HEADER = '\t'.join(COLUMNS)
+
+
+def read_file(infile, *,
+              _read_tsv=read_tsv,
+              ):
+    """Yield (kind, id, decl) for each row in the data file.
+
+    The caller is responsible for validating each row.
+    """
+    for row in _read_tsv(infile, HEADER):
+        filename, funcname, name, kind, declaration = row
+        if not funcname or funcname == '-':
+            funcname = None
+        id = ID(filename, funcname, name)
+        yield kind, id, declaration
+
+
+def from_file(infile, *,
+              handle_var=Variable.from_id,
+              _read_file=read_file,
+              ):
+    """Return the info for known declarations in the given file."""
+    known = {
+        'variables': {},
+        #'types': {},
+        #'constants': {},
+        #'macros': {},
+        }
+    for kind, id, decl in _read_file(infile):
+        if kind == 'variable':
+            values = known['variables']
+            value = handle_var(id, decl)
+        else:
+            raise ValueError(f'unsupported kind in row {row}')
+        value.validate()
+        values[id] = value
+    return known
+
+
+def look_up_variable(varid, knownvars, *,
+                     match_files=(lambda f1, f2: f1 == f2),
+                     ):
+    """Return the known Variable matching the given ID.
+
+    "knownvars" is a mapping of ID to Variable.
+
+    "match_files" is used to verify if two filenames point to
+    the same file.
+
+    If no match is found then None is returned.
+    """
+    if not knownvars:
+        return None
+
+    if varid.funcname == UNKNOWN:
+        if not varid.filename or varid.filename == UNKNOWN:
+            for varid in knownvars:
+                if not varid.funcname:
+                    continue
+                if varid.name == varid.name:
+                    return knownvars[varid]
+            else:
+                return None
+        else:
+            for varid in knownvars:
+                if not varid.funcname:
+                    continue
+                if not match_files(varid.filename, varid.filename):
+                    continue
+                if varid.name == varid.name:
+                    return knownvars[varid]
+            else:
+                return None
+    elif not varid.filename or varid.filename == UNKNOWN:
+        raise NotImplementedError
+    else:
+        return knownvars.get(varid.id)
diff --git a/Tools/c-analyzer/c_analyzer_common/__init__.py b/Tools/c-analyzer/c_analyzer_common/__init__.py
deleted file mode 100644
index 888b16ff41d10..0000000000000
--- a/Tools/c-analyzer/c_analyzer_common/__init__.py
+++ /dev/null
@@ -1,19 +0,0 @@
-import os.path
-
-
-PKG_ROOT = os.path.dirname(__file__)
-DATA_DIR = os.path.dirname(PKG_ROOT)
-REPO_ROOT = os.path.dirname(
-        os.path.dirname(DATA_DIR))
-
-SOURCE_DIRS = [os.path.join(REPO_ROOT, name) for name in [
-        'Include',
-        'Python',
-        'Parser',
-        'Objects',
-        'Modules',
-        ]]
-
-
-# Clean up the namespace.
-del os
diff --git a/Tools/c-analyzer/c_analyzer_common/info.py b/Tools/c-analyzer/c_analyzer_common/info.py
deleted file mode 100644
index e217380406457..0000000000000
--- a/Tools/c-analyzer/c_analyzer_common/info.py
+++ /dev/null
@@ -1,69 +0,0 @@
-from collections import namedtuple
-import re
-
-from .util import classonly, _NTBase
-
-
-UNKNOWN = '???'
-
-NAME_RE = re.compile(r'^([a-zA-Z]|_\w*[a-zA-Z]\w*|[a-zA-Z]\w*)$')
-
-
-class ID(_NTBase, namedtuple('ID', 'filename funcname name')):
-    """A unique ID for a single symbol or declaration."""
-
-    __slots__ = ()
-    # XXX Add optional conditions (tuple of strings) field.
-    #conditions = Slot()
-
-    @classonly
-    def from_raw(cls, raw):
-        if not raw:
-            return None
-        if isinstance(raw, str):
-            return cls(None, None, raw)
-        try:
-            name, = raw
-            filename = None
-        except ValueError:
-            try:
-                filename, name = raw
-            except ValueError:
-                return super().from_raw(raw)
-        return cls(filename, None, name)
-
-    def __new__(cls, filename, funcname, name):
-        self = super().__new__(
-                cls,
-                filename=str(filename) if filename else None,
-                funcname=str(funcname) if funcname else None,
-                name=str(name) if name else None,
-                )
-        #cls.conditions.set(self, tuple(str(s) if s else None
-        #                               for s in conditions or ()))
-        return self
-
-    def validate(self):
-        """Fail if the object is invalid (i.e. init with bad data)."""
-        if not self.name:
-            raise TypeError('missing name')
-        else:
-            if not NAME_RE.match(self.name):
-                raise ValueError(
-                        f'name must be an identifier, got {self.name!r}')
-
-        # Symbols from a binary might not have filename/funcname info.
-
-        if self.funcname:
-            if not self.filename:
-                raise TypeError('missing filename')
-            if not NAME_RE.match(self.funcname) and self.funcname != UNKNOWN:
-                raise ValueError(
-                        f'name must be an identifier, got {self.funcname!r}')
-
-        # XXX Require the filename (at least UNKONWN)?
-        # XXX Check the filename?
-
-    @property
-    def islocal(self):
-        return self.funcname is not None
diff --git a/Tools/c-analyzer/c_analyzer_common/known.py b/Tools/c-analyzer/c_analyzer_common/known.py
deleted file mode 100644
index dec1e1d2e0927..0000000000000
--- a/Tools/c-analyzer/c_analyzer_common/known.py
+++ /dev/null
@@ -1,74 +0,0 @@
-import csv
-import os.path
-
-from c_parser.info import Variable
-
-from . import DATA_DIR
-from .info import ID, UNKNOWN
-from .util import read_tsv
-
-
-DATA_FILE = os.path.join(DATA_DIR, 'known.tsv')
-
-COLUMNS = ('filename', 'funcname', 'name', 'kind', 'declaration')
-HEADER = '\t'.join(COLUMNS)
-
-
-# XXX need tests:
-# * from_file()
-
-def from_file(infile, *,
-              _read_tsv=read_tsv,
-              ):
-    """Return the info for known declarations in the given file."""
-    known = {
-        'variables': {},
-        #'types': {},
-        #'constants': {},
-        #'macros': {},
-        }
-    for row in _read_tsv(infile, HEADER):
-        filename, funcname, name, kind, declaration = row
-        if not funcname or funcname == '-':
-            funcname = None
-        id = ID(filename, funcname, name)
-        if kind == 'variable':
-            values = known['variables']
-            if funcname:
-                storage = _get_storage(declaration) or 'local'
-            else:
-                storage = _get_storage(declaration) or 'implicit'
-            value = Variable(id, storage, declaration)
-        else:
-            raise ValueError(f'unsupported kind in row {row}')
-        value.validate()
-#        if value.name == 'id' and declaration == UNKNOWN:
-#            # None of these are variables.
-#            declaration = 'int id';
-#        else:
-#            value.validate()
-        values[id] = value
-    return known
-
-
-def _get_storage(decl):
-    # statics
-    if decl.startswith('static '):
-        return 'static'
-    if decl.startswith(('Py_LOCAL(', 'Py_LOCAL_INLINE(')):
-        return 'static'
-    if decl.startswith(('_Py_IDENTIFIER(', '_Py_static_string(')):
-        return 'static'
-    if decl.startswith('PyDoc_VAR('):
-        return 'static'
-    if decl.startswith(('SLOT1BINFULL(', 'SLOT1BIN(')):
-        return 'static'
-    if decl.startswith('WRAP_METHOD('):
-        return 'static'
-    # public extern
-    if decl.startswith('extern '):
-        return 'extern'
-    if decl.startswith('PyAPI_DATA('):
-        return 'extern'
-    # implicit or local
-    return None
diff --git a/Tools/c-analyzer/c_globals/find.py b/Tools/c-analyzer/c_globals/find.py
deleted file mode 100644
index a51b947cbdf10..0000000000000
--- a/Tools/c-analyzer/c_globals/find.py
+++ /dev/null
@@ -1,95 +0,0 @@
-from c_analyzer_common import SOURCE_DIRS
-from c_analyzer_common.info import UNKNOWN
-from c_symbols import (
-        info as s_info,
-        binary as b_symbols,
-        source as s_symbols,
-        resolve,
-        )
-from c_parser import info, declarations
-
-
-# XXX needs tests:
-# * iter_variables
-
-def globals_from_binary(binfile=b_symbols.PYTHON, *,
-                        knownvars=None,
-                        dirnames=None,
-                        _iter_symbols=b_symbols.iter_symbols,
-                        _resolve=resolve.symbols_to_variables,
-                        _get_symbol_resolver=resolve.get_resolver,
-                        ):
-    """Yield a Variable for each found Symbol.
-
-    Details are filled in from the given "known" variables and types.
-    """
-    symbols = _iter_symbols(binfile, find_local_symbol=None)
-    #symbols = list(symbols)
-    for variable in _resolve(symbols,
-                             resolve=_get_symbol_resolver(knownvars, dirnames),
-                             ):
-        # Skip each non-global variable (unless we couldn't find it).
-        # XXX Drop the "UNKNOWN" condition?
-        if not variable.isglobal and variable.vartype != UNKNOWN:
-            continue
-        yield variable
-
-
-def globals_from_declarations(dirnames=SOURCE_DIRS, *,
-                              known=None,
-                              ):
-    """Yield a Variable for each found declaration.
-
-    Details are filled in from the given "known" variables and types.
-    """
-    raise NotImplementedError
-
-
-def iter_variables(kind='platform', *,
-                   known=None,
-                   dirnames=None,
-                   _resolve_symbols=resolve.symbols_to_variables,
-                   _get_symbol_resolver=resolve.get_resolver,
-                   _symbols_from_binary=b_symbols.iter_symbols,
-                   _symbols_from_source=s_symbols.iter_symbols,
-                   _iter_raw=declarations.iter_all,
-                   _iter_preprocessed=declarations.iter_preprocessed,
-                   ):
-    """Yield a Variable for each one found (e.g. in files)."""
-    kind = kind or 'platform'
-
-    if kind == 'symbols':
-        knownvars = (known or {}).get('variables')
-        yield from _resolve_symbols(
-                _symbols_from_source(dirnames, known),
-                resolve=_get_symbol_resolver(knownvars, dirnames),
-                )
-    elif kind == 'platform':
-        knownvars = (known or {}).get('variables')
-        yield from _resolve_symbols(
-                _symbols_from_binary(find_local_symbol=None),
-                resolve=_get_symbol_resolver(knownvars, dirnames),
-                )
-    elif kind == 'declarations':
-        for decl in _iter_raw(dirnames):
-            if not isinstance(decl, info.Variable):
-                continue
-            yield decl
-    elif kind == 'preprocessed':
-        for decl in _iter_preprocessed(dirnames):
-            if not isinstance(decl, info.Variable):
-                continue
-            yield decl
-    else:
-        raise ValueError(f'unsupported kind {kind!r}')
-
-
-def globals(dirnames, known, *,
-            kind=None,  # Use the default.
-            _iter_variables=iter_variables,
-            ):
-    """Return a list of (StaticVar, <supported>) for each found global var."""
-    for found in _iter_variables(kind, known=known, dirnames=dirnames):
-        if not found.isglobal:
-            continue
-        yield found
diff --git a/Tools/c-analyzer/c_globals/show.py b/Tools/c-analyzer/c_globals/show.py
deleted file mode 100644
index f4298b17b6788..0000000000000
--- a/Tools/c-analyzer/c_globals/show.py
+++ /dev/null
@@ -1,16 +0,0 @@
-
-def basic(globals, *,
-          _print=print):
-    """Print each row simply."""
-    for variable in globals:
-        if variable.funcname:
-            line = f'{variable.filename}:{variable.funcname}():{variable.name}'
-        else:
-            line = f'{variable.filename}:{variable.name}'
-        vartype = variable.vartype
-        #if vartype.startswith('static '):
-        #    vartype = vartype.partition(' ')[2]
-        #else:
-        #    vartype = '=' + vartype
-        line = f'{line:<64} {vartype}'
-        _print(line)
diff --git a/Tools/c-analyzer/c_symbols/resolve.py b/Tools/c-analyzer/c_symbols/resolve.py
deleted file mode 100644
index 56210cefeb826..0000000000000
--- a/Tools/c-analyzer/c_symbols/resolve.py
+++ /dev/null
@@ -1,147 +0,0 @@
-import os.path
-
-from c_analyzer_common import files
-from c_analyzer_common.info import UNKNOWN
-from c_parser import declarations, info
-from .info import Symbol
-from .source import _find_symbol
-
-
-# XXX need tests:
-# * look_up_known_symbol()
-# * symbol_from_source()
-# * get_resolver()
-# * symbols_to_variables()
-
-def look_up_known_symbol(symbol, knownvars, *,
-                         match_files=(lambda f1, f2: f1 == f2),
-                         ):
-    """Return the known variable matching the given symbol.
-
-    "knownvars" is a mapping of common.ID to parser.Variable.
-
-    "match_files" is used to verify if two filenames point to
-    the same file.
-    """
-    if not knownvars:
-        return None
-
-    if symbol.funcname == UNKNOWN:
-        if not symbol.filename or symbol.filename == UNKNOWN:
-            for varid in knownvars:
-                if not varid.funcname:
-                    continue
-                if varid.name == symbol.name:
-                    return knownvars[varid]
-            else:
-                return None
-        else:
-            for varid in knownvars:
-                if not varid.funcname:
-                    continue
-                if not match_files(varid.filename, symbol.filename):
-                    continue
-                if varid.name == symbol.name:
-                    return knownvars[varid]
-            else:
-                return None
-    elif not symbol.filename or symbol.filename == UNKNOWN:
-        raise NotImplementedError
-    else:
-        return knownvars.get(symbol.id)
-
-
-def find_in_source(symbol, dirnames, *,
-                   _perfilecache={},
-                   _find_symbol=_find_symbol,
-                   _iter_files=files.iter_files_by_suffix,
-                   ):
-    """Return the Variable matching the given Symbol.
-
-    If there is no match then return None.
-    """
-    if symbol.filename and symbol.filename != UNKNOWN:
-        filenames = [symbol.filename]
-    else:
-        filenames = _iter_files(dirnames, ('.c', '.h'))
-
-    if symbol.funcname and symbol.funcname != UNKNOWN:
-        raise NotImplementedError
-
-    (filename, funcname, decl
-     ) = _find_symbol(symbol.name, filenames, _perfilecache)
-    if filename == UNKNOWN:
-        return None
-    return info.Variable.from_parts(filename, funcname, symbol.name, decl)
-
-
-def get_resolver(knownvars=None, dirnames=None, *,
-                 _look_up_known=look_up_known_symbol,
-                 _from_source=find_in_source,
-                 ):
-    """Return a "resolver" func for the given known vars and dirnames.
-
-    The func takes a single Symbol and returns a corresponding Variable.
-    If the symbol was located then the variable will be valid, populated
-    with the corresponding information.  Otherwise None is returned.
-    """
-    if knownvars:
-        knownvars = dict(knownvars)  # a copy
-        def resolve_known(symbol):
-            found = _look_up_known(symbol, knownvars)
-            if found is None:
-                return None
-            elif symbol.funcname == UNKNOWN:
-                knownvars.pop(found.id)
-            elif not symbol.filename or symbol.filename == UNKNOWN:
-                knownvars.pop(found.id)
-            return found
-        if dirnames:
-            def resolve(symbol):
-                found = resolve_known(symbol)
-                if found is None:
-                    return None
-                    #return _from_source(symbol, dirnames)
-                else:
-                    for dirname in dirnames:
-                        if not dirname.endswith(os.path.sep):
-                            dirname += os.path.sep
-                        if found.filename.startswith(dirname):
-                            break
-                    else:
-                        return None
-                    return found
-        else:
-            resolve = resolve_known
-    elif dirnames:
-        def resolve(symbol):
-            return _from_source(symbol, dirnames)
-    else:
-        def resolve(symbol):
-            return None
-    return resolve
-
-
-def symbols_to_variables(symbols, *,
-                         resolve=(lambda s: look_up_known_symbol(s, None)),
-                         ):
-    """Yield the variable the matches each given symbol.
-
-    Use get_resolver() for a "resolve" func to use.
-    """
-    for symbol in symbols:
-        if isinstance(symbol, info.Variable):
-            # XXX validate?
-            yield symbol
-            continue
-        if symbol.kind != Symbol.KIND.VARIABLE:
-            continue
-        resolved = resolve(symbol)
-        if resolved is None:
-            #raise NotImplementedError(symbol)
-            resolved = info.Variable(
-                    id=symbol.id,
-                    storage=UNKNOWN,
-                    vartype=UNKNOWN,
-                    )
-        yield resolved
diff --git a/Tools/c-analyzer/c_symbols/source.py b/Tools/c-analyzer/c_symbols/source.py
deleted file mode 100644
index a7248104c94ca..0000000000000
--- a/Tools/c-analyzer/c_symbols/source.py
+++ /dev/null
@@ -1,58 +0,0 @@
-from c_analyzer_common import files
-from c_analyzer_common.info import UNKNOWN
-from c_parser import declarations
-
-
-# XXX need tests:
-# * find_symbol()
-
-def find_symbol(name, dirnames, *,
-                _perfilecache,
-                _iter_files=files.iter_files_by_suffix,
-                **kwargs
-                ):
-    """Return (filename, funcname, vartype) for the matching Symbol."""
-    filenames = _iter_files(dirnames, ('.c', '.h'))
-    return _find_symbol(name, filenames, _perfilecache, **kwargs)
-
-
-def _get_symbols(filename, *,
-                 _iter_variables=declarations.iter_variables,
-                 ):
-    """Return the list of Symbols found in the given file."""
-    symbols = {}
-    for funcname, name, vartype in _iter_variables(filename):
-        if not funcname:
-            continue
-        try:
-            instances = symbols[name]
-        except KeyError:
-            instances = symbols[name] = []
-        instances.append((funcname, vartype))
-    return symbols
-
-
-def _find_symbol(name, filenames, _perfilecache, *,
-                _get_local_symbols=_get_symbols,
-                ):
-    for filename in filenames:
-        try:
-            symbols = _perfilecache[filename]
-        except KeyError:
-            symbols = _perfilecache[filename] = _get_local_symbols(filename)
-
-        try:
-            instances = symbols[name]
-        except KeyError:
-            continue
-
-        funcname, vartype = instances.pop(0)
-        if not instances:
-            symbols.pop(name)
-        return filename, funcname, vartype
-    else:
-        return UNKNOWN, UNKNOWN, UNKNOWN
-
-
-def iter_symbols():
-    raise NotImplementedError
diff --git a/Tools/c-analyzer/c_globals/README b/Tools/c-analyzer/cpython/README
similarity index 100%
rename from Tools/c-analyzer/c_globals/README
rename to Tools/c-analyzer/cpython/README
diff --git a/Tools/c-analyzer/cpython/__init__.py b/Tools/c-analyzer/cpython/__init__.py
new file mode 100644
index 0000000000000..ae45b424e3cc8
--- /dev/null
+++ b/Tools/c-analyzer/cpython/__init__.py
@@ -0,0 +1,29 @@
+import os.path
+import sys
+
+
+TOOL_ROOT = os.path.abspath(
+        os.path.dirname(  # c-analyzer/
+            os.path.dirname(__file__)))  # cpython/
+DATA_DIR = TOOL_ROOT
+REPO_ROOT = (
+        os.path.dirname(  # ..
+            os.path.dirname(TOOL_ROOT)))  # Tools/
+
+INCLUDE_DIRS = [os.path.join(REPO_ROOT, name) for name in [
+        'Include',
+        ]]
+SOURCE_DIRS = [os.path.join(REPO_ROOT, name) for name in [
+        'Python',
+        'Parser',
+        'Objects',
+        'Modules',
+        ]]
+
+#PYTHON = os.path.join(REPO_ROOT, 'python')
+PYTHON = sys.executable
+
+
+# Clean up the namespace.
+del sys
+del os
diff --git a/Tools/c-analyzer/c_globals/__main__.py b/Tools/c-analyzer/cpython/__main__.py
similarity index 68%
rename from Tools/c-analyzer/c_globals/__main__.py
rename to Tools/c-analyzer/cpython/__main__.py
index 9570fb6a14c4e..6b0f9bcb9687f 100644
--- a/Tools/c-analyzer/c_globals/__main__.py
+++ b/Tools/c-analyzer/cpython/__main__.py
@@ -1,42 +1,42 @@
 import argparse
-import os.path
 import re
 import sys
 
-from c_analyzer_common import SOURCE_DIRS, REPO_ROOT
-from c_analyzer_common.info import UNKNOWN
-from c_analyzer_common.known import (
+from c_analyzer.common import show
+from c_analyzer.common.info import UNKNOWN
+
+from . import SOURCE_DIRS
+from .find import supported_vars
+from .known import (
     from_file as known_from_file,
     DATA_FILE as KNOWN_FILE,
     )
-from . import find, show
-from .supported import is_supported, ignored_from_file, IGNORED_FILE, _is_object
+from .supported import IGNORED_FILE
 
 
-def _match_unused_global(variable, knownvars, used):
-    found = []
-    for varid in knownvars:
-        if varid in used:
-            continue
-        if varid.funcname is not None:
-            continue
-        if varid.name != variable.name:
-            continue
-        if variable.filename and variable.filename != UNKNOWN:
-            if variable.filename == varid.filename:
+def _check_results(unknown, knownvars, used):
+    def _match_unused_global(variable):
+        found = []
+        for varid in knownvars:
+            if varid in used:
+                continue
+            if varid.funcname is not None:
+                continue
+            if varid.name != variable.name:
+                continue
+            if variable.filename and variable.filename != UNKNOWN:
+                if variable.filename == varid.filename:
+                    found.append(varid)
+            else:
                 found.append(varid)
-        else:
-            found.append(varid)
-    return found
-
+        return found
 
-def _check_results(unknown, knownvars, used):
     badknown = set()
     for variable in sorted(unknown):
         msg = None
         if variable.funcname != UNKNOWN:
             msg = f'could not find global symbol {variable.id}'
-        elif m := _match_unused_global(variable, knownvars, used):
+        elif m := _match_unused_global(variable):
             assert isinstance(m, list)
             badknown.update(m)
         elif variable.name in ('completed', 'id'):  # XXX Figure out where these variables are.
@@ -65,32 +65,29 @@ def _check_results(unknown, knownvars, used):
         raise Exception('could not find all symbols')
 
 
-def _find_globals(dirnames, known, ignored):
-    if dirnames == SOURCE_DIRS:
-        dirnames = [os.path.relpath(d, REPO_ROOT) for d in dirnames]
-
-    ignored = ignored_from_file(ignored)
-    known = known_from_file(known)
+# XXX Move this check to its own command.
+def cmd_check_cache(cmd, *,
+                    known=KNOWN_FILE,
+                    ignored=IGNORED_FILE,
+                    _known_from_file=known_from_file,
+                    _find=supported_vars,
+                    ):
+    known = _known_from_file(known)
 
     used = set()
     unknown = set()
-    knownvars = (known or {}).get('variables')
-    for variable in find.globals_from_binary(knownvars=knownvars,
-                                             dirnames=dirnames):
-    #for variable in find.globals(dirnames, known, kind='platform'):
-        if variable.vartype == UNKNOWN:
-            unknown.add(variable)
+    for var, supported in _find(known=known, ignored=ignored):
+        if supported is None:
+            unknown.add(var)
             continue
-        yield variable, is_supported(variable, ignored, known)
-        used.add(variable.id)
-
-    #_check_results(unknown, knownvars, used)
+        used.add(var.id)
+    _check_results(unknown, known['variables'], used)
 
 
-def cmd_check(cmd, dirs=SOURCE_DIRS, *,
-              ignored=IGNORED_FILE,
+def cmd_check(cmd, *,
               known=KNOWN_FILE,
-              _find=_find_globals,
+              ignored=IGNORED_FILE,
+              _find=supported_vars,
               _show=show.basic,
               _print=print,
               ):
@@ -100,7 +97,11 @@ def cmd_check(cmd, dirs=SOURCE_DIRS, *,
     In the failure case, the list of unsupported variables
     will be printed out.
     """
-    unsupported = [v for v, s in _find(dirs, known, ignored) if not s]
+    unsupported = []
+    for var, supported in _find(known=known, ignored=ignored):
+        if not supported:
+            unsupported.append(var)
+
     if not unsupported:
         #_print('okay')
         return
@@ -112,11 +113,11 @@ def cmd_check(cmd, dirs=SOURCE_DIRS, *,
     sys.exit(1)
 
 
-def cmd_show(cmd, dirs=SOURCE_DIRS, *,
-             ignored=IGNORED_FILE,
+def cmd_show(cmd, *,
              known=KNOWN_FILE,
+             ignored=IGNORED_FILE,
              skip_objects=False,
-              _find=_find_globals,
+              _find=supported_vars,
              _show=show.basic,
              _print=print,
              ):
@@ -127,10 +128,12 @@ def cmd_show(cmd, dirs=SOURCE_DIRS, *,
     """
     allsupported = []
     allunsupported = []
-    for found, supported in _find(dirs, known, ignored):
-        if skip_objects:  # XXX Support proper filters instead.
-            if _is_object(found.vartype):
-                continue
+    for found, supported in _find(known=known,
+                                  ignored=ignored,
+                                  skip_objects=skip_objects,
+                                  ):
+        if supported is None:
+            continue
         (allsupported if supported else allunsupported
          ).append(found)
 
@@ -165,9 +168,9 @@ def parse_args(prog=PROG, argv=sys.argv[1:], *, _fail=None):
     common.add_argument('--known', metavar='FILE',
                         default=KNOWN_FILE,
                         help='path to file that lists known types')
-    common.add_argument('dirs', metavar='DIR', nargs='*',
-                        default=SOURCE_DIRS,
-                        help='a directory to check')
+    #common.add_argument('dirs', metavar='DIR', nargs='*',
+    #                    default=SOURCE_DIRS,
+    #                    help='a directory to check')
 
     parser = argparse.ArgumentParser(
             prog=prog,
diff --git a/Tools/c-analyzer/c_analyzer_common/_generate.py b/Tools/c-analyzer/cpython/_generate.py
similarity index 97%
rename from Tools/c-analyzer/c_analyzer_common/_generate.py
rename to Tools/c-analyzer/cpython/_generate.py
index 9b2fc9edb5c82..4c340acf99e1c 100644
--- a/Tools/c-analyzer/c_analyzer_common/_generate.py
+++ b/Tools/c-analyzer/cpython/_generate.py
@@ -1,15 +1,16 @@
 # The code here consists of hacks for pre-populating the known.tsv file.
 
-from c_parser.preprocessor import _iter_clean_lines
-from c_parser.naive import (
+from c_analyzer.parser.preprocessor import _iter_clean_lines
+from c_analyzer.parser.naive import (
         iter_variables, parse_variable_declaration, find_variables,
         )
-from c_parser.info import Variable
+from c_analyzer.common.known import HEADER as KNOWN_HEADER
+from c_analyzer.common.info import UNKNOWN, ID
+from c_analyzer.variables import Variable
+from c_analyzer.util import write_tsv
 
 from . import SOURCE_DIRS, REPO_ROOT
-from .known import DATA_FILE as KNOWN_FILE, HEADER as KNOWN_HEADER
-from .info import UNKNOWN, ID
-from .util import write_tsv
+from .known import DATA_FILE as KNOWN_FILE
 from .files import iter_cpython_files
 
 
diff --git a/Tools/c-analyzer/cpython/files.py b/Tools/c-analyzer/cpython/files.py
new file mode 100644
index 0000000000000..543097af7bcd5
--- /dev/null
+++ b/Tools/c-analyzer/cpython/files.py
@@ -0,0 +1,29 @@
+from c_analyzer.common.files import (
+        C_SOURCE_SUFFIXES, walk_tree, iter_files_by_suffix,
+        )
+
+from . import SOURCE_DIRS, REPO_ROOT
+
+# XXX need tests:
+# * iter_files()
+
+
+def iter_files(*,
+               walk=walk_tree,
+               _files=iter_files_by_suffix,
+               ):
+    """Yield each file in the tree for each of the given directory names."""
+    excludedtrees = [
+        os.path.join('Include', 'cpython', ''),
+        ]
+    def is_excluded(filename):
+        for root in excludedtrees:
+            if filename.startswith(root):
+                return True
+        return False
+    for filename in _files(SOURCE_DIRS, C_SOURCE_SUFFIXES, REPO_ROOT,
+                           walk=walk,
+                           ):
+        if is_excluded(filename):
+            continue
+        yield filename
diff --git a/Tools/c-analyzer/cpython/find.py b/Tools/c-analyzer/cpython/find.py
new file mode 100644
index 0000000000000..a7bc0b477b839
--- /dev/null
+++ b/Tools/c-analyzer/cpython/find.py
@@ -0,0 +1,101 @@
+import os.path
+
+from c_analyzer.common import files
+from c_analyzer.common.info import UNKNOWN, ID
+from c_analyzer.variables import find as _common
+
+from . import SOURCE_DIRS, PYTHON, REPO_ROOT
+from .known import (
+    from_file as known_from_file,
+    DATA_FILE as KNOWN_FILE,
+    )
+from .supported import (
+        ignored_from_file, IGNORED_FILE, is_supported, _is_object,
+        )
+
+# XXX need tests:
+# * vars_from_binary()
+# * vars_from_source()
+# * supported_vars()
+
+
+def _handle_id(filename, funcname, name, *,
+               _relpath=os.path.relpath,
+               ):
+    filename = _relpath(filename, REPO_ROOT)
+    return ID(filename, funcname, name)
+
+
+def vars_from_binary(*,
+                     known=KNOWN_FILE,
+                     _known_from_file=known_from_file,
+                     _iter_files=files.iter_files_by_suffix,
+                     _iter_vars=_common.vars_from_binary,
+                     ):
+    """Yield a Variable for each found Symbol.
+
+    Details are filled in from the given "known" variables and types.
+    """
+    if isinstance(known, str):
+        known = _known_from_file(known)
+    dirnames = SOURCE_DIRS
+    suffixes = ('.c',)
+    filenames = _iter_files(dirnames, suffixes)
+    # XXX For now we only use known variables (no source lookup).
+    filenames = None
+    yield from _iter_vars(PYTHON,
+                          known=known,
+                          filenames=filenames,
+                          handle_id=_handle_id,
+                          check_filename=(lambda n: True),
+                          )
+
+
+def vars_from_source(*,
+                     preprocessed=None,
+                     known=KNOWN_FILE,
+                     _known_from_file=known_from_file,
+                     _iter_files=files.iter_files_by_suffix,
+                     _iter_vars=_common.vars_from_source,
+                     ):
+    """Yield a Variable for each declaration in the raw source code.
+
+    Details are filled in from the given "known" variables and types.
+    """
+    if isinstance(known, str):
+        known = _known_from_file(known)
+    dirnames = SOURCE_DIRS
+    suffixes = ('.c',)
+    filenames = _iter_files(dirnames, suffixes)
+    yield from _iter_vars(filenames,
+                          preprocessed=preprocessed,
+                          known=known,
+                          handle_id=_handle_id,
+                          )
+
+
+def supported_vars(*,
+                   known=KNOWN_FILE,
+                   ignored=IGNORED_FILE,
+                   skip_objects=False,
+                   _known_from_file=known_from_file,
+                   _ignored_from_file=ignored_from_file,
+                   _iter_vars=vars_from_binary,
+                   _is_supported=is_supported,
+                   ):
+    """Yield (var, is supported) for each found variable."""
+    if isinstance(known, str):
+        known = _known_from_file(known)
+    if isinstance(ignored, str):
+        ignored = _ignored_from_file(ignored)
+
+    for var in _iter_vars(known=known):
+        if not var.isglobal:
+            continue
+        elif var.vartype == UNKNOWN:
+            yield var, None
+        # XXX Support proper filters instead.
+        elif skip_objects and _is_object(found.vartype):
+            continue
+        else:
+            yield var, _is_supported(var, ignored, known)
diff --git a/Tools/c-analyzer/cpython/known.py b/Tools/c-analyzer/cpython/known.py
new file mode 100644
index 0000000000000..c3cc2c06026ce
--- /dev/null
+++ b/Tools/c-analyzer/cpython/known.py
@@ -0,0 +1,66 @@
+import csv
+import os.path
+
+from c_analyzer.parser.declarations import extract_storage
+from c_analyzer.variables import known as _common
+from c_analyzer.variables.info import Variable
+
+from . import DATA_DIR
+
+
+# XXX need tests:
+# * from_file()
+# * look_up_variable()
+
+
+DATA_FILE = os.path.join(DATA_DIR, 'known.tsv')
+
+
+def _get_storage(decl, infunc):
+    # statics
+    if decl.startswith(('Py_LOCAL(', 'Py_LOCAL_INLINE(')):
+        return 'static'
+    if decl.startswith(('_Py_IDENTIFIER(', '_Py_static_string(')):
+        return 'static'
+    if decl.startswith('PyDoc_VAR('):
+        return 'static'
+    if decl.startswith(('SLOT1BINFULL(', 'SLOT1BIN(')):
+        return 'static'
+    if decl.startswith('WRAP_METHOD('):
+        return 'static'
+    # public extern
+    if decl.startswith('PyAPI_DATA('):
+        return 'extern'
+    # Fall back to the normal handler.
+    return extract_storage(decl, infunc=infunc)
+
+
+def _handle_var(varid, decl):
+#    if varid.name == 'id' and decl == UNKNOWN:
+#        # None of these are variables.
+#        decl = 'int id';
+    storage = _get_storage(decl, varid.funcname)
+    return Variable(varid, storage, decl)
+
+
+def from_file(infile=DATA_FILE, *,
+              _from_file=_common.from_file,
+              _handle_var=_handle_var,
+              ):
+    """Return the info for known declarations in the given file."""
+    return _from_file(infile, handle_var=_handle_var)
+
+
+def look_up_variable(varid, knownvars, *,
+                     _lookup=_common.look_up_variable,
+                     ):
+    """Return the known variable matching the given ID.
+
+    "knownvars" is a mapping of ID to Variable.
+
+    "match_files" is used to verify if two filenames point to
+    the same file.
+
+    If no match is found then None is returned.
+    """
+    return _lookup(varid, knownvars)
diff --git a/Tools/c-analyzer/c_globals/supported.py b/Tools/c-analyzer/cpython/supported.py
similarity index 97%
rename from Tools/c-analyzer/c_globals/supported.py
rename to Tools/c-analyzer/cpython/supported.py
index d185daa2463bb..18786eefd8ded 100644
--- a/Tools/c-analyzer/c_globals/supported.py
+++ b/Tools/c-analyzer/cpython/supported.py
@@ -1,9 +1,13 @@
 import os.path
 import re
 
-from c_analyzer_common import DATA_DIR
-from c_analyzer_common.info import ID
-from c_analyzer_common.util import read_tsv, write_tsv
+from c_analyzer.common.info import ID
+from c_analyzer.common.util import read_tsv, write_tsv
+
+from . import DATA_DIR
+
+# XXX need tests:
+# * generate / script
 
 
 IGNORED_FILE = os.path.join(DATA_DIR, 'ignored.tsv')
@@ -379,11 +383,12 @@ def _generate_ignored_file(variables, filename=None, *,
 
 
 if __name__ == '__main__':
-    from c_analyzer_common import SOURCE_DIRS
-    from c_analyzer_common.known import (
+    from cpython import SOURCE_DIRS
+    from cpython.known import (
         from_file as known_from_file,
         DATA_FILE as KNOWN_FILE,
         )
+    # XXX This is wrong!
     from . import find
     known = known_from_file(KNOWN_FILE)
     knownvars = (known or {}).get('variables')


From webhook-mailer at python.org  Sat Oct 19 05:35:53 2019
From: webhook-mailer at python.org (Ned Deily)
Date: Sat, 19 Oct 2019 09:35:53 -0000
Subject: [Python-checkins] Update build docs for macOS (GH-16844)
Message-ID: <mailman.69.1571477753.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/4ae38ba52ab1f28d183feffb835f45b9485b73f4
commit: 4ae38ba52ab1f28d183feffb835f45b9485b73f4
branch: 2.7
author: Ned Deily <nad at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-19T05:35:44-04:00
summary:

Update build docs for macOS (GH-16844)

files:
A Mac/BuildScript/README.rst
A Mac/README.rst
D Mac/BuildScript/README.txt
D Mac/README
M README

diff --git a/Mac/BuildScript/README.txt b/Mac/BuildScript/README.rst
similarity index 100%
rename from Mac/BuildScript/README.txt
rename to Mac/BuildScript/README.rst
diff --git a/Mac/README b/Mac/README.rst
similarity index 75%
rename from Mac/README
rename to Mac/README.rst
index 51e69ef6e86a3..8397e9c8e6831 100644
--- a/Mac/README
+++ b/Mac/README.rst
@@ -1,25 +1,23 @@
-=========================
-Python on Mac OS X README
-=========================
+======================
+Python on macOS README
+======================
 
 :Authors:
     Jack Jansen (2004-07),
     Ronald Oussoren (2010-04),
-    Ned Deily (2014-05)
+    Ned Deily (2012-06)
 
-:Version: 2.7.15
-
-This document provides a quick overview of some Mac OS X specific features in
+This document provides a quick overview of some macOS specific features in
 the Python distribution.
 
-OS X specific arguments to configure
-====================================
+macOS specific arguments to configure
+=====================================
 
 * ``--enable-framework[=DIR]``
 
   If this argument is specified the build will create a Python.framework rather
   than a traditional Unix install. See the section
-  _`Building and using a framework-based Python on Mac OS X` for more
+  _`Building and using a framework-based Python on macOS` for more
   information on frameworks.
 
   If the optional directory argument is specified the framework is installed
@@ -43,41 +41,51 @@ OS X specific arguments to configure
   Create a universal binary build of Python. This can be used with both
   regular and framework builds.
 
-  The optional argument specifies which OS X SDK should be used to perform the
-  build.  If xcodebuild is available and configured, this defaults to
-  the Xcode default MacOS X SDK, otherwise ``/Developer/SDKs/MacOSX.10.4u.sdk``
-  if available or ``/`` if not.  When building on OS X 10.5 or later, you can
-  specify ``/`` to use the installed system headers rather than an SDK.  As of
-  OS X 10.9, you should install the optional system headers from the Command
-  Line Tools component using ``xcode-select``::
-
-     $ sudo xcode-select --install
-
-  See the section _`Building and using a universal binary of Python on Mac OS X`
-  for more information.
+  The optional argument specifies which macOS SDK should be used to perform the
+  build.  In most cases on current systems, you do not need to specify PATH or
+  you can just use ``/``; the default MacOSX SDK for the active Xcode or Command
+  Line Tools developer directory will be used.  See the macOS ``xcrun`` man page
+  for more information.  Current versions of macOS and Xcode no longer install
+  system header files in their traditional locations, like ``/usr/include`` and
+  ``/System/Library/Frameworks``; instead they are found within a MacOSX SDK.
+  The Apple-supplied build tools handle this transparently and current
+  versiona of Python now handle this as well.  So it is no longer necessary,
+  and since macOS 10.14, no longer possible to force the installation of system
+  headers with ``xcode-select``.
 
 * ``--with-universal-archs=VALUE``
 
   Specify the kind of universal binary that should be created. This option is
   only valid when ``--enable-universalsdk`` is specified.  The default is
-  ``32-bit`` if a building with a SDK that supports PPC, otherwise defaults
-  to ``intel``.
+  ``32-bit`` if building with a SDK that supports PPC, otherwise defaults
+  to ``intel``.  Note that ``intel`` means a universal build of both 32-bit
+  and 64-bit binaries and that may not be what you want; for example,
+  as of macOS 10.15 Catalina, 32-bit execution is no longer supported by
+  the operating system.  Thus it is best to either explicitly specify
+  values for ``--with-universal-archs``:
+
+      ``--enable-universalsdk --with-universal-archs=intel-64``
 
+  or avoid using either.
 
-Building and using a universal binary of Python on Mac OS X
-===========================================================
+
+Building and using a universal binary of Python on macOS
+========================================================
 
 1. What is a universal binary
 -----------------------------
 
 A universal binary build of Python contains object code for more than one
-CPU architecture.  A universal OS X executable file or library combines the
+CPU architecture.  A universal macOS executable file or library combines the
 architecture-specific code into one file and can therefore run at native
 speed on all supported architectures.  Universal files were introduced in
-OS X 10.4 to add support for Intel-based Macs to the existing PowerPC (PPC)
-machines.  In OS X 10.5 support was extended to 64-bit Intel and 64-bit PPC
+macOS 10.4 to add support for Intel-based Macs to the existing PowerPC (PPC)
+machines.  In macOS 10.5 support was extended to 64-bit Intel and 64-bit PPC
 architectures.  It is possible to build Python with various combinations
-of architectures depending on the build tools and OS X version in use.
+of architectures depending on the build tools and macOS version in use.
+Note that PPC support was removed in macOS 10.7 and 32-bit Intel support
+was removed in macOS 10.15.  So currently as of macOS 10.15, macOS only
+supports one execution architecture, 64-bit Intel (``x86_64``).
 
 2. How do I build a universal binary
 ------------------------------------
@@ -90,14 +98,14 @@ flag to configure::
   $ make install
 
 This flag can be used with a framework build of python, but also with a classic
-unix build. Universal builds were first supported with OS X 10.4 with Xcode 2.1
-and the 10.4u SDK.  Starting with Xcode 3 and OS X 10.5, more configurations are
+unix build. Universal builds were first supported with macOS 10.4 with Xcode 2.1
+and the 10.4u SDK.  Starting with Xcode 3 and macOS 10.5, more configurations are
 available.
 
 In general, universal builds depend on specific features provided by the
 Apple-supplied compilers and other build tools included in Apple's Xcode
-development tools.  You should install Xcode and the command line tools
-component appropriate for the OS X release you are running on.  See the
+development tools.  You should install Xcode or the command line tools
+component appropriate for the macOS release you are running on.  See the
 Python Developer's Guide (https://devguide.python.org/setup/)
 for more information.
 
@@ -127,28 +135,29 @@ values are available:
   * ``all``:      ``ppc``, ``ppc64``, ``i386``, ``x86_64``
 
 To build a universal binary that includes a 64-bit architecture, you must build
-on a system running OS X 10.5 or later.  The ``all`` and ``64-bit`` flavors can
+on a system running macOS 10.5 or later.  The ``all`` and ``64-bit`` flavors can
 only be built with a 10.5 SDK because ``ppc64`` support was only included with
-OS X 10.5.  Although legacy ``ppc`` support was included with Xcode 3 on OS X
-10.6, it was removed in Xcode 4, versions of which were released on OS X 10.6
-and which is the standard for OS X 10.7.  To summarize, the
+macOS 10.5.  Although legacy ``ppc`` support was included with Xcode 3 on macOS
+10.6, it was removed in Xcode 4, versions of which were released on macOS 10.6
+and which is the standard for macOS 10.7.  To summarize, the
 following combinations of SDKs and universal-archs flavors are available:
 
   * 10.4u SDK with Xcode 2 supports ``32-bit`` only
 
   * 10.5 SDK with Xcode 3.1.x supports all flavors
 
-  * 10.6 SDK with Xcode 3.2.x supports ``intel``, ``3-way``, and ``32-bit``
+  * 10.6 SDK with Xcode 3.2.x supports ``intel``, ``intel-32``,
+    ``intel-64``, ``3-way``, and ``32-bit``
 
-  * 10.6 SDK with Xcode 4 supports ``intel`` only
+  * 10.6 SDK with Xcode 4 supports ``intel``, ``intel-32``, and ``intel-64``
 
-  * 10.7 and 10.8 SDKs with Xcode 4 support ``intel`` only
+  * 10.7 through 10.14 SDKs support ``intel``, ``intel-32``, and ``intel-64``
 
-  * 10.8 and 10.9 SDKs with Xcode 5 support ``intel`` only
+  * 10.15 and later SDKs support ``intel-64`` only
 
 The makefile for a framework build will also install ``python2.7-32``
 binaries when the universal architecture includes at least one 32-bit
-architecture (that is, for all flavors but ``64-bit``).
+architecture (that is, for all flavors but ``64-bit`` and ``intel-64``).
 
 Running a specific architecture
 ...............................
@@ -161,11 +170,6 @@ Or to explicitly run in 32-bit mode, regardless of the machine hardware::
 
    $ arch -i386 -ppc python
 
-NOTE: When you're using a framework install of Python this requires at least
-Python 2.7 or 3.2, in earlier versions the python (and pythonw) commands are
-wrapper tools that execute the real interpreter without ensuring that the
-real interpreter runs with the same architecture.
-
 Using ``arch`` is not a perfect solution as the selected architecture will
 not automatically carry through to subprocesses launched by programs and tests
 under that Python.  If you want to ensure that Python interpreters launched in
@@ -173,8 +177,8 @@ subprocesses also run in 32-bit-mode if the main interpreter does, use
 a ``python2.7-32`` binary and use the value of ``sys.executable`` as the
 ``subprocess`` ``Popen`` executable value.
 
-Building and using a framework-based Python on Mac OS X.
-========================================================
+Building and using a framework-based Python on macOS
+====================================================
 
 
 1. Why would I want a framework Python instead of a normal static Python?
@@ -182,18 +186,18 @@ Building and using a framework-based Python on Mac OS X.
 
 The main reason is because you want to create GUI programs in Python. With the
 exception of X11/XDarwin-based GUI toolkits all GUI programs need to be run
-from a Mac OS X application bundle (".app").
+from a macOS application bundle (".app").
 
 While it is technically possible to create a .app without using frameworks you
 will have to do the work yourself if you really want this.
 
 A second reason for using frameworks is that they put Python-related items in
 only two places: "/Library/Framework/Python.framework" and
-"/Applications/Python <VERSION>" where ``<VERSION>`` can be e.g. "3.4",
+"/Applications/Python <VERSION>" where ``<VERSION>`` can be e.g. "3.8",
 "2.7", etc.  This simplifies matters for users installing
 Python from a binary distribution if they want to get rid of it again. Moreover,
-due to the way frameworks work, a user without admin privileges can install a
-binary distribution in his or her home directory without recompilation.
+due to the way frameworks work, usera without admin privileges can install a
+binary distribution in their home directory without recompilation.
 
 2. How does a framework Python differ from a normal static Python?
 ------------------------------------------------------------------
@@ -207,12 +211,12 @@ Versions/Current and you will see the familiar bin and lib directories.
 3. Do I need extra packages?
 ----------------------------
 
-Yes, probably.  If you want Tkinter support you need to get the OS X AquaTk
-distribution, this is installed by default on Mac OS X 10.4 or later.  Be
-aware, though, that the Cocoa-based AquaTk's supplied starting with OS X
+Yes, probably.  If you want Tkinter support you need to get the macOS AquaTk
+distribution, this is installed by default on macOS 10.4 or later.  Be
+aware, though, that the Cocoa-based AquaTk's supplied starting with macOS
 10.6 have proven to be unstable.  If possible, you should consider
-installing a newer version before building on OS X 10.6 or later, such as
-the ActiveTcl 8.5.  See http://www.python.org/download/mac/tcltk/.  If you
+installing a newer version before building on macOS 10.6 or later, such as
+the ActiveTcl 8.6.  See http://www.python.org/download/mac/tcltk/.  If you
 are building with an SDK, ensure that the newer Tcl and Tk frameworks are
 seen in the SDK's ``Library/Frameworks`` directory; you may need to
 manually create symlinks to their installed location, ``/Library/Frameworks``.
@@ -223,7 +227,7 @@ If you want Cocoa you need to get PyObjC.
 -------------------------------------
 
 This directory contains a Makefile that will create a couple of python-related
-applications (full-blown OS X .app applications, that is) in
+applications (full-blown macOS .app applications, that is) in
 "/Applications/Python <VERSION>", and a hidden helper application Python.app
 inside the Python.framework, and unix tools including "python" into
 /usr/local/bin.  In addition it has a target "installmacsubtree" that installs
@@ -291,7 +295,7 @@ How do I create a binary distribution?
 Download and unpack the source release from http://www.python.org/download/.
 Go to the directory ``Mac/BuildScript``. There you will find a script
 ``build-installer.py`` that does all the work. This will download and build
-a number of 3rd-party libaries, configures and builds a framework Python,
+a number of 3rd-party libraries, configures and builds a framework Python,
 installs it, creates the installer package files and then packs this in a
 DMG image.  The script also builds an HTML copy of the current Python
 documentation set for this release for inclusion in the framework.  The
@@ -299,18 +303,17 @@ installer package will create links to the documentation for use by IDLE,
 pydoc, shell users, and Finder user.
 
 The script will build a universal binary so you'll therefore have to run this
-script on Mac OS X 10.4 or later and with Xcode 2.1 or later installed.
+script on macOS 10.4 or later and with Xcode 2.1 or later installed.
 However, the Python build process itself has several build dependencies not
-available out of the box with OS X 10.4 so you may have to install
-additional software beyond what is provided with Xcode 2.  OS X 10.5
-provides a recent enough system Python (in ``/usr/bin``) to build
-the Python documentation set.  It should be possible to use SDKs and/or older
+available out of the box with macOS 10.4 so you may have to install
+additional software beyond what is provided with Xcode 2.
+It should be possible to use SDKs and/or older
 versions of Xcode to build installers that are compatible with older systems
 on a newer system but this may not be completely foolproof so the resulting
 executables, shared libraries, and ``.so`` bundles should be carefully
 examined and tested on all supported systems for proper dynamic linking
 dependencies.  It is safest to build the distribution on a system running the
-minimum OS X version supported.
+minimum macOS version supported.
 
 All of this is normally done completely isolated in /tmp/_py, so it does not
 use your normal build directory nor does it install into /.
@@ -344,7 +347,7 @@ Uninstalling a framework install, including the binary installer
 
 Uninstalling a framework can be done by manually removing all bits that got installed.
 That's true for both installations from source and installations using the binary installer.
-OS X does not provide a central uninstaller.
+macOS does not provide a central uninstaller.
 
 The main bit of a framework install is the framework itself, installed in
 ``/Library/Frameworks/Python.framework``. This can contain multiple versions
diff --git a/README b/README
index 9f03976cba5ec..a2b0543dcdc5a 100644
--- a/README
+++ b/README
@@ -582,19 +582,30 @@ Reliant UNIX: The thread support does not compile on Reliant UNIX, and
         platform as well.  This should be resolved in time for a
         future release.
 
-MacOSX: The tests will crash on both 10.1 and 10.2 with SEGV in
+macOS:  Building a complete Python installation requires the use of various
+        additional third-party libraries, depending on your build platform and
+        configure options.  Not all standard library modules are buildable or
+        useable on all platforms.  Refer to the "Install Dependencies" section
+        section of the "Developer Guide" for current detailed information on
+        dependencies for macOS:
+            https://devguide.python.org/setup/#install-dependencies
+
+        On macOS, there are additional configure and build options related
+        to macOS framework and universal builds.  Refer to Mac/README.rst.
+
+        The tests will crash on both 10.1 and 10.2 with SEGV in
         test_re and test_sre due to the small default stack size.  If
         you set the stack size to 2048 before doing a "make test" the
         failure can be avoided.  If you're using the tcsh or csh shells,
         use "limit stacksize 2048" and for the bash shell (the default
-        as of OSX 10.3), use "ulimit -s 2048".
+        as of macOS 10.3), use "ulimit -s 2048".
 
         On naked Darwin you may want to add the configure option
         "--disable-toolbox-glue" to disable the glue code for the Carbon
         interface modules. The modules themselves are currently only built
         if you add the --enable-framework option, see below.
 
-        On a clean OSX /usr/local does not exist. Do a
+        On a clean macOS /usr/local does not exist. Do a
         "sudo mkdir -m 775 /usr/local"
         before you do a make install. It is probably not a good idea to
         do "sudo make install" which installs everything as superuser,
@@ -605,20 +616,6 @@ MacOSX: The tests will crash on both 10.1 and 10.2 with SEGV in
         to install additional unix software. Disabling fink (remove all 
         references to /sw from your .profile or .login) should solve this.
 
-        You may want to try the configure option "--enable-framework"
-        which installs Python as a framework. The location can be set
-        as argument to the --enable-framework option (default
-        /Library/Frameworks). A framework install is probably needed if you
-        want to use any Aqua-based GUI toolkit (whether Tkinter, wxPython,
-        Carbon, Cocoa or anything else).
-
-        You may also want to try the configure option "--enable-universalsdk"
-        which builds Python as a universal binary with support for the 
-        i386 and PPC architetures. This requires Xcode 2.1 or later to build.
-
-        See Mac/README for more information on framework and 
-        universal builds.
-
 Cygwin: With recent (relative to the time of writing, 2001-12-19)
         Cygwin installations, there are problems with the interaction
         of dynamic linking and fork().  This manifests itself in build
@@ -949,11 +946,6 @@ Emacs found in Misc/python-mode.el.  (But then again, more recent
 versions of Emacs may already have it.)  Follow the instructions that
 came with Emacs for installation of site-specific files.
 
-On Mac OS X, if you have configured Python with --enable-framework, you
-should use "make frameworkinstall" to do the installation. Note that this
-installs the Python executable in a place that is not normally on your
-PATH, you may want to set up a symlink in /usr/local/bin.
-
 
 Installing multiple versions
 ----------------------------


From webhook-mailer at python.org  Sat Oct 19 13:52:10 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Sat, 19 Oct 2019 17:52:10 -0000
Subject: [Python-checkins] Update doc switcher list for 3.8.0 (GH-16809)
Message-ID: <mailman.70.1571507531.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/6c4f841d8c13efcc40459ace97754131f250e3a9
commit: 6c4f841d8c13efcc40459ace97754131f250e3a9
branch: 2.7
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-19T10:52:07-07:00
summary:

Update doc switcher list for 3.8.0 (GH-16809)

(cherry picked from commit 3f36043db22361500f52634f2b8de49dde0e7da9)

Co-authored-by: Ned Deily <nad at python.org>

files:
M Doc/tools/static/switchers.js

diff --git a/Doc/tools/static/switchers.js b/Doc/tools/static/switchers.js
index 7f995fef8083b..d4fc07d3ab032 100644
--- a/Doc/tools/static/switchers.js
+++ b/Doc/tools/static/switchers.js
@@ -11,7 +11,7 @@
 
   var all_versions = {
     '3.9': 'dev (3.9)',
-    '3.8': 'pre (3.8)',
+    '3.8': '3.8',
     '3.7': '3.7',
     '3.6': '3.6',
     '3.5': '3.5',


From webhook-mailer at python.org  Sat Oct 19 15:02:41 2019
From: webhook-mailer at python.org (Benjamin Peterson)
Date: Sat, 19 Oct 2019 19:02:41 -0000
Subject: [Python-checkins] Bump version to 2.7.17 final.
Message-ID: <mailman.71.1571511761.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/74ceb358c70d3183644271abcf711100c9695dbc
commit: 74ceb358c70d3183644271abcf711100c9695dbc
branch: 2.7
author: Benjamin Peterson <benjamin at python.org>
committer: Benjamin Peterson <benjamin at python.org>
date: 2019-10-19T11:37:52-07:00
summary:

Bump version to 2.7.17 final.

files:
M Include/patchlevel.h

diff --git a/Include/patchlevel.h b/Include/patchlevel.h
index de01a84abbdbe..4bfcdafcd2590 100644
--- a/Include/patchlevel.h
+++ b/Include/patchlevel.h
@@ -23,11 +23,11 @@
 #define PY_MAJOR_VERSION	2
 #define PY_MINOR_VERSION	7
 #define PY_MICRO_VERSION	17
-#define PY_RELEASE_LEVEL	PY_RELEASE_LEVEL_GAMMA
-#define PY_RELEASE_SERIAL	1
+#define PY_RELEASE_LEVEL	PY_RELEASE_LEVEL_FINAL
+#define PY_RELEASE_SERIAL	0
 
 /* Version as a string */
-#define PY_VERSION      	"2.7.17rc1"
+#define PY_VERSION      	"2.7.17"
 /*--end constants--*/
 
 /* Subversion Revision number of this file (not of the repository). Empty


From webhook-mailer at python.org  Sat Oct 19 15:22:24 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Sat, 19 Oct 2019 19:22:24 -0000
Subject: [Python-checkins] Remove doc reference to unmaitained Nose package
 (GH-16849)
Message-ID: <mailman.72.1571512946.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/88eeda6311a8e3bf57136da5f73c70bc91ad79f3
commit: 88eeda6311a8e3bf57136da5f73c70bc91ad79f3
branch: master
author: Jon Dufresne <jon.dufresne at gmail.com>
committer: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
date: 2019-10-19T12:22:20-07:00
summary:

Remove doc reference to unmaitained Nose package (GH-16849)



The Nose package is no longer maintained.

files:
M Doc/library/unittest.rst

diff --git a/Doc/library/unittest.rst b/Doc/library/unittest.rst
index 8c9affe9db952..9e66a2c1c220a 100644
--- a/Doc/library/unittest.rst
+++ b/Doc/library/unittest.rst
@@ -56,8 +56,8 @@ test runner
       Kent Beck's original paper on testing frameworks using the pattern shared
       by :mod:`unittest`.
 
-   `Nose <https://nose.readthedocs.io/>`_ and `pytest <https://docs.pytest.org/>`_
-      Third-party unittest frameworks with a lighter-weight syntax for writing
+   `pytest <https://docs.pytest.org/>`_
+      Third-party unittest framework with a lighter-weight syntax for writing
       tests.  For example, ``assert func(10) == 42``.
 
    `The Python Testing Tools Taxonomy <https://wiki.python.org/moin/PythonTestingToolsTaxonomy>`_


From webhook-mailer at python.org  Sat Oct 19 15:29:24 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Sat, 19 Oct 2019 19:29:24 -0000
Subject: [Python-checkins] Remove doc reference to unmaitained Nose package
 (GH-16849)
Message-ID: <mailman.73.1571513365.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/e945052e07bb7f3061b5fdb4735c892cdda3aefc
commit: e945052e07bb7f3061b5fdb4735c892cdda3aefc
branch: 3.7
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-19T12:29:20-07:00
summary:

Remove doc reference to unmaitained Nose package (GH-16849)


The Nose package is no longer maintained.
(cherry picked from commit 88eeda6311a8e3bf57136da5f73c70bc91ad79f3)

Co-authored-by: Jon Dufresne <jon.dufresne at gmail.com>

files:
M Doc/library/unittest.rst

diff --git a/Doc/library/unittest.rst b/Doc/library/unittest.rst
index bbe14299edfb7..0cd6918b1d253 100644
--- a/Doc/library/unittest.rst
+++ b/Doc/library/unittest.rst
@@ -56,8 +56,8 @@ test runner
       Kent Beck's original paper on testing frameworks using the pattern shared
       by :mod:`unittest`.
 
-   `Nose <https://nose.readthedocs.io/>`_ and `pytest <https://docs.pytest.org/>`_
-      Third-party unittest frameworks with a lighter-weight syntax for writing
+   `pytest <https://docs.pytest.org/>`_
+      Third-party unittest framework with a lighter-weight syntax for writing
       tests.  For example, ``assert func(10) == 42``.
 
    `The Python Testing Tools Taxonomy <https://wiki.python.org/moin/PythonTestingToolsTaxonomy>`_


From webhook-mailer at python.org  Sat Oct 19 15:30:31 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Sat, 19 Oct 2019 19:30:31 -0000
Subject: [Python-checkins] Remove doc reference to unmaitained Nose package
 (GH-16849)
Message-ID: <mailman.74.1571513433.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/bbd600a17b24155f0b36f2cd473e17b09029ade6
commit: bbd600a17b24155f0b36f2cd473e17b09029ade6
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-19T12:30:28-07:00
summary:

Remove doc reference to unmaitained Nose package (GH-16849)


The Nose package is no longer maintained.
(cherry picked from commit 88eeda6311a8e3bf57136da5f73c70bc91ad79f3)

Co-authored-by: Jon Dufresne <jon.dufresne at gmail.com>

files:
M Doc/library/unittest.rst

diff --git a/Doc/library/unittest.rst b/Doc/library/unittest.rst
index cbee94e52c0ae..eb6714a8c8ff9 100644
--- a/Doc/library/unittest.rst
+++ b/Doc/library/unittest.rst
@@ -56,8 +56,8 @@ test runner
       Kent Beck's original paper on testing frameworks using the pattern shared
       by :mod:`unittest`.
 
-   `Nose <https://nose.readthedocs.io/>`_ and `pytest <https://docs.pytest.org/>`_
-      Third-party unittest frameworks with a lighter-weight syntax for writing
+   `pytest <https://docs.pytest.org/>`_
+      Third-party unittest framework with a lighter-weight syntax for writing
       tests.  For example, ``assert func(10) == 42``.
 
    `The Python Testing Tools Taxonomy <https://wiki.python.org/moin/PythonTestingToolsTaxonomy>`_


From webhook-mailer at python.org  Sat Oct 19 16:01:12 2019
From: webhook-mailer at python.org (Pablo Galindo)
Date: Sat, 19 Oct 2019 20:01:12 -0000
Subject: [Python-checkins] bpo-38525: Fix a segmentation fault when using
 reverse iterators of empty dict (GH-16846)
Message-ID: <mailman.75.1571515273.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/24dc2f8c56697f9ee51a4887cf0814b6600c1815
commit: 24dc2f8c56697f9ee51a4887cf0814b6600c1815
branch: master
author: Dong-hee Na <donghee.na92 at gmail.com>
committer: Pablo Galindo <Pablogsal at gmail.com>
date: 2019-10-19T21:01:08+01:00
summary:

bpo-38525: Fix a segmentation fault when using reverse iterators of empty dict (GH-16846)

The reverse iterator for empty dictionaries was not handling correctly shared-key dictionaries.

files:
A Misc/NEWS.d/next/Core and Builtins/2019-10-20-00-36-18.bpo-38525.Vty1cA.rst
M Lib/test/test_dict.py
M Objects/dictobject.c

diff --git a/Lib/test/test_dict.py b/Lib/test/test_dict.py
index 13be857f7ab60..5b513765f7b08 100644
--- a/Lib/test/test_dict.py
+++ b/Lib/test/test_dict.py
@@ -1312,6 +1312,31 @@ def test_reversed(self):
         self.assertEqual(list(r), list('dcba'))
         self.assertRaises(StopIteration, next, r)
 
+    def test_reverse_iterator_for_empty_dict(self):
+        # bpo-38525: revered iterator should work properly
+
+        # empty dict is directly used for reference count test
+        self.assertEqual(list(reversed({})), [])
+        self.assertEqual(list(reversed({}.items())), [])
+        self.assertEqual(list(reversed({}.values())), [])
+        self.assertEqual(list(reversed({}.keys())), [])
+
+        # dict() and {} don't trigger the same code path
+        self.assertEqual(list(reversed(dict())), [])
+        self.assertEqual(list(reversed(dict().items())), [])
+        self.assertEqual(list(reversed(dict().values())), [])
+        self.assertEqual(list(reversed(dict().keys())), [])
+
+    def test_reverse_iterator_for_shared_shared_dicts(self):
+        class A:
+            def __init__(self, x, y):
+                if x: self.x = x
+                if y: self.y = y
+
+        self.assertEqual(list(reversed(A(1, 2).__dict__)), ['y', 'x'])
+        self.assertEqual(list(reversed(A(1, 0).__dict__)), ['x'])
+        self.assertEqual(list(reversed(A(0, 1).__dict__)), ['y'])
+
     def test_dict_copy_order(self):
         # bpo-34320
         od = collections.OrderedDict([('a', 1), ('b', 2)])
diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-10-20-00-36-18.bpo-38525.Vty1cA.rst b/Misc/NEWS.d/next/Core and Builtins/2019-10-20-00-36-18.bpo-38525.Vty1cA.rst
new file mode 100644
index 0000000000000..c74d143762253
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2019-10-20-00-36-18.bpo-38525.Vty1cA.rst	
@@ -0,0 +1,2 @@
+Fix a segmentation fault when using reverse iterators of empty ``dict`` objects.
+Patch by Dong-hee Na and Inada Naoki.
diff --git a/Objects/dictobject.c b/Objects/dictobject.c
index 64876e05191e2..5ac7bb102b211 100644
--- a/Objects/dictobject.c
+++ b/Objects/dictobject.c
@@ -3452,10 +3452,15 @@ dictiter_new(PyDictObject *dict, PyTypeObject *itertype)
     di->di_dict = dict;
     di->di_used = dict->ma_used;
     di->len = dict->ma_used;
-    if ((itertype == &PyDictRevIterKey_Type ||
+    if (itertype == &PyDictRevIterKey_Type ||
          itertype == &PyDictRevIterItem_Type ||
-         itertype == &PyDictRevIterValue_Type) && dict->ma_used) {
+         itertype == &PyDictRevIterValue_Type) {
+        if (dict->ma_values) {
+            di->di_pos = dict->ma_used - 1;
+        }
+        else {
             di->di_pos = dict->ma_keys->dk_nentries - 1;
+        }
     }
     else {
         di->di_pos = 0;


From webhook-mailer at python.org  Sat Oct 19 16:04:52 2019
From: webhook-mailer at python.org (Benjamin Peterson)
Date: Sat, 19 Oct 2019 20:04:52 -0000
Subject: [Python-checkins] 2.2.17+
Message-ID: <mailman.76.1571515493.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/b02f6929344ec3650d859d8e6182cce9d8e37c74
commit: b02f6929344ec3650d859d8e6182cce9d8e37c74
branch: 2.7
author: Benjamin Peterson <benjamin at python.org>
committer: Benjamin Peterson <benjamin at python.org>
date: 2019-10-19T13:03:22-07:00
summary:

2.2.17+

files:
M Include/patchlevel.h

diff --git a/Include/patchlevel.h b/Include/patchlevel.h
index 4bfcdafcd2590..c5a4c14b48f73 100644
--- a/Include/patchlevel.h
+++ b/Include/patchlevel.h
@@ -27,7 +27,7 @@
 #define PY_RELEASE_SERIAL	0
 
 /* Version as a string */
-#define PY_VERSION      	"2.7.17"
+#define PY_VERSION      	"2.7.17+"
 /*--end constants--*/
 
 /* Subversion Revision number of this file (not of the repository). Empty


From webhook-mailer at python.org  Sat Oct 19 16:20:59 2019
From: webhook-mailer at python.org (Miss Islington (bot))
Date: Sat, 19 Oct 2019 20:20:59 -0000
Subject: [Python-checkins] bpo-38525: Fix a segmentation fault when using
 reverse iterators of empty dict (GH-16846)
Message-ID: <mailman.77.1571516460.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/d73205d788a32148ba9a2beaa27badbd94ab65ff
commit: d73205d788a32148ba9a2beaa27badbd94ab65ff
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-19T13:20:51-07:00
summary:

bpo-38525: Fix a segmentation fault when using reverse iterators of empty dict (GH-16846)


The reverse iterator for empty dictionaries was not handling correctly shared-key dictionaries.
(cherry picked from commit 24dc2f8c56697f9ee51a4887cf0814b6600c1815)

Co-authored-by: Dong-hee Na <donghee.na92 at gmail.com>

files:
A Misc/NEWS.d/next/Core and Builtins/2019-10-20-00-36-18.bpo-38525.Vty1cA.rst
M Lib/test/test_dict.py
M Objects/dictobject.c

diff --git a/Lib/test/test_dict.py b/Lib/test/test_dict.py
index 13be857f7ab60..5b513765f7b08 100644
--- a/Lib/test/test_dict.py
+++ b/Lib/test/test_dict.py
@@ -1312,6 +1312,31 @@ def test_reversed(self):
         self.assertEqual(list(r), list('dcba'))
         self.assertRaises(StopIteration, next, r)
 
+    def test_reverse_iterator_for_empty_dict(self):
+        # bpo-38525: revered iterator should work properly
+
+        # empty dict is directly used for reference count test
+        self.assertEqual(list(reversed({})), [])
+        self.assertEqual(list(reversed({}.items())), [])
+        self.assertEqual(list(reversed({}.values())), [])
+        self.assertEqual(list(reversed({}.keys())), [])
+
+        # dict() and {} don't trigger the same code path
+        self.assertEqual(list(reversed(dict())), [])
+        self.assertEqual(list(reversed(dict().items())), [])
+        self.assertEqual(list(reversed(dict().values())), [])
+        self.assertEqual(list(reversed(dict().keys())), [])
+
+    def test_reverse_iterator_for_shared_shared_dicts(self):
+        class A:
+            def __init__(self, x, y):
+                if x: self.x = x
+                if y: self.y = y
+
+        self.assertEqual(list(reversed(A(1, 2).__dict__)), ['y', 'x'])
+        self.assertEqual(list(reversed(A(1, 0).__dict__)), ['x'])
+        self.assertEqual(list(reversed(A(0, 1).__dict__)), ['y'])
+
     def test_dict_copy_order(self):
         # bpo-34320
         od = collections.OrderedDict([('a', 1), ('b', 2)])
diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-10-20-00-36-18.bpo-38525.Vty1cA.rst b/Misc/NEWS.d/next/Core and Builtins/2019-10-20-00-36-18.bpo-38525.Vty1cA.rst
new file mode 100644
index 0000000000000..c74d143762253
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2019-10-20-00-36-18.bpo-38525.Vty1cA.rst	
@@ -0,0 +1,2 @@
+Fix a segmentation fault when using reverse iterators of empty ``dict`` objects.
+Patch by Dong-hee Na and Inada Naoki.
diff --git a/Objects/dictobject.c b/Objects/dictobject.c
index 5dad1b6d9208c..a0208be9246d9 100644
--- a/Objects/dictobject.c
+++ b/Objects/dictobject.c
@@ -3462,10 +3462,15 @@ dictiter_new(PyDictObject *dict, PyTypeObject *itertype)
     di->di_dict = dict;
     di->di_used = dict->ma_used;
     di->len = dict->ma_used;
-    if ((itertype == &PyDictRevIterKey_Type ||
+    if (itertype == &PyDictRevIterKey_Type ||
          itertype == &PyDictRevIterItem_Type ||
-         itertype == &PyDictRevIterValue_Type) && dict->ma_used) {
+         itertype == &PyDictRevIterValue_Type) {
+        if (dict->ma_values) {
+            di->di_pos = dict->ma_used - 1;
+        }
+        else {
             di->di_pos = dict->ma_keys->dk_nentries - 1;
+        }
     }
     else {
         di->di_pos = 0;


From webhook-mailer at python.org  Sat Oct 19 21:25:39 2019
From: webhook-mailer at python.org (Steve Dower)
Date: Sun, 20 Oct 2019 01:25:39 -0000
Subject: [Python-checkins] Work around Path.glob() issue when creating nuget
 package (GH-16855)
Message-ID: <mailman.78.1571534739.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/c9ed34f5ad6cd9e86175a52588f9aacd7bf844ae
commit: c9ed34f5ad6cd9e86175a52588f9aacd7bf844ae
branch: 2.7
author: Steve Dower <steve.dower at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-19T18:25:35-07:00
summary:

Work around Path.glob() issue when creating nuget package (GH-16855)

files:
M Tools/nuget/make_zip.py

diff --git a/Tools/nuget/make_zip.py b/Tools/nuget/make_zip.py
index 8844cacb096e9..efc06bc306528 100644
--- a/Tools/nuget/make_zip.py
+++ b/Tools/nuget/make_zip.py
@@ -103,7 +103,7 @@ def include_in_tools(p):
     ('Lib/', 'Lib', '**/*', include_in_lib),
     ('libs/', 'PCBuild/$arch', '*.lib', include_in_libs),
     ('Tools/', 'Tools', '**/*', include_in_tools),
-    ('/', '', 'LICENSE', None),
+    ('/', '', 'LICENSE*', None),
 ]
 
 EMBED_LAYOUT = [


From webhook-mailer at python.org  Sun Oct 20 13:19:51 2019
From: webhook-mailer at python.org (Raymond Hettinger)
Date: Sun, 20 Oct 2019 17:19:51 -0000
Subject: [Python-checkins] bpo-36321: Fix misspelled attribute name in
 namedtuple() (GH-16858)
Message-ID: <mailman.79.1571591992.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/58ccd201fa74287ca9293c03136fcf1e19800ef9
commit: 58ccd201fa74287ca9293c03136fcf1e19800ef9
branch: master
author: Raymond Hettinger <rhettinger at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-20T10:19:47-07:00
summary:

bpo-36321: Fix misspelled attribute name in namedtuple() (GH-16858)

files:
A Misc/NEWS.d/next/Library/2019-10-19-21-41-20.bpo-36321.CFlxfy.rst
M Lib/collections/__init__.py

diff --git a/Lib/collections/__init__.py b/Lib/collections/__init__.py
index 2fc71e2991ab6..cec6c9781a15e 100644
--- a/Lib/collections/__init__.py
+++ b/Lib/collections/__init__.py
@@ -425,8 +425,6 @@ def __getnewargs__(self):
         '__slots__': (),
         '_fields': field_names,
         '_field_defaults': field_defaults,
-        # alternate spelling for backward compatibility
-        '_fields_defaults': field_defaults,
         '__new__': __new__,
         '_make': _make,
         '_replace': _replace,
diff --git a/Misc/NEWS.d/next/Library/2019-10-19-21-41-20.bpo-36321.CFlxfy.rst b/Misc/NEWS.d/next/Library/2019-10-19-21-41-20.bpo-36321.CFlxfy.rst
new file mode 100644
index 0000000000000..cf51d71da17fa
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-10-19-21-41-20.bpo-36321.CFlxfy.rst
@@ -0,0 +1,2 @@
+Remove misspelled attribute.  The 3.8 changelog noted that this would be
+removed in 3.9.


From webhook-mailer at python.org  Sun Oct 20 14:25:26 2019
From: webhook-mailer at python.org (Raymond Hettinger)
Date: Sun, 20 Oct 2019 18:25:26 -0000
Subject: [Python-checkins] bpo-37759: More updates to Whatsnew 3.8 (GH-16854)
Message-ID: <mailman.80.1571595928.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/c93883c6afbd99929516764263fc49c3e996b10a
commit: c93883c6afbd99929516764263fc49c3e996b10a
branch: master
author: Raymond Hettinger <rhettinger at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-20T11:25:16-07:00
summary:

bpo-37759: More updates to Whatsnew 3.8 (GH-16854)

* math.perm() and math.comb()

* math.isqrt()

* Add singledispatchmethod()

* itertools.accumulate()

* Optional headers for xmlrpc.client.ServerProxy

* IDLE non-BMP characters

* import collections.abc directly

* @coroutine is deprecated

* pprint.pp()

* New options for object.__reduce__()

* DictReader no longer returns OrderedDicts

* "force" option for logging.basicConfig()

* Fix spelling

* cProfile context manager

* Various markup/grammar fixes from Kyle Stanley.
Other minor fixes as well.
Also, dedup the __reduce__ entry.

* Fix markup

* Fix grammar nits found by MS Word

files:
M Doc/whatsnew/3.8.rst

diff --git a/Doc/whatsnew/3.8.rst b/Doc/whatsnew/3.8.rst
index ed450051170a6..db40ce756287f 100644
--- a/Doc/whatsnew/3.8.rst
+++ b/Doc/whatsnew/3.8.rst
@@ -355,8 +355,8 @@ It is meant to formalize existing optimizations which were already done
 for various classes.
 Any extension type implementing a callable can use this protocol.
 
-This is currently provisional,
-the aim is to make it fully public in Python 3.9.
+This is currently provisional.
+The aim is to make it fully public in Python 3.9.
 
 See :pep:`590` for a full description.
 
@@ -445,7 +445,7 @@ Other Language Changes
   an instance of the subclass, rather than the base class. This also affects
   the return type of operations whose implementation (directly or indirectly)
   uses :class:`datetime.timedelta` arithmetic, such as
-  :meth:`datetime.datetime.astimezone`.
+  :meth:`~datetime.datetime.astimezone`.
   (Contributed by Paul Ganssle in :issue:`32417`.)
 
 * When the Python interpreter is interrupted by Ctrl-C (SIGINT) and the
@@ -529,6 +529,13 @@ Other Language Changes
 
   (Contributed by J?rn Heissler in :issue:`35224`.)
 
+* The :meth:`object.__reduce__` method can now return a tuple from two to
+  six elements long. Formerly, five was the limit.  The new, optional sixth
+  element is a callable with a ``(obj, state)`` signature.  This allows the
+  direct control over the state-updating behavior of a specific object.  If
+  not *None*, this callable will have priority over the object's
+  :meth:`~__setstate__` method.
+  (Contributed by Pierre Glaser and Olivier Grisel in :issue:`35900`.)
 
 New Modules
 ===========
@@ -579,8 +586,8 @@ The :func:`ast.parse` function has some new flags:
   comments" (returned for function definition AST nodes);
 
 * ``feature_version=(3, N)`` allows specifying an earlier Python 3
-  version. (For example, ``feature_version=(3, 4)`` will treat
-  ``async`` and ``await`` as non-reserved words.)
+  version.  For example, ``feature_version=(3, 4)`` will treat
+  :keyword:`async` and :keyword:`await` as non-reserved words.
 
 (Contributed by Guido van Rossum in :issue:`35766`.)
 
@@ -630,14 +637,39 @@ marked with the ``CO_COROUTINE`` flag may then be returned.
 collections
 -----------
 
-The :meth:`_asdict()` method for :func:`collections.namedtuple` now returns
-a :class:`dict` instead of a :class:`collections.OrderedDict`. This works because
-regular dicts have guaranteed ordering since Python 3.7. If the extra
-features of :class:`OrderedDict` are required, the suggested remediation is
-to cast the result to the desired type: ``OrderedDict(nt._asdict())``.
+The :meth:`~collections.somenamedtuple._asdict` method for
+:func:`collections.namedtuple` now returns a :class:`dict` instead of a
+:class:`collections.OrderedDict`. This works because regular dicts have
+guaranteed ordering since Python 3.7. If the extra features of
+:class:`OrderedDict` are required, the suggested remediation is to cast the
+result to the desired type: ``OrderedDict(nt._asdict())``.
 (Contributed by Raymond Hettinger in :issue:`35864`.)
 
 
+cProfile
+--------
+
+The :class:`cProfile.Profile <profile.Profile>` class can now be used as a context manager.
+Profile a block of code by running::
+
+      import cProfile
+
+      with cProfile.Profile() as profiler:
+            # code to be profiled
+            ...
+
+(Contributed by Scott Sanderson in :issue:`29235`.)
+
+
+csv
+---
+
+The :class:`csv.DictReader` now returns instances of :class:`dict` instead of
+a :class:`collections.OrderedDict`.  The tool is now faster and uses less
+memory while still preserving the field order.
+(Contributed by Michael Seek in :issue:`34003`.)
+
+
 curses
 -------
 
@@ -700,6 +732,30 @@ cached for the life of the instance. ::
 (Contributed by Carl Meyer in :issue:`21145`)
 
 
+Added a new :func:`functools.singledispatchmethod` decorator that converts
+methods into :term:`generic functions <generic function>` using
+:term:`single dispatch`::
+
+    from functools import singledispatchmethod
+    from contextlib import suppress
+
+    class TaskManager:
+
+        def __init__(self, tasks):
+            self.tasks = list(tasks)
+
+        @singledispatchmethod
+        def discard(self, value):
+            with suppress(ValueError):
+                self.tasks.remove(value)
+
+        @discard.register(list)
+        def _(self, tasks):
+            targets = set(tasks)
+            self.tasks = [x for x in self.tasks if x not in targets]
+
+(Contributed by Ethan Smith in :issue:`32380`)
+
 gc
 --
 
@@ -727,7 +783,7 @@ for certain types of invalid or corrupt gzip files.
 :issue:`6584`.)
 
 
-idlelib and IDLE
+IDLE and idlelib
 ----------------
 
 Output over N lines (50 by default) is squeezed down to a button.
@@ -743,12 +799,19 @@ They also re-appear in the box for the next customized run.  One can also
 suppress the normal Shell main module restart.  (Contributed by Cheryl
 Sabella, Terry Jan Reedy, and others in :issue:`5680` and :issue:`37627`.)
 
-Add optional line numbers for IDLE editor windows. Windows
+Added optional line numbers for IDLE editor windows. Windows
 open without line numbers unless set otherwise in the General
 tab of the configuration dialog.  Line numbers for an existing
 window are shown and hidden in the Options menu.
 (Contributed by Tal Einat and Saimadhav Heblikar in :issue:`17535`.)
 
+OS native encoding is now used for converting between Python strings and Tcl
+objects. This allows IDLE to work with emoji and other non-BMP characters.
+These characters can be displayed or copied and pasted to or from the
+clipboard.  Converting strings from Tcl to Python and back now never fails.
+(Many people worked on this for eight years but the problem was finally
+solved by Serhiy Storchaka in :issue:`13153`.)
+
 The changes above have been backported to 3.7 maintenance releases.
 
 
@@ -779,13 +842,44 @@ fails. The exception is ignored silently by default in release build.
 (Contributed by Victor Stinner in :issue:`18748`.)
 
 
+itertools
+---------
+
+The :func:`itertools.accumulate` function added an option *initial* keyword
+argument to specify an initial value::
+
+    >>> from itertools import accumulate
+    >>> list(accumulate([10, 5, 30, 15], initial=1000))
+    [1000, 1010, 1015, 1045, 1060]
+
+(Contributed by Lisa Roach in :issue:`34659`.)
+
+
 json.tool
 ---------
 
-Add option ``--json-lines`` to parse every input line as separate JSON object.
+Add option ``--json-lines`` to parse every input line as a separate JSON object.
 (Contributed by Weipeng Hong in :issue:`31553`.)
 
 
+logging
+-------
+
+Added a *force* keyword argument to :func:`logging.basicConfig()`
+When set to *True*, any existing handlers attached
+to the root logger are removed and closed before carrying out the
+configuration specified by the other arguments.
+
+This solves a long-standing problem.  Once a logger or *basicConfig()* had
+been called, subsequent calls to *basicConfig()* were silently ignored.
+This made it difficult to update, experiment with, or teach the various
+logging configuration options using the interactive prompt or a Jupyter
+notebook.
+
+(Suggested by Raymond Hettinger, implemented by Dong-hee Na, and
+reviewed by Vinay Sajip in :issue:`33897`.)
+
+
 math
 ----
 
@@ -807,7 +901,28 @@ numbers::
 
 (Contributed by Pablo Galindo in :issue:`35606`.)
 
-Added new function :func:`math.isqrt` for computing integer square roots.
+Added two new combinatoric functions :func:`math.perm` and :func:`math.comb`::
+
+    >>> math.perm(10, 3)    # Permutations of 10 things taken 3 at a time
+    720
+    >>> math.comb(10, 3)    # Combinations of 10 things taken 3 at a time
+    120
+
+(Contributed by Yash Aggarwal, Keller Fuchs, Serhiy Storchaka, and Raymond
+Hettinger in :issue:`37128`, :issue:`37178`, and :issue:`35431`.)
+
+Added a new function :func:`math.isqrt` for computing accurate integer square
+roots without conversion to floating point.  The new function supports
+arbitrarily large integers.  It is faster than ``floor(sqrt(n))`` but slower
+than :func:`math.sqrt`::
+
+    >>> r = 650320427
+    >>> s = r ** 2
+    >>> isqrt(s - 1)         # correct
+    650320426
+    >>> floor(sqrt(s - 1))   # incorrect
+    650320427
+
 (Contributed by Mark Dickinson in :issue:`36887`.)
 
 The function :func:`math.factorial` no longer accepts arguments that are not
@@ -908,11 +1023,6 @@ to a path.
 pickle
 ------
 
-Reduction methods can now include a 6th item in the tuple they return. This
-item should specify a custom state-setting method that's called instead of the
-regular ``__setstate__`` method.
-(Contributed by Pierre Glaser and Olivier Grisel in :issue:`35900`.)
-
 :mod:`pickle` extensions subclassing the C-optimized :class:`~pickle.Pickler`
 can now override the pickling logic of functions and classes by defining the
 special :meth:`~pickle.Pickler.reducer_override` method.
@@ -927,6 +1037,32 @@ NSKeyedArchiver-encoded binary plists.
 (Contributed by Jon Janzen in :issue:`26707`.)
 
 
+pprint
+------
+
+The :mod:`pprint` module added a *sort_dicts* parameter to several functions.
+By default, those functions continue to sort dictionaries before rendering or
+printing.  However, if *sort_dicts* is set to *False*, the dictionaries retain
+the order that keys were inserted.  This can be useful for comparison to JSON
+inputs during debugging.
+
+In addition, there is a convenience new function, :func:`pprint.pp` that is
+like :func:`pprint.pprint` but with *sort_dicts* defaulting to *False*::
+
+    >>> from pprint import pprint, pp
+    >>> d = dict(source='input.txt', operation='filter', destination='output.txt')
+    >>> pp(d, width=40)                  # Original order
+    {'source': 'input.txt',
+     'operation': 'filter',
+     'destination': 'output.txt'}
+    >>> pprint(d, width=40)              # Keys sorted alphabetically
+    {'destination': 'output.txt',
+     'operation': 'filter',
+     'source': 'input.txt'}
+
+(Contributed by R?mi Lapeyre in :issue:`30670`.)
+
+
 py_compile
 ----------
 
@@ -973,8 +1109,8 @@ The :func:`socket.if_nameindex()`, :func:`socket.if_nametoindex()`, and
 ssl
 ---
 
-Added :attr:`ssl.SSLContext.post_handshake_auth` to enable and
-:meth:`ssl.SSLSocket.verify_client_post_handshake` to initiate TLS 1.3
+Added :attr:`~ssl.SSLContext.post_handshake_auth` to enable and
+:meth:`~ssl.SSLSocket.verify_client_post_handshake` to initiate TLS 1.3
 post-handshake authentication.
 (Contributed by Christian Heimes in :issue:`34670`.)
 
@@ -1152,8 +1288,9 @@ the string.  (Contributed by Max Belanger, David Euresti, and Greg Price in
 unittest
 --------
 
-Added :class:`AsyncMock` to support an asynchronous version of :class:`Mock`.
-Appropriate new assert functions for testing have been added as well.
+Added :class:`~unittest.mock.AsyncMock` to support an asynchronous version of
+:class:`~unittest.mock.Mock`.  Appropriate new assert functions for testing
+have been added as well.
 (Contributed by Lisa Roach in :issue:`26467`).
 
 Added :func:`~unittest.addModuleCleanup()` and
@@ -1233,6 +1370,16 @@ them in the generated tree.
 (Contributed by Stefan Behnel in :issue:`36676` and :issue:`36673`.)
 
 
+xmlrpc
+------
+
+:class:`xmlrpc.client.ServerProxy` now supports an optional *headers* keyword
+argument for a sequence of HTTP headers to be sent with each request.  Among
+other things, this makes it possible to upgrade from default basic
+authentication to faster session authentication.
+(Contributed by C?dric Krier in :issue:`35153`.)
+
+
 Optimizations
 =============
 
@@ -1456,6 +1603,11 @@ Deprecated
   constant nodes.
   (Contributed by Serhiy Storchaka in :issue:`36917`.)
 
+* The :func:`asyncio.coroutine` :term:`decorator` is deprecated and will be
+  removed in version 3.10.  Instead of ``@asyncio.coroutine``, use
+  :keyword:`async def` instead.
+  (Contributed by Andrew Svetlov in :issue:`36921`.)
+
 * The following functions and methods are deprecated in the :mod:`gettext`
   module: :func:`~gettext.lgettext`, :func:`~gettext.ldgettext`,
   :func:`~gettext.lngettext` and :func:`~gettext.ldngettext`.
@@ -1502,7 +1654,7 @@ Deprecated
     :class:`multiprocessing.managers.SharedMemoryServer`.
   - *obj* in :func:`weakref.finalize`.
 
-  In future releases of Python they will be :ref:`positional-only
+  In future releases of Python, they will be :ref:`positional-only
   <positional-only_parameter>`.
   (Contributed by Serhiy Storchaka in :issue:`36492`.)
 
@@ -1512,6 +1664,11 @@ API and Feature Removals
 
 The following features and APIs have been removed from Python 3.8:
 
+*  Starting with Python 3.3, importing ABCs from :mod:`collections` was
+   deprecated, and importing should be done from :mod:`collections.abc`. Being
+   able to import from collections was marked for removal in 3.8, but has been
+   delayed to 3.9. (See :issue:`36952`.)
+
 * The :mod:`macpath` module, deprecated in Python 3.7, has been removed.
   (Contributed by Victor Stinner in :issue:`35471`.)
 
@@ -1630,7 +1787,7 @@ Changes in the Python API
   (Contributed by Eric Snow in :issue:`34651`, modified by Christian Heimes
   in :issue:`37951`.)
 
-* The :meth:`imap.IMAP4.logout` method no longer ignores silently arbitrary
+* The :meth:`imap.IMAP4.logout` method no longer silently ignores arbitrary
   exceptions.
   (Contributed by Victor Stinner in :issue:`36348`.)
 
@@ -1685,7 +1842,7 @@ Changes in the Python API
 * The ``PyGC_Head`` struct has changed completely.  All code that touched the
   struct member should be rewritten.  (See :issue:`33597`.)
 
-* The ``PyInterpreterState`` struct has been moved into the "internal"
+* The :c:type:`PyInterpreterState` struct has been moved into the "internal"
   header files (specifically Include/internal/pycore_pystate.h).  An
   opaque ``PyInterpreterState`` is still available as part of the public
   API (and stable ABI).  The docs indicate that none of the struct's


From webhook-mailer at python.org  Sun Oct 20 14:35:32 2019
From: webhook-mailer at python.org (Raymond Hettinger)
Date: Sun, 20 Oct 2019 18:35:32 -0000
Subject: [Python-checkins] bpo-37759: More updates to Whatsnew 3.8
 (GH-16854) (GH-16867)
Message-ID: <mailman.81.1571596533.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/c39bc5c9e8f96a8197d1ce2b70746b7189135566
commit: c39bc5c9e8f96a8197d1ce2b70746b7189135566
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Raymond Hettinger <rhettinger at users.noreply.github.com>
date: 2019-10-20T11:35:28-07:00
summary:

bpo-37759: More updates to Whatsnew 3.8 (GH-16854) (GH-16867)

* math.perm() and math.comb()

* math.isqrt()

* Add singledispatchmethod()

* itertools.accumulate()

* Optional headers for xmlrpc.client.ServerProxy

* IDLE non-BMP characters

* import collections.abc directly

* @coroutine is deprecated

* pprint.pp()

* New options for object.__reduce__()

* DictReader no longer returns OrderedDicts

* "force" option for logging.basicConfig()

* Fix spelling

* cProfile context manager

* Various markup/grammar fixes from Kyle Stanley.
Other minor fixes as well.
Also, dedup the __reduce__ entry.

* Fix markup

* Fix grammar nits found by MS Word
(cherry picked from commit c93883c6afbd99929516764263fc49c3e996b10a)

Co-authored-by: Raymond Hettinger <rhettinger at users.noreply.github.com>

files:
M Doc/whatsnew/3.8.rst

diff --git a/Doc/whatsnew/3.8.rst b/Doc/whatsnew/3.8.rst
index b1497880c6323..d076e3f5b453e 100644
--- a/Doc/whatsnew/3.8.rst
+++ b/Doc/whatsnew/3.8.rst
@@ -357,8 +357,8 @@ It is meant to formalize existing optimizations which were already done
 for various classes.
 Any extension type implementing a callable can use this protocol.
 
-This is currently provisional,
-the aim is to make it fully public in Python 3.9.
+This is currently provisional.
+The aim is to make it fully public in Python 3.9.
 
 See :pep:`590` for a full description.
 
@@ -447,7 +447,7 @@ Other Language Changes
   an instance of the subclass, rather than the base class. This also affects
   the return type of operations whose implementation (directly or indirectly)
   uses :class:`datetime.timedelta` arithmetic, such as
-  :meth:`datetime.datetime.astimezone`.
+  :meth:`~datetime.datetime.astimezone`.
   (Contributed by Paul Ganssle in :issue:`32417`.)
 
 * When the Python interpreter is interrupted by Ctrl-C (SIGINT) and the
@@ -531,6 +531,13 @@ Other Language Changes
 
   (Contributed by J?rn Heissler in :issue:`35224`.)
 
+* The :meth:`object.__reduce__` method can now return a tuple from two to
+  six elements long. Formerly, five was the limit.  The new, optional sixth
+  element is a callable with a ``(obj, state)`` signature.  This allows the
+  direct control over the state-updating behavior of a specific object.  If
+  not *None*, this callable will have priority over the object's
+  :meth:`~__setstate__` method.
+  (Contributed by Pierre Glaser and Olivier Grisel in :issue:`35900`.)
 
 New Modules
 ===========
@@ -581,8 +588,8 @@ The :func:`ast.parse` function has some new flags:
   comments" (returned for function definition AST nodes);
 
 * ``feature_version=(3, N)`` allows specifying an earlier Python 3
-  version. (For example, ``feature_version=(3, 4)`` will treat
-  ``async`` and ``await`` as non-reserved words.)
+  version.  For example, ``feature_version=(3, 4)`` will treat
+  :keyword:`async` and :keyword:`await` as non-reserved words.
 
 (Contributed by Guido van Rossum in :issue:`35766`.)
 
@@ -632,14 +639,39 @@ marked with the ``CO_COROUTINE`` flag may then be returned.
 collections
 -----------
 
-The :meth:`_asdict()` method for :func:`collections.namedtuple` now returns
-a :class:`dict` instead of a :class:`collections.OrderedDict`. This works because
-regular dicts have guaranteed ordering since Python 3.7. If the extra
-features of :class:`OrderedDict` are required, the suggested remediation is
-to cast the result to the desired type: ``OrderedDict(nt._asdict())``.
+The :meth:`~collections.somenamedtuple._asdict` method for
+:func:`collections.namedtuple` now returns a :class:`dict` instead of a
+:class:`collections.OrderedDict`. This works because regular dicts have
+guaranteed ordering since Python 3.7. If the extra features of
+:class:`OrderedDict` are required, the suggested remediation is to cast the
+result to the desired type: ``OrderedDict(nt._asdict())``.
 (Contributed by Raymond Hettinger in :issue:`35864`.)
 
 
+cProfile
+--------
+
+The :class:`cProfile.Profile <profile.Profile>` class can now be used as a context manager.
+Profile a block of code by running::
+
+      import cProfile
+
+      with cProfile.Profile() as profiler:
+            # code to be profiled
+            ...
+
+(Contributed by Scott Sanderson in :issue:`29235`.)
+
+
+csv
+---
+
+The :class:`csv.DictReader` now returns instances of :class:`dict` instead of
+a :class:`collections.OrderedDict`.  The tool is now faster and uses less
+memory while still preserving the field order.
+(Contributed by Michael Seek in :issue:`34003`.)
+
+
 curses
 -------
 
@@ -702,6 +734,30 @@ cached for the life of the instance. ::
 (Contributed by Carl Meyer in :issue:`21145`)
 
 
+Added a new :func:`functools.singledispatchmethod` decorator that converts
+methods into :term:`generic functions <generic function>` using
+:term:`single dispatch`::
+
+    from functools import singledispatchmethod
+    from contextlib import suppress
+
+    class TaskManager:
+
+        def __init__(self, tasks):
+            self.tasks = list(tasks)
+
+        @singledispatchmethod
+        def discard(self, value):
+            with suppress(ValueError):
+                self.tasks.remove(value)
+
+        @discard.register(list)
+        def _(self, tasks):
+            targets = set(tasks)
+            self.tasks = [x for x in self.tasks if x not in targets]
+
+(Contributed by Ethan Smith in :issue:`32380`)
+
 gc
 --
 
@@ -729,7 +785,7 @@ for certain types of invalid or corrupt gzip files.
 :issue:`6584`.)
 
 
-idlelib and IDLE
+IDLE and idlelib
 ----------------
 
 Output over N lines (50 by default) is squeezed down to a button.
@@ -745,12 +801,19 @@ They also re-appear in the box for the next customized run.  One can also
 suppress the normal Shell main module restart.  (Contributed by Cheryl
 Sabella, Terry Jan Reedy, and others in :issue:`5680` and :issue:`37627`.)
 
-Add optional line numbers for IDLE editor windows. Windows
+Added optional line numbers for IDLE editor windows. Windows
 open without line numbers unless set otherwise in the General
 tab of the configuration dialog.  Line numbers for an existing
 window are shown and hidden in the Options menu.
 (Contributed by Tal Einat and Saimadhav Heblikar in :issue:`17535`.)
 
+OS native encoding is now used for converting between Python strings and Tcl
+objects. This allows IDLE to work with emoji and other non-BMP characters.
+These characters can be displayed or copied and pasted to or from the
+clipboard.  Converting strings from Tcl to Python and back now never fails.
+(Many people worked on this for eight years but the problem was finally
+solved by Serhiy Storchaka in :issue:`13153`.)
+
 The changes above have been backported to 3.7 maintenance releases.
 
 
@@ -781,13 +844,44 @@ fails. The exception is ignored silently by default in release build.
 (Contributed by Victor Stinner in :issue:`18748`.)
 
 
+itertools
+---------
+
+The :func:`itertools.accumulate` function added an option *initial* keyword
+argument to specify an initial value::
+
+    >>> from itertools import accumulate
+    >>> list(accumulate([10, 5, 30, 15], initial=1000))
+    [1000, 1010, 1015, 1045, 1060]
+
+(Contributed by Lisa Roach in :issue:`34659`.)
+
+
 json.tool
 ---------
 
-Add option ``--json-lines`` to parse every input line as separate JSON object.
+Add option ``--json-lines`` to parse every input line as a separate JSON object.
 (Contributed by Weipeng Hong in :issue:`31553`.)
 
 
+logging
+-------
+
+Added a *force* keyword argument to :func:`logging.basicConfig()`
+When set to *True*, any existing handlers attached
+to the root logger are removed and closed before carrying out the
+configuration specified by the other arguments.
+
+This solves a long-standing problem.  Once a logger or *basicConfig()* had
+been called, subsequent calls to *basicConfig()* were silently ignored.
+This made it difficult to update, experiment with, or teach the various
+logging configuration options using the interactive prompt or a Jupyter
+notebook.
+
+(Suggested by Raymond Hettinger, implemented by Dong-hee Na, and
+reviewed by Vinay Sajip in :issue:`33897`.)
+
+
 math
 ----
 
@@ -809,7 +903,28 @@ numbers::
 
 (Contributed by Pablo Galindo in :issue:`35606`.)
 
-Added new function :func:`math.isqrt` for computing integer square roots.
+Added two new combinatoric functions :func:`math.perm` and :func:`math.comb`::
+
+    >>> math.perm(10, 3)    # Permutations of 10 things taken 3 at a time
+    720
+    >>> math.comb(10, 3)    # Combinations of 10 things taken 3 at a time
+    120
+
+(Contributed by Yash Aggarwal, Keller Fuchs, Serhiy Storchaka, and Raymond
+Hettinger in :issue:`37128`, :issue:`37178`, and :issue:`35431`.)
+
+Added a new function :func:`math.isqrt` for computing accurate integer square
+roots without conversion to floating point.  The new function supports
+arbitrarily large integers.  It is faster than ``floor(sqrt(n))`` but slower
+than :func:`math.sqrt`::
+
+    >>> r = 650320427
+    >>> s = r ** 2
+    >>> isqrt(s - 1)         # correct
+    650320426
+    >>> floor(sqrt(s - 1))   # incorrect
+    650320427
+
 (Contributed by Mark Dickinson in :issue:`36887`.)
 
 The function :func:`math.factorial` no longer accepts arguments that are not
@@ -910,11 +1025,6 @@ to a path.
 pickle
 ------
 
-Reduction methods can now include a 6th item in the tuple they return. This
-item should specify a custom state-setting method that's called instead of the
-regular ``__setstate__`` method.
-(Contributed by Pierre Glaser and Olivier Grisel in :issue:`35900`.)
-
 :mod:`pickle` extensions subclassing the C-optimized :class:`~pickle.Pickler`
 can now override the pickling logic of functions and classes by defining the
 special :meth:`~pickle.Pickler.reducer_override` method.
@@ -929,6 +1039,32 @@ NSKeyedArchiver-encoded binary plists.
 (Contributed by Jon Janzen in :issue:`26707`.)
 
 
+pprint
+------
+
+The :mod:`pprint` module added a *sort_dicts* parameter to several functions.
+By default, those functions continue to sort dictionaries before rendering or
+printing.  However, if *sort_dicts* is set to *False*, the dictionaries retain
+the order that keys were inserted.  This can be useful for comparison to JSON
+inputs during debugging.
+
+In addition, there is a convenience new function, :func:`pprint.pp` that is
+like :func:`pprint.pprint` but with *sort_dicts* defaulting to *False*::
+
+    >>> from pprint import pprint, pp
+    >>> d = dict(source='input.txt', operation='filter', destination='output.txt')
+    >>> pp(d, width=40)                  # Original order
+    {'source': 'input.txt',
+     'operation': 'filter',
+     'destination': 'output.txt'}
+    >>> pprint(d, width=40)              # Keys sorted alphabetically
+    {'destination': 'output.txt',
+     'operation': 'filter',
+     'source': 'input.txt'}
+
+(Contributed by R?mi Lapeyre in :issue:`30670`.)
+
+
 py_compile
 ----------
 
@@ -975,8 +1111,8 @@ The :func:`socket.if_nameindex()`, :func:`socket.if_nametoindex()`, and
 ssl
 ---
 
-Added :attr:`ssl.SSLContext.post_handshake_auth` to enable and
-:meth:`ssl.SSLSocket.verify_client_post_handshake` to initiate TLS 1.3
+Added :attr:`~ssl.SSLContext.post_handshake_auth` to enable and
+:meth:`~ssl.SSLSocket.verify_client_post_handshake` to initiate TLS 1.3
 post-handshake authentication.
 (Contributed by Christian Heimes in :issue:`34670`.)
 
@@ -1154,8 +1290,9 @@ the string.  (Contributed by Max Belanger, David Euresti, and Greg Price in
 unittest
 --------
 
-Added :class:`AsyncMock` to support an asynchronous version of :class:`Mock`.
-Appropriate new assert functions for testing have been added as well.
+Added :class:`~unittest.mock.AsyncMock` to support an asynchronous version of
+:class:`~unittest.mock.Mock`.  Appropriate new assert functions for testing
+have been added as well.
 (Contributed by Lisa Roach in :issue:`26467`).
 
 Added :func:`~unittest.addModuleCleanup()` and
@@ -1235,6 +1372,16 @@ them in the generated tree.
 (Contributed by Stefan Behnel in :issue:`36676` and :issue:`36673`.)
 
 
+xmlrpc
+------
+
+:class:`xmlrpc.client.ServerProxy` now supports an optional *headers* keyword
+argument for a sequence of HTTP headers to be sent with each request.  Among
+other things, this makes it possible to upgrade from default basic
+authentication to faster session authentication.
+(Contributed by C?dric Krier in :issue:`35153`.)
+
+
 Optimizations
 =============
 
@@ -1458,6 +1605,11 @@ Deprecated
   constant nodes.
   (Contributed by Serhiy Storchaka in :issue:`36917`.)
 
+* The :func:`asyncio.coroutine` :term:`decorator` is deprecated and will be
+  removed in version 3.10.  Instead of ``@asyncio.coroutine``, use
+  :keyword:`async def` instead.
+  (Contributed by Andrew Svetlov in :issue:`36921`.)
+
 * The following functions and methods are deprecated in the :mod:`gettext`
   module: :func:`~gettext.lgettext`, :func:`~gettext.ldgettext`,
   :func:`~gettext.lngettext` and :func:`~gettext.ldngettext`.
@@ -1504,7 +1656,7 @@ Deprecated
     :class:`multiprocessing.managers.SharedMemoryServer`.
   - *obj* in :func:`weakref.finalize`.
 
-  In future releases of Python they will be :ref:`positional-only
+  In future releases of Python, they will be :ref:`positional-only
   <positional-only_parameter>`.
   (Contributed by Serhiy Storchaka in :issue:`36492`.)
 
@@ -1514,6 +1666,11 @@ API and Feature Removals
 
 The following features and APIs have been removed from Python 3.8:
 
+*  Starting with Python 3.3, importing ABCs from :mod:`collections` was
+   deprecated, and importing should be done from :mod:`collections.abc`. Being
+   able to import from collections was marked for removal in 3.8, but has been
+   delayed to 3.9. (See :issue:`36952`.)
+
 * The :mod:`macpath` module, deprecated in Python 3.7, has been removed.
   (Contributed by Victor Stinner in :issue:`35471`.)
 
@@ -1632,7 +1789,7 @@ Changes in the Python API
   (Contributed by Eric Snow in :issue:`34651`, modified by Christian Heimes
   in :issue:`37951`.)
 
-* The :meth:`imap.IMAP4.logout` method no longer ignores silently arbitrary
+* The :meth:`imap.IMAP4.logout` method no longer silently ignores arbitrary
   exceptions.
   (Contributed by Victor Stinner in :issue:`36348`.)
 
@@ -1687,7 +1844,7 @@ Changes in the Python API
 * The ``PyGC_Head`` struct has changed completely.  All code that touched the
   struct member should be rewritten.  (See :issue:`33597`.)
 
-* The ``PyInterpreterState`` struct has been moved into the "internal"
+* The :c:type:`PyInterpreterState` struct has been moved into the "internal"
   header files (specifically Include/internal/pycore_pystate.h).  An
   opaque ``PyInterpreterState`` is still available as part of the public
   API (and stable ABI).  The docs indicate that none of the struct's


From webhook-mailer at python.org  Sun Oct 20 16:14:04 2019
From: webhook-mailer at python.org (Raymond Hettinger)
Date: Sun, 20 Oct 2019 20:14:04 -0000
Subject: [Python-checkins] bpo-38531: document extend action's added version
 (GH-16865)
Message-ID: <mailman.82.1571602445.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/74142078b3b78fea7b4cd791e5c577c0c0964eb7
commit: 74142078b3b78fea7b4cd791e5c577c0c0964eb7
branch: master
author: Batuhan Ta?kaya <47358913+isidentical at users.noreply.github.com>
committer: Raymond Hettinger <rhettinger at users.noreply.github.com>
date: 2019-10-20T13:13:54-07:00
summary:

bpo-38531: document extend action's added version (GH-16865)

files:
M Doc/library/argparse.rst

diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst
index 00115d5cd1e9b..818acf484cdf3 100644
--- a/Doc/library/argparse.rst
+++ b/Doc/library/argparse.rst
@@ -838,6 +838,8 @@ how the command-line arguments should be handled. The supplied actions are:
     >>> parser.parse_args(["--foo", "f1", "--foo", "f2", "f3", "f4"])
     Namespace(foo=['f1', 'f2', 'f3', 'f4'])
 
+  .. versionadded:: 3.8
+
 You may also specify an arbitrary action by passing an Action subclass or
 other object that implements the same interface. The ``BooleanOptionalAction``
 is available in ``argparse`` and adds support for boolean actions such as


From webhook-mailer at python.org  Sun Oct 20 16:25:28 2019
From: webhook-mailer at python.org (Raymond Hettinger)
Date: Sun, 20 Oct 2019 20:25:28 -0000
Subject: [Python-checkins] bpo-38531: document extend action's added version
 (GH-16865) (GH-16868)
Message-ID: <mailman.83.1571603130.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/db385229645dbaaa9863e259b2fae67b9da873fe
commit: db385229645dbaaa9863e259b2fae67b9da873fe
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Raymond Hettinger <rhettinger at users.noreply.github.com>
date: 2019-10-20T13:25:25-07:00
summary:

bpo-38531: document extend action's added version (GH-16865) (GH-16868)

(cherry picked from commit 74142078b3b78fea7b4cd791e5c577c0c0964eb7)

Co-authored-by: Batuhan Ta?kaya <47358913+isidentical at users.noreply.github.com>

files:
M Doc/library/argparse.rst

diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst
index ee00559485c1d..5a61a5bf8ed90 100644
--- a/Doc/library/argparse.rst
+++ b/Doc/library/argparse.rst
@@ -810,6 +810,8 @@ how the command-line arguments should be handled. The supplied actions are:
     >>> parser.parse_args(["--foo", "f1", "--foo", "f2", "f3", "f4"])
     Namespace(foo=['f1', 'f2', 'f3', 'f4'])
 
+  .. versionadded:: 3.8
+
 You may also specify an arbitrary action by passing an Action subclass or
 other object that implements the same interface.  The recommended way to do
 this is to extend :class:`Action`, overriding the ``__call__`` method


From webhook-mailer at python.org  Mon Oct 21 02:31:51 2019
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Mon, 21 Oct 2019 06:31:51 -0000
Subject: [Python-checkins] bpo-38465: Convert the type of exports counters
 to Py_ssize_t. (GH-16746)
Message-ID: <mailman.84.1571639512.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/06cd5b6acd98205adae1a3ea4223e88f38ad55ab
commit: 06cd5b6acd98205adae1a3ea4223e88f38ad55ab
branch: master
author: Hai Shi <shihai1992 at gmail.com>
committer: Serhiy Storchaka <storchaka at gmail.com>
date: 2019-10-21T09:31:46+03:00
summary:

bpo-38465: Convert the type of exports counters to Py_ssize_t. (GH-16746)

files:
A Misc/NEWS.d/next/Core and Builtins/2019-10-19-12-44-13.bpo-38465.V1L8c4.rst
M Include/bytearrayobject.h
M Modules/arraymodule.c
M Modules/mmapmodule.c

diff --git a/Include/bytearrayobject.h b/Include/bytearrayobject.h
index a757b88059297..647a17a819d23 100644
--- a/Include/bytearrayobject.h
+++ b/Include/bytearrayobject.h
@@ -22,11 +22,10 @@ extern "C" {
 #ifndef Py_LIMITED_API
 typedef struct {
     PyObject_VAR_HEAD
-    Py_ssize_t ob_alloc; /* How many bytes allocated in ob_bytes */
-    char *ob_bytes;      /* Physical backing buffer */
-    char *ob_start;      /* Logical start inside ob_bytes */
-    /* XXX(nnorwitz): should ob_exports be Py_ssize_t? */
-    int ob_exports;      /* How many buffer exports */
+    Py_ssize_t ob_alloc;   /* How many bytes allocated in ob_bytes */
+    char *ob_bytes;        /* Physical backing buffer */
+    char *ob_start;        /* Logical start inside ob_bytes */
+    Py_ssize_t ob_exports; /* How many buffer exports */
 } PyByteArrayObject;
 #endif
 
diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-10-19-12-44-13.bpo-38465.V1L8c4.rst b/Misc/NEWS.d/next/Core and Builtins/2019-10-19-12-44-13.bpo-38465.V1L8c4.rst
new file mode 100644
index 0000000000000..4f77ebbfd02b8
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2019-10-19-12-44-13.bpo-38465.V1L8c4.rst	
@@ -0,0 +1 @@
+:class:`bytearray`, :class:`~array.array` and :class:`~mmap.mmap` objects allow now to export more than 2**31 buffers at a time.
\ No newline at end of file
diff --git a/Modules/arraymodule.c b/Modules/arraymodule.c
index 6aa981daca1d3..5ba261819db2c 100644
--- a/Modules/arraymodule.c
+++ b/Modules/arraymodule.c
@@ -43,7 +43,7 @@ typedef struct arrayobject {
     Py_ssize_t allocated;
     const struct arraydescr *ob_descr;
     PyObject *weakreflist; /* List of weak references */
-    int ob_exports;  /* Number of exported buffers */
+    Py_ssize_t ob_exports;  /* Number of exported buffers */
 } arrayobject;
 
 static PyTypeObject Arraytype;
diff --git a/Modules/mmapmodule.c b/Modules/mmapmodule.c
index a3b9e4df95305..a5c0ae0eaf065 100644
--- a/Modules/mmapmodule.c
+++ b/Modules/mmapmodule.c
@@ -97,7 +97,7 @@ typedef struct {
 #else
     off_t       offset;
 #endif
-    int     exports;
+    Py_ssize_t  exports;
 
 #ifdef MS_WINDOWS
     HANDLE      map_handle;


From webhook-mailer at python.org  Mon Oct 21 02:36:25 2019
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Mon, 21 Oct 2019 06:36:25 -0000
Subject: [Python-checkins] bpo-38208: Simplify string.Template by using
 __init_subclass__(). (GH-16256)
Message-ID: <mailman.85.1571639786.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/919f0bc8c904d3aa13eedb2dd1fe9c6b0555a591
commit: 919f0bc8c904d3aa13eedb2dd1fe9c6b0555a591
branch: master
author: Serhiy Storchaka <storchaka at gmail.com>
committer: GitHub <noreply at github.com>
date: 2019-10-21T09:36:21+03:00
summary:

bpo-38208: Simplify string.Template by using __init_subclass__(). (GH-16256)

files:
M Lib/string.py

diff --git a/Lib/string.py b/Lib/string.py
index b423ff5dc6f69..489777b10c25d 100644
--- a/Lib/string.py
+++ b/Lib/string.py
@@ -54,30 +54,7 @@ def capwords(s, sep=None):
 
 _sentinel_dict = {}
 
-class _TemplateMetaclass(type):
-    pattern = r"""
-    %(delim)s(?:
-      (?P<escaped>%(delim)s) |   # Escape sequence of two delimiters
-      (?P<named>%(id)s)      |   # delimiter and a Python identifier
-      {(?P<braced>%(bid)s)}  |   # delimiter and a braced identifier
-      (?P<invalid>)              # Other ill-formed delimiter exprs
-    )
-    """
-
-    def __init__(cls, name, bases, dct):
-        super(_TemplateMetaclass, cls).__init__(name, bases, dct)
-        if 'pattern' in dct:
-            pattern = cls.pattern
-        else:
-            pattern = _TemplateMetaclass.pattern % {
-                'delim' : _re.escape(cls.delimiter),
-                'id'    : cls.idpattern,
-                'bid'   : cls.braceidpattern or cls.idpattern,
-                }
-        cls.pattern = _re.compile(pattern, cls.flags | _re.VERBOSE)
-
-
-class Template(metaclass=_TemplateMetaclass):
+class Template:
     """A string class for supporting $-substitutions."""
 
     delimiter = '$'
@@ -89,6 +66,24 @@ class Template(metaclass=_TemplateMetaclass):
     braceidpattern = None
     flags = _re.IGNORECASE
 
+    def __init_subclass__(cls):
+        super().__init_subclass__()
+        if 'pattern' in cls.__dict__:
+            pattern = cls.pattern
+        else:
+            delim = _re.escape(cls.delimiter)
+            id = cls.idpattern
+            bid = cls.braceidpattern or cls.idpattern
+            pattern = fr"""
+            {delim}(?:
+              (?P<escaped>{delim})  |   # Escape sequence of two delimiters
+              (?P<named>{id})       |   # delimiter and a Python identifier
+              {{(?P<braced>{bid})}} |   # delimiter and a braced identifier
+              (?P<invalid>)             # Other ill-formed delimiter exprs
+            )
+            """
+        cls.pattern = _re.compile(pattern, cls.flags | _re.VERBOSE)
+
     def __init__(self, template):
         self.template = template
 
@@ -146,6 +141,9 @@ def convert(mo):
                              self.pattern)
         return self.pattern.sub(convert, self.template)
 
+# Initialize Template.pattern.  __init_subclass__() is automatically called
+# only for subclasses, not for the Template class itself.
+Template.__init_subclass__()
 
 
 ########################################################################


From webhook-mailer at python.org  Mon Oct 21 02:49:54 2019
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Mon, 21 Oct 2019 06:49:54 -0000
Subject: [Python-checkins] bpo-27961: Replace PY_LONG_LONG with long long.
 (GH-15386)
Message-ID: <mailman.86.1571640595.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/a9ed91e6c2f0f1a9960b1382321918448228a801
commit: a9ed91e6c2f0f1a9960b1382321918448228a801
branch: master
author: Sergey Fedoseev <fedoseev.sergey at gmail.com>
committer: Serhiy Storchaka <storchaka at gmail.com>
date: 2019-10-21T09:49:48+03:00
summary:

bpo-27961: Replace PY_LONG_LONG with long long. (GH-15386)

files:
M Lib/test/clinic.test
M Modules/_testcapimodule.c
M PC/pyconfig.h
M Tools/clinic/clinic.py

diff --git a/Lib/test/clinic.test b/Lib/test/clinic.test
index 0d84d5ef5359a..6ee9dc5a84b93 100644
--- a/Lib/test/clinic.test
+++ b/Lib/test/clinic.test
@@ -1269,7 +1269,7 @@ test_long_long_converter(PyObject *module, PyObject *const *args, Py_ssize_t nar
         goto exit;
     }
     a = PyLong_AsLongLong(args[0]);
-    if (a == (PY_LONG_LONG)-1 && PyErr_Occurred()) {
+    if (a == -1 && PyErr_Occurred()) {
         goto exit;
     }
 skip_optional:
@@ -1281,7 +1281,7 @@ exit:
 
 static PyObject *
 test_long_long_converter_impl(PyObject *module, long long a)
-/*[clinic end generated code: output=3e8083f3aee4f18a input=d5fc81577ff4dd02]*/
+/*[clinic end generated code: output=7143b585d7e433e8 input=d5fc81577ff4dd02]*/
 
 
 /*[clinic input]
diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c
index a18a8e7553ef2..baa6907b7e4f6 100644
--- a/Modules/_testcapimodule.c
+++ b/Modules/_testcapimodule.c
@@ -4763,8 +4763,8 @@ dict_get_version(PyObject *self, PyObject *args)
 
     version = dict->ma_version_tag;
 
-    Py_BUILD_ASSERT(sizeof(unsigned PY_LONG_LONG) >= sizeof(version));
-    return PyLong_FromUnsignedLongLong((unsigned PY_LONG_LONG)version);
+    Py_BUILD_ASSERT(sizeof(unsigned long long) >= sizeof(version));
+    return PyLong_FromUnsignedLongLong((unsigned long long)version);
 }
 
 
diff --git a/PC/pyconfig.h b/PC/pyconfig.h
index d407e1d4632af..6a437ce24bb3d 100644
--- a/PC/pyconfig.h
+++ b/PC/pyconfig.h
@@ -296,7 +296,7 @@ Py_NO_ENABLE_SHARED to find out.  Also support MS_NO_COREDLL for b/w compat */
 #       define SIZEOF_HKEY 8
 #       define SIZEOF_SIZE_T 8
 /* configure.ac defines HAVE_LARGEFILE_SUPPORT iff
-   sizeof(off_t) > sizeof(long), and sizeof(PY_LONG_LONG) >= sizeof(off_t).
+   sizeof(off_t) > sizeof(long), and sizeof(long long) >= sizeof(off_t).
    On Win64 the second condition is not true, but if fpos_t replaces off_t
    then this is true. The uses of HAVE_LARGEFILE_SUPPORT imply that Win64
    should define this. */
diff --git a/Tools/clinic/clinic.py b/Tools/clinic/clinic.py
index c5edc7c9336fb..403d8a4765690 100755
--- a/Tools/clinic/clinic.py
+++ b/Tools/clinic/clinic.py
@@ -3016,7 +3016,7 @@ def parse_arg(self, argname, displayname):
                     goto exit;
                 }}}}
                 {paramname} = PyLong_AsLongLong({argname});
-                if ({paramname} == (PY_LONG_LONG)-1 && PyErr_Occurred()) {{{{
+                if ({paramname} == -1 && PyErr_Occurred()) {{{{
                     goto exit;
                 }}}}
                 """.format(argname=argname, paramname=self.name)


From webhook-mailer at python.org  Mon Oct 21 03:01:09 2019
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Mon, 21 Oct 2019 07:01:09 -0000
Subject: [Python-checkins] bpo-38493: Add os.CLD_KILLED and os.CLD_STOPPED.
 (GH-16821)
Message-ID: <mailman.87.1571641270.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/2eba6ad7bf3a5beeed54209a0107be8e1ac77767
commit: 2eba6ad7bf3a5beeed54209a0107be8e1ac77767
branch: master
author: Dong-hee Na <donghee.na92 at gmail.com>
committer: Serhiy Storchaka <storchaka at gmail.com>
date: 2019-10-21T10:01:05+03:00
summary:

bpo-38493: Add os.CLD_KILLED and os.CLD_STOPPED. (GH-16821)

files:
A Misc/NEWS.d/next/Library/2019-10-16-19-56-51.bpo-38493.86ExWB.rst
M Doc/library/os.rst
M Doc/whatsnew/3.9.rst
M Lib/test/test_posix.py
M Modules/posixmodule.c

diff --git a/Doc/library/os.rst b/Doc/library/os.rst
index fe9531b170177..8e9d9e6f034e3 100644
--- a/Doc/library/os.rst
+++ b/Doc/library/os.rst
@@ -3946,8 +3946,10 @@ written in Python, such as a mail server's external command delivery program.
 
 
 .. data:: CLD_EXITED
+          CLD_KILLED
           CLD_DUMPED
           CLD_TRAPPED
+          CLD_STOPPED
           CLD_CONTINUED
 
    These are the possible values for :attr:`si_code` in the result returned by
@@ -3957,6 +3959,9 @@ written in Python, such as a mail server's external command delivery program.
 
    .. versionadded:: 3.3
 
+   .. versionchanged:: 3.9
+      Added :data:`CLD_KILLED` and :data:`CLD_STOPPED` values.
+
 
 .. function:: waitpid(pid, options)
 
diff --git a/Doc/whatsnew/3.9.rst b/Doc/whatsnew/3.9.rst
index 6b4abc0567c30..f203930fd1b7b 100644
--- a/Doc/whatsnew/3.9.rst
+++ b/Doc/whatsnew/3.9.rst
@@ -125,6 +125,12 @@ that schedules a shutdown for the default executor that waits on the
 :func:`asyncio.run` has been updated to use the new :term:`coroutine`.
 (Contributed by Kyle Stanley in :issue:`34037`.)
 
+os
+__
+
+Added :data:`~os.CLD_KILLED` and :data:`~os.CLD_STOPPED` for :attr:`si_code`.
+(Contributed by Dong-hee Na in :issue:`38493`.)
+
 threading
 ---------
 
diff --git a/Lib/test/test_posix.py b/Lib/test/test_posix.py
index 9bdd2848afc81..17e4ded2e2d69 100644
--- a/Lib/test/test_posix.py
+++ b/Lib/test/test_posix.py
@@ -1226,6 +1226,16 @@ def test_rename_dir_fd(self):
         finally:
             posix.close(f)
 
+    @unittest.skipUnless(hasattr(signal, 'SIGCHLD'), 'CLD_XXXX be placed in si_code for a SIGCHLD signal')
+    @unittest.skipUnless(hasattr(os, 'waitid_result'), "test needs os.waitid_result")
+    def test_cld_xxxx_constants(self):
+        os.CLD_EXITED
+        os.CLD_KILLED
+        os.CLD_DUMPED
+        os.CLD_TRAPPED
+        os.CLD_STOPPED
+        os.CLD_CONTINUED
+
     @unittest.skipUnless(os.symlink in os.supports_dir_fd, "test needs dir_fd support in os.symlink()")
     def test_symlink_dir_fd(self):
         f = posix.open(posix.getcwd(), posix.O_RDONLY)
diff --git a/Misc/NEWS.d/next/Library/2019-10-16-19-56-51.bpo-38493.86ExWB.rst b/Misc/NEWS.d/next/Library/2019-10-16-19-56-51.bpo-38493.86ExWB.rst
new file mode 100644
index 0000000000000..1a4bef65d4318
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-10-16-19-56-51.bpo-38493.86ExWB.rst
@@ -0,0 +1,2 @@
+Added :data:`~os.CLD_KILLED` and :data:`~os.CLD_STOPPED` for :attr:`si_code`.
+Patch by Dong-hee Na.
diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
index 686497090f88b..dcd90d3a5152d 100644
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -14057,12 +14057,18 @@ all_ins(PyObject *m)
 #ifdef CLD_EXITED
     if (PyModule_AddIntMacro(m, CLD_EXITED)) return -1;
 #endif
+#ifdef CLD_KILLED
+    if (PyModule_AddIntMacro(m, CLD_KILLED)) return -1;
+#endif
 #ifdef CLD_DUMPED
     if (PyModule_AddIntMacro(m, CLD_DUMPED)) return -1;
 #endif
 #ifdef CLD_TRAPPED
     if (PyModule_AddIntMacro(m, CLD_TRAPPED)) return -1;
 #endif
+#ifdef CLD_STOPPED
+    if (PyModule_AddIntMacro(m, CLD_STOPPED)) return -1;
+#endif
 #ifdef CLD_CONTINUED
     if (PyModule_AddIntMacro(m, CLD_CONTINUED)) return -1;
 #endif


From webhook-mailer at python.org  Mon Oct 21 04:35:11 2019
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Mon, 21 Oct 2019 08:35:11 -0000
Subject: [Python-checkins] bpo-38540: Fix possible leak in PyArg_Parse for
 "es#" and "et#". (GH-16869)
Message-ID: <mailman.88.1571646912.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/5bc6a7c06eda20ba131ecba6752be0506d310181
commit: 5bc6a7c06eda20ba131ecba6752be0506d310181
branch: master
author: Serhiy Storchaka <storchaka at gmail.com>
committer: GitHub <noreply at github.com>
date: 2019-10-21T11:35:07+03:00
summary:

bpo-38540: Fix possible leak in PyArg_Parse for "es#" and "et#". (GH-16869)

files:
A Misc/NEWS.d/next/C API/2019-10-21-09-24-03.bpo-38540.314N_T.rst
M Python/getargs.c

diff --git a/Misc/NEWS.d/next/C API/2019-10-21-09-24-03.bpo-38540.314N_T.rst b/Misc/NEWS.d/next/C API/2019-10-21-09-24-03.bpo-38540.314N_T.rst
new file mode 100644
index 0000000000000..1d73ad8fe96e6
--- /dev/null
+++ b/Misc/NEWS.d/next/C API/2019-10-21-09-24-03.bpo-38540.314N_T.rst	
@@ -0,0 +1,3 @@
+Fixed possible leak in :c:func:`PyArg_Parse` and similar functions for
+format units ``"es#"`` and ``"et#"`` when the macro
+:c:macro:`PY_SSIZE_T_CLEAN` is not defined.
diff --git a/Python/getargs.c b/Python/getargs.c
index 0ca0862912c87..351889f8e5189 100644
--- a/Python/getargs.c
+++ b/Python/getargs.c
@@ -1199,7 +1199,19 @@ convertsimple(PyObject *arg, const char **p_format, va_list *p_va, int flags,
                trailing 0-byte
 
             */
-            FETCH_SIZE;
+            int *q = NULL; Py_ssize_t *q2 = NULL;
+            if (flags & FLAG_SIZE_T) {
+                q2 = va_arg(*p_va, Py_ssize_t*);
+            }
+            else {
+                if (PyErr_WarnEx(PyExc_DeprecationWarning,
+                            "PY_SSIZE_T_CLEAN will be required for '#' formats", 1))
+                {
+                    Py_DECREF(s);
+                    return NULL;
+                }
+                q = va_arg(*p_va, int*);
+            }
 
             format++;
             if (q == NULL && q2 == NULL) {
@@ -1232,7 +1244,19 @@ convertsimple(PyObject *arg, const char **p_format, va_list *p_va, int flags,
                 }
             }
             memcpy(*buffer, ptr, size+1);
-            STORE_SIZE(size);
+
+            if (flags & FLAG_SIZE_T) {
+                *q2 = size;
+            }
+            else {
+                if (INT_MAX < size) {
+                    Py_DECREF(s);
+                    PyErr_SetString(PyExc_OverflowError,
+                                    "size does not fit in an int");
+                    return converterr("", arg, msgbuf, bufsize);
+                }
+                *q = (int)size;
+            }
         } else {
             /* Using a 0-terminated buffer:
 


From webhook-mailer at python.org  Mon Oct 21 04:56:54 2019
From: webhook-mailer at python.org (Miss Skeleton (bot))
Date: Mon, 21 Oct 2019 08:56:54 -0000
Subject: [Python-checkins] bpo-38540: Fix possible leak in PyArg_Parse for
 "esGH-" and "etGH-". (GH-16869)
Message-ID: <mailman.89.1571648215.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/3dec84f40ef49bab994a1af4e6082bf81021feab
commit: 3dec84f40ef49bab994a1af4e6082bf81021feab
branch: 3.7
author: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-21T01:56:50-07:00
summary:

bpo-38540: Fix possible leak in PyArg_Parse for "esGH-" and "etGH-". (GH-16869)

(cherry picked from commit 5bc6a7c06eda20ba131ecba6752be0506d310181)

Co-authored-by: Serhiy Storchaka <storchaka at gmail.com>

files:
A Misc/NEWS.d/next/C API/2019-10-21-09-24-03.bpo-38540.314N_T.rst
M Python/getargs.c

diff --git a/Misc/NEWS.d/next/C API/2019-10-21-09-24-03.bpo-38540.314N_T.rst b/Misc/NEWS.d/next/C API/2019-10-21-09-24-03.bpo-38540.314N_T.rst
new file mode 100644
index 0000000000000..1d73ad8fe96e6
--- /dev/null
+++ b/Misc/NEWS.d/next/C API/2019-10-21-09-24-03.bpo-38540.314N_T.rst	
@@ -0,0 +1,3 @@
+Fixed possible leak in :c:func:`PyArg_Parse` and similar functions for
+format units ``"es#"`` and ``"et#"`` when the macro
+:c:macro:`PY_SSIZE_T_CLEAN` is not defined.
diff --git a/Python/getargs.c b/Python/getargs.c
index 73e47f84561b8..c3895d1d6c594 100644
--- a/Python/getargs.c
+++ b/Python/getargs.c
@@ -1176,7 +1176,19 @@ convertsimple(PyObject *arg, const char **p_format, va_list *p_va, int flags,
                trailing 0-byte
 
             */
-            FETCH_SIZE;
+            int *q = NULL; Py_ssize_t *q2 = NULL;
+            if (flags & FLAG_SIZE_T) {
+                q2 = va_arg(*p_va, Py_ssize_t*);
+            }
+            else {
+                if (PyErr_WarnEx(PyExc_DeprecationWarning,
+                            "PY_SSIZE_T_CLEAN will be required for '#' formats", 1))
+                {
+                    Py_DECREF(s);
+                    return NULL;
+                }
+                q = va_arg(*p_va, int*);
+            }
 
             format++;
             if (q == NULL && q2 == NULL) {
@@ -1209,7 +1221,19 @@ convertsimple(PyObject *arg, const char **p_format, va_list *p_va, int flags,
                 }
             }
             memcpy(*buffer, ptr, size+1);
-            STORE_SIZE(size);
+
+            if (flags & FLAG_SIZE_T) {
+                *q2 = size;
+            }
+            else {
+                if (INT_MAX < size) {
+                    Py_DECREF(s);
+                    PyErr_SetString(PyExc_OverflowError,
+                                    "size does not fit in an int");
+                    return converterr("", arg, msgbuf, bufsize);
+                }
+                *q = (int)size;
+            }
         } else {
             /* Using a 0-terminated buffer:
 


From webhook-mailer at python.org  Mon Oct 21 05:12:44 2019
From: webhook-mailer at python.org (Miss Skeleton (bot))
Date: Mon, 21 Oct 2019 09:12:44 -0000
Subject: [Python-checkins] bpo-38540: Fix possible leak in PyArg_Parse for
 "esGH-" and "etGH-". (GH-16869)
Message-ID: <mailman.90.1571649165.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/b1fc8c0c8aec7ef8edd119034e076bc48882ffbb
commit: b1fc8c0c8aec7ef8edd119034e076bc48882ffbb
branch: 3.8
author: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-21T02:12:37-07:00
summary:

bpo-38540: Fix possible leak in PyArg_Parse for "esGH-" and "etGH-". (GH-16869)

(cherry picked from commit 5bc6a7c06eda20ba131ecba6752be0506d310181)

Co-authored-by: Serhiy Storchaka <storchaka at gmail.com>

files:
A Misc/NEWS.d/next/C API/2019-10-21-09-24-03.bpo-38540.314N_T.rst
M Python/getargs.c

diff --git a/Misc/NEWS.d/next/C API/2019-10-21-09-24-03.bpo-38540.314N_T.rst b/Misc/NEWS.d/next/C API/2019-10-21-09-24-03.bpo-38540.314N_T.rst
new file mode 100644
index 0000000000000..1d73ad8fe96e6
--- /dev/null
+++ b/Misc/NEWS.d/next/C API/2019-10-21-09-24-03.bpo-38540.314N_T.rst	
@@ -0,0 +1,3 @@
+Fixed possible leak in :c:func:`PyArg_Parse` and similar functions for
+format units ``"es#"`` and ``"et#"`` when the macro
+:c:macro:`PY_SSIZE_T_CLEAN` is not defined.
diff --git a/Python/getargs.c b/Python/getargs.c
index c65530d859c32..c1b7b1a275b00 100644
--- a/Python/getargs.c
+++ b/Python/getargs.c
@@ -1199,7 +1199,19 @@ convertsimple(PyObject *arg, const char **p_format, va_list *p_va, int flags,
                trailing 0-byte
 
             */
-            FETCH_SIZE;
+            int *q = NULL; Py_ssize_t *q2 = NULL;
+            if (flags & FLAG_SIZE_T) {
+                q2 = va_arg(*p_va, Py_ssize_t*);
+            }
+            else {
+                if (PyErr_WarnEx(PyExc_DeprecationWarning,
+                            "PY_SSIZE_T_CLEAN will be required for '#' formats", 1))
+                {
+                    Py_DECREF(s);
+                    return NULL;
+                }
+                q = va_arg(*p_va, int*);
+            }
 
             format++;
             if (q == NULL && q2 == NULL) {
@@ -1232,7 +1244,19 @@ convertsimple(PyObject *arg, const char **p_format, va_list *p_va, int flags,
                 }
             }
             memcpy(*buffer, ptr, size+1);
-            STORE_SIZE(size);
+
+            if (flags & FLAG_SIZE_T) {
+                *q2 = size;
+            }
+            else {
+                if (INT_MAX < size) {
+                    Py_DECREF(s);
+                    PyErr_SetString(PyExc_OverflowError,
+                                    "size does not fit in an int");
+                    return converterr("", arg, msgbuf, bufsize);
+                }
+                *q = (int)size;
+            }
         } else {
             /* Using a 0-terminated buffer:
 


From webhook-mailer at python.org  Mon Oct 21 12:21:46 2019
From: webhook-mailer at python.org (Tim Peters)
Date: Mon, 21 Oct 2019 16:21:46 -0000
Subject: [Python-checkins] visit_reachable: replace release-mode test with
 an assert. (GH-16866)
Message-ID: <mailman.91.1571674908.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/1e73945470644202262e1ddee2b49e2708a29794
commit: 1e73945470644202262e1ddee2b49e2708a29794
branch: master
author: Tim Peters <tim.peters at gmail.com>
committer: GitHub <noreply at github.com>
date: 2019-10-21T11:21:35-05:00
summary:

visit_reachable: replace release-mode test with an assert. (GH-16866)

It should be impossible for an untracked object to have the collecting
flag set.  Back when state was stored in gc_refs, it obviously was
impossible (gc_refs couldn't possibly have a positive & negative value
simultaneously).  While the _implementation_ of "state" has gotten much
more complicated, it's still _logically_ just as impossible.

files:
M Modules/gcmodule.c

diff --git a/Modules/gcmodule.c b/Modules/gcmodule.c
index 9750d2a6f3fb2..1307aa3dc12fd 100644
--- a/Modules/gcmodule.c
+++ b/Modules/gcmodule.c
@@ -468,13 +468,16 @@ visit_reachable(PyObject *op, PyGC_Head *reachable)
     PyGC_Head *gc = AS_GC(op);
     const Py_ssize_t gc_refs = gc_get_refs(gc);
 
-    // Ignore untracked objects and objects in other generation.
+    // Ignore objects in other generation.
     // This also skips objects "to the left" of the current position in
     // move_unreachable's scan of the 'young' list - they've already been
     // traversed, and no longer have the PREV_MASK_COLLECTING flag.
-    if (gc->_gc_next == 0 || !gc_is_collecting(gc)) {
+    if (! gc_is_collecting(gc)) {
         return 0;
     }
+    // It would be a logic error elsewhere if the collecting flag were set on
+    // an untracked object.
+    assert(gc->_gc_next != 0);
 
     if (gc->_gc_next & NEXT_MASK_UNREACHABLE) {
         /* This had gc_refs = 0 when move_unreachable got


From webhook-mailer at python.org  Mon Oct 21 13:37:23 2019
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Mon, 21 Oct 2019 17:37:23 -0000
Subject: [Python-checkins] bpo-31202: Preserve case of literal parts in
 Path.glob() on Windows. (GH-16860)
Message-ID: <mailman.92.1571679443.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/10ecbadb799ddf3393d1fc80119a3db14724d381
commit: 10ecbadb799ddf3393d1fc80119a3db14724d381
branch: master
author: Serhiy Storchaka <storchaka at gmail.com>
committer: GitHub <noreply at github.com>
date: 2019-10-21T20:37:15+03:00
summary:

bpo-31202: Preserve case of literal parts in Path.glob() on Windows. (GH-16860)

files:
A Misc/NEWS.d/next/Library/2019-10-20-12-04-48.bpo-31202.NfdIus.rst
M Lib/pathlib.py
M Lib/test/test_pathlib.py

diff --git a/Lib/pathlib.py b/Lib/pathlib.py
index 7bd66c10cbdb6..67b94e0154541 100644
--- a/Lib/pathlib.py
+++ b/Lib/pathlib.py
@@ -187,6 +187,9 @@ def casefold(self, s):
     def casefold_parts(self, parts):
         return [p.lower() for p in parts]
 
+    def compile_pattern(self, pattern):
+        return re.compile(fnmatch.translate(pattern), re.IGNORECASE).fullmatch
+
     def resolve(self, path, strict=False):
         s = str(path)
         if not s:
@@ -309,6 +312,9 @@ def casefold(self, s):
     def casefold_parts(self, parts):
         return parts
 
+    def compile_pattern(self, pattern):
+        return re.compile(fnmatch.translate(pattern)).fullmatch
+
     def resolve(self, path, strict=False):
         sep = self.sep
         accessor = path._accessor
@@ -446,7 +452,7 @@ def readlink(self, path):
 # Globbing helpers
 #
 
-def _make_selector(pattern_parts):
+def _make_selector(pattern_parts, flavour):
     pat = pattern_parts[0]
     child_parts = pattern_parts[1:]
     if pat == '**':
@@ -457,7 +463,7 @@ def _make_selector(pattern_parts):
         cls = _WildcardSelector
     else:
         cls = _PreciseSelector
-    return cls(pat, child_parts)
+    return cls(pat, child_parts, flavour)
 
 if hasattr(functools, "lru_cache"):
     _make_selector = functools.lru_cache()(_make_selector)
@@ -467,10 +473,10 @@ class _Selector:
     """A selector matches a specific glob pattern part against the children
     of a given path."""
 
-    def __init__(self, child_parts):
+    def __init__(self, child_parts, flavour):
         self.child_parts = child_parts
         if child_parts:
-            self.successor = _make_selector(child_parts)
+            self.successor = _make_selector(child_parts, flavour)
             self.dironly = True
         else:
             self.successor = _TerminatingSelector()
@@ -496,9 +502,9 @@ def _select_from(self, parent_path, is_dir, exists, scandir):
 
 class _PreciseSelector(_Selector):
 
-    def __init__(self, name, child_parts):
+    def __init__(self, name, child_parts, flavour):
         self.name = name
-        _Selector.__init__(self, child_parts)
+        _Selector.__init__(self, child_parts, flavour)
 
     def _select_from(self, parent_path, is_dir, exists, scandir):
         try:
@@ -512,13 +518,12 @@ def _select_from(self, parent_path, is_dir, exists, scandir):
 
 class _WildcardSelector(_Selector):
 
-    def __init__(self, pat, child_parts):
-        self.pat = re.compile(fnmatch.translate(pat))
-        _Selector.__init__(self, child_parts)
+    def __init__(self, pat, child_parts, flavour):
+        self.match = flavour.compile_pattern(pat)
+        _Selector.__init__(self, child_parts, flavour)
 
     def _select_from(self, parent_path, is_dir, exists, scandir):
         try:
-            cf = parent_path._flavour.casefold
             entries = list(scandir(parent_path))
             for entry in entries:
                 entry_is_dir = False
@@ -529,8 +534,7 @@ def _select_from(self, parent_path, is_dir, exists, scandir):
                         raise
                 if not self.dironly or entry_is_dir:
                     name = entry.name
-                    casefolded = cf(name)
-                    if self.pat.match(casefolded):
+                    if self.match(name):
                         path = parent_path._make_child_relpath(name)
                         for p in self.successor._select_from(path, is_dir, exists, scandir):
                             yield p
@@ -541,8 +545,8 @@ def _select_from(self, parent_path, is_dir, exists, scandir):
 
 class _RecursiveWildcardSelector(_Selector):
 
-    def __init__(self, pat, child_parts):
-        _Selector.__init__(self, child_parts)
+    def __init__(self, pat, child_parts, flavour):
+        _Selector.__init__(self, child_parts, flavour)
 
     def _iterate_directories(self, parent_path, is_dir, scandir):
         yield parent_path
@@ -1118,11 +1122,10 @@ def glob(self, pattern):
         """
         if not pattern:
             raise ValueError("Unacceptable pattern: {!r}".format(pattern))
-        pattern = self._flavour.casefold(pattern)
         drv, root, pattern_parts = self._flavour.parse_parts((pattern,))
         if drv or root:
             raise NotImplementedError("Non-relative patterns are unsupported")
-        selector = _make_selector(tuple(pattern_parts))
+        selector = _make_selector(tuple(pattern_parts), self._flavour)
         for p in selector.select_from(self):
             yield p
 
@@ -1131,11 +1134,10 @@ def rglob(self, pattern):
         directories) matching the given relative pattern, anywhere in
         this subtree.
         """
-        pattern = self._flavour.casefold(pattern)
         drv, root, pattern_parts = self._flavour.parse_parts((pattern,))
         if drv or root:
             raise NotImplementedError("Non-relative patterns are unsupported")
-        selector = _make_selector(("**",) + tuple(pattern_parts))
+        selector = _make_selector(("**",) + tuple(pattern_parts), self._flavour)
         for p in selector.select_from(self):
             yield p
 
diff --git a/Lib/test/test_pathlib.py b/Lib/test/test_pathlib.py
index f3304f01b4bee..221c2722d0a9d 100644
--- a/Lib/test/test_pathlib.py
+++ b/Lib/test/test_pathlib.py
@@ -2378,11 +2378,15 @@ def test_glob(self):
         P = self.cls
         p = P(BASE)
         self.assertEqual(set(p.glob("FILEa")), { P(BASE, "fileA") })
+        self.assertEqual(set(p.glob("F*a")), { P(BASE, "fileA") })
+        self.assertEqual(set(map(str, p.glob("FILEa"))), {f"{p}\\FILEa"})
+        self.assertEqual(set(map(str, p.glob("F*a"))), {f"{p}\\fileA"})
 
     def test_rglob(self):
         P = self.cls
         p = P(BASE, "dirC")
         self.assertEqual(set(p.rglob("FILEd")), { P(BASE, "dirC/dirD/fileD") })
+        self.assertEqual(set(map(str, p.rglob("FILEd"))), {f"{p}\\dirD\\FILEd"})
 
     def test_expanduser(self):
         P = self.cls
diff --git a/Misc/NEWS.d/next/Library/2019-10-20-12-04-48.bpo-31202.NfdIus.rst b/Misc/NEWS.d/next/Library/2019-10-20-12-04-48.bpo-31202.NfdIus.rst
new file mode 100644
index 0000000000000..8edb09d61317b
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-10-20-12-04-48.bpo-31202.NfdIus.rst
@@ -0,0 +1,2 @@
+The case the result of :func:`pathlib.WindowsPath.glob` matches now the case
+of the pattern for literal parts.


From webhook-mailer at python.org  Mon Oct 21 14:12:21 2019
From: webhook-mailer at python.org (Miss Skeleton (bot))
Date: Mon, 21 Oct 2019 18:12:21 -0000
Subject: [Python-checkins] bpo-31202: Preserve case of literal parts in
 Path.glob() on Windows. (GH-16860)
Message-ID: <mailman.93.1571681542.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/175abccbbfccb2f6489dc5c73f4630c1b25ce504
commit: 175abccbbfccb2f6489dc5c73f4630c1b25ce504
branch: 3.7
author: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-21T11:12:17-07:00
summary:

bpo-31202: Preserve case of literal parts in Path.glob() on Windows. (GH-16860)

(cherry picked from commit 10ecbadb799ddf3393d1fc80119a3db14724d381)

Co-authored-by: Serhiy Storchaka <storchaka at gmail.com>

files:
A Misc/NEWS.d/next/Library/2019-10-20-12-04-48.bpo-31202.NfdIus.rst
M Lib/pathlib.py
M Lib/test/test_pathlib.py

diff --git a/Lib/pathlib.py b/Lib/pathlib.py
index 24437f8f95d13..af9747ba3cb99 100644
--- a/Lib/pathlib.py
+++ b/Lib/pathlib.py
@@ -187,6 +187,9 @@ def casefold(self, s):
     def casefold_parts(self, parts):
         return [p.lower() for p in parts]
 
+    def compile_pattern(self, pattern):
+        return re.compile(fnmatch.translate(pattern), re.IGNORECASE).fullmatch
+
     def resolve(self, path, strict=False):
         s = str(path)
         if not s:
@@ -309,6 +312,9 @@ def casefold(self, s):
     def casefold_parts(self, parts):
         return parts
 
+    def compile_pattern(self, pattern):
+        return re.compile(fnmatch.translate(pattern)).fullmatch
+
     def resolve(self, path, strict=False):
         sep = self.sep
         accessor = path._accessor
@@ -444,7 +450,7 @@ def readlink(self, path):
 # Globbing helpers
 #
 
-def _make_selector(pattern_parts):
+def _make_selector(pattern_parts, flavour):
     pat = pattern_parts[0]
     child_parts = pattern_parts[1:]
     if pat == '**':
@@ -455,7 +461,7 @@ def _make_selector(pattern_parts):
         cls = _WildcardSelector
     else:
         cls = _PreciseSelector
-    return cls(pat, child_parts)
+    return cls(pat, child_parts, flavour)
 
 if hasattr(functools, "lru_cache"):
     _make_selector = functools.lru_cache()(_make_selector)
@@ -465,10 +471,10 @@ class _Selector:
     """A selector matches a specific glob pattern part against the children
     of a given path."""
 
-    def __init__(self, child_parts):
+    def __init__(self, child_parts, flavour):
         self.child_parts = child_parts
         if child_parts:
-            self.successor = _make_selector(child_parts)
+            self.successor = _make_selector(child_parts, flavour)
             self.dironly = True
         else:
             self.successor = _TerminatingSelector()
@@ -494,9 +500,9 @@ def _select_from(self, parent_path, is_dir, exists, scandir):
 
 class _PreciseSelector(_Selector):
 
-    def __init__(self, name, child_parts):
+    def __init__(self, name, child_parts, flavour):
         self.name = name
-        _Selector.__init__(self, child_parts)
+        _Selector.__init__(self, child_parts, flavour)
 
     def _select_from(self, parent_path, is_dir, exists, scandir):
         try:
@@ -510,13 +516,12 @@ def _select_from(self, parent_path, is_dir, exists, scandir):
 
 class _WildcardSelector(_Selector):
 
-    def __init__(self, pat, child_parts):
-        self.pat = re.compile(fnmatch.translate(pat))
-        _Selector.__init__(self, child_parts)
+    def __init__(self, pat, child_parts, flavour):
+        self.match = flavour.compile_pattern(pat)
+        _Selector.__init__(self, child_parts, flavour)
 
     def _select_from(self, parent_path, is_dir, exists, scandir):
         try:
-            cf = parent_path._flavour.casefold
             entries = list(scandir(parent_path))
             for entry in entries:
                 entry_is_dir = False
@@ -527,8 +532,7 @@ def _select_from(self, parent_path, is_dir, exists, scandir):
                         raise
                 if not self.dironly or entry_is_dir:
                     name = entry.name
-                    casefolded = cf(name)
-                    if self.pat.match(casefolded):
+                    if self.match(name):
                         path = parent_path._make_child_relpath(name)
                         for p in self.successor._select_from(path, is_dir, exists, scandir):
                             yield p
@@ -539,8 +543,8 @@ def _select_from(self, parent_path, is_dir, exists, scandir):
 
 class _RecursiveWildcardSelector(_Selector):
 
-    def __init__(self, pat, child_parts):
-        _Selector.__init__(self, child_parts)
+    def __init__(self, pat, child_parts, flavour):
+        _Selector.__init__(self, child_parts, flavour)
 
     def _iterate_directories(self, parent_path, is_dir, scandir):
         yield parent_path
@@ -1101,11 +1105,10 @@ def glob(self, pattern):
         """
         if not pattern:
             raise ValueError("Unacceptable pattern: {!r}".format(pattern))
-        pattern = self._flavour.casefold(pattern)
         drv, root, pattern_parts = self._flavour.parse_parts((pattern,))
         if drv or root:
             raise NotImplementedError("Non-relative patterns are unsupported")
-        selector = _make_selector(tuple(pattern_parts))
+        selector = _make_selector(tuple(pattern_parts), self._flavour)
         for p in selector.select_from(self):
             yield p
 
@@ -1114,11 +1117,10 @@ def rglob(self, pattern):
         directories) matching the given relative pattern, anywhere in
         this subtree.
         """
-        pattern = self._flavour.casefold(pattern)
         drv, root, pattern_parts = self._flavour.parse_parts((pattern,))
         if drv or root:
             raise NotImplementedError("Non-relative patterns are unsupported")
-        selector = _make_selector(("**",) + tuple(pattern_parts))
+        selector = _make_selector(("**",) + tuple(pattern_parts), self._flavour)
         for p in selector.select_from(self):
             yield p
 
diff --git a/Lib/test/test_pathlib.py b/Lib/test/test_pathlib.py
index 9c82c6451479c..e1d699a0d5246 100644
--- a/Lib/test/test_pathlib.py
+++ b/Lib/test/test_pathlib.py
@@ -2216,11 +2216,15 @@ def test_glob(self):
         P = self.cls
         p = P(BASE)
         self.assertEqual(set(p.glob("FILEa")), { P(BASE, "fileA") })
+        self.assertEqual(set(p.glob("F*a")), { P(BASE, "fileA") })
+        self.assertEqual(set(map(str, p.glob("FILEa"))), {f"{p}\\FILEa"})
+        self.assertEqual(set(map(str, p.glob("F*a"))), {f"{p}\\fileA"})
 
     def test_rglob(self):
         P = self.cls
         p = P(BASE, "dirC")
         self.assertEqual(set(p.rglob("FILEd")), { P(BASE, "dirC/dirD/fileD") })
+        self.assertEqual(set(map(str, p.rglob("FILEd"))), {f"{p}\\dirD\\FILEd"})
 
     def test_expanduser(self):
         P = self.cls
diff --git a/Misc/NEWS.d/next/Library/2019-10-20-12-04-48.bpo-31202.NfdIus.rst b/Misc/NEWS.d/next/Library/2019-10-20-12-04-48.bpo-31202.NfdIus.rst
new file mode 100644
index 0000000000000..8edb09d61317b
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-10-20-12-04-48.bpo-31202.NfdIus.rst
@@ -0,0 +1,2 @@
+The case the result of :func:`pathlib.WindowsPath.glob` matches now the case
+of the pattern for literal parts.


From webhook-mailer at python.org  Mon Oct 21 14:18:06 2019
From: webhook-mailer at python.org (Miss Skeleton (bot))
Date: Mon, 21 Oct 2019 18:18:06 -0000
Subject: [Python-checkins] bpo-31202: Preserve case of literal parts in
 Path.glob() on Windows. (GH-16860)
Message-ID: <mailman.94.1571681886.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/2f8d4f08e2fa62cd5c3f6f824be3e7513ff87e07
commit: 2f8d4f08e2fa62cd5c3f6f824be3e7513ff87e07
branch: 3.8
author: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-21T11:17:57-07:00
summary:

bpo-31202: Preserve case of literal parts in Path.glob() on Windows. (GH-16860)

(cherry picked from commit 10ecbadb799ddf3393d1fc80119a3db14724d381)

Co-authored-by: Serhiy Storchaka <storchaka at gmail.com>

files:
A Misc/NEWS.d/next/Library/2019-10-20-12-04-48.bpo-31202.NfdIus.rst
M Lib/pathlib.py
M Lib/test/test_pathlib.py

diff --git a/Lib/pathlib.py b/Lib/pathlib.py
index 91ce4a125740a..70b011b0e8c4f 100644
--- a/Lib/pathlib.py
+++ b/Lib/pathlib.py
@@ -187,6 +187,9 @@ def casefold(self, s):
     def casefold_parts(self, parts):
         return [p.lower() for p in parts]
 
+    def compile_pattern(self, pattern):
+        return re.compile(fnmatch.translate(pattern), re.IGNORECASE).fullmatch
+
     def resolve(self, path, strict=False):
         s = str(path)
         if not s:
@@ -309,6 +312,9 @@ def casefold(self, s):
     def casefold_parts(self, parts):
         return parts
 
+    def compile_pattern(self, pattern):
+        return re.compile(fnmatch.translate(pattern)).fullmatch
+
     def resolve(self, path, strict=False):
         sep = self.sep
         accessor = path._accessor
@@ -446,7 +452,7 @@ def readlink(self, path):
 # Globbing helpers
 #
 
-def _make_selector(pattern_parts):
+def _make_selector(pattern_parts, flavour):
     pat = pattern_parts[0]
     child_parts = pattern_parts[1:]
     if pat == '**':
@@ -457,7 +463,7 @@ def _make_selector(pattern_parts):
         cls = _WildcardSelector
     else:
         cls = _PreciseSelector
-    return cls(pat, child_parts)
+    return cls(pat, child_parts, flavour)
 
 if hasattr(functools, "lru_cache"):
     _make_selector = functools.lru_cache()(_make_selector)
@@ -467,10 +473,10 @@ class _Selector:
     """A selector matches a specific glob pattern part against the children
     of a given path."""
 
-    def __init__(self, child_parts):
+    def __init__(self, child_parts, flavour):
         self.child_parts = child_parts
         if child_parts:
-            self.successor = _make_selector(child_parts)
+            self.successor = _make_selector(child_parts, flavour)
             self.dironly = True
         else:
             self.successor = _TerminatingSelector()
@@ -496,9 +502,9 @@ def _select_from(self, parent_path, is_dir, exists, scandir):
 
 class _PreciseSelector(_Selector):
 
-    def __init__(self, name, child_parts):
+    def __init__(self, name, child_parts, flavour):
         self.name = name
-        _Selector.__init__(self, child_parts)
+        _Selector.__init__(self, child_parts, flavour)
 
     def _select_from(self, parent_path, is_dir, exists, scandir):
         try:
@@ -512,13 +518,12 @@ def _select_from(self, parent_path, is_dir, exists, scandir):
 
 class _WildcardSelector(_Selector):
 
-    def __init__(self, pat, child_parts):
-        self.pat = re.compile(fnmatch.translate(pat))
-        _Selector.__init__(self, child_parts)
+    def __init__(self, pat, child_parts, flavour):
+        self.match = flavour.compile_pattern(pat)
+        _Selector.__init__(self, child_parts, flavour)
 
     def _select_from(self, parent_path, is_dir, exists, scandir):
         try:
-            cf = parent_path._flavour.casefold
             entries = list(scandir(parent_path))
             for entry in entries:
                 entry_is_dir = False
@@ -529,8 +534,7 @@ def _select_from(self, parent_path, is_dir, exists, scandir):
                         raise
                 if not self.dironly or entry_is_dir:
                     name = entry.name
-                    casefolded = cf(name)
-                    if self.pat.match(casefolded):
+                    if self.match(name):
                         path = parent_path._make_child_relpath(name)
                         for p in self.successor._select_from(path, is_dir, exists, scandir):
                             yield p
@@ -541,8 +545,8 @@ def _select_from(self, parent_path, is_dir, exists, scandir):
 
 class _RecursiveWildcardSelector(_Selector):
 
-    def __init__(self, pat, child_parts):
-        _Selector.__init__(self, child_parts)
+    def __init__(self, pat, child_parts, flavour):
+        _Selector.__init__(self, child_parts, flavour)
 
     def _iterate_directories(self, parent_path, is_dir, scandir):
         yield parent_path
@@ -1109,11 +1113,10 @@ def glob(self, pattern):
         """
         if not pattern:
             raise ValueError("Unacceptable pattern: {!r}".format(pattern))
-        pattern = self._flavour.casefold(pattern)
         drv, root, pattern_parts = self._flavour.parse_parts((pattern,))
         if drv or root:
             raise NotImplementedError("Non-relative patterns are unsupported")
-        selector = _make_selector(tuple(pattern_parts))
+        selector = _make_selector(tuple(pattern_parts), self._flavour)
         for p in selector.select_from(self):
             yield p
 
@@ -1122,11 +1125,10 @@ def rglob(self, pattern):
         directories) matching the given relative pattern, anywhere in
         this subtree.
         """
-        pattern = self._flavour.casefold(pattern)
         drv, root, pattern_parts = self._flavour.parse_parts((pattern,))
         if drv or root:
             raise NotImplementedError("Non-relative patterns are unsupported")
-        selector = _make_selector(("**",) + tuple(pattern_parts))
+        selector = _make_selector(("**",) + tuple(pattern_parts), self._flavour)
         for p in selector.select_from(self):
             yield p
 
diff --git a/Lib/test/test_pathlib.py b/Lib/test/test_pathlib.py
index 95d0b09882f6c..4543e3773dbf8 100644
--- a/Lib/test/test_pathlib.py
+++ b/Lib/test/test_pathlib.py
@@ -2291,11 +2291,15 @@ def test_glob(self):
         P = self.cls
         p = P(BASE)
         self.assertEqual(set(p.glob("FILEa")), { P(BASE, "fileA") })
+        self.assertEqual(set(p.glob("F*a")), { P(BASE, "fileA") })
+        self.assertEqual(set(map(str, p.glob("FILEa"))), {f"{p}\\FILEa"})
+        self.assertEqual(set(map(str, p.glob("F*a"))), {f"{p}\\fileA"})
 
     def test_rglob(self):
         P = self.cls
         p = P(BASE, "dirC")
         self.assertEqual(set(p.rglob("FILEd")), { P(BASE, "dirC/dirD/fileD") })
+        self.assertEqual(set(map(str, p.rglob("FILEd"))), {f"{p}\\dirD\\FILEd"})
 
     def test_expanduser(self):
         P = self.cls
diff --git a/Misc/NEWS.d/next/Library/2019-10-20-12-04-48.bpo-31202.NfdIus.rst b/Misc/NEWS.d/next/Library/2019-10-20-12-04-48.bpo-31202.NfdIus.rst
new file mode 100644
index 0000000000000..8edb09d61317b
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-10-20-12-04-48.bpo-31202.NfdIus.rst
@@ -0,0 +1,2 @@
+The case the result of :func:`pathlib.WindowsPath.glob` matches now the case
+of the pattern for literal parts.


From webhook-mailer at python.org  Mon Oct 21 14:40:38 2019
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Mon, 21 Oct 2019 18:40:38 -0000
Subject: [Python-checkins] [2.7] bpo-38540: Fix possible leak in PyArg_Parse
 for "es#" and "et#". (GH-16869). (GH-16877)
Message-ID: <mailman.95.1571683239.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/ccdfeb7e969bf3aafd43dbe6581c30f66f2b0890
commit: ccdfeb7e969bf3aafd43dbe6581c30f66f2b0890
branch: 2.7
author: Serhiy Storchaka <storchaka at gmail.com>
committer: GitHub <noreply at github.com>
date: 2019-10-21T21:40:30+03:00
summary:

[2.7] bpo-38540: Fix possible leak in PyArg_Parse for "es#" and "et#". (GH-16869). (GH-16877)

(cherry picked from commit 5bc6a7c06eda20ba131ecba6752be0506d310181)

files:
A Misc/NEWS.d/next/C API/2019-10-21-09-24-03.bpo-38540.314N_T.rst
M Python/getargs.c

diff --git a/Misc/NEWS.d/next/C API/2019-10-21-09-24-03.bpo-38540.314N_T.rst b/Misc/NEWS.d/next/C API/2019-10-21-09-24-03.bpo-38540.314N_T.rst
new file mode 100644
index 0000000000000..1d73ad8fe96e6
--- /dev/null
+++ b/Misc/NEWS.d/next/C API/2019-10-21-09-24-03.bpo-38540.314N_T.rst	
@@ -0,0 +1,3 @@
+Fixed possible leak in :c:func:`PyArg_Parse` and similar functions for
+format units ``"es#"`` and ``"et#"`` when the macro
+:c:macro:`PY_SSIZE_T_CLEAN` is not defined.
diff --git a/Python/getargs.c b/Python/getargs.c
index cc1ddde977f55..12ee2486377f6 100644
--- a/Python/getargs.c
+++ b/Python/getargs.c
@@ -1156,7 +1156,19 @@ convertsimple(PyObject *arg, const char **p_format, va_list *p_va, int flags,
             memcpy(*buffer,
                    PyString_AS_STRING(s),
                    size + 1);
-            STORE_SIZE(size);
+
+            if (flags & FLAG_SIZE_T) {
+                *q2 = size;
+            }
+            else {
+                if (INT_MAX < size) {
+                    Py_DECREF(s);
+                    PyErr_SetString(PyExc_OverflowError,
+                                    "size does not fit in an int");
+                    return converterr("", arg, msgbuf, bufsize);
+                }
+                *q = (int)size;
+            }
         } else {
             /* Using a 0-terminated buffer:
 


From webhook-mailer at python.org  Mon Oct 21 14:41:13 2019
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Mon, 21 Oct 2019 18:41:13 -0000
Subject: [Python-checkins] bpo-38540: Revert a warning if PY_SSIZE_T_CLEAN
 is not defined. (GH-16876)
Message-ID: <mailman.96.1571683274.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/928c68eee620cc6b624a195375143c2aa55644a4
commit: 928c68eee620cc6b624a195375143c2aa55644a4
branch: 3.7
author: Serhiy Storchaka <storchaka at gmail.com>
committer: GitHub <noreply at github.com>
date: 2019-10-21T21:41:09+03:00
summary:

bpo-38540: Revert a warning if PY_SSIZE_T_CLEAN is not defined. (GH-16876)

files:
M Python/getargs.c

diff --git a/Python/getargs.c b/Python/getargs.c
index c3895d1d6c594..1c53361c5e6e1 100644
--- a/Python/getargs.c
+++ b/Python/getargs.c
@@ -1176,19 +1176,7 @@ convertsimple(PyObject *arg, const char **p_format, va_list *p_va, int flags,
                trailing 0-byte
 
             */
-            int *q = NULL; Py_ssize_t *q2 = NULL;
-            if (flags & FLAG_SIZE_T) {
-                q2 = va_arg(*p_va, Py_ssize_t*);
-            }
-            else {
-                if (PyErr_WarnEx(PyExc_DeprecationWarning,
-                            "PY_SSIZE_T_CLEAN will be required for '#' formats", 1))
-                {
-                    Py_DECREF(s);
-                    return NULL;
-                }
-                q = va_arg(*p_va, int*);
-            }
+            FETCH_SIZE;
 
             format++;
             if (q == NULL && q2 == NULL) {


From webhook-mailer at python.org  Tue Oct 22 05:43:16 2019
From: webhook-mailer at python.org (Terry Jan Reedy)
Date: Tue, 22 Oct 2019 09:43:16 -0000
Subject: [Python-checkins] Fix Zope URL (GH-16880)
Message-ID: <mailman.97.1571737397.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/dfe726b1ace03f206f45253b93ed7610473ae20f
commit: dfe726b1ace03f206f45253b93ed7610473ae20f
branch: master
author: Kyle Stanley <aeros167 at gmail.com>
committer: Terry Jan Reedy <tjreedy at udel.edu>
date: 2019-10-22T05:43:10-04:00
summary:

Fix Zope URL (GH-16880)

files:
M Doc/license.rst

diff --git a/Doc/license.rst b/Doc/license.rst
index d877f45677549..810d2e63fd48c 100644
--- a/Doc/license.rst
+++ b/Doc/license.rst
@@ -22,7 +22,7 @@ Virginia where he released several versions of the software.
 In May 2000, Guido and the Python core development team moved to BeOpen.com to
 form the BeOpen PythonLabs team.  In October of the same year, the PythonLabs
 team moved to Digital Creations (now Zope Corporation; see
-http://www.zope.com/).  In 2001, the Python Software Foundation (PSF, see
+https://www.zope.org/).  In 2001, the Python Software Foundation (PSF, see
 https://www.python.org/psf/) was formed, a non-profit organization created
 specifically to own Python-related Intellectual Property.  Zope Corporation is a
 sponsoring member of the PSF.


From webhook-mailer at python.org  Tue Oct 22 05:48:37 2019
From: webhook-mailer at python.org (Miss Skeleton (bot))
Date: Tue, 22 Oct 2019 09:48:37 -0000
Subject: [Python-checkins] Fix Zope URL (GH-16880)
Message-ID: <mailman.98.1571737718.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/9978a9553b5273ca8f55c4130383c4ee151fbe2a
commit: 9978a9553b5273ca8f55c4130383c4ee151fbe2a
branch: 2.7
author: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-22T02:48:33-07:00
summary:

Fix Zope URL (GH-16880)

(cherry picked from commit dfe726b1ace03f206f45253b93ed7610473ae20f)

Co-authored-by: Kyle Stanley <aeros167 at gmail.com>

files:
M Doc/license.rst

diff --git a/Doc/license.rst b/Doc/license.rst
index eded4a935ddfc..56687ac224850 100644
--- a/Doc/license.rst
+++ b/Doc/license.rst
@@ -22,7 +22,7 @@ Virginia where he released several versions of the software.
 In May 2000, Guido and the Python core development team moved to BeOpen.com to
 form the BeOpen PythonLabs team.  In October of the same year, the PythonLabs
 team moved to Digital Creations (now Zope Corporation; see
-http://www.zope.com/).  In 2001, the Python Software Foundation (PSF, see
+https://www.zope.org/).  In 2001, the Python Software Foundation (PSF, see
 https://www.python.org/psf/) was formed, a non-profit organization created
 specifically to own Python-related Intellectual Property.  Zope Corporation is a
 sponsoring member of the PSF.


From webhook-mailer at python.org  Tue Oct 22 05:49:36 2019
From: webhook-mailer at python.org (Miss Skeleton (bot))
Date: Tue, 22 Oct 2019 09:49:36 -0000
Subject: [Python-checkins] Fix Zope URL (GH-16880)
Message-ID: <mailman.99.1571737777.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/d4dc4a58f5c124463308701c3aba985c1fe5cc0e
commit: d4dc4a58f5c124463308701c3aba985c1fe5cc0e
branch: 3.7
author: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-22T02:49:32-07:00
summary:

Fix Zope URL (GH-16880)

(cherry picked from commit dfe726b1ace03f206f45253b93ed7610473ae20f)

Co-authored-by: Kyle Stanley <aeros167 at gmail.com>

files:
M Doc/license.rst

diff --git a/Doc/license.rst b/Doc/license.rst
index ac0e2b30bbbfd..d9a4314f057f8 100644
--- a/Doc/license.rst
+++ b/Doc/license.rst
@@ -22,7 +22,7 @@ Virginia where he released several versions of the software.
 In May 2000, Guido and the Python core development team moved to BeOpen.com to
 form the BeOpen PythonLabs team.  In October of the same year, the PythonLabs
 team moved to Digital Creations (now Zope Corporation; see
-http://www.zope.com/).  In 2001, the Python Software Foundation (PSF, see
+https://www.zope.org/).  In 2001, the Python Software Foundation (PSF, see
 https://www.python.org/psf/) was formed, a non-profit organization created
 specifically to own Python-related Intellectual Property.  Zope Corporation is a
 sponsoring member of the PSF.


From webhook-mailer at python.org  Tue Oct 22 09:07:15 2019
From: webhook-mailer at python.org (Miss Skeleton (bot))
Date: Tue, 22 Oct 2019 13:07:15 -0000
Subject: [Python-checkins] bpo-38303: Make audioop extension module PEP-384
 compatible (GH-16497)
Message-ID: <mailman.100.1571749637.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/f548a3e4a28c5496755f9f8d057fc8d031b27655
commit: f548a3e4a28c5496755f9f8d057fc8d031b27655
branch: master
author: Tyler Kieft <tyler.kieft at gmail.com>
committer: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
date: 2019-10-22T06:07:03-07:00
summary:

bpo-38303: Make audioop extension module PEP-384 compatible (GH-16497)



https://bugs.python.org/issue38303



Automerge-Triggered-By: @encukou

files:
A Misc/NEWS.d/next/C API/2019-09-30-16-53-30.bpo-38303.YoIs0M.rst
M Misc/ACKS
M Modules/audioop.c

diff --git a/Misc/ACKS b/Misc/ACKS
index d8e2630814a86..78beee5f34eec 100644
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -847,6 +847,7 @@ Dhiru Kholia
 Artem Khramov
 Akshit Khurana
 Sanyam Khurana
+Tyler Kieft
 Mads Kiilerich
 Jason Killen
 Jan Kim
diff --git a/Misc/NEWS.d/next/C API/2019-09-30-16-53-30.bpo-38303.YoIs0M.rst b/Misc/NEWS.d/next/C API/2019-09-30-16-53-30.bpo-38303.YoIs0M.rst
new file mode 100644
index 0000000000000..03048bbfce533
--- /dev/null
+++ b/Misc/NEWS.d/next/C API/2019-09-30-16-53-30.bpo-38303.YoIs0M.rst	
@@ -0,0 +1 @@
+Update audioop extension module to use the stable ABI (PEP-384). Patch by Tyler Kieft.
diff --git a/Modules/audioop.c b/Modules/audioop.c
index f4fdeb23ffa95..7726c88b1a9d4 100644
--- a/Modules/audioop.c
+++ b/Modules/audioop.c
@@ -371,14 +371,19 @@ static const int stepsizeTable[89] = {
             SETINT32((cp), (i), (val));         \
     } while(0)
 
+static PyModuleDef audioopmodule;
 
-static PyObject *AudioopError;
+typedef struct {
+    PyObject *AudioopError;
+} _audioopstate;
+
+#define _audioopstate(o) ((_audioopstate *)PyModule_GetState(o))
 
 static int
-audioop_check_size(int size)
+audioop_check_size(PyObject *module, int size)
 {
     if (size < 1 || size > 4) {
-        PyErr_SetString(AudioopError, "Size should be 1, 2, 3 or 4");
+        PyErr_SetString(_audioopstate(module)->AudioopError, "Size should be 1, 2, 3 or 4");
         return 0;
     }
     else
@@ -386,12 +391,12 @@ audioop_check_size(int size)
 }
 
 static int
-audioop_check_parameters(Py_ssize_t len, int size)
+audioop_check_parameters(PyObject *module, Py_ssize_t len, int size)
 {
-    if (!audioop_check_size(size))
+    if (!audioop_check_size(module, size))
         return 0;
     if (len % size != 0) {
-        PyErr_SetString(AudioopError, "not a whole number of frames");
+        PyErr_SetString(_audioopstate(module)->AudioopError, "not a whole number of frames");
         return 0;
     }
     return 1;
@@ -420,10 +425,10 @@ audioop_getsample_impl(PyObject *module, Py_buffer *fragment, int width,
 {
     int val;
 
-    if (!audioop_check_parameters(fragment->len, width))
+    if (!audioop_check_parameters(module, fragment->len, width))
         return NULL;
     if (index < 0 || index >= fragment->len/width) {
-        PyErr_SetString(AudioopError, "Index out of range");
+        PyErr_SetString(_audioopstate(module)->AudioopError, "Index out of range");
         return NULL;
     }
     val = GETRAWSAMPLE(width, fragment->buf, index*width);
@@ -447,7 +452,7 @@ audioop_max_impl(PyObject *module, Py_buffer *fragment, int width)
     Py_ssize_t i;
     unsigned int absval, max = 0;
 
-    if (!audioop_check_parameters(fragment->len, width))
+    if (!audioop_check_parameters(module, fragment->len, width))
         return NULL;
     for (i = 0; i < fragment->len; i += width) {
         int val = GETRAWSAMPLE(width, fragment->buf, i);
@@ -479,7 +484,7 @@ audioop_minmax_impl(PyObject *module, Py_buffer *fragment, int width)
     a warning */
     int min = 0x7fffffff, max = -0x7FFFFFFF-1;
 
-    if (!audioop_check_parameters(fragment->len, width))
+    if (!audioop_check_parameters(module, fragment->len, width))
         return NULL;
     for (i = 0; i < fragment->len; i += width) {
         int val = GETRAWSAMPLE(width, fragment->buf, i);
@@ -507,7 +512,7 @@ audioop_avg_impl(PyObject *module, Py_buffer *fragment, int width)
     int avg;
     double sum = 0.0;
 
-    if (!audioop_check_parameters(fragment->len, width))
+    if (!audioop_check_parameters(module, fragment->len, width))
         return NULL;
     for (i = 0; i < fragment->len; i += width)
         sum += GETRAWSAMPLE(width, fragment->buf, i);
@@ -536,7 +541,7 @@ audioop_rms_impl(PyObject *module, Py_buffer *fragment, int width)
     unsigned int res;
     double sum_squares = 0.0;
 
-    if (!audioop_check_parameters(fragment->len, width))
+    if (!audioop_check_parameters(module, fragment->len, width))
         return NULL;
     for (i = 0; i < fragment->len; i += width) {
         double val = GETRAWSAMPLE(width, fragment->buf, i);
@@ -614,7 +619,7 @@ audioop_findfit_impl(PyObject *module, Py_buffer *fragment,
     double sum_ri_2, sum_aij_2, sum_aij_ri, result, best_result, factor;
 
     if (fragment->len & 1 || reference->len & 1) {
-        PyErr_SetString(AudioopError, "Strings should be even-sized");
+        PyErr_SetString(_audioopstate(module)->AudioopError, "Strings should be even-sized");
         return NULL;
     }
     cp1 = (const int16_t *)fragment->buf;
@@ -623,7 +628,7 @@ audioop_findfit_impl(PyObject *module, Py_buffer *fragment,
     len2 = reference->len >> 1;
 
     if (len1 < len2) {
-        PyErr_SetString(AudioopError, "First sample should be longer");
+        PyErr_SetString(_audioopstate(module)->AudioopError, "First sample should be longer");
         return NULL;
     }
     sum_ri_2 = _sum2(cp2, cp2, len2);
@@ -681,11 +686,11 @@ audioop_findfactor_impl(PyObject *module, Py_buffer *fragment,
     double sum_ri_2, sum_aij_ri, result;
 
     if (fragment->len & 1 || reference->len & 1) {
-        PyErr_SetString(AudioopError, "Strings should be even-sized");
+        PyErr_SetString(_audioopstate(module)->AudioopError, "Strings should be even-sized");
         return NULL;
     }
     if (fragment->len != reference->len) {
-        PyErr_SetString(AudioopError, "Samples should be same size");
+        PyErr_SetString(_audioopstate(module)->AudioopError, "Samples should be same size");
         return NULL;
     }
     cp1 = (const int16_t *)fragment->buf;
@@ -725,14 +730,14 @@ audioop_findmax_impl(PyObject *module, Py_buffer *fragment,
     double result, best_result;
 
     if (fragment->len & 1) {
-        PyErr_SetString(AudioopError, "Strings should be even-sized");
+        PyErr_SetString(_audioopstate(module)->AudioopError, "Strings should be even-sized");
         return NULL;
     }
     cp1 = (const int16_t *)fragment->buf;
     len1 = fragment->len >> 1;
 
     if (length < 0 || len1 < length) {
-        PyErr_SetString(AudioopError, "Input sample should be longer");
+        PyErr_SetString(_audioopstate(module)->AudioopError, "Input sample should be longer");
         return NULL;
     }
 
@@ -777,7 +782,7 @@ audioop_avgpp_impl(PyObject *module, Py_buffer *fragment, int width)
     unsigned int avg;
     int diff, prevdiff, nextreme = 0;
 
-    if (!audioop_check_parameters(fragment->len, width))
+    if (!audioop_check_parameters(module, fragment->len, width))
         return NULL;
     if (fragment->len <= width)
         return PyLong_FromLong(0);
@@ -833,7 +838,7 @@ audioop_maxpp_impl(PyObject *module, Py_buffer *fragment, int width)
     unsigned int max = 0, extremediff;
     int diff, prevdiff;
 
-    if (!audioop_check_parameters(fragment->len, width))
+    if (!audioop_check_parameters(module, fragment->len, width))
         return NULL;
     if (fragment->len <= width)
         return PyLong_FromLong(0);
@@ -885,7 +890,7 @@ audioop_cross_impl(PyObject *module, Py_buffer *fragment, int width)
     int prevval;
     Py_ssize_t ncross;
 
-    if (!audioop_check_parameters(fragment->len, width))
+    if (!audioop_check_parameters(module, fragment->len, width))
         return NULL;
     ncross = -1;
     prevval = 17; /* Anything <> 0,1 */
@@ -918,7 +923,7 @@ audioop_mul_impl(PyObject *module, Py_buffer *fragment, int width,
     double maxval, minval;
     PyObject *rv;
 
-    if (!audioop_check_parameters(fragment->len, width))
+    if (!audioop_check_parameters(module, fragment->len, width))
         return NULL;
 
     maxval = (double) maxvals[width];
@@ -961,10 +966,10 @@ audioop_tomono_impl(PyObject *module, Py_buffer *fragment, int width,
 
     cp = fragment->buf;
     len = fragment->len;
-    if (!audioop_check_parameters(len, width))
+    if (!audioop_check_parameters(module, len, width))
         return NULL;
     if (((len / width) & 1) != 0) {
-        PyErr_SetString(AudioopError, "not a whole number of frames");
+        PyErr_SetString(_audioopstate(module)->AudioopError, "not a whole number of frames");
         return NULL;
     }
 
@@ -1008,7 +1013,7 @@ audioop_tostereo_impl(PyObject *module, Py_buffer *fragment, int width,
     double maxval, minval;
     PyObject *rv;
 
-    if (!audioop_check_parameters(fragment->len, width))
+    if (!audioop_check_parameters(module, fragment->len, width))
         return NULL;
 
     maxval = (double) maxvals[width];
@@ -1056,10 +1061,10 @@ audioop_add_impl(PyObject *module, Py_buffer *fragment1,
     int minval, maxval, newval;
     PyObject *rv;
 
-    if (!audioop_check_parameters(fragment1->len, width))
+    if (!audioop_check_parameters(module, fragment1->len, width))
         return NULL;
     if (fragment1->len != fragment2->len) {
-        PyErr_SetString(AudioopError, "Lengths should be the same");
+        PyErr_SetString(_audioopstate(module)->AudioopError, "Lengths should be the same");
         return NULL;
     }
 
@@ -1114,7 +1119,7 @@ audioop_bias_impl(PyObject *module, Py_buffer *fragment, int width, int bias)
     unsigned int val = 0, mask;
     PyObject *rv;
 
-    if (!audioop_check_parameters(fragment->len, width))
+    if (!audioop_check_parameters(module, fragment->len, width))
         return NULL;
 
     rv = PyBytes_FromStringAndSize(NULL, fragment->len);
@@ -1172,7 +1177,7 @@ audioop_reverse_impl(PyObject *module, Py_buffer *fragment, int width)
     Py_ssize_t i;
     PyObject *rv;
 
-    if (!audioop_check_parameters(fragment->len, width))
+    if (!audioop_check_parameters(module, fragment->len, width))
         return NULL;
 
     rv = PyBytes_FromStringAndSize(NULL, fragment->len);
@@ -1205,7 +1210,7 @@ audioop_byteswap_impl(PyObject *module, Py_buffer *fragment, int width)
     Py_ssize_t i;
     PyObject *rv;
 
-    if (!audioop_check_parameters(fragment->len, width))
+    if (!audioop_check_parameters(module, fragment->len, width))
         return NULL;
 
     rv = PyBytes_FromStringAndSize(NULL, fragment->len);
@@ -1241,9 +1246,9 @@ audioop_lin2lin_impl(PyObject *module, Py_buffer *fragment, int width,
     Py_ssize_t i, j;
     PyObject *rv;
 
-    if (!audioop_check_parameters(fragment->len, width))
+    if (!audioop_check_parameters(module, fragment->len, width))
         return NULL;
-    if (!audioop_check_size(newwidth))
+    if (!audioop_check_size(module, newwidth))
         return NULL;
 
     if (fragment->len/width > PY_SSIZE_T_MAX/newwidth) {
@@ -1302,10 +1307,10 @@ audioop_ratecv_impl(PyObject *module, Py_buffer *fragment, int width,
     PyObject *samps, *str, *rv = NULL, *channel;
     int bytes_per_frame;
 
-    if (!audioop_check_size(width))
+    if (!audioop_check_size(module, width))
         return NULL;
     if (nchannels < 1) {
-        PyErr_SetString(AudioopError, "# of channels should be >= 1");
+        PyErr_SetString(_audioopstate(module)->AudioopError, "# of channels should be >= 1");
         return NULL;
     }
     if (width > INT_MAX / nchannels) {
@@ -1318,17 +1323,17 @@ audioop_ratecv_impl(PyObject *module, Py_buffer *fragment, int width,
     }
     bytes_per_frame = width * nchannels;
     if (weightA < 1 || weightB < 0) {
-        PyErr_SetString(AudioopError,
+        PyErr_SetString(_audioopstate(module)->AudioopError,
             "weightA should be >= 1, weightB should be >= 0");
         return NULL;
     }
     assert(fragment->len >= 0);
     if (fragment->len % bytes_per_frame != 0) {
-        PyErr_SetString(AudioopError, "not a whole number of frames");
+        PyErr_SetString(_audioopstate(module)->AudioopError, "not a whole number of frames");
         return NULL;
     }
     if (inrate <= 0 || outrate <= 0) {
-        PyErr_SetString(AudioopError, "sampling rate not > 0");
+        PyErr_SetString(_audioopstate(module)->AudioopError, "sampling rate not > 0");
         return NULL;
     }
     /* divide inrate and outrate by their greatest common divisor */
@@ -1369,7 +1374,7 @@ audioop_ratecv_impl(PyObject *module, Py_buffer *fragment, int width,
                         &d, &PyTuple_Type, &samps))
             goto exit;
         if (PyTuple_Size(samps) != nchannels) {
-            PyErr_SetString(AudioopError,
+            PyErr_SetString(_audioopstate(module)->AudioopError,
                             "illegal state argument");
             goto exit;
         }
@@ -1491,7 +1496,7 @@ audioop_lin2ulaw_impl(PyObject *module, Py_buffer *fragment, int width)
     Py_ssize_t i;
     PyObject *rv;
 
-    if (!audioop_check_parameters(fragment->len, width))
+    if (!audioop_check_parameters(module, fragment->len, width))
         return NULL;
 
     rv = PyBytes_FromStringAndSize(NULL, fragment->len/width);
@@ -1525,7 +1530,7 @@ audioop_ulaw2lin_impl(PyObject *module, Py_buffer *fragment, int width)
     Py_ssize_t i;
     PyObject *rv;
 
-    if (!audioop_check_size(width))
+    if (!audioop_check_size(module, width))
         return NULL;
 
     if (fragment->len > PY_SSIZE_T_MAX/width) {
@@ -1564,7 +1569,7 @@ audioop_lin2alaw_impl(PyObject *module, Py_buffer *fragment, int width)
     Py_ssize_t i;
     PyObject *rv;
 
-    if (!audioop_check_parameters(fragment->len, width))
+    if (!audioop_check_parameters(module, fragment->len, width))
         return NULL;
 
     rv = PyBytes_FromStringAndSize(NULL, fragment->len/width);
@@ -1599,7 +1604,7 @@ audioop_alaw2lin_impl(PyObject *module, Py_buffer *fragment, int width)
     int val;
     PyObject *rv;
 
-    if (!audioop_check_size(width))
+    if (!audioop_check_size(module, width))
         return NULL;
 
     if (fragment->len > PY_SSIZE_T_MAX/width) {
@@ -1643,7 +1648,7 @@ audioop_lin2adpcm_impl(PyObject *module, Py_buffer *fragment, int width,
     PyObject *rv = NULL, *str;
     int outputbuffer = 0, bufferstep;
 
-    if (!audioop_check_parameters(fragment->len, width))
+    if (!audioop_check_parameters(module, fragment->len, width))
         return NULL;
 
     /* Decode state, should have (value, step) */
@@ -1773,7 +1778,7 @@ audioop_adpcm2lin_impl(PyObject *module, Py_buffer *fragment, int width,
     PyObject *rv, *str;
     int inputbuffer = 0, bufferstep;
 
-    if (!audioop_check_size(width))
+    if (!audioop_check_size(module, width))
         return NULL;
 
     /* Decode state, should have (value, step) */
@@ -1897,31 +1902,48 @@ static PyMethodDef audioop_methods[] = {
     { 0,          0 }
 };
 
+static int
+audioop_traverse(PyObject *m, visitproc visit, void *arg) {
+    _audioopstate *state = _audioopstate(m);
+    if (state != NULL)
+        Py_VISIT(state->AudioopError);
+    return 0;
+}
+static int
+audioop_clear(PyObject *m) {
+    _audioopstate *state = _audioopstate(m);
+    if (state != NULL)
+        Py_CLEAR(state->AudioopError);
+    return 0;
+}
+static void
+audioop_free(void *m) {
+    audioop_clear((PyObject *)m);
+}
 
 static struct PyModuleDef audioopmodule = {
     PyModuleDef_HEAD_INIT,
     "audioop",
     NULL,
-    -1,
+    sizeof(_audioopstate),
     audioop_methods,
     NULL,
-    NULL,
-    NULL,
-    NULL
+    audioop_traverse,
+    audioop_clear,
+    audioop_free
 };
 
 PyMODINIT_FUNC
 PyInit_audioop(void)
 {
-    PyObject *m, *d;
-    m = PyModule_Create(&audioopmodule);
+    PyObject *m = PyModule_Create(&audioopmodule);
     if (m == NULL)
         return NULL;
-    d = PyModule_GetDict(m);
-    if (d == NULL)
+    PyObject *AudioopError = PyErr_NewException("audioop.error", NULL, NULL);
+    if (AudioopError == NULL)
         return NULL;
-    AudioopError = PyErr_NewException("audioop.error", NULL, NULL);
-    if (AudioopError != NULL)
-         PyDict_SetItemString(d,"error",AudioopError);
+    Py_INCREF(AudioopError);
+    PyModule_AddObject(m, "error", AudioopError);
+    _audioopstate(m)->AudioopError = AudioopError;
     return m;
 }


From webhook-mailer at python.org  Tue Oct 22 12:14:07 2019
From: webhook-mailer at python.org (Julien Palard)
Date: Tue, 22 Oct 2019 16:14:07 -0000
Subject: [Python-checkins] Bump Sphinx to 2.2.0. (GH-16532)
Message-ID: <mailman.101.1571760847.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/c987090c731a363c58a02d0efcb3c2d49ccf61ed
commit: c987090c731a363c58a02d0efcb3c2d49ccf61ed
branch: master
author: Julien Palard <julien at palard.fr>
committer: GitHub <noreply at github.com>
date: 2019-10-22T18:13:41+02:00
summary:

Bump Sphinx to 2.2.0. (GH-16532)

files:
M .azure-pipelines/docs-steps.yml
M .travis.yml
M Doc/Makefile
M Doc/requirements.txt
M Mac/BuildScript/build-installer.py

diff --git a/.azure-pipelines/docs-steps.yml b/.azure-pipelines/docs-steps.yml
index 96361961ea75e..eaf4ad01927ae 100644
--- a/.azure-pipelines/docs-steps.yml
+++ b/.azure-pipelines/docs-steps.yml
@@ -12,7 +12,7 @@ steps:
   inputs:
     versionSpec: '>=3.6'
 
-- script: python -m pip install sphinx==2.0.1 blurb python-docs-theme
+- script: python -m pip install sphinx==2.2.0 blurb python-docs-theme
   displayName: 'Install build dependencies'
 
 - ${{ if ne(parameters.latex, 'true') }}:
@@ -21,7 +21,7 @@ steps:
     displayName: 'Build documentation'
 
 - ${{ if eq(parameters.latex, 'true') }}:
-  - script: sudo apt-get update && sudo apt-get install -qy --force-yes texlive-full 
+  - script: sudo apt-get update && sudo apt-get install -qy --force-yes texlive-full
     displayName: 'Install LaTeX'
 
   - script: make dist PYTHON=python SPHINXBUILD='python -m sphinx' BLURB='python -m blurb'
diff --git a/.travis.yml b/.travis.yml
index d4275fde64243..807a7b2f24a12 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -55,7 +55,7 @@ matrix:
         # Sphinx is pinned so that new versions that introduce new warnings won't suddenly cause build failures.
         # (Updating the version is fine as long as no warnings are raised by doing so.)
         # The theme used by the docs is stored separately, so we need to install that as well.
-        - python -m pip install sphinx==2.0.1 blurb python-docs-theme
+        - python -m pip install sphinx==2.2.0 blurb python-docs-theme
       script:
         - make check suspicious html SPHINXOPTS="-q -W -j4"
     - name: "Documentation tests"
diff --git a/Doc/Makefile b/Doc/Makefile
index 86207fe57a0f6..05361f2ee2c82 100644
--- a/Doc/Makefile
+++ b/Doc/Makefile
@@ -143,7 +143,7 @@ clean:
 venv:
 	$(PYTHON) -m venv $(VENVDIR)
 	$(VENVDIR)/bin/python3 -m pip install -U pip setuptools
-	$(VENVDIR)/bin/python3 -m pip install -U Sphinx==2.0.1 blurb python-docs-theme
+	$(VENVDIR)/bin/python3 -m pip install -U Sphinx==2.2.0 blurb python-docs-theme
 	@echo "The venv has been created in the $(VENVDIR) directory"
 
 dist:
diff --git a/Doc/requirements.txt b/Doc/requirements.txt
index e0d7ca869af8c..198446b350ff2 100644
--- a/Doc/requirements.txt
+++ b/Doc/requirements.txt
@@ -1,5 +1,5 @@
 # Requirements for docs build on netlify
 # Pin sphinx to version specified in .travis.yml
-sphinx==2.0.1
+sphinx==2.2.0
 blurb
 python-docs-theme
diff --git a/Mac/BuildScript/build-installer.py b/Mac/BuildScript/build-installer.py
index 0dd53647dd33e..3d1381b895b2f 100755
--- a/Mac/BuildScript/build-installer.py
+++ b/Mac/BuildScript/build-installer.py
@@ -1068,7 +1068,7 @@ def buildPythonDocs():
     runCommand('make clean')
     # Create virtual environment for docs builds with blurb and sphinx
     runCommand('make venv')
-    runCommand('venv/bin/python3 -m pip install -U Sphinx==2.0.1')
+    runCommand('venv/bin/python3 -m pip install -U Sphinx==2.2.0')
     runCommand('make html PYTHON=venv/bin/python')
     os.chdir(curDir)
     if not os.path.exists(docdir):


From webhook-mailer at python.org  Tue Oct 22 15:53:56 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Tue, 22 Oct 2019 19:53:56 -0000
Subject: [Python-checkins] bpo-37415: Fix stdatomic.h header check for ICC
 compiler (GH-16717)
Message-ID: <mailman.102.1571774037.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/028f7349a0f6eaea0fec31becb587fcdf6e3cb28
commit: 028f7349a0f6eaea0fec31becb587fcdf6e3cb28
branch: master
author: Victor Stinner <vstinner at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-22T21:53:50+02:00
summary:

bpo-37415: Fix stdatomic.h header check for ICC compiler (GH-16717)

Fix stdatomic.h header check for ICC compiler: the ICC implementation
lacks atomic_uintptr_t type which is needed by Python.

Test:

* atomic_int and atomic_uintptr_t types
* atomic_load_explicit() and atomic_store_explicit()
* memory_order_relaxed and memory_order_seq_cst constants

But don't test ATOMIC_VAR_INIT(): it's not used in Python.

files:
A Misc/NEWS.d/next/Build/2019-10-11-15-32-58.bpo-37415.D9RXrq.rst
M configure
M configure.ac
M pyconfig.h.in

diff --git a/Misc/NEWS.d/next/Build/2019-10-11-15-32-58.bpo-37415.D9RXrq.rst b/Misc/NEWS.d/next/Build/2019-10-11-15-32-58.bpo-37415.D9RXrq.rst
new file mode 100644
index 0000000000000..98f4a3bf4f545
--- /dev/null
+++ b/Misc/NEWS.d/next/Build/2019-10-11-15-32-58.bpo-37415.D9RXrq.rst
@@ -0,0 +1,2 @@
+Fix stdatomic.h header check for ICC compiler: the ICC implementation lacks
+atomic_uintptr_t type which is needed by Python.
diff --git a/configure b/configure
index 4f094ea650702..50840ac152105 100755
--- a/configure
+++ b/configure
@@ -782,7 +782,6 @@ infodir
 docdir
 oldincludedir
 includedir
-runstatedir
 localstatedir
 sharedstatedir
 sysconfdir
@@ -896,7 +895,6 @@ datadir='${datarootdir}'
 sysconfdir='${prefix}/etc'
 sharedstatedir='${prefix}/com'
 localstatedir='${prefix}/var'
-runstatedir='${localstatedir}/run'
 includedir='${prefix}/include'
 oldincludedir='/usr/include'
 docdir='${datarootdir}/doc/${PACKAGE_TARNAME}'
@@ -1149,15 +1147,6 @@ do
   | -silent | --silent | --silen | --sile | --sil)
     silent=yes ;;
 
-  -runstatedir | --runstatedir | --runstatedi | --runstated \
-  | --runstate | --runstat | --runsta | --runst | --runs \
-  | --run | --ru | --r)
-    ac_prev=runstatedir ;;
-  -runstatedir=* | --runstatedir=* | --runstatedi=* | --runstated=* \
-  | --runstate=* | --runstat=* | --runsta=* | --runst=* | --runs=* \
-  | --run=* | --ru=* | --r=*)
-    runstatedir=$ac_optarg ;;
-
   -sbindir | --sbindir | --sbindi | --sbind | --sbin | --sbi | --sb)
     ac_prev=sbindir ;;
   -sbindir=* | --sbindir=* | --sbindi=* | --sbind=* | --sbin=* \
@@ -1295,7 +1284,7 @@ fi
 for ac_var in	exec_prefix prefix bindir sbindir libexecdir datarootdir \
 		datadir sysconfdir sharedstatedir localstatedir includedir \
 		oldincludedir docdir infodir htmldir dvidir pdfdir psdir \
-		libdir localedir mandir runstatedir
+		libdir localedir mandir
 do
   eval ac_val=\$$ac_var
   # Remove trailing slashes.
@@ -1448,7 +1437,6 @@ Fine tuning of the installation directories:
   --sysconfdir=DIR        read-only single-machine data [PREFIX/etc]
   --sharedstatedir=DIR    modifiable architecture-independent data [PREFIX/com]
   --localstatedir=DIR     modifiable single-machine data [PREFIX/var]
-  --runstatedir=DIR       modifiable per-process data [LOCALSTATEDIR/run]
   --libdir=DIR            object code libraries [EPREFIX/lib]
   --includedir=DIR        C header files [PREFIX/include]
   --oldincludedir=DIR     C header files for non-gcc [/usr/include]
@@ -16775,9 +16763,12 @@ cat confdefs.h - <<_ACEOF >conftest.$ac_ext
 
 
     #include <stdatomic.h>
-    atomic_int value = ATOMIC_VAR_INIT(1);
+    atomic_int int_var;
+    atomic_uintptr_t uintptr_var;
     int main() {
-      int loaded_value = atomic_load(&value);
+      atomic_store_explicit(&int_var, 5, memory_order_relaxed);
+      atomic_store_explicit(&uintptr_var, 0, memory_order_relaxed);
+      int loaded_value = atomic_load_explicit(&int_var, memory_order_seq_cst);
       return 0;
     }
 
diff --git a/configure.ac b/configure.ac
index 8de53408eea74..20d8a5239f173 100644
--- a/configure.ac
+++ b/configure.ac
@@ -5414,9 +5414,12 @@ AC_LINK_IFELSE(
 [
   AC_LANG_SOURCE([[
     #include <stdatomic.h>
-    atomic_int value = ATOMIC_VAR_INIT(1);
+    atomic_int int_var;
+    atomic_uintptr_t uintptr_var;
     int main() {
-      int loaded_value = atomic_load(&value);
+      atomic_store_explicit(&int_var, 5, memory_order_relaxed);
+      atomic_store_explicit(&uintptr_var, 0, memory_order_relaxed);
+      int loaded_value = atomic_load_explicit(&int_var, memory_order_seq_cst);
       return 0;
     }
   ]])
@@ -5426,7 +5429,7 @@ AC_MSG_RESULT($have_stdatomic_h)
 
 if test "$have_stdatomic_h" = yes; then
     AC_DEFINE(HAVE_STD_ATOMIC, 1,
-              [Has stdatomic.h with atomic_int])
+              [Has stdatomic.h with atomic_int and atomic_uintptr_t])
 fi
 
 # Check for GCC >= 4.7 __atomic builtins
diff --git a/pyconfig.h.in b/pyconfig.h.in
index aca9751d4b756..9595d7537de47 100644
--- a/pyconfig.h.in
+++ b/pyconfig.h.in
@@ -1018,7 +1018,7 @@
 /* Define to 1 if you have the <stdlib.h> header file. */
 #undef HAVE_STDLIB_H
 
-/* Has stdatomic.h with atomic_int */
+/* Has stdatomic.h with atomic_int and atomic_uintptr_t */
 #undef HAVE_STD_ATOMIC
 
 /* Define to 1 if you have the `strdup' function. */


From webhook-mailer at python.org  Tue Oct 22 16:17:26 2019
From: webhook-mailer at python.org (Miss Skeleton (bot))
Date: Tue, 22 Oct 2019 20:17:26 -0000
Subject: [Python-checkins] bpo-37415: Fix stdatomic.h header check for ICC
 compiler (GH-16717)
Message-ID: <mailman.103.1571775447.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/b102e4f05278c1b06130885eba961bd0193733b4
commit: b102e4f05278c1b06130885eba961bd0193733b4
branch: 3.7
author: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-22T13:17:21-07:00
summary:

bpo-37415: Fix stdatomic.h header check for ICC compiler (GH-16717)


Fix stdatomic.h header check for ICC compiler: the ICC implementation
lacks atomic_uintptr_t type which is needed by Python.

Test:

* atomic_int and atomic_uintptr_t types
* atomic_load_explicit() and atomic_store_explicit()
* memory_order_relaxed and memory_order_seq_cst constants

But don't test ATOMIC_VAR_INIT(): it's not used in Python.
(cherry picked from commit 028f7349a0f6eaea0fec31becb587fcdf6e3cb28)

Co-authored-by: Victor Stinner <vstinner at python.org>

files:
A Misc/NEWS.d/next/Build/2019-10-11-15-32-58.bpo-37415.D9RXrq.rst
M configure
M configure.ac
M pyconfig.h.in

diff --git a/Misc/NEWS.d/next/Build/2019-10-11-15-32-58.bpo-37415.D9RXrq.rst b/Misc/NEWS.d/next/Build/2019-10-11-15-32-58.bpo-37415.D9RXrq.rst
new file mode 100644
index 0000000000000..98f4a3bf4f545
--- /dev/null
+++ b/Misc/NEWS.d/next/Build/2019-10-11-15-32-58.bpo-37415.D9RXrq.rst
@@ -0,0 +1,2 @@
+Fix stdatomic.h header check for ICC compiler: the ICC implementation lacks
+atomic_uintptr_t type which is needed by Python.
diff --git a/configure b/configure
index 0f466db4aa0a8..b769d59629637 100755
--- a/configure
+++ b/configure
@@ -16849,9 +16849,12 @@ cat confdefs.h - <<_ACEOF >conftest.$ac_ext
 
 
     #include <stdatomic.h>
-    atomic_int value = ATOMIC_VAR_INIT(1);
+    atomic_int int_var;
+    atomic_uintptr_t uintptr_var;
     int main() {
-      int loaded_value = atomic_load(&value);
+      atomic_store_explicit(&int_var, 5, memory_order_relaxed);
+      atomic_store_explicit(&uintptr_var, 0, memory_order_relaxed);
+      int loaded_value = atomic_load_explicit(&int_var, memory_order_seq_cst);
       return 0;
     }
 
diff --git a/configure.ac b/configure.ac
index 1ef0df70e686a..49acff31363b0 100644
--- a/configure.ac
+++ b/configure.ac
@@ -5492,9 +5492,12 @@ AC_LINK_IFELSE(
 [
   AC_LANG_SOURCE([[
     #include <stdatomic.h>
-    atomic_int value = ATOMIC_VAR_INIT(1);
+    atomic_int int_var;
+    atomic_uintptr_t uintptr_var;
     int main() {
-      int loaded_value = atomic_load(&value);
+      atomic_store_explicit(&int_var, 5, memory_order_relaxed);
+      atomic_store_explicit(&uintptr_var, 0, memory_order_relaxed);
+      int loaded_value = atomic_load_explicit(&int_var, memory_order_seq_cst);
       return 0;
     }
   ]])
@@ -5504,7 +5507,7 @@ AC_MSG_RESULT($have_stdatomic_h)
 
 if test "$have_stdatomic_h" = yes; then
     AC_DEFINE(HAVE_STD_ATOMIC, 1,
-              [Has stdatomic.h with atomic_int])
+              [Has stdatomic.h with atomic_int and atomic_uintptr_t])
 fi
 
 # Check for GCC >= 4.7 __atomic builtins
diff --git a/pyconfig.h.in b/pyconfig.h.in
index d36fc933f48cc..f4816fdcdfc4b 100644
--- a/pyconfig.h.in
+++ b/pyconfig.h.in
@@ -963,7 +963,7 @@
 /* Define to 1 if you have the <stdlib.h> header file. */
 #undef HAVE_STDLIB_H
 
-/* Has stdatomic.h with atomic_int */
+/* Has stdatomic.h with atomic_int and atomic_uintptr_t */
 #undef HAVE_STD_ATOMIC
 
 /* Define to 1 if you have the `strdup' function. */


From webhook-mailer at python.org  Tue Oct 22 16:19:36 2019
From: webhook-mailer at python.org (Miss Skeleton (bot))
Date: Tue, 22 Oct 2019 20:19:36 -0000
Subject: [Python-checkins] bpo-37415: Fix stdatomic.h header check for ICC
 compiler (GH-16717)
Message-ID: <mailman.104.1571775577.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/dbcea39ba713cc5b31fa1a243b16a0128c231c98
commit: dbcea39ba713cc5b31fa1a243b16a0128c231c98
branch: 3.8
author: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-22T13:19:32-07:00
summary:

bpo-37415: Fix stdatomic.h header check for ICC compiler (GH-16717)


Fix stdatomic.h header check for ICC compiler: the ICC implementation
lacks atomic_uintptr_t type which is needed by Python.

Test:

* atomic_int and atomic_uintptr_t types
* atomic_load_explicit() and atomic_store_explicit()
* memory_order_relaxed and memory_order_seq_cst constants

But don't test ATOMIC_VAR_INIT(): it's not used in Python.
(cherry picked from commit 028f7349a0f6eaea0fec31becb587fcdf6e3cb28)

Co-authored-by: Victor Stinner <vstinner at python.org>

files:
A Misc/NEWS.d/next/Build/2019-10-11-15-32-58.bpo-37415.D9RXrq.rst
M configure
M configure.ac
M pyconfig.h.in

diff --git a/Misc/NEWS.d/next/Build/2019-10-11-15-32-58.bpo-37415.D9RXrq.rst b/Misc/NEWS.d/next/Build/2019-10-11-15-32-58.bpo-37415.D9RXrq.rst
new file mode 100644
index 0000000000000..98f4a3bf4f545
--- /dev/null
+++ b/Misc/NEWS.d/next/Build/2019-10-11-15-32-58.bpo-37415.D9RXrq.rst
@@ -0,0 +1,2 @@
+Fix stdatomic.h header check for ICC compiler: the ICC implementation lacks
+atomic_uintptr_t type which is needed by Python.
diff --git a/configure b/configure
index 2a933cdbeb612..0914e24704383 100755
--- a/configure
+++ b/configure
@@ -782,7 +782,6 @@ infodir
 docdir
 oldincludedir
 includedir
-runstatedir
 localstatedir
 sharedstatedir
 sysconfdir
@@ -896,7 +895,6 @@ datadir='${datarootdir}'
 sysconfdir='${prefix}/etc'
 sharedstatedir='${prefix}/com'
 localstatedir='${prefix}/var'
-runstatedir='${localstatedir}/run'
 includedir='${prefix}/include'
 oldincludedir='/usr/include'
 docdir='${datarootdir}/doc/${PACKAGE_TARNAME}'
@@ -1149,15 +1147,6 @@ do
   | -silent | --silent | --silen | --sile | --sil)
     silent=yes ;;
 
-  -runstatedir | --runstatedir | --runstatedi | --runstated \
-  | --runstate | --runstat | --runsta | --runst | --runs \
-  | --run | --ru | --r)
-    ac_prev=runstatedir ;;
-  -runstatedir=* | --runstatedir=* | --runstatedi=* | --runstated=* \
-  | --runstate=* | --runstat=* | --runsta=* | --runst=* | --runs=* \
-  | --run=* | --ru=* | --r=*)
-    runstatedir=$ac_optarg ;;
-
   -sbindir | --sbindir | --sbindi | --sbind | --sbin | --sbi | --sb)
     ac_prev=sbindir ;;
   -sbindir=* | --sbindir=* | --sbindi=* | --sbind=* | --sbin=* \
@@ -1295,7 +1284,7 @@ fi
 for ac_var in	exec_prefix prefix bindir sbindir libexecdir datarootdir \
 		datadir sysconfdir sharedstatedir localstatedir includedir \
 		oldincludedir docdir infodir htmldir dvidir pdfdir psdir \
-		libdir localedir mandir runstatedir
+		libdir localedir mandir
 do
   eval ac_val=\$$ac_var
   # Remove trailing slashes.
@@ -1448,7 +1437,6 @@ Fine tuning of the installation directories:
   --sysconfdir=DIR        read-only single-machine data [PREFIX/etc]
   --sharedstatedir=DIR    modifiable architecture-independent data [PREFIX/com]
   --localstatedir=DIR     modifiable single-machine data [PREFIX/var]
-  --runstatedir=DIR       modifiable per-process data [LOCALSTATEDIR/run]
   --libdir=DIR            object code libraries [EPREFIX/lib]
   --includedir=DIR        C header files [PREFIX/include]
   --oldincludedir=DIR     C header files for non-gcc [/usr/include]
@@ -16765,9 +16753,12 @@ cat confdefs.h - <<_ACEOF >conftest.$ac_ext
 
 
     #include <stdatomic.h>
-    atomic_int value = ATOMIC_VAR_INIT(1);
+    atomic_int int_var;
+    atomic_uintptr_t uintptr_var;
     int main() {
-      int loaded_value = atomic_load(&value);
+      atomic_store_explicit(&int_var, 5, memory_order_relaxed);
+      atomic_store_explicit(&uintptr_var, 0, memory_order_relaxed);
+      int loaded_value = atomic_load_explicit(&int_var, memory_order_seq_cst);
       return 0;
     }
 
diff --git a/configure.ac b/configure.ac
index a189d42c2c367..7051dc109a725 100644
--- a/configure.ac
+++ b/configure.ac
@@ -5403,9 +5403,12 @@ AC_LINK_IFELSE(
 [
   AC_LANG_SOURCE([[
     #include <stdatomic.h>
-    atomic_int value = ATOMIC_VAR_INIT(1);
+    atomic_int int_var;
+    atomic_uintptr_t uintptr_var;
     int main() {
-      int loaded_value = atomic_load(&value);
+      atomic_store_explicit(&int_var, 5, memory_order_relaxed);
+      atomic_store_explicit(&uintptr_var, 0, memory_order_relaxed);
+      int loaded_value = atomic_load_explicit(&int_var, memory_order_seq_cst);
       return 0;
     }
   ]])
@@ -5415,7 +5418,7 @@ AC_MSG_RESULT($have_stdatomic_h)
 
 if test "$have_stdatomic_h" = yes; then
     AC_DEFINE(HAVE_STD_ATOMIC, 1,
-              [Has stdatomic.h with atomic_int])
+              [Has stdatomic.h with atomic_int and atomic_uintptr_t])
 fi
 
 # Check for GCC >= 4.7 __atomic builtins
diff --git a/pyconfig.h.in b/pyconfig.h.in
index 5f952d2cfc365..aaa7b72cac884 100644
--- a/pyconfig.h.in
+++ b/pyconfig.h.in
@@ -1025,7 +1025,7 @@
 /* Define to 1 if you have the <stdlib.h> header file. */
 #undef HAVE_STDLIB_H
 
-/* Has stdatomic.h with atomic_int */
+/* Has stdatomic.h with atomic_int and atomic_uintptr_t */
 #undef HAVE_STD_ATOMIC
 
 /* Define to 1 if you have the `strdup' function. */


From webhook-mailer at python.org  Tue Oct 22 16:37:53 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Tue, 22 Oct 2019 20:37:53 -0000
Subject: [Python-checkins] bpo-38466: Fix threading.excepthook doc (GH-16891)
Message-ID: <mailman.105.1571776674.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/91528f40c30717563a478920861c81d18da5bf63
commit: 91528f40c30717563a478920861c81d18da5bf63
branch: master
author: Daniel Baskal <MagnificentUrukHai at users.noreply.github.com>
committer: Victor Stinner <vstinner at python.org>
date: 2019-10-22T22:37:47+02:00
summary:

bpo-38466: Fix threading.excepthook doc (GH-16891)

files:
M Doc/library/threading.rst

diff --git a/Doc/library/threading.rst b/Doc/library/threading.rst
index 0489421cdf60e..905862e7747fb 100644
--- a/Doc/library/threading.rst
+++ b/Doc/library/threading.rst
@@ -62,8 +62,8 @@ This module defines the following functions:
    should be cleared explicitly to break the reference cycle when the
    exception is no longer needed.
 
-   Storing *object* using a custom hook can resurrect it if it is set to an
-   object which is being finalized. Avoid storing *object* after the custom
+   Storing *thread* using a custom hook can resurrect it if it is set to an
+   object which is being finalized. Avoid storing *thread* after the custom
    hook completes to avoid resurrecting objects.
 
    .. seealso::


From webhook-mailer at python.org  Tue Oct 22 16:45:07 2019
From: webhook-mailer at python.org (Miss Skeleton (bot))
Date: Tue, 22 Oct 2019 20:45:07 -0000
Subject: [Python-checkins] bpo-38466: Fix threading.excepthook doc (GH-16891)
Message-ID: <mailman.106.1571777109.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/6329a56ca7f99075caa95a075024736a90ad2857
commit: 6329a56ca7f99075caa95a075024736a90ad2857
branch: 3.8
author: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-22T13:44:45-07:00
summary:

bpo-38466: Fix threading.excepthook doc (GH-16891)

(cherry picked from commit 91528f40c30717563a478920861c81d18da5bf63)

Co-authored-by: Daniel Baskal <MagnificentUrukHai at users.noreply.github.com>

files:
M Doc/library/threading.rst

diff --git a/Doc/library/threading.rst b/Doc/library/threading.rst
index eb9da2b911974..f7c82729e51d7 100644
--- a/Doc/library/threading.rst
+++ b/Doc/library/threading.rst
@@ -62,8 +62,8 @@ This module defines the following functions:
    should be cleared explicitly to break the reference cycle when the
    exception is no longer needed.
 
-   Storing *object* using a custom hook can resurrect it if it is set to an
-   object which is being finalized. Avoid storing *object* after the custom
+   Storing *thread* using a custom hook can resurrect it if it is set to an
+   object which is being finalized. Avoid storing *thread* after the custom
    hook completes to avoid resurrecting objects.
 
    .. seealso::


From webhook-mailer at python.org  Tue Oct 22 18:55:18 2019
From: webhook-mailer at python.org (Miss Skeleton (bot))
Date: Tue, 22 Oct 2019 22:55:18 -0000
Subject: [Python-checkins] Fix Zope URL (GH-16880)
Message-ID: <mailman.107.1571784919.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/e113b5c3e7c837bd7113c21d00fe242f8ed581da
commit: e113b5c3e7c837bd7113c21d00fe242f8ed581da
branch: 3.8
author: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-22T15:55:12-07:00
summary:

Fix Zope URL (GH-16880)

(cherry picked from commit dfe726b1ace03f206f45253b93ed7610473ae20f)

Co-authored-by: Kyle Stanley <aeros167 at gmail.com>

files:
M Doc/license.rst

diff --git a/Doc/license.rst b/Doc/license.rst
index d877f45677549..810d2e63fd48c 100644
--- a/Doc/license.rst
+++ b/Doc/license.rst
@@ -22,7 +22,7 @@ Virginia where he released several versions of the software.
 In May 2000, Guido and the Python core development team moved to BeOpen.com to
 form the BeOpen PythonLabs team.  In October of the same year, the PythonLabs
 team moved to Digital Creations (now Zope Corporation; see
-http://www.zope.com/).  In 2001, the Python Software Foundation (PSF, see
+https://www.zope.org/).  In 2001, the Python Software Foundation (PSF, see
 https://www.python.org/psf/) was formed, a non-profit organization created
 specifically to own Python-related Intellectual Property.  Zope Corporation is a
 sponsoring member of the PSF.


From webhook-mailer at python.org  Tue Oct 22 20:06:04 2019
From: webhook-mailer at python.org (Carol Willing)
Date: Wed, 23 Oct 2019 00:06:04 -0000
Subject: [Python-checkins] bpo-38419: fix "check-c-globals" path (GH-16680)
Message-ID: <mailman.108.1571789166.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/20bf8e08a18c0f1eab49c54f3bd56f8364a2f5cc
commit: 20bf8e08a18c0f1eab49c54f3bd56f8364a2f5cc
branch: master
author: AMIR <31338382+amiremohamadi at users.noreply.github.com>
committer: Carol Willing <carolcode at willingconsulting.com>
date: 2019-10-22T21:05:54-03:00
summary:

 bpo-38419: fix "check-c-globals" path (GH-16680)

files:
M Tools/c-analyzer/README

diff --git a/Tools/c-analyzer/README b/Tools/c-analyzer/README
index 0ee8ac3800bc6..8cf20e276d927 100644
--- a/Tools/c-analyzer/README
+++ b/Tools/c-analyzer/README
@@ -34,7 +34,7 @@ should be added for runtime state.  Instead, they should be added to
 _PyRuntimeState or one of its sub-structs.  The check-c-globals script
 should be run to ensure that no new globals have been added:
 
-  python3 Tools/c-globals/check-c-globals.py
+  python3 Tools/c-analyzer/check-c-globals.py
 
 If it reports any globals then they should be resolved.  If the globals
 are runtime state then they should be folded into _PyRuntimeState.


From webhook-mailer at python.org  Tue Oct 22 23:23:53 2019
From: webhook-mailer at python.org (Raymond Hettinger)
Date: Wed, 23 Oct 2019 03:23:53 -0000
Subject: [Python-checkins] bpo-38539: Update demo files (GH-16890)
Message-ID: <mailman.109.1571801034.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/a4aeb336697c85996d781271cc6b42bc6d4c2908
commit: a4aeb336697c85996d781271cc6b42bc6d4c2908
branch: master
author: Julin S <48789920+ju-sh at users.noreply.github.com>
committer: Raymond Hettinger <rhettinger at users.noreply.github.com>
date: 2019-10-22T20:23:48-07:00
summary:

bpo-38539: Update demo files (GH-16890)

files:
A Tools/demo/spreadsheet.py
M Tools/demo/README
M Tools/demo/hanoi.py
M Tools/demo/rpythond.py
M Tools/demo/sortvisu.py

diff --git a/Tools/demo/README b/Tools/demo/README
index e914358a26df4..9fccb97d956cb 100644
--- a/Tools/demo/README
+++ b/Tools/demo/README
@@ -1,16 +1,16 @@
 This directory contains a collection of demonstration scripts for
 various aspects of Python programming.
 
-beer.py        Well-known programming example: Bottles of beer.
-eiffel.py      Python advanced magic: A metaclass for Eiffel post/preconditions.
-hanoi.py       Well-known programming example: Towers of Hanoi.
-life.py        Curses programming: Simple game-of-life.
-markov.py      Algorithms: Markov chain simulation.
-mcast.py       Network programming: Send and receive UDP multicast packets.
-queens.py      Well-known programming example: N-Queens problem.
-redemo.py      Regular Expressions: GUI script to test regexes.
-rpython.py     Network programming: Small client for remote code execution.
-rpythond.py    Network programming: Small server for remote code execution.
-sortvisu.py    GUI programming: Visualization of different sort algorithms.
-ss1.py         GUI/Application programming: A simple spreadsheet application.
-vector.py      Python basics: A vector class with demonstrating special methods.
\ No newline at end of file
+beer.py           Well-known programming example: Bottles of beer.
+eiffel.py         Python advanced magic: A metaclass for Eiffel post/preconditions.
+hanoi.py          Well-known programming example: Towers of Hanoi.
+life.py           Curses programming: Simple game-of-life.
+markov.py         Algorithms: Markov chain simulation.
+mcast.py          Network programming: Send and receive UDP multicast packets.
+queens.py         Well-known programming example: N-Queens problem.
+redemo.py         Regular Expressions: GUI script to test regexes.
+rpython.py        Network programming: Small client for remote code execution.
+rpythond.py       Network programming: Small server for remote code execution.
+sortvisu.py       GUI programming: Visualization of different sort algorithms.
+spreadsheet.py    GUI/Application programming: A simple spreadsheet application.
+vector.py         Python basics: A vector class demonstrating special methods.
diff --git a/Tools/demo/hanoi.py b/Tools/demo/hanoi.py
index dad02342ae337..8db895c244581 100755
--- a/Tools/demo/hanoi.py
+++ b/Tools/demo/hanoi.py
@@ -27,7 +27,7 @@ def hanoi(n, a, b, c, report):
 class Tkhanoi:
 
     # Create our objects
-    def __init__(self, n, bitmap = None):
+    def __init__(self, n, bitmap=None):
         self.n = n
         self.tk = tk = Tk()
         self.canvas = c = Canvas(tk)
@@ -77,7 +77,7 @@ def __init__(self, n, bitmap = None):
 
     # Run -- never returns
     def run(self):
-        while 1:
+        while True:
             hanoi(self.n, 0, 1, 2, self.report)
             hanoi(self.n, 1, 2, 0, self.report)
             hanoi(self.n, 2, 0, 1, self.report)
@@ -94,7 +94,7 @@ def report(self, i, a, b):
 
         # Lift the piece above peg a
         ax1, ay1, ax2, ay2 = c.bbox(self.pegs[a])
-        while 1:
+        while True:
             x1, y1, x2, y2 = c.bbox(p)
             if y2 < ay1: break
             c.move(p, 0, -1)
@@ -103,7 +103,7 @@ def report(self, i, a, b):
         # Move it towards peg b
         bx1, by1, bx2, by2 = c.bbox(self.pegs[b])
         newcenter = (bx1+bx2)//2
-        while 1:
+        while True:
             x1, y1, x2, y2 = c.bbox(p)
             center = (x1+x2)//2
             if center == newcenter: break
@@ -114,7 +114,7 @@ def report(self, i, a, b):
         # Move it down on top of the previous piece
         pieceheight = y2-y1
         newbottom = by2 - pieceheight*len(self.pegstate[b]) - 2
-        while 1:
+        while True:
             x1, y1, x2, y2 = c.bbox(p)
             if y2 >= newbottom: break
             c.move(p, 0, 1)
diff --git a/Tools/demo/rpythond.py b/Tools/demo/rpythond.py
index a885b3e946a1f..a18de137fa1cf 100755
--- a/Tools/demo/rpythond.py
+++ b/Tools/demo/rpythond.py
@@ -29,7 +29,7 @@ def main():
         with conn:
             print('connection from', remotehost, remoteport)
             request = b''
-            while 1:
+            while True:
                 data = conn.recv(BUFSIZE)
                 if not data:
                     break
diff --git a/Tools/demo/sortvisu.py b/Tools/demo/sortvisu.py
index 8447bc757e630..056a0e05fb1ed 100755
--- a/Tools/demo/sortvisu.py
+++ b/Tools/demo/sortvisu.py
@@ -444,7 +444,7 @@ def quicksort(array):
             array.wait(1000)
             left = first
             right = last
-            while 1:
+            while True:
                 array.message("Sweep right pointer")
                 right = right-1
                 array.show_right(right)
@@ -473,7 +473,7 @@ def quicksort(array):
         array.hide_partition()
 
 def demosort(array):
-    while 1:
+    while True:
         for alg in [quicksort, insertionsort, selectionsort, bubblesort]:
             randomize(array)
             alg(array)
diff --git a/Tools/demo/spreadsheet.py b/Tools/demo/spreadsheet.py
new file mode 100755
index 0000000000000..bf88820dca347
--- /dev/null
+++ b/Tools/demo/spreadsheet.py
@@ -0,0 +1,829 @@
+#!/usr/bin/env python3
+
+"""
+SS1 -- a spreadsheet-like application.
+"""
+
+import os
+import re
+import sys
+from xml.parsers import expat
+from xml.sax.saxutils import escape
+
+LEFT, CENTER, RIGHT = "LEFT", "CENTER", "RIGHT"
+
+def ljust(x, n):
+    return x.ljust(n)
+def center(x, n):
+    return x.center(n)
+def rjust(x, n):
+    return x.rjust(n)
+align2action = {LEFT: ljust, CENTER: center, RIGHT: rjust}
+
+align2xml = {LEFT: "left", CENTER: "center", RIGHT: "right"}
+xml2align = {"left": LEFT, "center": CENTER, "right": RIGHT}
+
+align2anchor = {LEFT: "w", CENTER: "center", RIGHT: "e"}
+
+def sum(seq):
+    total = 0
+    for x in seq:
+        if x is not None:
+            total += x
+    return total
+
+class Sheet:
+
+    def __init__(self):
+        self.cells = {} # {(x, y): cell, ...}
+        self.ns = dict(
+            cell = self.cellvalue,
+            cells = self.multicellvalue,
+            sum = sum,
+        )
+
+    def cellvalue(self, x, y):
+        cell = self.getcell(x, y)
+        if hasattr(cell, 'recalc'):
+            return cell.recalc(self.ns)
+        else:
+            return cell
+
+    def multicellvalue(self, x1, y1, x2, y2):
+        if x1 > x2:
+            x1, x2 = x2, x1
+        if y1 > y2:
+            y1, y2 = y2, y1
+        seq = []
+        for y in range(y1, y2+1):
+            for x in range(x1, x2+1):
+                seq.append(self.cellvalue(x, y))
+        return seq
+
+    def getcell(self, x, y):
+        return self.cells.get((x, y))
+
+    def setcell(self, x, y, cell):
+        assert x > 0 and y > 0
+        assert isinstance(cell, BaseCell)
+        self.cells[x, y] = cell
+
+    def clearcell(self, x, y):
+        try:
+            del self.cells[x, y]
+        except KeyError:
+            pass
+
+    def clearcells(self, x1, y1, x2, y2):
+        for xy in self.selectcells(x1, y1, x2, y2):
+            del self.cells[xy]
+
+    def clearrows(self, y1, y2):
+        self.clearcells(0, y1, sys.maxsize, y2)
+
+    def clearcolumns(self, x1, x2):
+        self.clearcells(x1, 0, x2, sys.maxsize)
+
+    def selectcells(self, x1, y1, x2, y2):
+        if x1 > x2:
+            x1, x2 = x2, x1
+        if y1 > y2:
+            y1, y2 = y2, y1
+        return [(x, y) for x, y in self.cells
+                if x1 <= x <= x2 and y1 <= y <= y2]
+
+    def movecells(self, x1, y1, x2, y2, dx, dy):
+        if dx == 0 and dy == 0:
+            return
+        if x1 > x2:
+            x1, x2 = x2, x1
+        if y1 > y2:
+            y1, y2 = y2, y1
+        assert x1+dx > 0 and y1+dy > 0
+        new = {}
+        for x, y in self.cells:
+            cell = self.cells[x, y]
+            if hasattr(cell, 'renumber'):
+                cell = cell.renumber(x1, y1, x2, y2, dx, dy)
+            if x1 <= x <= x2 and y1 <= y <= y2:
+                x += dx
+                y += dy
+            new[x, y] = cell
+        self.cells = new
+
+    def insertrows(self, y, n):
+        assert n > 0
+        self.movecells(0, y, sys.maxsize, sys.maxsize, 0, n)
+
+    def deleterows(self, y1, y2):
+        if y1 > y2:
+            y1, y2 = y2, y1
+        self.clearrows(y1, y2)
+        self.movecells(0, y2+1, sys.maxsize, sys.maxsize, 0, y1-y2-1)
+
+    def insertcolumns(self, x, n):
+        assert n > 0
+        self.movecells(x, 0, sys.maxsize, sys.maxsize, n, 0)
+
+    def deletecolumns(self, x1, x2):
+        if x1 > x2:
+            x1, x2 = x2, x1
+        self.clearcells(x1, x2)
+        self.movecells(x2+1, 0, sys.maxsize, sys.maxsize, x1-x2-1, 0)
+
+    def getsize(self):
+        maxx = maxy = 0
+        for x, y in self.cells:
+            maxx = max(maxx, x)
+            maxy = max(maxy, y)
+        return maxx, maxy
+
+    def reset(self):
+        for cell in self.cells.values():
+            if hasattr(cell, 'reset'):
+                cell.reset()
+
+    def recalc(self):
+        self.reset()
+        for cell in self.cells.values():
+            if hasattr(cell, 'recalc'):
+                cell.recalc(self.ns)
+
+    def display(self):
+        maxx, maxy = self.getsize()
+        width, height = maxx+1, maxy+1
+        colwidth = [1] * width
+        full = {}
+        # Add column heading labels in row 0
+        for x in range(1, width):
+            full[x, 0] = text, alignment = colnum2name(x), RIGHT
+            colwidth[x] = max(colwidth[x], len(text))
+        # Add row labels in column 0
+        for y in range(1, height):
+            full[0, y] = text, alignment = str(y), RIGHT
+            colwidth[0] = max(colwidth[0], len(text))
+        # Add sheet cells in columns with x>0 and y>0
+        for (x, y), cell in self.cells.items():
+            if x <= 0 or y <= 0:
+                continue
+            if hasattr(cell, 'recalc'):
+                cell.recalc(self.ns)
+            if hasattr(cell, 'format'):
+                text, alignment = cell.format()
+                assert isinstance(text, str)
+                assert alignment in (LEFT, CENTER, RIGHT)
+            else:
+                text = str(cell)
+                if isinstance(cell, str):
+                    alignment = LEFT
+                else:
+                    alignment = RIGHT
+            full[x, y] = (text, alignment)
+            colwidth[x] = max(colwidth[x], len(text))
+        # Calculate the horizontal separator line (dashes and dots)
+        sep = ""
+        for x in range(width):
+            if sep:
+                sep += "+"
+            sep += "-"*colwidth[x]
+        # Now print The full grid
+        for y in range(height):
+            line = ""
+            for x in range(width):
+                text, alignment = full.get((x, y)) or ("", LEFT)
+                text = align2action[alignment](text, colwidth[x])
+                if line:
+                    line += '|'
+                line += text
+            print(line)
+            if y == 0:
+                print(sep)
+
+    def xml(self):
+        out = ['<spreadsheet>']
+        for (x, y), cell in self.cells.items():
+            if hasattr(cell, 'xml'):
+                cellxml = cell.xml()
+            else:
+                cellxml = '<value>%s</value>' % escape(cell)
+            out.append('<cell row="%s" col="%s">\n  %s\n</cell>' %
+                       (y, x, cellxml))
+        out.append('</spreadsheet>')
+        return '\n'.join(out)
+
+    def save(self, filename):
+        text = self.xml()
+        with open(filename, "w", encoding='utf-8') as f:
+            f.write(text)
+            if text and not text.endswith('\n'):
+                f.write('\n')
+
+    def load(self, filename):
+        with open(filename, 'rb') as f:
+            SheetParser(self).parsefile(f)
+
+class SheetParser:
+
+    def __init__(self, sheet):
+        self.sheet = sheet
+
+    def parsefile(self, f):
+        parser = expat.ParserCreate()
+        parser.StartElementHandler = self.startelement
+        parser.EndElementHandler = self.endelement
+        parser.CharacterDataHandler = self.data
+        parser.ParseFile(f)
+
+    def startelement(self, tag, attrs):
+        method = getattr(self, 'start_'+tag, None)
+        if method:
+            method(attrs)
+        self.texts = []
+
+    def data(self, text):
+        self.texts.append(text)
+
+    def endelement(self, tag):
+        method = getattr(self, 'end_'+tag, None)
+        if method:
+            method("".join(self.texts))
+
+    def start_cell(self, attrs):
+        self.y = int(attrs.get("row"))
+        self.x = int(attrs.get("col"))
+
+    def start_value(self, attrs):
+        self.fmt = attrs.get('format')
+        self.alignment = xml2align.get(attrs.get('align'))
+
+    start_formula = start_value
+
+    def end_int(self, text):
+        try:
+            self.value = int(text)
+        except (TypeError, ValueError):
+            self.value = None
+
+    end_long = end_int
+
+    def end_double(self, text):
+        try:
+            self.value = float(text)
+        except (TypeError, ValueError):
+            self.value = None
+
+    def end_complex(self, text):
+        try:
+            self.value = complex(text)
+        except (TypeError, ValueError):
+            self.value = None
+
+    def end_string(self, text):
+        self.value = text
+
+    def end_value(self, text):
+        if isinstance(self.value, BaseCell):
+            self.cell = self.value
+        elif isinstance(self.value, str):
+            self.cell = StringCell(self.value,
+                                   self.fmt or "%s",
+                                   self.alignment or LEFT)
+        else:
+            self.cell = NumericCell(self.value,
+                                    self.fmt or "%s",
+                                    self.alignment or RIGHT)
+
+    def end_formula(self, text):
+        self.cell = FormulaCell(text,
+                                self.fmt or "%s",
+                                self.alignment or RIGHT)
+
+    def end_cell(self, text):
+        self.sheet.setcell(self.x, self.y, self.cell)
+
+class BaseCell:
+    __init__ = None # Must provide
+    """Abstract base class for sheet cells.
+
+    Subclasses may but needn't provide the following APIs:
+
+    cell.reset() -- prepare for recalculation
+    cell.recalc(ns) -> value -- recalculate formula
+    cell.format() -> (value, alignment) -- return formatted value
+    cell.xml() -> string -- return XML
+    """
+
+class NumericCell(BaseCell):
+
+    def __init__(self, value, fmt="%s", alignment=RIGHT):
+        assert isinstance(value, (int, float, complex))
+        assert alignment in (LEFT, CENTER, RIGHT)
+        self.value = value
+        self.fmt = fmt
+        self.alignment = alignment
+
+    def recalc(self, ns):
+        return self.value
+
+    def format(self):
+        try:
+            text = self.fmt % self.value
+        except:
+            text = str(self.value)
+        return text, self.alignment
+
+    def xml(self):
+        method = getattr(self, '_xml_' + type(self.value).__name__)
+        return '<value align="%s" format="%s">%s</value>' % (
+                align2xml[self.alignment],
+                self.fmt,
+                method())
+
+    def _xml_int(self):
+        if -2**31 <= self.value < 2**31:
+            return '<int>%s</int>' % self.value
+        else:
+            return '<long>%s</long>' % self.value
+
+    def _xml_float(self):
+        return '<double>%r</double>' % self.value
+
+    def _xml_complex(self):
+        return '<complex>%r</complex>' % self.value
+
+class StringCell(BaseCell):
+
+    def __init__(self, text, fmt="%s", alignment=LEFT):
+        assert isinstance(text, str)
+        assert alignment in (LEFT, CENTER, RIGHT)
+        self.text = text
+        self.fmt = fmt
+        self.alignment = alignment
+
+    def recalc(self, ns):
+        return self.text
+
+    def format(self):
+        return self.text, self.alignment
+
+    def xml(self):
+        s = '<value align="%s" format="%s"><string>%s</string></value>'
+        return s % (
+            align2xml[self.alignment],
+            self.fmt,
+            escape(self.text))
+
+class FormulaCell(BaseCell):
+
+    def __init__(self, formula, fmt="%s", alignment=RIGHT):
+        assert alignment in (LEFT, CENTER, RIGHT)
+        self.formula = formula
+        self.translated = translate(self.formula)
+        self.fmt = fmt
+        self.alignment = alignment
+        self.reset()
+
+    def reset(self):
+        self.value = None
+
+    def recalc(self, ns):
+        if self.value is None:
+            try:
+                self.value = eval(self.translated, ns)
+            except:
+                exc = sys.exc_info()[0]
+                if hasattr(exc, "__name__"):
+                    self.value = exc.__name__
+                else:
+                    self.value = str(exc)
+        return self.value
+
+    def format(self):
+        try:
+            text = self.fmt % self.value
+        except:
+            text = str(self.value)
+        return text, self.alignment
+
+    def xml(self):
+        return '<formula align="%s" format="%s">%s</formula>' % (
+            align2xml[self.alignment],
+            self.fmt,
+            escape(self.formula))
+
+    def renumber(self, x1, y1, x2, y2, dx, dy):
+        out = []
+        for part in re.split(r'(\w+)', self.formula):
+            m = re.match('^([A-Z]+)([1-9][0-9]*)$', part)
+            if m is not None:
+                sx, sy = m.groups()
+                x = colname2num(sx)
+                y = int(sy)
+                if x1 <= x <= x2 and y1 <= y <= y2:
+                    part = cellname(x+dx, y+dy)
+            out.append(part)
+        return FormulaCell("".join(out), self.fmt, self.alignment)
+
+def translate(formula):
+    """Translate a formula containing fancy cell names to valid Python code.
+
+    Examples:
+        B4 -> cell(2, 4)
+        B4:Z100 -> cells(2, 4, 26, 100)
+    """
+    out = []
+    for part in re.split(r"(\w+(?::\w+)?)", formula):
+        m = re.match(r"^([A-Z]+)([1-9][0-9]*)(?::([A-Z]+)([1-9][0-9]*))?$", part)
+        if m is None:
+            out.append(part)
+        else:
+            x1, y1, x2, y2 = m.groups()
+            x1 = colname2num(x1)
+            if x2 is None:
+                s = "cell(%s, %s)" % (x1, y1)
+            else:
+                x2 = colname2num(x2)
+                s = "cells(%s, %s, %s, %s)" % (x1, y1, x2, y2)
+            out.append(s)
+    return "".join(out)
+
+def cellname(x, y):
+    "Translate a cell coordinate to a fancy cell name (e.g. (1, 1)->'A1')."
+    assert x > 0 # Column 0 has an empty name, so can't use that
+    return colnum2name(x) + str(y)
+
+def colname2num(s):
+    "Translate a column name to number (e.g. 'A'->1, 'Z'->26, 'AA'->27)."
+    s = s.upper()
+    n = 0
+    for c in s:
+        assert 'A' <= c <= 'Z'
+        n = n*26 + ord(c) - ord('A') + 1
+    return n
+
+def colnum2name(n):
+    "Translate a column number to name (e.g. 1->'A', etc.)."
+    assert n > 0
+    s = ""
+    while n:
+        n, m = divmod(n-1, 26)
+        s = chr(m+ord('A')) + s
+    return s
+
+import tkinter as Tk
+
+class SheetGUI:
+
+    """Beginnings of a GUI for a spreadsheet.
+
+    TO DO:
+    - clear multiple cells
+    - Insert, clear, remove rows or columns
+    - Show new contents while typing
+    - Scroll bars
+    - Grow grid when window is grown
+    - Proper menus
+    - Undo, redo
+    - Cut, copy and paste
+    - Formatting and alignment
+    """
+
+    def __init__(self, filename="sheet1.xml", rows=10, columns=5):
+        """Constructor.
+
+        Load the sheet from the filename argument.
+        Set up the Tk widget tree.
+        """
+        # Create and load the sheet
+        self.filename = filename
+        self.sheet = Sheet()
+        if os.path.isfile(filename):
+            self.sheet.load(filename)
+        # Calculate the needed grid size
+        maxx, maxy = self.sheet.getsize()
+        rows = max(rows, maxy)
+        columns = max(columns, maxx)
+        # Create the widgets
+        self.root = Tk.Tk()
+        self.root.wm_title("Spreadsheet: %s" % self.filename)
+        self.beacon = Tk.Label(self.root, text="A1",
+                               font=('helvetica', 16, 'bold'))
+        self.entry = Tk.Entry(self.root)
+        self.savebutton = Tk.Button(self.root, text="Save",
+                                    command=self.save)
+        self.cellgrid = Tk.Frame(self.root)
+        # Configure the widget lay-out
+        self.cellgrid.pack(side="bottom", expand=1, fill="both")
+        self.beacon.pack(side="left")
+        self.savebutton.pack(side="right")
+        self.entry.pack(side="left", expand=1, fill="x")
+        # Bind some events
+        self.entry.bind("<Return>", self.return_event)
+        self.entry.bind("<Shift-Return>", self.shift_return_event)
+        self.entry.bind("<Tab>", self.tab_event)
+        self.entry.bind("<Shift-Tab>", self.shift_tab_event)
+        self.entry.bind("<Delete>", self.delete_event)
+        self.entry.bind("<Escape>", self.escape_event)
+        # Now create the cell grid
+        self.makegrid(rows, columns)
+        # Select the top-left cell
+        self.currentxy = None
+        self.cornerxy = None
+        self.setcurrent(1, 1)
+        # Copy the sheet cells to the GUI cells
+        self.sync()
+
+    def delete_event(self, event):
+        if self.cornerxy != self.currentxy and self.cornerxy is not None:
+            self.sheet.clearcells(*(self.currentxy + self.cornerxy))
+        else:
+            self.sheet.clearcell(*self.currentxy)
+        self.sync()
+        self.entry.delete(0, 'end')
+        return "break"
+
+    def escape_event(self, event):
+        x, y = self.currentxy
+        self.load_entry(x, y)
+
+    def load_entry(self, x, y):
+        cell = self.sheet.getcell(x, y)
+        if cell is None:
+            text = ""
+        elif isinstance(cell, FormulaCell):
+            text = '=' + cell.formula
+        else:
+            text, alignment = cell.format()
+        self.entry.delete(0, 'end')
+        self.entry.insert(0, text)
+        self.entry.selection_range(0, 'end')
+
+    def makegrid(self, rows, columns):
+        """Helper to create the grid of GUI cells.
+
+        The edge (x==0 or y==0) is filled with labels; the rest is real cells.
+        """
+        self.rows = rows
+        self.columns = columns
+        self.gridcells = {}
+        # Create the top left corner cell (which selects all)
+        cell = Tk.Label(self.cellgrid, relief='raised')
+        cell.grid_configure(column=0, row=0, sticky='NSWE')
+        cell.bind("<ButtonPress-1>", self.selectall)
+        # Create the top row of labels, and configure the grid columns
+        for x in range(1, columns+1):
+            self.cellgrid.grid_columnconfigure(x, minsize=64)
+            cell = Tk.Label(self.cellgrid, text=colnum2name(x), relief='raised')
+            cell.grid_configure(column=x, row=0, sticky='WE')
+            self.gridcells[x, 0] = cell
+            cell.__x = x
+            cell.__y = 0
+            cell.bind("<ButtonPress-1>", self.selectcolumn)
+            cell.bind("<B1-Motion>", self.extendcolumn)
+            cell.bind("<ButtonRelease-1>", self.extendcolumn)
+            cell.bind("<Shift-Button-1>", self.extendcolumn)
+        # Create the leftmost column of labels
+        for y in range(1, rows+1):
+            cell = Tk.Label(self.cellgrid, text=str(y), relief='raised')
+            cell.grid_configure(column=0, row=y, sticky='WE')
+            self.gridcells[0, y] = cell
+            cell.__x = 0
+            cell.__y = y
+            cell.bind("<ButtonPress-1>", self.selectrow)
+            cell.bind("<B1-Motion>", self.extendrow)
+            cell.bind("<ButtonRelease-1>", self.extendrow)
+            cell.bind("<Shift-Button-1>", self.extendrow)
+        # Create the real cells
+        for x in range(1, columns+1):
+            for y in range(1, rows+1):
+                cell = Tk.Label(self.cellgrid, relief='sunken',
+                                bg='white', fg='black')
+                cell.grid_configure(column=x, row=y, sticky='NSWE')
+                self.gridcells[x, y] = cell
+                cell.__x = x
+                cell.__y = y
+                # Bind mouse events
+                cell.bind("<ButtonPress-1>", self.press)
+                cell.bind("<B1-Motion>", self.motion)
+                cell.bind("<ButtonRelease-1>", self.release)
+                cell.bind("<Shift-Button-1>", self.release)
+
+    def selectall(self, event):
+        self.setcurrent(1, 1)
+        self.setcorner(sys.maxsize, sys.maxsize)
+
+    def selectcolumn(self, event):
+        x, y = self.whichxy(event)
+        self.setcurrent(x, 1)
+        self.setcorner(x, sys.maxsize)
+
+    def extendcolumn(self, event):
+        x, y = self.whichxy(event)
+        if x > 0:
+            self.setcurrent(self.currentxy[0], 1)
+            self.setcorner(x, sys.maxsize)
+
+    def selectrow(self, event):
+        x, y = self.whichxy(event)
+        self.setcurrent(1, y)
+        self.setcorner(sys.maxsize, y)
+
+    def extendrow(self, event):
+        x, y = self.whichxy(event)
+        if y > 0:
+            self.setcurrent(1, self.currentxy[1])
+            self.setcorner(sys.maxsize, y)
+
+    def press(self, event):
+        x, y = self.whichxy(event)
+        if x > 0 and y > 0:
+            self.setcurrent(x, y)
+
+    def motion(self, event):
+        x, y = self.whichxy(event)
+        if x > 0 and y > 0:
+            self.setcorner(x, y)
+
+    release = motion
+
+    def whichxy(self, event):
+        w = self.cellgrid.winfo_containing(event.x_root, event.y_root)
+        if w is not None and isinstance(w, Tk.Label):
+            try:
+                return w.__x, w.__y
+            except AttributeError:
+                pass
+        return 0, 0
+
+    def save(self):
+        self.sheet.save(self.filename)
+
+    def setcurrent(self, x, y):
+        "Make (x, y) the current cell."
+        if self.currentxy is not None:
+            self.change_cell()
+        self.clearfocus()
+        self.beacon['text'] = cellname(x, y)
+        self.load_entry(x, y)
+        self.entry.focus_set()
+        self.currentxy = x, y
+        self.cornerxy = None
+        gridcell = self.gridcells.get(self.currentxy)
+        if gridcell is not None:
+            gridcell['bg'] = 'yellow'
+
+    def setcorner(self, x, y):
+        if self.currentxy is None or self.currentxy == (x, y):
+            self.setcurrent(x, y)
+            return
+        self.clearfocus()
+        self.cornerxy = x, y
+        x1, y1 = self.currentxy
+        x2, y2 = self.cornerxy or self.currentxy
+        if x1 > x2:
+            x1, x2 = x2, x1
+        if y1 > y2:
+            y1, y2 = y2, y1
+        for (x, y), cell in self.gridcells.items():
+            if x1 <= x <= x2 and y1 <= y <= y2:
+                cell['bg'] = 'lightBlue'
+        gridcell = self.gridcells.get(self.currentxy)
+        if gridcell is not None:
+            gridcell['bg'] = 'yellow'
+        self.setbeacon(x1, y1, x2, y2)
+
+    def setbeacon(self, x1, y1, x2, y2):
+        if x1 == y1 == 1 and x2 == y2 == sys.maxsize:
+            name = ":"
+        elif (x1, x2) == (1, sys.maxsize):
+            if y1 == y2:
+                name = "%d" % y1
+            else:
+                name = "%d:%d" % (y1, y2)
+        elif (y1, y2) == (1, sys.maxsize):
+            if x1 == x2:
+                name = "%s" % colnum2name(x1)
+            else:
+                name = "%s:%s" % (colnum2name(x1), colnum2name(x2))
+        else:
+            name1 = cellname(*self.currentxy)
+            name2 = cellname(*self.cornerxy)
+            name = "%s:%s" % (name1, name2)
+        self.beacon['text'] = name
+
+
+    def clearfocus(self):
+        if self.currentxy is not None:
+            x1, y1 = self.currentxy
+            x2, y2 = self.cornerxy or self.currentxy
+            if x1 > x2:
+                x1, x2 = x2, x1
+            if y1 > y2:
+                y1, y2 = y2, y1
+            for (x, y), cell in self.gridcells.items():
+                if x1 <= x <= x2 and y1 <= y <= y2:
+                    cell['bg'] = 'white'
+
+    def return_event(self, event):
+        "Callback for the Return key."
+        self.change_cell()
+        x, y = self.currentxy
+        self.setcurrent(x, y+1)
+        return "break"
+
+    def shift_return_event(self, event):
+        "Callback for the Return key with Shift modifier."
+        self.change_cell()
+        x, y = self.currentxy
+        self.setcurrent(x, max(1, y-1))
+        return "break"
+
+    def tab_event(self, event):
+        "Callback for the Tab key."
+        self.change_cell()
+        x, y = self.currentxy
+        self.setcurrent(x+1, y)
+        return "break"
+
+    def shift_tab_event(self, event):
+        "Callback for the Tab key with Shift modifier."
+        self.change_cell()
+        x, y = self.currentxy
+        self.setcurrent(max(1, x-1), y)
+        return "break"
+
+    def change_cell(self):
+        "Set the current cell from the entry widget."
+        x, y = self.currentxy
+        text = self.entry.get()
+        cell = None
+        if text.startswith('='):
+            cell = FormulaCell(text[1:])
+        else:
+            for cls in int, float, complex:
+                try:
+                    value = cls(text)
+                except (TypeError, ValueError):
+                    continue
+                else:
+                    cell = NumericCell(value)
+                    break
+        if cell is None and text:
+            cell = StringCell(text)
+        if cell is None:
+            self.sheet.clearcell(x, y)
+        else:
+            self.sheet.setcell(x, y, cell)
+        self.sync()
+
+    def sync(self):
+        "Fill the GUI cells from the sheet cells."
+        self.sheet.recalc()
+        for (x, y), gridcell in self.gridcells.items():
+            if x == 0 or y == 0:
+                continue
+            cell = self.sheet.getcell(x, y)
+            if cell is None:
+                gridcell['text'] = ""
+            else:
+                if hasattr(cell, 'format'):
+                    text, alignment = cell.format()
+                else:
+                    text, alignment = str(cell), LEFT
+                gridcell['text'] = text
+                gridcell['anchor'] = align2anchor[alignment]
+
+
+def test_basic():
+    "Basic non-gui self-test."
+    a = Sheet()
+    for x in range(1, 11):
+        for y in range(1, 11):
+            if x == 1:
+                cell = NumericCell(y)
+            elif y == 1:
+                cell = NumericCell(x)
+            else:
+                c1 = cellname(x, 1)
+                c2 = cellname(1, y)
+                formula = "%s*%s" % (c1, c2)
+                cell = FormulaCell(formula)
+            a.setcell(x, y, cell)
+##    if os.path.isfile("sheet1.xml"):
+##        print "Loading from sheet1.xml"
+##        a.load("sheet1.xml")
+    a.display()
+    a.save("sheet1.xml")
+
+def test_gui():
+    "GUI test."
+    if sys.argv[1:]:
+        filename = sys.argv[1]
+    else:
+        filename = "sheet1.xml"
+    g = SheetGUI(filename)
+    g.root.mainloop()
+
+if __name__ == '__main__':
+    #test_basic()
+    test_gui()


From webhook-mailer at python.org  Wed Oct 23 00:05:11 2019
From: webhook-mailer at python.org (Raymond Hettinger)
Date: Wed, 23 Oct 2019 04:05:11 -0000
Subject: [Python-checkins] bpo-38539: Finish rename of ss1.py to
 spreadsheet.py (GH-16896)
Message-ID: <mailman.110.1571803512.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/ea6041cd7ff0d752296d1759927eff898ceba864
commit: ea6041cd7ff0d752296d1759927eff898ceba864
branch: master
author: Raymond Hettinger <rhettinger at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-22T21:05:02-07:00
summary:

bpo-38539:  Finish rename of ss1.py to spreadsheet.py (GH-16896)

files:
D Tools/demo/ss1.py

diff --git a/Tools/demo/ss1.py b/Tools/demo/ss1.py
deleted file mode 100755
index bf88820dca347..0000000000000
--- a/Tools/demo/ss1.py
+++ /dev/null
@@ -1,829 +0,0 @@
-#!/usr/bin/env python3
-
-"""
-SS1 -- a spreadsheet-like application.
-"""
-
-import os
-import re
-import sys
-from xml.parsers import expat
-from xml.sax.saxutils import escape
-
-LEFT, CENTER, RIGHT = "LEFT", "CENTER", "RIGHT"
-
-def ljust(x, n):
-    return x.ljust(n)
-def center(x, n):
-    return x.center(n)
-def rjust(x, n):
-    return x.rjust(n)
-align2action = {LEFT: ljust, CENTER: center, RIGHT: rjust}
-
-align2xml = {LEFT: "left", CENTER: "center", RIGHT: "right"}
-xml2align = {"left": LEFT, "center": CENTER, "right": RIGHT}
-
-align2anchor = {LEFT: "w", CENTER: "center", RIGHT: "e"}
-
-def sum(seq):
-    total = 0
-    for x in seq:
-        if x is not None:
-            total += x
-    return total
-
-class Sheet:
-
-    def __init__(self):
-        self.cells = {} # {(x, y): cell, ...}
-        self.ns = dict(
-            cell = self.cellvalue,
-            cells = self.multicellvalue,
-            sum = sum,
-        )
-
-    def cellvalue(self, x, y):
-        cell = self.getcell(x, y)
-        if hasattr(cell, 'recalc'):
-            return cell.recalc(self.ns)
-        else:
-            return cell
-
-    def multicellvalue(self, x1, y1, x2, y2):
-        if x1 > x2:
-            x1, x2 = x2, x1
-        if y1 > y2:
-            y1, y2 = y2, y1
-        seq = []
-        for y in range(y1, y2+1):
-            for x in range(x1, x2+1):
-                seq.append(self.cellvalue(x, y))
-        return seq
-
-    def getcell(self, x, y):
-        return self.cells.get((x, y))
-
-    def setcell(self, x, y, cell):
-        assert x > 0 and y > 0
-        assert isinstance(cell, BaseCell)
-        self.cells[x, y] = cell
-
-    def clearcell(self, x, y):
-        try:
-            del self.cells[x, y]
-        except KeyError:
-            pass
-
-    def clearcells(self, x1, y1, x2, y2):
-        for xy in self.selectcells(x1, y1, x2, y2):
-            del self.cells[xy]
-
-    def clearrows(self, y1, y2):
-        self.clearcells(0, y1, sys.maxsize, y2)
-
-    def clearcolumns(self, x1, x2):
-        self.clearcells(x1, 0, x2, sys.maxsize)
-
-    def selectcells(self, x1, y1, x2, y2):
-        if x1 > x2:
-            x1, x2 = x2, x1
-        if y1 > y2:
-            y1, y2 = y2, y1
-        return [(x, y) for x, y in self.cells
-                if x1 <= x <= x2 and y1 <= y <= y2]
-
-    def movecells(self, x1, y1, x2, y2, dx, dy):
-        if dx == 0 and dy == 0:
-            return
-        if x1 > x2:
-            x1, x2 = x2, x1
-        if y1 > y2:
-            y1, y2 = y2, y1
-        assert x1+dx > 0 and y1+dy > 0
-        new = {}
-        for x, y in self.cells:
-            cell = self.cells[x, y]
-            if hasattr(cell, 'renumber'):
-                cell = cell.renumber(x1, y1, x2, y2, dx, dy)
-            if x1 <= x <= x2 and y1 <= y <= y2:
-                x += dx
-                y += dy
-            new[x, y] = cell
-        self.cells = new
-
-    def insertrows(self, y, n):
-        assert n > 0
-        self.movecells(0, y, sys.maxsize, sys.maxsize, 0, n)
-
-    def deleterows(self, y1, y2):
-        if y1 > y2:
-            y1, y2 = y2, y1
-        self.clearrows(y1, y2)
-        self.movecells(0, y2+1, sys.maxsize, sys.maxsize, 0, y1-y2-1)
-
-    def insertcolumns(self, x, n):
-        assert n > 0
-        self.movecells(x, 0, sys.maxsize, sys.maxsize, n, 0)
-
-    def deletecolumns(self, x1, x2):
-        if x1 > x2:
-            x1, x2 = x2, x1
-        self.clearcells(x1, x2)
-        self.movecells(x2+1, 0, sys.maxsize, sys.maxsize, x1-x2-1, 0)
-
-    def getsize(self):
-        maxx = maxy = 0
-        for x, y in self.cells:
-            maxx = max(maxx, x)
-            maxy = max(maxy, y)
-        return maxx, maxy
-
-    def reset(self):
-        for cell in self.cells.values():
-            if hasattr(cell, 'reset'):
-                cell.reset()
-
-    def recalc(self):
-        self.reset()
-        for cell in self.cells.values():
-            if hasattr(cell, 'recalc'):
-                cell.recalc(self.ns)
-
-    def display(self):
-        maxx, maxy = self.getsize()
-        width, height = maxx+1, maxy+1
-        colwidth = [1] * width
-        full = {}
-        # Add column heading labels in row 0
-        for x in range(1, width):
-            full[x, 0] = text, alignment = colnum2name(x), RIGHT
-            colwidth[x] = max(colwidth[x], len(text))
-        # Add row labels in column 0
-        for y in range(1, height):
-            full[0, y] = text, alignment = str(y), RIGHT
-            colwidth[0] = max(colwidth[0], len(text))
-        # Add sheet cells in columns with x>0 and y>0
-        for (x, y), cell in self.cells.items():
-            if x <= 0 or y <= 0:
-                continue
-            if hasattr(cell, 'recalc'):
-                cell.recalc(self.ns)
-            if hasattr(cell, 'format'):
-                text, alignment = cell.format()
-                assert isinstance(text, str)
-                assert alignment in (LEFT, CENTER, RIGHT)
-            else:
-                text = str(cell)
-                if isinstance(cell, str):
-                    alignment = LEFT
-                else:
-                    alignment = RIGHT
-            full[x, y] = (text, alignment)
-            colwidth[x] = max(colwidth[x], len(text))
-        # Calculate the horizontal separator line (dashes and dots)
-        sep = ""
-        for x in range(width):
-            if sep:
-                sep += "+"
-            sep += "-"*colwidth[x]
-        # Now print The full grid
-        for y in range(height):
-            line = ""
-            for x in range(width):
-                text, alignment = full.get((x, y)) or ("", LEFT)
-                text = align2action[alignment](text, colwidth[x])
-                if line:
-                    line += '|'
-                line += text
-            print(line)
-            if y == 0:
-                print(sep)
-
-    def xml(self):
-        out = ['<spreadsheet>']
-        for (x, y), cell in self.cells.items():
-            if hasattr(cell, 'xml'):
-                cellxml = cell.xml()
-            else:
-                cellxml = '<value>%s</value>' % escape(cell)
-            out.append('<cell row="%s" col="%s">\n  %s\n</cell>' %
-                       (y, x, cellxml))
-        out.append('</spreadsheet>')
-        return '\n'.join(out)
-
-    def save(self, filename):
-        text = self.xml()
-        with open(filename, "w", encoding='utf-8') as f:
-            f.write(text)
-            if text and not text.endswith('\n'):
-                f.write('\n')
-
-    def load(self, filename):
-        with open(filename, 'rb') as f:
-            SheetParser(self).parsefile(f)
-
-class SheetParser:
-
-    def __init__(self, sheet):
-        self.sheet = sheet
-
-    def parsefile(self, f):
-        parser = expat.ParserCreate()
-        parser.StartElementHandler = self.startelement
-        parser.EndElementHandler = self.endelement
-        parser.CharacterDataHandler = self.data
-        parser.ParseFile(f)
-
-    def startelement(self, tag, attrs):
-        method = getattr(self, 'start_'+tag, None)
-        if method:
-            method(attrs)
-        self.texts = []
-
-    def data(self, text):
-        self.texts.append(text)
-
-    def endelement(self, tag):
-        method = getattr(self, 'end_'+tag, None)
-        if method:
-            method("".join(self.texts))
-
-    def start_cell(self, attrs):
-        self.y = int(attrs.get("row"))
-        self.x = int(attrs.get("col"))
-
-    def start_value(self, attrs):
-        self.fmt = attrs.get('format')
-        self.alignment = xml2align.get(attrs.get('align'))
-
-    start_formula = start_value
-
-    def end_int(self, text):
-        try:
-            self.value = int(text)
-        except (TypeError, ValueError):
-            self.value = None
-
-    end_long = end_int
-
-    def end_double(self, text):
-        try:
-            self.value = float(text)
-        except (TypeError, ValueError):
-            self.value = None
-
-    def end_complex(self, text):
-        try:
-            self.value = complex(text)
-        except (TypeError, ValueError):
-            self.value = None
-
-    def end_string(self, text):
-        self.value = text
-
-    def end_value(self, text):
-        if isinstance(self.value, BaseCell):
-            self.cell = self.value
-        elif isinstance(self.value, str):
-            self.cell = StringCell(self.value,
-                                   self.fmt or "%s",
-                                   self.alignment or LEFT)
-        else:
-            self.cell = NumericCell(self.value,
-                                    self.fmt or "%s",
-                                    self.alignment or RIGHT)
-
-    def end_formula(self, text):
-        self.cell = FormulaCell(text,
-                                self.fmt or "%s",
-                                self.alignment or RIGHT)
-
-    def end_cell(self, text):
-        self.sheet.setcell(self.x, self.y, self.cell)
-
-class BaseCell:
-    __init__ = None # Must provide
-    """Abstract base class for sheet cells.
-
-    Subclasses may but needn't provide the following APIs:
-
-    cell.reset() -- prepare for recalculation
-    cell.recalc(ns) -> value -- recalculate formula
-    cell.format() -> (value, alignment) -- return formatted value
-    cell.xml() -> string -- return XML
-    """
-
-class NumericCell(BaseCell):
-
-    def __init__(self, value, fmt="%s", alignment=RIGHT):
-        assert isinstance(value, (int, float, complex))
-        assert alignment in (LEFT, CENTER, RIGHT)
-        self.value = value
-        self.fmt = fmt
-        self.alignment = alignment
-
-    def recalc(self, ns):
-        return self.value
-
-    def format(self):
-        try:
-            text = self.fmt % self.value
-        except:
-            text = str(self.value)
-        return text, self.alignment
-
-    def xml(self):
-        method = getattr(self, '_xml_' + type(self.value).__name__)
-        return '<value align="%s" format="%s">%s</value>' % (
-                align2xml[self.alignment],
-                self.fmt,
-                method())
-
-    def _xml_int(self):
-        if -2**31 <= self.value < 2**31:
-            return '<int>%s</int>' % self.value
-        else:
-            return '<long>%s</long>' % self.value
-
-    def _xml_float(self):
-        return '<double>%r</double>' % self.value
-
-    def _xml_complex(self):
-        return '<complex>%r</complex>' % self.value
-
-class StringCell(BaseCell):
-
-    def __init__(self, text, fmt="%s", alignment=LEFT):
-        assert isinstance(text, str)
-        assert alignment in (LEFT, CENTER, RIGHT)
-        self.text = text
-        self.fmt = fmt
-        self.alignment = alignment
-
-    def recalc(self, ns):
-        return self.text
-
-    def format(self):
-        return self.text, self.alignment
-
-    def xml(self):
-        s = '<value align="%s" format="%s"><string>%s</string></value>'
-        return s % (
-            align2xml[self.alignment],
-            self.fmt,
-            escape(self.text))
-
-class FormulaCell(BaseCell):
-
-    def __init__(self, formula, fmt="%s", alignment=RIGHT):
-        assert alignment in (LEFT, CENTER, RIGHT)
-        self.formula = formula
-        self.translated = translate(self.formula)
-        self.fmt = fmt
-        self.alignment = alignment
-        self.reset()
-
-    def reset(self):
-        self.value = None
-
-    def recalc(self, ns):
-        if self.value is None:
-            try:
-                self.value = eval(self.translated, ns)
-            except:
-                exc = sys.exc_info()[0]
-                if hasattr(exc, "__name__"):
-                    self.value = exc.__name__
-                else:
-                    self.value = str(exc)
-        return self.value
-
-    def format(self):
-        try:
-            text = self.fmt % self.value
-        except:
-            text = str(self.value)
-        return text, self.alignment
-
-    def xml(self):
-        return '<formula align="%s" format="%s">%s</formula>' % (
-            align2xml[self.alignment],
-            self.fmt,
-            escape(self.formula))
-
-    def renumber(self, x1, y1, x2, y2, dx, dy):
-        out = []
-        for part in re.split(r'(\w+)', self.formula):
-            m = re.match('^([A-Z]+)([1-9][0-9]*)$', part)
-            if m is not None:
-                sx, sy = m.groups()
-                x = colname2num(sx)
-                y = int(sy)
-                if x1 <= x <= x2 and y1 <= y <= y2:
-                    part = cellname(x+dx, y+dy)
-            out.append(part)
-        return FormulaCell("".join(out), self.fmt, self.alignment)
-
-def translate(formula):
-    """Translate a formula containing fancy cell names to valid Python code.
-
-    Examples:
-        B4 -> cell(2, 4)
-        B4:Z100 -> cells(2, 4, 26, 100)
-    """
-    out = []
-    for part in re.split(r"(\w+(?::\w+)?)", formula):
-        m = re.match(r"^([A-Z]+)([1-9][0-9]*)(?::([A-Z]+)([1-9][0-9]*))?$", part)
-        if m is None:
-            out.append(part)
-        else:
-            x1, y1, x2, y2 = m.groups()
-            x1 = colname2num(x1)
-            if x2 is None:
-                s = "cell(%s, %s)" % (x1, y1)
-            else:
-                x2 = colname2num(x2)
-                s = "cells(%s, %s, %s, %s)" % (x1, y1, x2, y2)
-            out.append(s)
-    return "".join(out)
-
-def cellname(x, y):
-    "Translate a cell coordinate to a fancy cell name (e.g. (1, 1)->'A1')."
-    assert x > 0 # Column 0 has an empty name, so can't use that
-    return colnum2name(x) + str(y)
-
-def colname2num(s):
-    "Translate a column name to number (e.g. 'A'->1, 'Z'->26, 'AA'->27)."
-    s = s.upper()
-    n = 0
-    for c in s:
-        assert 'A' <= c <= 'Z'
-        n = n*26 + ord(c) - ord('A') + 1
-    return n
-
-def colnum2name(n):
-    "Translate a column number to name (e.g. 1->'A', etc.)."
-    assert n > 0
-    s = ""
-    while n:
-        n, m = divmod(n-1, 26)
-        s = chr(m+ord('A')) + s
-    return s
-
-import tkinter as Tk
-
-class SheetGUI:
-
-    """Beginnings of a GUI for a spreadsheet.
-
-    TO DO:
-    - clear multiple cells
-    - Insert, clear, remove rows or columns
-    - Show new contents while typing
-    - Scroll bars
-    - Grow grid when window is grown
-    - Proper menus
-    - Undo, redo
-    - Cut, copy and paste
-    - Formatting and alignment
-    """
-
-    def __init__(self, filename="sheet1.xml", rows=10, columns=5):
-        """Constructor.
-
-        Load the sheet from the filename argument.
-        Set up the Tk widget tree.
-        """
-        # Create and load the sheet
-        self.filename = filename
-        self.sheet = Sheet()
-        if os.path.isfile(filename):
-            self.sheet.load(filename)
-        # Calculate the needed grid size
-        maxx, maxy = self.sheet.getsize()
-        rows = max(rows, maxy)
-        columns = max(columns, maxx)
-        # Create the widgets
-        self.root = Tk.Tk()
-        self.root.wm_title("Spreadsheet: %s" % self.filename)
-        self.beacon = Tk.Label(self.root, text="A1",
-                               font=('helvetica', 16, 'bold'))
-        self.entry = Tk.Entry(self.root)
-        self.savebutton = Tk.Button(self.root, text="Save",
-                                    command=self.save)
-        self.cellgrid = Tk.Frame(self.root)
-        # Configure the widget lay-out
-        self.cellgrid.pack(side="bottom", expand=1, fill="both")
-        self.beacon.pack(side="left")
-        self.savebutton.pack(side="right")
-        self.entry.pack(side="left", expand=1, fill="x")
-        # Bind some events
-        self.entry.bind("<Return>", self.return_event)
-        self.entry.bind("<Shift-Return>", self.shift_return_event)
-        self.entry.bind("<Tab>", self.tab_event)
-        self.entry.bind("<Shift-Tab>", self.shift_tab_event)
-        self.entry.bind("<Delete>", self.delete_event)
-        self.entry.bind("<Escape>", self.escape_event)
-        # Now create the cell grid
-        self.makegrid(rows, columns)
-        # Select the top-left cell
-        self.currentxy = None
-        self.cornerxy = None
-        self.setcurrent(1, 1)
-        # Copy the sheet cells to the GUI cells
-        self.sync()
-
-    def delete_event(self, event):
-        if self.cornerxy != self.currentxy and self.cornerxy is not None:
-            self.sheet.clearcells(*(self.currentxy + self.cornerxy))
-        else:
-            self.sheet.clearcell(*self.currentxy)
-        self.sync()
-        self.entry.delete(0, 'end')
-        return "break"
-
-    def escape_event(self, event):
-        x, y = self.currentxy
-        self.load_entry(x, y)
-
-    def load_entry(self, x, y):
-        cell = self.sheet.getcell(x, y)
-        if cell is None:
-            text = ""
-        elif isinstance(cell, FormulaCell):
-            text = '=' + cell.formula
-        else:
-            text, alignment = cell.format()
-        self.entry.delete(0, 'end')
-        self.entry.insert(0, text)
-        self.entry.selection_range(0, 'end')
-
-    def makegrid(self, rows, columns):
-        """Helper to create the grid of GUI cells.
-
-        The edge (x==0 or y==0) is filled with labels; the rest is real cells.
-        """
-        self.rows = rows
-        self.columns = columns
-        self.gridcells = {}
-        # Create the top left corner cell (which selects all)
-        cell = Tk.Label(self.cellgrid, relief='raised')
-        cell.grid_configure(column=0, row=0, sticky='NSWE')
-        cell.bind("<ButtonPress-1>", self.selectall)
-        # Create the top row of labels, and configure the grid columns
-        for x in range(1, columns+1):
-            self.cellgrid.grid_columnconfigure(x, minsize=64)
-            cell = Tk.Label(self.cellgrid, text=colnum2name(x), relief='raised')
-            cell.grid_configure(column=x, row=0, sticky='WE')
-            self.gridcells[x, 0] = cell
-            cell.__x = x
-            cell.__y = 0
-            cell.bind("<ButtonPress-1>", self.selectcolumn)
-            cell.bind("<B1-Motion>", self.extendcolumn)
-            cell.bind("<ButtonRelease-1>", self.extendcolumn)
-            cell.bind("<Shift-Button-1>", self.extendcolumn)
-        # Create the leftmost column of labels
-        for y in range(1, rows+1):
-            cell = Tk.Label(self.cellgrid, text=str(y), relief='raised')
-            cell.grid_configure(column=0, row=y, sticky='WE')
-            self.gridcells[0, y] = cell
-            cell.__x = 0
-            cell.__y = y
-            cell.bind("<ButtonPress-1>", self.selectrow)
-            cell.bind("<B1-Motion>", self.extendrow)
-            cell.bind("<ButtonRelease-1>", self.extendrow)
-            cell.bind("<Shift-Button-1>", self.extendrow)
-        # Create the real cells
-        for x in range(1, columns+1):
-            for y in range(1, rows+1):
-                cell = Tk.Label(self.cellgrid, relief='sunken',
-                                bg='white', fg='black')
-                cell.grid_configure(column=x, row=y, sticky='NSWE')
-                self.gridcells[x, y] = cell
-                cell.__x = x
-                cell.__y = y
-                # Bind mouse events
-                cell.bind("<ButtonPress-1>", self.press)
-                cell.bind("<B1-Motion>", self.motion)
-                cell.bind("<ButtonRelease-1>", self.release)
-                cell.bind("<Shift-Button-1>", self.release)
-
-    def selectall(self, event):
-        self.setcurrent(1, 1)
-        self.setcorner(sys.maxsize, sys.maxsize)
-
-    def selectcolumn(self, event):
-        x, y = self.whichxy(event)
-        self.setcurrent(x, 1)
-        self.setcorner(x, sys.maxsize)
-
-    def extendcolumn(self, event):
-        x, y = self.whichxy(event)
-        if x > 0:
-            self.setcurrent(self.currentxy[0], 1)
-            self.setcorner(x, sys.maxsize)
-
-    def selectrow(self, event):
-        x, y = self.whichxy(event)
-        self.setcurrent(1, y)
-        self.setcorner(sys.maxsize, y)
-
-    def extendrow(self, event):
-        x, y = self.whichxy(event)
-        if y > 0:
-            self.setcurrent(1, self.currentxy[1])
-            self.setcorner(sys.maxsize, y)
-
-    def press(self, event):
-        x, y = self.whichxy(event)
-        if x > 0 and y > 0:
-            self.setcurrent(x, y)
-
-    def motion(self, event):
-        x, y = self.whichxy(event)
-        if x > 0 and y > 0:
-            self.setcorner(x, y)
-
-    release = motion
-
-    def whichxy(self, event):
-        w = self.cellgrid.winfo_containing(event.x_root, event.y_root)
-        if w is not None and isinstance(w, Tk.Label):
-            try:
-                return w.__x, w.__y
-            except AttributeError:
-                pass
-        return 0, 0
-
-    def save(self):
-        self.sheet.save(self.filename)
-
-    def setcurrent(self, x, y):
-        "Make (x, y) the current cell."
-        if self.currentxy is not None:
-            self.change_cell()
-        self.clearfocus()
-        self.beacon['text'] = cellname(x, y)
-        self.load_entry(x, y)
-        self.entry.focus_set()
-        self.currentxy = x, y
-        self.cornerxy = None
-        gridcell = self.gridcells.get(self.currentxy)
-        if gridcell is not None:
-            gridcell['bg'] = 'yellow'
-
-    def setcorner(self, x, y):
-        if self.currentxy is None or self.currentxy == (x, y):
-            self.setcurrent(x, y)
-            return
-        self.clearfocus()
-        self.cornerxy = x, y
-        x1, y1 = self.currentxy
-        x2, y2 = self.cornerxy or self.currentxy
-        if x1 > x2:
-            x1, x2 = x2, x1
-        if y1 > y2:
-            y1, y2 = y2, y1
-        for (x, y), cell in self.gridcells.items():
-            if x1 <= x <= x2 and y1 <= y <= y2:
-                cell['bg'] = 'lightBlue'
-        gridcell = self.gridcells.get(self.currentxy)
-        if gridcell is not None:
-            gridcell['bg'] = 'yellow'
-        self.setbeacon(x1, y1, x2, y2)
-
-    def setbeacon(self, x1, y1, x2, y2):
-        if x1 == y1 == 1 and x2 == y2 == sys.maxsize:
-            name = ":"
-        elif (x1, x2) == (1, sys.maxsize):
-            if y1 == y2:
-                name = "%d" % y1
-            else:
-                name = "%d:%d" % (y1, y2)
-        elif (y1, y2) == (1, sys.maxsize):
-            if x1 == x2:
-                name = "%s" % colnum2name(x1)
-            else:
-                name = "%s:%s" % (colnum2name(x1), colnum2name(x2))
-        else:
-            name1 = cellname(*self.currentxy)
-            name2 = cellname(*self.cornerxy)
-            name = "%s:%s" % (name1, name2)
-        self.beacon['text'] = name
-
-
-    def clearfocus(self):
-        if self.currentxy is not None:
-            x1, y1 = self.currentxy
-            x2, y2 = self.cornerxy or self.currentxy
-            if x1 > x2:
-                x1, x2 = x2, x1
-            if y1 > y2:
-                y1, y2 = y2, y1
-            for (x, y), cell in self.gridcells.items():
-                if x1 <= x <= x2 and y1 <= y <= y2:
-                    cell['bg'] = 'white'
-
-    def return_event(self, event):
-        "Callback for the Return key."
-        self.change_cell()
-        x, y = self.currentxy
-        self.setcurrent(x, y+1)
-        return "break"
-
-    def shift_return_event(self, event):
-        "Callback for the Return key with Shift modifier."
-        self.change_cell()
-        x, y = self.currentxy
-        self.setcurrent(x, max(1, y-1))
-        return "break"
-
-    def tab_event(self, event):
-        "Callback for the Tab key."
-        self.change_cell()
-        x, y = self.currentxy
-        self.setcurrent(x+1, y)
-        return "break"
-
-    def shift_tab_event(self, event):
-        "Callback for the Tab key with Shift modifier."
-        self.change_cell()
-        x, y = self.currentxy
-        self.setcurrent(max(1, x-1), y)
-        return "break"
-
-    def change_cell(self):
-        "Set the current cell from the entry widget."
-        x, y = self.currentxy
-        text = self.entry.get()
-        cell = None
-        if text.startswith('='):
-            cell = FormulaCell(text[1:])
-        else:
-            for cls in int, float, complex:
-                try:
-                    value = cls(text)
-                except (TypeError, ValueError):
-                    continue
-                else:
-                    cell = NumericCell(value)
-                    break
-        if cell is None and text:
-            cell = StringCell(text)
-        if cell is None:
-            self.sheet.clearcell(x, y)
-        else:
-            self.sheet.setcell(x, y, cell)
-        self.sync()
-
-    def sync(self):
-        "Fill the GUI cells from the sheet cells."
-        self.sheet.recalc()
-        for (x, y), gridcell in self.gridcells.items():
-            if x == 0 or y == 0:
-                continue
-            cell = self.sheet.getcell(x, y)
-            if cell is None:
-                gridcell['text'] = ""
-            else:
-                if hasattr(cell, 'format'):
-                    text, alignment = cell.format()
-                else:
-                    text, alignment = str(cell), LEFT
-                gridcell['text'] = text
-                gridcell['anchor'] = align2anchor[alignment]
-
-
-def test_basic():
-    "Basic non-gui self-test."
-    a = Sheet()
-    for x in range(1, 11):
-        for y in range(1, 11):
-            if x == 1:
-                cell = NumericCell(y)
-            elif y == 1:
-                cell = NumericCell(x)
-            else:
-                c1 = cellname(x, 1)
-                c2 = cellname(1, y)
-                formula = "%s*%s" % (c1, c2)
-                cell = FormulaCell(formula)
-            a.setcell(x, y, cell)
-##    if os.path.isfile("sheet1.xml"):
-##        print "Loading from sheet1.xml"
-##        a.load("sheet1.xml")
-    a.display()
-    a.save("sheet1.xml")
-
-def test_gui():
-    "GUI test."
-    if sys.argv[1:]:
-        filename = sys.argv[1]
-    else:
-        filename = "sheet1.xml"
-    g = SheetGUI(filename)
-    g.root.mainloop()
-
-if __name__ == '__main__':
-    #test_basic()
-    test_gui()


From webhook-mailer at python.org  Wed Oct 23 04:09:12 2019
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Wed, 23 Oct 2019 08:09:12 -0000
Subject: [Python-checkins] Replace _pysqlite_long_from_int64() with
 PyLong_FromLongLong() (GH-16882)
Message-ID: <mailman.111.1571818154.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/b6f5b9dd21a37fa97af38521941025259f1556e0
commit: b6f5b9dd21a37fa97af38521941025259f1556e0
branch: master
author: Sergey Fedoseev <fedoseev.sergey at gmail.com>
committer: Serhiy Storchaka <storchaka at gmail.com>
date: 2019-10-23T11:09:01+03:00
summary:

Replace _pysqlite_long_from_int64() with PyLong_FromLongLong() (GH-16882)

files:
M Modules/_sqlite/connection.c
M Modules/_sqlite/cursor.c
M Modules/_sqlite/util.c
M Modules/_sqlite/util.h

diff --git a/Modules/_sqlite/connection.c b/Modules/_sqlite/connection.c
index ae03aed8f0b88..64051669185ba 100644
--- a/Modules/_sqlite/connection.c
+++ b/Modules/_sqlite/connection.c
@@ -550,7 +550,7 @@ PyObject* _pysqlite_build_py_params(sqlite3_context *context, int argc, sqlite3_
         cur_value = argv[i];
         switch (sqlite3_value_type(argv[i])) {
             case SQLITE_INTEGER:
-                cur_py_value = _pysqlite_long_from_int64(sqlite3_value_int64(cur_value));
+                cur_py_value = PyLong_FromLongLong(sqlite3_value_int64(cur_value));
                 break;
             case SQLITE_FLOAT:
                 cur_py_value = PyFloat_FromDouble(sqlite3_value_double(cur_value));
diff --git a/Modules/_sqlite/cursor.c b/Modules/_sqlite/cursor.c
index 6d9685b296425..45f5865590d69 100644
--- a/Modules/_sqlite/cursor.c
+++ b/Modules/_sqlite/cursor.c
@@ -277,7 +277,7 @@ _pysqlite_fetch_one_row(pysqlite_Cursor* self)
                 Py_INCREF(Py_None);
                 converted = Py_None;
             } else if (coltype == SQLITE_INTEGER) {
-                converted = _pysqlite_long_from_int64(sqlite3_column_int64(self->statement->st, i));
+                converted = PyLong_FromLongLong(sqlite3_column_int64(self->statement->st, i));
             } else if (coltype == SQLITE_FLOAT) {
                 converted = PyFloat_FromDouble(sqlite3_column_double(self->statement->st, i));
             } else if (coltype == SQLITE_TEXT) {
@@ -558,7 +558,7 @@ _pysqlite_query_execute(pysqlite_Cursor* self, int multiple, PyObject* args)
             Py_BEGIN_ALLOW_THREADS
             lastrowid = sqlite3_last_insert_rowid(self->connection->db);
             Py_END_ALLOW_THREADS
-            self->lastrowid = _pysqlite_long_from_int64(lastrowid);
+            self->lastrowid = PyLong_FromLongLong(lastrowid);
         }
 
         if (rc == SQLITE_ROW) {
diff --git a/Modules/_sqlite/util.c b/Modules/_sqlite/util.c
index 3fa671d052b0d..1dbabcdd94a81 100644
--- a/Modules/_sqlite/util.c
+++ b/Modules/_sqlite/util.c
@@ -103,22 +103,6 @@ int _pysqlite_seterror(sqlite3* db, sqlite3_stmt* st)
 # define IS_LITTLE_ENDIAN 1
 #endif
 
-PyObject *
-_pysqlite_long_from_int64(sqlite_int64 value)
-{
-# if SIZEOF_LONG_LONG < 8
-    if (value > PY_LLONG_MAX || value < PY_LLONG_MIN) {
-        return _PyLong_FromByteArray(&value, sizeof(value),
-                                     IS_LITTLE_ENDIAN, 1 /* signed */);
-    }
-# endif
-# if SIZEOF_LONG < SIZEOF_LONG_LONG
-    if (value > LONG_MAX || value < LONG_MIN)
-        return PyLong_FromLongLong(value);
-# endif
-    return PyLong_FromLong(Py_SAFE_DOWNCAST(value, sqlite_int64, long));
-}
-
 sqlite_int64
 _pysqlite_long_as_int64(PyObject * py_val)
 {
diff --git a/Modules/_sqlite/util.h b/Modules/_sqlite/util.h
index 626191118f925..c5a220e9b0aa7 100644
--- a/Modules/_sqlite/util.h
+++ b/Modules/_sqlite/util.h
@@ -37,7 +37,6 @@ int pysqlite_step(sqlite3_stmt* statement, pysqlite_Connection* connection);
  */
 int _pysqlite_seterror(sqlite3* db, sqlite3_stmt* st);
 
-PyObject * _pysqlite_long_from_int64(sqlite_int64 value);
 sqlite_int64 _pysqlite_long_as_int64(PyObject * value);
 
 #if SQLITE_VERSION_NUMBER >= 3007014


From webhook-mailer at python.org  Wed Oct 23 05:17:45 2019
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Wed, 23 Oct 2019 09:17:45 -0000
Subject: [Python-checkins] Spell Bitbucket correctly. (GH-16862)
Message-ID: <mailman.112.1571822266.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/d34ac305327420bd68f05e201b63d2a665b073f8
commit: d34ac305327420bd68f05e201b63d2a665b073f8
branch: master
author: Peter Bittner <django at bittner.it>
committer: Serhiy Storchaka <storchaka at gmail.com>
date: 2019-10-23T12:17:30+03:00
summary:

Spell Bitbucket correctly. (GH-16862)

files:
M Doc/distributing/index.rst
M Doc/installing/index.rst

diff --git a/Doc/distributing/index.rst b/Doc/distributing/index.rst
index 2e46c7ac8635e..5f7b3bbc4f917 100644
--- a/Doc/distributing/index.rst
+++ b/Doc/distributing/index.rst
@@ -40,7 +40,7 @@ Key terms
   evolution of the standard packaging tools and the associated metadata and
   file format standards. They maintain a variety of tools, documentation
   and issue trackers on both `GitHub <https://github.com/pypa>`__ and
-  `BitBucket <https://bitbucket.org/pypa/>`__.
+  `Bitbucket <https://bitbucket.org/pypa/>`__.
 * :mod:`distutils` is the original build and distribution system first added
   to the Python standard library in 1998. While direct use of :mod:`distutils`
   is being phased out, it still laid the foundation for the current packaging
@@ -148,7 +148,7 @@ These are quick answers or links for some common tasks.
 This isn't an easy topic, but here are a few tips:
 
 * check the Python Packaging Index to see if the name is already in use
-* check popular hosting sites like GitHub, BitBucket, etc to see if there
+* check popular hosting sites like GitHub, Bitbucket, etc to see if there
   is already a project with that name
 * check what comes up in a web search for the name you're considering
 * avoid particularly common words, especially ones with multiple meanings,
diff --git a/Doc/installing/index.rst b/Doc/installing/index.rst
index dc44aa64b8e04..31e9b0bde0724 100644
--- a/Doc/installing/index.rst
+++ b/Doc/installing/index.rst
@@ -53,7 +53,7 @@ Key terms
   evolution of the standard packaging tools and the associated metadata and
   file format standards. They maintain a variety of tools, documentation,
   and issue trackers on both `GitHub <https://github.com/pypa>`__ and
-  `BitBucket <https://bitbucket.org/pypa/>`__.
+  `Bitbucket <https://bitbucket.org/pypa/>`__.
 * ``distutils`` is the original build and distribution system first added to
   the Python standard library in 1998. While direct use of ``distutils`` is
   being phased out, it still laid the foundation for the current packaging


From webhook-mailer at python.org  Wed Oct 23 07:45:16 2019
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Wed, 23 Oct 2019 11:45:16 -0000
Subject: [Python-checkins] Spell Bitbucket correctly. (GH-16862) (GH-16899)
Message-ID: <mailman.113.1571831117.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/2e7eaf9ff03161d2d982a22307fb031e3c8a73c3
commit: 2e7eaf9ff03161d2d982a22307fb031e3c8a73c3
branch: 3.7
author: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Serhiy Storchaka <storchaka at gmail.com>
date: 2019-10-23T14:45:09+03:00
summary:

Spell Bitbucket correctly. (GH-16862) (GH-16899)

(cherry picked from commit d34ac305327420bd68f05e201b63d2a665b073f8)

Co-authored-by: Peter Bittner <django at bittner.it>

files:
M Doc/distributing/index.rst
M Doc/installing/index.rst

diff --git a/Doc/distributing/index.rst b/Doc/distributing/index.rst
index 2e46c7ac8635e..5f7b3bbc4f917 100644
--- a/Doc/distributing/index.rst
+++ b/Doc/distributing/index.rst
@@ -40,7 +40,7 @@ Key terms
   evolution of the standard packaging tools and the associated metadata and
   file format standards. They maintain a variety of tools, documentation
   and issue trackers on both `GitHub <https://github.com/pypa>`__ and
-  `BitBucket <https://bitbucket.org/pypa/>`__.
+  `Bitbucket <https://bitbucket.org/pypa/>`__.
 * :mod:`distutils` is the original build and distribution system first added
   to the Python standard library in 1998. While direct use of :mod:`distutils`
   is being phased out, it still laid the foundation for the current packaging
@@ -148,7 +148,7 @@ These are quick answers or links for some common tasks.
 This isn't an easy topic, but here are a few tips:
 
 * check the Python Packaging Index to see if the name is already in use
-* check popular hosting sites like GitHub, BitBucket, etc to see if there
+* check popular hosting sites like GitHub, Bitbucket, etc to see if there
   is already a project with that name
 * check what comes up in a web search for the name you're considering
 * avoid particularly common words, especially ones with multiple meanings,
diff --git a/Doc/installing/index.rst b/Doc/installing/index.rst
index 8af828bb8a105..c1a9a1f0a58d7 100644
--- a/Doc/installing/index.rst
+++ b/Doc/installing/index.rst
@@ -53,7 +53,7 @@ Key terms
   evolution of the standard packaging tools and the associated metadata and
   file format standards. They maintain a variety of tools, documentation,
   and issue trackers on both `GitHub <https://github.com/pypa>`__ and
-  `BitBucket <https://bitbucket.org/pypa/>`__.
+  `Bitbucket <https://bitbucket.org/pypa/>`__.
 * ``distutils`` is the original build and distribution system first added to
   the Python standard library in 1998. While direct use of ``distutils`` is
   being phased out, it still laid the foundation for the current packaging


From webhook-mailer at python.org  Wed Oct 23 07:48:12 2019
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Wed, 23 Oct 2019 11:48:12 -0000
Subject: [Python-checkins] bpo-38555: Fix an undefined behavior. (GH-16883)
Message-ID: <mailman.114.1571831293.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/2e3d873d3bd0ef4708c4fa06b6cd6972574cb9af
commit: 2e3d873d3bd0ef4708c4fa06b6cd6972574cb9af
branch: master
author: Serhiy Storchaka <storchaka at gmail.com>
committer: GitHub <noreply at github.com>
date: 2019-10-23T14:48:08+03:00
summary:

bpo-38555: Fix an undefined behavior. (GH-16883)

files:
M Objects/dictobject.c

diff --git a/Objects/dictobject.c b/Objects/dictobject.c
index 5ac7bb102b211..d909f220a984b 100644
--- a/Objects/dictobject.c
+++ b/Objects/dictobject.c
@@ -3830,22 +3830,21 @@ dictreviter_iternext(dictiterobject *di)
     PyDictKeysObject *k = d->ma_keys;
     PyObject *key, *value, *result;
 
+    if (i < 0) {
+        goto fail;
+    }
     if (d->ma_values) {
-        if (i < 0) {
-            goto fail;
-        }
         key = DK_ENTRIES(k)[i].me_key;
         value = d->ma_values[i];
         assert (value != NULL);
     }
     else {
         PyDictKeyEntry *entry_ptr = &DK_ENTRIES(k)[i];
-        while (i >= 0 && entry_ptr->me_value == NULL) {
+        while (entry_ptr->me_value == NULL) {
+            if (--i < 0) {
+                goto fail;
+            }
             entry_ptr--;
-            i--;
-        }
-        if (i < 0) {
-            goto fail;
         }
         key = entry_ptr->me_key;
         value = entry_ptr->me_value;


From webhook-mailer at python.org  Wed Oct 23 08:07:27 2019
From: webhook-mailer at python.org (Miss Skeleton (bot))
Date: Wed, 23 Oct 2019 12:07:27 -0000
Subject: [Python-checkins] bpo-38555: Fix an undefined behavior. (GH-16883)
Message-ID: <mailman.115.1571832448.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/c5d3ea89ee5244714f221dcfcd3be96de2f1da8d
commit: c5d3ea89ee5244714f221dcfcd3be96de2f1da8d
branch: 3.8
author: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-23T05:07:23-07:00
summary:

bpo-38555: Fix an undefined behavior. (GH-16883)

(cherry picked from commit 2e3d873d3bd0ef4708c4fa06b6cd6972574cb9af)

Co-authored-by: Serhiy Storchaka <storchaka at gmail.com>

files:
M Objects/dictobject.c

diff --git a/Objects/dictobject.c b/Objects/dictobject.c
index a0208be9246d9..76f4fefe0979f 100644
--- a/Objects/dictobject.c
+++ b/Objects/dictobject.c
@@ -3840,22 +3840,21 @@ dictreviter_iternext(dictiterobject *di)
     PyDictKeysObject *k = d->ma_keys;
     PyObject *key, *value, *result;
 
+    if (i < 0) {
+        goto fail;
+    }
     if (d->ma_values) {
-        if (i < 0) {
-            goto fail;
-        }
         key = DK_ENTRIES(k)[i].me_key;
         value = d->ma_values[i];
         assert (value != NULL);
     }
     else {
         PyDictKeyEntry *entry_ptr = &DK_ENTRIES(k)[i];
-        while (i >= 0 && entry_ptr->me_value == NULL) {
+        while (entry_ptr->me_value == NULL) {
+            if (--i < 0) {
+                goto fail;
+            }
             entry_ptr--;
-            i--;
-        }
-        if (i < 0) {
-            goto fail;
         }
         key = entry_ptr->me_key;
         value = entry_ptr->me_value;


From webhook-mailer at python.org  Wed Oct 23 08:18:14 2019
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Wed, 23 Oct 2019 12:18:14 -0000
Subject: [Python-checkins] Spell Bitbucket correctly. (GH-16862) (GH-16898)
Message-ID: <mailman.116.1571833095.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/c54b54b49d330caf856c05b9e4cbd28967c3413a
commit: c54b54b49d330caf856c05b9e4cbd28967c3413a
branch: 3.8
author: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Serhiy Storchaka <storchaka at gmail.com>
date: 2019-10-23T15:18:10+03:00
summary:

Spell Bitbucket correctly. (GH-16862) (GH-16898)

(cherry picked from commit d34ac305327420bd68f05e201b63d2a665b073f8)

Co-authored-by: Peter Bittner <django at bittner.it>

files:
M Doc/distributing/index.rst
M Doc/installing/index.rst

diff --git a/Doc/distributing/index.rst b/Doc/distributing/index.rst
index 2e46c7ac8635e..5f7b3bbc4f917 100644
--- a/Doc/distributing/index.rst
+++ b/Doc/distributing/index.rst
@@ -40,7 +40,7 @@ Key terms
   evolution of the standard packaging tools and the associated metadata and
   file format standards. They maintain a variety of tools, documentation
   and issue trackers on both `GitHub <https://github.com/pypa>`__ and
-  `BitBucket <https://bitbucket.org/pypa/>`__.
+  `Bitbucket <https://bitbucket.org/pypa/>`__.
 * :mod:`distutils` is the original build and distribution system first added
   to the Python standard library in 1998. While direct use of :mod:`distutils`
   is being phased out, it still laid the foundation for the current packaging
@@ -148,7 +148,7 @@ These are quick answers or links for some common tasks.
 This isn't an easy topic, but here are a few tips:
 
 * check the Python Packaging Index to see if the name is already in use
-* check popular hosting sites like GitHub, BitBucket, etc to see if there
+* check popular hosting sites like GitHub, Bitbucket, etc to see if there
   is already a project with that name
 * check what comes up in a web search for the name you're considering
 * avoid particularly common words, especially ones with multiple meanings,
diff --git a/Doc/installing/index.rst b/Doc/installing/index.rst
index dc44aa64b8e04..31e9b0bde0724 100644
--- a/Doc/installing/index.rst
+++ b/Doc/installing/index.rst
@@ -53,7 +53,7 @@ Key terms
   evolution of the standard packaging tools and the associated metadata and
   file format standards. They maintain a variety of tools, documentation,
   and issue trackers on both `GitHub <https://github.com/pypa>`__ and
-  `BitBucket <https://bitbucket.org/pypa/>`__.
+  `Bitbucket <https://bitbucket.org/pypa/>`__.
 * ``distutils`` is the original build and distribution system first added to
   the Python standard library in 1998. While direct use of ``distutils`` is
   being phased out, it still laid the foundation for the current packaging


From webhook-mailer at python.org  Wed Oct 23 11:25:38 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Wed, 23 Oct 2019 15:25:38 -0000
Subject: [Python-checkins] bpo-34679: ProactorEventLoop only uses
 set_wakeup_fd() in main thread (GH-16901)
Message-ID: <mailman.117.1571844340.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/1b53a24fb4417c764dd5933bce505f5c94249ca6
commit: 1b53a24fb4417c764dd5933bce505f5c94249ca6
branch: master
author: Victor Stinner <vstinner at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-23T17:25:29+02:00
summary:

bpo-34679: ProactorEventLoop only uses set_wakeup_fd() in main thread (GH-16901)

bpo-34679, bpo-38563: asyncio.ProactorEventLoop.close() now only calls
signal.set_wakeup_fd() in the main thread.

files:
A Misc/NEWS.d/next/Library/2019-10-23-16-25-12.bpo-34679.Bnw8o3.rst
M Lib/asyncio/proactor_events.py
M Lib/test/test_asyncio/test_windows_events.py

diff --git a/Lib/asyncio/proactor_events.py b/Lib/asyncio/proactor_events.py
index 229f56e6bb9e2..830d8edc32f90 100644
--- a/Lib/asyncio/proactor_events.py
+++ b/Lib/asyncio/proactor_events.py
@@ -627,10 +627,9 @@ def __init__(self, proactor):
         self._accept_futures = {}   # socket file descriptor => Future
         proactor.set_loop(self)
         self._make_self_pipe()
-        self_no = self._csock.fileno()
         if threading.current_thread() is threading.main_thread():
             # wakeup fd can only be installed to a file descriptor from the main thread
-            signal.set_wakeup_fd(self_no)
+            signal.set_wakeup_fd(self._csock.fileno())
 
     def _make_socket_transport(self, sock, protocol, waiter=None,
                                extra=None, server=None):
@@ -676,7 +675,8 @@ def close(self):
         if self.is_closed():
             return
 
-        signal.set_wakeup_fd(-1)
+        if threading.current_thread() is threading.main_thread():
+            signal.set_wakeup_fd(-1)
         # Call these methods before closing the event loop (before calling
         # BaseEventLoop.close), because they can schedule callbacks with
         # call_soon(), which is forbidden when the event loop is closed.
diff --git a/Lib/test/test_asyncio/test_windows_events.py b/Lib/test/test_asyncio/test_windows_events.py
index 9ed10fc20f817..6b005702c9be7 100644
--- a/Lib/test/test_asyncio/test_windows_events.py
+++ b/Lib/test/test_asyncio/test_windows_events.py
@@ -69,6 +69,8 @@ def func():
             nonlocal finished
             loop = asyncio.new_event_loop()
             loop.run_until_complete(coro())
+            # close() must not call signal.set_wakeup_fd()
+            loop.close()
             finished = True
 
         thread = threading.Thread(target=func)
diff --git a/Misc/NEWS.d/next/Library/2019-10-23-16-25-12.bpo-34679.Bnw8o3.rst b/Misc/NEWS.d/next/Library/2019-10-23-16-25-12.bpo-34679.Bnw8o3.rst
new file mode 100644
index 0000000000000..34334db6032b5
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-10-23-16-25-12.bpo-34679.Bnw8o3.rst
@@ -0,0 +1,2 @@
+asynci.ProactorEventLoop.close() now only calls signal.set_wakeup_fd() in the
+main thread.


From webhook-mailer at python.org  Wed Oct 23 11:44:04 2019
From: webhook-mailer at python.org (Miss Skeleton (bot))
Date: Wed, 23 Oct 2019 15:44:04 -0000
Subject: [Python-checkins] bpo-34679: ProactorEventLoop only uses
 set_wakeup_fd() in main thread (GH-16901)
Message-ID: <mailman.118.1571845445.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/cbf474c98e702d12c97cd16a1e44ede10ea52b5b
commit: cbf474c98e702d12c97cd16a1e44ede10ea52b5b
branch: 3.8
author: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-23T08:43:57-07:00
summary:

bpo-34679: ProactorEventLoop only uses set_wakeup_fd() in main thread (GH-16901)


bpo-34679, bpo-38563: asyncio.ProactorEventLoop.close() now only calls
signal.set_wakeup_fd() in the main thread.
(cherry picked from commit 1b53a24fb4417c764dd5933bce505f5c94249ca6)

Co-authored-by: Victor Stinner <vstinner at python.org>

files:
A Misc/NEWS.d/next/Library/2019-10-23-16-25-12.bpo-34679.Bnw8o3.rst
M Lib/asyncio/proactor_events.py
M Lib/test/test_asyncio/test_windows_events.py

diff --git a/Lib/asyncio/proactor_events.py b/Lib/asyncio/proactor_events.py
index 229f56e6bb9e2..830d8edc32f90 100644
--- a/Lib/asyncio/proactor_events.py
+++ b/Lib/asyncio/proactor_events.py
@@ -627,10 +627,9 @@ def __init__(self, proactor):
         self._accept_futures = {}   # socket file descriptor => Future
         proactor.set_loop(self)
         self._make_self_pipe()
-        self_no = self._csock.fileno()
         if threading.current_thread() is threading.main_thread():
             # wakeup fd can only be installed to a file descriptor from the main thread
-            signal.set_wakeup_fd(self_no)
+            signal.set_wakeup_fd(self._csock.fileno())
 
     def _make_socket_transport(self, sock, protocol, waiter=None,
                                extra=None, server=None):
@@ -676,7 +675,8 @@ def close(self):
         if self.is_closed():
             return
 
-        signal.set_wakeup_fd(-1)
+        if threading.current_thread() is threading.main_thread():
+            signal.set_wakeup_fd(-1)
         # Call these methods before closing the event loop (before calling
         # BaseEventLoop.close), because they can schedule callbacks with
         # call_soon(), which is forbidden when the event loop is closed.
diff --git a/Lib/test/test_asyncio/test_windows_events.py b/Lib/test/test_asyncio/test_windows_events.py
index 9ed10fc20f817..6b005702c9be7 100644
--- a/Lib/test/test_asyncio/test_windows_events.py
+++ b/Lib/test/test_asyncio/test_windows_events.py
@@ -69,6 +69,8 @@ def func():
             nonlocal finished
             loop = asyncio.new_event_loop()
             loop.run_until_complete(coro())
+            # close() must not call signal.set_wakeup_fd()
+            loop.close()
             finished = True
 
         thread = threading.Thread(target=func)
diff --git a/Misc/NEWS.d/next/Library/2019-10-23-16-25-12.bpo-34679.Bnw8o3.rst b/Misc/NEWS.d/next/Library/2019-10-23-16-25-12.bpo-34679.Bnw8o3.rst
new file mode 100644
index 0000000000000..34334db6032b5
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-10-23-16-25-12.bpo-34679.Bnw8o3.rst
@@ -0,0 +1,2 @@
+asynci.ProactorEventLoop.close() now only calls signal.set_wakeup_fd() in the
+main thread.


From webhook-mailer at python.org  Wed Oct 23 14:16:04 2019
From: webhook-mailer at python.org (Steve Dower)
Date: Wed, 23 Oct 2019 18:16:04 -0000
Subject: [Python-checkins] bpo-37025: AddRefActCtx() shouldn't be checked
 for failure (GH-16897)
Message-ID: <mailman.119.1571854565.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/009a6928727b23344613ab6a9a52b9da56ab905c
commit: 009a6928727b23344613ab6a9a52b9da56ab905c
branch: 2.7
author: Zackery Spytz <zspytz at gmail.com>
committer: Steve Dower <steve.dower at python.org>
date: 2019-10-23T11:15:55-07:00
summary:

bpo-37025: AddRefActCtx() shouldn't be checked for failure (GH-16897)

AddRefActCtx() does not return a value.

files:
A Misc/NEWS.d/next/Windows/2019-10-04-03-46-36.bpo-37025.tLheEe.rst
M PC/dl_nt.c

diff --git a/Misc/NEWS.d/next/Windows/2019-10-04-03-46-36.bpo-37025.tLheEe.rst b/Misc/NEWS.d/next/Windows/2019-10-04-03-46-36.bpo-37025.tLheEe.rst
new file mode 100644
index 0000000000000..7c0f9dc017705
--- /dev/null
+++ b/Misc/NEWS.d/next/Windows/2019-10-04-03-46-36.bpo-37025.tLheEe.rst
@@ -0,0 +1,2 @@
+``AddRefActCtx()`` was needlessly being checked for failure in
+``PC/dl_nt.c``.
diff --git a/PC/dl_nt.c b/PC/dl_nt.c
index ef1ce0934c186..3e58bacb55ff2 100644
--- a/PC/dl_nt.c
+++ b/PC/dl_nt.c
@@ -33,8 +33,8 @@ const char *PyWin_DLLVersionString = dllVersionBuffer;
 typedef BOOL (WINAPI * PFN_GETCURRENTACTCTX)(HANDLE *);
 typedef BOOL (WINAPI * PFN_ACTIVATEACTCTX)(HANDLE, ULONG_PTR *);
 typedef BOOL (WINAPI * PFN_DEACTIVATEACTCTX)(DWORD, ULONG_PTR);
-typedef BOOL (WINAPI * PFN_ADDREFACTCTX)(HANDLE);
-typedef BOOL (WINAPI * PFN_RELEASEACTCTX)(HANDLE);
+typedef void (WINAPI * PFN_ADDREFACTCTX)(HANDLE);
+typedef void (WINAPI * PFN_RELEASEACTCTX)(HANDLE);
 
 // locals and function pointers for this activation context magic.
 static HANDLE PyWin_DLLhActivationContext = NULL; // one day it might be public
@@ -90,9 +90,14 @@ BOOL    WINAPI  DllMain (HANDLE hInst,
             // and capture our activation context for use when loading extensions.
             _LoadActCtxPointers();
             if (pfnGetCurrentActCtx && pfnAddRefActCtx)
-                if ((*pfnGetCurrentActCtx)(&PyWin_DLLhActivationContext))
-                    if (!(*pfnAddRefActCtx)(PyWin_DLLhActivationContext))
-                        OutputDebugString("Python failed to load the default activation context\n");
+                if ((*pfnGetCurrentActCtx)(&PyWin_DLLhActivationContext)) {
+                    (*pfnAddRefActCtx)(PyWin_DLLhActivationContext);
+                }
+                else {
+                    OutputDebugString("Python failed to load the default "
+                                      "activation context\n");
+                    return FALSE;
+                }
             break;
 
         case DLL_PROCESS_DETACH:


From webhook-mailer at python.org  Wed Oct 23 15:34:02 2019
From: webhook-mailer at python.org (Ned Deily)
Date: Wed, 23 Oct 2019 19:34:02 -0000
Subject: [Python-checkins] [3.6] Fix Zope URL (GH-16880) (GH-16904)
Message-ID: <mailman.120.1571859243.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/293fc1730b378159194e260d946ffa53501fc95f
commit: 293fc1730b378159194e260d946ffa53501fc95f
branch: 3.6
author: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Ned Deily <nad at python.org>
date: 2019-10-23T15:33:56-04:00
summary:

[3.6] Fix Zope URL (GH-16880) (GH-16904)

(cherry picked from commit dfe726b1ace03f206f45253b93ed7610473ae20f)

Co-authored-by: Kyle Stanley <aeros167 at gmail.com>

files:
M Doc/license.rst

diff --git a/Doc/license.rst b/Doc/license.rst
index a7df1b9920448..ed8b8bc65348e 100644
--- a/Doc/license.rst
+++ b/Doc/license.rst
@@ -22,7 +22,7 @@ Virginia where he released several versions of the software.
 In May 2000, Guido and the Python core development team moved to BeOpen.com to
 form the BeOpen PythonLabs team.  In October of the same year, the PythonLabs
 team moved to Digital Creations (now Zope Corporation; see
-http://www.zope.com/).  In 2001, the Python Software Foundation (PSF, see
+https://www.zope.org/).  In 2001, the Python Software Foundation (PSF, see
 https://www.python.org/psf/) was formed, a non-profit organization created
 specifically to own Python-related Intellectual Property.  Zope Corporation is a
 sponsoring member of the PSF.


From webhook-mailer at python.org  Wed Oct 23 16:05:07 2019
From: webhook-mailer at python.org (Ned Deily)
Date: Wed, 23 Oct 2019 20:05:07 -0000
Subject: [Python-checkins] Update URL in macOS installer copy of license
 (GH-16905)
Message-ID: <mailman.121.1571861108.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/01659ca62c4508518478a74615ac91c0009427ad
commit: 01659ca62c4508518478a74615ac91c0009427ad
branch: master
author: Ned Deily <nad at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-23T16:04:59-04:00
summary:

Update URL in macOS installer copy of license (GH-16905)

files:
M Mac/BuildScript/resources/License.rtf

diff --git a/Mac/BuildScript/resources/License.rtf b/Mac/BuildScript/resources/License.rtf
index 86e54f6662a20..7566cf3b44618 100644
--- a/Mac/BuildScript/resources/License.rtf
+++ b/Mac/BuildScript/resources/License.rtf
@@ -16,7 +16,7 @@ Python was created in the early 1990s by Guido van Rossum at Stichting Mathemati
 \
 In 1995, Guido continued his work on Python at the Corporation for National Research Initiatives (CNRI, see http://www.cnri.reston.va.us) in Reston, Virginia where he released several versions of the software.\
 \
-In May 2000, Guido and the Python core development team moved to BeOpen.com to form the BeOpen PythonLabs team. In October of the same year, the PythonLabs team moved to Digital Creations (now Zope Corporation, see http://www.zope.com). In 2001, the Python Software Foundation (PSF, see http://www.python.org/psf/) was formed, a non-profit organization created specifically to own Python-related Intellectual Property. Zope Corporation is a sponsoring member of the PSF.\
+In May 2000, Guido and the Python core development team moved to BeOpen.com to form the BeOpen PythonLabs team. In October of the same year, the PythonLabs team moved to Digital Creations (now Zope Corporation, see http://www.zope.org). In 2001, the Python Software Foundation (PSF, see http://www.python.org/psf/) was formed, a non-profit organization created specifically to own Python-related Intellectual Property. Zope Corporation is a sponsoring member of the PSF.\
 \
 All Python releases are Open Source (see http://www.opensource.org for the Open Source Definition). Historically, most, but not all, Python releases have also been GPL-compatible; the table below summarizes the various releases.\
 \


From webhook-mailer at python.org  Wed Oct 23 16:16:00 2019
From: webhook-mailer at python.org (Miss Skeleton (bot))
Date: Wed, 23 Oct 2019 20:16:00 -0000
Subject: [Python-checkins] Update URL in macOS installer copy of license
 (GH-16905)
Message-ID: <mailman.122.1571861761.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/c80955cdee60c2688819a99a4c54252d77998263
commit: c80955cdee60c2688819a99a4c54252d77998263
branch: 2.7
author: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-23T13:15:56-07:00
summary:

Update URL in macOS installer copy of license (GH-16905)

(cherry picked from commit 01659ca62c4508518478a74615ac91c0009427ad)

Co-authored-by: Ned Deily <nad at python.org>

files:
M Mac/BuildScript/resources/License.rtf

diff --git a/Mac/BuildScript/resources/License.rtf b/Mac/BuildScript/resources/License.rtf
index 86e54f6662a20..7566cf3b44618 100644
--- a/Mac/BuildScript/resources/License.rtf
+++ b/Mac/BuildScript/resources/License.rtf
@@ -16,7 +16,7 @@ Python was created in the early 1990s by Guido van Rossum at Stichting Mathemati
 \
 In 1995, Guido continued his work on Python at the Corporation for National Research Initiatives (CNRI, see http://www.cnri.reston.va.us) in Reston, Virginia where he released several versions of the software.\
 \
-In May 2000, Guido and the Python core development team moved to BeOpen.com to form the BeOpen PythonLabs team. In October of the same year, the PythonLabs team moved to Digital Creations (now Zope Corporation, see http://www.zope.com). In 2001, the Python Software Foundation (PSF, see http://www.python.org/psf/) was formed, a non-profit organization created specifically to own Python-related Intellectual Property. Zope Corporation is a sponsoring member of the PSF.\
+In May 2000, Guido and the Python core development team moved to BeOpen.com to form the BeOpen PythonLabs team. In October of the same year, the PythonLabs team moved to Digital Creations (now Zope Corporation, see http://www.zope.org). In 2001, the Python Software Foundation (PSF, see http://www.python.org/psf/) was formed, a non-profit organization created specifically to own Python-related Intellectual Property. Zope Corporation is a sponsoring member of the PSF.\
 \
 All Python releases are Open Source (see http://www.opensource.org for the Open Source Definition). Historically, most, but not all, Python releases have also been GPL-compatible; the table below summarizes the various releases.\
 \


From webhook-mailer at python.org  Wed Oct 23 16:27:26 2019
From: webhook-mailer at python.org (Miss Skeleton (bot))
Date: Wed, 23 Oct 2019 20:27:26 -0000
Subject: [Python-checkins] Update URL in macOS installer copy of license
 (GH-16905)
Message-ID: <mailman.123.1571862447.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/dd81b7adce447aa1f8f34455e6fb367502b65f21
commit: dd81b7adce447aa1f8f34455e6fb367502b65f21
branch: 3.7
author: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-23T13:27:22-07:00
summary:

Update URL in macOS installer copy of license (GH-16905)

(cherry picked from commit 01659ca62c4508518478a74615ac91c0009427ad)

Co-authored-by: Ned Deily <nad at python.org>

files:
M Mac/BuildScript/resources/License.rtf

diff --git a/Mac/BuildScript/resources/License.rtf b/Mac/BuildScript/resources/License.rtf
index 86e54f6662a20..7566cf3b44618 100644
--- a/Mac/BuildScript/resources/License.rtf
+++ b/Mac/BuildScript/resources/License.rtf
@@ -16,7 +16,7 @@ Python was created in the early 1990s by Guido van Rossum at Stichting Mathemati
 \
 In 1995, Guido continued his work on Python at the Corporation for National Research Initiatives (CNRI, see http://www.cnri.reston.va.us) in Reston, Virginia where he released several versions of the software.\
 \
-In May 2000, Guido and the Python core development team moved to BeOpen.com to form the BeOpen PythonLabs team. In October of the same year, the PythonLabs team moved to Digital Creations (now Zope Corporation, see http://www.zope.com). In 2001, the Python Software Foundation (PSF, see http://www.python.org/psf/) was formed, a non-profit organization created specifically to own Python-related Intellectual Property. Zope Corporation is a sponsoring member of the PSF.\
+In May 2000, Guido and the Python core development team moved to BeOpen.com to form the BeOpen PythonLabs team. In October of the same year, the PythonLabs team moved to Digital Creations (now Zope Corporation, see http://www.zope.org). In 2001, the Python Software Foundation (PSF, see http://www.python.org/psf/) was formed, a non-profit organization created specifically to own Python-related Intellectual Property. Zope Corporation is a sponsoring member of the PSF.\
 \
 All Python releases are Open Source (see http://www.opensource.org for the Open Source Definition). Historically, most, but not all, Python releases have also been GPL-compatible; the table below summarizes the various releases.\
 \


From webhook-mailer at python.org  Wed Oct 23 16:29:59 2019
From: webhook-mailer at python.org (Ned Deily)
Date: Wed, 23 Oct 2019 20:29:59 -0000
Subject: [Python-checkins] Update URL in macOS installer copy of license
 (GH-16905) (GH-16908)
Message-ID: <mailman.124.1571862599.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/86c17c06c9420040c79c29ecf924741f37975342
commit: 86c17c06c9420040c79c29ecf924741f37975342
branch: 3.6
author: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Ned Deily <nad at python.org>
date: 2019-10-23T16:29:55-04:00
summary:

Update URL in macOS installer copy of license (GH-16905) (GH-16908)

(cherry picked from commit 01659ca62c4508518478a74615ac91c0009427ad)

Co-authored-by: Ned Deily <nad at python.org>

files:
M Mac/BuildScript/resources/License.rtf

diff --git a/Mac/BuildScript/resources/License.rtf b/Mac/BuildScript/resources/License.rtf
index 20982a46cc36a..5475d7255bd19 100644
--- a/Mac/BuildScript/resources/License.rtf
+++ b/Mac/BuildScript/resources/License.rtf
@@ -15,7 +15,7 @@ Python was created in the early 1990s by Guido van Rossum at Stichting Mathemati
 \
 In 1995, Guido continued his work on Python at the Corporation for National Research Initiatives (CNRI, see http://www.cnri.reston.va.us) in Reston, Virginia where he released several versions of the software.\
 \
-In May 2000, Guido and the Python core development team moved to BeOpen.com to form the BeOpen PythonLabs team. In October of the same year, the PythonLabs team moved to Digital Creations (now Zope Corporation, see http://www.zope.com). In 2001, the Python Software Foundation (PSF, see http://www.python.org/psf/) was formed, a non-profit organization created specifically to own Python-related Intellectual Property. Zope Corporation is a sponsoring member of the PSF.\
+In May 2000, Guido and the Python core development team moved to BeOpen.com to form the BeOpen PythonLabs team. In October of the same year, the PythonLabs team moved to Digital Creations (now Zope Corporation, see http://www.zope.org). In 2001, the Python Software Foundation (PSF, see http://www.python.org/psf/) was formed, a non-profit organization created specifically to own Python-related Intellectual Property. Zope Corporation is a sponsoring member of the PSF.\
 \
 All Python releases are Open Source (see http://www.opensource.org for the Open Source Definition). Historically, most, but not all, Python releases have also been GPL-compatible; the table below summarizes the various releases.\
 \


From webhook-mailer at python.org  Wed Oct 23 16:31:26 2019
From: webhook-mailer at python.org (Miss Skeleton (bot))
Date: Wed, 23 Oct 2019 20:31:26 -0000
Subject: [Python-checkins] Update URL in macOS installer copy of license
 (GH-16905)
Message-ID: <mailman.125.1571862687.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/229335ceef32e561cf6b5070f1b1175785702719
commit: 229335ceef32e561cf6b5070f1b1175785702719
branch: 3.8
author: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-23T13:31:22-07:00
summary:

Update URL in macOS installer copy of license (GH-16905)

(cherry picked from commit 01659ca62c4508518478a74615ac91c0009427ad)

Co-authored-by: Ned Deily <nad at python.org>

files:
M Mac/BuildScript/resources/License.rtf

diff --git a/Mac/BuildScript/resources/License.rtf b/Mac/BuildScript/resources/License.rtf
index 86e54f6662a20..7566cf3b44618 100644
--- a/Mac/BuildScript/resources/License.rtf
+++ b/Mac/BuildScript/resources/License.rtf
@@ -16,7 +16,7 @@ Python was created in the early 1990s by Guido van Rossum at Stichting Mathemati
 \
 In 1995, Guido continued his work on Python at the Corporation for National Research Initiatives (CNRI, see http://www.cnri.reston.va.us) in Reston, Virginia where he released several versions of the software.\
 \
-In May 2000, Guido and the Python core development team moved to BeOpen.com to form the BeOpen PythonLabs team. In October of the same year, the PythonLabs team moved to Digital Creations (now Zope Corporation, see http://www.zope.com). In 2001, the Python Software Foundation (PSF, see http://www.python.org/psf/) was formed, a non-profit organization created specifically to own Python-related Intellectual Property. Zope Corporation is a sponsoring member of the PSF.\
+In May 2000, Guido and the Python core development team moved to BeOpen.com to form the BeOpen PythonLabs team. In October of the same year, the PythonLabs team moved to Digital Creations (now Zope Corporation, see http://www.zope.org). In 2001, the Python Software Foundation (PSF, see http://www.python.org/psf/) was formed, a non-profit organization created specifically to own Python-related Intellectual Property. Zope Corporation is a sponsoring member of the PSF.\
 \
 All Python releases are Open Source (see http://www.opensource.org for the Open Source Definition). Historically, most, but not all, Python releases have also been GPL-compatible; the table below summarizes the various releases.\
 \


From webhook-mailer at python.org  Wed Oct 23 17:18:47 2019
From: webhook-mailer at python.org (Miss Skeleton (bot))
Date: Wed, 23 Oct 2019 21:18:47 -0000
Subject: [Python-checkins] bpo-30618: add readlink to pathlib.Path (GH-8285)
Message-ID: <mailman.126.1571865528.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/a01ba333affcc0677146dc8af57179bdb808d608
commit: a01ba333affcc0677146dc8af57179bdb808d608
branch: master
author: Girts <girtsf at users.noreply.github.com>
committer: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
date: 2019-10-23T14:18:40-07:00
summary:

bpo-30618: add readlink to pathlib.Path (GH-8285)



This adds a "readlink" method to pathlib.Path objects that calls through
to os.readlink.


https://bugs.python.org/issue30618



Automerge-Triggered-By: @gpshead

files:
A Misc/NEWS.d/next/Library/2018-07-14-13-48-56.bpo-30618.T5AUF6.rst
M Doc/library/pathlib.rst
M Doc/whatsnew/3.9.rst
M Lib/pathlib.py
M Lib/test/test_pathlib.py

diff --git a/Doc/library/pathlib.rst b/Doc/library/pathlib.rst
index acbd0e48af242..5906a33bdea11 100644
--- a/Doc/library/pathlib.rst
+++ b/Doc/library/pathlib.rst
@@ -944,6 +944,19 @@ call fails (for example because the path doesn't exist).
    .. versionadded:: 3.5
 
 
+.. method:: Path.readlink()
+
+   Return the path to which the symbolic link points (as returned by
+   :func:`os.readlink`)::
+
+      >>> p = Path('mylink')
+      >>> p.symlink_to('setup.py')
+      >>> p.readlink()
+      PosixPath('setup.py')
+
+   .. versionadded:: 3.9
+
+
 .. method:: Path.rename(target)
 
    Rename this file or directory to the given *target*, and return a new Path
@@ -1153,6 +1166,7 @@ os and os.path                         pathlib
 :func:`os.path.isdir`                  :meth:`Path.is_dir`
 :func:`os.path.isfile`                 :meth:`Path.is_file`
 :func:`os.path.islink`                 :meth:`Path.is_symlink`
+:func:`os.readlink`                    :meth:`Path.readlink`
 :func:`os.stat`                        :meth:`Path.stat`,
                                        :meth:`Path.owner`,
                                        :meth:`Path.group`
diff --git a/Doc/whatsnew/3.9.rst b/Doc/whatsnew/3.9.rst
index f203930fd1b7b..b95b0999cb952 100644
--- a/Doc/whatsnew/3.9.rst
+++ b/Doc/whatsnew/3.9.rst
@@ -150,6 +150,13 @@ customization consistently by always using the value specified by
 case), and one used ``__VENV_NAME__`` instead.
 (Contributed by Brett Cannon in :issue:`37663`.)
 
+pathlib
+-------
+
+Added :meth:`~pathlib.Path.readlink()` which acts similar to
+:func:`~os.readlink`.
+(Contributed by Girts Folkmanis in :issue:`30618`)
+
 pprint
 ------
 
diff --git a/Lib/pathlib.py b/Lib/pathlib.py
index 67b94e0154541..825533d8d5f97 100644
--- a/Lib/pathlib.py
+++ b/Lib/pathlib.py
@@ -1244,6 +1244,15 @@ def write_text(self, data, encoding=None, errors=None):
         with self.open(mode='w', encoding=encoding, errors=errors) as f:
             return f.write(data)
 
+    def readlink(self):
+        """
+        Return the path to which the symbolic link points.
+        """
+        path = self._accessor.readlink(self)
+        obj = self._from_parts((path,), init=False)
+        obj._init(template=self)
+        return obj
+
     def touch(self, mode=0o666, exist_ok=True):
         """
         Create this file with the given access mode, if it doesn't exist.
diff --git a/Lib/test/test_pathlib.py b/Lib/test/test_pathlib.py
index 221c2722d0a9d..058a201aebc1d 100644
--- a/Lib/test/test_pathlib.py
+++ b/Lib/test/test_pathlib.py
@@ -1812,6 +1812,16 @@ def test_replace(self):
         self.assertEqual(os.stat(r).st_size, size)
         self.assertFileNotFound(q.stat)
 
+    @support.skip_unless_symlink
+    def test_readlink(self):
+        P = self.cls(BASE)
+        self.assertEqual((P / 'linkA').readlink(), self.cls('fileA'))
+        self.assertEqual((P / 'brokenLink').readlink(),
+                         self.cls('non-existing'))
+        self.assertEqual((P / 'linkB').readlink(), self.cls('dirB'))
+        with self.assertRaises(OSError):
+            (P / 'fileA').readlink()
+
     def test_touch_common(self):
         P = self.cls(BASE)
         p = P / 'newfileA'
diff --git a/Misc/NEWS.d/next/Library/2018-07-14-13-48-56.bpo-30618.T5AUF6.rst b/Misc/NEWS.d/next/Library/2018-07-14-13-48-56.bpo-30618.T5AUF6.rst
new file mode 100644
index 0000000000000..35bdb4ef25647
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2018-07-14-13-48-56.bpo-30618.T5AUF6.rst
@@ -0,0 +1 @@
+Add :meth:`~pathlib.Path.readlink`. Patch by Girts Folkmanis.


From webhook-mailer at python.org  Thu Oct 24 00:15:30 2019
From: webhook-mailer at python.org (Yury Selivanov)
Date: Thu, 24 Oct 2019 04:15:30 -0000
Subject: [Python-checkins] Add missing asyncio changes from 3.8 whatsnew
 (GH-16911)
Message-ID: <mailman.127.1571890531.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/3bbb6db545eff73ba4031bd9b8f2ef71b84c906e
commit: 3bbb6db545eff73ba4031bd9b8f2ef71b84c906e
branch: master
author: Kyle Stanley <aeros167 at gmail.com>
committer: Yury Selivanov <yury at magic.io>
date: 2019-10-24T00:15:25-04:00
summary:

Add missing asyncio changes from 3.8 whatsnew (GH-16911)

files:
M Doc/whatsnew/3.8.rst

diff --git a/Doc/whatsnew/3.8.rst b/Doc/whatsnew/3.8.rst
index db40ce756287f..80ab565762c60 100644
--- a/Doc/whatsnew/3.8.rst
+++ b/Doc/whatsnew/3.8.rst
@@ -595,6 +595,40 @@ The :func:`ast.parse` function has some new flags:
 asyncio
 -------
 
+:func:`asyncio.run` has graduated from the provisional to stable API. This
+function can be used to execute a :term:`coroutine` and return the result while
+automatically managing the event loop. For example::
+
+    import asyncio
+
+    async def main():
+        await asyncio.sleep(0)
+        return 42
+
+    asyncio.run(main())
+
+This is *roughly* equivalent to::
+
+    import asyncio
+
+    async def main():
+        await asyncio.sleep(0)
+        return 42
+
+    loop = asyncio.new_event_loop()
+    asyncio.set_event_loop(loop)
+    try:
+        loop.run_until_complete(main())
+    finally:
+        asyncio.set_event_loop(None)
+        loop.close()
+
+
+The actual implementation is significantly more complex. Thus,
+:func:`asyncio.run` should be the preferred way of running asyncio programs.
+
+(Contributed by Yury Selivanov in :issue:`32314`.)
+
 Running ``python -m asyncio`` launches a natively async REPL.  This allows rapid
 experimentation with code that has a top-level :keyword:`await`.  There is no
 longer a need to directly call ``asyncio.run()`` which would spawn a new event
@@ -612,6 +646,10 @@ loop on every invocation:
 
 (Contributed by Yury Selivanov in :issue:`37028`.)
 
+The exception :class:`asyncio.CancelledError` now inherits from
+:class:`BaseException` rather than :class:`Exception`.
+(Contributed by Yury Selivanov in :issue:`32528`.)
+
 On Windows, the default event loop is now :class:`~asyncio.ProactorEventLoop`.
 (Contributed by Victor Stinner in :issue:`34687`.)
 
@@ -622,6 +660,26 @@ On Windows, the default event loop is now :class:`~asyncio.ProactorEventLoop`.
 :exc:`KeyboardInterrupt` ("CTRL+C").
 (Contributed by Vladimir Matveev in :issue:`23057`.)
 
+Added :meth:`asyncio.Task.get_coro` for getting the wrapped coroutine
+within an :class:`asyncio.Task`.
+(Contributed by Alex Gr?nholm in :issue:`36999`.)
+
+Asyncio tasks can now be named, either by passing the ``name`` keyword
+argument to :func:`asyncio.create_task` or
+the :meth:`~asyncio.loop.create_task` event loop method, or by
+calling the :meth:`~asyncio.Task.set_name` method on the task object. The
+task name is visible in the ``repr()`` output of :class:`asyncio.Task` and
+can also be retrieved using the :meth:`~asyncio.Task.get_name` method.
+(Contributed by Alex Gr?nholm in :issue:`34270`.)
+
+Added support for
+`Happy Eyeballs <https://en.wikipedia.org/wiki/Happy_Eyeballs>`_ to
+:func:`asyncio.loop.create_connection`. To specify the behavior, two new
+parameters have been added: *happy_eyeballs_delay* and *interleave*. The Happy
+Eyeballs algorithm improves responsiveness in applications that support IPv4
+and IPv6 by attempting to simultaneously connect using both.
+(Contributed by twisteroid ambassador in :issue:`33530`.)
+
 
 builtins
 --------
@@ -1575,7 +1633,7 @@ Deprecated
 
 * Passing an object that is not an instance of
   :class:`concurrent.futures.ThreadPoolExecutor` to
-  :meth:`asyncio.loop.set_default_executor()` is
+  :meth:`loop.set_default_executor() <asyncio.loop.set_default_executor>` is
   deprecated and will be prohibited in Python 3.9.
   (Contributed by Elvis Pranskevichus in :issue:`34075`.)
 
@@ -1608,6 +1666,19 @@ Deprecated
   :keyword:`async def` instead.
   (Contributed by Andrew Svetlov in :issue:`36921`.)
 
+* In :mod:`asyncio`, the explicit passing of a *loop* argument has been
+  deprecated and will be removed in version 3.10 for the following:
+  :func:`asyncio.sleep`, :func:`asyncio.gather`, :func:`asyncio.shield`,
+  :func:`asyncio.wait_for`, :func:`asyncio.wait`, :func:`asyncio.as_completed`,
+  :class:`asyncio.Task`, :class:`asyncio.Lock`, :class:`asyncio.Event`,
+  :class:`asyncio.Condition`, :class:`asyncio.Semaphore`,
+  :class:`asyncio.BoundedSemaphore`, :class:`asyncio.Queue`,
+  :func:`asyncio.create_subprocess_exec`, and
+  :func:`asyncio.create_subprocess_shell`.
+
+* The explicit passing of coroutine objects to :func:`asyncio.wait` has been
+  deprecated and will be removed in version 3.10.
+
 * The following functions and methods are deprecated in the :mod:`gettext`
   module: :func:`~gettext.lgettext`, :func:`~gettext.ldgettext`,
   :func:`~gettext.lngettext` and :func:`~gettext.ldngettext`.
@@ -1852,13 +1923,6 @@ Changes in the Python API
   you adjust (possibly including adding accessor functions to the
   public API).  (See :issue:`35886`.)
 
-* Asyncio tasks can now be named, either by passing the ``name`` keyword
-  argument to :func:`asyncio.create_task` or
-  the :meth:`~asyncio.loop.create_task` event loop method, or by
-  calling the :meth:`~asyncio.Task.set_name` method on the task object. The
-  task name is visible in the ``repr()`` output of :class:`asyncio.Task` and
-  can also be retrieved using the :meth:`~asyncio.Task.get_name` method.
-
 * The :meth:`mmap.flush() <mmap.mmap.flush>` method now returns ``None`` on
   success and raises an exception on error under all platforms.  Previously,
   its behavior was platform-dependent: a nonzero value was returned on success;
@@ -1881,8 +1945,19 @@ Changes in the Python API
   (Contributed by Anthony Sottile in :issue:`36264`.)
 
 * The exception :class:`asyncio.CancelledError` now inherits from
-  :class:`BaseException` rather than a :class:`Exception`.
-  (Contributed by Yury Selivanov in :issue:`13528`.)
+  :class:`BaseException` rather than :class:`Exception`.
+  (Contributed by Yury Selivanov in :issue:`32528`.)
+
+* The function :func:`asyncio.wait_for` now correctly waits for cancellation
+  when using an instance of :class:`asyncio.Task`. Previously, upon reaching
+  *timeout*, it was cancelled and immediately returned.
+  (Contributed by Elvis Pranskevichus in :issue:`32751`.)
+
+* The function :func:`asyncio.BaseTransport.get_extra_info` now returns a safe
+  to use socket object when 'socket' is passed to the *name* parameter.
+  (Contributed by Yury Selivanov in :issue:`37027`.)
+
+* :class:`asyncio.BufferedProtocol` has graduated to the stable API.
 
 .. _bpo-36085-whatsnew:
 


From webhook-mailer at python.org  Thu Oct 24 00:22:11 2019
From: webhook-mailer at python.org (Miss Skeleton (bot))
Date: Thu, 24 Oct 2019 04:22:11 -0000
Subject: [Python-checkins] Add missing asyncio changes from 3.8 whatsnew
 (GH-16911)
Message-ID: <mailman.128.1571890931.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/d8fc9c843e1e87460c4d3765dc9b5d8a25eb565a
commit: d8fc9c843e1e87460c4d3765dc9b5d8a25eb565a
branch: 3.8
author: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-23T21:22:06-07:00
summary:

Add missing asyncio changes from 3.8 whatsnew (GH-16911)

(cherry picked from commit 3bbb6db545eff73ba4031bd9b8f2ef71b84c906e)

Co-authored-by: Kyle Stanley <aeros167 at gmail.com>

files:
M Doc/whatsnew/3.8.rst

diff --git a/Doc/whatsnew/3.8.rst b/Doc/whatsnew/3.8.rst
index d076e3f5b453e..1d547a1fbf83a 100644
--- a/Doc/whatsnew/3.8.rst
+++ b/Doc/whatsnew/3.8.rst
@@ -597,6 +597,40 @@ The :func:`ast.parse` function has some new flags:
 asyncio
 -------
 
+:func:`asyncio.run` has graduated from the provisional to stable API. This
+function can be used to execute a :term:`coroutine` and return the result while
+automatically managing the event loop. For example::
+
+    import asyncio
+
+    async def main():
+        await asyncio.sleep(0)
+        return 42
+
+    asyncio.run(main())
+
+This is *roughly* equivalent to::
+
+    import asyncio
+
+    async def main():
+        await asyncio.sleep(0)
+        return 42
+
+    loop = asyncio.new_event_loop()
+    asyncio.set_event_loop(loop)
+    try:
+        loop.run_until_complete(main())
+    finally:
+        asyncio.set_event_loop(None)
+        loop.close()
+
+
+The actual implementation is significantly more complex. Thus,
+:func:`asyncio.run` should be the preferred way of running asyncio programs.
+
+(Contributed by Yury Selivanov in :issue:`32314`.)
+
 Running ``python -m asyncio`` launches a natively async REPL.  This allows rapid
 experimentation with code that has a top-level :keyword:`await`.  There is no
 longer a need to directly call ``asyncio.run()`` which would spawn a new event
@@ -614,6 +648,10 @@ loop on every invocation:
 
 (Contributed by Yury Selivanov in :issue:`37028`.)
 
+The exception :class:`asyncio.CancelledError` now inherits from
+:class:`BaseException` rather than :class:`Exception`.
+(Contributed by Yury Selivanov in :issue:`32528`.)
+
 On Windows, the default event loop is now :class:`~asyncio.ProactorEventLoop`.
 (Contributed by Victor Stinner in :issue:`34687`.)
 
@@ -624,6 +662,26 @@ On Windows, the default event loop is now :class:`~asyncio.ProactorEventLoop`.
 :exc:`KeyboardInterrupt` ("CTRL+C").
 (Contributed by Vladimir Matveev in :issue:`23057`.)
 
+Added :meth:`asyncio.Task.get_coro` for getting the wrapped coroutine
+within an :class:`asyncio.Task`.
+(Contributed by Alex Gr?nholm in :issue:`36999`.)
+
+Asyncio tasks can now be named, either by passing the ``name`` keyword
+argument to :func:`asyncio.create_task` or
+the :meth:`~asyncio.loop.create_task` event loop method, or by
+calling the :meth:`~asyncio.Task.set_name` method on the task object. The
+task name is visible in the ``repr()`` output of :class:`asyncio.Task` and
+can also be retrieved using the :meth:`~asyncio.Task.get_name` method.
+(Contributed by Alex Gr?nholm in :issue:`34270`.)
+
+Added support for
+`Happy Eyeballs <https://en.wikipedia.org/wiki/Happy_Eyeballs>`_ to
+:func:`asyncio.loop.create_connection`. To specify the behavior, two new
+parameters have been added: *happy_eyeballs_delay* and *interleave*. The Happy
+Eyeballs algorithm improves responsiveness in applications that support IPv4
+and IPv6 by attempting to simultaneously connect using both.
+(Contributed by twisteroid ambassador in :issue:`33530`.)
+
 
 builtins
 --------
@@ -1577,7 +1635,7 @@ Deprecated
 
 * Passing an object that is not an instance of
   :class:`concurrent.futures.ThreadPoolExecutor` to
-  :meth:`asyncio.loop.set_default_executor()` is
+  :meth:`loop.set_default_executor() <asyncio.loop.set_default_executor>` is
   deprecated and will be prohibited in Python 3.9.
   (Contributed by Elvis Pranskevichus in :issue:`34075`.)
 
@@ -1610,6 +1668,19 @@ Deprecated
   :keyword:`async def` instead.
   (Contributed by Andrew Svetlov in :issue:`36921`.)
 
+* In :mod:`asyncio`, the explicit passing of a *loop* argument has been
+  deprecated and will be removed in version 3.10 for the following:
+  :func:`asyncio.sleep`, :func:`asyncio.gather`, :func:`asyncio.shield`,
+  :func:`asyncio.wait_for`, :func:`asyncio.wait`, :func:`asyncio.as_completed`,
+  :class:`asyncio.Task`, :class:`asyncio.Lock`, :class:`asyncio.Event`,
+  :class:`asyncio.Condition`, :class:`asyncio.Semaphore`,
+  :class:`asyncio.BoundedSemaphore`, :class:`asyncio.Queue`,
+  :func:`asyncio.create_subprocess_exec`, and
+  :func:`asyncio.create_subprocess_shell`.
+
+* The explicit passing of coroutine objects to :func:`asyncio.wait` has been
+  deprecated and will be removed in version 3.10.
+
 * The following functions and methods are deprecated in the :mod:`gettext`
   module: :func:`~gettext.lgettext`, :func:`~gettext.ldgettext`,
   :func:`~gettext.lngettext` and :func:`~gettext.ldngettext`.
@@ -1854,13 +1925,6 @@ Changes in the Python API
   you adjust (possibly including adding accessor functions to the
   public API).  (See :issue:`35886`.)
 
-* Asyncio tasks can now be named, either by passing the ``name`` keyword
-  argument to :func:`asyncio.create_task` or
-  the :meth:`~asyncio.loop.create_task` event loop method, or by
-  calling the :meth:`~asyncio.Task.set_name` method on the task object. The
-  task name is visible in the ``repr()`` output of :class:`asyncio.Task` and
-  can also be retrieved using the :meth:`~asyncio.Task.get_name` method.
-
 * The :meth:`mmap.flush() <mmap.mmap.flush>` method now returns ``None`` on
   success and raises an exception on error under all platforms.  Previously,
   its behavior was platform-dependent: a nonzero value was returned on success;
@@ -1883,8 +1947,19 @@ Changes in the Python API
   (Contributed by Anthony Sottile in :issue:`36264`.)
 
 * The exception :class:`asyncio.CancelledError` now inherits from
-  :class:`BaseException` rather than a :class:`Exception`.
-  (Contributed by Yury Selivanov in :issue:`13528`.)
+  :class:`BaseException` rather than :class:`Exception`.
+  (Contributed by Yury Selivanov in :issue:`32528`.)
+
+* The function :func:`asyncio.wait_for` now correctly waits for cancellation
+  when using an instance of :class:`asyncio.Task`. Previously, upon reaching
+  *timeout*, it was cancelled and immediately returned.
+  (Contributed by Elvis Pranskevichus in :issue:`32751`.)
+
+* The function :func:`asyncio.BaseTransport.get_extra_info` now returns a safe
+  to use socket object when 'socket' is passed to the *name* parameter.
+  (Contributed by Yury Selivanov in :issue:`37027`.)
+
+* :class:`asyncio.BufferedProtocol` has graduated to the stable API.
 
 .. _bpo-36085-whatsnew:
 


From webhook-mailer at python.org  Thu Oct 24 02:19:14 2019
From: webhook-mailer at python.org (Miss Skeleton (bot))
Date: Thu, 24 Oct 2019 06:19:14 -0000
Subject: [Python-checkins] bpo-33348: parse expressions after * and ** in
 lib2to3 (GH-6586)
Message-ID: <mailman.129.1571897955.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/96b06aefe23521b61e4e9cdd44f5d30b00c7eb95
commit: 96b06aefe23521b61e4e9cdd44f5d30b00c7eb95
branch: master
author: Zsolt Dollenstein <zsol.zsol at gmail.com>
committer: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
date: 2019-10-23T23:19:07-07:00
summary:

bpo-33348: parse expressions after * and ** in lib2to3 (GH-6586)



These are valid even in python 2.7


https://bugs.python.org/issue33348



Automerge-Triggered-By: @gpshead

files:
A Misc/NEWS.d/next/Library/2018-04-24-13-18-48.bpo-33348.XaJDei.rst
M Lib/lib2to3/Grammar.txt
M Lib/lib2to3/fixes/fix_apply.py
M Lib/lib2to3/fixes/fix_intern.py
M Lib/lib2to3/fixes/fix_reload.py
M Lib/lib2to3/tests/test_parser.py

diff --git a/Lib/lib2to3/Grammar.txt b/Lib/lib2to3/Grammar.txt
index a7ddad3cf322c..68b73868b5828 100644
--- a/Lib/lib2to3/Grammar.txt
+++ b/Lib/lib2to3/Grammar.txt
@@ -138,8 +138,8 @@ arglist: argument (',' argument)* [',']
 # that precede iterable unpackings are blocked; etc.
 argument: ( test [comp_for] |
             test '=' test |
-	    '**' expr |
-	    star_expr )
+            '**' test |
+	        '*' test )
 
 comp_iter: comp_for | comp_if
 comp_for: [ASYNC] 'for' exprlist 'in' testlist_safe [comp_iter]
diff --git a/Lib/lib2to3/fixes/fix_apply.py b/Lib/lib2to3/fixes/fix_apply.py
index 826ec8c9b62ab..6408582c42647 100644
--- a/Lib/lib2to3/fixes/fix_apply.py
+++ b/Lib/lib2to3/fixes/fix_apply.py
@@ -37,10 +37,8 @@ def transform(self, node, results):
         # I feel like we should be able to express this logic in the
         # PATTERN above but I don't know how to do it so...
         if args:
-            if args.type == self.syms.star_expr:
-                return  # Make no change.
             if (args.type == self.syms.argument and
-                args.children[0].value == '**'):
+                args.children[0].value in {'**', '*'}):
                 return  # Make no change.
         if kwds and (kwds.type == self.syms.argument and
                      kwds.children[0].value == '**'):
diff --git a/Lib/lib2to3/fixes/fix_intern.py b/Lib/lib2to3/fixes/fix_intern.py
index a852330908bd9..d752843092aac 100644
--- a/Lib/lib2to3/fixes/fix_intern.py
+++ b/Lib/lib2to3/fixes/fix_intern.py
@@ -30,10 +30,8 @@ def transform(self, node, results):
             # PATTERN above but I don't know how to do it so...
             obj = results['obj']
             if obj:
-                if obj.type == self.syms.star_expr:
-                    return  # Make no change.
                 if (obj.type == self.syms.argument and
-                    obj.children[0].value == '**'):
+                    obj.children[0].value in {'**', '*'}):
                     return  # Make no change.
         names = ('sys', 'intern')
         new = ImportAndCall(node, results, names)
diff --git a/Lib/lib2to3/fixes/fix_reload.py b/Lib/lib2to3/fixes/fix_reload.py
index 6c7fbbd3be3fb..b30841131c51f 100644
--- a/Lib/lib2to3/fixes/fix_reload.py
+++ b/Lib/lib2to3/fixes/fix_reload.py
@@ -27,10 +27,8 @@ def transform(self, node, results):
             # PATTERN above but I don't know how to do it so...
             obj = results['obj']
             if obj:
-                if obj.type == self.syms.star_expr:
-                    return  # Make no change.
                 if (obj.type == self.syms.argument and
-                    obj.children[0].value == '**'):
+                    obj.children[0].value in {'**', '*'}):
                     return  # Make no change.
         names = ('importlib', 'reload')
         new = ImportAndCall(node, results, names)
diff --git a/Lib/lib2to3/tests/test_parser.py b/Lib/lib2to3/tests/test_parser.py
index f22d399b3403f..a0c31e8f5300d 100644
--- a/Lib/lib2to3/tests/test_parser.py
+++ b/Lib/lib2to3/tests/test_parser.py
@@ -253,6 +253,13 @@ def test_double_star_dict_literal(self):
     def test_double_star_dict_literal_after_keywords(self):
         self.validate("""func(spam='fried', **{'eggs':'scrambled'})""")
 
+    def test_double_star_expression(self):
+        self.validate("""func(**{'a':2} or {})""")
+        self.validate("""func(**() or {})""")
+
+    def test_star_expression(self):
+        self.validate("""func(*[] or [2])""")
+
     def test_list_display(self):
         self.validate("""[*{2}, 3, *[4]]""")
 
diff --git a/Misc/NEWS.d/next/Library/2018-04-24-13-18-48.bpo-33348.XaJDei.rst b/Misc/NEWS.d/next/Library/2018-04-24-13-18-48.bpo-33348.XaJDei.rst
new file mode 100644
index 0000000000000..f95a73fb57ed4
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2018-04-24-13-18-48.bpo-33348.XaJDei.rst
@@ -0,0 +1,2 @@
+lib2to3 now recognizes expressions after ``*`` and `**` like in ``f(*[] or
+[])``.


From webhook-mailer at python.org  Thu Oct 24 02:36:32 2019
From: webhook-mailer at python.org (Miss Skeleton (bot))
Date: Thu, 24 Oct 2019 06:36:32 -0000
Subject: [Python-checkins] bpo-33348: parse expressions after * and ** in
 lib2to3 (GH-6586)
Message-ID: <mailman.130.1571898994.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/0160a70acbd9c530802effdf320ab08c7132f4bf
commit: 0160a70acbd9c530802effdf320ab08c7132f4bf
branch: 3.7
author: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-23T23:36:26-07:00
summary:

bpo-33348: parse expressions after * and ** in lib2to3 (GH-6586)


These are valid even in python 2.7

https://bugs.python.org/issue33348

Automerge-Triggered-By: @gpshead
(cherry picked from commit 96b06aefe23521b61e4e9cdd44f5d30b00c7eb95)

Co-authored-by: Zsolt Dollenstein <zsol.zsol at gmail.com>

files:
A Misc/NEWS.d/next/Library/2018-04-24-13-18-48.bpo-33348.XaJDei.rst
M Lib/lib2to3/Grammar.txt
M Lib/lib2to3/fixes/fix_apply.py
M Lib/lib2to3/fixes/fix_intern.py
M Lib/lib2to3/fixes/fix_reload.py
M Lib/lib2to3/tests/test_parser.py

diff --git a/Lib/lib2to3/Grammar.txt b/Lib/lib2to3/Grammar.txt
index a7ddad3cf322c..68b73868b5828 100644
--- a/Lib/lib2to3/Grammar.txt
+++ b/Lib/lib2to3/Grammar.txt
@@ -138,8 +138,8 @@ arglist: argument (',' argument)* [',']
 # that precede iterable unpackings are blocked; etc.
 argument: ( test [comp_for] |
             test '=' test |
-	    '**' expr |
-	    star_expr )
+            '**' test |
+	        '*' test )
 
 comp_iter: comp_for | comp_if
 comp_for: [ASYNC] 'for' exprlist 'in' testlist_safe [comp_iter]
diff --git a/Lib/lib2to3/fixes/fix_apply.py b/Lib/lib2to3/fixes/fix_apply.py
index 826ec8c9b62ab..6408582c42647 100644
--- a/Lib/lib2to3/fixes/fix_apply.py
+++ b/Lib/lib2to3/fixes/fix_apply.py
@@ -37,10 +37,8 @@ def transform(self, node, results):
         # I feel like we should be able to express this logic in the
         # PATTERN above but I don't know how to do it so...
         if args:
-            if args.type == self.syms.star_expr:
-                return  # Make no change.
             if (args.type == self.syms.argument and
-                args.children[0].value == '**'):
+                args.children[0].value in {'**', '*'}):
                 return  # Make no change.
         if kwds and (kwds.type == self.syms.argument and
                      kwds.children[0].value == '**'):
diff --git a/Lib/lib2to3/fixes/fix_intern.py b/Lib/lib2to3/fixes/fix_intern.py
index a852330908bd9..d752843092aac 100644
--- a/Lib/lib2to3/fixes/fix_intern.py
+++ b/Lib/lib2to3/fixes/fix_intern.py
@@ -30,10 +30,8 @@ def transform(self, node, results):
             # PATTERN above but I don't know how to do it so...
             obj = results['obj']
             if obj:
-                if obj.type == self.syms.star_expr:
-                    return  # Make no change.
                 if (obj.type == self.syms.argument and
-                    obj.children[0].value == '**'):
+                    obj.children[0].value in {'**', '*'}):
                     return  # Make no change.
         names = ('sys', 'intern')
         new = ImportAndCall(node, results, names)
diff --git a/Lib/lib2to3/fixes/fix_reload.py b/Lib/lib2to3/fixes/fix_reload.py
index 6c7fbbd3be3fb..b30841131c51f 100644
--- a/Lib/lib2to3/fixes/fix_reload.py
+++ b/Lib/lib2to3/fixes/fix_reload.py
@@ -27,10 +27,8 @@ def transform(self, node, results):
             # PATTERN above but I don't know how to do it so...
             obj = results['obj']
             if obj:
-                if obj.type == self.syms.star_expr:
-                    return  # Make no change.
                 if (obj.type == self.syms.argument and
-                    obj.children[0].value == '**'):
+                    obj.children[0].value in {'**', '*'}):
                     return  # Make no change.
         names = ('importlib', 'reload')
         new = ImportAndCall(node, results, names)
diff --git a/Lib/lib2to3/tests/test_parser.py b/Lib/lib2to3/tests/test_parser.py
index 829e5a72924aa..7ec881e01a9f8 100644
--- a/Lib/lib2to3/tests/test_parser.py
+++ b/Lib/lib2to3/tests/test_parser.py
@@ -253,6 +253,13 @@ def test_double_star_dict_literal(self):
     def test_double_star_dict_literal_after_keywords(self):
         self.validate("""func(spam='fried', **{'eggs':'scrambled'})""")
 
+    def test_double_star_expression(self):
+        self.validate("""func(**{'a':2} or {})""")
+        self.validate("""func(**() or {})""")
+
+    def test_star_expression(self):
+        self.validate("""func(*[] or [2])""")
+
     def test_list_display(self):
         self.validate("""[*{2}, 3, *[4]]""")
 
diff --git a/Misc/NEWS.d/next/Library/2018-04-24-13-18-48.bpo-33348.XaJDei.rst b/Misc/NEWS.d/next/Library/2018-04-24-13-18-48.bpo-33348.XaJDei.rst
new file mode 100644
index 0000000000000..f95a73fb57ed4
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2018-04-24-13-18-48.bpo-33348.XaJDei.rst
@@ -0,0 +1,2 @@
+lib2to3 now recognizes expressions after ``*`` and `**` like in ``f(*[] or
+[])``.


From webhook-mailer at python.org  Thu Oct 24 02:37:26 2019
From: webhook-mailer at python.org (Miss Skeleton (bot))
Date: Thu, 24 Oct 2019 06:37:26 -0000
Subject: [Python-checkins] bpo-33348: parse expressions after * and ** in
 lib2to3 (GH-6586)
Message-ID: <mailman.131.1571899047.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/d04661f9ca4a082a2dd9de52df687600d574407e
commit: d04661f9ca4a082a2dd9de52df687600d574407e
branch: 3.8
author: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-23T23:37:21-07:00
summary:

bpo-33348: parse expressions after * and ** in lib2to3 (GH-6586)


These are valid even in python 2.7

https://bugs.python.org/issue33348

Automerge-Triggered-By: @gpshead
(cherry picked from commit 96b06aefe23521b61e4e9cdd44f5d30b00c7eb95)

Co-authored-by: Zsolt Dollenstein <zsol.zsol at gmail.com>

files:
A Misc/NEWS.d/next/Library/2018-04-24-13-18-48.bpo-33348.XaJDei.rst
M Lib/lib2to3/Grammar.txt
M Lib/lib2to3/fixes/fix_apply.py
M Lib/lib2to3/fixes/fix_intern.py
M Lib/lib2to3/fixes/fix_reload.py
M Lib/lib2to3/tests/test_parser.py

diff --git a/Lib/lib2to3/Grammar.txt b/Lib/lib2to3/Grammar.txt
index a7ddad3cf322c..68b73868b5828 100644
--- a/Lib/lib2to3/Grammar.txt
+++ b/Lib/lib2to3/Grammar.txt
@@ -138,8 +138,8 @@ arglist: argument (',' argument)* [',']
 # that precede iterable unpackings are blocked; etc.
 argument: ( test [comp_for] |
             test '=' test |
-	    '**' expr |
-	    star_expr )
+            '**' test |
+	        '*' test )
 
 comp_iter: comp_for | comp_if
 comp_for: [ASYNC] 'for' exprlist 'in' testlist_safe [comp_iter]
diff --git a/Lib/lib2to3/fixes/fix_apply.py b/Lib/lib2to3/fixes/fix_apply.py
index 826ec8c9b62ab..6408582c42647 100644
--- a/Lib/lib2to3/fixes/fix_apply.py
+++ b/Lib/lib2to3/fixes/fix_apply.py
@@ -37,10 +37,8 @@ def transform(self, node, results):
         # I feel like we should be able to express this logic in the
         # PATTERN above but I don't know how to do it so...
         if args:
-            if args.type == self.syms.star_expr:
-                return  # Make no change.
             if (args.type == self.syms.argument and
-                args.children[0].value == '**'):
+                args.children[0].value in {'**', '*'}):
                 return  # Make no change.
         if kwds and (kwds.type == self.syms.argument and
                      kwds.children[0].value == '**'):
diff --git a/Lib/lib2to3/fixes/fix_intern.py b/Lib/lib2to3/fixes/fix_intern.py
index a852330908bd9..d752843092aac 100644
--- a/Lib/lib2to3/fixes/fix_intern.py
+++ b/Lib/lib2to3/fixes/fix_intern.py
@@ -30,10 +30,8 @@ def transform(self, node, results):
             # PATTERN above but I don't know how to do it so...
             obj = results['obj']
             if obj:
-                if obj.type == self.syms.star_expr:
-                    return  # Make no change.
                 if (obj.type == self.syms.argument and
-                    obj.children[0].value == '**'):
+                    obj.children[0].value in {'**', '*'}):
                     return  # Make no change.
         names = ('sys', 'intern')
         new = ImportAndCall(node, results, names)
diff --git a/Lib/lib2to3/fixes/fix_reload.py b/Lib/lib2to3/fixes/fix_reload.py
index 6c7fbbd3be3fb..b30841131c51f 100644
--- a/Lib/lib2to3/fixes/fix_reload.py
+++ b/Lib/lib2to3/fixes/fix_reload.py
@@ -27,10 +27,8 @@ def transform(self, node, results):
             # PATTERN above but I don't know how to do it so...
             obj = results['obj']
             if obj:
-                if obj.type == self.syms.star_expr:
-                    return  # Make no change.
                 if (obj.type == self.syms.argument and
-                    obj.children[0].value == '**'):
+                    obj.children[0].value in {'**', '*'}):
                     return  # Make no change.
         names = ('importlib', 'reload')
         new = ImportAndCall(node, results, names)
diff --git a/Lib/lib2to3/tests/test_parser.py b/Lib/lib2to3/tests/test_parser.py
index 01b2b51e4ab6c..868ada71cda87 100644
--- a/Lib/lib2to3/tests/test_parser.py
+++ b/Lib/lib2to3/tests/test_parser.py
@@ -253,6 +253,13 @@ def test_double_star_dict_literal(self):
     def test_double_star_dict_literal_after_keywords(self):
         self.validate("""func(spam='fried', **{'eggs':'scrambled'})""")
 
+    def test_double_star_expression(self):
+        self.validate("""func(**{'a':2} or {})""")
+        self.validate("""func(**() or {})""")
+
+    def test_star_expression(self):
+        self.validate("""func(*[] or [2])""")
+
     def test_list_display(self):
         self.validate("""[*{2}, 3, *[4]]""")
 
diff --git a/Misc/NEWS.d/next/Library/2018-04-24-13-18-48.bpo-33348.XaJDei.rst b/Misc/NEWS.d/next/Library/2018-04-24-13-18-48.bpo-33348.XaJDei.rst
new file mode 100644
index 0000000000000..f95a73fb57ed4
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2018-04-24-13-18-48.bpo-33348.XaJDei.rst
@@ -0,0 +1,2 @@
+lib2to3 now recognizes expressions after ``*`` and `**` like in ``f(*[] or
+[])``.


From webhook-mailer at python.org  Fri Oct 25 12:44:12 2019
From: webhook-mailer at python.org (Miss Skeleton (bot))
Date: Fri, 25 Oct 2019 16:44:12 -0000
Subject: [Python-checkins] Fix typo in formatter_unicode (GH-16831)
Message-ID: <mailman.132.1572021853.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/7320ec05f72fc27b25789fe76f8297644e7e7e0a
commit: 7320ec05f72fc27b25789fe76f8297644e7e7e0a
branch: master
author: Hansraj Das <raj.das.136 at gmail.com>
committer: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
date: 2019-10-25T09:44:02-07:00
summary:

Fix typo in formatter_unicode (GH-16831)



numbers's -> number's

files:
M Python/formatter_unicode.c

diff --git a/Python/formatter_unicode.c b/Python/formatter_unicode.c
index 3e9e9ba08602c..7c4ecf0b3e2cb 100644
--- a/Python/formatter_unicode.c
+++ b/Python/formatter_unicode.c
@@ -590,7 +590,7 @@ calc_number_widths(NumberFieldWidths *spec, Py_ssize_t n_prefix,
         spec->n_remainder + spec->n_rpadding;
 }
 
-/* Fill in the digit parts of a numbers's string representation,
+/* Fill in the digit parts of a number's string representation,
    as determined in calc_number_widths().
    Return -1 on error, or 0 on success. */
 static int


From webhook-mailer at python.org  Fri Oct 25 13:02:51 2019
From: webhook-mailer at python.org (Miss Skeleton (bot))
Date: Fri, 25 Oct 2019 17:02:51 -0000
Subject: [Python-checkins] Fix typo in formatter_unicode (GH-16831)
Message-ID: <mailman.133.1572022971.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/5207cc0ef0843c18cf3aab6fa918c9492ae4c1de
commit: 5207cc0ef0843c18cf3aab6fa918c9492ae4c1de
branch: 3.7
author: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-25T10:02:47-07:00
summary:

Fix typo in formatter_unicode (GH-16831)


numbers's -> number's
(cherry picked from commit 7320ec05f72fc27b25789fe76f8297644e7e7e0a)

Co-authored-by: Hansraj Das <raj.das.136 at gmail.com>

files:
M Python/formatter_unicode.c

diff --git a/Python/formatter_unicode.c b/Python/formatter_unicode.c
index ef81d15bc568b..e0de8cb9cdb27 100644
--- a/Python/formatter_unicode.c
+++ b/Python/formatter_unicode.c
@@ -589,7 +589,7 @@ calc_number_widths(NumberFieldWidths *spec, Py_ssize_t n_prefix,
         spec->n_remainder + spec->n_rpadding;
 }
 
-/* Fill in the digit parts of a numbers's string representation,
+/* Fill in the digit parts of a number's string representation,
    as determined in calc_number_widths().
    Return -1 on error, or 0 on success. */
 static int


From webhook-mailer at python.org  Fri Oct 25 13:03:27 2019
From: webhook-mailer at python.org (Miss Skeleton (bot))
Date: Fri, 25 Oct 2019 17:03:27 -0000
Subject: [Python-checkins] Fix typo in formatter_unicode (GH-16831)
Message-ID: <mailman.134.1572023008.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/edf9e8e98a487229800da832292703a9b3855139
commit: edf9e8e98a487229800da832292703a9b3855139
branch: 3.8
author: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-25T10:03:23-07:00
summary:

Fix typo in formatter_unicode (GH-16831)


numbers's -> number's
(cherry picked from commit 7320ec05f72fc27b25789fe76f8297644e7e7e0a)

Co-authored-by: Hansraj Das <raj.das.136 at gmail.com>

files:
M Python/formatter_unicode.c

diff --git a/Python/formatter_unicode.c b/Python/formatter_unicode.c
index 3e9e9ba08602c..7c4ecf0b3e2cb 100644
--- a/Python/formatter_unicode.c
+++ b/Python/formatter_unicode.c
@@ -590,7 +590,7 @@ calc_number_widths(NumberFieldWidths *spec, Py_ssize_t n_prefix,
         spec->n_remainder + spec->n_rpadding;
 }
 
-/* Fill in the digit parts of a numbers's string representation,
+/* Fill in the digit parts of a number's string representation,
    as determined in calc_number_widths().
    Return -1 on error, or 0 on success. */
 static int


From webhook-mailer at python.org  Fri Oct 25 18:20:11 2019
From: webhook-mailer at python.org (Zachary Ware)
Date: Fri, 25 Oct 2019 22:20:11 -0000
Subject: [Python-checkins] bpo-38558: Mention `:=` in conditions tutorial
 (GH-16919)
Message-ID: <mailman.135.1572042012.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/cb2cf06b0aad1851f54999497c1b50c381d1fdd8
commit: cb2cf06b0aad1851f54999497c1b50c381d1fdd8
branch: master
author: Ammar Askar <ammar at ammaraskar.com>
committer: Zachary Ware <zach at python.org>
date: 2019-10-25T17:20:05-05:00
summary:

bpo-38558: Mention `:=` in conditions tutorial (GH-16919)

files:
M Doc/tutorial/datastructures.rst

diff --git a/Doc/tutorial/datastructures.rst b/Doc/tutorial/datastructures.rst
index a0d56274fd7e8..2f7afb088f3bb 100644
--- a/Doc/tutorial/datastructures.rst
+++ b/Doc/tutorial/datastructures.rst
@@ -675,10 +675,10 @@ to a variable.  For example, ::
    >>> non_null
    'Trondheim'
 
-Note that in Python, unlike C, assignment cannot occur inside expressions. C
-programmers may grumble about this, but it avoids a common class of problems
-encountered in C programs: typing ``=`` in an expression when ``==`` was
-intended.
+Note that in Python, unlike C, assignment inside expressions must be done
+explicitly with the walrus operator ``:=``. This avoids a common class of
+problems encountered in C programs: typing ``=`` in an expression when ``==``
+was intended.
 
 
 .. _tut-comparing:


From webhook-mailer at python.org  Fri Oct 25 18:27:11 2019
From: webhook-mailer at python.org (Miss Skeleton (bot))
Date: Fri, 25 Oct 2019 22:27:11 -0000
Subject: [Python-checkins] bpo-38558: Mention `:=` in conditions tutorial
 (GH-16919)
Message-ID: <mailman.136.1572042431.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/cf028b57169499f82b73dd0977fe0bab40778a60
commit: cf028b57169499f82b73dd0977fe0bab40778a60
branch: 3.8
author: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-25T15:27:07-07:00
summary:

bpo-38558: Mention `:=` in conditions tutorial (GH-16919)

(cherry picked from commit cb2cf06b0aad1851f54999497c1b50c381d1fdd8)

Co-authored-by: Ammar Askar <ammar at ammaraskar.com>

files:
M Doc/tutorial/datastructures.rst

diff --git a/Doc/tutorial/datastructures.rst b/Doc/tutorial/datastructures.rst
index a0d56274fd7e8..2f7afb088f3bb 100644
--- a/Doc/tutorial/datastructures.rst
+++ b/Doc/tutorial/datastructures.rst
@@ -675,10 +675,10 @@ to a variable.  For example, ::
    >>> non_null
    'Trondheim'
 
-Note that in Python, unlike C, assignment cannot occur inside expressions. C
-programmers may grumble about this, but it avoids a common class of problems
-encountered in C programs: typing ``=`` in an expression when ``==`` was
-intended.
+Note that in Python, unlike C, assignment inside expressions must be done
+explicitly with the walrus operator ``:=``. This avoids a common class of
+problems encountered in C programs: typing ``=`` in an expression when ``==``
+was intended.
 
 
 .. _tut-comparing:


From webhook-mailer at python.org  Sat Oct 26 09:46:10 2019
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Sat, 26 Oct 2019 13:46:10 -0000
Subject: [Python-checkins] bpo-38535: Fix positions for AST nodes for calls
 without arguments in decorators. (GH-16861)
Message-ID: <mailman.137.1572097571.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/26ae9f6d3d755734c9f371b9356325afe5764813
commit: 26ae9f6d3d755734c9f371b9356325afe5764813
branch: master
author: Serhiy Storchaka <storchaka at gmail.com>
committer: GitHub <noreply at github.com>
date: 2019-10-26T16:46:05+03:00
summary:

bpo-38535: Fix positions for AST nodes for calls without arguments in decorators. (GH-16861)

files:
A Misc/NEWS.d/next/Core and Builtins/2019-10-20-12-43-48.bpo-38535.ESMkVN.rst
M Lib/test/test_ast.py
M Python/ast.c

diff --git a/Lib/test/test_ast.py b/Lib/test/test_ast.py
index 47e259eb2656c..ca75a4ab4570e 100644
--- a/Lib/test/test_ast.py
+++ b/Lib/test/test_ast.py
@@ -129,11 +129,11 @@ def to_tuple(t):
     # Asynchronous comprehensions
     "async def f():\n [i async for b in c]",
     # Decorated FunctionDef
-    "@deco1\n at deco2()\ndef f(): pass",
+    "@deco1\n at deco2()\n at deco3(1)\ndef f(): pass",
     # Decorated AsyncFunctionDef
-    "@deco1\n at deco2()\nasync def f(): pass",
+    "@deco1\n at deco2()\n at deco3(1)\nasync def f(): pass",
     # Decorated ClassDef
-    "@deco1\n at deco2()\nclass C: pass",
+    "@deco1\n at deco2()\n at deco3(1)\nclass C: pass",
     # Decorator with generator argument
     "@deco(a for a in b)\ndef f(): pass",
     # Simple assignment expression
@@ -1870,9 +1870,9 @@ def main():
 ('Module', [('Expr', (1, 0), ('Dict', (1, 0), [None, ('Constant', (1, 10), 2, None)], [('Dict', (1, 3), [('Constant', (1, 4), 1, None)], [('Constant', (1, 6), 2, None)]), ('Constant', (1, 12), 3, None)]))], []),
 ('Module', [('Expr', (1, 0), ('Set', (1, 0), [('Starred', (1, 1), ('Set', (1, 2), [('Constant', (1, 3), 1, None), ('Constant', (1, 6), 2, None)]), ('Load',)), ('Constant', (1, 10), 3, None)]))], []),
 ('Module', [('AsyncFunctionDef', (1, 0), 'f', ('arguments', [], [], None, [], [], None, []), [('Expr', (2, 1), ('ListComp', (2, 1), ('Name', (2, 2), 'i', ('Load',)), [('comprehension', ('Name', (2, 14), 'b', ('Store',)), ('Name', (2, 19), 'c', ('Load',)), [], 1)]))], [], None, None)], []),
-('Module', [('FunctionDef', (3, 0), 'f', ('arguments', [], [], None, [], [], None, []), [('Pass', (3, 9))], [('Name', (1, 1), 'deco1', ('Load',)), ('Call', (2, 0), ('Name', (2, 1), 'deco2', ('Load',)), [], [])], None, None)], []),
-('Module', [('AsyncFunctionDef', (3, 0), 'f', ('arguments', [], [], None, [], [], None, []), [('Pass', (3, 15))], [('Name', (1, 1), 'deco1', ('Load',)), ('Call', (2, 0), ('Name', (2, 1), 'deco2', ('Load',)), [], [])], None, None)], []),
-('Module', [('ClassDef', (3, 0), 'C', [], [], [('Pass', (3, 9))], [('Name', (1, 1), 'deco1', ('Load',)), ('Call', (2, 0), ('Name', (2, 1), 'deco2', ('Load',)), [], [])])], []),
+('Module', [('FunctionDef', (4, 0), 'f', ('arguments', [], [], None, [], [], None, []), [('Pass', (4, 9))], [('Name', (1, 1), 'deco1', ('Load',)), ('Call', (2, 1), ('Name', (2, 1), 'deco2', ('Load',)), [], []), ('Call', (3, 1), ('Name', (3, 1), 'deco3', ('Load',)), [('Constant', (3, 7), 1, None)], [])], None, None)], []),
+('Module', [('AsyncFunctionDef', (4, 0), 'f', ('arguments', [], [], None, [], [], None, []), [('Pass', (4, 15))], [('Name', (1, 1), 'deco1', ('Load',)), ('Call', (2, 1), ('Name', (2, 1), 'deco2', ('Load',)), [], []), ('Call', (3, 1), ('Name', (3, 1), 'deco3', ('Load',)), [('Constant', (3, 7), 1, None)], [])], None, None)], []),
+('Module', [('ClassDef', (4, 0), 'C', [], [], [('Pass', (4, 9))], [('Name', (1, 1), 'deco1', ('Load',)), ('Call', (2, 1), ('Name', (2, 1), 'deco2', ('Load',)), [], []), ('Call', (3, 1), ('Name', (3, 1), 'deco3', ('Load',)), [('Constant', (3, 7), 1, None)], [])])], []),
 ('Module', [('FunctionDef', (2, 0), 'f', ('arguments', [], [], None, [], [], None, []), [('Pass', (2, 9))], [('Call', (1, 1), ('Name', (1, 1), 'deco', ('Load',)), [('GeneratorExp', (1, 5), ('Name', (1, 6), 'a', ('Load',)), [('comprehension', ('Name', (1, 12), 'a', ('Store',)), ('Name', (1, 17), 'b', ('Load',)), [], 0)])], [])], None, None)], []),
 ('Module', [('Expr', (1, 0), ('NamedExpr', (1, 1), ('Name', (1, 1), 'a', ('Store',)), ('Constant', (1, 6), 1, None)))], []),
 ('Module', [('FunctionDef', (1, 0), 'f', ('arguments', [('arg', (1, 6), 'a', None, None)], [], None, [], [], None, []), [('Pass', (1, 14))], [], None, None)], []),
diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-10-20-12-43-48.bpo-38535.ESMkVN.rst b/Misc/NEWS.d/next/Core and Builtins/2019-10-20-12-43-48.bpo-38535.ESMkVN.rst
new file mode 100644
index 0000000000000..7671fd06474ba
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2019-10-20-12-43-48.bpo-38535.ESMkVN.rst	
@@ -0,0 +1,2 @@
+Fixed line numbers and column offsets for AST nodes for calls without
+arguments in decorators.
diff --git a/Python/ast.c b/Python/ast.c
index 05147a49fc1dc..417b347f98741 100644
--- a/Python/ast.c
+++ b/Python/ast.c
@@ -1747,8 +1747,10 @@ ast_for_decorator(struct compiling *c, const node *n)
         name_expr = NULL;
     }
     else if (NCH(n) == 5) { /* Call with no arguments */
-        d = Call(name_expr, NULL, NULL, LINENO(n),
-                 n->n_col_offset, n->n_end_lineno, n->n_end_col_offset, c->c_arena);
+        d = Call(name_expr, NULL, NULL,
+                 name_expr->lineno, name_expr->col_offset,
+                 CHILD(n, 3)->n_end_lineno, CHILD(n, 3)->n_end_col_offset,
+                 c->c_arena);
         if (!d)
             return NULL;
         name_expr = NULL;


From webhook-mailer at python.org  Sat Oct 26 10:06:44 2019
From: webhook-mailer at python.org (Miss Skeleton (bot))
Date: Sat, 26 Oct 2019 14:06:44 -0000
Subject: [Python-checkins] bpo-38535: Fix positions for AST nodes for calls
 without arguments in decorators. (GH-16861)
Message-ID: <mailman.138.1572098805.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/ba3a566328e8df49741059b24a41480e248bf6d7
commit: ba3a566328e8df49741059b24a41480e248bf6d7
branch: 3.8
author: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-26T07:06:40-07:00
summary:

bpo-38535: Fix positions for AST nodes for calls without arguments in decorators. (GH-16861)

(cherry picked from commit 26ae9f6d3d755734c9f371b9356325afe5764813)

Co-authored-by: Serhiy Storchaka <storchaka at gmail.com>

files:
A Misc/NEWS.d/next/Core and Builtins/2019-10-20-12-43-48.bpo-38535.ESMkVN.rst
M Lib/test/test_ast.py
M Python/ast.c

diff --git a/Lib/test/test_ast.py b/Lib/test/test_ast.py
index 30460e72268cb..266075094bcba 100644
--- a/Lib/test/test_ast.py
+++ b/Lib/test/test_ast.py
@@ -129,11 +129,11 @@ def to_tuple(t):
     # Asynchronous comprehensions
     "async def f():\n [i async for b in c]",
     # Decorated FunctionDef
-    "@deco1\n at deco2()\ndef f(): pass",
+    "@deco1\n at deco2()\n at deco3(1)\ndef f(): pass",
     # Decorated AsyncFunctionDef
-    "@deco1\n at deco2()\nasync def f(): pass",
+    "@deco1\n at deco2()\n at deco3(1)\nasync def f(): pass",
     # Decorated ClassDef
-    "@deco1\n at deco2()\nclass C: pass",
+    "@deco1\n at deco2()\n at deco3(1)\nclass C: pass",
     # Decorator with generator argument
     "@deco(a for a in b)\ndef f(): pass",
     # Simple assignment expression
@@ -1808,9 +1808,9 @@ def main():
 ('Module', [('Expr', (1, 0), ('Dict', (1, 0), [None, ('Constant', (1, 10), 2, None)], [('Dict', (1, 3), [('Constant', (1, 4), 1, None)], [('Constant', (1, 6), 2, None)]), ('Constant', (1, 12), 3, None)]))], []),
 ('Module', [('Expr', (1, 0), ('Set', (1, 0), [('Starred', (1, 1), ('Set', (1, 2), [('Constant', (1, 3), 1, None), ('Constant', (1, 6), 2, None)]), ('Load',)), ('Constant', (1, 10), 3, None)]))], []),
 ('Module', [('AsyncFunctionDef', (1, 0), 'f', ('arguments', [], [], None, [], [], None, []), [('Expr', (2, 1), ('ListComp', (2, 1), ('Name', (2, 2), 'i', ('Load',)), [('comprehension', ('Name', (2, 14), 'b', ('Store',)), ('Name', (2, 19), 'c', ('Load',)), [], 1)]))], [], None, None)], []),
-('Module', [('FunctionDef', (3, 0), 'f', ('arguments', [], [], None, [], [], None, []), [('Pass', (3, 9))], [('Name', (1, 1), 'deco1', ('Load',)), ('Call', (2, 0), ('Name', (2, 1), 'deco2', ('Load',)), [], [])], None, None)], []),
-('Module', [('AsyncFunctionDef', (3, 0), 'f', ('arguments', [], [], None, [], [], None, []), [('Pass', (3, 15))], [('Name', (1, 1), 'deco1', ('Load',)), ('Call', (2, 0), ('Name', (2, 1), 'deco2', ('Load',)), [], [])], None, None)], []),
-('Module', [('ClassDef', (3, 0), 'C', [], [], [('Pass', (3, 9))], [('Name', (1, 1), 'deco1', ('Load',)), ('Call', (2, 0), ('Name', (2, 1), 'deco2', ('Load',)), [], [])])], []),
+('Module', [('FunctionDef', (4, 0), 'f', ('arguments', [], [], None, [], [], None, []), [('Pass', (4, 9))], [('Name', (1, 1), 'deco1', ('Load',)), ('Call', (2, 1), ('Name', (2, 1), 'deco2', ('Load',)), [], []), ('Call', (3, 1), ('Name', (3, 1), 'deco3', ('Load',)), [('Constant', (3, 7), 1, None)], [])], None, None)], []),
+('Module', [('AsyncFunctionDef', (4, 0), 'f', ('arguments', [], [], None, [], [], None, []), [('Pass', (4, 15))], [('Name', (1, 1), 'deco1', ('Load',)), ('Call', (2, 1), ('Name', (2, 1), 'deco2', ('Load',)), [], []), ('Call', (3, 1), ('Name', (3, 1), 'deco3', ('Load',)), [('Constant', (3, 7), 1, None)], [])], None, None)], []),
+('Module', [('ClassDef', (4, 0), 'C', [], [], [('Pass', (4, 9))], [('Name', (1, 1), 'deco1', ('Load',)), ('Call', (2, 1), ('Name', (2, 1), 'deco2', ('Load',)), [], []), ('Call', (3, 1), ('Name', (3, 1), 'deco3', ('Load',)), [('Constant', (3, 7), 1, None)], [])])], []),
 ('Module', [('FunctionDef', (2, 0), 'f', ('arguments', [], [], None, [], [], None, []), [('Pass', (2, 9))], [('Call', (1, 1), ('Name', (1, 1), 'deco', ('Load',)), [('GeneratorExp', (1, 5), ('Name', (1, 6), 'a', ('Load',)), [('comprehension', ('Name', (1, 12), 'a', ('Store',)), ('Name', (1, 17), 'b', ('Load',)), [], 0)])], [])], None, None)], []),
 ('Module', [('Expr', (1, 0), ('NamedExpr', (1, 1), ('Name', (1, 1), 'a', ('Store',)), ('Constant', (1, 6), 1, None)))], []),
 ('Module', [('FunctionDef', (1, 0), 'f', ('arguments', [('arg', (1, 6), 'a', None, None)], [], None, [], [], None, []), [('Pass', (1, 14))], [], None, None)], []),
diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-10-20-12-43-48.bpo-38535.ESMkVN.rst b/Misc/NEWS.d/next/Core and Builtins/2019-10-20-12-43-48.bpo-38535.ESMkVN.rst
new file mode 100644
index 0000000000000..7671fd06474ba
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2019-10-20-12-43-48.bpo-38535.ESMkVN.rst	
@@ -0,0 +1,2 @@
+Fixed line numbers and column offsets for AST nodes for calls without
+arguments in decorators.
diff --git a/Python/ast.c b/Python/ast.c
index 79a29a6628ad3..2031b88359363 100644
--- a/Python/ast.c
+++ b/Python/ast.c
@@ -1747,8 +1747,10 @@ ast_for_decorator(struct compiling *c, const node *n)
         name_expr = NULL;
     }
     else if (NCH(n) == 5) { /* Call with no arguments */
-        d = Call(name_expr, NULL, NULL, LINENO(n),
-                 n->n_col_offset, n->n_end_lineno, n->n_end_col_offset, c->c_arena);
+        d = Call(name_expr, NULL, NULL,
+                 name_expr->lineno, name_expr->col_offset,
+                 CHILD(n, 3)->n_end_lineno, CHILD(n, 3)->n_end_col_offset,
+                 c->c_arena);
         if (!d)
             return NULL;
         name_expr = NULL;


From webhook-mailer at python.org  Sat Oct 26 10:30:18 2019
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Sat, 26 Oct 2019 14:30:18 -0000
Subject: [Python-checkins] [3.7] bpo-38535: Fix positions for AST nodes for
 calls without arguments in decorators. (GH-16861). (GH-16930)
Message-ID: <mailman.139.1572100219.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/91fc9cf43cc2e3a2f236ef9944cf1f4bed701545
commit: 91fc9cf43cc2e3a2f236ef9944cf1f4bed701545
branch: 3.7
author: Serhiy Storchaka <storchaka at gmail.com>
committer: GitHub <noreply at github.com>
date: 2019-10-26T17:30:14+03:00
summary:

[3.7] bpo-38535: Fix positions for AST nodes for calls without arguments in decorators. (GH-16861). (GH-16930)

(cherry picked from commit 26ae9f6d3d755734c9f371b9356325afe5764813)

files:
A Misc/NEWS.d/next/Core and Builtins/2019-10-20-12-43-48.bpo-38535.ESMkVN.rst
M Lib/test/test_ast.py
M Python/ast.c

diff --git a/Lib/test/test_ast.py b/Lib/test/test_ast.py
index 2d0b5b24ee7dc..2d2fb26baa9c0 100644
--- a/Lib/test/test_ast.py
+++ b/Lib/test/test_ast.py
@@ -125,6 +125,14 @@ def to_tuple(t):
     "{*{1, 2}, 3}",
     # Asynchronous comprehensions
     "async def f():\n [i async for b in c]",
+    # Decorated FunctionDef
+    "@deco1\n at deco2()\n at deco3(1)\ndef f(): pass",
+    # Decorated AsyncFunctionDef
+    "@deco1\n at deco2()\n at deco3(1)\nasync def f(): pass",
+    # Decorated ClassDef
+    "@deco1\n at deco2()\n at deco3(1)\nclass C: pass",
+    # Decorator with generator argument
+    "@deco(a for a in b)\ndef f(): pass",
 ]
 
 # These are compiled through "single"
@@ -1255,6 +1263,10 @@ def main():
 ('Module', [('Expr', (1, 0), ('Dict', (1, 0), [None, ('Num', (1, 10), 2)], [('Dict', (1, 3), [('Num', (1, 4), 1)], [('Num', (1, 6), 2)]), ('Num', (1, 12), 3)]))]),
 ('Module', [('Expr', (1, 0), ('Set', (1, 0), [('Starred', (1, 1), ('Set', (1, 2), [('Num', (1, 3), 1), ('Num', (1, 6), 2)]), ('Load',)), ('Num', (1, 10), 3)]))]),
 ('Module', [('AsyncFunctionDef', (1, 0), 'f', ('arguments', [], None, [], [], None, []), [('Expr', (2, 1), ('ListComp', (2, 2), ('Name', (2, 2), 'i', ('Load',)), [('comprehension', ('Name', (2, 14), 'b', ('Store',)), ('Name', (2, 19), 'c', ('Load',)), [], 1)]))], [], None)]),
+('Module', [('FunctionDef', (1, 0), 'f', ('arguments', [], None, [], [], None, []), [('Pass', (4, 9))], [('Name', (1, 1), 'deco1', ('Load',)), ('Call', (2, 1), ('Name', (2, 1), 'deco2', ('Load',)), [], []), ('Call', (3, 1), ('Name', (3, 1), 'deco3', ('Load',)), [('Num', (3, 7), 1)], [])], None)]),
+('Module', [('AsyncFunctionDef', (1, 0), 'f', ('arguments', [], None, [], [], None, []), [('Pass', (4, 15))], [('Name', (1, 1), 'deco1', ('Load',)), ('Call', (2, 1), ('Name', (2, 1), 'deco2', ('Load',)), [], []), ('Call', (3, 1), ('Name', (3, 1), 'deco3', ('Load',)), [('Num', (3, 7), 1)], [])], None)]),
+('Module', [('ClassDef', (1, 0), 'C', [], [], [('Pass', (4, 9))], [('Name', (1, 1), 'deco1', ('Load',)), ('Call', (2, 1), ('Name', (2, 1), 'deco2', ('Load',)), [], []), ('Call', (3, 1), ('Name', (3, 1), 'deco3', ('Load',)), [('Num', (3, 7), 1)], [])])]),
+('Module', [('FunctionDef', (1, 0), 'f', ('arguments', [], None, [], [], None, []), [('Pass', (2, 9))], [('Call', (1, 1), ('Name', (1, 1), 'deco', ('Load',)), [('GeneratorExp', (1, 6), ('Name', (1, 6), 'a', ('Load',)), [('comprehension', ('Name', (1, 12), 'a', ('Store',)), ('Name', (1, 17), 'b', ('Load',)), [], 0)])], [])], None)]),
 ]
 single_results = [
 ('Interactive', [('Expr', (1, 0), ('BinOp', (1, 0), ('Num', (1, 0), 1), ('Add',), ('Num', (1, 2), 2)))]),
diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-10-20-12-43-48.bpo-38535.ESMkVN.rst b/Misc/NEWS.d/next/Core and Builtins/2019-10-20-12-43-48.bpo-38535.ESMkVN.rst
new file mode 100644
index 0000000000000..7671fd06474ba
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2019-10-20-12-43-48.bpo-38535.ESMkVN.rst	
@@ -0,0 +1,2 @@
+Fixed line numbers and column offsets for AST nodes for calls without
+arguments in decorators.
diff --git a/Python/ast.c b/Python/ast.c
index ce61375ea951d..5a60d6994d6a6 100644
--- a/Python/ast.c
+++ b/Python/ast.c
@@ -1531,8 +1531,9 @@ ast_for_decorator(struct compiling *c, const node *n)
         name_expr = NULL;
     }
     else if (NCH(n) == 5) { /* Call with no arguments */
-        d = Call(name_expr, NULL, NULL, LINENO(n),
-                 n->n_col_offset, c->c_arena);
+        d = Call(name_expr, NULL, NULL,
+                 name_expr->lineno, name_expr->col_offset,
+                 c->c_arena);
         if (!d)
             return NULL;
         name_expr = NULL;


From webhook-mailer at python.org  Sat Oct 26 10:30:34 2019
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Sat, 26 Oct 2019 14:30:34 -0000
Subject: [Python-checkins] [2.7] bpo-38535: Fix positions for AST nodes for
 calls without arguments in decorators. (GH-16861). (GH-16931)
Message-ID: <mailman.140.1572100236.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/493fef60a7600f83fe6916ed89d0fb0c0aab484d
commit: 493fef60a7600f83fe6916ed89d0fb0c0aab484d
branch: 2.7
author: Serhiy Storchaka <storchaka at gmail.com>
committer: GitHub <noreply at github.com>
date: 2019-10-26T17:30:30+03:00
summary:

[2.7] bpo-38535: Fix positions for AST nodes for calls without arguments in decorators. (GH-16861). (GH-16931)

(cherry picked from commit 26ae9f6d3d755734c9f371b9356325afe5764813)

files:
A Misc/NEWS.d/next/Core and Builtins/2019-10-20-12-43-48.bpo-38535.ESMkVN.rst
M Lib/test/test_ast.py
M Python/ast.c

diff --git a/Lib/test/test_ast.py b/Lib/test/test_ast.py
index 0a1ca4168a590..3cfe6188ac134 100644
--- a/Lib/test/test_ast.py
+++ b/Lib/test/test_ast.py
@@ -103,6 +103,12 @@ def to_tuple(t):
     "{r for l in x if g}",
     # setcomp with naked tuple
     "{r for l,m in x}",
+    # Decorated FunctionDef
+    "@deco1\n at deco2()\n at deco3(1)\ndef f(): pass",
+    # Decorated ClassDef
+    "@deco1\n at deco2()\n at deco3(1)\nclass C: pass",
+    # Decorator with generator argument
+    "@deco(a for a in b)\ndef f(): pass",
 ]
 
 # These are compiled through "single"
@@ -546,6 +552,9 @@ def main():
 ('Module', [('Expr', (1, 0), ('DictComp', (1, 1), ('Name', (1, 1), 'a', ('Load',)), ('Name', (1, 5), 'b', ('Load',)), [('comprehension', ('Tuple', (1, 11), [('Name', (1, 11), 'v', ('Store',)), ('Name', (1, 13), 'w', ('Store',))], ('Store',)), ('Name', (1, 18), 'x', ('Load',)), [])]))]),
 ('Module', [('Expr', (1, 0), ('SetComp', (1, 1), ('Name', (1, 1), 'r', ('Load',)), [('comprehension', ('Name', (1, 7), 'l', ('Store',)), ('Name', (1, 12), 'x', ('Load',)), [('Name', (1, 17), 'g', ('Load',))])]))]),
 ('Module', [('Expr', (1, 0), ('SetComp', (1, 1), ('Name', (1, 1), 'r', ('Load',)), [('comprehension', ('Tuple', (1, 7), [('Name', (1, 7), 'l', ('Store',)), ('Name', (1, 9), 'm', ('Store',))], ('Store',)), ('Name', (1, 14), 'x', ('Load',)), [])]))]),
+('Module', [('FunctionDef', (1, 0), 'f', ('arguments', [], None, None, []), [('Pass', (4, 9))], [('Name', (1, 1), 'deco1', ('Load',)), ('Call', (2, 1), ('Name', (2, 1), 'deco2', ('Load',)), [], [], None, None), ('Call', (3, 1), ('Name', (3, 1), 'deco3', ('Load',)), [('Num', (3, 7), 1)], [], None, None)])]),
+('Module', [('ClassDef', (1, 0), 'C', [], [('Pass', (4, 9))], [('Name', (1, 1), 'deco1', ('Load',)), ('Call', (2, 1), ('Name', (2, 1), 'deco2', ('Load',)), [], [], None, None), ('Call', (3, 1), ('Name', (3, 1), 'deco3', ('Load',)), [('Num', (3, 7), 1)], [], None, None)])]),
+('Module', [('FunctionDef', (1, 0), 'f', ('arguments', [], None, None, []), [('Pass', (2, 9))], [('Call', (1, 1), ('Name', (1, 1), 'deco', ('Load',)), [('GeneratorExp', (1, 6), ('Name', (1, 6), 'a', ('Load',)), [('comprehension', ('Name', (1, 12), 'a', ('Store',)), ('Name', (1, 17), 'b', ('Load',)), [])])], [], None, None)])]),
 ]
 single_results = [
 ('Interactive', [('Expr', (1, 0), ('BinOp', (1, 0), ('Num', (1, 0), 1), ('Add',), ('Num', (1, 2), 2)))]),
diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-10-20-12-43-48.bpo-38535.ESMkVN.rst b/Misc/NEWS.d/next/Core and Builtins/2019-10-20-12-43-48.bpo-38535.ESMkVN.rst
new file mode 100644
index 0000000000000..7671fd06474ba
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2019-10-20-12-43-48.bpo-38535.ESMkVN.rst	
@@ -0,0 +1,2 @@
+Fixed line numbers and column offsets for AST nodes for calls without
+arguments in decorators.
diff --git a/Python/ast.c b/Python/ast.c
index 946032589f4b0..10571a3ec2428 100644
--- a/Python/ast.c
+++ b/Python/ast.c
@@ -852,8 +852,9 @@ ast_for_decorator(struct compiling *c, const node *n)
         name_expr = NULL;
     }
     else if (NCH(n) == 5) { /* Call with no arguments */
-        d = Call(name_expr, NULL, NULL, NULL, NULL, LINENO(n),
-                 n->n_col_offset, c->c_arena);
+        d = Call(name_expr, NULL, NULL, NULL, NULL,
+                 name_expr->lineno, name_expr->col_offset,
+                 c->c_arena);
         if (!d)
             return NULL;
         name_expr = NULL;


From webhook-mailer at python.org  Sat Oct 26 15:59:23 2019
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Sat, 26 Oct 2019 19:59:23 -0000
Subject: [Python-checkins] bpo-38557: Improve documentation for list and
 tuple C API. (GH-16925)
Message-ID: <mailman.141.1572119964.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/d898d20e8c228229eb68e545f544db13f246f216
commit: d898d20e8c228229eb68e545f544db13f246f216
branch: master
author: Serhiy Storchaka <storchaka at gmail.com>
committer: GitHub <noreply at github.com>
date: 2019-10-26T22:59:18+03:00
summary:

bpo-38557: Improve documentation for list and tuple C API. (GH-16925)

files:
M Doc/c-api/list.rst
M Doc/c-api/tuple.rst
M Doc/tools/susp-ignored.csv

diff --git a/Doc/c-api/list.rst b/Doc/c-api/list.rst
index dc9026605be35..646f29d08a04b 100644
--- a/Doc/c-api/list.rst
+++ b/Doc/c-api/list.rst
@@ -71,8 +71,9 @@ List Objects
 
 .. c:function:: int PyList_SetItem(PyObject *list, Py_ssize_t index, PyObject *item)
 
-   Set the item at index *index* in list to *item*.  Return ``0`` on success
-   or ``-1`` on failure.
+   Set the item at index *index* in list to *item*.  Return ``0`` on success.
+   If *index* is out of bounds, return ``-1`` and set an :exc:`IndexError`
+   exception.
 
    .. note::
 
@@ -111,8 +112,7 @@ List Objects
 
    Return a list of the objects in *list* containing the objects *between* *low*
    and *high*.  Return *NULL* and set an exception if unsuccessful.  Analogous
-   to ``list[low:high]``.  Negative indices, as when slicing from Python, are not
-   supported.
+   to ``list[low:high]``.  Indexing from the end of the list is not supported.
 
 
 .. c:function:: int PyList_SetSlice(PyObject *list, Py_ssize_t low, Py_ssize_t high, PyObject *itemlist)
@@ -120,8 +120,8 @@ List Objects
    Set the slice of *list* between *low* and *high* to the contents of
    *itemlist*.  Analogous to ``list[low:high] = itemlist``. The *itemlist* may
    be *NULL*, indicating the assignment of an empty list (slice deletion).
-   Return ``0`` on success, ``-1`` on failure.  Negative indices, as when
-   slicing from Python, are not supported.
+   Return ``0`` on success, ``-1`` on failure.  Indexing from the end of the
+   list is not supported.
 
 
 .. c:function:: int PyList_Sort(PyObject *list)
diff --git a/Doc/c-api/tuple.rst b/Doc/c-api/tuple.rst
index 259ec4fb48515..a0d53e2030e63 100644
--- a/Doc/c-api/tuple.rst
+++ b/Doc/c-api/tuple.rst
@@ -57,7 +57,7 @@ Tuple Objects
 .. c:function:: PyObject* PyTuple_GetItem(PyObject *p, Py_ssize_t pos)
 
    Return the object at position *pos* in the tuple pointed to by *p*.  If *pos* is
-   out of bounds, return *NULL* and sets an :exc:`IndexError` exception.
+   out of bounds, return *NULL* and set an :exc:`IndexError` exception.
 
 
 .. c:function:: PyObject* PyTuple_GET_ITEM(PyObject *p, Py_ssize_t pos)
@@ -67,18 +67,21 @@ Tuple Objects
 
 .. c:function:: PyObject* PyTuple_GetSlice(PyObject *p, Py_ssize_t low, Py_ssize_t high)
 
-   Take a slice of the tuple pointed to by *p* from *low* to *high* and return it
-   as a new tuple.
+   Return the slice of the tuple pointed to by *p* between *low* and *high*,
+   or *NULL* on failure.  This is the equivalent of the Python expression
+   ``p[low:high]``.  Indexing from the end of the list is not supported.
 
 
 .. c:function:: int PyTuple_SetItem(PyObject *p, Py_ssize_t pos, PyObject *o)
 
    Insert a reference to object *o* at position *pos* of the tuple pointed to by
-   *p*. Return ``0`` on success.
+   *p*.  Return ``0`` on success.  If *pos* is out of bounds, return ``-1``
+   and set an :exc:`IndexError` exception.
 
    .. note::
 
-      This function "steals" a reference to *o*.
+      This function "steals" a reference to *o* and discards a reference to
+      an item already in the tuple at the affected position.
 
 
 .. c:function:: void PyTuple_SET_ITEM(PyObject *p, Py_ssize_t pos, PyObject *o)
@@ -88,7 +91,10 @@ Tuple Objects
 
    .. note::
 
-      This function "steals" a reference to *o*.
+      This macro "steals" a reference to *o*, and, unlike
+      :c:func:`PyTuple_SetItem`, does *not* discard a reference to any item that
+      is being replaced; any reference in the tuple at position *pos* will be
+      leaked.
 
 
 .. c:function:: int _PyTuple_Resize(PyObject **p, Py_ssize_t newsize)
diff --git a/Doc/tools/susp-ignored.csv b/Doc/tools/susp-ignored.csv
index 740850cae847c..fe15aae796fe6 100644
--- a/Doc/tools/susp-ignored.csv
+++ b/Doc/tools/susp-ignored.csv
@@ -2,6 +2,7 @@ c-api/arg,,:ref,"PyArg_ParseTuple(args, ""O|O:ref"", &object, &callback)"
 c-api/list,,:high,list[low:high]
 c-api/sequence,,:i2,del o[i1:i2]
 c-api/sequence,,:i2,o[i1:i2]
+c-api/tuple,,:high,p[low:high]
 c-api/unicode,,:end,str[start:end]
 c-api/unicode,,:start,unicode[start:start+length]
 distutils/examples,267,`,This is the description of the ``foobar`` package.


From webhook-mailer at python.org  Sat Oct 26 16:02:34 2019
From: webhook-mailer at python.org (Steve Dower)
Date: Sat, 26 Oct 2019 20:02:34 -0000
Subject: [Python-checkins] bpo-38434: Fixes some audit event documentation
 (GH-16932)
Message-ID: <mailman.142.1572120155.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/894e30ce0bcc1c509eb01c8ffa9ba6d7701aeaaf
commit: 894e30ce0bcc1c509eb01c8ffa9ba6d7701aeaaf
branch: master
author: Steve Dower <steve.dower at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-26T13:02:30-07:00
summary:

bpo-38434: Fixes some audit event documentation (GH-16932)

files:
M Doc/c-api/code.rst
M Doc/c-api/sys.rst
M Doc/library/sys.rst
M Doc/library/types.rst
M Doc/reference/simple_stmts.rst

diff --git a/Doc/c-api/code.rst b/Doc/c-api/code.rst
index 3c4f66923da5a..45a6b4a753a72 100644
--- a/Doc/c-api/code.rst
+++ b/Doc/c-api/code.rst
@@ -40,8 +40,6 @@ bound into a function.
    can bind you to a precise Python version since the definition of the bytecode
    changes often.
 
-   .. audit-event:: code.__new__ code,filename,name,argcount,posonlyargcount,kwonlyargcount,nlocals,stacksize,flags c.PyCode_New
-
 .. c:function:: PyCodeObject* PyCode_NewWithPosOnlyArgs(int argcount, int posonlyargcount, int kwonlyargcount, int nlocals, int stacksize, int flags, PyObject *code, PyObject *consts, PyObject *names, PyObject *varnames, PyObject *freevars, PyObject *cellvars, PyObject *filename, PyObject *name, int firstlineno, PyObject *lnotab)
 
    Similar to :c:func:`PyCode_New`, but with an extra "posonlyargcount" for positonal-only arguments.
diff --git a/Doc/c-api/sys.rst b/Doc/c-api/sys.rst
index 838a97cacfdf1..87d2bd13d31b8 100644
--- a/Doc/c-api/sys.rst
+++ b/Doc/c-api/sys.rst
@@ -330,7 +330,16 @@ accessible to C code.  They all work with the current interpreter thread's
 
    See :pep:`578` for a detailed description of auditing. Functions in the
    runtime and standard library that raise events include the details in each
-   function's documentation.
+   function's documentation and listed in the :ref:`audit events table
+   <audit-events>`.
+
+   .. audit-event:: sys.addaudithook "" c.PySys_AddAuditHook
+
+      If the interpreter is initialized, this function raises a auditing event
+      ``sys.addaudithook`` with no arguments. If any existing hooks raise an
+      exception derived from :class:`Exception`, the new hook will not be
+      added and the exception is cleared. As a result, callers cannot assume
+      that their hook has been added unless they control all existing hooks.
 
    .. versionadded:: 3.8
 
diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst
index c6c643b38de42..a2e930b35d57f 100644
--- a/Doc/library/sys.rst
+++ b/Doc/library/sys.rst
@@ -33,18 +33,21 @@ always available.
    tuple of arguments. Native hooks added by :c:func:`PySys_AddAuditHook` are
    called first, followed by hooks added in the current interpreter.
 
-   Calling this function will trigger an event for all existing hooks, and if
-   any raise an exception derived from :class:`Exception`, the add will be
-   silently ignored. As a result, callers cannot assume that their hook has been
-   added unless they control all existing hooks.
+   .. audit-event:: sys.addaudithook "" sys.addaudithook
+
+      Raises a auditing event ``sys.addaudithook`` with no arguments. If any
+      existing hooks raise an exception derived from :class:`Exception`, the
+      new hook will not be added and the exception suppressed. As a result,
+      callers cannot assume that their hook has been added unless they control
+      all existing hooks.
 
    .. versionadded:: 3.8
 
    .. impl-detail::
 
-      When tracing is enabled, Python hooks are only traced if the callable has
-      a ``__cantrace__`` member that is set to a true value. Otherwise, trace
-      functions will not see the hook.
+      When tracing is enabled (see :func:`settrace`), Python hooks are only
+      traced if the callable has a ``__cantrace__`` member that is set to a
+      true value. Otherwise, trace functions will skip the hook.
 
 
 .. data:: argv
@@ -87,7 +90,7 @@ always available.
    native function is preferred when possible.
 
    See the :ref:`audit events table <audit-events>` for all events raised by
-   ``CPython``.
+   CPython.
 
    .. versionadded:: 3.8
 
diff --git a/Doc/library/types.rst b/Doc/library/types.rst
index a21fb44dda5de..9393f9e6db990 100644
--- a/Doc/library/types.rst
+++ b/Doc/library/types.rst
@@ -138,6 +138,10 @@ Standard names are defined for the following types:
 
    The type for code objects such as returned by :func:`compile`.
 
+   .. audit-event:: code.__new__ code,filename,name,argcount,posonlyargcount,kwonlyargcount,nlocals,stacksize,flags CodeType
+
+   Note that the audited arguments may not match the names or positions
+   required by the initializer.
 
 .. data:: CellType
 
diff --git a/Doc/reference/simple_stmts.rst b/Doc/reference/simple_stmts.rst
index 0a043a90050c4..a8ec0fbe8b732 100644
--- a/Doc/reference/simple_stmts.rst
+++ b/Doc/reference/simple_stmts.rst
@@ -839,6 +839,7 @@ the :ref:`relativeimports` section.
 :func:`importlib.import_module` is provided to support applications that
 determine dynamically the modules to be loaded.
 
+.. audit-event:: import module,filename,sys.path,sys.meta_path,sys.path_hooks import
 
 .. _future:
 


From webhook-mailer at python.org  Sat Oct 26 16:04:17 2019
From: webhook-mailer at python.org (Miss Skeleton (bot))
Date: Sat, 26 Oct 2019 20:04:17 -0000
Subject: [Python-checkins] bpo-38557: Improve documentation for list and
 tuple C API. (GH-16925)
Message-ID: <mailman.143.1572120258.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/7356e10820b160d14b0ce0aba5427a8f9e757aa7
commit: 7356e10820b160d14b0ce0aba5427a8f9e757aa7
branch: 2.7
author: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-26T13:04:13-07:00
summary:

bpo-38557: Improve documentation for list and tuple C API. (GH-16925)

(cherry picked from commit d898d20e8c228229eb68e545f544db13f246f216)

Co-authored-by: Serhiy Storchaka <storchaka at gmail.com>

files:
M Doc/c-api/list.rst
M Doc/c-api/tuple.rst
M Doc/tools/susp-ignored.csv

diff --git a/Doc/c-api/list.rst b/Doc/c-api/list.rst
index a5e4a45492b69..c28d6b0c93055 100644
--- a/Doc/c-api/list.rst
+++ b/Doc/c-api/list.rst
@@ -96,8 +96,9 @@ List Objects
 
 .. c:function:: int PyList_SetItem(PyObject *list, Py_ssize_t index, PyObject *item)
 
-   Set the item at index *index* in list to *item*.  Return ``0`` on success
-   or ``-1`` on failure.
+   Set the item at index *index* in list to *item*.  Return ``0`` on success.
+   If *index* is out of bounds, return ``-1`` and set an :exc:`IndexError`
+   exception.
 
    .. note::
 
@@ -148,8 +149,7 @@ List Objects
 
    Return a list of the objects in *list* containing the objects *between* *low*
    and *high*.  Return *NULL* and set an exception if unsuccessful.  Analogous
-   to ``list[low:high]``.  Negative indices, as when slicing from Python, are not
-   supported.
+   to ``list[low:high]``.  Indexing from the end of the list is not supported.
 
    .. versionchanged:: 2.5
       This function used an :c:type:`int` for *low* and *high*. This might
@@ -161,8 +161,8 @@ List Objects
    Set the slice of *list* between *low* and *high* to the contents of
    *itemlist*.  Analogous to ``list[low:high] = itemlist``. The *itemlist* may
    be *NULL*, indicating the assignment of an empty list (slice deletion).
-   Return ``0`` on success, ``-1`` on failure.  Negative indices, as when
-   slicing from Python, are not supported.
+   Return ``0`` on success, ``-1`` on failure.  Indexing from the end of the
+   list is not supported.
 
    .. versionchanged:: 2.5
       This function used an :c:type:`int` for *low* and *high*. This might
diff --git a/Doc/c-api/tuple.rst b/Doc/c-api/tuple.rst
index 16de45fb31c43..fd85663a21cff 100644
--- a/Doc/c-api/tuple.rst
+++ b/Doc/c-api/tuple.rst
@@ -82,7 +82,7 @@ Tuple Objects
 .. c:function:: PyObject* PyTuple_GetItem(PyObject *p, Py_ssize_t pos)
 
    Return the object at position *pos* in the tuple pointed to by *p*.  If *pos* is
-   out of bounds, return *NULL* and sets an :exc:`IndexError` exception.
+   out of bounds, return *NULL* and set an :exc:`IndexError` exception.
 
    .. versionchanged:: 2.5
       This function used an :c:type:`int` type for *pos*. This might require
@@ -100,8 +100,9 @@ Tuple Objects
 
 .. c:function:: PyObject* PyTuple_GetSlice(PyObject *p, Py_ssize_t low, Py_ssize_t high)
 
-   Take a slice of the tuple pointed to by *p* from *low* to *high* and return it
-   as a new tuple.
+   Return the slice of the tuple pointed to by *p* between *low* and *high*,
+   or *NULL* on failure.  This is the equivalent of the Python expression
+   ``p[low:high]``.  Indexing from the end of the list is not supported.
 
    .. versionchanged:: 2.5
       This function used an :c:type:`int` type for *low* and *high*. This might
@@ -111,11 +112,13 @@ Tuple Objects
 .. c:function:: int PyTuple_SetItem(PyObject *p, Py_ssize_t pos, PyObject *o)
 
    Insert a reference to object *o* at position *pos* of the tuple pointed to by
-   *p*. Return ``0`` on success.
+   *p*.  Return ``0`` on success.  If *pos* is out of bounds, return ``-1``
+   and set an :exc:`IndexError` exception.
 
    .. note::
 
-      This function "steals" a reference to *o*.
+      This function "steals" a reference to *o* and discards a reference to
+      an item already in the tuple at the affected position.
 
    .. versionchanged:: 2.5
       This function used an :c:type:`int` type for *pos*. This might require
@@ -129,7 +132,10 @@ Tuple Objects
 
    .. note::
 
-      This function "steals" a reference to *o*.
+      This macro "steals" a reference to *o*, and, unlike
+      :c:func:`PyTuple_SetItem`, does *not* discard a reference to any item that
+      is being replaced; any reference in the tuple at position *pos* will be
+      leaked.
 
    .. versionchanged:: 2.5
       This function used an :c:type:`int` type for *pos*. This might require
diff --git a/Doc/tools/susp-ignored.csv b/Doc/tools/susp-ignored.csv
index f7d23e22baeac..0efbbabebfe40 100644
--- a/Doc/tools/susp-ignored.csv
+++ b/Doc/tools/susp-ignored.csv
@@ -1,6 +1,7 @@
 c-api/arg,,:ref,"PyArg_ParseTuple(args, ""O|O:ref"", &object, &callback)"
 c-api/list,,:high,list[low:high]
 c-api/sequence,,:i2,o[i1:i2]
+c-api/tuple,,:high,p[low:high]
 c-api/unicode,,:end,str[start:end]
 distutils/setupscript,,::,
 extending/embedding,,:numargs,"if(!PyArg_ParseTuple(args, "":numargs""))"


From webhook-mailer at python.org  Sat Oct 26 16:05:22 2019
From: webhook-mailer at python.org (Miss Skeleton (bot))
Date: Sat, 26 Oct 2019 20:05:22 -0000
Subject: [Python-checkins] bpo-38557: Improve documentation for list and
 tuple C API. (GH-16925)
Message-ID: <mailman.144.1572120323.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/334fc923b3a7a8d0d163692befd2a27b98b481df
commit: 334fc923b3a7a8d0d163692befd2a27b98b481df
branch: 3.7
author: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-26T13:05:18-07:00
summary:

bpo-38557: Improve documentation for list and tuple C API. (GH-16925)

(cherry picked from commit d898d20e8c228229eb68e545f544db13f246f216)

Co-authored-by: Serhiy Storchaka <storchaka at gmail.com>

files:
M Doc/c-api/list.rst
M Doc/c-api/tuple.rst
M Doc/tools/susp-ignored.csv

diff --git a/Doc/c-api/list.rst b/Doc/c-api/list.rst
index a5cd634a454a9..c7cae550a46fa 100644
--- a/Doc/c-api/list.rst
+++ b/Doc/c-api/list.rst
@@ -71,8 +71,9 @@ List Objects
 
 .. c:function:: int PyList_SetItem(PyObject *list, Py_ssize_t index, PyObject *item)
 
-   Set the item at index *index* in list to *item*.  Return ``0`` on success
-   or ``-1`` on failure.
+   Set the item at index *index* in list to *item*.  Return ``0`` on success.
+   If *index* is out of bounds, return ``-1`` and set an :exc:`IndexError`
+   exception.
 
    .. note::
 
@@ -111,8 +112,7 @@ List Objects
 
    Return a list of the objects in *list* containing the objects *between* *low*
    and *high*.  Return *NULL* and set an exception if unsuccessful.  Analogous
-   to ``list[low:high]``.  Negative indices, as when slicing from Python, are not
-   supported.
+   to ``list[low:high]``.  Indexing from the end of the list is not supported.
 
 
 .. c:function:: int PyList_SetSlice(PyObject *list, Py_ssize_t low, Py_ssize_t high, PyObject *itemlist)
@@ -120,8 +120,8 @@ List Objects
    Set the slice of *list* between *low* and *high* to the contents of
    *itemlist*.  Analogous to ``list[low:high] = itemlist``. The *itemlist* may
    be *NULL*, indicating the assignment of an empty list (slice deletion).
-   Return ``0`` on success, ``-1`` on failure.  Negative indices, as when
-   slicing from Python, are not supported.
+   Return ``0`` on success, ``-1`` on failure.  Indexing from the end of the
+   list is not supported.
 
 
 .. c:function:: int PyList_Sort(PyObject *list)
diff --git a/Doc/c-api/tuple.rst b/Doc/c-api/tuple.rst
index 20bf9f0b0804c..512d00db0066f 100644
--- a/Doc/c-api/tuple.rst
+++ b/Doc/c-api/tuple.rst
@@ -57,7 +57,7 @@ Tuple Objects
 .. c:function:: PyObject* PyTuple_GetItem(PyObject *p, Py_ssize_t pos)
 
    Return the object at position *pos* in the tuple pointed to by *p*.  If *pos* is
-   out of bounds, return *NULL* and sets an :exc:`IndexError` exception.
+   out of bounds, return *NULL* and set an :exc:`IndexError` exception.
 
 
 .. c:function:: PyObject* PyTuple_GET_ITEM(PyObject *p, Py_ssize_t pos)
@@ -67,18 +67,21 @@ Tuple Objects
 
 .. c:function:: PyObject* PyTuple_GetSlice(PyObject *p, Py_ssize_t low, Py_ssize_t high)
 
-   Take a slice of the tuple pointed to by *p* from *low* to *high* and return it
-   as a new tuple.
+   Return the slice of the tuple pointed to by *p* between *low* and *high*,
+   or *NULL* on failure.  This is the equivalent of the Python expression
+   ``p[low:high]``.  Indexing from the end of the list is not supported.
 
 
 .. c:function:: int PyTuple_SetItem(PyObject *p, Py_ssize_t pos, PyObject *o)
 
    Insert a reference to object *o* at position *pos* of the tuple pointed to by
-   *p*. Return ``0`` on success.
+   *p*.  Return ``0`` on success.  If *pos* is out of bounds, return ``-1``
+   and set an :exc:`IndexError` exception.
 
    .. note::
 
-      This function "steals" a reference to *o*.
+      This function "steals" a reference to *o* and discards a reference to
+      an item already in the tuple at the affected position.
 
 
 .. c:function:: void PyTuple_SET_ITEM(PyObject *p, Py_ssize_t pos, PyObject *o)
@@ -88,7 +91,10 @@ Tuple Objects
 
    .. note::
 
-      This function "steals" a reference to *o*.
+      This macro "steals" a reference to *o*, and, unlike
+      :c:func:`PyTuple_SetItem`, does *not* discard a reference to any item that
+      is being replaced; any reference in the tuple at position *pos* will be
+      leaked.
 
 
 .. c:function:: int _PyTuple_Resize(PyObject **p, Py_ssize_t newsize)
diff --git a/Doc/tools/susp-ignored.csv b/Doc/tools/susp-ignored.csv
index 36d93a45b0209..0eb9901d3ccf5 100644
--- a/Doc/tools/susp-ignored.csv
+++ b/Doc/tools/susp-ignored.csv
@@ -2,6 +2,7 @@ c-api/arg,,:ref,"PyArg_ParseTuple(args, ""O|O:ref"", &object, &callback)"
 c-api/list,,:high,list[low:high]
 c-api/sequence,,:i2,del o[i1:i2]
 c-api/sequence,,:i2,o[i1:i2]
+c-api/tuple,,:high,p[low:high]
 c-api/unicode,,:end,str[start:end]
 c-api/unicode,,:start,unicode[start:start+length]
 distutils/examples,267,`,This is the description of the ``foobar`` package.


From webhook-mailer at python.org  Sat Oct 26 16:06:38 2019
From: webhook-mailer at python.org (Miss Skeleton (bot))
Date: Sat, 26 Oct 2019 20:06:38 -0000
Subject: [Python-checkins] bpo-38557: Improve documentation for list and
 tuple C API. (GH-16925)
Message-ID: <mailman.145.1572120399.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/4992dc6610fb354e36c0012a47ea9613b61c9038
commit: 4992dc6610fb354e36c0012a47ea9613b61c9038
branch: 3.8
author: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-26T13:06:34-07:00
summary:

bpo-38557: Improve documentation for list and tuple C API. (GH-16925)

(cherry picked from commit d898d20e8c228229eb68e545f544db13f246f216)

Co-authored-by: Serhiy Storchaka <storchaka at gmail.com>

files:
M Doc/c-api/list.rst
M Doc/c-api/tuple.rst
M Doc/tools/susp-ignored.csv

diff --git a/Doc/c-api/list.rst b/Doc/c-api/list.rst
index dc9026605be35..646f29d08a04b 100644
--- a/Doc/c-api/list.rst
+++ b/Doc/c-api/list.rst
@@ -71,8 +71,9 @@ List Objects
 
 .. c:function:: int PyList_SetItem(PyObject *list, Py_ssize_t index, PyObject *item)
 
-   Set the item at index *index* in list to *item*.  Return ``0`` on success
-   or ``-1`` on failure.
+   Set the item at index *index* in list to *item*.  Return ``0`` on success.
+   If *index* is out of bounds, return ``-1`` and set an :exc:`IndexError`
+   exception.
 
    .. note::
 
@@ -111,8 +112,7 @@ List Objects
 
    Return a list of the objects in *list* containing the objects *between* *low*
    and *high*.  Return *NULL* and set an exception if unsuccessful.  Analogous
-   to ``list[low:high]``.  Negative indices, as when slicing from Python, are not
-   supported.
+   to ``list[low:high]``.  Indexing from the end of the list is not supported.
 
 
 .. c:function:: int PyList_SetSlice(PyObject *list, Py_ssize_t low, Py_ssize_t high, PyObject *itemlist)
@@ -120,8 +120,8 @@ List Objects
    Set the slice of *list* between *low* and *high* to the contents of
    *itemlist*.  Analogous to ``list[low:high] = itemlist``. The *itemlist* may
    be *NULL*, indicating the assignment of an empty list (slice deletion).
-   Return ``0`` on success, ``-1`` on failure.  Negative indices, as when
-   slicing from Python, are not supported.
+   Return ``0`` on success, ``-1`` on failure.  Indexing from the end of the
+   list is not supported.
 
 
 .. c:function:: int PyList_Sort(PyObject *list)
diff --git a/Doc/c-api/tuple.rst b/Doc/c-api/tuple.rst
index 259ec4fb48515..a0d53e2030e63 100644
--- a/Doc/c-api/tuple.rst
+++ b/Doc/c-api/tuple.rst
@@ -57,7 +57,7 @@ Tuple Objects
 .. c:function:: PyObject* PyTuple_GetItem(PyObject *p, Py_ssize_t pos)
 
    Return the object at position *pos* in the tuple pointed to by *p*.  If *pos* is
-   out of bounds, return *NULL* and sets an :exc:`IndexError` exception.
+   out of bounds, return *NULL* and set an :exc:`IndexError` exception.
 
 
 .. c:function:: PyObject* PyTuple_GET_ITEM(PyObject *p, Py_ssize_t pos)
@@ -67,18 +67,21 @@ Tuple Objects
 
 .. c:function:: PyObject* PyTuple_GetSlice(PyObject *p, Py_ssize_t low, Py_ssize_t high)
 
-   Take a slice of the tuple pointed to by *p* from *low* to *high* and return it
-   as a new tuple.
+   Return the slice of the tuple pointed to by *p* between *low* and *high*,
+   or *NULL* on failure.  This is the equivalent of the Python expression
+   ``p[low:high]``.  Indexing from the end of the list is not supported.
 
 
 .. c:function:: int PyTuple_SetItem(PyObject *p, Py_ssize_t pos, PyObject *o)
 
    Insert a reference to object *o* at position *pos* of the tuple pointed to by
-   *p*. Return ``0`` on success.
+   *p*.  Return ``0`` on success.  If *pos* is out of bounds, return ``-1``
+   and set an :exc:`IndexError` exception.
 
    .. note::
 
-      This function "steals" a reference to *o*.
+      This function "steals" a reference to *o* and discards a reference to
+      an item already in the tuple at the affected position.
 
 
 .. c:function:: void PyTuple_SET_ITEM(PyObject *p, Py_ssize_t pos, PyObject *o)
@@ -88,7 +91,10 @@ Tuple Objects
 
    .. note::
 
-      This function "steals" a reference to *o*.
+      This macro "steals" a reference to *o*, and, unlike
+      :c:func:`PyTuple_SetItem`, does *not* discard a reference to any item that
+      is being replaced; any reference in the tuple at position *pos* will be
+      leaked.
 
 
 .. c:function:: int _PyTuple_Resize(PyObject **p, Py_ssize_t newsize)
diff --git a/Doc/tools/susp-ignored.csv b/Doc/tools/susp-ignored.csv
index a0fc874829f5f..e86786a2cc81b 100644
--- a/Doc/tools/susp-ignored.csv
+++ b/Doc/tools/susp-ignored.csv
@@ -2,6 +2,7 @@ c-api/arg,,:ref,"PyArg_ParseTuple(args, ""O|O:ref"", &object, &callback)"
 c-api/list,,:high,list[low:high]
 c-api/sequence,,:i2,del o[i1:i2]
 c-api/sequence,,:i2,o[i1:i2]
+c-api/tuple,,:high,p[low:high]
 c-api/unicode,,:end,str[start:end]
 c-api/unicode,,:start,unicode[start:start+length]
 distutils/examples,267,`,This is the description of the ``foobar`` package.


From webhook-mailer at python.org  Sat Oct 26 16:09:39 2019
From: webhook-mailer at python.org (Miss Skeleton (bot))
Date: Sat, 26 Oct 2019 20:09:39 -0000
Subject: [Python-checkins] bpo-38434: Fixes some audit event documentation
 (GH-16932)
Message-ID: <mailman.146.1572120580.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/11f0f11c4859f5ca201cd40b379b13c65f05ec91
commit: 11f0f11c4859f5ca201cd40b379b13c65f05ec91
branch: 3.8
author: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-26T13:09:35-07:00
summary:

bpo-38434: Fixes some audit event documentation (GH-16932)

(cherry picked from commit 894e30ce0bcc1c509eb01c8ffa9ba6d7701aeaaf)

Co-authored-by: Steve Dower <steve.dower at python.org>

files:
M Doc/c-api/code.rst
M Doc/c-api/sys.rst
M Doc/library/sys.rst
M Doc/library/types.rst
M Doc/reference/simple_stmts.rst

diff --git a/Doc/c-api/code.rst b/Doc/c-api/code.rst
index 3c4f66923da5a..45a6b4a753a72 100644
--- a/Doc/c-api/code.rst
+++ b/Doc/c-api/code.rst
@@ -40,8 +40,6 @@ bound into a function.
    can bind you to a precise Python version since the definition of the bytecode
    changes often.
 
-   .. audit-event:: code.__new__ code,filename,name,argcount,posonlyargcount,kwonlyargcount,nlocals,stacksize,flags c.PyCode_New
-
 .. c:function:: PyCodeObject* PyCode_NewWithPosOnlyArgs(int argcount, int posonlyargcount, int kwonlyargcount, int nlocals, int stacksize, int flags, PyObject *code, PyObject *consts, PyObject *names, PyObject *varnames, PyObject *freevars, PyObject *cellvars, PyObject *filename, PyObject *name, int firstlineno, PyObject *lnotab)
 
    Similar to :c:func:`PyCode_New`, but with an extra "posonlyargcount" for positonal-only arguments.
diff --git a/Doc/c-api/sys.rst b/Doc/c-api/sys.rst
index 838a97cacfdf1..87d2bd13d31b8 100644
--- a/Doc/c-api/sys.rst
+++ b/Doc/c-api/sys.rst
@@ -330,7 +330,16 @@ accessible to C code.  They all work with the current interpreter thread's
 
    See :pep:`578` for a detailed description of auditing. Functions in the
    runtime and standard library that raise events include the details in each
-   function's documentation.
+   function's documentation and listed in the :ref:`audit events table
+   <audit-events>`.
+
+   .. audit-event:: sys.addaudithook "" c.PySys_AddAuditHook
+
+      If the interpreter is initialized, this function raises a auditing event
+      ``sys.addaudithook`` with no arguments. If any existing hooks raise an
+      exception derived from :class:`Exception`, the new hook will not be
+      added and the exception is cleared. As a result, callers cannot assume
+      that their hook has been added unless they control all existing hooks.
 
    .. versionadded:: 3.8
 
diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst
index a698ecb64d859..34b808406866b 100644
--- a/Doc/library/sys.rst
+++ b/Doc/library/sys.rst
@@ -33,18 +33,21 @@ always available.
    tuple of arguments. Native hooks added by :c:func:`PySys_AddAuditHook` are
    called first, followed by hooks added in the current interpreter.
 
-   Calling this function will trigger an event for all existing hooks, and if
-   any raise an exception derived from :class:`Exception`, the add will be
-   silently ignored. As a result, callers cannot assume that their hook has been
-   added unless they control all existing hooks.
+   .. audit-event:: sys.addaudithook "" sys.addaudithook
+
+      Raises a auditing event ``sys.addaudithook`` with no arguments. If any
+      existing hooks raise an exception derived from :class:`Exception`, the
+      new hook will not be added and the exception suppressed. As a result,
+      callers cannot assume that their hook has been added unless they control
+      all existing hooks.
 
    .. versionadded:: 3.8
 
    .. impl-detail::
 
-      When tracing is enabled, Python hooks are only traced if the callable has
-      a ``__cantrace__`` member that is set to a true value. Otherwise, trace
-      functions will not see the hook.
+      When tracing is enabled (see :func:`settrace`), Python hooks are only
+      traced if the callable has a ``__cantrace__`` member that is set to a
+      true value. Otherwise, trace functions will skip the hook.
 
 
 .. data:: argv
@@ -87,7 +90,7 @@ always available.
    native function is preferred when possible.
 
    See the :ref:`audit events table <audit-events>` for all events raised by
-   ``CPython``.
+   CPython.
 
    .. versionadded:: 3.8
 
diff --git a/Doc/library/types.rst b/Doc/library/types.rst
index a21fb44dda5de..9393f9e6db990 100644
--- a/Doc/library/types.rst
+++ b/Doc/library/types.rst
@@ -138,6 +138,10 @@ Standard names are defined for the following types:
 
    The type for code objects such as returned by :func:`compile`.
 
+   .. audit-event:: code.__new__ code,filename,name,argcount,posonlyargcount,kwonlyargcount,nlocals,stacksize,flags CodeType
+
+   Note that the audited arguments may not match the names or positions
+   required by the initializer.
 
 .. data:: CellType
 
diff --git a/Doc/reference/simple_stmts.rst b/Doc/reference/simple_stmts.rst
index 0a043a90050c4..a8ec0fbe8b732 100644
--- a/Doc/reference/simple_stmts.rst
+++ b/Doc/reference/simple_stmts.rst
@@ -839,6 +839,7 @@ the :ref:`relativeimports` section.
 :func:`importlib.import_module` is provided to support applications that
 determine dynamically the modules to be loaded.
 
+.. audit-event:: import module,filename,sys.path,sys.meta_path,sys.path_hooks import
 
 .. _future:
 


From webhook-mailer at python.org  Sat Oct 26 21:15:27 2019
From: webhook-mailer at python.org (Terry Jan Reedy)
Date: Sun, 27 Oct 2019 01:15:27 -0000
Subject: [Python-checkins] bpo-38598: Do not try to compile IDLE shell or
 output windows (GH-16939)
Message-ID: <mailman.147.1572138927.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/e3f90b217a5152275b180b466bd503658a734462
commit: e3f90b217a5152275b180b466bd503658a734462
branch: master
author: Terry Jan Reedy <tjreedy at udel.edu>
committer: GitHub <noreply at github.com>
date: 2019-10-26T21:15:10-04:00
summary:

bpo-38598: Do not try to compile IDLE shell or output windows (GH-16939)

files:
A Misc/NEWS.d/next/IDLE/2019-10-26-18-16-24.bpo-38598.6kH9FY.rst
M Lib/idlelib/runscript.py

diff --git a/Lib/idlelib/runscript.py b/Lib/idlelib/runscript.py
index e99d0d2e027fc..a54108794ab59 100644
--- a/Lib/idlelib/runscript.py
+++ b/Lib/idlelib/runscript.py
@@ -19,6 +19,7 @@
 from idlelib import macosx
 from idlelib import pyshell
 from idlelib.query import CustomRun
+from idlelib import outwin
 
 indent_message = """Error: Inconsistent indentation detected!
 
@@ -46,6 +47,9 @@ def __init__(self, editwin):
             self.editwin.text_frame.bind('<<run-module-event-2>>', self._run_module_event)
 
     def check_module_event(self, event):
+        if isinstance(self.editwin, outwin.OutputWindow):
+            self.editwin.text.bell()
+            return 'break'
         filename = self.getfilename()
         if not filename:
             return 'break'
@@ -129,6 +133,9 @@ def _run_module_event(self, event, *, customize=False):
         module being executed and also add that directory to its
         sys.path if not already included.
         """
+        if isinstance(self.editwin, outwin.OutputWindow):
+            self.editwin.text.bell()
+            return 'break'
         filename = self.getfilename()
         if not filename:
             return 'break'
diff --git a/Misc/NEWS.d/next/IDLE/2019-10-26-18-16-24.bpo-38598.6kH9FY.rst b/Misc/NEWS.d/next/IDLE/2019-10-26-18-16-24.bpo-38598.6kH9FY.rst
new file mode 100644
index 0000000000000..5d04e4a79b622
--- /dev/null
+++ b/Misc/NEWS.d/next/IDLE/2019-10-26-18-16-24.bpo-38598.6kH9FY.rst
@@ -0,0 +1 @@
+Do not try to compile IDLE shell or output windows


From webhook-mailer at python.org  Sat Oct 26 21:32:15 2019
From: webhook-mailer at python.org (Miss Skeleton (bot))
Date: Sun, 27 Oct 2019 01:32:15 -0000
Subject: [Python-checkins] bpo-38598: Do not try to compile IDLE shell or
 output windows (GH-16939)
Message-ID: <mailman.148.1572139937.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/849b1b9f6c538085839f8059e0d6a5536d2fdb52
commit: 849b1b9f6c538085839f8059e0d6a5536d2fdb52
branch: 3.7
author: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-26T18:32:11-07:00
summary:

bpo-38598: Do not try to compile IDLE shell or output windows (GH-16939)

(cherry picked from commit e3f90b217a5152275b180b466bd503658a734462)

Co-authored-by: Terry Jan Reedy <tjreedy at udel.edu>

files:
A Misc/NEWS.d/next/IDLE/2019-10-26-18-16-24.bpo-38598.6kH9FY.rst
M Lib/idlelib/runscript.py

diff --git a/Lib/idlelib/runscript.py b/Lib/idlelib/runscript.py
index e99d0d2e027fc..a54108794ab59 100644
--- a/Lib/idlelib/runscript.py
+++ b/Lib/idlelib/runscript.py
@@ -19,6 +19,7 @@
 from idlelib import macosx
 from idlelib import pyshell
 from idlelib.query import CustomRun
+from idlelib import outwin
 
 indent_message = """Error: Inconsistent indentation detected!
 
@@ -46,6 +47,9 @@ def __init__(self, editwin):
             self.editwin.text_frame.bind('<<run-module-event-2>>', self._run_module_event)
 
     def check_module_event(self, event):
+        if isinstance(self.editwin, outwin.OutputWindow):
+            self.editwin.text.bell()
+            return 'break'
         filename = self.getfilename()
         if not filename:
             return 'break'
@@ -129,6 +133,9 @@ def _run_module_event(self, event, *, customize=False):
         module being executed and also add that directory to its
         sys.path if not already included.
         """
+        if isinstance(self.editwin, outwin.OutputWindow):
+            self.editwin.text.bell()
+            return 'break'
         filename = self.getfilename()
         if not filename:
             return 'break'
diff --git a/Misc/NEWS.d/next/IDLE/2019-10-26-18-16-24.bpo-38598.6kH9FY.rst b/Misc/NEWS.d/next/IDLE/2019-10-26-18-16-24.bpo-38598.6kH9FY.rst
new file mode 100644
index 0000000000000..5d04e4a79b622
--- /dev/null
+++ b/Misc/NEWS.d/next/IDLE/2019-10-26-18-16-24.bpo-38598.6kH9FY.rst
@@ -0,0 +1 @@
+Do not try to compile IDLE shell or output windows


From webhook-mailer at python.org  Sat Oct 26 21:41:30 2019
From: webhook-mailer at python.org (Miss Skeleton (bot))
Date: Sun, 27 Oct 2019 01:41:30 -0000
Subject: [Python-checkins] bpo-38598: Do not try to compile IDLE shell or
 output windows (GH-16939)
Message-ID: <mailman.149.1572140490.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/e3a477aa072a737d5119b0778d14d9f3ddb9f587
commit: e3a477aa072a737d5119b0778d14d9f3ddb9f587
branch: 3.8
author: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-26T18:41:26-07:00
summary:

bpo-38598: Do not try to compile IDLE shell or output windows (GH-16939)

(cherry picked from commit e3f90b217a5152275b180b466bd503658a734462)

Co-authored-by: Terry Jan Reedy <tjreedy at udel.edu>

files:
A Misc/NEWS.d/next/IDLE/2019-10-26-18-16-24.bpo-38598.6kH9FY.rst
M Lib/idlelib/runscript.py

diff --git a/Lib/idlelib/runscript.py b/Lib/idlelib/runscript.py
index e99d0d2e027fc..a54108794ab59 100644
--- a/Lib/idlelib/runscript.py
+++ b/Lib/idlelib/runscript.py
@@ -19,6 +19,7 @@
 from idlelib import macosx
 from idlelib import pyshell
 from idlelib.query import CustomRun
+from idlelib import outwin
 
 indent_message = """Error: Inconsistent indentation detected!
 
@@ -46,6 +47,9 @@ def __init__(self, editwin):
             self.editwin.text_frame.bind('<<run-module-event-2>>', self._run_module_event)
 
     def check_module_event(self, event):
+        if isinstance(self.editwin, outwin.OutputWindow):
+            self.editwin.text.bell()
+            return 'break'
         filename = self.getfilename()
         if not filename:
             return 'break'
@@ -129,6 +133,9 @@ def _run_module_event(self, event, *, customize=False):
         module being executed and also add that directory to its
         sys.path if not already included.
         """
+        if isinstance(self.editwin, outwin.OutputWindow):
+            self.editwin.text.bell()
+            return 'break'
         filename = self.getfilename()
         if not filename:
             return 'break'
diff --git a/Misc/NEWS.d/next/IDLE/2019-10-26-18-16-24.bpo-38598.6kH9FY.rst b/Misc/NEWS.d/next/IDLE/2019-10-26-18-16-24.bpo-38598.6kH9FY.rst
new file mode 100644
index 0000000000000..5d04e4a79b622
--- /dev/null
+++ b/Misc/NEWS.d/next/IDLE/2019-10-26-18-16-24.bpo-38598.6kH9FY.rst
@@ -0,0 +1 @@
+Do not try to compile IDLE shell or output windows


From webhook-mailer at python.org  Sat Oct 26 22:20:00 2019
From: webhook-mailer at python.org (Terry Jan Reedy)
Date: Sun, 27 Oct 2019 02:20:00 -0000
Subject: [Python-checkins] bpo-34162: Last idlelib/NEWS.txt items for 3.8.0.
 (GH-16943)
Message-ID: <mailman.150.1572142801.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/e31a79a5b44357b409d71949dc5308889970f9ab
commit: e31a79a5b44357b409d71949dc5308889970f9ab
branch: master
author: Terry Jan Reedy <tjreedy at udel.edu>
committer: GitHub <noreply at github.com>
date: 2019-10-26T22:19:57-04:00
summary:

bpo-34162: Last idlelib/NEWS.txt items for 3.8.0. (GH-16943)

files:
M Lib/idlelib/NEWS.txt

diff --git a/Lib/idlelib/NEWS.txt b/Lib/idlelib/NEWS.txt
index 9d26bda42594e..1597eb960b2b4 100644
--- a/Lib/idlelib/NEWS.txt
+++ b/Lib/idlelib/NEWS.txt
@@ -2,6 +2,15 @@ What's New in IDLE 3.8.0 (since 3.7.0)
 Released on 2019-10-20?
 ======================================
 
+bpo-36698: IDLE no longer fails when writing non-encodable characters
+to stderr.  It now escapes them with a backslash, like the regular
+Python interpreter.  Add an errors field to the standard streams.
+
+bpo-13153: Improve tkinter's handing of non-BMP (astral) unicode
+characters, such as 'rocket \U0001f680'.  Whether a proper glyph or
+replacement char is displayed depends on the OS and font.  For IDLE,
+astral chars in code interfere with editing.
+
 bpo-35379: When exiting IDLE, catch any AttributeError.  One happens
 when EditorWindow.close is called twice.  Printing a traceback, when
 IDLE is run from a terminal, is useless and annoying.


From webhook-mailer at python.org  Sat Oct 26 22:37:29 2019
From: webhook-mailer at python.org (Miss Skeleton (bot))
Date: Sun, 27 Oct 2019 02:37:29 -0000
Subject: [Python-checkins] bpo-34162: Last idlelib/NEWS.txt items for 3.8.0.
 (GH-16943)
Message-ID: <mailman.151.1572143849.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/7a3b1a6f0b0a5ac6e9af4c5ee8b89a2fca96efb6
commit: 7a3b1a6f0b0a5ac6e9af4c5ee8b89a2fca96efb6
branch: 3.8
author: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-26T19:37:25-07:00
summary:

bpo-34162: Last idlelib/NEWS.txt items for 3.8.0. (GH-16943)

(cherry picked from commit e31a79a5b44357b409d71949dc5308889970f9ab)

Co-authored-by: Terry Jan Reedy <tjreedy at udel.edu>

files:
M Lib/idlelib/NEWS.txt

diff --git a/Lib/idlelib/NEWS.txt b/Lib/idlelib/NEWS.txt
index 9d26bda42594e..1597eb960b2b4 100644
--- a/Lib/idlelib/NEWS.txt
+++ b/Lib/idlelib/NEWS.txt
@@ -2,6 +2,15 @@ What's New in IDLE 3.8.0 (since 3.7.0)
 Released on 2019-10-20?
 ======================================
 
+bpo-36698: IDLE no longer fails when writing non-encodable characters
+to stderr.  It now escapes them with a backslash, like the regular
+Python interpreter.  Add an errors field to the standard streams.
+
+bpo-13153: Improve tkinter's handing of non-BMP (astral) unicode
+characters, such as 'rocket \U0001f680'.  Whether a proper glyph or
+replacement char is displayed depends on the OS and font.  For IDLE,
+astral chars in code interfere with editing.
+
 bpo-35379: When exiting IDLE, catch any AttributeError.  One happens
 when EditorWindow.close is called twice.  Printing a traceback, when
 IDLE is run from a terminal, is useless and annoying.


From webhook-mailer at python.org  Sat Oct 26 22:38:30 2019
From: webhook-mailer at python.org (Miss Skeleton (bot))
Date: Sun, 27 Oct 2019 02:38:30 -0000
Subject: [Python-checkins] bpo-34162: Last idlelib/NEWS.txt items for 3.8.0.
 (GH-16943)
Message-ID: <mailman.152.1572143910.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/bc30db1ee70b6708aa436868e548aef919567448
commit: bc30db1ee70b6708aa436868e548aef919567448
branch: 3.7
author: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-26T19:38:26-07:00
summary:

bpo-34162: Last idlelib/NEWS.txt items for 3.8.0. (GH-16943)

(cherry picked from commit e31a79a5b44357b409d71949dc5308889970f9ab)

Co-authored-by: Terry Jan Reedy <tjreedy at udel.edu>

files:
M Lib/idlelib/NEWS.txt

diff --git a/Lib/idlelib/NEWS.txt b/Lib/idlelib/NEWS.txt
index 32fda8c75bb8f..00d00a160b017 100644
--- a/Lib/idlelib/NEWS.txt
+++ b/Lib/idlelib/NEWS.txt
@@ -2,6 +2,15 @@ What's New in IDLE 3.7.5
 Released on 2019-09-30?
 ======================================
 
+bpo-36698: IDLE no longer fails when writing non-encodable characters
+to stderr.  It now escapes them with a backslash, like the regular
+Python interpreter.  Add an errors field to the standard streams.
+
+bpo-13153: Improve tkinter's handing of non-BMP (astral) unicode
+characters, such as 'rocket \U0001f680'.  Whether a proper glyph or
+replacement char is displayed depends on the OS and font.  For IDLE,
+astral chars in code interfere with editing.
+
 bpo-35379: When exiting IDLE, catch any AttributeError.  One happens
 when EditorWindow.close is called twice.  Printing a traceback, when
 IDLE is run from a terminal, is useless and annoying.


From webhook-mailer at python.org  Sun Oct 27 01:23:34 2019
From: webhook-mailer at python.org (Terry Jan Reedy)
Date: Sun, 27 Oct 2019 05:23:34 -0000
Subject: [Python-checkins] bpo-37309: First idlelib/NEWS.txt for 3.9.0
 (GH-16947)
Message-ID: <mailman.153.1572153815.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/a8fb9327fb9387b404d4b6ce55c608125f66b9ae
commit: a8fb9327fb9387b404d4b6ce55c608125f66b9ae
branch: master
author: Terry Jan Reedy <tjreedy at udel.edu>
committer: GitHub <noreply at github.com>
date: 2019-10-27T01:23:30-04:00
summary:

bpo-37309: First idlelib/NEWS.txt for 3.9.0 (GH-16947)

files:
M Lib/idlelib/NEWS.txt

diff --git a/Lib/idlelib/NEWS.txt b/Lib/idlelib/NEWS.txt
index 1597eb960b2b4..ce96baf32859c 100644
--- a/Lib/idlelib/NEWS.txt
+++ b/Lib/idlelib/NEWS.txt
@@ -1,5 +1,13 @@
+What's New in IDLE 3.9.0 (since 3.8.0)
+Released on 2020-10-05?
+======================================
+
+
+bpo-38598: Do not try to compile IDLE shell or output windows.
+
+
 What's New in IDLE 3.8.0 (since 3.7.0)
-Released on 2019-10-20?
+Released on 2019-10-14
 ======================================
 
 bpo-36698: IDLE no longer fails when writing non-encodable characters


From webhook-mailer at python.org  Sun Oct 27 01:25:31 2019
From: webhook-mailer at python.org (Terry Jan Reedy)
Date: Sun, 27 Oct 2019 05:25:31 -0000
Subject: [Python-checkins] [3.8] bpo-37309: First idlelib/NEWS.txt for 3.8.1
 (GH-16947)
Message-ID: <mailman.154.1572153932.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/6d6418a075f272aeab93c6f2ec15c10257b94e1d
commit: 6d6418a075f272aeab93c6f2ec15c10257b94e1d
branch: 3.8
author: Terry Jan Reedy <tjreedy at udel.edu>
committer: GitHub <noreply at github.com>
date: 2019-10-27T01:25:27-04:00
summary:

[3.8] bpo-37309: First idlelib/NEWS.txt for 3.8.1 (GH-16947)

files:
M Lib/idlelib/NEWS.txt

diff --git a/Lib/idlelib/NEWS.txt b/Lib/idlelib/NEWS.txt
index 1597eb960b2b4..04d9bb84b7920 100644
--- a/Lib/idlelib/NEWS.txt
+++ b/Lib/idlelib/NEWS.txt
@@ -1,5 +1,13 @@
+What's New in IDLE 3.8.1
+Released on 2019-12-16?
+======================================
+
+
+bpo-38598: Do not try to compile IDLE shell or output windows
+
+
 What's New in IDLE 3.8.0 (since 3.7.0)
-Released on 2019-10-20?
+Released on 2019-10-14
 ======================================
 
 bpo-36698: IDLE no longer fails when writing non-encodable characters


From webhook-mailer at python.org  Sun Oct 27 02:38:24 2019
From: webhook-mailer at python.org (Terry Jan Reedy)
Date: Sun, 27 Oct 2019 06:38:24 -0000
Subject: [Python-checkins] [3.7] bpo-37309: First idlelib/NEWS.txt for 3.7.6
 (GH-) (#16949)
Message-ID: <mailman.155.1572158305.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/baf2657eff82f5c33ae8194499d97ba963871655
commit: baf2657eff82f5c33ae8194499d97ba963871655
branch: 3.7
author: Terry Jan Reedy <tjreedy at udel.edu>
committer: GitHub <noreply at github.com>
date: 2019-10-27T02:38:20-04:00
summary:

[3.7] bpo-37309: First idlelib/NEWS.txt for 3.7.6 (GH-) (#16949)

files:
M Lib/idlelib/NEWS.txt

diff --git a/Lib/idlelib/NEWS.txt b/Lib/idlelib/NEWS.txt
index 00d00a160b017..6ced11f23729a 100644
--- a/Lib/idlelib/NEWS.txt
+++ b/Lib/idlelib/NEWS.txt
@@ -1,5 +1,13 @@
+What's New in IDLE 3.7.6
+Released on 2019-12-16?
+======================================
+
+
+bpo-38598: Do not try to compile IDLE shell or output windows
+
+
 What's New in IDLE 3.7.5
-Released on 2019-09-30?
+Released on 2019-10-15
 ======================================
 
 bpo-36698: IDLE no longer fails when writing non-encodable characters


From webhook-mailer at python.org  Sun Oct 27 04:22:20 2019
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Sun, 27 Oct 2019 08:22:20 -0000
Subject: [Python-checkins] bpo-38334: Fix seeking backward on an encrypted
 zipfile.ZipExtFile. (GH-16937)
Message-ID: <mailman.156.1572164541.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/5c32af7522d908e8c7da0243af37618433289cc5
commit: 5c32af7522d908e8c7da0243af37618433289cc5
branch: master
author: Serhiy Storchaka <storchaka at gmail.com>
committer: GitHub <noreply at github.com>
date: 2019-10-27T10:22:14+02:00
summary:

bpo-38334: Fix seeking backward on an encrypted zipfile.ZipExtFile. (GH-16937)

Test by Daniel Hillier.

files:
A Misc/NEWS.d/next/Library/2019-10-27-00-08-49.bpo-38334.pfLLmc.rst
M Lib/test/test_zipfile.py
M Lib/zipfile.py

diff --git a/Lib/test/test_zipfile.py b/Lib/test/test_zipfile.py
index f812188a7108e..6e1291e524f29 100644
--- a/Lib/test/test_zipfile.py
+++ b/Lib/test/test_zipfile.py
@@ -1934,6 +1934,44 @@ def test_unicode_password(self):
         self.assertRaises(TypeError, self.zip.open, "test.txt", pwd="python")
         self.assertRaises(TypeError, self.zip.extract, "test.txt", pwd="python")
 
+    def test_seek_tell(self):
+        self.zip.setpassword(b"python")
+        txt = self.plain
+        test_word = b'encryption'
+        bloc = txt.find(test_word)
+        bloc_len = len(test_word)
+        with self.zip.open("test.txt", "r") as fp:
+            fp.seek(bloc, os.SEEK_SET)
+            self.assertEqual(fp.tell(), bloc)
+            fp.seek(-bloc, os.SEEK_CUR)
+            self.assertEqual(fp.tell(), 0)
+            fp.seek(bloc, os.SEEK_CUR)
+            self.assertEqual(fp.tell(), bloc)
+            self.assertEqual(fp.read(bloc_len), txt[bloc:bloc+bloc_len])
+
+            # Make sure that the second read after seeking back beyond
+            # _readbuffer returns the same content (ie. rewind to the start of
+            # the file to read forward to the required position).
+            old_read_size = fp.MIN_READ_SIZE
+            fp.MIN_READ_SIZE = 1
+            fp._readbuffer = b''
+            fp._offset = 0
+            fp.seek(0, os.SEEK_SET)
+            self.assertEqual(fp.tell(), 0)
+            fp.seek(bloc, os.SEEK_CUR)
+            self.assertEqual(fp.read(bloc_len), txt[bloc:bloc+bloc_len])
+            fp.MIN_READ_SIZE = old_read_size
+
+            fp.seek(0, os.SEEK_END)
+            self.assertEqual(fp.tell(), len(txt))
+            fp.seek(0, os.SEEK_SET)
+            self.assertEqual(fp.tell(), 0)
+
+            # Read the file completely to definitely call any eof integrity
+            # checks (crc) and make sure they still pass.
+            fp.read()
+
+
 class AbstractTestsWithRandomBinaryFiles:
     @classmethod
     def setUpClass(cls):
diff --git a/Lib/zipfile.py b/Lib/zipfile.py
index 97a5ebf753b58..6201edc8d7386 100644
--- a/Lib/zipfile.py
+++ b/Lib/zipfile.py
@@ -792,10 +792,10 @@ class ZipExtFile(io.BufferedIOBase):
     # Chunk size to read during seek
     MAX_SEEK_READ = 1 << 24
 
-    def __init__(self, fileobj, mode, zipinfo, decrypter=None,
+    def __init__(self, fileobj, mode, zipinfo, pwd=None,
                  close_fileobj=False):
         self._fileobj = fileobj
-        self._decrypter = decrypter
+        self._pwd = pwd
         self._close_fileobj = close_fileobj
 
         self._compress_type = zipinfo.compress_type
@@ -810,11 +810,6 @@ def __init__(self, fileobj, mode, zipinfo, decrypter=None,
 
         self.newlines = None
 
-        # Adjust read size for encrypted files since the first 12 bytes
-        # are for the encryption/password information.
-        if self._decrypter is not None:
-            self._compress_left -= 12
-
         self.mode = mode
         self.name = zipinfo.filename
 
@@ -835,6 +830,30 @@ def __init__(self, fileobj, mode, zipinfo, decrypter=None,
         except AttributeError:
             pass
 
+        self._decrypter = None
+        if pwd:
+            if zipinfo.flag_bits & 0x8:
+                # compare against the file type from extended local headers
+                check_byte = (zipinfo._raw_time >> 8) & 0xff
+            else:
+                # compare against the CRC otherwise
+                check_byte = (zipinfo.CRC >> 24) & 0xff
+            h = self._init_decrypter()
+            if h != check_byte:
+                raise RuntimeError("Bad password for file %r" % zipinfo.orig_filename)
+
+
+    def _init_decrypter(self):
+        self._decrypter = _ZipDecrypter(self._pwd)
+        # The first 12 bytes in the cypher stream is an encryption header
+        #  used to strengthen the algorithm. The first 11 bytes are
+        #  completely random, while the 12th contains the MSB of the CRC,
+        #  or the MSB of the file time depending on the header type
+        #  and is used to check the correctness of the password.
+        header = self._fileobj.read(12)
+        self._compress_left -= 12
+        return self._decrypter(header)[11]
+
     def __repr__(self):
         result = ['<%s.%s' % (self.__class__.__module__,
                               self.__class__.__qualname__)]
@@ -1061,6 +1080,8 @@ def seek(self, offset, whence=0):
             self._decompressor = _get_decompressor(self._compress_type)
             self._eof = False
             read_offset = new_pos
+            if self._decrypter is not None:
+                self._init_decrypter()
 
         while read_offset > 0:
             read_len = min(self.MAX_SEEK_READ, read_offset)
@@ -1524,32 +1545,16 @@ def open(self, name, mode="r", pwd=None, *, force_zip64=False):
 
             # check for encrypted flag & handle password
             is_encrypted = zinfo.flag_bits & 0x1
-            zd = None
             if is_encrypted:
                 if not pwd:
                     pwd = self.pwd
                 if not pwd:
                     raise RuntimeError("File %r is encrypted, password "
                                        "required for extraction" % name)
+            else:
+                pwd = None
 
-                zd = _ZipDecrypter(pwd)
-                # The first 12 bytes in the cypher stream is an encryption header
-                #  used to strengthen the algorithm. The first 11 bytes are
-                #  completely random, while the 12th contains the MSB of the CRC,
-                #  or the MSB of the file time depending on the header type
-                #  and is used to check the correctness of the password.
-                header = zef_file.read(12)
-                h = zd(header[0:12])
-                if zinfo.flag_bits & 0x8:
-                    # compare against the file type from extended local headers
-                    check_byte = (zinfo._raw_time >> 8) & 0xff
-                else:
-                    # compare against the CRC otherwise
-                    check_byte = (zinfo.CRC >> 24) & 0xff
-                if h[11] != check_byte:
-                    raise RuntimeError("Bad password for file %r" % name)
-
-            return ZipExtFile(zef_file, mode, zinfo, zd, True)
+            return ZipExtFile(zef_file, mode, zinfo, pwd, True)
         except:
             zef_file.close()
             raise
diff --git a/Misc/NEWS.d/next/Library/2019-10-27-00-08-49.bpo-38334.pfLLmc.rst b/Misc/NEWS.d/next/Library/2019-10-27-00-08-49.bpo-38334.pfLLmc.rst
new file mode 100644
index 0000000000000..0d05d3f6e6c09
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-10-27-00-08-49.bpo-38334.pfLLmc.rst
@@ -0,0 +1 @@
+Fixed seeking backward on an encrypted :class:`zipfile.ZipExtFile`.


From webhook-mailer at python.org  Sun Oct 27 04:30:16 2019
From: webhook-mailer at python.org (Julien Palard)
Date: Sun, 27 Oct 2019 08:30:16 -0000
Subject: [Python-checkins] bpo-38592 Add pt-br switcher to Python Docs
 website (GH-16924)
Message-ID: <mailman.157.1572165017.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/85c6f8c65cd4f7219522c1f304bdfff19f785e7a
commit: 85c6f8c65cd4f7219522c1f304bdfff19f785e7a
branch: master
author: Marco Rougeth <marco at rougeth.com>
committer: Julien Palard <julien at palard.fr>
date: 2019-10-27T09:30:12+01:00
summary:

bpo-38592 Add pt-br switcher to Python Docs website (GH-16924)



* ?? Added by blurb_it.

files:
A Misc/NEWS.d/next/Documentation/2019-10-26-13-19-07.bpo-38592.Y96BYO.rst
M Doc/tools/static/switchers.js

diff --git a/Doc/tools/static/switchers.js b/Doc/tools/static/switchers.js
index ef6f473452065..e1ef91a8dfc68 100644
--- a/Doc/tools/static/switchers.js
+++ b/Doc/tools/static/switchers.js
@@ -23,6 +23,7 @@
       'fr': 'French',
       'ja': 'Japanese',
       'ko': 'Korean',
+      'pt-br': 'Brazilian Portuguese',
       'zh-cn': 'Simplified Chinese',
   };
 
diff --git a/Misc/NEWS.d/next/Documentation/2019-10-26-13-19-07.bpo-38592.Y96BYO.rst b/Misc/NEWS.d/next/Documentation/2019-10-26-13-19-07.bpo-38592.Y96BYO.rst
new file mode 100644
index 0000000000000..3752d48a7cb4f
--- /dev/null
+++ b/Misc/NEWS.d/next/Documentation/2019-10-26-13-19-07.bpo-38592.Y96BYO.rst
@@ -0,0 +1 @@
+Add Brazilian Portuguese to the language switcher at Python Documentation website.
\ No newline at end of file


From webhook-mailer at python.org  Sun Oct 27 04:40:48 2019
From: webhook-mailer at python.org (Miss Skeleton (bot))
Date: Sun, 27 Oct 2019 08:40:48 -0000
Subject: [Python-checkins] bpo-38334: Fix seeking backward on an encrypted
 zipfile.ZipExtFile. (GH-16937)
Message-ID: <mailman.158.1572165648.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/76fbdaa2a693caaa1b8eb34104720fc774ff80df
commit: 76fbdaa2a693caaa1b8eb34104720fc774ff80df
branch: 3.8
author: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-27T01:40:44-07:00
summary:

bpo-38334: Fix seeking backward on an encrypted zipfile.ZipExtFile. (GH-16937)


Test by Daniel Hillier.
(cherry picked from commit 5c32af7522d908e8c7da0243af37618433289cc5)

Co-authored-by: Serhiy Storchaka <storchaka at gmail.com>

files:
A Misc/NEWS.d/next/Library/2019-10-27-00-08-49.bpo-38334.pfLLmc.rst
M Lib/test/test_zipfile.py
M Lib/zipfile.py

diff --git a/Lib/test/test_zipfile.py b/Lib/test/test_zipfile.py
index 76e2f647c607b..e9ef89dd3eb2b 100644
--- a/Lib/test/test_zipfile.py
+++ b/Lib/test/test_zipfile.py
@@ -1907,6 +1907,44 @@ def test_unicode_password(self):
         self.assertRaises(TypeError, self.zip.open, "test.txt", pwd="python")
         self.assertRaises(TypeError, self.zip.extract, "test.txt", pwd="python")
 
+    def test_seek_tell(self):
+        self.zip.setpassword(b"python")
+        txt = self.plain
+        test_word = b'encryption'
+        bloc = txt.find(test_word)
+        bloc_len = len(test_word)
+        with self.zip.open("test.txt", "r") as fp:
+            fp.seek(bloc, os.SEEK_SET)
+            self.assertEqual(fp.tell(), bloc)
+            fp.seek(-bloc, os.SEEK_CUR)
+            self.assertEqual(fp.tell(), 0)
+            fp.seek(bloc, os.SEEK_CUR)
+            self.assertEqual(fp.tell(), bloc)
+            self.assertEqual(fp.read(bloc_len), txt[bloc:bloc+bloc_len])
+
+            # Make sure that the second read after seeking back beyond
+            # _readbuffer returns the same content (ie. rewind to the start of
+            # the file to read forward to the required position).
+            old_read_size = fp.MIN_READ_SIZE
+            fp.MIN_READ_SIZE = 1
+            fp._readbuffer = b''
+            fp._offset = 0
+            fp.seek(0, os.SEEK_SET)
+            self.assertEqual(fp.tell(), 0)
+            fp.seek(bloc, os.SEEK_CUR)
+            self.assertEqual(fp.read(bloc_len), txt[bloc:bloc+bloc_len])
+            fp.MIN_READ_SIZE = old_read_size
+
+            fp.seek(0, os.SEEK_END)
+            self.assertEqual(fp.tell(), len(txt))
+            fp.seek(0, os.SEEK_SET)
+            self.assertEqual(fp.tell(), 0)
+
+            # Read the file completely to definitely call any eof integrity
+            # checks (crc) and make sure they still pass.
+            fp.read()
+
+
 class AbstractTestsWithRandomBinaryFiles:
     @classmethod
     def setUpClass(cls):
diff --git a/Lib/zipfile.py b/Lib/zipfile.py
index 59435064eabf2..e1735416ec8eb 100644
--- a/Lib/zipfile.py
+++ b/Lib/zipfile.py
@@ -792,10 +792,10 @@ class ZipExtFile(io.BufferedIOBase):
     # Chunk size to read during seek
     MAX_SEEK_READ = 1 << 24
 
-    def __init__(self, fileobj, mode, zipinfo, decrypter=None,
+    def __init__(self, fileobj, mode, zipinfo, pwd=None,
                  close_fileobj=False):
         self._fileobj = fileobj
-        self._decrypter = decrypter
+        self._pwd = pwd
         self._close_fileobj = close_fileobj
 
         self._compress_type = zipinfo.compress_type
@@ -810,11 +810,6 @@ def __init__(self, fileobj, mode, zipinfo, decrypter=None,
 
         self.newlines = None
 
-        # Adjust read size for encrypted files since the first 12 bytes
-        # are for the encryption/password information.
-        if self._decrypter is not None:
-            self._compress_left -= 12
-
         self.mode = mode
         self.name = zipinfo.filename
 
@@ -835,6 +830,30 @@ def __init__(self, fileobj, mode, zipinfo, decrypter=None,
         except AttributeError:
             pass
 
+        self._decrypter = None
+        if pwd:
+            if zipinfo.flag_bits & 0x8:
+                # compare against the file type from extended local headers
+                check_byte = (zipinfo._raw_time >> 8) & 0xff
+            else:
+                # compare against the CRC otherwise
+                check_byte = (zipinfo.CRC >> 24) & 0xff
+            h = self._init_decrypter()
+            if h != check_byte:
+                raise RuntimeError("Bad password for file %r" % zipinfo.orig_filename)
+
+
+    def _init_decrypter(self):
+        self._decrypter = _ZipDecrypter(self._pwd)
+        # The first 12 bytes in the cypher stream is an encryption header
+        #  used to strengthen the algorithm. The first 11 bytes are
+        #  completely random, while the 12th contains the MSB of the CRC,
+        #  or the MSB of the file time depending on the header type
+        #  and is used to check the correctness of the password.
+        header = self._fileobj.read(12)
+        self._compress_left -= 12
+        return self._decrypter(header)[11]
+
     def __repr__(self):
         result = ['<%s.%s' % (self.__class__.__module__,
                               self.__class__.__qualname__)]
@@ -1061,6 +1080,8 @@ def seek(self, offset, whence=0):
             self._decompressor = _get_decompressor(self._compress_type)
             self._eof = False
             read_offset = new_pos
+            if self._decrypter is not None:
+                self._init_decrypter()
 
         while read_offset > 0:
             read_len = min(self.MAX_SEEK_READ, read_offset)
@@ -1524,32 +1545,16 @@ def open(self, name, mode="r", pwd=None, *, force_zip64=False):
 
             # check for encrypted flag & handle password
             is_encrypted = zinfo.flag_bits & 0x1
-            zd = None
             if is_encrypted:
                 if not pwd:
                     pwd = self.pwd
                 if not pwd:
                     raise RuntimeError("File %r is encrypted, password "
                                        "required for extraction" % name)
+            else:
+                pwd = None
 
-                zd = _ZipDecrypter(pwd)
-                # The first 12 bytes in the cypher stream is an encryption header
-                #  used to strengthen the algorithm. The first 11 bytes are
-                #  completely random, while the 12th contains the MSB of the CRC,
-                #  or the MSB of the file time depending on the header type
-                #  and is used to check the correctness of the password.
-                header = zef_file.read(12)
-                h = zd(header[0:12])
-                if zinfo.flag_bits & 0x8:
-                    # compare against the file type from extended local headers
-                    check_byte = (zinfo._raw_time >> 8) & 0xff
-                else:
-                    # compare against the CRC otherwise
-                    check_byte = (zinfo.CRC >> 24) & 0xff
-                if h[11] != check_byte:
-                    raise RuntimeError("Bad password for file %r" % name)
-
-            return ZipExtFile(zef_file, mode, zinfo, zd, True)
+            return ZipExtFile(zef_file, mode, zinfo, pwd, True)
         except:
             zef_file.close()
             raise
diff --git a/Misc/NEWS.d/next/Library/2019-10-27-00-08-49.bpo-38334.pfLLmc.rst b/Misc/NEWS.d/next/Library/2019-10-27-00-08-49.bpo-38334.pfLLmc.rst
new file mode 100644
index 0000000000000..0d05d3f6e6c09
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-10-27-00-08-49.bpo-38334.pfLLmc.rst
@@ -0,0 +1 @@
+Fixed seeking backward on an encrypted :class:`zipfile.ZipExtFile`.


From webhook-mailer at python.org  Sun Oct 27 04:41:31 2019
From: webhook-mailer at python.org (Miss Skeleton (bot))
Date: Sun, 27 Oct 2019 08:41:31 -0000
Subject: [Python-checkins] bpo-38334: Fix seeking backward on an encrypted
 zipfile.ZipExtFile. (GH-16937)
Message-ID: <mailman.159.1572165691.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/ed2db3113d54a5f8af1b419a9f5fdf3642285c82
commit: ed2db3113d54a5f8af1b419a9f5fdf3642285c82
branch: 3.7
author: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-27T01:41:27-07:00
summary:

bpo-38334: Fix seeking backward on an encrypted zipfile.ZipExtFile. (GH-16937)


Test by Daniel Hillier.
(cherry picked from commit 5c32af7522d908e8c7da0243af37618433289cc5)

Co-authored-by: Serhiy Storchaka <storchaka at gmail.com>

files:
A Misc/NEWS.d/next/Library/2019-10-27-00-08-49.bpo-38334.pfLLmc.rst
M Lib/test/test_zipfile.py
M Lib/zipfile.py

diff --git a/Lib/test/test_zipfile.py b/Lib/test/test_zipfile.py
index ac8f64ce22f6b..523a2e1dadcb9 100644
--- a/Lib/test/test_zipfile.py
+++ b/Lib/test/test_zipfile.py
@@ -1874,6 +1874,44 @@ def test_unicode_password(self):
         self.assertRaises(TypeError, self.zip.open, "test.txt", pwd="python")
         self.assertRaises(TypeError, self.zip.extract, "test.txt", pwd="python")
 
+    def test_seek_tell(self):
+        self.zip.setpassword(b"python")
+        txt = self.plain
+        test_word = b'encryption'
+        bloc = txt.find(test_word)
+        bloc_len = len(test_word)
+        with self.zip.open("test.txt", "r") as fp:
+            fp.seek(bloc, os.SEEK_SET)
+            self.assertEqual(fp.tell(), bloc)
+            fp.seek(-bloc, os.SEEK_CUR)
+            self.assertEqual(fp.tell(), 0)
+            fp.seek(bloc, os.SEEK_CUR)
+            self.assertEqual(fp.tell(), bloc)
+            self.assertEqual(fp.read(bloc_len), txt[bloc:bloc+bloc_len])
+
+            # Make sure that the second read after seeking back beyond
+            # _readbuffer returns the same content (ie. rewind to the start of
+            # the file to read forward to the required position).
+            old_read_size = fp.MIN_READ_SIZE
+            fp.MIN_READ_SIZE = 1
+            fp._readbuffer = b''
+            fp._offset = 0
+            fp.seek(0, os.SEEK_SET)
+            self.assertEqual(fp.tell(), 0)
+            fp.seek(bloc, os.SEEK_CUR)
+            self.assertEqual(fp.read(bloc_len), txt[bloc:bloc+bloc_len])
+            fp.MIN_READ_SIZE = old_read_size
+
+            fp.seek(0, os.SEEK_END)
+            self.assertEqual(fp.tell(), len(txt))
+            fp.seek(0, os.SEEK_SET)
+            self.assertEqual(fp.tell(), 0)
+
+            # Read the file completely to definitely call any eof integrity
+            # checks (crc) and make sure they still pass.
+            fp.read()
+
+
 class AbstractTestsWithRandomBinaryFiles:
     @classmethod
     def setUpClass(cls):
diff --git a/Lib/zipfile.py b/Lib/zipfile.py
index 955d91fd0a2bf..22176126a9b90 100644
--- a/Lib/zipfile.py
+++ b/Lib/zipfile.py
@@ -784,10 +784,10 @@ class ZipExtFile(io.BufferedIOBase):
     # Chunk size to read during seek
     MAX_SEEK_READ = 1 << 24
 
-    def __init__(self, fileobj, mode, zipinfo, decrypter=None,
+    def __init__(self, fileobj, mode, zipinfo, pwd=None,
                  close_fileobj=False):
         self._fileobj = fileobj
-        self._decrypter = decrypter
+        self._pwd = pwd
         self._close_fileobj = close_fileobj
 
         self._compress_type = zipinfo.compress_type
@@ -802,11 +802,6 @@ def __init__(self, fileobj, mode, zipinfo, decrypter=None,
 
         self.newlines = None
 
-        # Adjust read size for encrypted files since the first 12 bytes
-        # are for the encryption/password information.
-        if self._decrypter is not None:
-            self._compress_left -= 12
-
         self.mode = mode
         self.name = zipinfo.filename
 
@@ -827,6 +822,30 @@ def __init__(self, fileobj, mode, zipinfo, decrypter=None,
         except AttributeError:
             pass
 
+        self._decrypter = None
+        if pwd:
+            if zipinfo.flag_bits & 0x8:
+                # compare against the file type from extended local headers
+                check_byte = (zipinfo._raw_time >> 8) & 0xff
+            else:
+                # compare against the CRC otherwise
+                check_byte = (zipinfo.CRC >> 24) & 0xff
+            h = self._init_decrypter()
+            if h != check_byte:
+                raise RuntimeError("Bad password for file %r" % zipinfo.orig_filename)
+
+
+    def _init_decrypter(self):
+        self._decrypter = _ZipDecrypter(self._pwd)
+        # The first 12 bytes in the cypher stream is an encryption header
+        #  used to strengthen the algorithm. The first 11 bytes are
+        #  completely random, while the 12th contains the MSB of the CRC,
+        #  or the MSB of the file time depending on the header type
+        #  and is used to check the correctness of the password.
+        header = self._fileobj.read(12)
+        self._compress_left -= 12
+        return self._decrypter(header)[11]
+
     def __repr__(self):
         result = ['<%s.%s' % (self.__class__.__module__,
                               self.__class__.__qualname__)]
@@ -1053,6 +1072,8 @@ def seek(self, offset, whence=0):
             self._decompressor = _get_decompressor(self._compress_type)
             self._eof = False
             read_offset = new_pos
+            if self._decrypter is not None:
+                self._init_decrypter()
 
         while read_offset > 0:
             read_len = min(self.MAX_SEEK_READ, read_offset)
@@ -1515,32 +1536,16 @@ def open(self, name, mode="r", pwd=None, *, force_zip64=False):
 
             # check for encrypted flag & handle password
             is_encrypted = zinfo.flag_bits & 0x1
-            zd = None
             if is_encrypted:
                 if not pwd:
                     pwd = self.pwd
                 if not pwd:
                     raise RuntimeError("File %r is encrypted, password "
                                        "required for extraction" % name)
+            else:
+                pwd = None
 
-                zd = _ZipDecrypter(pwd)
-                # The first 12 bytes in the cypher stream is an encryption header
-                #  used to strengthen the algorithm. The first 11 bytes are
-                #  completely random, while the 12th contains the MSB of the CRC,
-                #  or the MSB of the file time depending on the header type
-                #  and is used to check the correctness of the password.
-                header = zef_file.read(12)
-                h = zd(header[0:12])
-                if zinfo.flag_bits & 0x8:
-                    # compare against the file type from extended local headers
-                    check_byte = (zinfo._raw_time >> 8) & 0xff
-                else:
-                    # compare against the CRC otherwise
-                    check_byte = (zinfo.CRC >> 24) & 0xff
-                if h[11] != check_byte:
-                    raise RuntimeError("Bad password for file %r" % name)
-
-            return ZipExtFile(zef_file, mode, zinfo, zd, True)
+            return ZipExtFile(zef_file, mode, zinfo, pwd, True)
         except:
             zef_file.close()
             raise
diff --git a/Misc/NEWS.d/next/Library/2019-10-27-00-08-49.bpo-38334.pfLLmc.rst b/Misc/NEWS.d/next/Library/2019-10-27-00-08-49.bpo-38334.pfLLmc.rst
new file mode 100644
index 0000000000000..0d05d3f6e6c09
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-10-27-00-08-49.bpo-38334.pfLLmc.rst
@@ -0,0 +1 @@
+Fixed seeking backward on an encrypted :class:`zipfile.ZipExtFile`.


From webhook-mailer at python.org  Sun Oct 27 06:11:19 2019
From: webhook-mailer at python.org (Julien Palard)
Date: Sun, 27 Oct 2019 10:11:19 -0000
Subject: [Python-checkins] bpo-38592 Add pt-br switcher to Python Docs
 website (GH-16924) (GH-16953)
Message-ID: <mailman.160.1572171080.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/955f96f6aae7c1a54d88c3f7a51c2e142ac7e4d4
commit: 955f96f6aae7c1a54d88c3f7a51c2e142ac7e4d4
branch: 3.8
author: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Julien Palard <julien at palard.fr>
date: 2019-10-27T11:11:14+01:00
summary:

bpo-38592 Add pt-br switcher to Python Docs website (GH-16924) (GH-16953)


(cherry picked from commit 85c6f8c65cd4f7219522c1f304bdfff19f785e7a)
Co-authored-by: Marco Rougeth <marco at rougeth.com>

files:
A Misc/NEWS.d/next/Documentation/2019-10-26-13-19-07.bpo-38592.Y96BYO.rst
M Doc/tools/static/switchers.js

diff --git a/Doc/tools/static/switchers.js b/Doc/tools/static/switchers.js
index ef6f473452065..e1ef91a8dfc68 100644
--- a/Doc/tools/static/switchers.js
+++ b/Doc/tools/static/switchers.js
@@ -23,6 +23,7 @@
       'fr': 'French',
       'ja': 'Japanese',
       'ko': 'Korean',
+      'pt-br': 'Brazilian Portuguese',
       'zh-cn': 'Simplified Chinese',
   };
 
diff --git a/Misc/NEWS.d/next/Documentation/2019-10-26-13-19-07.bpo-38592.Y96BYO.rst b/Misc/NEWS.d/next/Documentation/2019-10-26-13-19-07.bpo-38592.Y96BYO.rst
new file mode 100644
index 0000000000000..3752d48a7cb4f
--- /dev/null
+++ b/Misc/NEWS.d/next/Documentation/2019-10-26-13-19-07.bpo-38592.Y96BYO.rst
@@ -0,0 +1 @@
+Add Brazilian Portuguese to the language switcher at Python Documentation website.
\ No newline at end of file


From webhook-mailer at python.org  Sun Oct 27 06:13:14 2019
From: webhook-mailer at python.org (Julien Palard)
Date: Sun, 27 Oct 2019 10:13:14 -0000
Subject: [Python-checkins] bpo-38592 Add pt-br switcher to Python Docs
 website (GH-16924) (GH-16954)
Message-ID: <mailman.161.1572171195.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/155a9dc129c129270763525cd3eb2704990e406d
commit: 155a9dc129c129270763525cd3eb2704990e406d
branch: 3.7
author: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Julien Palard <julien at palard.fr>
date: 2019-10-27T11:13:10+01:00
summary:

bpo-38592 Add pt-br switcher to Python Docs website (GH-16924) (GH-16954)

(cherry picked from commit 85c6f8c65cd4f7219522c1f304bdfff19f785e7a)

Co-authored-by: Marco Rougeth <marco at rougeth.com>

files:
A Misc/NEWS.d/next/Documentation/2019-10-26-13-19-07.bpo-38592.Y96BYO.rst
M Doc/tools/static/switchers.js

diff --git a/Doc/tools/static/switchers.js b/Doc/tools/static/switchers.js
index ef6f473452065..e1ef91a8dfc68 100644
--- a/Doc/tools/static/switchers.js
+++ b/Doc/tools/static/switchers.js
@@ -23,6 +23,7 @@
       'fr': 'French',
       'ja': 'Japanese',
       'ko': 'Korean',
+      'pt-br': 'Brazilian Portuguese',
       'zh-cn': 'Simplified Chinese',
   };
 
diff --git a/Misc/NEWS.d/next/Documentation/2019-10-26-13-19-07.bpo-38592.Y96BYO.rst b/Misc/NEWS.d/next/Documentation/2019-10-26-13-19-07.bpo-38592.Y96BYO.rst
new file mode 100644
index 0000000000000..3752d48a7cb4f
--- /dev/null
+++ b/Misc/NEWS.d/next/Documentation/2019-10-26-13-19-07.bpo-38592.Y96BYO.rst
@@ -0,0 +1 @@
+Add Brazilian Portuguese to the language switcher at Python Documentation website.
\ No newline at end of file


From webhook-mailer at python.org  Mon Oct 28 03:31:35 2019
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Mon, 28 Oct 2019 07:31:35 -0000
Subject: [Python-checkins] bpo-38602: Add fcntl.F_OFD_XXXX for fcntlmodule
 (GH-16956)
Message-ID: <mailman.162.1572247896.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/3bfc8e0fcc707d200c267ff05b052fd6a63c985a
commit: 3bfc8e0fcc707d200c267ff05b052fd6a63c985a
branch: master
author: Dong-hee Na <donghee.na92 at gmail.com>
committer: Serhiy Storchaka <storchaka at gmail.com>
date: 2019-10-28T09:31:15+02:00
summary:

bpo-38602: Add fcntl.F_OFD_XXXX for fcntlmodule (GH-16956)

files:
A Misc/NEWS.d/next/Library/2019-10-27-22-29-45.bpo-38602.7jvYFA.rst
M Doc/library/fcntl.rst
M Doc/whatsnew/3.9.rst
M Modules/fcntlmodule.c

diff --git a/Doc/library/fcntl.rst b/Doc/library/fcntl.rst
index da39c3187a1e3..5c172b836acca 100644
--- a/Doc/library/fcntl.rst
+++ b/Doc/library/fcntl.rst
@@ -36,6 +36,8 @@ descriptor.
 .. versionchanged:: 3.9
    On macOS, the fcntl module exposes the ``F_GETPATH`` constant, which obtains
    the path of a file from a file descriptor.
+   On Linux(>=3.15), the fcntl module exposes the ``F_OFD_GETLK``, ``F_OFD_SETLK``
+   and ``F_OFD_SETLKW`` constants, which working with open file description locks.
 
 The module defines the following functions:
 
diff --git a/Doc/whatsnew/3.9.rst b/Doc/whatsnew/3.9.rst
index b95b0999cb952..886c555c7b1d6 100644
--- a/Doc/whatsnew/3.9.rst
+++ b/Doc/whatsnew/3.9.rst
@@ -125,8 +125,15 @@ that schedules a shutdown for the default executor that waits on the
 :func:`asyncio.run` has been updated to use the new :term:`coroutine`.
 (Contributed by Kyle Stanley in :issue:`34037`.)
 
+fcntl
+-----
+
+Added constants :data:`~fcntl.F_OFD_GETLK`, :data:`~fcntl.F_OFD_SETLK`
+and :data:`~fcntl.F_OFD_SETLKW`.
+(Contributed by Dong-hee Na in :issue:`38602`.)
+
 os
-__
+--
 
 Added :data:`~os.CLD_KILLED` and :data:`~os.CLD_STOPPED` for :attr:`si_code`.
 (Contributed by Dong-hee Na in :issue:`38493`.)
diff --git a/Misc/NEWS.d/next/Library/2019-10-27-22-29-45.bpo-38602.7jvYFA.rst b/Misc/NEWS.d/next/Library/2019-10-27-22-29-45.bpo-38602.7jvYFA.rst
new file mode 100644
index 0000000000000..9b8229b4a024c
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-10-27-22-29-45.bpo-38602.7jvYFA.rst
@@ -0,0 +1,3 @@
+Added constants :data:`~fcntl.F_OFD_GETLK`, :data:`~fcntl.F_OFD_SETLK`
+and :data:`~fcntl.F_OFD_SETLKW` to the :mod:`fcntl` module.
+Patch by Dong-hee Na.
diff --git a/Modules/fcntlmodule.c b/Modules/fcntlmodule.c
index cfa1225684049..1e5b0f7faea6f 100644
--- a/Modules/fcntlmodule.c
+++ b/Modules/fcntlmodule.c
@@ -495,6 +495,15 @@ all_ins(PyObject* m)
 #ifdef F_SETLKW
     if (PyModule_AddIntMacro(m, F_SETLKW)) return -1;
 #endif
+#ifdef F_OFD_GETLK
+    if (PyModule_AddIntMacro(m, F_OFD_GETLK)) return -1;
+#endif
+#ifdef F_OFD_SETLK
+    if (PyModule_AddIntMacro(m, F_OFD_SETLK)) return -1;
+#endif
+#ifdef F_OFD_SETLKW
+    if (PyModule_AddIntMacro(m, F_OFD_SETLKW)) return -1;
+#endif
 #ifdef F_GETOWN
     if (PyModule_AddIntMacro(m, F_GETOWN)) return -1;
 #endif


From webhook-mailer at python.org  Mon Oct 28 10:40:22 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Mon, 28 Oct 2019 14:40:22 -0000
Subject: [Python-checkins] bpo-37330: open() no longer accept 'U' in file
 mode (GH-16959)
Message-ID: <mailman.163.1572273623.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/e471e72977c83664f13d041c78549140c86c92de
commit: e471e72977c83664f13d041c78549140c86c92de
branch: master
author: Victor Stinner <vstinner at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-28T15:40:08+01:00
summary:

bpo-37330: open() no longer accept 'U' in file mode (GH-16959)

open(), io.open(), codecs.open() and fileinput.FileInput no longer
accept "U" ("universal newline") in the file mode. This flag was
deprecated since Python 3.3.

files:
A Misc/NEWS.d/next/Core and Builtins/2019-06-18-17-53-06.bpo-37330.wAvHmz.rst
M Doc/library/codecs.rst
M Doc/library/fileinput.rst
M Doc/library/functions.rst
M Doc/whatsnew/3.9.rst
M Lib/_pyio.py
M Lib/fileinput.py
M Lib/imp.py
M Lib/test/test_codecs.py
M Lib/test/test_fileinput.py
M Lib/test/test_io.py
M Modules/_io/_iomodule.c
M Modules/_io/clinic/_iomodule.c.h

diff --git a/Doc/library/codecs.rst b/Doc/library/codecs.rst
index f071057293eec..ec6a0533033bd 100644
--- a/Doc/library/codecs.rst
+++ b/Doc/library/codecs.rst
@@ -197,6 +197,9 @@ wider range of codecs when working with binary files:
    *buffering* has the same meaning as for the built-in :func:`open` function.
    It defaults to -1 which means that the default buffer size will be used.
 
+   .. versionchanged:: 3.9
+      The ``'U'`` mode has been removed.
+
 
 .. function:: EncodedFile(file, data_encoding, file_encoding=None, errors='strict')
 
diff --git a/Doc/library/fileinput.rst b/Doc/library/fileinput.rst
index f5e5280a13639..8870c177fb1e1 100644
--- a/Doc/library/fileinput.rst
+++ b/Doc/library/fileinput.rst
@@ -148,8 +148,8 @@ available for subclassing as well:
    The sequence must be accessed in strictly sequential order; random access
    and :meth:`~io.TextIOBase.readline` cannot be mixed.
 
-   With *mode* you can specify which file mode will be passed to :func:`open`. It
-   must be one of ``'r'``, ``'rU'``, ``'U'`` and ``'rb'``.
+   With *mode* you can specify which file mode will be passed to :func:`open`.
+   It must be ``'r'`` or ``'rb'``.
 
    The *openhook*, when given, must be a function that takes two arguments,
    *filename* and *mode*, and returns an accordingly opened file-like object. You
@@ -166,15 +166,14 @@ available for subclassing as well:
    .. versionchanged:: 3.2
       Can be used as a context manager.
 
-   .. deprecated:: 3.4
-      The ``'rU'`` and ``'U'`` modes.
-
    .. deprecated:: 3.8
       Support for :meth:`__getitem__` method is deprecated.
 
    .. versionchanged:: 3.8
       The keyword parameter *mode* and *openhook* are now keyword-only.
 
+   .. versionchanged:: 3.9
+      The ``'rU'`` and ``'U'`` modes have been removed.
 
 
 **Optional in-place filtering:** if the keyword argument ``inplace=True`` is
diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst
index 28d9c7b99e6f5..acf922dcd1378 100644
--- a/Doc/library/functions.rst
+++ b/Doc/library/functions.rst
@@ -1085,12 +1085,6 @@ are always available.  They are listed here in alphabetical order.
    first decoded using a platform-dependent encoding or using the specified
    *encoding* if given.
 
-   There is an additional mode character permitted, ``'U'``, which no longer
-   has any effect, and is considered deprecated. It previously enabled
-   :term:`universal newlines` in text mode, which became the default behaviour
-   in Python 3.0. Refer to the documentation of the
-   :ref:`newline <open-newline-parameter>` parameter for further details.
-
    .. note::
 
       Python doesn't depend on the underlying operating system's notion of text
@@ -1247,10 +1241,6 @@ are always available.  They are listed here in alphabetical order.
 
          * The file is now non-inheritable.
 
-   .. deprecated-removed:: 3.4 4.0
-
-      The ``'U'`` mode.
-
    .. versionchanged::
       3.5
 
@@ -1266,6 +1256,10 @@ are always available.  They are listed here in alphabetical order.
          * On Windows, opening a console buffer may return a subclass of
            :class:`io.RawIOBase` other than :class:`io.FileIO`.
 
+   .. versionchanged:: 3.9
+      The ``'U'`` mode has been removed.
+
+
 .. function:: ord(c)
 
    Given a string representing one Unicode character, return an integer
diff --git a/Doc/whatsnew/3.9.rst b/Doc/whatsnew/3.9.rst
index 886c555c7b1d6..bca222955668d 100644
--- a/Doc/whatsnew/3.9.rst
+++ b/Doc/whatsnew/3.9.rst
@@ -284,6 +284,14 @@ that may require changes to your code.
 Changes in the Python API
 -------------------------
 
+* :func:`open`, :func:`io.open`, :func:`codecs.open` and
+  :class:`fileinput.FileInput` no longer accept ``'U'`` ("universal newline")
+  in the file mode. This flag was deprecated since Python 3.3. In Python 3, the
+  "universal newline" is used by default when a file is open in text mode.  The
+  :ref:`newline parameter <open-newline-parameter>` of :func:`open` controls
+  how universal newlines works.
+  (Contributed by Victor Stinner in :issue:`37330`.)
+
 * :func:`__import__` and :func:`importlib.util.resolve_name` now raise
   :exc:`ImportError` where it previously raised :exc:`ValueError`. Callers
   catching the specific exception type and supporting both Python 3.9 and
diff --git a/Lib/_pyio.py b/Lib/_pyio.py
index c1bdac7913193..e819b0a3cce85 100644
--- a/Lib/_pyio.py
+++ b/Lib/_pyio.py
@@ -71,7 +71,6 @@ def open(file, mode="r", buffering=-1, encoding=None, errors=None,
     'b'       binary mode
     't'       text mode (default)
     '+'       open a disk file for updating (reading and writing)
-    'U'       universal newline mode (deprecated)
     ========= ===============================================================
 
     The default mode is 'rt' (open for reading text). For binary random
@@ -87,10 +86,6 @@ def open(file, mode="r", buffering=-1, encoding=None, errors=None,
     returned as strings, the bytes having been first decoded using a
     platform-dependent encoding or using the specified encoding if given.
 
-    'U' mode is deprecated and will raise an exception in future versions
-    of Python.  It has no effect in Python 3.  Use newline to control
-    universal newlines mode.
-
     buffering is an optional integer used to set the buffering policy.
     Pass 0 to switch buffering off (only allowed in binary mode), 1 to select
     line buffering (only usable in text mode), and an integer > 1 to indicate
@@ -176,7 +171,7 @@ def open(file, mode="r", buffering=-1, encoding=None, errors=None,
     if errors is not None and not isinstance(errors, str):
         raise TypeError("invalid errors: %r" % errors)
     modes = set(mode)
-    if modes - set("axrwb+tU") or len(mode) > len(modes):
+    if modes - set("axrwb+t") or len(mode) > len(modes):
         raise ValueError("invalid mode: %r" % mode)
     creating = "x" in modes
     reading = "r" in modes
@@ -185,13 +180,6 @@ def open(file, mode="r", buffering=-1, encoding=None, errors=None,
     updating = "+" in modes
     text = "t" in modes
     binary = "b" in modes
-    if "U" in modes:
-        if creating or writing or appending or updating:
-            raise ValueError("mode U cannot be combined with 'x', 'w', 'a', or '+'")
-        import warnings
-        warnings.warn("'U' mode is deprecated",
-                      DeprecationWarning, 2)
-        reading = True
     if text and binary:
         raise ValueError("can't have text and binary mode at once")
     if creating + reading + writing + appending > 1:
diff --git a/Lib/fileinput.py b/Lib/fileinput.py
index c1b0ec9a8ed08..166c631689747 100644
--- a/Lib/fileinput.py
+++ b/Lib/fileinput.py
@@ -209,15 +209,10 @@ def __init__(self, files=None, inplace=False, backup="", *,
         self._isstdin = False
         self._backupfilename = None
         # restrict mode argument to reading modes
-        if mode not in ('r', 'rU', 'U', 'rb'):
-            raise ValueError("FileInput opening mode must be one of "
-                             "'r', 'rU', 'U' and 'rb'")
-        if 'U' in mode:
-            import warnings
-            warnings.warn("'U' mode is deprecated",
-                          DeprecationWarning, 2)
+        if mode not in ('r', 'rb'):
+            raise ValueError("FileInput opening mode must be 'r' or 'rb'")
         self._mode = mode
-        self._write_mode = mode.replace('r', 'w') if 'U' not in mode else 'w'
+        self._write_mode = mode.replace('r', 'w')
         if openhook:
             if inplace:
                 raise ValueError("FileInput cannot use an opening hook in inplace mode")
diff --git a/Lib/imp.py b/Lib/imp.py
index 31f8c766381ad..a6f6fc8434902 100644
--- a/Lib/imp.py
+++ b/Lib/imp.py
@@ -225,7 +225,7 @@ def load_module(name, file, filename, details):
 
     """
     suffix, mode, type_ = details
-    if mode and (not mode.startswith(('r', 'U')) or '+' in mode):
+    if mode and (not mode.startswith('r') or '+' in mode):
         raise ValueError('invalid file open mode {!r}'.format(mode))
     elif file is None and type_ in {PY_SOURCE, PY_COMPILED}:
         msg = 'file object required for import (type code {})'.format(type_)
diff --git a/Lib/test/test_codecs.py b/Lib/test/test_codecs.py
index b37525bf66043..e1638c111681c 100644
--- a/Lib/test/test_codecs.py
+++ b/Lib/test/test_codecs.py
@@ -712,11 +712,23 @@ def test_bug691291(self):
         self.addCleanup(support.unlink, support.TESTFN)
         with open(support.TESTFN, 'wb') as fp:
             fp.write(s)
-        with support.check_warnings(('', DeprecationWarning)):
-            reader = codecs.open(support.TESTFN, 'U', encoding=self.encoding)
-        with reader:
+        with codecs.open(support.TESTFN, 'r',
+                         encoding=self.encoding) as reader:
             self.assertEqual(reader.read(), s1)
 
+    def test_invalid_modes(self):
+        for mode in ('U', 'rU', 'r+U'):
+            with self.assertRaises(ValueError) as cm:
+                codecs.open(support.TESTFN, mode, encoding=self.encoding)
+            self.assertIn('invalid mode', str(cm.exception))
+
+        for mode in ('rt', 'wt', 'at', 'r+t'):
+            with self.assertRaises(ValueError) as cm:
+                codecs.open(support.TESTFN, mode, encoding=self.encoding)
+            self.assertIn("can't have text and binary mode at once",
+                          str(cm.exception))
+
+
 class UTF16LETest(ReadTest, unittest.TestCase):
     encoding = "utf-16-le"
     ill_formed_sequence = b"\x80\xdc"
diff --git a/Lib/test/test_fileinput.py b/Lib/test/test_fileinput.py
index 014f19e6cbdb1..819557d5e8686 100644
--- a/Lib/test/test_fileinput.py
+++ b/Lib/test/test_fileinput.py
@@ -226,19 +226,11 @@ def test_fileno(self):
         self.assertEqual(fi.fileno(), -1)
 
     def test_opening_mode(self):
-        try:
-            # invalid mode, should raise ValueError
-            fi = FileInput(mode="w")
-            self.fail("FileInput should reject invalid mode argument")
-        except ValueError:
-            pass
-        # try opening in universal newline mode
-        t1 = self.writeTmp(b"A\nB\r\nC\rD", mode="wb")
-        with check_warnings(('', DeprecationWarning)):
-            fi = FileInput(files=t1, mode="U")
-        with check_warnings(('', DeprecationWarning)):
-            lines = list(fi)
-        self.assertEqual(lines, ["A\n", "B\n", "C\n", "D"])
+        # invalid modes
+        for mode in ('w', 'rU', 'U'):
+            with self.subTest(mode=mode):
+                with self.assertRaises(ValueError):
+                    FileInput(mode=mode)
 
     def test_stdin_binary_mode(self):
         with mock.patch('sys.stdin') as m_stdin:
@@ -985,10 +977,6 @@ def check(mode, expected_lines):
             self.assertEqual(lines, expected_lines)
 
         check('r', ['A\n', 'B\n', 'C\n', 'D\u20ac'])
-        with self.assertWarns(DeprecationWarning):
-            check('rU', ['A\n', 'B\n', 'C\n', 'D\u20ac'])
-        with self.assertWarns(DeprecationWarning):
-            check('U', ['A\n', 'B\n', 'C\n', 'D\u20ac'])
         with self.assertRaises(ValueError):
             check('rb', ['A\n', 'B\r\n', 'C\r', 'D\u20ac'])
 
diff --git a/Lib/test/test_io.py b/Lib/test/test_io.py
index 1fe1cba5167fc..ad22dfe9b2910 100644
--- a/Lib/test/test_io.py
+++ b/Lib/test/test_io.py
@@ -3886,16 +3886,6 @@ def test_attributes(self):
         self.assertEqual(f.mode, "wb")
         f.close()
 
-        with support.check_warnings(('', DeprecationWarning)):
-            f = self.open(support.TESTFN, "U")
-        self.assertEqual(f.name,            support.TESTFN)
-        self.assertEqual(f.buffer.name,     support.TESTFN)
-        self.assertEqual(f.buffer.raw.name, support.TESTFN)
-        self.assertEqual(f.mode,            "U")
-        self.assertEqual(f.buffer.mode,     "rb")
-        self.assertEqual(f.buffer.raw.mode, "rb")
-        f.close()
-
         f = self.open(support.TESTFN, "w+")
         self.assertEqual(f.mode,            "w+")
         self.assertEqual(f.buffer.mode,     "rb+") # Does it really matter?
@@ -3909,6 +3899,13 @@ def test_attributes(self):
         f.close()
         g.close()
 
+    def test_removed_u_mode(self):
+        # "U" mode has been removed in Python 3.9
+        for mode in ("U", "rU", "r+U"):
+            with self.assertRaises(ValueError) as cm:
+                self.open(support.TESTFN, mode)
+            self.assertIn('invalid mode', str(cm.exception))
+
     def test_io_after_close(self):
         for kwargs in [
                 {"mode": "w"},
diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-06-18-17-53-06.bpo-37330.wAvHmz.rst b/Misc/NEWS.d/next/Core and Builtins/2019-06-18-17-53-06.bpo-37330.wAvHmz.rst
new file mode 100644
index 0000000000000..aec6d018600a5
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2019-06-18-17-53-06.bpo-37330.wAvHmz.rst	
@@ -0,0 +1,3 @@
+:func:`open`, :func:`io.open`, :func:`codecs.open` and
+:class:`fileinput.FileInput` no longer accept ``'U'`` ("universal newline") in
+the file mode. This flag was deprecated since Python 3.3.
diff --git a/Modules/_io/_iomodule.c b/Modules/_io/_iomodule.c
index b4cc894004027..5932363f3af35 100644
--- a/Modules/_io/_iomodule.c
+++ b/Modules/_io/_iomodule.c
@@ -138,7 +138,6 @@ Character Meaning
 'b'       binary mode
 't'       text mode (default)
 '+'       open a disk file for updating (reading and writing)
-'U'       universal newline mode (deprecated)
 ========= ===============================================================
 
 The default mode is 'rt' (open for reading text). For binary random
@@ -154,10 +153,6 @@ bytes objects without any decoding. In text mode (the default, or when
 returned as strings, the bytes having been first decoded using a
 platform-dependent encoding or using the specified encoding if given.
 
-'U' mode is deprecated and will raise an exception in future versions
-of Python.  It has no effect in Python 3.  Use newline to control
-universal newlines mode.
-
 buffering is an optional integer used to set the buffering policy.
 Pass 0 to switch buffering off (only allowed in binary mode), 1 to select
 line buffering (only usable in text mode), and an integer > 1 to indicate
@@ -233,12 +228,12 @@ static PyObject *
 _io_open_impl(PyObject *module, PyObject *file, const char *mode,
               int buffering, const char *encoding, const char *errors,
               const char *newline, int closefd, PyObject *opener)
-/*[clinic end generated code: output=aefafc4ce2b46dc0 input=7295902222e6b311]*/
+/*[clinic end generated code: output=aefafc4ce2b46dc0 input=1543f4511d2356a5]*/
 {
     unsigned i;
 
     int creating = 0, reading = 0, writing = 0, appending = 0, updating = 0;
-    int text = 0, binary = 0, universal = 0;
+    int text = 0, binary = 0;
 
     char rawmode[6], *m;
     int line_buffering, is_number;
@@ -296,10 +291,6 @@ _io_open_impl(PyObject *module, PyObject *file, const char *mode,
         case 'b':
             binary = 1;
             break;
-        case 'U':
-            universal = 1;
-            reading = 1;
-            break;
         default:
             goto invalid_mode;
         }
@@ -322,18 +313,6 @@ _io_open_impl(PyObject *module, PyObject *file, const char *mode,
     *m = '\0';
 
     /* Parameters validation */
-    if (universal) {
-        if (creating || writing || appending || updating) {
-            PyErr_SetString(PyExc_ValueError,
-                            "mode U cannot be combined with 'x', 'w', 'a', or '+'");
-            goto error;
-        }
-        if (PyErr_WarnEx(PyExc_DeprecationWarning,
-                         "'U' mode is deprecated", 1) < 0)
-            goto error;
-        reading = 1;
-    }
-
     if (text && binary) {
         PyErr_SetString(PyExc_ValueError,
                         "can't have text and binary mode at once");
diff --git a/Modules/_io/clinic/_iomodule.c.h b/Modules/_io/clinic/_iomodule.c.h
index 1a9651d340813..c1f518ff60273 100644
--- a/Modules/_io/clinic/_iomodule.c.h
+++ b/Modules/_io/clinic/_iomodule.c.h
@@ -36,7 +36,6 @@ PyDoc_STRVAR(_io_open__doc__,
 "\'b\'       binary mode\n"
 "\'t\'       text mode (default)\n"
 "\'+\'       open a disk file for updating (reading and writing)\n"
-"\'U\'       universal newline mode (deprecated)\n"
 "========= ===============================================================\n"
 "\n"
 "The default mode is \'rt\' (open for reading text). For binary random\n"
@@ -52,10 +51,6 @@ PyDoc_STRVAR(_io_open__doc__,
 "returned as strings, the bytes having been first decoded using a\n"
 "platform-dependent encoding or using the specified encoding if given.\n"
 "\n"
-"\'U\' mode is deprecated and will raise an exception in future versions\n"
-"of Python.  It has no effect in Python 3.  Use newline to control\n"
-"universal newlines mode.\n"
-"\n"
 "buffering is an optional integer used to set the buffering policy.\n"
 "Pass 0 to switch buffering off (only allowed in binary mode), 1 to select\n"
 "line buffering (only usable in text mode), and an integer > 1 to indicate\n"
@@ -323,4 +318,4 @@ _io_open_code(PyObject *module, PyObject *const *args, Py_ssize_t nargs, PyObjec
 exit:
     return return_value;
 }
-/*[clinic end generated code: output=3df6bc6d91697545 input=a9049054013a1b77]*/
+/*[clinic end generated code: output=680e4b488c7da8a1 input=a9049054013a1b77]*/


From webhook-mailer at python.org  Mon Oct 28 13:03:38 2019
From: webhook-mailer at python.org (Steve Dower)
Date: Mon, 28 Oct 2019 17:03:38 -0000
Subject: [Python-checkins] bpo-38519: Internal include files missing on
 Windows (GH-16921)
Message-ID: <mailman.164.1572282218.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/edb172a87296d9359593a23cd9a09f5867ea1f0e
commit: edb172a87296d9359593a23cd9a09f5867ea1f0e
branch: master
author: Zackery Spytz <zspytz at gmail.com>
committer: Steve Dower <steve.dower at python.org>
date: 2019-10-28T10:03:27-07:00
summary:

bpo-38519: Internal include files missing on Windows (GH-16921)

files:
A Misc/NEWS.d/next/Windows/2019-10-28-05-01-29.bpo-38519.dCkY66.rst
M PC/layout/main.py

diff --git a/Misc/NEWS.d/next/Windows/2019-10-28-05-01-29.bpo-38519.dCkY66.rst b/Misc/NEWS.d/next/Windows/2019-10-28-05-01-29.bpo-38519.dCkY66.rst
new file mode 100644
index 0000000000000..56d8bb2b34c10
--- /dev/null
+++ b/Misc/NEWS.d/next/Windows/2019-10-28-05-01-29.bpo-38519.dCkY66.rst
@@ -0,0 +1,2 @@
+Restores the internal C headers that were missing from the nuget.org and
+Microsoft Store packages.
diff --git a/PC/layout/main.py b/PC/layout/main.py
index e59858196249a..3ca49d08e8f98 100644
--- a/PC/layout/main.py
+++ b/PC/layout/main.py
@@ -216,12 +216,7 @@ def _c(d):
 
     if ns.include_dev:
 
-        def _c(d):
-            if d.is_dir():
-                return d.name != "internal"
-            return True
-
-        for dest, src in rglob(ns.source / "Include", "**/*.h", _c):
+        for dest, src in rglob(ns.source / "Include", "**/*.h"):
             yield "include/{}".format(dest), src
         src = ns.source / "PC" / "pyconfig.h"
         yield "include/pyconfig.h", src


From webhook-mailer at python.org  Mon Oct 28 13:23:14 2019
From: webhook-mailer at python.org (Miss Skeleton (bot))
Date: Mon, 28 Oct 2019 17:23:14 -0000
Subject: [Python-checkins] bpo-38519: Internal include files missing on
 Windows (GH-16921)
Message-ID: <mailman.165.1572283395.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/07eee640cb3a4eaf1df942bc31a891e7517de763
commit: 07eee640cb3a4eaf1df942bc31a891e7517de763
branch: 3.8
author: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-28T10:23:07-07:00
summary:

bpo-38519: Internal include files missing on Windows (GH-16921)

(cherry picked from commit edb172a87296d9359593a23cd9a09f5867ea1f0e)

Co-authored-by: Zackery Spytz <zspytz at gmail.com>

files:
A Misc/NEWS.d/next/Windows/2019-10-28-05-01-29.bpo-38519.dCkY66.rst
M PC/layout/main.py

diff --git a/Misc/NEWS.d/next/Windows/2019-10-28-05-01-29.bpo-38519.dCkY66.rst b/Misc/NEWS.d/next/Windows/2019-10-28-05-01-29.bpo-38519.dCkY66.rst
new file mode 100644
index 0000000000000..56d8bb2b34c10
--- /dev/null
+++ b/Misc/NEWS.d/next/Windows/2019-10-28-05-01-29.bpo-38519.dCkY66.rst
@@ -0,0 +1,2 @@
+Restores the internal C headers that were missing from the nuget.org and
+Microsoft Store packages.
diff --git a/PC/layout/main.py b/PC/layout/main.py
index e59858196249a..3ca49d08e8f98 100644
--- a/PC/layout/main.py
+++ b/PC/layout/main.py
@@ -216,12 +216,7 @@ def _c(d):
 
     if ns.include_dev:
 
-        def _c(d):
-            if d.is_dir():
-                return d.name != "internal"
-            return True
-
-        for dest, src in rglob(ns.source / "Include", "**/*.h", _c):
+        for dest, src in rglob(ns.source / "Include", "**/*.h"):
             yield "include/{}".format(dest), src
         src = ns.source / "PC" / "pyconfig.h"
         yield "include/pyconfig.h", src


From webhook-mailer at python.org  Mon Oct 28 13:53:55 2019
From: webhook-mailer at python.org (Steve Dower)
Date: Mon, 28 Oct 2019 17:53:55 -0000
Subject: [Python-checkins] bpo-38534: Replace wrong KB number references
 (GH-16955)
Message-ID: <mailman.166.1572285236.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/794616f837c254c68d8384ab48fb78123a3c8a8b
commit: 794616f837c254c68d8384ab48fb78123a3c8a8b
branch: master
author: benedwards14 <53377856+benedwards14 at users.noreply.github.com>
committer: Steve Dower <steve.dower at python.org>
date: 2019-10-28T10:53:51-07:00
summary:

bpo-38534: Replace wrong KB number references (GH-16955)

files:
M Doc/whatsnew/3.8.rst
M Tools/msi/bundle/bootstrap/PythonBootstrapperApplication.cpp

diff --git a/Doc/whatsnew/3.8.rst b/Doc/whatsnew/3.8.rst
index 80ab565762c60..a3c5f5eab8cab 100644
--- a/Doc/whatsnew/3.8.rst
+++ b/Doc/whatsnew/3.8.rst
@@ -1970,7 +1970,7 @@ Changes in the Python API
   resolution. If your application relies on these mechanisms, you should check
   for :func:`~os.add_dll_directory` and if it exists, use it to add your DLLs
   directory while loading your library. Note that Windows 7 users will need to
-  ensure that Windows Update KB2533625 has been installed (this is also verified
+  ensure that Windows Update KB2533623 has been installed (this is also verified
   by the installer).
   (Contributed by Steve Dower in :issue:`36085`.)
 
diff --git a/Tools/msi/bundle/bootstrap/PythonBootstrapperApplication.cpp b/Tools/msi/bundle/bootstrap/PythonBootstrapperApplication.cpp
index 570798de1aafa..fbdf7b6424d61 100644
--- a/Tools/msi/bundle/bootstrap/PythonBootstrapperApplication.cpp
+++ b/Tools/msi/bundle/bootstrap/PythonBootstrapperApplication.cpp
@@ -3013,8 +3013,8 @@ class PythonBootstrapperApplication : public CBalBaseBootstrapperApplication {
             } else if (IsWindowsVersionOrGreater(6, 1, 1)) {
                 HMODULE hKernel32 = GetModuleHandleW(L"kernel32");
                 if (hKernel32 && !GetProcAddress(hKernel32, "AddDllDirectory")) {
-                    BalLog(BOOTSTRAPPER_LOG_LEVEL_ERROR, "Detected Windows Server 2008 R2 without KB2533625");
-                    BalLog(BOOTSTRAPPER_LOG_LEVEL_ERROR, "KB2533625 update is required to continue.");
+                    BalLog(BOOTSTRAPPER_LOG_LEVEL_ERROR, "Detected Windows Server 2008 R2 without KB2533623");
+                    BalLog(BOOTSTRAPPER_LOG_LEVEL_ERROR, "KB2533623 update is required to continue.");
                     /* The "MissingSP1" error also specifies updates are required */
                     LocGetString(_wixLoc, L"#(loc.FailureWS2K8R2MissingSP1)", &pLocString);
                 } else {
@@ -3044,8 +3044,8 @@ class PythonBootstrapperApplication : public CBalBaseBootstrapperApplication {
             } else if (IsWindows7SP1OrGreater()) {
                 HMODULE hKernel32 = GetModuleHandleW(L"kernel32");
                 if (hKernel32 && !GetProcAddress(hKernel32, "AddDllDirectory")) {
-                    BalLog(BOOTSTRAPPER_LOG_LEVEL_ERROR, "Detected Windows 7 SP1 without KB2533625");
-                    BalLog(BOOTSTRAPPER_LOG_LEVEL_ERROR, "KB2533625 update is required to continue.");
+                    BalLog(BOOTSTRAPPER_LOG_LEVEL_ERROR, "Detected Windows 7 SP1 without KB2533623");
+                    BalLog(BOOTSTRAPPER_LOG_LEVEL_ERROR, "KB2533623 update is required to continue.");
                     /* The "MissingSP1" error also specifies updates are required */
                     LocGetString(_wixLoc, L"#(loc.FailureWin7MissingSP1)", &pLocString);
                 } else {


From webhook-mailer at python.org  Mon Oct 28 14:15:27 2019
From: webhook-mailer at python.org (Miss Skeleton (bot))
Date: Mon, 28 Oct 2019 18:15:27 -0000
Subject: [Python-checkins] bpo-38534: Replace wrong KB number references
 (GH-16955)
Message-ID: <mailman.167.1572286529.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/a4ee7abd2056c4956df45125fcf451a531247017
commit: a4ee7abd2056c4956df45125fcf451a531247017
branch: 3.8
author: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-28T11:15:24-07:00
summary:

bpo-38534: Replace wrong KB number references (GH-16955)

(cherry picked from commit 794616f837c254c68d8384ab48fb78123a3c8a8b)

Co-authored-by: benedwards14 <53377856+benedwards14 at users.noreply.github.com>

files:
M Doc/whatsnew/3.8.rst
M Tools/msi/bundle/bootstrap/PythonBootstrapperApplication.cpp

diff --git a/Doc/whatsnew/3.8.rst b/Doc/whatsnew/3.8.rst
index 1d547a1fbf83a..21a6a00d8aa0d 100644
--- a/Doc/whatsnew/3.8.rst
+++ b/Doc/whatsnew/3.8.rst
@@ -1972,7 +1972,7 @@ Changes in the Python API
   resolution. If your application relies on these mechanisms, you should check
   for :func:`~os.add_dll_directory` and if it exists, use it to add your DLLs
   directory while loading your library. Note that Windows 7 users will need to
-  ensure that Windows Update KB2533625 has been installed (this is also verified
+  ensure that Windows Update KB2533623 has been installed (this is also verified
   by the installer).
   (Contributed by Steve Dower in :issue:`36085`.)
 
diff --git a/Tools/msi/bundle/bootstrap/PythonBootstrapperApplication.cpp b/Tools/msi/bundle/bootstrap/PythonBootstrapperApplication.cpp
index 570798de1aafa..fbdf7b6424d61 100644
--- a/Tools/msi/bundle/bootstrap/PythonBootstrapperApplication.cpp
+++ b/Tools/msi/bundle/bootstrap/PythonBootstrapperApplication.cpp
@@ -3013,8 +3013,8 @@ class PythonBootstrapperApplication : public CBalBaseBootstrapperApplication {
             } else if (IsWindowsVersionOrGreater(6, 1, 1)) {
                 HMODULE hKernel32 = GetModuleHandleW(L"kernel32");
                 if (hKernel32 && !GetProcAddress(hKernel32, "AddDllDirectory")) {
-                    BalLog(BOOTSTRAPPER_LOG_LEVEL_ERROR, "Detected Windows Server 2008 R2 without KB2533625");
-                    BalLog(BOOTSTRAPPER_LOG_LEVEL_ERROR, "KB2533625 update is required to continue.");
+                    BalLog(BOOTSTRAPPER_LOG_LEVEL_ERROR, "Detected Windows Server 2008 R2 without KB2533623");
+                    BalLog(BOOTSTRAPPER_LOG_LEVEL_ERROR, "KB2533623 update is required to continue.");
                     /* The "MissingSP1" error also specifies updates are required */
                     LocGetString(_wixLoc, L"#(loc.FailureWS2K8R2MissingSP1)", &pLocString);
                 } else {
@@ -3044,8 +3044,8 @@ class PythonBootstrapperApplication : public CBalBaseBootstrapperApplication {
             } else if (IsWindows7SP1OrGreater()) {
                 HMODULE hKernel32 = GetModuleHandleW(L"kernel32");
                 if (hKernel32 && !GetProcAddress(hKernel32, "AddDllDirectory")) {
-                    BalLog(BOOTSTRAPPER_LOG_LEVEL_ERROR, "Detected Windows 7 SP1 without KB2533625");
-                    BalLog(BOOTSTRAPPER_LOG_LEVEL_ERROR, "KB2533625 update is required to continue.");
+                    BalLog(BOOTSTRAPPER_LOG_LEVEL_ERROR, "Detected Windows 7 SP1 without KB2533623");
+                    BalLog(BOOTSTRAPPER_LOG_LEVEL_ERROR, "KB2533623 update is required to continue.");
                     /* The "MissingSP1" error also specifies updates are required */
                     LocGetString(_wixLoc, L"#(loc.FailureWin7MissingSP1)", &pLocString);
                 } else {


From webhook-mailer at python.org  Mon Oct 28 17:34:32 2019
From: webhook-mailer at python.org (Steve Dower)
Date: Mon, 28 Oct 2019 21:34:32 -0000
Subject: [Python-checkins] bpo-38589: Fixes HTML Help shortcut when Windows
 is not installed to C drive (GH-16968)
Message-ID: <mailman.168.1572298473.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/0ac6137dd3d1e2c8f9558153ad63021f57e05e73
commit: 0ac6137dd3d1e2c8f9558153ad63021f57e05e73
branch: master
author: Steve Dower <steve.dower at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-28T14:34:25-07:00
summary:

bpo-38589: Fixes HTML Help shortcut when Windows is not installed to C drive (GH-16968)

files:
A Misc/NEWS.d/next/Windows/2019-10-28-10-48-16.bpo-38589.V69Q1a.rst
M Tools/msi/doc/doc.wxs

diff --git a/Misc/NEWS.d/next/Windows/2019-10-28-10-48-16.bpo-38589.V69Q1a.rst b/Misc/NEWS.d/next/Windows/2019-10-28-10-48-16.bpo-38589.V69Q1a.rst
new file mode 100644
index 0000000000000..e864290723e07
--- /dev/null
+++ b/Misc/NEWS.d/next/Windows/2019-10-28-10-48-16.bpo-38589.V69Q1a.rst
@@ -0,0 +1 @@
+Fixes HTML Help shortcut when Windows is not installed to C drive
diff --git a/Tools/msi/doc/doc.wxs b/Tools/msi/doc/doc.wxs
index 306fb11dc6619..cd1a68cc2601a 100644
--- a/Tools/msi/doc/doc.wxs
+++ b/Tools/msi/doc/doc.wxs
@@ -8,7 +8,7 @@
         <PropertyRef Id="REGISTRYKEY" />
         
         <Property Id="HHExe" Value="C:\Windows\hh.exe" />
-        <CustomAction Id="SetHHExe" Property="HHCExe" Value='[WindowsFolder]\hh.exe' Execute="immediate" />
+        <CustomAction Id="SetHHExe" Property="HHExe" Value='[WindowsFolder]\hh.exe' Execute="immediate" />
         <InstallExecuteSequence>
             <Custom Action="SetHHExe" Before="CostFinalize">1</Custom>
         </InstallExecuteSequence>


From webhook-mailer at python.org  Mon Oct 28 17:35:35 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Mon, 28 Oct 2019 21:35:35 -0000
Subject: [Python-checkins] Remove code commented for more than 10 years
 (GH-16965)
Message-ID: <mailman.169.1572298536.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/ae7aa4277437566b55aa6941d8d50cce35848ace
commit: ae7aa4277437566b55aa6941d8d50cce35848ace
branch: master
author: Victor Stinner <vstinner at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-28T22:35:31+01:00
summary:

Remove code commented for more than 10 years (GH-16965)

test_urllib commented since 2007:

commit d9880d07fc032884e8bd5cf3ac636d3a3778d5f7
Author: Facundo Batista <facundobatista at gmail.com>
Date:   Fri May 25 04:20:22 2007 +0000

    Commenting out the tests until find out who can test them in
    one of the problematic enviroments.

pynche code commented since 1998 and 2001:

commit ef30092207e684bd3e74fe3d6172b97327a0a089
Author: Barry Warsaw <barry at python.org>
Date:   Tue Dec 15 01:04:38 1998 +0000

    Added most of the mechanism to change the strips from color variations
    to color constants (i.e. red constant, green constant, blue
    constant).  But I haven't hooked this up yet because the UI gets more
    crowded and the arrows don't reflect the correct values.

    Added "Go to Black" and "Go to White" buttons.

commit 741eae0b3129c123ed4319d8d9e5b877b09aefa8
Author: Barry Warsaw <barry at python.org>
Date:   Wed Apr 18 03:51:55 2001 +0000

    StripWidget.__init__(), update_yourself(): Removed some unused local
    variables reported by PyChecker.

    __togglegentype(): PyChecker accurately reported that the variable
    __gentypevar was unused -- actually this whole method is currently
    unused so comment it out.

files:
M Lib/test/test_urllib.py
M Tools/pynche/StripViewer.py

diff --git a/Lib/test/test_urllib.py b/Lib/test/test_urllib.py
index 3f59c66084593..95c4ecc4dcf29 100644
--- a/Lib/test/test_urllib.py
+++ b/Lib/test/test_urllib.py
@@ -1544,85 +1544,6 @@ def open_local_file(self, url):
             self.assertRaises(OSError, DummyURLopener().retrieve, url)
 
 
-# Just commented them out.
-# Can't really tell why keep failing in windows and sparc.
-# Everywhere else they work ok, but on those machines, sometimes
-# fail in one of the tests, sometimes in other. I have a linux, and
-# the tests go ok.
-# If anybody has one of the problematic environments, please help!
-# .   Facundo
-#
-# def server(evt):
-#     import socket, time
-#     serv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
-#     serv.settimeout(3)
-#     serv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
-#     serv.bind(("", 9093))
-#     serv.listen()
-#     try:
-#         conn, addr = serv.accept()
-#         conn.send("1 Hola mundo\n")
-#         cantdata = 0
-#         while cantdata < 13:
-#             data = conn.recv(13-cantdata)
-#             cantdata += len(data)
-#             time.sleep(.3)
-#         conn.send("2 No more lines\n")
-#         conn.close()
-#     except socket.timeout:
-#         pass
-#     finally:
-#         serv.close()
-#         evt.set()
-#
-# class FTPWrapperTests(unittest.TestCase):
-#
-#     def setUp(self):
-#         import ftplib, time, threading
-#         ftplib.FTP.port = 9093
-#         self.evt = threading.Event()
-#         threading.Thread(target=server, args=(self.evt,)).start()
-#         time.sleep(.1)
-#
-#     def tearDown(self):
-#         self.evt.wait()
-#
-#     def testBasic(self):
-#         # connects
-#         ftp = urllib.ftpwrapper("myuser", "mypass", "localhost", 9093, [])
-#         ftp.close()
-#
-#     def testTimeoutNone(self):
-#         # global default timeout is ignored
-#         import socket
-#         self.assertIsNone(socket.getdefaulttimeout())
-#         socket.setdefaulttimeout(30)
-#         try:
-#             ftp = urllib.ftpwrapper("myuser", "mypass", "localhost", 9093, [])
-#         finally:
-#             socket.setdefaulttimeout(None)
-#         self.assertEqual(ftp.ftp.sock.gettimeout(), 30)
-#         ftp.close()
-#
-#     def testTimeoutDefault(self):
-#         # global default timeout is used
-#         import socket
-#         self.assertIsNone(socket.getdefaulttimeout())
-#         socket.setdefaulttimeout(30)
-#         try:
-#             ftp = urllib.ftpwrapper("myuser", "mypass", "localhost", 9093, [])
-#         finally:
-#             socket.setdefaulttimeout(None)
-#         self.assertEqual(ftp.ftp.sock.gettimeout(), 30)
-#         ftp.close()
-#
-#     def testTimeoutValue(self):
-#         ftp = urllib.ftpwrapper("myuser", "mypass", "localhost", 9093, [],
-#                                 timeout=30)
-#         self.assertEqual(ftp.ftp.sock.gettimeout(), 30)
-#         ftp.close()
-
-
 class RequestTests(unittest.TestCase):
     """Unit tests for urllib.request.Request."""
 
diff --git a/Tools/pynche/StripViewer.py b/Tools/pynche/StripViewer.py
index 2b0ede39eeeff..6914ca94b2bd3 100644
--- a/Tools/pynche/StripViewer.py
+++ b/Tools/pynche/StripViewer.py
@@ -371,22 +371,6 @@ def __init__(self, switchboard, master=None):
                              command=self.__togglehex)
         hexbtn.grid(row=1, column=1, sticky=W)
 
-        # XXX: ignore this feature for now; it doesn't work quite right yet
-
-##        gentypevar = self.__gentypevar = IntVar()
-##        self.__variations = Radiobutton(frame,
-##                                        text='Variations',
-##                                        variable=gentypevar,
-##                                        value=0,
-##                                        command=self.__togglegentype)
-##        self.__variations.grid(row=0, column=1, sticky=W)
-##        self.__constants = Radiobutton(frame,
-##                                       text='Constants',
-##                                       variable=gentypevar,
-##                                       value=1,
-##                                       command=self.__togglegentype)
-##        self.__constants.grid(row=1, column=1, sticky=W)
-
         # create the white button
         whitebtn = Button(frame2,
                           text='White',
@@ -402,26 +386,6 @@ def __togglehex(self, event=None):
         red, green, blue = self.__sb.current_rgb()
         self.update_yourself(red, green, blue)
 
-##    def __togglegentype(self, event=None):
-##        which = self.__gentypevar.get()
-##        if which == 0:
-##            self.__reds.set(label='Red Variations',
-##                            generator=constant_cyan_generator)
-##            self.__greens.set(label='Green Variations',
-##                              generator=constant_magenta_generator)
-##            self.__blues.set(label='Blue Variations',
-##                             generator=constant_yellow_generator)
-##        elif which == 1:
-##            self.__reds.set(label='Red Constant',
-##                            generator=constant_red_generator)
-##            self.__greens.set(label='Green Constant',
-##                              generator=constant_green_generator)
-##            self.__blues.set(label='Blue Constant',
-##                             generator=constant_blue_generator)
-##        else:
-##            assert 0
-##        self.__sb.update_views_current()
-
     def __toblack(self, event=None):
         self.__sb.update_views(0, 0, 0)
 


From webhook-mailer at python.org  Mon Oct 28 17:52:04 2019
From: webhook-mailer at python.org (Miss Skeleton (bot))
Date: Mon, 28 Oct 2019 21:52:04 -0000
Subject: [Python-checkins] bpo-38589: Fixes HTML Help shortcut when Windows
 is not installed to C drive (GH-16968)
Message-ID: <mailman.170.1572299526.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/379b55b526a04e6db40c7070fd70f0d7d37bba64
commit: 379b55b526a04e6db40c7070fd70f0d7d37bba64
branch: 3.7
author: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-28T14:52:00-07:00
summary:

bpo-38589: Fixes HTML Help shortcut when Windows is not installed to C drive (GH-16968)

(cherry picked from commit 0ac6137dd3d1e2c8f9558153ad63021f57e05e73)

Co-authored-by: Steve Dower <steve.dower at python.org>

files:
A Misc/NEWS.d/next/Windows/2019-10-28-10-48-16.bpo-38589.V69Q1a.rst
M Tools/msi/doc/doc.wxs

diff --git a/Misc/NEWS.d/next/Windows/2019-10-28-10-48-16.bpo-38589.V69Q1a.rst b/Misc/NEWS.d/next/Windows/2019-10-28-10-48-16.bpo-38589.V69Q1a.rst
new file mode 100644
index 0000000000000..e864290723e07
--- /dev/null
+++ b/Misc/NEWS.d/next/Windows/2019-10-28-10-48-16.bpo-38589.V69Q1a.rst
@@ -0,0 +1 @@
+Fixes HTML Help shortcut when Windows is not installed to C drive
diff --git a/Tools/msi/doc/doc.wxs b/Tools/msi/doc/doc.wxs
index 306fb11dc6619..cd1a68cc2601a 100644
--- a/Tools/msi/doc/doc.wxs
+++ b/Tools/msi/doc/doc.wxs
@@ -8,7 +8,7 @@
         <PropertyRef Id="REGISTRYKEY" />
         
         <Property Id="HHExe" Value="C:\Windows\hh.exe" />
-        <CustomAction Id="SetHHExe" Property="HHCExe" Value='[WindowsFolder]\hh.exe' Execute="immediate" />
+        <CustomAction Id="SetHHExe" Property="HHExe" Value='[WindowsFolder]\hh.exe' Execute="immediate" />
         <InstallExecuteSequence>
             <Custom Action="SetHHExe" Before="CostFinalize">1</Custom>
         </InstallExecuteSequence>


From webhook-mailer at python.org  Mon Oct 28 17:55:14 2019
From: webhook-mailer at python.org (Miss Skeleton (bot))
Date: Mon, 28 Oct 2019 21:55:14 -0000
Subject: [Python-checkins] bpo-38589: Fixes HTML Help shortcut when Windows
 is not installed to C drive (GH-16968)
Message-ID: <mailman.171.1572299715.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/2267cb9da4a1fd8bb4962beb893e1f69a37909a7
commit: 2267cb9da4a1fd8bb4962beb893e1f69a37909a7
branch: 3.8
author: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-28T14:55:10-07:00
summary:

bpo-38589: Fixes HTML Help shortcut when Windows is not installed to C drive (GH-16968)

(cherry picked from commit 0ac6137dd3d1e2c8f9558153ad63021f57e05e73)

Co-authored-by: Steve Dower <steve.dower at python.org>

files:
A Misc/NEWS.d/next/Windows/2019-10-28-10-48-16.bpo-38589.V69Q1a.rst
M Tools/msi/doc/doc.wxs

diff --git a/Misc/NEWS.d/next/Windows/2019-10-28-10-48-16.bpo-38589.V69Q1a.rst b/Misc/NEWS.d/next/Windows/2019-10-28-10-48-16.bpo-38589.V69Q1a.rst
new file mode 100644
index 0000000000000..e864290723e07
--- /dev/null
+++ b/Misc/NEWS.d/next/Windows/2019-10-28-10-48-16.bpo-38589.V69Q1a.rst
@@ -0,0 +1 @@
+Fixes HTML Help shortcut when Windows is not installed to C drive
diff --git a/Tools/msi/doc/doc.wxs b/Tools/msi/doc/doc.wxs
index 306fb11dc6619..cd1a68cc2601a 100644
--- a/Tools/msi/doc/doc.wxs
+++ b/Tools/msi/doc/doc.wxs
@@ -8,7 +8,7 @@
         <PropertyRef Id="REGISTRYKEY" />
         
         <Property Id="HHExe" Value="C:\Windows\hh.exe" />
-        <CustomAction Id="SetHHExe" Property="HHCExe" Value='[WindowsFolder]\hh.exe' Execute="immediate" />
+        <CustomAction Id="SetHHExe" Property="HHExe" Value='[WindowsFolder]\hh.exe' Execute="immediate" />
         <InstallExecuteSequence>
             <Custom Action="SetHHExe" Before="CostFinalize">1</Custom>
         </InstallExecuteSequence>


From webhook-mailer at python.org  Mon Oct 28 21:53:37 2019
From: webhook-mailer at python.org (Yury Selivanov)
Date: Tue, 29 Oct 2019 01:53:37 -0000
Subject: [Python-checkins] Fix asyncio.wait() 3.8 whatsnew entry (GH-16975)
Message-ID: <mailman.172.1572314017.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/457306bddbc0021396504b7349fe0c322b65f7a7
commit: 457306bddbc0021396504b7349fe0c322b65f7a7
branch: master
author: Kyle Stanley <aeros167 at gmail.com>
committer: Yury Selivanov <yury at magic.io>
date: 2019-10-28T21:53:22-04:00
summary:

Fix asyncio.wait() 3.8 whatsnew entry (GH-16975)

files:
M Doc/whatsnew/3.8.rst

diff --git a/Doc/whatsnew/3.8.rst b/Doc/whatsnew/3.8.rst
index a3c5f5eab8cab..e26709696f956 100644
--- a/Doc/whatsnew/3.8.rst
+++ b/Doc/whatsnew/3.8.rst
@@ -1677,7 +1677,8 @@ Deprecated
   :func:`asyncio.create_subprocess_shell`.
 
 * The explicit passing of coroutine objects to :func:`asyncio.wait` has been
-  deprecated and will be removed in version 3.10.
+  deprecated and will be removed in version 3.11.
+  (Contributed by Yury Selivanov in :issue:`34790`.)
 
 * The following functions and methods are deprecated in the :mod:`gettext`
   module: :func:`~gettext.lgettext`, :func:`~gettext.ldgettext`,


From webhook-mailer at python.org  Mon Oct 28 22:00:30 2019
From: webhook-mailer at python.org (Miss Skeleton (bot))
Date: Tue, 29 Oct 2019 02:00:30 -0000
Subject: [Python-checkins] Fix asyncio.wait() 3.8 whatsnew entry (GH-16975)
Message-ID: <mailman.173.1572314432.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/31c19e68fe483e3f06269873e0081bd2556483d1
commit: 31c19e68fe483e3f06269873e0081bd2556483d1
branch: 3.8
author: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-28T19:00:26-07:00
summary:

Fix asyncio.wait() 3.8 whatsnew entry (GH-16975)

(cherry picked from commit 457306bddbc0021396504b7349fe0c322b65f7a7)

Co-authored-by: Kyle Stanley <aeros167 at gmail.com>

files:
M Doc/whatsnew/3.8.rst

diff --git a/Doc/whatsnew/3.8.rst b/Doc/whatsnew/3.8.rst
index 21a6a00d8aa0d..e50ccae954921 100644
--- a/Doc/whatsnew/3.8.rst
+++ b/Doc/whatsnew/3.8.rst
@@ -1679,7 +1679,8 @@ Deprecated
   :func:`asyncio.create_subprocess_shell`.
 
 * The explicit passing of coroutine objects to :func:`asyncio.wait` has been
-  deprecated and will be removed in version 3.10.
+  deprecated and will be removed in version 3.11.
+  (Contributed by Yury Selivanov in :issue:`34790`.)
 
 * The following functions and methods are deprecated in the :mod:`gettext`
   module: :func:`~gettext.lgettext`, :func:`~gettext.ldgettext`,


From webhook-mailer at python.org  Tue Oct 29 00:38:57 2019
From: webhook-mailer at python.org (Miss Skeleton (bot))
Date: Tue, 29 Oct 2019 04:38:57 -0000
Subject: [Python-checkins] bpo-38626: Add comment explaining why __lt__ is
 used. (GH-16978)
Message-ID: <mailman.174.1572323938.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/3c88199e0be352c0813f145d7c4c83af044268aa
commit: 3c88199e0be352c0813f145d7c4c83af044268aa
branch: master
author: Raymond Hettinger <rhettinger at users.noreply.github.com>
committer: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
date: 2019-10-28T21:38:50-07:00
summary:

bpo-38626:  Add comment explaining why __lt__ is used. (GH-16978)



https://bugs.python.org/issue38626

files:
M Lib/bisect.py

diff --git a/Lib/bisect.py b/Lib/bisect.py
index 9786fc9d87c5e..8f3f6a3fe35ff 100644
--- a/Lib/bisect.py
+++ b/Lib/bisect.py
@@ -29,6 +29,7 @@ def bisect_right(a, x, lo=0, hi=None):
         hi = len(a)
     while lo < hi:
         mid = (lo+hi)//2
+        # Use __lt__ to match the logic in list.sort() and in heapq
         if x < a[mid]: hi = mid
         else: lo = mid+1
     return lo
@@ -63,6 +64,7 @@ def bisect_left(a, x, lo=0, hi=None):
         hi = len(a)
     while lo < hi:
         mid = (lo+hi)//2
+        # Use __lt__ to match the logic in list.sort() and in heapq
         if a[mid] < x: lo = mid+1
         else: hi = mid
     return lo


From webhook-mailer at python.org  Tue Oct 29 01:10:31 2019
From: webhook-mailer at python.org (Raymond Hettinger)
Date: Tue, 29 Oct 2019 05:10:31 -0000
Subject: [Python-checkins] Permutations Python code equivalent in comment
 was invalid for Python 3 (GH-16927)
Message-ID: <mailman.175.1572325833.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/0078a0c2a59a358fa032ec9847c108378b66b656
commit: 0078a0c2a59a358fa032ec9847c108378b66b656
branch: master
author: Sergey <bizywiz at gmail.com>
committer: Raymond Hettinger <rhettinger at users.noreply.github.com>
date: 2019-10-28T22:10:24-07:00
summary:

Permutations Python code equivalent in comment was invalid for Python 3 (GH-16927)

files:
M Modules/itertoolsmodule.c

diff --git a/Modules/itertoolsmodule.c b/Modules/itertoolsmodule.c
index e60ad5bec43c6..3d39fa2e4737b 100644
--- a/Modules/itertoolsmodule.c
+++ b/Modules/itertoolsmodule.c
@@ -3076,12 +3076,15 @@ static PyTypeObject cwr_type = {
 /* permutations object ********************************************************
 
 def permutations(iterable, r=None):
-    'permutations(range(3), 2) --> (0,1) (0,2) (1,0) (1,2) (2,0) (2,1)'
+    # permutations('ABCD', 2) --> AB AC AD BA BC BD CA CB CD DA DB DC
+    # permutations(range(3)) --> 012 021 102 120 201 210
     pool = tuple(iterable)
     n = len(pool)
     r = n if r is None else r
-    indices = range(n)
-    cycles = range(n-r+1, n+1)[::-1]
+    if r > n:
+        return
+    indices = list(range(n))
+    cycles = list(range(n, n-r, -1))
     yield tuple(pool[i] for i in indices[:r])
     while n:
         for i in reversed(range(r)):


From webhook-mailer at python.org  Tue Oct 29 01:40:19 2019
From: webhook-mailer at python.org (larryhastings)
Date: Tue, 29 Oct 2019 05:40:19 -0000
Subject: [Python-checkins] bpo-38243,
 xmlrpc.server: Escape the server_title (GH-16373) (GH-16441)
 (#16516)
Message-ID: <mailman.176.1572327619.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/3fe1b19265b55c290fc956e9aafcf661803782de
commit: 3fe1b19265b55c290fc956e9aafcf661803782de
branch: 3.5
author: Victor Stinner <vstinner at redhat.com>
committer: larryhastings <larry at hastings.org>
date: 2019-10-28T22:40:15-07:00
summary:

bpo-38243, xmlrpc.server: Escape the server_title (GH-16373) (GH-16441) (#16516)

Escape the server title of xmlrpc.server.DocXMLRPCServer
when rendering the document page as HTML.

(cherry picked from commit e8650a4f8c7fb76f570d4ca9c1fbe44e91c8dfaa)

files:
A Misc/NEWS.d/next/Security/2019-09-25-13-21-09.bpo-38243.1pfz24.rst
M Lib/test/test_docxmlrpc.py
M Lib/xmlrpc/server.py

diff --git a/Lib/test/test_docxmlrpc.py b/Lib/test/test_docxmlrpc.py
index 00903337c07c2..d2adb21af0fb3 100644
--- a/Lib/test/test_docxmlrpc.py
+++ b/Lib/test/test_docxmlrpc.py
@@ -1,5 +1,6 @@
 from xmlrpc.server import DocXMLRPCServer
 import http.client
+import re
 import sys
 from test import support
 threading = support.import_module('threading')
@@ -193,6 +194,21 @@ def test_annotations(self):
              b'method_annotation</strong></a>(x: bytes)</dt></dl>'),
             response.read())
 
+    def test_server_title_escape(self):
+        # bpo-38243: Ensure that the server title and documentation
+        # are escaped for HTML.
+        self.serv.set_server_title('test_title<script>')
+        self.serv.set_server_documentation('test_documentation<script>')
+        self.assertEqual('test_title<script>', self.serv.server_title)
+        self.assertEqual('test_documentation<script>',
+                self.serv.server_documentation)
+
+        generated = self.serv.generate_html_documentation()
+        title = re.search(r'<title>(.+?)</title>', generated).group()
+        documentation = re.search(r'<p><tt>(.+?)</tt></p>', generated).group()
+        self.assertEqual('<title>Python: test_title&lt;script&gt;</title>', title)
+        self.assertEqual('<p><tt>test_documentation&lt;script&gt;</tt></p>', documentation)
+
 
 if __name__ == '__main__':
     unittest.main()
diff --git a/Lib/xmlrpc/server.py b/Lib/xmlrpc/server.py
index 276394dfbf33d..ca39d7f88f8a9 100644
--- a/Lib/xmlrpc/server.py
+++ b/Lib/xmlrpc/server.py
@@ -106,6 +106,7 @@ def export_add(self, x, y):
 
 from xmlrpc.client import Fault, dumps, loads, gzip_encode, gzip_decode
 from http.server import BaseHTTPRequestHandler
+import html
 import http.server
 import socketserver
 import sys
@@ -892,7 +893,7 @@ def generate_html_documentation(self):
                                 methods
                             )
 
-        return documenter.page(self.server_title, documentation)
+        return documenter.page(html.escape(self.server_title), documentation)
 
 class DocXMLRPCRequestHandler(SimpleXMLRPCRequestHandler):
     """XML-RPC and documentation request handler class.
diff --git a/Misc/NEWS.d/next/Security/2019-09-25-13-21-09.bpo-38243.1pfz24.rst b/Misc/NEWS.d/next/Security/2019-09-25-13-21-09.bpo-38243.1pfz24.rst
new file mode 100644
index 0000000000000..98d7be129573a
--- /dev/null
+++ b/Misc/NEWS.d/next/Security/2019-09-25-13-21-09.bpo-38243.1pfz24.rst
@@ -0,0 +1,3 @@
+Escape the server title of :class:`xmlrpc.server.DocXMLRPCServer`
+when rendering the document page as HTML.
+(Contributed by Dong-hee Na in :issue:`38243`.)


From webhook-mailer at python.org  Tue Oct 29 03:04:08 2019
From: webhook-mailer at python.org (Raymond Hettinger)
Date: Tue, 29 Oct 2019 07:04:08 -0000
Subject: [Python-checkins] bpo-38336: Remove the __set__ method restriction
 on data descriptors for attribute lookup precedence (GH-16520)
Message-ID: <mailman.177.1572332649.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/4c155f738dc2e70ccb574f5169ad89c01753c6f7
commit: 4c155f738dc2e70ccb574f5169ad89c01753c6f7
branch: master
author: G?ry Ogam <gery.ogam at gmail.com>
committer: Raymond Hettinger <rhettinger at users.noreply.github.com>
date: 2019-10-29T00:04:01-07:00
summary:

bpo-38336: Remove the __set__ method restriction on data descriptors for attribute lookup precedence (GH-16520)

files:
M Doc/reference/datamodel.rst

diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst
index c6c6e4075039c..2499aeb6996bb 100644
--- a/Doc/reference/datamodel.rst
+++ b/Doc/reference/datamodel.rst
@@ -1716,7 +1716,7 @@ the descriptor defines :meth:`__set__` and/or :meth:`__delete__`, it is a data
 descriptor; if it defines neither, it is a non-data descriptor.  Normally, data
 descriptors define both :meth:`__get__` and :meth:`__set__`, while non-data
 descriptors have just the :meth:`__get__` method.  Data descriptors with
-:meth:`__set__` and :meth:`__get__` defined always override a redefinition in an
+:meth:`__get__` and :meth:`__set__` (and/or :meth:`__delete__`) defined always override a redefinition in an
 instance dictionary.  In contrast, non-data descriptors can be overridden by
 instances.
 


From webhook-mailer at python.org  Tue Oct 29 03:24:22 2019
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Tue, 29 Oct 2019 07:24:22 -0000
Subject: [Python-checkins] bpo-36993: Improve error reporting for zipfiles
 with bad zip64 extra data. (GH-14656)
Message-ID: <mailman.178.1572333863.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/da6ce58dd5ac109485af45878fca6bfd265b43e9
commit: da6ce58dd5ac109485af45878fca6bfd265b43e9
branch: master
author: Daniel Hillier <daniel.hillier at gmail.com>
committer: Serhiy Storchaka <storchaka at gmail.com>
date: 2019-10-29T09:24:18+02:00
summary:

bpo-36993: Improve error reporting for zipfiles with bad zip64 extra data. (GH-14656)

files:
A Misc/NEWS.d/next/Library/2019-07-09-05-44-39.bpo-36993.4javqu.rst
M Lib/test/test_zipfile.py
M Lib/zipfile.py

diff --git a/Lib/test/test_zipfile.py b/Lib/test/test_zipfile.py
index 6e1291e524f29..1e1854be7109b 100644
--- a/Lib/test/test_zipfile.py
+++ b/Lib/test/test_zipfile.py
@@ -1,6 +1,7 @@
 import contextlib
 import importlib.util
 import io
+import itertools
 import os
 import pathlib
 import posixpath
@@ -824,6 +825,227 @@ def test_append(self):
             zinfo = zipfp.getinfo("strfile")
             self.assertEqual(zinfo.extra, extra)
 
+    def make_zip64_file(
+        self, file_size_64_set=False, file_size_extra=False,
+        compress_size_64_set=False, compress_size_extra=False,
+        header_offset_64_set=False, header_offset_extra=False,
+    ):
+        """Generate bytes sequence for a zip with (incomplete) zip64 data.
+
+        The actual values (not the zip 64 0xffffffff values) stored in the file
+        are:
+        file_size: 8
+        compress_size: 8
+        header_offset: 0
+        """
+        actual_size = 8
+        actual_header_offset = 0
+        local_zip64_fields = []
+        central_zip64_fields = []
+
+        file_size = actual_size
+        if file_size_64_set:
+            file_size = 0xffffffff
+            if file_size_extra:
+                local_zip64_fields.append(actual_size)
+                central_zip64_fields.append(actual_size)
+        file_size = struct.pack("<L", file_size)
+
+        compress_size = actual_size
+        if compress_size_64_set:
+            compress_size = 0xffffffff
+            if compress_size_extra:
+                local_zip64_fields.append(actual_size)
+                central_zip64_fields.append(actual_size)
+        compress_size = struct.pack("<L", compress_size)
+
+        header_offset = actual_header_offset
+        if header_offset_64_set:
+            header_offset = 0xffffffff
+            if header_offset_extra:
+                central_zip64_fields.append(actual_header_offset)
+        header_offset = struct.pack("<L", header_offset)
+
+        local_extra = struct.pack(
+            '<HH' + 'Q'*len(local_zip64_fields),
+            0x0001,
+            8*len(local_zip64_fields),
+            *local_zip64_fields
+        )
+
+        central_extra = struct.pack(
+            '<HH' + 'Q'*len(central_zip64_fields),
+            0x0001,
+            8*len(central_zip64_fields),
+            *central_zip64_fields
+        )
+
+        central_dir_size = struct.pack('<Q', 58 + 8 * len(central_zip64_fields))
+        offset_to_central_dir = struct.pack('<Q', 50 + 8 * len(local_zip64_fields))
+
+        local_extra_length = struct.pack("<H", 4 + 8 * len(local_zip64_fields))
+        central_extra_length = struct.pack("<H", 4 + 8 * len(central_zip64_fields))
+
+        filename = b"test.txt"
+        content = b"test1234"
+        filename_length = struct.pack("<H", len(filename))
+        zip64_contents = (
+            # Local file header
+            b"PK\x03\x04\x14\x00\x00\x00\x00\x00\x00\x00!\x00\x9e%\xf5\xaf"
+            + compress_size
+            + file_size
+            + filename_length
+            + local_extra_length
+            + filename
+            + local_extra
+            + content
+            # Central directory:
+            + b"PK\x01\x02-\x03-\x00\x00\x00\x00\x00\x00\x00!\x00\x9e%\xf5\xaf"
+            + compress_size
+            + file_size
+            + filename_length
+            + central_extra_length
+            + b"\x00\x00\x00\x00\x00\x00\x00\x00\x80\x01"
+            + header_offset
+            + filename
+            + central_extra
+            # Zip64 end of central directory
+            + b"PK\x06\x06,\x00\x00\x00\x00\x00\x00\x00-\x00-"
+            + b"\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00"
+            + b"\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00"
+            + central_dir_size
+            + offset_to_central_dir
+            # Zip64 end of central directory locator
+            + b"PK\x06\x07\x00\x00\x00\x00l\x00\x00\x00\x00\x00\x00\x00\x01"
+            + b"\x00\x00\x00"
+            # end of central directory
+            + b"PK\x05\x06\x00\x00\x00\x00\x01\x00\x01\x00:\x00\x00\x002\x00"
+            + b"\x00\x00\x00\x00"
+        )
+        return zip64_contents
+
+    def test_bad_zip64_extra(self):
+        """Missing zip64 extra records raises an exception.
+
+        There are 4 fields that the zip64 format handles (the disk number is
+        not used in this module and so is ignored here). According to the zip
+        spec:
+              The order of the fields in the zip64 extended
+              information record is fixed, but the fields MUST
+              only appear if the corresponding Local or Central
+              directory record field is set to 0xFFFF or 0xFFFFFFFF.
+
+        If the zip64 extra content doesn't contain enough entries for the
+        number of fields marked with 0xFFFF or 0xFFFFFFFF, we raise an error.
+        This test mismatches the length of the zip64 extra field and the number
+        of fields set to indicate the presence of zip64 data.
+        """
+        # zip64 file size present, no fields in extra, expecting one, equals
+        # missing file size.
+        missing_file_size_extra = self.make_zip64_file(
+            file_size_64_set=True,
+        )
+        with self.assertRaises(zipfile.BadZipFile) as e:
+            zipfile.ZipFile(io.BytesIO(missing_file_size_extra))
+        self.assertIn('file size', str(e.exception).lower())
+
+        # zip64 file size present, zip64 compress size present, one field in
+        # extra, expecting two, equals missing compress size.
+        missing_compress_size_extra = self.make_zip64_file(
+            file_size_64_set=True,
+            file_size_extra=True,
+            compress_size_64_set=True,
+        )
+        with self.assertRaises(zipfile.BadZipFile) as e:
+            zipfile.ZipFile(io.BytesIO(missing_compress_size_extra))
+        self.assertIn('compress size', str(e.exception).lower())
+
+        # zip64 compress size present, no fields in extra, expecting one,
+        # equals missing compress size.
+        missing_compress_size_extra = self.make_zip64_file(
+            compress_size_64_set=True,
+        )
+        with self.assertRaises(zipfile.BadZipFile) as e:
+            zipfile.ZipFile(io.BytesIO(missing_compress_size_extra))
+        self.assertIn('compress size', str(e.exception).lower())
+
+        # zip64 file size present, zip64 compress size present, zip64 header
+        # offset present, two fields in extra, expecting three, equals missing
+        # header offset
+        missing_header_offset_extra = self.make_zip64_file(
+            file_size_64_set=True,
+            file_size_extra=True,
+            compress_size_64_set=True,
+            compress_size_extra=True,
+            header_offset_64_set=True,
+        )
+        with self.assertRaises(zipfile.BadZipFile) as e:
+            zipfile.ZipFile(io.BytesIO(missing_header_offset_extra))
+        self.assertIn('header offset', str(e.exception).lower())
+
+        # zip64 compress size present, zip64 header offset present, one field
+        # in extra, expecting two, equals missing header offset
+        missing_header_offset_extra = self.make_zip64_file(
+            file_size_64_set=False,
+            compress_size_64_set=True,
+            compress_size_extra=True,
+            header_offset_64_set=True,
+        )
+        with self.assertRaises(zipfile.BadZipFile) as e:
+            zipfile.ZipFile(io.BytesIO(missing_header_offset_extra))
+        self.assertIn('header offset', str(e.exception).lower())
+
+        # zip64 file size present, zip64 header offset present, one field in
+        # extra, expecting two, equals missing header offset
+        missing_header_offset_extra = self.make_zip64_file(
+            file_size_64_set=True,
+            file_size_extra=True,
+            compress_size_64_set=False,
+            header_offset_64_set=True,
+        )
+        with self.assertRaises(zipfile.BadZipFile) as e:
+            zipfile.ZipFile(io.BytesIO(missing_header_offset_extra))
+        self.assertIn('header offset', str(e.exception).lower())
+
+        # zip64 header offset present, no fields in extra, expecting one,
+        # equals missing header offset
+        missing_header_offset_extra = self.make_zip64_file(
+            file_size_64_set=False,
+            compress_size_64_set=False,
+            header_offset_64_set=True,
+        )
+        with self.assertRaises(zipfile.BadZipFile) as e:
+            zipfile.ZipFile(io.BytesIO(missing_header_offset_extra))
+        self.assertIn('header offset', str(e.exception).lower())
+
+    def test_generated_valid_zip64_extra(self):
+        # These values are what is set in the make_zip64_file method.
+        expected_file_size = 8
+        expected_compress_size = 8
+        expected_header_offset = 0
+        expected_content = b"test1234"
+
+        # Loop through the various valid combinations of zip64 masks
+        # present and extra fields present.
+        params = (
+            {"file_size_64_set": True, "file_size_extra": True},
+            {"compress_size_64_set": True, "compress_size_extra": True},
+            {"header_offset_64_set": True, "header_offset_extra": True},
+        )
+
+        for r in range(1, len(params) + 1):
+            for combo in itertools.combinations(params, r):
+                kwargs = {}
+                for c in combo:
+                    kwargs.update(c)
+                with zipfile.ZipFile(io.BytesIO(self.make_zip64_file(**kwargs))) as zf:
+                    zinfo = zf.infolist()[0]
+                    self.assertEqual(zinfo.file_size, expected_file_size)
+                    self.assertEqual(zinfo.compress_size, expected_compress_size)
+                    self.assertEqual(zinfo.header_offset, expected_header_offset)
+                    self.assertEqual(zf.read(zinfo), expected_content)
+
+
 @requires_zlib
 class DeflateTestZip64InSmallFiles(AbstractTestZip64InSmallFiles,
                                    unittest.TestCase):
diff --git a/Lib/zipfile.py b/Lib/zipfile.py
index 6201edc8d7386..6504e0eee8b5a 100644
--- a/Lib/zipfile.py
+++ b/Lib/zipfile.py
@@ -480,14 +480,26 @@ def _decodeExtra(self):
 
                 # ZIP64 extension (large files and/or large archives)
                 if self.file_size in (0xffffffffffffffff, 0xffffffff):
+                    if len(counts) <= idx:
+                        raise BadZipFile(
+                            "Corrupt zip64 extra field. File size not found."
+                        )
                     self.file_size = counts[idx]
                     idx += 1
 
                 if self.compress_size == 0xFFFFFFFF:
+                    if len(counts) <= idx:
+                        raise BadZipFile(
+                            "Corrupt zip64 extra field. Compress size not found."
+                        )
                     self.compress_size = counts[idx]
                     idx += 1
 
                 if self.header_offset == 0xffffffff:
+                    if len(counts) <= idx:
+                        raise BadZipFile(
+                            "Corrupt zip64 extra field. Header offset not found."
+                        )
                     old = self.header_offset
                     self.header_offset = counts[idx]
                     idx+=1
diff --git a/Misc/NEWS.d/next/Library/2019-07-09-05-44-39.bpo-36993.4javqu.rst b/Misc/NEWS.d/next/Library/2019-07-09-05-44-39.bpo-36993.4javqu.rst
new file mode 100644
index 0000000000000..009e07b92d22d
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-07-09-05-44-39.bpo-36993.4javqu.rst
@@ -0,0 +1,2 @@
+Improve error reporting for corrupt zip files with bad zip64 extra data. Patch
+by Daniel Hillier.


From webhook-mailer at python.org  Tue Oct 29 03:43:41 2019
From: webhook-mailer at python.org (Miss Skeleton (bot))
Date: Tue, 29 Oct 2019 07:43:41 -0000
Subject: [Python-checkins] bpo-36993: Improve error reporting for zipfiles
 with bad zip64 extra data. (GH-14656)
Message-ID: <mailman.179.1572335023.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/f7d50f8f997fbfce1556991a3700826536871fe7
commit: f7d50f8f997fbfce1556991a3700826536871fe7
branch: 3.7
author: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-29T00:43:37-07:00
summary:

bpo-36993: Improve error reporting for zipfiles with bad zip64 extra data. (GH-14656)

(cherry picked from commit da6ce58dd5ac109485af45878fca6bfd265b43e9)

Co-authored-by: Daniel Hillier <daniel.hillier at gmail.com>

files:
A Misc/NEWS.d/next/Library/2019-07-09-05-44-39.bpo-36993.4javqu.rst
M Lib/test/test_zipfile.py
M Lib/zipfile.py

diff --git a/Lib/test/test_zipfile.py b/Lib/test/test_zipfile.py
index 523a2e1dadcb9..7e8e8d2c89f08 100644
--- a/Lib/test/test_zipfile.py
+++ b/Lib/test/test_zipfile.py
@@ -1,5 +1,6 @@
 import contextlib
 import io
+import itertools
 import os
 import importlib.util
 import pathlib
@@ -801,6 +802,227 @@ def test_append(self):
             zinfo = zipfp.getinfo("strfile")
             self.assertEqual(zinfo.extra, extra)
 
+    def make_zip64_file(
+        self, file_size_64_set=False, file_size_extra=False,
+        compress_size_64_set=False, compress_size_extra=False,
+        header_offset_64_set=False, header_offset_extra=False,
+    ):
+        """Generate bytes sequence for a zip with (incomplete) zip64 data.
+
+        The actual values (not the zip 64 0xffffffff values) stored in the file
+        are:
+        file_size: 8
+        compress_size: 8
+        header_offset: 0
+        """
+        actual_size = 8
+        actual_header_offset = 0
+        local_zip64_fields = []
+        central_zip64_fields = []
+
+        file_size = actual_size
+        if file_size_64_set:
+            file_size = 0xffffffff
+            if file_size_extra:
+                local_zip64_fields.append(actual_size)
+                central_zip64_fields.append(actual_size)
+        file_size = struct.pack("<L", file_size)
+
+        compress_size = actual_size
+        if compress_size_64_set:
+            compress_size = 0xffffffff
+            if compress_size_extra:
+                local_zip64_fields.append(actual_size)
+                central_zip64_fields.append(actual_size)
+        compress_size = struct.pack("<L", compress_size)
+
+        header_offset = actual_header_offset
+        if header_offset_64_set:
+            header_offset = 0xffffffff
+            if header_offset_extra:
+                central_zip64_fields.append(actual_header_offset)
+        header_offset = struct.pack("<L", header_offset)
+
+        local_extra = struct.pack(
+            '<HH' + 'Q'*len(local_zip64_fields),
+            0x0001,
+            8*len(local_zip64_fields),
+            *local_zip64_fields
+        )
+
+        central_extra = struct.pack(
+            '<HH' + 'Q'*len(central_zip64_fields),
+            0x0001,
+            8*len(central_zip64_fields),
+            *central_zip64_fields
+        )
+
+        central_dir_size = struct.pack('<Q', 58 + 8 * len(central_zip64_fields))
+        offset_to_central_dir = struct.pack('<Q', 50 + 8 * len(local_zip64_fields))
+
+        local_extra_length = struct.pack("<H", 4 + 8 * len(local_zip64_fields))
+        central_extra_length = struct.pack("<H", 4 + 8 * len(central_zip64_fields))
+
+        filename = b"test.txt"
+        content = b"test1234"
+        filename_length = struct.pack("<H", len(filename))
+        zip64_contents = (
+            # Local file header
+            b"PK\x03\x04\x14\x00\x00\x00\x00\x00\x00\x00!\x00\x9e%\xf5\xaf"
+            + compress_size
+            + file_size
+            + filename_length
+            + local_extra_length
+            + filename
+            + local_extra
+            + content
+            # Central directory:
+            + b"PK\x01\x02-\x03-\x00\x00\x00\x00\x00\x00\x00!\x00\x9e%\xf5\xaf"
+            + compress_size
+            + file_size
+            + filename_length
+            + central_extra_length
+            + b"\x00\x00\x00\x00\x00\x00\x00\x00\x80\x01"
+            + header_offset
+            + filename
+            + central_extra
+            # Zip64 end of central directory
+            + b"PK\x06\x06,\x00\x00\x00\x00\x00\x00\x00-\x00-"
+            + b"\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00"
+            + b"\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00"
+            + central_dir_size
+            + offset_to_central_dir
+            # Zip64 end of central directory locator
+            + b"PK\x06\x07\x00\x00\x00\x00l\x00\x00\x00\x00\x00\x00\x00\x01"
+            + b"\x00\x00\x00"
+            # end of central directory
+            + b"PK\x05\x06\x00\x00\x00\x00\x01\x00\x01\x00:\x00\x00\x002\x00"
+            + b"\x00\x00\x00\x00"
+        )
+        return zip64_contents
+
+    def test_bad_zip64_extra(self):
+        """Missing zip64 extra records raises an exception.
+
+        There are 4 fields that the zip64 format handles (the disk number is
+        not used in this module and so is ignored here). According to the zip
+        spec:
+              The order of the fields in the zip64 extended
+              information record is fixed, but the fields MUST
+              only appear if the corresponding Local or Central
+              directory record field is set to 0xFFFF or 0xFFFFFFFF.
+
+        If the zip64 extra content doesn't contain enough entries for the
+        number of fields marked with 0xFFFF or 0xFFFFFFFF, we raise an error.
+        This test mismatches the length of the zip64 extra field and the number
+        of fields set to indicate the presence of zip64 data.
+        """
+        # zip64 file size present, no fields in extra, expecting one, equals
+        # missing file size.
+        missing_file_size_extra = self.make_zip64_file(
+            file_size_64_set=True,
+        )
+        with self.assertRaises(zipfile.BadZipFile) as e:
+            zipfile.ZipFile(io.BytesIO(missing_file_size_extra))
+        self.assertIn('file size', str(e.exception).lower())
+
+        # zip64 file size present, zip64 compress size present, one field in
+        # extra, expecting two, equals missing compress size.
+        missing_compress_size_extra = self.make_zip64_file(
+            file_size_64_set=True,
+            file_size_extra=True,
+            compress_size_64_set=True,
+        )
+        with self.assertRaises(zipfile.BadZipFile) as e:
+            zipfile.ZipFile(io.BytesIO(missing_compress_size_extra))
+        self.assertIn('compress size', str(e.exception).lower())
+
+        # zip64 compress size present, no fields in extra, expecting one,
+        # equals missing compress size.
+        missing_compress_size_extra = self.make_zip64_file(
+            compress_size_64_set=True,
+        )
+        with self.assertRaises(zipfile.BadZipFile) as e:
+            zipfile.ZipFile(io.BytesIO(missing_compress_size_extra))
+        self.assertIn('compress size', str(e.exception).lower())
+
+        # zip64 file size present, zip64 compress size present, zip64 header
+        # offset present, two fields in extra, expecting three, equals missing
+        # header offset
+        missing_header_offset_extra = self.make_zip64_file(
+            file_size_64_set=True,
+            file_size_extra=True,
+            compress_size_64_set=True,
+            compress_size_extra=True,
+            header_offset_64_set=True,
+        )
+        with self.assertRaises(zipfile.BadZipFile) as e:
+            zipfile.ZipFile(io.BytesIO(missing_header_offset_extra))
+        self.assertIn('header offset', str(e.exception).lower())
+
+        # zip64 compress size present, zip64 header offset present, one field
+        # in extra, expecting two, equals missing header offset
+        missing_header_offset_extra = self.make_zip64_file(
+            file_size_64_set=False,
+            compress_size_64_set=True,
+            compress_size_extra=True,
+            header_offset_64_set=True,
+        )
+        with self.assertRaises(zipfile.BadZipFile) as e:
+            zipfile.ZipFile(io.BytesIO(missing_header_offset_extra))
+        self.assertIn('header offset', str(e.exception).lower())
+
+        # zip64 file size present, zip64 header offset present, one field in
+        # extra, expecting two, equals missing header offset
+        missing_header_offset_extra = self.make_zip64_file(
+            file_size_64_set=True,
+            file_size_extra=True,
+            compress_size_64_set=False,
+            header_offset_64_set=True,
+        )
+        with self.assertRaises(zipfile.BadZipFile) as e:
+            zipfile.ZipFile(io.BytesIO(missing_header_offset_extra))
+        self.assertIn('header offset', str(e.exception).lower())
+
+        # zip64 header offset present, no fields in extra, expecting one,
+        # equals missing header offset
+        missing_header_offset_extra = self.make_zip64_file(
+            file_size_64_set=False,
+            compress_size_64_set=False,
+            header_offset_64_set=True,
+        )
+        with self.assertRaises(zipfile.BadZipFile) as e:
+            zipfile.ZipFile(io.BytesIO(missing_header_offset_extra))
+        self.assertIn('header offset', str(e.exception).lower())
+
+    def test_generated_valid_zip64_extra(self):
+        # These values are what is set in the make_zip64_file method.
+        expected_file_size = 8
+        expected_compress_size = 8
+        expected_header_offset = 0
+        expected_content = b"test1234"
+
+        # Loop through the various valid combinations of zip64 masks
+        # present and extra fields present.
+        params = (
+            {"file_size_64_set": True, "file_size_extra": True},
+            {"compress_size_64_set": True, "compress_size_extra": True},
+            {"header_offset_64_set": True, "header_offset_extra": True},
+        )
+
+        for r in range(1, len(params) + 1):
+            for combo in itertools.combinations(params, r):
+                kwargs = {}
+                for c in combo:
+                    kwargs.update(c)
+                with zipfile.ZipFile(io.BytesIO(self.make_zip64_file(**kwargs))) as zf:
+                    zinfo = zf.infolist()[0]
+                    self.assertEqual(zinfo.file_size, expected_file_size)
+                    self.assertEqual(zinfo.compress_size, expected_compress_size)
+                    self.assertEqual(zinfo.header_offset, expected_header_offset)
+                    self.assertEqual(zf.read(zinfo), expected_content)
+
+
 @requires_zlib
 class DeflateTestZip64InSmallFiles(AbstractTestZip64InSmallFiles,
                                    unittest.TestCase):
diff --git a/Lib/zipfile.py b/Lib/zipfile.py
index 22176126a9b90..63748d371f0e7 100644
--- a/Lib/zipfile.py
+++ b/Lib/zipfile.py
@@ -477,14 +477,26 @@ def _decodeExtra(self):
 
                 # ZIP64 extension (large files and/or large archives)
                 if self.file_size in (0xffffffffffffffff, 0xffffffff):
+                    if len(counts) <= idx:
+                        raise BadZipFile(
+                            "Corrupt zip64 extra field. File size not found."
+                        )
                     self.file_size = counts[idx]
                     idx += 1
 
                 if self.compress_size == 0xFFFFFFFF:
+                    if len(counts) <= idx:
+                        raise BadZipFile(
+                            "Corrupt zip64 extra field. Compress size not found."
+                        )
                     self.compress_size = counts[idx]
                     idx += 1
 
                 if self.header_offset == 0xffffffff:
+                    if len(counts) <= idx:
+                        raise BadZipFile(
+                            "Corrupt zip64 extra field. Header offset not found."
+                        )
                     old = self.header_offset
                     self.header_offset = counts[idx]
                     idx+=1
diff --git a/Misc/NEWS.d/next/Library/2019-07-09-05-44-39.bpo-36993.4javqu.rst b/Misc/NEWS.d/next/Library/2019-07-09-05-44-39.bpo-36993.4javqu.rst
new file mode 100644
index 0000000000000..009e07b92d22d
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-07-09-05-44-39.bpo-36993.4javqu.rst
@@ -0,0 +1,2 @@
+Improve error reporting for corrupt zip files with bad zip64 extra data. Patch
+by Daniel Hillier.


From webhook-mailer at python.org  Tue Oct 29 03:44:11 2019
From: webhook-mailer at python.org (Miss Skeleton (bot))
Date: Tue, 29 Oct 2019 07:44:11 -0000
Subject: [Python-checkins] bpo-36993: Improve error reporting for zipfiles
 with bad zip64 extra data. (GH-14656)
Message-ID: <mailman.180.1572335051.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/3801b2699eb9441ca31c6ec8fa956fc0fe755ef7
commit: 3801b2699eb9441ca31c6ec8fa956fc0fe755ef7
branch: 3.8
author: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-29T00:44:07-07:00
summary:

bpo-36993: Improve error reporting for zipfiles with bad zip64 extra data. (GH-14656)

(cherry picked from commit da6ce58dd5ac109485af45878fca6bfd265b43e9)

Co-authored-by: Daniel Hillier <daniel.hillier at gmail.com>

files:
A Misc/NEWS.d/next/Library/2019-07-09-05-44-39.bpo-36993.4javqu.rst
M Lib/test/test_zipfile.py
M Lib/zipfile.py

diff --git a/Lib/test/test_zipfile.py b/Lib/test/test_zipfile.py
index e9ef89dd3eb2b..f03c044eae3a6 100644
--- a/Lib/test/test_zipfile.py
+++ b/Lib/test/test_zipfile.py
@@ -1,6 +1,7 @@
 import contextlib
 import importlib.util
 import io
+import itertools
 import os
 import pathlib
 import posixpath
@@ -824,6 +825,227 @@ def test_append(self):
             zinfo = zipfp.getinfo("strfile")
             self.assertEqual(zinfo.extra, extra)
 
+    def make_zip64_file(
+        self, file_size_64_set=False, file_size_extra=False,
+        compress_size_64_set=False, compress_size_extra=False,
+        header_offset_64_set=False, header_offset_extra=False,
+    ):
+        """Generate bytes sequence for a zip with (incomplete) zip64 data.
+
+        The actual values (not the zip 64 0xffffffff values) stored in the file
+        are:
+        file_size: 8
+        compress_size: 8
+        header_offset: 0
+        """
+        actual_size = 8
+        actual_header_offset = 0
+        local_zip64_fields = []
+        central_zip64_fields = []
+
+        file_size = actual_size
+        if file_size_64_set:
+            file_size = 0xffffffff
+            if file_size_extra:
+                local_zip64_fields.append(actual_size)
+                central_zip64_fields.append(actual_size)
+        file_size = struct.pack("<L", file_size)
+
+        compress_size = actual_size
+        if compress_size_64_set:
+            compress_size = 0xffffffff
+            if compress_size_extra:
+                local_zip64_fields.append(actual_size)
+                central_zip64_fields.append(actual_size)
+        compress_size = struct.pack("<L", compress_size)
+
+        header_offset = actual_header_offset
+        if header_offset_64_set:
+            header_offset = 0xffffffff
+            if header_offset_extra:
+                central_zip64_fields.append(actual_header_offset)
+        header_offset = struct.pack("<L", header_offset)
+
+        local_extra = struct.pack(
+            '<HH' + 'Q'*len(local_zip64_fields),
+            0x0001,
+            8*len(local_zip64_fields),
+            *local_zip64_fields
+        )
+
+        central_extra = struct.pack(
+            '<HH' + 'Q'*len(central_zip64_fields),
+            0x0001,
+            8*len(central_zip64_fields),
+            *central_zip64_fields
+        )
+
+        central_dir_size = struct.pack('<Q', 58 + 8 * len(central_zip64_fields))
+        offset_to_central_dir = struct.pack('<Q', 50 + 8 * len(local_zip64_fields))
+
+        local_extra_length = struct.pack("<H", 4 + 8 * len(local_zip64_fields))
+        central_extra_length = struct.pack("<H", 4 + 8 * len(central_zip64_fields))
+
+        filename = b"test.txt"
+        content = b"test1234"
+        filename_length = struct.pack("<H", len(filename))
+        zip64_contents = (
+            # Local file header
+            b"PK\x03\x04\x14\x00\x00\x00\x00\x00\x00\x00!\x00\x9e%\xf5\xaf"
+            + compress_size
+            + file_size
+            + filename_length
+            + local_extra_length
+            + filename
+            + local_extra
+            + content
+            # Central directory:
+            + b"PK\x01\x02-\x03-\x00\x00\x00\x00\x00\x00\x00!\x00\x9e%\xf5\xaf"
+            + compress_size
+            + file_size
+            + filename_length
+            + central_extra_length
+            + b"\x00\x00\x00\x00\x00\x00\x00\x00\x80\x01"
+            + header_offset
+            + filename
+            + central_extra
+            # Zip64 end of central directory
+            + b"PK\x06\x06,\x00\x00\x00\x00\x00\x00\x00-\x00-"
+            + b"\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00"
+            + b"\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00"
+            + central_dir_size
+            + offset_to_central_dir
+            # Zip64 end of central directory locator
+            + b"PK\x06\x07\x00\x00\x00\x00l\x00\x00\x00\x00\x00\x00\x00\x01"
+            + b"\x00\x00\x00"
+            # end of central directory
+            + b"PK\x05\x06\x00\x00\x00\x00\x01\x00\x01\x00:\x00\x00\x002\x00"
+            + b"\x00\x00\x00\x00"
+        )
+        return zip64_contents
+
+    def test_bad_zip64_extra(self):
+        """Missing zip64 extra records raises an exception.
+
+        There are 4 fields that the zip64 format handles (the disk number is
+        not used in this module and so is ignored here). According to the zip
+        spec:
+              The order of the fields in the zip64 extended
+              information record is fixed, but the fields MUST
+              only appear if the corresponding Local or Central
+              directory record field is set to 0xFFFF or 0xFFFFFFFF.
+
+        If the zip64 extra content doesn't contain enough entries for the
+        number of fields marked with 0xFFFF or 0xFFFFFFFF, we raise an error.
+        This test mismatches the length of the zip64 extra field and the number
+        of fields set to indicate the presence of zip64 data.
+        """
+        # zip64 file size present, no fields in extra, expecting one, equals
+        # missing file size.
+        missing_file_size_extra = self.make_zip64_file(
+            file_size_64_set=True,
+        )
+        with self.assertRaises(zipfile.BadZipFile) as e:
+            zipfile.ZipFile(io.BytesIO(missing_file_size_extra))
+        self.assertIn('file size', str(e.exception).lower())
+
+        # zip64 file size present, zip64 compress size present, one field in
+        # extra, expecting two, equals missing compress size.
+        missing_compress_size_extra = self.make_zip64_file(
+            file_size_64_set=True,
+            file_size_extra=True,
+            compress_size_64_set=True,
+        )
+        with self.assertRaises(zipfile.BadZipFile) as e:
+            zipfile.ZipFile(io.BytesIO(missing_compress_size_extra))
+        self.assertIn('compress size', str(e.exception).lower())
+
+        # zip64 compress size present, no fields in extra, expecting one,
+        # equals missing compress size.
+        missing_compress_size_extra = self.make_zip64_file(
+            compress_size_64_set=True,
+        )
+        with self.assertRaises(zipfile.BadZipFile) as e:
+            zipfile.ZipFile(io.BytesIO(missing_compress_size_extra))
+        self.assertIn('compress size', str(e.exception).lower())
+
+        # zip64 file size present, zip64 compress size present, zip64 header
+        # offset present, two fields in extra, expecting three, equals missing
+        # header offset
+        missing_header_offset_extra = self.make_zip64_file(
+            file_size_64_set=True,
+            file_size_extra=True,
+            compress_size_64_set=True,
+            compress_size_extra=True,
+            header_offset_64_set=True,
+        )
+        with self.assertRaises(zipfile.BadZipFile) as e:
+            zipfile.ZipFile(io.BytesIO(missing_header_offset_extra))
+        self.assertIn('header offset', str(e.exception).lower())
+
+        # zip64 compress size present, zip64 header offset present, one field
+        # in extra, expecting two, equals missing header offset
+        missing_header_offset_extra = self.make_zip64_file(
+            file_size_64_set=False,
+            compress_size_64_set=True,
+            compress_size_extra=True,
+            header_offset_64_set=True,
+        )
+        with self.assertRaises(zipfile.BadZipFile) as e:
+            zipfile.ZipFile(io.BytesIO(missing_header_offset_extra))
+        self.assertIn('header offset', str(e.exception).lower())
+
+        # zip64 file size present, zip64 header offset present, one field in
+        # extra, expecting two, equals missing header offset
+        missing_header_offset_extra = self.make_zip64_file(
+            file_size_64_set=True,
+            file_size_extra=True,
+            compress_size_64_set=False,
+            header_offset_64_set=True,
+        )
+        with self.assertRaises(zipfile.BadZipFile) as e:
+            zipfile.ZipFile(io.BytesIO(missing_header_offset_extra))
+        self.assertIn('header offset', str(e.exception).lower())
+
+        # zip64 header offset present, no fields in extra, expecting one,
+        # equals missing header offset
+        missing_header_offset_extra = self.make_zip64_file(
+            file_size_64_set=False,
+            compress_size_64_set=False,
+            header_offset_64_set=True,
+        )
+        with self.assertRaises(zipfile.BadZipFile) as e:
+            zipfile.ZipFile(io.BytesIO(missing_header_offset_extra))
+        self.assertIn('header offset', str(e.exception).lower())
+
+    def test_generated_valid_zip64_extra(self):
+        # These values are what is set in the make_zip64_file method.
+        expected_file_size = 8
+        expected_compress_size = 8
+        expected_header_offset = 0
+        expected_content = b"test1234"
+
+        # Loop through the various valid combinations of zip64 masks
+        # present and extra fields present.
+        params = (
+            {"file_size_64_set": True, "file_size_extra": True},
+            {"compress_size_64_set": True, "compress_size_extra": True},
+            {"header_offset_64_set": True, "header_offset_extra": True},
+        )
+
+        for r in range(1, len(params) + 1):
+            for combo in itertools.combinations(params, r):
+                kwargs = {}
+                for c in combo:
+                    kwargs.update(c)
+                with zipfile.ZipFile(io.BytesIO(self.make_zip64_file(**kwargs))) as zf:
+                    zinfo = zf.infolist()[0]
+                    self.assertEqual(zinfo.file_size, expected_file_size)
+                    self.assertEqual(zinfo.compress_size, expected_compress_size)
+                    self.assertEqual(zinfo.header_offset, expected_header_offset)
+                    self.assertEqual(zf.read(zinfo), expected_content)
+
+
 @requires_zlib
 class DeflateTestZip64InSmallFiles(AbstractTestZip64InSmallFiles,
                                    unittest.TestCase):
diff --git a/Lib/zipfile.py b/Lib/zipfile.py
index e1735416ec8eb..8b99c1189baa8 100644
--- a/Lib/zipfile.py
+++ b/Lib/zipfile.py
@@ -480,14 +480,26 @@ def _decodeExtra(self):
 
                 # ZIP64 extension (large files and/or large archives)
                 if self.file_size in (0xffffffffffffffff, 0xffffffff):
+                    if len(counts) <= idx:
+                        raise BadZipFile(
+                            "Corrupt zip64 extra field. File size not found."
+                        )
                     self.file_size = counts[idx]
                     idx += 1
 
                 if self.compress_size == 0xFFFFFFFF:
+                    if len(counts) <= idx:
+                        raise BadZipFile(
+                            "Corrupt zip64 extra field. Compress size not found."
+                        )
                     self.compress_size = counts[idx]
                     idx += 1
 
                 if self.header_offset == 0xffffffff:
+                    if len(counts) <= idx:
+                        raise BadZipFile(
+                            "Corrupt zip64 extra field. Header offset not found."
+                        )
                     old = self.header_offset
                     self.header_offset = counts[idx]
                     idx+=1
diff --git a/Misc/NEWS.d/next/Library/2019-07-09-05-44-39.bpo-36993.4javqu.rst b/Misc/NEWS.d/next/Library/2019-07-09-05-44-39.bpo-36993.4javqu.rst
new file mode 100644
index 0000000000000..009e07b92d22d
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-07-09-05-44-39.bpo-36993.4javqu.rst
@@ -0,0 +1,2 @@
+Improve error reporting for corrupt zip files with bad zip64 extra data. Patch
+by Daniel Hillier.


From webhook-mailer at python.org  Tue Oct 29 04:00:51 2019
From: webhook-mailer at python.org (Larry Hastings)
Date: Tue, 29 Oct 2019 08:00:51 -0000
Subject: [Python-checkins] [3.5] bpo-31026: Fix test_dbm if dbm.ndbm is
 build with Berkeley DB. (GH-6632)
Message-ID: <mailman.181.1572336052.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/d7b336fe5d54f73c758802df426e06e8a674bd63
commit: d7b336fe5d54f73c758802df426e06e8a674bd63
branch: 3.5
author: Serhiy Storchaka <storchaka at gmail.com>
committer: Larry Hastings <larry at hastings.org>
date: 2019-10-28T22:56:33-07:00
summary:

[3.5] bpo-31026: Fix test_dbm if dbm.ndbm is build with Berkeley DB. (GH-6632)
(cherry picked from commit 70af06cdc4e8fbee0b9d7d46bdc193097d4bc71f)

files:
M Lib/test/test_dbm.py
M Lib/test/test_dbm_ndbm.py

diff --git a/Lib/test/test_dbm.py b/Lib/test/test_dbm.py
index 623d9929d58f7..81857e6f88525 100644
--- a/Lib/test/test_dbm.py
+++ b/Lib/test/test_dbm.py
@@ -39,8 +39,7 @@ def delete_files():
 
 
 class AnyDBMTestCase:
-    _dict = {'0': b'',
-             'a': b'Python:',
+    _dict = {'a': b'Python:',
              'b': b'Programming',
              'c': b'the',
              'd': b'way',
@@ -101,6 +100,20 @@ def test_anydbm_keys(self):
         keys = self.keys_helper(f)
         f.close()
 
+    def test_empty_value(self):
+        if getattr(dbm._defaultmod, 'library', None) == 'Berkeley DB':
+            self.skipTest("Berkeley DB doesn't distinguish the empty value "
+                          "from the absent one")
+        f = dbm.open(_fname, 'c')
+        self.assertEqual(f.keys(), [])
+        f[b'empty'] = b''
+        self.assertEqual(f.keys(), [b'empty'])
+        self.assertIn(b'empty', f)
+        self.assertEqual(f[b'empty'], b'')
+        self.assertEqual(f.get(b'empty'), b'')
+        self.assertEqual(f.setdefault(b'empty'), b'')
+        f.close()
+
     def test_anydbm_access(self):
         self.init_db()
         f = dbm.open(_fname, 'r')
diff --git a/Lib/test/test_dbm_ndbm.py b/Lib/test/test_dbm_ndbm.py
index 2291561defc4d..463dd38f254ec 100644
--- a/Lib/test/test_dbm_ndbm.py
+++ b/Lib/test/test_dbm_ndbm.py
@@ -29,6 +29,20 @@ def test_keys(self):
         self.assertEqual(self.d[b'bytes'], b'data')
         self.d.close()
 
+    def test_empty_value(self):
+        if dbm.ndbm.library == 'Berkeley DB':
+            self.skipTest("Berkeley DB doesn't distinguish the empty value "
+                          "from the absent one")
+        self.d = dbm.ndbm.open(self.filename, 'c')
+        self.assertEqual(self.d.keys(), [])
+        self.d['empty'] = ''
+        self.assertEqual(self.d.keys(), [b'empty'])
+        self.assertIn(b'empty', self.d)
+        self.assertEqual(self.d[b'empty'], b'')
+        self.assertEqual(self.d.get(b'empty'), b'')
+        self.assertEqual(self.d.setdefault(b'empty'), b'')
+        self.d.close()
+
     def test_modes(self):
         for mode in ['r', 'rw', 'w', 'n']:
             try:


From webhook-mailer at python.org  Tue Oct 29 05:32:52 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Tue, 29 Oct 2019 09:32:52 -0000
Subject: [Python-checkins] bpo-37330: open(): "U" mode is removed in Python
 3.9 (GH-16972)
Message-ID: <mailman.182.1572341573.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/1d2862a323ae1387e306942253b1e487018e2b7f
commit: 1d2862a323ae1387e306942253b1e487018e2b7f
branch: 3.8
author: Victor Stinner <vstinner at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-29T10:32:39+01:00
summary:

bpo-37330: open(): "U" mode is removed in Python 3.9 (GH-16972)

Update open() documentation.

files:
M Doc/library/functions.rst

diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst
index 0236ff03da074..bc46844246cf0 100644
--- a/Doc/library/functions.rst
+++ b/Doc/library/functions.rst
@@ -1245,7 +1245,7 @@ are always available.  They are listed here in alphabetical order.
 
          * The file is now non-inheritable.
 
-   .. deprecated-removed:: 3.4 4.0
+   .. deprecated-removed:: 3.4 3.9
 
       The ``'U'`` mode.
 


From webhook-mailer at python.org  Tue Oct 29 05:38:42 2019
From: webhook-mailer at python.org (Miss Skeleton (bot))
Date: Tue, 29 Oct 2019 09:38:42 -0000
Subject: [Python-checkins] bpo-37330: open(): "U" mode is removed in Python
 3.9 (GH-16972)
Message-ID: <mailman.183.1572341922.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/4cb08b6c0ae6989d169dd48c2b24087941f6d0b0
commit: 4cb08b6c0ae6989d169dd48c2b24087941f6d0b0
branch: 3.7
author: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-29T02:38:37-07:00
summary:

bpo-37330: open(): "U" mode is removed in Python 3.9 (GH-16972)


Update open() documentation.
(cherry picked from commit 1d2862a323ae1387e306942253b1e487018e2b7f)

Co-authored-by: Victor Stinner <vstinner at python.org>

files:
M Doc/library/functions.rst

diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst
index 06ba323ca30c6..ff51a4a32cc57 100644
--- a/Doc/library/functions.rst
+++ b/Doc/library/functions.rst
@@ -1173,7 +1173,7 @@ are always available.  They are listed here in alphabetical order.
 
          * The file is now non-inheritable.
 
-   .. deprecated-removed:: 3.4 4.0
+   .. deprecated-removed:: 3.4 3.9
 
       The ``'U'`` mode.
 


From webhook-mailer at python.org  Wed Oct 30 06:03:25 2019
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Wed, 30 Oct 2019 10:03:25 -0000
Subject: [Python-checkins] bpo-38600: Change the mark up of NULL in the C
 API documentation. (GH-16950)
Message-ID: <mailman.184.1572429806.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/25fc088607c855060ed142296dc1bd0125fad1af
commit: 25fc088607c855060ed142296dc1bd0125fad1af
branch: master
author: Serhiy Storchaka <storchaka at gmail.com>
committer: GitHub <noreply at github.com>
date: 2019-10-30T12:03:20+02:00
summary:

bpo-38600: Change the mark up of NULL in the C API documentation. (GH-16950)

Replace all *NULL* with ``NULL``.

files:
M Doc/c-api/arg.rst
M Doc/c-api/buffer.rst
M Doc/c-api/bytearray.rst
M Doc/c-api/bytes.rst
M Doc/c-api/capsule.rst
M Doc/c-api/cell.rst
M Doc/c-api/codec.rst
M Doc/c-api/concrete.rst
M Doc/c-api/contextvars.rst
M Doc/c-api/conversion.rst
M Doc/c-api/coro.rst
M Doc/c-api/datetime.rst
M Doc/c-api/dict.rst
M Doc/c-api/exceptions.rst
M Doc/c-api/file.rst
M Doc/c-api/float.rst
M Doc/c-api/function.rst
M Doc/c-api/gcsupport.rst
M Doc/c-api/gen.rst
M Doc/c-api/import.rst
M Doc/c-api/init.rst
M Doc/c-api/intro.rst
M Doc/c-api/iter.rst
M Doc/c-api/list.rst
M Doc/c-api/long.rst
M Doc/c-api/mapping.rst
M Doc/c-api/marshal.rst
M Doc/c-api/memory.rst
M Doc/c-api/memoryview.rst
M Doc/c-api/method.rst
M Doc/c-api/module.rst
M Doc/c-api/number.rst
M Doc/c-api/object.rst
M Doc/c-api/refcounting.rst
M Doc/c-api/reflection.rst
M Doc/c-api/sequence.rst
M Doc/c-api/set.rst
M Doc/c-api/slice.rst
M Doc/c-api/structures.rst
M Doc/c-api/sys.rst
M Doc/c-api/tuple.rst
M Doc/c-api/type.rst
M Doc/c-api/typeobj.rst
M Doc/c-api/unicode.rst
M Doc/c-api/veryhigh.rst
M Doc/c-api/weakref.rst
M Doc/extending/embedding.rst
M Doc/extending/extending.rst
M Doc/extending/newtypes.rst
M Doc/extending/newtypes_tutorial.rst
M Doc/faq/extending.rst
M Doc/library/ctypes.rst
M Doc/whatsnew/3.4.rst
M Doc/whatsnew/3.7.rst
M Include/abstract.h

diff --git a/Doc/c-api/arg.rst b/Doc/c-api/arg.rst
index ba9ca5e0d30b9..594fef29f2a33 100644
--- a/Doc/c-api/arg.rst
+++ b/Doc/c-api/arg.rst
@@ -50,7 +50,7 @@ Unless otherwise stated, buffers are not NUL-terminated.
 
 Some formats require a read-only :term:`bytes-like object`, and set a
 pointer instead of a buffer structure.  They work by checking that
-the object's :c:member:`PyBufferProcs.bf_releasebuffer` field is *NULL*,
+the object's :c:member:`PyBufferProcs.bf_releasebuffer` field is ``NULL``,
 which disallows mutable objects such as :class:`bytearray`.
 
 .. note::
@@ -99,15 +99,15 @@ which disallows mutable objects such as :class:`bytearray`.
 
 ``z`` (:class:`str` or ``None``) [const char \*]
    Like ``s``, but the Python object may also be ``None``, in which case the C
-   pointer is set to *NULL*.
+   pointer is set to ``NULL``.
 
 ``z*`` (:class:`str`, :term:`bytes-like object` or ``None``) [Py_buffer]
    Like ``s*``, but the Python object may also be ``None``, in which case the
-   ``buf`` member of the :c:type:`Py_buffer` structure is set to *NULL*.
+   ``buf`` member of the :c:type:`Py_buffer` structure is set to ``NULL``.
 
 ``z#`` (:class:`str`, read-only :term:`bytes-like object` or ``None``) [const char \*, int]
    Like ``s#``, but the Python object may also be ``None``, in which case the C
-   pointer is set to *NULL*.
+   pointer is set to ``NULL``.
 
 ``y`` (read-only :term:`bytes-like object`) [const char \*]
    This format converts a bytes-like object to a C pointer to a character
@@ -166,7 +166,7 @@ which disallows mutable objects such as :class:`bytearray`.
 
 ``Z`` (:class:`str` or ``None``) [const Py_UNICODE \*]
    Like ``u``, but the Python object may also be ``None``, in which case the
-   :c:type:`Py_UNICODE` pointer is set to *NULL*.
+   :c:type:`Py_UNICODE` pointer is set to ``NULL``.
 
    .. deprecated-removed:: 3.3 4.0
       Part of the old-style :c:type:`Py_UNICODE` API; please migrate to using
@@ -174,7 +174,7 @@ which disallows mutable objects such as :class:`bytearray`.
 
 ``Z#`` (:class:`str` or ``None``) [const Py_UNICODE \*, int]
    Like ``u#``, but the Python object may also be ``None``, in which case the
-   :c:type:`Py_UNICODE` pointer is set to *NULL*.
+   :c:type:`Py_UNICODE` pointer is set to ``NULL``.
 
    .. deprecated-removed:: 3.3 4.0
       Part of the old-style :c:type:`Py_UNICODE` API; please migrate to using
@@ -197,7 +197,7 @@ which disallows mutable objects such as :class:`bytearray`.
 
    This format requires two arguments.  The first is only used as input, and
    must be a :c:type:`const char\*` which points to the name of an encoding as a
-   NUL-terminated string, or *NULL*, in which case ``'utf-8'`` encoding is used.
+   NUL-terminated string, or ``NULL``, in which case ``'utf-8'`` encoding is used.
    An exception is raised if the named encoding is not known to Python.  The
    second argument must be a :c:type:`char\*\*`; the value of the pointer it
    references will be set to a buffer with the contents of the argument text.
@@ -220,7 +220,7 @@ which disallows mutable objects such as :class:`bytearray`.
 
    It requires three arguments.  The first is only used as input, and must be a
    :c:type:`const char\*` which points to the name of an encoding as a
-   NUL-terminated string, or *NULL*, in which case ``'utf-8'`` encoding is used.
+   NUL-terminated string, or ``NULL``, in which case ``'utf-8'`` encoding is used.
    An exception is raised if the named encoding is not known to Python.  The
    second argument must be a :c:type:`char\*\*`; the value of the pointer it
    references will be set to a buffer with the contents of the argument text.
@@ -230,12 +230,12 @@ which disallows mutable objects such as :class:`bytearray`.
 
    There are two modes of operation:
 
-   If *\*buffer* points a *NULL* pointer, the function will allocate a buffer of
+   If *\*buffer* points a ``NULL`` pointer, the function will allocate a buffer of
    the needed size, copy the encoded data into this buffer and set *\*buffer* to
    reference the newly allocated storage.  The caller is responsible for calling
    :c:func:`PyMem_Free` to free the allocated buffer after usage.
 
-   If *\*buffer* points to a non-*NULL* pointer (an already allocated buffer),
+   If *\*buffer* points to a non-``NULL`` pointer (an already allocated buffer),
    :c:func:`PyArg_ParseTuple` will use this location as the buffer and interpret the
    initial value of *\*buffer_length* as the buffer size.  It will then copy the
    encoded data into the buffer and NUL-terminate it.  If the buffer is not large
@@ -317,7 +317,7 @@ Other objects
 ``O`` (object) [PyObject \*]
    Store a Python object (without any conversion) in a C object pointer.  The C
    program thus receives the actual object that was passed.  The object's reference
-   count is not increased.  The pointer stored is not *NULL*.
+   count is not increased.  The pointer stored is not ``NULL``.
 
 ``O!`` (object) [*typeobject*, PyObject \*]
    Store a Python object in a C object pointer.  This is similar to ``O``, but
@@ -437,7 +437,7 @@ API Functions
 
    Parse the parameters of a function that takes both positional and keyword
    parameters into local variables.  The *keywords* argument is a
-   *NULL*-terminated array of keyword parameter names.  Empty names denote
+   ``NULL``-terminated array of keyword parameter names.  Empty names denote
    :ref:`positional-only parameters <positional-only_parameter>`.
    Returns true on success; on failure, it returns false and raises the
    appropriate exception.
@@ -520,8 +520,8 @@ Building values
 
    Create a new value based on a format string similar to those accepted by the
    :c:func:`PyArg_Parse\*` family of functions and a sequence of values.  Returns
-   the value or *NULL* in the case of an error; an exception will be raised if
-   *NULL* is returned.
+   the value or ``NULL`` in the case of an error; an exception will be raised if
+   ``NULL`` is returned.
 
    :c:func:`Py_BuildValue` does not always build a tuple.  It builds a tuple only if
    its format string contains two or more format units.  If the format string is
@@ -547,20 +547,20 @@ Building values
 
    ``s`` (:class:`str` or ``None``) [const char \*]
       Convert a null-terminated C string to a Python :class:`str` object using ``'utf-8'``
-      encoding. If the C string pointer is *NULL*, ``None`` is used.
+      encoding. If the C string pointer is ``NULL``, ``None`` is used.
 
    ``s#`` (:class:`str` or ``None``) [const char \*, int]
       Convert a C string and its length to a Python :class:`str` object using ``'utf-8'``
-      encoding. If the C string pointer is *NULL*, the length is ignored and
+      encoding. If the C string pointer is ``NULL``, the length is ignored and
       ``None`` is returned.
 
    ``y`` (:class:`bytes`) [const char \*]
       This converts a C string to a Python :class:`bytes` object.  If the C
-      string pointer is *NULL*, ``None`` is returned.
+      string pointer is ``NULL``, ``None`` is returned.
 
    ``y#`` (:class:`bytes`) [const char \*, int]
       This converts a C string and its lengths to a Python object.  If the C
-      string pointer is *NULL*, ``None`` is returned.
+      string pointer is ``NULL``, ``None`` is returned.
 
    ``z`` (:class:`str` or ``None``) [const char \*]
       Same as ``s``.
@@ -570,12 +570,12 @@ Building values
 
    ``u`` (:class:`str`) [const wchar_t \*]
       Convert a null-terminated :c:type:`wchar_t` buffer of Unicode (UTF-16 or UCS-4)
-      data to a Python Unicode object.  If the Unicode buffer pointer is *NULL*,
+      data to a Python Unicode object.  If the Unicode buffer pointer is ``NULL``,
       ``None`` is returned.
 
    ``u#`` (:class:`str`) [const wchar_t \*, int]
       Convert a Unicode (UTF-16 or UCS-4) data buffer and its length to a Python
-      Unicode object.   If the Unicode buffer pointer is *NULL*, the length is ignored
+      Unicode object.   If the Unicode buffer pointer is ``NULL``, the length is ignored
       and ``None`` is returned.
 
    ``U`` (:class:`str` or ``None``) [const char \*]
@@ -636,9 +636,9 @@ Building values
 
    ``O`` (object) [PyObject \*]
       Pass a Python object untouched (except for its reference count, which is
-      incremented by one).  If the object passed in is a *NULL* pointer, it is assumed
+      incremented by one).  If the object passed in is a ``NULL`` pointer, it is assumed
       that this was caused because the call producing the argument found an error and
-      set an exception. Therefore, :c:func:`Py_BuildValue` will return *NULL* but won't
+      set an exception. Therefore, :c:func:`Py_BuildValue` will return ``NULL`` but won't
       raise an exception.  If no exception has been raised yet, :exc:`SystemError` is
       set.
 
@@ -653,7 +653,7 @@ Building values
    ``O&`` (object) [*converter*, *anything*]
       Convert *anything* to a Python object through a *converter* function.  The
       function is called with *anything* (which should be compatible with :c:type:`void
-      \*`) as its argument and should return a "new" Python object, or *NULL* if an
+      \*`) as its argument and should return a "new" Python object, or ``NULL`` if an
       error occurred.
 
    ``(items)`` (:class:`tuple`) [*matching-items*]
@@ -668,7 +668,7 @@ Building values
       respectively.
 
    If there is an error in the format string, the :exc:`SystemError` exception is
-   set and *NULL* returned.
+   set and ``NULL`` returned.
 
 .. c:function:: PyObject* Py_VaBuildValue(const char *format, va_list vargs)
 
diff --git a/Doc/c-api/buffer.rst b/Doc/c-api/buffer.rst
index e660716856b6d..f37b0db07cd70 100644
--- a/Doc/c-api/buffer.rst
+++ b/Doc/c-api/buffer.rst
@@ -102,13 +102,13 @@ a buffer, see :c:func:`PyObject_GetBuffer`.
    .. c:member:: void \*obj
 
       A new reference to the exporting object. The reference is owned by
-      the consumer and automatically decremented and set to *NULL* by
+      the consumer and automatically decremented and set to ``NULL`` by
       :c:func:`PyBuffer_Release`. The field is the equivalent of the return
       value of any standard C-API function.
 
       As a special case, for *temporary* buffers that are wrapped by
       :c:func:`PyMemoryView_FromBuffer` or :c:func:`PyBuffer_FillInfo`
-      this field is *NULL*. In general, exporting objects MUST NOT
+      this field is ``NULL``. In general, exporting objects MUST NOT
       use this scheme.
 
    .. c:member:: Py_ssize_t len
@@ -134,21 +134,21 @@ a buffer, see :c:func:`PyObject_GetBuffer`.
 
       Important exception: If a consumer requests a buffer without the
       :c:macro:`PyBUF_FORMAT` flag, :c:member:`~Py_buffer.format` will
-      be set to  *NULL*,  but :c:member:`~Py_buffer.itemsize` still has
+      be set to  ``NULL``,  but :c:member:`~Py_buffer.itemsize` still has
       the value for the original format.
 
       If :c:member:`~Py_buffer.shape` is present, the equality
       ``product(shape) * itemsize == len`` still holds and the consumer
       can use :c:member:`~Py_buffer.itemsize` to navigate the buffer.
 
-      If :c:member:`~Py_buffer.shape` is *NULL* as a result of a :c:macro:`PyBUF_SIMPLE`
+      If :c:member:`~Py_buffer.shape` is ``NULL`` as a result of a :c:macro:`PyBUF_SIMPLE`
       or a :c:macro:`PyBUF_WRITABLE` request, the consumer must disregard
       :c:member:`~Py_buffer.itemsize` and assume ``itemsize == 1``.
 
    .. c:member:: const char \*format
 
       A *NUL* terminated string in :mod:`struct` module style syntax describing
-      the contents of a single item. If this is *NULL*, ``"B"`` (unsigned bytes)
+      the contents of a single item. If this is ``NULL``, ``"B"`` (unsigned bytes)
       is assumed.
 
       This field is controlled by the :c:macro:`PyBUF_FORMAT` flag.
@@ -158,7 +158,7 @@ a buffer, see :c:func:`PyObject_GetBuffer`.
       The number of dimensions the memory represents as an n-dimensional array.
       If it is ``0``, :c:member:`~Py_buffer.buf` points to a single item representing
       a scalar. In this case, :c:member:`~Py_buffer.shape`, :c:member:`~Py_buffer.strides`
-      and :c:member:`~Py_buffer.suboffsets` MUST be *NULL*.
+      and :c:member:`~Py_buffer.suboffsets` MUST be ``NULL``.
 
       The macro :c:macro:`PyBUF_MAX_NDIM` limits the maximum number of dimensions
       to 64. Exporters MUST respect this limit, consumers of multi-dimensional
@@ -248,7 +248,7 @@ readonly, format
    .. c:macro:: PyBUF_FORMAT
 
       Controls the :c:member:`~Py_buffer.format` field. If set, this field MUST
-      be filled in correctly. Otherwise, this field MUST be *NULL*.
+      be filled in correctly. Otherwise, this field MUST be ``NULL``.
 
 
 :c:macro:`PyBUF_WRITABLE` can be \|'d to any of the flags in the next section.
@@ -349,9 +349,9 @@ The logical structure of NumPy-style arrays is defined by :c:member:`~Py_buffer.
 
 If ``ndim == 0``, the memory location pointed to by :c:member:`~Py_buffer.buf` is
 interpreted as a scalar of size :c:member:`~Py_buffer.itemsize`. In that case,
-both :c:member:`~Py_buffer.shape` and :c:member:`~Py_buffer.strides` are *NULL*.
+both :c:member:`~Py_buffer.shape` and :c:member:`~Py_buffer.strides` are ``NULL``.
 
-If :c:member:`~Py_buffer.strides` is *NULL*, the array is interpreted as
+If :c:member:`~Py_buffer.strides` is ``NULL``, the array is interpreted as
 a standard n-dimensional C-array. Otherwise, the consumer must access an
 n-dimensional array as follows:
 
@@ -438,7 +438,7 @@ Buffer-related functions
 
    Send a request to *exporter* to fill in *view* as specified by  *flags*.
    If the exporter cannot provide a buffer of the exact type, it MUST raise
-   :c:data:`PyExc_BufferError`, set :c:member:`view->obj` to *NULL* and
+   :c:data:`PyExc_BufferError`, set :c:member:`view->obj` to ``NULL`` and
    return ``-1``.
 
    On success, fill in *view*, set :c:member:`view->obj` to a new reference
@@ -518,7 +518,7 @@ Buffer-related functions
 
    On success, set :c:member:`view->obj` to a new reference to *exporter* and
    return 0. Otherwise, raise :c:data:`PyExc_BufferError`, set
-   :c:member:`view->obj` to *NULL* and return ``-1``;
+   :c:member:`view->obj` to ``NULL`` and return ``-1``;
 
    If this function is used as part of a :ref:`getbufferproc <buffer-structs>`,
    *exporter* MUST be set to the exporting object and *flags* must be passed
diff --git a/Doc/c-api/bytearray.rst b/Doc/c-api/bytearray.rst
index b4a9660a9169e..b2f409c15abb7 100644
--- a/Doc/c-api/bytearray.rst
+++ b/Doc/c-api/bytearray.rst
@@ -48,7 +48,7 @@ Direct API functions
 .. c:function:: PyObject* PyByteArray_FromStringAndSize(const char *string, Py_ssize_t len)
 
    Create a new bytearray object from *string* and its length, *len*.  On
-   failure, *NULL* is returned.
+   failure, ``NULL`` is returned.
 
 
 .. c:function:: PyObject* PyByteArray_Concat(PyObject *a, PyObject *b)
@@ -58,13 +58,13 @@ Direct API functions
 
 .. c:function:: Py_ssize_t PyByteArray_Size(PyObject *bytearray)
 
-   Return the size of *bytearray* after checking for a *NULL* pointer.
+   Return the size of *bytearray* after checking for a ``NULL`` pointer.
 
 
 .. c:function:: char* PyByteArray_AsString(PyObject *bytearray)
 
    Return the contents of *bytearray* as a char array after checking for a
-   *NULL* pointer.  The returned array always has an extra
+   ``NULL`` pointer.  The returned array always has an extra
    null byte appended.
 
 
diff --git a/Doc/c-api/bytes.rst b/Doc/c-api/bytes.rst
index 9a62fb682d689..0e33ed2c7c940 100644
--- a/Doc/c-api/bytes.rst
+++ b/Doc/c-api/bytes.rst
@@ -37,14 +37,14 @@ called with a non-bytes parameter.
 .. c:function:: PyObject* PyBytes_FromString(const char *v)
 
    Return a new bytes object with a copy of the string *v* as value on success,
-   and *NULL* on failure.  The parameter *v* must not be *NULL*; it will not be
+   and ``NULL`` on failure.  The parameter *v* must not be ``NULL``; it will not be
    checked.
 
 
 .. c:function:: PyObject* PyBytes_FromStringAndSize(const char *v, Py_ssize_t len)
 
    Return a new bytes object with a copy of the string *v* as value and length
-   *len* on success, and *NULL* on failure.  If *v* is *NULL*, the contents of
+   *len* on success, and ``NULL`` on failure.  If *v* is ``NULL``, the contents of
    the bytes object are uninitialized.
 
 
@@ -145,7 +145,7 @@ called with a non-bytes parameter.
    whether there are any other null bytes.  The data must not be
    modified in any way, unless the object was just created using
    ``PyBytes_FromStringAndSize(NULL, size)``. It must not be deallocated.  If
-   *o* is not a bytes object at all, :c:func:`PyBytes_AsString` returns *NULL*
+   *o* is not a bytes object at all, :c:func:`PyBytes_AsString` returns ``NULL``
    and raises :exc:`TypeError`.
 
 
@@ -159,7 +159,7 @@ called with a non-bytes parameter.
    Return the null-terminated contents of the object *obj*
    through the output variables *buffer* and *length*.
 
-   If *length* is *NULL*, the bytes object
+   If *length* is ``NULL``, the bytes object
    may not contain embedded null bytes;
    if it does, the function returns ``-1`` and a :exc:`ValueError` is raised.
 
@@ -181,7 +181,7 @@ called with a non-bytes parameter.
    appended to *bytes*; the caller will own the new reference.  The reference to
    the old value of *bytes* will be stolen.  If the new object cannot be
    created, the old reference to *bytes* will still be discarded and the value
-   of *\*bytes* will be set to *NULL*; the appropriate exception will be set.
+   of *\*bytes* will be set to ``NULL``; the appropriate exception will be set.
 
 
 .. c:function:: void PyBytes_ConcatAndDel(PyObject **bytes, PyObject *newpart)
@@ -201,5 +201,5 @@ called with a non-bytes parameter.
    desired.  On success, *\*bytes* holds the resized bytes object and ``0`` is
    returned; the address in *\*bytes* may differ from its input value.  If the
    reallocation fails, the original bytes object at *\*bytes* is deallocated,
-   *\*bytes* is set to *NULL*, :exc:`MemoryError` is set, and ``-1`` is
+   *\*bytes* is set to ``NULL``, :exc:`MemoryError` is set, and ``-1`` is
    returned.
diff --git a/Doc/c-api/capsule.rst b/Doc/c-api/capsule.rst
index 3c921bbde3486..78e21140b2f80 100644
--- a/Doc/c-api/capsule.rst
+++ b/Doc/c-api/capsule.rst
@@ -40,15 +40,15 @@ Refer to :ref:`using-capsules` for more information on using these objects.
 .. c:function:: PyObject* PyCapsule_New(void *pointer, const char *name, PyCapsule_Destructor destructor)
 
    Create a :c:type:`PyCapsule` encapsulating the *pointer*.  The *pointer*
-   argument may not be *NULL*.
+   argument may not be ``NULL``.
 
-   On failure, set an exception and return *NULL*.
+   On failure, set an exception and return ``NULL``.
 
-   The *name* string may either be *NULL* or a pointer to a valid C string.  If
-   non-*NULL*, this string must outlive the capsule.  (Though it is permitted to
+   The *name* string may either be ``NULL`` or a pointer to a valid C string.  If
+   non-``NULL``, this string must outlive the capsule.  (Though it is permitted to
    free it inside the *destructor*.)
 
-   If the *destructor* argument is not *NULL*, it will be called with the
+   If the *destructor* argument is not ``NULL``, it will be called with the
    capsule as its argument when it is destroyed.
 
    If this capsule will be stored as an attribute of a module, the *name* should
@@ -59,20 +59,20 @@ Refer to :ref:`using-capsules` for more information on using these objects.
 .. c:function:: void* PyCapsule_GetPointer(PyObject *capsule, const char *name)
 
    Retrieve the *pointer* stored in the capsule.  On failure, set an exception
-   and return *NULL*.
+   and return ``NULL``.
 
    The *name* parameter must compare exactly to the name stored in the capsule.
-   If the name stored in the capsule is *NULL*, the *name* passed in must also
-   be *NULL*.  Python uses the C function :c:func:`strcmp` to compare capsule
+   If the name stored in the capsule is ``NULL``, the *name* passed in must also
+   be ``NULL``.  Python uses the C function :c:func:`strcmp` to compare capsule
    names.
 
 
 .. c:function:: PyCapsule_Destructor PyCapsule_GetDestructor(PyObject *capsule)
 
    Return the current destructor stored in the capsule.  On failure, set an
-   exception and return *NULL*.
+   exception and return ``NULL``.
 
-   It is legal for a capsule to have a *NULL* destructor.  This makes a *NULL*
+   It is legal for a capsule to have a ``NULL`` destructor.  This makes a ``NULL``
    return code somewhat ambiguous; use :c:func:`PyCapsule_IsValid` or
    :c:func:`PyErr_Occurred` to disambiguate.
 
@@ -80,9 +80,9 @@ Refer to :ref:`using-capsules` for more information on using these objects.
 .. c:function:: void* PyCapsule_GetContext(PyObject *capsule)
 
    Return the current context stored in the capsule.  On failure, set an
-   exception and return *NULL*.
+   exception and return ``NULL``.
 
-   It is legal for a capsule to have a *NULL* context.  This makes a *NULL*
+   It is legal for a capsule to have a ``NULL`` context.  This makes a ``NULL``
    return code somewhat ambiguous; use :c:func:`PyCapsule_IsValid` or
    :c:func:`PyErr_Occurred` to disambiguate.
 
@@ -90,9 +90,9 @@ Refer to :ref:`using-capsules` for more information on using these objects.
 .. c:function:: const char* PyCapsule_GetName(PyObject *capsule)
 
    Return the current name stored in the capsule.  On failure, set an exception
-   and return *NULL*.
+   and return ``NULL``.
 
-   It is legal for a capsule to have a *NULL* name.  This makes a *NULL* return
+   It is legal for a capsule to have a ``NULL`` name.  This makes a ``NULL`` return
    code somewhat ambiguous; use :c:func:`PyCapsule_IsValid` or
    :c:func:`PyErr_Occurred` to disambiguate.
 
@@ -107,13 +107,13 @@ Refer to :ref:`using-capsules` for more information on using these objects.
    import the module conventionally (using :c:func:`PyImport_ImportModule`).
 
    Return the capsule's internal *pointer* on success.  On failure, set an
-   exception and return *NULL*.
+   exception and return ``NULL``.
 
 
 .. c:function:: int PyCapsule_IsValid(PyObject *capsule, const char *name)
 
    Determines whether or not *capsule* is a valid capsule.  A valid capsule is
-   non-*NULL*, passes :c:func:`PyCapsule_CheckExact`, has a non-*NULL* pointer
+   non-``NULL``, passes :c:func:`PyCapsule_CheckExact`, has a non-``NULL`` pointer
    stored in it, and its internal name matches the *name* parameter.  (See
    :c:func:`PyCapsule_GetPointer` for information on how capsule names are
    compared.)
@@ -142,9 +142,9 @@ Refer to :ref:`using-capsules` for more information on using these objects.
 
 .. c:function:: int PyCapsule_SetName(PyObject *capsule, const char *name)
 
-   Set the name inside *capsule* to *name*.  If non-*NULL*, the name must
+   Set the name inside *capsule* to *name*.  If non-``NULL``, the name must
    outlive the capsule.  If the previous *name* stored in the capsule was not
-   *NULL*, no attempt is made to free it.
+   ``NULL``, no attempt is made to free it.
 
    Return ``0`` on success.  Return nonzero and set an exception on failure.
 
@@ -152,6 +152,6 @@ Refer to :ref:`using-capsules` for more information on using these objects.
 .. c:function:: int PyCapsule_SetPointer(PyObject *capsule, void *pointer)
 
    Set the void pointer inside *capsule* to *pointer*.  The pointer may not be
-   *NULL*.
+   ``NULL``.
 
    Return ``0`` on success.  Return nonzero and set an exception on failure.
diff --git a/Doc/c-api/cell.rst b/Doc/c-api/cell.rst
index 8514afe928e7d..8408f7e398db7 100644
--- a/Doc/c-api/cell.rst
+++ b/Doc/c-api/cell.rst
@@ -27,13 +27,13 @@ Cell objects are not likely to be useful elsewhere.
 
 .. c:function:: int PyCell_Check(ob)
 
-   Return true if *ob* is a cell object; *ob* must not be *NULL*.
+   Return true if *ob* is a cell object; *ob* must not be ``NULL``.
 
 
 .. c:function:: PyObject* PyCell_New(PyObject *ob)
 
    Create and return a new cell object containing the value *ob*. The parameter may
-   be *NULL*.
+   be ``NULL``.
 
 
 .. c:function:: PyObject* PyCell_Get(PyObject *cell)
@@ -44,19 +44,19 @@ Cell objects are not likely to be useful elsewhere.
 .. c:function:: PyObject* PyCell_GET(PyObject *cell)
 
    Return the contents of the cell *cell*, but without checking that *cell* is
-   non-*NULL* and a cell object.
+   non-``NULL`` and a cell object.
 
 
 .. c:function:: int PyCell_Set(PyObject *cell, PyObject *value)
 
    Set the contents of the cell object *cell* to *value*.  This releases the
-   reference to any current content of the cell. *value* may be *NULL*.  *cell*
-   must be non-*NULL*; if it is not a cell object, ``-1`` will be returned.  On
+   reference to any current content of the cell. *value* may be ``NULL``.  *cell*
+   must be non-``NULL``; if it is not a cell object, ``-1`` will be returned.  On
    success, ``0`` will be returned.
 
 
 .. c:function:: void PyCell_SET(PyObject *cell, PyObject *value)
 
    Sets the value of the cell object *cell* to *value*.  No reference counts are
-   adjusted, and no checks are made for safety; *cell* must be non-*NULL* and must
+   adjusted, and no checks are made for safety; *cell* must be non-``NULL`` and must
    be a cell object.
diff --git a/Doc/c-api/codec.rst b/Doc/c-api/codec.rst
index c55f19970e125..172dcb326a4bc 100644
--- a/Doc/c-api/codec.rst
+++ b/Doc/c-api/codec.rst
@@ -21,7 +21,7 @@ Codec registry and support functions
 
    *object* is passed through the encoder function found for the given
    *encoding* using the error handling method defined by *errors*.  *errors* may
-   be *NULL* to use the default method defined for the codec.  Raises a
+   be ``NULL`` to use the default method defined for the codec.  Raises a
    :exc:`LookupError` if no encoder can be found.
 
 .. c:function:: PyObject* PyCodec_Decode(PyObject *object, const char *encoding, const char *errors)
@@ -30,7 +30,7 @@ Codec registry and support functions
 
    *object* is passed through the decoder function found for the given
    *encoding* using the error handling method defined by *errors*.  *errors* may
-   be *NULL* to use the default method defined for the codec.  Raises a
+   be ``NULL`` to use the default method defined for the codec.  Raises a
    :exc:`LookupError` if no encoder can be found.
 
 
@@ -40,7 +40,7 @@ Codec lookup API
 In the following functions, the *encoding* string is looked up converted to all
 lower-case characters, which makes encodings looked up through this mechanism
 effectively case-insensitive.  If no codec is found, a :exc:`KeyError` is set
-and *NULL* returned.
+and ``NULL`` returned.
 
 .. c:function:: PyObject* PyCodec_Encoder(const char *encoding)
 
@@ -92,7 +92,7 @@ Registry API for Unicode encoding error handlers
 .. c:function:: PyObject* PyCodec_LookupError(const char *name)
 
    Lookup the error handling callback function registered under *name*.  As a
-   special case *NULL* can be passed, in which case the error handling callback
+   special case ``NULL`` can be passed, in which case the error handling callback
    for "strict" will be returned.
 
 .. c:function:: PyObject* PyCodec_StrictErrors(PyObject *exc)
diff --git a/Doc/c-api/concrete.rst b/Doc/c-api/concrete.rst
index f83d711795c2c..c1d9fa1b41a3f 100644
--- a/Doc/c-api/concrete.rst
+++ b/Doc/c-api/concrete.rst
@@ -17,8 +17,8 @@ dictionary, use :c:func:`PyDict_Check`.  The chapter is structured like the
 .. warning::
 
    While the functions described in this chapter carefully check the type of the
-   objects which are passed in, many of them do not check for *NULL* being passed
-   instead of a valid object.  Allowing *NULL* to be passed in can cause memory
+   objects which are passed in, many of them do not check for ``NULL`` being passed
+   instead of a valid object.  Allowing ``NULL`` to be passed in can cause memory
    access violations and immediate termination of the interpreter.
 
 
diff --git a/Doc/c-api/contextvars.rst b/Doc/c-api/contextvars.rst
index a7cde7fb1958a..38256a3b0f2a0 100644
--- a/Doc/c-api/contextvars.rst
+++ b/Doc/c-api/contextvars.rst
@@ -60,17 +60,17 @@ Type-check macros:
 .. c:function:: int PyContext_CheckExact(PyObject *o)
 
    Return true if *o* is of type :c:data:`PyContext_Type`. *o* must not be
-   *NULL*.  This function always succeeds.
+   ``NULL``.  This function always succeeds.
 
 .. c:function:: int PyContextVar_CheckExact(PyObject *o)
 
    Return true if *o* is of type :c:data:`PyContextVar_Type`. *o* must not be
-   *NULL*.  This function always succeeds.
+   ``NULL``.  This function always succeeds.
 
 .. c:function:: int PyContextToken_CheckExact(PyObject *o)
 
    Return true if *o* is of type :c:data:`PyContextToken_Type`.
-   *o* must not be *NULL*.  This function always succeeds.
+   *o* must not be ``NULL``.  This function always succeeds.
 
 
 Context object management functions:
diff --git a/Doc/c-api/conversion.rst b/Doc/c-api/conversion.rst
index ed101c791ec74..c02fa60ae9a7c 100644
--- a/Doc/c-api/conversion.rst
+++ b/Doc/c-api/conversion.rst
@@ -113,7 +113,7 @@ The following functions provide locale-independent string to number conversions.
    *val* is a finite number, an infinite number, or not a number, respectively.
 
    The return value is a pointer to *buffer* with the converted string or
-   *NULL* if the conversion failed. The caller is responsible for freeing the
+   ``NULL`` if the conversion failed. The caller is responsible for freeing the
    returned string by calling :c:func:`PyMem_Free`.
 
    .. versionadded:: 3.1
diff --git a/Doc/c-api/coro.rst b/Doc/c-api/coro.rst
index 50428c7eb43e3..915c57eeb55b8 100644
--- a/Doc/c-api/coro.rst
+++ b/Doc/c-api/coro.rst
@@ -23,7 +23,7 @@ return.
 
 .. c:function:: int PyCoro_CheckExact(PyObject *ob)
 
-   Return true if *ob*'s type is *PyCoro_Type*; *ob* must not be *NULL*.
+   Return true if *ob*'s type is *PyCoro_Type*; *ob* must not be ``NULL``.
 
 
 .. c:function:: PyObject* PyCoro_New(PyFrameObject *frame, PyObject *name, PyObject *qualname)
@@ -31,4 +31,4 @@ return.
    Create and return a new coroutine object based on the *frame* object,
    with ``__name__`` and ``__qualname__`` set to *name* and *qualname*.
    A reference to *frame* is stolen by this function.  The *frame* argument
-   must not be *NULL*.
+   must not be ``NULL``.
diff --git a/Doc/c-api/datetime.rst b/Doc/c-api/datetime.rst
index 44a04373d1968..bd4f1ff446bcf 100644
--- a/Doc/c-api/datetime.rst
+++ b/Doc/c-api/datetime.rst
@@ -28,61 +28,61 @@ Type-check macros:
 .. c:function:: int PyDate_Check(PyObject *ob)
 
    Return true if *ob* is of type :c:data:`PyDateTime_DateType` or a subtype of
-   :c:data:`PyDateTime_DateType`.  *ob* must not be *NULL*.
+   :c:data:`PyDateTime_DateType`.  *ob* must not be ``NULL``.
 
 
 .. c:function:: int PyDate_CheckExact(PyObject *ob)
 
    Return true if *ob* is of type :c:data:`PyDateTime_DateType`. *ob* must not be
-   *NULL*.
+   ``NULL``.
 
 
 .. c:function:: int PyDateTime_Check(PyObject *ob)
 
    Return true if *ob* is of type :c:data:`PyDateTime_DateTimeType` or a subtype of
-   :c:data:`PyDateTime_DateTimeType`.  *ob* must not be *NULL*.
+   :c:data:`PyDateTime_DateTimeType`.  *ob* must not be ``NULL``.
 
 
 .. c:function:: int PyDateTime_CheckExact(PyObject *ob)
 
    Return true if *ob* is of type :c:data:`PyDateTime_DateTimeType`. *ob* must not
-   be *NULL*.
+   be ``NULL``.
 
 
 .. c:function:: int PyTime_Check(PyObject *ob)
 
    Return true if *ob* is of type :c:data:`PyDateTime_TimeType` or a subtype of
-   :c:data:`PyDateTime_TimeType`.  *ob* must not be *NULL*.
+   :c:data:`PyDateTime_TimeType`.  *ob* must not be ``NULL``.
 
 
 .. c:function:: int PyTime_CheckExact(PyObject *ob)
 
    Return true if *ob* is of type :c:data:`PyDateTime_TimeType`. *ob* must not be
-   *NULL*.
+   ``NULL``.
 
 
 .. c:function:: int PyDelta_Check(PyObject *ob)
 
    Return true if *ob* is of type :c:data:`PyDateTime_DeltaType` or a subtype of
-   :c:data:`PyDateTime_DeltaType`.  *ob* must not be *NULL*.
+   :c:data:`PyDateTime_DeltaType`.  *ob* must not be ``NULL``.
 
 
 .. c:function:: int PyDelta_CheckExact(PyObject *ob)
 
    Return true if *ob* is of type :c:data:`PyDateTime_DeltaType`. *ob* must not be
-   *NULL*.
+   ``NULL``.
 
 
 .. c:function:: int PyTZInfo_Check(PyObject *ob)
 
    Return true if *ob* is of type :c:data:`PyDateTime_TZInfoType` or a subtype of
-   :c:data:`PyDateTime_TZInfoType`.  *ob* must not be *NULL*.
+   :c:data:`PyDateTime_TZInfoType`.  *ob* must not be ``NULL``.
 
 
 .. c:function:: int PyTZInfo_CheckExact(PyObject *ob)
 
    Return true if *ob* is of type :c:data:`PyDateTime_TZInfoType`. *ob* must not be
-   *NULL*.
+   ``NULL``.
 
 
 Macros to create objects:
@@ -144,7 +144,7 @@ Macros to create objects:
 
 Macros to extract fields from date objects.  The argument must be an instance of
 :c:data:`PyDateTime_Date`, including subclasses (such as
-:c:data:`PyDateTime_DateTime`).  The argument must not be *NULL*, and the type is
+:c:data:`PyDateTime_DateTime`).  The argument must not be ``NULL``, and the type is
 not checked:
 
 .. c:function:: int PyDateTime_GET_YEAR(PyDateTime_Date *o)
@@ -164,7 +164,7 @@ not checked:
 
 Macros to extract fields from datetime objects.  The argument must be an
 instance of :c:data:`PyDateTime_DateTime`, including subclasses. The argument
-must not be *NULL*, and the type is not checked:
+must not be ``NULL``, and the type is not checked:
 
 .. c:function:: int PyDateTime_DATE_GET_HOUR(PyDateTime_DateTime *o)
 
@@ -187,7 +187,7 @@ must not be *NULL*, and the type is not checked:
 
 
 Macros to extract fields from time objects.  The argument must be an instance of
-:c:data:`PyDateTime_Time`, including subclasses. The argument must not be *NULL*,
+:c:data:`PyDateTime_Time`, including subclasses. The argument must not be ``NULL``,
 and the type is not checked:
 
 .. c:function:: int PyDateTime_TIME_GET_HOUR(PyDateTime_Time *o)
@@ -212,7 +212,7 @@ and the type is not checked:
 
 Macros to extract fields from time delta objects.  The argument must be an
 instance of :c:data:`PyDateTime_Delta`, including subclasses. The argument must
-not be *NULL*, and the type is not checked:
+not be ``NULL``, and the type is not checked:
 
 .. c:function:: int PyDateTime_DELTA_GET_DAYS(PyDateTime_Delta *o)
 
diff --git a/Doc/c-api/dict.rst b/Doc/c-api/dict.rst
index e970771893eb3..e7922dc0c73f2 100644
--- a/Doc/c-api/dict.rst
+++ b/Doc/c-api/dict.rst
@@ -33,7 +33,7 @@ Dictionary Objects
 
 .. c:function:: PyObject* PyDict_New()
 
-   Return a new empty dictionary, or *NULL* on failure.
+   Return a new empty dictionary, or ``NULL`` on failure.
 
 
 .. c:function:: PyObject* PyDictProxy_New(PyObject *mapping)
@@ -92,7 +92,7 @@ Dictionary Objects
 
 .. c:function:: PyObject* PyDict_GetItem(PyObject *p, PyObject *key)
 
-   Return the object from dictionary *p* which has a key *key*.  Return *NULL*
+   Return the object from dictionary *p* which has a key *key*.  Return ``NULL``
    if the key *key* is not present, but *without* setting an exception.
 
    Note that exceptions which occur while calling :meth:`__hash__` and
@@ -103,8 +103,8 @@ Dictionary Objects
 .. c:function:: PyObject* PyDict_GetItemWithError(PyObject *p, PyObject *key)
 
    Variant of :c:func:`PyDict_GetItem` that does not suppress
-   exceptions. Return *NULL* **with** an exception set if an exception
-   occurred.  Return *NULL* **without** an exception set if the key
+   exceptions. Return ``NULL`` **with** an exception set if an exception
+   occurred.  Return ``NULL`` **without** an exception set if the key
    wasn't present.
 
 
@@ -161,7 +161,7 @@ Dictionary Objects
    function returns true for each pair in the dictionary, and false once all
    pairs have been reported.  The parameters *pkey* and *pvalue* should either
    point to :c:type:`PyObject\*` variables that will be filled in with each key
-   and value, respectively, or may be *NULL*.  Any references returned through
+   and value, respectively, or may be ``NULL``.  Any references returned through
    them are borrowed.  *ppos* should not be altered during iteration. Its
    value represents offsets within the internal dictionary structure, and
    since the structure is sparse, the offsets are not consecutive.
diff --git a/Doc/c-api/exceptions.rst b/Doc/c-api/exceptions.rst
index 25bb657dca0e1..4cb309521d4d2 100644
--- a/Doc/c-api/exceptions.rst
+++ b/Doc/c-api/exceptions.rst
@@ -13,7 +13,7 @@ exception handling.  It works somewhat like the POSIX :c:data:`errno` variable:
 there is a global indicator (per thread) of the last error that occurred.  Most
 C API functions don't clear this on success, but will set it to indicate the
 cause of the error on failure.  Most C API functions also return an error
-indicator, usually *NULL* if they are supposed to return a pointer, or ``-1``
+indicator, usually ``NULL`` if they are supposed to return a pointer, or ``-1``
 if they return an integer (exception: the :c:func:`PyArg_\*` functions
 return ``1`` for success and ``0`` for failure).
 
@@ -111,7 +111,7 @@ NULL pointer for use in a ``return`` statement.
 
 .. c:function:: PyObject* PyErr_Format(PyObject *exception, const char *format, ...)
 
-   This function sets the error indicator and returns *NULL*.  *exception*
+   This function sets the error indicator and returns ``NULL``.  *exception*
    should be a Python exception class.  The *format* and subsequent
    parameters help format the error message; they have the same meaning and
    values as in :c:func:`PyUnicode_FromFormat`. *format* is an ASCII-encoded
@@ -140,7 +140,7 @@ NULL pointer for use in a ``return`` statement.
 
 .. c:function:: PyObject* PyErr_NoMemory()
 
-   This is a shorthand for ``PyErr_SetNone(PyExc_MemoryError)``; it returns *NULL*
+   This is a shorthand for ``PyErr_SetNone(PyExc_MemoryError)``; it returns ``NULL``
    so an object allocation function can write ``return PyErr_NoMemory();`` when it
    runs out of memory.
 
@@ -156,7 +156,7 @@ NULL pointer for use in a ``return`` statement.
    and then calls ``PyErr_SetObject(type, object)``.  On Unix, when the
    :c:data:`errno` value is :const:`EINTR`, indicating an interrupted system call,
    this calls :c:func:`PyErr_CheckSignals`, and if that set the error indicator,
-   leaves it set to that.  The function always returns *NULL*, so a wrapper
+   leaves it set to that.  The function always returns ``NULL``, so a wrapper
    function around a system call can write ``return PyErr_SetFromErrno(type);``
    when the system call returns an error.
 
@@ -164,7 +164,7 @@ NULL pointer for use in a ``return`` statement.
 .. c:function:: PyObject* PyErr_SetFromErrnoWithFilenameObject(PyObject *type, PyObject *filenameObject)
 
    Similar to :c:func:`PyErr_SetFromErrno`, with the additional behavior that if
-   *filenameObject* is not *NULL*, it is passed to the constructor of *type* as
+   *filenameObject* is not ``NULL``, it is passed to the constructor of *type* as
    a third parameter.  In the case of :exc:`OSError` exception,
    this is used to define the :attr:`filename` attribute of the
    exception instance.
@@ -195,7 +195,7 @@ NULL pointer for use in a ``return`` statement.
    then it constructs a tuple object whose first item is the *ierr* value and whose
    second item is the corresponding error message (gotten from
    :c:func:`FormatMessage`), and then calls ``PyErr_SetObject(PyExc_WindowsError,
-   object)``. This function always returns *NULL*.
+   object)``. This function always returns ``NULL``.
 
    .. availability:: Windows.
 
@@ -304,7 +304,7 @@ an error value).
 .. c:function:: int PyErr_WarnEx(PyObject *category, const char *message, Py_ssize_t stack_level)
 
    Issue a warning message.  The *category* argument is a warning category (see
-   below) or *NULL*; the *message* argument is a UTF-8 encoded string.  *stack_level* is a
+   below) or ``NULL``; the *message* argument is a UTF-8 encoded string.  *stack_level* is a
    positive number giving a number of stack frames; the warning will be issued from
    the  currently executing line of code in that stack frame.  A *stack_level* of 1
    is the function calling :c:func:`PyErr_WarnEx`, 2 is  the function above that,
@@ -333,7 +333,7 @@ an error value).
    Issue a warning message with explicit control over all warning attributes.  This
    is a straightforward wrapper around the Python function
    :func:`warnings.warn_explicit`, see there for more information.  The *module*
-   and *registry* arguments may be set to *NULL* to get the default effect
+   and *registry* arguments may be set to ``NULL`` to get the default effect
    described there.
 
    .. versionadded:: 3.4
@@ -370,7 +370,7 @@ Querying the error indicator
 
    Test whether the error indicator is set.  If set, return the exception *type*
    (the first argument to the last call to one of the :c:func:`PyErr_Set\*`
-   functions or to :c:func:`PyErr_Restore`).  If not set, return *NULL*.  You do not
+   functions or to :c:func:`PyErr_Restore`).  If not set, return ``NULL``.  You do not
    own a reference to the return value, so you do not need to :c:func:`Py_DECREF`
    it.
 
@@ -400,9 +400,9 @@ Querying the error indicator
 .. c:function:: void PyErr_Fetch(PyObject **ptype, PyObject **pvalue, PyObject **ptraceback)
 
    Retrieve the error indicator into three variables whose addresses are passed.
-   If the error indicator is not set, set all three variables to *NULL*.  If it is
+   If the error indicator is not set, set all three variables to ``NULL``.  If it is
    set, it will be cleared and you own a reference to each object retrieved.  The
-   value and traceback object may be *NULL* even when the type object is not.
+   value and traceback object may be ``NULL`` even when the type object is not.
 
    .. note::
 
@@ -422,8 +422,8 @@ Querying the error indicator
 .. c:function:: void PyErr_Restore(PyObject *type, PyObject *value, PyObject *traceback)
 
    Set  the error indicator from the three objects.  If the error indicator is
-   already set, it is cleared first.  If the objects are *NULL*, the error
-   indicator is cleared.  Do not pass a *NULL* type and non-*NULL* value or
+   already set, it is cleared first.  If the objects are ``NULL``, the error
+   indicator is cleared.  Do not pass a ``NULL`` type and non-``NULL`` value or
    traceback.  The exception type should be a class.  Do not pass an invalid
    exception type or value. (Violating these rules will cause subtle problems
    later.)  This call takes away a reference to each object: you must own a
@@ -462,7 +462,7 @@ Querying the error indicator
    Retrieve the exception info, as known from ``sys.exc_info()``.  This refers
    to an exception that was *already caught*, not to an exception that was
    freshly raised.  Returns new references for the three objects, any of which
-   may be *NULL*.  Does not modify the exception info state.
+   may be ``NULL``.  Does not modify the exception info state.
 
    .. note::
 
@@ -479,7 +479,7 @@ Querying the error indicator
    Set the exception info, as known from ``sys.exc_info()``.  This refers
    to an exception that was *already caught*, not to an exception that was
    freshly raised.  This function steals the references of the arguments.
-   To clear the exception state, pass *NULL* for all three arguments.
+   To clear the exception state, pass ``NULL`` for all three arguments.
    For general rules about the three arguments, see :c:func:`PyErr_Restore`.
 
    .. note::
@@ -549,7 +549,7 @@ Exception Classes
 
    This utility function creates and returns a new exception class. The *name*
    argument must be the name of the new exception, a C string of the form
-   ``module.classname``.  The *base* and *dict* arguments are normally *NULL*.
+   ``module.classname``.  The *base* and *dict* arguments are normally ``NULL``.
    This creates a class object derived from :exc:`Exception` (accessible in C as
    :c:data:`PyExc_Exception`).
 
@@ -563,7 +563,7 @@ Exception Classes
 .. c:function:: PyObject* PyErr_NewExceptionWithDoc(const char *name, const char *doc, PyObject *base, PyObject *dict)
 
    Same as :c:func:`PyErr_NewException`, except that the new exception class can
-   easily be given a docstring: If *doc* is non-*NULL*, it will be used as the
+   easily be given a docstring: If *doc* is non-``NULL``, it will be used as the
    docstring for the exception class.
 
    .. versionadded:: 3.2
@@ -576,7 +576,7 @@ Exception Objects
 
    Return the traceback associated with the exception as a new reference, as
    accessible from Python through :attr:`__traceback__`.  If there is no
-   traceback associated, this returns *NULL*.
+   traceback associated, this returns ``NULL``.
 
 
 .. c:function:: int PyException_SetTraceback(PyObject *ex, PyObject *tb)
@@ -590,12 +590,12 @@ Exception Objects
    Return the context (another exception instance during whose handling *ex* was
    raised) associated with the exception as a new reference, as accessible from
    Python through :attr:`__context__`.  If there is no context associated, this
-   returns *NULL*.
+   returns ``NULL``.
 
 
 .. c:function:: void PyException_SetContext(PyObject *ex, PyObject *ctx)
 
-   Set the context associated with the exception to *ctx*.  Use *NULL* to clear
+   Set the context associated with the exception to *ctx*.  Use ``NULL`` to clear
    it.  There is no type check to make sure that *ctx* is an exception instance.
    This steals a reference to *ctx*.
 
@@ -609,7 +609,7 @@ Exception Objects
 
 .. c:function:: void PyException_SetCause(PyObject *ex, PyObject *cause)
 
-   Set the cause associated with the exception to *cause*.  Use *NULL* to clear
+   Set the cause associated with the exception to *cause*.  Use ``NULL`` to clear
    it.  There is no type check to make sure that *cause* is either an exception
    instance or :const:`None`.  This steals a reference to *cause*.
 
@@ -656,7 +656,7 @@ The following functions are used to create and modify Unicode exceptions from C.
                 int PyUnicodeTranslateError_GetStart(PyObject *exc, Py_ssize_t *start)
 
    Get the *start* attribute of the given exception object and place it into
-   *\*start*.  *start* must not be *NULL*.  Return ``0`` on success, ``-1`` on
+   *\*start*.  *start* must not be ``NULL``.  Return ``0`` on success, ``-1`` on
    failure.
 
 .. c:function:: int PyUnicodeDecodeError_SetStart(PyObject *exc, Py_ssize_t start)
@@ -671,7 +671,7 @@ The following functions are used to create and modify Unicode exceptions from C.
                 int PyUnicodeTranslateError_GetEnd(PyObject *exc, Py_ssize_t *end)
 
    Get the *end* attribute of the given exception object and place it into
-   *\*end*.  *end* must not be *NULL*.  Return ``0`` on success, ``-1`` on
+   *\*end*.  *end* must not be ``NULL``.  Return ``0`` on success, ``-1`` on
    failure.
 
 .. c:function:: int PyUnicodeDecodeError_SetEnd(PyObject *exc, Py_ssize_t end)
diff --git a/Doc/c-api/file.rst b/Doc/c-api/file.rst
index 543dc60b9f55a..5370c4e350a0b 100644
--- a/Doc/c-api/file.rst
+++ b/Doc/c-api/file.rst
@@ -21,9 +21,9 @@ the :mod:`io` APIs instead.
 
    Create a Python file object from the file descriptor of an already
    opened file *fd*.  The arguments *name*, *encoding*, *errors* and *newline*
-   can be *NULL* to use the defaults; *buffering* can be *-1* to use the
+   can be ``NULL`` to use the defaults; *buffering* can be *-1* to use the
    default. *name* is ignored and kept for backward compatibility. Return
-   *NULL* on failure. For a more comprehensive description of the arguments,
+   ``NULL`` on failure. For a more comprehensive description of the arguments,
    please refer to the :func:`io.open` function documentation.
 
    .. warning::
diff --git a/Doc/c-api/float.rst b/Doc/c-api/float.rst
index 057ff522516a3..bfc28a79ecfdc 100644
--- a/Doc/c-api/float.rst
+++ b/Doc/c-api/float.rst
@@ -34,12 +34,12 @@ Floating Point Objects
 .. c:function:: PyObject* PyFloat_FromString(PyObject *str)
 
    Create a :c:type:`PyFloatObject` object based on the string value in *str*, or
-   *NULL* on failure.
+   ``NULL`` on failure.
 
 
 .. c:function:: PyObject* PyFloat_FromDouble(double v)
 
-   Create a :c:type:`PyFloatObject` object from *v*, or *NULL* on failure.
+   Create a :c:type:`PyFloatObject` object from *v*, or ``NULL`` on failure.
 
 
 .. c:function:: double PyFloat_AsDouble(PyObject *pyfloat)
diff --git a/Doc/c-api/function.rst b/Doc/c-api/function.rst
index 02c4ebdbed454..a3cbf569f76f2 100644
--- a/Doc/c-api/function.rst
+++ b/Doc/c-api/function.rst
@@ -26,7 +26,7 @@ There are a few functions specific to Python functions.
 .. c:function:: int PyFunction_Check(PyObject *o)
 
    Return true if *o* is a function object (has type :c:data:`PyFunction_Type`).
-   The parameter must not be *NULL*.
+   The parameter must not be ``NULL``.
 
 
 .. c:function:: PyObject* PyFunction_New(PyObject *code, PyObject *globals)
@@ -36,7 +36,7 @@ There are a few functions specific to Python functions.
 
    The function's docstring and name are retrieved from the code object. *__module__*
    is retrieved from *globals*. The argument defaults, annotations and closure are
-   set to *NULL*. *__qualname__* is set to the same value as the function's name.
+   set to ``NULL``. *__qualname__* is set to the same value as the function's name.
 
 
 .. c:function:: PyObject* PyFunction_NewWithQualName(PyObject *code, PyObject *globals, PyObject *qualname)
@@ -69,7 +69,7 @@ There are a few functions specific to Python functions.
 .. c:function:: PyObject* PyFunction_GetDefaults(PyObject *op)
 
    Return the argument default values of the function object *op*. This can be a
-   tuple of arguments or *NULL*.
+   tuple of arguments or ``NULL``.
 
 
 .. c:function:: int PyFunction_SetDefaults(PyObject *op, PyObject *defaults)
@@ -82,7 +82,7 @@ There are a few functions specific to Python functions.
 
 .. c:function:: PyObject* PyFunction_GetClosure(PyObject *op)
 
-   Return the closure associated with the function object *op*. This can be *NULL*
+   Return the closure associated with the function object *op*. This can be ``NULL``
    or a tuple of cell objects.
 
 
@@ -97,7 +97,7 @@ There are a few functions specific to Python functions.
 .. c:function:: PyObject *PyFunction_GetAnnotations(PyObject *op)
 
    Return the annotations of the function object *op*. This can be a
-   mutable dictionary or *NULL*.
+   mutable dictionary or ``NULL``.
 
 
 .. c:function:: int PyFunction_SetAnnotations(PyObject *op, PyObject *annotations)
diff --git a/Doc/c-api/gcsupport.rst b/Doc/c-api/gcsupport.rst
index b3f30b2ed9e99..924a7fd2fda4d 100644
--- a/Doc/c-api/gcsupport.rst
+++ b/Doc/c-api/gcsupport.rst
@@ -49,7 +49,7 @@ Constructors for container types must conform to two rules:
 .. c:function:: TYPE* PyObject_GC_Resize(TYPE, PyVarObject *op, Py_ssize_t newsize)
 
    Resize an object allocated by :c:func:`PyObject_NewVar`.  Returns the
-   resized object or *NULL* on failure.  *op* must not be tracked by the collector yet.
+   resized object or ``NULL`` on failure.  *op* must not be tracked by the collector yet.
 
 
 .. c:function:: void PyObject_GC_Track(PyObject *op)
@@ -110,7 +110,7 @@ The :c:member:`~PyTypeObject.tp_traverse` handler must have the following type:
    Traversal function for a container object.  Implementations must call the
    *visit* function for each object directly contained by *self*, with the
    parameters to *visit* being the contained object and the *arg* value passed
-   to the handler.  The *visit* function must not be called with a *NULL*
+   to the handler.  The *visit* function must not be called with a ``NULL``
    object argument.  If *visit* returns a non-zero value that value should be
    returned immediately.
 
@@ -121,7 +121,7 @@ must name its arguments exactly *visit* and *arg*:
 
 .. c:function:: void Py_VISIT(PyObject *o)
 
-   If *o* is not *NULL*, call the *visit* callback, with arguments *o*
+   If *o* is not ``NULL``, call the *visit* callback, with arguments *o*
    and *arg*.  If *visit* returns a non-zero value, then return it.
    Using this macro, :c:member:`~PyTypeObject.tp_traverse` handlers
    look like::
@@ -134,7 +134,7 @@ must name its arguments exactly *visit* and *arg*:
           return 0;
       }
 
-The :c:member:`~PyTypeObject.tp_clear` handler must be of the :c:type:`inquiry` type, or *NULL*
+The :c:member:`~PyTypeObject.tp_clear` handler must be of the :c:type:`inquiry` type, or ``NULL``
 if the object is immutable.
 
 
diff --git a/Doc/c-api/gen.rst b/Doc/c-api/gen.rst
index 8d54021c18145..e4b8d255dadba 100644
--- a/Doc/c-api/gen.rst
+++ b/Doc/c-api/gen.rst
@@ -22,23 +22,23 @@ than explicitly calling :c:func:`PyGen_New` or :c:func:`PyGen_NewWithQualName`.
 
 .. c:function:: int PyGen_Check(PyObject *ob)
 
-   Return true if *ob* is a generator object; *ob* must not be *NULL*.
+   Return true if *ob* is a generator object; *ob* must not be ``NULL``.
 
 
 .. c:function:: int PyGen_CheckExact(PyObject *ob)
 
-   Return true if *ob*'s type is *PyGen_Type*; *ob* must not be *NULL*.
+   Return true if *ob*'s type is *PyGen_Type*; *ob* must not be ``NULL``.
 
 
 .. c:function:: PyObject* PyGen_New(PyFrameObject *frame)
 
    Create and return a new generator object based on the *frame* object.
    A reference to *frame* is stolen by this function. The argument must not be
-   *NULL*.
+   ``NULL``.
 
 .. c:function:: PyObject* PyGen_NewWithQualName(PyFrameObject *frame, PyObject *name, PyObject *qualname)
 
    Create and return a new generator object based on the *frame* object,
    with ``__name__`` and ``__qualname__`` set to *name* and *qualname*.
    A reference to *frame* is stolen by this function.  The *frame* argument
-   must not be *NULL*.
+   must not be ``NULL``.
diff --git a/Doc/c-api/import.rst b/Doc/c-api/import.rst
index 3bc50609714a5..ee098768b124e 100644
--- a/Doc/c-api/import.rst
+++ b/Doc/c-api/import.rst
@@ -14,7 +14,7 @@ Importing Modules
       single: modules (in module sys)
 
    This is a simplified interface to :c:func:`PyImport_ImportModuleEx` below,
-   leaving the *globals* and *locals* arguments set to *NULL* and *level* set
+   leaving the *globals* and *locals* arguments set to ``NULL`` and *level* set
    to 0.  When the *name*
    argument contains a dot (when it specifies a submodule of a package), the
    *fromlist* argument is set to the list ``['*']`` so that the return value is the
@@ -22,7 +22,7 @@ Importing Modules
    be the case.  (Unfortunately, this has an additional side effect when *name* in
    fact specifies a subpackage instead of a submodule: the submodules specified in
    the package's ``__all__`` variable are  loaded.)  Return a new reference to the
-   imported module, or *NULL* with an exception set on failure.  A failing
+   imported module, or ``NULL`` with an exception set on failure.  A failing
    import of a module doesn't leave the module in :data:`sys.modules`.
 
    This function always uses absolute imports.
@@ -47,7 +47,7 @@ Importing Modules
    function :func:`__import__`.
 
    The return value is a new reference to the imported module or top-level
-   package, or *NULL* with an exception set on failure.  Like for
+   package, or ``NULL`` with an exception set on failure.  Like for
    :func:`__import__`, the return value when a submodule of a package was
    requested is normally the top-level package, unless a non-empty *fromlist*
    was given.
@@ -63,7 +63,7 @@ Importing Modules
    this function directly.
 
    The return value is a new reference to the imported module or top-level package,
-   or *NULL* with an exception set on failure.  Like for :func:`__import__`,
+   or ``NULL`` with an exception set on failure.  Like for :func:`__import__`,
    the return value when a submodule of a package was requested is normally the
    top-level package, unless a non-empty *fromlist* was given.
 
@@ -91,7 +91,7 @@ Importing Modules
 
 .. c:function:: PyObject* PyImport_ReloadModule(PyObject *m)
 
-   Reload a module.  Return a new reference to the reloaded module, or *NULL* with
+   Reload a module.  Return a new reference to the reloaded module, or ``NULL`` with
    an exception set on failure (the module still exists in this case).
 
 
@@ -100,7 +100,7 @@ Importing Modules
    Return the module object corresponding to a module name.  The *name* argument
    may be of the form ``package.module``. First check the modules dictionary if
    there's one there, and if not, create a new one and insert it in the modules
-   dictionary. Return *NULL* with an exception set on failure.
+   dictionary. Return ``NULL`` with an exception set on failure.
 
    .. note::
 
@@ -125,7 +125,7 @@ Importing Modules
    Given a module name (possibly of the form ``package.module``) and a code object
    read from a Python bytecode file or obtained from the built-in function
    :func:`compile`, load the module.  Return a new reference to the module object,
-   or *NULL* with an exception set if an error occurred.  *name*
+   or ``NULL`` with an exception set if an error occurred.  *name*
    is removed from :attr:`sys.modules` in error cases, even if *name* was already
    in :attr:`sys.modules` on entry to :c:func:`PyImport_ExecCodeModule`.  Leaving
    incompletely initialized modules in :attr:`sys.modules` is dangerous, as imports of
@@ -262,7 +262,7 @@ Importing Modules
 .. c:var:: const struct _frozen* PyImport_FrozenModules
 
    This pointer is initialized to point to an array of :c:type:`struct _frozen`
-   records, terminated by one whose members are all *NULL* or zero.  When a frozen
+   records, terminated by one whose members are all ``NULL`` or zero.  When a frozen
    module is imported, it is searched in this table.  Third-party code could play
    tricks with this to provide a dynamically created collection of frozen modules.
 
@@ -295,7 +295,7 @@ Importing Modules
 .. c:function:: int PyImport_ExtendInittab(struct _inittab *newtab)
 
    Add a collection of modules to the table of built-in modules.  The *newtab*
-   array must end with a sentinel entry which contains *NULL* for the :attr:`name`
+   array must end with a sentinel entry which contains ``NULL`` for the :attr:`name`
    field; failure to provide the sentinel value can result in a memory fault.
    Returns ``0`` on success or ``-1`` if insufficient memory could be allocated to
    extend the internal table.  In the event of failure, no modules are added to the
diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst
index 5456ad446cf23..61912b968fd4b 100644
--- a/Doc/c-api/init.rst
+++ b/Doc/c-api/init.rst
@@ -851,15 +851,15 @@ code, or when embedding the Python interpreter:
 .. c:function:: PyThreadState* PyEval_SaveThread()
 
    Release the global interpreter lock (if it has been created) and reset the
-   thread state to *NULL*, returning the previous thread state (which is not
-   *NULL*).  If the lock has been created, the current thread must have
+   thread state to ``NULL``, returning the previous thread state (which is not
+   ``NULL``).  If the lock has been created, the current thread must have
    acquired it.
 
 
 .. c:function:: void PyEval_RestoreThread(PyThreadState *tstate)
 
    Acquire the global interpreter lock (if it has been created) and set the
-   thread state to *tstate*, which must not be *NULL*.  If the lock has been
+   thread state to *tstate*, which must not be ``NULL``.  If the lock has been
    created, the current thread must not have acquired it, otherwise deadlock
    ensues.
 
@@ -873,14 +873,14 @@ code, or when embedding the Python interpreter:
 .. c:function:: PyThreadState* PyThreadState_Get()
 
    Return the current thread state.  The global interpreter lock must be held.
-   When the current thread state is *NULL*, this issues a fatal error (so that
-   the caller needn't check for *NULL*).
+   When the current thread state is ``NULL``, this issues a fatal error (so that
+   the caller needn't check for ``NULL``).
 
 
 .. c:function:: PyThreadState* PyThreadState_Swap(PyThreadState *tstate)
 
    Swap the current thread state with the thread state given by the argument
-   *tstate*, which may be *NULL*.  The global interpreter lock must be held
+   *tstate*, which may be ``NULL``.  The global interpreter lock must be held
    and is not released.
 
 
@@ -1054,7 +1054,7 @@ All of the following functions must be called after :c:func:`Py_Initialize`.
 .. c:function:: PyObject* PyInterpreterState_GetDict(PyInterpreterState *interp)
 
    Return a dictionary in which interpreter-specific data may be stored.
-   If this function returns *NULL* then no exception has been raised and
+   If this function returns ``NULL`` then no exception has been raised and
    the caller should assume no interpreter-specific dict is available.
 
    This is not a replacement for :c:func:`PyModule_GetState()`, which
@@ -1068,7 +1068,7 @@ All of the following functions must be called after :c:func:`Py_Initialize`.
    Return a dictionary in which extensions can store thread-specific state
    information.  Each extension should use a unique key to use to store state in
    the dictionary.  It is okay to call this function when no current thread state
-   is available. If this function returns *NULL*, no exception has been raised and
+   is available. If this function returns ``NULL``, no exception has been raised and
    the caller should assume no current thread state is available.
 
 
@@ -1089,7 +1089,7 @@ All of the following functions must be called after :c:func:`Py_Initialize`.
 .. c:function:: void PyEval_AcquireThread(PyThreadState *tstate)
 
    Acquire the global interpreter lock and set the current thread state to
-   *tstate*, which should not be *NULL*.  The lock must have been created earlier.
+   *tstate*, which should not be ``NULL``.  The lock must have been created earlier.
    If this thread already has the lock, deadlock ensues.
 
    .. note::
@@ -1110,9 +1110,9 @@ All of the following functions must be called after :c:func:`Py_Initialize`.
 
 .. c:function:: void PyEval_ReleaseThread(PyThreadState *tstate)
 
-   Reset the current thread state to *NULL* and release the global interpreter
+   Reset the current thread state to ``NULL`` and release the global interpreter
    lock.  The lock must have been created earlier and must be held by the current
-   thread.  The *tstate* argument, which must not be *NULL*, is only used to check
+   thread.  The *tstate* argument, which must not be ``NULL``, is only used to check
    that it represents the current thread state --- if it isn't, a fatal error is
    reported.
 
@@ -1197,7 +1197,7 @@ function. You can create and destroy them using the following functions:
    The return value points to the first thread state created in the new
    sub-interpreter.  This thread state is made in the current thread state.
    Note that no actual thread is created; see the discussion of thread states
-   below.  If creation of the new interpreter is unsuccessful, *NULL* is
+   below.  If creation of the new interpreter is unsuccessful, ``NULL`` is
    returned; no exception is set since the exception state is stored in the
    current thread state and there may not be a current thread state.  (Like all
    other Python/C API functions, the global interpreter lock must be held before
@@ -1228,7 +1228,7 @@ function. You can create and destroy them using the following functions:
 
    Destroy the (sub-)interpreter represented by the given thread state. The given
    thread state must be the current thread state.  See the discussion of thread
-   states below.  When the call returns, the current thread state is *NULL*.  All
+   states below.  When the call returns, the current thread state is ``NULL``.  All
    thread states associated with this interpreter are destroyed.  (The global
    interpreter lock must be held before calling this function and is still held
    when it returns.)  :c:func:`Py_FinalizeEx` will destroy all sub-interpreters that
@@ -1335,27 +1335,27 @@ Python-level trace functions in previous versions.
    :const:`PyTrace_C_CALL`, :const:`PyTrace_C_EXCEPTION`, :const:`PyTrace_C_RETURN`,
    or :const:`PyTrace_OPCODE`, and *arg* depends on the value of *what*:
 
-   +------------------------------+--------------------------------------+
-   | Value of *what*              | Meaning of *arg*                     |
-   +==============================+======================================+
-   | :const:`PyTrace_CALL`        | Always :c:data:`Py_None`.            |
-   +------------------------------+--------------------------------------+
-   | :const:`PyTrace_EXCEPTION`   | Exception information as returned by |
-   |                              | :func:`sys.exc_info`.                |
-   +------------------------------+--------------------------------------+
-   | :const:`PyTrace_LINE`        | Always :c:data:`Py_None`.            |
-   +------------------------------+--------------------------------------+
-   | :const:`PyTrace_RETURN`      | Value being returned to the caller,  |
-   |                              | or *NULL* if caused by an exception. |
-   +------------------------------+--------------------------------------+
-   | :const:`PyTrace_C_CALL`      | Function object being called.        |
-   +------------------------------+--------------------------------------+
-   | :const:`PyTrace_C_EXCEPTION` | Function object being called.        |
-   +------------------------------+--------------------------------------+
-   | :const:`PyTrace_C_RETURN`    | Function object being called.        |
-   +------------------------------+--------------------------------------+
-   | :const:`PyTrace_OPCODE`      | Always :c:data:`Py_None`.            |
-   +------------------------------+--------------------------------------+
+   +------------------------------+----------------------------------------+
+   | Value of *what*              | Meaning of *arg*                       |
+   +==============================+========================================+
+   | :const:`PyTrace_CALL`        | Always :c:data:`Py_None`.              |
+   +------------------------------+----------------------------------------+
+   | :const:`PyTrace_EXCEPTION`   | Exception information as returned by   |
+   |                              | :func:`sys.exc_info`.                  |
+   +------------------------------+----------------------------------------+
+   | :const:`PyTrace_LINE`        | Always :c:data:`Py_None`.              |
+   +------------------------------+----------------------------------------+
+   | :const:`PyTrace_RETURN`      | Value being returned to the caller,    |
+   |                              | or ``NULL`` if caused by an exception. |
+   +------------------------------+----------------------------------------+
+   | :const:`PyTrace_C_CALL`      | Function object being called.          |
+   +------------------------------+----------------------------------------+
+   | :const:`PyTrace_C_EXCEPTION` | Function object being called.          |
+   +------------------------------+----------------------------------------+
+   | :const:`PyTrace_C_RETURN`    | Function object being called.          |
+   +------------------------------+----------------------------------------+
+   | :const:`PyTrace_OPCODE`      | Always :c:data:`Py_None`.              |
+   +------------------------------+----------------------------------------+
 
 .. c:var:: int PyTrace_CALL
 
@@ -1419,7 +1419,7 @@ Python-level trace functions in previous versions.
 .. c:function:: void PyEval_SetProfile(Py_tracefunc func, PyObject *obj)
 
    Set the profiler function to *func*.  The *obj* parameter is passed to the
-   function as its first parameter, and may be any Python object, or *NULL*.  If
+   function as its first parameter, and may be any Python object, or ``NULL``.  If
    the profile function needs to maintain state, using a different value for *obj*
    for each thread provides a convenient and thread-safe place to store it.  The
    profile function is called for all monitored events except :const:`PyTrace_LINE`
@@ -1542,7 +1542,7 @@ is not possible due to its implementation being opaque at build time.
 .. c:function:: Py_tss_t* PyThread_tss_alloc()
 
    Return a value which is the same state as a value initialized with
-   :c:macro:`Py_tss_NEEDS_INIT`, or *NULL* in the case of dynamic allocation
+   :c:macro:`Py_tss_NEEDS_INIT`, or ``NULL`` in the case of dynamic allocation
    failure.
 
 
@@ -1561,7 +1561,7 @@ is not possible due to its implementation being opaque at build time.
 Methods
 ~~~~~~~
 
-The parameter *key* of these functions must not be *NULL*.  Moreover, the
+The parameter *key* of these functions must not be ``NULL``.  Moreover, the
 behaviors of :c:func:`PyThread_tss_set` and :c:func:`PyThread_tss_get` are
 undefined if the given :c:type:`Py_tss_t` has not been initialized by
 :c:func:`PyThread_tss_create`.
@@ -1601,7 +1601,7 @@ undefined if the given :c:type:`Py_tss_t` has not been initialized by
 .. c:function:: void* PyThread_tss_get(Py_tss_t *key)
 
    Return the :c:type:`void\*` value associated with a TSS key in the current
-   thread.  This returns *NULL* if no value is associated with the key in the
+   thread.  This returns ``NULL`` if no value is associated with the key in the
    current thread.
 
 
diff --git a/Doc/c-api/intro.rst b/Doc/c-api/intro.rst
index 80eebd89ad3f2..d08d4f97a308e 100644
--- a/Doc/c-api/intro.rst
+++ b/Doc/c-api/intro.rst
@@ -152,7 +152,7 @@ complete listing.
 
 .. c:macro:: Py_GETENV(s)
 
-   Like ``getenv(s)``, but returns *NULL* if :option:`-E` was passed on the
+   Like ``getenv(s)``, but returns ``NULL`` if :option:`-E` was passed on the
    command line (i.e. if ``Py_IgnoreEnvironmentFlag`` is set).
 
 .. c:macro:: Py_UNUSED(arg)
@@ -475,7 +475,7 @@ functions in the Python/C API can raise exceptions, unless an explicit claim is
 made otherwise in a function's documentation.  In general, when a function
 encounters an error, it sets an exception, discards any object references that
 it owns, and returns an error indicator.  If not documented otherwise, this
-indicator is either *NULL* or ``-1``, depending on the function's return type.
+indicator is either ``NULL`` or ``-1``, depending on the function's return type.
 A few functions return a Boolean true/false result, with false indicating an
 error.  Very few functions return no explicit error indicator or have an
 ambiguous return value, and require explicit testing for errors with
@@ -490,13 +490,13 @@ using global storage in an unthreaded application).  A  thread can be in one of
 two states: an exception has occurred, or not. The function
 :c:func:`PyErr_Occurred` can be used to check for this: it returns a borrowed
 reference to the exception type object when an exception has occurred, and
-*NULL* otherwise.  There are a number of functions to set the exception state:
+``NULL`` otherwise.  There are a number of functions to set the exception state:
 :c:func:`PyErr_SetString` is the most common (though not the most general)
 function to set the exception state, and :c:func:`PyErr_Clear` clears the
 exception state.
 
 The full exception state consists of three objects (all of which can  be
-*NULL*): the exception type, the corresponding exception  value, and the
+``NULL``): the exception type, the corresponding exception  value, and the
 traceback.  These have the same meanings as the Python result of
 ``sys.exc_info()``; however, they are not the same: the Python objects represent
 the last exception being handled by a Python  :keyword:`try` ...
@@ -597,10 +597,10 @@ Here is the corresponding C code, in all its glory::
 This example represents an endorsed use of the ``goto`` statement  in C!
 It illustrates the use of :c:func:`PyErr_ExceptionMatches` and
 :c:func:`PyErr_Clear` to handle specific exceptions, and the use of
-:c:func:`Py_XDECREF` to dispose of owned references that may be *NULL* (note the
+:c:func:`Py_XDECREF` to dispose of owned references that may be ``NULL`` (note the
 ``'X'`` in the name; :c:func:`Py_DECREF` would crash when confronted with a
-*NULL* reference).  It is important that the variables used to hold owned
-references are initialized to *NULL* for this to work; likewise, the proposed
+``NULL`` reference).  It is important that the variables used to hold owned
+references are initialized to ``NULL`` for this to work; likewise, the proposed
 return value is initialized to ``-1`` (failure) and only set to success after
 the final call made is successful.
 
diff --git a/Doc/c-api/iter.rst b/Doc/c-api/iter.rst
index 546efb518a7af..0224d37f1a41a 100644
--- a/Doc/c-api/iter.rst
+++ b/Doc/c-api/iter.rst
@@ -16,8 +16,8 @@ There are two functions specifically for working with iterators.
 
    Return the next value from the iteration *o*.  The object must be an iterator
    (it is up to the caller to check this).  If there are no remaining values,
-   returns *NULL* with no exception set.  If an error occurs while retrieving
-   the item, returns *NULL* and passes along the exception.
+   returns ``NULL`` with no exception set.  If an error occurs while retrieving
+   the item, returns ``NULL`` and passes along the exception.
 
 To write a loop which iterates over an iterator, the C code should look
 something like this::
diff --git a/Doc/c-api/list.rst b/Doc/c-api/list.rst
index 646f29d08a04b..b247cdfba0187 100644
--- a/Doc/c-api/list.rst
+++ b/Doc/c-api/list.rst
@@ -33,7 +33,7 @@ List Objects
 
 .. c:function:: PyObject* PyList_New(Py_ssize_t len)
 
-   Return a new list of length *len* on success, or *NULL* on failure.
+   Return a new list of length *len* on success, or ``NULL`` on failure.
 
    .. note::
 
@@ -61,7 +61,7 @@ List Objects
    Return the object at position *index* in the list pointed to by *list*.  The
    position must be non-negative; indexing from the end of the list is not
    supported.  If *index* is out of bounds (<0 or >=len(list)),
-   return *NULL* and set an :exc:`IndexError` exception.
+   return ``NULL`` and set an :exc:`IndexError` exception.
 
 
 .. c:function:: PyObject* PyList_GET_ITEM(PyObject *list, Py_ssize_t i)
@@ -111,7 +111,7 @@ List Objects
 .. c:function:: PyObject* PyList_GetSlice(PyObject *list, Py_ssize_t low, Py_ssize_t high)
 
    Return a list of the objects in *list* containing the objects *between* *low*
-   and *high*.  Return *NULL* and set an exception if unsuccessful.  Analogous
+   and *high*.  Return ``NULL`` and set an exception if unsuccessful.  Analogous
    to ``list[low:high]``.  Indexing from the end of the list is not supported.
 
 
@@ -119,7 +119,7 @@ List Objects
 
    Set the slice of *list* between *low* and *high* to the contents of
    *itemlist*.  Analogous to ``list[low:high] = itemlist``. The *itemlist* may
-   be *NULL*, indicating the assignment of an empty list (slice deletion).
+   be ``NULL``, indicating the assignment of an empty list (slice deletion).
    Return ``0`` on success, ``-1`` on failure.  Indexing from the end of the
    list is not supported.
 
diff --git a/Doc/c-api/long.rst b/Doc/c-api/long.rst
index aeebf3060eb4e..83d59de6a0d58 100644
--- a/Doc/c-api/long.rst
+++ b/Doc/c-api/long.rst
@@ -38,7 +38,7 @@ distinguished from a number.  Use :c:func:`PyErr_Occurred` to disambiguate.
 
 .. c:function:: PyObject* PyLong_FromLong(long v)
 
-   Return a new :c:type:`PyLongObject` object from *v*, or *NULL* on failure.
+   Return a new :c:type:`PyLongObject` object from *v*, or ``NULL`` on failure.
 
    The current implementation keeps an array of integer objects for all integers
    between ``-5`` and ``256``, when you create an int in that range you actually
@@ -48,43 +48,43 @@ distinguished from a number.  Use :c:func:`PyErr_Occurred` to disambiguate.
 .. c:function:: PyObject* PyLong_FromUnsignedLong(unsigned long v)
 
    Return a new :c:type:`PyLongObject` object from a C :c:type:`unsigned long`, or
-   *NULL* on failure.
+   ``NULL`` on failure.
 
 
 .. c:function:: PyObject* PyLong_FromSsize_t(Py_ssize_t v)
 
    Return a new :c:type:`PyLongObject` object from a C :c:type:`Py_ssize_t`, or
-   *NULL* on failure.
+   ``NULL`` on failure.
 
 
 .. c:function:: PyObject* PyLong_FromSize_t(size_t v)
 
    Return a new :c:type:`PyLongObject` object from a C :c:type:`size_t`, or
-   *NULL* on failure.
+   ``NULL`` on failure.
 
 
 .. c:function:: PyObject* PyLong_FromLongLong(long long v)
 
-   Return a new :c:type:`PyLongObject` object from a C :c:type:`long long`, or *NULL*
+   Return a new :c:type:`PyLongObject` object from a C :c:type:`long long`, or ``NULL``
    on failure.
 
 
 .. c:function:: PyObject* PyLong_FromUnsignedLongLong(unsigned long long v)
 
    Return a new :c:type:`PyLongObject` object from a C :c:type:`unsigned long long`,
-   or *NULL* on failure.
+   or ``NULL`` on failure.
 
 
 .. c:function:: PyObject* PyLong_FromDouble(double v)
 
    Return a new :c:type:`PyLongObject` object from the integer part of *v*, or
-   *NULL* on failure.
+   ``NULL`` on failure.
 
 
 .. c:function:: PyObject* PyLong_FromString(const char *str, char **pend, int base)
 
    Return a new :c:type:`PyLongObject` based on the string value in *str*, which
-   is interpreted according to the radix in *base*.  If *pend* is non-*NULL*,
+   is interpreted according to the radix in *base*.  If *pend* is non-``NULL``,
    *\*pend* will point to the first character in *str* which follows the
    representation of the number.  If *base* is ``0``, *str* is interpreted using
    the :ref:`integers` definition; in this case, leading zeros in a
@@ -334,4 +334,4 @@ distinguished from a number.  Use :c:func:`PyErr_Occurred` to disambiguate.
    is only assured to produce a usable :c:type:`void` pointer for values created
    with :c:func:`PyLong_FromVoidPtr`.
 
-   Returns *NULL* on error.  Use :c:func:`PyErr_Occurred` to disambiguate.
+   Returns ``NULL`` on error.  Use :c:func:`PyErr_Occurred` to disambiguate.
diff --git a/Doc/c-api/mapping.rst b/Doc/c-api/mapping.rst
index 4244b47af75f1..4015f43d8f1ee 100644
--- a/Doc/c-api/mapping.rst
+++ b/Doc/c-api/mapping.rst
@@ -29,7 +29,7 @@ See also :c:func:`PyObject_GetItem`, :c:func:`PyObject_SetItem` and
 
 .. c:function:: PyObject* PyMapping_GetItemString(PyObject *o, const char *key)
 
-   Return element of *o* corresponding to the string *key* or *NULL* on failure.
+   Return element of *o* corresponding to the string *key* or ``NULL`` on failure.
    This is the equivalent of the Python expression ``o[key]``.
    See also :c:func:`PyObject_GetItem`.
 
@@ -79,7 +79,7 @@ See also :c:func:`PyObject_GetItem`, :c:func:`PyObject_SetItem` and
 .. c:function:: PyObject* PyMapping_Keys(PyObject *o)
 
    On success, return a list of the keys in object *o*.  On failure, return
-   *NULL*.
+   ``NULL``.
 
    .. versionchanged:: 3.7
       Previously, the function returned a list or a tuple.
@@ -88,7 +88,7 @@ See also :c:func:`PyObject_GetItem`, :c:func:`PyObject_SetItem` and
 .. c:function:: PyObject* PyMapping_Values(PyObject *o)
 
    On success, return a list of the values in object *o*.  On failure, return
-   *NULL*.
+   ``NULL``.
 
    .. versionchanged:: 3.7
       Previously, the function returned a list or a tuple.
@@ -97,7 +97,7 @@ See also :c:func:`PyObject_GetItem`, :c:func:`PyObject_SetItem` and
 .. c:function:: PyObject* PyMapping_Items(PyObject *o)
 
    On success, return a list of the items in object *o*, where each item is a
-   tuple containing a key-value pair.  On failure, return *NULL*.
+   tuple containing a key-value pair.  On failure, return ``NULL``.
 
    .. versionchanged:: 3.7
       Previously, the function returned a list or a tuple.
diff --git a/Doc/c-api/marshal.rst b/Doc/c-api/marshal.rst
index b086830feb14f..2f580430237af 100644
--- a/Doc/c-api/marshal.rst
+++ b/Doc/c-api/marshal.rst
@@ -67,7 +67,7 @@ The following functions allow marshalled values to be read back in.
    reading.
 
    On error, sets the appropriate exception (:exc:`EOFError`, :exc:`ValueError`
-   or :exc:`TypeError`) and returns *NULL*.
+   or :exc:`TypeError`) and returns ``NULL``.
 
 
 .. c:function:: PyObject* PyMarshal_ReadLastObjectFromFile(FILE *file)
@@ -81,7 +81,7 @@ The following functions allow marshalled values to be read back in.
    anything else from the file.
 
    On error, sets the appropriate exception (:exc:`EOFError`, :exc:`ValueError`
-   or :exc:`TypeError`) and returns *NULL*.
+   or :exc:`TypeError`) and returns ``NULL``.
 
 
 .. c:function:: PyObject* PyMarshal_ReadObjectFromString(const char *data, Py_ssize_t len)
@@ -90,5 +90,5 @@ The following functions allow marshalled values to be read back in.
    containing *len* bytes pointed to by *data*.
 
    On error, sets the appropriate exception (:exc:`EOFError`, :exc:`ValueError`
-   or :exc:`TypeError`) and returns *NULL*.
+   or :exc:`TypeError`) and returns ``NULL``.
 
diff --git a/Doc/c-api/memory.rst b/Doc/c-api/memory.rst
index d3c8b30d360ec..c5b3af64c16a4 100644
--- a/Doc/c-api/memory.rst
+++ b/Doc/c-api/memory.rst
@@ -110,9 +110,9 @@ zero bytes.
 .. c:function:: void* PyMem_RawMalloc(size_t n)
 
    Allocates *n* bytes and returns a pointer of type :c:type:`void\*` to the
-   allocated memory, or *NULL* if the request fails.
+   allocated memory, or ``NULL`` if the request fails.
 
-   Requesting zero bytes returns a distinct non-*NULL* pointer if possible, as
+   Requesting zero bytes returns a distinct non-``NULL`` pointer if possible, as
    if ``PyMem_RawMalloc(1)`` had been called instead. The memory will not have
    been initialized in any way.
 
@@ -120,11 +120,11 @@ zero bytes.
 .. c:function:: void* PyMem_RawCalloc(size_t nelem, size_t elsize)
 
    Allocates *nelem* elements each whose size in bytes is *elsize* and returns
-   a pointer of type :c:type:`void\*` to the allocated memory, or *NULL* if the
+   a pointer of type :c:type:`void\*` to the allocated memory, or ``NULL`` if the
    request fails. The memory is initialized to zeros.
 
    Requesting zero elements or elements of size zero bytes returns a distinct
-   non-*NULL* pointer if possible, as if ``PyMem_RawCalloc(1, 1)`` had been
+   non-``NULL`` pointer if possible, as if ``PyMem_RawCalloc(1, 1)`` had been
    called instead.
 
    .. versionadded:: 3.5
@@ -135,15 +135,15 @@ zero bytes.
    Resizes the memory block pointed to by *p* to *n* bytes. The contents will
    be unchanged to the minimum of the old and the new sizes.
 
-   If *p* is *NULL*, the call is equivalent to ``PyMem_RawMalloc(n)``; else if
+   If *p* is ``NULL``, the call is equivalent to ``PyMem_RawMalloc(n)``; else if
    *n* is equal to zero, the memory block is resized but is not freed, and the
-   returned pointer is non-*NULL*.
+   returned pointer is non-``NULL``.
 
-   Unless *p* is *NULL*, it must have been returned by a previous call to
+   Unless *p* is ``NULL``, it must have been returned by a previous call to
    :c:func:`PyMem_RawMalloc`, :c:func:`PyMem_RawRealloc` or
    :c:func:`PyMem_RawCalloc`.
 
-   If the request fails, :c:func:`PyMem_RawRealloc` returns *NULL* and *p*
+   If the request fails, :c:func:`PyMem_RawRealloc` returns ``NULL`` and *p*
    remains a valid pointer to the previous memory area.
 
 
@@ -154,7 +154,7 @@ zero bytes.
    :c:func:`PyMem_RawCalloc`.  Otherwise, or if ``PyMem_RawFree(p)`` has been
    called before, undefined behavior occurs.
 
-   If *p* is *NULL*, no operation is performed.
+   If *p* is ``NULL``, no operation is performed.
 
 
 .. _memoryinterface:
@@ -181,9 +181,9 @@ The :ref:`default memory allocator <default-memory-allocators>` uses the
 .. c:function:: void* PyMem_Malloc(size_t n)
 
    Allocates *n* bytes and returns a pointer of type :c:type:`void\*` to the
-   allocated memory, or *NULL* if the request fails.
+   allocated memory, or ``NULL`` if the request fails.
 
-   Requesting zero bytes returns a distinct non-*NULL* pointer if possible, as
+   Requesting zero bytes returns a distinct non-``NULL`` pointer if possible, as
    if ``PyMem_Malloc(1)`` had been called instead. The memory will not have
    been initialized in any way.
 
@@ -191,11 +191,11 @@ The :ref:`default memory allocator <default-memory-allocators>` uses the
 .. c:function:: void* PyMem_Calloc(size_t nelem, size_t elsize)
 
    Allocates *nelem* elements each whose size in bytes is *elsize* and returns
-   a pointer of type :c:type:`void\*` to the allocated memory, or *NULL* if the
+   a pointer of type :c:type:`void\*` to the allocated memory, or ``NULL`` if the
    request fails. The memory is initialized to zeros.
 
    Requesting zero elements or elements of size zero bytes returns a distinct
-   non-*NULL* pointer if possible, as if ``PyMem_Calloc(1, 1)`` had been called
+   non-``NULL`` pointer if possible, as if ``PyMem_Calloc(1, 1)`` had been called
    instead.
 
    .. versionadded:: 3.5
@@ -206,14 +206,14 @@ The :ref:`default memory allocator <default-memory-allocators>` uses the
    Resizes the memory block pointed to by *p* to *n* bytes. The contents will be
    unchanged to the minimum of the old and the new sizes.
 
-   If *p* is *NULL*, the call is equivalent to ``PyMem_Malloc(n)``; else if *n*
+   If *p* is ``NULL``, the call is equivalent to ``PyMem_Malloc(n)``; else if *n*
    is equal to zero, the memory block is resized but is not freed, and the
-   returned pointer is non-*NULL*.
+   returned pointer is non-``NULL``.
 
-   Unless *p* is *NULL*, it must have been returned by a previous call to
+   Unless *p* is ``NULL``, it must have been returned by a previous call to
    :c:func:`PyMem_Malloc`, :c:func:`PyMem_Realloc` or :c:func:`PyMem_Calloc`.
 
-   If the request fails, :c:func:`PyMem_Realloc` returns *NULL* and *p* remains
+   If the request fails, :c:func:`PyMem_Realloc` returns ``NULL`` and *p* remains
    a valid pointer to the previous memory area.
 
 
@@ -224,7 +224,7 @@ The :ref:`default memory allocator <default-memory-allocators>` uses the
    :c:func:`PyMem_Calloc`.  Otherwise, or if ``PyMem_Free(p)`` has been called
    before, undefined behavior occurs.
 
-   If *p* is *NULL*, no operation is performed.
+   If *p* is ``NULL``, no operation is performed.
 
 The following type-oriented macros are provided for convenience.  Note  that
 *TYPE* refers to any C type.
@@ -241,7 +241,7 @@ The following type-oriented macros are provided for convenience.  Note  that
 
    Same as :c:func:`PyMem_Realloc`, but the memory block is resized to ``(n *
    sizeof(TYPE))`` bytes.  Returns a pointer cast to :c:type:`TYPE\*`. On return,
-   *p* will be a pointer to the new memory area, or *NULL* in the event of
+   *p* will be a pointer to the new memory area, or ``NULL`` in the event of
    failure.
 
    This is a C preprocessor macro; *p* is always reassigned.  Save the original
@@ -283,9 +283,9 @@ The :ref:`default object allocator <default-memory-allocators>` uses the
 .. c:function:: void* PyObject_Malloc(size_t n)
 
    Allocates *n* bytes and returns a pointer of type :c:type:`void\*` to the
-   allocated memory, or *NULL* if the request fails.
+   allocated memory, or ``NULL`` if the request fails.
 
-   Requesting zero bytes returns a distinct non-*NULL* pointer if possible, as
+   Requesting zero bytes returns a distinct non-``NULL`` pointer if possible, as
    if ``PyObject_Malloc(1)`` had been called instead. The memory will not have
    been initialized in any way.
 
@@ -293,11 +293,11 @@ The :ref:`default object allocator <default-memory-allocators>` uses the
 .. c:function:: void* PyObject_Calloc(size_t nelem, size_t elsize)
 
    Allocates *nelem* elements each whose size in bytes is *elsize* and returns
-   a pointer of type :c:type:`void\*` to the allocated memory, or *NULL* if the
+   a pointer of type :c:type:`void\*` to the allocated memory, or ``NULL`` if the
    request fails. The memory is initialized to zeros.
 
    Requesting zero elements or elements of size zero bytes returns a distinct
-   non-*NULL* pointer if possible, as if ``PyObject_Calloc(1, 1)`` had been called
+   non-``NULL`` pointer if possible, as if ``PyObject_Calloc(1, 1)`` had been called
    instead.
 
    .. versionadded:: 3.5
@@ -308,14 +308,14 @@ The :ref:`default object allocator <default-memory-allocators>` uses the
    Resizes the memory block pointed to by *p* to *n* bytes. The contents will be
    unchanged to the minimum of the old and the new sizes.
 
-   If *p* is *NULL*, the call is equivalent to ``PyObject_Malloc(n)``; else if *n*
+   If *p* is ``NULL``, the call is equivalent to ``PyObject_Malloc(n)``; else if *n*
    is equal to zero, the memory block is resized but is not freed, and the
-   returned pointer is non-*NULL*.
+   returned pointer is non-``NULL``.
 
-   Unless *p* is *NULL*, it must have been returned by a previous call to
+   Unless *p* is ``NULL``, it must have been returned by a previous call to
    :c:func:`PyObject_Malloc`, :c:func:`PyObject_Realloc` or :c:func:`PyObject_Calloc`.
 
-   If the request fails, :c:func:`PyObject_Realloc` returns *NULL* and *p* remains
+   If the request fails, :c:func:`PyObject_Realloc` returns ``NULL`` and *p* remains
    a valid pointer to the previous memory area.
 
 
@@ -326,7 +326,7 @@ The :ref:`default object allocator <default-memory-allocators>` uses the
    :c:func:`PyObject_Calloc`.  Otherwise, or if ``PyObject_Free(p)`` has been called
    before, undefined behavior occurs.
 
-   If *p* is *NULL*, no operation is performed.
+   If *p* is ``NULL``, no operation is performed.
 
 
 .. _default-memory-allocators:
diff --git a/Doc/c-api/memoryview.rst b/Doc/c-api/memoryview.rst
index 77afb020405a7..de429e5c11dc7 100644
--- a/Doc/c-api/memoryview.rst
+++ b/Doc/c-api/memoryview.rst
@@ -57,7 +57,7 @@ any other object.
 .. c:function:: Py_buffer *PyMemoryView_GET_BASE(PyObject *mview)
 
    Return either a pointer to the exporting object that the memoryview is based
-   on or *NULL* if the memoryview has been created by one of the functions
+   on or ``NULL`` if the memoryview has been created by one of the functions
    :c:func:`PyMemoryView_FromMemory` or :c:func:`PyMemoryView_FromBuffer`.
    *mview* **must** be a memoryview instance.
 
diff --git a/Doc/c-api/method.rst b/Doc/c-api/method.rst
index 1ad805e269aa2..b1862d796c9f4 100644
--- a/Doc/c-api/method.rst
+++ b/Doc/c-api/method.rst
@@ -21,7 +21,7 @@ to bind a :c:data:`PyCFunction` to a class object. It replaces the former call
 .. c:function:: int PyInstanceMethod_Check(PyObject *o)
 
    Return true if *o* is an instance method object (has type
-   :c:data:`PyInstanceMethod_Type`).  The parameter must not be *NULL*.
+   :c:data:`PyInstanceMethod_Type`).  The parameter must not be ``NULL``.
 
 
 .. c:function:: PyObject* PyInstanceMethod_New(PyObject *func)
@@ -64,14 +64,14 @@ no longer available.
 .. c:function:: int PyMethod_Check(PyObject *o)
 
    Return true if *o* is a method object (has type :c:data:`PyMethod_Type`).  The
-   parameter must not be *NULL*.
+   parameter must not be ``NULL``.
 
 
 .. c:function:: PyObject* PyMethod_New(PyObject *func, PyObject *self)
 
    Return a new method object, with *func* being any callable object and *self*
    the instance the method should be bound. *func* is the function that will
-   be called when the method is called. *self* must not be *NULL*.
+   be called when the method is called. *self* must not be ``NULL``.
 
 
 .. c:function:: PyObject* PyMethod_Function(PyObject *meth)
diff --git a/Doc/c-api/module.rst b/Doc/c-api/module.rst
index feca1ec2a057a..46306b3917cce 100644
--- a/Doc/c-api/module.rst
+++ b/Doc/c-api/module.rst
@@ -61,7 +61,7 @@ Module Objects
    Return the dictionary object that implements *module*'s namespace; this object
    is the same as the :attr:`~object.__dict__` attribute of the module object.
    If *module* is not a module object (or a subtype of a module object),
-   :exc:`SystemError` is raised and *NULL* is returned.
+   :exc:`SystemError` is raised and ``NULL`` is returned.
 
    It is recommended extensions use other :c:func:`PyModule_\*` and
    :c:func:`PyObject_\*` functions rather than directly manipulate a module's
@@ -75,7 +75,7 @@ Module Objects
       single: SystemError (built-in exception)
 
    Return *module*'s :attr:`__name__` value.  If the module does not provide one,
-   or if it is not a string, :exc:`SystemError` is raised and *NULL* is returned.
+   or if it is not a string, :exc:`SystemError` is raised and ``NULL`` is returned.
 
    .. versionadded:: 3.3
 
@@ -88,14 +88,14 @@ Module Objects
 .. c:function:: void* PyModule_GetState(PyObject *module)
 
    Return the "state" of the module, that is, a pointer to the block of memory
-   allocated at module creation time, or *NULL*.  See
+   allocated at module creation time, or ``NULL``.  See
    :c:member:`PyModuleDef.m_size`.
 
 
 .. c:function:: PyModuleDef* PyModule_GetDef(PyObject *module)
 
    Return a pointer to the :c:type:`PyModuleDef` struct from which the module was
-   created, or *NULL* if the module wasn't created from a definition.
+   created, or ``NULL`` if the module wasn't created from a definition.
 
 
 .. c:function:: PyObject* PyModule_GetFilenameObject(PyObject *module)
@@ -106,7 +106,7 @@ Module Objects
 
    Return the name of the file from which *module* was loaded using *module*'s
    :attr:`__file__` attribute.  If this is not defined, or if it is not a
-   unicode string, raise :exc:`SystemError` and return *NULL*; otherwise return
+   unicode string, raise :exc:`SystemError` and return ``NULL``; otherwise return
    a reference to a Unicode object.
 
    .. versionadded:: 3.2
@@ -178,17 +178,17 @@ or request "multi-phase initialization" by returning the definition struct itsel
    .. c:member:: PyMethodDef* m_methods
 
       A pointer to a table of module-level functions, described by
-      :c:type:`PyMethodDef` values.  Can be *NULL* if no functions are present.
+      :c:type:`PyMethodDef` values.  Can be ``NULL`` if no functions are present.
 
    .. c:member:: PyModuleDef_Slot* m_slots
 
       An array of slot definitions for multi-phase initialization, terminated by
       a ``{0, NULL}`` entry.
-      When using single-phase initialization, *m_slots* must be *NULL*.
+      When using single-phase initialization, *m_slots* must be ``NULL``.
 
       .. versionchanged:: 3.5
 
-         Prior to version 3.5, this member was always set to *NULL*,
+         Prior to version 3.5, this member was always set to ``NULL``,
          and was defined as:
 
            .. c:member:: inquiry m_reload
@@ -196,20 +196,20 @@ or request "multi-phase initialization" by returning the definition struct itsel
    .. c:member:: traverseproc m_traverse
 
       A traversal function to call during GC traversal of the module object, or
-      *NULL* if not needed. This function may be called before module state
+      ``NULL`` if not needed. This function may be called before module state
       is allocated (:c:func:`PyModule_GetState()` may return `NULL`),
       and before the :c:member:`Py_mod_exec` function is executed.
 
    .. c:member:: inquiry m_clear
 
       A clear function to call during GC clearing of the module object, or
-      *NULL* if not needed. This function may be called before module state
+      ``NULL`` if not needed. This function may be called before module state
       is allocated (:c:func:`PyModule_GetState()` may return `NULL`),
       and before the :c:member:`Py_mod_exec` function is executed.
 
    .. c:member:: freefunc m_free
 
-      A function to call during deallocation of the module object, or *NULL* if
+      A function to call during deallocation of the module object, or ``NULL`` if
       not needed. This function may be called before module state
       is allocated (:c:func:`PyModule_GetState()` may return `NULL`),
       and before the :c:member:`Py_mod_exec` function is executed.
@@ -278,7 +278,7 @@ instance must be initialized with the following function:
    Ensures a module definition is a properly initialized Python object that
    correctly reports its type and reference count.
 
-   Returns *def* cast to ``PyObject*``, or *NULL* if an error occurred.
+   Returns *def* cast to ``PyObject*``, or ``NULL`` if an error occurred.
 
    .. versionadded:: 3.5
 
@@ -311,7 +311,7 @@ The available slot types are:
    The function receives a :py:class:`~importlib.machinery.ModuleSpec`
    instance, as defined in :PEP:`451`, and the module definition.
    It should return a new module object, or set an error
-   and return *NULL*.
+   and return ``NULL``.
 
    This function should be kept minimal. In particular, it should not
    call arbitrary Python code, as trying to import the same module again may
@@ -330,7 +330,7 @@ The available slot types are:
    :c:type:`PyModule_Type`. Any type can be used, as long as it supports
    setting and getting import-related attributes.
    However, only ``PyModule_Type`` instances may be returned if the
-   ``PyModuleDef`` has non-*NULL* ``m_traverse``, ``m_clear``,
+   ``PyModuleDef`` has non-``NULL`` ``m_traverse``, ``m_clear``,
    ``m_free``; non-zero ``m_size``; or slots other than ``Py_mod_create``.
 
 .. c:var:: Py_mod_exec
@@ -394,7 +394,7 @@ objects dynamically. Note that both ``PyModule_FromDefAndSpec`` and
 
 .. c:function:: int PyModule_AddFunctions(PyObject *module, PyMethodDef *functions)
 
-   Add the functions from the *NULL* terminated *functions* array to *module*.
+   Add the functions from the ``NULL`` terminated *functions* array to *module*.
    Refer to the :c:type:`PyMethodDef` documentation for details on individual
    entries (due to the lack of a shared module namespace, module level
    "functions" implemented in C typically receive the module as their first
@@ -445,7 +445,7 @@ state:
 
    Add a string constant to *module* as *name*.  This convenience function can be
    used from the module's initialization function.  The string *value* must be
-   *NULL*-terminated.  Return ``-1`` on error, ``0`` on success.
+   ``NULL``-terminated.  Return ``-1`` on error, ``0`` on success.
 
 
 .. c:function:: int PyModule_AddIntMacro(PyObject *module, macro)
@@ -476,7 +476,7 @@ since multiple such modules can be created from a single definition.
    Returns the module object that was created from *def* for the current interpreter.
    This method requires that the module object has been attached to the interpreter state with
    :c:func:`PyState_AddModule` beforehand. In case the corresponding module object is not
-   found or has not been attached to the interpreter state yet, it returns *NULL*.
+   found or has not been attached to the interpreter state yet, it returns ``NULL``.
 
 .. c:function:: int PyState_AddModule(PyObject *module, PyModuleDef *def)
 
diff --git a/Doc/c-api/number.rst b/Doc/c-api/number.rst
index 74932f6e7445f..55ddb9c0c8206 100644
--- a/Doc/c-api/number.rst
+++ b/Doc/c-api/number.rst
@@ -17,25 +17,25 @@ Number Protocol
 
 .. c:function:: PyObject* PyNumber_Add(PyObject *o1, PyObject *o2)
 
-   Returns the result of adding *o1* and *o2*, or *NULL* on failure.  This is the
+   Returns the result of adding *o1* and *o2*, or ``NULL`` on failure.  This is the
    equivalent of the Python expression ``o1 + o2``.
 
 
 .. c:function:: PyObject* PyNumber_Subtract(PyObject *o1, PyObject *o2)
 
-   Returns the result of subtracting *o2* from *o1*, or *NULL* on failure.  This is
+   Returns the result of subtracting *o2* from *o1*, or ``NULL`` on failure.  This is
    the equivalent of the Python expression ``o1 - o2``.
 
 
 .. c:function:: PyObject* PyNumber_Multiply(PyObject *o1, PyObject *o2)
 
-   Returns the result of multiplying *o1* and *o2*, or *NULL* on failure.  This is
+   Returns the result of multiplying *o1* and *o2*, or ``NULL`` on failure.  This is
    the equivalent of the Python expression ``o1 * o2``.
 
 
 .. c:function:: PyObject* PyNumber_MatrixMultiply(PyObject *o1, PyObject *o2)
 
-   Returns the result of matrix multiplication on *o1* and *o2*, or *NULL* on
+   Returns the result of matrix multiplication on *o1* and *o2*, or ``NULL`` on
    failure.  This is the equivalent of the Python expression ``o1 @ o2``.
 
    .. versionadded:: 3.5
@@ -43,14 +43,14 @@ Number Protocol
 
 .. c:function:: PyObject* PyNumber_FloorDivide(PyObject *o1, PyObject *o2)
 
-   Return the floor of *o1* divided by *o2*, or *NULL* on failure.  This is
+   Return the floor of *o1* divided by *o2*, or ``NULL`` on failure.  This is
    equivalent to the "classic" division of integers.
 
 
 .. c:function:: PyObject* PyNumber_TrueDivide(PyObject *o1, PyObject *o2)
 
    Return a reasonable approximation for the mathematical value of *o1* divided by
-   *o2*, or *NULL* on failure.  The return value is "approximate" because binary
+   *o2*, or ``NULL`` on failure.  The return value is "approximate" because binary
    floating point numbers are approximate; it is not possible to represent all real
    numbers in base two.  This function can return a floating point value when
    passed two integers.
@@ -58,7 +58,7 @@ Number Protocol
 
 .. c:function:: PyObject* PyNumber_Remainder(PyObject *o1, PyObject *o2)
 
-   Returns the remainder of dividing *o1* by *o2*, or *NULL* on failure.  This is
+   Returns the remainder of dividing *o1* by *o2*, or ``NULL`` on failure.  This is
    the equivalent of the Python expression ``o1 % o2``.
 
 
@@ -66,7 +66,7 @@ Number Protocol
 
    .. index:: builtin: divmod
 
-   See the built-in function :func:`divmod`. Returns *NULL* on failure.  This is
+   See the built-in function :func:`divmod`. Returns ``NULL`` on failure.  This is
    the equivalent of the Python expression ``divmod(o1, o2)``.
 
 
@@ -74,21 +74,21 @@ Number Protocol
 
    .. index:: builtin: pow
 
-   See the built-in function :func:`pow`. Returns *NULL* on failure.  This is the
+   See the built-in function :func:`pow`. Returns ``NULL`` on failure.  This is the
    equivalent of the Python expression ``pow(o1, o2, o3)``, where *o3* is optional.
-   If *o3* is to be ignored, pass :c:data:`Py_None` in its place (passing *NULL* for
+   If *o3* is to be ignored, pass :c:data:`Py_None` in its place (passing ``NULL`` for
    *o3* would cause an illegal memory access).
 
 
 .. c:function:: PyObject* PyNumber_Negative(PyObject *o)
 
-   Returns the negation of *o* on success, or *NULL* on failure. This is the
+   Returns the negation of *o* on success, or ``NULL`` on failure. This is the
    equivalent of the Python expression ``-o``.
 
 
 .. c:function:: PyObject* PyNumber_Positive(PyObject *o)
 
-   Returns *o* on success, or *NULL* on failure.  This is the equivalent of the
+   Returns *o* on success, or ``NULL`` on failure.  This is the equivalent of the
    Python expression ``+o``.
 
 
@@ -96,70 +96,70 @@ Number Protocol
 
    .. index:: builtin: abs
 
-   Returns the absolute value of *o*, or *NULL* on failure.  This is the equivalent
+   Returns the absolute value of *o*, or ``NULL`` on failure.  This is the equivalent
    of the Python expression ``abs(o)``.
 
 
 .. c:function:: PyObject* PyNumber_Invert(PyObject *o)
 
-   Returns the bitwise negation of *o* on success, or *NULL* on failure.  This is
+   Returns the bitwise negation of *o* on success, or ``NULL`` on failure.  This is
    the equivalent of the Python expression ``~o``.
 
 
 .. c:function:: PyObject* PyNumber_Lshift(PyObject *o1, PyObject *o2)
 
-   Returns the result of left shifting *o1* by *o2* on success, or *NULL* on
+   Returns the result of left shifting *o1* by *o2* on success, or ``NULL`` on
    failure.  This is the equivalent of the Python expression ``o1 << o2``.
 
 
 .. c:function:: PyObject* PyNumber_Rshift(PyObject *o1, PyObject *o2)
 
-   Returns the result of right shifting *o1* by *o2* on success, or *NULL* on
+   Returns the result of right shifting *o1* by *o2* on success, or ``NULL`` on
    failure.  This is the equivalent of the Python expression ``o1 >> o2``.
 
 
 .. c:function:: PyObject* PyNumber_And(PyObject *o1, PyObject *o2)
 
-   Returns the "bitwise and" of *o1* and *o2* on success and *NULL* on failure.
+   Returns the "bitwise and" of *o1* and *o2* on success and ``NULL`` on failure.
    This is the equivalent of the Python expression ``o1 & o2``.
 
 
 .. c:function:: PyObject* PyNumber_Xor(PyObject *o1, PyObject *o2)
 
-   Returns the "bitwise exclusive or" of *o1* by *o2* on success, or *NULL* on
+   Returns the "bitwise exclusive or" of *o1* by *o2* on success, or ``NULL`` on
    failure.  This is the equivalent of the Python expression ``o1 ^ o2``.
 
 
 .. c:function:: PyObject* PyNumber_Or(PyObject *o1, PyObject *o2)
 
-   Returns the "bitwise or" of *o1* and *o2* on success, or *NULL* on failure.
+   Returns the "bitwise or" of *o1* and *o2* on success, or ``NULL`` on failure.
    This is the equivalent of the Python expression ``o1 | o2``.
 
 
 .. c:function:: PyObject* PyNumber_InPlaceAdd(PyObject *o1, PyObject *o2)
 
-   Returns the result of adding *o1* and *o2*, or *NULL* on failure.  The operation
+   Returns the result of adding *o1* and *o2*, or ``NULL`` on failure.  The operation
    is done *in-place* when *o1* supports it.  This is the equivalent of the Python
    statement ``o1 += o2``.
 
 
 .. c:function:: PyObject* PyNumber_InPlaceSubtract(PyObject *o1, PyObject *o2)
 
-   Returns the result of subtracting *o2* from *o1*, or *NULL* on failure.  The
+   Returns the result of subtracting *o2* from *o1*, or ``NULL`` on failure.  The
    operation is done *in-place* when *o1* supports it.  This is the equivalent of
    the Python statement ``o1 -= o2``.
 
 
 .. c:function:: PyObject* PyNumber_InPlaceMultiply(PyObject *o1, PyObject *o2)
 
-   Returns the result of multiplying *o1* and *o2*, or *NULL* on failure.  The
+   Returns the result of multiplying *o1* and *o2*, or ``NULL`` on failure.  The
    operation is done *in-place* when *o1* supports it.  This is the equivalent of
    the Python statement ``o1 *= o2``.
 
 
 .. c:function:: PyObject* PyNumber_InPlaceMatrixMultiply(PyObject *o1, PyObject *o2)
 
-   Returns the result of matrix multiplication on *o1* and *o2*, or *NULL* on
+   Returns the result of matrix multiplication on *o1* and *o2*, or ``NULL`` on
    failure.  The operation is done *in-place* when *o1* supports it.  This is
    the equivalent of the Python statement ``o1 @= o2``.
 
@@ -168,7 +168,7 @@ Number Protocol
 
 .. c:function:: PyObject* PyNumber_InPlaceFloorDivide(PyObject *o1, PyObject *o2)
 
-   Returns the mathematical floor of dividing *o1* by *o2*, or *NULL* on failure.
+   Returns the mathematical floor of dividing *o1* by *o2*, or ``NULL`` on failure.
    The operation is done *in-place* when *o1* supports it.  This is the equivalent
    of the Python statement ``o1 //= o2``.
 
@@ -176,7 +176,7 @@ Number Protocol
 .. c:function:: PyObject* PyNumber_InPlaceTrueDivide(PyObject *o1, PyObject *o2)
 
    Return a reasonable approximation for the mathematical value of *o1* divided by
-   *o2*, or *NULL* on failure.  The return value is "approximate" because binary
+   *o2*, or ``NULL`` on failure.  The return value is "approximate" because binary
    floating point numbers are approximate; it is not possible to represent all real
    numbers in base two.  This function can return a floating point value when
    passed two integers.  The operation is done *in-place* when *o1* supports it.
@@ -184,7 +184,7 @@ Number Protocol
 
 .. c:function:: PyObject* PyNumber_InPlaceRemainder(PyObject *o1, PyObject *o2)
 
-   Returns the remainder of dividing *o1* by *o2*, or *NULL* on failure.  The
+   Returns the remainder of dividing *o1* by *o2*, or ``NULL`` on failure.  The
    operation is done *in-place* when *o1* supports it.  This is the equivalent of
    the Python statement ``o1 %= o2``.
 
@@ -193,44 +193,44 @@ Number Protocol
 
    .. index:: builtin: pow
 
-   See the built-in function :func:`pow`. Returns *NULL* on failure.  The operation
+   See the built-in function :func:`pow`. Returns ``NULL`` on failure.  The operation
    is done *in-place* when *o1* supports it.  This is the equivalent of the Python
    statement ``o1 **= o2`` when o3 is :c:data:`Py_None`, or an in-place variant of
    ``pow(o1, o2, o3)`` otherwise. If *o3* is to be ignored, pass :c:data:`Py_None`
-   in its place (passing *NULL* for *o3* would cause an illegal memory access).
+   in its place (passing ``NULL`` for *o3* would cause an illegal memory access).
 
 
 .. c:function:: PyObject* PyNumber_InPlaceLshift(PyObject *o1, PyObject *o2)
 
-   Returns the result of left shifting *o1* by *o2* on success, or *NULL* on
+   Returns the result of left shifting *o1* by *o2* on success, or ``NULL`` on
    failure.  The operation is done *in-place* when *o1* supports it.  This is the
    equivalent of the Python statement ``o1 <<= o2``.
 
 
 .. c:function:: PyObject* PyNumber_InPlaceRshift(PyObject *o1, PyObject *o2)
 
-   Returns the result of right shifting *o1* by *o2* on success, or *NULL* on
+   Returns the result of right shifting *o1* by *o2* on success, or ``NULL`` on
    failure.  The operation is done *in-place* when *o1* supports it.  This is the
    equivalent of the Python statement ``o1 >>= o2``.
 
 
 .. c:function:: PyObject* PyNumber_InPlaceAnd(PyObject *o1, PyObject *o2)
 
-   Returns the "bitwise and" of *o1* and *o2* on success and *NULL* on failure. The
+   Returns the "bitwise and" of *o1* and *o2* on success and ``NULL`` on failure. The
    operation is done *in-place* when *o1* supports it.  This is the equivalent of
    the Python statement ``o1 &= o2``.
 
 
 .. c:function:: PyObject* PyNumber_InPlaceXor(PyObject *o1, PyObject *o2)
 
-   Returns the "bitwise exclusive or" of *o1* by *o2* on success, or *NULL* on
+   Returns the "bitwise exclusive or" of *o1* by *o2* on success, or ``NULL`` on
    failure.  The operation is done *in-place* when *o1* supports it.  This is the
    equivalent of the Python statement ``o1 ^= o2``.
 
 
 .. c:function:: PyObject* PyNumber_InPlaceOr(PyObject *o1, PyObject *o2)
 
-   Returns the "bitwise or" of *o1* and *o2* on success, or *NULL* on failure.  The
+   Returns the "bitwise or" of *o1* and *o2* on success, or ``NULL`` on failure.  The
    operation is done *in-place* when *o1* supports it.  This is the equivalent of
    the Python statement ``o1 |= o2``.
 
@@ -239,7 +239,7 @@ Number Protocol
 
    .. index:: builtin: int
 
-   Returns the *o* converted to an integer object on success, or *NULL* on
+   Returns the *o* converted to an integer object on success, or ``NULL`` on
    failure.  This is the equivalent of the Python expression ``int(o)``.
 
 
@@ -247,13 +247,13 @@ Number Protocol
 
    .. index:: builtin: float
 
-   Returns the *o* converted to a float object on success, or *NULL* on failure.
+   Returns the *o* converted to a float object on success, or ``NULL`` on failure.
    This is the equivalent of the Python expression ``float(o)``.
 
 
 .. c:function:: PyObject* PyNumber_Index(PyObject *o)
 
-   Returns the *o* converted to a Python int on success or *NULL* with a
+   Returns the *o* converted to a Python int on success or ``NULL`` with a
    :exc:`TypeError` exception raised on failure.
 
 
@@ -274,7 +274,7 @@ Number Protocol
    If *o* can be converted to a Python int but the attempt to
    convert to a Py_ssize_t value would raise an :exc:`OverflowError`, then the
    *exc* argument is the type of exception that will be raised (usually
-   :exc:`IndexError` or :exc:`OverflowError`).  If *exc* is *NULL*, then the
+   :exc:`IndexError` or :exc:`OverflowError`).  If *exc* is ``NULL``, then the
    exception is cleared and the value is clipped to *PY_SSIZE_T_MIN* for a negative
    integer or *PY_SSIZE_T_MAX* for a positive integer.
 
diff --git a/Doc/c-api/object.rst b/Doc/c-api/object.rst
index fd1e9c65aaba9..0b93064028b8e 100644
--- a/Doc/c-api/object.rst
+++ b/Doc/c-api/object.rst
@@ -53,14 +53,14 @@ Object Protocol
 .. c:function:: PyObject* PyObject_GetAttr(PyObject *o, PyObject *attr_name)
 
    Retrieve an attribute named *attr_name* from object *o*. Returns the attribute
-   value on success, or *NULL* on failure.  This is the equivalent of the Python
+   value on success, or ``NULL`` on failure.  This is the equivalent of the Python
    expression ``o.attr_name``.
 
 
 .. c:function:: PyObject* PyObject_GetAttrString(PyObject *o, const char *attr_name)
 
    Retrieve an attribute named *attr_name* from object *o*. Returns the attribute
-   value on success, or *NULL* on failure. This is the equivalent of the Python
+   value on success, or ``NULL`` on failure. This is the equivalent of the Python
    expression ``o.attr_name``.
 
 
@@ -81,7 +81,7 @@ Object Protocol
    return ``0`` on success.  This is the equivalent of the Python statement
    ``o.attr_name = v``.
 
-   If *v* is *NULL*, the attribute is deleted, however this feature is
+   If *v* is ``NULL``, the attribute is deleted, however this feature is
    deprecated in favour of using :c:func:`PyObject_DelAttr`.
 
 
@@ -92,7 +92,7 @@ Object Protocol
    return ``0`` on success.  This is the equivalent of the Python statement
    ``o.attr_name = v``.
 
-   If *v* is *NULL*, the attribute is deleted, however this feature is
+   If *v* is ``NULL``, the attribute is deleted, however this feature is
    deprecated in favour of using :c:func:`PyObject_DelAttrString`.
 
 
@@ -143,7 +143,7 @@ Object Protocol
    :const:`Py_NE`, :const:`Py_GT`, or :const:`Py_GE`, corresponding to ``<``,
    ``<=``, ``==``, ``!=``, ``>``, or ``>=`` respectively. This is the equivalent of
    the Python expression ``o1 op o2``, where ``op`` is the operator corresponding
-   to *opid*. Returns the value of the comparison on success, or *NULL* on failure.
+   to *opid*. Returns the value of the comparison on success, or ``NULL`` on failure.
 
 
 .. c:function:: int PyObject_RichCompareBool(PyObject *o1, PyObject *o2, int opid)
@@ -165,7 +165,7 @@ Object Protocol
    .. index:: builtin: repr
 
    Compute a string representation of object *o*.  Returns the string
-   representation on success, *NULL* on failure.  This is the equivalent of the
+   representation on success, ``NULL`` on failure.  This is the equivalent of the
    Python expression ``repr(o)``.  Called by the :func:`repr` built-in function.
 
    .. versionchanged:: 3.4
@@ -188,7 +188,7 @@ Object Protocol
 .. c:function:: PyObject* PyObject_Str(PyObject *o)
 
    Compute a string representation of object *o*.  Returns the string
-   representation on success, *NULL* on failure.  This is the equivalent of the
+   representation on success, ``NULL`` on failure.  This is the equivalent of the
    Python expression ``str(o)``.  Called by the :func:`str` built-in function
    and, therefore, by the :func:`print` function.
 
@@ -200,7 +200,7 @@ Object Protocol
 
    .. index:: builtin: bytes
 
-   Compute a bytes representation of object *o*.  *NULL* is returned on
+   Compute a bytes representation of object *o*.  ``NULL`` is returned on
    failure and a bytes object on success.  This is equivalent to the Python
    expression ``bytes(o)``, when *o* is not an integer.  Unlike ``bytes(o)``,
    a TypeError is raised when *o* is an integer instead of a zero-initialized
@@ -259,7 +259,7 @@ Object Protocol
    most efficient way to call a callable Python object without any argument.
 
    Return the result of the call on success, or raise an exception and return
-   *NULL* on failure.
+   ``NULL`` on failure.
 
    .. versionadded:: 3.9
 
@@ -270,7 +270,7 @@ Object Protocol
    *arg* and no keyword arguments.
 
    Return the result of the call on success, or raise an exception and return
-   *NULL* on failure.
+   ``NULL`` on failure.
 
    .. versionadded:: 3.9
 
@@ -280,11 +280,11 @@ Object Protocol
    Call a callable Python object *callable*, with arguments given by the
    tuple *args*, and named arguments given by the dictionary *kwargs*.
 
-   *args* must not be *NULL*, use an empty tuple if no arguments are needed.
-   If no named arguments are needed, *kwargs* can be *NULL*.
+   *args* must not be ``NULL``, use an empty tuple if no arguments are needed.
+   If no named arguments are needed, *kwargs* can be ``NULL``.
 
    Return the result of the call on success, or raise an exception and return
-   *NULL* on failure.
+   ``NULL`` on failure.
 
    This is the equivalent of the Python expression:
    ``callable(*args, **kwargs)``.
@@ -293,10 +293,10 @@ Object Protocol
 .. c:function:: PyObject* PyObject_CallObject(PyObject *callable, PyObject *args)
 
    Call a callable Python object *callable*, with arguments given by the
-   tuple *args*.  If no arguments are needed, then *args* can be *NULL*.
+   tuple *args*.  If no arguments are needed, then *args* can be ``NULL``.
 
    Return the result of the call on success, or raise an exception and return
-   *NULL* on failure.
+   ``NULL`` on failure.
 
    This is the equivalent of the Python expression: ``callable(*args)``.
 
@@ -305,10 +305,10 @@ Object Protocol
 
    Call a callable Python object *callable*, with a variable number of C arguments.
    The C arguments are described using a :c:func:`Py_BuildValue` style format
-   string.  The format can be *NULL*, indicating that no arguments are provided.
+   string.  The format can be ``NULL``, indicating that no arguments are provided.
 
    Return the result of the call on success, or raise an exception and return
-   *NULL* on failure.
+   ``NULL`` on failure.
 
    This is the equivalent of the Python expression: ``callable(*args)``.
 
@@ -325,10 +325,10 @@ Object Protocol
    arguments.  The C arguments are described by a :c:func:`Py_BuildValue` format
    string that should  produce a tuple.
 
-   The format can be *NULL*, indicating that no arguments are provided.
+   The format can be ``NULL``, indicating that no arguments are provided.
 
    Return the result of the call on success, or raise an exception and return
-   *NULL* on failure.
+   ``NULL`` on failure.
 
    This is the equivalent of the Python expression:
    ``obj.name(arg1, arg2, ...)``.
@@ -344,10 +344,10 @@ Object Protocol
 
    Call a callable Python object *callable*, with a variable number of
    :c:type:`PyObject\*` arguments.  The arguments are provided as a variable number
-   of parameters followed by *NULL*.
+   of parameters followed by ``NULL``.
 
    Return the result of the call on success, or raise an exception and return
-   *NULL* on failure.
+   ``NULL`` on failure.
 
    This is the equivalent of the Python expression:
    ``callable(arg1, arg2, ...)``.
@@ -358,10 +358,10 @@ Object Protocol
    Calls a method of the Python object *obj*, where the name of the method is given as a
    Python string object in *name*.  It is called with a variable number of
    :c:type:`PyObject\*` arguments.  The arguments are provided as a variable number
-   of parameters followed by *NULL*.
+   of parameters followed by ``NULL``.
 
    Return the result of the call on success, or raise an exception and return
-   *NULL* on failure.
+   ``NULL`` on failure.
 
 
 .. c:function:: PyObject* _PyObject_CallMethodNoArgs(PyObject *obj, PyObject *name)
@@ -370,7 +370,7 @@ Object Protocol
    where the name of the method is given as a Python string object in *name*.
 
    Return the result of the call on success, or raise an exception and return
-   *NULL* on failure.
+   ``NULL`` on failure.
 
    .. versionadded:: 3.9
 
@@ -382,7 +382,7 @@ Object Protocol
    *name*.
 
    Return the result of the call on success, or raise an exception and return
-   *NULL* on failure.
+   ``NULL`` on failure.
 
    .. versionadded:: 3.9
 
@@ -408,7 +408,7 @@ Object Protocol
    and all keys must be unique.
 
    Return the result of the call on success, or raise an exception and return
-   *NULL* on failure.
+   ``NULL`` on failure.
 
    This uses the vectorcall protocol if the callable supports it;
    otherwise, the arguments are converted to use
@@ -450,7 +450,7 @@ Object Protocol
 .. c:function:: PyObject* _PyObject_FastCallDict(PyObject *callable, PyObject *const *args, size_t nargsf, PyObject *kwdict)
 
    Same as :c:func:`_PyObject_Vectorcall` except that the keyword arguments
-   are passed as a dictionary in *kwdict*. This may be *NULL* if there
+   are passed as a dictionary in *kwdict*. This may be ``NULL`` if there
    are no keyword arguments.
 
    For callables supporting :c:data:`vectorcall <PyTypeObject.tp_vectorcall_offset>`,
@@ -483,7 +483,7 @@ Object Protocol
    *args* vector as arguments.
 
    Return the result of the call on success, or raise an exception and return
-   *NULL* on failure.
+   ``NULL`` on failure.
 
    .. versionadded:: 3.9
 
@@ -525,8 +525,8 @@ Object Protocol
 
    .. index:: builtin: type
 
-   When *o* is non-*NULL*, returns a type object corresponding to the object type
-   of object *o*. On failure, raises :exc:`SystemError` and returns *NULL*.  This
+   When *o* is non-``NULL``, returns a type object corresponding to the object type
+   of object *o*. On failure, raises :exc:`SystemError` and returns ``NULL``.  This
    is equivalent to the Python expression ``type(o)``. This function increments the
    reference count of the return value. There's really no reason to use this
    function instead of the common expression ``o->ob_type``, which returns a
@@ -537,7 +537,7 @@ Object Protocol
 .. c:function:: int PyObject_TypeCheck(PyObject *o, PyTypeObject *type)
 
    Return true if the object *o* is of type *type* or a subtype of *type*.  Both
-   parameters must be non-*NULL*.
+   parameters must be non-``NULL``.
 
 
 .. c:function:: Py_ssize_t PyObject_Size(PyObject *o)
@@ -562,7 +562,7 @@ Object Protocol
 
 .. c:function:: PyObject* PyObject_GetItem(PyObject *o, PyObject *key)
 
-   Return element of *o* corresponding to the object *key* or *NULL* on failure.
+   Return element of *o* corresponding to the object *key* or ``NULL`` on failure.
    This is the equivalent of the Python expression ``o[key]``.
 
 
@@ -582,15 +582,15 @@ Object Protocol
 .. c:function:: PyObject* PyObject_Dir(PyObject *o)
 
    This is equivalent to the Python expression ``dir(o)``, returning a (possibly
-   empty) list of strings appropriate for the object argument, or *NULL* if there
-   was an error.  If the argument is *NULL*, this is like the Python ``dir()``,
+   empty) list of strings appropriate for the object argument, or ``NULL`` if there
+   was an error.  If the argument is ``NULL``, this is like the Python ``dir()``,
    returning the names of the current locals; in this case, if no execution frame
-   is active then *NULL* is returned but :c:func:`PyErr_Occurred` will return false.
+   is active then ``NULL`` is returned but :c:func:`PyErr_Occurred` will return false.
 
 
 .. c:function:: PyObject* PyObject_GetIter(PyObject *o)
 
    This is equivalent to the Python expression ``iter(o)``. It returns a new
    iterator for the object argument, or the object  itself if the object is already
-   an iterator.  Raises :exc:`TypeError` and returns *NULL* if the object cannot be
+   an iterator.  Raises :exc:`TypeError` and returns ``NULL`` if the object cannot be
    iterated.
diff --git a/Doc/c-api/refcounting.rst b/Doc/c-api/refcounting.rst
index 6b07c87d62ede..0df12c67f40bc 100644
--- a/Doc/c-api/refcounting.rst
+++ b/Doc/c-api/refcounting.rst
@@ -13,22 +13,22 @@ objects.
 
 .. c:function:: void Py_INCREF(PyObject *o)
 
-   Increment the reference count for object *o*.  The object must not be *NULL*; if
-   you aren't sure that it isn't *NULL*, use :c:func:`Py_XINCREF`.
+   Increment the reference count for object *o*.  The object must not be ``NULL``; if
+   you aren't sure that it isn't ``NULL``, use :c:func:`Py_XINCREF`.
 
 
 .. c:function:: void Py_XINCREF(PyObject *o)
 
-   Increment the reference count for object *o*.  The object may be *NULL*, in
+   Increment the reference count for object *o*.  The object may be ``NULL``, in
    which case the macro has no effect.
 
 
 .. c:function:: void Py_DECREF(PyObject *o)
 
-   Decrement the reference count for object *o*.  The object must not be *NULL*; if
-   you aren't sure that it isn't *NULL*, use :c:func:`Py_XDECREF`.  If the reference
+   Decrement the reference count for object *o*.  The object must not be ``NULL``; if
+   you aren't sure that it isn't ``NULL``, use :c:func:`Py_XDECREF`.  If the reference
    count reaches zero, the object's type's deallocation function (which must not be
-   *NULL*) is invoked.
+   ``NULL``) is invoked.
 
    .. warning::
 
@@ -44,18 +44,18 @@ objects.
 
 .. c:function:: void Py_XDECREF(PyObject *o)
 
-   Decrement the reference count for object *o*.  The object may be *NULL*, in
+   Decrement the reference count for object *o*.  The object may be ``NULL``, in
    which case the macro has no effect; otherwise the effect is the same as for
    :c:func:`Py_DECREF`, and the same warning applies.
 
 
 .. c:function:: void Py_CLEAR(PyObject *o)
 
-   Decrement the reference count for object *o*.  The object may be *NULL*, in
+   Decrement the reference count for object *o*.  The object may be ``NULL``, in
    which case the macro has no effect; otherwise the effect is the same as for
-   :c:func:`Py_DECREF`, except that the argument is also set to *NULL*.  The warning
+   :c:func:`Py_DECREF`, except that the argument is also set to ``NULL``.  The warning
    for :c:func:`Py_DECREF` does not apply with respect to the object passed because
-   the macro carefully uses a temporary variable and sets the argument to *NULL*
+   the macro carefully uses a temporary variable and sets the argument to ``NULL``
    before decrementing its reference count.
 
    It is a good idea to use this macro whenever decrementing the reference
diff --git a/Doc/c-api/reflection.rst b/Doc/c-api/reflection.rst
index 080ea3222cfbb..1d86de66ed3ed 100644
--- a/Doc/c-api/reflection.rst
+++ b/Doc/c-api/reflection.rst
@@ -14,18 +14,18 @@ Reflection
 .. c:function:: PyObject* PyEval_GetLocals()
 
    Return a dictionary of the local variables in the current execution frame,
-   or *NULL* if no frame is currently executing.
+   or ``NULL`` if no frame is currently executing.
 
 
 .. c:function:: PyObject* PyEval_GetGlobals()
 
    Return a dictionary of the global variables in the current execution frame,
-   or *NULL* if no frame is currently executing.
+   or ``NULL`` if no frame is currently executing.
 
 
 .. c:function:: PyFrameObject* PyEval_GetFrame()
 
-   Return the current thread state's frame, which is *NULL* if no frame is
+   Return the current thread state's frame, which is ``NULL`` if no frame is
    currently executing.
 
 
diff --git a/Doc/c-api/sequence.rst b/Doc/c-api/sequence.rst
index fe1c252ce9ba3..6581885904117 100644
--- a/Doc/c-api/sequence.rst
+++ b/Doc/c-api/sequence.rst
@@ -26,39 +26,39 @@ Sequence Protocol
 
 .. c:function:: PyObject* PySequence_Concat(PyObject *o1, PyObject *o2)
 
-   Return the concatenation of *o1* and *o2* on success, and *NULL* on failure.
+   Return the concatenation of *o1* and *o2* on success, and ``NULL`` on failure.
    This is the equivalent of the Python expression ``o1 + o2``.
 
 
 .. c:function:: PyObject* PySequence_Repeat(PyObject *o, Py_ssize_t count)
 
-   Return the result of repeating sequence object *o* *count* times, or *NULL* on
+   Return the result of repeating sequence object *o* *count* times, or ``NULL`` on
    failure.  This is the equivalent of the Python expression ``o * count``.
 
 
 .. c:function:: PyObject* PySequence_InPlaceConcat(PyObject *o1, PyObject *o2)
 
-   Return the concatenation of *o1* and *o2* on success, and *NULL* on failure.
+   Return the concatenation of *o1* and *o2* on success, and ``NULL`` on failure.
    The operation is done *in-place* when *o1* supports it.  This is the equivalent
    of the Python expression ``o1 += o2``.
 
 
 .. c:function:: PyObject* PySequence_InPlaceRepeat(PyObject *o, Py_ssize_t count)
 
-   Return the result of repeating sequence object *o* *count* times, or *NULL* on
+   Return the result of repeating sequence object *o* *count* times, or ``NULL`` on
    failure.  The operation is done *in-place* when *o* supports it.  This is the
    equivalent of the Python expression ``o *= count``.
 
 
 .. c:function:: PyObject* PySequence_GetItem(PyObject *o, Py_ssize_t i)
 
-   Return the *i*\ th element of *o*, or *NULL* on failure. This is the equivalent of
+   Return the *i*\ th element of *o*, or ``NULL`` on failure. This is the equivalent of
    the Python expression ``o[i]``.
 
 
 .. c:function:: PyObject* PySequence_GetSlice(PyObject *o, Py_ssize_t i1, Py_ssize_t i2)
 
-   Return the slice of sequence object *o* between *i1* and *i2*, or *NULL* on
+   Return the slice of sequence object *o* between *i1* and *i2*, or ``NULL`` on
    failure. This is the equivalent of the Python expression ``o[i1:i2]``.
 
 
@@ -69,7 +69,7 @@ Sequence Protocol
    is the equivalent of the Python statement ``o[i] = v``.  This function *does
    not* steal a reference to *v*.
 
-   If *v* is *NULL*, the element is deleted, however this feature is
+   If *v* is ``NULL``, the element is deleted, however this feature is
    deprecated in favour of using :c:func:`PySequence_DelItem`.
 
 
@@ -114,7 +114,7 @@ Sequence Protocol
 .. c:function:: PyObject* PySequence_List(PyObject *o)
 
    Return a list object with the same contents as the sequence or iterable *o*,
-   or *NULL* on failure.  The returned list is guaranteed to be new.  This is
+   or ``NULL`` on failure.  The returned list is guaranteed to be new.  This is
    equivalent to the Python expression ``list(o)``.
 
 
@@ -123,7 +123,7 @@ Sequence Protocol
    .. index:: builtin: tuple
 
    Return a tuple object with the same contents as the sequence or iterable *o*,
-   or *NULL* on failure.  If *o* is a tuple, a new reference will be returned,
+   or ``NULL`` on failure.  If *o* is a tuple, a new reference will be returned,
    otherwise a tuple will be constructed with the appropriate contents.  This is
    equivalent to the Python expression ``tuple(o)``.
 
@@ -133,7 +133,7 @@ Sequence Protocol
    Return the sequence or iterable *o* as an object usable by the other
    ``PySequence_Fast*`` family of functions. If the object is not a sequence or
    iterable, raises :exc:`TypeError` with *m* as the message text. Returns
-   *NULL* on failure.
+   ``NULL`` on failure.
 
    The ``PySequence_Fast*`` functions are thus named because they assume
    *o* is a :c:type:`PyTupleObject` or a :c:type:`PyListObject` and access
@@ -146,7 +146,7 @@ Sequence Protocol
 .. c:function:: Py_ssize_t PySequence_Fast_GET_SIZE(PyObject *o)
 
    Returns the length of *o*, assuming that *o* was returned by
-   :c:func:`PySequence_Fast` and that *o* is not *NULL*.  The size can also be
+   :c:func:`PySequence_Fast` and that *o* is not ``NULL``.  The size can also be
    gotten by calling :c:func:`PySequence_Size` on *o*, but
    :c:func:`PySequence_Fast_GET_SIZE` is faster because it can assume *o* is a
    list or tuple.
@@ -155,13 +155,13 @@ Sequence Protocol
 .. c:function:: PyObject* PySequence_Fast_GET_ITEM(PyObject *o, Py_ssize_t i)
 
    Return the *i*\ th element of *o*, assuming that *o* was returned by
-   :c:func:`PySequence_Fast`, *o* is not *NULL*, and that *i* is within bounds.
+   :c:func:`PySequence_Fast`, *o* is not ``NULL``, and that *i* is within bounds.
 
 
 .. c:function:: PyObject** PySequence_Fast_ITEMS(PyObject *o)
 
    Return the underlying array of PyObject pointers.  Assumes that *o* was returned
-   by :c:func:`PySequence_Fast` and *o* is not *NULL*.
+   by :c:func:`PySequence_Fast` and *o* is not ``NULL``.
 
    Note, if a list gets resized, the reallocation may relocate the items array.
    So, only use the underlying array pointer in contexts where the sequence
@@ -170,7 +170,7 @@ Sequence Protocol
 
 .. c:function:: PyObject* PySequence_ITEM(PyObject *o, Py_ssize_t i)
 
-   Return the *i*\ th element of *o* or *NULL* on failure. Faster form of
+   Return the *i*\ th element of *o* or ``NULL`` on failure. Faster form of
    :c:func:`PySequence_GetItem` but without checking that
    :c:func:`PySequence_Check` on *o* is true and without adjustment for negative
    indices.
diff --git a/Doc/c-api/set.rst b/Doc/c-api/set.rst
index 074fcb877e1da..54819e8fd6cbd 100644
--- a/Doc/c-api/set.rst
+++ b/Doc/c-api/set.rst
@@ -80,8 +80,8 @@ the constructor functions work with any iterable Python object.
 .. c:function:: PyObject* PySet_New(PyObject *iterable)
 
    Return a new :class:`set` containing objects returned by the *iterable*.  The
-   *iterable* may be *NULL* to create a new empty set.  Return the new set on
-   success or *NULL* on failure.  Raise :exc:`TypeError` if *iterable* is not
+   *iterable* may be ``NULL`` to create a new empty set.  Return the new set on
+   success or ``NULL`` on failure.  Raise :exc:`TypeError` if *iterable* is not
    actually iterable.  The constructor is also useful for copying a set
    (``c=set(s)``).
 
@@ -89,8 +89,8 @@ the constructor functions work with any iterable Python object.
 .. c:function:: PyObject* PyFrozenSet_New(PyObject *iterable)
 
    Return a new :class:`frozenset` containing objects returned by the *iterable*.
-   The *iterable* may be *NULL* to create a new empty frozenset.  Return the new
-   set on success or *NULL* on failure.  Raise :exc:`TypeError` if *iterable* is
+   The *iterable* may be ``NULL`` to create a new empty frozenset.  Return the new
+   set on success or ``NULL`` on failure.  Raise :exc:`TypeError` if *iterable* is
    not actually iterable.
 
 
@@ -149,7 +149,7 @@ subtypes but not for instances of :class:`frozenset` or its subtypes.
 .. c:function:: PyObject* PySet_Pop(PyObject *set)
 
    Return a new reference to an arbitrary object in the *set*, and removes the
-   object from the *set*.  Return *NULL* on failure.  Raise :exc:`KeyError` if the
+   object from the *set*.  Return ``NULL`` on failure.  Raise :exc:`KeyError` if the
    set is empty. Raise a :exc:`SystemError` if *set* is not an instance of
    :class:`set` or its subtype.
 
diff --git a/Doc/c-api/slice.rst b/Doc/c-api/slice.rst
index d924308890b8d..48a58c6c6f7e3 100644
--- a/Doc/c-api/slice.rst
+++ b/Doc/c-api/slice.rst
@@ -14,15 +14,15 @@ Slice Objects
 
 .. c:function:: int PySlice_Check(PyObject *ob)
 
-   Return true if *ob* is a slice object; *ob* must not be *NULL*.
+   Return true if *ob* is a slice object; *ob* must not be ``NULL``.
 
 
 .. c:function:: PyObject* PySlice_New(PyObject *start, PyObject *stop, PyObject *step)
 
    Return a new slice object with the given values.  The *start*, *stop*, and
    *step* parameters are used as the values of the slice object attributes of
-   the same names.  Any of the values may be *NULL*, in which case the
-   ``None`` will be used for the corresponding attribute.  Return *NULL* if
+   the same names.  Any of the values may be ``NULL``, in which case the
+   ``None`` will be used for the corresponding attribute.  Return ``NULL`` if
    the new object could not be allocated.
 
 
diff --git a/Doc/c-api/structures.rst b/Doc/c-api/structures.rst
index b0a193876c537..10352778f33b7 100644
--- a/Doc/c-api/structures.rst
+++ b/Doc/c-api/structures.rst
@@ -113,8 +113,8 @@ Implementing functions and methods
 
    Type of the functions used to implement most Python callables in C.
    Functions of this type take two :c:type:`PyObject\*` parameters and return
-   one such value.  If the return value is *NULL*, an exception shall have
-   been set.  If not *NULL*, the return value is interpreted as the return
+   one such value.  If the return value is ``NULL``, an exception shall have
+   been set.  If not ``NULL``, the return value is interpreted as the return
    value of the function as exposed in Python.  The function must return a new
    reference.
 
@@ -186,7 +186,7 @@ also keyword arguments.  So there are a total of 6 calling conventions:
 
    Methods with these flags must be of type :c:type:`PyCFunctionWithKeywords`.
    The function expects three parameters: *self*, *args*, *kwargs* where
-   *kwargs* is a dictionary of all the keyword arguments or possibly *NULL*
+   *kwargs* is a dictionary of all the keyword arguments or possibly ``NULL``
    if there are no keyword arguments.  The parameters are typically processed
    using :c:func:`PyArg_ParseTupleAndKeywords`.
 
@@ -212,7 +212,7 @@ also keyword arguments.  So there are a total of 6 calling conventions:
    there is an additional fourth :c:type:`PyObject\*` parameter
    which is a tuple representing the names of the keyword arguments
    (which are guaranteed to be strings)
-   or possibly *NULL* if there are no keywords.  The values of the keyword
+   or possibly ``NULL`` if there are no keywords.  The values of the keyword
    arguments are stored in the *args* array, after the positional arguments.
 
    This is not part of the :ref:`limited API <stable>`.
@@ -226,7 +226,7 @@ also keyword arguments.  So there are a total of 6 calling conventions:
    they are listed with the :const:`METH_NOARGS` flag.  They need to be of type
    :c:type:`PyCFunction`.  The first parameter is typically named *self* and will
    hold a reference to the module or object instance.  In all cases the second
-   parameter will be *NULL*.
+   parameter will be ``NULL``.
 
 
 .. data:: METH_O
@@ -257,7 +257,7 @@ method.
 
    .. index:: builtin: staticmethod
 
-   The method will be passed *NULL* as the first parameter rather than an
+   The method will be passed ``NULL`` as the first parameter rather than an
    instance of the type.  This is used to create *static methods*, similar to
    what is created when using the :func:`staticmethod` built-in function.
 
@@ -334,7 +334,7 @@ Accessing attributes of extension types
    =============== ==================
 
    :c:macro:`T_OBJECT` and :c:macro:`T_OBJECT_EX` differ in that
-   :c:macro:`T_OBJECT` returns ``None`` if the member is *NULL* and
+   :c:macro:`T_OBJECT` returns ``None`` if the member is ``NULL`` and
    :c:macro:`T_OBJECT_EX` raises an :exc:`AttributeError`.  Try to use
    :c:macro:`T_OBJECT_EX` over :c:macro:`T_OBJECT` because :c:macro:`T_OBJECT_EX`
    handles use of the :keyword:`del` statement on that attribute more correctly
@@ -344,7 +344,7 @@ Accessing attributes of extension types
    read-only access.  Using :c:macro:`T_STRING` for :attr:`type` implies
    :c:macro:`READONLY`.  :c:macro:`T_STRING` data is interpreted as UTF-8.
    Only :c:macro:`T_OBJECT` and :c:macro:`T_OBJECT_EX`
-   members can be deleted.  (They are set to *NULL*).
+   members can be deleted.  (They are set to ``NULL``).
 
    .. _pymemberdef-offsets:
 
@@ -388,7 +388,7 @@ Accessing attributes of extension types
 
       typedef PyObject *(*getter)(PyObject *, void *);
 
-   It should return a new reference on success or *NULL* with a set exception
+   It should return a new reference on success or ``NULL`` with a set exception
    on failure.
 
    ``set`` functions take two :c:type:`PyObject\*` parameters (the instance and
@@ -396,5 +396,5 @@ Accessing attributes of extension types
 
       typedef int (*setter)(PyObject *, PyObject *, void *);
 
-   In case the attribute should be deleted the second parameter is *NULL*.
+   In case the attribute should be deleted the second parameter is ``NULL``.
    Should return ``0`` on success or ``-1`` with a set exception on failure.
diff --git a/Doc/c-api/sys.rst b/Doc/c-api/sys.rst
index 87d2bd13d31b8..212f4d1066c9c 100644
--- a/Doc/c-api/sys.rst
+++ b/Doc/c-api/sys.rst
@@ -22,7 +22,7 @@ Operating System Utilities
    Return true (nonzero) if the standard I/O file *fp* with name *filename* is
    deemed interactive.  This is the case for files for which ``isatty(fileno(fp))``
    is true.  If the global flag :c:data:`Py_InteractiveFlag` is true, this function
-   also returns true if the *filename* pointer is *NULL* or if the name is equal to
+   also returns true if the *filename* pointer is ``NULL`` or if the name is equal to
    one of the strings ``'<stdin>'`` or ``'???'``.
 
 
@@ -201,12 +201,12 @@ accessible to C code.  They all work with the current interpreter thread's
 
 .. c:function:: PyObject *PySys_GetObject(const char *name)
 
-   Return the object *name* from the :mod:`sys` module or *NULL* if it does
+   Return the object *name* from the :mod:`sys` module or ``NULL`` if it does
    not exist, without setting an exception.
 
 .. c:function:: int PySys_SetObject(const char *name, PyObject *v)
 
-   Set *name* in the :mod:`sys` module to *v* unless *v* is *NULL*, in which
+   Set *name* in the :mod:`sys` module to *v* unless *v* is ``NULL``, in which
    case *name* is deleted from the sys module. Returns ``0`` on success, ``-1``
    on error.
 
@@ -283,7 +283,7 @@ accessible to C code.  They all work with the current interpreter thread's
 .. c:function:: PyObject *PySys_GetXOptions()
 
    Return the current dictionary of :option:`-X` options, similarly to
-   :data:`sys._xoptions`.  On error, *NULL* is returned and an exception is
+   :data:`sys._xoptions`.  On error, ``NULL`` is returned and an exception is
    set.
 
    .. versionadded:: 3.2
diff --git a/Doc/c-api/tuple.rst b/Doc/c-api/tuple.rst
index a0d53e2030e63..d74d8626d1e92 100644
--- a/Doc/c-api/tuple.rst
+++ b/Doc/c-api/tuple.rst
@@ -33,12 +33,12 @@ Tuple Objects
 
 .. c:function:: PyObject* PyTuple_New(Py_ssize_t len)
 
-   Return a new tuple object of size *len*, or *NULL* on failure.
+   Return a new tuple object of size *len*, or ``NULL`` on failure.
 
 
 .. c:function:: PyObject* PyTuple_Pack(Py_ssize_t n, ...)
 
-   Return a new tuple object of size *n*, or *NULL* on failure. The tuple values
+   Return a new tuple object of size *n*, or ``NULL`` on failure. The tuple values
    are initialized to the subsequent *n* C arguments pointing to Python objects.
    ``PyTuple_Pack(2, a, b)`` is equivalent to ``Py_BuildValue("(OO)", a, b)``.
 
@@ -50,14 +50,14 @@ Tuple Objects
 
 .. c:function:: Py_ssize_t PyTuple_GET_SIZE(PyObject *p)
 
-   Return the size of the tuple *p*, which must be non-*NULL* and point to a tuple;
+   Return the size of the tuple *p*, which must be non-``NULL`` and point to a tuple;
    no error checking is performed.
 
 
 .. c:function:: PyObject* PyTuple_GetItem(PyObject *p, Py_ssize_t pos)
 
    Return the object at position *pos* in the tuple pointed to by *p*.  If *pos* is
-   out of bounds, return *NULL* and set an :exc:`IndexError` exception.
+   out of bounds, return ``NULL`` and set an :exc:`IndexError` exception.
 
 
 .. c:function:: PyObject* PyTuple_GET_ITEM(PyObject *p, Py_ssize_t pos)
@@ -68,7 +68,7 @@ Tuple Objects
 .. c:function:: PyObject* PyTuple_GetSlice(PyObject *p, Py_ssize_t low, Py_ssize_t high)
 
    Return the slice of the tuple pointed to by *p* between *low* and *high*,
-   or *NULL* on failure.  This is the equivalent of the Python expression
+   or ``NULL`` on failure.  This is the equivalent of the Python expression
    ``p[low:high]``.  Indexing from the end of the list is not supported.
 
 
@@ -107,7 +107,7 @@ Tuple Objects
    only more efficiently.  Returns ``0`` on success. Client code should never
    assume that the resulting value of ``*p`` will be the same as before calling
    this function. If the object referenced by ``*p`` is replaced, the original
-   ``*p`` is destroyed.  On failure, returns ``-1`` and sets ``*p`` to *NULL*, and
+   ``*p`` is destroyed.  On failure, returns ``-1`` and sets ``*p`` to ``NULL``, and
    raises :exc:`MemoryError` or :exc:`SystemError`.
 
 
@@ -147,20 +147,20 @@ type.
 
    Contains the meta information of a struct sequence type to create.
 
-   +-------------------+------------------------------+------------------------------------+
-   | Field             | C Type                       | Meaning                            |
-   +===================+==============================+====================================+
-   | ``name``          | ``const char *``             | name of the struct sequence type   |
-   +-------------------+------------------------------+------------------------------------+
-   | ``doc``           | ``const char *``             | pointer to docstring for the type  |
-   |                   |                              | or NULL to omit                    |
-   +-------------------+------------------------------+------------------------------------+
-   | ``fields``        | ``PyStructSequence_Field *`` | pointer to *NULL*-terminated array |
-   |                   |                              | with field names of the new type   |
-   +-------------------+------------------------------+------------------------------------+
-   | ``n_in_sequence`` | ``int``                      | number of fields visible to the    |
-   |                   |                              | Python side (if used as tuple)     |
-   +-------------------+------------------------------+------------------------------------+
+   +-------------------+------------------------------+--------------------------------------+
+   | Field             | C Type                       | Meaning                              |
+   +===================+==============================+======================================+
+   | ``name``          | ``const char *``             | name of the struct sequence type     |
+   +-------------------+------------------------------+--------------------------------------+
+   | ``doc``           | ``const char *``             | pointer to docstring for the type    |
+   |                   |                              | or NULL to omit                      |
+   +-------------------+------------------------------+--------------------------------------+
+   | ``fields``        | ``PyStructSequence_Field *`` | pointer to ``NULL``-terminated array |
+   |                   |                              | with field names of the new type     |
+   +-------------------+------------------------------+--------------------------------------+
+   | ``n_in_sequence`` | ``int``                      | number of fields visible to the      |
+   |                   |                              | Python side (if used as tuple)       |
+   +-------------------+------------------------------+--------------------------------------+
 
 
 .. c:type:: PyStructSequence_Field
@@ -170,16 +170,16 @@ type.
    :attr:`fields` array of the :c:type:`PyStructSequence_Desc` determines which
    field of the struct sequence is described.
 
-   +-----------+------------------+--------------------------------------+
-   | Field     | C Type           | Meaning                              |
-   +===========+==================+======================================+
-   | ``name``  | ``const char *`` | name for the field or *NULL* to end  |
-   |           |                  | the list of named fields, set to     |
-   |           |                  | PyStructSequence_UnnamedField to     |
-   |           |                  | leave unnamed                        |
-   +-----------+------------------+--------------------------------------+
-   | ``doc``   | ``const char *`` | field docstring or *NULL* to omit    |
-   +-----------+------------------+--------------------------------------+
+   +-----------+------------------+----------------------------------------+
+   | Field     | C Type           | Meaning                                |
+   +===========+==================+========================================+
+   | ``name``  | ``const char *`` | name for the field or ``NULL`` to end  |
+   |           |                  | the list of named fields, set to       |
+   |           |                  | PyStructSequence_UnnamedField to       |
+   |           |                  | leave unnamed                          |
+   +-----------+------------------+----------------------------------------+
+   | ``doc``   | ``const char *`` | field docstring or ``NULL`` to omit    |
+   +-----------+------------------+----------------------------------------+
 
 
 .. c:var:: char* PyStructSequence_UnnamedField
diff --git a/Doc/c-api/type.rst b/Doc/c-api/type.rst
index 73311ec083a85..34b52f27ecb83 100644
--- a/Doc/c-api/type.rst
+++ b/Doc/c-api/type.rst
@@ -81,7 +81,7 @@ Type Objects
 
    Generic handler for the :c:member:`~PyTypeObject.tp_alloc` slot of a type object.  Use
    Python's default memory allocation mechanism to allocate a new instance and
-   initialize all its contents to *NULL*.
+   initialize all its contents to ``NULL``.
 
 .. c:function:: PyObject* PyType_GenericNew(PyTypeObject *type, PyObject *args, PyObject *kwds)
 
@@ -98,7 +98,7 @@ Type Objects
 .. c:function:: void* PyType_GetSlot(PyTypeObject *type, int slot)
 
    Return the function pointer stored in the given slot. If the
-   result is *NULL*, this indicates that either the slot is *NULL*,
+   result is ``NULL``, this indicates that either the slot is ``NULL``,
    or that the function was called with invalid parameters.
    Callers will typically cast the result pointer into the appropriate
    function type.
@@ -124,8 +124,8 @@ The following functions and structs are used to create
    If *bases* is a tuple, the created heap type contains all types contained
    in it as base types.
 
-   If *bases* is *NULL*, the *Py_tp_base* slot is used instead.
-   If that also is *NULL*, the new type derives from :class:`object`.
+   If *bases* is ``NULL``, the *Py_tp_base* slot is used instead.
+   If that also is ``NULL``, the new type derives from :class:`object`.
 
    This function calls :c:func:`PyType_Ready` on the new type.
 
@@ -205,4 +205,4 @@ The following functions and structs are used to create
       The desired value of the slot. In most cases, this is a pointer
       to a function.
 
-      May not be *NULL*.
+      May not be ``NULL``.
diff --git a/Doc/c-api/typeobj.rst b/Doc/c-api/typeobj.rst
index bb76750074309..181157e08ec43 100644
--- a/Doc/c-api/typeobj.rst
+++ b/Doc/c-api/typeobj.rst
@@ -159,20 +159,20 @@ fields exist as well:
    A slot name in parentheses indicates it is (effectively) deprecated.
    Names in angle brackets should be treated as read-only.
    Names in square brackets are for internal use only.
-   "<R>" (as a prefix) means the field is required (must be non-*NULL*).
+   "<R>" (as a prefix) means the field is required (must be non-``NULL``).
 .. [#cols] Columns:
 
    **"O"**:  set on :c:type:`PyBaseObject_Type`
 
    **"T"**:  set on :c:type:`PyType_Type`
 
-   **"D"**:  default (if slot is set to *NULL*)
+   **"D"**:  default (if slot is set to ``NULL``)
 
    .. code-block:: none
 
-      X - *PyType_Ready* sets this value if it is *NULL*
-      ~ - *PyType_Ready* always sets this value (it should be *NULL*)
-      ? - *PyType_Ready* may set this value depending on other slots
+      X - PyType_Ready sets this value if it is NULL
+      ~ - PyType_Ready always sets this value (it should be NULL)
+      ? - PyType_Ready may set this value depending on other slots
 
       Also see the inheritance column ("I").
 
@@ -180,7 +180,7 @@ fields exist as well:
 
    .. code-block:: none
 
-      X - type slot is inherited via *PyType_Ready* if defined with a *NULL* value
+      X - type slot is inherited via PyType_Ready if defined with a NULL value
       % - the slots of the sub-struct are inherited individually
       G - inherited, but only in combination with other slots; see the slot's description
       ? - it's complicated; see the slot's description
@@ -491,8 +491,8 @@ type objects) *must* have the :attr:`ob_size` field.
              PyObject* PyObject._ob_prev
 
    These fields are only present when the macro ``Py_TRACE_REFS`` is defined.
-   Their initialization to *NULL* is taken care of by the ``PyObject_HEAD_INIT``
-   macro.  For statically allocated objects, these fields always remain *NULL*.
+   Their initialization to ``NULL`` is taken care of by the ``PyObject_HEAD_INIT``
+   macro.  For statically allocated objects, these fields always remain ``NULL``.
    For dynamically allocated objects, these two fields are used to link the object
    into a doubly-linked list of *all* live objects on the heap.  This could be used
    for various debugging purposes; currently the only use is to print the objects
@@ -523,7 +523,7 @@ type objects) *must* have the :attr:`ob_size` field.
    argument to the ``PyObject_HEAD_INIT`` macro, and its value should normally be
    ``&PyType_Type``.  However, for dynamically loadable extension modules that must
    be usable on Windows (at least), the compiler complains that this is not a valid
-   initializer.  Therefore, the convention is to pass *NULL* to the
+   initializer.  Therefore, the convention is to pass ``NULL`` to the
    ``PyObject_HEAD_INIT`` macro and to initialize this field explicitly at the
    start of the module's initialization function, before doing anything else.  This
    is typically done like this::
@@ -531,7 +531,7 @@ type objects) *must* have the :attr:`ob_size` field.
       Foo_Type.ob_type = &PyType_Type;
 
    This should be done before any instances of the type are created.
-   :c:func:`PyType_Ready` checks if :attr:`ob_type` is *NULL*, and if so,
+   :c:func:`PyType_Ready` checks if :attr:`ob_type` is ``NULL``, and if so,
    initializes it to the :attr:`ob_type` field of the base class.
    :c:func:`PyType_Ready` will not change this field if it is non-zero.
 
@@ -557,7 +557,7 @@ PyTypeObject Slots
 ------------------
 
 Each slot has a section describing inheritance.  If :c:func:`PyType_Ready`
-may set a value when the field is set to *NULL* then there will also be
+may set a value when the field is set to ``NULL`` then there will also be
 a "Default" section.  (Note that many fields set on :c:type:`PyBaseObject_Type`
 and :c:type:`PyType_Type` effectively act as defaults.)
 
@@ -586,7 +586,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
    type will be impossible to pickle.  Additionally, it will not be listed in
    module documentations created with pydoc.
 
-   This field must not be *NULL*.  It is the only required field
+   This field must not be ``NULL``.  It is the only required field
    in :c:func:`PyTypeObject` (other than potentially
    :c:member:`~PyTypeObject.tp_itemsize`).
 
@@ -631,7 +631,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
    :c:member:`~PyTypeObject.tp_basicsize` is a multiple of ``sizeof(double)`` (assuming this is the
    alignment requirement for ``double``).
 
-   For any type with variable-length instances, this field must not be *NULL*.
+   For any type with variable-length instances, this field must not be ``NULL``.
 
    **Inheritance:**
 
@@ -760,7 +760,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
 
    This field is inherited by subtypes together with :c:member:`~PyTypeObject.tp_getattro`: a subtype
    inherits both :c:member:`~PyTypeObject.tp_getattr` and :c:member:`~PyTypeObject.tp_getattro` from its base type when
-   the subtype's :c:member:`~PyTypeObject.tp_getattr` and :c:member:`~PyTypeObject.tp_getattro` are both *NULL*.
+   the subtype's :c:member:`~PyTypeObject.tp_getattr` and :c:member:`~PyTypeObject.tp_getattro` are both ``NULL``.
 
 
 .. c:member:: setattrfunc PyTypeObject.tp_setattr
@@ -777,7 +777,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
 
    This field is inherited by subtypes together with :c:member:`~PyTypeObject.tp_setattro`: a subtype
    inherits both :c:member:`~PyTypeObject.tp_setattr` and :c:member:`~PyTypeObject.tp_setattro` from its base type when
-   the subtype's :c:member:`~PyTypeObject.tp_setattr` and :c:member:`~PyTypeObject.tp_setattro` are both *NULL*.
+   the subtype's :c:member:`~PyTypeObject.tp_setattr` and :c:member:`~PyTypeObject.tp_setattro` are both ``NULL``.
 
 
 .. c:member:: PyAsyncMethods* PyTypeObject.tp_as_async
@@ -894,13 +894,13 @@ and :c:type:`PyType_Type` effectively act as defaults.)
    This field is inherited by subtypes together with
    :c:member:`~PyTypeObject.tp_richcompare`: a subtype inherits both of
    :c:member:`~PyTypeObject.tp_richcompare` and :c:member:`~PyTypeObject.tp_hash`, when the subtype's
-   :c:member:`~PyTypeObject.tp_richcompare` and :c:member:`~PyTypeObject.tp_hash` are both *NULL*.
+   :c:member:`~PyTypeObject.tp_richcompare` and :c:member:`~PyTypeObject.tp_hash` are both ``NULL``.
 
 
 .. c:member:: ternaryfunc PyTypeObject.tp_call
 
    An optional pointer to a function that implements calling the object.  This
-   should be *NULL* if the object is not callable.  The signature is the same as
+   should be ``NULL`` if the object is not callable.  The signature is the same as
    for :c:func:`PyObject_Call`::
 
       PyObject *tp_call(PyObject *self, PyObject *args, PyObject *kwargs);
@@ -952,7 +952,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
 
    This field is inherited by subtypes together with :c:member:`~PyTypeObject.tp_getattr`: a subtype
    inherits both :c:member:`~PyTypeObject.tp_getattr` and :c:member:`~PyTypeObject.tp_getattro` from its base type when
-   the subtype's :c:member:`~PyTypeObject.tp_getattr` and :c:member:`~PyTypeObject.tp_getattro` are both *NULL*.
+   the subtype's :c:member:`~PyTypeObject.tp_getattr` and :c:member:`~PyTypeObject.tp_getattro` are both ``NULL``.
 
    **Default:**
 
@@ -967,7 +967,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
 
       PyObject *tp_setattro(PyObject *self, PyObject *attr, PyObject *value);
 
-   In addition, setting *value* to *NULL* to delete an attribute must be
+   In addition, setting *value* to ``NULL`` to delete an attribute must be
    supported.  It is usually convenient to set this field to
    :c:func:`PyObject_GenericSetAttr`, which implements the normal
    way of setting object attributes.
@@ -978,7 +978,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
 
    This field is inherited by subtypes together with :c:member:`~PyTypeObject.tp_setattr`: a subtype
    inherits both :c:member:`~PyTypeObject.tp_setattr` and :c:member:`~PyTypeObject.tp_setattro` from its base type when
-   the subtype's :c:member:`~PyTypeObject.tp_setattr` and :c:member:`~PyTypeObject.tp_setattro` are both *NULL*.
+   the subtype's :c:member:`~PyTypeObject.tp_setattr` and :c:member:`~PyTypeObject.tp_setattro` are both ``NULL``.
 
    **Default:**
 
@@ -1005,7 +1005,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
    :c:member:`~PyTypeObject.tp_as_number`, :c:member:`~PyTypeObject.tp_as_sequence`, :c:member:`~PyTypeObject.tp_as_mapping`, and
    :c:member:`~PyTypeObject.tp_as_buffer`) that were historically not always present are valid; if
    such a flag bit is clear, the type fields it guards must not be accessed and
-   must be considered to have a zero or *NULL* value instead.
+   must be considered to have a zero or ``NULL`` value instead.
 
    **Inheritance:**
 
@@ -1018,7 +1018,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
    the :c:member:`~PyTypeObject.tp_traverse` and :c:member:`~PyTypeObject.tp_clear` fields, i.e. if the
    :const:`Py_TPFLAGS_HAVE_GC` flag bit is clear in the subtype and the
    :c:member:`~PyTypeObject.tp_traverse` and :c:member:`~PyTypeObject.tp_clear` fields in the subtype exist and have
-   *NULL* values.
+   ``NULL`` values.
 
    .. XXX are most flag bits *really* inherited individually?
 
@@ -1097,7 +1097,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
       together with the :attr:`tp_traverse` and :attr:`tp_clear`
       fields, i.e.  if the :const:`Py_TPFLAGS_HAVE_GC` flag bit is
       clear in the subtype and the :attr:`tp_traverse` and
-      :attr:`tp_clear` fields in the subtype exist and have *NULL*
+      :attr:`tp_clear` fields in the subtype exist and have ``NULL``
       values.
 
 
@@ -1232,7 +1232,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
 
    Note that :c:func:`Py_VISIT` is called only on those members that can participate
    in reference cycles.  Although there is also a ``self->key`` member, it can only
-   be *NULL* or a Python string and therefore cannot be part of a reference cycle.
+   be ``NULL`` or a Python string and therefore cannot be part of a reference cycle.
 
    On the other hand, even if you know a member can never be part of a cycle, as a
    debugging aid you may want to visit it anyway just so the :mod:`gc` module's
@@ -1271,7 +1271,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
 
    Implementations of :c:member:`~PyTypeObject.tp_clear` should drop the instance's references to
    those of its members that may be Python objects, and set its pointers to those
-   members to *NULL*, as in the following example::
+   members to ``NULL``, as in the following example::
 
       static int
       local_clear(localobject *self)
@@ -1285,12 +1285,12 @@ and :c:type:`PyType_Type` effectively act as defaults.)
 
    The :c:func:`Py_CLEAR` macro should be used, because clearing references is
    delicate:  the reference to the contained object must not be decremented until
-   after the pointer to the contained object is set to *NULL*.  This is because
+   after the pointer to the contained object is set to ``NULL``.  This is because
    decrementing the reference count may cause the contained object to become trash,
    triggering a chain of reclamation activity that may include invoking arbitrary
    Python code (due to finalizers, or weakref callbacks, associated with the
    contained object). If it's possible for such code to reference *self* again,
-   it's important that the pointer to the contained object be *NULL* at that time,
+   it's important that the pointer to the contained object be ``NULL`` at that time,
    so that *self* knows the contained object can no longer be used.  The
    :c:func:`Py_CLEAR` macro performs the operations in a safe order.
 
@@ -1324,7 +1324,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
 
    The function should return the result of the comparison (usually ``Py_True``
    or ``Py_False``).  If the comparison is undefined, it must return
-   ``Py_NotImplemented``, if another error occurred it must return *NULL* and
+   ``Py_NotImplemented``, if another error occurred it must return ``NULL`` and
    set an exception condition.
 
    The following constants are defined to be used as the third argument for
@@ -1358,7 +1358,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
 
       The return value's reference count is properly incremented.
 
-      On error, sets an exception and returns *NULL* from the function.
+      On error, sets an exception and returns ``NULL`` from the function.
 
       .. versionadded:: 3.7
 
@@ -1369,7 +1369,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
    This field is inherited by subtypes together with :c:member:`~PyTypeObject.tp_hash`:
    a subtype inherits :c:member:`~PyTypeObject.tp_richcompare` and :c:member:`~PyTypeObject.tp_hash` when
    the subtype's :c:member:`~PyTypeObject.tp_richcompare` and :c:member:`~PyTypeObject.tp_hash` are both
-   *NULL*.
+   ``NULL``.
 
    **Default:**
 
@@ -1387,7 +1387,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
    reference list head (ignoring the GC header, if present); this offset is used by
    :c:func:`PyObject_ClearWeakRefs` and the :c:func:`PyWeakref_\*` functions.  The
    instance structure needs to include a field of type :c:type:`PyObject\*` which is
-   initialized to *NULL*.
+   initialized to ``NULL``.
 
    Do not confuse this field with :c:member:`~PyTypeObject.tp_weaklist`; that is the list head for
    weak references to the type object itself.
@@ -1436,9 +1436,9 @@ and :c:type:`PyType_Type` effectively act as defaults.)
 
       PyObject *tp_iternext(PyObject *self);
 
-   When the iterator is exhausted, it must return *NULL*; a :exc:`StopIteration`
+   When the iterator is exhausted, it must return ``NULL``; a :exc:`StopIteration`
    exception may or may not be set.  When another error occurs, it must return
-   *NULL* too.  Its presence signals that the instances of this type are
+   ``NULL`` too.  Its presence signals that the instances of this type are
    iterators.
 
    Iterator types should also define the :c:member:`~PyTypeObject.tp_iter` function, and that
@@ -1454,7 +1454,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
 
 .. c:member:: struct PyMethodDef* PyTypeObject.tp_methods
 
-   An optional pointer to a static *NULL*-terminated array of :c:type:`PyMethodDef`
+   An optional pointer to a static ``NULL``-terminated array of :c:type:`PyMethodDef`
    structures, declaring regular methods of this type.
 
    For each entry in the array, an entry is added to the type's dictionary (see
@@ -1468,7 +1468,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
 
 .. c:member:: struct PyMemberDef* PyTypeObject.tp_members
 
-   An optional pointer to a static *NULL*-terminated array of :c:type:`PyMemberDef`
+   An optional pointer to a static ``NULL``-terminated array of :c:type:`PyMemberDef`
    structures, declaring regular data members (fields or slots) of instances of
    this type.
 
@@ -1483,7 +1483,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
 
 .. c:member:: struct PyGetSetDef* PyTypeObject.tp_getset
 
-   An optional pointer to a static *NULL*-terminated array of :c:type:`PyGetSetDef`
+   An optional pointer to a static ``NULL``-terminated array of :c:type:`PyGetSetDef`
    structures, declaring computed attributes of instances of this type.
 
    For each entry in the array, an entry is added to the type's dictionary (see
@@ -1533,7 +1533,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
 
    The type's dictionary is stored here by :c:func:`PyType_Ready`.
 
-   This field should normally be initialized to *NULL* before PyType_Ready is
+   This field should normally be initialized to ``NULL`` before PyType_Ready is
    called; it may also be initialized to a dictionary containing initial attributes
    for the type.  Once :c:func:`PyType_Ready` has initialized the type, extra
    attributes for the type may be added to this dictionary only if they don't
@@ -1546,7 +1546,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
 
    **Default:**
 
-   If this field is *NULL*, :c:func:`PyType_Ready` will assign a new
+   If this field is ``NULL``, :c:func:`PyType_Ready` will assign a new
    dictionary to it.
 
    .. warning::
@@ -1579,7 +1579,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
 
       int tp_descr_set(PyObject *self, PyObject *obj, PyObject *value);
 
-   The *value* argument is set to *NULL* to delete the value.
+   The *value* argument is set to ``NULL`` to delete the value.
 
    .. XXX explain more?
 
@@ -1645,7 +1645,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
    **Default:**
 
    This slot has no default.  For static types, if the field is
-   *NULL* then no :attr:`__dict__` gets created for instances.
+   ``NULL`` then no :attr:`__dict__` gets created for instances.
 
 
 .. c:member:: initproc PyTypeObject.tp_init
@@ -1665,7 +1665,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
    arguments represent positional and keyword arguments of the call to
    :meth:`__init__`.
 
-   The :c:member:`~PyTypeObject.tp_init` function, if not *NULL*, is called when an instance is
+   The :c:member:`~PyTypeObject.tp_init` function, if not ``NULL``, is called when an instance is
    created normally by calling its type, after the type's :c:member:`~PyTypeObject.tp_new` function
    has returned an instance of the type.  If the :c:member:`~PyTypeObject.tp_new` function returns an
    instance of some other type that is not a subtype of the original type, no
@@ -1732,12 +1732,12 @@ and :c:type:`PyType_Type` effectively act as defaults.)
    **Inheritance:**
 
    This field is inherited by subtypes, except it is not inherited by static types
-   whose :c:member:`~PyTypeObject.tp_base` is *NULL* or ``&PyBaseObject_Type``.
+   whose :c:member:`~PyTypeObject.tp_base` is ``NULL`` or ``&PyBaseObject_Type``.
 
    **Default:**
 
    For static types this field has no default.  This means if the
-   slot is defined as *NULL*, the type cannot be called to create new
+   slot is defined as ``NULL``, the type cannot be called to create new
    instances; presumably there is some other way to create
    instances, like a factory function.
 
@@ -1788,7 +1788,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
 
    **Default:**
 
-   This slot has no default.  If this field is *NULL*,
+   This slot has no default.  If this field is ``NULL``,
    :const:`Py_TPFLAGS_HAVE_GC` is used as the functional equivalent.
 
 
@@ -1796,7 +1796,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
 
    Tuple of base types.
 
-   This is set for types created by a class statement.  It should be *NULL* for
+   This is set for types created by a class statement.  It should be ``NULL`` for
    statically defined types.
 
    **Inheritance:**
@@ -2034,12 +2034,12 @@ Number Object Structures
       and implement the necessary conversions (at least one of the operands is
       an instance of the defined type).  If the operation is not defined for the
       given operands, binary and ternary functions must return
-      ``Py_NotImplemented``, if another error occurred they must return *NULL*
+      ``Py_NotImplemented``, if another error occurred they must return ``NULL``
       and set an exception.
 
    .. note::
 
-      The :c:data:`nb_reserved` field should always be *NULL*.  It
+      The :c:data:`nb_reserved` field should always be ``NULL``.  It
       was previously called :c:data:`nb_long`, and was renamed in
       Python 3.0.1.
 
@@ -2098,14 +2098,14 @@ Mapping Object Structures
 
    This function is used by :c:func:`PyMapping_Size` and
    :c:func:`PyObject_Size`, and has the same signature.  This slot may be set to
-   *NULL* if the object has no defined length.
+   ``NULL`` if the object has no defined length.
 
 .. c:member:: binaryfunc PyMappingMethods.mp_subscript
 
    This function is used by :c:func:`PyObject_GetItem` and
    :c:func:`PySequence_GetSlice`, and has the same signature as
    :c:func:`!PyObject_GetItem`.  This slot must be filled for the
-   :c:func:`PyMapping_Check` function to return ``1``, it can be *NULL*
+   :c:func:`PyMapping_Check` function to return ``1``, it can be ``NULL``
    otherwise.
 
 .. c:member:: objobjargproc PyMappingMethods.mp_ass_subscript
@@ -2113,8 +2113,8 @@ Mapping Object Structures
    This function is used by :c:func:`PyObject_SetItem`,
    :c:func:`PyObject_DelItem`, :c:func:`PyObject_SetSlice` and
    :c:func:`PyObject_DelSlice`.  It has the same signature as
-   :c:func:`!PyObject_SetItem`, but *v* can also be set to *NULL* to delete
-   an item.  If this slot is *NULL*, the object does not support item
+   :c:func:`!PyObject_SetItem`, but *v* can also be set to ``NULL`` to delete
+   an item.  If this slot is ``NULL``, the object does not support item
    assignment and deletion.
 
 
@@ -2156,11 +2156,11 @@ Sequence Object Structures
    signature.  It is also used by :c:func:`PyObject_GetItem`, after trying
    the subscription via the :c:member:`~PyMappingMethods.mp_subscript` slot.
    This slot must be filled for the :c:func:`PySequence_Check`
-   function to return ``1``, it can be *NULL* otherwise.
+   function to return ``1``, it can be ``NULL`` otherwise.
 
    Negative indexes are handled as follows: if the :attr:`sq_length` slot is
    filled, it is called and the sequence length is used to compute a positive
-   index which is passed to :attr:`sq_item`.  If :attr:`sq_length` is *NULL*,
+   index which is passed to :attr:`sq_item`.  If :attr:`sq_length` is ``NULL``,
    the index is passed as is to the function.
 
 .. c:member:: ssizeobjargproc PySequenceMethods.sq_ass_item
@@ -2169,13 +2169,13 @@ Sequence Object Structures
    signature.  It is also used by :c:func:`PyObject_SetItem` and
    :c:func:`PyObject_DelItem`, after trying the item assignment and deletion
    via the :c:member:`~PyMappingMethods.mp_ass_subscript` slot.
-   This slot may be left to *NULL* if the object does not support
+   This slot may be left to ``NULL`` if the object does not support
    item assignment and deletion.
 
 .. c:member:: objobjproc PySequenceMethods.sq_contains
 
    This function may be used by :c:func:`PySequence_Contains` and has the same
-   signature.  This slot may be left to *NULL*, in this case
+   signature.  This slot may be left to ``NULL``, in this case
    :c:func:`!PySequence_Contains` simply traverses the sequence until it
    finds a match.
 
@@ -2183,7 +2183,7 @@ Sequence Object Structures
 
    This function is used by :c:func:`PySequence_InPlaceConcat` and has the same
    signature.  It should modify its first operand, and return it.  This slot
-   may be left to *NULL*, in this case :c:func:`!PySequence_InPlaceConcat`
+   may be left to ``NULL``, in this case :c:func:`!PySequence_InPlaceConcat`
    will fall back to :c:func:`PySequence_Concat`.  It is also used by the
    augmented assignment ``+=``, after trying numeric in-place addition
    via the :c:member:`~PyNumberMethods.nb_inplace_add` slot.
@@ -2192,7 +2192,7 @@ Sequence Object Structures
 
    This function is used by :c:func:`PySequence_InPlaceRepeat` and has the same
    signature.  It should modify its first operand, and return it.  This slot
-   may be left to *NULL*, in this case :c:func:`!PySequence_InPlaceRepeat`
+   may be left to ``NULL``, in this case :c:func:`!PySequence_InPlaceRepeat`
    will fall back to :c:func:`PySequence_Repeat`.  It is also used by the
    augmented assignment ``*=``, after trying numeric in-place multiplication
    via the :c:member:`~PyNumberMethods.nb_inplace_multiply` slot.
@@ -2224,7 +2224,7 @@ Buffer Object Structures
    steps:
 
    (1) Check if the request can be met. If not, raise :c:data:`PyExc_BufferError`,
-       set :c:data:`view->obj` to *NULL* and return ``-1``.
+       set :c:data:`view->obj` to ``NULL`` and return ``-1``.
 
    (2) Fill in the requested fields.
 
@@ -2270,7 +2270,7 @@ Buffer Object Structures
 
    Handle a request to release the resources of the buffer. If no resources
    need to be released, :c:member:`PyBufferProcs.bf_releasebuffer` may be
-   *NULL*. Otherwise, a standard implementation of this function will take
+   ``NULL``. Otherwise, a standard implementation of this function will take
    these optional steps:
 
    (1) Decrement an internal counter for the number of exports.
@@ -2324,7 +2324,7 @@ Async Object Structures
    The returned object must be an iterator, i.e. :c:func:`PyIter_Check` must
    return ``1`` for it.
 
-   This slot may be set to *NULL* if an object is not an :term:`awaitable`.
+   This slot may be set to ``NULL`` if an object is not an :term:`awaitable`.
 
 .. c:member:: unaryfunc PyAsyncMethods.am_aiter
 
@@ -2334,7 +2334,7 @@ Async Object Structures
 
    Must return an :term:`awaitable` object.  See :meth:`__anext__` for details.
 
-   This slot may be set to *NULL* if an object does not implement
+   This slot may be set to ``NULL`` if an object does not implement
    asynchronous iteration protocol.
 
 .. c:member:: unaryfunc PyAsyncMethods.am_anext
@@ -2344,7 +2344,7 @@ Async Object Structures
       PyObject *am_anext(PyObject *self);
 
    Must return an :term:`awaitable` object.  See :meth:`__anext__` for details.
-   This slot may be set to *NULL*.
+   This slot may be set to ``NULL``.
 
 
 .. _slot-typedefs:
@@ -2400,7 +2400,7 @@ Slot Type typedefs
 .. c:type:: int (*setattrfunc)(PyObject *self, char *attr, PyObject *value)
 
    Set the value of the named attribute for the object.
-   The value argument is set to *NULL* to delete the attribute.
+   The value argument is set to ``NULL`` to delete the attribute.
 
 .. c:type:: PyObject *(*getattrofunc)(PyObject *self, PyObject *attr)
 
@@ -2411,7 +2411,7 @@ Slot Type typedefs
 .. c:type:: int (*setattrofunc)(PyObject *self, PyObject *attr, PyObject *value)
 
    Set the value of the named attribute for the object.
-   The value argument is set to *NULL* to delete the attribute.
+   The value argument is set to ``NULL`` to delete the attribute.
 
    See :c:member:`~PyTypeObject.tp_setattro`.
 
diff --git a/Doc/c-api/unicode.rst b/Doc/c-api/unicode.rst
index a150a9c39d880..0359f5ec36bed 100644
--- a/Doc/c-api/unicode.rst
+++ b/Doc/c-api/unicode.rst
@@ -236,7 +236,7 @@ access internal read-only data of Unicode objects:
    .. versionchanged:: 3.3
       This macro is now inefficient -- because in many cases the
       :c:type:`Py_UNICODE` representation does not exist and needs to be created
-      -- and can fail (return *NULL* with an exception set).  Try to port the
+      -- and can fail (return ``NULL`` with an exception set).  Try to port the
       code to use the new :c:func:`PyUnicode_nBYTE_DATA` macros or use
       :c:func:`PyUnicode_WRITE` or :c:func:`PyUnicode_READ`.
 
@@ -413,11 +413,11 @@ APIs:
 
    Create a Unicode object from the char buffer *u*.  The bytes will be
    interpreted as being UTF-8 encoded.  The buffer is copied into the new
-   object. If the buffer is not *NULL*, the return value might be a shared
+   object. If the buffer is not ``NULL``, the return value might be a shared
    object, i.e. modification of the data is not allowed.
 
-   If *u* is *NULL*, this function behaves like :c:func:`PyUnicode_FromUnicode`
-   with the buffer set to *NULL*.  This usage is deprecated in favor of
+   If *u* is ``NULL``, this function behaves like :c:func:`PyUnicode_FromUnicode`
+   with the buffer set to ``NULL``.  This usage is deprecated in favor of
    :c:func:`PyUnicode_New`.
 
 
@@ -443,82 +443,82 @@ APIs:
 
    .. tabularcolumns:: |l|l|L|
 
-   +-------------------+---------------------+--------------------------------+
-   | Format Characters | Type                | Comment                        |
-   +===================+=====================+================================+
-   | :attr:`%%`        | *n/a*               | The literal % character.       |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%c`        | int                 | A single character,            |
-   |                   |                     | represented as a C int.        |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%d`        | int                 | Equivalent to                  |
-   |                   |                     | ``printf("%d")``. [1]_         |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%u`        | unsigned int        | Equivalent to                  |
-   |                   |                     | ``printf("%u")``. [1]_         |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%ld`       | long                | Equivalent to                  |
-   |                   |                     | ``printf("%ld")``. [1]_        |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%li`       | long                | Equivalent to                  |
-   |                   |                     | ``printf("%li")``. [1]_        |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%lu`       | unsigned long       | Equivalent to                  |
-   |                   |                     | ``printf("%lu")``. [1]_        |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%lld`      | long long           | Equivalent to                  |
-   |                   |                     | ``printf("%lld")``. [1]_       |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%lli`      | long long           | Equivalent to                  |
-   |                   |                     | ``printf("%lli")``. [1]_       |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%llu`      | unsigned long long  | Equivalent to                  |
-   |                   |                     | ``printf("%llu")``. [1]_       |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%zd`       | Py_ssize_t          | Equivalent to                  |
-   |                   |                     | ``printf("%zd")``. [1]_        |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%zi`       | Py_ssize_t          | Equivalent to                  |
-   |                   |                     | ``printf("%zi")``. [1]_        |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%zu`       | size_t              | Equivalent to                  |
-   |                   |                     | ``printf("%zu")``. [1]_        |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%i`        | int                 | Equivalent to                  |
-   |                   |                     | ``printf("%i")``. [1]_         |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%x`        | int                 | Equivalent to                  |
-   |                   |                     | ``printf("%x")``. [1]_         |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%s`        | const char\*        | A null-terminated C character  |
-   |                   |                     | array.                         |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%p`        | const void\*        | The hex representation of a C  |
-   |                   |                     | pointer. Mostly equivalent to  |
-   |                   |                     | ``printf("%p")`` except that   |
-   |                   |                     | it is guaranteed to start with |
-   |                   |                     | the literal ``0x`` regardless  |
-   |                   |                     | of what the platform's         |
-   |                   |                     | ``printf`` yields.             |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%A`        | PyObject\*          | The result of calling          |
-   |                   |                     | :func:`ascii`.                 |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%U`        | PyObject\*          | A Unicode object.              |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%V`        | PyObject\*,         | A Unicode object (which may be |
-   |                   | const char\*        | *NULL*) and a null-terminated  |
-   |                   |                     | C character array as a second  |
-   |                   |                     | parameter (which will be used, |
-   |                   |                     | if the first parameter is      |
-   |                   |                     | *NULL*).                       |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%S`        | PyObject\*          | The result of calling          |
-   |                   |                     | :c:func:`PyObject_Str`.        |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%R`        | PyObject\*          | The result of calling          |
-   |                   |                     | :c:func:`PyObject_Repr`.       |
-   +-------------------+---------------------+--------------------------------+
+   +-------------------+---------------------+----------------------------------+
+   | Format Characters | Type                | Comment                          |
+   +===================+=====================+==================================+
+   | :attr:`%%`        | *n/a*               | The literal % character.         |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%c`        | int                 | A single character,              |
+   |                   |                     | represented as a C int.          |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%d`        | int                 | Equivalent to                    |
+   |                   |                     | ``printf("%d")``. [1]_           |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%u`        | unsigned int        | Equivalent to                    |
+   |                   |                     | ``printf("%u")``. [1]_           |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%ld`       | long                | Equivalent to                    |
+   |                   |                     | ``printf("%ld")``. [1]_          |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%li`       | long                | Equivalent to                    |
+   |                   |                     | ``printf("%li")``. [1]_          |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%lu`       | unsigned long       | Equivalent to                    |
+   |                   |                     | ``printf("%lu")``. [1]_          |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%lld`      | long long           | Equivalent to                    |
+   |                   |                     | ``printf("%lld")``. [1]_         |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%lli`      | long long           | Equivalent to                    |
+   |                   |                     | ``printf("%lli")``. [1]_         |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%llu`      | unsigned long long  | Equivalent to                    |
+   |                   |                     | ``printf("%llu")``. [1]_         |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%zd`       | Py_ssize_t          | Equivalent to                    |
+   |                   |                     | ``printf("%zd")``. [1]_          |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%zi`       | Py_ssize_t          | Equivalent to                    |
+   |                   |                     | ``printf("%zi")``. [1]_          |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%zu`       | size_t              | Equivalent to                    |
+   |                   |                     | ``printf("%zu")``. [1]_          |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%i`        | int                 | Equivalent to                    |
+   |                   |                     | ``printf("%i")``. [1]_           |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%x`        | int                 | Equivalent to                    |
+   |                   |                     | ``printf("%x")``. [1]_           |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%s`        | const char\*        | A null-terminated C character    |
+   |                   |                     | array.                           |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%p`        | const void\*        | The hex representation of a C    |
+   |                   |                     | pointer. Mostly equivalent to    |
+   |                   |                     | ``printf("%p")`` except that     |
+   |                   |                     | it is guaranteed to start with   |
+   |                   |                     | the literal ``0x`` regardless    |
+   |                   |                     | of what the platform's           |
+   |                   |                     | ``printf`` yields.               |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%A`        | PyObject\*          | The result of calling            |
+   |                   |                     | :func:`ascii`.                   |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%U`        | PyObject\*          | A Unicode object.                |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%V`        | PyObject\*,         | A Unicode object (which may be   |
+   |                   | const char\*        | ``NULL``) and a null-terminated  |
+   |                   |                     | C character array as a second    |
+   |                   |                     | parameter (which will be used,   |
+   |                   |                     | if the first parameter is        |
+   |                   |                     | ``NULL``).                       |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%S`        | PyObject\*          | The result of calling            |
+   |                   |                     | :c:func:`PyObject_Str`.          |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%R`        | PyObject\*          | The result of calling            |
+   |                   |                     | :c:func:`PyObject_Repr`.         |
+   +-------------------+---------------------+----------------------------------+
 
    An unrecognized format character causes all the rest of the format string to be
    copied as-is to the result string, and any extra arguments discarded.
@@ -558,13 +558,13 @@ APIs:
    :class:`bytes`, :class:`bytearray` and other
    :term:`bytes-like objects <bytes-like object>`
    are decoded according to the given *encoding* and using the error handling
-   defined by *errors*. Both can be *NULL* to have the interface use the default
+   defined by *errors*. Both can be ``NULL`` to have the interface use the default
    values (see :ref:`builtincodecs` for details).
 
    All other objects, including Unicode objects, cause a :exc:`TypeError` to be
    set.
 
-   The API returns *NULL* if there was an error.  The caller is responsible for
+   The API returns ``NULL`` if there was an error.  The caller is responsible for
    decref'ing the returned objects.
 
 
@@ -640,7 +640,7 @@ APIs:
                                           Py_ssize_t buflen, int copy_null)
 
    Copy the string *u* into a UCS4 buffer, including a null character, if
-   *copy_null* is set.  Returns *NULL* and sets an exception on error (in
+   *copy_null* is set.  Returns ``NULL`` and sets an exception on error (in
    particular, a :exc:`SystemError` if *buflen* is smaller than the length of
    *u*).  *buffer* is returned on success.
 
@@ -650,7 +650,7 @@ APIs:
 .. c:function:: Py_UCS4* PyUnicode_AsUCS4Copy(PyObject *u)
 
    Copy the string *u* into a new UCS4 buffer that is allocated using
-   :c:func:`PyMem_Malloc`.  If this fails, *NULL* is returned with a
+   :c:func:`PyMem_Malloc`.  If this fails, ``NULL`` is returned with a
    :exc:`MemoryError` set.  The returned buffer always has an extra
    null code point appended.
 
@@ -670,15 +670,15 @@ Extension modules can continue using them, as they will not be removed in Python
 .. c:function:: PyObject* PyUnicode_FromUnicode(const Py_UNICODE *u, Py_ssize_t size)
 
    Create a Unicode object from the Py_UNICODE buffer *u* of the given size. *u*
-   may be *NULL* which causes the contents to be undefined. It is the user's
+   may be ``NULL`` which causes the contents to be undefined. It is the user's
    responsibility to fill in the needed data.  The buffer is copied into the new
    object.
 
-   If the buffer is not *NULL*, the return value might be a shared object.
+   If the buffer is not ``NULL``, the return value might be a shared object.
    Therefore, modification of the resulting Unicode object is only allowed when
-   *u* is *NULL*.
+   *u* is ``NULL``.
 
-   If the buffer is *NULL*, :c:func:`PyUnicode_READY` must be called once the
+   If the buffer is ``NULL``, :c:func:`PyUnicode_READY` must be called once the
    string content has been filled before using any of the access macros such as
    :c:func:`PyUnicode_KIND`.
 
@@ -689,7 +689,7 @@ Extension modules can continue using them, as they will not be removed in Python
 .. c:function:: Py_UNICODE* PyUnicode_AsUnicode(PyObject *unicode)
 
    Return a read-only pointer to the Unicode object's internal
-   :c:type:`Py_UNICODE` buffer, or *NULL* on error. This will create the
+   :c:type:`Py_UNICODE` buffer, or ``NULL`` on error. This will create the
    :c:type:`Py_UNICODE*` representation of the object if it is not yet
    available. The buffer is always terminated with an extra null code point.
    Note that the resulting :c:type:`Py_UNICODE` string may also contain
@@ -705,7 +705,7 @@ Extension modules can continue using them, as they will not be removed in Python
 
    Create a Unicode object by replacing all decimal digits in
    :c:type:`Py_UNICODE` buffer of the given *size* by ASCII digits 0--9
-   according to their decimal value.  Return *NULL* if an exception occurs.
+   according to their decimal value.  Return ``NULL`` if an exception occurs.
 
 
 .. c:function:: Py_UNICODE* PyUnicode_AsUnicodeAndSize(PyObject *unicode, Py_ssize_t *size)
@@ -721,7 +721,7 @@ Extension modules can continue using them, as they will not be removed in Python
 
 .. c:function:: Py_UNICODE* PyUnicode_AsUnicodeCopy(PyObject *unicode)
 
-   Create a copy of a Unicode string ending with a null code point. Return *NULL*
+   Create a copy of a Unicode string ending with a null code point. Return ``NULL``
    and raise a :exc:`MemoryError` exception on memory allocation failure,
    otherwise return a new allocated buffer (use :c:func:`PyMem_Free` to free
    the buffer). Note that the resulting :c:type:`Py_UNICODE*` string may
@@ -932,7 +932,7 @@ wchar_t Support
    Create a Unicode object from the :c:type:`wchar_t` buffer *w* of the given *size*.
    Passing ``-1`` as the *size* indicates that the function must itself compute the length,
    using wcslen.
-   Return *NULL* on failure.
+   Return ``NULL`` on failure.
 
 
 .. c:function:: Py_ssize_t PyUnicode_AsWideChar(PyObject *unicode, wchar_t *w, Py_ssize_t size)
@@ -951,22 +951,22 @@ wchar_t Support
 .. c:function:: wchar_t* PyUnicode_AsWideCharString(PyObject *unicode, Py_ssize_t *size)
 
    Convert the Unicode object to a wide character string. The output string
-   always ends with a null character. If *size* is not *NULL*, write the number
+   always ends with a null character. If *size* is not ``NULL``, write the number
    of wide characters (excluding the trailing null termination character) into
    *\*size*. Note that the resulting :c:type:`wchar_t` string might contain
    null characters, which would cause the string to be truncated when used with
-   most C functions. If *size* is *NULL* and the :c:type:`wchar_t*` string
+   most C functions. If *size* is ``NULL`` and the :c:type:`wchar_t*` string
    contains null characters a :exc:`ValueError` is raised.
 
    Returns a buffer allocated by :c:func:`PyMem_Alloc` (use
-   :c:func:`PyMem_Free` to free it) on success. On error, returns *NULL*
+   :c:func:`PyMem_Free` to free it) on success. On error, returns ``NULL``
    and *\*size* is undefined. Raises a :exc:`MemoryError` if memory allocation
    is failed.
 
    .. versionadded:: 3.2
 
    .. versionchanged:: 3.7
-      Raises a :exc:`ValueError` if *size* is *NULL* and the :c:type:`wchar_t*`
+      Raises a :exc:`ValueError` if *size* is ``NULL`` and the :c:type:`wchar_t*`
       string contains null characters.
 
 
@@ -982,7 +982,7 @@ Many of the following APIs take two arguments encoding and errors, and they
 have the same semantics as the ones of the built-in :func:`str` string object
 constructor.
 
-Setting encoding to *NULL* causes the default encoding to be used
+Setting encoding to ``NULL`` causes the default encoding to be used
 which is ASCII.  The file system calls should use
 :c:func:`PyUnicode_FSConverter` for encoding file names. This uses the
 variable :c:data:`Py_FileSystemDefaultEncoding` internally. This
@@ -990,7 +990,7 @@ variable should be treated as read-only: on some systems, it will be a
 pointer to a static string, on others, it will change at run-time
 (such as when the application invokes setlocale).
 
-Error handling is set by errors which may also be set to *NULL* meaning to use
+Error handling is set by errors which may also be set to ``NULL`` meaning to use
 the default handling defined for the codec.  Default error handling for all
 built-in codecs is "strict" (:exc:`ValueError` is raised).
 
@@ -1010,7 +1010,7 @@ These are the generic codec APIs:
    Create a Unicode object by decoding *size* bytes of the encoded string *s*.
    *encoding* and *errors* have the same meaning as the parameters of the same name
    in the :func:`str` built-in function.  The codec to be used is looked up
-   using the Python codec registry.  Return *NULL* if an exception was raised by
+   using the Python codec registry.  Return ``NULL`` if an exception was raised by
    the codec.
 
 
@@ -1020,7 +1020,7 @@ These are the generic codec APIs:
    Encode a Unicode object and return the result as Python bytes object.
    *encoding* and *errors* have the same meaning as the parameters of the same
    name in the Unicode :meth:`~str.encode` method. The codec to be used is looked up
-   using the Python codec registry. Return *NULL* if an exception was raised by
+   using the Python codec registry. Return ``NULL`` if an exception was raised by
    the codec.
 
 
@@ -1030,7 +1030,7 @@ These are the generic codec APIs:
    Encode the :c:type:`Py_UNICODE` buffer *s* of the given *size* and return a Python
    bytes object.  *encoding* and *errors* have the same meaning as the
    parameters of the same name in the Unicode :meth:`~str.encode` method.  The codec
-   to be used is looked up using the Python codec registry.  Return *NULL* if an
+   to be used is looked up using the Python codec registry.  Return ``NULL`` if an
    exception was raised by the codec.
 
    .. deprecated-removed:: 3.3 4.0
@@ -1047,14 +1047,14 @@ These are the UTF-8 codec APIs:
 .. c:function:: PyObject* PyUnicode_DecodeUTF8(const char *s, Py_ssize_t size, const char *errors)
 
    Create a Unicode object by decoding *size* bytes of the UTF-8 encoded string
-   *s*. Return *NULL* if an exception was raised by the codec.
+   *s*. Return ``NULL`` if an exception was raised by the codec.
 
 
 .. c:function:: PyObject* PyUnicode_DecodeUTF8Stateful(const char *s, Py_ssize_t size, \
                               const char *errors, Py_ssize_t *consumed)
 
-   If *consumed* is *NULL*, behave like :c:func:`PyUnicode_DecodeUTF8`. If
-   *consumed* is not *NULL*, trailing incomplete UTF-8 byte sequences will not be
+   If *consumed* is ``NULL``, behave like :c:func:`PyUnicode_DecodeUTF8`. If
+   *consumed* is not ``NULL``, trailing incomplete UTF-8 byte sequences will not be
    treated as an error. Those bytes will not be decoded and the number of bytes
    that have been decoded will be stored in *consumed*.
 
@@ -1062,7 +1062,7 @@ These are the UTF-8 codec APIs:
 .. c:function:: PyObject* PyUnicode_AsUTF8String(PyObject *unicode)
 
    Encode a Unicode object using UTF-8 and return the result as Python bytes
-   object.  Error handling is "strict".  Return *NULL* if an exception was
+   object.  Error handling is "strict".  Return ``NULL`` if an exception was
    raised by the codec.
 
 
@@ -1070,11 +1070,11 @@ These are the UTF-8 codec APIs:
 
    Return a pointer to the UTF-8 encoding of the Unicode object, and
    store the size of the encoded representation (in bytes) in *size*.  The
-   *size* argument can be *NULL*; in this case no size will be stored.  The
+   *size* argument can be ``NULL``; in this case no size will be stored.  The
    returned buffer always has an extra null byte appended (not included in
    *size*), regardless of whether there are any other null code points.
 
-   In the case of an error, *NULL* is returned with an exception set and no
+   In the case of an error, ``NULL`` is returned with an exception set and no
    *size* is stored.
 
    This caches the UTF-8 representation of the string in the Unicode object, and
@@ -1100,7 +1100,7 @@ These are the UTF-8 codec APIs:
 .. c:function:: PyObject* PyUnicode_EncodeUTF8(const Py_UNICODE *s, Py_ssize_t size, const char *errors)
 
    Encode the :c:type:`Py_UNICODE` buffer *s* of the given *size* using UTF-8 and
-   return a Python bytes object.  Return *NULL* if an exception was raised by
+   return a Python bytes object.  Return ``NULL`` if an exception was raised by
    the codec.
 
    .. deprecated-removed:: 3.3 4.0
@@ -1119,10 +1119,10 @@ These are the UTF-32 codec APIs:
                               const char *errors, int *byteorder)
 
    Decode *size* bytes from a UTF-32 encoded buffer string and return the
-   corresponding Unicode object.  *errors* (if non-*NULL*) defines the error
+   corresponding Unicode object.  *errors* (if non-``NULL``) defines the error
    handling. It defaults to "strict".
 
-   If *byteorder* is non-*NULL*, the decoder starts decoding using the given byte
+   If *byteorder* is non-``NULL``, the decoder starts decoding using the given byte
    order::
 
       *byteorder == -1: little endian
@@ -1137,16 +1137,16 @@ These are the UTF-32 codec APIs:
    After completion, *\*byteorder* is set to the current byte order at the end
    of input data.
 
-   If *byteorder* is *NULL*, the codec starts in native order mode.
+   If *byteorder* is ``NULL``, the codec starts in native order mode.
 
-   Return *NULL* if an exception was raised by the codec.
+   Return ``NULL`` if an exception was raised by the codec.
 
 
 .. c:function:: PyObject* PyUnicode_DecodeUTF32Stateful(const char *s, Py_ssize_t size, \
                               const char *errors, int *byteorder, Py_ssize_t *consumed)
 
-   If *consumed* is *NULL*, behave like :c:func:`PyUnicode_DecodeUTF32`. If
-   *consumed* is not *NULL*, :c:func:`PyUnicode_DecodeUTF32Stateful` will not treat
+   If *consumed* is ``NULL``, behave like :c:func:`PyUnicode_DecodeUTF32`. If
+   *consumed* is not ``NULL``, :c:func:`PyUnicode_DecodeUTF32Stateful` will not treat
    trailing incomplete UTF-32 byte sequences (such as a number of bytes not divisible
    by four) as an error. Those bytes will not be decoded and the number of bytes
    that have been decoded will be stored in *consumed*.
@@ -1156,7 +1156,7 @@ These are the UTF-32 codec APIs:
 
    Return a Python byte string using the UTF-32 encoding in native byte
    order. The string always starts with a BOM mark.  Error handling is "strict".
-   Return *NULL* if an exception was raised by the codec.
+   Return ``NULL`` if an exception was raised by the codec.
 
 
 .. c:function:: PyObject* PyUnicode_EncodeUTF32(const Py_UNICODE *s, Py_ssize_t size, \
@@ -1175,7 +1175,7 @@ These are the UTF-32 codec APIs:
    If *Py_UNICODE_WIDE* is not defined, surrogate pairs will be output
    as a single code point.
 
-   Return *NULL* if an exception was raised by the codec.
+   Return ``NULL`` if an exception was raised by the codec.
 
    .. deprecated-removed:: 3.3 4.0
       Part of the old-style :c:type:`Py_UNICODE` API; please migrate to using
@@ -1192,10 +1192,10 @@ These are the UTF-16 codec APIs:
                               const char *errors, int *byteorder)
 
    Decode *size* bytes from a UTF-16 encoded buffer string and return the
-   corresponding Unicode object.  *errors* (if non-*NULL*) defines the error
+   corresponding Unicode object.  *errors* (if non-``NULL``) defines the error
    handling. It defaults to "strict".
 
-   If *byteorder* is non-*NULL*, the decoder starts decoding using the given byte
+   If *byteorder* is non-``NULL``, the decoder starts decoding using the given byte
    order::
 
       *byteorder == -1: little endian
@@ -1211,16 +1211,16 @@ These are the UTF-16 codec APIs:
    After completion, *\*byteorder* is set to the current byte order at the end
    of input data.
 
-   If *byteorder* is *NULL*, the codec starts in native order mode.
+   If *byteorder* is ``NULL``, the codec starts in native order mode.
 
-   Return *NULL* if an exception was raised by the codec.
+   Return ``NULL`` if an exception was raised by the codec.
 
 
 .. c:function:: PyObject* PyUnicode_DecodeUTF16Stateful(const char *s, Py_ssize_t size, \
                               const char *errors, int *byteorder, Py_ssize_t *consumed)
 
-   If *consumed* is *NULL*, behave like :c:func:`PyUnicode_DecodeUTF16`. If
-   *consumed* is not *NULL*, :c:func:`PyUnicode_DecodeUTF16Stateful` will not treat
+   If *consumed* is ``NULL``, behave like :c:func:`PyUnicode_DecodeUTF16`. If
+   *consumed* is not ``NULL``, :c:func:`PyUnicode_DecodeUTF16Stateful` will not treat
    trailing incomplete UTF-16 byte sequences (such as an odd number of bytes or a
    split surrogate pair) as an error. Those bytes will not be decoded and the
    number of bytes that have been decoded will be stored in *consumed*.
@@ -1230,7 +1230,7 @@ These are the UTF-16 codec APIs:
 
    Return a Python byte string using the UTF-16 encoding in native byte
    order. The string always starts with a BOM mark.  Error handling is "strict".
-   Return *NULL* if an exception was raised by the codec.
+   Return ``NULL`` if an exception was raised by the codec.
 
 
 .. c:function:: PyObject* PyUnicode_EncodeUTF16(const Py_UNICODE *s, Py_ssize_t size, \
@@ -1250,7 +1250,7 @@ These are the UTF-16 codec APIs:
    represented as a surrogate pair. If it is not defined, each :c:type:`Py_UNICODE`
    values is interpreted as a UCS-2 character.
 
-   Return *NULL* if an exception was raised by the codec.
+   Return ``NULL`` if an exception was raised by the codec.
 
    .. deprecated-removed:: 3.3 4.0
       Part of the old-style :c:type:`Py_UNICODE` API; please migrate to using
@@ -1266,14 +1266,14 @@ These are the UTF-7 codec APIs:
 .. c:function:: PyObject* PyUnicode_DecodeUTF7(const char *s, Py_ssize_t size, const char *errors)
 
    Create a Unicode object by decoding *size* bytes of the UTF-7 encoded string
-   *s*.  Return *NULL* if an exception was raised by the codec.
+   *s*.  Return ``NULL`` if an exception was raised by the codec.
 
 
 .. c:function:: PyObject* PyUnicode_DecodeUTF7Stateful(const char *s, Py_ssize_t size, \
                               const char *errors, Py_ssize_t *consumed)
 
-   If *consumed* is *NULL*, behave like :c:func:`PyUnicode_DecodeUTF7`.  If
-   *consumed* is not *NULL*, trailing incomplete UTF-7 base-64 sections will not
+   If *consumed* is ``NULL``, behave like :c:func:`PyUnicode_DecodeUTF7`.  If
+   *consumed* is not ``NULL``, trailing incomplete UTF-7 base-64 sections will not
    be treated as an error.  Those bytes will not be decoded and the number of
    bytes that have been decoded will be stored in *consumed*.
 
@@ -1282,7 +1282,7 @@ These are the UTF-7 codec APIs:
                               int base64SetO, int base64WhiteSpace, const char *errors)
 
    Encode the :c:type:`Py_UNICODE` buffer of the given size using UTF-7 and
-   return a Python bytes object.  Return *NULL* if an exception was raised by
+   return a Python bytes object.  Return ``NULL`` if an exception was raised by
    the codec.
 
    If *base64SetO* is nonzero, "Set O" (punctuation that has no otherwise
@@ -1305,20 +1305,20 @@ These are the "Unicode Escape" codec APIs:
                               Py_ssize_t size, const char *errors)
 
    Create a Unicode object by decoding *size* bytes of the Unicode-Escape encoded
-   string *s*.  Return *NULL* if an exception was raised by the codec.
+   string *s*.  Return ``NULL`` if an exception was raised by the codec.
 
 
 .. c:function:: PyObject* PyUnicode_AsUnicodeEscapeString(PyObject *unicode)
 
    Encode a Unicode object using Unicode-Escape and return the result as a
-   bytes object.  Error handling is "strict".  Return *NULL* if an exception was
+   bytes object.  Error handling is "strict".  Return ``NULL`` if an exception was
    raised by the codec.
 
 
 .. c:function:: PyObject* PyUnicode_EncodeUnicodeEscape(const Py_UNICODE *s, Py_ssize_t size)
 
    Encode the :c:type:`Py_UNICODE` buffer of the given *size* using Unicode-Escape and
-   return a bytes object.  Return *NULL* if an exception was raised by the codec.
+   return a bytes object.  Return ``NULL`` if an exception was raised by the codec.
 
    .. deprecated-removed:: 3.3 4.0
       Part of the old-style :c:type:`Py_UNICODE` API; please migrate to using
@@ -1335,13 +1335,13 @@ These are the "Raw Unicode Escape" codec APIs:
                               Py_ssize_t size, const char *errors)
 
    Create a Unicode object by decoding *size* bytes of the Raw-Unicode-Escape
-   encoded string *s*.  Return *NULL* if an exception was raised by the codec.
+   encoded string *s*.  Return ``NULL`` if an exception was raised by the codec.
 
 
 .. c:function:: PyObject* PyUnicode_AsRawUnicodeEscapeString(PyObject *unicode)
 
    Encode a Unicode object using Raw-Unicode-Escape and return the result as
-   a bytes object.  Error handling is "strict".  Return *NULL* if an exception
+   a bytes object.  Error handling is "strict".  Return ``NULL`` if an exception
    was raised by the codec.
 
 
@@ -1349,7 +1349,7 @@ These are the "Raw Unicode Escape" codec APIs:
                               Py_ssize_t size)
 
    Encode the :c:type:`Py_UNICODE` buffer of the given *size* using Raw-Unicode-Escape
-   and return a bytes object.  Return *NULL* if an exception was raised by the codec.
+   and return a bytes object.  Return ``NULL`` if an exception was raised by the codec.
 
    .. deprecated-removed:: 3.3 4.0
       Part of the old-style :c:type:`Py_UNICODE` API; please migrate to using
@@ -1367,20 +1367,20 @@ ordinals and only these are accepted by the codecs during encoding.
 .. c:function:: PyObject* PyUnicode_DecodeLatin1(const char *s, Py_ssize_t size, const char *errors)
 
    Create a Unicode object by decoding *size* bytes of the Latin-1 encoded string
-   *s*.  Return *NULL* if an exception was raised by the codec.
+   *s*.  Return ``NULL`` if an exception was raised by the codec.
 
 
 .. c:function:: PyObject* PyUnicode_AsLatin1String(PyObject *unicode)
 
    Encode a Unicode object using Latin-1 and return the result as Python bytes
-   object.  Error handling is "strict".  Return *NULL* if an exception was
+   object.  Error handling is "strict".  Return ``NULL`` if an exception was
    raised by the codec.
 
 
 .. c:function:: PyObject* PyUnicode_EncodeLatin1(const Py_UNICODE *s, Py_ssize_t size, const char *errors)
 
    Encode the :c:type:`Py_UNICODE` buffer of the given *size* using Latin-1 and
-   return a Python bytes object.  Return *NULL* if an exception was raised by
+   return a Python bytes object.  Return ``NULL`` if an exception was raised by
    the codec.
 
    .. deprecated-removed:: 3.3 4.0
@@ -1399,20 +1399,20 @@ codes generate errors.
 .. c:function:: PyObject* PyUnicode_DecodeASCII(const char *s, Py_ssize_t size, const char *errors)
 
    Create a Unicode object by decoding *size* bytes of the ASCII encoded string
-   *s*.  Return *NULL* if an exception was raised by the codec.
+   *s*.  Return ``NULL`` if an exception was raised by the codec.
 
 
 .. c:function:: PyObject* PyUnicode_AsASCIIString(PyObject *unicode)
 
    Encode a Unicode object using ASCII and return the result as Python bytes
-   object.  Error handling is "strict".  Return *NULL* if an exception was
+   object.  Error handling is "strict".  Return ``NULL`` if an exception was
    raised by the codec.
 
 
 .. c:function:: PyObject* PyUnicode_EncodeASCII(const Py_UNICODE *s, Py_ssize_t size, const char *errors)
 
    Encode the :c:type:`Py_UNICODE` buffer of the given *size* using ASCII and
-   return a Python bytes object.  Return *NULL* if an exception was raised by
+   return a Python bytes object.  Return ``NULL`` if an exception was raised by
    the codec.
 
    .. deprecated-removed:: 3.3 4.0
@@ -1436,10 +1436,10 @@ These are the mapping codec APIs:
                               PyObject *mapping, const char *errors)
 
    Create a Unicode object by decoding *size* bytes of the encoded string *s*
-   using the given *mapping* object.  Return *NULL* if an exception was raised
+   using the given *mapping* object.  Return ``NULL`` if an exception was raised
    by the codec.
 
-   If *mapping* is *NULL*, Latin-1 decoding will be applied.  Else
+   If *mapping* is ``NULL``, Latin-1 decoding will be applied.  Else
    *mapping* must map bytes ordinals (integers in the range from 0 to 255)
    to Unicode strings, integers (which are then interpreted as Unicode
    ordinals) or ``None``.  Unmapped data bytes -- ones which cause a
@@ -1451,7 +1451,7 @@ These are the mapping codec APIs:
 .. c:function:: PyObject* PyUnicode_AsCharmapString(PyObject *unicode, PyObject *mapping)
 
    Encode a Unicode object using the given *mapping* object and return the
-   result as a bytes object.  Error handling is "strict".  Return *NULL* if an
+   result as a bytes object.  Error handling is "strict".  Return ``NULL`` if an
    exception was raised by the codec.
 
    The *mapping* object must map Unicode ordinal integers to bytes objects,
@@ -1464,7 +1464,7 @@ These are the mapping codec APIs:
                               PyObject *mapping, const char *errors)
 
    Encode the :c:type:`Py_UNICODE` buffer of the given *size* using the given
-   *mapping* object and return the result as a bytes object.  Return *NULL* if
+   *mapping* object and return the result as a bytes object.  Return ``NULL`` if
    an exception was raised by the codec.
 
    .. deprecated-removed:: 3.3 4.0
@@ -1479,7 +1479,7 @@ The following codec API is special in that maps Unicode to Unicode.
                               PyObject *mapping, const char *errors)
 
    Translate a Unicode object using the given *mapping* object and return the
-   resulting Unicode object.  Return *NULL* if an exception was raised by the
+   resulting Unicode object.  Return ``NULL`` if an exception was raised by the
    codec.
 
    The *mapping* object must map Unicode ordinal integers to Unicode strings,
@@ -1493,7 +1493,7 @@ The following codec API is special in that maps Unicode to Unicode.
 
    Translate a :c:type:`Py_UNICODE` buffer of the given *size* by applying a
    character *mapping* table to it and return the resulting Unicode object.
-   Return *NULL* when an exception was raised by the codec.
+   Return ``NULL`` when an exception was raised by the codec.
 
    .. deprecated-removed:: 3.3 4.0
       Part of the old-style :c:type:`Py_UNICODE` API; please migrate to using
@@ -1512,14 +1512,14 @@ the user settings on the machine running the codec.
 .. c:function:: PyObject* PyUnicode_DecodeMBCS(const char *s, Py_ssize_t size, const char *errors)
 
    Create a Unicode object by decoding *size* bytes of the MBCS encoded string *s*.
-   Return *NULL* if an exception was raised by the codec.
+   Return ``NULL`` if an exception was raised by the codec.
 
 
 .. c:function:: PyObject* PyUnicode_DecodeMBCSStateful(const char *s, Py_ssize_t size, \
                               const char *errors, Py_ssize_t *consumed)
 
-   If *consumed* is *NULL*, behave like :c:func:`PyUnicode_DecodeMBCS`. If
-   *consumed* is not *NULL*, :c:func:`PyUnicode_DecodeMBCSStateful` will not decode
+   If *consumed* is ``NULL``, behave like :c:func:`PyUnicode_DecodeMBCS`. If
+   *consumed* is not ``NULL``, :c:func:`PyUnicode_DecodeMBCSStateful` will not decode
    trailing lead byte and the number of bytes that have been decoded will be stored
    in *consumed*.
 
@@ -1527,14 +1527,14 @@ the user settings on the machine running the codec.
 .. c:function:: PyObject* PyUnicode_AsMBCSString(PyObject *unicode)
 
    Encode a Unicode object using MBCS and return the result as Python bytes
-   object.  Error handling is "strict".  Return *NULL* if an exception was
+   object.  Error handling is "strict".  Return ``NULL`` if an exception was
    raised by the codec.
 
 
 .. c:function:: PyObject* PyUnicode_EncodeCodePage(int code_page, PyObject *unicode, const char *errors)
 
    Encode the Unicode object using the specified code page and return a Python
-   bytes object.  Return *NULL* if an exception was raised by the codec. Use
+   bytes object.  Return ``NULL`` if an exception was raised by the codec. Use
    :c:data:`CP_ACP` code page to get the MBCS encoder.
 
    .. versionadded:: 3.3
@@ -1543,7 +1543,7 @@ the user settings on the machine running the codec.
 .. c:function:: PyObject* PyUnicode_EncodeMBCS(const Py_UNICODE *s, Py_ssize_t size, const char *errors)
 
    Encode the :c:type:`Py_UNICODE` buffer of the given *size* using MBCS and return
-   a Python bytes object.  Return *NULL* if an exception was raised by the
+   a Python bytes object.  Return ``NULL`` if an exception was raised by the
    codec.
 
    .. deprecated-removed:: 3.3 4.0
@@ -1565,7 +1565,7 @@ The following APIs are capable of handling Unicode objects and strings on input
 (we refer to them as strings in the descriptions) and return Unicode objects or
 integers as appropriate.
 
-They all return *NULL* or ``-1`` if an exception occurs.
+They all return ``NULL`` or ``-1`` if an exception occurs.
 
 
 .. c:function:: PyObject* PyUnicode_Concat(PyObject *left, PyObject *right)
@@ -1575,7 +1575,7 @@ They all return *NULL* or ``-1`` if an exception occurs.
 
 .. c:function:: PyObject* PyUnicode_Split(PyObject *s, PyObject *sep, Py_ssize_t maxsplit)
 
-   Split a string giving a list of Unicode strings.  If *sep* is *NULL*, splitting
+   Split a string giving a list of Unicode strings.  If *sep* is ``NULL``, splitting
    will be done at all whitespace substrings.  Otherwise, splits occur at the given
    separator.  At most *maxsplit* splits will be done.  If negative, no limit is
    set.  Separators are not included in the resulting list.
@@ -1601,7 +1601,7 @@ They all return *NULL* or ``-1`` if an exception occurs.
    and sequences work well.  Unmapped character ordinals (ones which cause a
    :exc:`LookupError`) are left untouched and are copied as-is.
 
-   *errors* has the usual meaning for codecs. It may be *NULL* which indicates to
+   *errors* has the usual meaning for codecs. It may be ``NULL`` which indicates to
    use the default error handling.
 
 
diff --git a/Doc/c-api/veryhigh.rst b/Doc/c-api/veryhigh.rst
index cc194c200a37c..fce39b5c81b2a 100644
--- a/Doc/c-api/veryhigh.rst
+++ b/Doc/c-api/veryhigh.rst
@@ -52,7 +52,7 @@ the same library that the Python runtime is using.
 .. c:function:: int PyRun_AnyFile(FILE *fp, const char *filename)
 
    This is a simplified interface to :c:func:`PyRun_AnyFileExFlags` below, leaving
-   *closeit* set to ``0`` and *flags* set to *NULL*.
+   *closeit* set to ``0`` and *flags* set to ``NULL``.
 
 
 .. c:function:: int PyRun_AnyFileFlags(FILE *fp, const char *filename, PyCompilerFlags *flags)
@@ -64,7 +64,7 @@ the same library that the Python runtime is using.
 .. c:function:: int PyRun_AnyFileEx(FILE *fp, const char *filename, int closeit)
 
    This is a simplified interface to :c:func:`PyRun_AnyFileExFlags` below, leaving
-   the *flags* argument set to *NULL*.
+   the *flags* argument set to ``NULL``.
 
 
 .. c:function:: int PyRun_AnyFileExFlags(FILE *fp, const char *filename, int closeit, PyCompilerFlags *flags)
@@ -73,7 +73,7 @@ the same library that the Python runtime is using.
    terminal input or Unix pseudo-terminal), return the value of
    :c:func:`PyRun_InteractiveLoop`, otherwise return the result of
    :c:func:`PyRun_SimpleFile`.  *filename* is decoded from the filesystem
-   encoding (:func:`sys.getfilesystemencoding`).  If *filename* is *NULL*, this
+   encoding (:func:`sys.getfilesystemencoding`).  If *filename* is ``NULL``, this
    function uses ``"???"`` as the filename.
 
 
@@ -99,13 +99,13 @@ the same library that the Python runtime is using.
 .. c:function:: int PyRun_SimpleFile(FILE *fp, const char *filename)
 
    This is a simplified interface to :c:func:`PyRun_SimpleFileExFlags` below,
-   leaving *closeit* set to ``0`` and *flags* set to *NULL*.
+   leaving *closeit* set to ``0`` and *flags* set to ``NULL``.
 
 
 .. c:function:: int PyRun_SimpleFileEx(FILE *fp, const char *filename, int closeit)
 
    This is a simplified interface to :c:func:`PyRun_SimpleFileExFlags` below,
-   leaving *flags* set to *NULL*.
+   leaving *flags* set to ``NULL``.
 
 
 .. c:function:: int PyRun_SimpleFileExFlags(FILE *fp, const char *filename, int closeit, PyCompilerFlags *flags)
@@ -124,7 +124,7 @@ the same library that the Python runtime is using.
 .. c:function:: int PyRun_InteractiveOne(FILE *fp, const char *filename)
 
    This is a simplified interface to :c:func:`PyRun_InteractiveOneFlags` below,
-   leaving *flags* set to *NULL*.
+   leaving *flags* set to ``NULL``.
 
 
 .. c:function:: int PyRun_InteractiveOneFlags(FILE *fp, const char *filename, PyCompilerFlags *flags)
@@ -144,7 +144,7 @@ the same library that the Python runtime is using.
 .. c:function:: int PyRun_InteractiveLoop(FILE *fp, const char *filename)
 
    This is a simplified interface to :c:func:`PyRun_InteractiveLoopFlags` below,
-   leaving *flags* set to *NULL*.
+   leaving *flags* set to ``NULL``.
 
 
 .. c:function:: int PyRun_InteractiveLoopFlags(FILE *fp, const char *filename, PyCompilerFlags *flags)
@@ -173,13 +173,13 @@ the same library that the Python runtime is using.
    ``char *func(FILE *stdin, FILE *stdout, char *prompt)``,
    overriding the default function used to read a single line of input
    at the interpreter's prompt.  The function is expected to output
-   the string *prompt* if it's not *NULL*, and then read a line of
+   the string *prompt* if it's not ``NULL``, and then read a line of
    input from the provided standard input file, returning the
    resulting string.  For example, The :mod:`readline` module sets
    this hook to provide line-editing and tab-completion features.
 
    The result must be a string allocated by :c:func:`PyMem_RawMalloc` or
-   :c:func:`PyMem_RawRealloc`, or *NULL* if an error occurred.
+   :c:func:`PyMem_RawRealloc`, or ``NULL`` if an error occurred.
 
    .. versionchanged:: 3.4
       The result must be allocated by :c:func:`PyMem_RawMalloc` or
@@ -191,14 +191,14 @@ the same library that the Python runtime is using.
 
    This is a simplified interface to
    :c:func:`PyParser_SimpleParseStringFlagsFilename` below, leaving  *filename* set
-   to *NULL* and *flags* set to ``0``.
+   to ``NULL`` and *flags* set to ``0``.
 
 
 .. c:function:: struct _node* PyParser_SimpleParseStringFlags( const char *str, int start, int flags)
 
    This is a simplified interface to
    :c:func:`PyParser_SimpleParseStringFlagsFilename` below, leaving  *filename* set
-   to *NULL*.
+   to ``NULL``.
 
 
 .. c:function:: struct _node* PyParser_SimpleParseStringFlagsFilename( const char *str, const char *filename, int start, int flags)
@@ -225,7 +225,7 @@ the same library that the Python runtime is using.
 .. c:function:: PyObject* PyRun_String(const char *str, int start, PyObject *globals, PyObject *locals)
 
    This is a simplified interface to :c:func:`PyRun_StringFlags` below, leaving
-   *flags* set to *NULL*.
+   *flags* set to ``NULL``.
 
 
 .. c:function:: PyObject* PyRun_StringFlags(const char *str, int start, PyObject *globals, PyObject *locals, PyCompilerFlags *flags)
@@ -236,20 +236,20 @@ the same library that the Python runtime is using.
    that implements the mapping protocol.  The parameter *start* specifies
    the start token that should be used to parse the source code.
 
-   Returns the result of executing the code as a Python object, or *NULL* if an
+   Returns the result of executing the code as a Python object, or ``NULL`` if an
    exception was raised.
 
 
 .. c:function:: PyObject* PyRun_File(FILE *fp, const char *filename, int start, PyObject *globals, PyObject *locals)
 
    This is a simplified interface to :c:func:`PyRun_FileExFlags` below, leaving
-   *closeit* set to ``0`` and *flags* set to *NULL*.
+   *closeit* set to ``0`` and *flags* set to ``NULL``.
 
 
 .. c:function:: PyObject* PyRun_FileEx(FILE *fp, const char *filename, int start, PyObject *globals, PyObject *locals, int closeit)
 
    This is a simplified interface to :c:func:`PyRun_FileExFlags` below, leaving
-   *flags* set to *NULL*.
+   *flags* set to ``NULL``.
 
 
 .. c:function:: PyObject* PyRun_FileFlags(FILE *fp, const char *filename, int start, PyObject *globals, PyObject *locals, PyCompilerFlags *flags)
@@ -270,7 +270,7 @@ the same library that the Python runtime is using.
 .. c:function:: PyObject* Py_CompileString(const char *str, const char *filename, int start)
 
    This is a simplified interface to :c:func:`Py_CompileStringFlags` below, leaving
-   *flags* set to *NULL*.
+   *flags* set to ``NULL``.
 
 
 .. c:function:: PyObject* Py_CompileStringFlags(const char *str, const char *filename, int start, PyCompilerFlags *flags)
@@ -286,7 +286,7 @@ the same library that the Python runtime is using.
    code which can be compiled and should be :const:`Py_eval_input`,
    :const:`Py_file_input`, or :const:`Py_single_input`.  The filename specified by
    *filename* is used to construct the code object and may appear in tracebacks or
-   :exc:`SyntaxError` exception messages.  This returns *NULL* if the code
+   :exc:`SyntaxError` exception messages.  This returns ``NULL`` if the code
    cannot be parsed or compiled.
 
    The integer *optimize* specifies the optimization level of the compiler; a
@@ -309,7 +309,7 @@ the same library that the Python runtime is using.
 
    This is a simplified interface to :c:func:`PyEval_EvalCodeEx`, with just
    the code object, and global and local variables.  The other arguments are
-   set to *NULL*.
+   set to ``NULL``.
 
 
 .. c:function:: PyObject* PyEval_EvalCodeEx(PyObject *co, PyObject *globals, PyObject *locals, PyObject *const *args, int argcount, PyObject *const *kws, int kwcount, PyObject *const *defs, int defcount, PyObject *kwdefs, PyObject *closure)
@@ -386,7 +386,7 @@ the same library that the Python runtime is using.
    executed, it is passed as ``PyCompilerFlags *flags``.  In this case, ``from
    __future__ import`` can modify *flags*.
 
-   Whenever ``PyCompilerFlags *flags`` is *NULL*, :attr:`cf_flags` is treated as
+   Whenever ``PyCompilerFlags *flags`` is ``NULL``, :attr:`cf_flags` is treated as
    equal to ``0``, and any modification due to ``from __future__ import`` is
    discarded.
 
diff --git a/Doc/c-api/weakref.rst b/Doc/c-api/weakref.rst
index 7a4f8615b9c32..e3a9bda54d671 100644
--- a/Doc/c-api/weakref.rst
+++ b/Doc/c-api/weakref.rst
@@ -33,9 +33,9 @@ as much as it can.
    reference object may be returned.  The second parameter, *callback*, can be a
    callable object that receives notification when *ob* is garbage collected; it
    should accept a single parameter, which will be the weak reference object
-   itself. *callback* may also be ``None`` or *NULL*.  If *ob* is not a
+   itself. *callback* may also be ``None`` or ``NULL``.  If *ob* is not a
    weakly-referencable object, or if *callback* is not callable, ``None``, or
-   *NULL*, this will return *NULL* and raise :exc:`TypeError`.
+   ``NULL``, this will return ``NULL`` and raise :exc:`TypeError`.
 
 
 .. c:function:: PyObject* PyWeakref_NewProxy(PyObject *ob, PyObject *callback)
@@ -45,9 +45,9 @@ as much as it can.
    existing proxy object may be returned.  The second parameter, *callback*, can
    be a callable object that receives notification when *ob* is garbage
    collected; it should accept a single parameter, which will be the weak
-   reference object itself. *callback* may also be ``None`` or *NULL*.  If *ob*
+   reference object itself. *callback* may also be ``None`` or ``NULL``.  If *ob*
    is not a weakly-referencable object, or if *callback* is not callable,
-   ``None``, or *NULL*, this will return *NULL* and raise :exc:`TypeError`.
+   ``None``, or ``NULL``, this will return ``NULL`` and raise :exc:`TypeError`.
 
 
 .. c:function:: PyObject* PyWeakref_GetObject(PyObject *ref)
diff --git a/Doc/extending/embedding.rst b/Doc/extending/embedding.rst
index 483bc852f6c77..5f5abdf9c1506 100644
--- a/Doc/extending/embedding.rst
+++ b/Doc/extending/embedding.rst
@@ -196,7 +196,7 @@ function is then made with::
 
    pValue = PyObject_CallObject(pFunc, pArgs);
 
-Upon return of the function, ``pValue`` is either *NULL* or it contains a
+Upon return of the function, ``pValue`` is either ``NULL`` or it contains a
 reference to the return value of the function.  Be sure to release the reference
 after examining the value.
 
diff --git a/Doc/extending/extending.rst b/Doc/extending/extending.rst
index 5b4ea8220e237..7986ea1e15045 100644
--- a/Doc/extending/extending.rst
+++ b/Doc/extending/extending.rst
@@ -117,7 +117,7 @@ store the converted values.  More about this later.
 type and its components have been stored in the variables whose addresses are
 passed.  It returns false (zero) if an invalid argument list was passed.  In the
 latter case it also raises an appropriate exception so the calling function can
-return *NULL* immediately (as we saw in the example).
+return ``NULL`` immediately (as we saw in the example).
 
 
 .. _extending-errors:
@@ -127,8 +127,8 @@ Intermezzo: Errors and Exceptions
 
 An important convention throughout the Python interpreter is the following: when
 a function fails, it should set an exception condition and return an error value
-(usually a *NULL* pointer).  Exceptions are stored in a static global variable
-inside the interpreter; if this variable is *NULL* no exception has occurred.  A
+(usually a ``NULL`` pointer).  Exceptions are stored in a static global variable
+inside the interpreter; if this variable is ``NULL`` no exception has occurred.  A
 second global variable stores the "associated value" of the exception (the
 second argument to :keyword:`raise`).  A third variable contains the stack
 traceback in case the error originated in Python code.  These three variables
@@ -152,13 +152,13 @@ its associated value.  You don't need to :c:func:`Py_INCREF` the objects passed
 to any of these functions.
 
 You can test non-destructively whether an exception has been set with
-:c:func:`PyErr_Occurred`.  This returns the current exception object, or *NULL*
+:c:func:`PyErr_Occurred`.  This returns the current exception object, or ``NULL``
 if no exception has occurred.  You normally don't need to call
 :c:func:`PyErr_Occurred` to see whether an error occurred in a function call,
 since you should be able to tell from the return value.
 
 When a function *f* that calls another function *g* detects that the latter
-fails, *f* should itself return an error value (usually *NULL* or ``-1``).  It
+fails, *f* should itself return an error value (usually ``NULL`` or ``-1``).  It
 should *not* call one of the :c:func:`PyErr_\*` functions --- one has already
 been called by *g*. *f*'s caller is then supposed to also return an error
 indication to *its* caller, again *without* calling :c:func:`PyErr_\*`, and so on
@@ -234,7 +234,7 @@ with an exception object::
 
 Note that the Python name for the exception object is :exc:`spam.error`.  The
 :c:func:`PyErr_NewException` function may create a class with the base class
-being :exc:`Exception` (unless another class is passed in instead of *NULL*),
+being :exc:`Exception` (unless another class is passed in instead of ``NULL``),
 described in :ref:`bltin-exceptions`.
 
 Note also that the :c:data:`SpamError` variable retains a reference to the newly
@@ -278,7 +278,7 @@ statement::
    if (!PyArg_ParseTuple(args, "s", &command))
        return NULL;
 
-It returns *NULL* (the error indicator for functions returning object pointers)
+It returns ``NULL`` (the error indicator for functions returning object pointers)
 if an error is detected in the argument list, relying on the exception set by
 :c:func:`PyArg_ParseTuple`.  Otherwise the string value of the argument has been
 copied to the local variable :c:data:`command`.  This is a pointer assignment and
@@ -308,7 +308,7 @@ macro)::
    return Py_None;
 
 :c:data:`Py_None` is the C name for the special Python object ``None``.  It is a
-genuine Python object rather than a *NULL* pointer, which means "error" in most
+genuine Python object rather than a ``NULL`` pointer, which means "error" in most
 contexts, as we have seen.
 
 
@@ -376,7 +376,7 @@ inserts built-in function objects into the newly created module based upon the
 table (an array of :c:type:`PyMethodDef` structures) found in the module definition.
 :c:func:`PyModule_Create` returns a pointer to the module object
 that it creates.  It may abort with a fatal error for
-certain errors, or return *NULL* if the module could not be initialized
+certain errors, or return ``NULL`` if the module could not be initialized
 satisfactorily. The init function must return the module object to its caller,
 so that it then gets inserted into ``sys.modules``.
 
@@ -526,8 +526,8 @@ This function must be registered with the interpreter using the
 :ref:`parsetuple`.
 
 The macros :c:func:`Py_XINCREF` and :c:func:`Py_XDECREF` increment/decrement the
-reference count of an object and are safe in the presence of *NULL* pointers
-(but note that *temp* will not be  *NULL* in this context).  More info on them
+reference count of an object and are safe in the presence of ``NULL`` pointers
+(but note that *temp* will not be  ``NULL`` in this context).  More info on them
 in section :ref:`refcounts`.
 
 .. index:: single: PyObject_CallObject()
@@ -566,7 +566,7 @@ somehow :c:func:`Py_DECREF` the result, even (especially!) if you are not
 interested in its value.
 
 Before you do this, however, it is important to check that the return value
-isn't *NULL*.  If it is, the Python function terminated by raising an exception.
+isn't ``NULL``.  If it is, the Python function terminated by raising an exception.
 If the C code that called :c:func:`PyObject_CallObject` is called from Python, it
 should now return an error indication to its Python caller, so the interpreter
 can print a stack trace, or the calling Python code can handle the exception.
@@ -723,7 +723,7 @@ The :c:func:`PyArg_ParseTupleAndKeywords` function is declared as follows::
 The *arg* and *format* parameters are identical to those of the
 :c:func:`PyArg_ParseTuple` function.  The *kwdict* parameter is the dictionary of
 keywords received as the third parameter from the Python runtime.  The *kwlist*
-parameter is a *NULL*-terminated list of strings which identify the parameters;
+parameter is a ``NULL``-terminated list of strings which identify the parameters;
 the names are matched with the type information from *format* from left to
 right.  On success, :c:func:`PyArg_ParseTupleAndKeywords` returns true, otherwise
 it returns false and raises an appropriate exception.
@@ -1084,32 +1084,32 @@ NULL Pointers
 -------------
 
 In general, functions that take object references as arguments do not expect you
-to pass them *NULL* pointers, and will dump core (or cause later core dumps) if
-you do so.  Functions that return object references generally return *NULL* only
-to indicate that an exception occurred.  The reason for not testing for *NULL*
+to pass them ``NULL`` pointers, and will dump core (or cause later core dumps) if
+you do so.  Functions that return object references generally return ``NULL`` only
+to indicate that an exception occurred.  The reason for not testing for ``NULL``
 arguments is that functions often pass the objects they receive on to other
-function --- if each function were to test for *NULL*, there would be a lot of
+function --- if each function were to test for ``NULL``, there would be a lot of
 redundant tests and the code would run more slowly.
 
-It is better to test for *NULL* only at the "source:" when a pointer that may be
-*NULL* is received, for example, from :c:func:`malloc` or from a function that
+It is better to test for ``NULL`` only at the "source:" when a pointer that may be
+``NULL`` is received, for example, from :c:func:`malloc` or from a function that
 may raise an exception.
 
-The macros :c:func:`Py_INCREF` and :c:func:`Py_DECREF` do not check for *NULL*
+The macros :c:func:`Py_INCREF` and :c:func:`Py_DECREF` do not check for ``NULL``
 pointers --- however, their variants :c:func:`Py_XINCREF` and :c:func:`Py_XDECREF`
 do.
 
 The macros for checking for a particular object type (``Pytype_Check()``) don't
-check for *NULL* pointers --- again, there is much code that calls several of
+check for ``NULL`` pointers --- again, there is much code that calls several of
 these in a row to test an object against various different expected types, and
-this would generate redundant tests.  There are no variants with *NULL*
+this would generate redundant tests.  There are no variants with ``NULL``
 checking.
 
 The C function calling mechanism guarantees that the argument list passed to C
-functions (``args`` in the examples) is never *NULL* --- in fact it guarantees
+functions (``args`` in the examples) is never ``NULL`` --- in fact it guarantees
 that it is always a tuple [#]_.
 
-It is a severe error to ever let a *NULL* pointer "escape" to the Python user.
+It is a severe error to ever let a ``NULL`` pointer "escape" to the Python user.
 
 .. Frank Stajano:
    A pedagogically buggy example, along the lines of the previous listing, would
@@ -1184,7 +1184,7 @@ different ways between the module providing the code and the client modules.
 
 Whichever method you choose, it's important to name your Capsules properly.
 The function :c:func:`PyCapsule_New` takes a name parameter
-(:c:type:`const char \*`); you're permitted to pass in a *NULL* name, but
+(:c:type:`const char \*`); you're permitted to pass in a ``NULL`` name, but
 we strongly encourage you to specify a name.  Properly named Capsules provide
 a degree of runtime type-safety; there is no feasible way to tell one unnamed
 Capsule from another.
diff --git a/Doc/extending/newtypes.rst b/Doc/extending/newtypes.rst
index 7e0b18d6299fb..d9023709fddc8 100644
--- a/Doc/extending/newtypes.rst
+++ b/Doc/extending/newtypes.rst
@@ -191,7 +191,7 @@ For every object which can support attributes, the corresponding type must
 provide the functions that control how the attributes are resolved.  There needs
 to be a function which can retrieve attributes (if any are defined), and another
 to set attributes (if setting attributes is allowed).  Removing an attribute is
-a special case, for which the new value passed to the handler is *NULL*.
+a special case, for which the new value passed to the handler is ``NULL``.
 
 Python supports two pairs of attribute handlers; a type that supports attributes
 only needs to implement the functions for one pair.  The difference is that one
@@ -233,9 +233,9 @@ attributes, when the values are computed, or how relevant data is stored.
 When :c:func:`PyType_Ready` is called, it uses three tables referenced by the
 type object to create :term:`descriptor`\s which are placed in the dictionary of the
 type object.  Each descriptor controls access to one attribute of the instance
-object.  Each of the tables is optional; if all three are *NULL*, instances of
+object.  Each of the tables is optional; if all three are ``NULL``, instances of
 the type will only have attributes that are inherited from their base type, and
-should leave the :c:member:`~PyTypeObject.tp_getattro` and :c:member:`~PyTypeObject.tp_setattro` fields *NULL* as
+should leave the :c:member:`~PyTypeObject.tp_getattro` and :c:member:`~PyTypeObject.tp_setattro` fields ``NULL`` as
 well, allowing the base type to handle attributes.
 
 The tables are declared as three fields of the type object::
@@ -244,7 +244,7 @@ The tables are declared as three fields of the type object::
    struct PyMemberDef *tp_members;
    struct PyGetSetDef *tp_getset;
 
-If :c:member:`~PyTypeObject.tp_methods` is not *NULL*, it must refer to an array of
+If :c:member:`~PyTypeObject.tp_methods` is not ``NULL``, it must refer to an array of
 :c:type:`PyMethodDef` structures.  Each entry in the table is an instance of this
 structure::
 
@@ -258,7 +258,7 @@ structure::
 One entry should be defined for each method provided by the type; no entries are
 needed for methods inherited from a base type.  One additional entry is needed
 at the end; it is a sentinel that marks the end of the array.  The
-:attr:`ml_name` field of the sentinel must be *NULL*.
+:attr:`ml_name` field of the sentinel must be ``NULL``.
 
 The second table is used to define attributes which map directly to data stored
 in the instance.  A variety of primitive C types are supported, and access may
@@ -307,7 +307,7 @@ application can use the introspection API to retrieve the descriptor from the
 class object, and get the doc string using its :attr:`__doc__` attribute.
 
 As with the :c:member:`~PyTypeObject.tp_methods` table, a sentinel entry with a :attr:`name` value
-of *NULL* is required.
+of ``NULL`` is required.
 
 .. XXX Descriptors need to be explained in more detail somewhere, but not here.
 
@@ -352,9 +352,9 @@ Here is an example::
 
 The :c:member:`~PyTypeObject.tp_setattr` handler is called when the :meth:`__setattr__` or
 :meth:`__delattr__` method of a class instance would be called.  When an
-attribute should be deleted, the third parameter will be *NULL*.  Here is an
+attribute should be deleted, the third parameter will be ``NULL``.  Here is an
 example that simply raises an exception; if this were really all you wanted, the
-:c:member:`~PyTypeObject.tp_setattr` handler should be set to *NULL*. ::
+:c:member:`~PyTypeObject.tp_setattr` handler should be set to ``NULL``. ::
 
    static int
    newdatatype_setattr(newdatatypeobject *obj, char *name, PyObject *v)
@@ -380,7 +380,7 @@ where the operator is one of ``Py_EQ``, ``Py_NE``, ``Py_LE``, ``Py_GT``,
 ``Py_LT`` or ``Py_GT``.  It should compare the two objects with respect to the
 specified operator and return ``Py_True`` or ``Py_False`` if the comparison is
 successful, ``Py_NotImplemented`` to indicate that comparison is not
-implemented and the other object's comparison method should be tried, or *NULL*
+implemented and the other object's comparison method should be tried, or ``NULL``
 if an exception was set.
 
 Here is a sample implementation, for a datatype that is considered equal if the
@@ -427,7 +427,7 @@ from the type implementation, the older protocols have been defined as optional
 blocks of handlers referenced by the type object.  For newer protocols there are
 additional slots in the main type object, with a flag bit being set to indicate
 that the slots are present and should be checked by the interpreter.  (The flag
-bit does not indicate that the slot values are non-*NULL*. The flag may be set
+bit does not indicate that the slot values are non-``NULL``. The flag may be set
 to indicate the presence of a slot, but a slot may still be unfilled.) ::
 
    PyNumberMethods   *tp_as_number;
@@ -478,9 +478,9 @@ This function takes three arguments:
    :c:func:`PyArg_ParseTuple` to extract the arguments.
 
 #. *kwds* is a dictionary of keyword arguments that were passed. If this is
-   non-*NULL* and you support keyword arguments, use
+   non-``NULL`` and you support keyword arguments, use
    :c:func:`PyArg_ParseTupleAndKeywords` to extract the arguments.  If you
-   do not want to support keyword arguments and this is non-*NULL*, raise a
+   do not want to support keyword arguments and this is non-``NULL``, raise a
    :exc:`TypeError` with a message saying that keyword arguments are not supported.
 
 Here is a toy ``tp_call`` implementation::
@@ -512,7 +512,7 @@ Here is a toy ``tp_call`` implementation::
 These functions provide support for the iterator protocol.  Both handlers
 take exactly one parameter, the instance for which they are being called,
 and return a new reference.  In the case of an error, they should set an
-exception and return *NULL*.  :c:member:`~PyTypeObject.tp_iter` corresponds
+exception and return ``NULL``.  :c:member:`~PyTypeObject.tp_iter` corresponds
 to the Python :meth:`__iter__` method, while :c:member:`~PyTypeObject.tp_iternext`
 corresponds to the Python :meth:`~iterator.__next__` method.
 
@@ -534,11 +534,11 @@ and :c:member:`~PyTypeObject.tp_iternext`.  An iterator's
 to the iterator.  Its :c:member:`~PyTypeObject.tp_iternext` handler should
 return a new reference to the next object in the iteration, if there is one.
 If the iteration has reached the end, :c:member:`~PyTypeObject.tp_iternext`
-may return *NULL* without setting an exception, or it may set
-:exc:`StopIteration` *in addition* to returning *NULL*; avoiding
+may return ``NULL`` without setting an exception, or it may set
+:exc:`StopIteration` *in addition* to returning ``NULL``; avoiding
 the exception can yield slightly better performance.  If an actual error
 occurs, :c:member:`~PyTypeObject.tp_iternext` should always set an exception
-and return *NULL*.
+and return ``NULL``.
 
 
 .. _weakref-support:
@@ -557,7 +557,7 @@ For an object to be weakly referencable, the extension type must do two things:
 
 #. Include a :c:type:`PyObject\*` field in the C object structure dedicated to
    the weak reference mechanism.  The object's constructor should leave it
-   *NULL* (which is automatic when using the default
+   ``NULL`` (which is automatic when using the default
    :c:member:`~PyTypeObject.tp_alloc`).
 
 #. Set the :c:member:`~PyTypeObject.tp_weaklistoffset` type member
@@ -582,7 +582,7 @@ And the corresponding member in the statically-declared type object::
 
 The only further addition is that ``tp_dealloc`` needs to clear any weak
 references (by calling :c:func:`PyObject_ClearWeakRefs`) if the field is
-non-*NULL*::
+non-``NULL``::
 
    static void
    Trivial_dealloc(TrivialObject *self)
diff --git a/Doc/extending/newtypes_tutorial.rst b/Doc/extending/newtypes_tutorial.rst
index b2c819c858237..0eb6ffd026f49 100644
--- a/Doc/extending/newtypes_tutorial.rst
+++ b/Doc/extending/newtypes_tutorial.rst
@@ -177,7 +177,7 @@ Everything else in the file should be familiar, except for some code in
 
 This initializes the :class:`Custom` type, filling in a number of members
 to the appropriate default values, including :attr:`ob_type` that we initially
-set to *NULL*. ::
+set to ``NULL``. ::
 
    Py_INCREF(&CustomType);
    if (PyModule_AddObject(m, "Custom", (PyObject *) &CustomType) < 0) {
@@ -275,7 +275,7 @@ which is assigned to the :c:member:`~PyTypeObject.tp_dealloc` member::
 
 This method first clears the reference counts of the two Python attributes.
 :c:func:`Py_XDECREF` correctly handles the case where its argument is
-*NULL* (which might happen here if ``tp_new`` failed midway).  It then
+``NULL`` (which might happen here if ``tp_new`` failed midway).  It then
 calls the :c:member:`~PyTypeObject.tp_free` member of the object's type
 (computed by ``Py_TYPE(self)``) to free the object's memory.  Note that
 the object's type might not be :class:`CustomType`, because the object may
@@ -321,7 +321,7 @@ objects of the type.  It is exposed in Python as the :meth:`__new__` method.
 It is not required to define a ``tp_new`` member, and indeed many extension
 types will simply reuse :c:func:`PyType_GenericNew` as done in the first
 version of the ``Custom`` type above.  In this case, we use the ``tp_new``
-handler to initialize the ``first`` and ``last`` attributes to non-*NULL*
+handler to initialize the ``first`` and ``last`` attributes to non-``NULL``
 default values.
 
 ``tp_new`` is passed the type being instantiated (not necessarily ``CustomType``,
@@ -341,7 +341,7 @@ slot to allocate memory::
    self = (CustomObject *) type->tp_alloc(type, 0);
 
 Since memory allocation may fail, we must check the :c:member:`~PyTypeObject.tp_alloc`
-result against *NULL* before proceeding.
+result against ``NULL`` before proceeding.
 
 .. note::
    We didn't fill the :c:member:`~PyTypeObject.tp_alloc` slot ourselves. Rather
@@ -455,9 +455,9 @@ below for details.
 A disadvantage of this approach is that it doesn't provide a way to restrict the
 types of objects that can be assigned to the Python attributes.  We expect the
 first and last names to be strings, but any Python objects can be assigned.
-Further, the attributes can be deleted, setting the C pointers to *NULL*.  Even
-though we can make sure the members are initialized to non-*NULL* values, the
-members can be set to *NULL* if the attributes are deleted.
+Further, the attributes can be deleted, setting the C pointers to ``NULL``.  Even
+though we can make sure the members are initialized to non-``NULL`` values, the
+members can be set to ``NULL`` if the attributes are deleted.
 
 We define a single method, :meth:`Custom.name()`, that outputs the objects name as the
 concatenation of the first and last names. ::
@@ -489,8 +489,8 @@ equivalent to the Python method:
        return "%s %s" % (self.first, self.last)
 
 Note that we have to check for the possibility that our :attr:`first` and
-:attr:`last` members are *NULL*.  This is because they can be deleted, in which
-case they are set to *NULL*.  It would be better to prevent deletion of these
+:attr:`last` members are ``NULL``.  This is because they can be deleted, in which
+case they are set to ``NULL``.  It would be better to prevent deletion of these
 attributes and to restrict the attribute values to be strings.  We'll see how to
 do that in the next section.
 
@@ -584,7 +584,7 @@ could, for example, be used to allow a single set of getter and setter functions
 that decide the attribute to get or set based on data in the closure.)
 
 The setter function is passed the :class:`Custom` object, the new value, and the
-closure.  The new value may be *NULL*, in which case the attribute is being
+closure.  The new value may be ``NULL``, in which case the attribute is being
 deleted.  In our setter, we raise an error if the attribute is deleted or if its
 new value is not a string.
 
@@ -603,7 +603,7 @@ and register it in the :c:member:`~PyTypeObject.tp_getset` slot::
    .tp_getset = Custom_getsetters,
 
 The last item in a :c:type:`PyGetSetDef` structure is the "closure" mentioned
-above.  In this case, we aren't using a closure, so we just pass *NULL*.
+above.  In this case, we aren't using a closure, so we just pass ``NULL``.
 
 We also remove the member definitions for these attributes::
 
@@ -643,7 +643,7 @@ allow strings [#]_ to be passed::
    }
 
 With these changes, we can assure that the ``first`` and ``last`` members are
-never *NULL* so we can remove checks for *NULL* values in almost all cases.
+never ``NULL`` so we can remove checks for ``NULL`` values in almost all cases.
 This means that most of the :c:func:`Py_XDECREF` calls can be converted to
 :c:func:`Py_DECREF` calls.  The only place we can't change these calls is in
 the ``tp_dealloc`` implementation, where there is the possibility that the
@@ -749,7 +749,7 @@ participate in cycles::
 Notice the use of the :c:func:`Py_CLEAR` macro.  It is the recommended and safe
 way to clear data attributes of arbitrary types while decrementing
 their reference counts.  If you were to call :c:func:`Py_XDECREF` instead
-on the attribute before setting it to *NULL*, there is a possibility
+on the attribute before setting it to ``NULL``, there is a possibility
 that the attribute's destructor would call back into code that reads the
 attribute again (*especially* if there is a reference cycle).
 
diff --git a/Doc/faq/extending.rst b/Doc/faq/extending.rst
index 2ad2765870525..aecb56eaa4fd2 100644
--- a/Doc/faq/extending.rst
+++ b/Doc/faq/extending.rst
@@ -89,7 +89,7 @@ For bytes, :c:func:`PyBytes_Size` returns its length and
 length.  Note that Python bytes objects may contain null bytes so C's
 :c:func:`strlen` should not be used.
 
-To test the type of an object, first make sure it isn't *NULL*, and then use
+To test the type of an object, first make sure it isn't ``NULL``, and then use
 :c:func:`PyBytes_Check`, :c:func:`PyTuple_Check`, :c:func:`PyList_Check`, etc.
 
 There is also a high-level API to Python objects which is provided by the
diff --git a/Doc/library/ctypes.rst b/Doc/library/ctypes.rst
index eb0bcf4ae2e36..4504a800c43ca 100644
--- a/Doc/library/ctypes.rst
+++ b/Doc/library/ctypes.rst
@@ -1083,7 +1083,7 @@ An extended example which also demonstrates the use of pointers accesses the
 Quoting the docs for that value:
 
    This pointer is initialized to point to an array of :c:type:`struct _frozen`
-   records, terminated by one whose members are all *NULL* or zero.  When a frozen
+   records, terminated by one whose members are all ``NULL`` or zero.  When a frozen
    module is imported, it is searched in this table.  Third-party code could play
    tricks with this to provide a dynamically created collection of frozen modules.
 
diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst
index 822ba81be8a7f..7b886e7c27a48 100644
--- a/Doc/whatsnew/3.4.rst
+++ b/Doc/whatsnew/3.4.rst
@@ -2503,7 +2503,7 @@ Changes in the C API
 
 * The result of the :c:data:`PyOS_ReadlineFunctionPointer` callback must
   now be a string allocated by :c:func:`PyMem_RawMalloc` or
-  :c:func:`PyMem_RawRealloc`, or *NULL* if an error occurred, instead of a
+  :c:func:`PyMem_RawRealloc`, or ``NULL`` if an error occurred, instead of a
   string allocated by :c:func:`PyMem_Malloc` or :c:func:`PyMem_Realloc`
   (:issue:`16742`)
 
diff --git a/Doc/whatsnew/3.7.rst b/Doc/whatsnew/3.7.rst
index 1ec8e0c04c76b..2197bd9739299 100644
--- a/Doc/whatsnew/3.7.rst
+++ b/Doc/whatsnew/3.7.rst
@@ -1707,7 +1707,7 @@ The type of results of :c:func:`PyThread_start_new_thread` and
 (Contributed by Serhiy Storchaka in :issue:`6532`.)
 
 :c:func:`PyUnicode_AsWideCharString` now raises a :exc:`ValueError` if the
-second argument is *NULL* and the :c:type:`wchar_t*` string contains null
+second argument is ``NULL`` and the :c:type:`wchar_t*` string contains null
 characters.  (Contributed by Serhiy Storchaka in :issue:`30708`.)
 
 Changes to the startup sequence and the management of dynamic memory
diff --git a/Include/abstract.h b/Include/abstract.h
index 8951f638a4ec9..bb51c668ac698 100644
--- a/Include/abstract.h
+++ b/Include/abstract.h
@@ -150,7 +150,7 @@ PyAPI_FUNC(PyObject *) PyObject_CallNoArgs(PyObject *func);
 /* Call a callable Python object 'callable' with arguments given by the
    tuple 'args' and keywords arguments given by the dictionary 'kwargs'.
 
-   'args' must not be *NULL*, use an empty tuple if no arguments are
+   'args' must not be NULL, use an empty tuple if no arguments are
    needed. If no named arguments are needed, 'kwargs' can be NULL.
 
    This is the equivalent of the Python expression:
@@ -160,9 +160,9 @@ PyAPI_FUNC(PyObject *) PyObject_Call(PyObject *callable,
 
 
 /* Call a callable Python object 'callable', with arguments given by the
-   tuple 'args'.  If no arguments are needed, then 'args' can be *NULL*.
+   tuple 'args'.  If no arguments are needed, then 'args' can be NULL.
 
-   Returns the result of the call on success, or *NULL* on failure.
+   Returns the result of the call on success, or NULL on failure.
 
    This is the equivalent of the Python expression:
    callable(*args). */


From webhook-mailer at python.org  Wed Oct 30 06:03:57 2019
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Wed, 30 Oct 2019 10:03:57 -0000
Subject: [Python-checkins] bpo-28029: Make "".replace("", s,
 n) returning s for any n != 0. (GH-16981)
Message-ID: <mailman.185.1572429838.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/865c3b257fe38154a4320c7ee6afb416f665b9c2
commit: 865c3b257fe38154a4320c7ee6afb416f665b9c2
branch: master
author: Serhiy Storchaka <storchaka at gmail.com>
committer: GitHub <noreply at github.com>
date: 2019-10-30T12:03:53+02:00
summary:

bpo-28029: Make "".replace("", s, n) returning s for any n != 0. (GH-16981)

files:
A Misc/NEWS.d/next/Core and Builtins/2019-10-29-09-38-54.bpo-28029.AmRMEF.rst
M Doc/whatsnew/3.9.rst
M Lib/test/string_tests.py
M Objects/stringlib/transmogrify.h
M Objects/unicodeobject.c

diff --git a/Doc/whatsnew/3.9.rst b/Doc/whatsnew/3.9.rst
index bca222955668d..d41e70808d6e1 100644
--- a/Doc/whatsnew/3.9.rst
+++ b/Doc/whatsnew/3.9.rst
@@ -99,6 +99,11 @@ Other Language Changes
   ignored for empty strings.
   (Contributed by Victor Stinner in :issue:`37388`.)
 
+* ``"".replace("", s, n)`` now returns ``s`` instead of an empty string for
+  all non-zero ``n``.  It is now consistent with ``"".replace("", s)``.
+  There are similar changes for :class:`bytes` and :class:`bytearray` objects.
+  (Contributed by Serhiy Storchaka in :issue:`28029`.)
+
 
 New Modules
 ===========
diff --git a/Lib/test/string_tests.py b/Lib/test/string_tests.py
index 38da941f7f50e..948e2a3c3c56b 100644
--- a/Lib/test/string_tests.py
+++ b/Lib/test/string_tests.py
@@ -505,6 +505,7 @@ def test_replace(self):
         EQ("", "", "replace", "A", "")
         EQ("", "", "replace", "A", "A")
         EQ("", "", "replace", "", "", 100)
+        EQ("A", "", "replace", "", "A", 100)
         EQ("", "", "replace", "", "", sys.maxsize)
 
         # interleave (from=="", 'to' gets inserted everywhere)
diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-10-29-09-38-54.bpo-28029.AmRMEF.rst b/Misc/NEWS.d/next/Core and Builtins/2019-10-29-09-38-54.bpo-28029.AmRMEF.rst
new file mode 100644
index 0000000000000..420e53aec9be0
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2019-10-29-09-38-54.bpo-28029.AmRMEF.rst	
@@ -0,0 +1,3 @@
+``"".replace("", s, n)`` now returns ``s`` instead of an empty string for
+all non-zero ``n``.  There are similar changes for :class:`bytes` and
+:class:`bytearray` objects.
diff --git a/Objects/stringlib/transmogrify.h b/Objects/stringlib/transmogrify.h
index 9506019d5af2b..e1165ea38e825 100644
--- a/Objects/stringlib/transmogrify.h
+++ b/Objects/stringlib/transmogrify.h
@@ -680,9 +680,13 @@ stringlib_replace(PyObject *self,
                   const char *to_s, Py_ssize_t to_len,
                   Py_ssize_t maxcount)
 {
+    if (STRINGLIB_LEN(self) < from_len) {
+        /* nothing to do; return the original bytes */
+        return return_self(self);
+    }
     if (maxcount < 0) {
         maxcount = PY_SSIZE_T_MAX;
-    } else if (maxcount == 0 || STRINGLIB_LEN(self) == 0) {
+    } else if (maxcount == 0) {
         /* nothing to do; return the original bytes */
         return return_self(self);
     }
@@ -699,13 +703,6 @@ stringlib_replace(PyObject *self,
         return stringlib_replace_interleave(self, to_s, to_len, maxcount);
     }
 
-    /* Except for b"".replace(b"", b"A") == b"A" there is no way beyond this */
-    /* point for an empty self bytes to generate a non-empty bytes */
-    /* Special case so the remaining code always gets a non-empty bytes */
-    if (STRINGLIB_LEN(self) == 0) {
-        return return_self(self);
-    }
-
     if (to_len == 0) {
         /* delete all occurrences of 'from' bytes */
         if (from_len == 1) {
diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c
index 2d60627b19193..5ae0af8ea336a 100644
--- a/Objects/unicodeobject.c
+++ b/Objects/unicodeobject.c
@@ -10572,9 +10572,12 @@ replace(PyObject *self, PyObject *str1,
     int mayshrink;
     Py_UCS4 maxchar, maxchar_str1, maxchar_str2;
 
+    if (slen < len1)
+        goto nothing;
+
     if (maxcount < 0)
         maxcount = PY_SSIZE_T_MAX;
-    else if (maxcount == 0 || slen == 0)
+    else if (maxcount == 0)
         goto nothing;
 
     if (str1 == str2)


From webhook-mailer at python.org  Wed Oct 30 07:41:47 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Wed, 30 Oct 2019 11:41:47 -0000
Subject: [Python-checkins] bpo-38614: Add timeout constants to test.support
 (GH-16964)
Message-ID: <mailman.186.1572435708.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/24c6258269acd842914450f55491690ba87dded9
commit: 24c6258269acd842914450f55491690ba87dded9
branch: master
author: Victor Stinner <vstinner at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-30T12:41:43+01:00
summary:

bpo-38614: Add timeout constants to test.support (GH-16964)

Add timeout constants to test.support:

* LOOPBACK_TIMEOUT
* INTERNET_TIMEOUT
* SHORT_TIMEOUT
* LONG_TIMEOUT

files:
A Misc/NEWS.d/next/Tests/2019-10-28-15-56-02.bpo-38614.aDdDYE.rst
M Doc/library/test.rst
M Lib/test/_test_multiprocessing.py
M Lib/test/libregrtest/setup.py
M Lib/test/support/__init__.py
M Lib/test/test_socket.py

diff --git a/Doc/library/test.rst b/Doc/library/test.rst
index 5dde55cdf951a..4a61566c2239a 100644
--- a/Doc/library/test.rst
+++ b/Doc/library/test.rst
@@ -287,6 +287,67 @@ The :mod:`test.support` module defines the following constants:
    Set to a filename containing the :data:`FS_NONASCII` character.
 
 
+.. data:: LOOPBACK_TIMEOUT
+
+   Timeout in seconds for tests using a network server listening on the network
+   local loopback interface like ``127.0.0.1``.
+
+   The timeout is long enough to prevent test failure: it takes into account
+   that the client and the server can run in different threads or even
+   different processes.
+
+   The timeout should be long enough for :meth:`~socket.socket.connect`,
+   :meth:`~socket.socket.recv` and :meth:`~socket.socket.send` methods of
+   :class:`socket.socket`.
+
+   Its default value is 5 seconds.
+
+   See also :data:`INTERNET_TIMEOUT`.
+
+
+.. data:: INTERNET_TIMEOUT
+
+   Timeout in seconds for network requests going to the Internet.
+
+   The timeout is short enough to prevent a test to wait for too long if the
+   Internet request is blocked for whatever reason.
+
+   Usually, a timeout using :data:`INTERNET_TIMEOUT` should not mark a test as
+   failed, but skip the test instead: see
+   :func:`~test.support.transient_internet`.
+
+   Its default value is 1 minute.
+
+   See also :data:`LOOPBACK_TIMEOUT`.
+
+
+.. data:: SHORT_TIMEOUT
+
+   Timeout in seconds to mark a test as failed if the test takes "too long".
+
+   The timeout value depends on the regrtest ``--timeout`` command line option.
+
+   If a test using :data:`SHORT_TIMEOUT` starts to fail randomly on slow
+   buildbots, use :data:`LONG_TIMEOUT` instead.
+
+   Its default value is 30 seconds.
+
+
+.. data:: LONG_TIMEOUT
+
+   Timeout in seconds to detect when a test hangs.
+
+   It is long enough to reduce the risk of test failure on the slowest Python
+   buildbots. It should not be used to mark a test as failed if the test takes
+   "too long".  The timeout value depends on the regrtest ``--timeout`` command
+   line option.
+
+   Its default value is 5 minutes.
+
+   See also :data:`LOOPBACK_TIMEOUT`, :data:`INTERNET_TIMEOUT` and
+   :data:`SHORT_TIMEOUT`.
+
+
 .. data:: IPV6_ENABLED
 
     Set to ``True`` if IPV6 is enabled on this host, ``False`` otherwise.
diff --git a/Lib/test/_test_multiprocessing.py b/Lib/test/_test_multiprocessing.py
index bfaa02b0fcc88..f7bebc6e8370b 100644
--- a/Lib/test/_test_multiprocessing.py
+++ b/Lib/test/_test_multiprocessing.py
@@ -66,12 +66,6 @@
 except ImportError:
     msvcrt = None
 
-#
-#
-#
-
-# Timeout to wait until a process completes
-TIMEOUT = 60.0 # seconds
 
 def latin(s):
     return s.encode('latin')
@@ -86,7 +80,7 @@ def close_queue(queue):
 def join_process(process):
     # Since multiprocessing.Process has the same API than threading.Thread
     # (join() and is_alive(), the support function can be reused
-    support.join_thread(process, timeout=TIMEOUT)
+    support.join_thread(process)
 
 
 if os.name == "posix":
@@ -1128,7 +1122,7 @@ def __reduce__(self):
             q = self.Queue()
             q.put(NotSerializable())
             q.put(True)
-            self.assertTrue(q.get(timeout=TIMEOUT))
+            self.assertTrue(q.get(timeout=support.LONG_TIMEOUT))
             close_queue(q)
 
         with test.support.captured_stderr():
@@ -1531,7 +1525,7 @@ def test_waitfor_timeout(self):
                          args=(cond, state, success, sem))
         p.daemon = True
         p.start()
-        self.assertTrue(sem.acquire(timeout=TIMEOUT))
+        self.assertTrue(sem.acquire(timeout=support.LONG_TIMEOUT))
 
         # Only increment 3 times, so state == 4 is never reached.
         for i in range(3):
@@ -3388,7 +3382,7 @@ class _TestPicklingConnections(BaseTestCase):
     @classmethod
     def tearDownClass(cls):
         from multiprocessing import resource_sharer
-        resource_sharer.stop(timeout=TIMEOUT)
+        resource_sharer.stop(timeout=support.LONG_TIMEOUT)
 
     @classmethod
     def _listener(cls, conn, families):
@@ -4033,7 +4027,7 @@ def test_shared_memory_cleaned_after_process_termination(self):
             p.terminate()
             p.wait()
 
-            deadline = time.monotonic() + 60
+            deadline = time.monotonic() + support.LONG_TIMEOUT
             t = 0.1
             while time.monotonic() < deadline:
                 time.sleep(t)
@@ -5040,7 +5034,7 @@ def create_and_register_resource(rtype):
                 p.terminate()
                 p.wait()
 
-                deadline = time.monotonic() + 60
+                deadline = time.monotonic() + support.LONG_TIMEOUT
                 while time.monotonic() < deadline:
                     time.sleep(.5)
                     try:
diff --git a/Lib/test/libregrtest/setup.py b/Lib/test/libregrtest/setup.py
index 5254e7fc6fe0c..ce8149677e0b9 100644
--- a/Lib/test/libregrtest/setup.py
+++ b/Lib/test/libregrtest/setup.py
@@ -81,6 +81,17 @@ def _test_audit_hook(name, args):
 
     setup_unraisable_hook()
 
+    if ns.timeout is not None:
+        # For a slow buildbot worker, increase SHORT_TIMEOUT and LONG_TIMEOUT
+        support.SHORT_TIMEOUT = max(support.SHORT_TIMEOUT, ns.timeout / 40)
+        support.LONG_TIMEOUT = max(support.LONG_TIMEOUT, ns.timeout / 4)
+
+        # If --timeout is short: reduce timeouts
+        support.LOOPBACK_TIMEOUT = min(support.LOOPBACK_TIMEOUT, ns.timeout)
+        support.INTERNET_TIMEOUT = min(support.INTERNET_TIMEOUT, ns.timeout)
+        support.SHORT_TIMEOUT = min(support.SHORT_TIMEOUT, ns.timeout)
+        support.LONG_TIMEOUT = min(support.LONG_TIMEOUT, ns.timeout)
+
 
 def suppress_msvcrt_asserts(verbose):
     try:
diff --git a/Lib/test/support/__init__.py b/Lib/test/support/__init__.py
index 0f294c5b0f456..5ad32b8f4081c 100644
--- a/Lib/test/support/__init__.py
+++ b/Lib/test/support/__init__.py
@@ -119,9 +119,53 @@
     "run_with_locale", "swap_item",
     "swap_attr", "Matcher", "set_memlimit", "SuppressCrashReport", "sortdict",
     "run_with_tz", "PGO", "missing_compiler_executable", "fd_count",
-    "ALWAYS_EQ", "NEVER_EQ", "LARGEST", "SMALLEST"
+    "ALWAYS_EQ", "NEVER_EQ", "LARGEST", "SMALLEST",
+    "LOOPBACK_TIMEOUT", "INTERNET_TIMEOUT", "SHORT_TIMEOUT", "LONG_TIMEOUT",
     ]
 
+
+# Timeout in seconds for tests using a network server listening on the network
+# local loopback interface like 127.0.0.1.
+#
+# The timeout is long enough to prevent test failure: it takes into account
+# that the client and the server can run in different threads or even different
+# processes.
+#
+# The timeout should be long enough for connect(), recv() and send() methods
+# of socket.socket.
+LOOPBACK_TIMEOUT = 5.0
+if sys.platform == 'win32' and platform.machine() == 'ARM':
+    # bpo-37553: test_socket.SendfileUsingSendTest is taking longer than 2
+    # seconds on Windows ARM32 buildbot
+    LOOPBACK_TIMEOUT = 10
+
+# Timeout in seconds for network requests going to the Internet. The timeout is
+# short enough to prevent a test to wait for too long if the Internet request
+# is blocked for whatever reason.
+#
+# Usually, a timeout using INTERNET_TIMEOUT should not mark a test as failed,
+# but skip the test instead: see transient_internet().
+INTERNET_TIMEOUT = 60.0
+
+# Timeout in seconds to mark a test as failed if the test takes "too long".
+#
+# The timeout value depends on the regrtest --timeout command line option.
+#
+# If a test using SHORT_TIMEOUT starts to fail randomly on slow buildbots, use
+# LONG_TIMEOUT instead.
+SHORT_TIMEOUT = 30.0
+
+# Timeout in seconds to detect when a test hangs.
+#
+# It is long enough to reduce the risk of test failure on the slowest Python
+# buildbots. It should not be used to mark a test as failed if the test takes
+# "too long". The timeout value depends on the regrtest --timeout command line
+# option.
+LONG_TIMEOUT = 5 * 60.0
+
+_NOT_SET = object()
+
+
 class Error(Exception):
     """Base class for regression test exceptions."""
 
@@ -1231,7 +1275,7 @@ def check_valid_file(fn):
     opener = urllib.request.build_opener()
     if gzip:
         opener.addheaders.append(('Accept-Encoding', 'gzip'))
-    f = opener.open(url, timeout=15)
+    f = opener.open(url, timeout=INTERNET_TIMEOUT)
     if gzip and f.headers.get('Content-Encoding') == 'gzip':
         f = gzip.GzipFile(fileobj=f)
     try:
@@ -1542,9 +1586,12 @@ def get_socket_conn_refused_errs():
 
 
 @contextlib.contextmanager
-def transient_internet(resource_name, *, timeout=30.0, errnos=()):
+def transient_internet(resource_name, *, timeout=_NOT_SET, errnos=()):
     """Return a context manager that raises ResourceDenied when various issues
     with the Internet connection manifest themselves as exceptions."""
+    if timeout is _NOT_SET:
+        timeout = INTERNET_TIMEOUT
+
     default_errnos = [
         ('ECONNREFUSED', 111),
         ('ECONNRESET', 104),
@@ -2264,7 +2311,7 @@ def decorator(*args):
 
 
 @contextlib.contextmanager
-def wait_threads_exit(timeout=60.0):
+def wait_threads_exit(timeout=None):
     """
     bpo-31234: Context manager to wait until all threads created in the with
     statement exit.
@@ -2278,6 +2325,8 @@ def wait_threads_exit(timeout=60.0):
     which doesn't allow to wait for thread exit, whereas thread.Thread has a
     join() method.
     """
+    if timeout is None:
+        timeout = SHORT_TIMEOUT
     old_count = _thread._count()
     try:
         yield
@@ -2298,10 +2347,12 @@ def wait_threads_exit(timeout=60.0):
             gc_collect()
 
 
-def join_thread(thread, timeout=30.0):
+def join_thread(thread, timeout=None):
     """Join a thread. Raise an AssertionError if the thread is still alive
     after timeout seconds.
     """
+    if timeout is None:
+        timeout = SHORT_TIMEOUT
     thread.join(timeout)
     if thread.is_alive():
         msg = f"failed to join the thread in {timeout:.1f} seconds"
diff --git a/Lib/test/test_socket.py b/Lib/test/test_socket.py
index b74549024b789..184c67c56ebd0 100755
--- a/Lib/test/test_socket.py
+++ b/Lib/test/test_socket.py
@@ -37,7 +37,6 @@
 HOST = support.HOST
 # test unicode string and carriage return
 MSG = 'Michael Gilfix was here\u1234\r\n'.encode('utf-8')
-MAIN_TIMEOUT = 60.0
 
 VSOCKPORT = 1234
 AIX = platform.system() == "AIX"
@@ -2527,7 +2526,7 @@ class SendrecvmsgBase(ThreadSafeCleanupTestCase):
 
     # Time in seconds to wait before considering a test failed, or
     # None for no timeout.  Not all tests actually set a timeout.
-    fail_timeout = 3.0
+    fail_timeout = support.LOOPBACK_TIMEOUT
 
     def setUp(self):
         self.misc_event = threading.Event()
@@ -4320,7 +4319,7 @@ def setUp(self):
         self.addCleanup(signal.signal, signal.SIGALRM, orig_alrm_handler)
 
     # Timeout for socket operations
-    timeout = 4.0
+    timeout = support.LOOPBACK_TIMEOUT
 
     # Provide setAlarm() method to schedule delivery of SIGALRM after
     # given number of seconds, or cancel it if zero, and an
@@ -4610,7 +4609,7 @@ def testAccept(self):
 
         self.event.set()
 
-        read, write, err = select.select([self.serv], [], [], MAIN_TIMEOUT)
+        read, write, err = select.select([self.serv], [], [], support.LONG_TIMEOUT)
         if self.serv not in read:
             self.fail("Error trying to do accept after select.")
 
@@ -4638,7 +4637,7 @@ def testRecv(self):
 
         self.event.set()
 
-        read, write, err = select.select([conn], [], [], MAIN_TIMEOUT)
+        read, write, err = select.select([conn], [], [], support.LONG_TIMEOUT)
         if conn not in read:
             self.fail("Error during select call to non-blocking socket.")
 
@@ -5838,8 +5837,7 @@ class SendfileUsingSendTest(ThreadedTCPSocketTest):
     FILESIZE = (10 * 1024 * 1024)  # 10 MiB
     BUFSIZE = 8192
     FILEDATA = b""
-    # bpo-37553: This is taking longer than 2 seconds on Windows ARM32 buildbot
-    TIMEOUT = 10 if sys.platform == 'win32' and platform.machine() == 'ARM' else 2
+    TIMEOUT = support.LOOPBACK_TIMEOUT
 
     @classmethod
     def setUpClass(cls):
@@ -5865,7 +5863,7 @@ def tearDownClass(cls):
         support.unlink(support.TESTFN)
 
     def accept_conn(self):
-        self.serv.settimeout(MAIN_TIMEOUT)
+        self.serv.settimeout(support.LONG_TIMEOUT)
         conn, addr = self.serv.accept()
         conn.settimeout(self.TIMEOUT)
         self.addCleanup(conn.close)
@@ -6369,7 +6367,7 @@ def test_dualstack_ipv6_family(self):
 
 
 class CreateServerFunctionalTest(unittest.TestCase):
-    timeout = 3
+    timeout = support.LOOPBACK_TIMEOUT
 
     def setUp(self):
         self.thread = None
diff --git a/Misc/NEWS.d/next/Tests/2019-10-28-15-56-02.bpo-38614.aDdDYE.rst b/Misc/NEWS.d/next/Tests/2019-10-28-15-56-02.bpo-38614.aDdDYE.rst
new file mode 100644
index 0000000000000..42ff853a12be1
--- /dev/null
+++ b/Misc/NEWS.d/next/Tests/2019-10-28-15-56-02.bpo-38614.aDdDYE.rst
@@ -0,0 +1,4 @@
+Add timeout constants to :mod:`test.support`:
+:data:`~test.support.LOOPBACK_TIMEOUT`,
+:data:`~test.support.INTERNET_TIMEOUT`, :data:`~test.support.SHORT_TIMEOUT`
+and :data:`~test.support.LONG_TIMEOUT`.


From webhook-mailer at python.org  Wed Oct 30 07:53:30 2019
From: webhook-mailer at python.org (Pablo Galindo)
Date: Wed, 30 Oct 2019 11:53:30 -0000
Subject: [Python-checkins] bpo-38640: Allow break and continue in always
 false while loops (GH-16992)
Message-ID: <mailman.187.1572436411.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/6c3e66a34b95fff07df0ad5086104dd637a091ce
commit: 6c3e66a34b95fff07df0ad5086104dd637a091ce
branch: master
author: Pablo Galindo <Pablogsal at gmail.com>
committer: GitHub <noreply at github.com>
date: 2019-10-30T11:53:26Z
summary:

bpo-38640: Allow break and continue in always false while loops (GH-16992)

files:
A Misc/NEWS.d/next/Core and Builtins/2019-10-30-11-25-25.bpo-38640.4sAFh5.rst
M Lib/test/test_compile.py
M Python/compile.c

diff --git a/Lib/test/test_compile.py b/Lib/test/test_compile.py
index 9d77f7af05d6f..566ca27fca893 100644
--- a/Lib/test/test_compile.py
+++ b/Lib/test/test_compile.py
@@ -731,6 +731,24 @@ def unused_block_while_else():
             self.assertEqual(None, opcodes[0].argval)
             self.assertEqual('RETURN_VALUE', opcodes[1].opname)
 
+    def test_false_while_loop(self):
+        def break_in_while():
+            while False:
+                break
+
+        def continue_in_while():
+            while False:
+                continue
+
+        funcs = [break_in_while, continue_in_while]
+
+        # Check that we did not raise but we also don't generate bytecode
+        for func in funcs:
+            opcodes = list(dis.get_instructions(func))
+            self.assertEqual(2, len(opcodes))
+            self.assertEqual('LOAD_CONST', opcodes[0].opname)
+            self.assertEqual(None, opcodes[0].argval)
+            self.assertEqual('RETURN_VALUE', opcodes[1].opname)
 
 class TestExpressionStackSize(unittest.TestCase):
     # These tests check that the computed stack size for a code object
diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-10-30-11-25-25.bpo-38640.4sAFh5.rst b/Misc/NEWS.d/next/Core and Builtins/2019-10-30-11-25-25.bpo-38640.4sAFh5.rst
new file mode 100644
index 0000000000000..d99db3cce49a6
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2019-10-30-11-25-25.bpo-38640.4sAFh5.rst	
@@ -0,0 +1,3 @@
+Fixed a bug in the compiler that was causing to raise in the presence of
+break statements and continue statements inside always false while loops.
+Patch by Pablo Galindo.
diff --git a/Python/compile.c b/Python/compile.c
index 3b2188e6e7438..f26ad3c775255 100644
--- a/Python/compile.c
+++ b/Python/compile.c
@@ -2738,7 +2738,15 @@ compiler_while(struct compiler *c, stmt_ty s)
 
     if (constant == 0) {
         BEGIN_DO_NOT_EMIT_BYTECODE
+        // Push a dummy block so the VISIT_SEQ knows that we are
+        // inside a while loop so it can correctly evaluate syntax
+        // errors.
+        if (!compiler_push_fblock(c, WHILE_LOOP, NULL, NULL)) {
+            return 0;
+        }
         VISIT_SEQ(c, stmt, s->v.While.body);
+        // Remove the dummy block now that is not needed.
+        compiler_pop_fblock(c, WHILE_LOOP, NULL);
         END_DO_NOT_EMIT_BYTECODE
         if (s->v.While.orelse) {
             VISIT_SEQ(c, stmt, s->v.While.orelse);


From webhook-mailer at python.org  Wed Oct 30 08:11:44 2019
From: webhook-mailer at python.org (Miss Skeleton (bot))
Date: Wed, 30 Oct 2019 12:11:44 -0000
Subject: [Python-checkins] bpo-38640: Allow break and continue in always
 false while loops (GH-16992)
Message-ID: <mailman.188.1572437506.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/dcb338ea1b8f14e21dde3564658b9040df996349
commit: dcb338ea1b8f14e21dde3564658b9040df996349
branch: 3.8
author: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-30T05:11:41-07:00
summary:

bpo-38640: Allow break and continue in always false while loops (GH-16992)

(cherry picked from commit 6c3e66a34b95fff07df0ad5086104dd637a091ce)

Co-authored-by: Pablo Galindo <Pablogsal at gmail.com>

files:
A Misc/NEWS.d/next/Core and Builtins/2019-10-30-11-25-25.bpo-38640.4sAFh5.rst
M Lib/test/test_compile.py
M Python/compile.c

diff --git a/Lib/test/test_compile.py b/Lib/test/test_compile.py
index 9d77f7af05d6f..566ca27fca893 100644
--- a/Lib/test/test_compile.py
+++ b/Lib/test/test_compile.py
@@ -731,6 +731,24 @@ def unused_block_while_else():
             self.assertEqual(None, opcodes[0].argval)
             self.assertEqual('RETURN_VALUE', opcodes[1].opname)
 
+    def test_false_while_loop(self):
+        def break_in_while():
+            while False:
+                break
+
+        def continue_in_while():
+            while False:
+                continue
+
+        funcs = [break_in_while, continue_in_while]
+
+        # Check that we did not raise but we also don't generate bytecode
+        for func in funcs:
+            opcodes = list(dis.get_instructions(func))
+            self.assertEqual(2, len(opcodes))
+            self.assertEqual('LOAD_CONST', opcodes[0].opname)
+            self.assertEqual(None, opcodes[0].argval)
+            self.assertEqual('RETURN_VALUE', opcodes[1].opname)
 
 class TestExpressionStackSize(unittest.TestCase):
     # These tests check that the computed stack size for a code object
diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-10-30-11-25-25.bpo-38640.4sAFh5.rst b/Misc/NEWS.d/next/Core and Builtins/2019-10-30-11-25-25.bpo-38640.4sAFh5.rst
new file mode 100644
index 0000000000000..d99db3cce49a6
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2019-10-30-11-25-25.bpo-38640.4sAFh5.rst	
@@ -0,0 +1,3 @@
+Fixed a bug in the compiler that was causing to raise in the presence of
+break statements and continue statements inside always false while loops.
+Patch by Pablo Galindo.
diff --git a/Python/compile.c b/Python/compile.c
index a512e9cbd13a1..f09e99fd4a43d 100644
--- a/Python/compile.c
+++ b/Python/compile.c
@@ -2736,7 +2736,15 @@ compiler_while(struct compiler *c, stmt_ty s)
 
     if (constant == 0) {
         BEGIN_DO_NOT_EMIT_BYTECODE
+        // Push a dummy block so the VISIT_SEQ knows that we are
+        // inside a while loop so it can correctly evaluate syntax
+        // errors.
+        if (!compiler_push_fblock(c, WHILE_LOOP, NULL, NULL)) {
+            return 0;
+        }
         VISIT_SEQ(c, stmt, s->v.While.body);
+        // Remove the dummy block now that is not needed.
+        compiler_pop_fblock(c, WHILE_LOOP, NULL);
         END_DO_NOT_EMIT_BYTECODE
         if (s->v.While.orelse) {
             VISIT_SEQ(c, stmt, s->v.While.orelse);


From webhook-mailer at python.org  Wed Oct 30 11:00:48 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Wed, 30 Oct 2019 15:00:48 -0000
Subject: [Python-checkins] bpo-38614: Increase asyncio test_communicate()
 timeout (GH-16995)
Message-ID: <mailman.189.1572447649.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/a4ed6ed9f3779b6eda41bb65f5c09004a2b937ef
commit: a4ed6ed9f3779b6eda41bb65f5c09004a2b937ef
branch: master
author: Victor Stinner <vstinner at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-30T16:00:44+01:00
summary:

bpo-38614: Increase asyncio test_communicate() timeout (GH-16995)

Fix test_communicate() of test_asyncio.test_subprocess: use
support.LONG_TIMEOUT (5 minutes), instead of 1 minute.

files:
A Misc/NEWS.d/next/Tests/2019-10-30-15-12-32.bpo-38614.M6UnLB.rst
M Lib/test/test_asyncio/test_subprocess.py

diff --git a/Lib/test/test_asyncio/test_subprocess.py b/Lib/test/test_asyncio/test_subprocess.py
index fe8cfa61b1b2d..17552d03f5f04 100644
--- a/Lib/test/test_asyncio/test_subprocess.py
+++ b/Lib/test/test_asyncio/test_subprocess.py
@@ -143,7 +143,7 @@ def test_communicate(self):
             return proc.returncode, stdout
 
         task = run(b'some data')
-        task = asyncio.wait_for(task, 60.0)
+        task = asyncio.wait_for(task, support.LONG_TIMEOUT)
         exitcode, stdout = self.loop.run_until_complete(task)
         self.assertEqual(exitcode, 0)
         self.assertEqual(stdout, b'some data')
diff --git a/Misc/NEWS.d/next/Tests/2019-10-30-15-12-32.bpo-38614.M6UnLB.rst b/Misc/NEWS.d/next/Tests/2019-10-30-15-12-32.bpo-38614.M6UnLB.rst
new file mode 100644
index 0000000000000..89f6847e689dd
--- /dev/null
+++ b/Misc/NEWS.d/next/Tests/2019-10-30-15-12-32.bpo-38614.M6UnLB.rst
@@ -0,0 +1,2 @@
+Fix test_communicate() of test_asyncio.test_subprocess: use
+``support.LONG_TIMEOUT`` (5 minutes), instead of just 1 minute.


From webhook-mailer at python.org  Wed Oct 30 11:39:31 2019
From: webhook-mailer at python.org (Victor Stinner)
Date: Wed, 30 Oct 2019 15:39:31 -0000
Subject: [Python-checkins] bpo-38631: Avoid Py_FatalError() in readline
 (GH-16998)
Message-ID: <mailman.190.1572449973.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/1d8da61f5ad26274556e0bbce260ce292d0402a1
commit: 1d8da61f5ad26274556e0bbce260ce292d0402a1
branch: master
author: Victor Stinner <vstinner at python.org>
committer: GitHub <noreply at github.com>
date: 2019-10-30T16:39:27+01:00
summary:

bpo-38631: Avoid Py_FatalError() in readline (GH-16998)

readline now calls PyErr_NoMemory() rather than Py_FatalError() on
memory allocation failure, when importing the module.

files:
M Modules/readline.c

diff --git a/Modules/readline.c b/Modules/readline.c
index 930e63975c5ce..b76861f7b0407 100644
--- a/Modules/readline.c
+++ b/Modules/readline.c
@@ -1066,15 +1066,16 @@ flex_complete(const char *text, int start, int end)
 }
 
 
-/* Helper to initialize GNU readline properly. */
-
-static void
+/* Helper to initialize GNU readline properly.
+   Return -1 on memory allocation failure, return 0 on success. */
+static int
 setup_readline(readlinestate *mod_state)
 {
 #ifdef SAVE_LOCALE
     char *saved_locale = strdup(setlocale(LC_CTYPE, NULL));
-    if (!saved_locale)
-        Py_FatalError("not enough memory to save locale");
+    if (!saved_locale) {
+        return -1;
+    }
 #endif
 
     /* The name must be defined before initialization */
@@ -1156,6 +1157,7 @@ setup_readline(readlinestate *mod_state)
         rl_initialize();
 
     RESTORE_LOCALE(saved_locale)
+    return 0;
 }
 
 /* Wrapper around GNU readline that handles signals differently. */
@@ -1369,7 +1371,10 @@ PyInit_readline(void)
 
     mod_state = (readlinestate *) PyModule_GetState(m);
     PyOS_ReadlineFunctionPointer = call_readline;
-    setup_readline(mod_state);
+    if (setup_readline(mod_state) < 0) {
+        PyErr_NoMemory();
+        goto error;
+    }
 
     return m;
 


From webhook-mailer at python.org  Wed Oct 30 15:36:35 2019
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Wed, 30 Oct 2019 19:36:35 -0000
Subject: [Python-checkins] [3.7] bpo-38600: Change the mark up of NULL in
 the C API documentation. (GH-16950) (GH-17000)
Message-ID: <mailman.191.1572464197.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/7f7f986cc485c40d80bae504333e3946ce9e96ba
commit: 7f7f986cc485c40d80bae504333e3946ce9e96ba
branch: 3.7
author: Serhiy Storchaka <storchaka at gmail.com>
committer: GitHub <noreply at github.com>
date: 2019-10-30T21:36:19+02:00
summary:

[3.7] bpo-38600: Change the mark up of NULL in the C API documentation. (GH-16950) (GH-17000)

Replace all *NULL* with ``NULL``.
(cherry picked from commit 25fc088607c855060ed142296dc1bd0125fad1af)

files:
M Doc/c-api/arg.rst
M Doc/c-api/buffer.rst
M Doc/c-api/bytearray.rst
M Doc/c-api/bytes.rst
M Doc/c-api/capsule.rst
M Doc/c-api/cell.rst
M Doc/c-api/codec.rst
M Doc/c-api/concrete.rst
M Doc/c-api/contextvars.rst
M Doc/c-api/conversion.rst
M Doc/c-api/coro.rst
M Doc/c-api/datetime.rst
M Doc/c-api/dict.rst
M Doc/c-api/exceptions.rst
M Doc/c-api/file.rst
M Doc/c-api/float.rst
M Doc/c-api/function.rst
M Doc/c-api/gcsupport.rst
M Doc/c-api/gen.rst
M Doc/c-api/import.rst
M Doc/c-api/init.rst
M Doc/c-api/intro.rst
M Doc/c-api/iter.rst
M Doc/c-api/list.rst
M Doc/c-api/long.rst
M Doc/c-api/mapping.rst
M Doc/c-api/marshal.rst
M Doc/c-api/memory.rst
M Doc/c-api/memoryview.rst
M Doc/c-api/method.rst
M Doc/c-api/module.rst
M Doc/c-api/number.rst
M Doc/c-api/object.rst
M Doc/c-api/refcounting.rst
M Doc/c-api/reflection.rst
M Doc/c-api/sequence.rst
M Doc/c-api/set.rst
M Doc/c-api/slice.rst
M Doc/c-api/structures.rst
M Doc/c-api/sys.rst
M Doc/c-api/tuple.rst
M Doc/c-api/type.rst
M Doc/c-api/typeobj.rst
M Doc/c-api/unicode.rst
M Doc/c-api/veryhigh.rst
M Doc/c-api/weakref.rst
M Doc/extending/embedding.rst
M Doc/extending/extending.rst
M Doc/extending/newtypes.rst
M Doc/extending/newtypes_tutorial.rst
M Doc/faq/extending.rst
M Doc/library/ctypes.rst
M Doc/whatsnew/3.4.rst
M Doc/whatsnew/3.7.rst
M Include/abstract.h

diff --git a/Doc/c-api/arg.rst b/Doc/c-api/arg.rst
index b41130ede416e..c3c8a03a79e08 100644
--- a/Doc/c-api/arg.rst
+++ b/Doc/c-api/arg.rst
@@ -50,7 +50,7 @@ Unless otherwise stated, buffers are not NUL-terminated.
 
 Some formats require a read-only :term:`bytes-like object`, and set a
 pointer instead of a buffer structure.  They work by checking that
-the object's :c:member:`PyBufferProcs.bf_releasebuffer` field is *NULL*,
+the object's :c:member:`PyBufferProcs.bf_releasebuffer` field is ``NULL``,
 which disallows mutable objects such as :class:`bytearray`.
 
 .. note::
@@ -99,15 +99,15 @@ which disallows mutable objects such as :class:`bytearray`.
 
 ``z`` (:class:`str` or ``None``) [const char \*]
    Like ``s``, but the Python object may also be ``None``, in which case the C
-   pointer is set to *NULL*.
+   pointer is set to ``NULL``.
 
 ``z*`` (:class:`str`, :term:`bytes-like object` or ``None``) [Py_buffer]
    Like ``s*``, but the Python object may also be ``None``, in which case the
-   ``buf`` member of the :c:type:`Py_buffer` structure is set to *NULL*.
+   ``buf`` member of the :c:type:`Py_buffer` structure is set to ``NULL``.
 
 ``z#`` (:class:`str`, read-only :term:`bytes-like object` or ``None``) [const char \*, int]
    Like ``s#``, but the Python object may also be ``None``, in which case the C
-   pointer is set to *NULL*.
+   pointer is set to ``NULL``.
 
 ``y`` (read-only :term:`bytes-like object`) [const char \*]
    This format converts a bytes-like object to a C pointer to a character
@@ -166,7 +166,7 @@ which disallows mutable objects such as :class:`bytearray`.
 
 ``Z`` (:class:`str` or ``None``) [const Py_UNICODE \*]
    Like ``u``, but the Python object may also be ``None``, in which case the
-   :c:type:`Py_UNICODE` pointer is set to *NULL*.
+   :c:type:`Py_UNICODE` pointer is set to ``NULL``.
 
    .. deprecated-removed:: 3.3 4.0
       Part of the old-style :c:type:`Py_UNICODE` API; please migrate to using
@@ -174,7 +174,7 @@ which disallows mutable objects such as :class:`bytearray`.
 
 ``Z#`` (:class:`str` or ``None``) [const Py_UNICODE \*, int]
    Like ``u#``, but the Python object may also be ``None``, in which case the
-   :c:type:`Py_UNICODE` pointer is set to *NULL*.
+   :c:type:`Py_UNICODE` pointer is set to ``NULL``.
 
    .. deprecated-removed:: 3.3 4.0
       Part of the old-style :c:type:`Py_UNICODE` API; please migrate to using
@@ -197,7 +197,7 @@ which disallows mutable objects such as :class:`bytearray`.
 
    This format requires two arguments.  The first is only used as input, and
    must be a :c:type:`const char\*` which points to the name of an encoding as a
-   NUL-terminated string, or *NULL*, in which case ``'utf-8'`` encoding is used.
+   NUL-terminated string, or ``NULL``, in which case ``'utf-8'`` encoding is used.
    An exception is raised if the named encoding is not known to Python.  The
    second argument must be a :c:type:`char\*\*`; the value of the pointer it
    references will be set to a buffer with the contents of the argument text.
@@ -220,7 +220,7 @@ which disallows mutable objects such as :class:`bytearray`.
 
    It requires three arguments.  The first is only used as input, and must be a
    :c:type:`const char\*` which points to the name of an encoding as a
-   NUL-terminated string, or *NULL*, in which case ``'utf-8'`` encoding is used.
+   NUL-terminated string, or ``NULL``, in which case ``'utf-8'`` encoding is used.
    An exception is raised if the named encoding is not known to Python.  The
    second argument must be a :c:type:`char\*\*`; the value of the pointer it
    references will be set to a buffer with the contents of the argument text.
@@ -230,12 +230,12 @@ which disallows mutable objects such as :class:`bytearray`.
 
    There are two modes of operation:
 
-   If *\*buffer* points a *NULL* pointer, the function will allocate a buffer of
+   If *\*buffer* points a ``NULL`` pointer, the function will allocate a buffer of
    the needed size, copy the encoded data into this buffer and set *\*buffer* to
    reference the newly allocated storage.  The caller is responsible for calling
    :c:func:`PyMem_Free` to free the allocated buffer after usage.
 
-   If *\*buffer* points to a non-*NULL* pointer (an already allocated buffer),
+   If *\*buffer* points to a non-``NULL`` pointer (an already allocated buffer),
    :c:func:`PyArg_ParseTuple` will use this location as the buffer and interpret the
    initial value of *\*buffer_length* as the buffer size.  It will then copy the
    encoded data into the buffer and NUL-terminate it.  If the buffer is not large
@@ -317,7 +317,7 @@ Other objects
 ``O`` (object) [PyObject \*]
    Store a Python object (without any conversion) in a C object pointer.  The C
    program thus receives the actual object that was passed.  The object's reference
-   count is not increased.  The pointer stored is not *NULL*.
+   count is not increased.  The pointer stored is not ``NULL``.
 
 ``O!`` (object) [*typeobject*, PyObject \*]
    Store a Python object in a C object pointer.  This is similar to ``O``, but
@@ -437,7 +437,7 @@ API Functions
 
    Parse the parameters of a function that takes both positional and keyword
    parameters into local variables.  The *keywords* argument is a
-   *NULL*-terminated array of keyword parameter names.  Empty names denote
+   ``NULL``-terminated array of keyword parameter names.  Empty names denote
    :ref:`positional-only parameters <positional-only_parameter>`.
    Returns true on success; on failure, it returns false and raises the
    appropriate exception.
@@ -520,8 +520,8 @@ Building values
 
    Create a new value based on a format string similar to those accepted by the
    :c:func:`PyArg_Parse\*` family of functions and a sequence of values.  Returns
-   the value or *NULL* in the case of an error; an exception will be raised if
-   *NULL* is returned.
+   the value or ``NULL`` in the case of an error; an exception will be raised if
+   ``NULL`` is returned.
 
    :c:func:`Py_BuildValue` does not always build a tuple.  It builds a tuple only if
    its format string contains two or more format units.  If the format string is
@@ -547,20 +547,20 @@ Building values
 
    ``s`` (:class:`str` or ``None``) [const char \*]
       Convert a null-terminated C string to a Python :class:`str` object using ``'utf-8'``
-      encoding. If the C string pointer is *NULL*, ``None`` is used.
+      encoding. If the C string pointer is ``NULL``, ``None`` is used.
 
    ``s#`` (:class:`str` or ``None``) [const char \*, int]
       Convert a C string and its length to a Python :class:`str` object using ``'utf-8'``
-      encoding. If the C string pointer is *NULL*, the length is ignored and
+      encoding. If the C string pointer is ``NULL``, the length is ignored and
       ``None`` is returned.
 
    ``y`` (:class:`bytes`) [const char \*]
       This converts a C string to a Python :class:`bytes` object.  If the C
-      string pointer is *NULL*, ``None`` is returned.
+      string pointer is ``NULL``, ``None`` is returned.
 
    ``y#`` (:class:`bytes`) [const char \*, int]
       This converts a C string and its lengths to a Python object.  If the C
-      string pointer is *NULL*, ``None`` is returned.
+      string pointer is ``NULL``, ``None`` is returned.
 
    ``z`` (:class:`str` or ``None``) [const char \*]
       Same as ``s``.
@@ -570,12 +570,12 @@ Building values
 
    ``u`` (:class:`str`) [const wchar_t \*]
       Convert a null-terminated :c:type:`wchar_t` buffer of Unicode (UTF-16 or UCS-4)
-      data to a Python Unicode object.  If the Unicode buffer pointer is *NULL*,
+      data to a Python Unicode object.  If the Unicode buffer pointer is ``NULL``,
       ``None`` is returned.
 
    ``u#`` (:class:`str`) [const wchar_t \*, int]
       Convert a Unicode (UTF-16 or UCS-4) data buffer and its length to a Python
-      Unicode object.   If the Unicode buffer pointer is *NULL*, the length is ignored
+      Unicode object.   If the Unicode buffer pointer is ``NULL``, the length is ignored
       and ``None`` is returned.
 
    ``U`` (:class:`str` or ``None``) [const char \*]
@@ -636,9 +636,9 @@ Building values
 
    ``O`` (object) [PyObject \*]
       Pass a Python object untouched (except for its reference count, which is
-      incremented by one).  If the object passed in is a *NULL* pointer, it is assumed
+      incremented by one).  If the object passed in is a ``NULL`` pointer, it is assumed
       that this was caused because the call producing the argument found an error and
-      set an exception. Therefore, :c:func:`Py_BuildValue` will return *NULL* but won't
+      set an exception. Therefore, :c:func:`Py_BuildValue` will return ``NULL`` but won't
       raise an exception.  If no exception has been raised yet, :exc:`SystemError` is
       set.
 
@@ -653,7 +653,7 @@ Building values
    ``O&`` (object) [*converter*, *anything*]
       Convert *anything* to a Python object through a *converter* function.  The
       function is called with *anything* (which should be compatible with :c:type:`void
-      \*`) as its argument and should return a "new" Python object, or *NULL* if an
+      \*`) as its argument and should return a "new" Python object, or ``NULL`` if an
       error occurred.
 
    ``(items)`` (:class:`tuple`) [*matching-items*]
@@ -668,7 +668,7 @@ Building values
       respectively.
 
    If there is an error in the format string, the :exc:`SystemError` exception is
-   set and *NULL* returned.
+   set and ``NULL`` returned.
 
 .. c:function:: PyObject* Py_VaBuildValue(const char *format, va_list vargs)
 
diff --git a/Doc/c-api/buffer.rst b/Doc/c-api/buffer.rst
index b639ba6ff7877..6b59ec5d4c9fd 100644
--- a/Doc/c-api/buffer.rst
+++ b/Doc/c-api/buffer.rst
@@ -102,13 +102,13 @@ a buffer, see :c:func:`PyObject_GetBuffer`.
    .. c:member:: void \*obj
 
       A new reference to the exporting object. The reference is owned by
-      the consumer and automatically decremented and set to *NULL* by
+      the consumer and automatically decremented and set to ``NULL`` by
       :c:func:`PyBuffer_Release`. The field is the equivalent of the return
       value of any standard C-API function.
 
       As a special case, for *temporary* buffers that are wrapped by
       :c:func:`PyMemoryView_FromBuffer` or :c:func:`PyBuffer_FillInfo`
-      this field is *NULL*. In general, exporting objects MUST NOT
+      this field is ``NULL``. In general, exporting objects MUST NOT
       use this scheme.
 
    .. c:member:: Py_ssize_t len
@@ -134,21 +134,21 @@ a buffer, see :c:func:`PyObject_GetBuffer`.
 
       Important exception: If a consumer requests a buffer without the
       :c:macro:`PyBUF_FORMAT` flag, :c:member:`~Py_buffer.format` will
-      be set to  *NULL*,  but :c:member:`~Py_buffer.itemsize` still has
+      be set to  ``NULL``,  but :c:member:`~Py_buffer.itemsize` still has
       the value for the original format.
 
       If :c:member:`~Py_buffer.shape` is present, the equality
       ``product(shape) * itemsize == len`` still holds and the consumer
       can use :c:member:`~Py_buffer.itemsize` to navigate the buffer.
 
-      If :c:member:`~Py_buffer.shape` is *NULL* as a result of a :c:macro:`PyBUF_SIMPLE`
+      If :c:member:`~Py_buffer.shape` is ``NULL`` as a result of a :c:macro:`PyBUF_SIMPLE`
       or a :c:macro:`PyBUF_WRITABLE` request, the consumer must disregard
       :c:member:`~Py_buffer.itemsize` and assume ``itemsize == 1``.
 
    .. c:member:: const char \*format
 
       A *NUL* terminated string in :mod:`struct` module style syntax describing
-      the contents of a single item. If this is *NULL*, ``"B"`` (unsigned bytes)
+      the contents of a single item. If this is ``NULL``, ``"B"`` (unsigned bytes)
       is assumed.
 
       This field is controlled by the :c:macro:`PyBUF_FORMAT` flag.
@@ -158,7 +158,7 @@ a buffer, see :c:func:`PyObject_GetBuffer`.
       The number of dimensions the memory represents as an n-dimensional array.
       If it is ``0``, :c:member:`~Py_buffer.buf` points to a single item representing
       a scalar. In this case, :c:member:`~Py_buffer.shape`, :c:member:`~Py_buffer.strides`
-      and :c:member:`~Py_buffer.suboffsets` MUST be *NULL*.
+      and :c:member:`~Py_buffer.suboffsets` MUST be ``NULL``.
 
       The macro :c:macro:`PyBUF_MAX_NDIM` limits the maximum number of dimensions
       to 64. Exporters MUST respect this limit, consumers of multi-dimensional
@@ -248,7 +248,7 @@ readonly, format
    .. c:macro:: PyBUF_FORMAT
 
       Controls the :c:member:`~Py_buffer.format` field. If set, this field MUST
-      be filled in correctly. Otherwise, this field MUST be *NULL*.
+      be filled in correctly. Otherwise, this field MUST be ``NULL``.
 
 
 :c:macro:`PyBUF_WRITABLE` can be \|'d to any of the flags in the next section.
@@ -349,9 +349,9 @@ The logical structure of NumPy-style arrays is defined by :c:member:`~Py_buffer.
 
 If ``ndim == 0``, the memory location pointed to by :c:member:`~Py_buffer.buf` is
 interpreted as a scalar of size :c:member:`~Py_buffer.itemsize`. In that case,
-both :c:member:`~Py_buffer.shape` and :c:member:`~Py_buffer.strides` are *NULL*.
+both :c:member:`~Py_buffer.shape` and :c:member:`~Py_buffer.strides` are ``NULL``.
 
-If :c:member:`~Py_buffer.strides` is *NULL*, the array is interpreted as
+If :c:member:`~Py_buffer.strides` is ``NULL``, the array is interpreted as
 a standard n-dimensional C-array. Otherwise, the consumer must access an
 n-dimensional array as follows:
 
@@ -438,7 +438,7 @@ Buffer-related functions
 
    Send a request to *exporter* to fill in *view* as specified by  *flags*.
    If the exporter cannot provide a buffer of the exact type, it MUST raise
-   :c:data:`PyExc_BufferError`, set :c:member:`view->obj` to *NULL* and
+   :c:data:`PyExc_BufferError`, set :c:member:`view->obj` to ``NULL`` and
    return ``-1``.
 
    On success, fill in *view*, set :c:member:`view->obj` to a new reference
@@ -516,7 +516,7 @@ Buffer-related functions
 
    On success, set :c:member:`view->obj` to a new reference to *exporter* and
    return 0. Otherwise, raise :c:data:`PyExc_BufferError`, set
-   :c:member:`view->obj` to *NULL* and return ``-1``;
+   :c:member:`view->obj` to ``NULL`` and return ``-1``;
 
    If this function is used as part of a :ref:`getbufferproc <buffer-structs>`,
    *exporter* MUST be set to the exporting object and *flags* must be passed
diff --git a/Doc/c-api/bytearray.rst b/Doc/c-api/bytearray.rst
index 41b6e3c71be53..e6872ce0cb288 100644
--- a/Doc/c-api/bytearray.rst
+++ b/Doc/c-api/bytearray.rst
@@ -48,7 +48,7 @@ Direct API functions
 .. c:function:: PyObject* PyByteArray_FromStringAndSize(const char *string, Py_ssize_t len)
 
    Create a new bytearray object from *string* and its length, *len*.  On
-   failure, *NULL* is returned.
+   failure, ``NULL`` is returned.
 
 
 .. c:function:: PyObject* PyByteArray_Concat(PyObject *a, PyObject *b)
@@ -58,13 +58,13 @@ Direct API functions
 
 .. c:function:: Py_ssize_t PyByteArray_Size(PyObject *bytearray)
 
-   Return the size of *bytearray* after checking for a *NULL* pointer.
+   Return the size of *bytearray* after checking for a ``NULL`` pointer.
 
 
 .. c:function:: char* PyByteArray_AsString(PyObject *bytearray)
 
    Return the contents of *bytearray* as a char array after checking for a
-   *NULL* pointer.  The returned array always has an extra
+   ``NULL`` pointer.  The returned array always has an extra
    null byte appended.
 
 
diff --git a/Doc/c-api/bytes.rst b/Doc/c-api/bytes.rst
index 5b9ebf6b6af5f..6699e35742f9d 100644
--- a/Doc/c-api/bytes.rst
+++ b/Doc/c-api/bytes.rst
@@ -37,14 +37,14 @@ called with a non-bytes parameter.
 .. c:function:: PyObject* PyBytes_FromString(const char *v)
 
    Return a new bytes object with a copy of the string *v* as value on success,
-   and *NULL* on failure.  The parameter *v* must not be *NULL*; it will not be
+   and ``NULL`` on failure.  The parameter *v* must not be ``NULL``; it will not be
    checked.
 
 
 .. c:function:: PyObject* PyBytes_FromStringAndSize(const char *v, Py_ssize_t len)
 
    Return a new bytes object with a copy of the string *v* as value and length
-   *len* on success, and *NULL* on failure.  If *v* is *NULL*, the contents of
+   *len* on success, and ``NULL`` on failure.  If *v* is ``NULL``, the contents of
    the bytes object are uninitialized.
 
 
@@ -145,7 +145,7 @@ called with a non-bytes parameter.
    whether there are any other null bytes.  The data must not be
    modified in any way, unless the object was just created using
    ``PyBytes_FromStringAndSize(NULL, size)``. It must not be deallocated.  If
-   *o* is not a bytes object at all, :c:func:`PyBytes_AsString` returns *NULL*
+   *o* is not a bytes object at all, :c:func:`PyBytes_AsString` returns ``NULL``
    and raises :exc:`TypeError`.
 
 
@@ -159,7 +159,7 @@ called with a non-bytes parameter.
    Return the null-terminated contents of the object *obj*
    through the output variables *buffer* and *length*.
 
-   If *length* is *NULL*, the bytes object
+   If *length* is ``NULL``, the bytes object
    may not contain embedded null bytes;
    if it does, the function returns ``-1`` and a :exc:`ValueError` is raised.
 
@@ -181,7 +181,7 @@ called with a non-bytes parameter.
    appended to *bytes*; the caller will own the new reference.  The reference to
    the old value of *bytes* will be stolen.  If the new object cannot be
    created, the old reference to *bytes* will still be discarded and the value
-   of *\*bytes* will be set to *NULL*; the appropriate exception will be set.
+   of *\*bytes* will be set to ``NULL``; the appropriate exception will be set.
 
 
 .. c:function:: void PyBytes_ConcatAndDel(PyObject **bytes, PyObject *newpart)
@@ -201,5 +201,5 @@ called with a non-bytes parameter.
    desired.  On success, *\*bytes* holds the resized bytes object and ``0`` is
    returned; the address in *\*bytes* may differ from its input value.  If the
    reallocation fails, the original bytes object at *\*bytes* is deallocated,
-   *\*bytes* is set to *NULL*, :exc:`MemoryError` is set, and ``-1`` is
+   *\*bytes* is set to ``NULL``, :exc:`MemoryError` is set, and ``-1`` is
    returned.
diff --git a/Doc/c-api/capsule.rst b/Doc/c-api/capsule.rst
index 8eb6695e22de1..07efb9e10e264 100644
--- a/Doc/c-api/capsule.rst
+++ b/Doc/c-api/capsule.rst
@@ -40,15 +40,15 @@ Refer to :ref:`using-capsules` for more information on using these objects.
 .. c:function:: PyObject* PyCapsule_New(void *pointer, const char *name, PyCapsule_Destructor destructor)
 
    Create a :c:type:`PyCapsule` encapsulating the *pointer*.  The *pointer*
-   argument may not be *NULL*.
+   argument may not be ``NULL``.
 
-   On failure, set an exception and return *NULL*.
+   On failure, set an exception and return ``NULL``.
 
-   The *name* string may either be *NULL* or a pointer to a valid C string.  If
-   non-*NULL*, this string must outlive the capsule.  (Though it is permitted to
+   The *name* string may either be ``NULL`` or a pointer to a valid C string.  If
+   non-``NULL``, this string must outlive the capsule.  (Though it is permitted to
    free it inside the *destructor*.)
 
-   If the *destructor* argument is not *NULL*, it will be called with the
+   If the *destructor* argument is not ``NULL``, it will be called with the
    capsule as its argument when it is destroyed.
 
    If this capsule will be stored as an attribute of a module, the *name* should
@@ -59,20 +59,20 @@ Refer to :ref:`using-capsules` for more information on using these objects.
 .. c:function:: void* PyCapsule_GetPointer(PyObject *capsule, const char *name)
 
    Retrieve the *pointer* stored in the capsule.  On failure, set an exception
-   and return *NULL*.
+   and return ``NULL``.
 
    The *name* parameter must compare exactly to the name stored in the capsule.
-   If the name stored in the capsule is *NULL*, the *name* passed in must also
-   be *NULL*.  Python uses the C function :c:func:`strcmp` to compare capsule
+   If the name stored in the capsule is ``NULL``, the *name* passed in must also
+   be ``NULL``.  Python uses the C function :c:func:`strcmp` to compare capsule
    names.
 
 
 .. c:function:: PyCapsule_Destructor PyCapsule_GetDestructor(PyObject *capsule)
 
    Return the current destructor stored in the capsule.  On failure, set an
-   exception and return *NULL*.
+   exception and return ``NULL``.
 
-   It is legal for a capsule to have a *NULL* destructor.  This makes a *NULL*
+   It is legal for a capsule to have a ``NULL`` destructor.  This makes a ``NULL``
    return code somewhat ambiguous; use :c:func:`PyCapsule_IsValid` or
    :c:func:`PyErr_Occurred` to disambiguate.
 
@@ -80,9 +80,9 @@ Refer to :ref:`using-capsules` for more information on using these objects.
 .. c:function:: void* PyCapsule_GetContext(PyObject *capsule)
 
    Return the current context stored in the capsule.  On failure, set an
-   exception and return *NULL*.
+   exception and return ``NULL``.
 
-   It is legal for a capsule to have a *NULL* context.  This makes a *NULL*
+   It is legal for a capsule to have a ``NULL`` context.  This makes a ``NULL``
    return code somewhat ambiguous; use :c:func:`PyCapsule_IsValid` or
    :c:func:`PyErr_Occurred` to disambiguate.
 
@@ -90,9 +90,9 @@ Refer to :ref:`using-capsules` for more information on using these objects.
 .. c:function:: const char* PyCapsule_GetName(PyObject *capsule)
 
    Return the current name stored in the capsule.  On failure, set an exception
-   and return *NULL*.
+   and return ``NULL``.
 
-   It is legal for a capsule to have a *NULL* name.  This makes a *NULL* return
+   It is legal for a capsule to have a ``NULL`` name.  This makes a ``NULL`` return
    code somewhat ambiguous; use :c:func:`PyCapsule_IsValid` or
    :c:func:`PyErr_Occurred` to disambiguate.
 
@@ -107,13 +107,13 @@ Refer to :ref:`using-capsules` for more information on using these objects.
    import the module conventionally (using :c:func:`PyImport_ImportModule`).
 
    Return the capsule's internal *pointer* on success.  On failure, set an
-   exception and return *NULL*.
+   exception and return ``NULL``.
 
 
 .. c:function:: int PyCapsule_IsValid(PyObject *capsule, const char *name)
 
    Determines whether or not *capsule* is a valid capsule.  A valid capsule is
-   non-*NULL*, passes :c:func:`PyCapsule_CheckExact`, has a non-*NULL* pointer
+   non-``NULL``, passes :c:func:`PyCapsule_CheckExact`, has a non-``NULL`` pointer
    stored in it, and its internal name matches the *name* parameter.  (See
    :c:func:`PyCapsule_GetPointer` for information on how capsule names are
    compared.)
@@ -142,9 +142,9 @@ Refer to :ref:`using-capsules` for more information on using these objects.
 
 .. c:function:: int PyCapsule_SetName(PyObject *capsule, const char *name)
 
-   Set the name inside *capsule* to *name*.  If non-*NULL*, the name must
+   Set the name inside *capsule* to *name*.  If non-``NULL``, the name must
    outlive the capsule.  If the previous *name* stored in the capsule was not
-   *NULL*, no attempt is made to free it.
+   ``NULL``, no attempt is made to free it.
 
    Return ``0`` on success.  Return nonzero and set an exception on failure.
 
@@ -152,6 +152,6 @@ Refer to :ref:`using-capsules` for more information on using these objects.
 .. c:function:: int PyCapsule_SetPointer(PyObject *capsule, void *pointer)
 
    Set the void pointer inside *capsule* to *pointer*.  The pointer may not be
-   *NULL*.
+   ``NULL``.
 
    Return ``0`` on success.  Return nonzero and set an exception on failure.
diff --git a/Doc/c-api/cell.rst b/Doc/c-api/cell.rst
index 427259cc24d8d..624dfe8508e7e 100644
--- a/Doc/c-api/cell.rst
+++ b/Doc/c-api/cell.rst
@@ -27,13 +27,13 @@ Cell objects are not likely to be useful elsewhere.
 
 .. c:function:: int PyCell_Check(ob)
 
-   Return true if *ob* is a cell object; *ob* must not be *NULL*.
+   Return true if *ob* is a cell object; *ob* must not be ``NULL``.
 
 
 .. c:function:: PyObject* PyCell_New(PyObject *ob)
 
    Create and return a new cell object containing the value *ob*. The parameter may
-   be *NULL*.
+   be ``NULL``.
 
 
 .. c:function:: PyObject* PyCell_Get(PyObject *cell)
@@ -44,19 +44,19 @@ Cell objects are not likely to be useful elsewhere.
 .. c:function:: PyObject* PyCell_GET(PyObject *cell)
 
    Return the contents of the cell *cell*, but without checking that *cell* is
-   non-*NULL* and a cell object.
+   non-``NULL`` and a cell object.
 
 
 .. c:function:: int PyCell_Set(PyObject *cell, PyObject *value)
 
    Set the contents of the cell object *cell* to *value*.  This releases the
-   reference to any current content of the cell. *value* may be *NULL*.  *cell*
-   must be non-*NULL*; if it is not a cell object, ``-1`` will be returned.  On
+   reference to any current content of the cell. *value* may be ``NULL``.  *cell*
+   must be non-``NULL``; if it is not a cell object, ``-1`` will be returned.  On
    success, ``0`` will be returned.
 
 
 .. c:function:: void PyCell_SET(PyObject *cell, PyObject *value)
 
    Sets the value of the cell object *cell* to *value*.  No reference counts are
-   adjusted, and no checks are made for safety; *cell* must be non-*NULL* and must
+   adjusted, and no checks are made for safety; *cell* must be non-``NULL`` and must
    be a cell object.
diff --git a/Doc/c-api/codec.rst b/Doc/c-api/codec.rst
index c55f19970e125..172dcb326a4bc 100644
--- a/Doc/c-api/codec.rst
+++ b/Doc/c-api/codec.rst
@@ -21,7 +21,7 @@ Codec registry and support functions
 
    *object* is passed through the encoder function found for the given
    *encoding* using the error handling method defined by *errors*.  *errors* may
-   be *NULL* to use the default method defined for the codec.  Raises a
+   be ``NULL`` to use the default method defined for the codec.  Raises a
    :exc:`LookupError` if no encoder can be found.
 
 .. c:function:: PyObject* PyCodec_Decode(PyObject *object, const char *encoding, const char *errors)
@@ -30,7 +30,7 @@ Codec registry and support functions
 
    *object* is passed through the decoder function found for the given
    *encoding* using the error handling method defined by *errors*.  *errors* may
-   be *NULL* to use the default method defined for the codec.  Raises a
+   be ``NULL`` to use the default method defined for the codec.  Raises a
    :exc:`LookupError` if no encoder can be found.
 
 
@@ -40,7 +40,7 @@ Codec lookup API
 In the following functions, the *encoding* string is looked up converted to all
 lower-case characters, which makes encodings looked up through this mechanism
 effectively case-insensitive.  If no codec is found, a :exc:`KeyError` is set
-and *NULL* returned.
+and ``NULL`` returned.
 
 .. c:function:: PyObject* PyCodec_Encoder(const char *encoding)
 
@@ -92,7 +92,7 @@ Registry API for Unicode encoding error handlers
 .. c:function:: PyObject* PyCodec_LookupError(const char *name)
 
    Lookup the error handling callback function registered under *name*.  As a
-   special case *NULL* can be passed, in which case the error handling callback
+   special case ``NULL`` can be passed, in which case the error handling callback
    for "strict" will be returned.
 
 .. c:function:: PyObject* PyCodec_StrictErrors(PyObject *exc)
diff --git a/Doc/c-api/concrete.rst b/Doc/c-api/concrete.rst
index 9558a4a583cba..b4c1372c2394c 100644
--- a/Doc/c-api/concrete.rst
+++ b/Doc/c-api/concrete.rst
@@ -17,8 +17,8 @@ dictionary, use :c:func:`PyDict_Check`.  The chapter is structured like the
 .. warning::
 
    While the functions described in this chapter carefully check the type of the
-   objects which are passed in, many of them do not check for *NULL* being passed
-   instead of a valid object.  Allowing *NULL* to be passed in can cause memory
+   objects which are passed in, many of them do not check for ``NULL`` being passed
+   instead of a valid object.  Allowing ``NULL`` to be passed in can cause memory
    access violations and immediate termination of the interpreter.
 
 
diff --git a/Doc/c-api/contextvars.rst b/Doc/c-api/contextvars.rst
index c344c8d71ae32..c0532f2d09220 100644
--- a/Doc/c-api/contextvars.rst
+++ b/Doc/c-api/contextvars.rst
@@ -60,17 +60,17 @@ Type-check macros:
 .. c:function:: int PyContext_CheckExact(PyObject *o)
 
    Return true if *o* is of type :c:data:`PyContext_Type`. *o* must not be
-   *NULL*.  This function always succeeds.
+   ``NULL``.  This function always succeeds.
 
 .. c:function:: int PyContextVar_CheckExact(PyObject *o)
 
    Return true if *o* is of type :c:data:`PyContextVar_Type`. *o* must not be
-   *NULL*.  This function always succeeds.
+   ``NULL``.  This function always succeeds.
 
 .. c:function:: int PyContextToken_CheckExact(PyObject *o)
 
    Return true if *o* is of type :c:data:`PyContextToken_Type`.
-   *o* must not be *NULL*.  This function always succeeds.
+   *o* must not be ``NULL``.  This function always succeeds.
 
 
 Context object management functions:
diff --git a/Doc/c-api/conversion.rst b/Doc/c-api/conversion.rst
index c46722d782a2f..3a4182a608948 100644
--- a/Doc/c-api/conversion.rst
+++ b/Doc/c-api/conversion.rst
@@ -113,7 +113,7 @@ The following functions provide locale-independent string to number conversions.
    *val* is a finite number, an infinite number, or not a number, respectively.
 
    The return value is a pointer to *buffer* with the converted string or
-   *NULL* if the conversion failed. The caller is responsible for freeing the
+   ``NULL`` if the conversion failed. The caller is responsible for freeing the
    returned string by calling :c:func:`PyMem_Free`.
 
    .. versionadded:: 3.1
diff --git a/Doc/c-api/coro.rst b/Doc/c-api/coro.rst
index 2fe50b5d8c447..de81b4ea5dc21 100644
--- a/Doc/c-api/coro.rst
+++ b/Doc/c-api/coro.rst
@@ -23,7 +23,7 @@ return.
 
 .. c:function:: int PyCoro_CheckExact(PyObject *ob)
 
-   Return true if *ob*'s type is *PyCoro_Type*; *ob* must not be *NULL*.
+   Return true if *ob*'s type is *PyCoro_Type*; *ob* must not be ``NULL``.
 
 
 .. c:function:: PyObject* PyCoro_New(PyFrameObject *frame, PyObject *name, PyObject *qualname)
@@ -31,4 +31,4 @@ return.
    Create and return a new coroutine object based on the *frame* object,
    with ``__name__`` and ``__qualname__`` set to *name* and *qualname*.
    A reference to *frame* is stolen by this function.  The *frame* argument
-   must not be *NULL*.
+   must not be ``NULL``.
diff --git a/Doc/c-api/datetime.rst b/Doc/c-api/datetime.rst
index 77b1b216744bf..55de81665e04b 100644
--- a/Doc/c-api/datetime.rst
+++ b/Doc/c-api/datetime.rst
@@ -28,61 +28,61 @@ Type-check macros:
 .. c:function:: int PyDate_Check(PyObject *ob)
 
    Return true if *ob* is of type :c:data:`PyDateTime_DateType` or a subtype of
-   :c:data:`PyDateTime_DateType`.  *ob* must not be *NULL*.
+   :c:data:`PyDateTime_DateType`.  *ob* must not be ``NULL``.
 
 
 .. c:function:: int PyDate_CheckExact(PyObject *ob)
 
    Return true if *ob* is of type :c:data:`PyDateTime_DateType`. *ob* must not be
-   *NULL*.
+   ``NULL``.
 
 
 .. c:function:: int PyDateTime_Check(PyObject *ob)
 
    Return true if *ob* is of type :c:data:`PyDateTime_DateTimeType` or a subtype of
-   :c:data:`PyDateTime_DateTimeType`.  *ob* must not be *NULL*.
+   :c:data:`PyDateTime_DateTimeType`.  *ob* must not be ``NULL``.
 
 
 .. c:function:: int PyDateTime_CheckExact(PyObject *ob)
 
    Return true if *ob* is of type :c:data:`PyDateTime_DateTimeType`. *ob* must not
-   be *NULL*.
+   be ``NULL``.
 
 
 .. c:function:: int PyTime_Check(PyObject *ob)
 
    Return true if *ob* is of type :c:data:`PyDateTime_TimeType` or a subtype of
-   :c:data:`PyDateTime_TimeType`.  *ob* must not be *NULL*.
+   :c:data:`PyDateTime_TimeType`.  *ob* must not be ``NULL``.
 
 
 .. c:function:: int PyTime_CheckExact(PyObject *ob)
 
    Return true if *ob* is of type :c:data:`PyDateTime_TimeType`. *ob* must not be
-   *NULL*.
+   ``NULL``.
 
 
 .. c:function:: int PyDelta_Check(PyObject *ob)
 
    Return true if *ob* is of type :c:data:`PyDateTime_DeltaType` or a subtype of
-   :c:data:`PyDateTime_DeltaType`.  *ob* must not be *NULL*.
+   :c:data:`PyDateTime_DeltaType`.  *ob* must not be ``NULL``.
 
 
 .. c:function:: int PyDelta_CheckExact(PyObject *ob)
 
    Return true if *ob* is of type :c:data:`PyDateTime_DeltaType`. *ob* must not be
-   *NULL*.
+   ``NULL``.
 
 
 .. c:function:: int PyTZInfo_Check(PyObject *ob)
 
    Return true if *ob* is of type :c:data:`PyDateTime_TZInfoType` or a subtype of
-   :c:data:`PyDateTime_TZInfoType`.  *ob* must not be *NULL*.
+   :c:data:`PyDateTime_TZInfoType`.  *ob* must not be ``NULL``.
 
 
 .. c:function:: int PyTZInfo_CheckExact(PyObject *ob)
 
    Return true if *ob* is of type :c:data:`PyDateTime_TZInfoType`. *ob* must not be
-   *NULL*.
+   ``NULL``.
 
 
 Macros to create objects:
@@ -144,7 +144,7 @@ Macros to create objects:
 
 Macros to extract fields from date objects.  The argument must be an instance of
 :c:data:`PyDateTime_Date`, including subclasses (such as
-:c:data:`PyDateTime_DateTime`).  The argument must not be *NULL*, and the type is
+:c:data:`PyDateTime_DateTime`).  The argument must not be ``NULL``, and the type is
 not checked:
 
 .. c:function:: int PyDateTime_GET_YEAR(PyDateTime_Date *o)
@@ -164,7 +164,7 @@ not checked:
 
 Macros to extract fields from datetime objects.  The argument must be an
 instance of :c:data:`PyDateTime_DateTime`, including subclasses. The argument
-must not be *NULL*, and the type is not checked:
+must not be ``NULL``, and the type is not checked:
 
 .. c:function:: int PyDateTime_DATE_GET_HOUR(PyDateTime_DateTime *o)
 
@@ -187,7 +187,7 @@ must not be *NULL*, and the type is not checked:
 
 
 Macros to extract fields from time objects.  The argument must be an instance of
-:c:data:`PyDateTime_Time`, including subclasses. The argument must not be *NULL*,
+:c:data:`PyDateTime_Time`, including subclasses. The argument must not be ``NULL``,
 and the type is not checked:
 
 .. c:function:: int PyDateTime_TIME_GET_HOUR(PyDateTime_Time *o)
@@ -212,7 +212,7 @@ and the type is not checked:
 
 Macros to extract fields from time delta objects.  The argument must be an
 instance of :c:data:`PyDateTime_Delta`, including subclasses. The argument must
-not be *NULL*, and the type is not checked:
+not be ``NULL``, and the type is not checked:
 
 .. c:function:: int PyDateTime_DELTA_GET_DAYS(PyDateTime_Delta *o)
 
diff --git a/Doc/c-api/dict.rst b/Doc/c-api/dict.rst
index 0ced5a5fd0017..ea7baf891e05e 100644
--- a/Doc/c-api/dict.rst
+++ b/Doc/c-api/dict.rst
@@ -33,7 +33,7 @@ Dictionary Objects
 
 .. c:function:: PyObject* PyDict_New()
 
-   Return a new empty dictionary, or *NULL* on failure.
+   Return a new empty dictionary, or ``NULL`` on failure.
 
 
 .. c:function:: PyObject* PyDictProxy_New(PyObject *mapping)
@@ -92,7 +92,7 @@ Dictionary Objects
 
 .. c:function:: PyObject* PyDict_GetItem(PyObject *p, PyObject *key)
 
-   Return the object from dictionary *p* which has a key *key*.  Return *NULL*
+   Return the object from dictionary *p* which has a key *key*.  Return ``NULL``
    if the key *key* is not present, but *without* setting an exception.
 
    Note that exceptions which occur while calling :meth:`__hash__` and
@@ -103,8 +103,8 @@ Dictionary Objects
 .. c:function:: PyObject* PyDict_GetItemWithError(PyObject *p, PyObject *key)
 
    Variant of :c:func:`PyDict_GetItem` that does not suppress
-   exceptions. Return *NULL* **with** an exception set if an exception
-   occurred.  Return *NULL* **without** an exception set if the key
+   exceptions. Return ``NULL`` **with** an exception set if an exception
+   occurred.  Return ``NULL`` **without** an exception set if the key
    wasn't present.
 
 
@@ -161,7 +161,7 @@ Dictionary Objects
    function returns true for each pair in the dictionary, and false once all
    pairs have been reported.  The parameters *pkey* and *pvalue* should either
    point to :c:type:`PyObject\*` variables that will be filled in with each key
-   and value, respectively, or may be *NULL*.  Any references returned through
+   and value, respectively, or may be ``NULL``.  Any references returned through
    them are borrowed.  *ppos* should not be altered during iteration. Its
    value represents offsets within the internal dictionary structure, and
    since the structure is sparse, the offsets are not consecutive.
diff --git a/Doc/c-api/exceptions.rst b/Doc/c-api/exceptions.rst
index 14ca2b409969c..a763fd7751307 100644
--- a/Doc/c-api/exceptions.rst
+++ b/Doc/c-api/exceptions.rst
@@ -13,7 +13,7 @@ exception handling.  It works somewhat like the POSIX :c:data:`errno` variable:
 there is a global indicator (per thread) of the last error that occurred.  Most
 C API functions don't clear this on success, but will set it to indicate the
 cause of the error on failure.  Most C API functions also return an error
-indicator, usually *NULL* if they are supposed to return a pointer, or ``-1``
+indicator, usually ``NULL`` if they are supposed to return a pointer, or ``-1``
 if they return an integer (exception: the :c:func:`PyArg_\*` functions
 return ``1`` for success and ``0`` for failure).
 
@@ -108,7 +108,7 @@ NULL pointer for use in a ``return`` statement.
 
 .. c:function:: PyObject* PyErr_Format(PyObject *exception, const char *format, ...)
 
-   This function sets the error indicator and returns *NULL*.  *exception*
+   This function sets the error indicator and returns ``NULL``.  *exception*
    should be a Python exception class.  The *format* and subsequent
    parameters help format the error message; they have the same meaning and
    values as in :c:func:`PyUnicode_FromFormat`. *format* is an ASCII-encoded
@@ -137,7 +137,7 @@ NULL pointer for use in a ``return`` statement.
 
 .. c:function:: PyObject* PyErr_NoMemory()
 
-   This is a shorthand for ``PyErr_SetNone(PyExc_MemoryError)``; it returns *NULL*
+   This is a shorthand for ``PyErr_SetNone(PyExc_MemoryError)``; it returns ``NULL``
    so an object allocation function can write ``return PyErr_NoMemory();`` when it
    runs out of memory.
 
@@ -153,7 +153,7 @@ NULL pointer for use in a ``return`` statement.
    and then calls ``PyErr_SetObject(type, object)``.  On Unix, when the
    :c:data:`errno` value is :const:`EINTR`, indicating an interrupted system call,
    this calls :c:func:`PyErr_CheckSignals`, and if that set the error indicator,
-   leaves it set to that.  The function always returns *NULL*, so a wrapper
+   leaves it set to that.  The function always returns ``NULL``, so a wrapper
    function around a system call can write ``return PyErr_SetFromErrno(type);``
    when the system call returns an error.
 
@@ -161,7 +161,7 @@ NULL pointer for use in a ``return`` statement.
 .. c:function:: PyObject* PyErr_SetFromErrnoWithFilenameObject(PyObject *type, PyObject *filenameObject)
 
    Similar to :c:func:`PyErr_SetFromErrno`, with the additional behavior that if
-   *filenameObject* is not *NULL*, it is passed to the constructor of *type* as
+   *filenameObject* is not ``NULL``, it is passed to the constructor of *type* as
    a third parameter.  In the case of :exc:`OSError` exception,
    this is used to define the :attr:`filename` attribute of the
    exception instance.
@@ -192,7 +192,7 @@ NULL pointer for use in a ``return`` statement.
    then it constructs a tuple object whose first item is the *ierr* value and whose
    second item is the corresponding error message (gotten from
    :c:func:`FormatMessage`), and then calls ``PyErr_SetObject(PyExc_WindowsError,
-   object)``. This function always returns *NULL*.
+   object)``. This function always returns ``NULL``.
 
    .. availability:: Windows.
 
@@ -301,7 +301,7 @@ an error value).
 .. c:function:: int PyErr_WarnEx(PyObject *category, const char *message, Py_ssize_t stack_level)
 
    Issue a warning message.  The *category* argument is a warning category (see
-   below) or *NULL*; the *message* argument is a UTF-8 encoded string.  *stack_level* is a
+   below) or ``NULL``; the *message* argument is a UTF-8 encoded string.  *stack_level* is a
    positive number giving a number of stack frames; the warning will be issued from
    the  currently executing line of code in that stack frame.  A *stack_level* of 1
    is the function calling :c:func:`PyErr_WarnEx`, 2 is  the function above that,
@@ -330,7 +330,7 @@ an error value).
    Issue a warning message with explicit control over all warning attributes.  This
    is a straightforward wrapper around the Python function
    :func:`warnings.warn_explicit`, see there for more information.  The *module*
-   and *registry* arguments may be set to *NULL* to get the default effect
+   and *registry* arguments may be set to ``NULL`` to get the default effect
    described there.
 
    .. versionadded:: 3.4
@@ -367,7 +367,7 @@ Querying the error indicator
 
    Test whether the error indicator is set.  If set, return the exception *type*
    (the first argument to the last call to one of the :c:func:`PyErr_Set\*`
-   functions or to :c:func:`PyErr_Restore`).  If not set, return *NULL*.  You do not
+   functions or to :c:func:`PyErr_Restore`).  If not set, return ``NULL``.  You do not
    own a reference to the return value, so you do not need to :c:func:`Py_DECREF`
    it.
 
@@ -397,9 +397,9 @@ Querying the error indicator
 .. c:function:: void PyErr_Fetch(PyObject **ptype, PyObject **pvalue, PyObject **ptraceback)
 
    Retrieve the error indicator into three variables whose addresses are passed.
-   If the error indicator is not set, set all three variables to *NULL*.  If it is
+   If the error indicator is not set, set all three variables to ``NULL``.  If it is
    set, it will be cleared and you own a reference to each object retrieved.  The
-   value and traceback object may be *NULL* even when the type object is not.
+   value and traceback object may be ``NULL`` even when the type object is not.
 
    .. note::
 
@@ -419,8 +419,8 @@ Querying the error indicator
 .. c:function:: void PyErr_Restore(PyObject *type, PyObject *value, PyObject *traceback)
 
    Set  the error indicator from the three objects.  If the error indicator is
-   already set, it is cleared first.  If the objects are *NULL*, the error
-   indicator is cleared.  Do not pass a *NULL* type and non-*NULL* value or
+   already set, it is cleared first.  If the objects are ``NULL``, the error
+   indicator is cleared.  Do not pass a ``NULL`` type and non-``NULL`` value or
    traceback.  The exception type should be a class.  Do not pass an invalid
    exception type or value. (Violating these rules will cause subtle problems
    later.)  This call takes away a reference to each object: you must own a
@@ -459,7 +459,7 @@ Querying the error indicator
    Retrieve the exception info, as known from ``sys.exc_info()``.  This refers
    to an exception that was *already caught*, not to an exception that was
    freshly raised.  Returns new references for the three objects, any of which
-   may be *NULL*.  Does not modify the exception info state.
+   may be ``NULL``.  Does not modify the exception info state.
 
    .. note::
 
@@ -476,7 +476,7 @@ Querying the error indicator
    Set the exception info, as known from ``sys.exc_info()``.  This refers
    to an exception that was *already caught*, not to an exception that was
    freshly raised.  This function steals the references of the arguments.
-   To clear the exception state, pass *NULL* for all three arguments.
+   To clear the exception state, pass ``NULL`` for all three arguments.
    For general rules about the three arguments, see :c:func:`PyErr_Restore`.
 
    .. note::
@@ -546,7 +546,7 @@ Exception Classes
 
    This utility function creates and returns a new exception class. The *name*
    argument must be the name of the new exception, a C string of the form
-   ``module.classname``.  The *base* and *dict* arguments are normally *NULL*.
+   ``module.classname``.  The *base* and *dict* arguments are normally ``NULL``.
    This creates a class object derived from :exc:`Exception` (accessible in C as
    :c:data:`PyExc_Exception`).
 
@@ -560,7 +560,7 @@ Exception Classes
 .. c:function:: PyObject* PyErr_NewExceptionWithDoc(const char *name, const char *doc, PyObject *base, PyObject *dict)
 
    Same as :c:func:`PyErr_NewException`, except that the new exception class can
-   easily be given a docstring: If *doc* is non-*NULL*, it will be used as the
+   easily be given a docstring: If *doc* is non-``NULL``, it will be used as the
    docstring for the exception class.
 
    .. versionadded:: 3.2
@@ -573,7 +573,7 @@ Exception Objects
 
    Return the traceback associated with the exception as a new reference, as
    accessible from Python through :attr:`__traceback__`.  If there is no
-   traceback associated, this returns *NULL*.
+   traceback associated, this returns ``NULL``.
 
 
 .. c:function:: int PyException_SetTraceback(PyObject *ex, PyObject *tb)
@@ -587,12 +587,12 @@ Exception Objects
    Return the context (another exception instance during whose handling *ex* was
    raised) associated with the exception as a new reference, as accessible from
    Python through :attr:`__context__`.  If there is no context associated, this
-   returns *NULL*.
+   returns ``NULL``.
 
 
 .. c:function:: void PyException_SetContext(PyObject *ex, PyObject *ctx)
 
-   Set the context associated with the exception to *ctx*.  Use *NULL* to clear
+   Set the context associated with the exception to *ctx*.  Use ``NULL`` to clear
    it.  There is no type check to make sure that *ctx* is an exception instance.
    This steals a reference to *ctx*.
 
@@ -606,7 +606,7 @@ Exception Objects
 
 .. c:function:: void PyException_SetCause(PyObject *ex, PyObject *cause)
 
-   Set the cause associated with the exception to *cause*.  Use *NULL* to clear
+   Set the cause associated with the exception to *cause*.  Use ``NULL`` to clear
    it.  There is no type check to make sure that *cause* is either an exception
    instance or :const:`None`.  This steals a reference to *cause*.
 
@@ -653,7 +653,7 @@ The following functions are used to create and modify Unicode exceptions from C.
                 int PyUnicodeTranslateError_GetStart(PyObject *exc, Py_ssize_t *start)
 
    Get the *start* attribute of the given exception object and place it into
-   *\*start*.  *start* must not be *NULL*.  Return ``0`` on success, ``-1`` on
+   *\*start*.  *start* must not be ``NULL``.  Return ``0`` on success, ``-1`` on
    failure.
 
 .. c:function:: int PyUnicodeDecodeError_SetStart(PyObject *exc, Py_ssize_t start)
@@ -668,7 +668,7 @@ The following functions are used to create and modify Unicode exceptions from C.
                 int PyUnicodeTranslateError_GetEnd(PyObject *exc, Py_ssize_t *end)
 
    Get the *end* attribute of the given exception object and place it into
-   *\*end*.  *end* must not be *NULL*.  Return ``0`` on success, ``-1`` on
+   *\*end*.  *end* must not be ``NULL``.  Return ``0`` on success, ``-1`` on
    failure.
 
 .. c:function:: int PyUnicodeDecodeError_SetEnd(PyObject *exc, Py_ssize_t end)
diff --git a/Doc/c-api/file.rst b/Doc/c-api/file.rst
index 6f2ecee51fd86..63f1101caf515 100644
--- a/Doc/c-api/file.rst
+++ b/Doc/c-api/file.rst
@@ -21,9 +21,9 @@ the :mod:`io` APIs instead.
 
    Create a Python file object from the file descriptor of an already
    opened file *fd*.  The arguments *name*, *encoding*, *errors* and *newline*
-   can be *NULL* to use the defaults; *buffering* can be *-1* to use the
+   can be ``NULL`` to use the defaults; *buffering* can be *-1* to use the
    default. *name* is ignored and kept for backward compatibility. Return
-   *NULL* on failure. For a more comprehensive description of the arguments,
+   ``NULL`` on failure. For a more comprehensive description of the arguments,
    please refer to the :func:`io.open` function documentation.
 
    .. warning::
diff --git a/Doc/c-api/float.rst b/Doc/c-api/float.rst
index 27a75e3e0c2ee..fae321c250c41 100644
--- a/Doc/c-api/float.rst
+++ b/Doc/c-api/float.rst
@@ -34,12 +34,12 @@ Floating Point Objects
 .. c:function:: PyObject* PyFloat_FromString(PyObject *str)
 
    Create a :c:type:`PyFloatObject` object based on the string value in *str*, or
-   *NULL* on failure.
+   ``NULL`` on failure.
 
 
 .. c:function:: PyObject* PyFloat_FromDouble(double v)
 
-   Create a :c:type:`PyFloatObject` object from *v*, or *NULL* on failure.
+   Create a :c:type:`PyFloatObject` object from *v*, or ``NULL`` on failure.
 
 
 .. c:function:: double PyFloat_AsDouble(PyObject *pyfloat)
diff --git a/Doc/c-api/function.rst b/Doc/c-api/function.rst
index 17279c732ae07..a5045ab56f46c 100644
--- a/Doc/c-api/function.rst
+++ b/Doc/c-api/function.rst
@@ -26,7 +26,7 @@ There are a few functions specific to Python functions.
 .. c:function:: int PyFunction_Check(PyObject *o)
 
    Return true if *o* is a function object (has type :c:data:`PyFunction_Type`).
-   The parameter must not be *NULL*.
+   The parameter must not be ``NULL``.
 
 
 .. c:function:: PyObject* PyFunction_New(PyObject *code, PyObject *globals)
@@ -36,7 +36,7 @@ There are a few functions specific to Python functions.
 
    The function's docstring and name are retrieved from the code object. *__module__*
    is retrieved from *globals*. The argument defaults, annotations and closure are
-   set to *NULL*. *__qualname__* is set to the same value as the function's name.
+   set to ``NULL``. *__qualname__* is set to the same value as the function's name.
 
 
 .. c:function:: PyObject* PyFunction_NewWithQualName(PyObject *code, PyObject *globals, PyObject *qualname)
@@ -69,7 +69,7 @@ There are a few functions specific to Python functions.
 .. c:function:: PyObject* PyFunction_GetDefaults(PyObject *op)
 
    Return the argument default values of the function object *op*. This can be a
-   tuple of arguments or *NULL*.
+   tuple of arguments or ``NULL``.
 
 
 .. c:function:: int PyFunction_SetDefaults(PyObject *op, PyObject *defaults)
@@ -82,7 +82,7 @@ There are a few functions specific to Python functions.
 
 .. c:function:: PyObject* PyFunction_GetClosure(PyObject *op)
 
-   Return the closure associated with the function object *op*. This can be *NULL*
+   Return the closure associated with the function object *op*. This can be ``NULL``
    or a tuple of cell objects.
 
 
@@ -97,7 +97,7 @@ There are a few functions specific to Python functions.
 .. c:function:: PyObject *PyFunction_GetAnnotations(PyObject *op)
 
    Return the annotations of the function object *op*. This can be a
-   mutable dictionary or *NULL*.
+   mutable dictionary or ``NULL``.
 
 
 .. c:function:: int PyFunction_SetAnnotations(PyObject *op, PyObject *annotations)
diff --git a/Doc/c-api/gcsupport.rst b/Doc/c-api/gcsupport.rst
index 472cd93ec3f7e..95116d50bc465 100644
--- a/Doc/c-api/gcsupport.rst
+++ b/Doc/c-api/gcsupport.rst
@@ -49,7 +49,7 @@ Constructors for container types must conform to two rules:
 .. c:function:: TYPE* PyObject_GC_Resize(TYPE, PyVarObject *op, Py_ssize_t newsize)
 
    Resize an object allocated by :c:func:`PyObject_NewVar`.  Returns the
-   resized object or *NULL* on failure.  *op* must not be tracked by the collector yet.
+   resized object or ``NULL`` on failure.  *op* must not be tracked by the collector yet.
 
 
 .. c:function:: void PyObject_GC_Track(PyObject *op)
@@ -121,7 +121,7 @@ The :c:member:`~PyTypeObject.tp_traverse` handler must have the following type:
    Traversal function for a container object.  Implementations must call the
    *visit* function for each object directly contained by *self*, with the
    parameters to *visit* being the contained object and the *arg* value passed
-   to the handler.  The *visit* function must not be called with a *NULL*
+   to the handler.  The *visit* function must not be called with a ``NULL``
    object argument.  If *visit* returns a non-zero value that value should be
    returned immediately.
 
@@ -132,7 +132,7 @@ must name its arguments exactly *visit* and *arg*:
 
 .. c:function:: void Py_VISIT(PyObject *o)
 
-   If *o* is not *NULL*, call the *visit* callback, with arguments *o*
+   If *o* is not ``NULL``, call the *visit* callback, with arguments *o*
    and *arg*.  If *visit* returns a non-zero value, then return it.
    Using this macro, :c:member:`~PyTypeObject.tp_traverse` handlers
    look like::
@@ -145,7 +145,7 @@ must name its arguments exactly *visit* and *arg*:
           return 0;
       }
 
-The :c:member:`~PyTypeObject.tp_clear` handler must be of the :c:type:`inquiry` type, or *NULL*
+The :c:member:`~PyTypeObject.tp_clear` handler must be of the :c:type:`inquiry` type, or ``NULL``
 if the object is immutable.
 
 
diff --git a/Doc/c-api/gen.rst b/Doc/c-api/gen.rst
index 1efbae4fcba09..e007ced158545 100644
--- a/Doc/c-api/gen.rst
+++ b/Doc/c-api/gen.rst
@@ -22,23 +22,23 @@ than explicitly calling :c:func:`PyGen_New` or :c:func:`PyGen_NewWithQualName`.
 
 .. c:function:: int PyGen_Check(PyObject *ob)
 
-   Return true if *ob* is a generator object; *ob* must not be *NULL*.
+   Return true if *ob* is a generator object; *ob* must not be ``NULL``.
 
 
 .. c:function:: int PyGen_CheckExact(PyObject *ob)
 
-   Return true if *ob*'s type is *PyGen_Type*; *ob* must not be *NULL*.
+   Return true if *ob*'s type is *PyGen_Type*; *ob* must not be ``NULL``.
 
 
 .. c:function:: PyObject* PyGen_New(PyFrameObject *frame)
 
    Create and return a new generator object based on the *frame* object.
    A reference to *frame* is stolen by this function. The argument must not be
-   *NULL*.
+   ``NULL``.
 
 .. c:function:: PyObject* PyGen_NewWithQualName(PyFrameObject *frame, PyObject *name, PyObject *qualname)
 
    Create and return a new generator object based on the *frame* object,
    with ``__name__`` and ``__qualname__`` set to *name* and *qualname*.
    A reference to *frame* is stolen by this function.  The *frame* argument
-   must not be *NULL*.
+   must not be ``NULL``.
diff --git a/Doc/c-api/import.rst b/Doc/c-api/import.rst
index 8cdc256e7c9e0..daa704ce2a474 100644
--- a/Doc/c-api/import.rst
+++ b/Doc/c-api/import.rst
@@ -14,7 +14,7 @@ Importing Modules
       single: modules (in module sys)
 
    This is a simplified interface to :c:func:`PyImport_ImportModuleEx` below,
-   leaving the *globals* and *locals* arguments set to *NULL* and *level* set
+   leaving the *globals* and *locals* arguments set to ``NULL`` and *level* set
    to 0.  When the *name*
    argument contains a dot (when it specifies a submodule of a package), the
    *fromlist* argument is set to the list ``['*']`` so that the return value is the
@@ -22,7 +22,7 @@ Importing Modules
    be the case.  (Unfortunately, this has an additional side effect when *name* in
    fact specifies a subpackage instead of a submodule: the submodules specified in
    the package's ``__all__`` variable are  loaded.)  Return a new reference to the
-   imported module, or *NULL* with an exception set on failure.  A failing
+   imported module, or ``NULL`` with an exception set on failure.  A failing
    import of a module doesn't leave the module in :data:`sys.modules`.
 
    This function always uses absolute imports.
@@ -47,7 +47,7 @@ Importing Modules
    function :func:`__import__`.
 
    The return value is a new reference to the imported module or top-level
-   package, or *NULL* with an exception set on failure.  Like for
+   package, or ``NULL`` with an exception set on failure.  Like for
    :func:`__import__`, the return value when a submodule of a package was
    requested is normally the top-level package, unless a non-empty *fromlist*
    was given.
@@ -63,7 +63,7 @@ Importing Modules
    this function directly.
 
    The return value is a new reference to the imported module or top-level package,
-   or *NULL* with an exception set on failure.  Like for :func:`__import__`,
+   or ``NULL`` with an exception set on failure.  Like for :func:`__import__`,
    the return value when a submodule of a package was requested is normally the
    top-level package, unless a non-empty *fromlist* was given.
 
@@ -91,7 +91,7 @@ Importing Modules
 
 .. c:function:: PyObject* PyImport_ReloadModule(PyObject *m)
 
-   Reload a module.  Return a new reference to the reloaded module, or *NULL* with
+   Reload a module.  Return a new reference to the reloaded module, or ``NULL`` with
    an exception set on failure (the module still exists in this case).
 
 
@@ -100,7 +100,7 @@ Importing Modules
    Return the module object corresponding to a module name.  The *name* argument
    may be of the form ``package.module``. First check the modules dictionary if
    there's one there, and if not, create a new one and insert it in the modules
-   dictionary. Return *NULL* with an exception set on failure.
+   dictionary. Return ``NULL`` with an exception set on failure.
 
    .. note::
 
@@ -125,7 +125,7 @@ Importing Modules
    Given a module name (possibly of the form ``package.module``) and a code object
    read from a Python bytecode file or obtained from the built-in function
    :func:`compile`, load the module.  Return a new reference to the module object,
-   or *NULL* with an exception set if an error occurred.  *name*
+   or ``NULL`` with an exception set if an error occurred.  *name*
    is removed from :attr:`sys.modules` in error cases, even if *name* was already
    in :attr:`sys.modules` on entry to :c:func:`PyImport_ExecCodeModule`.  Leaving
    incompletely initialized modules in :attr:`sys.modules` is dangerous, as imports of
@@ -277,7 +277,7 @@ Importing Modules
 .. c:var:: const struct _frozen* PyImport_FrozenModules
 
    This pointer is initialized to point to an array of :c:type:`struct _frozen`
-   records, terminated by one whose members are all *NULL* or zero.  When a frozen
+   records, terminated by one whose members are all ``NULL`` or zero.  When a frozen
    module is imported, it is searched in this table.  Third-party code could play
    tricks with this to provide a dynamically created collection of frozen modules.
 
@@ -310,7 +310,7 @@ Importing Modules
 .. c:function:: int PyImport_ExtendInittab(struct _inittab *newtab)
 
    Add a collection of modules to the table of built-in modules.  The *newtab*
-   array must end with a sentinel entry which contains *NULL* for the :attr:`name`
+   array must end with a sentinel entry which contains ``NULL`` for the :attr:`name`
    field; failure to provide the sentinel value can result in a memory fault.
    Returns ``0`` on success or ``-1`` if insufficient memory could be allocated to
    extend the internal table.  In the event of failure, no modules are added to the
diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst
index 93fcfe6260782..8c7485168572f 100644
--- a/Doc/c-api/init.rst
+++ b/Doc/c-api/init.rst
@@ -844,8 +844,8 @@ code, or when embedding the Python interpreter:
 .. c:function:: PyThreadState* PyEval_SaveThread()
 
    Release the global interpreter lock (if it has been created and thread
-   support is enabled) and reset the thread state to *NULL*, returning the
-   previous thread state (which is not *NULL*).  If the lock has been created,
+   support is enabled) and reset the thread state to ``NULL``, returning the
+   previous thread state (which is not ``NULL``).  If the lock has been created,
    the current thread must have acquired it.
 
 
@@ -853,7 +853,7 @@ code, or when embedding the Python interpreter:
 
    Acquire the global interpreter lock (if it has been created and thread
    support is enabled) and set the thread state to *tstate*, which must not be
-   *NULL*.  If the lock has been created, the current thread must not have
+   ``NULL``.  If the lock has been created, the current thread must not have
    acquired it, otherwise deadlock ensues.
 
    .. note::
@@ -866,14 +866,14 @@ code, or when embedding the Python interpreter:
 .. c:function:: PyThreadState* PyThreadState_Get()
 
    Return the current thread state.  The global interpreter lock must be held.
-   When the current thread state is *NULL*, this issues a fatal error (so that
-   the caller needn't check for *NULL*).
+   When the current thread state is ``NULL``, this issues a fatal error (so that
+   the caller needn't check for ``NULL``).
 
 
 .. c:function:: PyThreadState* PyThreadState_Swap(PyThreadState *tstate)
 
    Swap the current thread state with the thread state given by the argument
-   *tstate*, which may be *NULL*.  The global interpreter lock must be held
+   *tstate*, which may be ``NULL``.  The global interpreter lock must be held
    and is not released.
 
 
@@ -1044,7 +1044,7 @@ All of the following functions must be called after :c:func:`Py_Initialize`.
    Return a dictionary in which extensions can store thread-specific state
    information.  Each extension should use a unique key to use to store state in
    the dictionary.  It is okay to call this function when no current thread state
-   is available. If this function returns *NULL*, no exception has been raised and
+   is available. If this function returns ``NULL``, no exception has been raised and
    the caller should assume no current thread state is available.
 
 
@@ -1065,7 +1065,7 @@ All of the following functions must be called after :c:func:`Py_Initialize`.
 .. c:function:: void PyEval_AcquireThread(PyThreadState *tstate)
 
    Acquire the global interpreter lock and set the current thread state to
-   *tstate*, which should not be *NULL*.  The lock must have been created earlier.
+   *tstate*, which should not be ``NULL``.  The lock must have been created earlier.
    If this thread already has the lock, deadlock ensues.
 
    :c:func:`PyEval_RestoreThread` is a higher-level function which is always
@@ -1074,9 +1074,9 @@ All of the following functions must be called after :c:func:`Py_Initialize`.
 
 .. c:function:: void PyEval_ReleaseThread(PyThreadState *tstate)
 
-   Reset the current thread state to *NULL* and release the global interpreter
+   Reset the current thread state to ``NULL`` and release the global interpreter
    lock.  The lock must have been created earlier and must be held by the current
-   thread.  The *tstate* argument, which must not be *NULL*, is only used to check
+   thread.  The *tstate* argument, which must not be ``NULL``, is only used to check
    that it represents the current thread state --- if it isn't, a fatal error is
    reported.
 
@@ -1141,7 +1141,7 @@ using the following functions:
    The return value points to the first thread state created in the new
    sub-interpreter.  This thread state is made in the current thread state.
    Note that no actual thread is created; see the discussion of thread states
-   below.  If creation of the new interpreter is unsuccessful, *NULL* is
+   below.  If creation of the new interpreter is unsuccessful, ``NULL`` is
    returned; no exception is set since the exception state is stored in the
    current thread state and there may not be a current thread state.  (Like all
    other Python/C API functions, the global interpreter lock must be held before
@@ -1172,7 +1172,7 @@ using the following functions:
 
    Destroy the (sub-)interpreter represented by the given thread state. The given
    thread state must be the current thread state.  See the discussion of thread
-   states below.  When the call returns, the current thread state is *NULL*.  All
+   states below.  When the call returns, the current thread state is ``NULL``.  All
    thread states associated with this interpreter are destroyed.  (The global
    interpreter lock must be held before calling this function and is still held
    when it returns.)  :c:func:`Py_FinalizeEx` will destroy all sub-interpreters that
@@ -1279,27 +1279,27 @@ Python-level trace functions in previous versions.
    :const:`PyTrace_C_CALL`, :const:`PyTrace_C_EXCEPTION`, :const:`PyTrace_C_RETURN`,
    or :const:`PyTrace_OPCODE`, and *arg* depends on the value of *what*:
 
-   +------------------------------+--------------------------------------+
-   | Value of *what*              | Meaning of *arg*                     |
-   +==============================+======================================+
-   | :const:`PyTrace_CALL`        | Always :c:data:`Py_None`.            |
-   +------------------------------+--------------------------------------+
-   | :const:`PyTrace_EXCEPTION`   | Exception information as returned by |
-   |                              | :func:`sys.exc_info`.                |
-   +------------------------------+--------------------------------------+
-   | :const:`PyTrace_LINE`        | Always :c:data:`Py_None`.            |
-   +------------------------------+--------------------------------------+
-   | :const:`PyTrace_RETURN`      | Value being returned to the caller,  |
-   |                              | or *NULL* if caused by an exception. |
-   +------------------------------+--------------------------------------+
-   | :const:`PyTrace_C_CALL`      | Function object being called.        |
-   +------------------------------+--------------------------------------+
-   | :const:`PyTrace_C_EXCEPTION` | Function object being called.        |
-   +------------------------------+--------------------------------------+
-   | :const:`PyTrace_C_RETURN`    | Function object being called.        |
-   +------------------------------+--------------------------------------+
-   | :const:`PyTrace_OPCODE`      | Always :c:data:`Py_None`.            |
-   +------------------------------+--------------------------------------+
+   +------------------------------+----------------------------------------+
+   | Value of *what*              | Meaning of *arg*                       |
+   +==============================+========================================+
+   | :const:`PyTrace_CALL`        | Always :c:data:`Py_None`.              |
+   +------------------------------+----------------------------------------+
+   | :const:`PyTrace_EXCEPTION`   | Exception information as returned by   |
+   |                              | :func:`sys.exc_info`.                  |
+   +------------------------------+----------------------------------------+
+   | :const:`PyTrace_LINE`        | Always :c:data:`Py_None`.              |
+   +------------------------------+----------------------------------------+
+   | :const:`PyTrace_RETURN`      | Value being returned to the caller,    |
+   |                              | or ``NULL`` if caused by an exception. |
+   +------------------------------+----------------------------------------+
+   | :const:`PyTrace_C_CALL`      | Function object being called.          |
+   +------------------------------+----------------------------------------+
+   | :const:`PyTrace_C_EXCEPTION` | Function object being called.          |
+   +------------------------------+----------------------------------------+
+   | :const:`PyTrace_C_RETURN`    | Function object being called.          |
+   +------------------------------+----------------------------------------+
+   | :const:`PyTrace_OPCODE`      | Always :c:data:`Py_None`.              |
+   +------------------------------+----------------------------------------+
 
 .. c:var:: int PyTrace_CALL
 
@@ -1363,7 +1363,7 @@ Python-level trace functions in previous versions.
 .. c:function:: void PyEval_SetProfile(Py_tracefunc func, PyObject *obj)
 
    Set the profiler function to *func*.  The *obj* parameter is passed to the
-   function as its first parameter, and may be any Python object, or *NULL*.  If
+   function as its first parameter, and may be any Python object, or ``NULL``.  If
    the profile function needs to maintain state, using a different value for *obj*
    for each thread provides a convenient and thread-safe place to store it.  The
    profile function is called for all monitored events except :const:`PyTrace_LINE`
@@ -1486,7 +1486,7 @@ is not possible due to its implementation being opaque at build time.
 .. c:function:: Py_tss_t* PyThread_tss_alloc()
 
    Return a value which is the same state as a value initialized with
-   :c:macro:`Py_tss_NEEDS_INIT`, or *NULL* in the case of dynamic allocation
+   :c:macro:`Py_tss_NEEDS_INIT`, or ``NULL`` in the case of dynamic allocation
    failure.
 
 
@@ -1505,7 +1505,7 @@ is not possible due to its implementation being opaque at build time.
 Methods
 ~~~~~~~
 
-The parameter *key* of these functions must not be *NULL*.  Moreover, the
+The parameter *key* of these functions must not be ``NULL``.  Moreover, the
 behaviors of :c:func:`PyThread_tss_set` and :c:func:`PyThread_tss_get` are
 undefined if the given :c:type:`Py_tss_t` has not been initialized by
 :c:func:`PyThread_tss_create`.
@@ -1545,7 +1545,7 @@ undefined if the given :c:type:`Py_tss_t` has not been initialized by
 .. c:function:: void* PyThread_tss_get(Py_tss_t *key)
 
    Return the :c:type:`void\*` value associated with a TSS key in the current
-   thread.  This returns *NULL* if no value is associated with the key in the
+   thread.  This returns ``NULL`` if no value is associated with the key in the
    current thread.
 
 
diff --git a/Doc/c-api/intro.rst b/Doc/c-api/intro.rst
index 964d348c2df18..e4aad1f6e0134 100644
--- a/Doc/c-api/intro.rst
+++ b/Doc/c-api/intro.rst
@@ -152,7 +152,7 @@ complete listing.
 
 .. c:macro:: Py_GETENV(s)
 
-   Like ``getenv(s)``, but returns *NULL* if :option:`-E` was passed on the
+   Like ``getenv(s)``, but returns ``NULL`` if :option:`-E` was passed on the
    command line (i.e. if ``Py_IgnoreEnvironmentFlag`` is set).
 
 .. c:macro:: Py_UNUSED(arg)
@@ -463,7 +463,7 @@ functions in the Python/C API can raise exceptions, unless an explicit claim is
 made otherwise in a function's documentation.  In general, when a function
 encounters an error, it sets an exception, discards any object references that
 it owns, and returns an error indicator.  If not documented otherwise, this
-indicator is either *NULL* or ``-1``, depending on the function's return type.
+indicator is either ``NULL`` or ``-1``, depending on the function's return type.
 A few functions return a Boolean true/false result, with false indicating an
 error.  Very few functions return no explicit error indicator or have an
 ambiguous return value, and require explicit testing for errors with
@@ -478,13 +478,13 @@ using global storage in an unthreaded application).  A  thread can be in one of
 two states: an exception has occurred, or not. The function
 :c:func:`PyErr_Occurred` can be used to check for this: it returns a borrowed
 reference to the exception type object when an exception has occurred, and
-*NULL* otherwise.  There are a number of functions to set the exception state:
+``NULL`` otherwise.  There are a number of functions to set the exception state:
 :c:func:`PyErr_SetString` is the most common (though not the most general)
 function to set the exception state, and :c:func:`PyErr_Clear` clears the
 exception state.
 
 The full exception state consists of three objects (all of which can  be
-*NULL*): the exception type, the corresponding exception  value, and the
+``NULL``): the exception type, the corresponding exception  value, and the
 traceback.  These have the same meanings as the Python result of
 ``sys.exc_info()``; however, they are not the same: the Python objects represent
 the last exception being handled by a Python  :keyword:`try` ...
@@ -585,10 +585,10 @@ Here is the corresponding C code, in all its glory::
 This example represents an endorsed use of the ``goto`` statement  in C!
 It illustrates the use of :c:func:`PyErr_ExceptionMatches` and
 :c:func:`PyErr_Clear` to handle specific exceptions, and the use of
-:c:func:`Py_XDECREF` to dispose of owned references that may be *NULL* (note the
+:c:func:`Py_XDECREF` to dispose of owned references that may be ``NULL`` (note the
 ``'X'`` in the name; :c:func:`Py_DECREF` would crash when confronted with a
-*NULL* reference).  It is important that the variables used to hold owned
-references are initialized to *NULL* for this to work; likewise, the proposed
+``NULL`` reference).  It is important that the variables used to hold owned
+references are initialized to ``NULL`` for this to work; likewise, the proposed
 return value is initialized to ``-1`` (failure) and only set to success after
 the final call made is successful.
 
diff --git a/Doc/c-api/iter.rst b/Doc/c-api/iter.rst
index 2ba444d1de6d2..62ca082713b68 100644
--- a/Doc/c-api/iter.rst
+++ b/Doc/c-api/iter.rst
@@ -16,8 +16,8 @@ There are two functions specifically for working with iterators.
 
    Return the next value from the iteration *o*.  The object must be an iterator
    (it is up to the caller to check this).  If there are no remaining values,
-   returns *NULL* with no exception set.  If an error occurs while retrieving
-   the item, returns *NULL* and passes along the exception.
+   returns ``NULL`` with no exception set.  If an error occurs while retrieving
+   the item, returns ``NULL`` and passes along the exception.
 
 To write a loop which iterates over an iterator, the C code should look
 something like this::
diff --git a/Doc/c-api/list.rst b/Doc/c-api/list.rst
index c7cae550a46fa..0dde88470e3a3 100644
--- a/Doc/c-api/list.rst
+++ b/Doc/c-api/list.rst
@@ -33,7 +33,7 @@ List Objects
 
 .. c:function:: PyObject* PyList_New(Py_ssize_t len)
 
-   Return a new list of length *len* on success, or *NULL* on failure.
+   Return a new list of length *len* on success, or ``NULL`` on failure.
 
    .. note::
 
@@ -61,7 +61,7 @@ List Objects
    Return the object at position *index* in the list pointed to by *list*.  The
    position must be non-negative; indexing from the end of the list is not
    supported.  If *index* is out of bounds (<0 or >=len(list)),
-   return *NULL* and set an :exc:`IndexError` exception.
+   return ``NULL`` and set an :exc:`IndexError` exception.
 
 
 .. c:function:: PyObject* PyList_GET_ITEM(PyObject *list, Py_ssize_t i)
@@ -111,7 +111,7 @@ List Objects
 .. c:function:: PyObject* PyList_GetSlice(PyObject *list, Py_ssize_t low, Py_ssize_t high)
 
    Return a list of the objects in *list* containing the objects *between* *low*
-   and *high*.  Return *NULL* and set an exception if unsuccessful.  Analogous
+   and *high*.  Return ``NULL`` and set an exception if unsuccessful.  Analogous
    to ``list[low:high]``.  Indexing from the end of the list is not supported.
 
 
@@ -119,7 +119,7 @@ List Objects
 
    Set the slice of *list* between *low* and *high* to the contents of
    *itemlist*.  Analogous to ``list[low:high] = itemlist``. The *itemlist* may
-   be *NULL*, indicating the assignment of an empty list (slice deletion).
+   be ``NULL``, indicating the assignment of an empty list (slice deletion).
    Return ``0`` on success, ``-1`` on failure.  Indexing from the end of the
    list is not supported.
 
diff --git a/Doc/c-api/long.rst b/Doc/c-api/long.rst
index 71144f16e395f..15fccb86a403e 100644
--- a/Doc/c-api/long.rst
+++ b/Doc/c-api/long.rst
@@ -38,7 +38,7 @@ distinguished from a number.  Use :c:func:`PyErr_Occurred` to disambiguate.
 
 .. c:function:: PyObject* PyLong_FromLong(long v)
 
-   Return a new :c:type:`PyLongObject` object from *v*, or *NULL* on failure.
+   Return a new :c:type:`PyLongObject` object from *v*, or ``NULL`` on failure.
 
    The current implementation keeps an array of integer objects for all integers
    between ``-5`` and ``256``, when you create an int in that range you actually
@@ -50,43 +50,43 @@ distinguished from a number.  Use :c:func:`PyErr_Occurred` to disambiguate.
 .. c:function:: PyObject* PyLong_FromUnsignedLong(unsigned long v)
 
    Return a new :c:type:`PyLongObject` object from a C :c:type:`unsigned long`, or
-   *NULL* on failure.
+   ``NULL`` on failure.
 
 
 .. c:function:: PyObject* PyLong_FromSsize_t(Py_ssize_t v)
 
    Return a new :c:type:`PyLongObject` object from a C :c:type:`Py_ssize_t`, or
-   *NULL* on failure.
+   ``NULL`` on failure.
 
 
 .. c:function:: PyObject* PyLong_FromSize_t(size_t v)
 
    Return a new :c:type:`PyLongObject` object from a C :c:type:`size_t`, or
-   *NULL* on failure.
+   ``NULL`` on failure.
 
 
 .. c:function:: PyObject* PyLong_FromLongLong(long long v)
 
-   Return a new :c:type:`PyLongObject` object from a C :c:type:`long long`, or *NULL*
+   Return a new :c:type:`PyLongObject` object from a C :c:type:`long long`, or ``NULL``
    on failure.
 
 
 .. c:function:: PyObject* PyLong_FromUnsignedLongLong(unsigned long long v)
 
    Return a new :c:type:`PyLongObject` object from a C :c:type:`unsigned long long`,
-   or *NULL* on failure.
+   or ``NULL`` on failure.
 
 
 .. c:function:: PyObject* PyLong_FromDouble(double v)
 
    Return a new :c:type:`PyLongObject` object from the integer part of *v*, or
-   *NULL* on failure.
+   ``NULL`` on failure.
 
 
 .. c:function:: PyObject* PyLong_FromString(const char *str, char **pend, int base)
 
    Return a new :c:type:`PyLongObject` based on the string value in *str*, which
-   is interpreted according to the radix in *base*.  If *pend* is non-*NULL*,
+   is interpreted according to the radix in *base*.  If *pend* is non-``NULL``,
    *\*pend* will point to the first character in *str* which follows the
    representation of the number.  If *base* is ``0``, *str* is interpreted using
    the :ref:`integers` definition; in this case, leading zeros in a
@@ -294,4 +294,4 @@ distinguished from a number.  Use :c:func:`PyErr_Occurred` to disambiguate.
    is only assured to produce a usable :c:type:`void` pointer for values created
    with :c:func:`PyLong_FromVoidPtr`.
 
-   Returns *NULL* on error.  Use :c:func:`PyErr_Occurred` to disambiguate.
+   Returns ``NULL`` on error.  Use :c:func:`PyErr_Occurred` to disambiguate.
diff --git a/Doc/c-api/mapping.rst b/Doc/c-api/mapping.rst
index e37dec9949ab6..44cf2e7690892 100644
--- a/Doc/c-api/mapping.rst
+++ b/Doc/c-api/mapping.rst
@@ -29,7 +29,7 @@ See also :c:func:`PyObject_GetItem`, :c:func:`PyObject_SetItem` and
 
 .. c:function:: PyObject* PyMapping_GetItemString(PyObject *o, const char *key)
 
-   Return element of *o* corresponding to the string *key* or *NULL* on failure.
+   Return element of *o* corresponding to the string *key* or ``NULL`` on failure.
    This is the equivalent of the Python expression ``o[key]``.
    See also :c:func:`PyObject_GetItem`.
 
@@ -79,7 +79,7 @@ See also :c:func:`PyObject_GetItem`, :c:func:`PyObject_SetItem` and
 .. c:function:: PyObject* PyMapping_Keys(PyObject *o)
 
    On success, return a list of the keys in object *o*.  On failure, return
-   *NULL*.
+   ``NULL``.
 
    .. versionchanged:: 3.7
       Previously, the function returned a list or a tuple.
@@ -88,7 +88,7 @@ See also :c:func:`PyObject_GetItem`, :c:func:`PyObject_SetItem` and
 .. c:function:: PyObject* PyMapping_Values(PyObject *o)
 
    On success, return a list of the values in object *o*.  On failure, return
-   *NULL*.
+   ``NULL``.
 
    .. versionchanged:: 3.7
       Previously, the function returned a list or a tuple.
@@ -97,7 +97,7 @@ See also :c:func:`PyObject_GetItem`, :c:func:`PyObject_SetItem` and
 .. c:function:: PyObject* PyMapping_Items(PyObject *o)
 
    On success, return a list of the items in object *o*, where each item is a
-   tuple containing a key-value pair.  On failure, return *NULL*.
+   tuple containing a key-value pair.  On failure, return ``NULL``.
 
    .. versionchanged:: 3.7
       Previously, the function returned a list or a tuple.
diff --git a/Doc/c-api/marshal.rst b/Doc/c-api/marshal.rst
index 17ec621610b5e..0556a2e74ab53 100644
--- a/Doc/c-api/marshal.rst
+++ b/Doc/c-api/marshal.rst
@@ -67,7 +67,7 @@ The following functions allow marshalled values to be read back in.
    reading.
 
    On error, sets the appropriate exception (:exc:`EOFError`, :exc:`ValueError`
-   or :exc:`TypeError`) and returns *NULL*.
+   or :exc:`TypeError`) and returns ``NULL``.
 
 
 .. c:function:: PyObject* PyMarshal_ReadLastObjectFromFile(FILE *file)
@@ -81,7 +81,7 @@ The following functions allow marshalled values to be read back in.
    anything else from the file.
 
    On error, sets the appropriate exception (:exc:`EOFError`, :exc:`ValueError`
-   or :exc:`TypeError`) and returns *NULL*.
+   or :exc:`TypeError`) and returns ``NULL``.
 
 
 .. c:function:: PyObject* PyMarshal_ReadObjectFromString(const char *data, Py_ssize_t len)
@@ -90,5 +90,5 @@ The following functions allow marshalled values to be read back in.
    containing *len* bytes pointed to by *data*.
 
    On error, sets the appropriate exception (:exc:`EOFError`, :exc:`ValueError`
-   or :exc:`TypeError`) and returns *NULL*.
+   or :exc:`TypeError`) and returns ``NULL``.
 
diff --git a/Doc/c-api/memory.rst b/Doc/c-api/memory.rst
index 01f7f4209fe9e..1ce7e3cf16cde 100644
--- a/Doc/c-api/memory.rst
+++ b/Doc/c-api/memory.rst
@@ -110,9 +110,9 @@ zero bytes.
 .. c:function:: void* PyMem_RawMalloc(size_t n)
 
    Allocates *n* bytes and returns a pointer of type :c:type:`void\*` to the
-   allocated memory, or *NULL* if the request fails.
+   allocated memory, or ``NULL`` if the request fails.
 
-   Requesting zero bytes returns a distinct non-*NULL* pointer if possible, as
+   Requesting zero bytes returns a distinct non-``NULL`` pointer if possible, as
    if ``PyMem_RawMalloc(1)`` had been called instead. The memory will not have
    been initialized in any way.
 
@@ -120,11 +120,11 @@ zero bytes.
 .. c:function:: void* PyMem_RawCalloc(size_t nelem, size_t elsize)
 
    Allocates *nelem* elements each whose size in bytes is *elsize* and returns
-   a pointer of type :c:type:`void\*` to the allocated memory, or *NULL* if the
+   a pointer of type :c:type:`void\*` to the allocated memory, or ``NULL`` if the
    request fails. The memory is initialized to zeros.
 
    Requesting zero elements or elements of size zero bytes returns a distinct
-   non-*NULL* pointer if possible, as if ``PyMem_RawCalloc(1, 1)`` had been
+   non-``NULL`` pointer if possible, as if ``PyMem_RawCalloc(1, 1)`` had been
    called instead.
 
    .. versionadded:: 3.5
@@ -135,15 +135,15 @@ zero bytes.
    Resizes the memory block pointed to by *p* to *n* bytes. The contents will
    be unchanged to the minimum of the old and the new sizes.
 
-   If *p* is *NULL*, the call is equivalent to ``PyMem_RawMalloc(n)``; else if
+   If *p* is ``NULL``, the call is equivalent to ``PyMem_RawMalloc(n)``; else if
    *n* is equal to zero, the memory block is resized but is not freed, and the
-   returned pointer is non-*NULL*.
+   returned pointer is non-``NULL``.
 
-   Unless *p* is *NULL*, it must have been returned by a previous call to
+   Unless *p* is ``NULL``, it must have been returned by a previous call to
    :c:func:`PyMem_RawMalloc`, :c:func:`PyMem_RawRealloc` or
    :c:func:`PyMem_RawCalloc`.
 
-   If the request fails, :c:func:`PyMem_RawRealloc` returns *NULL* and *p*
+   If the request fails, :c:func:`PyMem_RawRealloc` returns ``NULL`` and *p*
    remains a valid pointer to the previous memory area.
 
 
@@ -154,7 +154,7 @@ zero bytes.
    :c:func:`PyMem_RawCalloc`.  Otherwise, or if ``PyMem_RawFree(p)`` has been
    called before, undefined behavior occurs.
 
-   If *p* is *NULL*, no operation is performed.
+   If *p* is ``NULL``, no operation is performed.
 
 
 .. _memoryinterface:
@@ -181,9 +181,9 @@ The :ref:`default memory allocator <default-memory-allocators>` uses the
 .. c:function:: void* PyMem_Malloc(size_t n)
 
    Allocates *n* bytes and returns a pointer of type :c:type:`void\*` to the
-   allocated memory, or *NULL* if the request fails.
+   allocated memory, or ``NULL`` if the request fails.
 
-   Requesting zero bytes returns a distinct non-*NULL* pointer if possible, as
+   Requesting zero bytes returns a distinct non-``NULL`` pointer if possible, as
    if ``PyMem_Malloc(1)`` had been called instead. The memory will not have
    been initialized in any way.
 
@@ -191,11 +191,11 @@ The :ref:`default memory allocator <default-memory-allocators>` uses the
 .. c:function:: void* PyMem_Calloc(size_t nelem, size_t elsize)
 
    Allocates *nelem* elements each whose size in bytes is *elsize* and returns
-   a pointer of type :c:type:`void\*` to the allocated memory, or *NULL* if the
+   a pointer of type :c:type:`void\*` to the allocated memory, or ``NULL`` if the
    request fails. The memory is initialized to zeros.
 
    Requesting zero elements or elements of size zero bytes returns a distinct
-   non-*NULL* pointer if possible, as if ``PyMem_Calloc(1, 1)`` had been called
+   non-``NULL`` pointer if possible, as if ``PyMem_Calloc(1, 1)`` had been called
    instead.
 
    .. versionadded:: 3.5
@@ -206,14 +206,14 @@ The :ref:`default memory allocator <default-memory-allocators>` uses the
    Resizes the memory block pointed to by *p* to *n* bytes. The contents will be
    unchanged to the minimum of the old and the new sizes.
 
-   If *p* is *NULL*, the call is equivalent to ``PyMem_Malloc(n)``; else if *n*
+   If *p* is ``NULL``, the call is equivalent to ``PyMem_Malloc(n)``; else if *n*
    is equal to zero, the memory block is resized but is not freed, and the
-   returned pointer is non-*NULL*.
+   returned pointer is non-``NULL``.
 
-   Unless *p* is *NULL*, it must have been returned by a previous call to
+   Unless *p* is ``NULL``, it must have been returned by a previous call to
    :c:func:`PyMem_Malloc`, :c:func:`PyMem_Realloc` or :c:func:`PyMem_Calloc`.
 
-   If the request fails, :c:func:`PyMem_Realloc` returns *NULL* and *p* remains
+   If the request fails, :c:func:`PyMem_Realloc` returns ``NULL`` and *p* remains
    a valid pointer to the previous memory area.
 
 
@@ -224,7 +224,7 @@ The :ref:`default memory allocator <default-memory-allocators>` uses the
    :c:func:`PyMem_Calloc`.  Otherwise, or if ``PyMem_Free(p)`` has been called
    before, undefined behavior occurs.
 
-   If *p* is *NULL*, no operation is performed.
+   If *p* is ``NULL``, no operation is performed.
 
 The following type-oriented macros are provided for convenience.  Note  that
 *TYPE* refers to any C type.
@@ -241,7 +241,7 @@ The following type-oriented macros are provided for convenience.  Note  that
 
    Same as :c:func:`PyMem_Realloc`, but the memory block is resized to ``(n *
    sizeof(TYPE))`` bytes.  Returns a pointer cast to :c:type:`TYPE\*`. On return,
-   *p* will be a pointer to the new memory area, or *NULL* in the event of
+   *p* will be a pointer to the new memory area, or ``NULL`` in the event of
    failure.
 
    This is a C preprocessor macro; *p* is always reassigned.  Save the original
@@ -283,9 +283,9 @@ The :ref:`default object allocator <default-memory-allocators>` uses the
 .. c:function:: void* PyObject_Malloc(size_t n)
 
    Allocates *n* bytes and returns a pointer of type :c:type:`void\*` to the
-   allocated memory, or *NULL* if the request fails.
+   allocated memory, or ``NULL`` if the request fails.
 
-   Requesting zero bytes returns a distinct non-*NULL* pointer if possible, as
+   Requesting zero bytes returns a distinct non-``NULL`` pointer if possible, as
    if ``PyObject_Malloc(1)`` had been called instead. The memory will not have
    been initialized in any way.
 
@@ -293,11 +293,11 @@ The :ref:`default object allocator <default-memory-allocators>` uses the
 .. c:function:: void* PyObject_Calloc(size_t nelem, size_t elsize)
 
    Allocates *nelem* elements each whose size in bytes is *elsize* and returns
-   a pointer of type :c:type:`void\*` to the allocated memory, or *NULL* if the
+   a pointer of type :c:type:`void\*` to the allocated memory, or ``NULL`` if the
    request fails. The memory is initialized to zeros.
 
    Requesting zero elements or elements of size zero bytes returns a distinct
-   non-*NULL* pointer if possible, as if ``PyObject_Calloc(1, 1)`` had been called
+   non-``NULL`` pointer if possible, as if ``PyObject_Calloc(1, 1)`` had been called
    instead.
 
    .. versionadded:: 3.5
@@ -308,14 +308,14 @@ The :ref:`default object allocator <default-memory-allocators>` uses the
    Resizes the memory block pointed to by *p* to *n* bytes. The contents will be
    unchanged to the minimum of the old and the new sizes.
 
-   If *p* is *NULL*, the call is equivalent to ``PyObject_Malloc(n)``; else if *n*
+   If *p* is ``NULL``, the call is equivalent to ``PyObject_Malloc(n)``; else if *n*
    is equal to zero, the memory block is resized but is not freed, and the
-   returned pointer is non-*NULL*.
+   returned pointer is non-``NULL``.
 
-   Unless *p* is *NULL*, it must have been returned by a previous call to
+   Unless *p* is ``NULL``, it must have been returned by a previous call to
    :c:func:`PyObject_Malloc`, :c:func:`PyObject_Realloc` or :c:func:`PyObject_Calloc`.
 
-   If the request fails, :c:func:`PyObject_Realloc` returns *NULL* and *p* remains
+   If the request fails, :c:func:`PyObject_Realloc` returns ``NULL`` and *p* remains
    a valid pointer to the previous memory area.
 
 
@@ -326,7 +326,7 @@ The :ref:`default object allocator <default-memory-allocators>` uses the
    :c:func:`PyObject_Calloc`.  Otherwise, or if ``PyObject_Free(p)`` has been called
    before, undefined behavior occurs.
 
-   If *p* is *NULL*, no operation is performed.
+   If *p* is ``NULL``, no operation is performed.
 
 
 .. _default-memory-allocators:
diff --git a/Doc/c-api/memoryview.rst b/Doc/c-api/memoryview.rst
index 9f6bfd751ade5..0fcd96dacfa6a 100644
--- a/Doc/c-api/memoryview.rst
+++ b/Doc/c-api/memoryview.rst
@@ -57,7 +57,7 @@ any other object.
 .. c:function:: Py_buffer *PyMemoryView_GET_BASE(PyObject *mview)
 
    Return either a pointer to the exporting object that the memoryview is based
-   on or *NULL* if the memoryview has been created by one of the functions
+   on or ``NULL`` if the memoryview has been created by one of the functions
    :c:func:`PyMemoryView_FromMemory` or :c:func:`PyMemoryView_FromBuffer`.
    *mview* **must** be a memoryview instance.
 
diff --git a/Doc/c-api/method.rst b/Doc/c-api/method.rst
index 7a2a84fe110dc..c08bc32ac83a2 100644
--- a/Doc/c-api/method.rst
+++ b/Doc/c-api/method.rst
@@ -21,7 +21,7 @@ to bind a :c:data:`PyCFunction` to a class object. It replaces the former call
 .. c:function:: int PyInstanceMethod_Check(PyObject *o)
 
    Return true if *o* is an instance method object (has type
-   :c:data:`PyInstanceMethod_Type`).  The parameter must not be *NULL*.
+   :c:data:`PyInstanceMethod_Type`).  The parameter must not be ``NULL``.
 
 
 .. c:function:: PyObject* PyInstanceMethod_New(PyObject *func)
@@ -64,14 +64,14 @@ no longer available.
 .. c:function:: int PyMethod_Check(PyObject *o)
 
    Return true if *o* is a method object (has type :c:data:`PyMethod_Type`).  The
-   parameter must not be *NULL*.
+   parameter must not be ``NULL``.
 
 
 .. c:function:: PyObject* PyMethod_New(PyObject *func, PyObject *self)
 
    Return a new method object, with *func* being any callable object and *self*
    the instance the method should be bound. *func* is the function that will
-   be called when the method is called. *self* must not be *NULL*.
+   be called when the method is called. *self* must not be ``NULL``.
 
 
 .. c:function:: PyObject* PyMethod_Function(PyObject *meth)
diff --git a/Doc/c-api/module.rst b/Doc/c-api/module.rst
index 7168dc7363cae..60c1bf48c60cb 100644
--- a/Doc/c-api/module.rst
+++ b/Doc/c-api/module.rst
@@ -61,7 +61,7 @@ Module Objects
    Return the dictionary object that implements *module*'s namespace; this object
    is the same as the :attr:`~object.__dict__` attribute of the module object.
    If *module* is not a module object (or a subtype of a module object),
-   :exc:`SystemError` is raised and *NULL* is returned.
+   :exc:`SystemError` is raised and ``NULL`` is returned.
 
    It is recommended extensions use other :c:func:`PyModule_\*` and
    :c:func:`PyObject_\*` functions rather than directly manipulate a module's
@@ -75,7 +75,7 @@ Module Objects
       single: SystemError (built-in exception)
 
    Return *module*'s :attr:`__name__` value.  If the module does not provide one,
-   or if it is not a string, :exc:`SystemError` is raised and *NULL* is returned.
+   or if it is not a string, :exc:`SystemError` is raised and ``NULL`` is returned.
 
    .. versionadded:: 3.3
 
@@ -88,14 +88,14 @@ Module Objects
 .. c:function:: void* PyModule_GetState(PyObject *module)
 
    Return the "state" of the module, that is, a pointer to the block of memory
-   allocated at module creation time, or *NULL*.  See
+   allocated at module creation time, or ``NULL``.  See
    :c:member:`PyModuleDef.m_size`.
 
 
 .. c:function:: PyModuleDef* PyModule_GetDef(PyObject *module)
 
    Return a pointer to the :c:type:`PyModuleDef` struct from which the module was
-   created, or *NULL* if the module wasn't created from a definition.
+   created, or ``NULL`` if the module wasn't created from a definition.
 
 
 .. c:function:: PyObject* PyModule_GetFilenameObject(PyObject *module)
@@ -106,7 +106,7 @@ Module Objects
 
    Return the name of the file from which *module* was loaded using *module*'s
    :attr:`__file__` attribute.  If this is not defined, or if it is not a
-   unicode string, raise :exc:`SystemError` and return *NULL*; otherwise return
+   unicode string, raise :exc:`SystemError` and return ``NULL``; otherwise return
    a reference to a Unicode object.
 
    .. versionadded:: 3.2
@@ -178,17 +178,17 @@ or request "multi-phase initialization" by returning the definition struct itsel
    .. c:member:: PyMethodDef* m_methods
 
       A pointer to a table of module-level functions, described by
-      :c:type:`PyMethodDef` values.  Can be *NULL* if no functions are present.
+      :c:type:`PyMethodDef` values.  Can be ``NULL`` if no functions are present.
 
    .. c:member:: PyModuleDef_Slot* m_slots
 
       An array of slot definitions for multi-phase initialization, terminated by
       a ``{0, NULL}`` entry.
-      When using single-phase initialization, *m_slots* must be *NULL*.
+      When using single-phase initialization, *m_slots* must be ``NULL``.
 
       .. versionchanged:: 3.5
 
-         Prior to version 3.5, this member was always set to *NULL*,
+         Prior to version 3.5, this member was always set to ``NULL``,
          and was defined as:
 
            .. c:member:: inquiry m_reload
@@ -196,20 +196,20 @@ or request "multi-phase initialization" by returning the definition struct itsel
    .. c:member:: traverseproc m_traverse
 
       A traversal function to call during GC traversal of the module object, or
-      *NULL* if not needed. This function may be called before module state
+      ``NULL`` if not needed. This function may be called before module state
       is allocated (:c:func:`PyModule_GetState()` may return `NULL`),
       and before the :c:member:`Py_mod_exec` function is executed.
 
    .. c:member:: inquiry m_clear
 
       A clear function to call during GC clearing of the module object, or
-      *NULL* if not needed. This function may be called before module state
+      ``NULL`` if not needed. This function may be called before module state
       is allocated (:c:func:`PyModule_GetState()` may return `NULL`),
       and before the :c:member:`Py_mod_exec` function is executed.
 
    .. c:member:: freefunc m_free
 
-      A function to call during deallocation of the module object, or *NULL* if
+      A function to call during deallocation of the module object, or ``NULL`` if
       not needed. This function may be called before module state
       is allocated (:c:func:`PyModule_GetState()` may return `NULL`),
       and before the :c:member:`Py_mod_exec` function is executed.
@@ -278,7 +278,7 @@ instance must be initialized with the following function:
    Ensures a module definition is a properly initialized Python object that
    correctly reports its type and reference count.
 
-   Returns *def* cast to ``PyObject*``, or *NULL* if an error occurred.
+   Returns *def* cast to ``PyObject*``, or ``NULL`` if an error occurred.
 
    .. versionadded:: 3.5
 
@@ -311,7 +311,7 @@ The available slot types are:
    The function receives a :py:class:`~importlib.machinery.ModuleSpec`
    instance, as defined in :PEP:`451`, and the module definition.
    It should return a new module object, or set an error
-   and return *NULL*.
+   and return ``NULL``.
 
    This function should be kept minimal. In particular, it should not
    call arbitrary Python code, as trying to import the same module again may
@@ -330,7 +330,7 @@ The available slot types are:
    :c:type:`PyModule_Type`. Any type can be used, as long as it supports
    setting and getting import-related attributes.
    However, only ``PyModule_Type`` instances may be returned if the
-   ``PyModuleDef`` has non-*NULL* ``m_traverse``, ``m_clear``,
+   ``PyModuleDef`` has non-``NULL`` ``m_traverse``, ``m_clear``,
    ``m_free``; non-zero ``m_size``; or slots other than ``Py_mod_create``.
 
 .. c:var:: Py_mod_exec
@@ -394,7 +394,7 @@ objects dynamically. Note that both ``PyModule_FromDefAndSpec`` and
 
 .. c:function:: int PyModule_AddFunctions(PyObject *module, PyMethodDef *functions)
 
-   Add the functions from the *NULL* terminated *functions* array to *module*.
+   Add the functions from the ``NULL`` terminated *functions* array to *module*.
    Refer to the :c:type:`PyMethodDef` documentation for details on individual
    entries (due to the lack of a shared module namespace, module level
    "functions" implemented in C typically receive the module as their first
@@ -445,7 +445,7 @@ state:
 
    Add a string constant to *module* as *name*.  This convenience function can be
    used from the module's initialization function.  The string *value* must be
-   *NULL*-terminated.  Return ``-1`` on error, ``0`` on success.
+   ``NULL``-terminated.  Return ``-1`` on error, ``0`` on success.
 
 
 .. c:function:: int PyModule_AddIntMacro(PyObject *module, macro)
@@ -476,7 +476,7 @@ since multiple such modules can be created from a single definition.
    Returns the module object that was created from *def* for the current interpreter.
    This method requires that the module object has been attached to the interpreter state with
    :c:func:`PyState_AddModule` beforehand. In case the corresponding module object is not
-   found or has not been attached to the interpreter state yet, it returns *NULL*.
+   found or has not been attached to the interpreter state yet, it returns ``NULL``.
 
 .. c:function:: int PyState_AddModule(PyObject *module, PyModuleDef *def)
 
diff --git a/Doc/c-api/number.rst b/Doc/c-api/number.rst
index 296b21c132b79..32c7a98052eae 100644
--- a/Doc/c-api/number.rst
+++ b/Doc/c-api/number.rst
@@ -14,25 +14,25 @@ Number Protocol
 
 .. c:function:: PyObject* PyNumber_Add(PyObject *o1, PyObject *o2)
 
-   Returns the result of adding *o1* and *o2*, or *NULL* on failure.  This is the
+   Returns the result of adding *o1* and *o2*, or ``NULL`` on failure.  This is the
    equivalent of the Python expression ``o1 + o2``.
 
 
 .. c:function:: PyObject* PyNumber_Subtract(PyObject *o1, PyObject *o2)
 
-   Returns the result of subtracting *o2* from *o1*, or *NULL* on failure.  This is
+   Returns the result of subtracting *o2* from *o1*, or ``NULL`` on failure.  This is
    the equivalent of the Python expression ``o1 - o2``.
 
 
 .. c:function:: PyObject* PyNumber_Multiply(PyObject *o1, PyObject *o2)
 
-   Returns the result of multiplying *o1* and *o2*, or *NULL* on failure.  This is
+   Returns the result of multiplying *o1* and *o2*, or ``NULL`` on failure.  This is
    the equivalent of the Python expression ``o1 * o2``.
 
 
 .. c:function:: PyObject* PyNumber_MatrixMultiply(PyObject *o1, PyObject *o2)
 
-   Returns the result of matrix multiplication on *o1* and *o2*, or *NULL* on
+   Returns the result of matrix multiplication on *o1* and *o2*, or ``NULL`` on
    failure.  This is the equivalent of the Python expression ``o1 @ o2``.
 
    .. versionadded:: 3.5
@@ -40,14 +40,14 @@ Number Protocol
 
 .. c:function:: PyObject* PyNumber_FloorDivide(PyObject *o1, PyObject *o2)
 
-   Return the floor of *o1* divided by *o2*, or *NULL* on failure.  This is
+   Return the floor of *o1* divided by *o2*, or ``NULL`` on failure.  This is
    equivalent to the "classic" division of integers.
 
 
 .. c:function:: PyObject* PyNumber_TrueDivide(PyObject *o1, PyObject *o2)
 
    Return a reasonable approximation for the mathematical value of *o1* divided by
-   *o2*, or *NULL* on failure.  The return value is "approximate" because binary
+   *o2*, or ``NULL`` on failure.  The return value is "approximate" because binary
    floating point numbers are approximate; it is not possible to represent all real
    numbers in base two.  This function can return a floating point value when
    passed two integers.
@@ -55,7 +55,7 @@ Number Protocol
 
 .. c:function:: PyObject* PyNumber_Remainder(PyObject *o1, PyObject *o2)
 
-   Returns the remainder of dividing *o1* by *o2*, or *NULL* on failure.  This is
+   Returns the remainder of dividing *o1* by *o2*, or ``NULL`` on failure.  This is
    the equivalent of the Python expression ``o1 % o2``.
 
 
@@ -63,7 +63,7 @@ Number Protocol
 
    .. index:: builtin: divmod
 
-   See the built-in function :func:`divmod`. Returns *NULL* on failure.  This is
+   See the built-in function :func:`divmod`. Returns ``NULL`` on failure.  This is
    the equivalent of the Python expression ``divmod(o1, o2)``.
 
 
@@ -71,21 +71,21 @@ Number Protocol
 
    .. index:: builtin: pow
 
-   See the built-in function :func:`pow`. Returns *NULL* on failure.  This is the
+   See the built-in function :func:`pow`. Returns ``NULL`` on failure.  This is the
    equivalent of the Python expression ``pow(o1, o2, o3)``, where *o3* is optional.
-   If *o3* is to be ignored, pass :c:data:`Py_None` in its place (passing *NULL* for
+   If *o3* is to be ignored, pass :c:data:`Py_None` in its place (passing ``NULL`` for
    *o3* would cause an illegal memory access).
 
 
 .. c:function:: PyObject* PyNumber_Negative(PyObject *o)
 
-   Returns the negation of *o* on success, or *NULL* on failure. This is the
+   Returns the negation of *o* on success, or ``NULL`` on failure. This is the
    equivalent of the Python expression ``-o``.
 
 
 .. c:function:: PyObject* PyNumber_Positive(PyObject *o)
 
-   Returns *o* on success, or *NULL* on failure.  This is the equivalent of the
+   Returns *o* on success, or ``NULL`` on failure.  This is the equivalent of the
    Python expression ``+o``.
 
 
@@ -93,70 +93,70 @@ Number Protocol
 
    .. index:: builtin: abs
 
-   Returns the absolute value of *o*, or *NULL* on failure.  This is the equivalent
+   Returns the absolute value of *o*, or ``NULL`` on failure.  This is the equivalent
    of the Python expression ``abs(o)``.
 
 
 .. c:function:: PyObject* PyNumber_Invert(PyObject *o)
 
-   Returns the bitwise negation of *o* on success, or *NULL* on failure.  This is
+   Returns the bitwise negation of *o* on success, or ``NULL`` on failure.  This is
    the equivalent of the Python expression ``~o``.
 
 
 .. c:function:: PyObject* PyNumber_Lshift(PyObject *o1, PyObject *o2)
 
-   Returns the result of left shifting *o1* by *o2* on success, or *NULL* on
+   Returns the result of left shifting *o1* by *o2* on success, or ``NULL`` on
    failure.  This is the equivalent of the Python expression ``o1 << o2``.
 
 
 .. c:function:: PyObject* PyNumber_Rshift(PyObject *o1, PyObject *o2)
 
-   Returns the result of right shifting *o1* by *o2* on success, or *NULL* on
+   Returns the result of right shifting *o1* by *o2* on success, or ``NULL`` on
    failure.  This is the equivalent of the Python expression ``o1 >> o2``.
 
 
 .. c:function:: PyObject* PyNumber_And(PyObject *o1, PyObject *o2)
 
-   Returns the "bitwise and" of *o1* and *o2* on success and *NULL* on failure.
+   Returns the "bitwise and" of *o1* and *o2* on success and ``NULL`` on failure.
    This is the equivalent of the Python expression ``o1 & o2``.
 
 
 .. c:function:: PyObject* PyNumber_Xor(PyObject *o1, PyObject *o2)
 
-   Returns the "bitwise exclusive or" of *o1* by *o2* on success, or *NULL* on
+   Returns the "bitwise exclusive or" of *o1* by *o2* on success, or ``NULL`` on
    failure.  This is the equivalent of the Python expression ``o1 ^ o2``.
 
 
 .. c:function:: PyObject* PyNumber_Or(PyObject *o1, PyObject *o2)
 
-   Returns the "bitwise or" of *o1* and *o2* on success, or *NULL* on failure.
+   Returns the "bitwise or" of *o1* and *o2* on success, or ``NULL`` on failure.
    This is the equivalent of the Python expression ``o1 | o2``.
 
 
 .. c:function:: PyObject* PyNumber_InPlaceAdd(PyObject *o1, PyObject *o2)
 
-   Returns the result of adding *o1* and *o2*, or *NULL* on failure.  The operation
+   Returns the result of adding *o1* and *o2*, or ``NULL`` on failure.  The operation
    is done *in-place* when *o1* supports it.  This is the equivalent of the Python
    statement ``o1 += o2``.
 
 
 .. c:function:: PyObject* PyNumber_InPlaceSubtract(PyObject *o1, PyObject *o2)
 
-   Returns the result of subtracting *o2* from *o1*, or *NULL* on failure.  The
+   Returns the result of subtracting *o2* from *o1*, or ``NULL`` on failure.  The
    operation is done *in-place* when *o1* supports it.  This is the equivalent of
    the Python statement ``o1 -= o2``.
 
 
 .. c:function:: PyObject* PyNumber_InPlaceMultiply(PyObject *o1, PyObject *o2)
 
-   Returns the result of multiplying *o1* and *o2*, or *NULL* on failure.  The
+   Returns the result of multiplying *o1* and *o2*, or ``NULL`` on failure.  The
    operation is done *in-place* when *o1* supports it.  This is the equivalent of
    the Python statement ``o1 *= o2``.
 
 
 .. c:function:: PyObject* PyNumber_InPlaceMatrixMultiply(PyObject *o1, PyObject *o2)
 
-   Returns the result of matrix multiplication on *o1* and *o2*, or *NULL* on
+   Returns the result of matrix multiplication on *o1* and *o2*, or ``NULL`` on
    failure.  The operation is done *in-place* when *o1* supports it.  This is
    the equivalent of the Python statement ``o1 @= o2``.
 
@@ -165,7 +165,7 @@ Number Protocol
 
 .. c:function:: PyObject* PyNumber_InPlaceFloorDivide(PyObject *o1, PyObject *o2)
 
-   Returns the mathematical floor of dividing *o1* by *o2*, or *NULL* on failure.
+   Returns the mathematical floor of dividing *o1* by *o2*, or ``NULL`` on failure.
    The operation is done *in-place* when *o1* supports it.  This is the equivalent
    of the Python statement ``o1 //= o2``.
 
@@ -173,7 +173,7 @@ Number Protocol
 .. c:function:: PyObject* PyNumber_InPlaceTrueDivide(PyObject *o1, PyObject *o2)
 
    Return a reasonable approximation for the mathematical value of *o1* divided by
-   *o2*, or *NULL* on failure.  The return value is "approximate" because binary
+   *o2*, or ``NULL`` on failure.  The return value is "approximate" because binary
    floating point numbers are approximate; it is not possible to represent all real
    numbers in base two.  This function can return a floating point value when
    passed two integers.  The operation is done *in-place* when *o1* supports it.
@@ -181,7 +181,7 @@ Number Protocol
 
 .. c:function:: PyObject* PyNumber_InPlaceRemainder(PyObject *o1, PyObject *o2)
 
-   Returns the remainder of dividing *o1* by *o2*, or *NULL* on failure.  The
+   Returns the remainder of dividing *o1* by *o2*, or ``NULL`` on failure.  The
    operation is done *in-place* when *o1* supports it.  This is the equivalent of
    the Python statement ``o1 %= o2``.
 
@@ -190,44 +190,44 @@ Number Protocol
 
    .. index:: builtin: pow
 
-   See the built-in function :func:`pow`. Returns *NULL* on failure.  The operation
+   See the built-in function :func:`pow`. Returns ``NULL`` on failure.  The operation
    is done *in-place* when *o1* supports it.  This is the equivalent of the Python
    statement ``o1 **= o2`` when o3 is :c:data:`Py_None`, or an in-place variant of
    ``pow(o1, o2, o3)`` otherwise. If *o3* is to be ignored, pass :c:data:`Py_None`
-   in its place (passing *NULL* for *o3* would cause an illegal memory access).
+   in its place (passing ``NULL`` for *o3* would cause an illegal memory access).
 
 
 .. c:function:: PyObject* PyNumber_InPlaceLshift(PyObject *o1, PyObject *o2)
 
-   Returns the result of left shifting *o1* by *o2* on success, or *NULL* on
+   Returns the result of left shifting *o1* by *o2* on success, or ``NULL`` on
    failure.  The operation is done *in-place* when *o1* supports it.  This is the
    equivalent of the Python statement ``o1 <<= o2``.
 
 
 .. c:function:: PyObject* PyNumber_InPlaceRshift(PyObject *o1, PyObject *o2)
 
-   Returns the result of right shifting *o1* by *o2* on success, or *NULL* on
+   Returns the result of right shifting *o1* by *o2* on success, or ``NULL`` on
    failure.  The operation is done *in-place* when *o1* supports it.  This is the
    equivalent of the Python statement ``o1 >>= o2``.
 
 
 .. c:function:: PyObject* PyNumber_InPlaceAnd(PyObject *o1, PyObject *o2)
 
-   Returns the "bitwise and" of *o1* and *o2* on success and *NULL* on failure. The
+   Returns the "bitwise and" of *o1* and *o2* on success and ``NULL`` on failure. The
    operation is done *in-place* when *o1* supports it.  This is the equivalent of
    the Python statement ``o1 &= o2``.
 
 
 .. c:function:: PyObject* PyNumber_InPlaceXor(PyObject *o1, PyObject *o2)
 
-   Returns the "bitwise exclusive or" of *o1* by *o2* on success, or *NULL* on
+   Returns the "bitwise exclusive or" of *o1* by *o2* on success, or ``NULL`` on
    failure.  The operation is done *in-place* when *o1* supports it.  This is the
    equivalent of the Python statement ``o1 ^= o2``.
 
 
 .. c:function:: PyObject* PyNumber_InPlaceOr(PyObject *o1, PyObject *o2)
 
-   Returns the "bitwise or" of *o1* and *o2* on success, or *NULL* on failure.  The
+   Returns the "bitwise or" of *o1* and *o2* on success, or ``NULL`` on failure.  The
    operation is done *in-place* when *o1* supports it.  This is the equivalent of
    the Python statement ``o1 |= o2``.
 
@@ -236,7 +236,7 @@ Number Protocol
 
    .. index:: builtin: int
 
-   Returns the *o* converted to an integer object on success, or *NULL* on
+   Returns the *o* converted to an integer object on success, or ``NULL`` on
    failure.  This is the equivalent of the Python expression ``int(o)``.
 
 
@@ -244,13 +244,13 @@ Number Protocol
 
    .. index:: builtin: float
 
-   Returns the *o* converted to a float object on success, or *NULL* on failure.
+   Returns the *o* converted to a float object on success, or ``NULL`` on failure.
    This is the equivalent of the Python expression ``float(o)``.
 
 
 .. c:function:: PyObject* PyNumber_Index(PyObject *o)
 
-   Returns the *o* converted to a Python int on success or *NULL* with a
+   Returns the *o* converted to a Python int on success or ``NULL`` with a
    :exc:`TypeError` exception raised on failure.
 
 
@@ -271,7 +271,7 @@ Number Protocol
    If *o* can be converted to a Python int but the attempt to
    convert to a Py_ssize_t value would raise an :exc:`OverflowError`, then the
    *exc* argument is the type of exception that will be raised (usually
-   :exc:`IndexError` or :exc:`OverflowError`).  If *exc* is *NULL*, then the
+   :exc:`IndexError` or :exc:`OverflowError`).  If *exc* is ``NULL``, then the
    exception is cleared and the value is clipped to *PY_SSIZE_T_MIN* for a negative
    integer or *PY_SSIZE_T_MAX* for a positive integer.
 
diff --git a/Doc/c-api/object.rst b/Doc/c-api/object.rst
index c09d97a66c013..d81dc938e4f96 100644
--- a/Doc/c-api/object.rst
+++ b/Doc/c-api/object.rst
@@ -53,14 +53,14 @@ Object Protocol
 .. c:function:: PyObject* PyObject_GetAttr(PyObject *o, PyObject *attr_name)
 
    Retrieve an attribute named *attr_name* from object *o*. Returns the attribute
-   value on success, or *NULL* on failure.  This is the equivalent of the Python
+   value on success, or ``NULL`` on failure.  This is the equivalent of the Python
    expression ``o.attr_name``.
 
 
 .. c:function:: PyObject* PyObject_GetAttrString(PyObject *o, const char *attr_name)
 
    Retrieve an attribute named *attr_name* from object *o*. Returns the attribute
-   value on success, or *NULL* on failure. This is the equivalent of the Python
+   value on success, or ``NULL`` on failure. This is the equivalent of the Python
    expression ``o.attr_name``.
 
 
@@ -81,7 +81,7 @@ Object Protocol
    return ``0`` on success.  This is the equivalent of the Python statement
    ``o.attr_name = v``.
 
-   If *v* is *NULL*, the attribute is deleted, however this feature is
+   If *v* is ``NULL``, the attribute is deleted, however this feature is
    deprecated in favour of using :c:func:`PyObject_DelAttr`.
 
 
@@ -92,7 +92,7 @@ Object Protocol
    return ``0`` on success.  This is the equivalent of the Python statement
    ``o.attr_name = v``.
 
-   If *v* is *NULL*, the attribute is deleted, however this feature is
+   If *v* is ``NULL``, the attribute is deleted, however this feature is
    deprecated in favour of using :c:func:`PyObject_DelAttrString`.
 
 
@@ -143,7 +143,7 @@ Object Protocol
    :const:`Py_NE`, :const:`Py_GT`, or :const:`Py_GE`, corresponding to ``<``,
    ``<=``, ``==``, ``!=``, ``>``, or ``>=`` respectively. This is the equivalent of
    the Python expression ``o1 op o2``, where ``op`` is the operator corresponding
-   to *opid*. Returns the value of the comparison on success, or *NULL* on failure.
+   to *opid*. Returns the value of the comparison on success, or ``NULL`` on failure.
 
 
 .. c:function:: int PyObject_RichCompareBool(PyObject *o1, PyObject *o2, int opid)
@@ -165,7 +165,7 @@ Object Protocol
    .. index:: builtin: repr
 
    Compute a string representation of object *o*.  Returns the string
-   representation on success, *NULL* on failure.  This is the equivalent of the
+   representation on success, ``NULL`` on failure.  This is the equivalent of the
    Python expression ``repr(o)``.  Called by the :func:`repr` built-in function.
 
    .. versionchanged:: 3.4
@@ -188,7 +188,7 @@ Object Protocol
 .. c:function:: PyObject* PyObject_Str(PyObject *o)
 
    Compute a string representation of object *o*.  Returns the string
-   representation on success, *NULL* on failure.  This is the equivalent of the
+   representation on success, ``NULL`` on failure.  This is the equivalent of the
    Python expression ``str(o)``.  Called by the :func:`str` built-in function
    and, therefore, by the :func:`print` function.
 
@@ -200,7 +200,7 @@ Object Protocol
 
    .. index:: builtin: bytes
 
-   Compute a bytes representation of object *o*.  *NULL* is returned on
+   Compute a bytes representation of object *o*.  ``NULL`` is returned on
    failure and a bytes object on success.  This is equivalent to the Python
    expression ``bytes(o)``, when *o* is not an integer.  Unlike ``bytes(o)``,
    a TypeError is raised when *o* is an integer instead of a zero-initialized
@@ -258,11 +258,11 @@ Object Protocol
    Call a callable Python object *callable*, with arguments given by the
    tuple *args*, and named arguments given by the dictionary *kwargs*.
 
-   *args* must not be *NULL*, use an empty tuple if no arguments are needed.
-   If no named arguments are needed, *kwargs* can be *NULL*.
+   *args* must not be ``NULL``, use an empty tuple if no arguments are needed.
+   If no named arguments are needed, *kwargs* can be ``NULL``.
 
    Return the result of the call on success, or raise an exception and return
-   *NULL* on failure.
+   ``NULL`` on failure.
 
    This is the equivalent of the Python expression:
    ``callable(*args, **kwargs)``.
@@ -271,10 +271,10 @@ Object Protocol
 .. c:function:: PyObject* PyObject_CallObject(PyObject *callable, PyObject *args)
 
    Call a callable Python object *callable*, with arguments given by the
-   tuple *args*.  If no arguments are needed, then *args* can be *NULL*.
+   tuple *args*.  If no arguments are needed, then *args* can be ``NULL``.
 
    Return the result of the call on success, or raise an exception and return
-   *NULL* on failure.
+   ``NULL`` on failure.
 
    This is the equivalent of the Python expression: ``callable(*args)``.
 
@@ -283,10 +283,10 @@ Object Protocol
 
    Call a callable Python object *callable*, with a variable number of C arguments.
    The C arguments are described using a :c:func:`Py_BuildValue` style format
-   string.  The format can be *NULL*, indicating that no arguments are provided.
+   string.  The format can be ``NULL``, indicating that no arguments are provided.
 
    Return the result of the call on success, or raise an exception and return
-   *NULL* on failure.
+   ``NULL`` on failure.
 
    This is the equivalent of the Python expression: ``callable(*args)``.
 
@@ -303,10 +303,10 @@ Object Protocol
    arguments.  The C arguments are described by a :c:func:`Py_BuildValue` format
    string that should  produce a tuple.
 
-   The format can be *NULL*, indicating that no arguments are provided.
+   The format can be ``NULL``, indicating that no arguments are provided.
 
    Return the result of the call on success, or raise an exception and return
-   *NULL* on failure.
+   ``NULL`` on failure.
 
    This is the equivalent of the Python expression:
    ``obj.name(arg1, arg2, ...)``.
@@ -322,10 +322,10 @@ Object Protocol
 
    Call a callable Python object *callable*, with a variable number of
    :c:type:`PyObject\*` arguments.  The arguments are provided as a variable number
-   of parameters followed by *NULL*.
+   of parameters followed by ``NULL``.
 
    Return the result of the call on success, or raise an exception and return
-   *NULL* on failure.
+   ``NULL`` on failure.
 
    This is the equivalent of the Python expression:
    ``callable(arg1, arg2, ...)``.
@@ -336,10 +336,10 @@ Object Protocol
    Calls a method of the Python object *obj*, where the name of the method is given as a
    Python string object in *name*.  It is called with a variable number of
    :c:type:`PyObject\*` arguments.  The arguments are provided as a variable number
-   of parameters followed by *NULL*.
+   of parameters followed by ``NULL``.
 
    Return the result of the call on success, or raise an exception and return
-   *NULL* on failure.
+   ``NULL`` on failure.
 
 
 .. c:function:: Py_hash_t PyObject_Hash(PyObject *o)
@@ -380,8 +380,8 @@ Object Protocol
 
    .. index:: builtin: type
 
-   When *o* is non-*NULL*, returns a type object corresponding to the object type
-   of object *o*. On failure, raises :exc:`SystemError` and returns *NULL*.  This
+   When *o* is non-``NULL``, returns a type object corresponding to the object type
+   of object *o*. On failure, raises :exc:`SystemError` and returns ``NULL``.  This
    is equivalent to the Python expression ``type(o)``. This function increments the
    reference count of the return value. There's really no reason to use this
    function instead of the common expression ``o->ob_type``, which returns a
@@ -392,7 +392,7 @@ Object Protocol
 .. c:function:: int PyObject_TypeCheck(PyObject *o, PyTypeObject *type)
 
    Return true if the object *o* is of type *type* or a subtype of *type*.  Both
-   parameters must be non-*NULL*.
+   parameters must be non-``NULL``.
 
 
 .. c:function:: Py_ssize_t PyObject_Size(PyObject *o)
@@ -417,7 +417,7 @@ Object Protocol
 
 .. c:function:: PyObject* PyObject_GetItem(PyObject *o, PyObject *key)
 
-   Return element of *o* corresponding to the object *key* or *NULL* on failure.
+   Return element of *o* corresponding to the object *key* or ``NULL`` on failure.
    This is the equivalent of the Python expression ``o[key]``.
 
 
@@ -437,15 +437,15 @@ Object Protocol
 .. c:function:: PyObject* PyObject_Dir(PyObject *o)
 
    This is equivalent to the Python expression ``dir(o)``, returning a (possibly
-   empty) list of strings appropriate for the object argument, or *NULL* if there
-   was an error.  If the argument is *NULL*, this is like the Python ``dir()``,
+   empty) list of strings appropriate for the object argument, or ``NULL`` if there
+   was an error.  If the argument is ``NULL``, this is like the Python ``dir()``,
    returning the names of the current locals; in this case, if no execution frame
-   is active then *NULL* is returned but :c:func:`PyErr_Occurred` will return false.
+   is active then ``NULL`` is returned but :c:func:`PyErr_Occurred` will return false.
 
 
 .. c:function:: PyObject* PyObject_GetIter(PyObject *o)
 
    This is equivalent to the Python expression ``iter(o)``. It returns a new
    iterator for the object argument, or the object  itself if the object is already
-   an iterator.  Raises :exc:`TypeError` and returns *NULL* if the object cannot be
+   an iterator.  Raises :exc:`TypeError` and returns ``NULL`` if the object cannot be
    iterated.
diff --git a/Doc/c-api/refcounting.rst b/Doc/c-api/refcounting.rst
index 4f512ecdbe9fb..1479db42d70f4 100644
--- a/Doc/c-api/refcounting.rst
+++ b/Doc/c-api/refcounting.rst
@@ -13,22 +13,22 @@ objects.
 
 .. c:function:: void Py_INCREF(PyObject *o)
 
-   Increment the reference count for object *o*.  The object must not be *NULL*; if
-   you aren't sure that it isn't *NULL*, use :c:func:`Py_XINCREF`.
+   Increment the reference count for object *o*.  The object must not be ``NULL``; if
+   you aren't sure that it isn't ``NULL``, use :c:func:`Py_XINCREF`.
 
 
 .. c:function:: void Py_XINCREF(PyObject *o)
 
-   Increment the reference count for object *o*.  The object may be *NULL*, in
+   Increment the reference count for object *o*.  The object may be ``NULL``, in
    which case the macro has no effect.
 
 
 .. c:function:: void Py_DECREF(PyObject *o)
 
-   Decrement the reference count for object *o*.  The object must not be *NULL*; if
-   you aren't sure that it isn't *NULL*, use :c:func:`Py_XDECREF`.  If the reference
+   Decrement the reference count for object *o*.  The object must not be ``NULL``; if
+   you aren't sure that it isn't ``NULL``, use :c:func:`Py_XDECREF`.  If the reference
    count reaches zero, the object's type's deallocation function (which must not be
-   *NULL*) is invoked.
+   ``NULL``) is invoked.
 
    .. warning::
 
@@ -44,18 +44,18 @@ objects.
 
 .. c:function:: void Py_XDECREF(PyObject *o)
 
-   Decrement the reference count for object *o*.  The object may be *NULL*, in
+   Decrement the reference count for object *o*.  The object may be ``NULL``, in
    which case the macro has no effect; otherwise the effect is the same as for
    :c:func:`Py_DECREF`, and the same warning applies.
 
 
 .. c:function:: void Py_CLEAR(PyObject *o)
 
-   Decrement the reference count for object *o*.  The object may be *NULL*, in
+   Decrement the reference count for object *o*.  The object may be ``NULL``, in
    which case the macro has no effect; otherwise the effect is the same as for
-   :c:func:`Py_DECREF`, except that the argument is also set to *NULL*.  The warning
+   :c:func:`Py_DECREF`, except that the argument is also set to ``NULL``.  The warning
    for :c:func:`Py_DECREF` does not apply with respect to the object passed because
-   the macro carefully uses a temporary variable and sets the argument to *NULL*
+   the macro carefully uses a temporary variable and sets the argument to ``NULL``
    before decrementing its reference count.
 
    It is a good idea to use this macro whenever decrementing the value of a
diff --git a/Doc/c-api/reflection.rst b/Doc/c-api/reflection.rst
index 96893652f730c..cb9a8f8a180cf 100644
--- a/Doc/c-api/reflection.rst
+++ b/Doc/c-api/reflection.rst
@@ -14,18 +14,18 @@ Reflection
 .. c:function:: PyObject* PyEval_GetLocals()
 
    Return a dictionary of the local variables in the current execution frame,
-   or *NULL* if no frame is currently executing.
+   or ``NULL`` if no frame is currently executing.
 
 
 .. c:function:: PyObject* PyEval_GetGlobals()
 
    Return a dictionary of the global variables in the current execution frame,
-   or *NULL* if no frame is currently executing.
+   or ``NULL`` if no frame is currently executing.
 
 
 .. c:function:: PyFrameObject* PyEval_GetFrame()
 
-   Return the current thread state's frame, which is *NULL* if no frame is
+   Return the current thread state's frame, which is ``NULL`` if no frame is
    currently executing.
 
 
diff --git a/Doc/c-api/sequence.rst b/Doc/c-api/sequence.rst
index d653319f8064b..7db618a1004fa 100644
--- a/Doc/c-api/sequence.rst
+++ b/Doc/c-api/sequence.rst
@@ -26,39 +26,39 @@ Sequence Protocol
 
 .. c:function:: PyObject* PySequence_Concat(PyObject *o1, PyObject *o2)
 
-   Return the concatenation of *o1* and *o2* on success, and *NULL* on failure.
+   Return the concatenation of *o1* and *o2* on success, and ``NULL`` on failure.
    This is the equivalent of the Python expression ``o1 + o2``.
 
 
 .. c:function:: PyObject* PySequence_Repeat(PyObject *o, Py_ssize_t count)
 
-   Return the result of repeating sequence object *o* *count* times, or *NULL* on
+   Return the result of repeating sequence object *o* *count* times, or ``NULL`` on
    failure.  This is the equivalent of the Python expression ``o * count``.
 
 
 .. c:function:: PyObject* PySequence_InPlaceConcat(PyObject *o1, PyObject *o2)
 
-   Return the concatenation of *o1* and *o2* on success, and *NULL* on failure.
+   Return the concatenation of *o1* and *o2* on success, and ``NULL`` on failure.
    The operation is done *in-place* when *o1* supports it.  This is the equivalent
    of the Python expression ``o1 += o2``.
 
 
 .. c:function:: PyObject* PySequence_InPlaceRepeat(PyObject *o, Py_ssize_t count)
 
-   Return the result of repeating sequence object *o* *count* times, or *NULL* on
+   Return the result of repeating sequence object *o* *count* times, or ``NULL`` on
    failure.  The operation is done *in-place* when *o* supports it.  This is the
    equivalent of the Python expression ``o *= count``.
 
 
 .. c:function:: PyObject* PySequence_GetItem(PyObject *o, Py_ssize_t i)
 
-   Return the *i*\ th element of *o*, or *NULL* on failure. This is the equivalent of
+   Return the *i*\ th element of *o*, or ``NULL`` on failure. This is the equivalent of
    the Python expression ``o[i]``.
 
 
 .. c:function:: PyObject* PySequence_GetSlice(PyObject *o, Py_ssize_t i1, Py_ssize_t i2)
 
-   Return the slice of sequence object *o* between *i1* and *i2*, or *NULL* on
+   Return the slice of sequence object *o* between *i1* and *i2*, or ``NULL`` on
    failure. This is the equivalent of the Python expression ``o[i1:i2]``.
 
 
@@ -69,7 +69,7 @@ Sequence Protocol
    is the equivalent of the Python statement ``o[i] = v``.  This function *does
    not* steal a reference to *v*.
 
-   If *v* is *NULL*, the element is deleted, however this feature is
+   If *v* is ``NULL``, the element is deleted, however this feature is
    deprecated in favour of using :c:func:`PySequence_DelItem`.
 
 
@@ -114,7 +114,7 @@ Sequence Protocol
 .. c:function:: PyObject* PySequence_List(PyObject *o)
 
    Return a list object with the same contents as the sequence or iterable *o*,
-   or *NULL* on failure.  The returned list is guaranteed to be new.  This is
+   or ``NULL`` on failure.  The returned list is guaranteed to be new.  This is
    equivalent to the Python expression ``list(o)``.
 
 
@@ -123,7 +123,7 @@ Sequence Protocol
    .. index:: builtin: tuple
 
    Return a tuple object with the same contents as the sequence or iterable *o*,
-   or *NULL* on failure.  If *o* is a tuple, a new reference will be returned,
+   or ``NULL`` on failure.  If *o* is a tuple, a new reference will be returned,
    otherwise a tuple will be constructed with the appropriate contents.  This is
    equivalent to the Python expression ``tuple(o)``.
 
@@ -133,7 +133,7 @@ Sequence Protocol
    Return the sequence or iterable *o* as an object usable by the other
    ``PySequence_Fast*`` family of functions. If the object is not a sequence or
    iterable, raises :exc:`TypeError` with *m* as the message text. Returns
-   *NULL* on failure.
+   ``NULL`` on failure.
 
    The ``PySequence_Fast*`` functions are thus named because they assume
    *o* is a :c:type:`PyTupleObject` or a :c:type:`PyListObject` and access
@@ -146,7 +146,7 @@ Sequence Protocol
 .. c:function:: Py_ssize_t PySequence_Fast_GET_SIZE(PyObject *o)
 
    Returns the length of *o*, assuming that *o* was returned by
-   :c:func:`PySequence_Fast` and that *o* is not *NULL*.  The size can also be
+   :c:func:`PySequence_Fast` and that *o* is not ``NULL``.  The size can also be
    gotten by calling :c:func:`PySequence_Size` on *o*, but
    :c:func:`PySequence_Fast_GET_SIZE` is faster because it can assume *o* is a
    list or tuple.
@@ -155,13 +155,13 @@ Sequence Protocol
 .. c:function:: PyObject* PySequence_Fast_GET_ITEM(PyObject *o, Py_ssize_t i)
 
    Return the *i*\ th element of *o*, assuming that *o* was returned by
-   :c:func:`PySequence_Fast`, *o* is not *NULL*, and that *i* is within bounds.
+   :c:func:`PySequence_Fast`, *o* is not ``NULL``, and that *i* is within bounds.
 
 
 .. c:function:: PyObject** PySequence_Fast_ITEMS(PyObject *o)
 
    Return the underlying array of PyObject pointers.  Assumes that *o* was returned
-   by :c:func:`PySequence_Fast` and *o* is not *NULL*.
+   by :c:func:`PySequence_Fast` and *o* is not ``NULL``.
 
    Note, if a list gets resized, the reallocation may relocate the items array.
    So, only use the underlying array pointer in contexts where the sequence
@@ -170,7 +170,7 @@ Sequence Protocol
 
 .. c:function:: PyObject* PySequence_ITEM(PyObject *o, Py_ssize_t i)
 
-   Return the *i*\ th element of *o* or *NULL* on failure. Faster form of
+   Return the *i*\ th element of *o* or ``NULL`` on failure. Faster form of
    :c:func:`PySequence_GetItem` but without checking that
    :c:func:`PySequence_Check` on *o* is true and without adjustment for negative
    indices.
diff --git a/Doc/c-api/set.rst b/Doc/c-api/set.rst
index 64b6dde3b7482..34ee077b88de9 100644
--- a/Doc/c-api/set.rst
+++ b/Doc/c-api/set.rst
@@ -80,8 +80,8 @@ the constructor functions work with any iterable Python object.
 .. c:function:: PyObject* PySet_New(PyObject *iterable)
 
    Return a new :class:`set` containing objects returned by the *iterable*.  The
-   *iterable* may be *NULL* to create a new empty set.  Return the new set on
-   success or *NULL* on failure.  Raise :exc:`TypeError` if *iterable* is not
+   *iterable* may be ``NULL`` to create a new empty set.  Return the new set on
+   success or ``NULL`` on failure.  Raise :exc:`TypeError` if *iterable* is not
    actually iterable.  The constructor is also useful for copying a set
    (``c=set(s)``).
 
@@ -89,8 +89,8 @@ the constructor functions work with any iterable Python object.
 .. c:function:: PyObject* PyFrozenSet_New(PyObject *iterable)
 
    Return a new :class:`frozenset` containing objects returned by the *iterable*.
-   The *iterable* may be *NULL* to create a new empty frozenset.  Return the new
-   set on success or *NULL* on failure.  Raise :exc:`TypeError` if *iterable* is
+   The *iterable* may be ``NULL`` to create a new empty frozenset.  Return the new
+   set on success or ``NULL`` on failure.  Raise :exc:`TypeError` if *iterable* is
    not actually iterable.
 
 
@@ -149,7 +149,7 @@ subtypes but not for instances of :class:`frozenset` or its subtypes.
 .. c:function:: PyObject* PySet_Pop(PyObject *set)
 
    Return a new reference to an arbitrary object in the *set*, and removes the
-   object from the *set*.  Return *NULL* on failure.  Raise :exc:`KeyError` if the
+   object from the *set*.  Return ``NULL`` on failure.  Raise :exc:`KeyError` if the
    set is empty. Raise a :exc:`SystemError` if *set* is not an instance of
    :class:`set` or its subtype.
 
diff --git a/Doc/c-api/slice.rst b/Doc/c-api/slice.rst
index 8ad9a29b256eb..c72ab04cbc543 100644
--- a/Doc/c-api/slice.rst
+++ b/Doc/c-api/slice.rst
@@ -14,15 +14,15 @@ Slice Objects
 
 .. c:function:: int PySlice_Check(PyObject *ob)
 
-   Return true if *ob* is a slice object; *ob* must not be *NULL*.
+   Return true if *ob* is a slice object; *ob* must not be ``NULL``.
 
 
 .. c:function:: PyObject* PySlice_New(PyObject *start, PyObject *stop, PyObject *step)
 
    Return a new slice object with the given values.  The *start*, *stop*, and
    *step* parameters are used as the values of the slice object attributes of
-   the same names.  Any of the values may be *NULL*, in which case the
-   ``None`` will be used for the corresponding attribute.  Return *NULL* if
+   the same names.  Any of the values may be ``NULL``, in which case the
+   ``None`` will be used for the corresponding attribute.  Return ``NULL`` if
    the new object could not be allocated.
 
 
diff --git a/Doc/c-api/structures.rst b/Doc/c-api/structures.rst
index 274beeef5d5bb..0aa1ef026c14e 100644
--- a/Doc/c-api/structures.rst
+++ b/Doc/c-api/structures.rst
@@ -106,8 +106,8 @@ the definition of all other Python objects.
 
    Type of the functions used to implement most Python callables in C.
    Functions of this type take two :c:type:`PyObject\*` parameters and return
-   one such value.  If the return value is *NULL*, an exception shall have
-   been set.  If not *NULL*, the return value is interpreted as the return
+   one such value.  If the return value is ``NULL``, an exception shall have
+   been set.  If not ``NULL``, the return value is interpreted as the return
    value of the function as exposed in Python.  The function must return a new
    reference.
 
@@ -179,7 +179,7 @@ also keyword arguments.  So there are a total of 6 calling conventions:
 
    Methods with these flags must be of type :c:type:`PyCFunctionWithKeywords`.
    The function expects three parameters: *self*, *args*, *kwargs* where
-   *kwargs* is a dictionary of all the keyword arguments or possibly *NULL*
+   *kwargs* is a dictionary of all the keyword arguments or possibly ``NULL``
    if there are no keyword arguments.  The parameters are typically processed
    using :c:func:`PyArg_ParseTupleAndKeywords`.
 
@@ -204,7 +204,7 @@ also keyword arguments.  So there are a total of 6 calling conventions:
    Keyword arguments are passed the same way as in the vectorcall protocol:
    there is an additional fourth :c:type:`PyObject\*` parameter
    which is a tuple representing the names of the keyword arguments
-   or possibly *NULL* if there are no keywords.  The values of the keyword
+   or possibly ``NULL`` if there are no keywords.  The values of the keyword
    arguments are stored in the *args* array, after the positional arguments.
 
    This is not part of the :ref:`limited API <stable>`.
@@ -218,7 +218,7 @@ also keyword arguments.  So there are a total of 6 calling conventions:
    they are listed with the :const:`METH_NOARGS` flag.  They need to be of type
    :c:type:`PyCFunction`.  The first parameter is typically named *self* and will
    hold a reference to the module or object instance.  In all cases the second
-   parameter will be *NULL*.
+   parameter will be ``NULL``.
 
 
 .. data:: METH_O
@@ -249,7 +249,7 @@ method.
 
    .. index:: builtin: staticmethod
 
-   The method will be passed *NULL* as the first parameter rather than an
+   The method will be passed ``NULL`` as the first parameter rather than an
    instance of the type.  This is used to create *static methods*, similar to
    what is created when using the :func:`staticmethod` built-in function.
 
@@ -323,7 +323,7 @@ definition with the same method name.
    =============== ==================
 
    :c:macro:`T_OBJECT` and :c:macro:`T_OBJECT_EX` differ in that
-   :c:macro:`T_OBJECT` returns ``None`` if the member is *NULL* and
+   :c:macro:`T_OBJECT` returns ``None`` if the member is ``NULL`` and
    :c:macro:`T_OBJECT_EX` raises an :exc:`AttributeError`.  Try to use
    :c:macro:`T_OBJECT_EX` over :c:macro:`T_OBJECT` because :c:macro:`T_OBJECT_EX`
    handles use of the :keyword:`del` statement on that attribute more correctly
@@ -333,7 +333,7 @@ definition with the same method name.
    read-only access.  Using :c:macro:`T_STRING` for :attr:`type` implies
    :c:macro:`READONLY`.  :c:macro:`T_STRING` data is interpreted as UTF-8.
    Only :c:macro:`T_OBJECT` and :c:macro:`T_OBJECT_EX`
-   members can be deleted.  (They are set to *NULL*).
+   members can be deleted.  (They are set to ``NULL``).
 
 
 .. c:type:: PyGetSetDef
@@ -364,7 +364,7 @@ definition with the same method name.
 
       typedef PyObject *(*getter)(PyObject *, void *);
 
-   It should return a new reference on success or *NULL* with a set exception
+   It should return a new reference on success or ``NULL`` with a set exception
    on failure.
 
    ``set`` functions take two :c:type:`PyObject\*` parameters (the instance and
@@ -372,5 +372,5 @@ definition with the same method name.
 
       typedef int (*setter)(PyObject *, PyObject *, void *);
 
-   In case the attribute should be deleted the second parameter is *NULL*.
+   In case the attribute should be deleted the second parameter is ``NULL``.
    Should return ``0`` on success or ``-1`` with a set exception on failure.
diff --git a/Doc/c-api/sys.rst b/Doc/c-api/sys.rst
index 994509aa50f2a..e1715bccf55a5 100644
--- a/Doc/c-api/sys.rst
+++ b/Doc/c-api/sys.rst
@@ -22,7 +22,7 @@ Operating System Utilities
    Return true (nonzero) if the standard I/O file *fp* with name *filename* is
    deemed interactive.  This is the case for files for which ``isatty(fileno(fp))``
    is true.  If the global flag :c:data:`Py_InteractiveFlag` is true, this function
-   also returns true if the *filename* pointer is *NULL* or if the name is equal to
+   also returns true if the *filename* pointer is ``NULL`` or if the name is equal to
    one of the strings ``'<stdin>'`` or ``'???'``.
 
 
@@ -194,12 +194,12 @@ accessible to C code.  They all work with the current interpreter thread's
 
 .. c:function:: PyObject *PySys_GetObject(const char *name)
 
-   Return the object *name* from the :mod:`sys` module or *NULL* if it does
+   Return the object *name* from the :mod:`sys` module or ``NULL`` if it does
    not exist, without setting an exception.
 
 .. c:function:: int PySys_SetObject(const char *name, PyObject *v)
 
-   Set *name* in the :mod:`sys` module to *v* unless *v* is *NULL*, in which
+   Set *name* in the :mod:`sys` module to *v* unless *v* is ``NULL``, in which
    case *name* is deleted from the sys module. Returns ``0`` on success, ``-1``
    on error.
 
@@ -276,7 +276,7 @@ accessible to C code.  They all work with the current interpreter thread's
 .. c:function:: PyObject *PySys_GetXOptions()
 
    Return the current dictionary of :option:`-X` options, similarly to
-   :data:`sys._xoptions`.  On error, *NULL* is returned and an exception is
+   :data:`sys._xoptions`.  On error, ``NULL`` is returned and an exception is
    set.
 
    .. versionadded:: 3.2
diff --git a/Doc/c-api/tuple.rst b/Doc/c-api/tuple.rst
index 512d00db0066f..4dc79d0e7d053 100644
--- a/Doc/c-api/tuple.rst
+++ b/Doc/c-api/tuple.rst
@@ -33,12 +33,12 @@ Tuple Objects
 
 .. c:function:: PyObject* PyTuple_New(Py_ssize_t len)
 
-   Return a new tuple object of size *len*, or *NULL* on failure.
+   Return a new tuple object of size *len*, or ``NULL`` on failure.
 
 
 .. c:function:: PyObject* PyTuple_Pack(Py_ssize_t n, ...)
 
-   Return a new tuple object of size *n*, or *NULL* on failure. The tuple values
+   Return a new tuple object of size *n*, or ``NULL`` on failure. The tuple values
    are initialized to the subsequent *n* C arguments pointing to Python objects.
    ``PyTuple_Pack(2, a, b)`` is equivalent to ``Py_BuildValue("(OO)", a, b)``.
 
@@ -50,14 +50,14 @@ Tuple Objects
 
 .. c:function:: Py_ssize_t PyTuple_GET_SIZE(PyObject *p)
 
-   Return the size of the tuple *p*, which must be non-*NULL* and point to a tuple;
+   Return the size of the tuple *p*, which must be non-``NULL`` and point to a tuple;
    no error checking is performed.
 
 
 .. c:function:: PyObject* PyTuple_GetItem(PyObject *p, Py_ssize_t pos)
 
    Return the object at position *pos* in the tuple pointed to by *p*.  If *pos* is
-   out of bounds, return *NULL* and set an :exc:`IndexError` exception.
+   out of bounds, return ``NULL`` and set an :exc:`IndexError` exception.
 
 
 .. c:function:: PyObject* PyTuple_GET_ITEM(PyObject *p, Py_ssize_t pos)
@@ -68,7 +68,7 @@ Tuple Objects
 .. c:function:: PyObject* PyTuple_GetSlice(PyObject *p, Py_ssize_t low, Py_ssize_t high)
 
    Return the slice of the tuple pointed to by *p* between *low* and *high*,
-   or *NULL* on failure.  This is the equivalent of the Python expression
+   or ``NULL`` on failure.  This is the equivalent of the Python expression
    ``p[low:high]``.  Indexing from the end of the list is not supported.
 
 
@@ -107,7 +107,7 @@ Tuple Objects
    only more efficiently.  Returns ``0`` on success. Client code should never
    assume that the resulting value of ``*p`` will be the same as before calling
    this function. If the object referenced by ``*p`` is replaced, the original
-   ``*p`` is destroyed.  On failure, returns ``-1`` and sets ``*p`` to *NULL*, and
+   ``*p`` is destroyed.  On failure, returns ``-1`` and sets ``*p`` to ``NULL``, and
    raises :exc:`MemoryError` or :exc:`SystemError`.
 
 
@@ -147,20 +147,20 @@ type.
 
    Contains the meta information of a struct sequence type to create.
 
-   +-------------------+------------------------------+------------------------------------+
-   | Field             | C Type                       | Meaning                            |
-   +===================+==============================+====================================+
-   | ``name``          | ``const char *``             | name of the struct sequence type   |
-   +-------------------+------------------------------+------------------------------------+
-   | ``doc``           | ``const char *``             | pointer to docstring for the type  |
-   |                   |                              | or NULL to omit                    |
-   +-------------------+------------------------------+------------------------------------+
-   | ``fields``        | ``PyStructSequence_Field *`` | pointer to *NULL*-terminated array |
-   |                   |                              | with field names of the new type   |
-   +-------------------+------------------------------+------------------------------------+
-   | ``n_in_sequence`` | ``int``                      | number of fields visible to the    |
-   |                   |                              | Python side (if used as tuple)     |
-   +-------------------+------------------------------+------------------------------------+
+   +-------------------+------------------------------+--------------------------------------+
+   | Field             | C Type                       | Meaning                              |
+   +===================+==============================+======================================+
+   | ``name``          | ``const char *``             | name of the struct sequence type     |
+   +-------------------+------------------------------+--------------------------------------+
+   | ``doc``           | ``const char *``             | pointer to docstring for the type    |
+   |                   |                              | or NULL to omit                      |
+   +-------------------+------------------------------+--------------------------------------+
+   | ``fields``        | ``PyStructSequence_Field *`` | pointer to ``NULL``-terminated array |
+   |                   |                              | with field names of the new type     |
+   +-------------------+------------------------------+--------------------------------------+
+   | ``n_in_sequence`` | ``int``                      | number of fields visible to the      |
+   |                   |                              | Python side (if used as tuple)       |
+   +-------------------+------------------------------+--------------------------------------+
 
 
 .. c:type:: PyStructSequence_Field
@@ -170,16 +170,16 @@ type.
    :attr:`fields` array of the :c:type:`PyStructSequence_Desc` determines which
    field of the struct sequence is described.
 
-   +-----------+------------------+--------------------------------------+
-   | Field     | C Type           | Meaning                              |
-   +===========+==================+======================================+
-   | ``name``  | ``const char *`` | name for the field or *NULL* to end  |
-   |           |                  | the list of named fields, set to     |
-   |           |                  | PyStructSequence_UnnamedField to     |
-   |           |                  | leave unnamed                        |
-   +-----------+------------------+--------------------------------------+
-   | ``doc``   | ``const char *`` | field docstring or *NULL* to omit    |
-   +-----------+------------------+--------------------------------------+
+   +-----------+------------------+----------------------------------------+
+   | Field     | C Type           | Meaning                                |
+   +===========+==================+========================================+
+   | ``name``  | ``const char *`` | name for the field or ``NULL`` to end  |
+   |           |                  | the list of named fields, set to       |
+   |           |                  | PyStructSequence_UnnamedField to       |
+   |           |                  | leave unnamed                          |
+   +-----------+------------------+----------------------------------------+
+   | ``doc``   | ``const char *`` | field docstring or ``NULL`` to omit    |
+   +-----------+------------------+----------------------------------------+
 
 
 .. c:var:: char* PyStructSequence_UnnamedField
diff --git a/Doc/c-api/type.rst b/Doc/c-api/type.rst
index 4dfd53fb9f076..5a3d749a0a258 100644
--- a/Doc/c-api/type.rst
+++ b/Doc/c-api/type.rst
@@ -81,7 +81,7 @@ Type Objects
 
    Generic handler for the :c:member:`~PyTypeObject.tp_alloc` slot of a type object.  Use
    Python's default memory allocation mechanism to allocate a new instance and
-   initialize all its contents to *NULL*.
+   initialize all its contents to ``NULL``.
 
 .. c:function:: PyObject* PyType_GenericNew(PyTypeObject *type, PyObject *args, PyObject *kwds)
 
@@ -110,7 +110,7 @@ Type Objects
 .. c:function:: void* PyType_GetSlot(PyTypeObject *type, int slot)
 
    Return the function pointer stored in the given slot. If the
-   result is *NULL*, this indicates that either the slot is *NULL*,
+   result is ``NULL``, this indicates that either the slot is ``NULL``,
    or that the function was called with invalid parameters.
    Callers will typically cast the result pointer into the appropriate
    function type.
diff --git a/Doc/c-api/typeobj.rst b/Doc/c-api/typeobj.rst
index 532508ee72281..71b45a6e7158f 100644
--- a/Doc/c-api/typeobj.rst
+++ b/Doc/c-api/typeobj.rst
@@ -43,8 +43,8 @@ type objects) *must* have the :attr:`ob_size` field.
              PyObject* PyObject._ob_prev
 
    These fields are only present when the macro ``Py_TRACE_REFS`` is defined.
-   Their initialization to *NULL* is taken care of by the ``PyObject_HEAD_INIT``
-   macro.  For statically allocated objects, these fields always remain *NULL*.
+   Their initialization to ``NULL`` is taken care of by the ``PyObject_HEAD_INIT``
+   macro.  For statically allocated objects, these fields always remain ``NULL``.
    For dynamically allocated objects, these two fields are used to link the object
    into a doubly-linked list of *all* live objects on the heap.  This could be used
    for various debugging purposes; currently the only use is to print the objects
@@ -71,7 +71,7 @@ type objects) *must* have the :attr:`ob_size` field.
    argument to the ``PyObject_HEAD_INIT`` macro, and its value should normally be
    ``&PyType_Type``.  However, for dynamically loadable extension modules that must
    be usable on Windows (at least), the compiler complains that this is not a valid
-   initializer.  Therefore, the convention is to pass *NULL* to the
+   initializer.  Therefore, the convention is to pass ``NULL`` to the
    ``PyObject_HEAD_INIT`` macro and to initialize this field explicitly at the
    start of the module's initialization function, before doing anything else.  This
    is typically done like this::
@@ -79,7 +79,7 @@ type objects) *must* have the :attr:`ob_size` field.
       Foo_Type.ob_type = &PyType_Type;
 
    This should be done before any instances of the type are created.
-   :c:func:`PyType_Ready` checks if :attr:`ob_type` is *NULL*, and if so,
+   :c:func:`PyType_Ready` checks if :attr:`ob_type` is ``NULL``, and if so,
    initializes it to the :attr:`ob_type` field of the base class.
    :c:func:`PyType_Ready` will not change this field if it is non-zero.
 
@@ -205,7 +205,7 @@ type objects) *must* have the :attr:`ob_size` field.
 
    This field is inherited by subtypes together with :c:member:`~PyTypeObject.tp_getattro`: a subtype
    inherits both :c:member:`~PyTypeObject.tp_getattr` and :c:member:`~PyTypeObject.tp_getattro` from its base type when
-   the subtype's :c:member:`~PyTypeObject.tp_getattr` and :c:member:`~PyTypeObject.tp_getattro` are both *NULL*.
+   the subtype's :c:member:`~PyTypeObject.tp_getattr` and :c:member:`~PyTypeObject.tp_getattro` are both ``NULL``.
 
 
 .. c:member:: setattrfunc PyTypeObject.tp_setattr
@@ -218,10 +218,10 @@ type objects) *must* have the :attr:`ob_size` field.
 
       PyObject * tp_setattr(PyObject *o, char *attr_name, PyObject *v);
 
-   The *v* argument is set to *NULL* to delete the attribute.
+   The *v* argument is set to ``NULL`` to delete the attribute.
    This field is inherited by subtypes together with :c:member:`~PyTypeObject.tp_setattro`: a subtype
    inherits both :c:member:`~PyTypeObject.tp_setattr` and :c:member:`~PyTypeObject.tp_setattro` from its base type when
-   the subtype's :c:member:`~PyTypeObject.tp_setattr` and :c:member:`~PyTypeObject.tp_setattro` are both *NULL*.
+   the subtype's :c:member:`~PyTypeObject.tp_setattr` and :c:member:`~PyTypeObject.tp_setattro` are both ``NULL``.
 
 
 .. c:member:: PyAsyncMethods* tp_as_async
@@ -310,13 +310,13 @@ type objects) *must* have the :attr:`ob_size` field.
    This field is inherited by subtypes together with
    :c:member:`~PyTypeObject.tp_richcompare`: a subtype inherits both of
    :c:member:`~PyTypeObject.tp_richcompare` and :c:member:`~PyTypeObject.tp_hash`, when the subtype's
-   :c:member:`~PyTypeObject.tp_richcompare` and :c:member:`~PyTypeObject.tp_hash` are both *NULL*.
+   :c:member:`~PyTypeObject.tp_richcompare` and :c:member:`~PyTypeObject.tp_hash` are both ``NULL``.
 
 
 .. c:member:: ternaryfunc PyTypeObject.tp_call
 
    An optional pointer to a function that implements calling the object.  This
-   should be *NULL* if the object is not callable.  The signature is the same as
+   should be ``NULL`` if the object is not callable.  The signature is the same as
    for :c:func:`PyObject_Call`.
 
    This field is inherited by subtypes.
@@ -350,7 +350,7 @@ type objects) *must* have the :attr:`ob_size` field.
 
    This field is inherited by subtypes together with :c:member:`~PyTypeObject.tp_getattr`: a subtype
    inherits both :c:member:`~PyTypeObject.tp_getattr` and :c:member:`~PyTypeObject.tp_getattro` from its base type when
-   the subtype's :c:member:`~PyTypeObject.tp_getattr` and :c:member:`~PyTypeObject.tp_getattro` are both *NULL*.
+   the subtype's :c:member:`~PyTypeObject.tp_getattr` and :c:member:`~PyTypeObject.tp_getattro` are both ``NULL``.
 
 
 .. c:member:: setattrofunc PyTypeObject.tp_setattro
@@ -358,13 +358,13 @@ type objects) *must* have the :attr:`ob_size` field.
    An optional pointer to the function for setting and deleting attributes.
 
    The signature is the same as for :c:func:`PyObject_SetAttr`, but setting
-   *v* to *NULL* to delete an attribute must be supported.  It is usually
+   *v* to ``NULL`` to delete an attribute must be supported.  It is usually
    convenient to set this field to :c:func:`PyObject_GenericSetAttr`, which
    implements the normal way of setting object attributes.
 
    This field is inherited by subtypes together with :c:member:`~PyTypeObject.tp_setattr`: a subtype
    inherits both :c:member:`~PyTypeObject.tp_setattr` and :c:member:`~PyTypeObject.tp_setattro` from its base type when
-   the subtype's :c:member:`~PyTypeObject.tp_setattr` and :c:member:`~PyTypeObject.tp_setattro` are both *NULL*.
+   the subtype's :c:member:`~PyTypeObject.tp_setattr` and :c:member:`~PyTypeObject.tp_setattro` are both ``NULL``.
 
 
 .. c:member:: PyBufferProcs* PyTypeObject.tp_as_buffer
@@ -385,7 +385,7 @@ type objects) *must* have the :attr:`ob_size` field.
    :c:member:`~PyTypeObject.tp_as_number`, :c:member:`~PyTypeObject.tp_as_sequence`, :c:member:`~PyTypeObject.tp_as_mapping`, and
    :c:member:`~PyTypeObject.tp_as_buffer`) that were historically not always present are valid; if
    such a flag bit is clear, the type fields it guards must not be accessed and
-   must be considered to have a zero or *NULL* value instead.
+   must be considered to have a zero or ``NULL`` value instead.
 
    Inheritance of this field is complicated.  Most flag bits are inherited
    individually, i.e. if the base type has a flag bit set, the subtype inherits
@@ -396,7 +396,7 @@ type objects) *must* have the :attr:`ob_size` field.
    the :c:member:`~PyTypeObject.tp_traverse` and :c:member:`~PyTypeObject.tp_clear` fields, i.e. if the
    :const:`Py_TPFLAGS_HAVE_GC` flag bit is clear in the subtype and the
    :c:member:`~PyTypeObject.tp_traverse` and :c:member:`~PyTypeObject.tp_clear` fields in the subtype exist and have
-   *NULL* values.
+   ``NULL`` values.
 
    The following bit masks are currently defined; these can be ORed together using
    the ``|`` operator to form the value of the :c:member:`~PyTypeObject.tp_flags` field.  The macro
@@ -510,7 +510,7 @@ type objects) *must* have the :attr:`ob_size` field.
 
    Note that :c:func:`Py_VISIT` is called only on those members that can participate
    in reference cycles.  Although there is also a ``self->key`` member, it can only
-   be *NULL* or a Python string and therefore cannot be part of a reference cycle.
+   be ``NULL`` or a Python string and therefore cannot be part of a reference cycle.
 
    On the other hand, even if you know a member can never be part of a cycle, as a
    debugging aid you may want to visit it anyway just so the :mod:`gc` module's
@@ -543,7 +543,7 @@ type objects) *must* have the :attr:`ob_size` field.
 
    Implementations of :c:member:`~PyTypeObject.tp_clear` should drop the instance's references to
    those of its members that may be Python objects, and set its pointers to those
-   members to *NULL*, as in the following example::
+   members to ``NULL``, as in the following example::
 
       static int
       local_clear(localobject *self)
@@ -557,12 +557,12 @@ type objects) *must* have the :attr:`ob_size` field.
 
    The :c:func:`Py_CLEAR` macro should be used, because clearing references is
    delicate:  the reference to the contained object must not be decremented until
-   after the pointer to the contained object is set to *NULL*.  This is because
+   after the pointer to the contained object is set to ``NULL``.  This is because
    decrementing the reference count may cause the contained object to become trash,
    triggering a chain of reclamation activity that may include invoking arbitrary
    Python code (due to finalizers, or weakref callbacks, associated with the
    contained object). If it's possible for such code to reference *self* again,
-   it's important that the pointer to the contained object be *NULL* at that time,
+   it's important that the pointer to the contained object be ``NULL`` at that time,
    so that *self* knows the contained object can no longer be used.  The
    :c:func:`Py_CLEAR` macro performs the operations in a safe order.
 
@@ -602,7 +602,7 @@ type objects) *must* have the :attr:`ob_size` field.
    This field is inherited by subtypes together with :c:member:`~PyTypeObject.tp_hash`:
    a subtype inherits :c:member:`~PyTypeObject.tp_richcompare` and :c:member:`~PyTypeObject.tp_hash` when
    the subtype's :c:member:`~PyTypeObject.tp_richcompare` and :c:member:`~PyTypeObject.tp_hash` are both
-   *NULL*.
+   ``NULL``.
 
    The following constants are defined to be used as the third argument for
    :c:member:`~PyTypeObject.tp_richcompare` and for :c:func:`PyObject_RichCompare`:
@@ -647,7 +647,7 @@ type objects) *must* have the :attr:`ob_size` field.
    reference list head (ignoring the GC header, if present); this offset is used by
    :c:func:`PyObject_ClearWeakRefs` and the :c:func:`PyWeakref_\*` functions.  The
    instance structure needs to include a field of type :c:type:`PyObject\*` which is
-   initialized to *NULL*.
+   initialized to ``NULL``.
 
    Do not confuse this field with :c:member:`~PyTypeObject.tp_weaklist`; that is the list head for
    weak references to the type object itself.
@@ -685,9 +685,9 @@ type objects) *must* have the :attr:`ob_size` field.
 .. c:member:: iternextfunc PyTypeObject.tp_iternext
 
    An optional pointer to a function that returns the next item in an iterator.
-   When the iterator is exhausted, it must return *NULL*; a :exc:`StopIteration`
+   When the iterator is exhausted, it must return ``NULL``; a :exc:`StopIteration`
    exception may or may not be set.  When another error occurs, it must return
-   *NULL* too.  Its presence signals that the instances of this type are
+   ``NULL`` too.  Its presence signals that the instances of this type are
    iterators.
 
    Iterator types should also define the :c:member:`~PyTypeObject.tp_iter` function, and that
@@ -701,7 +701,7 @@ type objects) *must* have the :attr:`ob_size` field.
 
 .. c:member:: struct PyMethodDef* PyTypeObject.tp_methods
 
-   An optional pointer to a static *NULL*-terminated array of :c:type:`PyMethodDef`
+   An optional pointer to a static ``NULL``-terminated array of :c:type:`PyMethodDef`
    structures, declaring regular methods of this type.
 
    For each entry in the array, an entry is added to the type's dictionary (see
@@ -713,7 +713,7 @@ type objects) *must* have the :attr:`ob_size` field.
 
 .. c:member:: struct PyMemberDef* PyTypeObject.tp_members
 
-   An optional pointer to a static *NULL*-terminated array of :c:type:`PyMemberDef`
+   An optional pointer to a static ``NULL``-terminated array of :c:type:`PyMemberDef`
    structures, declaring regular data members (fields or slots) of instances of
    this type.
 
@@ -726,7 +726,7 @@ type objects) *must* have the :attr:`ob_size` field.
 
 .. c:member:: struct PyGetSetDef* PyTypeObject.tp_getset
 
-   An optional pointer to a static *NULL*-terminated array of :c:type:`PyGetSetDef`
+   An optional pointer to a static ``NULL``-terminated array of :c:type:`PyGetSetDef`
    structures, declaring computed attributes of instances of this type.
 
    For each entry in the array, an entry is added to the type's dictionary (see
@@ -751,7 +751,7 @@ type objects) *must* have the :attr:`ob_size` field.
 
    The type's dictionary is stored here by :c:func:`PyType_Ready`.
 
-   This field should normally be initialized to *NULL* before PyType_Ready is
+   This field should normally be initialized to ``NULL`` before PyType_Ready is
    called; it may also be initialized to a dictionary containing initial attributes
    for the type.  Once :c:func:`PyType_Ready` has initialized the type, extra
    attributes for the type may be added to this dictionary only if they don't
@@ -788,7 +788,7 @@ type objects) *must* have the :attr:`ob_size` field.
 
       int tp_descr_set(PyObject *self, PyObject *obj, PyObject *value);
 
-   The *value* argument is set to *NULL* to delete the value.
+   The *value* argument is set to ``NULL`` to delete the value.
    This field is inherited by subtypes.
 
    .. XXX explain.
@@ -864,7 +864,7 @@ type objects) *must* have the :attr:`ob_size` field.
    arguments represent positional and keyword arguments of the call to
    :meth:`__init__`.
 
-   The :c:member:`~PyTypeObject.tp_init` function, if not *NULL*, is called when an instance is
+   The :c:member:`~PyTypeObject.tp_init` function, if not ``NULL``, is called when an instance is
    created normally by calling its type, after the type's :c:member:`~PyTypeObject.tp_new` function
    has returned an instance of the type.  If the :c:member:`~PyTypeObject.tp_new` function returns an
    instance of some other type that is not a subtype of the original type, no
@@ -905,7 +905,7 @@ type objects) *must* have the :attr:`ob_size` field.
 
    An optional pointer to an instance creation function.
 
-   If this function is *NULL* for a particular type, that type cannot be called to
+   If this function is ``NULL`` for a particular type, that type cannot be called to
    create new instances; presumably there is some other way to create instances,
    like a factory function.
 
@@ -928,7 +928,7 @@ type objects) *must* have the :attr:`ob_size` field.
    deferred to :c:member:`~PyTypeObject.tp_init`.
 
    This field is inherited by subtypes, except it is not inherited by static types
-   whose :c:member:`~PyTypeObject.tp_base` is *NULL* or ``&PyBaseObject_Type``.
+   whose :c:member:`~PyTypeObject.tp_base` is ``NULL`` or ``&PyBaseObject_Type``.
 
 
 .. c:member:: destructor PyTypeObject.tp_free
@@ -971,7 +971,7 @@ type objects) *must* have the :attr:`ob_size` field.
 
    Tuple of base types.
 
-   This is set for types created by a class statement.  It should be *NULL* for
+   This is set for types created by a class statement.  It should be ``NULL`` for
    statically defined types.
 
    This field is not inherited.
@@ -1169,14 +1169,14 @@ Mapping Object Structures
 
    This function is used by :c:func:`PyMapping_Size` and
    :c:func:`PyObject_Size`, and has the same signature.  This slot may be set to
-   *NULL* if the object has no defined length.
+   ``NULL`` if the object has no defined length.
 
 .. c:member:: binaryfunc PyMappingMethods.mp_subscript
 
    This function is used by :c:func:`PyObject_GetItem` and
    :c:func:`PySequence_GetSlice`, and has the same signature as
    :c:func:`!PyObject_GetItem`.  This slot must be filled for the
-   :c:func:`PyMapping_Check` function to return ``1``, it can be *NULL*
+   :c:func:`PyMapping_Check` function to return ``1``, it can be ``NULL``
    otherwise.
 
 .. c:member:: objobjargproc PyMappingMethods.mp_ass_subscript
@@ -1184,8 +1184,8 @@ Mapping Object Structures
    This function is used by :c:func:`PyObject_SetItem`,
    :c:func:`PyObject_DelItem`, :c:func:`PyObject_SetSlice` and
    :c:func:`PyObject_DelSlice`.  It has the same signature as
-   :c:func:`!PyObject_SetItem`, but *v* can also be set to *NULL* to delete
-   an item.  If this slot is *NULL*, the object does not support item
+   :c:func:`!PyObject_SetItem`, but *v* can also be set to ``NULL`` to delete
+   an item.  If this slot is ``NULL``, the object does not support item
    assignment and deletion.
 
 
@@ -1227,11 +1227,11 @@ Sequence Object Structures
    signature.  It is also used by :c:func:`PyObject_GetItem`, after trying
    the subscription via the :c:member:`~PyMappingMethods.mp_subscript` slot.
    This slot must be filled for the :c:func:`PySequence_Check`
-   function to return ``1``, it can be *NULL* otherwise.
+   function to return ``1``, it can be ``NULL`` otherwise.
 
    Negative indexes are handled as follows: if the :attr:`sq_length` slot is
    filled, it is called and the sequence length is used to compute a positive
-   index which is passed to :attr:`sq_item`.  If :attr:`sq_length` is *NULL*,
+   index which is passed to :attr:`sq_item`.  If :attr:`sq_length` is ``NULL``,
    the index is passed as is to the function.
 
 .. c:member:: ssizeobjargproc PySequenceMethods.sq_ass_item
@@ -1240,13 +1240,13 @@ Sequence Object Structures
    signature.  It is also used by :c:func:`PyObject_SetItem` and
    :c:func:`PyObject_DelItem`, after trying the item assignment and deletion
    via the :c:member:`~PyMappingMethods.mp_ass_subscript` slot.
-   This slot may be left to *NULL* if the object does not support
+   This slot may be left to ``NULL`` if the object does not support
    item assignment and deletion.
 
 .. c:member:: objobjproc PySequenceMethods.sq_contains
 
    This function may be used by :c:func:`PySequence_Contains` and has the same
-   signature.  This slot may be left to *NULL*, in this case
+   signature.  This slot may be left to ``NULL``, in this case
    :c:func:`!PySequence_Contains` simply traverses the sequence until it
    finds a match.
 
@@ -1254,7 +1254,7 @@ Sequence Object Structures
 
    This function is used by :c:func:`PySequence_InPlaceConcat` and has the same
    signature.  It should modify its first operand, and return it.  This slot
-   may be left to *NULL*, in this case :c:func:`!PySequence_InPlaceConcat`
+   may be left to ``NULL``, in this case :c:func:`!PySequence_InPlaceConcat`
    will fall back to :c:func:`PySequence_Concat`.  It is also used by the
    augmented assignment ``+=``, after trying numeric in-place addition
    via the :c:member:`~PyNumberMethods.nb_inplace_add` slot.
@@ -1263,7 +1263,7 @@ Sequence Object Structures
 
    This function is used by :c:func:`PySequence_InPlaceRepeat` and has the same
    signature.  It should modify its first operand, and return it.  This slot
-   may be left to *NULL*, in this case :c:func:`!PySequence_InPlaceRepeat`
+   may be left to ``NULL``, in this case :c:func:`!PySequence_InPlaceRepeat`
    will fall back to :c:func:`PySequence_Repeat`.  It is also used by the
    augmented assignment ``*=``, after trying numeric in-place multiplication
    via the :c:member:`~PyNumberMethods.nb_inplace_multiply` slot.
@@ -1295,7 +1295,7 @@ Buffer Object Structures
    steps:
 
    (1) Check if the request can be met. If not, raise :c:data:`PyExc_BufferError`,
-       set :c:data:`view->obj` to *NULL* and return ``-1``.
+       set :c:data:`view->obj` to ``NULL`` and return ``-1``.
 
    (2) Fill in the requested fields.
 
@@ -1341,7 +1341,7 @@ Buffer Object Structures
 
    Handle a request to release the resources of the buffer. If no resources
    need to be released, :c:member:`PyBufferProcs.bf_releasebuffer` may be
-   *NULL*. Otherwise, a standard implementation of this function will take
+   ``NULL``. Otherwise, a standard implementation of this function will take
    these optional steps:
 
    (1) Decrement an internal counter for the number of exports.
@@ -1395,7 +1395,7 @@ Async Object Structures
    The returned object must be an iterator, i.e. :c:func:`PyIter_Check` must
    return ``1`` for it.
 
-   This slot may be set to *NULL* if an object is not an :term:`awaitable`.
+   This slot may be set to ``NULL`` if an object is not an :term:`awaitable`.
 
 .. c:member:: unaryfunc PyAsyncMethods.am_aiter
 
@@ -1405,7 +1405,7 @@ Async Object Structures
 
    Must return an :term:`awaitable` object.  See :meth:`__anext__` for details.
 
-   This slot may be set to *NULL* if an object does not implement
+   This slot may be set to ``NULL`` if an object does not implement
    asynchronous iteration protocol.
 
 .. c:member:: unaryfunc PyAsyncMethods.am_anext
@@ -1415,4 +1415,4 @@ Async Object Structures
       PyObject *am_anext(PyObject *self)
 
    Must return an :term:`awaitable` object.  See :meth:`__anext__` for details.
-   This slot may be set to *NULL*.
+   This slot may be set to ``NULL``.
diff --git a/Doc/c-api/unicode.rst b/Doc/c-api/unicode.rst
index 1d724a325745e..f6c50fb468d4d 100644
--- a/Doc/c-api/unicode.rst
+++ b/Doc/c-api/unicode.rst
@@ -236,7 +236,7 @@ access internal read-only data of Unicode objects:
    .. versionchanged:: 3.3
       This macro is now inefficient -- because in many cases the
       :c:type:`Py_UNICODE` representation does not exist and needs to be created
-      -- and can fail (return *NULL* with an exception set).  Try to port the
+      -- and can fail (return ``NULL`` with an exception set).  Try to port the
       code to use the new :c:func:`PyUnicode_nBYTE_DATA` macros or use
       :c:func:`PyUnicode_WRITE` or :c:func:`PyUnicode_READ`.
 
@@ -413,11 +413,11 @@ APIs:
 
    Create a Unicode object from the char buffer *u*.  The bytes will be
    interpreted as being UTF-8 encoded.  The buffer is copied into the new
-   object. If the buffer is not *NULL*, the return value might be a shared
+   object. If the buffer is not ``NULL``, the return value might be a shared
    object, i.e. modification of the data is not allowed.
 
-   If *u* is *NULL*, this function behaves like :c:func:`PyUnicode_FromUnicode`
-   with the buffer set to *NULL*.  This usage is deprecated in favor of
+   If *u* is ``NULL``, this function behaves like :c:func:`PyUnicode_FromUnicode`
+   with the buffer set to ``NULL``.  This usage is deprecated in favor of
    :c:func:`PyUnicode_New`.
 
 
@@ -443,82 +443,82 @@ APIs:
 
    .. tabularcolumns:: |l|l|L|
 
-   +-------------------+---------------------+--------------------------------+
-   | Format Characters | Type                | Comment                        |
-   +===================+=====================+================================+
-   | :attr:`%%`        | *n/a*               | The literal % character.       |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%c`        | int                 | A single character,            |
-   |                   |                     | represented as a C int.        |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%d`        | int                 | Equivalent to                  |
-   |                   |                     | ``printf("%d")``. [1]_         |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%u`        | unsigned int        | Equivalent to                  |
-   |                   |                     | ``printf("%u")``. [1]_         |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%ld`       | long                | Equivalent to                  |
-   |                   |                     | ``printf("%ld")``. [1]_        |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%li`       | long                | Equivalent to                  |
-   |                   |                     | ``printf("%li")``. [1]_        |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%lu`       | unsigned long       | Equivalent to                  |
-   |                   |                     | ``printf("%lu")``. [1]_        |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%lld`      | long long           | Equivalent to                  |
-   |                   |                     | ``printf("%lld")``. [1]_       |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%lli`      | long long           | Equivalent to                  |
-   |                   |                     | ``printf("%lli")``. [1]_       |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%llu`      | unsigned long long  | Equivalent to                  |
-   |                   |                     | ``printf("%llu")``. [1]_       |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%zd`       | Py_ssize_t          | Equivalent to                  |
-   |                   |                     | ``printf("%zd")``. [1]_        |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%zi`       | Py_ssize_t          | Equivalent to                  |
-   |                   |                     | ``printf("%zi")``. [1]_        |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%zu`       | size_t              | Equivalent to                  |
-   |                   |                     | ``printf("%zu")``. [1]_        |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%i`        | int                 | Equivalent to                  |
-   |                   |                     | ``printf("%i")``. [1]_         |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%x`        | int                 | Equivalent to                  |
-   |                   |                     | ``printf("%x")``. [1]_         |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%s`        | const char\*        | A null-terminated C character  |
-   |                   |                     | array.                         |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%p`        | const void\*        | The hex representation of a C  |
-   |                   |                     | pointer. Mostly equivalent to  |
-   |                   |                     | ``printf("%p")`` except that   |
-   |                   |                     | it is guaranteed to start with |
-   |                   |                     | the literal ``0x`` regardless  |
-   |                   |                     | of what the platform's         |
-   |                   |                     | ``printf`` yields.             |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%A`        | PyObject\*          | The result of calling          |
-   |                   |                     | :func:`ascii`.                 |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%U`        | PyObject\*          | A Unicode object.              |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%V`        | PyObject\*,         | A Unicode object (which may be |
-   |                   | const char\*        | *NULL*) and a null-terminated  |
-   |                   |                     | C character array as a second  |
-   |                   |                     | parameter (which will be used, |
-   |                   |                     | if the first parameter is      |
-   |                   |                     | *NULL*).                       |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%S`        | PyObject\*          | The result of calling          |
-   |                   |                     | :c:func:`PyObject_Str`.        |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%R`        | PyObject\*          | The result of calling          |
-   |                   |                     | :c:func:`PyObject_Repr`.       |
-   +-------------------+---------------------+--------------------------------+
+   +-------------------+---------------------+----------------------------------+
+   | Format Characters | Type                | Comment                          |
+   +===================+=====================+==================================+
+   | :attr:`%%`        | *n/a*               | The literal % character.         |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%c`        | int                 | A single character,              |
+   |                   |                     | represented as a C int.          |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%d`        | int                 | Equivalent to                    |
+   |                   |                     | ``printf("%d")``. [1]_           |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%u`        | unsigned int        | Equivalent to                    |
+   |                   |                     | ``printf("%u")``. [1]_           |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%ld`       | long                | Equivalent to                    |
+   |                   |                     | ``printf("%ld")``. [1]_          |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%li`       | long                | Equivalent to                    |
+   |                   |                     | ``printf("%li")``. [1]_          |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%lu`       | unsigned long       | Equivalent to                    |
+   |                   |                     | ``printf("%lu")``. [1]_          |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%lld`      | long long           | Equivalent to                    |
+   |                   |                     | ``printf("%lld")``. [1]_         |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%lli`      | long long           | Equivalent to                    |
+   |                   |                     | ``printf("%lli")``. [1]_         |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%llu`      | unsigned long long  | Equivalent to                    |
+   |                   |                     | ``printf("%llu")``. [1]_         |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%zd`       | Py_ssize_t          | Equivalent to                    |
+   |                   |                     | ``printf("%zd")``. [1]_          |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%zi`       | Py_ssize_t          | Equivalent to                    |
+   |                   |                     | ``printf("%zi")``. [1]_          |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%zu`       | size_t              | Equivalent to                    |
+   |                   |                     | ``printf("%zu")``. [1]_          |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%i`        | int                 | Equivalent to                    |
+   |                   |                     | ``printf("%i")``. [1]_           |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%x`        | int                 | Equivalent to                    |
+   |                   |                     | ``printf("%x")``. [1]_           |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%s`        | const char\*        | A null-terminated C character    |
+   |                   |                     | array.                           |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%p`        | const void\*        | The hex representation of a C    |
+   |                   |                     | pointer. Mostly equivalent to    |
+   |                   |                     | ``printf("%p")`` except that     |
+   |                   |                     | it is guaranteed to start with   |
+   |                   |                     | the literal ``0x`` regardless    |
+   |                   |                     | of what the platform's           |
+   |                   |                     | ``printf`` yields.               |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%A`        | PyObject\*          | The result of calling            |
+   |                   |                     | :func:`ascii`.                   |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%U`        | PyObject\*          | A Unicode object.                |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%V`        | PyObject\*,         | A Unicode object (which may be   |
+   |                   | const char\*        | ``NULL``) and a null-terminated  |
+   |                   |                     | C character array as a second    |
+   |                   |                     | parameter (which will be used,   |
+   |                   |                     | if the first parameter is        |
+   |                   |                     | ``NULL``).                       |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%S`        | PyObject\*          | The result of calling            |
+   |                   |                     | :c:func:`PyObject_Str`.          |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%R`        | PyObject\*          | The result of calling            |
+   |                   |                     | :c:func:`PyObject_Repr`.         |
+   +-------------------+---------------------+----------------------------------+
 
    An unrecognized format character causes all the rest of the format string to be
    copied as-is to the result string, and any extra arguments discarded.
@@ -558,13 +558,13 @@ APIs:
    :class:`bytes`, :class:`bytearray` and other
    :term:`bytes-like objects <bytes-like object>`
    are decoded according to the given *encoding* and using the error handling
-   defined by *errors*. Both can be *NULL* to have the interface use the default
+   defined by *errors*. Both can be ``NULL`` to have the interface use the default
    values (see :ref:`builtincodecs` for details).
 
    All other objects, including Unicode objects, cause a :exc:`TypeError` to be
    set.
 
-   The API returns *NULL* if there was an error.  The caller is responsible for
+   The API returns ``NULL`` if there was an error.  The caller is responsible for
    decref'ing the returned objects.
 
 
@@ -640,7 +640,7 @@ APIs:
                                           Py_ssize_t buflen, int copy_null)
 
    Copy the string *u* into a UCS4 buffer, including a null character, if
-   *copy_null* is set.  Returns *NULL* and sets an exception on error (in
+   *copy_null* is set.  Returns ``NULL`` and sets an exception on error (in
    particular, a :exc:`SystemError` if *buflen* is smaller than the length of
    *u*).  *buffer* is returned on success.
 
@@ -650,7 +650,7 @@ APIs:
 .. c:function:: Py_UCS4* PyUnicode_AsUCS4Copy(PyObject *u)
 
    Copy the string *u* into a new UCS4 buffer that is allocated using
-   :c:func:`PyMem_Malloc`.  If this fails, *NULL* is returned with a
+   :c:func:`PyMem_Malloc`.  If this fails, ``NULL`` is returned with a
    :exc:`MemoryError` set.  The returned buffer always has an extra
    null code point appended.
 
@@ -670,15 +670,15 @@ Extension modules can continue using them, as they will not be removed in Python
 .. c:function:: PyObject* PyUnicode_FromUnicode(const Py_UNICODE *u, Py_ssize_t size)
 
    Create a Unicode object from the Py_UNICODE buffer *u* of the given size. *u*
-   may be *NULL* which causes the contents to be undefined. It is the user's
+   may be ``NULL`` which causes the contents to be undefined. It is the user's
    responsibility to fill in the needed data.  The buffer is copied into the new
    object.
 
-   If the buffer is not *NULL*, the return value might be a shared object.
+   If the buffer is not ``NULL``, the return value might be a shared object.
    Therefore, modification of the resulting Unicode object is only allowed when
-   *u* is *NULL*.
+   *u* is ``NULL``.
 
-   If the buffer is *NULL*, :c:func:`PyUnicode_READY` must be called once the
+   If the buffer is ``NULL``, :c:func:`PyUnicode_READY` must be called once the
    string content has been filled before using any of the access macros such as
    :c:func:`PyUnicode_KIND`.
 
@@ -689,7 +689,7 @@ Extension modules can continue using them, as they will not be removed in Python
 .. c:function:: Py_UNICODE* PyUnicode_AsUnicode(PyObject *unicode)
 
    Return a read-only pointer to the Unicode object's internal
-   :c:type:`Py_UNICODE` buffer, or *NULL* on error. This will create the
+   :c:type:`Py_UNICODE` buffer, or ``NULL`` on error. This will create the
    :c:type:`Py_UNICODE*` representation of the object if it is not yet
    available. The buffer is always terminated with an extra null code point.
    Note that the resulting :c:type:`Py_UNICODE` string may also contain
@@ -705,7 +705,7 @@ Extension modules can continue using them, as they will not be removed in Python
 
    Create a Unicode object by replacing all decimal digits in
    :c:type:`Py_UNICODE` buffer of the given *size* by ASCII digits 0--9
-   according to their decimal value.  Return *NULL* if an exception occurs.
+   according to their decimal value.  Return ``NULL`` if an exception occurs.
 
 
 .. c:function:: Py_UNICODE* PyUnicode_AsUnicodeAndSize(PyObject *unicode, Py_ssize_t *size)
@@ -721,7 +721,7 @@ Extension modules can continue using them, as they will not be removed in Python
 
 .. c:function:: Py_UNICODE* PyUnicode_AsUnicodeCopy(PyObject *unicode)
 
-   Create a copy of a Unicode string ending with a null code point. Return *NULL*
+   Create a copy of a Unicode string ending with a null code point. Return ``NULL``
    and raise a :exc:`MemoryError` exception on memory allocation failure,
    otherwise return a new allocated buffer (use :c:func:`PyMem_Free` to free
    the buffer). Note that the resulting :c:type:`Py_UNICODE*` string may
@@ -932,7 +932,7 @@ wchar_t Support
    Create a Unicode object from the :c:type:`wchar_t` buffer *w* of the given *size*.
    Passing ``-1`` as the *size* indicates that the function must itself compute the length,
    using wcslen.
-   Return *NULL* on failure.
+   Return ``NULL`` on failure.
 
 
 .. c:function:: Py_ssize_t PyUnicode_AsWideChar(PyObject *unicode, wchar_t *w, Py_ssize_t size)
@@ -951,22 +951,22 @@ wchar_t Support
 .. c:function:: wchar_t* PyUnicode_AsWideCharString(PyObject *unicode, Py_ssize_t *size)
 
    Convert the Unicode object to a wide character string. The output string
-   always ends with a null character. If *size* is not *NULL*, write the number
+   always ends with a null character. If *size* is not ``NULL``, write the number
    of wide characters (excluding the trailing null termination character) into
    *\*size*. Note that the resulting :c:type:`wchar_t` string might contain
    null characters, which would cause the string to be truncated when used with
-   most C functions. If *size* is *NULL* and the :c:type:`wchar_t*` string
+   most C functions. If *size* is ``NULL`` and the :c:type:`wchar_t*` string
    contains null characters a :exc:`ValueError` is raised.
 
    Returns a buffer allocated by :c:func:`PyMem_Alloc` (use
-   :c:func:`PyMem_Free` to free it) on success. On error, returns *NULL*
+   :c:func:`PyMem_Free` to free it) on success. On error, returns ``NULL``
    and *\*size* is undefined. Raises a :exc:`MemoryError` if memory allocation
    is failed.
 
    .. versionadded:: 3.2
 
    .. versionchanged:: 3.7
-      Raises a :exc:`ValueError` if *size* is *NULL* and the :c:type:`wchar_t*`
+      Raises a :exc:`ValueError` if *size* is ``NULL`` and the :c:type:`wchar_t*`
       string contains null characters.
 
 
@@ -982,7 +982,7 @@ Many of the following APIs take two arguments encoding and errors, and they
 have the same semantics as the ones of the built-in :func:`str` string object
 constructor.
 
-Setting encoding to *NULL* causes the default encoding to be used
+Setting encoding to ``NULL`` causes the default encoding to be used
 which is ASCII.  The file system calls should use
 :c:func:`PyUnicode_FSConverter` for encoding file names. This uses the
 variable :c:data:`Py_FileSystemDefaultEncoding` internally. This
@@ -990,7 +990,7 @@ variable should be treated as read-only: on some systems, it will be a
 pointer to a static string, on others, it will change at run-time
 (such as when the application invokes setlocale).
 
-Error handling is set by errors which may also be set to *NULL* meaning to use
+Error handling is set by errors which may also be set to ``NULL`` meaning to use
 the default handling defined for the codec.  Default error handling for all
 built-in codecs is "strict" (:exc:`ValueError` is raised).
 
@@ -1010,7 +1010,7 @@ These are the generic codec APIs:
    Create a Unicode object by decoding *size* bytes of the encoded string *s*.
    *encoding* and *errors* have the same meaning as the parameters of the same name
    in the :func:`str` built-in function.  The codec to be used is looked up
-   using the Python codec registry.  Return *NULL* if an exception was raised by
+   using the Python codec registry.  Return ``NULL`` if an exception was raised by
    the codec.
 
 
@@ -1020,7 +1020,7 @@ These are the generic codec APIs:
    Encode a Unicode object and return the result as Python bytes object.
    *encoding* and *errors* have the same meaning as the parameters of the same
    name in the Unicode :meth:`~str.encode` method. The codec to be used is looked up
-   using the Python codec registry. Return *NULL* if an exception was raised by
+   using the Python codec registry. Return ``NULL`` if an exception was raised by
    the codec.
 
 
@@ -1030,7 +1030,7 @@ These are the generic codec APIs:
    Encode the :c:type:`Py_UNICODE` buffer *s* of the given *size* and return a Python
    bytes object.  *encoding* and *errors* have the same meaning as the
    parameters of the same name in the Unicode :meth:`~str.encode` method.  The codec
-   to be used is looked up using the Python codec registry.  Return *NULL* if an
+   to be used is looked up using the Python codec registry.  Return ``NULL`` if an
    exception was raised by the codec.
 
    .. deprecated-removed:: 3.3 4.0
@@ -1047,14 +1047,14 @@ These are the UTF-8 codec APIs:
 .. c:function:: PyObject* PyUnicode_DecodeUTF8(const char *s, Py_ssize_t size, const char *errors)
 
    Create a Unicode object by decoding *size* bytes of the UTF-8 encoded string
-   *s*. Return *NULL* if an exception was raised by the codec.
+   *s*. Return ``NULL`` if an exception was raised by the codec.
 
 
 .. c:function:: PyObject* PyUnicode_DecodeUTF8Stateful(const char *s, Py_ssize_t size, \
                               const char *errors, Py_ssize_t *consumed)
 
-   If *consumed* is *NULL*, behave like :c:func:`PyUnicode_DecodeUTF8`. If
-   *consumed* is not *NULL*, trailing incomplete UTF-8 byte sequences will not be
+   If *consumed* is ``NULL``, behave like :c:func:`PyUnicode_DecodeUTF8`. If
+   *consumed* is not ``NULL``, trailing incomplete UTF-8 byte sequences will not be
    treated as an error. Those bytes will not be decoded and the number of bytes
    that have been decoded will be stored in *consumed*.
 
@@ -1062,7 +1062,7 @@ These are the UTF-8 codec APIs:
 .. c:function:: PyObject* PyUnicode_AsUTF8String(PyObject *unicode)
 
    Encode a Unicode object using UTF-8 and return the result as Python bytes
-   object.  Error handling is "strict".  Return *NULL* if an exception was
+   object.  Error handling is "strict".  Return ``NULL`` if an exception was
    raised by the codec.
 
 
@@ -1070,11 +1070,11 @@ These are the UTF-8 codec APIs:
 
    Return a pointer to the UTF-8 encoding of the Unicode object, and
    store the size of the encoded representation (in bytes) in *size*.  The
-   *size* argument can be *NULL*; in this case no size will be stored.  The
+   *size* argument can be ``NULL``; in this case no size will be stored.  The
    returned buffer always has an extra null byte appended (not included in
    *size*), regardless of whether there are any other null code points.
 
-   In the case of an error, *NULL* is returned with an exception set and no
+   In the case of an error, ``NULL`` is returned with an exception set and no
    *size* is stored.
 
    This caches the UTF-8 representation of the string in the Unicode object, and
@@ -1100,7 +1100,7 @@ These are the UTF-8 codec APIs:
 .. c:function:: PyObject* PyUnicode_EncodeUTF8(const Py_UNICODE *s, Py_ssize_t size, const char *errors)
 
    Encode the :c:type:`Py_UNICODE` buffer *s* of the given *size* using UTF-8 and
-   return a Python bytes object.  Return *NULL* if an exception was raised by
+   return a Python bytes object.  Return ``NULL`` if an exception was raised by
    the codec.
 
    .. deprecated-removed:: 3.3 4.0
@@ -1119,10 +1119,10 @@ These are the UTF-32 codec APIs:
                               const char *errors, int *byteorder)
 
    Decode *size* bytes from a UTF-32 encoded buffer string and return the
-   corresponding Unicode object.  *errors* (if non-*NULL*) defines the error
+   corresponding Unicode object.  *errors* (if non-``NULL``) defines the error
    handling. It defaults to "strict".
 
-   If *byteorder* is non-*NULL*, the decoder starts decoding using the given byte
+   If *byteorder* is non-``NULL``, the decoder starts decoding using the given byte
    order::
 
       *byteorder == -1: little endian
@@ -1137,16 +1137,16 @@ These are the UTF-32 codec APIs:
    After completion, *\*byteorder* is set to the current byte order at the end
    of input data.
 
-   If *byteorder* is *NULL*, the codec starts in native order mode.
+   If *byteorder* is ``NULL``, the codec starts in native order mode.
 
-   Return *NULL* if an exception was raised by the codec.
+   Return ``NULL`` if an exception was raised by the codec.
 
 
 .. c:function:: PyObject* PyUnicode_DecodeUTF32Stateful(const char *s, Py_ssize_t size, \
                               const char *errors, int *byteorder, Py_ssize_t *consumed)
 
-   If *consumed* is *NULL*, behave like :c:func:`PyUnicode_DecodeUTF32`. If
-   *consumed* is not *NULL*, :c:func:`PyUnicode_DecodeUTF32Stateful` will not treat
+   If *consumed* is ``NULL``, behave like :c:func:`PyUnicode_DecodeUTF32`. If
+   *consumed* is not ``NULL``, :c:func:`PyUnicode_DecodeUTF32Stateful` will not treat
    trailing incomplete UTF-32 byte sequences (such as a number of bytes not divisible
    by four) as an error. Those bytes will not be decoded and the number of bytes
    that have been decoded will be stored in *consumed*.
@@ -1156,7 +1156,7 @@ These are the UTF-32 codec APIs:
 
    Return a Python byte string using the UTF-32 encoding in native byte
    order. The string always starts with a BOM mark.  Error handling is "strict".
-   Return *NULL* if an exception was raised by the codec.
+   Return ``NULL`` if an exception was raised by the codec.
 
 
 .. c:function:: PyObject* PyUnicode_EncodeUTF32(const Py_UNICODE *s, Py_ssize_t size, \
@@ -1175,7 +1175,7 @@ These are the UTF-32 codec APIs:
    If *Py_UNICODE_WIDE* is not defined, surrogate pairs will be output
    as a single code point.
 
-   Return *NULL* if an exception was raised by the codec.
+   Return ``NULL`` if an exception was raised by the codec.
 
    .. deprecated-removed:: 3.3 4.0
       Part of the old-style :c:type:`Py_UNICODE` API; please migrate to using
@@ -1192,10 +1192,10 @@ These are the UTF-16 codec APIs:
                               const char *errors, int *byteorder)
 
    Decode *size* bytes from a UTF-16 encoded buffer string and return the
-   corresponding Unicode object.  *errors* (if non-*NULL*) defines the error
+   corresponding Unicode object.  *errors* (if non-``NULL``) defines the error
    handling. It defaults to "strict".
 
-   If *byteorder* is non-*NULL*, the decoder starts decoding using the given byte
+   If *byteorder* is non-``NULL``, the decoder starts decoding using the given byte
    order::
 
       *byteorder == -1: little endian
@@ -1211,16 +1211,16 @@ These are the UTF-16 codec APIs:
    After completion, *\*byteorder* is set to the current byte order at the end
    of input data.
 
-   If *byteorder* is *NULL*, the codec starts in native order mode.
+   If *byteorder* is ``NULL``, the codec starts in native order mode.
 
-   Return *NULL* if an exception was raised by the codec.
+   Return ``NULL`` if an exception was raised by the codec.
 
 
 .. c:function:: PyObject* PyUnicode_DecodeUTF16Stateful(const char *s, Py_ssize_t size, \
                               const char *errors, int *byteorder, Py_ssize_t *consumed)
 
-   If *consumed* is *NULL*, behave like :c:func:`PyUnicode_DecodeUTF16`. If
-   *consumed* is not *NULL*, :c:func:`PyUnicode_DecodeUTF16Stateful` will not treat
+   If *consumed* is ``NULL``, behave like :c:func:`PyUnicode_DecodeUTF16`. If
+   *consumed* is not ``NULL``, :c:func:`PyUnicode_DecodeUTF16Stateful` will not treat
    trailing incomplete UTF-16 byte sequences (such as an odd number of bytes or a
    split surrogate pair) as an error. Those bytes will not be decoded and the
    number of bytes that have been decoded will be stored in *consumed*.
@@ -1230,7 +1230,7 @@ These are the UTF-16 codec APIs:
 
    Return a Python byte string using the UTF-16 encoding in native byte
    order. The string always starts with a BOM mark.  Error handling is "strict".
-   Return *NULL* if an exception was raised by the codec.
+   Return ``NULL`` if an exception was raised by the codec.
 
 
 .. c:function:: PyObject* PyUnicode_EncodeUTF16(const Py_UNICODE *s, Py_ssize_t size, \
@@ -1250,7 +1250,7 @@ These are the UTF-16 codec APIs:
    represented as a surrogate pair. If it is not defined, each :c:type:`Py_UNICODE`
    values is interpreted as a UCS-2 character.
 
-   Return *NULL* if an exception was raised by the codec.
+   Return ``NULL`` if an exception was raised by the codec.
 
    .. deprecated-removed:: 3.3 4.0
       Part of the old-style :c:type:`Py_UNICODE` API; please migrate to using
@@ -1266,14 +1266,14 @@ These are the UTF-7 codec APIs:
 .. c:function:: PyObject* PyUnicode_DecodeUTF7(const char *s, Py_ssize_t size, const char *errors)
 
    Create a Unicode object by decoding *size* bytes of the UTF-7 encoded string
-   *s*.  Return *NULL* if an exception was raised by the codec.
+   *s*.  Return ``NULL`` if an exception was raised by the codec.
 
 
 .. c:function:: PyObject* PyUnicode_DecodeUTF7Stateful(const char *s, Py_ssize_t size, \
                               const char *errors, Py_ssize_t *consumed)
 
-   If *consumed* is *NULL*, behave like :c:func:`PyUnicode_DecodeUTF7`.  If
-   *consumed* is not *NULL*, trailing incomplete UTF-7 base-64 sections will not
+   If *consumed* is ``NULL``, behave like :c:func:`PyUnicode_DecodeUTF7`.  If
+   *consumed* is not ``NULL``, trailing incomplete UTF-7 base-64 sections will not
    be treated as an error.  Those bytes will not be decoded and the number of
    bytes that have been decoded will be stored in *consumed*.
 
@@ -1282,7 +1282,7 @@ These are the UTF-7 codec APIs:
                               int base64SetO, int base64WhiteSpace, const char *errors)
 
    Encode the :c:type:`Py_UNICODE` buffer of the given size using UTF-7 and
-   return a Python bytes object.  Return *NULL* if an exception was raised by
+   return a Python bytes object.  Return ``NULL`` if an exception was raised by
    the codec.
 
    If *base64SetO* is nonzero, "Set O" (punctuation that has no otherwise
@@ -1305,20 +1305,20 @@ These are the "Unicode Escape" codec APIs:
                               Py_ssize_t size, const char *errors)
 
    Create a Unicode object by decoding *size* bytes of the Unicode-Escape encoded
-   string *s*.  Return *NULL* if an exception was raised by the codec.
+   string *s*.  Return ``NULL`` if an exception was raised by the codec.
 
 
 .. c:function:: PyObject* PyUnicode_AsUnicodeEscapeString(PyObject *unicode)
 
    Encode a Unicode object using Unicode-Escape and return the result as a
-   bytes object.  Error handling is "strict".  Return *NULL* if an exception was
+   bytes object.  Error handling is "strict".  Return ``NULL`` if an exception was
    raised by the codec.
 
 
 .. c:function:: PyObject* PyUnicode_EncodeUnicodeEscape(const Py_UNICODE *s, Py_ssize_t size)
 
    Encode the :c:type:`Py_UNICODE` buffer of the given *size* using Unicode-Escape and
-   return a bytes object.  Return *NULL* if an exception was raised by the codec.
+   return a bytes object.  Return ``NULL`` if an exception was raised by the codec.
 
    .. deprecated-removed:: 3.3 4.0
       Part of the old-style :c:type:`Py_UNICODE` API; please migrate to using
@@ -1335,13 +1335,13 @@ These are the "Raw Unicode Escape" codec APIs:
                               Py_ssize_t size, const char *errors)
 
    Create a Unicode object by decoding *size* bytes of the Raw-Unicode-Escape
-   encoded string *s*.  Return *NULL* if an exception was raised by the codec.
+   encoded string *s*.  Return ``NULL`` if an exception was raised by the codec.
 
 
 .. c:function:: PyObject* PyUnicode_AsRawUnicodeEscapeString(PyObject *unicode)
 
    Encode a Unicode object using Raw-Unicode-Escape and return the result as
-   a bytes object.  Error handling is "strict".  Return *NULL* if an exception
+   a bytes object.  Error handling is "strict".  Return ``NULL`` if an exception
    was raised by the codec.
 
 
@@ -1349,7 +1349,7 @@ These are the "Raw Unicode Escape" codec APIs:
                               Py_ssize_t size)
 
    Encode the :c:type:`Py_UNICODE` buffer of the given *size* using Raw-Unicode-Escape
-   and return a bytes object.  Return *NULL* if an exception was raised by the codec.
+   and return a bytes object.  Return ``NULL`` if an exception was raised by the codec.
 
    .. deprecated-removed:: 3.3 4.0
       Part of the old-style :c:type:`Py_UNICODE` API; please migrate to using
@@ -1367,20 +1367,20 @@ ordinals and only these are accepted by the codecs during encoding.
 .. c:function:: PyObject* PyUnicode_DecodeLatin1(const char *s, Py_ssize_t size, const char *errors)
 
    Create a Unicode object by decoding *size* bytes of the Latin-1 encoded string
-   *s*.  Return *NULL* if an exception was raised by the codec.
+   *s*.  Return ``NULL`` if an exception was raised by the codec.
 
 
 .. c:function:: PyObject* PyUnicode_AsLatin1String(PyObject *unicode)
 
    Encode a Unicode object using Latin-1 and return the result as Python bytes
-   object.  Error handling is "strict".  Return *NULL* if an exception was
+   object.  Error handling is "strict".  Return ``NULL`` if an exception was
    raised by the codec.
 
 
 .. c:function:: PyObject* PyUnicode_EncodeLatin1(const Py_UNICODE *s, Py_ssize_t size, const char *errors)
 
    Encode the :c:type:`Py_UNICODE` buffer of the given *size* using Latin-1 and
-   return a Python bytes object.  Return *NULL* if an exception was raised by
+   return a Python bytes object.  Return ``NULL`` if an exception was raised by
    the codec.
 
    .. deprecated-removed:: 3.3 4.0
@@ -1399,20 +1399,20 @@ codes generate errors.
 .. c:function:: PyObject* PyUnicode_DecodeASCII(const char *s, Py_ssize_t size, const char *errors)
 
    Create a Unicode object by decoding *size* bytes of the ASCII encoded string
-   *s*.  Return *NULL* if an exception was raised by the codec.
+   *s*.  Return ``NULL`` if an exception was raised by the codec.
 
 
 .. c:function:: PyObject* PyUnicode_AsASCIIString(PyObject *unicode)
 
    Encode a Unicode object using ASCII and return the result as Python bytes
-   object.  Error handling is "strict".  Return *NULL* if an exception was
+   object.  Error handling is "strict".  Return ``NULL`` if an exception was
    raised by the codec.
 
 
 .. c:function:: PyObject* PyUnicode_EncodeASCII(const Py_UNICODE *s, Py_ssize_t size, const char *errors)
 
    Encode the :c:type:`Py_UNICODE` buffer of the given *size* using ASCII and
-   return a Python bytes object.  Return *NULL* if an exception was raised by
+   return a Python bytes object.  Return ``NULL`` if an exception was raised by
    the codec.
 
    .. deprecated-removed:: 3.3 4.0
@@ -1436,10 +1436,10 @@ These are the mapping codec APIs:
                               PyObject *mapping, const char *errors)
 
    Create a Unicode object by decoding *size* bytes of the encoded string *s*
-   using the given *mapping* object.  Return *NULL* if an exception was raised
+   using the given *mapping* object.  Return ``NULL`` if an exception was raised
    by the codec.
 
-   If *mapping* is *NULL*, Latin-1 decoding will be applied.  Else
+   If *mapping* is ``NULL``, Latin-1 decoding will be applied.  Else
    *mapping* must map bytes ordinals (integers in the range from 0 to 255)
    to Unicode strings, integers (which are then interpreted as Unicode
    ordinals) or ``None``.  Unmapped data bytes -- ones which cause a
@@ -1451,7 +1451,7 @@ These are the mapping codec APIs:
 .. c:function:: PyObject* PyUnicode_AsCharmapString(PyObject *unicode, PyObject *mapping)
 
    Encode a Unicode object using the given *mapping* object and return the
-   result as a bytes object.  Error handling is "strict".  Return *NULL* if an
+   result as a bytes object.  Error handling is "strict".  Return ``NULL`` if an
    exception was raised by the codec.
 
    The *mapping* object must map Unicode ordinal integers to bytes objects,
@@ -1464,7 +1464,7 @@ These are the mapping codec APIs:
                               PyObject *mapping, const char *errors)
 
    Encode the :c:type:`Py_UNICODE` buffer of the given *size* using the given
-   *mapping* object and return the result as a bytes object.  Return *NULL* if
+   *mapping* object and return the result as a bytes object.  Return ``NULL`` if
    an exception was raised by the codec.
 
    .. deprecated-removed:: 3.3 4.0
@@ -1479,7 +1479,7 @@ The following codec API is special in that maps Unicode to Unicode.
                               PyObject *mapping, const char *errors)
 
    Translate a Unicode object using the given *mapping* object and return the
-   resulting Unicode object.  Return *NULL* if an exception was raised by the
+   resulting Unicode object.  Return ``NULL`` if an exception was raised by the
    codec.
 
    The *mapping* object must map Unicode ordinal integers to Unicode strings,
@@ -1493,7 +1493,7 @@ The following codec API is special in that maps Unicode to Unicode.
 
    Translate a :c:type:`Py_UNICODE` buffer of the given *size* by applying a
    character *mapping* table to it and return the resulting Unicode object.
-   Return *NULL* when an exception was raised by the codec.
+   Return ``NULL`` when an exception was raised by the codec.
 
    .. deprecated-removed:: 3.3 4.0
       Part of the old-style :c:type:`Py_UNICODE` API; please migrate to using
@@ -1512,14 +1512,14 @@ the user settings on the machine running the codec.
 .. c:function:: PyObject* PyUnicode_DecodeMBCS(const char *s, Py_ssize_t size, const char *errors)
 
    Create a Unicode object by decoding *size* bytes of the MBCS encoded string *s*.
-   Return *NULL* if an exception was raised by the codec.
+   Return ``NULL`` if an exception was raised by the codec.
 
 
 .. c:function:: PyObject* PyUnicode_DecodeMBCSStateful(const char *s, Py_ssize_t size, \
                               const char *errors, Py_ssize_t *consumed)
 
-   If *consumed* is *NULL*, behave like :c:func:`PyUnicode_DecodeMBCS`. If
-   *consumed* is not *NULL*, :c:func:`PyUnicode_DecodeMBCSStateful` will not decode
+   If *consumed* is ``NULL``, behave like :c:func:`PyUnicode_DecodeMBCS`. If
+   *consumed* is not ``NULL``, :c:func:`PyUnicode_DecodeMBCSStateful` will not decode
    trailing lead byte and the number of bytes that have been decoded will be stored
    in *consumed*.
 
@@ -1527,14 +1527,14 @@ the user settings on the machine running the codec.
 .. c:function:: PyObject* PyUnicode_AsMBCSString(PyObject *unicode)
 
    Encode a Unicode object using MBCS and return the result as Python bytes
-   object.  Error handling is "strict".  Return *NULL* if an exception was
+   object.  Error handling is "strict".  Return ``NULL`` if an exception was
    raised by the codec.
 
 
 .. c:function:: PyObject* PyUnicode_EncodeCodePage(int code_page, PyObject *unicode, const char *errors)
 
    Encode the Unicode object using the specified code page and return a Python
-   bytes object.  Return *NULL* if an exception was raised by the codec. Use
+   bytes object.  Return ``NULL`` if an exception was raised by the codec. Use
    :c:data:`CP_ACP` code page to get the MBCS encoder.
 
    .. versionadded:: 3.3
@@ -1543,7 +1543,7 @@ the user settings on the machine running the codec.
 .. c:function:: PyObject* PyUnicode_EncodeMBCS(const Py_UNICODE *s, Py_ssize_t size, const char *errors)
 
    Encode the :c:type:`Py_UNICODE` buffer of the given *size* using MBCS and return
-   a Python bytes object.  Return *NULL* if an exception was raised by the
+   a Python bytes object.  Return ``NULL`` if an exception was raised by the
    codec.
 
    .. deprecated-removed:: 3.3 4.0
@@ -1565,7 +1565,7 @@ The following APIs are capable of handling Unicode objects and strings on input
 (we refer to them as strings in the descriptions) and return Unicode objects or
 integers as appropriate.
 
-They all return *NULL* or ``-1`` if an exception occurs.
+They all return ``NULL`` or ``-1`` if an exception occurs.
 
 
 .. c:function:: PyObject* PyUnicode_Concat(PyObject *left, PyObject *right)
@@ -1575,7 +1575,7 @@ They all return *NULL* or ``-1`` if an exception occurs.
 
 .. c:function:: PyObject* PyUnicode_Split(PyObject *s, PyObject *sep, Py_ssize_t maxsplit)
 
-   Split a string giving a list of Unicode strings.  If *sep* is *NULL*, splitting
+   Split a string giving a list of Unicode strings.  If *sep* is ``NULL``, splitting
    will be done at all whitespace substrings.  Otherwise, splits occur at the given
    separator.  At most *maxsplit* splits will be done.  If negative, no limit is
    set.  Separators are not included in the resulting list.
@@ -1601,7 +1601,7 @@ They all return *NULL* or ``-1`` if an exception occurs.
    and sequences work well.  Unmapped character ordinals (ones which cause a
    :exc:`LookupError`) are left untouched and are copied as-is.
 
-   *errors* has the usual meaning for codecs. It may be *NULL* which indicates to
+   *errors* has the usual meaning for codecs. It may be ``NULL`` which indicates to
    use the default error handling.
 
 
diff --git a/Doc/c-api/veryhigh.rst b/Doc/c-api/veryhigh.rst
index 317093e95615e..3d92fe9acb58c 100644
--- a/Doc/c-api/veryhigh.rst
+++ b/Doc/c-api/veryhigh.rst
@@ -45,7 +45,7 @@ the same library that the Python runtime is using.
 .. c:function:: int PyRun_AnyFile(FILE *fp, const char *filename)
 
    This is a simplified interface to :c:func:`PyRun_AnyFileExFlags` below, leaving
-   *closeit* set to ``0`` and *flags* set to *NULL*.
+   *closeit* set to ``0`` and *flags* set to ``NULL``.
 
 
 .. c:function:: int PyRun_AnyFileFlags(FILE *fp, const char *filename, PyCompilerFlags *flags)
@@ -57,7 +57,7 @@ the same library that the Python runtime is using.
 .. c:function:: int PyRun_AnyFileEx(FILE *fp, const char *filename, int closeit)
 
    This is a simplified interface to :c:func:`PyRun_AnyFileExFlags` below, leaving
-   the *flags* argument set to *NULL*.
+   the *flags* argument set to ``NULL``.
 
 
 .. c:function:: int PyRun_AnyFileExFlags(FILE *fp, const char *filename, int closeit, PyCompilerFlags *flags)
@@ -66,7 +66,7 @@ the same library that the Python runtime is using.
    terminal input or Unix pseudo-terminal), return the value of
    :c:func:`PyRun_InteractiveLoop`, otherwise return the result of
    :c:func:`PyRun_SimpleFile`.  *filename* is decoded from the filesystem
-   encoding (:func:`sys.getfilesystemencoding`).  If *filename* is *NULL*, this
+   encoding (:func:`sys.getfilesystemencoding`).  If *filename* is ``NULL``, this
    function uses ``"???"`` as the filename.
 
 
@@ -92,13 +92,13 @@ the same library that the Python runtime is using.
 .. c:function:: int PyRun_SimpleFile(FILE *fp, const char *filename)
 
    This is a simplified interface to :c:func:`PyRun_SimpleFileExFlags` below,
-   leaving *closeit* set to ``0`` and *flags* set to *NULL*.
+   leaving *closeit* set to ``0`` and *flags* set to ``NULL``.
 
 
 .. c:function:: int PyRun_SimpleFileEx(FILE *fp, const char *filename, int closeit)
 
    This is a simplified interface to :c:func:`PyRun_SimpleFileExFlags` below,
-   leaving *flags* set to *NULL*.
+   leaving *flags* set to ``NULL``.
 
 
 .. c:function:: int PyRun_SimpleFileExFlags(FILE *fp, const char *filename, int closeit, PyCompilerFlags *flags)
@@ -117,7 +117,7 @@ the same library that the Python runtime is using.
 .. c:function:: int PyRun_InteractiveOne(FILE *fp, const char *filename)
 
    This is a simplified interface to :c:func:`PyRun_InteractiveOneFlags` below,
-   leaving *flags* set to *NULL*.
+   leaving *flags* set to ``NULL``.
 
 
 .. c:function:: int PyRun_InteractiveOneFlags(FILE *fp, const char *filename, PyCompilerFlags *flags)
@@ -137,7 +137,7 @@ the same library that the Python runtime is using.
 .. c:function:: int PyRun_InteractiveLoop(FILE *fp, const char *filename)
 
    This is a simplified interface to :c:func:`PyRun_InteractiveLoopFlags` below,
-   leaving *flags* set to *NULL*.
+   leaving *flags* set to ``NULL``.
 
 
 .. c:function:: int PyRun_InteractiveLoopFlags(FILE *fp, const char *filename, PyCompilerFlags *flags)
@@ -166,13 +166,13 @@ the same library that the Python runtime is using.
    ``char *func(FILE *stdin, FILE *stdout, char *prompt)``,
    overriding the default function used to read a single line of input
    at the interpreter's prompt.  The function is expected to output
-   the string *prompt* if it's not *NULL*, and then read a line of
+   the string *prompt* if it's not ``NULL``, and then read a line of
    input from the provided standard input file, returning the
    resulting string.  For example, The :mod:`readline` module sets
    this hook to provide line-editing and tab-completion features.
 
    The result must be a string allocated by :c:func:`PyMem_RawMalloc` or
-   :c:func:`PyMem_RawRealloc`, or *NULL* if an error occurred.
+   :c:func:`PyMem_RawRealloc`, or ``NULL`` if an error occurred.
 
    .. versionchanged:: 3.4
       The result must be allocated by :c:func:`PyMem_RawMalloc` or
@@ -184,14 +184,14 @@ the same library that the Python runtime is using.
 
    This is a simplified interface to
    :c:func:`PyParser_SimpleParseStringFlagsFilename` below, leaving  *filename* set
-   to *NULL* and *flags* set to ``0``.
+   to ``NULL`` and *flags* set to ``0``.
 
 
 .. c:function:: struct _node* PyParser_SimpleParseStringFlags( const char *str, int start, int flags)
 
    This is a simplified interface to
    :c:func:`PyParser_SimpleParseStringFlagsFilename` below, leaving  *filename* set
-   to *NULL*.
+   to ``NULL``.
 
 
 .. c:function:: struct _node* PyParser_SimpleParseStringFlagsFilename( const char *str, const char *filename, int start, int flags)
@@ -218,7 +218,7 @@ the same library that the Python runtime is using.
 .. c:function:: PyObject* PyRun_String(const char *str, int start, PyObject *globals, PyObject *locals)
 
    This is a simplified interface to :c:func:`PyRun_StringFlags` below, leaving
-   *flags* set to *NULL*.
+   *flags* set to ``NULL``.
 
 
 .. c:function:: PyObject* PyRun_StringFlags(const char *str, int start, PyObject *globals, PyObject *locals, PyCompilerFlags *flags)
@@ -229,20 +229,20 @@ the same library that the Python runtime is using.
    that implements the mapping protocol.  The parameter *start* specifies
    the start token that should be used to parse the source code.
 
-   Returns the result of executing the code as a Python object, or *NULL* if an
+   Returns the result of executing the code as a Python object, or ``NULL`` if an
    exception was raised.
 
 
 .. c:function:: PyObject* PyRun_File(FILE *fp, const char *filename, int start, PyObject *globals, PyObject *locals)
 
    This is a simplified interface to :c:func:`PyRun_FileExFlags` below, leaving
-   *closeit* set to ``0`` and *flags* set to *NULL*.
+   *closeit* set to ``0`` and *flags* set to ``NULL``.
 
 
 .. c:function:: PyObject* PyRun_FileEx(FILE *fp, const char *filename, int start, PyObject *globals, PyObject *locals, int closeit)
 
    This is a simplified interface to :c:func:`PyRun_FileExFlags` below, leaving
-   *flags* set to *NULL*.
+   *flags* set to ``NULL``.
 
 
 .. c:function:: PyObject* PyRun_FileFlags(FILE *fp, const char *filename, int start, PyObject *globals, PyObject *locals, PyCompilerFlags *flags)
@@ -263,7 +263,7 @@ the same library that the Python runtime is using.
 .. c:function:: PyObject* Py_CompileString(const char *str, const char *filename, int start)
 
    This is a simplified interface to :c:func:`Py_CompileStringFlags` below, leaving
-   *flags* set to *NULL*.
+   *flags* set to ``NULL``.
 
 
 .. c:function:: PyObject* Py_CompileStringFlags(const char *str, const char *filename, int start, PyCompilerFlags *flags)
@@ -279,7 +279,7 @@ the same library that the Python runtime is using.
    code which can be compiled and should be :const:`Py_eval_input`,
    :const:`Py_file_input`, or :const:`Py_single_input`.  The filename specified by
    *filename* is used to construct the code object and may appear in tracebacks or
-   :exc:`SyntaxError` exception messages.  This returns *NULL* if the code
+   :exc:`SyntaxError` exception messages.  This returns ``NULL`` if the code
    cannot be parsed or compiled.
 
    The integer *optimize* specifies the optimization level of the compiler; a
@@ -302,7 +302,7 @@ the same library that the Python runtime is using.
 
    This is a simplified interface to :c:func:`PyEval_EvalCodeEx`, with just
    the code object, and global and local variables.  The other arguments are
-   set to *NULL*.
+   set to ``NULL``.
 
 
 .. c:function:: PyObject* PyEval_EvalCodeEx(PyObject *co, PyObject *globals, PyObject *locals, PyObject *const *args, int argcount, PyObject *const *kws, int kwcount, PyObject *const *defs, int defcount, PyObject *kwdefs, PyObject *closure)
@@ -379,7 +379,7 @@ the same library that the Python runtime is using.
    executed, it is passed as ``PyCompilerFlags *flags``.  In this case, ``from
    __future__ import`` can modify *flags*.
 
-   Whenever ``PyCompilerFlags *flags`` is *NULL*, :attr:`cf_flags` is treated as
+   Whenever ``PyCompilerFlags *flags`` is ``NULL``, :attr:`cf_flags` is treated as
    equal to ``0``, and any modification due to ``from __future__ import`` is
    discarded.  ::
 
diff --git a/Doc/c-api/weakref.rst b/Doc/c-api/weakref.rst
index 6cb3e33fe843a..0938ee1928a0b 100644
--- a/Doc/c-api/weakref.rst
+++ b/Doc/c-api/weakref.rst
@@ -33,9 +33,9 @@ as much as it can.
    reference object may be returned.  The second parameter, *callback*, can be a
    callable object that receives notification when *ob* is garbage collected; it
    should accept a single parameter, which will be the weak reference object
-   itself. *callback* may also be ``None`` or *NULL*.  If *ob* is not a
+   itself. *callback* may also be ``None`` or ``NULL``.  If *ob* is not a
    weakly-referencable object, or if *callback* is not callable, ``None``, or
-   *NULL*, this will return *NULL* and raise :exc:`TypeError`.
+   ``NULL``, this will return ``NULL`` and raise :exc:`TypeError`.
 
 
 .. c:function:: PyObject* PyWeakref_NewProxy(PyObject *ob, PyObject *callback)
@@ -45,9 +45,9 @@ as much as it can.
    existing proxy object may be returned.  The second parameter, *callback*, can
    be a callable object that receives notification when *ob* is garbage
    collected; it should accept a single parameter, which will be the weak
-   reference object itself. *callback* may also be ``None`` or *NULL*.  If *ob*
+   reference object itself. *callback* may also be ``None`` or ``NULL``.  If *ob*
    is not a weakly-referencable object, or if *callback* is not callable,
-   ``None``, or *NULL*, this will return *NULL* and raise :exc:`TypeError`.
+   ``None``, or ``NULL``, this will return ``NULL`` and raise :exc:`TypeError`.
 
 
 .. c:function:: PyObject* PyWeakref_GetObject(PyObject *ref)
diff --git a/Doc/extending/embedding.rst b/Doc/extending/embedding.rst
index 13d83b72f82ab..881390da0e275 100644
--- a/Doc/extending/embedding.rst
+++ b/Doc/extending/embedding.rst
@@ -196,7 +196,7 @@ function is then made with::
 
    pValue = PyObject_CallObject(pFunc, pArgs);
 
-Upon return of the function, ``pValue`` is either *NULL* or it contains a
+Upon return of the function, ``pValue`` is either ``NULL`` or it contains a
 reference to the return value of the function.  Be sure to release the reference
 after examining the value.
 
diff --git a/Doc/extending/extending.rst b/Doc/extending/extending.rst
index 73a77c781bc29..da463e4f23ede 100644
--- a/Doc/extending/extending.rst
+++ b/Doc/extending/extending.rst
@@ -117,7 +117,7 @@ store the converted values.  More about this later.
 type and its components have been stored in the variables whose addresses are
 passed.  It returns false (zero) if an invalid argument list was passed.  In the
 latter case it also raises an appropriate exception so the calling function can
-return *NULL* immediately (as we saw in the example).
+return ``NULL`` immediately (as we saw in the example).
 
 
 .. _extending-errors:
@@ -127,8 +127,8 @@ Intermezzo: Errors and Exceptions
 
 An important convention throughout the Python interpreter is the following: when
 a function fails, it should set an exception condition and return an error value
-(usually a *NULL* pointer).  Exceptions are stored in a static global variable
-inside the interpreter; if this variable is *NULL* no exception has occurred.  A
+(usually a ``NULL`` pointer).  Exceptions are stored in a static global variable
+inside the interpreter; if this variable is ``NULL`` no exception has occurred.  A
 second global variable stores the "associated value" of the exception (the
 second argument to :keyword:`raise`).  A third variable contains the stack
 traceback in case the error originated in Python code.  These three variables
@@ -152,13 +152,13 @@ its associated value.  You don't need to :c:func:`Py_INCREF` the objects passed
 to any of these functions.
 
 You can test non-destructively whether an exception has been set with
-:c:func:`PyErr_Occurred`.  This returns the current exception object, or *NULL*
+:c:func:`PyErr_Occurred`.  This returns the current exception object, or ``NULL``
 if no exception has occurred.  You normally don't need to call
 :c:func:`PyErr_Occurred` to see whether an error occurred in a function call,
 since you should be able to tell from the return value.
 
 When a function *f* that calls another function *g* detects that the latter
-fails, *f* should itself return an error value (usually *NULL* or ``-1``).  It
+fails, *f* should itself return an error value (usually ``NULL`` or ``-1``).  It
 should *not* call one of the :c:func:`PyErr_\*` functions --- one has already
 been called by *g*. *f*'s caller is then supposed to also return an error
 indication to *its* caller, again *without* calling :c:func:`PyErr_\*`, and so on
@@ -234,7 +234,7 @@ with an exception object::
 
 Note that the Python name for the exception object is :exc:`spam.error`.  The
 :c:func:`PyErr_NewException` function may create a class with the base class
-being :exc:`Exception` (unless another class is passed in instead of *NULL*),
+being :exc:`Exception` (unless another class is passed in instead of ``NULL``),
 described in :ref:`bltin-exceptions`.
 
 Note also that the :c:data:`SpamError` variable retains a reference to the newly
@@ -278,7 +278,7 @@ statement::
    if (!PyArg_ParseTuple(args, "s", &command))
        return NULL;
 
-It returns *NULL* (the error indicator for functions returning object pointers)
+It returns ``NULL`` (the error indicator for functions returning object pointers)
 if an error is detected in the argument list, relying on the exception set by
 :c:func:`PyArg_ParseTuple`.  Otherwise the string value of the argument has been
 copied to the local variable :c:data:`command`.  This is a pointer assignment and
@@ -308,7 +308,7 @@ macro)::
    return Py_None;
 
 :c:data:`Py_None` is the C name for the special Python object ``None``.  It is a
-genuine Python object rather than a *NULL* pointer, which means "error" in most
+genuine Python object rather than a ``NULL`` pointer, which means "error" in most
 contexts, as we have seen.
 
 
@@ -376,7 +376,7 @@ inserts built-in function objects into the newly created module based upon the
 table (an array of :c:type:`PyMethodDef` structures) found in the module definition.
 :c:func:`PyModule_Create` returns a pointer to the module object
 that it creates.  It may abort with a fatal error for
-certain errors, or return *NULL* if the module could not be initialized
+certain errors, or return ``NULL`` if the module could not be initialized
 satisfactorily. The init function must return the module object to its caller,
 so that it then gets inserted into ``sys.modules``.
 
@@ -526,8 +526,8 @@ This function must be registered with the interpreter using the
 :ref:`parsetuple`.
 
 The macros :c:func:`Py_XINCREF` and :c:func:`Py_XDECREF` increment/decrement the
-reference count of an object and are safe in the presence of *NULL* pointers
-(but note that *temp* will not be  *NULL* in this context).  More info on them
+reference count of an object and are safe in the presence of ``NULL`` pointers
+(but note that *temp* will not be  ``NULL`` in this context).  More info on them
 in section :ref:`refcounts`.
 
 .. index:: single: PyObject_CallObject()
@@ -566,7 +566,7 @@ somehow :c:func:`Py_DECREF` the result, even (especially!) if you are not
 interested in its value.
 
 Before you do this, however, it is important to check that the return value
-isn't *NULL*.  If it is, the Python function terminated by raising an exception.
+isn't ``NULL``.  If it is, the Python function terminated by raising an exception.
 If the C code that called :c:func:`PyObject_CallObject` is called from Python, it
 should now return an error indication to its Python caller, so the interpreter
 can print a stack trace, or the calling Python code can handle the exception.
@@ -723,7 +723,7 @@ The :c:func:`PyArg_ParseTupleAndKeywords` function is declared as follows::
 The *arg* and *format* parameters are identical to those of the
 :c:func:`PyArg_ParseTuple` function.  The *kwdict* parameter is the dictionary of
 keywords received as the third parameter from the Python runtime.  The *kwlist*
-parameter is a *NULL*-terminated list of strings which identify the parameters;
+parameter is a ``NULL``-terminated list of strings which identify the parameters;
 the names are matched with the type information from *format* from left to
 right.  On success, :c:func:`PyArg_ParseTupleAndKeywords` returns true, otherwise
 it returns false and raises an appropriate exception.
@@ -1084,32 +1084,32 @@ NULL Pointers
 -------------
 
 In general, functions that take object references as arguments do not expect you
-to pass them *NULL* pointers, and will dump core (or cause later core dumps) if
-you do so.  Functions that return object references generally return *NULL* only
-to indicate that an exception occurred.  The reason for not testing for *NULL*
+to pass them ``NULL`` pointers, and will dump core (or cause later core dumps) if
+you do so.  Functions that return object references generally return ``NULL`` only
+to indicate that an exception occurred.  The reason for not testing for ``NULL``
 arguments is that functions often pass the objects they receive on to other
-function --- if each function were to test for *NULL*, there would be a lot of
+function --- if each function were to test for ``NULL``, there would be a lot of
 redundant tests and the code would run more slowly.
 
-It is better to test for *NULL* only at the "source:" when a pointer that may be
-*NULL* is received, for example, from :c:func:`malloc` or from a function that
+It is better to test for ``NULL`` only at the "source:" when a pointer that may be
+``NULL`` is received, for example, from :c:func:`malloc` or from a function that
 may raise an exception.
 
-The macros :c:func:`Py_INCREF` and :c:func:`Py_DECREF` do not check for *NULL*
+The macros :c:func:`Py_INCREF` and :c:func:`Py_DECREF` do not check for ``NULL``
 pointers --- however, their variants :c:func:`Py_XINCREF` and :c:func:`Py_XDECREF`
 do.
 
 The macros for checking for a particular object type (``Pytype_Check()``) don't
-check for *NULL* pointers --- again, there is much code that calls several of
+check for ``NULL`` pointers --- again, there is much code that calls several of
 these in a row to test an object against various different expected types, and
-this would generate redundant tests.  There are no variants with *NULL*
+this would generate redundant tests.  There are no variants with ``NULL``
 checking.
 
 The C function calling mechanism guarantees that the argument list passed to C
-functions (``args`` in the examples) is never *NULL* --- in fact it guarantees
+functions (``args`` in the examples) is never ``NULL`` --- in fact it guarantees
 that it is always a tuple [#]_.
 
-It is a severe error to ever let a *NULL* pointer "escape" to the Python user.
+It is a severe error to ever let a ``NULL`` pointer "escape" to the Python user.
 
 .. Frank Stajano:
    A pedagogically buggy example, along the lines of the previous listing, would
@@ -1184,7 +1184,7 @@ different ways between the module providing the code and the client modules.
 
 Whichever method you choose, it's important to name your Capsules properly.
 The function :c:func:`PyCapsule_New` takes a name parameter
-(:c:type:`const char \*`); you're permitted to pass in a *NULL* name, but
+(:c:type:`const char \*`); you're permitted to pass in a ``NULL`` name, but
 we strongly encourage you to specify a name.  Properly named Capsules provide
 a degree of runtime type-safety; there is no feasible way to tell one unnamed
 Capsule from another.
diff --git a/Doc/extending/newtypes.rst b/Doc/extending/newtypes.rst
index d0d2ec1f88207..315300e192576 100644
--- a/Doc/extending/newtypes.rst
+++ b/Doc/extending/newtypes.rst
@@ -189,7 +189,7 @@ For every object which can support attributes, the corresponding type must
 provide the functions that control how the attributes are resolved.  There needs
 to be a function which can retrieve attributes (if any are defined), and another
 to set attributes (if setting attributes is allowed).  Removing an attribute is
-a special case, for which the new value passed to the handler is *NULL*.
+a special case, for which the new value passed to the handler is ``NULL``.
 
 Python supports two pairs of attribute handlers; a type that supports attributes
 only needs to implement the functions for one pair.  The difference is that one
@@ -231,9 +231,9 @@ attributes, when the values are computed, or how relevant data is stored.
 When :c:func:`PyType_Ready` is called, it uses three tables referenced by the
 type object to create :term:`descriptor`\s which are placed in the dictionary of the
 type object.  Each descriptor controls access to one attribute of the instance
-object.  Each of the tables is optional; if all three are *NULL*, instances of
+object.  Each of the tables is optional; if all three are ``NULL``, instances of
 the type will only have attributes that are inherited from their base type, and
-should leave the :c:member:`~PyTypeObject.tp_getattro` and :c:member:`~PyTypeObject.tp_setattro` fields *NULL* as
+should leave the :c:member:`~PyTypeObject.tp_getattro` and :c:member:`~PyTypeObject.tp_setattro` fields ``NULL`` as
 well, allowing the base type to handle attributes.
 
 The tables are declared as three fields of the type object::
@@ -242,7 +242,7 @@ The tables are declared as three fields of the type object::
    struct PyMemberDef *tp_members;
    struct PyGetSetDef *tp_getset;
 
-If :c:member:`~PyTypeObject.tp_methods` is not *NULL*, it must refer to an array of
+If :c:member:`~PyTypeObject.tp_methods` is not ``NULL``, it must refer to an array of
 :c:type:`PyMethodDef` structures.  Each entry in the table is an instance of this
 structure::
 
@@ -256,7 +256,7 @@ structure::
 One entry should be defined for each method provided by the type; no entries are
 needed for methods inherited from a base type.  One additional entry is needed
 at the end; it is a sentinel that marks the end of the array.  The
-:attr:`ml_name` field of the sentinel must be *NULL*.
+:attr:`ml_name` field of the sentinel must be ``NULL``.
 
 The second table is used to define attributes which map directly to data stored
 in the instance.  A variety of primitive C types are supported, and access may
@@ -305,7 +305,7 @@ application can use the introspection API to retrieve the descriptor from the
 class object, and get the doc string using its :attr:`__doc__` attribute.
 
 As with the :c:member:`~PyTypeObject.tp_methods` table, a sentinel entry with a :attr:`name` value
-of *NULL* is required.
+of ``NULL`` is required.
 
 .. XXX Descriptors need to be explained in more detail somewhere, but not here.
 
@@ -350,9 +350,9 @@ Here is an example::
 
 The :c:member:`~PyTypeObject.tp_setattr` handler is called when the :meth:`__setattr__` or
 :meth:`__delattr__` method of a class instance would be called.  When an
-attribute should be deleted, the third parameter will be *NULL*.  Here is an
+attribute should be deleted, the third parameter will be ``NULL``.  Here is an
 example that simply raises an exception; if this were really all you wanted, the
-:c:member:`~PyTypeObject.tp_setattr` handler should be set to *NULL*. ::
+:c:member:`~PyTypeObject.tp_setattr` handler should be set to ``NULL``. ::
 
    static int
    newdatatype_setattr(newdatatypeobject *obj, char *name, PyObject *v)
@@ -378,7 +378,7 @@ where the operator is one of ``Py_EQ``, ``Py_NE``, ``Py_LE``, ``Py_GT``,
 ``Py_LT`` or ``Py_GT``.  It should compare the two objects with respect to the
 specified operator and return ``Py_True`` or ``Py_False`` if the comparison is
 successful, ``Py_NotImplemented`` to indicate that comparison is not
-implemented and the other object's comparison method should be tried, or *NULL*
+implemented and the other object's comparison method should be tried, or ``NULL``
 if an exception was set.
 
 Here is a sample implementation, for a datatype that is considered equal if the
@@ -425,7 +425,7 @@ from the type implementation, the older protocols have been defined as optional
 blocks of handlers referenced by the type object.  For newer protocols there are
 additional slots in the main type object, with a flag bit being set to indicate
 that the slots are present and should be checked by the interpreter.  (The flag
-bit does not indicate that the slot values are non-*NULL*. The flag may be set
+bit does not indicate that the slot values are non-``NULL``. The flag may be set
 to indicate the presence of a slot, but a slot may still be unfilled.) ::
 
    PyNumberMethods   *tp_as_number;
@@ -476,9 +476,9 @@ This function takes three arguments:
    :c:func:`PyArg_ParseTuple` to extract the arguments.
 
 #. *kwds* is a dictionary of keyword arguments that were passed. If this is
-   non-*NULL* and you support keyword arguments, use
+   non-``NULL`` and you support keyword arguments, use
    :c:func:`PyArg_ParseTupleAndKeywords` to extract the arguments.  If you
-   do not want to support keyword arguments and this is non-*NULL*, raise a
+   do not want to support keyword arguments and this is non-``NULL``, raise a
    :exc:`TypeError` with a message saying that keyword arguments are not supported.
 
 Here is a toy ``tp_call`` implementation::
@@ -510,7 +510,7 @@ Here is a toy ``tp_call`` implementation::
 These functions provide support for the iterator protocol.  Both handlers
 take exactly one parameter, the instance for which they are being called,
 and return a new reference.  In the case of an error, they should set an
-exception and return *NULL*.  :c:member:`~PyTypeObject.tp_iter` corresponds
+exception and return ``NULL``.  :c:member:`~PyTypeObject.tp_iter` corresponds
 to the Python :meth:`__iter__` method, while :c:member:`~PyTypeObject.tp_iternext`
 corresponds to the Python :meth:`~iterator.__next__` method.
 
@@ -532,11 +532,11 @@ and :c:member:`~PyTypeObject.tp_iternext`.  An iterator's
 to the iterator.  Its :c:member:`~PyTypeObject.tp_iternext` handler should
 return a new reference to the next object in the iteration, if there is one.
 If the iteration has reached the end, :c:member:`~PyTypeObject.tp_iternext`
-may return *NULL* without setting an exception, or it may set
-:exc:`StopIteration` *in addition* to returning *NULL*; avoiding
+may return ``NULL`` without setting an exception, or it may set
+:exc:`StopIteration` *in addition* to returning ``NULL``; avoiding
 the exception can yield slightly better performance.  If an actual error
 occurs, :c:member:`~PyTypeObject.tp_iternext` should always set an exception
-and return *NULL*.
+and return ``NULL``.
 
 
 .. _weakref-support:
@@ -555,7 +555,7 @@ For an object to be weakly referencable, the extension type must do two things:
 
 #. Include a :c:type:`PyObject\*` field in the C object structure dedicated to
    the weak reference mechanism.  The object's constructor should leave it
-   *NULL* (which is automatic when using the default
+   ``NULL`` (which is automatic when using the default
    :c:member:`~PyTypeObject.tp_alloc`).
 
 #. Set the :c:member:`~PyTypeObject.tp_weaklistoffset` type member
@@ -580,7 +580,7 @@ And the corresponding member in the statically-declared type object::
 
 The only further addition is that ``tp_dealloc`` needs to clear any weak
 references (by calling :c:func:`PyObject_ClearWeakRefs`) if the field is
-non-*NULL*::
+non-``NULL``::
 
    static void
    Trivial_dealloc(TrivialObject *self)
diff --git a/Doc/extending/newtypes_tutorial.rst b/Doc/extending/newtypes_tutorial.rst
index 50be28fbcba28..b314e9e2e1549 100644
--- a/Doc/extending/newtypes_tutorial.rst
+++ b/Doc/extending/newtypes_tutorial.rst
@@ -177,7 +177,7 @@ Everything else in the file should be familiar, except for some code in
 
 This initializes the :class:`Custom` type, filling in a number of members
 to the appropriate default values, including :attr:`ob_type` that we initially
-set to *NULL*. ::
+set to ``NULL``. ::
 
    Py_INCREF(&CustomType);
    if (PyModule_AddObject(m, "Custom", (PyObject *) &CustomType) < 0) {
@@ -275,7 +275,7 @@ which is assigned to the :c:member:`~PyTypeObject.tp_dealloc` member::
 
 This method first clears the reference counts of the two Python attributes.
 :c:func:`Py_XDECREF` correctly handles the case where its argument is
-*NULL* (which might happen here if ``tp_new`` failed midway).  It then
+``NULL`` (which might happen here if ``tp_new`` failed midway).  It then
 calls the :c:member:`~PyTypeObject.tp_free` member of the object's type
 (computed by ``Py_TYPE(self)``) to free the object's memory.  Note that
 the object's type might not be :class:`CustomType`, because the object may
@@ -321,7 +321,7 @@ objects of the type.  It is exposed in Python as the :meth:`__new__` method.
 It is not required to define a ``tp_new`` member, and indeed many extension
 types will simply reuse :c:func:`PyType_GenericNew` as done in the first
 version of the ``Custom`` type above.  In this case, we use the ``tp_new``
-handler to initialize the ``first`` and ``last`` attributes to non-*NULL*
+handler to initialize the ``first`` and ``last`` attributes to non-``NULL``
 default values.
 
 ``tp_new`` is passed the type being instantiated (not necessarily ``CustomType``,
@@ -341,7 +341,7 @@ slot to allocate memory::
    self = (CustomObject *) type->tp_alloc(type, 0);
 
 Since memory allocation may fail, we must check the :c:member:`~PyTypeObject.tp_alloc`
-result against *NULL* before proceeding.
+result against ``NULL`` before proceeding.
 
 .. note::
    We didn't fill the :c:member:`~PyTypeObject.tp_alloc` slot ourselves. Rather
@@ -455,9 +455,9 @@ below for details.
 A disadvantage of this approach is that it doesn't provide a way to restrict the
 types of objects that can be assigned to the Python attributes.  We expect the
 first and last names to be strings, but any Python objects can be assigned.
-Further, the attributes can be deleted, setting the C pointers to *NULL*.  Even
-though we can make sure the members are initialized to non-*NULL* values, the
-members can be set to *NULL* if the attributes are deleted.
+Further, the attributes can be deleted, setting the C pointers to ``NULL``.  Even
+though we can make sure the members are initialized to non-``NULL`` values, the
+members can be set to ``NULL`` if the attributes are deleted.
 
 We define a single method, :meth:`Custom.name()`, that outputs the objects name as the
 concatenation of the first and last names. ::
@@ -489,8 +489,8 @@ equivalent to the Python method:
        return "%s %s" % (self.first, self.last)
 
 Note that we have to check for the possibility that our :attr:`first` and
-:attr:`last` members are *NULL*.  This is because they can be deleted, in which
-case they are set to *NULL*.  It would be better to prevent deletion of these
+:attr:`last` members are ``NULL``.  This is because they can be deleted, in which
+case they are set to ``NULL``.  It would be better to prevent deletion of these
 attributes and to restrict the attribute values to be strings.  We'll see how to
 do that in the next section.
 
@@ -584,7 +584,7 @@ could, for example, be used to allow a single set of getter and setter functions
 that decide the attribute to get or set based on data in the closure.)
 
 The setter function is passed the :class:`Custom` object, the new value, and the
-closure.  The new value may be *NULL*, in which case the attribute is being
+closure.  The new value may be ``NULL``, in which case the attribute is being
 deleted.  In our setter, we raise an error if the attribute is deleted or if its
 new value is not a string.
 
@@ -603,7 +603,7 @@ and register it in the :c:member:`~PyTypeObject.tp_getset` slot::
    .tp_getset = Custom_getsetters,
 
 The last item in a :c:type:`PyGetSetDef` structure is the "closure" mentioned
-above.  In this case, we aren't using a closure, so we just pass *NULL*.
+above.  In this case, we aren't using a closure, so we just pass ``NULL``.
 
 We also remove the member definitions for these attributes::
 
@@ -643,7 +643,7 @@ allow strings [#]_ to be passed::
    }
 
 With these changes, we can assure that the ``first`` and ``last`` members are
-never *NULL* so we can remove checks for *NULL* values in almost all cases.
+never ``NULL`` so we can remove checks for ``NULL`` values in almost all cases.
 This means that most of the :c:func:`Py_XDECREF` calls can be converted to
 :c:func:`Py_DECREF` calls.  The only place we can't change these calls is in
 the ``tp_dealloc`` implementation, where there is the possibility that the
@@ -749,7 +749,7 @@ participate in cycles::
 Notice the use of the :c:func:`Py_CLEAR` macro.  It is the recommended and safe
 way to clear data attributes of arbitrary types while decrementing
 their reference counts.  If you were to call :c:func:`Py_XDECREF` instead
-on the attribute before setting it to *NULL*, there is a possibility
+on the attribute before setting it to ``NULL``, there is a possibility
 that the attribute's destructor would call back into code that reads the
 attribute again (*especially* if there is a reference cycle).
 
diff --git a/Doc/faq/extending.rst b/Doc/faq/extending.rst
index 2ad2765870525..aecb56eaa4fd2 100644
--- a/Doc/faq/extending.rst
+++ b/Doc/faq/extending.rst
@@ -89,7 +89,7 @@ For bytes, :c:func:`PyBytes_Size` returns its length and
 length.  Note that Python bytes objects may contain null bytes so C's
 :c:func:`strlen` should not be used.
 
-To test the type of an object, first make sure it isn't *NULL*, and then use
+To test the type of an object, first make sure it isn't ``NULL``, and then use
 :c:func:`PyBytes_Check`, :c:func:`PyTuple_Check`, :c:func:`PyList_Check`, etc.
 
 There is also a high-level API to Python objects which is provided by the
diff --git a/Doc/library/ctypes.rst b/Doc/library/ctypes.rst
index 2eafa654e2e64..2be004b79168b 100644
--- a/Doc/library/ctypes.rst
+++ b/Doc/library/ctypes.rst
@@ -1083,7 +1083,7 @@ An extended example which also demonstrates the use of pointers accesses the
 Quoting the docs for that value:
 
    This pointer is initialized to point to an array of :c:type:`struct _frozen`
-   records, terminated by one whose members are all *NULL* or zero.  When a frozen
+   records, terminated by one whose members are all ``NULL`` or zero.  When a frozen
    module is imported, it is searched in this table.  Third-party code could play
    tricks with this to provide a dynamically created collection of frozen modules.
 
diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst
index 822ba81be8a7f..7b886e7c27a48 100644
--- a/Doc/whatsnew/3.4.rst
+++ b/Doc/whatsnew/3.4.rst
@@ -2503,7 +2503,7 @@ Changes in the C API
 
 * The result of the :c:data:`PyOS_ReadlineFunctionPointer` callback must
   now be a string allocated by :c:func:`PyMem_RawMalloc` or
-  :c:func:`PyMem_RawRealloc`, or *NULL* if an error occurred, instead of a
+  :c:func:`PyMem_RawRealloc`, or ``NULL`` if an error occurred, instead of a
   string allocated by :c:func:`PyMem_Malloc` or :c:func:`PyMem_Realloc`
   (:issue:`16742`)
 
diff --git a/Doc/whatsnew/3.7.rst b/Doc/whatsnew/3.7.rst
index f6476a30f889f..12e1d9be2a1cf 100644
--- a/Doc/whatsnew/3.7.rst
+++ b/Doc/whatsnew/3.7.rst
@@ -1720,7 +1720,7 @@ The type of results of :c:func:`PyThread_start_new_thread` and
 (Contributed by Serhiy Storchaka in :issue:`6532`.)
 
 :c:func:`PyUnicode_AsWideCharString` now raises a :exc:`ValueError` if the
-second argument is *NULL* and the :c:type:`wchar_t*` string contains null
+second argument is ``NULL`` and the :c:type:`wchar_t*` string contains null
 characters.  (Contributed by Serhiy Storchaka in :issue:`30708`.)
 
 Changes to the startup sequence and the management of dynamic memory
diff --git a/Include/abstract.h b/Include/abstract.h
index 3fe5a0064532e..d8f648e26f43b 100644
--- a/Include/abstract.h
+++ b/Include/abstract.h
@@ -147,7 +147,7 @@ extern "C" {
 /* Call a callable Python object 'callable' with arguments given by the
    tuple 'args' and keywords arguments given by the dictionary 'kwargs'.
 
-   'args' must not be *NULL*, use an empty tuple if no arguments are
+   'args' must not be NULL, use an empty tuple if no arguments are
    needed. If no named arguments are needed, 'kwargs' can be NULL.
 
    This is the equivalent of the Python expression:
@@ -274,9 +274,9 @@ PyAPI_FUNC(PyObject *) _Py_CheckFunctionResult(PyObject *callable,
 
 
 /* Call a callable Python object 'callable', with arguments given by the
-   tuple 'args'.  If no arguments are needed, then 'args' can be *NULL*.
+   tuple 'args'.  If no arguments are needed, then 'args' can be NULL.
 
-   Returns the result of the call on success, or *NULL* on failure.
+   Returns the result of the call on success, or NULL on failure.
 
    This is the equivalent of the Python expression:
    callable(*args). */


From webhook-mailer at python.org  Wed Oct 30 15:36:40 2019
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Wed, 30 Oct 2019 19:36:40 -0000
Subject: [Python-checkins] [3.8] bpo-38600: Change the mark up of NULL in
 the C API documentation. (GH-16950) (GH-16999)
Message-ID: <mailman.192.1572464204.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/f2ba17be2aec873741c2d8a55b1342afa360aec2
commit: f2ba17be2aec873741c2d8a55b1342afa360aec2
branch: 3.8
author: Serhiy Storchaka <storchaka at gmail.com>
committer: GitHub <noreply at github.com>
date: 2019-10-30T21:36:33+02:00
summary:

[3.8] bpo-38600: Change the mark up of NULL in the C API documentation. (GH-16950) (GH-16999)

Replace all *NULL* with ``NULL``.
(cherry picked from commit 25fc088607c855060ed142296dc1bd0125fad1af)

files:
M Doc/c-api/arg.rst
M Doc/c-api/buffer.rst
M Doc/c-api/bytearray.rst
M Doc/c-api/bytes.rst
M Doc/c-api/capsule.rst
M Doc/c-api/cell.rst
M Doc/c-api/codec.rst
M Doc/c-api/concrete.rst
M Doc/c-api/contextvars.rst
M Doc/c-api/conversion.rst
M Doc/c-api/coro.rst
M Doc/c-api/datetime.rst
M Doc/c-api/dict.rst
M Doc/c-api/exceptions.rst
M Doc/c-api/file.rst
M Doc/c-api/float.rst
M Doc/c-api/function.rst
M Doc/c-api/gcsupport.rst
M Doc/c-api/gen.rst
M Doc/c-api/import.rst
M Doc/c-api/init.rst
M Doc/c-api/intro.rst
M Doc/c-api/iter.rst
M Doc/c-api/list.rst
M Doc/c-api/long.rst
M Doc/c-api/mapping.rst
M Doc/c-api/marshal.rst
M Doc/c-api/memory.rst
M Doc/c-api/memoryview.rst
M Doc/c-api/method.rst
M Doc/c-api/module.rst
M Doc/c-api/number.rst
M Doc/c-api/object.rst
M Doc/c-api/refcounting.rst
M Doc/c-api/reflection.rst
M Doc/c-api/sequence.rst
M Doc/c-api/set.rst
M Doc/c-api/slice.rst
M Doc/c-api/structures.rst
M Doc/c-api/sys.rst
M Doc/c-api/tuple.rst
M Doc/c-api/type.rst
M Doc/c-api/typeobj.rst
M Doc/c-api/unicode.rst
M Doc/c-api/veryhigh.rst
M Doc/c-api/weakref.rst
M Doc/extending/embedding.rst
M Doc/extending/extending.rst
M Doc/extending/newtypes.rst
M Doc/extending/newtypes_tutorial.rst
M Doc/faq/extending.rst
M Doc/library/ctypes.rst
M Doc/whatsnew/3.4.rst
M Doc/whatsnew/3.7.rst
M Include/abstract.h

diff --git a/Doc/c-api/arg.rst b/Doc/c-api/arg.rst
index ba9ca5e0d30b9..594fef29f2a33 100644
--- a/Doc/c-api/arg.rst
+++ b/Doc/c-api/arg.rst
@@ -50,7 +50,7 @@ Unless otherwise stated, buffers are not NUL-terminated.
 
 Some formats require a read-only :term:`bytes-like object`, and set a
 pointer instead of a buffer structure.  They work by checking that
-the object's :c:member:`PyBufferProcs.bf_releasebuffer` field is *NULL*,
+the object's :c:member:`PyBufferProcs.bf_releasebuffer` field is ``NULL``,
 which disallows mutable objects such as :class:`bytearray`.
 
 .. note::
@@ -99,15 +99,15 @@ which disallows mutable objects such as :class:`bytearray`.
 
 ``z`` (:class:`str` or ``None``) [const char \*]
    Like ``s``, but the Python object may also be ``None``, in which case the C
-   pointer is set to *NULL*.
+   pointer is set to ``NULL``.
 
 ``z*`` (:class:`str`, :term:`bytes-like object` or ``None``) [Py_buffer]
    Like ``s*``, but the Python object may also be ``None``, in which case the
-   ``buf`` member of the :c:type:`Py_buffer` structure is set to *NULL*.
+   ``buf`` member of the :c:type:`Py_buffer` structure is set to ``NULL``.
 
 ``z#`` (:class:`str`, read-only :term:`bytes-like object` or ``None``) [const char \*, int]
    Like ``s#``, but the Python object may also be ``None``, in which case the C
-   pointer is set to *NULL*.
+   pointer is set to ``NULL``.
 
 ``y`` (read-only :term:`bytes-like object`) [const char \*]
    This format converts a bytes-like object to a C pointer to a character
@@ -166,7 +166,7 @@ which disallows mutable objects such as :class:`bytearray`.
 
 ``Z`` (:class:`str` or ``None``) [const Py_UNICODE \*]
    Like ``u``, but the Python object may also be ``None``, in which case the
-   :c:type:`Py_UNICODE` pointer is set to *NULL*.
+   :c:type:`Py_UNICODE` pointer is set to ``NULL``.
 
    .. deprecated-removed:: 3.3 4.0
       Part of the old-style :c:type:`Py_UNICODE` API; please migrate to using
@@ -174,7 +174,7 @@ which disallows mutable objects such as :class:`bytearray`.
 
 ``Z#`` (:class:`str` or ``None``) [const Py_UNICODE \*, int]
    Like ``u#``, but the Python object may also be ``None``, in which case the
-   :c:type:`Py_UNICODE` pointer is set to *NULL*.
+   :c:type:`Py_UNICODE` pointer is set to ``NULL``.
 
    .. deprecated-removed:: 3.3 4.0
       Part of the old-style :c:type:`Py_UNICODE` API; please migrate to using
@@ -197,7 +197,7 @@ which disallows mutable objects such as :class:`bytearray`.
 
    This format requires two arguments.  The first is only used as input, and
    must be a :c:type:`const char\*` which points to the name of an encoding as a
-   NUL-terminated string, or *NULL*, in which case ``'utf-8'`` encoding is used.
+   NUL-terminated string, or ``NULL``, in which case ``'utf-8'`` encoding is used.
    An exception is raised if the named encoding is not known to Python.  The
    second argument must be a :c:type:`char\*\*`; the value of the pointer it
    references will be set to a buffer with the contents of the argument text.
@@ -220,7 +220,7 @@ which disallows mutable objects such as :class:`bytearray`.
 
    It requires three arguments.  The first is only used as input, and must be a
    :c:type:`const char\*` which points to the name of an encoding as a
-   NUL-terminated string, or *NULL*, in which case ``'utf-8'`` encoding is used.
+   NUL-terminated string, or ``NULL``, in which case ``'utf-8'`` encoding is used.
    An exception is raised if the named encoding is not known to Python.  The
    second argument must be a :c:type:`char\*\*`; the value of the pointer it
    references will be set to a buffer with the contents of the argument text.
@@ -230,12 +230,12 @@ which disallows mutable objects such as :class:`bytearray`.
 
    There are two modes of operation:
 
-   If *\*buffer* points a *NULL* pointer, the function will allocate a buffer of
+   If *\*buffer* points a ``NULL`` pointer, the function will allocate a buffer of
    the needed size, copy the encoded data into this buffer and set *\*buffer* to
    reference the newly allocated storage.  The caller is responsible for calling
    :c:func:`PyMem_Free` to free the allocated buffer after usage.
 
-   If *\*buffer* points to a non-*NULL* pointer (an already allocated buffer),
+   If *\*buffer* points to a non-``NULL`` pointer (an already allocated buffer),
    :c:func:`PyArg_ParseTuple` will use this location as the buffer and interpret the
    initial value of *\*buffer_length* as the buffer size.  It will then copy the
    encoded data into the buffer and NUL-terminate it.  If the buffer is not large
@@ -317,7 +317,7 @@ Other objects
 ``O`` (object) [PyObject \*]
    Store a Python object (without any conversion) in a C object pointer.  The C
    program thus receives the actual object that was passed.  The object's reference
-   count is not increased.  The pointer stored is not *NULL*.
+   count is not increased.  The pointer stored is not ``NULL``.
 
 ``O!`` (object) [*typeobject*, PyObject \*]
    Store a Python object in a C object pointer.  This is similar to ``O``, but
@@ -437,7 +437,7 @@ API Functions
 
    Parse the parameters of a function that takes both positional and keyword
    parameters into local variables.  The *keywords* argument is a
-   *NULL*-terminated array of keyword parameter names.  Empty names denote
+   ``NULL``-terminated array of keyword parameter names.  Empty names denote
    :ref:`positional-only parameters <positional-only_parameter>`.
    Returns true on success; on failure, it returns false and raises the
    appropriate exception.
@@ -520,8 +520,8 @@ Building values
 
    Create a new value based on a format string similar to those accepted by the
    :c:func:`PyArg_Parse\*` family of functions and a sequence of values.  Returns
-   the value or *NULL* in the case of an error; an exception will be raised if
-   *NULL* is returned.
+   the value or ``NULL`` in the case of an error; an exception will be raised if
+   ``NULL`` is returned.
 
    :c:func:`Py_BuildValue` does not always build a tuple.  It builds a tuple only if
    its format string contains two or more format units.  If the format string is
@@ -547,20 +547,20 @@ Building values
 
    ``s`` (:class:`str` or ``None``) [const char \*]
       Convert a null-terminated C string to a Python :class:`str` object using ``'utf-8'``
-      encoding. If the C string pointer is *NULL*, ``None`` is used.
+      encoding. If the C string pointer is ``NULL``, ``None`` is used.
 
    ``s#`` (:class:`str` or ``None``) [const char \*, int]
       Convert a C string and its length to a Python :class:`str` object using ``'utf-8'``
-      encoding. If the C string pointer is *NULL*, the length is ignored and
+      encoding. If the C string pointer is ``NULL``, the length is ignored and
       ``None`` is returned.
 
    ``y`` (:class:`bytes`) [const char \*]
       This converts a C string to a Python :class:`bytes` object.  If the C
-      string pointer is *NULL*, ``None`` is returned.
+      string pointer is ``NULL``, ``None`` is returned.
 
    ``y#`` (:class:`bytes`) [const char \*, int]
       This converts a C string and its lengths to a Python object.  If the C
-      string pointer is *NULL*, ``None`` is returned.
+      string pointer is ``NULL``, ``None`` is returned.
 
    ``z`` (:class:`str` or ``None``) [const char \*]
       Same as ``s``.
@@ -570,12 +570,12 @@ Building values
 
    ``u`` (:class:`str`) [const wchar_t \*]
       Convert a null-terminated :c:type:`wchar_t` buffer of Unicode (UTF-16 or UCS-4)
-      data to a Python Unicode object.  If the Unicode buffer pointer is *NULL*,
+      data to a Python Unicode object.  If the Unicode buffer pointer is ``NULL``,
       ``None`` is returned.
 
    ``u#`` (:class:`str`) [const wchar_t \*, int]
       Convert a Unicode (UTF-16 or UCS-4) data buffer and its length to a Python
-      Unicode object.   If the Unicode buffer pointer is *NULL*, the length is ignored
+      Unicode object.   If the Unicode buffer pointer is ``NULL``, the length is ignored
       and ``None`` is returned.
 
    ``U`` (:class:`str` or ``None``) [const char \*]
@@ -636,9 +636,9 @@ Building values
 
    ``O`` (object) [PyObject \*]
       Pass a Python object untouched (except for its reference count, which is
-      incremented by one).  If the object passed in is a *NULL* pointer, it is assumed
+      incremented by one).  If the object passed in is a ``NULL`` pointer, it is assumed
       that this was caused because the call producing the argument found an error and
-      set an exception. Therefore, :c:func:`Py_BuildValue` will return *NULL* but won't
+      set an exception. Therefore, :c:func:`Py_BuildValue` will return ``NULL`` but won't
       raise an exception.  If no exception has been raised yet, :exc:`SystemError` is
       set.
 
@@ -653,7 +653,7 @@ Building values
    ``O&`` (object) [*converter*, *anything*]
       Convert *anything* to a Python object through a *converter* function.  The
       function is called with *anything* (which should be compatible with :c:type:`void
-      \*`) as its argument and should return a "new" Python object, or *NULL* if an
+      \*`) as its argument and should return a "new" Python object, or ``NULL`` if an
       error occurred.
 
    ``(items)`` (:class:`tuple`) [*matching-items*]
@@ -668,7 +668,7 @@ Building values
       respectively.
 
    If there is an error in the format string, the :exc:`SystemError` exception is
-   set and *NULL* returned.
+   set and ``NULL`` returned.
 
 .. c:function:: PyObject* Py_VaBuildValue(const char *format, va_list vargs)
 
diff --git a/Doc/c-api/buffer.rst b/Doc/c-api/buffer.rst
index 67140511ef74f..58d874734e8ac 100644
--- a/Doc/c-api/buffer.rst
+++ b/Doc/c-api/buffer.rst
@@ -102,13 +102,13 @@ a buffer, see :c:func:`PyObject_GetBuffer`.
    .. c:member:: void \*obj
 
       A new reference to the exporting object. The reference is owned by
-      the consumer and automatically decremented and set to *NULL* by
+      the consumer and automatically decremented and set to ``NULL`` by
       :c:func:`PyBuffer_Release`. The field is the equivalent of the return
       value of any standard C-API function.
 
       As a special case, for *temporary* buffers that are wrapped by
       :c:func:`PyMemoryView_FromBuffer` or :c:func:`PyBuffer_FillInfo`
-      this field is *NULL*. In general, exporting objects MUST NOT
+      this field is ``NULL``. In general, exporting objects MUST NOT
       use this scheme.
 
    .. c:member:: Py_ssize_t len
@@ -134,21 +134,21 @@ a buffer, see :c:func:`PyObject_GetBuffer`.
 
       Important exception: If a consumer requests a buffer without the
       :c:macro:`PyBUF_FORMAT` flag, :c:member:`~Py_buffer.format` will
-      be set to  *NULL*,  but :c:member:`~Py_buffer.itemsize` still has
+      be set to  ``NULL``,  but :c:member:`~Py_buffer.itemsize` still has
       the value for the original format.
 
       If :c:member:`~Py_buffer.shape` is present, the equality
       ``product(shape) * itemsize == len`` still holds and the consumer
       can use :c:member:`~Py_buffer.itemsize` to navigate the buffer.
 
-      If :c:member:`~Py_buffer.shape` is *NULL* as a result of a :c:macro:`PyBUF_SIMPLE`
+      If :c:member:`~Py_buffer.shape` is ``NULL`` as a result of a :c:macro:`PyBUF_SIMPLE`
       or a :c:macro:`PyBUF_WRITABLE` request, the consumer must disregard
       :c:member:`~Py_buffer.itemsize` and assume ``itemsize == 1``.
 
    .. c:member:: const char \*format
 
       A *NUL* terminated string in :mod:`struct` module style syntax describing
-      the contents of a single item. If this is *NULL*, ``"B"`` (unsigned bytes)
+      the contents of a single item. If this is ``NULL``, ``"B"`` (unsigned bytes)
       is assumed.
 
       This field is controlled by the :c:macro:`PyBUF_FORMAT` flag.
@@ -158,7 +158,7 @@ a buffer, see :c:func:`PyObject_GetBuffer`.
       The number of dimensions the memory represents as an n-dimensional array.
       If it is ``0``, :c:member:`~Py_buffer.buf` points to a single item representing
       a scalar. In this case, :c:member:`~Py_buffer.shape`, :c:member:`~Py_buffer.strides`
-      and :c:member:`~Py_buffer.suboffsets` MUST be *NULL*.
+      and :c:member:`~Py_buffer.suboffsets` MUST be ``NULL``.
 
       The macro :c:macro:`PyBUF_MAX_NDIM` limits the maximum number of dimensions
       to 64. Exporters MUST respect this limit, consumers of multi-dimensional
@@ -248,7 +248,7 @@ readonly, format
    .. c:macro:: PyBUF_FORMAT
 
       Controls the :c:member:`~Py_buffer.format` field. If set, this field MUST
-      be filled in correctly. Otherwise, this field MUST be *NULL*.
+      be filled in correctly. Otherwise, this field MUST be ``NULL``.
 
 
 :c:macro:`PyBUF_WRITABLE` can be \|'d to any of the flags in the next section.
@@ -349,9 +349,9 @@ The logical structure of NumPy-style arrays is defined by :c:member:`~Py_buffer.
 
 If ``ndim == 0``, the memory location pointed to by :c:member:`~Py_buffer.buf` is
 interpreted as a scalar of size :c:member:`~Py_buffer.itemsize`. In that case,
-both :c:member:`~Py_buffer.shape` and :c:member:`~Py_buffer.strides` are *NULL*.
+both :c:member:`~Py_buffer.shape` and :c:member:`~Py_buffer.strides` are ``NULL``.
 
-If :c:member:`~Py_buffer.strides` is *NULL*, the array is interpreted as
+If :c:member:`~Py_buffer.strides` is ``NULL``, the array is interpreted as
 a standard n-dimensional C-array. Otherwise, the consumer must access an
 n-dimensional array as follows:
 
@@ -438,7 +438,7 @@ Buffer-related functions
 
    Send a request to *exporter* to fill in *view* as specified by  *flags*.
    If the exporter cannot provide a buffer of the exact type, it MUST raise
-   :c:data:`PyExc_BufferError`, set :c:member:`view->obj` to *NULL* and
+   :c:data:`PyExc_BufferError`, set :c:member:`view->obj` to ``NULL`` and
    return ``-1``.
 
    On success, fill in *view*, set :c:member:`view->obj` to a new reference
@@ -516,7 +516,7 @@ Buffer-related functions
 
    On success, set :c:member:`view->obj` to a new reference to *exporter* and
    return 0. Otherwise, raise :c:data:`PyExc_BufferError`, set
-   :c:member:`view->obj` to *NULL* and return ``-1``;
+   :c:member:`view->obj` to ``NULL`` and return ``-1``;
 
    If this function is used as part of a :ref:`getbufferproc <buffer-structs>`,
    *exporter* MUST be set to the exporting object and *flags* must be passed
diff --git a/Doc/c-api/bytearray.rst b/Doc/c-api/bytearray.rst
index b4a9660a9169e..b2f409c15abb7 100644
--- a/Doc/c-api/bytearray.rst
+++ b/Doc/c-api/bytearray.rst
@@ -48,7 +48,7 @@ Direct API functions
 .. c:function:: PyObject* PyByteArray_FromStringAndSize(const char *string, Py_ssize_t len)
 
    Create a new bytearray object from *string* and its length, *len*.  On
-   failure, *NULL* is returned.
+   failure, ``NULL`` is returned.
 
 
 .. c:function:: PyObject* PyByteArray_Concat(PyObject *a, PyObject *b)
@@ -58,13 +58,13 @@ Direct API functions
 
 .. c:function:: Py_ssize_t PyByteArray_Size(PyObject *bytearray)
 
-   Return the size of *bytearray* after checking for a *NULL* pointer.
+   Return the size of *bytearray* after checking for a ``NULL`` pointer.
 
 
 .. c:function:: char* PyByteArray_AsString(PyObject *bytearray)
 
    Return the contents of *bytearray* as a char array after checking for a
-   *NULL* pointer.  The returned array always has an extra
+   ``NULL`` pointer.  The returned array always has an extra
    null byte appended.
 
 
diff --git a/Doc/c-api/bytes.rst b/Doc/c-api/bytes.rst
index 9a62fb682d689..0e33ed2c7c940 100644
--- a/Doc/c-api/bytes.rst
+++ b/Doc/c-api/bytes.rst
@@ -37,14 +37,14 @@ called with a non-bytes parameter.
 .. c:function:: PyObject* PyBytes_FromString(const char *v)
 
    Return a new bytes object with a copy of the string *v* as value on success,
-   and *NULL* on failure.  The parameter *v* must not be *NULL*; it will not be
+   and ``NULL`` on failure.  The parameter *v* must not be ``NULL``; it will not be
    checked.
 
 
 .. c:function:: PyObject* PyBytes_FromStringAndSize(const char *v, Py_ssize_t len)
 
    Return a new bytes object with a copy of the string *v* as value and length
-   *len* on success, and *NULL* on failure.  If *v* is *NULL*, the contents of
+   *len* on success, and ``NULL`` on failure.  If *v* is ``NULL``, the contents of
    the bytes object are uninitialized.
 
 
@@ -145,7 +145,7 @@ called with a non-bytes parameter.
    whether there are any other null bytes.  The data must not be
    modified in any way, unless the object was just created using
    ``PyBytes_FromStringAndSize(NULL, size)``. It must not be deallocated.  If
-   *o* is not a bytes object at all, :c:func:`PyBytes_AsString` returns *NULL*
+   *o* is not a bytes object at all, :c:func:`PyBytes_AsString` returns ``NULL``
    and raises :exc:`TypeError`.
 
 
@@ -159,7 +159,7 @@ called with a non-bytes parameter.
    Return the null-terminated contents of the object *obj*
    through the output variables *buffer* and *length*.
 
-   If *length* is *NULL*, the bytes object
+   If *length* is ``NULL``, the bytes object
    may not contain embedded null bytes;
    if it does, the function returns ``-1`` and a :exc:`ValueError` is raised.
 
@@ -181,7 +181,7 @@ called with a non-bytes parameter.
    appended to *bytes*; the caller will own the new reference.  The reference to
    the old value of *bytes* will be stolen.  If the new object cannot be
    created, the old reference to *bytes* will still be discarded and the value
-   of *\*bytes* will be set to *NULL*; the appropriate exception will be set.
+   of *\*bytes* will be set to ``NULL``; the appropriate exception will be set.
 
 
 .. c:function:: void PyBytes_ConcatAndDel(PyObject **bytes, PyObject *newpart)
@@ -201,5 +201,5 @@ called with a non-bytes parameter.
    desired.  On success, *\*bytes* holds the resized bytes object and ``0`` is
    returned; the address in *\*bytes* may differ from its input value.  If the
    reallocation fails, the original bytes object at *\*bytes* is deallocated,
-   *\*bytes* is set to *NULL*, :exc:`MemoryError` is set, and ``-1`` is
+   *\*bytes* is set to ``NULL``, :exc:`MemoryError` is set, and ``-1`` is
    returned.
diff --git a/Doc/c-api/capsule.rst b/Doc/c-api/capsule.rst
index 3c921bbde3486..78e21140b2f80 100644
--- a/Doc/c-api/capsule.rst
+++ b/Doc/c-api/capsule.rst
@@ -40,15 +40,15 @@ Refer to :ref:`using-capsules` for more information on using these objects.
 .. c:function:: PyObject* PyCapsule_New(void *pointer, const char *name, PyCapsule_Destructor destructor)
 
    Create a :c:type:`PyCapsule` encapsulating the *pointer*.  The *pointer*
-   argument may not be *NULL*.
+   argument may not be ``NULL``.
 
-   On failure, set an exception and return *NULL*.
+   On failure, set an exception and return ``NULL``.
 
-   The *name* string may either be *NULL* or a pointer to a valid C string.  If
-   non-*NULL*, this string must outlive the capsule.  (Though it is permitted to
+   The *name* string may either be ``NULL`` or a pointer to a valid C string.  If
+   non-``NULL``, this string must outlive the capsule.  (Though it is permitted to
    free it inside the *destructor*.)
 
-   If the *destructor* argument is not *NULL*, it will be called with the
+   If the *destructor* argument is not ``NULL``, it will be called with the
    capsule as its argument when it is destroyed.
 
    If this capsule will be stored as an attribute of a module, the *name* should
@@ -59,20 +59,20 @@ Refer to :ref:`using-capsules` for more information on using these objects.
 .. c:function:: void* PyCapsule_GetPointer(PyObject *capsule, const char *name)
 
    Retrieve the *pointer* stored in the capsule.  On failure, set an exception
-   and return *NULL*.
+   and return ``NULL``.
 
    The *name* parameter must compare exactly to the name stored in the capsule.
-   If the name stored in the capsule is *NULL*, the *name* passed in must also
-   be *NULL*.  Python uses the C function :c:func:`strcmp` to compare capsule
+   If the name stored in the capsule is ``NULL``, the *name* passed in must also
+   be ``NULL``.  Python uses the C function :c:func:`strcmp` to compare capsule
    names.
 
 
 .. c:function:: PyCapsule_Destructor PyCapsule_GetDestructor(PyObject *capsule)
 
    Return the current destructor stored in the capsule.  On failure, set an
-   exception and return *NULL*.
+   exception and return ``NULL``.
 
-   It is legal for a capsule to have a *NULL* destructor.  This makes a *NULL*
+   It is legal for a capsule to have a ``NULL`` destructor.  This makes a ``NULL``
    return code somewhat ambiguous; use :c:func:`PyCapsule_IsValid` or
    :c:func:`PyErr_Occurred` to disambiguate.
 
@@ -80,9 +80,9 @@ Refer to :ref:`using-capsules` for more information on using these objects.
 .. c:function:: void* PyCapsule_GetContext(PyObject *capsule)
 
    Return the current context stored in the capsule.  On failure, set an
-   exception and return *NULL*.
+   exception and return ``NULL``.
 
-   It is legal for a capsule to have a *NULL* context.  This makes a *NULL*
+   It is legal for a capsule to have a ``NULL`` context.  This makes a ``NULL``
    return code somewhat ambiguous; use :c:func:`PyCapsule_IsValid` or
    :c:func:`PyErr_Occurred` to disambiguate.
 
@@ -90,9 +90,9 @@ Refer to :ref:`using-capsules` for more information on using these objects.
 .. c:function:: const char* PyCapsule_GetName(PyObject *capsule)
 
    Return the current name stored in the capsule.  On failure, set an exception
-   and return *NULL*.
+   and return ``NULL``.
 
-   It is legal for a capsule to have a *NULL* name.  This makes a *NULL* return
+   It is legal for a capsule to have a ``NULL`` name.  This makes a ``NULL`` return
    code somewhat ambiguous; use :c:func:`PyCapsule_IsValid` or
    :c:func:`PyErr_Occurred` to disambiguate.
 
@@ -107,13 +107,13 @@ Refer to :ref:`using-capsules` for more information on using these objects.
    import the module conventionally (using :c:func:`PyImport_ImportModule`).
 
    Return the capsule's internal *pointer* on success.  On failure, set an
-   exception and return *NULL*.
+   exception and return ``NULL``.
 
 
 .. c:function:: int PyCapsule_IsValid(PyObject *capsule, const char *name)
 
    Determines whether or not *capsule* is a valid capsule.  A valid capsule is
-   non-*NULL*, passes :c:func:`PyCapsule_CheckExact`, has a non-*NULL* pointer
+   non-``NULL``, passes :c:func:`PyCapsule_CheckExact`, has a non-``NULL`` pointer
    stored in it, and its internal name matches the *name* parameter.  (See
    :c:func:`PyCapsule_GetPointer` for information on how capsule names are
    compared.)
@@ -142,9 +142,9 @@ Refer to :ref:`using-capsules` for more information on using these objects.
 
 .. c:function:: int PyCapsule_SetName(PyObject *capsule, const char *name)
 
-   Set the name inside *capsule* to *name*.  If non-*NULL*, the name must
+   Set the name inside *capsule* to *name*.  If non-``NULL``, the name must
    outlive the capsule.  If the previous *name* stored in the capsule was not
-   *NULL*, no attempt is made to free it.
+   ``NULL``, no attempt is made to free it.
 
    Return ``0`` on success.  Return nonzero and set an exception on failure.
 
@@ -152,6 +152,6 @@ Refer to :ref:`using-capsules` for more information on using these objects.
 .. c:function:: int PyCapsule_SetPointer(PyObject *capsule, void *pointer)
 
    Set the void pointer inside *capsule* to *pointer*.  The pointer may not be
-   *NULL*.
+   ``NULL``.
 
    Return ``0`` on success.  Return nonzero and set an exception on failure.
diff --git a/Doc/c-api/cell.rst b/Doc/c-api/cell.rst
index 8514afe928e7d..8408f7e398db7 100644
--- a/Doc/c-api/cell.rst
+++ b/Doc/c-api/cell.rst
@@ -27,13 +27,13 @@ Cell objects are not likely to be useful elsewhere.
 
 .. c:function:: int PyCell_Check(ob)
 
-   Return true if *ob* is a cell object; *ob* must not be *NULL*.
+   Return true if *ob* is a cell object; *ob* must not be ``NULL``.
 
 
 .. c:function:: PyObject* PyCell_New(PyObject *ob)
 
    Create and return a new cell object containing the value *ob*. The parameter may
-   be *NULL*.
+   be ``NULL``.
 
 
 .. c:function:: PyObject* PyCell_Get(PyObject *cell)
@@ -44,19 +44,19 @@ Cell objects are not likely to be useful elsewhere.
 .. c:function:: PyObject* PyCell_GET(PyObject *cell)
 
    Return the contents of the cell *cell*, but without checking that *cell* is
-   non-*NULL* and a cell object.
+   non-``NULL`` and a cell object.
 
 
 .. c:function:: int PyCell_Set(PyObject *cell, PyObject *value)
 
    Set the contents of the cell object *cell* to *value*.  This releases the
-   reference to any current content of the cell. *value* may be *NULL*.  *cell*
-   must be non-*NULL*; if it is not a cell object, ``-1`` will be returned.  On
+   reference to any current content of the cell. *value* may be ``NULL``.  *cell*
+   must be non-``NULL``; if it is not a cell object, ``-1`` will be returned.  On
    success, ``0`` will be returned.
 
 
 .. c:function:: void PyCell_SET(PyObject *cell, PyObject *value)
 
    Sets the value of the cell object *cell* to *value*.  No reference counts are
-   adjusted, and no checks are made for safety; *cell* must be non-*NULL* and must
+   adjusted, and no checks are made for safety; *cell* must be non-``NULL`` and must
    be a cell object.
diff --git a/Doc/c-api/codec.rst b/Doc/c-api/codec.rst
index c55f19970e125..172dcb326a4bc 100644
--- a/Doc/c-api/codec.rst
+++ b/Doc/c-api/codec.rst
@@ -21,7 +21,7 @@ Codec registry and support functions
 
    *object* is passed through the encoder function found for the given
    *encoding* using the error handling method defined by *errors*.  *errors* may
-   be *NULL* to use the default method defined for the codec.  Raises a
+   be ``NULL`` to use the default method defined for the codec.  Raises a
    :exc:`LookupError` if no encoder can be found.
 
 .. c:function:: PyObject* PyCodec_Decode(PyObject *object, const char *encoding, const char *errors)
@@ -30,7 +30,7 @@ Codec registry and support functions
 
    *object* is passed through the decoder function found for the given
    *encoding* using the error handling method defined by *errors*.  *errors* may
-   be *NULL* to use the default method defined for the codec.  Raises a
+   be ``NULL`` to use the default method defined for the codec.  Raises a
    :exc:`LookupError` if no encoder can be found.
 
 
@@ -40,7 +40,7 @@ Codec lookup API
 In the following functions, the *encoding* string is looked up converted to all
 lower-case characters, which makes encodings looked up through this mechanism
 effectively case-insensitive.  If no codec is found, a :exc:`KeyError` is set
-and *NULL* returned.
+and ``NULL`` returned.
 
 .. c:function:: PyObject* PyCodec_Encoder(const char *encoding)
 
@@ -92,7 +92,7 @@ Registry API for Unicode encoding error handlers
 .. c:function:: PyObject* PyCodec_LookupError(const char *name)
 
    Lookup the error handling callback function registered under *name*.  As a
-   special case *NULL* can be passed, in which case the error handling callback
+   special case ``NULL`` can be passed, in which case the error handling callback
    for "strict" will be returned.
 
 .. c:function:: PyObject* PyCodec_StrictErrors(PyObject *exc)
diff --git a/Doc/c-api/concrete.rst b/Doc/c-api/concrete.rst
index f83d711795c2c..c1d9fa1b41a3f 100644
--- a/Doc/c-api/concrete.rst
+++ b/Doc/c-api/concrete.rst
@@ -17,8 +17,8 @@ dictionary, use :c:func:`PyDict_Check`.  The chapter is structured like the
 .. warning::
 
    While the functions described in this chapter carefully check the type of the
-   objects which are passed in, many of them do not check for *NULL* being passed
-   instead of a valid object.  Allowing *NULL* to be passed in can cause memory
+   objects which are passed in, many of them do not check for ``NULL`` being passed
+   instead of a valid object.  Allowing ``NULL`` to be passed in can cause memory
    access violations and immediate termination of the interpreter.
 
 
diff --git a/Doc/c-api/contextvars.rst b/Doc/c-api/contextvars.rst
index a7cde7fb1958a..38256a3b0f2a0 100644
--- a/Doc/c-api/contextvars.rst
+++ b/Doc/c-api/contextvars.rst
@@ -60,17 +60,17 @@ Type-check macros:
 .. c:function:: int PyContext_CheckExact(PyObject *o)
 
    Return true if *o* is of type :c:data:`PyContext_Type`. *o* must not be
-   *NULL*.  This function always succeeds.
+   ``NULL``.  This function always succeeds.
 
 .. c:function:: int PyContextVar_CheckExact(PyObject *o)
 
    Return true if *o* is of type :c:data:`PyContextVar_Type`. *o* must not be
-   *NULL*.  This function always succeeds.
+   ``NULL``.  This function always succeeds.
 
 .. c:function:: int PyContextToken_CheckExact(PyObject *o)
 
    Return true if *o* is of type :c:data:`PyContextToken_Type`.
-   *o* must not be *NULL*.  This function always succeeds.
+   *o* must not be ``NULL``.  This function always succeeds.
 
 
 Context object management functions:
diff --git a/Doc/c-api/conversion.rst b/Doc/c-api/conversion.rst
index ed101c791ec74..c02fa60ae9a7c 100644
--- a/Doc/c-api/conversion.rst
+++ b/Doc/c-api/conversion.rst
@@ -113,7 +113,7 @@ The following functions provide locale-independent string to number conversions.
    *val* is a finite number, an infinite number, or not a number, respectively.
 
    The return value is a pointer to *buffer* with the converted string or
-   *NULL* if the conversion failed. The caller is responsible for freeing the
+   ``NULL`` if the conversion failed. The caller is responsible for freeing the
    returned string by calling :c:func:`PyMem_Free`.
 
    .. versionadded:: 3.1
diff --git a/Doc/c-api/coro.rst b/Doc/c-api/coro.rst
index 50428c7eb43e3..915c57eeb55b8 100644
--- a/Doc/c-api/coro.rst
+++ b/Doc/c-api/coro.rst
@@ -23,7 +23,7 @@ return.
 
 .. c:function:: int PyCoro_CheckExact(PyObject *ob)
 
-   Return true if *ob*'s type is *PyCoro_Type*; *ob* must not be *NULL*.
+   Return true if *ob*'s type is *PyCoro_Type*; *ob* must not be ``NULL``.
 
 
 .. c:function:: PyObject* PyCoro_New(PyFrameObject *frame, PyObject *name, PyObject *qualname)
@@ -31,4 +31,4 @@ return.
    Create and return a new coroutine object based on the *frame* object,
    with ``__name__`` and ``__qualname__`` set to *name* and *qualname*.
    A reference to *frame* is stolen by this function.  The *frame* argument
-   must not be *NULL*.
+   must not be ``NULL``.
diff --git a/Doc/c-api/datetime.rst b/Doc/c-api/datetime.rst
index 44a04373d1968..bd4f1ff446bcf 100644
--- a/Doc/c-api/datetime.rst
+++ b/Doc/c-api/datetime.rst
@@ -28,61 +28,61 @@ Type-check macros:
 .. c:function:: int PyDate_Check(PyObject *ob)
 
    Return true if *ob* is of type :c:data:`PyDateTime_DateType` or a subtype of
-   :c:data:`PyDateTime_DateType`.  *ob* must not be *NULL*.
+   :c:data:`PyDateTime_DateType`.  *ob* must not be ``NULL``.
 
 
 .. c:function:: int PyDate_CheckExact(PyObject *ob)
 
    Return true if *ob* is of type :c:data:`PyDateTime_DateType`. *ob* must not be
-   *NULL*.
+   ``NULL``.
 
 
 .. c:function:: int PyDateTime_Check(PyObject *ob)
 
    Return true if *ob* is of type :c:data:`PyDateTime_DateTimeType` or a subtype of
-   :c:data:`PyDateTime_DateTimeType`.  *ob* must not be *NULL*.
+   :c:data:`PyDateTime_DateTimeType`.  *ob* must not be ``NULL``.
 
 
 .. c:function:: int PyDateTime_CheckExact(PyObject *ob)
 
    Return true if *ob* is of type :c:data:`PyDateTime_DateTimeType`. *ob* must not
-   be *NULL*.
+   be ``NULL``.
 
 
 .. c:function:: int PyTime_Check(PyObject *ob)
 
    Return true if *ob* is of type :c:data:`PyDateTime_TimeType` or a subtype of
-   :c:data:`PyDateTime_TimeType`.  *ob* must not be *NULL*.
+   :c:data:`PyDateTime_TimeType`.  *ob* must not be ``NULL``.
 
 
 .. c:function:: int PyTime_CheckExact(PyObject *ob)
 
    Return true if *ob* is of type :c:data:`PyDateTime_TimeType`. *ob* must not be
-   *NULL*.
+   ``NULL``.
 
 
 .. c:function:: int PyDelta_Check(PyObject *ob)
 
    Return true if *ob* is of type :c:data:`PyDateTime_DeltaType` or a subtype of
-   :c:data:`PyDateTime_DeltaType`.  *ob* must not be *NULL*.
+   :c:data:`PyDateTime_DeltaType`.  *ob* must not be ``NULL``.
 
 
 .. c:function:: int PyDelta_CheckExact(PyObject *ob)
 
    Return true if *ob* is of type :c:data:`PyDateTime_DeltaType`. *ob* must not be
-   *NULL*.
+   ``NULL``.
 
 
 .. c:function:: int PyTZInfo_Check(PyObject *ob)
 
    Return true if *ob* is of type :c:data:`PyDateTime_TZInfoType` or a subtype of
-   :c:data:`PyDateTime_TZInfoType`.  *ob* must not be *NULL*.
+   :c:data:`PyDateTime_TZInfoType`.  *ob* must not be ``NULL``.
 
 
 .. c:function:: int PyTZInfo_CheckExact(PyObject *ob)
 
    Return true if *ob* is of type :c:data:`PyDateTime_TZInfoType`. *ob* must not be
-   *NULL*.
+   ``NULL``.
 
 
 Macros to create objects:
@@ -144,7 +144,7 @@ Macros to create objects:
 
 Macros to extract fields from date objects.  The argument must be an instance of
 :c:data:`PyDateTime_Date`, including subclasses (such as
-:c:data:`PyDateTime_DateTime`).  The argument must not be *NULL*, and the type is
+:c:data:`PyDateTime_DateTime`).  The argument must not be ``NULL``, and the type is
 not checked:
 
 .. c:function:: int PyDateTime_GET_YEAR(PyDateTime_Date *o)
@@ -164,7 +164,7 @@ not checked:
 
 Macros to extract fields from datetime objects.  The argument must be an
 instance of :c:data:`PyDateTime_DateTime`, including subclasses. The argument
-must not be *NULL*, and the type is not checked:
+must not be ``NULL``, and the type is not checked:
 
 .. c:function:: int PyDateTime_DATE_GET_HOUR(PyDateTime_DateTime *o)
 
@@ -187,7 +187,7 @@ must not be *NULL*, and the type is not checked:
 
 
 Macros to extract fields from time objects.  The argument must be an instance of
-:c:data:`PyDateTime_Time`, including subclasses. The argument must not be *NULL*,
+:c:data:`PyDateTime_Time`, including subclasses. The argument must not be ``NULL``,
 and the type is not checked:
 
 .. c:function:: int PyDateTime_TIME_GET_HOUR(PyDateTime_Time *o)
@@ -212,7 +212,7 @@ and the type is not checked:
 
 Macros to extract fields from time delta objects.  The argument must be an
 instance of :c:data:`PyDateTime_Delta`, including subclasses. The argument must
-not be *NULL*, and the type is not checked:
+not be ``NULL``, and the type is not checked:
 
 .. c:function:: int PyDateTime_DELTA_GET_DAYS(PyDateTime_Delta *o)
 
diff --git a/Doc/c-api/dict.rst b/Doc/c-api/dict.rst
index e970771893eb3..e7922dc0c73f2 100644
--- a/Doc/c-api/dict.rst
+++ b/Doc/c-api/dict.rst
@@ -33,7 +33,7 @@ Dictionary Objects
 
 .. c:function:: PyObject* PyDict_New()
 
-   Return a new empty dictionary, or *NULL* on failure.
+   Return a new empty dictionary, or ``NULL`` on failure.
 
 
 .. c:function:: PyObject* PyDictProxy_New(PyObject *mapping)
@@ -92,7 +92,7 @@ Dictionary Objects
 
 .. c:function:: PyObject* PyDict_GetItem(PyObject *p, PyObject *key)
 
-   Return the object from dictionary *p* which has a key *key*.  Return *NULL*
+   Return the object from dictionary *p* which has a key *key*.  Return ``NULL``
    if the key *key* is not present, but *without* setting an exception.
 
    Note that exceptions which occur while calling :meth:`__hash__` and
@@ -103,8 +103,8 @@ Dictionary Objects
 .. c:function:: PyObject* PyDict_GetItemWithError(PyObject *p, PyObject *key)
 
    Variant of :c:func:`PyDict_GetItem` that does not suppress
-   exceptions. Return *NULL* **with** an exception set if an exception
-   occurred.  Return *NULL* **without** an exception set if the key
+   exceptions. Return ``NULL`` **with** an exception set if an exception
+   occurred.  Return ``NULL`` **without** an exception set if the key
    wasn't present.
 
 
@@ -161,7 +161,7 @@ Dictionary Objects
    function returns true for each pair in the dictionary, and false once all
    pairs have been reported.  The parameters *pkey* and *pvalue* should either
    point to :c:type:`PyObject\*` variables that will be filled in with each key
-   and value, respectively, or may be *NULL*.  Any references returned through
+   and value, respectively, or may be ``NULL``.  Any references returned through
    them are borrowed.  *ppos* should not be altered during iteration. Its
    value represents offsets within the internal dictionary structure, and
    since the structure is sparse, the offsets are not consecutive.
diff --git a/Doc/c-api/exceptions.rst b/Doc/c-api/exceptions.rst
index 25bb657dca0e1..4cb309521d4d2 100644
--- a/Doc/c-api/exceptions.rst
+++ b/Doc/c-api/exceptions.rst
@@ -13,7 +13,7 @@ exception handling.  It works somewhat like the POSIX :c:data:`errno` variable:
 there is a global indicator (per thread) of the last error that occurred.  Most
 C API functions don't clear this on success, but will set it to indicate the
 cause of the error on failure.  Most C API functions also return an error
-indicator, usually *NULL* if they are supposed to return a pointer, or ``-1``
+indicator, usually ``NULL`` if they are supposed to return a pointer, or ``-1``
 if they return an integer (exception: the :c:func:`PyArg_\*` functions
 return ``1`` for success and ``0`` for failure).
 
@@ -111,7 +111,7 @@ NULL pointer for use in a ``return`` statement.
 
 .. c:function:: PyObject* PyErr_Format(PyObject *exception, const char *format, ...)
 
-   This function sets the error indicator and returns *NULL*.  *exception*
+   This function sets the error indicator and returns ``NULL``.  *exception*
    should be a Python exception class.  The *format* and subsequent
    parameters help format the error message; they have the same meaning and
    values as in :c:func:`PyUnicode_FromFormat`. *format* is an ASCII-encoded
@@ -140,7 +140,7 @@ NULL pointer for use in a ``return`` statement.
 
 .. c:function:: PyObject* PyErr_NoMemory()
 
-   This is a shorthand for ``PyErr_SetNone(PyExc_MemoryError)``; it returns *NULL*
+   This is a shorthand for ``PyErr_SetNone(PyExc_MemoryError)``; it returns ``NULL``
    so an object allocation function can write ``return PyErr_NoMemory();`` when it
    runs out of memory.
 
@@ -156,7 +156,7 @@ NULL pointer for use in a ``return`` statement.
    and then calls ``PyErr_SetObject(type, object)``.  On Unix, when the
    :c:data:`errno` value is :const:`EINTR`, indicating an interrupted system call,
    this calls :c:func:`PyErr_CheckSignals`, and if that set the error indicator,
-   leaves it set to that.  The function always returns *NULL*, so a wrapper
+   leaves it set to that.  The function always returns ``NULL``, so a wrapper
    function around a system call can write ``return PyErr_SetFromErrno(type);``
    when the system call returns an error.
 
@@ -164,7 +164,7 @@ NULL pointer for use in a ``return`` statement.
 .. c:function:: PyObject* PyErr_SetFromErrnoWithFilenameObject(PyObject *type, PyObject *filenameObject)
 
    Similar to :c:func:`PyErr_SetFromErrno`, with the additional behavior that if
-   *filenameObject* is not *NULL*, it is passed to the constructor of *type* as
+   *filenameObject* is not ``NULL``, it is passed to the constructor of *type* as
    a third parameter.  In the case of :exc:`OSError` exception,
    this is used to define the :attr:`filename` attribute of the
    exception instance.
@@ -195,7 +195,7 @@ NULL pointer for use in a ``return`` statement.
    then it constructs a tuple object whose first item is the *ierr* value and whose
    second item is the corresponding error message (gotten from
    :c:func:`FormatMessage`), and then calls ``PyErr_SetObject(PyExc_WindowsError,
-   object)``. This function always returns *NULL*.
+   object)``. This function always returns ``NULL``.
 
    .. availability:: Windows.
 
@@ -304,7 +304,7 @@ an error value).
 .. c:function:: int PyErr_WarnEx(PyObject *category, const char *message, Py_ssize_t stack_level)
 
    Issue a warning message.  The *category* argument is a warning category (see
-   below) or *NULL*; the *message* argument is a UTF-8 encoded string.  *stack_level* is a
+   below) or ``NULL``; the *message* argument is a UTF-8 encoded string.  *stack_level* is a
    positive number giving a number of stack frames; the warning will be issued from
    the  currently executing line of code in that stack frame.  A *stack_level* of 1
    is the function calling :c:func:`PyErr_WarnEx`, 2 is  the function above that,
@@ -333,7 +333,7 @@ an error value).
    Issue a warning message with explicit control over all warning attributes.  This
    is a straightforward wrapper around the Python function
    :func:`warnings.warn_explicit`, see there for more information.  The *module*
-   and *registry* arguments may be set to *NULL* to get the default effect
+   and *registry* arguments may be set to ``NULL`` to get the default effect
    described there.
 
    .. versionadded:: 3.4
@@ -370,7 +370,7 @@ Querying the error indicator
 
    Test whether the error indicator is set.  If set, return the exception *type*
    (the first argument to the last call to one of the :c:func:`PyErr_Set\*`
-   functions or to :c:func:`PyErr_Restore`).  If not set, return *NULL*.  You do not
+   functions or to :c:func:`PyErr_Restore`).  If not set, return ``NULL``.  You do not
    own a reference to the return value, so you do not need to :c:func:`Py_DECREF`
    it.
 
@@ -400,9 +400,9 @@ Querying the error indicator
 .. c:function:: void PyErr_Fetch(PyObject **ptype, PyObject **pvalue, PyObject **ptraceback)
 
    Retrieve the error indicator into three variables whose addresses are passed.
-   If the error indicator is not set, set all three variables to *NULL*.  If it is
+   If the error indicator is not set, set all three variables to ``NULL``.  If it is
    set, it will be cleared and you own a reference to each object retrieved.  The
-   value and traceback object may be *NULL* even when the type object is not.
+   value and traceback object may be ``NULL`` even when the type object is not.
 
    .. note::
 
@@ -422,8 +422,8 @@ Querying the error indicator
 .. c:function:: void PyErr_Restore(PyObject *type, PyObject *value, PyObject *traceback)
 
    Set  the error indicator from the three objects.  If the error indicator is
-   already set, it is cleared first.  If the objects are *NULL*, the error
-   indicator is cleared.  Do not pass a *NULL* type and non-*NULL* value or
+   already set, it is cleared first.  If the objects are ``NULL``, the error
+   indicator is cleared.  Do not pass a ``NULL`` type and non-``NULL`` value or
    traceback.  The exception type should be a class.  Do not pass an invalid
    exception type or value. (Violating these rules will cause subtle problems
    later.)  This call takes away a reference to each object: you must own a
@@ -462,7 +462,7 @@ Querying the error indicator
    Retrieve the exception info, as known from ``sys.exc_info()``.  This refers
    to an exception that was *already caught*, not to an exception that was
    freshly raised.  Returns new references for the three objects, any of which
-   may be *NULL*.  Does not modify the exception info state.
+   may be ``NULL``.  Does not modify the exception info state.
 
    .. note::
 
@@ -479,7 +479,7 @@ Querying the error indicator
    Set the exception info, as known from ``sys.exc_info()``.  This refers
    to an exception that was *already caught*, not to an exception that was
    freshly raised.  This function steals the references of the arguments.
-   To clear the exception state, pass *NULL* for all three arguments.
+   To clear the exception state, pass ``NULL`` for all three arguments.
    For general rules about the three arguments, see :c:func:`PyErr_Restore`.
 
    .. note::
@@ -549,7 +549,7 @@ Exception Classes
 
    This utility function creates and returns a new exception class. The *name*
    argument must be the name of the new exception, a C string of the form
-   ``module.classname``.  The *base* and *dict* arguments are normally *NULL*.
+   ``module.classname``.  The *base* and *dict* arguments are normally ``NULL``.
    This creates a class object derived from :exc:`Exception` (accessible in C as
    :c:data:`PyExc_Exception`).
 
@@ -563,7 +563,7 @@ Exception Classes
 .. c:function:: PyObject* PyErr_NewExceptionWithDoc(const char *name, const char *doc, PyObject *base, PyObject *dict)
 
    Same as :c:func:`PyErr_NewException`, except that the new exception class can
-   easily be given a docstring: If *doc* is non-*NULL*, it will be used as the
+   easily be given a docstring: If *doc* is non-``NULL``, it will be used as the
    docstring for the exception class.
 
    .. versionadded:: 3.2
@@ -576,7 +576,7 @@ Exception Objects
 
    Return the traceback associated with the exception as a new reference, as
    accessible from Python through :attr:`__traceback__`.  If there is no
-   traceback associated, this returns *NULL*.
+   traceback associated, this returns ``NULL``.
 
 
 .. c:function:: int PyException_SetTraceback(PyObject *ex, PyObject *tb)
@@ -590,12 +590,12 @@ Exception Objects
    Return the context (another exception instance during whose handling *ex* was
    raised) associated with the exception as a new reference, as accessible from
    Python through :attr:`__context__`.  If there is no context associated, this
-   returns *NULL*.
+   returns ``NULL``.
 
 
 .. c:function:: void PyException_SetContext(PyObject *ex, PyObject *ctx)
 
-   Set the context associated with the exception to *ctx*.  Use *NULL* to clear
+   Set the context associated with the exception to *ctx*.  Use ``NULL`` to clear
    it.  There is no type check to make sure that *ctx* is an exception instance.
    This steals a reference to *ctx*.
 
@@ -609,7 +609,7 @@ Exception Objects
 
 .. c:function:: void PyException_SetCause(PyObject *ex, PyObject *cause)
 
-   Set the cause associated with the exception to *cause*.  Use *NULL* to clear
+   Set the cause associated with the exception to *cause*.  Use ``NULL`` to clear
    it.  There is no type check to make sure that *cause* is either an exception
    instance or :const:`None`.  This steals a reference to *cause*.
 
@@ -656,7 +656,7 @@ The following functions are used to create and modify Unicode exceptions from C.
                 int PyUnicodeTranslateError_GetStart(PyObject *exc, Py_ssize_t *start)
 
    Get the *start* attribute of the given exception object and place it into
-   *\*start*.  *start* must not be *NULL*.  Return ``0`` on success, ``-1`` on
+   *\*start*.  *start* must not be ``NULL``.  Return ``0`` on success, ``-1`` on
    failure.
 
 .. c:function:: int PyUnicodeDecodeError_SetStart(PyObject *exc, Py_ssize_t start)
@@ -671,7 +671,7 @@ The following functions are used to create and modify Unicode exceptions from C.
                 int PyUnicodeTranslateError_GetEnd(PyObject *exc, Py_ssize_t *end)
 
    Get the *end* attribute of the given exception object and place it into
-   *\*end*.  *end* must not be *NULL*.  Return ``0`` on success, ``-1`` on
+   *\*end*.  *end* must not be ``NULL``.  Return ``0`` on success, ``-1`` on
    failure.
 
 .. c:function:: int PyUnicodeDecodeError_SetEnd(PyObject *exc, Py_ssize_t end)
diff --git a/Doc/c-api/file.rst b/Doc/c-api/file.rst
index 543dc60b9f55a..5370c4e350a0b 100644
--- a/Doc/c-api/file.rst
+++ b/Doc/c-api/file.rst
@@ -21,9 +21,9 @@ the :mod:`io` APIs instead.
 
    Create a Python file object from the file descriptor of an already
    opened file *fd*.  The arguments *name*, *encoding*, *errors* and *newline*
-   can be *NULL* to use the defaults; *buffering* can be *-1* to use the
+   can be ``NULL`` to use the defaults; *buffering* can be *-1* to use the
    default. *name* is ignored and kept for backward compatibility. Return
-   *NULL* on failure. For a more comprehensive description of the arguments,
+   ``NULL`` on failure. For a more comprehensive description of the arguments,
    please refer to the :func:`io.open` function documentation.
 
    .. warning::
diff --git a/Doc/c-api/float.rst b/Doc/c-api/float.rst
index 057ff522516a3..bfc28a79ecfdc 100644
--- a/Doc/c-api/float.rst
+++ b/Doc/c-api/float.rst
@@ -34,12 +34,12 @@ Floating Point Objects
 .. c:function:: PyObject* PyFloat_FromString(PyObject *str)
 
    Create a :c:type:`PyFloatObject` object based on the string value in *str*, or
-   *NULL* on failure.
+   ``NULL`` on failure.
 
 
 .. c:function:: PyObject* PyFloat_FromDouble(double v)
 
-   Create a :c:type:`PyFloatObject` object from *v*, or *NULL* on failure.
+   Create a :c:type:`PyFloatObject` object from *v*, or ``NULL`` on failure.
 
 
 .. c:function:: double PyFloat_AsDouble(PyObject *pyfloat)
diff --git a/Doc/c-api/function.rst b/Doc/c-api/function.rst
index 02c4ebdbed454..a3cbf569f76f2 100644
--- a/Doc/c-api/function.rst
+++ b/Doc/c-api/function.rst
@@ -26,7 +26,7 @@ There are a few functions specific to Python functions.
 .. c:function:: int PyFunction_Check(PyObject *o)
 
    Return true if *o* is a function object (has type :c:data:`PyFunction_Type`).
-   The parameter must not be *NULL*.
+   The parameter must not be ``NULL``.
 
 
 .. c:function:: PyObject* PyFunction_New(PyObject *code, PyObject *globals)
@@ -36,7 +36,7 @@ There are a few functions specific to Python functions.
 
    The function's docstring and name are retrieved from the code object. *__module__*
    is retrieved from *globals*. The argument defaults, annotations and closure are
-   set to *NULL*. *__qualname__* is set to the same value as the function's name.
+   set to ``NULL``. *__qualname__* is set to the same value as the function's name.
 
 
 .. c:function:: PyObject* PyFunction_NewWithQualName(PyObject *code, PyObject *globals, PyObject *qualname)
@@ -69,7 +69,7 @@ There are a few functions specific to Python functions.
 .. c:function:: PyObject* PyFunction_GetDefaults(PyObject *op)
 
    Return the argument default values of the function object *op*. This can be a
-   tuple of arguments or *NULL*.
+   tuple of arguments or ``NULL``.
 
 
 .. c:function:: int PyFunction_SetDefaults(PyObject *op, PyObject *defaults)
@@ -82,7 +82,7 @@ There are a few functions specific to Python functions.
 
 .. c:function:: PyObject* PyFunction_GetClosure(PyObject *op)
 
-   Return the closure associated with the function object *op*. This can be *NULL*
+   Return the closure associated with the function object *op*. This can be ``NULL``
    or a tuple of cell objects.
 
 
@@ -97,7 +97,7 @@ There are a few functions specific to Python functions.
 .. c:function:: PyObject *PyFunction_GetAnnotations(PyObject *op)
 
    Return the annotations of the function object *op*. This can be a
-   mutable dictionary or *NULL*.
+   mutable dictionary or ``NULL``.
 
 
 .. c:function:: int PyFunction_SetAnnotations(PyObject *op, PyObject *annotations)
diff --git a/Doc/c-api/gcsupport.rst b/Doc/c-api/gcsupport.rst
index b3f30b2ed9e99..924a7fd2fda4d 100644
--- a/Doc/c-api/gcsupport.rst
+++ b/Doc/c-api/gcsupport.rst
@@ -49,7 +49,7 @@ Constructors for container types must conform to two rules:
 .. c:function:: TYPE* PyObject_GC_Resize(TYPE, PyVarObject *op, Py_ssize_t newsize)
 
    Resize an object allocated by :c:func:`PyObject_NewVar`.  Returns the
-   resized object or *NULL* on failure.  *op* must not be tracked by the collector yet.
+   resized object or ``NULL`` on failure.  *op* must not be tracked by the collector yet.
 
 
 .. c:function:: void PyObject_GC_Track(PyObject *op)
@@ -110,7 +110,7 @@ The :c:member:`~PyTypeObject.tp_traverse` handler must have the following type:
    Traversal function for a container object.  Implementations must call the
    *visit* function for each object directly contained by *self*, with the
    parameters to *visit* being the contained object and the *arg* value passed
-   to the handler.  The *visit* function must not be called with a *NULL*
+   to the handler.  The *visit* function must not be called with a ``NULL``
    object argument.  If *visit* returns a non-zero value that value should be
    returned immediately.
 
@@ -121,7 +121,7 @@ must name its arguments exactly *visit* and *arg*:
 
 .. c:function:: void Py_VISIT(PyObject *o)
 
-   If *o* is not *NULL*, call the *visit* callback, with arguments *o*
+   If *o* is not ``NULL``, call the *visit* callback, with arguments *o*
    and *arg*.  If *visit* returns a non-zero value, then return it.
    Using this macro, :c:member:`~PyTypeObject.tp_traverse` handlers
    look like::
@@ -134,7 +134,7 @@ must name its arguments exactly *visit* and *arg*:
           return 0;
       }
 
-The :c:member:`~PyTypeObject.tp_clear` handler must be of the :c:type:`inquiry` type, or *NULL*
+The :c:member:`~PyTypeObject.tp_clear` handler must be of the :c:type:`inquiry` type, or ``NULL``
 if the object is immutable.
 
 
diff --git a/Doc/c-api/gen.rst b/Doc/c-api/gen.rst
index 8d54021c18145..e4b8d255dadba 100644
--- a/Doc/c-api/gen.rst
+++ b/Doc/c-api/gen.rst
@@ -22,23 +22,23 @@ than explicitly calling :c:func:`PyGen_New` or :c:func:`PyGen_NewWithQualName`.
 
 .. c:function:: int PyGen_Check(PyObject *ob)
 
-   Return true if *ob* is a generator object; *ob* must not be *NULL*.
+   Return true if *ob* is a generator object; *ob* must not be ``NULL``.
 
 
 .. c:function:: int PyGen_CheckExact(PyObject *ob)
 
-   Return true if *ob*'s type is *PyGen_Type*; *ob* must not be *NULL*.
+   Return true if *ob*'s type is *PyGen_Type*; *ob* must not be ``NULL``.
 
 
 .. c:function:: PyObject* PyGen_New(PyFrameObject *frame)
 
    Create and return a new generator object based on the *frame* object.
    A reference to *frame* is stolen by this function. The argument must not be
-   *NULL*.
+   ``NULL``.
 
 .. c:function:: PyObject* PyGen_NewWithQualName(PyFrameObject *frame, PyObject *name, PyObject *qualname)
 
    Create and return a new generator object based on the *frame* object,
    with ``__name__`` and ``__qualname__`` set to *name* and *qualname*.
    A reference to *frame* is stolen by this function.  The *frame* argument
-   must not be *NULL*.
+   must not be ``NULL``.
diff --git a/Doc/c-api/import.rst b/Doc/c-api/import.rst
index 86cc4031610b7..862ad5151eb0f 100644
--- a/Doc/c-api/import.rst
+++ b/Doc/c-api/import.rst
@@ -14,7 +14,7 @@ Importing Modules
       single: modules (in module sys)
 
    This is a simplified interface to :c:func:`PyImport_ImportModuleEx` below,
-   leaving the *globals* and *locals* arguments set to *NULL* and *level* set
+   leaving the *globals* and *locals* arguments set to ``NULL`` and *level* set
    to 0.  When the *name*
    argument contains a dot (when it specifies a submodule of a package), the
    *fromlist* argument is set to the list ``['*']`` so that the return value is the
@@ -22,7 +22,7 @@ Importing Modules
    be the case.  (Unfortunately, this has an additional side effect when *name* in
    fact specifies a subpackage instead of a submodule: the submodules specified in
    the package's ``__all__`` variable are  loaded.)  Return a new reference to the
-   imported module, or *NULL* with an exception set on failure.  A failing
+   imported module, or ``NULL`` with an exception set on failure.  A failing
    import of a module doesn't leave the module in :data:`sys.modules`.
 
    This function always uses absolute imports.
@@ -47,7 +47,7 @@ Importing Modules
    function :func:`__import__`.
 
    The return value is a new reference to the imported module or top-level
-   package, or *NULL* with an exception set on failure.  Like for
+   package, or ``NULL`` with an exception set on failure.  Like for
    :func:`__import__`, the return value when a submodule of a package was
    requested is normally the top-level package, unless a non-empty *fromlist*
    was given.
@@ -63,7 +63,7 @@ Importing Modules
    this function directly.
 
    The return value is a new reference to the imported module or top-level package,
-   or *NULL* with an exception set on failure.  Like for :func:`__import__`,
+   or ``NULL`` with an exception set on failure.  Like for :func:`__import__`,
    the return value when a submodule of a package was requested is normally the
    top-level package, unless a non-empty *fromlist* was given.
 
@@ -91,7 +91,7 @@ Importing Modules
 
 .. c:function:: PyObject* PyImport_ReloadModule(PyObject *m)
 
-   Reload a module.  Return a new reference to the reloaded module, or *NULL* with
+   Reload a module.  Return a new reference to the reloaded module, or ``NULL`` with
    an exception set on failure (the module still exists in this case).
 
 
@@ -100,7 +100,7 @@ Importing Modules
    Return the module object corresponding to a module name.  The *name* argument
    may be of the form ``package.module``. First check the modules dictionary if
    there's one there, and if not, create a new one and insert it in the modules
-   dictionary. Return *NULL* with an exception set on failure.
+   dictionary. Return ``NULL`` with an exception set on failure.
 
    .. note::
 
@@ -125,7 +125,7 @@ Importing Modules
    Given a module name (possibly of the form ``package.module``) and a code object
    read from a Python bytecode file or obtained from the built-in function
    :func:`compile`, load the module.  Return a new reference to the module object,
-   or *NULL* with an exception set if an error occurred.  *name*
+   or ``NULL`` with an exception set if an error occurred.  *name*
    is removed from :attr:`sys.modules` in error cases, even if *name* was already
    in :attr:`sys.modules` on entry to :c:func:`PyImport_ExecCodeModule`.  Leaving
    incompletely initialized modules in :attr:`sys.modules` is dangerous, as imports of
@@ -277,7 +277,7 @@ Importing Modules
 .. c:var:: const struct _frozen* PyImport_FrozenModules
 
    This pointer is initialized to point to an array of :c:type:`struct _frozen`
-   records, terminated by one whose members are all *NULL* or zero.  When a frozen
+   records, terminated by one whose members are all ``NULL`` or zero.  When a frozen
    module is imported, it is searched in this table.  Third-party code could play
    tricks with this to provide a dynamically created collection of frozen modules.
 
@@ -310,7 +310,7 @@ Importing Modules
 .. c:function:: int PyImport_ExtendInittab(struct _inittab *newtab)
 
    Add a collection of modules to the table of built-in modules.  The *newtab*
-   array must end with a sentinel entry which contains *NULL* for the :attr:`name`
+   array must end with a sentinel entry which contains ``NULL`` for the :attr:`name`
    field; failure to provide the sentinel value can result in a memory fault.
    Returns ``0`` on success or ``-1`` if insufficient memory could be allocated to
    extend the internal table.  In the event of failure, no modules are added to the
diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst
index 0b7a84d031532..e04302044989d 100644
--- a/Doc/c-api/init.rst
+++ b/Doc/c-api/init.rst
@@ -851,15 +851,15 @@ code, or when embedding the Python interpreter:
 .. c:function:: PyThreadState* PyEval_SaveThread()
 
    Release the global interpreter lock (if it has been created) and reset the
-   thread state to *NULL*, returning the previous thread state (which is not
-   *NULL*).  If the lock has been created, the current thread must have
+   thread state to ``NULL``, returning the previous thread state (which is not
+   ``NULL``).  If the lock has been created, the current thread must have
    acquired it.
 
 
 .. c:function:: void PyEval_RestoreThread(PyThreadState *tstate)
 
    Acquire the global interpreter lock (if it has been created) and set the
-   thread state to *tstate*, which must not be *NULL*.  If the lock has been
+   thread state to *tstate*, which must not be ``NULL``.  If the lock has been
    created, the current thread must not have acquired it, otherwise deadlock
    ensues.
 
@@ -873,14 +873,14 @@ code, or when embedding the Python interpreter:
 .. c:function:: PyThreadState* PyThreadState_Get()
 
    Return the current thread state.  The global interpreter lock must be held.
-   When the current thread state is *NULL*, this issues a fatal error (so that
-   the caller needn't check for *NULL*).
+   When the current thread state is ``NULL``, this issues a fatal error (so that
+   the caller needn't check for ``NULL``).
 
 
 .. c:function:: PyThreadState* PyThreadState_Swap(PyThreadState *tstate)
 
    Swap the current thread state with the thread state given by the argument
-   *tstate*, which may be *NULL*.  The global interpreter lock must be held
+   *tstate*, which may be ``NULL``.  The global interpreter lock must be held
    and is not released.
 
 
@@ -1046,7 +1046,7 @@ All of the following functions must be called after :c:func:`Py_Initialize`.
 .. c:function:: PyObject* PyInterpreterState_GetDict(PyInterpreterState *interp)
 
    Return a dictionary in which interpreter-specific data may be stored.
-   If this function returns *NULL* then no exception has been raised and
+   If this function returns ``NULL`` then no exception has been raised and
    the caller should assume no interpreter-specific dict is available.
 
    This is not a replacement for :c:func:`PyModule_GetState()`, which
@@ -1060,7 +1060,7 @@ All of the following functions must be called after :c:func:`Py_Initialize`.
    Return a dictionary in which extensions can store thread-specific state
    information.  Each extension should use a unique key to use to store state in
    the dictionary.  It is okay to call this function when no current thread state
-   is available. If this function returns *NULL*, no exception has been raised and
+   is available. If this function returns ``NULL``, no exception has been raised and
    the caller should assume no current thread state is available.
 
 
@@ -1081,7 +1081,7 @@ All of the following functions must be called after :c:func:`Py_Initialize`.
 .. c:function:: void PyEval_AcquireThread(PyThreadState *tstate)
 
    Acquire the global interpreter lock and set the current thread state to
-   *tstate*, which should not be *NULL*.  The lock must have been created earlier.
+   *tstate*, which should not be ``NULL``.  The lock must have been created earlier.
    If this thread already has the lock, deadlock ensues.
 
    .. note::
@@ -1102,9 +1102,9 @@ All of the following functions must be called after :c:func:`Py_Initialize`.
 
 .. c:function:: void PyEval_ReleaseThread(PyThreadState *tstate)
 
-   Reset the current thread state to *NULL* and release the global interpreter
+   Reset the current thread state to ``NULL`` and release the global interpreter
    lock.  The lock must have been created earlier and must be held by the current
-   thread.  The *tstate* argument, which must not be *NULL*, is only used to check
+   thread.  The *tstate* argument, which must not be ``NULL``, is only used to check
    that it represents the current thread state --- if it isn't, a fatal error is
    reported.
 
@@ -1189,7 +1189,7 @@ function. You can create and destroy them using the following functions:
    The return value points to the first thread state created in the new
    sub-interpreter.  This thread state is made in the current thread state.
    Note that no actual thread is created; see the discussion of thread states
-   below.  If creation of the new interpreter is unsuccessful, *NULL* is
+   below.  If creation of the new interpreter is unsuccessful, ``NULL`` is
    returned; no exception is set since the exception state is stored in the
    current thread state and there may not be a current thread state.  (Like all
    other Python/C API functions, the global interpreter lock must be held before
@@ -1220,7 +1220,7 @@ function. You can create and destroy them using the following functions:
 
    Destroy the (sub-)interpreter represented by the given thread state. The given
    thread state must be the current thread state.  See the discussion of thread
-   states below.  When the call returns, the current thread state is *NULL*.  All
+   states below.  When the call returns, the current thread state is ``NULL``.  All
    thread states associated with this interpreter are destroyed.  (The global
    interpreter lock must be held before calling this function and is still held
    when it returns.)  :c:func:`Py_FinalizeEx` will destroy all sub-interpreters that
@@ -1327,27 +1327,27 @@ Python-level trace functions in previous versions.
    :const:`PyTrace_C_CALL`, :const:`PyTrace_C_EXCEPTION`, :const:`PyTrace_C_RETURN`,
    or :const:`PyTrace_OPCODE`, and *arg* depends on the value of *what*:
 
-   +------------------------------+--------------------------------------+
-   | Value of *what*              | Meaning of *arg*                     |
-   +==============================+======================================+
-   | :const:`PyTrace_CALL`        | Always :c:data:`Py_None`.            |
-   +------------------------------+--------------------------------------+
-   | :const:`PyTrace_EXCEPTION`   | Exception information as returned by |
-   |                              | :func:`sys.exc_info`.                |
-   +------------------------------+--------------------------------------+
-   | :const:`PyTrace_LINE`        | Always :c:data:`Py_None`.            |
-   +------------------------------+--------------------------------------+
-   | :const:`PyTrace_RETURN`      | Value being returned to the caller,  |
-   |                              | or *NULL* if caused by an exception. |
-   +------------------------------+--------------------------------------+
-   | :const:`PyTrace_C_CALL`      | Function object being called.        |
-   +------------------------------+--------------------------------------+
-   | :const:`PyTrace_C_EXCEPTION` | Function object being called.        |
-   +------------------------------+--------------------------------------+
-   | :const:`PyTrace_C_RETURN`    | Function object being called.        |
-   +------------------------------+--------------------------------------+
-   | :const:`PyTrace_OPCODE`      | Always :c:data:`Py_None`.            |
-   +------------------------------+--------------------------------------+
+   +------------------------------+----------------------------------------+
+   | Value of *what*              | Meaning of *arg*                       |
+   +==============================+========================================+
+   | :const:`PyTrace_CALL`        | Always :c:data:`Py_None`.              |
+   +------------------------------+----------------------------------------+
+   | :const:`PyTrace_EXCEPTION`   | Exception information as returned by   |
+   |                              | :func:`sys.exc_info`.                  |
+   +------------------------------+----------------------------------------+
+   | :const:`PyTrace_LINE`        | Always :c:data:`Py_None`.              |
+   +------------------------------+----------------------------------------+
+   | :const:`PyTrace_RETURN`      | Value being returned to the caller,    |
+   |                              | or ``NULL`` if caused by an exception. |
+   +------------------------------+----------------------------------------+
+   | :const:`PyTrace_C_CALL`      | Function object being called.          |
+   +------------------------------+----------------------------------------+
+   | :const:`PyTrace_C_EXCEPTION` | Function object being called.          |
+   +------------------------------+----------------------------------------+
+   | :const:`PyTrace_C_RETURN`    | Function object being called.          |
+   +------------------------------+----------------------------------------+
+   | :const:`PyTrace_OPCODE`      | Always :c:data:`Py_None`.              |
+   +------------------------------+----------------------------------------+
 
 .. c:var:: int PyTrace_CALL
 
@@ -1411,7 +1411,7 @@ Python-level trace functions in previous versions.
 .. c:function:: void PyEval_SetProfile(Py_tracefunc func, PyObject *obj)
 
    Set the profiler function to *func*.  The *obj* parameter is passed to the
-   function as its first parameter, and may be any Python object, or *NULL*.  If
+   function as its first parameter, and may be any Python object, or ``NULL``.  If
    the profile function needs to maintain state, using a different value for *obj*
    for each thread provides a convenient and thread-safe place to store it.  The
    profile function is called for all monitored events except :const:`PyTrace_LINE`
@@ -1534,7 +1534,7 @@ is not possible due to its implementation being opaque at build time.
 .. c:function:: Py_tss_t* PyThread_tss_alloc()
 
    Return a value which is the same state as a value initialized with
-   :c:macro:`Py_tss_NEEDS_INIT`, or *NULL* in the case of dynamic allocation
+   :c:macro:`Py_tss_NEEDS_INIT`, or ``NULL`` in the case of dynamic allocation
    failure.
 
 
@@ -1553,7 +1553,7 @@ is not possible due to its implementation being opaque at build time.
 Methods
 ~~~~~~~
 
-The parameter *key* of these functions must not be *NULL*.  Moreover, the
+The parameter *key* of these functions must not be ``NULL``.  Moreover, the
 behaviors of :c:func:`PyThread_tss_set` and :c:func:`PyThread_tss_get` are
 undefined if the given :c:type:`Py_tss_t` has not been initialized by
 :c:func:`PyThread_tss_create`.
@@ -1593,7 +1593,7 @@ undefined if the given :c:type:`Py_tss_t` has not been initialized by
 .. c:function:: void* PyThread_tss_get(Py_tss_t *key)
 
    Return the :c:type:`void\*` value associated with a TSS key in the current
-   thread.  This returns *NULL* if no value is associated with the key in the
+   thread.  This returns ``NULL`` if no value is associated with the key in the
    current thread.
 
 
diff --git a/Doc/c-api/intro.rst b/Doc/c-api/intro.rst
index 80eebd89ad3f2..d08d4f97a308e 100644
--- a/Doc/c-api/intro.rst
+++ b/Doc/c-api/intro.rst
@@ -152,7 +152,7 @@ complete listing.
 
 .. c:macro:: Py_GETENV(s)
 
-   Like ``getenv(s)``, but returns *NULL* if :option:`-E` was passed on the
+   Like ``getenv(s)``, but returns ``NULL`` if :option:`-E` was passed on the
    command line (i.e. if ``Py_IgnoreEnvironmentFlag`` is set).
 
 .. c:macro:: Py_UNUSED(arg)
@@ -475,7 +475,7 @@ functions in the Python/C API can raise exceptions, unless an explicit claim is
 made otherwise in a function's documentation.  In general, when a function
 encounters an error, it sets an exception, discards any object references that
 it owns, and returns an error indicator.  If not documented otherwise, this
-indicator is either *NULL* or ``-1``, depending on the function's return type.
+indicator is either ``NULL`` or ``-1``, depending on the function's return type.
 A few functions return a Boolean true/false result, with false indicating an
 error.  Very few functions return no explicit error indicator or have an
 ambiguous return value, and require explicit testing for errors with
@@ -490,13 +490,13 @@ using global storage in an unthreaded application).  A  thread can be in one of
 two states: an exception has occurred, or not. The function
 :c:func:`PyErr_Occurred` can be used to check for this: it returns a borrowed
 reference to the exception type object when an exception has occurred, and
-*NULL* otherwise.  There are a number of functions to set the exception state:
+``NULL`` otherwise.  There are a number of functions to set the exception state:
 :c:func:`PyErr_SetString` is the most common (though not the most general)
 function to set the exception state, and :c:func:`PyErr_Clear` clears the
 exception state.
 
 The full exception state consists of three objects (all of which can  be
-*NULL*): the exception type, the corresponding exception  value, and the
+``NULL``): the exception type, the corresponding exception  value, and the
 traceback.  These have the same meanings as the Python result of
 ``sys.exc_info()``; however, they are not the same: the Python objects represent
 the last exception being handled by a Python  :keyword:`try` ...
@@ -597,10 +597,10 @@ Here is the corresponding C code, in all its glory::
 This example represents an endorsed use of the ``goto`` statement  in C!
 It illustrates the use of :c:func:`PyErr_ExceptionMatches` and
 :c:func:`PyErr_Clear` to handle specific exceptions, and the use of
-:c:func:`Py_XDECREF` to dispose of owned references that may be *NULL* (note the
+:c:func:`Py_XDECREF` to dispose of owned references that may be ``NULL`` (note the
 ``'X'`` in the name; :c:func:`Py_DECREF` would crash when confronted with a
-*NULL* reference).  It is important that the variables used to hold owned
-references are initialized to *NULL* for this to work; likewise, the proposed
+``NULL`` reference).  It is important that the variables used to hold owned
+references are initialized to ``NULL`` for this to work; likewise, the proposed
 return value is initialized to ``-1`` (failure) and only set to success after
 the final call made is successful.
 
diff --git a/Doc/c-api/iter.rst b/Doc/c-api/iter.rst
index 546efb518a7af..0224d37f1a41a 100644
--- a/Doc/c-api/iter.rst
+++ b/Doc/c-api/iter.rst
@@ -16,8 +16,8 @@ There are two functions specifically for working with iterators.
 
    Return the next value from the iteration *o*.  The object must be an iterator
    (it is up to the caller to check this).  If there are no remaining values,
-   returns *NULL* with no exception set.  If an error occurs while retrieving
-   the item, returns *NULL* and passes along the exception.
+   returns ``NULL`` with no exception set.  If an error occurs while retrieving
+   the item, returns ``NULL`` and passes along the exception.
 
 To write a loop which iterates over an iterator, the C code should look
 something like this::
diff --git a/Doc/c-api/list.rst b/Doc/c-api/list.rst
index 646f29d08a04b..b247cdfba0187 100644
--- a/Doc/c-api/list.rst
+++ b/Doc/c-api/list.rst
@@ -33,7 +33,7 @@ List Objects
 
 .. c:function:: PyObject* PyList_New(Py_ssize_t len)
 
-   Return a new list of length *len* on success, or *NULL* on failure.
+   Return a new list of length *len* on success, or ``NULL`` on failure.
 
    .. note::
 
@@ -61,7 +61,7 @@ List Objects
    Return the object at position *index* in the list pointed to by *list*.  The
    position must be non-negative; indexing from the end of the list is not
    supported.  If *index* is out of bounds (<0 or >=len(list)),
-   return *NULL* and set an :exc:`IndexError` exception.
+   return ``NULL`` and set an :exc:`IndexError` exception.
 
 
 .. c:function:: PyObject* PyList_GET_ITEM(PyObject *list, Py_ssize_t i)
@@ -111,7 +111,7 @@ List Objects
 .. c:function:: PyObject* PyList_GetSlice(PyObject *list, Py_ssize_t low, Py_ssize_t high)
 
    Return a list of the objects in *list* containing the objects *between* *low*
-   and *high*.  Return *NULL* and set an exception if unsuccessful.  Analogous
+   and *high*.  Return ``NULL`` and set an exception if unsuccessful.  Analogous
    to ``list[low:high]``.  Indexing from the end of the list is not supported.
 
 
@@ -119,7 +119,7 @@ List Objects
 
    Set the slice of *list* between *low* and *high* to the contents of
    *itemlist*.  Analogous to ``list[low:high] = itemlist``. The *itemlist* may
-   be *NULL*, indicating the assignment of an empty list (slice deletion).
+   be ``NULL``, indicating the assignment of an empty list (slice deletion).
    Return ``0`` on success, ``-1`` on failure.  Indexing from the end of the
    list is not supported.
 
diff --git a/Doc/c-api/long.rst b/Doc/c-api/long.rst
index fdaefafe21ba0..33f6b3be58307 100644
--- a/Doc/c-api/long.rst
+++ b/Doc/c-api/long.rst
@@ -38,7 +38,7 @@ distinguished from a number.  Use :c:func:`PyErr_Occurred` to disambiguate.
 
 .. c:function:: PyObject* PyLong_FromLong(long v)
 
-   Return a new :c:type:`PyLongObject` object from *v*, or *NULL* on failure.
+   Return a new :c:type:`PyLongObject` object from *v*, or ``NULL`` on failure.
 
    The current implementation keeps an array of integer objects for all integers
    between ``-5`` and ``256``, when you create an int in that range you actually
@@ -50,43 +50,43 @@ distinguished from a number.  Use :c:func:`PyErr_Occurred` to disambiguate.
 .. c:function:: PyObject* PyLong_FromUnsignedLong(unsigned long v)
 
    Return a new :c:type:`PyLongObject` object from a C :c:type:`unsigned long`, or
-   *NULL* on failure.
+   ``NULL`` on failure.
 
 
 .. c:function:: PyObject* PyLong_FromSsize_t(Py_ssize_t v)
 
    Return a new :c:type:`PyLongObject` object from a C :c:type:`Py_ssize_t`, or
-   *NULL* on failure.
+   ``NULL`` on failure.
 
 
 .. c:function:: PyObject* PyLong_FromSize_t(size_t v)
 
    Return a new :c:type:`PyLongObject` object from a C :c:type:`size_t`, or
-   *NULL* on failure.
+   ``NULL`` on failure.
 
 
 .. c:function:: PyObject* PyLong_FromLongLong(long long v)
 
-   Return a new :c:type:`PyLongObject` object from a C :c:type:`long long`, or *NULL*
+   Return a new :c:type:`PyLongObject` object from a C :c:type:`long long`, or ``NULL``
    on failure.
 
 
 .. c:function:: PyObject* PyLong_FromUnsignedLongLong(unsigned long long v)
 
    Return a new :c:type:`PyLongObject` object from a C :c:type:`unsigned long long`,
-   or *NULL* on failure.
+   or ``NULL`` on failure.
 
 
 .. c:function:: PyObject* PyLong_FromDouble(double v)
 
    Return a new :c:type:`PyLongObject` object from the integer part of *v*, or
-   *NULL* on failure.
+   ``NULL`` on failure.
 
 
 .. c:function:: PyObject* PyLong_FromString(const char *str, char **pend, int base)
 
    Return a new :c:type:`PyLongObject` based on the string value in *str*, which
-   is interpreted according to the radix in *base*.  If *pend* is non-*NULL*,
+   is interpreted according to the radix in *base*.  If *pend* is non-``NULL``,
    *\*pend* will point to the first character in *str* which follows the
    representation of the number.  If *base* is ``0``, *str* is interpreted using
    the :ref:`integers` definition; in this case, leading zeros in a
@@ -336,4 +336,4 @@ distinguished from a number.  Use :c:func:`PyErr_Occurred` to disambiguate.
    is only assured to produce a usable :c:type:`void` pointer for values created
    with :c:func:`PyLong_FromVoidPtr`.
 
-   Returns *NULL* on error.  Use :c:func:`PyErr_Occurred` to disambiguate.
+   Returns ``NULL`` on error.  Use :c:func:`PyErr_Occurred` to disambiguate.
diff --git a/Doc/c-api/mapping.rst b/Doc/c-api/mapping.rst
index 4244b47af75f1..4015f43d8f1ee 100644
--- a/Doc/c-api/mapping.rst
+++ b/Doc/c-api/mapping.rst
@@ -29,7 +29,7 @@ See also :c:func:`PyObject_GetItem`, :c:func:`PyObject_SetItem` and
 
 .. c:function:: PyObject* PyMapping_GetItemString(PyObject *o, const char *key)
 
-   Return element of *o* corresponding to the string *key* or *NULL* on failure.
+   Return element of *o* corresponding to the string *key* or ``NULL`` on failure.
    This is the equivalent of the Python expression ``o[key]``.
    See also :c:func:`PyObject_GetItem`.
 
@@ -79,7 +79,7 @@ See also :c:func:`PyObject_GetItem`, :c:func:`PyObject_SetItem` and
 .. c:function:: PyObject* PyMapping_Keys(PyObject *o)
 
    On success, return a list of the keys in object *o*.  On failure, return
-   *NULL*.
+   ``NULL``.
 
    .. versionchanged:: 3.7
       Previously, the function returned a list or a tuple.
@@ -88,7 +88,7 @@ See also :c:func:`PyObject_GetItem`, :c:func:`PyObject_SetItem` and
 .. c:function:: PyObject* PyMapping_Values(PyObject *o)
 
    On success, return a list of the values in object *o*.  On failure, return
-   *NULL*.
+   ``NULL``.
 
    .. versionchanged:: 3.7
       Previously, the function returned a list or a tuple.
@@ -97,7 +97,7 @@ See also :c:func:`PyObject_GetItem`, :c:func:`PyObject_SetItem` and
 .. c:function:: PyObject* PyMapping_Items(PyObject *o)
 
    On success, return a list of the items in object *o*, where each item is a
-   tuple containing a key-value pair.  On failure, return *NULL*.
+   tuple containing a key-value pair.  On failure, return ``NULL``.
 
    .. versionchanged:: 3.7
       Previously, the function returned a list or a tuple.
diff --git a/Doc/c-api/marshal.rst b/Doc/c-api/marshal.rst
index b086830feb14f..2f580430237af 100644
--- a/Doc/c-api/marshal.rst
+++ b/Doc/c-api/marshal.rst
@@ -67,7 +67,7 @@ The following functions allow marshalled values to be read back in.
    reading.
 
    On error, sets the appropriate exception (:exc:`EOFError`, :exc:`ValueError`
-   or :exc:`TypeError`) and returns *NULL*.
+   or :exc:`TypeError`) and returns ``NULL``.
 
 
 .. c:function:: PyObject* PyMarshal_ReadLastObjectFromFile(FILE *file)
@@ -81,7 +81,7 @@ The following functions allow marshalled values to be read back in.
    anything else from the file.
 
    On error, sets the appropriate exception (:exc:`EOFError`, :exc:`ValueError`
-   or :exc:`TypeError`) and returns *NULL*.
+   or :exc:`TypeError`) and returns ``NULL``.
 
 
 .. c:function:: PyObject* PyMarshal_ReadObjectFromString(const char *data, Py_ssize_t len)
@@ -90,5 +90,5 @@ The following functions allow marshalled values to be read back in.
    containing *len* bytes pointed to by *data*.
 
    On error, sets the appropriate exception (:exc:`EOFError`, :exc:`ValueError`
-   or :exc:`TypeError`) and returns *NULL*.
+   or :exc:`TypeError`) and returns ``NULL``.
 
diff --git a/Doc/c-api/memory.rst b/Doc/c-api/memory.rst
index d3c8b30d360ec..c5b3af64c16a4 100644
--- a/Doc/c-api/memory.rst
+++ b/Doc/c-api/memory.rst
@@ -110,9 +110,9 @@ zero bytes.
 .. c:function:: void* PyMem_RawMalloc(size_t n)
 
    Allocates *n* bytes and returns a pointer of type :c:type:`void\*` to the
-   allocated memory, or *NULL* if the request fails.
+   allocated memory, or ``NULL`` if the request fails.
 
-   Requesting zero bytes returns a distinct non-*NULL* pointer if possible, as
+   Requesting zero bytes returns a distinct non-``NULL`` pointer if possible, as
    if ``PyMem_RawMalloc(1)`` had been called instead. The memory will not have
    been initialized in any way.
 
@@ -120,11 +120,11 @@ zero bytes.
 .. c:function:: void* PyMem_RawCalloc(size_t nelem, size_t elsize)
 
    Allocates *nelem* elements each whose size in bytes is *elsize* and returns
-   a pointer of type :c:type:`void\*` to the allocated memory, or *NULL* if the
+   a pointer of type :c:type:`void\*` to the allocated memory, or ``NULL`` if the
    request fails. The memory is initialized to zeros.
 
    Requesting zero elements or elements of size zero bytes returns a distinct
-   non-*NULL* pointer if possible, as if ``PyMem_RawCalloc(1, 1)`` had been
+   non-``NULL`` pointer if possible, as if ``PyMem_RawCalloc(1, 1)`` had been
    called instead.
 
    .. versionadded:: 3.5
@@ -135,15 +135,15 @@ zero bytes.
    Resizes the memory block pointed to by *p* to *n* bytes. The contents will
    be unchanged to the minimum of the old and the new sizes.
 
-   If *p* is *NULL*, the call is equivalent to ``PyMem_RawMalloc(n)``; else if
+   If *p* is ``NULL``, the call is equivalent to ``PyMem_RawMalloc(n)``; else if
    *n* is equal to zero, the memory block is resized but is not freed, and the
-   returned pointer is non-*NULL*.
+   returned pointer is non-``NULL``.
 
-   Unless *p* is *NULL*, it must have been returned by a previous call to
+   Unless *p* is ``NULL``, it must have been returned by a previous call to
    :c:func:`PyMem_RawMalloc`, :c:func:`PyMem_RawRealloc` or
    :c:func:`PyMem_RawCalloc`.
 
-   If the request fails, :c:func:`PyMem_RawRealloc` returns *NULL* and *p*
+   If the request fails, :c:func:`PyMem_RawRealloc` returns ``NULL`` and *p*
    remains a valid pointer to the previous memory area.
 
 
@@ -154,7 +154,7 @@ zero bytes.
    :c:func:`PyMem_RawCalloc`.  Otherwise, or if ``PyMem_RawFree(p)`` has been
    called before, undefined behavior occurs.
 
-   If *p* is *NULL*, no operation is performed.
+   If *p* is ``NULL``, no operation is performed.
 
 
 .. _memoryinterface:
@@ -181,9 +181,9 @@ The :ref:`default memory allocator <default-memory-allocators>` uses the
 .. c:function:: void* PyMem_Malloc(size_t n)
 
    Allocates *n* bytes and returns a pointer of type :c:type:`void\*` to the
-   allocated memory, or *NULL* if the request fails.
+   allocated memory, or ``NULL`` if the request fails.
 
-   Requesting zero bytes returns a distinct non-*NULL* pointer if possible, as
+   Requesting zero bytes returns a distinct non-``NULL`` pointer if possible, as
    if ``PyMem_Malloc(1)`` had been called instead. The memory will not have
    been initialized in any way.
 
@@ -191,11 +191,11 @@ The :ref:`default memory allocator <default-memory-allocators>` uses the
 .. c:function:: void* PyMem_Calloc(size_t nelem, size_t elsize)
 
    Allocates *nelem* elements each whose size in bytes is *elsize* and returns
-   a pointer of type :c:type:`void\*` to the allocated memory, or *NULL* if the
+   a pointer of type :c:type:`void\*` to the allocated memory, or ``NULL`` if the
    request fails. The memory is initialized to zeros.
 
    Requesting zero elements or elements of size zero bytes returns a distinct
-   non-*NULL* pointer if possible, as if ``PyMem_Calloc(1, 1)`` had been called
+   non-``NULL`` pointer if possible, as if ``PyMem_Calloc(1, 1)`` had been called
    instead.
 
    .. versionadded:: 3.5
@@ -206,14 +206,14 @@ The :ref:`default memory allocator <default-memory-allocators>` uses the
    Resizes the memory block pointed to by *p* to *n* bytes. The contents will be
    unchanged to the minimum of the old and the new sizes.
 
-   If *p* is *NULL*, the call is equivalent to ``PyMem_Malloc(n)``; else if *n*
+   If *p* is ``NULL``, the call is equivalent to ``PyMem_Malloc(n)``; else if *n*
    is equal to zero, the memory block is resized but is not freed, and the
-   returned pointer is non-*NULL*.
+   returned pointer is non-``NULL``.
 
-   Unless *p* is *NULL*, it must have been returned by a previous call to
+   Unless *p* is ``NULL``, it must have been returned by a previous call to
    :c:func:`PyMem_Malloc`, :c:func:`PyMem_Realloc` or :c:func:`PyMem_Calloc`.
 
-   If the request fails, :c:func:`PyMem_Realloc` returns *NULL* and *p* remains
+   If the request fails, :c:func:`PyMem_Realloc` returns ``NULL`` and *p* remains
    a valid pointer to the previous memory area.
 
 
@@ -224,7 +224,7 @@ The :ref:`default memory allocator <default-memory-allocators>` uses the
    :c:func:`PyMem_Calloc`.  Otherwise, or if ``PyMem_Free(p)`` has been called
    before, undefined behavior occurs.
 
-   If *p* is *NULL*, no operation is performed.
+   If *p* is ``NULL``, no operation is performed.
 
 The following type-oriented macros are provided for convenience.  Note  that
 *TYPE* refers to any C type.
@@ -241,7 +241,7 @@ The following type-oriented macros are provided for convenience.  Note  that
 
    Same as :c:func:`PyMem_Realloc`, but the memory block is resized to ``(n *
    sizeof(TYPE))`` bytes.  Returns a pointer cast to :c:type:`TYPE\*`. On return,
-   *p* will be a pointer to the new memory area, or *NULL* in the event of
+   *p* will be a pointer to the new memory area, or ``NULL`` in the event of
    failure.
 
    This is a C preprocessor macro; *p* is always reassigned.  Save the original
@@ -283,9 +283,9 @@ The :ref:`default object allocator <default-memory-allocators>` uses the
 .. c:function:: void* PyObject_Malloc(size_t n)
 
    Allocates *n* bytes and returns a pointer of type :c:type:`void\*` to the
-   allocated memory, or *NULL* if the request fails.
+   allocated memory, or ``NULL`` if the request fails.
 
-   Requesting zero bytes returns a distinct non-*NULL* pointer if possible, as
+   Requesting zero bytes returns a distinct non-``NULL`` pointer if possible, as
    if ``PyObject_Malloc(1)`` had been called instead. The memory will not have
    been initialized in any way.
 
@@ -293,11 +293,11 @@ The :ref:`default object allocator <default-memory-allocators>` uses the
 .. c:function:: void* PyObject_Calloc(size_t nelem, size_t elsize)
 
    Allocates *nelem* elements each whose size in bytes is *elsize* and returns
-   a pointer of type :c:type:`void\*` to the allocated memory, or *NULL* if the
+   a pointer of type :c:type:`void\*` to the allocated memory, or ``NULL`` if the
    request fails. The memory is initialized to zeros.
 
    Requesting zero elements or elements of size zero bytes returns a distinct
-   non-*NULL* pointer if possible, as if ``PyObject_Calloc(1, 1)`` had been called
+   non-``NULL`` pointer if possible, as if ``PyObject_Calloc(1, 1)`` had been called
    instead.
 
    .. versionadded:: 3.5
@@ -308,14 +308,14 @@ The :ref:`default object allocator <default-memory-allocators>` uses the
    Resizes the memory block pointed to by *p* to *n* bytes. The contents will be
    unchanged to the minimum of the old and the new sizes.
 
-   If *p* is *NULL*, the call is equivalent to ``PyObject_Malloc(n)``; else if *n*
+   If *p* is ``NULL``, the call is equivalent to ``PyObject_Malloc(n)``; else if *n*
    is equal to zero, the memory block is resized but is not freed, and the
-   returned pointer is non-*NULL*.
+   returned pointer is non-``NULL``.
 
-   Unless *p* is *NULL*, it must have been returned by a previous call to
+   Unless *p* is ``NULL``, it must have been returned by a previous call to
    :c:func:`PyObject_Malloc`, :c:func:`PyObject_Realloc` or :c:func:`PyObject_Calloc`.
 
-   If the request fails, :c:func:`PyObject_Realloc` returns *NULL* and *p* remains
+   If the request fails, :c:func:`PyObject_Realloc` returns ``NULL`` and *p* remains
    a valid pointer to the previous memory area.
 
 
@@ -326,7 +326,7 @@ The :ref:`default object allocator <default-memory-allocators>` uses the
    :c:func:`PyObject_Calloc`.  Otherwise, or if ``PyObject_Free(p)`` has been called
    before, undefined behavior occurs.
 
-   If *p* is *NULL*, no operation is performed.
+   If *p* is ``NULL``, no operation is performed.
 
 
 .. _default-memory-allocators:
diff --git a/Doc/c-api/memoryview.rst b/Doc/c-api/memoryview.rst
index 77afb020405a7..de429e5c11dc7 100644
--- a/Doc/c-api/memoryview.rst
+++ b/Doc/c-api/memoryview.rst
@@ -57,7 +57,7 @@ any other object.
 .. c:function:: Py_buffer *PyMemoryView_GET_BASE(PyObject *mview)
 
    Return either a pointer to the exporting object that the memoryview is based
-   on or *NULL* if the memoryview has been created by one of the functions
+   on or ``NULL`` if the memoryview has been created by one of the functions
    :c:func:`PyMemoryView_FromMemory` or :c:func:`PyMemoryView_FromBuffer`.
    *mview* **must** be a memoryview instance.
 
diff --git a/Doc/c-api/method.rst b/Doc/c-api/method.rst
index 1ad805e269aa2..b1862d796c9f4 100644
--- a/Doc/c-api/method.rst
+++ b/Doc/c-api/method.rst
@@ -21,7 +21,7 @@ to bind a :c:data:`PyCFunction` to a class object. It replaces the former call
 .. c:function:: int PyInstanceMethod_Check(PyObject *o)
 
    Return true if *o* is an instance method object (has type
-   :c:data:`PyInstanceMethod_Type`).  The parameter must not be *NULL*.
+   :c:data:`PyInstanceMethod_Type`).  The parameter must not be ``NULL``.
 
 
 .. c:function:: PyObject* PyInstanceMethod_New(PyObject *func)
@@ -64,14 +64,14 @@ no longer available.
 .. c:function:: int PyMethod_Check(PyObject *o)
 
    Return true if *o* is a method object (has type :c:data:`PyMethod_Type`).  The
-   parameter must not be *NULL*.
+   parameter must not be ``NULL``.
 
 
 .. c:function:: PyObject* PyMethod_New(PyObject *func, PyObject *self)
 
    Return a new method object, with *func* being any callable object and *self*
    the instance the method should be bound. *func* is the function that will
-   be called when the method is called. *self* must not be *NULL*.
+   be called when the method is called. *self* must not be ``NULL``.
 
 
 .. c:function:: PyObject* PyMethod_Function(PyObject *meth)
diff --git a/Doc/c-api/module.rst b/Doc/c-api/module.rst
index feca1ec2a057a..46306b3917cce 100644
--- a/Doc/c-api/module.rst
+++ b/Doc/c-api/module.rst
@@ -61,7 +61,7 @@ Module Objects
    Return the dictionary object that implements *module*'s namespace; this object
    is the same as the :attr:`~object.__dict__` attribute of the module object.
    If *module* is not a module object (or a subtype of a module object),
-   :exc:`SystemError` is raised and *NULL* is returned.
+   :exc:`SystemError` is raised and ``NULL`` is returned.
 
    It is recommended extensions use other :c:func:`PyModule_\*` and
    :c:func:`PyObject_\*` functions rather than directly manipulate a module's
@@ -75,7 +75,7 @@ Module Objects
       single: SystemError (built-in exception)
 
    Return *module*'s :attr:`__name__` value.  If the module does not provide one,
-   or if it is not a string, :exc:`SystemError` is raised and *NULL* is returned.
+   or if it is not a string, :exc:`SystemError` is raised and ``NULL`` is returned.
 
    .. versionadded:: 3.3
 
@@ -88,14 +88,14 @@ Module Objects
 .. c:function:: void* PyModule_GetState(PyObject *module)
 
    Return the "state" of the module, that is, a pointer to the block of memory
-   allocated at module creation time, or *NULL*.  See
+   allocated at module creation time, or ``NULL``.  See
    :c:member:`PyModuleDef.m_size`.
 
 
 .. c:function:: PyModuleDef* PyModule_GetDef(PyObject *module)
 
    Return a pointer to the :c:type:`PyModuleDef` struct from which the module was
-   created, or *NULL* if the module wasn't created from a definition.
+   created, or ``NULL`` if the module wasn't created from a definition.
 
 
 .. c:function:: PyObject* PyModule_GetFilenameObject(PyObject *module)
@@ -106,7 +106,7 @@ Module Objects
 
    Return the name of the file from which *module* was loaded using *module*'s
    :attr:`__file__` attribute.  If this is not defined, or if it is not a
-   unicode string, raise :exc:`SystemError` and return *NULL*; otherwise return
+   unicode string, raise :exc:`SystemError` and return ``NULL``; otherwise return
    a reference to a Unicode object.
 
    .. versionadded:: 3.2
@@ -178,17 +178,17 @@ or request "multi-phase initialization" by returning the definition struct itsel
    .. c:member:: PyMethodDef* m_methods
 
       A pointer to a table of module-level functions, described by
-      :c:type:`PyMethodDef` values.  Can be *NULL* if no functions are present.
+      :c:type:`PyMethodDef` values.  Can be ``NULL`` if no functions are present.
 
    .. c:member:: PyModuleDef_Slot* m_slots
 
       An array of slot definitions for multi-phase initialization, terminated by
       a ``{0, NULL}`` entry.
-      When using single-phase initialization, *m_slots* must be *NULL*.
+      When using single-phase initialization, *m_slots* must be ``NULL``.
 
       .. versionchanged:: 3.5
 
-         Prior to version 3.5, this member was always set to *NULL*,
+         Prior to version 3.5, this member was always set to ``NULL``,
          and was defined as:
 
            .. c:member:: inquiry m_reload
@@ -196,20 +196,20 @@ or request "multi-phase initialization" by returning the definition struct itsel
    .. c:member:: traverseproc m_traverse
 
       A traversal function to call during GC traversal of the module object, or
-      *NULL* if not needed. This function may be called before module state
+      ``NULL`` if not needed. This function may be called before module state
       is allocated (:c:func:`PyModule_GetState()` may return `NULL`),
       and before the :c:member:`Py_mod_exec` function is executed.
 
    .. c:member:: inquiry m_clear
 
       A clear function to call during GC clearing of the module object, or
-      *NULL* if not needed. This function may be called before module state
+      ``NULL`` if not needed. This function may be called before module state
       is allocated (:c:func:`PyModule_GetState()` may return `NULL`),
       and before the :c:member:`Py_mod_exec` function is executed.
 
    .. c:member:: freefunc m_free
 
-      A function to call during deallocation of the module object, or *NULL* if
+      A function to call during deallocation of the module object, or ``NULL`` if
       not needed. This function may be called before module state
       is allocated (:c:func:`PyModule_GetState()` may return `NULL`),
       and before the :c:member:`Py_mod_exec` function is executed.
@@ -278,7 +278,7 @@ instance must be initialized with the following function:
    Ensures a module definition is a properly initialized Python object that
    correctly reports its type and reference count.
 
-   Returns *def* cast to ``PyObject*``, or *NULL* if an error occurred.
+   Returns *def* cast to ``PyObject*``, or ``NULL`` if an error occurred.
 
    .. versionadded:: 3.5
 
@@ -311,7 +311,7 @@ The available slot types are:
    The function receives a :py:class:`~importlib.machinery.ModuleSpec`
    instance, as defined in :PEP:`451`, and the module definition.
    It should return a new module object, or set an error
-   and return *NULL*.
+   and return ``NULL``.
 
    This function should be kept minimal. In particular, it should not
    call arbitrary Python code, as trying to import the same module again may
@@ -330,7 +330,7 @@ The available slot types are:
    :c:type:`PyModule_Type`. Any type can be used, as long as it supports
    setting and getting import-related attributes.
    However, only ``PyModule_Type`` instances may be returned if the
-   ``PyModuleDef`` has non-*NULL* ``m_traverse``, ``m_clear``,
+   ``PyModuleDef`` has non-``NULL`` ``m_traverse``, ``m_clear``,
    ``m_free``; non-zero ``m_size``; or slots other than ``Py_mod_create``.
 
 .. c:var:: Py_mod_exec
@@ -394,7 +394,7 @@ objects dynamically. Note that both ``PyModule_FromDefAndSpec`` and
 
 .. c:function:: int PyModule_AddFunctions(PyObject *module, PyMethodDef *functions)
 
-   Add the functions from the *NULL* terminated *functions* array to *module*.
+   Add the functions from the ``NULL`` terminated *functions* array to *module*.
    Refer to the :c:type:`PyMethodDef` documentation for details on individual
    entries (due to the lack of a shared module namespace, module level
    "functions" implemented in C typically receive the module as their first
@@ -445,7 +445,7 @@ state:
 
    Add a string constant to *module* as *name*.  This convenience function can be
    used from the module's initialization function.  The string *value* must be
-   *NULL*-terminated.  Return ``-1`` on error, ``0`` on success.
+   ``NULL``-terminated.  Return ``-1`` on error, ``0`` on success.
 
 
 .. c:function:: int PyModule_AddIntMacro(PyObject *module, macro)
@@ -476,7 +476,7 @@ since multiple such modules can be created from a single definition.
    Returns the module object that was created from *def* for the current interpreter.
    This method requires that the module object has been attached to the interpreter state with
    :c:func:`PyState_AddModule` beforehand. In case the corresponding module object is not
-   found or has not been attached to the interpreter state yet, it returns *NULL*.
+   found or has not been attached to the interpreter state yet, it returns ``NULL``.
 
 .. c:function:: int PyState_AddModule(PyObject *module, PyModuleDef *def)
 
diff --git a/Doc/c-api/number.rst b/Doc/c-api/number.rst
index 74932f6e7445f..55ddb9c0c8206 100644
--- a/Doc/c-api/number.rst
+++ b/Doc/c-api/number.rst
@@ -17,25 +17,25 @@ Number Protocol
 
 .. c:function:: PyObject* PyNumber_Add(PyObject *o1, PyObject *o2)
 
-   Returns the result of adding *o1* and *o2*, or *NULL* on failure.  This is the
+   Returns the result of adding *o1* and *o2*, or ``NULL`` on failure.  This is the
    equivalent of the Python expression ``o1 + o2``.
 
 
 .. c:function:: PyObject* PyNumber_Subtract(PyObject *o1, PyObject *o2)
 
-   Returns the result of subtracting *o2* from *o1*, or *NULL* on failure.  This is
+   Returns the result of subtracting *o2* from *o1*, or ``NULL`` on failure.  This is
    the equivalent of the Python expression ``o1 - o2``.
 
 
 .. c:function:: PyObject* PyNumber_Multiply(PyObject *o1, PyObject *o2)
 
-   Returns the result of multiplying *o1* and *o2*, or *NULL* on failure.  This is
+   Returns the result of multiplying *o1* and *o2*, or ``NULL`` on failure.  This is
    the equivalent of the Python expression ``o1 * o2``.
 
 
 .. c:function:: PyObject* PyNumber_MatrixMultiply(PyObject *o1, PyObject *o2)
 
-   Returns the result of matrix multiplication on *o1* and *o2*, or *NULL* on
+   Returns the result of matrix multiplication on *o1* and *o2*, or ``NULL`` on
    failure.  This is the equivalent of the Python expression ``o1 @ o2``.
 
    .. versionadded:: 3.5
@@ -43,14 +43,14 @@ Number Protocol
 
 .. c:function:: PyObject* PyNumber_FloorDivide(PyObject *o1, PyObject *o2)
 
-   Return the floor of *o1* divided by *o2*, or *NULL* on failure.  This is
+   Return the floor of *o1* divided by *o2*, or ``NULL`` on failure.  This is
    equivalent to the "classic" division of integers.
 
 
 .. c:function:: PyObject* PyNumber_TrueDivide(PyObject *o1, PyObject *o2)
 
    Return a reasonable approximation for the mathematical value of *o1* divided by
-   *o2*, or *NULL* on failure.  The return value is "approximate" because binary
+   *o2*, or ``NULL`` on failure.  The return value is "approximate" because binary
    floating point numbers are approximate; it is not possible to represent all real
    numbers in base two.  This function can return a floating point value when
    passed two integers.
@@ -58,7 +58,7 @@ Number Protocol
 
 .. c:function:: PyObject* PyNumber_Remainder(PyObject *o1, PyObject *o2)
 
-   Returns the remainder of dividing *o1* by *o2*, or *NULL* on failure.  This is
+   Returns the remainder of dividing *o1* by *o2*, or ``NULL`` on failure.  This is
    the equivalent of the Python expression ``o1 % o2``.
 
 
@@ -66,7 +66,7 @@ Number Protocol
 
    .. index:: builtin: divmod
 
-   See the built-in function :func:`divmod`. Returns *NULL* on failure.  This is
+   See the built-in function :func:`divmod`. Returns ``NULL`` on failure.  This is
    the equivalent of the Python expression ``divmod(o1, o2)``.
 
 
@@ -74,21 +74,21 @@ Number Protocol
 
    .. index:: builtin: pow
 
-   See the built-in function :func:`pow`. Returns *NULL* on failure.  This is the
+   See the built-in function :func:`pow`. Returns ``NULL`` on failure.  This is the
    equivalent of the Python expression ``pow(o1, o2, o3)``, where *o3* is optional.
-   If *o3* is to be ignored, pass :c:data:`Py_None` in its place (passing *NULL* for
+   If *o3* is to be ignored, pass :c:data:`Py_None` in its place (passing ``NULL`` for
    *o3* would cause an illegal memory access).
 
 
 .. c:function:: PyObject* PyNumber_Negative(PyObject *o)
 
-   Returns the negation of *o* on success, or *NULL* on failure. This is the
+   Returns the negation of *o* on success, or ``NULL`` on failure. This is the
    equivalent of the Python expression ``-o``.
 
 
 .. c:function:: PyObject* PyNumber_Positive(PyObject *o)
 
-   Returns *o* on success, or *NULL* on failure.  This is the equivalent of the
+   Returns *o* on success, or ``NULL`` on failure.  This is the equivalent of the
    Python expression ``+o``.
 
 
@@ -96,70 +96,70 @@ Number Protocol
 
    .. index:: builtin: abs
 
-   Returns the absolute value of *o*, or *NULL* on failure.  This is the equivalent
+   Returns the absolute value of *o*, or ``NULL`` on failure.  This is the equivalent
    of the Python expression ``abs(o)``.
 
 
 .. c:function:: PyObject* PyNumber_Invert(PyObject *o)
 
-   Returns the bitwise negation of *o* on success, or *NULL* on failure.  This is
+   Returns the bitwise negation of *o* on success, or ``NULL`` on failure.  This is
    the equivalent of the Python expression ``~o``.
 
 
 .. c:function:: PyObject* PyNumber_Lshift(PyObject *o1, PyObject *o2)
 
-   Returns the result of left shifting *o1* by *o2* on success, or *NULL* on
+   Returns the result of left shifting *o1* by *o2* on success, or ``NULL`` on
    failure.  This is the equivalent of the Python expression ``o1 << o2``.
 
 
 .. c:function:: PyObject* PyNumber_Rshift(PyObject *o1, PyObject *o2)
 
-   Returns the result of right shifting *o1* by *o2* on success, or *NULL* on
+   Returns the result of right shifting *o1* by *o2* on success, or ``NULL`` on
    failure.  This is the equivalent of the Python expression ``o1 >> o2``.
 
 
 .. c:function:: PyObject* PyNumber_And(PyObject *o1, PyObject *o2)
 
-   Returns the "bitwise and" of *o1* and *o2* on success and *NULL* on failure.
+   Returns the "bitwise and" of *o1* and *o2* on success and ``NULL`` on failure.
    This is the equivalent of the Python expression ``o1 & o2``.
 
 
 .. c:function:: PyObject* PyNumber_Xor(PyObject *o1, PyObject *o2)
 
-   Returns the "bitwise exclusive or" of *o1* by *o2* on success, or *NULL* on
+   Returns the "bitwise exclusive or" of *o1* by *o2* on success, or ``NULL`` on
    failure.  This is the equivalent of the Python expression ``o1 ^ o2``.
 
 
 .. c:function:: PyObject* PyNumber_Or(PyObject *o1, PyObject *o2)
 
-   Returns the "bitwise or" of *o1* and *o2* on success, or *NULL* on failure.
+   Returns the "bitwise or" of *o1* and *o2* on success, or ``NULL`` on failure.
    This is the equivalent of the Python expression ``o1 | o2``.
 
 
 .. c:function:: PyObject* PyNumber_InPlaceAdd(PyObject *o1, PyObject *o2)
 
-   Returns the result of adding *o1* and *o2*, or *NULL* on failure.  The operation
+   Returns the result of adding *o1* and *o2*, or ``NULL`` on failure.  The operation
    is done *in-place* when *o1* supports it.  This is the equivalent of the Python
    statement ``o1 += o2``.
 
 
 .. c:function:: PyObject* PyNumber_InPlaceSubtract(PyObject *o1, PyObject *o2)
 
-   Returns the result of subtracting *o2* from *o1*, or *NULL* on failure.  The
+   Returns the result of subtracting *o2* from *o1*, or ``NULL`` on failure.  The
    operation is done *in-place* when *o1* supports it.  This is the equivalent of
    the Python statement ``o1 -= o2``.
 
 
 .. c:function:: PyObject* PyNumber_InPlaceMultiply(PyObject *o1, PyObject *o2)
 
-   Returns the result of multiplying *o1* and *o2*, or *NULL* on failure.  The
+   Returns the result of multiplying *o1* and *o2*, or ``NULL`` on failure.  The
    operation is done *in-place* when *o1* supports it.  This is the equivalent of
    the Python statement ``o1 *= o2``.
 
 
 .. c:function:: PyObject* PyNumber_InPlaceMatrixMultiply(PyObject *o1, PyObject *o2)
 
-   Returns the result of matrix multiplication on *o1* and *o2*, or *NULL* on
+   Returns the result of matrix multiplication on *o1* and *o2*, or ``NULL`` on
    failure.  The operation is done *in-place* when *o1* supports it.  This is
    the equivalent of the Python statement ``o1 @= o2``.
 
@@ -168,7 +168,7 @@ Number Protocol
 
 .. c:function:: PyObject* PyNumber_InPlaceFloorDivide(PyObject *o1, PyObject *o2)
 
-   Returns the mathematical floor of dividing *o1* by *o2*, or *NULL* on failure.
+   Returns the mathematical floor of dividing *o1* by *o2*, or ``NULL`` on failure.
    The operation is done *in-place* when *o1* supports it.  This is the equivalent
    of the Python statement ``o1 //= o2``.
 
@@ -176,7 +176,7 @@ Number Protocol
 .. c:function:: PyObject* PyNumber_InPlaceTrueDivide(PyObject *o1, PyObject *o2)
 
    Return a reasonable approximation for the mathematical value of *o1* divided by
-   *o2*, or *NULL* on failure.  The return value is "approximate" because binary
+   *o2*, or ``NULL`` on failure.  The return value is "approximate" because binary
    floating point numbers are approximate; it is not possible to represent all real
    numbers in base two.  This function can return a floating point value when
    passed two integers.  The operation is done *in-place* when *o1* supports it.
@@ -184,7 +184,7 @@ Number Protocol
 
 .. c:function:: PyObject* PyNumber_InPlaceRemainder(PyObject *o1, PyObject *o2)
 
-   Returns the remainder of dividing *o1* by *o2*, or *NULL* on failure.  The
+   Returns the remainder of dividing *o1* by *o2*, or ``NULL`` on failure.  The
    operation is done *in-place* when *o1* supports it.  This is the equivalent of
    the Python statement ``o1 %= o2``.
 
@@ -193,44 +193,44 @@ Number Protocol
 
    .. index:: builtin: pow
 
-   See the built-in function :func:`pow`. Returns *NULL* on failure.  The operation
+   See the built-in function :func:`pow`. Returns ``NULL`` on failure.  The operation
    is done *in-place* when *o1* supports it.  This is the equivalent of the Python
    statement ``o1 **= o2`` when o3 is :c:data:`Py_None`, or an in-place variant of
    ``pow(o1, o2, o3)`` otherwise. If *o3* is to be ignored, pass :c:data:`Py_None`
-   in its place (passing *NULL* for *o3* would cause an illegal memory access).
+   in its place (passing ``NULL`` for *o3* would cause an illegal memory access).
 
 
 .. c:function:: PyObject* PyNumber_InPlaceLshift(PyObject *o1, PyObject *o2)
 
-   Returns the result of left shifting *o1* by *o2* on success, or *NULL* on
+   Returns the result of left shifting *o1* by *o2* on success, or ``NULL`` on
    failure.  The operation is done *in-place* when *o1* supports it.  This is the
    equivalent of the Python statement ``o1 <<= o2``.
 
 
 .. c:function:: PyObject* PyNumber_InPlaceRshift(PyObject *o1, PyObject *o2)
 
-   Returns the result of right shifting *o1* by *o2* on success, or *NULL* on
+   Returns the result of right shifting *o1* by *o2* on success, or ``NULL`` on
    failure.  The operation is done *in-place* when *o1* supports it.  This is the
    equivalent of the Python statement ``o1 >>= o2``.
 
 
 .. c:function:: PyObject* PyNumber_InPlaceAnd(PyObject *o1, PyObject *o2)
 
-   Returns the "bitwise and" of *o1* and *o2* on success and *NULL* on failure. The
+   Returns the "bitwise and" of *o1* and *o2* on success and ``NULL`` on failure. The
    operation is done *in-place* when *o1* supports it.  This is the equivalent of
    the Python statement ``o1 &= o2``.
 
 
 .. c:function:: PyObject* PyNumber_InPlaceXor(PyObject *o1, PyObject *o2)
 
-   Returns the "bitwise exclusive or" of *o1* by *o2* on success, or *NULL* on
+   Returns the "bitwise exclusive or" of *o1* by *o2* on success, or ``NULL`` on
    failure.  The operation is done *in-place* when *o1* supports it.  This is the
    equivalent of the Python statement ``o1 ^= o2``.
 
 
 .. c:function:: PyObject* PyNumber_InPlaceOr(PyObject *o1, PyObject *o2)
 
-   Returns the "bitwise or" of *o1* and *o2* on success, or *NULL* on failure.  The
+   Returns the "bitwise or" of *o1* and *o2* on success, or ``NULL`` on failure.  The
    operation is done *in-place* when *o1* supports it.  This is the equivalent of
    the Python statement ``o1 |= o2``.
 
@@ -239,7 +239,7 @@ Number Protocol
 
    .. index:: builtin: int
 
-   Returns the *o* converted to an integer object on success, or *NULL* on
+   Returns the *o* converted to an integer object on success, or ``NULL`` on
    failure.  This is the equivalent of the Python expression ``int(o)``.
 
 
@@ -247,13 +247,13 @@ Number Protocol
 
    .. index:: builtin: float
 
-   Returns the *o* converted to a float object on success, or *NULL* on failure.
+   Returns the *o* converted to a float object on success, or ``NULL`` on failure.
    This is the equivalent of the Python expression ``float(o)``.
 
 
 .. c:function:: PyObject* PyNumber_Index(PyObject *o)
 
-   Returns the *o* converted to a Python int on success or *NULL* with a
+   Returns the *o* converted to a Python int on success or ``NULL`` with a
    :exc:`TypeError` exception raised on failure.
 
 
@@ -274,7 +274,7 @@ Number Protocol
    If *o* can be converted to a Python int but the attempt to
    convert to a Py_ssize_t value would raise an :exc:`OverflowError`, then the
    *exc* argument is the type of exception that will be raised (usually
-   :exc:`IndexError` or :exc:`OverflowError`).  If *exc* is *NULL*, then the
+   :exc:`IndexError` or :exc:`OverflowError`).  If *exc* is ``NULL``, then the
    exception is cleared and the value is clipped to *PY_SSIZE_T_MIN* for a negative
    integer or *PY_SSIZE_T_MAX* for a positive integer.
 
diff --git a/Doc/c-api/object.rst b/Doc/c-api/object.rst
index 187a025a67cb6..992dca6e1c8d2 100644
--- a/Doc/c-api/object.rst
+++ b/Doc/c-api/object.rst
@@ -53,14 +53,14 @@ Object Protocol
 .. c:function:: PyObject* PyObject_GetAttr(PyObject *o, PyObject *attr_name)
 
    Retrieve an attribute named *attr_name* from object *o*. Returns the attribute
-   value on success, or *NULL* on failure.  This is the equivalent of the Python
+   value on success, or ``NULL`` on failure.  This is the equivalent of the Python
    expression ``o.attr_name``.
 
 
 .. c:function:: PyObject* PyObject_GetAttrString(PyObject *o, const char *attr_name)
 
    Retrieve an attribute named *attr_name* from object *o*. Returns the attribute
-   value on success, or *NULL* on failure. This is the equivalent of the Python
+   value on success, or ``NULL`` on failure. This is the equivalent of the Python
    expression ``o.attr_name``.
 
 
@@ -81,7 +81,7 @@ Object Protocol
    return ``0`` on success.  This is the equivalent of the Python statement
    ``o.attr_name = v``.
 
-   If *v* is *NULL*, the attribute is deleted, however this feature is
+   If *v* is ``NULL``, the attribute is deleted, however this feature is
    deprecated in favour of using :c:func:`PyObject_DelAttr`.
 
 
@@ -92,7 +92,7 @@ Object Protocol
    return ``0`` on success.  This is the equivalent of the Python statement
    ``o.attr_name = v``.
 
-   If *v* is *NULL*, the attribute is deleted, however this feature is
+   If *v* is ``NULL``, the attribute is deleted, however this feature is
    deprecated in favour of using :c:func:`PyObject_DelAttrString`.
 
 
@@ -143,7 +143,7 @@ Object Protocol
    :const:`Py_NE`, :const:`Py_GT`, or :const:`Py_GE`, corresponding to ``<``,
    ``<=``, ``==``, ``!=``, ``>``, or ``>=`` respectively. This is the equivalent of
    the Python expression ``o1 op o2``, where ``op`` is the operator corresponding
-   to *opid*. Returns the value of the comparison on success, or *NULL* on failure.
+   to *opid*. Returns the value of the comparison on success, or ``NULL`` on failure.
 
 
 .. c:function:: int PyObject_RichCompareBool(PyObject *o1, PyObject *o2, int opid)
@@ -165,7 +165,7 @@ Object Protocol
    .. index:: builtin: repr
 
    Compute a string representation of object *o*.  Returns the string
-   representation on success, *NULL* on failure.  This is the equivalent of the
+   representation on success, ``NULL`` on failure.  This is the equivalent of the
    Python expression ``repr(o)``.  Called by the :func:`repr` built-in function.
 
    .. versionchanged:: 3.4
@@ -188,7 +188,7 @@ Object Protocol
 .. c:function:: PyObject* PyObject_Str(PyObject *o)
 
    Compute a string representation of object *o*.  Returns the string
-   representation on success, *NULL* on failure.  This is the equivalent of the
+   representation on success, ``NULL`` on failure.  This is the equivalent of the
    Python expression ``str(o)``.  Called by the :func:`str` built-in function
    and, therefore, by the :func:`print` function.
 
@@ -200,7 +200,7 @@ Object Protocol
 
    .. index:: builtin: bytes
 
-   Compute a bytes representation of object *o*.  *NULL* is returned on
+   Compute a bytes representation of object *o*.  ``NULL`` is returned on
    failure and a bytes object on success.  This is equivalent to the Python
    expression ``bytes(o)``, when *o* is not an integer.  Unlike ``bytes(o)``,
    a TypeError is raised when *o* is an integer instead of a zero-initialized
@@ -258,11 +258,11 @@ Object Protocol
    Call a callable Python object *callable*, with arguments given by the
    tuple *args*, and named arguments given by the dictionary *kwargs*.
 
-   *args* must not be *NULL*, use an empty tuple if no arguments are needed.
-   If no named arguments are needed, *kwargs* can be *NULL*.
+   *args* must not be ``NULL``, use an empty tuple if no arguments are needed.
+   If no named arguments are needed, *kwargs* can be ``NULL``.
 
    Return the result of the call on success, or raise an exception and return
-   *NULL* on failure.
+   ``NULL`` on failure.
 
    This is the equivalent of the Python expression:
    ``callable(*args, **kwargs)``.
@@ -271,10 +271,10 @@ Object Protocol
 .. c:function:: PyObject* PyObject_CallObject(PyObject *callable, PyObject *args)
 
    Call a callable Python object *callable*, with arguments given by the
-   tuple *args*.  If no arguments are needed, then *args* can be *NULL*.
+   tuple *args*.  If no arguments are needed, then *args* can be ``NULL``.
 
    Return the result of the call on success, or raise an exception and return
-   *NULL* on failure.
+   ``NULL`` on failure.
 
    This is the equivalent of the Python expression: ``callable(*args)``.
 
@@ -283,10 +283,10 @@ Object Protocol
 
    Call a callable Python object *callable*, with a variable number of C arguments.
    The C arguments are described using a :c:func:`Py_BuildValue` style format
-   string.  The format can be *NULL*, indicating that no arguments are provided.
+   string.  The format can be ``NULL``, indicating that no arguments are provided.
 
    Return the result of the call on success, or raise an exception and return
-   *NULL* on failure.
+   ``NULL`` on failure.
 
    This is the equivalent of the Python expression: ``callable(*args)``.
 
@@ -303,10 +303,10 @@ Object Protocol
    arguments.  The C arguments are described by a :c:func:`Py_BuildValue` format
    string that should  produce a tuple.
 
-   The format can be *NULL*, indicating that no arguments are provided.
+   The format can be ``NULL``, indicating that no arguments are provided.
 
    Return the result of the call on success, or raise an exception and return
-   *NULL* on failure.
+   ``NULL`` on failure.
 
    This is the equivalent of the Python expression:
    ``obj.name(arg1, arg2, ...)``.
@@ -322,10 +322,10 @@ Object Protocol
 
    Call a callable Python object *callable*, with a variable number of
    :c:type:`PyObject\*` arguments.  The arguments are provided as a variable number
-   of parameters followed by *NULL*.
+   of parameters followed by ``NULL``.
 
    Return the result of the call on success, or raise an exception and return
-   *NULL* on failure.
+   ``NULL`` on failure.
 
    This is the equivalent of the Python expression:
    ``callable(arg1, arg2, ...)``.
@@ -336,10 +336,10 @@ Object Protocol
    Calls a method of the Python object *obj*, where the name of the method is given as a
    Python string object in *name*.  It is called with a variable number of
    :c:type:`PyObject\*` arguments.  The arguments are provided as a variable number
-   of parameters followed by *NULL*.
+   of parameters followed by ``NULL``.
 
    Return the result of the call on success, or raise an exception and return
-   *NULL* on failure.
+   ``NULL`` on failure.
 
 
 .. c:function:: PyObject* _PyObject_Vectorcall(PyObject *callable, PyObject *const *args, size_t nargsf, PyObject *kwnames)
@@ -363,7 +363,7 @@ Object Protocol
    and all keys must be unique.
 
    Return the result of the call on success, or raise an exception and return
-   *NULL* on failure.
+   ``NULL`` on failure.
 
    This uses the vectorcall protocol if the callable supports it;
    otherwise, the arguments are converted to use
@@ -402,7 +402,7 @@ Object Protocol
 .. c:function:: PyObject* _PyObject_FastCallDict(PyObject *callable, PyObject *const *args, size_t nargsf, PyObject *kwdict)
 
    Same as :c:func:`_PyObject_Vectorcall` except that the keyword arguments
-   are passed as a dictionary in *kwdict*. This may be *NULL* if there
+   are passed as a dictionary in *kwdict*. This may be ``NULL`` if there
    are no keyword arguments.
 
    For callables supporting :c:data:`vectorcall <PyTypeObject.tp_vectorcall_offset>`,
@@ -458,8 +458,8 @@ Object Protocol
 
    .. index:: builtin: type
 
-   When *o* is non-*NULL*, returns a type object corresponding to the object type
-   of object *o*. On failure, raises :exc:`SystemError` and returns *NULL*.  This
+   When *o* is non-``NULL``, returns a type object corresponding to the object type
+   of object *o*. On failure, raises :exc:`SystemError` and returns ``NULL``.  This
    is equivalent to the Python expression ``type(o)``. This function increments the
    reference count of the return value. There's really no reason to use this
    function instead of the common expression ``o->ob_type``, which returns a
@@ -470,7 +470,7 @@ Object Protocol
 .. c:function:: int PyObject_TypeCheck(PyObject *o, PyTypeObject *type)
 
    Return true if the object *o* is of type *type* or a subtype of *type*.  Both
-   parameters must be non-*NULL*.
+   parameters must be non-``NULL``.
 
 
 .. c:function:: Py_ssize_t PyObject_Size(PyObject *o)
@@ -495,7 +495,7 @@ Object Protocol
 
 .. c:function:: PyObject* PyObject_GetItem(PyObject *o, PyObject *key)
 
-   Return element of *o* corresponding to the object *key* or *NULL* on failure.
+   Return element of *o* corresponding to the object *key* or ``NULL`` on failure.
    This is the equivalent of the Python expression ``o[key]``.
 
 
@@ -515,15 +515,15 @@ Object Protocol
 .. c:function:: PyObject* PyObject_Dir(PyObject *o)
 
    This is equivalent to the Python expression ``dir(o)``, returning a (possibly
-   empty) list of strings appropriate for the object argument, or *NULL* if there
-   was an error.  If the argument is *NULL*, this is like the Python ``dir()``,
+   empty) list of strings appropriate for the object argument, or ``NULL`` if there
+   was an error.  If the argument is ``NULL``, this is like the Python ``dir()``,
    returning the names of the current locals; in this case, if no execution frame
-   is active then *NULL* is returned but :c:func:`PyErr_Occurred` will return false.
+   is active then ``NULL`` is returned but :c:func:`PyErr_Occurred` will return false.
 
 
 .. c:function:: PyObject* PyObject_GetIter(PyObject *o)
 
    This is equivalent to the Python expression ``iter(o)``. It returns a new
    iterator for the object argument, or the object  itself if the object is already
-   an iterator.  Raises :exc:`TypeError` and returns *NULL* if the object cannot be
+   an iterator.  Raises :exc:`TypeError` and returns ``NULL`` if the object cannot be
    iterated.
diff --git a/Doc/c-api/refcounting.rst b/Doc/c-api/refcounting.rst
index 6b07c87d62ede..0df12c67f40bc 100644
--- a/Doc/c-api/refcounting.rst
+++ b/Doc/c-api/refcounting.rst
@@ -13,22 +13,22 @@ objects.
 
 .. c:function:: void Py_INCREF(PyObject *o)
 
-   Increment the reference count for object *o*.  The object must not be *NULL*; if
-   you aren't sure that it isn't *NULL*, use :c:func:`Py_XINCREF`.
+   Increment the reference count for object *o*.  The object must not be ``NULL``; if
+   you aren't sure that it isn't ``NULL``, use :c:func:`Py_XINCREF`.
 
 
 .. c:function:: void Py_XINCREF(PyObject *o)
 
-   Increment the reference count for object *o*.  The object may be *NULL*, in
+   Increment the reference count for object *o*.  The object may be ``NULL``, in
    which case the macro has no effect.
 
 
 .. c:function:: void Py_DECREF(PyObject *o)
 
-   Decrement the reference count for object *o*.  The object must not be *NULL*; if
-   you aren't sure that it isn't *NULL*, use :c:func:`Py_XDECREF`.  If the reference
+   Decrement the reference count for object *o*.  The object must not be ``NULL``; if
+   you aren't sure that it isn't ``NULL``, use :c:func:`Py_XDECREF`.  If the reference
    count reaches zero, the object's type's deallocation function (which must not be
-   *NULL*) is invoked.
+   ``NULL``) is invoked.
 
    .. warning::
 
@@ -44,18 +44,18 @@ objects.
 
 .. c:function:: void Py_XDECREF(PyObject *o)
 
-   Decrement the reference count for object *o*.  The object may be *NULL*, in
+   Decrement the reference count for object *o*.  The object may be ``NULL``, in
    which case the macro has no effect; otherwise the effect is the same as for
    :c:func:`Py_DECREF`, and the same warning applies.
 
 
 .. c:function:: void Py_CLEAR(PyObject *o)
 
-   Decrement the reference count for object *o*.  The object may be *NULL*, in
+   Decrement the reference count for object *o*.  The object may be ``NULL``, in
    which case the macro has no effect; otherwise the effect is the same as for
-   :c:func:`Py_DECREF`, except that the argument is also set to *NULL*.  The warning
+   :c:func:`Py_DECREF`, except that the argument is also set to ``NULL``.  The warning
    for :c:func:`Py_DECREF` does not apply with respect to the object passed because
-   the macro carefully uses a temporary variable and sets the argument to *NULL*
+   the macro carefully uses a temporary variable and sets the argument to ``NULL``
    before decrementing its reference count.
 
    It is a good idea to use this macro whenever decrementing the reference
diff --git a/Doc/c-api/reflection.rst b/Doc/c-api/reflection.rst
index 080ea3222cfbb..1d86de66ed3ed 100644
--- a/Doc/c-api/reflection.rst
+++ b/Doc/c-api/reflection.rst
@@ -14,18 +14,18 @@ Reflection
 .. c:function:: PyObject* PyEval_GetLocals()
 
    Return a dictionary of the local variables in the current execution frame,
-   or *NULL* if no frame is currently executing.
+   or ``NULL`` if no frame is currently executing.
 
 
 .. c:function:: PyObject* PyEval_GetGlobals()
 
    Return a dictionary of the global variables in the current execution frame,
-   or *NULL* if no frame is currently executing.
+   or ``NULL`` if no frame is currently executing.
 
 
 .. c:function:: PyFrameObject* PyEval_GetFrame()
 
-   Return the current thread state's frame, which is *NULL* if no frame is
+   Return the current thread state's frame, which is ``NULL`` if no frame is
    currently executing.
 
 
diff --git a/Doc/c-api/sequence.rst b/Doc/c-api/sequence.rst
index fe1c252ce9ba3..6581885904117 100644
--- a/Doc/c-api/sequence.rst
+++ b/Doc/c-api/sequence.rst
@@ -26,39 +26,39 @@ Sequence Protocol
 
 .. c:function:: PyObject* PySequence_Concat(PyObject *o1, PyObject *o2)
 
-   Return the concatenation of *o1* and *o2* on success, and *NULL* on failure.
+   Return the concatenation of *o1* and *o2* on success, and ``NULL`` on failure.
    This is the equivalent of the Python expression ``o1 + o2``.
 
 
 .. c:function:: PyObject* PySequence_Repeat(PyObject *o, Py_ssize_t count)
 
-   Return the result of repeating sequence object *o* *count* times, or *NULL* on
+   Return the result of repeating sequence object *o* *count* times, or ``NULL`` on
    failure.  This is the equivalent of the Python expression ``o * count``.
 
 
 .. c:function:: PyObject* PySequence_InPlaceConcat(PyObject *o1, PyObject *o2)
 
-   Return the concatenation of *o1* and *o2* on success, and *NULL* on failure.
+   Return the concatenation of *o1* and *o2* on success, and ``NULL`` on failure.
    The operation is done *in-place* when *o1* supports it.  This is the equivalent
    of the Python expression ``o1 += o2``.
 
 
 .. c:function:: PyObject* PySequence_InPlaceRepeat(PyObject *o, Py_ssize_t count)
 
-   Return the result of repeating sequence object *o* *count* times, or *NULL* on
+   Return the result of repeating sequence object *o* *count* times, or ``NULL`` on
    failure.  The operation is done *in-place* when *o* supports it.  This is the
    equivalent of the Python expression ``o *= count``.
 
 
 .. c:function:: PyObject* PySequence_GetItem(PyObject *o, Py_ssize_t i)
 
-   Return the *i*\ th element of *o*, or *NULL* on failure. This is the equivalent of
+   Return the *i*\ th element of *o*, or ``NULL`` on failure. This is the equivalent of
    the Python expression ``o[i]``.
 
 
 .. c:function:: PyObject* PySequence_GetSlice(PyObject *o, Py_ssize_t i1, Py_ssize_t i2)
 
-   Return the slice of sequence object *o* between *i1* and *i2*, or *NULL* on
+   Return the slice of sequence object *o* between *i1* and *i2*, or ``NULL`` on
    failure. This is the equivalent of the Python expression ``o[i1:i2]``.
 
 
@@ -69,7 +69,7 @@ Sequence Protocol
    is the equivalent of the Python statement ``o[i] = v``.  This function *does
    not* steal a reference to *v*.
 
-   If *v* is *NULL*, the element is deleted, however this feature is
+   If *v* is ``NULL``, the element is deleted, however this feature is
    deprecated in favour of using :c:func:`PySequence_DelItem`.
 
 
@@ -114,7 +114,7 @@ Sequence Protocol
 .. c:function:: PyObject* PySequence_List(PyObject *o)
 
    Return a list object with the same contents as the sequence or iterable *o*,
-   or *NULL* on failure.  The returned list is guaranteed to be new.  This is
+   or ``NULL`` on failure.  The returned list is guaranteed to be new.  This is
    equivalent to the Python expression ``list(o)``.
 
 
@@ -123,7 +123,7 @@ Sequence Protocol
    .. index:: builtin: tuple
 
    Return a tuple object with the same contents as the sequence or iterable *o*,
-   or *NULL* on failure.  If *o* is a tuple, a new reference will be returned,
+   or ``NULL`` on failure.  If *o* is a tuple, a new reference will be returned,
    otherwise a tuple will be constructed with the appropriate contents.  This is
    equivalent to the Python expression ``tuple(o)``.
 
@@ -133,7 +133,7 @@ Sequence Protocol
    Return the sequence or iterable *o* as an object usable by the other
    ``PySequence_Fast*`` family of functions. If the object is not a sequence or
    iterable, raises :exc:`TypeError` with *m* as the message text. Returns
-   *NULL* on failure.
+   ``NULL`` on failure.
 
    The ``PySequence_Fast*`` functions are thus named because they assume
    *o* is a :c:type:`PyTupleObject` or a :c:type:`PyListObject` and access
@@ -146,7 +146,7 @@ Sequence Protocol
 .. c:function:: Py_ssize_t PySequence_Fast_GET_SIZE(PyObject *o)
 
    Returns the length of *o*, assuming that *o* was returned by
-   :c:func:`PySequence_Fast` and that *o* is not *NULL*.  The size can also be
+   :c:func:`PySequence_Fast` and that *o* is not ``NULL``.  The size can also be
    gotten by calling :c:func:`PySequence_Size` on *o*, but
    :c:func:`PySequence_Fast_GET_SIZE` is faster because it can assume *o* is a
    list or tuple.
@@ -155,13 +155,13 @@ Sequence Protocol
 .. c:function:: PyObject* PySequence_Fast_GET_ITEM(PyObject *o, Py_ssize_t i)
 
    Return the *i*\ th element of *o*, assuming that *o* was returned by
-   :c:func:`PySequence_Fast`, *o* is not *NULL*, and that *i* is within bounds.
+   :c:func:`PySequence_Fast`, *o* is not ``NULL``, and that *i* is within bounds.
 
 
 .. c:function:: PyObject** PySequence_Fast_ITEMS(PyObject *o)
 
    Return the underlying array of PyObject pointers.  Assumes that *o* was returned
-   by :c:func:`PySequence_Fast` and *o* is not *NULL*.
+   by :c:func:`PySequence_Fast` and *o* is not ``NULL``.
 
    Note, if a list gets resized, the reallocation may relocate the items array.
    So, only use the underlying array pointer in contexts where the sequence
@@ -170,7 +170,7 @@ Sequence Protocol
 
 .. c:function:: PyObject* PySequence_ITEM(PyObject *o, Py_ssize_t i)
 
-   Return the *i*\ th element of *o* or *NULL* on failure. Faster form of
+   Return the *i*\ th element of *o* or ``NULL`` on failure. Faster form of
    :c:func:`PySequence_GetItem` but without checking that
    :c:func:`PySequence_Check` on *o* is true and without adjustment for negative
    indices.
diff --git a/Doc/c-api/set.rst b/Doc/c-api/set.rst
index 074fcb877e1da..54819e8fd6cbd 100644
--- a/Doc/c-api/set.rst
+++ b/Doc/c-api/set.rst
@@ -80,8 +80,8 @@ the constructor functions work with any iterable Python object.
 .. c:function:: PyObject* PySet_New(PyObject *iterable)
 
    Return a new :class:`set` containing objects returned by the *iterable*.  The
-   *iterable* may be *NULL* to create a new empty set.  Return the new set on
-   success or *NULL* on failure.  Raise :exc:`TypeError` if *iterable* is not
+   *iterable* may be ``NULL`` to create a new empty set.  Return the new set on
+   success or ``NULL`` on failure.  Raise :exc:`TypeError` if *iterable* is not
    actually iterable.  The constructor is also useful for copying a set
    (``c=set(s)``).
 
@@ -89,8 +89,8 @@ the constructor functions work with any iterable Python object.
 .. c:function:: PyObject* PyFrozenSet_New(PyObject *iterable)
 
    Return a new :class:`frozenset` containing objects returned by the *iterable*.
-   The *iterable* may be *NULL* to create a new empty frozenset.  Return the new
-   set on success or *NULL* on failure.  Raise :exc:`TypeError` if *iterable* is
+   The *iterable* may be ``NULL`` to create a new empty frozenset.  Return the new
+   set on success or ``NULL`` on failure.  Raise :exc:`TypeError` if *iterable* is
    not actually iterable.
 
 
@@ -149,7 +149,7 @@ subtypes but not for instances of :class:`frozenset` or its subtypes.
 .. c:function:: PyObject* PySet_Pop(PyObject *set)
 
    Return a new reference to an arbitrary object in the *set*, and removes the
-   object from the *set*.  Return *NULL* on failure.  Raise :exc:`KeyError` if the
+   object from the *set*.  Return ``NULL`` on failure.  Raise :exc:`KeyError` if the
    set is empty. Raise a :exc:`SystemError` if *set* is not an instance of
    :class:`set` or its subtype.
 
diff --git a/Doc/c-api/slice.rst b/Doc/c-api/slice.rst
index d924308890b8d..48a58c6c6f7e3 100644
--- a/Doc/c-api/slice.rst
+++ b/Doc/c-api/slice.rst
@@ -14,15 +14,15 @@ Slice Objects
 
 .. c:function:: int PySlice_Check(PyObject *ob)
 
-   Return true if *ob* is a slice object; *ob* must not be *NULL*.
+   Return true if *ob* is a slice object; *ob* must not be ``NULL``.
 
 
 .. c:function:: PyObject* PySlice_New(PyObject *start, PyObject *stop, PyObject *step)
 
    Return a new slice object with the given values.  The *start*, *stop*, and
    *step* parameters are used as the values of the slice object attributes of
-   the same names.  Any of the values may be *NULL*, in which case the
-   ``None`` will be used for the corresponding attribute.  Return *NULL* if
+   the same names.  Any of the values may be ``NULL``, in which case the
+   ``None`` will be used for the corresponding attribute.  Return ``NULL`` if
    the new object could not be allocated.
 
 
diff --git a/Doc/c-api/structures.rst b/Doc/c-api/structures.rst
index 5184ad511cd9b..2dc0762ee7fb8 100644
--- a/Doc/c-api/structures.rst
+++ b/Doc/c-api/structures.rst
@@ -106,8 +106,8 @@ the definition of all other Python objects.
 
    Type of the functions used to implement most Python callables in C.
    Functions of this type take two :c:type:`PyObject\*` parameters and return
-   one such value.  If the return value is *NULL*, an exception shall have
-   been set.  If not *NULL*, the return value is interpreted as the return
+   one such value.  If the return value is ``NULL``, an exception shall have
+   been set.  If not ``NULL``, the return value is interpreted as the return
    value of the function as exposed in Python.  The function must return a new
    reference.
 
@@ -179,7 +179,7 @@ also keyword arguments.  So there are a total of 6 calling conventions:
 
    Methods with these flags must be of type :c:type:`PyCFunctionWithKeywords`.
    The function expects three parameters: *self*, *args*, *kwargs* where
-   *kwargs* is a dictionary of all the keyword arguments or possibly *NULL*
+   *kwargs* is a dictionary of all the keyword arguments or possibly ``NULL``
    if there are no keyword arguments.  The parameters are typically processed
    using :c:func:`PyArg_ParseTupleAndKeywords`.
 
@@ -204,7 +204,7 @@ also keyword arguments.  So there are a total of 6 calling conventions:
    Keyword arguments are passed the same way as in the vectorcall protocol:
    there is an additional fourth :c:type:`PyObject\*` parameter
    which is a tuple representing the names of the keyword arguments
-   or possibly *NULL* if there are no keywords.  The values of the keyword
+   or possibly ``NULL`` if there are no keywords.  The values of the keyword
    arguments are stored in the *args* array, after the positional arguments.
 
    This is not part of the :ref:`limited API <stable>`.
@@ -218,7 +218,7 @@ also keyword arguments.  So there are a total of 6 calling conventions:
    they are listed with the :const:`METH_NOARGS` flag.  They need to be of type
    :c:type:`PyCFunction`.  The first parameter is typically named *self* and will
    hold a reference to the module or object instance.  In all cases the second
-   parameter will be *NULL*.
+   parameter will be ``NULL``.
 
 
 .. data:: METH_O
@@ -249,7 +249,7 @@ method.
 
    .. index:: builtin: staticmethod
 
-   The method will be passed *NULL* as the first parameter rather than an
+   The method will be passed ``NULL`` as the first parameter rather than an
    instance of the type.  This is used to create *static methods*, similar to
    what is created when using the :func:`staticmethod` built-in function.
 
@@ -323,7 +323,7 @@ definition with the same method name.
    =============== ==================
 
    :c:macro:`T_OBJECT` and :c:macro:`T_OBJECT_EX` differ in that
-   :c:macro:`T_OBJECT` returns ``None`` if the member is *NULL* and
+   :c:macro:`T_OBJECT` returns ``None`` if the member is ``NULL`` and
    :c:macro:`T_OBJECT_EX` raises an :exc:`AttributeError`.  Try to use
    :c:macro:`T_OBJECT_EX` over :c:macro:`T_OBJECT` because :c:macro:`T_OBJECT_EX`
    handles use of the :keyword:`del` statement on that attribute more correctly
@@ -333,7 +333,7 @@ definition with the same method name.
    read-only access.  Using :c:macro:`T_STRING` for :attr:`type` implies
    :c:macro:`READONLY`.  :c:macro:`T_STRING` data is interpreted as UTF-8.
    Only :c:macro:`T_OBJECT` and :c:macro:`T_OBJECT_EX`
-   members can be deleted.  (They are set to *NULL*).
+   members can be deleted.  (They are set to ``NULL``).
 
 
 .. c:type:: PyGetSetDef
@@ -364,7 +364,7 @@ definition with the same method name.
 
       typedef PyObject *(*getter)(PyObject *, void *);
 
-   It should return a new reference on success or *NULL* with a set exception
+   It should return a new reference on success or ``NULL`` with a set exception
    on failure.
 
    ``set`` functions take two :c:type:`PyObject\*` parameters (the instance and
@@ -372,5 +372,5 @@ definition with the same method name.
 
       typedef int (*setter)(PyObject *, PyObject *, void *);
 
-   In case the attribute should be deleted the second parameter is *NULL*.
+   In case the attribute should be deleted the second parameter is ``NULL``.
    Should return ``0`` on success or ``-1`` with a set exception on failure.
diff --git a/Doc/c-api/sys.rst b/Doc/c-api/sys.rst
index 87d2bd13d31b8..212f4d1066c9c 100644
--- a/Doc/c-api/sys.rst
+++ b/Doc/c-api/sys.rst
@@ -22,7 +22,7 @@ Operating System Utilities
    Return true (nonzero) if the standard I/O file *fp* with name *filename* is
    deemed interactive.  This is the case for files for which ``isatty(fileno(fp))``
    is true.  If the global flag :c:data:`Py_InteractiveFlag` is true, this function
-   also returns true if the *filename* pointer is *NULL* or if the name is equal to
+   also returns true if the *filename* pointer is ``NULL`` or if the name is equal to
    one of the strings ``'<stdin>'`` or ``'???'``.
 
 
@@ -201,12 +201,12 @@ accessible to C code.  They all work with the current interpreter thread's
 
 .. c:function:: PyObject *PySys_GetObject(const char *name)
 
-   Return the object *name* from the :mod:`sys` module or *NULL* if it does
+   Return the object *name* from the :mod:`sys` module or ``NULL`` if it does
    not exist, without setting an exception.
 
 .. c:function:: int PySys_SetObject(const char *name, PyObject *v)
 
-   Set *name* in the :mod:`sys` module to *v* unless *v* is *NULL*, in which
+   Set *name* in the :mod:`sys` module to *v* unless *v* is ``NULL``, in which
    case *name* is deleted from the sys module. Returns ``0`` on success, ``-1``
    on error.
 
@@ -283,7 +283,7 @@ accessible to C code.  They all work with the current interpreter thread's
 .. c:function:: PyObject *PySys_GetXOptions()
 
    Return the current dictionary of :option:`-X` options, similarly to
-   :data:`sys._xoptions`.  On error, *NULL* is returned and an exception is
+   :data:`sys._xoptions`.  On error, ``NULL`` is returned and an exception is
    set.
 
    .. versionadded:: 3.2
diff --git a/Doc/c-api/tuple.rst b/Doc/c-api/tuple.rst
index a0d53e2030e63..d74d8626d1e92 100644
--- a/Doc/c-api/tuple.rst
+++ b/Doc/c-api/tuple.rst
@@ -33,12 +33,12 @@ Tuple Objects
 
 .. c:function:: PyObject* PyTuple_New(Py_ssize_t len)
 
-   Return a new tuple object of size *len*, or *NULL* on failure.
+   Return a new tuple object of size *len*, or ``NULL`` on failure.
 
 
 .. c:function:: PyObject* PyTuple_Pack(Py_ssize_t n, ...)
 
-   Return a new tuple object of size *n*, or *NULL* on failure. The tuple values
+   Return a new tuple object of size *n*, or ``NULL`` on failure. The tuple values
    are initialized to the subsequent *n* C arguments pointing to Python objects.
    ``PyTuple_Pack(2, a, b)`` is equivalent to ``Py_BuildValue("(OO)", a, b)``.
 
@@ -50,14 +50,14 @@ Tuple Objects
 
 .. c:function:: Py_ssize_t PyTuple_GET_SIZE(PyObject *p)
 
-   Return the size of the tuple *p*, which must be non-*NULL* and point to a tuple;
+   Return the size of the tuple *p*, which must be non-``NULL`` and point to a tuple;
    no error checking is performed.
 
 
 .. c:function:: PyObject* PyTuple_GetItem(PyObject *p, Py_ssize_t pos)
 
    Return the object at position *pos* in the tuple pointed to by *p*.  If *pos* is
-   out of bounds, return *NULL* and set an :exc:`IndexError` exception.
+   out of bounds, return ``NULL`` and set an :exc:`IndexError` exception.
 
 
 .. c:function:: PyObject* PyTuple_GET_ITEM(PyObject *p, Py_ssize_t pos)
@@ -68,7 +68,7 @@ Tuple Objects
 .. c:function:: PyObject* PyTuple_GetSlice(PyObject *p, Py_ssize_t low, Py_ssize_t high)
 
    Return the slice of the tuple pointed to by *p* between *low* and *high*,
-   or *NULL* on failure.  This is the equivalent of the Python expression
+   or ``NULL`` on failure.  This is the equivalent of the Python expression
    ``p[low:high]``.  Indexing from the end of the list is not supported.
 
 
@@ -107,7 +107,7 @@ Tuple Objects
    only more efficiently.  Returns ``0`` on success. Client code should never
    assume that the resulting value of ``*p`` will be the same as before calling
    this function. If the object referenced by ``*p`` is replaced, the original
-   ``*p`` is destroyed.  On failure, returns ``-1`` and sets ``*p`` to *NULL*, and
+   ``*p`` is destroyed.  On failure, returns ``-1`` and sets ``*p`` to ``NULL``, and
    raises :exc:`MemoryError` or :exc:`SystemError`.
 
 
@@ -147,20 +147,20 @@ type.
 
    Contains the meta information of a struct sequence type to create.
 
-   +-------------------+------------------------------+------------------------------------+
-   | Field             | C Type                       | Meaning                            |
-   +===================+==============================+====================================+
-   | ``name``          | ``const char *``             | name of the struct sequence type   |
-   +-------------------+------------------------------+------------------------------------+
-   | ``doc``           | ``const char *``             | pointer to docstring for the type  |
-   |                   |                              | or NULL to omit                    |
-   +-------------------+------------------------------+------------------------------------+
-   | ``fields``        | ``PyStructSequence_Field *`` | pointer to *NULL*-terminated array |
-   |                   |                              | with field names of the new type   |
-   +-------------------+------------------------------+------------------------------------+
-   | ``n_in_sequence`` | ``int``                      | number of fields visible to the    |
-   |                   |                              | Python side (if used as tuple)     |
-   +-------------------+------------------------------+------------------------------------+
+   +-------------------+------------------------------+--------------------------------------+
+   | Field             | C Type                       | Meaning                              |
+   +===================+==============================+======================================+
+   | ``name``          | ``const char *``             | name of the struct sequence type     |
+   +-------------------+------------------------------+--------------------------------------+
+   | ``doc``           | ``const char *``             | pointer to docstring for the type    |
+   |                   |                              | or NULL to omit                      |
+   +-------------------+------------------------------+--------------------------------------+
+   | ``fields``        | ``PyStructSequence_Field *`` | pointer to ``NULL``-terminated array |
+   |                   |                              | with field names of the new type     |
+   +-------------------+------------------------------+--------------------------------------+
+   | ``n_in_sequence`` | ``int``                      | number of fields visible to the      |
+   |                   |                              | Python side (if used as tuple)       |
+   +-------------------+------------------------------+--------------------------------------+
 
 
 .. c:type:: PyStructSequence_Field
@@ -170,16 +170,16 @@ type.
    :attr:`fields` array of the :c:type:`PyStructSequence_Desc` determines which
    field of the struct sequence is described.
 
-   +-----------+------------------+--------------------------------------+
-   | Field     | C Type           | Meaning                              |
-   +===========+==================+======================================+
-   | ``name``  | ``const char *`` | name for the field or *NULL* to end  |
-   |           |                  | the list of named fields, set to     |
-   |           |                  | PyStructSequence_UnnamedField to     |
-   |           |                  | leave unnamed                        |
-   +-----------+------------------+--------------------------------------+
-   | ``doc``   | ``const char *`` | field docstring or *NULL* to omit    |
-   +-----------+------------------+--------------------------------------+
+   +-----------+------------------+----------------------------------------+
+   | Field     | C Type           | Meaning                                |
+   +===========+==================+========================================+
+   | ``name``  | ``const char *`` | name for the field or ``NULL`` to end  |
+   |           |                  | the list of named fields, set to       |
+   |           |                  | PyStructSequence_UnnamedField to       |
+   |           |                  | leave unnamed                          |
+   +-----------+------------------+----------------------------------------+
+   | ``doc``   | ``const char *`` | field docstring or ``NULL`` to omit    |
+   +-----------+------------------+----------------------------------------+
 
 
 .. c:var:: char* PyStructSequence_UnnamedField
diff --git a/Doc/c-api/type.rst b/Doc/c-api/type.rst
index 239355a802413..0bb806d21421b 100644
--- a/Doc/c-api/type.rst
+++ b/Doc/c-api/type.rst
@@ -81,7 +81,7 @@ Type Objects
 
    Generic handler for the :c:member:`~PyTypeObject.tp_alloc` slot of a type object.  Use
    Python's default memory allocation mechanism to allocate a new instance and
-   initialize all its contents to *NULL*.
+   initialize all its contents to ``NULL``.
 
 .. c:function:: PyObject* PyType_GenericNew(PyTypeObject *type, PyObject *args, PyObject *kwds)
 
@@ -98,7 +98,7 @@ Type Objects
 .. c:function:: void* PyType_GetSlot(PyTypeObject *type, int slot)
 
    Return the function pointer stored in the given slot. If the
-   result is *NULL*, this indicates that either the slot is *NULL*,
+   result is ``NULL``, this indicates that either the slot is ``NULL``,
    or that the function was called with invalid parameters.
    Callers will typically cast the result pointer into the appropriate
    function type.
@@ -124,8 +124,8 @@ The following functions and structs are used to create
    If *bases* is a tuple, the created heap type contains all types contained
    in it as base types.
 
-   If *bases* is *NULL*, the *Py_tp_base* slot is used instead.
-   If that also is *NULL*, the new type derives from :class:`object`.
+   If *bases* is ``NULL``, the *Py_tp_base* slot is used instead.
+   If that also is ``NULL``, the new type derives from :class:`object`.
 
    This function calls :c:func:`PyType_Ready` on the new type.
 
@@ -203,4 +203,4 @@ The following functions and structs are used to create
       The desired value of the slot. In most cases, this is a pointer
       to a function.
 
-      May not be *NULL*.
+      May not be ``NULL``.
diff --git a/Doc/c-api/typeobj.rst b/Doc/c-api/typeobj.rst
index bb76750074309..181157e08ec43 100644
--- a/Doc/c-api/typeobj.rst
+++ b/Doc/c-api/typeobj.rst
@@ -159,20 +159,20 @@ fields exist as well:
    A slot name in parentheses indicates it is (effectively) deprecated.
    Names in angle brackets should be treated as read-only.
    Names in square brackets are for internal use only.
-   "<R>" (as a prefix) means the field is required (must be non-*NULL*).
+   "<R>" (as a prefix) means the field is required (must be non-``NULL``).
 .. [#cols] Columns:
 
    **"O"**:  set on :c:type:`PyBaseObject_Type`
 
    **"T"**:  set on :c:type:`PyType_Type`
 
-   **"D"**:  default (if slot is set to *NULL*)
+   **"D"**:  default (if slot is set to ``NULL``)
 
    .. code-block:: none
 
-      X - *PyType_Ready* sets this value if it is *NULL*
-      ~ - *PyType_Ready* always sets this value (it should be *NULL*)
-      ? - *PyType_Ready* may set this value depending on other slots
+      X - PyType_Ready sets this value if it is NULL
+      ~ - PyType_Ready always sets this value (it should be NULL)
+      ? - PyType_Ready may set this value depending on other slots
 
       Also see the inheritance column ("I").
 
@@ -180,7 +180,7 @@ fields exist as well:
 
    .. code-block:: none
 
-      X - type slot is inherited via *PyType_Ready* if defined with a *NULL* value
+      X - type slot is inherited via PyType_Ready if defined with a NULL value
       % - the slots of the sub-struct are inherited individually
       G - inherited, but only in combination with other slots; see the slot's description
       ? - it's complicated; see the slot's description
@@ -491,8 +491,8 @@ type objects) *must* have the :attr:`ob_size` field.
              PyObject* PyObject._ob_prev
 
    These fields are only present when the macro ``Py_TRACE_REFS`` is defined.
-   Their initialization to *NULL* is taken care of by the ``PyObject_HEAD_INIT``
-   macro.  For statically allocated objects, these fields always remain *NULL*.
+   Their initialization to ``NULL`` is taken care of by the ``PyObject_HEAD_INIT``
+   macro.  For statically allocated objects, these fields always remain ``NULL``.
    For dynamically allocated objects, these two fields are used to link the object
    into a doubly-linked list of *all* live objects on the heap.  This could be used
    for various debugging purposes; currently the only use is to print the objects
@@ -523,7 +523,7 @@ type objects) *must* have the :attr:`ob_size` field.
    argument to the ``PyObject_HEAD_INIT`` macro, and its value should normally be
    ``&PyType_Type``.  However, for dynamically loadable extension modules that must
    be usable on Windows (at least), the compiler complains that this is not a valid
-   initializer.  Therefore, the convention is to pass *NULL* to the
+   initializer.  Therefore, the convention is to pass ``NULL`` to the
    ``PyObject_HEAD_INIT`` macro and to initialize this field explicitly at the
    start of the module's initialization function, before doing anything else.  This
    is typically done like this::
@@ -531,7 +531,7 @@ type objects) *must* have the :attr:`ob_size` field.
       Foo_Type.ob_type = &PyType_Type;
 
    This should be done before any instances of the type are created.
-   :c:func:`PyType_Ready` checks if :attr:`ob_type` is *NULL*, and if so,
+   :c:func:`PyType_Ready` checks if :attr:`ob_type` is ``NULL``, and if so,
    initializes it to the :attr:`ob_type` field of the base class.
    :c:func:`PyType_Ready` will not change this field if it is non-zero.
 
@@ -557,7 +557,7 @@ PyTypeObject Slots
 ------------------
 
 Each slot has a section describing inheritance.  If :c:func:`PyType_Ready`
-may set a value when the field is set to *NULL* then there will also be
+may set a value when the field is set to ``NULL`` then there will also be
 a "Default" section.  (Note that many fields set on :c:type:`PyBaseObject_Type`
 and :c:type:`PyType_Type` effectively act as defaults.)
 
@@ -586,7 +586,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
    type will be impossible to pickle.  Additionally, it will not be listed in
    module documentations created with pydoc.
 
-   This field must not be *NULL*.  It is the only required field
+   This field must not be ``NULL``.  It is the only required field
    in :c:func:`PyTypeObject` (other than potentially
    :c:member:`~PyTypeObject.tp_itemsize`).
 
@@ -631,7 +631,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
    :c:member:`~PyTypeObject.tp_basicsize` is a multiple of ``sizeof(double)`` (assuming this is the
    alignment requirement for ``double``).
 
-   For any type with variable-length instances, this field must not be *NULL*.
+   For any type with variable-length instances, this field must not be ``NULL``.
 
    **Inheritance:**
 
@@ -760,7 +760,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
 
    This field is inherited by subtypes together with :c:member:`~PyTypeObject.tp_getattro`: a subtype
    inherits both :c:member:`~PyTypeObject.tp_getattr` and :c:member:`~PyTypeObject.tp_getattro` from its base type when
-   the subtype's :c:member:`~PyTypeObject.tp_getattr` and :c:member:`~PyTypeObject.tp_getattro` are both *NULL*.
+   the subtype's :c:member:`~PyTypeObject.tp_getattr` and :c:member:`~PyTypeObject.tp_getattro` are both ``NULL``.
 
 
 .. c:member:: setattrfunc PyTypeObject.tp_setattr
@@ -777,7 +777,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
 
    This field is inherited by subtypes together with :c:member:`~PyTypeObject.tp_setattro`: a subtype
    inherits both :c:member:`~PyTypeObject.tp_setattr` and :c:member:`~PyTypeObject.tp_setattro` from its base type when
-   the subtype's :c:member:`~PyTypeObject.tp_setattr` and :c:member:`~PyTypeObject.tp_setattro` are both *NULL*.
+   the subtype's :c:member:`~PyTypeObject.tp_setattr` and :c:member:`~PyTypeObject.tp_setattro` are both ``NULL``.
 
 
 .. c:member:: PyAsyncMethods* PyTypeObject.tp_as_async
@@ -894,13 +894,13 @@ and :c:type:`PyType_Type` effectively act as defaults.)
    This field is inherited by subtypes together with
    :c:member:`~PyTypeObject.tp_richcompare`: a subtype inherits both of
    :c:member:`~PyTypeObject.tp_richcompare` and :c:member:`~PyTypeObject.tp_hash`, when the subtype's
-   :c:member:`~PyTypeObject.tp_richcompare` and :c:member:`~PyTypeObject.tp_hash` are both *NULL*.
+   :c:member:`~PyTypeObject.tp_richcompare` and :c:member:`~PyTypeObject.tp_hash` are both ``NULL``.
 
 
 .. c:member:: ternaryfunc PyTypeObject.tp_call
 
    An optional pointer to a function that implements calling the object.  This
-   should be *NULL* if the object is not callable.  The signature is the same as
+   should be ``NULL`` if the object is not callable.  The signature is the same as
    for :c:func:`PyObject_Call`::
 
       PyObject *tp_call(PyObject *self, PyObject *args, PyObject *kwargs);
@@ -952,7 +952,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
 
    This field is inherited by subtypes together with :c:member:`~PyTypeObject.tp_getattr`: a subtype
    inherits both :c:member:`~PyTypeObject.tp_getattr` and :c:member:`~PyTypeObject.tp_getattro` from its base type when
-   the subtype's :c:member:`~PyTypeObject.tp_getattr` and :c:member:`~PyTypeObject.tp_getattro` are both *NULL*.
+   the subtype's :c:member:`~PyTypeObject.tp_getattr` and :c:member:`~PyTypeObject.tp_getattro` are both ``NULL``.
 
    **Default:**
 
@@ -967,7 +967,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
 
       PyObject *tp_setattro(PyObject *self, PyObject *attr, PyObject *value);
 
-   In addition, setting *value* to *NULL* to delete an attribute must be
+   In addition, setting *value* to ``NULL`` to delete an attribute must be
    supported.  It is usually convenient to set this field to
    :c:func:`PyObject_GenericSetAttr`, which implements the normal
    way of setting object attributes.
@@ -978,7 +978,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
 
    This field is inherited by subtypes together with :c:member:`~PyTypeObject.tp_setattr`: a subtype
    inherits both :c:member:`~PyTypeObject.tp_setattr` and :c:member:`~PyTypeObject.tp_setattro` from its base type when
-   the subtype's :c:member:`~PyTypeObject.tp_setattr` and :c:member:`~PyTypeObject.tp_setattro` are both *NULL*.
+   the subtype's :c:member:`~PyTypeObject.tp_setattr` and :c:member:`~PyTypeObject.tp_setattro` are both ``NULL``.
 
    **Default:**
 
@@ -1005,7 +1005,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
    :c:member:`~PyTypeObject.tp_as_number`, :c:member:`~PyTypeObject.tp_as_sequence`, :c:member:`~PyTypeObject.tp_as_mapping`, and
    :c:member:`~PyTypeObject.tp_as_buffer`) that were historically not always present are valid; if
    such a flag bit is clear, the type fields it guards must not be accessed and
-   must be considered to have a zero or *NULL* value instead.
+   must be considered to have a zero or ``NULL`` value instead.
 
    **Inheritance:**
 
@@ -1018,7 +1018,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
    the :c:member:`~PyTypeObject.tp_traverse` and :c:member:`~PyTypeObject.tp_clear` fields, i.e. if the
    :const:`Py_TPFLAGS_HAVE_GC` flag bit is clear in the subtype and the
    :c:member:`~PyTypeObject.tp_traverse` and :c:member:`~PyTypeObject.tp_clear` fields in the subtype exist and have
-   *NULL* values.
+   ``NULL`` values.
 
    .. XXX are most flag bits *really* inherited individually?
 
@@ -1097,7 +1097,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
       together with the :attr:`tp_traverse` and :attr:`tp_clear`
       fields, i.e.  if the :const:`Py_TPFLAGS_HAVE_GC` flag bit is
       clear in the subtype and the :attr:`tp_traverse` and
-      :attr:`tp_clear` fields in the subtype exist and have *NULL*
+      :attr:`tp_clear` fields in the subtype exist and have ``NULL``
       values.
 
 
@@ -1232,7 +1232,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
 
    Note that :c:func:`Py_VISIT` is called only on those members that can participate
    in reference cycles.  Although there is also a ``self->key`` member, it can only
-   be *NULL* or a Python string and therefore cannot be part of a reference cycle.
+   be ``NULL`` or a Python string and therefore cannot be part of a reference cycle.
 
    On the other hand, even if you know a member can never be part of a cycle, as a
    debugging aid you may want to visit it anyway just so the :mod:`gc` module's
@@ -1271,7 +1271,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
 
    Implementations of :c:member:`~PyTypeObject.tp_clear` should drop the instance's references to
    those of its members that may be Python objects, and set its pointers to those
-   members to *NULL*, as in the following example::
+   members to ``NULL``, as in the following example::
 
       static int
       local_clear(localobject *self)
@@ -1285,12 +1285,12 @@ and :c:type:`PyType_Type` effectively act as defaults.)
 
    The :c:func:`Py_CLEAR` macro should be used, because clearing references is
    delicate:  the reference to the contained object must not be decremented until
-   after the pointer to the contained object is set to *NULL*.  This is because
+   after the pointer to the contained object is set to ``NULL``.  This is because
    decrementing the reference count may cause the contained object to become trash,
    triggering a chain of reclamation activity that may include invoking arbitrary
    Python code (due to finalizers, or weakref callbacks, associated with the
    contained object). If it's possible for such code to reference *self* again,
-   it's important that the pointer to the contained object be *NULL* at that time,
+   it's important that the pointer to the contained object be ``NULL`` at that time,
    so that *self* knows the contained object can no longer be used.  The
    :c:func:`Py_CLEAR` macro performs the operations in a safe order.
 
@@ -1324,7 +1324,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
 
    The function should return the result of the comparison (usually ``Py_True``
    or ``Py_False``).  If the comparison is undefined, it must return
-   ``Py_NotImplemented``, if another error occurred it must return *NULL* and
+   ``Py_NotImplemented``, if another error occurred it must return ``NULL`` and
    set an exception condition.
 
    The following constants are defined to be used as the third argument for
@@ -1358,7 +1358,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
 
       The return value's reference count is properly incremented.
 
-      On error, sets an exception and returns *NULL* from the function.
+      On error, sets an exception and returns ``NULL`` from the function.
 
       .. versionadded:: 3.7
 
@@ -1369,7 +1369,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
    This field is inherited by subtypes together with :c:member:`~PyTypeObject.tp_hash`:
    a subtype inherits :c:member:`~PyTypeObject.tp_richcompare` and :c:member:`~PyTypeObject.tp_hash` when
    the subtype's :c:member:`~PyTypeObject.tp_richcompare` and :c:member:`~PyTypeObject.tp_hash` are both
-   *NULL*.
+   ``NULL``.
 
    **Default:**
 
@@ -1387,7 +1387,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
    reference list head (ignoring the GC header, if present); this offset is used by
    :c:func:`PyObject_ClearWeakRefs` and the :c:func:`PyWeakref_\*` functions.  The
    instance structure needs to include a field of type :c:type:`PyObject\*` which is
-   initialized to *NULL*.
+   initialized to ``NULL``.
 
    Do not confuse this field with :c:member:`~PyTypeObject.tp_weaklist`; that is the list head for
    weak references to the type object itself.
@@ -1436,9 +1436,9 @@ and :c:type:`PyType_Type` effectively act as defaults.)
 
       PyObject *tp_iternext(PyObject *self);
 
-   When the iterator is exhausted, it must return *NULL*; a :exc:`StopIteration`
+   When the iterator is exhausted, it must return ``NULL``; a :exc:`StopIteration`
    exception may or may not be set.  When another error occurs, it must return
-   *NULL* too.  Its presence signals that the instances of this type are
+   ``NULL`` too.  Its presence signals that the instances of this type are
    iterators.
 
    Iterator types should also define the :c:member:`~PyTypeObject.tp_iter` function, and that
@@ -1454,7 +1454,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
 
 .. c:member:: struct PyMethodDef* PyTypeObject.tp_methods
 
-   An optional pointer to a static *NULL*-terminated array of :c:type:`PyMethodDef`
+   An optional pointer to a static ``NULL``-terminated array of :c:type:`PyMethodDef`
    structures, declaring regular methods of this type.
 
    For each entry in the array, an entry is added to the type's dictionary (see
@@ -1468,7 +1468,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
 
 .. c:member:: struct PyMemberDef* PyTypeObject.tp_members
 
-   An optional pointer to a static *NULL*-terminated array of :c:type:`PyMemberDef`
+   An optional pointer to a static ``NULL``-terminated array of :c:type:`PyMemberDef`
    structures, declaring regular data members (fields or slots) of instances of
    this type.
 
@@ -1483,7 +1483,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
 
 .. c:member:: struct PyGetSetDef* PyTypeObject.tp_getset
 
-   An optional pointer to a static *NULL*-terminated array of :c:type:`PyGetSetDef`
+   An optional pointer to a static ``NULL``-terminated array of :c:type:`PyGetSetDef`
    structures, declaring computed attributes of instances of this type.
 
    For each entry in the array, an entry is added to the type's dictionary (see
@@ -1533,7 +1533,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
 
    The type's dictionary is stored here by :c:func:`PyType_Ready`.
 
-   This field should normally be initialized to *NULL* before PyType_Ready is
+   This field should normally be initialized to ``NULL`` before PyType_Ready is
    called; it may also be initialized to a dictionary containing initial attributes
    for the type.  Once :c:func:`PyType_Ready` has initialized the type, extra
    attributes for the type may be added to this dictionary only if they don't
@@ -1546,7 +1546,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
 
    **Default:**
 
-   If this field is *NULL*, :c:func:`PyType_Ready` will assign a new
+   If this field is ``NULL``, :c:func:`PyType_Ready` will assign a new
    dictionary to it.
 
    .. warning::
@@ -1579,7 +1579,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
 
       int tp_descr_set(PyObject *self, PyObject *obj, PyObject *value);
 
-   The *value* argument is set to *NULL* to delete the value.
+   The *value* argument is set to ``NULL`` to delete the value.
 
    .. XXX explain more?
 
@@ -1645,7 +1645,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
    **Default:**
 
    This slot has no default.  For static types, if the field is
-   *NULL* then no :attr:`__dict__` gets created for instances.
+   ``NULL`` then no :attr:`__dict__` gets created for instances.
 
 
 .. c:member:: initproc PyTypeObject.tp_init
@@ -1665,7 +1665,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
    arguments represent positional and keyword arguments of the call to
    :meth:`__init__`.
 
-   The :c:member:`~PyTypeObject.tp_init` function, if not *NULL*, is called when an instance is
+   The :c:member:`~PyTypeObject.tp_init` function, if not ``NULL``, is called when an instance is
    created normally by calling its type, after the type's :c:member:`~PyTypeObject.tp_new` function
    has returned an instance of the type.  If the :c:member:`~PyTypeObject.tp_new` function returns an
    instance of some other type that is not a subtype of the original type, no
@@ -1732,12 +1732,12 @@ and :c:type:`PyType_Type` effectively act as defaults.)
    **Inheritance:**
 
    This field is inherited by subtypes, except it is not inherited by static types
-   whose :c:member:`~PyTypeObject.tp_base` is *NULL* or ``&PyBaseObject_Type``.
+   whose :c:member:`~PyTypeObject.tp_base` is ``NULL`` or ``&PyBaseObject_Type``.
 
    **Default:**
 
    For static types this field has no default.  This means if the
-   slot is defined as *NULL*, the type cannot be called to create new
+   slot is defined as ``NULL``, the type cannot be called to create new
    instances; presumably there is some other way to create
    instances, like a factory function.
 
@@ -1788,7 +1788,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
 
    **Default:**
 
-   This slot has no default.  If this field is *NULL*,
+   This slot has no default.  If this field is ``NULL``,
    :const:`Py_TPFLAGS_HAVE_GC` is used as the functional equivalent.
 
 
@@ -1796,7 +1796,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
 
    Tuple of base types.
 
-   This is set for types created by a class statement.  It should be *NULL* for
+   This is set for types created by a class statement.  It should be ``NULL`` for
    statically defined types.
 
    **Inheritance:**
@@ -2034,12 +2034,12 @@ Number Object Structures
       and implement the necessary conversions (at least one of the operands is
       an instance of the defined type).  If the operation is not defined for the
       given operands, binary and ternary functions must return
-      ``Py_NotImplemented``, if another error occurred they must return *NULL*
+      ``Py_NotImplemented``, if another error occurred they must return ``NULL``
       and set an exception.
 
    .. note::
 
-      The :c:data:`nb_reserved` field should always be *NULL*.  It
+      The :c:data:`nb_reserved` field should always be ``NULL``.  It
       was previously called :c:data:`nb_long`, and was renamed in
       Python 3.0.1.
 
@@ -2098,14 +2098,14 @@ Mapping Object Structures
 
    This function is used by :c:func:`PyMapping_Size` and
    :c:func:`PyObject_Size`, and has the same signature.  This slot may be set to
-   *NULL* if the object has no defined length.
+   ``NULL`` if the object has no defined length.
 
 .. c:member:: binaryfunc PyMappingMethods.mp_subscript
 
    This function is used by :c:func:`PyObject_GetItem` and
    :c:func:`PySequence_GetSlice`, and has the same signature as
    :c:func:`!PyObject_GetItem`.  This slot must be filled for the
-   :c:func:`PyMapping_Check` function to return ``1``, it can be *NULL*
+   :c:func:`PyMapping_Check` function to return ``1``, it can be ``NULL``
    otherwise.
 
 .. c:member:: objobjargproc PyMappingMethods.mp_ass_subscript
@@ -2113,8 +2113,8 @@ Mapping Object Structures
    This function is used by :c:func:`PyObject_SetItem`,
    :c:func:`PyObject_DelItem`, :c:func:`PyObject_SetSlice` and
    :c:func:`PyObject_DelSlice`.  It has the same signature as
-   :c:func:`!PyObject_SetItem`, but *v* can also be set to *NULL* to delete
-   an item.  If this slot is *NULL*, the object does not support item
+   :c:func:`!PyObject_SetItem`, but *v* can also be set to ``NULL`` to delete
+   an item.  If this slot is ``NULL``, the object does not support item
    assignment and deletion.
 
 
@@ -2156,11 +2156,11 @@ Sequence Object Structures
    signature.  It is also used by :c:func:`PyObject_GetItem`, after trying
    the subscription via the :c:member:`~PyMappingMethods.mp_subscript` slot.
    This slot must be filled for the :c:func:`PySequence_Check`
-   function to return ``1``, it can be *NULL* otherwise.
+   function to return ``1``, it can be ``NULL`` otherwise.
 
    Negative indexes are handled as follows: if the :attr:`sq_length` slot is
    filled, it is called and the sequence length is used to compute a positive
-   index which is passed to :attr:`sq_item`.  If :attr:`sq_length` is *NULL*,
+   index which is passed to :attr:`sq_item`.  If :attr:`sq_length` is ``NULL``,
    the index is passed as is to the function.
 
 .. c:member:: ssizeobjargproc PySequenceMethods.sq_ass_item
@@ -2169,13 +2169,13 @@ Sequence Object Structures
    signature.  It is also used by :c:func:`PyObject_SetItem` and
    :c:func:`PyObject_DelItem`, after trying the item assignment and deletion
    via the :c:member:`~PyMappingMethods.mp_ass_subscript` slot.
-   This slot may be left to *NULL* if the object does not support
+   This slot may be left to ``NULL`` if the object does not support
    item assignment and deletion.
 
 .. c:member:: objobjproc PySequenceMethods.sq_contains
 
    This function may be used by :c:func:`PySequence_Contains` and has the same
-   signature.  This slot may be left to *NULL*, in this case
+   signature.  This slot may be left to ``NULL``, in this case
    :c:func:`!PySequence_Contains` simply traverses the sequence until it
    finds a match.
 
@@ -2183,7 +2183,7 @@ Sequence Object Structures
 
    This function is used by :c:func:`PySequence_InPlaceConcat` and has the same
    signature.  It should modify its first operand, and return it.  This slot
-   may be left to *NULL*, in this case :c:func:`!PySequence_InPlaceConcat`
+   may be left to ``NULL``, in this case :c:func:`!PySequence_InPlaceConcat`
    will fall back to :c:func:`PySequence_Concat`.  It is also used by the
    augmented assignment ``+=``, after trying numeric in-place addition
    via the :c:member:`~PyNumberMethods.nb_inplace_add` slot.
@@ -2192,7 +2192,7 @@ Sequence Object Structures
 
    This function is used by :c:func:`PySequence_InPlaceRepeat` and has the same
    signature.  It should modify its first operand, and return it.  This slot
-   may be left to *NULL*, in this case :c:func:`!PySequence_InPlaceRepeat`
+   may be left to ``NULL``, in this case :c:func:`!PySequence_InPlaceRepeat`
    will fall back to :c:func:`PySequence_Repeat`.  It is also used by the
    augmented assignment ``*=``, after trying numeric in-place multiplication
    via the :c:member:`~PyNumberMethods.nb_inplace_multiply` slot.
@@ -2224,7 +2224,7 @@ Buffer Object Structures
    steps:
 
    (1) Check if the request can be met. If not, raise :c:data:`PyExc_BufferError`,
-       set :c:data:`view->obj` to *NULL* and return ``-1``.
+       set :c:data:`view->obj` to ``NULL`` and return ``-1``.
 
    (2) Fill in the requested fields.
 
@@ -2270,7 +2270,7 @@ Buffer Object Structures
 
    Handle a request to release the resources of the buffer. If no resources
    need to be released, :c:member:`PyBufferProcs.bf_releasebuffer` may be
-   *NULL*. Otherwise, a standard implementation of this function will take
+   ``NULL``. Otherwise, a standard implementation of this function will take
    these optional steps:
 
    (1) Decrement an internal counter for the number of exports.
@@ -2324,7 +2324,7 @@ Async Object Structures
    The returned object must be an iterator, i.e. :c:func:`PyIter_Check` must
    return ``1`` for it.
 
-   This slot may be set to *NULL* if an object is not an :term:`awaitable`.
+   This slot may be set to ``NULL`` if an object is not an :term:`awaitable`.
 
 .. c:member:: unaryfunc PyAsyncMethods.am_aiter
 
@@ -2334,7 +2334,7 @@ Async Object Structures
 
    Must return an :term:`awaitable` object.  See :meth:`__anext__` for details.
 
-   This slot may be set to *NULL* if an object does not implement
+   This slot may be set to ``NULL`` if an object does not implement
    asynchronous iteration protocol.
 
 .. c:member:: unaryfunc PyAsyncMethods.am_anext
@@ -2344,7 +2344,7 @@ Async Object Structures
       PyObject *am_anext(PyObject *self);
 
    Must return an :term:`awaitable` object.  See :meth:`__anext__` for details.
-   This slot may be set to *NULL*.
+   This slot may be set to ``NULL``.
 
 
 .. _slot-typedefs:
@@ -2400,7 +2400,7 @@ Slot Type typedefs
 .. c:type:: int (*setattrfunc)(PyObject *self, char *attr, PyObject *value)
 
    Set the value of the named attribute for the object.
-   The value argument is set to *NULL* to delete the attribute.
+   The value argument is set to ``NULL`` to delete the attribute.
 
 .. c:type:: PyObject *(*getattrofunc)(PyObject *self, PyObject *attr)
 
@@ -2411,7 +2411,7 @@ Slot Type typedefs
 .. c:type:: int (*setattrofunc)(PyObject *self, PyObject *attr, PyObject *value)
 
    Set the value of the named attribute for the object.
-   The value argument is set to *NULL* to delete the attribute.
+   The value argument is set to ``NULL`` to delete the attribute.
 
    See :c:member:`~PyTypeObject.tp_setattro`.
 
diff --git a/Doc/c-api/unicode.rst b/Doc/c-api/unicode.rst
index a150a9c39d880..0359f5ec36bed 100644
--- a/Doc/c-api/unicode.rst
+++ b/Doc/c-api/unicode.rst
@@ -236,7 +236,7 @@ access internal read-only data of Unicode objects:
    .. versionchanged:: 3.3
       This macro is now inefficient -- because in many cases the
       :c:type:`Py_UNICODE` representation does not exist and needs to be created
-      -- and can fail (return *NULL* with an exception set).  Try to port the
+      -- and can fail (return ``NULL`` with an exception set).  Try to port the
       code to use the new :c:func:`PyUnicode_nBYTE_DATA` macros or use
       :c:func:`PyUnicode_WRITE` or :c:func:`PyUnicode_READ`.
 
@@ -413,11 +413,11 @@ APIs:
 
    Create a Unicode object from the char buffer *u*.  The bytes will be
    interpreted as being UTF-8 encoded.  The buffer is copied into the new
-   object. If the buffer is not *NULL*, the return value might be a shared
+   object. If the buffer is not ``NULL``, the return value might be a shared
    object, i.e. modification of the data is not allowed.
 
-   If *u* is *NULL*, this function behaves like :c:func:`PyUnicode_FromUnicode`
-   with the buffer set to *NULL*.  This usage is deprecated in favor of
+   If *u* is ``NULL``, this function behaves like :c:func:`PyUnicode_FromUnicode`
+   with the buffer set to ``NULL``.  This usage is deprecated in favor of
    :c:func:`PyUnicode_New`.
 
 
@@ -443,82 +443,82 @@ APIs:
 
    .. tabularcolumns:: |l|l|L|
 
-   +-------------------+---------------------+--------------------------------+
-   | Format Characters | Type                | Comment                        |
-   +===================+=====================+================================+
-   | :attr:`%%`        | *n/a*               | The literal % character.       |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%c`        | int                 | A single character,            |
-   |                   |                     | represented as a C int.        |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%d`        | int                 | Equivalent to                  |
-   |                   |                     | ``printf("%d")``. [1]_         |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%u`        | unsigned int        | Equivalent to                  |
-   |                   |                     | ``printf("%u")``. [1]_         |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%ld`       | long                | Equivalent to                  |
-   |                   |                     | ``printf("%ld")``. [1]_        |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%li`       | long                | Equivalent to                  |
-   |                   |                     | ``printf("%li")``. [1]_        |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%lu`       | unsigned long       | Equivalent to                  |
-   |                   |                     | ``printf("%lu")``. [1]_        |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%lld`      | long long           | Equivalent to                  |
-   |                   |                     | ``printf("%lld")``. [1]_       |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%lli`      | long long           | Equivalent to                  |
-   |                   |                     | ``printf("%lli")``. [1]_       |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%llu`      | unsigned long long  | Equivalent to                  |
-   |                   |                     | ``printf("%llu")``. [1]_       |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%zd`       | Py_ssize_t          | Equivalent to                  |
-   |                   |                     | ``printf("%zd")``. [1]_        |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%zi`       | Py_ssize_t          | Equivalent to                  |
-   |                   |                     | ``printf("%zi")``. [1]_        |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%zu`       | size_t              | Equivalent to                  |
-   |                   |                     | ``printf("%zu")``. [1]_        |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%i`        | int                 | Equivalent to                  |
-   |                   |                     | ``printf("%i")``. [1]_         |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%x`        | int                 | Equivalent to                  |
-   |                   |                     | ``printf("%x")``. [1]_         |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%s`        | const char\*        | A null-terminated C character  |
-   |                   |                     | array.                         |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%p`        | const void\*        | The hex representation of a C  |
-   |                   |                     | pointer. Mostly equivalent to  |
-   |                   |                     | ``printf("%p")`` except that   |
-   |                   |                     | it is guaranteed to start with |
-   |                   |                     | the literal ``0x`` regardless  |
-   |                   |                     | of what the platform's         |
-   |                   |                     | ``printf`` yields.             |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%A`        | PyObject\*          | The result of calling          |
-   |                   |                     | :func:`ascii`.                 |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%U`        | PyObject\*          | A Unicode object.              |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%V`        | PyObject\*,         | A Unicode object (which may be |
-   |                   | const char\*        | *NULL*) and a null-terminated  |
-   |                   |                     | C character array as a second  |
-   |                   |                     | parameter (which will be used, |
-   |                   |                     | if the first parameter is      |
-   |                   |                     | *NULL*).                       |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%S`        | PyObject\*          | The result of calling          |
-   |                   |                     | :c:func:`PyObject_Str`.        |
-   +-------------------+---------------------+--------------------------------+
-   | :attr:`%R`        | PyObject\*          | The result of calling          |
-   |                   |                     | :c:func:`PyObject_Repr`.       |
-   +-------------------+---------------------+--------------------------------+
+   +-------------------+---------------------+----------------------------------+
+   | Format Characters | Type                | Comment                          |
+   +===================+=====================+==================================+
+   | :attr:`%%`        | *n/a*               | The literal % character.         |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%c`        | int                 | A single character,              |
+   |                   |                     | represented as a C int.          |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%d`        | int                 | Equivalent to                    |
+   |                   |                     | ``printf("%d")``. [1]_           |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%u`        | unsigned int        | Equivalent to                    |
+   |                   |                     | ``printf("%u")``. [1]_           |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%ld`       | long                | Equivalent to                    |
+   |                   |                     | ``printf("%ld")``. [1]_          |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%li`       | long                | Equivalent to                    |
+   |                   |                     | ``printf("%li")``. [1]_          |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%lu`       | unsigned long       | Equivalent to                    |
+   |                   |                     | ``printf("%lu")``. [1]_          |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%lld`      | long long           | Equivalent to                    |
+   |                   |                     | ``printf("%lld")``. [1]_         |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%lli`      | long long           | Equivalent to                    |
+   |                   |                     | ``printf("%lli")``. [1]_         |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%llu`      | unsigned long long  | Equivalent to                    |
+   |                   |                     | ``printf("%llu")``. [1]_         |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%zd`       | Py_ssize_t          | Equivalent to                    |
+   |                   |                     | ``printf("%zd")``. [1]_          |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%zi`       | Py_ssize_t          | Equivalent to                    |
+   |                   |                     | ``printf("%zi")``. [1]_          |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%zu`       | size_t              | Equivalent to                    |
+   |                   |                     | ``printf("%zu")``. [1]_          |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%i`        | int                 | Equivalent to                    |
+   |                   |                     | ``printf("%i")``. [1]_           |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%x`        | int                 | Equivalent to                    |
+   |                   |                     | ``printf("%x")``. [1]_           |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%s`        | const char\*        | A null-terminated C character    |
+   |                   |                     | array.                           |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%p`        | const void\*        | The hex representation of a C    |
+   |                   |                     | pointer. Mostly equivalent to    |
+   |                   |                     | ``printf("%p")`` except that     |
+   |                   |                     | it is guaranteed to start with   |
+   |                   |                     | the literal ``0x`` regardless    |
+   |                   |                     | of what the platform's           |
+   |                   |                     | ``printf`` yields.               |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%A`        | PyObject\*          | The result of calling            |
+   |                   |                     | :func:`ascii`.                   |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%U`        | PyObject\*          | A Unicode object.                |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%V`        | PyObject\*,         | A Unicode object (which may be   |
+   |                   | const char\*        | ``NULL``) and a null-terminated  |
+   |                   |                     | C character array as a second    |
+   |                   |                     | parameter (which will be used,   |
+   |                   |                     | if the first parameter is        |
+   |                   |                     | ``NULL``).                       |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%S`        | PyObject\*          | The result of calling            |
+   |                   |                     | :c:func:`PyObject_Str`.          |
+   +-------------------+---------------------+----------------------------------+
+   | :attr:`%R`        | PyObject\*          | The result of calling            |
+   |                   |                     | :c:func:`PyObject_Repr`.         |
+   +-------------------+---------------------+----------------------------------+
 
    An unrecognized format character causes all the rest of the format string to be
    copied as-is to the result string, and any extra arguments discarded.
@@ -558,13 +558,13 @@ APIs:
    :class:`bytes`, :class:`bytearray` and other
    :term:`bytes-like objects <bytes-like object>`
    are decoded according to the given *encoding* and using the error handling
-   defined by *errors*. Both can be *NULL* to have the interface use the default
+   defined by *errors*. Both can be ``NULL`` to have the interface use the default
    values (see :ref:`builtincodecs` for details).
 
    All other objects, including Unicode objects, cause a :exc:`TypeError` to be
    set.
 
-   The API returns *NULL* if there was an error.  The caller is responsible for
+   The API returns ``NULL`` if there was an error.  The caller is responsible for
    decref'ing the returned objects.
 
 
@@ -640,7 +640,7 @@ APIs:
                                           Py_ssize_t buflen, int copy_null)
 
    Copy the string *u* into a UCS4 buffer, including a null character, if
-   *copy_null* is set.  Returns *NULL* and sets an exception on error (in
+   *copy_null* is set.  Returns ``NULL`` and sets an exception on error (in
    particular, a :exc:`SystemError` if *buflen* is smaller than the length of
    *u*).  *buffer* is returned on success.
 
@@ -650,7 +650,7 @@ APIs:
 .. c:function:: Py_UCS4* PyUnicode_AsUCS4Copy(PyObject *u)
 
    Copy the string *u* into a new UCS4 buffer that is allocated using
-   :c:func:`PyMem_Malloc`.  If this fails, *NULL* is returned with a
+   :c:func:`PyMem_Malloc`.  If this fails, ``NULL`` is returned with a
    :exc:`MemoryError` set.  The returned buffer always has an extra
    null code point appended.
 
@@ -670,15 +670,15 @@ Extension modules can continue using them, as they will not be removed in Python
 .. c:function:: PyObject* PyUnicode_FromUnicode(const Py_UNICODE *u, Py_ssize_t size)
 
    Create a Unicode object from the Py_UNICODE buffer *u* of the given size. *u*
-   may be *NULL* which causes the contents to be undefined. It is the user's
+   may be ``NULL`` which causes the contents to be undefined. It is the user's
    responsibility to fill in the needed data.  The buffer is copied into the new
    object.
 
-   If the buffer is not *NULL*, the return value might be a shared object.
+   If the buffer is not ``NULL``, the return value might be a shared object.
    Therefore, modification of the resulting Unicode object is only allowed when
-   *u* is *NULL*.
+   *u* is ``NULL``.
 
-   If the buffer is *NULL*, :c:func:`PyUnicode_READY` must be called once the
+   If the buffer is ``NULL``, :c:func:`PyUnicode_READY` must be called once the
    string content has been filled before using any of the access macros such as
    :c:func:`PyUnicode_KIND`.
 
@@ -689,7 +689,7 @@ Extension modules can continue using them, as they will not be removed in Python
 .. c:function:: Py_UNICODE* PyUnicode_AsUnicode(PyObject *unicode)
 
    Return a read-only pointer to the Unicode object's internal
-   :c:type:`Py_UNICODE` buffer, or *NULL* on error. This will create the
+   :c:type:`Py_UNICODE` buffer, or ``NULL`` on error. This will create the
    :c:type:`Py_UNICODE*` representation of the object if it is not yet
    available. The buffer is always terminated with an extra null code point.
    Note that the resulting :c:type:`Py_UNICODE` string may also contain
@@ -705,7 +705,7 @@ Extension modules can continue using them, as they will not be removed in Python
 
    Create a Unicode object by replacing all decimal digits in
    :c:type:`Py_UNICODE` buffer of the given *size* by ASCII digits 0--9
-   according to their decimal value.  Return *NULL* if an exception occurs.
+   according to their decimal value.  Return ``NULL`` if an exception occurs.
 
 
 .. c:function:: Py_UNICODE* PyUnicode_AsUnicodeAndSize(PyObject *unicode, Py_ssize_t *size)
@@ -721,7 +721,7 @@ Extension modules can continue using them, as they will not be removed in Python
 
 .. c:function:: Py_UNICODE* PyUnicode_AsUnicodeCopy(PyObject *unicode)
 
-   Create a copy of a Unicode string ending with a null code point. Return *NULL*
+   Create a copy of a Unicode string ending with a null code point. Return ``NULL``
    and raise a :exc:`MemoryError` exception on memory allocation failure,
    otherwise return a new allocated buffer (use :c:func:`PyMem_Free` to free
    the buffer). Note that the resulting :c:type:`Py_UNICODE*` string may
@@ -932,7 +932,7 @@ wchar_t Support
    Create a Unicode object from the :c:type:`wchar_t` buffer *w* of the given *size*.
    Passing ``-1`` as the *size* indicates that the function must itself compute the length,
    using wcslen.
-   Return *NULL* on failure.
+   Return ``NULL`` on failure.
 
 
 .. c:function:: Py_ssize_t PyUnicode_AsWideChar(PyObject *unicode, wchar_t *w, Py_ssize_t size)
@@ -951,22 +951,22 @@ wchar_t Support
 .. c:function:: wchar_t* PyUnicode_AsWideCharString(PyObject *unicode, Py_ssize_t *size)
 
    Convert the Unicode object to a wide character string. The output string
-   always ends with a null character. If *size* is not *NULL*, write the number
+   always ends with a null character. If *size* is not ``NULL``, write the number
    of wide characters (excluding the trailing null termination character) into
    *\*size*. Note that the resulting :c:type:`wchar_t` string might contain
    null characters, which would cause the string to be truncated when used with
-   most C functions. If *size* is *NULL* and the :c:type:`wchar_t*` string
+   most C functions. If *size* is ``NULL`` and the :c:type:`wchar_t*` string
    contains null characters a :exc:`ValueError` is raised.
 
    Returns a buffer allocated by :c:func:`PyMem_Alloc` (use
-   :c:func:`PyMem_Free` to free it) on success. On error, returns *NULL*
+   :c:func:`PyMem_Free` to free it) on success. On error, returns ``NULL``
    and *\*size* is undefined. Raises a :exc:`MemoryError` if memory allocation
    is failed.
 
    .. versionadded:: 3.2
 
    .. versionchanged:: 3.7
-      Raises a :exc:`ValueError` if *size* is *NULL* and the :c:type:`wchar_t*`
+      Raises a :exc:`ValueError` if *size* is ``NULL`` and the :c:type:`wchar_t*`
       string contains null characters.
 
 
@@ -982,7 +982,7 @@ Many of the following APIs take two arguments encoding and errors, and they
 have the same semantics as the ones of the built-in :func:`str` string object
 constructor.
 
-Setting encoding to *NULL* causes the default encoding to be used
+Setting encoding to ``NULL`` causes the default encoding to be used
 which is ASCII.  The file system calls should use
 :c:func:`PyUnicode_FSConverter` for encoding file names. This uses the
 variable :c:data:`Py_FileSystemDefaultEncoding` internally. This
@@ -990,7 +990,7 @@ variable should be treated as read-only: on some systems, it will be a
 pointer to a static string, on others, it will change at run-time
 (such as when the application invokes setlocale).
 
-Error handling is set by errors which may also be set to *NULL* meaning to use
+Error handling is set by errors which may also be set to ``NULL`` meaning to use
 the default handling defined for the codec.  Default error handling for all
 built-in codecs is "strict" (:exc:`ValueError` is raised).
 
@@ -1010,7 +1010,7 @@ These are the generic codec APIs:
    Create a Unicode object by decoding *size* bytes of the encoded string *s*.
    *encoding* and *errors* have the same meaning as the parameters of the same name
    in the :func:`str` built-in function.  The codec to be used is looked up
-   using the Python codec registry.  Return *NULL* if an exception was raised by
+   using the Python codec registry.  Return ``NULL`` if an exception was raised by
    the codec.
 
 
@@ -1020,7 +1020,7 @@ These are the generic codec APIs:
    Encode a Unicode object and return the result as Python bytes object.
    *encoding* and *errors* have the same meaning as the parameters of the same
    name in the Unicode :meth:`~str.encode` method. The codec to be used is looked up
-   using the Python codec registry. Return *NULL* if an exception was raised by
+   using the Python codec registry. Return ``NULL`` if an exception was raised by
    the codec.
 
 
@@ -1030,7 +1030,7 @@ These are the generic codec APIs:
    Encode the :c:type:`Py_UNICODE` buffer *s* of the given *size* and return a Python
    bytes object.  *encoding* and *errors* have the same meaning as the
    parameters of the same name in the Unicode :meth:`~str.encode` method.  The codec
-   to be used is looked up using the Python codec registry.  Return *NULL* if an
+   to be used is looked up using the Python codec registry.  Return ``NULL`` if an
    exception was raised by the codec.
 
    .. deprecated-removed:: 3.3 4.0
@@ -1047,14 +1047,14 @@ These are the UTF-8 codec APIs:
 .. c:function:: PyObject* PyUnicode_DecodeUTF8(const char *s, Py_ssize_t size, const char *errors)
 
    Create a Unicode object by decoding *size* bytes of the UTF-8 encoded string
-   *s*. Return *NULL* if an exception was raised by the codec.
+   *s*. Return ``NULL`` if an exception was raised by the codec.
 
 
 .. c:function:: PyObject* PyUnicode_DecodeUTF8Stateful(const char *s, Py_ssize_t size, \
                               const char *errors, Py_ssize_t *consumed)
 
-   If *consumed* is *NULL*, behave like :c:func:`PyUnicode_DecodeUTF8`. If
-   *consumed* is not *NULL*, trailing incomplete UTF-8 byte sequences will not be
+   If *consumed* is ``NULL``, behave like :c:func:`PyUnicode_DecodeUTF8`. If
+   *consumed* is not ``NULL``, trailing incomplete UTF-8 byte sequences will not be
    treated as an error. Those bytes will not be decoded and the number of bytes
    that have been decoded will be stored in *consumed*.
 
@@ -1062,7 +1062,7 @@ These are the UTF-8 codec APIs:
 .. c:function:: PyObject* PyUnicode_AsUTF8String(PyObject *unicode)
 
    Encode a Unicode object using UTF-8 and return the result as Python bytes
-   object.  Error handling is "strict".  Return *NULL* if an exception was
+   object.  Error handling is "strict".  Return ``NULL`` if an exception was
    raised by the codec.
 
 
@@ -1070,11 +1070,11 @@ These are the UTF-8 codec APIs:
 
    Return a pointer to the UTF-8 encoding of the Unicode object, and
    store the size of the encoded representation (in bytes) in *size*.  The
-   *size* argument can be *NULL*; in this case no size will be stored.  The
+   *size* argument can be ``NULL``; in this case no size will be stored.  The
    returned buffer always has an extra null byte appended (not included in
    *size*), regardless of whether there are any other null code points.
 
-   In the case of an error, *NULL* is returned with an exception set and no
+   In the case of an error, ``NULL`` is returned with an exception set and no
    *size* is stored.
 
    This caches the UTF-8 representation of the string in the Unicode object, and
@@ -1100,7 +1100,7 @@ These are the UTF-8 codec APIs:
 .. c:function:: PyObject* PyUnicode_EncodeUTF8(const Py_UNICODE *s, Py_ssize_t size, const char *errors)
 
    Encode the :c:type:`Py_UNICODE` buffer *s* of the given *size* using UTF-8 and
-   return a Python bytes object.  Return *NULL* if an exception was raised by
+   return a Python bytes object.  Return ``NULL`` if an exception was raised by
    the codec.
 
    .. deprecated-removed:: 3.3 4.0
@@ -1119,10 +1119,10 @@ These are the UTF-32 codec APIs:
                               const char *errors, int *byteorder)
 
    Decode *size* bytes from a UTF-32 encoded buffer string and return the
-   corresponding Unicode object.  *errors* (if non-*NULL*) defines the error
+   corresponding Unicode object.  *errors* (if non-``NULL``) defines the error
    handling. It defaults to "strict".
 
-   If *byteorder* is non-*NULL*, the decoder starts decoding using the given byte
+   If *byteorder* is non-``NULL``, the decoder starts decoding using the given byte
    order::
 
       *byteorder == -1: little endian
@@ -1137,16 +1137,16 @@ These are the UTF-32 codec APIs:
    After completion, *\*byteorder* is set to the current byte order at the end
    of input data.
 
-   If *byteorder* is *NULL*, the codec starts in native order mode.
+   If *byteorder* is ``NULL``, the codec starts in native order mode.
 
-   Return *NULL* if an exception was raised by the codec.
+   Return ``NULL`` if an exception was raised by the codec.
 
 
 .. c:function:: PyObject* PyUnicode_DecodeUTF32Stateful(const char *s, Py_ssize_t size, \
                               const char *errors, int *byteorder, Py_ssize_t *consumed)
 
-   If *consumed* is *NULL*, behave like :c:func:`PyUnicode_DecodeUTF32`. If
-   *consumed* is not *NULL*, :c:func:`PyUnicode_DecodeUTF32Stateful` will not treat
+   If *consumed* is ``NULL``, behave like :c:func:`PyUnicode_DecodeUTF32`. If
+   *consumed* is not ``NULL``, :c:func:`PyUnicode_DecodeUTF32Stateful` will not treat
    trailing incomplete UTF-32 byte sequences (such as a number of bytes not divisible
    by four) as an error. Those bytes will not be decoded and the number of bytes
    that have been decoded will be stored in *consumed*.
@@ -1156,7 +1156,7 @@ These are the UTF-32 codec APIs:
 
    Return a Python byte string using the UTF-32 encoding in native byte
    order. The string always starts with a BOM mark.  Error handling is "strict".
-   Return *NULL* if an exception was raised by the codec.
+   Return ``NULL`` if an exception was raised by the codec.
 
 
 .. c:function:: PyObject* PyUnicode_EncodeUTF32(const Py_UNICODE *s, Py_ssize_t size, \
@@ -1175,7 +1175,7 @@ These are the UTF-32 codec APIs:
    If *Py_UNICODE_WIDE* is not defined, surrogate pairs will be output
    as a single code point.
 
-   Return *NULL* if an exception was raised by the codec.
+   Return ``NULL`` if an exception was raised by the codec.
 
    .. deprecated-removed:: 3.3 4.0
       Part of the old-style :c:type:`Py_UNICODE` API; please migrate to using
@@ -1192,10 +1192,10 @@ These are the UTF-16 codec APIs:
                               const char *errors, int *byteorder)
 
    Decode *size* bytes from a UTF-16 encoded buffer string and return the
-   corresponding Unicode object.  *errors* (if non-*NULL*) defines the error
+   corresponding Unicode object.  *errors* (if non-``NULL``) defines the error
    handling. It defaults to "strict".
 
-   If *byteorder* is non-*NULL*, the decoder starts decoding using the given byte
+   If *byteorder* is non-``NULL``, the decoder starts decoding using the given byte
    order::
 
       *byteorder == -1: little endian
@@ -1211,16 +1211,16 @@ These are the UTF-16 codec APIs:
    After completion, *\*byteorder* is set to the current byte order at the end
    of input data.
 
-   If *byteorder* is *NULL*, the codec starts in native order mode.
+   If *byteorder* is ``NULL``, the codec starts in native order mode.
 
-   Return *NULL* if an exception was raised by the codec.
+   Return ``NULL`` if an exception was raised by the codec.
 
 
 .. c:function:: PyObject* PyUnicode_DecodeUTF16Stateful(const char *s, Py_ssize_t size, \
                               const char *errors, int *byteorder, Py_ssize_t *consumed)
 
-   If *consumed* is *NULL*, behave like :c:func:`PyUnicode_DecodeUTF16`. If
-   *consumed* is not *NULL*, :c:func:`PyUnicode_DecodeUTF16Stateful` will not treat
+   If *consumed* is ``NULL``, behave like :c:func:`PyUnicode_DecodeUTF16`. If
+   *consumed* is not ``NULL``, :c:func:`PyUnicode_DecodeUTF16Stateful` will not treat
    trailing incomplete UTF-16 byte sequences (such as an odd number of bytes or a
    split surrogate pair) as an error. Those bytes will not be decoded and the
    number of bytes that have been decoded will be stored in *consumed*.
@@ -1230,7 +1230,7 @@ These are the UTF-16 codec APIs:
 
    Return a Python byte string using the UTF-16 encoding in native byte
    order. The string always starts with a BOM mark.  Error handling is "strict".
-   Return *NULL* if an exception was raised by the codec.
+   Return ``NULL`` if an exception was raised by the codec.
 
 
 .. c:function:: PyObject* PyUnicode_EncodeUTF16(const Py_UNICODE *s, Py_ssize_t size, \
@@ -1250,7 +1250,7 @@ These are the UTF-16 codec APIs:
    represented as a surrogate pair. If it is not defined, each :c:type:`Py_UNICODE`
    values is interpreted as a UCS-2 character.
 
-   Return *NULL* if an exception was raised by the codec.
+   Return ``NULL`` if an exception was raised by the codec.
 
    .. deprecated-removed:: 3.3 4.0
       Part of the old-style :c:type:`Py_UNICODE` API; please migrate to using
@@ -1266,14 +1266,14 @@ These are the UTF-7 codec APIs:
 .. c:function:: PyObject* PyUnicode_DecodeUTF7(const char *s, Py_ssize_t size, const char *errors)
 
    Create a Unicode object by decoding *size* bytes of the UTF-7 encoded string
-   *s*.  Return *NULL* if an exception was raised by the codec.
+   *s*.  Return ``NULL`` if an exception was raised by the codec.
 
 
 .. c:function:: PyObject* PyUnicode_DecodeUTF7Stateful(const char *s, Py_ssize_t size, \
                               const char *errors, Py_ssize_t *consumed)
 
-   If *consumed* is *NULL*, behave like :c:func:`PyUnicode_DecodeUTF7`.  If
-   *consumed* is not *NULL*, trailing incomplete UTF-7 base-64 sections will not
+   If *consumed* is ``NULL``, behave like :c:func:`PyUnicode_DecodeUTF7`.  If
+   *consumed* is not ``NULL``, trailing incomplete UTF-7 base-64 sections will not
    be treated as an error.  Those bytes will not be decoded and the number of
    bytes that have been decoded will be stored in *consumed*.
 
@@ -1282,7 +1282,7 @@ These are the UTF-7 codec APIs:
                               int base64SetO, int base64WhiteSpace, const char *errors)
 
    Encode the :c:type:`Py_UNICODE` buffer of the given size using UTF-7 and
-   return a Python bytes object.  Return *NULL* if an exception was raised by
+   return a Python bytes object.  Return ``NULL`` if an exception was raised by
    the codec.
 
    If *base64SetO* is nonzero, "Set O" (punctuation that has no otherwise
@@ -1305,20 +1305,20 @@ These are the "Unicode Escape" codec APIs:
                               Py_ssize_t size, const char *errors)
 
    Create a Unicode object by decoding *size* bytes of the Unicode-Escape encoded
-   string *s*.  Return *NULL* if an exception was raised by the codec.
+   string *s*.  Return ``NULL`` if an exception was raised by the codec.
 
 
 .. c:function:: PyObject* PyUnicode_AsUnicodeEscapeString(PyObject *unicode)
 
    Encode a Unicode object using Unicode-Escape and return the result as a
-   bytes object.  Error handling is "strict".  Return *NULL* if an exception was
+   bytes object.  Error handling is "strict".  Return ``NULL`` if an exception was
    raised by the codec.
 
 
 .. c:function:: PyObject* PyUnicode_EncodeUnicodeEscape(const Py_UNICODE *s, Py_ssize_t size)
 
    Encode the :c:type:`Py_UNICODE` buffer of the given *size* using Unicode-Escape and
-   return a bytes object.  Return *NULL* if an exception was raised by the codec.
+   return a bytes object.  Return ``NULL`` if an exception was raised by the codec.
 
    .. deprecated-removed:: 3.3 4.0
       Part of the old-style :c:type:`Py_UNICODE` API; please migrate to using
@@ -1335,13 +1335,13 @@ These are the "Raw Unicode Escape" codec APIs:
                               Py_ssize_t size, const char *errors)
 
    Create a Unicode object by decoding *size* bytes of the Raw-Unicode-Escape
-   encoded string *s*.  Return *NULL* if an exception was raised by the codec.
+   encoded string *s*.  Return ``NULL`` if an exception was raised by the codec.
 
 
 .. c:function:: PyObject* PyUnicode_AsRawUnicodeEscapeString(PyObject *unicode)
 
    Encode a Unicode object using Raw-Unicode-Escape and return the result as
-   a bytes object.  Error handling is "strict".  Return *NULL* if an exception
+   a bytes object.  Error handling is "strict".  Return ``NULL`` if an exception
    was raised by the codec.
 
 
@@ -1349,7 +1349,7 @@ These are the "Raw Unicode Escape" codec APIs:
                               Py_ssize_t size)
 
    Encode the :c:type:`Py_UNICODE` buffer of the given *size* using Raw-Unicode-Escape
-   and return a bytes object.  Return *NULL* if an exception was raised by the codec.
+   and return a bytes object.  Return ``NULL`` if an exception was raised by the codec.
 
    .. deprecated-removed:: 3.3 4.0
       Part of the old-style :c:type:`Py_UNICODE` API; please migrate to using
@@ -1367,20 +1367,20 @@ ordinals and only these are accepted by the codecs during encoding.
 .. c:function:: PyObject* PyUnicode_DecodeLatin1(const char *s, Py_ssize_t size, const char *errors)
 
    Create a Unicode object by decoding *size* bytes of the Latin-1 encoded string
-   *s*.  Return *NULL* if an exception was raised by the codec.
+   *s*.  Return ``NULL`` if an exception was raised by the codec.
 
 
 .. c:function:: PyObject* PyUnicode_AsLatin1String(PyObject *unicode)
 
    Encode a Unicode object using Latin-1 and return the result as Python bytes
-   object.  Error handling is "strict".  Return *NULL* if an exception was
+   object.  Error handling is "strict".  Return ``NULL`` if an exception was
    raised by the codec.
 
 
 .. c:function:: PyObject* PyUnicode_EncodeLatin1(const Py_UNICODE *s, Py_ssize_t size, const char *errors)
 
    Encode the :c:type:`Py_UNICODE` buffer of the given *size* using Latin-1 and
-   return a Python bytes object.  Return *NULL* if an exception was raised by
+   return a Python bytes object.  Return ``NULL`` if an exception was raised by
    the codec.
 
    .. deprecated-removed:: 3.3 4.0
@@ -1399,20 +1399,20 @@ codes generate errors.
 .. c:function:: PyObject* PyUnicode_DecodeASCII(const char *s, Py_ssize_t size, const char *errors)
 
    Create a Unicode object by decoding *size* bytes of the ASCII encoded string
-   *s*.  Return *NULL* if an exception was raised by the codec.
+   *s*.  Return ``NULL`` if an exception was raised by the codec.
 
 
 .. c:function:: PyObject* PyUnicode_AsASCIIString(PyObject *unicode)
 
    Encode a Unicode object using ASCII and return the result as Python bytes
-   object.  Error handling is "strict".  Return *NULL* if an exception was
+   object.  Error handling is "strict".  Return ``NULL`` if an exception was
    raised by the codec.
 
 
 .. c:function:: PyObject* PyUnicode_EncodeASCII(const Py_UNICODE *s, Py_ssize_t size, const char *errors)
 
    Encode the :c:type:`Py_UNICODE` buffer of the given *size* using ASCII and
-   return a Python bytes object.  Return *NULL* if an exception was raised by
+   return a Python bytes object.  Return ``NULL`` if an exception was raised by
    the codec.
 
    .. deprecated-removed:: 3.3 4.0
@@ -1436,10 +1436,10 @@ These are the mapping codec APIs:
                               PyObject *mapping, const char *errors)
 
    Create a Unicode object by decoding *size* bytes of the encoded string *s*
-   using the given *mapping* object.  Return *NULL* if an exception was raised
+   using the given *mapping* object.  Return ``NULL`` if an exception was raised
    by the codec.
 
-   If *mapping* is *NULL*, Latin-1 decoding will be applied.  Else
+   If *mapping* is ``NULL``, Latin-1 decoding will be applied.  Else
    *mapping* must map bytes ordinals (integers in the range from 0 to 255)
    to Unicode strings, integers (which are then interpreted as Unicode
    ordinals) or ``None``.  Unmapped data bytes -- ones which cause a
@@ -1451,7 +1451,7 @@ These are the mapping codec APIs:
 .. c:function:: PyObject* PyUnicode_AsCharmapString(PyObject *unicode, PyObject *mapping)
 
    Encode a Unicode object using the given *mapping* object and return the
-   result as a bytes object.  Error handling is "strict".  Return *NULL* if an
+   result as a bytes object.  Error handling is "strict".  Return ``NULL`` if an
    exception was raised by the codec.
 
    The *mapping* object must map Unicode ordinal integers to bytes objects,
@@ -1464,7 +1464,7 @@ These are the mapping codec APIs:
                               PyObject *mapping, const char *errors)
 
    Encode the :c:type:`Py_UNICODE` buffer of the given *size* using the given
-   *mapping* object and return the result as a bytes object.  Return *NULL* if
+   *mapping* object and return the result as a bytes object.  Return ``NULL`` if
    an exception was raised by the codec.
 
    .. deprecated-removed:: 3.3 4.0
@@ -1479,7 +1479,7 @@ The following codec API is special in that maps Unicode to Unicode.
                               PyObject *mapping, const char *errors)
 
    Translate a Unicode object using the given *mapping* object and return the
-   resulting Unicode object.  Return *NULL* if an exception was raised by the
+   resulting Unicode object.  Return ``NULL`` if an exception was raised by the
    codec.
 
    The *mapping* object must map Unicode ordinal integers to Unicode strings,
@@ -1493,7 +1493,7 @@ The following codec API is special in that maps Unicode to Unicode.
 
    Translate a :c:type:`Py_UNICODE` buffer of the given *size* by applying a
    character *mapping* table to it and return the resulting Unicode object.
-   Return *NULL* when an exception was raised by the codec.
+   Return ``NULL`` when an exception was raised by the codec.
 
    .. deprecated-removed:: 3.3 4.0
       Part of the old-style :c:type:`Py_UNICODE` API; please migrate to using
@@ -1512,14 +1512,14 @@ the user settings on the machine running the codec.
 .. c:function:: PyObject* PyUnicode_DecodeMBCS(const char *s, Py_ssize_t size, const char *errors)
 
    Create a Unicode object by decoding *size* bytes of the MBCS encoded string *s*.
-   Return *NULL* if an exception was raised by the codec.
+   Return ``NULL`` if an exception was raised by the codec.
 
 
 .. c:function:: PyObject* PyUnicode_DecodeMBCSStateful(const char *s, Py_ssize_t size, \
                               const char *errors, Py_ssize_t *consumed)
 
-   If *consumed* is *NULL*, behave like :c:func:`PyUnicode_DecodeMBCS`. If
-   *consumed* is not *NULL*, :c:func:`PyUnicode_DecodeMBCSStateful` will not decode
+   If *consumed* is ``NULL``, behave like :c:func:`PyUnicode_DecodeMBCS`. If
+   *consumed* is not ``NULL``, :c:func:`PyUnicode_DecodeMBCSStateful` will not decode
    trailing lead byte and the number of bytes that have been decoded will be stored
    in *consumed*.
 
@@ -1527,14 +1527,14 @@ the user settings on the machine running the codec.
 .. c:function:: PyObject* PyUnicode_AsMBCSString(PyObject *unicode)
 
    Encode a Unicode object using MBCS and return the result as Python bytes
-   object.  Error handling is "strict".  Return *NULL* if an exception was
+   object.  Error handling is "strict".  Return ``NULL`` if an exception was
    raised by the codec.
 
 
 .. c:function:: PyObject* PyUnicode_EncodeCodePage(int code_page, PyObject *unicode, const char *errors)
 
    Encode the Unicode object using the specified code page and return a Python
-   bytes object.  Return *NULL* if an exception was raised by the codec. Use
+   bytes object.  Return ``NULL`` if an exception was raised by the codec. Use
    :c:data:`CP_ACP` code page to get the MBCS encoder.
 
    .. versionadded:: 3.3
@@ -1543,7 +1543,7 @@ the user settings on the machine running the codec.
 .. c:function:: PyObject* PyUnicode_EncodeMBCS(const Py_UNICODE *s, Py_ssize_t size, const char *errors)
 
    Encode the :c:type:`Py_UNICODE` buffer of the given *size* using MBCS and return
-   a Python bytes object.  Return *NULL* if an exception was raised by the
+   a Python bytes object.  Return ``NULL`` if an exception was raised by the
    codec.
 
    .. deprecated-removed:: 3.3 4.0
@@ -1565,7 +1565,7 @@ The following APIs are capable of handling Unicode objects and strings on input
 (we refer to them as strings in the descriptions) and return Unicode objects or
 integers as appropriate.
 
-They all return *NULL* or ``-1`` if an exception occurs.
+They all return ``NULL`` or ``-1`` if an exception occurs.
 
 
 .. c:function:: PyObject* PyUnicode_Concat(PyObject *left, PyObject *right)
@@ -1575,7 +1575,7 @@ They all return *NULL* or ``-1`` if an exception occurs.
 
 .. c:function:: PyObject* PyUnicode_Split(PyObject *s, PyObject *sep, Py_ssize_t maxsplit)
 
-   Split a string giving a list of Unicode strings.  If *sep* is *NULL*, splitting
+   Split a string giving a list of Unicode strings.  If *sep* is ``NULL``, splitting
    will be done at all whitespace substrings.  Otherwise, splits occur at the given
    separator.  At most *maxsplit* splits will be done.  If negative, no limit is
    set.  Separators are not included in the resulting list.
@@ -1601,7 +1601,7 @@ They all return *NULL* or ``-1`` if an exception occurs.
    and sequences work well.  Unmapped character ordinals (ones which cause a
    :exc:`LookupError`) are left untouched and are copied as-is.
 
-   *errors* has the usual meaning for codecs. It may be *NULL* which indicates to
+   *errors* has the usual meaning for codecs. It may be ``NULL`` which indicates to
    use the default error handling.
 
 
diff --git a/Doc/c-api/veryhigh.rst b/Doc/c-api/veryhigh.rst
index 67dc11dfa9a53..4ca0e0bb8bfc4 100644
--- a/Doc/c-api/veryhigh.rst
+++ b/Doc/c-api/veryhigh.rst
@@ -52,7 +52,7 @@ the same library that the Python runtime is using.
 .. c:function:: int PyRun_AnyFile(FILE *fp, const char *filename)
 
    This is a simplified interface to :c:func:`PyRun_AnyFileExFlags` below, leaving
-   *closeit* set to ``0`` and *flags* set to *NULL*.
+   *closeit* set to ``0`` and *flags* set to ``NULL``.
 
 
 .. c:function:: int PyRun_AnyFileFlags(FILE *fp, const char *filename, PyCompilerFlags *flags)
@@ -64,7 +64,7 @@ the same library that the Python runtime is using.
 .. c:function:: int PyRun_AnyFileEx(FILE *fp, const char *filename, int closeit)
 
    This is a simplified interface to :c:func:`PyRun_AnyFileExFlags` below, leaving
-   the *flags* argument set to *NULL*.
+   the *flags* argument set to ``NULL``.
 
 
 .. c:function:: int PyRun_AnyFileExFlags(FILE *fp, const char *filename, int closeit, PyCompilerFlags *flags)
@@ -73,7 +73,7 @@ the same library that the Python runtime is using.
    terminal input or Unix pseudo-terminal), return the value of
    :c:func:`PyRun_InteractiveLoop`, otherwise return the result of
    :c:func:`PyRun_SimpleFile`.  *filename* is decoded from the filesystem
-   encoding (:func:`sys.getfilesystemencoding`).  If *filename* is *NULL*, this
+   encoding (:func:`sys.getfilesystemencoding`).  If *filename* is ``NULL``, this
    function uses ``"???"`` as the filename.
 
 
@@ -99,13 +99,13 @@ the same library that the Python runtime is using.
 .. c:function:: int PyRun_SimpleFile(FILE *fp, const char *filename)
 
    This is a simplified interface to :c:func:`PyRun_SimpleFileExFlags` below,
-   leaving *closeit* set to ``0`` and *flags* set to *NULL*.
+   leaving *closeit* set to ``0`` and *flags* set to ``NULL``.
 
 
 .. c:function:: int PyRun_SimpleFileEx(FILE *fp, const char *filename, int closeit)
 
    This is a simplified interface to :c:func:`PyRun_SimpleFileExFlags` below,
-   leaving *flags* set to *NULL*.
+   leaving *flags* set to ``NULL``.
 
 
 .. c:function:: int PyRun_SimpleFileExFlags(FILE *fp, const char *filename, int closeit, PyCompilerFlags *flags)
@@ -124,7 +124,7 @@ the same library that the Python runtime is using.
 .. c:function:: int PyRun_InteractiveOne(FILE *fp, const char *filename)
 
    This is a simplified interface to :c:func:`PyRun_InteractiveOneFlags` below,
-   leaving *flags* set to *NULL*.
+   leaving *flags* set to ``NULL``.
 
 
 .. c:function:: int PyRun_InteractiveOneFlags(FILE *fp, const char *filename, PyCompilerFlags *flags)
@@ -144,7 +144,7 @@ the same library that the Python runtime is using.
 .. c:function:: int PyRun_InteractiveLoop(FILE *fp, const char *filename)
 
    This is a simplified interface to :c:func:`PyRun_InteractiveLoopFlags` below,
-   leaving *flags* set to *NULL*.
+   leaving *flags* set to ``NULL``.
 
 
 .. c:function:: int PyRun_InteractiveLoopFlags(FILE *fp, const char *filename, PyCompilerFlags *flags)
@@ -173,13 +173,13 @@ the same library that the Python runtime is using.
    ``char *func(FILE *stdin, FILE *stdout, char *prompt)``,
    overriding the default function used to read a single line of input
    at the interpreter's prompt.  The function is expected to output
-   the string *prompt* if it's not *NULL*, and then read a line of
+   the string *prompt* if it's not ``NULL``, and then read a line of
    input from the provided standard input file, returning the
    resulting string.  For example, The :mod:`readline` module sets
    this hook to provide line-editing and tab-completion features.
 
    The result must be a string allocated by :c:func:`PyMem_RawMalloc` or
-   :c:func:`PyMem_RawRealloc`, or *NULL* if an error occurred.
+   :c:func:`PyMem_RawRealloc`, or ``NULL`` if an error occurred.
 
    .. versionchanged:: 3.4
       The result must be allocated by :c:func:`PyMem_RawMalloc` or
@@ -191,14 +191,14 @@ the same library that the Python runtime is using.
 
    This is a simplified interface to
    :c:func:`PyParser_SimpleParseStringFlagsFilename` below, leaving  *filename* set
-   to *NULL* and *flags* set to ``0``.
+   to ``NULL`` and *flags* set to ``0``.
 
 
 .. c:function:: struct _node* PyParser_SimpleParseStringFlags( const char *str, int start, int flags)
 
    This is a simplified interface to
    :c:func:`PyParser_SimpleParseStringFlagsFilename` below, leaving  *filename* set
-   to *NULL*.
+   to ``NULL``.
 
 
 .. c:function:: struct _node* PyParser_SimpleParseStringFlagsFilename( const char *str, const char *filename, int start, int flags)
@@ -225,7 +225,7 @@ the same library that the Python runtime is using.
 .. c:function:: PyObject* PyRun_String(const char *str, int start, PyObject *globals, PyObject *locals)
 
    This is a simplified interface to :c:func:`PyRun_StringFlags` below, leaving
-   *flags* set to *NULL*.
+   *flags* set to ``NULL``.
 
 
 .. c:function:: PyObject* PyRun_StringFlags(const char *str, int start, PyObject *globals, PyObject *locals, PyCompilerFlags *flags)
@@ -236,20 +236,20 @@ the same library that the Python runtime is using.
    that implements the mapping protocol.  The parameter *start* specifies
    the start token that should be used to parse the source code.
 
-   Returns the result of executing the code as a Python object, or *NULL* if an
+   Returns the result of executing the code as a Python object, or ``NULL`` if an
    exception was raised.
 
 
 .. c:function:: PyObject* PyRun_File(FILE *fp, const char *filename, int start, PyObject *globals, PyObject *locals)
 
    This is a simplified interface to :c:func:`PyRun_FileExFlags` below, leaving
-   *closeit* set to ``0`` and *flags* set to *NULL*.
+   *closeit* set to ``0`` and *flags* set to ``NULL``.
 
 
 .. c:function:: PyObject* PyRun_FileEx(FILE *fp, const char *filename, int start, PyObject *globals, PyObject *locals, int closeit)
 
    This is a simplified interface to :c:func:`PyRun_FileExFlags` below, leaving
-   *flags* set to *NULL*.
+   *flags* set to ``NULL``.
 
 
 .. c:function:: PyObject* PyRun_FileFlags(FILE *fp, const char *filename, int start, PyObject *globals, PyObject *locals, PyCompilerFlags *flags)
@@ -270,7 +270,7 @@ the same library that the Python runtime is using.
 .. c:function:: PyObject* Py_CompileString(const char *str, const char *filename, int start)
 
    This is a simplified interface to :c:func:`Py_CompileStringFlags` below, leaving
-   *flags* set to *NULL*.
+   *flags* set to ``NULL``.
 
 
 .. c:function:: PyObject* Py_CompileStringFlags(const char *str, const char *filename, int start, PyCompilerFlags *flags)
@@ -286,7 +286,7 @@ the same library that the Python runtime is using.
    code which can be compiled and should be :const:`Py_eval_input`,
    :const:`Py_file_input`, or :const:`Py_single_input`.  The filename specified by
    *filename* is used to construct the code object and may appear in tracebacks or
-   :exc:`SyntaxError` exception messages.  This returns *NULL* if the code
+   :exc:`SyntaxError` exception messages.  This returns ``NULL`` if the code
    cannot be parsed or compiled.
 
    The integer *optimize* specifies the optimization level of the compiler; a
@@ -309,7 +309,7 @@ the same library that the Python runtime is using.
 
    This is a simplified interface to :c:func:`PyEval_EvalCodeEx`, with just
    the code object, and global and local variables.  The other arguments are
-   set to *NULL*.
+   set to ``NULL``.
 
 
 .. c:function:: PyObject* PyEval_EvalCodeEx(PyObject *co, PyObject *globals, PyObject *locals, PyObject *const *args, int argcount, PyObject *const *kws, int kwcount, PyObject *const *defs, int defcount, PyObject *kwdefs, PyObject *closure)
@@ -386,7 +386,7 @@ the same library that the Python runtime is using.
    executed, it is passed as ``PyCompilerFlags *flags``.  In this case, ``from
    __future__ import`` can modify *flags*.
 
-   Whenever ``PyCompilerFlags *flags`` is *NULL*, :attr:`cf_flags` is treated as
+   Whenever ``PyCompilerFlags *flags`` is ``NULL``, :attr:`cf_flags` is treated as
    equal to ``0``, and any modification due to ``from __future__ import`` is
    discarded.
 
diff --git a/Doc/c-api/weakref.rst b/Doc/c-api/weakref.rst
index 7a4f8615b9c32..e3a9bda54d671 100644
--- a/Doc/c-api/weakref.rst
+++ b/Doc/c-api/weakref.rst
@@ -33,9 +33,9 @@ as much as it can.
    reference object may be returned.  The second parameter, *callback*, can be a
    callable object that receives notification when *ob* is garbage collected; it
    should accept a single parameter, which will be the weak reference object
-   itself. *callback* may also be ``None`` or *NULL*.  If *ob* is not a
+   itself. *callback* may also be ``None`` or ``NULL``.  If *ob* is not a
    weakly-referencable object, or if *callback* is not callable, ``None``, or
-   *NULL*, this will return *NULL* and raise :exc:`TypeError`.
+   ``NULL``, this will return ``NULL`` and raise :exc:`TypeError`.
 
 
 .. c:function:: PyObject* PyWeakref_NewProxy(PyObject *ob, PyObject *callback)
@@ -45,9 +45,9 @@ as much as it can.
    existing proxy object may be returned.  The second parameter, *callback*, can
    be a callable object that receives notification when *ob* is garbage
    collected; it should accept a single parameter, which will be the weak
-   reference object itself. *callback* may also be ``None`` or *NULL*.  If *ob*
+   reference object itself. *callback* may also be ``None`` or ``NULL``.  If *ob*
    is not a weakly-referencable object, or if *callback* is not callable,
-   ``None``, or *NULL*, this will return *NULL* and raise :exc:`TypeError`.
+   ``None``, or ``NULL``, this will return ``NULL`` and raise :exc:`TypeError`.
 
 
 .. c:function:: PyObject* PyWeakref_GetObject(PyObject *ref)
diff --git a/Doc/extending/embedding.rst b/Doc/extending/embedding.rst
index 483bc852f6c77..5f5abdf9c1506 100644
--- a/Doc/extending/embedding.rst
+++ b/Doc/extending/embedding.rst
@@ -196,7 +196,7 @@ function is then made with::
 
    pValue = PyObject_CallObject(pFunc, pArgs);
 
-Upon return of the function, ``pValue`` is either *NULL* or it contains a
+Upon return of the function, ``pValue`` is either ``NULL`` or it contains a
 reference to the return value of the function.  Be sure to release the reference
 after examining the value.
 
diff --git a/Doc/extending/extending.rst b/Doc/extending/extending.rst
index 5b4ea8220e237..7986ea1e15045 100644
--- a/Doc/extending/extending.rst
+++ b/Doc/extending/extending.rst
@@ -117,7 +117,7 @@ store the converted values.  More about this later.
 type and its components have been stored in the variables whose addresses are
 passed.  It returns false (zero) if an invalid argument list was passed.  In the
 latter case it also raises an appropriate exception so the calling function can
-return *NULL* immediately (as we saw in the example).
+return ``NULL`` immediately (as we saw in the example).
 
 
 .. _extending-errors:
@@ -127,8 +127,8 @@ Intermezzo: Errors and Exceptions
 
 An important convention throughout the Python interpreter is the following: when
 a function fails, it should set an exception condition and return an error value
-(usually a *NULL* pointer).  Exceptions are stored in a static global variable
-inside the interpreter; if this variable is *NULL* no exception has occurred.  A
+(usually a ``NULL`` pointer).  Exceptions are stored in a static global variable
+inside the interpreter; if this variable is ``NULL`` no exception has occurred.  A
 second global variable stores the "associated value" of the exception (the
 second argument to :keyword:`raise`).  A third variable contains the stack
 traceback in case the error originated in Python code.  These three variables
@@ -152,13 +152,13 @@ its associated value.  You don't need to :c:func:`Py_INCREF` the objects passed
 to any of these functions.
 
 You can test non-destructively whether an exception has been set with
-:c:func:`PyErr_Occurred`.  This returns the current exception object, or *NULL*
+:c:func:`PyErr_Occurred`.  This returns the current exception object, or ``NULL``
 if no exception has occurred.  You normally don't need to call
 :c:func:`PyErr_Occurred` to see whether an error occurred in a function call,
 since you should be able to tell from the return value.
 
 When a function *f* that calls another function *g* detects that the latter
-fails, *f* should itself return an error value (usually *NULL* or ``-1``).  It
+fails, *f* should itself return an error value (usually ``NULL`` or ``-1``).  It
 should *not* call one of the :c:func:`PyErr_\*` functions --- one has already
 been called by *g*. *f*'s caller is then supposed to also return an error
 indication to *its* caller, again *without* calling :c:func:`PyErr_\*`, and so on
@@ -234,7 +234,7 @@ with an exception object::
 
 Note that the Python name for the exception object is :exc:`spam.error`.  The
 :c:func:`PyErr_NewException` function may create a class with the base class
-being :exc:`Exception` (unless another class is passed in instead of *NULL*),
+being :exc:`Exception` (unless another class is passed in instead of ``NULL``),
 described in :ref:`bltin-exceptions`.
 
 Note also that the :c:data:`SpamError` variable retains a reference to the newly
@@ -278,7 +278,7 @@ statement::
    if (!PyArg_ParseTuple(args, "s", &command))
        return NULL;
 
-It returns *NULL* (the error indicator for functions returning object pointers)
+It returns ``NULL`` (the error indicator for functions returning object pointers)
 if an error is detected in the argument list, relying on the exception set by
 :c:func:`PyArg_ParseTuple`.  Otherwise the string value of the argument has been
 copied to the local variable :c:data:`command`.  This is a pointer assignment and
@@ -308,7 +308,7 @@ macro)::
    return Py_None;
 
 :c:data:`Py_None` is the C name for the special Python object ``None``.  It is a
-genuine Python object rather than a *NULL* pointer, which means "error" in most
+genuine Python object rather than a ``NULL`` pointer, which means "error" in most
 contexts, as we have seen.
 
 
@@ -376,7 +376,7 @@ inserts built-in function objects into the newly created module based upon the
 table (an array of :c:type:`PyMethodDef` structures) found in the module definition.
 :c:func:`PyModule_Create` returns a pointer to the module object
 that it creates.  It may abort with a fatal error for
-certain errors, or return *NULL* if the module could not be initialized
+certain errors, or return ``NULL`` if the module could not be initialized
 satisfactorily. The init function must return the module object to its caller,
 so that it then gets inserted into ``sys.modules``.
 
@@ -526,8 +526,8 @@ This function must be registered with the interpreter using the
 :ref:`parsetuple`.
 
 The macros :c:func:`Py_XINCREF` and :c:func:`Py_XDECREF` increment/decrement the
-reference count of an object and are safe in the presence of *NULL* pointers
-(but note that *temp* will not be  *NULL* in this context).  More info on them
+reference count of an object and are safe in the presence of ``NULL`` pointers
+(but note that *temp* will not be  ``NULL`` in this context).  More info on them
 in section :ref:`refcounts`.
 
 .. index:: single: PyObject_CallObject()
@@ -566,7 +566,7 @@ somehow :c:func:`Py_DECREF` the result, even (especially!) if you are not
 interested in its value.
 
 Before you do this, however, it is important to check that the return value
-isn't *NULL*.  If it is, the Python function terminated by raising an exception.
+isn't ``NULL``.  If it is, the Python function terminated by raising an exception.
 If the C code that called :c:func:`PyObject_CallObject` is called from Python, it
 should now return an error indication to its Python caller, so the interpreter
 can print a stack trace, or the calling Python code can handle the exception.
@@ -723,7 +723,7 @@ The :c:func:`PyArg_ParseTupleAndKeywords` function is declared as follows::
 The *arg* and *format* parameters are identical to those of the
 :c:func:`PyArg_ParseTuple` function.  The *kwdict* parameter is the dictionary of
 keywords received as the third parameter from the Python runtime.  The *kwlist*
-parameter is a *NULL*-terminated list of strings which identify the parameters;
+parameter is a ``NULL``-terminated list of strings which identify the parameters;
 the names are matched with the type information from *format* from left to
 right.  On success, :c:func:`PyArg_ParseTupleAndKeywords` returns true, otherwise
 it returns false and raises an appropriate exception.
@@ -1084,32 +1084,32 @@ NULL Pointers
 -------------
 
 In general, functions that take object references as arguments do not expect you
-to pass them *NULL* pointers, and will dump core (or cause later core dumps) if
-you do so.  Functions that return object references generally return *NULL* only
-to indicate that an exception occurred.  The reason for not testing for *NULL*
+to pass them ``NULL`` pointers, and will dump core (or cause later core dumps) if
+you do so.  Functions that return object references generally return ``NULL`` only
+to indicate that an exception occurred.  The reason for not testing for ``NULL``
 arguments is that functions often pass the objects they receive on to other
-function --- if each function were to test for *NULL*, there would be a lot of
+function --- if each function were to test for ``NULL``, there would be a lot of
 redundant tests and the code would run more slowly.
 
-It is better to test for *NULL* only at the "source:" when a pointer that may be
-*NULL* is received, for example, from :c:func:`malloc` or from a function that
+It is better to test for ``NULL`` only at the "source:" when a pointer that may be
+``NULL`` is received, for example, from :c:func:`malloc` or from a function that
 may raise an exception.
 
-The macros :c:func:`Py_INCREF` and :c:func:`Py_DECREF` do not check for *NULL*
+The macros :c:func:`Py_INCREF` and :c:func:`Py_DECREF` do not check for ``NULL``
 pointers --- however, their variants :c:func:`Py_XINCREF` and :c:func:`Py_XDECREF`
 do.
 
 The macros for checking for a particular object type (``Pytype_Check()``) don't
-check for *NULL* pointers --- again, there is much code that calls several of
+check for ``NULL`` pointers --- again, there is much code that calls several of
 these in a row to test an object against various different expected types, and
-this would generate redundant tests.  There are no variants with *NULL*
+this would generate redundant tests.  There are no variants with ``NULL``
 checking.
 
 The C function calling mechanism guarantees that the argument list passed to C
-functions (``args`` in the examples) is never *NULL* --- in fact it guarantees
+functions (``args`` in the examples) is never ``NULL`` --- in fact it guarantees
 that it is always a tuple [#]_.
 
-It is a severe error to ever let a *NULL* pointer "escape" to the Python user.
+It is a severe error to ever let a ``NULL`` pointer "escape" to the Python user.
 
 .. Frank Stajano:
    A pedagogically buggy example, along the lines of the previous listing, would
@@ -1184,7 +1184,7 @@ different ways between the module providing the code and the client modules.
 
 Whichever method you choose, it's important to name your Capsules properly.
 The function :c:func:`PyCapsule_New` takes a name parameter
-(:c:type:`const char \*`); you're permitted to pass in a *NULL* name, but
+(:c:type:`const char \*`); you're permitted to pass in a ``NULL`` name, but
 we strongly encourage you to specify a name.  Properly named Capsules provide
 a degree of runtime type-safety; there is no feasible way to tell one unnamed
 Capsule from another.
diff --git a/Doc/extending/newtypes.rst b/Doc/extending/newtypes.rst
index 308c06705e8d2..19c5e51537b19 100644
--- a/Doc/extending/newtypes.rst
+++ b/Doc/extending/newtypes.rst
@@ -191,7 +191,7 @@ For every object which can support attributes, the corresponding type must
 provide the functions that control how the attributes are resolved.  There needs
 to be a function which can retrieve attributes (if any are defined), and another
 to set attributes (if setting attributes is allowed).  Removing an attribute is
-a special case, for which the new value passed to the handler is *NULL*.
+a special case, for which the new value passed to the handler is ``NULL``.
 
 Python supports two pairs of attribute handlers; a type that supports attributes
 only needs to implement the functions for one pair.  The difference is that one
@@ -233,9 +233,9 @@ attributes, when the values are computed, or how relevant data is stored.
 When :c:func:`PyType_Ready` is called, it uses three tables referenced by the
 type object to create :term:`descriptor`\s which are placed in the dictionary of the
 type object.  Each descriptor controls access to one attribute of the instance
-object.  Each of the tables is optional; if all three are *NULL*, instances of
+object.  Each of the tables is optional; if all three are ``NULL``, instances of
 the type will only have attributes that are inherited from their base type, and
-should leave the :c:member:`~PyTypeObject.tp_getattro` and :c:member:`~PyTypeObject.tp_setattro` fields *NULL* as
+should leave the :c:member:`~PyTypeObject.tp_getattro` and :c:member:`~PyTypeObject.tp_setattro` fields ``NULL`` as
 well, allowing the base type to handle attributes.
 
 The tables are declared as three fields of the type object::
@@ -244,7 +244,7 @@ The tables are declared as three fields of the type object::
    struct PyMemberDef *tp_members;
    struct PyGetSetDef *tp_getset;
 
-If :c:member:`~PyTypeObject.tp_methods` is not *NULL*, it must refer to an array of
+If :c:member:`~PyTypeObject.tp_methods` is not ``NULL``, it must refer to an array of
 :c:type:`PyMethodDef` structures.  Each entry in the table is an instance of this
 structure::
 
@@ -258,7 +258,7 @@ structure::
 One entry should be defined for each method provided by the type; no entries are
 needed for methods inherited from a base type.  One additional entry is needed
 at the end; it is a sentinel that marks the end of the array.  The
-:attr:`ml_name` field of the sentinel must be *NULL*.
+:attr:`ml_name` field of the sentinel must be ``NULL``.
 
 The second table is used to define attributes which map directly to data stored
 in the instance.  A variety of primitive C types are supported, and access may
@@ -307,7 +307,7 @@ application can use the introspection API to retrieve the descriptor from the
 class object, and get the doc string using its :attr:`__doc__` attribute.
 
 As with the :c:member:`~PyTypeObject.tp_methods` table, a sentinel entry with a :attr:`name` value
-of *NULL* is required.
+of ``NULL`` is required.
 
 .. XXX Descriptors need to be explained in more detail somewhere, but not here.
 
@@ -352,9 +352,9 @@ Here is an example::
 
 The :c:member:`~PyTypeObject.tp_setattr` handler is called when the :meth:`__setattr__` or
 :meth:`__delattr__` method of a class instance would be called.  When an
-attribute should be deleted, the third parameter will be *NULL*.  Here is an
+attribute should be deleted, the third parameter will be ``NULL``.  Here is an
 example that simply raises an exception; if this were really all you wanted, the
-:c:member:`~PyTypeObject.tp_setattr` handler should be set to *NULL*. ::
+:c:member:`~PyTypeObject.tp_setattr` handler should be set to ``NULL``. ::
 
    static int
    newdatatype_setattr(newdatatypeobject *obj, char *name, PyObject *v)
@@ -380,7 +380,7 @@ where the operator is one of ``Py_EQ``, ``Py_NE``, ``Py_LE``, ``Py_GT``,
 ``Py_LT`` or ``Py_GT``.  It should compare the two objects with respect to the
 specified operator and return ``Py_True`` or ``Py_False`` if the comparison is
 successful, ``Py_NotImplemented`` to indicate that comparison is not
-implemented and the other object's comparison method should be tried, or *NULL*
+implemented and the other object's comparison method should be tried, or ``NULL``
 if an exception was set.
 
 Here is a sample implementation, for a datatype that is considered equal if the
@@ -427,7 +427,7 @@ from the type implementation, the older protocols have been defined as optional
 blocks of handlers referenced by the type object.  For newer protocols there are
 additional slots in the main type object, with a flag bit being set to indicate
 that the slots are present and should be checked by the interpreter.  (The flag
-bit does not indicate that the slot values are non-*NULL*. The flag may be set
+bit does not indicate that the slot values are non-``NULL``. The flag may be set
 to indicate the presence of a slot, but a slot may still be unfilled.) ::
 
    PyNumberMethods   *tp_as_number;
@@ -478,9 +478,9 @@ This function takes three arguments:
    :c:func:`PyArg_ParseTuple` to extract the arguments.
 
 #. *kwds* is a dictionary of keyword arguments that were passed. If this is
-   non-*NULL* and you support keyword arguments, use
+   non-``NULL`` and you support keyword arguments, use
    :c:func:`PyArg_ParseTupleAndKeywords` to extract the arguments.  If you
-   do not want to support keyword arguments and this is non-*NULL*, raise a
+   do not want to support keyword arguments and this is non-``NULL``, raise a
    :exc:`TypeError` with a message saying that keyword arguments are not supported.
 
 Here is a toy ``tp_call`` implementation::
@@ -512,7 +512,7 @@ Here is a toy ``tp_call`` implementation::
 These functions provide support for the iterator protocol.  Both handlers
 take exactly one parameter, the instance for which they are being called,
 and return a new reference.  In the case of an error, they should set an
-exception and return *NULL*.  :c:member:`~PyTypeObject.tp_iter` corresponds
+exception and return ``NULL``.  :c:member:`~PyTypeObject.tp_iter` corresponds
 to the Python :meth:`__iter__` method, while :c:member:`~PyTypeObject.tp_iternext`
 corresponds to the Python :meth:`~iterator.__next__` method.
 
@@ -534,11 +534,11 @@ and :c:member:`~PyTypeObject.tp_iternext`.  An iterator's
 to the iterator.  Its :c:member:`~PyTypeObject.tp_iternext` handler should
 return a new reference to the next object in the iteration, if there is one.
 If the iteration has reached the end, :c:member:`~PyTypeObject.tp_iternext`
-may return *NULL* without setting an exception, or it may set
-:exc:`StopIteration` *in addition* to returning *NULL*; avoiding
+may return ``NULL`` without setting an exception, or it may set
+:exc:`StopIteration` *in addition* to returning ``NULL``; avoiding
 the exception can yield slightly better performance.  If an actual error
 occurs, :c:member:`~PyTypeObject.tp_iternext` should always set an exception
-and return *NULL*.
+and return ``NULL``.
 
 
 .. _weakref-support:
@@ -557,7 +557,7 @@ For an object to be weakly referencable, the extension type must do two things:
 
 #. Include a :c:type:`PyObject\*` field in the C object structure dedicated to
    the weak reference mechanism.  The object's constructor should leave it
-   *NULL* (which is automatic when using the default
+   ``NULL`` (which is automatic when using the default
    :c:member:`~PyTypeObject.tp_alloc`).
 
 #. Set the :c:member:`~PyTypeObject.tp_weaklistoffset` type member
@@ -582,7 +582,7 @@ And the corresponding member in the statically-declared type object::
 
 The only further addition is that ``tp_dealloc`` needs to clear any weak
 references (by calling :c:func:`PyObject_ClearWeakRefs`) if the field is
-non-*NULL*::
+non-``NULL``::
 
    static void
    Trivial_dealloc(TrivialObject *self)
diff --git a/Doc/extending/newtypes_tutorial.rst b/Doc/extending/newtypes_tutorial.rst
index b2c819c858237..0eb6ffd026f49 100644
--- a/Doc/extending/newtypes_tutorial.rst
+++ b/Doc/extending/newtypes_tutorial.rst
@@ -177,7 +177,7 @@ Everything else in the file should be familiar, except for some code in
 
 This initializes the :class:`Custom` type, filling in a number of members
 to the appropriate default values, including :attr:`ob_type` that we initially
-set to *NULL*. ::
+set to ``NULL``. ::
 
    Py_INCREF(&CustomType);
    if (PyModule_AddObject(m, "Custom", (PyObject *) &CustomType) < 0) {
@@ -275,7 +275,7 @@ which is assigned to the :c:member:`~PyTypeObject.tp_dealloc` member::
 
 This method first clears the reference counts of the two Python attributes.
 :c:func:`Py_XDECREF` correctly handles the case where its argument is
-*NULL* (which might happen here if ``tp_new`` failed midway).  It then
+``NULL`` (which might happen here if ``tp_new`` failed midway).  It then
 calls the :c:member:`~PyTypeObject.tp_free` member of the object's type
 (computed by ``Py_TYPE(self)``) to free the object's memory.  Note that
 the object's type might not be :class:`CustomType`, because the object may
@@ -321,7 +321,7 @@ objects of the type.  It is exposed in Python as the :meth:`__new__` method.
 It is not required to define a ``tp_new`` member, and indeed many extension
 types will simply reuse :c:func:`PyType_GenericNew` as done in the first
 version of the ``Custom`` type above.  In this case, we use the ``tp_new``
-handler to initialize the ``first`` and ``last`` attributes to non-*NULL*
+handler to initialize the ``first`` and ``last`` attributes to non-``NULL``
 default values.
 
 ``tp_new`` is passed the type being instantiated (not necessarily ``CustomType``,
@@ -341,7 +341,7 @@ slot to allocate memory::
    self = (CustomObject *) type->tp_alloc(type, 0);
 
 Since memory allocation may fail, we must check the :c:member:`~PyTypeObject.tp_alloc`
-result against *NULL* before proceeding.
+result against ``NULL`` before proceeding.
 
 .. note::
    We didn't fill the :c:member:`~PyTypeObject.tp_alloc` slot ourselves. Rather
@@ -455,9 +455,9 @@ below for details.
 A disadvantage of this approach is that it doesn't provide a way to restrict the
 types of objects that can be assigned to the Python attributes.  We expect the
 first and last names to be strings, but any Python objects can be assigned.
-Further, the attributes can be deleted, setting the C pointers to *NULL*.  Even
-though we can make sure the members are initialized to non-*NULL* values, the
-members can be set to *NULL* if the attributes are deleted.
+Further, the attributes can be deleted, setting the C pointers to ``NULL``.  Even
+though we can make sure the members are initialized to non-``NULL`` values, the
+members can be set to ``NULL`` if the attributes are deleted.
 
 We define a single method, :meth:`Custom.name()`, that outputs the objects name as the
 concatenation of the first and last names. ::
@@ -489,8 +489,8 @@ equivalent to the Python method:
        return "%s %s" % (self.first, self.last)
 
 Note that we have to check for the possibility that our :attr:`first` and
-:attr:`last` members are *NULL*.  This is because they can be deleted, in which
-case they are set to *NULL*.  It would be better to prevent deletion of these
+:attr:`last` members are ``NULL``.  This is because they can be deleted, in which
+case they are set to ``NULL``.  It would be better to prevent deletion of these
 attributes and to restrict the attribute values to be strings.  We'll see how to
 do that in the next section.
 
@@ -584,7 +584,7 @@ could, for example, be used to allow a single set of getter and setter functions
 that decide the attribute to get or set based on data in the closure.)
 
 The setter function is passed the :class:`Custom` object, the new value, and the
-closure.  The new value may be *NULL*, in which case the attribute is being
+closure.  The new value may be ``NULL``, in which case the attribute is being
 deleted.  In our setter, we raise an error if the attribute is deleted or if its
 new value is not a string.
 
@@ -603,7 +603,7 @@ and register it in the :c:member:`~PyTypeObject.tp_getset` slot::
    .tp_getset = Custom_getsetters,
 
 The last item in a :c:type:`PyGetSetDef` structure is the "closure" mentioned
-above.  In this case, we aren't using a closure, so we just pass *NULL*.
+above.  In this case, we aren't using a closure, so we just pass ``NULL``.
 
 We also remove the member definitions for these attributes::
 
@@ -643,7 +643,7 @@ allow strings [#]_ to be passed::
    }
 
 With these changes, we can assure that the ``first`` and ``last`` members are
-never *NULL* so we can remove checks for *NULL* values in almost all cases.
+never ``NULL`` so we can remove checks for ``NULL`` values in almost all cases.
 This means that most of the :c:func:`Py_XDECREF` calls can be converted to
 :c:func:`Py_DECREF` calls.  The only place we can't change these calls is in
 the ``tp_dealloc`` implementation, where there is the possibility that the
@@ -749,7 +749,7 @@ participate in cycles::
 Notice the use of the :c:func:`Py_CLEAR` macro.  It is the recommended and safe
 way to clear data attributes of arbitrary types while decrementing
 their reference counts.  If you were to call :c:func:`Py_XDECREF` instead
-on the attribute before setting it to *NULL*, there is a possibility
+on the attribute before setting it to ``NULL``, there is a possibility
 that the attribute's destructor would call back into code that reads the
 attribute again (*especially* if there is a reference cycle).
 
diff --git a/Doc/faq/extending.rst b/Doc/faq/extending.rst
index 2ad2765870525..aecb56eaa4fd2 100644
--- a/Doc/faq/extending.rst
+++ b/Doc/faq/extending.rst
@@ -89,7 +89,7 @@ For bytes, :c:func:`PyBytes_Size` returns its length and
 length.  Note that Python bytes objects may contain null bytes so C's
 :c:func:`strlen` should not be used.
 
-To test the type of an object, first make sure it isn't *NULL*, and then use
+To test the type of an object, first make sure it isn't ``NULL``, and then use
 :c:func:`PyBytes_Check`, :c:func:`PyTuple_Check`, :c:func:`PyList_Check`, etc.
 
 There is also a high-level API to Python objects which is provided by the
diff --git a/Doc/library/ctypes.rst b/Doc/library/ctypes.rst
index eb0bcf4ae2e36..4504a800c43ca 100644
--- a/Doc/library/ctypes.rst
+++ b/Doc/library/ctypes.rst
@@ -1083,7 +1083,7 @@ An extended example which also demonstrates the use of pointers accesses the
 Quoting the docs for that value:
 
    This pointer is initialized to point to an array of :c:type:`struct _frozen`
-   records, terminated by one whose members are all *NULL* or zero.  When a frozen
+   records, terminated by one whose members are all ``NULL`` or zero.  When a frozen
    module is imported, it is searched in this table.  Third-party code could play
    tricks with this to provide a dynamically created collection of frozen modules.
 
diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst
index 822ba81be8a7f..7b886e7c27a48 100644
--- a/Doc/whatsnew/3.4.rst
+++ b/Doc/whatsnew/3.4.rst
@@ -2503,7 +2503,7 @@ Changes in the C API
 
 * The result of the :c:data:`PyOS_ReadlineFunctionPointer` callback must
   now be a string allocated by :c:func:`PyMem_RawMalloc` or
-  :c:func:`PyMem_RawRealloc`, or *NULL* if an error occurred, instead of a
+  :c:func:`PyMem_RawRealloc`, or ``NULL`` if an error occurred, instead of a
   string allocated by :c:func:`PyMem_Malloc` or :c:func:`PyMem_Realloc`
   (:issue:`16742`)
 
diff --git a/Doc/whatsnew/3.7.rst b/Doc/whatsnew/3.7.rst
index 1ec8e0c04c76b..2197bd9739299 100644
--- a/Doc/whatsnew/3.7.rst
+++ b/Doc/whatsnew/3.7.rst
@@ -1707,7 +1707,7 @@ The type of results of :c:func:`PyThread_start_new_thread` and
 (Contributed by Serhiy Storchaka in :issue:`6532`.)
 
 :c:func:`PyUnicode_AsWideCharString` now raises a :exc:`ValueError` if the
-second argument is *NULL* and the :c:type:`wchar_t*` string contains null
+second argument is ``NULL`` and the :c:type:`wchar_t*` string contains null
 characters.  (Contributed by Serhiy Storchaka in :issue:`30708`.)
 
 Changes to the startup sequence and the management of dynamic memory
diff --git a/Include/abstract.h b/Include/abstract.h
index c226aab9b7304..777fd7077c2d6 100644
--- a/Include/abstract.h
+++ b/Include/abstract.h
@@ -144,7 +144,7 @@ extern "C" {
 /* Call a callable Python object 'callable' with arguments given by the
    tuple 'args' and keywords arguments given by the dictionary 'kwargs'.
 
-   'args' must not be *NULL*, use an empty tuple if no arguments are
+   'args' must not be NULL, use an empty tuple if no arguments are
    needed. If no named arguments are needed, 'kwargs' can be NULL.
 
    This is the equivalent of the Python expression:
@@ -154,9 +154,9 @@ PyAPI_FUNC(PyObject *) PyObject_Call(PyObject *callable,
 
 
 /* Call a callable Python object 'callable', with arguments given by the
-   tuple 'args'.  If no arguments are needed, then 'args' can be *NULL*.
+   tuple 'args'.  If no arguments are needed, then 'args' can be NULL.
 
-   Returns the result of the call on success, or *NULL* on failure.
+   Returns the result of the call on success, or NULL on failure.
 
    This is the equivalent of the Python expression:
    callable(*args). */


From webhook-mailer at python.org  Wed Oct 30 15:37:22 2019
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Wed, 30 Oct 2019 19:37:22 -0000
Subject: [Python-checkins] bpo-38600: NULL -> ``NULL``. (GH-17001)
Message-ID: <mailman.193.1572464243.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/e835b31d2b212c3c7820364398979cae2a9740b2
commit: e835b31d2b212c3c7820364398979cae2a9740b2
branch: master
author: Serhiy Storchaka <storchaka at gmail.com>
committer: GitHub <noreply at github.com>
date: 2019-10-30T21:37:16+02:00
summary:

bpo-38600: NULL -> ``NULL``. (GH-17001)

Also fix some other formatting.

files:
M Doc/c-api/arg.rst
M Doc/c-api/buffer.rst
M Doc/c-api/conversion.rst
M Doc/c-api/coro.rst
M Doc/c-api/exceptions.rst
M Doc/c-api/function.rst
M Doc/c-api/gen.rst
M Doc/c-api/import.rst
M Doc/c-api/init.rst
M Doc/c-api/init_config.rst
M Doc/c-api/marshal.rst
M Doc/c-api/memory.rst
M Doc/c-api/number.rst
M Doc/c-api/object.rst
M Doc/c-api/tuple.rst
M Doc/c-api/type.rst
M Doc/c-api/typeobj.rst
M Doc/c-api/unicode.rst
M Doc/c-api/veryhigh.rst
M Doc/extending/extending.rst
M Doc/howto/clinic.rst
M Doc/library/ctypes.rst
M Doc/library/gc.rst
M Doc/library/socket.rst
M Doc/library/winreg.rst
M Doc/whatsnew/2.7.rst
M Doc/whatsnew/3.4.rst

diff --git a/Doc/c-api/arg.rst b/Doc/c-api/arg.rst
index 594fef29f2a33..f17c63d0b9dc9 100644
--- a/Doc/c-api/arg.rst
+++ b/Doc/c-api/arg.rst
@@ -345,7 +345,7 @@ Other objects
    If the *converter* returns ``Py_CLEANUP_SUPPORTED``, it may get called a
    second time if the argument parsing eventually fails, giving the converter a
    chance to release any memory that it had already allocated. In this second
-   call, the *object* parameter will be NULL; *address* will have the same value
+   call, the *object* parameter will be ``NULL``; *address* will have the same value
    as in the original call.
 
    .. versionchanged:: 3.1
diff --git a/Doc/c-api/buffer.rst b/Doc/c-api/buffer.rst
index f37b0db07cd70..fc1430efa8a4b 100644
--- a/Doc/c-api/buffer.rst
+++ b/Doc/c-api/buffer.rst
@@ -130,7 +130,7 @@ a buffer, see :c:func:`PyObject_GetBuffer`.
    .. c:member:: Py_ssize_t itemsize
 
       Item size in bytes of a single element. Same as the value of :func:`struct.calcsize`
-      called on non-NULL :c:member:`~Py_buffer.format` values.
+      called on non-``NULL`` :c:member:`~Py_buffer.format` values.
 
       Important exception: If a consumer requests a buffer without the
       :c:macro:`PyBUF_FORMAT` flag, :c:member:`~Py_buffer.format` will
@@ -199,7 +199,7 @@ a buffer, see :c:func:`PyObject_GetBuffer`.
       memory block).
 
       If all suboffsets are negative (i.e. no de-referencing is needed), then
-      this field must be NULL (the default value).
+      this field must be ``NULL`` (the default value).
 
       This type of array representation is used by the Python Imaging Library
       (PIL). See `complex arrays`_ for further information how to access elements
@@ -407,7 +407,7 @@ to two ``char x[2][3]`` arrays that can be located anywhere in memory.
 
 
 Here is a function that returns a pointer to the element in an N-D array
-pointed to by an N-dimensional index when there are both non-NULL strides
+pointed to by an N-dimensional index when there are both non-``NULL`` strides
 and suboffsets::
 
    void *get_item_pointer(int ndim, void *buf, Py_ssize_t *strides,
@@ -522,4 +522,4 @@ Buffer-related functions
 
    If this function is used as part of a :ref:`getbufferproc <buffer-structs>`,
    *exporter* MUST be set to the exporting object and *flags* must be passed
-   unmodified. Otherwise, *exporter* MUST be NULL.
+   unmodified. Otherwise, *exporter* MUST be ``NULL``.
diff --git a/Doc/c-api/conversion.rst b/Doc/c-api/conversion.rst
index c02fa60ae9a7c..b310fcb5e4f91 100644
--- a/Doc/c-api/conversion.rst
+++ b/Doc/c-api/conversion.rst
@@ -32,7 +32,7 @@ NULL``.
 
 If the platform doesn't have :c:func:`vsnprintf` and the buffer size needed to
 avoid truncation exceeds *size* by more than 512 bytes, Python aborts with a
-*Py_FatalError*.
+:c:func:`Py_FatalError`.
 
 The return value (*rv*) for these functions should be interpreted as follows:
 
@@ -95,21 +95,21 @@ The following functions provide locale-independent string to number conversions.
    must be 0 and is ignored.  The ``'r'`` format code specifies the
    standard :func:`repr` format.
 
-   *flags* can be zero or more of the values *Py_DTSF_SIGN*,
-   *Py_DTSF_ADD_DOT_0*, or *Py_DTSF_ALT*, or-ed together:
+   *flags* can be zero or more of the values ``Py_DTSF_SIGN``,
+   ``Py_DTSF_ADD_DOT_0``, or ``Py_DTSF_ALT``, or-ed together:
 
-   * *Py_DTSF_SIGN* means to always precede the returned string with a sign
+   * ``Py_DTSF_SIGN`` means to always precede the returned string with a sign
      character, even if *val* is non-negative.
 
-   * *Py_DTSF_ADD_DOT_0* means to ensure that the returned string will not look
+   * ``Py_DTSF_ADD_DOT_0`` means to ensure that the returned string will not look
      like an integer.
 
-   * *Py_DTSF_ALT* means to apply "alternate" formatting rules.  See the
+   * ``Py_DTSF_ALT`` means to apply "alternate" formatting rules.  See the
      documentation for the :c:func:`PyOS_snprintf` ``'#'`` specifier for
      details.
 
-   If *ptype* is non-NULL, then the value it points to will be set to one of
-   *Py_DTST_FINITE*, *Py_DTST_INFINITE*, or *Py_DTST_NAN*, signifying that
+   If *ptype* is non-``NULL``, then the value it points to will be set to one of
+   ``Py_DTST_FINITE``, ``Py_DTST_INFINITE``, or ``Py_DTST_NAN``, signifying that
    *val* is a finite number, an infinite number, or not a number, respectively.
 
    The return value is a pointer to *buffer* with the converted string or
diff --git a/Doc/c-api/coro.rst b/Doc/c-api/coro.rst
index 915c57eeb55b8..2260944a9a93a 100644
--- a/Doc/c-api/coro.rst
+++ b/Doc/c-api/coro.rst
@@ -23,7 +23,7 @@ return.
 
 .. c:function:: int PyCoro_CheckExact(PyObject *ob)
 
-   Return true if *ob*'s type is *PyCoro_Type*; *ob* must not be ``NULL``.
+   Return true if *ob*'s type is :c:type:`PyCoro_Type`; *ob* must not be ``NULL``.
 
 
 .. c:function:: PyObject* PyCoro_New(PyFrameObject *frame, PyObject *name, PyObject *qualname)
diff --git a/Doc/c-api/exceptions.rst b/Doc/c-api/exceptions.rst
index 4cb309521d4d2..c7ba74cc8d587 100644
--- a/Doc/c-api/exceptions.rst
+++ b/Doc/c-api/exceptions.rst
@@ -19,9 +19,9 @@ return ``1`` for success and ``0`` for failure).
 
 Concretely, the error indicator consists of three object pointers: the
 exception's type, the exception's value, and the traceback object.  Any
-of those pointers can be NULL if non-set (although some combinations are
-forbidden, for example you can't have a non-NULL traceback if the exception
-type is NULL).
+of those pointers can be ``NULL`` if non-set (although some combinations are
+forbidden, for example you can't have a non-``NULL`` traceback if the exception
+type is ``NULL``).
 
 When a function must fail because some function it called failed, it generally
 doesn't set the error indicator; the function it called already set it.  It is
@@ -92,7 +92,7 @@ Raising exceptions
 
 These functions help you set the current thread's error indicator.
 For convenience, some of these functions will always return a
-NULL pointer for use in a ``return`` statement.
+``NULL`` pointer for use in a ``return`` statement.
 
 
 .. c:function:: void PyErr_SetString(PyObject *type, const char *message)
diff --git a/Doc/c-api/function.rst b/Doc/c-api/function.rst
index a3cbf569f76f2..bb416f4bb63aa 100644
--- a/Doc/c-api/function.rst
+++ b/Doc/c-api/function.rst
@@ -42,8 +42,8 @@ There are a few functions specific to Python functions.
 .. c:function:: PyObject* PyFunction_NewWithQualName(PyObject *code, PyObject *globals, PyObject *qualname)
 
    As :c:func:`PyFunction_New`, but also allows setting the function object's
-   ``__qualname__`` attribute.  *qualname* should be a unicode object or NULL;
-   if NULL, the ``__qualname__`` attribute is set to the same value as its
+   ``__qualname__`` attribute.  *qualname* should be a unicode object or ``NULL``;
+   if ``NULL``, the ``__qualname__`` attribute is set to the same value as its
    ``__name__`` attribute.
 
    .. versionadded:: 3.3
@@ -75,7 +75,7 @@ There are a few functions specific to Python functions.
 .. c:function:: int PyFunction_SetDefaults(PyObject *op, PyObject *defaults)
 
    Set the argument default values for the function object *op*. *defaults* must be
-   *Py_None* or a tuple.
+   ``Py_None`` or a tuple.
 
    Raises :exc:`SystemError` and returns ``-1`` on failure.
 
@@ -89,7 +89,7 @@ There are a few functions specific to Python functions.
 .. c:function:: int PyFunction_SetClosure(PyObject *op, PyObject *closure)
 
    Set the closure associated with the function object *op*. *closure* must be
-   *Py_None* or a tuple of cell objects.
+   ``Py_None`` or a tuple of cell objects.
 
    Raises :exc:`SystemError` and returns ``-1`` on failure.
 
@@ -103,6 +103,6 @@ There are a few functions specific to Python functions.
 .. c:function:: int PyFunction_SetAnnotations(PyObject *op, PyObject *annotations)
 
    Set the annotations for the function object *op*. *annotations*
-   must be a dictionary or *Py_None*.
+   must be a dictionary or ``Py_None``.
 
    Raises :exc:`SystemError` and returns ``-1`` on failure.
diff --git a/Doc/c-api/gen.rst b/Doc/c-api/gen.rst
index e4b8d255dadba..74410927bfde1 100644
--- a/Doc/c-api/gen.rst
+++ b/Doc/c-api/gen.rst
@@ -27,7 +27,7 @@ than explicitly calling :c:func:`PyGen_New` or :c:func:`PyGen_NewWithQualName`.
 
 .. c:function:: int PyGen_CheckExact(PyObject *ob)
 
-   Return true if *ob*'s type is *PyGen_Type*; *ob* must not be ``NULL``.
+   Return true if *ob*'s type is :c:type:`PyGen_Type`; *ob* must not be ``NULL``.
 
 
 .. c:function:: PyObject* PyGen_New(PyFrameObject *frame)
diff --git a/Doc/c-api/import.rst b/Doc/c-api/import.rst
index ee098768b124e..c6fc33076f0f5 100644
--- a/Doc/c-api/import.rst
+++ b/Doc/c-api/import.rst
@@ -207,8 +207,8 @@ Importing Modules
 .. c:function:: PyObject* PyImport_GetModule(PyObject *name)
 
    Return the already imported module with the given name.  If the
-   module has not been imported yet then returns NULL but does not set
-   an error.  Returns NULL and sets an error if the lookup failed.
+   module has not been imported yet then returns ``NULL`` but does not set
+   an error.  Returns ``NULL`` and sets an error if the lookup failed.
 
    .. versionadded:: 3.7
 
diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst
index 61912b968fd4b..dc30e49682551 100644
--- a/Doc/c-api/init.rst
+++ b/Doc/c-api/init.rst
@@ -329,7 +329,7 @@ Process-wide parameters
    It overrides :envvar:`PYTHONIOENCODING` values, and allows embedding code
    to control IO encoding when the environment variable does not work.
 
-   ``encoding`` and/or ``errors`` may be NULL to use
+   *encoding* and/or *errors* may be ``NULL`` to use
    :envvar:`PYTHONIOENCODING` and/or default values (depending on other
    settings).
 
diff --git a/Doc/c-api/init_config.rst b/Doc/c-api/init_config.rst
index 5e18300471d46..6b16b5bb72859 100644
--- a/Doc/c-api/init_config.rst
+++ b/Doc/c-api/init_config.rst
@@ -61,8 +61,8 @@ PyWideStringList
 
    List of ``wchar_t*`` strings.
 
-   If *length* is non-zero, *items* must be non-NULL and all strings must be
-   non-NULL.
+   If *length* is non-zero, *items* must be non-``NULL`` and all strings must be
+   non-``NULL``.
 
    Methods:
 
@@ -608,7 +608,7 @@ PyConfig
 
       :data:`sys.pycache_prefix`: ``.pyc`` cache prefix.
 
-      If NULL, :data:`sys.pycache_prefix` is set to ``None``.
+      If ``NULL``, :data:`sys.pycache_prefix` is set to ``None``.
 
    .. c:member:: int quiet
 
diff --git a/Doc/c-api/marshal.rst b/Doc/c-api/marshal.rst
index 2f580430237af..7b179e22e290e 100644
--- a/Doc/c-api/marshal.rst
+++ b/Doc/c-api/marshal.rst
@@ -16,7 +16,7 @@ Numeric values are stored with the least significant byte first.
 The module supports two versions of the data format: version 0 is the
 historical version, version 1 shares interned strings in the file, and upon
 unmarshalling.  Version 2 uses a binary format for floating point numbers.
-*Py_MARSHAL_VERSION* indicates the current file format (currently 2).
+``Py_MARSHAL_VERSION`` indicates the current file format (currently 2).
 
 
 .. c:function:: void PyMarshal_WriteLongToFile(long value, FILE *file, int version)
diff --git a/Doc/c-api/memory.rst b/Doc/c-api/memory.rst
index c5b3af64c16a4..ba7bd3b9a5387 100644
--- a/Doc/c-api/memory.rst
+++ b/Doc/c-api/memory.rst
@@ -424,7 +424,7 @@ Customize Memory Allocators
 
    Set the memory block allocator of the specified domain.
 
-   The new allocator must return a distinct non-NULL pointer when requesting
+   The new allocator must return a distinct non-``NULL`` pointer when requesting
    zero bytes.
 
    For the :c:data:`PYMEM_DOMAIN_RAW` domain, the allocator must be
diff --git a/Doc/c-api/number.rst b/Doc/c-api/number.rst
index 55ddb9c0c8206..620204ca8e229 100644
--- a/Doc/c-api/number.rst
+++ b/Doc/c-api/number.rst
@@ -275,8 +275,8 @@ Number Protocol
    convert to a Py_ssize_t value would raise an :exc:`OverflowError`, then the
    *exc* argument is the type of exception that will be raised (usually
    :exc:`IndexError` or :exc:`OverflowError`).  If *exc* is ``NULL``, then the
-   exception is cleared and the value is clipped to *PY_SSIZE_T_MIN* for a negative
-   integer or *PY_SSIZE_T_MAX* for a positive integer.
+   exception is cleared and the value is clipped to ``PY_SSIZE_T_MIN`` for a negative
+   integer or ``PY_SSIZE_T_MAX`` for a positive integer.
 
 
 .. c:function:: int PyIndex_Check(PyObject *o)
diff --git a/Doc/c-api/object.rst b/Doc/c-api/object.rst
index 0b93064028b8e..9d1155184725b 100644
--- a/Doc/c-api/object.rst
+++ b/Doc/c-api/object.rst
@@ -399,7 +399,7 @@ Object Protocol
    To get actual number of arguments, use
    :c:func:`PyVectorcall_NARGS(nargsf) <PyVectorcall_NARGS>`.
 
-   *kwnames* can be either NULL (no keyword arguments) or a tuple of keyword
+   *kwnames* can be either ``NULL`` (no keyword arguments) or a tuple of keyword
    names, which must be strings. In the latter case, the values of the keyword
    arguments are stored in *args* after the positional arguments.
    The number of keyword arguments does not influence *nargsf*.
diff --git a/Doc/c-api/tuple.rst b/Doc/c-api/tuple.rst
index d74d8626d1e92..25df3974e870e 100644
--- a/Doc/c-api/tuple.rst
+++ b/Doc/c-api/tuple.rst
@@ -153,7 +153,7 @@ type.
    | ``name``          | ``const char *``             | name of the struct sequence type     |
    +-------------------+------------------------------+--------------------------------------+
    | ``doc``           | ``const char *``             | pointer to docstring for the type    |
-   |                   |                              | or NULL to omit                      |
+   |                   |                              | or ``NULL`` to omit                  |
    +-------------------+------------------------------+--------------------------------------+
    | ``fields``        | ``PyStructSequence_Field *`` | pointer to ``NULL``-terminated array |
    |                   |                              | with field names of the new type     |
@@ -170,16 +170,16 @@ type.
    :attr:`fields` array of the :c:type:`PyStructSequence_Desc` determines which
    field of the struct sequence is described.
 
-   +-----------+------------------+----------------------------------------+
-   | Field     | C Type           | Meaning                                |
-   +===========+==================+========================================+
-   | ``name``  | ``const char *`` | name for the field or ``NULL`` to end  |
-   |           |                  | the list of named fields, set to       |
-   |           |                  | PyStructSequence_UnnamedField to       |
-   |           |                  | leave unnamed                          |
-   +-----------+------------------+----------------------------------------+
-   | ``doc``   | ``const char *`` | field docstring or ``NULL`` to omit    |
-   +-----------+------------------+----------------------------------------+
+   +-----------+------------------+-----------------------------------------+
+   | Field     | C Type           | Meaning                                 |
+   +===========+==================+=========================================+
+   | ``name``  | ``const char *`` | name for the field or ``NULL`` to end   |
+   |           |                  | the list of named fields, set to        |
+   |           |                  | :c:data:`PyStructSequence_UnnamedField` |
+   |           |                  | to leave unnamed                        |
+   +-----------+------------------+-----------------------------------------+
+   | ``doc``   | ``const char *`` | field docstring or ``NULL`` to omit     |
+   +-----------+------------------+-----------------------------------------+
 
 
 .. c:var:: char* PyStructSequence_UnnamedField
diff --git a/Doc/c-api/type.rst b/Doc/c-api/type.rst
index 34b52f27ecb83..b1b2df9f19042 100644
--- a/Doc/c-api/type.rst
+++ b/Doc/c-api/type.rst
@@ -181,7 +181,7 @@ The following functions and structs are used to create
       * ``Py_nb_add`` to set :c:member:`PyNumberMethods.nb_add`
       * ``Py_sq_length`` to set :c:member:`PySequenceMethods.sq_length`
 
-      The following fields cannot be set using *PyType_Spec* and *PyType_Slot*:
+      The following fields cannot be set using :c:type:`PyType_Spec` and :c:type:`PyType_Slot`:
 
       * :c:member:`~PyTypeObject.tp_dict`
       * :c:member:`~PyTypeObject.tp_mro`
diff --git a/Doc/c-api/typeobj.rst b/Doc/c-api/typeobj.rst
index 181157e08ec43..bff5abfea48ee 100644
--- a/Doc/c-api/typeobj.rst
+++ b/Doc/c-api/typeobj.rst
@@ -740,7 +740,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
    This field is inherited by subtypes together with
    :c:member:`~PyTypeObject.tp_call`: a subtype inherits
    :c:member:`~PyTypeObject.tp_vectorcall_offset` from its base type when
-   the subtype?s :c:member:`~PyTypeObject.tp_call` is NULL.
+   the subtype?s :c:member:`~PyTypeObject.tp_call` is ``NULL``.
 
    Note that `heap types`_ (including subclasses defined in Python) do not
    inherit the :const:`_Py_TPFLAGS_HAVE_VECTORCALL` flag.
@@ -1180,7 +1180,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
 
       This bit is set on *static* subtypes if ``tp_flags`` is not overridden:
       a subtype inherits ``_Py_TPFLAGS_HAVE_VECTORCALL`` from its base type
-      when the subtype?s :c:member:`~PyTypeObject.tp_call` is NULL
+      when the subtype?s :c:member:`~PyTypeObject.tp_call` is ``NULL``
       and the subtype's ``Py_TPFLAGS_HEAPTYPE`` is not set.
 
       `Heap types`_ do not inherit ``_Py_TPFLAGS_HAVE_VECTORCALL``.
@@ -1955,7 +1955,7 @@ This results in types that are limited relative to types defined in Python:
   :ref:`sub-interpreters <sub-interpreter-support>`, so they should not
   include any subinterpreter-specific state.
 
-Also, since *PyTypeObject* is not part of the :ref:`stable ABI <stable>`,
+Also, since :c:type:`PyTypeObject` is not part of the :ref:`stable ABI <stable>`,
 any extension modules using static types must be compiled for a specific
 Python minor version.
 
diff --git a/Doc/c-api/unicode.rst b/Doc/c-api/unicode.rst
index 0359f5ec36bed..2bf4a0f56bc8a 100644
--- a/Doc/c-api/unicode.rst
+++ b/Doc/c-api/unicode.rst
@@ -526,9 +526,9 @@ APIs:
    .. note::
       The width formatter unit is number of characters rather than bytes.
       The precision formatter unit is number of bytes for ``"%s"`` and
-      ``"%V"`` (if the ``PyObject*`` argument is NULL), and a number of
+      ``"%V"`` (if the ``PyObject*`` argument is ``NULL``), and a number of
       characters for ``"%A"``, ``"%U"``, ``"%S"``, ``"%R"`` and ``"%V"``
-      (if the ``PyObject*`` argument is not NULL).
+      (if the ``PyObject*`` argument is not ``NULL``).
 
    .. [1] For integer specifiers (d, u, ld, li, lu, lld, lli, llu, zd, zi,
       zu, i, x): the 0-conversion flag has effect even when a precision is given.
@@ -1172,7 +1172,7 @@ These are the UTF-32 codec APIs:
    If byteorder is ``0``, the output string will always start with the Unicode BOM
    mark (U+FEFF). In the other two modes, no BOM mark is prepended.
 
-   If *Py_UNICODE_WIDE* is not defined, surrogate pairs will be output
+   If ``Py_UNICODE_WIDE`` is not defined, surrogate pairs will be output
    as a single code point.
 
    Return ``NULL`` if an exception was raised by the codec.
@@ -1246,7 +1246,7 @@ These are the UTF-16 codec APIs:
    If byteorder is ``0``, the output string will always start with the Unicode BOM
    mark (U+FEFF). In the other two modes, no BOM mark is prepended.
 
-   If *Py_UNICODE_WIDE* is defined, a single :c:type:`Py_UNICODE` value may get
+   If ``Py_UNICODE_WIDE`` is defined, a single :c:type:`Py_UNICODE` value may get
    represented as a surrogate pair. If it is not defined, each :c:type:`Py_UNICODE`
    values is interpreted as a UCS-2 character.
 
diff --git a/Doc/c-api/veryhigh.rst b/Doc/c-api/veryhigh.rst
index fce39b5c81b2a..d6aecc394c9e7 100644
--- a/Doc/c-api/veryhigh.rst
+++ b/Doc/c-api/veryhigh.rst
@@ -80,7 +80,7 @@ the same library that the Python runtime is using.
 .. c:function:: int PyRun_SimpleString(const char *command)
 
    This is a simplified interface to :c:func:`PyRun_SimpleStringFlags` below,
-   leaving the *PyCompilerFlags\** argument set to NULL.
+   leaving the :c:type:`PyCompilerFlags`\* argument set to ``NULL``.
 
 
 .. c:function:: int PyRun_SimpleStringFlags(const char *command, PyCompilerFlags *flags)
diff --git a/Doc/extending/extending.rst b/Doc/extending/extending.rst
index 7986ea1e15045..5b32a2cdc5506 100644
--- a/Doc/extending/extending.rst
+++ b/Doc/extending/extending.rst
@@ -536,7 +536,7 @@ Later, when it is time to call the function, you call the C function
 :c:func:`PyObject_CallObject`.  This function has two arguments, both pointers to
 arbitrary Python objects: the Python function, and the argument list.  The
 argument list must always be a tuple object, whose length is the number of
-arguments.  To call the Python function with no arguments, pass in NULL, or
+arguments.  To call the Python function with no arguments, pass in ``NULL``, or
 an empty tuple; to call it with one argument, pass a singleton tuple.
 :c:func:`Py_BuildValue` returns a tuple when its format string consists of zero
 or more format codes between parentheses.  For example::
diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst
index cfd9f2e4075c4..50041829b8c38 100644
--- a/Doc/howto/clinic.rst
+++ b/Doc/howto/clinic.rst
@@ -1070,7 +1070,7 @@ Currently Argument Clinic supports only a few return converters:
     DecodeFSDefault
 
 None of these take parameters.  For the first three, return -1 to indicate
-error.  For ``DecodeFSDefault``, the return type is ``const char *``; return a NULL
+error.  For ``DecodeFSDefault``, the return type is ``const char *``; return a ``NULL``
 pointer to indicate an error.
 
 (There's also an experimental ``NoneType`` converter, which lets you
diff --git a/Doc/library/ctypes.rst b/Doc/library/ctypes.rst
index 4504a800c43ca..1f84b7ab29839 100644
--- a/Doc/library/ctypes.rst
+++ b/Doc/library/ctypes.rst
@@ -152,8 +152,8 @@ the ``time()`` function, which returns system time in seconds since the Unix
 epoch, and the ``GetModuleHandleA()`` function, which returns a win32 module
 handle.
 
-This example calls both functions with a NULL pointer (``None`` should be used
-as the NULL pointer)::
+This example calls both functions with a ``NULL`` pointer (``None`` should be used
+as the ``NULL`` pointer)::
 
    >>> print(libc.time(None))  # doctest: +SKIP
    1150640792
@@ -1110,7 +1110,7 @@ Since ``table`` is a ``pointer`` to the array of ``struct_frozen`` records, we
 can iterate over it, but we just have to make sure that our loop terminates,
 because pointers have no size. Sooner or later it would probably crash with an
 access violation or whatever, so it's better to break out of the loop when we
-hit the NULL entry::
+hit the ``NULL`` entry::
 
    >>> for item in table:
    ...     if item.name is None:
diff --git a/Doc/library/gc.rst b/Doc/library/gc.rst
index 084cd6ac257eb..366861c524c3d 100644
--- a/Doc/library/gc.rst
+++ b/Doc/library/gc.rst
@@ -212,7 +212,7 @@ values but should not rebind them):
    A list of objects which the collector found to be unreachable but could
    not be freed (uncollectable objects).  Starting with Python 3.4, this
    list should be empty most of the time, except when using instances of
-   C extension types with a non-NULL ``tp_del`` slot.
+   C extension types with a non-``NULL`` ``tp_del`` slot.
 
    If :const:`DEBUG_SAVEALL` is set, then all unreachable objects will be
    added to this list rather than freed.
diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst
index d443e972fe98b..62552bcbe24c1 100755
--- a/Doc/library/socket.rst
+++ b/Doc/library/socket.rst
@@ -1675,9 +1675,9 @@ to sockets.
    ``None`` or a :term:`bytes-like object` representing a buffer. In the later
    case it is up to the caller to ensure that the bytestring contains the
    proper bits (see the optional built-in module :mod:`struct` for a way to
-   encode C structures as bytestrings). When value is set to ``None``,
-   optlen argument is required. It's equivalent to call setsockopt C
-   function with optval=NULL and optlen=optlen.
+   encode C structures as bytestrings). When *value* is set to ``None``,
+   *optlen* argument is required. It's equivalent to call :c:func:`setsockopt` C
+   function with ``optval=NULL`` and ``optlen=optlen``.
 
 
    .. versionchanged:: 3.5
diff --git a/Doc/library/winreg.rst b/Doc/library/winreg.rst
index cb67f2f39d760..5e810680b2643 100644
--- a/Doc/library/winreg.rst
+++ b/Doc/library/winreg.rst
@@ -343,7 +343,7 @@ This module offers the following functions:
    value set by the :func:`SetValue` method for the key identified by *key*.
 
    Values in the registry have name, type, and data components. This method
-   retrieves the data for a key's first value that has a NULL name. But the
+   retrieves the data for a key's first value that has a ``NULL`` name. But the
    underlying API call doesn't return the type, so always use
    :func:`QueryValueEx` if possible.
 
@@ -391,7 +391,7 @@ This module offers the following functions:
    <https://msdn.microsoft.com/en-us/library/ms724878%28v=VS.85%29.aspx>`__
    for more details.
 
-   This function passes NULL for *security_attributes* to the API.
+   This function passes ``NULL`` for *security_attributes* to the API.
 
 
 .. function:: SetValue(key, sub_key, type, value)
diff --git a/Doc/whatsnew/2.7.rst b/Doc/whatsnew/2.7.rst
index 9f8d9f202c434..d19c8e01ad8a0 100644
--- a/Doc/whatsnew/2.7.rst
+++ b/Doc/whatsnew/2.7.rst
@@ -1169,7 +1169,7 @@ changes, or look through the Subversion logs for all the details.
   correctly copy bound instance methods.  (Implemented by
   Robert Collins; :issue:`1515`.)
 
-* The :mod:`ctypes` module now always converts ``None`` to a C NULL
+* The :mod:`ctypes` module now always converts ``None`` to a C ``NULL``
   pointer for arguments declared as pointers.  (Changed by Thomas
   Heller; :issue:`4606`.)  The underlying `libffi library
   <https://sourceware.org/libffi/>`__ has been updated to version
diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst
index 7b886e7c27a48..99d040840d9fb 100644
--- a/Doc/whatsnew/3.4.rst
+++ b/Doc/whatsnew/3.4.rst
@@ -2509,7 +2509,7 @@ Changes in the C API
 
 * :c:func:`PyThread_set_key_value` now always set the value. In Python
   3.3, the function did nothing if the key already exists (if the current
-  value is a non-NULL pointer).
+  value is a non-``NULL`` pointer).
 
 * The ``f_tstate`` (thread state) field of the :c:type:`PyFrameObject`
   structure has been removed to fix a bug: see :issue:`14432` for the


From webhook-mailer at python.org  Wed Oct 30 16:45:01 2019
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Wed, 30 Oct 2019 20:45:01 -0000
Subject: [Python-checkins] [3.8] bpo-38600: NULL -> ``NULL``. (GH-17001)
 (GH-17003)
Message-ID: <mailman.194.1572468303.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/2c921c65e9d0aa7e3a38c96a22d5eec56724da72
commit: 2c921c65e9d0aa7e3a38c96a22d5eec56724da72
branch: 3.8
author: Serhiy Storchaka <storchaka at gmail.com>
committer: GitHub <noreply at github.com>
date: 2019-10-30T22:44:55+02:00
summary:

[3.8] bpo-38600: NULL -> ``NULL``. (GH-17001) (GH-17003)

Also fix some other formatting.
(cherry picked from commit e835b31d2b212c3c7820364398979cae2a9740b2)

files:
M Doc/c-api/arg.rst
M Doc/c-api/buffer.rst
M Doc/c-api/conversion.rst
M Doc/c-api/coro.rst
M Doc/c-api/exceptions.rst
M Doc/c-api/function.rst
M Doc/c-api/gen.rst
M Doc/c-api/import.rst
M Doc/c-api/init.rst
M Doc/c-api/init_config.rst
M Doc/c-api/marshal.rst
M Doc/c-api/memory.rst
M Doc/c-api/number.rst
M Doc/c-api/object.rst
M Doc/c-api/tuple.rst
M Doc/c-api/type.rst
M Doc/c-api/typeobj.rst
M Doc/c-api/unicode.rst
M Doc/c-api/veryhigh.rst
M Doc/extending/extending.rst
M Doc/howto/clinic.rst
M Doc/library/ctypes.rst
M Doc/library/gc.rst
M Doc/library/socket.rst
M Doc/library/winreg.rst
M Doc/whatsnew/2.7.rst
M Doc/whatsnew/3.4.rst

diff --git a/Doc/c-api/arg.rst b/Doc/c-api/arg.rst
index 594fef29f2a33..f17c63d0b9dc9 100644
--- a/Doc/c-api/arg.rst
+++ b/Doc/c-api/arg.rst
@@ -345,7 +345,7 @@ Other objects
    If the *converter* returns ``Py_CLEANUP_SUPPORTED``, it may get called a
    second time if the argument parsing eventually fails, giving the converter a
    chance to release any memory that it had already allocated. In this second
-   call, the *object* parameter will be NULL; *address* will have the same value
+   call, the *object* parameter will be ``NULL``; *address* will have the same value
    as in the original call.
 
    .. versionchanged:: 3.1
diff --git a/Doc/c-api/buffer.rst b/Doc/c-api/buffer.rst
index 58d874734e8ac..4e6fa0b32e669 100644
--- a/Doc/c-api/buffer.rst
+++ b/Doc/c-api/buffer.rst
@@ -130,7 +130,7 @@ a buffer, see :c:func:`PyObject_GetBuffer`.
    .. c:member:: Py_ssize_t itemsize
 
       Item size in bytes of a single element. Same as the value of :func:`struct.calcsize`
-      called on non-NULL :c:member:`~Py_buffer.format` values.
+      called on non-``NULL`` :c:member:`~Py_buffer.format` values.
 
       Important exception: If a consumer requests a buffer without the
       :c:macro:`PyBUF_FORMAT` flag, :c:member:`~Py_buffer.format` will
@@ -199,7 +199,7 @@ a buffer, see :c:func:`PyObject_GetBuffer`.
       memory block).
 
       If all suboffsets are negative (i.e. no de-referencing is needed), then
-      this field must be NULL (the default value).
+      this field must be ``NULL`` (the default value).
 
       This type of array representation is used by the Python Imaging Library
       (PIL). See `complex arrays`_ for further information how to access elements
@@ -407,7 +407,7 @@ to two ``char x[2][3]`` arrays that can be located anywhere in memory.
 
 
 Here is a function that returns a pointer to the element in an N-D array
-pointed to by an N-dimensional index when there are both non-NULL strides
+pointed to by an N-dimensional index when there are both non-``NULL`` strides
 and suboffsets::
 
    void *get_item_pointer(int ndim, void *buf, Py_ssize_t *strides,
@@ -520,4 +520,4 @@ Buffer-related functions
 
    If this function is used as part of a :ref:`getbufferproc <buffer-structs>`,
    *exporter* MUST be set to the exporting object and *flags* must be passed
-   unmodified. Otherwise, *exporter* MUST be NULL.
+   unmodified. Otherwise, *exporter* MUST be ``NULL``.
diff --git a/Doc/c-api/conversion.rst b/Doc/c-api/conversion.rst
index c02fa60ae9a7c..b310fcb5e4f91 100644
--- a/Doc/c-api/conversion.rst
+++ b/Doc/c-api/conversion.rst
@@ -32,7 +32,7 @@ NULL``.
 
 If the platform doesn't have :c:func:`vsnprintf` and the buffer size needed to
 avoid truncation exceeds *size* by more than 512 bytes, Python aborts with a
-*Py_FatalError*.
+:c:func:`Py_FatalError`.
 
 The return value (*rv*) for these functions should be interpreted as follows:
 
@@ -95,21 +95,21 @@ The following functions provide locale-independent string to number conversions.
    must be 0 and is ignored.  The ``'r'`` format code specifies the
    standard :func:`repr` format.
 
-   *flags* can be zero or more of the values *Py_DTSF_SIGN*,
-   *Py_DTSF_ADD_DOT_0*, or *Py_DTSF_ALT*, or-ed together:
+   *flags* can be zero or more of the values ``Py_DTSF_SIGN``,
+   ``Py_DTSF_ADD_DOT_0``, or ``Py_DTSF_ALT``, or-ed together:
 
-   * *Py_DTSF_SIGN* means to always precede the returned string with a sign
+   * ``Py_DTSF_SIGN`` means to always precede the returned string with a sign
      character, even if *val* is non-negative.
 
-   * *Py_DTSF_ADD_DOT_0* means to ensure that the returned string will not look
+   * ``Py_DTSF_ADD_DOT_0`` means to ensure that the returned string will not look
      like an integer.
 
-   * *Py_DTSF_ALT* means to apply "alternate" formatting rules.  See the
+   * ``Py_DTSF_ALT`` means to apply "alternate" formatting rules.  See the
      documentation for the :c:func:`PyOS_snprintf` ``'#'`` specifier for
      details.
 
-   If *ptype* is non-NULL, then the value it points to will be set to one of
-   *Py_DTST_FINITE*, *Py_DTST_INFINITE*, or *Py_DTST_NAN*, signifying that
+   If *ptype* is non-``NULL``, then the value it points to will be set to one of
+   ``Py_DTST_FINITE``, ``Py_DTST_INFINITE``, or ``Py_DTST_NAN``, signifying that
    *val* is a finite number, an infinite number, or not a number, respectively.
 
    The return value is a pointer to *buffer* with the converted string or
diff --git a/Doc/c-api/coro.rst b/Doc/c-api/coro.rst
index 915c57eeb55b8..2260944a9a93a 100644
--- a/Doc/c-api/coro.rst
+++ b/Doc/c-api/coro.rst
@@ -23,7 +23,7 @@ return.
 
 .. c:function:: int PyCoro_CheckExact(PyObject *ob)
 
-   Return true if *ob*'s type is *PyCoro_Type*; *ob* must not be ``NULL``.
+   Return true if *ob*'s type is :c:type:`PyCoro_Type`; *ob* must not be ``NULL``.
 
 
 .. c:function:: PyObject* PyCoro_New(PyFrameObject *frame, PyObject *name, PyObject *qualname)
diff --git a/Doc/c-api/exceptions.rst b/Doc/c-api/exceptions.rst
index 4cb309521d4d2..c7ba74cc8d587 100644
--- a/Doc/c-api/exceptions.rst
+++ b/Doc/c-api/exceptions.rst
@@ -19,9 +19,9 @@ return ``1`` for success and ``0`` for failure).
 
 Concretely, the error indicator consists of three object pointers: the
 exception's type, the exception's value, and the traceback object.  Any
-of those pointers can be NULL if non-set (although some combinations are
-forbidden, for example you can't have a non-NULL traceback if the exception
-type is NULL).
+of those pointers can be ``NULL`` if non-set (although some combinations are
+forbidden, for example you can't have a non-``NULL`` traceback if the exception
+type is ``NULL``).
 
 When a function must fail because some function it called failed, it generally
 doesn't set the error indicator; the function it called already set it.  It is
@@ -92,7 +92,7 @@ Raising exceptions
 
 These functions help you set the current thread's error indicator.
 For convenience, some of these functions will always return a
-NULL pointer for use in a ``return`` statement.
+``NULL`` pointer for use in a ``return`` statement.
 
 
 .. c:function:: void PyErr_SetString(PyObject *type, const char *message)
diff --git a/Doc/c-api/function.rst b/Doc/c-api/function.rst
index a3cbf569f76f2..bb416f4bb63aa 100644
--- a/Doc/c-api/function.rst
+++ b/Doc/c-api/function.rst
@@ -42,8 +42,8 @@ There are a few functions specific to Python functions.
 .. c:function:: PyObject* PyFunction_NewWithQualName(PyObject *code, PyObject *globals, PyObject *qualname)
 
    As :c:func:`PyFunction_New`, but also allows setting the function object's
-   ``__qualname__`` attribute.  *qualname* should be a unicode object or NULL;
-   if NULL, the ``__qualname__`` attribute is set to the same value as its
+   ``__qualname__`` attribute.  *qualname* should be a unicode object or ``NULL``;
+   if ``NULL``, the ``__qualname__`` attribute is set to the same value as its
    ``__name__`` attribute.
 
    .. versionadded:: 3.3
@@ -75,7 +75,7 @@ There are a few functions specific to Python functions.
 .. c:function:: int PyFunction_SetDefaults(PyObject *op, PyObject *defaults)
 
    Set the argument default values for the function object *op*. *defaults* must be
-   *Py_None* or a tuple.
+   ``Py_None`` or a tuple.
 
    Raises :exc:`SystemError` and returns ``-1`` on failure.
 
@@ -89,7 +89,7 @@ There are a few functions specific to Python functions.
 .. c:function:: int PyFunction_SetClosure(PyObject *op, PyObject *closure)
 
    Set the closure associated with the function object *op*. *closure* must be
-   *Py_None* or a tuple of cell objects.
+   ``Py_None`` or a tuple of cell objects.
 
    Raises :exc:`SystemError` and returns ``-1`` on failure.
 
@@ -103,6 +103,6 @@ There are a few functions specific to Python functions.
 .. c:function:: int PyFunction_SetAnnotations(PyObject *op, PyObject *annotations)
 
    Set the annotations for the function object *op*. *annotations*
-   must be a dictionary or *Py_None*.
+   must be a dictionary or ``Py_None``.
 
    Raises :exc:`SystemError` and returns ``-1`` on failure.
diff --git a/Doc/c-api/gen.rst b/Doc/c-api/gen.rst
index e4b8d255dadba..74410927bfde1 100644
--- a/Doc/c-api/gen.rst
+++ b/Doc/c-api/gen.rst
@@ -27,7 +27,7 @@ than explicitly calling :c:func:`PyGen_New` or :c:func:`PyGen_NewWithQualName`.
 
 .. c:function:: int PyGen_CheckExact(PyObject *ob)
 
-   Return true if *ob*'s type is *PyGen_Type*; *ob* must not be ``NULL``.
+   Return true if *ob*'s type is :c:type:`PyGen_Type`; *ob* must not be ``NULL``.
 
 
 .. c:function:: PyObject* PyGen_New(PyFrameObject *frame)
diff --git a/Doc/c-api/import.rst b/Doc/c-api/import.rst
index 862ad5151eb0f..78f935a2911bc 100644
--- a/Doc/c-api/import.rst
+++ b/Doc/c-api/import.rst
@@ -207,8 +207,8 @@ Importing Modules
 .. c:function:: PyObject* PyImport_GetModule(PyObject *name)
 
    Return the already imported module with the given name.  If the
-   module has not been imported yet then returns NULL but does not set
-   an error.  Returns NULL and sets an error if the lookup failed.
+   module has not been imported yet then returns ``NULL`` but does not set
+   an error.  Returns ``NULL`` and sets an error if the lookup failed.
 
    .. versionadded:: 3.7
 
diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst
index e04302044989d..c9a36ac4f5588 100644
--- a/Doc/c-api/init.rst
+++ b/Doc/c-api/init.rst
@@ -329,7 +329,7 @@ Process-wide parameters
    It overrides :envvar:`PYTHONIOENCODING` values, and allows embedding code
    to control IO encoding when the environment variable does not work.
 
-   ``encoding`` and/or ``errors`` may be NULL to use
+   *encoding* and/or *errors* may be ``NULL`` to use
    :envvar:`PYTHONIOENCODING` and/or default values (depending on other
    settings).
 
diff --git a/Doc/c-api/init_config.rst b/Doc/c-api/init_config.rst
index 5e18300471d46..6b16b5bb72859 100644
--- a/Doc/c-api/init_config.rst
+++ b/Doc/c-api/init_config.rst
@@ -61,8 +61,8 @@ PyWideStringList
 
    List of ``wchar_t*`` strings.
 
-   If *length* is non-zero, *items* must be non-NULL and all strings must be
-   non-NULL.
+   If *length* is non-zero, *items* must be non-``NULL`` and all strings must be
+   non-``NULL``.
 
    Methods:
 
@@ -608,7 +608,7 @@ PyConfig
 
       :data:`sys.pycache_prefix`: ``.pyc`` cache prefix.
 
-      If NULL, :data:`sys.pycache_prefix` is set to ``None``.
+      If ``NULL``, :data:`sys.pycache_prefix` is set to ``None``.
 
    .. c:member:: int quiet
 
diff --git a/Doc/c-api/marshal.rst b/Doc/c-api/marshal.rst
index 2f580430237af..7b179e22e290e 100644
--- a/Doc/c-api/marshal.rst
+++ b/Doc/c-api/marshal.rst
@@ -16,7 +16,7 @@ Numeric values are stored with the least significant byte first.
 The module supports two versions of the data format: version 0 is the
 historical version, version 1 shares interned strings in the file, and upon
 unmarshalling.  Version 2 uses a binary format for floating point numbers.
-*Py_MARSHAL_VERSION* indicates the current file format (currently 2).
+``Py_MARSHAL_VERSION`` indicates the current file format (currently 2).
 
 
 .. c:function:: void PyMarshal_WriteLongToFile(long value, FILE *file, int version)
diff --git a/Doc/c-api/memory.rst b/Doc/c-api/memory.rst
index c5b3af64c16a4..ba7bd3b9a5387 100644
--- a/Doc/c-api/memory.rst
+++ b/Doc/c-api/memory.rst
@@ -424,7 +424,7 @@ Customize Memory Allocators
 
    Set the memory block allocator of the specified domain.
 
-   The new allocator must return a distinct non-NULL pointer when requesting
+   The new allocator must return a distinct non-``NULL`` pointer when requesting
    zero bytes.
 
    For the :c:data:`PYMEM_DOMAIN_RAW` domain, the allocator must be
diff --git a/Doc/c-api/number.rst b/Doc/c-api/number.rst
index 55ddb9c0c8206..620204ca8e229 100644
--- a/Doc/c-api/number.rst
+++ b/Doc/c-api/number.rst
@@ -275,8 +275,8 @@ Number Protocol
    convert to a Py_ssize_t value would raise an :exc:`OverflowError`, then the
    *exc* argument is the type of exception that will be raised (usually
    :exc:`IndexError` or :exc:`OverflowError`).  If *exc* is ``NULL``, then the
-   exception is cleared and the value is clipped to *PY_SSIZE_T_MIN* for a negative
-   integer or *PY_SSIZE_T_MAX* for a positive integer.
+   exception is cleared and the value is clipped to ``PY_SSIZE_T_MIN`` for a negative
+   integer or ``PY_SSIZE_T_MAX`` for a positive integer.
 
 
 .. c:function:: int PyIndex_Check(PyObject *o)
diff --git a/Doc/c-api/object.rst b/Doc/c-api/object.rst
index 992dca6e1c8d2..404a98fb417bb 100644
--- a/Doc/c-api/object.rst
+++ b/Doc/c-api/object.rst
@@ -354,7 +354,7 @@ Object Protocol
    To get actual number of arguments, use
    :c:func:`PyVectorcall_NARGS(nargsf) <PyVectorcall_NARGS>`.
 
-   *kwnames* can be either NULL (no keyword arguments) or a tuple of keyword
+   *kwnames* can be either ``NULL`` (no keyword arguments) or a tuple of keyword
    names. In the latter case, the values of the keyword arguments are stored
    in *args* after the positional arguments.
    The number of keyword arguments does not influence *nargsf*.
diff --git a/Doc/c-api/tuple.rst b/Doc/c-api/tuple.rst
index d74d8626d1e92..25df3974e870e 100644
--- a/Doc/c-api/tuple.rst
+++ b/Doc/c-api/tuple.rst
@@ -153,7 +153,7 @@ type.
    | ``name``          | ``const char *``             | name of the struct sequence type     |
    +-------------------+------------------------------+--------------------------------------+
    | ``doc``           | ``const char *``             | pointer to docstring for the type    |
-   |                   |                              | or NULL to omit                      |
+   |                   |                              | or ``NULL`` to omit                  |
    +-------------------+------------------------------+--------------------------------------+
    | ``fields``        | ``PyStructSequence_Field *`` | pointer to ``NULL``-terminated array |
    |                   |                              | with field names of the new type     |
@@ -170,16 +170,16 @@ type.
    :attr:`fields` array of the :c:type:`PyStructSequence_Desc` determines which
    field of the struct sequence is described.
 
-   +-----------+------------------+----------------------------------------+
-   | Field     | C Type           | Meaning                                |
-   +===========+==================+========================================+
-   | ``name``  | ``const char *`` | name for the field or ``NULL`` to end  |
-   |           |                  | the list of named fields, set to       |
-   |           |                  | PyStructSequence_UnnamedField to       |
-   |           |                  | leave unnamed                          |
-   +-----------+------------------+----------------------------------------+
-   | ``doc``   | ``const char *`` | field docstring or ``NULL`` to omit    |
-   +-----------+------------------+----------------------------------------+
+   +-----------+------------------+-----------------------------------------+
+   | Field     | C Type           | Meaning                                 |
+   +===========+==================+=========================================+
+   | ``name``  | ``const char *`` | name for the field or ``NULL`` to end   |
+   |           |                  | the list of named fields, set to        |
+   |           |                  | :c:data:`PyStructSequence_UnnamedField` |
+   |           |                  | to leave unnamed                        |
+   +-----------+------------------+-----------------------------------------+
+   | ``doc``   | ``const char *`` | field docstring or ``NULL`` to omit     |
+   +-----------+------------------+-----------------------------------------+
 
 
 .. c:var:: char* PyStructSequence_UnnamedField
diff --git a/Doc/c-api/type.rst b/Doc/c-api/type.rst
index 0bb806d21421b..dcdec5329bdbc 100644
--- a/Doc/c-api/type.rst
+++ b/Doc/c-api/type.rst
@@ -181,7 +181,7 @@ The following functions and structs are used to create
       * ``Py_nb_add`` to set :c:member:`PyNumberMethods.nb_add`
       * ``Py_sq_length`` to set :c:member:`PySequenceMethods.sq_length`
 
-      The following fields cannot be set using *PyType_Spec* and *PyType_Slot*:
+      The following fields cannot be set using :c:type:`PyType_Spec` and :c:type:`PyType_Slot`:
 
       * :c:member:`~PyTypeObject.tp_dict`
       * :c:member:`~PyTypeObject.tp_mro`
diff --git a/Doc/c-api/typeobj.rst b/Doc/c-api/typeobj.rst
index 181157e08ec43..bff5abfea48ee 100644
--- a/Doc/c-api/typeobj.rst
+++ b/Doc/c-api/typeobj.rst
@@ -740,7 +740,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
    This field is inherited by subtypes together with
    :c:member:`~PyTypeObject.tp_call`: a subtype inherits
    :c:member:`~PyTypeObject.tp_vectorcall_offset` from its base type when
-   the subtype?s :c:member:`~PyTypeObject.tp_call` is NULL.
+   the subtype?s :c:member:`~PyTypeObject.tp_call` is ``NULL``.
 
    Note that `heap types`_ (including subclasses defined in Python) do not
    inherit the :const:`_Py_TPFLAGS_HAVE_VECTORCALL` flag.
@@ -1180,7 +1180,7 @@ and :c:type:`PyType_Type` effectively act as defaults.)
 
       This bit is set on *static* subtypes if ``tp_flags`` is not overridden:
       a subtype inherits ``_Py_TPFLAGS_HAVE_VECTORCALL`` from its base type
-      when the subtype?s :c:member:`~PyTypeObject.tp_call` is NULL
+      when the subtype?s :c:member:`~PyTypeObject.tp_call` is ``NULL``
       and the subtype's ``Py_TPFLAGS_HEAPTYPE`` is not set.
 
       `Heap types`_ do not inherit ``_Py_TPFLAGS_HAVE_VECTORCALL``.
@@ -1955,7 +1955,7 @@ This results in types that are limited relative to types defined in Python:
   :ref:`sub-interpreters <sub-interpreter-support>`, so they should not
   include any subinterpreter-specific state.
 
-Also, since *PyTypeObject* is not part of the :ref:`stable ABI <stable>`,
+Also, since :c:type:`PyTypeObject` is not part of the :ref:`stable ABI <stable>`,
 any extension modules using static types must be compiled for a specific
 Python minor version.
 
diff --git a/Doc/c-api/unicode.rst b/Doc/c-api/unicode.rst
index 0359f5ec36bed..2bf4a0f56bc8a 100644
--- a/Doc/c-api/unicode.rst
+++ b/Doc/c-api/unicode.rst
@@ -526,9 +526,9 @@ APIs:
    .. note::
       The width formatter unit is number of characters rather than bytes.
       The precision formatter unit is number of bytes for ``"%s"`` and
-      ``"%V"`` (if the ``PyObject*`` argument is NULL), and a number of
+      ``"%V"`` (if the ``PyObject*`` argument is ``NULL``), and a number of
       characters for ``"%A"``, ``"%U"``, ``"%S"``, ``"%R"`` and ``"%V"``
-      (if the ``PyObject*`` argument is not NULL).
+      (if the ``PyObject*`` argument is not ``NULL``).
 
    .. [1] For integer specifiers (d, u, ld, li, lu, lld, lli, llu, zd, zi,
       zu, i, x): the 0-conversion flag has effect even when a precision is given.
@@ -1172,7 +1172,7 @@ These are the UTF-32 codec APIs:
    If byteorder is ``0``, the output string will always start with the Unicode BOM
    mark (U+FEFF). In the other two modes, no BOM mark is prepended.
 
-   If *Py_UNICODE_WIDE* is not defined, surrogate pairs will be output
+   If ``Py_UNICODE_WIDE`` is not defined, surrogate pairs will be output
    as a single code point.
 
    Return ``NULL`` if an exception was raised by the codec.
@@ -1246,7 +1246,7 @@ These are the UTF-16 codec APIs:
    If byteorder is ``0``, the output string will always start with the Unicode BOM
    mark (U+FEFF). In the other two modes, no BOM mark is prepended.
 
-   If *Py_UNICODE_WIDE* is defined, a single :c:type:`Py_UNICODE` value may get
+   If ``Py_UNICODE_WIDE`` is defined, a single :c:type:`Py_UNICODE` value may get
    represented as a surrogate pair. If it is not defined, each :c:type:`Py_UNICODE`
    values is interpreted as a UCS-2 character.
 
diff --git a/Doc/c-api/veryhigh.rst b/Doc/c-api/veryhigh.rst
index 4ca0e0bb8bfc4..98cea52ab7193 100644
--- a/Doc/c-api/veryhigh.rst
+++ b/Doc/c-api/veryhigh.rst
@@ -80,7 +80,7 @@ the same library that the Python runtime is using.
 .. c:function:: int PyRun_SimpleString(const char *command)
 
    This is a simplified interface to :c:func:`PyRun_SimpleStringFlags` below,
-   leaving the *PyCompilerFlags\** argument set to NULL.
+   leaving the :c:type:`PyCompilerFlags`\* argument set to ``NULL``.
 
 
 .. c:function:: int PyRun_SimpleStringFlags(const char *command, PyCompilerFlags *flags)
diff --git a/Doc/extending/extending.rst b/Doc/extending/extending.rst
index 7986ea1e15045..5b32a2cdc5506 100644
--- a/Doc/extending/extending.rst
+++ b/Doc/extending/extending.rst
@@ -536,7 +536,7 @@ Later, when it is time to call the function, you call the C function
 :c:func:`PyObject_CallObject`.  This function has two arguments, both pointers to
 arbitrary Python objects: the Python function, and the argument list.  The
 argument list must always be a tuple object, whose length is the number of
-arguments.  To call the Python function with no arguments, pass in NULL, or
+arguments.  To call the Python function with no arguments, pass in ``NULL``, or
 an empty tuple; to call it with one argument, pass a singleton tuple.
 :c:func:`Py_BuildValue` returns a tuple when its format string consists of zero
 or more format codes between parentheses.  For example::
diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst
index cfd9f2e4075c4..50041829b8c38 100644
--- a/Doc/howto/clinic.rst
+++ b/Doc/howto/clinic.rst
@@ -1070,7 +1070,7 @@ Currently Argument Clinic supports only a few return converters:
     DecodeFSDefault
 
 None of these take parameters.  For the first three, return -1 to indicate
-error.  For ``DecodeFSDefault``, the return type is ``const char *``; return a NULL
+error.  For ``DecodeFSDefault``, the return type is ``const char *``; return a ``NULL``
 pointer to indicate an error.
 
 (There's also an experimental ``NoneType`` converter, which lets you
diff --git a/Doc/library/ctypes.rst b/Doc/library/ctypes.rst
index 4504a800c43ca..1f84b7ab29839 100644
--- a/Doc/library/ctypes.rst
+++ b/Doc/library/ctypes.rst
@@ -152,8 +152,8 @@ the ``time()`` function, which returns system time in seconds since the Unix
 epoch, and the ``GetModuleHandleA()`` function, which returns a win32 module
 handle.
 
-This example calls both functions with a NULL pointer (``None`` should be used
-as the NULL pointer)::
+This example calls both functions with a ``NULL`` pointer (``None`` should be used
+as the ``NULL`` pointer)::
 
    >>> print(libc.time(None))  # doctest: +SKIP
    1150640792
@@ -1110,7 +1110,7 @@ Since ``table`` is a ``pointer`` to the array of ``struct_frozen`` records, we
 can iterate over it, but we just have to make sure that our loop terminates,
 because pointers have no size. Sooner or later it would probably crash with an
 access violation or whatever, so it's better to break out of the loop when we
-hit the NULL entry::
+hit the ``NULL`` entry::
 
    >>> for item in table:
    ...     if item.name is None:
diff --git a/Doc/library/gc.rst b/Doc/library/gc.rst
index 084cd6ac257eb..366861c524c3d 100644
--- a/Doc/library/gc.rst
+++ b/Doc/library/gc.rst
@@ -212,7 +212,7 @@ values but should not rebind them):
    A list of objects which the collector found to be unreachable but could
    not be freed (uncollectable objects).  Starting with Python 3.4, this
    list should be empty most of the time, except when using instances of
-   C extension types with a non-NULL ``tp_del`` slot.
+   C extension types with a non-``NULL`` ``tp_del`` slot.
 
    If :const:`DEBUG_SAVEALL` is set, then all unreachable objects will be
    added to this list rather than freed.
diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst
index 12416e0f4a9fd..12b90914bae96 100644
--- a/Doc/library/socket.rst
+++ b/Doc/library/socket.rst
@@ -1635,9 +1635,9 @@ to sockets.
    ``None`` or a :term:`bytes-like object` representing a buffer. In the later
    case it is up to the caller to ensure that the bytestring contains the
    proper bits (see the optional built-in module :mod:`struct` for a way to
-   encode C structures as bytestrings). When value is set to ``None``,
-   optlen argument is required. It's equivalent to call setsockopt C
-   function with optval=NULL and optlen=optlen.
+   encode C structures as bytestrings). When *value* is set to ``None``,
+   *optlen* argument is required. It's equivalent to call :c:func:`setsockopt` C
+   function with ``optval=NULL`` and ``optlen=optlen``.
 
 
    .. versionchanged:: 3.5
diff --git a/Doc/library/winreg.rst b/Doc/library/winreg.rst
index cb67f2f39d760..5e810680b2643 100644
--- a/Doc/library/winreg.rst
+++ b/Doc/library/winreg.rst
@@ -343,7 +343,7 @@ This module offers the following functions:
    value set by the :func:`SetValue` method for the key identified by *key*.
 
    Values in the registry have name, type, and data components. This method
-   retrieves the data for a key's first value that has a NULL name. But the
+   retrieves the data for a key's first value that has a ``NULL`` name. But the
    underlying API call doesn't return the type, so always use
    :func:`QueryValueEx` if possible.
 
@@ -391,7 +391,7 @@ This module offers the following functions:
    <https://msdn.microsoft.com/en-us/library/ms724878%28v=VS.85%29.aspx>`__
    for more details.
 
-   This function passes NULL for *security_attributes* to the API.
+   This function passes ``NULL`` for *security_attributes* to the API.
 
 
 .. function:: SetValue(key, sub_key, type, value)
diff --git a/Doc/whatsnew/2.7.rst b/Doc/whatsnew/2.7.rst
index 9f8d9f202c434..d19c8e01ad8a0 100644
--- a/Doc/whatsnew/2.7.rst
+++ b/Doc/whatsnew/2.7.rst
@@ -1169,7 +1169,7 @@ changes, or look through the Subversion logs for all the details.
   correctly copy bound instance methods.  (Implemented by
   Robert Collins; :issue:`1515`.)
 
-* The :mod:`ctypes` module now always converts ``None`` to a C NULL
+* The :mod:`ctypes` module now always converts ``None`` to a C ``NULL``
   pointer for arguments declared as pointers.  (Changed by Thomas
   Heller; :issue:`4606`.)  The underlying `libffi library
   <https://sourceware.org/libffi/>`__ has been updated to version
diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst
index 7b886e7c27a48..99d040840d9fb 100644
--- a/Doc/whatsnew/3.4.rst
+++ b/Doc/whatsnew/3.4.rst
@@ -2509,7 +2509,7 @@ Changes in the C API
 
 * :c:func:`PyThread_set_key_value` now always set the value. In Python
   3.3, the function did nothing if the key already exists (if the current
-  value is a non-NULL pointer).
+  value is a non-``NULL`` pointer).
 
 * The ``f_tstate`` (thread state) field of the :c:type:`PyFrameObject`
   structure has been removed to fix a bug: see :issue:`14432` for the


From webhook-mailer at python.org  Wed Oct 30 16:45:18 2019
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Wed, 30 Oct 2019 20:45:18 -0000
Subject: [Python-checkins] [3.7] bpo-38600: NULL -> ``NULL``. (GH-17001)
 (GH-17004)
Message-ID: <mailman.195.1572468320.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/951b6c314a8567be3b5d901059fde866c79f5ff0
commit: 951b6c314a8567be3b5d901059fde866c79f5ff0
branch: 3.7
author: Serhiy Storchaka <storchaka at gmail.com>
committer: GitHub <noreply at github.com>
date: 2019-10-30T22:45:14+02:00
summary:

[3.7] bpo-38600: NULL -> ``NULL``. (GH-17001) (GH-17004)

Also fix some other formatting.
(cherry picked from commit e835b31d2b212c3c7820364398979cae2a9740b2)

files:
M Doc/c-api/arg.rst
M Doc/c-api/buffer.rst
M Doc/c-api/conversion.rst
M Doc/c-api/coro.rst
M Doc/c-api/exceptions.rst
M Doc/c-api/function.rst
M Doc/c-api/gen.rst
M Doc/c-api/import.rst
M Doc/c-api/init.rst
M Doc/c-api/marshal.rst
M Doc/c-api/memory.rst
M Doc/c-api/number.rst
M Doc/c-api/tuple.rst
M Doc/c-api/unicode.rst
M Doc/c-api/veryhigh.rst
M Doc/extending/extending.rst
M Doc/howto/clinic.rst
M Doc/library/ctypes.rst
M Doc/library/gc.rst
M Doc/library/socket.rst
M Doc/library/winreg.rst
M Doc/whatsnew/2.7.rst
M Doc/whatsnew/3.4.rst

diff --git a/Doc/c-api/arg.rst b/Doc/c-api/arg.rst
index c3c8a03a79e08..a5d1edc5f5b15 100644
--- a/Doc/c-api/arg.rst
+++ b/Doc/c-api/arg.rst
@@ -345,7 +345,7 @@ Other objects
    If the *converter* returns ``Py_CLEANUP_SUPPORTED``, it may get called a
    second time if the argument parsing eventually fails, giving the converter a
    chance to release any memory that it had already allocated. In this second
-   call, the *object* parameter will be NULL; *address* will have the same value
+   call, the *object* parameter will be ``NULL``; *address* will have the same value
    as in the original call.
 
    .. versionchanged:: 3.1
diff --git a/Doc/c-api/buffer.rst b/Doc/c-api/buffer.rst
index 6b59ec5d4c9fd..74f518bab6cde 100644
--- a/Doc/c-api/buffer.rst
+++ b/Doc/c-api/buffer.rst
@@ -130,7 +130,7 @@ a buffer, see :c:func:`PyObject_GetBuffer`.
    .. c:member:: Py_ssize_t itemsize
 
       Item size in bytes of a single element. Same as the value of :func:`struct.calcsize`
-      called on non-NULL :c:member:`~Py_buffer.format` values.
+      called on non-``NULL`` :c:member:`~Py_buffer.format` values.
 
       Important exception: If a consumer requests a buffer without the
       :c:macro:`PyBUF_FORMAT` flag, :c:member:`~Py_buffer.format` will
@@ -199,7 +199,7 @@ a buffer, see :c:func:`PyObject_GetBuffer`.
       memory block).
 
       If all suboffsets are negative (i.e. no de-referencing is needed), then
-      this field must be NULL (the default value).
+      this field must be ``NULL`` (the default value).
 
       This type of array representation is used by the Python Imaging Library
       (PIL). See `complex arrays`_ for further information how to access elements
@@ -407,7 +407,7 @@ to two ``char x[2][3]`` arrays that can be located anywhere in memory.
 
 
 Here is a function that returns a pointer to the element in an N-D array
-pointed to by an N-dimensional index when there are both non-NULL strides
+pointed to by an N-dimensional index when there are both non-``NULL`` strides
 and suboffsets::
 
    void *get_item_pointer(int ndim, void *buf, Py_ssize_t *strides,
@@ -520,4 +520,4 @@ Buffer-related functions
 
    If this function is used as part of a :ref:`getbufferproc <buffer-structs>`,
    *exporter* MUST be set to the exporting object and *flags* must be passed
-   unmodified. Otherwise, *exporter* MUST be NULL.
+   unmodified. Otherwise, *exporter* MUST be ``NULL``.
diff --git a/Doc/c-api/conversion.rst b/Doc/c-api/conversion.rst
index 3a4182a608948..8ea85895a4c41 100644
--- a/Doc/c-api/conversion.rst
+++ b/Doc/c-api/conversion.rst
@@ -32,7 +32,7 @@ NULL``.
 
 If the platform doesn't have :c:func:`vsnprintf` and the buffer size needed to
 avoid truncation exceeds *size* by more than 512 bytes, Python aborts with a
-*Py_FatalError*.
+:c:func:`Py_FatalError`.
 
 The return value (*rv*) for these functions should be interpreted as follows:
 
@@ -95,21 +95,21 @@ The following functions provide locale-independent string to number conversions.
    must be 0 and is ignored.  The ``'r'`` format code specifies the
    standard :func:`repr` format.
 
-   *flags* can be zero or more of the values *Py_DTSF_SIGN*,
-   *Py_DTSF_ADD_DOT_0*, or *Py_DTSF_ALT*, or-ed together:
+   *flags* can be zero or more of the values ``Py_DTSF_SIGN``,
+   ``Py_DTSF_ADD_DOT_0``, or ``Py_DTSF_ALT``, or-ed together:
 
-   * *Py_DTSF_SIGN* means to always precede the returned string with a sign
+   * ``Py_DTSF_SIGN`` means to always precede the returned string with a sign
      character, even if *val* is non-negative.
 
-   * *Py_DTSF_ADD_DOT_0* means to ensure that the returned string will not look
+   * ``Py_DTSF_ADD_DOT_0`` means to ensure that the returned string will not look
      like an integer.
 
-   * *Py_DTSF_ALT* means to apply "alternate" formatting rules.  See the
+   * ``Py_DTSF_ALT`` means to apply "alternate" formatting rules.  See the
      documentation for the :c:func:`PyOS_snprintf` ``'#'`` specifier for
      details.
 
-   If *ptype* is non-NULL, then the value it points to will be set to one of
-   *Py_DTST_FINITE*, *Py_DTST_INFINITE*, or *Py_DTST_NAN*, signifying that
+   If *ptype* is non-``NULL``, then the value it points to will be set to one of
+   ``Py_DTST_FINITE``, ``Py_DTST_INFINITE``, or ``Py_DTST_NAN``, signifying that
    *val* is a finite number, an infinite number, or not a number, respectively.
 
    The return value is a pointer to *buffer* with the converted string or
diff --git a/Doc/c-api/coro.rst b/Doc/c-api/coro.rst
index de81b4ea5dc21..af52d4f17374d 100644
--- a/Doc/c-api/coro.rst
+++ b/Doc/c-api/coro.rst
@@ -23,7 +23,7 @@ return.
 
 .. c:function:: int PyCoro_CheckExact(PyObject *ob)
 
-   Return true if *ob*'s type is *PyCoro_Type*; *ob* must not be ``NULL``.
+   Return true if *ob*'s type is :c:type:`PyCoro_Type`; *ob* must not be ``NULL``.
 
 
 .. c:function:: PyObject* PyCoro_New(PyFrameObject *frame, PyObject *name, PyObject *qualname)
diff --git a/Doc/c-api/exceptions.rst b/Doc/c-api/exceptions.rst
index a763fd7751307..a53c49a211036 100644
--- a/Doc/c-api/exceptions.rst
+++ b/Doc/c-api/exceptions.rst
@@ -19,9 +19,9 @@ return ``1`` for success and ``0`` for failure).
 
 Concretely, the error indicator consists of three object pointers: the
 exception's type, the exception's value, and the traceback object.  Any
-of those pointers can be NULL if non-set (although some combinations are
-forbidden, for example you can't have a non-NULL traceback if the exception
-type is NULL).
+of those pointers can be ``NULL`` if non-set (although some combinations are
+forbidden, for example you can't have a non-``NULL`` traceback if the exception
+type is ``NULL``).
 
 When a function must fail because some function it called failed, it generally
 doesn't set the error indicator; the function it called already set it.  It is
@@ -89,7 +89,7 @@ Raising exceptions
 
 These functions help you set the current thread's error indicator.
 For convenience, some of these functions will always return a
-NULL pointer for use in a ``return`` statement.
+``NULL`` pointer for use in a ``return`` statement.
 
 
 .. c:function:: void PyErr_SetString(PyObject *type, const char *message)
diff --git a/Doc/c-api/function.rst b/Doc/c-api/function.rst
index a5045ab56f46c..438381a7d2fe0 100644
--- a/Doc/c-api/function.rst
+++ b/Doc/c-api/function.rst
@@ -42,8 +42,8 @@ There are a few functions specific to Python functions.
 .. c:function:: PyObject* PyFunction_NewWithQualName(PyObject *code, PyObject *globals, PyObject *qualname)
 
    As :c:func:`PyFunction_New`, but also allows setting the function object's
-   ``__qualname__`` attribute.  *qualname* should be a unicode object or NULL;
-   if NULL, the ``__qualname__`` attribute is set to the same value as its
+   ``__qualname__`` attribute.  *qualname* should be a unicode object or ``NULL``;
+   if ``NULL``, the ``__qualname__`` attribute is set to the same value as its
    ``__name__`` attribute.
 
    .. versionadded:: 3.3
@@ -75,7 +75,7 @@ There are a few functions specific to Python functions.
 .. c:function:: int PyFunction_SetDefaults(PyObject *op, PyObject *defaults)
 
    Set the argument default values for the function object *op*. *defaults* must be
-   *Py_None* or a tuple.
+   ``Py_None`` or a tuple.
 
    Raises :exc:`SystemError` and returns ``-1`` on failure.
 
@@ -89,7 +89,7 @@ There are a few functions specific to Python functions.
 .. c:function:: int PyFunction_SetClosure(PyObject *op, PyObject *closure)
 
    Set the closure associated with the function object *op*. *closure* must be
-   *Py_None* or a tuple of cell objects.
+   ``Py_None`` or a tuple of cell objects.
 
    Raises :exc:`SystemError` and returns ``-1`` on failure.
 
@@ -103,6 +103,6 @@ There are a few functions specific to Python functions.
 .. c:function:: int PyFunction_SetAnnotations(PyObject *op, PyObject *annotations)
 
    Set the annotations for the function object *op*. *annotations*
-   must be a dictionary or *Py_None*.
+   must be a dictionary or ``Py_None``.
 
    Raises :exc:`SystemError` and returns ``-1`` on failure.
diff --git a/Doc/c-api/gen.rst b/Doc/c-api/gen.rst
index e007ced158545..2226f58bc4292 100644
--- a/Doc/c-api/gen.rst
+++ b/Doc/c-api/gen.rst
@@ -27,7 +27,7 @@ than explicitly calling :c:func:`PyGen_New` or :c:func:`PyGen_NewWithQualName`.
 
 .. c:function:: int PyGen_CheckExact(PyObject *ob)
 
-   Return true if *ob*'s type is *PyGen_Type*; *ob* must not be ``NULL``.
+   Return true if *ob*'s type is :c:type:`PyGen_Type`; *ob* must not be ``NULL``.
 
 
 .. c:function:: PyObject* PyGen_New(PyFrameObject *frame)
diff --git a/Doc/c-api/import.rst b/Doc/c-api/import.rst
index daa704ce2a474..764e2be444301 100644
--- a/Doc/c-api/import.rst
+++ b/Doc/c-api/import.rst
@@ -207,8 +207,8 @@ Importing Modules
 .. c:function:: PyObject* PyImport_GetModule(PyObject *name)
 
    Return the already imported module with the given name.  If the
-   module has not been imported yet then returns NULL but does not set
-   an error.  Returns NULL and sets an error if the lookup failed.
+   module has not been imported yet then returns ``NULL`` but does not set
+   an error.  Returns ``NULL`` and sets an error if the lookup failed.
 
    .. versionadded:: 3.7
 
diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst
index 8c7485168572f..2822b88cc3359 100644
--- a/Doc/c-api/init.rst
+++ b/Doc/c-api/init.rst
@@ -326,7 +326,7 @@ Process-wide parameters
    It overrides :envvar:`PYTHONIOENCODING` values, and allows embedding code
    to control IO encoding when the environment variable does not work.
 
-   ``encoding`` and/or ``errors`` may be NULL to use
+   *encoding* and/or *errors* may be ``NULL`` to use
    :envvar:`PYTHONIOENCODING` and/or default values (depending on other
    settings).
 
diff --git a/Doc/c-api/marshal.rst b/Doc/c-api/marshal.rst
index 0556a2e74ab53..bf5fb4ecbb7dc 100644
--- a/Doc/c-api/marshal.rst
+++ b/Doc/c-api/marshal.rst
@@ -16,7 +16,7 @@ Numeric values are stored with the least significant byte first.
 The module supports two versions of the data format: version 0 is the
 historical version, version 1 shares interned strings in the file, and upon
 unmarshalling.  Version 2 uses a binary format for floating point numbers.
-*Py_MARSHAL_VERSION* indicates the current file format (currently 2).
+``Py_MARSHAL_VERSION`` indicates the current file format (currently 2).
 
 
 .. c:function:: void PyMarshal_WriteLongToFile(long value, FILE *file, int version)
diff --git a/Doc/c-api/memory.rst b/Doc/c-api/memory.rst
index 1ce7e3cf16cde..f14e1e123a0df 100644
--- a/Doc/c-api/memory.rst
+++ b/Doc/c-api/memory.rst
@@ -424,7 +424,7 @@ Customize Memory Allocators
 
    Set the memory block allocator of the specified domain.
 
-   The new allocator must return a distinct non-NULL pointer when requesting
+   The new allocator must return a distinct non-``NULL`` pointer when requesting
    zero bytes.
 
    For the :c:data:`PYMEM_DOMAIN_RAW` domain, the allocator must be
diff --git a/Doc/c-api/number.rst b/Doc/c-api/number.rst
index 32c7a98052eae..82c43670ecc2e 100644
--- a/Doc/c-api/number.rst
+++ b/Doc/c-api/number.rst
@@ -272,8 +272,8 @@ Number Protocol
    convert to a Py_ssize_t value would raise an :exc:`OverflowError`, then the
    *exc* argument is the type of exception that will be raised (usually
    :exc:`IndexError` or :exc:`OverflowError`).  If *exc* is ``NULL``, then the
-   exception is cleared and the value is clipped to *PY_SSIZE_T_MIN* for a negative
-   integer or *PY_SSIZE_T_MAX* for a positive integer.
+   exception is cleared and the value is clipped to ``PY_SSIZE_T_MIN`` for a negative
+   integer or ``PY_SSIZE_T_MAX`` for a positive integer.
 
 
 .. c:function:: int PyIndex_Check(PyObject *o)
diff --git a/Doc/c-api/tuple.rst b/Doc/c-api/tuple.rst
index 4dc79d0e7d053..b7d4b7bb49646 100644
--- a/Doc/c-api/tuple.rst
+++ b/Doc/c-api/tuple.rst
@@ -153,7 +153,7 @@ type.
    | ``name``          | ``const char *``             | name of the struct sequence type     |
    +-------------------+------------------------------+--------------------------------------+
    | ``doc``           | ``const char *``             | pointer to docstring for the type    |
-   |                   |                              | or NULL to omit                      |
+   |                   |                              | or ``NULL`` to omit                  |
    +-------------------+------------------------------+--------------------------------------+
    | ``fields``        | ``PyStructSequence_Field *`` | pointer to ``NULL``-terminated array |
    |                   |                              | with field names of the new type     |
@@ -170,16 +170,16 @@ type.
    :attr:`fields` array of the :c:type:`PyStructSequence_Desc` determines which
    field of the struct sequence is described.
 
-   +-----------+------------------+----------------------------------------+
-   | Field     | C Type           | Meaning                                |
-   +===========+==================+========================================+
-   | ``name``  | ``const char *`` | name for the field or ``NULL`` to end  |
-   |           |                  | the list of named fields, set to       |
-   |           |                  | PyStructSequence_UnnamedField to       |
-   |           |                  | leave unnamed                          |
-   +-----------+------------------+----------------------------------------+
-   | ``doc``   | ``const char *`` | field docstring or ``NULL`` to omit    |
-   +-----------+------------------+----------------------------------------+
+   +-----------+------------------+-----------------------------------------+
+   | Field     | C Type           | Meaning                                 |
+   +===========+==================+=========================================+
+   | ``name``  | ``const char *`` | name for the field or ``NULL`` to end   |
+   |           |                  | the list of named fields, set to        |
+   |           |                  | :c:data:`PyStructSequence_UnnamedField` |
+   |           |                  | to leave unnamed                        |
+   +-----------+------------------+-----------------------------------------+
+   | ``doc``   | ``const char *`` | field docstring or ``NULL`` to omit     |
+   +-----------+------------------+-----------------------------------------+
 
 
 .. c:var:: char* PyStructSequence_UnnamedField
diff --git a/Doc/c-api/unicode.rst b/Doc/c-api/unicode.rst
index f6c50fb468d4d..35f67533badb1 100644
--- a/Doc/c-api/unicode.rst
+++ b/Doc/c-api/unicode.rst
@@ -526,9 +526,9 @@ APIs:
    .. note::
       The width formatter unit is number of characters rather than bytes.
       The precision formatter unit is number of bytes for ``"%s"`` and
-      ``"%V"`` (if the ``PyObject*`` argument is NULL), and a number of
+      ``"%V"`` (if the ``PyObject*`` argument is ``NULL``), and a number of
       characters for ``"%A"``, ``"%U"``, ``"%S"``, ``"%R"`` and ``"%V"``
-      (if the ``PyObject*`` argument is not NULL).
+      (if the ``PyObject*`` argument is not ``NULL``).
 
    .. [1] For integer specifiers (d, u, ld, li, lu, lld, lli, llu, zd, zi,
       zu, i, x): the 0-conversion flag has effect even when a precision is given.
@@ -1172,7 +1172,7 @@ These are the UTF-32 codec APIs:
    If byteorder is ``0``, the output string will always start with the Unicode BOM
    mark (U+FEFF). In the other two modes, no BOM mark is prepended.
 
-   If *Py_UNICODE_WIDE* is not defined, surrogate pairs will be output
+   If ``Py_UNICODE_WIDE`` is not defined, surrogate pairs will be output
    as a single code point.
 
    Return ``NULL`` if an exception was raised by the codec.
@@ -1246,7 +1246,7 @@ These are the UTF-16 codec APIs:
    If byteorder is ``0``, the output string will always start with the Unicode BOM
    mark (U+FEFF). In the other two modes, no BOM mark is prepended.
 
-   If *Py_UNICODE_WIDE* is defined, a single :c:type:`Py_UNICODE` value may get
+   If ``Py_UNICODE_WIDE`` is defined, a single :c:type:`Py_UNICODE` value may get
    represented as a surrogate pair. If it is not defined, each :c:type:`Py_UNICODE`
    values is interpreted as a UCS-2 character.
 
diff --git a/Doc/c-api/veryhigh.rst b/Doc/c-api/veryhigh.rst
index 3d92fe9acb58c..49a7a0c591a2c 100644
--- a/Doc/c-api/veryhigh.rst
+++ b/Doc/c-api/veryhigh.rst
@@ -73,7 +73,7 @@ the same library that the Python runtime is using.
 .. c:function:: int PyRun_SimpleString(const char *command)
 
    This is a simplified interface to :c:func:`PyRun_SimpleStringFlags` below,
-   leaving the *PyCompilerFlags\** argument set to NULL.
+   leaving the :c:type:`PyCompilerFlags`\* argument set to ``NULL``.
 
 
 .. c:function:: int PyRun_SimpleStringFlags(const char *command, PyCompilerFlags *flags)
diff --git a/Doc/extending/extending.rst b/Doc/extending/extending.rst
index da463e4f23ede..fb15ecf39b7cd 100644
--- a/Doc/extending/extending.rst
+++ b/Doc/extending/extending.rst
@@ -536,7 +536,7 @@ Later, when it is time to call the function, you call the C function
 :c:func:`PyObject_CallObject`.  This function has two arguments, both pointers to
 arbitrary Python objects: the Python function, and the argument list.  The
 argument list must always be a tuple object, whose length is the number of
-arguments.  To call the Python function with no arguments, pass in NULL, or
+arguments.  To call the Python function with no arguments, pass in ``NULL``, or
 an empty tuple; to call it with one argument, pass a singleton tuple.
 :c:func:`Py_BuildValue` returns a tuple when its format string consists of zero
 or more format codes between parentheses.  For example::
diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst
index 695fbb1be192b..ab5f0e4d039e2 100644
--- a/Doc/howto/clinic.rst
+++ b/Doc/howto/clinic.rst
@@ -1064,7 +1064,7 @@ Currently Argument Clinic supports only a few return converters:
     DecodeFSDefault
 
 None of these take parameters.  For the first three, return -1 to indicate
-error.  For ``DecodeFSDefault``, the return type is ``const char *``; return a NULL
+error.  For ``DecodeFSDefault``, the return type is ``const char *``; return a ``NULL``
 pointer to indicate an error.
 
 (There's also an experimental ``NoneType`` converter, which lets you
diff --git a/Doc/library/ctypes.rst b/Doc/library/ctypes.rst
index 2be004b79168b..b4f989d95ad5b 100644
--- a/Doc/library/ctypes.rst
+++ b/Doc/library/ctypes.rst
@@ -152,8 +152,8 @@ the ``time()`` function, which returns system time in seconds since the Unix
 epoch, and the ``GetModuleHandleA()`` function, which returns a win32 module
 handle.
 
-This example calls both functions with a NULL pointer (``None`` should be used
-as the NULL pointer)::
+This example calls both functions with a ``NULL`` pointer (``None`` should be used
+as the ``NULL`` pointer)::
 
    >>> print(libc.time(None))  # doctest: +SKIP
    1150640792
@@ -1110,7 +1110,7 @@ Since ``table`` is a ``pointer`` to the array of ``struct_frozen`` records, we
 can iterate over it, but we just have to make sure that our loop terminates,
 because pointers have no size. Sooner or later it would probably crash with an
 access violation or whatever, so it's better to break out of the loop when we
-hit the NULL entry::
+hit the ``NULL`` entry::
 
    >>> for item in table:
    ...     if item.name is None:
diff --git a/Doc/library/gc.rst b/Doc/library/gc.rst
index 153d8fb704562..d428c33e87679 100644
--- a/Doc/library/gc.rst
+++ b/Doc/library/gc.rst
@@ -209,7 +209,7 @@ values but should not rebind them):
    A list of objects which the collector found to be unreachable but could
    not be freed (uncollectable objects).  Starting with Python 3.4, this
    list should be empty most of the time, except when using instances of
-   C extension types with a non-NULL ``tp_del`` slot.
+   C extension types with a non-``NULL`` ``tp_del`` slot.
 
    If :const:`DEBUG_SAVEALL` is set, then all unreachable objects will be
    added to this list rather than freed.
diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst
index 15c65b97d993d..b7262774b67bb 100644
--- a/Doc/library/socket.rst
+++ b/Doc/library/socket.rst
@@ -1535,9 +1535,9 @@ to sockets.
    ``None`` or a :term:`bytes-like object` representing a buffer. In the later
    case it is up to the caller to ensure that the bytestring contains the
    proper bits (see the optional built-in module :mod:`struct` for a way to
-   encode C structures as bytestrings). When value is set to ``None``,
-   optlen argument is required. It's equivalent to call setsockopt C
-   function with optval=NULL and optlen=optlen.
+   encode C structures as bytestrings). When *value* is set to ``None``,
+   *optlen* argument is required. It's equivalent to call :c:func:`setsockopt` C
+   function with ``optval=NULL`` and ``optlen=optlen``.
 
 
    .. versionchanged:: 3.5
diff --git a/Doc/library/winreg.rst b/Doc/library/winreg.rst
index cb67f2f39d760..5e810680b2643 100644
--- a/Doc/library/winreg.rst
+++ b/Doc/library/winreg.rst
@@ -343,7 +343,7 @@ This module offers the following functions:
    value set by the :func:`SetValue` method for the key identified by *key*.
 
    Values in the registry have name, type, and data components. This method
-   retrieves the data for a key's first value that has a NULL name. But the
+   retrieves the data for a key's first value that has a ``NULL`` name. But the
    underlying API call doesn't return the type, so always use
    :func:`QueryValueEx` if possible.
 
@@ -391,7 +391,7 @@ This module offers the following functions:
    <https://msdn.microsoft.com/en-us/library/ms724878%28v=VS.85%29.aspx>`__
    for more details.
 
-   This function passes NULL for *security_attributes* to the API.
+   This function passes ``NULL`` for *security_attributes* to the API.
 
 
 .. function:: SetValue(key, sub_key, type, value)
diff --git a/Doc/whatsnew/2.7.rst b/Doc/whatsnew/2.7.rst
index 9f8d9f202c434..d19c8e01ad8a0 100644
--- a/Doc/whatsnew/2.7.rst
+++ b/Doc/whatsnew/2.7.rst
@@ -1169,7 +1169,7 @@ changes, or look through the Subversion logs for all the details.
   correctly copy bound instance methods.  (Implemented by
   Robert Collins; :issue:`1515`.)
 
-* The :mod:`ctypes` module now always converts ``None`` to a C NULL
+* The :mod:`ctypes` module now always converts ``None`` to a C ``NULL``
   pointer for arguments declared as pointers.  (Changed by Thomas
   Heller; :issue:`4606`.)  The underlying `libffi library
   <https://sourceware.org/libffi/>`__ has been updated to version
diff --git a/Doc/whatsnew/3.4.rst b/Doc/whatsnew/3.4.rst
index 7b886e7c27a48..99d040840d9fb 100644
--- a/Doc/whatsnew/3.4.rst
+++ b/Doc/whatsnew/3.4.rst
@@ -2509,7 +2509,7 @@ Changes in the C API
 
 * :c:func:`PyThread_set_key_value` now always set the value. In Python
   3.3, the function did nothing if the key already exists (if the current
-  value is a non-NULL pointer).
+  value is a non-``NULL`` pointer).
 
 * The ``f_tstate`` (thread state) field of the :c:type:`PyFrameObject`
   structure has been removed to fix a bug: see :issue:`14432` for the


From webhook-mailer at python.org  Wed Oct 30 21:08:21 2019
From: webhook-mailer at python.org (Benjamin Peterson)
Date: Thu, 31 Oct 2019 01:08:21 -0000
Subject: [Python-checkins] closes bpo-38648: Remove double tp_free slot in
 Python-ast.c. (GH-17002)
Message-ID: <mailman.196.1572484102.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/bdac32e9fe25fdb97a7172a93aabd1ffead89462
commit: bdac32e9fe25fdb97a7172a93aabd1ffead89462
branch: master
author: Max Bernstein <tekknolagi at users.noreply.github.com>
committer: Benjamin Peterson <benjamin at python.org>
date: 2019-10-30T18:08:06-07:00
summary:

closes bpo-38648: Remove double tp_free slot in Python-ast.c. (GH-17002)

This looks like a typo due to copy-paste.

files:
M Parser/asdl_c.py
M Python/Python-ast.c

diff --git a/Parser/asdl_c.py b/Parser/asdl_c.py
index 3a821cc9265c9..52495e9b451c1 100755
--- a/Parser/asdl_c.py
+++ b/Parser/asdl_c.py
@@ -751,7 +751,6 @@ def visitModule(self, mod):
     {Py_tp_init, ast_type_init},
     {Py_tp_alloc, PyType_GenericAlloc},
     {Py_tp_new, PyType_GenericNew},
-    {Py_tp_free, PyType_GenericNew},
     {Py_tp_free, PyObject_GC_Del},
     {0, 0},
 };
diff --git a/Python/Python-ast.c b/Python/Python-ast.c
index e2c703d815552..d5465d795cfa4 100644
--- a/Python/Python-ast.c
+++ b/Python/Python-ast.c
@@ -1244,7 +1244,6 @@ static PyType_Slot AST_type_slots[] = {
     {Py_tp_init, ast_type_init},
     {Py_tp_alloc, PyType_GenericAlloc},
     {Py_tp_new, PyType_GenericNew},
-    {Py_tp_free, PyType_GenericNew},
     {Py_tp_free, PyObject_GC_Del},
     {0, 0},
 };


From webhook-mailer at python.org  Thu Oct 31 04:04:03 2019
From: webhook-mailer at python.org (Vinay Sajip)
Date: Thu, 31 Oct 2019 08:04:03 -0000
Subject: [Python-checkins] bpo-16575: Add checks for unions passed by value
 to functions. (GH-16799)
Message-ID: <mailman.197.1572509043.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/79d4ed102a5069c6cebaed2627cb1645637f0429
commit: 79d4ed102a5069c6cebaed2627cb1645637f0429
branch: master
author: Vinay Sajip <vinay_sajip at yahoo.co.uk>
committer: GitHub <noreply at github.com>
date: 2019-10-31T08:03:54Z
summary:

bpo-16575: Add checks for unions passed by value to functions. (GH-16799)

files:
M Lib/ctypes/test/test_structures.py
M Modules/_ctypes/_ctypes.c
M Modules/_ctypes/_ctypes_test.c
M Modules/_ctypes/ctypes.h
M Modules/_ctypes/stgdict.c

diff --git a/Lib/ctypes/test/test_structures.py b/Lib/ctypes/test/test_structures.py
index cdf4a9182439f..ab23206df8a76 100644
--- a/Lib/ctypes/test/test_structures.py
+++ b/Lib/ctypes/test/test_structures.py
@@ -576,6 +576,86 @@ class U(Union):
             self.assertEqual(f2, [0x4567, 0x0123, 0xcdef, 0x89ab,
                                   0x3210, 0x7654, 0xba98, 0xfedc])
 
+    def test_union_by_value(self):
+        # See bpo-16575
+
+        # These should mirror the structures in Modules/_ctypes/_ctypes_test.c
+
+        class Nested1(Structure):
+            _fields_ = [
+                ('an_int', c_int),
+                ('another_int', c_int),
+            ]
+
+        class Test4(Union):
+            _fields_ = [
+                ('a_long', c_long),
+                ('a_struct', Nested1),
+            ]
+
+        class Nested2(Structure):
+            _fields_ = [
+                ('an_int', c_int),
+                ('a_union', Test4),
+            ]
+
+        class Test5(Structure):
+            _fields_ = [
+                ('an_int', c_int),
+                ('nested', Nested2),
+                ('another_int', c_int),
+            ]
+
+        test4 = Test4()
+        dll = CDLL(_ctypes_test.__file__)
+        with self.assertRaises(TypeError) as ctx:
+            func = dll._testfunc_union_by_value1
+            func.restype = c_long
+            func.argtypes = (Test4,)
+            result = func(test4)
+        self.assertEqual(ctx.exception.args[0], 'item 1 in _argtypes_ passes '
+                         'a union by value, which is unsupported.')
+        test5 = Test5()
+        with self.assertRaises(TypeError) as ctx:
+            func = dll._testfunc_union_by_value2
+            func.restype = c_long
+            func.argtypes = (Test5,)
+            result = func(test5)
+        self.assertEqual(ctx.exception.args[0], 'item 1 in _argtypes_ passes '
+                         'a union by value, which is unsupported.')
+
+        # passing by reference should be OK
+        test4.a_long = 12345;
+        func = dll._testfunc_union_by_reference1
+        func.restype = c_long
+        func.argtypes = (POINTER(Test4),)
+        result = func(byref(test4))
+        self.assertEqual(result, 12345)
+        self.assertEqual(test4.a_long, 0)
+        self.assertEqual(test4.a_struct.an_int, 0)
+        self.assertEqual(test4.a_struct.another_int, 0)
+        test4.a_struct.an_int = 0x12340000
+        test4.a_struct.another_int = 0x5678
+        func = dll._testfunc_union_by_reference2
+        func.restype = c_long
+        func.argtypes = (POINTER(Test4),)
+        result = func(byref(test4))
+        self.assertEqual(result, 0x12345678)
+        self.assertEqual(test4.a_long, 0)
+        self.assertEqual(test4.a_struct.an_int, 0)
+        self.assertEqual(test4.a_struct.another_int, 0)
+        test5.an_int = 0x12000000
+        test5.nested.an_int = 0x345600
+        test5.another_int = 0x78
+        func = dll._testfunc_union_by_reference3
+        func.restype = c_long
+        func.argtypes = (POINTER(Test5),)
+        result = func(byref(test5))
+        self.assertEqual(result, 0x12345678)
+        self.assertEqual(test5.an_int, 0)
+        self.assertEqual(test5.nested.an_int, 0)
+        self.assertEqual(test5.another_int, 0)
+
 class PointerMemberTestCase(unittest.TestCase):
 
     def test(self):
diff --git a/Modules/_ctypes/_ctypes.c b/Modules/_ctypes/_ctypes.c
index 16a0cfe8dd4dc..5e22b73155c5f 100644
--- a/Modules/_ctypes/_ctypes.c
+++ b/Modules/_ctypes/_ctypes.c
@@ -504,6 +504,9 @@ StructUnionType_new(PyTypeObject *type, PyObject *args, PyObject *kwds, int isSt
         Py_DECREF(result);
         return NULL;
     }
+    if (!isStruct) {
+        dict->flags |= TYPEFLAG_HASUNION;
+    }
     /* replace the class dict by our updated stgdict, which holds info
        about storage requirements of the instances */
     if (-1 == PyDict_Update((PyObject *)dict, result->tp_dict)) {
@@ -2383,6 +2386,27 @@ converters_from_argtypes(PyObject *ob)
     for (i = 0; i < nArgs; ++i) {
         PyObject *cnv;
         PyObject *tp = PyTuple_GET_ITEM(ob, i);
+        StgDictObject *stgdict = PyType_stgdict(tp);
+
+        if (stgdict != NULL) {
+            if (stgdict->flags & TYPEFLAG_HASUNION) {
+                Py_DECREF(converters);
+                Py_DECREF(ob);
+                if (!PyErr_Occurred()) {
+                    PyErr_Format(PyExc_TypeError,
+                                 "item %zd in _argtypes_ passes a union by "
+                                 "value, which is unsupported.",
+                                 i + 1);
+                }
+                return NULL;
+            }
+            /*
+            if (stgdict->flags & TYPEFLAG_HASBITFIELD) {
+                printf("found stgdict with bitfield\n");
+            }
+            */
+        }
+
         if (_PyObject_LookupAttrId(tp, &PyId_from_param, &cnv) <= 0) {
             Py_DECREF(converters);
             Py_DECREF(ob);
diff --git a/Modules/_ctypes/_ctypes_test.c b/Modules/_ctypes/_ctypes_test.c
index 40da652620271..49ed82004f604 100644
--- a/Modules/_ctypes/_ctypes_test.c
+++ b/Modules/_ctypes/_ctypes_test.c
@@ -131,6 +131,69 @@ _testfunc_array_in_struct2a(Test3B in)
     return result;
 }
 
+typedef union {
+    long a_long;
+    struct {
+        int an_int;
+        int another_int;
+    } a_struct;
+} Test4;
+
+typedef struct {
+    int an_int;
+    struct {
+        int an_int;
+        Test4 a_union;
+    } nested;
+    int another_int;
+} Test5;
+
+EXPORT(long)
+_testfunc_union_by_value1(Test4 in) {
+    long result = in.a_long + in.a_struct.an_int + in.a_struct.another_int;
+
+    /* As the union/struct are passed by value, changes to them shouldn't be
+     * reflected in the caller.
+     */
+    memset(&in, 0, sizeof(in));
+    return result;
+}
+
+EXPORT(long)
+_testfunc_union_by_value2(Test5 in) {
+    long result = in.an_int + in.nested.an_int;
+
+    /* As the union/struct are passed by value, changes to them shouldn't be
+     * reflected in the caller.
+     */
+    memset(&in, 0, sizeof(in));
+    return result;
+}
+
+EXPORT(long)
+_testfunc_union_by_reference1(Test4 *in) {
+    long result = in->a_long;
+
+    memset(in, 0, sizeof(Test4));
+    return result;
+}
+
+EXPORT(long)
+_testfunc_union_by_reference2(Test4 *in) {
+    long result = in->a_struct.an_int + in->a_struct.another_int;
+
+    memset(in, 0, sizeof(Test4));
+    return result;
+}
+
+EXPORT(long)
+_testfunc_union_by_reference3(Test5 *in) {
+    long result = in->an_int + in->nested.an_int + in->another_int;
+
+    memset(in, 0, sizeof(Test5));
+    return result;
+}
+
 EXPORT(void)testfunc_array(int values[4])
 {
     printf("testfunc_array %d %d %d %d\n",
diff --git a/Modules/_ctypes/ctypes.h b/Modules/_ctypes/ctypes.h
index 5d3b9663385f5..e58f85233cb71 100644
--- a/Modules/_ctypes/ctypes.h
+++ b/Modules/_ctypes/ctypes.h
@@ -288,6 +288,8 @@ PyObject *_ctypes_callproc(PPROC pProc,
 
 #define TYPEFLAG_ISPOINTER 0x100
 #define TYPEFLAG_HASPOINTER 0x200
+#define TYPEFLAG_HASUNION 0x400
+#define TYPEFLAG_HASBITFIELD 0x800
 
 #define DICTFLAG_FINAL 0x1000
 
diff --git a/Modules/_ctypes/stgdict.c b/Modules/_ctypes/stgdict.c
index 97bcf5539aeea..1d45ade5efd90 100644
--- a/Modules/_ctypes/stgdict.c
+++ b/Modules/_ctypes/stgdict.c
@@ -440,6 +440,13 @@ PyCStructUnionType_update_stgdict(PyObject *type, PyObject *fields, int isStruct
         PyMem_Free(stgdict->ffi_type_pointer.elements);
 
     basedict = PyType_stgdict((PyObject *)((PyTypeObject *)type)->tp_base);
+    if (basedict) {
+        stgdict->flags |= (basedict->flags &
+                           (TYPEFLAG_HASUNION | TYPEFLAG_HASBITFIELD));
+    }
+    if (!isStruct) {
+        stgdict->flags |= TYPEFLAG_HASUNION;
+    }
     if (basedict && !use_broken_old_ctypes_semantics) {
         size = offset = basedict->size;
         align = basedict->align;
@@ -515,8 +522,10 @@ PyCStructUnionType_update_stgdict(PyObject *type, PyObject *fields, int isStruct
         stgdict->ffi_type_pointer.elements[ffi_ofs + i] = &dict->ffi_type_pointer;
         if (dict->flags & (TYPEFLAG_ISPOINTER | TYPEFLAG_HASPOINTER))
             stgdict->flags |= TYPEFLAG_HASPOINTER;
+        stgdict->flags |= dict->flags & (TYPEFLAG_HASUNION | TYPEFLAG_HASBITFIELD);
         dict->flags |= DICTFLAG_FINAL; /* mark field type final */
         if (PyTuple_Size(pair) == 3) { /* bits specified */
+            stgdict->flags |= TYPEFLAG_HASBITFIELD;
             switch(dict->ffi_type_pointer.type) {
             case FFI_TYPE_UINT8:
             case FFI_TYPE_UINT16:


From webhook-mailer at python.org  Thu Oct 31 04:06:29 2019
From: webhook-mailer at python.org (Vinay Sajip)
Date: Thu, 31 Oct 2019 08:06:29 -0000
Subject: [Python-checkins] bpo-38586: setting logging.Handler .name property
 in fileConfig (GH-16918)
Message-ID: <mailman.198.1572509190.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/b15100fe7def8580c78ed16f0bb4b72b2ae7af3f
commit: b15100fe7def8580c78ed16f0bb4b72b2ae7af3f
branch: master
author: Lucas Cimon <lucas.cimon at gmail.com>
committer: Vinay Sajip <vinay_sajip at yahoo.co.uk>
date: 2019-10-31T08:06:25Z
summary:

bpo-38586: setting logging.Handler .name property in fileConfig (GH-16918)

files:
A Misc/NEWS.d/next/Library/2019-10-24-17-26-39.bpo-38586.cyq5nr.rst
M Lib/logging/config.py
M Lib/test/test_logging.py

diff --git a/Lib/logging/config.py b/Lib/logging/config.py
index 9dd35e11aab39..4a3b8966ede1c 100644
--- a/Lib/logging/config.py
+++ b/Lib/logging/config.py
@@ -143,6 +143,7 @@ def _install_handlers(cp, formatters):
         kwargs = section.get("kwargs", '{}')
         kwargs = eval(kwargs, vars(logging))
         h = klass(*args, **kwargs)
+        h.name = hand
         if "level" in section:
             level = section["level"]
             h.setLevel(level)
diff --git a/Lib/test/test_logging.py b/Lib/test/test_logging.py
index 74ccf48b7ebe2..53b5bfc93f33e 100644
--- a/Lib/test/test_logging.py
+++ b/Lib/test/test_logging.py
@@ -1591,6 +1591,30 @@ def test_logger_disabling(self):
         self.apply_config(self.disable_test, disable_existing_loggers=False)
         self.assertFalse(logger.disabled)
 
+    def test_config_set_handler_names(self):
+        test_config = """
+            [loggers]
+            keys=root
+
+            [handlers]
+            keys=hand1
+
+            [formatters]
+            keys=form1
+
+            [logger_root]
+            handlers=hand1
+
+            [handler_hand1]
+            class=StreamHandler
+            formatter=form1
+
+            [formatter_form1]
+            format=%(levelname)s ++ %(message)s
+            """
+        self.apply_config(test_config)
+        self.assertEquals(logging.getLogger().handlers[0].name, 'hand1')
+
     def test_defaults_do_no_interpolation(self):
         """bpo-33802 defaults should not get interpolated"""
         ini = textwrap.dedent("""
diff --git a/Misc/NEWS.d/next/Library/2019-10-24-17-26-39.bpo-38586.cyq5nr.rst b/Misc/NEWS.d/next/Library/2019-10-24-17-26-39.bpo-38586.cyq5nr.rst
new file mode 100644
index 0000000000000..eac7bba082566
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-10-24-17-26-39.bpo-38586.cyq5nr.rst
@@ -0,0 +1 @@
+Now :func:`~logging.config.fileConfig` correcty sets the .name of handlers loaded.


From webhook-mailer at python.org  Thu Oct 31 05:13:55 2019
From: webhook-mailer at python.org (Serhiy Storchaka)
Date: Thu, 31 Oct 2019 09:13:55 -0000
Subject: [Python-checkins] bpo-38312: Add curses.{get,
 set}_escdelay and curses.{get, set}_tabsize. (GH-16938)
Message-ID: <mailman.199.1572513237.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/b32cb97bce472dad337c6b2f071883f6234e21d8
commit: b32cb97bce472dad337c6b2f071883f6234e21d8
branch: master
author: Anthony Sottile <asottile at umich.edu>
committer: Serhiy Storchaka <storchaka at gmail.com>
date: 2019-10-31T11:13:48+02:00
summary:

bpo-38312: Add curses.{get,set}_escdelay and curses.{get,set}_tabsize. (GH-16938)

files:
A Misc/NEWS.d/next/Library/2019-10-26-14-42-20.bpo-38312.e_FVWh.rst
M Doc/library/curses.rst
M Doc/whatsnew/3.9.rst
M Lib/test/test_curses.py
M Modules/_cursesmodule.c
M Modules/clinic/_cursesmodule.c.h

diff --git a/Doc/library/curses.rst b/Doc/library/curses.rst
index f1803c685a9e2..1201e8972de63 100644
--- a/Doc/library/curses.rst
+++ b/Doc/library/curses.rst
@@ -511,6 +511,32 @@ The module :mod:`curses` defines the following functions:
    Save the current state of the terminal modes in a buffer, usable by
    :func:`resetty`.
 
+.. function:: get_escdelay()
+
+   Retrieves the value set by :func:`set_escdelay`.
+
+   .. versionadded:: 3.9
+
+.. function:: set_escdelay(ms)
+
+   Sets the number of milliseconds to wait after reading an escape character,
+   to distinguish between an individual escape character entered on the
+   keyboard from escape sequences sent by cursor and function keys.
+
+   .. versionadded:: 3.9
+
+.. function:: get_tabsize()
+
+   Retrieves the value set by :func:`set_tabsize`.
+
+   .. versionadded:: 3.9
+
+.. function:: set_tabsize(size)
+
+   Sets the number of columns used by the curses library when converting a tab
+   character to spaces as it adds the tab to a window.
+
+   .. versionadded:: 3.9
 
 .. function:: setsyx(y, x)
 
diff --git a/Doc/whatsnew/3.9.rst b/Doc/whatsnew/3.9.rst
index d41e70808d6e1..7d7c502459a9b 100644
--- a/Doc/whatsnew/3.9.rst
+++ b/Doc/whatsnew/3.9.rst
@@ -130,6 +130,13 @@ that schedules a shutdown for the default executor that waits on the
 :func:`asyncio.run` has been updated to use the new :term:`coroutine`.
 (Contributed by Kyle Stanley in :issue:`34037`.)
 
+curses
+------
+
+Add :func:`curses.get_escdelay`, :func:`curses.set_escdelay`,
+:func:`curses.get_tabsize`, and :func:`curses.set_tabsize` functions.
+(Contributed by Anthony Sottile in :issue:`38312`.)
+
 fcntl
 -----
 
diff --git a/Lib/test/test_curses.py b/Lib/test/test_curses.py
index 09738c8a41c94..5e619d13836d2 100644
--- a/Lib/test/test_curses.py
+++ b/Lib/test/test_curses.py
@@ -261,6 +261,10 @@ def test_module_funcs(self):
         curses.putp(b'abc')
         curses.qiflush()
         curses.raw() ; curses.raw(1)
+        curses.set_escdelay(25)
+        self.assertEqual(curses.get_escdelay(), 25)
+        curses.set_tabsize(4)
+        self.assertEqual(curses.get_tabsize(), 4)
         if hasattr(curses, 'setsyx'):
             curses.setsyx(5,5)
         curses.tigetflag('hc')
diff --git a/Misc/NEWS.d/next/Library/2019-10-26-14-42-20.bpo-38312.e_FVWh.rst b/Misc/NEWS.d/next/Library/2019-10-26-14-42-20.bpo-38312.e_FVWh.rst
new file mode 100644
index 0000000000000..2a1800fd2b00d
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-10-26-14-42-20.bpo-38312.e_FVWh.rst
@@ -0,0 +1,3 @@
+Add :func:`curses.get_escdelay`, :func:`curses.set_escdelay`,
+:func:`curses.get_tabsize`, and :func:`curses.set_tabsize` functions -
+by Anthony Sottile.
diff --git a/Modules/_cursesmodule.c b/Modules/_cursesmodule.c
index 98fd1f33b62f0..29840b52ffd81 100644
--- a/Modules/_cursesmodule.c
+++ b/Modules/_cursesmodule.c
@@ -3255,6 +3255,86 @@ _curses_setupterm_impl(PyObject *module, const char *term, int fd)
     Py_RETURN_NONE;
 }
 
+/*[clinic input]
+_curses.get_escdelay
+
+Gets the curses ESCDELAY setting.
+
+Gets the number of milliseconds to wait after reading an escape character,
+to distinguish between an individual escape character entered on the
+keyboard from escape sequences sent by cursor and function keys.
+[clinic start generated code]*/
+
+static PyObject *
+_curses_get_escdelay_impl(PyObject *module)
+/*[clinic end generated code: output=222fa1a822555d60 input=be2d5b3dd974d0a4]*/
+{
+    return PyLong_FromLong(ESCDELAY);
+}
+/*[clinic input]
+_curses.set_escdelay
+    ms: int
+        length of the delay in milliseconds.
+    /
+
+Sets the curses ESCDELAY setting.
+
+Sets the number of milliseconds to wait after reading an escape character,
+to distinguish between an individual escape character entered on the
+keyboard from escape sequences sent by cursor and function keys.
+[clinic start generated code]*/
+
+static PyObject *
+_curses_set_escdelay_impl(PyObject *module, int ms)
+/*[clinic end generated code: output=43818efbf7980ac4 input=7796fe19f111e250]*/
+{
+    if (ms <= 0) {
+        PyErr_SetString(PyExc_ValueError, "ms must be > 0");
+        return NULL;
+    }
+
+    return PyCursesCheckERR(set_escdelay(ms), "set_escdelay");
+}
+
+/*[clinic input]
+_curses.get_tabsize
+
+Gets the curses TABSIZE setting.
+
+Gets the number of columns used by the curses library when converting a tab
+character to spaces as it adds the tab to a window.
+[clinic start generated code]*/
+
+static PyObject *
+_curses_get_tabsize_impl(PyObject *module)
+/*[clinic end generated code: output=7e9e51fb6126fbdf input=74af86bf6c9f5d7e]*/
+{
+    return PyLong_FromLong(TABSIZE);
+}
+/*[clinic input]
+_curses.set_tabsize
+    size: int
+        rendered cell width of a tab character.
+    /
+
+Sets the curses TABSIZE setting.
+
+Sets the number of columns used by the curses library when converting a tab
+character to spaces as it adds the tab to a window.
+[clinic start generated code]*/
+
+static PyObject *
+_curses_set_tabsize_impl(PyObject *module, int size)
+/*[clinic end generated code: output=c1de5a76c0daab1e input=78cba6a3021ad061]*/
+{
+    if (size <= 0) {
+        PyErr_SetString(PyExc_ValueError, "size must be > 0");
+        return NULL;
+    }
+
+    return PyCursesCheckERR(set_tabsize(size), "set_tabsize");
+}
+
 /*[clinic input]
 _curses.intrflush
 
@@ -4415,6 +4495,10 @@ static PyMethodDef PyCurses_methods[] = {
     _CURSES_RESIZETERM_METHODDEF
     _CURSES_RESIZE_TERM_METHODDEF
     _CURSES_SAVETTY_METHODDEF
+    _CURSES_GET_ESCDELAY_METHODDEF
+    _CURSES_SET_ESCDELAY_METHODDEF
+    _CURSES_GET_TABSIZE_METHODDEF
+    _CURSES_SET_TABSIZE_METHODDEF
     _CURSES_SETSYX_METHODDEF
     _CURSES_SETUPTERM_METHODDEF
     _CURSES_START_COLOR_METHODDEF
diff --git a/Modules/clinic/_cursesmodule.c.h b/Modules/clinic/_cursesmodule.c.h
index ad93e6a0ca022..7b30a49182f9e 100644
--- a/Modules/clinic/_cursesmodule.c.h
+++ b/Modules/clinic/_cursesmodule.c.h
@@ -3040,6 +3040,128 @@ _curses_setupterm(PyObject *module, PyObject *const *args, Py_ssize_t nargs, PyO
     return return_value;
 }
 
+PyDoc_STRVAR(_curses_get_escdelay__doc__,
+"get_escdelay($module, /)\n"
+"--\n"
+"\n"
+"Gets the curses ESCDELAY setting.\n"
+"\n"
+"Gets the number of milliseconds to wait after reading an escape character,\n"
+"to distinguish between an individual escape character entered on the\n"
+"keyboard from escape sequences sent by cursor and function keys.");
+
+#define _CURSES_GET_ESCDELAY_METHODDEF    \
+    {"get_escdelay", (PyCFunction)_curses_get_escdelay, METH_NOARGS, _curses_get_escdelay__doc__},
+
+static PyObject *
+_curses_get_escdelay_impl(PyObject *module);
+
+static PyObject *
+_curses_get_escdelay(PyObject *module, PyObject *Py_UNUSED(ignored))
+{
+    return _curses_get_escdelay_impl(module);
+}
+
+PyDoc_STRVAR(_curses_set_escdelay__doc__,
+"set_escdelay($module, ms, /)\n"
+"--\n"
+"\n"
+"Sets the curses ESCDELAY setting.\n"
+"\n"
+"  ms\n"
+"    length of the delay in milliseconds.\n"
+"\n"
+"Sets the number of milliseconds to wait after reading an escape character,\n"
+"to distinguish between an individual escape character entered on the\n"
+"keyboard from escape sequences sent by cursor and function keys.");
+
+#define _CURSES_SET_ESCDELAY_METHODDEF    \
+    {"set_escdelay", (PyCFunction)_curses_set_escdelay, METH_O, _curses_set_escdelay__doc__},
+
+static PyObject *
+_curses_set_escdelay_impl(PyObject *module, int ms);
+
+static PyObject *
+_curses_set_escdelay(PyObject *module, PyObject *arg)
+{
+    PyObject *return_value = NULL;
+    int ms;
+
+    if (PyFloat_Check(arg)) {
+        PyErr_SetString(PyExc_TypeError,
+                        "integer argument expected, got float" );
+        goto exit;
+    }
+    ms = _PyLong_AsInt(arg);
+    if (ms == -1 && PyErr_Occurred()) {
+        goto exit;
+    }
+    return_value = _curses_set_escdelay_impl(module, ms);
+
+exit:
+    return return_value;
+}
+
+PyDoc_STRVAR(_curses_get_tabsize__doc__,
+"get_tabsize($module, /)\n"
+"--\n"
+"\n"
+"Gets the curses TABSIZE setting.\n"
+"\n"
+"Gets the number of columns used by the curses library when converting a tab\n"
+"character to spaces as it adds the tab to a window.");
+
+#define _CURSES_GET_TABSIZE_METHODDEF    \
+    {"get_tabsize", (PyCFunction)_curses_get_tabsize, METH_NOARGS, _curses_get_tabsize__doc__},
+
+static PyObject *
+_curses_get_tabsize_impl(PyObject *module);
+
+static PyObject *
+_curses_get_tabsize(PyObject *module, PyObject *Py_UNUSED(ignored))
+{
+    return _curses_get_tabsize_impl(module);
+}
+
+PyDoc_STRVAR(_curses_set_tabsize__doc__,
+"set_tabsize($module, size, /)\n"
+"--\n"
+"\n"
+"Sets the curses TABSIZE setting.\n"
+"\n"
+"  size\n"
+"    rendered cell width of a tab character.\n"
+"\n"
+"Sets the number of columns used by the curses library when converting a tab\n"
+"character to spaces as it adds the tab to a window.");
+
+#define _CURSES_SET_TABSIZE_METHODDEF    \
+    {"set_tabsize", (PyCFunction)_curses_set_tabsize, METH_O, _curses_set_tabsize__doc__},
+
+static PyObject *
+_curses_set_tabsize_impl(PyObject *module, int size);
+
+static PyObject *
+_curses_set_tabsize(PyObject *module, PyObject *arg)
+{
+    PyObject *return_value = NULL;
+    int size;
+
+    if (PyFloat_Check(arg)) {
+        PyErr_SetString(PyExc_TypeError,
+                        "integer argument expected, got float" );
+        goto exit;
+    }
+    size = _PyLong_AsInt(arg);
+    if (size == -1 && PyErr_Occurred()) {
+        goto exit;
+    }
+    return_value = _curses_set_tabsize_impl(module, size);
+
+exit:
+    return return_value;
+}
+
 PyDoc_STRVAR(_curses_intrflush__doc__,
 "intrflush($module, flag, /)\n"
 "--\n"
@@ -4569,4 +4691,4 @@ _curses_use_default_colors(PyObject *module, PyObject *Py_UNUSED(ignored))
 #ifndef _CURSES_USE_DEFAULT_COLORS_METHODDEF
     #define _CURSES_USE_DEFAULT_COLORS_METHODDEF
 #endif /* !defined(_CURSES_USE_DEFAULT_COLORS_METHODDEF) */
-/*[clinic end generated code: output=e5b3502f1d38dff0 input=a9049054013a1b77]*/
+/*[clinic end generated code: output=985c0849e841acec input=a9049054013a1b77]*/


From webhook-mailer at python.org  Thu Oct 31 06:23:24 2019
From: webhook-mailer at python.org (Vinay Sajip)
Date: Thu, 31 Oct 2019 10:23:24 -0000
Subject: [Python-checkins] Add docstring for shlex.split (GH-16740)
Message-ID: <mailman.200.1572517405.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/65c7382c47af07aea5c1ce86b76cf7f4b6acaaa2
commit: 65c7382c47af07aea5c1ce86b76cf7f4b6acaaa2
branch: master
author: MaT1g3R <peijun.ma at protonmail.com>
committer: Vinay Sajip <vinay_sajip at yahoo.co.uk>
date: 2019-10-31T10:23:20Z
summary:

Add docstring for shlex.split (GH-16740)

files:
M Lib/shlex.py

diff --git a/Lib/shlex.py b/Lib/shlex.py
index ae0f5ddec1871..c817274583135 100644
--- a/Lib/shlex.py
+++ b/Lib/shlex.py
@@ -303,6 +303,7 @@ def __next__(self):
         return token
 
 def split(s, comments=False, posix=True):
+    """Split the string *s* using shell-like syntax."""
     lex = shlex(s, posix=posix)
     lex.whitespace_split = True
     if not comments:


From webhook-mailer at python.org  Thu Oct 31 08:01:48 2019
From: webhook-mailer at python.org (Miss Skeleton (bot))
Date: Thu, 31 Oct 2019 12:01:48 -0000
Subject: [Python-checkins] Update the URL for the requests package (GH-17006)
Message-ID: <mailman.201.1572523309.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/112f2b805bc83429e8a66a54d088bbefc921abb7
commit: 112f2b805bc83429e8a66a54d088bbefc921abb7
branch: master
author: Simon Legner <Simon.Legner at gmail.com>
committer: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
date: 2019-10-31T05:01:44-07:00
summary:

Update the URL for the requests package (GH-17006)



Change the url from docs.python-requests.org to requests.readthedocs.io

files:
M Doc/library/http.client.rst
M Doc/library/urllib.request.rst

diff --git a/Doc/library/http.client.rst b/Doc/library/http.client.rst
index a458ab77efd82..807dd8bd379a0 100644
--- a/Doc/library/http.client.rst
+++ b/Doc/library/http.client.rst
@@ -20,7 +20,7 @@ HTTPS protocols.  It is normally not used directly --- the module
 
 .. seealso::
 
-    The `Requests package <http://docs.python-requests.org/>`_
+    The `Requests package <https://requests.readthedocs.io/en/master/>`_
     is recommended for a higher-level HTTP client interface.
 
 .. note::
diff --git a/Doc/library/urllib.request.rst b/Doc/library/urllib.request.rst
index a903d605cf4e9..03712c1f4a6ee 100644
--- a/Doc/library/urllib.request.rst
+++ b/Doc/library/urllib.request.rst
@@ -18,7 +18,7 @@ authentication, redirections, cookies and more.
 
 .. seealso::
 
-    The `Requests package <http://docs.python-requests.org/>`_
+    The `Requests package <https://requests.readthedocs.io/en/master/>`_
     is recommended for a higher-level HTTP client interface.
 
 


From webhook-mailer at python.org  Thu Oct 31 08:10:01 2019
From: webhook-mailer at python.org (Miss Skeleton (bot))
Date: Thu, 31 Oct 2019 12:10:01 -0000
Subject: [Python-checkins] Update the URL for the requests package (GH-17006)
Message-ID: <mailman.202.1572523803.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/f3aa1dc3b31906bc0fa74bdd339bf1bf14154361
commit: f3aa1dc3b31906bc0fa74bdd339bf1bf14154361
branch: 3.7
author: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-31T05:09:57-07:00
summary:

Update the URL for the requests package (GH-17006)


Change the url from docs.python-requests.org to requests.readthedocs.io
(cherry picked from commit 112f2b805bc83429e8a66a54d088bbefc921abb7)

Co-authored-by: Simon Legner <Simon.Legner at gmail.com>

files:
M Doc/library/http.client.rst
M Doc/library/urllib.request.rst

diff --git a/Doc/library/http.client.rst b/Doc/library/http.client.rst
index bc73c7a9bdf77..d7757d07a4ba4 100644
--- a/Doc/library/http.client.rst
+++ b/Doc/library/http.client.rst
@@ -20,7 +20,7 @@ HTTPS protocols.  It is normally not used directly --- the module
 
 .. seealso::
 
-    The `Requests package <http://docs.python-requests.org/>`_
+    The `Requests package <https://requests.readthedocs.io/en/master/>`_
     is recommended for a higher-level HTTP client interface.
 
 .. note::
diff --git a/Doc/library/urllib.request.rst b/Doc/library/urllib.request.rst
index 5d8c0ec4303cd..22bad2dc6a2eb 100644
--- a/Doc/library/urllib.request.rst
+++ b/Doc/library/urllib.request.rst
@@ -18,7 +18,7 @@ authentication, redirections, cookies and more.
 
 .. seealso::
 
-    The `Requests package <http://docs.python-requests.org/>`_
+    The `Requests package <https://requests.readthedocs.io/en/master/>`_
     is recommended for a higher-level HTTP client interface.
 
 


From webhook-mailer at python.org  Thu Oct 31 08:12:28 2019
From: webhook-mailer at python.org (Miss Skeleton (bot))
Date: Thu, 31 Oct 2019 12:12:28 -0000
Subject: [Python-checkins] Update the URL for the requests package (GH-17006)
Message-ID: <mailman.203.1572523949.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/2ed52b86aa28b2a10f4e2cc28a2b6d63e57bc73c
commit: 2ed52b86aa28b2a10f4e2cc28a2b6d63e57bc73c
branch: 3.8
author: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
committer: GitHub <noreply at github.com>
date: 2019-10-31T05:12:23-07:00
summary:

Update the URL for the requests package (GH-17006)


Change the url from docs.python-requests.org to requests.readthedocs.io
(cherry picked from commit 112f2b805bc83429e8a66a54d088bbefc921abb7)

Co-authored-by: Simon Legner <Simon.Legner at gmail.com>

files:
M Doc/library/http.client.rst
M Doc/library/urllib.request.rst

diff --git a/Doc/library/http.client.rst b/Doc/library/http.client.rst
index db26d56af30fb..9cdabc2a03a75 100644
--- a/Doc/library/http.client.rst
+++ b/Doc/library/http.client.rst
@@ -20,7 +20,7 @@ HTTPS protocols.  It is normally not used directly --- the module
 
 .. seealso::
 
-    The `Requests package <http://docs.python-requests.org/>`_
+    The `Requests package <https://requests.readthedocs.io/en/master/>`_
     is recommended for a higher-level HTTP client interface.
 
 .. note::
diff --git a/Doc/library/urllib.request.rst b/Doc/library/urllib.request.rst
index 448bc67854704..adc0593e8094a 100644
--- a/Doc/library/urllib.request.rst
+++ b/Doc/library/urllib.request.rst
@@ -18,7 +18,7 @@ authentication, redirections, cookies and more.
 
 .. seealso::
 
-    The `Requests package <http://docs.python-requests.org/>`_
+    The `Requests package <https://requests.readthedocs.io/en/master/>`_
     is recommended for a higher-level HTTP client interface.
 
 


From webhook-mailer at python.org  Thu Oct 31 08:50:16 2019
From: webhook-mailer at python.org (Vinay Sajip)
Date: Thu, 31 Oct 2019 12:50:16 -0000
Subject: [Python-checkins] Add docstring for shlex.split (GH-16740)
 (GH-17013)
Message-ID: <mailman.204.1572526217.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/c1ebe6a1259ce988dc7d1fc547386b142dfe4364
commit: c1ebe6a1259ce988dc7d1fc547386b142dfe4364
branch: 3.8
author: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Vinay Sajip <vinay_sajip at yahoo.co.uk>
date: 2019-10-31T12:50:04Z
summary:

Add docstring for shlex.split (GH-16740) (GH-17013)

(cherry picked from commit 65c7382c47af07aea5c1ce86b76cf7f4b6acaaa2)

Co-authored-by: MaT1g3R <peijun.ma at protonmail.com>

files:
M Lib/shlex.py

diff --git a/Lib/shlex.py b/Lib/shlex.py
index ae0f5ddec1871..c817274583135 100644
--- a/Lib/shlex.py
+++ b/Lib/shlex.py
@@ -303,6 +303,7 @@ def __next__(self):
         return token
 
 def split(s, comments=False, posix=True):
+    """Split the string *s* using shell-like syntax."""
     lex = shlex(s, posix=posix)
     lex.whitespace_split = True
     if not comments:


From webhook-mailer at python.org  Thu Oct 31 08:50:36 2019
From: webhook-mailer at python.org (Vinay Sajip)
Date: Thu, 31 Oct 2019 12:50:36 -0000
Subject: [Python-checkins] Add docstring for shlex.split (GH-16740)
 (GH-17012)
Message-ID: <mailman.205.1572526237.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/a28cf14deaa3e62c995f119036d2e63346cadbb8
commit: a28cf14deaa3e62c995f119036d2e63346cadbb8
branch: 3.7
author: Miss Skeleton (bot) <31488909+miss-islington at users.noreply.github.com>
committer: Vinay Sajip <vinay_sajip at yahoo.co.uk>
date: 2019-10-31T12:50:32Z
summary:

Add docstring for shlex.split (GH-16740) (GH-17012)

(cherry picked from commit 65c7382c47af07aea5c1ce86b76cf7f4b6acaaa2)

Co-authored-by: MaT1g3R <peijun.ma at protonmail.com>

files:
M Lib/shlex.py

diff --git a/Lib/shlex.py b/Lib/shlex.py
index 195dc12bbce95..48e31f4799b44 100644
--- a/Lib/shlex.py
+++ b/Lib/shlex.py
@@ -302,6 +302,7 @@ def __next__(self):
         return token
 
 def split(s, comments=False, posix=True):
+    """Split the string *s* using shell-like syntax."""
     lex = shlex(s, posix=posix)
     lex.whitespace_split = True
     if not comments:


From webhook-mailer at python.org  Thu Oct 31 09:34:12 2019
From: webhook-mailer at python.org (Vinay Sajip)
Date: Thu, 31 Oct 2019 13:34:12 -0000
Subject: [Python-checkins] [3.8] bpo-16575: Add checks for unions passed by
 value to functions. (GH-16799) (GH-17016)
Message-ID: <mailman.206.1572528852.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/952899704800a5aa438e84d50f0b6fc498c72e89
commit: 952899704800a5aa438e84d50f0b6fc498c72e89
branch: 3.8
author: Vinay Sajip <vinay_sajip at yahoo.co.uk>
committer: GitHub <noreply at github.com>
date: 2019-10-31T13:34:05Z
summary:

[3.8] bpo-16575: Add checks for unions passed by value to functions. (GH-16799) (GH-17016)

(cherry picked from commit 79d4ed102a5069c6cebaed2627cb1645637f0429)

files:
M Lib/ctypes/test/test_structures.py
M Modules/_ctypes/_ctypes.c
M Modules/_ctypes/_ctypes_test.c
M Modules/_ctypes/ctypes.h
M Modules/_ctypes/stgdict.c

diff --git a/Lib/ctypes/test/test_structures.py b/Lib/ctypes/test/test_structures.py
index 1efd36dd62f49..8541674066546 100644
--- a/Lib/ctypes/test/test_structures.py
+++ b/Lib/ctypes/test/test_structures.py
@@ -571,6 +571,86 @@ class U(Union):
             self.assertEqual(f2, [0x4567, 0x0123, 0xcdef, 0x89ab,
                                   0x3210, 0x7654, 0xba98, 0xfedc])
 
+    def test_union_by_value(self):
+        # See bpo-16575
+
+        # These should mirror the structures in Modules/_ctypes/_ctypes_test.c
+
+        class Nested1(Structure):
+            _fields_ = [
+                ('an_int', c_int),
+                ('another_int', c_int),
+            ]
+
+        class Test4(Union):
+            _fields_ = [
+                ('a_long', c_long),
+                ('a_struct', Nested1),
+            ]
+
+        class Nested2(Structure):
+            _fields_ = [
+                ('an_int', c_int),
+                ('a_union', Test4),
+            ]
+
+        class Test5(Structure):
+            _fields_ = [
+                ('an_int', c_int),
+                ('nested', Nested2),
+                ('another_int', c_int),
+            ]
+
+        test4 = Test4()
+        dll = CDLL(_ctypes_test.__file__)
+        with self.assertRaises(TypeError) as ctx:
+            func = dll._testfunc_union_by_value1
+            func.restype = c_long
+            func.argtypes = (Test4,)
+            result = func(test4)
+        self.assertEqual(ctx.exception.args[0], 'item 1 in _argtypes_ passes '
+                         'a union by value, which is unsupported.')
+        test5 = Test5()
+        with self.assertRaises(TypeError) as ctx:
+            func = dll._testfunc_union_by_value2
+            func.restype = c_long
+            func.argtypes = (Test5,)
+            result = func(test5)
+        self.assertEqual(ctx.exception.args[0], 'item 1 in _argtypes_ passes '
+                         'a union by value, which is unsupported.')
+
+        # passing by reference should be OK
+        test4.a_long = 12345;
+        func = dll._testfunc_union_by_reference1
+        func.restype = c_long
+        func.argtypes = (POINTER(Test4),)
+        result = func(byref(test4))
+        self.assertEqual(result, 12345)
+        self.assertEqual(test4.a_long, 0)
+        self.assertEqual(test4.a_struct.an_int, 0)
+        self.assertEqual(test4.a_struct.another_int, 0)
+        test4.a_struct.an_int = 0x12340000
+        test4.a_struct.another_int = 0x5678
+        func = dll._testfunc_union_by_reference2
+        func.restype = c_long
+        func.argtypes = (POINTER(Test4),)
+        result = func(byref(test4))
+        self.assertEqual(result, 0x12345678)
+        self.assertEqual(test4.a_long, 0)
+        self.assertEqual(test4.a_struct.an_int, 0)
+        self.assertEqual(test4.a_struct.another_int, 0)
+        test5.an_int = 0x12000000
+        test5.nested.an_int = 0x345600
+        test5.another_int = 0x78
+        func = dll._testfunc_union_by_reference3
+        func.restype = c_long
+        func.argtypes = (POINTER(Test5),)
+        result = func(byref(test5))
+        self.assertEqual(result, 0x12345678)
+        self.assertEqual(test5.an_int, 0)
+        self.assertEqual(test5.nested.an_int, 0)
+        self.assertEqual(test5.another_int, 0)
+
 class PointerMemberTestCase(unittest.TestCase):
 
     def test(self):
diff --git a/Modules/_ctypes/_ctypes.c b/Modules/_ctypes/_ctypes.c
index 0f11e01e11105..a2fddda87a53e 100644
--- a/Modules/_ctypes/_ctypes.c
+++ b/Modules/_ctypes/_ctypes.c
@@ -504,6 +504,9 @@ StructUnionType_new(PyTypeObject *type, PyObject *args, PyObject *kwds, int isSt
         Py_DECREF(result);
         return NULL;
     }
+    if (!isStruct) {
+        dict->flags |= TYPEFLAG_HASUNION;
+    }
     /* replace the class dict by our updated stgdict, which holds info
        about storage requirements of the instances */
     if (-1 == PyDict_Update((PyObject *)dict, result->tp_dict)) {
@@ -2383,6 +2386,27 @@ converters_from_argtypes(PyObject *ob)
     for (i = 0; i < nArgs; ++i) {
         PyObject *cnv;
         PyObject *tp = PyTuple_GET_ITEM(ob, i);
+        StgDictObject *stgdict = PyType_stgdict(tp);
+
+        if (stgdict != NULL) {
+            if (stgdict->flags & TYPEFLAG_HASUNION) {
+                Py_DECREF(converters);
+                Py_DECREF(ob);
+                if (!PyErr_Occurred()) {
+                    PyErr_Format(PyExc_TypeError,
+                                 "item %zd in _argtypes_ passes a union by "
+                                 "value, which is unsupported.",
+                                 i + 1);
+                }
+                return NULL;
+            }
+            /*
+            if (stgdict->flags & TYPEFLAG_HASBITFIELD) {
+                printf("found stgdict with bitfield\n");
+            }
+            */
+        }
+
         if (_PyObject_LookupAttrId(tp, &PyId_from_param, &cnv) <= 0) {
             Py_DECREF(converters);
             Py_DECREF(ob);
diff --git a/Modules/_ctypes/_ctypes_test.c b/Modules/_ctypes/_ctypes_test.c
index 19066b31c1c87..223be850a6d73 100644
--- a/Modules/_ctypes/_ctypes_test.c
+++ b/Modules/_ctypes/_ctypes_test.c
@@ -135,6 +135,69 @@ _testfunc_array_in_struct2a(Test3B in)
     return result;
 }
 
+typedef union {
+    long a_long;
+    struct {
+        int an_int;
+        int another_int;
+    } a_struct;
+} Test4;
+
+typedef struct {
+    int an_int;
+    struct {
+        int an_int;
+        Test4 a_union;
+    } nested;
+    int another_int;
+} Test5;
+
+EXPORT(long)
+_testfunc_union_by_value1(Test4 in) {
+    long result = in.a_long + in.a_struct.an_int + in.a_struct.another_int;
+
+    /* As the union/struct are passed by value, changes to them shouldn't be
+     * reflected in the caller.
+     */
+    memset(&in, 0, sizeof(in));
+    return result;
+}
+
+EXPORT(long)
+_testfunc_union_by_value2(Test5 in) {
+    long result = in.an_int + in.nested.an_int;
+
+    /* As the union/struct are passed by value, changes to them shouldn't be
+     * reflected in the caller.
+     */
+    memset(&in, 0, sizeof(in));
+    return result;
+}
+
+EXPORT(long)
+_testfunc_union_by_reference1(Test4 *in) {
+    long result = in->a_long;
+
+    memset(in, 0, sizeof(Test4));
+    return result;
+}
+
+EXPORT(long)
+_testfunc_union_by_reference2(Test4 *in) {
+    long result = in->a_struct.an_int + in->a_struct.another_int;
+
+    memset(in, 0, sizeof(Test4));
+    return result;
+}
+
+EXPORT(long)
+_testfunc_union_by_reference3(Test5 *in) {
+    long result = in->an_int + in->nested.an_int + in->another_int;
+
+    memset(in, 0, sizeof(Test5));
+    return result;
+}
+
 EXPORT(void)testfunc_array(int values[4])
 {
     printf("testfunc_array %d %d %d %d\n",
diff --git a/Modules/_ctypes/ctypes.h b/Modules/_ctypes/ctypes.h
index 5d3b9663385f5..e58f85233cb71 100644
--- a/Modules/_ctypes/ctypes.h
+++ b/Modules/_ctypes/ctypes.h
@@ -288,6 +288,8 @@ PyObject *_ctypes_callproc(PPROC pProc,
 
 #define TYPEFLAG_ISPOINTER 0x100
 #define TYPEFLAG_HASPOINTER 0x200
+#define TYPEFLAG_HASUNION 0x400
+#define TYPEFLAG_HASBITFIELD 0x800
 
 #define DICTFLAG_FINAL 0x1000
 
diff --git a/Modules/_ctypes/stgdict.c b/Modules/_ctypes/stgdict.c
index 97bcf5539aeea..1d45ade5efd90 100644
--- a/Modules/_ctypes/stgdict.c
+++ b/Modules/_ctypes/stgdict.c
@@ -440,6 +440,13 @@ PyCStructUnionType_update_stgdict(PyObject *type, PyObject *fields, int isStruct
         PyMem_Free(stgdict->ffi_type_pointer.elements);
 
     basedict = PyType_stgdict((PyObject *)((PyTypeObject *)type)->tp_base);
+    if (basedict) {
+        stgdict->flags |= (basedict->flags &
+                           (TYPEFLAG_HASUNION | TYPEFLAG_HASBITFIELD));
+    }
+    if (!isStruct) {
+        stgdict->flags |= TYPEFLAG_HASUNION;
+    }
     if (basedict && !use_broken_old_ctypes_semantics) {
         size = offset = basedict->size;
         align = basedict->align;
@@ -515,8 +522,10 @@ PyCStructUnionType_update_stgdict(PyObject *type, PyObject *fields, int isStruct
         stgdict->ffi_type_pointer.elements[ffi_ofs + i] = &dict->ffi_type_pointer;
         if (dict->flags & (TYPEFLAG_ISPOINTER | TYPEFLAG_HASPOINTER))
             stgdict->flags |= TYPEFLAG_HASPOINTER;
+        stgdict->flags |= dict->flags & (TYPEFLAG_HASUNION | TYPEFLAG_HASBITFIELD);
         dict->flags |= DICTFLAG_FINAL; /* mark field type final */
         if (PyTuple_Size(pair) == 3) { /* bits specified */
+            stgdict->flags |= TYPEFLAG_HASBITFIELD;
             switch(dict->ffi_type_pointer.type) {
             case FFI_TYPE_UINT8:
             case FFI_TYPE_UINT16:


From webhook-mailer at python.org  Thu Oct 31 10:15:19 2019
From: webhook-mailer at python.org (Vinay Sajip)
Date: Thu, 31 Oct 2019 14:15:19 -0000
Subject: [Python-checkins] [3.7] bpo-16575: Add checks for unions passed by
 value to functions. (GH-16799) (GH-17017)
Message-ID: <mailman.207.1572531319.32329.python-checkins@python.org>

https://github.com/python/cpython/commit/0118d109d54bf75c99a8b0fa9aeae1a478ac4b7e
commit: 0118d109d54bf75c99a8b0fa9aeae1a478ac4b7e
branch: 3.7
author: Vinay Sajip <vinay_sajip at yahoo.co.uk>
committer: GitHub <noreply at github.com>
date: 2019-10-31T14:15:04Z
summary:

[3.7] bpo-16575: Add checks for unions passed by value to functions. (GH-16799) (GH-17017)

(cherry picked from commit 79d4ed102a5069c6cebaed2627cb1645637f0429)

files:
M Lib/ctypes/test/test_structures.py
M Modules/_ctypes/_ctypes.c
M Modules/_ctypes/_ctypes_test.c
M Modules/_ctypes/ctypes.h
M Modules/_ctypes/stgdict.c

diff --git a/Lib/ctypes/test/test_structures.py b/Lib/ctypes/test/test_structures.py
index d8d6c657928bf..8e2897ad51230 100644
--- a/Lib/ctypes/test/test_structures.py
+++ b/Lib/ctypes/test/test_structures.py
@@ -532,6 +532,86 @@ class U(Union):
             self.assertEqual(f2, [0x4567, 0x0123, 0xcdef, 0x89ab,
                                   0x3210, 0x7654, 0xba98, 0xfedc])
 
+    def test_union_by_value(self):
+        # See bpo-16575
+
+        # These should mirror the structures in Modules/_ctypes/_ctypes_test.c
+
+        class Nested1(Structure):
+            _fields_ = [
+                ('an_int', c_int),
+                ('another_int', c_int),
+            ]
+
+        class Test4(Union):
+            _fields_ = [
+                ('a_long', c_long),
+                ('a_struct', Nested1),
+            ]
+
+        class Nested2(Structure):
+            _fields_ = [
+                ('an_int', c_int),
+                ('a_union', Test4),
+            ]
+
+        class Test5(Structure):
+            _fields_ = [
+                ('an_int', c_int),
+                ('nested', Nested2),
+                ('another_int', c_int),
+            ]
+
+        test4 = Test4()
+        dll = CDLL(_ctypes_test.__file__)
+        with self.assertRaises(TypeError) as ctx:
+            func = dll._testfunc_union_by_value1
+            func.restype = c_long
+            func.argtypes = (Test4,)
+            result = func(test4)
+        self.assertEqual(ctx.exception.args[0], 'item 1 in _argtypes_ passes '
+                         'a union by value, which is unsupported.')
+        test5 = Test5()
+        with self.assertRaises(TypeError) as ctx:
+            func = dll._testfunc_union_by_value2
+            func.restype = c_long
+            func.argtypes = (Test5,)
+            result = func(test5)
+        self.assertEqual(ctx.exception.args[0], 'item 1 in _argtypes_ passes '
+                         'a union by value, which is unsupported.')
+
+        # passing by reference should be OK
+        test4.a_long = 12345;
+        func = dll._testfunc_union_by_reference1
+        func.restype = c_long
+        func.argtypes = (POINTER(Test4),)
+        result = func(byref(test4))
+        self.assertEqual(result, 12345)
+        self.assertEqual(test4.a_long, 0)
+        self.assertEqual(test4.a_struct.an_int, 0)
+        self.assertEqual(test4.a_struct.another_int, 0)
+        test4.a_struct.an_int = 0x12340000
+        test4.a_struct.another_int = 0x5678
+        func = dll._testfunc_union_by_reference2
+        func.restype = c_long
+        func.argtypes = (POINTER(Test4),)
+        result = func(byref(test4))
+        self.assertEqual(result, 0x12345678)
+        self.assertEqual(test4.a_long, 0)
+        self.assertEqual(test4.a_struct.an_int, 0)
+        self.assertEqual(test4.a_struct.another_int, 0)
+        test5.an_int = 0x12000000
+        test5.nested.an_int = 0x345600
+        test5.another_int = 0x78
+        func = dll._testfunc_union_by_reference3
+        func.restype = c_long
+        func.argtypes = (POINTER(Test5),)
+        result = func(byref(test5))
+        self.assertEqual(result, 0x12345678)
+        self.assertEqual(test5.an_int, 0)
+        self.assertEqual(test5.nested.an_int, 0)
+        self.assertEqual(test5.another_int, 0)
+
 class PointerMemberTestCase(unittest.TestCase):
 
     def test(self):
diff --git a/Modules/_ctypes/_ctypes.c b/Modules/_ctypes/_ctypes.c
index f186e5082afbf..d2ce1b074348d 100644
--- a/Modules/_ctypes/_ctypes.c
+++ b/Modules/_ctypes/_ctypes.c
@@ -446,6 +446,9 @@ StructUnionType_new(PyTypeObject *type, PyObject *args, PyObject *kwds, int isSt
         Py_DECREF(result);
         return NULL;
     }
+    if (!isStruct) {
+        dict->flags |= TYPEFLAG_HASUNION;
+    }
     /* replace the class dict by our updated stgdict, which holds info
        about storage requirements of the instances */
     if (-1 == PyDict_Update((PyObject *)dict, result->tp_dict)) {
@@ -2276,6 +2279,26 @@ converters_from_argtypes(PyObject *ob)
         PyObject *cnv = PyObject_GetAttrString(tp, "from_param");
         if (!cnv)
             goto argtypes_error_1;
+        StgDictObject *stgdict = PyType_stgdict(tp);
+
+        if (stgdict != NULL) {
+            if (stgdict->flags & TYPEFLAG_HASUNION) {
+                Py_DECREF(converters);
+                Py_DECREF(ob);
+                if (!PyErr_Occurred()) {
+                    PyErr_Format(PyExc_TypeError,
+                                 "item %zd in _argtypes_ passes a union by "
+                                 "value, which is unsupported.",
+                                 i + 1);
+                }
+                return NULL;
+            }
+            /*
+            if (stgdict->flags & TYPEFLAG_HASBITFIELD) {
+                printf("found stgdict with bitfield\n");
+            }
+            */
+        }
         PyTuple_SET_ITEM(converters, i, cnv);
     }
     Py_DECREF(ob);
diff --git a/Modules/_ctypes/_ctypes_test.c b/Modules/_ctypes/_ctypes_test.c
index f59abcc451591..e06a045545578 100644
--- a/Modules/_ctypes/_ctypes_test.c
+++ b/Modules/_ctypes/_ctypes_test.c
@@ -135,6 +135,69 @@ _testfunc_array_in_struct2a(Test3B in)
     return result;
 }
 
+typedef union {
+    long a_long;
+    struct {
+        int an_int;
+        int another_int;
+    } a_struct;
+} Test4;
+
+typedef struct {
+    int an_int;
+    struct {
+        int an_int;
+        Test4 a_union;
+    } nested;
+    int another_int;
+} Test5;
+
+EXPORT(long)
+_testfunc_union_by_value1(Test4 in) {
+    long result = in.a_long + in.a_struct.an_int + in.a_struct.another_int;
+
+    /* As the union/struct are passed by value, changes to them shouldn't be
+     * reflected in the caller.
+     */
+    memset(&in, 0, sizeof(in));
+    return result;
+}
+
+EXPORT(long)
+_testfunc_union_by_value2(Test5 in) {
+    long result = in.an_int + in.nested.an_int;
+
+    /* As the union/struct are passed by value, changes to them shouldn't be
+     * reflected in the caller.
+     */
+    memset(&in, 0, sizeof(in));
+    return result;
+}
+
+EXPORT(long)
+_testfunc_union_by_reference1(Test4 *in) {
+    long result = in->a_long;
+
+    memset(in, 0, sizeof(Test4));
+    return result;
+}
+
+EXPORT(long)
+_testfunc_union_by_reference2(Test4 *in) {
+    long result = in->a_struct.an_int + in->a_struct.another_int;
+
+    memset(in, 0, sizeof(Test4));
+    return result;
+}
+
+EXPORT(long)
+_testfunc_union_by_reference3(Test5 *in) {
+    long result = in->an_int + in->nested.an_int + in->another_int;
+
+    memset(in, 0, sizeof(Test5));
+    return result;
+}
+
 EXPORT(void)testfunc_array(int values[4])
 {
     printf("testfunc_array %d %d %d %d\n",
diff --git a/Modules/_ctypes/ctypes.h b/Modules/_ctypes/ctypes.h
index 5d3b9663385f5..e58f85233cb71 100644
--- a/Modules/_ctypes/ctypes.h
+++ b/Modules/_ctypes/ctypes.h
@@ -288,6 +288,8 @@ PyObject *_ctypes_callproc(PPROC pProc,
 
 #define TYPEFLAG_ISPOINTER 0x100
 #define TYPEFLAG_HASPOINTER 0x200
+#define TYPEFLAG_HASUNION 0x400
+#define TYPEFLAG_HASBITFIELD 0x800
 
 #define DICTFLAG_FINAL 0x1000
 
diff --git a/Modules/_ctypes/stgdict.c b/Modules/_ctypes/stgdict.c
index af12d314ab98f..231c3feb8f335 100644
--- a/Modules/_ctypes/stgdict.c
+++ b/Modules/_ctypes/stgdict.c
@@ -407,6 +407,13 @@ PyCStructUnionType_update_stgdict(PyObject *type, PyObject *fields, int isStruct
         PyMem_Free(stgdict->ffi_type_pointer.elements);
 
     basedict = PyType_stgdict((PyObject *)((PyTypeObject *)type)->tp_base);
+    if (basedict) {
+        stgdict->flags |= (basedict->flags &
+                           (TYPEFLAG_HASUNION | TYPEFLAG_HASBITFIELD));
+    }
+    if (!isStruct) {
+        stgdict->flags |= TYPEFLAG_HASUNION;
+    }
     if (basedict && !use_broken_old_ctypes_semantics) {
         size = offset = basedict->size;
         align = basedict->align;
@@ -482,8 +489,10 @@ PyCStructUnionType_update_stgdict(PyObject *type, PyObject *fields, int isStruct
         stgdict->ffi_type_pointer.elements[ffi_ofs + i] = &dict->ffi_type_pointer;
         if (dict->flags & (TYPEFLAG_ISPOINTER | TYPEFLAG_HASPOINTER))
             stgdict->flags |= TYPEFLAG_HASPOINTER;
+        stgdict->flags |= dict->flags & (TYPEFLAG_HASUNION | TYPEFLAG_HASBITFIELD);
         dict->flags |= DICTFLAG_FINAL; /* mark field type final */
         if (PyTuple_Size(pair) == 3) { /* bits specified */
+            stgdict->flags |= TYPEFLAG_HASBITFIELD;
             switch(dict->ffi_type_pointer.type) {
             case FFI_TYPE_UINT8:
             case FFI_TYPE_UINT16:


From webhook-mailer at python.org  Tue Oct  1 03:49:42 2019
From: webhook-mailer at python.org (=?utf-8?q?=C5=81ukasz?= Langa)
Date: Tue, 01 Oct 2019 07:49:42 -0000
Subject: [Python-checkins] (no subject)
Message-ID: <mailman.575.1599367012.9553.python-checkins@python.org>




To: python-checkins at python.org
Subject: Restore tp_clear for function object. (#16502)
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

https://github.com/python/cpython/commit/b3612070b746f799901443b65725008bc035=
872b
commit: b3612070b746f799901443b65725008bc035872b
branch: 3.8
author: Neil Schemenauer <nas-github at arctrix.com>
committer: =C5=81ukasz Langa <lukasz at langa.pl>
date: 2019-10-01T09:49:36+02:00
summary:

Restore tp_clear for function object. (#16502)

This is a revert of the revert (GH-15826).  Having a tp_clear for
functions should be safe (and helpful) now that bpo-38006 has been
fixed.

files:
M Objects/funcobject.c

diff --git a/Objects/funcobject.c b/Objects/funcobject.c
index 53aebf12f550..df5cc2d3f570 100644
--- a/Objects/funcobject.c
+++ b/Objects/funcobject.c
@@ -570,23 +570,31 @@ func_new_impl(PyTypeObject *type, PyCodeObject *code, P=
yObject *globals,
     return (PyObject *)newfunc;
 }
=20
+static int
+func_clear(PyFunctionObject *op)
+{
+    Py_CLEAR(op->func_code);
+    Py_CLEAR(op->func_globals);
+    Py_CLEAR(op->func_module);
+    Py_CLEAR(op->func_name);
+    Py_CLEAR(op->func_defaults);
+    Py_CLEAR(op->func_kwdefaults);
+    Py_CLEAR(op->func_doc);
+    Py_CLEAR(op->func_dict);
+    Py_CLEAR(op->func_closure);
+    Py_CLEAR(op->func_annotations);
+    Py_CLEAR(op->func_qualname);
+    return 0;
+}
+
 static void
 func_dealloc(PyFunctionObject *op)
 {
     _PyObject_GC_UNTRACK(op);
-    if (op->func_weakreflist !=3D NULL)
+    if (op->func_weakreflist !=3D NULL) {
         PyObject_ClearWeakRefs((PyObject *) op);
-    Py_DECREF(op->func_code);
-    Py_DECREF(op->func_globals);
-    Py_XDECREF(op->func_module);
-    Py_DECREF(op->func_name);
-    Py_XDECREF(op->func_defaults);
-    Py_XDECREF(op->func_kwdefaults);
-    Py_XDECREF(op->func_doc);
-    Py_XDECREF(op->func_dict);
-    Py_XDECREF(op->func_closure);
-    Py_XDECREF(op->func_annotations);
-    Py_XDECREF(op->func_qualname);
+    }
+    (void)func_clear(op);
     PyObject_GC_Del(op);
 }
=20
@@ -661,7 +669,7 @@ PyTypeObject PyFunction_Type =3D {
     Py_TPFLAGS_METHOD_DESCRIPTOR,               /* tp_flags */
     func_new__doc__,                            /* tp_doc */
     (traverseproc)func_traverse,                /* tp_traverse */
-    0,                                          /* tp_clear */
+    (inquiry)func_clear,                        /* tp_clear */
     0,                                          /* tp_richcompare */
     offsetof(PyFunctionObject, func_weakreflist), /* tp_weaklistoffset */
     0,                                          /* tp_iter */


From webhook-mailer at python.org  Tue Oct  1 03:55:08 2019
From: webhook-mailer at python.org (=?utf-8?q?=C5=81ukasz?= Langa)
Date: Tue, 01 Oct 2019 07:55:08 -0000
Subject: [Python-checkins] (no subject)
Message-ID: <mailman.576.1599367013.9553.python-checkins@python.org>




To: python-checkins at python.org
Subject: bpo-38319: Fix shutil._fastcopy_sendfile(): set sendfile() max block
 size (GH-16491) (#16506)
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

https://github.com/python/cpython/commit/938c00ca9e4207a2531041edff2e82490b02=
047f
commit: 938c00ca9e4207a2531041edff2e82490b02047f
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.co=
m>
committer: =C5=81ukasz Langa <lukasz at langa.pl>
date: 2019-10-01T09:55:02+02:00
summary:

bpo-38319: Fix shutil._fastcopy_sendfile(): set sendfile() max block size (GH=
-16491) (#16506)

(cherry picked from commit 94e165096fd65e8237e60de570fb609604ab94c9)

Co-authored-by: Giampaolo Rodola <g.rodola at gmail.com>

files:
A Misc/NEWS.d/next/Library/2019-09-30-22-06-33.bpo-38319.5QjiDa.rst
M Lib/shutil.py
M Lib/socket.py

diff --git a/Lib/shutil.py b/Lib/shutil.py
index 5c1255a6713d..1e89256cc344 100644
--- a/Lib/shutil.py
+++ b/Lib/shutil.py
@@ -135,9 +135,13 @@ def _fastcopy_sendfile(fsrc, fdst):
     # should not make any difference, also in case the file content
     # changes while being copied.
     try:
-        blocksize =3D max(os.fstat(infd).st_size, 2 ** 23)  # min 8MB
-    except Exception:
-        blocksize =3D 2 ** 27  # 128MB
+        blocksize =3D max(os.fstat(infd).st_size, 2 ** 23)  # min 8MiB
+    except OSError:
+        blocksize =3D 2 ** 27  # 128MiB
+    # On 32-bit architectures truncate to 1GiB to avoid OverflowError,
+    # see bpo-38319.
+    if sys.maxsize < 2 ** 32:
+        blocksize =3D min(blocksize, 2 ** 30)
=20
     offset =3D 0
     while True:
diff --git a/Lib/socket.py b/Lib/socket.py
index af2ed0e76a49..813f4ef5c3e1 100644
--- a/Lib/socket.py
+++ b/Lib/socket.py
@@ -355,8 +355,8 @@ def _sendfile_use_sendfile(self, file, offset=3D0, count=
=3DNone):
                 raise _GiveupOnSendfile(err)  # not a regular file
             if not fsize:
                 return 0  # empty file
-            blocksize =3D fsize if not count else count
-
+            # Truncate to 1GiB to avoid OverflowError, see bpo-38319.
+            blocksize =3D min(count or fsize, 2 ** 30)
             timeout =3D self.gettimeout()
             if timeout =3D=3D 0:
                 raise ValueError("non-blocking sockets are not supported")
diff --git a/Misc/NEWS.d/next/Library/2019-09-30-22-06-33.bpo-38319.5QjiDa.rs=
t b/Misc/NEWS.d/next/Library/2019-09-30-22-06-33.bpo-38319.5QjiDa.rst
new file mode 100644
index 000000000000..376a9e459de2
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2019-09-30-22-06-33.bpo-38319.5QjiDa.rst
@@ -0,0 +1,2 @@
+sendfile() used in socket and shutil modules was raising OverflowError for
+files >=3D 2GiB on 32-bit architectures.  (patch by Giampaolo Rodola)


From webhook-mailer at python.org  Tue Oct  1 16:51:30 2019
From: webhook-mailer at python.org (=?utf-8?q?=C5=81ukasz?= Langa)
Date: Tue, 01 Oct 2019 20:51:30 -0000
Subject: [Python-checkins] (no subject)
Message-ID: <mailman.577.1599367015.9553.python-checkins@python.org>




To: python-checkins at python.org
Subject: v3.8.0rc1
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

https://github.com/python/cpython/commit/34214de6ab9ebcb5d63d0e116800d5ee535a=
38cb
commit: 34214de6ab9ebcb5d63d0e116800d5ee535a38cb
branch: 3.8
author: =C5=81ukasz Langa <lukasz at langa.pl>
committer: =C5=81ukasz Langa <lukasz at langa.pl>
date: 2019-10-01T14:58:26+02:00
summary:

v3.8.0rc1

files:
A Misc/NEWS.d/3.8.0rc1.rst
D Misc/NEWS.d/next/Build/2019-08-24-00-29-40.bpo-37936.QrORqA.rst
D Misc/NEWS.d/next/Build/2019-09-10-00-54-48.bpo-37936.E7XEwu.rst
D Misc/NEWS.d/next/Build/2019-09-13-14-12-36.bpo-36002.Bcl4oe.rst
D Misc/NEWS.d/next/Build/2019-09-24-22-47-47.bpo-36210.EmL9X1.rst
D Misc/NEWS.d/next/Build/2019-09-28-02-37-11.bpo-38301.123456.rst
D Misc/NEWS.d/next/C API/2019-08-17-13-50-21.bpo-37879.CZeUem.rst
D Misc/NEWS.d/next/C API/2019-09-19-18-26-29.bpo-38205.Db1OJL.rst
D Misc/NEWS.d/next/C API/2019-09-20-17-22-41.bpo-38234.ZbquVK.rst
D Misc/NEWS.d/next/C API/2019-09-24-17-09-48.bpo-38234.d0bhEA.rst
D Misc/NEWS.d/next/Core and Builtins/2018-06-07-01-01-20.bpo-30773.C31rVE.rst
D Misc/NEWS.d/next/Core and Builtins/2019-07-18-11-50-49.bpo-37619.X6Lulo.rst
D Misc/NEWS.d/next/Core and Builtins/2019-08-06-23-39-05.bpo-37409.1qwzn2.rst
D Misc/NEWS.d/next/Core and Builtins/2019-08-15-12-48-36.bpo-20490.-hXeEn.rst
D Misc/NEWS.d/next/Core and Builtins/2019-08-27-21-21-36.bpo-37966.5OBLez.rst
D Misc/NEWS.d/next/Core and Builtins/2019-08-31-09-22-33.bpo-37990.WDY2f-.rst
D Misc/NEWS.d/next/Core and Builtins/2019-08-31-11-13-25.bpo-37994.Rj6S4j.rst
D Misc/NEWS.d/next/Core and Builtins/2019-09-02-20-00-31.bpo-38005.e7VsTA.rst
D Misc/NEWS.d/next/Core and Builtins/2019-09-12-00-14-01.bpo-38124.n6E0H7.rst
D Misc/NEWS.d/next/Core and Builtins/2019-09-12-19-50-01.bpo-38013.I7btD0.rst
D Misc/NEWS.d/next/Core and Builtins/2019-09-20-19-06-23.bpo-38236.eQ0Tmj.rst
D Misc/NEWS.d/next/Core and Builtins/2019-09-24-18-45-46.bpo-36871.p47knk.rst
D Misc/NEWS.d/next/Core and Builtins/2019-09-30-00-56-21.bpo-38317.pmqlIQ.rst
D Misc/NEWS.d/next/Core and Builtins/2019-09-30-09-33-21.bpo-38006.UYlJum.rst
D Misc/NEWS.d/next/Documentation/2018-07-31-15-38-26.bpo-34293.yHupAL.rst
D Misc/NEWS.d/next/Documentation/2018-10-26-18-10-29.bpo-33944.V1YeOA.rst
D Misc/NEWS.d/next/Documentation/2019-01-21-14-30-59.bpo-35803.yae6Lq.rst
D Misc/NEWS.d/next/Documentation/2019-02-14-07-12-48.bpo-23460.Iqiqtm.rst
D Misc/NEWS.d/next/Documentation/2019-05-18-16-25-44.bpo-36960.xEKHXj.rst
D Misc/NEWS.d/next/Documentation/2019-06-04-09-29-00.bpo-36260.WrGuc-.rst
D Misc/NEWS.d/next/Documentation/2019-08-24-12-59-06.bpo-37937.F7fHbt.rst
D Misc/NEWS.d/next/Documentation/2019-08-29-14-38-01.bpo-37977.pML-UI.rst
D Misc/NEWS.d/next/Documentation/2019-09-05-14-47-51.bpo-36797.KN9Ga5.rst
D Misc/NEWS.d/next/Documentation/2019-09-07-15-55-46.bpo-26868.Raw0Gd.rst
D Misc/NEWS.d/next/IDLE/2019-08-21-16-02-49.bpo-37902._R_adE.rst
D Misc/NEWS.d/next/IDLE/2019-09-01-10-22-55.bpo-35771.tdbmbP.rst
D Misc/NEWS.d/next/IDLE/2019-09-05-23-12-13.bpo-38041.nxmGGK.rst
D Misc/NEWS.d/next/IDLE/2019-09-09-22-08-36.bpo-38077.Mzpfe2.rst
D Misc/NEWS.d/next/IDLE/2019-09-16-15-04-29.bpo-38183.eudCN1.rst
D Misc/NEWS.d/next/IDLE/2019-09-17-01-28-56.bpo-35379.yAECDr.rst
D Misc/NEWS.d/next/Library/2018-08-27-15-44-50.bpo-34519.cPlH1h.rst
D Misc/NEWS.d/next/Library/2018-11-21-18-05-50.bpo-20504.kG0ub5.rst
D Misc/NEWS.d/next/Library/2019-01-22-09-23-20.bpo-35168.UGv2yW.rst
D Misc/NEWS.d/next/Library/2019-03-09-16-04-12.bpo-36250.tSK4N1.rst
D Misc/NEWS.d/next/Library/2019-05-19-10-48-46.bpo-21315.PgXVqF.rst
D Misc/NEWS.d/next/Library/2019-05-22-04-52-35.bpo-25068.vR_rC-.rst
D Misc/NEWS.d/next/Library/2019-05-26-16-34-53.bpo-31163.21A802.rst
D Misc/NEWS.d/next/Library/2019-06-08-23-26-58.bpo-12144.Z7mz-q.rst
D Misc/NEWS.d/next/Library/2019-06-09-22-25-03.bpo-37206.2WBg4q.rst
D Misc/NEWS.d/next/Library/2019-06-12-08-56-22.bpo-21872.V9QGGN.rst
D Misc/NEWS.d/next/Library/2019-06-18-13-59-55.bpo-37305.fGzWlP.rst
D Misc/NEWS.d/next/Library/2019-06-22-22-00-35.bpo-37212.Zhv-tq.rst
D Misc/NEWS.d/next/Library/2019-07-04-13-00-20.bpo-37424.0i1MR-.rst
D Misc/NEWS.d/next/Library/2019-07-29-21-39-45.bpo-11953.4Hpwf9.rst
D Misc/NEWS.d/next/Library/2019-08-07-14-49-22.bpo-37785.y7OlT8.rst
D Misc/NEWS.d/next/Library/2019-08-19-10-31-41.bpo-37885.4Nc9sp.rst
D Misc/NEWS.d/next/Library/2019-08-27-01-03-26.bpo-22347._TRpYr.rst
D Misc/NEWS.d/next/Library/2019-08-28-21-40-12.bpo-37972.kP-n4L.rst
D Misc/NEWS.d/next/Library/2019-08-30-11-21-10.bpo-37140.cFAX-a.rst
D Misc/NEWS.d/next/Library/2019-08-31-01-52-59.bpo-34410.7KbWZQ.rst
D Misc/NEWS.d/next/Library/2019-09-02-13-37-27.bpo-38006.Y7vA0Q.rst
D Misc/NEWS.d/next/Library/2019-09-02-14-30-39.bpo-38010.JOnz9Z.rst
D Misc/NEWS.d/next/Library/2019-09-04-20-34-14.bpo-38026.0LLRX-.rst
D Misc/NEWS.d/next/Library/2019-09-06-17-40-34.bpo-37953.db5FQq.rst
D Misc/NEWS.d/next/Library/2019-09-08-11-36-50.bpo-38059.8SA6co.rst
D Misc/NEWS.d/next/Library/2019-09-09-14-39-47.bpo-38066.l9mWv-.rst
D Misc/NEWS.d/next/Library/2019-09-09-18-39-23.bpo-38037.B0UgFU.rst
D Misc/NEWS.d/next/Library/2019-09-10-10-59-50.bpo-37251.8zn2o3.rst
D Misc/NEWS.d/next/Library/2019-09-10-11-42-59.bpo-38086.w5TlG-.rst
D Misc/NEWS.d/next/Library/2019-09-11-11-44-16.bpo-37488.S8CJUL.rst
D Misc/NEWS.d/next/Library/2019-09-11-14-45-30.bpo-38093.yQ6k7y.rst
D Misc/NEWS.d/next/Library/2019-09-11-14-49-20.bpo-38110.A19Y-q.rst
D Misc/NEWS.d/next/Library/2019-09-11-16-54-57.bpo-38121.SrSDzB.rst
D Misc/NEWS.d/next/Library/2019-09-11-20-27-41.bpo-37405.MG5xiY.rst
D Misc/NEWS.d/next/Library/2019-09-12-10-47-34.bpo-38008.sH74Iy.rst
D Misc/NEWS.d/next/Library/2019-09-12-12-47-35.bpo-38132.KSFx1F.rst
D Misc/NEWS.d/next/Library/2019-09-12-13-18-55.bpo-38134.gXJTbP.rst
D Misc/NEWS.d/next/Library/2019-09-12-14-52-38.bpo-36991.1OcSm8.rst
D Misc/NEWS.d/next/Library/2019-09-13-08-55-43.bpo-38148.Lnww6D.rst
D Misc/NEWS.d/next/Library/2019-09-13-09-24-58.bpo-38115.BOO-Y1.rst
D Misc/NEWS.d/next/Library/2019-09-13-12-18-51.bpo-38153.nHAbuJ.rst
D Misc/NEWS.d/next/Library/2019-09-13-14-54-33.bpo-34706.HWVpOY.rst
D Misc/NEWS.d/next/Library/2019-09-14-10-34-00.bpo-33936.8wCI_n.rst
D Misc/NEWS.d/next/Library/2019-09-15-10-30-33.bpo-38175.61XlUv.rst
D Misc/NEWS.d/next/Library/2019-09-15-21-31-18.bpo-37828.gLLDX7.rst
D Misc/NEWS.d/next/Library/2019-09-16-09-54-42.bpo-38136.MdI-Zb.rst
D Misc/NEWS.d/next/Library/2019-09-16-19-12-57.bpo-38185.zYWppY.rst
D Misc/NEWS.d/next/Library/2019-09-17-12-28-27.bpo-38191.1TU0HV.rst
D Misc/NEWS.d/next/Library/2019-09-20-14-27-17.bpo-38237.xRUZbx.rst
D Misc/NEWS.d/next/Library/2019-09-22-13-05-36.bpo-38248.Yo3N_1.rst
D Misc/NEWS.d/next/Library/2019-09-25-21-37-02.bpo-38108.Jr9HU6.rst
D Misc/NEWS.d/next/Library/2019-09-27-15-24-45.bpo-38216.-7yvZR.rst
D Misc/NEWS.d/next/Library/2019-09-27-16-31-28.bpo-38161.zehai1.rst
D Misc/NEWS.d/next/Library/2019-09-28-20-16-40.bpo-38163.x51-vK.rst
D Misc/NEWS.d/next/Library/2019-09-29-13-50-24.bpo-38019.6MoOE3.rst
D Misc/NEWS.d/next/Library/2019-09-30-00-15-27.bpo-38242.uPIyAc.rst
D Misc/NEWS.d/next/Library/2019-09-30-22-06-33.bpo-38319.5QjiDa.rst
D Misc/NEWS.d/next/Security/2019-08-27-01-13-05.bpo-37764.qv67PQ.rst
D Misc/NEWS.d/next/Security/2019-09-23-21-02-46.bpo-38174.MeWuJd.rst
D Misc/NEWS.d/next/Security/2019-09-25-13-21-09.bpo-38243.1pfz24.rst
D Misc/NEWS.d/next/Tests/2018-09-07-01-18-27.bpo-34596.r2-EGd.rst
D Misc/NEWS.d/next/Tests/2019-05-28-15-41-34.bpo-36919.-vGt_m.rst
D Misc/NEWS.d/next/Tests/2019-06-03-20-47-10.bpo-34001.KvYx9z.rst
D Misc/NEWS.d/next/Tests/2019-06-12-14-30-29.bpo-37252.4o-uLs.rst
D Misc/NEWS.d/next/Tests/2019-08-16-16-15-14.bpo-37876.m3k1w3.rst
D Misc/NEWS.d/next/Tests/2019-08-20-19-24-19.bpo-37531.wRoXfU.rst
D Misc/NEWS.d/next/Tests/2019-09-11-17-22-32.bpo-38117.X7LgGY.rst
D Misc/NEWS.d/next/Tests/2019-09-24-12-24-05.bpo-38212.IWbhWz.rst
D Misc/NEWS.d/next/Tests/2019-09-24-12-30-55.bpo-37123.IoutBn.rst
D Misc/NEWS.d/next/Tests/2019-09-25-12-18-31.bpo-38270._x-9uH.rst
D Misc/NEWS.d/next/Tests/2019-09-25-13-11-29.bpo-38271.iHXNIg.rst
D Misc/NEWS.d/next/Tests/2019-09-25-14-40-57.bpo-38275.-kdveI.rst
D Misc/NEWS.d/next/Tests/2019-09-26-15-48-36.bpo-38239.MfoVzY.rst
D Misc/NEWS.d/next/Tools-Demos/2019-05-27-15-26-12.bpo-37064.k_SPW2.rst
D Misc/NEWS.d/next/Tools-Demos/2019-09-12-16-15-55.bpo-37803.chEizy.rst
D Misc/NEWS.d/next/Windows/2019-05-10-15-25-44.bpo-32587.-0g2O3.rst
D Misc/NEWS.d/next/Windows/2019-06-25-04-15-22.bpo-37380.tPxjuz.rst
D Misc/NEWS.d/next/Windows/2019-06-28-18-10-29.bpo-37445.LsdYO6.rst
D Misc/NEWS.d/next/Windows/2019-07-29-16-49-31.bpo-37702.Lj2f5e.rst
D Misc/NEWS.d/next/Windows/2019-08-30-15-15-22.bpo-37705.2o4NWW.rst
D Misc/NEWS.d/next/Windows/2019-09-03-11-47-37.bpo-38020.xFZ2j0.rst
D Misc/NEWS.d/next/Windows/2019-09-04-14-01-08.bpo-38030._USdtk.rst
D Misc/NEWS.d/next/Windows/2019-09-09-12-22-23.bpo-37283.8NvOkU.rst
D Misc/NEWS.d/next/Windows/2019-09-10-14-17-25.bpo-38088.FOvWSM.rst
D Misc/NEWS.d/next/Windows/2019-09-10-14-21-40.bpo-38087.--eIib.rst
D Misc/NEWS.d/next/Windows/2019-09-11-10-22-01.bpo-38081.8JhzjD.rst
D Misc/NEWS.d/next/Windows/2019-09-11-12-34-31.bpo-36634.xLaGgb.rst
D Misc/NEWS.d/next/Windows/2019-09-11-14-42-04.bpo-36634.8Un8ih.rst
D Misc/NEWS.d/next/Windows/2019-09-11-15-24-04.bpo-38114.cc0E5E.rst
D Misc/NEWS.d/next/Windows/2019-09-12-12-05-55.bpo-38133.yFeRGS.rst
D Misc/NEWS.d/next/Windows/2019-09-13-14-11-42.bpo-38092.x31ehI.rst
D Misc/NEWS.d/next/Windows/2019-09-16-14-07-11.bpo-38117.hJVf0C.rst
D Misc/NEWS.d/next/macOS/2019-09-10-14-24-35.bpo-38089.eedgyD.rst
D Misc/NEWS.d/next/macOS/2019-09-15-21-29-13.bpo-38117.ZLsoAZ.rst
M Include/patchlevel.h
M Lib/pydoc_data/topics.py
M README.rst

diff --git a/Include/patchlevel.h b/Include/patchlevel.h
index 58ee1692772d8..052cd45f47016 100644
--- a/Include/patchlevel.h
+++ b/Include/patchlevel.h
@@ -19,11 +19,11 @@
 #define PY_MAJOR_VERSION        3
 #define PY_MINOR_VERSION        8
 #define PY_MICRO_VERSION        0
-#define PY_RELEASE_LEVEL        PY_RELEASE_LEVEL_BETA
-#define PY_RELEASE_SERIAL       4
+#define PY_RELEASE_LEVEL        PY_RELEASE_LEVEL_GAMMA
+#define PY_RELEASE_SERIAL       1
=20
 /* Version as a string */
-#define PY_VERSION              "3.8.0b4+"
+#define PY_VERSION              "3.8.0rc1"
 /*--end constants--*/
=20
 /* Version as a single 4-byte hex number, e.g. 0x010502B2 =3D=3D 1.5.2b2.
diff --git a/Lib/pydoc_data/topics.py b/Lib/pydoc_data/topics.py
index 8992ea9b4ea76..7cb089f757181 100644
--- a/Lib/pydoc_data/topics.py
+++ b/Lib/pydoc_data/topics.py
@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Autogenerated by Sphinx on Thu Aug 29 23:57:58 2019
+# Autogenerated by Sphinx on Tue Oct  1 14:53:09 2019
 topics =3D {'assert': 'The "assert" statement\n'
            '**********************\n'
            '\n'
@@ -5375,30 +5375,34 @@
                   'the result      |\n'
                   '   |           | formatted with presentation type "\'e\'"=
 '
                   'and precision "p-1" |\n'
-                  '   |           | would have exponent "exp".  Then if "-4 =
<=3D '
-                  'exp < p", the   |\n'
-                  '   |           | number is formatted with presentation ty=
pe '
-                  '"\'f\'" and       |\n'
-                  '   |           | precision "p-1-exp".  Otherwise, the '
-                  'number is formatted   |\n'
-                  '   |           | with presentation type "\'e\'" and '
-                  'precision "p-1". In both  |\n'
-                  '   |           | cases insignificant trailing zeros are '
-                  'removed from the    |\n'
+                  '   |           | would have exponent "exp".  Then, if "m =
<=3D '
+                  'exp < p", where |\n'
+                  '   |           | "m" is -4 for floats and -6 for '
+                  '"Decimals", the number is  |\n'
+                  '   |           | formatted with presentation type "\'f\'"=
 '
+                  'and precision       |\n'
+                  '   |           | "p-1-exp".  Otherwise, the number is '
+                  'formatted with        |\n'
+                  '   |           | presentation type "\'e\'" and precision '
+                  '"p-1". In both cases |\n'
+                  '   |           | insignificant trailing zeros are removed=
 '
+                  'from the          |\n'
                   '   |           | significand, and the decimal point is al=
so '
                   'removed if      |\n'
                   '   |           | there are no remaining digits following '
-                  'it.  Positive and  |\n'
-                  '   |           | negative infinity, positive and negative=
 '
-                  'zero, and nans,   |\n'
-                  '   |           | are formatted as "inf", "-inf", "0", "-0=
" '
-                  'and "nan"        |\n'
-                  '   |           | respectively, regardless of the '
-                  'precision.  A precision of |\n'
-                  '   |           | "0" is treated as equivalent to a '
-                  'precision of "1". The    |\n'
-                  '   |           | default precision is '
-                  '"6".                                  |\n'
+                  'it, unless the     |\n'
+                  '   |           | "\'#\'" option is used.  Positive and '
+                  'negative infinity,     |\n'
+                  '   |           | positive and negative zero, and nans, ar=
e '
+                  'formatted as     |\n'
+                  '   |           | "inf", "-inf", "0", "-0" and "nan" '
+                  'respectively,           |\n'
+                  '   |           | regardless of the precision.  A precisio=
n '
+                  'of "0" is        |\n'
+                  '   |           | treated as equivalent to a precision of '
+                  '"1". The default   |\n'
+                  '   |           | precision is '
+                  '"6".                                          |\n'
                   '   '
                   '+-----------+--------------------------------------------=
----------------+\n'
                   '   | "\'G\'"     | General format. Same as "\'g\'" except=
 '
@@ -7048,6 +7052,9 @@
                      '| Operator                                        | '
                      'Description                           |\n'
                      '|=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D|=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D|\n'
+                     '| ":=3D"                                            | '
+                     'Assignment expression                 |\n'
+                     '+-------------------------------------------------+---=
------------------------------------+\n'
                      '| "lambda"                                        | '
                      'Lambda expression                     |\n'
                      '+-------------------------------------------------+---=
------------------------------------+\n'
@@ -7104,10 +7111,10 @@
                      '| "x(arguments...)", "x.attribute"                | '
                      'attribute reference                   |\n'
                      '+-------------------------------------------------+---=
------------------------------------+\n'
-                     '| "(expressions...)", "[expressions...]", "{key:  | '
-                     'Binding or tuple display, list        |\n'
-                     '| value...}", "{expressions...}"                  | '
-                     'display, dictionary display, set      |\n'
+                     '| "(expressions...)",  "[expressions...]", "{key: | '
+                     'Binding or parenthesized expression,  |\n'
+                     '| value...}", "{expressions...}"                  | li=
st '
+                     'display, dictionary display, set |\n'
                      '|                                                 | '
                      'display                               |\n'
                      '+-------------------------------------------------+---=
------------------------------------+\n'
@@ -7475,7 +7482,11 @@
                    '   estimated length for the object (which may be greater=
 '
                    'or less than\n'
                    '   the actual length). The length must be an integer ">=
=3D" '
-                   '0. This\n'
+                   '0. The\n'
+                   '   return value may also be "NotImplemented", which is '
+                   'treated the\n'
+                   '   same as if the "__length_hint__" method didn=E2=80=99=
t exist at '
+                   'all. This\n'
                    '   method is purely an optimization and is never require=
d '
                    'for\n'
                    '   correctness.\n'
@@ -9199,7 +9210,11 @@
                  '   estimated length for the object (which may be greater o=
r '
                  'less than\n'
                  '   the actual length). The length must be an integer ">=3D=
" 0. '
-                 'This\n'
+                 'The\n'
+                 '   return value may also be "NotImplemented", which is '
+                 'treated the\n'
+                 '   same as if the "__length_hint__" method didn=E2=80=99t =
exist at '
+                 'all. This\n'
                  '   method is purely an optimization and is never required '
                  'for\n'
                  '   correctness.\n'
@@ -12149,6 +12164,11 @@
                  'therefore,\n'
                  '   custom mapping types should support too):\n'
                  '\n'
+                 '   list(d)\n'
+                 '\n'
+                 '      Return a list of all the keys used in the dictionary=
 '
+                 '*d*.\n'
+                 '\n'
                  '   len(d)\n'
                  '\n'
                  '      Return the number of items in the dictionary *d*.\n'
@@ -12317,11 +12337,21 @@
                  'the\n'
                  '      documentation of view objects.\n'
                  '\n'
+                 '      An equality comparison between one "dict.values()" '
+                 'view and\n'
+                 '      another will always return "False". This also applie=
s '
+                 'when\n'
+                 '      comparing "dict.values()" to itself:\n'
+                 '\n'
+                 "         >>> d =3D {'a': 1}\n"
+                 '         >>> d.values() =3D=3D d.values()\n'
+                 '         False\n'
+                 '\n'
                  '   Dictionaries compare equal if and only if they have the=
 '
                  'same "(key,\n'
-                 '   value)" pairs. Order comparisons (=E2=80=98<=E2=80=99, =
=E2=80=98<=3D=E2=80=99, =E2=80=98>=3D=E2=80=99, =E2=80=98>=E2=80=99) '
-                 'raise\n'
-                 '   "TypeError".\n'
+                 '   value)" pairs (regardless of ordering). Order compariso=
ns '
+                 '(=E2=80=98<=E2=80=99,\n'
+                 '   =E2=80=98<=3D=E2=80=99, =E2=80=98>=3D=E2=80=99, =E2=80=
=98>=E2=80=99) raise "TypeError".\n'
                  '\n'
                  '   Dictionaries preserve insertion order.  Note that '
                  'updating a key\n'
diff --git a/Misc/NEWS.d/3.8.0rc1.rst b/Misc/NEWS.d/3.8.0rc1.rst
new file mode 100644
index 0000000000000..f7c3c9984c09e
--- /dev/null
+++ b/Misc/NEWS.d/3.8.0rc1.rst
@@ -0,0 +1,1349 @@
+.. bpo: 38243
+.. date: 2019-09-25-13-21-09
+.. nonce: 1pfz24
+.. release date: 2019-10-01
+.. section: Security
+
+Escape the server title of :class:`xmlrpc.server.DocXMLRPCServer` when
+rendering the document page as HTML. (Contributed by Dong-hee Na in
+:issue:`38243`.)
+
+..
+
+.. bpo: 38174
+.. date: 2019-09-23-21-02-46
+.. nonce: MeWuJd
+.. section: Security
+
+Update vendorized expat library version to 2.2.8, which resolves
+CVE-2019-15903.
+
+..
+
+.. bpo: 37764
+.. date: 2019-08-27-01-13-05
+.. nonce: qv67PQ
+.. section: Security
+
+Fixes email._header_value_parser.get_unstructured going into an infinite
+loop for a specific case in which the email header does not have trailing
+whitespace, and the case in which it contains an invalid encoded word. Patch
+by Ashwin Ramaswami.
+
+..
+
+.. bpo: 38006
+.. date: 2019-09-30-09-33-21
+.. nonce: UYlJum
+.. section: Core and Builtins
+
+Fix a bug due to the interaction of weakrefs and the cyclic garbage
+collector. We must clear any weakrefs in garbage in order to prevent their
+callbacks from executing and causing a crash.
+
+..
+
+.. bpo: 38317
+.. date: 2019-09-30-00-56-21
+.. nonce: pmqlIQ
+.. section: Core and Builtins
+
+Fix warnings options priority: ``PyConfig.warnoptions`` has the highest
+priority, as stated in the :pep:`587`.
+
+..
+
+.. bpo: 36871
+.. date: 2019-09-24-18-45-46
+.. nonce: p47knk
+.. section: Core and Builtins
+
+Improve error handling for the assert_has_calls and assert_has_awaits
+methods of mocks. Fixed a bug where any errors encountered while binding the
+expected calls to the mock's spec were silently swallowed, leading to
+misleading error output.
+
+..
+
+.. bpo: 38236
+.. date: 2019-09-20-19-06-23
+.. nonce: eQ0Tmj
+.. section: Core and Builtins
+
+Python now dumps path configuration if it fails to import the Python codecs
+of the filesystem and stdio encodings.
+
+..
+
+.. bpo: 38013
+.. date: 2019-09-12-19-50-01
+.. nonce: I7btD0
+.. section: Core and Builtins
+
+Allow to call ``async_generator_athrow().throw(...)`` even for non-started
+async generator helper. It fixes annoying warning at the end of
+:func:`asyncio.run` call.
+
+..
+
+.. bpo: 38124
+.. date: 2019-09-12-00-14-01
+.. nonce: n6E0H7
+.. section: Core and Builtins
+
+Fix an off-by-one error in PyState_AddModule that could cause out-of-bounds
+memory access.
+
+..
+
+.. bpo: 38005
+.. date: 2019-09-02-20-00-31
+.. nonce: e7VsTA
+.. section: Core and Builtins
+
+Fixed comparing and creating of InterpreterID and ChannelID.
+
+..
+
+.. bpo: 37994
+.. date: 2019-08-31-11-13-25
+.. nonce: Rj6S4j
+.. section: Core and Builtins
+
+Fixed silencing arbitrary errors if an attribute lookup fails in several
+sites. Only AttributeError should be silenced.
+
+..
+
+.. bpo: 37990
+.. date: 2019-08-31-09-22-33
+.. nonce: WDY2f-
+.. section: Core and Builtins
+
+Fix elapsed time in gc stats was not printed correctly.  This bug was a
+regression in 3.8b4.
+
+..
+
+.. bpo: 37966
+.. date: 2019-08-27-21-21-36
+.. nonce: 5OBLez
+.. section: Core and Builtins
+
+The implementation of :func:`~unicodedata.is_normalized` has been greatly
+sped up on strings that aren't normalized, by implementing the full
+normalization-quick-check algorithm from the Unicode standard.
+
+..
+
+.. bpo: 20490
+.. date: 2019-08-15-12-48-36
+.. nonce: -hXeEn
+.. section: Core and Builtins
+
+Improve import error message for partially initialized module on circular
+``from`` imports - by Anthony Sottile.
+
+..
+
+.. bpo: 37409
+.. date: 2019-08-06-23-39-05
+.. nonce: 1qwzn2
+.. section: Core and Builtins
+
+Ensure explicit relative imports from interactive sessions and scripts
+(having no parent package) always raise ImportError, rather than treating
+the current module as the package. Patch by Ben Lewis.
+
+..
+
+.. bpo: 37619
+.. date: 2019-07-18-11-50-49
+.. nonce: X6Lulo
+.. section: Core and Builtins
+
+When adding a wrapper descriptor from one class to a different class (for
+example, setting ``__add__ =3D str.__add__`` on an ``int`` subclass), an
+exception is correctly raised when the operator is called.
+
+..
+
+.. bpo: 30773
+.. date: 2018-06-07-01-01-20
+.. nonce: C31rVE
+.. section: Core and Builtins
+
+Prohibit parallel running of aclose() / asend() / athrow(). Fix ag_running
+to reflect the actual running status of the AG.
+
+..
+
+.. bpo: 38319
+.. date: 2019-09-30-22-06-33
+.. nonce: 5QjiDa
+.. section: Library
+
+sendfile() used in socket and shutil modules was raising OverflowError for
+files >=3D 2GiB on 32-bit architectures.  (patch by Giampaolo Rodola)
+
+..
+
+.. bpo: 38242
+.. date: 2019-09-30-00-15-27
+.. nonce: uPIyAc
+.. section: Library
+
+Revert the new asyncio Streams API
+
+..
+
+.. bpo: 38019
+.. date: 2019-09-29-13-50-24
+.. nonce: 6MoOE3
+.. section: Library
+
+Correctly handle pause/resume reading of closed asyncio unix pipe.
+
+..
+
+.. bpo: 38163
+.. date: 2019-09-28-20-16-40
+.. nonce: x51-vK
+.. section: Library
+
+Child mocks will now detect their type as either synchronous or
+asynchronous, asynchronous child mocks will be AsyncMocks and synchronous
+child mocks will be either MagicMock or Mock (depending on their parent
+type).
+
+..
+
+.. bpo: 38161
+.. date: 2019-09-27-16-31-28
+.. nonce: zehai1
+.. section: Library
+
+Removes _AwaitEvent from AsyncMock.
+
+..
+
+.. bpo: 38216
+.. date: 2019-09-27-15-24-45
+.. nonce: -7yvZR
+.. section: Library
+
+Allow the rare code that wants to send invalid http requests from the
+`http.client` library a way to do so.  The fixes for bpo-30458 led to
+breakage for some projects that were relying on this ability to test their
+own behavior in the face of bad requests.
+
+..
+
+.. bpo: 38108
+.. date: 2019-09-25-21-37-02
+.. nonce: Jr9HU6
+.. section: Library
+
+Any synchronous magic methods on an AsyncMock now return a MagicMock. Any
+asynchronous magic methods on a MagicMock now return an AsyncMock.
+
+..
+
+.. bpo: 38248
+.. date: 2019-09-22-13-05-36
+.. nonce: Yo3N_1
+.. section: Library
+
+asyncio: Fix inconsistent immediate Task cancellation
+
+..
+
+.. bpo: 38237
+.. date: 2019-09-20-14-27-17
+.. nonce: xRUZbx
+.. section: Library
+
+The arguments for the builtin pow function are more descriptive. They can
+now also be passed in as keywords.
+
+..
+
+.. bpo: 38191
+.. date: 2019-09-17-12-28-27
+.. nonce: 1TU0HV
+.. section: Library
+
+Constructors of :class:`~typing.NamedTuple` and :class:`~typing.TypedDict`
+types now accept arbitrary keyword argument names, including "cls", "self",
+"typename", "_typename", "fields" and "_fields".  Passing positional
+arguments by keyword is deprecated.
+
+..
+
+.. bpo: 38185
+.. date: 2019-09-16-19-12-57
+.. nonce: zYWppY
+.. section: Library
+
+Fixed case-insensitive string comparison in :class:`sqlite3.Row` indexing.
+
+..
+
+.. bpo: 38136
+.. date: 2019-09-16-09-54-42
+.. nonce: MdI-Zb
+.. section: Library
+
+Changes AsyncMock call count and await count to be two different counters.
+Now await count only counts when a coroutine has been awaited, not when it
+has been called, and vice-versa. Update the documentation around this.
+
+..
+
+.. bpo: 37828
+.. date: 2019-09-15-21-31-18
+.. nonce: gLLDX7
+.. section: Library
+
+Fix default mock name in :meth:`unittest.mock.Mock.assert_called`
+exceptions. Patch by Abraham Toriz Cruz.
+
+..
+
+.. bpo: 38175
+.. date: 2019-09-15-10-30-33
+.. nonce: 61XlUv
+.. section: Library
+
+Fix a memory leak in comparison of :class:`sqlite3.Row` objects.
+
+..
+
+.. bpo: 33936
+.. date: 2019-09-14-10-34-00
+.. nonce: 8wCI_n
+.. section: Library
+
+_hashlib no longer calls obsolete OpenSSL initialization function with
+OpenSSL 1.1.0+.
+
+..
+
+.. bpo: 34706
+.. date: 2019-09-13-14-54-33
+.. nonce: HWVpOY
+.. section: Library
+
+Preserve subclassing in inspect.Signature.from_callable.
+
+..
+
+.. bpo: 38153
+.. date: 2019-09-13-12-18-51
+.. nonce: nHAbuJ
+.. section: Library
+
+Names of hashing algorithms frome OpenSSL are now normalized to follow
+Python's naming conventions. For example OpenSSL uses sha3-512 instead of
+sha3_512 or blake2b512 instead of blake2b.
+
+..
+
+.. bpo: 38115
+.. date: 2019-09-13-09-24-58
+.. nonce: BOO-Y1
+.. section: Library
+
+Fix a bug in dis.findlinestarts() where it would return invalid bytecode
+offsets. Document that a code object's co_lnotab can contain invalid
+bytecode offsets.
+
+..
+
+.. bpo: 38148
+.. date: 2019-09-13-08-55-43
+.. nonce: Lnww6D
+.. section: Library
+
+Add slots to :mod:`asyncio` transport classes, which can reduce memory
+usage.
+
+..
+
+.. bpo: 36991
+.. date: 2019-09-12-14-52-38
+.. nonce: 1OcSm8
+.. section: Library
+
+Fixes a potential incorrect AttributeError exception escaping
+ZipFile.extract() in some unsupported input error situations.
+
+..
+
+.. bpo: 38134
+.. date: 2019-09-12-13-18-55
+.. nonce: gXJTbP
+.. section: Library
+
+Remove obsolete copy of PBKDF2_HMAC_fast. All supported OpenSSL versions
+contain a fast implementation.
+
+..
+
+.. bpo: 38132
+.. date: 2019-09-12-12-47-35
+.. nonce: KSFx1F
+.. section: Library
+
+The OpenSSL hashlib wrapper uses a simpler implementation. Several Macros
+and pointless caches are gone. The hash name now comes from OpenSSL's EVP.
+The algorithm name stays the same, except it is now always lower case.
+
+..
+
+.. bpo: 38008
+.. date: 2019-09-12-10-47-34
+.. nonce: sH74Iy
+.. section: Library
+
+Fix parent class check in protocols to correctly identify the module that
+provides a builtin protocol, instead of assuming they all come from the
+:mod:`collections.abc` module
+
+..
+
+.. bpo: 37405
+.. date: 2019-09-11-20-27-41
+.. nonce: MG5xiY
+.. section: Library
+
+Fixed regression bug for socket.getsockname() for non-CAN_ISOTP AF_CAN
+address family sockets by returning a 1-tuple instead of string.
+
+..
+
+.. bpo: 38121
+.. date: 2019-09-11-16-54-57
+.. nonce: SrSDzB
+.. section: Library
+
+Update parameter names on functions in importlib.metadata matching the
+changes in the 0.22 release of importlib_metadata.
+
+..
+
+.. bpo: 38110
+.. date: 2019-09-11-14-49-20
+.. nonce: A19Y-q
+.. section: Library
+
+The os.closewalk() implementation now uses the libc fdwalk() API on
+platforms where it is available.
+
+..
+
+.. bpo: 38093
+.. date: 2019-09-11-14-45-30
+.. nonce: yQ6k7y
+.. section: Library
+
+Fixes AsyncMock so it doesn't crash when used with AsyncContextManagers or
+AsyncIterators.
+
+..
+
+.. bpo: 37488
+.. date: 2019-09-11-11-44-16
+.. nonce: S8CJUL
+.. section: Library
+
+Add warning to :meth:`datetime.utctimetuple`,  :meth:`datetime.utcnow` and
+:meth:`datetime.utcfromtimestamp` .
+
+..
+
+.. bpo: 38086
+.. date: 2019-09-10-11-42-59
+.. nonce: w5TlG-
+.. section: Library
+
+Update importlib.metadata with changes from `importlib_metadata 0.21
+<https://gitlab.com/python-devs/importlib_metadata/blob/0.21/importlib_metad=
ata/docs/changelog.rst>`_.
+
+..
+
+.. bpo: 37251
+.. date: 2019-09-10-10-59-50
+.. nonce: 8zn2o3
+.. section: Library
+
+Remove `__code__` check in AsyncMock that incorrectly evaluated function
+specs as async objects but failed to evaluate classes with `__await__` but
+no `__code__` attribute defined as async objects.
+
+..
+
+.. bpo: 38037
+.. date: 2019-09-09-18-39-23
+.. nonce: B0UgFU
+.. section: Library
+
+Fix reference counters in the :mod:`signal` module.
+
+..
+
+.. bpo: 38066
+.. date: 2019-09-09-14-39-47
+.. nonce: l9mWv-
+.. section: Library
+
+Hide internal asyncio.Stream methods: feed_eof(), feed_data(),
+set_exception() and set_transport().
+
+..
+
+.. bpo: 38059
+.. date: 2019-09-08-11-36-50
+.. nonce: 8SA6co
+.. section: Library
+
+inspect.py now uses sys.exit() instead of exit()
+
+..
+
+.. bpo: 37953
+.. date: 2019-09-06-17-40-34
+.. nonce: db5FQq
+.. section: Library
+
+In :mod:`typing`, improved the ``__hash__`` and ``__eq__`` methods for
+:class:`ForwardReferences`.
+
+..
+
+.. bpo: 38026
+.. date: 2019-09-04-20-34-14
+.. nonce: 0LLRX-
+.. section: Library
+
+Fixed :func:`inspect.getattr_static` used ``isinstance`` while it should
+avoid dynamic lookup.
+
+..
+
+.. bpo: 38010
+.. date: 2019-09-02-14-30-39
+.. nonce: JOnz9Z
+.. section: Library
+
+In ``importlib.metadata`` sync with ``importlib_metadata`` 0.20, clarifying
+behavior of ``files()`` and fixing issue where only one requirement was
+returned for ``requires()`` on ``dist-info`` packages.
+
+..
+
+.. bpo: 38006
+.. date: 2019-09-02-13-37-27
+.. nonce: Y7vA0Q
+.. section: Library
+
+weakref.WeakValueDictionary defines a local remove() function used as
+callback for weak references. This function was created with a closure.
+Modify the implementation to avoid the closure.
+
+..
+
+.. bpo: 34410
+.. date: 2019-08-31-01-52-59
+.. nonce: 7KbWZQ
+.. section: Library
+
+Fixed a crash in the :func:`tee` iterator when re-enter it. RuntimeError is
+now raised in this case.
+
+..
+
+.. bpo: 37140
+.. date: 2019-08-30-11-21-10
+.. nonce: cFAX-a
+.. section: Library
+
+Fix a ctypes regression of Python 3.8. When a ctypes.Structure is passed by
+copy to a function, ctypes internals created a temporary object which had
+the side effect of calling the structure finalizer (__del__) twice. The
+Python semantics requires a finalizer to be called exactly once. Fix ctypes
+internals to no longer call the finalizer twice.
+
+..
+
+.. bpo: 37972
+.. date: 2019-08-28-21-40-12
+.. nonce: kP-n4L
+.. section: Library
+
+Subscripts to the `unittest.mock.call` objects now receive the same chaining
+mechanism as any other custom attributes, so that the following usage no
+longer raises a `TypeError`:
+
+call().foo().__getitem__('bar')
+
+Patch by blhsing
+
+..
+
+.. bpo: 22347
+.. date: 2019-08-27-01-03-26
+.. nonce: _TRpYr
+.. section: Library
+
+Update mimetypes.guess_type to allow proper parsing of URLs with only a host
+name. Patch by Dong-hee Na.
+
+..
+
+.. bpo: 37885
+.. date: 2019-08-19-10-31-41
+.. nonce: 4Nc9sp
+.. section: Library
+
+venv: Don't generate unset variable warning on deactivate.
+
+..
+
+.. bpo: 37785
+.. date: 2019-08-07-14-49-22
+.. nonce: y7OlT8
+.. section: Library
+
+Fix xgettext warnings in :mod:`argparse`.
+
+..
+
+.. bpo: 11953
+.. date: 2019-07-29-21-39-45
+.. nonce: 4Hpwf9
+.. section: Library
+
+Completing WSA* error codes in :mod:`socket`.
+
+..
+
+.. bpo: 37424
+.. date: 2019-07-04-13-00-20
+.. nonce: 0i1MR-
+.. section: Library
+
+Fixes a possible hang when using a timeout on `subprocess.run()` while
+capturing output.  If the child process spawned its own children or
+otherwise connected its stdout or stderr handles with another process, we
+could hang after the timeout was reached and our child was killed when
+attempting to read final output from the pipes.
+
+..
+
+.. bpo: 37212
+.. date: 2019-06-22-22-00-35
+.. nonce: Zhv-tq
+.. section: Library
+
+:func:`unittest.mock.call` now preserves the order of keyword arguments in
+repr output. Patch by Karthikeyan Singaravelan.
+
+..
+
+.. bpo: 37305
+.. date: 2019-06-18-13-59-55
+.. nonce: fGzWlP
+.. section: Library
+
+Add .webmanifest -> application/manifest+json to list of recognized file
+types and content type headers
+
+..
+
+.. bpo: 21872
+.. date: 2019-06-12-08-56-22
+.. nonce: V9QGGN
+.. section: Library
+
+Fix :mod:`lzma`: module decompresses data incompletely. When decompressing a
+FORMAT_ALONE format file, and it doesn't have the end marker, sometimes the
+last one to dozens bytes can't be output. Patch by Ma Lin.
+
+..
+
+.. bpo: 37206
+.. date: 2019-06-09-22-25-03
+.. nonce: 2WBg4q
+.. section: Library
+
+Default values which cannot be represented as Python objects no longer
+improperly represented as ``None`` in function signatures.
+
+..
+
+.. bpo: 12144
+.. date: 2019-06-08-23-26-58
+.. nonce: Z7mz-q
+.. section: Library
+
+Ensure cookies with ``expires`` attribute are handled in
+:meth:`CookieJar.make_cookies`.
+
+..
+
+.. bpo: 31163
+.. date: 2019-05-26-16-34-53
+.. nonce: 21A802
+.. section: Library
+
+pathlib.Path instance's rename and replace methods now return the new Path
+instance.
+
+..
+
+.. bpo: 25068
+.. date: 2019-05-22-04-52-35
+.. nonce: vR_rC-
+.. section: Library
+
+:class:`urllib.request.ProxyHandler` now lowercases the keys of the passed
+dictionary.
+
+..
+
+.. bpo: 21315
+.. date: 2019-05-19-10-48-46
+.. nonce: PgXVqF
+.. section: Library
+
+Email headers containing RFC2047 encoded words are parsed despite the
+missing whitespace, and a defect registered. Also missing trailing
+whitespace after encoded words is now registered as a defect.
+
+..
+
+.. bpo: 36250
+.. date: 2019-03-09-16-04-12
+.. nonce: tSK4N1
+.. section: Library
+
+Ignore ``ValueError`` from ``signal`` with ``interaction`` in non-main
+thread.
+
+..
+
+.. bpo: 35168
+.. date: 2019-01-22-09-23-20
+.. nonce: UGv2yW
+.. section: Library
+
+:attr:`shlex.shlex.punctuation_chars` is now a read-only property.
+
+..
+
+.. bpo: 20504
+.. date: 2018-11-21-18-05-50
+.. nonce: kG0ub5
+.. section: Library
+
+Fixes a bug in :mod:`cgi` module when a multipart/form-data request has no
+`Content-Length` header.
+
+..
+
+.. bpo: 34519
+.. date: 2018-08-27-15-44-50
+.. nonce: cPlH1h
+.. section: Library
+
+Add additional aliases for HP Roman 8. Patch by Michael Osipov.
+
+..
+
+.. bpo: 26868
+.. date: 2019-09-07-15-55-46
+.. nonce: Raw0Gd
+.. section: Documentation
+
+Fix example usage of :c:func:`PyModule_AddObject` to properly handle errors.
+
+..
+
+.. bpo: 36797
+.. date: 2019-09-05-14-47-51
+.. nonce: KN9Ga5
+.. section: Documentation
+
+Fix a dead link in the distutils API Reference.
+
+..
+
+.. bpo: 37977
+.. date: 2019-08-29-14-38-01
+.. nonce: pML-UI
+.. section: Documentation
+
+Warn more strongly and clearly about pickle insecurity
+
+..
+
+.. bpo: 37937
+.. date: 2019-08-24-12-59-06
+.. nonce: F7fHbt
+.. section: Documentation
+
+Mention ``frame.f_trace`` in :func:`sys.settrace` docs.
+
+..
+
+.. bpo: 36260
+.. date: 2019-06-04-09-29-00
+.. nonce: WrGuc-
+.. section: Documentation
+
+Add decompression pitfalls to zipfile module documentation.
+
+..
+
+.. bpo: 36960
+.. date: 2019-05-18-16-25-44
+.. nonce: xEKHXj
+.. section: Documentation
+
+Restructured the :mod:`datetime` docs in the interest of making them more
+user-friendly and improving readability. Patch by Brad Solomon.
+
+..
+
+.. bpo: 23460
+.. date: 2019-02-14-07-12-48
+.. nonce: Iqiqtm
+.. section: Documentation
+
+The documentation for decimal string formatting using the `:g` specifier has
+been updated to reflect the correct exponential notation cutoff point.
+Original patch contributed by Tuomas Suutari.
+
+..
+
+.. bpo: 35803
+.. date: 2019-01-21-14-30-59
+.. nonce: yae6Lq
+.. section: Documentation
+
+Document and test that ``tempfile`` functions may accept a :term:`path-like
+object` for the ``dir`` argument.  Patch by Anthony Sottile.
+
+..
+
+.. bpo: 33944
+.. date: 2018-10-26-18-10-29
+.. nonce: V1YeOA
+.. section: Documentation
+
+Added a note about the intended use of code in .pth files.
+
+..
+
+.. bpo: 34293
+.. date: 2018-07-31-15-38-26
+.. nonce: yHupAL
+.. section: Documentation
+
+Fix the Doc/Makefile regarding PAPER environment variable and PDF builds
+
+..
+
+.. bpo: 38239
+.. date: 2019-09-26-15-48-36
+.. nonce: MfoVzY
+.. section: Tests
+
+Fix test_gdb for Link Time Optimization (LTO) builds.
+
+..
+
+.. bpo: 38275
+.. date: 2019-09-25-14-40-57
+.. nonce: -kdveI
+.. section: Tests
+
+test_ssl now handles disabled TLS/SSL versions better. OpenSSL's crypto
+policy and run-time settings are recognized and tests for disabled versions
+are skipped. Tests also accept more TLS minimum_versions for platforms that
+override OpenSSL's default with strict settings.
+
+..
+
+.. bpo: 38271
+.. date: 2019-09-25-13-11-29
+.. nonce: iHXNIg
+.. section: Tests
+
+The private keys for test_ssl were encrypted with 3DES in traditional PKCS#5
+format. 3DES and the digest algorithm of PKCS#5 are blocked by some strict
+crypto policies. Use PKCS#8 format with AES256 encryption instead.
+
+..
+
+.. bpo: 38270
+.. date: 2019-09-25-12-18-31
+.. nonce: _x-9uH
+.. section: Tests
+
+test.support now has a helper function to check for availibility of a hash
+digest function. Several tests are refactored avoid MD5 and use SHA256
+instead. Other tests are marked to use MD5 and skipped when MD5 is disabled.
+
+..
+
+.. bpo: 37123
+.. date: 2019-09-24-12-30-55
+.. nonce: IoutBn
+.. section: Tests
+
+Multiprocessing test test_mymanager() now also expects -SIGTERM, not only
+exitcode 0. BaseManager._finalize_manager() sends SIGTERM to the manager
+process if it takes longer than 1 second to stop, which happens on slow
+buildbots.
+
+..
+
+.. bpo: 38212
+.. date: 2019-09-24-12-24-05
+.. nonce: IWbhWz
+.. section: Tests
+
+Multiprocessing tests: increase test_queue_feeder_donot_stop_onexc() timeout
+from 1 to 60 seconds.
+
+..
+
+.. bpo: 38117
+.. date: 2019-09-11-17-22-32
+.. nonce: X7LgGY
+.. section: Tests
+
+Test with OpenSSL 1.1.1d
+
+..
+
+.. bpo: 37531
+.. date: 2019-08-20-19-24-19
+.. nonce: wRoXfU
+.. section: Tests
+
+Enhance regrtest multiprocess timeout: write a message when killing a worker
+process, catch popen.kill() and popen.wait() exceptions, put a timeout on
+the second call to popen.communicate().
+
+..
+
+.. bpo: 37876
+.. date: 2019-08-16-16-15-14
+.. nonce: m3k1w3
+.. section: Tests
+
+Add tests for ROT-13 codec.
+
+..
+
+.. bpo: 37252
+.. date: 2019-06-12-14-30-29
+.. nonce: 4o-uLs
+.. section: Tests
+
+Fix assertions in ``test_close`` and ``test_events_mask_overflow`` devpoll
+tests.
+
+..
+
+.. bpo: 34001
+.. date: 2019-06-03-20-47-10
+.. nonce: KvYx9z
+.. section: Tests
+
+Make test_ssl pass with LibreSSL. LibreSSL handles minimum and maximum TLS
+version differently than OpenSSL.
+
+..
+
+.. bpo: 36919
+.. date: 2019-05-28-15-41-34
+.. nonce: -vGt_m
+.. section: Tests
+
+Make ``test_source_encoding.test_issue2301`` implementation independent. The
+test will work now for both CPython and IronPython.
+
+..
+
+.. bpo: 34596
+.. date: 2018-09-07-01-18-27
+.. nonce: r2-EGd
+.. section: Tests
+
+Fallback to a default reason when :func:`unittest.skip` is uncalled. Patch
+by Naitree Zhu.
+
+..
+
+.. bpo: 38301
+.. date: 2019-09-28-02-37-11
+.. nonce: 123456
+.. section: Build
+
+In Solaris family, we must be sure to use ``-D_REENTRANT``. Patch by Jes=C3=
=BAs
+Cea Avi=C3=B3n.
+
+..
+
+.. bpo: 36210
+.. date: 2019-09-24-22-47-47
+.. nonce: EmL9X1
+.. section: Build
+
+Update optional extension module detection for AIX. ossaudiodev and spwd are
+not applicable for AIX, and are no longer reported as missing. 3rd-party
+packaging of ncurses (with ASIS support) conflicts with officially supported
+AIX curses library, so configure AIX to use libcurses.a. However, skip
+trying to build _curses_panel.
+
+patch by M Felt
+
+..
+
+.. bpo: 36002
+.. date: 2019-09-13-14-12-36
+.. nonce: Bcl4oe
+.. section: Build
+
+Locate ``llvm-profdata`` and ``llvm-ar`` binaries using ``AC_PATH_TOOL``
+rather than ``AC_PATH_TARGET_TOOL``.
+
+..
+
+.. bpo: 37936
+.. date: 2019-09-10-00-54-48
+.. nonce: E7XEwu
+.. section: Build
+
+The :file:`.gitignore` file systematically keeps "rooted", with a
+non-trailing slash, all the rules that are meant to apply to files in a
+specific place in the repo.  Previously, when the intended file to ignore
+happened to be at the root of the repo, we'd most often accidentally also
+ignore files and directories with the same name anywhere in the tree.
+
+..
+
+.. bpo: 37936
+.. date: 2019-08-24-00-29-40
+.. nonce: QrORqA
+.. section: Build
+
+The :file:`.gitignore` file no longer applies to any files that are in fact
+tracked in the Git repository.  Patch by Greg Price.
+
+..
+
+.. bpo: 38117
+.. date: 2019-09-16-14-07-11
+.. nonce: hJVf0C
+.. section: Windows
+
+Update bundled OpenSSL to 1.1.1d
+
+..
+
+.. bpo: 38092
+.. date: 2019-09-13-14-11-42
+.. nonce: x31ehI
+.. section: Windows
+
+Reduce overhead when using multiprocessing in a Windows virtual environment.
+
+..
+
+.. bpo: 38133
+.. date: 2019-09-12-12-05-55
+.. nonce: yFeRGS
+.. section: Windows
+
+Allow py.exe launcher to locate installations from the Microsoft Store and
+improve display of active virtual environments.
+
+..
+
+.. bpo: 38114
+.. date: 2019-09-11-15-24-04
+.. nonce: cc0E5E
+.. section: Windows
+
+The ``pip.ini`` is no longer included in the Nuget package.
+
+..
+
+.. bpo: 36634
+.. date: 2019-09-11-14-42-04
+.. nonce: 8Un8ih
+.. section: Windows
+
+:func:`os.cpu_count` now returns active processors rather than maximum
+processors.
+
+..
+
+.. bpo: 36634
+.. date: 2019-09-11-12-34-31
+.. nonce: xLaGgb
+.. section: Windows
+
+venv activate.bat now works when the existing variables contain double quote
+characters.
+
+..
+
+.. bpo: 38081
+.. date: 2019-09-11-10-22-01
+.. nonce: 8JhzjD
+.. section: Windows
+
+Prevent error calling :func:`os.path.realpath` on ``'NUL'``.
+
+..
+
+.. bpo: 38087
+.. date: 2019-09-10-14-21-40
+.. nonce: --eIib
+.. section: Windows
+
+Fix case sensitivity in test_pathlib and test_ntpath.
+
+..
+
+.. bpo: 38088
+.. date: 2019-09-10-14-17-25
+.. nonce: FOvWSM
+.. section: Windows
+
+Fixes distutils not finding vcruntime140.dll with only the v142 toolset
+installed.
+
+..
+
+.. bpo: 37283
+.. date: 2019-09-09-12-22-23
+.. nonce: 8NvOkU
+.. section: Windows
+
+Ensure command-line and unattend.xml setting override previously detected
+states in Windows installer.
+
+..
+
+.. bpo: 38030
+.. date: 2019-09-04-14-01-08
+.. nonce: _USdtk
+.. section: Windows
+
+Fixes :func:`os.stat` failing for block devices on Windows
+
+..
+
+.. bpo: 38020
+.. date: 2019-09-03-11-47-37
+.. nonce: xFZ2j0
+.. section: Windows
+
+Fixes potential crash when calling :func:`os.readlink` (or indirectly
+through :func:`~os.path.realpath`) on a file that is not a supported link.
+
+..
+
+.. bpo: 37705
+.. date: 2019-08-30-15-15-22
+.. nonce: 2o4NWW
+.. section: Windows
+
+Improve the implementation of ``winerror_to_errno()``.
+
+..
+
+.. bpo: 37702
+.. date: 2019-07-29-16-49-31
+.. nonce: Lj2f5e
+.. section: Windows
+
+Fix memory leak on Windows in creating an SSLContext object or running
+urllib.request.urlopen('https://...').
+
+..
+
+.. bpo: 37445
+.. date: 2019-06-28-18-10-29
+.. nonce: LsdYO6
+.. section: Windows
+
+Include the ``FORMAT_MESSAGE_IGNORE_INSERTS`` flag in ``FormatMessageW()``
+calls.
+
+..
+
+.. bpo: 37380
+.. date: 2019-06-25-04-15-22
+.. nonce: tPxjuz
+.. section: Windows
+
+Don't collect unfinished processes with ``subprocess._active`` on Windows to
+cleanup later. Patch by Ruslan Kuprieiev.
+
+..
+
+.. bpo: 32587
+.. date: 2019-05-10-15-25-44
+.. nonce: -0g2O3
+.. section: Windows
+
+Make :data:`winreg.REG_MULTI_SZ` support zero-length strings.
+
+..
+
+.. bpo: 38117
+.. date: 2019-09-15-21-29-13
+.. nonce: ZLsoAZ
+.. section: macOS
+
+Updated OpenSSL to 1.1.1d in macOS installer.
+
+..
+
+.. bpo: 38089
+.. date: 2019-09-10-14-24-35
+.. nonce: eedgyD
+.. section: macOS
+
+Move Azure Pipelines to latest VM versions and make macOS tests optional
+
+..
+
+.. bpo: 35379
+.. date: 2019-09-17-01-28-56
+.. nonce: yAECDr
+.. section: IDLE
+
+When exiting IDLE, catch any AttributeError.  One happens when
+EditorWindow.close is called twice.  Printing a traceback, when IDLE is run
+from a terminal, is useless and annoying.
+
+..
+
+.. bpo: 38183
+.. date: 2019-09-16-15-04-29
+.. nonce: eudCN1
+.. section: IDLE
+
+To avoid problems, test_idle ignores the user config directory. It no longer
+tries to create or access .idlerc or any files within. Users must run IDLE
+to discover problems with saving settings.
+
+..
+
+.. bpo: 38077
+.. date: 2019-09-09-22-08-36
+.. nonce: Mzpfe2
+.. section: IDLE
+
+IDLE no longer adds 'argv' to the user namespace when initializing it.  This
+bug only affected 3.7.4 and 3.8.0b2 to 3.8.0b4.
+
+..
+
+.. bpo: 38041
+.. date: 2019-09-05-23-12-13
+.. nonce: nxmGGK
+.. section: IDLE
+
+Shell restart lines now fill the window width, always start with '=3D', and
+avoid wrapping unnecessarily. The line will still wrap if the included file
+name is long relative to the width.
+
+..
+
+.. bpo: 35771
+.. date: 2019-09-01-10-22-55
+.. nonce: tdbmbP
+.. section: IDLE
+
+To avoid occasional spurious test_idle failures on slower machines, increase
+the ``hover_delay`` in test_tooltip.
+
+..
+
+.. bpo: 37902
+.. date: 2019-08-21-16-02-49
+.. nonce: _R_adE
+.. section: IDLE
+
+Add mousewheel scrolling for IDLE module, path, and stack browsers. Patch by
+George Zhang.
+
+..
+
+.. bpo: 37803
+.. date: 2019-09-12-16-15-55
+.. nonce: chEizy
+.. section: Tools/Demos
+
+pdb's ``--help`` and ``--version`` long options now work.
+
+..
+
+.. bpo: 37064
+.. date: 2019-05-27-15-26-12
+.. nonce: k_SPW2
+.. section: Tools/Demos
+
+Add option -k to pathscript.py script: preserve shebang flags. Add option -a
+to pathscript.py script: add flags.
+
+..
+
+.. bpo: 38234
+.. date: 2019-09-24-17-09-48
+.. nonce: d0bhEA
+.. section: C API
+
+:c:func:`Py_SetPath` now sets :data:`sys.executable` to the program full
+path (:c:func:`Py_GetProgramFullPath`) rather than to the program name
+(:c:func:`Py_GetProgramName`).
+
+..
+
+.. bpo: 38234
+.. date: 2019-09-20-17-22-41
+.. nonce: ZbquVK
+.. section: C API
+
+Python ignored arguments passed to :c:func:`Py_SetPath`,
+:c:func:`Py_SetPythonHome` and :c:func:`Py_SetProgramName`: fix Python
+initialization to use specified arguments.
+
+..
+
+.. bpo: 38205
+.. date: 2019-09-19-18-26-29
+.. nonce: Db1OJL
+.. section: C API
+
+The :c:func:`Py_UNREACHABLE` macro now calls :c:func:`Py_FatalError`.
+
+..
+
+.. bpo: 37879
+.. date: 2019-08-17-13-50-21
+.. nonce: CZeUem
+.. section: C API
+
+Fix subtype_dealloc to suppress the type decref when the base type is a C
+heap type
diff --git a/Misc/NEWS.d/next/Build/2019-08-24-00-29-40.bpo-37936.QrORqA.rst =
b/Misc/NEWS.d/next/Build/2019-08-24-00-29-40.bpo-37936.QrORqA.rst
deleted file mode 100644
index 4c6486103881a..0000000000000
--- a/Misc/NEWS.d/next/Build/2019-08-24-00-29-40.bpo-37936.QrORqA.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-The :file:`.gitignore` file no longer applies to any files that are in fact
-tracked in the Git repository.  Patch by Greg Price.
diff --git a/Misc/NEWS.d/next/Build/2019-09-10-00-54-48.bpo-37936.E7XEwu.rst =
b/Misc/NEWS.d/next/Build/2019-09-10-00-54-48.bpo-37936.E7XEwu.rst
deleted file mode 100644
index 5ded61eafe947..0000000000000
--- a/Misc/NEWS.d/next/Build/2019-09-10-00-54-48.bpo-37936.E7XEwu.rst
+++ /dev/null
@@ -1,5 +0,0 @@
-The :file:`.gitignore` file systematically keeps "rooted", with a
-non-trailing slash, all the rules that are meant to apply to files in a
-specific place in the repo.  Previously, when the intended file to ignore
-happened to be at the root of the repo, we'd most often accidentally also
-ignore files and directories with the same name anywhere in the tree.
diff --git a/Misc/NEWS.d/next/Build/2019-09-13-14-12-36.bpo-36002.Bcl4oe.rst =
b/Misc/NEWS.d/next/Build/2019-09-13-14-12-36.bpo-36002.Bcl4oe.rst
deleted file mode 100644
index 374e7b234cfe6..0000000000000
--- a/Misc/NEWS.d/next/Build/2019-09-13-14-12-36.bpo-36002.Bcl4oe.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Locate ``llvm-profdata`` and ``llvm-ar`` binaries using ``AC_PATH_TOOL``
-rather than ``AC_PATH_TARGET_TOOL``.
diff --git a/Misc/NEWS.d/next/Build/2019-09-24-22-47-47.bpo-36210.EmL9X1.rst =
b/Misc/NEWS.d/next/Build/2019-09-24-22-47-47.bpo-36210.EmL9X1.rst
deleted file mode 100644
index aa9a56fe57aec..0000000000000
--- a/Misc/NEWS.d/next/Build/2019-09-24-22-47-47.bpo-36210.EmL9X1.rst
+++ /dev/null
@@ -1,9 +0,0 @@
-Update optional extension module detection for AIX.
-ossaudiodev and spwd are not applicable for AIX, and
-are no longer reported as missing.
-3rd-party packaging of ncurses (with ASIS support)
-conflicts with officially supported AIX curses library,
-so configure AIX to use libcurses.a. However, skip
-trying to build _curses_panel.
-
-patch by M Felt
diff --git a/Misc/NEWS.d/next/Build/2019-09-28-02-37-11.bpo-38301.123456.rst =
b/Misc/NEWS.d/next/Build/2019-09-28-02-37-11.bpo-38301.123456.rst
deleted file mode 100644
index 59c9a76385e2b..0000000000000
--- a/Misc/NEWS.d/next/Build/2019-09-28-02-37-11.bpo-38301.123456.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-In Solaris family, we must be sure to use ``-D_REENTRANT``.
-Patch by Jes=C3=BAs Cea Avi=C3=B3n.
diff --git a/Misc/NEWS.d/next/C API/2019-08-17-13-50-21.bpo-37879.CZeUem.rst =
b/Misc/NEWS.d/next/C API/2019-08-17-13-50-21.bpo-37879.CZeUem.rst
deleted file mode 100644
index 87322fbf5f239..0000000000000
--- a/Misc/NEWS.d/next/C API/2019-08-17-13-50-21.bpo-37879.CZeUem.rst=09
+++ /dev/null
@@ -1,2 +0,0 @@
-Fix subtype_dealloc to suppress the type decref when the base type is a C
-heap type
diff --git a/Misc/NEWS.d/next/C API/2019-09-19-18-26-29.bpo-38205.Db1OJL.rst =
b/Misc/NEWS.d/next/C API/2019-09-19-18-26-29.bpo-38205.Db1OJL.rst
deleted file mode 100644
index 1818e54d461a4..0000000000000
--- a/Misc/NEWS.d/next/C API/2019-09-19-18-26-29.bpo-38205.Db1OJL.rst=09
+++ /dev/null
@@ -1 +0,0 @@
-The :c:func:`Py_UNREACHABLE` macro now calls :c:func:`Py_FatalError`.
diff --git a/Misc/NEWS.d/next/C API/2019-09-20-17-22-41.bpo-38234.ZbquVK.rst =
b/Misc/NEWS.d/next/C API/2019-09-20-17-22-41.bpo-38234.ZbquVK.rst
deleted file mode 100644
index fe658ad75d89e..0000000000000
--- a/Misc/NEWS.d/next/C API/2019-09-20-17-22-41.bpo-38234.ZbquVK.rst=09
+++ /dev/null
@@ -1,3 +0,0 @@
-Python ignored arguments passed to :c:func:`Py_SetPath`,
-:c:func:`Py_SetPythonHome` and :c:func:`Py_SetProgramName`: fix Python
-initialization to use specified arguments.
diff --git a/Misc/NEWS.d/next/C API/2019-09-24-17-09-48.bpo-38234.d0bhEA.rst =
b/Misc/NEWS.d/next/C API/2019-09-24-17-09-48.bpo-38234.d0bhEA.rst
deleted file mode 100644
index ba4cc312e6928..0000000000000
--- a/Misc/NEWS.d/next/C API/2019-09-24-17-09-48.bpo-38234.d0bhEA.rst=09
+++ /dev/null
@@ -1,3 +0,0 @@
-:c:func:`Py_SetPath` now sets :data:`sys.executable` to the program full
-path (:c:func:`Py_GetProgramFullPath`) rather than to the program name
-(:c:func:`Py_GetProgramName`).
diff --git a/Misc/NEWS.d/next/Core and Builtins/2018-06-07-01-01-20.bpo-30773=
.C31rVE.rst b/Misc/NEWS.d/next/Core and Builtins/2018-06-07-01-01-20.bpo-3077=
3.C31rVE.rst
deleted file mode 100644
index 501ee527e8a9f..0000000000000
--- a/Misc/NEWS.d/next/Core and Builtins/2018-06-07-01-01-20.bpo-30773.C31rVE=
.rst=09
+++ /dev/null
@@ -1,2 +0,0 @@
-Prohibit parallel running of aclose() / asend() / athrow(). Fix ag_running
-to reflect the actual running status of the AG.
diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-07-18-11-50-49.bpo-37619=
.X6Lulo.rst b/Misc/NEWS.d/next/Core and Builtins/2019-07-18-11-50-49.bpo-3761=
9.X6Lulo.rst
deleted file mode 100644
index 8723d3d9e9da0..0000000000000
--- a/Misc/NEWS.d/next/Core and Builtins/2019-07-18-11-50-49.bpo-37619.X6Lulo=
.rst=09
+++ /dev/null
@@ -1,3 +0,0 @@
-When adding a wrapper descriptor from one class to a different class
-(for example, setting ``__add__ =3D str.__add__`` on an ``int`` subclass),
-an exception is correctly raised when the operator is called.
diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-08-06-23-39-05.bpo-37409=
.1qwzn2.rst b/Misc/NEWS.d/next/Core and Builtins/2019-08-06-23-39-05.bpo-3740=
9.1qwzn2.rst
deleted file mode 100644
index 9cfa715480201..0000000000000
--- a/Misc/NEWS.d/next/Core and Builtins/2019-08-06-23-39-05.bpo-37409.1qwzn2=
.rst=09
+++ /dev/null
@@ -1,2 +0,0 @@
-Ensure explicit relative imports from interactive sessions and scripts (havi=
ng no parent package) always raise ImportError, rather than treating the curr=
ent module as the package.
-Patch by Ben Lewis.
diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-08-15-12-48-36.bpo-20490=
.-hXeEn.rst b/Misc/NEWS.d/next/Core and Builtins/2019-08-15-12-48-36.bpo-2049=
0.-hXeEn.rst
deleted file mode 100644
index dfee480e1aa70..0000000000000
--- a/Misc/NEWS.d/next/Core and Builtins/2019-08-15-12-48-36.bpo-20490.-hXeEn=
.rst=09
+++ /dev/null
@@ -1,2 +0,0 @@
-Improve import error message for partially initialized module on circular
-``from`` imports - by Anthony Sottile.
diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-08-27-21-21-36.bpo-37966=
.5OBLez.rst b/Misc/NEWS.d/next/Core and Builtins/2019-08-27-21-21-36.bpo-3796=
6.5OBLez.rst
deleted file mode 100644
index 6b9d69c5b3a9a..0000000000000
--- a/Misc/NEWS.d/next/Core and Builtins/2019-08-27-21-21-36.bpo-37966.5OBLez=
.rst=09
+++ /dev/null
@@ -1,3 +0,0 @@
-The implementation of :func:`~unicodedata.is_normalized` has been greatly
-sped up on strings that aren't normalized, by implementing the full
-normalization-quick-check algorithm from the Unicode standard.
diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-08-31-09-22-33.bpo-37990=
.WDY2f-.rst b/Misc/NEWS.d/next/Core and Builtins/2019-08-31-09-22-33.bpo-3799=
0.WDY2f-.rst
deleted file mode 100644
index 39edc595418e4..0000000000000
--- a/Misc/NEWS.d/next/Core and Builtins/2019-08-31-09-22-33.bpo-37990.WDY2f-=
.rst=09
+++ /dev/null
@@ -1,2 +0,0 @@
-Fix elapsed time in gc stats was not printed correctly.  This bug was
-a regression in 3.8b4.
diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-08-31-11-13-25.bpo-37994=
.Rj6S4j.rst b/Misc/NEWS.d/next/Core and Builtins/2019-08-31-11-13-25.bpo-3799=
4.Rj6S4j.rst
deleted file mode 100644
index 103ac5a309057..0000000000000
--- a/Misc/NEWS.d/next/Core and Builtins/2019-08-31-11-13-25.bpo-37994.Rj6S4j=
.rst=09
+++ /dev/null
@@ -1,2 +0,0 @@
-Fixed silencing arbitrary errors if an attribute lookup fails in several
-sites. Only AttributeError should be silenced.
diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-09-02-20-00-31.bpo-38005=
.e7VsTA.rst b/Misc/NEWS.d/next/Core and Builtins/2019-09-02-20-00-31.bpo-3800=
5.e7VsTA.rst
deleted file mode 100644
index 706abf587b90a..0000000000000
--- a/Misc/NEWS.d/next/Core and Builtins/2019-09-02-20-00-31.bpo-38005.e7VsTA=
.rst=09
+++ /dev/null
@@ -1 +0,0 @@
-Fixed comparing and creating of InterpreterID and ChannelID.
diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-09-12-00-14-01.bpo-38124=
.n6E0H7.rst b/Misc/NEWS.d/next/Core and Builtins/2019-09-12-00-14-01.bpo-3812=
4.n6E0H7.rst
deleted file mode 100644
index dca0ba5bc0cc3..0000000000000
--- a/Misc/NEWS.d/next/Core and Builtins/2019-09-12-00-14-01.bpo-38124.n6E0H7=
.rst=09
+++ /dev/null
@@ -1,2 +0,0 @@
-Fix an off-by-one error in PyState_AddModule that could cause out-of-bounds
-memory access.
diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-09-12-19-50-01.bpo-38013=
.I7btD0.rst b/Misc/NEWS.d/next/Core and Builtins/2019-09-12-19-50-01.bpo-3801=
3.I7btD0.rst
deleted file mode 100644
index a61aa48ff1efc..0000000000000
--- a/Misc/NEWS.d/next/Core and Builtins/2019-09-12-19-50-01.bpo-38013.I7btD0=
.rst=09
+++ /dev/null
@@ -1,3 +0,0 @@
-Allow to call ``async_generator_athrow().throw(...)`` even for non-started
-async generator helper. It fixes annoying warning at the end of
-:func:`asyncio.run` call.
diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-09-20-19-06-23.bpo-38236=
.eQ0Tmj.rst b/Misc/NEWS.d/next/Core and Builtins/2019-09-20-19-06-23.bpo-3823=
6.eQ0Tmj.rst
deleted file mode 100644
index 5c7729296cee5..0000000000000
--- a/Misc/NEWS.d/next/Core and Builtins/2019-09-20-19-06-23.bpo-38236.eQ0Tmj=
.rst=09
+++ /dev/null
@@ -1,2 +0,0 @@
-Python now dumps path configuration if it fails to import the Python codecs
-of the filesystem and stdio encodings.
diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-09-24-18-45-46.bpo-36871=
.p47knk.rst b/Misc/NEWS.d/next/Core and Builtins/2019-09-24-18-45-46.bpo-3687=
1.p47knk.rst
deleted file mode 100644
index 6b7b19a0d57ee..0000000000000
--- a/Misc/NEWS.d/next/Core and Builtins/2019-09-24-18-45-46.bpo-36871.p47knk=
.rst=09
+++ /dev/null
@@ -1,3 +0,0 @@
-Improve error handling for the assert_has_calls and assert_has_awaits method=
s of
-mocks. Fixed a bug where any errors encountered while binding the expected c=
alls
-to the mock's spec were silently swallowed, leading to misleading error outp=
ut.
diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-09-30-00-56-21.bpo-38317=
.pmqlIQ.rst b/Misc/NEWS.d/next/Core and Builtins/2019-09-30-00-56-21.bpo-3831=
7.pmqlIQ.rst
deleted file mode 100644
index b6d07474cf520..0000000000000
--- a/Misc/NEWS.d/next/Core and Builtins/2019-09-30-00-56-21.bpo-38317.pmqlIQ=
.rst=09
+++ /dev/null
@@ -1,2 +0,0 @@
-Fix warnings options priority: ``PyConfig.warnoptions`` has the highest
-priority, as stated in the :pep:`587`.
diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-09-30-09-33-21.bpo-38006=
.UYlJum.rst b/Misc/NEWS.d/next/Core and Builtins/2019-09-30-09-33-21.bpo-3800=
6.UYlJum.rst
deleted file mode 100644
index c2dafae0afabe..0000000000000
--- a/Misc/NEWS.d/next/Core and Builtins/2019-09-30-09-33-21.bpo-38006.UYlJum=
.rst=09
+++ /dev/null
@@ -1,3 +0,0 @@
-Fix a bug due to the interaction of weakrefs and the cyclic garbage
-collector. We must clear any weakrefs in garbage in order to prevent their
-callbacks from executing and causing a crash.
diff --git a/Misc/NEWS.d/next/Documentation/2018-07-31-15-38-26.bpo-34293.yHu=
pAL.rst b/Misc/NEWS.d/next/Documentation/2018-07-31-15-38-26.bpo-34293.yHupAL=
.rst
deleted file mode 100644
index 912a3ad48d403..0000000000000
--- a/Misc/NEWS.d/next/Documentation/2018-07-31-15-38-26.bpo-34293.yHupAL.rst
+++ /dev/null
@@ -1 +0,0 @@
-Fix the Doc/Makefile regarding PAPER environment variable and PDF builds
diff --git a/Misc/NEWS.d/next/Documentation/2018-10-26-18-10-29.bpo-33944.V1Y=
eOA.rst b/Misc/NEWS.d/next/Documentation/2018-10-26-18-10-29.bpo-33944.V1YeOA=
.rst
deleted file mode 100644
index 29bec88c96495..0000000000000
--- a/Misc/NEWS.d/next/Documentation/2018-10-26-18-10-29.bpo-33944.V1YeOA.rst
+++ /dev/null
@@ -1 +0,0 @@
-Added a note about the intended use of code in .pth files.
diff --git a/Misc/NEWS.d/next/Documentation/2019-01-21-14-30-59.bpo-35803.yae=
6Lq.rst b/Misc/NEWS.d/next/Documentation/2019-01-21-14-30-59.bpo-35803.yae6Lq=
.rst
deleted file mode 100644
index b8394560e5486..0000000000000
--- a/Misc/NEWS.d/next/Documentation/2019-01-21-14-30-59.bpo-35803.yae6Lq.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Document and test that ``tempfile`` functions may accept a
-:term:`path-like object` for the ``dir`` argument.  Patch by Anthony Sottile.
diff --git a/Misc/NEWS.d/next/Documentation/2019-02-14-07-12-48.bpo-23460.Iqi=
qtm.rst b/Misc/NEWS.d/next/Documentation/2019-02-14-07-12-48.bpo-23460.Iqiqtm=
.rst
deleted file mode 100644
index 4674d551d3797..0000000000000
--- a/Misc/NEWS.d/next/Documentation/2019-02-14-07-12-48.bpo-23460.Iqiqtm.rst
+++ /dev/null
@@ -1 +0,0 @@
-The documentation for decimal string formatting using the `:g` specifier has=
 been updated to reflect the correct exponential notation cutoff point. Origi=
nal patch contributed by Tuomas Suutari.
\ No newline at end of file
diff --git a/Misc/NEWS.d/next/Documentation/2019-05-18-16-25-44.bpo-36960.xEK=
HXj.rst b/Misc/NEWS.d/next/Documentation/2019-05-18-16-25-44.bpo-36960.xEKHXj=
.rst
deleted file mode 100644
index 1d4b213367e76..0000000000000
--- a/Misc/NEWS.d/next/Documentation/2019-05-18-16-25-44.bpo-36960.xEKHXj.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Restructured the :mod:`datetime` docs in the interest of making them more
-user-friendly and improving readability. Patch by Brad Solomon.
diff --git a/Misc/NEWS.d/next/Documentation/2019-06-04-09-29-00.bpo-36260.WrG=
uc-.rst b/Misc/NEWS.d/next/Documentation/2019-06-04-09-29-00.bpo-36260.WrGuc-=
.rst
deleted file mode 100644
index 9276516a88239..0000000000000
--- a/Misc/NEWS.d/next/Documentation/2019-06-04-09-29-00.bpo-36260.WrGuc-.rst
+++ /dev/null
@@ -1 +0,0 @@
-Add decompression pitfalls to zipfile module documentation.
\ No newline at end of file
diff --git a/Misc/NEWS.d/next/Documentation/2019-08-24-12-59-06.bpo-37937.F7f=
Hbt.rst b/Misc/NEWS.d/next/Documentation/2019-08-24-12-59-06.bpo-37937.F7fHbt=
.rst
deleted file mode 100644
index 26cc16a74d1c7..0000000000000
--- a/Misc/NEWS.d/next/Documentation/2019-08-24-12-59-06.bpo-37937.F7fHbt.rst
+++ /dev/null
@@ -1 +0,0 @@
-Mention ``frame.f_trace`` in :func:`sys.settrace` docs.
diff --git a/Misc/NEWS.d/next/Documentation/2019-08-29-14-38-01.bpo-37977.pML=
-UI.rst b/Misc/NEWS.d/next/Documentation/2019-08-29-14-38-01.bpo-37977.pML-UI=
.rst
deleted file mode 100644
index cd0fa3c0584a3..0000000000000
--- a/Misc/NEWS.d/next/Documentation/2019-08-29-14-38-01.bpo-37977.pML-UI.rst
+++ /dev/null
@@ -1 +0,0 @@
-Warn more strongly and clearly about pickle insecurity
diff --git a/Misc/NEWS.d/next/Documentation/2019-09-05-14-47-51.bpo-36797.KN9=
Ga5.rst b/Misc/NEWS.d/next/Documentation/2019-09-05-14-47-51.bpo-36797.KN9Ga5=
.rst
deleted file mode 100644
index 62a3c176b1c7f..0000000000000
--- a/Misc/NEWS.d/next/Documentation/2019-09-05-14-47-51.bpo-36797.KN9Ga5.rst
+++ /dev/null
@@ -1 +0,0 @@
-Fix a dead link in the distutils API Reference.
diff --git a/Misc/NEWS.d/next/Documentation/2019-09-07-15-55-46.bpo-26868.Raw=
0Gd.rst b/Misc/NEWS.d/next/Documentation/2019-09-07-15-55-46.bpo-26868.Raw0Gd=
.rst
deleted file mode 100644
index c668092b41a66..0000000000000
--- a/Misc/NEWS.d/next/Documentation/2019-09-07-15-55-46.bpo-26868.Raw0Gd.rst
+++ /dev/null
@@ -1 +0,0 @@
-Fix example usage of :c:func:`PyModule_AddObject` to properly handle errors.
\ No newline at end of file
diff --git a/Misc/NEWS.d/next/IDLE/2019-08-21-16-02-49.bpo-37902._R_adE.rst b=
/Misc/NEWS.d/next/IDLE/2019-08-21-16-02-49.bpo-37902._R_adE.rst
deleted file mode 100644
index 24b4142484695..0000000000000
--- a/Misc/NEWS.d/next/IDLE/2019-08-21-16-02-49.bpo-37902._R_adE.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Add mousewheel scrolling for IDLE module, path, and stack browsers.
-Patch by George Zhang.
diff --git a/Misc/NEWS.d/next/IDLE/2019-09-01-10-22-55.bpo-35771.tdbmbP.rst b=
/Misc/NEWS.d/next/IDLE/2019-09-01-10-22-55.bpo-35771.tdbmbP.rst
deleted file mode 100644
index b96e53f5a5dfd..0000000000000
--- a/Misc/NEWS.d/next/IDLE/2019-09-01-10-22-55.bpo-35771.tdbmbP.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-To avoid occasional spurious test_idle failures on slower machines,=20
-increase the ``hover_delay`` in test_tooltip.
diff --git a/Misc/NEWS.d/next/IDLE/2019-09-05-23-12-13.bpo-38041.nxmGGK.rst b=
/Misc/NEWS.d/next/IDLE/2019-09-05-23-12-13.bpo-38041.nxmGGK.rst
deleted file mode 100644
index 0aa254e8ca70f..0000000000000
--- a/Misc/NEWS.d/next/IDLE/2019-09-05-23-12-13.bpo-38041.nxmGGK.rst
+++ /dev/null
@@ -1,3 +0,0 @@
-Shell restart lines now fill the window width, always start with '=3D',
-and avoid wrapping unnecessarily. The line will still wrap if the
-included file name is long relative to the width.
diff --git a/Misc/NEWS.d/next/IDLE/2019-09-09-22-08-36.bpo-38077.Mzpfe2.rst b=
/Misc/NEWS.d/next/IDLE/2019-09-09-22-08-36.bpo-38077.Mzpfe2.rst
deleted file mode 100644
index ba1fd55defaa2..0000000000000
--- a/Misc/NEWS.d/next/IDLE/2019-09-09-22-08-36.bpo-38077.Mzpfe2.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-IDLE no longer adds 'argv' to the user namespace when initializing it.  This
-bug only affected 3.7.4 and 3.8.0b2 to 3.8.0b4.
diff --git a/Misc/NEWS.d/next/IDLE/2019-09-16-15-04-29.bpo-38183.eudCN1.rst b=
/Misc/NEWS.d/next/IDLE/2019-09-16-15-04-29.bpo-38183.eudCN1.rst
deleted file mode 100644
index e545f49bc836e..0000000000000
--- a/Misc/NEWS.d/next/IDLE/2019-09-16-15-04-29.bpo-38183.eudCN1.rst
+++ /dev/null
@@ -1,3 +0,0 @@
-To avoid problems, test_idle ignores the user config directory.
-It no longer tries to create or access .idlerc or any files within.
-Users must run IDLE to discover problems with saving settings.
diff --git a/Misc/NEWS.d/next/IDLE/2019-09-17-01-28-56.bpo-35379.yAECDr.rst b=
/Misc/NEWS.d/next/IDLE/2019-09-17-01-28-56.bpo-35379.yAECDr.rst
deleted file mode 100644
index 98d41674b65d5..0000000000000
--- a/Misc/NEWS.d/next/IDLE/2019-09-17-01-28-56.bpo-35379.yAECDr.rst
+++ /dev/null
@@ -1,3 +0,0 @@
-When exiting IDLE, catch any AttributeError.  One happens when
-EditorWindow.close is called twice.  Printing a traceback, when IDLE is run
-from a terminal, is useless and annoying.
diff --git a/Misc/NEWS.d/next/Library/2018-08-27-15-44-50.bpo-34519.cPlH1h.rs=
t b/Misc/NEWS.d/next/Library/2018-08-27-15-44-50.bpo-34519.cPlH1h.rst
deleted file mode 100644
index e17e2dc0fdcce..0000000000000
--- a/Misc/NEWS.d/next/Library/2018-08-27-15-44-50.bpo-34519.cPlH1h.rst
+++ /dev/null
@@ -1 +0,0 @@
-Add additional aliases for HP Roman 8. Patch by Michael Osipov.
diff --git a/Misc/NEWS.d/next/Library/2018-11-21-18-05-50.bpo-20504.kG0ub5.rs=
t b/Misc/NEWS.d/next/Library/2018-11-21-18-05-50.bpo-20504.kG0ub5.rst
deleted file mode 100644
index 726329ad0d65a..0000000000000
--- a/Misc/NEWS.d/next/Library/2018-11-21-18-05-50.bpo-20504.kG0ub5.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Fixes a bug in :mod:`cgi` module when a multipart/form-data request has no
-`Content-Length` header.
diff --git a/Misc/NEWS.d/next/Library/2019-01-22-09-23-20.bpo-35168.UGv2yW.rs=
t b/Misc/NEWS.d/next/Library/2019-01-22-09-23-20.bpo-35168.UGv2yW.rst
deleted file mode 100644
index 10684fdbde2f1..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-01-22-09-23-20.bpo-35168.UGv2yW.rst
+++ /dev/null
@@ -1 +0,0 @@
-:attr:`shlex.shlex.punctuation_chars` is now a read-only property.
diff --git a/Misc/NEWS.d/next/Library/2019-03-09-16-04-12.bpo-36250.tSK4N1.rs=
t b/Misc/NEWS.d/next/Library/2019-03-09-16-04-12.bpo-36250.tSK4N1.rst
deleted file mode 100644
index 8d9fbcbb1cba5..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-03-09-16-04-12.bpo-36250.tSK4N1.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Ignore ``ValueError`` from ``signal`` with ``interaction`` in non-main
-thread.
diff --git a/Misc/NEWS.d/next/Library/2019-05-19-10-48-46.bpo-21315.PgXVqF.rs=
t b/Misc/NEWS.d/next/Library/2019-05-19-10-48-46.bpo-21315.PgXVqF.rst
deleted file mode 100644
index dd0dd7f72c0a3..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-05-19-10-48-46.bpo-21315.PgXVqF.rst
+++ /dev/null
@@ -1,4 +0,0 @@
-Email headers containing RFC2047 encoded words are parsed despite the missing
-whitespace, and a defect registered. Also missing trailing whitespace after
-encoded words is now registered as a defect.
-
diff --git a/Misc/NEWS.d/next/Library/2019-05-22-04-52-35.bpo-25068.vR_rC-.rs=
t b/Misc/NEWS.d/next/Library/2019-05-22-04-52-35.bpo-25068.vR_rC-.rst
deleted file mode 100644
index 89de83bce0290..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-05-22-04-52-35.bpo-25068.vR_rC-.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-:class:`urllib.request.ProxyHandler` now lowercases the keys of the passed
-dictionary.
diff --git a/Misc/NEWS.d/next/Library/2019-05-26-16-34-53.bpo-31163.21A802.rs=
t b/Misc/NEWS.d/next/Library/2019-05-26-16-34-53.bpo-31163.21A802.rst
deleted file mode 100644
index 6ec074612141c..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-05-26-16-34-53.bpo-31163.21A802.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-pathlib.Path instance's rename and replace methods now return the new Path
-instance.
diff --git a/Misc/NEWS.d/next/Library/2019-06-08-23-26-58.bpo-12144.Z7mz-q.rs=
t b/Misc/NEWS.d/next/Library/2019-06-08-23-26-58.bpo-12144.Z7mz-q.rst
deleted file mode 100644
index ee802f81477ec..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-06-08-23-26-58.bpo-12144.Z7mz-q.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Ensure cookies with ``expires`` attribute are handled in
-:meth:`CookieJar.make_cookies`.
diff --git a/Misc/NEWS.d/next/Library/2019-06-09-22-25-03.bpo-37206.2WBg4q.rs=
t b/Misc/NEWS.d/next/Library/2019-06-09-22-25-03.bpo-37206.2WBg4q.rst
deleted file mode 100644
index 2c6c6821e639e..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-06-09-22-25-03.bpo-37206.2WBg4q.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Default values which cannot be represented as Python objects no longer
-improperly represented as ``None`` in function signatures.
diff --git a/Misc/NEWS.d/next/Library/2019-06-12-08-56-22.bpo-21872.V9QGGN.rs=
t b/Misc/NEWS.d/next/Library/2019-06-12-08-56-22.bpo-21872.V9QGGN.rst
deleted file mode 100644
index 90df756f9a81b..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-06-12-08-56-22.bpo-21872.V9QGGN.rst
+++ /dev/null
@@ -1,3 +0,0 @@
-Fix :mod:`lzma`: module decompresses data incompletely. When decompressing a
-FORMAT_ALONE format file, and it doesn't have the end marker, sometimes the
-last one to dozens bytes can't be output. Patch by Ma Lin.
diff --git a/Misc/NEWS.d/next/Library/2019-06-18-13-59-55.bpo-37305.fGzWlP.rs=
t b/Misc/NEWS.d/next/Library/2019-06-18-13-59-55.bpo-37305.fGzWlP.rst
deleted file mode 100644
index 5bd2cd4d30afa..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-06-18-13-59-55.bpo-37305.fGzWlP.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Add .webmanifest -> application/manifest+json to list of recognized file typ=
es
-and content type headers
diff --git a/Misc/NEWS.d/next/Library/2019-06-22-22-00-35.bpo-37212.Zhv-tq.rs=
t b/Misc/NEWS.d/next/Library/2019-06-22-22-00-35.bpo-37212.Zhv-tq.rst
deleted file mode 100644
index 520a0229aa9d0..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-06-22-22-00-35.bpo-37212.Zhv-tq.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-:func:`unittest.mock.call` now preserves the order of keyword arguments in
-repr output. Patch by Karthikeyan Singaravelan.
diff --git a/Misc/NEWS.d/next/Library/2019-07-04-13-00-20.bpo-37424.0i1MR-.rs=
t b/Misc/NEWS.d/next/Library/2019-07-04-13-00-20.bpo-37424.0i1MR-.rst
deleted file mode 100644
index b98a17e241e07..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-07-04-13-00-20.bpo-37424.0i1MR-.rst
+++ /dev/null
@@ -1,5 +0,0 @@
-Fixes a possible hang when using a timeout on `subprocess.run()` while
-capturing output.  If the child process spawned its own children or
-otherwise connected its stdout or stderr handles with another process, we
-could hang after the timeout was reached and our child was killed when
-attempting to read final output from the pipes.
diff --git a/Misc/NEWS.d/next/Library/2019-07-29-21-39-45.bpo-11953.4Hpwf9.rs=
t b/Misc/NEWS.d/next/Library/2019-07-29-21-39-45.bpo-11953.4Hpwf9.rst
deleted file mode 100644
index 62f3c44c284ea..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-07-29-21-39-45.bpo-11953.4Hpwf9.rst
+++ /dev/null
@@ -1 +0,0 @@
-Completing WSA* error codes in :mod:`socket`.
diff --git a/Misc/NEWS.d/next/Library/2019-08-07-14-49-22.bpo-37785.y7OlT8.rs=
t b/Misc/NEWS.d/next/Library/2019-08-07-14-49-22.bpo-37785.y7OlT8.rst
deleted file mode 100644
index 2d3aa4f9a75a0..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-08-07-14-49-22.bpo-37785.y7OlT8.rst
+++ /dev/null
@@ -1 +0,0 @@
-Fix xgettext warnings in :mod:`argparse`.
diff --git a/Misc/NEWS.d/next/Library/2019-08-19-10-31-41.bpo-37885.4Nc9sp.rs=
t b/Misc/NEWS.d/next/Library/2019-08-19-10-31-41.bpo-37885.4Nc9sp.rst
deleted file mode 100644
index 055d02970146e..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-08-19-10-31-41.bpo-37885.4Nc9sp.rst
+++ /dev/null
@@ -1 +0,0 @@
-venv: Don't generate unset variable warning on deactivate.
\ No newline at end of file
diff --git a/Misc/NEWS.d/next/Library/2019-08-27-01-03-26.bpo-22347._TRpYr.rs=
t b/Misc/NEWS.d/next/Library/2019-08-27-01-03-26.bpo-22347._TRpYr.rst
deleted file mode 100644
index 1a3c19938217c..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-08-27-01-03-26.bpo-22347._TRpYr.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Update mimetypes.guess_type to allow proper parsing of URLs with only a host=
 name.
-Patch by Dong-hee Na.
diff --git a/Misc/NEWS.d/next/Library/2019-08-28-21-40-12.bpo-37972.kP-n4L.rs=
t b/Misc/NEWS.d/next/Library/2019-08-28-21-40-12.bpo-37972.kP-n4L.rst
deleted file mode 100644
index 73d9ef77762b7..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-08-28-21-40-12.bpo-37972.kP-n4L.rst
+++ /dev/null
@@ -1,5 +0,0 @@
-Subscripts to the `unittest.mock.call` objects now receive the same chaining=
 mechanism as any other custom attributes, so that the following usage no lon=
ger raises a `TypeError`:
-
-    call().foo().__getitem__('bar')
-
-Patch by blhsing
diff --git a/Misc/NEWS.d/next/Library/2019-08-30-11-21-10.bpo-37140.cFAX-a.rs=
t b/Misc/NEWS.d/next/Library/2019-08-30-11-21-10.bpo-37140.cFAX-a.rst
deleted file mode 100644
index 4eaa226147f9e..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-08-30-11-21-10.bpo-37140.cFAX-a.rst
+++ /dev/null
@@ -1,5 +0,0 @@
-Fix a ctypes regression of Python 3.8. When a ctypes.Structure is passed by
-copy to a function, ctypes internals created a temporary object which had
-the side effect of calling the structure finalizer (__del__) twice. The
-Python semantics requires a finalizer to be called exactly once. Fix ctypes
-internals to no longer call the finalizer twice.
diff --git a/Misc/NEWS.d/next/Library/2019-08-31-01-52-59.bpo-34410.7KbWZQ.rs=
t b/Misc/NEWS.d/next/Library/2019-08-31-01-52-59.bpo-34410.7KbWZQ.rst
deleted file mode 100644
index 64e778ee0913c..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-08-31-01-52-59.bpo-34410.7KbWZQ.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Fixed a crash in the :func:`tee` iterator when re-enter it. RuntimeError is
-now raised in this case.
diff --git a/Misc/NEWS.d/next/Library/2019-09-02-13-37-27.bpo-38006.Y7vA0Q.rs=
t b/Misc/NEWS.d/next/Library/2019-09-02-13-37-27.bpo-38006.Y7vA0Q.rst
deleted file mode 100644
index ff064ad3f1a49..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-09-02-13-37-27.bpo-38006.Y7vA0Q.rst
+++ /dev/null
@@ -1,3 +0,0 @@
-weakref.WeakValueDictionary defines a local remove() function used as
-callback for weak references. This function was created with a closure.
-Modify the implementation to avoid the closure.
diff --git a/Misc/NEWS.d/next/Library/2019-09-02-14-30-39.bpo-38010.JOnz9Z.rs=
t b/Misc/NEWS.d/next/Library/2019-09-02-14-30-39.bpo-38010.JOnz9Z.rst
deleted file mode 100644
index a97f9b7d9a401..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-09-02-14-30-39.bpo-38010.JOnz9Z.rst
+++ /dev/null
@@ -1 +0,0 @@
-In ``importlib.metadata`` sync with ``importlib_metadata`` 0.20, clarifying =
behavior of ``files()`` and fixing issue where only one requirement was retur=
ned for ``requires()`` on ``dist-info`` packages.
diff --git a/Misc/NEWS.d/next/Library/2019-09-04-20-34-14.bpo-38026.0LLRX-.rs=
t b/Misc/NEWS.d/next/Library/2019-09-04-20-34-14.bpo-38026.0LLRX-.rst
deleted file mode 100644
index 166cd867514ca..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-09-04-20-34-14.bpo-38026.0LLRX-.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Fixed :func:`inspect.getattr_static` used ``isinstance`` while it should
-avoid dynamic lookup.
diff --git a/Misc/NEWS.d/next/Library/2019-09-06-17-40-34.bpo-37953.db5FQq.rs=
t b/Misc/NEWS.d/next/Library/2019-09-06-17-40-34.bpo-37953.db5FQq.rst
deleted file mode 100644
index 4eff4f7479aad..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-09-06-17-40-34.bpo-37953.db5FQq.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-In :mod:`typing`, improved the ``__hash__`` and ``__eq__`` methods for
-:class:`ForwardReferences`.
diff --git a/Misc/NEWS.d/next/Library/2019-09-08-11-36-50.bpo-38059.8SA6co.rs=
t b/Misc/NEWS.d/next/Library/2019-09-08-11-36-50.bpo-38059.8SA6co.rst
deleted file mode 100644
index 001952ae12613..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-09-08-11-36-50.bpo-38059.8SA6co.rst
+++ /dev/null
@@ -1 +0,0 @@
-inspect.py now uses sys.exit() instead of exit()
diff --git a/Misc/NEWS.d/next/Library/2019-09-09-14-39-47.bpo-38066.l9mWv-.rs=
t b/Misc/NEWS.d/next/Library/2019-09-09-14-39-47.bpo-38066.l9mWv-.rst
deleted file mode 100644
index a69924fe2aa41..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-09-09-14-39-47.bpo-38066.l9mWv-.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Hide internal asyncio.Stream methods: feed_eof(), feed_data(),
-set_exception() and set_transport().
diff --git a/Misc/NEWS.d/next/Library/2019-09-09-18-39-23.bpo-38037.B0UgFU.rs=
t b/Misc/NEWS.d/next/Library/2019-09-09-18-39-23.bpo-38037.B0UgFU.rst
deleted file mode 100644
index 607673bd9994b..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-09-09-18-39-23.bpo-38037.B0UgFU.rst
+++ /dev/null
@@ -1 +0,0 @@
-Fix reference counters in the :mod:`signal` module.
diff --git a/Misc/NEWS.d/next/Library/2019-09-10-10-59-50.bpo-37251.8zn2o3.rs=
t b/Misc/NEWS.d/next/Library/2019-09-10-10-59-50.bpo-37251.8zn2o3.rst
deleted file mode 100644
index 27fd1e46963c7..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-09-10-10-59-50.bpo-37251.8zn2o3.rst
+++ /dev/null
@@ -1,3 +0,0 @@
-Remove `__code__` check in AsyncMock that incorrectly
-evaluated function specs as async objects but failed to evaluate classes
-with `__await__` but no `__code__` attribute defined as async objects.
diff --git a/Misc/NEWS.d/next/Library/2019-09-10-11-42-59.bpo-38086.w5TlG-.rs=
t b/Misc/NEWS.d/next/Library/2019-09-10-11-42-59.bpo-38086.w5TlG-.rst
deleted file mode 100644
index 40db53c97da4d..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-09-10-11-42-59.bpo-38086.w5TlG-.rst
+++ /dev/null
@@ -1 +0,0 @@
-Update importlib.metadata with changes from `importlib_metadata 0.21 <https:=
//gitlab.com/python-devs/importlib_metadata/blob/0.21/importlib_metadata/docs=
/changelog.rst>`_.
\ No newline at end of file
diff --git a/Misc/NEWS.d/next/Library/2019-09-11-11-44-16.bpo-37488.S8CJUL.rs=
t b/Misc/NEWS.d/next/Library/2019-09-11-11-44-16.bpo-37488.S8CJUL.rst
deleted file mode 100644
index 4f9041502a914..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-09-11-11-44-16.bpo-37488.S8CJUL.rst
+++ /dev/null
@@ -1 +0,0 @@
-Add warning to :meth:`datetime.utctimetuple`,  :meth:`datetime.utcnow` and :=
meth:`datetime.utcfromtimestamp` .
\ No newline at end of file
diff --git a/Misc/NEWS.d/next/Library/2019-09-11-14-45-30.bpo-38093.yQ6k7y.rs=
t b/Misc/NEWS.d/next/Library/2019-09-11-14-45-30.bpo-38093.yQ6k7y.rst
deleted file mode 100644
index 24a53013cec78..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-09-11-14-45-30.bpo-38093.yQ6k7y.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Fixes AsyncMock so it doesn't crash when used with AsyncContextManagers
-or AsyncIterators.
diff --git a/Misc/NEWS.d/next/Library/2019-09-11-14-49-20.bpo-38110.A19Y-q.rs=
t b/Misc/NEWS.d/next/Library/2019-09-11-14-49-20.bpo-38110.A19Y-q.rst
deleted file mode 100644
index 7b94a2e90a503..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-09-11-14-49-20.bpo-38110.A19Y-q.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-The os.closewalk() implementation now uses the libc fdwalk() API on
-platforms where it is available.
diff --git a/Misc/NEWS.d/next/Library/2019-09-11-16-54-57.bpo-38121.SrSDzB.rs=
t b/Misc/NEWS.d/next/Library/2019-09-11-16-54-57.bpo-38121.SrSDzB.rst
deleted file mode 100644
index 93eabcfaab397..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-09-11-16-54-57.bpo-38121.SrSDzB.rst
+++ /dev/null
@@ -1 +0,0 @@
-Update parameter names on functions in importlib.metadata matching the chang=
es in the 0.22 release of importlib_metadata.
\ No newline at end of file
diff --git a/Misc/NEWS.d/next/Library/2019-09-11-20-27-41.bpo-37405.MG5xiY.rs=
t b/Misc/NEWS.d/next/Library/2019-09-11-20-27-41.bpo-37405.MG5xiY.rst
deleted file mode 100644
index 09e1097342210..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-09-11-20-27-41.bpo-37405.MG5xiY.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Fixed regression bug for socket.getsockname() for non-CAN_ISOTP AF_CAN
-address family sockets by returning a 1-tuple instead of string.
diff --git a/Misc/NEWS.d/next/Library/2019-09-12-10-47-34.bpo-38008.sH74Iy.rs=
t b/Misc/NEWS.d/next/Library/2019-09-12-10-47-34.bpo-38008.sH74Iy.rst
deleted file mode 100644
index f696707eadf74..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-09-12-10-47-34.bpo-38008.sH74Iy.rst
+++ /dev/null
@@ -1,3 +0,0 @@
-Fix parent class check in protocols to correctly identify the module that
-provides a builtin protocol, instead of assuming they all come from the
-:mod:`collections.abc` module
diff --git a/Misc/NEWS.d/next/Library/2019-09-12-12-47-35.bpo-38132.KSFx1F.rs=
t b/Misc/NEWS.d/next/Library/2019-09-12-12-47-35.bpo-38132.KSFx1F.rst
deleted file mode 100644
index c2e3b6290d9ef..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-09-12-12-47-35.bpo-38132.KSFx1F.rst
+++ /dev/null
@@ -1,3 +0,0 @@
-The OpenSSL hashlib wrapper uses a simpler implementation. Several Macros
-and pointless caches are gone. The hash name now comes from OpenSSL's EVP.
-The algorithm name stays the same, except it is now always lower case.
diff --git a/Misc/NEWS.d/next/Library/2019-09-12-13-18-55.bpo-38134.gXJTbP.rs=
t b/Misc/NEWS.d/next/Library/2019-09-12-13-18-55.bpo-38134.gXJTbP.rst
deleted file mode 100644
index fc134c203fdd8..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-09-12-13-18-55.bpo-38134.gXJTbP.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Remove obsolete copy of PBKDF2_HMAC_fast. All supported OpenSSL versions
-contain a fast implementation.
diff --git a/Misc/NEWS.d/next/Library/2019-09-12-14-52-38.bpo-36991.1OcSm8.rs=
t b/Misc/NEWS.d/next/Library/2019-09-12-14-52-38.bpo-36991.1OcSm8.rst
deleted file mode 100644
index c6fa852e94ad9..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-09-12-14-52-38.bpo-36991.1OcSm8.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Fixes a potential incorrect AttributeError exception escaping
-ZipFile.extract() in some unsupported input error situations.
diff --git a/Misc/NEWS.d/next/Library/2019-09-13-08-55-43.bpo-38148.Lnww6D.rs=
t b/Misc/NEWS.d/next/Library/2019-09-13-08-55-43.bpo-38148.Lnww6D.rst
deleted file mode 100644
index 88c0809b1c2a4..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-09-13-08-55-43.bpo-38148.Lnww6D.rst
+++ /dev/null
@@ -1 +0,0 @@
-Add slots to :mod:`asyncio` transport classes, which can reduce memory usage.
diff --git a/Misc/NEWS.d/next/Library/2019-09-13-09-24-58.bpo-38115.BOO-Y1.rs=
t b/Misc/NEWS.d/next/Library/2019-09-13-09-24-58.bpo-38115.BOO-Y1.rst
deleted file mode 100644
index 5119c0546e369..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-09-13-09-24-58.bpo-38115.BOO-Y1.rst
+++ /dev/null
@@ -1 +0,0 @@
-Fix a bug in dis.findlinestarts() where it would return invalid bytecode off=
sets. Document that a code object's co_lnotab can contain invalid bytecode of=
fsets.
\ No newline at end of file
diff --git a/Misc/NEWS.d/next/Library/2019-09-13-12-18-51.bpo-38153.nHAbuJ.rs=
t b/Misc/NEWS.d/next/Library/2019-09-13-12-18-51.bpo-38153.nHAbuJ.rst
deleted file mode 100644
index 8a483c760adc7..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-09-13-12-18-51.bpo-38153.nHAbuJ.rst
+++ /dev/null
@@ -1,3 +0,0 @@
-Names of hashing algorithms frome OpenSSL are now normalized to follow
-Python's naming conventions. For example OpenSSL uses sha3-512 instead of
-sha3_512 or blake2b512 instead of blake2b.=20
diff --git a/Misc/NEWS.d/next/Library/2019-09-13-14-54-33.bpo-34706.HWVpOY.rs=
t b/Misc/NEWS.d/next/Library/2019-09-13-14-54-33.bpo-34706.HWVpOY.rst
deleted file mode 100644
index 1df3742abd272..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-09-13-14-54-33.bpo-34706.HWVpOY.rst
+++ /dev/null
@@ -1 +0,0 @@
-Preserve subclassing in inspect.Signature.from_callable.
diff --git a/Misc/NEWS.d/next/Library/2019-09-14-10-34-00.bpo-33936.8wCI_n.rs=
t b/Misc/NEWS.d/next/Library/2019-09-14-10-34-00.bpo-33936.8wCI_n.rst
deleted file mode 100644
index 7bc7fed483a2e..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-09-14-10-34-00.bpo-33936.8wCI_n.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-_hashlib no longer calls obsolete OpenSSL initialization function with
-OpenSSL 1.1.0+.
diff --git a/Misc/NEWS.d/next/Library/2019-09-15-10-30-33.bpo-38175.61XlUv.rs=
t b/Misc/NEWS.d/next/Library/2019-09-15-10-30-33.bpo-38175.61XlUv.rst
deleted file mode 100644
index 6d9f280bbd050..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-09-15-10-30-33.bpo-38175.61XlUv.rst
+++ /dev/null
@@ -1 +0,0 @@
-Fix a memory leak in comparison of :class:`sqlite3.Row` objects.
diff --git a/Misc/NEWS.d/next/Library/2019-09-15-21-31-18.bpo-37828.gLLDX7.rs=
t b/Misc/NEWS.d/next/Library/2019-09-15-21-31-18.bpo-37828.gLLDX7.rst
deleted file mode 100644
index c364009b24086..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-09-15-21-31-18.bpo-37828.gLLDX7.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Fix default mock name in :meth:`unittest.mock.Mock.assert_called` exceptions.
-Patch by Abraham Toriz Cruz.
diff --git a/Misc/NEWS.d/next/Library/2019-09-16-09-54-42.bpo-38136.MdI-Zb.rs=
t b/Misc/NEWS.d/next/Library/2019-09-16-09-54-42.bpo-38136.MdI-Zb.rst
deleted file mode 100644
index 78cad245e67d4..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-09-16-09-54-42.bpo-38136.MdI-Zb.rst
+++ /dev/null
@@ -1,3 +0,0 @@
-Changes AsyncMock call count and await count to be two different counters.
-Now await count only counts when a coroutine has been awaited, not when it
-has been called, and vice-versa. Update the documentation around this.
diff --git a/Misc/NEWS.d/next/Library/2019-09-16-19-12-57.bpo-38185.zYWppY.rs=
t b/Misc/NEWS.d/next/Library/2019-09-16-19-12-57.bpo-38185.zYWppY.rst
deleted file mode 100644
index 2260db68e8cf7..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-09-16-19-12-57.bpo-38185.zYWppY.rst
+++ /dev/null
@@ -1 +0,0 @@
-Fixed case-insensitive string comparison in :class:`sqlite3.Row` indexing.
diff --git a/Misc/NEWS.d/next/Library/2019-09-17-12-28-27.bpo-38191.1TU0HV.rs=
t b/Misc/NEWS.d/next/Library/2019-09-17-12-28-27.bpo-38191.1TU0HV.rst
deleted file mode 100644
index 1a6de60e46c18..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-09-17-12-28-27.bpo-38191.1TU0HV.rst
+++ /dev/null
@@ -1,4 +0,0 @@
-Constructors of :class:`~typing.NamedTuple` and :class:`~typing.TypedDict`
-types now accept arbitrary keyword argument names, including "cls", "self",
-"typename", "_typename", "fields" and "_fields".  Passing positional
-arguments by keyword is deprecated.
diff --git a/Misc/NEWS.d/next/Library/2019-09-20-14-27-17.bpo-38237.xRUZbx.rs=
t b/Misc/NEWS.d/next/Library/2019-09-20-14-27-17.bpo-38237.xRUZbx.rst
deleted file mode 100644
index 07e1bf712db34..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-09-20-14-27-17.bpo-38237.xRUZbx.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-The arguments for the builtin pow function are more descriptive. They can now
-also be passed in as keywords.
diff --git a/Misc/NEWS.d/next/Library/2019-09-22-13-05-36.bpo-38248.Yo3N_1.rs=
t b/Misc/NEWS.d/next/Library/2019-09-22-13-05-36.bpo-38248.Yo3N_1.rst
deleted file mode 100644
index fc92209af3875..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-09-22-13-05-36.bpo-38248.Yo3N_1.rst
+++ /dev/null
@@ -1 +0,0 @@
-asyncio: Fix inconsistent immediate Task cancellation
diff --git a/Misc/NEWS.d/next/Library/2019-09-25-21-37-02.bpo-38108.Jr9HU6.rs=
t b/Misc/NEWS.d/next/Library/2019-09-25-21-37-02.bpo-38108.Jr9HU6.rst
deleted file mode 100644
index d7eea367e482b..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-09-25-21-37-02.bpo-38108.Jr9HU6.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Any synchronous magic methods on an AsyncMock now return a MagicMock. Any
-asynchronous magic methods on a MagicMock now return an AsyncMock.
diff --git a/Misc/NEWS.d/next/Library/2019-09-27-15-24-45.bpo-38216.-7yvZR.rs=
t b/Misc/NEWS.d/next/Library/2019-09-27-15-24-45.bpo-38216.-7yvZR.rst
deleted file mode 100644
index ac8e2b042d929..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-09-27-15-24-45.bpo-38216.-7yvZR.rst
+++ /dev/null
@@ -1,4 +0,0 @@
-Allow the rare code that wants to send invalid http requests from the
-`http.client` library a way to do so.  The fixes for bpo-30458 led to
-breakage for some projects that were relying on this ability to test their
-own behavior in the face of bad requests.
diff --git a/Misc/NEWS.d/next/Library/2019-09-27-16-31-28.bpo-38161.zehai1.rs=
t b/Misc/NEWS.d/next/Library/2019-09-27-16-31-28.bpo-38161.zehai1.rst
deleted file mode 100644
index 0077033c654f5..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-09-27-16-31-28.bpo-38161.zehai1.rst
+++ /dev/null
@@ -1 +0,0 @@
-Removes _AwaitEvent from AsyncMock.
diff --git a/Misc/NEWS.d/next/Library/2019-09-28-20-16-40.bpo-38163.x51-vK.rs=
t b/Misc/NEWS.d/next/Library/2019-09-28-20-16-40.bpo-38163.x51-vK.rst
deleted file mode 100644
index 5f7db26e05ce9..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-09-28-20-16-40.bpo-38163.x51-vK.rst
+++ /dev/null
@@ -1,4 +0,0 @@
-Child mocks will now detect their type as either synchronous or
-asynchronous, asynchronous child mocks will be AsyncMocks and synchronous
-child mocks will be either MagicMock or Mock (depending on their parent
-type).
diff --git a/Misc/NEWS.d/next/Library/2019-09-29-13-50-24.bpo-38019.6MoOE3.rs=
t b/Misc/NEWS.d/next/Library/2019-09-29-13-50-24.bpo-38019.6MoOE3.rst
deleted file mode 100644
index aa5a23de24609..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-09-29-13-50-24.bpo-38019.6MoOE3.rst
+++ /dev/null
@@ -1 +0,0 @@
-Correctly handle pause/resume reading of closed asyncio unix pipe.
diff --git a/Misc/NEWS.d/next/Library/2019-09-30-00-15-27.bpo-38242.uPIyAc.rs=
t b/Misc/NEWS.d/next/Library/2019-09-30-00-15-27.bpo-38242.uPIyAc.rst
deleted file mode 100644
index be9da891b866f..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-09-30-00-15-27.bpo-38242.uPIyAc.rst
+++ /dev/null
@@ -1 +0,0 @@
-Revert the new asyncio Streams API
diff --git a/Misc/NEWS.d/next/Library/2019-09-30-22-06-33.bpo-38319.5QjiDa.rs=
t b/Misc/NEWS.d/next/Library/2019-09-30-22-06-33.bpo-38319.5QjiDa.rst
deleted file mode 100644
index 376a9e459de27..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-09-30-22-06-33.bpo-38319.5QjiDa.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-sendfile() used in socket and shutil modules was raising OverflowError for
-files >=3D 2GiB on 32-bit architectures.  (patch by Giampaolo Rodola)
diff --git a/Misc/NEWS.d/next/Security/2019-08-27-01-13-05.bpo-37764.qv67PQ.r=
st b/Misc/NEWS.d/next/Security/2019-08-27-01-13-05.bpo-37764.qv67PQ.rst
deleted file mode 100644
index 27fa8e192f0c0..0000000000000
--- a/Misc/NEWS.d/next/Security/2019-08-27-01-13-05.bpo-37764.qv67PQ.rst
+++ /dev/null
@@ -1 +0,0 @@
-Fixes email._header_value_parser.get_unstructured going into an infinite loo=
p for a specific case in which the email header does not have trailing whites=
pace, and the case in which it contains an invalid encoded word. Patch by Ash=
win Ramaswami.
\ No newline at end of file
diff --git a/Misc/NEWS.d/next/Security/2019-09-23-21-02-46.bpo-38174.MeWuJd.r=
st b/Misc/NEWS.d/next/Security/2019-09-23-21-02-46.bpo-38174.MeWuJd.rst
deleted file mode 100644
index e3da6fcdd139f..0000000000000
--- a/Misc/NEWS.d/next/Security/2019-09-23-21-02-46.bpo-38174.MeWuJd.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Update vendorized expat library version to 2.2.8, which resolves
-CVE-2019-15903.
diff --git a/Misc/NEWS.d/next/Security/2019-09-25-13-21-09.bpo-38243.1pfz24.r=
st b/Misc/NEWS.d/next/Security/2019-09-25-13-21-09.bpo-38243.1pfz24.rst
deleted file mode 100644
index 98d7be129573a..0000000000000
--- a/Misc/NEWS.d/next/Security/2019-09-25-13-21-09.bpo-38243.1pfz24.rst
+++ /dev/null
@@ -1,3 +0,0 @@
-Escape the server title of :class:`xmlrpc.server.DocXMLRPCServer`
-when rendering the document page as HTML.
-(Contributed by Dong-hee Na in :issue:`38243`.)
diff --git a/Misc/NEWS.d/next/Tests/2018-09-07-01-18-27.bpo-34596.r2-EGd.rst =
b/Misc/NEWS.d/next/Tests/2018-09-07-01-18-27.bpo-34596.r2-EGd.rst
deleted file mode 100644
index 156e8aa8945de..0000000000000
--- a/Misc/NEWS.d/next/Tests/2018-09-07-01-18-27.bpo-34596.r2-EGd.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Fallback to a default reason when :func:`unittest.skip` is uncalled. Patch by
-Naitree Zhu.
diff --git a/Misc/NEWS.d/next/Tests/2019-05-28-15-41-34.bpo-36919.-vGt_m.rst =
b/Misc/NEWS.d/next/Tests/2019-05-28-15-41-34.bpo-36919.-vGt_m.rst
deleted file mode 100644
index 9400bdda5da18..0000000000000
--- a/Misc/NEWS.d/next/Tests/2019-05-28-15-41-34.bpo-36919.-vGt_m.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Make ``test_source_encoding.test_issue2301`` implementation independent. The
-test will work now for both CPython and IronPython.
diff --git a/Misc/NEWS.d/next/Tests/2019-06-03-20-47-10.bpo-34001.KvYx9z.rst =
b/Misc/NEWS.d/next/Tests/2019-06-03-20-47-10.bpo-34001.KvYx9z.rst
deleted file mode 100644
index 35c65a8fbf0fa..0000000000000
--- a/Misc/NEWS.d/next/Tests/2019-06-03-20-47-10.bpo-34001.KvYx9z.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Make test_ssl pass with LibreSSL. LibreSSL handles minimum and maximum TLS
-version differently than OpenSSL.
diff --git a/Misc/NEWS.d/next/Tests/2019-06-12-14-30-29.bpo-37252.4o-uLs.rst =
b/Misc/NEWS.d/next/Tests/2019-06-12-14-30-29.bpo-37252.4o-uLs.rst
deleted file mode 100644
index 1bd7d2872f13b..0000000000000
--- a/Misc/NEWS.d/next/Tests/2019-06-12-14-30-29.bpo-37252.4o-uLs.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Fix assertions in ``test_close`` and ``test_events_mask_overflow`` devpoll
-tests.
diff --git a/Misc/NEWS.d/next/Tests/2019-08-16-16-15-14.bpo-37876.m3k1w3.rst =
b/Misc/NEWS.d/next/Tests/2019-08-16-16-15-14.bpo-37876.m3k1w3.rst
deleted file mode 100644
index 45702fc635103..0000000000000
--- a/Misc/NEWS.d/next/Tests/2019-08-16-16-15-14.bpo-37876.m3k1w3.rst
+++ /dev/null
@@ -1 +0,0 @@
-Add tests for ROT-13 codec.
\ No newline at end of file
diff --git a/Misc/NEWS.d/next/Tests/2019-08-20-19-24-19.bpo-37531.wRoXfU.rst =
b/Misc/NEWS.d/next/Tests/2019-08-20-19-24-19.bpo-37531.wRoXfU.rst
deleted file mode 100644
index 59500ce67a01e..0000000000000
--- a/Misc/NEWS.d/next/Tests/2019-08-20-19-24-19.bpo-37531.wRoXfU.rst
+++ /dev/null
@@ -1,3 +0,0 @@
-Enhance regrtest multiprocess timeout: write a message when killing a worker
-process, catch popen.kill() and popen.wait() exceptions, put a timeout on the
-second call to popen.communicate().
diff --git a/Misc/NEWS.d/next/Tests/2019-09-11-17-22-32.bpo-38117.X7LgGY.rst =
b/Misc/NEWS.d/next/Tests/2019-09-11-17-22-32.bpo-38117.X7LgGY.rst
deleted file mode 100644
index d3280375516a0..0000000000000
--- a/Misc/NEWS.d/next/Tests/2019-09-11-17-22-32.bpo-38117.X7LgGY.rst
+++ /dev/null
@@ -1 +0,0 @@
-Test with OpenSSL 1.1.1d
diff --git a/Misc/NEWS.d/next/Tests/2019-09-24-12-24-05.bpo-38212.IWbhWz.rst =
b/Misc/NEWS.d/next/Tests/2019-09-24-12-24-05.bpo-38212.IWbhWz.rst
deleted file mode 100644
index 83a812d469981..0000000000000
--- a/Misc/NEWS.d/next/Tests/2019-09-24-12-24-05.bpo-38212.IWbhWz.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Multiprocessing tests: increase test_queue_feeder_donot_stop_onexc() timeout
-from 1 to 60 seconds.
diff --git a/Misc/NEWS.d/next/Tests/2019-09-24-12-30-55.bpo-37123.IoutBn.rst =
b/Misc/NEWS.d/next/Tests/2019-09-24-12-30-55.bpo-37123.IoutBn.rst
deleted file mode 100644
index 200a5c324d6de..0000000000000
--- a/Misc/NEWS.d/next/Tests/2019-09-24-12-30-55.bpo-37123.IoutBn.rst
+++ /dev/null
@@ -1,4 +0,0 @@
-Multiprocessing test test_mymanager() now also expects -SIGTERM, not only
-exitcode 0. BaseManager._finalize_manager() sends SIGTERM to the manager
-process if it takes longer than 1 second to stop, which happens on slow
-buildbots.
diff --git a/Misc/NEWS.d/next/Tests/2019-09-25-12-18-31.bpo-38270._x-9uH.rst =
b/Misc/NEWS.d/next/Tests/2019-09-25-12-18-31.bpo-38270._x-9uH.rst
deleted file mode 100644
index efb1b98b36d00..0000000000000
--- a/Misc/NEWS.d/next/Tests/2019-09-25-12-18-31.bpo-38270._x-9uH.rst
+++ /dev/null
@@ -1,4 +0,0 @@
-test.support now has a helper function to check for availibility of a
-hash digest function. Several tests are refactored avoid MD5 and use
-SHA256 instead. Other tests are marked to use MD5 and skipped when MD5 is
-disabled.
diff --git a/Misc/NEWS.d/next/Tests/2019-09-25-13-11-29.bpo-38271.iHXNIg.rst =
b/Misc/NEWS.d/next/Tests/2019-09-25-13-11-29.bpo-38271.iHXNIg.rst
deleted file mode 100644
index 8f43d321d371a..0000000000000
--- a/Misc/NEWS.d/next/Tests/2019-09-25-13-11-29.bpo-38271.iHXNIg.rst
+++ /dev/null
@@ -1,4 +0,0 @@
-The private keys for test_ssl were encrypted with 3DES in traditional
-PKCS#5 format. 3DES and the digest algorithm of PKCS#5 are blocked by
-some strict crypto policies. Use PKCS#8 format with AES256 encryption
-instead.
diff --git a/Misc/NEWS.d/next/Tests/2019-09-25-14-40-57.bpo-38275.-kdveI.rst =
b/Misc/NEWS.d/next/Tests/2019-09-25-14-40-57.bpo-38275.-kdveI.rst
deleted file mode 100644
index 893c0f137aeaf..0000000000000
--- a/Misc/NEWS.d/next/Tests/2019-09-25-14-40-57.bpo-38275.-kdveI.rst
+++ /dev/null
@@ -1,4 +0,0 @@
-test_ssl now handles disabled TLS/SSL versions better. OpenSSL's crypto
-policy and run-time settings are recognized and tests for disabled versions
-are skipped. Tests also accept more TLS minimum_versions for platforms that
-override OpenSSL's default with strict settings.
diff --git a/Misc/NEWS.d/next/Tests/2019-09-26-15-48-36.bpo-38239.MfoVzY.rst =
b/Misc/NEWS.d/next/Tests/2019-09-26-15-48-36.bpo-38239.MfoVzY.rst
deleted file mode 100644
index f79da29fa1828..0000000000000
--- a/Misc/NEWS.d/next/Tests/2019-09-26-15-48-36.bpo-38239.MfoVzY.rst
+++ /dev/null
@@ -1 +0,0 @@
-Fix test_gdb for Link Time Optimization (LTO) builds.
diff --git a/Misc/NEWS.d/next/Tools-Demos/2019-05-27-15-26-12.bpo-37064.k_SPW=
2.rst b/Misc/NEWS.d/next/Tools-Demos/2019-05-27-15-26-12.bpo-37064.k_SPW2.rst
deleted file mode 100644
index d1210e29532bc..0000000000000
--- a/Misc/NEWS.d/next/Tools-Demos/2019-05-27-15-26-12.bpo-37064.k_SPW2.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Add option -k to pathscript.py script: preserve shebang flags.
-Add option -a to pathscript.py script: add flags.
diff --git a/Misc/NEWS.d/next/Tools-Demos/2019-09-12-16-15-55.bpo-37803.chEiz=
y.rst b/Misc/NEWS.d/next/Tools-Demos/2019-09-12-16-15-55.bpo-37803.chEizy.rst
deleted file mode 100644
index 5c0eedb937096..0000000000000
--- a/Misc/NEWS.d/next/Tools-Demos/2019-09-12-16-15-55.bpo-37803.chEizy.rst
+++ /dev/null
@@ -1 +0,0 @@
-pdb's ``--help`` and ``--version`` long options now work.
diff --git a/Misc/NEWS.d/next/Windows/2019-05-10-15-25-44.bpo-32587.-0g2O3.rs=
t b/Misc/NEWS.d/next/Windows/2019-05-10-15-25-44.bpo-32587.-0g2O3.rst
deleted file mode 100644
index 41483aa8b74a9..0000000000000
--- a/Misc/NEWS.d/next/Windows/2019-05-10-15-25-44.bpo-32587.-0g2O3.rst
+++ /dev/null
@@ -1 +0,0 @@
-Make :data:`winreg.REG_MULTI_SZ` support zero-length strings.
diff --git a/Misc/NEWS.d/next/Windows/2019-06-25-04-15-22.bpo-37380.tPxjuz.rs=
t b/Misc/NEWS.d/next/Windows/2019-06-25-04-15-22.bpo-37380.tPxjuz.rst
deleted file mode 100644
index facce27954a63..0000000000000
--- a/Misc/NEWS.d/next/Windows/2019-06-25-04-15-22.bpo-37380.tPxjuz.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Don't collect unfinished processes with ``subprocess._active`` on Windows to
-cleanup later. Patch by Ruslan Kuprieiev.
diff --git a/Misc/NEWS.d/next/Windows/2019-06-28-18-10-29.bpo-37445.LsdYO6.rs=
t b/Misc/NEWS.d/next/Windows/2019-06-28-18-10-29.bpo-37445.LsdYO6.rst
deleted file mode 100644
index e4805b4e02fb1..0000000000000
--- a/Misc/NEWS.d/next/Windows/2019-06-28-18-10-29.bpo-37445.LsdYO6.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Include the ``FORMAT_MESSAGE_IGNORE_INSERTS`` flag in ``FormatMessageW()``
-calls.
diff --git a/Misc/NEWS.d/next/Windows/2019-07-29-16-49-31.bpo-37702.Lj2f5e.rs=
t b/Misc/NEWS.d/next/Windows/2019-07-29-16-49-31.bpo-37702.Lj2f5e.rst
deleted file mode 100644
index 67d53d4c46276..0000000000000
--- a/Misc/NEWS.d/next/Windows/2019-07-29-16-49-31.bpo-37702.Lj2f5e.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Fix memory leak on Windows in creating an SSLContext object or
-running urllib.request.urlopen('https://...').
\ No newline at end of file
diff --git a/Misc/NEWS.d/next/Windows/2019-08-30-15-15-22.bpo-37705.2o4NWW.rs=
t b/Misc/NEWS.d/next/Windows/2019-08-30-15-15-22.bpo-37705.2o4NWW.rst
deleted file mode 100644
index a374c3a2965a6..0000000000000
--- a/Misc/NEWS.d/next/Windows/2019-08-30-15-15-22.bpo-37705.2o4NWW.rst
+++ /dev/null
@@ -1 +0,0 @@
-Improve the implementation of ``winerror_to_errno()``.
diff --git a/Misc/NEWS.d/next/Windows/2019-09-03-11-47-37.bpo-38020.xFZ2j0.rs=
t b/Misc/NEWS.d/next/Windows/2019-09-03-11-47-37.bpo-38020.xFZ2j0.rst
deleted file mode 100644
index e6ddaba2f67b0..0000000000000
--- a/Misc/NEWS.d/next/Windows/2019-09-03-11-47-37.bpo-38020.xFZ2j0.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Fixes potential crash when calling :func:`os.readlink` (or indirectly
-through :func:`~os.path.realpath`) on a file that is not a supported link.
diff --git a/Misc/NEWS.d/next/Windows/2019-09-04-14-01-08.bpo-38030._USdtk.rs=
t b/Misc/NEWS.d/next/Windows/2019-09-04-14-01-08.bpo-38030._USdtk.rst
deleted file mode 100644
index f1be8a1e1c86f..0000000000000
--- a/Misc/NEWS.d/next/Windows/2019-09-04-14-01-08.bpo-38030._USdtk.rst
+++ /dev/null
@@ -1 +0,0 @@
-Fixes :func:`os.stat` failing for block devices on Windows
diff --git a/Misc/NEWS.d/next/Windows/2019-09-09-12-22-23.bpo-37283.8NvOkU.rs=
t b/Misc/NEWS.d/next/Windows/2019-09-09-12-22-23.bpo-37283.8NvOkU.rst
deleted file mode 100644
index 973047839301e..0000000000000
--- a/Misc/NEWS.d/next/Windows/2019-09-09-12-22-23.bpo-37283.8NvOkU.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Ensure command-line and unattend.xml setting override previously detected
-states in Windows installer.
diff --git a/Misc/NEWS.d/next/Windows/2019-09-10-14-17-25.bpo-38088.FOvWSM.rs=
t b/Misc/NEWS.d/next/Windows/2019-09-10-14-17-25.bpo-38088.FOvWSM.rst
deleted file mode 100644
index 37bdeeadd894a..0000000000000
--- a/Misc/NEWS.d/next/Windows/2019-09-10-14-17-25.bpo-38088.FOvWSM.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Fixes distutils not finding vcruntime140.dll with only the v142 toolset
-installed.
diff --git a/Misc/NEWS.d/next/Windows/2019-09-10-14-21-40.bpo-38087.--eIib.rs=
t b/Misc/NEWS.d/next/Windows/2019-09-10-14-21-40.bpo-38087.--eIib.rst
deleted file mode 100644
index ca625cc775a20..0000000000000
--- a/Misc/NEWS.d/next/Windows/2019-09-10-14-21-40.bpo-38087.--eIib.rst
+++ /dev/null
@@ -1 +0,0 @@
-Fix case sensitivity in test_pathlib and test_ntpath.
diff --git a/Misc/NEWS.d/next/Windows/2019-09-11-10-22-01.bpo-38081.8JhzjD.rs=
t b/Misc/NEWS.d/next/Windows/2019-09-11-10-22-01.bpo-38081.8JhzjD.rst
deleted file mode 100644
index e9d7a30eed9f9..0000000000000
--- a/Misc/NEWS.d/next/Windows/2019-09-11-10-22-01.bpo-38081.8JhzjD.rst
+++ /dev/null
@@ -1 +0,0 @@
-Prevent error calling :func:`os.path.realpath` on ``'NUL'``.
diff --git a/Misc/NEWS.d/next/Windows/2019-09-11-12-34-31.bpo-36634.xLaGgb.rs=
t b/Misc/NEWS.d/next/Windows/2019-09-11-12-34-31.bpo-36634.xLaGgb.rst
deleted file mode 100644
index c0bd2beaae2df..0000000000000
--- a/Misc/NEWS.d/next/Windows/2019-09-11-12-34-31.bpo-36634.xLaGgb.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-venv activate.bat now works when the existing variables contain double quote
-characters.
diff --git a/Misc/NEWS.d/next/Windows/2019-09-11-14-42-04.bpo-36634.8Un8ih.rs=
t b/Misc/NEWS.d/next/Windows/2019-09-11-14-42-04.bpo-36634.8Un8ih.rst
deleted file mode 100644
index 1affdd81cbfac..0000000000000
--- a/Misc/NEWS.d/next/Windows/2019-09-11-14-42-04.bpo-36634.8Un8ih.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-:func:`os.cpu_count` now returns active processors rather than maximum
-processors.
diff --git a/Misc/NEWS.d/next/Windows/2019-09-11-15-24-04.bpo-38114.cc0E5E.rs=
t b/Misc/NEWS.d/next/Windows/2019-09-11-15-24-04.bpo-38114.cc0E5E.rst
deleted file mode 100644
index 5dab8d728e881..0000000000000
--- a/Misc/NEWS.d/next/Windows/2019-09-11-15-24-04.bpo-38114.cc0E5E.rst
+++ /dev/null
@@ -1 +0,0 @@
-The ``pip.ini`` is no longer included in the Nuget package.
diff --git a/Misc/NEWS.d/next/Windows/2019-09-12-12-05-55.bpo-38133.yFeRGS.rs=
t b/Misc/NEWS.d/next/Windows/2019-09-12-12-05-55.bpo-38133.yFeRGS.rst
deleted file mode 100644
index 3fbf01693d550..0000000000000
--- a/Misc/NEWS.d/next/Windows/2019-09-12-12-05-55.bpo-38133.yFeRGS.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Allow py.exe launcher to locate installations from the Microsoft Store and
-improve display of active virtual environments.
diff --git a/Misc/NEWS.d/next/Windows/2019-09-13-14-11-42.bpo-38092.x31ehI.rs=
t b/Misc/NEWS.d/next/Windows/2019-09-13-14-11-42.bpo-38092.x31ehI.rst
deleted file mode 100644
index 48e83c8d15672..0000000000000
--- a/Misc/NEWS.d/next/Windows/2019-09-13-14-11-42.bpo-38092.x31ehI.rst
+++ /dev/null
@@ -1 +0,0 @@
-Reduce overhead when using multiprocessing in a Windows virtual environment.
diff --git a/Misc/NEWS.d/next/Windows/2019-09-16-14-07-11.bpo-38117.hJVf0C.rs=
t b/Misc/NEWS.d/next/Windows/2019-09-16-14-07-11.bpo-38117.hJVf0C.rst
deleted file mode 100644
index 4fd8a66d279e9..0000000000000
--- a/Misc/NEWS.d/next/Windows/2019-09-16-14-07-11.bpo-38117.hJVf0C.rst
+++ /dev/null
@@ -1 +0,0 @@
-Update bundled OpenSSL to 1.1.1d
diff --git a/Misc/NEWS.d/next/macOS/2019-09-10-14-24-35.bpo-38089.eedgyD.rst =
b/Misc/NEWS.d/next/macOS/2019-09-10-14-24-35.bpo-38089.eedgyD.rst
deleted file mode 100644
index 41349848621ec..0000000000000
--- a/Misc/NEWS.d/next/macOS/2019-09-10-14-24-35.bpo-38089.eedgyD.rst
+++ /dev/null
@@ -1 +0,0 @@
-Move Azure Pipelines to latest VM versions and make macOS tests optional
diff --git a/Misc/NEWS.d/next/macOS/2019-09-15-21-29-13.bpo-38117.ZLsoAZ.rst =
b/Misc/NEWS.d/next/macOS/2019-09-15-21-29-13.bpo-38117.ZLsoAZ.rst
deleted file mode 100644
index 644f147c95535..0000000000000
--- a/Misc/NEWS.d/next/macOS/2019-09-15-21-29-13.bpo-38117.ZLsoAZ.rst
+++ /dev/null
@@ -1 +0,0 @@
-Updated OpenSSL to 1.1.1d in macOS installer.
diff --git a/README.rst b/README.rst
index 52c79d5677034..6482de800f230 100644
--- a/README.rst
+++ b/README.rst
@@ -1,5 +1,5 @@
-This is Python version 3.8.0 beta 4
-=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
+This is Python version 3.8.0 release candidate 1
+=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
=20
 .. image:: https://travis-ci.org/python/cpython.svg?branch=3Dmaster
    :alt: CPython build status on Travis CI


From webhook-mailer at python.org  Sun Oct 13 16:35:25 2019
From: webhook-mailer at python.org (=?utf-8?q?=C5=81ukasz?= Langa)
Date: Sun, 13 Oct 2019 20:35:25 -0000
Subject: [Python-checkins] (no subject)
Message-ID: <mailman.578.1599367016.9553.python-checkins@python.org>




To: python-checkins at python.org
Subject: Rebased version of what's new PR (GH-16745) (#16748)
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

https://github.com/python/cpython/commit/02d6f4f9bcf147fa34a0f5e00ffba92225c0=
ced3
commit: 02d6f4f9bcf147fa34a0f5e00ffba92225c0ced3
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.co=
m>
committer: =C5=81ukasz Langa <lukasz at langa.pl>
date: 2019-10-13T22:35:10+02:00
summary:

Rebased version of what's new PR (GH-16745) (#16748)

* Use Unicode character for accent
* Various grammar fixes
* Sort library modules alphabetically; remove duplicated idlelib/IDLE section
(cherry picked from commit bb78f6cfa6f2b84fa4611d39c35baf1c7dce7f8d)

Co-authored-by: Andrew Kuchling <amk at amk.ca>

files:
M Doc/whatsnew/3.8.rst

diff --git a/Doc/whatsnew/3.8.rst b/Doc/whatsnew/3.8.rst
index 9bf028deaa9de..c53bffb08edea 100644
--- a/Doc/whatsnew/3.8.rst
+++ b/Doc/whatsnew/3.8.rst
@@ -82,7 +82,7 @@ Assignment expressions
 ----------------------
=20
 There is new syntax ``:=3D`` that assigns values to variables as part of a l=
arger
-expression. It is affectionately known as "walrus operator" due to
+expression. It is affectionately known as "the walrus operator" due to
 its resemblance to `the eyes and tusks of a walrus
 <https://en.wikipedia.org/wiki/Walrus#/media/File:Pacific_Walrus_-_Bull_(824=
7646168).jpg>`_.
=20
@@ -620,6 +620,16 @@ where the DLL is stored (if a full or partial path is us=
ed to load the initial
 DLL) and paths added by :func:`~os.add_dll_directory`.
=20
=20
+datetime
+--------
+
+Added new alternate constructors :meth:`datetime.date.fromisocalendar` and
+:meth:`datetime.datetime.fromisocalendar`, which construct :class:`date` and
+:class:`datetime` objects respectively from ISO year, week number, and weekd=
ay;
+these are the inverse of each class's ``isocalendar`` method.
+(Contributed by Paul Ganssle in :issue:`36004`.)
+
+
 functools
 ---------
=20
@@ -637,14 +647,12 @@ than as a function returning a decorator.  So both of t=
hese are now supported::
 (Contributed by Raymond Hettinger in :issue:`36772`.)
=20
=20
-datetime
---------
+gc
+--
=20
-Added new alternate constructors :meth:`datetime.date.fromisocalendar` and
-:meth:`datetime.datetime.fromisocalendar`, which construct :class:`date` and
-:class:`datetime` objects respectively from ISO year, week number and weekda=
y;
-these are the inverse of each class's ``isocalendar`` method.
-(Contributed by Paul Ganssle in :issue:`36004`.)
+:func:`~gc.get_objects` can now receive an optional *generation* parameter
+indicating a generation to get objects from. Contributed in
+:issue:`36016` by Pablo Galindo.
=20
=20
 gettext
@@ -654,6 +662,18 @@ Added :func:`~gettext.pgettext` and its variants.
 (Contributed by Franz Glasner, =C3=89ric Araujo, and Cheryl Sabella in :issu=
e:`2504`.)
=20
=20
+gzip
+----
+
+Added the *mtime* parameter to :func:`gzip.compress` for reproducible output.
+(Contributed by Guo Ci Teo in :issue:`34898`.)
+
+A :exc:`~gzip.BadGzipFile` exception is now raised instead of :exc:`OSError`
+for certain types of invalid or corrupt gzip files.
+(Contributed by Filip Gruszczy=C5=84ski, Michele Orr=C3=B9, and Zackery Spyt=
z in
+:issue:`6584`.)
+
+
 idlelib and IDLE
 ----------------
=20
@@ -704,44 +724,6 @@ fails. The exception is ignored silently by default in r=
elease build.
 (Contributed by Victor Stinner in :issue:`18748`.)
=20
=20
-gc
---
-
-:func:`~gc.get_objects` can now receive an optional *generation* parameter
-indicating a generation to get objects from. Contributed in
-:issue:`36016` by Pablo Galindo.
-
-
-gzip
-----
-
-Added the *mtime* parameter to :func:`gzip.compress` for reproducible output.
-(Contributed by Guo Ci Teo in :issue:`34898`.)
-
-A :exc:`~gzip.BadGzipFile` exception is now raised instead of :exc:`OSError`
-for certain types of invalid or corrupt gzip files.
-(Contributed by Filip Gruszczy=C5=84ski, Michele Orr=C3=B9, and Zackery Spyt=
z in
-:issue:`6584`.)
-
-
-idlelib and IDLE
-----------------
-
-Add optional line numbers for IDLE editor windows. Windows
-open without line numbers unless set otherwise in the General
-tab of the configuration dialog.
-(Contributed by Tal Einat and Saimadhav Heblikar in :issue:`17535`.)
-
-Output over N lines (50 by default) is squeezed down to a button.
-N can be changed in the PyShell section of the General page of the
-Settings dialog.  Fewer, but possibly extra long, lines can be squeezed by
-right clicking on the output.  Squeezed output can be expanded in place
-by double-clicking the button or into the clipboard or a separate window
-by right-clicking the button.  (Contributed by Tal Einat in :issue:`1529353`=
.)
-
-The changes above have been backported to 3.7 maintenance releases.
-
-
 json.tool
 ---------
=20
@@ -899,18 +881,6 @@ py_compile
 (Contributed by Joannah Nanjekye in :issue:`22640`.)
=20
=20
-socket
-------
-
-Added :meth:`~socket.create_server()` and :meth:`~socket.has_dualstack_ipv6(=
)`
-convenience functions to automate the necessary tasks usually involved when
-creating a server socket, including accepting both IPv4 and IPv6 connections
-on the same socket.  (Contributed by Giampaolo Rodola in :issue:`17561`.)
-
-The :func:`socket.if_nameindex()`, :func:`socket.if_nametoindex()`, and
-:func:`socket.if_indextoname()` functions have been implemented on Windows.
-(Contributed by Zackery Spytz in :issue:`37007`.)
-
 shlex
 ----------
=20
@@ -932,6 +902,19 @@ inherited from the corresponding change to the :mod:`tar=
file` module.
 recursively removing their contents first.
=20
=20
+socket
+------
+
+Added :meth:`~socket.create_server()` and :meth:`~socket.has_dualstack_ipv6(=
)`
+convenience functions to automate the necessary tasks usually involved when
+creating a server socket, including accepting both IPv4 and IPv6 connections
+on the same socket.  (Contributed by Giampaolo Rodol=C3=A0 in :issue:`17561`=
.)
+
+The :func:`socket.if_nameindex()`, :func:`socket.if_nametoindex()`, and
+:func:`socket.if_indextoname()` functions have been implemented on Windows.
+(Contributed by Zackery Spytz in :issue:`37007`.)
+
+
 ssl
 ---
=20
@@ -1202,14 +1185,14 @@ Optimizations
   +26% on Linux, +50% on macOS and +40% on Windows. Also, much less CPU cycl=
es
   are consumed.
   See :ref:`shutil-platform-dependent-efficient-copy-operations` section.
-  (Contributed by Giampaolo Rodola' in :issue:`33671`.)
+  (Contributed by Giampaolo Rodol=C3=A0 in :issue:`33671`.)
=20
 * :func:`shutil.copytree` uses :func:`os.scandir` function and all copy
   functions depending from it use cached :func:`os.stat` values. The speedup
   for copying a directory with 8000 files is around +9% on Linux, +20% on
   Windows and +30% on a Windows SMB share. Also the number of :func:`os.stat`
   syscalls is reduced by 38% making :func:`shutil.copytree` especially faster
-  on network filesystems. (Contributed by Giampaolo Rodola' in :issue:`33695=
`.)
+  on network filesystems. (Contributed by Giampaolo Rodol=C3=A0 in :issue:`3=
3695`.)
=20
 * The default protocol in the :mod:`pickle` module is now Protocol 4,
   first introduced in Python 3.4.  It offers better performance and smaller
@@ -1361,7 +1344,7 @@ Deprecated
   (Contributed by Victor Stinner in :issue:`37481`.)
=20
 * Deprecated methods ``getchildren()`` and ``getiterator()`` in
-  the :mod:`~xml.etree.ElementTree` module emit now a
+  the :mod:`~xml.etree.ElementTree` module now emit a
   :exc:`DeprecationWarning` instead of :exc:`PendingDeprecationWarning`.
   They will be removed in Python 3.9.
   (Contributed by Serhiy Storchaka in :issue:`29209`.)
@@ -1456,28 +1439,29 @@ The following features and APIs have been removed fro=
m Python 3.8:
 * The :mod:`macpath` module, deprecated in Python 3.7, has been removed.
   (Contributed by Victor Stinner in :issue:`35471`.)
=20
-* The function :func:`platform.popen` has been removed, it was deprecated si=
nce
-  Python 3.3: use :func:`os.popen` instead.
+* The function :func:`platform.popen` has been removed, after having been
+  deprecated since Python 3.3: use :func:`os.popen` instead.
   (Contributed by Victor Stinner in :issue:`35345`.)
=20
-* The function :func:`time.clock` has been removed, it was deprecated since =
Python
-  3.3: use :func:`time.perf_counter` or :func:`time.process_time` instead, d=
epending
-  on your requirements, to have a well defined behavior.
+* The function :func:`time.clock` has been removed, after having been
+  deprecated since Python 3.3: use :func:`time.perf_counter` or
+  :func:`time.process_time` instead, depending
+  on your requirements, to have well-defined behavior.
   (Contributed by Matthias Bussonnier in :issue:`36895`.)
=20
 * The ``pyvenv`` script has been removed in favor of ``python3.8 -m venv``
   to help eliminate confusion as to what Python interpreter the ``pyvenv``
   script is tied to. (Contributed by Brett Cannon in :issue:`25427`.)
=20
-* ``parse_qs``, ``parse_qsl``, and ``escape`` are removed from :mod:`cgi`
-  module.  They are deprecated from Python 3.2 or older. They should be impo=
rted
+* ``parse_qs``, ``parse_qsl``, and ``escape`` are removed from the :mod:`cgi`
+  module.  They are deprecated in Python 3.2 or older. They should be import=
ed
   from the ``urllib.parse`` and ``html`` modules instead.
=20
-* ``filemode`` function is removed from :mod:`tarfile` module.
+* ``filemode`` function is removed from the :mod:`tarfile` module.
   It is not documented and deprecated since Python 3.3.
=20
 * The :class:`~xml.etree.ElementTree.XMLParser` constructor no longer accepts
-  the *html* argument.  It never had effect and was deprecated in Python 3.4.
+  the *html* argument.  It never had an effect and was deprecated in Python =
3.4.
   All other parameters are now :ref:`keyword-only <keyword-only_parameter>`.
   (Contributed by Serhiy Storchaka in :issue:`29209`.)
=20
@@ -1536,8 +1520,8 @@ Changes in Python behavior
=20
 * On AIX, :attr:`sys.platform` doesn't contain the major version anymore.
   It is always ``'aix'``, instead of ``'aix3'`` .. ``'aix7'``.  Since
-  older Python versions include the version number, it is recommended to
-  always use the ``sys.platform.startswith('aix')``.
+  older Python versions include the version number, so it is recommended to
+  always use ``sys.platform.startswith('aix')``.
   (Contributed by M. Felt in :issue:`36588`.)
=20
 * :c:func:`PyEval_AcquireLock` and :c:func:`PyEval_AcquireThread` now
@@ -1557,8 +1541,8 @@ Changes in the Python API
=20
 * :class:`subprocess.Popen` can now use :func:`os.posix_spawn` in some cases
   for better performance. On Windows Subsystem for Linux and QEMU User
-  Emulation, Popen constructor using :func:`os.posix_spawn` no longer raise =
an
-  exception on errors like missing program, but the child process fails with=
 a
+  Emulation, the :class:`Popen` constructor using :func:`os.posix_spawn` no =
longer raises an
+  exception on errors like "missing program".  Instead the child process fai=
ls with a
   non-zero :attr:`~Popen.returncode`.
   (Contributed by Joannah Nanjekye and Victor Stinner in :issue:`35537`.)
=20
@@ -1571,7 +1555,7 @@ Changes in the Python API
 * The :meth:`imap.IMAP4.logout` method no longer ignores silently arbitrary
   exceptions.
=20
-* The function :func:`platform.popen` has been removed, it was deprecated si=
nce
+* The function :func:`platform.popen` has been removed, after having been de=
precated since
   Python 3.3: use :func:`os.popen` instead.
   (Contributed by Victor Stinner in :issue:`35345`.)
=20
@@ -1586,9 +1570,9 @@ Changes in the Python API
   specialized methods like :meth:`~tkinter.ttk.Treeview.selection_set` for
   changing the selection.  (Contributed by Serhiy Storchaka in :issue:`31508=
`.)
=20
-* The :meth:`writexml`, :meth:`toxml` and :meth:`toprettyxml` methods of the
-  :mod:`xml.dom.minidom` module, and :mod:`xml.etree` now preserve the attri=
bute
-  order specified by the user.
+* The :meth:`writexml`, :meth:`toxml` and :meth:`toprettyxml` methods of
+  :mod:`xml.dom.minidom`, and the :meth:`write` method of :mod:`xml.etree`,
+  now preserve the attribute order specified by the user.
   (Contributed by Diego Rojas and Raymond Hettinger in :issue:`34160`.)
=20
 * A :mod:`dbm.dumb` database opened with flags ``'r'`` is now read-only.
@@ -1598,7 +1582,7 @@ Changes in the Python API
=20
 * The ``doctype()`` method defined in a subclass of
   :class:`~xml.etree.ElementTree.XMLParser` will no longer be called and will
-  cause emitting a :exc:`RuntimeWarning` instead of a :exc:`DeprecationWarni=
ng`.
+  emit a :exc:`RuntimeWarning` instead of a :exc:`DeprecationWarning`.
   Define the :meth:`doctype() <xml.etree.ElementTree.TreeBuilder.doctype>`
   method on a target for handling an XML doctype declaration.
   (Contributed by Serhiy Storchaka in :issue:`29209`.)
@@ -1641,7 +1625,7 @@ Changes in the Python API
=20
 * The :meth:`mmap.flush() <mmap.mmap.flush>` method now returns ``None`` on
   success and raises an exception on error under all platforms.  Previously,
-  its behavior was platform-depended: a nonzero value was returned on succes=
s;
+  its behavior was platform-dependent: a nonzero value was returned on succe=
ss;
   zero was returned on error under Windows.  A zero value was returned on
   success; an exception was raised on error under Unix.
   (Contributed by Berker Peksag in :issue:`2122`.)
@@ -1685,7 +1669,7 @@ Changes in the Python API
 * :class:`types.CodeType` has a new parameter in the second position of the
   constructor (*posonlyargcount*) to support positional-only arguments defin=
ed
   in :pep:`570`. The first argument (*argcount*) now represents the total
-  number of positional arguments (including positional-only arguments). A new
+  number of positional arguments (including positional-only arguments). The =
new
   ``replace()`` method of :class:`types.CodeType` can be used to make the co=
de
   future-proof.
=20
@@ -1693,9 +1677,9 @@ Changes in the Python API
 Changes in the C API
 --------------------
=20
-* The :c:type:`PyCompilerFlags` structure gets a new *cf_feature_version*
+* The :c:type:`PyCompilerFlags` structure got a new *cf_feature_version*
   field. It should be initialized to ``PY_MINOR_VERSION``. The field is igno=
red
-  by default, it is used if and only if ``PyCF_ONLY_AST`` flag is set in
+  by default, and is used if and only if ``PyCF_ONLY_AST`` flag is set in
   *cf_flags*.
=20
 * The :c:func:`PyEval_ReInitThreads` function has been removed from the C AP=
I.


From webhook-mailer at python.org  Mon Oct 14 07:40:19 2019
From: webhook-mailer at python.org (=?utf-8?q?=C5=81ukasz?= Langa)
Date: Mon, 14 Oct 2019 11:40:19 -0000
Subject: [Python-checkins] (no subject)
Message-ID: <mailman.579.1599367017.9553.python-checkins@python.org>




To: python-checkins at python.org
Subject: bpo-38469: Handle named expression scope with global/nonlocal
 keywords (GH-16755) (#16760)
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

https://github.com/python/cpython/commit/927f07c816aad9f8ed961d7ace6a367837f7=
fa8f
commit: 927f07c816aad9f8ed961d7ace6a367837f7fa8f
branch: 3.8
author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.co=
m>
committer: =C5=81ukasz Langa <lukasz at langa.pl>
date: 2019-10-14T13:40:15+02:00
summary:

bpo-38469: Handle named expression scope with global/nonlocal keywords (GH-16=
755) (#16760)

The symbol table handing of PEP572's assignment expressions is not resolving =
correctly the scope of some variables in presence of global/nonlocal keywords=
 in conjunction with comprehensions.
(cherry picked from commit fd5c414880b2e05720b9cf14ab0b0d7ae2b7d925)

Co-authored-by: Pablo Galindo <Pablogsal at gmail.com>

files:
A Misc/NEWS.d/next/Core and Builtins/2019-10-13-23-41-38.bpo-38469.9kmuQj.rst
M Lib/test/test_named_expressions.py
M Python/symtable.c

diff --git a/Lib/test/test_named_expressions.py b/Lib/test/test_named_express=
ions.py
index b1027ce78006d..01e26c8dfaf25 100644
--- a/Lib/test/test_named_expressions.py
+++ b/Lib/test/test_named_expressions.py
@@ -1,6 +1,7 @@
 import os
 import unittest
=20
+GLOBAL_VAR =3D None
=20
 class NamedExpressionInvalidTest(unittest.TestCase):
=20
@@ -470,5 +471,49 @@ def test_named_expression_variable_reuse_in_comprehensio=
ns(self):
                 self.assertEqual(ns["x"], 2)
                 self.assertEqual(ns["result"], [0, 1, 2])
=20
+    def test_named_expression_global_scope(self):
+        sentinel =3D object()
+        global GLOBAL_VAR
+        def f():
+            global GLOBAL_VAR
+            [GLOBAL_VAR :=3D sentinel for _ in range(1)]
+            self.assertEqual(GLOBAL_VAR, sentinel)
+        try:
+            f()
+            self.assertEqual(GLOBAL_VAR, sentinel)
+        finally:
+            GLOBAL_VAR =3D None
+
+    def test_named_expression_global_scope_no_global_keyword(self):
+        sentinel =3D object()
+        def f():
+            GLOBAL_VAR =3D None
+            [GLOBAL_VAR :=3D sentinel for _ in range(1)]
+            self.assertEqual(GLOBAL_VAR, sentinel)
+        f()
+        self.assertEqual(GLOBAL_VAR, None)
+
+    def test_named_expression_nonlocal_scope(self):
+        sentinel =3D object()
+        def f():
+            nonlocal_var =3D None
+            def g():
+                nonlocal nonlocal_var
+                [nonlocal_var :=3D sentinel for _ in range(1)]
+            g()
+            self.assertEqual(nonlocal_var, sentinel)
+        f()
+
+    def test_named_expression_nonlocal_scope_no_nonlocal_keyword(self):
+        sentinel =3D object()
+        def f():
+            nonlocal_var =3D None
+            def g():
+                [nonlocal_var :=3D sentinel for _ in range(1)]
+            g()
+            self.assertEqual(nonlocal_var, None)
+        f()
+
+
 if __name__ =3D=3D "__main__":
     unittest.main()
diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-10-13-23-41-38.bpo-38469=
.9kmuQj.rst b/Misc/NEWS.d/next/Core and Builtins/2019-10-13-23-41-38.bpo-3846=
9.9kmuQj.rst
new file mode 100644
index 0000000000000..328a1b70825af
--- /dev/null
+++ b/Misc/NEWS.d/next/Core and Builtins/2019-10-13-23-41-38.bpo-38469.9kmuQj=
.rst=09
@@ -0,0 +1,2 @@
+Fixed a bug where the scope of named expressions was not being resolved
+correctly in the presence of the *global* keyword. Patch by Pablo Galindo.
diff --git a/Python/symtable.c b/Python/symtable.c
index f2453db69dd7d..b8713588b9a91 100644
--- a/Python/symtable.c
+++ b/Python/symtable.c
@@ -1467,10 +1467,16 @@ symtable_extend_namedexpr_scope(struct symtable *st, =
expr_ty e)
             continue;
         }
=20
-        /* If we find a FunctionBlock entry, add as NONLOCAL/LOCAL */
+        /* If we find a FunctionBlock entry, add as GLOBAL/LOCAL or NONLOCAL=
/LOCAL */
         if (ste->ste_type =3D=3D FunctionBlock) {
-            if (!symtable_add_def(st, target_name, DEF_NONLOCAL))
-                VISIT_QUIT(st, 0);
+            long target_in_scope =3D _PyST_GetSymbol(ste, target_name);
+            if (target_in_scope & DEF_GLOBAL) {
+                if (!symtable_add_def(st, target_name, DEF_GLOBAL))
+                    VISIT_QUIT(st, 0);
+            } else {
+                if (!symtable_add_def(st, target_name, DEF_NONLOCAL))
+                    VISIT_QUIT(st, 0);
+            }
             if (!symtable_record_directive(st, target_name, e->lineno, e->co=
l_offset))
                 VISIT_QUIT(st, 0);
=20


From webhook-mailer at python.org  Mon Oct 14 09:10:44 2019
From: webhook-mailer at python.org (=?utf-8?q?=C5=81ukasz?= Langa)
Date: Mon, 14 Oct 2019 13:10:44 -0000
Subject: [Python-checkins] (no subject)
Message-ID: <mailman.580.1599367018.9553.python-checkins@python.org>




To: python-checkins at python.org
Subject: bpo-37759: Polish What's New in Python 3.8. (#16769)
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

https://github.com/python/cpython/commit/298439ce3fb01de6c3110cc4847e6afe3253=
ba7a
commit: 298439ce3fb01de6c3110cc4847e6afe3253ba7a
branch: master
author: Serhiy Storchaka <storchaka at gmail.com>
committer: =C5=81ukasz Langa <lukasz at langa.pl>
date: 2019-10-14T15:10:40+02:00
summary:

bpo-37759: Polish What's New in Python 3.8. (#16769)

files:
M Doc/whatsnew/3.8.rst

diff --git a/Doc/whatsnew/3.8.rst b/Doc/whatsnew/3.8.rst
index 0c59ef8015a58..8108fb7e14704 100644
--- a/Doc/whatsnew/3.8.rst
+++ b/Doc/whatsnew/3.8.rst
@@ -212,6 +212,7 @@ subdirectories).
=20
 (Contributed by Carl Meyer in :issue:`33499`.)
=20
+
 Debug build uses the same ABI as release build
 -----------------------------------------------
=20
@@ -285,6 +286,7 @@ calculations can be shown::
=20
 (Contributed by Eric V. Smith and Larry Hastings in :issue:`36817`.)
=20
+
 PEP 587: Python Initialization Configuration
 --------------------------------------------
=20
@@ -411,7 +413,8 @@ Other Language Changes
 * The syntax allowed for keyword names in function calls was further
   restricted. In particular, ``f((keyword)=3Darg)`` is no longer allowed. It=
 was
   never intended to permit more than a bare name on the left-hand side of a
-  keyword argument assignment term. See :issue:`34641`.
+  keyword argument assignment term.
+  (Contributed by Benjamin Peterson in :issue:`34641`.)
=20
 * Generalized iterable unpacking in :keyword:`yield` and
   :keyword:`return` statements no longer requires enclosing parentheses.
@@ -425,7 +428,6 @@ Other Language Changes
     >>> parse('simpsons homer marge bart lisa sally')
     ('SIMPSONS', 'homer', 'marge', 'bart', 'lisa', 'sally')
=20
-
   (Contributed by David Cuthbert and Jordan Chapman in :issue:`32117`.)
=20
 * When a comma is missed in code such as ``[(10, 20) (30, 40)]``, the
@@ -521,6 +523,8 @@ Other Language Changes
      '=C5=82ukasz langa': '=C5=81ukasz Langa',
      'walter d=C3=B6rwald': 'Walter D=C3=B6rwald'}
=20
+  (Contributed by J=C3=B6rn Heissler in :issue:`35224`.)
+
=20
 New Modules
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
@@ -544,13 +548,12 @@ New Modules
      PackagePath('requests-2.22.0.dist-info/RECORD'),
      PackagePath('requests-2.22.0.dist-info/WHEEL')]
=20
-  (Contributed in :issue:`34632` by Barry Warsaw and Jason R. Coombs.)
+  (Contributed by Barry Warsaw and Jason R. Coombs in :issue:`34632`.)
=20
=20
 Improved Modules
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
=20
-
 ast
 ---
=20
@@ -558,6 +561,11 @@ AST nodes now have ``end_lineno`` and ``end_col_offset``=
 attributes,
 which give the precise location of the end of the node.  (This only
 applies to nodes that have ``lineno`` and ``col_offset`` attributes.)
=20
+New function :func:`ast.get_source_segment` returns the source code
+for a specific AST node.
+
+(Contributed by Ivan Levkivskyi in :issue:`33416`.)
+
 The :func:`ast.parse` function has some new flags:
=20
 * ``type_comments=3DTrue`` causes it to return the text of :pep:`484` and
@@ -570,8 +578,7 @@ The :func:`ast.parse` function has some new flags:
   version. (For example, ``feature_version=3D(3, 4)`` will treat
   ``async`` and ``await`` as non-reserved words.)
=20
-New function :func:`ast.get_source_segment` returns the source code
-for a specific AST node.
+(Contributed by Guido van Rossum in :issue:`35766`.)
=20
=20
 asyncio
@@ -596,7 +603,6 @@ The :func:`compile` built-in has been improved to accept =
the
 :func:`compile` will allow top-level ``await``, ``async for`` and ``async wi=
th``
 constructs that are usually considered invalid syntax. Asynchronous code obj=
ect
 marked with the ``CO_COROUTINE`` flag may then be returned.
-
 (Contributed by Matthias Bussonnier in :issue:`34616`)
=20
=20
@@ -627,6 +633,7 @@ to specify flags for the underlying ``LoadLibraryEx`` cal=
l. The default flags ar
 set to only load DLL dependencies from trusted locations, including the path
 where the DLL is stored (if a full or partial path is used to load the initi=
al
 DLL) and paths added by :func:`~os.add_dll_directory`.
+(Contributed by Steve Dower in :issue:`36085`.)
=20
=20
 datetime
@@ -660,8 +667,8 @@ gc
 --
=20
 :func:`~gc.get_objects` can now receive an optional *generation* parameter
-indicating a generation to get objects from. Contributed in
-:issue:`36016` by Pablo Galindo.
+indicating a generation to get objects from.
+(Contributed by Pablo Galindo in :issue:`36016`.)
=20
=20
 gettext
@@ -723,6 +730,8 @@ for :func:`property`, :func:`classmethod`, and :func:`sta=
ticmethod`::
           self.bit_rate =3D round(bit_rate / 1000.0, 1)
           self.duration =3D ceil(duration)
=20
+(Contributed by Raymond Hettinger in :issue:`36326`.)
+
=20
 io
 --
@@ -780,7 +789,7 @@ multiprocessing
 ---------------
=20
 Added new :mod:`multiprocessing.shared_memory` module.
-(Contributed Davin Potts in :issue:`35813`.)
+(Contributed by Davin Potts in :issue:`35813`.)
=20
 On macOS, the *spawn* start method is now used by default.
 (Contributed by Victor Stinner in :issue:`33725`.)
@@ -792,6 +801,7 @@ os
 Added new function :func:`~os.add_dll_directory` on Windows for providing
 additional search paths for native dependencies when importing extension
 modules or loading DLLs using :mod:`ctypes`.
+(Contributed by Steve Dower in :issue:`36085`.)
=20
 A new :func:`os.memfd_create` function was added to wrap the
 ``memfd_create()`` syscall.
@@ -812,6 +822,8 @@ and so code that checks ``islink`` first will continue to=
 treat junctions as
 directories, while code that handles errors from :func:`os.readlink` may now
 treat junctions as links.
=20
+(Contributed by Steve Dower in :issue:`37834`.)
+
=20
 os.path
 -------
@@ -827,6 +839,7 @@ characters or bytes unrepresentable at the OS level.
 :func:`~os.path.expanduser` on Windows now prefers the :envvar:`USERPROFILE`
 environment variable and does not use :envvar:`HOME`, which is not normally =
set
 for regular user accounts.
+(Contributed by Anthony Sottile in :issue:`36264`.)
=20
 :func:`~os.path.isdir` on Windows no longer returns true for a link to a
 non-existent directory.
@@ -834,6 +847,8 @@ non-existent directory.
 :func:`~os.path.realpath` on Windows now resolves reparse points, including
 symlinks and directory junctions.
=20
+(Contributed by Steve Dower in :issue:`37834`.)
+
=20
 pathlib
 -------
@@ -883,11 +898,12 @@ py_compile
=20
=20
 shlex
-----------
+-----
=20
 The new :func:`shlex.join` function acts as the inverse of :func:`shlex.spli=
t`.
 (Contributed by Bo Bayles in :issue:`32102`.)
=20
+
 shutil
 ------
=20
@@ -901,6 +917,7 @@ inherited from the corresponding change to the :mod:`tarf=
ile` module.
=20
 :func:`shutil.rmtree` on Windows now removes directory junctions without
 recursively removing their contents first.
+(Contributed by Steve Dower in :issue:`37834`.)
=20
=20
 socket
@@ -995,18 +1012,18 @@ in a standardized and extensible format, and offers se=
veral other benefits.
 threading
 ---------
=20
-* Add a new :func:`threading.excepthook` function which handles uncaught
-  :meth:`threading.Thread.run` exception. It can be overridden to control how
-  uncaught :meth:`threading.Thread.run` exceptions are handled.
-  (Contributed by Victor Stinner in :issue:`1230540`.)
+Add a new :func:`threading.excepthook` function which handles uncaught
+:meth:`threading.Thread.run` exception. It can be overridden to control how
+uncaught :meth:`threading.Thread.run` exceptions are handled.
+(Contributed by Victor Stinner in :issue:`1230540`.)
=20
-* Add a new
-  :func:`threading.get_native_id` function and a :data:`~threading.Thread.na=
tive_id`
-  attribute to the :class:`threading.Thread` class. These return the native
-  integral Thread ID of the current thread assigned by the kernel.
-  This feature is only available on certain platforms, see
-  :func:`get_native_id <threading.get_native_id>` for more information.
-  (Contributed by Jake Tesler in :issue:`36084`.)
+Add a new :func:`threading.get_native_id` function and
+a :data:`~threading.Thread.native_id`
+attribute to the :class:`threading.Thread` class. These return the native
+integral Thread ID of the current thread assigned by the kernel.
+This feature is only available on certain platforms, see
+:func:`get_native_id <threading.get_native_id>` for more information.
+(Contributed by Jake Tesler in :issue:`36084`.)
=20
=20
 tokenize
@@ -1017,6 +1034,7 @@ provided with input that does not have a trailing new l=
ine.  This behavior
 now matches what the C tokenizer does internally.
 (Contributed by Ammar Askar in :issue:`33899`.)
=20
+
 tkinter
 -------
=20
@@ -1036,6 +1054,7 @@ The :class:`tkinter.PhotoImage` class now has
 :meth:`~tkinter.PhotoImage.transparency_set` methods.  (Contributed by
 Zackery Spytz in :issue:`25451`.)
=20
+
 time
 ----
=20
@@ -1084,95 +1103,98 @@ The :mod:`typing` module incorporates several new fea=
tures:
 unicodedata
 -----------
=20
-* The :mod:`unicodedata` module has been upgraded to use the `Unicode 12.1.0
-  <http://blog.unicode.org/2019/05/unicode-12-1-en.html>`_ release.
+The :mod:`unicodedata` module has been upgraded to use the `Unicode 12.1.0
+<http://blog.unicode.org/2019/05/unicode-12-1-en.html>`_ release.
=20
-* New function :func:`~unicodedata.is_normalized` can be used to verify a st=
ring
-  is in a specific normal form, often much faster than by actually normalizi=
ng
-  the string.  (Contributed by Max Belanger, David Euresti, and Greg Price in
-  :issue:`32285` and :issue:`37966`).
+New function :func:`~unicodedata.is_normalized` can be used to verify a stri=
ng
+is in a specific normal form, often much faster than by actually normalizing
+the string.  (Contributed by Max Belanger, David Euresti, and Greg Price in
+:issue:`32285` and :issue:`37966`).
=20
=20
 unittest
 --------
=20
-* Added :class:`AsyncMock` to support an asynchronous version of :class:`Moc=
k`.
-  Appropriate new assert functions for testing have been added as well.
-  (Contributed by Lisa Roach in :issue:`26467`).
+Added :class:`AsyncMock` to support an asynchronous version of :class:`Mock`.
+Appropriate new assert functions for testing have been added as well.
+(Contributed by Lisa Roach in :issue:`26467`).
=20
-* Added :func:`~unittest.addModuleCleanup()` and
-  :meth:`~unittest.TestCase.addClassCleanup()` to unittest to support
-  cleanups for :func:`~unittest.setUpModule()` and
-  :meth:`~unittest.TestCase.setUpClass()`.
-  (Contributed by Lisa Roach in :issue:`24412`.)
+Added :func:`~unittest.addModuleCleanup()` and
+:meth:`~unittest.TestCase.addClassCleanup()` to unittest to support
+cleanups for :func:`~unittest.setUpModule()` and
+:meth:`~unittest.TestCase.setUpClass()`.
+(Contributed by Lisa Roach in :issue:`24412`.)
=20
-* Several mock assert functions now also print a list of actual calls upon
-  failure. (Contributed by Petter Strandmark in :issue:`35047`.)
+Several mock assert functions now also print a list of actual calls upon
+failure. (Contributed by Petter Strandmark in :issue:`35047`.)
=20
-* :mod:`unittest` module gained support for coroutines to be used as test ca=
ses
-  with :class:`unittest.IsolatedAsyncioTestCase`.
-  (Contributed by Andrew Svetlov in :issue:`32972`.)
+:mod:`unittest` module gained support for coroutines to be used as test cases
+with :class:`unittest.IsolatedAsyncioTestCase`.
+(Contributed by Andrew Svetlov in :issue:`32972`.)
=20
-  Example::
+Example::
=20
-    import unittest
+   import unittest
=20
=20
-    class TestRequest(unittest.IsolatedAsyncioTestCase):
+   class TestRequest(unittest.IsolatedAsyncioTestCase):
=20
-        async def asyncSetUp(self):
-            self.connection =3D await AsyncConnection()
+       async def asyncSetUp(self):
+           self.connection =3D await AsyncConnection()
=20
-        async def test_get(self):
-            response =3D await self.connection.get("https://example.com")
-            self.assertEqual(response.status_code, 200)
+       async def test_get(self):
+           response =3D await self.connection.get("https://example.com")
+           self.assertEqual(response.status_code, 200)
=20
-        async def asyncTearDown(self):
-            await self.connection.close()
+       async def asyncTearDown(self):
+           await self.connection.close()
=20
=20
-    if __name__ =3D=3D "__main__":
-        unittest.main()
+   if __name__ =3D=3D "__main__":
+       unittest.main()
=20
=20
 venv
 ----
=20
-* :mod:`venv` now includes an ``Activate.ps1`` script on all platforms for
-  activating virtual environments under PowerShell Core 6.1.
-  (Contributed by Brett Cannon in :issue:`32718`.)
+:mod:`venv` now includes an ``Activate.ps1`` script on all platforms for
+activating virtual environments under PowerShell Core 6.1.
+(Contributed by Brett Cannon in :issue:`32718`.)
+
=20
 weakref
 -------
=20
-* The proxy objects returned by :func:`weakref.proxy` now support the matrix
-  multiplication operators ``@`` and ``@=3D`` in addition to the other
-  numeric operators. (Contributed by Mark Dickinson in :issue:`36669`.)
+The proxy objects returned by :func:`weakref.proxy` now support the matrix
+multiplication operators ``@`` and ``@=3D`` in addition to the other
+numeric operators. (Contributed by Mark Dickinson in :issue:`36669`.)
+
=20
 xml
 ---
=20
-* As mitigation against DTD and external entity retrieval, the
-  :mod:`xml.dom.minidom` and :mod:`xml.sax` modules no longer process
-  external entities by default.
-  (Contributed by Christian Heimes in :issue:`17239`.)
+As mitigation against DTD and external entity retrieval, the
+:mod:`xml.dom.minidom` and :mod:`xml.sax` modules no longer process
+external entities by default.
+(Contributed by Christian Heimes in :issue:`17239`.)
=20
-* The ``.find*()`` methods in the :mod:`xml.etree.ElementTree` module
-  support wildcard searches like ``{*}tag`` which ignores the namespace
-  and ``{namespace}*`` which returns all tags in the given namespace.
-  (Contributed by Stefan Behnel in :issue:`28238`.)
+The ``.find*()`` methods in the :mod:`xml.etree.ElementTree` module
+support wildcard searches like ``{*}tag`` which ignores the namespace
+and ``{namespace}*`` which returns all tags in the given namespace.
+(Contributed by Stefan Behnel in :issue:`28238`.)
=20
-* The :mod:`xml.etree.ElementTree` module provides a new function
-  :func:`=E2=80=93xml.etree.ElementTree.canonicalize()` that implements C14N=
 2.0.
-  (Contributed by Stefan Behnel in :issue:`13611`.)
+The :mod:`xml.etree.ElementTree` module provides a new function
+:func:`=E2=80=93xml.etree.ElementTree.canonicalize()` that implements C14N 2=
.0.
+(Contributed by Stefan Behnel in :issue:`13611`.)
+
+The target object of :class:`xml.etree.ElementTree.XMLParser` can
+receive namespace declaration events through the new callback methods
+``start_ns()`` and ``end_ns()``.  Additionally, the
+:class:`xml.etree.ElementTree.TreeBuilder` target can be configured
+to process events about comments and processing instructions to include
+them in the generated tree.
+(Contributed by Stefan Behnel in :issue:`36676` and :issue:`36673`.)
=20
-* The target object of :class:`xml.etree.ElementTree.XMLParser` can
-  receive namespace declaration events through the new callback methods
-  ``start_ns()`` and ``end_ns()``.  Additionally, the
-  :class:`xml.etree.ElementTree.TreeBuilder` target can be configured
-  to process events about comments and processing instructions to include
-  them in the generated tree.
-  (Contributed by Stefan Behnel in :issue:`36676` and :issue:`36673`.)
=20
 Optimizations
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
@@ -1220,6 +1242,7 @@ Optimizations
   (Contributed by Inada Naoki in :issue:`33597`)
=20
 * :class:`uuid.UUID` now uses ``__slots__`` to reduce its memory footprint.
+  (Contributed by Wouter Bolsterlee and Tal Einat in :issue:`30977`)
=20
 * Improved performance of :func:`operator.itemgetter` by 33%.  Optimized
   argument handling and added a fast path for the common case of a single
@@ -1412,7 +1435,8 @@ Deprecated
   for the ``l*gettext()`` functions.
   (Contributed by Serhiy Storchaka in :issue:`33710`.)
=20
-* The :meth:`~threading.Thread.isAlive()` method of :class:`threading.Thread=
` has been deprecated.
+* The :meth:`~threading.Thread.isAlive()` method of :class:`threading.Thread`
+  has been deprecated.
   (Contributed by Dong-hee Na in :issue:`35283`.)
=20
 * Many builtin and extension functions that take integer arguments will
@@ -1516,7 +1540,7 @@ Changes in Python behavior
=20
 * The compiler now produces a :exc:`SyntaxWarning` when identity checks
   (``is`` and ``is not``) are used with certain types of literals
-  (e.g. strings, ints).  These can often work by accident in CPython,
+  (e.g. strings, numbers).  These can often work by accident in CPython,
   but are not guaranteed by the language spec.  The warning advises users
   to use equality tests (``=3D=3D`` and ``!=3D``) instead.
   (Contributed by Serhiy Storchaka in :issue:`34850`.)
@@ -1545,6 +1569,8 @@ Changes in Python behavior
   :c:func:`Py_END_ALLOW_THREADS`, and :c:func:`PyGILState_Ensure`. If this
   behavior is not desired, guard the call by checking :c:func:`_Py_IsFinaliz=
ing`
   or :c:func:`sys.is_finalizing`.
+  (Contributed by Joannah Nanjekye in :issue:`36475`.)
+
=20
 Changes in the Python API
 -------------------------
@@ -1569,6 +1595,7 @@ Changes in the Python API
=20
 * The :meth:`imap.IMAP4.logout` method no longer ignores silently arbitrary
   exceptions.
+  (Contributed by Victor Stinner in :issue:`36348`.)
=20
 * The function :func:`platform.popen` has been removed, after having been de=
precated since
   Python 3.3: use :func:`os.popen` instead.
@@ -1657,8 +1684,9 @@ Changes in the Python API
 * :func:`~os.path.expanduser` on Windows now prefers the :envvar:`USERPROFIL=
E`
   environment variable and does not use :envvar:`HOME`, which is not normally
   set for regular user accounts.
+  (Contributed by Anthony Sottile in :issue:`36264`.)
=20
-* The Exception :class:`asyncio.CancelledError` now inherits from
+* The exception :class:`asyncio.CancelledError` now inherits from
   :class:`BaseException` rather than a :class:`Exception`.
   (Contributed by Yury Selivanov in :issue:`13528`.)
=20
@@ -1675,7 +1703,7 @@ Changes in the Python API
   directory while loading your library. Note that Windows 7 users will need =
to
   ensure that Windows Update KB2533625 has been installed (this is also veri=
fied
   by the installer).
-  (See :issue:`36085`.)
+  (Contributed by Steve Dower in :issue:`36085`.)
=20
 * The header files and functions related to pgen have been removed after its
   replacement by a pure Python implementation. (Contributed by Pablo Galindo
@@ -1696,6 +1724,7 @@ Changes in the C API
   field. It should be initialized to ``PY_MINOR_VERSION``. The field is igno=
red
   by default, and is used if and only if ``PyCF_ONLY_AST`` flag is set in
   *cf_flags*.
+  (Contributed by Guido van Rossum in :issue:`35766`.)
=20
 * The :c:func:`PyEval_ReInitThreads` function has been removed from the C AP=
I.
   It should not be called explicitly: use :c:func:`PyOS_AfterFork_Child`
@@ -1742,7 +1771,9 @@ Changes in the C API
     :c:func:`PyObject_GC_NewVar`, or any other custom allocator that uses
     :c:func:`PyObject_Init` or :c:func:`PyObject_INIT`.
=20
-    Example::
+    Example:
+
+    .. code-block:: c
=20
         static foo_struct *
         foo_new(PyObject *type) {
@@ -1759,7 +1790,9 @@ Changes in the C API
   * Ensure that all custom ``tp_dealloc`` functions of heap-allocated types
     decrease the type's reference count.
=20
-    Example::
+    Example:
+
+    .. code-block:: c
=20
         static void
         foo_dealloc(foo_struct *instance) {
@@ -1776,7 +1809,9 @@ Changes in the C API
 * The :c:macro:`Py_DEPRECATED()` macro has been implemented for MSVC.
   The macro now must be placed before the symbol name.
=20
-  Example::
+  Example:
+
+  .. code-block:: c
=20
       Py_DEPRECATED(3.8) PyAPI_FUNC(int) Py_OldFunction(void);
=20
@@ -1797,12 +1832,12 @@ Changes in the C API
 * The functions :c:func:`PyNode_AddChild` and :c:func:`PyParser_AddToken` no=
w accept
   two additional ``int`` arguments *end_lineno* and *end_col_offset*.
=20
-.. highlight:: shell
-
 * The :file:`libpython38.a` file to allow MinGW tools to link directly again=
st
   :file:`python38.dll` is no longer included in the regular Windows distribu=
tion.
   If you require this file, it may be generated with the ``gendef`` and
-  ``dlltool`` tools, which are part of the MinGW binutils package::
+  ``dlltool`` tools, which are part of the MinGW binutils package:
+
+  .. code-block:: shell
=20
       gendef python38.dll > tmp.def
       dlltool --dllname python38.dll --def tmp.def --output-lib libpython38.a
@@ -1813,7 +1848,7 @@ Changes in the C API
   placed in the same directory as :file:`pythonXY.lib`, which is generally t=
he
   :file:`libs` directory under your Python installation.
=20
-.. highlight:: python3
+  (Contributed by Steve Dower in :issue:`37351`.)
=20
=20
 CPython bytecode changes


From webhook-mailer at python.org  Mon Oct 14 09:25:50 2019
From: webhook-mailer at python.org (=?utf-8?q?=C5=81ukasz?= Langa)
Date: Mon, 14 Oct 2019 13:25:50 -0000
Subject: [Python-checkins] (no subject)
Message-ID: <mailman.581.1599367019.9553.python-checkins@python.org>




To: python-checkins at python.org
Subject: [3.8] bpo-37759: Polish What's New in Python 3.8. (GH-16769) (#16777)
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

https://github.com/python/cpython/commit/e498329ebe0d86a5d80471112d766502b144=
a476
commit: e498329ebe0d86a5d80471112d766502b144a476
branch: 3.8
author: =C5=81ukasz Langa <lukasz at langa.pl>
committer: GitHub <noreply at github.com>
date: 2019-10-14T15:25:46+02:00
summary:

[3.8] bpo-37759: Polish What's New in Python 3.8. (GH-16769) (#16777)

(cherry picked from commit 298439ce3fb01de6c3110cc4847e6afe3253ba7a)

Co-authored-by: Serhiy Storchaka <storchaka at gmail.com>

files:
M Doc/whatsnew/3.8.rst

diff --git a/Doc/whatsnew/3.8.rst b/Doc/whatsnew/3.8.rst
index 0c59ef8015a58..8108fb7e14704 100644
--- a/Doc/whatsnew/3.8.rst
+++ b/Doc/whatsnew/3.8.rst
@@ -212,6 +212,7 @@ subdirectories).
=20
 (Contributed by Carl Meyer in :issue:`33499`.)
=20
+
 Debug build uses the same ABI as release build
 -----------------------------------------------
=20
@@ -285,6 +286,7 @@ calculations can be shown::
=20
 (Contributed by Eric V. Smith and Larry Hastings in :issue:`36817`.)
=20
+
 PEP 587: Python Initialization Configuration
 --------------------------------------------
=20
@@ -411,7 +413,8 @@ Other Language Changes
 * The syntax allowed for keyword names in function calls was further
   restricted. In particular, ``f((keyword)=3Darg)`` is no longer allowed. It=
 was
   never intended to permit more than a bare name on the left-hand side of a
-  keyword argument assignment term. See :issue:`34641`.
+  keyword argument assignment term.
+  (Contributed by Benjamin Peterson in :issue:`34641`.)
=20
 * Generalized iterable unpacking in :keyword:`yield` and
   :keyword:`return` statements no longer requires enclosing parentheses.
@@ -425,7 +428,6 @@ Other Language Changes
     >>> parse('simpsons homer marge bart lisa sally')
     ('SIMPSONS', 'homer', 'marge', 'bart', 'lisa', 'sally')
=20
-
   (Contributed by David Cuthbert and Jordan Chapman in :issue:`32117`.)
=20
 * When a comma is missed in code such as ``[(10, 20) (30, 40)]``, the
@@ -521,6 +523,8 @@ Other Language Changes
      '=C5=82ukasz langa': '=C5=81ukasz Langa',
      'walter d=C3=B6rwald': 'Walter D=C3=B6rwald'}
=20
+  (Contributed by J=C3=B6rn Heissler in :issue:`35224`.)
+
=20
 New Modules
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
@@ -544,13 +548,12 @@ New Modules
      PackagePath('requests-2.22.0.dist-info/RECORD'),
      PackagePath('requests-2.22.0.dist-info/WHEEL')]
=20
-  (Contributed in :issue:`34632` by Barry Warsaw and Jason R. Coombs.)
+  (Contributed by Barry Warsaw and Jason R. Coombs in :issue:`34632`.)
=20
=20
 Improved Modules
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
=20
-
 ast
 ---
=20
@@ -558,6 +561,11 @@ AST nodes now have ``end_lineno`` and ``end_col_offset``=
 attributes,
 which give the precise location of the end of the node.  (This only
 applies to nodes that have ``lineno`` and ``col_offset`` attributes.)
=20
+New function :func:`ast.get_source_segment` returns the source code
+for a specific AST node.
+
+(Contributed by Ivan Levkivskyi in :issue:`33416`.)
+
 The :func:`ast.parse` function has some new flags:
=20
 * ``type_comments=3DTrue`` causes it to return the text of :pep:`484` and
@@ -570,8 +578,7 @@ The :func:`ast.parse` function has some new flags:
   version. (For example, ``feature_version=3D(3, 4)`` will treat
   ``async`` and ``await`` as non-reserved words.)
=20
-New function :func:`ast.get_source_segment` returns the source code
-for a specific AST node.
+(Contributed by Guido van Rossum in :issue:`35766`.)
=20
=20
 asyncio
@@ -596,7 +603,6 @@ The :func:`compile` built-in has been improved to accept =
the
 :func:`compile` will allow top-level ``await``, ``async for`` and ``async wi=
th``
 constructs that are usually considered invalid syntax. Asynchronous code obj=
ect
 marked with the ``CO_COROUTINE`` flag may then be returned.
-
 (Contributed by Matthias Bussonnier in :issue:`34616`)
=20
=20
@@ -627,6 +633,7 @@ to specify flags for the underlying ``LoadLibraryEx`` cal=
l. The default flags ar
 set to only load DLL dependencies from trusted locations, including the path
 where the DLL is stored (if a full or partial path is used to load the initi=
al
 DLL) and paths added by :func:`~os.add_dll_directory`.
+(Contributed by Steve Dower in :issue:`36085`.)
=20
=20
 datetime
@@ -660,8 +667,8 @@ gc
 --
=20
 :func:`~gc.get_objects` can now receive an optional *generation* parameter
-indicating a generation to get objects from. Contributed in
-:issue:`36016` by Pablo Galindo.
+indicating a generation to get objects from.
+(Contributed by Pablo Galindo in :issue:`36016`.)
=20
=20
 gettext
@@ -723,6 +730,8 @@ for :func:`property`, :func:`classmethod`, and :func:`sta=
ticmethod`::
           self.bit_rate =3D round(bit_rate / 1000.0, 1)
           self.duration =3D ceil(duration)
=20
+(Contributed by Raymond Hettinger in :issue:`36326`.)
+
=20
 io
 --
@@ -780,7 +789,7 @@ multiprocessing
 ---------------
=20
 Added new :mod:`multiprocessing.shared_memory` module.
-(Contributed Davin Potts in :issue:`35813`.)
+(Contributed by Davin Potts in :issue:`35813`.)
=20
 On macOS, the *spawn* start method is now used by default.
 (Contributed by Victor Stinner in :issue:`33725`.)
@@ -792,6 +801,7 @@ os
 Added new function :func:`~os.add_dll_directory` on Windows for providing
 additional search paths for native dependencies when importing extension
 modules or loading DLLs using :mod:`ctypes`.
+(Contributed by Steve Dower in :issue:`36085`.)
=20
 A new :func:`os.memfd_create` function was added to wrap the
 ``memfd_create()`` syscall.
@@ -812,6 +822,8 @@ and so code that checks ``islink`` first will continue to=
 treat junctions as
 directories, while code that handles errors from :func:`os.readlink` may now
 treat junctions as links.
=20
+(Contributed by Steve Dower in :issue:`37834`.)
+
=20
 os.path
 -------
@@ -827,6 +839,7 @@ characters or bytes unrepresentable at the OS level.
 :func:`~os.path.expanduser` on Windows now prefers the :envvar:`USERPROFILE`
 environment variable and does not use :envvar:`HOME`, which is not normally =
set
 for regular user accounts.
+(Contributed by Anthony Sottile in :issue:`36264`.)
=20
 :func:`~os.path.isdir` on Windows no longer returns true for a link to a
 non-existent directory.
@@ -834,6 +847,8 @@ non-existent directory.
 :func:`~os.path.realpath` on Windows now resolves reparse points, including
 symlinks and directory junctions.
=20
+(Contributed by Steve Dower in :issue:`37834`.)
+
=20
 pathlib
 -------
@@ -883,11 +898,12 @@ py_compile
=20
=20
 shlex
-----------
+-----
=20
 The new :func:`shlex.join` function acts as the inverse of :func:`shlex.spli=
t`.
 (Contributed by Bo Bayles in :issue:`32102`.)
=20
+
 shutil
 ------
=20
@@ -901,6 +917,7 @@ inherited from the corresponding change to the :mod:`tarf=
ile` module.
=20
 :func:`shutil.rmtree` on Windows now removes directory junctions without
 recursively removing their contents first.
+(Contributed by Steve Dower in :issue:`37834`.)
=20
=20
 socket
@@ -995,18 +1012,18 @@ in a standardized and extensible format, and offers se=
veral other benefits.
 threading
 ---------
=20
-* Add a new :func:`threading.excepthook` function which handles uncaught
-  :meth:`threading.Thread.run` exception. It can be overridden to control how
-  uncaught :meth:`threading.Thread.run` exceptions are handled.
-  (Contributed by Victor Stinner in :issue:`1230540`.)
+Add a new :func:`threading.excepthook` function which handles uncaught
+:meth:`threading.Thread.run` exception. It can be overridden to control how
+uncaught :meth:`threading.Thread.run` exceptions are handled.
+(Contributed by Victor Stinner in :issue:`1230540`.)
=20
-* Add a new
-  :func:`threading.get_native_id` function and a :data:`~threading.Thread.na=
tive_id`
-  attribute to the :class:`threading.Thread` class. These return the native
-  integral Thread ID of the current thread assigned by the kernel.
-  This feature is only available on certain platforms, see
-  :func:`get_native_id <threading.get_native_id>` for more information.
-  (Contributed by Jake Tesler in :issue:`36084`.)
+Add a new :func:`threading.get_native_id` function and
+a :data:`~threading.Thread.native_id`
+attribute to the :class:`threading.Thread` class. These return the native
+integral Thread ID of the current thread assigned by the kernel.
+This feature is only available on certain platforms, see
+:func:`get_native_id <threading.get_native_id>` for more information.
+(Contributed by Jake Tesler in :issue:`36084`.)
=20
=20
 tokenize
@@ -1017,6 +1034,7 @@ provided with input that does not have a trailing new l=
ine.  This behavior
 now matches what the C tokenizer does internally.
 (Contributed by Ammar Askar in :issue:`33899`.)
=20
+
 tkinter
 -------
=20
@@ -1036,6 +1054,7 @@ The :class:`tkinter.PhotoImage` class now has
 :meth:`~tkinter.PhotoImage.transparency_set` methods.  (Contributed by
 Zackery Spytz in :issue:`25451`.)
=20
+
 time
 ----
=20
@@ -1084,95 +1103,98 @@ The :mod:`typing` module incorporates several new fea=
tures:
 unicodedata
 -----------
=20
-* The :mod:`unicodedata` module has been upgraded to use the `Unicode 12.1.0
-  <http://blog.unicode.org/2019/05/unicode-12-1-en.html>`_ release.
+The :mod:`unicodedata` module has been upgraded to use the `Unicode 12.1.0
+<http://blog.unicode.org/2019/05/unicode-12-1-en.html>`_ release.
=20
-* New function :func:`~unicodedata.is_normalized` can be used to verify a st=
ring
-  is in a specific normal form, often much faster than by actually normalizi=
ng
-  the string.  (Contributed by Max Belanger, David Euresti, and Greg Price in
-  :issue:`32285` and :issue:`37966`).
+New function :func:`~unicodedata.is_normalized` can be used to verify a stri=
ng
+is in a specific normal form, often much faster than by actually normalizing
+the string.  (Contributed by Max Belanger, David Euresti, and Greg Price in
+:issue:`32285` and :issue:`37966`).
=20
=20
 unittest
 --------
=20
-* Added :class:`AsyncMock` to support an asynchronous version of :class:`Moc=
k`.
-  Appropriate new assert functions for testing have been added as well.
-  (Contributed by Lisa Roach in :issue:`26467`).
+Added :class:`AsyncMock` to support an asynchronous version of :class:`Mock`.
+Appropriate new assert functions for testing have been added as well.
+(Contributed by Lisa Roach in :issue:`26467`).
=20
-* Added :func:`~unittest.addModuleCleanup()` and
-  :meth:`~unittest.TestCase.addClassCleanup()` to unittest to support
-  cleanups for :func:`~unittest.setUpModule()` and
-  :meth:`~unittest.TestCase.setUpClass()`.
-  (Contributed by Lisa Roach in :issue:`24412`.)
+Added :func:`~unittest.addModuleCleanup()` and
+:meth:`~unittest.TestCase.addClassCleanup()` to unittest to support
+cleanups for :func:`~unittest.setUpModule()` and
+:meth:`~unittest.TestCase.setUpClass()`.
+(Contributed by Lisa Roach in :issue:`24412`.)
=20
-* Several mock assert functions now also print a list of actual calls upon
-  failure. (Contributed by Petter Strandmark in :issue:`35047`.)
+Several mock assert functions now also print a list of actual calls upon
+failure. (Contributed by Petter Strandmark in :issue:`35047`.)
=20
-* :mod:`unittest` module gained support for coroutines to be used as test ca=
ses
-  with :class:`unittest.IsolatedAsyncioTestCase`.
-  (Contributed by Andrew Svetlov in :issue:`32972`.)
+:mod:`unittest` module gained support for coroutines to be used as test cases
+with :class:`unittest.IsolatedAsyncioTestCase`.
+(Contributed by Andrew Svetlov in :issue:`32972`.)
=20
-  Example::
+Example::
=20
-    import unittest
+   import unittest
=20
=20
-    class TestRequest(unittest.IsolatedAsyncioTestCase):
+   class TestRequest(unittest.IsolatedAsyncioTestCase):
=20
-        async def asyncSetUp(self):
-            self.connection =3D await AsyncConnection()
+       async def asyncSetUp(self):
+           self.connection =3D await AsyncConnection()
=20
-        async def test_get(self):
-            response =3D await self.connection.get("https://example.com")
-            self.assertEqual(response.status_code, 200)
+       async def test_get(self):
+           response =3D await self.connection.get("https://example.com")
+           self.assertEqual(response.status_code, 200)
=20
-        async def asyncTearDown(self):
-            await self.connection.close()
+       async def asyncTearDown(self):
+           await self.connection.close()
=20
=20
-    if __name__ =3D=3D "__main__":
-        unittest.main()
+   if __name__ =3D=3D "__main__":
+       unittest.main()
=20
=20
 venv
 ----
=20
-* :mod:`venv` now includes an ``Activate.ps1`` script on all platforms for
-  activating virtual environments under PowerShell Core 6.1.
-  (Contributed by Brett Cannon in :issue:`32718`.)
+:mod:`venv` now includes an ``Activate.ps1`` script on all platforms for
+activating virtual environments under PowerShell Core 6.1.
+(Contributed by Brett Cannon in :issue:`32718`.)
+
=20
 weakref
 -------
=20
-* The proxy objects returned by :func:`weakref.proxy` now support the matrix
-  multiplication operators ``@`` and ``@=3D`` in addition to the other
-  numeric operators. (Contributed by Mark Dickinson in :issue:`36669`.)
+The proxy objects returned by :func:`weakref.proxy` now support the matrix
+multiplication operators ``@`` and ``@=3D`` in addition to the other
+numeric operators. (Contributed by Mark Dickinson in :issue:`36669`.)
+
=20
 xml
 ---
=20
-* As mitigation against DTD and external entity retrieval, the
-  :mod:`xml.dom.minidom` and :mod:`xml.sax` modules no longer process
-  external entities by default.
-  (Contributed by Christian Heimes in :issue:`17239`.)
+As mitigation against DTD and external entity retrieval, the
+:mod:`xml.dom.minidom` and :mod:`xml.sax` modules no longer process
+external entities by default.
+(Contributed by Christian Heimes in :issue:`17239`.)
=20
-* The ``.find*()`` methods in the :mod:`xml.etree.ElementTree` module
-  support wildcard searches like ``{*}tag`` which ignores the namespace
-  and ``{namespace}*`` which returns all tags in the given namespace.
-  (Contributed by Stefan Behnel in :issue:`28238`.)
+The ``.find*()`` methods in the :mod:`xml.etree.ElementTree` module
+support wildcard searches like ``{*}tag`` which ignores the namespace
+and ``{namespace}*`` which returns all tags in the given namespace.
+(Contributed by Stefan Behnel in :issue:`28238`.)
=20
-* The :mod:`xml.etree.ElementTree` module provides a new function
-  :func:`=E2=80=93xml.etree.ElementTree.canonicalize()` that implements C14N=
 2.0.
-  (Contributed by Stefan Behnel in :issue:`13611`.)
+The :mod:`xml.etree.ElementTree` module provides a new function
+:func:`=E2=80=93xml.etree.ElementTree.canonicalize()` that implements C14N 2=
.0.
+(Contributed by Stefan Behnel in :issue:`13611`.)
+
+The target object of :class:`xml.etree.ElementTree.XMLParser` can
+receive namespace declaration events through the new callback methods
+``start_ns()`` and ``end_ns()``.  Additionally, the
+:class:`xml.etree.ElementTree.TreeBuilder` target can be configured
+to process events about comments and processing instructions to include
+them in the generated tree.
+(Contributed by Stefan Behnel in :issue:`36676` and :issue:`36673`.)
=20
-* The target object of :class:`xml.etree.ElementTree.XMLParser` can
-  receive namespace declaration events through the new callback methods
-  ``start_ns()`` and ``end_ns()``.  Additionally, the
-  :class:`xml.etree.ElementTree.TreeBuilder` target can be configured
-  to process events about comments and processing instructions to include
-  them in the generated tree.
-  (Contributed by Stefan Behnel in :issue:`36676` and :issue:`36673`.)
=20
 Optimizations
 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
@@ -1220,6 +1242,7 @@ Optimizations
   (Contributed by Inada Naoki in :issue:`33597`)
=20
 * :class:`uuid.UUID` now uses ``__slots__`` to reduce its memory footprint.
+  (Contributed by Wouter Bolsterlee and Tal Einat in :issue:`30977`)
=20
 * Improved performance of :func:`operator.itemgetter` by 33%.  Optimized
   argument handling and added a fast path for the common case of a single
@@ -1412,7 +1435,8 @@ Deprecated
   for the ``l*gettext()`` functions.
   (Contributed by Serhiy Storchaka in :issue:`33710`.)
=20
-* The :meth:`~threading.Thread.isAlive()` method of :class:`threading.Thread=
` has been deprecated.
+* The :meth:`~threading.Thread.isAlive()` method of :class:`threading.Thread`
+  has been deprecated.
   (Contributed by Dong-hee Na in :issue:`35283`.)
=20
 * Many builtin and extension functions that take integer arguments will
@@ -1516,7 +1540,7 @@ Changes in Python behavior
=20
 * The compiler now produces a :exc:`SyntaxWarning` when identity checks
   (``is`` and ``is not``) are used with certain types of literals
-  (e.g. strings, ints).  These can often work by accident in CPython,
+  (e.g. strings, numbers).  These can often work by accident in CPython,
   but are not guaranteed by the language spec.  The warning advises users
   to use equality tests (``=3D=3D`` and ``!=3D``) instead.
   (Contributed by Serhiy Storchaka in :issue:`34850`.)
@@ -1545,6 +1569,8 @@ Changes in Python behavior
   :c:func:`Py_END_ALLOW_THREADS`, and :c:func:`PyGILState_Ensure`. If this
   behavior is not desired, guard the call by checking :c:func:`_Py_IsFinaliz=
ing`
   or :c:func:`sys.is_finalizing`.
+  (Contributed by Joannah Nanjekye in :issue:`36475`.)
+
=20
 Changes in the Python API
 -------------------------
@@ -1569,6 +1595,7 @@ Changes in the Python API
=20
 * The :meth:`imap.IMAP4.logout` method no longer ignores silently arbitrary
   exceptions.
+  (Contributed by Victor Stinner in :issue:`36348`.)
=20
 * The function :func:`platform.popen` has been removed, after having been de=
precated since
   Python 3.3: use :func:`os.popen` instead.
@@ -1657,8 +1684,9 @@ Changes in the Python API
 * :func:`~os.path.expanduser` on Windows now prefers the :envvar:`USERPROFIL=
E`
   environment variable and does not use :envvar:`HOME`, which is not normally
   set for regular user accounts.
+  (Contributed by Anthony Sottile in :issue:`36264`.)
=20
-* The Exception :class:`asyncio.CancelledError` now inherits from
+* The exception :class:`asyncio.CancelledError` now inherits from
   :class:`BaseException` rather than a :class:`Exception`.
   (Contributed by Yury Selivanov in :issue:`13528`.)
=20
@@ -1675,7 +1703,7 @@ Changes in the Python API
   directory while loading your library. Note that Windows 7 users will need =
to
   ensure that Windows Update KB2533625 has been installed (this is also veri=
fied
   by the installer).
-  (See :issue:`36085`.)
+  (Contributed by Steve Dower in :issue:`36085`.)
=20
 * The header files and functions related to pgen have been removed after its
   replacement by a pure Python implementation. (Contributed by Pablo Galindo
@@ -1696,6 +1724,7 @@ Changes in the C API
   field. It should be initialized to ``PY_MINOR_VERSION``. The field is igno=
red
   by default, and is used if and only if ``PyCF_ONLY_AST`` flag is set in
   *cf_flags*.
+  (Contributed by Guido van Rossum in :issue:`35766`.)
=20
 * The :c:func:`PyEval_ReInitThreads` function has been removed from the C AP=
I.
   It should not be called explicitly: use :c:func:`PyOS_AfterFork_Child`
@@ -1742,7 +1771,9 @@ Changes in the C API
     :c:func:`PyObject_GC_NewVar`, or any other custom allocator that uses
     :c:func:`PyObject_Init` or :c:func:`PyObject_INIT`.
=20
-    Example::
+    Example:
+
+    .. code-block:: c
=20
         static foo_struct *
         foo_new(PyObject *type) {
@@ -1759,7 +1790,9 @@ Changes in the C API
   * Ensure that all custom ``tp_dealloc`` functions of heap-allocated types
     decrease the type's reference count.
=20
-    Example::
+    Example:
+
+    .. code-block:: c
=20
         static void
         foo_dealloc(foo_struct *instance) {
@@ -1776,7 +1809,9 @@ Changes in the C API
 * The :c:macro:`Py_DEPRECATED()` macro has been implemented for MSVC.
   The macro now must be placed before the symbol name.
=20
-  Example::
+  Example:
+
+  .. code-block:: c
=20
       Py_DEPRECATED(3.8) PyAPI_FUNC(int) Py_OldFunction(void);
=20
@@ -1797,12 +1832,12 @@ Changes in the C API
 * The functions :c:func:`PyNode_AddChild` and :c:func:`PyParser_AddToken` no=
w accept
   two additional ``int`` arguments *end_lineno* and *end_col_offset*.
=20
-.. highlight:: shell
-
 * The :file:`libpython38.a` file to allow MinGW tools to link directly again=
st
   :file:`python38.dll` is no longer included in the regular Windows distribu=
tion.
   If you require this file, it may be generated with the ``gendef`` and
-  ``dlltool`` tools, which are part of the MinGW binutils package::
+  ``dlltool`` tools, which are part of the MinGW binutils package:
+
+  .. code-block:: shell
=20
       gendef python38.dll > tmp.def
       dlltool --dllname python38.dll --def tmp.def --output-lib libpython38.a
@@ -1813,7 +1848,7 @@ Changes in the C API
   placed in the same directory as :file:`pythonXY.lib`, which is generally t=
he
   :file:`libs` directory under your Python installation.
=20
-.. highlight:: python3
+  (Contributed by Steve Dower in :issue:`37351`.)
=20
=20
 CPython bytecode changes


From webhook-mailer at python.org  Mon Oct 14 16:04:52 2019
From: webhook-mailer at python.org (=?utf-8?q?=C5=81ukasz?= Langa)
Date: Mon, 14 Oct 2019 20:04:52 -0000
Subject: [Python-checkins] (no subject)
Message-ID: <mailman.582.1599367020.9553.python-checkins@python.org>




To: python-checkins at python.org
Subject: v3.8.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

https://github.com/python/cpython/commit/fa919fdf2583bdfead1df00e842f24f30b2a=
34bf
commit: fa919fdf2583bdfead1df00e842f24f30b2a34bf
branch: 3.8
author: =C5=81ukasz Langa <lukasz at langa.pl>
committer: =C5=81ukasz Langa <lukasz at langa.pl>
date: 2019-10-14T15:34:47+02:00
summary:

v3.8.0

files:
A Misc/NEWS.d/3.8.0.rst
D Misc/NEWS.d/next/C API/2019-10-08-01-23-24.bpo-38395.MJ6Ey9.rst
D Misc/NEWS.d/next/Core and Builtins/2019-10-09-16-50-52.bpo-38379.oz5qZx.rst
D Misc/NEWS.d/next/Core and Builtins/2019-10-13-23-41-38.bpo-38469.9kmuQj.rst
D Misc/NEWS.d/next/Documentation/2019-09-27-23-37-41.bpo-38294.go_jFf.rst
D Misc/NEWS.d/next/IDLE/2019-10-04-18-03-09.bpo-36698.BKcmom.rst
D Misc/NEWS.d/next/Library/2019-09-29-22-47-37.bpo-13153.0mO9qR.rst
D Misc/NEWS.d/next/Library/2019-10-01-21-06-18.bpo-38341.uqwgU_.rst
D Misc/NEWS.d/next/Library/2019-10-05-02-07-52.bpo-38332.hwrPN7.rst
D Misc/NEWS.d/next/Library/2019-10-08-11-18-40.bpo-38405.0-7e7s.rst
D Misc/NEWS.d/next/Library/2019-10-10-00-25-28.bpo-38109.9w-IGF.rst
D Misc/NEWS.d/next/Library/2019-10-10-16-53-00.bpo-38431.d5wzNp.rst
D Misc/NEWS.d/next/Library/2019-10-11-18-49-00.bpo-38449.9TWMlz.rst
D Misc/NEWS.d/next/Tests/2019-10-08-16-42-05.bpo-37531.7v-_Ca.rst
D Misc/NEWS.d/next/Tools-Demos/2019-10-02-09-48-42.bpo-38347.2Tq5D1.rst
D Misc/NEWS.d/next/Tools-Demos/2019-10-08-15-07-52.bpo-38118.pIZD6H.rst
D Misc/NEWS.d/next/Windows/2019-10-02-15-38-49.bpo-38355.n3AWX6.rst
D Misc/NEWS.d/next/Windows/2019-10-03-08-04-14.bpo-38359.wzwsl_.rst
D Misc/NEWS.d/next/Windows/2019-10-05-05-50-58.bpo-38344.scr2LO.rst
M Doc/whatsnew/3.8.rst
M Include/patchlevel.h
M Lib/pydoc_data/topics.py
M README.rst

diff --git a/Doc/whatsnew/3.8.rst b/Doc/whatsnew/3.8.rst
index 8108fb7e14704..d80005fda3673 100644
--- a/Doc/whatsnew/3.8.rst
+++ b/Doc/whatsnew/3.8.rst
@@ -47,6 +47,8 @@
 This article explains the new features in Python 3.8, compared to 3.7.
 For full details, see the :ref:`changelog <changelog>`.
=20
+Python 3.8 was released on October 14th, 2019.
+
 Prerelease users should be aware that this document is currently in
 draft form. It will be updated as Python 3.8 moves towards release, so
 it's worth checking back even after reading earlier versions. Some
diff --git a/Include/patchlevel.h b/Include/patchlevel.h
index a441413074f4a..d685a85709e82 100644
--- a/Include/patchlevel.h
+++ b/Include/patchlevel.h
@@ -19,11 +19,11 @@
 #define PY_MAJOR_VERSION        3
 #define PY_MINOR_VERSION        8
 #define PY_MICRO_VERSION        0
-#define PY_RELEASE_LEVEL        PY_RELEASE_LEVEL_GAMMA
-#define PY_RELEASE_SERIAL       1
+#define PY_RELEASE_LEVEL        PY_RELEASE_LEVEL_FINAL
+#define PY_RELEASE_SERIAL       0
=20
 /* Version as a string */
-#define PY_VERSION              "3.8.0rc1+"
+#define PY_VERSION              "3.8.0"
 /*--end constants--*/
=20
 /* Version as a single 4-byte hex number, e.g. 0x010502B2 =3D=3D 1.5.2b2.
diff --git a/Lib/pydoc_data/topics.py b/Lib/pydoc_data/topics.py
index 7cb089f757181..327a421bbb6bc 100644
--- a/Lib/pydoc_data/topics.py
+++ b/Lib/pydoc_data/topics.py
@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-# Autogenerated by Sphinx on Tue Oct  1 14:53:09 2019
+# Autogenerated by Sphinx on Mon Oct 14 14:33:10 2019
 topics =3D {'assert': 'The "assert" statement\n'
            '**********************\n'
            '\n'
diff --git a/Misc/NEWS.d/3.8.0.rst b/Misc/NEWS.d/3.8.0.rst
new file mode 100644
index 0000000000000..b7e681f8333bd
--- /dev/null
+++ b/Misc/NEWS.d/3.8.0.rst
@@ -0,0 +1,186 @@
+.. bpo: 38469
+.. date: 2019-10-13-23-41-38
+.. nonce: 9kmuQj
+.. release date: 2019-10-14
+.. section: Core and Builtins
+
+Fixed a bug where the scope of named expressions was not being resolved
+correctly in the presence of the *global* keyword. Patch by Pablo Galindo.
+
+..
+
+.. bpo: 38379
+.. date: 2019-10-09-16-50-52
+.. nonce: oz5qZx
+.. section: Core and Builtins
+
+When cyclic garbage collection (gc) runs finalizers that resurrect
+unreachable objects, the current gc run ends, without collecting any cyclic
+trash.  However, the statistics reported by ``collect()`` and
+``get_stats()`` claimed that all cyclic trash found was collected, and that
+the resurrected objects were collected.   Changed the stats to report that
+none were collected.
+
+..
+
+.. bpo: 38449
+.. date: 2019-10-11-18-49-00
+.. nonce: 9TWMlz
+.. section: Library
+
+Revert GH-15522, which introduces a regression in
+:meth:`mimetypes.guess_type` due to improper handling of filenames as urls.
+
+..
+
+.. bpo: 38431
+.. date: 2019-10-10-16-53-00
+.. nonce: d5wzNp
+.. section: Library
+
+Fix ``__repr__`` method for :class:`dataclasses.InitVar` to support typing
+objects, patch by Samuel Colvin.
+
+..
+
+.. bpo: 38109
+.. date: 2019-10-10-00-25-28
+.. nonce: 9w-IGF
+.. section: Library
+
+Add missing :data:`stat.S_IFDOOR`, :data:`stat.S_IFPORT`,
+:data:`stat.S_IFWHT`, :func:`stat.S_ISDOOR`, :func:`stat.S_ISPORT`, and
+:func:`stat.S_ISWHT` values to the Python implementation of :mod:`stat`.
+
+..
+
+.. bpo: 38405
+.. date: 2019-10-08-11-18-40
+.. nonce: 0-7e7s
+.. section: Library
+
+Nested subclasses of :class:`typing.NamedTuple` are now pickleable.
+
+..
+
+.. bpo: 38332
+.. date: 2019-10-05-02-07-52
+.. nonce: hwrPN7
+.. section: Library
+
+Prevent :exc:`KeyError` thrown by :func:`_encoded_words.decode` when given
+an encoded-word with invalid content-type encoding from propagating all the
+way to :func:`email.message.get`.
+
+..
+
+.. bpo: 38341
+.. date: 2019-10-01-21-06-18
+.. nonce: uqwgU_
+.. section: Library
+
+Add :exc:`smtplib.SMTPNotSupportedError` to the :mod:`smtplib` exported
+names.
+
+..
+
+.. bpo: 13153
+.. date: 2019-09-29-22-47-37
+.. nonce: 0mO9qR
+.. section: Library
+
+OS native encoding is now used for converting between Python strings and Tcl
+objects.  This allows to display, copy and paste to clipboard emoji and
+other non-BMP characters.  Converting strings from Tcl to Python and back
+now never fails (except MemoryError).
+
+..
+
+.. bpo: 38294
+.. date: 2019-09-27-23-37-41
+.. nonce: go_jFf
+.. section: Documentation
+
+Add list of no-longer-escaped chars to re.escape documentation
+
+..
+
+.. bpo: 37531
+.. date: 2019-10-08-16-42-05
+.. nonce: 7v-_Ca
+.. section: Tests
+
+On timeout, regrtest no longer attempts to call ``popen.communicate()``
+again: it can hang until all child processes using stdout and stderr pipes
+completes. Kill the worker process and ignores its output. Change also the
+faulthandler timeout of the main process from 1 minute to 5 minutes, for
+Python slowest buildbots.
+
+..
+
+.. bpo: 38344
+.. date: 2019-10-05-05-50-58
+.. nonce: scr2LO
+.. section: Windows
+
+Fix error message in activate.bat
+
+..
+
+.. bpo: 38359
+.. date: 2019-10-03-08-04-14
+.. nonce: wzwsl_
+.. section: Windows
+
+Ensures ``pyw.exe`` launcher reads correct registry key.
+
+..
+
+.. bpo: 38355
+.. date: 2019-10-02-15-38-49
+.. nonce: n3AWX6
+.. section: Windows
+
+Fixes ``ntpath.realpath`` failing on ``sys.executable``.
+
+..
+
+.. bpo: 36698
+.. date: 2019-10-04-18-03-09
+.. nonce: BKcmom
+.. section: IDLE
+
+IDLE no longer fails when write non-encodable characters to stderr.  It now
+escapes them with a backslash, as the regular Python interpreter. Added the
+``errors`` field to the standard streams.
+
+..
+
+.. bpo: 38118
+.. date: 2019-10-08-15-07-52
+.. nonce: pIZD6H
+.. section: Tools/Demos
+
+Update Valgrind suppression file to ignore a false alarm in
+:c:func:`PyUnicode_Decode` when using GCC builtin strcmp().
+
+..
+
+.. bpo: 38347
+.. date: 2019-10-02-09-48-42
+.. nonce: 2Tq5D1
+.. section: Tools/Demos
+
+pathfix.py: Assume all files that end on '.py' are Python scripts when
+working recursively.
+
+..
+
+.. bpo: 38395
+.. date: 2019-10-08-01-23-24
+.. nonce: MJ6Ey9
+.. section: C API
+
+Fix a crash in :class:`weakref.proxy` objects due to incorrect lifetime
+management when calling some associated methods that may delete the last
+reference to object being referenced by the proxy. Patch by Pablo Galindo.
diff --git a/Misc/NEWS.d/next/C API/2019-10-08-01-23-24.bpo-38395.MJ6Ey9.rst =
b/Misc/NEWS.d/next/C API/2019-10-08-01-23-24.bpo-38395.MJ6Ey9.rst
deleted file mode 100644
index 4bc30600ca758..0000000000000
--- a/Misc/NEWS.d/next/C API/2019-10-08-01-23-24.bpo-38395.MJ6Ey9.rst=09
+++ /dev/null
@@ -1,3 +0,0 @@
-Fix a crash in :class:`weakref.proxy` objects due to incorrect lifetime
-management when calling some associated methods that may delete the last
-reference to object being referenced by the proxy. Patch by Pablo Galindo.
diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-10-09-16-50-52.bpo-38379=
.oz5qZx.rst b/Misc/NEWS.d/next/Core and Builtins/2019-10-09-16-50-52.bpo-3837=
9.oz5qZx.rst
deleted file mode 100644
index 82dcb525dd49d..0000000000000
--- a/Misc/NEWS.d/next/Core and Builtins/2019-10-09-16-50-52.bpo-38379.oz5qZx=
.rst=09
+++ /dev/null
@@ -1 +0,0 @@
-When cyclic garbage collection (gc) runs finalizers that resurrect unreachab=
le objects, the current gc run ends, without collecting any cyclic trash.  Ho=
wever, the statistics reported by ``collect()`` and ``get_stats()`` claimed t=
hat all cyclic trash found was collected, and that the resurrected objects we=
re collected.   Changed the stats to report that none were collected.
diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-10-13-23-41-38.bpo-38469=
.9kmuQj.rst b/Misc/NEWS.d/next/Core and Builtins/2019-10-13-23-41-38.bpo-3846=
9.9kmuQj.rst
deleted file mode 100644
index 328a1b70825af..0000000000000
--- a/Misc/NEWS.d/next/Core and Builtins/2019-10-13-23-41-38.bpo-38469.9kmuQj=
.rst=09
+++ /dev/null
@@ -1,2 +0,0 @@
-Fixed a bug where the scope of named expressions was not being resolved
-correctly in the presence of the *global* keyword. Patch by Pablo Galindo.
diff --git a/Misc/NEWS.d/next/Documentation/2019-09-27-23-37-41.bpo-38294.go_=
jFf.rst b/Misc/NEWS.d/next/Documentation/2019-09-27-23-37-41.bpo-38294.go_jFf=
.rst
deleted file mode 100644
index 0bde8c868c9ec..0000000000000
--- a/Misc/NEWS.d/next/Documentation/2019-09-27-23-37-41.bpo-38294.go_jFf.rst
+++ /dev/null
@@ -1 +0,0 @@
-Add list of no-longer-escaped chars to re.escape documentation
\ No newline at end of file
diff --git a/Misc/NEWS.d/next/IDLE/2019-10-04-18-03-09.bpo-36698.BKcmom.rst b=
/Misc/NEWS.d/next/IDLE/2019-10-04-18-03-09.bpo-36698.BKcmom.rst
deleted file mode 100644
index 5aaa3c92646a1..0000000000000
--- a/Misc/NEWS.d/next/IDLE/2019-10-04-18-03-09.bpo-36698.BKcmom.rst
+++ /dev/null
@@ -1,3 +0,0 @@
-IDLE no longer fails when write non-encodable characters to stderr.  It now
-escapes them with a backslash, as the regular Python interpreter. Added the
-``errors`` field to the standard streams.
diff --git a/Misc/NEWS.d/next/Library/2019-09-29-22-47-37.bpo-13153.0mO9qR.rs=
t b/Misc/NEWS.d/next/Library/2019-09-29-22-47-37.bpo-13153.0mO9qR.rst
deleted file mode 100644
index 6f9561539d8d3..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-09-29-22-47-37.bpo-13153.0mO9qR.rst
+++ /dev/null
@@ -1,4 +0,0 @@
-OS native encoding is now used for converting between Python strings and
-Tcl objects.  This allows to display, copy and paste to clipboard emoji and
-other non-BMP characters.  Converting strings from Tcl to Python and back
-now never fails (except MemoryError).
diff --git a/Misc/NEWS.d/next/Library/2019-10-01-21-06-18.bpo-38341.uqwgU_.rs=
t b/Misc/NEWS.d/next/Library/2019-10-01-21-06-18.bpo-38341.uqwgU_.rst
deleted file mode 100644
index dd1a6b080b627..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-10-01-21-06-18.bpo-38341.uqwgU_.rst
+++ /dev/null
@@ -1 +0,0 @@
-Add :exc:`smtplib.SMTPNotSupportedError` to the :mod:`smtplib` exported name=
s.
diff --git a/Misc/NEWS.d/next/Library/2019-10-05-02-07-52.bpo-38332.hwrPN7.rs=
t b/Misc/NEWS.d/next/Library/2019-10-05-02-07-52.bpo-38332.hwrPN7.rst
deleted file mode 100644
index 600c702cf3bbd..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-10-05-02-07-52.bpo-38332.hwrPN7.rst
+++ /dev/null
@@ -1,3 +0,0 @@
-Prevent :exc:`KeyError` thrown by :func:`_encoded_words.decode` when given
-an encoded-word with invalid content-type encoding from propagating all the
-way to :func:`email.message.get`.
diff --git a/Misc/NEWS.d/next/Library/2019-10-08-11-18-40.bpo-38405.0-7e7s.rs=
t b/Misc/NEWS.d/next/Library/2019-10-08-11-18-40.bpo-38405.0-7e7s.rst
deleted file mode 100644
index ee346a30ec416..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-10-08-11-18-40.bpo-38405.0-7e7s.rst
+++ /dev/null
@@ -1 +0,0 @@
-Nested subclasses of :class:`typing.NamedTuple` are now pickleable.
diff --git a/Misc/NEWS.d/next/Library/2019-10-10-00-25-28.bpo-38109.9w-IGF.rs=
t b/Misc/NEWS.d/next/Library/2019-10-10-00-25-28.bpo-38109.9w-IGF.rst
deleted file mode 100644
index 3f4484dc644fa..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-10-10-00-25-28.bpo-38109.9w-IGF.rst
+++ /dev/null
@@ -1,3 +0,0 @@
-Add missing :data:`stat.S_IFDOOR`, :data:`stat.S_IFPORT`, :data:`stat.S_IFWH=
T`,
-:func:`stat.S_ISDOOR`, :func:`stat.S_ISPORT`, and :func:`stat.S_ISWHT` value=
s to
-the Python implementation of :mod:`stat`.
\ No newline at end of file
diff --git a/Misc/NEWS.d/next/Library/2019-10-10-16-53-00.bpo-38431.d5wzNp.rs=
t b/Misc/NEWS.d/next/Library/2019-10-10-16-53-00.bpo-38431.d5wzNp.rst
deleted file mode 100644
index c2f860d804c13..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-10-10-16-53-00.bpo-38431.d5wzNp.rst
+++ /dev/null
@@ -1 +0,0 @@
-Fix ``__repr__`` method for :class:`dataclasses.InitVar` to support typing o=
bjects, patch by Samuel Colvin.
diff --git a/Misc/NEWS.d/next/Library/2019-10-11-18-49-00.bpo-38449.9TWMlz.rs=
t b/Misc/NEWS.d/next/Library/2019-10-11-18-49-00.bpo-38449.9TWMlz.rst
deleted file mode 100644
index f7b1dbf8d8951..0000000000000
--- a/Misc/NEWS.d/next/Library/2019-10-11-18-49-00.bpo-38449.9TWMlz.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Revert GH-15522, which introduces a regression in
-:meth:`mimetypes.guess_type` due to improper handling of filenames as urls.
diff --git a/Misc/NEWS.d/next/Tests/2019-10-08-16-42-05.bpo-37531.7v-_Ca.rst =
b/Misc/NEWS.d/next/Tests/2019-10-08-16-42-05.bpo-37531.7v-_Ca.rst
deleted file mode 100644
index 78ab636516514..0000000000000
--- a/Misc/NEWS.d/next/Tests/2019-10-08-16-42-05.bpo-37531.7v-_Ca.rst
+++ /dev/null
@@ -1,5 +0,0 @@
-On timeout, regrtest no longer attempts to call ``popen.communicate()``
-again: it can hang until all child processes using stdout and stderr pipes
-completes. Kill the worker process and ignores its output. Change also the
-faulthandler timeout of the main process from 1 minute to 5 minutes, for Pyt=
hon
-slowest buildbots.
diff --git a/Misc/NEWS.d/next/Tools-Demos/2019-10-02-09-48-42.bpo-38347.2Tq5D=
1.rst b/Misc/NEWS.d/next/Tools-Demos/2019-10-02-09-48-42.bpo-38347.2Tq5D1.rst
deleted file mode 100644
index ae64a319b5756..0000000000000
--- a/Misc/NEWS.d/next/Tools-Demos/2019-10-02-09-48-42.bpo-38347.2Tq5D1.rst
+++ /dev/null
@@ -1 +0,0 @@
-pathfix.py: Assume all files that end on '.py' are Python scripts when worki=
ng recursively.
diff --git a/Misc/NEWS.d/next/Tools-Demos/2019-10-08-15-07-52.bpo-38118.pIZD6=
H.rst b/Misc/NEWS.d/next/Tools-Demos/2019-10-08-15-07-52.bpo-38118.pIZD6H.rst
deleted file mode 100644
index 13adadb0b8f7a..0000000000000
--- a/Misc/NEWS.d/next/Tools-Demos/2019-10-08-15-07-52.bpo-38118.pIZD6H.rst
+++ /dev/null
@@ -1,2 +0,0 @@
-Update Valgrind suppression file to ignore a false alarm in
-:c:func:`PyUnicode_Decode` when using GCC builtin strcmp().
diff --git a/Misc/NEWS.d/next/Windows/2019-10-02-15-38-49.bpo-38355.n3AWX6.rs=
t b/Misc/NEWS.d/next/Windows/2019-10-02-15-38-49.bpo-38355.n3AWX6.rst
deleted file mode 100644
index 56e0f56505522..0000000000000
--- a/Misc/NEWS.d/next/Windows/2019-10-02-15-38-49.bpo-38355.n3AWX6.rst
+++ /dev/null
@@ -1 +0,0 @@
-Fixes ``ntpath.realpath`` failing on ``sys.executable``.
diff --git a/Misc/NEWS.d/next/Windows/2019-10-03-08-04-14.bpo-38359.wzwsl_.rs=
t b/Misc/NEWS.d/next/Windows/2019-10-03-08-04-14.bpo-38359.wzwsl_.rst
deleted file mode 100644
index ab6f29baf5358..0000000000000
--- a/Misc/NEWS.d/next/Windows/2019-10-03-08-04-14.bpo-38359.wzwsl_.rst
+++ /dev/null
@@ -1 +0,0 @@
-Ensures ``pyw.exe`` launcher reads correct registry key.
diff --git a/Misc/NEWS.d/next/Windows/2019-10-05-05-50-58.bpo-38344.scr2LO.rs=
t b/Misc/NEWS.d/next/Windows/2019-10-05-05-50-58.bpo-38344.scr2LO.rst
deleted file mode 100644
index b5854d1ecb0d3..0000000000000
--- a/Misc/NEWS.d/next/Windows/2019-10-05-05-50-58.bpo-38344.scr2LO.rst
+++ /dev/null
@@ -1 +0,0 @@
-Fix error message in activate.bat
diff --git a/README.rst b/README.rst
index 4bc5214480347..76d70b3e2d6d1 100644
--- a/README.rst
+++ b/README.rst
@@ -1,5 +1,5 @@
-This is Python version 3.8.0 release candidate 1
-=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
+This is Python version 3.8.0
+=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D
=20
 .. image:: https://travis-ci.org/python/cpython.svg?branch=3D3.8
    :alt: CPython build status on Travis CI


From webhook-mailer at python.org  Fri Oct 18 03:14:45 2019
From: webhook-mailer at python.org (=?utf-8?q?St=C3=A9phane?= Wirtel)
Date: Fri, 18 Oct 2019 07:14:45 -0000
Subject: [Python-checkins] (no subject)
Message-ID: <mailman.583.1599367021.9553.python-checkins@python.org>




To: python-checkins at python.org
Subject: Doc: Add missing entry for functools.cached_property (GH-16803)
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

https://github.com/python/cpython/commit/93b81e1fcbeb61c1b49ac2fa52c5a0dff929=
940b
commit: 93b81e1fcbeb61c1b49ac2fa52c5a0dff929940b
branch: master
author: St=C3=A9phane Wirtel <stephane at wirtel.be>
committer: GitHub <noreply at github.com>
date: 2019-10-18T09:14:18+02:00
summary:

Doc: Add missing entry for functools.cached_property (GH-16803)

files:
M Doc/whatsnew/3.8.rst

diff --git a/Doc/whatsnew/3.8.rst b/Doc/whatsnew/3.8.rst
index 85351e10e6994..fef712d2ab296 100644
--- a/Doc/whatsnew/3.8.rst
+++ b/Doc/whatsnew/3.8.rst
@@ -683,6 +683,22 @@ than as a function returning a decorator.  So both of th=
ese are now supported::
=20
 (Contributed by Raymond Hettinger in :issue:`36772`.)
=20
+Added a new :func:`functools.cached_property` decorator, for computed proper=
ties
+cached for the life of the instance. ::
+
+   import functools
+   import statistics
+
+   class Dataset:
+      def __init__(self, sequence_of_numbers):
+         self.data =3D sequence_of_numbers
+
+      @functools.cached_property
+      def variance(self):
+         return statistics.variance(self.data)
+
+(Contributed by Carl Meyer in :issue:`21145`)
+
=20
 gc
 --