From webhook-mailer at python.org Fri Apr 1 04:18:09 2022 From: webhook-mailer at python.org (vstinner) Date: Fri, 01 Apr 2022 08:18:09 -0000 Subject: [Python-checkins] bpo-46850: Move _PyEval_EvalFrameDefault() to internal C API (GH-32052) Message-ID: https://github.com/python/cpython/commit/b9a5522dd952125a99ff554f01f311cae25f5a91 commit: b9a5522dd952125a99ff554f01f311cae25f5a91 branch: main author: Victor Stinner committer: vstinner date: 2022-04-01T10:17:57+02:00 summary: bpo-46850: Move _PyEval_EvalFrameDefault() to internal C API (GH-32052) Move the private undocumented _PyEval_EvalFrameDefault() function to the internal C API. The function now uses the _PyInterpreterFrame type which is part of the internal C API. files: A Misc/NEWS.d/next/C API/2022-03-22-16-48-02.bpo-46850.7M5dO7.rst M Doc/whatsnew/3.11.rst M Include/cpython/ceval.h M Include/internal/pycore_ceval.h diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index 16715c32502e4..15f765dbe806e 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -1097,6 +1097,11 @@ Porting to Python 3.11 * Distributors are encouraged to build Python with the optimized Blake2 library `libb2`_. +* Move the private undocumented ``_PyEval_EvalFrameDefault()`` function to the + internal C API. The function now uses the ``_PyInterpreterFrame`` type which + is part of the internal C API. + (Contributed by Victor Stinner in :issue:`46850`.) + Deprecated ---------- diff --git a/Include/cpython/ceval.h b/Include/cpython/ceval.h index 9d4eeafb427eb..65aae2d669a52 100644 --- a/Include/cpython/ceval.h +++ b/Include/cpython/ceval.h @@ -15,8 +15,6 @@ PyAPI_FUNC(PyObject *) _PyEval_GetBuiltinId(_Py_Identifier *); flag was set, else return 0. */ PyAPI_FUNC(int) PyEval_MergeCompilerFlags(PyCompilerFlags *cf); -PyAPI_FUNC(PyObject *) _PyEval_EvalFrameDefault(PyThreadState *tstate, struct _PyInterpreterFrame *f, int exc); - PyAPI_FUNC(void) _PyEval_SetSwitchInterval(unsigned long microseconds); PyAPI_FUNC(unsigned long) _PyEval_GetSwitchInterval(void); diff --git a/Include/internal/pycore_ceval.h b/Include/internal/pycore_ceval.h index 45d26a37a34c6..b29b496ee3f58 100644 --- a/Include/internal/pycore_ceval.h +++ b/Include/internal/pycore_ceval.h @@ -59,6 +59,11 @@ extern PyObject* _PyEval_BuiltinsFromGlobals( PyObject *globals); +PyAPI_FUNC(PyObject *) _PyEval_EvalFrameDefault( + PyThreadState *tstate, + struct _PyInterpreterFrame *frame, + int throwflag); + static inline PyObject* _PyEval_EvalFrame(PyThreadState *tstate, struct _PyInterpreterFrame *frame, int throwflag) { diff --git a/Misc/NEWS.d/next/C API/2022-03-22-16-48-02.bpo-46850.7M5dO7.rst b/Misc/NEWS.d/next/C API/2022-03-22-16-48-02.bpo-46850.7M5dO7.rst new file mode 100644 index 0000000000000..1519ac7890e37 --- /dev/null +++ b/Misc/NEWS.d/next/C API/2022-03-22-16-48-02.bpo-46850.7M5dO7.rst @@ -0,0 +1,3 @@ +Move the private undocumented ``_PyEval_EvalFrameDefault()`` function to the +internal C API. The function now uses the ``_PyInterpreterFrame`` type which is +part of the internal C API. Patch by Victor Stinner. From webhook-mailer at python.org Fri Apr 1 04:45:06 2022 From: webhook-mailer at python.org (tiran) Date: Fri, 01 Apr 2022 08:45:06 -0000 Subject: [Python-checkins] [3.10] bpo-47182: Fix crash by named unicode characters after interpreter reinitialization (GH-32212) (GH-32216) Message-ID: https://github.com/python/cpython/commit/55d5c96c57738766eb6f3b5ccfa6599d5f094c18 commit: 55d5c96c57738766eb6f3b5ccfa6599d5f094c18 branch: 3.10 author: Christian Heimes committer: tiran date: 2022-04-01T10:44:56+02:00 summary: [3.10] bpo-47182: Fix crash by named unicode characters after interpreter reinitialization (GH-32212) (GH-32216) Co-authored-by: Christian Heimes files: A Misc/NEWS.d/next/Core and Builtins/2022-03-31-15-37-02.bpo-47182.e_4SsC.rst M Objects/unicodeobject.c diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-03-31-15-37-02.bpo-47182.e_4SsC.rst b/Misc/NEWS.d/next/Core and Builtins/2022-03-31-15-37-02.bpo-47182.e_4SsC.rst new file mode 100644 index 0000000000000..08036bc680933 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2022-03-31-15-37-02.bpo-47182.e_4SsC.rst @@ -0,0 +1,2 @@ +Fix a crash when using a named unicode character like ``"\N{digit nine}"`` +after the main interpreter has been initialized a second time. diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c index 077cf8d7f4560..377fa6c8e2a28 100644 --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -16352,6 +16352,9 @@ _PyUnicode_Fini(PyInterpreterState *interp) if (_Py_IsMainInterpreter(interp)) { // _PyUnicode_ClearInterned() must be called before _PyUnicode_Fini() assert(interned == NULL); + // bpo-47182: force a unicodedata CAPI capsule re-import on + // subsequent initialization of main interpreter. + ucnhash_capi = NULL; } _PyUnicode_FiniEncodings(&state->fs_codec); From webhook-mailer at python.org Fri Apr 1 04:55:14 2022 From: webhook-mailer at python.org (vstinner) Date: Fri, 01 Apr 2022 08:55:14 -0000 Subject: [Python-checkins] bpo-46850: Move _PyInterpreterState_SetEvalFrameFunc() to internal C API (GH-32054) Message-ID: https://github.com/python/cpython/commit/f877b40e3f7e0d97878884d80fbec879a85ab7e8 commit: f877b40e3f7e0d97878884d80fbec879a85ab7e8 branch: main author: Victor Stinner committer: vstinner date: 2022-04-01T10:55:00+02:00 summary: bpo-46850: Move _PyInterpreterState_SetEvalFrameFunc() to internal C API (GH-32054) Move the private _PyFrameEvalFunction type, and private _PyInterpreterState_GetEvalFrameFunc() and _PyInterpreterState_SetEvalFrameFunc() functions to the internal C API. The _PyFrameEvalFunction callback function type now uses the _PyInterpreterFrame type which is part of the internal C API. Update the _PyFrameEvalFunction documentation. files: A Misc/NEWS.d/next/C API/2022-03-22-16-59-34.bpo-46850.lmEKLy.rst M Doc/c-api/init.rst M Doc/whatsnew/3.11.rst M Include/cpython/pystate.h M Include/internal/pycore_interp.h M Include/internal/pycore_pystate.h diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst index 322b9e4d251e7..152d4c8e5036b 100644 --- a/Doc/c-api/init.rst +++ b/Doc/c-api/init.rst @@ -1228,18 +1228,25 @@ All of the following functions must be called after :c:func:`Py_Initialize`. .. versionadded:: 3.8 -.. c:type:: PyObject* (*_PyFrameEvalFunction)(PyThreadState *tstate, PyFrameObject *frame, int throwflag) +.. c:type:: PyObject* (*_PyFrameEvalFunction)(PyThreadState *tstate, _PyInterpreterFrame *frame, int throwflag) + + Internal C API. Type of a frame evaluation function. The *throwflag* parameter is used by the ``throw()`` method of generators: if non-zero, handle the current exception. + .. versionchanged:: 3.11 + The second parameter type becomes ``_PyInterpreterFrame``. + .. versionchanged:: 3.9 The function now takes a *tstate* parameter. .. c:function:: _PyFrameEvalFunction _PyInterpreterState_GetEvalFrameFunc(PyInterpreterState *interp) + Internal C API. + Get the frame evaluation function. See the :pep:`523` "Adding a frame evaluation API to CPython". @@ -1248,6 +1255,8 @@ All of the following functions must be called after :c:func:`Py_Initialize`. .. c:function:: void _PyInterpreterState_SetEvalFrameFunc(PyInterpreterState *interp, _PyFrameEvalFunction eval_frame) + Internal C API. + Set the frame evaluation function. See the :pep:`523` "Adding a frame evaluation API to CPython". diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index 15f765dbe806e..15808679f7fd5 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -1102,6 +1102,12 @@ Porting to Python 3.11 is part of the internal C API. (Contributed by Victor Stinner in :issue:`46850`.) +* Move the private ``_PyFrameEvalFunction`` type, and private + ``_PyInterpreterState_GetEvalFrameFunc()`` and + ``_PyInterpreterState_SetEvalFrameFunc()`` functions to the internal C API. + The ``_PyFrameEvalFunction`` callback function type now uses the + ``_PyInterpreterFrame`` type which is part of the internal C API. + (Contributed by Victor Stinner in :issue:`46850`.) Deprecated ---------- diff --git a/Include/cpython/pystate.h b/Include/cpython/pystate.h index 1af21a2c947d9..e346d744b6f12 100644 --- a/Include/cpython/pystate.h +++ b/Include/cpython/pystate.h @@ -259,16 +259,6 @@ PyAPI_FUNC(PyThreadState *) PyInterpreterState_ThreadHead(PyInterpreterState *); PyAPI_FUNC(PyThreadState *) PyThreadState_Next(PyThreadState *); PyAPI_FUNC(void) PyThreadState_DeleteCurrent(void); -/* Frame evaluation API */ - -typedef PyObject* (*_PyFrameEvalFunction)(PyThreadState *tstate, struct _PyInterpreterFrame *, int); - -PyAPI_FUNC(_PyFrameEvalFunction) _PyInterpreterState_GetEvalFrameFunc( - PyInterpreterState *interp); -PyAPI_FUNC(void) _PyInterpreterState_SetEvalFrameFunc( - PyInterpreterState *interp, - _PyFrameEvalFunction eval_frame); - PyAPI_FUNC(const PyConfig*) _PyInterpreterState_GetConfig(PyInterpreterState *interp); /* Get a copy of the current interpreter configuration. diff --git a/Include/internal/pycore_interp.h b/Include/internal/pycore_interp.h index d55627908a28f..592d438bcf1d2 100644 --- a/Include/internal/pycore_interp.h +++ b/Include/internal/pycore_interp.h @@ -17,9 +17,9 @@ extern "C" { #include "pycore_dict.h" // struct _Py_dict_state #include "pycore_exceptions.h" // struct _Py_exc_state #include "pycore_floatobject.h" // struct _Py_float_state +#include "pycore_gc.h" // struct _gc_runtime_state #include "pycore_genobject.h" // struct _Py_async_gen_state #include "pycore_gil.h" // struct _gil_runtime_state -#include "pycore_gc.h" // struct _gc_runtime_state #include "pycore_list.h" // struct _Py_list_state #include "pycore_tuple.h" // struct _Py_tuple_state #include "pycore_typeobject.h" // struct type_cache @@ -71,6 +71,20 @@ struct atexit_state { }; +/* Frame evaluation API (PEP 523) */ + +typedef PyObject* (*_PyFrameEvalFunction) ( + PyThreadState *tstate, + struct _PyInterpreterFrame *frame, + int throwflag); + +PyAPI_FUNC(_PyFrameEvalFunction) _PyInterpreterState_GetEvalFrameFunc( + PyInterpreterState *interp); +PyAPI_FUNC(void) _PyInterpreterState_SetEvalFrameFunc( + PyInterpreterState *interp, + _PyFrameEvalFunction eval_frame); + + /* interpreter state */ /* PyInterpreterState holds the global state for one of the runtime's diff --git a/Include/internal/pycore_pystate.h b/Include/internal/pycore_pystate.h index c4bc53c707fda..c463347dbf06a 100644 --- a/Include/internal/pycore_pystate.h +++ b/Include/internal/pycore_pystate.h @@ -8,7 +8,7 @@ extern "C" { # error "this header requires Py_BUILD_CORE define" #endif -#include "pycore_runtime.h" /* PyRuntimeState */ +#include "pycore_runtime.h" // _PyRuntime /* Check if the current thread is the main thread. diff --git a/Misc/NEWS.d/next/C API/2022-03-22-16-59-34.bpo-46850.lmEKLy.rst b/Misc/NEWS.d/next/C API/2022-03-22-16-59-34.bpo-46850.lmEKLy.rst new file mode 100644 index 0000000000000..d32cc3448721e --- /dev/null +++ b/Misc/NEWS.d/next/C API/2022-03-22-16-59-34.bpo-46850.lmEKLy.rst @@ -0,0 +1,6 @@ +Move the private ``_PyFrameEvalFunction`` type, and private +``_PyInterpreterState_GetEvalFrameFunc()`` and +``_PyInterpreterState_SetEvalFrameFunc()`` functions to the internal C API. The +``_PyFrameEvalFunction`` callback function type now uses the +``_PyInterpreterFrame`` type which is part of the internal C API. Patch by +Victor Stinner. From webhook-mailer at python.org Fri Apr 1 06:24:14 2022 From: webhook-mailer at python.org (markshannon) Date: Fri, 01 Apr 2022 10:24:14 -0000 Subject: [Python-checkins] bpo-47009: Streamline list.append for the common case (GH-31864) Message-ID: https://github.com/python/cpython/commit/a0ea7a116ce52a178c02d42b684089758bd7f355 commit: a0ea7a116ce52a178c02d42b684089758bd7f355 branch: main author: Dennis Sweeney <36520290+sweeneyde at users.noreply.github.com> committer: markshannon date: 2022-04-01T11:23:42+01:00 summary: bpo-47009: Streamline list.append for the common case (GH-31864) files: A Misc/NEWS.d/next/Core and Builtins/2022-03-14-09-45-10.bpo-47009.ZI05b5.rst M Include/internal/pycore_list.h M Objects/listobject.c M Python/ceval.c diff --git a/Include/internal/pycore_list.h b/Include/internal/pycore_list.h index 0717a1f9563a2..860dce1fd5d39 100644 --- a/Include/internal/pycore_list.h +++ b/Include/internal/pycore_list.h @@ -37,6 +37,24 @@ struct _Py_list_state { #define _PyList_ITEMS(op) (_PyList_CAST(op)->ob_item) +extern int +_PyList_AppendTakeRefListResize(PyListObject *self, PyObject *newitem); + +static inline int +_PyList_AppendTakeRef(PyListObject *self, PyObject *newitem) +{ + assert(self != NULL && newitem != NULL); + assert(PyList_Check(self)); + Py_ssize_t len = PyList_GET_SIZE(self); + Py_ssize_t allocated = self->allocated; + assert((size_t)len + 1 < PY_SSIZE_T_MAX); + if (allocated > len) { + PyList_SET_ITEM(self, len, newitem); + Py_SET_SIZE(self, len + 1); + return 0; + } + return _PyList_AppendTakeRefListResize(self, newitem); +} #ifdef __cplusplus } diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-03-14-09-45-10.bpo-47009.ZI05b5.rst b/Misc/NEWS.d/next/Core and Builtins/2022-03-14-09-45-10.bpo-47009.ZI05b5.rst new file mode 100644 index 0000000000000..0c65c34d310ec --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2022-03-14-09-45-10.bpo-47009.ZI05b5.rst @@ -0,0 +1 @@ +Improved the performance of :meth:`list.append()` and list comprehensions by optimizing for the common case, where no resize is needed. Patch by Dennis Sweeney. diff --git a/Objects/listobject.c b/Objects/listobject.c index d50633d2b3132..ccb9b91ba930d 100644 --- a/Objects/listobject.c +++ b/Objects/listobject.c @@ -301,26 +301,27 @@ PyList_Insert(PyObject *op, Py_ssize_t where, PyObject *newitem) return ins1((PyListObject *)op, where, newitem); } -static int -app1(PyListObject *self, PyObject *v) +/* internal, used by _PyList_AppendTakeRef */ +int +_PyList_AppendTakeRefListResize(PyListObject *self, PyObject *newitem) { - Py_ssize_t n = PyList_GET_SIZE(self); - - assert (v != NULL); - assert((size_t)n + 1 < PY_SSIZE_T_MAX); - if (list_resize(self, n+1) < 0) + Py_ssize_t len = PyList_GET_SIZE(self); + assert(self->allocated == -1 || self->allocated == len); + if (list_resize(self, len + 1) < 0) { + Py_DECREF(newitem); return -1; - - Py_INCREF(v); - PyList_SET_ITEM(self, n, v); + } + PyList_SET_ITEM(self, len, newitem); return 0; } int PyList_Append(PyObject *op, PyObject *newitem) { - if (PyList_Check(op) && (newitem != NULL)) - return app1((PyListObject *)op, newitem); + if (PyList_Check(op) && (newitem != NULL)) { + Py_INCREF(newitem); + return _PyList_AppendTakeRef((PyListObject *)op, newitem); + } PyErr_BadInternalCall(); return -1; } @@ -844,9 +845,10 @@ static PyObject * list_append(PyListObject *self, PyObject *object) /*[clinic end generated code: output=7c096003a29c0eae input=43a3fe48a7066e91]*/ { - if (app1(self, object) == 0) - Py_RETURN_NONE; - return NULL; + if (_PyList_AppendTakeRef(self, Py_NewRef(object)) < 0) { + return NULL; + } + Py_RETURN_NONE; } /*[clinic input] @@ -963,9 +965,7 @@ list_extend(PyListObject *self, PyObject *iterable) Py_SET_SIZE(self, Py_SIZE(self) + 1); } else { - int status = app1(self, item); - Py_DECREF(item); /* append creates a new ref */ - if (status < 0) + if (_PyList_AppendTakeRef(self, item) < 0) goto error; } } diff --git a/Python/ceval.c b/Python/ceval.c index 8f73ea1c01ac5..8c1f21b086da9 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -2213,10 +2213,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int TARGET(LIST_APPEND) { PyObject *v = POP(); PyObject *list = PEEK(oparg); - int err; - err = PyList_Append(list, v); - Py_DECREF(v); - if (err != 0) + if (_PyList_AppendTakeRef((PyListObject *)list, v) < 0) goto error; PREDICT(JUMP_BACKWARD_QUICK); DISPATCH(); @@ -5044,14 +5041,12 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int DEOPT_IF(!PyList_Check(list), PRECALL); STAT_INC(PRECALL, hit); SKIP_CALL(); - PyObject *arg = TOP(); - int err = PyList_Append(list, arg); - if (err) { + PyObject *arg = POP(); + if (_PyList_AppendTakeRef((PyListObject *)list, arg) < 0) { goto error; } - Py_DECREF(arg); Py_DECREF(list); - STACK_SHRINK(2); + STACK_SHRINK(1); Py_INCREF(Py_None); SET_TOP(Py_None); Py_DECREF(callable); From webhook-mailer at python.org Fri Apr 1 06:42:55 2022 From: webhook-mailer at python.org (markshannon) Date: Fri, 01 Apr 2022 10:42:55 -0000 Subject: [Python-checkins] bpo-46841: Avoid unnecessary allocations in code object comparisons (GH-32222) Message-ID: https://github.com/python/cpython/commit/bd2e47c8830d1b2869f2b4345945a5e0c3b4e3fb commit: bd2e47c8830d1b2869f2b4345945a5e0c3b4e3fb branch: main author: Brandt Bucher committer: markshannon date: 2022-04-01T11:42:46+01:00 summary: bpo-46841: Avoid unnecessary allocations in code object comparisons (GH-32222) files: A Misc/NEWS.d/next/Core and Builtins/2022-03-31-15-57-42.bpo-46841.U-25Z6.rst M Objects/codeobject.c diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-03-31-15-57-42.bpo-46841.U-25Z6.rst b/Misc/NEWS.d/next/Core and Builtins/2022-03-31-15-57-42.bpo-46841.U-25Z6.rst new file mode 100644 index 0000000000000..42711cd40f396 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2022-03-31-15-57-42.bpo-46841.U-25Z6.rst @@ -0,0 +1 @@ +Avoid unnecessary allocations when comparing code objects. diff --git a/Objects/codeobject.c b/Objects/codeobject.c index 224493edb19ea..987cdef3c90d4 100644 --- a/Objects/codeobject.c +++ b/Objects/codeobject.c @@ -1398,21 +1398,21 @@ code_richcompare(PyObject *self, PyObject *other, int op) if (!eq) goto unequal; eq = co->co_firstlineno == cp->co_firstlineno; if (!eq) goto unequal; - PyObject *co_code = _PyCode_GetCode(co); - if (co_code == NULL) { - return NULL; - } - PyObject *cp_code = _PyCode_GetCode(cp); - if (cp_code == NULL) { - Py_DECREF(co_code); - return NULL; - } - eq = PyObject_RichCompareBool(co_code, cp_code, Py_EQ); - Py_DECREF(co_code); - Py_DECREF(cp_code); - if (eq <= 0) { + eq = Py_SIZE(co) == Py_SIZE(cp); + if (!eq) { goto unequal; } + for (int i = 0; i < Py_SIZE(co); i++) { + _Py_CODEUNIT co_instr = _PyCode_CODE(co)[i]; + _Py_CODEUNIT cp_instr = _PyCode_CODE(cp)[i]; + _Py_SET_OPCODE(co_instr, _PyOpcode_Deopt[_Py_OPCODE(co_instr)]); + _Py_SET_OPCODE(cp_instr, _PyOpcode_Deopt[_Py_OPCODE(cp_instr)]); + eq = co_instr == cp_instr; + if (!eq) { + goto unequal; + } + i += _PyOpcode_Caches[_Py_OPCODE(co_instr)]; + } /* compare constants */ consts1 = _PyCode_ConstantKey(co->co_consts); From webhook-mailer at python.org Fri Apr 1 07:29:29 2022 From: webhook-mailer at python.org (markshannon) Date: Fri, 01 Apr 2022 11:29:29 -0000 Subject: [Python-checkins] bpo-46841: Use a `bytes` object for `_co_code_adaptive` (GH-32205) Message-ID: https://github.com/python/cpython/commit/ae9de82e321581e1906c6ef2a7ad83ab30ae3325 commit: ae9de82e321581e1906c6ef2a7ad83ab30ae3325 branch: main author: Brandt Bucher committer: markshannon date: 2022-04-01T12:28:50+01:00 summary: bpo-46841: Use a `bytes` object for `_co_code_adaptive` (GH-32205) files: M Objects/codeobject.c diff --git a/Objects/codeobject.c b/Objects/codeobject.c index 987cdef3c90d4..7d50b40ec4536 100644 --- a/Objects/codeobject.c +++ b/Objects/codeobject.c @@ -1523,8 +1523,8 @@ code_getfreevars(PyCodeObject *code, void *closure) static PyObject * code_getcodeadaptive(PyCodeObject *code, void *closure) { - return PyMemoryView_FromMemory(code->co_code_adaptive, _PyCode_NBYTES(code), - PyBUF_READ); + return PyBytes_FromStringAndSize(code->co_code_adaptive, + _PyCode_NBYTES(code)); } static PyObject * From webhook-mailer at python.org Fri Apr 1 08:59:49 2022 From: webhook-mailer at python.org (markshannon) Date: Fri, 01 Apr 2022 12:59:49 -0000 Subject: [Python-checkins] bpo-47186: Replace JUMP_IF_NOT_EXC_MATCH by CHECK_EXC_MATCH + jump (GH-32231) Message-ID: https://github.com/python/cpython/commit/04e07c258f4f2ac85e25355242a113f98a706f04 commit: 04e07c258f4f2ac85e25355242a113f98a706f04 branch: main author: Irit Katriel <1055913+iritkatriel at users.noreply.github.com> committer: markshannon date: 2022-04-01T13:59:38+01:00 summary: bpo-47186: Replace JUMP_IF_NOT_EXC_MATCH by CHECK_EXC_MATCH + jump (GH-32231) files: A Misc/NEWS.d/next/Core and Builtins/2022-04-01-11-53-59.bpo-47186.RBCPk8.rst M Doc/library/dis.rst M Doc/whatsnew/3.11.rst M Include/opcode.h M Lib/importlib/_bootstrap_external.py M Lib/opcode.py M Lib/test/test_dis.py M Objects/frameobject.c M Python/ceval.c M Python/compile.c M Python/opcode_targets.h diff --git a/Doc/library/dis.rst b/Doc/library/dis.rst index d1a0cecd82841..b364e3f031b3d 100644 --- a/Doc/library/dis.rst +++ b/Doc/library/dis.rst @@ -628,6 +628,12 @@ iterations of the loop. .. versionadded:: 3.11 +.. opcode:: CHECK_EXC_MATCH + + Performs exception matching for ``except``. Tests whether the TOS1 is an exception + matching TOS. Pops TOS and pushes the boolean result of the test. + + .. versionadded:: 3.11 .. opcode:: WITH_EXCEPT_START @@ -916,18 +922,6 @@ iterations of the loop. .. versionadded:: 3.1 -.. opcode:: JUMP_IF_NOT_EXC_MATCH (target) - - Performs exception matching for ``except``. - Tests whether the second value on the stack is an exception matching TOS, - and jumps if it is not. Pops one value from the stack. - - .. versionadded:: 3.9 - - .. versionchanged:: 3.11 - This opcode no longer pops the active exception. - - .. opcode:: JUMP_IF_NOT_EG_MATCH (target) Performs exception matching for ``except*``. Applies ``split(TOS)`` on diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index 15808679f7fd5..dc2d4b0937c77 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -520,7 +520,8 @@ CPython bytecode changes * Add :opcode:`POP_JUMP_IF_NOT_NONE` and :opcode:`POP_JUMP_IF_NONE` opcodes to speed up conditional jumps. -* :opcode:`JUMP_IF_NOT_EXC_MATCH` no longer pops the active exception. +* Replaced :opcode:`JUMP_IF_NOT_EXC_MATCH` by :opcode:`CHECK_EXC_MATCH` which + performs the check but does not jump. * Replaced :opcode:`JUMP_ABSOLUTE` by the relative :opcode:`JUMP_BACKWARD`. diff --git a/Include/opcode.h b/Include/opcode.h index 3a7db438ede1f..c82d1fd1fae2b 100644 --- a/Include/opcode.h +++ b/Include/opcode.h @@ -21,6 +21,7 @@ extern "C" { #define MATCH_SEQUENCE 32 #define MATCH_KEYS 33 #define PUSH_EXC_INFO 35 +#define CHECK_EXC_MATCH 36 #define WITH_EXCEPT_START 49 #define GET_AITER 50 #define GET_ANEXT 51 @@ -74,7 +75,6 @@ extern "C" { #define CONTAINS_OP 118 #define RERAISE 119 #define COPY 120 -#define JUMP_IF_NOT_EXC_MATCH 121 #define BINARY_OP 122 #define SEND 123 #define LOAD_FAST 124 @@ -136,39 +136,39 @@ extern "C" { #define COMPARE_OP_INT_JUMP 28 #define COMPARE_OP_STR_JUMP 29 #define JUMP_BACKWARD_QUICK 34 -#define LOAD_ATTR_ADAPTIVE 36 -#define LOAD_ATTR_INSTANCE_VALUE 37 -#define LOAD_ATTR_MODULE 38 -#define LOAD_ATTR_SLOT 39 -#define LOAD_ATTR_WITH_HINT 40 -#define LOAD_CONST__LOAD_FAST 41 -#define LOAD_FAST__LOAD_CONST 42 -#define LOAD_FAST__LOAD_FAST 43 -#define LOAD_GLOBAL_ADAPTIVE 44 -#define LOAD_GLOBAL_BUILTIN 45 -#define LOAD_GLOBAL_MODULE 46 -#define LOAD_METHOD_ADAPTIVE 47 -#define LOAD_METHOD_CLASS 48 -#define LOAD_METHOD_MODULE 55 -#define LOAD_METHOD_NO_DICT 56 -#define LOAD_METHOD_WITH_DICT 57 -#define LOAD_METHOD_WITH_VALUES 58 -#define PRECALL_ADAPTIVE 59 -#define PRECALL_BOUND_METHOD 62 -#define PRECALL_BUILTIN_CLASS 63 -#define PRECALL_BUILTIN_FAST_WITH_KEYWORDS 64 -#define PRECALL_METHOD_DESCRIPTOR_FAST_WITH_KEYWORDS 65 -#define PRECALL_NO_KW_BUILTIN_FAST 66 -#define PRECALL_NO_KW_BUILTIN_O 67 -#define PRECALL_NO_KW_ISINSTANCE 72 -#define PRECALL_NO_KW_LEN 73 -#define PRECALL_NO_KW_LIST_APPEND 76 -#define PRECALL_NO_KW_METHOD_DESCRIPTOR_FAST 77 -#define PRECALL_NO_KW_METHOD_DESCRIPTOR_NOARGS 78 -#define PRECALL_NO_KW_METHOD_DESCRIPTOR_O 79 -#define PRECALL_NO_KW_STR_1 80 -#define PRECALL_NO_KW_TUPLE_1 81 -#define PRECALL_NO_KW_TYPE_1 113 +#define LOAD_ATTR_ADAPTIVE 37 +#define LOAD_ATTR_INSTANCE_VALUE 38 +#define LOAD_ATTR_MODULE 39 +#define LOAD_ATTR_SLOT 40 +#define LOAD_ATTR_WITH_HINT 41 +#define LOAD_CONST__LOAD_FAST 42 +#define LOAD_FAST__LOAD_CONST 43 +#define LOAD_FAST__LOAD_FAST 44 +#define LOAD_GLOBAL_ADAPTIVE 45 +#define LOAD_GLOBAL_BUILTIN 46 +#define LOAD_GLOBAL_MODULE 47 +#define LOAD_METHOD_ADAPTIVE 48 +#define LOAD_METHOD_CLASS 55 +#define LOAD_METHOD_MODULE 56 +#define LOAD_METHOD_NO_DICT 57 +#define LOAD_METHOD_WITH_DICT 58 +#define LOAD_METHOD_WITH_VALUES 59 +#define PRECALL_ADAPTIVE 62 +#define PRECALL_BOUND_METHOD 63 +#define PRECALL_BUILTIN_CLASS 64 +#define PRECALL_BUILTIN_FAST_WITH_KEYWORDS 65 +#define PRECALL_METHOD_DESCRIPTOR_FAST_WITH_KEYWORDS 66 +#define PRECALL_NO_KW_BUILTIN_FAST 67 +#define PRECALL_NO_KW_BUILTIN_O 72 +#define PRECALL_NO_KW_ISINSTANCE 73 +#define PRECALL_NO_KW_LEN 76 +#define PRECALL_NO_KW_LIST_APPEND 77 +#define PRECALL_NO_KW_METHOD_DESCRIPTOR_FAST 78 +#define PRECALL_NO_KW_METHOD_DESCRIPTOR_NOARGS 79 +#define PRECALL_NO_KW_METHOD_DESCRIPTOR_O 80 +#define PRECALL_NO_KW_STR_1 81 +#define PRECALL_NO_KW_TUPLE_1 113 +#define PRECALL_NO_KW_TYPE_1 121 #define PRECALL_PYFUNC 141 #define RESUME_QUICK 143 #define STORE_ATTR_ADAPTIVE 150 @@ -205,7 +205,7 @@ static const uint32_t _PyOpcode_Jump[8] = { 0U, 0U, 536870912U, - 2316156928U, + 2282602496U, 4163U, 0U, 0U, @@ -259,6 +259,7 @@ const uint8_t _PyOpcode_Deopt[256] = { [CALL_FUNCTION_EX] = CALL_FUNCTION_EX, [CALL_PY_EXACT_ARGS] = CALL, [CALL_PY_WITH_DEFAULTS] = CALL, + [CHECK_EXC_MATCH] = CHECK_EXC_MATCH, [COMPARE_OP] = COMPARE_OP, [COMPARE_OP_ADAPTIVE] = COMPARE_OP, [COMPARE_OP_FLOAT_JUMP] = COMPARE_OP, @@ -294,7 +295,6 @@ const uint8_t _PyOpcode_Deopt[256] = { [JUMP_FORWARD] = JUMP_FORWARD, [JUMP_IF_FALSE_OR_POP] = JUMP_IF_FALSE_OR_POP, [JUMP_IF_NOT_EG_MATCH] = JUMP_IF_NOT_EG_MATCH, - [JUMP_IF_NOT_EXC_MATCH] = JUMP_IF_NOT_EXC_MATCH, [JUMP_IF_TRUE_OR_POP] = JUMP_IF_TRUE_OR_POP, [JUMP_NO_INTERRUPT] = JUMP_NO_INTERRUPT, [KW_NAMES] = KW_NAMES, diff --git a/Lib/importlib/_bootstrap_external.py b/Lib/importlib/_bootstrap_external.py index 744fefd5e21e7..39a348aecef6c 100644 --- a/Lib/importlib/_bootstrap_external.py +++ b/Lib/importlib/_bootstrap_external.py @@ -397,6 +397,7 @@ def _write_atomic(path, data, mode=0o666): # Python 3.11a6 3487 (Remove the adaptive "oparg counter" mechanism) # Python 3.11a6 3488 (LOAD_GLOBAL can push additional NULL) # Python 3.11a6 3489 (Add JUMP_BACKWARD, remove JUMP_ABSOLUTE) +# Python 3.11a6 3490 (remove JUMP_IF_NOT_EXC_MATCH, add CHECK_EXC_MATCH) # Python 3.12 will start with magic number 3500 @@ -411,7 +412,7 @@ def _write_atomic(path, data, mode=0o666): # Whenever MAGIC_NUMBER is changed, the ranges in the magic_values array # in PC/launcher.c must also be updated. -MAGIC_NUMBER = (3489).to_bytes(2, 'little') + b'\r\n' +MAGIC_NUMBER = (3490).to_bytes(2, 'little') + b'\r\n' _RAW_MAGIC_NUMBER = int.from_bytes(MAGIC_NUMBER, 'little') # For import.c _PYCACHE = '__pycache__' diff --git a/Lib/opcode.py b/Lib/opcode.py index 6bc64177ac8fc..e993a5c1ff16a 100644 --- a/Lib/opcode.py +++ b/Lib/opcode.py @@ -76,6 +76,7 @@ def jabs_op(name, op, entries=0): def_op('MATCH_KEYS', 33) def_op('PUSH_EXC_INFO', 35) +def_op('CHECK_EXC_MATCH', 36) def_op('WITH_EXCEPT_START', 49) def_op('GET_AITER', 50) @@ -138,7 +139,6 @@ def jabs_op(name, op, entries=0): def_op('CONTAINS_OP', 118) def_op('RERAISE', 119) def_op('COPY', 120) -jabs_op('JUMP_IF_NOT_EXC_MATCH', 121) def_op('BINARY_OP', 122, 1) jrel_op('SEND', 123) # Number of bytes to skip def_op('LOAD_FAST', 124) # Local variable number diff --git a/Lib/test/test_dis.py b/Lib/test/test_dis.py index 99db8ba0ac3b9..544e1350cb7f9 100644 --- a/Lib/test/test_dis.py +++ b/Lib/test/test_dis.py @@ -372,7 +372,8 @@ def bug42562(): >> PUSH_EXC_INFO %3d LOAD_GLOBAL 0 (Exception) - JUMP_IF_NOT_EXC_MATCH 35 (to 70) + CHECK_EXC_MATCH + POP_JUMP_IF_FALSE 36 (to 72) STORE_FAST 0 (e) %3d LOAD_FAST 0 (e) @@ -683,8 +684,7 @@ def test_boundaries(self): def test_widths(self): for opcode, opname in enumerate(dis.opname): if opname in ('BUILD_MAP_UNPACK_WITH_CALL', - 'BUILD_TUPLE_UNPACK_WITH_CALL', - 'JUMP_IF_NOT_EXC_MATCH'): + 'BUILD_TUPLE_UNPACK_WITH_CALL'): continue with self.subTest(opname=opname): width = dis._OPNAME_WIDTH @@ -1299,47 +1299,48 @@ def _prepare_test_cases(): Instruction(opname='POP_EXCEPT', opcode=89, arg=None, argval=None, argrepr='', offset=322, starts_line=None, is_jump_target=False, positions=None), Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, argrepr='', offset=324, starts_line=None, is_jump_target=False, positions=None), Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, argrepr='', offset=326, starts_line=None, is_jump_target=False, positions=None), - Instruction(opname='JUMP_FORWARD', opcode=110, arg=30, argval=390, argrepr='to 390', offset=328, starts_line=None, is_jump_target=True, positions=None), + Instruction(opname='JUMP_FORWARD', opcode=110, arg=31, argval=392, argrepr='to 392', offset=328, starts_line=None, is_jump_target=True, positions=None), Instruction(opname='PUSH_EXC_INFO', opcode=35, arg=None, argval=None, argrepr='', offset=330, starts_line=None, is_jump_target=False, positions=None), Instruction(opname='LOAD_GLOBAL', opcode=116, arg=4, argval='ZeroDivisionError', argrepr='ZeroDivisionError', offset=332, starts_line=22, is_jump_target=False, positions=None), - Instruction(opname='JUMP_IF_NOT_EXC_MATCH', opcode=121, arg=191, argval=382, argrepr='to 382', offset=344, starts_line=None, is_jump_target=False, positions=None), - Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, argrepr='', offset=346, starts_line=None, is_jump_target=False, positions=None), - Instruction(opname='LOAD_GLOBAL', opcode=116, arg=3, argval='print', argrepr='NULL + print', offset=348, starts_line=23, is_jump_target=False, positions=None), - Instruction(opname='LOAD_CONST', opcode=100, arg=9, argval='Here we go, here we go, here we go...', argrepr="'Here we go, here we go, here we go...'", offset=360, starts_line=None, is_jump_target=False, positions=None), - Instruction(opname='PRECALL', opcode=166, arg=1, argval=1, argrepr='', offset=362, starts_line=None, is_jump_target=False, positions=None), - Instruction(opname='CALL', opcode=171, arg=1, argval=1, argrepr='', offset=366, starts_line=None, is_jump_target=False, positions=None), - Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, argrepr='', offset=376, starts_line=None, is_jump_target=False, positions=None), - Instruction(opname='POP_EXCEPT', opcode=89, arg=None, argval=None, argrepr='', offset=378, starts_line=None, is_jump_target=False, positions=None), - Instruction(opname='JUMP_FORWARD', opcode=110, arg=21, argval=424, argrepr='to 424', offset=380, starts_line=None, is_jump_target=False, positions=None), - Instruction(opname='RERAISE', opcode=119, arg=0, argval=0, argrepr='', offset=382, starts_line=22, is_jump_target=True, positions=None), - Instruction(opname='COPY', opcode=120, arg=3, argval=3, argrepr='', offset=384, starts_line=None, is_jump_target=False, positions=None), - Instruction(opname='POP_EXCEPT', opcode=89, arg=None, argval=None, argrepr='', offset=386, starts_line=None, is_jump_target=False, positions=None), - Instruction(opname='RERAISE', opcode=119, arg=1, argval=1, argrepr='', offset=388, starts_line=None, is_jump_target=False, positions=None), - Instruction(opname='LOAD_GLOBAL', opcode=116, arg=3, argval='print', argrepr='NULL + print', offset=390, starts_line=28, is_jump_target=True, positions=None), - Instruction(opname='LOAD_CONST', opcode=100, arg=10, argval="OK, now we're done", argrepr='"OK, now we\'re done"', offset=402, starts_line=None, is_jump_target=False, positions=None), - Instruction(opname='PRECALL', opcode=166, arg=1, argval=1, argrepr='', offset=404, starts_line=None, is_jump_target=False, positions=None), - Instruction(opname='CALL', opcode=171, arg=1, argval=1, argrepr='', offset=408, starts_line=None, is_jump_target=False, positions=None), - Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, argrepr='', offset=418, starts_line=None, is_jump_target=False, positions=None), - Instruction(opname='LOAD_CONST', opcode=100, arg=0, argval=None, argrepr='None', offset=420, starts_line=None, is_jump_target=False, positions=None), - Instruction(opname='RETURN_VALUE', opcode=83, arg=None, argval=None, argrepr='', offset=422, starts_line=None, is_jump_target=False, positions=None), - Instruction(opname='NOP', opcode=9, arg=None, argval=None, argrepr='', offset=424, starts_line=23, is_jump_target=True, positions=None), - Instruction(opname='LOAD_GLOBAL', opcode=116, arg=3, argval='print', argrepr='NULL + print', offset=426, starts_line=28, is_jump_target=False, positions=None), - Instruction(opname='LOAD_CONST', opcode=100, arg=10, argval="OK, now we're done", argrepr='"OK, now we\'re done"', offset=438, starts_line=None, is_jump_target=False, positions=None), - Instruction(opname='PRECALL', opcode=166, arg=1, argval=1, argrepr='', offset=440, starts_line=None, is_jump_target=False, positions=None), - Instruction(opname='CALL', opcode=171, arg=1, argval=1, argrepr='', offset=444, starts_line=None, is_jump_target=False, positions=None), - Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, argrepr='', offset=454, starts_line=None, is_jump_target=False, positions=None), - Instruction(opname='LOAD_CONST', opcode=100, arg=0, argval=None, argrepr='None', offset=456, starts_line=None, is_jump_target=False, positions=None), - Instruction(opname='RETURN_VALUE', opcode=83, arg=None, argval=None, argrepr='', offset=458, starts_line=None, is_jump_target=False, positions=None), - Instruction(opname='PUSH_EXC_INFO', opcode=35, arg=None, argval=None, argrepr='', offset=460, starts_line=None, is_jump_target=False, positions=None), - Instruction(opname='LOAD_GLOBAL', opcode=116, arg=3, argval='print', argrepr='NULL + print', offset=462, starts_line=None, is_jump_target=False, positions=None), - Instruction(opname='LOAD_CONST', opcode=100, arg=10, argval="OK, now we're done", argrepr='"OK, now we\'re done"', offset=474, starts_line=None, is_jump_target=False, positions=None), - Instruction(opname='PRECALL', opcode=166, arg=1, argval=1, argrepr='', offset=476, starts_line=None, is_jump_target=False, positions=None), - Instruction(opname='CALL', opcode=171, arg=1, argval=1, argrepr='', offset=480, starts_line=None, is_jump_target=False, positions=None), - Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, argrepr='', offset=490, starts_line=None, is_jump_target=False, positions=None), - Instruction(opname='RERAISE', opcode=119, arg=0, argval=0, argrepr='', offset=492, starts_line=None, is_jump_target=False, positions=None), - Instruction(opname='COPY', opcode=120, arg=3, argval=3, argrepr='', offset=494, starts_line=None, is_jump_target=False, positions=None), - Instruction(opname='POP_EXCEPT', opcode=89, arg=None, argval=None, argrepr='', offset=496, starts_line=None, is_jump_target=False, positions=None), - Instruction(opname='RERAISE', opcode=119, arg=1, argval=1, argrepr='', offset=498, starts_line=None, is_jump_target=False, positions=None), + Instruction(opname='CHECK_EXC_MATCH', opcode=36, arg=None, argval=None, argrepr='', offset=344, starts_line=None, is_jump_target=False, positions=None), + Instruction(opname='POP_JUMP_IF_FALSE', opcode=114, arg=192, argval=384, argrepr='to 384', offset=346, starts_line=None, is_jump_target=False, positions=None), + Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, argrepr='', offset=348, starts_line=None, is_jump_target=False, positions=None), + Instruction(opname='LOAD_GLOBAL', opcode=116, arg=3, argval='print', argrepr='NULL + print', offset=350, starts_line=23, is_jump_target=False, positions=None), + Instruction(opname='LOAD_CONST', opcode=100, arg=9, argval='Here we go, here we go, here we go...', argrepr="'Here we go, here we go, here we go...'", offset=362, starts_line=None, is_jump_target=False, positions=None), + Instruction(opname='PRECALL', opcode=166, arg=1, argval=1, argrepr='', offset=364, starts_line=None, is_jump_target=False, positions=None), + Instruction(opname='CALL', opcode=171, arg=1, argval=1, argrepr='', offset=368, starts_line=None, is_jump_target=False, positions=None), + Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, argrepr='', offset=378, starts_line=None, is_jump_target=False, positions=None), + Instruction(opname='POP_EXCEPT', opcode=89, arg=None, argval=None, argrepr='', offset=380, starts_line=None, is_jump_target=False, positions=None), + Instruction(opname='JUMP_FORWARD', opcode=110, arg=21, argval=426, argrepr='to 426', offset=382, starts_line=None, is_jump_target=False, positions=None), + Instruction(opname='RERAISE', opcode=119, arg=0, argval=0, argrepr='', offset=384, starts_line=22, is_jump_target=True, positions=None), + Instruction(opname='COPY', opcode=120, arg=3, argval=3, argrepr='', offset=386, starts_line=None, is_jump_target=False, positions=None), + Instruction(opname='POP_EXCEPT', opcode=89, arg=None, argval=None, argrepr='', offset=388, starts_line=None, is_jump_target=False, positions=None), + Instruction(opname='RERAISE', opcode=119, arg=1, argval=1, argrepr='', offset=390, starts_line=None, is_jump_target=False, positions=None), + Instruction(opname='LOAD_GLOBAL', opcode=116, arg=3, argval='print', argrepr='NULL + print', offset=392, starts_line=28, is_jump_target=True, positions=None), + Instruction(opname='LOAD_CONST', opcode=100, arg=10, argval="OK, now we're done", argrepr='"OK, now we\'re done"', offset=404, starts_line=None, is_jump_target=False, positions=None), + Instruction(opname='PRECALL', opcode=166, arg=1, argval=1, argrepr='', offset=406, starts_line=None, is_jump_target=False, positions=None), + Instruction(opname='CALL', opcode=171, arg=1, argval=1, argrepr='', offset=410, starts_line=None, is_jump_target=False, positions=None), + Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, argrepr='', offset=420, starts_line=None, is_jump_target=False, positions=None), + Instruction(opname='LOAD_CONST', opcode=100, arg=0, argval=None, argrepr='None', offset=422, starts_line=None, is_jump_target=False, positions=None), + Instruction(opname='RETURN_VALUE', opcode=83, arg=None, argval=None, argrepr='', offset=424, starts_line=None, is_jump_target=False, positions=None), + Instruction(opname='NOP', opcode=9, arg=None, argval=None, argrepr='', offset=426, starts_line=23, is_jump_target=True, positions=None), + Instruction(opname='LOAD_GLOBAL', opcode=116, arg=3, argval='print', argrepr='NULL + print', offset=428, starts_line=28, is_jump_target=False, positions=None), + Instruction(opname='LOAD_CONST', opcode=100, arg=10, argval="OK, now we're done", argrepr='"OK, now we\'re done"', offset=440, starts_line=None, is_jump_target=False, positions=None), + Instruction(opname='PRECALL', opcode=166, arg=1, argval=1, argrepr='', offset=442, starts_line=None, is_jump_target=False, positions=None), + Instruction(opname='CALL', opcode=171, arg=1, argval=1, argrepr='', offset=446, starts_line=None, is_jump_target=False, positions=None), + Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, argrepr='', offset=456, starts_line=None, is_jump_target=False, positions=None), + Instruction(opname='LOAD_CONST', opcode=100, arg=0, argval=None, argrepr='None', offset=458, starts_line=None, is_jump_target=False, positions=None), + Instruction(opname='RETURN_VALUE', opcode=83, arg=None, argval=None, argrepr='', offset=460, starts_line=None, is_jump_target=False, positions=None), + Instruction(opname='PUSH_EXC_INFO', opcode=35, arg=None, argval=None, argrepr='', offset=462, starts_line=None, is_jump_target=False, positions=None), + Instruction(opname='LOAD_GLOBAL', opcode=116, arg=3, argval='print', argrepr='NULL + print', offset=464, starts_line=None, is_jump_target=False, positions=None), + Instruction(opname='LOAD_CONST', opcode=100, arg=10, argval="OK, now we're done", argrepr='"OK, now we\'re done"', offset=476, starts_line=None, is_jump_target=False, positions=None), + Instruction(opname='PRECALL', opcode=166, arg=1, argval=1, argrepr='', offset=478, starts_line=None, is_jump_target=False, positions=None), + Instruction(opname='CALL', opcode=171, arg=1, argval=1, argrepr='', offset=482, starts_line=None, is_jump_target=False, positions=None), + Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, argrepr='', offset=492, starts_line=None, is_jump_target=False, positions=None), + Instruction(opname='RERAISE', opcode=119, arg=0, argval=0, argrepr='', offset=494, starts_line=None, is_jump_target=False, positions=None), + Instruction(opname='COPY', opcode=120, arg=3, argval=3, argrepr='', offset=496, starts_line=None, is_jump_target=False, positions=None), + Instruction(opname='POP_EXCEPT', opcode=89, arg=None, argval=None, argrepr='', offset=498, starts_line=None, is_jump_target=False, positions=None), + Instruction(opname='RERAISE', opcode=119, arg=1, argval=1, argrepr='', offset=500, starts_line=None, is_jump_target=False, positions=None), ] # One last piece of inspect fodder to check the default line number handling diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-04-01-11-53-59.bpo-47186.RBCPk8.rst b/Misc/NEWS.d/next/Core and Builtins/2022-04-01-11-53-59.bpo-47186.RBCPk8.rst new file mode 100644 index 0000000000000..002da2ba3791a --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2022-04-01-11-53-59.bpo-47186.RBCPk8.rst @@ -0,0 +1 @@ +Replace :opcode:`JUMP_IF_NOT_EXC_MATCH` by :opcode:`CHECK_EXC_MATCH` + jump. diff --git a/Objects/frameobject.c b/Objects/frameobject.c index 581de22587219..fe374bf0632cd 100644 --- a/Objects/frameobject.c +++ b/Objects/frameobject.c @@ -207,7 +207,6 @@ mark_stacks(PyCodeObject *code_obj, int len) case JUMP_IF_TRUE_OR_POP: case POP_JUMP_IF_FALSE: case POP_JUMP_IF_TRUE: - case JUMP_IF_NOT_EXC_MATCH: case JUMP_IF_NOT_EG_MATCH: { int64_t target_stack; @@ -216,8 +215,7 @@ mark_stacks(PyCodeObject *code_obj, int len) if (stacks[j] == UNINITIALIZED && j < i) { todo = 1; } - if (opcode == JUMP_IF_NOT_EXC_MATCH || - opcode == JUMP_IF_NOT_EG_MATCH) + if (opcode == JUMP_IF_NOT_EG_MATCH) { next_stack = pop_value(pop_value(next_stack)); target_stack = next_stack; diff --git a/Python/ceval.c b/Python/ceval.c index 8c1f21b086da9..43080f8db0422 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -3853,7 +3853,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int DISPATCH(); } - TARGET(JUMP_IF_NOT_EXC_MATCH) { + TARGET(CHECK_EXC_MATCH) { PyObject *right = POP(); PyObject *left = TOP(); assert(PyExceptionInstance_Check(left)); @@ -3864,9 +3864,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int int res = PyErr_GivenExceptionMatches(left, right); Py_DECREF(right); - if (res == 0) { - JUMPTO(oparg); - } + PUSH(Py_NewRef(res ? Py_True : Py_False)); DISPATCH(); } diff --git a/Python/compile.c b/Python/compile.c index 7a073777ee1cf..bdf886bf42352 100644 --- a/Python/compile.c +++ b/Python/compile.c @@ -977,8 +977,8 @@ stack_effect(int opcode, int oparg, int jump) case IS_OP: case CONTAINS_OP: return -1; - case JUMP_IF_NOT_EXC_MATCH: - return -1; + case CHECK_EXC_MATCH: + return 0; case JUMP_IF_NOT_EG_MATCH: return jump > 0 ? -1 : 0; case IMPORT_NAME: @@ -3352,7 +3352,8 @@ compiler_try_star_finally(struct compiler *c, stmt_ty s) [] JUMP L0 [exc] L1: ) - [exc, E1] JUMP_IF_NOT_EXC_MATCH L2 ) only if E1 + [exc, E1] CHECK_EXC_MATCH ) + [exc, bool] POP_JUMP_IF_FALSE L2 ) only if E1 [exc] (or POP if no V1) [] JUMP L0 @@ -3410,7 +3411,8 @@ compiler_try_except(struct compiler *c, stmt_ty s) return 0; if (handler->v.ExceptHandler.type) { VISIT(c, expr, handler->v.ExceptHandler.type); - ADDOP_JUMP(c, JUMP_IF_NOT_EXC_MATCH, except); + ADDOP(c, CHECK_EXC_MATCH); + ADDOP_JUMP(c, POP_JUMP_IF_FALSE, except); } if (handler->v.ExceptHandler.name) { basicblock *cleanup_end, *cleanup_body; diff --git a/Python/opcode_targets.h b/Python/opcode_targets.h index 3afaf0b029831..2eba5531723fb 100644 --- a/Python/opcode_targets.h +++ b/Python/opcode_targets.h @@ -35,6 +35,7 @@ static void *opcode_targets[256] = { &&TARGET_MATCH_KEYS, &&TARGET_JUMP_BACKWARD_QUICK, &&TARGET_PUSH_EXC_INFO, + &&TARGET_CHECK_EXC_MATCH, &&TARGET_LOAD_ATTR_ADAPTIVE, &&TARGET_LOAD_ATTR_INSTANCE_VALUE, &&TARGET_LOAD_ATTR_MODULE, @@ -47,40 +48,39 @@ static void *opcode_targets[256] = { &&TARGET_LOAD_GLOBAL_BUILTIN, &&TARGET_LOAD_GLOBAL_MODULE, &&TARGET_LOAD_METHOD_ADAPTIVE, - &&TARGET_LOAD_METHOD_CLASS, &&TARGET_WITH_EXCEPT_START, &&TARGET_GET_AITER, &&TARGET_GET_ANEXT, &&TARGET_BEFORE_ASYNC_WITH, &&TARGET_BEFORE_WITH, &&TARGET_END_ASYNC_FOR, + &&TARGET_LOAD_METHOD_CLASS, &&TARGET_LOAD_METHOD_MODULE, &&TARGET_LOAD_METHOD_NO_DICT, &&TARGET_LOAD_METHOD_WITH_DICT, &&TARGET_LOAD_METHOD_WITH_VALUES, - &&TARGET_PRECALL_ADAPTIVE, &&TARGET_STORE_SUBSCR, &&TARGET_DELETE_SUBSCR, + &&TARGET_PRECALL_ADAPTIVE, &&TARGET_PRECALL_BOUND_METHOD, &&TARGET_PRECALL_BUILTIN_CLASS, &&TARGET_PRECALL_BUILTIN_FAST_WITH_KEYWORDS, &&TARGET_PRECALL_METHOD_DESCRIPTOR_FAST_WITH_KEYWORDS, &&TARGET_PRECALL_NO_KW_BUILTIN_FAST, - &&TARGET_PRECALL_NO_KW_BUILTIN_O, &&TARGET_GET_ITER, &&TARGET_GET_YIELD_FROM_ITER, &&TARGET_PRINT_EXPR, &&TARGET_LOAD_BUILD_CLASS, + &&TARGET_PRECALL_NO_KW_BUILTIN_O, &&TARGET_PRECALL_NO_KW_ISINSTANCE, - &&TARGET_PRECALL_NO_KW_LEN, &&TARGET_LOAD_ASSERTION_ERROR, &&TARGET_RETURN_GENERATOR, + &&TARGET_PRECALL_NO_KW_LEN, &&TARGET_PRECALL_NO_KW_LIST_APPEND, &&TARGET_PRECALL_NO_KW_METHOD_DESCRIPTOR_FAST, &&TARGET_PRECALL_NO_KW_METHOD_DESCRIPTOR_NOARGS, &&TARGET_PRECALL_NO_KW_METHOD_DESCRIPTOR_O, &&TARGET_PRECALL_NO_KW_STR_1, - &&TARGET_PRECALL_NO_KW_TUPLE_1, &&TARGET_LIST_TO_TUPLE, &&TARGET_RETURN_VALUE, &&TARGET_IMPORT_STAR, @@ -112,7 +112,7 @@ static void *opcode_targets[256] = { &&TARGET_JUMP_FORWARD, &&TARGET_JUMP_IF_FALSE_OR_POP, &&TARGET_JUMP_IF_TRUE_OR_POP, - &&TARGET_PRECALL_NO_KW_TYPE_1, + &&TARGET_PRECALL_NO_KW_TUPLE_1, &&TARGET_POP_JUMP_IF_FALSE, &&TARGET_POP_JUMP_IF_TRUE, &&TARGET_LOAD_GLOBAL, @@ -120,7 +120,7 @@ static void *opcode_targets[256] = { &&TARGET_CONTAINS_OP, &&TARGET_RERAISE, &&TARGET_COPY, - &&TARGET_JUMP_IF_NOT_EXC_MATCH, + &&TARGET_PRECALL_NO_KW_TYPE_1, &&TARGET_BINARY_OP, &&TARGET_SEND, &&TARGET_LOAD_FAST, From webhook-mailer at python.org Fri Apr 1 10:50:28 2022 From: webhook-mailer at python.org (markshannon) Date: Fri, 01 Apr 2022 14:50:28 -0000 Subject: [Python-checkins] bpo-47172: Compiler enhancements (GH-32200) Message-ID: https://github.com/python/cpython/commit/997ba5d126f5040d5b7536f73bc89049e9f9421d commit: 997ba5d126f5040d5b7536f73bc89049e9f9421d branch: main author: Irit Katriel <1055913+iritkatriel at users.noreply.github.com> committer: markshannon date: 2022-04-01T15:50:15+01:00 summary: bpo-47172: Compiler enhancements (GH-32200) * Make virtual opcodes negative. * Make is_jump detect only actual jumps. * Use is_block_push for the exception block setup opcodes. files: M Python/compile.c diff --git a/Python/compile.c b/Python/compile.c index bdf886bf42352..c3773a4f48929 100644 --- a/Python/compile.c +++ b/Python/compile.c @@ -72,18 +72,26 @@ /* Pseudo-instructions used in the compiler, * but turned into NOPs by the assembler. */ -#define SETUP_FINALLY 255 -#define SETUP_CLEANUP 254 -#define SETUP_WITH 253 -#define POP_BLOCK 252 -#define JUMP 251 +#define SETUP_FINALLY -1 +#define SETUP_CLEANUP -2 +#define SETUP_WITH -3 +#define POP_BLOCK -4 +#define JUMP -5 + +#define MIN_VIRTUAL_OPCODE -5 +#define MAX_ALLOWED_OPCODE 254 + +#define IS_WITHIN_OPCODE_RANGE(opcode) \ + ((opcode) >= MIN_VIRTUAL_OPCODE && (opcode) <= MAX_ALLOWED_OPCODE) + +#define IS_VIRTUAL_OPCODE(opcode) ((opcode) < 0) #define IS_TOP_LEVEL_AWAIT(c) ( \ (c->c_flags->cf_flags & PyCF_ALLOW_TOP_LEVEL_AWAIT) \ && (c->u->u_ste->ste_type == ModuleBlock)) struct instr { - unsigned char i_opcode; + int i_opcode; int i_oparg; /* target block (if jump instruction) */ struct basicblock_ *i_target; @@ -115,8 +123,13 @@ is_bit_set_in_table(const uint32_t *table, int bitindex) { * Word is indexed by (bitindex>>ln(size of int in bits)). * Bit within word is the low bits of bitindex. */ - uint32_t word = table[bitindex >> LOG_BITS_PER_INT]; - return (word >> (bitindex & MASK_LOW_LOG_BITS)) & 1; + if (bitindex >= 0 && bitindex < 256) { + uint32_t word = table[bitindex >> LOG_BITS_PER_INT]; + return (word >> (bitindex & MASK_LOW_LOG_BITS)) & 1; + } + else { + return 0; + } } static inline int @@ -125,11 +138,17 @@ is_relative_jump(struct instr *i) return is_bit_set_in_table(_PyOpcode_RelativeJump, i->i_opcode); } +static inline int +is_block_push(struct instr *instr) +{ + int opcode = instr->i_opcode; + return opcode == SETUP_FINALLY || opcode == SETUP_WITH || opcode == SETUP_CLEANUP; +} + static inline int is_jump(struct instr *i) { - return i->i_opcode >= SETUP_WITH || - i->i_opcode == JUMP || + return i->i_opcode == JUMP || is_bit_set_in_table(_PyOpcode_Jump, i->i_opcode); } @@ -137,6 +156,7 @@ static int instr_size(struct instr *instruction) { int opcode = instruction->i_opcode; + assert(!IS_VIRTUAL_OPCODE(opcode)); int oparg = HAS_ARG(opcode) ? instruction->i_oparg : 0; int extended_args = (0xFFFFFF < oparg) + (0xFFFF < oparg) + (0xFF < oparg); int caches = _PyOpcode_Caches[opcode]; @@ -147,6 +167,7 @@ static void write_instr(_Py_CODEUNIT *codestr, struct instr *instruction, int ilen) { int opcode = instruction->i_opcode; + assert(!IS_VIRTUAL_OPCODE(opcode)); int oparg = HAS_ARG(opcode) ? instruction->i_oparg : 0; int caches = _PyOpcode_Caches[opcode]; switch (ilen - caches) { @@ -1177,6 +1198,7 @@ static int compiler_addop_line(struct compiler *c, int opcode, int line, int end_line, int col_offset, int end_col_offset) { + assert(IS_WITHIN_OPCODE_RANGE(opcode)); assert(!HAS_ARG(opcode) || IS_ARTIFICIAL(opcode)); if (compiler_use_new_implicit_block_if_needed(c) < 0) { @@ -1419,6 +1441,7 @@ compiler_addop_i_line(struct compiler *c, int opcode, Py_ssize_t oparg, The argument of a concrete bytecode instruction is limited to 8-bit. EXTENDED_ARG is used for 16, 24, and 32-bit arguments. */ + assert(IS_WITHIN_OPCODE_RANGE(opcode)); assert(HAS_ARG(opcode)); assert(0 <= oparg && oparg <= 2147483647); @@ -1462,7 +1485,8 @@ static int add_jump_to_block(struct compiler *c, int opcode, int col_offset, int end_col_offset, basicblock *target) { - assert(HAS_ARG(opcode)); + assert(IS_WITHIN_OPCODE_RANGE(opcode)); + assert(HAS_ARG(opcode) || IS_VIRTUAL_OPCODE(opcode)); assert(target != NULL); if (compiler_use_new_implicit_block_if_needed(c) < 0) { @@ -7040,7 +7064,7 @@ stackdepth(struct compiler *c) maxdepth = new_depth; } assert(depth >= 0); /* invalid code or bug in stackdepth() */ - if (is_jump(instr)) { + if (is_jump(instr) || is_block_push(instr)) { effect = stack_effect(instr->i_opcode, instr->i_oparg, 1); assert(effect != PY_INVALID_STACK_EFFECT); int target_depth = depth + effect; @@ -7159,13 +7183,6 @@ assemble_emit_table_pair(struct assembler* a, PyObject** table, int* offset, return 1; } -static int -is_block_push(struct instr *instr) -{ - int opcode = instr->i_opcode; - return opcode == SETUP_FINALLY || opcode == SETUP_WITH || opcode == SETUP_CLEANUP; -} - static basicblock * push_except_block(ExceptStack *stack, struct instr *setup) { assert(is_block_push(setup)); @@ -8600,6 +8617,7 @@ apply_static_swaps(basicblock *block, int i) static bool jump_thread(struct instr *inst, struct instr *target, int opcode) { + assert(!IS_VIRTUAL_OPCODE(opcode) || opcode == JUMP); assert(is_jump(inst)); assert(is_jump(target)); // bpo-45773: If inst->i_target == target->i_target, then nothing actually @@ -8629,7 +8647,7 @@ optimize_basic_block(struct compiler *c, basicblock *bb, PyObject *consts) struct instr *inst = &bb->b_instr[i]; int oparg = inst->i_oparg; int nextop = i+1 < bb->b_iused ? bb->b_instr[i+1].i_opcode : 0; - if (is_jump(inst)) { + if (is_jump(inst) || is_block_push(inst)) { /* Skip over empty basic blocks. */ while (inst->i_target->b_iused == 0) { inst->i_target = inst->i_target->b_next; @@ -8996,8 +9014,9 @@ mark_reachable(struct assembler *a) { } for (int i = 0; i < b->b_iused; i++) { basicblock *target; - if (is_jump(&b->b_instr[i])) { - target = b->b_instr[i].i_target; + struct instr *instr = &b->b_instr[i]; + if (is_jump(instr) || is_block_push(instr)) { + target = instr->i_target; if (target->b_predecessors == 0) { *sp++ = target; } @@ -9073,13 +9092,6 @@ propagate_line_numbers(struct assembler *a) { } } if (is_jump(&b->b_instr[b->b_iused-1])) { - switch (b->b_instr[b->b_iused-1].i_opcode) { - /* Note: Only actual jumps, not exception handlers */ - case SETUP_WITH: - case SETUP_FINALLY: - case SETUP_CLEANUP: - continue; - } basicblock *target = b->b_instr[b->b_iused-1].i_target; if (target->b_predecessors == 1) { if (target->b_instr[0].i_lineno < 0) { @@ -9205,13 +9217,6 @@ duplicate_exits_without_lineno(struct compiler *c) */ for (basicblock *b = c->u->u_blocks; b != NULL; b = b->b_list) { if (b->b_iused > 0 && is_jump(&b->b_instr[b->b_iused-1])) { - switch (b->b_instr[b->b_iused-1].i_opcode) { - /* Note: Only actual jumps, not exception handlers */ - case SETUP_WITH: - case SETUP_FINALLY: - case SETUP_CLEANUP: - continue; - } basicblock *target = b->b_instr[b->b_iused-1].i_target; if (is_exit_without_lineno(target) && target->b_predecessors > 1) { basicblock *new_target = compiler_copy_block(c, target); From webhook-mailer at python.org Fri Apr 1 11:19:21 2022 From: webhook-mailer at python.org (encukou) Date: Fri, 01 Apr 2022 15:19:21 -0000 Subject: [Python-checkins] bpo-47168: Mark files generated by `make regen-limited-abi` as generated (GH-32195) Message-ID: https://github.com/python/cpython/commit/079143df7e40c4d336cb5c385b166aa91058d050 commit: 079143df7e40c4d336cb5c385b166aa91058d050 branch: main author: Petr Viktorin committer: encukou date: 2022-04-01T17:19:05+02:00 summary: bpo-47168: Mark files generated by `make regen-limited-abi` as generated (GH-32195) files: M .gitattributes diff --git a/.gitattributes b/.gitattributes index 2718e63e5748f..bc513e437e018 100644 --- a/.gitattributes +++ b/.gitattributes @@ -60,14 +60,17 @@ PC/readme.txt text eol=crlf **/clinic/*.c.h generated *_db.h generated +Doc/data/stable_abi.dat generated Doc/library/token-list.inc generated Include/internal/pycore_ast.h generated Include/internal/pycore_ast_state.h generated Include/opcode.h generated Include/token.h generated Lib/keyword.py generated +Lib/test/test_stable_abi_ctypes.py generated Lib/token.py generated Objects/typeslots.inc generated +PC/python3dll.c generated Parser/parser.c generated Parser/token.c generated Programs/test_frozenmain.h generated From webhook-mailer at python.org Fri Apr 1 11:23:21 2022 From: webhook-mailer at python.org (tiran) Date: Fri, 01 Apr 2022 15:23:21 -0000 Subject: [Python-checkins] bpo-46023: makesetup: skip all duplicate modules (GH-32234) Message-ID: https://github.com/python/cpython/commit/abdd69c95c1711c2dc75be4e784c6d6c80a409b9 commit: abdd69c95c1711c2dc75be4e784c6d6c80a409b9 branch: main author: Christian Heimes committer: tiran date: 2022-04-01T17:23:12+02:00 summary: bpo-46023: makesetup: skip all duplicate modules (GH-32234) files: A Misc/NEWS.d/next/Build/2022-04-01-16-12-53.bpo-46023.1Z1OcC.rst M Modules/makesetup diff --git a/Misc/NEWS.d/next/Build/2022-04-01-16-12-53.bpo-46023.1Z1OcC.rst b/Misc/NEWS.d/next/Build/2022-04-01-16-12-53.bpo-46023.1Z1OcC.rst new file mode 100644 index 0000000000000..cb2f7b760e1ec --- /dev/null +++ b/Misc/NEWS.d/next/Build/2022-04-01-16-12-53.bpo-46023.1Z1OcC.rst @@ -0,0 +1,2 @@ +``makesetup`` now detects and skips all duplicated module definitions. The +first entry wins. diff --git a/Modules/makesetup b/Modules/makesetup index 3909650ed7c41..9b20e3c9f6344 100755 --- a/Modules/makesetup +++ b/Modules/makesetup @@ -117,6 +117,7 @@ sed -e 's/[ ]*#.*//' -e '/^[ ]*$/d' | BUILT= BUILT_SHARED= DISABLED= + CONFIGURED= MODS= SHAREDMODS= OBJS= @@ -206,12 +207,17 @@ sed -e 's/[ ]*#.*//' -e '/^[ ]*$/d' | cpps="\$(MODULE_${mods_upper}_CFLAGS)" libs="\$(MODULE_${mods_upper}_LDFLAGS)" fi - case $DISABLED in - *$mods*) - # disabled by previous rule / Setup file - continue - ;; - esac + for mod in $mods + do + case $CONFIGURED in + *,${mod},*) + # Detected multiple rules for a module, first rule wins. This + # allows users to disable modules in Setup.local. + echo 1>&2 "maksetup: '$mod' was handled by previous rule." + continue 2;; + esac + CONFIGURED="$CONFIGURED,${mod}," + done case $doconfig in yes) LIBS="$LIBS $libs" From webhook-mailer at python.org Fri Apr 1 11:24:05 2022 From: webhook-mailer at python.org (tiran) Date: Fri, 01 Apr 2022 15:24:05 -0000 Subject: [Python-checkins] bpo-40280: Add debug Emscripten flavors (GH-32233) Message-ID: https://github.com/python/cpython/commit/17245c815e44e79d4dad6a99b54000956a4a7229 commit: 17245c815e44e79d4dad6a99b54000956a4a7229 branch: main author: Christian Heimes committer: tiran date: 2022-04-01T17:24:00+02:00 summary: bpo-40280: Add debug Emscripten flavors (GH-32233) files: M configure M configure.ac diff --git a/configure b/configure index f261a86a28d1d..f08a01caafdd5 100755 --- a/configure +++ b/configure @@ -6262,6 +6262,10 @@ if test "${with_emscripten_target+set}" = set; then : ac_sys_emscripten_target=browser ;; #( node) : ac_sys_emscripten_target=node ;; #( + browser-debug) : + ac_sys_emscripten_target=browser-debug ;; #( + node-debug) : + ac_sys_emscripten_target=node-debug ;; #( *) : as_fn_error $? "Invalid argument: --with-emscripten-target=browser|node" "$LINENO" 5 ;; @@ -6303,9 +6307,9 @@ esac else case $ac_sys_system/$ac_sys_emscripten_target in #( - Emscripten/browser) : + Emscripten/browser*) : EXEEXT=.html ;; #( - Emscripten/node) : + Emscripten/node*) : EXEEXT=.js ;; #( WASI/*) : EXEEXT=.wasm ;; #( @@ -6615,7 +6619,7 @@ $as_echo "$LDLIBRARY" >&6; } # LIBRARY_DEPS, LINK_PYTHON_OBJS and LINK_PYTHON_DEPS variable case $ac_sys_system/$ac_sys_emscripten_target in #( - Emscripten/browser) : + Emscripten/browser*) : LIBRARY_DEPS='$(PY3LIBRARY) $(WASM_STDLIB)' ;; #( *) : LIBRARY_DEPS='$(PY3LIBRARY) $(EXPORTSYMS)' @@ -7758,29 +7762,29 @@ fi # The option disables code elimination, which increases code size of main # binary. All objects must be built with -fPIC. case $ac_sys_system/$ac_sys_emscripten_target in #( - Emscripten/browser) : + Emscripten/browser*) : LDFLAGS_NODIST="$LDFLAGS_NODIST -s ALLOW_MEMORY_GROWTH=1" LINKFORSHARED="--preload-file \$(WASM_ASSETS_DIR)" WASM_ASSETS_DIR=".\$(prefix)" WASM_STDLIB="\$(WASM_ASSETS_DIR)/local/lib/python\$(VERSION)/os.py" - if test "$Py_DEBUG" = 'true'; then + if test "$Py_DEBUG" = 'true' -o "$ac_sys_emscripten_target" = "browser-debug"; then LDFLAGS_NODIST="$LDFLAGS_NODIST -s ASSERTIONS=1" LINKFORSHARED="$LINKFORSHARED -gsource-map --emit-symbol-map" else LINKFORSHARED="$LINKFORSHARED -O2 -g0" fi ;; #( - Emscripten/node) : + Emscripten/node*) : LDFLAGS_NODIST="$LDFLAGS_NODIST -s ALLOW_MEMORY_GROWTH=1 -s NODERAWFS=1 -s USE_PTHREADS=1" LINKFORSHARED="-s PROXY_TO_PTHREAD=1 -s EXIT_RUNTIME=1" CFLAGS_NODIST="$CFLAGS_NODIST -pthread" - if test "$Py_DEBUG" = 'true'; then + if test "$Py_DEBUG" = 'true' -o "$ac_sys_emscripten_target" = "node-debug"; then LDFLAGS_NODIST="$LDFLAGS_NODIST -s ASSERTIONS=1" LINKFORSHARED="$LINKFORSHARED -gseparate-dwarf --emit-symbol-map" else - LINKFORSHARED="$LINKFORSHARED -O2 -gseparate-dwarf" + LINKFORSHARED="$LINKFORSHARED -O2 -g0" fi ;; #( WASI/*) : @@ -21672,7 +21676,7 @@ if test "$enable_test_modules" = no; then TEST_MODULES=no else case $ac_sys_system/$ac_sys_emscripten_target in #( - Emscripten/browser) : + Emscripten/browser*) : TEST_MODULES=no ;; #( *) : TEST_MODULES=yes @@ -21731,7 +21735,7 @@ case $ac_sys_system/$ac_sys_emscripten_target in #( py_cv_module__scproxy=n/a py_cv_module_spwd=n/a ;; #( - Emscripten/browser) : + Emscripten/browser*) : @@ -21759,7 +21763,7 @@ case $ac_sys_system/$ac_sys_emscripten_target in #( py_cv_module_=n/a ;; #( - Emscripten/node) : + Emscripten/node*) : diff --git a/configure.ac b/configure.ac index 7e8203ba85d1b..bc3d8b87def9c 100644 --- a/configure.ac +++ b/configure.ac @@ -1091,6 +1091,12 @@ AC_ARG_WITH([emscripten-target], AS_CASE([$with_emscripten_target], [browser], [ac_sys_emscripten_target=browser], [node], [ac_sys_emscripten_target=node], +dnl Debug builds with source map / dwarf symbols. Py_DEBUG builds easily +dnl run out of stack space. Detached sybmols and map prohibit some +dnl optimizations and increase file size. Options are undocumented so we +dnl are free to remove them in the future. + [browser-debug], [ac_sys_emscripten_target=browser-debug], + [node-debug], [ac_sys_emscripten_target=node-debug], [AC_MSG_ERROR([Invalid argument: --with-emscripten-target=browser|node])] ) ], [ @@ -1112,8 +1118,8 @@ AC_ARG_WITH([suffix], ) ], [ AS_CASE([$ac_sys_system/$ac_sys_emscripten_target], - [Emscripten/browser], [EXEEXT=.html], - [Emscripten/node], [EXEEXT=.js], + [Emscripten/browser*], [EXEEXT=.html], + [Emscripten/node*], [EXEEXT=.js], [WASI/*], [EXEEXT=.wasm], [EXEEXT=] ) @@ -1376,7 +1382,7 @@ AC_MSG_RESULT($LDLIBRARY) # LIBRARY_DEPS, LINK_PYTHON_OBJS and LINK_PYTHON_DEPS variable AS_CASE([$ac_sys_system/$ac_sys_emscripten_target], - [Emscripten/browser], [LIBRARY_DEPS='$(PY3LIBRARY) $(WASM_STDLIB)'], + [Emscripten/browser*], [LIBRARY_DEPS='$(PY3LIBRARY) $(WASM_STDLIB)'], [LIBRARY_DEPS='$(PY3LIBRARY) $(EXPORTSYMS)'] ) LINK_PYTHON_DEPS='$(LIBRARY_DEPS)' @@ -1888,28 +1894,28 @@ fi # The option disables code elimination, which increases code size of main # binary. All objects must be built with -fPIC. AS_CASE([$ac_sys_system/$ac_sys_emscripten_target], - [Emscripten/browser], [ + [Emscripten/browser*], [ LDFLAGS_NODIST="$LDFLAGS_NODIST -s ALLOW_MEMORY_GROWTH=1" LINKFORSHARED="--preload-file \$(WASM_ASSETS_DIR)" WASM_ASSETS_DIR=".\$(prefix)" WASM_STDLIB="\$(WASM_ASSETS_DIR)/local/lib/python\$(VERSION)/os.py" dnl separate-dwarf does not seem to work in Chrome DevTools Support. - if test "$Py_DEBUG" = 'true'; then + if test "$Py_DEBUG" = 'true' -o "$ac_sys_emscripten_target" = "browser-debug"; then LDFLAGS_NODIST="$LDFLAGS_NODIST -s ASSERTIONS=1" LINKFORSHARED="$LINKFORSHARED -gsource-map --emit-symbol-map" else LINKFORSHARED="$LINKFORSHARED -O2 -g0" fi ], - [Emscripten/node], [ + [Emscripten/node*], [ LDFLAGS_NODIST="$LDFLAGS_NODIST -s ALLOW_MEMORY_GROWTH=1 -s NODERAWFS=1 -s USE_PTHREADS=1" LINKFORSHARED="-s PROXY_TO_PTHREAD=1 -s EXIT_RUNTIME=1" CFLAGS_NODIST="$CFLAGS_NODIST -pthread" - if test "$Py_DEBUG" = 'true'; then + if test "$Py_DEBUG" = 'true' -o "$ac_sys_emscripten_target" = "node-debug"; then LDFLAGS_NODIST="$LDFLAGS_NODIST -s ASSERTIONS=1" LINKFORSHARED="$LINKFORSHARED -gseparate-dwarf --emit-symbol-map" else - LINKFORSHARED="$LINKFORSHARED -O2 -gseparate-dwarf" + LINKFORSHARED="$LINKFORSHARED -O2 -g0" fi ], [WASI/*], [ @@ -6453,7 +6459,7 @@ if test "$enable_test_modules" = no; then TEST_MODULES=no else AS_CASE([$ac_sys_system/$ac_sys_emscripten_target], - [Emscripten/browser], [TEST_MODULES=no], + [Emscripten/browser*], [TEST_MODULES=no], [TEST_MODULES=yes] ) fi @@ -6478,7 +6484,7 @@ AS_CASE([$ac_sys_system/$ac_sys_emscripten_target], [CYGWIN*/*], [PY_STDLIB_MOD_SET_NA([_scproxy], [nis])], [QNX*/*], [PY_STDLIB_MOD_SET_NA([_scproxy], [nis])], [FreeBSD*/*], [PY_STDLIB_MOD_SET_NA([_scproxy], [spwd])], - [Emscripten/browser], [ + [Emscripten/browser*], [ PY_STDLIB_MOD_SET_NA( [_ctypes], [_curses], @@ -6505,7 +6511,7 @@ AS_CASE([$ac_sys_system/$ac_sys_emscripten_target], ], dnl Some modules like _posixsubprocess do not work. We build them anyway dnl so imports in tests do not fail. - [Emscripten/node], [ + [Emscripten/node*], [ PY_STDLIB_MOD_SET_NA( [_ctypes], [_curses], From webhook-mailer at python.org Fri Apr 1 14:13:11 2022 From: webhook-mailer at python.org (vstinner) Date: Fri, 01 Apr 2022 18:13:11 -0000 Subject: [Python-checkins] bpo-47089: Avoid test_compileall failures on Windows (GH-32037) Message-ID: https://github.com/python/cpython/commit/76b8a075b8a79b08468fd0ed06a489a5c815bc11 commit: 76b8a075b8a79b08468fd0ed06a489a5c815bc11 branch: main author: Jeremy Kloth committer: vstinner date: 2022-04-01T20:13:01+02:00 summary: bpo-47089: Avoid test_compileall failures on Windows (GH-32037) files: M Lib/test/test_compileall.py diff --git a/Lib/test/test_compileall.py b/Lib/test/test_compileall.py index 98dab339caa1c..1a9fc973c3aa1 100644 --- a/Lib/test/test_compileall.py +++ b/Lib/test/test_compileall.py @@ -460,31 +460,29 @@ def test_error(self): class CommandLineTestsBase: """Test compileall's CLI.""" - @classmethod - def setUpClass(cls): - for path in filter(os.path.isdir, sys.path): - directory_created = False - directory = pathlib.Path(path) / '__pycache__' - path = directory / 'test.try' - try: - if not directory.is_dir(): - directory.mkdir() - directory_created = True - path.write_text('# for test_compileall', encoding="utf-8") - except OSError: - sys_path_writable = False - break - finally: - os_helper.unlink(str(path)) - if directory_created: - directory.rmdir() - else: - sys_path_writable = True - cls._sys_path_writable = sys_path_writable - - def _skip_if_sys_path_not_writable(self): - if not self._sys_path_writable: - raise unittest.SkipTest('not all entries on sys.path are writable') + def setUp(self): + self.directory = tempfile.mkdtemp() + self.addCleanup(os_helper.rmtree, self.directory) + self.pkgdir = os.path.join(self.directory, 'foo') + os.mkdir(self.pkgdir) + self.pkgdir_cachedir = os.path.join(self.pkgdir, '__pycache__') + # Create the __init__.py and a package module. + self.initfn = script_helper.make_script(self.pkgdir, '__init__', '') + self.barfn = script_helper.make_script(self.pkgdir, 'bar', '') + + @contextlib.contextmanager + def temporary_pycache_prefix(self): + """Adjust and restore sys.pycache_prefix.""" + old_prefix = sys.pycache_prefix + new_prefix = os.path.join(self.directory, '__testcache__') + try: + sys.pycache_prefix = new_prefix + yield { + 'PYTHONPATH': self.directory, + 'PYTHONPYCACHEPREFIX': new_prefix, + } + finally: + sys.pycache_prefix = old_prefix def _get_run_args(self, args): return [*support.optim_args_from_interpreter_flags(), @@ -512,49 +510,39 @@ def assertNotCompiled(self, fn): path = importlib.util.cache_from_source(fn) self.assertFalse(os.path.exists(path)) - def setUp(self): - self.directory = tempfile.mkdtemp() - self.addCleanup(os_helper.rmtree, self.directory) - self.pkgdir = os.path.join(self.directory, 'foo') - os.mkdir(self.pkgdir) - self.pkgdir_cachedir = os.path.join(self.pkgdir, '__pycache__') - # Create the __init__.py and a package module. - self.initfn = script_helper.make_script(self.pkgdir, '__init__', '') - self.barfn = script_helper.make_script(self.pkgdir, 'bar', '') - def test_no_args_compiles_path(self): # Note that -l is implied for the no args case. - self._skip_if_sys_path_not_writable() bazfn = script_helper.make_script(self.directory, 'baz', '') - self.assertRunOK(PYTHONPATH=self.directory) - self.assertCompiled(bazfn) - self.assertNotCompiled(self.initfn) - self.assertNotCompiled(self.barfn) + with self.temporary_pycache_prefix() as env: + self.assertRunOK(**env) + self.assertCompiled(bazfn) + self.assertNotCompiled(self.initfn) + self.assertNotCompiled(self.barfn) @without_source_date_epoch # timestamp invalidation test def test_no_args_respects_force_flag(self): - self._skip_if_sys_path_not_writable() bazfn = script_helper.make_script(self.directory, 'baz', '') - self.assertRunOK(PYTHONPATH=self.directory) - pycpath = importlib.util.cache_from_source(bazfn) + with self.temporary_pycache_prefix() as env: + self.assertRunOK(**env) + pycpath = importlib.util.cache_from_source(bazfn) # Set atime/mtime backward to avoid file timestamp resolution issues os.utime(pycpath, (time.time()-60,)*2) mtime = os.stat(pycpath).st_mtime # Without force, no recompilation - self.assertRunOK(PYTHONPATH=self.directory) + self.assertRunOK(**env) mtime2 = os.stat(pycpath).st_mtime self.assertEqual(mtime, mtime2) # Now force it. - self.assertRunOK('-f', PYTHONPATH=self.directory) + self.assertRunOK('-f', **env) mtime2 = os.stat(pycpath).st_mtime self.assertNotEqual(mtime, mtime2) def test_no_args_respects_quiet_flag(self): - self._skip_if_sys_path_not_writable() script_helper.make_script(self.directory, 'baz', '') - noisy = self.assertRunOK(PYTHONPATH=self.directory) + with self.temporary_pycache_prefix() as env: + noisy = self.assertRunOK(**env) self.assertIn(b'Listing ', noisy) - quiet = self.assertRunOK('-q', PYTHONPATH=self.directory) + quiet = self.assertRunOK('-q', **env) self.assertNotIn(b'Listing ', quiet) # Ensure that the default behavior of compileall's CLI is to create From webhook-mailer at python.org Fri Apr 1 15:21:12 2022 From: webhook-mailer at python.org (tiran) Date: Fri, 01 Apr 2022 19:21:12 -0000 Subject: [Python-checkins] bpo-40280: Emscripten fork_exec now fails early (GH-32224) Message-ID: https://github.com/python/cpython/commit/082d3495d0c820972f09f6109a98ed7eb5a7b79f commit: 082d3495d0c820972f09f6109a98ed7eb5a7b79f branch: main author: Christian Heimes committer: tiran date: 2022-04-01T21:20:56+02:00 summary: bpo-40280: Emscripten fork_exec now fails early (GH-32224) files: M Lib/subprocess.py M Lib/test/test_subprocess.py M configure M configure.ac diff --git a/Lib/subprocess.py b/Lib/subprocess.py index ad08339b25ddc..b58c578428390 100644 --- a/Lib/subprocess.py +++ b/Lib/subprocess.py @@ -96,7 +96,13 @@ "CREATE_NO_WINDOW", "DETACHED_PROCESS", "CREATE_DEFAULT_ERROR_MODE", "CREATE_BREAKAWAY_FROM_JOB"]) else: - import _posixsubprocess + if sys.platform in {"emscripten", "wasi"}: + def _fork_exec(*args, **kwargs): + raise OSError( + errno.ENOTSUP, f"{sys.platform} does not support processes." + ) + else: + from _posixsubprocess import fork_exec as _fork_exec import select import selectors @@ -1777,7 +1783,7 @@ def _execute_child(self, args, executable, preexec_fn, close_fds, for dir in os.get_exec_path(env)) fds_to_keep = set(pass_fds) fds_to_keep.add(errpipe_write) - self.pid = _posixsubprocess.fork_exec( + self.pid = _fork_exec( args, executable_list, close_fds, tuple(sorted(map(int, fds_to_keep))), cwd, env_list, diff --git a/Lib/test/test_subprocess.py b/Lib/test/test_subprocess.py index 99a25e279df92..8603b9888bb98 100644 --- a/Lib/test/test_subprocess.py +++ b/Lib/test/test_subprocess.py @@ -1805,7 +1805,7 @@ class PopenNoDestructor(subprocess.Popen): def __del__(self): pass - @mock.patch("subprocess._posixsubprocess.fork_exec") + @mock.patch("subprocess._fork_exec") def test_exception_errpipe_normal(self, fork_exec): """Test error passing done through errpipe_write in the good case""" def proper_error(*args): @@ -1822,7 +1822,7 @@ def proper_error(*args): with self.assertRaises(IsADirectoryError): self.PopenNoDestructor(["non_existent_command"]) - @mock.patch("subprocess._posixsubprocess.fork_exec") + @mock.patch("subprocess._fork_exec") def test_exception_errpipe_bad_data(self, fork_exec): """Test error passing done through errpipe_write where its not in the expected format""" @@ -2112,7 +2112,7 @@ def raise_it(): preexec_fn=raise_it) except subprocess.SubprocessError as e: self.assertTrue( - subprocess._posixsubprocess, + subprocess._fork_exec, "Expected a ValueError from the preexec_fn") except ValueError as e: self.assertIn("coconut", e.args[0]) @@ -2600,11 +2600,11 @@ def prepare(): preexec_fn=prepare) except ValueError as err: # Pure Python implementations keeps the message - self.assertIsNone(subprocess._posixsubprocess) + self.assertIsNone(subprocess._fork_exec) self.assertEqual(str(err), "surrogate:\uDCff") except subprocess.SubprocessError as err: # _posixsubprocess uses a default message - self.assertIsNotNone(subprocess._posixsubprocess) + self.assertIsNotNone(subprocess._fork_exec) self.assertEqual(str(err), "Exception occurred in preexec_fn.") else: self.fail("Expected ValueError or subprocess.SubprocessError") diff --git a/configure b/configure index f08a01caafdd5..a5062d7b81573 100755 --- a/configure +++ b/configure @@ -21772,6 +21772,9 @@ case $ac_sys_system/$ac_sys_emscripten_target in #( py_cv_module__curses_panel=n/a py_cv_module__dbm=n/a py_cv_module__gdbm=n/a + py_cv_module__multiprocessing=n/a + py_cv_module__posixshmem=n/a + py_cv_module__posixsubprocess=n/a py_cv_module__scproxy=n/a py_cv_module__tkinter=n/a py_cv_module__xxsubinterpreters=n/a diff --git a/configure.ac b/configure.ac index bc3d8b87def9c..84bc9b3ca5cf0 100644 --- a/configure.ac +++ b/configure.ac @@ -6518,6 +6518,9 @@ AS_CASE([$ac_sys_system/$ac_sys_emscripten_target], [_curses_panel], [_dbm], [_gdbm], + [_multiprocessing], + [_posixshmem], + [_posixsubprocess], [_scproxy], [_tkinter], [_xxsubinterpreters], From webhook-mailer at python.org Fri Apr 1 16:38:53 2022 From: webhook-mailer at python.org (miss-islington) Date: Fri, 01 Apr 2022 20:38:53 -0000 Subject: [Python-checkins] bpo-47089: Avoid test_compileall failures on Windows (GH-32037) Message-ID: https://github.com/python/cpython/commit/1069d529590850e87a11b8c559a7fb296e9c626a commit: 1069d529590850e87a11b8c559a7fb296e9c626a branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-01T13:38:35-07:00 summary: bpo-47089: Avoid test_compileall failures on Windows (GH-32037) (cherry picked from commit 76b8a075b8a79b08468fd0ed06a489a5c815bc11) Co-authored-by: Jeremy Kloth files: M Lib/test/test_compileall.py diff --git a/Lib/test/test_compileall.py b/Lib/test/test_compileall.py index 33f0c939325f5..a7318bf404c3b 100644 --- a/Lib/test/test_compileall.py +++ b/Lib/test/test_compileall.py @@ -457,31 +457,29 @@ def test_error(self): class CommandLineTestsBase: """Test compileall's CLI.""" - @classmethod - def setUpClass(cls): - for path in filter(os.path.isdir, sys.path): - directory_created = False - directory = pathlib.Path(path) / '__pycache__' - path = directory / 'test.try' - try: - if not directory.is_dir(): - directory.mkdir() - directory_created = True - path.write_text('# for test_compileall', encoding="utf-8") - except OSError: - sys_path_writable = False - break - finally: - os_helper.unlink(str(path)) - if directory_created: - directory.rmdir() - else: - sys_path_writable = True - cls._sys_path_writable = sys_path_writable - - def _skip_if_sys_path_not_writable(self): - if not self._sys_path_writable: - raise unittest.SkipTest('not all entries on sys.path are writable') + def setUp(self): + self.directory = tempfile.mkdtemp() + self.addCleanup(os_helper.rmtree, self.directory) + self.pkgdir = os.path.join(self.directory, 'foo') + os.mkdir(self.pkgdir) + self.pkgdir_cachedir = os.path.join(self.pkgdir, '__pycache__') + # Create the __init__.py and a package module. + self.initfn = script_helper.make_script(self.pkgdir, '__init__', '') + self.barfn = script_helper.make_script(self.pkgdir, 'bar', '') + + @contextlib.contextmanager + def temporary_pycache_prefix(self): + """Adjust and restore sys.pycache_prefix.""" + old_prefix = sys.pycache_prefix + new_prefix = os.path.join(self.directory, '__testcache__') + try: + sys.pycache_prefix = new_prefix + yield { + 'PYTHONPATH': self.directory, + 'PYTHONPYCACHEPREFIX': new_prefix, + } + finally: + sys.pycache_prefix = old_prefix def _get_run_args(self, args): return [*support.optim_args_from_interpreter_flags(), @@ -509,49 +507,39 @@ def assertNotCompiled(self, fn): path = importlib.util.cache_from_source(fn) self.assertFalse(os.path.exists(path)) - def setUp(self): - self.directory = tempfile.mkdtemp() - self.addCleanup(os_helper.rmtree, self.directory) - self.pkgdir = os.path.join(self.directory, 'foo') - os.mkdir(self.pkgdir) - self.pkgdir_cachedir = os.path.join(self.pkgdir, '__pycache__') - # Create the __init__.py and a package module. - self.initfn = script_helper.make_script(self.pkgdir, '__init__', '') - self.barfn = script_helper.make_script(self.pkgdir, 'bar', '') - def test_no_args_compiles_path(self): # Note that -l is implied for the no args case. - self._skip_if_sys_path_not_writable() bazfn = script_helper.make_script(self.directory, 'baz', '') - self.assertRunOK(PYTHONPATH=self.directory) - self.assertCompiled(bazfn) - self.assertNotCompiled(self.initfn) - self.assertNotCompiled(self.barfn) + with self.temporary_pycache_prefix() as env: + self.assertRunOK(**env) + self.assertCompiled(bazfn) + self.assertNotCompiled(self.initfn) + self.assertNotCompiled(self.barfn) @without_source_date_epoch # timestamp invalidation test def test_no_args_respects_force_flag(self): - self._skip_if_sys_path_not_writable() bazfn = script_helper.make_script(self.directory, 'baz', '') - self.assertRunOK(PYTHONPATH=self.directory) - pycpath = importlib.util.cache_from_source(bazfn) + with self.temporary_pycache_prefix() as env: + self.assertRunOK(**env) + pycpath = importlib.util.cache_from_source(bazfn) # Set atime/mtime backward to avoid file timestamp resolution issues os.utime(pycpath, (time.time()-60,)*2) mtime = os.stat(pycpath).st_mtime # Without force, no recompilation - self.assertRunOK(PYTHONPATH=self.directory) + self.assertRunOK(**env) mtime2 = os.stat(pycpath).st_mtime self.assertEqual(mtime, mtime2) # Now force it. - self.assertRunOK('-f', PYTHONPATH=self.directory) + self.assertRunOK('-f', **env) mtime2 = os.stat(pycpath).st_mtime self.assertNotEqual(mtime, mtime2) def test_no_args_respects_quiet_flag(self): - self._skip_if_sys_path_not_writable() script_helper.make_script(self.directory, 'baz', '') - noisy = self.assertRunOK(PYTHONPATH=self.directory) + with self.temporary_pycache_prefix() as env: + noisy = self.assertRunOK(**env) self.assertIn(b'Listing ', noisy) - quiet = self.assertRunOK('-q', PYTHONPATH=self.directory) + quiet = self.assertRunOK('-q', **env) self.assertNotIn(b'Listing ', quiet) # Ensure that the default behavior of compileall's CLI is to create From webhook-mailer at python.org Fri Apr 1 20:56:35 2022 From: webhook-mailer at python.org (corona10) Date: Sat, 02 Apr 2022 00:56:35 -0000 Subject: [Python-checkins] no-issue: Add assertion to PyModule_GetName for understanding (GH-32236) Message-ID: https://github.com/python/cpython/commit/b183f486493e7e4c332566392ef18c6b346a6561 commit: b183f486493e7e4c332566392ef18c6b346a6561 branch: main author: Dong-hee Na committer: corona10 date: 2022-04-02T09:56:30+09:00 summary: no-issue: Add assertion to PyModule_GetName for understanding (GH-32236) files: M Objects/moduleobject.c diff --git a/Objects/moduleobject.c b/Objects/moduleobject.c index 738b262288bcd..dba20a0b36463 100644 --- a/Objects/moduleobject.c +++ b/Objects/moduleobject.c @@ -510,8 +510,10 @@ const char * PyModule_GetName(PyObject *m) { PyObject *name = PyModule_GetNameObject(m); - if (name == NULL) + if (name == NULL) { return NULL; + } + assert(Py_REFCNT(name) >= 2); Py_DECREF(name); /* module dict has still a reference */ return PyUnicode_AsUTF8(name); } From webhook-mailer at python.org Fri Apr 1 21:40:32 2022 From: webhook-mailer at python.org (vstinner) Date: Sat, 02 Apr 2022 01:40:32 -0000 Subject: [Python-checkins] [3.9] bpo-47089: Avoid test_compileall failures on Windows (GH-32037). (GH-32240) Message-ID: https://github.com/python/cpython/commit/306a93b4819db611041da735ca3b34117a3bc961 commit: 306a93b4819db611041da735ca3b34117a3bc961 branch: 3.9 author: Jeremy Kloth committer: vstinner date: 2022-04-02T03:40:27+02:00 summary: [3.9] bpo-47089: Avoid test_compileall failures on Windows (GH-32037). (GH-32240) * [3.9] bpo-47089: Avoid test_compileall failures on Windows (GH-32037). (cherry picked from commit 76b8a075b8a79b08468fd0ed06a489a5c815bc11) Co-authored-by: Jeremy Kloth files: M Lib/test/test_compileall.py diff --git a/Lib/test/test_compileall.py b/Lib/test/test_compileall.py index a904f426b172a..ab647d66f0d12 100644 --- a/Lib/test/test_compileall.py +++ b/Lib/test/test_compileall.py @@ -449,32 +449,29 @@ def test_error(self): class CommandLineTestsBase: """Test compileall's CLI.""" - @classmethod - def setUpClass(cls): - for path in filter(os.path.isdir, sys.path): - directory_created = False - directory = pathlib.Path(path) / '__pycache__' - path = directory / 'test.try' - try: - if not directory.is_dir(): - directory.mkdir() - directory_created = True - with path.open('w') as file: - file.write('# for test_compileall') - except OSError: - sys_path_writable = False - break - finally: - support.unlink(str(path)) - if directory_created: - directory.rmdir() - else: - sys_path_writable = True - cls._sys_path_writable = sys_path_writable - - def _skip_if_sys_path_not_writable(self): - if not self._sys_path_writable: - raise unittest.SkipTest('not all entries on sys.path are writable') + def setUp(self): + self.directory = tempfile.mkdtemp() + self.addCleanup(shutil.rmtree, self.directory) + self.pkgdir = os.path.join(self.directory, 'foo') + os.mkdir(self.pkgdir) + self.pkgdir_cachedir = os.path.join(self.pkgdir, '__pycache__') + # Create the __init__.py and a package module. + self.initfn = script_helper.make_script(self.pkgdir, '__init__', '') + self.barfn = script_helper.make_script(self.pkgdir, 'bar', '') + + @contextlib.contextmanager + def temporary_pycache_prefix(self): + """Adjust and restore sys.pycache_prefix.""" + old_prefix = sys.pycache_prefix + new_prefix = os.path.join(self.directory, '__testcache__') + try: + sys.pycache_prefix = new_prefix + yield { + 'PYTHONPATH': self.directory, + 'PYTHONPYCACHEPREFIX': new_prefix, + } + finally: + sys.pycache_prefix = old_prefix def _get_run_args(self, args): return [*support.optim_args_from_interpreter_flags(), @@ -502,49 +499,39 @@ def assertNotCompiled(self, fn): path = importlib.util.cache_from_source(fn) self.assertFalse(os.path.exists(path)) - def setUp(self): - self.directory = tempfile.mkdtemp() - self.addCleanup(support.rmtree, self.directory) - self.pkgdir = os.path.join(self.directory, 'foo') - os.mkdir(self.pkgdir) - self.pkgdir_cachedir = os.path.join(self.pkgdir, '__pycache__') - # Create the __init__.py and a package module. - self.initfn = script_helper.make_script(self.pkgdir, '__init__', '') - self.barfn = script_helper.make_script(self.pkgdir, 'bar', '') - def test_no_args_compiles_path(self): # Note that -l is implied for the no args case. - self._skip_if_sys_path_not_writable() bazfn = script_helper.make_script(self.directory, 'baz', '') - self.assertRunOK(PYTHONPATH=self.directory) - self.assertCompiled(bazfn) - self.assertNotCompiled(self.initfn) - self.assertNotCompiled(self.barfn) + with self.temporary_pycache_prefix() as env: + self.assertRunOK(**env) + self.assertCompiled(bazfn) + self.assertNotCompiled(self.initfn) + self.assertNotCompiled(self.barfn) @without_source_date_epoch # timestamp invalidation test def test_no_args_respects_force_flag(self): - self._skip_if_sys_path_not_writable() bazfn = script_helper.make_script(self.directory, 'baz', '') - self.assertRunOK(PYTHONPATH=self.directory) - pycpath = importlib.util.cache_from_source(bazfn) + with self.temporary_pycache_prefix() as env: + self.assertRunOK(**env) + pycpath = importlib.util.cache_from_source(bazfn) # Set atime/mtime backward to avoid file timestamp resolution issues os.utime(pycpath, (time.time()-60,)*2) mtime = os.stat(pycpath).st_mtime # Without force, no recompilation - self.assertRunOK(PYTHONPATH=self.directory) + self.assertRunOK(**env) mtime2 = os.stat(pycpath).st_mtime self.assertEqual(mtime, mtime2) # Now force it. - self.assertRunOK('-f', PYTHONPATH=self.directory) + self.assertRunOK('-f', **env) mtime2 = os.stat(pycpath).st_mtime self.assertNotEqual(mtime, mtime2) def test_no_args_respects_quiet_flag(self): - self._skip_if_sys_path_not_writable() script_helper.make_script(self.directory, 'baz', '') - noisy = self.assertRunOK(PYTHONPATH=self.directory) + with self.temporary_pycache_prefix() as env: + noisy = self.assertRunOK(**env) self.assertIn(b'Listing ', noisy) - quiet = self.assertRunOK('-q', PYTHONPATH=self.directory) + quiet = self.assertRunOK('-q', **env) self.assertNotIn(b'Listing ', quiet) # Ensure that the default behavior of compileall's CLI is to create From webhook-mailer at python.org Fri Apr 1 21:54:09 2022 From: webhook-mailer at python.org (pablogsal) Date: Sat, 02 Apr 2022 01:54:09 -0000 Subject: [Python-checkins] bpo-47131: Speedup AST comparisons in test_unparse by using node traversal (GH-32132) Message-ID: https://github.com/python/cpython/commit/0f68c208fa6a36b6c8ad3d775e64292a665ba108 commit: 0f68c208fa6a36b6c8ad3d775e64292a665ba108 branch: main author: Jeremy Kloth committer: pablogsal date: 2022-04-02T02:54:04+01:00 summary: bpo-47131: Speedup AST comparisons in test_unparse by using node traversal (GH-32132) files: M Lib/test/test_unparse.py diff --git a/Lib/test/test_unparse.py b/Lib/test/test_unparse.py index e38b33574cccc..f999ae8c16cea 100644 --- a/Lib/test/test_unparse.py +++ b/Lib/test/test_unparse.py @@ -130,7 +130,43 @@ class Foo: pass class ASTTestCase(unittest.TestCase): def assertASTEqual(self, ast1, ast2): - self.assertEqual(ast.dump(ast1), ast.dump(ast2)) + # Ensure the comparisons start at an AST node + self.assertIsInstance(ast1, ast.AST) + self.assertIsInstance(ast2, ast.AST) + + # An AST comparison routine modeled after ast.dump(), but + # instead of string building, it traverses the two trees + # in lock-step. + def traverse_compare(a, b, missing=object()): + if type(a) is not type(b): + self.fail(f"{type(a)!r} is not {type(b)!r}") + if isinstance(a, ast.AST): + for field in a._fields: + value1 = getattr(a, field, missing) + value2 = getattr(b, field, missing) + # Singletons are equal by definition, so further + # testing can be skipped. + if value1 is not value2: + traverse_compare(value1, value2) + elif isinstance(a, list): + try: + for node1, node2 in zip(a, b, strict=True): + traverse_compare(node1, node2) + except ValueError: + # Attempt a "pretty" error ala assertSequenceEqual() + len1 = len(a) + len2 = len(b) + if len1 > len2: + what = "First" + diff = len1 - len2 + else: + what = "Second" + diff = len2 - len1 + msg = f"{what} list contains {diff} additional elements." + raise self.failureException(msg) from None + elif a != b: + self.fail(f"{a!r} != {b!r}") + traverse_compare(ast1, ast2) def check_ast_roundtrip(self, code1, **kwargs): with self.subTest(code1=code1, ast_parse_kwargs=kwargs): From webhook-mailer at python.org Sat Apr 2 04:00:57 2022 From: webhook-mailer at python.org (miss-islington) Date: Sat, 02 Apr 2022 08:00:57 -0000 Subject: [Python-checkins] bpo-47196: Fix function pointer cast in test_imp (GH-32244) Message-ID: https://github.com/python/cpython/commit/7000cd70164707e28ba045b5c036ca7ed73f5dc4 commit: 7000cd70164707e28ba045b5c036ca7ed73f5dc4 branch: main author: Hood Chatham committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-02T01:00:49-07:00 summary: bpo-47196: Fix function pointer cast in test_imp (GH-32244) The function PyInit_imp_dummy is declared as void f(PyObject* spec) but called as void f(void). On wasm targets without the call trampolines this causes a fatal error. Automerge-Triggered-By: GH:tiran files: M Modules/_testmultiphase.c diff --git a/Modules/_testmultiphase.c b/Modules/_testmultiphase.c index 4905269177bf3..e9a37a019134f 100644 --- a/Modules/_testmultiphase.c +++ b/Modules/_testmultiphase.c @@ -450,7 +450,7 @@ static PyModuleDef_Slot main_slots[] = { static PyModuleDef main_def = TEST_MODULE_DEF("main", main_slots, testexport_methods); PyMODINIT_FUNC -PyInit__testmultiphase(PyObject *spec) +PyInit__testmultiphase(void) { return PyModuleDef_Init(&main_def); } @@ -495,7 +495,7 @@ static PyModuleDef def_nonmodule = TEST_MODULE_DEF( "_testmultiphase_nonmodule", slots_create_nonmodule, NULL); PyMODINIT_FUNC -PyInit__testmultiphase_nonmodule(PyObject *spec) +PyInit__testmultiphase_nonmodule(void) { return PyModuleDef_Init(&def_nonmodule); } @@ -525,7 +525,7 @@ static PyModuleDef def_nonmodule_with_methods = TEST_MODULE_DEF( "_testmultiphase_nonmodule_with_methods", slots_create_nonmodule, nonmodule_methods); PyMODINIT_FUNC -PyInit__testmultiphase_nonmodule_with_methods(PyObject *spec) +PyInit__testmultiphase_nonmodule_with_methods(void) { return PyModuleDef_Init(&def_nonmodule_with_methods); } @@ -545,7 +545,7 @@ static PyModuleDef def_nonascii_latin = { \ }; PyMODINIT_FUNC -PyInitU__testmultiphase_zkouka_naten_evc07gi8e(PyObject *spec) +PyInitU__testmultiphase_zkouka_naten_evc07gi8e(void) { return PyModuleDef_Init(&def_nonascii_latin); } @@ -563,7 +563,7 @@ static PyModuleDef def_nonascii_kana = { \ }; PyMODINIT_FUNC -PyInitU_eckzbwbhc6jpgzcx415x(PyObject *spec) +PyInitU_eckzbwbhc6jpgzcx415x(void) { return PyModuleDef_Init(&def_nonascii_kana); } @@ -571,7 +571,7 @@ PyInitU_eckzbwbhc6jpgzcx415x(PyObject *spec) /*** Module with a single-character name ***/ PyMODINIT_FUNC -PyInit_x(PyObject *spec) +PyInit_x(void) { return PyModuleDef_Init(&main_def); } @@ -582,7 +582,7 @@ static PyModuleDef null_slots_def = TEST_MODULE_DEF( "_testmultiphase_null_slots", NULL, NULL); PyMODINIT_FUNC -PyInit__testmultiphase_null_slots(PyObject *spec) +PyInit__testmultiphase_null_slots(void) { return PyModuleDef_Init(&null_slots_def); } @@ -598,7 +598,7 @@ static PyModuleDef def_bad_large = TEST_MODULE_DEF( "_testmultiphase_bad_slot_large", slots_bad_large, NULL); PyMODINIT_FUNC -PyInit__testmultiphase_bad_slot_large(PyObject *spec) +PyInit__testmultiphase_bad_slot_large(void) { return PyModuleDef_Init(&def_bad_large); } @@ -612,7 +612,7 @@ static PyModuleDef def_bad_negative = TEST_MODULE_DEF( "_testmultiphase_bad_slot_negative", slots_bad_negative, NULL); PyMODINIT_FUNC -PyInit__testmultiphase_bad_slot_negative(PyObject *spec) +PyInit__testmultiphase_bad_slot_negative(void) { return PyModuleDef_Init(&def_bad_negative); } @@ -630,7 +630,7 @@ static PyModuleDef def_create_int_with_state = { \ }; PyMODINIT_FUNC -PyInit__testmultiphase_create_int_with_state(PyObject *spec) +PyInit__testmultiphase_create_int_with_state(void) { return PyModuleDef_Init(&def_create_int_with_state); } @@ -649,7 +649,7 @@ static PyModuleDef def_negative_size = { \ }; PyMODINIT_FUNC -PyInit__testmultiphase_negative_size(PyObject *spec) +PyInit__testmultiphase_negative_size(void) { return PyModuleDef_Init(&def_negative_size); } @@ -658,26 +658,26 @@ PyInit__testmultiphase_negative_size(PyObject *spec) static PyModuleDef uninitialized_def = TEST_MODULE_DEF("main", main_slots, testexport_methods); PyMODINIT_FUNC -PyInit__testmultiphase_export_uninitialized(PyObject *spec) +PyInit__testmultiphase_export_uninitialized(void) { return (PyObject*) &uninitialized_def; } PyMODINIT_FUNC -PyInit__testmultiphase_export_null(PyObject *spec) +PyInit__testmultiphase_export_null(void) { return NULL; } PyMODINIT_FUNC -PyInit__testmultiphase_export_raise(PyObject *spec) +PyInit__testmultiphase_export_raise(void) { PyErr_SetString(PyExc_SystemError, "bad export function"); return NULL; } PyMODINIT_FUNC -PyInit__testmultiphase_export_unreported_exception(PyObject *spec) +PyInit__testmultiphase_export_unreported_exception(void) { PyErr_SetString(PyExc_SystemError, "bad export function"); return PyModuleDef_Init(&main_def); @@ -698,7 +698,7 @@ static PyModuleDef def_create_null = TEST_MODULE_DEF( "_testmultiphase_create_null", slots_create_null, NULL); PyMODINIT_FUNC -PyInit__testmultiphase_create_null(PyObject *spec) +PyInit__testmultiphase_create_null(void) { return PyModuleDef_Init(&def_create_null); } @@ -719,7 +719,7 @@ static PyModuleDef def_create_raise = TEST_MODULE_DEF( "_testmultiphase_create_null", slots_create_raise, NULL); PyMODINIT_FUNC -PyInit__testmultiphase_create_raise(PyObject *spec) +PyInit__testmultiphase_create_raise(void) { return PyModuleDef_Init(&def_create_raise); } @@ -740,7 +740,7 @@ static PyModuleDef def_create_unreported_exception = TEST_MODULE_DEF( "_testmultiphase_create_unreported_exception", slots_create_unreported_exception, NULL); PyMODINIT_FUNC -PyInit__testmultiphase_create_unreported_exception(PyObject *spec) +PyInit__testmultiphase_create_unreported_exception(void) { return PyModuleDef_Init(&def_create_unreported_exception); } @@ -755,7 +755,7 @@ static PyModuleDef def_nonmodule_with_exec_slots = TEST_MODULE_DEF( "_testmultiphase_nonmodule_with_exec_slots", slots_nonmodule_with_exec_slots, NULL); PyMODINIT_FUNC -PyInit__testmultiphase_nonmodule_with_exec_slots(PyObject *spec) +PyInit__testmultiphase_nonmodule_with_exec_slots(void) { return PyModuleDef_Init(&def_nonmodule_with_exec_slots); } @@ -775,7 +775,7 @@ static PyModuleDef def_exec_err = TEST_MODULE_DEF( "_testmultiphase_exec_err", slots_exec_err, NULL); PyMODINIT_FUNC -PyInit__testmultiphase_exec_err(PyObject *spec) +PyInit__testmultiphase_exec_err(void) { return PyModuleDef_Init(&def_exec_err); } @@ -817,7 +817,7 @@ static PyModuleDef def_exec_unreported_exception = TEST_MODULE_DEF( "_testmultiphase_exec_unreported_exception", slots_exec_unreported_exception, NULL); PyMODINIT_FUNC -PyInit__testmultiphase_exec_unreported_exception(PyObject *spec) +PyInit__testmultiphase_exec_unreported_exception(void) { return PyModuleDef_Init(&def_exec_unreported_exception); } @@ -861,7 +861,7 @@ static PyModuleDef def_meth_state_access = { }; PyMODINIT_FUNC -PyInit__testmultiphase_meth_state_access(PyObject *spec) +PyInit__testmultiphase_meth_state_access(void) { return PyModuleDef_Init(&def_meth_state_access); } @@ -874,7 +874,7 @@ static PyModuleDef def_module_state_shared = { }; PyMODINIT_FUNC -PyInit__test_module_state_shared(PyObject *spec) +PyInit__test_module_state_shared(void) { PyObject *module = PyModule_Create(&def_module_state_shared); if (module == NULL) { @@ -894,7 +894,7 @@ PyInit__test_module_state_shared(PyObject *spec) static PyModuleDef imp_dummy_def = TEST_MODULE_DEF("imp_dummy", main_slots, testexport_methods); PyMODINIT_FUNC -PyInit_imp_dummy(PyObject *spec) +PyInit_imp_dummy(void) { return PyModuleDef_Init(&imp_dummy_def); } From webhook-mailer at python.org Sat Apr 2 04:12:52 2022 From: webhook-mailer at python.org (miss-islington) Date: Sat, 02 Apr 2022 08:12:52 -0000 Subject: [Python-checkins] bpo-40280: Detect if WASM platform supports threading (GH-32243) Message-ID: https://github.com/python/cpython/commit/59be9cd748728b03ac61287681c3010bcec5e558 commit: 59be9cd748728b03ac61287681c3010bcec5e558 branch: main author: Christian Heimes committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-02T01:12:44-07:00 summary: bpo-40280: Detect if WASM platform supports threading (GH-32243) Automerge-Triggered-By: GH:tiran files: M Lib/test/libregrtest/runtest.py M Lib/test/support/threading_helper.py diff --git a/Lib/test/libregrtest/runtest.py b/Lib/test/libregrtest/runtest.py index 83c5f249841cc..62cf1a3f1175e 100644 --- a/Lib/test/libregrtest/runtest.py +++ b/Lib/test/libregrtest/runtest.py @@ -11,6 +11,7 @@ from test import support from test.support import os_helper +from test.support import threading_helper from test.libregrtest.cmdline import Namespace from test.libregrtest.save_env import saved_test_environment from test.libregrtest.utils import clear_caches, format_duration, print_warning @@ -179,7 +180,9 @@ def _runtest(ns: Namespace, test_name: str) -> TestResult: output_on_failure = ns.verbose3 - use_timeout = (ns.timeout is not None) + use_timeout = ( + ns.timeout is not None and threading_helper.can_start_thread + ) if use_timeout: faulthandler.dump_traceback_later(ns.timeout, exit=True) diff --git a/Lib/test/support/threading_helper.py b/Lib/test/support/threading_helper.py index 92a64e8354acb..2ae4577f41f7b 100644 --- a/Lib/test/support/threading_helper.py +++ b/Lib/test/support/threading_helper.py @@ -207,3 +207,30 @@ def __exit__(self, *exc_info): del self.exc_value del self.exc_traceback del self.thread + + +def _can_start_thread() -> bool: + """Detect if Python can start new threads. + + Some WebAssembly platforms do not provide a working pthread + implementation. Thread support is stubbed and any attempt + to create a new thread fails. + + - wasm32-wasi does not have threading. + - wasm32-emscripten can be compiled with or without pthread + support (-s USE_PTHREADS / __EMSCRIPTEN_PTHREADS__). + """ + if sys.platform == "emscripten": + try: + _thread.start_new_thread(lambda: None, ()) + except RuntimeError: + return False + else: + return True + elif sys.platform == "wasi": + return False + else: + # assume all other platforms have working thread support. + return True + +can_start_thread = _can_start_thread() From webhook-mailer at python.org Sat Apr 2 04:13:55 2022 From: webhook-mailer at python.org (tiran) Date: Sat, 02 Apr 2022 08:13:55 -0000 Subject: [Python-checkins] bpo-40280: Enable most file-at() and nanosleep APIs again (GH-32238) Message-ID: https://github.com/python/cpython/commit/4ed8a9a589d2eee7442e0c9417515a707e504faa commit: 4ed8a9a589d2eee7442e0c9417515a707e504faa branch: main author: Christian Heimes committer: tiran date: 2022-04-02T10:13:44+02:00 summary: bpo-40280: Enable most file-at() and nanosleep APIs again (GH-32238) files: M Lib/test/test_os.py M Tools/wasm/config.site-wasm32-emscripten diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py index 9a60bf5f7fdd5..5f73af81d4560 100644 --- a/Lib/test/test_os.py +++ b/Lib/test/test_os.py @@ -1334,7 +1334,9 @@ def setUp(self): else: self.sub2_tree = (sub2_path, ["SUB21"], ["tmp3"]) - os.chmod(sub21_path, 0) + if not support.is_emscripten: + # Emscripten fails with inaccessible directory + os.chmod(sub21_path, 0) try: os.listdir(sub21_path) except PermissionError: @@ -1517,6 +1519,9 @@ def test_yields_correct_dir_fd(self): # check that listdir() returns consistent information self.assertEqual(set(os.listdir(rootfd)), set(dirs) | set(files)) + @unittest.skipIf( + support.is_emscripten, "Cannot dup stdout on Emscripten" + ) def test_fd_leak(self): # Since we're opening a lot of FDs, we must be careful to avoid leaks: # we both check that calling fwalk() a large number of times doesn't @@ -2935,6 +2940,9 @@ def test_bad_fd(self): @unittest.skipUnless(os.isatty(0) and not win32_is_iot() and (sys.platform.startswith('win') or (hasattr(locale, 'nl_langinfo') and hasattr(locale, 'CODESET'))), 'test requires a tty and either Windows or nl_langinfo(CODESET)') + @unittest.skipIf( + support.is_emscripten, "Cannot get encoding of stdin on Emscripten" + ) def test_device_encoding(self): encoding = os.device_encoding(0) self.assertIsNotNone(encoding) diff --git a/Tools/wasm/config.site-wasm32-emscripten b/Tools/wasm/config.site-wasm32-emscripten index 7f2df3a47723f..60ede49eb3745 100644 --- a/Tools/wasm/config.site-wasm32-emscripten +++ b/Tools/wasm/config.site-wasm32-emscripten @@ -27,11 +27,6 @@ ac_cv_func_prlimit=no # unsupported syscall, https://github.com/emscripten-core/emscripten/issues/13393 ac_cv_func_shutdown=no -# clock_nanosleep() causes time.sleep() to sleep forever. -# nanosleep() works correctly -ac_cv_func_clock_nanosleep=no -ac_cv_lib_rt_clock_nanosleep=no - # The rest is based on pyodide # https://github.com/pyodide/pyodide/blob/main/cpython/pyconfig.undefs.h @@ -39,20 +34,11 @@ ac_cv_func_epoll_create=no ac_cv_func_epoll_create1=no ac_cv_header_linux_vm_sockets_h=no ac_cv_func_socketpair=no -ac_cv_func_utimensat=no ac_cv_func_sigaction=no -# Untested or failing syscalls in emscripten -ac_cv_func_openat=no -ac_cv_func_mkdirat=no -ac_cv_func_faccessat=no -ac_cv_func_fchownat=no -ac_cv_func_renameat=no -ac_cv_func_linkat=no +# symlinkat is implemented, but fails ac_cv_func_symlinkat=no -ac_cv_func_readlinkat=no -ac_cv_func_fchmodat=no -ac_cv_func_dup3=no + # lchmod/lchown are implemented, but fail with ENOTSUP. ac_cv_func_lchmod=no ac_cv_func_lchown=no From webhook-mailer at python.org Sat Apr 2 04:35:28 2022 From: webhook-mailer at python.org (serhiy-storchaka) Date: Sat, 02 Apr 2022 08:35:28 -0000 Subject: [Python-checkins] bpo-47152: Convert the re module into a package (GH-32177) Message-ID: https://github.com/python/cpython/commit/1be3260a90f16aae334d993aecf7b70426f98013 commit: 1be3260a90f16aae334d993aecf7b70426f98013 branch: main author: Serhiy Storchaka committer: serhiy-storchaka date: 2022-04-02T11:35:13+03:00 summary: bpo-47152: Convert the re module into a package (GH-32177) The sre_* modules are now deprecated. files: A Lib/re/__init__.py A Lib/re/_compiler.py A Lib/re/_constants.py A Lib/re/_parser.py A Misc/NEWS.d/next/Library/2022-03-29-19-14-53.bpo-47152.5rl5ZK.rst D Lib/re.py M Doc/library/modulefinder.rst M Doc/library/profile.rst M Doc/whatsnew/3.11.rst M Lib/sre_compile.py M Lib/sre_constants.py M Lib/sre_parse.py M Lib/test/test_pyclbr.py M Lib/test/test_re.py M Lib/test/test_site.py M Makefile.pre.in M Modules/sre_constants.h diff --git a/Doc/library/modulefinder.rst b/Doc/library/modulefinder.rst index 7b39ce7d1aae5..526f0ff868c2b 100644 --- a/Doc/library/modulefinder.rst +++ b/Doc/library/modulefinder.rst @@ -96,14 +96,14 @@ Sample output (may vary depending on the architecture):: Loaded modules: _types: copyreg: _inverted_registry,_slotnames,__all__ - sre_compile: isstring,_sre,_optimize_unicode + re._compiler: isstring,_sre,_optimize_unicode _sre: - sre_constants: REPEAT_ONE,makedict,AT_END_LINE + re._constants: REPEAT_ONE,makedict,AT_END_LINE sys: re: __module__,finditer,_expand itertools: __main__: re,itertools,baconhameggs - sre_parse: _PATTERNENDERS,SRE_FLAG_UNICODE + re._parser: _PATTERNENDERS,SRE_FLAG_UNICODE array: types: __module__,IntType,TypeType --------------------------------------------------- diff --git a/Doc/library/profile.rst b/Doc/library/profile.rst index 5278d1a58802e..2d95096f4cb83 100644 --- a/Doc/library/profile.rst +++ b/Doc/library/profile.rst @@ -73,12 +73,12 @@ the following:: ncalls tottime percall cumtime percall filename:lineno(function) 1 0.000 0.000 0.002 0.002 {built-in method builtins.exec} 1 0.000 0.000 0.001 0.001 :1() - 1 0.000 0.000 0.001 0.001 re.py:250(compile) - 1 0.000 0.000 0.001 0.001 re.py:289(_compile) - 1 0.000 0.000 0.000 0.000 sre_compile.py:759(compile) - 1 0.000 0.000 0.000 0.000 sre_parse.py:937(parse) - 1 0.000 0.000 0.000 0.000 sre_compile.py:598(_code) - 1 0.000 0.000 0.000 0.000 sre_parse.py:435(_parse_sub) + 1 0.000 0.000 0.001 0.001 __init__.py:250(compile) + 1 0.000 0.000 0.001 0.001 __init__.py:289(_compile) + 1 0.000 0.000 0.000 0.000 _compiler.py:759(compile) + 1 0.000 0.000 0.000 0.000 _parser.py:937(parse) + 1 0.000 0.000 0.000 0.000 _compiler.py:598(_code) + 1 0.000 0.000 0.000 0.000 _parser.py:435(_parse_sub) The first line indicates that 214 calls were monitored. Of those calls, 207 were :dfn:`primitive`, meaning that the call was not induced via recursion. The diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index dc2d4b0937c77..0c7f4afa694ad 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -532,6 +532,10 @@ Deprecated be able to parse Python 3.10 or newer. See the :pep:`617` (New PEG parser for CPython). (Contributed by Victor Stinner in :issue:`40360`.) +* Undocumented modules ``sre_compile``, ``sre_constants`` and ``sre_parse`` + are now deprecated. + (Contributed by Serhiy Storchaka in :issue:`47152`.) + * :class:`webbrowser.MacOSX` is deprecated and will be removed in Python 3.13. It is untested and undocumented and also not used by webbrowser itself. (Contributed by Dong-hee Na in :issue:`42255`.) diff --git a/Lib/re.py b/Lib/re/__init__.py similarity index 90% rename from Lib/re.py rename to Lib/re/__init__.py index e9a745dc581a6..c47a2650e32f5 100644 --- a/Lib/re.py +++ b/Lib/re/__init__.py @@ -122,8 +122,7 @@ """ import enum -import sre_compile -import sre_parse +from . import _compiler, _parser import functools try: import _locale @@ -146,21 +145,21 @@ @enum._simple_enum(enum.IntFlag, boundary=enum.KEEP) class RegexFlag: NOFLAG = 0 - ASCII = A = sre_compile.SRE_FLAG_ASCII # assume ascii "locale" - IGNORECASE = I = sre_compile.SRE_FLAG_IGNORECASE # ignore case - LOCALE = L = sre_compile.SRE_FLAG_LOCALE # assume current 8-bit locale - UNICODE = U = sre_compile.SRE_FLAG_UNICODE # assume unicode "locale" - MULTILINE = M = sre_compile.SRE_FLAG_MULTILINE # make anchors look for newline - DOTALL = S = sre_compile.SRE_FLAG_DOTALL # make dot match newline - VERBOSE = X = sre_compile.SRE_FLAG_VERBOSE # ignore whitespace and comments + ASCII = A = _compiler.SRE_FLAG_ASCII # assume ascii "locale" + IGNORECASE = I = _compiler.SRE_FLAG_IGNORECASE # ignore case + LOCALE = L = _compiler.SRE_FLAG_LOCALE # assume current 8-bit locale + UNICODE = U = _compiler.SRE_FLAG_UNICODE # assume unicode "locale" + MULTILINE = M = _compiler.SRE_FLAG_MULTILINE # make anchors look for newline + DOTALL = S = _compiler.SRE_FLAG_DOTALL # make dot match newline + VERBOSE = X = _compiler.SRE_FLAG_VERBOSE # ignore whitespace and comments # sre extensions (experimental, don't rely on these) - TEMPLATE = T = sre_compile.SRE_FLAG_TEMPLATE # disable backtracking - DEBUG = sre_compile.SRE_FLAG_DEBUG # dump pattern after compilation + TEMPLATE = T = _compiler.SRE_FLAG_TEMPLATE # disable backtracking + DEBUG = _compiler.SRE_FLAG_DEBUG # dump pattern after compilation __str__ = object.__str__ _numeric_repr_ = hex # sre exception -error = sre_compile.error +error = _compiler.error # -------------------------------------------------------------------- # public interface @@ -257,8 +256,8 @@ def escape(pattern): pattern = str(pattern, 'latin1') return pattern.translate(_special_chars_map).encode('latin1') -Pattern = type(sre_compile.compile('', 0)) -Match = type(sre_compile.compile('', 0).match('')) +Pattern = type(_compiler.compile('', 0)) +Match = type(_compiler.compile('', 0).match('')) # -------------------------------------------------------------------- # internals @@ -279,9 +278,9 @@ def _compile(pattern, flags): raise ValueError( "cannot process flags argument with a compiled pattern") return pattern - if not sre_compile.isstring(pattern): + if not _compiler.isstring(pattern): raise TypeError("first argument must be string or compiled pattern") - p = sre_compile.compile(pattern, flags) + p = _compiler.compile(pattern, flags) if not (flags & DEBUG): if len(_cache) >= _MAXCACHE: # Drop the oldest item @@ -295,12 +294,12 @@ def _compile(pattern, flags): @functools.lru_cache(_MAXCACHE) def _compile_repl(repl, pattern): # internal: compile replacement pattern - return sre_parse.parse_template(repl, pattern) + return _parser.parse_template(repl, pattern) def _expand(pattern, match, template): # internal: Match.expand implementation hook - template = sre_parse.parse_template(template, pattern) - return sre_parse.expand_template(template, match) + template = _parser.parse_template(template, pattern) + return _parser.expand_template(template, match) def _subx(pattern, template): # internal: Pattern.sub/subn implementation helper @@ -309,7 +308,7 @@ def _subx(pattern, template): # literal replacement return template[1][0] def filter(match, template=template): - return sre_parse.expand_template(template, match) + return _parser.expand_template(template, match) return filter # register myself for pickling @@ -326,22 +325,22 @@ def _pickle(p): class Scanner: def __init__(self, lexicon, flags=0): - from sre_constants import BRANCH, SUBPATTERN + from ._constants import BRANCH, SUBPATTERN if isinstance(flags, RegexFlag): flags = flags.value self.lexicon = lexicon # combine phrases into a compound pattern p = [] - s = sre_parse.State() + s = _parser.State() s.flags = flags for phrase, action in lexicon: gid = s.opengroup() - p.append(sre_parse.SubPattern(s, [ - (SUBPATTERN, (gid, 0, 0, sre_parse.parse(phrase, flags))), + p.append(_parser.SubPattern(s, [ + (SUBPATTERN, (gid, 0, 0, _parser.parse(phrase, flags))), ])) s.closegroup(gid, p[-1]) - p = sre_parse.SubPattern(s, [(BRANCH, (None, p))]) - self.scanner = sre_compile.compile(p) + p = _parser.SubPattern(s, [(BRANCH, (None, p))]) + self.scanner = _compiler.compile(p) def scan(self, string): result = [] append = result.append diff --git a/Lib/re/_compiler.py b/Lib/re/_compiler.py new file mode 100644 index 0000000000000..62da8e55d72ab --- /dev/null +++ b/Lib/re/_compiler.py @@ -0,0 +1,800 @@ +# +# Secret Labs' Regular Expression Engine +# +# convert template to internal format +# +# Copyright (c) 1997-2001 by Secret Labs AB. All rights reserved. +# +# See the __init__.py file for information on usage and redistribution. +# + +"""Internal support module for sre""" + +import _sre +from . import _parser +from ._constants import * + +assert _sre.MAGIC == MAGIC, "SRE module mismatch" + +_LITERAL_CODES = {LITERAL, NOT_LITERAL} +_SUCCESS_CODES = {SUCCESS, FAILURE} +_ASSERT_CODES = {ASSERT, ASSERT_NOT} +_UNIT_CODES = _LITERAL_CODES | {ANY, IN} + +_REPEATING_CODES = { + MIN_REPEAT: (REPEAT, MIN_UNTIL, MIN_REPEAT_ONE), + MAX_REPEAT: (REPEAT, MAX_UNTIL, REPEAT_ONE), + POSSESSIVE_REPEAT: (POSSESSIVE_REPEAT, SUCCESS, POSSESSIVE_REPEAT_ONE), +} + +# Sets of lowercase characters which have the same uppercase. +_equivalences = ( + # LATIN SMALL LETTER I, LATIN SMALL LETTER DOTLESS I + (0x69, 0x131), # i? + # LATIN SMALL LETTER S, LATIN SMALL LETTER LONG S + (0x73, 0x17f), # s? + # MICRO SIGN, GREEK SMALL LETTER MU + (0xb5, 0x3bc), # ?? + # COMBINING GREEK YPOGEGRAMMENI, GREEK SMALL LETTER IOTA, GREEK PROSGEGRAMMENI + (0x345, 0x3b9, 0x1fbe), # \u0345?? + # GREEK SMALL LETTER IOTA WITH DIALYTIKA AND TONOS, GREEK SMALL LETTER IOTA WITH DIALYTIKA AND OXIA + (0x390, 0x1fd3), # ?? + # GREEK SMALL LETTER UPSILON WITH DIALYTIKA AND TONOS, GREEK SMALL LETTER UPSILON WITH DIALYTIKA AND OXIA + (0x3b0, 0x1fe3), # ?? + # GREEK SMALL LETTER BETA, GREEK BETA SYMBOL + (0x3b2, 0x3d0), # ?? + # GREEK SMALL LETTER EPSILON, GREEK LUNATE EPSILON SYMBOL + (0x3b5, 0x3f5), # ?? + # GREEK SMALL LETTER THETA, GREEK THETA SYMBOL + (0x3b8, 0x3d1), # ?? + # GREEK SMALL LETTER KAPPA, GREEK KAPPA SYMBOL + (0x3ba, 0x3f0), # ?? + # GREEK SMALL LETTER PI, GREEK PI SYMBOL + (0x3c0, 0x3d6), # ?? + # GREEK SMALL LETTER RHO, GREEK RHO SYMBOL + (0x3c1, 0x3f1), # ?? + # GREEK SMALL LETTER FINAL SIGMA, GREEK SMALL LETTER SIGMA + (0x3c2, 0x3c3), # ?? + # GREEK SMALL LETTER PHI, GREEK PHI SYMBOL + (0x3c6, 0x3d5), # ?? + # LATIN SMALL LETTER S WITH DOT ABOVE, LATIN SMALL LETTER LONG S WITH DOT ABOVE + (0x1e61, 0x1e9b), # ?? + # LATIN SMALL LIGATURE LONG S T, LATIN SMALL LIGATURE ST + (0xfb05, 0xfb06), # ?? +) + +# Maps the lowercase code to lowercase codes which have the same uppercase. +_ignorecase_fixes = {i: tuple(j for j in t if i != j) + for t in _equivalences for i in t} + +def _combine_flags(flags, add_flags, del_flags, + TYPE_FLAGS=_parser.TYPE_FLAGS): + if add_flags & TYPE_FLAGS: + flags &= ~TYPE_FLAGS + return (flags | add_flags) & ~del_flags + +def _compile(code, pattern, flags): + # internal: compile a (sub)pattern + emit = code.append + _len = len + LITERAL_CODES = _LITERAL_CODES + REPEATING_CODES = _REPEATING_CODES + SUCCESS_CODES = _SUCCESS_CODES + ASSERT_CODES = _ASSERT_CODES + iscased = None + tolower = None + fixes = None + if flags & SRE_FLAG_IGNORECASE and not flags & SRE_FLAG_LOCALE: + if flags & SRE_FLAG_UNICODE: + iscased = _sre.unicode_iscased + tolower = _sre.unicode_tolower + fixes = _ignorecase_fixes + else: + iscased = _sre.ascii_iscased + tolower = _sre.ascii_tolower + for op, av in pattern: + if op in LITERAL_CODES: + if not flags & SRE_FLAG_IGNORECASE: + emit(op) + emit(av) + elif flags & SRE_FLAG_LOCALE: + emit(OP_LOCALE_IGNORE[op]) + emit(av) + elif not iscased(av): + emit(op) + emit(av) + else: + lo = tolower(av) + if not fixes: # ascii + emit(OP_IGNORE[op]) + emit(lo) + elif lo not in fixes: + emit(OP_UNICODE_IGNORE[op]) + emit(lo) + else: + emit(IN_UNI_IGNORE) + skip = _len(code); emit(0) + if op is NOT_LITERAL: + emit(NEGATE) + for k in (lo,) + fixes[lo]: + emit(LITERAL) + emit(k) + emit(FAILURE) + code[skip] = _len(code) - skip + elif op is IN: + charset, hascased = _optimize_charset(av, iscased, tolower, fixes) + if flags & SRE_FLAG_IGNORECASE and flags & SRE_FLAG_LOCALE: + emit(IN_LOC_IGNORE) + elif not hascased: + emit(IN) + elif not fixes: # ascii + emit(IN_IGNORE) + else: + emit(IN_UNI_IGNORE) + skip = _len(code); emit(0) + _compile_charset(charset, flags, code) + code[skip] = _len(code) - skip + elif op is ANY: + if flags & SRE_FLAG_DOTALL: + emit(ANY_ALL) + else: + emit(ANY) + elif op in REPEATING_CODES: + if flags & SRE_FLAG_TEMPLATE: + raise error("internal: unsupported template operator %r" % (op,)) + if _simple(av[2]): + emit(REPEATING_CODES[op][2]) + skip = _len(code); emit(0) + emit(av[0]) + emit(av[1]) + _compile(code, av[2], flags) + emit(SUCCESS) + code[skip] = _len(code) - skip + else: + emit(REPEATING_CODES[op][0]) + skip = _len(code); emit(0) + emit(av[0]) + emit(av[1]) + _compile(code, av[2], flags) + code[skip] = _len(code) - skip + emit(REPEATING_CODES[op][1]) + elif op is SUBPATTERN: + group, add_flags, del_flags, p = av + if group: + emit(MARK) + emit((group-1)*2) + # _compile_info(code, p, _combine_flags(flags, add_flags, del_flags)) + _compile(code, p, _combine_flags(flags, add_flags, del_flags)) + if group: + emit(MARK) + emit((group-1)*2+1) + elif op is ATOMIC_GROUP: + # Atomic Groups are handled by starting with an Atomic + # Group op code, then putting in the atomic group pattern + # and finally a success op code to tell any repeat + # operations within the Atomic Group to stop eating and + # pop their stack if they reach it + emit(ATOMIC_GROUP) + skip = _len(code); emit(0) + _compile(code, av, flags) + emit(SUCCESS) + code[skip] = _len(code) - skip + elif op in SUCCESS_CODES: + emit(op) + elif op in ASSERT_CODES: + emit(op) + skip = _len(code); emit(0) + if av[0] >= 0: + emit(0) # look ahead + else: + lo, hi = av[1].getwidth() + if lo != hi: + raise error("look-behind requires fixed-width pattern") + emit(lo) # look behind + _compile(code, av[1], flags) + emit(SUCCESS) + code[skip] = _len(code) - skip + elif op is CALL: + emit(op) + skip = _len(code); emit(0) + _compile(code, av, flags) + emit(SUCCESS) + code[skip] = _len(code) - skip + elif op is AT: + emit(op) + if flags & SRE_FLAG_MULTILINE: + av = AT_MULTILINE.get(av, av) + if flags & SRE_FLAG_LOCALE: + av = AT_LOCALE.get(av, av) + elif flags & SRE_FLAG_UNICODE: + av = AT_UNICODE.get(av, av) + emit(av) + elif op is BRANCH: + emit(op) + tail = [] + tailappend = tail.append + for av in av[1]: + skip = _len(code); emit(0) + # _compile_info(code, av, flags) + _compile(code, av, flags) + emit(JUMP) + tailappend(_len(code)); emit(0) + code[skip] = _len(code) - skip + emit(FAILURE) # end of branch + for tail in tail: + code[tail] = _len(code) - tail + elif op is CATEGORY: + emit(op) + if flags & SRE_FLAG_LOCALE: + av = CH_LOCALE[av] + elif flags & SRE_FLAG_UNICODE: + av = CH_UNICODE[av] + emit(av) + elif op is GROUPREF: + if not flags & SRE_FLAG_IGNORECASE: + emit(op) + elif flags & SRE_FLAG_LOCALE: + emit(GROUPREF_LOC_IGNORE) + elif not fixes: # ascii + emit(GROUPREF_IGNORE) + else: + emit(GROUPREF_UNI_IGNORE) + emit(av-1) + elif op is GROUPREF_EXISTS: + emit(op) + emit(av[0]-1) + skipyes = _len(code); emit(0) + _compile(code, av[1], flags) + if av[2]: + emit(JUMP) + skipno = _len(code); emit(0) + code[skipyes] = _len(code) - skipyes + 1 + _compile(code, av[2], flags) + code[skipno] = _len(code) - skipno + else: + code[skipyes] = _len(code) - skipyes + 1 + else: + raise error("internal: unsupported operand type %r" % (op,)) + +def _compile_charset(charset, flags, code): + # compile charset subprogram + emit = code.append + for op, av in charset: + emit(op) + if op is NEGATE: + pass + elif op is LITERAL: + emit(av) + elif op is RANGE or op is RANGE_UNI_IGNORE: + emit(av[0]) + emit(av[1]) + elif op is CHARSET: + code.extend(av) + elif op is BIGCHARSET: + code.extend(av) + elif op is CATEGORY: + if flags & SRE_FLAG_LOCALE: + emit(CH_LOCALE[av]) + elif flags & SRE_FLAG_UNICODE: + emit(CH_UNICODE[av]) + else: + emit(av) + else: + raise error("internal: unsupported set operator %r" % (op,)) + emit(FAILURE) + +def _optimize_charset(charset, iscased=None, fixup=None, fixes=None): + # internal: optimize character set + out = [] + tail = [] + charmap = bytearray(256) + hascased = False + for op, av in charset: + while True: + try: + if op is LITERAL: + if fixup: + lo = fixup(av) + charmap[lo] = 1 + if fixes and lo in fixes: + for k in fixes[lo]: + charmap[k] = 1 + if not hascased and iscased(av): + hascased = True + else: + charmap[av] = 1 + elif op is RANGE: + r = range(av[0], av[1]+1) + if fixup: + if fixes: + for i in map(fixup, r): + charmap[i] = 1 + if i in fixes: + for k in fixes[i]: + charmap[k] = 1 + else: + for i in map(fixup, r): + charmap[i] = 1 + if not hascased: + hascased = any(map(iscased, r)) + else: + for i in r: + charmap[i] = 1 + elif op is NEGATE: + out.append((op, av)) + else: + tail.append((op, av)) + except IndexError: + if len(charmap) == 256: + # character set contains non-UCS1 character codes + charmap += b'\0' * 0xff00 + continue + # Character set contains non-BMP character codes. + if fixup: + hascased = True + # There are only two ranges of cased non-BMP characters: + # 10400-1044F (Deseret) and 118A0-118DF (Warang Citi), + # and for both ranges RANGE_UNI_IGNORE works. + if op is RANGE: + op = RANGE_UNI_IGNORE + tail.append((op, av)) + break + + # compress character map + runs = [] + q = 0 + while True: + p = charmap.find(1, q) + if p < 0: + break + if len(runs) >= 2: + runs = None + break + q = charmap.find(0, p) + if q < 0: + runs.append((p, len(charmap))) + break + runs.append((p, q)) + if runs is not None: + # use literal/range + for p, q in runs: + if q - p == 1: + out.append((LITERAL, p)) + else: + out.append((RANGE, (p, q - 1))) + out += tail + # if the case was changed or new representation is more compact + if hascased or len(out) < len(charset): + return out, hascased + # else original character set is good enough + return charset, hascased + + # use bitmap + if len(charmap) == 256: + data = _mk_bitmap(charmap) + out.append((CHARSET, data)) + out += tail + return out, hascased + + # To represent a big charset, first a bitmap of all characters in the + # set is constructed. Then, this bitmap is sliced into chunks of 256 + # characters, duplicate chunks are eliminated, and each chunk is + # given a number. In the compiled expression, the charset is + # represented by a 32-bit word sequence, consisting of one word for + # the number of different chunks, a sequence of 256 bytes (64 words) + # of chunk numbers indexed by their original chunk position, and a + # sequence of 256-bit chunks (8 words each). + + # Compression is normally good: in a typical charset, large ranges of + # Unicode will be either completely excluded (e.g. if only cyrillic + # letters are to be matched), or completely included (e.g. if large + # subranges of Kanji match). These ranges will be represented by + # chunks of all one-bits or all zero-bits. + + # Matching can be also done efficiently: the more significant byte of + # the Unicode character is an index into the chunk number, and the + # less significant byte is a bit index in the chunk (just like the + # CHARSET matching). + + charmap = bytes(charmap) # should be hashable + comps = {} + mapping = bytearray(256) + block = 0 + data = bytearray() + for i in range(0, 65536, 256): + chunk = charmap[i: i + 256] + if chunk in comps: + mapping[i // 256] = comps[chunk] + else: + mapping[i // 256] = comps[chunk] = block + block += 1 + data += chunk + data = _mk_bitmap(data) + data[0:0] = [block] + _bytes_to_codes(mapping) + out.append((BIGCHARSET, data)) + out += tail + return out, hascased + +_CODEBITS = _sre.CODESIZE * 8 +MAXCODE = (1 << _CODEBITS) - 1 +_BITS_TRANS = b'0' + b'1' * 255 +def _mk_bitmap(bits, _CODEBITS=_CODEBITS, _int=int): + s = bits.translate(_BITS_TRANS)[::-1] + return [_int(s[i - _CODEBITS: i], 2) + for i in range(len(s), 0, -_CODEBITS)] + +def _bytes_to_codes(b): + # Convert block indices to word array + a = memoryview(b).cast('I') + assert a.itemsize == _sre.CODESIZE + assert len(a) * a.itemsize == len(b) + return a.tolist() + +def _simple(p): + # check if this subpattern is a "simple" operator + if len(p) != 1: + return False + op, av = p[0] + if op is SUBPATTERN: + return av[0] is None and _simple(av[-1]) + return op in _UNIT_CODES + +def _generate_overlap_table(prefix): + """ + Generate an overlap table for the following prefix. + An overlap table is a table of the same size as the prefix which + informs about the potential self-overlap for each index in the prefix: + - if overlap[i] == 0, prefix[i:] can't overlap prefix[0:...] + - if overlap[i] == k with 0 < k <= i, prefix[i-k+1:i+1] overlaps with + prefix[0:k] + """ + table = [0] * len(prefix) + for i in range(1, len(prefix)): + idx = table[i - 1] + while prefix[i] != prefix[idx]: + if idx == 0: + table[i] = 0 + break + idx = table[idx - 1] + else: + table[i] = idx + 1 + return table + +def _get_iscased(flags): + if not flags & SRE_FLAG_IGNORECASE: + return None + elif flags & SRE_FLAG_UNICODE: + return _sre.unicode_iscased + else: + return _sre.ascii_iscased + +def _get_literal_prefix(pattern, flags): + # look for literal prefix + prefix = [] + prefixappend = prefix.append + prefix_skip = None + iscased = _get_iscased(flags) + for op, av in pattern.data: + if op is LITERAL: + if iscased and iscased(av): + break + prefixappend(av) + elif op is SUBPATTERN: + group, add_flags, del_flags, p = av + flags1 = _combine_flags(flags, add_flags, del_flags) + if flags1 & SRE_FLAG_IGNORECASE and flags1 & SRE_FLAG_LOCALE: + break + prefix1, prefix_skip1, got_all = _get_literal_prefix(p, flags1) + if prefix_skip is None: + if group is not None: + prefix_skip = len(prefix) + elif prefix_skip1 is not None: + prefix_skip = len(prefix) + prefix_skip1 + prefix.extend(prefix1) + if not got_all: + break + else: + break + else: + return prefix, prefix_skip, True + return prefix, prefix_skip, False + +def _get_charset_prefix(pattern, flags): + while True: + if not pattern.data: + return None + op, av = pattern.data[0] + if op is not SUBPATTERN: + break + group, add_flags, del_flags, pattern = av + flags = _combine_flags(flags, add_flags, del_flags) + if flags & SRE_FLAG_IGNORECASE and flags & SRE_FLAG_LOCALE: + return None + + iscased = _get_iscased(flags) + if op is LITERAL: + if iscased and iscased(av): + return None + return [(op, av)] + elif op is BRANCH: + charset = [] + charsetappend = charset.append + for p in av[1]: + if not p: + return None + op, av = p[0] + if op is LITERAL and not (iscased and iscased(av)): + charsetappend((op, av)) + else: + return None + return charset + elif op is IN: + charset = av + if iscased: + for op, av in charset: + if op is LITERAL: + if iscased(av): + return None + elif op is RANGE: + if av[1] > 0xffff: + return None + if any(map(iscased, range(av[0], av[1]+1))): + return None + return charset + return None + +def _compile_info(code, pattern, flags): + # internal: compile an info block. in the current version, + # this contains min/max pattern width, and an optional literal + # prefix or a character map + lo, hi = pattern.getwidth() + if hi > MAXCODE: + hi = MAXCODE + if lo == 0: + code.extend([INFO, 4, 0, lo, hi]) + return + # look for a literal prefix + prefix = [] + prefix_skip = 0 + charset = [] # not used + if not (flags & SRE_FLAG_IGNORECASE and flags & SRE_FLAG_LOCALE): + # look for literal prefix + prefix, prefix_skip, got_all = _get_literal_prefix(pattern, flags) + # if no prefix, look for charset prefix + if not prefix: + charset = _get_charset_prefix(pattern, flags) +## if prefix: +## print("*** PREFIX", prefix, prefix_skip) +## if charset: +## print("*** CHARSET", charset) + # add an info block + emit = code.append + emit(INFO) + skip = len(code); emit(0) + # literal flag + mask = 0 + if prefix: + mask = SRE_INFO_PREFIX + if prefix_skip is None and got_all: + mask = mask | SRE_INFO_LITERAL + elif charset: + mask = mask | SRE_INFO_CHARSET + emit(mask) + # pattern length + if lo < MAXCODE: + emit(lo) + else: + emit(MAXCODE) + prefix = prefix[:MAXCODE] + emit(min(hi, MAXCODE)) + # add literal prefix + if prefix: + emit(len(prefix)) # length + if prefix_skip is None: + prefix_skip = len(prefix) + emit(prefix_skip) # skip + code.extend(prefix) + # generate overlap table + code.extend(_generate_overlap_table(prefix)) + elif charset: + charset, hascased = _optimize_charset(charset) + assert not hascased + _compile_charset(charset, flags, code) + code[skip] = len(code) - skip + +def isstring(obj): + return isinstance(obj, (str, bytes)) + +def _code(p, flags): + + flags = p.state.flags | flags + code = [] + + # compile info block + _compile_info(code, p, flags) + + # compile the pattern + _compile(code, p.data, flags) + + code.append(SUCCESS) + + return code + +def _hex_code(code): + return '[%s]' % ', '.join('%#0*x' % (_sre.CODESIZE*2+2, x) for x in code) + +def dis(code): + import sys + + labels = set() + level = 0 + offset_width = len(str(len(code) - 1)) + + def dis_(start, end): + def print_(*args, to=None): + if to is not None: + labels.add(to) + args += ('(to %d)' % (to,),) + print('%*d%s ' % (offset_width, start, ':' if start in labels else '.'), + end=' '*(level-1)) + print(*args) + + def print_2(*args): + print(end=' '*(offset_width + 2*level)) + print(*args) + + nonlocal level + level += 1 + i = start + while i < end: + start = i + op = code[i] + i += 1 + op = OPCODES[op] + if op in (SUCCESS, FAILURE, ANY, ANY_ALL, + MAX_UNTIL, MIN_UNTIL, NEGATE): + print_(op) + elif op in (LITERAL, NOT_LITERAL, + LITERAL_IGNORE, NOT_LITERAL_IGNORE, + LITERAL_UNI_IGNORE, NOT_LITERAL_UNI_IGNORE, + LITERAL_LOC_IGNORE, NOT_LITERAL_LOC_IGNORE): + arg = code[i] + i += 1 + print_(op, '%#02x (%r)' % (arg, chr(arg))) + elif op is AT: + arg = code[i] + i += 1 + arg = str(ATCODES[arg]) + assert arg[:3] == 'AT_' + print_(op, arg[3:]) + elif op is CATEGORY: + arg = code[i] + i += 1 + arg = str(CHCODES[arg]) + assert arg[:9] == 'CATEGORY_' + print_(op, arg[9:]) + elif op in (IN, IN_IGNORE, IN_UNI_IGNORE, IN_LOC_IGNORE): + skip = code[i] + print_(op, skip, to=i+skip) + dis_(i+1, i+skip) + i += skip + elif op in (RANGE, RANGE_UNI_IGNORE): + lo, hi = code[i: i+2] + i += 2 + print_(op, '%#02x %#02x (%r-%r)' % (lo, hi, chr(lo), chr(hi))) + elif op is CHARSET: + print_(op, _hex_code(code[i: i + 256//_CODEBITS])) + i += 256//_CODEBITS + elif op is BIGCHARSET: + arg = code[i] + i += 1 + mapping = list(b''.join(x.to_bytes(_sre.CODESIZE, sys.byteorder) + for x in code[i: i + 256//_sre.CODESIZE])) + print_(op, arg, mapping) + i += 256//_sre.CODESIZE + level += 1 + for j in range(arg): + print_2(_hex_code(code[i: i + 256//_CODEBITS])) + i += 256//_CODEBITS + level -= 1 + elif op in (MARK, GROUPREF, GROUPREF_IGNORE, GROUPREF_UNI_IGNORE, + GROUPREF_LOC_IGNORE): + arg = code[i] + i += 1 + print_(op, arg) + elif op is JUMP: + skip = code[i] + print_(op, skip, to=i+skip) + i += 1 + elif op is BRANCH: + skip = code[i] + print_(op, skip, to=i+skip) + while skip: + dis_(i+1, i+skip) + i += skip + start = i + skip = code[i] + if skip: + print_('branch', skip, to=i+skip) + else: + print_(FAILURE) + i += 1 + elif op in (REPEAT, REPEAT_ONE, MIN_REPEAT_ONE, + POSSESSIVE_REPEAT, POSSESSIVE_REPEAT_ONE): + skip, min, max = code[i: i+3] + if max == MAXREPEAT: + max = 'MAXREPEAT' + print_(op, skip, min, max, to=i+skip) + dis_(i+3, i+skip) + i += skip + elif op is GROUPREF_EXISTS: + arg, skip = code[i: i+2] + print_(op, arg, skip, to=i+skip) + i += 2 + elif op in (ASSERT, ASSERT_NOT): + skip, arg = code[i: i+2] + print_(op, skip, arg, to=i+skip) + dis_(i+2, i+skip) + i += skip + elif op is ATOMIC_GROUP: + skip = code[i] + print_(op, skip, to=i+skip) + dis_(i+1, i+skip) + i += skip + elif op is INFO: + skip, flags, min, max = code[i: i+4] + if max == MAXREPEAT: + max = 'MAXREPEAT' + print_(op, skip, bin(flags), min, max, to=i+skip) + start = i+4 + if flags & SRE_INFO_PREFIX: + prefix_len, prefix_skip = code[i+4: i+6] + print_2(' prefix_skip', prefix_skip) + start = i + 6 + prefix = code[start: start+prefix_len] + print_2(' prefix', + '[%s]' % ', '.join('%#02x' % x for x in prefix), + '(%r)' % ''.join(map(chr, prefix))) + start += prefix_len + print_2(' overlap', code[start: start+prefix_len]) + start += prefix_len + if flags & SRE_INFO_CHARSET: + level += 1 + print_2('in') + dis_(start, i+skip) + level -= 1 + i += skip + else: + raise ValueError(op) + + level -= 1 + + dis_(0, len(code)) + + +def compile(p, flags=0): + # internal: convert pattern list to internal format + + if isstring(p): + pattern = p + p = _parser.parse(p, flags) + else: + pattern = None + + code = _code(p, flags) + + if flags & SRE_FLAG_DEBUG: + print() + dis(code) + + # map in either direction + groupindex = p.state.groupdict + indexgroup = [None] * p.state.groups + for k, i in groupindex.items(): + indexgroup[i] = k + + return _sre.compile( + pattern, flags | p.state.flags, code, + p.state.groups-1, + groupindex, tuple(indexgroup) + ) diff --git a/Lib/re/_constants.py b/Lib/re/_constants.py new file mode 100644 index 0000000000000..c735edfea1f13 --- /dev/null +++ b/Lib/re/_constants.py @@ -0,0 +1,262 @@ +# +# Secret Labs' Regular Expression Engine +# +# various symbols used by the regular expression engine. +# run this script to update the _sre include files! +# +# Copyright (c) 1998-2001 by Secret Labs AB. All rights reserved. +# +# See the __init__.py file for information on usage and redistribution. +# + +"""Internal support module for sre""" + +# update when constants are added or removed + +MAGIC = 20220318 + +from _sre import MAXREPEAT, MAXGROUPS + +# SRE standard exception (access as sre.error) +# should this really be here? + +class error(Exception): + """Exception raised for invalid regular expressions. + + Attributes: + + msg: The unformatted error message + pattern: The regular expression pattern + pos: The index in the pattern where compilation failed (may be None) + lineno: The line corresponding to pos (may be None) + colno: The column corresponding to pos (may be None) + """ + + __module__ = 're' + + def __init__(self, msg, pattern=None, pos=None): + self.msg = msg + self.pattern = pattern + self.pos = pos + if pattern is not None and pos is not None: + msg = '%s at position %d' % (msg, pos) + if isinstance(pattern, str): + newline = '\n' + else: + newline = b'\n' + self.lineno = pattern.count(newline, 0, pos) + 1 + self.colno = pos - pattern.rfind(newline, 0, pos) + if newline in pattern: + msg = '%s (line %d, column %d)' % (msg, self.lineno, self.colno) + else: + self.lineno = self.colno = None + super().__init__(msg) + + +class _NamedIntConstant(int): + def __new__(cls, value, name): + self = super(_NamedIntConstant, cls).__new__(cls, value) + self.name = name + return self + + def __repr__(self): + return self.name + +MAXREPEAT = _NamedIntConstant(MAXREPEAT, 'MAXREPEAT') + +def _makecodes(names): + names = names.strip().split() + items = [_NamedIntConstant(i, name) for i, name in enumerate(names)] + globals().update({item.name: item for item in items}) + return items + +# operators +# failure=0 success=1 (just because it looks better that way :-) +OPCODES = _makecodes(""" + FAILURE SUCCESS + + ANY ANY_ALL + ASSERT ASSERT_NOT + AT + BRANCH + CALL + CATEGORY + CHARSET BIGCHARSET + GROUPREF GROUPREF_EXISTS + IN + INFO + JUMP + LITERAL + MARK + MAX_UNTIL + MIN_UNTIL + NOT_LITERAL + NEGATE + RANGE + REPEAT + REPEAT_ONE + SUBPATTERN + MIN_REPEAT_ONE + ATOMIC_GROUP + POSSESSIVE_REPEAT + POSSESSIVE_REPEAT_ONE + + GROUPREF_IGNORE + IN_IGNORE + LITERAL_IGNORE + NOT_LITERAL_IGNORE + + GROUPREF_LOC_IGNORE + IN_LOC_IGNORE + LITERAL_LOC_IGNORE + NOT_LITERAL_LOC_IGNORE + + GROUPREF_UNI_IGNORE + IN_UNI_IGNORE + LITERAL_UNI_IGNORE + NOT_LITERAL_UNI_IGNORE + RANGE_UNI_IGNORE + + MIN_REPEAT MAX_REPEAT +""") +del OPCODES[-2:] # remove MIN_REPEAT and MAX_REPEAT + +# positions +ATCODES = _makecodes(""" + AT_BEGINNING AT_BEGINNING_LINE AT_BEGINNING_STRING + AT_BOUNDARY AT_NON_BOUNDARY + AT_END AT_END_LINE AT_END_STRING + + AT_LOC_BOUNDARY AT_LOC_NON_BOUNDARY + + AT_UNI_BOUNDARY AT_UNI_NON_BOUNDARY +""") + +# categories +CHCODES = _makecodes(""" + CATEGORY_DIGIT CATEGORY_NOT_DIGIT + CATEGORY_SPACE CATEGORY_NOT_SPACE + CATEGORY_WORD CATEGORY_NOT_WORD + CATEGORY_LINEBREAK CATEGORY_NOT_LINEBREAK + + CATEGORY_LOC_WORD CATEGORY_LOC_NOT_WORD + + CATEGORY_UNI_DIGIT CATEGORY_UNI_NOT_DIGIT + CATEGORY_UNI_SPACE CATEGORY_UNI_NOT_SPACE + CATEGORY_UNI_WORD CATEGORY_UNI_NOT_WORD + CATEGORY_UNI_LINEBREAK CATEGORY_UNI_NOT_LINEBREAK +""") + + +# replacement operations for "ignore case" mode +OP_IGNORE = { + LITERAL: LITERAL_IGNORE, + NOT_LITERAL: NOT_LITERAL_IGNORE, +} + +OP_LOCALE_IGNORE = { + LITERAL: LITERAL_LOC_IGNORE, + NOT_LITERAL: NOT_LITERAL_LOC_IGNORE, +} + +OP_UNICODE_IGNORE = { + LITERAL: LITERAL_UNI_IGNORE, + NOT_LITERAL: NOT_LITERAL_UNI_IGNORE, +} + +AT_MULTILINE = { + AT_BEGINNING: AT_BEGINNING_LINE, + AT_END: AT_END_LINE +} + +AT_LOCALE = { + AT_BOUNDARY: AT_LOC_BOUNDARY, + AT_NON_BOUNDARY: AT_LOC_NON_BOUNDARY +} + +AT_UNICODE = { + AT_BOUNDARY: AT_UNI_BOUNDARY, + AT_NON_BOUNDARY: AT_UNI_NON_BOUNDARY +} + +CH_LOCALE = { + CATEGORY_DIGIT: CATEGORY_DIGIT, + CATEGORY_NOT_DIGIT: CATEGORY_NOT_DIGIT, + CATEGORY_SPACE: CATEGORY_SPACE, + CATEGORY_NOT_SPACE: CATEGORY_NOT_SPACE, + CATEGORY_WORD: CATEGORY_LOC_WORD, + CATEGORY_NOT_WORD: CATEGORY_LOC_NOT_WORD, + CATEGORY_LINEBREAK: CATEGORY_LINEBREAK, + CATEGORY_NOT_LINEBREAK: CATEGORY_NOT_LINEBREAK +} + +CH_UNICODE = { + CATEGORY_DIGIT: CATEGORY_UNI_DIGIT, + CATEGORY_NOT_DIGIT: CATEGORY_UNI_NOT_DIGIT, + CATEGORY_SPACE: CATEGORY_UNI_SPACE, + CATEGORY_NOT_SPACE: CATEGORY_UNI_NOT_SPACE, + CATEGORY_WORD: CATEGORY_UNI_WORD, + CATEGORY_NOT_WORD: CATEGORY_UNI_NOT_WORD, + CATEGORY_LINEBREAK: CATEGORY_UNI_LINEBREAK, + CATEGORY_NOT_LINEBREAK: CATEGORY_UNI_NOT_LINEBREAK +} + +# flags +SRE_FLAG_TEMPLATE = 1 # template mode (disable backtracking) +SRE_FLAG_IGNORECASE = 2 # case insensitive +SRE_FLAG_LOCALE = 4 # honour system locale +SRE_FLAG_MULTILINE = 8 # treat target as multiline string +SRE_FLAG_DOTALL = 16 # treat target as a single string +SRE_FLAG_UNICODE = 32 # use unicode "locale" +SRE_FLAG_VERBOSE = 64 # ignore whitespace and comments +SRE_FLAG_DEBUG = 128 # debugging +SRE_FLAG_ASCII = 256 # use ascii "locale" + +# flags for INFO primitive +SRE_INFO_PREFIX = 1 # has prefix +SRE_INFO_LITERAL = 2 # entire pattern is literal (given by prefix) +SRE_INFO_CHARSET = 4 # pattern starts with character from given set + +if __name__ == "__main__": + def dump(f, d, prefix): + items = sorted(d) + for item in items: + f.write("#define %s_%s %d\n" % (prefix, item, item)) + with open("sre_constants.h", "w") as f: + f.write("""\ +/* + * Secret Labs' Regular Expression Engine + * + * regular expression matching engine + * + * NOTE: This file is generated by Lib/re/_constants.py. If you need + * to change anything in here, edit Lib/re/_constants.py and run it. + * + * Copyright (c) 1997-2001 by Secret Labs AB. All rights reserved. + * + * See the _sre.c file for information on usage and redistribution. + */ + +""") + + f.write("#define SRE_MAGIC %d\n" % MAGIC) + + dump(f, OPCODES, "SRE_OP") + dump(f, ATCODES, "SRE") + dump(f, CHCODES, "SRE") + + f.write("#define SRE_FLAG_TEMPLATE %d\n" % SRE_FLAG_TEMPLATE) + f.write("#define SRE_FLAG_IGNORECASE %d\n" % SRE_FLAG_IGNORECASE) + f.write("#define SRE_FLAG_LOCALE %d\n" % SRE_FLAG_LOCALE) + f.write("#define SRE_FLAG_MULTILINE %d\n" % SRE_FLAG_MULTILINE) + f.write("#define SRE_FLAG_DOTALL %d\n" % SRE_FLAG_DOTALL) + f.write("#define SRE_FLAG_UNICODE %d\n" % SRE_FLAG_UNICODE) + f.write("#define SRE_FLAG_VERBOSE %d\n" % SRE_FLAG_VERBOSE) + f.write("#define SRE_FLAG_DEBUG %d\n" % SRE_FLAG_DEBUG) + f.write("#define SRE_FLAG_ASCII %d\n" % SRE_FLAG_ASCII) + + f.write("#define SRE_INFO_PREFIX %d\n" % SRE_INFO_PREFIX) + f.write("#define SRE_INFO_LITERAL %d\n" % SRE_INFO_LITERAL) + f.write("#define SRE_INFO_CHARSET %d\n" % SRE_INFO_CHARSET) + + print("done") diff --git a/Lib/re/_parser.py b/Lib/re/_parser.py new file mode 100644 index 0000000000000..ae44118564eb7 --- /dev/null +++ b/Lib/re/_parser.py @@ -0,0 +1,1079 @@ +# +# Secret Labs' Regular Expression Engine +# +# convert re-style regular expression to sre pattern +# +# Copyright (c) 1998-2001 by Secret Labs AB. All rights reserved. +# +# See the __init__.py file for information on usage and redistribution. +# + +"""Internal support module for sre""" + +# XXX: show string offset and offending character for all errors + +from ._constants import * + +SPECIAL_CHARS = ".\\[{()*+?^$|" +REPEAT_CHARS = "*+?{" + +DIGITS = frozenset("0123456789") + +OCTDIGITS = frozenset("01234567") +HEXDIGITS = frozenset("0123456789abcdefABCDEF") +ASCIILETTERS = frozenset("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ") + +WHITESPACE = frozenset(" \t\n\r\v\f") + +_REPEATCODES = frozenset({MIN_REPEAT, MAX_REPEAT, POSSESSIVE_REPEAT}) +_UNITCODES = frozenset({ANY, RANGE, IN, LITERAL, NOT_LITERAL, CATEGORY}) + +ESCAPES = { + r"\a": (LITERAL, ord("\a")), + r"\b": (LITERAL, ord("\b")), + r"\f": (LITERAL, ord("\f")), + r"\n": (LITERAL, ord("\n")), + r"\r": (LITERAL, ord("\r")), + r"\t": (LITERAL, ord("\t")), + r"\v": (LITERAL, ord("\v")), + r"\\": (LITERAL, ord("\\")) +} + +CATEGORIES = { + r"\A": (AT, AT_BEGINNING_STRING), # start of string + r"\b": (AT, AT_BOUNDARY), + r"\B": (AT, AT_NON_BOUNDARY), + r"\d": (IN, [(CATEGORY, CATEGORY_DIGIT)]), + r"\D": (IN, [(CATEGORY, CATEGORY_NOT_DIGIT)]), + r"\s": (IN, [(CATEGORY, CATEGORY_SPACE)]), + r"\S": (IN, [(CATEGORY, CATEGORY_NOT_SPACE)]), + r"\w": (IN, [(CATEGORY, CATEGORY_WORD)]), + r"\W": (IN, [(CATEGORY, CATEGORY_NOT_WORD)]), + r"\Z": (AT, AT_END_STRING), # end of string +} + +FLAGS = { + # standard flags + "i": SRE_FLAG_IGNORECASE, + "L": SRE_FLAG_LOCALE, + "m": SRE_FLAG_MULTILINE, + "s": SRE_FLAG_DOTALL, + "x": SRE_FLAG_VERBOSE, + # extensions + "a": SRE_FLAG_ASCII, + "t": SRE_FLAG_TEMPLATE, + "u": SRE_FLAG_UNICODE, +} + +TYPE_FLAGS = SRE_FLAG_ASCII | SRE_FLAG_LOCALE | SRE_FLAG_UNICODE +GLOBAL_FLAGS = SRE_FLAG_DEBUG | SRE_FLAG_TEMPLATE + +class Verbose(Exception): + pass + +class State: + # keeps track of state for parsing + def __init__(self): + self.flags = 0 + self.groupdict = {} + self.groupwidths = [None] # group 0 + self.lookbehindgroups = None + @property + def groups(self): + return len(self.groupwidths) + def opengroup(self, name=None): + gid = self.groups + self.groupwidths.append(None) + if self.groups > MAXGROUPS: + raise error("too many groups") + if name is not None: + ogid = self.groupdict.get(name, None) + if ogid is not None: + raise error("redefinition of group name %r as group %d; " + "was group %d" % (name, gid, ogid)) + self.groupdict[name] = gid + return gid + def closegroup(self, gid, p): + self.groupwidths[gid] = p.getwidth() + def checkgroup(self, gid): + return gid < self.groups and self.groupwidths[gid] is not None + + def checklookbehindgroup(self, gid, source): + if self.lookbehindgroups is not None: + if not self.checkgroup(gid): + raise source.error('cannot refer to an open group') + if gid >= self.lookbehindgroups: + raise source.error('cannot refer to group defined in the same ' + 'lookbehind subpattern') + +class SubPattern: + # a subpattern, in intermediate form + def __init__(self, state, data=None): + self.state = state + if data is None: + data = [] + self.data = data + self.width = None + + def dump(self, level=0): + nl = True + seqtypes = (tuple, list) + for op, av in self.data: + print(level*" " + str(op), end='') + if op is IN: + # member sublanguage + print() + for op, a in av: + print((level+1)*" " + str(op), a) + elif op is BRANCH: + print() + for i, a in enumerate(av[1]): + if i: + print(level*" " + "OR") + a.dump(level+1) + elif op is GROUPREF_EXISTS: + condgroup, item_yes, item_no = av + print('', condgroup) + item_yes.dump(level+1) + if item_no: + print(level*" " + "ELSE") + item_no.dump(level+1) + elif isinstance(av, seqtypes): + nl = False + for a in av: + if isinstance(a, SubPattern): + if not nl: + print() + a.dump(level+1) + nl = True + else: + if not nl: + print(' ', end='') + print(a, end='') + nl = False + if not nl: + print() + else: + print('', av) + def __repr__(self): + return repr(self.data) + def __len__(self): + return len(self.data) + def __delitem__(self, index): + del self.data[index] + def __getitem__(self, index): + if isinstance(index, slice): + return SubPattern(self.state, self.data[index]) + return self.data[index] + def __setitem__(self, index, code): + self.data[index] = code + def insert(self, index, code): + self.data.insert(index, code) + def append(self, code): + self.data.append(code) + def getwidth(self): + # determine the width (min, max) for this subpattern + if self.width is not None: + return self.width + lo = hi = 0 + for op, av in self.data: + if op is BRANCH: + i = MAXREPEAT - 1 + j = 0 + for av in av[1]: + l, h = av.getwidth() + i = min(i, l) + j = max(j, h) + lo = lo + i + hi = hi + j + elif op is CALL: + i, j = av.getwidth() + lo = lo + i + hi = hi + j + elif op is ATOMIC_GROUP: + i, j = av.getwidth() + lo = lo + i + hi = hi + j + elif op is SUBPATTERN: + i, j = av[-1].getwidth() + lo = lo + i + hi = hi + j + elif op in _REPEATCODES: + i, j = av[2].getwidth() + lo = lo + i * av[0] + hi = hi + j * av[1] + elif op in _UNITCODES: + lo = lo + 1 + hi = hi + 1 + elif op is GROUPREF: + i, j = self.state.groupwidths[av] + lo = lo + i + hi = hi + j + elif op is GROUPREF_EXISTS: + i, j = av[1].getwidth() + if av[2] is not None: + l, h = av[2].getwidth() + i = min(i, l) + j = max(j, h) + else: + i = 0 + lo = lo + i + hi = hi + j + elif op is SUCCESS: + break + self.width = min(lo, MAXREPEAT - 1), min(hi, MAXREPEAT) + return self.width + +class Tokenizer: + def __init__(self, string): + self.istext = isinstance(string, str) + self.string = string + if not self.istext: + string = str(string, 'latin1') + self.decoded_string = string + self.index = 0 + self.next = None + self.__next() + def __next(self): + index = self.index + try: + char = self.decoded_string[index] + except IndexError: + self.next = None + return + if char == "\\": + index += 1 + try: + char += self.decoded_string[index] + except IndexError: + raise error("bad escape (end of pattern)", + self.string, len(self.string) - 1) from None + self.index = index + 1 + self.next = char + def match(self, char): + if char == self.next: + self.__next() + return True + return False + def get(self): + this = self.next + self.__next() + return this + def getwhile(self, n, charset): + result = '' + for _ in range(n): + c = self.next + if c not in charset: + break + result += c + self.__next() + return result + def getuntil(self, terminator, name): + result = '' + while True: + c = self.next + self.__next() + if c is None: + if not result: + raise self.error("missing " + name) + raise self.error("missing %s, unterminated name" % terminator, + len(result)) + if c == terminator: + if not result: + raise self.error("missing " + name, 1) + break + result += c + return result + @property + def pos(self): + return self.index - len(self.next or '') + def tell(self): + return self.index - len(self.next or '') + def seek(self, index): + self.index = index + self.__next() + + def error(self, msg, offset=0): + return error(msg, self.string, self.tell() - offset) + +def _class_escape(source, escape): + # handle escape code inside character class + code = ESCAPES.get(escape) + if code: + return code + code = CATEGORIES.get(escape) + if code and code[0] is IN: + return code + try: + c = escape[1:2] + if c == "x": + # hexadecimal escape (exactly two digits) + escape += source.getwhile(2, HEXDIGITS) + if len(escape) != 4: + raise source.error("incomplete escape %s" % escape, len(escape)) + return LITERAL, int(escape[2:], 16) + elif c == "u" and source.istext: + # unicode escape (exactly four digits) + escape += source.getwhile(4, HEXDIGITS) + if len(escape) != 6: + raise source.error("incomplete escape %s" % escape, len(escape)) + return LITERAL, int(escape[2:], 16) + elif c == "U" and source.istext: + # unicode escape (exactly eight digits) + escape += source.getwhile(8, HEXDIGITS) + if len(escape) != 10: + raise source.error("incomplete escape %s" % escape, len(escape)) + c = int(escape[2:], 16) + chr(c) # raise ValueError for invalid code + return LITERAL, c + elif c == "N" and source.istext: + import unicodedata + # named unicode escape e.g. \N{EM DASH} + if not source.match('{'): + raise source.error("missing {") + charname = source.getuntil('}', 'character name') + try: + c = ord(unicodedata.lookup(charname)) + except KeyError: + raise source.error("undefined character name %r" % charname, + len(charname) + len(r'\N{}')) + return LITERAL, c + elif c in OCTDIGITS: + # octal escape (up to three digits) + escape += source.getwhile(2, OCTDIGITS) + c = int(escape[1:], 8) + if c > 0o377: + raise source.error('octal escape value %s outside of ' + 'range 0-0o377' % escape, len(escape)) + return LITERAL, c + elif c in DIGITS: + raise ValueError + if len(escape) == 2: + if c in ASCIILETTERS: + raise source.error('bad escape %s' % escape, len(escape)) + return LITERAL, ord(escape[1]) + except ValueError: + pass + raise source.error("bad escape %s" % escape, len(escape)) + +def _escape(source, escape, state): + # handle escape code in expression + code = CATEGORIES.get(escape) + if code: + return code + code = ESCAPES.get(escape) + if code: + return code + try: + c = escape[1:2] + if c == "x": + # hexadecimal escape + escape += source.getwhile(2, HEXDIGITS) + if len(escape) != 4: + raise source.error("incomplete escape %s" % escape, len(escape)) + return LITERAL, int(escape[2:], 16) + elif c == "u" and source.istext: + # unicode escape (exactly four digits) + escape += source.getwhile(4, HEXDIGITS) + if len(escape) != 6: + raise source.error("incomplete escape %s" % escape, len(escape)) + return LITERAL, int(escape[2:], 16) + elif c == "U" and source.istext: + # unicode escape (exactly eight digits) + escape += source.getwhile(8, HEXDIGITS) + if len(escape) != 10: + raise source.error("incomplete escape %s" % escape, len(escape)) + c = int(escape[2:], 16) + chr(c) # raise ValueError for invalid code + return LITERAL, c + elif c == "N" and source.istext: + import unicodedata + # named unicode escape e.g. \N{EM DASH} + if not source.match('{'): + raise source.error("missing {") + charname = source.getuntil('}', 'character name') + try: + c = ord(unicodedata.lookup(charname)) + except KeyError: + raise source.error("undefined character name %r" % charname, + len(charname) + len(r'\N{}')) + return LITERAL, c + elif c == "0": + # octal escape + escape += source.getwhile(2, OCTDIGITS) + return LITERAL, int(escape[1:], 8) + elif c in DIGITS: + # octal escape *or* decimal group reference (sigh) + if source.next in DIGITS: + escape += source.get() + if (escape[1] in OCTDIGITS and escape[2] in OCTDIGITS and + source.next in OCTDIGITS): + # got three octal digits; this is an octal escape + escape += source.get() + c = int(escape[1:], 8) + if c > 0o377: + raise source.error('octal escape value %s outside of ' + 'range 0-0o377' % escape, + len(escape)) + return LITERAL, c + # not an octal escape, so this is a group reference + group = int(escape[1:]) + if group < state.groups: + if not state.checkgroup(group): + raise source.error("cannot refer to an open group", + len(escape)) + state.checklookbehindgroup(group, source) + return GROUPREF, group + raise source.error("invalid group reference %d" % group, len(escape) - 1) + if len(escape) == 2: + if c in ASCIILETTERS: + raise source.error("bad escape %s" % escape, len(escape)) + return LITERAL, ord(escape[1]) + except ValueError: + pass + raise source.error("bad escape %s" % escape, len(escape)) + +def _uniq(items): + return list(dict.fromkeys(items)) + +def _parse_sub(source, state, verbose, nested): + # parse an alternation: a|b|c + + items = [] + itemsappend = items.append + sourcematch = source.match + start = source.tell() + while True: + itemsappend(_parse(source, state, verbose, nested + 1, + not nested and not items)) + if not sourcematch("|"): + break + + if len(items) == 1: + return items[0] + + subpattern = SubPattern(state) + + # check if all items share a common prefix + while True: + prefix = None + for item in items: + if not item: + break + if prefix is None: + prefix = item[0] + elif item[0] != prefix: + break + else: + # all subitems start with a common "prefix". + # move it out of the branch + for item in items: + del item[0] + subpattern.append(prefix) + continue # check next one + break + + # check if the branch can be replaced by a character set + set = [] + for item in items: + if len(item) != 1: + break + op, av = item[0] + if op is LITERAL: + set.append((op, av)) + elif op is IN and av[0][0] is not NEGATE: + set.extend(av) + else: + break + else: + # we can store this as a character set instead of a + # branch (the compiler may optimize this even more) + subpattern.append((IN, _uniq(set))) + return subpattern + + subpattern.append((BRANCH, (None, items))) + return subpattern + +def _parse(source, state, verbose, nested, first=False): + # parse a simple pattern + subpattern = SubPattern(state) + + # precompute constants into local variables + subpatternappend = subpattern.append + sourceget = source.get + sourcematch = source.match + _len = len + _ord = ord + + while True: + + this = source.next + if this is None: + break # end of pattern + if this in "|)": + break # end of subpattern + sourceget() + + if verbose: + # skip whitespace and comments + if this in WHITESPACE: + continue + if this == "#": + while True: + this = sourceget() + if this is None or this == "\n": + break + continue + + if this[0] == "\\": + code = _escape(source, this, state) + subpatternappend(code) + + elif this not in SPECIAL_CHARS: + subpatternappend((LITERAL, _ord(this))) + + elif this == "[": + here = source.tell() - 1 + # character set + set = [] + setappend = set.append +## if sourcematch(":"): +## pass # handle character classes + if source.next == '[': + import warnings + warnings.warn( + 'Possible nested set at position %d' % source.tell(), + FutureWarning, stacklevel=nested + 6 + ) + negate = sourcematch("^") + # check remaining characters + while True: + this = sourceget() + if this is None: + raise source.error("unterminated character set", + source.tell() - here) + if this == "]" and set: + break + elif this[0] == "\\": + code1 = _class_escape(source, this) + else: + if set and this in '-&~|' and source.next == this: + import warnings + warnings.warn( + 'Possible set %s at position %d' % ( + 'difference' if this == '-' else + 'intersection' if this == '&' else + 'symmetric difference' if this == '~' else + 'union', + source.tell() - 1), + FutureWarning, stacklevel=nested + 6 + ) + code1 = LITERAL, _ord(this) + if sourcematch("-"): + # potential range + that = sourceget() + if that is None: + raise source.error("unterminated character set", + source.tell() - here) + if that == "]": + if code1[0] is IN: + code1 = code1[1][0] + setappend(code1) + setappend((LITERAL, _ord("-"))) + break + if that[0] == "\\": + code2 = _class_escape(source, that) + else: + if that == '-': + import warnings + warnings.warn( + 'Possible set difference at position %d' % ( + source.tell() - 2), + FutureWarning, stacklevel=nested + 6 + ) + code2 = LITERAL, _ord(that) + if code1[0] != LITERAL or code2[0] != LITERAL: + msg = "bad character range %s-%s" % (this, that) + raise source.error(msg, len(this) + 1 + len(that)) + lo = code1[1] + hi = code2[1] + if hi < lo: + msg = "bad character range %s-%s" % (this, that) + raise source.error(msg, len(this) + 1 + len(that)) + setappend((RANGE, (lo, hi))) + else: + if code1[0] is IN: + code1 = code1[1][0] + setappend(code1) + + set = _uniq(set) + # XXX: should move set optimization to compiler! + if _len(set) == 1 and set[0][0] is LITERAL: + # optimization + if negate: + subpatternappend((NOT_LITERAL, set[0][1])) + else: + subpatternappend(set[0]) + else: + if negate: + set.insert(0, (NEGATE, None)) + # charmap optimization can't be added here because + # global flags still are not known + subpatternappend((IN, set)) + + elif this in REPEAT_CHARS: + # repeat previous item + here = source.tell() + if this == "?": + min, max = 0, 1 + elif this == "*": + min, max = 0, MAXREPEAT + + elif this == "+": + min, max = 1, MAXREPEAT + elif this == "{": + if source.next == "}": + subpatternappend((LITERAL, _ord(this))) + continue + + min, max = 0, MAXREPEAT + lo = hi = "" + while source.next in DIGITS: + lo += sourceget() + if sourcematch(","): + while source.next in DIGITS: + hi += sourceget() + else: + hi = lo + if not sourcematch("}"): + subpatternappend((LITERAL, _ord(this))) + source.seek(here) + continue + + if lo: + min = int(lo) + if min >= MAXREPEAT: + raise OverflowError("the repetition number is too large") + if hi: + max = int(hi) + if max >= MAXREPEAT: + raise OverflowError("the repetition number is too large") + if max < min: + raise source.error("min repeat greater than max repeat", + source.tell() - here) + else: + raise AssertionError("unsupported quantifier %r" % (char,)) + # figure out which item to repeat + if subpattern: + item = subpattern[-1:] + else: + item = None + if not item or item[0][0] is AT: + raise source.error("nothing to repeat", + source.tell() - here + len(this)) + if item[0][0] in _REPEATCODES: + raise source.error("multiple repeat", + source.tell() - here + len(this)) + if item[0][0] is SUBPATTERN: + group, add_flags, del_flags, p = item[0][1] + if group is None and not add_flags and not del_flags: + item = p + if sourcematch("?"): + # Non-Greedy Match + subpattern[-1] = (MIN_REPEAT, (min, max, item)) + elif sourcematch("+"): + # Possessive Match (Always Greedy) + subpattern[-1] = (POSSESSIVE_REPEAT, (min, max, item)) + else: + # Greedy Match + subpattern[-1] = (MAX_REPEAT, (min, max, item)) + + elif this == ".": + subpatternappend((ANY, None)) + + elif this == "(": + start = source.tell() - 1 + capture = True + atomic = False + name = None + add_flags = 0 + del_flags = 0 + if sourcematch("?"): + # options + char = sourceget() + if char is None: + raise source.error("unexpected end of pattern") + if char == "P": + # python extensions + if sourcematch("<"): + # named group: skip forward to end of name + name = source.getuntil(">", "group name") + if not name.isidentifier(): + msg = "bad character in group name %r" % name + raise source.error(msg, len(name) + 1) + elif sourcematch("="): + # named backreference + name = source.getuntil(")", "group name") + if not name.isidentifier(): + msg = "bad character in group name %r" % name + raise source.error(msg, len(name) + 1) + gid = state.groupdict.get(name) + if gid is None: + msg = "unknown group name %r" % name + raise source.error(msg, len(name) + 1) + if not state.checkgroup(gid): + raise source.error("cannot refer to an open group", + len(name) + 1) + state.checklookbehindgroup(gid, source) + subpatternappend((GROUPREF, gid)) + continue + + else: + char = sourceget() + if char is None: + raise source.error("unexpected end of pattern") + raise source.error("unknown extension ?P" + char, + len(char) + 2) + elif char == ":": + # non-capturing group + capture = False + elif char == "#": + # comment + while True: + if source.next is None: + raise source.error("missing ), unterminated comment", + source.tell() - start) + if sourceget() == ")": + break + continue + + elif char in "=!<": + # lookahead assertions + dir = 1 + if char == "<": + char = sourceget() + if char is None: + raise source.error("unexpected end of pattern") + if char not in "=!": + raise source.error("unknown extension ?<" + char, + len(char) + 2) + dir = -1 # lookbehind + lookbehindgroups = state.lookbehindgroups + if lookbehindgroups is None: + state.lookbehindgroups = state.groups + p = _parse_sub(source, state, verbose, nested + 1) + if dir < 0: + if lookbehindgroups is None: + state.lookbehindgroups = None + if not sourcematch(")"): + raise source.error("missing ), unterminated subpattern", + source.tell() - start) + if char == "=": + subpatternappend((ASSERT, (dir, p))) + else: + subpatternappend((ASSERT_NOT, (dir, p))) + continue + + elif char == "(": + # conditional backreference group + condname = source.getuntil(")", "group name") + if condname.isidentifier(): + condgroup = state.groupdict.get(condname) + if condgroup is None: + msg = "unknown group name %r" % condname + raise source.error(msg, len(condname) + 1) + else: + try: + condgroup = int(condname) + if condgroup < 0: + raise ValueError + except ValueError: + msg = "bad character in group name %r" % condname + raise source.error(msg, len(condname) + 1) from None + if not condgroup: + raise source.error("bad group number", + len(condname) + 1) + if condgroup >= MAXGROUPS: + msg = "invalid group reference %d" % condgroup + raise source.error(msg, len(condname) + 1) + state.checklookbehindgroup(condgroup, source) + item_yes = _parse(source, state, verbose, nested + 1) + if source.match("|"): + item_no = _parse(source, state, verbose, nested + 1) + if source.next == "|": + raise source.error("conditional backref with more than two branches") + else: + item_no = None + if not source.match(")"): + raise source.error("missing ), unterminated subpattern", + source.tell() - start) + subpatternappend((GROUPREF_EXISTS, (condgroup, item_yes, item_no))) + continue + + elif char == ">": + # non-capturing, atomic group + capture = False + atomic = True + elif char in FLAGS or char == "-": + # flags + flags = _parse_flags(source, state, char) + if flags is None: # global flags + if not first or subpattern: + raise source.error('global flags not at the start ' + 'of the expression', + source.tell() - start) + if (state.flags & SRE_FLAG_VERBOSE) and not verbose: + raise Verbose + continue + + add_flags, del_flags = flags + capture = False + else: + raise source.error("unknown extension ?" + char, + len(char) + 1) + + # parse group contents + if capture: + try: + group = state.opengroup(name) + except error as err: + raise source.error(err.msg, len(name) + 1) from None + else: + group = None + sub_verbose = ((verbose or (add_flags & SRE_FLAG_VERBOSE)) and + not (del_flags & SRE_FLAG_VERBOSE)) + p = _parse_sub(source, state, sub_verbose, nested + 1) + if not source.match(")"): + raise source.error("missing ), unterminated subpattern", + source.tell() - start) + if group is not None: + state.closegroup(group, p) + if atomic: + assert group is None + subpatternappend((ATOMIC_GROUP, p)) + else: + subpatternappend((SUBPATTERN, (group, add_flags, del_flags, p))) + + elif this == "^": + subpatternappend((AT, AT_BEGINNING)) + + elif this == "$": + subpatternappend((AT, AT_END)) + + else: + raise AssertionError("unsupported special character %r" % (char,)) + + # unpack non-capturing groups + for i in range(len(subpattern))[::-1]: + op, av = subpattern[i] + if op is SUBPATTERN: + group, add_flags, del_flags, p = av + if group is None and not add_flags and not del_flags: + subpattern[i: i+1] = p + + return subpattern + +def _parse_flags(source, state, char): + sourceget = source.get + add_flags = 0 + del_flags = 0 + if char != "-": + while True: + flag = FLAGS[char] + if source.istext: + if char == 'L': + msg = "bad inline flags: cannot use 'L' flag with a str pattern" + raise source.error(msg) + else: + if char == 'u': + msg = "bad inline flags: cannot use 'u' flag with a bytes pattern" + raise source.error(msg) + add_flags |= flag + if (flag & TYPE_FLAGS) and (add_flags & TYPE_FLAGS) != flag: + msg = "bad inline flags: flags 'a', 'u' and 'L' are incompatible" + raise source.error(msg) + char = sourceget() + if char is None: + raise source.error("missing -, : or )") + if char in ")-:": + break + if char not in FLAGS: + msg = "unknown flag" if char.isalpha() else "missing -, : or )" + raise source.error(msg, len(char)) + if char == ")": + state.flags |= add_flags + return None + if add_flags & GLOBAL_FLAGS: + raise source.error("bad inline flags: cannot turn on global flag", 1) + if char == "-": + char = sourceget() + if char is None: + raise source.error("missing flag") + if char not in FLAGS: + msg = "unknown flag" if char.isalpha() else "missing flag" + raise source.error(msg, len(char)) + while True: + flag = FLAGS[char] + if flag & TYPE_FLAGS: + msg = "bad inline flags: cannot turn off flags 'a', 'u' and 'L'" + raise source.error(msg) + del_flags |= flag + char = sourceget() + if char is None: + raise source.error("missing :") + if char == ":": + break + if char not in FLAGS: + msg = "unknown flag" if char.isalpha() else "missing :" + raise source.error(msg, len(char)) + assert char == ":" + if del_flags & GLOBAL_FLAGS: + raise source.error("bad inline flags: cannot turn off global flag", 1) + if add_flags & del_flags: + raise source.error("bad inline flags: flag turned on and off", 1) + return add_flags, del_flags + +def fix_flags(src, flags): + # Check and fix flags according to the type of pattern (str or bytes) + if isinstance(src, str): + if flags & SRE_FLAG_LOCALE: + raise ValueError("cannot use LOCALE flag with a str pattern") + if not flags & SRE_FLAG_ASCII: + flags |= SRE_FLAG_UNICODE + elif flags & SRE_FLAG_UNICODE: + raise ValueError("ASCII and UNICODE flags are incompatible") + else: + if flags & SRE_FLAG_UNICODE: + raise ValueError("cannot use UNICODE flag with a bytes pattern") + if flags & SRE_FLAG_LOCALE and flags & SRE_FLAG_ASCII: + raise ValueError("ASCII and LOCALE flags are incompatible") + return flags + +def parse(str, flags=0, state=None): + # parse 're' pattern into list of (opcode, argument) tuples + + source = Tokenizer(str) + + if state is None: + state = State() + state.flags = flags + state.str = str + + try: + p = _parse_sub(source, state, flags & SRE_FLAG_VERBOSE, 0) + except Verbose: + # the VERBOSE flag was switched on inside the pattern. to be + # on the safe side, we'll parse the whole thing again... + state = State() + state.flags = flags | SRE_FLAG_VERBOSE + state.str = str + source.seek(0) + p = _parse_sub(source, state, True, 0) + + p.state.flags = fix_flags(str, p.state.flags) + + if source.next is not None: + assert source.next == ")" + raise source.error("unbalanced parenthesis") + + if flags & SRE_FLAG_DEBUG: + p.dump() + + return p + +def parse_template(source, state): + # parse 're' replacement string into list of literals and + # group references + s = Tokenizer(source) + sget = s.get + groups = [] + literals = [] + literal = [] + lappend = literal.append + def addgroup(index, pos): + if index > state.groups: + raise s.error("invalid group reference %d" % index, pos) + if literal: + literals.append(''.join(literal)) + del literal[:] + groups.append((len(literals), index)) + literals.append(None) + groupindex = state.groupindex + while True: + this = sget() + if this is None: + break # end of replacement string + if this[0] == "\\": + # group + c = this[1] + if c == "g": + name = "" + if not s.match("<"): + raise s.error("missing <") + name = s.getuntil(">", "group name") + if name.isidentifier(): + try: + index = groupindex[name] + except KeyError: + raise IndexError("unknown group name %r" % name) + else: + try: + index = int(name) + if index < 0: + raise ValueError + except ValueError: + raise s.error("bad character in group name %r" % name, + len(name) + 1) from None + if index >= MAXGROUPS: + raise s.error("invalid group reference %d" % index, + len(name) + 1) + addgroup(index, len(name) + 1) + elif c == "0": + if s.next in OCTDIGITS: + this += sget() + if s.next in OCTDIGITS: + this += sget() + lappend(chr(int(this[1:], 8) & 0xff)) + elif c in DIGITS: + isoctal = False + if s.next in DIGITS: + this += sget() + if (c in OCTDIGITS and this[2] in OCTDIGITS and + s.next in OCTDIGITS): + this += sget() + isoctal = True + c = int(this[1:], 8) + if c > 0o377: + raise s.error('octal escape value %s outside of ' + 'range 0-0o377' % this, len(this)) + lappend(chr(c)) + if not isoctal: + addgroup(int(this[1:]), len(this) - 1) + else: + try: + this = chr(ESCAPES[this][1]) + except KeyError: + if c in ASCIILETTERS: + raise s.error('bad escape %s' % this, len(this)) + lappend(this) + else: + lappend(this) + if literal: + literals.append(''.join(literal)) + if not isinstance(source, str): + # The tokenizer implicitly decodes bytes objects as latin-1, we must + # therefore re-encode the final representation. + literals = [None if s is None else s.encode('latin-1') for s in literals] + return groups, literals + +def expand_template(template, match): + g = match.group + empty = match.string[:0] + groups, literals = template + literals = literals[:] + try: + for index, group in groups: + literals[index] = g(group) or empty + except IndexError: + raise error("invalid group reference %d" % index) + return empty.join(literals) diff --git a/Lib/sre_compile.py b/Lib/sre_compile.py index 0867200a59a23..f9da61e648786 100644 --- a/Lib/sre_compile.py +++ b/Lib/sre_compile.py @@ -1,800 +1,7 @@ -# -# Secret Labs' Regular Expression Engine -# -# convert template to internal format -# -# Copyright (c) 1997-2001 by Secret Labs AB. All rights reserved. -# -# See the sre.py file for information on usage and redistribution. -# +import warnings +warnings.warn(f"module {__name__!r} is deprecated", + DeprecationWarning, + stacklevel=2) -"""Internal support module for sre""" - -import _sre -import sre_parse -from sre_constants import * - -assert _sre.MAGIC == MAGIC, "SRE module mismatch" - -_LITERAL_CODES = {LITERAL, NOT_LITERAL} -_SUCCESS_CODES = {SUCCESS, FAILURE} -_ASSERT_CODES = {ASSERT, ASSERT_NOT} -_UNIT_CODES = _LITERAL_CODES | {ANY, IN} - -_REPEATING_CODES = { - MIN_REPEAT: (REPEAT, MIN_UNTIL, MIN_REPEAT_ONE), - MAX_REPEAT: (REPEAT, MAX_UNTIL, REPEAT_ONE), - POSSESSIVE_REPEAT: (POSSESSIVE_REPEAT, SUCCESS, POSSESSIVE_REPEAT_ONE), -} - -# Sets of lowercase characters which have the same uppercase. -_equivalences = ( - # LATIN SMALL LETTER I, LATIN SMALL LETTER DOTLESS I - (0x69, 0x131), # i? - # LATIN SMALL LETTER S, LATIN SMALL LETTER LONG S - (0x73, 0x17f), # s? - # MICRO SIGN, GREEK SMALL LETTER MU - (0xb5, 0x3bc), # ?? - # COMBINING GREEK YPOGEGRAMMENI, GREEK SMALL LETTER IOTA, GREEK PROSGEGRAMMENI - (0x345, 0x3b9, 0x1fbe), # \u0345?? - # GREEK SMALL LETTER IOTA WITH DIALYTIKA AND TONOS, GREEK SMALL LETTER IOTA WITH DIALYTIKA AND OXIA - (0x390, 0x1fd3), # ?? - # GREEK SMALL LETTER UPSILON WITH DIALYTIKA AND TONOS, GREEK SMALL LETTER UPSILON WITH DIALYTIKA AND OXIA - (0x3b0, 0x1fe3), # ?? - # GREEK SMALL LETTER BETA, GREEK BETA SYMBOL - (0x3b2, 0x3d0), # ?? - # GREEK SMALL LETTER EPSILON, GREEK LUNATE EPSILON SYMBOL - (0x3b5, 0x3f5), # ?? - # GREEK SMALL LETTER THETA, GREEK THETA SYMBOL - (0x3b8, 0x3d1), # ?? - # GREEK SMALL LETTER KAPPA, GREEK KAPPA SYMBOL - (0x3ba, 0x3f0), # ?? - # GREEK SMALL LETTER PI, GREEK PI SYMBOL - (0x3c0, 0x3d6), # ?? - # GREEK SMALL LETTER RHO, GREEK RHO SYMBOL - (0x3c1, 0x3f1), # ?? - # GREEK SMALL LETTER FINAL SIGMA, GREEK SMALL LETTER SIGMA - (0x3c2, 0x3c3), # ?? - # GREEK SMALL LETTER PHI, GREEK PHI SYMBOL - (0x3c6, 0x3d5), # ?? - # LATIN SMALL LETTER S WITH DOT ABOVE, LATIN SMALL LETTER LONG S WITH DOT ABOVE - (0x1e61, 0x1e9b), # ?? - # LATIN SMALL LIGATURE LONG S T, LATIN SMALL LIGATURE ST - (0xfb05, 0xfb06), # ?? -) - -# Maps the lowercase code to lowercase codes which have the same uppercase. -_ignorecase_fixes = {i: tuple(j for j in t if i != j) - for t in _equivalences for i in t} - -def _combine_flags(flags, add_flags, del_flags, - TYPE_FLAGS=sre_parse.TYPE_FLAGS): - if add_flags & TYPE_FLAGS: - flags &= ~TYPE_FLAGS - return (flags | add_flags) & ~del_flags - -def _compile(code, pattern, flags): - # internal: compile a (sub)pattern - emit = code.append - _len = len - LITERAL_CODES = _LITERAL_CODES - REPEATING_CODES = _REPEATING_CODES - SUCCESS_CODES = _SUCCESS_CODES - ASSERT_CODES = _ASSERT_CODES - iscased = None - tolower = None - fixes = None - if flags & SRE_FLAG_IGNORECASE and not flags & SRE_FLAG_LOCALE: - if flags & SRE_FLAG_UNICODE: - iscased = _sre.unicode_iscased - tolower = _sre.unicode_tolower - fixes = _ignorecase_fixes - else: - iscased = _sre.ascii_iscased - tolower = _sre.ascii_tolower - for op, av in pattern: - if op in LITERAL_CODES: - if not flags & SRE_FLAG_IGNORECASE: - emit(op) - emit(av) - elif flags & SRE_FLAG_LOCALE: - emit(OP_LOCALE_IGNORE[op]) - emit(av) - elif not iscased(av): - emit(op) - emit(av) - else: - lo = tolower(av) - if not fixes: # ascii - emit(OP_IGNORE[op]) - emit(lo) - elif lo not in fixes: - emit(OP_UNICODE_IGNORE[op]) - emit(lo) - else: - emit(IN_UNI_IGNORE) - skip = _len(code); emit(0) - if op is NOT_LITERAL: - emit(NEGATE) - for k in (lo,) + fixes[lo]: - emit(LITERAL) - emit(k) - emit(FAILURE) - code[skip] = _len(code) - skip - elif op is IN: - charset, hascased = _optimize_charset(av, iscased, tolower, fixes) - if flags & SRE_FLAG_IGNORECASE and flags & SRE_FLAG_LOCALE: - emit(IN_LOC_IGNORE) - elif not hascased: - emit(IN) - elif not fixes: # ascii - emit(IN_IGNORE) - else: - emit(IN_UNI_IGNORE) - skip = _len(code); emit(0) - _compile_charset(charset, flags, code) - code[skip] = _len(code) - skip - elif op is ANY: - if flags & SRE_FLAG_DOTALL: - emit(ANY_ALL) - else: - emit(ANY) - elif op in REPEATING_CODES: - if flags & SRE_FLAG_TEMPLATE: - raise error("internal: unsupported template operator %r" % (op,)) - if _simple(av[2]): - emit(REPEATING_CODES[op][2]) - skip = _len(code); emit(0) - emit(av[0]) - emit(av[1]) - _compile(code, av[2], flags) - emit(SUCCESS) - code[skip] = _len(code) - skip - else: - emit(REPEATING_CODES[op][0]) - skip = _len(code); emit(0) - emit(av[0]) - emit(av[1]) - _compile(code, av[2], flags) - code[skip] = _len(code) - skip - emit(REPEATING_CODES[op][1]) - elif op is SUBPATTERN: - group, add_flags, del_flags, p = av - if group: - emit(MARK) - emit((group-1)*2) - # _compile_info(code, p, _combine_flags(flags, add_flags, del_flags)) - _compile(code, p, _combine_flags(flags, add_flags, del_flags)) - if group: - emit(MARK) - emit((group-1)*2+1) - elif op is ATOMIC_GROUP: - # Atomic Groups are handled by starting with an Atomic - # Group op code, then putting in the atomic group pattern - # and finally a success op code to tell any repeat - # operations within the Atomic Group to stop eating and - # pop their stack if they reach it - emit(ATOMIC_GROUP) - skip = _len(code); emit(0) - _compile(code, av, flags) - emit(SUCCESS) - code[skip] = _len(code) - skip - elif op in SUCCESS_CODES: - emit(op) - elif op in ASSERT_CODES: - emit(op) - skip = _len(code); emit(0) - if av[0] >= 0: - emit(0) # look ahead - else: - lo, hi = av[1].getwidth() - if lo != hi: - raise error("look-behind requires fixed-width pattern") - emit(lo) # look behind - _compile(code, av[1], flags) - emit(SUCCESS) - code[skip] = _len(code) - skip - elif op is CALL: - emit(op) - skip = _len(code); emit(0) - _compile(code, av, flags) - emit(SUCCESS) - code[skip] = _len(code) - skip - elif op is AT: - emit(op) - if flags & SRE_FLAG_MULTILINE: - av = AT_MULTILINE.get(av, av) - if flags & SRE_FLAG_LOCALE: - av = AT_LOCALE.get(av, av) - elif flags & SRE_FLAG_UNICODE: - av = AT_UNICODE.get(av, av) - emit(av) - elif op is BRANCH: - emit(op) - tail = [] - tailappend = tail.append - for av in av[1]: - skip = _len(code); emit(0) - # _compile_info(code, av, flags) - _compile(code, av, flags) - emit(JUMP) - tailappend(_len(code)); emit(0) - code[skip] = _len(code) - skip - emit(FAILURE) # end of branch - for tail in tail: - code[tail] = _len(code) - tail - elif op is CATEGORY: - emit(op) - if flags & SRE_FLAG_LOCALE: - av = CH_LOCALE[av] - elif flags & SRE_FLAG_UNICODE: - av = CH_UNICODE[av] - emit(av) - elif op is GROUPREF: - if not flags & SRE_FLAG_IGNORECASE: - emit(op) - elif flags & SRE_FLAG_LOCALE: - emit(GROUPREF_LOC_IGNORE) - elif not fixes: # ascii - emit(GROUPREF_IGNORE) - else: - emit(GROUPREF_UNI_IGNORE) - emit(av-1) - elif op is GROUPREF_EXISTS: - emit(op) - emit(av[0]-1) - skipyes = _len(code); emit(0) - _compile(code, av[1], flags) - if av[2]: - emit(JUMP) - skipno = _len(code); emit(0) - code[skipyes] = _len(code) - skipyes + 1 - _compile(code, av[2], flags) - code[skipno] = _len(code) - skipno - else: - code[skipyes] = _len(code) - skipyes + 1 - else: - raise error("internal: unsupported operand type %r" % (op,)) - -def _compile_charset(charset, flags, code): - # compile charset subprogram - emit = code.append - for op, av in charset: - emit(op) - if op is NEGATE: - pass - elif op is LITERAL: - emit(av) - elif op is RANGE or op is RANGE_UNI_IGNORE: - emit(av[0]) - emit(av[1]) - elif op is CHARSET: - code.extend(av) - elif op is BIGCHARSET: - code.extend(av) - elif op is CATEGORY: - if flags & SRE_FLAG_LOCALE: - emit(CH_LOCALE[av]) - elif flags & SRE_FLAG_UNICODE: - emit(CH_UNICODE[av]) - else: - emit(av) - else: - raise error("internal: unsupported set operator %r" % (op,)) - emit(FAILURE) - -def _optimize_charset(charset, iscased=None, fixup=None, fixes=None): - # internal: optimize character set - out = [] - tail = [] - charmap = bytearray(256) - hascased = False - for op, av in charset: - while True: - try: - if op is LITERAL: - if fixup: - lo = fixup(av) - charmap[lo] = 1 - if fixes and lo in fixes: - for k in fixes[lo]: - charmap[k] = 1 - if not hascased and iscased(av): - hascased = True - else: - charmap[av] = 1 - elif op is RANGE: - r = range(av[0], av[1]+1) - if fixup: - if fixes: - for i in map(fixup, r): - charmap[i] = 1 - if i in fixes: - for k in fixes[i]: - charmap[k] = 1 - else: - for i in map(fixup, r): - charmap[i] = 1 - if not hascased: - hascased = any(map(iscased, r)) - else: - for i in r: - charmap[i] = 1 - elif op is NEGATE: - out.append((op, av)) - else: - tail.append((op, av)) - except IndexError: - if len(charmap) == 256: - # character set contains non-UCS1 character codes - charmap += b'\0' * 0xff00 - continue - # Character set contains non-BMP character codes. - if fixup: - hascased = True - # There are only two ranges of cased non-BMP characters: - # 10400-1044F (Deseret) and 118A0-118DF (Warang Citi), - # and for both ranges RANGE_UNI_IGNORE works. - if op is RANGE: - op = RANGE_UNI_IGNORE - tail.append((op, av)) - break - - # compress character map - runs = [] - q = 0 - while True: - p = charmap.find(1, q) - if p < 0: - break - if len(runs) >= 2: - runs = None - break - q = charmap.find(0, p) - if q < 0: - runs.append((p, len(charmap))) - break - runs.append((p, q)) - if runs is not None: - # use literal/range - for p, q in runs: - if q - p == 1: - out.append((LITERAL, p)) - else: - out.append((RANGE, (p, q - 1))) - out += tail - # if the case was changed or new representation is more compact - if hascased or len(out) < len(charset): - return out, hascased - # else original character set is good enough - return charset, hascased - - # use bitmap - if len(charmap) == 256: - data = _mk_bitmap(charmap) - out.append((CHARSET, data)) - out += tail - return out, hascased - - # To represent a big charset, first a bitmap of all characters in the - # set is constructed. Then, this bitmap is sliced into chunks of 256 - # characters, duplicate chunks are eliminated, and each chunk is - # given a number. In the compiled expression, the charset is - # represented by a 32-bit word sequence, consisting of one word for - # the number of different chunks, a sequence of 256 bytes (64 words) - # of chunk numbers indexed by their original chunk position, and a - # sequence of 256-bit chunks (8 words each). - - # Compression is normally good: in a typical charset, large ranges of - # Unicode will be either completely excluded (e.g. if only cyrillic - # letters are to be matched), or completely included (e.g. if large - # subranges of Kanji match). These ranges will be represented by - # chunks of all one-bits or all zero-bits. - - # Matching can be also done efficiently: the more significant byte of - # the Unicode character is an index into the chunk number, and the - # less significant byte is a bit index in the chunk (just like the - # CHARSET matching). - - charmap = bytes(charmap) # should be hashable - comps = {} - mapping = bytearray(256) - block = 0 - data = bytearray() - for i in range(0, 65536, 256): - chunk = charmap[i: i + 256] - if chunk in comps: - mapping[i // 256] = comps[chunk] - else: - mapping[i // 256] = comps[chunk] = block - block += 1 - data += chunk - data = _mk_bitmap(data) - data[0:0] = [block] + _bytes_to_codes(mapping) - out.append((BIGCHARSET, data)) - out += tail - return out, hascased - -_CODEBITS = _sre.CODESIZE * 8 -MAXCODE = (1 << _CODEBITS) - 1 -_BITS_TRANS = b'0' + b'1' * 255 -def _mk_bitmap(bits, _CODEBITS=_CODEBITS, _int=int): - s = bits.translate(_BITS_TRANS)[::-1] - return [_int(s[i - _CODEBITS: i], 2) - for i in range(len(s), 0, -_CODEBITS)] - -def _bytes_to_codes(b): - # Convert block indices to word array - a = memoryview(b).cast('I') - assert a.itemsize == _sre.CODESIZE - assert len(a) * a.itemsize == len(b) - return a.tolist() - -def _simple(p): - # check if this subpattern is a "simple" operator - if len(p) != 1: - return False - op, av = p[0] - if op is SUBPATTERN: - return av[0] is None and _simple(av[-1]) - return op in _UNIT_CODES - -def _generate_overlap_table(prefix): - """ - Generate an overlap table for the following prefix. - An overlap table is a table of the same size as the prefix which - informs about the potential self-overlap for each index in the prefix: - - if overlap[i] == 0, prefix[i:] can't overlap prefix[0:...] - - if overlap[i] == k with 0 < k <= i, prefix[i-k+1:i+1] overlaps with - prefix[0:k] - """ - table = [0] * len(prefix) - for i in range(1, len(prefix)): - idx = table[i - 1] - while prefix[i] != prefix[idx]: - if idx == 0: - table[i] = 0 - break - idx = table[idx - 1] - else: - table[i] = idx + 1 - return table - -def _get_iscased(flags): - if not flags & SRE_FLAG_IGNORECASE: - return None - elif flags & SRE_FLAG_UNICODE: - return _sre.unicode_iscased - else: - return _sre.ascii_iscased - -def _get_literal_prefix(pattern, flags): - # look for literal prefix - prefix = [] - prefixappend = prefix.append - prefix_skip = None - iscased = _get_iscased(flags) - for op, av in pattern.data: - if op is LITERAL: - if iscased and iscased(av): - break - prefixappend(av) - elif op is SUBPATTERN: - group, add_flags, del_flags, p = av - flags1 = _combine_flags(flags, add_flags, del_flags) - if flags1 & SRE_FLAG_IGNORECASE and flags1 & SRE_FLAG_LOCALE: - break - prefix1, prefix_skip1, got_all = _get_literal_prefix(p, flags1) - if prefix_skip is None: - if group is not None: - prefix_skip = len(prefix) - elif prefix_skip1 is not None: - prefix_skip = len(prefix) + prefix_skip1 - prefix.extend(prefix1) - if not got_all: - break - else: - break - else: - return prefix, prefix_skip, True - return prefix, prefix_skip, False - -def _get_charset_prefix(pattern, flags): - while True: - if not pattern.data: - return None - op, av = pattern.data[0] - if op is not SUBPATTERN: - break - group, add_flags, del_flags, pattern = av - flags = _combine_flags(flags, add_flags, del_flags) - if flags & SRE_FLAG_IGNORECASE and flags & SRE_FLAG_LOCALE: - return None - - iscased = _get_iscased(flags) - if op is LITERAL: - if iscased and iscased(av): - return None - return [(op, av)] - elif op is BRANCH: - charset = [] - charsetappend = charset.append - for p in av[1]: - if not p: - return None - op, av = p[0] - if op is LITERAL and not (iscased and iscased(av)): - charsetappend((op, av)) - else: - return None - return charset - elif op is IN: - charset = av - if iscased: - for op, av in charset: - if op is LITERAL: - if iscased(av): - return None - elif op is RANGE: - if av[1] > 0xffff: - return None - if any(map(iscased, range(av[0], av[1]+1))): - return None - return charset - return None - -def _compile_info(code, pattern, flags): - # internal: compile an info block. in the current version, - # this contains min/max pattern width, and an optional literal - # prefix or a character map - lo, hi = pattern.getwidth() - if hi > MAXCODE: - hi = MAXCODE - if lo == 0: - code.extend([INFO, 4, 0, lo, hi]) - return - # look for a literal prefix - prefix = [] - prefix_skip = 0 - charset = [] # not used - if not (flags & SRE_FLAG_IGNORECASE and flags & SRE_FLAG_LOCALE): - # look for literal prefix - prefix, prefix_skip, got_all = _get_literal_prefix(pattern, flags) - # if no prefix, look for charset prefix - if not prefix: - charset = _get_charset_prefix(pattern, flags) -## if prefix: -## print("*** PREFIX", prefix, prefix_skip) -## if charset: -## print("*** CHARSET", charset) - # add an info block - emit = code.append - emit(INFO) - skip = len(code); emit(0) - # literal flag - mask = 0 - if prefix: - mask = SRE_INFO_PREFIX - if prefix_skip is None and got_all: - mask = mask | SRE_INFO_LITERAL - elif charset: - mask = mask | SRE_INFO_CHARSET - emit(mask) - # pattern length - if lo < MAXCODE: - emit(lo) - else: - emit(MAXCODE) - prefix = prefix[:MAXCODE] - emit(min(hi, MAXCODE)) - # add literal prefix - if prefix: - emit(len(prefix)) # length - if prefix_skip is None: - prefix_skip = len(prefix) - emit(prefix_skip) # skip - code.extend(prefix) - # generate overlap table - code.extend(_generate_overlap_table(prefix)) - elif charset: - charset, hascased = _optimize_charset(charset) - assert not hascased - _compile_charset(charset, flags, code) - code[skip] = len(code) - skip - -def isstring(obj): - return isinstance(obj, (str, bytes)) - -def _code(p, flags): - - flags = p.state.flags | flags - code = [] - - # compile info block - _compile_info(code, p, flags) - - # compile the pattern - _compile(code, p.data, flags) - - code.append(SUCCESS) - - return code - -def _hex_code(code): - return '[%s]' % ', '.join('%#0*x' % (_sre.CODESIZE*2+2, x) for x in code) - -def dis(code): - import sys - - labels = set() - level = 0 - offset_width = len(str(len(code) - 1)) - - def dis_(start, end): - def print_(*args, to=None): - if to is not None: - labels.add(to) - args += ('(to %d)' % (to,),) - print('%*d%s ' % (offset_width, start, ':' if start in labels else '.'), - end=' '*(level-1)) - print(*args) - - def print_2(*args): - print(end=' '*(offset_width + 2*level)) - print(*args) - - nonlocal level - level += 1 - i = start - while i < end: - start = i - op = code[i] - i += 1 - op = OPCODES[op] - if op in (SUCCESS, FAILURE, ANY, ANY_ALL, - MAX_UNTIL, MIN_UNTIL, NEGATE): - print_(op) - elif op in (LITERAL, NOT_LITERAL, - LITERAL_IGNORE, NOT_LITERAL_IGNORE, - LITERAL_UNI_IGNORE, NOT_LITERAL_UNI_IGNORE, - LITERAL_LOC_IGNORE, NOT_LITERAL_LOC_IGNORE): - arg = code[i] - i += 1 - print_(op, '%#02x (%r)' % (arg, chr(arg))) - elif op is AT: - arg = code[i] - i += 1 - arg = str(ATCODES[arg]) - assert arg[:3] == 'AT_' - print_(op, arg[3:]) - elif op is CATEGORY: - arg = code[i] - i += 1 - arg = str(CHCODES[arg]) - assert arg[:9] == 'CATEGORY_' - print_(op, arg[9:]) - elif op in (IN, IN_IGNORE, IN_UNI_IGNORE, IN_LOC_IGNORE): - skip = code[i] - print_(op, skip, to=i+skip) - dis_(i+1, i+skip) - i += skip - elif op in (RANGE, RANGE_UNI_IGNORE): - lo, hi = code[i: i+2] - i += 2 - print_(op, '%#02x %#02x (%r-%r)' % (lo, hi, chr(lo), chr(hi))) - elif op is CHARSET: - print_(op, _hex_code(code[i: i + 256//_CODEBITS])) - i += 256//_CODEBITS - elif op is BIGCHARSET: - arg = code[i] - i += 1 - mapping = list(b''.join(x.to_bytes(_sre.CODESIZE, sys.byteorder) - for x in code[i: i + 256//_sre.CODESIZE])) - print_(op, arg, mapping) - i += 256//_sre.CODESIZE - level += 1 - for j in range(arg): - print_2(_hex_code(code[i: i + 256//_CODEBITS])) - i += 256//_CODEBITS - level -= 1 - elif op in (MARK, GROUPREF, GROUPREF_IGNORE, GROUPREF_UNI_IGNORE, - GROUPREF_LOC_IGNORE): - arg = code[i] - i += 1 - print_(op, arg) - elif op is JUMP: - skip = code[i] - print_(op, skip, to=i+skip) - i += 1 - elif op is BRANCH: - skip = code[i] - print_(op, skip, to=i+skip) - while skip: - dis_(i+1, i+skip) - i += skip - start = i - skip = code[i] - if skip: - print_('branch', skip, to=i+skip) - else: - print_(FAILURE) - i += 1 - elif op in (REPEAT, REPEAT_ONE, MIN_REPEAT_ONE, - POSSESSIVE_REPEAT, POSSESSIVE_REPEAT_ONE): - skip, min, max = code[i: i+3] - if max == MAXREPEAT: - max = 'MAXREPEAT' - print_(op, skip, min, max, to=i+skip) - dis_(i+3, i+skip) - i += skip - elif op is GROUPREF_EXISTS: - arg, skip = code[i: i+2] - print_(op, arg, skip, to=i+skip) - i += 2 - elif op in (ASSERT, ASSERT_NOT): - skip, arg = code[i: i+2] - print_(op, skip, arg, to=i+skip) - dis_(i+2, i+skip) - i += skip - elif op is ATOMIC_GROUP: - skip = code[i] - print_(op, skip, to=i+skip) - dis_(i+1, i+skip) - i += skip - elif op is INFO: - skip, flags, min, max = code[i: i+4] - if max == MAXREPEAT: - max = 'MAXREPEAT' - print_(op, skip, bin(flags), min, max, to=i+skip) - start = i+4 - if flags & SRE_INFO_PREFIX: - prefix_len, prefix_skip = code[i+4: i+6] - print_2(' prefix_skip', prefix_skip) - start = i + 6 - prefix = code[start: start+prefix_len] - print_2(' prefix', - '[%s]' % ', '.join('%#02x' % x for x in prefix), - '(%r)' % ''.join(map(chr, prefix))) - start += prefix_len - print_2(' overlap', code[start: start+prefix_len]) - start += prefix_len - if flags & SRE_INFO_CHARSET: - level += 1 - print_2('in') - dis_(start, i+skip) - level -= 1 - i += skip - else: - raise ValueError(op) - - level -= 1 - - dis_(0, len(code)) - - -def compile(p, flags=0): - # internal: convert pattern list to internal format - - if isstring(p): - pattern = p - p = sre_parse.parse(p, flags) - else: - pattern = None - - code = _code(p, flags) - - if flags & SRE_FLAG_DEBUG: - print() - dis(code) - - # map in either direction - groupindex = p.state.groupdict - indexgroup = [None] * p.state.groups - for k, i in groupindex.items(): - indexgroup[i] = k - - return _sre.compile( - pattern, flags | p.state.flags, code, - p.state.groups-1, - groupindex, tuple(indexgroup) - ) +from re import _compiler as _ +globals().update({k: v for k, v in vars(_).items() if k[:2] != '__'}) diff --git a/Lib/sre_constants.py b/Lib/sre_constants.py index a00b0170607b5..fa09d04429296 100644 --- a/Lib/sre_constants.py +++ b/Lib/sre_constants.py @@ -1,262 +1,7 @@ -# -# Secret Labs' Regular Expression Engine -# -# various symbols used by the regular expression engine. -# run this script to update the _sre include files! -# -# Copyright (c) 1998-2001 by Secret Labs AB. All rights reserved. -# -# See the sre.py file for information on usage and redistribution. -# +import warnings +warnings.warn(f"module {__name__!r} is deprecated", + DeprecationWarning, + stacklevel=2) -"""Internal support module for sre""" - -# update when constants are added or removed - -MAGIC = 20220318 - -from _sre import MAXREPEAT, MAXGROUPS - -# SRE standard exception (access as sre.error) -# should this really be here? - -class error(Exception): - """Exception raised for invalid regular expressions. - - Attributes: - - msg: The unformatted error message - pattern: The regular expression pattern - pos: The index in the pattern where compilation failed (may be None) - lineno: The line corresponding to pos (may be None) - colno: The column corresponding to pos (may be None) - """ - - __module__ = 're' - - def __init__(self, msg, pattern=None, pos=None): - self.msg = msg - self.pattern = pattern - self.pos = pos - if pattern is not None and pos is not None: - msg = '%s at position %d' % (msg, pos) - if isinstance(pattern, str): - newline = '\n' - else: - newline = b'\n' - self.lineno = pattern.count(newline, 0, pos) + 1 - self.colno = pos - pattern.rfind(newline, 0, pos) - if newline in pattern: - msg = '%s (line %d, column %d)' % (msg, self.lineno, self.colno) - else: - self.lineno = self.colno = None - super().__init__(msg) - - -class _NamedIntConstant(int): - def __new__(cls, value, name): - self = super(_NamedIntConstant, cls).__new__(cls, value) - self.name = name - return self - - def __repr__(self): - return self.name - -MAXREPEAT = _NamedIntConstant(MAXREPEAT, 'MAXREPEAT') - -def _makecodes(names): - names = names.strip().split() - items = [_NamedIntConstant(i, name) for i, name in enumerate(names)] - globals().update({item.name: item for item in items}) - return items - -# operators -# failure=0 success=1 (just because it looks better that way :-) -OPCODES = _makecodes(""" - FAILURE SUCCESS - - ANY ANY_ALL - ASSERT ASSERT_NOT - AT - BRANCH - CALL - CATEGORY - CHARSET BIGCHARSET - GROUPREF GROUPREF_EXISTS - IN - INFO - JUMP - LITERAL - MARK - MAX_UNTIL - MIN_UNTIL - NOT_LITERAL - NEGATE - RANGE - REPEAT - REPEAT_ONE - SUBPATTERN - MIN_REPEAT_ONE - ATOMIC_GROUP - POSSESSIVE_REPEAT - POSSESSIVE_REPEAT_ONE - - GROUPREF_IGNORE - IN_IGNORE - LITERAL_IGNORE - NOT_LITERAL_IGNORE - - GROUPREF_LOC_IGNORE - IN_LOC_IGNORE - LITERAL_LOC_IGNORE - NOT_LITERAL_LOC_IGNORE - - GROUPREF_UNI_IGNORE - IN_UNI_IGNORE - LITERAL_UNI_IGNORE - NOT_LITERAL_UNI_IGNORE - RANGE_UNI_IGNORE - - MIN_REPEAT MAX_REPEAT -""") -del OPCODES[-2:] # remove MIN_REPEAT and MAX_REPEAT - -# positions -ATCODES = _makecodes(""" - AT_BEGINNING AT_BEGINNING_LINE AT_BEGINNING_STRING - AT_BOUNDARY AT_NON_BOUNDARY - AT_END AT_END_LINE AT_END_STRING - - AT_LOC_BOUNDARY AT_LOC_NON_BOUNDARY - - AT_UNI_BOUNDARY AT_UNI_NON_BOUNDARY -""") - -# categories -CHCODES = _makecodes(""" - CATEGORY_DIGIT CATEGORY_NOT_DIGIT - CATEGORY_SPACE CATEGORY_NOT_SPACE - CATEGORY_WORD CATEGORY_NOT_WORD - CATEGORY_LINEBREAK CATEGORY_NOT_LINEBREAK - - CATEGORY_LOC_WORD CATEGORY_LOC_NOT_WORD - - CATEGORY_UNI_DIGIT CATEGORY_UNI_NOT_DIGIT - CATEGORY_UNI_SPACE CATEGORY_UNI_NOT_SPACE - CATEGORY_UNI_WORD CATEGORY_UNI_NOT_WORD - CATEGORY_UNI_LINEBREAK CATEGORY_UNI_NOT_LINEBREAK -""") - - -# replacement operations for "ignore case" mode -OP_IGNORE = { - LITERAL: LITERAL_IGNORE, - NOT_LITERAL: NOT_LITERAL_IGNORE, -} - -OP_LOCALE_IGNORE = { - LITERAL: LITERAL_LOC_IGNORE, - NOT_LITERAL: NOT_LITERAL_LOC_IGNORE, -} - -OP_UNICODE_IGNORE = { - LITERAL: LITERAL_UNI_IGNORE, - NOT_LITERAL: NOT_LITERAL_UNI_IGNORE, -} - -AT_MULTILINE = { - AT_BEGINNING: AT_BEGINNING_LINE, - AT_END: AT_END_LINE -} - -AT_LOCALE = { - AT_BOUNDARY: AT_LOC_BOUNDARY, - AT_NON_BOUNDARY: AT_LOC_NON_BOUNDARY -} - -AT_UNICODE = { - AT_BOUNDARY: AT_UNI_BOUNDARY, - AT_NON_BOUNDARY: AT_UNI_NON_BOUNDARY -} - -CH_LOCALE = { - CATEGORY_DIGIT: CATEGORY_DIGIT, - CATEGORY_NOT_DIGIT: CATEGORY_NOT_DIGIT, - CATEGORY_SPACE: CATEGORY_SPACE, - CATEGORY_NOT_SPACE: CATEGORY_NOT_SPACE, - CATEGORY_WORD: CATEGORY_LOC_WORD, - CATEGORY_NOT_WORD: CATEGORY_LOC_NOT_WORD, - CATEGORY_LINEBREAK: CATEGORY_LINEBREAK, - CATEGORY_NOT_LINEBREAK: CATEGORY_NOT_LINEBREAK -} - -CH_UNICODE = { - CATEGORY_DIGIT: CATEGORY_UNI_DIGIT, - CATEGORY_NOT_DIGIT: CATEGORY_UNI_NOT_DIGIT, - CATEGORY_SPACE: CATEGORY_UNI_SPACE, - CATEGORY_NOT_SPACE: CATEGORY_UNI_NOT_SPACE, - CATEGORY_WORD: CATEGORY_UNI_WORD, - CATEGORY_NOT_WORD: CATEGORY_UNI_NOT_WORD, - CATEGORY_LINEBREAK: CATEGORY_UNI_LINEBREAK, - CATEGORY_NOT_LINEBREAK: CATEGORY_UNI_NOT_LINEBREAK -} - -# flags -SRE_FLAG_TEMPLATE = 1 # template mode (disable backtracking) -SRE_FLAG_IGNORECASE = 2 # case insensitive -SRE_FLAG_LOCALE = 4 # honour system locale -SRE_FLAG_MULTILINE = 8 # treat target as multiline string -SRE_FLAG_DOTALL = 16 # treat target as a single string -SRE_FLAG_UNICODE = 32 # use unicode "locale" -SRE_FLAG_VERBOSE = 64 # ignore whitespace and comments -SRE_FLAG_DEBUG = 128 # debugging -SRE_FLAG_ASCII = 256 # use ascii "locale" - -# flags for INFO primitive -SRE_INFO_PREFIX = 1 # has prefix -SRE_INFO_LITERAL = 2 # entire pattern is literal (given by prefix) -SRE_INFO_CHARSET = 4 # pattern starts with character from given set - -if __name__ == "__main__": - def dump(f, d, prefix): - items = sorted(d) - for item in items: - f.write("#define %s_%s %d\n" % (prefix, item, item)) - with open("sre_constants.h", "w") as f: - f.write("""\ -/* - * Secret Labs' Regular Expression Engine - * - * regular expression matching engine - * - * NOTE: This file is generated by sre_constants.py. If you need - * to change anything in here, edit sre_constants.py and run it. - * - * Copyright (c) 1997-2001 by Secret Labs AB. All rights reserved. - * - * See the _sre.c file for information on usage and redistribution. - */ - -""") - - f.write("#define SRE_MAGIC %d\n" % MAGIC) - - dump(f, OPCODES, "SRE_OP") - dump(f, ATCODES, "SRE") - dump(f, CHCODES, "SRE") - - f.write("#define SRE_FLAG_TEMPLATE %d\n" % SRE_FLAG_TEMPLATE) - f.write("#define SRE_FLAG_IGNORECASE %d\n" % SRE_FLAG_IGNORECASE) - f.write("#define SRE_FLAG_LOCALE %d\n" % SRE_FLAG_LOCALE) - f.write("#define SRE_FLAG_MULTILINE %d\n" % SRE_FLAG_MULTILINE) - f.write("#define SRE_FLAG_DOTALL %d\n" % SRE_FLAG_DOTALL) - f.write("#define SRE_FLAG_UNICODE %d\n" % SRE_FLAG_UNICODE) - f.write("#define SRE_FLAG_VERBOSE %d\n" % SRE_FLAG_VERBOSE) - f.write("#define SRE_FLAG_DEBUG %d\n" % SRE_FLAG_DEBUG) - f.write("#define SRE_FLAG_ASCII %d\n" % SRE_FLAG_ASCII) - - f.write("#define SRE_INFO_PREFIX %d\n" % SRE_INFO_PREFIX) - f.write("#define SRE_INFO_LITERAL %d\n" % SRE_INFO_LITERAL) - f.write("#define SRE_INFO_CHARSET %d\n" % SRE_INFO_CHARSET) - - print("done") +from re import _constants as _ +globals().update({k: v for k, v in vars(_).items() if k[:2] != '__'}) diff --git a/Lib/sre_parse.py b/Lib/sre_parse.py index b91082e48fafe..25a3f557d44c6 100644 --- a/Lib/sre_parse.py +++ b/Lib/sre_parse.py @@ -1,1079 +1,7 @@ -# -# Secret Labs' Regular Expression Engine -# -# convert re-style regular expression to sre pattern -# -# Copyright (c) 1998-2001 by Secret Labs AB. All rights reserved. -# -# See the sre.py file for information on usage and redistribution. -# +import warnings +warnings.warn(f"module {__name__!r} is deprecated", + DeprecationWarning, + stacklevel=2) -"""Internal support module for sre""" - -# XXX: show string offset and offending character for all errors - -from sre_constants import * - -SPECIAL_CHARS = ".\\[{()*+?^$|" -REPEAT_CHARS = "*+?{" - -DIGITS = frozenset("0123456789") - -OCTDIGITS = frozenset("01234567") -HEXDIGITS = frozenset("0123456789abcdefABCDEF") -ASCIILETTERS = frozenset("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ") - -WHITESPACE = frozenset(" \t\n\r\v\f") - -_REPEATCODES = frozenset({MIN_REPEAT, MAX_REPEAT, POSSESSIVE_REPEAT}) -_UNITCODES = frozenset({ANY, RANGE, IN, LITERAL, NOT_LITERAL, CATEGORY}) - -ESCAPES = { - r"\a": (LITERAL, ord("\a")), - r"\b": (LITERAL, ord("\b")), - r"\f": (LITERAL, ord("\f")), - r"\n": (LITERAL, ord("\n")), - r"\r": (LITERAL, ord("\r")), - r"\t": (LITERAL, ord("\t")), - r"\v": (LITERAL, ord("\v")), - r"\\": (LITERAL, ord("\\")) -} - -CATEGORIES = { - r"\A": (AT, AT_BEGINNING_STRING), # start of string - r"\b": (AT, AT_BOUNDARY), - r"\B": (AT, AT_NON_BOUNDARY), - r"\d": (IN, [(CATEGORY, CATEGORY_DIGIT)]), - r"\D": (IN, [(CATEGORY, CATEGORY_NOT_DIGIT)]), - r"\s": (IN, [(CATEGORY, CATEGORY_SPACE)]), - r"\S": (IN, [(CATEGORY, CATEGORY_NOT_SPACE)]), - r"\w": (IN, [(CATEGORY, CATEGORY_WORD)]), - r"\W": (IN, [(CATEGORY, CATEGORY_NOT_WORD)]), - r"\Z": (AT, AT_END_STRING), # end of string -} - -FLAGS = { - # standard flags - "i": SRE_FLAG_IGNORECASE, - "L": SRE_FLAG_LOCALE, - "m": SRE_FLAG_MULTILINE, - "s": SRE_FLAG_DOTALL, - "x": SRE_FLAG_VERBOSE, - # extensions - "a": SRE_FLAG_ASCII, - "t": SRE_FLAG_TEMPLATE, - "u": SRE_FLAG_UNICODE, -} - -TYPE_FLAGS = SRE_FLAG_ASCII | SRE_FLAG_LOCALE | SRE_FLAG_UNICODE -GLOBAL_FLAGS = SRE_FLAG_DEBUG | SRE_FLAG_TEMPLATE - -class Verbose(Exception): - pass - -class State: - # keeps track of state for parsing - def __init__(self): - self.flags = 0 - self.groupdict = {} - self.groupwidths = [None] # group 0 - self.lookbehindgroups = None - @property - def groups(self): - return len(self.groupwidths) - def opengroup(self, name=None): - gid = self.groups - self.groupwidths.append(None) - if self.groups > MAXGROUPS: - raise error("too many groups") - if name is not None: - ogid = self.groupdict.get(name, None) - if ogid is not None: - raise error("redefinition of group name %r as group %d; " - "was group %d" % (name, gid, ogid)) - self.groupdict[name] = gid - return gid - def closegroup(self, gid, p): - self.groupwidths[gid] = p.getwidth() - def checkgroup(self, gid): - return gid < self.groups and self.groupwidths[gid] is not None - - def checklookbehindgroup(self, gid, source): - if self.lookbehindgroups is not None: - if not self.checkgroup(gid): - raise source.error('cannot refer to an open group') - if gid >= self.lookbehindgroups: - raise source.error('cannot refer to group defined in the same ' - 'lookbehind subpattern') - -class SubPattern: - # a subpattern, in intermediate form - def __init__(self, state, data=None): - self.state = state - if data is None: - data = [] - self.data = data - self.width = None - - def dump(self, level=0): - nl = True - seqtypes = (tuple, list) - for op, av in self.data: - print(level*" " + str(op), end='') - if op is IN: - # member sublanguage - print() - for op, a in av: - print((level+1)*" " + str(op), a) - elif op is BRANCH: - print() - for i, a in enumerate(av[1]): - if i: - print(level*" " + "OR") - a.dump(level+1) - elif op is GROUPREF_EXISTS: - condgroup, item_yes, item_no = av - print('', condgroup) - item_yes.dump(level+1) - if item_no: - print(level*" " + "ELSE") - item_no.dump(level+1) - elif isinstance(av, seqtypes): - nl = False - for a in av: - if isinstance(a, SubPattern): - if not nl: - print() - a.dump(level+1) - nl = True - else: - if not nl: - print(' ', end='') - print(a, end='') - nl = False - if not nl: - print() - else: - print('', av) - def __repr__(self): - return repr(self.data) - def __len__(self): - return len(self.data) - def __delitem__(self, index): - del self.data[index] - def __getitem__(self, index): - if isinstance(index, slice): - return SubPattern(self.state, self.data[index]) - return self.data[index] - def __setitem__(self, index, code): - self.data[index] = code - def insert(self, index, code): - self.data.insert(index, code) - def append(self, code): - self.data.append(code) - def getwidth(self): - # determine the width (min, max) for this subpattern - if self.width is not None: - return self.width - lo = hi = 0 - for op, av in self.data: - if op is BRANCH: - i = MAXREPEAT - 1 - j = 0 - for av in av[1]: - l, h = av.getwidth() - i = min(i, l) - j = max(j, h) - lo = lo + i - hi = hi + j - elif op is CALL: - i, j = av.getwidth() - lo = lo + i - hi = hi + j - elif op is ATOMIC_GROUP: - i, j = av.getwidth() - lo = lo + i - hi = hi + j - elif op is SUBPATTERN: - i, j = av[-1].getwidth() - lo = lo + i - hi = hi + j - elif op in _REPEATCODES: - i, j = av[2].getwidth() - lo = lo + i * av[0] - hi = hi + j * av[1] - elif op in _UNITCODES: - lo = lo + 1 - hi = hi + 1 - elif op is GROUPREF: - i, j = self.state.groupwidths[av] - lo = lo + i - hi = hi + j - elif op is GROUPREF_EXISTS: - i, j = av[1].getwidth() - if av[2] is not None: - l, h = av[2].getwidth() - i = min(i, l) - j = max(j, h) - else: - i = 0 - lo = lo + i - hi = hi + j - elif op is SUCCESS: - break - self.width = min(lo, MAXREPEAT - 1), min(hi, MAXREPEAT) - return self.width - -class Tokenizer: - def __init__(self, string): - self.istext = isinstance(string, str) - self.string = string - if not self.istext: - string = str(string, 'latin1') - self.decoded_string = string - self.index = 0 - self.next = None - self.__next() - def __next(self): - index = self.index - try: - char = self.decoded_string[index] - except IndexError: - self.next = None - return - if char == "\\": - index += 1 - try: - char += self.decoded_string[index] - except IndexError: - raise error("bad escape (end of pattern)", - self.string, len(self.string) - 1) from None - self.index = index + 1 - self.next = char - def match(self, char): - if char == self.next: - self.__next() - return True - return False - def get(self): - this = self.next - self.__next() - return this - def getwhile(self, n, charset): - result = '' - for _ in range(n): - c = self.next - if c not in charset: - break - result += c - self.__next() - return result - def getuntil(self, terminator, name): - result = '' - while True: - c = self.next - self.__next() - if c is None: - if not result: - raise self.error("missing " + name) - raise self.error("missing %s, unterminated name" % terminator, - len(result)) - if c == terminator: - if not result: - raise self.error("missing " + name, 1) - break - result += c - return result - @property - def pos(self): - return self.index - len(self.next or '') - def tell(self): - return self.index - len(self.next or '') - def seek(self, index): - self.index = index - self.__next() - - def error(self, msg, offset=0): - return error(msg, self.string, self.tell() - offset) - -def _class_escape(source, escape): - # handle escape code inside character class - code = ESCAPES.get(escape) - if code: - return code - code = CATEGORIES.get(escape) - if code and code[0] is IN: - return code - try: - c = escape[1:2] - if c == "x": - # hexadecimal escape (exactly two digits) - escape += source.getwhile(2, HEXDIGITS) - if len(escape) != 4: - raise source.error("incomplete escape %s" % escape, len(escape)) - return LITERAL, int(escape[2:], 16) - elif c == "u" and source.istext: - # unicode escape (exactly four digits) - escape += source.getwhile(4, HEXDIGITS) - if len(escape) != 6: - raise source.error("incomplete escape %s" % escape, len(escape)) - return LITERAL, int(escape[2:], 16) - elif c == "U" and source.istext: - # unicode escape (exactly eight digits) - escape += source.getwhile(8, HEXDIGITS) - if len(escape) != 10: - raise source.error("incomplete escape %s" % escape, len(escape)) - c = int(escape[2:], 16) - chr(c) # raise ValueError for invalid code - return LITERAL, c - elif c == "N" and source.istext: - import unicodedata - # named unicode escape e.g. \N{EM DASH} - if not source.match('{'): - raise source.error("missing {") - charname = source.getuntil('}', 'character name') - try: - c = ord(unicodedata.lookup(charname)) - except KeyError: - raise source.error("undefined character name %r" % charname, - len(charname) + len(r'\N{}')) - return LITERAL, c - elif c in OCTDIGITS: - # octal escape (up to three digits) - escape += source.getwhile(2, OCTDIGITS) - c = int(escape[1:], 8) - if c > 0o377: - raise source.error('octal escape value %s outside of ' - 'range 0-0o377' % escape, len(escape)) - return LITERAL, c - elif c in DIGITS: - raise ValueError - if len(escape) == 2: - if c in ASCIILETTERS: - raise source.error('bad escape %s' % escape, len(escape)) - return LITERAL, ord(escape[1]) - except ValueError: - pass - raise source.error("bad escape %s" % escape, len(escape)) - -def _escape(source, escape, state): - # handle escape code in expression - code = CATEGORIES.get(escape) - if code: - return code - code = ESCAPES.get(escape) - if code: - return code - try: - c = escape[1:2] - if c == "x": - # hexadecimal escape - escape += source.getwhile(2, HEXDIGITS) - if len(escape) != 4: - raise source.error("incomplete escape %s" % escape, len(escape)) - return LITERAL, int(escape[2:], 16) - elif c == "u" and source.istext: - # unicode escape (exactly four digits) - escape += source.getwhile(4, HEXDIGITS) - if len(escape) != 6: - raise source.error("incomplete escape %s" % escape, len(escape)) - return LITERAL, int(escape[2:], 16) - elif c == "U" and source.istext: - # unicode escape (exactly eight digits) - escape += source.getwhile(8, HEXDIGITS) - if len(escape) != 10: - raise source.error("incomplete escape %s" % escape, len(escape)) - c = int(escape[2:], 16) - chr(c) # raise ValueError for invalid code - return LITERAL, c - elif c == "N" and source.istext: - import unicodedata - # named unicode escape e.g. \N{EM DASH} - if not source.match('{'): - raise source.error("missing {") - charname = source.getuntil('}', 'character name') - try: - c = ord(unicodedata.lookup(charname)) - except KeyError: - raise source.error("undefined character name %r" % charname, - len(charname) + len(r'\N{}')) - return LITERAL, c - elif c == "0": - # octal escape - escape += source.getwhile(2, OCTDIGITS) - return LITERAL, int(escape[1:], 8) - elif c in DIGITS: - # octal escape *or* decimal group reference (sigh) - if source.next in DIGITS: - escape += source.get() - if (escape[1] in OCTDIGITS and escape[2] in OCTDIGITS and - source.next in OCTDIGITS): - # got three octal digits; this is an octal escape - escape += source.get() - c = int(escape[1:], 8) - if c > 0o377: - raise source.error('octal escape value %s outside of ' - 'range 0-0o377' % escape, - len(escape)) - return LITERAL, c - # not an octal escape, so this is a group reference - group = int(escape[1:]) - if group < state.groups: - if not state.checkgroup(group): - raise source.error("cannot refer to an open group", - len(escape)) - state.checklookbehindgroup(group, source) - return GROUPREF, group - raise source.error("invalid group reference %d" % group, len(escape) - 1) - if len(escape) == 2: - if c in ASCIILETTERS: - raise source.error("bad escape %s" % escape, len(escape)) - return LITERAL, ord(escape[1]) - except ValueError: - pass - raise source.error("bad escape %s" % escape, len(escape)) - -def _uniq(items): - return list(dict.fromkeys(items)) - -def _parse_sub(source, state, verbose, nested): - # parse an alternation: a|b|c - - items = [] - itemsappend = items.append - sourcematch = source.match - start = source.tell() - while True: - itemsappend(_parse(source, state, verbose, nested + 1, - not nested and not items)) - if not sourcematch("|"): - break - - if len(items) == 1: - return items[0] - - subpattern = SubPattern(state) - - # check if all items share a common prefix - while True: - prefix = None - for item in items: - if not item: - break - if prefix is None: - prefix = item[0] - elif item[0] != prefix: - break - else: - # all subitems start with a common "prefix". - # move it out of the branch - for item in items: - del item[0] - subpattern.append(prefix) - continue # check next one - break - - # check if the branch can be replaced by a character set - set = [] - for item in items: - if len(item) != 1: - break - op, av = item[0] - if op is LITERAL: - set.append((op, av)) - elif op is IN and av[0][0] is not NEGATE: - set.extend(av) - else: - break - else: - # we can store this as a character set instead of a - # branch (the compiler may optimize this even more) - subpattern.append((IN, _uniq(set))) - return subpattern - - subpattern.append((BRANCH, (None, items))) - return subpattern - -def _parse(source, state, verbose, nested, first=False): - # parse a simple pattern - subpattern = SubPattern(state) - - # precompute constants into local variables - subpatternappend = subpattern.append - sourceget = source.get - sourcematch = source.match - _len = len - _ord = ord - - while True: - - this = source.next - if this is None: - break # end of pattern - if this in "|)": - break # end of subpattern - sourceget() - - if verbose: - # skip whitespace and comments - if this in WHITESPACE: - continue - if this == "#": - while True: - this = sourceget() - if this is None or this == "\n": - break - continue - - if this[0] == "\\": - code = _escape(source, this, state) - subpatternappend(code) - - elif this not in SPECIAL_CHARS: - subpatternappend((LITERAL, _ord(this))) - - elif this == "[": - here = source.tell() - 1 - # character set - set = [] - setappend = set.append -## if sourcematch(":"): -## pass # handle character classes - if source.next == '[': - import warnings - warnings.warn( - 'Possible nested set at position %d' % source.tell(), - FutureWarning, stacklevel=nested + 6 - ) - negate = sourcematch("^") - # check remaining characters - while True: - this = sourceget() - if this is None: - raise source.error("unterminated character set", - source.tell() - here) - if this == "]" and set: - break - elif this[0] == "\\": - code1 = _class_escape(source, this) - else: - if set and this in '-&~|' and source.next == this: - import warnings - warnings.warn( - 'Possible set %s at position %d' % ( - 'difference' if this == '-' else - 'intersection' if this == '&' else - 'symmetric difference' if this == '~' else - 'union', - source.tell() - 1), - FutureWarning, stacklevel=nested + 6 - ) - code1 = LITERAL, _ord(this) - if sourcematch("-"): - # potential range - that = sourceget() - if that is None: - raise source.error("unterminated character set", - source.tell() - here) - if that == "]": - if code1[0] is IN: - code1 = code1[1][0] - setappend(code1) - setappend((LITERAL, _ord("-"))) - break - if that[0] == "\\": - code2 = _class_escape(source, that) - else: - if that == '-': - import warnings - warnings.warn( - 'Possible set difference at position %d' % ( - source.tell() - 2), - FutureWarning, stacklevel=nested + 6 - ) - code2 = LITERAL, _ord(that) - if code1[0] != LITERAL or code2[0] != LITERAL: - msg = "bad character range %s-%s" % (this, that) - raise source.error(msg, len(this) + 1 + len(that)) - lo = code1[1] - hi = code2[1] - if hi < lo: - msg = "bad character range %s-%s" % (this, that) - raise source.error(msg, len(this) + 1 + len(that)) - setappend((RANGE, (lo, hi))) - else: - if code1[0] is IN: - code1 = code1[1][0] - setappend(code1) - - set = _uniq(set) - # XXX: should move set optimization to compiler! - if _len(set) == 1 and set[0][0] is LITERAL: - # optimization - if negate: - subpatternappend((NOT_LITERAL, set[0][1])) - else: - subpatternappend(set[0]) - else: - if negate: - set.insert(0, (NEGATE, None)) - # charmap optimization can't be added here because - # global flags still are not known - subpatternappend((IN, set)) - - elif this in REPEAT_CHARS: - # repeat previous item - here = source.tell() - if this == "?": - min, max = 0, 1 - elif this == "*": - min, max = 0, MAXREPEAT - - elif this == "+": - min, max = 1, MAXREPEAT - elif this == "{": - if source.next == "}": - subpatternappend((LITERAL, _ord(this))) - continue - - min, max = 0, MAXREPEAT - lo = hi = "" - while source.next in DIGITS: - lo += sourceget() - if sourcematch(","): - while source.next in DIGITS: - hi += sourceget() - else: - hi = lo - if not sourcematch("}"): - subpatternappend((LITERAL, _ord(this))) - source.seek(here) - continue - - if lo: - min = int(lo) - if min >= MAXREPEAT: - raise OverflowError("the repetition number is too large") - if hi: - max = int(hi) - if max >= MAXREPEAT: - raise OverflowError("the repetition number is too large") - if max < min: - raise source.error("min repeat greater than max repeat", - source.tell() - here) - else: - raise AssertionError("unsupported quantifier %r" % (char,)) - # figure out which item to repeat - if subpattern: - item = subpattern[-1:] - else: - item = None - if not item or item[0][0] is AT: - raise source.error("nothing to repeat", - source.tell() - here + len(this)) - if item[0][0] in _REPEATCODES: - raise source.error("multiple repeat", - source.tell() - here + len(this)) - if item[0][0] is SUBPATTERN: - group, add_flags, del_flags, p = item[0][1] - if group is None and not add_flags and not del_flags: - item = p - if sourcematch("?"): - # Non-Greedy Match - subpattern[-1] = (MIN_REPEAT, (min, max, item)) - elif sourcematch("+"): - # Possessive Match (Always Greedy) - subpattern[-1] = (POSSESSIVE_REPEAT, (min, max, item)) - else: - # Greedy Match - subpattern[-1] = (MAX_REPEAT, (min, max, item)) - - elif this == ".": - subpatternappend((ANY, None)) - - elif this == "(": - start = source.tell() - 1 - capture = True - atomic = False - name = None - add_flags = 0 - del_flags = 0 - if sourcematch("?"): - # options - char = sourceget() - if char is None: - raise source.error("unexpected end of pattern") - if char == "P": - # python extensions - if sourcematch("<"): - # named group: skip forward to end of name - name = source.getuntil(">", "group name") - if not name.isidentifier(): - msg = "bad character in group name %r" % name - raise source.error(msg, len(name) + 1) - elif sourcematch("="): - # named backreference - name = source.getuntil(")", "group name") - if not name.isidentifier(): - msg = "bad character in group name %r" % name - raise source.error(msg, len(name) + 1) - gid = state.groupdict.get(name) - if gid is None: - msg = "unknown group name %r" % name - raise source.error(msg, len(name) + 1) - if not state.checkgroup(gid): - raise source.error("cannot refer to an open group", - len(name) + 1) - state.checklookbehindgroup(gid, source) - subpatternappend((GROUPREF, gid)) - continue - - else: - char = sourceget() - if char is None: - raise source.error("unexpected end of pattern") - raise source.error("unknown extension ?P" + char, - len(char) + 2) - elif char == ":": - # non-capturing group - capture = False - elif char == "#": - # comment - while True: - if source.next is None: - raise source.error("missing ), unterminated comment", - source.tell() - start) - if sourceget() == ")": - break - continue - - elif char in "=!<": - # lookahead assertions - dir = 1 - if char == "<": - char = sourceget() - if char is None: - raise source.error("unexpected end of pattern") - if char not in "=!": - raise source.error("unknown extension ?<" + char, - len(char) + 2) - dir = -1 # lookbehind - lookbehindgroups = state.lookbehindgroups - if lookbehindgroups is None: - state.lookbehindgroups = state.groups - p = _parse_sub(source, state, verbose, nested + 1) - if dir < 0: - if lookbehindgroups is None: - state.lookbehindgroups = None - if not sourcematch(")"): - raise source.error("missing ), unterminated subpattern", - source.tell() - start) - if char == "=": - subpatternappend((ASSERT, (dir, p))) - else: - subpatternappend((ASSERT_NOT, (dir, p))) - continue - - elif char == "(": - # conditional backreference group - condname = source.getuntil(")", "group name") - if condname.isidentifier(): - condgroup = state.groupdict.get(condname) - if condgroup is None: - msg = "unknown group name %r" % condname - raise source.error(msg, len(condname) + 1) - else: - try: - condgroup = int(condname) - if condgroup < 0: - raise ValueError - except ValueError: - msg = "bad character in group name %r" % condname - raise source.error(msg, len(condname) + 1) from None - if not condgroup: - raise source.error("bad group number", - len(condname) + 1) - if condgroup >= MAXGROUPS: - msg = "invalid group reference %d" % condgroup - raise source.error(msg, len(condname) + 1) - state.checklookbehindgroup(condgroup, source) - item_yes = _parse(source, state, verbose, nested + 1) - if source.match("|"): - item_no = _parse(source, state, verbose, nested + 1) - if source.next == "|": - raise source.error("conditional backref with more than two branches") - else: - item_no = None - if not source.match(")"): - raise source.error("missing ), unterminated subpattern", - source.tell() - start) - subpatternappend((GROUPREF_EXISTS, (condgroup, item_yes, item_no))) - continue - - elif char == ">": - # non-capturing, atomic group - capture = False - atomic = True - elif char in FLAGS or char == "-": - # flags - flags = _parse_flags(source, state, char) - if flags is None: # global flags - if not first or subpattern: - raise source.error('global flags not at the start ' - 'of the expression', - source.tell() - start) - if (state.flags & SRE_FLAG_VERBOSE) and not verbose: - raise Verbose - continue - - add_flags, del_flags = flags - capture = False - else: - raise source.error("unknown extension ?" + char, - len(char) + 1) - - # parse group contents - if capture: - try: - group = state.opengroup(name) - except error as err: - raise source.error(err.msg, len(name) + 1) from None - else: - group = None - sub_verbose = ((verbose or (add_flags & SRE_FLAG_VERBOSE)) and - not (del_flags & SRE_FLAG_VERBOSE)) - p = _parse_sub(source, state, sub_verbose, nested + 1) - if not source.match(")"): - raise source.error("missing ), unterminated subpattern", - source.tell() - start) - if group is not None: - state.closegroup(group, p) - if atomic: - assert group is None - subpatternappend((ATOMIC_GROUP, p)) - else: - subpatternappend((SUBPATTERN, (group, add_flags, del_flags, p))) - - elif this == "^": - subpatternappend((AT, AT_BEGINNING)) - - elif this == "$": - subpatternappend((AT, AT_END)) - - else: - raise AssertionError("unsupported special character %r" % (char,)) - - # unpack non-capturing groups - for i in range(len(subpattern))[::-1]: - op, av = subpattern[i] - if op is SUBPATTERN: - group, add_flags, del_flags, p = av - if group is None and not add_flags and not del_flags: - subpattern[i: i+1] = p - - return subpattern - -def _parse_flags(source, state, char): - sourceget = source.get - add_flags = 0 - del_flags = 0 - if char != "-": - while True: - flag = FLAGS[char] - if source.istext: - if char == 'L': - msg = "bad inline flags: cannot use 'L' flag with a str pattern" - raise source.error(msg) - else: - if char == 'u': - msg = "bad inline flags: cannot use 'u' flag with a bytes pattern" - raise source.error(msg) - add_flags |= flag - if (flag & TYPE_FLAGS) and (add_flags & TYPE_FLAGS) != flag: - msg = "bad inline flags: flags 'a', 'u' and 'L' are incompatible" - raise source.error(msg) - char = sourceget() - if char is None: - raise source.error("missing -, : or )") - if char in ")-:": - break - if char not in FLAGS: - msg = "unknown flag" if char.isalpha() else "missing -, : or )" - raise source.error(msg, len(char)) - if char == ")": - state.flags |= add_flags - return None - if add_flags & GLOBAL_FLAGS: - raise source.error("bad inline flags: cannot turn on global flag", 1) - if char == "-": - char = sourceget() - if char is None: - raise source.error("missing flag") - if char not in FLAGS: - msg = "unknown flag" if char.isalpha() else "missing flag" - raise source.error(msg, len(char)) - while True: - flag = FLAGS[char] - if flag & TYPE_FLAGS: - msg = "bad inline flags: cannot turn off flags 'a', 'u' and 'L'" - raise source.error(msg) - del_flags |= flag - char = sourceget() - if char is None: - raise source.error("missing :") - if char == ":": - break - if char not in FLAGS: - msg = "unknown flag" if char.isalpha() else "missing :" - raise source.error(msg, len(char)) - assert char == ":" - if del_flags & GLOBAL_FLAGS: - raise source.error("bad inline flags: cannot turn off global flag", 1) - if add_flags & del_flags: - raise source.error("bad inline flags: flag turned on and off", 1) - return add_flags, del_flags - -def fix_flags(src, flags): - # Check and fix flags according to the type of pattern (str or bytes) - if isinstance(src, str): - if flags & SRE_FLAG_LOCALE: - raise ValueError("cannot use LOCALE flag with a str pattern") - if not flags & SRE_FLAG_ASCII: - flags |= SRE_FLAG_UNICODE - elif flags & SRE_FLAG_UNICODE: - raise ValueError("ASCII and UNICODE flags are incompatible") - else: - if flags & SRE_FLAG_UNICODE: - raise ValueError("cannot use UNICODE flag with a bytes pattern") - if flags & SRE_FLAG_LOCALE and flags & SRE_FLAG_ASCII: - raise ValueError("ASCII and LOCALE flags are incompatible") - return flags - -def parse(str, flags=0, state=None): - # parse 're' pattern into list of (opcode, argument) tuples - - source = Tokenizer(str) - - if state is None: - state = State() - state.flags = flags - state.str = str - - try: - p = _parse_sub(source, state, flags & SRE_FLAG_VERBOSE, 0) - except Verbose: - # the VERBOSE flag was switched on inside the pattern. to be - # on the safe side, we'll parse the whole thing again... - state = State() - state.flags = flags | SRE_FLAG_VERBOSE - state.str = str - source.seek(0) - p = _parse_sub(source, state, True, 0) - - p.state.flags = fix_flags(str, p.state.flags) - - if source.next is not None: - assert source.next == ")" - raise source.error("unbalanced parenthesis") - - if flags & SRE_FLAG_DEBUG: - p.dump() - - return p - -def parse_template(source, state): - # parse 're' replacement string into list of literals and - # group references - s = Tokenizer(source) - sget = s.get - groups = [] - literals = [] - literal = [] - lappend = literal.append - def addgroup(index, pos): - if index > state.groups: - raise s.error("invalid group reference %d" % index, pos) - if literal: - literals.append(''.join(literal)) - del literal[:] - groups.append((len(literals), index)) - literals.append(None) - groupindex = state.groupindex - while True: - this = sget() - if this is None: - break # end of replacement string - if this[0] == "\\": - # group - c = this[1] - if c == "g": - name = "" - if not s.match("<"): - raise s.error("missing <") - name = s.getuntil(">", "group name") - if name.isidentifier(): - try: - index = groupindex[name] - except KeyError: - raise IndexError("unknown group name %r" % name) - else: - try: - index = int(name) - if index < 0: - raise ValueError - except ValueError: - raise s.error("bad character in group name %r" % name, - len(name) + 1) from None - if index >= MAXGROUPS: - raise s.error("invalid group reference %d" % index, - len(name) + 1) - addgroup(index, len(name) + 1) - elif c == "0": - if s.next in OCTDIGITS: - this += sget() - if s.next in OCTDIGITS: - this += sget() - lappend(chr(int(this[1:], 8) & 0xff)) - elif c in DIGITS: - isoctal = False - if s.next in DIGITS: - this += sget() - if (c in OCTDIGITS and this[2] in OCTDIGITS and - s.next in OCTDIGITS): - this += sget() - isoctal = True - c = int(this[1:], 8) - if c > 0o377: - raise s.error('octal escape value %s outside of ' - 'range 0-0o377' % this, len(this)) - lappend(chr(c)) - if not isoctal: - addgroup(int(this[1:]), len(this) - 1) - else: - try: - this = chr(ESCAPES[this][1]) - except KeyError: - if c in ASCIILETTERS: - raise s.error('bad escape %s' % this, len(this)) - lappend(this) - else: - lappend(this) - if literal: - literals.append(''.join(literal)) - if not isinstance(source, str): - # The tokenizer implicitly decodes bytes objects as latin-1, we must - # therefore re-encode the final representation. - literals = [None if s is None else s.encode('latin-1') for s in literals] - return groups, literals - -def expand_template(template, match): - g = match.group - empty = match.string[:0] - groups, literals = template - literals = literals[:] - try: - for index, group in groups: - literals[index] = g(group) or empty - except IndexError: - raise error("invalid group reference %d" % index) - return empty.join(literals) +from re import _parser as _ +globals().update({k: v for k, v in vars(_).items() if k[:2] != '__'}) diff --git a/Lib/test/test_pyclbr.py b/Lib/test/test_pyclbr.py index 4bb9cfcad9a76..157c522b63bd0 100644 --- a/Lib/test/test_pyclbr.py +++ b/Lib/test/test_pyclbr.py @@ -221,7 +221,7 @@ def test_others(self): cm('cgi', ignore=('log',)) # set with = in module cm('pickle', ignore=('partial', 'PickleBuffer')) cm('aifc', ignore=('_aifc_params',)) # set with = in module - cm('sre_parse', ignore=('dump', 'groups', 'pos')) # from sre_constants import *; property + cm('re._parser', ignore=('dump', 'groups', 'pos')) # from ._constants import *; property cm( 'pdb', # pyclbr does not handle elegantly `typing` or properties diff --git a/Lib/test/test_re.py b/Lib/test/test_re.py index 85716fbe2a8e8..f1e5af452d8e0 100644 --- a/Lib/test/test_re.py +++ b/Lib/test/test_re.py @@ -3,8 +3,8 @@ check_disallow_instantiation, is_emscripten) import locale import re -import sre_compile import string +import sys import time import unittest import warnings @@ -569,7 +569,7 @@ def test_re_groupref_exists(self): 'two branches', 10) def test_re_groupref_overflow(self): - from sre_constants import MAXGROUPS + from re._constants import MAXGROUPS self.checkTemplateError('()', r'\g<%s>' % MAXGROUPS, 'xx', 'invalid group reference %d' % MAXGROUPS, 3) self.checkPatternError(r'(?P)(?(%d))' % MAXGROUPS, @@ -2433,7 +2433,7 @@ def test_immutable(self): tp.foo = 1 def test_overlap_table(self): - f = sre_compile._generate_overlap_table + f = re._compiler._generate_overlap_table self.assertEqual(f(""), []) self.assertEqual(f("a"), [0]) self.assertEqual(f("abcd"), [0, 0, 0, 0]) @@ -2442,8 +2442,8 @@ def test_overlap_table(self): self.assertEqual(f("abcabdac"), [0, 0, 0, 1, 2, 0, 1, 0]) def test_signedness(self): - self.assertGreaterEqual(sre_compile.MAXREPEAT, 0) - self.assertGreaterEqual(sre_compile.MAXGROUPS, 0) + self.assertGreaterEqual(re._compiler.MAXREPEAT, 0) + self.assertGreaterEqual(re._compiler.MAXGROUPS, 0) @cpython_only def test_disallow_instantiation(self): @@ -2453,6 +2453,32 @@ def test_disallow_instantiation(self): pat = re.compile("") check_disallow_instantiation(self, type(pat.scanner(""))) + def test_deprecated_modules(self): + deprecated = { + 'sre_compile': ['compile', 'error', + 'SRE_FLAG_IGNORECASE', 'SUBPATTERN', + '_compile_info'], + 'sre_constants': ['error', 'SRE_FLAG_IGNORECASE', 'SUBPATTERN', + '_NamedIntConstant'], + 'sre_parse': ['SubPattern', 'parse', + 'SRE_FLAG_IGNORECASE', 'SUBPATTERN', + '_parse_sub'], + } + for name in deprecated: + with self.subTest(module=name): + sys.modules.pop(name, None) + with self.assertWarns(DeprecationWarning) as cm: + __import__(name) + self.assertEqual(str(cm.warnings[0].message), + f"module {name!r} is deprecated") + self.assertEqual(cm.warnings[0].filename, __file__) + self.assertIn(name, sys.modules) + mod = sys.modules[name] + self.assertEqual(mod.__name__, name) + self.assertEqual(mod.__package__, '') + for attr in deprecated[name]: + self.assertTrue(hasattr(mod, attr)) + del sys.modules[name] class ExternalTests(unittest.TestCase): diff --git a/Lib/test/test_site.py b/Lib/test/test_site.py index a67cfec72aee4..dd018d6b38a86 100644 --- a/Lib/test/test_site.py +++ b/Lib/test/test_site.py @@ -523,7 +523,7 @@ def test_startup_imports(self): self.assertIn('site', modules) # http://bugs.python.org/issue19205 - re_mods = {'re', '_sre', 'sre_compile', 'sre_constants', 'sre_parse'} + re_mods = {'re', '_sre', 're._compiler', 're._constants', 're._parser'} self.assertFalse(modules.intersection(re_mods), stderr) # http://bugs.python.org/issue9548 diff --git a/Makefile.pre.in b/Makefile.pre.in index 6dda71bc49cff..5318a41dc857a 100644 --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -1862,6 +1862,7 @@ LIBSUBDIRS= asyncio \ logging \ multiprocessing multiprocessing/dummy \ pydoc_data \ + re \ site-packages \ sqlite3 \ tkinter \ diff --git a/Misc/NEWS.d/next/Library/2022-03-29-19-14-53.bpo-47152.5rl5ZK.rst b/Misc/NEWS.d/next/Library/2022-03-29-19-14-53.bpo-47152.5rl5ZK.rst new file mode 100644 index 0000000000000..1e1633daae597 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-03-29-19-14-53.bpo-47152.5rl5ZK.rst @@ -0,0 +1,2 @@ +Convert the :mod:`re` module into a package. Deprecate modules ``sre_compile``, +``sre_constants`` and ``sre_parse``. diff --git a/Modules/sre_constants.h b/Modules/sre_constants.h index 8b9125b75b456..45395dcea807a 100644 --- a/Modules/sre_constants.h +++ b/Modules/sre_constants.h @@ -3,8 +3,8 @@ * * regular expression matching engine * - * NOTE: This file is generated by sre_constants.py. If you need - * to change anything in here, edit sre_constants.py and run it. + * NOTE: This file is generated by Lib/re/_constants.py. If you need + * to change anything in here, edit Lib/re/_constants.py and run it. * * Copyright (c) 1997-2001 by Secret Labs AB. All rights reserved. * From webhook-mailer at python.org Sat Apr 2 09:08:44 2022 From: webhook-mailer at python.org (JulienPalard) Date: Sat, 02 Apr 2022 13:08:44 -0000 Subject: [Python-checkins] [doc] fix superfluous backtick in front of role. (GH-32220) Message-ID: https://github.com/python/cpython/commit/7f9c084fdec7ddcfe8855aa79f98545591ae2261 commit: 7f9c084fdec7ddcfe8855aa79f98545591ae2261 branch: main author: Julien Palard committer: JulienPalard date: 2022-04-02T15:08:36+02:00 summary: [doc] fix superfluous backtick in front of role. (GH-32220) files: M Doc/whatsnew/3.11.rst diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index 0c7f4afa694ad..809cbd556c8c7 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -1017,8 +1017,8 @@ Porting to Python 3.11 instead. Debuggers that accessed the ``f_locals`` directly *must* call - `:c:func:`PyFrame_GetLocals` instead. They no longer need to call - `:c:func:`PyFrame_FastToLocalsWithError` or :c:func:`PyFrame_LocalsToFast`, + :c:func:`PyFrame_GetLocals` instead. They no longer need to call + :c:func:`PyFrame_FastToLocalsWithError` or :c:func:`PyFrame_LocalsToFast`, in fact they should not call those functions. The necessary updating of the frame is now managed by the virtual machine. From webhook-mailer at python.org Sat Apr 2 09:10:27 2022 From: webhook-mailer at python.org (zooba) Date: Sat, 02 Apr 2022 13:10:27 -0000 Subject: [Python-checkins] bpo-47194: Update zlib to v1.2.12 on Windows to resolve CVE-2018-25032 (GH-32241) Message-ID: https://github.com/python/cpython/commit/6066739ff7794e54c98c08b953a699cbc961cd28 commit: 6066739ff7794e54c98c08b953a699cbc961cd28 branch: main author: Zachary Ware committer: zooba date: 2022-04-02T14:10:23+01:00 summary: bpo-47194: Update zlib to v1.2.12 on Windows to resolve CVE-2018-25032 (GH-32241) files: A Misc/NEWS.d/next/Windows/2022-04-01-14-57-40.bpo-47194.IB0XL4.rst M PCbuild/get_externals.bat M PCbuild/python.props diff --git a/Misc/NEWS.d/next/Windows/2022-04-01-14-57-40.bpo-47194.IB0XL4.rst b/Misc/NEWS.d/next/Windows/2022-04-01-14-57-40.bpo-47194.IB0XL4.rst new file mode 100644 index 0000000000000..7e76add45fa95 --- /dev/null +++ b/Misc/NEWS.d/next/Windows/2022-04-01-14-57-40.bpo-47194.IB0XL4.rst @@ -0,0 +1 @@ +Update ``zlib`` to v1.2.12 to resolve CVE-2018-25032. diff --git a/PCbuild/get_externals.bat b/PCbuild/get_externals.bat index 189b31246d545..481e06d9fcd06 100644 --- a/PCbuild/get_externals.bat +++ b/PCbuild/get_externals.bat @@ -59,7 +59,7 @@ if NOT "%IncludeTkinterSrc%"=="false" set libraries=%libraries% tcl-core-8.6.12. if NOT "%IncludeTkinterSrc%"=="false" set libraries=%libraries% tk-8.6.12.1 if NOT "%IncludeTkinterSrc%"=="false" set libraries=%libraries% tix-8.4.3.6 set libraries=%libraries% xz-5.2.2 -set libraries=%libraries% zlib-1.2.11 +set libraries=%libraries% zlib-1.2.12 for %%e in (%libraries%) do ( if exist "%EXTERNALS_DIR%\%%e" ( diff --git a/PCbuild/python.props b/PCbuild/python.props index e64173737cf72..b9211f60d86d4 100644 --- a/PCbuild/python.props +++ b/PCbuild/python.props @@ -71,7 +71,7 @@ $(ExternalsDir)openssl-bin-1.1.1n\$(ArchName)\ $(opensslOutDir)include $(ExternalsDir)\nasm-2.11.06\ - $(ExternalsDir)\zlib-1.2.11\ + $(ExternalsDir)\zlib-1.2.12\ _d From webhook-mailer at python.org Sat Apr 2 09:37:57 2022 From: webhook-mailer at python.org (miss-islington) Date: Sat, 02 Apr 2022 13:37:57 -0000 Subject: [Python-checkins] bpo-47194: Update zlib to v1.2.12 on Windows to resolve CVE-2018-25032 (GH-32241) Message-ID: https://github.com/python/cpython/commit/0f0f85e9d8088eb789cda35477900df32adff546 commit: 0f0f85e9d8088eb789cda35477900df32adff546 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-02T06:37:39-07:00 summary: bpo-47194: Update zlib to v1.2.12 on Windows to resolve CVE-2018-25032 (GH-32241) (cherry picked from commit 6066739ff7794e54c98c08b953a699cbc961cd28) Co-authored-by: Zachary Ware files: A Misc/NEWS.d/next/Windows/2022-04-01-14-57-40.bpo-47194.IB0XL4.rst M PCbuild/get_externals.bat M PCbuild/python.props diff --git a/Misc/NEWS.d/next/Windows/2022-04-01-14-57-40.bpo-47194.IB0XL4.rst b/Misc/NEWS.d/next/Windows/2022-04-01-14-57-40.bpo-47194.IB0XL4.rst new file mode 100644 index 0000000000000..7e76add45fa95 --- /dev/null +++ b/Misc/NEWS.d/next/Windows/2022-04-01-14-57-40.bpo-47194.IB0XL4.rst @@ -0,0 +1 @@ +Update ``zlib`` to v1.2.12 to resolve CVE-2018-25032. diff --git a/PCbuild/get_externals.bat b/PCbuild/get_externals.bat index 7129898c6bc62..dc5c909de7441 100644 --- a/PCbuild/get_externals.bat +++ b/PCbuild/get_externals.bat @@ -59,7 +59,7 @@ if NOT "%IncludeTkinterSrc%"=="false" set libraries=%libraries% tcl-core-8.6.12. if NOT "%IncludeTkinterSrc%"=="false" set libraries=%libraries% tk-8.6.12.0 if NOT "%IncludeTkinterSrc%"=="false" set libraries=%libraries% tix-8.4.3.6 set libraries=%libraries% xz-5.2.2 -set libraries=%libraries% zlib-1.2.11 +set libraries=%libraries% zlib-1.2.12 for %%e in (%libraries%) do ( if exist "%EXTERNALS_DIR%\%%e" ( diff --git a/PCbuild/python.props b/PCbuild/python.props index 7a11cfafedf5e..7b42037b7cdf8 100644 --- a/PCbuild/python.props +++ b/PCbuild/python.props @@ -67,7 +67,7 @@ $(ExternalsDir)openssl-bin-1.1.1n\$(ArchName)\ $(opensslOutDir)include $(ExternalsDir)\nasm-2.11.06\ - $(ExternalsDir)\zlib-1.2.11\ + $(ExternalsDir)\zlib-1.2.12\ _d From webhook-mailer at python.org Sat Apr 2 09:39:11 2022 From: webhook-mailer at python.org (miss-islington) Date: Sat, 02 Apr 2022 13:39:11 -0000 Subject: [Python-checkins] bpo-47194: Update zlib to v1.2.12 on Windows to resolve CVE-2018-25032 (GH-32241) Message-ID: https://github.com/python/cpython/commit/16a809ffb7af14898ce9ec8165960d96cbcd4ec3 commit: 16a809ffb7af14898ce9ec8165960d96cbcd4ec3 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-02T06:39:03-07:00 summary: bpo-47194: Update zlib to v1.2.12 on Windows to resolve CVE-2018-25032 (GH-32241) (cherry picked from commit 6066739ff7794e54c98c08b953a699cbc961cd28) Co-authored-by: Zachary Ware files: A Misc/NEWS.d/next/Windows/2022-04-01-14-57-40.bpo-47194.IB0XL4.rst M PCbuild/get_externals.bat M PCbuild/python.props diff --git a/Misc/NEWS.d/next/Windows/2022-04-01-14-57-40.bpo-47194.IB0XL4.rst b/Misc/NEWS.d/next/Windows/2022-04-01-14-57-40.bpo-47194.IB0XL4.rst new file mode 100644 index 0000000000000..7e76add45fa95 --- /dev/null +++ b/Misc/NEWS.d/next/Windows/2022-04-01-14-57-40.bpo-47194.IB0XL4.rst @@ -0,0 +1 @@ +Update ``zlib`` to v1.2.12 to resolve CVE-2018-25032. diff --git a/PCbuild/get_externals.bat b/PCbuild/get_externals.bat index 7129898c6bc62..dc5c909de7441 100644 --- a/PCbuild/get_externals.bat +++ b/PCbuild/get_externals.bat @@ -59,7 +59,7 @@ if NOT "%IncludeTkinterSrc%"=="false" set libraries=%libraries% tcl-core-8.6.12. if NOT "%IncludeTkinterSrc%"=="false" set libraries=%libraries% tk-8.6.12.0 if NOT "%IncludeTkinterSrc%"=="false" set libraries=%libraries% tix-8.4.3.6 set libraries=%libraries% xz-5.2.2 -set libraries=%libraries% zlib-1.2.11 +set libraries=%libraries% zlib-1.2.12 for %%e in (%libraries%) do ( if exist "%EXTERNALS_DIR%\%%e" ( diff --git a/PCbuild/python.props b/PCbuild/python.props index 7a11cfafedf5e..7b42037b7cdf8 100644 --- a/PCbuild/python.props +++ b/PCbuild/python.props @@ -67,7 +67,7 @@ $(ExternalsDir)openssl-bin-1.1.1n\$(ArchName)\ $(opensslOutDir)include $(ExternalsDir)\nasm-2.11.06\ - $(ExternalsDir)\zlib-1.2.11\ + $(ExternalsDir)\zlib-1.2.12\ _d From webhook-mailer at python.org Sat Apr 2 15:31:15 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Sat, 02 Apr 2022 19:31:15 -0000 Subject: [Python-checkins] More minor fixes to C API docs (GH-31525) Message-ID: https://github.com/python/cpython/commit/897bc6f9282238d5fb32d232ab62d30675244736 commit: 897bc6f9282238d5fb32d232ab62d30675244736 branch: main author: Jelle Zijlstra committer: JelleZijlstra date: 2022-04-02T12:31:05-07:00 summary: More minor fixes to C API docs (GH-31525) * wording fixes in type.rst * grammar and punctuation in sys.rst * set: grammar fixes * structures: capitalization fix * grammar fixes for sequence * objects: point to Py_TYPE instead of direct object access * numbers: add more explicit Python equivalences * method: add missing period * memory: grammar fix * mapping: grammar fixes * long: grammar fix * iter: fix grammar for PyAIter_Check * init: grammar fix files: M Doc/c-api/init.rst M Doc/c-api/iter.rst M Doc/c-api/long.rst M Doc/c-api/mapping.rst M Doc/c-api/memory.rst M Doc/c-api/method.rst M Doc/c-api/number.rst M Doc/c-api/object.rst M Doc/c-api/sequence.rst M Doc/c-api/set.rst M Doc/c-api/structures.rst M Doc/c-api/sys.rst M Doc/c-api/type.rst diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst index 152d4c8e5036b..82c4acebf1bc9 100644 --- a/Doc/c-api/init.rst +++ b/Doc/c-api/init.rst @@ -1792,7 +1792,7 @@ is not possible due to its implementation being opaque at build time. argument is `NULL`. .. note:: - A freed key becomes a dangling pointer, you should reset the key to + A freed key becomes a dangling pointer. You should reset the key to `NULL`. diff --git a/Doc/c-api/iter.rst b/Doc/c-api/iter.rst index 3e388bb917a02..434d2021cea8e 100644 --- a/Doc/c-api/iter.rst +++ b/Doc/c-api/iter.rst @@ -14,8 +14,8 @@ There are two functions specifically for working with iterators. .. c:function:: int PyAIter_Check(PyObject *o) - Returns non-zero if the object 'obj' provides :class:`AsyncIterator` - protocols, and ``0`` otherwise. This function always succeeds. + Return non-zero if the object *o* provides the :class:`AsyncIterator` + protocol, and ``0`` otherwise. This function always succeeds. .. versionadded:: 3.10 diff --git a/Doc/c-api/long.rst b/Doc/c-api/long.rst index 4201490286b82..620344e71373b 100644 --- a/Doc/c-api/long.rst +++ b/Doc/c-api/long.rst @@ -41,7 +41,7 @@ distinguished from a number. Use :c:func:`PyErr_Occurred` to disambiguate. Return a new :c:type:`PyLongObject` object from *v*, or ``NULL`` on failure. The current implementation keeps an array of integer objects for all integers - between ``-5`` and ``256``, when you create an int in that range you actually + between ``-5`` and ``256``. When you create an int in that range you actually just get back a reference to the existing object. diff --git a/Doc/c-api/mapping.rst b/Doc/c-api/mapping.rst index 682160d1475c1..3c9d282c6d0ab 100644 --- a/Doc/c-api/mapping.rst +++ b/Doc/c-api/mapping.rst @@ -11,10 +11,10 @@ See also :c:func:`PyObject_GetItem`, :c:func:`PyObject_SetItem` and .. c:function:: int PyMapping_Check(PyObject *o) - Return ``1`` if the object provides mapping protocol or supports slicing, + Return ``1`` if the object provides the mapping protocol or supports slicing, and ``0`` otherwise. Note that it returns ``1`` for Python classes with - a :meth:`__getitem__` method since in general case it is impossible to - determine what type of keys it supports. This function always succeeds. + a :meth:`__getitem__` method, since in general it is impossible to + determine what type of keys the class supports. This function always succeeds. .. c:function:: Py_ssize_t PyMapping_Size(PyObject *o) diff --git a/Doc/c-api/memory.rst b/Doc/c-api/memory.rst index 01b646c226e6d..e81a246cb75cc 100644 --- a/Doc/c-api/memory.rst +++ b/Doc/c-api/memory.rst @@ -306,7 +306,7 @@ memory from the Python heap. .. note:: There is no guarantee that the memory returned by these allocators can be - successfully casted to a Python object when intercepting the allocating + successfully cast to a Python object when intercepting the allocating functions in this domain by the methods described in the :ref:`Customize Memory Allocators ` section. diff --git a/Doc/c-api/method.rst b/Doc/c-api/method.rst index 23852251dfe02..6e7e1e21aa93f 100644 --- a/Doc/c-api/method.rst +++ b/Doc/c-api/method.rst @@ -27,7 +27,7 @@ to bind a :c:data:`PyCFunction` to a class object. It replaces the former call .. c:function:: PyObject* PyInstanceMethod_New(PyObject *func) - Return a new instance method object, with *func* being any callable object + Return a new instance method object, with *func* being any callable object. *func* is the function that will be called when the instance method is called. diff --git a/Doc/c-api/number.rst b/Doc/c-api/number.rst index 37979bb506bcf..11c9c67d36a67 100644 --- a/Doc/c-api/number.rst +++ b/Doc/c-api/number.rst @@ -44,7 +44,7 @@ Number Protocol .. c:function:: PyObject* PyNumber_FloorDivide(PyObject *o1, PyObject *o2) Return the floor of *o1* divided by *o2*, or ``NULL`` on failure. This is - equivalent to the "classic" division of integers. + the equivalent of the Python expression ``o1 // o2``. .. c:function:: PyObject* PyNumber_TrueDivide(PyObject *o1, PyObject *o2) @@ -53,7 +53,7 @@ Number Protocol *o2*, or ``NULL`` on failure. The return value is "approximate" because binary floating point numbers are approximate; it is not possible to represent all real numbers in base two. This function can return a floating point value when - passed two integers. + passed two integers. This is the equivalent of the Python expression ``o1 / o2``. .. c:function:: PyObject* PyNumber_Remainder(PyObject *o1, PyObject *o2) @@ -180,6 +180,7 @@ Number Protocol floating point numbers are approximate; it is not possible to represent all real numbers in base two. This function can return a floating point value when passed two integers. The operation is done *in-place* when *o1* supports it. + This is the equivalent of the Python statement ``o1 /= o2``. .. c:function:: PyObject* PyNumber_InPlaceRemainder(PyObject *o1, PyObject *o2) @@ -285,6 +286,6 @@ Number Protocol .. c:function:: int PyIndex_Check(PyObject *o) - Returns ``1`` if *o* is an index integer (has the nb_index slot of the - tp_as_number structure filled in), and ``0`` otherwise. + Returns ``1`` if *o* is an index integer (has the ``nb_index`` slot of the + ``tp_as_number`` structure filled in), and ``0`` otherwise. This function always succeeds. diff --git a/Doc/c-api/object.rst b/Doc/c-api/object.rst index 41a3affcf9842..9dcfd769c64a0 100644 --- a/Doc/c-api/object.rst +++ b/Doc/c-api/object.rst @@ -93,7 +93,7 @@ Object Protocol return ``0`` on success. This is the equivalent of the Python statement ``o.attr_name = v``. - If *v* is ``NULL``, the attribute is deleted, however this feature is + If *v* is ``NULL``, the attribute is deleted, but this feature is deprecated in favour of using :c:func:`PyObject_DelAttrString`. @@ -291,7 +291,7 @@ Object Protocol of object *o*. On failure, raises :exc:`SystemError` and returns ``NULL``. This is equivalent to the Python expression ``type(o)``. This function increments the reference count of the return value. There's really no reason to use this - function instead of the common expression ``o->ob_type``, which returns a + function instead of the :c:func:`Py_TYPE()` function, which returns a pointer of type :c:type:`PyTypeObject*`, except when the incremented reference count is needed. diff --git a/Doc/c-api/sequence.rst b/Doc/c-api/sequence.rst index 6581885904117..c78d273f9f149 100644 --- a/Doc/c-api/sequence.rst +++ b/Doc/c-api/sequence.rst @@ -8,10 +8,10 @@ Sequence Protocol .. c:function:: int PySequence_Check(PyObject *o) - Return ``1`` if the object provides sequence protocol, and ``0`` otherwise. + Return ``1`` if the object provides the sequence protocol, and ``0`` otherwise. Note that it returns ``1`` for Python classes with a :meth:`__getitem__` - method unless they are :class:`dict` subclasses since in general case it - is impossible to determine what the type of keys it supports. This + method, unless they are :class:`dict` subclasses, since in general it + is impossible to determine what type of keys the class supports. This function always succeeds. @@ -69,7 +69,7 @@ Sequence Protocol is the equivalent of the Python statement ``o[i] = v``. This function *does not* steal a reference to *v*. - If *v* is ``NULL``, the element is deleted, however this feature is + If *v* is ``NULL``, the element is deleted, but this feature is deprecated in favour of using :c:func:`PySequence_DelItem`. @@ -147,7 +147,7 @@ Sequence Protocol Returns the length of *o*, assuming that *o* was returned by :c:func:`PySequence_Fast` and that *o* is not ``NULL``. The size can also be - gotten by calling :c:func:`PySequence_Size` on *o*, but + retrieved by calling :c:func:`PySequence_Size` on *o*, but :c:func:`PySequence_Fast_GET_SIZE` is faster because it can assume *o* is a list or tuple. diff --git a/Doc/c-api/set.rst b/Doc/c-api/set.rst index eca19c4d81647..f0d905bae8ae4 100644 --- a/Doc/c-api/set.rst +++ b/Doc/c-api/set.rst @@ -13,7 +13,7 @@ Set Objects object: frozenset This section details the public API for :class:`set` and :class:`frozenset` -objects. Any functionality not listed below is best accessed using the either +objects. Any functionality not listed below is best accessed using either the abstract object protocol (including :c:func:`PyObject_CallMethod`, :c:func:`PyObject_RichCompareBool`, :c:func:`PyObject_Hash`, :c:func:`PyObject_Repr`, :c:func:`PyObject_IsTrue`, :c:func:`PyObject_Print`, and @@ -31,7 +31,7 @@ the abstract object protocol (including :c:func:`PyObject_CallMethod`, in that it is a fixed size for small sets (much like tuple storage) and will point to a separate, variable sized block of memory for medium and large sized sets (much like list storage). None of the fields of this structure should be - considered public and are subject to change. All access should be done through + considered public and all are subject to change. All access should be done through the documented API rather than by manipulating the values in the structure. @@ -131,7 +131,7 @@ or :class:`frozenset` or instances of their subtypes. .. c:function:: int PySet_Add(PyObject *set, PyObject *key) Add *key* to a :class:`set` instance. Also works with :class:`frozenset` - instances (like :c:func:`PyTuple_SetItem` it can be used to fill-in the values + instances (like :c:func:`PyTuple_SetItem` it can be used to fill in the values of brand new frozensets before they are exposed to other code). Return ``0`` on success or ``-1`` on failure. Raise a :exc:`TypeError` if the *key* is unhashable. Raise a :exc:`MemoryError` if there is no room to grow. Raise a diff --git a/Doc/c-api/structures.rst b/Doc/c-api/structures.rst index 3270d7d8fba45..aa74f6cce1ac7 100644 --- a/Doc/c-api/structures.rst +++ b/Doc/c-api/structures.rst @@ -504,7 +504,7 @@ Accessing attributes of extension types +=============+==================+===================================+ | name | const char \* | attribute name | +-------------+------------------+-----------------------------------+ - | get | getter | C Function to get the attribute | + | get | getter | C function to get the attribute | +-------------+------------------+-----------------------------------+ | set | setter | optional C function to set or | | | | delete the attribute, if omitted | diff --git a/Doc/c-api/sys.rst b/Doc/c-api/sys.rst index cca8b7bb6d644..de94e3f0180f0 100644 --- a/Doc/c-api/sys.rst +++ b/Doc/c-api/sys.rst @@ -177,7 +177,7 @@ Operating System Utilities Return a pointer to a newly allocated byte string, use :c:func:`PyMem_Free` to free the memory. Return ``NULL`` on encoding error or memory allocation - error + error. If error_pos is not ``NULL``, ``*error_pos`` is set to ``(size_t)-1`` on success, or set to the index of the invalid character on encoding error. @@ -207,7 +207,7 @@ Operating System Utilities .. versionchanged:: 3.8 The function now uses the UTF-8 encoding on Windows if - :c:data:`Py_LegacyWindowsFSEncodingFlag` is zero; + :c:data:`Py_LegacyWindowsFSEncodingFlag` is zero. .. _systemfunctions: @@ -356,7 +356,7 @@ accessible to C code. They all work with the current interpreter thread's .. c:function:: int PySys_AddAuditHook(Py_AuditHookFunction hook, void *userData) Append the callable *hook* to the list of active auditing hooks. - Return zero for success + Return zero on success and non-zero on failure. If the runtime has been initialized, also set an error on failure. Hooks added through this API are called for all interpreters created by the runtime. diff --git a/Doc/c-api/type.rst b/Doc/c-api/type.rst index e6a5a0ea9c121..d740e4eb0897e 100644 --- a/Doc/c-api/type.rst +++ b/Doc/c-api/type.rst @@ -296,12 +296,12 @@ The following functions and structs are used to create .. versionchanged:: 3.9 - Slots in :c:type:`PyBufferProcs` in may be set in the unlimited API. + Slots in :c:type:`PyBufferProcs` may be set in the unlimited API. .. versionchanged:: 3.11 :c:member:`~PyBufferProcs.bf_getbuffer` and :c:member:`~PyBufferProcs.bf_releasebuffer` are now available - under limited API. + under the limited API. .. c:member:: void *PyType_Slot.pfunc From webhook-mailer at python.org Sat Apr 2 15:33:09 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Sat, 02 Apr 2022 19:33:09 -0000 Subject: [Python-checkins] More minor fixes to C API docs (GH-31714) Message-ID: https://github.com/python/cpython/commit/677a87946630c5fbd9998969669b4dd4f4b32545 commit: 677a87946630c5fbd9998969669b4dd4f4b32545 branch: main author: Jelle Zijlstra committer: JelleZijlstra date: 2022-04-02T12:32:59-07:00 summary: More minor fixes to C API docs (GH-31714) * init_config: wording fixes * bytearray: remove XXX, there is a good link to the buffer docs * bytes, call, exceptions: minor wording fixes files: M Doc/c-api/bytearray.rst M Doc/c-api/bytes.rst M Doc/c-api/call.rst M Doc/c-api/exceptions.rst M Doc/c-api/init_config.rst diff --git a/Doc/c-api/bytearray.rst b/Doc/c-api/bytearray.rst index 30bcfc7cf9f50..85a7d1373c2a8 100644 --- a/Doc/c-api/bytearray.rst +++ b/Doc/c-api/bytearray.rst @@ -42,8 +42,6 @@ Direct API functions Return a new bytearray object from any object, *o*, that implements the :ref:`buffer protocol `. - .. XXX expand about the buffer protocol, at least somewhere - .. c:function:: PyObject* PyByteArray_FromStringAndSize(const char *string, Py_ssize_t len) diff --git a/Doc/c-api/bytes.rst b/Doc/c-api/bytes.rst index de65701037a7c..32c7b80dc7ac6 100644 --- a/Doc/c-api/bytes.rst +++ b/Doc/c-api/bytes.rst @@ -5,7 +5,7 @@ Bytes Objects ------------- -These functions raise :exc:`TypeError` when expecting a bytes parameter and are +These functions raise :exc:`TypeError` when expecting a bytes parameter and called with a non-bytes parameter. .. index:: object: bytes diff --git a/Doc/c-api/call.rst b/Doc/c-api/call.rst index 739b5e97d1515..cdf72bc1f4714 100644 --- a/Doc/c-api/call.rst +++ b/Doc/c-api/call.rst @@ -26,7 +26,7 @@ This convention is not only used by *tp_call*: :c:member:`~PyTypeObject.tp_new` and :c:member:`~PyTypeObject.tp_init` also pass arguments this way. -To call an object, use :c:func:`PyObject_Call` or other +To call an object, use :c:func:`PyObject_Call` or another :ref:`call API `. diff --git a/Doc/c-api/exceptions.rst b/Doc/c-api/exceptions.rst index f1807f913aace..a5a93d0ebbf28 100644 --- a/Doc/c-api/exceptions.rst +++ b/Doc/c-api/exceptions.rst @@ -281,7 +281,7 @@ For convenience, some of these functions will always return a .. c:function:: void PyErr_SyntaxLocation(const char *filename, int lineno) - Like :c:func:`PyErr_SyntaxLocationEx`, but the col_offset parameter is + Like :c:func:`PyErr_SyntaxLocationEx`, but the *col_offset* parameter is omitted. @@ -333,7 +333,7 @@ an error value). Issue a warning message with explicit control over all warning attributes. This is a straightforward wrapper around the Python function - :func:`warnings.warn_explicit`, see there for more information. The *module* + :func:`warnings.warn_explicit`; see there for more information. The *module* and *registry* arguments may be set to ``NULL`` to get the default effect described there. @@ -441,7 +441,7 @@ Querying the error indicator error indicator. -.. c:function:: void PyErr_NormalizeException(PyObject**exc, PyObject**val, PyObject**tb) +.. c:function:: void PyErr_NormalizeException(PyObject **exc, PyObject **val, PyObject **tb) Under certain circumstances, the values returned by :c:func:`PyErr_Fetch` below can be "unnormalized", meaning that ``*exc`` is a class object but ``*val`` is diff --git a/Doc/c-api/init_config.rst b/Doc/c-api/init_config.rst index 922412c142302..bab5313dd9ec7 100644 --- a/Doc/c-api/init_config.rst +++ b/Doc/c-api/init_config.rst @@ -16,12 +16,12 @@ There are two kinds of configuration: * The :ref:`Python Configuration ` can be used to build a customized Python which behaves as the regular Python. For example, - environments variables and command line arguments are used to configure + environment variables and command line arguments are used to configure Python. * The :ref:`Isolated Configuration ` can be used to embed Python into an application. It isolates Python from the system. For example, - environments variables are ignored, the LC_CTYPE locale is left unchanged and + environment variables are ignored, the LC_CTYPE locale is left unchanged and no signal handler is registered. The :c:func:`Py_RunMain` function can be used to write a customized Python @@ -1316,7 +1316,7 @@ Isolated Configuration isolate Python from the system. For example, to embed Python into an application. -This configuration ignores global configuration variables, environments +This configuration ignores global configuration variables, environment variables, command line arguments (:c:member:`PyConfig.argv` is not parsed) and user site directory. The C standard streams (ex: ``stdout``) and the LC_CTYPE locale are left unchanged. Signal handlers are not installed. @@ -1460,7 +1460,7 @@ Multi-Phase Initialization Private Provisional API ================================================== This section is a private provisional API introducing multi-phase -initialization, the core feature of the :pep:`432`: +initialization, the core feature of :pep:`432`: * "Core" initialization phase, "bare minimum Python": From webhook-mailer at python.org Sat Apr 2 15:52:15 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Sat, 02 Apr 2022 19:52:15 -0000 Subject: [Python-checkins] Document func parameter of locale.atof (GH-18183) Message-ID: https://github.com/python/cpython/commit/208da6d508bb2683732151f4ae288dfc8001267c commit: 208da6d508bb2683732151f4ae288dfc8001267c branch: main author: Kevin Locke committer: JelleZijlstra date: 2022-04-02T12:52:05-07:00 summary: Document func parameter of locale.atof (GH-18183) The second parameter (named `func`) has been present since the `locale` module was introduced in eef1d4e8b1, but has never been documented. This commit updates the documentation for `locale.atof` to clarify the behavior of the function and how the `func` parameter is used. Signed-off-by: Kevin Locke files: M Doc/library/locale.rst diff --git a/Doc/library/locale.rst b/Doc/library/locale.rst index 1b147342cef14..01e14a151d299 100644 --- a/Doc/library/locale.rst +++ b/Doc/library/locale.rst @@ -437,10 +437,10 @@ The :mod:`locale` module defines the following exception and functions: .. versionadded:: 3.10 -.. function:: atof(string) +.. function:: atof(string, func=float) - Converts a string to a floating point number, following the :const:`LC_NUMERIC` - settings. + Converts a string to a number, following the :const:`LC_NUMERIC` settings, + by calling *func* on the result of calling :func:`delocalize` on *string*. .. function:: atoi(string) From webhook-mailer at python.org Sat Apr 2 15:54:08 2022 From: webhook-mailer at python.org (miss-islington) Date: Sat, 02 Apr 2022 19:54:08 -0000 Subject: [Python-checkins] More minor fixes to C API docs (GH-31714) Message-ID: https://github.com/python/cpython/commit/5830a288abc00fc4fca57ebe18e42dc78334a279 commit: 5830a288abc00fc4fca57ebe18e42dc78334a279 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-02T12:54:04-07:00 summary: More minor fixes to C API docs (GH-31714) * init_config: wording fixes * bytearray: remove XXX, there is a good link to the buffer docs * bytes, call, exceptions: minor wording fixes (cherry picked from commit 677a87946630c5fbd9998969669b4dd4f4b32545) Co-authored-by: Jelle Zijlstra files: M Doc/c-api/bytearray.rst M Doc/c-api/bytes.rst M Doc/c-api/call.rst M Doc/c-api/exceptions.rst M Doc/c-api/init_config.rst diff --git a/Doc/c-api/bytearray.rst b/Doc/c-api/bytearray.rst index 30bcfc7cf9f50..85a7d1373c2a8 100644 --- a/Doc/c-api/bytearray.rst +++ b/Doc/c-api/bytearray.rst @@ -42,8 +42,6 @@ Direct API functions Return a new bytearray object from any object, *o*, that implements the :ref:`buffer protocol `. - .. XXX expand about the buffer protocol, at least somewhere - .. c:function:: PyObject* PyByteArray_FromStringAndSize(const char *string, Py_ssize_t len) diff --git a/Doc/c-api/bytes.rst b/Doc/c-api/bytes.rst index de65701037a7c..32c7b80dc7ac6 100644 --- a/Doc/c-api/bytes.rst +++ b/Doc/c-api/bytes.rst @@ -5,7 +5,7 @@ Bytes Objects ------------- -These functions raise :exc:`TypeError` when expecting a bytes parameter and are +These functions raise :exc:`TypeError` when expecting a bytes parameter and called with a non-bytes parameter. .. index:: object: bytes diff --git a/Doc/c-api/call.rst b/Doc/c-api/call.rst index 739b5e97d1515..cdf72bc1f4714 100644 --- a/Doc/c-api/call.rst +++ b/Doc/c-api/call.rst @@ -26,7 +26,7 @@ This convention is not only used by *tp_call*: :c:member:`~PyTypeObject.tp_new` and :c:member:`~PyTypeObject.tp_init` also pass arguments this way. -To call an object, use :c:func:`PyObject_Call` or other +To call an object, use :c:func:`PyObject_Call` or another :ref:`call API `. diff --git a/Doc/c-api/exceptions.rst b/Doc/c-api/exceptions.rst index bcd9621681c2e..7deef83b2789a 100644 --- a/Doc/c-api/exceptions.rst +++ b/Doc/c-api/exceptions.rst @@ -281,7 +281,7 @@ For convenience, some of these functions will always return a .. c:function:: void PyErr_SyntaxLocation(const char *filename, int lineno) - Like :c:func:`PyErr_SyntaxLocationEx`, but the col_offset parameter is + Like :c:func:`PyErr_SyntaxLocationEx`, but the *col_offset* parameter is omitted. @@ -333,7 +333,7 @@ an error value). Issue a warning message with explicit control over all warning attributes. This is a straightforward wrapper around the Python function - :func:`warnings.warn_explicit`, see there for more information. The *module* + :func:`warnings.warn_explicit`; see there for more information. The *module* and *registry* arguments may be set to ``NULL`` to get the default effect described there. @@ -441,7 +441,7 @@ Querying the error indicator error indicator. -.. c:function:: void PyErr_NormalizeException(PyObject**exc, PyObject**val, PyObject**tb) +.. c:function:: void PyErr_NormalizeException(PyObject **exc, PyObject **val, PyObject **tb) Under certain circumstances, the values returned by :c:func:`PyErr_Fetch` below can be "unnormalized", meaning that ``*exc`` is a class object but ``*val`` is diff --git a/Doc/c-api/init_config.rst b/Doc/c-api/init_config.rst index b8b41510b89ec..657d8e9caf1ea 100644 --- a/Doc/c-api/init_config.rst +++ b/Doc/c-api/init_config.rst @@ -16,12 +16,12 @@ There are two kinds of configuration: * The :ref:`Python Configuration ` can be used to build a customized Python which behaves as the regular Python. For example, - environments variables and command line arguments are used to configure + environment variables and command line arguments are used to configure Python. * The :ref:`Isolated Configuration ` can be used to embed Python into an application. It isolates Python from the system. For example, - environments variables are ignored, the LC_CTYPE locale is left unchanged and + environment variables are ignored, the LC_CTYPE locale is left unchanged and no signal handler is registered. The :c:func:`Py_RunMain` function can be used to write a customized Python @@ -1287,7 +1287,7 @@ Isolated Configuration isolate Python from the system. For example, to embed Python into an application. -This configuration ignores global configuration variables, environments +This configuration ignores global configuration variables, environment variables, command line arguments (:c:member:`PyConfig.argv` is not parsed) and user site directory. The C standard streams (ex: ``stdout``) and the LC_CTYPE locale are left unchanged. Signal handlers are not installed. @@ -1432,7 +1432,7 @@ Multi-Phase Initialization Private Provisional API ================================================== This section is a private provisional API introducing multi-phase -initialization, the core feature of the :pep:`432`: +initialization, the core feature of :pep:`432`: * "Core" initialization phase, "bare minimum Python": From webhook-mailer at python.org Sat Apr 2 15:58:08 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Sat, 02 Apr 2022 19:58:08 -0000 Subject: [Python-checkins] bpo-47031: Improve documentation for `math.nan` (GH-32170) Message-ID: https://github.com/python/cpython/commit/182e93c3f57b0c72e765c9896066d32e461c0865 commit: 182e93c3f57b0c72e765c9896066d32e461c0865 branch: main author: Charlie Zhao committer: JelleZijlstra date: 2022-04-02T12:58:03-07:00 summary: bpo-47031: Improve documentation for `math.nan` (GH-32170) Co-authored-by: Jelle Zijlstra files: M Doc/library/math.rst diff --git a/Doc/library/math.rst b/Doc/library/math.rst index bcbcdef51d3fc..c0f9614231912 100644 --- a/Doc/library/math.rst +++ b/Doc/library/math.rst @@ -646,8 +646,23 @@ Constants .. data:: nan - A floating-point "not a number" (NaN) value. Equivalent to the output of - ``float('nan')``. + A floating-point "not a number" (NaN) value. Equivalent to the output of + ``float('nan')``. Due to the requirements of the `IEEE-754 standard + `_, ``math.nan`` and ``float('nan')`` are + not considered to equal to any other numeric value, including themselves. To check + whether a number is a NaN, use the :func:`isnan` function to test + for NaNs instead of ``is`` or ``==``. + Example:: + + >>> import math + >>> math.nan == math.nan + False + >>> float('nan') == float('nan') + False + >>> math.isnan(math.nan) + True + >>> math.isnan(float('nan')) + True .. versionchanged:: 3.11 It is now always available. From webhook-mailer at python.org Sat Apr 2 16:19:24 2022 From: webhook-mailer at python.org (miss-islington) Date: Sat, 02 Apr 2022 20:19:24 -0000 Subject: [Python-checkins] bpo-47031: Improve documentation for `math.nan` (GH-32170) Message-ID: https://github.com/python/cpython/commit/319a70cf99c9866c7fa47deecf04f6ebcfe35a54 commit: 319a70cf99c9866c7fa47deecf04f6ebcfe35a54 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-02T13:19:20-07:00 summary: bpo-47031: Improve documentation for `math.nan` (GH-32170) Co-authored-by: Jelle Zijlstra (cherry picked from commit 182e93c3f57b0c72e765c9896066d32e461c0865) Co-authored-by: Charlie Zhao files: M Doc/library/math.rst diff --git a/Doc/library/math.rst b/Doc/library/math.rst index 145bac4966e18..d485ae5c2706a 100644 --- a/Doc/library/math.rst +++ b/Doc/library/math.rst @@ -627,8 +627,23 @@ Constants .. data:: nan - A floating-point "not a number" (NaN) value. Equivalent to the output of - ``float('nan')``. + A floating-point "not a number" (NaN) value. Equivalent to the output of + ``float('nan')``. Due to the requirements of the `IEEE-754 standard + `_, ``math.nan`` and ``float('nan')`` are + not considered to equal to any other numeric value, including themselves. To check + whether a number is a NaN, use the :func:`isnan` function to test + for NaNs instead of ``is`` or ``==``. + Example:: + + >>> import math + >>> math.nan == math.nan + False + >>> float('nan') == float('nan') + False + >>> math.isnan(math.nan) + True + >>> math.isnan(float('nan')) + True .. versionadded:: 3.5 From webhook-mailer at python.org Sat Apr 2 16:23:30 2022 From: webhook-mailer at python.org (miss-islington) Date: Sat, 02 Apr 2022 20:23:30 -0000 Subject: [Python-checkins] bpo-47031: Improve documentation for `math.nan` (GH-32170) Message-ID: https://github.com/python/cpython/commit/5b80031fb0d2ea14f0d42a33309ce5464c4a6042 commit: 5b80031fb0d2ea14f0d42a33309ce5464c4a6042 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-02T13:23:26-07:00 summary: bpo-47031: Improve documentation for `math.nan` (GH-32170) Co-authored-by: Jelle Zijlstra (cherry picked from commit 182e93c3f57b0c72e765c9896066d32e461c0865) Co-authored-by: Charlie Zhao files: M Doc/library/math.rst diff --git a/Doc/library/math.rst b/Doc/library/math.rst index b20e557b5c610..9783e9ef825d5 100644 --- a/Doc/library/math.rst +++ b/Doc/library/math.rst @@ -622,8 +622,23 @@ Constants .. data:: nan - A floating-point "not a number" (NaN) value. Equivalent to the output of - ``float('nan')``. + A floating-point "not a number" (NaN) value. Equivalent to the output of + ``float('nan')``. Due to the requirements of the `IEEE-754 standard + `_, ``math.nan`` and ``float('nan')`` are + not considered to equal to any other numeric value, including themselves. To check + whether a number is a NaN, use the :func:`isnan` function to test + for NaNs instead of ``is`` or ``==``. + Example:: + + >>> import math + >>> math.nan == math.nan + False + >>> float('nan') == float('nan') + False + >>> math.isnan(math.nan) + True + >>> math.isnan(float('nan')) + True .. versionadded:: 3.5 From webhook-mailer at python.org Sat Apr 2 16:58:35 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Sat, 02 Apr 2022 20:58:35 -0000 Subject: [Python-checkins] bpo-45114: Use lstat() instead of stat() in stat docs example (GH-29845) Message-ID: https://github.com/python/cpython/commit/c93a0ac6972221787d8bea1c41a9feb667ed3d2c commit: c93a0ac6972221787d8bea1c41a9feb667ed3d2c branch: main author: 180909 committer: JelleZijlstra date: 2022-04-02T13:58:26-07:00 summary: bpo-45114: Use lstat() instead of stat() in stat docs example (GH-29845) files: M Doc/library/stat.rst diff --git a/Doc/library/stat.rst b/Doc/library/stat.rst index 98219eaee9761..083dc5e3bcfd6 100644 --- a/Doc/library/stat.rst +++ b/Doc/library/stat.rst @@ -109,7 +109,7 @@ Example:: for f in os.listdir(top): pathname = os.path.join(top, f) - mode = os.stat(pathname).st_mode + mode = os.lstat(pathname).st_mode if S_ISDIR(mode): # It's a directory, recurse into it walktree(pathname, callback) From webhook-mailer at python.org Sat Apr 2 17:11:47 2022 From: webhook-mailer at python.org (tiran) Date: Sat, 02 Apr 2022 21:11:47 -0000 Subject: [Python-checkins] bpo-46315: Use fopencookie only on Emscripten 3.x and newer (GH-32266) Message-ID: https://github.com/python/cpython/commit/3df0e63aabef905b72fad78256f24b9270c63172 commit: 3df0e63aabef905b72fad78256f24b9270c63172 branch: main author: Christian Heimes committer: tiran date: 2022-04-02T23:11:38+02:00 summary: bpo-46315: Use fopencookie only on Emscripten 3.x and newer (GH-32266) files: M Parser/tokenizer.c diff --git a/Parser/tokenizer.c b/Parser/tokenizer.c index 13116d052ea59..db84e2e92e167 100644 --- a/Parser/tokenizer.c +++ b/Parser/tokenizer.c @@ -2072,7 +2072,7 @@ _PyTokenizer_Get(struct tok_state *tok, return result; } -#if defined(__wasi__) || defined(__EMSCRIPTEN__) +#if defined(__wasi__) || (defined(__EMSCRIPTEN__) && (__EMSCRIPTEN_major__ >= 3)) // fdopen() with borrowed fd. WASI does not provide dup() and Emscripten's // dup() emulation with open() is slow. typedef union { From webhook-mailer at python.org Sat Apr 2 17:12:46 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Sat, 02 Apr 2022 21:12:46 -0000 Subject: [Python-checkins] codecs docs: fix grammar mistake (GH-29462) Message-ID: https://github.com/python/cpython/commit/ea56845744e815ed468dfbdd835110254c3be997 commit: ea56845744e815ed468dfbdd835110254c3be997 branch: main author: 180909 committer: JelleZijlstra date: 2022-04-02T14:12:32-07:00 summary: codecs docs: fix grammar mistake (GH-29462) files: M Doc/library/codecs.rst diff --git a/Doc/library/codecs.rst b/Doc/library/codecs.rst index 76710974dd427..e6f5954d33b59 100644 --- a/Doc/library/codecs.rst +++ b/Doc/library/codecs.rst @@ -915,7 +915,7 @@ there's the so called BOM ("Byte Order Mark"). This is the Unicode character ``U+FEFF``. This character can be prepended to every ``UTF-16`` or ``UTF-32`` byte sequence. The byte swapped version of this character (``0xFFFE``) is an illegal character that may not appear in a Unicode text. So when the -first character in an ``UTF-16`` or ``UTF-32`` byte sequence +first character in a ``UTF-16`` or ``UTF-32`` byte sequence appears to be a ``U+FFFE`` the bytes have to be swapped on decoding. Unfortunately the character ``U+FEFF`` had a second purpose as a ``ZERO WIDTH NO-BREAK SPACE``: a character that has no width and doesn't allow From webhook-mailer at python.org Sat Apr 2 17:19:52 2022 From: webhook-mailer at python.org (miss-islington) Date: Sat, 02 Apr 2022 21:19:52 -0000 Subject: [Python-checkins] bpo-45114: Use lstat() instead of stat() in stat docs example (GH-29845) Message-ID: https://github.com/python/cpython/commit/23c0200c436714a3f9b38d5e6cac230ceacd4c43 commit: 23c0200c436714a3f9b38d5e6cac230ceacd4c43 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-02T14:19:48-07:00 summary: bpo-45114: Use lstat() instead of stat() in stat docs example (GH-29845) (cherry picked from commit c93a0ac6972221787d8bea1c41a9feb667ed3d2c) Co-authored-by: 180909 files: M Doc/library/stat.rst diff --git a/Doc/library/stat.rst b/Doc/library/stat.rst index 98219eaee9761..083dc5e3bcfd6 100644 --- a/Doc/library/stat.rst +++ b/Doc/library/stat.rst @@ -109,7 +109,7 @@ Example:: for f in os.listdir(top): pathname = os.path.join(top, f) - mode = os.stat(pathname).st_mode + mode = os.lstat(pathname).st_mode if S_ISDIR(mode): # It's a directory, recurse into it walktree(pathname, callback) From webhook-mailer at python.org Sat Apr 2 17:21:57 2022 From: webhook-mailer at python.org (miss-islington) Date: Sat, 02 Apr 2022 21:21:57 -0000 Subject: [Python-checkins] bpo-45114: Use lstat() instead of stat() in stat docs example (GH-29845) Message-ID: https://github.com/python/cpython/commit/dc9322a91d48c80afe8329bb509673a2f74fb925 commit: dc9322a91d48c80afe8329bb509673a2f74fb925 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-02T14:21:45-07:00 summary: bpo-45114: Use lstat() instead of stat() in stat docs example (GH-29845) (cherry picked from commit c93a0ac6972221787d8bea1c41a9feb667ed3d2c) Co-authored-by: 180909 files: M Doc/library/stat.rst diff --git a/Doc/library/stat.rst b/Doc/library/stat.rst index 98219eaee9761..083dc5e3bcfd6 100644 --- a/Doc/library/stat.rst +++ b/Doc/library/stat.rst @@ -109,7 +109,7 @@ Example:: for f in os.listdir(top): pathname = os.path.join(top, f) - mode = os.stat(pathname).st_mode + mode = os.lstat(pathname).st_mode if S_ISDIR(mode): # It's a directory, recurse into it walktree(pathname, callback) From webhook-mailer at python.org Sat Apr 2 17:34:16 2022 From: webhook-mailer at python.org (miss-islington) Date: Sat, 02 Apr 2022 21:34:16 -0000 Subject: [Python-checkins] codecs docs: fix grammar mistake (GH-29462) Message-ID: https://github.com/python/cpython/commit/2d936a42ac2b45ddc160eb90178cc9b4107639db commit: 2d936a42ac2b45ddc160eb90178cc9b4107639db branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-02T14:33:52-07:00 summary: codecs docs: fix grammar mistake (GH-29462) (cherry picked from commit ea56845744e815ed468dfbdd835110254c3be997) Co-authored-by: 180909 files: M Doc/library/codecs.rst diff --git a/Doc/library/codecs.rst b/Doc/library/codecs.rst index 1a1ce9237b010..9e2eb63479282 100644 --- a/Doc/library/codecs.rst +++ b/Doc/library/codecs.rst @@ -912,7 +912,7 @@ there's the so called BOM ("Byte Order Mark"). This is the Unicode character ``U+FEFF``. This character can be prepended to every ``UTF-16`` or ``UTF-32`` byte sequence. The byte swapped version of this character (``0xFFFE``) is an illegal character that may not appear in a Unicode text. So when the -first character in an ``UTF-16`` or ``UTF-32`` byte sequence +first character in a ``UTF-16`` or ``UTF-32`` byte sequence appears to be a ``U+FFFE`` the bytes have to be swapped on decoding. Unfortunately the character ``U+FEFF`` had a second purpose as a ``ZERO WIDTH NO-BREAK SPACE``: a character that has no width and doesn't allow From webhook-mailer at python.org Sat Apr 2 17:34:22 2022 From: webhook-mailer at python.org (miss-islington) Date: Sat, 02 Apr 2022 21:34:22 -0000 Subject: [Python-checkins] codecs docs: fix grammar mistake (GH-29462) Message-ID: https://github.com/python/cpython/commit/bf0f04f87139bceac1a5f84326d1a3ed3a5c4061 commit: bf0f04f87139bceac1a5f84326d1a3ed3a5c4061 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-02T14:34:18-07:00 summary: codecs docs: fix grammar mistake (GH-29462) (cherry picked from commit ea56845744e815ed468dfbdd835110254c3be997) Co-authored-by: 180909 files: M Doc/library/codecs.rst diff --git a/Doc/library/codecs.rst b/Doc/library/codecs.rst index 3386208d5ef5f..1a71e5791170d 100644 --- a/Doc/library/codecs.rst +++ b/Doc/library/codecs.rst @@ -909,7 +909,7 @@ there's the so called BOM ("Byte Order Mark"). This is the Unicode character ``U+FEFF``. This character can be prepended to every ``UTF-16`` or ``UTF-32`` byte sequence. The byte swapped version of this character (``0xFFFE``) is an illegal character that may not appear in a Unicode text. So when the -first character in an ``UTF-16`` or ``UTF-32`` byte sequence +first character in a ``UTF-16`` or ``UTF-32`` byte sequence appears to be a ``U+FFFE`` the bytes have to be swapped on decoding. Unfortunately the character ``U+FEFF`` had a second purpose as a ``ZERO WIDTH NO-BREAK SPACE``: a character that has no width and doesn't allow From webhook-mailer at python.org Sat Apr 2 17:38:41 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Sat, 02 Apr 2022 21:38:41 -0000 Subject: [Python-checkins] os docs: fix typo (GH-28996) Message-ID: https://github.com/python/cpython/commit/1f80dcd2442f2a354097797fedc077592984903b commit: 1f80dcd2442f2a354097797fedc077592984903b branch: main author: Vitor Buxbaum Orlandi committer: JelleZijlstra date: 2022-04-02T14:38:33-07:00 summary: os docs: fix typo (GH-28996) Co-authored-by: Jacob Walls files: M Doc/library/os.rst diff --git a/Doc/library/os.rst b/Doc/library/os.rst index 05e0303614ed1..60eac2f78572e 100644 --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -113,7 +113,7 @@ of the UTF-8 encoding: :ref:`error handler ` being enabled for :data:`sys.stdin` and :data:`sys.stdout` (:data:`sys.stderr` continues to use ``backslashreplace`` as it does in the default locale-aware mode) -* On Unix, :func:`os.device_encoding` returns ``'UTF-8'``. rather than the +* On Unix, :func:`os.device_encoding` returns ``'UTF-8'`` rather than the device encoding. Note that the standard stream settings in UTF-8 mode can be overridden by From webhook-mailer at python.org Sat Apr 2 18:00:16 2022 From: webhook-mailer at python.org (miss-islington) Date: Sat, 02 Apr 2022 22:00:16 -0000 Subject: [Python-checkins] os docs: fix typo (GH-28996) Message-ID: https://github.com/python/cpython/commit/8d9a75b206d40b096d9a8f9bcc40d95b7687d6eb commit: 8d9a75b206d40b096d9a8f9bcc40d95b7687d6eb branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-02T15:00:12-07:00 summary: os docs: fix typo (GH-28996) Co-authored-by: Jacob Walls (cherry picked from commit 1f80dcd2442f2a354097797fedc077592984903b) Co-authored-by: Vitor Buxbaum Orlandi files: M Doc/library/os.rst diff --git a/Doc/library/os.rst b/Doc/library/os.rst index 2a1ea05c5c833..203995c80fd32 100644 --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -113,7 +113,7 @@ of the UTF-8 encoding: :ref:`error handler ` being enabled for :data:`sys.stdin` and :data:`sys.stdout` (:data:`sys.stderr` continues to use ``backslashreplace`` as it does in the default locale-aware mode) -* On Unix, :func:`os.device_encoding` returns ``'UTF-8'``. rather than the +* On Unix, :func:`os.device_encoding` returns ``'UTF-8'`` rather than the device encoding. Note that the standard stream settings in UTF-8 mode can be overridden by From webhook-mailer at python.org Sat Apr 2 18:11:28 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Sat, 02 Apr 2022 22:11:28 -0000 Subject: [Python-checkins] bpo-45584: Clarify `math.trunc` documentation (GH-29183) Message-ID: https://github.com/python/cpython/commit/ebbdbbff5d6840807e46ec61b8a323e94ee88de2 commit: ebbdbbff5d6840807e46ec61b8a323e94ee88de2 branch: main author: Arthur Milchior committer: JelleZijlstra date: 2022-04-02T15:11:20-07:00 summary: bpo-45584: Clarify `math.trunc` documentation (GH-29183) While floor/ceil 's documentation are very precise, `truncate` was not explained. I actually had to search online to understand the difference between `truncate` and `floor` (admittedly, once I remembered that numbers are signed, and that floating numbers actually uses a bit for negation symbol instead of two complement, it became obvious) Co-authored-by: Alex Waygood Co-authored-by: ?ric Araujo Co-authored-by: Terry Jan Reedy Co-authored-by: Jelle Zijlstra files: M Doc/library/math.rst diff --git a/Doc/library/math.rst b/Doc/library/math.rst index c0f9614231912..48030fae189ba 100644 --- a/Doc/library/math.rst +++ b/Doc/library/math.rst @@ -32,8 +32,8 @@ Number-theoretic and representation functions .. function:: ceil(x) Return the ceiling of *x*, the smallest integer greater than or equal to *x*. - If *x* is not a float, delegates to ``x.__ceil__()``, which should return an - :class:`~numbers.Integral` value. + If *x* is not a float, delegates to :meth:`x.__ceil__ `, + which should return an :class:`~numbers.Integral` value. .. function:: comb(n, k) @@ -77,9 +77,9 @@ Number-theoretic and representation functions .. function:: floor(x) - Return the floor of *x*, the largest integer less than or equal to *x*. - If *x* is not a float, delegates to ``x.__floor__()``, which should return an - :class:`~numbers.Integral` value. + Return the floor of *x*, the largest integer less than or equal to *x*. If + *x* is not a float, delegates to :meth:`x.__floor__ `, which + should return an :class:`~numbers.Integral` value. .. function:: fmod(x, y) @@ -298,9 +298,11 @@ Number-theoretic and representation functions .. function:: trunc(x) - Return the :class:`~numbers.Real` value *x* truncated to an - :class:`~numbers.Integral` (usually an integer). Delegates to - :meth:`x.__trunc__() `. + Return *x* with the fractional part + removed, leaving the integer part. This rounds toward 0: ``trunc()`` is + equivalent to :func:`floor` for positive *x*, and equivalent to :func:`ceil` + for negative *x*. If *x* is not a float, delegates to :meth:`x.__trunc__ + `, which should return an :class:`~numbers.Integral` value. .. function:: ulp(x) From webhook-mailer at python.org Sat Apr 2 18:31:17 2022 From: webhook-mailer at python.org (miss-islington) Date: Sat, 02 Apr 2022 22:31:17 -0000 Subject: [Python-checkins] bpo-45584: Clarify `math.trunc` documentation (GH-29183) Message-ID: https://github.com/python/cpython/commit/3031b867531009d270d5d7d3e53e739c4cba8816 commit: 3031b867531009d270d5d7d3e53e739c4cba8816 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-02T15:30:59-07:00 summary: bpo-45584: Clarify `math.trunc` documentation (GH-29183) While floor/ceil 's documentation are very precise, `truncate` was not explained. I actually had to search online to understand the difference between `truncate` and `floor` (admittedly, once I remembered that numbers are signed, and that floating numbers actually uses a bit for negation symbol instead of two complement, it became obvious) Co-authored-by: Alex Waygood Co-authored-by: ?ric Araujo Co-authored-by: Terry Jan Reedy Co-authored-by: Jelle Zijlstra (cherry picked from commit ebbdbbff5d6840807e46ec61b8a323e94ee88de2) Co-authored-by: Arthur Milchior files: M Doc/library/math.rst diff --git a/Doc/library/math.rst b/Doc/library/math.rst index d485ae5c2706a..7ba5fa4a0b4e7 100644 --- a/Doc/library/math.rst +++ b/Doc/library/math.rst @@ -32,8 +32,8 @@ Number-theoretic and representation functions .. function:: ceil(x) Return the ceiling of *x*, the smallest integer greater than or equal to *x*. - If *x* is not a float, delegates to ``x.__ceil__()``, which should return an - :class:`~numbers.Integral` value. + If *x* is not a float, delegates to :meth:`x.__ceil__ `, + which should return an :class:`~numbers.Integral` value. .. function:: comb(n, k) @@ -77,9 +77,9 @@ Number-theoretic and representation functions .. function:: floor(x) - Return the floor of *x*, the largest integer less than or equal to *x*. - If *x* is not a float, delegates to ``x.__floor__()``, which should return an - :class:`~numbers.Integral` value. + Return the floor of *x*, the largest integer less than or equal to *x*. If + *x* is not a float, delegates to :meth:`x.__floor__ `, which + should return an :class:`~numbers.Integral` value. .. function:: fmod(x, y) @@ -298,9 +298,11 @@ Number-theoretic and representation functions .. function:: trunc(x) - Return the :class:`~numbers.Real` value *x* truncated to an - :class:`~numbers.Integral` (usually an integer). Delegates to - :meth:`x.__trunc__() `. + Return *x* with the fractional part + removed, leaving the integer part. This rounds toward 0: ``trunc()`` is + equivalent to :func:`floor` for positive *x*, and equivalent to :func:`ceil` + for negative *x*. If *x* is not a float, delegates to :meth:`x.__trunc__ + `, which should return an :class:`~numbers.Integral` value. .. function:: ulp(x) From webhook-mailer at python.org Sat Apr 2 18:36:49 2022 From: webhook-mailer at python.org (miss-islington) Date: Sat, 02 Apr 2022 22:36:49 -0000 Subject: [Python-checkins] bpo-45584: Clarify `math.trunc` documentation (GH-29183) Message-ID: https://github.com/python/cpython/commit/694425817ba2b3a796acb3413a7111f6de4bd086 commit: 694425817ba2b3a796acb3413a7111f6de4bd086 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-02T15:36:33-07:00 summary: bpo-45584: Clarify `math.trunc` documentation (GH-29183) While floor/ceil 's documentation are very precise, `truncate` was not explained. I actually had to search online to understand the difference between `truncate` and `floor` (admittedly, once I remembered that numbers are signed, and that floating numbers actually uses a bit for negation symbol instead of two complement, it became obvious) Co-authored-by: Alex Waygood Co-authored-by: ?ric Araujo Co-authored-by: Terry Jan Reedy Co-authored-by: Jelle Zijlstra (cherry picked from commit ebbdbbff5d6840807e46ec61b8a323e94ee88de2) Co-authored-by: Arthur Milchior files: M Doc/library/math.rst diff --git a/Doc/library/math.rst b/Doc/library/math.rst index 9783e9ef825d50..ad9116b63e6066 100644 --- a/Doc/library/math.rst +++ b/Doc/library/math.rst @@ -32,8 +32,8 @@ Number-theoretic and representation functions .. function:: ceil(x) Return the ceiling of *x*, the smallest integer greater than or equal to *x*. - If *x* is not a float, delegates to ``x.__ceil__()``, which should return an - :class:`~numbers.Integral` value. + If *x* is not a float, delegates to :meth:`x.__ceil__ `, + which should return an :class:`~numbers.Integral` value. .. function:: comb(n, k) @@ -77,9 +77,9 @@ Number-theoretic and representation functions .. function:: floor(x) - Return the floor of *x*, the largest integer less than or equal to *x*. - If *x* is not a float, delegates to ``x.__floor__()``, which should return an - :class:`~numbers.Integral` value. + Return the floor of *x*, the largest integer less than or equal to *x*. If + *x* is not a float, delegates to :meth:`x.__floor__ `, which + should return an :class:`~numbers.Integral` value. .. function:: fmod(x, y) @@ -298,9 +298,11 @@ Number-theoretic and representation functions .. function:: trunc(x) - Return the :class:`~numbers.Real` value *x* truncated to an - :class:`~numbers.Integral` (usually an integer). Delegates to - :meth:`x.__trunc__() `. + Return *x* with the fractional part + removed, leaving the integer part. This rounds toward 0: ``trunc()`` is + equivalent to :func:`floor` for positive *x*, and equivalent to :func:`ceil` + for negative *x*. If *x* is not a float, delegates to :meth:`x.__trunc__ + `, which should return an :class:`~numbers.Integral` value. .. function:: ulp(x) From webhook-mailer at python.org Sat Apr 2 18:59:46 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Sat, 02 Apr 2022 22:59:46 -0000 Subject: [Python-checkins] [3.10] Document func parameter of locale.atof (GH-18183) (GH-32262) Message-ID: https://github.com/python/cpython/commit/a930062d257b066705c9a681aa6c7f0f342dea11 commit: a930062d257b066705c9a681aa6c7f0f342dea11 branch: 3.10 author: Jelle Zijlstra committer: JelleZijlstra date: 2022-04-02T15:59:41-07:00 summary: [3.10] Document func parameter of locale.atof (GH-18183) (GH-32262) The second parameter (named `func`) has been present since the `locale` module was introduced in eef1d4e8b1, but has never been documented. This commit updates the documentation for `locale.atof` to clarify the behavior of the function and how the `func` parameter is used. Signed-off-by: Kevin Locke (cherry picked from commit 208da6d508bb2683732151f4ae288dfc8001267c) Co-authored-by: Kevin Locke files: M Doc/library/locale.rst diff --git a/Doc/library/locale.rst b/Doc/library/locale.rst index 60d0c59d017c7..dd14a379b0c6c 100644 --- a/Doc/library/locale.rst +++ b/Doc/library/locale.rst @@ -435,10 +435,10 @@ The :mod:`locale` module defines the following exception and functions: .. versionadded:: 3.10 -.. function:: atof(string) +.. function:: atof(string, func=float) - Converts a string to a floating point number, following the :const:`LC_NUMERIC` - settings. + Converts a string to a number, following the :const:`LC_NUMERIC` settings, + by calling *func* on the result of calling :func:`delocalize` on *string*. .. function:: atoi(string) From webhook-mailer at python.org Sat Apr 2 19:00:02 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Sat, 02 Apr 2022 23:00:02 -0000 Subject: [Python-checkins] [3.9] Document func parameter of locale.atof (GH-18183) (GH-32261) Message-ID: https://github.com/python/cpython/commit/62da258a455974014ddc3a40466724bb815f862d commit: 62da258a455974014ddc3a40466724bb815f862d branch: 3.9 author: Jelle Zijlstra committer: JelleZijlstra date: 2022-04-02T15:59:57-07:00 summary: [3.9] Document func parameter of locale.atof (GH-18183) (GH-32261) The second parameter (named `func`) has been present since the `locale` module was introduced in eef1d4e8b1, but has never been documented. This commit updates the documentation for `locale.atof` to clarify the behavior of the function and how the `func` parameter is used. Signed-off-by: Kevin Locke . (cherry picked from commit 208da6d508bb2683732151f4ae288dfc8001267c) Co-authored-by: Kevin Locke files: M Doc/library/locale.rst diff --git a/Doc/library/locale.rst b/Doc/library/locale.rst index bf57a08355916..5b5a0c7552323 100644 --- a/Doc/library/locale.rst +++ b/Doc/library/locale.rst @@ -423,10 +423,10 @@ The :mod:`locale` module defines the following exception and functions: .. versionadded:: 3.5 -.. function:: atof(string) +.. function:: atof(string, func=float) - Converts a string to a floating point number, following the :const:`LC_NUMERIC` - settings. + Converts a string to a number, following the :const:`LC_NUMERIC` settings, + by calling *func* on the result of calling :func:`delocalize` on *string*. .. function:: atoi(string) From webhook-mailer at python.org Sat Apr 2 19:00:18 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Sat, 02 Apr 2022 23:00:18 -0000 Subject: [Python-checkins] [3.9] More minor fixes to C API docs (GH-31714) (GH-32260) Message-ID: https://github.com/python/cpython/commit/74138ac954f6ee04aefcfbb96240a5185bd944c9 commit: 74138ac954f6ee04aefcfbb96240a5185bd944c9 branch: 3.9 author: Jelle Zijlstra committer: JelleZijlstra date: 2022-04-02T16:00:13-07:00 summary: [3.9] More minor fixes to C API docs (GH-31714) (GH-32260) * init_config: wording fixes * bytearray: remove XXX, there is a good link to the buffer docs * bytes, call, exceptions: minor wording fixes. (cherry picked from commit 677a87946630c5fbd9998969669b4dd4f4b32545) Co-authored-by: Jelle Zijlstra files: M Doc/c-api/bytearray.rst M Doc/c-api/bytes.rst M Doc/c-api/call.rst M Doc/c-api/exceptions.rst M Doc/c-api/init_config.rst diff --git a/Doc/c-api/bytearray.rst b/Doc/c-api/bytearray.rst index 30bcfc7cf9f50..85a7d1373c2a8 100644 --- a/Doc/c-api/bytearray.rst +++ b/Doc/c-api/bytearray.rst @@ -42,8 +42,6 @@ Direct API functions Return a new bytearray object from any object, *o*, that implements the :ref:`buffer protocol `. - .. XXX expand about the buffer protocol, at least somewhere - .. c:function:: PyObject* PyByteArray_FromStringAndSize(const char *string, Py_ssize_t len) diff --git a/Doc/c-api/bytes.rst b/Doc/c-api/bytes.rst index de65701037a7c..32c7b80dc7ac6 100644 --- a/Doc/c-api/bytes.rst +++ b/Doc/c-api/bytes.rst @@ -5,7 +5,7 @@ Bytes Objects ------------- -These functions raise :exc:`TypeError` when expecting a bytes parameter and are +These functions raise :exc:`TypeError` when expecting a bytes parameter and called with a non-bytes parameter. .. index:: object: bytes diff --git a/Doc/c-api/call.rst b/Doc/c-api/call.rst index 739b5e97d1515..cdf72bc1f4714 100644 --- a/Doc/c-api/call.rst +++ b/Doc/c-api/call.rst @@ -26,7 +26,7 @@ This convention is not only used by *tp_call*: :c:member:`~PyTypeObject.tp_new` and :c:member:`~PyTypeObject.tp_init` also pass arguments this way. -To call an object, use :c:func:`PyObject_Call` or other +To call an object, use :c:func:`PyObject_Call` or another :ref:`call API `. diff --git a/Doc/c-api/exceptions.rst b/Doc/c-api/exceptions.rst index df38ba43ec431..646dc5a543473 100644 --- a/Doc/c-api/exceptions.rst +++ b/Doc/c-api/exceptions.rst @@ -281,7 +281,7 @@ For convenience, some of these functions will always return a .. c:function:: void PyErr_SyntaxLocation(const char *filename, int lineno) - Like :c:func:`PyErr_SyntaxLocationEx`, but the col_offset parameter is + Like :c:func:`PyErr_SyntaxLocationEx`, but the *col_offset* parameter is omitted. @@ -333,7 +333,7 @@ an error value). Issue a warning message with explicit control over all warning attributes. This is a straightforward wrapper around the Python function - :func:`warnings.warn_explicit`, see there for more information. The *module* + :func:`warnings.warn_explicit`; see there for more information. The *module* and *registry* arguments may be set to ``NULL`` to get the default effect described there. @@ -441,7 +441,7 @@ Querying the error indicator error indicator. -.. c:function:: void PyErr_NormalizeException(PyObject**exc, PyObject**val, PyObject**tb) +.. c:function:: void PyErr_NormalizeException(PyObject **exc, PyObject **val, PyObject **tb) Under certain circumstances, the values returned by :c:func:`PyErr_Fetch` below can be "unnormalized", meaning that ``*exc`` is a class object but ``*val`` is diff --git a/Doc/c-api/init_config.rst b/Doc/c-api/init_config.rst index 23009e4191f88..f44c7bb5cd5a8 100644 --- a/Doc/c-api/init_config.rst +++ b/Doc/c-api/init_config.rst @@ -823,7 +823,7 @@ Isolated Configuration isolate Python from the system. For example, to embed Python into an application. -This configuration ignores global configuration variables, environments +This configuration ignores global configuration variables, environment variables, command line arguments (:c:member:`PyConfig.argv` is not parsed) and user site directory. The C standard streams (ex: ``stdout``) and the LC_CTYPE locale are left unchanged. Signal handlers are not installed. @@ -1000,7 +1000,7 @@ Multi-Phase Initialization Private Provisional API -------------------------------------------------- This section is a private provisional API introducing multi-phase -initialization, the core feature of the :pep:`432`: +initialization, the core feature of :pep:`432`: * "Core" initialization phase, "bare minimum Python": From webhook-mailer at python.org Sat Apr 2 19:00:40 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Sat, 02 Apr 2022 23:00:40 -0000 Subject: [Python-checkins] [3.9] More minor fixes to C API docs (GH-31525) (GH-32259) Message-ID: https://github.com/python/cpython/commit/6eff004b96ae9cdca1b36600fe1098498228f66d commit: 6eff004b96ae9cdca1b36600fe1098498228f66d branch: 3.9 author: Jelle Zijlstra committer: JelleZijlstra date: 2022-04-02T16:00:36-07:00 summary: [3.9] More minor fixes to C API docs (GH-31525) (GH-32259) * wording fixes in type.rst * grammar and punctuation in sys.rst * set: grammar fixes * structures: capitalization fix * grammar fixes for sequence * objects: point to Py_TYPE instead of direct object access * numbers: add more explicit Python equivalences * method: add missing period * memory: grammar fix * mapping: grammar fixes * long: grammar fix * iter: fix grammar for PyAIter_Check * init: grammar fix. (cherry picked from commit 897bc6f9282238d5fb32d232ab62d30675244736) Co-authored-by: Jelle Zijlstra files: M Doc/c-api/init.rst M Doc/c-api/long.rst M Doc/c-api/mapping.rst M Doc/c-api/method.rst M Doc/c-api/number.rst M Doc/c-api/object.rst M Doc/c-api/sequence.rst M Doc/c-api/set.rst M Doc/c-api/structures.rst M Doc/c-api/sys.rst M Doc/c-api/type.rst diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst index 9d7876e055981..8f1382e008c1e 100644 --- a/Doc/c-api/init.rst +++ b/Doc/c-api/init.rst @@ -1688,7 +1688,7 @@ is not possible due to its implementation being opaque at build time. argument is `NULL`. .. note:: - A freed key becomes a dangling pointer, you should reset the key to + A freed key becomes a dangling pointer. You should reset the key to `NULL`. diff --git a/Doc/c-api/long.rst b/Doc/c-api/long.rst index 60e1791df9a45..95796e0fa506a 100644 --- a/Doc/c-api/long.rst +++ b/Doc/c-api/long.rst @@ -41,7 +41,7 @@ distinguished from a number. Use :c:func:`PyErr_Occurred` to disambiguate. Return a new :c:type:`PyLongObject` object from *v*, or ``NULL`` on failure. The current implementation keeps an array of integer objects for all integers - between ``-5`` and ``256``, when you create an int in that range you actually + between ``-5`` and ``256``. When you create an int in that range you actually just get back a reference to the existing object. diff --git a/Doc/c-api/mapping.rst b/Doc/c-api/mapping.rst index 682160d1475c1..3c9d282c6d0ab 100644 --- a/Doc/c-api/mapping.rst +++ b/Doc/c-api/mapping.rst @@ -11,10 +11,10 @@ See also :c:func:`PyObject_GetItem`, :c:func:`PyObject_SetItem` and .. c:function:: int PyMapping_Check(PyObject *o) - Return ``1`` if the object provides mapping protocol or supports slicing, + Return ``1`` if the object provides the mapping protocol or supports slicing, and ``0`` otherwise. Note that it returns ``1`` for Python classes with - a :meth:`__getitem__` method since in general case it is impossible to - determine what type of keys it supports. This function always succeeds. + a :meth:`__getitem__` method, since in general it is impossible to + determine what type of keys the class supports. This function always succeeds. .. c:function:: Py_ssize_t PyMapping_Size(PyObject *o) diff --git a/Doc/c-api/method.rst b/Doc/c-api/method.rst index 23852251dfe02..6e7e1e21aa93f 100644 --- a/Doc/c-api/method.rst +++ b/Doc/c-api/method.rst @@ -27,7 +27,7 @@ to bind a :c:data:`PyCFunction` to a class object. It replaces the former call .. c:function:: PyObject* PyInstanceMethod_New(PyObject *func) - Return a new instance method object, with *func* being any callable object + Return a new instance method object, with *func* being any callable object. *func* is the function that will be called when the instance method is called. diff --git a/Doc/c-api/number.rst b/Doc/c-api/number.rst index 620204ca8e229..1058b2c77f308 100644 --- a/Doc/c-api/number.rst +++ b/Doc/c-api/number.rst @@ -44,7 +44,7 @@ Number Protocol .. c:function:: PyObject* PyNumber_FloorDivide(PyObject *o1, PyObject *o2) Return the floor of *o1* divided by *o2*, or ``NULL`` on failure. This is - equivalent to the "classic" division of integers. + the equivalent of the Python expression ``o1 // o2``. .. c:function:: PyObject* PyNumber_TrueDivide(PyObject *o1, PyObject *o2) @@ -53,7 +53,7 @@ Number Protocol *o2*, or ``NULL`` on failure. The return value is "approximate" because binary floating point numbers are approximate; it is not possible to represent all real numbers in base two. This function can return a floating point value when - passed two integers. + passed two integers. This is the equivalent of the Python expression ``o1 / o2``. .. c:function:: PyObject* PyNumber_Remainder(PyObject *o1, PyObject *o2) @@ -180,6 +180,7 @@ Number Protocol floating point numbers are approximate; it is not possible to represent all real numbers in base two. This function can return a floating point value when passed two integers. The operation is done *in-place* when *o1* supports it. + This is the equivalent of the Python statement ``o1 /= o2``. .. c:function:: PyObject* PyNumber_InPlaceRemainder(PyObject *o1, PyObject *o2) @@ -281,6 +282,6 @@ Number Protocol .. c:function:: int PyIndex_Check(PyObject *o) - Returns ``1`` if *o* is an index integer (has the nb_index slot of the - tp_as_number structure filled in), and ``0`` otherwise. + Returns ``1`` if *o* is an index integer (has the ``nb_index`` slot of the + ``tp_as_number`` structure filled in), and ``0`` otherwise. This function always succeeds. diff --git a/Doc/c-api/object.rst b/Doc/c-api/object.rst index 9f874382e85bb..43ccb44c35ba0 100644 --- a/Doc/c-api/object.rst +++ b/Doc/c-api/object.rst @@ -93,7 +93,7 @@ Object Protocol return ``0`` on success. This is the equivalent of the Python statement ``o.attr_name = v``. - If *v* is ``NULL``, the attribute is deleted, however this feature is + If *v* is ``NULL``, the attribute is deleted, but this feature is deprecated in favour of using :c:func:`PyObject_DelAttrString`. @@ -291,7 +291,7 @@ Object Protocol of object *o*. On failure, raises :exc:`SystemError` and returns ``NULL``. This is equivalent to the Python expression ``type(o)``. This function increments the reference count of the return value. There's really no reason to use this - function instead of the common expression ``o->ob_type``, which returns a + function instead of the :c:func:`Py_TYPE()` function, which returns a pointer of type :c:type:`PyTypeObject*`, except when the incremented reference count is needed. diff --git a/Doc/c-api/sequence.rst b/Doc/c-api/sequence.rst index 6581885904117..c78d273f9f149 100644 --- a/Doc/c-api/sequence.rst +++ b/Doc/c-api/sequence.rst @@ -8,10 +8,10 @@ Sequence Protocol .. c:function:: int PySequence_Check(PyObject *o) - Return ``1`` if the object provides sequence protocol, and ``0`` otherwise. + Return ``1`` if the object provides the sequence protocol, and ``0`` otherwise. Note that it returns ``1`` for Python classes with a :meth:`__getitem__` - method unless they are :class:`dict` subclasses since in general case it - is impossible to determine what the type of keys it supports. This + method, unless they are :class:`dict` subclasses, since in general it + is impossible to determine what type of keys the class supports. This function always succeeds. @@ -69,7 +69,7 @@ Sequence Protocol is the equivalent of the Python statement ``o[i] = v``. This function *does not* steal a reference to *v*. - If *v* is ``NULL``, the element is deleted, however this feature is + If *v* is ``NULL``, the element is deleted, but this feature is deprecated in favour of using :c:func:`PySequence_DelItem`. @@ -147,7 +147,7 @@ Sequence Protocol Returns the length of *o*, assuming that *o* was returned by :c:func:`PySequence_Fast` and that *o* is not ``NULL``. The size can also be - gotten by calling :c:func:`PySequence_Size` on *o*, but + retrieved by calling :c:func:`PySequence_Size` on *o*, but :c:func:`PySequence_Fast_GET_SIZE` is faster because it can assume *o* is a list or tuple. diff --git a/Doc/c-api/set.rst b/Doc/c-api/set.rst index 84f34e7dae80b..9e3045e796eec 100644 --- a/Doc/c-api/set.rst +++ b/Doc/c-api/set.rst @@ -13,7 +13,7 @@ Set Objects object: frozenset This section details the public API for :class:`set` and :class:`frozenset` -objects. Any functionality not listed below is best accessed using the either +objects. Any functionality not listed below is best accessed using either the abstract object protocol (including :c:func:`PyObject_CallMethod`, :c:func:`PyObject_RichCompareBool`, :c:func:`PyObject_Hash`, :c:func:`PyObject_Repr`, :c:func:`PyObject_IsTrue`, :c:func:`PyObject_Print`, and @@ -31,7 +31,7 @@ the abstract object protocol (including :c:func:`PyObject_CallMethod`, in that it is a fixed size for small sets (much like tuple storage) and will point to a separate, variable sized block of memory for medium and large sized sets (much like list storage). None of the fields of this structure should be - considered public and are subject to change. All access should be done through + considered public and all are subject to change. All access should be done through the documented API rather than by manipulating the values in the structure. @@ -125,7 +125,7 @@ or :class:`frozenset` or instances of their subtypes. .. c:function:: int PySet_Add(PyObject *set, PyObject *key) Add *key* to a :class:`set` instance. Also works with :class:`frozenset` - instances (like :c:func:`PyTuple_SetItem` it can be used to fill-in the values + instances (like :c:func:`PyTuple_SetItem` it can be used to fill in the values of brand new frozensets before they are exposed to other code). Return ``0`` on success or ``-1`` on failure. Raise a :exc:`TypeError` if the *key* is unhashable. Raise a :exc:`MemoryError` if there is no room to grow. Raise a diff --git a/Doc/c-api/structures.rst b/Doc/c-api/structures.rst index 1e30d408ab6d1..1599b88e4aa27 100644 --- a/Doc/c-api/structures.rst +++ b/Doc/c-api/structures.rst @@ -461,7 +461,7 @@ Accessing attributes of extension types +=============+==================+===================================+ | name | const char \* | attribute name | +-------------+------------------+-----------------------------------+ - | get | getter | C Function to get the attribute | + | get | getter | C function to get the attribute | +-------------+------------------+-----------------------------------+ | set | setter | optional C function to set or | | | | delete the attribute, if omitted | diff --git a/Doc/c-api/sys.rst b/Doc/c-api/sys.rst index 9ac9179097892..37591a495fd4d 100644 --- a/Doc/c-api/sys.rst +++ b/Doc/c-api/sys.rst @@ -185,7 +185,7 @@ Operating System Utilities Return a pointer to a newly allocated byte string, use :c:func:`PyMem_Free` to free the memory. Return ``NULL`` on encoding error or memory allocation - error + error. If error_pos is not ``NULL``, ``*error_pos`` is set to ``(size_t)-1`` on success, or set to the index of the invalid character on encoding error. @@ -205,7 +205,7 @@ Operating System Utilities .. versionchanged:: 3.8 The function now uses the UTF-8 encoding on Windows if - :c:data:`Py_LegacyWindowsFSEncodingFlag` is zero; + :c:data:`Py_LegacyWindowsFSEncodingFlag` is zero. .. _systemfunctions: @@ -336,7 +336,7 @@ accessible to C code. They all work with the current interpreter thread's .. c:function:: int PySys_AddAuditHook(Py_AuditHookFunction hook, void *userData) Append the callable *hook* to the list of active auditing hooks. - Return zero for success + Return zero on success and non-zero on failure. If the runtime has been initialized, also set an error on failure. Hooks added through this API are called for all interpreters created by the runtime. diff --git a/Doc/c-api/type.rst b/Doc/c-api/type.rst index cc2d8fdaa513c..119a908171eb3 100644 --- a/Doc/c-api/type.rst +++ b/Doc/c-api/type.rst @@ -266,7 +266,7 @@ The following functions and structs are used to create .. versionchanged:: 3.9 - Slots in :c:type:`PyBufferProcs` in may be set in the unlimited API. + Slots in :c:type:`PyBufferProcs` may be set in the unlimited API. .. c:member:: void *PyType_Slot.pfunc From webhook-mailer at python.org Sat Apr 2 19:00:56 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Sat, 02 Apr 2022 23:00:56 -0000 Subject: [Python-checkins] [3.10] More minor fixes to C API docs (GH-31525) (GH-32258) Message-ID: https://github.com/python/cpython/commit/11f5fd1135065908c281ddfe767daa357c61e598 commit: 11f5fd1135065908c281ddfe767daa357c61e598 branch: 3.10 author: Jelle Zijlstra committer: JelleZijlstra date: 2022-04-02T16:00:51-07:00 summary: [3.10] More minor fixes to C API docs (GH-31525) (GH-32258) * wording fixes in type.rst * grammar and punctuation in sys.rst * set: grammar fixes * structures: capitalization fix * grammar fixes for sequence * objects: point to Py_TYPE instead of direct object access * numbers: add more explicit Python equivalences * method: add missing period * memory: grammar fix * mapping: grammar fixes * long: grammar fix * iter: fix grammar for PyAIter_Check * init: grammar fix. (cherry picked from commit 897bc6f9282238d5fb32d232ab62d30675244736) Co-authored-by: Jelle Zijlstra files: M Doc/c-api/init.rst M Doc/c-api/iter.rst M Doc/c-api/long.rst M Doc/c-api/mapping.rst M Doc/c-api/memory.rst M Doc/c-api/method.rst M Doc/c-api/number.rst M Doc/c-api/object.rst M Doc/c-api/sequence.rst M Doc/c-api/set.rst M Doc/c-api/structures.rst M Doc/c-api/sys.rst M Doc/c-api/type.rst diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst index 9ac3039d02ec3..a522790446001 100644 --- a/Doc/c-api/init.rst +++ b/Doc/c-api/init.rst @@ -1725,7 +1725,7 @@ is not possible due to its implementation being opaque at build time. argument is `NULL`. .. note:: - A freed key becomes a dangling pointer, you should reset the key to + A freed key becomes a dangling pointer. You should reset the key to `NULL`. diff --git a/Doc/c-api/iter.rst b/Doc/c-api/iter.rst index 3e388bb917a02..434d2021cea8e 100644 --- a/Doc/c-api/iter.rst +++ b/Doc/c-api/iter.rst @@ -14,8 +14,8 @@ There are two functions specifically for working with iterators. .. c:function:: int PyAIter_Check(PyObject *o) - Returns non-zero if the object 'obj' provides :class:`AsyncIterator` - protocols, and ``0`` otherwise. This function always succeeds. + Return non-zero if the object *o* provides the :class:`AsyncIterator` + protocol, and ``0`` otherwise. This function always succeeds. .. versionadded:: 3.10 diff --git a/Doc/c-api/long.rst b/Doc/c-api/long.rst index 4201490286b82..620344e71373b 100644 --- a/Doc/c-api/long.rst +++ b/Doc/c-api/long.rst @@ -41,7 +41,7 @@ distinguished from a number. Use :c:func:`PyErr_Occurred` to disambiguate. Return a new :c:type:`PyLongObject` object from *v*, or ``NULL`` on failure. The current implementation keeps an array of integer objects for all integers - between ``-5`` and ``256``, when you create an int in that range you actually + between ``-5`` and ``256``. When you create an int in that range you actually just get back a reference to the existing object. diff --git a/Doc/c-api/mapping.rst b/Doc/c-api/mapping.rst index 682160d1475c1..3c9d282c6d0ab 100644 --- a/Doc/c-api/mapping.rst +++ b/Doc/c-api/mapping.rst @@ -11,10 +11,10 @@ See also :c:func:`PyObject_GetItem`, :c:func:`PyObject_SetItem` and .. c:function:: int PyMapping_Check(PyObject *o) - Return ``1`` if the object provides mapping protocol or supports slicing, + Return ``1`` if the object provides the mapping protocol or supports slicing, and ``0`` otherwise. Note that it returns ``1`` for Python classes with - a :meth:`__getitem__` method since in general case it is impossible to - determine what type of keys it supports. This function always succeeds. + a :meth:`__getitem__` method, since in general it is impossible to + determine what type of keys the class supports. This function always succeeds. .. c:function:: Py_ssize_t PyMapping_Size(PyObject *o) diff --git a/Doc/c-api/memory.rst b/Doc/c-api/memory.rst index 0746d9ac52a7b..ced7ca79ca590 100644 --- a/Doc/c-api/memory.rst +++ b/Doc/c-api/memory.rst @@ -306,7 +306,7 @@ memory from the Python heap. .. note:: There is no guarantee that the memory returned by these allocators can be - successfully casted to a Python object when intercepting the allocating + successfully cast to a Python object when intercepting the allocating functions in this domain by the methods described in the :ref:`Customize Memory Allocators ` section. diff --git a/Doc/c-api/method.rst b/Doc/c-api/method.rst index 23852251dfe02..6e7e1e21aa93f 100644 --- a/Doc/c-api/method.rst +++ b/Doc/c-api/method.rst @@ -27,7 +27,7 @@ to bind a :c:data:`PyCFunction` to a class object. It replaces the former call .. c:function:: PyObject* PyInstanceMethod_New(PyObject *func) - Return a new instance method object, with *func* being any callable object + Return a new instance method object, with *func* being any callable object. *func* is the function that will be called when the instance method is called. diff --git a/Doc/c-api/number.rst b/Doc/c-api/number.rst index 37979bb506bcf..11c9c67d36a67 100644 --- a/Doc/c-api/number.rst +++ b/Doc/c-api/number.rst @@ -44,7 +44,7 @@ Number Protocol .. c:function:: PyObject* PyNumber_FloorDivide(PyObject *o1, PyObject *o2) Return the floor of *o1* divided by *o2*, or ``NULL`` on failure. This is - equivalent to the "classic" division of integers. + the equivalent of the Python expression ``o1 // o2``. .. c:function:: PyObject* PyNumber_TrueDivide(PyObject *o1, PyObject *o2) @@ -53,7 +53,7 @@ Number Protocol *o2*, or ``NULL`` on failure. The return value is "approximate" because binary floating point numbers are approximate; it is not possible to represent all real numbers in base two. This function can return a floating point value when - passed two integers. + passed two integers. This is the equivalent of the Python expression ``o1 / o2``. .. c:function:: PyObject* PyNumber_Remainder(PyObject *o1, PyObject *o2) @@ -180,6 +180,7 @@ Number Protocol floating point numbers are approximate; it is not possible to represent all real numbers in base two. This function can return a floating point value when passed two integers. The operation is done *in-place* when *o1* supports it. + This is the equivalent of the Python statement ``o1 /= o2``. .. c:function:: PyObject* PyNumber_InPlaceRemainder(PyObject *o1, PyObject *o2) @@ -285,6 +286,6 @@ Number Protocol .. c:function:: int PyIndex_Check(PyObject *o) - Returns ``1`` if *o* is an index integer (has the nb_index slot of the - tp_as_number structure filled in), and ``0`` otherwise. + Returns ``1`` if *o* is an index integer (has the ``nb_index`` slot of the + ``tp_as_number`` structure filled in), and ``0`` otherwise. This function always succeeds. diff --git a/Doc/c-api/object.rst b/Doc/c-api/object.rst index 41a3affcf9842..9dcfd769c64a0 100644 --- a/Doc/c-api/object.rst +++ b/Doc/c-api/object.rst @@ -93,7 +93,7 @@ Object Protocol return ``0`` on success. This is the equivalent of the Python statement ``o.attr_name = v``. - If *v* is ``NULL``, the attribute is deleted, however this feature is + If *v* is ``NULL``, the attribute is deleted, but this feature is deprecated in favour of using :c:func:`PyObject_DelAttrString`. @@ -291,7 +291,7 @@ Object Protocol of object *o*. On failure, raises :exc:`SystemError` and returns ``NULL``. This is equivalent to the Python expression ``type(o)``. This function increments the reference count of the return value. There's really no reason to use this - function instead of the common expression ``o->ob_type``, which returns a + function instead of the :c:func:`Py_TYPE()` function, which returns a pointer of type :c:type:`PyTypeObject*`, except when the incremented reference count is needed. diff --git a/Doc/c-api/sequence.rst b/Doc/c-api/sequence.rst index 6581885904117..c78d273f9f149 100644 --- a/Doc/c-api/sequence.rst +++ b/Doc/c-api/sequence.rst @@ -8,10 +8,10 @@ Sequence Protocol .. c:function:: int PySequence_Check(PyObject *o) - Return ``1`` if the object provides sequence protocol, and ``0`` otherwise. + Return ``1`` if the object provides the sequence protocol, and ``0`` otherwise. Note that it returns ``1`` for Python classes with a :meth:`__getitem__` - method unless they are :class:`dict` subclasses since in general case it - is impossible to determine what the type of keys it supports. This + method, unless they are :class:`dict` subclasses, since in general it + is impossible to determine what type of keys the class supports. This function always succeeds. @@ -69,7 +69,7 @@ Sequence Protocol is the equivalent of the Python statement ``o[i] = v``. This function *does not* steal a reference to *v*. - If *v* is ``NULL``, the element is deleted, however this feature is + If *v* is ``NULL``, the element is deleted, but this feature is deprecated in favour of using :c:func:`PySequence_DelItem`. @@ -147,7 +147,7 @@ Sequence Protocol Returns the length of *o*, assuming that *o* was returned by :c:func:`PySequence_Fast` and that *o* is not ``NULL``. The size can also be - gotten by calling :c:func:`PySequence_Size` on *o*, but + retrieved by calling :c:func:`PySequence_Size` on *o*, but :c:func:`PySequence_Fast_GET_SIZE` is faster because it can assume *o* is a list or tuple. diff --git a/Doc/c-api/set.rst b/Doc/c-api/set.rst index eca19c4d81647..f0d905bae8ae4 100644 --- a/Doc/c-api/set.rst +++ b/Doc/c-api/set.rst @@ -13,7 +13,7 @@ Set Objects object: frozenset This section details the public API for :class:`set` and :class:`frozenset` -objects. Any functionality not listed below is best accessed using the either +objects. Any functionality not listed below is best accessed using either the abstract object protocol (including :c:func:`PyObject_CallMethod`, :c:func:`PyObject_RichCompareBool`, :c:func:`PyObject_Hash`, :c:func:`PyObject_Repr`, :c:func:`PyObject_IsTrue`, :c:func:`PyObject_Print`, and @@ -31,7 +31,7 @@ the abstract object protocol (including :c:func:`PyObject_CallMethod`, in that it is a fixed size for small sets (much like tuple storage) and will point to a separate, variable sized block of memory for medium and large sized sets (much like list storage). None of the fields of this structure should be - considered public and are subject to change. All access should be done through + considered public and all are subject to change. All access should be done through the documented API rather than by manipulating the values in the structure. @@ -131,7 +131,7 @@ or :class:`frozenset` or instances of their subtypes. .. c:function:: int PySet_Add(PyObject *set, PyObject *key) Add *key* to a :class:`set` instance. Also works with :class:`frozenset` - instances (like :c:func:`PyTuple_SetItem` it can be used to fill-in the values + instances (like :c:func:`PyTuple_SetItem` it can be used to fill in the values of brand new frozensets before they are exposed to other code). Return ``0`` on success or ``-1`` on failure. Raise a :exc:`TypeError` if the *key* is unhashable. Raise a :exc:`MemoryError` if there is no room to grow. Raise a diff --git a/Doc/c-api/structures.rst b/Doc/c-api/structures.rst index 05c54ccc8dab5..839c889a6c8a7 100644 --- a/Doc/c-api/structures.rst +++ b/Doc/c-api/structures.rst @@ -494,7 +494,7 @@ Accessing attributes of extension types +=============+==================+===================================+ | name | const char \* | attribute name | +-------------+------------------+-----------------------------------+ - | get | getter | C Function to get the attribute | + | get | getter | C function to get the attribute | +-------------+------------------+-----------------------------------+ | set | setter | optional C function to set or | | | | delete the attribute, if omitted | diff --git a/Doc/c-api/sys.rst b/Doc/c-api/sys.rst index 97717f5fc1923..350d4e1de41ba 100644 --- a/Doc/c-api/sys.rst +++ b/Doc/c-api/sys.rst @@ -177,7 +177,7 @@ Operating System Utilities Return a pointer to a newly allocated byte string, use :c:func:`PyMem_Free` to free the memory. Return ``NULL`` on encoding error or memory allocation - error + error. If error_pos is not ``NULL``, ``*error_pos`` is set to ``(size_t)-1`` on success, or set to the index of the invalid character on encoding error. @@ -207,7 +207,7 @@ Operating System Utilities .. versionchanged:: 3.8 The function now uses the UTF-8 encoding on Windows if - :c:data:`Py_LegacyWindowsFSEncodingFlag` is zero; + :c:data:`Py_LegacyWindowsFSEncodingFlag` is zero. .. _systemfunctions: @@ -338,7 +338,7 @@ accessible to C code. They all work with the current interpreter thread's .. c:function:: int PySys_AddAuditHook(Py_AuditHookFunction hook, void *userData) Append the callable *hook* to the list of active auditing hooks. - Return zero for success + Return zero on success and non-zero on failure. If the runtime has been initialized, also set an error on failure. Hooks added through this API are called for all interpreters created by the runtime. diff --git a/Doc/c-api/type.rst b/Doc/c-api/type.rst index c472e8df9cf43..01d00bede544d 100644 --- a/Doc/c-api/type.rst +++ b/Doc/c-api/type.rst @@ -272,7 +272,7 @@ The following functions and structs are used to create .. versionchanged:: 3.9 - Slots in :c:type:`PyBufferProcs` in may be set in the unlimited API. + Slots in :c:type:`PyBufferProcs` may be set in the unlimited API. .. c:member:: void *PyType_Slot.pfunc From webhook-mailer at python.org Sat Apr 2 20:13:46 2022 From: webhook-mailer at python.org (terryjreedy) Date: Sun, 03 Apr 2022 00:13:46 -0000 Subject: [Python-checkins] bpo-24563: Link encoding names to encoding declarations (GH-32274) Message-ID: https://github.com/python/cpython/commit/01be5d6446abbdd95d0c18bd19a58a62b05568d8 commit: 01be5d6446abbdd95d0c18bd19a58a62b05568d8 branch: main author: Terry Jan Reedy committer: terryjreedy date: 2022-04-02T20:13:37-04:00 summary: bpo-24563: Link encoding names to encoding declarations (GH-32274) files: M Doc/reference/lexical_analysis.rst diff --git a/Doc/reference/lexical_analysis.rst b/Doc/reference/lexical_analysis.rst index b69d0fbdd41f8..dba1a9dd2e15a 100644 --- a/Doc/reference/lexical_analysis.rst +++ b/Doc/reference/lexical_analysis.rst @@ -101,12 +101,11 @@ addition, if the first bytes of the file are the UTF-8 byte-order mark (``b'\xef\xbb\xbf'``), the declared file encoding is UTF-8 (this is supported, among others, by Microsoft's :program:`notepad`). -If an encoding is declared, the encoding name must be recognized by Python. The +If an encoding is declared, the encoding name must be recognized by Python +(see :ref:`standard-encodings`). The encoding is used for all lexical analysis, including string literals, comments and identifiers. -.. XXX there should be a list of supported encodings. - .. _explicit-joining: From webhook-mailer at python.org Sat Apr 2 20:40:39 2022 From: webhook-mailer at python.org (miss-islington) Date: Sun, 03 Apr 2022 00:40:39 -0000 Subject: [Python-checkins] bpo-24563: Link encoding names to encoding declarations (GH-32274) Message-ID: https://github.com/python/cpython/commit/f4e280d8c5461fa5e928fa07f6cd1779a4ac43ba commit: f4e280d8c5461fa5e928fa07f6cd1779a4ac43ba branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-02T17:40:31-07:00 summary: bpo-24563: Link encoding names to encoding declarations (GH-32274) (cherry picked from commit 01be5d6446abbdd95d0c18bd19a58a62b05568d8) Co-authored-by: Terry Jan Reedy files: M Doc/reference/lexical_analysis.rst diff --git a/Doc/reference/lexical_analysis.rst b/Doc/reference/lexical_analysis.rst index 77e0578f5d89b..17c0fb4a51eb4 100644 --- a/Doc/reference/lexical_analysis.rst +++ b/Doc/reference/lexical_analysis.rst @@ -101,12 +101,11 @@ addition, if the first bytes of the file are the UTF-8 byte-order mark (``b'\xef\xbb\xbf'``), the declared file encoding is UTF-8 (this is supported, among others, by Microsoft's :program:`notepad`). -If an encoding is declared, the encoding name must be recognized by Python. The +If an encoding is declared, the encoding name must be recognized by Python +(see :ref:`standard-encodings`). The encoding is used for all lexical analysis, including string literals, comments and identifiers. -.. XXX there should be a list of supported encodings. - .. _explicit-joining: From webhook-mailer at python.org Sat Apr 2 20:41:00 2022 From: webhook-mailer at python.org (miss-islington) Date: Sun, 03 Apr 2022 00:41:00 -0000 Subject: [Python-checkins] bpo-24563: Link encoding names to encoding declarations (GH-32274) Message-ID: https://github.com/python/cpython/commit/ce6af314ca8529d5ed0f307deb9c33029a69f4cb commit: ce6af314ca8529d5ed0f307deb9c33029a69f4cb branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-02T17:40:55-07:00 summary: bpo-24563: Link encoding names to encoding declarations (GH-32274) (cherry picked from commit 01be5d6446abbdd95d0c18bd19a58a62b05568d8) Co-authored-by: Terry Jan Reedy files: M Doc/reference/lexical_analysis.rst diff --git a/Doc/reference/lexical_analysis.rst b/Doc/reference/lexical_analysis.rst index 21ad3731a3246..d9e2cead958c9 100644 --- a/Doc/reference/lexical_analysis.rst +++ b/Doc/reference/lexical_analysis.rst @@ -101,12 +101,11 @@ addition, if the first bytes of the file are the UTF-8 byte-order mark (``b'\xef\xbb\xbf'``), the declared file encoding is UTF-8 (this is supported, among others, by Microsoft's :program:`notepad`). -If an encoding is declared, the encoding name must be recognized by Python. The +If an encoding is declared, the encoding name must be recognized by Python +(see :ref:`standard-encodings`). The encoding is used for all lexical analysis, including string literals, comments and identifiers. -.. XXX there should be a list of supported encodings. - .. _explicit-joining: From webhook-mailer at python.org Sat Apr 2 20:52:27 2022 From: webhook-mailer at python.org (terryjreedy) Date: Sun, 03 Apr 2022 00:52:27 -0000 Subject: [Python-checkins] bpo-46033: Clarify for-statement execution (GH-30025) Message-ID: https://github.com/python/cpython/commit/281f980d354d1709018a2dc77f79388faf3e56c0 commit: 281f980d354d1709018a2dc77f79388faf3e56c0 branch: main author: Micha? D committer: terryjreedy date: 2022-04-02T20:52:20-04:00 summary: bpo-46033: Clarify for-statement execution (GH-30025) Co-authored-by: Terry Jan Reedy Co-authored-by: Jelle Zijlstra files: A Misc/NEWS.d/next/Documentation/2022-01-03-18-50-39.bpo-46033.7WeF0f.rst M Doc/reference/compound_stmts.rst diff --git a/Doc/reference/compound_stmts.rst b/Doc/reference/compound_stmts.rst index 92024e51ef4fc..688407195f05d 100644 --- a/Doc/reference/compound_stmts.rst +++ b/Doc/reference/compound_stmts.rst @@ -157,17 +157,14 @@ The :keyword:`for` statement is used to iterate over the elements of a sequence for_stmt: "for" `target_list` "in" `starred_list` ":" `suite` : ["else" ":" `suite`] -The expression list is evaluated once; it should yield an iterable object. An -iterator is created for the result of the ``starred_list``. The expression -list can contain starred elements (``*x, *y``) that will be unpacked in the -final iterator (as when constructing a ``tuple`` or ``list`` literal). The -suite is then executed once for each item provided by the iterator, in the -order returned by the iterator. Each item in turn is assigned to the target -list using the standard rules for assignments (see :ref:`assignment`), and then -the suite is executed. When the items are exhausted (which is immediately when -the sequence is empty or an iterator raises a :exc:`StopIteration` exception), -the suite in the :keyword:`!else` clause, if present, is executed, and the loop -terminates. +The ``starred_list`` expression is evaluated once; it should yield an +:term:`iterable` object. An :term:`iterator` is created for that iterable. +The first item provided +by the iterator is then assigned to the target list using the standard +rules for assignments (see :ref:`assignment`), and the suite is executed. This +repeats for each item provided by the iterator. When the iterator is exhausted, +the suite in the :keyword:`!else` clause, +if present, is executed, and the loop terminates. .. index:: statement: break diff --git a/Misc/NEWS.d/next/Documentation/2022-01-03-18-50-39.bpo-46033.7WeF0f.rst b/Misc/NEWS.d/next/Documentation/2022-01-03-18-50-39.bpo-46033.7WeF0f.rst new file mode 100644 index 0000000000000..a484def239d8b --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2022-01-03-18-50-39.bpo-46033.7WeF0f.rst @@ -0,0 +1 @@ +Clarify ``for`` statement execution in its doc. From webhook-mailer at python.org Sat Apr 2 22:41:20 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Sun, 03 Apr 2022 02:41:20 -0000 Subject: [Python-checkins] Language reference: Remove duplicated text about iterable unpacking (GH-25212) Message-ID: https://github.com/python/cpython/commit/4f5d56f8f33196f5ed8ffad0ab2f012afda2f9b3 commit: 4f5d56f8f33196f5ed8ffad0ab2f012afda2f9b3 branch: main author: Jiashuo Li <4003950+jiasli at users.noreply.github.com> committer: JelleZijlstra date: 2022-04-02T19:41:16-07:00 summary: Language reference: Remove duplicated text about iterable unpacking (GH-25212) files: M Doc/reference/simple_stmts.rst diff --git a/Doc/reference/simple_stmts.rst b/Doc/reference/simple_stmts.rst index e9795d8deffc7..5f19a3822ba9f 100644 --- a/Doc/reference/simple_stmts.rst +++ b/Doc/reference/simple_stmts.rst @@ -124,9 +124,7 @@ square brackets, is recursively defined as follows. * If the target list is a single target with no trailing comma, optionally in parentheses, the object is assigned to that target. -* Else: The object must be an iterable with the same number of - items as there are targets in the target list, and the items are assigned, - from left to right, to the corresponding targets. +* Else: * If the target list contains one target prefixed with an asterisk, called a "starred" target: The object must be an iterable with at least as many items From webhook-mailer at python.org Sat Apr 2 23:05:29 2022 From: webhook-mailer at python.org (miss-islington) Date: Sun, 03 Apr 2022 03:05:29 -0000 Subject: [Python-checkins] Language reference: Remove duplicated text about iterable unpacking (GH-25212) Message-ID: https://github.com/python/cpython/commit/a5c90784be0bf70287bac4195573e85e956c2332 commit: a5c90784be0bf70287bac4195573e85e956c2332 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-02T20:05:11-07:00 summary: Language reference: Remove duplicated text about iterable unpacking (GH-25212) (cherry picked from commit 4f5d56f8f33196f5ed8ffad0ab2f012afda2f9b3) Co-authored-by: Jiashuo Li <4003950+jiasli at users.noreply.github.com> files: M Doc/reference/simple_stmts.rst diff --git a/Doc/reference/simple_stmts.rst b/Doc/reference/simple_stmts.rst index 92dfcb0461e60..8dfc01db263f5 100644 --- a/Doc/reference/simple_stmts.rst +++ b/Doc/reference/simple_stmts.rst @@ -124,9 +124,7 @@ square brackets, is recursively defined as follows. * If the target list is a single target with no trailing comma, optionally in parentheses, the object is assigned to that target. -* Else: The object must be an iterable with the same number of - items as there are targets in the target list, and the items are assigned, - from left to right, to the corresponding targets. +* Else: * If the target list contains one target prefixed with an asterisk, called a "starred" target: The object must be an iterable with at least as many items From webhook-mailer at python.org Sat Apr 2 23:07:07 2022 From: webhook-mailer at python.org (miss-islington) Date: Sun, 03 Apr 2022 03:07:07 -0000 Subject: [Python-checkins] Language reference: Remove duplicated text about iterable unpacking (GH-25212) Message-ID: https://github.com/python/cpython/commit/b5092350b2f70ec3519761c60f7399ebcc04ebed commit: b5092350b2f70ec3519761c60f7399ebcc04ebed branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-02T20:06:54-07:00 summary: Language reference: Remove duplicated text about iterable unpacking (GH-25212) (cherry picked from commit 4f5d56f8f33196f5ed8ffad0ab2f012afda2f9b3) Co-authored-by: Jiashuo Li <4003950+jiasli at users.noreply.github.com> files: M Doc/reference/simple_stmts.rst diff --git a/Doc/reference/simple_stmts.rst b/Doc/reference/simple_stmts.rst index 79f4011307ed8..5f9e8dc1791d5 100644 --- a/Doc/reference/simple_stmts.rst +++ b/Doc/reference/simple_stmts.rst @@ -124,9 +124,7 @@ square brackets, is recursively defined as follows. * If the target list is a single target with no trailing comma, optionally in parentheses, the object is assigned to that target. -* Else: The object must be an iterable with the same number of - items as there are targets in the target list, and the items are assigned, - from left to right, to the corresponding targets. +* Else: * If the target list contains one target prefixed with an asterisk, called a "starred" target: The object must be an iterable with at least as many items From webhook-mailer at python.org Sun Apr 3 02:56:05 2022 From: webhook-mailer at python.org (ncoghlan) Date: Sun, 03 Apr 2022 06:56:05 -0000 Subject: [Python-checkins] bpo-44800: Document internal frame naming conventions (GH-32281) Message-ID: https://github.com/python/cpython/commit/124227c95f310d2ecd4b567271ab1919fc7000cb commit: 124227c95f310d2ecd4b567271ab1919fc7000cb branch: main author: Nick Coghlan committer: ncoghlan date: 2022-04-03T16:55:55+10:00 summary: bpo-44800: Document internal frame naming conventions (GH-32281) The fact interpreter frames were split out from full frame objects rather than always being part of the eval loop implementation means that it's tricky to infer the expected naming conventions simply from looking at the code. Documenting the de facto conventions in pycore_frame.h means future readers of the code will have a clear explanation of the rationale for those conventions (i.e. minimising non-functional code churn). files: M Include/internal/pycore_frame.h diff --git a/Include/internal/pycore_frame.h b/Include/internal/pycore_frame.h index 211831a6e497f..3ea538ccc5500 100644 --- a/Include/internal/pycore_frame.h +++ b/Include/internal/pycore_frame.h @@ -7,6 +7,75 @@ extern "C" { #include #include +/* Starting in CPython 3.11, CPython separates the frame state between the + * full frame objects exposed by the Python and C runtime state introspection + * APIs, and internal lighter weight interpreter frames, which are simple C + * structures owned by either the interpreter eval loop (while executing + * ordinary functions), by a generator or coroutine object (for frames that + * are able to be suspended), or by their corresponding full frame object (if + * a state instrospection API has been invoked and the full frame object has + * taken responsibility for the lifecycle of the interpreter frame). + * + * This split storage eliminates a lot of allocation and deallocation of full + * Python objects during code execution, providing a significant speed gain + * over the previous approach of using full Python objects for both + * introspection and code execution. + * + * Struct names: + * + * * PyFrameObject: the full Python frame object + * * _PyInterpreterFrame: the lightweight frame struct used by the eval loop + * * _PyCFrame: a struct that lives on the C stack and allows Python level + * recursive evaluation to be decoupled from recursive C level invocation + * of the bytecode eval loop + * * See pystate.h for more details on this struct + * + * Field naming conventions: + * + * * full frame object fields have an "f_*" (or "_f_*") prefix + * * new interpreter frame fields have no prefix + * * Several interpreter frame fields have the "f_*" prefix as a result of + * trying to keep diffs as small as was feasible when splitting the original + * frame struct definition in two. The following are all interpreter frame + * fields, NOT full frame object fields: + * * f_func + * * f_globals + * * f_builtins + * * f_locals + * * f_code + * * f_lasti + * * f_state + * * Renaming those fields was considered but ultimately deemed too disruptive + * to key third party projects that were trying to keep up with the Python + * 3.11 code evaluation changes during the alpha release cycle + * (see bpo-44800 for details) + * + * Naming conventions for local variables, function parameters and fields in other structs: + * + * * "frame" and "f" may refer to either full frame objects or interpreter frames + * * the context of use or the field naming conventions usually make the + * type being referenced unambiguous in code reviews + * * the following alternative names are used when more clarity is needed: + * * full frame objects: "frame_obj" (and variants like "frameobj" or "fobj") + * * interpreter frame structs: "frame_data" or "iframe" + * * "current frame" should NOT be abbreviated as "cframe", as the latter now + * typically refers to _PyCFrame structs + * + * Function/macro parameter types: + * + * * "PyFrame_*" functions and other public C API functions that relate to + * frames accept full frame object pointers + * * "_PyFrame_*" functions and other private C API functions that relate to + * frames accept either full frame object or interpreter frame pointers. + * Check the specific function signatures for details. + * + * Function return types: + * + * * Public C API functions will only ever return full frame object pointers + * * Private C API functions with an underscore prefix may return interpreter + * frame pointers instead. Check the specific function signatures for details. + */ + struct _frame { PyObject_HEAD PyFrameObject *f_back; /* previous frame, or NULL */ From webhook-mailer at python.org Sun Apr 3 03:45:37 2022 From: webhook-mailer at python.org (miss-islington) Date: Sun, 03 Apr 2022 07:45:37 -0000 Subject: [Python-checkins] bpo-47196: Fix one more PyInit function signature (GH-32280) Message-ID: https://github.com/python/cpython/commit/3faa9f78d4b9a8c0fd4657b434bdb08ae1f28800 commit: 3faa9f78d4b9a8c0fd4657b434bdb08ae1f28800 branch: main author: Hood Chatham committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-03T00:45:26-07:00 summary: bpo-47196: Fix one more PyInit function signature (GH-32280) I missed one PyInit function in #32244. Automerge-Triggered-By: GH:tiran files: M Modules/_testmultiphase.c diff --git a/Modules/_testmultiphase.c b/Modules/_testmultiphase.c index e9a37a019134f..1b4f58eb1c0b4 100644 --- a/Modules/_testmultiphase.c +++ b/Modules/_testmultiphase.c @@ -796,7 +796,7 @@ static PyModuleDef def_exec_raise = TEST_MODULE_DEF( "_testmultiphase_exec_raise", slots_exec_raise, NULL); PyMODINIT_FUNC -PyInit__testmultiphase_exec_raise(PyObject *mod) +PyInit__testmultiphase_exec_raise(void) { return PyModuleDef_Init(&def_exec_raise); } From webhook-mailer at python.org Sun Apr 3 11:04:16 2022 From: webhook-mailer at python.org (tiran) Date: Sun, 03 Apr 2022 15:04:16 -0000 Subject: [Python-checkins] bpo-47205: Skip error check of sched_get/setaffinity on FreeBSD (GH-32285) Message-ID: https://github.com/python/cpython/commit/b82cdd1dac9a9be52051abd90a1ce69236ac41f4 commit: b82cdd1dac9a9be52051abd90a1ce69236ac41f4 branch: main author: Christian Heimes committer: tiran date: 2022-04-03T17:03:49+02:00 summary: bpo-47205: Skip error check of sched_get/setaffinity on FreeBSD (GH-32285) files: A Misc/NEWS.d/next/Tests/2022-04-03-14-38-21.bpo-47205.hbbTnh.rst M Lib/test/test_posix.py diff --git a/Lib/test/test_posix.py b/Lib/test/test_posix.py index c10039b17f7f8..f44b8d0403ff2 100644 --- a/Lib/test/test_posix.py +++ b/Lib/test/test_posix.py @@ -1194,7 +1194,9 @@ def test_sched_getaffinity(self): mask = posix.sched_getaffinity(0) self.assertIsInstance(mask, set) self.assertGreaterEqual(len(mask), 1) - self.assertRaises(OSError, posix.sched_getaffinity, -1) + if not sys.platform.startswith("freebsd"): + # bpo-47205: does not raise OSError on FreeBSD + self.assertRaises(OSError, posix.sched_getaffinity, -1) for cpu in mask: self.assertIsInstance(cpu, int) self.assertGreaterEqual(cpu, 0) @@ -1212,7 +1214,9 @@ def test_sched_setaffinity(self): self.assertRaises(ValueError, posix.sched_setaffinity, 0, [-10]) self.assertRaises(ValueError, posix.sched_setaffinity, 0, map(int, "0X")) self.assertRaises(OverflowError, posix.sched_setaffinity, 0, [1<<128]) - self.assertRaises(OSError, posix.sched_setaffinity, -1, mask) + if not sys.platform.startswith("freebsd"): + # bpo-47205: does not raise OSError on FreeBSD + self.assertRaises(OSError, posix.sched_setaffinity, -1, mask) def test_rtld_constants(self): # check presence of major RTLD_* constants diff --git a/Misc/NEWS.d/next/Tests/2022-04-03-14-38-21.bpo-47205.hbbTnh.rst b/Misc/NEWS.d/next/Tests/2022-04-03-14-38-21.bpo-47205.hbbTnh.rst new file mode 100644 index 0000000000000..35fd94421326e --- /dev/null +++ b/Misc/NEWS.d/next/Tests/2022-04-03-14-38-21.bpo-47205.hbbTnh.rst @@ -0,0 +1,2 @@ +Skip test for :func:`~os.sched_getaffinity` and +:func:`~os.sched_setaffinity` error case on FreeBSD. From webhook-mailer at python.org Sun Apr 3 11:30:54 2022 From: webhook-mailer at python.org (miss-islington) Date: Sun, 03 Apr 2022 15:30:54 -0000 Subject: [Python-checkins] bpo-47205: Skip error check of sched_get/setaffinity on FreeBSD (GH-32285) Message-ID: https://github.com/python/cpython/commit/490ccbd6e0e5aad07a40c79a6b0c45ffca91724b commit: 490ccbd6e0e5aad07a40c79a6b0c45ffca91724b branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-03T08:30:39-07:00 summary: bpo-47205: Skip error check of sched_get/setaffinity on FreeBSD (GH-32285) (cherry picked from commit b82cdd1dac9a9be52051abd90a1ce69236ac41f4) Co-authored-by: Christian Heimes files: A Misc/NEWS.d/next/Tests/2022-04-03-14-38-21.bpo-47205.hbbTnh.rst M Lib/test/test_posix.py diff --git a/Lib/test/test_posix.py b/Lib/test/test_posix.py index a8db30679af8d..6ba14547bdd6c 100644 --- a/Lib/test/test_posix.py +++ b/Lib/test/test_posix.py @@ -1153,7 +1153,9 @@ def test_sched_getaffinity(self): mask = posix.sched_getaffinity(0) self.assertIsInstance(mask, set) self.assertGreaterEqual(len(mask), 1) - self.assertRaises(OSError, posix.sched_getaffinity, -1) + if not sys.platform.startswith("freebsd"): + # bpo-47205: does not raise OSError on FreeBSD + self.assertRaises(OSError, posix.sched_getaffinity, -1) for cpu in mask: self.assertIsInstance(cpu, int) self.assertGreaterEqual(cpu, 0) @@ -1171,7 +1173,9 @@ def test_sched_setaffinity(self): self.assertRaises(ValueError, posix.sched_setaffinity, 0, [-10]) self.assertRaises(ValueError, posix.sched_setaffinity, 0, map(int, "0X")) self.assertRaises(OverflowError, posix.sched_setaffinity, 0, [1<<128]) - self.assertRaises(OSError, posix.sched_setaffinity, -1, mask) + if not sys.platform.startswith("freebsd"): + # bpo-47205: does not raise OSError on FreeBSD + self.assertRaises(OSError, posix.sched_setaffinity, -1, mask) def test_rtld_constants(self): # check presence of major RTLD_* constants diff --git a/Misc/NEWS.d/next/Tests/2022-04-03-14-38-21.bpo-47205.hbbTnh.rst b/Misc/NEWS.d/next/Tests/2022-04-03-14-38-21.bpo-47205.hbbTnh.rst new file mode 100644 index 0000000000000..35fd94421326e --- /dev/null +++ b/Misc/NEWS.d/next/Tests/2022-04-03-14-38-21.bpo-47205.hbbTnh.rst @@ -0,0 +1,2 @@ +Skip test for :func:`~os.sched_getaffinity` and +:func:`~os.sched_setaffinity` error case on FreeBSD. From webhook-mailer at python.org Sun Apr 3 12:16:41 2022 From: webhook-mailer at python.org (serhiy-storchaka) Date: Sun, 03 Apr 2022 16:16:41 -0000 Subject: [Python-checkins] bpo-23689: re module, fix memory leak when a match is terminated by a signal or memory allocation failure (GH-32283) Message-ID: https://github.com/python/cpython/commit/6e3eee5c11b539e9aab39cff783acf57838c355a commit: 6e3eee5c11b539e9aab39cff783acf57838c355a branch: main author: Ma Lin committer: serhiy-storchaka date: 2022-04-03T19:16:20+03:00 summary: bpo-23689: re module, fix memory leak when a match is terminated by a signal or memory allocation failure (GH-32283) files: A Misc/NEWS.d/next/Library/2022-04-03-13-19-08.bpo-23689.TFSc3E.rst M Lib/re/_compiler.py M Lib/re/_constants.py M Lib/test/test_re.py M Modules/_sre.c M Modules/clinic/_sre.c.h M Modules/sre.h M Modules/sre_constants.h M Modules/sre_lib.h diff --git a/Lib/re/_compiler.py b/Lib/re/_compiler.py index 62da8e55d72ab..bedd4b8f40016 100644 --- a/Lib/re/_compiler.py +++ b/Lib/re/_compiler.py @@ -67,14 +67,21 @@ _ignorecase_fixes = {i: tuple(j for j in t if i != j) for t in _equivalences for i in t} +class _CompileData: + __slots__ = ('code', 'repeat_count') + def __init__(self): + self.code = [] + self.repeat_count = 0 + def _combine_flags(flags, add_flags, del_flags, TYPE_FLAGS=_parser.TYPE_FLAGS): if add_flags & TYPE_FLAGS: flags &= ~TYPE_FLAGS return (flags | add_flags) & ~del_flags -def _compile(code, pattern, flags): +def _compile(data, pattern, flags): # internal: compile a (sub)pattern + code = data.code emit = code.append _len = len LITERAL_CODES = _LITERAL_CODES @@ -147,7 +154,7 @@ def _compile(code, pattern, flags): skip = _len(code); emit(0) emit(av[0]) emit(av[1]) - _compile(code, av[2], flags) + _compile(data, av[2], flags) emit(SUCCESS) code[skip] = _len(code) - skip else: @@ -155,7 +162,11 @@ def _compile(code, pattern, flags): skip = _len(code); emit(0) emit(av[0]) emit(av[1]) - _compile(code, av[2], flags) + # now op is in (MIN_REPEAT, MAX_REPEAT, POSSESSIVE_REPEAT) + if op != POSSESSIVE_REPEAT: + emit(data.repeat_count) + data.repeat_count += 1 + _compile(data, av[2], flags) code[skip] = _len(code) - skip emit(REPEATING_CODES[op][1]) elif op is SUBPATTERN: @@ -164,7 +175,7 @@ def _compile(code, pattern, flags): emit(MARK) emit((group-1)*2) # _compile_info(code, p, _combine_flags(flags, add_flags, del_flags)) - _compile(code, p, _combine_flags(flags, add_flags, del_flags)) + _compile(data, p, _combine_flags(flags, add_flags, del_flags)) if group: emit(MARK) emit((group-1)*2+1) @@ -176,7 +187,7 @@ def _compile(code, pattern, flags): # pop their stack if they reach it emit(ATOMIC_GROUP) skip = _len(code); emit(0) - _compile(code, av, flags) + _compile(data, av, flags) emit(SUCCESS) code[skip] = _len(code) - skip elif op in SUCCESS_CODES: @@ -191,13 +202,13 @@ def _compile(code, pattern, flags): if lo != hi: raise error("look-behind requires fixed-width pattern") emit(lo) # look behind - _compile(code, av[1], flags) + _compile(data, av[1], flags) emit(SUCCESS) code[skip] = _len(code) - skip elif op is CALL: emit(op) skip = _len(code); emit(0) - _compile(code, av, flags) + _compile(data, av, flags) emit(SUCCESS) code[skip] = _len(code) - skip elif op is AT: @@ -216,7 +227,7 @@ def _compile(code, pattern, flags): for av in av[1]: skip = _len(code); emit(0) # _compile_info(code, av, flags) - _compile(code, av, flags) + _compile(data, av, flags) emit(JUMP) tailappend(_len(code)); emit(0) code[skip] = _len(code) - skip @@ -244,12 +255,12 @@ def _compile(code, pattern, flags): emit(op) emit(av[0]-1) skipyes = _len(code); emit(0) - _compile(code, av[1], flags) + _compile(data, av[1], flags) if av[2]: emit(JUMP) skipno = _len(code); emit(0) code[skipyes] = _len(code) - skipyes + 1 - _compile(code, av[2], flags) + _compile(data, av[2], flags) code[skipno] = _len(code) - skipno else: code[skipyes] = _len(code) - skipyes + 1 @@ -608,17 +619,17 @@ def isstring(obj): def _code(p, flags): flags = p.state.flags | flags - code = [] + data = _CompileData() # compile info block - _compile_info(code, p, flags) + _compile_info(data.code, p, flags) # compile the pattern - _compile(code, p.data, flags) + _compile(data, p.data, flags) - code.append(SUCCESS) + data.code.append(SUCCESS) - return code + return data def _hex_code(code): return '[%s]' % ', '.join('%#0*x' % (_sre.CODESIZE*2+2, x) for x in code) @@ -719,7 +730,7 @@ def print_2(*args): else: print_(FAILURE) i += 1 - elif op in (REPEAT, REPEAT_ONE, MIN_REPEAT_ONE, + elif op in (REPEAT_ONE, MIN_REPEAT_ONE, POSSESSIVE_REPEAT, POSSESSIVE_REPEAT_ONE): skip, min, max = code[i: i+3] if max == MAXREPEAT: @@ -727,6 +738,13 @@ def print_2(*args): print_(op, skip, min, max, to=i+skip) dis_(i+3, i+skip) i += skip + elif op is REPEAT: + skip, min, max, repeat_index = code[i: i+4] + if max == MAXREPEAT: + max = 'MAXREPEAT' + print_(op, skip, min, max, repeat_index, to=i+skip) + dis_(i+4, i+skip) + i += skip elif op is GROUPREF_EXISTS: arg, skip = code[i: i+2] print_(op, arg, skip, to=i+skip) @@ -781,11 +799,11 @@ def compile(p, flags=0): else: pattern = None - code = _code(p, flags) + data = _code(p, flags) if flags & SRE_FLAG_DEBUG: print() - dis(code) + dis(data.code) # map in either direction groupindex = p.state.groupdict @@ -794,7 +812,6 @@ def compile(p, flags=0): indexgroup[i] = k return _sre.compile( - pattern, flags | p.state.flags, code, - p.state.groups-1, - groupindex, tuple(indexgroup) - ) + pattern, flags | p.state.flags, data.code, + p.state.groups-1, groupindex, tuple(indexgroup), + data.repeat_count) diff --git a/Lib/re/_constants.py b/Lib/re/_constants.py index c735edfea1f13..5317fd53e9c5a 100644 --- a/Lib/re/_constants.py +++ b/Lib/re/_constants.py @@ -13,7 +13,7 @@ # update when constants are added or removed -MAGIC = 20220318 +MAGIC = 20220402 from _sre import MAXREPEAT, MAXGROUPS diff --git a/Lib/test/test_re.py b/Lib/test/test_re.py index f1e5af452d8e0..553eb4cfe85b6 100644 --- a/Lib/test/test_re.py +++ b/Lib/test/test_re.py @@ -1643,9 +1643,12 @@ def test_dealloc(self): long_overflow = 2**128 self.assertRaises(TypeError, re.finditer, "a", {}) with self.assertRaises(OverflowError): - _sre.compile("abc", 0, [long_overflow], 0, {}, ()) + _sre.compile("abc", 0, [long_overflow], 0, {}, (), 0) with self.assertRaises(TypeError): - _sre.compile({}, 0, [], 0, [], []) + _sre.compile({}, 0, [], 0, [], [], 0) + with self.assertRaises(RuntimeError): + # invalid repeat_count -1 + _sre.compile("abc", 0, [1], 0, {}, (), -1) def test_search_dot_unicode(self): self.assertTrue(re.search("123.*-", '123abc-')) @@ -2334,6 +2337,27 @@ def test_possesive_repeat(self): 14. SUCCESS ''') + def test_repeat_index(self): + self.assertEqual(get_debug_out(r'(?:ab)*?(?:cd)*'), '''\ +MIN_REPEAT 0 MAXREPEAT + LITERAL 97 + LITERAL 98 +MAX_REPEAT 0 MAXREPEAT + LITERAL 99 + LITERAL 100 + + 0. INFO 4 0b0 0 MAXREPEAT (to 5) + 5: REPEAT 8 0 MAXREPEAT 0 (to 14) +10. LITERAL 0x61 ('a') +12. LITERAL 0x62 ('b') +14: MIN_UNTIL +15. REPEAT 8 0 MAXREPEAT 1 (to 24) +20. LITERAL 0x63 ('c') +22. LITERAL 0x64 ('d') +24: MAX_UNTIL +25. SUCCESS +''') + class PatternReprTests(unittest.TestCase): def check(self, pattern, expected): diff --git a/Misc/NEWS.d/next/Library/2022-04-03-13-19-08.bpo-23689.TFSc3E.rst b/Misc/NEWS.d/next/Library/2022-04-03-13-19-08.bpo-23689.TFSc3E.rst new file mode 100644 index 0000000000000..1032087d9b850 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-03-13-19-08.bpo-23689.TFSc3E.rst @@ -0,0 +1,2 @@ +:mod:`re` module: fix memory leak when a match is terminated by a signal or +memory allocation failure. Patch by Ma Lin. diff --git a/Modules/_sre.c b/Modules/_sre.c index 48193f82475a4..506363d6fbf6d 100644 --- a/Modules/_sre.c +++ b/Modules/_sre.c @@ -427,6 +427,12 @@ state_init(SRE_STATE* state, PatternObject* pattern, PyObject* string, state->lastmark = -1; state->lastindex = -1; + state->repeats_array = PyMem_New(SRE_REPEAT, pattern->repeat_count); + if (!state->repeats_array) { + PyErr_NoMemory(); + goto err; + } + state->buffer.buf = NULL; ptr = getstring(string, &length, &isbytes, &charsize, &state->buffer); if (!ptr) @@ -476,6 +482,9 @@ state_init(SRE_STATE* state, PatternObject* pattern, PyObject* string, safely casted to `void*`, see bpo-39943 for details. */ PyMem_Free((void*) state->mark); state->mark = NULL; + PyMem_Free(state->repeats_array); + state->repeats_array = NULL; + if (state->buffer.buf) PyBuffer_Release(&state->buffer); return NULL; @@ -491,6 +500,8 @@ state_fini(SRE_STATE* state) /* See above PyMem_Del for why we explicitly cast here. */ PyMem_Free((void*) state->mark); state->mark = NULL; + PyMem_Free(state->repeats_array); + state->repeats_array = NULL; } /* calculate offset from start of string */ @@ -1407,14 +1418,15 @@ _sre.compile groups: Py_ssize_t groupindex: object(subclass_of='&PyDict_Type') indexgroup: object(subclass_of='&PyTuple_Type') + repeat_count: Py_ssize_t [clinic start generated code]*/ static PyObject * _sre_compile_impl(PyObject *module, PyObject *pattern, int flags, PyObject *code, Py_ssize_t groups, PyObject *groupindex, - PyObject *indexgroup) -/*[clinic end generated code: output=ef9c2b3693776404 input=0a68476dbbe5db30]*/ + PyObject *indexgroup, Py_ssize_t repeat_count) +/*[clinic end generated code: output=922af562d51b1657 input=77e39c322501ec2a]*/ { /* "compile" pattern descriptor to pattern object */ @@ -1472,8 +1484,8 @@ _sre_compile_impl(PyObject *module, PyObject *pattern, int flags, self->pattern = pattern; self->flags = flags; - self->groups = groups; + self->repeat_count = repeat_count; if (PyDict_GET_SIZE(groupindex) > 0) { Py_INCREF(groupindex); @@ -1645,7 +1657,7 @@ _validate_charset(SRE_CODE *code, SRE_CODE *end) } static int -_validate_inner(SRE_CODE *code, SRE_CODE *end, Py_ssize_t groups) +_validate_inner(SRE_CODE *code, SRE_CODE *end, PatternObject *self) { /* Some variables are manipulated by the macros above */ SRE_CODE op; @@ -1666,8 +1678,8 @@ _validate_inner(SRE_CODE *code, SRE_CODE *end, Py_ssize_t groups) sre_match() code is robust even if they don't, and the worst you can get is nonsensical match results. */ GET_ARG; - if (arg > 2 * (size_t)groups + 1) { - VTRACE(("arg=%d, groups=%d\n", (int)arg, (int)groups)); + if (arg > 2 * (size_t)self->groups + 1) { + VTRACE(("arg=%d, groups=%d\n", (int)arg, (int)self->groups)); FAIL; } break; @@ -1796,7 +1808,7 @@ _validate_inner(SRE_CODE *code, SRE_CODE *end, Py_ssize_t groups) if (skip == 0) break; /* Stop 2 before the end; we check the JUMP below */ - if (!_validate_inner(code, code+skip-3, groups)) + if (!_validate_inner(code, code+skip-3, self)) FAIL; code += skip-3; /* Check that it ends with a JUMP, and that each JUMP @@ -1825,7 +1837,7 @@ _validate_inner(SRE_CODE *code, SRE_CODE *end, Py_ssize_t groups) FAIL; if (max > SRE_MAXREPEAT) FAIL; - if (!_validate_inner(code, code+skip-4, groups)) + if (!_validate_inner(code, code+skip-4, self)) FAIL; code += skip-4; GET_OP; @@ -1837,7 +1849,7 @@ _validate_inner(SRE_CODE *code, SRE_CODE *end, Py_ssize_t groups) case SRE_OP_REPEAT: case SRE_OP_POSSESSIVE_REPEAT: { - SRE_CODE op1 = op, min, max; + SRE_CODE op1 = op, min, max, repeat_index; GET_SKIP; GET_ARG; min = arg; GET_ARG; max = arg; @@ -1845,9 +1857,17 @@ _validate_inner(SRE_CODE *code, SRE_CODE *end, Py_ssize_t groups) FAIL; if (max > SRE_MAXREPEAT) FAIL; - if (!_validate_inner(code, code+skip-3, groups)) + if (op1 == SRE_OP_REPEAT) { + GET_ARG; repeat_index = arg; + if (repeat_index >= (size_t)self->repeat_count) + FAIL; + skip -= 4; + } else { + skip -= 3; + } + if (!_validate_inner(code, code+skip, self)) FAIL; - code += skip-3; + code += skip; GET_OP; if (op1 == SRE_OP_POSSESSIVE_REPEAT) { if (op != SRE_OP_SUCCESS) @@ -1863,7 +1883,7 @@ _validate_inner(SRE_CODE *code, SRE_CODE *end, Py_ssize_t groups) case SRE_OP_ATOMIC_GROUP: { GET_SKIP; - if (!_validate_inner(code, code+skip-2, groups)) + if (!_validate_inner(code, code+skip-2, self)) FAIL; code += skip-2; GET_OP; @@ -1877,7 +1897,7 @@ _validate_inner(SRE_CODE *code, SRE_CODE *end, Py_ssize_t groups) case SRE_OP_GROUPREF_UNI_IGNORE: case SRE_OP_GROUPREF_LOC_IGNORE: GET_ARG; - if (arg >= (size_t)groups) + if (arg >= (size_t)self->groups) FAIL; break; @@ -1886,7 +1906,7 @@ _validate_inner(SRE_CODE *code, SRE_CODE *end, Py_ssize_t groups) 'group' is either an integer group number or a group name, 'then' and 'else' are sub-regexes, and 'else' is optional. */ GET_ARG; - if (arg >= (size_t)groups) + if (arg >= (size_t)self->groups) FAIL; GET_SKIP_ADJ(1); code--; /* The skip is relative to the first arg! */ @@ -1919,17 +1939,17 @@ _validate_inner(SRE_CODE *code, SRE_CODE *end, Py_ssize_t groups) code[skip-3] == SRE_OP_JUMP) { VTRACE(("both then and else parts present\n")); - if (!_validate_inner(code+1, code+skip-3, groups)) + if (!_validate_inner(code+1, code+skip-3, self)) FAIL; code += skip-2; /* Position after JUMP, at */ GET_SKIP; - if (!_validate_inner(code, code+skip-1, groups)) + if (!_validate_inner(code, code+skip-1, self)) FAIL; code += skip-1; } else { VTRACE(("only a then part present\n")); - if (!_validate_inner(code+1, code+skip-1, groups)) + if (!_validate_inner(code+1, code+skip-1, self)) FAIL; code += skip-1; } @@ -1943,7 +1963,7 @@ _validate_inner(SRE_CODE *code, SRE_CODE *end, Py_ssize_t groups) if (arg & 0x80000000) FAIL; /* Width too large */ /* Stop 1 before the end; we check the SUCCESS below */ - if (!_validate_inner(code+1, code+skip-2, groups)) + if (!_validate_inner(code+1, code+skip-2, self)) FAIL; code += skip-2; GET_OP; @@ -1962,18 +1982,19 @@ _validate_inner(SRE_CODE *code, SRE_CODE *end, Py_ssize_t groups) } static int -_validate_outer(SRE_CODE *code, SRE_CODE *end, Py_ssize_t groups) +_validate_outer(SRE_CODE *code, SRE_CODE *end, PatternObject *self) { - if (groups < 0 || (size_t)groups > SRE_MAXGROUPS || + if (self->groups < 0 || (size_t)self->groups > SRE_MAXGROUPS || + self->repeat_count < 0 || code >= end || end[-1] != SRE_OP_SUCCESS) FAIL; - return _validate_inner(code, end-1, groups); + return _validate_inner(code, end-1, self); } static int _validate(PatternObject *self) { - if (!_validate_outer(self->code, self->code+self->codesize, self->groups)) + if (!_validate_outer(self->code, self->code+self->codesize, self)) { PyErr_SetString(PyExc_RuntimeError, "invalid SRE code"); return 0; diff --git a/Modules/clinic/_sre.c.h b/Modules/clinic/_sre.c.h index 72d772c289ae8..34cbe21f14071 100644 --- a/Modules/clinic/_sre.c.h +++ b/Modules/clinic/_sre.c.h @@ -544,7 +544,7 @@ PyDoc_STRVAR(_sre_SRE_Pattern___deepcopy____doc__, PyDoc_STRVAR(_sre_compile__doc__, "compile($module, /, pattern, flags, code, groups, groupindex,\n" -" indexgroup)\n" +" indexgroup, repeat_count)\n" "--\n" "\n"); @@ -554,23 +554,24 @@ PyDoc_STRVAR(_sre_compile__doc__, static PyObject * _sre_compile_impl(PyObject *module, PyObject *pattern, int flags, PyObject *code, Py_ssize_t groups, PyObject *groupindex, - PyObject *indexgroup); + PyObject *indexgroup, Py_ssize_t repeat_count); static PyObject * _sre_compile(PyObject *module, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) { PyObject *return_value = NULL; - static const char * const _keywords[] = {"pattern", "flags", "code", "groups", "groupindex", "indexgroup", NULL}; + static const char * const _keywords[] = {"pattern", "flags", "code", "groups", "groupindex", "indexgroup", "repeat_count", NULL}; static _PyArg_Parser _parser = {NULL, _keywords, "compile", 0}; - PyObject *argsbuf[6]; + PyObject *argsbuf[7]; PyObject *pattern; int flags; PyObject *code; Py_ssize_t groups; PyObject *groupindex; PyObject *indexgroup; + Py_ssize_t repeat_count; - args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 6, 6, 0, argsbuf); + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 7, 7, 0, argsbuf); if (!args) { goto exit; } @@ -606,7 +607,19 @@ _sre_compile(PyObject *module, PyObject *const *args, Py_ssize_t nargs, PyObject goto exit; } indexgroup = args[5]; - return_value = _sre_compile_impl(module, pattern, flags, code, groups, groupindex, indexgroup); + { + Py_ssize_t ival = -1; + PyObject *iobj = _PyNumber_Index(args[6]); + if (iobj != NULL) { + ival = PyLong_AsSsize_t(iobj); + Py_DECREF(iobj); + } + if (ival == -1 && PyErr_Occurred()) { + goto exit; + } + repeat_count = ival; + } + return_value = _sre_compile_impl(module, pattern, flags, code, groups, groupindex, indexgroup, repeat_count); exit: return return_value; @@ -910,4 +923,4 @@ _sre_SRE_Scanner_search(ScannerObject *self, PyTypeObject *cls, PyObject *const exit: return return_value; } -/*[clinic end generated code: output=518f7bb775c1184f input=a9049054013a1b77]*/ +/*[clinic end generated code: output=9d7510a57a157a38 input=a9049054013a1b77]*/ diff --git a/Modules/sre.h b/Modules/sre.h index 785adbd003e7f..e2c5277aefb5d 100644 --- a/Modules/sre.h +++ b/Modules/sre.h @@ -29,6 +29,8 @@ typedef struct { Py_ssize_t groups; /* must be first! */ PyObject* groupindex; /* dict */ PyObject* indexgroup; /* tuple */ + /* the number of REPEATs */ + Py_ssize_t repeat_count; /* compatibility */ PyObject* pattern; /* pattern source (or None) */ int flags; /* flags used when compiling pattern source */ @@ -83,6 +85,8 @@ typedef struct { size_t data_stack_base; /* current repeat context */ SRE_REPEAT *repeat; + /* repeat contexts array */ + SRE_REPEAT *repeats_array; } SRE_STATE; typedef struct { diff --git a/Modules/sre_constants.h b/Modules/sre_constants.h index 45395dcea807a..8b249493bd5cd 100644 --- a/Modules/sre_constants.h +++ b/Modules/sre_constants.h @@ -11,7 +11,7 @@ * See the _sre.c file for information on usage and redistribution. */ -#define SRE_MAGIC 20220318 +#define SRE_MAGIC 20220402 #define SRE_OP_FAILURE 0 #define SRE_OP_SUCCESS 1 #define SRE_OP_ANY 2 diff --git a/Modules/sre_lib.h b/Modules/sre_lib.h index 8e4e714eada38..1cc926d956c63 100644 --- a/Modules/sre_lib.h +++ b/Modules/sre_lib.h @@ -1032,16 +1032,14 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) case SRE_OP_REPEAT: /* create repeat context. all the hard work is done by the UNTIL operator (MAX_UNTIL, MIN_UNTIL) */ - /* <1=min> <2=max> item tail */ - TRACE(("|%p|%p|REPEAT %d %d\n", ctx->pattern, ctx->ptr, - ctx->pattern[1], ctx->pattern[2])); + /* <1=min> <2=max> + <3=repeat_index> item tail */ + TRACE(("|%p|%p|REPEAT %d %d %d\n", ctx->pattern, ctx->ptr, + ctx->pattern[1], ctx->pattern[2], ctx->pattern[3])); + + /* install repeat context */ + ctx->u.rep = &state->repeats_array[ctx->pattern[3]]; - /* install new repeat context */ - ctx->u.rep = (SRE_REPEAT*) PyObject_Malloc(sizeof(*ctx->u.rep)); - if (!ctx->u.rep) { - PyErr_NoMemory(); - RETURN_FAILURE; - } ctx->u.rep->count = -1; ctx->u.rep->pattern = ctx->pattern; ctx->u.rep->prev = state->repeat; @@ -1051,7 +1049,6 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) state->ptr = ctx->ptr; DO_JUMP(JUMP_REPEAT, jump_repeat, ctx->pattern+ctx->pattern[0]); state->repeat = ctx->u.rep->prev; - PyObject_Free(ctx->u.rep); if (ret) { RETURN_ON_ERROR(ret); @@ -1061,7 +1058,8 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) case SRE_OP_MAX_UNTIL: /* maximizing repeat */ - /* <1=min> <2=max> item tail */ + /* <1=min> <2=max> + <3=repeat_index> item tail */ /* FIXME: we probably need to deal with zero-width matches in here... */ @@ -1081,7 +1079,7 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) /* not enough matches */ ctx->u.rep->count = ctx->count; DO_JUMP(JUMP_MAX_UNTIL_1, jump_max_until_1, - ctx->u.rep->pattern+3); + ctx->u.rep->pattern+4); if (ret) { RETURN_ON_ERROR(ret); RETURN_SUCCESS; @@ -1103,7 +1101,7 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) DATA_PUSH(&ctx->u.rep->last_ptr); ctx->u.rep->last_ptr = state->ptr; DO_JUMP(JUMP_MAX_UNTIL_2, jump_max_until_2, - ctx->u.rep->pattern+3); + ctx->u.rep->pattern+4); DATA_POP(&ctx->u.rep->last_ptr); if (ret) { MARK_POP_DISCARD(ctx->lastmark); @@ -1128,7 +1126,8 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) case SRE_OP_MIN_UNTIL: /* minimizing repeat */ - /* <1=min> <2=max> item tail */ + /* <1=min> <2=max> + <3=repeat_index> item tail */ ctx->u.rep = state->repeat; if (!ctx->u.rep) @@ -1145,7 +1144,7 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) /* not enough matches */ ctx->u.rep->count = ctx->count; DO_JUMP(JUMP_MIN_UNTIL_1, jump_min_until_1, - ctx->u.rep->pattern+3); + ctx->u.rep->pattern+4); if (ret) { RETURN_ON_ERROR(ret); RETURN_SUCCESS; @@ -1188,7 +1187,7 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) DATA_PUSH(&ctx->u.rep->last_ptr); ctx->u.rep->last_ptr = state->ptr; DO_JUMP(JUMP_MIN_UNTIL_3,jump_min_until_3, - ctx->u.rep->pattern+3); + ctx->u.rep->pattern+4); DATA_POP(&ctx->u.rep->last_ptr); if (ret) { RETURN_ON_ERROR(ret); From webhook-mailer at python.org Sun Apr 3 15:27:48 2022 From: webhook-mailer at python.org (miss-islington) Date: Sun, 03 Apr 2022 19:27:48 -0000 Subject: [Python-checkins] bpo-47205: Skip error check of sched_get/setaffinity on FreeBSD (GH-32285) Message-ID: https://github.com/python/cpython/commit/470dfe20cb6e741c42c52619e122fc218e27aebd commit: 470dfe20cb6e741c42c52619e122fc218e27aebd branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-03T12:27:32-07:00 summary: bpo-47205: Skip error check of sched_get/setaffinity on FreeBSD (GH-32285) (cherry picked from commit b82cdd1dac9a9be52051abd90a1ce69236ac41f4) Co-authored-by: Christian Heimes files: A Misc/NEWS.d/next/Tests/2022-04-03-14-38-21.bpo-47205.hbbTnh.rst M Lib/test/test_posix.py diff --git a/Lib/test/test_posix.py b/Lib/test/test_posix.py index 974edd766cc80..701543bb6ac66 100644 --- a/Lib/test/test_posix.py +++ b/Lib/test/test_posix.py @@ -1179,7 +1179,9 @@ def test_sched_getaffinity(self): mask = posix.sched_getaffinity(0) self.assertIsInstance(mask, set) self.assertGreaterEqual(len(mask), 1) - self.assertRaises(OSError, posix.sched_getaffinity, -1) + if not sys.platform.startswith("freebsd"): + # bpo-47205: does not raise OSError on FreeBSD + self.assertRaises(OSError, posix.sched_getaffinity, -1) for cpu in mask: self.assertIsInstance(cpu, int) self.assertGreaterEqual(cpu, 0) @@ -1197,7 +1199,9 @@ def test_sched_setaffinity(self): self.assertRaises(ValueError, posix.sched_setaffinity, 0, [-10]) self.assertRaises(ValueError, posix.sched_setaffinity, 0, map(int, "0X")) self.assertRaises(OverflowError, posix.sched_setaffinity, 0, [1<<128]) - self.assertRaises(OSError, posix.sched_setaffinity, -1, mask) + if not sys.platform.startswith("freebsd"): + # bpo-47205: does not raise OSError on FreeBSD + self.assertRaises(OSError, posix.sched_setaffinity, -1, mask) def test_rtld_constants(self): # check presence of major RTLD_* constants diff --git a/Misc/NEWS.d/next/Tests/2022-04-03-14-38-21.bpo-47205.hbbTnh.rst b/Misc/NEWS.d/next/Tests/2022-04-03-14-38-21.bpo-47205.hbbTnh.rst new file mode 100644 index 0000000000000..35fd94421326e --- /dev/null +++ b/Misc/NEWS.d/next/Tests/2022-04-03-14-38-21.bpo-47205.hbbTnh.rst @@ -0,0 +1,2 @@ +Skip test for :func:`~os.sched_getaffinity` and +:func:`~os.sched_setaffinity` error case on FreeBSD. From webhook-mailer at python.org Sun Apr 3 15:33:38 2022 From: webhook-mailer at python.org (miss-islington) Date: Sun, 03 Apr 2022 19:33:38 -0000 Subject: [Python-checkins] bpo-46126: Restore 'descriptions' when running tests internally. (GH-32128) Message-ID: https://github.com/python/cpython/commit/84acb5cad1b871bb8729cbf1d036b84cbe1636f0 commit: 84acb5cad1b871bb8729cbf1d036b84cbe1636f0 branch: main author: Jason R. Coombs committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-03T12:33:28-07:00 summary: bpo-46126: Restore 'descriptions' when running tests internally. (GH-32128) This reverts commit a941e5927f7f2540946813606c61c6aea38db426 (GH-30194). Automerge-Triggered-By: GH:jaraco files: A Misc/NEWS.d/next/Tests/2022-03-26-11-41-19.bpo-46126.q14Ioy.rst M Lib/test/support/testresult.py diff --git a/Lib/test/support/testresult.py b/Lib/test/support/testresult.py index eb2279a88f9a0..2cd1366cd8a9e 100644 --- a/Lib/test/support/testresult.py +++ b/Lib/test/support/testresult.py @@ -145,11 +145,7 @@ def get_test_runner_class(verbosity, buffer=False): return functools.partial(unittest.TextTestRunner, resultclass=RegressionTestResult, buffer=buffer, - verbosity=verbosity, - # disable descriptions so errors are - # readily traceable. bpo-46126 - descriptions=False, - ) + verbosity=verbosity) return functools.partial(QuietRegressionTestRunner, buffer=buffer) def get_test_runner(stream, verbosity, capture_output=False): diff --git a/Misc/NEWS.d/next/Tests/2022-03-26-11-41-19.bpo-46126.q14Ioy.rst b/Misc/NEWS.d/next/Tests/2022-03-26-11-41-19.bpo-46126.q14Ioy.rst new file mode 100644 index 0000000000000..0877b0e385743 --- /dev/null +++ b/Misc/NEWS.d/next/Tests/2022-03-26-11-41-19.bpo-46126.q14Ioy.rst @@ -0,0 +1 @@ +Restore 'descriptions' when running tests internally. From webhook-mailer at python.org Sun Apr 3 16:08:40 2022 From: webhook-mailer at python.org (JulienPalard) Date: Sun, 03 Apr 2022 20:08:40 -0000 Subject: [Python-checkins] bpo-42238: [doc]: make suspicious: false positive. (GH-32292) Message-ID: https://github.com/python/cpython/commit/bdc497496548e30fa208a8d98c30bf6d1833ac4c commit: bdc497496548e30fa208a8d98c30bf6d1833ac4c branch: main author: Julien Palard committer: JulienPalard date: 2022-04-03T22:08:29+02:00 summary: bpo-42238: [doc]: make suspicious: false positive. (GH-32292) files: M Doc/tools/susp-ignored.csv diff --git a/Doc/tools/susp-ignored.csv b/Doc/tools/susp-ignored.csv index a5ceff4be0196..88f401fc8370a 100644 --- a/Doc/tools/susp-ignored.csv +++ b/Doc/tools/susp-ignored.csv @@ -392,3 +392,4 @@ whatsnew/changelog,,:CON,": os.path.abspath(?C:CON?) is now fixed to return whatsnew/changelog,,::,Lib/email/mime/nonmultipart.py::MIMENonMultipart library/typing,,`,"assert_type(name, str) # OK, inferred type of `name` is `str`" library/typing,,`,# after which we hope the inferred type will be `int` +whatsnew/changelog,,:company,-V:company/tag From webhook-mailer at python.org Sun Apr 3 16:59:24 2022 From: webhook-mailer at python.org (tiran) Date: Sun, 03 Apr 2022 20:59:24 -0000 Subject: [Python-checkins] bpo-47176: Interrupt handling for wasm32-emscripten builds without pthreads (GH-32209) Message-ID: https://github.com/python/cpython/commit/087d0fa5b97796560c0d8ceab4f0360fd54baf4f commit: 087d0fa5b97796560c0d8ceab4f0360fd54baf4f branch: main author: Hood Chatham committer: tiran date: 2022-04-03T22:58:52+02:00 summary: bpo-47176: Interrupt handling for wasm32-emscripten builds without pthreads (GH-32209) Co-authored-by: Christian Heimes Co-authored-by: Brett Cannon files: A Include/internal/pycore_emscripten_signal.h A Misc/NEWS.d/next/Core and Builtins/2022-04-02-14-32-21.bpo-47176.kTygYI.rst A Python/emscripten_signal.c M Makefile.pre.in M Modules/signalmodule.c M Python/ceval.c M configure M configure.ac diff --git a/Include/internal/pycore_emscripten_signal.h b/Include/internal/pycore_emscripten_signal.h new file mode 100644 index 0000000000000..8b3287d85da4b --- /dev/null +++ b/Include/internal/pycore_emscripten_signal.h @@ -0,0 +1,25 @@ +#ifndef Py_EMSCRIPTEN_SIGNAL_H +#define Py_EMSCRIPTEN_SIGNAL_H + +#if defined(__EMSCRIPTEN__) + +void +_Py_CheckEmscriptenSignals(void); + +void +_Py_CheckEmscriptenSignalsPeriodically(void); + +#define _Py_CHECK_EMSCRIPTEN_SIGNALS() _Py_CheckEmscriptenSignals() + +#define _Py_CHECK_EMSCRIPTEN_SIGNALS_PERIODICALLY() _Py_CheckEmscriptenSignalsPeriodically() + +extern int Py_EMSCRIPTEN_SIGNAL_HANDLING; + +#else + +#define _Py_CHECK_EMSCRIPTEN_SIGNALS() +#define _Py_CHECK_EMSCRIPTEN_SIGNALS_PERIODICALLY() + +#endif // defined(__EMSCRIPTEN__) + +#endif // ndef Py_EMSCRIPTEN_SIGNAL_H diff --git a/Makefile.pre.in b/Makefile.pre.in index 5318a41dc857a..f94ba93cff91c 100644 --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -429,7 +429,8 @@ PYTHON_OBJS= \ Python/$(DYNLOADFILE) \ $(LIBOBJS) \ $(MACHDEP_OBJS) \ - $(DTRACE_OBJS) + $(DTRACE_OBJS) \ + @PLATFORM_OBJS@ ########################################################################## @@ -1608,6 +1609,7 @@ PYTHON_HEADERS= \ $(srcdir)/Include/internal/pycore_unicodeobject.h \ $(srcdir)/Include/internal/pycore_warnings.h \ $(DTRACE_HEADERS) \ + @PLATFORM_HEADERS@ \ \ $(srcdir)/Python/stdlib_module_names.h diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-04-02-14-32-21.bpo-47176.kTygYI.rst b/Misc/NEWS.d/next/Core and Builtins/2022-04-02-14-32-21.bpo-47176.kTygYI.rst new file mode 100644 index 0000000000000..03fe54a372552 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2022-04-02-14-32-21.bpo-47176.kTygYI.rst @@ -0,0 +1,6 @@ +Emscripten builds cannot handle signals in the usual way due to platform +limitations. Python can now handle signals. To use, set +Module.Py_EmscriptenSignalBuffer to be a single byte SharedArrayBuffer and +set Py_EMSCRIPTEN_SIGNAL_HANDLING to 1. Writing a number into the +SharedArrayBuffer will cause the corresponding signal to be raised into the +Python thread. diff --git a/Modules/signalmodule.c b/Modules/signalmodule.c index 9566263a0dd87..1ee5c669df015 100644 --- a/Modules/signalmodule.c +++ b/Modules/signalmodule.c @@ -13,6 +13,7 @@ #include "pycore_pyerrors.h" // _PyErr_SetString() #include "pycore_pylifecycle.h" // NSIG #include "pycore_pystate.h" // _PyThreadState_GET() +#include "pycore_emscripten_signal.h" // _Py_CHECK_EMSCRIPTEN_SIGNALS #ifndef MS_WINDOWS # include "posixmodule.h" @@ -1797,6 +1798,7 @@ PyErr_CheckSignals(void) int _PyErr_CheckSignalsTstate(PyThreadState *tstate) { + _Py_CHECK_EMSCRIPTEN_SIGNALS(); if (!_Py_atomic_load(&is_tripped)) { return 0; } diff --git a/Python/ceval.c b/Python/ceval.c index 43080f8db0422..68d2920727ab0 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -21,6 +21,7 @@ #include "pycore_pystate.h" // _PyInterpreterState_GET() #include "pycore_sysmodule.h" // _PySys_Audit() #include "pycore_tuple.h" // _PyTuple_ITEMS() +#include "pycore_emscripten_signal.h" // _Py_CHECK_EMSCRIPTEN_SIGNALS #include "code.h" #include "pycore_dict.h" @@ -1292,6 +1293,7 @@ eval_frame_handle_pending(PyThreadState *tstate) } #define CHECK_EVAL_BREAKER() \ + _Py_CHECK_EMSCRIPTEN_SIGNALS_PERIODICALLY(); \ if (_Py_atomic_load_relaxed(eval_breaker)) { \ goto handle_eval_breaker; \ } diff --git a/Python/emscripten_signal.c b/Python/emscripten_signal.c new file mode 100644 index 0000000000000..d617ddfeb37c5 --- /dev/null +++ b/Python/emscripten_signal.c @@ -0,0 +1,56 @@ +// To enable signal handling, the embedder should: +// 1. set Module.Py_EmscriptenSignalBuffer = some_shared_array_buffer; +// 2. set the Py_EMSCRIPTEN_SIGNAL_HANDLING flag to 1 as follows: +// Module.HEAP8[Module._Py_EMSCRIPTEN_SIGNAL_HANDLING] = 1 +// +// The address &Py_EMSCRIPTEN_SIGNAL_HANDLING is exported as +// Module._Py_EMSCRIPTEN_SIGNAL_HANDLING. +#include +#include "Python.h" + +EM_JS(int, _Py_CheckEmscriptenSignals_Helper, (void), { + if (!Module.Py_EmscriptenSignalBuffer) { + return 0; + } + try { + let result = Module.Py_EmscriptenSignalBuffer[0]; + Module.Py_EmscriptenSignalBuffer[0] = 0; + return result; + } catch(e) { +#if !defined(NDEBUG) + console.warn("Error occurred while trying to read signal buffer:", e); +#endif + return 0; + } +}); + +EMSCRIPTEN_KEEPALIVE int Py_EMSCRIPTEN_SIGNAL_HANDLING = 0; + +void +_Py_CheckEmscriptenSignals(void) +{ + if (!Py_EMSCRIPTEN_SIGNAL_HANDLING) { + return; + } + int signal = _Py_CheckEmscriptenSignals_Helper(); + if (signal) { + PyErr_SetInterruptEx(signal); + } +} + + +#define PY_EMSCRIPTEN_SIGNAL_INTERVAL 50 +static int emscripten_signal_clock = PY_EMSCRIPTEN_SIGNAL_INTERVAL; + +void +_Py_CheckEmscriptenSignalsPeriodically(void) +{ + if (!Py_EMSCRIPTEN_SIGNAL_HANDLING) { + return; + } + emscripten_signal_clock--; + if (emscripten_signal_clock == 0) { + emscripten_signal_clock = PY_EMSCRIPTEN_SIGNAL_INTERVAL; + _Py_CheckEmscriptenSignals(); + } +} diff --git a/configure b/configure index a5062d7b81573..bb1aa7568233d 100755 --- a/configure +++ b/configure @@ -817,6 +817,8 @@ TRUE MACHDEP_OBJS DYNLOADFILE DLINCLDIR +PLATFORM_OBJS +PLATFORM_HEADERS DTRACE_OBJS DTRACE_HEADERS DFLAGS @@ -13993,6 +13995,21 @@ $as_echo "$ac_cv_dtrace_link" >&6; } fi fi +PLATFORM_HEADERS= +PLATFORM_OBJS= + +case $ac_sys_system in #( + Emscripten) : + + as_fn_append PLATFORM_OBJS ' Python/emscripten_signal.o' + as_fn_append PLATFORM_HEADERS ' $(srcdir)/Include/internal/pycore_emscripten_signal.h' + ;; #( + *) : + ;; +esac + + + # -I${DLINCLDIR} is added to the compile rule for importdl.o DLINCLDIR=. diff --git a/configure.ac b/configure.ac index 84bc9b3ca5cf0..9f0a50ec852b6 100644 --- a/configure.ac +++ b/configure.ac @@ -4187,6 +4187,19 @@ then fi fi +dnl Platform-specific C and header files. +PLATFORM_HEADERS= +PLATFORM_OBJS= + +AS_CASE([$ac_sys_system], + [Emscripten], [ + AS_VAR_APPEND([PLATFORM_OBJS], [' Python/emscripten_signal.o']) + AS_VAR_APPEND([PLATFORM_HEADERS], [' $(srcdir)/Include/internal/pycore_emscripten_signal.h']) + ], +) +AC_SUBST([PLATFORM_HEADERS]) +AC_SUBST([PLATFORM_OBJS]) + # -I${DLINCLDIR} is added to the compile rule for importdl.o AC_SUBST(DLINCLDIR) DLINCLDIR=. From webhook-mailer at python.org Sun Apr 3 18:31:29 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Sun, 03 Apr 2022 22:31:29 -0000 Subject: [Python-checkins] Follow PEP-8 guidelines in tutorial for standard library (GH-26127) Message-ID: https://github.com/python/cpython/commit/6db2db91b96aaa1270c200ec931a2250fe2799c7 commit: 6db2db91b96aaa1270c200ec931a2250fe2799c7 branch: main author: Bob Kline committer: JelleZijlstra date: 2022-04-03T15:31:03-07:00 summary: Follow PEP-8 guidelines in tutorial for standard library (GH-26127) files: M Doc/tutorial/stdlib.rst diff --git a/Doc/tutorial/stdlib.rst b/Doc/tutorial/stdlib.rst index d90dc51c71927..227a6d8651ff5 100644 --- a/Doc/tutorial/stdlib.rst +++ b/Doc/tutorial/stdlib.rst @@ -78,8 +78,9 @@ and an optional number of lines to be displayed:: import argparse - parser = argparse.ArgumentParser(prog = 'top', - description = 'Show top lines from each file') + parser = argparse.ArgumentParser( + prog='top', + description='Show top lines from each file') parser.add_argument('filenames', nargs='+') parser.add_argument('-l', '--lines', type=int, default=10) args = parser.parse_args() From webhook-mailer at python.org Sun Apr 3 18:49:56 2022 From: webhook-mailer at python.org (miss-islington) Date: Sun, 03 Apr 2022 22:49:56 -0000 Subject: [Python-checkins] Follow PEP-8 guidelines in tutorial for standard library (GH-26127) Message-ID: https://github.com/python/cpython/commit/a331d0f627406d621fa8ef1a1407f2c8a4cee4b5 commit: a331d0f627406d621fa8ef1a1407f2c8a4cee4b5 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-03T15:49:52-07:00 summary: Follow PEP-8 guidelines in tutorial for standard library (GH-26127) (cherry picked from commit 6db2db91b96aaa1270c200ec931a2250fe2799c7) Co-authored-by: Bob Kline files: M Doc/tutorial/stdlib.rst diff --git a/Doc/tutorial/stdlib.rst b/Doc/tutorial/stdlib.rst index ac16160b23439..aac1ae3a8a6b4 100644 --- a/Doc/tutorial/stdlib.rst +++ b/Doc/tutorial/stdlib.rst @@ -78,8 +78,9 @@ and an optional number of lines to be displayed:: import argparse - parser = argparse.ArgumentParser(prog = 'top', - description = 'Show top lines from each file') + parser = argparse.ArgumentParser( + prog='top', + description='Show top lines from each file') parser.add_argument('filenames', nargs='+') parser.add_argument('-l', '--lines', type=int, default=10) args = parser.parse_args() From webhook-mailer at python.org Sun Apr 3 18:58:35 2022 From: webhook-mailer at python.org (miss-islington) Date: Sun, 03 Apr 2022 22:58:35 -0000 Subject: [Python-checkins] Follow PEP-8 guidelines in tutorial for standard library (GH-26127) Message-ID: https://github.com/python/cpython/commit/0d0a6f18a09d04244af3a09b50b65c3d0ca43fd2 commit: 0d0a6f18a09d04244af3a09b50b65c3d0ca43fd2 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-03T15:58:21-07:00 summary: Follow PEP-8 guidelines in tutorial for standard library (GH-26127) (cherry picked from commit 6db2db91b96aaa1270c200ec931a2250fe2799c7) Co-authored-by: Bob Kline files: M Doc/tutorial/stdlib.rst diff --git a/Doc/tutorial/stdlib.rst b/Doc/tutorial/stdlib.rst index ab64ca6d400b8..a6399dc1af3a9 100644 --- a/Doc/tutorial/stdlib.rst +++ b/Doc/tutorial/stdlib.rst @@ -78,8 +78,9 @@ and an optional number of lines to be displayed:: import argparse - parser = argparse.ArgumentParser(prog = 'top', - description = 'Show top lines from each file') + parser = argparse.ArgumentParser( + prog='top', + description='Show top lines from each file') parser.add_argument('filenames', nargs='+') parser.add_argument('-l', '--lines', type=int, default=10) args = parser.parse_args() From webhook-mailer at python.org Sun Apr 3 22:47:06 2022 From: webhook-mailer at python.org (methane) Date: Mon, 04 Apr 2022 02:47:06 -0000 Subject: [Python-checkins] bpo-47000: Make `io.text_encoding()` respects UTF-8 mode (GH-32003) Message-ID: https://github.com/python/cpython/commit/4216dce04b7d3f329beaaafc82a77c4ac6cf4d57 commit: 4216dce04b7d3f329beaaafc82a77c4ac6cf4d57 branch: main author: Inada Naoki committer: methane date: 2022-04-04T11:46:57+09:00 summary: bpo-47000: Make `io.text_encoding()` respects UTF-8 mode (GH-32003) Co-authored-by: Eric Snow files: A Misc/NEWS.d/next/Library/2022-03-20-13-00-08.bpo-47000.p8HpG0.rst M Doc/library/io.rst M Include/internal/pycore_global_strings.h M Include/internal/pycore_runtime_init.h M Lib/_pyio.py M Lib/test/test_io.py M Lib/test/test_utf8_mode.py M Modules/_io/_iomodule.c M Modules/_io/clinic/_iomodule.c.h M Python/sysmodule.c diff --git a/Doc/library/io.rst b/Doc/library/io.rst index d5123348195bd..80107d539505c 100644 --- a/Doc/library/io.rst +++ b/Doc/library/io.rst @@ -198,12 +198,13 @@ High-level Module Interface This is a helper function for callables that use :func:`open` or :class:`TextIOWrapper` and have an ``encoding=None`` parameter. - This function returns *encoding* if it is not ``None`` and ``"locale"`` if - *encoding* is ``None``. + This function returns *encoding* if it is not ``None``. + Otherwise, it returns ``"locale"`` or ``"utf-8"`` depending on + :ref:`UTF-8 Mode `. This function emits an :class:`EncodingWarning` if :data:`sys.flags.warn_default_encoding ` is true and *encoding* - is None. *stacklevel* specifies where the warning is emitted. + is ``None``. *stacklevel* specifies where the warning is emitted. For example:: def read_text(path, encoding=None): @@ -218,6 +219,10 @@ High-level Module Interface .. versionadded:: 3.10 + .. versionchanged:: 3.11 + :func:`text_encoding` returns "utf-8" when UTF-8 mode is enabled and + *encoding* is ``None``. + .. exception:: BlockingIOError diff --git a/Include/internal/pycore_global_strings.h b/Include/internal/pycore_global_strings.h index 3e533fd16509f..833ff2710a787 100644 --- a/Include/internal/pycore_global_strings.h +++ b/Include/internal/pycore_global_strings.h @@ -48,6 +48,7 @@ struct _Py_global_strings { STRUCT_FOR_STR(newline, "\n") STRUCT_FOR_STR(open_br, "{") STRUCT_FOR_STR(percent, "%") + STRUCT_FOR_STR(utf_8, "utf-8") } literals; struct { diff --git a/Include/internal/pycore_runtime_init.h b/Include/internal/pycore_runtime_init.h index d5690d83a0482..fd925b3e060df 100644 --- a/Include/internal/pycore_runtime_init.h +++ b/Include/internal/pycore_runtime_init.h @@ -672,6 +672,7 @@ extern "C" { INIT_STR(newline, "\n"), \ INIT_STR(open_br, "{"), \ INIT_STR(percent, "%"), \ + INIT_STR(utf_8, "utf-8"), \ }, \ .identifiers = { \ INIT_ID(False), \ diff --git a/Lib/_pyio.py b/Lib/_pyio.py index fd00d6536c076..e3ff59eb1adb1 100644 --- a/Lib/_pyio.py +++ b/Lib/_pyio.py @@ -44,8 +44,9 @@ def text_encoding(encoding, stacklevel=2): """ A helper function to choose the text encoding. - When encoding is not None, just return it. - Otherwise, return the default text encoding (i.e. "locale"). + When encoding is not None, this function returns it. + Otherwise, this function returns the default text encoding + (i.e. "locale" or "utf-8" depends on UTF-8 mode). This function emits an EncodingWarning if *encoding* is None and sys.flags.warn_default_encoding is true. @@ -55,7 +56,10 @@ def text_encoding(encoding, stacklevel=2): However, please consider using encoding="utf-8" for new APIs. """ if encoding is None: - encoding = "locale" + if sys.flags.utf8_mode: + encoding = "utf-8" + else: + encoding = "locale" if sys.flags.warn_default_encoding: import warnings warnings.warn("'encoding' argument not specified.", diff --git a/Lib/test/test_io.py b/Lib/test/test_io.py index 2d0ca878788f2..67be108d2526f 100644 --- a/Lib/test/test_io.py +++ b/Lib/test/test_io.py @@ -4289,6 +4289,17 @@ def test_check_encoding_warning(self): self.assertTrue( warnings[1].startswith(b":8: EncodingWarning: ")) + def test_text_encoding(self): + # PEP 597, bpo-47000. io.text_encoding() returns "locale" or "utf-8" + # based on sys.flags.utf8_mode + code = "import io; print(io.text_encoding(None))" + + proc = assert_python_ok('-X', 'utf8=0', '-c', code) + self.assertEqual(b"locale", proc.out.strip()) + + proc = assert_python_ok('-X', 'utf8=1', '-c', code) + self.assertEqual(b"utf-8", proc.out.strip()) + @support.cpython_only # Depending if OpenWrapper was already created or not, the warning is # emitted or not. For example, the attribute is already created when this diff --git a/Lib/test/test_utf8_mode.py b/Lib/test/test_utf8_mode.py index 2b96f76df305f..308e8e8aea6c2 100644 --- a/Lib/test/test_utf8_mode.py +++ b/Lib/test/test_utf8_mode.py @@ -161,7 +161,7 @@ def test_io(self): filename = __file__ out = self.get_output('-c', code, filename, PYTHONUTF8='1') - self.assertEqual(out, 'UTF-8/strict') + self.assertEqual(out.lower(), 'utf-8/strict') def _check_io_encoding(self, module, encoding=None, errors=None): filename = __file__ @@ -183,10 +183,10 @@ def _check_io_encoding(self, module, encoding=None, errors=None): PYTHONUTF8='1') if not encoding: - encoding = 'UTF-8' + encoding = 'utf-8' if not errors: errors = 'strict' - self.assertEqual(out, f'{encoding}/{errors}') + self.assertEqual(out.lower(), f'{encoding}/{errors}') def check_io_encoding(self, module): self._check_io_encoding(module, encoding="latin1") diff --git a/Misc/NEWS.d/next/Library/2022-03-20-13-00-08.bpo-47000.p8HpG0.rst b/Misc/NEWS.d/next/Library/2022-03-20-13-00-08.bpo-47000.p8HpG0.rst new file mode 100644 index 0000000000000..f96b6e627ed11 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-03-20-13-00-08.bpo-47000.p8HpG0.rst @@ -0,0 +1 @@ +Make :func:`io.text_encoding` returns "utf-8" when UTF-8 mode is enabled. diff --git a/Modules/_io/_iomodule.c b/Modules/_io/_iomodule.c index 7f029f26078b8..065f5e29c315b 100644 --- a/Modules/_io/_iomodule.c +++ b/Modules/_io/_iomodule.c @@ -457,8 +457,9 @@ _io.text_encoding A helper function to choose the text encoding. -When encoding is not None, just return it. -Otherwise, return the default text encoding (i.e. "locale"). +When encoding is not None, this function returns it. +Otherwise, this function returns the default text encoding +(i.e. "locale" or "utf-8" depends on UTF-8 mode). This function emits an EncodingWarning if encoding is None and sys.flags.warn_default_encoding is true. @@ -469,7 +470,7 @@ However, please consider using encoding="utf-8" for new APIs. static PyObject * _io_text_encoding_impl(PyObject *module, PyObject *encoding, int stacklevel) -/*[clinic end generated code: output=91b2cfea6934cc0c input=bf70231213e2a7b4]*/ +/*[clinic end generated code: output=91b2cfea6934cc0c input=4999aa8b3d90f3d4]*/ { if (encoding == NULL || encoding == Py_None) { PyInterpreterState *interp = _PyInterpreterState_GET(); @@ -479,7 +480,14 @@ _io_text_encoding_impl(PyObject *module, PyObject *encoding, int stacklevel) return NULL; } } - return &_Py_ID(locale); + const PyPreConfig *preconfig = &_PyRuntime.preconfig; + if (preconfig->utf8_mode) { + _Py_DECLARE_STR(utf_8, "utf-8"); + encoding = &_Py_STR(utf_8); + } + else { + encoding = &_Py_ID(locale); + } } Py_INCREF(encoding); return encoding; diff --git a/Modules/_io/clinic/_iomodule.c.h b/Modules/_io/clinic/_iomodule.c.h index d5fb176eb66be..e4a6b8c42e1d8 100644 --- a/Modules/_io/clinic/_iomodule.c.h +++ b/Modules/_io/clinic/_iomodule.c.h @@ -273,8 +273,9 @@ PyDoc_STRVAR(_io_text_encoding__doc__, "\n" "A helper function to choose the text encoding.\n" "\n" -"When encoding is not None, just return it.\n" -"Otherwise, return the default text encoding (i.e. \"locale\").\n" +"When encoding is not None, this function returns it.\n" +"Otherwise, this function returns the default text encoding\n" +"(i.e. \"locale\" or \"utf-8\" depends on UTF-8 mode).\n" "\n" "This function emits an EncodingWarning if encoding is None and\n" "sys.flags.warn_default_encoding is true.\n" @@ -354,4 +355,4 @@ _io_open_code(PyObject *module, PyObject *const *args, Py_ssize_t nargs, PyObjec exit: return return_value; } -/*[clinic end generated code: output=6ea315343f6a94ba input=a9049054013a1b77]*/ +/*[clinic end generated code: output=1a7fd7755c9a9609 input=a9049054013a1b77]*/ diff --git a/Python/sysmodule.c b/Python/sysmodule.c index 5765e9ef6577c..de4e10a7e110c 100644 --- a/Python/sysmodule.c +++ b/Python/sysmodule.c @@ -841,7 +841,10 @@ static PyObject * sys_getdefaultencoding_impl(PyObject *module) /*[clinic end generated code: output=256d19dfcc0711e6 input=d416856ddbef6909]*/ { - return PyUnicode_FromString(PyUnicode_GetDefaultEncoding()); + _Py_DECLARE_STR(utf_8, "utf-8"); + PyObject *ret = &_Py_STR(utf_8); + Py_INCREF(ret); + return ret; } /*[clinic input] From webhook-mailer at python.org Sun Apr 3 23:27:27 2022 From: webhook-mailer at python.org (ned-deily) Date: Mon, 04 Apr 2022 03:27:27 -0000 Subject: [Python-checkins] bpo-47194: Update zlib to v1.2.12 on Windows to resolve CVE-2018-25032 (GH-32241) (GH-32251) Message-ID: https://github.com/python/cpython/commit/387f93c156288c170ff0016a75af06e109d48ee1 commit: 387f93c156288c170ff0016a75af06e109d48ee1 branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ned-deily date: 2022-04-03T23:27:22-04:00 summary: bpo-47194: Update zlib to v1.2.12 on Windows to resolve CVE-2018-25032 (GH-32241) (GH-32251) (cherry picked from commit 6066739ff7794e54c98c08b953a699cbc961cd28) Co-authored-by: Zachary Ware files: A Misc/NEWS.d/next/Windows/2022-04-01-14-57-40.bpo-47194.IB0XL4.rst M PCbuild/get_externals.bat M PCbuild/python.props diff --git a/Misc/NEWS.d/next/Windows/2022-04-01-14-57-40.bpo-47194.IB0XL4.rst b/Misc/NEWS.d/next/Windows/2022-04-01-14-57-40.bpo-47194.IB0XL4.rst new file mode 100644 index 0000000000000..7e76add45fa95 --- /dev/null +++ b/Misc/NEWS.d/next/Windows/2022-04-01-14-57-40.bpo-47194.IB0XL4.rst @@ -0,0 +1 @@ +Update ``zlib`` to v1.2.12 to resolve CVE-2018-25032. diff --git a/PCbuild/get_externals.bat b/PCbuild/get_externals.bat index 677a6a41ab611..57628396ed06a 100644 --- a/PCbuild/get_externals.bat +++ b/PCbuild/get_externals.bat @@ -55,7 +55,7 @@ if NOT "%IncludeTkinterSrc%"=="false" set libraries=%libraries% tcl-core-8.6.9.0 if NOT "%IncludeTkinterSrc%"=="false" set libraries=%libraries% tk-8.6.9.0 if NOT "%IncludeTkinterSrc%"=="false" set libraries=%libraries% tix-8.4.3.6 set libraries=%libraries% xz-5.2.2 -set libraries=%libraries% zlib-1.2.11 +set libraries=%libraries% zlib-1.2.12 for %%e in (%libraries%) do ( if exist "%EXTERNALS_DIR%\%%e" ( diff --git a/PCbuild/python.props b/PCbuild/python.props index 296bfd637bf9e..1226d623831ba 100644 --- a/PCbuild/python.props +++ b/PCbuild/python.props @@ -53,7 +53,7 @@ $(ExternalsDir)openssl-bin-1.1.1n\$(ArchName)\ $(opensslOutDir)include $(ExternalsDir)\nasm-2.11.06\ - $(ExternalsDir)\zlib-1.2.11\ + $(ExternalsDir)\zlib-1.2.12\ _d From webhook-mailer at python.org Mon Apr 4 03:53:35 2022 From: webhook-mailer at python.org (serhiy-storchaka) Date: Mon, 04 Apr 2022 07:53:35 -0000 Subject: [Python-checkins] bpo-47152: Move sources of the _sre module into a subdirectory (GH-32290) Message-ID: https://github.com/python/cpython/commit/1578f06c1c69fbbb942b90bfbacd512784b599fa commit: 1578f06c1c69fbbb942b90bfbacd512784b599fa branch: main author: Serhiy Storchaka committer: serhiy-storchaka date: 2022-04-04T10:53:26+03:00 summary: bpo-47152: Move sources of the _sre module into a subdirectory (GH-32290) files: A Modules/_sre/clinic/sre.c.h A Modules/_sre/sre.c A Modules/_sre/sre.h A Modules/_sre/sre_constants.h A Modules/_sre/sre_lib.h D Modules/_sre.c D Modules/clinic/_sre.c.h D Modules/sre.h D Modules/sre_constants.h D Modules/sre_lib.h M Lib/re/_constants.py M Makefile.pre.in M Modules/Setup.bootstrap.in M PCbuild/pythoncore.vcxproj M PCbuild/pythoncore.vcxproj.filters M Tools/c-analyzer/cpython/_parser.py M configure M configure.ac diff --git a/Lib/re/_constants.py b/Lib/re/_constants.py index 5317fd53e9c5a..327ba548118b3 100644 --- a/Lib/re/_constants.py +++ b/Lib/re/_constants.py @@ -234,7 +234,7 @@ def dump(f, d, prefix): * * Copyright (c) 1997-2001 by Secret Labs AB. All rights reserved. * - * See the _sre.c file for information on usage and redistribution. + * See the sre.c file for information on usage and redistribution. */ """) diff --git a/Makefile.pre.in b/Makefile.pre.in index f94ba93cff91c..c1e58f7315f49 100644 --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -1226,7 +1226,7 @@ Programs/python.o: $(srcdir)/Programs/python.c Programs/_testembed.o: $(srcdir)/Programs/_testembed.c Programs/test_frozenmain.h $(MAINCC) -c $(PY_CORE_CFLAGS) -o $@ $(srcdir)/Programs/_testembed.c -Modules/_sre.o: $(srcdir)/Modules/_sre.c $(srcdir)/Modules/sre.h $(srcdir)/Modules/sre_constants.h $(srcdir)/Modules/sre_lib.h +Modules/_sre/sre.o: $(srcdir)/Modules/_sre/sre.c $(srcdir)/Modules/_sre/sre.h $(srcdir)/Modules/_sre/sre_constants.h $(srcdir)/Modules/_sre/sre_lib.h Modules/posixmodule.o: $(srcdir)/Modules/posixmodule.c $(srcdir)/Modules/posixmodule.h diff --git a/Modules/Setup.bootstrap.in b/Modules/Setup.bootstrap.in index ec724978f319b..e3e9b96b0630d 100644 --- a/Modules/Setup.bootstrap.in +++ b/Modules/Setup.bootstrap.in @@ -18,7 +18,7 @@ _collections _collectionsmodule.c errno errnomodule.c _io _io/_iomodule.c _io/iobase.c _io/fileio.c _io/bytesio.c _io/bufferedio.c _io/textio.c _io/stringio.c itertools itertoolsmodule.c -_sre _sre.c +_sre _sre/sre.c _thread _threadmodule.c time timemodule.c _weakref _weakref.c diff --git a/Modules/clinic/_sre.c.h b/Modules/_sre/clinic/sre.c.h similarity index 100% rename from Modules/clinic/_sre.c.h rename to Modules/_sre/clinic/sre.c.h diff --git a/Modules/_sre.c b/Modules/_sre/sre.c similarity index 99% rename from Modules/_sre.c rename to Modules/_sre/sre.c index 506363d6fbf6d..491734f243849 100644 --- a/Modules/_sre.c +++ b/Modules/_sre/sre.c @@ -2788,7 +2788,7 @@ pattern_richcompare(PyObject *lefto, PyObject *righto, int op) return PyBool_FromLong(cmp); } -#include "clinic/_sre.c.h" +#include "clinic/sre.c.h" static PyMethodDef pattern_methods[] = { _SRE_SRE_PATTERN_MATCH_METHODDEF diff --git a/Modules/sre.h b/Modules/_sre/sre.h similarity index 97% rename from Modules/sre.h rename to Modules/_sre/sre.h index e2c5277aefb5d..129f5595269f5 100644 --- a/Modules/sre.h +++ b/Modules/_sre/sre.h @@ -5,7 +5,7 @@ * * Copyright (c) 1997-2001 by Secret Labs AB. All rights reserved. * - * See the _sre.c file for information on usage and redistribution. + * See the sre.c file for information on usage and redistribution. */ #ifndef SRE_INCLUDED diff --git a/Modules/sre_constants.h b/Modules/_sre/sre_constants.h similarity index 97% rename from Modules/sre_constants.h rename to Modules/_sre/sre_constants.h index 8b249493bd5cd..3e3643144a92c 100644 --- a/Modules/sre_constants.h +++ b/Modules/_sre/sre_constants.h @@ -8,7 +8,7 @@ * * Copyright (c) 1997-2001 by Secret Labs AB. All rights reserved. * - * See the _sre.c file for information on usage and redistribution. + * See the sre.c file for information on usage and redistribution. */ #define SRE_MAGIC 20220402 diff --git a/Modules/sre_lib.h b/Modules/_sre/sre_lib.h similarity index 99% rename from Modules/sre_lib.h rename to Modules/_sre/sre_lib.h index 1cc926d956c63..34cd0552532f7 100644 --- a/Modules/sre_lib.h +++ b/Modules/_sre/sre_lib.h @@ -5,7 +5,7 @@ * * Copyright (c) 1997-2001 by Secret Labs AB. All rights reserved. * - * See the _sre.c file for information on usage and redistribution. + * See the sre.c file for information on usage and redistribution. */ /* String matching engine */ diff --git a/PCbuild/pythoncore.vcxproj b/PCbuild/pythoncore.vcxproj index 5e6e703df9123..9894e37cb78e1 100644 --- a/PCbuild/pythoncore.vcxproj +++ b/PCbuild/pythoncore.vcxproj @@ -296,9 +296,6 @@ - - - @@ -359,7 +356,10 @@ - + + + + diff --git a/PCbuild/pythoncore.vcxproj.filters b/PCbuild/pythoncore.vcxproj.filters index 86049a2a5df5d..55fca4982e0f3 100644 --- a/PCbuild/pythoncore.vcxproj.filters +++ b/PCbuild/pythoncore.vcxproj.filters @@ -240,15 +240,6 @@ Modules - - Modules - - - Modules - - - Modules - Modules\_io @@ -731,9 +722,18 @@ Modules - + Modules + + Modules + + + Modules + + + Modules + Modules diff --git a/Tools/c-analyzer/cpython/_parser.py b/Tools/c-analyzer/cpython/_parser.py index e58871cf73dce..19000096fc9c3 100644 --- a/Tools/c-analyzer/cpython/_parser.py +++ b/Tools/c-analyzer/cpython/_parser.py @@ -148,7 +148,7 @@ def clean_lines(text): Modules/_localemodule.c Py_BUILD_CORE 1 Modules/_operator.c Py_BUILD_CORE 1 Modules/_posixsubprocess.c Py_BUILD_CORE 1 -Modules/_sre.c Py_BUILD_CORE 1 +Modules/_sre/sre.c Py_BUILD_CORE 1 Modules/_threadmodule.c Py_BUILD_CORE 1 Modules/_tracemalloc.c Py_BUILD_CORE 1 Modules/_weakref.c Py_BUILD_CORE 1 @@ -262,8 +262,8 @@ def clean_lines(text): Modules/_dbmmodule.c HAVE_GDBM_DASH_NDBM_H 1 # others -Modules/sre_lib.h LOCAL(type) static inline type -Modules/sre_lib.h SRE(F) sre_ucs2_##F +Modules/_sre/sre_lib.h LOCAL(type) static inline type +Modules/_sre/sre_lib.h SRE(F) sre_ucs2_##F Objects/stringlib/codecs.h STRINGLIB_IS_UNICODE 1 Include/internal/pycore_bitutils.h _Py__has_builtin(B) 0 diff --git a/configure b/configure index bb1aa7568233d..a06d4c9b41d88 100755 --- a/configure +++ b/configure @@ -20557,6 +20557,7 @@ SRCDIRS="\ Modules/_multiprocessing \ Modules/_sha3 \ Modules/_sqlite \ + Modules/_sre \ Modules/_xxtestfuzz \ Modules/cjkcodecs \ Modules/expat \ diff --git a/configure.ac b/configure.ac index 9f0a50ec852b6..abcd379d705fb 100644 --- a/configure.ac +++ b/configure.ac @@ -5992,6 +5992,7 @@ SRCDIRS="\ Modules/_multiprocessing \ Modules/_sha3 \ Modules/_sqlite \ + Modules/_sre \ Modules/_xxtestfuzz \ Modules/cjkcodecs \ Modules/expat \ From webhook-mailer at python.org Mon Apr 4 05:00:58 2022 From: webhook-mailer at python.org (serhiy-storchaka) Date: Mon, 04 Apr 2022 09:00:58 -0000 Subject: [Python-checkins] bpo-47152: Remove unused import in re (GH-32298) Message-ID: https://github.com/python/cpython/commit/ff2cf1d7d5fb25224f3ff2e0c678d36f78e1f3cb commit: ff2cf1d7d5fb25224f3ff2e0c678d36f78e1f3cb branch: main author: Serhiy Storchaka committer: serhiy-storchaka date: 2022-04-04T12:00:53+03:00 summary: bpo-47152: Remove unused import in re (GH-32298) files: M Lib/re/__init__.py diff --git a/Lib/re/__init__.py b/Lib/re/__init__.py index c47a2650e32f5..b887722bbb2cd 100644 --- a/Lib/re/__init__.py +++ b/Lib/re/__init__.py @@ -124,10 +124,6 @@ import enum from . import _compiler, _parser import functools -try: - import _locale -except ImportError: - _locale = None # public symbols From webhook-mailer at python.org Mon Apr 4 10:09:53 2022 From: webhook-mailer at python.org (markshannon) Date: Mon, 04 Apr 2022 14:09:53 -0000 Subject: [Python-checkins] Revert "bpo-44800: Document internal frame naming conventions (GH-32281)" (#32301) Message-ID: https://github.com/python/cpython/commit/8a349eb30b54bab9a7146fc10e3379c3cacaa19e commit: 8a349eb30b54bab9a7146fc10e3379c3cacaa19e branch: main author: Mark Shannon committer: markshannon date: 2022-04-04T15:09:40+01:00 summary: Revert "bpo-44800: Document internal frame naming conventions (GH-32281)" (#32301) This reverts commit 124227c95f310d2ecd4b567271ab1919fc7000cb. files: M Include/internal/pycore_frame.h diff --git a/Include/internal/pycore_frame.h b/Include/internal/pycore_frame.h index 3ea538ccc5500..211831a6e497f 100644 --- a/Include/internal/pycore_frame.h +++ b/Include/internal/pycore_frame.h @@ -7,75 +7,6 @@ extern "C" { #include #include -/* Starting in CPython 3.11, CPython separates the frame state between the - * full frame objects exposed by the Python and C runtime state introspection - * APIs, and internal lighter weight interpreter frames, which are simple C - * structures owned by either the interpreter eval loop (while executing - * ordinary functions), by a generator or coroutine object (for frames that - * are able to be suspended), or by their corresponding full frame object (if - * a state instrospection API has been invoked and the full frame object has - * taken responsibility for the lifecycle of the interpreter frame). - * - * This split storage eliminates a lot of allocation and deallocation of full - * Python objects during code execution, providing a significant speed gain - * over the previous approach of using full Python objects for both - * introspection and code execution. - * - * Struct names: - * - * * PyFrameObject: the full Python frame object - * * _PyInterpreterFrame: the lightweight frame struct used by the eval loop - * * _PyCFrame: a struct that lives on the C stack and allows Python level - * recursive evaluation to be decoupled from recursive C level invocation - * of the bytecode eval loop - * * See pystate.h for more details on this struct - * - * Field naming conventions: - * - * * full frame object fields have an "f_*" (or "_f_*") prefix - * * new interpreter frame fields have no prefix - * * Several interpreter frame fields have the "f_*" prefix as a result of - * trying to keep diffs as small as was feasible when splitting the original - * frame struct definition in two. The following are all interpreter frame - * fields, NOT full frame object fields: - * * f_func - * * f_globals - * * f_builtins - * * f_locals - * * f_code - * * f_lasti - * * f_state - * * Renaming those fields was considered but ultimately deemed too disruptive - * to key third party projects that were trying to keep up with the Python - * 3.11 code evaluation changes during the alpha release cycle - * (see bpo-44800 for details) - * - * Naming conventions for local variables, function parameters and fields in other structs: - * - * * "frame" and "f" may refer to either full frame objects or interpreter frames - * * the context of use or the field naming conventions usually make the - * type being referenced unambiguous in code reviews - * * the following alternative names are used when more clarity is needed: - * * full frame objects: "frame_obj" (and variants like "frameobj" or "fobj") - * * interpreter frame structs: "frame_data" or "iframe" - * * "current frame" should NOT be abbreviated as "cframe", as the latter now - * typically refers to _PyCFrame structs - * - * Function/macro parameter types: - * - * * "PyFrame_*" functions and other public C API functions that relate to - * frames accept full frame object pointers - * * "_PyFrame_*" functions and other private C API functions that relate to - * frames accept either full frame object or interpreter frame pointers. - * Check the specific function signatures for details. - * - * Function return types: - * - * * Public C API functions will only ever return full frame object pointers - * * Private C API functions with an underscore prefix may return interpreter - * frame pointers instead. Check the specific function signatures for details. - */ - struct _frame { PyObject_HEAD PyFrameObject *f_back; /* previous frame, or NULL */ From webhook-mailer at python.org Mon Apr 4 11:36:22 2022 From: webhook-mailer at python.org (zooba) Date: Mon, 04 Apr 2022 15:36:22 -0000 Subject: [Python-checkins] Demonstrate `py --list` in the quickstart section of the Windows doc page (GH-29383) Message-ID: https://github.com/python/cpython/commit/1ecfe3d5ae4ddec4e73a6cfc93fed6df43fe0be5 commit: 1ecfe3d5ae4ddec4e73a6cfc93fed6df43fe0be5 branch: main author: Christian Clauss committer: zooba date: 2022-04-04T16:35:51+01:00 summary: Demonstrate `py --list` in the quickstart section of the Windows doc page (GH-29383) files: M Doc/using/windows.rst diff --git a/Doc/using/windows.rst b/Doc/using/windows.rst index 7e7be63d7da39..83eee281d4e5c 100644 --- a/Doc/using/windows.rst +++ b/Doc/using/windows.rst @@ -714,6 +714,12 @@ If you see the following error, you do not have the launcher installed: Per-user installations of Python do not add the launcher to :envvar:`PATH` unless the option was selected on installation. +:: + + py --list + +You should see the currently installed versions of Python. + Virtual environments ^^^^^^^^^^^^^^^^^^^^ From webhook-mailer at python.org Mon Apr 4 11:58:53 2022 From: webhook-mailer at python.org (miss-islington) Date: Mon, 04 Apr 2022 15:58:53 -0000 Subject: [Python-checkins] Demonstrate `py --list` in the quickstart section of the Windows doc page (GH-29383) Message-ID: https://github.com/python/cpython/commit/29ffac2d39f7e45834880e359e1bda9a4152f016 commit: 29ffac2d39f7e45834880e359e1bda9a4152f016 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-04T08:58:44-07:00 summary: Demonstrate `py --list` in the quickstart section of the Windows doc page (GH-29383) (cherry picked from commit 1ecfe3d5ae4ddec4e73a6cfc93fed6df43fe0be5) Co-authored-by: Christian Clauss files: M Doc/using/windows.rst diff --git a/Doc/using/windows.rst b/Doc/using/windows.rst index cad7f69806d03..2c348aff12a62 100644 --- a/Doc/using/windows.rst +++ b/Doc/using/windows.rst @@ -710,6 +710,12 @@ If you see the following error, you do not have the launcher installed: Per-user installations of Python do not add the launcher to :envvar:`PATH` unless the option was selected on installation. +:: + + py --list + +You should see the currently installed versions of Python. + Virtual environments ^^^^^^^^^^^^^^^^^^^^ From webhook-mailer at python.org Mon Apr 4 12:02:33 2022 From: webhook-mailer at python.org (miss-islington) Date: Mon, 04 Apr 2022 16:02:33 -0000 Subject: [Python-checkins] Demonstrate `py --list` in the quickstart section of the Windows doc page (GH-29383) Message-ID: https://github.com/python/cpython/commit/a8f29b887e1d8212398508836bf6aaf81dc4f149 commit: a8f29b887e1d8212398508836bf6aaf81dc4f149 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-04T09:02:23-07:00 summary: Demonstrate `py --list` in the quickstart section of the Windows doc page (GH-29383) (cherry picked from commit 1ecfe3d5ae4ddec4e73a6cfc93fed6df43fe0be5) Co-authored-by: Christian Clauss files: M Doc/using/windows.rst diff --git a/Doc/using/windows.rst b/Doc/using/windows.rst index be21b55fe801f..f5f33b917a481 100644 --- a/Doc/using/windows.rst +++ b/Doc/using/windows.rst @@ -716,6 +716,12 @@ If you see the following error, you do not have the launcher installed: Per-user installations of Python do not add the launcher to :envvar:`PATH` unless the option was selected on installation. +:: + + py --list + +You should see the currently installed versions of Python. + Virtual environments ^^^^^^^^^^^^^^^^^^^^ From webhook-mailer at python.org Mon Apr 4 13:13:51 2022 From: webhook-mailer at python.org (tiran) Date: Mon, 04 Apr 2022 17:13:51 -0000 Subject: [Python-checkins] bpo-47208: Allow vendors to override CTYPES_MAX_ARGCOUNT (GH-32297) Message-ID: https://github.com/python/cpython/commit/d1b1c885d8f276a0b1ff2e327270916396a8b842 commit: d1b1c885d8f276a0b1ff2e327270916396a8b842 branch: main author: Christian Heimes committer: tiran date: 2022-04-04T19:13:42+02:00 summary: bpo-47208: Allow vendors to override CTYPES_MAX_ARGCOUNT (GH-32297) files: A Misc/NEWS.d/next/Library/2022-04-04-08-54-31.bpo-47208.cOh9xZ.rst M Lib/ctypes/test/test_callbacks.py M Modules/_ctypes/_ctypes.c M Modules/_ctypes/ctypes.h diff --git a/Lib/ctypes/test/test_callbacks.py b/Lib/ctypes/test/test_callbacks.py index 5561ffefe12f7..1099cf9a69c6b 100644 --- a/Lib/ctypes/test/test_callbacks.py +++ b/Lib/ctypes/test/test_callbacks.py @@ -4,6 +4,7 @@ from ctypes import * from ctypes.test import need_symbol +from _ctypes import CTYPES_MAX_ARGCOUNT import _ctypes_test class Callbacks(unittest.TestCase): @@ -293,8 +294,6 @@ def test_callback_too_many_args(self): def func(*args): return len(args) - CTYPES_MAX_ARGCOUNT = 1024 - # valid call with nargs <= CTYPES_MAX_ARGCOUNT proto = CFUNCTYPE(c_int, *(c_int,) * CTYPES_MAX_ARGCOUNT) cb = proto(func) diff --git a/Misc/NEWS.d/next/Library/2022-04-04-08-54-31.bpo-47208.cOh9xZ.rst b/Misc/NEWS.d/next/Library/2022-04-04-08-54-31.bpo-47208.cOh9xZ.rst new file mode 100644 index 0000000000000..a5da321f77a6a --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-04-08-54-31.bpo-47208.cOh9xZ.rst @@ -0,0 +1 @@ +Allow vendors to override :const:`CTYPES_MAX_ARGCOUNT`. diff --git a/Modules/_ctypes/_ctypes.c b/Modules/_ctypes/_ctypes.c index 9177225f3f929..0415923694129 100644 --- a/Modules/_ctypes/_ctypes.c +++ b/Modules/_ctypes/_ctypes.c @@ -5781,6 +5781,7 @@ _ctypes_add_objects(PyObject *mod) #endif MOD_ADD("RTLD_LOCAL", PyLong_FromLong(RTLD_LOCAL)); MOD_ADD("RTLD_GLOBAL", PyLong_FromLong(RTLD_GLOBAL)); + MOD_ADD("CTYPES_MAX_ARGCOUNT", PyLong_FromLong(CTYPES_MAX_ARGCOUNT)); MOD_ADD("ArgumentError", Py_NewRef(PyExc_ArgError)); return 0; #undef MOD_ADD diff --git a/Modules/_ctypes/ctypes.h b/Modules/_ctypes/ctypes.h index 0badb48b2a410..da1941caf3927 100644 --- a/Modules/_ctypes/ctypes.h +++ b/Modules/_ctypes/ctypes.h @@ -18,7 +18,9 @@ * This limit is enforced for the `alloca()` call in `_ctypes_callproc`, * to avoid allocating a massive buffer on the stack. */ -#define CTYPES_MAX_ARGCOUNT 1024 +#ifndef CTYPES_MAX_ARGCOUNT + #define CTYPES_MAX_ARGCOUNT 1024 +#endif typedef struct tagPyCArgObject PyCArgObject; typedef struct tagCDataObject CDataObject; From webhook-mailer at python.org Mon Apr 4 13:17:05 2022 From: webhook-mailer at python.org (iritkatriel) Date: Mon, 04 Apr 2022 17:17:05 -0000 Subject: [Python-checkins] bpo-46484:Add test for Calendar.iterweekdays (GH-30825) Message-ID: https://github.com/python/cpython/commit/48269ea9fdbc5804f80962364f95e69097c417ba commit: 48269ea9fdbc5804f80962364f95e69097c417ba branch: main author: 180909 <734461790 at qq.com> committer: iritkatriel <1055913+iritkatriel at users.noreply.github.com> date: 2022-04-04T18:16:56+01:00 summary: bpo-46484:Add test for Calendar.iterweekdays (GH-30825) files: M Lib/test/test_calendar.py M Misc/ACKS diff --git a/Lib/test/test_calendar.py b/Lib/test/test_calendar.py index e6bd4d03e0f63..f76cbc9472a6e 100644 --- a/Lib/test/test_calendar.py +++ b/Lib/test/test_calendar.py @@ -619,6 +619,14 @@ def test_itermonthdays2(self): self.assertEqual(days[0][1], firstweekday) self.assertEqual(days[-1][1], (firstweekday - 1) % 7) + def test_iterweekdays(self): + week0 = list(range(7)) + for firstweekday in range(7): + cal = calendar.Calendar(firstweekday) + week = list(cal.iterweekdays()) + expected = week0[firstweekday:] + week0[:firstweekday] + self.assertEqual(week, expected) + class MonthCalendarTestCase(unittest.TestCase): def setUp(self): diff --git a/Misc/ACKS b/Misc/ACKS index 72e2cfe13775a..33e1c4c492983 100644 --- a/Misc/ACKS +++ b/Misc/ACKS @@ -1886,6 +1886,7 @@ Jacob Walls Kevin Walzer Rodrigo Steinmuller Wanderley Dingyuan Wang +Jiahua Wang Ke Wang Liang-Bo Wang Greg Ward From webhook-mailer at python.org Mon Apr 4 13:31:42 2022 From: webhook-mailer at python.org (tiran) Date: Mon, 04 Apr 2022 17:31:42 -0000 Subject: [Python-checkins] bpo-40280: Add --enable-wasm-dynamic-linking (GH-32253) Message-ID: https://github.com/python/cpython/commit/c9844cb8aa9615cdc8770d1e43ce6e2ac3efd836 commit: c9844cb8aa9615cdc8770d1e43ce6e2ac3efd836 branch: main author: Christian Heimes committer: tiran date: 2022-04-04T19:31:31+02:00 summary: bpo-40280: Add --enable-wasm-dynamic-linking (GH-32253) files: A Misc/NEWS.d/next/Build/2022-04-02-17-52-38.bpo-40280.U8Dd0H.rst M Doc/using/configure.rst M Tools/wasm/README.md M Tools/wasm/config.site-wasm32-emscripten M configure M configure.ac diff --git a/Doc/using/configure.rst b/Doc/using/configure.rst index debbee7117f58..b46157cc6ad59 100644 --- a/Doc/using/configure.rst +++ b/Doc/using/configure.rst @@ -52,11 +52,13 @@ General Options Set the Python executable suffix to *SUFFIX*. The default suffix is ``.exe`` on Windows and macOS (``python.exe`` - executable), ``.wasm`` on Emscripten (``python.wasm`` executable), and - an empty string on other platforms (``python`` executable). + executable), ``.js`` on Emscripten node, ``.html`` on Emscripten browser, + ``.wasm`` on WASI, and an empty string on other platforms (``python`` + executable). .. versionchanged:: 3.11 - The default suffix on Emscripten platform is ``.wasm``. + The default suffix on WASM platform is one of ``.js``, ``.html`` + or ``.wasm``. .. cmdoption:: --with-tzpath= @@ -141,6 +143,27 @@ General Options .. versionadded:: 3.11 +WebAssemby Options +------------------ + +.. cmdoption:: --with-emscripten-target=[browser|node] + + Set build flavor for ``wasm32-emscripten``. + + * ``browser`` (default): preload minimal stdlib, default MEMFS. + * ``node``: NODERAWFS and pthread support. + + .. versionadded:: 3.11 + +.. cmdoption:: --enable-wasm-dynamic-linking + + Turn on dynamic linking support for WASM. + + Dynamic linking enables ``dlopen``. File size of the executable + increases due to limited dead code elimination and additional features. + + .. versionadded:: 3.11 + Install Options --------------- diff --git a/Misc/NEWS.d/next/Build/2022-04-02-17-52-38.bpo-40280.U8Dd0H.rst b/Misc/NEWS.d/next/Build/2022-04-02-17-52-38.bpo-40280.U8Dd0H.rst new file mode 100644 index 0000000000000..74fe5c7e49b4a --- /dev/null +++ b/Misc/NEWS.d/next/Build/2022-04-02-17-52-38.bpo-40280.U8Dd0H.rst @@ -0,0 +1,2 @@ +Add configure option :option:`--enable-wasm-dynamic-linking` to enable +``dlopen`` and MAIN_MODULE / SIDE_MODULE on ``wasm32-emscripten``. diff --git a/Tools/wasm/README.md b/Tools/wasm/README.md index 1cdaa4efffb89..6b1e7b03df1e1 100644 --- a/Tools/wasm/README.md +++ b/Tools/wasm/README.md @@ -81,24 +81,31 @@ node --experimental-wasm-threads --experimental-wasm-bulk-memory builddir/emscri ## wasm32-emscripten limitations and issues +- Heap and stack are limited. - Most stdlib modules with a dependency on external libraries are missing: ``ctypes``, ``readline``, ``sqlite3``, ``ssl``, and more. - Shared extension modules are not implemented yet. All extension modules are statically linked into the main binary. + The experimental configure option ``--enable-wasm-dynamic-linking`` enables + dynamic extensions. - Processes are not supported. System calls like fork, popen, and subprocess fail with ``ENOSYS`` or ``ENOSUP``. +- Only ``AF_INET`` and ``AF_INET6`` with ``SOCK_STREAM`` (TCP) or + ``SOCK_DGRAM`` (UDP) are available. ``AF_UNIX`` is not supported. +- ``socketpair`` does not work. - Blocking sockets are not available and non-blocking sockets don't work correctly, e.g. ``socket.accept`` crashes the runtime. ``gethostbyname`` does not resolve to a real IP address. IPv6 is not available. - The ``select`` module is limited. ``select.select()`` crashes the runtime due to lack of exectfd support. -- The ``*at`` variants of functions (e.g. ``openat``) are not available. - The ``dir_fd`` argument of *os* module functions can't be used. - Signal support is limited. ``signal.alarm``, ``itimer``, ``sigaction`` are not available or do not work correctly. ``SIGTERM`` exits the runtime. - Most user, group, and permission related function and modules are not supported or don't work as expected, e.g.``pwd`` module, ``grp`` module, - ``os.setgroups``, ``os.chown``, and so on. + ``os.setgroups``, ``os.chown``, and so on. ``lchown`` and `lchmod`` are + not available. +- ``umask`` is a no-op. +- hard links (``os.link``) are not supported. - Offset and iovec I/O functions (e.g. ``os.pread``, ``os.preadv``) are not available. - ``os.mknod`` and ``os.mkfifo`` @@ -108,17 +115,9 @@ node --experimental-wasm-threads --experimental-wasm-bulk-memory builddir/emscri - ``mmap`` module is unstable. flush (``msync``) can crash the runtime. - Resource-related functions like ``os.nice`` and most functions of the ``resource`` module are not available. -- Some time and datetime features are broken. ``strftime`` and ``strptime`` - have known bugs, e.g. - [%% quoting](https://github.com/emscripten-core/emscripten/issues/16155), - [%U off-by-one](https://github.com/emscripten-core/emscripten/issues/16156). - Extended glibc formatting features are not available. +- glibc extensions for date and time formatting are not available. - ``locales`` module is affected by musl libc issues, [bpo-46390](https://bugs.python.org/issue46390). -- ``uuid`` module is affected by - [memory leak](https://github.com/emscripten-core/emscripten/issues/16081) - and crasher in Emscripten's ``freeaddrinfo``, -- Recursive ``glob`` leaks file descriptors. - Python's object allocator ``obmalloc`` is disabled by default. - ``ensurepip`` is not available. diff --git a/Tools/wasm/config.site-wasm32-emscripten b/Tools/wasm/config.site-wasm32-emscripten index 60ede49eb3745..33636648eaa52 100644 --- a/Tools/wasm/config.site-wasm32-emscripten +++ b/Tools/wasm/config.site-wasm32-emscripten @@ -91,7 +91,3 @@ ac_cv_func_linkat=no # alarm signal is not delivered, may need a callback into the event loop? ac_cv_func_alarm=no - -# To use dlopen, you need to use Emscripten's linking support, -# see https://emscripten.org/docs/compiling/Dynamic-Linking.html -ac_cv_func_dlopen=no diff --git a/configure b/configure index a06d4c9b41d88..72d88806190e7 100755 --- a/configure +++ b/configure @@ -1017,6 +1017,7 @@ with_framework_name enable_framework with_cxx_main with_emscripten_target +enable_wasm_dynamic_linking with_suffix enable_shared with_static_libpython @@ -1730,6 +1731,9 @@ Optional Features: Unix install. optional INSTALLDIR specifies the installation path. see Mac/README.rst (default is no) + --enable-wasm-dynamic-linking + Enable dynamic linking support for WebAssembly + (default is no) --enable-shared enable building a shared Python library (default is no) --enable-profiling enable C-level code profiling with gprof (default is @@ -6290,6 +6294,30 @@ fi { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_sys_emscripten_target" >&5 $as_echo "$ac_sys_emscripten_target" >&6; } +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for --enable-wasm-dynamic-linking" >&5 +$as_echo_n "checking for --enable-wasm-dynamic-linking... " >&6; } +# Check whether --enable-wasm-dynamic-linking was given. +if test "${enable_wasm_dynamic_linking+set}" = set; then : + enableval=$enable_wasm_dynamic_linking; + case $ac_sys_system in #( + Emscripten) : + ;; #( + WASI) : + as_fn_error $? "WASI dynamic linking is not implemented yet." "$LINENO" 5 ;; #( + *) : + as_fn_error $? "--enable-wasm-dynamic-linking only applies to Emscripten and WASI" "$LINENO" 5 + ;; +esac + +else + + enable_wasm_dynamic_linking=missing + +fi + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $enable_wasm_dynamic_linking" >&5 +$as_echo "$enable_wasm_dynamic_linking" >&6; } + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for --with-suffix" >&5 $as_echo_n "checking for --with-suffix... " >&6; } @@ -7760,18 +7788,20 @@ then fi # WASM flags -# TODO: Add -s MAIN_MODULE=2 for dlopen() support. -# The option disables code elimination, which increases code size of main -# binary. All objects must be built with -fPIC. case $ac_sys_system/$ac_sys_emscripten_target in #( Emscripten/browser*) : - LDFLAGS_NODIST="$LDFLAGS_NODIST -s ALLOW_MEMORY_GROWTH=1" - LINKFORSHARED="--preload-file \$(WASM_ASSETS_DIR)" + LDFLAGS_NODIST="$LDFLAGS_NODIST -sALLOW_MEMORY_GROWTH=1" + LINKFORSHARED="--preload-file=\$(WASM_ASSETS_DIR)" + if test "x$enable_wasm_dynamic_linking" = xyes; then : + + as_fn_append LINKFORSHARED " -sMAIN_MODULE=1" + +fi WASM_ASSETS_DIR=".\$(prefix)" WASM_STDLIB="\$(WASM_ASSETS_DIR)/local/lib/python\$(VERSION)/os.py" if test "$Py_DEBUG" = 'true' -o "$ac_sys_emscripten_target" = "browser-debug"; then - LDFLAGS_NODIST="$LDFLAGS_NODIST -s ASSERTIONS=1" + LDFLAGS_NODIST="$LDFLAGS_NODIST -sASSERTIONS=1" LINKFORSHARED="$LINKFORSHARED -gsource-map --emit-symbol-map" else LINKFORSHARED="$LINKFORSHARED -O2 -g0" @@ -7779,11 +7809,16 @@ case $ac_sys_system/$ac_sys_emscripten_target in #( ;; #( Emscripten/node*) : - LDFLAGS_NODIST="$LDFLAGS_NODIST -s ALLOW_MEMORY_GROWTH=1 -s NODERAWFS=1 -s USE_PTHREADS=1" - LINKFORSHARED="-s PROXY_TO_PTHREAD=1 -s EXIT_RUNTIME=1" + LDFLAGS_NODIST="$LDFLAGS_NODIST -sALLOW_MEMORY_GROWTH=1 -sNODERAWFS=1 -sUSE_PTHREADS=1" + LINKFORSHARED="-sPROXY_TO_PTHREAD=1 -sEXIT_RUNTIME=1" + if test "x$enable_wasm_dynamic_linking" = xyes; then : + + as_fn_append LINKFORSHARED " -sMAIN_MODULE=1" + +fi CFLAGS_NODIST="$CFLAGS_NODIST -pthread" if test "$Py_DEBUG" = 'true' -o "$ac_sys_emscripten_target" = "node-debug"; then - LDFLAGS_NODIST="$LDFLAGS_NODIST -s ASSERTIONS=1" + LDFLAGS_NODIST="$LDFLAGS_NODIST -sASSERTIONS=1" LINKFORSHARED="$LINKFORSHARED -gseparate-dwarf --emit-symbol-map" else LINKFORSHARED="$LINKFORSHARED -O2 -g0" @@ -7808,6 +7843,18 @@ $as_echo "#define _WASI_EMULATED_PROCESS_CLOCKS 1" >>confdefs.h ;; esac +case $enable_wasm_dynamic_linking in #( + yes) : + ac_cv_func_dlopen=yes ;; #( + no) : + ac_cv_func_dlopen=no ;; #( + missing) : + + ;; #( + *) : + ;; +esac + @@ -10469,10 +10516,6 @@ then Linux*|GNU*|QNX*|VxWorks*|Haiku*) LDSHARED='$(CC) -shared' LDCXXSHARED='$(CXX) -shared';; - Emscripten*) - LDSHARED='$(CC) -shared -s SIDE_MODULE=1' - LDCXXSHARED='$(CXX) -shared -s SIDE_MODULE=1' - ;; FreeBSD*) if [ "`$CC -dM -E - &5 $as_echo "$LDSHARED" >&6; } LDCXXSHARED=${LDCXXSHARED-$LDSHARED} + +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking BLDSHARED flags" >&5 +$as_echo_n "checking BLDSHARED flags... " >&6; } BLDSHARED=${BLDSHARED-$LDSHARED} +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $BLDSHARED" >&5 +$as_echo "$BLDSHARED" >&6; } + # CCSHARED are the C *flags* used to create objects to go into a shared # library (module) -- this is only needed for a few systems { $as_echo "$as_me:${as_lineno-$LINENO}: checking CCSHARED" >&5 @@ -10540,6 +10599,12 @@ then fi;; Linux-android*) ;; Linux*|GNU*) CCSHARED="-fPIC";; + Emscripten*|WASI*) + if test "x$enable_wasm_dynamic_linking" = xyes; then : + + CCSHARED="-fPIC" + +fi;; FreeBSD*|NetBSD*|OpenBSD*|DragonFly*) CCSHARED="-fPIC";; Haiku*) CCSHARED="-fPIC";; OpenUNIX*|UnixWare*) @@ -10647,6 +10712,13 @@ then CFLAGSFORSHARED='$(CCSHARED)' esac fi + +if test "x$enable_wasm_dynamic_linking" = xyes; then : + + CFLAGSFORSHARED='$(CCSHARED)' + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: result: $CFLAGSFORSHARED" >&5 $as_echo "$CFLAGSFORSHARED" >&6; } @@ -15379,7 +15451,8 @@ $as_echo "yes" >&6; } fi if test "$have_zlib" = "yes" -a "$ac_sys_system" = "Emscripten" -a "$ZLIB_LIBS" = "-lz"; then - ZLIB_LIBS="-s USE_ZLIB=1" + ZLIB_CFLAGS="-sUSE_ZLIB=1" + ZLIB_LIBS="-sUSE_ZLIB=1" fi if test "x$have_zlib" = xyes; then : @@ -15612,7 +15685,8 @@ $as_echo "yes" >&6; } fi if test "$have_bzip2" = "yes" -a "$ac_sys_system" = "Emscripten" -a "$BZIP2_LIBS" = "-lbz2"; then - BZIP2_LIBS="-s USE_BZIP2=1" + BZIP2_CFLAGS="-sUSE_BZIP2=1" + BZIP2_LIBS="-sUSE_BZIP2=1" fi diff --git a/configure.ac b/configure.ac index abcd379d705fb..fda231214b3f7 100644 --- a/configure.ac +++ b/configure.ac @@ -1107,6 +1107,25 @@ dnl are free to remove them in the future. ]) AC_MSG_RESULT([$ac_sys_emscripten_target]) +dnl On Emscripten dlopen() requires -s MAIN_MODULE and -fPIC. The flags +dnl disables dead code elimination and increases the size of the WASM module +dnl by about 1.5 to 2MB. MAIN_MODULE defines __wasm_mutable_globals__. +dnl See https://emscripten.org/docs/compiling/Dynamic-Linking.html +AC_MSG_CHECKING([for --enable-wasm-dynamic-linking]) +AC_ARG_ENABLE([wasm-dynamic-linking], + [AS_HELP_STRING([--enable-wasm-dynamic-linking], + [Enable dynamic linking support for WebAssembly (default is no)])], +[ + AS_CASE([$ac_sys_system], + [Emscripten], [], + [WASI], [AC_MSG_ERROR([WASI dynamic linking is not implemented yet.])], + [AC_MSG_ERROR([--enable-wasm-dynamic-linking only applies to Emscripten and WASI])] + ) +], [ + enable_wasm_dynamic_linking=missing +]) +AC_MSG_RESULT([$enable_wasm_dynamic_linking]) + AC_MSG_CHECKING([for --with-suffix]) AC_ARG_WITH([suffix], [AS_HELP_STRING([--with-suffix=SUFFIX], [set executable suffix to SUFFIX (default is empty, yes is mapped to '.exe')])], @@ -1890,29 +1909,32 @@ then fi # WASM flags -# TODO: Add -s MAIN_MODULE=2 for dlopen() support. -# The option disables code elimination, which increases code size of main -# binary. All objects must be built with -fPIC. AS_CASE([$ac_sys_system/$ac_sys_emscripten_target], [Emscripten/browser*], [ - LDFLAGS_NODIST="$LDFLAGS_NODIST -s ALLOW_MEMORY_GROWTH=1" - LINKFORSHARED="--preload-file \$(WASM_ASSETS_DIR)" + LDFLAGS_NODIST="$LDFLAGS_NODIST -sALLOW_MEMORY_GROWTH=1" + LINKFORSHARED="--preload-file=\$(WASM_ASSETS_DIR)" + AS_VAR_IF([enable_wasm_dynamic_linking], [yes], [ + AS_VAR_APPEND([LINKFORSHARED], [" -sMAIN_MODULE=1"]) + ]) WASM_ASSETS_DIR=".\$(prefix)" WASM_STDLIB="\$(WASM_ASSETS_DIR)/local/lib/python\$(VERSION)/os.py" dnl separate-dwarf does not seem to work in Chrome DevTools Support. if test "$Py_DEBUG" = 'true' -o "$ac_sys_emscripten_target" = "browser-debug"; then - LDFLAGS_NODIST="$LDFLAGS_NODIST -s ASSERTIONS=1" + LDFLAGS_NODIST="$LDFLAGS_NODIST -sASSERTIONS=1" LINKFORSHARED="$LINKFORSHARED -gsource-map --emit-symbol-map" else LINKFORSHARED="$LINKFORSHARED -O2 -g0" fi ], [Emscripten/node*], [ - LDFLAGS_NODIST="$LDFLAGS_NODIST -s ALLOW_MEMORY_GROWTH=1 -s NODERAWFS=1 -s USE_PTHREADS=1" - LINKFORSHARED="-s PROXY_TO_PTHREAD=1 -s EXIT_RUNTIME=1" + LDFLAGS_NODIST="$LDFLAGS_NODIST -sALLOW_MEMORY_GROWTH=1 -sNODERAWFS=1 -sUSE_PTHREADS=1" + LINKFORSHARED="-sPROXY_TO_PTHREAD=1 -sEXIT_RUNTIME=1" + AS_VAR_IF([enable_wasm_dynamic_linking], [yes], [ + AS_VAR_APPEND([LINKFORSHARED], [" -sMAIN_MODULE=1"]) + ]) CFLAGS_NODIST="$CFLAGS_NODIST -pthread" if test "$Py_DEBUG" = 'true' -o "$ac_sys_emscripten_target" = "node-debug"; then - LDFLAGS_NODIST="$LDFLAGS_NODIST -s ASSERTIONS=1" + LDFLAGS_NODIST="$LDFLAGS_NODIST -sASSERTIONS=1" LINKFORSHARED="$LINKFORSHARED -gseparate-dwarf --emit-symbol-map" else LINKFORSHARED="$LINKFORSHARED -O2 -g0" @@ -1927,6 +1949,12 @@ AS_CASE([$ac_sys_system/$ac_sys_emscripten_target], ] ) +AS_CASE([$enable_wasm_dynamic_linking], + [yes], [ac_cv_func_dlopen=yes], + [no], [ac_cv_func_dlopen=no], + [missing], [] +) + AC_SUBST(BASECFLAGS) AC_SUBST(CFLAGS_NODIST) AC_SUBST(LDFLAGS_NODIST) @@ -2929,10 +2957,6 @@ then Linux*|GNU*|QNX*|VxWorks*|Haiku*) LDSHARED='$(CC) -shared' LDCXXSHARED='$(CXX) -shared';; - Emscripten*) - LDSHARED='$(CC) -shared -s SIDE_MODULE=1' - LDCXXSHARED='$(CXX) -shared -s SIDE_MODULE=1' - ;; FreeBSD*) if [[ "`$CC -dM -E - = 1.2.0], [ ]) if test "$have_zlib" = "yes" -a "$ac_sys_system" = "Emscripten" -a "$ZLIB_LIBS" = "-lz"; then - ZLIB_LIBS="-s USE_ZLIB=1" + ZLIB_CFLAGS="-sUSE_ZLIB=1" + ZLIB_LIBS="-sUSE_ZLIB=1" fi dnl binascii can use zlib for optimized crc32. @@ -4482,7 +4532,8 @@ PKG_CHECK_MODULES([BZIP2], [bzip2], [have_bzip2=yes], [ ]) if test "$have_bzip2" = "yes" -a "$ac_sys_system" = "Emscripten" -a "$BZIP2_LIBS" = "-lbz2"; then - BZIP2_LIBS="-s USE_BZIP2=1" + BZIP2_CFLAGS="-sUSE_BZIP2=1" + BZIP2_LIBS="-sUSE_BZIP2=1" fi From webhook-mailer at python.org Mon Apr 4 13:41:55 2022 From: webhook-mailer at python.org (miss-islington) Date: Mon, 04 Apr 2022 17:41:55 -0000 Subject: [Python-checkins] bpo-46484:Add test for Calendar.iterweekdays (GH-30825) Message-ID: https://github.com/python/cpython/commit/5b4bc61d897c956c245a907807d0073aaa35f1c3 commit: 5b4bc61d897c956c245a907807d0073aaa35f1c3 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-04T10:41:50-07:00 summary: bpo-46484:Add test for Calendar.iterweekdays (GH-30825) (cherry picked from commit 48269ea9fdbc5804f80962364f95e69097c417ba) Co-authored-by: 180909 <734461790 at qq.com> files: M Lib/test/test_calendar.py M Misc/ACKS diff --git a/Lib/test/test_calendar.py b/Lib/test/test_calendar.py index 6241d114d3382..d6170c7e814fa 100644 --- a/Lib/test/test_calendar.py +++ b/Lib/test/test_calendar.py @@ -595,6 +595,14 @@ def test_itermonthdays2(self): self.assertEqual(days[0][1], firstweekday) self.assertEqual(days[-1][1], (firstweekday - 1) % 7) + def test_iterweekdays(self): + week0 = list(range(7)) + for firstweekday in range(7): + cal = calendar.Calendar(firstweekday) + week = list(cal.iterweekdays()) + expected = week0[firstweekday:] + week0[:firstweekday] + self.assertEqual(week, expected) + class MonthCalendarTestCase(unittest.TestCase): def setUp(self): diff --git a/Misc/ACKS b/Misc/ACKS index d043195a7b550..d8bb5e465015d 100644 --- a/Misc/ACKS +++ b/Misc/ACKS @@ -1840,6 +1840,7 @@ Jacob Walls Kevin Walzer Rodrigo Steinmuller Wanderley Dingyuan Wang +Jiahua Wang Ke Wang Liang-Bo Wang Greg Ward From webhook-mailer at python.org Mon Apr 4 13:44:03 2022 From: webhook-mailer at python.org (miss-islington) Date: Mon, 04 Apr 2022 17:44:03 -0000 Subject: [Python-checkins] bpo-46484:Add test for Calendar.iterweekdays (GH-30825) Message-ID: https://github.com/python/cpython/commit/6b4b892e0962a7050c5064133c59955691f9776c commit: 6b4b892e0962a7050c5064133c59955691f9776c branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-04T10:43:38-07:00 summary: bpo-46484:Add test for Calendar.iterweekdays (GH-30825) (cherry picked from commit 48269ea9fdbc5804f80962364f95e69097c417ba) Co-authored-by: 180909 <734461790 at qq.com> files: M Lib/test/test_calendar.py M Misc/ACKS diff --git a/Lib/test/test_calendar.py b/Lib/test/test_calendar.py index 39094ad6fd9ab..5ae2b66ff3b5f 100644 --- a/Lib/test/test_calendar.py +++ b/Lib/test/test_calendar.py @@ -619,6 +619,14 @@ def test_itermonthdays2(self): self.assertEqual(days[0][1], firstweekday) self.assertEqual(days[-1][1], (firstweekday - 1) % 7) + def test_iterweekdays(self): + week0 = list(range(7)) + for firstweekday in range(7): + cal = calendar.Calendar(firstweekday) + week = list(cal.iterweekdays()) + expected = week0[firstweekday:] + week0[:firstweekday] + self.assertEqual(week, expected) + class MonthCalendarTestCase(unittest.TestCase): def setUp(self): diff --git a/Misc/ACKS b/Misc/ACKS index 1f1364e4f0bb8..ab4a037d298b5 100644 --- a/Misc/ACKS +++ b/Misc/ACKS @@ -1875,6 +1875,7 @@ Jacob Walls Kevin Walzer Rodrigo Steinmuller Wanderley Dingyuan Wang +Jiahua Wang Ke Wang Liang-Bo Wang Greg Ward From webhook-mailer at python.org Mon Apr 4 19:37:11 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Mon, 04 Apr 2022 23:37:11 -0000 Subject: [Python-checkins] bpo-47097: Add documentation for TypeVarTuple (#32103) Message-ID: https://github.com/python/cpython/commit/38ae5b8c0c0b64ae6100b0dee8707d5ab769e381 commit: 38ae5b8c0c0b64ae6100b0dee8707d5ab769e381 branch: main author: Matthew Rahtz committer: JelleZijlstra date: 2022-04-04T16:37:01-07:00 summary: bpo-47097: Add documentation for TypeVarTuple (#32103) Co-authored-by: Alex Waygood Co-authored-by: Jelle Zijlstra files: M Doc/library/typing.rst diff --git a/Doc/library/typing.rst b/Doc/library/typing.rst index 4d833dc497f17..37c17c429fa47 100644 --- a/Doc/library/typing.rst +++ b/Doc/library/typing.rst @@ -70,6 +70,8 @@ annotations. These include: *Introducing* :class:`ParamSpec` and :data:`Concatenate` * :pep:`613`: Explicit Type Aliases *Introducing* :data:`TypeAlias` +* :pep:`646`: Variadic Generics + *Introducing* :data:`TypeVarTuple` * :pep:`647`: User-Defined Type Guards *Introducing* :data:`TypeGuard` * :pep:`673`: Self type @@ -1230,6 +1232,123 @@ These are not used in annotations. They are building blocks for creating generic ``covariant=True`` or ``contravariant=True``. See :pep:`484` for more details. By default, type variables are invariant. +.. class:: TypeVarTuple + + Type variable tuple. A specialized form of :class:`Type variable ` + that enables *variadic* generics. + + A normal type variable enables parameterization with a single type. A type + variable tuple, in contrast, allows parameterization with an + *arbitrary* number of types by acting like an *arbitrary* number of type + variables wrapped in a tuple. For example:: + + T = TypeVar('T') + Ts = TypeVarTuple('Ts') + + def remove_first_element(tup: tuple[T, *Ts]) -> tuple[*Ts]: + return tup[1:] + + # T is bound to int, Ts is bound to () + # Return value is (), which has type tuple[()] + remove_first_element(tup=(1,)) + + # T is bound to int, Ts is bound to (str,) + # Return value is ('spam',), which has type tuple[str] + remove_first_element(tup=(1, 'spam')) + + # T is bound to int, Ts is bound to (str, float) + # Return value is ('spam', 3.0), which has type tuple[str, float] + remove_first_element(tup=(1, 'spam', 3.0)) + + Note the use of the unpacking operator ``*`` in ``tuple[T, *Ts]``. + Conceptually, you can think of ``Ts`` as a tuple of type variables + ``(T1, T2, ...)``. ``tuple[T, *Ts]`` would then become + ``tuple[T, *(T1, T2, ...)]``, which is equivalent to + ``tuple[T, T1, T2, ...]``. (Note that in older versions of Python, you might + see this written using :data:`Unpack ` instead, as + ``Unpack[Ts]``.) + + Type variable tuples must *always* be unpacked. This helps distinguish type + variable types from normal type variables:: + + x: Ts # Not valid + x: tuple[Ts] # Not valid + x: tuple[*Ts] # The correct way to to do it + + Type variable tuples can be used in the same contexts as normal type + variables. For example, in class definitions, arguments, and return types:: + + Shape = TypeVarTuple('Shape') + class Array(Generic[*Shape]): + def __getitem__(self, key: tuple[*Shape]) -> float: ... + def __abs__(self) -> Array[*Shape]: ... + def get_shape(self) -> tuple[*Shape]: ... + + Type variable tuples can be happily combined with normal type variables:: + + DType = TypeVar('DType') + + class Array(Generic[DType, *Shape]): # This is fine + pass + + class Array2(Generic[*Shape, DType]): # This would also be fine + pass + + float_array_1d: Array[float, Height] = Array() # Totally fine + int_array_2d: Array[int, Height, Width] = Array() # Yup, fine too + + However, note that at most one type variable tuple may appear in a single + list of type arguments or type parameters:: + + x: tuple[*Ts, *Ts] # Not valid + class Array(Generic[*Shape, *Shape]): # Not valid + pass + + Finally, an unpacked type variable tuple can be used as the type annotation + of ``*args``:: + + def call_soon( + callback: Callable[[*Ts], None], + *args: *Ts + ) -> None: + ... + callback(*args) + + In contrast to non-unpacked annotations of ``*args`` - e.g. ``*args: int``, + which would specify that *all* arguments are ``int`` - ``*args: *Ts`` + enables reference to the types of the *individual* arguments in ``*args``. + Here, this allows us to ensure the types of the ``*args`` passed + to ``call_soon`` match the types of the (positional) arguments of + ``callback``. + + For more details on type variable tuples, see :pep:`646`. + + .. versionadded:: 3.11 + +.. data:: Unpack + + A typing operator that conceptually marks an object as having been + unpacked. For example, using the unpack operator ``*`` on a + :class:`type variable tuple ` is equivalent to using ``Unpack`` + to mark the type variable tuple as having been unpacked:: + + Ts = TypeVarTuple('Ts') + tup: tuple[*Ts] + # Effectively does: + tup: tuple[Unpack[Ts]] + + In fact, ``Unpack`` can be used interchangeably with ``*`` in the context + of types. You might see ``Unpack`` being used explicitly in older versions + of Python, where ``*`` couldn't be used in certain places:: + + # In older versions of Python, TypeVarTuple and Unpack + # are located in the `typing_extensions` backports package. + from typing_extensions import TypeVarTuple, Unpack + + Ts = TypeVarTuple('Ts') + tup: tuple[*Ts] # Syntax error on Python <= 3.10! + tup: tuple[Unpack[Ts]] # Semantically equivalent, and backwards-compatible + .. class:: ParamSpec(name, *, bound=None, covariant=False, contravariant=False) Parameter specification variable. A specialized version of From webhook-mailer at python.org Mon Apr 4 21:18:23 2022 From: webhook-mailer at python.org (ned-deily) Date: Tue, 05 Apr 2022 01:18:23 -0000 Subject: [Python-checkins] Fix "Contributed ... in bpo-bpo-45847" (GH-32299) Message-ID: https://github.com/python/cpython/commit/5a7506de7374c10e6eb9ae5a232cc88d4d588450 commit: 5a7506de7374c10e6eb9ae5a232cc88d4d588450 branch: main author: Oleg Iarygin committer: ned-deily date: 2022-04-04T21:17:54-04:00 summary: Fix "Contributed ... in bpo-bpo-45847" (GH-32299) files: M Doc/whatsnew/3.11.rst diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index 809cbd556c8c7..c607aadb4457c 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -779,7 +779,7 @@ Build Changes libsqlite3, zlib, bzip2, liblzma, libcrypt, Tcl/Tk libs, and uuid flags are detected by ``pkg-config`` (when available). (Contributed by Christian Heimes and Erlend Egeberg Aasland in - :issue:`bpo-45847`, :issue:`45747`, and :issue:`45763`.) + :issue:`45847`, :issue:`45747`, and :issue:`45763`.) .. note:: Use the environment variables ``TCLTK_CFLAGS`` and ``TCLTK_LIBS`` to From webhook-mailer at python.org Mon Apr 4 22:01:42 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Tue, 05 Apr 2022 02:01:42 -0000 Subject: [Python-checkins] bpo-41233: Add links to errnos referenced in exceptions docs (GH-21380) Message-ID: https://github.com/python/cpython/commit/a74892cb2168d249d9a8c53fad605a5def9b41d4 commit: a74892cb2168d249d9a8c53fad605a5def9b41d4 branch: main author: yyyyyyyan <24644216+yyyyyyyan at users.noreply.github.com> committer: JelleZijlstra date: 2022-04-04T19:01:36-07:00 summary: bpo-41233: Add links to errnos referenced in exceptions docs (GH-21380) Co-authored-by: Andrew Kuchling Co-authored-by: Jelle Zijlstra files: A Misc/NEWS.d/next/Documentation/2020-07-07-22-54-51.bpo-41233.lyUJ8L.rst M Doc/library/errno.rst M Doc/library/exceptions.rst M Misc/ACKS diff --git a/Doc/library/errno.rst b/Doc/library/errno.rst index 93bdb6ca9b8e3..035340e256874 100644 --- a/Doc/library/errno.rst +++ b/Doc/library/errno.rst @@ -8,7 +8,7 @@ This module makes available standard ``errno`` system symbols. The value of each symbol is the corresponding integer value. The names and descriptions are -borrowed from :file:`linux/include/errno.h`, which should be pretty +borrowed from :file:`linux/include/errno.h`, which should be all-inclusive. @@ -27,25 +27,26 @@ defined by the module. The specific list of defined symbols is available as .. data:: EPERM - Operation not permitted + Operation not permitted. This error is mapped to the exception + :exc:`PermissionError`. .. data:: ENOENT - No such file or directory + No such file or directory. This error is mapped to the exception + :exc:`FileNotFoundError`. .. data:: ESRCH - No such process + No such process. This error is mapped to the exception + :exc:`ProcessLookupError`. .. data:: EINTR - Interrupted system call. - - .. seealso:: - This error is mapped to the exception :exc:`InterruptedError`. + Interrupted system call. This error is mapped to the exception + :exc:`InterruptedError`. .. data:: EIO @@ -75,12 +76,13 @@ defined by the module. The specific list of defined symbols is available as .. data:: ECHILD - No child processes + No child processes. This error is mapped to the exception + :exc:`ChildProcessError`. .. data:: EAGAIN - Try again + Try again. This error is mapped to the exception :exc:`BlockingIOError`. .. data:: ENOMEM @@ -90,7 +92,8 @@ defined by the module. The specific list of defined symbols is available as .. data:: EACCES - Permission denied + Permission denied. This error is mapped to the exception + :exc:`PermissionError`. .. data:: EFAULT @@ -110,7 +113,8 @@ defined by the module. The specific list of defined symbols is available as .. data:: EEXIST - File exists + File exists. This error is mapped to the exception + :exc:`FileExistsError`. .. data:: EXDEV @@ -125,12 +129,14 @@ defined by the module. The specific list of defined symbols is available as .. data:: ENOTDIR - Not a directory + Not a directory. This error is mapped to the exception + :exc:`NotADirectoryError`. .. data:: EISDIR - Is a directory + Is a directory. This error is mapped to the exception + :exc:`IsADirectoryError`. .. data:: EINVAL @@ -185,7 +191,8 @@ defined by the module. The specific list of defined symbols is available as .. data:: EPIPE - Broken pipe + Broken pipe. This error is mapped to the exception + :exc:`BrokenPipeError`. .. data:: EDOM @@ -230,7 +237,8 @@ defined by the module. The specific list of defined symbols is available as .. data:: EWOULDBLOCK - Operation would block + Operation would block. This error is mapped to the exception + :exc:`BlockingIOError`. .. data:: ENOMSG @@ -540,12 +548,14 @@ defined by the module. The specific list of defined symbols is available as .. data:: ECONNABORTED - Software caused connection abort + Software caused connection abort. This error is mapped to the + exception :exc:`ConnectionAbortedError`. .. data:: ECONNRESET - Connection reset by peer + Connection reset by peer. This error is mapped to the exception + :exc:`ConnectionResetError`. .. data:: ENOBUFS @@ -565,7 +575,8 @@ defined by the module. The specific list of defined symbols is available as .. data:: ESHUTDOWN - Cannot send after transport endpoint shutdown + Cannot send after transport endpoint shutdown. This error is mapped + to the exception :exc:`BrokenPipeError`. .. data:: ETOOMANYREFS @@ -575,12 +586,14 @@ defined by the module. The specific list of defined symbols is available as .. data:: ETIMEDOUT - Connection timed out + Connection timed out. This error is mapped to the exception + :exc:`TimeoutError`. .. data:: ECONNREFUSED - Connection refused + Connection refused. This error is mapped to the exception + :exc:`ConnectionRefusedError`. .. data:: EHOSTDOWN @@ -595,12 +608,14 @@ defined by the module. The specific list of defined symbols is available as .. data:: EALREADY - Operation already in progress + Operation already in progress. This error is mapped to the + exception :exc:`BlockingIOError`. .. data:: EINPROGRESS - Operation now in progress + Operation now in progress. This error is mapped to the exception + :exc:`BlockingIOError`. .. data:: ESTALE diff --git a/Doc/library/exceptions.rst b/Doc/library/exceptions.rst index 9e001c8dcd757..137566e079d20 100644 --- a/Doc/library/exceptions.rst +++ b/Doc/library/exceptions.rst @@ -652,8 +652,8 @@ depending on the system error code. Raised when an operation would block on an object (e.g. socket) set for non-blocking operation. - Corresponds to :c:data:`errno` ``EAGAIN``, ``EALREADY``, - ``EWOULDBLOCK`` and ``EINPROGRESS``. + Corresponds to :c:data:`errno` :py:data:`~errno.EAGAIN`, :py:data:`~errno.EALREADY`, + :py:data:`~errno.EWOULDBLOCK` and :py:data:`~errno.EINPROGRESS`. In addition to those of :exc:`OSError`, :exc:`BlockingIOError` can have one more attribute: @@ -667,7 +667,7 @@ depending on the system error code. .. exception:: ChildProcessError Raised when an operation on a child process failed. - Corresponds to :c:data:`errno` ``ECHILD``. + Corresponds to :c:data:`errno` :py:data:`~errno.ECHILD`. .. exception:: ConnectionError @@ -681,35 +681,35 @@ depending on the system error code. A subclass of :exc:`ConnectionError`, raised when trying to write on a pipe while the other end has been closed, or trying to write on a socket which has been shutdown for writing. - Corresponds to :c:data:`errno` ``EPIPE`` and ``ESHUTDOWN``. + Corresponds to :c:data:`errno` :py:data:`~errno.EPIPE` and :py:data:`~errno.ESHUTDOWN`. .. exception:: ConnectionAbortedError A subclass of :exc:`ConnectionError`, raised when a connection attempt is aborted by the peer. - Corresponds to :c:data:`errno` ``ECONNABORTED``. + Corresponds to :c:data:`errno` :py:data:`~errno.ECONNABORTED`. .. exception:: ConnectionRefusedError A subclass of :exc:`ConnectionError`, raised when a connection attempt is refused by the peer. - Corresponds to :c:data:`errno` ``ECONNREFUSED``. + Corresponds to :c:data:`errno` :py:data:`~errno.ECONNREFUSED`. .. exception:: ConnectionResetError A subclass of :exc:`ConnectionError`, raised when a connection is reset by the peer. - Corresponds to :c:data:`errno` ``ECONNRESET``. + Corresponds to :c:data:`errno` :py:data:`~errno.ECONNRESET`. .. exception:: FileExistsError Raised when trying to create a file or directory which already exists. - Corresponds to :c:data:`errno` ``EEXIST``. + Corresponds to :c:data:`errno` :py:data:`~errno.EEXIST`. .. exception:: FileNotFoundError Raised when a file or directory is requested but doesn't exist. - Corresponds to :c:data:`errno` ``ENOENT``. + Corresponds to :c:data:`errno` :py:data:`~errno.ENOENT`. .. exception:: InterruptedError @@ -725,7 +725,7 @@ depending on the system error code. Raised when a file operation (such as :func:`os.remove`) is requested on a directory. - Corresponds to :c:data:`errno` ``EISDIR``. + Corresponds to :c:data:`errno` :py:data:`~errno.EISDIR`. .. exception:: NotADirectoryError @@ -733,23 +733,23 @@ depending on the system error code. something which is not a directory. On most POSIX platforms, it may also be raised if an operation attempts to open or traverse a non-directory file as if it were a directory. - Corresponds to :c:data:`errno` ``ENOTDIR``. + Corresponds to :c:data:`errno` :py:data:`~errno.ENOTDIR`. .. exception:: PermissionError Raised when trying to run an operation without the adequate access rights - for example filesystem permissions. - Corresponds to :c:data:`errno` ``EACCES`` and ``EPERM``. + Corresponds to :c:data:`errno` :py:data:`~errno.EACCES` and :py:data:`~errno.EPERM`. .. exception:: ProcessLookupError Raised when a given process doesn't exist. - Corresponds to :c:data:`errno` ``ESRCH``. + Corresponds to :c:data:`errno` :py:data:`~errno.ESRCH`. .. exception:: TimeoutError Raised when a system function timed out at the system level. - Corresponds to :c:data:`errno` ``ETIMEDOUT``. + Corresponds to :c:data:`errno` :py:data:`~errno.ETIMEDOUT`. .. versionadded:: 3.3 All the above :exc:`OSError` subclasses were added. diff --git a/Misc/ACKS b/Misc/ACKS index 33e1c4c492983..5e66a2e757adf 100644 --- a/Misc/ACKS +++ b/Misc/ACKS @@ -1298,6 +1298,7 @@ Ken Jin Ooi Piet van Oostrum Tomas Oppelstrup Jason Orendorff +Yan "yyyyyyyan" Orestes Bastien Orivel orlnub123 Douglas Orr diff --git a/Misc/NEWS.d/next/Documentation/2020-07-07-22-54-51.bpo-41233.lyUJ8L.rst b/Misc/NEWS.d/next/Documentation/2020-07-07-22-54-51.bpo-41233.lyUJ8L.rst new file mode 100644 index 0000000000000..ea0643aa00ee0 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2020-07-07-22-54-51.bpo-41233.lyUJ8L.rst @@ -0,0 +1 @@ +Link the errnos referenced in ``Doc/library/exceptions.rst`` to their respective section in ``Doc/library/errno.rst``, and vice versa. Previously this was only done for EINTR and InterruptedError. Patch by Yan "yyyyyyyan" Orestes. From webhook-mailer at python.org Mon Apr 4 22:30:15 2022 From: webhook-mailer at python.org (miss-islington) Date: Tue, 05 Apr 2022 02:30:15 -0000 Subject: [Python-checkins] bpo-41233: Add links to errnos referenced in exceptions docs (GH-21380) Message-ID: https://github.com/python/cpython/commit/3fa800d7a7a405f51e0e8c9b7dac2f2a75c17bb4 commit: 3fa800d7a7a405f51e0e8c9b7dac2f2a75c17bb4 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-04T19:30:06-07:00 summary: bpo-41233: Add links to errnos referenced in exceptions docs (GH-21380) Co-authored-by: Andrew Kuchling Co-authored-by: Jelle Zijlstra (cherry picked from commit a74892cb2168d249d9a8c53fad605a5def9b41d4) Co-authored-by: yyyyyyyan <24644216+yyyyyyyan at users.noreply.github.com> files: A Misc/NEWS.d/next/Documentation/2020-07-07-22-54-51.bpo-41233.lyUJ8L.rst M Doc/library/errno.rst M Doc/library/exceptions.rst M Misc/ACKS diff --git a/Doc/library/errno.rst b/Doc/library/errno.rst index 1cbd51c582c0c..c87da091bf84f 100644 --- a/Doc/library/errno.rst +++ b/Doc/library/errno.rst @@ -8,7 +8,7 @@ This module makes available standard ``errno`` system symbols. The value of each symbol is the corresponding integer value. The names and descriptions are -borrowed from :file:`linux/include/errno.h`, which should be pretty +borrowed from :file:`linux/include/errno.h`, which should be all-inclusive. @@ -27,25 +27,26 @@ defined by the module. The specific list of defined symbols is available as .. data:: EPERM - Operation not permitted + Operation not permitted. This error is mapped to the exception + :exc:`PermissionError`. .. data:: ENOENT - No such file or directory + No such file or directory. This error is mapped to the exception + :exc:`FileNotFoundError`. .. data:: ESRCH - No such process + No such process. This error is mapped to the exception + :exc:`ProcessLookupError`. .. data:: EINTR - Interrupted system call. - - .. seealso:: - This error is mapped to the exception :exc:`InterruptedError`. + Interrupted system call. This error is mapped to the exception + :exc:`InterruptedError`. .. data:: EIO @@ -75,12 +76,13 @@ defined by the module. The specific list of defined symbols is available as .. data:: ECHILD - No child processes + No child processes. This error is mapped to the exception + :exc:`ChildProcessError`. .. data:: EAGAIN - Try again + Try again. This error is mapped to the exception :exc:`BlockingIOError`. .. data:: ENOMEM @@ -90,7 +92,8 @@ defined by the module. The specific list of defined symbols is available as .. data:: EACCES - Permission denied + Permission denied. This error is mapped to the exception + :exc:`PermissionError`. .. data:: EFAULT @@ -110,7 +113,8 @@ defined by the module. The specific list of defined symbols is available as .. data:: EEXIST - File exists + File exists. This error is mapped to the exception + :exc:`FileExistsError`. .. data:: EXDEV @@ -125,12 +129,14 @@ defined by the module. The specific list of defined symbols is available as .. data:: ENOTDIR - Not a directory + Not a directory. This error is mapped to the exception + :exc:`NotADirectoryError`. .. data:: EISDIR - Is a directory + Is a directory. This error is mapped to the exception + :exc:`IsADirectoryError`. .. data:: EINVAL @@ -185,7 +191,8 @@ defined by the module. The specific list of defined symbols is available as .. data:: EPIPE - Broken pipe + Broken pipe. This error is mapped to the exception + :exc:`BrokenPipeError`. .. data:: EDOM @@ -230,7 +237,8 @@ defined by the module. The specific list of defined symbols is available as .. data:: EWOULDBLOCK - Operation would block + Operation would block. This error is mapped to the exception + :exc:`BlockingIOError`. .. data:: ENOMSG @@ -540,12 +548,14 @@ defined by the module. The specific list of defined symbols is available as .. data:: ECONNABORTED - Software caused connection abort + Software caused connection abort. This error is mapped to the + exception :exc:`ConnectionAbortedError`. .. data:: ECONNRESET - Connection reset by peer + Connection reset by peer. This error is mapped to the exception + :exc:`ConnectionResetError`. .. data:: ENOBUFS @@ -565,7 +575,8 @@ defined by the module. The specific list of defined symbols is available as .. data:: ESHUTDOWN - Cannot send after transport endpoint shutdown + Cannot send after transport endpoint shutdown. This error is mapped + to the exception :exc:`BrokenPipeError`. .. data:: ETOOMANYREFS @@ -575,12 +586,14 @@ defined by the module. The specific list of defined symbols is available as .. data:: ETIMEDOUT - Connection timed out + Connection timed out. This error is mapped to the exception + :exc:`TimeoutError`. .. data:: ECONNREFUSED - Connection refused + Connection refused. This error is mapped to the exception + :exc:`ConnectionRefusedError`. .. data:: EHOSTDOWN @@ -595,12 +608,14 @@ defined by the module. The specific list of defined symbols is available as .. data:: EALREADY - Operation already in progress + Operation already in progress. This error is mapped to the + exception :exc:`BlockingIOError`. .. data:: EINPROGRESS - Operation now in progress + Operation now in progress. This error is mapped to the exception + :exc:`BlockingIOError`. .. data:: ESTALE diff --git a/Doc/library/exceptions.rst b/Doc/library/exceptions.rst index 50a3ff3452c99..9a3c92554df63 100644 --- a/Doc/library/exceptions.rst +++ b/Doc/library/exceptions.rst @@ -644,8 +644,8 @@ depending on the system error code. Raised when an operation would block on an object (e.g. socket) set for non-blocking operation. - Corresponds to :c:data:`errno` ``EAGAIN``, ``EALREADY``, - ``EWOULDBLOCK`` and ``EINPROGRESS``. + Corresponds to :c:data:`errno` :py:data:`~errno.EAGAIN`, :py:data:`~errno.EALREADY`, + :py:data:`~errno.EWOULDBLOCK` and :py:data:`~errno.EINPROGRESS`. In addition to those of :exc:`OSError`, :exc:`BlockingIOError` can have one more attribute: @@ -659,7 +659,7 @@ depending on the system error code. .. exception:: ChildProcessError Raised when an operation on a child process failed. - Corresponds to :c:data:`errno` ``ECHILD``. + Corresponds to :c:data:`errno` :py:data:`~errno.ECHILD`. .. exception:: ConnectionError @@ -673,35 +673,35 @@ depending on the system error code. A subclass of :exc:`ConnectionError`, raised when trying to write on a pipe while the other end has been closed, or trying to write on a socket which has been shutdown for writing. - Corresponds to :c:data:`errno` ``EPIPE`` and ``ESHUTDOWN``. + Corresponds to :c:data:`errno` :py:data:`~errno.EPIPE` and :py:data:`~errno.ESHUTDOWN`. .. exception:: ConnectionAbortedError A subclass of :exc:`ConnectionError`, raised when a connection attempt is aborted by the peer. - Corresponds to :c:data:`errno` ``ECONNABORTED``. + Corresponds to :c:data:`errno` :py:data:`~errno.ECONNABORTED`. .. exception:: ConnectionRefusedError A subclass of :exc:`ConnectionError`, raised when a connection attempt is refused by the peer. - Corresponds to :c:data:`errno` ``ECONNREFUSED``. + Corresponds to :c:data:`errno` :py:data:`~errno.ECONNREFUSED`. .. exception:: ConnectionResetError A subclass of :exc:`ConnectionError`, raised when a connection is reset by the peer. - Corresponds to :c:data:`errno` ``ECONNRESET``. + Corresponds to :c:data:`errno` :py:data:`~errno.ECONNRESET`. .. exception:: FileExistsError Raised when trying to create a file or directory which already exists. - Corresponds to :c:data:`errno` ``EEXIST``. + Corresponds to :c:data:`errno` :py:data:`~errno.EEXIST`. .. exception:: FileNotFoundError Raised when a file or directory is requested but doesn't exist. - Corresponds to :c:data:`errno` ``ENOENT``. + Corresponds to :c:data:`errno` :py:data:`~errno.ENOENT`. .. exception:: InterruptedError @@ -717,7 +717,7 @@ depending on the system error code. Raised when a file operation (such as :func:`os.remove`) is requested on a directory. - Corresponds to :c:data:`errno` ``EISDIR``. + Corresponds to :c:data:`errno` :py:data:`~errno.EISDIR`. .. exception:: NotADirectoryError @@ -725,23 +725,23 @@ depending on the system error code. something which is not a directory. On most POSIX platforms, it may also be raised if an operation attempts to open or traverse a non-directory file as if it were a directory. - Corresponds to :c:data:`errno` ``ENOTDIR``. + Corresponds to :c:data:`errno` :py:data:`~errno.ENOTDIR`. .. exception:: PermissionError Raised when trying to run an operation without the adequate access rights - for example filesystem permissions. - Corresponds to :c:data:`errno` ``EACCES`` and ``EPERM``. + Corresponds to :c:data:`errno` :py:data:`~errno.EACCES` and :py:data:`~errno.EPERM`. .. exception:: ProcessLookupError Raised when a given process doesn't exist. - Corresponds to :c:data:`errno` ``ESRCH``. + Corresponds to :c:data:`errno` :py:data:`~errno.ESRCH`. .. exception:: TimeoutError Raised when a system function timed out at the system level. - Corresponds to :c:data:`errno` ``ETIMEDOUT``. + Corresponds to :c:data:`errno` :py:data:`~errno.ETIMEDOUT`. .. versionadded:: 3.3 All the above :exc:`OSError` subclasses were added. diff --git a/Misc/ACKS b/Misc/ACKS index ab4a037d298b5..c6abfac531cbe 100644 --- a/Misc/ACKS +++ b/Misc/ACKS @@ -1291,6 +1291,7 @@ Ken Jin Ooi Piet van Oostrum Tomas Oppelstrup Jason Orendorff +Yan "yyyyyyyan" Orestes Bastien Orivel orlnub123 Douglas Orr diff --git a/Misc/NEWS.d/next/Documentation/2020-07-07-22-54-51.bpo-41233.lyUJ8L.rst b/Misc/NEWS.d/next/Documentation/2020-07-07-22-54-51.bpo-41233.lyUJ8L.rst new file mode 100644 index 0000000000000..ea0643aa00ee0 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2020-07-07-22-54-51.bpo-41233.lyUJ8L.rst @@ -0,0 +1 @@ +Link the errnos referenced in ``Doc/library/exceptions.rst`` to their respective section in ``Doc/library/errno.rst``, and vice versa. Previously this was only done for EINTR and InterruptedError. Patch by Yan "yyyyyyyan" Orestes. From webhook-mailer at python.org Mon Apr 4 22:30:29 2022 From: webhook-mailer at python.org (miss-islington) Date: Tue, 05 Apr 2022 02:30:29 -0000 Subject: [Python-checkins] bpo-41233: Add links to errnos referenced in exceptions docs (GH-21380) Message-ID: https://github.com/python/cpython/commit/e47e6ffed36eb1dd82da3bcadf8b45b894ef4ce2 commit: e47e6ffed36eb1dd82da3bcadf8b45b894ef4ce2 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-04T19:30:25-07:00 summary: bpo-41233: Add links to errnos referenced in exceptions docs (GH-21380) Co-authored-by: Andrew Kuchling Co-authored-by: Jelle Zijlstra (cherry picked from commit a74892cb2168d249d9a8c53fad605a5def9b41d4) Co-authored-by: yyyyyyyan <24644216+yyyyyyyan at users.noreply.github.com> files: A Misc/NEWS.d/next/Documentation/2020-07-07-22-54-51.bpo-41233.lyUJ8L.rst M Doc/library/errno.rst M Doc/library/exceptions.rst M Misc/ACKS diff --git a/Doc/library/errno.rst b/Doc/library/errno.rst index 1cbd51c582c0c..c87da091bf84f 100644 --- a/Doc/library/errno.rst +++ b/Doc/library/errno.rst @@ -8,7 +8,7 @@ This module makes available standard ``errno`` system symbols. The value of each symbol is the corresponding integer value. The names and descriptions are -borrowed from :file:`linux/include/errno.h`, which should be pretty +borrowed from :file:`linux/include/errno.h`, which should be all-inclusive. @@ -27,25 +27,26 @@ defined by the module. The specific list of defined symbols is available as .. data:: EPERM - Operation not permitted + Operation not permitted. This error is mapped to the exception + :exc:`PermissionError`. .. data:: ENOENT - No such file or directory + No such file or directory. This error is mapped to the exception + :exc:`FileNotFoundError`. .. data:: ESRCH - No such process + No such process. This error is mapped to the exception + :exc:`ProcessLookupError`. .. data:: EINTR - Interrupted system call. - - .. seealso:: - This error is mapped to the exception :exc:`InterruptedError`. + Interrupted system call. This error is mapped to the exception + :exc:`InterruptedError`. .. data:: EIO @@ -75,12 +76,13 @@ defined by the module. The specific list of defined symbols is available as .. data:: ECHILD - No child processes + No child processes. This error is mapped to the exception + :exc:`ChildProcessError`. .. data:: EAGAIN - Try again + Try again. This error is mapped to the exception :exc:`BlockingIOError`. .. data:: ENOMEM @@ -90,7 +92,8 @@ defined by the module. The specific list of defined symbols is available as .. data:: EACCES - Permission denied + Permission denied. This error is mapped to the exception + :exc:`PermissionError`. .. data:: EFAULT @@ -110,7 +113,8 @@ defined by the module. The specific list of defined symbols is available as .. data:: EEXIST - File exists + File exists. This error is mapped to the exception + :exc:`FileExistsError`. .. data:: EXDEV @@ -125,12 +129,14 @@ defined by the module. The specific list of defined symbols is available as .. data:: ENOTDIR - Not a directory + Not a directory. This error is mapped to the exception + :exc:`NotADirectoryError`. .. data:: EISDIR - Is a directory + Is a directory. This error is mapped to the exception + :exc:`IsADirectoryError`. .. data:: EINVAL @@ -185,7 +191,8 @@ defined by the module. The specific list of defined symbols is available as .. data:: EPIPE - Broken pipe + Broken pipe. This error is mapped to the exception + :exc:`BrokenPipeError`. .. data:: EDOM @@ -230,7 +237,8 @@ defined by the module. The specific list of defined symbols is available as .. data:: EWOULDBLOCK - Operation would block + Operation would block. This error is mapped to the exception + :exc:`BlockingIOError`. .. data:: ENOMSG @@ -540,12 +548,14 @@ defined by the module. The specific list of defined symbols is available as .. data:: ECONNABORTED - Software caused connection abort + Software caused connection abort. This error is mapped to the + exception :exc:`ConnectionAbortedError`. .. data:: ECONNRESET - Connection reset by peer + Connection reset by peer. This error is mapped to the exception + :exc:`ConnectionResetError`. .. data:: ENOBUFS @@ -565,7 +575,8 @@ defined by the module. The specific list of defined symbols is available as .. data:: ESHUTDOWN - Cannot send after transport endpoint shutdown + Cannot send after transport endpoint shutdown. This error is mapped + to the exception :exc:`BrokenPipeError`. .. data:: ETOOMANYREFS @@ -575,12 +586,14 @@ defined by the module. The specific list of defined symbols is available as .. data:: ETIMEDOUT - Connection timed out + Connection timed out. This error is mapped to the exception + :exc:`TimeoutError`. .. data:: ECONNREFUSED - Connection refused + Connection refused. This error is mapped to the exception + :exc:`ConnectionRefusedError`. .. data:: EHOSTDOWN @@ -595,12 +608,14 @@ defined by the module. The specific list of defined symbols is available as .. data:: EALREADY - Operation already in progress + Operation already in progress. This error is mapped to the + exception :exc:`BlockingIOError`. .. data:: EINPROGRESS - Operation now in progress + Operation now in progress. This error is mapped to the exception + :exc:`BlockingIOError`. .. data:: ESTALE diff --git a/Doc/library/exceptions.rst b/Doc/library/exceptions.rst index 688eeb31b2ab0..b6a5df8775dbd 100644 --- a/Doc/library/exceptions.rst +++ b/Doc/library/exceptions.rst @@ -613,8 +613,8 @@ depending on the system error code. Raised when an operation would block on an object (e.g. socket) set for non-blocking operation. - Corresponds to :c:data:`errno` ``EAGAIN``, ``EALREADY``, - ``EWOULDBLOCK`` and ``EINPROGRESS``. + Corresponds to :c:data:`errno` :py:data:`~errno.EAGAIN`, :py:data:`~errno.EALREADY`, + :py:data:`~errno.EWOULDBLOCK` and :py:data:`~errno.EINPROGRESS`. In addition to those of :exc:`OSError`, :exc:`BlockingIOError` can have one more attribute: @@ -628,7 +628,7 @@ depending on the system error code. .. exception:: ChildProcessError Raised when an operation on a child process failed. - Corresponds to :c:data:`errno` ``ECHILD``. + Corresponds to :c:data:`errno` :py:data:`~errno.ECHILD`. .. exception:: ConnectionError @@ -642,35 +642,35 @@ depending on the system error code. A subclass of :exc:`ConnectionError`, raised when trying to write on a pipe while the other end has been closed, or trying to write on a socket which has been shutdown for writing. - Corresponds to :c:data:`errno` ``EPIPE`` and ``ESHUTDOWN``. + Corresponds to :c:data:`errno` :py:data:`~errno.EPIPE` and :py:data:`~errno.ESHUTDOWN`. .. exception:: ConnectionAbortedError A subclass of :exc:`ConnectionError`, raised when a connection attempt is aborted by the peer. - Corresponds to :c:data:`errno` ``ECONNABORTED``. + Corresponds to :c:data:`errno` :py:data:`~errno.ECONNABORTED`. .. exception:: ConnectionRefusedError A subclass of :exc:`ConnectionError`, raised when a connection attempt is refused by the peer. - Corresponds to :c:data:`errno` ``ECONNREFUSED``. + Corresponds to :c:data:`errno` :py:data:`~errno.ECONNREFUSED`. .. exception:: ConnectionResetError A subclass of :exc:`ConnectionError`, raised when a connection is reset by the peer. - Corresponds to :c:data:`errno` ``ECONNRESET``. + Corresponds to :c:data:`errno` :py:data:`~errno.ECONNRESET`. .. exception:: FileExistsError Raised when trying to create a file or directory which already exists. - Corresponds to :c:data:`errno` ``EEXIST``. + Corresponds to :c:data:`errno` :py:data:`~errno.EEXIST`. .. exception:: FileNotFoundError Raised when a file or directory is requested but doesn't exist. - Corresponds to :c:data:`errno` ``ENOENT``. + Corresponds to :c:data:`errno` :py:data:`~errno.ENOENT`. .. exception:: InterruptedError @@ -686,7 +686,7 @@ depending on the system error code. Raised when a file operation (such as :func:`os.remove`) is requested on a directory. - Corresponds to :c:data:`errno` ``EISDIR``. + Corresponds to :c:data:`errno` :py:data:`~errno.EISDIR`. .. exception:: NotADirectoryError @@ -694,23 +694,23 @@ depending on the system error code. something which is not a directory. On most POSIX platforms, it may also be raised if an operation attempts to open or traverse a non-directory file as if it were a directory. - Corresponds to :c:data:`errno` ``ENOTDIR``. + Corresponds to :c:data:`errno` :py:data:`~errno.ENOTDIR`. .. exception:: PermissionError Raised when trying to run an operation without the adequate access rights - for example filesystem permissions. - Corresponds to :c:data:`errno` ``EACCES`` and ``EPERM``. + Corresponds to :c:data:`errno` :py:data:`~errno.EACCES` and :py:data:`~errno.EPERM`. .. exception:: ProcessLookupError Raised when a given process doesn't exist. - Corresponds to :c:data:`errno` ``ESRCH``. + Corresponds to :c:data:`errno` :py:data:`~errno.ESRCH`. .. exception:: TimeoutError Raised when a system function timed out at the system level. - Corresponds to :c:data:`errno` ``ETIMEDOUT``. + Corresponds to :c:data:`errno` :py:data:`~errno.ETIMEDOUT`. .. versionadded:: 3.3 All the above :exc:`OSError` subclasses were added. diff --git a/Misc/ACKS b/Misc/ACKS index d8bb5e465015d..d6b9650f642a9 100644 --- a/Misc/ACKS +++ b/Misc/ACKS @@ -1269,6 +1269,7 @@ Ethan Onstott Piet van Oostrum Tomas Oppelstrup Jason Orendorff +Yan "yyyyyyyan" Orestes Bastien Orivel orlnub123 Douglas Orr diff --git a/Misc/NEWS.d/next/Documentation/2020-07-07-22-54-51.bpo-41233.lyUJ8L.rst b/Misc/NEWS.d/next/Documentation/2020-07-07-22-54-51.bpo-41233.lyUJ8L.rst new file mode 100644 index 0000000000000..ea0643aa00ee0 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2020-07-07-22-54-51.bpo-41233.lyUJ8L.rst @@ -0,0 +1 @@ +Link the errnos referenced in ``Doc/library/exceptions.rst`` to their respective section in ``Doc/library/errno.rst``, and vice versa. Previously this was only done for EINTR and InterruptedError. Patch by Yan "yyyyyyyan" Orestes. From webhook-mailer at python.org Mon Apr 4 22:34:40 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Tue, 05 Apr 2022 02:34:40 -0000 Subject: [Python-checkins] bpo-47007: [doc] `str` special method lookup (GH-31863) Message-ID: https://github.com/python/cpython/commit/bb86d1d9fbd1888524e04475383f4ea764277f67 commit: bb86d1d9fbd1888524e04475383f4ea764277f67 branch: main author: Vanshaj Singhania <8797467+itsvs at users.noreply.github.com> committer: JelleZijlstra date: 2022-04-04T19:34:28-07:00 summary: bpo-47007: [doc] `str` special method lookup (GH-31863) Clarify the `str()` docs to point out that `object.__str__()` follows special method lookup. Co-authored-by: Jelle Zijlstra files: M Doc/library/stdtypes.rst diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst index cdc4dad067814..6608b7b60ec50 100644 --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -1517,7 +1517,8 @@ multiple fragments. depends on whether *encoding* or *errors* is given, as follows. If neither *encoding* nor *errors* is given, ``str(object)`` returns - :meth:`object.__str__() `, which is the "informal" or nicely + :meth:`type(object).__str__(object) `, + which is the "informal" or nicely printable string representation of *object*. For string objects, this is the string itself. If *object* does not have a :meth:`~object.__str__` method, then :func:`str` falls back to returning From webhook-mailer at python.org Mon Apr 4 22:35:39 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Tue, 05 Apr 2022 02:35:39 -0000 Subject: [Python-checkins] bpo-46998: Allow subclassing Any at runtime (GH-31841) Message-ID: https://github.com/python/cpython/commit/5a4973e29f2f5c4ee8c086f40325786c62381540 commit: 5a4973e29f2f5c4ee8c086f40325786c62381540 branch: main author: Shantanu <12621235+hauntsaninja at users.noreply.github.com> committer: JelleZijlstra date: 2022-04-04T19:35:29-07:00 summary: bpo-46998: Allow subclassing Any at runtime (GH-31841) Co-authored-by: Jelle Zijlstra files: A Misc/NEWS.d/next/Library/2022-03-13-08-52-58.bpo-46998.cHh-9O.rst M Doc/library/typing.rst M Lib/test/test_functools.py M Lib/test/test_pydoc.py M Lib/test/test_typing.py M Lib/typing.py diff --git a/Doc/library/typing.rst b/Doc/library/typing.rst index 37c17c429fa47..0a4e848c67736 100644 --- a/Doc/library/typing.rst +++ b/Doc/library/typing.rst @@ -580,6 +580,11 @@ These can be used as types in annotations and do not support ``[]``. * Every type is compatible with :data:`Any`. * :data:`Any` is compatible with every type. + .. versionchanged:: 3.11 + :data:`Any` can now be used as a base class. This can be useful for + avoiding type checker errors with classes that can duck type anywhere or + are highly dynamic. + .. data:: Never The `bottom type `_, diff --git a/Lib/test/test_functools.py b/Lib/test/test_functools.py index abbd50a47f395..82e73f46a3fba 100644 --- a/Lib/test/test_functools.py +++ b/Lib/test/test_functools.py @@ -2802,8 +2802,6 @@ def f(arg): f.register(list[int] | str, lambda arg: "types.UnionTypes(types.GenericAlias)") with self.assertRaisesRegex(TypeError, "Invalid first argument to "): f.register(typing.List[float] | bytes, lambda arg: "typing.Union[typing.GenericAlias]") - with self.assertRaisesRegex(TypeError, "Invalid first argument to "): - f.register(typing.Any, lambda arg: "typing.Any") self.assertEqual(f([1]), "default") self.assertEqual(f([1.0]), "default") @@ -2823,8 +2821,6 @@ def f(arg): f.register(list[int] | str) with self.assertRaisesRegex(TypeError, "Invalid first argument to "): f.register(typing.List[int] | str) - with self.assertRaisesRegex(TypeError, "Invalid first argument to "): - f.register(typing.Any) def test_register_genericalias_annotation(self): @functools.singledispatch @@ -2847,10 +2843,6 @@ def _(arg: list[int] | str): @f.register def _(arg: typing.List[float] | bytes): return "typing.Union[typing.GenericAlias]" - with self.assertRaisesRegex(TypeError, "Invalid annotation for 'arg'"): - @f.register - def _(arg: typing.Any): - return "typing.Any" self.assertEqual(f([1]), "default") self.assertEqual(f([1.0]), "default") diff --git a/Lib/test/test_pydoc.py b/Lib/test/test_pydoc.py index 9c900c3e8ee0a..13c77b6fa6822 100644 --- a/Lib/test/test_pydoc.py +++ b/Lib/test/test_pydoc.py @@ -1066,14 +1066,14 @@ def test_union_type(self): self.assertIn(types.UnionType.__doc__.strip().splitlines()[0], doc) def test_special_form(self): - self.assertEqual(pydoc.describe(typing.Any), '_SpecialForm') - doc = pydoc.render_doc(typing.Any, renderer=pydoc.plaintext) + self.assertEqual(pydoc.describe(typing.NoReturn), '_SpecialForm') + doc = pydoc.render_doc(typing.NoReturn, renderer=pydoc.plaintext) self.assertIn('_SpecialForm in module typing', doc) - if typing.Any.__doc__: - self.assertIn('Any = typing.Any', doc) - self.assertIn(typing.Any.__doc__.strip().splitlines()[0], doc) + if typing.NoReturn.__doc__: + self.assertIn('NoReturn = typing.NoReturn', doc) + self.assertIn(typing.NoReturn.__doc__.strip().splitlines()[0], doc) else: - self.assertIn('Any = class _SpecialForm(_Final)', doc) + self.assertIn('NoReturn = class _SpecialForm(_Final)', doc) def test_typing_pydoc(self): def foo(data: typing.List[typing.Any], diff --git a/Lib/test/test_typing.py b/Lib/test/test_typing.py index 0e28655296d14..041b6ad9ed6dd 100644 --- a/Lib/test/test_typing.py +++ b/Lib/test/test_typing.py @@ -89,12 +89,6 @@ def test_any_instance_type_error(self): with self.assertRaises(TypeError): isinstance(42, Any) - def test_any_subclass_type_error(self): - with self.assertRaises(TypeError): - issubclass(Employee, Any) - with self.assertRaises(TypeError): - issubclass(Any, Employee) - def test_repr(self): self.assertEqual(repr(Any), 'typing.Any') @@ -104,13 +98,21 @@ def test_errors(self): with self.assertRaises(TypeError): Any[int] # Any is not a generic type. - def test_cannot_subclass(self): - with self.assertRaises(TypeError): - class A(Any): - pass - with self.assertRaises(TypeError): - class A(type(Any)): - pass + def test_can_subclass(self): + class Mock(Any): pass + self.assertTrue(issubclass(Mock, Any)) + self.assertIsInstance(Mock(), Mock) + + class Something: pass + self.assertFalse(issubclass(Something, Any)) + self.assertNotIsInstance(Something(), Mock) + + class MockSomething(Something, Mock): pass + self.assertTrue(issubclass(MockSomething, Any)) + ms = MockSomething() + self.assertIsInstance(ms, MockSomething) + self.assertIsInstance(ms, Something) + self.assertIsInstance(ms, Mock) def test_cannot_instantiate(self): with self.assertRaises(TypeError): diff --git a/Lib/typing.py b/Lib/typing.py index 36f9eceb38c7c..4636798bd6956 100644 --- a/Lib/typing.py +++ b/Lib/typing.py @@ -429,8 +429,17 @@ def __getitem__(self, parameters): return self._getitem(self, *parameters) - at _SpecialForm -def Any(self, parameters): +class _AnyMeta(type): + def __instancecheck__(self, obj): + if self is Any: + raise TypeError("typing.Any cannot be used with isinstance()") + return super().__instancecheck__(obj) + + def __repr__(self): + return "typing.Any" + + +class Any(metaclass=_AnyMeta): """Special type indicating an unconstrained type. - Any is compatible with every type. @@ -439,9 +448,13 @@ def Any(self, parameters): Note that all the above statements are true from the point of view of static type checkers. At runtime, Any should not be used with instance - or class checks. + checks. """ - raise TypeError(f"{self} is not subscriptable") + def __new__(cls, *args, **kwargs): + if cls is Any: + raise TypeError("Any cannot be instantiated") + return super().__new__(cls, *args, **kwargs) + @_SpecialForm def NoReturn(self, parameters): diff --git a/Misc/NEWS.d/next/Library/2022-03-13-08-52-58.bpo-46998.cHh-9O.rst b/Misc/NEWS.d/next/Library/2022-03-13-08-52-58.bpo-46998.cHh-9O.rst new file mode 100644 index 0000000000000..25b82b5370846 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-03-13-08-52-58.bpo-46998.cHh-9O.rst @@ -0,0 +1 @@ +Allow subclassing of :class:`typing.Any`. Patch by Shantanu Jain. From webhook-mailer at python.org Mon Apr 4 22:36:46 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Tue, 05 Apr 2022 02:36:46 -0000 Subject: [Python-checkins] bpo-45790: List macros in same order in which fields are described (GH-29529) Message-ID: https://github.com/python/cpython/commit/b275267aa7d44ec90fa435c9cb1610c549da745a commit: b275267aa7d44ec90fa435c9cb1610c549da745a branch: main author: rtobar committer: JelleZijlstra date: 2022-04-04T19:36:33-07:00 summary: bpo-45790: List macros in same order in which fields are described (GH-29529) Signed-off-by: Rodrigo Tobar Co-authored-by: Jelle Zijlstra files: A Misc/NEWS.d/next/Documentation/2021-11-12-11-03-55.bpo-45790.6yuhe8.rst M Doc/extending/newtypes_tutorial.rst diff --git a/Doc/extending/newtypes_tutorial.rst b/Doc/extending/newtypes_tutorial.rst index 530e2c4d35f84..904915306f1f3 100644 --- a/Doc/extending/newtypes_tutorial.rst +++ b/Doc/extending/newtypes_tutorial.rst @@ -67,8 +67,8 @@ The first bit is:: This is what a Custom object will contain. ``PyObject_HEAD`` is mandatory at the start of each object struct and defines a field called ``ob_base`` of type :c:type:`PyObject`, containing a pointer to a type object and a -reference count (these can be accessed using the macros :c:macro:`Py_REFCNT` -and :c:macro:`Py_TYPE` respectively). The reason for the macro is to +reference count (these can be accessed using the macros :c:macro:`Py_TYPE` +and :c:macro:`Py_REFCNT` respectively). The reason for the macro is to abstract away the layout and to enable additional fields in :ref:`debug builds `. diff --git a/Misc/NEWS.d/next/Documentation/2021-11-12-11-03-55.bpo-45790.6yuhe8.rst b/Misc/NEWS.d/next/Documentation/2021-11-12-11-03-55.bpo-45790.6yuhe8.rst new file mode 100644 index 0000000000000..41cf2cb91525f --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2021-11-12-11-03-55.bpo-45790.6yuhe8.rst @@ -0,0 +1,2 @@ +Adjust inaccurate phrasing in :doc:`../extending/newtypes_tutorial` about the +``ob_base`` field and the macros used to access its contents. From webhook-mailer at python.org Mon Apr 4 22:47:15 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Tue, 05 Apr 2022 02:47:15 -0000 Subject: [Python-checkins] bpo-32658: Regex docs: Fix metacharacter reference (GH-32230) Message-ID: https://github.com/python/cpython/commit/43571a3eea8b5931769376daf4bdad1c9184ae0d commit: 43571a3eea8b5931769376daf4bdad1c9184ae0d branch: main author: Mike cm committer: JelleZijlstra date: 2022-04-04T19:46:54-07:00 summary: bpo-32658: Regex docs: Fix metacharacter reference (GH-32230) Co-authored-by: Jelle Zijlstra files: M Doc/howto/regex.rst diff --git a/Doc/howto/regex.rst b/Doc/howto/regex.rst index 33dca2fc41f9d..c4eed8fb1fbe7 100644 --- a/Doc/howto/regex.rst +++ b/Doc/howto/regex.rst @@ -89,7 +89,7 @@ is the same as ``[a-c]``, which uses a range to express the same set of characters. If you wanted to match only lowercase letters, your RE would be ``[a-z]``. -Metacharacters are not active inside classes. For example, ``[akm$]`` will +Metacharacters (except ``\``) are not active inside classes. For example, ``[akm$]`` will match any of the characters ``'a'``, ``'k'``, ``'m'``, or ``'$'``; ``'$'`` is usually a metacharacter, but inside a character class it's stripped of its special nature. From webhook-mailer at python.org Mon Apr 4 22:50:35 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Tue, 05 Apr 2022 02:50:35 -0000 Subject: [Python-checkins] crypt docs: Fix references to `methods` attr (GH-26806) Message-ID: https://github.com/python/cpython/commit/cae0f5d3dad6db0d13690e5952ae2015ad8b3a05 commit: cae0f5d3dad6db0d13690e5952ae2015ad8b3a05 branch: main author: andrei kulakov committer: JelleZijlstra date: 2022-04-04T19:50:13-07:00 summary: crypt docs: Fix references to `methods` attr (GH-26806) Co-authored-by: Jelle Zijlstra files: M Doc/library/crypt.rst diff --git a/Doc/library/crypt.rst b/Doc/library/crypt.rst index 73df87ca0db8d..3189ece048a26 100644 --- a/Doc/library/crypt.rst +++ b/Doc/library/crypt.rst @@ -96,8 +96,7 @@ The :mod:`crypt` module defines the following functions: :func:`mksalt`, one of the ``crypt.METHOD_*`` values (though not all may be available on all platforms), or a full encrypted password including salt, as returned by this function. If *salt* is not - provided, the strongest method will be used (as returned by - :func:`methods`). + provided, the strongest method available in :attr:`methods` will be used. Checking a password is usually done by passing the plain-text password as *word* and the full results of a previous :func:`crypt` call, @@ -125,8 +124,8 @@ The :mod:`crypt` module defines the following functions: .. function:: mksalt(method=None, *, rounds=None) Return a randomly generated salt of the specified method. If no - *method* is given, the strongest method available as returned by - :func:`methods` is used. + *method* is given, the strongest method available in :attr:`methods` is + used. The return value is a string suitable for passing as the *salt* argument to :func:`crypt`. From webhook-mailer at python.org Mon Apr 4 22:52:51 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Tue, 05 Apr 2022 02:52:51 -0000 Subject: [Python-checkins] bpo-43224: typing: Add tests for pickling and copying of unpacked native tuple (GH-32159) Message-ID: https://github.com/python/cpython/commit/772d8080c9fd635c3999673ca9fad8b674385c7f commit: 772d8080c9fd635c3999673ca9fad8b674385c7f branch: main author: Matthew Rahtz committer: JelleZijlstra date: 2022-04-04T19:52:42-07:00 summary: bpo-43224: typing: Add tests for pickling and copying of unpacked native tuple (GH-32159) files: M Lib/test/test_genericalias.py diff --git a/Lib/test/test_genericalias.py b/Lib/test/test_genericalias.py index 39c56f2290bd3..bf96ba065fbb0 100644 --- a/Lib/test/test_genericalias.py +++ b/Lib/test/test_genericalias.py @@ -47,12 +47,44 @@ from queue import Queue, SimpleQueue from weakref import WeakSet, ReferenceType, ref import typing +from typing import Unpack from typing import TypeVar T = TypeVar('T') K = TypeVar('K') V = TypeVar('V') +_UNPACKED_TUPLES = [ + # Unpacked tuple using `*` + (*tuple[int],)[0], + (*tuple[T],)[0], + (*tuple[int, str],)[0], + (*tuple[int, ...],)[0], + (*tuple[T, ...],)[0], + tuple[*tuple[int, ...]], + tuple[*tuple[T, ...]], + tuple[str, *tuple[int, ...]], + tuple[*tuple[int, ...], str], + tuple[float, *tuple[int, ...], str], + tuple[*tuple[*tuple[int, ...]]], + # Unpacked tuple using `Unpack` + Unpack[tuple[int]], + Unpack[tuple[T]], + Unpack[tuple[int, str]], + Unpack[tuple[int, ...]], + Unpack[tuple[T, ...]], + tuple[Unpack[tuple[int, ...]]], + tuple[Unpack[tuple[T, ...]]], + tuple[str, Unpack[tuple[int, ...]]], + tuple[Unpack[tuple[int, ...]], str], + tuple[float, Unpack[tuple[int, ...]], str], + tuple[Unpack[tuple[Unpack[tuple[int, ...]]]]], + # Unpacked tuple using `*` AND `Unpack` + tuple[Unpack[tuple[*tuple[int, ...]]]], + tuple[*tuple[Unpack[tuple[int, ...]]]], +] + + class BaseTest(unittest.TestCase): """Test basics.""" generic_types = [type, tuple, list, dict, set, frozenset, enumerate, @@ -351,13 +383,15 @@ class MyType(type): MyType[int] def test_pickle(self): - alias = GenericAlias(list, T) - for proto in range(pickle.HIGHEST_PROTOCOL + 1): - s = pickle.dumps(alias, proto) - loaded = pickle.loads(s) - self.assertEqual(loaded.__origin__, alias.__origin__) - self.assertEqual(loaded.__args__, alias.__args__) - self.assertEqual(loaded.__parameters__, alias.__parameters__) + aliases = [GenericAlias(list, T)] + _UNPACKED_TUPLES + for alias in aliases: + for proto in range(pickle.HIGHEST_PROTOCOL + 1): + with self.subTest(alias=alias, proto=proto): + s = pickle.dumps(alias, proto) + loaded = pickle.loads(s) + self.assertEqual(loaded.__origin__, alias.__origin__) + self.assertEqual(loaded.__args__, alias.__args__) + self.assertEqual(loaded.__parameters__, alias.__parameters__) def test_copy(self): class X(list): @@ -366,16 +400,21 @@ def __copy__(self): def __deepcopy__(self, memo): return self - for origin in list, deque, X: - alias = GenericAlias(origin, T) - copied = copy.copy(alias) - self.assertEqual(copied.__origin__, alias.__origin__) - self.assertEqual(copied.__args__, alias.__args__) - self.assertEqual(copied.__parameters__, alias.__parameters__) - copied = copy.deepcopy(alias) - self.assertEqual(copied.__origin__, alias.__origin__) - self.assertEqual(copied.__args__, alias.__args__) - self.assertEqual(copied.__parameters__, alias.__parameters__) + aliases = [ + GenericAlias(list, T), + GenericAlias(deque, T), + GenericAlias(X, T) + ] + _UNPACKED_TUPLES + for alias in aliases: + with self.subTest(alias=alias): + copied = copy.copy(alias) + self.assertEqual(copied.__origin__, alias.__origin__) + self.assertEqual(copied.__args__, alias.__args__) + self.assertEqual(copied.__parameters__, alias.__parameters__) + copied = copy.deepcopy(alias) + self.assertEqual(copied.__origin__, alias.__origin__) + self.assertEqual(copied.__args__, alias.__args__) + self.assertEqual(copied.__parameters__, alias.__parameters__) def test_union(self): a = typing.Union[list[int], list[str]] From webhook-mailer at python.org Mon Apr 4 22:56:38 2022 From: webhook-mailer at python.org (miss-islington) Date: Tue, 05 Apr 2022 02:56:38 -0000 Subject: [Python-checkins] bpo-47007: [doc] `str` special method lookup (GH-31863) Message-ID: https://github.com/python/cpython/commit/f502dadb332f911fa3b6c531bbc5065795cca242 commit: f502dadb332f911fa3b6c531bbc5065795cca242 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-04T19:56:29-07:00 summary: bpo-47007: [doc] `str` special method lookup (GH-31863) Clarify the `str()` docs to point out that `object.__str__()` follows special method lookup. Co-authored-by: Jelle Zijlstra (cherry picked from commit bb86d1d9fbd1888524e04475383f4ea764277f67) Co-authored-by: Vanshaj Singhania <8797467+itsvs at users.noreply.github.com> files: M Doc/library/stdtypes.rst diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst index b330335773103..28082b42d7923 100644 --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -1479,7 +1479,8 @@ multiple fragments. depends on whether *encoding* or *errors* is given, as follows. If neither *encoding* nor *errors* is given, ``str(object)`` returns - :meth:`object.__str__() `, which is the "informal" or nicely + :meth:`type(object).__str__(object) `, + which is the "informal" or nicely printable string representation of *object*. For string objects, this is the string itself. If *object* does not have a :meth:`~object.__str__` method, then :func:`str` falls back to returning From webhook-mailer at python.org Mon Apr 4 22:57:27 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Tue, 05 Apr 2022 02:57:27 -0000 Subject: [Python-checkins] bpo-40982: shutil docs: Remove outdated copytree() example (GH-24778) Message-ID: https://github.com/python/cpython/commit/e06f920c5bc6e9fad29082ba0d84043722806e17 commit: e06f920c5bc6e9fad29082ba0d84043722806e17 branch: main author: Zackery Spytz committer: JelleZijlstra date: 2022-04-04T19:57:17-07:00 summary: bpo-40982: shutil docs: Remove outdated copytree() example (GH-24778) It is not preferable to keep a copy of the implementation in the docs. files: M Doc/library/shutil.rst diff --git a/Doc/library/shutil.rst b/Doc/library/shutil.rst index 16b8d3cdeebc8..ac271ce9ffbe4 100644 --- a/Doc/library/shutil.rst +++ b/Doc/library/shutil.rst @@ -476,42 +476,7 @@ file then shutil will silently fallback on using less efficient copytree example ~~~~~~~~~~~~~~~~ -This example is the implementation of the :func:`copytree` function, described -above, with the docstring omitted. It demonstrates many of the other functions -provided by this module. :: - - def copytree(src, dst, symlinks=False): - names = os.listdir(src) - os.makedirs(dst) - errors = [] - for name in names: - srcname = os.path.join(src, name) - dstname = os.path.join(dst, name) - try: - if symlinks and os.path.islink(srcname): - linkto = os.readlink(srcname) - os.symlink(linkto, dstname) - elif os.path.isdir(srcname): - copytree(srcname, dstname, symlinks) - else: - copy2(srcname, dstname) - # XXX What about devices, sockets etc.? - except OSError as why: - errors.append((srcname, dstname, str(why))) - # catch the Error from the recursive copytree so that we can - # continue with other files - except Error as err: - errors.extend(err.args[0]) - try: - copystat(src, dst) - except OSError as why: - # can't copy file access times on Windows - if why.winerror is None: - errors.extend((src, dst, str(why))) - if errors: - raise Error(errors) - -Another example that uses the :func:`ignore_patterns` helper:: +An example that uses the :func:`ignore_patterns` helper:: from shutil import copytree, ignore_patterns From webhook-mailer at python.org Mon Apr 4 22:57:27 2022 From: webhook-mailer at python.org (miss-islington) Date: Tue, 05 Apr 2022 02:57:27 -0000 Subject: [Python-checkins] bpo-45790: List macros in same order in which fields are described (GH-29529) Message-ID: https://github.com/python/cpython/commit/f4e711bb49881deb1f07a685878646cd5cdee50f commit: f4e711bb49881deb1f07a685878646cd5cdee50f branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-04T19:57:22-07:00 summary: bpo-45790: List macros in same order in which fields are described (GH-29529) Signed-off-by: Rodrigo Tobar Co-authored-by: Jelle Zijlstra (cherry picked from commit b275267aa7d44ec90fa435c9cb1610c549da745a) Co-authored-by: rtobar files: A Misc/NEWS.d/next/Documentation/2021-11-12-11-03-55.bpo-45790.6yuhe8.rst M Doc/extending/newtypes_tutorial.rst diff --git a/Doc/extending/newtypes_tutorial.rst b/Doc/extending/newtypes_tutorial.rst index 530e2c4d35f84..904915306f1f3 100644 --- a/Doc/extending/newtypes_tutorial.rst +++ b/Doc/extending/newtypes_tutorial.rst @@ -67,8 +67,8 @@ The first bit is:: This is what a Custom object will contain. ``PyObject_HEAD`` is mandatory at the start of each object struct and defines a field called ``ob_base`` of type :c:type:`PyObject`, containing a pointer to a type object and a -reference count (these can be accessed using the macros :c:macro:`Py_REFCNT` -and :c:macro:`Py_TYPE` respectively). The reason for the macro is to +reference count (these can be accessed using the macros :c:macro:`Py_TYPE` +and :c:macro:`Py_REFCNT` respectively). The reason for the macro is to abstract away the layout and to enable additional fields in :ref:`debug builds `. diff --git a/Misc/NEWS.d/next/Documentation/2021-11-12-11-03-55.bpo-45790.6yuhe8.rst b/Misc/NEWS.d/next/Documentation/2021-11-12-11-03-55.bpo-45790.6yuhe8.rst new file mode 100644 index 0000000000000..41cf2cb91525f --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2021-11-12-11-03-55.bpo-45790.6yuhe8.rst @@ -0,0 +1,2 @@ +Adjust inaccurate phrasing in :doc:`../extending/newtypes_tutorial` about the +``ob_base`` field and the macros used to access its contents. From webhook-mailer at python.org Mon Apr 4 22:59:13 2022 From: webhook-mailer at python.org (miss-islington) Date: Tue, 05 Apr 2022 02:59:13 -0000 Subject: [Python-checkins] bpo-47007: [doc] `str` special method lookup (GH-31863) Message-ID: https://github.com/python/cpython/commit/c0063bdc7b5fed98c6799f3da0a954a775e3d89e commit: c0063bdc7b5fed98c6799f3da0a954a775e3d89e branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-04T19:59:04-07:00 summary: bpo-47007: [doc] `str` special method lookup (GH-31863) Clarify the `str()` docs to point out that `object.__str__()` follows special method lookup. Co-authored-by: Jelle Zijlstra (cherry picked from commit bb86d1d9fbd1888524e04475383f4ea764277f67) Co-authored-by: Vanshaj Singhania <8797467+itsvs at users.noreply.github.com> files: M Doc/library/stdtypes.rst diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst index b10909d81af0a..4be4d1daa831d 100644 --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -1448,7 +1448,8 @@ multiple fragments. depends on whether *encoding* or *errors* is given, as follows. If neither *encoding* nor *errors* is given, ``str(object)`` returns - :meth:`object.__str__() `, which is the "informal" or nicely + :meth:`type(object).__str__(object) `, + which is the "informal" or nicely printable string representation of *object*. For string objects, this is the string itself. If *object* does not have a :meth:`~object.__str__` method, then :func:`str` falls back to returning From webhook-mailer at python.org Mon Apr 4 23:08:25 2022 From: webhook-mailer at python.org (miss-islington) Date: Tue, 05 Apr 2022 03:08:25 -0000 Subject: [Python-checkins] bpo-32658: Regex docs: Fix metacharacter reference (GH-32230) Message-ID: https://github.com/python/cpython/commit/d95e072c419e40b0fb67b8cc8a84087c8a0276ee commit: d95e072c419e40b0fb67b8cc8a84087c8a0276ee branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-04T20:08:17-07:00 summary: bpo-32658: Regex docs: Fix metacharacter reference (GH-32230) Co-authored-by: Jelle Zijlstra (cherry picked from commit 43571a3eea8b5931769376daf4bdad1c9184ae0d) Co-authored-by: Mike cm files: M Doc/howto/regex.rst diff --git a/Doc/howto/regex.rst b/Doc/howto/regex.rst index d574c3736b1cb..c4ebbd311e0fe 100644 --- a/Doc/howto/regex.rst +++ b/Doc/howto/regex.rst @@ -89,7 +89,7 @@ is the same as ``[a-c]``, which uses a range to express the same set of characters. If you wanted to match only lowercase letters, your RE would be ``[a-z]``. -Metacharacters are not active inside classes. For example, ``[akm$]`` will +Metacharacters (except ``\``) are not active inside classes. For example, ``[akm$]`` will match any of the characters ``'a'``, ``'k'``, ``'m'``, or ``'$'``; ``'$'`` is usually a metacharacter, but inside a character class it's stripped of its special nature. From webhook-mailer at python.org Mon Apr 4 23:09:50 2022 From: webhook-mailer at python.org (miss-islington) Date: Tue, 05 Apr 2022 03:09:50 -0000 Subject: [Python-checkins] crypt docs: Fix references to `methods` attr (GH-26806) Message-ID: https://github.com/python/cpython/commit/97151e1e3a683d7b5d196e66b8a3482896eb8c9b commit: 97151e1e3a683d7b5d196e66b8a3482896eb8c9b branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-04T20:09:45-07:00 summary: crypt docs: Fix references to `methods` attr (GH-26806) Co-authored-by: Jelle Zijlstra (cherry picked from commit cae0f5d3dad6db0d13690e5952ae2015ad8b3a05) Co-authored-by: andrei kulakov files: M Doc/library/crypt.rst diff --git a/Doc/library/crypt.rst b/Doc/library/crypt.rst index 73df87ca0db8d..3189ece048a26 100644 --- a/Doc/library/crypt.rst +++ b/Doc/library/crypt.rst @@ -96,8 +96,7 @@ The :mod:`crypt` module defines the following functions: :func:`mksalt`, one of the ``crypt.METHOD_*`` values (though not all may be available on all platforms), or a full encrypted password including salt, as returned by this function. If *salt* is not - provided, the strongest method will be used (as returned by - :func:`methods`). + provided, the strongest method available in :attr:`methods` will be used. Checking a password is usually done by passing the plain-text password as *word* and the full results of a previous :func:`crypt` call, @@ -125,8 +124,8 @@ The :mod:`crypt` module defines the following functions: .. function:: mksalt(method=None, *, rounds=None) Return a randomly generated salt of the specified method. If no - *method* is given, the strongest method available as returned by - :func:`methods` is used. + *method* is given, the strongest method available in :attr:`methods` is + used. The return value is a string suitable for passing as the *salt* argument to :func:`crypt`. From webhook-mailer at python.org Mon Apr 4 23:11:44 2022 From: webhook-mailer at python.org (miss-islington) Date: Tue, 05 Apr 2022 03:11:44 -0000 Subject: [Python-checkins] bpo-32658: Regex docs: Fix metacharacter reference (GH-32230) Message-ID: https://github.com/python/cpython/commit/31c9e35e5360ba91fc6b81b23eb21dd3c39e2257 commit: 31c9e35e5360ba91fc6b81b23eb21dd3c39e2257 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-04T20:11:36-07:00 summary: bpo-32658: Regex docs: Fix metacharacter reference (GH-32230) Co-authored-by: Jelle Zijlstra (cherry picked from commit 43571a3eea8b5931769376daf4bdad1c9184ae0d) Co-authored-by: Mike cm files: M Doc/howto/regex.rst diff --git a/Doc/howto/regex.rst b/Doc/howto/regex.rst index d574c3736b1cb..c4ebbd311e0fe 100644 --- a/Doc/howto/regex.rst +++ b/Doc/howto/regex.rst @@ -89,7 +89,7 @@ is the same as ``[a-c]``, which uses a range to express the same set of characters. If you wanted to match only lowercase letters, your RE would be ``[a-z]``. -Metacharacters are not active inside classes. For example, ``[akm$]`` will +Metacharacters (except ``\``) are not active inside classes. For example, ``[akm$]`` will match any of the characters ``'a'``, ``'k'``, ``'m'``, or ``'$'``; ``'$'`` is usually a metacharacter, but inside a character class it's stripped of its special nature. From webhook-mailer at python.org Mon Apr 4 23:15:44 2022 From: webhook-mailer at python.org (miss-islington) Date: Tue, 05 Apr 2022 03:15:44 -0000 Subject: [Python-checkins] crypt docs: Fix references to `methods` attr (GH-26806) Message-ID: https://github.com/python/cpython/commit/c1984f39c5a34cbffc137676fa0b624ca598599f commit: c1984f39c5a34cbffc137676fa0b624ca598599f branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-04T20:15:30-07:00 summary: crypt docs: Fix references to `methods` attr (GH-26806) Co-authored-by: Jelle Zijlstra (cherry picked from commit cae0f5d3dad6db0d13690e5952ae2015ad8b3a05) Co-authored-by: andrei kulakov files: M Doc/library/crypt.rst diff --git a/Doc/library/crypt.rst b/Doc/library/crypt.rst index 73df87ca0db8d..3189ece048a26 100644 --- a/Doc/library/crypt.rst +++ b/Doc/library/crypt.rst @@ -96,8 +96,7 @@ The :mod:`crypt` module defines the following functions: :func:`mksalt`, one of the ``crypt.METHOD_*`` values (though not all may be available on all platforms), or a full encrypted password including salt, as returned by this function. If *salt* is not - provided, the strongest method will be used (as returned by - :func:`methods`). + provided, the strongest method available in :attr:`methods` will be used. Checking a password is usually done by passing the plain-text password as *word* and the full results of a previous :func:`crypt` call, @@ -125,8 +124,8 @@ The :mod:`crypt` module defines the following functions: .. function:: mksalt(method=None, *, rounds=None) Return a randomly generated salt of the specified method. If no - *method* is given, the strongest method available as returned by - :func:`methods` is used. + *method* is given, the strongest method available in :attr:`methods` is + used. The return value is a string suitable for passing as the *salt* argument to :func:`crypt`. From webhook-mailer at python.org Mon Apr 4 23:16:23 2022 From: webhook-mailer at python.org (miss-islington) Date: Tue, 05 Apr 2022 03:16:23 -0000 Subject: [Python-checkins] bpo-40982: shutil docs: Remove outdated copytree() example (GH-24778) Message-ID: https://github.com/python/cpython/commit/857cf55cbdd65b7a9534dc35d89a19dfe8cbdba5 commit: 857cf55cbdd65b7a9534dc35d89a19dfe8cbdba5 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-04T20:16:16-07:00 summary: bpo-40982: shutil docs: Remove outdated copytree() example (GH-24778) It is not preferable to keep a copy of the implementation in the docs. (cherry picked from commit e06f920c5bc6e9fad29082ba0d84043722806e17) Co-authored-by: Zackery Spytz files: M Doc/library/shutil.rst diff --git a/Doc/library/shutil.rst b/Doc/library/shutil.rst index 11c6707492167..5f71049f918e0 100644 --- a/Doc/library/shutil.rst +++ b/Doc/library/shutil.rst @@ -470,42 +470,7 @@ file then shutil will silently fallback on using less efficient copytree example ~~~~~~~~~~~~~~~~ -This example is the implementation of the :func:`copytree` function, described -above, with the docstring omitted. It demonstrates many of the other functions -provided by this module. :: - - def copytree(src, dst, symlinks=False): - names = os.listdir(src) - os.makedirs(dst) - errors = [] - for name in names: - srcname = os.path.join(src, name) - dstname = os.path.join(dst, name) - try: - if symlinks and os.path.islink(srcname): - linkto = os.readlink(srcname) - os.symlink(linkto, dstname) - elif os.path.isdir(srcname): - copytree(srcname, dstname, symlinks) - else: - copy2(srcname, dstname) - # XXX What about devices, sockets etc.? - except OSError as why: - errors.append((srcname, dstname, str(why))) - # catch the Error from the recursive copytree so that we can - # continue with other files - except Error as err: - errors.extend(err.args[0]) - try: - copystat(src, dst) - except OSError as why: - # can't copy file access times on Windows - if why.winerror is None: - errors.extend((src, dst, str(why))) - if errors: - raise Error(errors) - -Another example that uses the :func:`ignore_patterns` helper:: +An example that uses the :func:`ignore_patterns` helper:: from shutil import copytree, ignore_patterns From webhook-mailer at python.org Mon Apr 4 23:22:12 2022 From: webhook-mailer at python.org (miss-islington) Date: Tue, 05 Apr 2022 03:22:12 -0000 Subject: [Python-checkins] bpo-40982: shutil docs: Remove outdated copytree() example (GH-24778) Message-ID: https://github.com/python/cpython/commit/bc9ec24290d0fa823fa2ac19ac9767b2b8498faa commit: bc9ec24290d0fa823fa2ac19ac9767b2b8498faa branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-04T20:22:04-07:00 summary: bpo-40982: shutil docs: Remove outdated copytree() example (GH-24778) It is not preferable to keep a copy of the implementation in the docs. (cherry picked from commit e06f920c5bc6e9fad29082ba0d84043722806e17) Co-authored-by: Zackery Spytz files: M Doc/library/shutil.rst diff --git a/Doc/library/shutil.rst b/Doc/library/shutil.rst index 11c6707492167..5f71049f918e0 100644 --- a/Doc/library/shutil.rst +++ b/Doc/library/shutil.rst @@ -470,42 +470,7 @@ file then shutil will silently fallback on using less efficient copytree example ~~~~~~~~~~~~~~~~ -This example is the implementation of the :func:`copytree` function, described -above, with the docstring omitted. It demonstrates many of the other functions -provided by this module. :: - - def copytree(src, dst, symlinks=False): - names = os.listdir(src) - os.makedirs(dst) - errors = [] - for name in names: - srcname = os.path.join(src, name) - dstname = os.path.join(dst, name) - try: - if symlinks and os.path.islink(srcname): - linkto = os.readlink(srcname) - os.symlink(linkto, dstname) - elif os.path.isdir(srcname): - copytree(srcname, dstname, symlinks) - else: - copy2(srcname, dstname) - # XXX What about devices, sockets etc.? - except OSError as why: - errors.append((srcname, dstname, str(why))) - # catch the Error from the recursive copytree so that we can - # continue with other files - except Error as err: - errors.extend(err.args[0]) - try: - copystat(src, dst) - except OSError as why: - # can't copy file access times on Windows - if why.winerror is None: - errors.extend((src, dst, str(why))) - if errors: - raise Error(errors) - -Another example that uses the :func:`ignore_patterns` helper:: +An example that uses the :func:`ignore_patterns` helper:: from shutil import copytree, ignore_patterns From webhook-mailer at python.org Tue Apr 5 00:46:19 2022 From: webhook-mailer at python.org (ned-deily) Date: Tue, 05 Apr 2022 04:46:19 -0000 Subject: [Python-checkins] bpo-45847: Adapt macOS installer build to use new tkinter configure vars (GH-32328) Message-ID: https://github.com/python/cpython/commit/a0c700480b52dffab830d52e9c4eba15d4b57a89 commit: a0c700480b52dffab830d52e9c4eba15d4b57a89 branch: main author: Ned Deily committer: ned-deily date: 2022-04-05T00:46:09-04:00 summary: bpo-45847: Adapt macOS installer build to use new tkinter configure vars (GH-32328) files: M Mac/BuildScript/build-installer.py diff --git a/Mac/BuildScript/build-installer.py b/Mac/BuildScript/build-installer.py index 27719b85f1f50..1fb6749fc7d5b 100755 --- a/Mac/BuildScript/build-installer.py +++ b/Mac/BuildScript/build-installer.py @@ -1157,11 +1157,11 @@ def buildPython(): (' ', '--without-ensurepip ')[PYTHON_3], (' ', "--with-openssl='%s/libraries/usr/local'"%( shellQuote(WORKDIR)[1:-1],))[PYTHON_3], - (' ', "--with-tcltk-includes='-I%s/libraries/usr/local/include'"%( + (' ', "--enable-optimizations --with-lto")[compilerCanOptimize()], + (' ', "TCLTK_CFLAGS='-I%s/libraries/usr/local/include'"%( shellQuote(WORKDIR)[1:-1],))[internalTk()], - (' ', "--with-tcltk-libs='-L%s/libraries/usr/local/lib -ltcl8.6 -ltk8.6'"%( + (' ', "TCLTK_LIBS='-L%s/libraries/usr/local/lib -ltcl8.6 -ltk8.6'"%( shellQuote(WORKDIR)[1:-1],))[internalTk()], - (' ', "--enable-optimizations --with-lto")[compilerCanOptimize()], shellQuote(WORKDIR)[1:-1], shellQuote(WORKDIR)[1:-1])) From webhook-mailer at python.org Tue Apr 5 01:28:25 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Tue, 05 Apr 2022 05:28:25 -0000 Subject: [Python-checkins] [3.9] bpo-45790: List macros in same order in which fields are described (GH-29529) (GH-32321) Message-ID: https://github.com/python/cpython/commit/d1fb16ae286795abe3e9da86332c891b4b18826f commit: d1fb16ae286795abe3e9da86332c891b4b18826f branch: 3.9 author: Jelle Zijlstra committer: JelleZijlstra date: 2022-04-04T22:28:21-07:00 summary: [3.9] bpo-45790: List macros in same order in which fields are described (GH-29529) (GH-32321) Signed-off-by: Rodrigo Tobar Co-authored-by: Jelle Zijlstra . (cherry picked from commit b275267aa7d44ec90fa435c9cb1610c549da745a) Co-authored-by: rtobar files: A Misc/NEWS.d/next/Documentation/2021-11-12-11-03-55.bpo-45790.6yuhe8.rst M Doc/extending/newtypes_tutorial.rst diff --git a/Doc/extending/newtypes_tutorial.rst b/Doc/extending/newtypes_tutorial.rst index 4da77e797d222..4b5eb69ba7701 100644 --- a/Doc/extending/newtypes_tutorial.rst +++ b/Doc/extending/newtypes_tutorial.rst @@ -67,8 +67,8 @@ The first bit is:: This is what a Custom object will contain. ``PyObject_HEAD`` is mandatory at the start of each object struct and defines a field called ``ob_base`` of type :c:type:`PyObject`, containing a pointer to a type object and a -reference count (these can be accessed using the macros :c:macro:`Py_REFCNT` -and :c:macro:`Py_TYPE` respectively). The reason for the macro is to +reference count (these can be accessed using the macros :c:macro:`Py_TYPE` +and :c:macro:`Py_REFCNT` respectively). The reason for the macro is to abstract away the layout and to enable additional fields in debug builds. .. note:: diff --git a/Misc/NEWS.d/next/Documentation/2021-11-12-11-03-55.bpo-45790.6yuhe8.rst b/Misc/NEWS.d/next/Documentation/2021-11-12-11-03-55.bpo-45790.6yuhe8.rst new file mode 100644 index 0000000000000..41cf2cb91525f --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2021-11-12-11-03-55.bpo-45790.6yuhe8.rst @@ -0,0 +1,2 @@ +Adjust inaccurate phrasing in :doc:`../extending/newtypes_tutorial` about the +``ob_base`` field and the macros used to access its contents. From webhook-mailer at python.org Tue Apr 5 02:05:45 2022 From: webhook-mailer at python.org (ned-deily) Date: Tue, 05 Apr 2022 06:05:45 -0000 Subject: [Python-checkins] bpo-46890: Fix setting of sys._base_executable with framework builds on macOS (GH-31958) Message-ID: https://github.com/python/cpython/commit/6aaf4cd866f7c8f065d30d2a3fb4fffa8461d1d8 commit: 6aaf4cd866f7c8f065d30d2a3fb4fffa8461d1d8 branch: main author: Ronald Oussoren committer: ned-deily date: 2022-04-05T02:05:36-04:00 summary: bpo-46890: Fix setting of sys._base_executable with framework builds on macOS (GH-31958) The side effect of this bug was that venv environments directly used the main interpreter instead of the intermediate stub executable, which can cause problems when a script uses system APIs that require the use of an application bundle. files: A Misc/NEWS.d/next/macOS/2022-03-17-09-55-02.bpo-46890.GX-3OO.rst M Lib/test/test_getpath.py M Makefile.pre.in M Modules/getpath.c M Modules/getpath.py diff --git a/Lib/test/test_getpath.py b/Lib/test/test_getpath.py index eaf4a99279663..5208374e20013 100644 --- a/Lib/test/test_getpath.py +++ b/Lib/test/test_getpath.py @@ -446,6 +446,182 @@ def test_custom_platlibdir_posix(self): actual = getpath(ns, expected) self.assertEqual(expected, actual) + def test_framework_macos(self): + """ Test framework layout on macOS + + This layout is primarily detected using a compile-time option + (WITH_NEXT_FRAMEWORK). + """ + ns = MockPosixNamespace( + os_name="darwin", + argv0="/Library/Frameworks/Python.framework/Versions/9.8/Resources/Python.app/Contents/MacOS/Python", + WITH_NEXT_FRAMEWORK=1, + PREFIX="/Library/Frameworks/Python.framework/Versions/9.8", + EXEC_PREFIX="/Library/Frameworks/Python.framework/Versions/9.8", + ENV___PYVENV_LAUNCHER__="/Library/Frameworks/Python.framework/Versions/9.8/bin/python9.8", + real_executable="/Library/Frameworks/Python.framework/Versions/9.8/Resources/Python.app/Contents/MacOS/Python", + library="/Library/Frameworks/Python.framework/Versions/9.8/Python", + ) + ns.add_known_xfile("/Library/Frameworks/Python.framework/Versions/9.8/Resources/Python.app/Contents/MacOS/Python") + ns.add_known_xfile("/Library/Frameworks/Python.framework/Versions/9.8/bin/python9.8") + ns.add_known_dir("/Library/Frameworks/Python.framework/Versions/9.8/lib/python9.8/lib-dynload") + ns.add_known_file("/Library/Frameworks/Python.framework/Versions/9.8/lib/python9.8/os.py") + + # This is definitely not the stdlib (see discusion in bpo-46890) + #ns.add_known_file("/Library/Frameworks/lib/python98.zip") + + expected = dict( + executable="/Library/Frameworks/Python.framework/Versions/9.8/bin/python9.8", + prefix="/Library/Frameworks/Python.framework/Versions/9.8", + exec_prefix="/Library/Frameworks/Python.framework/Versions/9.8", + base_executable="/Library/Frameworks/Python.framework/Versions/9.8/bin/python9.8", + base_prefix="/Library/Frameworks/Python.framework/Versions/9.8", + base_exec_prefix="/Library/Frameworks/Python.framework/Versions/9.8", + module_search_paths_set=1, + module_search_paths=[ + "/Library/Frameworks/Python.framework/Versions/9.8/lib/python98.zip", + "/Library/Frameworks/Python.framework/Versions/9.8/lib/python9.8", + "/Library/Frameworks/Python.framework/Versions/9.8/lib/python9.8/lib-dynload", + ], + ) + actual = getpath(ns, expected) + self.assertEqual(expected, actual) + + def test_alt_framework_macos(self): + """ Test framework layout on macOS with alternate framework name + + ``--with-framework-name=DebugPython`` + + This layout is primarily detected using a compile-time option + (WITH_NEXT_FRAMEWORK). + """ + ns = MockPosixNamespace( + argv0="/Library/Frameworks/DebugPython.framework/Versions/9.8/Resources/Python.app/Contents/MacOS/DebugPython", + os_name="darwin", + WITH_NEXT_FRAMEWORK=1, + PREFIX="/Library/Frameworks/DebugPython.framework/Versions/9.8", + EXEC_PREFIX="/Library/Frameworks/DebugPython.framework/Versions/9.8", + ENV___PYVENV_LAUNCHER__="/Library/Frameworks/DebugPython.framework/Versions/9.8/bin/python9.8", + real_executable="/Library/Frameworks/DebugPython.framework/Versions/9.8/Resources/Python.app/Contents/MacOS/DebugPython", + library="/Library/Frameworks/DebugPython.framework/Versions/9.8/DebugPython", + PYTHONPATH=None, + ENV_PYTHONHOME=None, + ENV_PYTHONEXECUTABLE=None, + executable_dir=None, + py_setpath=None, + ) + ns.add_known_xfile("/Library/Frameworks/DebugPython.framework/Versions/9.8/Resources/Python.app/Contents/MacOS/DebugPython") + ns.add_known_xfile("/Library/Frameworks/DebugPython.framework/Versions/9.8/bin/python9.8") + ns.add_known_dir("/Library/Frameworks/DebugPython.framework/Versions/9.8/lib/python9.8/lib-dynload") + ns.add_known_xfile("/Library/Frameworks/DebugPython.framework/Versions/9.8/lib/python9.8/os.py") + + # This is definitely not the stdlib (see discusion in bpo-46890) + #ns.add_known_xfile("/Library/lib/python98.zip") + expected = dict( + executable="/Library/Frameworks/DebugPython.framework/Versions/9.8/bin/python9.8", + prefix="/Library/Frameworks/DebugPython.framework/Versions/9.8", + exec_prefix="/Library/Frameworks/DebugPython.framework/Versions/9.8", + base_executable="/Library/Frameworks/DebugPython.framework/Versions/9.8/bin/python9.8", + base_prefix="/Library/Frameworks/DebugPython.framework/Versions/9.8", + base_exec_prefix="/Library/Frameworks/DebugPython.framework/Versions/9.8", + module_search_paths_set=1, + module_search_paths=[ + "/Library/Frameworks/DebugPython.framework/Versions/9.8/lib/python98.zip", + "/Library/Frameworks/DebugPython.framework/Versions/9.8/lib/python9.8", + "/Library/Frameworks/DebugPython.framework/Versions/9.8/lib/python9.8/lib-dynload", + ], + ) + actual = getpath(ns, expected) + self.assertEqual(expected, actual) + + def test_venv_framework_macos(self): + """Test a venv layout on macOS using a framework build + """ + venv_path = "/tmp/workdir/venv" + ns = MockPosixNamespace( + os_name="darwin", + argv0="/Library/Frameworks/Python.framework/Versions/9.8/Resources/Python.app/Contents/MacOS/Python", + WITH_NEXT_FRAMEWORK=1, + PREFIX="/Library/Frameworks/Python.framework/Versions/9.8", + EXEC_PREFIX="/Library/Frameworks/Python.framework/Versions/9.8", + ENV___PYVENV_LAUNCHER__=f"{venv_path}/bin/python", + real_executable="/Library/Frameworks/Python.framework/Versions/9.8/Resources/Python.app/Contents/MacOS/Python", + library="/Library/Frameworks/Python.framework/Versions/9.8/Python", + ) + ns.add_known_dir(venv_path) + ns.add_known_dir(f"{venv_path}/bin") + ns.add_known_dir(f"{venv_path}/lib") + ns.add_known_dir(f"{venv_path}/lib/python9.8") + ns.add_known_xfile(f"{venv_path}/bin/python") + ns.add_known_xfile("/Library/Frameworks/Python.framework/Versions/9.8/Resources/Python.app/Contents/MacOS/Python") + ns.add_known_xfile("/Library/Frameworks/Python.framework/Versions/9.8/bin/python9.8") + ns.add_known_dir("/Library/Frameworks/Python.framework/Versions/9.8/lib/python9.8/lib-dynload") + ns.add_known_xfile("/Library/Frameworks/Python.framework/Versions/9.8/lib/python9.8/os.py") + ns.add_known_file(f"{venv_path}/pyvenv.cfg", [ + "home = /Library/Frameworks/Python.framework/Versions/9.8/bin" + ]) + expected = dict( + executable=f"{venv_path}/bin/python", + prefix="/Library/Frameworks/Python.framework/Versions/9.8", + exec_prefix="/Library/Frameworks/Python.framework/Versions/9.8", + base_executable="/Library/Frameworks/Python.framework/Versions/9.8/bin/python9.8", + base_prefix="/Library/Frameworks/Python.framework/Versions/9.8", + base_exec_prefix="/Library/Frameworks/Python.framework/Versions/9.8", + module_search_paths_set=1, + module_search_paths=[ + "/Library/Frameworks/Python.framework/Versions/9.8/lib/python98.zip", + "/Library/Frameworks/Python.framework/Versions/9.8/lib/python9.8", + "/Library/Frameworks/Python.framework/Versions/9.8/lib/python9.8/lib-dynload", + ], + ) + actual = getpath(ns, expected) + self.assertEqual(expected, actual) + + def test_venv_alt_framework_macos(self): + """Test a venv layout on macOS using a framework build + + ``--with-framework-name=DebugPython`` + """ + venv_path = "/tmp/workdir/venv" + ns = MockPosixNamespace( + os_name="darwin", + argv0="/Library/Frameworks/DebugPython.framework/Versions/9.8/Resources/Python.app/Contents/MacOS/DebugPython", + WITH_NEXT_FRAMEWORK=1, + PREFIX="/Library/Frameworks/DebugPython.framework/Versions/9.8", + EXEC_PREFIX="/Library/Frameworks/DebugPython.framework/Versions/9.8", + ENV___PYVENV_LAUNCHER__=f"{venv_path}/bin/python", + real_executable="/Library/Frameworks/DebugPython.framework/Versions/9.8/Resources/Python.app/Contents/MacOS/DebugPython", + library="/Library/Frameworks/DebugPython.framework/Versions/9.8/DebugPython", + ) + ns.add_known_dir(venv_path) + ns.add_known_dir(f"{venv_path}/bin") + ns.add_known_dir(f"{venv_path}/lib") + ns.add_known_dir(f"{venv_path}/lib/python9.8") + ns.add_known_xfile(f"{venv_path}/bin/python") + ns.add_known_xfile("/Library/Frameworks/DebugPython.framework/Versions/9.8/Resources/Python.app/Contents/MacOS/DebugPython") + ns.add_known_xfile("/Library/Frameworks/DebugPython.framework/Versions/9.8/bin/python9.8") + ns.add_known_dir("/Library/Frameworks/DebugPython.framework/Versions/9.8/lib/python9.8/lib-dynload") + ns.add_known_xfile("/Library/Frameworks/DebugPython.framework/Versions/9.8/lib/python9.8/os.py") + ns.add_known_file(f"{venv_path}/pyvenv.cfg", [ + "home = /Library/Frameworks/DebugPython.framework/Versions/9.8/bin" + ]) + expected = dict( + executable=f"{venv_path}/bin/python", + prefix="/Library/Frameworks/DebugPython.framework/Versions/9.8", + exec_prefix="/Library/Frameworks/DebugPython.framework/Versions/9.8", + base_executable="/Library/Frameworks/DebugPython.framework/Versions/9.8/bin/python9.8", + base_prefix="/Library/Frameworks/DebugPython.framework/Versions/9.8", + base_exec_prefix="/Library/Frameworks/DebugPython.framework/Versions/9.8", + module_search_paths_set=1, + module_search_paths=[ + "/Library/Frameworks/DebugPython.framework/Versions/9.8/lib/python98.zip", + "/Library/Frameworks/DebugPython.framework/Versions/9.8/lib/python9.8", + "/Library/Frameworks/DebugPython.framework/Versions/9.8/lib/python9.8/lib-dynload", + ], + ) + actual = getpath(ns, expected) + self.assertEqual(expected, actual) + def test_venv_macos(self): """Test a venv layout on macOS. @@ -787,6 +963,7 @@ def __init__(self, *a, argv0=None, config=None, **kw): self["config"] = DEFAULT_CONFIG.copy() self["os_name"] = "posix" self["PLATLIBDIR"] = "lib" + self["WITH_NEXT_FRAMEWORK"] = 0 super().__init__(*a, **kw) if argv0: self["config"]["orig_argv"] = [argv0] diff --git a/Makefile.pre.in b/Makefile.pre.in index c1e58f7315f49..9e0dae0e33bfe 100644 --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -1218,6 +1218,7 @@ Modules/getpath.o: $(srcdir)/Modules/getpath.c Python/frozen_modules/getpath.h M -DVERSION='"$(VERSION)"' \ -DVPATH='"$(VPATH)"' \ -DPLATLIBDIR='"$(PLATLIBDIR)"' \ + -DPYTHONFRAMEWORK='"$(PYTHONFRAMEWORK)"' \ -o $@ $(srcdir)/Modules/getpath.c Programs/python.o: $(srcdir)/Programs/python.c diff --git a/Misc/NEWS.d/next/macOS/2022-03-17-09-55-02.bpo-46890.GX-3OO.rst b/Misc/NEWS.d/next/macOS/2022-03-17-09-55-02.bpo-46890.GX-3OO.rst new file mode 100644 index 0000000000000..a3d7d3e4ede02 --- /dev/null +++ b/Misc/NEWS.d/next/macOS/2022-03-17-09-55-02.bpo-46890.GX-3OO.rst @@ -0,0 +1,3 @@ +Fix a regression in the setting of ``sys._base_executable`` in framework +builds, and thereby fix a regression in :mod:`venv` virtual environments +with such builds. diff --git a/Modules/getpath.c b/Modules/getpath.c index 5c646c9c83cbf..94479887cf850 100644 --- a/Modules/getpath.c +++ b/Modules/getpath.c @@ -875,6 +875,11 @@ _PyConfig_InitPathConfig(PyConfig *config, int compute_path_config) !decode_to_dict(dict, "os_name", "darwin") || #else !decode_to_dict(dict, "os_name", "posix") || +#endif +#ifdef WITH_NEXT_FRAMEWORK + !int_to_dict(dict, "WITH_NEXT_FRAMEWORK", 1) || +#else + !int_to_dict(dict, "WITH_NEXT_FRAMEWORK", 0) || #endif !decode_to_dict(dict, "PREFIX", PREFIX) || !decode_to_dict(dict, "EXEC_PREFIX", EXEC_PREFIX) || @@ -943,3 +948,4 @@ _PyConfig_InitPathConfig(PyConfig *config, int compute_path_config) return _PyStatus_OK(); } + diff --git a/Modules/getpath.py b/Modules/getpath.py index 3a13bfdf491a1..26465c88aaea5 100644 --- a/Modules/getpath.py +++ b/Modules/getpath.py @@ -33,6 +33,7 @@ # PREFIX -- [in] sysconfig.get_config_var(...) # EXEC_PREFIX -- [in] sysconfig.get_config_var(...) # PYTHONPATH -- [in] sysconfig.get_config_var(...) +# WITH_NEXT_FRAMEWORK -- [in] sysconfig.get_config_var(...) # VPATH -- [in] sysconfig.get_config_var(...) # PLATLIBDIR -- [in] sysconfig.get_config_var(...) # PYDEBUGEXT -- [in, opt] '_d' on Windows for debug builds @@ -301,9 +302,19 @@ def search_up(prefix, *landmarks, test=isfile): # If set, these variables imply that we should be using them as # sys.executable and when searching for venvs. However, we should # use the argv0 path for prefix calculation - base_executable = executable + + if os_name == 'darwin' and WITH_NEXT_FRAMEWORK: + # In a framework build the binary in {sys.exec_prefix}/bin is + # a stub executable that execs the real interpreter in an + # embedded app bundle. That bundle is an implementation detail + # and should not affect base_executable. + base_executable = f"{dirname(library)}/bin/python{VERSION_MAJOR}.{VERSION_MINOR}" + else: + base_executable = executable + if not real_executable: - real_executable = executable + real_executable = base_executable + #real_executable_dir = dirname(real_executable) executable = ENV_PYTHONEXECUTABLE or ENV___PYVENV_LAUNCHER__ executable_dir = dirname(executable) From webhook-mailer at python.org Tue Apr 5 03:57:51 2022 From: webhook-mailer at python.org (JulienPalard) Date: Tue, 05 Apr 2022 07:57:51 -0000 Subject: [Python-checkins] bpo-42238: [doc]: A make sucpicious false positive. (GH-32329) Message-ID: https://github.com/python/cpython/commit/d0e696e05d4aaca1f1cde72a5c3326ca3d0f5c24 commit: d0e696e05d4aaca1f1cde72a5c3326ca3d0f5c24 branch: main author: Julien Palard committer: JulienPalard date: 2022-04-05T09:57:42+02:00 summary: bpo-42238: [doc]: A make sucpicious false positive. (GH-32329) files: M Doc/tools/susp-ignored.csv diff --git a/Doc/tools/susp-ignored.csv b/Doc/tools/susp-ignored.csv index 88f401fc8370a..932c3800c9fa1 100644 --- a/Doc/tools/susp-ignored.csv +++ b/Doc/tools/susp-ignored.csv @@ -393,3 +393,4 @@ whatsnew/changelog,,::,Lib/email/mime/nonmultipart.py::MIMENonMultipart library/typing,,`,"assert_type(name, str) # OK, inferred type of `name` is `str`" library/typing,,`,# after which we hope the inferred type will be `int` whatsnew/changelog,,:company,-V:company/tag +library/typing,,`,# are located in the `typing_extensions` backports package. From webhook-mailer at python.org Tue Apr 5 05:08:16 2022 From: webhook-mailer at python.org (miss-islington) Date: Tue, 05 Apr 2022 09:08:16 -0000 Subject: [Python-checkins] chore/docs: fix rst style and typo (GH-32331) Message-ID: https://github.com/python/cpython/commit/faa12088c179dd896fde713448a7f142f820c1aa commit: faa12088c179dd896fde713448a7f142f820c1aa branch: main author: ??? <109224573 at qq.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-05T02:08:00-07:00 summary: chore/docs: fix rst style and typo (GH-32331) Current: ![??](https://user-images.githubusercontent.com/24759802/161704413-30fc91e8-ccd1-4617-8483-bc54ec970f30.png) After this change: ![??](https://user-images.githubusercontent.com/24759802/161704636-a5458192-a93a-40af-8bde-90ba80fdb53f.png) Trivial so I don't think it needs news or issue Automerge-Triggered-By: GH:JulienPalard files: M Doc/library/re.rst diff --git a/Doc/library/re.rst b/Doc/library/re.rst index b72d72f8355f2..89de9286ace79 100644 --- a/Doc/library/re.rst +++ b/Doc/library/re.rst @@ -166,15 +166,15 @@ The special characters are: back-tracking when the expression following it fails to match. These are known as :dfn:`possessive` quantifiers. For example, ``a*a`` will match ``'aaaa'`` because the ``a*`` will match - all 4 ``'a'``s, but, when the final ``'a'`` is encountered, the + all 4 ``'a'``\ s, but, when the final ``'a'`` is encountered, the expression is backtracked so that in the end the ``a*`` ends up matching - 3 ``'a'``s total, and the fourth ``'a'`` is matched by the final ``'a'``. + 3 ``'a'``\ s total, and the fourth ``'a'`` is matched by the final ``'a'``. However, when ``a*+a`` is used to match ``'aaaa'``, the ``a*+`` will match all 4 ``'a'``, but when the final ``'a'`` fails to find any more characters to match, the expression cannot be backtracked and will thus fail to match. ``x*+``, ``x++`` and ``x?+`` are equivalent to ``(?>x*)``, ``(?>x+)`` - and ``(?>x?)`` correspondigly. + and ``(?>x?)`` correspondingly. .. versionadded:: 3.11 @@ -208,10 +208,10 @@ The special characters are: *without* establishing any backtracking points. This is the possessive version of the quantifier above. For example, on the 6-character string ``'aaaaaa'``, ``a{3,5}+aa`` - attempt to match 5 ``'a'`` characters, then, requiring 2 more ``'a'``s, + attempt to match 5 ``'a'`` characters, then, requiring 2 more ``'a'``\ s, will need more characters than available and thus fail, while - ``a{3,5}aa`` will match with ``a{3,5}`` capturing 5, then 4 ``'a'``s - by backtracking and then the final 2 ``'a'``s are matched by the final + ``a{3,5}aa`` will match with ``a{3,5}`` capturing 5, then 4 ``'a'``\ s + by backtracking and then the final 2 ``'a'``\ s are matched by the final ``aa`` in the pattern. ``x{m,n}+`` is equivalent to ``(?>x{m,n})``. From webhook-mailer at python.org Tue Apr 5 05:21:40 2022 From: webhook-mailer at python.org (tiran) Date: Tue, 05 Apr 2022 09:21:40 -0000 Subject: [Python-checkins] bpo-40280: Add limited Emscripten REPL (GH-32284) Message-ID: https://github.com/python/cpython/commit/96e09837fb8031aebe8d823dd19ef664a34bcfad commit: 96e09837fb8031aebe8d823dd19ef664a34bcfad branch: main author: Christian Heimes committer: tiran date: 2022-04-05T11:21:11+02:00 summary: bpo-40280: Add limited Emscripten REPL (GH-32284) Co-authored-by: Katie Bell files: A Misc/NEWS.d/next/Tools-Demos/2022-04-03-11-47-45.bpo-40280.Q_IJik.rst A Tools/wasm/python.html A Tools/wasm/python.worker.js A Tools/wasm/wasm_webserver.py M Makefile.pre.in M Tools/wasm/README.md M configure M configure.ac diff --git a/Makefile.pre.in b/Makefile.pre.in index 9e0dae0e33bfe..d9b96f52ec9f7 100644 --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -807,15 +807,22 @@ $(DLLLIBRARY) libpython$(LDVERSION).dll.a: $(LIBRARY_OBJS) else true; \ fi -# wasm32-emscripten build +# wasm32-emscripten browser build # wasm assets directory is relative to current build dir, e.g. "./usr/local". # --preload-file turns a relative asset path into an absolute path. $(WASM_STDLIB): $(srcdir)/Lib/*.py $(srcdir)/Lib/*/*.py \ - pybuilddir.txt $(srcdir)/Tools/wasm/wasm_assets.py + pybuilddir.txt $(srcdir)/Tools/wasm/wasm_assets.py \ + python.html python.worker.js $(PYTHON_FOR_BUILD) $(srcdir)/Tools/wasm/wasm_assets.py \ --builddir . --prefix $(prefix) +python.html: $(srcdir)/Tools/wasm/python.html python.worker.js + @cp $(srcdir)/Tools/wasm/python.html $@ + +python.worker.js: $(srcdir)/Tools/wasm/python.worker.js + @cp $(srcdir)/Tools/wasm/python.worker.js $@ + ########################################################################## # Build static libmpdec.a LIBMPDEC_CFLAGS=$(PY_STDMODULE_CFLAGS) $(CCSHARED) @LIBMPDEC_CFLAGS@ diff --git a/Misc/NEWS.d/next/Tools-Demos/2022-04-03-11-47-45.bpo-40280.Q_IJik.rst b/Misc/NEWS.d/next/Tools-Demos/2022-04-03-11-47-45.bpo-40280.Q_IJik.rst new file mode 100644 index 0000000000000..07a968617117c --- /dev/null +++ b/Misc/NEWS.d/next/Tools-Demos/2022-04-03-11-47-45.bpo-40280.Q_IJik.rst @@ -0,0 +1,2 @@ +Replace Emscripten's limited shell with Katie Bell's browser-ui REPL from +python-wasm project. diff --git a/Tools/wasm/README.md b/Tools/wasm/README.md index 6b1e7b03df1e1..40b82e8f9397a 100644 --- a/Tools/wasm/README.md +++ b/Tools/wasm/README.md @@ -55,9 +55,13 @@ emrun builddir/emscripten-browser/python.html or ```shell -python3 -m http.server +./Tools/wasm/wasm_webserver.py ``` +and open http://localhost:8000/builddir/emscripten-browser/python.html . This +directory structure enables the *C/C++ DevTools Support (DWARF)* to load C +and header files with debug builds. + ### Cross compile to wasm32-emscripten for node ``` @@ -79,17 +83,17 @@ popd node --experimental-wasm-threads --experimental-wasm-bulk-memory builddir/emscripten-node/python.js ``` -## wasm32-emscripten limitations and issues +# wasm32-emscripten limitations and issues -- Heap and stack are limited. -- Most stdlib modules with a dependency on external libraries are missing: - ``ctypes``, ``readline``, ``sqlite3``, ``ssl``, and more. -- Shared extension modules are not implemented yet. All extension modules - are statically linked into the main binary. - The experimental configure option ``--enable-wasm-dynamic-linking`` enables - dynamic extensions. -- Processes are not supported. System calls like fork, popen, and subprocess - fail with ``ENOSYS`` or ``ENOSUP``. +Emscripten before 3.1.8 has known bugs that can cause memory corruption and +resource leaks. 3.1.8 contains several fixes for bugs in date and time +functions. + +## Network stack + +- Python's socket module does not work with Emscripten's emulated POSIX + sockets yet. Network modules like ``asyncio``, ``urllib``, ``selectors``, + etc. are not available. - Only ``AF_INET`` and ``AF_INET6`` with ``SOCK_STREAM`` (TCP) or ``SOCK_DGRAM`` (UDP) are available. ``AF_UNIX`` is not supported. - ``socketpair`` does not work. @@ -98,8 +102,21 @@ node --experimental-wasm-threads --experimental-wasm-bulk-memory builddir/emscri does not resolve to a real IP address. IPv6 is not available. - The ``select`` module is limited. ``select.select()`` crashes the runtime due to lack of exectfd support. + +## processes, threads, signals + +- Processes are not supported. System calls like fork, popen, and subprocess + fail with ``ENOSYS`` or ``ENOSUP``. - Signal support is limited. ``signal.alarm``, ``itimer``, ``sigaction`` are not available or do not work correctly. ``SIGTERM`` exits the runtime. +- Keyboard interrupt (CTRL+C) handling is not implemented yet. +- Browser builds cannot start new threads. Node's web workers consume + extra file descriptors. +- Resource-related functions like ``os.nice`` and most functions of the + ``resource`` module are not available. + +## file system + - Most user, group, and permission related function and modules are not supported or don't work as expected, e.g.``pwd`` module, ``grp`` module, ``os.setgroups``, ``os.chown``, and so on. ``lchown`` and `lchmod`` are @@ -113,23 +130,35 @@ node --experimental-wasm-threads --experimental-wasm-bulk-memory builddir/emscri and are disabled. - Large file support crashes the runtime and is disabled. - ``mmap`` module is unstable. flush (``msync``) can crash the runtime. -- Resource-related functions like ``os.nice`` and most functions of the - ``resource`` module are not available. + +## Misc + +- Heap memory and stack size are limited. Recursion or extensive memory + consumption can crash Python. +- Most stdlib modules with a dependency on external libraries are missing, + e.g. ``ctypes``, ``readline``, ``sqlite3``, ``ssl``, and more. +- Shared extension modules are not implemented yet. All extension modules + are statically linked into the main binary. + The experimental configure option ``--enable-wasm-dynamic-linking`` enables + dynamic extensions. - glibc extensions for date and time formatting are not available. - ``locales`` module is affected by musl libc issues, [bpo-46390](https://bugs.python.org/issue46390). - Python's object allocator ``obmalloc`` is disabled by default. - ``ensurepip`` is not available. -### wasm32-emscripten in browsers +## wasm32-emscripten in browsers +- The interactive shell does not handle copy 'n paste and unicode support + well. - The bundled stdlib is limited. Network-related modules, distutils, multiprocessing, dbm, tests and similar modules are not shipped. All other modules are bundled as pre-compiled ``pyc`` files. - Threading is not supported. +- In-memory file system (MEMFS) is not persistent and limited. -### wasm32-emscripten in node +## wasm32-emscripten in node Node builds use ``NODERAWFS``, ``USE_PTHREADS`` and ``PROXY_TO_PTHREAD`` linker options. diff --git a/Tools/wasm/python.html b/Tools/wasm/python.html new file mode 100644 index 0000000000000..c8d17488b2e70 --- /dev/null +++ b/Tools/wasm/python.html @@ -0,0 +1,245 @@ + + + + + + + + + wasm-python terminal + + + + + + +

Simple REPL for Python WASM

+
+
+ + +
+
+ The simple REPL provides a limited Python experience in the browser. + + Tools/wasm/README.md contains a list of known limitations and + issues. Networking, subprocesses, and threading are not available. +
+ + diff --git a/Tools/wasm/python.worker.js b/Tools/wasm/python.worker.js new file mode 100644 index 0000000000000..c3a8bdf7d2fc2 --- /dev/null +++ b/Tools/wasm/python.worker.js @@ -0,0 +1,87 @@ +class StdinBuffer { + constructor() { + this.sab = new SharedArrayBuffer(128 * Int32Array.BYTES_PER_ELEMENT) + this.buffer = new Int32Array(this.sab) + this.readIndex = 1; + this.numberOfCharacters = 0; + this.sentNull = true + } + + prompt() { + this.readIndex = 1 + Atomics.store(this.buffer, 0, -1) + postMessage({ + type: 'stdin', + buffer: this.sab + }) + Atomics.wait(this.buffer, 0, -1) + this.numberOfCharacters = this.buffer[0] + } + + stdin = () => { + if (this.numberOfCharacters + 1 === this.readIndex) { + if (!this.sentNull) { + // Must return null once to indicate we're done for now. + this.sentNull = true + return null + } + this.sentNull = false + this.prompt() + } + const char = this.buffer[this.readIndex] + this.readIndex += 1 + // How do I send an EOF?? + return char + } +} + +const stdoutBufSize = 128; +const stdoutBuf = new Int32Array() +let index = 0; + +const stdout = (charCode) => { + if (charCode) { + postMessage({ + type: 'stdout', + stdout: String.fromCharCode(charCode), + }) + } else { + console.log(typeof charCode, charCode) + } +} + +const stderr = (charCode) => { + if (charCode) { + postMessage({ + type: 'stderr', + stderr: String.fromCharCode(charCode), + }) + } else { + console.log(typeof charCode, charCode) + } +} + +const stdinBuffer = new StdinBuffer() + +var Module = { + noInitialRun: true, + stdin: stdinBuffer.stdin, + stdout: stdout, + stderr: stderr, + onRuntimeInitialized: () => { + postMessage({type: 'ready', stdinBuffer: stdinBuffer.sab}) + } +} + +onmessage = (event) => { + if (event.data.type === 'run') { + // TODO: Set up files from event.data.files + const ret = callMain(event.data.args) + postMessage({ + type: 'finished', + returnCode: ret + }) + } +} + +importScripts('python.js') diff --git a/Tools/wasm/wasm_webserver.py b/Tools/wasm/wasm_webserver.py new file mode 100755 index 0000000000000..ef642bf8a5be8 --- /dev/null +++ b/Tools/wasm/wasm_webserver.py @@ -0,0 +1,39 @@ +#!/usr/bin/env python +import argparse +from http import server + +parser = argparse.ArgumentParser( + description="Start a local webserver with a Python terminal." +) +parser.add_argument( + "--port", type=int, default=8000, help="port for the http server to listen on" +) +parser.add_argument( + "--bind", type=str, default="127.0.0.1", help="Bind address (empty for all)" +) + + +class MyHTTPRequestHandler(server.SimpleHTTPRequestHandler): + def end_headers(self): + self.send_my_headers() + super().end_headers() + + def send_my_headers(self): + self.send_header("Cross-Origin-Opener-Policy", "same-origin") + self.send_header("Cross-Origin-Embedder-Policy", "require-corp") + + +def main(): + args = parser.parse_args() + if not args.bind: + args.bind = None + + server.test( + HandlerClass=MyHTTPRequestHandler, + protocol="HTTP/1.1", + port=args.port, + bind=args.bind, + ) + +if __name__ == "__main__": + main() diff --git a/configure b/configure index 72d88806190e7..d6bc7175c2fa0 100755 --- a/configure +++ b/configure @@ -6338,7 +6338,7 @@ else case $ac_sys_system/$ac_sys_emscripten_target in #( Emscripten/browser*) : - EXEEXT=.html ;; #( + EXEEXT=.js ;; #( Emscripten/node*) : EXEEXT=.js ;; #( WASI/*) : diff --git a/configure.ac b/configure.ac index fda231214b3f7..53bbc3e7b199c 100644 --- a/configure.ac +++ b/configure.ac @@ -1137,7 +1137,7 @@ AC_ARG_WITH([suffix], ) ], [ AS_CASE([$ac_sys_system/$ac_sys_emscripten_target], - [Emscripten/browser*], [EXEEXT=.html], + [Emscripten/browser*], [EXEEXT=.js], [Emscripten/node*], [EXEEXT=.js], [WASI/*], [EXEEXT=.wasm], [EXEEXT=] From webhook-mailer at python.org Tue Apr 5 06:18:54 2022 From: webhook-mailer at python.org (markshannon) Date: Tue, 05 Apr 2022 10:18:54 -0000 Subject: [Python-checkins] bpo-47009: Let PRECALL_NO_KW_LIST_APPEND do its own POP_TOP (GH-32239) Message-ID: https://github.com/python/cpython/commit/6c6e0408a663c1f53dad403f54a18d444da39cb7 commit: 6c6e0408a663c1f53dad403f54a18d444da39cb7 branch: main author: Dennis Sweeney <36520290+sweeneyde at users.noreply.github.com> committer: markshannon date: 2022-04-05T11:18:30+01:00 summary: bpo-47009: Let PRECALL_NO_KW_LIST_APPEND do its own POP_TOP (GH-32239) files: M Python/ceval.c M Python/specialize.c diff --git a/Python/ceval.c b/Python/ceval.c index 68d2920727ab0..ce4abd5da2414 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -5040,15 +5040,15 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int PyObject *list = SECOND(); DEOPT_IF(!PyList_Check(list), PRECALL); STAT_INC(PRECALL, hit); - SKIP_CALL(); + // PRECALL + CALL + POP_TOP + JUMPBY(INLINE_CACHE_ENTRIES_PRECALL + 1 + INLINE_CACHE_ENTRIES_CALL + 1); + assert(next_instr[-1] == POP_TOP); PyObject *arg = POP(); if (_PyList_AppendTakeRef((PyListObject *)list, arg) < 0) { goto error; } + STACK_SHRINK(2); Py_DECREF(list); - STACK_SHRINK(1); - Py_INCREF(Py_None); - SET_TOP(Py_None); Py_DECREF(callable); NOTRACE_DISPATCH(); } diff --git a/Python/specialize.c b/Python/specialize.c index 08c69041e78b0..36b05026489cf 100644 --- a/Python/specialize.c +++ b/Python/specialize.c @@ -1435,7 +1435,10 @@ specialize_method_descriptor(PyMethodDescrObject *descr, _Py_CODEUNIT *instr, } PyInterpreterState *interp = _PyInterpreterState_GET(); PyObject *list_append = interp->callable_cache.list_append; - if ((PyObject *)descr == list_append && oparg == 1) { + _Py_CODEUNIT next = instr[INLINE_CACHE_ENTRIES_PRECALL + 1 + + INLINE_CACHE_ENTRIES_CALL + 1]; + bool pop = (_Py_OPCODE(next) == POP_TOP); + if ((PyObject *)descr == list_append && oparg == 1 && pop) { _Py_SET_OPCODE(*instr, PRECALL_NO_KW_LIST_APPEND); return 0; } From webhook-mailer at python.org Tue Apr 5 07:06:42 2022 From: webhook-mailer at python.org (iritkatriel) Date: Tue, 05 Apr 2022 11:06:42 -0000 Subject: [Python-checkins] bpo-47186: Replace JUMP_IF_NOT_EG_MATCH by CHECK_EG_MATCH + jump (GH-32309) Message-ID: https://github.com/python/cpython/commit/32091df41ce6e3a71df2cf37dc74b728c0d885f2 commit: 32091df41ce6e3a71df2cf37dc74b728c0d885f2 branch: main author: Irit Katriel <1055913+iritkatriel at users.noreply.github.com> committer: iritkatriel <1055913+iritkatriel at users.noreply.github.com> date: 2022-04-05T12:06:22+01:00 summary: bpo-47186: Replace JUMP_IF_NOT_EG_MATCH by CHECK_EG_MATCH + jump (GH-32309) files: A Misc/NEWS.d/next/Core and Builtins/2022-04-04-17-41-10.bpo-47186.aQWoSh.rst M Doc/library/dis.rst M Doc/whatsnew/3.11.rst M Include/opcode.h M Lib/importlib/_bootstrap_external.py M Lib/opcode.py M Objects/frameobject.c M Python/ceval.c M Python/compile.c M Python/opcode_targets.h diff --git a/Doc/library/dis.rst b/Doc/library/dis.rst index b364e3f031b3d..63d846715b165 100644 --- a/Doc/library/dis.rst +++ b/Doc/library/dis.rst @@ -635,6 +635,28 @@ iterations of the loop. .. versionadded:: 3.11 +.. opcode:: CHECK_EG_MATCH + + Performs exception matching for ``except*``. Applies ``split(TOS)`` on + the exception group representing TOS1. + + In case of a match, pops two items from the stack and pushes the + non-matching subgroup (``None`` in case of full match) followed by the + matching subgroup. When there is no match, pops one item (the match + type) and pushes ``None``. + + .. versionadded:: 3.11 + +.. opcode:: PREP_RERAISE_STAR + + Combines the raised and reraised exceptions list from TOS, into an exception + group to propagate from a try-except* block. Uses the original exception + group from TOS1 to reconstruct the structure of reraised exceptions. Pops + two items from the stack and pushes the exception to reraise or ``None`` + if there isn't one. + + .. versionadded:: 3.11 + .. opcode:: WITH_EXCEPT_START Calls the function in position 4 on the stack with arguments (type, val, tb) @@ -922,18 +944,6 @@ iterations of the loop. .. versionadded:: 3.1 -.. opcode:: JUMP_IF_NOT_EG_MATCH (target) - - Performs exception matching for ``except*``. Applies ``split(TOS)`` on - the exception group representing TOS1. Jumps if no match is found. - - Pops one item from the stack (the match type). If a match was found, - next item (the exception) and pushes the non-matching part of the - exception group followed by the matching part. - - .. versionadded:: 3.11 - - .. opcode:: POP_JUMP_IF_NOT_NONE (target) If TOS is not none, sets the bytecode counter to *target*. TOS is popped. @@ -948,17 +958,6 @@ iterations of the loop. .. versionadded:: 3.11 -.. opcode:: PREP_RERAISE_STAR - - Combines the raised and reraised exceptions list from TOS, into an exception - group to propagate from a try-except* block. Uses the original exception - group from TOS1 to reconstruct the structure of reraised exceptions. Pops - two items from the stack and pushes the exception to reraise or ``None`` - if there isn't one. - - .. versionadded:: 3.11 - - .. opcode:: JUMP_IF_TRUE_OR_POP (target) If TOS is true, sets the bytecode counter to *target* and leaves TOS on the diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index c607aadb4457c..fae85d867f91b 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -523,6 +523,9 @@ CPython bytecode changes * Replaced :opcode:`JUMP_IF_NOT_EXC_MATCH` by :opcode:`CHECK_EXC_MATCH` which performs the check but does not jump. +* Replaced :opcode:`JUMP_IF_NOT_EG_MATCH` by :opcode:`CHECK_EG_MATCH` which + performs the check but does not jump. + * Replaced :opcode:`JUMP_ABSOLUTE` by the relative :opcode:`JUMP_BACKWARD`. Deprecated diff --git a/Include/opcode.h b/Include/opcode.h index c82d1fd1fae2b..fd49dfed27ecd 100644 --- a/Include/opcode.h +++ b/Include/opcode.h @@ -22,6 +22,7 @@ extern "C" { #define MATCH_KEYS 33 #define PUSH_EXC_INFO 35 #define CHECK_EXC_MATCH 36 +#define CHECK_EG_MATCH 37 #define WITH_EXCEPT_START 49 #define GET_AITER 50 #define GET_ANEXT 51 @@ -80,7 +81,6 @@ extern "C" { #define LOAD_FAST 124 #define STORE_FAST 125 #define DELETE_FAST 126 -#define JUMP_IF_NOT_EG_MATCH 127 #define POP_JUMP_IF_NOT_NONE 128 #define POP_JUMP_IF_NONE 129 #define RAISE_VARARGS 130 @@ -136,39 +136,39 @@ extern "C" { #define COMPARE_OP_INT_JUMP 28 #define COMPARE_OP_STR_JUMP 29 #define JUMP_BACKWARD_QUICK 34 -#define LOAD_ATTR_ADAPTIVE 37 -#define LOAD_ATTR_INSTANCE_VALUE 38 -#define LOAD_ATTR_MODULE 39 -#define LOAD_ATTR_SLOT 40 -#define LOAD_ATTR_WITH_HINT 41 -#define LOAD_CONST__LOAD_FAST 42 -#define LOAD_FAST__LOAD_CONST 43 -#define LOAD_FAST__LOAD_FAST 44 -#define LOAD_GLOBAL_ADAPTIVE 45 -#define LOAD_GLOBAL_BUILTIN 46 -#define LOAD_GLOBAL_MODULE 47 -#define LOAD_METHOD_ADAPTIVE 48 -#define LOAD_METHOD_CLASS 55 -#define LOAD_METHOD_MODULE 56 -#define LOAD_METHOD_NO_DICT 57 -#define LOAD_METHOD_WITH_DICT 58 -#define LOAD_METHOD_WITH_VALUES 59 -#define PRECALL_ADAPTIVE 62 -#define PRECALL_BOUND_METHOD 63 -#define PRECALL_BUILTIN_CLASS 64 -#define PRECALL_BUILTIN_FAST_WITH_KEYWORDS 65 -#define PRECALL_METHOD_DESCRIPTOR_FAST_WITH_KEYWORDS 66 -#define PRECALL_NO_KW_BUILTIN_FAST 67 -#define PRECALL_NO_KW_BUILTIN_O 72 -#define PRECALL_NO_KW_ISINSTANCE 73 -#define PRECALL_NO_KW_LEN 76 -#define PRECALL_NO_KW_LIST_APPEND 77 -#define PRECALL_NO_KW_METHOD_DESCRIPTOR_FAST 78 -#define PRECALL_NO_KW_METHOD_DESCRIPTOR_NOARGS 79 -#define PRECALL_NO_KW_METHOD_DESCRIPTOR_O 80 -#define PRECALL_NO_KW_STR_1 81 -#define PRECALL_NO_KW_TUPLE_1 113 -#define PRECALL_NO_KW_TYPE_1 121 +#define LOAD_ATTR_ADAPTIVE 38 +#define LOAD_ATTR_INSTANCE_VALUE 39 +#define LOAD_ATTR_MODULE 40 +#define LOAD_ATTR_SLOT 41 +#define LOAD_ATTR_WITH_HINT 42 +#define LOAD_CONST__LOAD_FAST 43 +#define LOAD_FAST__LOAD_CONST 44 +#define LOAD_FAST__LOAD_FAST 45 +#define LOAD_GLOBAL_ADAPTIVE 46 +#define LOAD_GLOBAL_BUILTIN 47 +#define LOAD_GLOBAL_MODULE 48 +#define LOAD_METHOD_ADAPTIVE 55 +#define LOAD_METHOD_CLASS 56 +#define LOAD_METHOD_MODULE 57 +#define LOAD_METHOD_NO_DICT 58 +#define LOAD_METHOD_WITH_DICT 59 +#define LOAD_METHOD_WITH_VALUES 62 +#define PRECALL_ADAPTIVE 63 +#define PRECALL_BOUND_METHOD 64 +#define PRECALL_BUILTIN_CLASS 65 +#define PRECALL_BUILTIN_FAST_WITH_KEYWORDS 66 +#define PRECALL_METHOD_DESCRIPTOR_FAST_WITH_KEYWORDS 67 +#define PRECALL_NO_KW_BUILTIN_FAST 72 +#define PRECALL_NO_KW_BUILTIN_O 73 +#define PRECALL_NO_KW_ISINSTANCE 76 +#define PRECALL_NO_KW_LEN 77 +#define PRECALL_NO_KW_LIST_APPEND 78 +#define PRECALL_NO_KW_METHOD_DESCRIPTOR_FAST 79 +#define PRECALL_NO_KW_METHOD_DESCRIPTOR_NOARGS 80 +#define PRECALL_NO_KW_METHOD_DESCRIPTOR_O 81 +#define PRECALL_NO_KW_STR_1 113 +#define PRECALL_NO_KW_TUPLE_1 121 +#define PRECALL_NO_KW_TYPE_1 127 #define PRECALL_PYFUNC 141 #define RESUME_QUICK 143 #define STORE_ATTR_ADAPTIVE 150 @@ -205,7 +205,7 @@ static const uint32_t _PyOpcode_Jump[8] = { 0U, 0U, 536870912U, - 2282602496U, + 135118848U, 4163U, 0U, 0U, @@ -259,6 +259,7 @@ const uint8_t _PyOpcode_Deopt[256] = { [CALL_FUNCTION_EX] = CALL_FUNCTION_EX, [CALL_PY_EXACT_ARGS] = CALL, [CALL_PY_WITH_DEFAULTS] = CALL, + [CHECK_EG_MATCH] = CHECK_EG_MATCH, [CHECK_EXC_MATCH] = CHECK_EXC_MATCH, [COMPARE_OP] = COMPARE_OP, [COMPARE_OP_ADAPTIVE] = COMPARE_OP, @@ -294,7 +295,6 @@ const uint8_t _PyOpcode_Deopt[256] = { [JUMP_BACKWARD_QUICK] = JUMP_BACKWARD, [JUMP_FORWARD] = JUMP_FORWARD, [JUMP_IF_FALSE_OR_POP] = JUMP_IF_FALSE_OR_POP, - [JUMP_IF_NOT_EG_MATCH] = JUMP_IF_NOT_EG_MATCH, [JUMP_IF_TRUE_OR_POP] = JUMP_IF_TRUE_OR_POP, [JUMP_NO_INTERRUPT] = JUMP_NO_INTERRUPT, [KW_NAMES] = KW_NAMES, diff --git a/Lib/importlib/_bootstrap_external.py b/Lib/importlib/_bootstrap_external.py index 39a348aecef6c..189547cc16a8c 100644 --- a/Lib/importlib/_bootstrap_external.py +++ b/Lib/importlib/_bootstrap_external.py @@ -398,6 +398,7 @@ def _write_atomic(path, data, mode=0o666): # Python 3.11a6 3488 (LOAD_GLOBAL can push additional NULL) # Python 3.11a6 3489 (Add JUMP_BACKWARD, remove JUMP_ABSOLUTE) # Python 3.11a6 3490 (remove JUMP_IF_NOT_EXC_MATCH, add CHECK_EXC_MATCH) +# Python 3.11a6 3491 (remove JUMP_IF_NOT_EG_MATCH, add CHECK_EG_MATCH) # Python 3.12 will start with magic number 3500 @@ -412,7 +413,7 @@ def _write_atomic(path, data, mode=0o666): # Whenever MAGIC_NUMBER is changed, the ranges in the magic_values array # in PC/launcher.c must also be updated. -MAGIC_NUMBER = (3490).to_bytes(2, 'little') + b'\r\n' +MAGIC_NUMBER = (3491).to_bytes(2, 'little') + b'\r\n' _RAW_MAGIC_NUMBER = int.from_bytes(MAGIC_NUMBER, 'little') # For import.c _PYCACHE = '__pycache__' diff --git a/Lib/opcode.py b/Lib/opcode.py index e993a5c1ff16a..23d98df5061f4 100644 --- a/Lib/opcode.py +++ b/Lib/opcode.py @@ -77,6 +77,7 @@ def jabs_op(name, op, entries=0): def_op('PUSH_EXC_INFO', 35) def_op('CHECK_EXC_MATCH', 36) +def_op('CHECK_EG_MATCH', 37) def_op('WITH_EXCEPT_START', 49) def_op('GET_AITER', 50) @@ -147,7 +148,6 @@ def jabs_op(name, op, entries=0): haslocal.append(125) def_op('DELETE_FAST', 126) # Local variable number haslocal.append(126) -jabs_op('JUMP_IF_NOT_EG_MATCH', 127) jabs_op('POP_JUMP_IF_NOT_NONE', 128) jabs_op('POP_JUMP_IF_NONE', 129) def_op('RAISE_VARARGS', 130) # Number of raise arguments (1, 2, or 3) diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-04-04-17-41-10.bpo-47186.aQWoSh.rst b/Misc/NEWS.d/next/Core and Builtins/2022-04-04-17-41-10.bpo-47186.aQWoSh.rst new file mode 100644 index 0000000000000..2282c485b29e8 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2022-04-04-17-41-10.bpo-47186.aQWoSh.rst @@ -0,0 +1 @@ +Replace :opcode:`JUMP_IF_NOT_EG_MATCH` by :opcode:`CHECK_EG_MATCH` + jump. diff --git a/Objects/frameobject.c b/Objects/frameobject.c index fe374bf0632cd..c257c0a3b43f2 100644 --- a/Objects/frameobject.c +++ b/Objects/frameobject.c @@ -207,7 +207,6 @@ mark_stacks(PyCodeObject *code_obj, int len) case JUMP_IF_TRUE_OR_POP: case POP_JUMP_IF_FALSE: case POP_JUMP_IF_TRUE: - case JUMP_IF_NOT_EG_MATCH: { int64_t target_stack; int j = get_arg(code, i); @@ -215,13 +214,8 @@ mark_stacks(PyCodeObject *code_obj, int len) if (stacks[j] == UNINITIALIZED && j < i) { todo = 1; } - if (opcode == JUMP_IF_NOT_EG_MATCH) - { - next_stack = pop_value(pop_value(next_stack)); - target_stack = next_stack; - } - else if (opcode == JUMP_IF_FALSE_OR_POP || - opcode == JUMP_IF_TRUE_OR_POP) + if (opcode == JUMP_IF_FALSE_OR_POP || + opcode == JUMP_IF_TRUE_OR_POP) { target_stack = next_stack; next_stack = pop_value(next_stack); diff --git a/Python/ceval.c b/Python/ceval.c index ce4abd5da2414..f7e08c6ee813a 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -3804,7 +3804,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int DISPATCH(); } - TARGET(JUMP_IF_NOT_EG_MATCH) { + TARGET(CHECK_EG_MATCH) { PyObject *match_type = POP(); if (check_except_star_type_valid(tstate, match_type) < 0) { Py_DECREF(match_type); @@ -3825,15 +3825,11 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int assert(rest == NULL); goto error; } - if (Py_IsNone(match)) { - Py_DECREF(match); + PUSH(match); Py_XDECREF(rest); - /* no match - jump to target */ - JUMPTO(oparg); } else { - /* Total or partial match - update the stack from * [val] * to @@ -3841,17 +3837,11 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int * (rest can be Py_None) */ - PyObject *exc = TOP(); - SET_TOP(rest); PUSH(match); - PyErr_SetExcInfo(NULL, Py_NewRef(match), NULL); - - Py_DECREF(exc); - + Py_DECREF(exc_value); } - DISPATCH(); } diff --git a/Python/compile.c b/Python/compile.c index c3773a4f48929..c92b267dd03a1 100644 --- a/Python/compile.c +++ b/Python/compile.c @@ -1000,8 +1000,8 @@ stack_effect(int opcode, int oparg, int jump) return -1; case CHECK_EXC_MATCH: return 0; - case JUMP_IF_NOT_EG_MATCH: - return jump > 0 ? -1 : 0; + case CHECK_EG_MATCH: + return 0; case IMPORT_NAME: return -1; case IMPORT_FROM: @@ -3533,14 +3533,18 @@ compiler_try_except(struct compiler *c, stmt_ty s) [] POP_BLOCK [] JUMP L0 - [exc] L1: COPY 1 ) save copy of the original exception + [exc] L1: COPY 1 ) save copy of the original exception [orig, exc] BUILD_LIST ) list for raised/reraised excs ("result") [orig, exc, res] SWAP 2 [orig, res, exc] - [orig, res, exc, E1] JUMP_IF_NOT_EG_MATCH L2 + [orig, res, exc, E1] CHECK_EG_MATCH + [orig, red, rest/exc, match?] COPY 1 + [orig, red, rest/exc, match?, match?] POP_JUMP_IF_NOT_NONE H1 + [orig, red, exc, None] POP_TOP + [orig, red, exc] JUMP L2 - [orig, res, rest, match] (or POP if no V1) + [orig, res, rest, match] H1: (or POP if no V1) [orig, res, rest] SETUP_FINALLY R1 [orig, res, rest] @@ -3622,6 +3626,10 @@ compiler_try_star_except(struct compiler *c, stmt_ty s) if (except == NULL) { return 0; } + basicblock *handle_match = compiler_new_block(c); + if (handle_match == NULL) { + return 0; + } if (i == 0) { /* Push the original EG into the stack */ /* @@ -3641,9 +3649,15 @@ compiler_try_star_except(struct compiler *c, stmt_ty s) } if (handler->v.ExceptHandler.type) { VISIT(c, expr, handler->v.ExceptHandler.type); - ADDOP_JUMP(c, JUMP_IF_NOT_EG_MATCH, except); + ADDOP(c, CHECK_EG_MATCH); + ADDOP_I(c, COPY, 1); + ADDOP_JUMP(c, POP_JUMP_IF_NOT_NONE, handle_match); + ADDOP(c, POP_TOP); // match + ADDOP_JUMP(c, JUMP, except); } + compiler_use_next_block(c, handle_match); + basicblock *cleanup_end = compiler_new_block(c); if (cleanup_end == NULL) { return 0; @@ -3657,7 +3671,7 @@ compiler_try_star_except(struct compiler *c, stmt_ty s) compiler_nameop(c, handler->v.ExceptHandler.name, Store); } else { - ADDOP(c, POP_TOP); // exc + ADDOP(c, POP_TOP); // match } /* diff --git a/Python/opcode_targets.h b/Python/opcode_targets.h index 2eba5531723fb..e71b3e2120f7a 100644 --- a/Python/opcode_targets.h +++ b/Python/opcode_targets.h @@ -36,6 +36,7 @@ static void *opcode_targets[256] = { &&TARGET_JUMP_BACKWARD_QUICK, &&TARGET_PUSH_EXC_INFO, &&TARGET_CHECK_EXC_MATCH, + &&TARGET_CHECK_EG_MATCH, &&TARGET_LOAD_ATTR_ADAPTIVE, &&TARGET_LOAD_ATTR_INSTANCE_VALUE, &&TARGET_LOAD_ATTR_MODULE, @@ -47,40 +48,39 @@ static void *opcode_targets[256] = { &&TARGET_LOAD_GLOBAL_ADAPTIVE, &&TARGET_LOAD_GLOBAL_BUILTIN, &&TARGET_LOAD_GLOBAL_MODULE, - &&TARGET_LOAD_METHOD_ADAPTIVE, &&TARGET_WITH_EXCEPT_START, &&TARGET_GET_AITER, &&TARGET_GET_ANEXT, &&TARGET_BEFORE_ASYNC_WITH, &&TARGET_BEFORE_WITH, &&TARGET_END_ASYNC_FOR, + &&TARGET_LOAD_METHOD_ADAPTIVE, &&TARGET_LOAD_METHOD_CLASS, &&TARGET_LOAD_METHOD_MODULE, &&TARGET_LOAD_METHOD_NO_DICT, &&TARGET_LOAD_METHOD_WITH_DICT, - &&TARGET_LOAD_METHOD_WITH_VALUES, &&TARGET_STORE_SUBSCR, &&TARGET_DELETE_SUBSCR, + &&TARGET_LOAD_METHOD_WITH_VALUES, &&TARGET_PRECALL_ADAPTIVE, &&TARGET_PRECALL_BOUND_METHOD, &&TARGET_PRECALL_BUILTIN_CLASS, &&TARGET_PRECALL_BUILTIN_FAST_WITH_KEYWORDS, &&TARGET_PRECALL_METHOD_DESCRIPTOR_FAST_WITH_KEYWORDS, - &&TARGET_PRECALL_NO_KW_BUILTIN_FAST, &&TARGET_GET_ITER, &&TARGET_GET_YIELD_FROM_ITER, &&TARGET_PRINT_EXPR, &&TARGET_LOAD_BUILD_CLASS, + &&TARGET_PRECALL_NO_KW_BUILTIN_FAST, &&TARGET_PRECALL_NO_KW_BUILTIN_O, - &&TARGET_PRECALL_NO_KW_ISINSTANCE, &&TARGET_LOAD_ASSERTION_ERROR, &&TARGET_RETURN_GENERATOR, + &&TARGET_PRECALL_NO_KW_ISINSTANCE, &&TARGET_PRECALL_NO_KW_LEN, &&TARGET_PRECALL_NO_KW_LIST_APPEND, &&TARGET_PRECALL_NO_KW_METHOD_DESCRIPTOR_FAST, &&TARGET_PRECALL_NO_KW_METHOD_DESCRIPTOR_NOARGS, &&TARGET_PRECALL_NO_KW_METHOD_DESCRIPTOR_O, - &&TARGET_PRECALL_NO_KW_STR_1, &&TARGET_LIST_TO_TUPLE, &&TARGET_RETURN_VALUE, &&TARGET_IMPORT_STAR, @@ -112,7 +112,7 @@ static void *opcode_targets[256] = { &&TARGET_JUMP_FORWARD, &&TARGET_JUMP_IF_FALSE_OR_POP, &&TARGET_JUMP_IF_TRUE_OR_POP, - &&TARGET_PRECALL_NO_KW_TUPLE_1, + &&TARGET_PRECALL_NO_KW_STR_1, &&TARGET_POP_JUMP_IF_FALSE, &&TARGET_POP_JUMP_IF_TRUE, &&TARGET_LOAD_GLOBAL, @@ -120,13 +120,13 @@ static void *opcode_targets[256] = { &&TARGET_CONTAINS_OP, &&TARGET_RERAISE, &&TARGET_COPY, - &&TARGET_PRECALL_NO_KW_TYPE_1, + &&TARGET_PRECALL_NO_KW_TUPLE_1, &&TARGET_BINARY_OP, &&TARGET_SEND, &&TARGET_LOAD_FAST, &&TARGET_STORE_FAST, &&TARGET_DELETE_FAST, - &&TARGET_JUMP_IF_NOT_EG_MATCH, + &&TARGET_PRECALL_NO_KW_TYPE_1, &&TARGET_POP_JUMP_IF_NOT_NONE, &&TARGET_POP_JUMP_IF_NONE, &&TARGET_RAISE_VARARGS, From webhook-mailer at python.org Tue Apr 5 07:49:24 2022 From: webhook-mailer at python.org (iritkatriel) Date: Tue, 05 Apr 2022 11:49:24 -0000 Subject: [Python-checkins] bpo-47120: make JUMP_NO_INTERRUPT relative (GH-32221) Message-ID: https://github.com/python/cpython/commit/0aa8d5cbd89cf3b61d7e8626f3a7b9c4881dfd70 commit: 0aa8d5cbd89cf3b61d7e8626f3a7b9c4881dfd70 branch: main author: Irit Katriel <1055913+iritkatriel at users.noreply.github.com> committer: iritkatriel <1055913+iritkatriel at users.noreply.github.com> date: 2022-04-05T12:49:08+01:00 summary: bpo-47120: make JUMP_NO_INTERRUPT relative (GH-32221) files: A Misc/NEWS.d/next/Core and Builtins/2022-03-31-21-43-57.bpo-47120.NgxQbA.rst M Doc/library/dis.rst M Doc/whatsnew/3.11.rst M Include/opcode.h M Lib/importlib/_bootstrap_external.py M Lib/opcode.py M Lib/test/test_dis.py M Objects/frameobject.c M Python/ceval.c M Python/compile.c M Python/opcode_targets.h diff --git a/Doc/library/dis.rst b/Doc/library/dis.rst index 63d846715b165..fa0e23a6c9693 100644 --- a/Doc/library/dis.rst +++ b/Doc/library/dis.rst @@ -925,7 +925,14 @@ iterations of the loop. .. opcode:: JUMP_BACKWARD (delta) - Decrements bytecode counter by *delta*. + Decrements bytecode counter by *delta*. Checks for interrupts. + + .. versionadded:: 3.11 + + +.. opcode:: JUMP_BACKWARD_NO_INTERRUPT (delta) + + Decrements bytecode counter by *delta*. Does not check for interrupts. .. versionadded:: 3.11 @@ -974,13 +981,6 @@ iterations of the loop. .. versionadded:: 3.1 -.. opcode:: JUMP_NO_INTERRUPT (target) - - Set bytecode counter to *target*. Do not check for interrupts. - - .. versionadded:: 3.11 - - .. opcode:: FOR_ITER (delta) TOS is an :term:`iterator`. Call its :meth:`~iterator.__next__` method. If diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index fae85d867f91b..d0c10a9100997 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -528,6 +528,8 @@ CPython bytecode changes * Replaced :opcode:`JUMP_ABSOLUTE` by the relative :opcode:`JUMP_BACKWARD`. +* Added :opcode:`JUMP_BACKWARD_NO_INTERRUPT`, which is used in certain loops where it is undesirable to handle interrupts. + Deprecated ========== diff --git a/Include/opcode.h b/Include/opcode.h index fd49dfed27ecd..ff3ffddda2147 100644 --- a/Include/opcode.h +++ b/Include/opcode.h @@ -87,7 +87,7 @@ extern "C" { #define GET_AWAITABLE 131 #define MAKE_FUNCTION 132 #define BUILD_SLICE 133 -#define JUMP_NO_INTERRUPT 134 +#define JUMP_BACKWARD_NO_INTERRUPT 134 #define MAKE_CELL 135 #define LOAD_CLOSURE 136 #define LOAD_DEREF 137 @@ -196,7 +196,7 @@ static const uint32_t _PyOpcode_RelativeJump[8] = { 0U, 536870912U, 134234112U, - 4096U, + 4160U, 0U, 0U, 0U, @@ -292,11 +292,11 @@ const uint8_t _PyOpcode_Deopt[256] = { [IMPORT_STAR] = IMPORT_STAR, [IS_OP] = IS_OP, [JUMP_BACKWARD] = JUMP_BACKWARD, + [JUMP_BACKWARD_NO_INTERRUPT] = JUMP_BACKWARD_NO_INTERRUPT, [JUMP_BACKWARD_QUICK] = JUMP_BACKWARD, [JUMP_FORWARD] = JUMP_FORWARD, [JUMP_IF_FALSE_OR_POP] = JUMP_IF_FALSE_OR_POP, [JUMP_IF_TRUE_OR_POP] = JUMP_IF_TRUE_OR_POP, - [JUMP_NO_INTERRUPT] = JUMP_NO_INTERRUPT, [KW_NAMES] = KW_NAMES, [LIST_APPEND] = LIST_APPEND, [LIST_EXTEND] = LIST_EXTEND, diff --git a/Lib/importlib/_bootstrap_external.py b/Lib/importlib/_bootstrap_external.py index 189547cc16a8c..45be177df76a9 100644 --- a/Lib/importlib/_bootstrap_external.py +++ b/Lib/importlib/_bootstrap_external.py @@ -398,7 +398,8 @@ def _write_atomic(path, data, mode=0o666): # Python 3.11a6 3488 (LOAD_GLOBAL can push additional NULL) # Python 3.11a6 3489 (Add JUMP_BACKWARD, remove JUMP_ABSOLUTE) # Python 3.11a6 3490 (remove JUMP_IF_NOT_EXC_MATCH, add CHECK_EXC_MATCH) -# Python 3.11a6 3491 (remove JUMP_IF_NOT_EG_MATCH, add CHECK_EG_MATCH) +# Python 3.11a6 3491 (remove JUMP_IF_NOT_EG_MATCH, add CHECK_EG_MATCH, +# add JUMP_BACKWARD_NO_INTERRUPT, make JUMP_NO_INTERRUPT virtual) # Python 3.12 will start with magic number 3500 diff --git a/Lib/opcode.py b/Lib/opcode.py index 23d98df5061f4..97b580532cb94 100644 --- a/Lib/opcode.py +++ b/Lib/opcode.py @@ -154,7 +154,7 @@ def jabs_op(name, op, entries=0): def_op('GET_AWAITABLE', 131) def_op('MAKE_FUNCTION', 132) # Flags def_op('BUILD_SLICE', 133) # Number of items -jabs_op('JUMP_NO_INTERRUPT', 134) # Target byte offset from beginning of code +jrel_op('JUMP_BACKWARD_NO_INTERRUPT', 134) # Number of words to skip (backwards) def_op('MAKE_CELL', 135) hasfree.append(135) def_op('LOAD_CLOSURE', 136) diff --git a/Lib/test/test_dis.py b/Lib/test/test_dis.py index 544e1350cb7f9..2f78d42cc724a 100644 --- a/Lib/test/test_dis.py +++ b/Lib/test/test_dis.py @@ -684,7 +684,8 @@ def test_boundaries(self): def test_widths(self): for opcode, opname in enumerate(dis.opname): if opname in ('BUILD_MAP_UNPACK_WITH_CALL', - 'BUILD_TUPLE_UNPACK_WITH_CALL'): + 'BUILD_TUPLE_UNPACK_WITH_CALL', + 'JUMP_BACKWARD_NO_INTERRUPT'): continue with self.subTest(opname=opname): width = dis._OPNAME_WIDTH diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-03-31-21-43-57.bpo-47120.NgxQbA.rst b/Misc/NEWS.d/next/Core and Builtins/2022-03-31-21-43-57.bpo-47120.NgxQbA.rst new file mode 100644 index 0000000000000..236ad94795056 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2022-03-31-21-43-57.bpo-47120.NgxQbA.rst @@ -0,0 +1 @@ +Replace the absolute jump opcode :opcode:`JUMP_NO_INTERRUPT` by the relative :opcode:`JUMP_BACKWARD_NO_INTERRUPT`. diff --git a/Objects/frameobject.c b/Objects/frameobject.c index c257c0a3b43f2..6842e62839fd1 100644 --- a/Objects/frameobject.c +++ b/Objects/frameobject.c @@ -229,15 +229,6 @@ mark_stacks(PyCodeObject *code_obj, int len) stacks[i+1] = next_stack; break; } - case JUMP_NO_INTERRUPT: - j = get_arg(code, i); - assert(j < len); - if (stacks[j] == UNINITIALIZED && j < i) { - todo = 1; - } - assert(stacks[j] == UNINITIALIZED || stacks[j] == next_stack); - stacks[j] = next_stack; - break; case POP_EXCEPT: next_stack = pop_value(pop_value(pop_value(next_stack))); stacks[i+1] = next_stack; @@ -256,8 +247,13 @@ mark_stacks(PyCodeObject *code_obj, int len) stacks[j] = next_stack; break; case JUMP_BACKWARD: + case JUMP_BACKWARD_NO_INTERRUPT: j = i + 1 - get_arg(code, i); assert(j >= 0); + assert(j < len); + if (stacks[j] == UNINITIALIZED && j < i) { + todo = 1; + } assert(stacks[j] == UNINITIALIZED || stacks[j] == next_stack); stacks[j] = next_stack; break; diff --git a/Python/ceval.c b/Python/ceval.c index f7e08c6ee813a..9b7c42cbe4b11 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -4045,13 +4045,13 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int DISPATCH(); } - TARGET(JUMP_NO_INTERRUPT) { + TARGET(JUMP_BACKWARD_NO_INTERRUPT) { /* This bytecode is used in the `yield from` or `await` loop. * If there is an interrupt, we want it handled in the innermost * generator or coroutine, so we deliberately do not check it here. * (see bpo-30039). */ - JUMPTO(oparg); + JUMPBY(-oparg); DISPATCH(); } diff --git a/Python/compile.c b/Python/compile.c index c92b267dd03a1..f04ba9ec50f6f 100644 --- a/Python/compile.c +++ b/Python/compile.c @@ -77,8 +77,9 @@ #define SETUP_WITH -3 #define POP_BLOCK -4 #define JUMP -5 +#define JUMP_NO_INTERRUPT -6 -#define MIN_VIRTUAL_OPCODE -5 +#define MIN_VIRTUAL_OPCODE -6 #define MAX_ALLOWED_OPCODE 254 #define IS_WITHIN_OPCODE_RANGE(opcode) \ @@ -86,6 +87,13 @@ #define IS_VIRTUAL_OPCODE(opcode) ((opcode) < 0) +/* opcodes which are not emitted in codegen stage, only by the assembler */ +#define IS_ASSEMBLER_OPCODE(opcode) \ + ((opcode) == JUMP_FORWARD || \ + (opcode) == JUMP_BACKWARD || \ + (opcode) == JUMP_BACKWARD_NO_INTERRUPT) + + #define IS_TOP_LEVEL_AWAIT(c) ( \ (c->c_flags->cf_flags & PyCF_ALLOW_TOP_LEVEL_AWAIT) \ && (c->u->u_ste->ste_type == ModuleBlock)) @@ -1011,6 +1019,7 @@ stack_effect(int opcode, int oparg, int jump) case JUMP_FORWARD: case JUMP_BACKWARD: case JUMP: + case JUMP_BACKWARD_NO_INTERRUPT: case JUMP_NO_INTERRUPT: return 0; @@ -1199,6 +1208,7 @@ compiler_addop_line(struct compiler *c, int opcode, int line, int end_line, int col_offset, int end_col_offset) { assert(IS_WITHIN_OPCODE_RANGE(opcode)); + assert(!IS_ASSEMBLER_OPCODE(opcode)); assert(!HAS_ARG(opcode) || IS_ARTIFICIAL(opcode)); if (compiler_use_new_implicit_block_if_needed(c) < 0) { @@ -1442,6 +1452,7 @@ compiler_addop_i_line(struct compiler *c, int opcode, Py_ssize_t oparg, EXTENDED_ARG is used for 16, 24, and 32-bit arguments. */ assert(IS_WITHIN_OPCODE_RANGE(opcode)); + assert(!IS_ASSEMBLER_OPCODE(opcode)); assert(HAS_ARG(opcode)); assert(0 <= oparg && oparg <= 2147483647); @@ -1486,6 +1497,7 @@ static int add_jump_to_block(struct compiler *c, int opcode, basicblock *target) { assert(IS_WITHIN_OPCODE_RANGE(opcode)); + assert(!IS_ASSEMBLER_OPCODE(opcode)); assert(HAS_ARG(opcode) || IS_VIRTUAL_OPCODE(opcode)); assert(target != NULL); @@ -7089,8 +7101,7 @@ stackdepth(struct compiler *c) stackdepth_push(&sp, instr->i_target, target_depth); } depth = new_depth; - assert(instr->i_opcode != JUMP_FORWARD); - assert(instr->i_opcode != JUMP_BACKWARD); + assert(!IS_ASSEMBLER_OPCODE(instr->i_opcode)); if (instr->i_opcode == JUMP_NO_INTERRUPT || instr->i_opcode == JUMP || instr->i_opcode == RETURN_VALUE || @@ -7597,15 +7608,15 @@ normalize_jumps(struct assembler *a) continue; } struct instr *last = &b->b_instr[b->b_iused-1]; - assert(last->i_opcode != JUMP_FORWARD); - assert(last->i_opcode != JUMP_BACKWARD); + assert(!IS_ASSEMBLER_OPCODE(last->i_opcode)); if (last->i_opcode == JUMP) { - if (last->i_target->b_visited == 0) { - last->i_opcode = JUMP_FORWARD; - } - else { - last->i_opcode = JUMP_BACKWARD; - } + bool is_forward = last->i_target->b_visited == 0; + last->i_opcode = is_forward ? JUMP_FORWARD : JUMP_BACKWARD; + } + if (last->i_opcode == JUMP_NO_INTERRUPT) { + bool is_forward = last->i_target->b_visited == 0; + last->i_opcode = is_forward ? + JUMP_FORWARD : JUMP_BACKWARD_NO_INTERRUPT; } } } @@ -7641,11 +7652,13 @@ assemble_jump_offsets(struct assembler *a, struct compiler *c) instr->i_oparg = instr->i_target->b_offset; if (is_relative_jump(instr)) { if (instr->i_oparg < bsize) { - assert(instr->i_opcode == JUMP_BACKWARD); + assert(instr->i_opcode == JUMP_BACKWARD || + instr->i_opcode == JUMP_BACKWARD_NO_INTERRUPT); instr->i_oparg = bsize - instr->i_oparg; } else { assert(instr->i_opcode != JUMP_BACKWARD); + assert(instr->i_opcode != JUMP_BACKWARD_NO_INTERRUPT); instr->i_oparg -= bsize; } } @@ -8667,14 +8680,12 @@ optimize_basic_block(struct compiler *c, basicblock *bb, PyObject *consts) inst->i_target = inst->i_target->b_next; } target = &inst->i_target->b_instr[0]; - assert(target->i_opcode != JUMP_FORWARD); - assert(target->i_opcode != JUMP_BACKWARD); + assert(!IS_ASSEMBLER_OPCODE(target->i_opcode)); } else { target = &nop; } - assert(inst->i_opcode != JUMP_FORWARD); - assert(inst->i_opcode != JUMP_BACKWARD); + assert(!IS_ASSEMBLER_OPCODE(inst->i_opcode)); switch (inst->i_opcode) { /* Remove LOAD_CONST const; conditional jump */ case LOAD_CONST: @@ -8975,8 +8986,7 @@ normalize_basic_block(basicblock *bb) { /* Mark blocks as exit and/or nofallthrough. Raise SystemError if CFG is malformed. */ for (int i = 0; i < bb->b_iused; i++) { - assert(bb->b_instr[i].i_opcode != JUMP_FORWARD); - assert(bb->b_instr[i].i_opcode != JUMP_BACKWARD); + assert(!IS_ASSEMBLER_OPCODE(bb->b_instr[i].i_opcode)); switch(bb->b_instr[i].i_opcode) { case RETURN_VALUE: case RAISE_VARARGS: @@ -9163,8 +9173,7 @@ optimize_cfg(struct compiler *c, struct assembler *a, PyObject *consts) for (basicblock *b = a->a_entry; b != NULL; b = b->b_next) { if (b->b_iused > 0) { struct instr *b_last_instr = &b->b_instr[b->b_iused - 1]; - assert(b_last_instr->i_opcode != JUMP_FORWARD); - assert(b_last_instr->i_opcode != JUMP_BACKWARD); + assert(!IS_ASSEMBLER_OPCODE(b_last_instr->i_opcode)); if (b_last_instr->i_opcode == JUMP || b_last_instr->i_opcode == JUMP_NO_INTERRUPT) { if (b_last_instr->i_target == b->b_next) { diff --git a/Python/opcode_targets.h b/Python/opcode_targets.h index e71b3e2120f7a..064aa060c8428 100644 --- a/Python/opcode_targets.h +++ b/Python/opcode_targets.h @@ -133,7 +133,7 @@ static void *opcode_targets[256] = { &&TARGET_GET_AWAITABLE, &&TARGET_MAKE_FUNCTION, &&TARGET_BUILD_SLICE, - &&TARGET_JUMP_NO_INTERRUPT, + &&TARGET_JUMP_BACKWARD_NO_INTERRUPT, &&TARGET_MAKE_CELL, &&TARGET_LOAD_CLOSURE, &&TARGET_LOAD_DEREF, From webhook-mailer at python.org Tue Apr 5 08:54:56 2022 From: webhook-mailer at python.org (tiran) Date: Tue, 05 Apr 2022 12:54:56 -0000 Subject: [Python-checkins] bpo-45774: Harden SQLite detection (GH-30016) Message-ID: https://github.com/python/cpython/commit/f1606a5ba50bdc4e7d335d62297b4b4043a25e6e commit: f1606a5ba50bdc4e7d335d62297b4b4043a25e6e branch: main author: Erlend Egeberg Aasland committer: tiran date: 2022-04-05T14:54:47+02:00 summary: bpo-45774: Harden SQLite detection (GH-30016) files: A Misc/NEWS.d/next/Build/2022-03-04-21-24-02.bpo-45774.9AhC0r.rst M configure M configure.ac M pyconfig.h.in diff --git a/Misc/NEWS.d/next/Build/2022-03-04-21-24-02.bpo-45774.9AhC0r.rst b/Misc/NEWS.d/next/Build/2022-03-04-21-24-02.bpo-45774.9AhC0r.rst new file mode 100644 index 0000000000000..0bfc9862ef5c7 --- /dev/null +++ b/Misc/NEWS.d/next/Build/2022-03-04-21-24-02.bpo-45774.9AhC0r.rst @@ -0,0 +1,2 @@ +``configure`` now verifies that all SQLite C APIs needed for the +:mod:`sqlite3` extension module are found. diff --git a/configure b/configure index d6bc7175c2fa0..44912b9c34df8 100755 --- a/configure +++ b/configure @@ -1742,8 +1742,8 @@ Optional Features: --enable-optimizations enable expensive, stable optimizations (PGO, etc.) (default is no) --enable-loadable-sqlite-extensions - support loadable extensions in _sqlite module, see - Doc/library/sqlite3.rst (default is no) + support loadable extensions in the sqlite3 module, + see Doc/library/sqlite3.rst (default is no) --enable-ipv6 enable ipv6 (with ipv4) support, see Doc/library/socket.rst (default is yes if supported) --enable-big-digits[=15|30] @@ -12262,20 +12262,47 @@ $as_echo "yes" >&6; } fi as_fn_append LIBSQLITE3_CFLAGS ' -I$(srcdir)/Modules/_sqlite' + + save_CFLAGS=$CFLAGS save_CPPFLAGS=$CPPFLAGS save_LDFLAGS=$LDFLAGS save_LIBS=$LIBS + CPPFLAGS="$LIBSQLITE3_CFLAGS $CFLAGS" LDFLAGS="$LIBSQLITE3_LIBS $LDFLAGS" ac_fn_c_check_header_mongrel "$LINENO" "sqlite3.h" "ac_cv_header_sqlite3_h" "$ac_includes_default" if test "x$ac_cv_header_sqlite3_h" = xyes; then : - { $as_echo "$as_me:${as_lineno-$LINENO}: checking for sqlite3_open_v2 in -lsqlite3" >&5 -$as_echo_n "checking for sqlite3_open_v2 in -lsqlite3... " >&6; } -if ${ac_cv_lib_sqlite3_sqlite3_open_v2+:} false; then : + have_sqlite3=yes + + cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + + + #include + #if SQLITE_VERSION_NUMBER < 3007015 + # error "SQLite 3.7.15 or higher required" + #endif + +int +main () +{ + + ; + return 0; +} + +_ACEOF +if ac_fn_c_try_compile "$LINENO"; then : + + have_supported_sqlite3=yes + + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for sqlite3_bind_double in -lsqlite3" >&5 +$as_echo_n "checking for sqlite3_bind_double in -lsqlite3... " >&6; } +if ${ac_cv_lib_sqlite3_sqlite3_bind_double+:} false; then : $as_echo_n "(cached) " >&6 else ac_check_lib_save_LIBS=$LIBS @@ -12289,59 +12316,552 @@ cat confdefs.h - <<_ACEOF >conftest.$ac_ext #ifdef __cplusplus extern "C" #endif -char sqlite3_open_v2 (); +char sqlite3_bind_double (); int main () { -return sqlite3_open_v2 (); +return sqlite3_bind_double (); ; return 0; } _ACEOF if ac_fn_c_try_link "$LINENO"; then : - ac_cv_lib_sqlite3_sqlite3_open_v2=yes + ac_cv_lib_sqlite3_sqlite3_bind_double=yes else - ac_cv_lib_sqlite3_sqlite3_open_v2=no + ac_cv_lib_sqlite3_sqlite3_bind_double=no fi rm -f core conftest.err conftest.$ac_objext \ conftest$ac_exeext conftest.$ac_ext LIBS=$ac_check_lib_save_LIBS fi -{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_sqlite3_sqlite3_open_v2" >&5 -$as_echo "$ac_cv_lib_sqlite3_sqlite3_open_v2" >&6; } -if test "x$ac_cv_lib_sqlite3_sqlite3_open_v2" = xyes; then : +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_sqlite3_sqlite3_bind_double" >&5 +$as_echo "$ac_cv_lib_sqlite3_sqlite3_bind_double" >&6; } +if test "x$ac_cv_lib_sqlite3_sqlite3_bind_double" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBSQLITE3 1 +_ACEOF - have_sqlite3=yes - cat confdefs.h - <<_ACEOF >conftest.$ac_ext + LIBS="-lsqlite3 $LIBS" + +else + + have_supported_sqlite3=no + +fi + + + + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for sqlite3_column_decltype in -lsqlite3" >&5 +$as_echo_n "checking for sqlite3_column_decltype in -lsqlite3... " >&6; } +if ${ac_cv_lib_sqlite3_sqlite3_column_decltype+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-lsqlite3 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext /* end confdefs.h. */ +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char sqlite3_column_decltype (); +int +main () +{ +return sqlite3_column_decltype (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_sqlite3_sqlite3_column_decltype=yes +else + ac_cv_lib_sqlite3_sqlite3_column_decltype=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_sqlite3_sqlite3_column_decltype" >&5 +$as_echo "$ac_cv_lib_sqlite3_sqlite3_column_decltype" >&6; } +if test "x$ac_cv_lib_sqlite3_sqlite3_column_decltype" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBSQLITE3 1 +_ACEOF + + LIBS="-lsqlite3 $LIBS" + +else + + have_supported_sqlite3=no + +fi - #include - #if SQLITE_VERSION_NUMBER < 3007015 - # error "SQLite 3.7.15 or higher required" - #endif + + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for sqlite3_column_double in -lsqlite3" >&5 +$as_echo_n "checking for sqlite3_column_double in -lsqlite3... " >&6; } +if ${ac_cv_lib_sqlite3_sqlite3_column_double+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-lsqlite3 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char sqlite3_column_double (); int main () { +return sqlite3_column_double (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_sqlite3_sqlite3_column_double=yes +else + ac_cv_lib_sqlite3_sqlite3_column_double=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_sqlite3_sqlite3_column_double" >&5 +$as_echo "$ac_cv_lib_sqlite3_sqlite3_column_double" >&6; } +if test "x$ac_cv_lib_sqlite3_sqlite3_column_double" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBSQLITE3 1 +_ACEOF + + LIBS="-lsqlite3 $LIBS" + +else + + have_supported_sqlite3=no + +fi + + + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for sqlite3_complete in -lsqlite3" >&5 +$as_echo_n "checking for sqlite3_complete in -lsqlite3... " >&6; } +if ${ac_cv_lib_sqlite3_sqlite3_complete+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-lsqlite3 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char sqlite3_complete (); +int +main () +{ +return sqlite3_complete (); ; return 0; } +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_sqlite3_sqlite3_complete=yes +else + ac_cv_lib_sqlite3_sqlite3_complete=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_sqlite3_sqlite3_complete" >&5 +$as_echo "$ac_cv_lib_sqlite3_sqlite3_complete" >&6; } +if test "x$ac_cv_lib_sqlite3_sqlite3_complete" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBSQLITE3 1 +_ACEOF + + LIBS="-lsqlite3 $LIBS" + +else + + have_supported_sqlite3=no +fi + + + + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for sqlite3_enable_shared_cache in -lsqlite3" >&5 +$as_echo_n "checking for sqlite3_enable_shared_cache in -lsqlite3... " >&6; } +if ${ac_cv_lib_sqlite3_sqlite3_enable_shared_cache+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-lsqlite3 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char sqlite3_enable_shared_cache (); +int +main () +{ +return sqlite3_enable_shared_cache (); + ; + return 0; +} _ACEOF -if ac_fn_c_try_compile "$LINENO"; then : - have_supported_sqlite3=yes +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_sqlite3_sqlite3_enable_shared_cache=yes else - have_supported_sqlite3=no + ac_cv_lib_sqlite3_sqlite3_enable_shared_cache=no fi -rm -f core conftest.err conftest.$ac_objext conftest.$ac_ext +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_sqlite3_sqlite3_enable_shared_cache" >&5 +$as_echo "$ac_cv_lib_sqlite3_sqlite3_enable_shared_cache" >&6; } +if test "x$ac_cv_lib_sqlite3_sqlite3_enable_shared_cache" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBSQLITE3 1 +_ACEOF + + LIBS="-lsqlite3 $LIBS" + +else + + have_supported_sqlite3=no + +fi + + + + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for sqlite3_progress_handler in -lsqlite3" >&5 +$as_echo_n "checking for sqlite3_progress_handler in -lsqlite3... " >&6; } +if ${ac_cv_lib_sqlite3_sqlite3_progress_handler+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-lsqlite3 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char sqlite3_progress_handler (); +int +main () +{ +return sqlite3_progress_handler (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_sqlite3_sqlite3_progress_handler=yes +else + ac_cv_lib_sqlite3_sqlite3_progress_handler=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_sqlite3_sqlite3_progress_handler" >&5 +$as_echo "$ac_cv_lib_sqlite3_sqlite3_progress_handler" >&6; } +if test "x$ac_cv_lib_sqlite3_sqlite3_progress_handler" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBSQLITE3 1 +_ACEOF + + LIBS="-lsqlite3 $LIBS" + +else + + have_supported_sqlite3=no + +fi + + + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for sqlite3_result_double in -lsqlite3" >&5 +$as_echo_n "checking for sqlite3_result_double in -lsqlite3... " >&6; } +if ${ac_cv_lib_sqlite3_sqlite3_result_double+:} false; then : + $as_echo_n "(cached) " >&6 else - have_sqlite3=no + ac_check_lib_save_LIBS=$LIBS +LIBS="-lsqlite3 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char sqlite3_result_double (); +int +main () +{ +return sqlite3_result_double (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_sqlite3_sqlite3_result_double=yes +else + ac_cv_lib_sqlite3_sqlite3_result_double=no fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_sqlite3_sqlite3_result_double" >&5 +$as_echo "$ac_cv_lib_sqlite3_sqlite3_result_double" >&6; } +if test "x$ac_cv_lib_sqlite3_sqlite3_result_double" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBSQLITE3 1 +_ACEOF - { $as_echo "$as_me:${as_lineno-$LINENO}: checking for sqlite3_load_extension in -lsqlite3" >&5 + LIBS="-lsqlite3 $LIBS" + +else + + have_supported_sqlite3=no + +fi + + + + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for sqlite3_set_authorizer in -lsqlite3" >&5 +$as_echo_n "checking for sqlite3_set_authorizer in -lsqlite3... " >&6; } +if ${ac_cv_lib_sqlite3_sqlite3_set_authorizer+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-lsqlite3 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char sqlite3_set_authorizer (); +int +main () +{ +return sqlite3_set_authorizer (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_sqlite3_sqlite3_set_authorizer=yes +else + ac_cv_lib_sqlite3_sqlite3_set_authorizer=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_sqlite3_sqlite3_set_authorizer" >&5 +$as_echo "$ac_cv_lib_sqlite3_sqlite3_set_authorizer" >&6; } +if test "x$ac_cv_lib_sqlite3_sqlite3_set_authorizer" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBSQLITE3 1 +_ACEOF + + LIBS="-lsqlite3 $LIBS" + +else + + have_supported_sqlite3=no + +fi + + + + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for sqlite3_trace_v2 in -lsqlite3" >&5 +$as_echo_n "checking for sqlite3_trace_v2 in -lsqlite3... " >&6; } +if ${ac_cv_lib_sqlite3_sqlite3_trace_v2+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-lsqlite3 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char sqlite3_trace_v2 (); +int +main () +{ +return sqlite3_trace_v2 (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_sqlite3_sqlite3_trace_v2=yes +else + ac_cv_lib_sqlite3_sqlite3_trace_v2=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_sqlite3_sqlite3_trace_v2" >&5 +$as_echo "$ac_cv_lib_sqlite3_sqlite3_trace_v2" >&6; } +if test "x$ac_cv_lib_sqlite3_sqlite3_trace_v2" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBSQLITE3 1 +_ACEOF + + LIBS="-lsqlite3 $LIBS" + +else + + + + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for sqlite3_trace in -lsqlite3" >&5 +$as_echo_n "checking for sqlite3_trace in -lsqlite3... " >&6; } +if ${ac_cv_lib_sqlite3_sqlite3_trace+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-lsqlite3 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char sqlite3_trace (); +int +main () +{ +return sqlite3_trace (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_sqlite3_sqlite3_trace=yes +else + ac_cv_lib_sqlite3_sqlite3_trace=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_sqlite3_sqlite3_trace" >&5 +$as_echo "$ac_cv_lib_sqlite3_sqlite3_trace" >&6; } +if test "x$ac_cv_lib_sqlite3_sqlite3_trace" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBSQLITE3 1 +_ACEOF + + LIBS="-lsqlite3 $LIBS" + +else + + have_supported_sqlite3=no + +fi + + + + +fi + + + + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for sqlite3_value_double in -lsqlite3" >&5 +$as_echo_n "checking for sqlite3_value_double in -lsqlite3... " >&6; } +if ${ac_cv_lib_sqlite3_sqlite3_value_double+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-lsqlite3 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char sqlite3_value_double (); +int +main () +{ +return sqlite3_value_double (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_sqlite3_sqlite3_value_double=yes +else + ac_cv_lib_sqlite3_sqlite3_value_double=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_sqlite3_sqlite3_value_double" >&5 +$as_echo "$ac_cv_lib_sqlite3_sqlite3_value_double" >&6; } +if test "x$ac_cv_lib_sqlite3_sqlite3_value_double" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBSQLITE3 1 +_ACEOF + + LIBS="-lsqlite3 $LIBS" + +else + + have_supported_sqlite3=no + +fi + + + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for sqlite3_load_extension in -lsqlite3" >&5 $as_echo_n "checking for sqlite3_load_extension in -lsqlite3... " >&6; } if ${ac_cv_lib_sqlite3_sqlite3_load_extension+:} false; then : $as_echo_n "(cached) " >&6 @@ -12381,9 +12901,17 @@ if test "x$ac_cv_lib_sqlite3_sqlite3_load_extension" = xyes; then : have_sqlite3_load_extension=yes else have_sqlite3_load_extension=no + fi +else + + have_supported_sqlite3=no + +fi +rm -f core conftest.err conftest.$ac_objext conftest.$ac_ext + fi @@ -12395,30 +12923,37 @@ LIBS=$save_LIBS -# Check for support for loadable sqlite extensions { $as_echo "$as_me:${as_lineno-$LINENO}: checking for --enable-loadable-sqlite-extensions" >&5 $as_echo_n "checking for --enable-loadable-sqlite-extensions... " >&6; } # Check whether --enable-loadable-sqlite-extensions was given. if test "${enable_loadable_sqlite_extensions+set}" = set; then : - enableval=$enable_loadable_sqlite_extensions; if test "x$have_sqlite3_load_extension" = xno; then : - { $as_echo "$as_me:${as_lineno-$LINENO}: WARNING: Your version of SQLite does not support loadable extensions" >&5 + enableval=$enable_loadable_sqlite_extensions; + if test "x$have_sqlite3_load_extension" = xno; then : + + { $as_echo "$as_me:${as_lineno-$LINENO}: result: n/a" >&5 +$as_echo "n/a" >&6; } + { $as_echo "$as_me:${as_lineno-$LINENO}: WARNING: Your version of SQLite does not support loadable extensions" >&5 $as_echo "$as_me: WARNING: Your version of SQLite does not support loadable extensions" >&2;} -fi + else - enable_loadable_sqlite_extensions=no -fi -{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $enable_loadable_sqlite_extensions" >&5 -$as_echo "$enable_loadable_sqlite_extensions" >&6; } + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } -if test "x$enable_loadable_sqlite_extensions" = xyes; then : +$as_echo "#define PY_SQLITE_ENABLE_LOAD_EXTENSION 1" >>confdefs.h -$as_echo "#define PY_SQLITE_ENABLE_LOAD_EXTENSION 1" >>confdefs.h +fi + +else + + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } fi + found_tcltk=no for _QUERY in \ "tcl >= 8.5.12 tk >= 8.5.12" \ diff --git a/configure.ac b/configure.ac index 53bbc3e7b199c..c02adf7bf3f14 100644 --- a/configure.ac +++ b/configure.ac @@ -3562,45 +3562,79 @@ PKG_CHECK_MODULES( ) AS_VAR_APPEND([LIBSQLITE3_CFLAGS], [' -I$(srcdir)/Modules/_sqlite']) -WITH_SAVE_ENV( +dnl PY_CHECK_SQLITE_FUNC(FUNCTION, IF-FOUND, IF-NOT-FOUND) +AC_DEFUN([PY_CHECK_SQLITE_FUNC], [ + AC_CHECK_LIB([sqlite3], [$1], [$2], [ + m4_ifblank([$3], [have_supported_sqlite3=no], [$3]) + ]) +]) + +WITH_SAVE_ENV([ dnl bpo-45774/GH-29507: The CPP check in AC_CHECK_HEADER can fail on FreeBSD, dnl hence CPPFLAGS instead of CFLAGS. CPPFLAGS="$LIBSQLITE3_CFLAGS $CFLAGS" LDFLAGS="$LIBSQLITE3_LIBS $LDFLAGS" AC_CHECK_HEADER([sqlite3.h], [ - AC_CHECK_LIB([sqlite3], [sqlite3_open_v2], [ - have_sqlite3=yes - AC_COMPILE_IFELSE([ - AC_LANG_PROGRAM([ - #include - #if SQLITE_VERSION_NUMBER < 3007015 - # error "SQLite 3.7.15 or higher required" - #endif - ], []) - ], [have_supported_sqlite3=yes], [have_supported_sqlite3=no]) - ], [have_sqlite3=no]) - AC_CHECK_LIB([sqlite3], [sqlite3_load_extension], - [have_sqlite3_load_extension=yes], - [have_sqlite3_load_extension=no]) - ]) -) + have_sqlite3=yes -# Check for support for loadable sqlite extensions -AC_MSG_CHECKING(for --enable-loadable-sqlite-extensions) -AC_ARG_ENABLE(loadable-sqlite-extensions, - AS_HELP_STRING([--enable-loadable-sqlite-extensions], - [support loadable extensions in _sqlite module, see Doc/library/sqlite3.rst (default is no)]), - [AS_VAR_IF([have_sqlite3_load_extension], [no], - [AC_MSG_WARN([Your version of SQLite does not support loadable extensions])])], - [enable_loadable_sqlite_extensions=no]) -AC_MSG_RESULT($enable_loadable_sqlite_extensions) - -AS_VAR_IF([enable_loadable_sqlite_extensions], [yes], [ - AC_DEFINE(PY_SQLITE_ENABLE_LOAD_EXTENSION, 1, - [Define to 1 to build the sqlite module with loadable extensions support.]) + AC_COMPILE_IFELSE([ + AC_LANG_PROGRAM([ + #include + #if SQLITE_VERSION_NUMBER < 3007015 + # error "SQLite 3.7.15 or higher required" + #endif + ], []) + ], [ + have_supported_sqlite3=yes + dnl Check that required functions are in place. A lot of stuff may be + dnl omitted with SQLITE_OMIT_* compile time defines. + PY_CHECK_SQLITE_FUNC([sqlite3_bind_double]) + PY_CHECK_SQLITE_FUNC([sqlite3_column_decltype]) + PY_CHECK_SQLITE_FUNC([sqlite3_column_double]) + PY_CHECK_SQLITE_FUNC([sqlite3_complete]) + PY_CHECK_SQLITE_FUNC([sqlite3_enable_shared_cache]) + PY_CHECK_SQLITE_FUNC([sqlite3_progress_handler]) + PY_CHECK_SQLITE_FUNC([sqlite3_result_double]) + PY_CHECK_SQLITE_FUNC([sqlite3_set_authorizer]) + PY_CHECK_SQLITE_FUNC([sqlite3_trace_v2], [], [ + PY_CHECK_SQLITE_FUNC([sqlite3_trace]) + ]) + PY_CHECK_SQLITE_FUNC([sqlite3_value_double]) + AC_CHECK_LIB([sqlite3], [sqlite3_load_extension], + [have_sqlite3_load_extension=yes], + [have_sqlite3_load_extension=no] + ) + ], [ + have_supported_sqlite3=no + ]) + ]) ]) +dnl Check for support for loadable sqlite extensions +AC_MSG_CHECKING([for --enable-loadable-sqlite-extensions]) +AC_ARG_ENABLE([loadable-sqlite-extensions], + AS_HELP_STRING( + [--enable-loadable-sqlite-extensions], [ + support loadable extensions in the sqlite3 module, see + Doc/library/sqlite3.rst (default is no) + ] + ), [ + AS_VAR_IF([have_sqlite3_load_extension], [no], [ + AC_MSG_RESULT([n/a]) + AC_MSG_WARN([Your version of SQLite does not support loadable extensions]) + ], [ + AC_MSG_RESULT([yes]) + AC_DEFINE( + [PY_SQLITE_ENABLE_LOAD_EXTENSION], [1], + [Define to 1 to build the sqlite module with loadable extensions support.] + ) + ]) + ], [ + AC_MSG_RESULT([no]) + ] +) + dnl dnl Detect Tcl/Tk. Use pkg-config if available. dnl diff --git a/pyconfig.h.in b/pyconfig.h.in index 3d2020c896934..be776f734163f 100644 --- a/pyconfig.h.in +++ b/pyconfig.h.in @@ -655,6 +655,9 @@ /* Define to 1 if you have the `sendfile' library (-lsendfile). */ #undef HAVE_LIBSENDFILE +/* Define to 1 if you have the `sqlite3' library (-lsqlite3). */ +#undef HAVE_LIBSQLITE3 + /* Define to 1 if you have the header file. */ #undef HAVE_LIBUTIL_H From webhook-mailer at python.org Tue Apr 5 09:47:24 2022 From: webhook-mailer at python.org (pablogsal) Date: Tue, 05 Apr 2022 13:47:24 -0000 Subject: [Python-checkins] bpo-47212: Improve error messages for un-parenthesized generator expressions (GH-32302) Message-ID: https://github.com/python/cpython/commit/aa0f056a00c4bcaef83d729e042359ddae903382 commit: aa0f056a00c4bcaef83d729e042359ddae903382 branch: main author: Matthieu Dartiailh committer: pablogsal date: 2022-04-05T14:47:13+01:00 summary: bpo-47212: Improve error messages for un-parenthesized generator expressions (GH-32302) files: A Misc/NEWS.d/next/Core and Builtins/2022-04-05-11-29-21.bpo-47212.leF4pz.rst M Grammar/python.gram M Lib/test/test_exceptions.py M Lib/test/test_syntax.py M Parser/action_helpers.c M Parser/parser.c M Parser/pegen.h diff --git a/Grammar/python.gram b/Grammar/python.gram index b9965d224a85b..15c40b6bbbacd 100644 --- a/Grammar/python.gram +++ b/Grammar/python.gram @@ -1073,12 +1073,12 @@ func_type_comment[Token*]: invalid_arguments: | a=args ',' '*' { RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "iterable argument unpacking follows keyword argument unpacking") } | a=expression b=for_if_clauses ',' [args | expression for_if_clauses] { - RAISE_SYNTAX_ERROR_KNOWN_RANGE(a, PyPegen_last_item(b, comprehension_ty)->target, "Generator expression must be parenthesized") } + RAISE_SYNTAX_ERROR_KNOWN_RANGE(a, _PyPegen_get_last_comprehension_item(PyPegen_last_item(b, comprehension_ty)), "Generator expression must be parenthesized") } | a=NAME b='=' expression for_if_clauses { RAISE_SYNTAX_ERROR_KNOWN_RANGE(a, b, "invalid syntax. Maybe you meant '==' or ':=' instead of '='?")} | a=args b=for_if_clauses { _PyPegen_nonparen_genexp_in_call(p, a, b) } | args ',' a=expression b=for_if_clauses { - RAISE_SYNTAX_ERROR_KNOWN_RANGE(a, asdl_seq_GET(b, b->size-1)->target, "Generator expression must be parenthesized") } + RAISE_SYNTAX_ERROR_KNOWN_RANGE(a, _PyPegen_get_last_comprehension_item(PyPegen_last_item(b, comprehension_ty)), "Generator expression must be parenthesized") } | a=args ',' args { _PyPegen_arguments_parsing_error(p, a) } invalid_kwarg: | a[Token*]=('True'|'False'|'None') b='=' { @@ -1257,7 +1257,7 @@ invalid_finally_stmt: invalid_except_stmt_indent: | a='except' expression ['as' NAME ] ':' NEWLINE !INDENT { RAISE_INDENTATION_ERROR("expected an indented block after 'except' statement on line %d", a->lineno) } - | a='except' ':' NEWLINE !INDENT { RAISE_SYNTAX_ERROR("expected an indented block after except statement on line %d", a->lineno) } + | a='except' ':' NEWLINE !INDENT { RAISE_INDENTATION_ERROR("expected an indented block after 'except' statement on line %d", a->lineno) } invalid_except_star_stmt_indent: | a='except' '*' expression ['as' NAME ] ':' NEWLINE !INDENT { RAISE_INDENTATION_ERROR("expected an indented block after 'except*' statement on line %d", a->lineno) } diff --git a/Lib/test/test_exceptions.py b/Lib/test/test_exceptions.py index a75b7fae5518e..6dca79efef180 100644 --- a/Lib/test/test_exceptions.py +++ b/Lib/test/test_exceptions.py @@ -198,12 +198,17 @@ def ckmsg(src, msg, exception=SyntaxError): s = '''if True:\n print()\n\texec "mixed tabs and spaces"''' ckmsg(s, "inconsistent use of tabs and spaces in indentation", TabError) - def check(self, src, lineno, offset, encoding='utf-8'): + def check(self, src, lineno, offset, end_lineno=None, end_offset=None, encoding='utf-8'): with self.subTest(source=src, lineno=lineno, offset=offset): with self.assertRaises(SyntaxError) as cm: compile(src, '', 'exec') self.assertEqual(cm.exception.lineno, lineno) self.assertEqual(cm.exception.offset, offset) + if end_lineno is not None: + self.assertEqual(cm.exception.end_lineno, end_lineno) + if end_offset is not None: + self.assertEqual(cm.exception.end_offset, end_offset) + if cm.exception.text is not None: if not isinstance(src, str): src = src.decode(encoding, 'replace') @@ -235,6 +240,10 @@ def testSyntaxErrorOffset(self): check('match ...:\n case {**rest, "key": value}:\n ...', 2, 19) check("[a b c d e f]", 1, 2) check("for x yfff:", 1, 7) + check("f(a for a in b, c)", 1, 3, 1, 15) + check("f(a for a in b if a, c)", 1, 3, 1, 20) + check("f(a, b for b in c)", 1, 6, 1, 18) + check("f(a, b for b in c, d)", 1, 6, 1, 18) # Errors thrown by compile.c check('class foo:return 1', 1, 11) diff --git a/Lib/test/test_syntax.py b/Lib/test/test_syntax.py index 3e79ebfc68351..96e5c129c6599 100644 --- a/Lib/test/test_syntax.py +++ b/Lib/test/test_syntax.py @@ -1307,6 +1307,13 @@ Traceback (most recent call last): IndentationError: expected an indented block after 'try' statement on line 1 + >>> try: + ... something() + ... except: + ... pass + Traceback (most recent call last): + IndentationError: expected an indented block after 'except' statement on line 3 + >>> try: ... something() ... except A: diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-04-05-11-29-21.bpo-47212.leF4pz.rst b/Misc/NEWS.d/next/Core and Builtins/2022-04-05-11-29-21.bpo-47212.leF4pz.rst new file mode 100644 index 0000000000000..8f1f6b6cfbbb8 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2022-04-05-11-29-21.bpo-47212.leF4pz.rst @@ -0,0 +1,3 @@ +Raise :exc:`IndentationError` instead of :exc:`SyntaxError` for a bare +``except`` with no following indent. Improve :exc:`SyntaxError` locations for +an un-parenthesized generator used as arguments. Patch by Matthieu Dartiailh. diff --git a/Parser/action_helpers.c b/Parser/action_helpers.c index e5d7b667f7f5e..d1be679aff2e7 100644 --- a/Parser/action_helpers.c +++ b/Parser/action_helpers.c @@ -1145,7 +1145,7 @@ _PyPegen_get_expr_name(expr_ty e) } } -static inline expr_ty +expr_ty _PyPegen_get_last_comprehension_item(comprehension_ty comprehension) { if (comprehension->ifs == NULL || asdl_seq_LEN(comprehension->ifs) == 0) { return comprehension->iter; diff --git a/Parser/parser.c b/Parser/parser.c index 40c5d62fe49d7..adc8d509eb7d7 100644 --- a/Parser/parser.c +++ b/Parser/parser.c @@ -18968,7 +18968,7 @@ invalid_arguments_rule(Parser *p) ) { D(fprintf(stderr, "%*c+ invalid_arguments[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "expression for_if_clauses ',' [args | expression for_if_clauses]")); - _res = RAISE_SYNTAX_ERROR_KNOWN_RANGE ( a , PyPegen_last_item ( b , comprehension_ty ) -> target , "Generator expression must be parenthesized" ); + _res = RAISE_SYNTAX_ERROR_KNOWN_RANGE ( a , _PyPegen_get_last_comprehension_item ( PyPegen_last_item ( b , comprehension_ty ) ) , "Generator expression must be parenthesized" ); if (_res == NULL && PyErr_Occurred()) { p->error_indicator = 1; p->level--; @@ -19061,7 +19061,7 @@ invalid_arguments_rule(Parser *p) ) { D(fprintf(stderr, "%*c+ invalid_arguments[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "args ',' expression for_if_clauses")); - _res = RAISE_SYNTAX_ERROR_KNOWN_RANGE ( a , asdl_seq_GET ( b , b -> size - 1 ) -> target , "Generator expression must be parenthesized" ); + _res = RAISE_SYNTAX_ERROR_KNOWN_RANGE ( a , _PyPegen_get_last_comprehension_item ( PyPegen_last_item ( b , comprehension_ty ) ) , "Generator expression must be parenthesized" ); if (_res == NULL && PyErr_Occurred()) { p->error_indicator = 1; p->level--; @@ -22190,7 +22190,7 @@ invalid_except_stmt_indent_rule(Parser *p) ) { D(fprintf(stderr, "%*c+ invalid_except_stmt_indent[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "'except' ':' NEWLINE !INDENT")); - _res = RAISE_SYNTAX_ERROR ( "expected an indented block after except statement on line %d" , a -> lineno ); + _res = RAISE_INDENTATION_ERROR ( "expected an indented block after 'except' statement on line %d" , a -> lineno ); if (_res == NULL && PyErr_Occurred()) { p->error_indicator = 1; p->level--; diff --git a/Parser/pegen.h b/Parser/pegen.h index 77d5ca8418a97..fe0c327b87556 100644 --- a/Parser/pegen.h +++ b/Parser/pegen.h @@ -324,6 +324,7 @@ int _PyPegen_check_barry_as_flufl(Parser *, Token *); int _PyPegen_check_legacy_stmt(Parser *p, expr_ty t); mod_ty _PyPegen_make_module(Parser *, asdl_stmt_seq *); void *_PyPegen_arguments_parsing_error(Parser *, expr_ty); +expr_ty _PyPegen_get_last_comprehension_item(comprehension_ty comprehension); void *_PyPegen_nonparen_genexp_in_call(Parser *p, expr_ty args, asdl_comprehension_seq *comprehensions); // Parser API From webhook-mailer at python.org Tue Apr 5 10:15:35 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Tue, 05 Apr 2022 14:15:35 -0000 Subject: [Python-checkins] bpo-41930: Add support for SQLite serialise/deserialise API (GH-26728) Message-ID: https://github.com/python/cpython/commit/a7551247e7cb7010fb4735281f1afa4abeb8a9cc commit: a7551247e7cb7010fb4735281f1afa4abeb8a9cc branch: main author: Erlend Egeberg Aasland committer: JelleZijlstra date: 2022-04-05T07:15:25-07:00 summary: bpo-41930: Add support for SQLite serialise/deserialise API (GH-26728) Co-authored-by: Jelle Zijlstra Co-authored-by: Kumar Aditya <59607654+kumaraditya303 at users.noreply.github.com> files: A Misc/NEWS.d/next/Library/2021-06-17-00-02-58.bpo-41930.JS6fsd.rst M Doc/library/sqlite3.rst M Doc/whatsnew/3.11.rst M Lib/test/test_sqlite3/test_dbapi.py M Modules/_sqlite/clinic/connection.c.h M Modules/_sqlite/connection.c M PCbuild/_sqlite3.vcxproj M configure M configure.ac M pyconfig.h.in diff --git a/Doc/library/sqlite3.rst b/Doc/library/sqlite3.rst index e70d038e61d82..852b68437a265 100644 --- a/Doc/library/sqlite3.rst +++ b/Doc/library/sqlite3.rst @@ -748,6 +748,44 @@ Connection Objects .. versionadded:: 3.11 + .. method:: serialize(*, name="main") + + This method serializes a database into a :class:`bytes` object. For an + ordinary on-disk database file, the serialization is just a copy of the + disk file. For an in-memory database or a "temp" database, the + serialization is the same sequence of bytes which would be written to + disk if that database were backed up to disk. + + *name* is the database to be serialized, and defaults to the main + database. + + .. note:: + + This method is only available if the underlying SQLite library has the + serialize API. + + .. versionadded:: 3.11 + + + .. method:: deserialize(data, /, *, name="main") + + This method causes the database connection to disconnect from database + *name*, and reopen *name* as an in-memory database based on the + serialization contained in *data*. Deserialization will raise + :exc:`OperationalError` if the database connection is currently involved + in a read transaction or a backup operation. :exc:`DataError` will be + raised if ``len(data)`` is larger than ``2**63 - 1``, and + :exc:`DatabaseError` will be raised if *data* does not contain a valid + SQLite database. + + .. note:: + + This method is only available if the underlying SQLite library has the + deserialize API. + + .. versionadded:: 3.11 + + .. _sqlite3-cursor-objects: Cursor Objects diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index d0c10a9100997..c312645c31cd7 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -366,6 +366,11 @@ sqlite3 Instead we leave it to the SQLite library to handle these cases. (Contributed by Erlend E. Aasland in :issue:`44092`.) +* Add :meth:`~sqlite3.Connection.serialize` and + :meth:`~sqlite3.Connection.deserialize` to :class:`sqlite3.Connection` for + serializing and deserializing databases. + (Contributed by Erlend E. Aasland in :issue:`41930`.) + sys --- diff --git a/Lib/test/test_sqlite3/test_dbapi.py b/Lib/test/test_sqlite3/test_dbapi.py index 177c2cd327ff3..02482816cb933 100644 --- a/Lib/test/test_sqlite3/test_dbapi.py +++ b/Lib/test/test_sqlite3/test_dbapi.py @@ -29,6 +29,7 @@ from test.support import ( SHORT_TIMEOUT, + bigmemtest, check_disallow_instantiation, threading_helper, ) @@ -603,6 +604,56 @@ def test_uninit_operations(self): func) + at unittest.skipUnless(hasattr(sqlite.Connection, "serialize"), + "Needs SQLite serialize API") +class SerializeTests(unittest.TestCase): + def test_serialize_deserialize(self): + with memory_database() as cx: + with cx: + cx.execute("create table t(t)") + data = cx.serialize() + self.assertEqual(len(data), 8192) + + # Remove test table, verify that it was removed. + with cx: + cx.execute("drop table t") + regex = "no such table" + with self.assertRaisesRegex(sqlite.OperationalError, regex): + cx.execute("select t from t") + + # Deserialize and verify that test table is restored. + cx.deserialize(data) + cx.execute("select t from t") + + def test_deserialize_wrong_args(self): + dataset = ( + (BufferError, memoryview(b"blob")[::2]), + (TypeError, []), + (TypeError, 1), + (TypeError, None), + ) + for exc, arg in dataset: + with self.subTest(exc=exc, arg=arg): + with memory_database() as cx: + self.assertRaises(exc, cx.deserialize, arg) + + def test_deserialize_corrupt_database(self): + with memory_database() as cx: + regex = "file is not a database" + with self.assertRaisesRegex(sqlite.DatabaseError, regex): + cx.deserialize(b"\0\1\3") + # SQLite does not generate an error until you try to query the + # deserialized database. + cx.execute("create table fail(f)") + + @unittest.skipUnless(sys.maxsize > 2**32, 'requires 64bit platform') + @bigmemtest(size=2**63, memuse=3, dry_run=False) + def test_deserialize_too_much_data_64bit(self): + with memory_database() as cx: + with self.assertRaisesRegex(OverflowError, "'data' is too large"): + cx.deserialize(b"b" * size) + + class OpenTests(unittest.TestCase): _sql = "create table test(id integer)" @@ -1030,6 +1081,10 @@ def test_check_connection_thread(self): lambda: self.con.setlimit(sqlite.SQLITE_LIMIT_LENGTH, -1), lambda: self.con.getlimit(sqlite.SQLITE_LIMIT_LENGTH), ] + if hasattr(sqlite.Connection, "serialize"): + fns.append(lambda: self.con.serialize()) + fns.append(lambda: self.con.deserialize(b"")) + for fn in fns: with self.subTest(fn=fn): self._run_test(fn) diff --git a/Misc/NEWS.d/next/Library/2021-06-17-00-02-58.bpo-41930.JS6fsd.rst b/Misc/NEWS.d/next/Library/2021-06-17-00-02-58.bpo-41930.JS6fsd.rst new file mode 100644 index 0000000000000..ce494e7225e22 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-06-17-00-02-58.bpo-41930.JS6fsd.rst @@ -0,0 +1,3 @@ +Add :meth:`~sqlite3.Connection.serialize` and +:meth:`~sqlite3.Connection.deserialize` support to :mod:`sqlite3`. Patch by +Erlend E. Aasland. diff --git a/Modules/_sqlite/clinic/connection.c.h b/Modules/_sqlite/clinic/connection.c.h index 111e344fd2ae1..99ef94ecd71ec 100644 --- a/Modules/_sqlite/clinic/connection.c.h +++ b/Modules/_sqlite/clinic/connection.c.h @@ -693,6 +693,156 @@ pysqlite_connection_create_collation(pysqlite_Connection *self, PyTypeObject *cl return return_value; } +#if defined(PY_SQLITE_HAVE_SERIALIZE) + +PyDoc_STRVAR(serialize__doc__, +"serialize($self, /, *, name=\'main\')\n" +"--\n" +"\n" +"Serialize a database into a byte string.\n" +"\n" +" name\n" +" Which database to serialize.\n" +"\n" +"For an ordinary on-disk database file, the serialization is just a copy of the\n" +"disk file. For an in-memory database or a \"temp\" database, the serialization is\n" +"the same sequence of bytes which would be written to disk if that database\n" +"were backed up to disk."); + +#define SERIALIZE_METHODDEF \ + {"serialize", (PyCFunction)(void(*)(void))serialize, METH_FASTCALL|METH_KEYWORDS, serialize__doc__}, + +static PyObject * +serialize_impl(pysqlite_Connection *self, const char *name); + +static PyObject * +serialize(pysqlite_Connection *self, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) +{ + PyObject *return_value = NULL; + static const char * const _keywords[] = {"name", NULL}; + static _PyArg_Parser _parser = {NULL, _keywords, "serialize", 0}; + PyObject *argsbuf[1]; + Py_ssize_t noptargs = nargs + (kwnames ? PyTuple_GET_SIZE(kwnames) : 0) - 0; + const char *name = "main"; + + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 0, 0, 0, argsbuf); + if (!args) { + goto exit; + } + if (!noptargs) { + goto skip_optional_kwonly; + } + if (!PyUnicode_Check(args[0])) { + _PyArg_BadArgument("serialize", "argument 'name'", "str", args[0]); + goto exit; + } + Py_ssize_t name_length; + name = PyUnicode_AsUTF8AndSize(args[0], &name_length); + if (name == NULL) { + goto exit; + } + if (strlen(name) != (size_t)name_length) { + PyErr_SetString(PyExc_ValueError, "embedded null character"); + goto exit; + } +skip_optional_kwonly: + return_value = serialize_impl(self, name); + +exit: + return return_value; +} + +#endif /* defined(PY_SQLITE_HAVE_SERIALIZE) */ + +#if defined(PY_SQLITE_HAVE_SERIALIZE) + +PyDoc_STRVAR(deserialize__doc__, +"deserialize($self, data, /, *, name=\'main\')\n" +"--\n" +"\n" +"Load a serialized database.\n" +"\n" +" data\n" +" The serialized database content.\n" +" name\n" +" Which database to reopen with the deserialization.\n" +"\n" +"The deserialize interface causes the database connection to disconnect from the\n" +"target database, and then reopen it as an in-memory database based on the given\n" +"serialized data.\n" +"\n" +"The deserialize interface will fail with SQLITE_BUSY if the database is\n" +"currently in a read transaction or is involved in a backup operation."); + +#define DESERIALIZE_METHODDEF \ + {"deserialize", (PyCFunction)(void(*)(void))deserialize, METH_FASTCALL|METH_KEYWORDS, deserialize__doc__}, + +static PyObject * +deserialize_impl(pysqlite_Connection *self, Py_buffer *data, + const char *name); + +static PyObject * +deserialize(pysqlite_Connection *self, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) +{ + PyObject *return_value = NULL; + static const char * const _keywords[] = {"", "name", NULL}; + static _PyArg_Parser _parser = {NULL, _keywords, "deserialize", 0}; + PyObject *argsbuf[2]; + Py_ssize_t noptargs = nargs + (kwnames ? PyTuple_GET_SIZE(kwnames) : 0) - 1; + Py_buffer data = {NULL, NULL}; + const char *name = "main"; + + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 1, 1, 0, argsbuf); + if (!args) { + goto exit; + } + if (PyUnicode_Check(args[0])) { + Py_ssize_t len; + const char *ptr = PyUnicode_AsUTF8AndSize(args[0], &len); + if (ptr == NULL) { + goto exit; + } + PyBuffer_FillInfo(&data, args[0], (void *)ptr, len, 1, 0); + } + else { /* any bytes-like object */ + if (PyObject_GetBuffer(args[0], &data, PyBUF_SIMPLE) != 0) { + goto exit; + } + if (!PyBuffer_IsContiguous(&data, 'C')) { + _PyArg_BadArgument("deserialize", "argument 1", "contiguous buffer", args[0]); + goto exit; + } + } + if (!noptargs) { + goto skip_optional_kwonly; + } + if (!PyUnicode_Check(args[1])) { + _PyArg_BadArgument("deserialize", "argument 'name'", "str", args[1]); + goto exit; + } + Py_ssize_t name_length; + name = PyUnicode_AsUTF8AndSize(args[1], &name_length); + if (name == NULL) { + goto exit; + } + if (strlen(name) != (size_t)name_length) { + PyErr_SetString(PyExc_ValueError, "embedded null character"); + goto exit; + } +skip_optional_kwonly: + return_value = deserialize_impl(self, &data, name); + +exit: + /* Cleanup for data */ + if (data.obj) { + PyBuffer_Release(&data); + } + + return return_value; +} + +#endif /* defined(PY_SQLITE_HAVE_SERIALIZE) */ + PyDoc_STRVAR(pysqlite_connection_enter__doc__, "__enter__($self, /)\n" "--\n" @@ -832,4 +982,12 @@ getlimit(pysqlite_Connection *self, PyObject *arg) #ifndef PYSQLITE_CONNECTION_LOAD_EXTENSION_METHODDEF #define PYSQLITE_CONNECTION_LOAD_EXTENSION_METHODDEF #endif /* !defined(PYSQLITE_CONNECTION_LOAD_EXTENSION_METHODDEF) */ -/*[clinic end generated code: output=176c9095219b17c4 input=a9049054013a1b77]*/ + +#ifndef SERIALIZE_METHODDEF + #define SERIALIZE_METHODDEF +#endif /* !defined(SERIALIZE_METHODDEF) */ + +#ifndef DESERIALIZE_METHODDEF + #define DESERIALIZE_METHODDEF +#endif /* !defined(DESERIALIZE_METHODDEF) */ +/*[clinic end generated code: output=d965a68f9229a56c input=a9049054013a1b77]*/ diff --git a/Modules/_sqlite/connection.c b/Modules/_sqlite/connection.c index 37f6d0fa5a502..9d187cfa99d23 100644 --- a/Modules/_sqlite/connection.c +++ b/Modules/_sqlite/connection.c @@ -1818,6 +1818,125 @@ pysqlite_connection_create_collation_impl(pysqlite_Connection *self, Py_RETURN_NONE; } +#ifdef PY_SQLITE_HAVE_SERIALIZE +/*[clinic input] +_sqlite3.Connection.serialize as serialize + + * + name: str = "main" + Which database to serialize. + +Serialize a database into a byte string. + +For an ordinary on-disk database file, the serialization is just a copy of the +disk file. For an in-memory database or a "temp" database, the serialization is +the same sequence of bytes which would be written to disk if that database +were backed up to disk. +[clinic start generated code]*/ + +static PyObject * +serialize_impl(pysqlite_Connection *self, const char *name) +/*[clinic end generated code: output=97342b0e55239dd3 input=d2eb5194a65abe2b]*/ +{ + if (!pysqlite_check_thread(self) || !pysqlite_check_connection(self)) { + return NULL; + } + + /* If SQLite has a contiguous memory representation of the database, we can + * avoid memory allocations, so we try with the no-copy flag first. + */ + sqlite3_int64 size; + unsigned int flags = SQLITE_SERIALIZE_NOCOPY; + const char *data; + + Py_BEGIN_ALLOW_THREADS + data = (const char *)sqlite3_serialize(self->db, name, &size, flags); + if (data == NULL) { + flags &= ~SQLITE_SERIALIZE_NOCOPY; + data = (const char *)sqlite3_serialize(self->db, name, &size, flags); + } + Py_END_ALLOW_THREADS + + if (data == NULL) { + PyErr_Format(self->OperationalError, "unable to serialize '%s'", + name); + return NULL; + } + PyObject *res = PyBytes_FromStringAndSize(data, size); + if (!(flags & SQLITE_SERIALIZE_NOCOPY)) { + sqlite3_free((void *)data); + } + return res; +} + +/*[clinic input] +_sqlite3.Connection.deserialize as deserialize + + data: Py_buffer(accept={buffer, str}) + The serialized database content. + / + * + name: str = "main" + Which database to reopen with the deserialization. + +Load a serialized database. + +The deserialize interface causes the database connection to disconnect from the +target database, and then reopen it as an in-memory database based on the given +serialized data. + +The deserialize interface will fail with SQLITE_BUSY if the database is +currently in a read transaction or is involved in a backup operation. +[clinic start generated code]*/ + +static PyObject * +deserialize_impl(pysqlite_Connection *self, Py_buffer *data, + const char *name) +/*[clinic end generated code: output=e394c798b98bad89 input=1be4ca1faacf28f2]*/ +{ + if (!pysqlite_check_thread(self) || !pysqlite_check_connection(self)) { + return NULL; + } + + /* Transfer ownership of the buffer to SQLite: + * - Move buffer from Py to SQLite + * - Tell SQLite to free buffer memory + * - Tell SQLite that it is permitted to grow the resulting database + * + * Make sure we don't overflow sqlite3_deserialize(); it accepts a signed + * 64-bit int as its data size argument. + * + * We can safely use sqlite3_malloc64 here, since it was introduced before + * the serialize APIs. + */ + if (data->len > 9223372036854775807) { // (1 << 63) - 1 + PyErr_SetString(PyExc_OverflowError, "'data' is too large"); + return NULL; + } + + sqlite3_int64 size = (sqlite3_int64)data->len; + unsigned char *buf = sqlite3_malloc64(size); + if (buf == NULL) { + return PyErr_NoMemory(); + } + + const unsigned int flags = SQLITE_DESERIALIZE_FREEONCLOSE | + SQLITE_DESERIALIZE_RESIZEABLE; + int rc; + Py_BEGIN_ALLOW_THREADS + (void)memcpy(buf, data->buf, data->len); + rc = sqlite3_deserialize(self->db, name, buf, size, size, flags); + Py_END_ALLOW_THREADS + + if (rc != SQLITE_OK) { + (void)_pysqlite_seterror(self->state, self->db); + return NULL; + } + Py_RETURN_NONE; +} +#endif // PY_SQLITE_HAVE_SERIALIZE + + /*[clinic input] _sqlite3.Connection.__enter__ as pysqlite_connection_enter @@ -1971,6 +2090,8 @@ static PyMethodDef connection_methods[] = { PYSQLITE_CONNECTION_SET_TRACE_CALLBACK_METHODDEF SETLIMIT_METHODDEF GETLIMIT_METHODDEF + SERIALIZE_METHODDEF + DESERIALIZE_METHODDEF {NULL, NULL} }; diff --git a/PCbuild/_sqlite3.vcxproj b/PCbuild/_sqlite3.vcxproj index e268c473f4c98..9cff43f73e5be 100644 --- a/PCbuild/_sqlite3.vcxproj +++ b/PCbuild/_sqlite3.vcxproj @@ -94,6 +94,7 @@ $(sqlite3Dir);%(AdditionalIncludeDirectories) + PY_SQLITE_HAVE_SERIALIZE;%(PreprocessorDefinitions) diff --git a/configure b/configure index 44912b9c34df8..69b12309de578 100755 --- a/configure +++ b/configure @@ -12902,6 +12902,50 @@ if test "x$ac_cv_lib_sqlite3_sqlite3_load_extension" = xyes; then : else have_sqlite3_load_extension=no +fi + + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for sqlite3_serialize in -lsqlite3" >&5 +$as_echo_n "checking for sqlite3_serialize in -lsqlite3... " >&6; } +if ${ac_cv_lib_sqlite3_sqlite3_serialize+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-lsqlite3 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char sqlite3_serialize (); +int +main () +{ +return sqlite3_serialize (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_sqlite3_sqlite3_serialize=yes +else + ac_cv_lib_sqlite3_sqlite3_serialize=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_sqlite3_sqlite3_serialize" >&5 +$as_echo "$ac_cv_lib_sqlite3_sqlite3_serialize" >&6; } +if test "x$ac_cv_lib_sqlite3_sqlite3_serialize" = xyes; then : + + +$as_echo "#define PY_SQLITE_HAVE_SERIALIZE 1" >>confdefs.h + + fi diff --git a/configure.ac b/configure.ac index c02adf7bf3f14..5860595b752c8 100644 --- a/configure.ac +++ b/configure.ac @@ -3605,6 +3605,12 @@ dnl hence CPPFLAGS instead of CFLAGS. [have_sqlite3_load_extension=yes], [have_sqlite3_load_extension=no] ) + AC_CHECK_LIB([sqlite3], [sqlite3_serialize], [ + AC_DEFINE( + [PY_SQLITE_HAVE_SERIALIZE], [1], + [Define if SQLite was compiled with the serialize API] + ) + ]) ], [ have_supported_sqlite3=no ]) diff --git a/pyconfig.h.in b/pyconfig.h.in index be776f734163f..4ac054a28d8ef 100644 --- a/pyconfig.h.in +++ b/pyconfig.h.in @@ -1506,6 +1506,9 @@ /* Define to 1 to build the sqlite module with loadable extensions support. */ #undef PY_SQLITE_ENABLE_LOAD_EXTENSION +/* Define if SQLite was compiled with the serialize API */ +#undef PY_SQLITE_HAVE_SERIALIZE + /* Default cipher suites list for ssl module. 1: Python's preferred selection, 2: leave OpenSSL defaults untouched, 0: custom string */ #undef PY_SSL_DEFAULT_CIPHERS From webhook-mailer at python.org Tue Apr 5 10:21:12 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Tue, 05 Apr 2022 14:21:12 -0000 Subject: [Python-checkins] bpo-47088: Add typing.LiteralString (PEP 675) (GH-32064) Message-ID: https://github.com/python/cpython/commit/cfb849a326e52a4edc577112ebf60e1d9d0d7fdb commit: cfb849a326e52a4edc577112ebf60e1d9d0d7fdb branch: main author: Jelle Zijlstra committer: JelleZijlstra date: 2022-04-05T07:21:03-07:00 summary: bpo-47088: Add typing.LiteralString (PEP 675) (GH-32064) Co-authored-by: Nick Pope files: A Misc/NEWS.d/next/Library/2022-03-22-19-18-31.bpo-47088.JM1kNI.rst M Doc/library/typing.rst M Lib/test/test_typing.py M Lib/typing.py diff --git a/Doc/library/typing.rst b/Doc/library/typing.rst index 0a4e848c67736..fdd00a277b507 100644 --- a/Doc/library/typing.rst +++ b/Doc/library/typing.rst @@ -76,6 +76,8 @@ annotations. These include: *Introducing* :data:`TypeGuard` * :pep:`673`: Self type *Introducing* :data:`Self` +* :pep:`675`: Arbitrary Literal String Type + *Introducing* :data:`LiteralString` .. _type-aliases: @@ -585,6 +587,33 @@ These can be used as types in annotations and do not support ``[]``. avoiding type checker errors with classes that can duck type anywhere or are highly dynamic. +.. data:: LiteralString + + Special type that includes only literal strings. A string + literal is compatible with ``LiteralString``, as is another + ``LiteralString``, but an object typed as just ``str`` is not. + A string created by composing ``LiteralString``-typed objects + is also acceptable as a ``LiteralString``. + + Example:: + + def run_query(sql: LiteralString) -> ... + ... + + def caller(arbitrary_string: str, literal_string: LiteralString) -> None: + run_query("SELECT * FROM students") # ok + run_query(literal_string) # ok + run_query("SELECT * FROM " + literal_string) # ok + run_query(arbitrary_string) # type checker error + run_query( # type checker error + f"SELECT * FROM students WHERE name = {arbitrary_string}" + ) + + This is useful for sensitive APIs where arbitrary user-generated + strings could generate problems. For example, the two cases above + that generate type checker errors could be vulnerable to an SQL + injection attack. + .. data:: Never The `bottom type `_, diff --git a/Lib/test/test_typing.py b/Lib/test/test_typing.py index 041b6ad9ed6dd..e09f8aa3fb849 100644 --- a/Lib/test/test_typing.py +++ b/Lib/test/test_typing.py @@ -27,7 +27,7 @@ from typing import IO, TextIO, BinaryIO from typing import Pattern, Match from typing import Annotated, ForwardRef -from typing import Self +from typing import Self, LiteralString from typing import TypeAlias from typing import ParamSpec, Concatenate, ParamSpecArgs, ParamSpecKwargs from typing import TypeGuard @@ -265,6 +265,60 @@ def test_alias(self): self.assertEqual(get_args(alias_3), (Self,)) +class LiteralStringTests(BaseTestCase): + def test_equality(self): + self.assertEqual(LiteralString, LiteralString) + self.assertIs(LiteralString, LiteralString) + self.assertNotEqual(LiteralString, None) + + def test_basics(self): + class Foo: + def bar(self) -> LiteralString: ... + class FooStr: + def bar(self) -> 'LiteralString': ... + class FooStrTyping: + def bar(self) -> 'typing.LiteralString': ... + + for target in [Foo, FooStr, FooStrTyping]: + with self.subTest(target=target): + self.assertEqual(gth(target.bar), {'return': LiteralString}) + self.assertIs(get_origin(LiteralString), None) + + def test_repr(self): + self.assertEqual(repr(LiteralString), 'typing.LiteralString') + + def test_cannot_subscript(self): + with self.assertRaises(TypeError): + LiteralString[int] + + def test_cannot_subclass(self): + with self.assertRaises(TypeError): + class C(type(LiteralString)): + pass + with self.assertRaises(TypeError): + class C(LiteralString): + pass + + def test_cannot_init(self): + with self.assertRaises(TypeError): + LiteralString() + with self.assertRaises(TypeError): + type(LiteralString)() + + def test_no_isinstance(self): + with self.assertRaises(TypeError): + isinstance(1, LiteralString) + with self.assertRaises(TypeError): + issubclass(int, LiteralString) + + def test_alias(self): + alias_1 = Tuple[LiteralString, LiteralString] + alias_2 = List[LiteralString] + alias_3 = ClassVar[LiteralString] + self.assertEqual(get_args(alias_1), (LiteralString, LiteralString)) + self.assertEqual(get_args(alias_2), (LiteralString,)) + self.assertEqual(get_args(alias_3), (LiteralString,)) + class TypeVarTests(BaseTestCase): def test_basic_plain(self): T = TypeVar('T') diff --git a/Lib/typing.py b/Lib/typing.py index 4636798bd6956..26c6b8c278b73 100644 --- a/Lib/typing.py +++ b/Lib/typing.py @@ -126,6 +126,7 @@ def _idfunc(_, x): 'get_origin', 'get_type_hints', 'is_typeddict', + 'LiteralString', 'Never', 'NewType', 'no_type_check', @@ -180,7 +181,7 @@ def _type_check(arg, msg, is_argument=True, module=None, *, allow_special_forms= if (isinstance(arg, _GenericAlias) and arg.__origin__ in invalid_generic_forms): raise TypeError(f"{arg} is not valid as type argument") - if arg in (Any, NoReturn, Never, Self, TypeAlias): + if arg in (Any, LiteralString, NoReturn, Never, Self, TypeAlias): return arg if allow_special_forms and arg in (ClassVar, Final): return arg @@ -523,6 +524,34 @@ def returns_self(self) -> Self: raise TypeError(f"{self} is not subscriptable") + at _SpecialForm +def LiteralString(self, parameters): + """Represents an arbitrary literal string. + + Example:: + + from typing import LiteralString + + def run_query(sql: LiteralString) -> ... + ... + + def caller(arbitrary_string: str, literal_string: LiteralString) -> None: + run_query("SELECT * FROM students") # ok + run_query(literal_string) # ok + run_query("SELECT * FROM " + literal_string) # ok + run_query(arbitrary_string) # type checker error + run_query( # type checker error + f"SELECT * FROM students WHERE name = {arbitrary_string}" + ) + + Only string literals and other LiteralStrings are compatible + with LiteralString. This provides a tool to help prevent + security issues such as SQL injection. + + """ + raise TypeError(f"{self} is not subscriptable") + + @_SpecialForm def ClassVar(self, parameters): """Special type construct to mark class variables. diff --git a/Misc/NEWS.d/next/Library/2022-03-22-19-18-31.bpo-47088.JM1kNI.rst b/Misc/NEWS.d/next/Library/2022-03-22-19-18-31.bpo-47088.JM1kNI.rst new file mode 100644 index 0000000000000..10a814e018245 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-03-22-19-18-31.bpo-47088.JM1kNI.rst @@ -0,0 +1,2 @@ +Implement :data:`typing.LiteralString`, part of :pep:`675`. Patch by Jelle +Zijlstra. From webhook-mailer at python.org Tue Apr 5 11:15:16 2022 From: webhook-mailer at python.org (gpshead) Date: Tue, 05 Apr 2022 15:15:16 -0000 Subject: [Python-checkins] bpo-46607: Add DeprecationWarning for LegacyInterpolation, deprecated in docs since 3.2 (GH-30927) Message-ID: https://github.com/python/cpython/commit/75280944e5ca957eec7f814b9d0608fc84fc5811 commit: 75280944e5ca957eec7f814b9d0608fc84fc5811 branch: main author: Hugo van Kemenade committer: gpshead date: 2022-04-05T08:15:11-07:00 summary: bpo-46607: Add DeprecationWarning for LegacyInterpolation, deprecated in docs since 3.2 (GH-30927) files: A Misc/NEWS.d/next/Library/2022-01-26-18-30-34.bpo-46607.xnhT4a.rst M Doc/whatsnew/3.11.rst M Lib/configparser.py M Lib/test/test_configparser.py diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index c312645c31cd7..537fa49fd2f28 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -590,6 +590,12 @@ Deprecated (Contributed by Hugo van Kemenade in :issue:`45173`.) +* :class:`configparser.LegacyInterpolation` has been deprecated in the docstring + since Python 3.2. It now emits a :exc:`DeprecationWarning` and will be removed + in Python 3.13. Use :class:`configparser.BasicInterpolation` or + :class:`configparser.ExtendedInterpolation instead. + (Contributed by Hugo van Kemenade in :issue:`46607`.) + * The :func:`locale.getdefaultlocale` function is deprecated and will be removed in Python 3.13. Use :func:`locale.setlocale`, :func:`locale.getpreferredencoding(False) ` and diff --git a/Lib/configparser.py b/Lib/configparser.py index f5666f518f121..de9ee537ac188 100644 --- a/Lib/configparser.py +++ b/Lib/configparser.py @@ -525,6 +525,15 @@ class LegacyInterpolation(Interpolation): _KEYCRE = re.compile(r"%\(([^)]*)\)s|.") + def __init__(self, *args, **kwargs): + super().__init__(*args, **kwargs) + warnings.warn( + "LegacyInterpolation has been deprecated since Python 3.2 " + "and will be removed from the configparser module in Python 3.13. " + "Use BasicInterpolation or ExtendedInterpolation instead.", + DeprecationWarning, stacklevel=2 + ) + def before_get(self, parser, section, option, value, vars): rawval = value depth = MAX_INTERPOLATION_DEPTH diff --git a/Lib/test/test_configparser.py b/Lib/test/test_configparser.py index 569959c3ed919..efd98ffb67a54 100644 --- a/Lib/test/test_configparser.py +++ b/Lib/test/test_configparser.py @@ -1666,6 +1666,14 @@ def test_safeconfigparser_deprecation(self): for warning in w: self.assertTrue(warning.category is DeprecationWarning) + def test_legacyinterpolation_deprecation(self): + with warnings.catch_warnings(record=True) as w: + warnings.simplefilter("always", DeprecationWarning) + configparser.LegacyInterpolation() + self.assertGreaterEqual(len(w), 1) + for warning in w: + self.assertIs(warning.category, DeprecationWarning) + def test_sectionproxy_repr(self): parser = configparser.ConfigParser() parser.read_string(""" diff --git a/Misc/NEWS.d/next/Library/2022-01-26-18-30-34.bpo-46607.xnhT4a.rst b/Misc/NEWS.d/next/Library/2022-01-26-18-30-34.bpo-46607.xnhT4a.rst new file mode 100644 index 0000000000000..e0c7ed0531b57 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-01-26-18-30-34.bpo-46607.xnhT4a.rst @@ -0,0 +1,3 @@ +Add :exc:`DeprecationWarning` to :class:`LegacyInterpolation`, deprecated in +the docstring since Python 3.2. Will be removed in Python 3.13. Use +:class:`BasicInterpolation` or :class:`ExtendedInterpolation` instead. From webhook-mailer at python.org Tue Apr 5 12:18:12 2022 From: webhook-mailer at python.org (tiran) Date: Tue, 05 Apr 2022 16:18:12 -0000 Subject: [Python-checkins] bpo-47009: Fix assert on big endian (GH-32332) Message-ID: https://github.com/python/cpython/commit/9e88b572fb904b172f9e344069fb7118f1cee517 commit: 9e88b572fb904b172f9e344069fb7118f1cee517 branch: main author: Christian Heimes committer: tiran date: 2022-04-05T18:18:02+02:00 summary: bpo-47009: Fix assert on big endian (GH-32332) files: M Python/ceval.c diff --git a/Python/ceval.c b/Python/ceval.c index 9b7c42cbe4b11..487e09bc66417 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -5032,7 +5032,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int STAT_INC(PRECALL, hit); // PRECALL + CALL + POP_TOP JUMPBY(INLINE_CACHE_ENTRIES_PRECALL + 1 + INLINE_CACHE_ENTRIES_CALL + 1); - assert(next_instr[-1] == POP_TOP); + assert(_Py_OPCODE(next_instr[-1]) == POP_TOP); PyObject *arg = POP(); if (_PyList_AppendTakeRef((PyListObject *)list, arg) < 0) { goto error; From webhook-mailer at python.org Tue Apr 5 12:21:59 2022 From: webhook-mailer at python.org (miss-islington) Date: Tue, 05 Apr 2022 16:21:59 -0000 Subject: [Python-checkins] [3.10] Backport bpo-47212 (GH-32302) to Python 3.10 (GH-32334) Message-ID: https://github.com/python/cpython/commit/94609e3192f15636c760a48c4c1c2c236fac3217 commit: 94609e3192f15636c760a48c4c1c2c236fac3217 branch: 3.10 author: Matthieu Dartiailh committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-05T09:21:49-07:00 summary: [3.10] Backport bpo-47212 (GH-32302) to Python 3.10 (GH-32334) (cherry picked from commit aa0f056a00c4bcaef83d729e042359ddae903382) # Conflicts: # Grammar/python.gram # Parser/action_helpers.c Automerge-Triggered-By: GH:pablogsal files: A Misc/NEWS.d/next/Core and Builtins/2022-04-05-11-29-21.bpo-47212.leF4pz.rst M Grammar/python.gram M Lib/test/test_exceptions.py M Lib/test/test_syntax.py M Parser/parser.c M Parser/pegen.c M Parser/pegen.h diff --git a/Grammar/python.gram b/Grammar/python.gram index 99e01355a582c..84b9c9b82720b 100644 --- a/Grammar/python.gram +++ b/Grammar/python.gram @@ -830,12 +830,12 @@ t_lookahead: '(' | '[' | '.' invalid_arguments: | a=args ',' '*' { RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "iterable argument unpacking follows keyword argument unpacking") } | a=expression b=for_if_clauses ',' [args | expression for_if_clauses] { - RAISE_SYNTAX_ERROR_KNOWN_RANGE(a, PyPegen_last_item(b, comprehension_ty)->target, "Generator expression must be parenthesized") } + RAISE_SYNTAX_ERROR_KNOWN_RANGE(a, _PyPegen_get_last_comprehension_item(PyPegen_last_item(b, comprehension_ty)), "Generator expression must be parenthesized") } | a=NAME b='=' expression for_if_clauses { RAISE_SYNTAX_ERROR_KNOWN_RANGE(a, b, "invalid syntax. Maybe you meant '==' or ':=' instead of '='?")} | a=args b=for_if_clauses { _PyPegen_nonparen_genexp_in_call(p, a, b) } | args ',' a=expression b=for_if_clauses { - RAISE_SYNTAX_ERROR_KNOWN_RANGE(a, asdl_seq_GET(b, b->size-1)->target, "Generator expression must be parenthesized") } + RAISE_SYNTAX_ERROR_KNOWN_RANGE(a, _PyPegen_get_last_comprehension_item(PyPegen_last_item(b, comprehension_ty)), "Generator expression must be parenthesized") } | a=args ',' args { _PyPegen_arguments_parsing_error(p, a) } invalid_kwarg: | a[Token*]=('True'|'False'|'None') b='=' { @@ -975,7 +975,7 @@ invalid_finally_stmt: invalid_except_stmt_indent: | a='except' expression ['as' NAME ] ':' NEWLINE !INDENT { RAISE_INDENTATION_ERROR("expected an indented block after 'except' statement on line %d", a->lineno) } - | a='except' ':' NEWLINE !INDENT { RAISE_SYNTAX_ERROR("expected an indented block after except statement on line %d", a->lineno) } + | a='except' ':' NEWLINE !INDENT { RAISE_INDENTATION_ERROR("expected an indented block after 'except' statement on line %d", a->lineno) } invalid_match_stmt: | "match" subject_expr !':' { CHECK_VERSION(void*, 10, "Pattern matching is", RAISE_SYNTAX_ERROR("expected ':'") ) } | a="match" subject=subject_expr ':' NEWLINE !INDENT { diff --git a/Lib/test/test_exceptions.py b/Lib/test/test_exceptions.py index 2bdd7214b0505..d9ea8a4872d71 100644 --- a/Lib/test/test_exceptions.py +++ b/Lib/test/test_exceptions.py @@ -198,12 +198,17 @@ def ckmsg(src, msg, exception=SyntaxError): s = '''if True:\n print()\n\texec "mixed tabs and spaces"''' ckmsg(s, "inconsistent use of tabs and spaces in indentation", TabError) - def check(self, src, lineno, offset, encoding='utf-8'): + def check(self, src, lineno, offset, end_lineno=None, end_offset=None, encoding='utf-8'): with self.subTest(source=src, lineno=lineno, offset=offset): with self.assertRaises(SyntaxError) as cm: compile(src, '', 'exec') self.assertEqual(cm.exception.lineno, lineno) self.assertEqual(cm.exception.offset, offset) + if end_lineno is not None: + self.assertEqual(cm.exception.end_lineno, end_lineno) + if end_offset is not None: + self.assertEqual(cm.exception.end_offset, end_offset) + if cm.exception.text is not None: if not isinstance(src, str): src = src.decode(encoding, 'replace') @@ -235,6 +240,10 @@ def testSyntaxErrorOffset(self): check('match ...:\n case {**rest, "key": value}:\n ...', 2, 19) check("[a b c d e f]", 1, 2) check("for x yfff:", 1, 7) + check("f(a for a in b, c)", 1, 3, 1, 15) + check("f(a for a in b if a, c)", 1, 3, 1, 20) + check("f(a, b for b in c)", 1, 6, 1, 18) + check("f(a, b for b in c, d)", 1, 6, 1, 18) # Errors thrown by compile.c check('class foo:return 1', 1, 11) diff --git a/Lib/test/test_syntax.py b/Lib/test/test_syntax.py index 09c0d5682a7e4..006374e7cd912 100644 --- a/Lib/test/test_syntax.py +++ b/Lib/test/test_syntax.py @@ -1012,6 +1012,13 @@ Traceback (most recent call last): IndentationError: expected an indented block after 'try' statement on line 1 + >>> try: + ... something() + ... except: + ... pass + Traceback (most recent call last): + IndentationError: expected an indented block after 'except' statement on line 3 + >>> try: ... something() ... except A: diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-04-05-11-29-21.bpo-47212.leF4pz.rst b/Misc/NEWS.d/next/Core and Builtins/2022-04-05-11-29-21.bpo-47212.leF4pz.rst new file mode 100644 index 0000000000000..8f1f6b6cfbbb8 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2022-04-05-11-29-21.bpo-47212.leF4pz.rst @@ -0,0 +1,3 @@ +Raise :exc:`IndentationError` instead of :exc:`SyntaxError` for a bare +``except`` with no following indent. Improve :exc:`SyntaxError` locations for +an un-parenthesized generator used as arguments. Patch by Matthieu Dartiailh. diff --git a/Parser/parser.c b/Parser/parser.c index 3f73003b78973..862950018a312 100644 --- a/Parser/parser.c +++ b/Parser/parser.c @@ -18419,7 +18419,7 @@ invalid_arguments_rule(Parser *p) ) { D(fprintf(stderr, "%*c+ invalid_arguments[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "expression for_if_clauses ',' [args | expression for_if_clauses]")); - _res = RAISE_SYNTAX_ERROR_KNOWN_RANGE ( a , PyPegen_last_item ( b , comprehension_ty ) -> target , "Generator expression must be parenthesized" ); + _res = RAISE_SYNTAX_ERROR_KNOWN_RANGE ( a , _PyPegen_get_last_comprehension_item ( PyPegen_last_item ( b , comprehension_ty ) ) , "Generator expression must be parenthesized" ); if (_res == NULL && PyErr_Occurred()) { p->error_indicator = 1; p->level--; @@ -18512,7 +18512,7 @@ invalid_arguments_rule(Parser *p) ) { D(fprintf(stderr, "%*c+ invalid_arguments[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "args ',' expression for_if_clauses")); - _res = RAISE_SYNTAX_ERROR_KNOWN_RANGE ( a , asdl_seq_GET ( b , b -> size - 1 ) -> target , "Generator expression must be parenthesized" ); + _res = RAISE_SYNTAX_ERROR_KNOWN_RANGE ( a , _PyPegen_get_last_comprehension_item ( PyPegen_last_item ( b , comprehension_ty ) ) , "Generator expression must be parenthesized" ); if (_res == NULL && PyErr_Occurred()) { p->error_indicator = 1; p->level--; @@ -20804,7 +20804,7 @@ invalid_except_stmt_indent_rule(Parser *p) ) { D(fprintf(stderr, "%*c+ invalid_except_stmt_indent[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "'except' ':' NEWLINE !INDENT")); - _res = RAISE_SYNTAX_ERROR ( "expected an indented block after except statement on line %d" , a -> lineno ); + _res = RAISE_INDENTATION_ERROR ( "expected an indented block after 'except' statement on line %d" , a -> lineno ); if (_res == NULL && PyErr_Occurred()) { p->error_indicator = 1; p->level--; diff --git a/Parser/pegen.c b/Parser/pegen.c index 90a6ab98cf42e..c04824315c7a3 100644 --- a/Parser/pegen.c +++ b/Parser/pegen.c @@ -2568,7 +2568,7 @@ void *_PyPegen_arguments_parsing_error(Parser *p, expr_ty e) { } -static inline expr_ty +expr_ty _PyPegen_get_last_comprehension_item(comprehension_ty comprehension) { if (comprehension->ifs == NULL || asdl_seq_LEN(comprehension->ifs) == 0) { return comprehension->iter; diff --git a/Parser/pegen.h b/Parser/pegen.h index 0e5a057c1f196..118fbc7b3b78a 100644 --- a/Parser/pegen.h +++ b/Parser/pegen.h @@ -327,6 +327,7 @@ _RAISE_SYNTAX_ERROR_INVALID_TARGET(Parser *p, TARGETS_TYPE type, void *e) } void *_PyPegen_arguments_parsing_error(Parser *, expr_ty); +expr_ty _PyPegen_get_last_comprehension_item(comprehension_ty comprehension); void *_PyPegen_nonparen_genexp_in_call(Parser *p, expr_ty args, asdl_comprehension_seq *comprehensions); From webhook-mailer at python.org Tue Apr 5 14:41:42 2022 From: webhook-mailer at python.org (ethanfurman) Date: Tue, 05 Apr 2022 18:41:42 -0000 Subject: [Python-checkins] bpo-4833: Add ZipFile.mkdir (GH-32160) Message-ID: https://github.com/python/cpython/commit/050a8f94c678a05d506fe192c863c4a572178c42 commit: 050a8f94c678a05d506fe192c863c4a572178c42 branch: main author: Sam Ezeh committer: ethanfurman date: 2022-04-05T11:41:38-07:00 summary: bpo-4833: Add ZipFile.mkdir (GH-32160) files: A Misc/NEWS.d/next/Library/2022-03-28-20-16-37.bpo-4833.2vSUE5.rst M Doc/library/zipfile.rst M Lib/test/test_zipfile.py M Lib/zipfile.py diff --git a/Doc/library/zipfile.rst b/Doc/library/zipfile.rst index bfcc883de6927..d6a1fce49c545 100644 --- a/Doc/library/zipfile.rst +++ b/Doc/library/zipfile.rst @@ -478,6 +478,17 @@ ZipFile Objects a closed ZipFile will raise a :exc:`ValueError`. Previously, a :exc:`RuntimeError` was raised. +.. method:: ZipFile.mkdir(zinfo_or_directory, mode=511) + + Create a directory inside the archive. If *zinfo_or_directory* is a string, + a directory is created inside the archive with the mode that is specified in + the *mode* argument. If, however, *zinfo_or_directory* is + a :class:`ZipInfo` instance then the *mode* argument is ignored. + + The archive must be opened with mode ``'w'``, ``'x'`` or ``'a'``. + + .. versionadded:: 3.11 + The following data attributes are also available: diff --git a/Lib/test/test_zipfile.py b/Lib/test/test_zipfile.py index 26c40457e62a0..17111b3a40fef 100644 --- a/Lib/test/test_zipfile.py +++ b/Lib/test/test_zipfile.py @@ -2637,6 +2637,59 @@ def test_writestr_dir(self): self.assertTrue(os.path.isdir(os.path.join(target, "x"))) self.assertEqual(os.listdir(target), ["x"]) + def test_mkdir(self): + with zipfile.ZipFile(TESTFN, "w") as zf: + zf.mkdir("directory") + zinfo = zf.filelist[0] + self.assertEqual(zinfo.filename, "directory/") + self.assertEqual(zinfo.external_attr, (0o40777 << 16) | 0x10) + + zf.mkdir("directory2/") + zinfo = zf.filelist[1] + self.assertEqual(zinfo.filename, "directory2/") + self.assertEqual(zinfo.external_attr, (0o40777 << 16) | 0x10) + + zf.mkdir("directory3", mode=0o777) + zinfo = zf.filelist[2] + self.assertEqual(zinfo.filename, "directory3/") + self.assertEqual(zinfo.external_attr, (0o40777 << 16) | 0x10) + + old_zinfo = zipfile.ZipInfo("directory4/") + old_zinfo.external_attr = (0o40777 << 16) | 0x10 + old_zinfo.CRC = 0 + old_zinfo.file_size = 0 + old_zinfo.compress_size = 0 + zf.mkdir(old_zinfo) + new_zinfo = zf.filelist[3] + self.assertEqual(old_zinfo.filename, "directory4/") + self.assertEqual(old_zinfo.external_attr, new_zinfo.external_attr) + + target = os.path.join(TESTFN2, "target") + os.mkdir(target) + zf.extractall(target) + self.assertEqual(set(os.listdir(target)), {"directory", "directory2", "directory3", "directory4"}) + + def test_create_directory_with_write(self): + with zipfile.ZipFile(TESTFN, "w") as zf: + zf.writestr(zipfile.ZipInfo('directory/'), '') + + zinfo = zf.filelist[0] + self.assertEqual(zinfo.filename, "directory/") + + directory = os.path.join(TESTFN2, "directory2") + os.mkdir(directory) + mode = os.stat(directory).st_mode + zf.write(directory, arcname="directory2/") + zinfo = zf.filelist[1] + self.assertEqual(zinfo.filename, "directory2/") + self.assertEqual(zinfo.external_attr, (mode << 16) | 0x10) + + target = os.path.join(TESTFN2, "target") + os.mkdir(target) + zf.extractall(target) + + self.assertEqual(set(os.listdir(target)), {"directory", "directory2"}) + def tearDown(self): rmtree(TESTFN2) if os.path.exists(TESTFN): diff --git a/Lib/zipfile.py b/Lib/zipfile.py index 721834aff13a7..dc02011084329 100644 --- a/Lib/zipfile.py +++ b/Lib/zipfile.py @@ -1772,6 +1772,7 @@ def write(self, filename, arcname=None, if zinfo.is_dir(): zinfo.compress_size = 0 zinfo.CRC = 0 + self.mkdir(zinfo) else: if compress_type is not None: zinfo.compress_type = compress_type @@ -1783,23 +1784,6 @@ def write(self, filename, arcname=None, else: zinfo._compresslevel = self.compresslevel - if zinfo.is_dir(): - with self._lock: - if self._seekable: - self.fp.seek(self.start_dir) - zinfo.header_offset = self.fp.tell() # Start of header bytes - if zinfo.compress_type == ZIP_LZMA: - # Compressed data includes an end-of-stream (EOS) marker - zinfo.flag_bits |= _MASK_COMPRESS_OPTION_1 - - self._writecheck(zinfo) - self._didModify = True - - self.filelist.append(zinfo) - self.NameToInfo[zinfo.filename] = zinfo - self.fp.write(zinfo.FileHeader(False)) - self.start_dir = self.fp.tell() - else: with open(filename, "rb") as src, self.open(zinfo, 'w') as dest: shutil.copyfileobj(src, dest, 1024*8) @@ -1844,6 +1828,41 @@ def writestr(self, zinfo_or_arcname, data, with self.open(zinfo, mode='w') as dest: dest.write(data) + def mkdir(self, zinfo_or_directory_name, mode=511): + """Creates a directory inside the zip archive.""" + if isinstance(zinfo_or_directory_name, ZipInfo): + zinfo = zinfo_or_directory_name + if not zinfo.is_dir(): + raise ValueError("The given ZipInfo does not describe a directory") + elif isinstance(zinfo_or_directory_name, str): + directory_name = zinfo_or_directory_name + if not directory_name.endswith("/"): + directory_name += "/" + zinfo = ZipInfo(directory_name) + zinfo.compress_size = 0 + zinfo.CRC = 0 + zinfo.external_attr = ((0o40000 | mode) & 0xFFFF) << 16 + zinfo.file_size = 0 + zinfo.external_attr |= 0x10 + else: + raise TypeError("Expected type str or ZipInfo") + + with self._lock: + if self._seekable: + self.fp.seek(self.start_dir) + zinfo.header_offset = self.fp.tell() # Start of header bytes + if zinfo.compress_type == ZIP_LZMA: + # Compressed data includes an end-of-stream (EOS) marker + zinfo.flag_bits |= _MASK_COMPRESS_OPTION_1 + + self._writecheck(zinfo) + self._didModify = True + + self.filelist.append(zinfo) + self.NameToInfo[zinfo.filename] = zinfo + self.fp.write(zinfo.FileHeader(False)) + self.start_dir = self.fp.tell() + def __del__(self): """Call the "close()" method in case the user forgot.""" self.close() diff --git a/Misc/NEWS.d/next/Library/2022-03-28-20-16-37.bpo-4833.2vSUE5.rst b/Misc/NEWS.d/next/Library/2022-03-28-20-16-37.bpo-4833.2vSUE5.rst new file mode 100644 index 0000000000000..7696091221cb5 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-03-28-20-16-37.bpo-4833.2vSUE5.rst @@ -0,0 +1 @@ +Add :meth:`ZipFile.mkdir` From webhook-mailer at python.org Tue Apr 5 15:03:41 2022 From: webhook-mailer at python.org (zooba) Date: Tue, 05 Apr 2022 19:03:41 -0000 Subject: [Python-checkins] bpo-47230: Silence compiler warnings on Windows from zlib 1.2.12 (GH-32337) Message-ID: https://github.com/python/cpython/commit/944f09adfcc59f54432ac2947cf95f3465d90e1e commit: 944f09adfcc59f54432ac2947cf95f3465d90e1e branch: main author: Jeremy Kloth committer: zooba date: 2022-04-05T20:03:17+01:00 summary: bpo-47230: Silence compiler warnings on Windows from zlib 1.2.12 (GH-32337) files: M PCbuild/pythoncore.vcxproj diff --git a/PCbuild/pythoncore.vcxproj b/PCbuild/pythoncore.vcxproj index 9894e37cb78e1..4fc6aa80f6fae 100644 --- a/PCbuild/pythoncore.vcxproj +++ b/PCbuild/pythoncore.vcxproj @@ -534,13 +534,15 @@ - + - + + 4244 + From webhook-mailer at python.org Tue Apr 5 15:06:00 2022 From: webhook-mailer at python.org (brettcannon) Date: Tue, 05 Apr 2022 19:06:00 -0000 Subject: [Python-checkins] bpo-47061: deprecate the `aifc` module (GH-32134) Message-ID: https://github.com/python/cpython/commit/c1d93b6411f975d67e43942f1a2745a22983c18c commit: c1d93b6411f975d67e43942f1a2745a22983c18c branch: main author: Brett Cannon committer: brettcannon date: 2022-04-05T12:05:48-07:00 summary: bpo-47061: deprecate the `aifc` module (GH-32134) Co-authored-by: Christian Heimes files: A Misc/NEWS.d/next/Library/2022-03-26-13-14-43.bpo-47061.QLxbC6.rst M Doc/whatsnew/3.11.rst M Lib/aifc.py M Lib/sndhdr.py M Lib/test/support/warnings_helper.py M Lib/test/test___all__.py M Lib/test/test_aifc.py M Lib/test/test_pyclbr.py M Lib/test/test_warnings/__init__.py M Lib/warnings.py diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index 537fa49fd2f28..a2c57eb909ed1 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -608,6 +608,14 @@ Deprecated (:pep:`594`). (Contributed by Hugo van Kemenade in :issue:`47022`.) +* :pep:`594` led to the deprecations of the following modules which are + slated for removal in Python 3.13: + + * :mod:`aifc` + + (Contributed by Brett Cannon in :issue:`47061`.) + + Removed ======= diff --git a/Lib/aifc.py b/Lib/aifc.py index d50f258e1acf8..b5eab9215d6ef 100644 --- a/Lib/aifc.py +++ b/Lib/aifc.py @@ -140,6 +140,10 @@ __all__ = ["Error", "open"] + +warnings._deprecated(__name__, remove=(3, 13)) + + class Error(Exception): pass diff --git a/Lib/sndhdr.py b/Lib/sndhdr.py index 96595c6974468..a63b6fd20220c 100644 --- a/Lib/sndhdr.py +++ b/Lib/sndhdr.py @@ -33,6 +33,7 @@ __all__ = ['what', 'whathdr'] from collections import namedtuple +import warnings SndHeaders = namedtuple('SndHeaders', 'filetype framerate nchannels nframes sampwidth') @@ -73,7 +74,9 @@ def whathdr(filename): tests = [] def test_aifc(h, f): - import aifc + with warnings.catch_warnings(): + warnings.simplefilter('ignore', category=DeprecationWarning) + import aifc if not h.startswith(b'FORM'): return None if h[8:12] == b'AIFC': diff --git a/Lib/test/support/warnings_helper.py b/Lib/test/support/warnings_helper.py index a024fbe5beaa0..28e96f88b2444 100644 --- a/Lib/test/support/warnings_helper.py +++ b/Lib/test/support/warnings_helper.py @@ -1,10 +1,18 @@ import contextlib import functools +import importlib import re import sys import warnings +def import_deprecated(name): + """Import *name* while suppressing DeprecationWarning.""" + with warnings.catch_warnings(): + warnings.simplefilter('ignore', category=DeprecationWarning) + return importlib.import_module(name) + + def check_syntax_warning(testcase, statement, errtext='', *, lineno=1, offset=None): # Test also that a warning is emitted only once. diff --git a/Lib/test/test___all__.py b/Lib/test/test___all__.py index a1a3d899e4e03..1ec83cb0b1440 100644 --- a/Lib/test/test___all__.py +++ b/Lib/test/test___all__.py @@ -41,6 +41,7 @@ def tearDown(self): def check_all(self, modname): names = {} with warnings_helper.check_warnings( + (f".*{modname}", DeprecationWarning), (".* (module|package)", DeprecationWarning), (".* (module|package)", PendingDeprecationWarning), ("", ResourceWarning), diff --git a/Lib/test/test_aifc.py b/Lib/test/test_aifc.py index fb6da4136f4c5..ad8a7ee053cca 100644 --- a/Lib/test/test_aifc.py +++ b/Lib/test/test_aifc.py @@ -1,6 +1,6 @@ from test.support import findfile from test.support.os_helper import TESTFN, unlink -from test.support.warnings_helper import check_no_resource_warning +from test.support.warnings_helper import check_no_resource_warning, import_deprecated import unittest from unittest import mock from test import audiotests @@ -8,7 +8,10 @@ import io import sys import struct -import aifc + + +aifc = import_deprecated("aifc") + class AifcTest(audiotests.AudioWriteTests, audiotests.AudioTestsWithSourceFile): diff --git a/Lib/test/test_pyclbr.py b/Lib/test/test_pyclbr.py index 157c522b63bd0..329acf0c64295 100644 --- a/Lib/test/test_pyclbr.py +++ b/Lib/test/test_pyclbr.py @@ -216,12 +216,12 @@ def compare(parent1, children1, parent2, children2): def test_others(self): cm = self.checkModule - # These were once about the 10 longest modules + # These were once some of the longest modules. + cm('aifc', ignore=('_aifc_params',)) # set with = in module cm('random', ignore=('Random',)) # from _random import Random as CoreGenerator cm('cgi', ignore=('log',)) # set with = in module cm('pickle', ignore=('partial', 'PickleBuffer')) - cm('aifc', ignore=('_aifc_params',)) # set with = in module - cm('re._parser', ignore=('dump', 'groups', 'pos')) # from ._constants import *; property + cm('sre_parse', ignore=('dump', 'groups', 'pos')) # from sre_constants import *; property cm( 'pdb', # pyclbr does not handle elegantly `typing` or properties diff --git a/Lib/test/test_warnings/__init__.py b/Lib/test/test_warnings/__init__.py index 4b1b4e193cb16..f7f931130714c 100644 --- a/Lib/test/test_warnings/__init__.py +++ b/Lib/test/test_warnings/__init__.py @@ -1219,7 +1219,47 @@ class PyEnvironmentVariableTests(EnvironmentVariableTests, unittest.TestCase): module = py_warnings +class _DeprecatedTest(BaseTest, unittest.TestCase): + + """Test _deprecated().""" + + module = original_warnings + + def test_warning(self): + version = (3, 11, 0, "final", 0) + test = [(4, 12), (4, 11), (4, 0), (3, 12)] + for remove in test: + msg = rf".*test_warnings.*{remove[0]}\.{remove[1]}" + filter = msg, DeprecationWarning + with self.subTest(remove=remove): + with warnings_helper.check_warnings(filter, quiet=False): + self.module._deprecated("test_warnings", remove=remove, + _version=version) + + version = (3, 11, 0, "alpha", 0) + msg = r".*test_warnings.*3\.11" + with warnings_helper.check_warnings((msg, DeprecationWarning), quiet=False): + self.module._deprecated("test_warnings", remove=(3, 11), + _version=version) + + def test_RuntimeError(self): + version = (3, 11, 0, "final", 0) + test = [(2, 0), (2, 12), (3, 10)] + for remove in test: + with self.subTest(remove=remove): + with self.assertRaises(RuntimeError): + self.module._deprecated("test_warnings", remove=remove, + _version=version) + for level in ["beta", "candidate", "final"]: + version = (3, 11, 0, level, 0) + with self.subTest(releaselevel=level): + with self.assertRaises(RuntimeError): + self.module._deprecated("test_warnings", remove=(3, 11), + _version=version) + + class BootstrapTest(unittest.TestCase): + def test_issue_8766(self): # "import encodings" emits a warning whereas the warnings is not loaded # or not completely loaded (warnings imports indirectly encodings by diff --git a/Lib/warnings.py b/Lib/warnings.py index 691ccddfa450a..887ca6ecd1a72 100644 --- a/Lib/warnings.py +++ b/Lib/warnings.py @@ -483,6 +483,27 @@ def __exit__(self, *exc_info): self._module._showwarnmsg_impl = self._showwarnmsg_impl +_DEPRECATED_MSG = "{name!r} is deprecated and slated for removal in Python {remove}" + +def _deprecated(name, message=_DEPRECATED_MSG, *, remove, _version=sys.version_info): + """Warn that *name* is deprecated or should be removed. + + RuntimeError is raised if *remove* specifies a major/minor tuple older than + the current Python version or the same version but past the alpha. + + The *message* argument is formatted with *name* and *remove* as a Python + version (e.g. "3.11"). + + """ + remove_formatted = f"{remove[0]}.{remove[1]}" + if (_version[:2] > remove) or (_version[:2] == remove and _version[3] != "alpha"): + msg = f"{name!r} was slated for removal after Python {remove_formatted} alpha" + raise RuntimeError(msg) + else: + msg = message.format(name=name, remove=remove_formatted) + warn(msg, DeprecationWarning, stacklevel=3) + + # Private utility function called by _PyErr_WarnUnawaitedCoroutine def _warn_unawaited_coroutine(coro): msg_lines = [ diff --git a/Misc/NEWS.d/next/Library/2022-03-26-13-14-43.bpo-47061.QLxbC6.rst b/Misc/NEWS.d/next/Library/2022-03-26-13-14-43.bpo-47061.QLxbC6.rst new file mode 100644 index 0000000000000..17180861a88cd --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-03-26-13-14-43.bpo-47061.QLxbC6.rst @@ -0,0 +1 @@ +Deprecate the aifc module. From webhook-mailer at python.org Tue Apr 5 19:36:32 2022 From: webhook-mailer at python.org (zooba) Date: Tue, 05 Apr 2022 23:36:32 -0000 Subject: [Python-checkins] bpo-47230: Silence compiler warnings on Windows from zlib 1.2.12 (GH-32337) Message-ID: https://github.com/python/cpython/commit/8bce3cb16df5b8ac06ac6b2cae177dd221780b2f commit: 8bce3cb16df5b8ac06ac6b2cae177dd221780b2f branch: 3.10 author: Jeremy Kloth committer: zooba date: 2022-04-06T00:36:18+01:00 summary: bpo-47230: Silence compiler warnings on Windows from zlib 1.2.12 (GH-32337) (cherry picked from commit 944f09adfcc59f54432ac2947cf95f3465d90e1e) Co-authored-by: Jeremy Kloth files: M PCbuild/pythoncore.vcxproj diff --git a/PCbuild/pythoncore.vcxproj b/PCbuild/pythoncore.vcxproj index c39ba3e1a9f41..3ba63587642d1 100644 --- a/PCbuild/pythoncore.vcxproj +++ b/PCbuild/pythoncore.vcxproj @@ -499,7 +499,9 @@ - + + 4244 + From webhook-mailer at python.org Tue Apr 5 19:36:57 2022 From: webhook-mailer at python.org (pablogsal) Date: Tue, 05 Apr 2022 23:36:57 -0000 Subject: [Python-checkins] Python 3.11.0a7 Message-ID: https://github.com/python/cpython/commit/2e49bd06c5ffab7d154042da4b5bcd5ae6e4220b commit: 2e49bd06c5ffab7d154042da4b5bcd5ae6e4220b branch: main author: Pablo Galindo committer: pablogsal date: 2022-04-05T20:54:03+01:00 summary: Python 3.11.0a7 files: A Misc/NEWS.d/3.11.0a7.rst D Misc/NEWS.d/next/Build/2022-03-04-21-24-02.bpo-45774.9AhC0r.rst D Misc/NEWS.d/next/Build/2022-03-10-09-37-05.bpo-46917.fry4aK.rst D Misc/NEWS.d/next/Build/2022-03-10-14-30-39.bpo-46973._LEvnc.rst D Misc/NEWS.d/next/Build/2022-03-12-18-09-31.bpo-46996.SygzVz.rst D Misc/NEWS.d/next/Build/2022-03-15-09-28-55.bpo-47024.t7-dcu.rst D Misc/NEWS.d/next/Build/2022-03-16-00-37-40.bpo-47032.tsS9KE.rst D Misc/NEWS.d/next/Build/2022-03-24-12-12-35.bpo-40280.eAQWrM.rst D Misc/NEWS.d/next/Build/2022-04-01-16-12-53.bpo-46023.1Z1OcC.rst D Misc/NEWS.d/next/Build/2022-04-02-17-52-38.bpo-40280.U8Dd0H.rst D Misc/NEWS.d/next/C API/2022-03-03-11-12-33.bpo-46906.-olyBI.rst D Misc/NEWS.d/next/C API/2022-03-12-18-37-06.bpo-46987.LWcwyN.rst D Misc/NEWS.d/next/C API/2022-03-21-00-41-29.bpo-46850.rOt771.rst D Misc/NEWS.d/next/C API/2022-03-21-01-30-14.bpo-46850.Tfxde5.rst D Misc/NEWS.d/next/C API/2022-03-21-02-26-27.bpo-46850.hU3c-O.rst D Misc/NEWS.d/next/C API/2022-03-22-16-48-02.bpo-46850.7M5dO7.rst D Misc/NEWS.d/next/C API/2022-03-22-16-59-34.bpo-46850.lmEKLy.rst D Misc/NEWS.d/next/C API/2022-03-25-13-40-46.bpo-40421.wJREl2.rst D Misc/NEWS.d/next/Core and Builtins/2022-01-20-16-48-09.bpo-43224.WDihrT.rst D Misc/NEWS.d/next/Core and Builtins/2022-02-01-10-05-27.bpo-43721.-1XAIo.rst D Misc/NEWS.d/next/Core and Builtins/2022-02-25-22-42-30.bpo-46838.RB6kEy.rst D Misc/NEWS.d/next/Core and Builtins/2022-03-03-09-08-17.bpo-46881.ckD4tT.rst D Misc/NEWS.d/next/Core and Builtins/2022-03-06-10-37-36.bpo-46841.O12Pba.rst D Misc/NEWS.d/next/Core and Builtins/2022-03-07-15-54-39.bpo-46841.7wG92r.rst D Misc/NEWS.d/next/Core and Builtins/2022-03-08-10-50-42.bpo-46944.cnaIK3.rst D Misc/NEWS.d/next/Core and Builtins/2022-03-11-09-39-01.bpo-39829.mlW3Su.rst D Misc/NEWS.d/next/Core and Builtins/2022-03-12-09-44-31.bpo-46993.-13hGo.rst D Misc/NEWS.d/next/Core and Builtins/2022-03-12-21-07-21.bpo-46829.cpGoPV.rst D Misc/NEWS.d/next/Core and Builtins/2022-03-13-21-04-20.bpo-47005.OHBfCc.rst D Misc/NEWS.d/next/Core and Builtins/2022-03-14-09-45-10.bpo-47009.ZI05b5.rst D Misc/NEWS.d/next/Core and Builtins/2022-03-14-11-15-11.bpo-47012.5L6NoE.rst D Misc/NEWS.d/next/Core and Builtins/2022-03-16-11-05-35.bpo-46841.yUoIHg.rst D Misc/NEWS.d/next/Core and Builtins/2022-03-16-12-19-25.bpo-46329.9oS0HT.rst D Misc/NEWS.d/next/Core and Builtins/2022-03-17-14-22-23.bpo-46968.4gz4NA.rst D Misc/NEWS.d/next/Core and Builtins/2022-03-17-16-25-57.bpo-47045.xQgHul.rst D Misc/NEWS.d/next/Core and Builtins/2022-03-17-22-47-29.bpo-47053.QAXk8Q.rst D Misc/NEWS.d/next/Core and Builtins/2022-03-19-21-50-59.bpo-47070.wPcsQh.rst D Misc/NEWS.d/next/Core and Builtins/2022-03-22-15-12-28.bpo-42197.SwrrFO.rst D Misc/NEWS.d/next/Core and Builtins/2022-03-25-21-51-10.bpo-47120.9YJ-Xw.rst D Misc/NEWS.d/next/Core and Builtins/2022-03-26-12-21-53.bpo-47127.Mh86RB.rst D Misc/NEWS.d/next/Core and Builtins/2022-03-26-15-45-57.bpo-47117.60W6GQ.rst D Misc/NEWS.d/next/Core and Builtins/2022-03-26-16-35-57.bpo-47129.hDg2Vt.rst D Misc/NEWS.d/next/Core and Builtins/2022-03-30-02-36-25.bpo-46775.e3Oxqf.rst D Misc/NEWS.d/next/Core and Builtins/2022-03-30-13-13-25.bpo-47162.yDJMUm.rst D Misc/NEWS.d/next/Core and Builtins/2022-03-31-15-37-02.bpo-47182.e_4SsC.rst D Misc/NEWS.d/next/Core and Builtins/2022-03-31-15-57-42.bpo-46841.U-25Z6.rst D Misc/NEWS.d/next/Core and Builtins/2022-03-31-21-43-57.bpo-47120.NgxQbA.rst D Misc/NEWS.d/next/Core and Builtins/2022-04-01-11-53-59.bpo-47186.RBCPk8.rst D Misc/NEWS.d/next/Core and Builtins/2022-04-02-14-32-21.bpo-47176.kTygYI.rst D Misc/NEWS.d/next/Core and Builtins/2022-04-04-17-41-10.bpo-47186.aQWoSh.rst D Misc/NEWS.d/next/Core and Builtins/2022-04-05-11-29-21.bpo-47212.leF4pz.rst D Misc/NEWS.d/next/Documentation/2020-07-07-22-54-51.bpo-41233.lyUJ8L.rst D Misc/NEWS.d/next/Documentation/2020-11-12-21-26-31.bpo-42340.apumUL.rst D Misc/NEWS.d/next/Documentation/2021-11-12-11-03-55.bpo-45790.6yuhe8.rst D Misc/NEWS.d/next/Documentation/2022-01-03-18-50-39.bpo-46033.7WeF0f.rst D Misc/NEWS.d/next/Documentation/2022-03-17-13-35-28.bpo-47040.4Dn48U.rst D Misc/NEWS.d/next/Documentation/2022-03-26-12-20-16.bpo-47126.p6_Ovm.rst D Misc/NEWS.d/next/Documentation/2022-03-29-13-25-49.bpo-45099.dagdhx.rst D Misc/NEWS.d/next/Library/2019-03-14-09-08-25.bpo-35859.8lFdLe.rst D Misc/NEWS.d/next/Library/2021-03-31-15-22-45.bpo-43352.nSjMuE.rst D Misc/NEWS.d/next/Library/2021-04-20-16-48-07.bpo-33178.kSnWwb.rst D Misc/NEWS.d/next/Library/2021-06-17-00-02-58.bpo-41930.JS6fsd.rst D Misc/NEWS.d/next/Library/2021-07-26-10-46-49.bpo-44493.xp3CRH.rst D Misc/NEWS.d/next/Library/2021-08-10-00-05-53.bpo-44859.9e9_3V.rst D Misc/NEWS.d/next/Library/2021-09-06-15-46-53.bpo-24959.UVFgiO.rst D Misc/NEWS.d/next/Library/2021-09-11-16-06-54.bpo-45171.ec597j.rst D Misc/NEWS.d/next/Library/2021-11-08-20-27-41.bpo-44439.I_8qro.rst D Misc/NEWS.d/next/Library/2021-12-10-07-07-47.bpo-46030.UN349J.rst D Misc/NEWS.d/next/Library/2021-12-22-12-02-27.bpo-20392.CLAFIp.rst D Misc/NEWS.d/next/Library/2021-12-25-14-13-14.bpo-40296.p0YVGB.rst D Misc/NEWS.d/next/Library/2021-12-26-14-45-51.bpo-46170.AQ7kSM.rst D Misc/NEWS.d/next/Library/2021-12-29-19-37-49.bpo-22859.AixHW7.rst D Misc/NEWS.d/next/Library/2022-01-03-20-12-14.bpo-46245.3w4RlA.rst D Misc/NEWS.d/next/Library/2022-01-18-01-29-38.bpo-46421.9LdmNr.rst D Misc/NEWS.d/next/Library/2022-01-23-16-33-07.bpo-46480.E4jHlh.rst D Misc/NEWS.d/next/Library/2022-01-25-15-45-04.bpo-26120.YzrBMO.rst D Misc/NEWS.d/next/Library/2022-01-26-18-30-34.bpo-46607.xnhT4a.rst D Misc/NEWS.d/next/Library/2022-01-27-11-54-16.bpo-41370.gYxCPE.rst D Misc/NEWS.d/next/Library/2022-01-28-01-23-25.bpo-46557.XSbhyQ.rst D Misc/NEWS.d/next/Library/2022-01-30-20-32-40.bpo-43224.zqrQsj.rst D Misc/NEWS.d/next/Library/2022-01-30-22-05-53.bpo-43224.E-eT22.rst D Misc/NEWS.d/next/Library/2022-01-31-15-19-38.bpo-45413.1vaS0V.rst D Misc/NEWS.d/next/Library/2022-02-01-11-32-47.bpo-46581.t7Zw65.rst D Misc/NEWS.d/next/Library/2022-02-05-22-14-44.bpo-46644.P--1Cz.rst D Misc/NEWS.d/next/Library/2022-02-20-23-03-32.bpo-46805.HZ8xWG.rst D Misc/NEWS.d/next/Library/2022-02-21-11-41-23.bpo-464471.fL06TV.rst D Misc/NEWS.d/next/Library/2022-02-23-01-11-08.bpo-40059.Iwc9UH.rst D Misc/NEWS.d/next/Library/2022-03-05-21-51-31.bpo-46933.6yzWtb.rst D Misc/NEWS.d/next/Library/2022-03-07-20-20-34.bpo-46932.xbarAs.rst D Misc/NEWS.d/next/Library/2022-03-08-11-34-06.bpo-23325.3VQnfo.rst D Misc/NEWS.d/next/Library/2022-03-08-22-41-59.bpo-46955.IOoonN.rst D Misc/NEWS.d/next/Library/2022-03-10-14-47-16.bpo-46917.s19zcy.rst D Misc/NEWS.d/next/Library/2022-03-10-14-51-11.bpo-46968.ym2QxL.rst D Misc/NEWS.d/next/Library/2022-03-11-13-34-16.bpo-46985.BgoMr2.rst D Misc/NEWS.d/next/Library/2022-03-11-17-56-25.bpo-46968.pPVvNo.rst D Misc/NEWS.d/next/Library/2022-03-12-11-30-42.bpo-46981.ltWCxH.rst D Misc/NEWS.d/next/Library/2022-03-12-12-34-13.bpo-46994.d7hPdz.rst D Misc/NEWS.d/next/Library/2022-03-12-13-50-42.bpo-46995.2kdNDg.rst D Misc/NEWS.d/next/Library/2022-03-13-08-52-58.bpo-46998.cHh-9O.rst D Misc/NEWS.d/next/Library/2022-03-13-15-04-05.bpo-47004.SyYpxd.rst D Misc/NEWS.d/next/Library/2022-03-14-09-26-42.bpo-40280.2-k8TV.rst D Misc/NEWS.d/next/Library/2022-03-15-07-53-45.bpo-43253.rjdLFj.rst D Misc/NEWS.d/next/Library/2022-03-15-09-29-52.bpo-47022.uaEDcI.rst D Misc/NEWS.d/next/Library/2022-03-15-18-32-12.bpo-45997.4n2aVU.rst D Misc/NEWS.d/next/Library/2022-03-16-08-49-12.bpo-34861.p8ugVg.rst D Misc/NEWS.d/next/Library/2022-03-16-11-52-52.bpo-45150.kYbIME.rst D Misc/NEWS.d/next/Library/2022-03-16-18-25-19.bpo-2604.jeopdL.rst D Misc/NEWS.d/next/Library/2022-03-17-01-54-13.bpo-47039.0Yxv0K.rst D Misc/NEWS.d/next/Library/2022-03-17-19-38-40.bpo-34790.zQIiVJ.rst D Misc/NEWS.d/next/Library/2022-03-18-13-30-40.bpo-47061.etLHK5.rst D Misc/NEWS.d/next/Library/2022-03-18-14-22-38.bpo-47057.n-IHbt.rst D Misc/NEWS.d/next/Library/2022-03-18-17-25-57.bpo-46382.zQUJ66.rst D Misc/NEWS.d/next/Library/2022-03-18-22-46-18.bpo-47062.RNc99_.rst D Misc/NEWS.d/next/Library/2022-03-19-08-42-57.bpo-433030.UTwRX7.rst D Misc/NEWS.d/next/Library/2022-03-19-13-38-29.bpo-39394.7j6WL6.rst D Misc/NEWS.d/next/Library/2022-03-19-14-12-23.bpo-47066.we3YFx.rst D Misc/NEWS.d/next/Library/2022-03-19-15-54-41.bpo-38256.FoMbjE.rst D Misc/NEWS.d/next/Library/2022-03-19-19-56-04.bpo-42369.Ok828t.rst D Misc/NEWS.d/next/Library/2022-03-20-13-00-08.bpo-47000.p8HpG0.rst D Misc/NEWS.d/next/Library/2022-03-20-15-54-41.bpo-28080.kn35Vk.rst D Misc/NEWS.d/next/Library/2022-03-20-17-15-56.bpo-47067.XXLnje.rst D Misc/NEWS.d/next/Library/2022-03-20-22-13-24.bpo-23691.Nc2TrW.rst D Misc/NEWS.d/next/Library/2022-03-21-08-32-19.bpo-42885.LCnTTp.rst D Misc/NEWS.d/next/Library/2022-03-22-19-18-31.bpo-47088.JM1kNI.rst D Misc/NEWS.d/next/Library/2022-03-23-10-07-41.bpo-47098.7AN_qp.rst D Misc/NEWS.d/next/Library/2022-03-23-12-07-26.bpo-47095.P3YTrh.rst D Misc/NEWS.d/next/Library/2022-03-23-13-55-41.bpo-47099.P6quRP.rst D Misc/NEWS.d/next/Library/2022-03-23-14-16-38.bpo-47099.2raait.rst D Misc/NEWS.d/next/Library/2022-03-23-15-31-02.bpo-47101.rVSld-.rst D Misc/NEWS.d/next/Library/2022-03-25-01-27-25.bpo-39622.ieBIMp.rst D Misc/NEWS.d/next/Library/2022-03-26-13-14-43.bpo-47061.QLxbC6.rst D Misc/NEWS.d/next/Library/2022-03-27-10-41-24.bpo-14265.OBMlAi.rst D Misc/NEWS.d/next/Library/2022-03-28-13-35-50.bpo-27929.j5mAmV.rst D Misc/NEWS.d/next/Library/2022-03-28-20-16-37.bpo-4833.2vSUE5.rst D Misc/NEWS.d/next/Library/2022-03-29-19-14-53.bpo-47152.5rl5ZK.rst D Misc/NEWS.d/next/Library/2022-03-30-01-17-43.bpo-47151.z-nQkR.rst D Misc/NEWS.d/next/Library/2022-03-30-18-35-50.bpo-47167.nCNHsB.rst D Misc/NEWS.d/next/Library/2022-04-03-13-19-08.bpo-23689.TFSc3E.rst D Misc/NEWS.d/next/Library/2022-04-04-08-54-31.bpo-47208.cOh9xZ.rst D Misc/NEWS.d/next/Tests/2022-03-13-23-43-40.bpo-47015.FjmCsz.rst D Misc/NEWS.d/next/Tests/2022-03-14-17-10-35.bpo-46587.ASDsJX.rst D Misc/NEWS.d/next/Tests/2022-03-16-21-29-30.bpo-47037.xcrLpJ.rst D Misc/NEWS.d/next/Tests/2022-03-19-10-25-04.bpo-40280.wBRSel.rst D Misc/NEWS.d/next/Tests/2022-03-23-22-45-51.bpo-47104._esUq8.rst D Misc/NEWS.d/next/Tests/2022-03-26-11-41-19.bpo-46126.q14Ioy.rst D Misc/NEWS.d/next/Tests/2022-04-03-14-38-21.bpo-47205.hbbTnh.rst D Misc/NEWS.d/next/Tools-Demos/2022-04-03-11-47-45.bpo-40280.Q_IJik.rst D Misc/NEWS.d/next/Windows/2022-03-07-16-34-11.bpo-46948.Ufd4tG.rst D Misc/NEWS.d/next/Windows/2022-03-07-17-46-40.bpo-44549.SPrGS9.rst D Misc/NEWS.d/next/Windows/2022-03-13-11-18-41.bpo-46907.YLzxBM.rst D Misc/NEWS.d/next/Windows/2022-03-21-20-45-01.bpo-47086.bIuKlF.rst D Misc/NEWS.d/next/Windows/2022-03-23-12-51-46.bpo-46566.4x4a7e.rst D Misc/NEWS.d/next/Windows/2022-03-30-19-55-00.bpo-47171.MbqCWn.rst D Misc/NEWS.d/next/Windows/2022-04-01-14-57-40.bpo-47194.IB0XL4.rst D Misc/NEWS.d/next/macOS/2022-03-13-11-11-31.bpo-46907.Ql0z1E.rst D Misc/NEWS.d/next/macOS/2022-03-17-09-55-02.bpo-46890.GX-3OO.rst M Include/patchlevel.h M Lib/pydoc_data/topics.py M README.rst diff --git a/Include/patchlevel.h b/Include/patchlevel.h index fae787d04e2f7..9850a117aa9bd 100644 --- a/Include/patchlevel.h +++ b/Include/patchlevel.h @@ -20,10 +20,10 @@ #define PY_MINOR_VERSION 11 #define PY_MICRO_VERSION 0 #define PY_RELEASE_LEVEL PY_RELEASE_LEVEL_ALPHA -#define PY_RELEASE_SERIAL 6 +#define PY_RELEASE_SERIAL 7 /* Version as a string */ -#define PY_VERSION "3.11.0a6+" +#define PY_VERSION "3.11.0a7" /*--end constants--*/ /* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2. diff --git a/Lib/pydoc_data/topics.py b/Lib/pydoc_data/topics.py index 433c905096e9e..9b684c6dfc296 100644 --- a/Lib/pydoc_data/topics.py +++ b/Lib/pydoc_data/topics.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Autogenerated by Sphinx on Mon Mar 7 12:29:42 2022 +# Autogenerated by Sphinx on Tue Apr 5 20:53:43 2022 topics = {'assert': 'The "assert" statement\n' '**********************\n' '\n' @@ -93,11 +93,7 @@ ' optionally in parentheses, the object is assigned to that ' 'target.\n' '\n' - '* Else: The object must be an iterable with the same number of ' - 'items\n' - ' as there are targets in the target list, and the items are ' - 'assigned,\n' - ' from left to right, to the corresponding targets.\n' + '* Else:\n' '\n' ' * If the target list contains one target prefixed with an ' 'asterisk,\n' @@ -2328,27 +2324,18 @@ ' for_stmt ::= "for" target_list "in" starred_list ":" suite\n' ' ["else" ":" suite]\n' '\n' - 'The expression list is evaluated once; it should yield an ' - 'iterable\n' - 'object. An iterator is created for the result of the ' - '"starred_list".\n' - 'The expression list can contain starred elements ("*x, *y") that ' - 'will\n' - 'be unpacked in the final iterator (as when constructing a ' - '"tuple" or\n' - '"list" literal). The suite is then executed once for each item\n' - 'provided by the iterator, in the order returned by the ' - 'iterator. Each\n' - 'item in turn is assigned to the target list using the standard ' - 'rules\n' - 'for assignments (see Assignment statements), and then the suite ' - 'is\n' - 'executed. When the items are exhausted (which is immediately ' - 'when the\n' - 'sequence is empty or an iterator raises a "StopIteration" ' - 'exception),\n' - 'the suite in the "else" clause, if present, is executed, and the ' - 'loop\n' + 'The "starred_list" expression is evaluated once; it should yield ' + 'an\n' + '*iterable* object. An *iterator* is created for that iterable. ' + 'The\n' + 'first item provided by the iterator is then assigned to the ' + 'target\n' + 'list using the standard rules for assignments (see Assignment\n' + 'statements), and the suite is executed. This repeats for each ' + 'item\n' + 'provided by the iterator. When the iterator is exhausted, the ' + 'suite\n' + 'in the "else" clause, if present, is executed, and the loop\n' 'terminates.\n' '\n' 'A "break" statement executed in the first suite terminates the ' @@ -2427,11 +2414,11 @@ 'resulting\n' 'object is ?compatible? with the exception. An object is ' 'compatible\n' - 'with an exception if the object is the class or a base class of ' - 'the\n' - 'exception object, or a tuple containing an item that is the ' - 'class or a\n' - 'base class of the exception object.\n' + 'with an exception if the object is the class or a *non-virtual ' + 'base\n' + 'class* of the exception object, or a tuple containing an item ' + 'that is\n' + 'the class or a non-virtual base class of the exception object.\n' '\n' 'If no except clause matches the exception, the search for an ' 'exception\n' @@ -4461,15 +4448,17 @@ 'on members\n' ' of hashed collections including "set", "frozenset", and ' '"dict".\n' - ' "__hash__()" should return an integer. The only required ' - 'property\n' - ' is that objects which compare equal have the same hash ' - 'value; it is\n' - ' advised to mix together the hash values of the ' - 'components of the\n' - ' object that also play a part in comparison of objects by ' - 'packing\n' - ' them into a tuple and hashing the tuple. Example:\n' + ' The "__hash__()" method should return an integer. The ' + 'only required\n' + ' property is that objects which compare equal have the ' + 'same hash\n' + ' value; it is advised to mix together the hash values of ' + 'the\n' + ' components of the object that also play a part in ' + 'comparison of\n' + ' objects by packing them into a tuple and hashing the ' + 'tuple.\n' + ' Example:\n' '\n' ' def __hash__(self):\n' ' return hash((self.name, self.nick, self.color))\n' @@ -5458,11 +5447,11 @@ 'clause is\n' 'selected depending on the class of the instance: it must ' 'reference the\n' - 'class of the instance or a base class thereof. The instance ' - 'can be\n' - 'received by the handler and can carry additional information ' - 'about the\n' - 'exceptional condition.\n' + 'class of the instance or a *non-virtual base class* thereof. ' + 'The\n' + 'instance can be received by the handler and can carry ' + 'additional\n' + 'information about the exceptional condition.\n' '\n' 'Note:\n' '\n' @@ -5797,11 +5786,11 @@ 'clause is\n' 'selected depending on the class of the instance: it must ' 'reference the\n' - 'class of the instance or a base class thereof. The instance ' - 'can be\n' - 'received by the handler and can carry additional information ' - 'about the\n' - 'exceptional condition.\n' + 'class of the instance or a *non-virtual base class* thereof. ' + 'The\n' + 'instance can be received by the handler and can carry ' + 'additional\n' + 'information about the exceptional condition.\n' '\n' 'Note:\n' '\n' @@ -5901,23 +5890,13 @@ ' for_stmt ::= "for" target_list "in" starred_list ":" suite\n' ' ["else" ":" suite]\n' '\n' - 'The expression list is evaluated once; it should yield an iterable\n' - 'object. An iterator is created for the result of the ' - '"starred_list".\n' - 'The expression list can contain starred elements ("*x, *y") that ' - 'will\n' - 'be unpacked in the final iterator (as when constructing a "tuple" or\n' - '"list" literal). The suite is then executed once for each item\n' - 'provided by the iterator, in the order returned by the iterator. ' - 'Each\n' - 'item in turn is assigned to the target list using the standard rules\n' - 'for assignments (see Assignment statements), and then the suite is\n' - 'executed. When the items are exhausted (which is immediately when ' - 'the\n' - 'sequence is empty or an iterator raises a "StopIteration" ' - 'exception),\n' - 'the suite in the "else" clause, if present, is executed, and the ' - 'loop\n' + 'The "starred_list" expression is evaluated once; it should yield an\n' + '*iterable* object. An *iterator* is created for that iterable. The\n' + 'first item provided by the iterator is then assigned to the target\n' + 'list using the standard rules for assignments (see Assignment\n' + 'statements), and the suite is executed. This repeats for each item\n' + 'provided by the iterator. When the iterator is exhausted, the suite\n' + 'in the "else" clause, if present, is executed, and the loop\n' 'terminates.\n' '\n' 'A "break" statement executed in the first suite terminates the loop\n' @@ -9388,15 +9367,17 @@ 'on members\n' ' of hashed collections including "set", "frozenset", and ' '"dict".\n' - ' "__hash__()" should return an integer. The only required ' - 'property\n' - ' is that objects which compare equal have the same hash ' - 'value; it is\n' - ' advised to mix together the hash values of the components ' - 'of the\n' - ' object that also play a part in comparison of objects by ' - 'packing\n' - ' them into a tuple and hashing the tuple. Example:\n' + ' The "__hash__()" method should return an integer. The ' + 'only required\n' + ' property is that objects which compare equal have the ' + 'same hash\n' + ' value; it is advised to mix together the hash values of ' + 'the\n' + ' components of the object that also play a part in ' + 'comparison of\n' + ' objects by packing them into a tuple and hashing the ' + 'tuple.\n' + ' Example:\n' '\n' ' def __hash__(self):\n' ' return hash((self.name, self.nick, self.color))\n' @@ -10003,14 +9984,14 @@ '\n' 'Whenever a class inherits from another class, ' '"__init_subclass__()" is\n' - 'called on that class. This way, it is possible to write ' - 'classes which\n' - 'change the behavior of subclasses. This is closely related ' - 'to class\n' - 'decorators, but where class decorators only affect the ' - 'specific class\n' - 'they?re applied to, "__init_subclass__" solely applies to ' - 'future\n' + 'called on the parent class. This way, it is possible to ' + 'write classes\n' + 'which change the behavior of subclasses. This is closely ' + 'related to\n' + 'class decorators, but where class decorators only affect the ' + 'specific\n' + 'class they?re applied to, "__init_subclass__" solely applies ' + 'to future\n' 'subclasses of the class defining the method.\n' '\n' 'classmethod object.__init_subclass__(cls)\n' @@ -12381,67 +12362,86 @@ 'subscriptions': 'Subscriptions\n' '*************\n' '\n' - 'Subscription of a sequence (string, tuple or list) or ' - 'mapping\n' - '(dictionary) object usually selects an item from the ' - 'collection:\n' + 'The subscription of an instance of a container class will ' + 'generally\n' + 'select an element from the container. The subscription of a ' + '*generic\n' + 'class* will generally return a GenericAlias object.\n' '\n' ' subscription ::= primary "[" expression_list "]"\n' '\n' + 'When an object is subscripted, the interpreter will ' + 'evaluate the\n' + 'primary and the expression list.\n' + '\n' 'The primary must evaluate to an object that supports ' - 'subscription\n' - '(lists or dictionaries for example). User-defined objects ' - 'can support\n' - 'subscription by defining a "__getitem__()" method.\n' + 'subscription. An\n' + 'object may support subscription through defining one or ' + 'both of\n' + '"__getitem__()" and "__class_getitem__()". When the primary ' + 'is\n' + 'subscripted, the evaluated result of the expression list ' + 'will be\n' + 'passed to one of these methods. For more details on when\n' + '"__class_getitem__" is called instead of "__getitem__", ' + 'see\n' + '__class_getitem__ versus __getitem__.\n' + '\n' + 'If the expression list contains at least one comma, it will ' + 'evaluate\n' + 'to a "tuple" containing the items of the expression list. ' + 'Otherwise,\n' + 'the expression list will evaluate to the value of the ' + 'list?s sole\n' + 'member.\n' '\n' 'For built-in objects, there are two types of objects that ' 'support\n' - 'subscription:\n' + 'subscription via "__getitem__()":\n' '\n' - 'If the primary is a mapping, the expression list must ' - 'evaluate to an\n' - 'object whose value is one of the keys of the mapping, and ' + '1. Mappings. If the primary is a *mapping*, the expression ' + 'list must\n' + ' evaluate to an object whose value is one of the keys of ' 'the\n' - 'subscription selects the value in the mapping that ' - 'corresponds to that\n' - 'key. (The expression list is a tuple except if it has ' - 'exactly one\n' - 'item.)\n' - '\n' - 'If the primary is a sequence, the expression list must ' - 'evaluate to an\n' - 'integer or a slice (as discussed in the following ' - 'section).\n' + ' mapping, and the subscription selects the value in the ' + 'mapping that\n' + ' corresponds to that key. An example of a builtin mapping ' + 'class is\n' + ' the "dict" class.\n' + '\n' + '2. Sequences. If the primary is a *sequence*, the ' + 'expression list must\n' + ' evaluate to an "int" or a "slice" (as discussed in the ' + 'following\n' + ' section). Examples of builtin sequence classes include ' + 'the "str",\n' + ' "list" and "tuple" classes.\n' '\n' 'The formal syntax makes no special provision for negative ' 'indices in\n' - 'sequences; however, built-in sequences all provide a ' + '*sequences*. However, built-in sequences all provide a ' '"__getitem__()"\n' 'method that interprets negative indices by adding the ' 'length of the\n' - 'sequence to the index (so that "x[-1]" selects the last ' - 'item of "x").\n' - 'The resulting value must be a nonnegative integer less than ' - 'the number\n' - 'of items in the sequence, and the subscription selects the ' - 'item whose\n' - 'index is that value (counting from zero). Since the support ' - 'for\n' - 'negative indices and slicing occurs in the object?s ' - '"__getitem__()"\n' - 'method, subclasses overriding this method will need to ' - 'explicitly add\n' - 'that support.\n' - '\n' - 'A string?s items are characters. A character is not a ' - 'separate data\n' - 'type but a string of exactly one character.\n' - '\n' - 'Subscription of certain *classes* or *types* creates a ' - 'generic alias.\n' - 'In this case, user-defined classes can support subscription ' - 'by\n' - 'providing a "__class_getitem__()" classmethod.\n', + 'sequence to the index so that, for example, "x[-1]" selects ' + 'the last\n' + 'item of "x". The resulting value must be a nonnegative ' + 'integer less\n' + 'than the number of items in the sequence, and the ' + 'subscription selects\n' + 'the item whose index is that value (counting from zero). ' + 'Since the\n' + 'support for negative indices and slicing occurs in the ' + 'object?s\n' + '"__getitem__()" method, subclasses overriding this method ' + 'will need to\n' + 'explicitly add that support.\n' + '\n' + 'A "string" is a special kind of sequence whose items are ' + '*characters*.\n' + 'A character is not a separate data type but a string of ' + 'exactly one\n' + 'character.\n', 'truth': 'Truth Value Testing\n' '*******************\n' '\n' @@ -12502,10 +12502,10 @@ 'exception. For an except clause with an expression, that expression\n' 'is evaluated, and the clause matches the exception if the resulting\n' 'object is ?compatible? with the exception. An object is compatible\n' - 'with an exception if the object is the class or a base class of the\n' - 'exception object, or a tuple containing an item that is the class or ' - 'a\n' - 'base class of the exception object.\n' + 'with an exception if the object is the class or a *non-virtual base\n' + 'class* of the exception object, or a tuple containing an item that ' + 'is\n' + 'the class or a non-virtual base class of the exception object.\n' '\n' 'If no except clause matches the exception, the search for an ' 'exception\n' diff --git a/Misc/NEWS.d/3.11.0a7.rst b/Misc/NEWS.d/3.11.0a7.rst new file mode 100644 index 0000000000000..5eaf8ec1f63c0 --- /dev/null +++ b/Misc/NEWS.d/3.11.0a7.rst @@ -0,0 +1,1614 @@ +.. bpo: 47212 +.. date: 2022-04-05-11-29-21 +.. nonce: leF4pz +.. release date: 2022-04-05 +.. section: Core and Builtins + +Raise :exc:`IndentationError` instead of :exc:`SyntaxError` for a bare +``except`` with no following indent. Improve :exc:`SyntaxError` locations +for an un-parenthesized generator used as arguments. Patch by Matthieu +Dartiailh. + +.. + +.. bpo: 47186 +.. date: 2022-04-04-17-41-10 +.. nonce: aQWoSh +.. section: Core and Builtins + +Replace :opcode:`JUMP_IF_NOT_EG_MATCH` by :opcode:`CHECK_EG_MATCH` + jump. + +.. + +.. bpo: 47176 +.. date: 2022-04-02-14-32-21 +.. nonce: kTygYI +.. section: Core and Builtins + +Emscripten builds cannot handle signals in the usual way due to platform +limitations. Python can now handle signals. To use, set +Module.Py_EmscriptenSignalBuffer to be a single byte SharedArrayBuffer and +set Py_EMSCRIPTEN_SIGNAL_HANDLING to 1. Writing a number into the +SharedArrayBuffer will cause the corresponding signal to be raised into the +Python thread. + +.. + +.. bpo: 47186 +.. date: 2022-04-01-11-53-59 +.. nonce: RBCPk8 +.. section: Core and Builtins + +Replace :opcode:`JUMP_IF_NOT_EXC_MATCH` by :opcode:`CHECK_EXC_MATCH` + jump. + +.. + +.. bpo: 47120 +.. date: 2022-03-31-21-43-57 +.. nonce: NgxQbA +.. section: Core and Builtins + +Replace the absolute jump opcode :opcode:`JUMP_NO_INTERRUPT` by the relative +:opcode:`JUMP_BACKWARD_NO_INTERRUPT`. + +.. + +.. bpo: 46841 +.. date: 2022-03-31-15-57-42 +.. nonce: U-25Z6 +.. section: Core and Builtins + +Avoid unnecessary allocations when comparing code objects. + +.. + +.. bpo: 47182 +.. date: 2022-03-31-15-37-02 +.. nonce: e_4SsC +.. section: Core and Builtins + +Fix a crash when using a named unicode character like ``"\N{digit nine}"`` +after the main interpreter has been initialized a second time. + +.. + +.. bpo: 47162 +.. date: 2022-03-30-13-13-25 +.. nonce: yDJMUm +.. section: Core and Builtins + +WebAssembly cannot deal with bad function pointer casts (different count or +types of arguments). Python can now use call trampolines to mitigate the +problem. Define :c:macro:`PY_CALL_TRAMPOLINE` to enable call trampolines. + +.. + +.. bpo: 46775 +.. date: 2022-03-30-02-36-25 +.. nonce: e3Oxqf +.. section: Core and Builtins + +Some Windows system error codes(>= 10000) are now mapped into the correct +errno and may now raise a subclass of :exc:`OSError`. Patch by Dong-hee Na. + +.. + +.. bpo: 47129 +.. date: 2022-03-26-16-35-57 +.. nonce: hDg2Vt +.. section: Core and Builtins + +Improve error messages in f-string syntax errors concerning empty +expressions. + +.. + +.. bpo: 47117 +.. date: 2022-03-26-15-45-57 +.. nonce: 60W6GQ +.. section: Core and Builtins + +Fix a crash if we fail to decode characters in interactive mode if the +tokenizer buffers are uninitialized. Patch by Pablo Galindo. + +.. + +.. bpo: 47127 +.. date: 2022-03-26-12-21-53 +.. nonce: Mh86RB +.. section: Core and Builtins + +Speed up calls to c functions with keyword arguments by 25% with +specialization. Patch by Kumar Aditya. + +.. + +.. bpo: 47120 +.. date: 2022-03-25-21-51-10 +.. nonce: 9YJ-Xw +.. section: Core and Builtins + +Replaced :opcode:`JUMP_ABSOLUTE` by the relative jump +:opcode:`JUMP_BACKWARD`. + +.. + +.. bpo: 42197 +.. date: 2022-03-22-15-12-28 +.. nonce: SwrrFO +.. section: Core and Builtins + +:c:func:`PyFrame_FastToLocalsWithError` and :c:func:`PyFrame_LocalsToFast` +are no longer called during profiling nor tracing. C code can access the +``f_locals`` attribute of :c:type:`PyFrameObject` by calling +:c:func:`PyFrame_GetLocals`. + +.. + +.. bpo: 47070 +.. date: 2022-03-19-21-50-59 +.. nonce: wPcsQh +.. section: Core and Builtins + +Improve performance of ``array_inplace_repeat`` by reducing the number of +invocations of ``memcpy``. Refactor the ``repeat`` and inplace ``repeat`` +methods of ``array``, ``bytes``, ``bytearray`` and ``unicodeobject`` to use +the common ``_PyBytes_Repeat``. + +.. + +.. bpo: 47053 +.. date: 2022-03-17-22-47-29 +.. nonce: QAXk8Q +.. section: Core and Builtins + +Reduce de-optimization in the specialized ``BINARY_OP_INPLACE_ADD_UNICODE`` +opcode. + +.. + +.. bpo: 47045 +.. date: 2022-03-17-16-25-57 +.. nonce: xQgHul +.. section: Core and Builtins + +Remove the ``f_state`` field from the _PyInterpreterFrame struct. Add the +``owner`` field to the _PyInterpreterFrame struct to make ownership explicit +to simplify clearing and deallocing frames and generators. + +.. + +.. bpo: 46968 +.. date: 2022-03-17-14-22-23 +.. nonce: 4gz4NA +.. section: Core and Builtins + +Check for the existence of the "sys/auxv.h" header in :mod:`faulthandler` to +avoid compilation problems in systems where this header doesn't exist. Patch +by Pablo Galindo + +.. + +.. bpo: 46329 +.. date: 2022-03-16-12-19-25 +.. nonce: 9oS0HT +.. section: Core and Builtins + +Use low bit of ``LOAD_GLOBAL`` to indicate whether to push a ``NULL`` before +the global. Helps streamline the call sequence a bit. + +.. + +.. bpo: 46841 +.. date: 2022-03-16-11-05-35 +.. nonce: yUoIHg +.. section: Core and Builtins + +Quicken bytecode in-place by storing it as part of the corresponding +``PyCodeObject``. + +.. + +.. bpo: 47012 +.. date: 2022-03-14-11-15-11 +.. nonce: 5L6NoE +.. section: Core and Builtins + +Speed up iteration of :class:`bytes` and :class:`bytearray` by 30%. Patch by +Kumar Aditya. + +.. + +.. bpo: 47009 +.. date: 2022-03-14-09-45-10 +.. nonce: ZI05b5 +.. section: Core and Builtins + +Improved the performance of :meth:`list.append()` and list comprehensions by +optimizing for the common case, where no resize is needed. Patch by Dennis +Sweeney. + +.. + +.. bpo: 47005 +.. date: 2022-03-13-21-04-20 +.. nonce: OHBfCc +.. section: Core and Builtins + +Improve performance of ``bytearray_repeat`` and ``bytearray_irepeat`` by +reducing the number of invocations of ``memcpy``. + +.. + +.. bpo: 46829 +.. date: 2022-03-12-21-07-21 +.. nonce: cpGoPV +.. section: Core and Builtins + +Deprecate passing a message into :meth:`asyncio.Future.cancel` and +:meth:`asyncio.Task.cancel` + +.. + +.. bpo: 46993 +.. date: 2022-03-12-09-44-31 +.. nonce: -13hGo +.. section: Core and Builtins + +Speed up :class:`bytearray` creation from :class:`list` and :class:`tuple` +by 40%. Patch by Kumar Aditya. + +.. + +.. bpo: 39829 +.. date: 2022-03-11-09-39-01 +.. nonce: mlW3Su +.. section: Core and Builtins + +Removed the ``__len__()`` call when initializing a list and moved +initializing to ``list_extend``. Patch by Jeremiah Pascual. + +.. + +.. bpo: 46944 +.. date: 2022-03-08-10-50-42 +.. nonce: cnaIK3 +.. section: Core and Builtins + +Speed up throwing exception in generator with :const:`METH_FASTCALL` calling +convention. Patch by Kumar Aditya. + +.. + +.. bpo: 46841 +.. date: 2022-03-07-15-54-39 +.. nonce: 7wG92r +.. section: Core and Builtins + +Modify :opcode:`STORE_SUBSCR` to use an inline cache entry (rather than its +oparg) as an adaptive counter. + +.. + +.. bpo: 46841 +.. date: 2022-03-06-10-37-36 +.. nonce: O12Pba +.. section: Core and Builtins + +Use inline caching for :opcode:`PRECALL` and :opcode:`CALL`, and remove the +internal machinery for managing the (now unused) non-inline caches. + +.. + +.. bpo: 46881 +.. date: 2022-03-03-09-08-17 +.. nonce: ckD4tT +.. section: Core and Builtins + +Statically allocate and initialize the latin1 characters. + +.. + +.. bpo: 46838 +.. date: 2022-02-25-22-42-30 +.. nonce: RB6kEy +.. section: Core and Builtins + +Improve syntax errors for incorrect function definitions. Patch by Pablo +Galindo + +.. + +.. bpo: 43721 +.. date: 2022-02-01-10-05-27 +.. nonce: -1XAIo +.. section: Core and Builtins + +Fix docstrings of :attr:`~property.getter`, :attr:`~property.setter`, and +:attr:`~property.deleter` to clarify that they create a new copy of the +property. + +.. + +.. bpo: 43224 +.. date: 2022-01-20-16-48-09 +.. nonce: WDihrT +.. section: Core and Builtins + +Make grammar changes required for PEP 646. + +.. + +.. bpo: 47208 +.. date: 2022-04-04-08-54-31 +.. nonce: cOh9xZ +.. section: Library + +Allow vendors to override :const:`CTYPES_MAX_ARGCOUNT`. + +.. + +.. bpo: 23689 +.. date: 2022-04-03-13-19-08 +.. nonce: TFSc3E +.. section: Library + +:mod:`re` module: fix memory leak when a match is terminated by a signal or +memory allocation failure. Patch by Ma Lin. + +.. + +.. bpo: 47167 +.. date: 2022-03-30-18-35-50 +.. nonce: nCNHsB +.. section: Library + +Allow overriding a future compliance check in :class:`asyncio.Task`. + +.. + +.. bpo: 47151 +.. date: 2022-03-30-01-17-43 +.. nonce: z-nQkR +.. section: Library + +When subprocess tries to use vfork, it now falls back to fork if vfork +returns an error. This allows use in situations where vfork isn't allowed by +the OS kernel. + +.. + +.. bpo: 47152 +.. date: 2022-03-29-19-14-53 +.. nonce: 5rl5ZK +.. section: Library + +Convert the :mod:`re` module into a package. Deprecate modules +``sre_compile``, ``sre_constants`` and ``sre_parse``. + +.. + +.. bpo: 4833 +.. date: 2022-03-28-20-16-37 +.. nonce: 2vSUE5 +.. section: Library + +Add :meth:`ZipFile.mkdir` + +.. + +.. bpo: 27929 +.. date: 2022-03-28-13-35-50 +.. nonce: j5mAmV +.. section: Library + +Fix :meth:`asyncio.loop.sock_connect` to only resolve names for +:const:`socket.AF_INET` or :const:`socket.AF_INET6` families. Resolution may +not make sense for other families, like :const:`socket.AF_BLUETOOTH` and +:const:`socket.AF_UNIX`. + +.. + +.. bpo: 14265 +.. date: 2022-03-27-10-41-24 +.. nonce: OBMlAi +.. section: Library + +Adds the fully qualified test name to unittest output + +.. + +.. bpo: 47061 +.. date: 2022-03-26-13-14-43 +.. nonce: QLxbC6 +.. section: Library + +Deprecate the aifc module. + +.. + +.. bpo: 39622 +.. date: 2022-03-25-01-27-25 +.. nonce: ieBIMp +.. section: Library + +Handle Ctrl+C in asyncio programs to interrupt the main task. + +.. + +.. bpo: 47101 +.. date: 2022-03-23-15-31-02 +.. nonce: rVSld- +.. section: Library + +:const:`hashlib.algorithms_available` now lists only algorithms that are +provided by activated crypto providers on OpenSSL 3.0. Legacy algorithms are +not listed unless the legacy provider has been loaded into the default OSSL +context. + +.. + +.. bpo: 47099 +.. date: 2022-03-23-14-16-38 +.. nonce: 2raait +.. section: Library + +All :exc:`URLError` exception messages raised in +:class:`urllib.request.URLopener` now contain a colon between ``ftp error`` +and the rest of the message. Previously, +:func:`~urllib.request.URLopener.open_ftp` missed the colon. Patch by Oleg +Iarygin. + +.. + +.. bpo: 47099 +.. date: 2022-03-23-13-55-41 +.. nonce: P6quRP +.. section: Library + +Exception chaining is changed from +:func:`Exception.with_traceback`/:func:`sys.exc_info` to :pep:`3134`. Patch +by Oleg Iarygin. + +.. + +.. bpo: 47095 +.. date: 2022-03-23-12-07-26 +.. nonce: P3YTrh +.. section: Library + +:mod:`hashlib`'s internal ``_blake2`` module now prefers ``libb2`` from +https://www.blake2.net/ over Python's vendored copy of blake2. + +.. + +.. bpo: 47098 +.. date: 2022-03-23-10-07-41 +.. nonce: 7AN_qp +.. section: Library + +The Keccak Code Package for :mod:`hashlib`'s internal ``_sha3`` module has +been replaced with tiny_sha3. The module is used as fallback when Python is +built without OpenSSL. + +.. + +.. bpo: 47088 +.. date: 2022-03-22-19-18-31 +.. nonce: JM1kNI +.. section: Library + +Implement :data:`typing.LiteralString`, part of :pep:`675`. Patch by Jelle +Zijlstra. + +.. + +.. bpo: 42885 +.. date: 2022-03-21-08-32-19 +.. nonce: LCnTTp +.. section: Library + +Optimize :func:`re.search`, :func:`re.split`, :func:`re.findall`, +:func:`re.finditer` and :func:`re.sub` for regular expressions starting with +``\A`` or ``^``. + +.. + +.. bpo: 23691 +.. date: 2022-03-20-22-13-24 +.. nonce: Nc2TrW +.. section: Library + +Protect the :func:`re.finditer` iterator from re-entering. + +.. + +.. bpo: 47067 +.. date: 2022-03-20-17-15-56 +.. nonce: XXLnje +.. section: Library + +Optimize calling ``GenericAlias`` objects by using :pep:`590` ``vectorcall`` +and by replacing ``PyObject_SetAttrString`` with ``PyObject_SetAttr``. + +.. + +.. bpo: 28080 +.. date: 2022-03-20-15-54-41 +.. nonce: kn35Vk +.. section: Library + +Add the *metadata_encoding* parameter in the :class:`zipfile.ZipFile` +constructor and the ``--metadata-encoding`` option in the :mod:`zipfile` CLI +to allow reading zipfiles using non-standard codecs to encode the filenames +within the archive. + +.. + +.. bpo: 47000 +.. date: 2022-03-20-13-00-08 +.. nonce: p8HpG0 +.. section: Library + +Make :func:`io.text_encoding` returns "utf-8" when UTF-8 mode is enabled. + +.. + +.. bpo: 42369 +.. date: 2022-03-19-19-56-04 +.. nonce: Ok828t +.. section: Library + +Fix thread safety of :meth:`zipfile._SharedFile.tell` to avoid a +"zipfile.BadZipFile: Bad CRC-32 for file" exception when reading a +:class:`ZipFile` from multiple threads. + +.. + +.. bpo: 38256 +.. date: 2022-03-19-15-54-41 +.. nonce: FoMbjE +.. section: Library + +Fix :func:`binascii.crc32` when it is compiled to use zlib'c crc32 to work +properly on inputs 4+GiB in length instead of returning the wrong result. +The workaround prior to this was to always feed the function data in +increments smaller than 4GiB or to just call the zlib module function. + +We also have :func:`binascii.crc32` release the GIL when computing on larger +inputs as :func:`zlib.crc32` and :mod:`hashlib` do. + +This also boosts performance on Windows as it now uses the zlib crc32 +implementation for :func:`binascii.crc32` for a 2-3x speedup. + +That the stdlib has a crc32 API in two modules is a known historical oddity. +This moves us closer to a single implementation behind them. + +.. + +.. bpo: 47066 +.. date: 2022-03-19-14-12-23 +.. nonce: we3YFx +.. section: Library + +Global inline flags (e.g. ``(?i)``) can now only be used at the start of the +regular expressions. Using them not at the start of expression was +deprecated since Python 3.6. + +.. + +.. bpo: 39394 +.. date: 2022-03-19-13-38-29 +.. nonce: 7j6WL6 +.. section: Library + +A warning about inline flags not at the start of the regular expression now +contains the position of the flag. + +.. + +.. bpo: 433030 +.. date: 2022-03-19-08-42-57 +.. nonce: UTwRX7 +.. section: Library + +Add support of atomic grouping (``(?>...)``) and possessive quantifiers +(``*+``, ``++``, ``?+``, ``{m,n}+``) in :mod:`regular expressions `. + +.. + +.. bpo: 47062 +.. date: 2022-03-18-22-46-18 +.. nonce: RNc99_ +.. section: Library + +Implement :class:`asyncio.Runner` context manager. + +.. + +.. bpo: 46382 +.. date: 2022-03-18-17-25-57 +.. nonce: zQUJ66 +.. section: Library + +:func:`~dataclasses.dataclass` ``slots=True`` now correctly omits slots +already defined in base classes. Patch by Arie Bovenberg. + +.. + +.. bpo: 47057 +.. date: 2022-03-18-14-22-38 +.. nonce: n-IHbt +.. section: Library + +Use FASTCALL convention for ``FutureIter.throw()`` + +.. + +.. bpo: 47061 +.. date: 2022-03-18-13-30-40 +.. nonce: etLHK5 +.. section: Library + +Deprecate the various modules listed by :pep:`594`: + +aifc, asynchat, asyncore, audioop, cgi, cgitb, chunk, crypt, imghdr, msilib, +nntplib, nis, ossaudiodev, pipes, smtpd, sndhdr, spwd, sunau, telnetlib, uu, +xdrlib + +.. + +.. bpo: 34790 +.. date: 2022-03-17-19-38-40 +.. nonce: zQIiVJ +.. section: Library + +Remove passing coroutine objects to :func:`asyncio.wait`. + +.. + +.. bpo: 47039 +.. date: 2022-03-17-01-54-13 +.. nonce: 0Yxv0K +.. section: Library + +Normalize ``repr()`` of asyncio future and task objects. + +.. + +.. bpo: 2604 +.. date: 2022-03-16-18-25-19 +.. nonce: jeopdL +.. section: Library + +Fix bug where doctests using globals would fail when run multiple times. + +.. + +.. bpo: 45150 +.. date: 2022-03-16-11-52-52 +.. nonce: kYbIME +.. section: Library + +Add :func:`hashlib.file_digest` helper for efficient hashing of file object. + +.. + +.. bpo: 34861 +.. date: 2022-03-16-08-49-12 +.. nonce: p8ugVg +.. section: Library + +Made cumtime the default sorting key for cProfile + +.. + +.. bpo: 45997 +.. date: 2022-03-15-18-32-12 +.. nonce: 4n2aVU +.. section: Library + +Fix :class:`asyncio.Semaphore` re-aquiring FIFO order. + +.. + +.. bpo: 47022 +.. date: 2022-03-15-09-29-52 +.. nonce: uaEDcI +.. section: Library + +The :mod:`asynchat`, :mod:`asyncore` and :mod:`smtpd` modules have been +deprecated since at least Python 3.6. Their documentation and deprecation +warnings and have now been updated to note they will removed in Python 3.12 +(:pep:`594`). + +.. + +.. bpo: 43253 +.. date: 2022-03-15-07-53-45 +.. nonce: rjdLFj +.. section: Library + +Fix a crash when closing transports where the underlying socket handle is +already invalid on the Proactor event loop. + +.. + +.. bpo: 40280 +.. date: 2022-03-14-09-26-42 +.. nonce: 2-k8TV +.. section: Library + +:func:`select.select` now passes ``NULL`` to ``select`` for each empty +fdset. + +.. + +.. bpo: 47004 +.. date: 2022-03-13-15-04-05 +.. nonce: SyYpxd +.. section: Library + +Apply bugfixes from importlib_metadata 4.11.3, including bugfix for +EntryPoint.extras, which was returning match objects and not the extras +strings. + +.. + +.. bpo: 46998 +.. date: 2022-03-13-08-52-58 +.. nonce: cHh-9O +.. section: Library + +Allow subclassing of :class:`typing.Any`. Patch by Shantanu Jain. + +.. + +.. bpo: 46995 +.. date: 2022-03-12-13-50-42 +.. nonce: 2kdNDg +.. section: Library + +Deprecate missing :meth:`asyncio.Task.set_name` for third-party task +implementations, schedule making it mandatory in Python 3.13. + +.. + +.. bpo: 46994 +.. date: 2022-03-12-12-34-13 +.. nonce: d7hPdz +.. section: Library + +Accept explicit contextvars.Context in :func:`asyncio.create_task` and +:meth:`asyncio.loop.create_task`. + +.. + +.. bpo: 46981 +.. date: 2022-03-12-11-30-42 +.. nonce: ltWCxH +.. section: Library + +``typing.get_args(typing.Tuple[()])`` now returns ``()`` instead of +``((),)``. + +.. + +.. bpo: 46968 +.. date: 2022-03-11-17-56-25 +.. nonce: pPVvNo +.. section: Library + +Add ``os.sysconf_names['SC_MINSIGSTKSZ']``. + +.. + +.. bpo: 46985 +.. date: 2022-03-11-13-34-16 +.. nonce: BgoMr2 +.. section: Library + +Upgrade pip wheel bundled with ensurepip (pip 22.0.4) + +.. + +.. bpo: 46968 +.. date: 2022-03-10-14-51-11 +.. nonce: ym2QxL +.. section: Library + +:mod:`faulthandler`: On Linux 5.14 and newer, dynamically determine size of +signal handler stack size CPython allocates using +``getauxval(AT_MINSIGSTKSZ)``. This changes allows for Python extension's +request to Linux kernel to use AMX_TILE instruction set on Sapphire Rapids +Xeon processor to succeed, unblocking use of the ISA in frameworks. + +.. + +.. bpo: 46917 +.. date: 2022-03-10-14-47-16 +.. nonce: s19zcy +.. section: Library + +The :data:`math.nan` value is now always available. Patch by Victor Stinner. + +.. + +.. bpo: 46955 +.. date: 2022-03-08-22-41-59 +.. nonce: IOoonN +.. section: Library + +Expose :class:`asyncio.base_events.Server` as :class:`asyncio.Server`. Patch +by Stefan Zabka. + +.. + +.. bpo: 23325 +.. date: 2022-03-08-11-34-06 +.. nonce: 3VQnfo +.. section: Library + +The :mod:`signal` module no longer assumes that :const:`~signal.SIG_IGN` and +:const:`~signal.SIG_DFL` are small int singletons. + +.. + +.. bpo: 46932 +.. date: 2022-03-07-20-20-34 +.. nonce: xbarAs +.. section: Library + +Update bundled libexpat to 2.4.7 + +.. + +.. bpo: 46933 +.. date: 2022-03-05-21-51-31 +.. nonce: 6yzWtb +.. section: Library + +The :mod:`pwd` module is now optional. :func:`os.path.expanduser` returns +the path when the :mod:`pwd` module is not available. + +.. + +.. bpo: 40059 +.. date: 2022-02-23-01-11-08 +.. nonce: Iwc9UH +.. section: Library + +:pep:`680`, the :mod:`tomllib` module. Adds support for parsing TOML. + +.. + +.. bpo: 464471 +.. date: 2022-02-21-11-41-23 +.. nonce: fL06TV +.. section: Library + +:func:`asyncio.timeout` and :func:`asyncio.timeout_at` context managers +added. Patch by Tin Tvrtkovi? and Andrew Svetlov. + +.. + +.. bpo: 46805 +.. date: 2022-02-20-23-03-32 +.. nonce: HZ8xWG +.. section: Library + +Added raw datagram socket functions for asyncio: +:meth:`~asyncio.AbstractEventLoop.sock_sendto`, +:meth:`~asyncio.AbstractEventLoop.sock_recvfrom` and +:meth:`~asyncio.AbstractEventLoop.sock_recvfrom_into`. + +.. + +.. bpo: 46644 +.. date: 2022-02-05-22-14-44 +.. nonce: P--1Cz +.. section: Library + +No longer require valid typeforms to be callable. This allows +:data:`typing.Annotated` to wrap :data:`typing.ParamSpecArgs` and +:data:`dataclasses.InitVar`. Patch by Gregory Beauregard. + +.. + +.. bpo: 46581 +.. date: 2022-02-01-11-32-47 +.. nonce: t7Zw65 +.. section: Library + +Brings :class:`ParamSpec` propagation for :class:`GenericAlias` in line with +:class:`Concatenate` (and others). + +.. + +.. bpo: 45413 +.. date: 2022-01-31-15-19-38 +.. nonce: 1vaS0V +.. section: Library + +Define *posix_venv* and *nt_venv* :ref:`sysconfig installation schemes +` to be used for bootstrapping new virtual environments. +Add *venv* sysconfig installation scheme to get the appropriate one of the +above. The schemes are identical to the pre-existing *posix_prefix* and *nt* +install schemes. The :mod:`venv` module now uses the *venv* scheme to create +new virtual environments instead of hardcoding the paths depending only on +the platform. Downstream Python distributors customizing the *posix_prefix* +or *nt* install scheme in a way that is not compatible with the install +scheme used in virtual environments are encouraged not to customize the +*venv* schemes. When Python itself runs in a virtual environment, +:func:`sysconfig.get_default_scheme` and +:func:`sysconfig.get_preferred_scheme` with ``key="prefix"`` returns *venv*. + +.. + +.. bpo: 43224 +.. date: 2022-01-30-22-05-53 +.. nonce: E-eT22 +.. section: Library + +Implement support for PEP 646 in typing.py. + +.. + +.. bpo: 43224 +.. date: 2022-01-30-20-32-40 +.. nonce: zqrQsj +.. section: Library + +Allow unpacking types.GenericAlias objects, e.g. ``*tuple[int, str]``. + +.. + +.. bpo: 46557 +.. date: 2022-01-28-01-23-25 +.. nonce: XSbhyQ +.. section: Library + +Warnings captured by the logging module are now logged without a format +string to prevent systems that group logs by the msg argument from grouping +captured warnings together. + +.. + +.. bpo: 41370 +.. date: 2022-01-27-11-54-16 +.. nonce: gYxCPE +.. section: Library + +:func:`typing.get_type_hints` now supports evaluating strings as forward +references in :ref:`PEP 585 generic aliases `. + +.. + +.. bpo: 46607 +.. date: 2022-01-26-18-30-34 +.. nonce: xnhT4a +.. section: Library + +Add :exc:`DeprecationWarning` to :class:`LegacyInterpolation`, deprecated in +the docstring since Python 3.2. Will be removed in Python 3.13. Use +:class:`BasicInterpolation` or :class:`ExtendedInterpolation` instead. + +.. + +.. bpo: 26120 +.. date: 2022-01-25-15-45-04 +.. nonce: YzrBMO +.. section: Library + +:mod:`pydoc` now excludes __future__ imports from the module's data items. + +.. + +.. bpo: 46480 +.. date: 2022-01-23-16-33-07 +.. nonce: E4jHlh +.. section: Library + +Add :func:`typing.assert_type`. Patch by Jelle Zijlstra. + +.. + +.. bpo: 46421 +.. date: 2022-01-18-01-29-38 +.. nonce: 9LdmNr +.. section: Library + +Fix a unittest issue where if the command was invoked as ``python -m +unittest`` and the filename(s) began with a dot (.), a ``ValueError`` is +returned. + +.. + +.. bpo: 46245 +.. date: 2022-01-03-20-12-14 +.. nonce: 3w4RlA +.. section: Library + +Add optional parameter *dir_fd* in :func:`shutil.rmtree`. + +.. + +.. bpo: 22859 +.. date: 2021-12-29-19-37-49 +.. nonce: AixHW7 +.. section: Library + +:meth:`~unittest.TestProgram.usageExit` is marked deprecated, to be removed +in 3.13. + +.. + +.. bpo: 46170 +.. date: 2021-12-26-14-45-51 +.. nonce: AQ7kSM +.. section: Library + +Improve the error message when you try to subclass an instance of +:class:`typing.NewType`. + +.. + +.. bpo: 40296 +.. date: 2021-12-25-14-13-14 +.. nonce: p0YVGB +.. section: Library + +Fix supporting generic aliases in :mod:`pydoc`. + +.. + +.. bpo: 20392 +.. date: 2021-12-22-12-02-27 +.. nonce: CLAFIp +.. section: Library + +Fix inconsistency with uppercase file extensions in +:meth:`MimeTypes.guess_type`. Patch by Kumar Aditya. + +.. + +.. bpo: 46030 +.. date: 2021-12-10-07-07-47 +.. nonce: UN349J +.. section: Library + +Add ``LOCAL_CREDS``, ``LOCAL_CREDS_PERSISTENT`` and ``SCM_CREDS2`` FreeBSD +constants to the socket module. + +.. + +.. bpo: 44439 +.. date: 2021-11-08-20-27-41 +.. nonce: I_8qro +.. section: Library + +Fix ``.write()`` method of a member file in ``ZipFile``, when the input data +is an object that supports the buffer protocol, the file length may be +wrong. + +.. + +.. bpo: 45171 +.. date: 2021-09-11-16-06-54 +.. nonce: ec597j +.. section: Library + +Fix handling of the ``stacklevel`` argument to logging functions in the +:mod:`logging` module so that it is consistent accross all logging functions +and, as advertised, similar to the ``stacklevel`` argument used in +:meth:`~warnings.warn`. + +.. + +.. bpo: 24959 +.. date: 2021-09-06-15-46-53 +.. nonce: UVFgiO +.. section: Library + +Fix bug where :mod:`unittest` sometimes drops frames from tracebacks of +exceptions raised in tests. + +.. + +.. bpo: 44859 +.. date: 2021-08-10-00-05-53 +.. nonce: 9e9_3V +.. section: Library + +Raise more accurate and :pep:`249` compatible exceptions in :mod:`sqlite3`. + +* Raise :exc:`~sqlite3.InterfaceError` instead of + :exc:`~sqlite3.ProgrammingError` for ``SQLITE_MISUSE`` errors. +* Don't overwrite :exc:`BufferError` with :exc:`ValueError` when conversion to + BLOB fails. +* Raise :exc:`~sqlite3.ProgrammingError` instead of :exc:`~sqlite3.Warning` if + user tries to :meth:`~sqlite3.Cursor.execute()` more than one SQL statement. +* Raise :exc:`~sqlite3.ProgrammingError` instead of :exc:`ValueError` if an SQL + query contains null characters. + +.. + +.. bpo: 44493 +.. date: 2021-07-26-10-46-49 +.. nonce: xp3CRH +.. section: Library + +Add missing terminated NUL in sockaddr_un's length + +This was potentially observable when using non-abstract AF_UNIX datagram +sockets to processes written in another programming language. + +.. + +.. bpo: 41930 +.. date: 2021-06-17-00-02-58 +.. nonce: JS6fsd +.. section: Library + +Add :meth:`~sqlite3.Connection.serialize` and +:meth:`~sqlite3.Connection.deserialize` support to :mod:`sqlite3`. Patch by +Erlend E. Aasland. + +.. + +.. bpo: 33178 +.. date: 2021-04-20-16-48-07 +.. nonce: kSnWwb +.. section: Library + +Added :class:`ctypes.BigEndianUnion` and :class:`ctypes.LittleEndianUnion` +classes, as originally documented in the library docs but not yet +implemented. + +.. + +.. bpo: 43352 +.. date: 2021-03-31-15-22-45 +.. nonce: nSjMuE +.. section: Library + +Add an Barrier object in synchronization primitives of *asyncio* Lib in +order to be consistant with Barrier from *threading* and *multiprocessing* +libs* + +.. + +.. bpo: 35859 +.. date: 2019-03-14-09-08-25 +.. nonce: 8lFdLe +.. section: Library + +:mod:`re` module, fix a few bugs about capturing group. In rare cases, +capturing group gets an incorrect string. Patch by Ma Lin. + +.. + +.. bpo: 45099 +.. date: 2022-03-29-13-25-49 +.. nonce: dagdhx +.. section: Documentation + +Document internal :mod:`asyncio` API. + +.. + +.. bpo: 47126 +.. date: 2022-03-26-12-20-16 +.. nonce: p6_Ovm +.. section: Documentation + +Update PEP URLs to :pep:`676`'s new canonical form. + +.. + +.. bpo: 47040 +.. date: 2022-03-17-13-35-28 +.. nonce: 4Dn48U +.. section: Documentation + +Clarified the old Python versions compatiblity note of +:func:`binascii.crc32` / :func:`zlib.adler32` / :func:`zlib.crc32` +functions. + +.. + +.. bpo: 46033 +.. date: 2022-01-03-18-50-39 +.. nonce: 7WeF0f +.. section: Documentation + +Clarify ``for`` statement execution in its doc. + +.. + +.. bpo: 45790 +.. date: 2021-11-12-11-03-55 +.. nonce: 6yuhe8 +.. section: Documentation + +Adjust inaccurate phrasing in :doc:`../extending/newtypes_tutorial` about +the ``ob_base`` field and the macros used to access its contents. + +.. + +.. bpo: 42340 +.. date: 2020-11-12-21-26-31 +.. nonce: apumUL +.. section: Documentation + +Document that in some circumstances :exc:`KeyboardInterrupt` may cause the +code to enter an inconsistent state. Provided a sample workaround to avoid +it if needed. + +.. + +.. bpo: 41233 +.. date: 2020-07-07-22-54-51 +.. nonce: lyUJ8L +.. section: Documentation + +Link the errnos referenced in ``Doc/library/exceptions.rst`` to their +respective section in ``Doc/library/errno.rst``, and vice versa. Previously +this was only done for EINTR and InterruptedError. Patch by Yan "yyyyyyyan" +Orestes. + +.. + +.. bpo: 47205 +.. date: 2022-04-03-14-38-21 +.. nonce: hbbTnh +.. section: Tests + +Skip test for :func:`~os.sched_getaffinity` and +:func:`~os.sched_setaffinity` error case on FreeBSD. + +.. + +.. bpo: 46126 +.. date: 2022-03-26-11-41-19 +.. nonce: q14Ioy +.. section: Tests + +Restore 'descriptions' when running tests internally. + +.. + +.. bpo: 47104 +.. date: 2022-03-23-22-45-51 +.. nonce: _esUq8 +.. section: Tests + +Rewrite :func:`asyncio.to_thread` tests to use +:class:`unittest.IsolatedAsyncioTestCase`. + +.. + +.. bpo: 40280 +.. date: 2022-03-19-10-25-04 +.. nonce: wBRSel +.. section: Tests + +The test suite is now passing on the Emscripten platform. All fork, socket, +and subprocess-based tests are skipped. + +.. + +.. bpo: 47037 +.. date: 2022-03-16-21-29-30 +.. nonce: xcrLpJ +.. section: Tests + +Skip ``strftime("%4Y")`` feature test on Windows. It can cause an assertion +error in debug builds. + +.. + +.. bpo: 46587 +.. date: 2022-03-14-17-10-35 +.. nonce: ASDsJX +.. section: Tests + +Skip tests if platform's ``strftime`` does not support non-portable glibc +extensions. + +.. + +.. bpo: 47015 +.. date: 2022-03-13-23-43-40 +.. nonce: FjmCsz +.. section: Tests + +A test case for :func:`os.sendfile` is converted from deprecated +:mod:`asyncore` (see :pep:`594`) to :mod:`asyncio`. Patch by Oleg Iarygin. + +.. + +.. bpo: 40280 +.. date: 2022-04-02-17-52-38 +.. nonce: U8Dd0H +.. section: Build + +Add configure option :option:`--enable-wasm-dynamic-linking` to enable +``dlopen`` and MAIN_MODULE / SIDE_MODULE on ``wasm32-emscripten``. + +.. + +.. bpo: 46023 +.. date: 2022-04-01-16-12-53 +.. nonce: 1Z1OcC +.. section: Build + +``makesetup`` now detects and skips all duplicated module definitions. The +first entry wins. + +.. + +.. bpo: 40280 +.. date: 2022-03-24-12-12-35 +.. nonce: eAQWrM +.. section: Build + +Add SOABI ``wasm32-emscripten`` for Emscripten and ``wasm32-wasi`` for WASI +on 32bit WASM as well as ``wasm64`` counter parts. + +.. + +.. bpo: 47032 +.. date: 2022-03-16-00-37-40 +.. nonce: tsS9KE +.. section: Build + +Ensure Windows install builds fail correctly with a non-zero exit code when +part of the build fails. + +.. + +.. bpo: 47024 +.. date: 2022-03-15-09-28-55 +.. nonce: t7-dcu +.. section: Build + +Update OpenSSL to 1.1.1n for macOS installers and all Windows builds. + +.. + +.. bpo: 46996 +.. date: 2022-03-12-18-09-31 +.. nonce: SygzVz +.. section: Build + +The :mod:`tkinter` package now requires Tcl/Tk version 8.5.12 or newer. + +.. + +.. bpo: 46973 +.. date: 2022-03-10-14-30-39 +.. nonce: _LEvnc +.. section: Build + +Add ``regen-configure`` make target to regenerate configure script with +Christian's container image ``quay.io/tiran/cpython_autoconf:269``. + +.. + +.. bpo: 46917 +.. date: 2022-03-10-09-37-05 +.. nonce: fry4aK +.. section: Build + +Building Python now requires support of IEEE 754 floating point numbers. +Patch by Victor Stinner. + +.. + +.. bpo: 45774 +.. date: 2022-03-04-21-24-02 +.. nonce: 9AhC0r +.. section: Build + +``configure`` now verifies that all SQLite C APIs needed for the +:mod:`sqlite3` extension module are found. + +.. + +.. bpo: 47194 +.. date: 2022-04-01-14-57-40 +.. nonce: IB0XL4 +.. section: Windows + +Update ``zlib`` to v1.2.12 to resolve CVE-2018-25032. + +.. + +.. bpo: 47171 +.. date: 2022-03-30-19-55-00 +.. nonce: MbqCWn +.. section: Windows + +Enables installing the :file:`py.exe` launcher on Windows ARM64. + +.. + +.. bpo: 46566 +.. date: 2022-03-23-12-51-46 +.. nonce: 4x4a7e +.. section: Windows + +Upgraded :ref:`launcher` to support a new ``-V:company/tag`` argument for +full :pep:`514` support and to detect ARM64 installs. The ``-64`` suffix on +arguments is deprecated, but still selects any non-32-bit install. Setting +:envvar:`PYLAUNCHER_ALLOW_INSTALL` and specifying a version that is not +installed will attempt to install the requested version from the Microsoft +Store. + +.. + +.. bpo: 47086 +.. date: 2022-03-21-20-45-01 +.. nonce: bIuKlF +.. section: Windows + +The installer for Windows now includes documentation as loose HTML files +rather than a single compiled :file:`.chm` file. + +.. + +.. bpo: 46907 +.. date: 2022-03-13-11-18-41 +.. nonce: YLzxBM +.. section: Windows + +Update Windows installer to use SQLite 3.38.1. + +.. + +.. bpo: 44549 +.. date: 2022-03-07-17-46-40 +.. nonce: SPrGS9 +.. section: Windows + +Update bzip2 to 1.0.8 in Windows builds to mitigate CVE-2016-3189 and +CVE-2019-12900 + +.. + +.. bpo: 46948 +.. date: 2022-03-07-16-34-11 +.. nonce: Ufd4tG +.. section: Windows + +Prevent CVE-2022-26488 by ensuring the Add to PATH option in the Windows +installer uses the correct path when being repaired. + +.. + +.. bpo: 46890 +.. date: 2022-03-17-09-55-02 +.. nonce: GX-3OO +.. section: macOS + +Fix a regression in the setting of ``sys._base_executable`` in framework +builds, and thereby fix a regression in :mod:`venv` virtual environments +with such builds. + +.. + +.. bpo: 46907 +.. date: 2022-03-13-11-11-31 +.. nonce: Ql0z1E +.. section: macOS + +Update macOS installer to SQLite 3.38.1. + +.. + +.. bpo: 40280 +.. date: 2022-04-03-11-47-45 +.. nonce: Q_IJik +.. section: Tools/Demos + +Replace Emscripten's limited shell with Katie Bell's browser-ui REPL from +python-wasm project. + +.. + +.. bpo: 40421 +.. date: 2022-03-25-13-40-46 +.. nonce: wJREl2 +.. section: C API + +Add ``PyFrame_GetBuiltins``, ``PyFrame_GetGenerator`` and +``PyFrame_GetGlobals`` C-API functions to access frame object attributes +safely from C code. + +.. + +.. bpo: 46850 +.. date: 2022-03-22-16-59-34 +.. nonce: lmEKLy +.. section: C API + +Move the private ``_PyFrameEvalFunction`` type, and private +``_PyInterpreterState_GetEvalFrameFunc()`` and +``_PyInterpreterState_SetEvalFrameFunc()`` functions to the internal C API. +The ``_PyFrameEvalFunction`` callback function type now uses the +``_PyInterpreterFrame`` type which is part of the internal C API. Patch by +Victor Stinner. + +.. + +.. bpo: 46850 +.. date: 2022-03-22-16-48-02 +.. nonce: 7M5dO7 +.. section: C API + +Move the private undocumented ``_PyEval_EvalFrameDefault()`` function to the +internal C API. The function now uses the ``_PyInterpreterFrame`` type which +is part of the internal C API. Patch by Victor Stinner. + +.. + +.. bpo: 46850 +.. date: 2022-03-21-02-26-27 +.. nonce: hU3c-O +.. section: C API + +Remove the private undocumented function ``_PyEval_CallTracing()`` from the +C API. Call the public :func:`sys.call_tracing` function instead. Patch by +Victor Stinner. + +.. + +.. bpo: 46850 +.. date: 2022-03-21-01-30-14 +.. nonce: Tfxde5 +.. section: C API + +Remove the private undocumented function +``_PyEval_GetCoroutineOriginTrackingDepth()`` from the C API. Call the +public :func:`sys.get_coroutine_origin_tracking_depth` function instead. +Patch by Victor Stinner. + +.. + +.. bpo: 46850 +.. date: 2022-03-21-00-41-29 +.. nonce: rOt771 +.. section: C API + +Remove the following private undocumented functions from the C API: + +* ``_PyEval_GetAsyncGenFirstiter()`` +* ``_PyEval_GetAsyncGenFinalizer()`` +* ``_PyEval_SetAsyncGenFirstiter()`` +* ``_PyEval_SetAsyncGenFinalizer()`` + +Call the public :func:`sys.get_asyncgen_hooks` and +:func:`sys.set_asyncgen_hooks` functions instead. Patch by Victor Stinner. + +.. + +.. bpo: 46987 +.. date: 2022-03-12-18-37-06 +.. nonce: LWcwyN +.. section: C API + +Remove private functions ``_PySys_GetObjectId()`` and +``_PySys_SetObjectId()``. Patch by Dong-hee Na. + +.. + +.. bpo: 46906 +.. date: 2022-03-03-11-12-33 +.. nonce: -olyBI +.. section: C API + +Add new functions to pack and unpack C double (serialize and deserialize): +:c:func:`PyFloat_Pack2`, :c:func:`PyFloat_Pack4`, :c:func:`PyFloat_Pack8`, +:c:func:`PyFloat_Unpack2`, :c:func:`PyFloat_Unpack4` and +:c:func:`PyFloat_Unpack8`. Patch by Victor Stinner. diff --git a/Misc/NEWS.d/next/Build/2022-03-04-21-24-02.bpo-45774.9AhC0r.rst b/Misc/NEWS.d/next/Build/2022-03-04-21-24-02.bpo-45774.9AhC0r.rst deleted file mode 100644 index 0bfc9862ef5c7..0000000000000 --- a/Misc/NEWS.d/next/Build/2022-03-04-21-24-02.bpo-45774.9AhC0r.rst +++ /dev/null @@ -1,2 +0,0 @@ -``configure`` now verifies that all SQLite C APIs needed for the -:mod:`sqlite3` extension module are found. diff --git a/Misc/NEWS.d/next/Build/2022-03-10-09-37-05.bpo-46917.fry4aK.rst b/Misc/NEWS.d/next/Build/2022-03-10-09-37-05.bpo-46917.fry4aK.rst deleted file mode 100644 index fbb7891cae1a5..0000000000000 --- a/Misc/NEWS.d/next/Build/2022-03-10-09-37-05.bpo-46917.fry4aK.rst +++ /dev/null @@ -1,2 +0,0 @@ -Building Python now requires support of IEEE 754 floating point numbers. -Patch by Victor Stinner. diff --git a/Misc/NEWS.d/next/Build/2022-03-10-14-30-39.bpo-46973._LEvnc.rst b/Misc/NEWS.d/next/Build/2022-03-10-14-30-39.bpo-46973._LEvnc.rst deleted file mode 100644 index 62d71d15fdcd2..0000000000000 --- a/Misc/NEWS.d/next/Build/2022-03-10-14-30-39.bpo-46973._LEvnc.rst +++ /dev/null @@ -1,2 +0,0 @@ -Add ``regen-configure`` make target to regenerate configure script with -Christian's container image ``quay.io/tiran/cpython_autoconf:269``. diff --git a/Misc/NEWS.d/next/Build/2022-03-12-18-09-31.bpo-46996.SygzVz.rst b/Misc/NEWS.d/next/Build/2022-03-12-18-09-31.bpo-46996.SygzVz.rst deleted file mode 100644 index 08138f2e3e935..0000000000000 --- a/Misc/NEWS.d/next/Build/2022-03-12-18-09-31.bpo-46996.SygzVz.rst +++ /dev/null @@ -1 +0,0 @@ -The :mod:`tkinter` package now requires Tcl/Tk version 8.5.12 or newer. diff --git a/Misc/NEWS.d/next/Build/2022-03-15-09-28-55.bpo-47024.t7-dcu.rst b/Misc/NEWS.d/next/Build/2022-03-15-09-28-55.bpo-47024.t7-dcu.rst deleted file mode 100644 index 1035cbab1ba61..0000000000000 --- a/Misc/NEWS.d/next/Build/2022-03-15-09-28-55.bpo-47024.t7-dcu.rst +++ /dev/null @@ -1 +0,0 @@ -Update OpenSSL to 1.1.1n for macOS installers and all Windows builds. diff --git a/Misc/NEWS.d/next/Build/2022-03-16-00-37-40.bpo-47032.tsS9KE.rst b/Misc/NEWS.d/next/Build/2022-03-16-00-37-40.bpo-47032.tsS9KE.rst deleted file mode 100644 index 4f2f1c8d0471a..0000000000000 --- a/Misc/NEWS.d/next/Build/2022-03-16-00-37-40.bpo-47032.tsS9KE.rst +++ /dev/null @@ -1,2 +0,0 @@ -Ensure Windows install builds fail correctly with a non-zero exit code when -part of the build fails. diff --git a/Misc/NEWS.d/next/Build/2022-03-24-12-12-35.bpo-40280.eAQWrM.rst b/Misc/NEWS.d/next/Build/2022-03-24-12-12-35.bpo-40280.eAQWrM.rst deleted file mode 100644 index 7c7dc6d24fae4..0000000000000 --- a/Misc/NEWS.d/next/Build/2022-03-24-12-12-35.bpo-40280.eAQWrM.rst +++ /dev/null @@ -1,2 +0,0 @@ -Add SOABI ``wasm32-emscripten`` for Emscripten and ``wasm32-wasi`` for WASI -on 32bit WASM as well as ``wasm64`` counter parts. diff --git a/Misc/NEWS.d/next/Build/2022-04-01-16-12-53.bpo-46023.1Z1OcC.rst b/Misc/NEWS.d/next/Build/2022-04-01-16-12-53.bpo-46023.1Z1OcC.rst deleted file mode 100644 index cb2f7b760e1ec..0000000000000 --- a/Misc/NEWS.d/next/Build/2022-04-01-16-12-53.bpo-46023.1Z1OcC.rst +++ /dev/null @@ -1,2 +0,0 @@ -``makesetup`` now detects and skips all duplicated module definitions. The -first entry wins. diff --git a/Misc/NEWS.d/next/Build/2022-04-02-17-52-38.bpo-40280.U8Dd0H.rst b/Misc/NEWS.d/next/Build/2022-04-02-17-52-38.bpo-40280.U8Dd0H.rst deleted file mode 100644 index 74fe5c7e49b4a..0000000000000 --- a/Misc/NEWS.d/next/Build/2022-04-02-17-52-38.bpo-40280.U8Dd0H.rst +++ /dev/null @@ -1,2 +0,0 @@ -Add configure option :option:`--enable-wasm-dynamic-linking` to enable -``dlopen`` and MAIN_MODULE / SIDE_MODULE on ``wasm32-emscripten``. diff --git a/Misc/NEWS.d/next/C API/2022-03-03-11-12-33.bpo-46906.-olyBI.rst b/Misc/NEWS.d/next/C API/2022-03-03-11-12-33.bpo-46906.-olyBI.rst deleted file mode 100644 index 9f9cbb5e914ab..0000000000000 --- a/Misc/NEWS.d/next/C API/2022-03-03-11-12-33.bpo-46906.-olyBI.rst +++ /dev/null @@ -1,4 +0,0 @@ -Add new functions to pack and unpack C double (serialize and deserialize): -:c:func:`PyFloat_Pack2`, :c:func:`PyFloat_Pack4`, :c:func:`PyFloat_Pack8`, -:c:func:`PyFloat_Unpack2`, :c:func:`PyFloat_Unpack4` and -:c:func:`PyFloat_Unpack8`. Patch by Victor Stinner. diff --git a/Misc/NEWS.d/next/C API/2022-03-12-18-37-06.bpo-46987.LWcwyN.rst b/Misc/NEWS.d/next/C API/2022-03-12-18-37-06.bpo-46987.LWcwyN.rst deleted file mode 100644 index 2c858afdf1f47..0000000000000 --- a/Misc/NEWS.d/next/C API/2022-03-12-18-37-06.bpo-46987.LWcwyN.rst +++ /dev/null @@ -1,2 +0,0 @@ -Remove private functions ``_PySys_GetObjectId()`` and ``_PySys_SetObjectId()``. -Patch by Dong-hee Na. diff --git a/Misc/NEWS.d/next/C API/2022-03-21-00-41-29.bpo-46850.rOt771.rst b/Misc/NEWS.d/next/C API/2022-03-21-00-41-29.bpo-46850.rOt771.rst deleted file mode 100644 index b3740ae740987..0000000000000 --- a/Misc/NEWS.d/next/C API/2022-03-21-00-41-29.bpo-46850.rOt771.rst +++ /dev/null @@ -1,9 +0,0 @@ -Remove the following private undocumented functions from the C API: - -* ``_PyEval_GetAsyncGenFirstiter()`` -* ``_PyEval_GetAsyncGenFinalizer()`` -* ``_PyEval_SetAsyncGenFirstiter()`` -* ``_PyEval_SetAsyncGenFinalizer()`` - -Call the public :func:`sys.get_asyncgen_hooks` and -:func:`sys.set_asyncgen_hooks` functions instead. Patch by Victor Stinner. diff --git a/Misc/NEWS.d/next/C API/2022-03-21-01-30-14.bpo-46850.Tfxde5.rst b/Misc/NEWS.d/next/C API/2022-03-21-01-30-14.bpo-46850.Tfxde5.rst deleted file mode 100644 index 0dc01fe6ea6ce..0000000000000 --- a/Misc/NEWS.d/next/C API/2022-03-21-01-30-14.bpo-46850.Tfxde5.rst +++ /dev/null @@ -1,4 +0,0 @@ -Remove the private undocumented function -``_PyEval_GetCoroutineOriginTrackingDepth()`` from the C API. Call the -public :func:`sys.get_coroutine_origin_tracking_depth` function instead. -Patch by Victor Stinner. diff --git a/Misc/NEWS.d/next/C API/2022-03-21-02-26-27.bpo-46850.hU3c-O.rst b/Misc/NEWS.d/next/C API/2022-03-21-02-26-27.bpo-46850.hU3c-O.rst deleted file mode 100644 index f600ea8ea24f3..0000000000000 --- a/Misc/NEWS.d/next/C API/2022-03-21-02-26-27.bpo-46850.hU3c-O.rst +++ /dev/null @@ -1,3 +0,0 @@ -Remove the private undocumented function ``_PyEval_CallTracing()`` from the -C API. Call the public :func:`sys.call_tracing` function instead. Patch by -Victor Stinner. diff --git a/Misc/NEWS.d/next/C API/2022-03-22-16-48-02.bpo-46850.7M5dO7.rst b/Misc/NEWS.d/next/C API/2022-03-22-16-48-02.bpo-46850.7M5dO7.rst deleted file mode 100644 index 1519ac7890e37..0000000000000 --- a/Misc/NEWS.d/next/C API/2022-03-22-16-48-02.bpo-46850.7M5dO7.rst +++ /dev/null @@ -1,3 +0,0 @@ -Move the private undocumented ``_PyEval_EvalFrameDefault()`` function to the -internal C API. The function now uses the ``_PyInterpreterFrame`` type which is -part of the internal C API. Patch by Victor Stinner. diff --git a/Misc/NEWS.d/next/C API/2022-03-22-16-59-34.bpo-46850.lmEKLy.rst b/Misc/NEWS.d/next/C API/2022-03-22-16-59-34.bpo-46850.lmEKLy.rst deleted file mode 100644 index d32cc3448721e..0000000000000 --- a/Misc/NEWS.d/next/C API/2022-03-22-16-59-34.bpo-46850.lmEKLy.rst +++ /dev/null @@ -1,6 +0,0 @@ -Move the private ``_PyFrameEvalFunction`` type, and private -``_PyInterpreterState_GetEvalFrameFunc()`` and -``_PyInterpreterState_SetEvalFrameFunc()`` functions to the internal C API. The -``_PyFrameEvalFunction`` callback function type now uses the -``_PyInterpreterFrame`` type which is part of the internal C API. Patch by -Victor Stinner. diff --git a/Misc/NEWS.d/next/C API/2022-03-25-13-40-46.bpo-40421.wJREl2.rst b/Misc/NEWS.d/next/C API/2022-03-25-13-40-46.bpo-40421.wJREl2.rst deleted file mode 100644 index 95b7b69347ce4..0000000000000 --- a/Misc/NEWS.d/next/C API/2022-03-25-13-40-46.bpo-40421.wJREl2.rst +++ /dev/null @@ -1,3 +0,0 @@ -Add ``PyFrame_GetBuiltins``, ``PyFrame_GetGenerator`` and -``PyFrame_GetGlobals`` C-API functions to access frame object attributes -safely from C code. diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-01-20-16-48-09.bpo-43224.WDihrT.rst b/Misc/NEWS.d/next/Core and Builtins/2022-01-20-16-48-09.bpo-43224.WDihrT.rst deleted file mode 100644 index 55affb26c1937..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2022-01-20-16-48-09.bpo-43224.WDihrT.rst +++ /dev/null @@ -1 +0,0 @@ -Make grammar changes required for PEP 646. diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-02-01-10-05-27.bpo-43721.-1XAIo.rst b/Misc/NEWS.d/next/Core and Builtins/2022-02-01-10-05-27.bpo-43721.-1XAIo.rst deleted file mode 100644 index cd3df72664823..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2022-02-01-10-05-27.bpo-43721.-1XAIo.rst +++ /dev/null @@ -1 +0,0 @@ -Fix docstrings of :attr:`~property.getter`, :attr:`~property.setter`, and :attr:`~property.deleter` to clarify that they create a new copy of the property. diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-02-25-22-42-30.bpo-46838.RB6kEy.rst b/Misc/NEWS.d/next/Core and Builtins/2022-02-25-22-42-30.bpo-46838.RB6kEy.rst deleted file mode 100644 index aa3cbca0cf9f2..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2022-02-25-22-42-30.bpo-46838.RB6kEy.rst +++ /dev/null @@ -1,2 +0,0 @@ -Improve syntax errors for incorrect function definitions. Patch by Pablo -Galindo diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-03-03-09-08-17.bpo-46881.ckD4tT.rst b/Misc/NEWS.d/next/Core and Builtins/2022-03-03-09-08-17.bpo-46881.ckD4tT.rst deleted file mode 100644 index 88434dd1dba97..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2022-03-03-09-08-17.bpo-46881.ckD4tT.rst +++ /dev/null @@ -1 +0,0 @@ -Statically allocate and initialize the latin1 characters. diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-03-06-10-37-36.bpo-46841.O12Pba.rst b/Misc/NEWS.d/next/Core and Builtins/2022-03-06-10-37-36.bpo-46841.O12Pba.rst deleted file mode 100644 index 835427437c287..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2022-03-06-10-37-36.bpo-46841.O12Pba.rst +++ /dev/null @@ -1,2 +0,0 @@ -Use inline caching for :opcode:`PRECALL` and :opcode:`CALL`, and remove the -internal machinery for managing the (now unused) non-inline caches. diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-03-07-15-54-39.bpo-46841.7wG92r.rst b/Misc/NEWS.d/next/Core and Builtins/2022-03-07-15-54-39.bpo-46841.7wG92r.rst deleted file mode 100644 index f863c75f715ee..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2022-03-07-15-54-39.bpo-46841.7wG92r.rst +++ /dev/null @@ -1,2 +0,0 @@ -Modify :opcode:`STORE_SUBSCR` to use an inline cache entry (rather than its -oparg) as an adaptive counter. diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-03-08-10-50-42.bpo-46944.cnaIK3.rst b/Misc/NEWS.d/next/Core and Builtins/2022-03-08-10-50-42.bpo-46944.cnaIK3.rst deleted file mode 100644 index f940ebbbd1f5c..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2022-03-08-10-50-42.bpo-46944.cnaIK3.rst +++ /dev/null @@ -1 +0,0 @@ -Speed up throwing exception in generator with :const:`METH_FASTCALL` calling convention. Patch by Kumar Aditya. diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-03-11-09-39-01.bpo-39829.mlW3Su.rst b/Misc/NEWS.d/next/Core and Builtins/2022-03-11-09-39-01.bpo-39829.mlW3Su.rst deleted file mode 100644 index 1f3d945188a32..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2022-03-11-09-39-01.bpo-39829.mlW3Su.rst +++ /dev/null @@ -1 +0,0 @@ -Removed the ``__len__()`` call when initializing a list and moved initializing to ``list_extend``. Patch by Jeremiah Pascual. diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-03-12-09-44-31.bpo-46993.-13hGo.rst b/Misc/NEWS.d/next/Core and Builtins/2022-03-12-09-44-31.bpo-46993.-13hGo.rst deleted file mode 100644 index b7f7078856bd5..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2022-03-12-09-44-31.bpo-46993.-13hGo.rst +++ /dev/null @@ -1 +0,0 @@ -Speed up :class:`bytearray` creation from :class:`list` and :class:`tuple` by 40%. Patch by Kumar Aditya. diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-03-12-21-07-21.bpo-46829.cpGoPV.rst b/Misc/NEWS.d/next/Core and Builtins/2022-03-12-21-07-21.bpo-46829.cpGoPV.rst deleted file mode 100644 index 9d260f5b1dddb..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2022-03-12-21-07-21.bpo-46829.cpGoPV.rst +++ /dev/null @@ -1,2 +0,0 @@ -Deprecate passing a message into :meth:`asyncio.Future.cancel` and -:meth:`asyncio.Task.cancel` diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-03-13-21-04-20.bpo-47005.OHBfCc.rst b/Misc/NEWS.d/next/Core and Builtins/2022-03-13-21-04-20.bpo-47005.OHBfCc.rst deleted file mode 100644 index bf8a4f92a1e7f..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2022-03-13-21-04-20.bpo-47005.OHBfCc.rst +++ /dev/null @@ -1 +0,0 @@ -Improve performance of ``bytearray_repeat`` and ``bytearray_irepeat`` by reducing the number of invocations of ``memcpy``. diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-03-14-09-45-10.bpo-47009.ZI05b5.rst b/Misc/NEWS.d/next/Core and Builtins/2022-03-14-09-45-10.bpo-47009.ZI05b5.rst deleted file mode 100644 index 0c65c34d310ec..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2022-03-14-09-45-10.bpo-47009.ZI05b5.rst +++ /dev/null @@ -1 +0,0 @@ -Improved the performance of :meth:`list.append()` and list comprehensions by optimizing for the common case, where no resize is needed. Patch by Dennis Sweeney. diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-03-14-11-15-11.bpo-47012.5L6NoE.rst b/Misc/NEWS.d/next/Core and Builtins/2022-03-14-11-15-11.bpo-47012.5L6NoE.rst deleted file mode 100644 index f85487f8d3ae7..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2022-03-14-11-15-11.bpo-47012.5L6NoE.rst +++ /dev/null @@ -1 +0,0 @@ -Speed up iteration of :class:`bytes` and :class:`bytearray` by 30%. Patch by Kumar Aditya. diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-03-16-11-05-35.bpo-46841.yUoIHg.rst b/Misc/NEWS.d/next/Core and Builtins/2022-03-16-11-05-35.bpo-46841.yUoIHg.rst deleted file mode 100644 index 99fad382d13bb..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2022-03-16-11-05-35.bpo-46841.yUoIHg.rst +++ /dev/null @@ -1,2 +0,0 @@ -Quicken bytecode in-place by storing it as part of the corresponding -``PyCodeObject``. diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-03-16-12-19-25.bpo-46329.9oS0HT.rst b/Misc/NEWS.d/next/Core and Builtins/2022-03-16-12-19-25.bpo-46329.9oS0HT.rst deleted file mode 100644 index 49a1886e88ba1..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2022-03-16-12-19-25.bpo-46329.9oS0HT.rst +++ /dev/null @@ -1,2 +0,0 @@ -Use low bit of ``LOAD_GLOBAL`` to indicate whether to push a ``NULL`` before -the global. Helps streamline the call sequence a bit. diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-03-17-14-22-23.bpo-46968.4gz4NA.rst b/Misc/NEWS.d/next/Core and Builtins/2022-03-17-14-22-23.bpo-46968.4gz4NA.rst deleted file mode 100644 index bef1d0532b098..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2022-03-17-14-22-23.bpo-46968.4gz4NA.rst +++ /dev/null @@ -1,3 +0,0 @@ -Check for the existence of the "sys/auxv.h" header in :mod:`faulthandler` to -avoid compilation problems in systems where this header doesn't exist. Patch -by Pablo Galindo diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-03-17-16-25-57.bpo-47045.xQgHul.rst b/Misc/NEWS.d/next/Core and Builtins/2022-03-17-16-25-57.bpo-47045.xQgHul.rst deleted file mode 100644 index 388258884299e..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2022-03-17-16-25-57.bpo-47045.xQgHul.rst +++ /dev/null @@ -1,3 +0,0 @@ -Remove the ``f_state`` field from the _PyInterpreterFrame struct. Add the -``owner`` field to the _PyInterpreterFrame struct to make ownership explicit -to simplify clearing and deallocing frames and generators. diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-03-17-22-47-29.bpo-47053.QAXk8Q.rst b/Misc/NEWS.d/next/Core and Builtins/2022-03-17-22-47-29.bpo-47053.QAXk8Q.rst deleted file mode 100644 index 097105b1c20f4..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2022-03-17-22-47-29.bpo-47053.QAXk8Q.rst +++ /dev/null @@ -1 +0,0 @@ -Reduce de-optimization in the specialized ``BINARY_OP_INPLACE_ADD_UNICODE`` opcode. diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-03-19-21-50-59.bpo-47070.wPcsQh.rst b/Misc/NEWS.d/next/Core and Builtins/2022-03-19-21-50-59.bpo-47070.wPcsQh.rst deleted file mode 100644 index 568974f251f32..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2022-03-19-21-50-59.bpo-47070.wPcsQh.rst +++ /dev/null @@ -1,3 +0,0 @@ -Improve performance of ``array_inplace_repeat`` by reducing the number of invocations of ``memcpy``. -Refactor the ``repeat`` and inplace ``repeat`` methods of ``array``, ``bytes``, ``bytearray`` -and ``unicodeobject`` to use the common ``_PyBytes_Repeat``. diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-03-22-15-12-28.bpo-42197.SwrrFO.rst b/Misc/NEWS.d/next/Core and Builtins/2022-03-22-15-12-28.bpo-42197.SwrrFO.rst deleted file mode 100644 index d54002a80e44f..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2022-03-22-15-12-28.bpo-42197.SwrrFO.rst +++ /dev/null @@ -1,2 +0,0 @@ -:c:func:`PyFrame_FastToLocalsWithError` and :c:func:`PyFrame_LocalsToFast` are no longer -called during profiling nor tracing. C code can access the ``f_locals`` attribute of :c:type:`PyFrameObject` by calling :c:func:`PyFrame_GetLocals`. diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-03-25-21-51-10.bpo-47120.9YJ-Xw.rst b/Misc/NEWS.d/next/Core and Builtins/2022-03-25-21-51-10.bpo-47120.9YJ-Xw.rst deleted file mode 100644 index 65208c73543d3..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2022-03-25-21-51-10.bpo-47120.9YJ-Xw.rst +++ /dev/null @@ -1,2 +0,0 @@ -Replaced :opcode:`JUMP_ABSOLUTE` by the relative jump :opcode:`JUMP_BACKWARD`. - diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-03-26-12-21-53.bpo-47127.Mh86RB.rst b/Misc/NEWS.d/next/Core and Builtins/2022-03-26-12-21-53.bpo-47127.Mh86RB.rst deleted file mode 100644 index c4ec9774429d4..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2022-03-26-12-21-53.bpo-47127.Mh86RB.rst +++ /dev/null @@ -1 +0,0 @@ -Speed up calls to c functions with keyword arguments by 25% with specialization. Patch by Kumar Aditya. diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-03-26-15-45-57.bpo-47117.60W6GQ.rst b/Misc/NEWS.d/next/Core and Builtins/2022-03-26-15-45-57.bpo-47117.60W6GQ.rst deleted file mode 100644 index 5098ed86d0793..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2022-03-26-15-45-57.bpo-47117.60W6GQ.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix a crash if we fail to decode characters in interactive mode if the -tokenizer buffers are uninitialized. Patch by Pablo Galindo. diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-03-26-16-35-57.bpo-47129.hDg2Vt.rst b/Misc/NEWS.d/next/Core and Builtins/2022-03-26-16-35-57.bpo-47129.hDg2Vt.rst deleted file mode 100644 index 1627aba41d636..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2022-03-26-16-35-57.bpo-47129.hDg2Vt.rst +++ /dev/null @@ -1 +0,0 @@ -Improve error messages in f-string syntax errors concerning empty expressions. diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-03-30-02-36-25.bpo-46775.e3Oxqf.rst b/Misc/NEWS.d/next/Core and Builtins/2022-03-30-02-36-25.bpo-46775.e3Oxqf.rst deleted file mode 100644 index da56ecd89367b..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2022-03-30-02-36-25.bpo-46775.e3Oxqf.rst +++ /dev/null @@ -1,3 +0,0 @@ -Some Windows system error codes(>= 10000) are now mapped into -the correct errno and may now raise a subclass of :exc:`OSError`. -Patch by Dong-hee Na. diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-03-30-13-13-25.bpo-47162.yDJMUm.rst b/Misc/NEWS.d/next/Core and Builtins/2022-03-30-13-13-25.bpo-47162.yDJMUm.rst deleted file mode 100644 index 7ecbfb37cd17b..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2022-03-30-13-13-25.bpo-47162.yDJMUm.rst +++ /dev/null @@ -1,4 +0,0 @@ -WebAssembly cannot deal with bad function pointer casts (different count -or types of arguments). Python can now use call trampolines to mitigate -the problem. Define :c:macro:`PY_CALL_TRAMPOLINE` to enable call -trampolines. diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-03-31-15-37-02.bpo-47182.e_4SsC.rst b/Misc/NEWS.d/next/Core and Builtins/2022-03-31-15-37-02.bpo-47182.e_4SsC.rst deleted file mode 100644 index 08036bc680933..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2022-03-31-15-37-02.bpo-47182.e_4SsC.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix a crash when using a named unicode character like ``"\N{digit nine}"`` -after the main interpreter has been initialized a second time. diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-03-31-15-57-42.bpo-46841.U-25Z6.rst b/Misc/NEWS.d/next/Core and Builtins/2022-03-31-15-57-42.bpo-46841.U-25Z6.rst deleted file mode 100644 index 42711cd40f396..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2022-03-31-15-57-42.bpo-46841.U-25Z6.rst +++ /dev/null @@ -1 +0,0 @@ -Avoid unnecessary allocations when comparing code objects. diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-03-31-21-43-57.bpo-47120.NgxQbA.rst b/Misc/NEWS.d/next/Core and Builtins/2022-03-31-21-43-57.bpo-47120.NgxQbA.rst deleted file mode 100644 index 236ad94795056..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2022-03-31-21-43-57.bpo-47120.NgxQbA.rst +++ /dev/null @@ -1 +0,0 @@ -Replace the absolute jump opcode :opcode:`JUMP_NO_INTERRUPT` by the relative :opcode:`JUMP_BACKWARD_NO_INTERRUPT`. diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-04-01-11-53-59.bpo-47186.RBCPk8.rst b/Misc/NEWS.d/next/Core and Builtins/2022-04-01-11-53-59.bpo-47186.RBCPk8.rst deleted file mode 100644 index 002da2ba3791a..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2022-04-01-11-53-59.bpo-47186.RBCPk8.rst +++ /dev/null @@ -1 +0,0 @@ -Replace :opcode:`JUMP_IF_NOT_EXC_MATCH` by :opcode:`CHECK_EXC_MATCH` + jump. diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-04-02-14-32-21.bpo-47176.kTygYI.rst b/Misc/NEWS.d/next/Core and Builtins/2022-04-02-14-32-21.bpo-47176.kTygYI.rst deleted file mode 100644 index 03fe54a372552..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2022-04-02-14-32-21.bpo-47176.kTygYI.rst +++ /dev/null @@ -1,6 +0,0 @@ -Emscripten builds cannot handle signals in the usual way due to platform -limitations. Python can now handle signals. To use, set -Module.Py_EmscriptenSignalBuffer to be a single byte SharedArrayBuffer and -set Py_EMSCRIPTEN_SIGNAL_HANDLING to 1. Writing a number into the -SharedArrayBuffer will cause the corresponding signal to be raised into the -Python thread. diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-04-04-17-41-10.bpo-47186.aQWoSh.rst b/Misc/NEWS.d/next/Core and Builtins/2022-04-04-17-41-10.bpo-47186.aQWoSh.rst deleted file mode 100644 index 2282c485b29e8..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2022-04-04-17-41-10.bpo-47186.aQWoSh.rst +++ /dev/null @@ -1 +0,0 @@ -Replace :opcode:`JUMP_IF_NOT_EG_MATCH` by :opcode:`CHECK_EG_MATCH` + jump. diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-04-05-11-29-21.bpo-47212.leF4pz.rst b/Misc/NEWS.d/next/Core and Builtins/2022-04-05-11-29-21.bpo-47212.leF4pz.rst deleted file mode 100644 index 8f1f6b6cfbbb8..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2022-04-05-11-29-21.bpo-47212.leF4pz.rst +++ /dev/null @@ -1,3 +0,0 @@ -Raise :exc:`IndentationError` instead of :exc:`SyntaxError` for a bare -``except`` with no following indent. Improve :exc:`SyntaxError` locations for -an un-parenthesized generator used as arguments. Patch by Matthieu Dartiailh. diff --git a/Misc/NEWS.d/next/Documentation/2020-07-07-22-54-51.bpo-41233.lyUJ8L.rst b/Misc/NEWS.d/next/Documentation/2020-07-07-22-54-51.bpo-41233.lyUJ8L.rst deleted file mode 100644 index ea0643aa00ee0..0000000000000 --- a/Misc/NEWS.d/next/Documentation/2020-07-07-22-54-51.bpo-41233.lyUJ8L.rst +++ /dev/null @@ -1 +0,0 @@ -Link the errnos referenced in ``Doc/library/exceptions.rst`` to their respective section in ``Doc/library/errno.rst``, and vice versa. Previously this was only done for EINTR and InterruptedError. Patch by Yan "yyyyyyyan" Orestes. diff --git a/Misc/NEWS.d/next/Documentation/2020-11-12-21-26-31.bpo-42340.apumUL.rst b/Misc/NEWS.d/next/Documentation/2020-11-12-21-26-31.bpo-42340.apumUL.rst deleted file mode 100644 index aa6857497383c..0000000000000 --- a/Misc/NEWS.d/next/Documentation/2020-11-12-21-26-31.bpo-42340.apumUL.rst +++ /dev/null @@ -1,3 +0,0 @@ -Document that in some circumstances :exc:`KeyboardInterrupt` may cause the -code to enter an inconsistent state. Provided a sample workaround to avoid -it if needed. diff --git a/Misc/NEWS.d/next/Documentation/2021-11-12-11-03-55.bpo-45790.6yuhe8.rst b/Misc/NEWS.d/next/Documentation/2021-11-12-11-03-55.bpo-45790.6yuhe8.rst deleted file mode 100644 index 41cf2cb91525f..0000000000000 --- a/Misc/NEWS.d/next/Documentation/2021-11-12-11-03-55.bpo-45790.6yuhe8.rst +++ /dev/null @@ -1,2 +0,0 @@ -Adjust inaccurate phrasing in :doc:`../extending/newtypes_tutorial` about the -``ob_base`` field and the macros used to access its contents. diff --git a/Misc/NEWS.d/next/Documentation/2022-01-03-18-50-39.bpo-46033.7WeF0f.rst b/Misc/NEWS.d/next/Documentation/2022-01-03-18-50-39.bpo-46033.7WeF0f.rst deleted file mode 100644 index a484def239d8b..0000000000000 --- a/Misc/NEWS.d/next/Documentation/2022-01-03-18-50-39.bpo-46033.7WeF0f.rst +++ /dev/null @@ -1 +0,0 @@ -Clarify ``for`` statement execution in its doc. diff --git a/Misc/NEWS.d/next/Documentation/2022-03-17-13-35-28.bpo-47040.4Dn48U.rst b/Misc/NEWS.d/next/Documentation/2022-03-17-13-35-28.bpo-47040.4Dn48U.rst deleted file mode 100644 index e977fb5f59fbc..0000000000000 --- a/Misc/NEWS.d/next/Documentation/2022-03-17-13-35-28.bpo-47040.4Dn48U.rst +++ /dev/null @@ -1,2 +0,0 @@ -Clarified the old Python versions compatiblity note of :func:`binascii.crc32` / -:func:`zlib.adler32` / :func:`zlib.crc32` functions. diff --git a/Misc/NEWS.d/next/Documentation/2022-03-26-12-20-16.bpo-47126.p6_Ovm.rst b/Misc/NEWS.d/next/Documentation/2022-03-26-12-20-16.bpo-47126.p6_Ovm.rst deleted file mode 100644 index 4cbd0154eac4e..0000000000000 --- a/Misc/NEWS.d/next/Documentation/2022-03-26-12-20-16.bpo-47126.p6_Ovm.rst +++ /dev/null @@ -1 +0,0 @@ -Update PEP URLs to :pep:`676`'s new canonical form. diff --git a/Misc/NEWS.d/next/Documentation/2022-03-29-13-25-49.bpo-45099.dagdhx.rst b/Misc/NEWS.d/next/Documentation/2022-03-29-13-25-49.bpo-45099.dagdhx.rst deleted file mode 100644 index 7b38682b452d2..0000000000000 --- a/Misc/NEWS.d/next/Documentation/2022-03-29-13-25-49.bpo-45099.dagdhx.rst +++ /dev/null @@ -1 +0,0 @@ -Document internal :mod:`asyncio` API. diff --git a/Misc/NEWS.d/next/Library/2019-03-14-09-08-25.bpo-35859.8lFdLe.rst b/Misc/NEWS.d/next/Library/2019-03-14-09-08-25.bpo-35859.8lFdLe.rst deleted file mode 100644 index 8c88ef01164e4..0000000000000 --- a/Misc/NEWS.d/next/Library/2019-03-14-09-08-25.bpo-35859.8lFdLe.rst +++ /dev/null @@ -1,2 +0,0 @@ -:mod:`re` module, fix a few bugs about capturing group. In rare cases, -capturing group gets an incorrect string. Patch by Ma Lin. diff --git a/Misc/NEWS.d/next/Library/2021-03-31-15-22-45.bpo-43352.nSjMuE.rst b/Misc/NEWS.d/next/Library/2021-03-31-15-22-45.bpo-43352.nSjMuE.rst deleted file mode 100644 index e53ba28b64099..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-03-31-15-22-45.bpo-43352.nSjMuE.rst +++ /dev/null @@ -1 +0,0 @@ -Add an Barrier object in synchronization primitives of *asyncio* Lib in order to be consistant with Barrier from *threading* and *multiprocessing* libs* diff --git a/Misc/NEWS.d/next/Library/2021-04-20-16-48-07.bpo-33178.kSnWwb.rst b/Misc/NEWS.d/next/Library/2021-04-20-16-48-07.bpo-33178.kSnWwb.rst deleted file mode 100644 index 3646e4a707d6a..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-04-20-16-48-07.bpo-33178.kSnWwb.rst +++ /dev/null @@ -1 +0,0 @@ -Added :class:`ctypes.BigEndianUnion` and :class:`ctypes.LittleEndianUnion` classes, as originally documented in the library docs but not yet implemented. diff --git a/Misc/NEWS.d/next/Library/2021-06-17-00-02-58.bpo-41930.JS6fsd.rst b/Misc/NEWS.d/next/Library/2021-06-17-00-02-58.bpo-41930.JS6fsd.rst deleted file mode 100644 index ce494e7225e22..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-06-17-00-02-58.bpo-41930.JS6fsd.rst +++ /dev/null @@ -1,3 +0,0 @@ -Add :meth:`~sqlite3.Connection.serialize` and -:meth:`~sqlite3.Connection.deserialize` support to :mod:`sqlite3`. Patch by -Erlend E. Aasland. diff --git a/Misc/NEWS.d/next/Library/2021-07-26-10-46-49.bpo-44493.xp3CRH.rst b/Misc/NEWS.d/next/Library/2021-07-26-10-46-49.bpo-44493.xp3CRH.rst deleted file mode 100644 index 390a7222bbf55..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-07-26-10-46-49.bpo-44493.xp3CRH.rst +++ /dev/null @@ -1,3 +0,0 @@ -Add missing terminated NUL in sockaddr_un's length - -This was potentially observable when using non-abstract AF_UNIX datagram sockets to processes written in another programming language. diff --git a/Misc/NEWS.d/next/Library/2021-08-10-00-05-53.bpo-44859.9e9_3V.rst b/Misc/NEWS.d/next/Library/2021-08-10-00-05-53.bpo-44859.9e9_3V.rst deleted file mode 100644 index 07d7eb0bafb62..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-08-10-00-05-53.bpo-44859.9e9_3V.rst +++ /dev/null @@ -1,10 +0,0 @@ -Raise more accurate and :pep:`249` compatible exceptions in :mod:`sqlite3`. - -* Raise :exc:`~sqlite3.InterfaceError` instead of - :exc:`~sqlite3.ProgrammingError` for ``SQLITE_MISUSE`` errors. -* Don't overwrite :exc:`BufferError` with :exc:`ValueError` when conversion to - BLOB fails. -* Raise :exc:`~sqlite3.ProgrammingError` instead of :exc:`~sqlite3.Warning` if - user tries to :meth:`~sqlite3.Cursor.execute()` more than one SQL statement. -* Raise :exc:`~sqlite3.ProgrammingError` instead of :exc:`ValueError` if an SQL - query contains null characters. diff --git a/Misc/NEWS.d/next/Library/2021-09-06-15-46-53.bpo-24959.UVFgiO.rst b/Misc/NEWS.d/next/Library/2021-09-06-15-46-53.bpo-24959.UVFgiO.rst deleted file mode 100644 index b702986f9468a..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-09-06-15-46-53.bpo-24959.UVFgiO.rst +++ /dev/null @@ -1 +0,0 @@ -Fix bug where :mod:`unittest` sometimes drops frames from tracebacks of exceptions raised in tests. diff --git a/Misc/NEWS.d/next/Library/2021-09-11-16-06-54.bpo-45171.ec597j.rst b/Misc/NEWS.d/next/Library/2021-09-11-16-06-54.bpo-45171.ec597j.rst deleted file mode 100644 index eaa3fb2915e52..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-09-11-16-06-54.bpo-45171.ec597j.rst +++ /dev/null @@ -1,4 +0,0 @@ -Fix handling of the ``stacklevel`` argument to logging functions in the -:mod:`logging` module so that it is consistent accross all logging functions -and, as advertised, similar to the ``stacklevel`` argument used in -:meth:`~warnings.warn`. diff --git a/Misc/NEWS.d/next/Library/2021-11-08-20-27-41.bpo-44439.I_8qro.rst b/Misc/NEWS.d/next/Library/2021-11-08-20-27-41.bpo-44439.I_8qro.rst deleted file mode 100644 index f4e562c4236d2..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-11-08-20-27-41.bpo-44439.I_8qro.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix ``.write()`` method of a member file in ``ZipFile``, when the input data is -an object that supports the buffer protocol, the file length may be wrong. diff --git a/Misc/NEWS.d/next/Library/2021-12-10-07-07-47.bpo-46030.UN349J.rst b/Misc/NEWS.d/next/Library/2021-12-10-07-07-47.bpo-46030.UN349J.rst deleted file mode 100644 index 4f91b17963157..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-12-10-07-07-47.bpo-46030.UN349J.rst +++ /dev/null @@ -1 +0,0 @@ -Add ``LOCAL_CREDS``, ``LOCAL_CREDS_PERSISTENT`` and ``SCM_CREDS2`` FreeBSD constants to the socket module. diff --git a/Misc/NEWS.d/next/Library/2021-12-22-12-02-27.bpo-20392.CLAFIp.rst b/Misc/NEWS.d/next/Library/2021-12-22-12-02-27.bpo-20392.CLAFIp.rst deleted file mode 100644 index 8973c4d433174..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-12-22-12-02-27.bpo-20392.CLAFIp.rst +++ /dev/null @@ -1 +0,0 @@ -Fix inconsistency with uppercase file extensions in :meth:`MimeTypes.guess_type`. Patch by Kumar Aditya. diff --git a/Misc/NEWS.d/next/Library/2021-12-25-14-13-14.bpo-40296.p0YVGB.rst b/Misc/NEWS.d/next/Library/2021-12-25-14-13-14.bpo-40296.p0YVGB.rst deleted file mode 100644 index ea469c916b9db..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-12-25-14-13-14.bpo-40296.p0YVGB.rst +++ /dev/null @@ -1 +0,0 @@ -Fix supporting generic aliases in :mod:`pydoc`. diff --git a/Misc/NEWS.d/next/Library/2021-12-26-14-45-51.bpo-46170.AQ7kSM.rst b/Misc/NEWS.d/next/Library/2021-12-26-14-45-51.bpo-46170.AQ7kSM.rst deleted file mode 100644 index 5f266a29ce1cc..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-12-26-14-45-51.bpo-46170.AQ7kSM.rst +++ /dev/null @@ -1 +0,0 @@ -Improve the error message when you try to subclass an instance of :class:`typing.NewType`. diff --git a/Misc/NEWS.d/next/Library/2021-12-29-19-37-49.bpo-22859.AixHW7.rst b/Misc/NEWS.d/next/Library/2021-12-29-19-37-49.bpo-22859.AixHW7.rst deleted file mode 100644 index f6380b0f904e8..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-12-29-19-37-49.bpo-22859.AixHW7.rst +++ /dev/null @@ -1 +0,0 @@ -:meth:`~unittest.TestProgram.usageExit` is marked deprecated, to be removed in 3.13. diff --git a/Misc/NEWS.d/next/Library/2022-01-03-20-12-14.bpo-46245.3w4RlA.rst b/Misc/NEWS.d/next/Library/2022-01-03-20-12-14.bpo-46245.3w4RlA.rst deleted file mode 100644 index 43e8660b2a3d3..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-01-03-20-12-14.bpo-46245.3w4RlA.rst +++ /dev/null @@ -1 +0,0 @@ -Add optional parameter *dir_fd* in :func:`shutil.rmtree`. diff --git a/Misc/NEWS.d/next/Library/2022-01-18-01-29-38.bpo-46421.9LdmNr.rst b/Misc/NEWS.d/next/Library/2022-01-18-01-29-38.bpo-46421.9LdmNr.rst deleted file mode 100644 index 03ff27fd7d1a7..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-01-18-01-29-38.bpo-46421.9LdmNr.rst +++ /dev/null @@ -1,3 +0,0 @@ -Fix a unittest issue where if the command was invoked as ``python -m -unittest`` and the filename(s) began with a dot (.), a ``ValueError`` is -returned. diff --git a/Misc/NEWS.d/next/Library/2022-01-23-16-33-07.bpo-46480.E4jHlh.rst b/Misc/NEWS.d/next/Library/2022-01-23-16-33-07.bpo-46480.E4jHlh.rst deleted file mode 100644 index fd18a8198edea..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-01-23-16-33-07.bpo-46480.E4jHlh.rst +++ /dev/null @@ -1 +0,0 @@ -Add :func:`typing.assert_type`. Patch by Jelle Zijlstra. diff --git a/Misc/NEWS.d/next/Library/2022-01-25-15-45-04.bpo-26120.YzrBMO.rst b/Misc/NEWS.d/next/Library/2022-01-25-15-45-04.bpo-26120.YzrBMO.rst deleted file mode 100644 index bc45b277d8d6f..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-01-25-15-45-04.bpo-26120.YzrBMO.rst +++ /dev/null @@ -1 +0,0 @@ -:mod:`pydoc` now excludes __future__ imports from the module's data items. diff --git a/Misc/NEWS.d/next/Library/2022-01-26-18-30-34.bpo-46607.xnhT4a.rst b/Misc/NEWS.d/next/Library/2022-01-26-18-30-34.bpo-46607.xnhT4a.rst deleted file mode 100644 index e0c7ed0531b57..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-01-26-18-30-34.bpo-46607.xnhT4a.rst +++ /dev/null @@ -1,3 +0,0 @@ -Add :exc:`DeprecationWarning` to :class:`LegacyInterpolation`, deprecated in -the docstring since Python 3.2. Will be removed in Python 3.13. Use -:class:`BasicInterpolation` or :class:`ExtendedInterpolation` instead. diff --git a/Misc/NEWS.d/next/Library/2022-01-27-11-54-16.bpo-41370.gYxCPE.rst b/Misc/NEWS.d/next/Library/2022-01-27-11-54-16.bpo-41370.gYxCPE.rst deleted file mode 100644 index d9ad2af156a4d..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-01-27-11-54-16.bpo-41370.gYxCPE.rst +++ /dev/null @@ -1 +0,0 @@ -:func:`typing.get_type_hints` now supports evaluating strings as forward references in :ref:`PEP 585 generic aliases `. diff --git a/Misc/NEWS.d/next/Library/2022-01-28-01-23-25.bpo-46557.XSbhyQ.rst b/Misc/NEWS.d/next/Library/2022-01-28-01-23-25.bpo-46557.XSbhyQ.rst deleted file mode 100644 index dd7d3f3d6c51f..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-01-28-01-23-25.bpo-46557.XSbhyQ.rst +++ /dev/null @@ -1 +0,0 @@ -Warnings captured by the logging module are now logged without a format string to prevent systems that group logs by the msg argument from grouping captured warnings together. diff --git a/Misc/NEWS.d/next/Library/2022-01-30-20-32-40.bpo-43224.zqrQsj.rst b/Misc/NEWS.d/next/Library/2022-01-30-20-32-40.bpo-43224.zqrQsj.rst deleted file mode 100644 index 55e9412671058..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-01-30-20-32-40.bpo-43224.zqrQsj.rst +++ /dev/null @@ -1 +0,0 @@ -Allow unpacking types.GenericAlias objects, e.g. ``*tuple[int, str]``. diff --git a/Misc/NEWS.d/next/Library/2022-01-30-22-05-53.bpo-43224.E-eT22.rst b/Misc/NEWS.d/next/Library/2022-01-30-22-05-53.bpo-43224.E-eT22.rst deleted file mode 100644 index c248dd7b28778..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-01-30-22-05-53.bpo-43224.E-eT22.rst +++ /dev/null @@ -1 +0,0 @@ -Implement support for PEP 646 in typing.py. diff --git a/Misc/NEWS.d/next/Library/2022-01-31-15-19-38.bpo-45413.1vaS0V.rst b/Misc/NEWS.d/next/Library/2022-01-31-15-19-38.bpo-45413.1vaS0V.rst deleted file mode 100644 index 6daff85781a5d..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-01-31-15-19-38.bpo-45413.1vaS0V.rst +++ /dev/null @@ -1,15 +0,0 @@ -Define *posix_venv* and *nt_venv* -:ref:`sysconfig installation schemes ` -to be used for bootstrapping new virtual environments. -Add *venv* sysconfig installation scheme to get the appropriate one of the above. -The schemes are identical to the pre-existing -*posix_prefix* and *nt* install schemes. -The :mod:`venv` module now uses the *venv* scheme to create new virtual environments -instead of hardcoding the paths depending only on the platform. Downstream -Python distributors customizing the *posix_prefix* or *nt* install -scheme in a way that is not compatible with the install scheme used in -virtual environments are encouraged not to customize the *venv* schemes. -When Python itself runs in a virtual environment, -:func:`sysconfig.get_default_scheme` and -:func:`sysconfig.get_preferred_scheme` with ``key="prefix"`` returns -*venv*. diff --git a/Misc/NEWS.d/next/Library/2022-02-01-11-32-47.bpo-46581.t7Zw65.rst b/Misc/NEWS.d/next/Library/2022-02-01-11-32-47.bpo-46581.t7Zw65.rst deleted file mode 100644 index 1982c1d70093f..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-02-01-11-32-47.bpo-46581.t7Zw65.rst +++ /dev/null @@ -1,2 +0,0 @@ -Brings :class:`ParamSpec` propagation for :class:`GenericAlias` in line with -:class:`Concatenate` (and others). diff --git a/Misc/NEWS.d/next/Library/2022-02-05-22-14-44.bpo-46644.P--1Cz.rst b/Misc/NEWS.d/next/Library/2022-02-05-22-14-44.bpo-46644.P--1Cz.rst deleted file mode 100644 index 25a999fac8d37..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-02-05-22-14-44.bpo-46644.P--1Cz.rst +++ /dev/null @@ -1 +0,0 @@ -No longer require valid typeforms to be callable. This allows :data:`typing.Annotated` to wrap :data:`typing.ParamSpecArgs` and :data:`dataclasses.InitVar`. Patch by Gregory Beauregard. diff --git a/Misc/NEWS.d/next/Library/2022-02-20-23-03-32.bpo-46805.HZ8xWG.rst b/Misc/NEWS.d/next/Library/2022-02-20-23-03-32.bpo-46805.HZ8xWG.rst deleted file mode 100644 index 3c877d5498cd6..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-02-20-23-03-32.bpo-46805.HZ8xWG.rst +++ /dev/null @@ -1,4 +0,0 @@ -Added raw datagram socket functions for asyncio: -:meth:`~asyncio.AbstractEventLoop.sock_sendto`, -:meth:`~asyncio.AbstractEventLoop.sock_recvfrom` and -:meth:`~asyncio.AbstractEventLoop.sock_recvfrom_into`. diff --git a/Misc/NEWS.d/next/Library/2022-02-21-11-41-23.bpo-464471.fL06TV.rst b/Misc/NEWS.d/next/Library/2022-02-21-11-41-23.bpo-464471.fL06TV.rst deleted file mode 100644 index b8a48d658250f..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-02-21-11-41-23.bpo-464471.fL06TV.rst +++ /dev/null @@ -1,2 +0,0 @@ -:func:`asyncio.timeout` and :func:`asyncio.timeout_at` context managers -added. Patch by Tin Tvrtkovi? and Andrew Svetlov. diff --git a/Misc/NEWS.d/next/Library/2022-02-23-01-11-08.bpo-40059.Iwc9UH.rst b/Misc/NEWS.d/next/Library/2022-02-23-01-11-08.bpo-40059.Iwc9UH.rst deleted file mode 100644 index d41ff1304e83b..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-02-23-01-11-08.bpo-40059.Iwc9UH.rst +++ /dev/null @@ -1 +0,0 @@ -:pep:`680`, the :mod:`tomllib` module. Adds support for parsing TOML. diff --git a/Misc/NEWS.d/next/Library/2022-03-05-21-51-31.bpo-46933.6yzWtb.rst b/Misc/NEWS.d/next/Library/2022-03-05-21-51-31.bpo-46933.6yzWtb.rst deleted file mode 100644 index c3d2e6b50c296..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-03-05-21-51-31.bpo-46933.6yzWtb.rst +++ /dev/null @@ -1 +0,0 @@ -The :mod:`pwd` module is now optional. :func:`os.path.expanduser` returns the path when the :mod:`pwd` module is not available. diff --git a/Misc/NEWS.d/next/Library/2022-03-07-20-20-34.bpo-46932.xbarAs.rst b/Misc/NEWS.d/next/Library/2022-03-07-20-20-34.bpo-46932.xbarAs.rst deleted file mode 100644 index 8545c656eab89..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-03-07-20-20-34.bpo-46932.xbarAs.rst +++ /dev/null @@ -1 +0,0 @@ -Update bundled libexpat to 2.4.7 diff --git a/Misc/NEWS.d/next/Library/2022-03-08-11-34-06.bpo-23325.3VQnfo.rst b/Misc/NEWS.d/next/Library/2022-03-08-11-34-06.bpo-23325.3VQnfo.rst deleted file mode 100644 index 0801cbb448225..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-03-08-11-34-06.bpo-23325.3VQnfo.rst +++ /dev/null @@ -1,2 +0,0 @@ -The :mod:`signal` module no longer assumes that :const:`~signal.SIG_IGN` and -:const:`~signal.SIG_DFL` are small int singletons. diff --git a/Misc/NEWS.d/next/Library/2022-03-08-22-41-59.bpo-46955.IOoonN.rst b/Misc/NEWS.d/next/Library/2022-03-08-22-41-59.bpo-46955.IOoonN.rst deleted file mode 100644 index 75fee1240d3cd..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-03-08-22-41-59.bpo-46955.IOoonN.rst +++ /dev/null @@ -1,2 +0,0 @@ -Expose :class:`asyncio.base_events.Server` as :class:`asyncio.Server`. Patch -by Stefan Zabka. diff --git a/Misc/NEWS.d/next/Library/2022-03-10-14-47-16.bpo-46917.s19zcy.rst b/Misc/NEWS.d/next/Library/2022-03-10-14-47-16.bpo-46917.s19zcy.rst deleted file mode 100644 index 3fbd84a428816..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-03-10-14-47-16.bpo-46917.s19zcy.rst +++ /dev/null @@ -1 +0,0 @@ -The :data:`math.nan` value is now always available. Patch by Victor Stinner. diff --git a/Misc/NEWS.d/next/Library/2022-03-10-14-51-11.bpo-46968.ym2QxL.rst b/Misc/NEWS.d/next/Library/2022-03-10-14-51-11.bpo-46968.ym2QxL.rst deleted file mode 100644 index 0da5ae76572ba..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-03-10-14-51-11.bpo-46968.ym2QxL.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`faulthandler`: On Linux 5.14 and newer, dynamically determine size of -signal handler stack size CPython allocates using ``getauxval(AT_MINSIGSTKSZ)``. -This changes allows for Python extension's request to Linux kernel to use -AMX_TILE instruction set on Sapphire Rapids Xeon processor to succeed, -unblocking use of the ISA in frameworks. diff --git a/Misc/NEWS.d/next/Library/2022-03-11-13-34-16.bpo-46985.BgoMr2.rst b/Misc/NEWS.d/next/Library/2022-03-11-13-34-16.bpo-46985.BgoMr2.rst deleted file mode 100644 index 2e08ee837f583..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-03-11-13-34-16.bpo-46985.BgoMr2.rst +++ /dev/null @@ -1 +0,0 @@ -Upgrade pip wheel bundled with ensurepip (pip 22.0.4) diff --git a/Misc/NEWS.d/next/Library/2022-03-11-17-56-25.bpo-46968.pPVvNo.rst b/Misc/NEWS.d/next/Library/2022-03-11-17-56-25.bpo-46968.pPVvNo.rst deleted file mode 100644 index f526fa30cd60e..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-03-11-17-56-25.bpo-46968.pPVvNo.rst +++ /dev/null @@ -1 +0,0 @@ -Add ``os.sysconf_names['SC_MINSIGSTKSZ']``. diff --git a/Misc/NEWS.d/next/Library/2022-03-12-11-30-42.bpo-46981.ltWCxH.rst b/Misc/NEWS.d/next/Library/2022-03-12-11-30-42.bpo-46981.ltWCxH.rst deleted file mode 100644 index 29f7c9376fe36..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-03-12-11-30-42.bpo-46981.ltWCxH.rst +++ /dev/null @@ -1,2 +0,0 @@ -``typing.get_args(typing.Tuple[()])`` now returns ``()`` instead of -``((),)``. diff --git a/Misc/NEWS.d/next/Library/2022-03-12-12-34-13.bpo-46994.d7hPdz.rst b/Misc/NEWS.d/next/Library/2022-03-12-12-34-13.bpo-46994.d7hPdz.rst deleted file mode 100644 index 765936f1efb59..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-03-12-12-34-13.bpo-46994.d7hPdz.rst +++ /dev/null @@ -1,2 +0,0 @@ -Accept explicit contextvars.Context in :func:`asyncio.create_task` and -:meth:`asyncio.loop.create_task`. diff --git a/Misc/NEWS.d/next/Library/2022-03-12-13-50-42.bpo-46995.2kdNDg.rst b/Misc/NEWS.d/next/Library/2022-03-12-13-50-42.bpo-46995.2kdNDg.rst deleted file mode 100644 index 021923f5889a7..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-03-12-13-50-42.bpo-46995.2kdNDg.rst +++ /dev/null @@ -1,2 +0,0 @@ -Deprecate missing :meth:`asyncio.Task.set_name` for third-party task -implementations, schedule making it mandatory in Python 3.13. diff --git a/Misc/NEWS.d/next/Library/2022-03-13-08-52-58.bpo-46998.cHh-9O.rst b/Misc/NEWS.d/next/Library/2022-03-13-08-52-58.bpo-46998.cHh-9O.rst deleted file mode 100644 index 25b82b5370846..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-03-13-08-52-58.bpo-46998.cHh-9O.rst +++ /dev/null @@ -1 +0,0 @@ -Allow subclassing of :class:`typing.Any`. Patch by Shantanu Jain. diff --git a/Misc/NEWS.d/next/Library/2022-03-13-15-04-05.bpo-47004.SyYpxd.rst b/Misc/NEWS.d/next/Library/2022-03-13-15-04-05.bpo-47004.SyYpxd.rst deleted file mode 100644 index 3cb3b212d89e9..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-03-13-15-04-05.bpo-47004.SyYpxd.rst +++ /dev/null @@ -1,3 +0,0 @@ -Apply bugfixes from importlib_metadata 4.11.3, including bugfix for -EntryPoint.extras, which was returning match objects and not the extras -strings. diff --git a/Misc/NEWS.d/next/Library/2022-03-14-09-26-42.bpo-40280.2-k8TV.rst b/Misc/NEWS.d/next/Library/2022-03-14-09-26-42.bpo-40280.2-k8TV.rst deleted file mode 100644 index f27c968623ff5..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-03-14-09-26-42.bpo-40280.2-k8TV.rst +++ /dev/null @@ -1 +0,0 @@ -:func:`select.select` now passes ``NULL`` to ``select`` for each empty fdset. diff --git a/Misc/NEWS.d/next/Library/2022-03-15-07-53-45.bpo-43253.rjdLFj.rst b/Misc/NEWS.d/next/Library/2022-03-15-07-53-45.bpo-43253.rjdLFj.rst deleted file mode 100644 index b9920cb821b35..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-03-15-07-53-45.bpo-43253.rjdLFj.rst +++ /dev/null @@ -1 +0,0 @@ -Fix a crash when closing transports where the underlying socket handle is already invalid on the Proactor event loop. diff --git a/Misc/NEWS.d/next/Library/2022-03-15-09-29-52.bpo-47022.uaEDcI.rst b/Misc/NEWS.d/next/Library/2022-03-15-09-29-52.bpo-47022.uaEDcI.rst deleted file mode 100644 index 0e867b9506484..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-03-15-09-29-52.bpo-47022.uaEDcI.rst +++ /dev/null @@ -1,4 +0,0 @@ -The :mod:`asynchat`, :mod:`asyncore` and :mod:`smtpd` modules have been -deprecated since at least Python 3.6. Their documentation and deprecation -warnings and have now been updated to note they will removed in Python 3.12 -(:pep:`594`). diff --git a/Misc/NEWS.d/next/Library/2022-03-15-18-32-12.bpo-45997.4n2aVU.rst b/Misc/NEWS.d/next/Library/2022-03-15-18-32-12.bpo-45997.4n2aVU.rst deleted file mode 100644 index 40d8504e5a946..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-03-15-18-32-12.bpo-45997.4n2aVU.rst +++ /dev/null @@ -1 +0,0 @@ -Fix :class:`asyncio.Semaphore` re-aquiring FIFO order. diff --git a/Misc/NEWS.d/next/Library/2022-03-16-08-49-12.bpo-34861.p8ugVg.rst b/Misc/NEWS.d/next/Library/2022-03-16-08-49-12.bpo-34861.p8ugVg.rst deleted file mode 100644 index 0277ffbf5508d..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-03-16-08-49-12.bpo-34861.p8ugVg.rst +++ /dev/null @@ -1 +0,0 @@ -Made cumtime the default sorting key for cProfile diff --git a/Misc/NEWS.d/next/Library/2022-03-16-11-52-52.bpo-45150.kYbIME.rst b/Misc/NEWS.d/next/Library/2022-03-16-11-52-52.bpo-45150.kYbIME.rst deleted file mode 100644 index 1c6ea5a8e43bc..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-03-16-11-52-52.bpo-45150.kYbIME.rst +++ /dev/null @@ -1 +0,0 @@ -Add :func:`hashlib.file_digest` helper for efficient hashing of file object. diff --git a/Misc/NEWS.d/next/Library/2022-03-16-18-25-19.bpo-2604.jeopdL.rst b/Misc/NEWS.d/next/Library/2022-03-16-18-25-19.bpo-2604.jeopdL.rst deleted file mode 100644 index c0fd000b2c660..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-03-16-18-25-19.bpo-2604.jeopdL.rst +++ /dev/null @@ -1 +0,0 @@ -Fix bug where doctests using globals would fail when run multiple times. diff --git a/Misc/NEWS.d/next/Library/2022-03-17-01-54-13.bpo-47039.0Yxv0K.rst b/Misc/NEWS.d/next/Library/2022-03-17-01-54-13.bpo-47039.0Yxv0K.rst deleted file mode 100644 index 66785e1617c7c..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-03-17-01-54-13.bpo-47039.0Yxv0K.rst +++ /dev/null @@ -1 +0,0 @@ -Normalize ``repr()`` of asyncio future and task objects. diff --git a/Misc/NEWS.d/next/Library/2022-03-17-19-38-40.bpo-34790.zQIiVJ.rst b/Misc/NEWS.d/next/Library/2022-03-17-19-38-40.bpo-34790.zQIiVJ.rst deleted file mode 100644 index 50a71b5877fc4..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-03-17-19-38-40.bpo-34790.zQIiVJ.rst +++ /dev/null @@ -1 +0,0 @@ -Remove passing coroutine objects to :func:`asyncio.wait`. diff --git a/Misc/NEWS.d/next/Library/2022-03-18-13-30-40.bpo-47061.etLHK5.rst b/Misc/NEWS.d/next/Library/2022-03-18-13-30-40.bpo-47061.etLHK5.rst deleted file mode 100644 index 5445089e53c39..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-03-18-13-30-40.bpo-47061.etLHK5.rst +++ /dev/null @@ -1,5 +0,0 @@ -Deprecate the various modules listed by :pep:`594`: - -aifc, asynchat, asyncore, audioop, cgi, cgitb, chunk, crypt, -imghdr, msilib, nntplib, nis, ossaudiodev, pipes, smtpd, -sndhdr, spwd, sunau, telnetlib, uu, xdrlib diff --git a/Misc/NEWS.d/next/Library/2022-03-18-14-22-38.bpo-47057.n-IHbt.rst b/Misc/NEWS.d/next/Library/2022-03-18-14-22-38.bpo-47057.n-IHbt.rst deleted file mode 100644 index b404b45e7cee4..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-03-18-14-22-38.bpo-47057.n-IHbt.rst +++ /dev/null @@ -1 +0,0 @@ -Use FASTCALL convention for ``FutureIter.throw()`` diff --git a/Misc/NEWS.d/next/Library/2022-03-18-17-25-57.bpo-46382.zQUJ66.rst b/Misc/NEWS.d/next/Library/2022-03-18-17-25-57.bpo-46382.zQUJ66.rst deleted file mode 100644 index 9bec94969cb4c..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-03-18-17-25-57.bpo-46382.zQUJ66.rst +++ /dev/null @@ -1,2 +0,0 @@ -:func:`~dataclasses.dataclass` ``slots=True`` now correctly omits slots already -defined in base classes. Patch by Arie Bovenberg. diff --git a/Misc/NEWS.d/next/Library/2022-03-18-22-46-18.bpo-47062.RNc99_.rst b/Misc/NEWS.d/next/Library/2022-03-18-22-46-18.bpo-47062.RNc99_.rst deleted file mode 100644 index 7d5bfc114a8d1..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-03-18-22-46-18.bpo-47062.RNc99_.rst +++ /dev/null @@ -1 +0,0 @@ -Implement :class:`asyncio.Runner` context manager. diff --git a/Misc/NEWS.d/next/Library/2022-03-19-08-42-57.bpo-433030.UTwRX7.rst b/Misc/NEWS.d/next/Library/2022-03-19-08-42-57.bpo-433030.UTwRX7.rst deleted file mode 100644 index f449306f2fae9..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-03-19-08-42-57.bpo-433030.UTwRX7.rst +++ /dev/null @@ -1,2 +0,0 @@ -Add support of atomic grouping (``(?>...)``) and possessive quantifiers -(``*+``, ``++``, ``?+``, ``{m,n}+``) in :mod:`regular expressions `. diff --git a/Misc/NEWS.d/next/Library/2022-03-19-13-38-29.bpo-39394.7j6WL6.rst b/Misc/NEWS.d/next/Library/2022-03-19-13-38-29.bpo-39394.7j6WL6.rst deleted file mode 100644 index 9285179c9fdca..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-03-19-13-38-29.bpo-39394.7j6WL6.rst +++ /dev/null @@ -1,2 +0,0 @@ -A warning about inline flags not at the start of the regular expression now -contains the position of the flag. diff --git a/Misc/NEWS.d/next/Library/2022-03-19-14-12-23.bpo-47066.we3YFx.rst b/Misc/NEWS.d/next/Library/2022-03-19-14-12-23.bpo-47066.we3YFx.rst deleted file mode 100644 index f28275b0706bf..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-03-19-14-12-23.bpo-47066.we3YFx.rst +++ /dev/null @@ -1,3 +0,0 @@ -Global inline flags (e.g. ``(?i)``) can now only be used at the start of the -regular expressions. Using them not at the start of expression was -deprecated since Python 3.6. diff --git a/Misc/NEWS.d/next/Library/2022-03-19-15-54-41.bpo-38256.FoMbjE.rst b/Misc/NEWS.d/next/Library/2022-03-19-15-54-41.bpo-38256.FoMbjE.rst deleted file mode 100644 index ea1763fca4138..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-03-19-15-54-41.bpo-38256.FoMbjE.rst +++ /dev/null @@ -1,14 +0,0 @@ -Fix :func:`binascii.crc32` when it is compiled to use zlib'c crc32 to -work properly on inputs 4+GiB in length instead of returning the wrong -result. The workaround prior to this was to always feed the function -data in increments smaller than 4GiB or to just call the zlib module -function. - -We also have :func:`binascii.crc32` release the GIL when computing -on larger inputs as :func:`zlib.crc32` and :mod:`hashlib` do. - -This also boosts performance on Windows as it now uses the zlib crc32 -implementation for :func:`binascii.crc32` for a 2-3x speedup. - -That the stdlib has a crc32 API in two modules is a known historical -oddity. This moves us closer to a single implementation behind them. diff --git a/Misc/NEWS.d/next/Library/2022-03-19-19-56-04.bpo-42369.Ok828t.rst b/Misc/NEWS.d/next/Library/2022-03-19-19-56-04.bpo-42369.Ok828t.rst deleted file mode 100644 index 86dc3a0b81b9c..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-03-19-19-56-04.bpo-42369.Ok828t.rst +++ /dev/null @@ -1 +0,0 @@ -Fix thread safety of :meth:`zipfile._SharedFile.tell` to avoid a "zipfile.BadZipFile: Bad CRC-32 for file" exception when reading a :class:`ZipFile` from multiple threads. diff --git a/Misc/NEWS.d/next/Library/2022-03-20-13-00-08.bpo-47000.p8HpG0.rst b/Misc/NEWS.d/next/Library/2022-03-20-13-00-08.bpo-47000.p8HpG0.rst deleted file mode 100644 index f96b6e627ed11..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-03-20-13-00-08.bpo-47000.p8HpG0.rst +++ /dev/null @@ -1 +0,0 @@ -Make :func:`io.text_encoding` returns "utf-8" when UTF-8 mode is enabled. diff --git a/Misc/NEWS.d/next/Library/2022-03-20-15-54-41.bpo-28080.kn35Vk.rst b/Misc/NEWS.d/next/Library/2022-03-20-15-54-41.bpo-28080.kn35Vk.rst deleted file mode 100644 index 08428e63be3d4..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-03-20-15-54-41.bpo-28080.kn35Vk.rst +++ /dev/null @@ -1,4 +0,0 @@ -Add the *metadata_encoding* parameter in the :class:`zipfile.ZipFile` -constructor and the ``--metadata-encoding`` option in the :mod:`zipfile` -CLI to allow reading zipfiles using non-standard codecs to encode the -filenames within the archive. diff --git a/Misc/NEWS.d/next/Library/2022-03-20-17-15-56.bpo-47067.XXLnje.rst b/Misc/NEWS.d/next/Library/2022-03-20-17-15-56.bpo-47067.XXLnje.rst deleted file mode 100644 index 28c0895681c40..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-03-20-17-15-56.bpo-47067.XXLnje.rst +++ /dev/null @@ -1 +0,0 @@ -Optimize calling ``GenericAlias`` objects by using :pep:`590` ``vectorcall`` and by replacing ``PyObject_SetAttrString`` with ``PyObject_SetAttr``. diff --git a/Misc/NEWS.d/next/Library/2022-03-20-22-13-24.bpo-23691.Nc2TrW.rst b/Misc/NEWS.d/next/Library/2022-03-20-22-13-24.bpo-23691.Nc2TrW.rst deleted file mode 100644 index 053a2b2709ee8..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-03-20-22-13-24.bpo-23691.Nc2TrW.rst +++ /dev/null @@ -1 +0,0 @@ -Protect the :func:`re.finditer` iterator from re-entering. diff --git a/Misc/NEWS.d/next/Library/2022-03-21-08-32-19.bpo-42885.LCnTTp.rst b/Misc/NEWS.d/next/Library/2022-03-21-08-32-19.bpo-42885.LCnTTp.rst deleted file mode 100644 index 5f9c1a19de221..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-03-21-08-32-19.bpo-42885.LCnTTp.rst +++ /dev/null @@ -1,3 +0,0 @@ -Optimize :func:`re.search`, :func:`re.split`, :func:`re.findall`, -:func:`re.finditer` and :func:`re.sub` for regular expressions starting with -``\A`` or ``^``. diff --git a/Misc/NEWS.d/next/Library/2022-03-22-19-18-31.bpo-47088.JM1kNI.rst b/Misc/NEWS.d/next/Library/2022-03-22-19-18-31.bpo-47088.JM1kNI.rst deleted file mode 100644 index 10a814e018245..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-03-22-19-18-31.bpo-47088.JM1kNI.rst +++ /dev/null @@ -1,2 +0,0 @@ -Implement :data:`typing.LiteralString`, part of :pep:`675`. Patch by Jelle -Zijlstra. diff --git a/Misc/NEWS.d/next/Library/2022-03-23-10-07-41.bpo-47098.7AN_qp.rst b/Misc/NEWS.d/next/Library/2022-03-23-10-07-41.bpo-47098.7AN_qp.rst deleted file mode 100644 index dbb71bfb0f28b..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-03-23-10-07-41.bpo-47098.7AN_qp.rst +++ /dev/null @@ -1,3 +0,0 @@ -The Keccak Code Package for :mod:`hashlib`'s internal ``_sha3`` module has -been replaced with tiny_sha3. The module is used as fallback when Python is -built without OpenSSL. diff --git a/Misc/NEWS.d/next/Library/2022-03-23-12-07-26.bpo-47095.P3YTrh.rst b/Misc/NEWS.d/next/Library/2022-03-23-12-07-26.bpo-47095.P3YTrh.rst deleted file mode 100644 index 2df0086e0f550..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-03-23-12-07-26.bpo-47095.P3YTrh.rst +++ /dev/null @@ -1,2 +0,0 @@ -:mod:`hashlib`'s internal ``_blake2`` module now prefers ``libb2`` from -https://www.blake2.net/ over Python's vendored copy of blake2. diff --git a/Misc/NEWS.d/next/Library/2022-03-23-13-55-41.bpo-47099.P6quRP.rst b/Misc/NEWS.d/next/Library/2022-03-23-13-55-41.bpo-47099.P6quRP.rst deleted file mode 100644 index fa2c87e941e8b..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-03-23-13-55-41.bpo-47099.P6quRP.rst +++ /dev/null @@ -1,3 +0,0 @@ -Exception chaining is changed from -:func:`Exception.with_traceback`/:func:`sys.exc_info` to :pep:`3134`. -Patch by Oleg Iarygin. diff --git a/Misc/NEWS.d/next/Library/2022-03-23-14-16-38.bpo-47099.2raait.rst b/Misc/NEWS.d/next/Library/2022-03-23-14-16-38.bpo-47099.2raait.rst deleted file mode 100644 index 785e53c123f91..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-03-23-14-16-38.bpo-47099.2raait.rst +++ /dev/null @@ -1,5 +0,0 @@ -All :exc:`URLError` exception messages raised in -:class:`urllib.request.URLopener` now contain a colon between ``ftp error`` -and the rest of the message. Previously, -:func:`~urllib.request.URLopener.open_ftp` missed the colon. Patch by Oleg -Iarygin. diff --git a/Misc/NEWS.d/next/Library/2022-03-23-15-31-02.bpo-47101.rVSld-.rst b/Misc/NEWS.d/next/Library/2022-03-23-15-31-02.bpo-47101.rVSld-.rst deleted file mode 100644 index 1a65024e69fbd..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-03-23-15-31-02.bpo-47101.rVSld-.rst +++ /dev/null @@ -1,4 +0,0 @@ -:const:`hashlib.algorithms_available` now lists only algorithms that are -provided by activated crypto providers on OpenSSL 3.0. Legacy algorithms are -not listed unless the legacy provider has been loaded into the default -OSSL context. diff --git a/Misc/NEWS.d/next/Library/2022-03-25-01-27-25.bpo-39622.ieBIMp.rst b/Misc/NEWS.d/next/Library/2022-03-25-01-27-25.bpo-39622.ieBIMp.rst deleted file mode 100644 index 25c6aa3703ad2..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-03-25-01-27-25.bpo-39622.ieBIMp.rst +++ /dev/null @@ -1 +0,0 @@ -Handle Ctrl+C in asyncio programs to interrupt the main task. diff --git a/Misc/NEWS.d/next/Library/2022-03-26-13-14-43.bpo-47061.QLxbC6.rst b/Misc/NEWS.d/next/Library/2022-03-26-13-14-43.bpo-47061.QLxbC6.rst deleted file mode 100644 index 17180861a88cd..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-03-26-13-14-43.bpo-47061.QLxbC6.rst +++ /dev/null @@ -1 +0,0 @@ -Deprecate the aifc module. diff --git a/Misc/NEWS.d/next/Library/2022-03-27-10-41-24.bpo-14265.OBMlAi.rst b/Misc/NEWS.d/next/Library/2022-03-27-10-41-24.bpo-14265.OBMlAi.rst deleted file mode 100644 index 308ce36c6ba50..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-03-27-10-41-24.bpo-14265.OBMlAi.rst +++ /dev/null @@ -1 +0,0 @@ -Adds the fully qualified test name to unittest output diff --git a/Misc/NEWS.d/next/Library/2022-03-28-13-35-50.bpo-27929.j5mAmV.rst b/Misc/NEWS.d/next/Library/2022-03-28-13-35-50.bpo-27929.j5mAmV.rst deleted file mode 100644 index 4c80a10bc5684..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-03-28-13-35-50.bpo-27929.j5mAmV.rst +++ /dev/null @@ -1,3 +0,0 @@ -Fix :meth:`asyncio.loop.sock_connect` to only resolve names for :const:`socket.AF_INET` or -:const:`socket.AF_INET6` families. Resolution may not make sense for other families, -like :const:`socket.AF_BLUETOOTH` and :const:`socket.AF_UNIX`. diff --git a/Misc/NEWS.d/next/Library/2022-03-28-20-16-37.bpo-4833.2vSUE5.rst b/Misc/NEWS.d/next/Library/2022-03-28-20-16-37.bpo-4833.2vSUE5.rst deleted file mode 100644 index 7696091221cb5..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-03-28-20-16-37.bpo-4833.2vSUE5.rst +++ /dev/null @@ -1 +0,0 @@ -Add :meth:`ZipFile.mkdir` diff --git a/Misc/NEWS.d/next/Library/2022-03-29-19-14-53.bpo-47152.5rl5ZK.rst b/Misc/NEWS.d/next/Library/2022-03-29-19-14-53.bpo-47152.5rl5ZK.rst deleted file mode 100644 index 1e1633daae597..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-03-29-19-14-53.bpo-47152.5rl5ZK.rst +++ /dev/null @@ -1,2 +0,0 @@ -Convert the :mod:`re` module into a package. Deprecate modules ``sre_compile``, -``sre_constants`` and ``sre_parse``. diff --git a/Misc/NEWS.d/next/Library/2022-03-30-01-17-43.bpo-47151.z-nQkR.rst b/Misc/NEWS.d/next/Library/2022-03-30-01-17-43.bpo-47151.z-nQkR.rst deleted file mode 100644 index d4d02459d35de..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-03-30-01-17-43.bpo-47151.z-nQkR.rst +++ /dev/null @@ -1,3 +0,0 @@ -When subprocess tries to use vfork, it now falls back to fork if vfork -returns an error. This allows use in situations where vfork isn't allowed -by the OS kernel. diff --git a/Misc/NEWS.d/next/Library/2022-03-30-18-35-50.bpo-47167.nCNHsB.rst b/Misc/NEWS.d/next/Library/2022-03-30-18-35-50.bpo-47167.nCNHsB.rst deleted file mode 100644 index a37dd25810f2e..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-03-30-18-35-50.bpo-47167.nCNHsB.rst +++ /dev/null @@ -1 +0,0 @@ -Allow overriding a future compliance check in :class:`asyncio.Task`. diff --git a/Misc/NEWS.d/next/Library/2022-04-03-13-19-08.bpo-23689.TFSc3E.rst b/Misc/NEWS.d/next/Library/2022-04-03-13-19-08.bpo-23689.TFSc3E.rst deleted file mode 100644 index 1032087d9b850..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-04-03-13-19-08.bpo-23689.TFSc3E.rst +++ /dev/null @@ -1,2 +0,0 @@ -:mod:`re` module: fix memory leak when a match is terminated by a signal or -memory allocation failure. Patch by Ma Lin. diff --git a/Misc/NEWS.d/next/Library/2022-04-04-08-54-31.bpo-47208.cOh9xZ.rst b/Misc/NEWS.d/next/Library/2022-04-04-08-54-31.bpo-47208.cOh9xZ.rst deleted file mode 100644 index a5da321f77a6a..0000000000000 --- a/Misc/NEWS.d/next/Library/2022-04-04-08-54-31.bpo-47208.cOh9xZ.rst +++ /dev/null @@ -1 +0,0 @@ -Allow vendors to override :const:`CTYPES_MAX_ARGCOUNT`. diff --git a/Misc/NEWS.d/next/Tests/2022-03-13-23-43-40.bpo-47015.FjmCsz.rst b/Misc/NEWS.d/next/Tests/2022-03-13-23-43-40.bpo-47015.FjmCsz.rst deleted file mode 100644 index 12c527ad1f23b..0000000000000 --- a/Misc/NEWS.d/next/Tests/2022-03-13-23-43-40.bpo-47015.FjmCsz.rst +++ /dev/null @@ -1,2 +0,0 @@ -A test case for :func:`os.sendfile` is converted from deprecated -:mod:`asyncore` (see :pep:`594`) to :mod:`asyncio`. Patch by Oleg Iarygin. diff --git a/Misc/NEWS.d/next/Tests/2022-03-14-17-10-35.bpo-46587.ASDsJX.rst b/Misc/NEWS.d/next/Tests/2022-03-14-17-10-35.bpo-46587.ASDsJX.rst deleted file mode 100644 index ebd94abe9ca4c..0000000000000 --- a/Misc/NEWS.d/next/Tests/2022-03-14-17-10-35.bpo-46587.ASDsJX.rst +++ /dev/null @@ -1,2 +0,0 @@ -Skip tests if platform's ``strftime`` does not support non-portable glibc -extensions. diff --git a/Misc/NEWS.d/next/Tests/2022-03-16-21-29-30.bpo-47037.xcrLpJ.rst b/Misc/NEWS.d/next/Tests/2022-03-16-21-29-30.bpo-47037.xcrLpJ.rst deleted file mode 100644 index f4f28d1e9a012..0000000000000 --- a/Misc/NEWS.d/next/Tests/2022-03-16-21-29-30.bpo-47037.xcrLpJ.rst +++ /dev/null @@ -1,2 +0,0 @@ -Skip ``strftime("%4Y")`` feature test on Windows. It can cause an assertion -error in debug builds. diff --git a/Misc/NEWS.d/next/Tests/2022-03-19-10-25-04.bpo-40280.wBRSel.rst b/Misc/NEWS.d/next/Tests/2022-03-19-10-25-04.bpo-40280.wBRSel.rst deleted file mode 100644 index 2572c27a8211a..0000000000000 --- a/Misc/NEWS.d/next/Tests/2022-03-19-10-25-04.bpo-40280.wBRSel.rst +++ /dev/null @@ -1,2 +0,0 @@ -The test suite is now passing on the Emscripten platform. All fork, socket, -and subprocess-based tests are skipped. diff --git a/Misc/NEWS.d/next/Tests/2022-03-23-22-45-51.bpo-47104._esUq8.rst b/Misc/NEWS.d/next/Tests/2022-03-23-22-45-51.bpo-47104._esUq8.rst deleted file mode 100644 index 1369bc227f5f2..0000000000000 --- a/Misc/NEWS.d/next/Tests/2022-03-23-22-45-51.bpo-47104._esUq8.rst +++ /dev/null @@ -1,2 +0,0 @@ -Rewrite :func:`asyncio.to_thread` tests to use -:class:`unittest.IsolatedAsyncioTestCase`. diff --git a/Misc/NEWS.d/next/Tests/2022-03-26-11-41-19.bpo-46126.q14Ioy.rst b/Misc/NEWS.d/next/Tests/2022-03-26-11-41-19.bpo-46126.q14Ioy.rst deleted file mode 100644 index 0877b0e385743..0000000000000 --- a/Misc/NEWS.d/next/Tests/2022-03-26-11-41-19.bpo-46126.q14Ioy.rst +++ /dev/null @@ -1 +0,0 @@ -Restore 'descriptions' when running tests internally. diff --git a/Misc/NEWS.d/next/Tests/2022-04-03-14-38-21.bpo-47205.hbbTnh.rst b/Misc/NEWS.d/next/Tests/2022-04-03-14-38-21.bpo-47205.hbbTnh.rst deleted file mode 100644 index 35fd94421326e..0000000000000 --- a/Misc/NEWS.d/next/Tests/2022-04-03-14-38-21.bpo-47205.hbbTnh.rst +++ /dev/null @@ -1,2 +0,0 @@ -Skip test for :func:`~os.sched_getaffinity` and -:func:`~os.sched_setaffinity` error case on FreeBSD. diff --git a/Misc/NEWS.d/next/Tools-Demos/2022-04-03-11-47-45.bpo-40280.Q_IJik.rst b/Misc/NEWS.d/next/Tools-Demos/2022-04-03-11-47-45.bpo-40280.Q_IJik.rst deleted file mode 100644 index 07a968617117c..0000000000000 --- a/Misc/NEWS.d/next/Tools-Demos/2022-04-03-11-47-45.bpo-40280.Q_IJik.rst +++ /dev/null @@ -1,2 +0,0 @@ -Replace Emscripten's limited shell with Katie Bell's browser-ui REPL from -python-wasm project. diff --git a/Misc/NEWS.d/next/Windows/2022-03-07-16-34-11.bpo-46948.Ufd4tG.rst b/Misc/NEWS.d/next/Windows/2022-03-07-16-34-11.bpo-46948.Ufd4tG.rst deleted file mode 100644 index cfc4827882ded..0000000000000 --- a/Misc/NEWS.d/next/Windows/2022-03-07-16-34-11.bpo-46948.Ufd4tG.rst +++ /dev/null @@ -1,2 +0,0 @@ -Prevent CVE-2022-26488 by ensuring the Add to PATH option in the Windows -installer uses the correct path when being repaired. diff --git a/Misc/NEWS.d/next/Windows/2022-03-07-17-46-40.bpo-44549.SPrGS9.rst b/Misc/NEWS.d/next/Windows/2022-03-07-17-46-40.bpo-44549.SPrGS9.rst deleted file mode 100644 index 0f1ef9af6c617..0000000000000 --- a/Misc/NEWS.d/next/Windows/2022-03-07-17-46-40.bpo-44549.SPrGS9.rst +++ /dev/null @@ -1,2 +0,0 @@ -Update bzip2 to 1.0.8 in Windows builds to mitigate CVE-2016-3189 and -CVE-2019-12900 diff --git a/Misc/NEWS.d/next/Windows/2022-03-13-11-18-41.bpo-46907.YLzxBM.rst b/Misc/NEWS.d/next/Windows/2022-03-13-11-18-41.bpo-46907.YLzxBM.rst deleted file mode 100644 index 7f3b875a59635..0000000000000 --- a/Misc/NEWS.d/next/Windows/2022-03-13-11-18-41.bpo-46907.YLzxBM.rst +++ /dev/null @@ -1 +0,0 @@ -Update Windows installer to use SQLite 3.38.1. diff --git a/Misc/NEWS.d/next/Windows/2022-03-21-20-45-01.bpo-47086.bIuKlF.rst b/Misc/NEWS.d/next/Windows/2022-03-21-20-45-01.bpo-47086.bIuKlF.rst deleted file mode 100644 index b73e1870c6de2..0000000000000 --- a/Misc/NEWS.d/next/Windows/2022-03-21-20-45-01.bpo-47086.bIuKlF.rst +++ /dev/null @@ -1,2 +0,0 @@ -The installer for Windows now includes documentation as loose HTML files -rather than a single compiled :file:`.chm` file. diff --git a/Misc/NEWS.d/next/Windows/2022-03-23-12-51-46.bpo-46566.4x4a7e.rst b/Misc/NEWS.d/next/Windows/2022-03-23-12-51-46.bpo-46566.4x4a7e.rst deleted file mode 100644 index b1822872113f7..0000000000000 --- a/Misc/NEWS.d/next/Windows/2022-03-23-12-51-46.bpo-46566.4x4a7e.rst +++ /dev/null @@ -1,6 +0,0 @@ -Upgraded :ref:`launcher` to support a new ``-V:company/tag`` argument for -full :pep:`514` support and to detect ARM64 installs. The ``-64`` suffix on -arguments is deprecated, but still selects any non-32-bit install. Setting -:envvar:`PYLAUNCHER_ALLOW_INSTALL` and specifying a version that is not -installed will attempt to install the requested version from the Microsoft -Store. diff --git a/Misc/NEWS.d/next/Windows/2022-03-30-19-55-00.bpo-47171.MbqCWn.rst b/Misc/NEWS.d/next/Windows/2022-03-30-19-55-00.bpo-47171.MbqCWn.rst deleted file mode 100644 index d9f1795f1e1f7..0000000000000 --- a/Misc/NEWS.d/next/Windows/2022-03-30-19-55-00.bpo-47171.MbqCWn.rst +++ /dev/null @@ -1 +0,0 @@ -Enables installing the :file:`py.exe` launcher on Windows ARM64. diff --git a/Misc/NEWS.d/next/Windows/2022-04-01-14-57-40.bpo-47194.IB0XL4.rst b/Misc/NEWS.d/next/Windows/2022-04-01-14-57-40.bpo-47194.IB0XL4.rst deleted file mode 100644 index 7e76add45fa95..0000000000000 --- a/Misc/NEWS.d/next/Windows/2022-04-01-14-57-40.bpo-47194.IB0XL4.rst +++ /dev/null @@ -1 +0,0 @@ -Update ``zlib`` to v1.2.12 to resolve CVE-2018-25032. diff --git a/Misc/NEWS.d/next/macOS/2022-03-13-11-11-31.bpo-46907.Ql0z1E.rst b/Misc/NEWS.d/next/macOS/2022-03-13-11-11-31.bpo-46907.Ql0z1E.rst deleted file mode 100644 index 365081f78e22c..0000000000000 --- a/Misc/NEWS.d/next/macOS/2022-03-13-11-11-31.bpo-46907.Ql0z1E.rst +++ /dev/null @@ -1 +0,0 @@ -Update macOS installer to SQLite 3.38.1. diff --git a/Misc/NEWS.d/next/macOS/2022-03-17-09-55-02.bpo-46890.GX-3OO.rst b/Misc/NEWS.d/next/macOS/2022-03-17-09-55-02.bpo-46890.GX-3OO.rst deleted file mode 100644 index a3d7d3e4ede02..0000000000000 --- a/Misc/NEWS.d/next/macOS/2022-03-17-09-55-02.bpo-46890.GX-3OO.rst +++ /dev/null @@ -1,3 +0,0 @@ -Fix a regression in the setting of ``sys._base_executable`` in framework -builds, and thereby fix a regression in :mod:`venv` virtual environments -with such builds. diff --git a/README.rst b/README.rst index d5cf8205c6b88..c0881ada4e2a0 100644 --- a/README.rst +++ b/README.rst @@ -1,4 +1,4 @@ -This is Python version 3.11.0 alpha 6 +This is Python version 3.11.0 alpha 7 ===================================== .. image:: https://github.com/python/cpython/workflows/Tests/badge.svg From webhook-mailer at python.org Tue Apr 5 19:37:23 2022 From: webhook-mailer at python.org (zooba) Date: Tue, 05 Apr 2022 23:37:23 -0000 Subject: [Python-checkins] bpo-47230: Silence compiler warnings on Windows from zlib 1.2.12 (GH-32337) Message-ID: https://github.com/python/cpython/commit/a4c7752f3e00b75fd1e4603458b8c77a9fa3d4cb commit: a4c7752f3e00b75fd1e4603458b8c77a9fa3d4cb branch: 3.9 author: Jeremy Kloth committer: zooba date: 2022-04-06T00:37:16+01:00 summary: bpo-47230: Silence compiler warnings on Windows from zlib 1.2.12 (GH-32337) (cherry picked from commit 944f09adfcc59f54432ac2947cf95f3465d90e1e) Co-authored-by: Jeremy Kloth files: M PCbuild/pythoncore.vcxproj diff --git a/PCbuild/pythoncore.vcxproj b/PCbuild/pythoncore.vcxproj index 9a40905fe01f8..c899a3b901018 100644 --- a/PCbuild/pythoncore.vcxproj +++ b/PCbuild/pythoncore.vcxproj @@ -500,7 +500,9 @@ - + + 4244 + From webhook-mailer at python.org Wed Apr 6 06:56:06 2022 From: webhook-mailer at python.org (zooba) Date: Wed, 06 Apr 2022 10:56:06 -0000 Subject: [Python-checkins] Fix generation of MD5 table at end of Windows release build (GH-32345) Message-ID: https://github.com/python/cpython/commit/35bcf9f3c19dfbb368c08e3aa75fda72a23c030d commit: 35bcf9f3c19dfbb368c08e3aa75fda72a23c030d branch: main author: Steve Dower committer: zooba date: 2022-04-06T11:55:47+01:00 summary: Fix generation of MD5 table at end of Windows release build (GH-32345) files: M .azure-pipelines/windows-release/stage-publish-pythonorg.yml diff --git a/.azure-pipelines/windows-release/stage-publish-pythonorg.yml b/.azure-pipelines/windows-release/stage-publish-pythonorg.yml index e8f12b64e5589..084134e009902 100644 --- a/.azure-pipelines/windows-release/stage-publish-pythonorg.yml +++ b/.azure-pipelines/windows-release/stage-publish-pythonorg.yml @@ -135,7 +135,7 @@ jobs: - powershell: | $failures = 0 - gci "msi\*\*-webinstall.exe" -File | %{ + gci "msi\*\*.exe" -File | %{ $d = mkdir "tests\$($_.BaseName)" -Force gci $d -r -File | del $ic = copy $_ $d -PassThru @@ -155,7 +155,11 @@ jobs: displayName: 'Test layouts' - powershell: | - $hashes = gci doc\htmlhelp\python*.chm, msi\*\*.exe, embed\*.zip | ` + $files = gci -File "msi\*\*.exe", "embed\*.zip" + if ("$(DoCHM)" -ieq "true") { + $files = $files + (gci -File "doc\htmlhelp\python*.chm") + } + $hashes = $files | ` Sort-Object Name | ` Format-Table Name, @{ Label="MD5"; @@ -170,9 +174,13 @@ jobs: - powershell: | "Copying:" - (gci msi\*\python*.asc, doc\htmlhelp\*.asc, embed\*.asc).FullName + $files = gci -File "msi\*\python*.asc", "embed\*.asc" + if ("$(DoCHM)" -ieq "true") { + $files = $files + (gci -File "doc\htmlhelp\*.asc") + } + $files.FullName $d = mkdir "$(Build.ArtifactStagingDirectory)\hashes" -Force - move msi\*\python*.asc, doc\htmlhelp\*.asc, embed\*.asc $d -Force + move $files $d -Force gci msi -Directory | %{ move "msi\$_\*.asc" (mkdir "$d\$_" -Force) } workingDirectory: $(Build.BinariesDirectory) displayName: 'Copy GPG signatures for build' From webhook-mailer at python.org Wed Apr 6 06:56:36 2022 From: webhook-mailer at python.org (zooba) Date: Wed, 06 Apr 2022 10:56:36 -0000 Subject: [Python-checkins] bpo-47103: Copy pgort140.dll into output directory when building PGInstrument on Windows (GH-32083) Message-ID: https://github.com/python/cpython/commit/074da788028c1f1e867dc81698efdcdc263f2288 commit: 074da788028c1f1e867dc81698efdcdc263f2288 branch: main author: Steve Dower committer: zooba date: 2022-04-06T11:56:31+01:00 summary: bpo-47103: Copy pgort140.dll into output directory when building PGInstrument on Windows (GH-32083) files: A Misc/NEWS.d/next/Build/2022-03-23-20-01-16.bpo-47103.b4-00F.rst M Lib/test/test_embed.py M PCbuild/python.vcxproj diff --git a/Lib/test/test_embed.py b/Lib/test/test_embed.py index 7e5e4c144851e..16d1c7dea6b4f 100644 --- a/Lib/test/test_embed.py +++ b/Lib/test/test_embed.py @@ -1206,20 +1206,11 @@ def tmpdir_with_python(self, subdir=None): if MS_WINDOWS: # Copy pythonXY.dll (or pythonXY_d.dll) - ver = sys.version_info - dll = f'python{ver.major}{ver.minor}' - dll3 = f'python{ver.major}' - if debug_build(sys.executable): - dll += '_d' - dll3 += '_d' - dll += '.dll' - dll3 += '.dll' - dll = os.path.join(os.path.dirname(self.test_exe), dll) - dll3 = os.path.join(os.path.dirname(self.test_exe), dll3) - dll_copy = os.path.join(tmpdir, os.path.basename(dll)) - dll3_copy = os.path.join(tmpdir, os.path.basename(dll3)) - shutil.copyfile(dll, dll_copy) - shutil.copyfile(dll3, dll3_copy) + import fnmatch + exedir = os.path.dirname(self.test_exe) + for f in os.listdir(exedir): + if fnmatch.fnmatch(f, '*.dll'): + shutil.copyfile(os.path.join(exedir, f), os.path.join(tmpdir, f)) # Copy Python program exec_copy = os.path.join(tmpdir, os.path.basename(self.test_exe)) diff --git a/Misc/NEWS.d/next/Build/2022-03-23-20-01-16.bpo-47103.b4-00F.rst b/Misc/NEWS.d/next/Build/2022-03-23-20-01-16.bpo-47103.b4-00F.rst new file mode 100644 index 0000000000000..c1e01adce0d26 --- /dev/null +++ b/Misc/NEWS.d/next/Build/2022-03-23-20-01-16.bpo-47103.b4-00F.rst @@ -0,0 +1,2 @@ +Windows ``PGInstrument`` builds now copy a required DLL into the output +directory, making it easier to run the profile stage of a PGO build. diff --git a/PCbuild/python.vcxproj b/PCbuild/python.vcxproj index 11f835aecea7d..d07db3a681505 100644 --- a/PCbuild/python.vcxproj +++ b/PCbuild/python.vcxproj @@ -133,9 +133,6 @@ set PYTHONPATH=$(PySourcePath)Lib - <_PGOPath Condition="$(Configuration) == 'PGInstrument' and $(Platform) == 'Win32'">@set PATH=%PATH%%3B$(VCInstallDir)bin - <_PGOPath Condition="$(Configuration) == 'PGInstrument' and $(Platform) == 'x64'">@set PATH=%PATH%%3B$(VCInstallDir)bin\amd64 - <_PGOPath Condition="$(Configuration) == 'PGInstrument' and $(VC_PGO_RunTime_Dir) != ''">@set PATH=%PATH%%3B$(VC_PGO_RunTime_Dir) <_Content>@rem This script invokes the most recently built Python with all arguments @rem passed through to the interpreter. This file is generated by the @rem build process and any changes *will* be thrown away by the next @@ -145,11 +142,21 @@ set PYTHONPATH=$(PySourcePath)Lib @echo Running $(Configuration)^|$(Platform) interpreter... @setlocal @set PYTHONHOME=$(PySourcePath) -$(_PGOPath) @"$(OutDir)python$(PyDebugExt).exe" %* <_ExistingContent Condition="Exists('$(PySourcePath)python.bat')">$([System.IO.File]::ReadAllText('$(PySourcePath)python.bat')) + + + <_PGORT Include="$(VCToolsInstallDir)bin\Hostx86\x86\pgort140.dll" Condition="$(Platform) == 'Win32'" /> + <_PGORT Include="$(VCToolsInstallDir)bin\Hostx64\x64\pgort140.dll" Condition="$(Platform) == 'x64'" /> + <_PGORT Include="$(VCToolsInstallDir)bin\arm64\pgort140.dll" Condition="$(Platform) == 'ARM64'" /> + + + + + + From webhook-mailer at python.org Wed Apr 6 07:38:53 2022 From: webhook-mailer at python.org (Fidget-Spinner) Date: Wed, 06 Apr 2022 11:38:53 -0000 Subject: [Python-checkins] bpo-47189: What's New in 3.11: Faster CPython (GH-32235) Message-ID: https://github.com/python/cpython/commit/9ffe47df5468a72603f730eae48c2fd4ec615ffa commit: 9ffe47df5468a72603f730eae48c2fd4ec615ffa branch: main author: Ken Jin committer: Fidget-Spinner date: 2022-04-06T19:38:25+08:00 summary: bpo-47189: What's New in 3.11: Faster CPython (GH-32235) Co-authored-by: Kumar Aditya <59607654+kumaraditya303 at users.noreply.github.com> Co-authored-by: Jelle Zijlstra Co-authored-by: Alex Waygood Co-authored-by: Guido van Rossum Co-authored-by: Irit Katriel <1055913+iritkatriel at users.noreply.github.com> files: A Misc/NEWS.d/next/Documentation/2022-04-01-23-56-13.bpo-47189.Nss0Y3.rst M Doc/tutorial/modules.rst M Doc/whatsnew/3.11.rst diff --git a/Doc/tutorial/modules.rst b/Doc/tutorial/modules.rst index a4e1cefb26b71..0d1fadadf83a0 100644 --- a/Doc/tutorial/modules.rst +++ b/Doc/tutorial/modules.rst @@ -211,6 +211,8 @@ directory. This is an error unless the replacement is intended. See section .. % Do we need stuff on zip files etc. ? DUBOIS +.. _tut-pycache: + "Compiled" Python files ----------------------- diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index a2c57eb909ed1..75b455ddf7e30 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -62,6 +62,8 @@ Summary -- Release highlights .. This section singles out the most important changes in Python 3.11. Brevity is key. +- Python 3.11 is up to 10-60% faster than Python 3.10. On average, we measured a + 1.22x speedup on the standard benchmark suite. See `Faster CPython`_ for details. .. PEP-sized items next. @@ -477,13 +479,6 @@ Optimizations almost eliminated when no exception is raised. (Contributed by Mark Shannon in :issue:`40222`.) -* Method calls with keywords are now faster due to bytecode - changes which avoid creating bound method instances. Previously, this - optimization was applied only to method calls with purely positional - arguments. - (Contributed by Ken Jin and Mark Shannon in :issue:`26110`, based on ideas - implemented in PyPy.) - * Pure ASCII strings are now normalized in constant time by :func:`unicodedata.normalize`. (Contributed by Dong-hee Na in :issue:`44987`.) @@ -498,6 +493,223 @@ Optimizations (Contributed by Inada Naoki in :issue:`46845`.) +Faster CPython +============== + +CPython 3.11 is on average `1.22x faster `_ +than CPython 3.10 when measured with the +`pyperformance `_ benchmark suite, +and compiled with GCC on Ubuntu Linux. Depending on your workload, the speedup +could be up to 10-60% faster. + +This project focuses on two major areas in Python: faster startup and faster +runtime. Other optimizations not under this project are listed in `Optimizations`_. + +Faster Startup +-------------- + +Frozen imports / Static code objects +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Python caches bytecode in the :ref:`__pycache__` directory to +speed up module loading. + +Previously in 3.10, Python module execution looked like this: + +.. code-block:: text + + Read __pycache__ -> Unmarshal -> Heap allocated code object -> Evaluate + +In Python 3.11, the core modules essential for Python startup are "frozen". +This means that their code objects (and bytecode) are statically allocated +by the interpreter. This reduces the steps in module execution process to this: + +.. code-block:: text + + Statically allocated code object -> Evaluate + +Interpreter startup is now 10-15% faster in Python 3.11. This has a big +impact for short-running programs using Python. + +(Contributed by Eric Snow, Guido van Rossum and Kumar Aditya in numerous issues.) + + +Faster Runtime +-------------- + +Cheaper, lazy Python frames +~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Python frames are created whenever Python calls a Python function. This frame +holds execution information. The following are new frame optimizations: + +- Streamlined the frame creation process. +- Avoided memory allocation by generously re-using frame space on the C stack. +- Streamlined the internal frame struct to contain only essential information. + Frames previously held extra debugging and memory management information. + +Old-style frame objects are now created only when required by debuggers. For +most user code, no frame objects are created at all. As a result, nearly all +Python functions calls have sped up significantly. We measured a 3-7% speedup +in pyperformance. + +(Contributed by Mark Shannon in :issue:`44590`.) + +.. _inline-calls: + +Inlined Python function calls +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +During a Python function call, Python will call an evaluating C function to +interpret that function's code. This effectively limits pure Python recursion to +what's safe for the C stack. + +In 3.11, when CPython detects Python code calling another Python function, +it sets up a new frame, and "jumps" to the new code inside the new frame. This +avoids calling the C interpreting function altogether. + +Most Python function calls now consume no C stack space. This speeds up +most of such calls. In simple recursive functions like fibonacci or +factorial, a 1.7x speedup was observed. This also means recursive functions +can recurse significantly deeper (if the user increases the recursion limit). +We measured a 1-3% improvement in pyperformance. + +(Contributed by Pablo Galindo and Mark Shannon in :issue:`45256`.) + +PEP 659: Specializing Adaptive Interpreter +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +:pep:`659` is one of the key parts of the faster CPython project. The general +idea is that while Python is a dynamic language, most code has regions where +objects and types rarely change. This concept is known as *type stability*. + +At runtime, Python will try to look for common patterns and type stability +in the executing code. Python will then replace the current operation with a +more specialized one. This specialized operation uses fast paths available only +to those use cases/types, which generally outperform their generic +counterparts. This also brings in another concept called *inline caching*, where +Python caches the results of expensive operations directly in the bytecode. + +The specializer will also combine certain common instruction pairs into one +superinstruction. This reduces the overhead during execution. + +Python will only specialize +when it sees code that is "hot" (executed multiple times). This prevents Python +from wasting time for run-once code. Python can also de-specialize when code is +too dynamic or when the use changes. Specialization is attempted periodically, +and specialization attempts are not too expensive. This allows specialization +to adapt to new circumstances. + +(PEP written by Mark Shannon, with ideas inspired by Stefan Brunthaler. +See :pep:`659` for more information.) + +.. + If I missed out anyone, please add them. + ++---------------+--------------------+-------------------------------------------------------+-------------------+-------------------+ +| Operation | Form | Specialization | Operation speedup | Contributor(s) | +| | | | (up to) | | ++===============+====================+=======================================================+===================+===================+ +| Binary | ``x+x; x*x; x-x;`` | Binary add, multiply and subtract for common types | 10% | Mark Shannon, | +| operations | | such as ``int``, ``float``, and ``str`` take custom | | Dong-hee Na, | +| | | fast paths for their underlying types. | | Brandt Bucher, | +| | | | | Dennis Sweeney | ++---------------+--------------------+-------------------------------------------------------+-------------------+-------------------+ +| Subscript | ``a[i]`` | Subscripting container types such as ``list``, | 10-25% | Irit Katriel, | +| | | ``tuple`` and ``dict`` directly index the underlying | | Mark Shannon | +| | | data structures. | | | +| | | | | | +| | | Subscripting custom ``__getitem__`` | | | +| | | is also inlined similar to :ref:`inline-calls`. | | | ++---------------+--------------------+-------------------------------------------------------+-------------------+-------------------+ +| Store | ``a[i] = z`` | Similar to subscripting specialization above. | 10-25% | Dennis Sweeney | +| subscript | | | | | ++---------------+--------------------+-------------------------------------------------------+-------------------+-------------------+ +| Calls | ``f(arg)`` | Calls to common builtin (C) functions and types such | 20% | Mark Shannon, | +| | ``C(arg)`` | as ``len`` and ``str`` directly call their underlying | | Ken Jin | +| | | C version. This avoids going through the internal | | | +| | | calling convention. | | | +| | | | | | ++---------------+--------------------+-------------------------------------------------------+-------------------+-------------------+ +| Load | ``print`` | The object's index in the globals/builtins namespace | [1]_ | Mark Shannon | +| global | ``len`` | is cached. Loading globals and builtins require | | | +| variable | | zero namespace lookups. | | | ++---------------+--------------------+-------------------------------------------------------+-------------------+-------------------+ +| Load | ``o.attr`` | Similar to loading global variables. The attribute's | [2]_ | Mark Shannon | +| attribute | | index inside the class/object's namespace is cached. | | | +| | | In most cases, attribute loading will require zero | | | +| | | namespace lookups. | | | ++---------------+--------------------+-------------------------------------------------------+-------------------+-------------------+ +| Load | ``o.meth()`` | The actual address of the method is cached. Method | 10-20% | Ken Jin, | +| methods for | | loading now has no namespace lookups -- even for | | Mark Shannon | +| call | | classes with long inheritance chains. | | | ++---------------+--------------------+-------------------------------------------------------+-------------------+-------------------+ +| Store | ``o.attr = z`` | Similar to load attribute optimization. | 2% | Mark Shannon | +| attribute | | | in pyperformance | | ++---------------+--------------------+-------------------------------------------------------+-------------------+-------------------+ +| Unpack | ``*seq`` | Specialized for common containers such as ``list`` | 8% | Brandt Bucher | +| Sequence | | and ``tuple``. Avoids internal calling convention. | | | ++---------------+--------------------+-------------------------------------------------------+-------------------+-------------------+ + +.. [1] A similar optimization already existed since Python 3.8. 3.11 + specializes for more forms and reduces some overhead. + +.. [2] A similar optimization already existed since Python 3.10. + 3.11 specializes for more forms. Furthermore, all attribute loads should + be sped up by :issue:`45947`. + + +Misc +---- + +* Objects now require less memory due to lazily created object namespaces. Their + namespace dictionaries now also share keys more freely. + (Contributed Mark Shannon in :issue:`45340` and :issue:`40116`.) + +* A more concise representation of exceptions in the interpreter reduced the + time required for catching an exception by about 10%. + (Contributed by Irit Katriel in :issue:`45711`.) + +FAQ +--- + +| Q: How should I write my code to utilize these speedups? +| +| A: You don't have to change your code. Write Pythonic code that follows common + best practices. The Faster CPython project optimizes for common code + patterns we observe. +| +| +| Q: Will CPython 3.11 use more memory? +| +| A: Maybe not. We don't expect memory use to exceed 20% more than 3.10. + This is offset by memory optimizations for frame objects and object + dictionaries as mentioned above. +| +| +| Q: I don't see any speedups in my workload. Why? +| +| A: Certain code won't have noticeable benefits. If your code spends most of + its time on I/O operations, or already does most of its + computation in a C extension library like numpy, there won't be significant + speedup. This project currently benefits pure-Python workloads the most. +| +| Furthermore, the pyperformance figures are a geometric mean. Even within the + pyperformance benchmarks, certain benchmarks have slowed down slightly, while + others have sped up by nearly 2x! +| +| +| Q: Is there a JIT compiler? +| +| A: No. We're still exploring other optimizations. + + +About +----- + +Faster CPython explores optimizations for :term:`CPython`. The main team is +funded by Microsoft to work on this full-time. Pablo Galindo Salgado is also +funded by Bloomberg LP to work on the project part-time. Finally, many +contributors are volunteers from the community. + + CPython bytecode changes ======================== diff --git a/Misc/NEWS.d/next/Documentation/2022-04-01-23-56-13.bpo-47189.Nss0Y3.rst b/Misc/NEWS.d/next/Documentation/2022-04-01-23-56-13.bpo-47189.Nss0Y3.rst new file mode 100644 index 0000000000000..8c400841ca1d1 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2022-04-01-23-56-13.bpo-47189.Nss0Y3.rst @@ -0,0 +1,2 @@ +Add a What's New in Python 3.11 entry for the Faster CPython project. +Documentation by Ken Jin and Kumar Aditya. From webhook-mailer at python.org Wed Apr 6 07:40:45 2022 From: webhook-mailer at python.org (JulienPalard) Date: Wed, 06 Apr 2022 11:40:45 -0000 Subject: [Python-checkins] [doc] Add missing backtick. (GH-32355) Message-ID: https://github.com/python/cpython/commit/1d3e743599673e9135d268fb72f0352d8ce23d97 commit: 1d3e743599673e9135d268fb72f0352d8ce23d97 branch: main author: Julien Palard committer: JulienPalard date: 2022-04-06T13:40:37+02:00 summary: [doc] Add missing backtick. (GH-32355) files: M Doc/whatsnew/3.11.rst diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index 75b455ddf7e30..a572ff6981462 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -805,7 +805,7 @@ Deprecated * :class:`configparser.LegacyInterpolation` has been deprecated in the docstring since Python 3.2. It now emits a :exc:`DeprecationWarning` and will be removed in Python 3.13. Use :class:`configparser.BasicInterpolation` or - :class:`configparser.ExtendedInterpolation instead. + :class:`configparser.ExtendedInterpolation` instead. (Contributed by Hugo van Kemenade in :issue:`46607`.) * The :func:`locale.getdefaultlocale` function is deprecated and will be From webhook-mailer at python.org Wed Apr 6 07:58:13 2022 From: webhook-mailer at python.org (vstinner) Date: Wed, 06 Apr 2022 11:58:13 -0000 Subject: [Python-checkins] Revert "bpo-46850: Move _PyEval_EvalFrameDefault() to internal C API (GH-32052)" (GH-32343) Message-ID: https://github.com/python/cpython/commit/2b4f2f5fa4d1123c19bf0643cfa5a4fe8df4175c commit: 2b4f2f5fa4d1123c19bf0643cfa5a4fe8df4175c branch: main author: Victor Stinner committer: vstinner date: 2022-04-06T13:58:07+02:00 summary: Revert "bpo-46850: Move _PyEval_EvalFrameDefault() to internal C API (GH-32052)" (GH-32343) * Revert "bpo-46850: Move _PyInterpreterState_SetEvalFrameFunc() to internal C API (GH-32054)" This reverts commit f877b40e3f7e0d97878884d80fbec879a85ab7e8. * Revert "bpo-46850: Move _PyEval_EvalFrameDefault() to internal C API (GH-32052)" This reverts commit b9a5522dd952125a99ff554f01f311cae25f5a91. files: M Doc/c-api/init.rst M Doc/whatsnew/3.11.rst M Include/cpython/ceval.h M Include/cpython/pystate.h M Include/internal/pycore_ceval.h M Include/internal/pycore_interp.h M Include/internal/pycore_pystate.h diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst index 82c4acebf1bc9..3fda9c3af4d2a 100644 --- a/Doc/c-api/init.rst +++ b/Doc/c-api/init.rst @@ -1228,25 +1228,18 @@ All of the following functions must be called after :c:func:`Py_Initialize`. .. versionadded:: 3.8 -.. c:type:: PyObject* (*_PyFrameEvalFunction)(PyThreadState *tstate, _PyInterpreterFrame *frame, int throwflag) - - Internal C API. +.. c:type:: PyObject* (*_PyFrameEvalFunction)(PyThreadState *tstate, PyFrameObject *frame, int throwflag) Type of a frame evaluation function. The *throwflag* parameter is used by the ``throw()`` method of generators: if non-zero, handle the current exception. - .. versionchanged:: 3.11 - The second parameter type becomes ``_PyInterpreterFrame``. - .. versionchanged:: 3.9 The function now takes a *tstate* parameter. .. c:function:: _PyFrameEvalFunction _PyInterpreterState_GetEvalFrameFunc(PyInterpreterState *interp) - Internal C API. - Get the frame evaluation function. See the :pep:`523` "Adding a frame evaluation API to CPython". @@ -1255,8 +1248,6 @@ All of the following functions must be called after :c:func:`Py_Initialize`. .. c:function:: void _PyInterpreterState_SetEvalFrameFunc(PyInterpreterState *interp, _PyFrameEvalFunction eval_frame) - Internal C API. - Set the frame evaluation function. See the :pep:`523` "Adding a frame evaluation API to CPython". diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index a572ff6981462..d58056394d864 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -1338,17 +1338,6 @@ Porting to Python 3.11 * Distributors are encouraged to build Python with the optimized Blake2 library `libb2`_. -* Move the private undocumented ``_PyEval_EvalFrameDefault()`` function to the - internal C API. The function now uses the ``_PyInterpreterFrame`` type which - is part of the internal C API. - (Contributed by Victor Stinner in :issue:`46850`.) - -* Move the private ``_PyFrameEvalFunction`` type, and private - ``_PyInterpreterState_GetEvalFrameFunc()`` and - ``_PyInterpreterState_SetEvalFrameFunc()`` functions to the internal C API. - The ``_PyFrameEvalFunction`` callback function type now uses the - ``_PyInterpreterFrame`` type which is part of the internal C API. - (Contributed by Victor Stinner in :issue:`46850`.) Deprecated ---------- diff --git a/Include/cpython/ceval.h b/Include/cpython/ceval.h index 65aae2d669a52..9d4eeafb427eb 100644 --- a/Include/cpython/ceval.h +++ b/Include/cpython/ceval.h @@ -15,6 +15,8 @@ PyAPI_FUNC(PyObject *) _PyEval_GetBuiltinId(_Py_Identifier *); flag was set, else return 0. */ PyAPI_FUNC(int) PyEval_MergeCompilerFlags(PyCompilerFlags *cf); +PyAPI_FUNC(PyObject *) _PyEval_EvalFrameDefault(PyThreadState *tstate, struct _PyInterpreterFrame *f, int exc); + PyAPI_FUNC(void) _PyEval_SetSwitchInterval(unsigned long microseconds); PyAPI_FUNC(unsigned long) _PyEval_GetSwitchInterval(void); diff --git a/Include/cpython/pystate.h b/Include/cpython/pystate.h index e346d744b6f12..1af21a2c947d9 100644 --- a/Include/cpython/pystate.h +++ b/Include/cpython/pystate.h @@ -259,6 +259,16 @@ PyAPI_FUNC(PyThreadState *) PyInterpreterState_ThreadHead(PyInterpreterState *); PyAPI_FUNC(PyThreadState *) PyThreadState_Next(PyThreadState *); PyAPI_FUNC(void) PyThreadState_DeleteCurrent(void); +/* Frame evaluation API */ + +typedef PyObject* (*_PyFrameEvalFunction)(PyThreadState *tstate, struct _PyInterpreterFrame *, int); + +PyAPI_FUNC(_PyFrameEvalFunction) _PyInterpreterState_GetEvalFrameFunc( + PyInterpreterState *interp); +PyAPI_FUNC(void) _PyInterpreterState_SetEvalFrameFunc( + PyInterpreterState *interp, + _PyFrameEvalFunction eval_frame); + PyAPI_FUNC(const PyConfig*) _PyInterpreterState_GetConfig(PyInterpreterState *interp); /* Get a copy of the current interpreter configuration. diff --git a/Include/internal/pycore_ceval.h b/Include/internal/pycore_ceval.h index b29b496ee3f58..45d26a37a34c6 100644 --- a/Include/internal/pycore_ceval.h +++ b/Include/internal/pycore_ceval.h @@ -59,11 +59,6 @@ extern PyObject* _PyEval_BuiltinsFromGlobals( PyObject *globals); -PyAPI_FUNC(PyObject *) _PyEval_EvalFrameDefault( - PyThreadState *tstate, - struct _PyInterpreterFrame *frame, - int throwflag); - static inline PyObject* _PyEval_EvalFrame(PyThreadState *tstate, struct _PyInterpreterFrame *frame, int throwflag) { diff --git a/Include/internal/pycore_interp.h b/Include/internal/pycore_interp.h index 592d438bcf1d2..d55627908a28f 100644 --- a/Include/internal/pycore_interp.h +++ b/Include/internal/pycore_interp.h @@ -17,9 +17,9 @@ extern "C" { #include "pycore_dict.h" // struct _Py_dict_state #include "pycore_exceptions.h" // struct _Py_exc_state #include "pycore_floatobject.h" // struct _Py_float_state -#include "pycore_gc.h" // struct _gc_runtime_state #include "pycore_genobject.h" // struct _Py_async_gen_state #include "pycore_gil.h" // struct _gil_runtime_state +#include "pycore_gc.h" // struct _gc_runtime_state #include "pycore_list.h" // struct _Py_list_state #include "pycore_tuple.h" // struct _Py_tuple_state #include "pycore_typeobject.h" // struct type_cache @@ -71,20 +71,6 @@ struct atexit_state { }; -/* Frame evaluation API (PEP 523) */ - -typedef PyObject* (*_PyFrameEvalFunction) ( - PyThreadState *tstate, - struct _PyInterpreterFrame *frame, - int throwflag); - -PyAPI_FUNC(_PyFrameEvalFunction) _PyInterpreterState_GetEvalFrameFunc( - PyInterpreterState *interp); -PyAPI_FUNC(void) _PyInterpreterState_SetEvalFrameFunc( - PyInterpreterState *interp, - _PyFrameEvalFunction eval_frame); - - /* interpreter state */ /* PyInterpreterState holds the global state for one of the runtime's diff --git a/Include/internal/pycore_pystate.h b/Include/internal/pycore_pystate.h index c463347dbf06a..c4bc53c707fda 100644 --- a/Include/internal/pycore_pystate.h +++ b/Include/internal/pycore_pystate.h @@ -8,7 +8,7 @@ extern "C" { # error "this header requires Py_BUILD_CORE define" #endif -#include "pycore_runtime.h" // _PyRuntime +#include "pycore_runtime.h" /* PyRuntimeState */ /* Check if the current thread is the main thread. From webhook-mailer at python.org Wed Apr 6 08:33:41 2022 From: webhook-mailer at python.org (tiran) Date: Wed, 06 Apr 2022 12:33:41 -0000 Subject: [Python-checkins] bpo-40280: WASM defaults to no dynamic linking (GH-32360) Message-ID: https://github.com/python/cpython/commit/765f6dee0fcf55c6ea258c2be4cc9dfb1b014f60 commit: 765f6dee0fcf55c6ea258c2be4cc9dfb1b014f60 branch: main author: Christian Heimes committer: tiran date: 2022-04-06T14:33:31+02:00 summary: bpo-40280: WASM defaults to no dynamic linking (GH-32360) files: M configure M configure.ac diff --git a/configure b/configure index 69b12309de578..e10462cf4dbfc 100755 --- a/configure +++ b/configure @@ -6311,7 +6311,15 @@ esac else - enable_wasm_dynamic_linking=missing + case $ac_sys_system in #( + Emscripten) : + enable_wasm_dynamic_linking=no ;; #( + WASI) : + enable_wasm_dynamic_linking=no ;; #( + *) : + enable_wasm_dynamic_linking=missing + ;; +esac fi diff --git a/configure.ac b/configure.ac index 5860595b752c8..4f256eeff5d7b 100644 --- a/configure.ac +++ b/configure.ac @@ -1122,7 +1122,11 @@ AC_ARG_ENABLE([wasm-dynamic-linking], [AC_MSG_ERROR([--enable-wasm-dynamic-linking only applies to Emscripten and WASI])] ) ], [ - enable_wasm_dynamic_linking=missing + AS_CASE([$ac_sys_system], + [Emscripten], [enable_wasm_dynamic_linking=no], + [WASI], [enable_wasm_dynamic_linking=no], + [enable_wasm_dynamic_linking=missing] + ) ]) AC_MSG_RESULT([$enable_wasm_dynamic_linking]) From webhook-mailer at python.org Wed Apr 6 09:12:46 2022 From: webhook-mailer at python.org (vstinner) Date: Wed, 06 Apr 2022 13:12:46 -0000 Subject: [Python-checkins] bpo-40421: test_capi uses assertEqual(), not assertEquals() (GH-32361) Message-ID: https://github.com/python/cpython/commit/14a9b4895b61bcd46ed968c43c5eec27670a0aac commit: 14a9b4895b61bcd46ed968c43c5eec27670a0aac branch: main author: Victor Stinner committer: vstinner date: 2022-04-06T15:12:38+02:00 summary: bpo-40421: test_capi uses assertEqual(), not assertEquals() (GH-32361) unittest.TestCase.assertEquals() alias is depracated. Fix the warning: Lib/test/test_capi.py:1100: DeprecationWarning: Please use assertEqual instead. self.assertEquals(frame.f_locals, _testcapi.frame_getlocals(frame)) files: M Lib/test/test_capi.py diff --git a/Lib/test/test_capi.py b/Lib/test/test_capi.py index 238acf94526a3..714a2d98e9fb5 100644 --- a/Lib/test/test_capi.py +++ b/Lib/test/test_capi.py @@ -1097,7 +1097,7 @@ def getgenframe(self): def test_frame_getters(self): frame = self.getframe() - self.assertEquals(frame.f_locals, _testcapi.frame_getlocals(frame)) + self.assertEqual(frame.f_locals, _testcapi.frame_getlocals(frame)) self.assertIs(frame.f_globals, _testcapi.frame_getglobals(frame)) self.assertIs(frame.f_builtins, _testcapi.frame_getbuiltins(frame)) From webhook-mailer at python.org Wed Apr 6 10:51:00 2022 From: webhook-mailer at python.org (encukou) Date: Wed, 06 Apr 2022 14:51:00 -0000 Subject: [Python-checkins] bpo-47115: Document which parts of structs are in limited API/stable ABI (GH-32196) Message-ID: https://github.com/python/cpython/commit/d79f118d044e9b4244b5dfda35448d39202d7f56 commit: d79f118d044e9b4244b5dfda35448d39202d7f56 branch: main author: Petr Viktorin committer: encukou date: 2022-04-06T16:50:45+02:00 summary: bpo-47115: Document which parts of structs are in limited API/stable ABI (GH-32196) Co-authored-by: Erlend Egeberg Aasland files: A Misc/NEWS.d/next/Documentation/2022-03-30-17-08-12.bpo-47115.R3wt3i.rst M Doc/c-api/frame.rst M Doc/data/stable_abi.dat M Doc/tools/extensions/c_annotations.py M Doc/whatsnew/3.11.rst M Misc/stable_abi.txt M Tools/scripts/stable_abi.py diff --git a/Doc/c-api/frame.rst b/Doc/c-api/frame.rst index 6d265e45659f9..f6c682c1e6c1c 100644 --- a/Doc/c-api/frame.rst +++ b/Doc/c-api/frame.rst @@ -7,10 +7,12 @@ Frame Objects The C structure of the objects used to describe frame objects. - The structure is not part of the C API. + There are no public members in this structure. .. versionchanged:: 3.11 - The structure moved to the internal C API headers. + The members of this structure were removed from the public C API. + Refer to the :ref:`What's New entry ` + for details. The :c:func:`PyEval_GetFrame` and :c:func:`PyThreadState_GetFrame` functions can be used to get a frame object. diff --git a/Doc/data/stable_abi.dat b/Doc/data/stable_abi.dat index 18bbf03187b30..7f42f9c12de71 100644 --- a/Doc/data/stable_abi.dat +++ b/Doc/data/stable_abi.dat @@ -1,881 +1,885 @@ -role,name,added,ifdef_note -function,PyAIter_Check,3.10, -function,PyArg_Parse,3.2, -function,PyArg_ParseTuple,3.2, -function,PyArg_ParseTupleAndKeywords,3.2, -function,PyArg_UnpackTuple,3.2, -function,PyArg_VaParse,3.2, -function,PyArg_VaParseTupleAndKeywords,3.2, -function,PyArg_ValidateKeywordArguments,3.2, -var,PyBaseObject_Type,3.2, -function,PyBool_FromLong,3.2, -var,PyBool_Type,3.2, -function,PyBuffer_FillContiguousStrides,3.11, -function,PyBuffer_FillInfo,3.11, -function,PyBuffer_FromContiguous,3.11, -function,PyBuffer_GetPointer,3.11, -function,PyBuffer_IsContiguous,3.11, -function,PyBuffer_Release,3.11, -function,PyBuffer_SizeFromFormat,3.11, -function,PyBuffer_ToContiguous,3.11, -var,PyByteArrayIter_Type,3.2, -function,PyByteArray_AsString,3.2, -function,PyByteArray_Concat,3.2, -function,PyByteArray_FromObject,3.2, -function,PyByteArray_FromStringAndSize,3.2, -function,PyByteArray_Resize,3.2, -function,PyByteArray_Size,3.2, -var,PyByteArray_Type,3.2, -var,PyBytesIter_Type,3.2, -function,PyBytes_AsString,3.2, -function,PyBytes_AsStringAndSize,3.2, -function,PyBytes_Concat,3.2, -function,PyBytes_ConcatAndDel,3.2, -function,PyBytes_DecodeEscape,3.2, -function,PyBytes_FromFormat,3.2, -function,PyBytes_FromFormatV,3.2, -function,PyBytes_FromObject,3.2, -function,PyBytes_FromString,3.2, -function,PyBytes_FromStringAndSize,3.2, -function,PyBytes_Repr,3.2, -function,PyBytes_Size,3.2, -var,PyBytes_Type,3.2, -type,PyCFunction,3.2, -type,PyCFunctionWithKeywords,3.2, -function,PyCFunction_Call,3.2, -function,PyCFunction_GetFlags,3.2, -function,PyCFunction_GetFunction,3.2, -function,PyCFunction_GetSelf,3.2, -function,PyCFunction_New,3.4, -function,PyCFunction_NewEx,3.2, -var,PyCFunction_Type,3.2, -function,PyCMethod_New,3.9, -function,PyCallIter_New,3.2, -var,PyCallIter_Type,3.2, -function,PyCallable_Check,3.2, -type,PyCapsule_Destructor,3.2, -function,PyCapsule_GetContext,3.2, -function,PyCapsule_GetDestructor,3.2, -function,PyCapsule_GetName,3.2, -function,PyCapsule_GetPointer,3.2, -function,PyCapsule_Import,3.2, -function,PyCapsule_IsValid,3.2, -function,PyCapsule_New,3.2, -function,PyCapsule_SetContext,3.2, -function,PyCapsule_SetDestructor,3.2, -function,PyCapsule_SetName,3.2, -function,PyCapsule_SetPointer,3.2, -var,PyCapsule_Type,3.2, -var,PyClassMethodDescr_Type,3.2, -function,PyCodec_BackslashReplaceErrors,3.2, -function,PyCodec_Decode,3.2, -function,PyCodec_Decoder,3.2, -function,PyCodec_Encode,3.2, -function,PyCodec_Encoder,3.2, -function,PyCodec_IgnoreErrors,3.2, -function,PyCodec_IncrementalDecoder,3.2, -function,PyCodec_IncrementalEncoder,3.2, -function,PyCodec_KnownEncoding,3.2, -function,PyCodec_LookupError,3.2, -function,PyCodec_NameReplaceErrors,3.7, -function,PyCodec_Register,3.2, -function,PyCodec_RegisterError,3.2, -function,PyCodec_ReplaceErrors,3.2, -function,PyCodec_StreamReader,3.2, -function,PyCodec_StreamWriter,3.2, -function,PyCodec_StrictErrors,3.2, -function,PyCodec_Unregister,3.10, -function,PyCodec_XMLCharRefReplaceErrors,3.2, -function,PyComplex_FromDoubles,3.2, -function,PyComplex_ImagAsDouble,3.2, -function,PyComplex_RealAsDouble,3.2, -var,PyComplex_Type,3.2, -function,PyDescr_NewClassMethod,3.2, -function,PyDescr_NewGetSet,3.2, -function,PyDescr_NewMember,3.2, -function,PyDescr_NewMethod,3.2, -var,PyDictItems_Type,3.2, -var,PyDictIterItem_Type,3.2, -var,PyDictIterKey_Type,3.2, -var,PyDictIterValue_Type,3.2, -var,PyDictKeys_Type,3.2, -function,PyDictProxy_New,3.2, -var,PyDictProxy_Type,3.2, -var,PyDictRevIterItem_Type,3.8, -var,PyDictRevIterKey_Type,3.8, -var,PyDictRevIterValue_Type,3.8, -var,PyDictValues_Type,3.2, -function,PyDict_Clear,3.2, -function,PyDict_Contains,3.2, -function,PyDict_Copy,3.2, -function,PyDict_DelItem,3.2, -function,PyDict_DelItemString,3.2, -function,PyDict_GetItem,3.2, -function,PyDict_GetItemString,3.2, -function,PyDict_GetItemWithError,3.2, -function,PyDict_Items,3.2, -function,PyDict_Keys,3.2, -function,PyDict_Merge,3.2, -function,PyDict_MergeFromSeq2,3.2, -function,PyDict_New,3.2, -function,PyDict_Next,3.2, -function,PyDict_SetItem,3.2, -function,PyDict_SetItemString,3.2, -function,PyDict_Size,3.2, -var,PyDict_Type,3.2, -function,PyDict_Update,3.2, -function,PyDict_Values,3.2, -var,PyEllipsis_Type,3.2, -var,PyEnum_Type,3.2, -function,PyErr_BadArgument,3.2, -function,PyErr_BadInternalCall,3.2, -function,PyErr_CheckSignals,3.2, -function,PyErr_Clear,3.2, -function,PyErr_Display,3.2, -function,PyErr_ExceptionMatches,3.2, -function,PyErr_Fetch,3.2, -function,PyErr_Format,3.2, -function,PyErr_FormatV,3.5, -function,PyErr_GetExcInfo,3.7, -function,PyErr_GivenExceptionMatches,3.2, -function,PyErr_NewException,3.2, -function,PyErr_NewExceptionWithDoc,3.2, -function,PyErr_NoMemory,3.2, -function,PyErr_NormalizeException,3.2, -function,PyErr_Occurred,3.2, -function,PyErr_Print,3.2, -function,PyErr_PrintEx,3.2, -function,PyErr_ProgramText,3.2, -function,PyErr_ResourceWarning,3.6, -function,PyErr_Restore,3.2, -function,PyErr_SetExcFromWindowsErr,3.7,on Windows -function,PyErr_SetExcFromWindowsErrWithFilename,3.7,on Windows -function,PyErr_SetExcFromWindowsErrWithFilenameObject,3.7,on Windows -function,PyErr_SetExcFromWindowsErrWithFilenameObjects,3.7,on Windows -function,PyErr_SetExcInfo,3.7, -function,PyErr_SetFromErrno,3.2, -function,PyErr_SetFromErrnoWithFilename,3.2, -function,PyErr_SetFromErrnoWithFilenameObject,3.2, -function,PyErr_SetFromErrnoWithFilenameObjects,3.7, -function,PyErr_SetFromWindowsErr,3.7,on Windows -function,PyErr_SetFromWindowsErrWithFilename,3.7,on Windows -function,PyErr_SetImportError,3.7, -function,PyErr_SetImportErrorSubclass,3.6, -function,PyErr_SetInterrupt,3.2, -function,PyErr_SetInterruptEx,3.10, -function,PyErr_SetNone,3.2, -function,PyErr_SetObject,3.2, -function,PyErr_SetString,3.2, -function,PyErr_SyntaxLocation,3.2, -function,PyErr_SyntaxLocationEx,3.7, -function,PyErr_WarnEx,3.2, -function,PyErr_WarnExplicit,3.2, -function,PyErr_WarnFormat,3.2, -function,PyErr_WriteUnraisable,3.2, -function,PyEval_AcquireLock,3.2, -function,PyEval_AcquireThread,3.2, -function,PyEval_CallFunction,3.2, -function,PyEval_CallMethod,3.2, -function,PyEval_CallObjectWithKeywords,3.2, -function,PyEval_EvalCode,3.2, -function,PyEval_EvalCodeEx,3.2, -function,PyEval_EvalFrame,3.2, -function,PyEval_EvalFrameEx,3.2, -function,PyEval_GetBuiltins,3.2, -function,PyEval_GetFrame,3.2, -function,PyEval_GetFuncDesc,3.2, -function,PyEval_GetFuncName,3.2, -function,PyEval_GetGlobals,3.2, -function,PyEval_GetLocals,3.2, -function,PyEval_InitThreads,3.2, -function,PyEval_ReleaseLock,3.2, -function,PyEval_ReleaseThread,3.2, -function,PyEval_RestoreThread,3.2, -function,PyEval_SaveThread,3.2, -function,PyEval_ThreadsInitialized,3.2, -var,PyExc_ArithmeticError,3.2, -var,PyExc_AssertionError,3.2, -var,PyExc_AttributeError,3.2, -var,PyExc_BaseException,3.2, -var,PyExc_BaseExceptionGroup,3.11, -var,PyExc_BlockingIOError,3.7, -var,PyExc_BrokenPipeError,3.7, -var,PyExc_BufferError,3.2, -var,PyExc_BytesWarning,3.2, -var,PyExc_ChildProcessError,3.7, -var,PyExc_ConnectionAbortedError,3.7, -var,PyExc_ConnectionError,3.7, -var,PyExc_ConnectionRefusedError,3.7, -var,PyExc_ConnectionResetError,3.7, -var,PyExc_DeprecationWarning,3.2, -var,PyExc_EOFError,3.2, -var,PyExc_EncodingWarning,3.10, -var,PyExc_EnvironmentError,3.2, -var,PyExc_Exception,3.2, -var,PyExc_FileExistsError,3.7, -var,PyExc_FileNotFoundError,3.7, -var,PyExc_FloatingPointError,3.2, -var,PyExc_FutureWarning,3.2, -var,PyExc_GeneratorExit,3.2, -var,PyExc_IOError,3.2, -var,PyExc_ImportError,3.2, -var,PyExc_ImportWarning,3.2, -var,PyExc_IndentationError,3.2, -var,PyExc_IndexError,3.2, -var,PyExc_InterruptedError,3.7, -var,PyExc_IsADirectoryError,3.7, -var,PyExc_KeyError,3.2, -var,PyExc_KeyboardInterrupt,3.2, -var,PyExc_LookupError,3.2, -var,PyExc_MemoryError,3.2, -var,PyExc_ModuleNotFoundError,3.6, -var,PyExc_NameError,3.2, -var,PyExc_NotADirectoryError,3.7, -var,PyExc_NotImplementedError,3.2, -var,PyExc_OSError,3.2, -var,PyExc_OverflowError,3.2, -var,PyExc_PendingDeprecationWarning,3.2, -var,PyExc_PermissionError,3.7, -var,PyExc_ProcessLookupError,3.7, -var,PyExc_RecursionError,3.7, -var,PyExc_ReferenceError,3.2, -var,PyExc_ResourceWarning,3.7, -var,PyExc_RuntimeError,3.2, -var,PyExc_RuntimeWarning,3.2, -var,PyExc_StopAsyncIteration,3.7, -var,PyExc_StopIteration,3.2, -var,PyExc_SyntaxError,3.2, -var,PyExc_SyntaxWarning,3.2, -var,PyExc_SystemError,3.2, -var,PyExc_SystemExit,3.2, -var,PyExc_TabError,3.2, -var,PyExc_TimeoutError,3.7, -var,PyExc_TypeError,3.2, -var,PyExc_UnboundLocalError,3.2, -var,PyExc_UnicodeDecodeError,3.2, -var,PyExc_UnicodeEncodeError,3.2, -var,PyExc_UnicodeError,3.2, -var,PyExc_UnicodeTranslateError,3.2, -var,PyExc_UnicodeWarning,3.2, -var,PyExc_UserWarning,3.2, -var,PyExc_ValueError,3.2, -var,PyExc_Warning,3.2, -var,PyExc_WindowsError,3.7,on Windows -var,PyExc_ZeroDivisionError,3.2, -function,PyExceptionClass_Name,3.8, -function,PyException_GetCause,3.2, -function,PyException_GetContext,3.2, -function,PyException_GetTraceback,3.2, -function,PyException_SetCause,3.2, -function,PyException_SetContext,3.2, -function,PyException_SetTraceback,3.2, -function,PyFile_FromFd,3.2, -function,PyFile_GetLine,3.2, -function,PyFile_WriteObject,3.2, -function,PyFile_WriteString,3.2, -var,PyFilter_Type,3.2, -function,PyFloat_AsDouble,3.2, -function,PyFloat_FromDouble,3.2, -function,PyFloat_FromString,3.2, -function,PyFloat_GetInfo,3.2, -function,PyFloat_GetMax,3.2, -function,PyFloat_GetMin,3.2, -var,PyFloat_Type,3.2, -type,PyFrameObject,3.2, -function,PyFrame_GetCode,3.10, -function,PyFrame_GetLineNumber,3.10, -function,PyFrozenSet_New,3.2, -var,PyFrozenSet_Type,3.2, -function,PyGC_Collect,3.2, -function,PyGC_Disable,3.10, -function,PyGC_Enable,3.10, -function,PyGC_IsEnabled,3.10, -function,PyGILState_Ensure,3.2, -function,PyGILState_GetThisThreadState,3.2, -function,PyGILState_Release,3.2, -type,PyGILState_STATE,3.2, -type,PyGetSetDef,3.2, -var,PyGetSetDescr_Type,3.2, -function,PyImport_AddModule,3.2, -function,PyImport_AddModuleObject,3.7, -function,PyImport_AppendInittab,3.2, -function,PyImport_ExecCodeModule,3.2, -function,PyImport_ExecCodeModuleEx,3.2, -function,PyImport_ExecCodeModuleObject,3.7, -function,PyImport_ExecCodeModuleWithPathnames,3.2, -function,PyImport_GetImporter,3.2, -function,PyImport_GetMagicNumber,3.2, -function,PyImport_GetMagicTag,3.2, -function,PyImport_GetModule,3.8, -function,PyImport_GetModuleDict,3.2, -function,PyImport_Import,3.2, -function,PyImport_ImportFrozenModule,3.2, -function,PyImport_ImportFrozenModuleObject,3.7, -function,PyImport_ImportModule,3.2, -function,PyImport_ImportModuleLevel,3.2, -function,PyImport_ImportModuleLevelObject,3.7, -function,PyImport_ImportModuleNoBlock,3.2, -function,PyImport_ReloadModule,3.2, -function,PyIndex_Check,3.8, -type,PyInterpreterState,3.2, -function,PyInterpreterState_Clear,3.2, -function,PyInterpreterState_Delete,3.2, -function,PyInterpreterState_Get,3.9, -function,PyInterpreterState_GetDict,3.8, -function,PyInterpreterState_GetID,3.7, -function,PyInterpreterState_New,3.2, -function,PyIter_Check,3.8, -function,PyIter_Next,3.2, -function,PyIter_Send,3.10, -var,PyListIter_Type,3.2, -var,PyListRevIter_Type,3.2, -function,PyList_Append,3.2, -function,PyList_AsTuple,3.2, -function,PyList_GetItem,3.2, -function,PyList_GetSlice,3.2, -function,PyList_Insert,3.2, -function,PyList_New,3.2, -function,PyList_Reverse,3.2, -function,PyList_SetItem,3.2, -function,PyList_SetSlice,3.2, -function,PyList_Size,3.2, -function,PyList_Sort,3.2, -var,PyList_Type,3.2, -type,PyLongObject,3.2, -var,PyLongRangeIter_Type,3.2, -function,PyLong_AsDouble,3.2, -function,PyLong_AsLong,3.2, -function,PyLong_AsLongAndOverflow,3.2, -function,PyLong_AsLongLong,3.2, -function,PyLong_AsLongLongAndOverflow,3.2, -function,PyLong_AsSize_t,3.2, -function,PyLong_AsSsize_t,3.2, -function,PyLong_AsUnsignedLong,3.2, -function,PyLong_AsUnsignedLongLong,3.2, -function,PyLong_AsUnsignedLongLongMask,3.2, -function,PyLong_AsUnsignedLongMask,3.2, -function,PyLong_AsVoidPtr,3.2, -function,PyLong_FromDouble,3.2, -function,PyLong_FromLong,3.2, -function,PyLong_FromLongLong,3.2, -function,PyLong_FromSize_t,3.2, -function,PyLong_FromSsize_t,3.2, -function,PyLong_FromString,3.2, -function,PyLong_FromUnsignedLong,3.2, -function,PyLong_FromUnsignedLongLong,3.2, -function,PyLong_FromVoidPtr,3.2, -function,PyLong_GetInfo,3.2, -var,PyLong_Type,3.2, -var,PyMap_Type,3.2, -function,PyMapping_Check,3.2, -function,PyMapping_GetItemString,3.2, -function,PyMapping_HasKey,3.2, -function,PyMapping_HasKeyString,3.2, -function,PyMapping_Items,3.2, -function,PyMapping_Keys,3.2, -function,PyMapping_Length,3.2, -function,PyMapping_SetItemString,3.2, -function,PyMapping_Size,3.2, -function,PyMapping_Values,3.2, -function,PyMem_Calloc,3.7, -function,PyMem_Free,3.2, -function,PyMem_Malloc,3.2, -function,PyMem_Realloc,3.2, -type,PyMemberDef,3.2, -var,PyMemberDescr_Type,3.2, -function,PyMemoryView_FromBuffer,3.11, -function,PyMemoryView_FromMemory,3.7, -function,PyMemoryView_FromObject,3.2, -function,PyMemoryView_GetContiguous,3.2, -var,PyMemoryView_Type,3.2, -type,PyMethodDef,3.2, -var,PyMethodDescr_Type,3.2, -type,PyModuleDef,3.2, -type,PyModuleDef_Base,3.2, -function,PyModuleDef_Init,3.5, -var,PyModuleDef_Type,3.5, -function,PyModule_AddFunctions,3.7, -function,PyModule_AddIntConstant,3.2, -function,PyModule_AddObject,3.2, -function,PyModule_AddObjectRef,3.10, -function,PyModule_AddStringConstant,3.2, -function,PyModule_AddType,3.10, -function,PyModule_Create2,3.2, -function,PyModule_ExecDef,3.7, -function,PyModule_FromDefAndSpec2,3.7, -function,PyModule_GetDef,3.2, -function,PyModule_GetDict,3.2, -function,PyModule_GetFilename,3.2, -function,PyModule_GetFilenameObject,3.2, -function,PyModule_GetName,3.2, -function,PyModule_GetNameObject,3.7, -function,PyModule_GetState,3.2, -function,PyModule_New,3.2, -function,PyModule_NewObject,3.7, -function,PyModule_SetDocString,3.7, -var,PyModule_Type,3.2, -function,PyNumber_Absolute,3.2, -function,PyNumber_Add,3.2, -function,PyNumber_And,3.2, -function,PyNumber_AsSsize_t,3.2, -function,PyNumber_Check,3.2, -function,PyNumber_Divmod,3.2, -function,PyNumber_Float,3.2, -function,PyNumber_FloorDivide,3.2, -function,PyNumber_InPlaceAdd,3.2, -function,PyNumber_InPlaceAnd,3.2, -function,PyNumber_InPlaceFloorDivide,3.2, -function,PyNumber_InPlaceLshift,3.2, -function,PyNumber_InPlaceMatrixMultiply,3.7, -function,PyNumber_InPlaceMultiply,3.2, -function,PyNumber_InPlaceOr,3.2, -function,PyNumber_InPlacePower,3.2, -function,PyNumber_InPlaceRemainder,3.2, -function,PyNumber_InPlaceRshift,3.2, -function,PyNumber_InPlaceSubtract,3.2, -function,PyNumber_InPlaceTrueDivide,3.2, -function,PyNumber_InPlaceXor,3.2, -function,PyNumber_Index,3.2, -function,PyNumber_Invert,3.2, -function,PyNumber_Long,3.2, -function,PyNumber_Lshift,3.2, -function,PyNumber_MatrixMultiply,3.7, -function,PyNumber_Multiply,3.2, -function,PyNumber_Negative,3.2, -function,PyNumber_Or,3.2, -function,PyNumber_Positive,3.2, -function,PyNumber_Power,3.2, -function,PyNumber_Remainder,3.2, -function,PyNumber_Rshift,3.2, -function,PyNumber_Subtract,3.2, -function,PyNumber_ToBase,3.2, -function,PyNumber_TrueDivide,3.2, -function,PyNumber_Xor,3.2, -function,PyOS_AfterFork,3.2,on platforms with fork() -function,PyOS_AfterFork_Child,3.7,on platforms with fork() -function,PyOS_AfterFork_Parent,3.7,on platforms with fork() -function,PyOS_BeforeFork,3.7,on platforms with fork() -function,PyOS_CheckStack,3.7,on platforms with USE_STACKCHECK -function,PyOS_FSPath,3.6, -var,PyOS_InputHook,3.2, -function,PyOS_InterruptOccurred,3.2, -function,PyOS_double_to_string,3.2, -function,PyOS_getsig,3.2, -function,PyOS_mystricmp,3.2, -function,PyOS_mystrnicmp,3.2, -function,PyOS_setsig,3.2, -type,PyOS_sighandler_t,3.2, -function,PyOS_snprintf,3.2, -function,PyOS_string_to_double,3.2, -function,PyOS_strtol,3.2, -function,PyOS_strtoul,3.2, -function,PyOS_vsnprintf,3.2, -type,PyObject,3.2, -function,PyObject_ASCII,3.2, -function,PyObject_AsCharBuffer,3.2, -function,PyObject_AsFileDescriptor,3.2, -function,PyObject_AsReadBuffer,3.2, -function,PyObject_AsWriteBuffer,3.2, -function,PyObject_Bytes,3.2, -function,PyObject_Call,3.2, -function,PyObject_CallFunction,3.2, -function,PyObject_CallFunctionObjArgs,3.2, -function,PyObject_CallMethod,3.2, -function,PyObject_CallMethodObjArgs,3.2, -function,PyObject_CallNoArgs,3.10, -function,PyObject_CallObject,3.2, -function,PyObject_Calloc,3.7, -function,PyObject_CheckBuffer,3.11, -function,PyObject_CheckReadBuffer,3.2, -function,PyObject_ClearWeakRefs,3.2, -function,PyObject_CopyData,3.11, -function,PyObject_DelItem,3.2, -function,PyObject_DelItemString,3.2, -function,PyObject_Dir,3.2, -function,PyObject_Format,3.2, -function,PyObject_Free,3.2, -function,PyObject_GC_Del,3.2, -function,PyObject_GC_IsFinalized,3.9, -function,PyObject_GC_IsTracked,3.9, -function,PyObject_GC_Track,3.2, -function,PyObject_GC_UnTrack,3.2, -function,PyObject_GenericGetAttr,3.2, -function,PyObject_GenericGetDict,3.10, -function,PyObject_GenericSetAttr,3.2, -function,PyObject_GenericSetDict,3.7, -function,PyObject_GetAIter,3.10, -function,PyObject_GetAttr,3.2, -function,PyObject_GetAttrString,3.2, -function,PyObject_GetBuffer,3.11, -function,PyObject_GetItem,3.2, -function,PyObject_GetIter,3.2, -function,PyObject_HasAttr,3.2, -function,PyObject_HasAttrString,3.2, -function,PyObject_Hash,3.2, -function,PyObject_HashNotImplemented,3.2, -function,PyObject_Init,3.2, -function,PyObject_InitVar,3.2, -function,PyObject_IsInstance,3.2, -function,PyObject_IsSubclass,3.2, -function,PyObject_IsTrue,3.2, -function,PyObject_Length,3.2, -function,PyObject_Malloc,3.2, -function,PyObject_Not,3.2, -function,PyObject_Realloc,3.2, -function,PyObject_Repr,3.2, -function,PyObject_RichCompare,3.2, -function,PyObject_RichCompareBool,3.2, -function,PyObject_SelfIter,3.2, -function,PyObject_SetAttr,3.2, -function,PyObject_SetAttrString,3.2, -function,PyObject_SetItem,3.2, -function,PyObject_Size,3.2, -function,PyObject_Str,3.2, -function,PyObject_Type,3.2, -var,PyProperty_Type,3.2, -var,PyRangeIter_Type,3.2, -var,PyRange_Type,3.2, -var,PyReversed_Type,3.2, -function,PySeqIter_New,3.2, -var,PySeqIter_Type,3.2, -function,PySequence_Check,3.2, -function,PySequence_Concat,3.2, -function,PySequence_Contains,3.2, -function,PySequence_Count,3.2, -function,PySequence_DelItem,3.2, -function,PySequence_DelSlice,3.2, -function,PySequence_Fast,3.2, -function,PySequence_GetItem,3.2, -function,PySequence_GetSlice,3.2, -function,PySequence_In,3.2, -function,PySequence_InPlaceConcat,3.2, -function,PySequence_InPlaceRepeat,3.2, -function,PySequence_Index,3.2, -function,PySequence_Length,3.2, -function,PySequence_List,3.2, -function,PySequence_Repeat,3.2, -function,PySequence_SetItem,3.2, -function,PySequence_SetSlice,3.2, -function,PySequence_Size,3.2, -function,PySequence_Tuple,3.2, -var,PySetIter_Type,3.2, -function,PySet_Add,3.2, -function,PySet_Clear,3.2, -function,PySet_Contains,3.2, -function,PySet_Discard,3.2, -function,PySet_New,3.2, -function,PySet_Pop,3.2, -function,PySet_Size,3.2, -var,PySet_Type,3.2, -function,PySlice_AdjustIndices,3.7, -function,PySlice_GetIndices,3.2, -function,PySlice_GetIndicesEx,3.2, -function,PySlice_New,3.2, -var,PySlice_Type,3.2, -function,PySlice_Unpack,3.7, -function,PyState_AddModule,3.3, -function,PyState_FindModule,3.2, -function,PyState_RemoveModule,3.3, -type,PyStructSequence_Desc,3.2, -type,PyStructSequence_Field,3.2, -function,PyStructSequence_GetItem,3.2, -function,PyStructSequence_New,3.2, -function,PyStructSequence_NewType,3.2, -function,PyStructSequence_SetItem,3.2, -var,PyStructSequence_UnnamedField,3.11, -var,PySuper_Type,3.2, -function,PySys_AddWarnOption,3.2, -function,PySys_AddWarnOptionUnicode,3.2, -function,PySys_AddXOption,3.7, -function,PySys_FormatStderr,3.2, -function,PySys_FormatStdout,3.2, -function,PySys_GetObject,3.2, -function,PySys_GetXOptions,3.7, -function,PySys_HasWarnOptions,3.2, -function,PySys_ResetWarnOptions,3.2, -function,PySys_SetArgv,3.2, -function,PySys_SetArgvEx,3.2, -function,PySys_SetObject,3.2, -function,PySys_SetPath,3.2, -function,PySys_WriteStderr,3.2, -function,PySys_WriteStdout,3.2, -type,PyThreadState,3.2, -function,PyThreadState_Clear,3.2, -function,PyThreadState_Delete,3.2, -function,PyThreadState_Get,3.2, -function,PyThreadState_GetDict,3.2, -function,PyThreadState_GetFrame,3.10, -function,PyThreadState_GetID,3.10, -function,PyThreadState_GetInterpreter,3.10, -function,PyThreadState_New,3.2, -function,PyThreadState_SetAsyncExc,3.2, -function,PyThreadState_Swap,3.2, -function,PyThread_GetInfo,3.3, -function,PyThread_ReInitTLS,3.2, -function,PyThread_acquire_lock,3.2, -function,PyThread_acquire_lock_timed,3.2, -function,PyThread_allocate_lock,3.2, -function,PyThread_create_key,3.2, -function,PyThread_delete_key,3.2, -function,PyThread_delete_key_value,3.2, -function,PyThread_exit_thread,3.2, -function,PyThread_free_lock,3.2, -function,PyThread_get_key_value,3.2, -function,PyThread_get_stacksize,3.2, -function,PyThread_get_thread_ident,3.2, -function,PyThread_get_thread_native_id,3.2, -function,PyThread_init_thread,3.2, -function,PyThread_release_lock,3.2, -function,PyThread_set_key_value,3.2, -function,PyThread_set_stacksize,3.2, -function,PyThread_start_new_thread,3.2, -function,PyThread_tss_alloc,3.7, -function,PyThread_tss_create,3.7, -function,PyThread_tss_delete,3.7, -function,PyThread_tss_free,3.7, -function,PyThread_tss_get,3.7, -function,PyThread_tss_is_created,3.7, -function,PyThread_tss_set,3.7, -function,PyTraceBack_Here,3.2, -function,PyTraceBack_Print,3.2, -var,PyTraceBack_Type,3.2, -var,PyTupleIter_Type,3.2, -function,PyTuple_GetItem,3.2, -function,PyTuple_GetSlice,3.2, -function,PyTuple_New,3.2, -function,PyTuple_Pack,3.2, -function,PyTuple_SetItem,3.2, -function,PyTuple_Size,3.2, -var,PyTuple_Type,3.2, -type,PyTypeObject,3.2, -function,PyType_ClearCache,3.2, -function,PyType_FromModuleAndSpec,3.10, -function,PyType_FromSpec,3.2, -function,PyType_FromSpecWithBases,3.3, -function,PyType_GenericAlloc,3.2, -function,PyType_GenericNew,3.2, -function,PyType_GetFlags,3.2, -function,PyType_GetModule,3.10, -function,PyType_GetModuleState,3.10, -function,PyType_GetName,3.11, -function,PyType_GetQualName,3.11, -function,PyType_GetSlot,3.4, -function,PyType_IsSubtype,3.2, -function,PyType_Modified,3.2, -function,PyType_Ready,3.2, -type,PyType_Slot,3.2, -type,PyType_Spec,3.2, -var,PyType_Type,3.2, -function,PyUnicodeDecodeError_Create,3.2, -function,PyUnicodeDecodeError_GetEncoding,3.2, -function,PyUnicodeDecodeError_GetEnd,3.2, -function,PyUnicodeDecodeError_GetObject,3.2, -function,PyUnicodeDecodeError_GetReason,3.2, -function,PyUnicodeDecodeError_GetStart,3.2, -function,PyUnicodeDecodeError_SetEnd,3.2, -function,PyUnicodeDecodeError_SetReason,3.2, -function,PyUnicodeDecodeError_SetStart,3.2, -function,PyUnicodeEncodeError_GetEncoding,3.2, -function,PyUnicodeEncodeError_GetEnd,3.2, -function,PyUnicodeEncodeError_GetObject,3.2, -function,PyUnicodeEncodeError_GetReason,3.2, -function,PyUnicodeEncodeError_GetStart,3.2, -function,PyUnicodeEncodeError_SetEnd,3.2, -function,PyUnicodeEncodeError_SetReason,3.2, -function,PyUnicodeEncodeError_SetStart,3.2, -var,PyUnicodeIter_Type,3.2, -function,PyUnicodeTranslateError_GetEnd,3.2, -function,PyUnicodeTranslateError_GetObject,3.2, -function,PyUnicodeTranslateError_GetReason,3.2, -function,PyUnicodeTranslateError_GetStart,3.2, -function,PyUnicodeTranslateError_SetEnd,3.2, -function,PyUnicodeTranslateError_SetReason,3.2, -function,PyUnicodeTranslateError_SetStart,3.2, -function,PyUnicode_Append,3.2, -function,PyUnicode_AppendAndDel,3.2, -function,PyUnicode_AsASCIIString,3.2, -function,PyUnicode_AsCharmapString,3.2, -function,PyUnicode_AsDecodedObject,3.2, -function,PyUnicode_AsDecodedUnicode,3.2, -function,PyUnicode_AsEncodedObject,3.2, -function,PyUnicode_AsEncodedString,3.2, -function,PyUnicode_AsEncodedUnicode,3.2, -function,PyUnicode_AsLatin1String,3.2, -function,PyUnicode_AsMBCSString,3.7,on Windows -function,PyUnicode_AsRawUnicodeEscapeString,3.2, -function,PyUnicode_AsUCS4,3.7, -function,PyUnicode_AsUCS4Copy,3.7, -function,PyUnicode_AsUTF16String,3.2, -function,PyUnicode_AsUTF32String,3.2, -function,PyUnicode_AsUTF8AndSize,3.10, -function,PyUnicode_AsUTF8String,3.2, -function,PyUnicode_AsUnicodeEscapeString,3.2, -function,PyUnicode_AsWideChar,3.2, -function,PyUnicode_AsWideCharString,3.7, -function,PyUnicode_BuildEncodingMap,3.2, -function,PyUnicode_Compare,3.2, -function,PyUnicode_CompareWithASCIIString,3.2, -function,PyUnicode_Concat,3.2, -function,PyUnicode_Contains,3.2, -function,PyUnicode_Count,3.2, -function,PyUnicode_Decode,3.2, -function,PyUnicode_DecodeASCII,3.2, -function,PyUnicode_DecodeCharmap,3.2, -function,PyUnicode_DecodeCodePageStateful,3.7,on Windows -function,PyUnicode_DecodeFSDefault,3.2, -function,PyUnicode_DecodeFSDefaultAndSize,3.2, -function,PyUnicode_DecodeLatin1,3.2, -function,PyUnicode_DecodeLocale,3.7, -function,PyUnicode_DecodeLocaleAndSize,3.7, -function,PyUnicode_DecodeMBCS,3.7,on Windows -function,PyUnicode_DecodeMBCSStateful,3.7,on Windows -function,PyUnicode_DecodeRawUnicodeEscape,3.2, -function,PyUnicode_DecodeUTF16,3.2, -function,PyUnicode_DecodeUTF16Stateful,3.2, -function,PyUnicode_DecodeUTF32,3.2, -function,PyUnicode_DecodeUTF32Stateful,3.2, -function,PyUnicode_DecodeUTF7,3.2, -function,PyUnicode_DecodeUTF7Stateful,3.2, -function,PyUnicode_DecodeUTF8,3.2, -function,PyUnicode_DecodeUTF8Stateful,3.2, -function,PyUnicode_DecodeUnicodeEscape,3.2, -function,PyUnicode_EncodeCodePage,3.7,on Windows -function,PyUnicode_EncodeFSDefault,3.2, -function,PyUnicode_EncodeLocale,3.7, -function,PyUnicode_FSConverter,3.2, -function,PyUnicode_FSDecoder,3.2, -function,PyUnicode_Find,3.2, -function,PyUnicode_FindChar,3.7, -function,PyUnicode_Format,3.2, -function,PyUnicode_FromEncodedObject,3.2, -function,PyUnicode_FromFormat,3.2, -function,PyUnicode_FromFormatV,3.2, -function,PyUnicode_FromObject,3.2, -function,PyUnicode_FromOrdinal,3.2, -function,PyUnicode_FromString,3.2, -function,PyUnicode_FromStringAndSize,3.2, -function,PyUnicode_FromWideChar,3.2, -function,PyUnicode_GetDefaultEncoding,3.2, -function,PyUnicode_GetLength,3.7, -function,PyUnicode_GetSize,3.2, -function,PyUnicode_InternFromString,3.2, -function,PyUnicode_InternImmortal,3.2, -function,PyUnicode_InternInPlace,3.2, -function,PyUnicode_IsIdentifier,3.2, -function,PyUnicode_Join,3.2, -function,PyUnicode_Partition,3.2, -function,PyUnicode_RPartition,3.2, -function,PyUnicode_RSplit,3.2, -function,PyUnicode_ReadChar,3.7, -function,PyUnicode_Replace,3.2, -function,PyUnicode_Resize,3.2, -function,PyUnicode_RichCompare,3.2, -function,PyUnicode_Split,3.2, -function,PyUnicode_Splitlines,3.2, -function,PyUnicode_Substring,3.7, -function,PyUnicode_Tailmatch,3.2, -function,PyUnicode_Translate,3.2, -var,PyUnicode_Type,3.2, -function,PyUnicode_WriteChar,3.7, -type,PyVarObject,3.2, -type,PyWeakReference,3.2, -function,PyWeakref_GetObject,3.2, -function,PyWeakref_NewProxy,3.2, -function,PyWeakref_NewRef,3.2, -var,PyWrapperDescr_Type,3.2, -function,PyWrapper_New,3.2, -var,PyZip_Type,3.2, -function,Py_AddPendingCall,3.2, -function,Py_AtExit,3.2, -macro,Py_BEGIN_ALLOW_THREADS,3.2, -macro,Py_BLOCK_THREADS,3.2, -function,Py_BuildValue,3.2, -function,Py_BytesMain,3.8, -function,Py_CompileString,3.2, -function,Py_DecRef,3.2, -function,Py_DecodeLocale,3.7, -macro,Py_END_ALLOW_THREADS,3.2, -function,Py_EncodeLocale,3.7, -function,Py_EndInterpreter,3.2, -function,Py_EnterRecursiveCall,3.9, -function,Py_Exit,3.2, -function,Py_FatalError,3.2, -var,Py_FileSystemDefaultEncodeErrors,3.10, -var,Py_FileSystemDefaultEncoding,3.2, -function,Py_Finalize,3.2, -function,Py_FinalizeEx,3.6, -function,Py_GenericAlias,3.9, -var,Py_GenericAliasType,3.9, -function,Py_GetBuildInfo,3.2, -function,Py_GetCompiler,3.2, -function,Py_GetCopyright,3.2, -function,Py_GetExecPrefix,3.2, -function,Py_GetPath,3.2, -function,Py_GetPlatform,3.2, -function,Py_GetPrefix,3.2, -function,Py_GetProgramFullPath,3.2, -function,Py_GetProgramName,3.2, -function,Py_GetPythonHome,3.2, -function,Py_GetRecursionLimit,3.2, -function,Py_GetVersion,3.2, -var,Py_HasFileSystemDefaultEncoding,3.2, -function,Py_IncRef,3.2, -function,Py_Initialize,3.2, -function,Py_InitializeEx,3.2, -function,Py_Is,3.10, -function,Py_IsFalse,3.10, -function,Py_IsInitialized,3.2, -function,Py_IsNone,3.10, -function,Py_IsTrue,3.10, -function,Py_LeaveRecursiveCall,3.9, -function,Py_Main,3.2, -function,Py_MakePendingCalls,3.2, -function,Py_NewInterpreter,3.2, -function,Py_NewRef,3.10, -function,Py_ReprEnter,3.2, -function,Py_ReprLeave,3.2, -function,Py_SetPath,3.7, -function,Py_SetProgramName,3.2, -function,Py_SetPythonHome,3.2, -function,Py_SetRecursionLimit,3.2, -type,Py_UCS4,3.2, -macro,Py_UNBLOCK_THREADS,3.2, -var,Py_UTF8Mode,3.8, -function,Py_VaBuildValue,3.2, -var,Py_Version,3.11, -function,Py_XNewRef,3.10, -type,Py_buffer,3.11, -type,Py_intptr_t,3.2, -type,Py_ssize_t,3.2, -type,Py_uintptr_t,3.2, -type,allocfunc,3.2, -type,binaryfunc,3.2, -type,descrgetfunc,3.2, -type,descrsetfunc,3.2, -type,destructor,3.2, -type,getattrfunc,3.2, -type,getattrofunc,3.2, -type,getiterfunc,3.2, -type,getter,3.2, -type,hashfunc,3.2, -type,initproc,3.2, -type,inquiry,3.2, -type,iternextfunc,3.2, -type,lenfunc,3.2, -type,newfunc,3.2, -type,objobjargproc,3.2, -type,objobjproc,3.2, -type,reprfunc,3.2, -type,richcmpfunc,3.2, -type,setattrfunc,3.2, -type,setattrofunc,3.2, -type,setter,3.2, -type,ssizeargfunc,3.2, -type,ssizeobjargproc,3.2, -type,ssizessizeargfunc,3.2, -type,ssizessizeobjargproc,3.2, -type,symtable,3.2, -type,ternaryfunc,3.2, -type,traverseproc,3.2, -type,unaryfunc,3.2, -type,visitproc,3.2, +role,name,added,ifdef_note,struct_abi_kind +function,PyAIter_Check,3.10,, +function,PyArg_Parse,3.2,, +function,PyArg_ParseTuple,3.2,, +function,PyArg_ParseTupleAndKeywords,3.2,, +function,PyArg_UnpackTuple,3.2,, +function,PyArg_VaParse,3.2,, +function,PyArg_VaParseTupleAndKeywords,3.2,, +function,PyArg_ValidateKeywordArguments,3.2,, +var,PyBaseObject_Type,3.2,, +function,PyBool_FromLong,3.2,, +var,PyBool_Type,3.2,, +function,PyBuffer_FillContiguousStrides,3.11,, +function,PyBuffer_FillInfo,3.11,, +function,PyBuffer_FromContiguous,3.11,, +function,PyBuffer_GetPointer,3.11,, +function,PyBuffer_IsContiguous,3.11,, +function,PyBuffer_Release,3.11,, +function,PyBuffer_SizeFromFormat,3.11,, +function,PyBuffer_ToContiguous,3.11,, +var,PyByteArrayIter_Type,3.2,, +function,PyByteArray_AsString,3.2,, +function,PyByteArray_Concat,3.2,, +function,PyByteArray_FromObject,3.2,, +function,PyByteArray_FromStringAndSize,3.2,, +function,PyByteArray_Resize,3.2,, +function,PyByteArray_Size,3.2,, +var,PyByteArray_Type,3.2,, +var,PyBytesIter_Type,3.2,, +function,PyBytes_AsString,3.2,, +function,PyBytes_AsStringAndSize,3.2,, +function,PyBytes_Concat,3.2,, +function,PyBytes_ConcatAndDel,3.2,, +function,PyBytes_DecodeEscape,3.2,, +function,PyBytes_FromFormat,3.2,, +function,PyBytes_FromFormatV,3.2,, +function,PyBytes_FromObject,3.2,, +function,PyBytes_FromString,3.2,, +function,PyBytes_FromStringAndSize,3.2,, +function,PyBytes_Repr,3.2,, +function,PyBytes_Size,3.2,, +var,PyBytes_Type,3.2,, +type,PyCFunction,3.2,, +type,PyCFunctionWithKeywords,3.2,, +function,PyCFunction_Call,3.2,, +function,PyCFunction_GetFlags,3.2,, +function,PyCFunction_GetFunction,3.2,, +function,PyCFunction_GetSelf,3.2,, +function,PyCFunction_New,3.4,, +function,PyCFunction_NewEx,3.2,, +var,PyCFunction_Type,3.2,, +function,PyCMethod_New,3.9,, +function,PyCallIter_New,3.2,, +var,PyCallIter_Type,3.2,, +function,PyCallable_Check,3.2,, +type,PyCapsule_Destructor,3.2,, +function,PyCapsule_GetContext,3.2,, +function,PyCapsule_GetDestructor,3.2,, +function,PyCapsule_GetName,3.2,, +function,PyCapsule_GetPointer,3.2,, +function,PyCapsule_Import,3.2,, +function,PyCapsule_IsValid,3.2,, +function,PyCapsule_New,3.2,, +function,PyCapsule_SetContext,3.2,, +function,PyCapsule_SetDestructor,3.2,, +function,PyCapsule_SetName,3.2,, +function,PyCapsule_SetPointer,3.2,, +var,PyCapsule_Type,3.2,, +var,PyClassMethodDescr_Type,3.2,, +function,PyCodec_BackslashReplaceErrors,3.2,, +function,PyCodec_Decode,3.2,, +function,PyCodec_Decoder,3.2,, +function,PyCodec_Encode,3.2,, +function,PyCodec_Encoder,3.2,, +function,PyCodec_IgnoreErrors,3.2,, +function,PyCodec_IncrementalDecoder,3.2,, +function,PyCodec_IncrementalEncoder,3.2,, +function,PyCodec_KnownEncoding,3.2,, +function,PyCodec_LookupError,3.2,, +function,PyCodec_NameReplaceErrors,3.7,, +function,PyCodec_Register,3.2,, +function,PyCodec_RegisterError,3.2,, +function,PyCodec_ReplaceErrors,3.2,, +function,PyCodec_StreamReader,3.2,, +function,PyCodec_StreamWriter,3.2,, +function,PyCodec_StrictErrors,3.2,, +function,PyCodec_Unregister,3.10,, +function,PyCodec_XMLCharRefReplaceErrors,3.2,, +function,PyComplex_FromDoubles,3.2,, +function,PyComplex_ImagAsDouble,3.2,, +function,PyComplex_RealAsDouble,3.2,, +var,PyComplex_Type,3.2,, +function,PyDescr_NewClassMethod,3.2,, +function,PyDescr_NewGetSet,3.2,, +function,PyDescr_NewMember,3.2,, +function,PyDescr_NewMethod,3.2,, +var,PyDictItems_Type,3.2,, +var,PyDictIterItem_Type,3.2,, +var,PyDictIterKey_Type,3.2,, +var,PyDictIterValue_Type,3.2,, +var,PyDictKeys_Type,3.2,, +function,PyDictProxy_New,3.2,, +var,PyDictProxy_Type,3.2,, +var,PyDictRevIterItem_Type,3.8,, +var,PyDictRevIterKey_Type,3.8,, +var,PyDictRevIterValue_Type,3.8,, +var,PyDictValues_Type,3.2,, +function,PyDict_Clear,3.2,, +function,PyDict_Contains,3.2,, +function,PyDict_Copy,3.2,, +function,PyDict_DelItem,3.2,, +function,PyDict_DelItemString,3.2,, +function,PyDict_GetItem,3.2,, +function,PyDict_GetItemString,3.2,, +function,PyDict_GetItemWithError,3.2,, +function,PyDict_Items,3.2,, +function,PyDict_Keys,3.2,, +function,PyDict_Merge,3.2,, +function,PyDict_MergeFromSeq2,3.2,, +function,PyDict_New,3.2,, +function,PyDict_Next,3.2,, +function,PyDict_SetItem,3.2,, +function,PyDict_SetItemString,3.2,, +function,PyDict_Size,3.2,, +var,PyDict_Type,3.2,, +function,PyDict_Update,3.2,, +function,PyDict_Values,3.2,, +var,PyEllipsis_Type,3.2,, +var,PyEnum_Type,3.2,, +function,PyErr_BadArgument,3.2,, +function,PyErr_BadInternalCall,3.2,, +function,PyErr_CheckSignals,3.2,, +function,PyErr_Clear,3.2,, +function,PyErr_Display,3.2,, +function,PyErr_ExceptionMatches,3.2,, +function,PyErr_Fetch,3.2,, +function,PyErr_Format,3.2,, +function,PyErr_FormatV,3.5,, +function,PyErr_GetExcInfo,3.7,, +function,PyErr_GivenExceptionMatches,3.2,, +function,PyErr_NewException,3.2,, +function,PyErr_NewExceptionWithDoc,3.2,, +function,PyErr_NoMemory,3.2,, +function,PyErr_NormalizeException,3.2,, +function,PyErr_Occurred,3.2,, +function,PyErr_Print,3.2,, +function,PyErr_PrintEx,3.2,, +function,PyErr_ProgramText,3.2,, +function,PyErr_ResourceWarning,3.6,, +function,PyErr_Restore,3.2,, +function,PyErr_SetExcFromWindowsErr,3.7,on Windows, +function,PyErr_SetExcFromWindowsErrWithFilename,3.7,on Windows, +function,PyErr_SetExcFromWindowsErrWithFilenameObject,3.7,on Windows, +function,PyErr_SetExcFromWindowsErrWithFilenameObjects,3.7,on Windows, +function,PyErr_SetExcInfo,3.7,, +function,PyErr_SetFromErrno,3.2,, +function,PyErr_SetFromErrnoWithFilename,3.2,, +function,PyErr_SetFromErrnoWithFilenameObject,3.2,, +function,PyErr_SetFromErrnoWithFilenameObjects,3.7,, +function,PyErr_SetFromWindowsErr,3.7,on Windows, +function,PyErr_SetFromWindowsErrWithFilename,3.7,on Windows, +function,PyErr_SetImportError,3.7,, +function,PyErr_SetImportErrorSubclass,3.6,, +function,PyErr_SetInterrupt,3.2,, +function,PyErr_SetInterruptEx,3.10,, +function,PyErr_SetNone,3.2,, +function,PyErr_SetObject,3.2,, +function,PyErr_SetString,3.2,, +function,PyErr_SyntaxLocation,3.2,, +function,PyErr_SyntaxLocationEx,3.7,, +function,PyErr_WarnEx,3.2,, +function,PyErr_WarnExplicit,3.2,, +function,PyErr_WarnFormat,3.2,, +function,PyErr_WriteUnraisable,3.2,, +function,PyEval_AcquireLock,3.2,, +function,PyEval_AcquireThread,3.2,, +function,PyEval_CallFunction,3.2,, +function,PyEval_CallMethod,3.2,, +function,PyEval_CallObjectWithKeywords,3.2,, +function,PyEval_EvalCode,3.2,, +function,PyEval_EvalCodeEx,3.2,, +function,PyEval_EvalFrame,3.2,, +function,PyEval_EvalFrameEx,3.2,, +function,PyEval_GetBuiltins,3.2,, +function,PyEval_GetFrame,3.2,, +function,PyEval_GetFuncDesc,3.2,, +function,PyEval_GetFuncName,3.2,, +function,PyEval_GetGlobals,3.2,, +function,PyEval_GetLocals,3.2,, +function,PyEval_InitThreads,3.2,, +function,PyEval_ReleaseLock,3.2,, +function,PyEval_ReleaseThread,3.2,, +function,PyEval_RestoreThread,3.2,, +function,PyEval_SaveThread,3.2,, +function,PyEval_ThreadsInitialized,3.2,, +var,PyExc_ArithmeticError,3.2,, +var,PyExc_AssertionError,3.2,, +var,PyExc_AttributeError,3.2,, +var,PyExc_BaseException,3.2,, +var,PyExc_BaseExceptionGroup,3.11,, +var,PyExc_BlockingIOError,3.7,, +var,PyExc_BrokenPipeError,3.7,, +var,PyExc_BufferError,3.2,, +var,PyExc_BytesWarning,3.2,, +var,PyExc_ChildProcessError,3.7,, +var,PyExc_ConnectionAbortedError,3.7,, +var,PyExc_ConnectionError,3.7,, +var,PyExc_ConnectionRefusedError,3.7,, +var,PyExc_ConnectionResetError,3.7,, +var,PyExc_DeprecationWarning,3.2,, +var,PyExc_EOFError,3.2,, +var,PyExc_EncodingWarning,3.10,, +var,PyExc_EnvironmentError,3.2,, +var,PyExc_Exception,3.2,, +var,PyExc_FileExistsError,3.7,, +var,PyExc_FileNotFoundError,3.7,, +var,PyExc_FloatingPointError,3.2,, +var,PyExc_FutureWarning,3.2,, +var,PyExc_GeneratorExit,3.2,, +var,PyExc_IOError,3.2,, +var,PyExc_ImportError,3.2,, +var,PyExc_ImportWarning,3.2,, +var,PyExc_IndentationError,3.2,, +var,PyExc_IndexError,3.2,, +var,PyExc_InterruptedError,3.7,, +var,PyExc_IsADirectoryError,3.7,, +var,PyExc_KeyError,3.2,, +var,PyExc_KeyboardInterrupt,3.2,, +var,PyExc_LookupError,3.2,, +var,PyExc_MemoryError,3.2,, +var,PyExc_ModuleNotFoundError,3.6,, +var,PyExc_NameError,3.2,, +var,PyExc_NotADirectoryError,3.7,, +var,PyExc_NotImplementedError,3.2,, +var,PyExc_OSError,3.2,, +var,PyExc_OverflowError,3.2,, +var,PyExc_PendingDeprecationWarning,3.2,, +var,PyExc_PermissionError,3.7,, +var,PyExc_ProcessLookupError,3.7,, +var,PyExc_RecursionError,3.7,, +var,PyExc_ReferenceError,3.2,, +var,PyExc_ResourceWarning,3.7,, +var,PyExc_RuntimeError,3.2,, +var,PyExc_RuntimeWarning,3.2,, +var,PyExc_StopAsyncIteration,3.7,, +var,PyExc_StopIteration,3.2,, +var,PyExc_SyntaxError,3.2,, +var,PyExc_SyntaxWarning,3.2,, +var,PyExc_SystemError,3.2,, +var,PyExc_SystemExit,3.2,, +var,PyExc_TabError,3.2,, +var,PyExc_TimeoutError,3.7,, +var,PyExc_TypeError,3.2,, +var,PyExc_UnboundLocalError,3.2,, +var,PyExc_UnicodeDecodeError,3.2,, +var,PyExc_UnicodeEncodeError,3.2,, +var,PyExc_UnicodeError,3.2,, +var,PyExc_UnicodeTranslateError,3.2,, +var,PyExc_UnicodeWarning,3.2,, +var,PyExc_UserWarning,3.2,, +var,PyExc_ValueError,3.2,, +var,PyExc_Warning,3.2,, +var,PyExc_WindowsError,3.7,on Windows, +var,PyExc_ZeroDivisionError,3.2,, +function,PyExceptionClass_Name,3.8,, +function,PyException_GetCause,3.2,, +function,PyException_GetContext,3.2,, +function,PyException_GetTraceback,3.2,, +function,PyException_SetCause,3.2,, +function,PyException_SetContext,3.2,, +function,PyException_SetTraceback,3.2,, +function,PyFile_FromFd,3.2,, +function,PyFile_GetLine,3.2,, +function,PyFile_WriteObject,3.2,, +function,PyFile_WriteString,3.2,, +var,PyFilter_Type,3.2,, +function,PyFloat_AsDouble,3.2,, +function,PyFloat_FromDouble,3.2,, +function,PyFloat_FromString,3.2,, +function,PyFloat_GetInfo,3.2,, +function,PyFloat_GetMax,3.2,, +function,PyFloat_GetMin,3.2,, +var,PyFloat_Type,3.2,, +type,PyFrameObject,3.2,,opaque +function,PyFrame_GetCode,3.10,, +function,PyFrame_GetLineNumber,3.10,, +function,PyFrozenSet_New,3.2,, +var,PyFrozenSet_Type,3.2,, +function,PyGC_Collect,3.2,, +function,PyGC_Disable,3.10,, +function,PyGC_Enable,3.10,, +function,PyGC_IsEnabled,3.10,, +function,PyGILState_Ensure,3.2,, +function,PyGILState_GetThisThreadState,3.2,, +function,PyGILState_Release,3.2,, +type,PyGILState_STATE,3.2,, +type,PyGetSetDef,3.2,,full-abi +var,PyGetSetDescr_Type,3.2,, +function,PyImport_AddModule,3.2,, +function,PyImport_AddModuleObject,3.7,, +function,PyImport_AppendInittab,3.2,, +function,PyImport_ExecCodeModule,3.2,, +function,PyImport_ExecCodeModuleEx,3.2,, +function,PyImport_ExecCodeModuleObject,3.7,, +function,PyImport_ExecCodeModuleWithPathnames,3.2,, +function,PyImport_GetImporter,3.2,, +function,PyImport_GetMagicNumber,3.2,, +function,PyImport_GetMagicTag,3.2,, +function,PyImport_GetModule,3.8,, +function,PyImport_GetModuleDict,3.2,, +function,PyImport_Import,3.2,, +function,PyImport_ImportFrozenModule,3.2,, +function,PyImport_ImportFrozenModuleObject,3.7,, +function,PyImport_ImportModule,3.2,, +function,PyImport_ImportModuleLevel,3.2,, +function,PyImport_ImportModuleLevelObject,3.7,, +function,PyImport_ImportModuleNoBlock,3.2,, +function,PyImport_ReloadModule,3.2,, +function,PyIndex_Check,3.8,, +type,PyInterpreterState,3.2,,opaque +function,PyInterpreterState_Clear,3.2,, +function,PyInterpreterState_Delete,3.2,, +function,PyInterpreterState_Get,3.9,, +function,PyInterpreterState_GetDict,3.8,, +function,PyInterpreterState_GetID,3.7,, +function,PyInterpreterState_New,3.2,, +function,PyIter_Check,3.8,, +function,PyIter_Next,3.2,, +function,PyIter_Send,3.10,, +var,PyListIter_Type,3.2,, +var,PyListRevIter_Type,3.2,, +function,PyList_Append,3.2,, +function,PyList_AsTuple,3.2,, +function,PyList_GetItem,3.2,, +function,PyList_GetSlice,3.2,, +function,PyList_Insert,3.2,, +function,PyList_New,3.2,, +function,PyList_Reverse,3.2,, +function,PyList_SetItem,3.2,, +function,PyList_SetSlice,3.2,, +function,PyList_Size,3.2,, +function,PyList_Sort,3.2,, +var,PyList_Type,3.2,, +type,PyLongObject,3.2,,opaque +var,PyLongRangeIter_Type,3.2,, +function,PyLong_AsDouble,3.2,, +function,PyLong_AsLong,3.2,, +function,PyLong_AsLongAndOverflow,3.2,, +function,PyLong_AsLongLong,3.2,, +function,PyLong_AsLongLongAndOverflow,3.2,, +function,PyLong_AsSize_t,3.2,, +function,PyLong_AsSsize_t,3.2,, +function,PyLong_AsUnsignedLong,3.2,, +function,PyLong_AsUnsignedLongLong,3.2,, +function,PyLong_AsUnsignedLongLongMask,3.2,, +function,PyLong_AsUnsignedLongMask,3.2,, +function,PyLong_AsVoidPtr,3.2,, +function,PyLong_FromDouble,3.2,, +function,PyLong_FromLong,3.2,, +function,PyLong_FromLongLong,3.2,, +function,PyLong_FromSize_t,3.2,, +function,PyLong_FromSsize_t,3.2,, +function,PyLong_FromString,3.2,, +function,PyLong_FromUnsignedLong,3.2,, +function,PyLong_FromUnsignedLongLong,3.2,, +function,PyLong_FromVoidPtr,3.2,, +function,PyLong_GetInfo,3.2,, +var,PyLong_Type,3.2,, +var,PyMap_Type,3.2,, +function,PyMapping_Check,3.2,, +function,PyMapping_GetItemString,3.2,, +function,PyMapping_HasKey,3.2,, +function,PyMapping_HasKeyString,3.2,, +function,PyMapping_Items,3.2,, +function,PyMapping_Keys,3.2,, +function,PyMapping_Length,3.2,, +function,PyMapping_SetItemString,3.2,, +function,PyMapping_Size,3.2,, +function,PyMapping_Values,3.2,, +function,PyMem_Calloc,3.7,, +function,PyMem_Free,3.2,, +function,PyMem_Malloc,3.2,, +function,PyMem_Realloc,3.2,, +type,PyMemberDef,3.2,,full-abi +var,PyMemberDescr_Type,3.2,, +function,PyMemoryView_FromBuffer,3.11,, +function,PyMemoryView_FromMemory,3.7,, +function,PyMemoryView_FromObject,3.2,, +function,PyMemoryView_GetContiguous,3.2,, +var,PyMemoryView_Type,3.2,, +type,PyMethodDef,3.2,,full-abi +var,PyMethodDescr_Type,3.2,, +type,PyModuleDef,3.2,,full-abi +type,PyModuleDef_Base,3.2,,full-abi +function,PyModuleDef_Init,3.5,, +var,PyModuleDef_Type,3.5,, +function,PyModule_AddFunctions,3.7,, +function,PyModule_AddIntConstant,3.2,, +function,PyModule_AddObject,3.2,, +function,PyModule_AddObjectRef,3.10,, +function,PyModule_AddStringConstant,3.2,, +function,PyModule_AddType,3.10,, +function,PyModule_Create2,3.2,, +function,PyModule_ExecDef,3.7,, +function,PyModule_FromDefAndSpec2,3.7,, +function,PyModule_GetDef,3.2,, +function,PyModule_GetDict,3.2,, +function,PyModule_GetFilename,3.2,, +function,PyModule_GetFilenameObject,3.2,, +function,PyModule_GetName,3.2,, +function,PyModule_GetNameObject,3.7,, +function,PyModule_GetState,3.2,, +function,PyModule_New,3.2,, +function,PyModule_NewObject,3.7,, +function,PyModule_SetDocString,3.7,, +var,PyModule_Type,3.2,, +function,PyNumber_Absolute,3.2,, +function,PyNumber_Add,3.2,, +function,PyNumber_And,3.2,, +function,PyNumber_AsSsize_t,3.2,, +function,PyNumber_Check,3.2,, +function,PyNumber_Divmod,3.2,, +function,PyNumber_Float,3.2,, +function,PyNumber_FloorDivide,3.2,, +function,PyNumber_InPlaceAdd,3.2,, +function,PyNumber_InPlaceAnd,3.2,, +function,PyNumber_InPlaceFloorDivide,3.2,, +function,PyNumber_InPlaceLshift,3.2,, +function,PyNumber_InPlaceMatrixMultiply,3.7,, +function,PyNumber_InPlaceMultiply,3.2,, +function,PyNumber_InPlaceOr,3.2,, +function,PyNumber_InPlacePower,3.2,, +function,PyNumber_InPlaceRemainder,3.2,, +function,PyNumber_InPlaceRshift,3.2,, +function,PyNumber_InPlaceSubtract,3.2,, +function,PyNumber_InPlaceTrueDivide,3.2,, +function,PyNumber_InPlaceXor,3.2,, +function,PyNumber_Index,3.2,, +function,PyNumber_Invert,3.2,, +function,PyNumber_Long,3.2,, +function,PyNumber_Lshift,3.2,, +function,PyNumber_MatrixMultiply,3.7,, +function,PyNumber_Multiply,3.2,, +function,PyNumber_Negative,3.2,, +function,PyNumber_Or,3.2,, +function,PyNumber_Positive,3.2,, +function,PyNumber_Power,3.2,, +function,PyNumber_Remainder,3.2,, +function,PyNumber_Rshift,3.2,, +function,PyNumber_Subtract,3.2,, +function,PyNumber_ToBase,3.2,, +function,PyNumber_TrueDivide,3.2,, +function,PyNumber_Xor,3.2,, +function,PyOS_AfterFork,3.2,on platforms with fork(), +function,PyOS_AfterFork_Child,3.7,on platforms with fork(), +function,PyOS_AfterFork_Parent,3.7,on platforms with fork(), +function,PyOS_BeforeFork,3.7,on platforms with fork(), +function,PyOS_CheckStack,3.7,on platforms with USE_STACKCHECK, +function,PyOS_FSPath,3.6,, +var,PyOS_InputHook,3.2,, +function,PyOS_InterruptOccurred,3.2,, +function,PyOS_double_to_string,3.2,, +function,PyOS_getsig,3.2,, +function,PyOS_mystricmp,3.2,, +function,PyOS_mystrnicmp,3.2,, +function,PyOS_setsig,3.2,, +type,PyOS_sighandler_t,3.2,, +function,PyOS_snprintf,3.2,, +function,PyOS_string_to_double,3.2,, +function,PyOS_strtol,3.2,, +function,PyOS_strtoul,3.2,, +function,PyOS_vsnprintf,3.2,, +type,PyObject,3.2,,members +member,PyObject.ob_refcnt,3.2,, +member,PyObject.ob_type,3.2,, +function,PyObject_ASCII,3.2,, +function,PyObject_AsCharBuffer,3.2,, +function,PyObject_AsFileDescriptor,3.2,, +function,PyObject_AsReadBuffer,3.2,, +function,PyObject_AsWriteBuffer,3.2,, +function,PyObject_Bytes,3.2,, +function,PyObject_Call,3.2,, +function,PyObject_CallFunction,3.2,, +function,PyObject_CallFunctionObjArgs,3.2,, +function,PyObject_CallMethod,3.2,, +function,PyObject_CallMethodObjArgs,3.2,, +function,PyObject_CallNoArgs,3.10,, +function,PyObject_CallObject,3.2,, +function,PyObject_Calloc,3.7,, +function,PyObject_CheckBuffer,3.11,, +function,PyObject_CheckReadBuffer,3.2,, +function,PyObject_ClearWeakRefs,3.2,, +function,PyObject_CopyData,3.11,, +function,PyObject_DelItem,3.2,, +function,PyObject_DelItemString,3.2,, +function,PyObject_Dir,3.2,, +function,PyObject_Format,3.2,, +function,PyObject_Free,3.2,, +function,PyObject_GC_Del,3.2,, +function,PyObject_GC_IsFinalized,3.9,, +function,PyObject_GC_IsTracked,3.9,, +function,PyObject_GC_Track,3.2,, +function,PyObject_GC_UnTrack,3.2,, +function,PyObject_GenericGetAttr,3.2,, +function,PyObject_GenericGetDict,3.10,, +function,PyObject_GenericSetAttr,3.2,, +function,PyObject_GenericSetDict,3.7,, +function,PyObject_GetAIter,3.10,, +function,PyObject_GetAttr,3.2,, +function,PyObject_GetAttrString,3.2,, +function,PyObject_GetBuffer,3.11,, +function,PyObject_GetItem,3.2,, +function,PyObject_GetIter,3.2,, +function,PyObject_HasAttr,3.2,, +function,PyObject_HasAttrString,3.2,, +function,PyObject_Hash,3.2,, +function,PyObject_HashNotImplemented,3.2,, +function,PyObject_Init,3.2,, +function,PyObject_InitVar,3.2,, +function,PyObject_IsInstance,3.2,, +function,PyObject_IsSubclass,3.2,, +function,PyObject_IsTrue,3.2,, +function,PyObject_Length,3.2,, +function,PyObject_Malloc,3.2,, +function,PyObject_Not,3.2,, +function,PyObject_Realloc,3.2,, +function,PyObject_Repr,3.2,, +function,PyObject_RichCompare,3.2,, +function,PyObject_RichCompareBool,3.2,, +function,PyObject_SelfIter,3.2,, +function,PyObject_SetAttr,3.2,, +function,PyObject_SetAttrString,3.2,, +function,PyObject_SetItem,3.2,, +function,PyObject_Size,3.2,, +function,PyObject_Str,3.2,, +function,PyObject_Type,3.2,, +var,PyProperty_Type,3.2,, +var,PyRangeIter_Type,3.2,, +var,PyRange_Type,3.2,, +var,PyReversed_Type,3.2,, +function,PySeqIter_New,3.2,, +var,PySeqIter_Type,3.2,, +function,PySequence_Check,3.2,, +function,PySequence_Concat,3.2,, +function,PySequence_Contains,3.2,, +function,PySequence_Count,3.2,, +function,PySequence_DelItem,3.2,, +function,PySequence_DelSlice,3.2,, +function,PySequence_Fast,3.2,, +function,PySequence_GetItem,3.2,, +function,PySequence_GetSlice,3.2,, +function,PySequence_In,3.2,, +function,PySequence_InPlaceConcat,3.2,, +function,PySequence_InPlaceRepeat,3.2,, +function,PySequence_Index,3.2,, +function,PySequence_Length,3.2,, +function,PySequence_List,3.2,, +function,PySequence_Repeat,3.2,, +function,PySequence_SetItem,3.2,, +function,PySequence_SetSlice,3.2,, +function,PySequence_Size,3.2,, +function,PySequence_Tuple,3.2,, +var,PySetIter_Type,3.2,, +function,PySet_Add,3.2,, +function,PySet_Clear,3.2,, +function,PySet_Contains,3.2,, +function,PySet_Discard,3.2,, +function,PySet_New,3.2,, +function,PySet_Pop,3.2,, +function,PySet_Size,3.2,, +var,PySet_Type,3.2,, +function,PySlice_AdjustIndices,3.7,, +function,PySlice_GetIndices,3.2,, +function,PySlice_GetIndicesEx,3.2,, +function,PySlice_New,3.2,, +var,PySlice_Type,3.2,, +function,PySlice_Unpack,3.7,, +function,PyState_AddModule,3.3,, +function,PyState_FindModule,3.2,, +function,PyState_RemoveModule,3.3,, +type,PyStructSequence_Desc,3.2,,full-abi +type,PyStructSequence_Field,3.2,,full-abi +function,PyStructSequence_GetItem,3.2,, +function,PyStructSequence_New,3.2,, +function,PyStructSequence_NewType,3.2,, +function,PyStructSequence_SetItem,3.2,, +var,PyStructSequence_UnnamedField,3.11,, +var,PySuper_Type,3.2,, +function,PySys_AddWarnOption,3.2,, +function,PySys_AddWarnOptionUnicode,3.2,, +function,PySys_AddXOption,3.7,, +function,PySys_FormatStderr,3.2,, +function,PySys_FormatStdout,3.2,, +function,PySys_GetObject,3.2,, +function,PySys_GetXOptions,3.7,, +function,PySys_HasWarnOptions,3.2,, +function,PySys_ResetWarnOptions,3.2,, +function,PySys_SetArgv,3.2,, +function,PySys_SetArgvEx,3.2,, +function,PySys_SetObject,3.2,, +function,PySys_SetPath,3.2,, +function,PySys_WriteStderr,3.2,, +function,PySys_WriteStdout,3.2,, +type,PyThreadState,3.2,,opaque +function,PyThreadState_Clear,3.2,, +function,PyThreadState_Delete,3.2,, +function,PyThreadState_Get,3.2,, +function,PyThreadState_GetDict,3.2,, +function,PyThreadState_GetFrame,3.10,, +function,PyThreadState_GetID,3.10,, +function,PyThreadState_GetInterpreter,3.10,, +function,PyThreadState_New,3.2,, +function,PyThreadState_SetAsyncExc,3.2,, +function,PyThreadState_Swap,3.2,, +function,PyThread_GetInfo,3.3,, +function,PyThread_ReInitTLS,3.2,, +function,PyThread_acquire_lock,3.2,, +function,PyThread_acquire_lock_timed,3.2,, +function,PyThread_allocate_lock,3.2,, +function,PyThread_create_key,3.2,, +function,PyThread_delete_key,3.2,, +function,PyThread_delete_key_value,3.2,, +function,PyThread_exit_thread,3.2,, +function,PyThread_free_lock,3.2,, +function,PyThread_get_key_value,3.2,, +function,PyThread_get_stacksize,3.2,, +function,PyThread_get_thread_ident,3.2,, +function,PyThread_get_thread_native_id,3.2,, +function,PyThread_init_thread,3.2,, +function,PyThread_release_lock,3.2,, +function,PyThread_set_key_value,3.2,, +function,PyThread_set_stacksize,3.2,, +function,PyThread_start_new_thread,3.2,, +function,PyThread_tss_alloc,3.7,, +function,PyThread_tss_create,3.7,, +function,PyThread_tss_delete,3.7,, +function,PyThread_tss_free,3.7,, +function,PyThread_tss_get,3.7,, +function,PyThread_tss_is_created,3.7,, +function,PyThread_tss_set,3.7,, +function,PyTraceBack_Here,3.2,, +function,PyTraceBack_Print,3.2,, +var,PyTraceBack_Type,3.2,, +var,PyTupleIter_Type,3.2,, +function,PyTuple_GetItem,3.2,, +function,PyTuple_GetSlice,3.2,, +function,PyTuple_New,3.2,, +function,PyTuple_Pack,3.2,, +function,PyTuple_SetItem,3.2,, +function,PyTuple_Size,3.2,, +var,PyTuple_Type,3.2,, +type,PyTypeObject,3.2,,opaque +function,PyType_ClearCache,3.2,, +function,PyType_FromModuleAndSpec,3.10,, +function,PyType_FromSpec,3.2,, +function,PyType_FromSpecWithBases,3.3,, +function,PyType_GenericAlloc,3.2,, +function,PyType_GenericNew,3.2,, +function,PyType_GetFlags,3.2,, +function,PyType_GetModule,3.10,, +function,PyType_GetModuleState,3.10,, +function,PyType_GetName,3.11,, +function,PyType_GetQualName,3.11,, +function,PyType_GetSlot,3.4,, +function,PyType_IsSubtype,3.2,, +function,PyType_Modified,3.2,, +function,PyType_Ready,3.2,, +type,PyType_Slot,3.2,,full-abi +type,PyType_Spec,3.2,,full-abi +var,PyType_Type,3.2,, +function,PyUnicodeDecodeError_Create,3.2,, +function,PyUnicodeDecodeError_GetEncoding,3.2,, +function,PyUnicodeDecodeError_GetEnd,3.2,, +function,PyUnicodeDecodeError_GetObject,3.2,, +function,PyUnicodeDecodeError_GetReason,3.2,, +function,PyUnicodeDecodeError_GetStart,3.2,, +function,PyUnicodeDecodeError_SetEnd,3.2,, +function,PyUnicodeDecodeError_SetReason,3.2,, +function,PyUnicodeDecodeError_SetStart,3.2,, +function,PyUnicodeEncodeError_GetEncoding,3.2,, +function,PyUnicodeEncodeError_GetEnd,3.2,, +function,PyUnicodeEncodeError_GetObject,3.2,, +function,PyUnicodeEncodeError_GetReason,3.2,, +function,PyUnicodeEncodeError_GetStart,3.2,, +function,PyUnicodeEncodeError_SetEnd,3.2,, +function,PyUnicodeEncodeError_SetReason,3.2,, +function,PyUnicodeEncodeError_SetStart,3.2,, +var,PyUnicodeIter_Type,3.2,, +function,PyUnicodeTranslateError_GetEnd,3.2,, +function,PyUnicodeTranslateError_GetObject,3.2,, +function,PyUnicodeTranslateError_GetReason,3.2,, +function,PyUnicodeTranslateError_GetStart,3.2,, +function,PyUnicodeTranslateError_SetEnd,3.2,, +function,PyUnicodeTranslateError_SetReason,3.2,, +function,PyUnicodeTranslateError_SetStart,3.2,, +function,PyUnicode_Append,3.2,, +function,PyUnicode_AppendAndDel,3.2,, +function,PyUnicode_AsASCIIString,3.2,, +function,PyUnicode_AsCharmapString,3.2,, +function,PyUnicode_AsDecodedObject,3.2,, +function,PyUnicode_AsDecodedUnicode,3.2,, +function,PyUnicode_AsEncodedObject,3.2,, +function,PyUnicode_AsEncodedString,3.2,, +function,PyUnicode_AsEncodedUnicode,3.2,, +function,PyUnicode_AsLatin1String,3.2,, +function,PyUnicode_AsMBCSString,3.7,on Windows, +function,PyUnicode_AsRawUnicodeEscapeString,3.2,, +function,PyUnicode_AsUCS4,3.7,, +function,PyUnicode_AsUCS4Copy,3.7,, +function,PyUnicode_AsUTF16String,3.2,, +function,PyUnicode_AsUTF32String,3.2,, +function,PyUnicode_AsUTF8AndSize,3.10,, +function,PyUnicode_AsUTF8String,3.2,, +function,PyUnicode_AsUnicodeEscapeString,3.2,, +function,PyUnicode_AsWideChar,3.2,, +function,PyUnicode_AsWideCharString,3.7,, +function,PyUnicode_BuildEncodingMap,3.2,, +function,PyUnicode_Compare,3.2,, +function,PyUnicode_CompareWithASCIIString,3.2,, +function,PyUnicode_Concat,3.2,, +function,PyUnicode_Contains,3.2,, +function,PyUnicode_Count,3.2,, +function,PyUnicode_Decode,3.2,, +function,PyUnicode_DecodeASCII,3.2,, +function,PyUnicode_DecodeCharmap,3.2,, +function,PyUnicode_DecodeCodePageStateful,3.7,on Windows, +function,PyUnicode_DecodeFSDefault,3.2,, +function,PyUnicode_DecodeFSDefaultAndSize,3.2,, +function,PyUnicode_DecodeLatin1,3.2,, +function,PyUnicode_DecodeLocale,3.7,, +function,PyUnicode_DecodeLocaleAndSize,3.7,, +function,PyUnicode_DecodeMBCS,3.7,on Windows, +function,PyUnicode_DecodeMBCSStateful,3.7,on Windows, +function,PyUnicode_DecodeRawUnicodeEscape,3.2,, +function,PyUnicode_DecodeUTF16,3.2,, +function,PyUnicode_DecodeUTF16Stateful,3.2,, +function,PyUnicode_DecodeUTF32,3.2,, +function,PyUnicode_DecodeUTF32Stateful,3.2,, +function,PyUnicode_DecodeUTF7,3.2,, +function,PyUnicode_DecodeUTF7Stateful,3.2,, +function,PyUnicode_DecodeUTF8,3.2,, +function,PyUnicode_DecodeUTF8Stateful,3.2,, +function,PyUnicode_DecodeUnicodeEscape,3.2,, +function,PyUnicode_EncodeCodePage,3.7,on Windows, +function,PyUnicode_EncodeFSDefault,3.2,, +function,PyUnicode_EncodeLocale,3.7,, +function,PyUnicode_FSConverter,3.2,, +function,PyUnicode_FSDecoder,3.2,, +function,PyUnicode_Find,3.2,, +function,PyUnicode_FindChar,3.7,, +function,PyUnicode_Format,3.2,, +function,PyUnicode_FromEncodedObject,3.2,, +function,PyUnicode_FromFormat,3.2,, +function,PyUnicode_FromFormatV,3.2,, +function,PyUnicode_FromObject,3.2,, +function,PyUnicode_FromOrdinal,3.2,, +function,PyUnicode_FromString,3.2,, +function,PyUnicode_FromStringAndSize,3.2,, +function,PyUnicode_FromWideChar,3.2,, +function,PyUnicode_GetDefaultEncoding,3.2,, +function,PyUnicode_GetLength,3.7,, +function,PyUnicode_GetSize,3.2,, +function,PyUnicode_InternFromString,3.2,, +function,PyUnicode_InternImmortal,3.2,, +function,PyUnicode_InternInPlace,3.2,, +function,PyUnicode_IsIdentifier,3.2,, +function,PyUnicode_Join,3.2,, +function,PyUnicode_Partition,3.2,, +function,PyUnicode_RPartition,3.2,, +function,PyUnicode_RSplit,3.2,, +function,PyUnicode_ReadChar,3.7,, +function,PyUnicode_Replace,3.2,, +function,PyUnicode_Resize,3.2,, +function,PyUnicode_RichCompare,3.2,, +function,PyUnicode_Split,3.2,, +function,PyUnicode_Splitlines,3.2,, +function,PyUnicode_Substring,3.7,, +function,PyUnicode_Tailmatch,3.2,, +function,PyUnicode_Translate,3.2,, +var,PyUnicode_Type,3.2,, +function,PyUnicode_WriteChar,3.7,, +type,PyVarObject,3.2,,members +member,PyVarObject.ob_base,3.2,, +member,PyVarObject.ob_size,3.2,, +type,PyWeakReference,3.2,,opaque +function,PyWeakref_GetObject,3.2,, +function,PyWeakref_NewProxy,3.2,, +function,PyWeakref_NewRef,3.2,, +var,PyWrapperDescr_Type,3.2,, +function,PyWrapper_New,3.2,, +var,PyZip_Type,3.2,, +function,Py_AddPendingCall,3.2,, +function,Py_AtExit,3.2,, +macro,Py_BEGIN_ALLOW_THREADS,3.2,, +macro,Py_BLOCK_THREADS,3.2,, +function,Py_BuildValue,3.2,, +function,Py_BytesMain,3.8,, +function,Py_CompileString,3.2,, +function,Py_DecRef,3.2,, +function,Py_DecodeLocale,3.7,, +macro,Py_END_ALLOW_THREADS,3.2,, +function,Py_EncodeLocale,3.7,, +function,Py_EndInterpreter,3.2,, +function,Py_EnterRecursiveCall,3.9,, +function,Py_Exit,3.2,, +function,Py_FatalError,3.2,, +var,Py_FileSystemDefaultEncodeErrors,3.10,, +var,Py_FileSystemDefaultEncoding,3.2,, +function,Py_Finalize,3.2,, +function,Py_FinalizeEx,3.6,, +function,Py_GenericAlias,3.9,, +var,Py_GenericAliasType,3.9,, +function,Py_GetBuildInfo,3.2,, +function,Py_GetCompiler,3.2,, +function,Py_GetCopyright,3.2,, +function,Py_GetExecPrefix,3.2,, +function,Py_GetPath,3.2,, +function,Py_GetPlatform,3.2,, +function,Py_GetPrefix,3.2,, +function,Py_GetProgramFullPath,3.2,, +function,Py_GetProgramName,3.2,, +function,Py_GetPythonHome,3.2,, +function,Py_GetRecursionLimit,3.2,, +function,Py_GetVersion,3.2,, +var,Py_HasFileSystemDefaultEncoding,3.2,, +function,Py_IncRef,3.2,, +function,Py_Initialize,3.2,, +function,Py_InitializeEx,3.2,, +function,Py_Is,3.10,, +function,Py_IsFalse,3.10,, +function,Py_IsInitialized,3.2,, +function,Py_IsNone,3.10,, +function,Py_IsTrue,3.10,, +function,Py_LeaveRecursiveCall,3.9,, +function,Py_Main,3.2,, +function,Py_MakePendingCalls,3.2,, +function,Py_NewInterpreter,3.2,, +function,Py_NewRef,3.10,, +function,Py_ReprEnter,3.2,, +function,Py_ReprLeave,3.2,, +function,Py_SetPath,3.7,, +function,Py_SetProgramName,3.2,, +function,Py_SetPythonHome,3.2,, +function,Py_SetRecursionLimit,3.2,, +type,Py_UCS4,3.2,, +macro,Py_UNBLOCK_THREADS,3.2,, +var,Py_UTF8Mode,3.8,, +function,Py_VaBuildValue,3.2,, +var,Py_Version,3.11,, +function,Py_XNewRef,3.10,, +type,Py_buffer,3.11,,full-abi +type,Py_intptr_t,3.2,, +type,Py_ssize_t,3.2,, +type,Py_uintptr_t,3.2,, +type,allocfunc,3.2,, +type,binaryfunc,3.2,, +type,descrgetfunc,3.2,, +type,descrsetfunc,3.2,, +type,destructor,3.2,, +type,getattrfunc,3.2,, +type,getattrofunc,3.2,, +type,getiterfunc,3.2,, +type,getter,3.2,, +type,hashfunc,3.2,, +type,initproc,3.2,, +type,inquiry,3.2,, +type,iternextfunc,3.2,, +type,lenfunc,3.2,, +type,newfunc,3.2,, +type,objobjargproc,3.2,, +type,objobjproc,3.2,, +type,reprfunc,3.2,, +type,richcmpfunc,3.2,, +type,setattrfunc,3.2,, +type,setattrofunc,3.2,, +type,setter,3.2,, +type,ssizeargfunc,3.2,, +type,ssizeobjargproc,3.2,, +type,ssizessizeargfunc,3.2,, +type,ssizessizeobjargproc,3.2,, +type,symtable,3.2,,opaque +type,ternaryfunc,3.2,, +type,traverseproc,3.2,, +type,unaryfunc,3.2,, +type,visitproc,3.2,, diff --git a/Doc/tools/extensions/c_annotations.py b/Doc/tools/extensions/c_annotations.py index 489f06613eb31..9defb24a0331e 100644 --- a/Doc/tools/extensions/c_annotations.py +++ b/Doc/tools/extensions/c_annotations.py @@ -36,6 +36,7 @@ 'type': 'type', 'macro': 'macro', 'type': 'type', + 'member': 'member', } @@ -100,6 +101,12 @@ def add_annotations(self, app, doctree): # Stable ABI annotation. These have two forms: # Part of the [Stable ABI](link). # Part of the [Stable ABI](link) since version X.Y. + # For structs, there's some more info in the message: + # Part of the [Limited API](link) (as an opaque struct). + # Part of the [Stable ABI](link) (including all members). + # Part of the [Limited API](link) (Only some members are part + # of the stable ABI.). + # ... all of which can have "since version X.Y" appended. record = self.stable_abi_data.get(name) if record: if record['role'] != objtype: @@ -113,15 +120,27 @@ def add_annotations(self, app, doctree): ref_node = addnodes.pending_xref( 'Stable ABI', refdomain="std", reftarget='stable', reftype='ref', refexplicit="False") - ref_node += nodes.Text('Stable ABI') + struct_abi_kind = record['struct_abi_kind'] + if struct_abi_kind in {'opaque', 'members'}: + ref_node += nodes.Text('Limited API') + else: + ref_node += nodes.Text('Stable ABI') emph_node += ref_node + if struct_abi_kind == 'opaque': + emph_node += nodes.Text(' (as an opaque struct)') + elif struct_abi_kind == 'full-abi': + emph_node += nodes.Text(' (including all members)') if record['ifdef_note']: emph_node += nodes.Text(' ' + record['ifdef_note']) if stable_added == '3.2': # Stable ABI was introduced in 3.2. - emph_node += nodes.Text('.') + pass else: - emph_node += nodes.Text(f' since version {stable_added}.') + emph_node += nodes.Text(f' since version {stable_added}') + emph_node += nodes.Text('.') + if struct_abi_kind == 'members': + emph_node += nodes.Text( + ' (Only some members are part of the stable ABI.)') node.insert(0, emph_node) # Return value annotation diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index d58056394d864..96902cf61b46d 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -1214,8 +1214,10 @@ Porting to Python 3.11 which are not available in the limited C API. (Contributed by Victor Stinner in :issue:`46007`.) -* The :c:type:`PyFrameObject` structure member has been moved to the internal C - API headers. +.. _pyframeobject-3.11-hiding: + +* The :c:type:`PyFrameObject` structure members have been removed from the + public C API. While the documentation notes that the :c:type:`PyFrameObject` fields are subject to change at any time, they have been stable for a long time and were diff --git a/Misc/NEWS.d/next/Documentation/2022-03-30-17-08-12.bpo-47115.R3wt3i.rst b/Misc/NEWS.d/next/Documentation/2022-03-30-17-08-12.bpo-47115.R3wt3i.rst new file mode 100644 index 0000000000000..ac7b6dcb32d5f --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2022-03-30-17-08-12.bpo-47115.R3wt3i.rst @@ -0,0 +1,2 @@ +The documentation now lists which members of C structs are part of the +:ref:`Limited API/Stable ABI `. diff --git a/Misc/stable_abi.txt b/Misc/stable_abi.txt index cc3cc56d472d9..04d22603cc19b 100644 --- a/Misc/stable_abi.txt +++ b/Misc/stable_abi.txt @@ -42,6 +42,15 @@ # - a combination of the above (functions that were called by macros that # were public in the past) +# For structs, one of the following must be set: +# - opaque: The struct name is available in the Limited API, but its members +# are not. Users must manipulate it via pointers. +# - members: Space-separated list of members which are part of the +# Limited API and Stable ABI. +# Members that aren't listed are not accessible to applications. +# - full-abi: The entire struct -- all its members and its size -- is part of +# the Stable ABI, and must not change. + # Removing items from this file is generally not allowed, and additions should # be considered with that in mind. See the devguide for exact rules: # https://devguide.python.org/c-api/#limited-api @@ -54,40 +63,58 @@ struct PyObject added 3.2 + members ob_refcnt ob_type struct PyVarObject added 3.2 + members ob_base ob_size struct PyMethodDef added 3.2 + full-abi struct PyMemberDef added 3.2 + full-abi struct PyGetSetDef added 3.2 + full-abi struct PyModuleDef_Base added 3.2 + full-abi struct PyModuleDef added 3.2 + full-abi struct PyStructSequence_Field added 3.2 + full-abi struct PyStructSequence_Desc added 3.2 + full-abi struct PyType_Slot added 3.2 + full-abi struct PyType_Spec added 3.2 + full-abi struct PyThreadState added 3.2 + opaque struct PyInterpreterState added 3.2 + opaque struct PyFrameObject added 3.2 + opaque struct symtable added 3.2 + opaque struct PyWeakReference added 3.2 + opaque struct PyLongObject added 3.2 + opaque struct PyTypeObject added 3.2 + opaque function PyType_FromSpec added 3.2 @@ -299,11 +326,11 @@ typedef newfunc added 3.2 typedef allocfunc added 3.2 -struct PyCFunction +typedef PyCFunction added 3.2 -struct PyCFunctionWithKeywords +typedef PyCFunctionWithKeywords added 3.2 -struct PyCapsule_Destructor +typedef PyCapsule_Destructor added 3.2 typedef getter added 3.2 @@ -2194,6 +2221,7 @@ data PyStructSequence_UnnamedField # Add stable Py_buffer API in Python 3.11 (https://bugs.python.org/issue45459) struct Py_buffer added 3.11 + full-abi function PyObject_CheckBuffer added 3.11 function PyObject_GetBuffer diff --git a/Tools/scripts/stable_abi.py b/Tools/scripts/stable_abi.py index 2d951788d12c4..feca9a2ba5a4c 100755 --- a/Tools/scripts/stable_abi.py +++ b/Tools/scripts/stable_abi.py @@ -118,6 +118,8 @@ class ABIItem: contents: list = dataclasses.field(default_factory=list) abi_only: bool = False ifdef: str = None + struct_abi_kind: str = None + members: list = None KINDS = frozenset({ 'struct', 'function', 'macro', 'data', 'const', 'typedef', @@ -172,6 +174,15 @@ def raise_error(msg): if parent.kind not in {'function', 'data'}: raise_error(f'{kind} cannot go in {parent.kind}') parent.abi_only = True + elif kind in {'members', 'full-abi', 'opaque'}: + if parent.kind not in {'struct'}: + raise_error(f'{kind} cannot go in {parent.kind}') + if prev := getattr(parent, 'struct_abi_kind', None): + raise_error( + f'{parent.name} already has {prev}, cannot add {kind}') + parent.struct_abi_kind = kind + if kind == 'members': + parent.members = content.split() else: raise_error(f"unknown kind {kind!r}") # When adding more, update the comment in stable_abi.txt. @@ -246,7 +257,9 @@ def sort_key(item): def gen_doc_annotations(manifest, args, outfile): """Generate/check the stable ABI list for documentation annotations""" writer = csv.DictWriter( - outfile, ['role', 'name', 'added', 'ifdef_note'], lineterminator='\n') + outfile, + ['role', 'name', 'added', 'ifdef_note', 'struct_abi_kind'], + lineterminator='\n') writer.writeheader() for item in manifest.select(REST_ROLES.keys(), include_abi_only=False): if item.ifdef: @@ -257,7 +270,13 @@ def gen_doc_annotations(manifest, args, outfile): 'role': REST_ROLES[item.kind], 'name': item.name, 'added': item.added, - 'ifdef_note': ifdef_note}) + 'ifdef_note': ifdef_note, + 'struct_abi_kind': item.struct_abi_kind}) + for member_name in item.members or (): + writer.writerow({ + 'role': 'member', + 'name': f'{item.name}.{member_name}', + 'added': item.added}) @generator("ctypes_test", 'Lib/test/test_stable_abi_ctypes.py') def gen_ctypes_test(manifest, args, outfile): From webhook-mailer at python.org Wed Apr 6 11:07:15 2022 From: webhook-mailer at python.org (gpshead) Date: Wed, 06 Apr 2022 15:07:15 -0000 Subject: [Python-checkins] bpo-47235: Note where a typo is intentional in code. (GH-32348) Message-ID: https://github.com/python/cpython/commit/ac1fb07b6ecb6b93446484f52894914e5199de63 commit: ac1fb07b6ecb6b93446484f52894914e5199de63 branch: main author: Gregory P. Smith committer: gpshead date: 2022-04-06T08:06:58-07:00 summary: bpo-47235: Note where a typo is intentional in code. (GH-32348) People keep popping up reporting these as typos in the docs despite being described as typos in the surrounding text. Hopefully a comment on the line itself makes it more obvious? Arguably some of the typo examples are not using the "right" typo as the "assret" one in particular is now detected by default due to how common it was in actual code. But I don't want to to typo chasing by changing these examples to be other not yet auto-detected typos as they still illustrate the point well enough. files: M Doc/library/unittest.mock.rst diff --git a/Doc/library/unittest.mock.rst b/Doc/library/unittest.mock.rst index dbea928443975..a3700ac07f1c1 100644 --- a/Doc/library/unittest.mock.rst +++ b/Doc/library/unittest.mock.rst @@ -2550,7 +2550,7 @@ your assertion is gone: >>> mock = Mock(name='Thing', return_value=None) >>> mock(1, 2, 3) - >>> mock.assret_called_once_with(4, 5, 6) + >>> mock.assret_called_once_with(4, 5, 6) # Intentional typo! Your tests can pass silently and incorrectly because of the typo. @@ -2570,7 +2570,7 @@ attributes on the mock that exist on the real class: >>> from urllib import request >>> mock = Mock(spec=request.Request) - >>> mock.assret_called_with + >>> mock.assret_called_with # Intentional typo! Traceback (most recent call last): ... AttributeError: Mock object has no attribute 'assret_called_with' @@ -2582,7 +2582,7 @@ with any methods on the mock: >>> mock.has_data() - >>> mock.has_data.assret_called_with() + >>> mock.has_data.assret_called_with() # Intentional typo! Auto-speccing solves this problem. You can either pass ``autospec=True`` to :func:`patch` / :func:`patch.object` or use the :func:`create_autospec` function to create a @@ -2625,7 +2625,7 @@ any typos in our asserts will raise the correct error:: >>> req.add_header('spam', 'eggs') - >>> req.add_header.assret_called_with + >>> req.add_header.assret_called_with # Intentional typo! Traceback (most recent call last): ... AttributeError: Mock object has no attribute 'assret_called_with' From webhook-mailer at python.org Wed Apr 6 11:30:13 2022 From: webhook-mailer at python.org (miss-islington) Date: Wed, 06 Apr 2022 15:30:13 -0000 Subject: [Python-checkins] bpo-47235: Note where a typo is intentional in code. (GH-32348) Message-ID: https://github.com/python/cpython/commit/3856b4995ec0e632d47b733cdecb5183ac830568 commit: 3856b4995ec0e632d47b733cdecb5183ac830568 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-06T08:30:01-07:00 summary: bpo-47235: Note where a typo is intentional in code. (GH-32348) People keep popping up reporting these as typos in the docs despite being described as typos in the surrounding text. Hopefully a comment on the line itself makes it more obvious? Arguably some of the typo examples are not using the "right" typo as the "assret" one in particular is now detected by default due to how common it was in actual code. But I don't want to to typo chasing by changing these examples to be other not yet auto-detected typos as they still illustrate the point well enough. (cherry picked from commit ac1fb07b6ecb6b93446484f52894914e5199de63) Co-authored-by: Gregory P. Smith files: M Doc/library/unittest.mock.rst diff --git a/Doc/library/unittest.mock.rst b/Doc/library/unittest.mock.rst index 4d74f9cee84ee..86a848edab889 100644 --- a/Doc/library/unittest.mock.rst +++ b/Doc/library/unittest.mock.rst @@ -2550,7 +2550,7 @@ your assertion is gone: >>> mock = Mock(name='Thing', return_value=None) >>> mock(1, 2, 3) - >>> mock.assret_called_once_with(4, 5, 6) + >>> mock.assret_called_once_with(4, 5, 6) # Intentional typo! Your tests can pass silently and incorrectly because of the typo. @@ -2570,7 +2570,7 @@ attributes on the mock that exist on the real class: >>> from urllib import request >>> mock = Mock(spec=request.Request) - >>> mock.assret_called_with + >>> mock.assret_called_with # Intentional typo! Traceback (most recent call last): ... AttributeError: Mock object has no attribute 'assret_called_with' @@ -2582,7 +2582,7 @@ with any methods on the mock: >>> mock.has_data() - >>> mock.has_data.assret_called_with() + >>> mock.has_data.assret_called_with() # Intentional typo! Auto-speccing solves this problem. You can either pass ``autospec=True`` to :func:`patch` / :func:`patch.object` or use the :func:`create_autospec` function to create a @@ -2625,7 +2625,7 @@ any typos in our asserts will raise the correct error:: >>> req.add_header('spam', 'eggs') - >>> req.add_header.assret_called_with + >>> req.add_header.assret_called_with # Intentional typo! Traceback (most recent call last): ... AttributeError: Mock object has no attribute 'assret_called_with' From webhook-mailer at python.org Wed Apr 6 11:32:30 2022 From: webhook-mailer at python.org (miss-islington) Date: Wed, 06 Apr 2022 15:32:30 -0000 Subject: [Python-checkins] bpo-47235: Note where a typo is intentional in code. (GH-32348) Message-ID: https://github.com/python/cpython/commit/770780e63e2a99f842670ef901a257ba64293d6b commit: 770780e63e2a99f842670ef901a257ba64293d6b branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-06T08:32:16-07:00 summary: bpo-47235: Note where a typo is intentional in code. (GH-32348) People keep popping up reporting these as typos in the docs despite being described as typos in the surrounding text. Hopefully a comment on the line itself makes it more obvious? Arguably some of the typo examples are not using the "right" typo as the "assret" one in particular is now detected by default due to how common it was in actual code. But I don't want to to typo chasing by changing these examples to be other not yet auto-detected typos as they still illustrate the point well enough. (cherry picked from commit ac1fb07b6ecb6b93446484f52894914e5199de63) Co-authored-by: Gregory P. Smith files: M Doc/library/unittest.mock.rst diff --git a/Doc/library/unittest.mock.rst b/Doc/library/unittest.mock.rst index 3fbb1b6b83a76..537fbcf4e0f61 100644 --- a/Doc/library/unittest.mock.rst +++ b/Doc/library/unittest.mock.rst @@ -2549,7 +2549,7 @@ your assertion is gone: >>> mock = Mock(name='Thing', return_value=None) >>> mock(1, 2, 3) - >>> mock.assret_called_once_with(4, 5, 6) + >>> mock.assret_called_once_with(4, 5, 6) # Intentional typo! Your tests can pass silently and incorrectly because of the typo. @@ -2569,7 +2569,7 @@ attributes on the mock that exist on the real class: >>> from urllib import request >>> mock = Mock(spec=request.Request) - >>> mock.assret_called_with + >>> mock.assret_called_with # Intentional typo! Traceback (most recent call last): ... AttributeError: Mock object has no attribute 'assret_called_with' @@ -2581,7 +2581,7 @@ with any methods on the mock: >>> mock.has_data() - >>> mock.has_data.assret_called_with() + >>> mock.has_data.assret_called_with() # Intentional typo! Auto-speccing solves this problem. You can either pass ``autospec=True`` to :func:`patch` / :func:`patch.object` or use the :func:`create_autospec` function to create a @@ -2624,7 +2624,7 @@ any typos in our asserts will raise the correct error:: >>> req.add_header('spam', 'eggs') - >>> req.add_header.assret_called_with + >>> req.add_header.assret_called_with # Intentional typo! Traceback (most recent call last): ... AttributeError: Mock object has no attribute 'assret_called_with' From webhook-mailer at python.org Wed Apr 6 11:39:37 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Wed, 06 Apr 2022 15:39:37 -0000 Subject: [Python-checkins] stdtypes docs: fix typo (GH-32349) Message-ID: https://github.com/python/cpython/commit/b33c4564aceeae8323bcb19167fbbd2d5f5002bc commit: b33c4564aceeae8323bcb19167fbbd2d5f5002bc branch: main author: Ian <40774387+isteptoe at users.noreply.github.com> committer: JelleZijlstra date: 2022-04-06T08:39:17-07:00 summary: stdtypes docs: fix typo (GH-32349) files: M Doc/library/stdtypes.rst diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst index 6608b7b60ec50..d6d90cd462c22 100644 --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -3618,7 +3618,7 @@ The conversion types are: | | be used for Python2/3 code bases. | | +------------+-----------------------------------------------------+-------+ | ``'a'`` | Bytes (converts any Python object using | \(5) | -| | ``repr(obj).encode('ascii','backslashreplace)``). | | +| | ``repr(obj).encode('ascii', 'backslashreplace')``). | | +------------+-----------------------------------------------------+-------+ | ``'r'`` | ``'r'`` is an alias for ``'a'`` and should only | \(7) | | | be used for Python2/3 code bases. | | From webhook-mailer at python.org Wed Apr 6 11:40:55 2022 From: webhook-mailer at python.org (Fidget-Spinner) Date: Wed, 06 Apr 2022 15:40:55 -0000 Subject: [Python-checkins] Fix whitespace/indentation issues in test_sys (GH-32369) Message-ID: https://github.com/python/cpython/commit/da922409ac3e65c6bf2911401c7dfdf8ee6e0036 commit: da922409ac3e65c6bf2911401c7dfdf8ee6e0036 branch: main author: Ken Jin committer: Fidget-Spinner date: 2022-04-06T23:40:51+08:00 summary: Fix whitespace/indentation issues in test_sys (GH-32369) files: M Lib/test/test_sys.py diff --git a/Lib/test/test_sys.py b/Lib/test/test_sys.py index f4deb1763b95f..445ff893ff6fc 100644 --- a/Lib/test/test_sys.py +++ b/Lib/test/test_sys.py @@ -1179,8 +1179,8 @@ class X(Exception): with test.support.captured_stderr() as stderr, \ test.support.swap_attr(sys, 'unraisablehook', sys.__unraisablehook__): - expected = self.write_unraisable_exc( - A.B.X(), "msg", "obj"); + expected = self.write_unraisable_exc( + A.B.X(), "msg", "obj"); report = stderr.getvalue() self.assertIn(A.B.X.__qualname__, report) if moduleName in ['builtins', '__main__']: From webhook-mailer at python.org Wed Apr 6 12:04:56 2022 From: webhook-mailer at python.org (miss-islington) Date: Wed, 06 Apr 2022 16:04:56 -0000 Subject: [Python-checkins] stdtypes docs: fix typo (GH-32349) Message-ID: https://github.com/python/cpython/commit/b5b3b9151ce0112502e0dc3b12503c28341aca31 commit: b5b3b9151ce0112502e0dc3b12503c28341aca31 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-06T09:04:38-07:00 summary: stdtypes docs: fix typo (GH-32349) (cherry picked from commit b33c4564aceeae8323bcb19167fbbd2d5f5002bc) Co-authored-by: Ian <40774387+isteptoe at users.noreply.github.com> files: M Doc/library/stdtypes.rst diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst index 4be4d1daa831d..d96fb1f2c9ca6 100644 --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -3546,7 +3546,7 @@ The conversion types are: | | be used for Python2/3 code bases. | | +------------+-----------------------------------------------------+-------+ | ``'a'`` | Bytes (converts any Python object using | \(5) | -| | ``repr(obj).encode('ascii','backslashreplace)``). | | +| | ``repr(obj).encode('ascii', 'backslashreplace')``). | | +------------+-----------------------------------------------------+-------+ | ``'r'`` | ``'r'`` is an alias for ``'a'`` and should only | \(7) | | | be used for Python2/3 code bases. | | From webhook-mailer at python.org Wed Apr 6 12:54:28 2022 From: webhook-mailer at python.org (serhiy-storchaka) Date: Wed, 06 Apr 2022 16:54:28 -0000 Subject: [Python-checkins] bpo-47211: Remove function re.template() and flag re.TEMPLATE (GH-32300) Message-ID: https://github.com/python/cpython/commit/b09184bf05b07b77c5ecfedd4daa846be3cbf0a9 commit: b09184bf05b07b77c5ecfedd4daa846be3cbf0a9 branch: main author: Serhiy Storchaka committer: serhiy-storchaka date: 2022-04-06T19:53:50+03:00 summary: bpo-47211: Remove function re.template() and flag re.TEMPLATE (GH-32300) They were undocumented and never working. files: A Misc/NEWS.d/next/Library/2022-04-04-11-58-07.bpo-47211.W4GFkB.rst M Lib/re/__init__.py M Lib/re/_compiler.py M Lib/re/_constants.py M Lib/re/_parser.py M Lib/test/test_re.py M Modules/_sre/sre.c M Modules/_sre/sre_constants.h diff --git a/Lib/re/__init__.py b/Lib/re/__init__.py index b887722bbb2cd..c9b511422f1e5 100644 --- a/Lib/re/__init__.py +++ b/Lib/re/__init__.py @@ -129,7 +129,7 @@ # public symbols __all__ = [ "match", "fullmatch", "search", "sub", "subn", "split", - "findall", "finditer", "compile", "purge", "template", "escape", + "findall", "finditer", "compile", "purge", "escape", "error", "Pattern", "Match", "A", "I", "L", "M", "S", "X", "U", "ASCII", "IGNORECASE", "LOCALE", "MULTILINE", "DOTALL", "VERBOSE", "UNICODE", "NOFLAG", "RegexFlag", @@ -148,8 +148,6 @@ class RegexFlag: MULTILINE = M = _compiler.SRE_FLAG_MULTILINE # make anchors look for newline DOTALL = S = _compiler.SRE_FLAG_DOTALL # make dot match newline VERBOSE = X = _compiler.SRE_FLAG_VERBOSE # ignore whitespace and comments - # sre extensions (experimental, don't rely on these) - TEMPLATE = T = _compiler.SRE_FLAG_TEMPLATE # disable backtracking DEBUG = _compiler.SRE_FLAG_DEBUG # dump pattern after compilation __str__ = object.__str__ _numeric_repr_ = hex @@ -231,10 +229,6 @@ def purge(): _cache.clear() _compile_repl.cache_clear() -def template(pattern, flags=0): - "Compile a template pattern, returning a Pattern object" - return _compile(pattern, flags|T) - # SPECIAL_CHARS # closing ')', '}' and ']' # '-' (a range in character set) diff --git a/Lib/re/_compiler.py b/Lib/re/_compiler.py index bedd4b8f40016..5b4c4a3f30a40 100644 --- a/Lib/re/_compiler.py +++ b/Lib/re/_compiler.py @@ -147,8 +147,6 @@ def _compile(data, pattern, flags): else: emit(ANY) elif op in REPEATING_CODES: - if flags & SRE_FLAG_TEMPLATE: - raise error("internal: unsupported template operator %r" % (op,)) if _simple(av[2]): emit(REPEATING_CODES[op][2]) skip = _len(code); emit(0) diff --git a/Lib/re/_constants.py b/Lib/re/_constants.py index 327ba548118b3..4c7e93e67ec60 100644 --- a/Lib/re/_constants.py +++ b/Lib/re/_constants.py @@ -202,7 +202,6 @@ def _makecodes(names): } # flags -SRE_FLAG_TEMPLATE = 1 # template mode (disable backtracking) SRE_FLAG_IGNORECASE = 2 # case insensitive SRE_FLAG_LOCALE = 4 # honour system locale SRE_FLAG_MULTILINE = 8 # treat target as multiline string @@ -245,7 +244,6 @@ def dump(f, d, prefix): dump(f, ATCODES, "SRE") dump(f, CHCODES, "SRE") - f.write("#define SRE_FLAG_TEMPLATE %d\n" % SRE_FLAG_TEMPLATE) f.write("#define SRE_FLAG_IGNORECASE %d\n" % SRE_FLAG_IGNORECASE) f.write("#define SRE_FLAG_LOCALE %d\n" % SRE_FLAG_LOCALE) f.write("#define SRE_FLAG_MULTILINE %d\n" % SRE_FLAG_MULTILINE) diff --git a/Lib/re/_parser.py b/Lib/re/_parser.py index ae44118564eb7..eca2ffc28b45b 100644 --- a/Lib/re/_parser.py +++ b/Lib/re/_parser.py @@ -61,12 +61,11 @@ "x": SRE_FLAG_VERBOSE, # extensions "a": SRE_FLAG_ASCII, - "t": SRE_FLAG_TEMPLATE, "u": SRE_FLAG_UNICODE, } TYPE_FLAGS = SRE_FLAG_ASCII | SRE_FLAG_LOCALE | SRE_FLAG_UNICODE -GLOBAL_FLAGS = SRE_FLAG_DEBUG | SRE_FLAG_TEMPLATE +GLOBAL_FLAGS = SRE_FLAG_DEBUG class Verbose(Exception): pass diff --git a/Lib/test/test_re.py b/Lib/test/test_re.py index 553eb4cfe85b6..959582e2f1257 100644 --- a/Lib/test/test_re.py +++ b/Lib/test/test_re.py @@ -2432,11 +2432,11 @@ def test_flags_repr(self): "re.IGNORECASE|re.DOTALL|re.VERBOSE|0x100000") self.assertEqual( repr(~re.I), - "re.ASCII|re.LOCALE|re.UNICODE|re.MULTILINE|re.DOTALL|re.VERBOSE|re.TEMPLATE|re.DEBUG") + "re.ASCII|re.LOCALE|re.UNICODE|re.MULTILINE|re.DOTALL|re.VERBOSE|re.DEBUG|0x1") self.assertEqual(repr(~(re.I|re.S|re.X)), - "re.ASCII|re.LOCALE|re.UNICODE|re.MULTILINE|re.TEMPLATE|re.DEBUG") + "re.ASCII|re.LOCALE|re.UNICODE|re.MULTILINE|re.DEBUG|0x1") self.assertEqual(repr(~(re.I|re.S|re.X|(1<<20))), - "re.ASCII|re.LOCALE|re.UNICODE|re.MULTILINE|re.TEMPLATE|re.DEBUG|0xffe00") + "re.ASCII|re.LOCALE|re.UNICODE|re.MULTILINE|re.DEBUG|0xffe01") class ImplementationTest(unittest.TestCase): diff --git a/Misc/NEWS.d/next/Library/2022-04-04-11-58-07.bpo-47211.W4GFkB.rst b/Misc/NEWS.d/next/Library/2022-04-04-11-58-07.bpo-47211.W4GFkB.rst new file mode 100644 index 0000000000000..0bd5d1619ff78 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-04-11-58-07.bpo-47211.W4GFkB.rst @@ -0,0 +1,2 @@ +Remove undocumented and never working function ``re.template()`` and flag +``re.TEMPLATE``. diff --git a/Modules/_sre/sre.c b/Modules/_sre/sre.c index 491734f243849..bd9204da428af 100644 --- a/Modules/_sre/sre.c +++ b/Modules/_sre/sre.c @@ -1323,7 +1323,6 @@ pattern_repr(PatternObject *obj) const char *name; int value; } flag_names[] = { - {"re.TEMPLATE", SRE_FLAG_TEMPLATE}, {"re.IGNORECASE", SRE_FLAG_IGNORECASE}, {"re.LOCALE", SRE_FLAG_LOCALE}, {"re.MULTILINE", SRE_FLAG_MULTILINE}, diff --git a/Modules/_sre/sre_constants.h b/Modules/_sre/sre_constants.h index 3e3643144a92c..e53fa392ec585 100644 --- a/Modules/_sre/sre_constants.h +++ b/Modules/_sre/sre_constants.h @@ -86,7 +86,6 @@ #define SRE_CATEGORY_UNI_NOT_WORD 15 #define SRE_CATEGORY_UNI_LINEBREAK 16 #define SRE_CATEGORY_UNI_NOT_LINEBREAK 17 -#define SRE_FLAG_TEMPLATE 1 #define SRE_FLAG_IGNORECASE 2 #define SRE_FLAG_LOCALE 4 #define SRE_FLAG_MULTILINE 8 From webhook-mailer at python.org Wed Apr 6 12:54:54 2022 From: webhook-mailer at python.org (serhiy-storchaka) Date: Wed, 06 Apr 2022 16:54:54 -0000 Subject: [Python-checkins] bpo-47227: Suppress expression chaining for more RE parsing errors (GH-32333) Message-ID: https://github.com/python/cpython/commit/50872dbadcba1f52867b6f76050cd7b5d0aa1e18 commit: 50872dbadcba1f52867b6f76050cd7b5d0aa1e18 branch: main author: Serhiy Storchaka committer: serhiy-storchaka date: 2022-04-06T19:54:44+03:00 summary: bpo-47227: Suppress expression chaining for more RE parsing errors (GH-32333) files: A Misc/NEWS.d/next/Library/2022-04-05-15-53-58.bpo-47227.1HWdp9.rst M Lib/re/_parser.py diff --git a/Lib/re/_parser.py b/Lib/re/_parser.py index eca2ffc28b45b..f191f809a1491 100644 --- a/Lib/re/_parser.py +++ b/Lib/re/_parser.py @@ -335,7 +335,7 @@ def _class_escape(source, escape): c = ord(unicodedata.lookup(charname)) except KeyError: raise source.error("undefined character name %r" % charname, - len(charname) + len(r'\N{}')) + len(charname) + len(r'\N{}')) from None return LITERAL, c elif c in OCTDIGITS: # octal escape (up to three digits) @@ -395,7 +395,7 @@ def _escape(source, escape, state): c = ord(unicodedata.lookup(charname)) except KeyError: raise source.error("undefined character name %r" % charname, - len(charname) + len(r'\N{}')) + len(charname) + len(r'\N{}')) from None return LITERAL, c elif c == "0": # octal escape @@ -1014,7 +1014,7 @@ def addgroup(index, pos): try: index = groupindex[name] except KeyError: - raise IndexError("unknown group name %r" % name) + raise IndexError("unknown group name %r" % name) from None else: try: index = int(name) @@ -1053,7 +1053,7 @@ def addgroup(index, pos): this = chr(ESCAPES[this][1]) except KeyError: if c in ASCIILETTERS: - raise s.error('bad escape %s' % this, len(this)) + raise s.error('bad escape %s' % this, len(this)) from None lappend(this) else: lappend(this) @@ -1074,5 +1074,5 @@ def expand_template(template, match): for index, group in groups: literals[index] = g(group) or empty except IndexError: - raise error("invalid group reference %d" % index) + raise error("invalid group reference %d" % index) from None return empty.join(literals) diff --git a/Misc/NEWS.d/next/Library/2022-04-05-15-53-58.bpo-47227.1HWdp9.rst b/Misc/NEWS.d/next/Library/2022-04-05-15-53-58.bpo-47227.1HWdp9.rst new file mode 100644 index 0000000000000..254d95150ef6c --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-05-15-53-58.bpo-47227.1HWdp9.rst @@ -0,0 +1 @@ +Suppress expression chaining for more :mod:`re` parsing errors. From webhook-mailer at python.org Wed Apr 6 12:56:42 2022 From: webhook-mailer at python.org (serhiy-storchaka) Date: Wed, 06 Apr 2022 16:56:42 -0000 Subject: [Python-checkins] bpo-43464: Optimize set.intersection() for non-set arguments (GH-31316) Message-ID: https://github.com/python/cpython/commit/31cd25f4e17cd68487dc76c1b2ec162a646818c2 commit: 31cd25f4e17cd68487dc76c1b2ec162a646818c2 branch: main author: Serhiy Storchaka committer: serhiy-storchaka date: 2022-04-06T19:56:28+03:00 summary: bpo-43464: Optimize set.intersection() for non-set arguments (GH-31316) files: A Misc/NEWS.d/next/Core and Builtins/2022-02-13-21-53-29.bpo-43464.yupHjd.rst M Objects/setobject.c diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-02-13-21-53-29.bpo-43464.yupHjd.rst b/Misc/NEWS.d/next/Core and Builtins/2022-02-13-21-53-29.bpo-43464.yupHjd.rst new file mode 100644 index 0000000000000..a67ce7c9688e2 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2022-02-13-21-53-29.bpo-43464.yupHjd.rst @@ -0,0 +1 @@ +Optimize :meth:`set.intersection` for non-set arguments. diff --git a/Objects/setobject.c b/Objects/setobject.c index 022ae8e7f9392..18dc49be93d6d 100644 --- a/Objects/setobject.c +++ b/Objects/setobject.c @@ -1240,6 +1240,10 @@ set_intersection(PySetObject *so, PyObject *other) if (rv) { if (set_add_entry(result, key, hash)) goto error; + if (PySet_GET_SIZE(result) >= PySet_GET_SIZE(so)) { + Py_DECREF(key); + break; + } } Py_DECREF(key); } From webhook-mailer at python.org Wed Apr 6 12:57:18 2022 From: webhook-mailer at python.org (serhiy-storchaka) Date: Wed, 06 Apr 2022 16:57:18 -0000 Subject: [Python-checkins] bpo-46721: Optimize set.issuperset() for non-set arguments (GH-31280) Message-ID: https://github.com/python/cpython/commit/a69a4a917c436579c2c4112081ea86a70f1f05d3 commit: a69a4a917c436579c2c4112081ea86a70f1f05d3 branch: main author: Serhiy Storchaka committer: serhiy-storchaka date: 2022-04-06T19:57:13+03:00 summary: bpo-46721: Optimize set.issuperset() for non-set arguments (GH-31280) files: A Misc/NEWS.d/next/Core and Builtins/2022-02-11-17-16-30.bpo-46721.JkHaLF.rst M Objects/setobject.c diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-02-11-17-16-30.bpo-46721.JkHaLF.rst b/Misc/NEWS.d/next/Core and Builtins/2022-02-11-17-16-30.bpo-46721.JkHaLF.rst new file mode 100644 index 0000000000000..5644f10fe5e3e --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2022-02-11-17-16-30.bpo-46721.JkHaLF.rst @@ -0,0 +1 @@ +Optimize :meth:`set.issuperset` for non-set argument. diff --git a/Objects/setobject.c b/Objects/setobject.c index 18dc49be93d6d..ef2190de914b3 100644 --- a/Objects/setobject.c +++ b/Objects/setobject.c @@ -1382,14 +1382,7 @@ set_isdisjoint(PySetObject *so, PyObject *other) return NULL; while ((key = PyIter_Next(it)) != NULL) { - Py_hash_t hash = PyObject_Hash(key); - - if (hash == -1) { - Py_DECREF(key); - Py_DECREF(it); - return NULL; - } - rv = set_contains_entry(so, key, hash); + rv = set_contains_key(so, key); Py_DECREF(key); if (rv < 0) { Py_DECREF(it); @@ -1773,17 +1766,31 @@ PyDoc_STRVAR(issubset_doc, "Report whether another set contains this set."); static PyObject * set_issuperset(PySetObject *so, PyObject *other) { - PyObject *tmp, *result; + if (PyAnySet_Check(other)) { + return set_issubset((PySetObject *)other, (PyObject *)so); + } - if (!PyAnySet_Check(other)) { - tmp = make_new_set(&PySet_Type, other); - if (tmp == NULL) + PyObject *key, *it = PyObject_GetIter(other); + if (it == NULL) { + return NULL; + } + while ((key = PyIter_Next(it)) != NULL) { + int rv = set_contains_key(so, key); + Py_DECREF(key); + if (rv < 0) { + Py_DECREF(it); return NULL; - result = set_issuperset(so, tmp); - Py_DECREF(tmp); - return result; + } + if (!rv) { + Py_DECREF(it); + Py_RETURN_FALSE; + } } - return set_issubset((PySetObject *)other, (PyObject *)so); + Py_DECREF(it); + if (PyErr_Occurred()) { + return NULL; + } + Py_RETURN_TRUE; } PyDoc_STRVAR(issuperset_doc, "Report whether this set contains another set."); From webhook-mailer at python.org Wed Apr 6 12:57:58 2022 From: webhook-mailer at python.org (vsajip) Date: Wed, 06 Apr 2022 16:57:58 -0000 Subject: [Python-checkins] Remove python2 support in logging cookbook example. (GH-32362) Message-ID: https://github.com/python/cpython/commit/f82f9ce3239b9a7e6ffa278658dd9858f64a3c14 commit: f82f9ce3239b9a7e6ffa278658dd9858f64a3c14 branch: main author: Mathieu Dupuy committer: vsajip date: 2022-04-06T17:57:54+01:00 summary: Remove python2 support in logging cookbook example. (GH-32362) files: M Doc/howto/logging-cookbook.rst diff --git a/Doc/howto/logging-cookbook.rst b/Doc/howto/logging-cookbook.rst index f0d944940fc28..704279240b357 100644 --- a/Doc/howto/logging-cookbook.rst +++ b/Doc/howto/logging-cookbook.rst @@ -1788,22 +1788,15 @@ Python used. If you need more specialised processing, you can use a custom JSON encoder, as in the following complete example:: - from __future__ import unicode_literals - import json import logging - # This next bit is to ensure the script runs unchanged on 2.x and 3.x - try: - unicode - except NameError: - unicode = str class Encoder(json.JSONEncoder): def default(self, o): if isinstance(o, set): return tuple(o) - elif isinstance(o, unicode): + elif isinstance(o, str): return o.encode('unicode_escape').decode('ascii') return super().default(o) From webhook-mailer at python.org Wed Apr 6 13:00:20 2022 From: webhook-mailer at python.org (serhiy-storchaka) Date: Wed, 06 Apr 2022 17:00:20 -0000 Subject: [Python-checkins] bpo-26579: Add object.__getstate__(). (GH-2821) Message-ID: https://github.com/python/cpython/commit/884eba3c76916889fd6bff3b37b8552bfb4f9566 commit: 884eba3c76916889fd6bff3b37b8552bfb4f9566 branch: main author: Serhiy Storchaka committer: serhiy-storchaka date: 2022-04-06T20:00:14+03:00 summary: bpo-26579: Add object.__getstate__(). (GH-2821) Copying and pickling instances of subclasses of builtin types bytearray, set, frozenset, collections.OrderedDict, collections.deque, weakref.WeakSet, and datetime.tzinfo now copies and pickles instance attributes implemented as slots. files: A Misc/NEWS.d/next/Core and Builtins/2017-07-23-11-28-45.bpo-26579.lpCY8R.rst M Doc/library/pickle.rst M Doc/whatsnew/3.11.rst M Include/object.h M Lib/_weakrefset.py M Lib/collections/__init__.py M Lib/copyreg.py M Lib/datetime.py M Lib/email/headerregistry.py M Lib/test/datetimetester.py M Lib/test/pickletester.py M Lib/test/test_bytes.py M Lib/test/test_deque.py M Lib/test/test_descrtut.py M Lib/test/test_ordered_dict.py M Lib/test/test_set.py M Lib/test/test_weakset.py M Lib/test/test_xml_etree.py M Modules/_collectionsmodule.c M Modules/_datetimemodule.c M Objects/bytearrayobject.c M Objects/clinic/typeobject.c.h M Objects/odictobject.c M Objects/setobject.c M Objects/typeobject.c diff --git a/Doc/library/pickle.rst b/Doc/library/pickle.rst index be48561ed10ac..a8ad5d437aaab 100644 --- a/Doc/library/pickle.rst +++ b/Doc/library/pickle.rst @@ -509,9 +509,8 @@ The following types can be pickled: * classes that are defined at the top level of a module -* instances of such classes whose :attr:`~object.__dict__` or the result of - calling :meth:`__getstate__` is picklable (see section :ref:`pickle-inst` for - details). +* instances of such classes whose the result of calling :meth:`__getstate__` + is picklable (see section :ref:`pickle-inst` for details). Attempts to pickle unpicklable objects will raise the :exc:`PicklingError` exception; when this happens, an unspecified number of bytes may have already @@ -611,11 +610,31 @@ methods: .. method:: object.__getstate__() - Classes can further influence how their instances are pickled; if the class - defines the method :meth:`__getstate__`, it is called and the returned object - is pickled as the contents for the instance, instead of the contents of the - instance's dictionary. If the :meth:`__getstate__` method is absent, the - instance's :attr:`~object.__dict__` is pickled as usual. + Classes can further influence how their instances are pickled by overriding + the method :meth:`__getstate__`. It is called and the returned object + is pickled as the contents for the instance, instead of a default state. + There are several cases: + + * For a class that has no instance :attr:`~object.__dict__` and no + :attr:`~object.__slots__`, the default state is ``None``. + + * For a class that has an instance :attr:`~object.__dict__` and no + :attr:`~object.__slots__`, the default state is ``self.__dict__``. + + * For a class that has an instance :attr:`~object.__dict__` and + :attr:`~object.__slots__`, the default state is a tuple consisting of two + dictionaries: ``self.__dict__``, and a dictionary mapping slot + names to slot values. Only slots that have a value are + included in the latter. + + * For a class that has :attr:`~object.__slots__` and no instance + :attr:`~object.__dict__`, the default state is a tuple whose first item + is ``None`` and whose second item is a dictionary mapping slot names + to slot values described in the previous bullet. + + .. versionchanged:: 3.11 + Added the default implementation of the ``__getstate__()`` method in the + :class:`object` class. .. method:: object.__setstate__(state) diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index 96902cf61b46d..4c9b32d9a9452 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -187,6 +187,15 @@ Other Language Changes protocols correspondingly. (Contributed by Serhiy Storchaka in :issue:`12022`.) +* Added :meth:`object.__getstate__` which provides the default + implementation of the ``__getstate__()`` method. :mod:`Copying ` + and :mod:`pickling ` instances of subclasses of builtin types + :class:`bytearray`, :class:`set`, :class:`frozenset`, + :class:`collections.OrderedDict`, :class:`collections.deque`, + :class:`weakref.WeakSet`, and :class:`datetime.tzinfo` now copies and + pickles instance attributes implemented as :term:`slots <__slots__>`. + (Contributed by Serhiy Storchaka in :issue:`26579`.) + Other CPython Implementation Changes ==================================== diff --git a/Include/object.h b/Include/object.h index 317515d2c8151..0b4b55ea1ded9 100644 --- a/Include/object.h +++ b/Include/object.h @@ -299,6 +299,11 @@ PyAPI_FUNC(void) PyObject_ClearWeakRefs(PyObject *); */ PyAPI_FUNC(PyObject *) PyObject_Dir(PyObject *); +/* Pickle support. */ +#ifndef Py_LIMITED_API +PyAPI_FUNC(PyObject *) _PyObject_GetState(PyObject *); +#endif + /* Helpers for printing recursive container types */ PyAPI_FUNC(int) Py_ReprEnter(PyObject *); diff --git a/Lib/_weakrefset.py b/Lib/_weakrefset.py index 2a27684324d80..489eec714e0d4 100644 --- a/Lib/_weakrefset.py +++ b/Lib/_weakrefset.py @@ -80,8 +80,7 @@ def __contains__(self, item): return wr in self.data def __reduce__(self): - return (self.__class__, (list(self),), - getattr(self, '__dict__', None)) + return self.__class__, (list(self),), self.__getstate__() def add(self, item): if self._pending_removals: diff --git a/Lib/collections/__init__.py b/Lib/collections/__init__.py index 264435cb41b2f..7af8dcd526df8 100644 --- a/Lib/collections/__init__.py +++ b/Lib/collections/__init__.py @@ -271,10 +271,22 @@ def __repr__(self): def __reduce__(self): 'Return state information for pickling' - inst_dict = vars(self).copy() - for k in vars(OrderedDict()): - inst_dict.pop(k, None) - return self.__class__, (), inst_dict or None, None, iter(self.items()) + state = self.__getstate__() + if state: + if isinstance(state, tuple): + state, slots = state + else: + slots = {} + state = state.copy() + slots = slots.copy() + for k in vars(OrderedDict()): + state.pop(k, None) + slots.pop(k, None) + if slots: + state = state, slots + else: + state = state or None + return self.__class__, (), state, None, iter(self.items()) def copy(self): 'od.copy() -> a shallow copy of od' diff --git a/Lib/copyreg.py b/Lib/copyreg.py index 356db6f083e39..c8a52a2dc63a2 100644 --- a/Lib/copyreg.py +++ b/Lib/copyreg.py @@ -89,6 +89,10 @@ def _reduce_ex(self, proto): except AttributeError: dict = None else: + if (type(self).__getstate__ is object.__getstate__ and + getattr(self, "__slots__", None)): + raise TypeError("a class that defines __slots__ without " + "defining __getstate__ cannot be pickled") dict = getstate() if dict: return _reconstructor, args, dict diff --git a/Lib/datetime.py b/Lib/datetime.py index 6bf37ccfab7ac..260b1de38877a 100644 --- a/Lib/datetime.py +++ b/Lib/datetime.py @@ -1169,15 +1169,7 @@ def __reduce__(self): args = getinitargs() else: args = () - getstate = getattr(self, "__getstate__", None) - if getstate: - state = getstate() - else: - state = getattr(self, "__dict__", None) or None - if state is None: - return (self.__class__, args) - else: - return (self.__class__, args, state) + return (self.__class__, args, self.__getstate__()) class IsoCalendarDate(tuple): diff --git a/Lib/email/headerregistry.py b/Lib/email/headerregistry.py index b590d69e8b744..543141dc427eb 100644 --- a/Lib/email/headerregistry.py +++ b/Lib/email/headerregistry.py @@ -218,7 +218,7 @@ def __reduce__(self): self.__class__.__bases__, str(self), ), - self.__dict__) + self.__getstate__()) @classmethod def _reconstruct(cls, value): diff --git a/Lib/test/datetimetester.py b/Lib/test/datetimetester.py index efea40d3f4ff1..335cded3b5fac 100644 --- a/Lib/test/datetimetester.py +++ b/Lib/test/datetimetester.py @@ -139,8 +139,8 @@ class PicklableFixedOffset(FixedOffset): def __init__(self, offset=None, name=None, dstoffset=None): FixedOffset.__init__(self, offset, name, dstoffset) - def __getstate__(self): - return self.__dict__ +class PicklableFixedOffsetWithSlots(PicklableFixedOffset): + __slots__ = '_FixedOffset__offset', '_FixedOffset__name', 'spam' class _TZInfo(tzinfo): def utcoffset(self, datetime_module): @@ -202,6 +202,7 @@ def test_pickling_subclass(self): offset = timedelta(minutes=-300) for otype, args in [ (PicklableFixedOffset, (offset, 'cookie')), + (PicklableFixedOffsetWithSlots, (offset, 'cookie')), (timezone, (offset,)), (timezone, (offset, "EST"))]: orig = otype(*args) @@ -217,6 +218,7 @@ def test_pickling_subclass(self): self.assertIs(type(derived), otype) self.assertEqual(derived.utcoffset(None), offset) self.assertEqual(derived.tzname(None), oname) + self.assertFalse(hasattr(derived, 'spam')) def test_issue23600(self): DSTDIFF = DSTOFFSET = timedelta(hours=1) diff --git a/Lib/test/pickletester.py b/Lib/test/pickletester.py index f13d42f664880..63fa7604fbfc9 100644 --- a/Lib/test/pickletester.py +++ b/Lib/test/pickletester.py @@ -2382,9 +2382,11 @@ def test_reduce_calls_base(self): def test_bad_getattr(self): # Issue #3514: crash when there is an infinite loop in __getattr__ x = BadGetattr() - for proto in protocols: + for proto in range(2): with support.infinite_recursion(): self.assertRaises(RuntimeError, self.dumps, x, proto) + for proto in range(2, pickle.HIGHEST_PROTOCOL + 1): + s = self.dumps(x, proto) def test_reduce_bad_iterator(self): # Issue4176: crash when 4th and 5th items of __reduce__() diff --git a/Lib/test/test_bytes.py b/Lib/test/test_bytes.py index fd8e1d4bdc44c..b457ff6ca849c 100644 --- a/Lib/test/test_bytes.py +++ b/Lib/test/test_bytes.py @@ -1940,28 +1940,30 @@ def test_join(self): def test_pickle(self): a = self.type2test(b"abcd") a.x = 10 - a.y = self.type2test(b"efgh") + a.z = self.type2test(b"efgh") for proto in range(pickle.HIGHEST_PROTOCOL + 1): b = pickle.loads(pickle.dumps(a, proto)) self.assertNotEqual(id(a), id(b)) self.assertEqual(a, b) self.assertEqual(a.x, b.x) - self.assertEqual(a.y, b.y) + self.assertEqual(a.z, b.z) self.assertEqual(type(a), type(b)) - self.assertEqual(type(a.y), type(b.y)) + self.assertEqual(type(a.z), type(b.z)) + self.assertFalse(hasattr(b, 'y')) def test_copy(self): a = self.type2test(b"abcd") a.x = 10 - a.y = self.type2test(b"efgh") + a.z = self.type2test(b"efgh") for copy_method in (copy.copy, copy.deepcopy): b = copy_method(a) self.assertNotEqual(id(a), id(b)) self.assertEqual(a, b) self.assertEqual(a.x, b.x) - self.assertEqual(a.y, b.y) + self.assertEqual(a.z, b.z) self.assertEqual(type(a), type(b)) - self.assertEqual(type(a.y), type(b.y)) + self.assertEqual(type(a.z), type(b.z)) + self.assertFalse(hasattr(b, 'y')) def test_fromhex(self): b = self.type2test.fromhex('1a2B30') @@ -1994,6 +1996,9 @@ def __init__(me, *args, **kwargs): class ByteArraySubclass(bytearray): pass +class ByteArraySubclassWithSlots(bytearray): + __slots__ = ('x', 'y', '__dict__') + class BytesSubclass(bytes): pass @@ -2014,6 +2019,9 @@ def __init__(me, newarg=1, *args, **kwargs): x = subclass(newarg=4, source=b"abcd") self.assertEqual(x, b"abcd") +class ByteArraySubclassWithSlotsTest(SubclassTest, unittest.TestCase): + basetype = bytearray + type2test = ByteArraySubclassWithSlots class BytesSubclassTest(SubclassTest, unittest.TestCase): basetype = bytes diff --git a/Lib/test/test_deque.py b/Lib/test/test_deque.py index 0be3feca01049..ae1dfacd7262e 100644 --- a/Lib/test/test_deque.py +++ b/Lib/test/test_deque.py @@ -781,6 +781,9 @@ def test_runtime_error_on_empty_deque(self): class Deque(deque): pass +class DequeWithSlots(deque): + __slots__ = ('x', 'y', '__dict__') + class DequeWithBadIter(deque): def __iter__(self): raise TypeError @@ -810,40 +813,28 @@ def test_basics(self): self.assertEqual(len(d), 0) def test_copy_pickle(self): - - d = Deque('abc') - - e = d.__copy__() - self.assertEqual(type(d), type(e)) - self.assertEqual(list(d), list(e)) - - e = Deque(d) - self.assertEqual(type(d), type(e)) - self.assertEqual(list(d), list(e)) - - for proto in range(pickle.HIGHEST_PROTOCOL + 1): - s = pickle.dumps(d, proto) - e = pickle.loads(s) - self.assertNotEqual(id(d), id(e)) - self.assertEqual(type(d), type(e)) - self.assertEqual(list(d), list(e)) - - d = Deque('abcde', maxlen=4) - - e = d.__copy__() - self.assertEqual(type(d), type(e)) - self.assertEqual(list(d), list(e)) - - e = Deque(d) - self.assertEqual(type(d), type(e)) - self.assertEqual(list(d), list(e)) - - for proto in range(pickle.HIGHEST_PROTOCOL + 1): - s = pickle.dumps(d, proto) - e = pickle.loads(s) - self.assertNotEqual(id(d), id(e)) - self.assertEqual(type(d), type(e)) - self.assertEqual(list(d), list(e)) + for cls in Deque, DequeWithSlots: + for d in cls('abc'), cls('abcde', maxlen=4): + d.x = ['x'] + d.z = ['z'] + + e = d.__copy__() + self.assertEqual(type(d), type(e)) + self.assertEqual(list(d), list(e)) + + e = cls(d) + self.assertEqual(type(d), type(e)) + self.assertEqual(list(d), list(e)) + + for proto in range(pickle.HIGHEST_PROTOCOL + 1): + s = pickle.dumps(d, proto) + e = pickle.loads(s) + self.assertNotEqual(id(d), id(e)) + self.assertEqual(type(d), type(e)) + self.assertEqual(list(d), list(e)) + self.assertEqual(e.x, d.x) + self.assertEqual(e.z, d.z) + self.assertFalse(hasattr(e, 'y')) def test_pickle_recursive(self): for proto in range(pickle.HIGHEST_PROTOCOL + 1): diff --git a/Lib/test/test_descrtut.py b/Lib/test/test_descrtut.py index 2af683e707ca8..e01a31a74695d 100644 --- a/Lib/test/test_descrtut.py +++ b/Lib/test/test_descrtut.py @@ -181,6 +181,7 @@ def merge(self, other): '__ge__', '__getattribute__', '__getitem__', + '__getstate__', '__gt__', '__hash__', '__iadd__', diff --git a/Lib/test/test_ordered_dict.py b/Lib/test/test_ordered_dict.py index d51296a2d1244..37447fd249b8c 100644 --- a/Lib/test/test_ordered_dict.py +++ b/Lib/test/test_ordered_dict.py @@ -287,6 +287,8 @@ def test_copying(self): # and have a repr/eval round-trip pairs = [('c', 1), ('b', 2), ('a', 3), ('d', 4), ('e', 5), ('f', 6)] od = OrderedDict(pairs) + od.x = ['x'] + od.z = ['z'] def check(dup): msg = "\ncopy: %s\nod: %s" % (dup, od) self.assertIsNot(dup, od, msg) @@ -295,13 +297,27 @@ def check(dup): self.assertEqual(len(dup), len(od)) self.assertEqual(type(dup), type(od)) check(od.copy()) - check(copy.copy(od)) - check(copy.deepcopy(od)) + dup = copy.copy(od) + check(dup) + self.assertIs(dup.x, od.x) + self.assertIs(dup.z, od.z) + self.assertFalse(hasattr(dup, 'y')) + dup = copy.deepcopy(od) + check(dup) + self.assertEqual(dup.x, od.x) + self.assertIsNot(dup.x, od.x) + self.assertEqual(dup.z, od.z) + self.assertIsNot(dup.z, od.z) + self.assertFalse(hasattr(dup, 'y')) # pickle directly pulls the module, so we have to fake it with replaced_module('collections', self.module): for proto in range(pickle.HIGHEST_PROTOCOL + 1): with self.subTest(proto=proto): - check(pickle.loads(pickle.dumps(od, proto))) + dup = pickle.loads(pickle.dumps(od, proto)) + check(dup) + self.assertEqual(dup.x, od.x) + self.assertEqual(dup.z, od.z) + self.assertFalse(hasattr(dup, 'y')) check(eval(repr(od))) update_test = OrderedDict() update_test.update(od) @@ -846,6 +862,23 @@ class OrderedDict(c_coll.OrderedDict): pass +class PurePythonOrderedDictWithSlotsCopyingTests(unittest.TestCase): + + module = py_coll + class OrderedDict(py_coll.OrderedDict): + __slots__ = ('x', 'y') + test_copying = OrderedDictTests.test_copying + + + at unittest.skipUnless(c_coll, 'requires the C version of the collections module') +class CPythonOrderedDictWithSlotsCopyingTests(unittest.TestCase): + + module = c_coll + class OrderedDict(c_coll.OrderedDict): + __slots__ = ('x', 'y') + test_copying = OrderedDictTests.test_copying + + class PurePythonGeneralMappingTests(mapping_tests.BasicTestMappingProtocol): @classmethod diff --git a/Lib/test/test_set.py b/Lib/test/test_set.py index 03b911920e179..3b57517a86101 100644 --- a/Lib/test/test_set.py +++ b/Lib/test/test_set.py @@ -227,14 +227,17 @@ def test_sub_and_super(self): def test_pickling(self): for i in range(pickle.HIGHEST_PROTOCOL + 1): + if type(self.s) not in (set, frozenset): + self.s.x = ['x'] + self.s.z = ['z'] p = pickle.dumps(self.s, i) dup = pickle.loads(p) self.assertEqual(self.s, dup, "%s != %s" % (self.s, dup)) if type(self.s) not in (set, frozenset): - self.s.x = 10 - p = pickle.dumps(self.s, i) - dup = pickle.loads(p) self.assertEqual(self.s.x, dup.x) + self.assertEqual(self.s.z, dup.z) + self.assertFalse(hasattr(self.s, 'y')) + del self.s.x, self.s.z def test_iterator_pickling(self): for proto in range(pickle.HIGHEST_PROTOCOL + 1): @@ -808,6 +811,21 @@ def test_singleton_empty_frozenset(self): # All empty frozenset subclass instances should have different ids self.assertEqual(len(set(map(id, efs))), len(efs)) + +class SetSubclassWithSlots(set): + __slots__ = ('x', 'y', '__dict__') + +class TestSetSubclassWithSlots(unittest.TestCase): + thetype = SetSubclassWithSlots + setUp = TestJointOps.setUp + test_pickling = TestJointOps.test_pickling + +class FrozenSetSubclassWithSlots(frozenset): + __slots__ = ('x', 'y', '__dict__') + +class TestFrozenSetSubclassWithSlots(TestSetSubclassWithSlots): + thetype = FrozenSetSubclassWithSlots + # Tests taken from test_sets.py ============================================= empty_set = set() diff --git a/Lib/test/test_weakset.py b/Lib/test/test_weakset.py index 9b31d5fce3472..d6c88596cff71 100644 --- a/Lib/test/test_weakset.py +++ b/Lib/test/test_weakset.py @@ -1,5 +1,6 @@ import unittest from weakref import WeakSet +import copy import string from collections import UserString as ustr from collections.abc import Set, MutableSet @@ -15,6 +16,12 @@ class RefCycle: def __init__(self): self.cycle = self +class WeakSetSubclass(WeakSet): + pass + +class WeakSetWithSlots(WeakSet): + __slots__ = ('x', 'y') + class TestWeakSet(unittest.TestCase): @@ -447,6 +454,30 @@ def test_abc(self): self.assertIsInstance(self.s, Set) self.assertIsInstance(self.s, MutableSet) + def test_copying(self): + for cls in WeakSet, WeakSetWithSlots: + s = cls(self.items) + s.x = ['x'] + s.z = ['z'] + + dup = copy.copy(s) + self.assertIsInstance(dup, cls) + self.assertEqual(dup, s) + self.assertIsNot(dup, s) + self.assertIs(dup.x, s.x) + self.assertIs(dup.z, s.z) + self.assertFalse(hasattr(dup, 'y')) + + dup = copy.deepcopy(s) + self.assertIsInstance(dup, cls) + self.assertEqual(dup, s) + self.assertIsNot(dup, s) + self.assertEqual(dup.x, s.x) + self.assertIsNot(dup.x, s.x) + self.assertEqual(dup.z, s.z) + self.assertIsNot(dup.z, s.z) + self.assertFalse(hasattr(dup, 'y')) + if __name__ == "__main__": unittest.main() diff --git a/Lib/test/test_xml_etree.py b/Lib/test/test_xml_etree.py index d2bdc4f7f0444..60a41506d8795 100644 --- a/Lib/test/test_xml_etree.py +++ b/Lib/test/test_xml_etree.py @@ -2524,8 +2524,7 @@ def test_pickle_issue18997(self): 4 """ e1 = dumper.fromstring(XMLTEXT) - if hasattr(e1, '__getstate__'): - self.assertEqual(e1.__getstate__()['tag'], 'group') + self.assertEqual(e1.__getstate__()['tag'], 'group') e2 = self.pickleRoundTrip(e1, 'xml.etree.ElementTree', dumper, loader, proto) self.assertEqual(e2.tag, 'group') diff --git a/Misc/NEWS.d/next/Core and Builtins/2017-07-23-11-28-45.bpo-26579.lpCY8R.rst b/Misc/NEWS.d/next/Core and Builtins/2017-07-23-11-28-45.bpo-26579.lpCY8R.rst new file mode 100644 index 0000000000000..9afd1bfa50084 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2017-07-23-11-28-45.bpo-26579.lpCY8R.rst @@ -0,0 +1,7 @@ +Added ``object.__getstate__`` which provides the default implementation of +the ``__getstate__()`` method. + +Copying and pickling instances of subclasses of builtin types bytearray, +set, frozenset, collections.OrderedDict, collections.deque, weakref.WeakSet, +and datetime.tzinfo now copies and pickles instance attributes implemented as +slots. diff --git a/Modules/_collectionsmodule.c b/Modules/_collectionsmodule.c index b47f977731758..f78e2613fc243 100644 --- a/Modules/_collectionsmodule.c +++ b/Modules/_collectionsmodule.c @@ -1347,27 +1347,24 @@ deque_traverse(dequeobject *deque, visitproc visit, void *arg) static PyObject * deque_reduce(dequeobject *deque, PyObject *Py_UNUSED(ignored)) { - PyObject *dict, *it; + PyObject *state, *it; - if (_PyObject_LookupAttr((PyObject *)deque, &_Py_ID(__dict__), &dict) < 0) { + state = _PyObject_GetState((PyObject *)deque); + if (state == NULL) { return NULL; } - if (dict == NULL) { - dict = Py_None; - Py_INCREF(dict); - } it = PyObject_GetIter((PyObject *)deque); if (it == NULL) { - Py_DECREF(dict); + Py_DECREF(state); return NULL; } if (deque->maxlen < 0) { - return Py_BuildValue("O()NN", Py_TYPE(deque), dict, it); + return Py_BuildValue("O()NN", Py_TYPE(deque), state, it); } else { - return Py_BuildValue("O(()n)NN", Py_TYPE(deque), deque->maxlen, dict, it); + return Py_BuildValue("O(()n)NN", Py_TYPE(deque), deque->maxlen, state, it); } } diff --git a/Modules/_datetimemodule.c b/Modules/_datetimemodule.c index ae97190bccbde..fc766c3c94360 100644 --- a/Modules/_datetimemodule.c +++ b/Modules/_datetimemodule.c @@ -3736,9 +3736,8 @@ static PyObject * tzinfo_reduce(PyObject *self, PyObject *Py_UNUSED(ignored)) { PyObject *args, *state; - PyObject *getinitargs, *getstate; + PyObject *getinitargs; _Py_IDENTIFIER(__getinitargs__); - _Py_IDENTIFIER(__getstate__); if (_PyObject_LookupAttrId(self, &PyId___getinitargs__, &getinitargs) < 0) { return NULL; @@ -3754,34 +3753,13 @@ tzinfo_reduce(PyObject *self, PyObject *Py_UNUSED(ignored)) return NULL; } - if (_PyObject_LookupAttrId(self, &PyId___getstate__, &getstate) < 0) { + state = _PyObject_GetState(self); + if (state == NULL) { Py_DECREF(args); return NULL; } - if (getstate != NULL) { - state = PyObject_CallNoArgs(getstate); - Py_DECREF(getstate); - if (state == NULL) { - Py_DECREF(args); - return NULL; - } - } - else { - PyObject **dictptr; - state = Py_None; - dictptr = _PyObject_GetDictPtr(self); - if (dictptr && *dictptr && PyDict_GET_SIZE(*dictptr)) { - state = *dictptr; - } - Py_INCREF(state); - } - if (state == Py_None) { - Py_DECREF(state); - return Py_BuildValue("(ON)", Py_TYPE(self), args); - } - else - return Py_BuildValue("(ONN)", Py_TYPE(self), args, state); + return Py_BuildValue("(ONN)", Py_TYPE(self), args, state); } static PyMethodDef tzinfo_methods[] = { diff --git a/Objects/bytearrayobject.c b/Objects/bytearrayobject.c index 3849337fbf77f..f784e0448191f 100644 --- a/Objects/bytearrayobject.c +++ b/Objects/bytearrayobject.c @@ -2122,35 +2122,26 @@ bytearray_hex_impl(PyByteArrayObject *self, PyObject *sep, int bytes_per_sep) static PyObject * _common_reduce(PyByteArrayObject *self, int proto) { - PyObject *dict; - char *buf; + PyObject *state; + const char *buf; - if (_PyObject_LookupAttr((PyObject *)self, &_Py_ID(__dict__), &dict) < 0) { + state = _PyObject_GetState((PyObject *)self); + if (state == NULL) { return NULL; } - if (dict == NULL) { - dict = Py_None; - Py_INCREF(dict); - } + if (!Py_SIZE(self)) { + return Py_BuildValue("(O()N)", Py_TYPE(self), state); + } buf = PyByteArray_AS_STRING(self); if (proto < 3) { /* use str based reduction for backwards compatibility with Python 2.x */ - PyObject *latin1; - if (Py_SIZE(self)) - latin1 = PyUnicode_DecodeLatin1(buf, Py_SIZE(self), NULL); - else - latin1 = PyUnicode_FromString(""); - return Py_BuildValue("(O(Ns)N)", Py_TYPE(self), latin1, "latin-1", dict); + PyObject *latin1 = PyUnicode_DecodeLatin1(buf, Py_SIZE(self), NULL); + return Py_BuildValue("(O(Ns)N)", Py_TYPE(self), latin1, "latin-1", state); } else { /* use more efficient byte based reduction */ - if (Py_SIZE(self)) { - return Py_BuildValue("(O(y#)N)", Py_TYPE(self), buf, Py_SIZE(self), dict); - } - else { - return Py_BuildValue("(O()N)", Py_TYPE(self), dict); - } + return Py_BuildValue("(O(y#)N)", Py_TYPE(self), buf, Py_SIZE(self), state); } } diff --git a/Objects/clinic/typeobject.c.h b/Objects/clinic/typeobject.c.h index 8c70d76d916db..dee3139bd3d82 100644 --- a/Objects/clinic/typeobject.c.h +++ b/Objects/clinic/typeobject.c.h @@ -130,6 +130,24 @@ type___sizeof__(PyTypeObject *self, PyObject *Py_UNUSED(ignored)) return type___sizeof___impl(self); } +PyDoc_STRVAR(object___getstate____doc__, +"__getstate__($self, /)\n" +"--\n" +"\n" +"Helper for pickle."); + +#define OBJECT___GETSTATE___METHODDEF \ + {"__getstate__", (PyCFunction)object___getstate__, METH_NOARGS, object___getstate____doc__}, + +static PyObject * +object___getstate___impl(PyObject *self); + +static PyObject * +object___getstate__(PyObject *self, PyObject *Py_UNUSED(ignored)) +{ + return object___getstate___impl(self); +} + PyDoc_STRVAR(object___reduce____doc__, "__reduce__($self, /)\n" "--\n" @@ -243,4 +261,4 @@ object___dir__(PyObject *self, PyObject *Py_UNUSED(ignored)) { return object___dir___impl(self); } -/*[clinic end generated code: output=b4fb62939b08baf9 input=a9049054013a1b77]*/ +/*[clinic end generated code: output=a30090032b8e6195 input=a9049054013a1b77]*/ diff --git a/Objects/odictobject.c b/Objects/odictobject.c index c207593ab79f7..f5b8b3e6cdd76 100644 --- a/Objects/odictobject.c +++ b/Objects/odictobject.c @@ -947,23 +947,13 @@ PyDoc_STRVAR(odict_reduce__doc__, "Return state information for pickling"); static PyObject * odict_reduce(register PyODictObject *od, PyObject *Py_UNUSED(ignored)) { - PyObject *dict = NULL, *result = NULL; + PyObject *state, *result = NULL; PyObject *items_iter, *items, *args = NULL; /* capture any instance state */ - dict = PyObject_GetAttr((PyObject *)od, &_Py_ID(__dict__)); - if (dict == NULL) + state = _PyObject_GetState((PyObject *)od); + if (state == NULL) goto Done; - else { - /* od.__dict__ isn't necessarily a dict... */ - Py_ssize_t dict_len = PyObject_Length(dict); - if (dict_len == -1) - goto Done; - if (!dict_len) { - /* nothing to pickle in od.__dict__ */ - Py_CLEAR(dict); - } - } /* build the result */ args = PyTuple_New(0); @@ -979,11 +969,11 @@ odict_reduce(register PyODictObject *od, PyObject *Py_UNUSED(ignored)) if (items_iter == NULL) goto Done; - result = PyTuple_Pack(5, Py_TYPE(od), args, dict ? dict : Py_None, Py_None, items_iter); + result = PyTuple_Pack(5, Py_TYPE(od), args, state, Py_None, items_iter); Py_DECREF(items_iter); Done: - Py_XDECREF(dict); + Py_XDECREF(state); Py_XDECREF(args); return result; diff --git a/Objects/setobject.c b/Objects/setobject.c index ef2190de914b3..4b6a8b8dfb679 100644 --- a/Objects/setobject.c +++ b/Objects/setobject.c @@ -1947,7 +1947,7 @@ an exception when an element is missing from the set."); static PyObject * set_reduce(PySetObject *so, PyObject *Py_UNUSED(ignored)) { - PyObject *keys=NULL, *args=NULL, *result=NULL, *dict=NULL; + PyObject *keys=NULL, *args=NULL, *result=NULL, *state=NULL; keys = PySequence_List((PyObject *)so); if (keys == NULL) @@ -1955,18 +1955,14 @@ set_reduce(PySetObject *so, PyObject *Py_UNUSED(ignored)) args = PyTuple_Pack(1, keys); if (args == NULL) goto done; - if (_PyObject_LookupAttr((PyObject *)so, &_Py_ID(__dict__), &dict) < 0) { + state = _PyObject_GetState((PyObject *)so); + if (state == NULL) goto done; - } - if (dict == NULL) { - dict = Py_None; - Py_INCREF(dict); - } - result = PyTuple_Pack(3, Py_TYPE(so), args, dict); + result = PyTuple_Pack(3, Py_TYPE(so), args, state); done: Py_XDECREF(args); Py_XDECREF(keys); - Py_XDECREF(dict); + Py_XDECREF(state); return result; } diff --git a/Objects/typeobject.c b/Objects/typeobject.c index 5de8c3d2ece43..53e4f0781d65b 100644 --- a/Objects/typeobject.c +++ b/Objects/typeobject.c @@ -4960,143 +4960,175 @@ _PyType_GetSlotNames(PyTypeObject *cls) } static PyObject * -_PyObject_GetState(PyObject *obj, int required) +object_getstate_default(PyObject *obj, int required) { PyObject *state; - PyObject *getstate; + PyObject *slotnames; - if (_PyObject_LookupAttr(obj, &_Py_ID(__getstate__), &getstate) < 0) { + if (required && Py_TYPE(obj)->tp_itemsize) { + PyErr_Format(PyExc_TypeError, + "cannot pickle %.200s objects", + Py_TYPE(obj)->tp_name); + return NULL; + } + + if (_PyObject_IsInstanceDictEmpty(obj)) { + state = Py_None; + Py_INCREF(state); + } + else { + state = PyObject_GenericGetDict(obj, NULL); + if (state == NULL) { + return NULL; + } + } + + slotnames = _PyType_GetSlotNames(Py_TYPE(obj)); + if (slotnames == NULL) { + Py_DECREF(state); return NULL; } - if (getstate == NULL) { - PyObject *slotnames; - if (required && Py_TYPE(obj)->tp_itemsize) { + assert(slotnames == Py_None || PyList_Check(slotnames)); + if (required) { + Py_ssize_t basicsize = PyBaseObject_Type.tp_basicsize; + if (Py_TYPE(obj)->tp_dictoffset && + (Py_TYPE(obj)->tp_flags & Py_TPFLAGS_MANAGED_DICT) == 0) + { + basicsize += sizeof(PyObject *); + } + if (Py_TYPE(obj)->tp_weaklistoffset) { + basicsize += sizeof(PyObject *); + } + if (slotnames != Py_None) { + basicsize += sizeof(PyObject *) * PyList_GET_SIZE(slotnames); + } + if (Py_TYPE(obj)->tp_basicsize > basicsize) { + Py_DECREF(slotnames); + Py_DECREF(state); PyErr_Format(PyExc_TypeError, "cannot pickle '%.200s' object", Py_TYPE(obj)->tp_name); return NULL; } - if (_PyObject_IsInstanceDictEmpty(obj)) { - state = Py_None; - Py_INCREF(state); - } - else { - state = PyObject_GenericGetDict(obj, NULL); - if (state == NULL) { - return NULL; - } - } + } - slotnames = _PyType_GetSlotNames(Py_TYPE(obj)); - if (slotnames == NULL) { + if (slotnames != Py_None && PyList_GET_SIZE(slotnames) > 0) { + PyObject *slots; + Py_ssize_t slotnames_size, i; + + slots = PyDict_New(); + if (slots == NULL) { + Py_DECREF(slotnames); Py_DECREF(state); return NULL; } - assert(slotnames == Py_None || PyList_Check(slotnames)); - if (required) { - Py_ssize_t basicsize = PyBaseObject_Type.tp_basicsize; - if (Py_TYPE(obj)->tp_dictoffset && - (Py_TYPE(obj)->tp_flags & Py_TPFLAGS_MANAGED_DICT) == 0) - { - basicsize += sizeof(PyObject *); + slotnames_size = PyList_GET_SIZE(slotnames); + for (i = 0; i < slotnames_size; i++) { + PyObject *name, *value; + + name = PyList_GET_ITEM(slotnames, i); + Py_INCREF(name); + value = PyObject_GetAttr(obj, name); + if (_PyObject_LookupAttr(obj, name, &value) < 0) { + Py_DECREF(name); + goto error; + } + if (value == NULL) { + Py_DECREF(name); + /* It is not an error if the attribute is not present. */ } - if (Py_TYPE(obj)->tp_weaklistoffset) { - basicsize += sizeof(PyObject *); + else { + int err = PyDict_SetItem(slots, name, value); + Py_DECREF(name); + Py_DECREF(value); + if (err) { + goto error; + } } - if (slotnames != Py_None) { - basicsize += sizeof(PyObject *) * PyList_GET_SIZE(slotnames); + + /* The list is stored on the class so it may mutate while we + iterate over it */ + if (slotnames_size != PyList_GET_SIZE(slotnames)) { + PyErr_Format(PyExc_RuntimeError, + "__slotsname__ changed size during iteration"); + goto error; } - if (Py_TYPE(obj)->tp_basicsize > basicsize) { + + /* We handle errors within the loop here. */ + if (0) { + error: Py_DECREF(slotnames); + Py_DECREF(slots); Py_DECREF(state); - PyErr_Format(PyExc_TypeError, - "cannot pickle '%.200s' object", - Py_TYPE(obj)->tp_name); return NULL; } } - if (slotnames != Py_None && PyList_GET_SIZE(slotnames) > 0) { - PyObject *slots; - Py_ssize_t slotnames_size, i; + /* If we found some slot attributes, pack them in a tuple along + the original attribute dictionary. */ + if (PyDict_GET_SIZE(slots) > 0) { + PyObject *state2; - slots = PyDict_New(); - if (slots == NULL) { + state2 = PyTuple_Pack(2, state, slots); + Py_DECREF(state); + if (state2 == NULL) { Py_DECREF(slotnames); - Py_DECREF(state); + Py_DECREF(slots); return NULL; } + state = state2; + } + Py_DECREF(slots); + } + Py_DECREF(slotnames); - slotnames_size = PyList_GET_SIZE(slotnames); - for (i = 0; i < slotnames_size; i++) { - PyObject *name, *value; - - name = PyList_GET_ITEM(slotnames, i); - Py_INCREF(name); - if (_PyObject_LookupAttr(obj, name, &value) < 0) { - goto error; - } - if (value == NULL) { - Py_DECREF(name); - /* It is not an error if the attribute is not present. */ - } - else { - int err = PyDict_SetItem(slots, name, value); - Py_DECREF(name); - Py_DECREF(value); - if (err) { - goto error; - } - } - - /* The list is stored on the class so it may mutate while we - iterate over it */ - if (slotnames_size != PyList_GET_SIZE(slotnames)) { - PyErr_Format(PyExc_RuntimeError, - "__slotsname__ changed size during iteration"); - goto error; - } - - /* We handle errors within the loop here. */ - if (0) { - error: - Py_DECREF(slotnames); - Py_DECREF(slots); - Py_DECREF(state); - return NULL; - } - } + return state; +} - /* If we found some slot attributes, pack them in a tuple along - the original attribute dictionary. */ - if (PyDict_GET_SIZE(slots) > 0) { - PyObject *state2; +static PyObject * +object_getstate(PyObject *obj, int required) +{ + PyObject *getstate, *state; - state2 = PyTuple_Pack(2, state, slots); - Py_DECREF(state); - if (state2 == NULL) { - Py_DECREF(slotnames); - Py_DECREF(slots); - return NULL; - } - state = state2; - } - Py_DECREF(slots); - } - Py_DECREF(slotnames); + getstate = PyObject_GetAttr(obj, &_Py_ID(__getstate__)); + if (getstate == NULL) { + return NULL; } - else { /* getstate != NULL */ + if (PyCFunction_Check(getstate) && + PyCFunction_GET_SELF(getstate) == obj && + PyCFunction_GET_FUNCTION(getstate) == object___getstate__) + { + /* If __getstate__ is not overriden pass the required argument. */ + state = object_getstate_default(obj, required); + } + else { state = _PyObject_CallNoArgs(getstate); - Py_DECREF(getstate); - if (state == NULL) - return NULL; } - + Py_DECREF(getstate); return state; } +PyObject * +_PyObject_GetState(PyObject *obj) +{ + return object_getstate(obj, 0); +} + +/*[clinic input] +object.__getstate__ + +Helper for pickle. +[clinic start generated code]*/ + +static PyObject * +object___getstate___impl(PyObject *self) +/*[clinic end generated code: output=5a2500dcb6217e9e input=692314d8fbe194ee]*/ +{ + return object_getstate_default(self, 0); +} + static int _PyObject_GetNewArguments(PyObject *obj, PyObject **args, PyObject **kwargs) { @@ -5309,8 +5341,7 @@ reduce_newobj(PyObject *obj) return NULL; } - state = _PyObject_GetState(obj, - !hasargs && !PyList_Check(obj) && !PyDict_Check(obj)); + state = object_getstate(obj, !(hasargs || PyList_Check(obj) || PyDict_Check(obj))); if (state == NULL) { Py_DECREF(newobj); Py_DECREF(newargs); @@ -5558,6 +5589,7 @@ object___dir___impl(PyObject *self) static PyMethodDef object_methods[] = { OBJECT___REDUCE_EX___METHODDEF OBJECT___REDUCE___METHODDEF + OBJECT___GETSTATE___METHODDEF {"__subclasshook__", object_subclasshook, METH_CLASS | METH_VARARGS, object_subclasshook_doc}, {"__init_subclass__", object_init_subclass, METH_CLASS | METH_NOARGS, From webhook-mailer at python.org Wed Apr 6 13:11:17 2022 From: webhook-mailer at python.org (Fidget-Spinner) Date: Wed, 06 Apr 2022 17:11:17 -0000 Subject: [Python-checkins] Fix whitespace/indentation issues in test_sys (GH-32369) (GH-32373) Message-ID: https://github.com/python/cpython/commit/1ab53511e96db42ade7bd8efdd4f237cbcb69252 commit: 1ab53511e96db42ade7bd8efdd4f237cbcb69252 branch: 3.9 author: Ken Jin committer: Fidget-Spinner date: 2022-04-07T01:11:07+08:00 summary: Fix whitespace/indentation issues in test_sys (GH-32369) (GH-32373) (cherry picked from commit da922409ac3e65c6bf2911401c7dfdf8ee6e0036) files: M Lib/test/test_sys.py diff --git a/Lib/test/test_sys.py b/Lib/test/test_sys.py index 2f1e5e971e8e9..ed85d185412ea 100644 --- a/Lib/test/test_sys.py +++ b/Lib/test/test_sys.py @@ -982,8 +982,8 @@ class X(Exception): with test.support.captured_stderr() as stderr, \ test.support.swap_attr(sys, 'unraisablehook', sys.__unraisablehook__): - expected = self.write_unraisable_exc( - A.B.X(), "msg", "obj"); + expected = self.write_unraisable_exc( + A.B.X(), "msg", "obj"); report = stderr.getvalue() testName = 'test_original_unraisablehook_exception_qualname' self.assertIn(f"{testName}..A.B.X", report) From webhook-mailer at python.org Wed Apr 6 14:00:35 2022 From: webhook-mailer at python.org (Fidget-Spinner) Date: Wed, 06 Apr 2022 18:00:35 -0000 Subject: [Python-checkins] [3.10] bpo-46769: Fix backticks in typing.rst to appease rstlint (GH-32374) Message-ID: https://github.com/python/cpython/commit/80af26d25af5568229d31ecb2a8f1bf9702b7791 commit: 80af26d25af5568229d31ecb2a8f1bf9702b7791 branch: 3.10 author: Ken Jin committer: Fidget-Spinner date: 2022-04-07T02:00:26+08:00 summary: [3.10] bpo-46769: Fix backticks in typing.rst to appease rstlint (GH-32374) * Use double backticks to appease rstlint * Update susp-ignored.csv files: M Doc/library/typing.rst M Doc/tools/susp-ignored.csv diff --git a/Doc/library/typing.rst b/Doc/library/typing.rst index ea0bcee81c50a..36d637f366509 100644 --- a/Doc/library/typing.rst +++ b/Doc/library/typing.rst @@ -1158,7 +1158,7 @@ These are not used in annotations. They are building blocks for creating generic self.radius = radius # Use a type variable to show that the return type - # will always be an instance of whatever `cls` is + # will always be an instance of whatever ``cls`` is @classmethod def with_circumference(cls: type[C], circumference: float) -> C: """Create a circle with the specified circumference""" diff --git a/Doc/tools/susp-ignored.csv b/Doc/tools/susp-ignored.csv index 0a8d702f96bce..d7d5b3547a9d9 100644 --- a/Doc/tools/susp-ignored.csv +++ b/Doc/tools/susp-ignored.csv @@ -384,4 +384,4 @@ library/typing,,`,# Type of ``val`` is narrowed to ``str`` library/typing,,`,"# Else, type of ``val`` is narrowed to ``float``." library/typing,,`,# Type of ``val`` is narrowed to ``List[str]``. library/typing,,`,# Type of ``val`` remains as ``List[object]``. -library/typing,,`, # will always be an instance of whatever `cls` is +library/typing,,`, # will always be an instance of whatever ``cls`` is From webhook-mailer at python.org Wed Apr 6 14:03:20 2022 From: webhook-mailer at python.org (Fidget-Spinner) Date: Wed, 06 Apr 2022 18:03:20 -0000 Subject: [Python-checkins] [3.9] bpo-46769: Fix backticks in typing.rst to appease rstlint (GH-32375) Message-ID: https://github.com/python/cpython/commit/d6a7ee69fb3263450ba47bed9104f4a68a08a9bd commit: d6a7ee69fb3263450ba47bed9104f4a68a08a9bd branch: 3.9 author: Ken Jin committer: Fidget-Spinner date: 2022-04-07T02:03:11+08:00 summary: [3.9] bpo-46769: Fix backticks in typing.rst to appease rstlint (GH-32375) * Use double backticks to appease rstlint * Update susp-ignored.csv files: M Doc/library/typing.rst M Doc/tools/susp-ignored.csv diff --git a/Doc/library/typing.rst b/Doc/library/typing.rst index 2689c7f293871..202f5b5e8fec4 100644 --- a/Doc/library/typing.rst +++ b/Doc/library/typing.rst @@ -930,7 +930,7 @@ These are not used in annotations. They are building blocks for creating generic self.radius = radius # Use a type variable to show that the return type - # will always be an instance of whatever `cls` is + # will always be an instance of whatever ``cls`` is @classmethod def with_circumference(cls: type[C], circumference: float) -> C: """Create a circle with the specified circumference""" diff --git a/Doc/tools/susp-ignored.csv b/Doc/tools/susp-ignored.csv index 67e449316e4a2..3eb3d7954f8fb 100644 --- a/Doc/tools/susp-ignored.csv +++ b/Doc/tools/susp-ignored.csv @@ -370,4 +370,4 @@ whatsnew/changelog,,::,default::DeprecationWarning library/importlib.metadata,,:main,"EntryPoint(name='wheel', value='wheel.cli:main', group='console_scripts')" library/importlib.metadata,,`,loading the metadata for packages for the indicated ``context``. library/re,,`,"`" -library/typing,,`, # will always be an instance of whatever `cls` is +library/typing,,`, # will always be an instance of whatever ``cls`` is From webhook-mailer at python.org Wed Apr 6 14:03:41 2022 From: webhook-mailer at python.org (rhettinger) Date: Wed, 06 Apr 2022 18:03:41 -0000 Subject: [Python-checkins] Minor code nit: Move an unrelated statement out of a try clause in Sequence.index (GH-32330) Message-ID: https://github.com/python/cpython/commit/59a99ae277e7d9f47edd4a538c1239d39f10db0c commit: 59a99ae277e7d9f47edd4a538c1239d39f10db0c branch: main author: G?ry Ogam committer: rhettinger date: 2022-04-06T13:03:36-05:00 summary: Minor code nit: Move an unrelated statement out of a try clause in Sequence.index (GH-32330) files: M Lib/_collections_abc.py diff --git a/Lib/_collections_abc.py b/Lib/_collections_abc.py index 86eb042e3a75a..e96e4c3535586 100644 --- a/Lib/_collections_abc.py +++ b/Lib/_collections_abc.py @@ -1022,10 +1022,10 @@ def index(self, value, start=0, stop=None): while stop is None or i < stop: try: v = self[i] - if v is value or v == value: - return i except IndexError: break + if v is value or v == value: + return i i += 1 raise ValueError From webhook-mailer at python.org Wed Apr 6 14:22:51 2022 From: webhook-mailer at python.org (brettcannon) Date: Wed, 06 Apr 2022 18:22:51 -0000 Subject: [Python-checkins] bpo-47061: use `warnings._deprecated()` with asynchat, asyncore, and smtpd (GH-32350) Message-ID: https://github.com/python/cpython/commit/32b33879c2e19cde735c1971b06869976200e1d8 commit: 32b33879c2e19cde735c1971b06869976200e1d8 branch: main author: Brett Cannon committer: brettcannon date: 2022-04-06T11:22:39-07:00 summary: bpo-47061: use `warnings._deprecated()` with asynchat, asyncore, and smtpd (GH-32350) files: M Lib/asynchat.py M Lib/asyncore.py M Lib/smtpd.py diff --git a/Lib/asynchat.py b/Lib/asynchat.py index e081e67c75acb..bed797e989e13 100644 --- a/Lib/asynchat.py +++ b/Lib/asynchat.py @@ -48,12 +48,11 @@ import asyncore from collections import deque -from warnings import warn -warn( - 'The asynchat module is deprecated and will be removed in Python 3.12. ' - 'The recommended replacement is asyncio', - DeprecationWarning, - stacklevel=2) +from warnings import _deprecated + +_DEPRECATION_MSG = ('The {name} module is deprecated and will be removed in ' + 'Python {remove}. The recommended replacement is asyncio') +_deprecated(__name__, _DEPRECATION_MSG, remove=(3, 12)) diff --git a/Lib/asyncore.py b/Lib/asyncore.py index a360d404395e5..57c86871f3dcf 100644 --- a/Lib/asyncore.py +++ b/Lib/asyncore.py @@ -57,11 +57,9 @@ ENOTCONN, ESHUTDOWN, EISCONN, EBADF, ECONNABORTED, EPIPE, EAGAIN, \ errorcode -warnings.warn( - 'The asyncore module is deprecated and will be removed in Python 3.12. ' - 'The recommended replacement is asyncio', - DeprecationWarning, - stacklevel=2) +_DEPRECATION_MSG = ('The {name} module is deprecated and will be removed in ' + 'Python {remove}. The recommended replacement is asyncio') +warnings._deprecated(__name__, _DEPRECATION_MSG, remove=(3, 12)) _DISCONNECTED = frozenset({ECONNRESET, ENOTCONN, ESHUTDOWN, ECONNABORTED, EPIPE, diff --git a/Lib/smtpd.py b/Lib/smtpd.py index eeda155b920f7..b23579f120716 100755 --- a/Lib/smtpd.py +++ b/Lib/smtpd.py @@ -77,19 +77,18 @@ import time import socket import collections -from warnings import warn +from warnings import _deprecated, warn from email._header_value_parser import get_addr_spec, get_angle_addr __all__ = [ "SMTPChannel", "SMTPServer", "DebuggingServer", "PureProxy", ] -warn( - 'The smtpd module is deprecated and unmaintained and will be removed ' - 'in Python 3.12. Please see aiosmtpd ' - '(https://aiosmtpd.readthedocs.io/) for the recommended replacement.', - DeprecationWarning, - stacklevel=2) +_DEPRECATION_MSG = ('The {name} module is deprecated and unmaintained and will ' + 'be removed in Python {remove}. Please see aiosmtpd ' + '(https://aiosmtpd.readthedocs.io/) for the recommended ' + 'replacement.') +_deprecated(__name__, _DEPRECATION_MSG, remove=(3, 12)) # These are imported after the above warning so that users get the correct From webhook-mailer at python.org Wed Apr 6 15:35:33 2022 From: webhook-mailer at python.org (rhettinger) Date: Wed, 06 Apr 2022 19:35:33 -0000 Subject: [Python-checkins] Change parameter name from *x* for reals to *n* for integers. (GH-32377) Message-ID: https://github.com/python/cpython/commit/1ba82d4419010994ddf68b9b4851509acc4d45e1 commit: 1ba82d4419010994ddf68b9b4851509acc4d45e1 branch: main author: Raymond Hettinger committer: rhettinger date: 2022-04-06T14:35:05-05:00 summary: Change parameter name from *x* for reals to *n* for integers. (GH-32377) files: M Doc/library/math.rst M Modules/clinic/mathmodule.c.h M Modules/mathmodule.c diff --git a/Doc/library/math.rst b/Doc/library/math.rst index 48030fae189ba..5efcc7f6a6efc 100644 --- a/Doc/library/math.rst +++ b/Doc/library/math.rst @@ -66,9 +66,9 @@ Number-theoretic and representation functions Return the absolute value of *x*. -.. function:: factorial(x) +.. function:: factorial(n) - Return *x* factorial as an integer. Raises :exc:`ValueError` if *x* is not integral or + Return *n* factorial as an integer. Raises :exc:`ValueError` if *n* is not integral or is negative. .. deprecated:: 3.9 diff --git a/Modules/clinic/mathmodule.c.h b/Modules/clinic/mathmodule.c.h index 65f3dd4f520ae..4ea0bb82b8e2c 100644 --- a/Modules/clinic/mathmodule.c.h +++ b/Modules/clinic/mathmodule.c.h @@ -45,10 +45,10 @@ PyDoc_STRVAR(math_isqrt__doc__, {"isqrt", (PyCFunction)math_isqrt, METH_O, math_isqrt__doc__}, PyDoc_STRVAR(math_factorial__doc__, -"factorial($module, x, /)\n" +"factorial($module, n, /)\n" "--\n" "\n" -"Find x!.\n" +"Find n!.\n" "\n" "Raise a ValueError if x is negative or non-integral."); @@ -865,4 +865,4 @@ math_ulp(PyObject *module, PyObject *arg) exit: return return_value; } -/*[clinic end generated code: output=1eae2b3ef19568fa input=a9049054013a1b77]*/ +/*[clinic end generated code: output=6381e7d982ff3711 input=a9049054013a1b77]*/ diff --git a/Modules/mathmodule.c b/Modules/mathmodule.c index f0aaf23845f6d..8dd763f2720ee 100644 --- a/Modules/mathmodule.c +++ b/Modules/mathmodule.c @@ -2093,17 +2093,17 @@ static const unsigned long SmallFactorials[] = { /*[clinic input] math.factorial - x as arg: object + n as arg: object / -Find x!. +Find n!. Raise a ValueError if x is negative or non-integral. [clinic start generated code]*/ static PyObject * math_factorial(PyObject *module, PyObject *arg) -/*[clinic end generated code: output=6686f26fae00e9ca input=6d1c8105c0d91fb4]*/ +/*[clinic end generated code: output=6686f26fae00e9ca input=713fb771677e8c31]*/ { long x, two_valuation; int overflow; From webhook-mailer at python.org Wed Apr 6 17:56:12 2022 From: webhook-mailer at python.org (gpshead) Date: Wed, 06 Apr 2022 21:56:12 -0000 Subject: [Python-checkins] bpo-46576: Speed up test_peg_generator by using a static library for shared sources (GH-32338) Message-ID: https://github.com/python/cpython/commit/612e422c6ea9df60d3382ed1491d8254c283e93f commit: 612e422c6ea9df60d3382ed1491d8254c283e93f branch: main author: Jeremy Kloth committer: gpshead date: 2022-04-06T14:55:58-07:00 summary: bpo-46576: Speed up test_peg_generator by using a static library for shared sources (GH-32338) Speed up test_peg_generator by using a static library for shared sources to avoid recompiling as much code. files: M Lib/test/test_peg_generator/test_c_parser.py M Tools/peg_generator/pegen/build.py M Tools/peg_generator/pegen/testutil.py diff --git a/Lib/test/test_peg_generator/test_c_parser.py b/Lib/test/test_peg_generator/test_c_parser.py index 51a4f7d7c07a0..13b83a9db9eb3 100644 --- a/Lib/test/test_peg_generator/test_c_parser.py +++ b/Lib/test/test_peg_generator/test_c_parser.py @@ -72,13 +72,30 @@ def test_parse(self): @support.requires_subprocess() class TestCParser(unittest.TestCase): + + @classmethod + def setUpClass(cls): + # When running under regtest, a seperate tempdir is used + # as the current directory and watched for left-overs. + # Reusing that as the base for temporary directories + # ensures everything is cleaned up properly and + # cleans up afterwards if not (with warnings). + cls.tmp_base = os.getcwd() + if os.path.samefile(cls.tmp_base, os_helper.SAVEDCWD): + cls.tmp_base = None + # Create a directory for the reuseable static library part of + # the pegen extension build process. This greatly reduces the + # runtime overhead of spawning compiler processes. + cls.library_dir = tempfile.mkdtemp(dir=cls.tmp_base) + cls.addClassCleanup(shutil.rmtree, cls.library_dir) + def setUp(self): self._backup_config_vars = dict(sysconfig._CONFIG_VARS) cmd = support.missing_compiler_executable() if cmd is not None: self.skipTest("The %r command is not found" % cmd) self.old_cwd = os.getcwd() - self.tmp_path = tempfile.mkdtemp() + self.tmp_path = tempfile.mkdtemp(dir=self.tmp_base) change_cwd = os_helper.change_cwd(self.tmp_path) change_cwd.__enter__() self.addCleanup(change_cwd.__exit__, None, None, None) @@ -91,7 +108,10 @@ def tearDown(self): def build_extension(self, grammar_source): grammar = parse_string(grammar_source, GrammarParser) - generate_parser_c_extension(grammar, Path(self.tmp_path)) + # Because setUp() already changes the current directory to the + # temporary path, use a relative path here to prevent excessive + # path lengths when compiling. + generate_parser_c_extension(grammar, Path('.'), library_dir=self.library_dir) def run_test(self, grammar_source, test_source): self.build_extension(grammar_source) diff --git a/Tools/peg_generator/pegen/build.py b/Tools/peg_generator/pegen/build.py index 78789b94df2e4..5805ff6371744 100644 --- a/Tools/peg_generator/pegen/build.py +++ b/Tools/peg_generator/pegen/build.py @@ -1,6 +1,5 @@ import itertools import pathlib -import shutil import sys import sysconfig import tempfile @@ -33,7 +32,8 @@ def compile_c_extension( build_dir: Optional[str] = None, verbose: bool = False, keep_asserts: bool = True, - disable_optimization: bool = True, # Significant test_peg_generator speedup. + disable_optimization: bool = False, + library_dir: Optional[str] = None, ) -> str: """Compile the generated source for a parser generator into an extension module. @@ -44,15 +44,21 @@ def compile_c_extension( If *build_dir* is provided, that path will be used as the temporary build directory of distutils (this is useful in case you want to use a temporary directory). + + If *library_dir* is provided, that path will be used as the directory for a + static library of the common parser sources (this is useful in case you are + creating multiple extensions). """ import distutils.log - from distutils.command.build_ext import build_ext # type: ignore - from distutils.command.clean import clean # type: ignore from distutils.core import Distribution, Extension from distutils.tests.support import fixup_build_ext # type: ignore + from distutils.ccompiler import new_compiler + from distutils.dep_util import newer_group + from distutils.sysconfig import customize_compiler + if verbose: - distutils.log.set_verbosity(distutils.log.DEBUG) + distutils.log.set_threshold(distutils.log.DEBUG) source_file_path = pathlib.Path(generated_source_path) extension_name = source_file_path.stem @@ -71,46 +77,92 @@ def compile_c_extension( extra_compile_args.append("-O0") if sysconfig.get_config_var("GNULD") == "yes": extra_link_args.append("-fno-lto") - extension = [ - Extension( - extension_name, - sources=[ - str(MOD_DIR.parent.parent.parent / "Python" / "Python-ast.c"), - str(MOD_DIR.parent.parent.parent / "Python" / "asdl.c"), - str(MOD_DIR.parent.parent.parent / "Parser" / "tokenizer.c"), - str(MOD_DIR.parent.parent.parent / "Parser" / "pegen.c"), - str(MOD_DIR.parent.parent.parent / "Parser" / "pegen_errors.c"), - str(MOD_DIR.parent.parent.parent / "Parser" / "action_helpers.c"), - str(MOD_DIR.parent.parent.parent / "Parser" / "string_parser.c"), - str(MOD_DIR.parent / "peg_extension" / "peg_extension.c"), - generated_source_path, - ], - include_dirs=[ - str(MOD_DIR.parent.parent.parent / "Include" / "internal"), - str(MOD_DIR.parent.parent.parent / "Parser"), - ], - extra_compile_args=extra_compile_args, - extra_link_args=extra_link_args, - ) + + common_sources = [ + str(MOD_DIR.parent.parent.parent / "Python" / "Python-ast.c"), + str(MOD_DIR.parent.parent.parent / "Python" / "asdl.c"), + str(MOD_DIR.parent.parent.parent / "Parser" / "tokenizer.c"), + str(MOD_DIR.parent.parent.parent / "Parser" / "pegen.c"), + str(MOD_DIR.parent.parent.parent / "Parser" / "pegen_errors.c"), + str(MOD_DIR.parent.parent.parent / "Parser" / "action_helpers.c"), + str(MOD_DIR.parent.parent.parent / "Parser" / "string_parser.c"), + str(MOD_DIR.parent / "peg_extension" / "peg_extension.c"), + ] + include_dirs = [ + str(MOD_DIR.parent.parent.parent / "Include" / "internal"), + str(MOD_DIR.parent.parent.parent / "Parser"), ] - dist = Distribution({"name": extension_name, "ext_modules": extension}) - cmd = build_ext(dist) + extension = Extension( + extension_name, + sources=[generated_source_path], + extra_compile_args=extra_compile_args, + extra_link_args=extra_link_args, + ) + dist = Distribution({"name": extension_name, "ext_modules": [extension]}) + cmd = dist.get_command_obj("build_ext") fixup_build_ext(cmd) - cmd.inplace = True + cmd.build_lib = str(source_file_path.parent) + cmd.include_dirs = include_dirs if build_dir: cmd.build_temp = build_dir - cmd.build_lib = build_dir cmd.ensure_finalized() - cmd.run() - - extension_path = source_file_path.parent / cmd.get_ext_filename(extension_name) - shutil.move(cmd.get_ext_fullpath(extension_name), extension_path) - - cmd = clean(dist) - cmd.finalize_options() - cmd.run() - return extension_path + compiler = new_compiler() + customize_compiler(compiler) + compiler.set_include_dirs(cmd.include_dirs) + compiler.set_library_dirs(cmd.library_dirs) + # build static lib + if library_dir: + library_filename = compiler.library_filename(extension_name, + output_dir=library_dir) + if newer_group(common_sources, library_filename, 'newer'): + if sys.platform == 'win32': + pdb = compiler.static_lib_format % (extension_name, '.pdb') + compile_opts = [f"/Fd{library_dir}\\{pdb}"] + compile_opts.extend(extra_compile_args) + else: + compile_opts = extra_compile_args + objects = compiler.compile(common_sources, + output_dir=library_dir, + debug=cmd.debug, + extra_postargs=compile_opts) + compiler.create_static_lib(objects, extension_name, + output_dir=library_dir, + debug=cmd.debug) + if sys.platform == 'win32': + compiler.add_library_dir(library_dir) + extension.libraries = [extension_name] + elif sys.platform == 'darwin': + compiler.set_link_objects([ + '-Wl,-force_load', library_filename, + ]) + else: + compiler.set_link_objects([ + '-Wl,--whole-archive', library_filename, '-Wl,--no-whole-archive', + ]) + else: + extension.sources[0:0] = common_sources + + # Compile the source code to object files. + ext_path = cmd.get_ext_fullpath(extension_name) + if newer_group(extension.sources, ext_path, 'newer'): + objects = compiler.compile(extension.sources, + output_dir=cmd.build_temp, + debug=cmd.debug, + extra_postargs=extra_compile_args) + else: + objects = compiler.object_filenames(extension.sources, + output_dir=cmd.build_temp) + # Now link the object files together into a "shared object" + compiler.link_shared_object( + objects, ext_path, + libraries=cmd.get_libraries(extension), + extra_postargs=extra_link_args, + export_symbols=cmd.get_export_symbols(extension), + debug=cmd.debug, + build_temp=cmd.build_temp) + + return pathlib.Path(ext_path) def build_parser( diff --git a/Tools/peg_generator/pegen/testutil.py b/Tools/peg_generator/pegen/testutil.py index 8e5dbc5cdbb33..473d208a78671 100644 --- a/Tools/peg_generator/pegen/testutil.py +++ b/Tools/peg_generator/pegen/testutil.py @@ -6,7 +6,7 @@ import textwrap import token import tokenize -from typing import IO, Any, Dict, Final, Type, cast +from typing import IO, Any, Dict, Final, Optional, Type, cast from pegen.build import compile_c_extension from pegen.c_generator import CParserGenerator @@ -83,7 +83,8 @@ def generate_c_parser_source(grammar: Grammar) -> str: def generate_parser_c_extension( - grammar: Grammar, path: pathlib.PurePath, debug: bool = False + grammar: Grammar, path: pathlib.PurePath, debug: bool = False, + library_dir: Optional[str] = None, ) -> Any: """Generate a parser c extension for the given grammar in the given path @@ -101,7 +102,13 @@ def generate_parser_c_extension( grammar, ALL_TOKENS, EXACT_TOKENS, NON_EXACT_TOKENS, file, debug=debug ) genr.generate("parse.c") - compile_c_extension(str(source), build_dir=str(path)) + compile_c_extension( + str(source), + build_dir=str(path), + # Significant test_peg_generator speedups + disable_optimization=True, + library_dir=library_dir, + ) def print_memstats() -> bool: From webhook-mailer at python.org Wed Apr 6 19:05:36 2022 From: webhook-mailer at python.org (vstinner) Date: Wed, 06 Apr 2022 23:05:36 -0000 Subject: [Python-checkins] bpo-35134: Add Include/cpython/complexobject.h header (GH-32383) Message-ID: https://github.com/python/cpython/commit/ca219f6dfc57f8f4984f96df0f733b7de92fe91c commit: ca219f6dfc57f8f4984f96df0f733b7de92fe91c branch: main author: Victor Stinner committer: vstinner date: 2022-04-07T01:05:27+02:00 summary: bpo-35134: Add Include/cpython/complexobject.h header (GH-32383) Move the private _PyComplex_FormatAdvancedWriter() function to the internal C API. This function is no longer exported. files: A Include/cpython/complexobject.h M Include/complexobject.h M Makefile.pre.in M PCbuild/pythoncore.vcxproj M PCbuild/pythoncore.vcxproj.filters diff --git a/Include/complexobject.h b/Include/complexobject.h index 9221f9c51d65b..c7764e43803d6 100644 --- a/Include/complexobject.h +++ b/Include/complexobject.h @@ -6,61 +6,22 @@ extern "C" { #endif -#ifndef Py_LIMITED_API -typedef struct { - double real; - double imag; -} Py_complex; - -/* Operations on complex numbers from complexmodule.c */ - -PyAPI_FUNC(Py_complex) _Py_c_sum(Py_complex, Py_complex); -PyAPI_FUNC(Py_complex) _Py_c_diff(Py_complex, Py_complex); -PyAPI_FUNC(Py_complex) _Py_c_neg(Py_complex); -PyAPI_FUNC(Py_complex) _Py_c_prod(Py_complex, Py_complex); -PyAPI_FUNC(Py_complex) _Py_c_quot(Py_complex, Py_complex); -PyAPI_FUNC(Py_complex) _Py_c_pow(Py_complex, Py_complex); -PyAPI_FUNC(double) _Py_c_abs(Py_complex); -#endif - /* Complex object interface */ -/* -PyComplexObject represents a complex number with double-precision -real and imaginary parts. -*/ -#ifndef Py_LIMITED_API -typedef struct { - PyObject_HEAD - Py_complex cval; -} PyComplexObject; -#endif - PyAPI_DATA(PyTypeObject) PyComplex_Type; #define PyComplex_Check(op) PyObject_TypeCheck(op, &PyComplex_Type) #define PyComplex_CheckExact(op) Py_IS_TYPE(op, &PyComplex_Type) -#ifndef Py_LIMITED_API -PyAPI_FUNC(PyObject *) PyComplex_FromCComplex(Py_complex); -#endif PyAPI_FUNC(PyObject *) PyComplex_FromDoubles(double real, double imag); PyAPI_FUNC(double) PyComplex_RealAsDouble(PyObject *op); PyAPI_FUNC(double) PyComplex_ImagAsDouble(PyObject *op); -#ifndef Py_LIMITED_API -PyAPI_FUNC(Py_complex) PyComplex_AsCComplex(PyObject *op); -#endif -/* Format the object based on the format_spec, as defined in PEP 3101 - (Advanced String Formatting). */ #ifndef Py_LIMITED_API -PyAPI_FUNC(int) _PyComplex_FormatAdvancedWriter( - _PyUnicodeWriter *writer, - PyObject *obj, - PyObject *format_spec, - Py_ssize_t start, - Py_ssize_t end); +# define Py_CPYTHON_COMPLEXOBJECT_H +# include "cpython/complexobject.h" +# undef Py_CPYTHON_COMPLEXOBJECT_H #endif #ifdef __cplusplus diff --git a/Include/cpython/complexobject.h b/Include/cpython/complexobject.h new file mode 100644 index 0000000000000..b7d7283ae8896 --- /dev/null +++ b/Include/cpython/complexobject.h @@ -0,0 +1,44 @@ +#ifndef Py_CPYTHON_COMPLEXOBJECT_H +# error "this header file must not be included directly" +#endif + +typedef struct { + double real; + double imag; +} Py_complex; + +/* Operations on complex numbers from complexmodule.c */ + +PyAPI_FUNC(Py_complex) _Py_c_sum(Py_complex, Py_complex); +PyAPI_FUNC(Py_complex) _Py_c_diff(Py_complex, Py_complex); +PyAPI_FUNC(Py_complex) _Py_c_neg(Py_complex); +PyAPI_FUNC(Py_complex) _Py_c_prod(Py_complex, Py_complex); +PyAPI_FUNC(Py_complex) _Py_c_quot(Py_complex, Py_complex); +PyAPI_FUNC(Py_complex) _Py_c_pow(Py_complex, Py_complex); +PyAPI_FUNC(double) _Py_c_abs(Py_complex); + +/* Complex object interface */ + +/* +PyComplexObject represents a complex number with double-precision +real and imaginary parts. +*/ +typedef struct { + PyObject_HEAD + Py_complex cval; +} PyComplexObject; + +PyAPI_FUNC(PyObject *) PyComplex_FromCComplex(Py_complex); + +PyAPI_FUNC(Py_complex) PyComplex_AsCComplex(PyObject *op); + +#ifdef Py_BUILD_CORE +/* Format the object based on the format_spec, as defined in PEP 3101 + (Advanced String Formatting). */ +extern int _PyComplex_FormatAdvancedWriter( + _PyUnicodeWriter *writer, + PyObject *obj, + PyObject *format_spec, + Py_ssize_t start, + Py_ssize_t end); +#endif // Py_BUILD_CORE diff --git a/Makefile.pre.in b/Makefile.pre.in index d9b96f52ec9f7..69e97234ae737 100644 --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -1522,6 +1522,7 @@ PYTHON_HEADERS= \ $(srcdir)/Include/cpython/classobject.h \ $(srcdir)/Include/cpython/code.h \ $(srcdir)/Include/cpython/compile.h \ + $(srcdir)/Include/cpython/complexobject.h \ $(srcdir)/Include/cpython/context.h \ $(srcdir)/Include/cpython/descrobject.h \ $(srcdir)/Include/cpython/dictobject.h \ diff --git a/PCbuild/pythoncore.vcxproj b/PCbuild/pythoncore.vcxproj index 4fc6aa80f6fae..1d7c08b2c8a71 100644 --- a/PCbuild/pythoncore.vcxproj +++ b/PCbuild/pythoncore.vcxproj @@ -141,6 +141,7 @@ + diff --git a/PCbuild/pythoncore.vcxproj.filters b/PCbuild/pythoncore.vcxproj.filters index 55fca4982e0f3..571a7f13bc1a0 100644 --- a/PCbuild/pythoncore.vcxproj.filters +++ b/PCbuild/pythoncore.vcxproj.filters @@ -357,6 +357,9 @@ Include + + Include + Include\cpython From webhook-mailer at python.org Wed Apr 6 19:10:09 2022 From: webhook-mailer at python.org (zooba) Date: Wed, 06 Apr 2022 23:10:09 -0000 Subject: [Python-checkins] bpo-47239: Fixes py.exe output when run in a virtual environment. (GH-32364) Message-ID: https://github.com/python/cpython/commit/2390b2236d4b6ea96217478221d6f7d4b4f344f8 commit: 2390b2236d4b6ea96217478221d6f7d4b4f344f8 branch: main author: Steve Dower committer: zooba date: 2022-04-07T00:09:54+01:00 summary: bpo-47239: Fixes py.exe output when run in a virtual environment. (GH-32364) files: A Misc/NEWS.d/next/Windows/2022-04-06-15-16-37.bpo-47239.B1HP7i.rst M Lib/test/test_launcher.py M PC/launcher2.c diff --git a/Lib/test/test_launcher.py b/Lib/test/test_launcher.py index 52b1cfa212b88..2e10c55e339d5 100644 --- a/Lib/test/test_launcher.py +++ b/Lib/test/test_launcher.py @@ -2,6 +2,7 @@ import itertools import os import re +import shutil import subprocess import sys import sysconfig @@ -59,12 +60,18 @@ } } -TEST_PY_COMMANDS = textwrap.dedent(""" - [defaults] - py_python=PythonTestSuite/3.100 - py_python2=PythonTestSuite/3.100-32 - py_python3=PythonTestSuite/3.100-arm64 -""") + +TEST_PY_ENV = dict( + PY_PYTHON="PythonTestSuite/3.100", + PY_PYTHON2="PythonTestSuite/3.100-32", + PY_PYTHON3="PythonTestSuite/3.100-arm64", +) + + +TEST_PY_COMMANDS = "\n".join([ + "[defaults]", + *[f"{k.lower()}={v}" for k, v in TEST_PY_ENV.items()] +]) def create_registry_data(root, data): @@ -185,8 +192,13 @@ def run_py(self, args, env=None, allow_fail=False, expect_returncode=0): if not self.py_exe: self.py_exe = self.find_py() - env = {**os.environ, **(env or {}), "PYLAUNCHER_DEBUG": "1", "PYLAUNCHER_DRYRUN": "1"} - env.pop("VIRTUAL_ENV", None) + ignore = {"VIRTUAL_ENV", "PY_PYTHON", "PY_PYTHON2", "PY_PYTHON3"} + env = { + **{k.upper(): v for k, v in os.environ.items() if k.upper() not in ignore}, + **{k.upper(): v for k, v in (env or {}).items()}, + "PYLAUNCHER_DEBUG": "1", + "PYLAUNCHER_DRYRUN": "1", + } with subprocess.Popen( [self.py_exe, *args], env=env, @@ -410,6 +422,60 @@ def test_py3_default(self): self.assertEqual("3.100-arm64", data["SearchInfo.tag"]) self.assertEqual("X.Y-arm64.exe -X fake_arg_for_test -arg", data["stdout"].strip()) + def test_py_default_env(self): + data = self.run_py(["-arg"], env=TEST_PY_ENV) + self.assertEqual("PythonTestSuite", data["SearchInfo.company"]) + self.assertEqual("3.100", data["SearchInfo.tag"]) + self.assertEqual("X.Y.exe -arg", data["stdout"].strip()) + + def test_py2_default_env(self): + data = self.run_py(["-2", "-arg"], env=TEST_PY_ENV) + self.assertEqual("PythonTestSuite", data["SearchInfo.company"]) + self.assertEqual("3.100-32", data["SearchInfo.tag"]) + self.assertEqual("X.Y-32.exe -arg", data["stdout"].strip()) + + def test_py3_default_env(self): + data = self.run_py(["-3", "-arg"], env=TEST_PY_ENV) + self.assertEqual("PythonTestSuite", data["SearchInfo.company"]) + self.assertEqual("3.100-arm64", data["SearchInfo.tag"]) + self.assertEqual("X.Y-arm64.exe -X fake_arg_for_test -arg", data["stdout"].strip()) + + def test_py_default_in_list(self): + data = self.run_py(["-0"], env=TEST_PY_ENV) + default = None + for line in data["stdout"].splitlines(): + m = re.match(r"\s*-V:(.+?)\s+?\*\s+(.+)$", line) + if m: + default = m.group(1) + break + self.assertEqual("PythonTestSuite/3.100", default) + + def test_virtualenv_in_list(self): + venv = Path.cwd() / "Scripts" + venv.mkdir(exist_ok=True, parents=True) + venv_exe = (venv / Path(sys.executable).name) + venv_exe.touch() + try: + data = self.run_py(["-0p"], env={"VIRTUAL_ENV": str(venv.parent)}) + for line in data["stdout"].splitlines(): + m = re.match(r"\s*\*\s+(.+)$", line) + if m: + self.assertEqual(str(venv_exe), m.group(1)) + break + else: + self.fail("did not find active venv path") + + data = self.run_py(["-0"], env={"VIRTUAL_ENV": str(venv.parent)}) + for line in data["stdout"].splitlines(): + m = re.match(r"\s*\*\s+(.+)$", line) + if m: + self.assertEqual("Active venv", m.group(1)) + break + else: + self.fail("did not find active venv entry") + finally: + shutil.rmtree(venv) + def test_py_shebang(self): with self.py_ini(TEST_PY_COMMANDS): with self.script("#! /usr/bin/env python -prearg") as script: diff --git a/Misc/NEWS.d/next/Windows/2022-04-06-15-16-37.bpo-47239.B1HP7i.rst b/Misc/NEWS.d/next/Windows/2022-04-06-15-16-37.bpo-47239.B1HP7i.rst new file mode 100644 index 0000000000000..d8018888c2c48 --- /dev/null +++ b/Misc/NEWS.d/next/Windows/2022-04-06-15-16-37.bpo-47239.B1HP7i.rst @@ -0,0 +1,2 @@ +Fixed --list and --list-paths output for :ref:`launcher` when used in an +active virtual environment. diff --git a/PC/launcher2.c b/PC/launcher2.c index 873538171d93e..35c932aa329c8 100644 --- a/PC/launcher2.c +++ b/PC/launcher2.c @@ -196,9 +196,9 @@ int _compare(const wchar_t *x, int xLen, const wchar_t *y, int yLen) { // Empty strings sort first - if (xLen == 0) { - return yLen == 0 ? 0 : -1; - } else if (yLen == 0) { + if (!x || !xLen) { + return (!y || !yLen) ? 0 : -1; + } else if (!y || !yLen) { return 1; } switch (CompareStringEx( @@ -223,9 +223,9 @@ int _compareArgument(const wchar_t *x, int xLen, const wchar_t *y, int yLen) { // Empty strings sort first - if (xLen == 0) { - return yLen == 0 ? 0 : -1; - } else if (yLen == 0) { + if (!x || !xLen) { + return (!y || !yLen) ? 0 : -1; + } else if (!y || !yLen) { return 1; } switch (CompareStringEx( @@ -249,9 +249,9 @@ int _comparePath(const wchar_t *x, int xLen, const wchar_t *y, int yLen) { // Empty strings sort first - if (xLen == 0) { - return yLen == 0 ? 0 : -1; - } else if (yLen == 0) { + if (!x || !xLen) { + return !y || !yLen ? 0 : -1; + } else if (!y || !yLen) { return 1; } switch (CompareStringOrdinal(x, xLen, y, yLen, TRUE)) { @@ -271,6 +271,9 @@ _comparePath(const wchar_t *x, int xLen, const wchar_t *y, int yLen) bool _startsWith(const wchar_t *x, int xLen, const wchar_t *y, int yLen) { + if (!x || !y) { + return false; + } yLen = yLen < 0 ? (int)wcsnlen_s(y, MAXLEN) : yLen; xLen = xLen < 0 ? (int)wcsnlen_s(x, MAXLEN) : xLen; return xLen >= yLen && 0 == _compare(x, yLen, y, yLen); @@ -280,6 +283,9 @@ _startsWith(const wchar_t *x, int xLen, const wchar_t *y, int yLen) bool _startsWithArgument(const wchar_t *x, int xLen, const wchar_t *y, int yLen) { + if (!x || !y) { + return false; + } yLen = yLen < 0 ? (int)wcsnlen_s(y, MAXLEN) : yLen; xLen = xLen < 0 ? (int)wcsnlen_s(x, MAXLEN) : xLen; return xLen >= yLen && 0 == _compareArgument(x, yLen, y, yLen); @@ -895,8 +901,13 @@ checkShebang(SearchInfo *search) search->oldStyleTag = true; search->executableArgs = &command[commandLength]; search->executableArgsLength = shebangLength - commandLength; - debug(L"# Treating shebang command '%.*s' as 'py -%.*s'\n", - commandLength, command, search->tagLength, search->tag); + if (search->tag && search->tagLength) { + debug(L"# Treating shebang command '%.*s' as 'py -%.*s'\n", + commandLength, command, search->tagLength, search->tag); + } else { + debug(L"# Treating shebang command '%.*s' as 'py'\n", + commandLength, command); + } } else { debug(L"# Found shebang command but could not execute it: %.*s\n", commandLength, command); @@ -912,32 +923,10 @@ checkShebang(SearchInfo *search) int checkDefaults(SearchInfo *search) { - if (!search->allowDefaults || search->list || search->listPaths) { + if (!search->allowDefaults) { return 0; } - wchar_t buffer[MAXLEN]; - int n; - - // If no tag was provided at all, look for an active virtual environment - if (!search->tag || !search->tagLength) { - n = GetEnvironmentVariableW(L"VIRTUAL_ENV", buffer, MAXLEN); - if (n && join(buffer, MAXLEN, L"Scripts") && join(buffer, MAXLEN, search->executable)) { - if (INVALID_FILE_ATTRIBUTES != GetFileAttributesW(buffer)) { - n = (int)wcsnlen_s(buffer, MAXLEN) + 1; - wchar_t *path = allocSearchInfoBuffer(search, n); - if (!path) { - return RC_NO_MEMORY; - } - search->executablePath = path; - wcscpy_s(path, n, buffer); - return 0; - } else { - debug(L"Python executable %s missing from virtual env\n", buffer); - } - } - } - // Only resolve old-style (or absent) tags to defaults if (search->tag && search->tagLength && !search->oldStyleTag) { return 0; @@ -958,7 +947,8 @@ checkDefaults(SearchInfo *search) } // First, try to read an environment variable - n = GetEnvironmentVariableW(settingName, buffer, MAXLEN); + wchar_t buffer[MAXLEN]; + int n = GetEnvironmentVariableW(settingName, buffer, MAXLEN); // If none found, check in our two .ini files instead if (!n) { @@ -1005,6 +995,7 @@ typedef struct EnvironmentInfo { const wchar_t *executableArgs; const wchar_t *architecture; const wchar_t *displayName; + bool isActiveVenv; } EnvironmentInfo; @@ -1076,6 +1067,14 @@ freeEnvironmentInfo(EnvironmentInfo *env) int _compareCompany(const wchar_t *x, const wchar_t *y) { + if (!x && !y) { + return 0; + } else if (!x) { + return -1; + } else if (!y) { + return 1; + } + bool coreX = 0 == _compare(x, -1, L"PythonCore", -1); bool coreY = 0 == _compare(y, -1, L"PythonCore", -1); if (coreX) { @@ -1090,6 +1089,14 @@ _compareCompany(const wchar_t *x, const wchar_t *y) int _compareTag(const wchar_t *x, const wchar_t *y) { + if (!x && !y) { + return 0; + } else if (!x) { + return -1; + } else if (!y) { + return 1; + } + // Compare up to the first dash. If not equal, that's our sort order const wchar_t *xDash = wcschr(x, L'-'); const wchar_t *yDash = wcschr(y, L'-'); @@ -1162,7 +1169,6 @@ addEnvironmentInfo(EnvironmentInfo **root, EnvironmentInfo *node) node->prev = r->prev; } else { debug(L"# not adding %s/%s/%i to tree\n", node->company, node->tag, node->internalSortKey); - freeEnvironmentInfo(node); return RC_DUPLICATE_ITEM; } return 0; @@ -1311,10 +1317,11 @@ _registrySearchTags(const SearchInfo *search, EnvironmentInfo **result, HKEY roo exitCode = 0; } else if (!exitCode) { exitCode = addEnvironmentInfo(result, env); - if (exitCode == RC_DUPLICATE_ITEM) { - exitCode = 0; - } else if (exitCode) { + if (exitCode) { freeEnvironmentInfo(env); + if (exitCode == RC_DUPLICATE_ITEM) { + exitCode = 0; + } } } } @@ -1398,10 +1405,11 @@ appxSearch(const SearchInfo *search, EnvironmentInfo **result, const wchar_t *pa } int exitCode = addEnvironmentInfo(result, env); - if (exitCode == RC_DUPLICATE_ITEM) { - exitCode = 0; - } else if (exitCode) { + if (exitCode) { freeEnvironmentInfo(env); + if (exitCode == RC_DUPLICATE_ITEM) { + exitCode = 0; + } } @@ -1409,6 +1417,94 @@ appxSearch(const SearchInfo *search, EnvironmentInfo **result, const wchar_t *pa } +/******************************************************************************\ + *** OVERRIDDEN EXECUTABLE PATH *** +\******************************************************************************/ + + +int +explicitOverrideSearch(const SearchInfo *search, EnvironmentInfo **result) +{ + if (!search->executablePath) { + return 0; + } + + EnvironmentInfo *env = newEnvironmentInfo(NULL, NULL); + if (!env) { + return RC_NO_MEMORY; + } + env->internalSortKey = 10; + int exitCode = copyWstr(&env->executablePath, search->executablePath); + if (exitCode) { + goto abort; + } + exitCode = copyWstr(&env->displayName, L"Explicit override"); + if (exitCode) { + goto abort; + } + exitCode = addEnvironmentInfo(result, env); + if (exitCode) { + goto abort; + } + return 0; + +abort: + freeEnvironmentInfo(env); + if (exitCode == RC_DUPLICATE_ITEM) { + exitCode = 0; + } + return exitCode; +} + + +/******************************************************************************\ + *** ACTIVE VIRTUAL ENVIRONMENT SEARCH *** +\******************************************************************************/ + +int +virtualenvSearch(const SearchInfo *search, EnvironmentInfo **result) +{ + int exitCode = 0; + EnvironmentInfo *env = NULL; + wchar_t buffer[MAXLEN]; + int n = GetEnvironmentVariableW(L"VIRTUAL_ENV", buffer, MAXLEN); + if (!n || !join(buffer, MAXLEN, L"Scripts") || !join(buffer, MAXLEN, search->executable)) { + return 0; + } + + if (INVALID_FILE_ATTRIBUTES == GetFileAttributesW(buffer)) { + debug(L"Python executable %s missing from virtual env\n", buffer); + return 0; + } + + env = newEnvironmentInfo(NULL, NULL); + if (!env) { + return RC_NO_MEMORY; + } + env->isActiveVenv = true; + env->internalSortKey = 20; + exitCode = copyWstr(&env->displayName, L"Active venv"); + if (exitCode) { + goto abort; + } + exitCode = copyWstr(&env->executablePath, buffer); + if (exitCode) { + goto abort; + } + exitCode = addEnvironmentInfo(result, env); + if (exitCode) { + goto abort; + } + return 0; + +abort: + freeEnvironmentInfo(env); + if (exitCode == RC_DUPLICATE_ITEM) { + return 0; + } + return exitCode; +} + /******************************************************************************\ *** COLLECT ENVIRONMENTS *** \******************************************************************************/ @@ -1492,18 +1588,28 @@ int collectEnvironments(const SearchInfo *search, EnvironmentInfo **result) { int exitCode = 0; + HKEY root; EnvironmentInfo *env = NULL; + if (!result) { return RC_INTERNAL_ERROR; } *result = NULL; - if (search->executablePath) { - env = newEnvironmentInfo(NULL, NULL); - return copyWstr(&env->executablePath, search->executablePath); + exitCode = explicitOverrideSearch(search, result); + if (exitCode) { + return exitCode; } - HKEY root; + exitCode = virtualenvSearch(search, result); + if (exitCode) { + return exitCode; + } + + // If we aren't collecting all items to list them, we can exit now. + if (env && !(search->list || search->listPaths)) { + return 0; + } for (struct RegistrySearchInfo *info = REGISTRY_SEARCH; info->subkey; ++info) { if (ERROR_SUCCESS == RegOpenKeyExW(info->hive, info->subkey, 0, info->flags, &root)) { @@ -1799,11 +1905,12 @@ _printEnvironment(const EnvironmentInfo *env, FILE *out, bool showPath, const wc int -_listAllEnvironments(EnvironmentInfo *env, FILE * out, bool showPath, bool *isDefault) +_listAllEnvironments(EnvironmentInfo *env, FILE * out, bool showPath, EnvironmentInfo *defaultEnv) { wchar_t buffer[256]; + const int bufferSize = 256; while (env) { - int exitCode = _listAllEnvironments(env->prev, out, showPath, isDefault); + int exitCode = _listAllEnvironments(env->prev, out, showPath, defaultEnv); if (exitCode) { return exitCode; } @@ -1811,15 +1918,16 @@ _listAllEnvironments(EnvironmentInfo *env, FILE * out, bool showPath, bool *isDe if (!env->company || !env->tag) { buffer[0] = L'\0'; } else if (0 == _compare(env->company, -1, L"PythonCore", -1)) { - swprintf_s(buffer, sizeof(buffer) / sizeof(buffer[0]), L"-V:%s", env->tag); + swprintf_s(buffer, bufferSize, L"-V:%s", env->tag); } else { - swprintf_s(buffer, sizeof(buffer) / sizeof(buffer[0]), L"-V:%s/%s", env->company, env->tag); + swprintf_s(buffer, bufferSize, L"-V:%s/%s", env->company, env->tag); } + + if (env == defaultEnv) { + wcscat_s(buffer, bufferSize, L" *"); + } + if (buffer[0]) { - if (*isDefault) { - wcscat_s(buffer, sizeof(buffer) / sizeof(buffer[0]), L" *"); - *isDefault = false; - } exitCode = _printEnvironment(env, out, showPath, buffer); if (exitCode) { return exitCode; @@ -1833,10 +1941,8 @@ _listAllEnvironments(EnvironmentInfo *env, FILE * out, bool showPath, bool *isDe int -listEnvironments(EnvironmentInfo *env, FILE * out, bool showPath) +listEnvironments(EnvironmentInfo *env, FILE * out, bool showPath, EnvironmentInfo *defaultEnv) { - bool isDefault = true; - if (!env) { fwprintf_s(stdout, L"No installed Pythons found!\n"); return 0; @@ -1878,7 +1984,7 @@ listEnvironments(EnvironmentInfo *env, FILE * out, bool showPath) */ int mode = _setmode(_fileno(out), _O_U8TEXT); - int exitCode = _listAllEnvironments(env, out, showPath, &isDefault); + int exitCode = _listAllEnvironments(env, out, showPath, defaultEnv); fflush(out); if (mode >= 0) { _setmode(_fileno(out), mode); @@ -2147,6 +2253,7 @@ int process(int argc, wchar_t ** argv) { int exitCode = 0; + int searchExitCode = 0; SearchInfo search = {0}; EnvironmentInfo *envs = NULL; EnvironmentInfo *env = NULL; @@ -2175,60 +2282,62 @@ process(int argc, wchar_t ** argv) } } + // Select best environment + // This is early so that we can show the default when listing, but all + // responses to any errors occur later. + searchExitCode = selectEnvironment(&search, envs, &env); + // List all environments, then exit if (search.list || search.listPaths) { - exitCode = listEnvironments(envs, stdout, search.listPaths); + exitCode = listEnvironments(envs, stdout, search.listPaths, env); goto abort; } // When debugging, list all discovered environments anyway if (log_fp) { - exitCode = listEnvironments(envs, log_fp, true); + exitCode = listEnvironments(envs, log_fp, true, NULL); if (exitCode) { goto abort; } } - // Select best environment - env = NULL; - if (search.executablePath == NULL) { - exitCode = selectEnvironment(&search, envs, &env); - // If none found, and if permitted, install it - if (exitCode == RC_NO_PYTHON && isEnvVarSet(L"PYLAUNCHER_ALLOW_INSTALL") || - isEnvVarSet(L"PYLAUNCHER_ALWAYS_INSTALL")) { - exitCode = installEnvironment(&search); - if (!exitCode) { - // Successful install, so we need to re-scan and select again - exitCode = performSearch(&search, &envs); - if (exitCode) { - goto abort; - } - env = NULL; - exitCode = selectEnvironment(&search, envs, &env); - } - } - if (exitCode == RC_NO_PYTHON) { - fputws(L"No suitable Python runtime found\n", stderr); - fputws(L"Pass --list (-0) to see all detected environments on your machine\n", stderr); - if (!isEnvVarSet(L"PYLAUNCHER_ALLOW_INSTALL") && search.oldStyleTag) { - fputws(L"or set environment variable PYLAUNCHER_ALLOW_INSTALL to use winget\n" - L"or open the Microsoft Store to the requested version.\n", stderr); + // We searched earlier, so if we didn't find anything, now we react + exitCode = searchExitCode; + // If none found, and if permitted, install it + if (exitCode == RC_NO_PYTHON && isEnvVarSet(L"PYLAUNCHER_ALLOW_INSTALL") || + isEnvVarSet(L"PYLAUNCHER_ALWAYS_INSTALL")) { + exitCode = installEnvironment(&search); + if (!exitCode) { + // Successful install, so we need to re-scan and select again + env = NULL; + exitCode = performSearch(&search, &envs); + if (exitCode) { + goto abort; } - goto abort; + exitCode = selectEnvironment(&search, envs, &env); } - if (exitCode == RC_NO_PYTHON_AT_ALL) { - fputws(L"No installed Python found!\n", stderr); - goto abort; - } - if (exitCode) { - goto abort; + } + if (exitCode == RC_NO_PYTHON) { + fputws(L"No suitable Python runtime found\n", stderr); + fputws(L"Pass --list (-0) to see all detected environments on your machine\n", stderr); + if (!isEnvVarSet(L"PYLAUNCHER_ALLOW_INSTALL") && search.oldStyleTag) { + fputws(L"or set environment variable PYLAUNCHER_ALLOW_INSTALL to use winget\n" + L"or open the Microsoft Store to the requested version.\n", stderr); } + goto abort; + } + if (exitCode == RC_NO_PYTHON_AT_ALL) { + fputws(L"No installed Python found!\n", stderr); + goto abort; + } + if (exitCode) { + goto abort; + } - if (env) { - debug(L"env.company: %s\nenv.tag: %s\n", env->company, env->tag); - } else { - debug(L"env.company: (null)\nenv.tag: (null)\n"); - } + if (env) { + debug(L"env.company: %s\nenv.tag: %s\n", env->company, env->tag); + } else { + debug(L"env.company: (null)\nenv.tag: (null)\n"); } exitCode = calculateCommandLine(&search, env, launchCommand, sizeof(launchCommand) / sizeof(launchCommand[0])); From webhook-mailer at python.org Wed Apr 6 19:26:28 2022 From: webhook-mailer at python.org (vstinner) Date: Wed, 06 Apr 2022 23:26:28 -0000 Subject: [Python-checkins] bpo-35134: Add Include/cpython/setobject.h header (GH-32384) Message-ID: https://github.com/python/cpython/commit/5c4d1f6e0e192653560ae2941a6677fbf4fbd1f2 commit: 5c4d1f6e0e192653560ae2941a6677fbf4fbd1f2 branch: main author: Victor Stinner committer: vstinner date: 2022-04-07T01:26:24+02:00 summary: bpo-35134: Add Include/cpython/setobject.h header (GH-32384) files: A Include/cpython/setobject.h M Include/setobject.h M Makefile.pre.in M PCbuild/pythoncore.vcxproj M PCbuild/pythoncore.vcxproj.filters diff --git a/Include/cpython/setobject.h b/Include/cpython/setobject.h new file mode 100644 index 0000000000000..b4443a678b7e7 --- /dev/null +++ b/Include/cpython/setobject.h @@ -0,0 +1,67 @@ +#ifndef Py_CPYTHON_SETOBJECT_H +# error "this header file must not be included directly" +#endif + +/* There are three kinds of entries in the table: + +1. Unused: key == NULL and hash == 0 +2. Dummy: key == dummy and hash == -1 +3. Active: key != NULL and key != dummy and hash != -1 + +The hash field of Unused slots is always zero. + +The hash field of Dummy slots are set to -1 +meaning that dummy entries can be detected by +either entry->key==dummy or by entry->hash==-1. +*/ + +#define PySet_MINSIZE 8 + +typedef struct { + PyObject *key; + Py_hash_t hash; /* Cached hash code of the key */ +} setentry; + +/* The SetObject data structure is shared by set and frozenset objects. + +Invariant for sets: + - hash is -1 + +Invariants for frozensets: + - data is immutable. + - hash is the hash of the frozenset or -1 if not computed yet. + +*/ + +typedef struct { + PyObject_HEAD + + Py_ssize_t fill; /* Number active and dummy entries*/ + Py_ssize_t used; /* Number active entries */ + + /* The table contains mask + 1 slots, and that's a power of 2. + * We store the mask instead of the size because the mask is more + * frequently needed. + */ + Py_ssize_t mask; + + /* The table points to a fixed-size smalltable for small tables + * or to additional malloc'ed memory for bigger tables. + * The table pointer is never NULL which saves us from repeated + * runtime null-tests. + */ + setentry *table; + Py_hash_t hash; /* Only used by frozenset objects */ + Py_ssize_t finger; /* Search finger for pop() */ + + setentry smalltable[PySet_MINSIZE]; + PyObject *weakreflist; /* List of weak references */ +} PySetObject; + +#define PySet_GET_SIZE(so) \ + (assert(PyAnySet_Check(so)), (((PySetObject *)(so))->used)) + +PyAPI_DATA(PyObject *) _PySet_Dummy; + +PyAPI_FUNC(int) _PySet_NextEntry(PyObject *set, Py_ssize_t *pos, PyObject **key, Py_hash_t *hash); +PyAPI_FUNC(int) _PySet_Update(PyObject *set, PyObject *iterable); diff --git a/Include/setobject.h b/Include/setobject.h index 62516be5ab29b..fdad70644c545 100644 --- a/Include/setobject.h +++ b/Include/setobject.h @@ -6,73 +6,6 @@ extern "C" { #endif -#ifndef Py_LIMITED_API - -/* There are three kinds of entries in the table: - -1. Unused: key == NULL and hash == 0 -2. Dummy: key == dummy and hash == -1 -3. Active: key != NULL and key != dummy and hash != -1 - -The hash field of Unused slots is always zero. - -The hash field of Dummy slots are set to -1 -meaning that dummy entries can be detected by -either entry->key==dummy or by entry->hash==-1. -*/ - -#define PySet_MINSIZE 8 - -typedef struct { - PyObject *key; - Py_hash_t hash; /* Cached hash code of the key */ -} setentry; - -/* The SetObject data structure is shared by set and frozenset objects. - -Invariant for sets: - - hash is -1 - -Invariants for frozensets: - - data is immutable. - - hash is the hash of the frozenset or -1 if not computed yet. - -*/ - -typedef struct { - PyObject_HEAD - - Py_ssize_t fill; /* Number active and dummy entries*/ - Py_ssize_t used; /* Number active entries */ - - /* The table contains mask + 1 slots, and that's a power of 2. - * We store the mask instead of the size because the mask is more - * frequently needed. - */ - Py_ssize_t mask; - - /* The table points to a fixed-size smalltable for small tables - * or to additional malloc'ed memory for bigger tables. - * The table pointer is never NULL which saves us from repeated - * runtime null-tests. - */ - setentry *table; - Py_hash_t hash; /* Only used by frozenset objects */ - Py_ssize_t finger; /* Search finger for pop() */ - - setentry smalltable[PySet_MINSIZE]; - PyObject *weakreflist; /* List of weak references */ -} PySetObject; - -#define PySet_GET_SIZE(so) (assert(PyAnySet_Check(so)),(((PySetObject *)(so))->used)) - -PyAPI_DATA(PyObject *) _PySet_Dummy; - -PyAPI_FUNC(int) _PySet_NextEntry(PyObject *set, Py_ssize_t *pos, PyObject **key, Py_hash_t *hash); -PyAPI_FUNC(int) _PySet_Update(PyObject *set, PyObject *iterable); - -#endif /* Section excluded by Py_LIMITED_API */ - PyAPI_DATA(PyTypeObject) PySet_Type; PyAPI_DATA(PyTypeObject) PyFrozenSet_Type; PyAPI_DATA(PyTypeObject) PySetIter_Type; @@ -104,6 +37,12 @@ PyAPI_FUNC(Py_ssize_t) PySet_Size(PyObject *anyset); (Py_IS_TYPE(ob, &PySet_Type) || \ PyType_IsSubtype(Py_TYPE(ob), &PySet_Type)) +#ifndef Py_LIMITED_API +# define Py_CPYTHON_SETOBJECT_H +# include "cpython/setobject.h" +# undef Py_CPYTHON_SETOBJECT_H +#endif + #ifdef __cplusplus } #endif diff --git a/Makefile.pre.in b/Makefile.pre.in index 69e97234ae737..2f61259e69b8c 100644 --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -1551,6 +1551,7 @@ PYTHON_HEADERS= \ $(srcdir)/Include/cpython/pystate.h \ $(srcdir)/Include/cpython/pythonrun.h \ $(srcdir)/Include/cpython/pytime.h \ + $(srcdir)/Include/cpython/setobject.h \ $(srcdir)/Include/cpython/sysmodule.h \ $(srcdir)/Include/cpython/traceback.h \ $(srcdir)/Include/cpython/tupleobject.h \ diff --git a/PCbuild/pythoncore.vcxproj b/PCbuild/pythoncore.vcxproj index 1d7c08b2c8a71..6e51e17194d4d 100644 --- a/PCbuild/pythoncore.vcxproj +++ b/PCbuild/pythoncore.vcxproj @@ -172,6 +172,7 @@ + diff --git a/PCbuild/pythoncore.vcxproj.filters b/PCbuild/pythoncore.vcxproj.filters index 571a7f13bc1a0..5fe6a9de21c13 100644 --- a/PCbuild/pythoncore.vcxproj.filters +++ b/PCbuild/pythoncore.vcxproj.filters @@ -459,6 +459,9 @@ Include\cpython + + Include\cpython + Include\cpython From webhook-mailer at python.org Wed Apr 6 20:29:56 2022 From: webhook-mailer at python.org (vstinner) Date: Thu, 07 Apr 2022 00:29:56 -0000 Subject: [Python-checkins] bpo-35134: Remove the Include/code.h header file (GH-32385) Message-ID: https://github.com/python/cpython/commit/85addfb9c6496eb3d26082348cf5aca848c877ef commit: 85addfb9c6496eb3d26082348cf5aca848c877ef branch: main author: Victor Stinner committer: vstinner date: 2022-04-07T02:29:52+02:00 summary: bpo-35134: Remove the Include/code.h header file (GH-32385) Remove the Include/code.h header file. C extensions should only include the main header file. Python.h includes directly Include/cpython/code.h instead. files: A Misc/NEWS.d/next/C API/2022-04-07-00-53-51.bpo-35134.zSjIzk.rst D Include/code.h M Doc/whatsnew/3.11.rst M Include/Python.h M Include/cpython/code.h M Makefile.pre.in M Objects/codeobject.c M PCbuild/pythoncore.vcxproj M PCbuild/pythoncore.vcxproj.filters M Python/ceval.c M Python/marshal.c M Python/pythonrun.c M Python/sysmodule.c M Python/traceback.c diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index 4c9b32d9a9452..31cdf10f66656 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -1210,7 +1210,7 @@ Porting to Python 3.11 explicitly include the header files after ``#include ``. (Contributed by Victor Stinner in :issue:`45434`.) -* The non-limited API files ``cellobject.h``, ``classobject.h``, ``context.h``, +* The non-limited API files ``cellobject.h``, ``classobject.h``, ``code.h``, ``context.h``, ``funcobject.h``, ``genobject.h`` and ``longintrepr.h`` have been moved to the ``Include/cpython`` directory. Moreover, the ``eval.h`` header file was removed. These files must not be included directly, as they are already diff --git a/Include/Python.h b/Include/Python.h index 4dc2edb4dbbe2..52a7aac6ba6cb 100644 --- a/Include/Python.h +++ b/Include/Python.h @@ -68,7 +68,7 @@ #include "cpython/classobject.h" #include "fileobject.h" #include "pycapsule.h" -#include "code.h" +#include "cpython/code.h" #include "pyframe.h" #include "traceback.h" #include "sliceobject.h" diff --git a/Include/code.h b/Include/code.h deleted file mode 100644 index 0245c32bc2544..0000000000000 --- a/Include/code.h +++ /dev/null @@ -1,18 +0,0 @@ -/* Definitions for bytecode */ - -#ifndef Py_CODE_H -#define Py_CODE_H -#ifdef __cplusplus -extern "C" { -#endif - -#ifndef Py_LIMITED_API -# define Py_CPYTHON_CODE_H -# include "cpython/code.h" -# undef Py_CPYTHON_CODE_H -#endif - -#ifdef __cplusplus -} -#endif -#endif /* !Py_CODE_H */ diff --git a/Include/cpython/code.h b/Include/cpython/code.h index 157678317931e..6dc2290ffeb5e 100644 --- a/Include/cpython/code.h +++ b/Include/cpython/code.h @@ -1,5 +1,10 @@ -#ifndef Py_CPYTHON_CODE_H -# error "this header file must not be included directly" +/* Definitions for bytecode */ + +#ifndef Py_LIMITED_API +#ifndef Py_CODE_H +#define Py_CODE_H +#ifdef __cplusplus +extern "C" { #endif /* Each instruction in a code object is a fixed-width value, @@ -204,3 +209,9 @@ PyAPI_FUNC(int) _PyCode_GetExtra(PyObject *code, Py_ssize_t index, void **extra); PyAPI_FUNC(int) _PyCode_SetExtra(PyObject *code, Py_ssize_t index, void *extra); + +#ifdef __cplusplus +} +#endif +#endif // !Py_CODE_H +#endif // !Py_LIMITED_API diff --git a/Makefile.pre.in b/Makefile.pre.in index 2f61259e69b8c..cb6e962045a04 100644 --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -1450,7 +1450,6 @@ PYTHON_HEADERS= \ $(srcdir)/Include/bytearrayobject.h \ $(srcdir)/Include/bytesobject.h \ $(srcdir)/Include/ceval.h \ - $(srcdir)/Include/code.h \ $(srcdir)/Include/codecs.h \ $(srcdir)/Include/compile.h \ $(srcdir)/Include/complexobject.h \ diff --git a/Misc/NEWS.d/next/C API/2022-04-07-00-53-51.bpo-35134.zSjIzk.rst b/Misc/NEWS.d/next/C API/2022-04-07-00-53-51.bpo-35134.zSjIzk.rst new file mode 100644 index 0000000000000..93e6e32160462 --- /dev/null +++ b/Misc/NEWS.d/next/C API/2022-04-07-00-53-51.bpo-35134.zSjIzk.rst @@ -0,0 +1,2 @@ +Remove the ``Include/code.h`` header file. C extensions should only include +the main ```` header file. Patch by Victor Stinner. diff --git a/Objects/codeobject.c b/Objects/codeobject.c index 7d50b40ec4536..e872b398e08c8 100644 --- a/Objects/codeobject.c +++ b/Objects/codeobject.c @@ -1,7 +1,6 @@ #include #include "Python.h" -#include "code.h" #include "opcode.h" #include "structmember.h" // PyMemberDef #include "pycore_code.h" // _PyCodeConstructor diff --git a/PCbuild/pythoncore.vcxproj b/PCbuild/pythoncore.vcxproj index 6e51e17194d4d..cbd3936860120 100644 --- a/PCbuild/pythoncore.vcxproj +++ b/PCbuild/pythoncore.vcxproj @@ -129,7 +129,6 @@ - diff --git a/PCbuild/pythoncore.vcxproj.filters b/PCbuild/pythoncore.vcxproj.filters index 5fe6a9de21c13..70af305222170 100644 --- a/PCbuild/pythoncore.vcxproj.filters +++ b/PCbuild/pythoncore.vcxproj.filters @@ -132,9 +132,6 @@ Include - - Include - Include diff --git a/Python/ceval.c b/Python/ceval.c index 487e09bc66417..5384aac5d6e6c 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -23,7 +23,6 @@ #include "pycore_tuple.h" // _PyTuple_ITEMS() #include "pycore_emscripten_signal.h" // _Py_CHECK_EMSCRIPTEN_SIGNALS -#include "code.h" #include "pycore_dict.h" #include "dictobject.h" #include "frameobject.h" diff --git a/Python/marshal.c b/Python/marshal.c index e7cf6553bd142..19abcc8ffe4b7 100644 --- a/Python/marshal.c +++ b/Python/marshal.c @@ -12,7 +12,6 @@ #include "pycore_call.h" // _PyObject_CallNoArgs() #include "pycore_code.h" // _PyCode_New() #include "pycore_hashtable.h" // _Py_hashtable_t -#include "code.h" #include "marshal.h" // Py_MARSHAL_VERSION /*[clinic input] diff --git a/Python/pythonrun.c b/Python/pythonrun.c index 38ca952838a1f..d117b790dfdaa 100644 --- a/Python/pythonrun.c +++ b/Python/pythonrun.c @@ -25,7 +25,6 @@ #include "token.h" // INDENT #include "errcode.h" // E_EOF -#include "code.h" // PyCodeObject #include "marshal.h" // PyMarshal_ReadLongFromFile() #ifdef MS_WINDOWS diff --git a/Python/sysmodule.c b/Python/sysmodule.c index de4e10a7e110c..ac44b803b23f5 100644 --- a/Python/sysmodule.c +++ b/Python/sysmodule.c @@ -31,7 +31,6 @@ Data members: #include "pycore_structseq.h" // _PyStructSequence_InitType() #include "pycore_tuple.h" // _PyTuple_FromArray() -#include "code.h" #include "frameobject.h" // PyFrame_GetBack() #include "pydtrace.h" #include "osdefs.h" // DELIM diff --git a/Python/traceback.c b/Python/traceback.c index 0d0eb954c232f..488c1b17cf51f 100644 --- a/Python/traceback.c +++ b/Python/traceback.c @@ -3,7 +3,6 @@ #include "Python.h" -#include "code.h" // PyCode_Addr2Line etc #include "frameobject.h" // PyFrame_GetBack() #include "pycore_ast.h" // asdl_seq_* #include "pycore_call.h" // _PyObject_CallMethodFormat() From webhook-mailer at python.org Wed Apr 6 20:50:09 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Thu, 07 Apr 2022 00:50:09 -0000 Subject: [Python-checkins] stdtypes docs: fix typo (GH-32349) (GH-32370) Message-ID: https://github.com/python/cpython/commit/55abb0ef25d9ecabc6fe93cf0d38f19268b5859c commit: 55abb0ef25d9ecabc6fe93cf0d38f19268b5859c branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: JelleZijlstra date: 2022-04-06T17:50:04-07:00 summary: stdtypes docs: fix typo (GH-32349) (GH-32370) (cherry picked from commit b33c4564aceeae8323bcb19167fbbd2d5f5002bc) Co-authored-by: Ian <40774387+isteptoe at users.noreply.github.com> files: M Doc/library/stdtypes.rst diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst index 28082b42d7923..f60e936089c23 100644 --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -3580,7 +3580,7 @@ The conversion types are: | | be used for Python2/3 code bases. | | +------------+-----------------------------------------------------+-------+ | ``'a'`` | Bytes (converts any Python object using | \(5) | -| | ``repr(obj).encode('ascii','backslashreplace)``). | | +| | ``repr(obj).encode('ascii', 'backslashreplace')``). | | +------------+-----------------------------------------------------+-------+ | ``'r'`` | ``'r'`` is an alias for ``'a'`` and should only | \(7) | | | be used for Python2/3 code bases. | | From webhook-mailer at python.org Wed Apr 6 20:51:10 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Thu, 07 Apr 2022 00:51:10 -0000 Subject: [Python-checkins] pkgutil docs: Link sys constants, add backticks (GH-32356) Message-ID: https://github.com/python/cpython/commit/63bd72448a5af01206c2a9aec5f1ed1e903f1e12 commit: 63bd72448a5af01206c2a9aec5f1ed1e903f1e12 branch: main author: Boris Verkhovskiy committer: JelleZijlstra date: 2022-04-06T17:51:05-07:00 summary: pkgutil docs: Link sys constants, add backticks (GH-32356) Co-authored-by: ?ric files: M Doc/library/pkgutil.rst diff --git a/Doc/library/pkgutil.rst b/Doc/library/pkgutil.rst index 3b17b9a621987..788a02dcb8922 100644 --- a/Doc/library/pkgutil.rst +++ b/Doc/library/pkgutil.rst @@ -26,7 +26,7 @@ support. __path__ = extend_path(__path__, __name__) This will add to the package's ``__path__`` all subdirectories of directories - on ``sys.path`` named after the package. This is useful if one wants to + on :data:`sys.path` named after the package. This is useful if one wants to distribute different parts of a single logical package as multiple directories. @@ -128,9 +128,9 @@ support. Yield :term:`finder` objects for the given module name. - If fullname contains a '.', the finders will be for the package + If fullname contains a ``'.'``, the finders will be for the package containing fullname, otherwise they will be all registered top level - finders (i.e. those on both sys.meta_path and sys.path_hooks). + finders (i.e. those on both :data:`sys.meta_path` and :data:`sys.path_hooks`). If the named module is in a package, that package is imported as a side effect of invoking this function. @@ -145,7 +145,7 @@ support. .. function:: iter_modules(path=None, prefix='') Yields :class:`ModuleInfo` for all submodules on *path*, or, if - *path* is ``None``, all top-level modules on ``sys.path``. + *path* is ``None``, all top-level modules on :data:`sys.path`. *path* should be either ``None`` or a list of paths to look for modules in. From webhook-mailer at python.org Wed Apr 6 20:51:39 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Thu, 07 Apr 2022 00:51:39 -0000 Subject: [Python-checkins] ssl docs: Fix typo (GH-32314) Message-ID: https://github.com/python/cpython/commit/1da9c38fd352465fd3d1a00e64dc90444b421730 commit: 1da9c38fd352465fd3d1a00e64dc90444b421730 branch: main author: Frederick committer: JelleZijlstra date: 2022-04-06T17:51:35-07:00 summary: ssl docs: Fix typo (GH-32314) files: M Doc/library/ssl.rst diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst index 8e5554ee4b842..ce9b71f49c64d 100644 --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -1397,7 +1397,7 @@ SSL sockets also have the following additional methods and attributes: .. method:: SSLSocket.version() Return the actual SSL protocol version negotiated by the connection - as a string, or ``None`` is no secure connection is established. + as a string, or ``None`` if no secure connection is established. As of this writing, possible return values include ``"SSLv2"``, ``"SSLv3"``, ``"TLSv1"``, ``"TLSv1.1"`` and ``"TLSv1.2"``. Recent OpenSSL versions may define more return values. From webhook-mailer at python.org Wed Apr 6 21:12:43 2022 From: webhook-mailer at python.org (miss-islington) Date: Thu, 07 Apr 2022 01:12:43 -0000 Subject: [Python-checkins] pkgutil docs: Link sys constants, add backticks (GH-32356) Message-ID: https://github.com/python/cpython/commit/08bd308dd2500616f28415113c4bce52a8065138 commit: 08bd308dd2500616f28415113c4bce52a8065138 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-06T18:12:34-07:00 summary: pkgutil docs: Link sys constants, add backticks (GH-32356) Co-authored-by: ?ric (cherry picked from commit 63bd72448a5af01206c2a9aec5f1ed1e903f1e12) Co-authored-by: Boris Verkhovskiy files: M Doc/library/pkgutil.rst diff --git a/Doc/library/pkgutil.rst b/Doc/library/pkgutil.rst index 3b17b9a621987..788a02dcb8922 100644 --- a/Doc/library/pkgutil.rst +++ b/Doc/library/pkgutil.rst @@ -26,7 +26,7 @@ support. __path__ = extend_path(__path__, __name__) This will add to the package's ``__path__`` all subdirectories of directories - on ``sys.path`` named after the package. This is useful if one wants to + on :data:`sys.path` named after the package. This is useful if one wants to distribute different parts of a single logical package as multiple directories. @@ -128,9 +128,9 @@ support. Yield :term:`finder` objects for the given module name. - If fullname contains a '.', the finders will be for the package + If fullname contains a ``'.'``, the finders will be for the package containing fullname, otherwise they will be all registered top level - finders (i.e. those on both sys.meta_path and sys.path_hooks). + finders (i.e. those on both :data:`sys.meta_path` and :data:`sys.path_hooks`). If the named module is in a package, that package is imported as a side effect of invoking this function. @@ -145,7 +145,7 @@ support. .. function:: iter_modules(path=None, prefix='') Yields :class:`ModuleInfo` for all submodules on *path*, or, if - *path* is ``None``, all top-level modules on ``sys.path``. + *path* is ``None``, all top-level modules on :data:`sys.path`. *path* should be either ``None`` or a list of paths to look for modules in. From webhook-mailer at python.org Wed Apr 6 21:13:54 2022 From: webhook-mailer at python.org (miss-islington) Date: Thu, 07 Apr 2022 01:13:54 -0000 Subject: [Python-checkins] ssl docs: Fix typo (GH-32314) Message-ID: https://github.com/python/cpython/commit/b217ba7371a780ad014e4ed5695ab8d0a2ea588e commit: b217ba7371a780ad014e4ed5695ab8d0a2ea588e branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-06T18:13:46-07:00 summary: ssl docs: Fix typo (GH-32314) (cherry picked from commit 1da9c38fd352465fd3d1a00e64dc90444b421730) Co-authored-by: Frederick files: M Doc/library/ssl.rst diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst index 3531cc3413afa..84abbaf935374 100644 --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -1397,7 +1397,7 @@ SSL sockets also have the following additional methods and attributes: .. method:: SSLSocket.version() Return the actual SSL protocol version negotiated by the connection - as a string, or ``None`` is no secure connection is established. + as a string, or ``None`` if no secure connection is established. As of this writing, possible return values include ``"SSLv2"``, ``"SSLv3"``, ``"TLSv1"``, ``"TLSv1.1"`` and ``"TLSv1.2"``. Recent OpenSSL versions may define more return values. From webhook-mailer at python.org Wed Apr 6 21:17:10 2022 From: webhook-mailer at python.org (miss-islington) Date: Thu, 07 Apr 2022 01:17:10 -0000 Subject: [Python-checkins] ssl docs: Fix typo (GH-32314) Message-ID: https://github.com/python/cpython/commit/30dfc46be43c2e6311699a18718cc97d784ed1a8 commit: 30dfc46be43c2e6311699a18718cc97d784ed1a8 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-06T18:16:53-07:00 summary: ssl docs: Fix typo (GH-32314) (cherry picked from commit 1da9c38fd352465fd3d1a00e64dc90444b421730) Co-authored-by: Frederick files: M Doc/library/ssl.rst diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst index c3acf61d2b527..0256e04a28c9e 100644 --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -1357,7 +1357,7 @@ SSL sockets also have the following additional methods and attributes: .. method:: SSLSocket.version() Return the actual SSL protocol version negotiated by the connection - as a string, or ``None`` is no secure connection is established. + as a string, or ``None`` if no secure connection is established. As of this writing, possible return values include ``"SSLv2"``, ``"SSLv3"``, ``"TLSv1"``, ``"TLSv1.1"`` and ``"TLSv1.2"``. Recent OpenSSL versions may define more return values. From webhook-mailer at python.org Wed Apr 6 21:19:04 2022 From: webhook-mailer at python.org (miss-islington) Date: Thu, 07 Apr 2022 01:19:04 -0000 Subject: [Python-checkins] pkgutil docs: Link sys constants, add backticks (GH-32356) Message-ID: https://github.com/python/cpython/commit/ea6dc5ec08d6efe8b04e4375ad0f14f926ba2188 commit: ea6dc5ec08d6efe8b04e4375ad0f14f926ba2188 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-06T18:18:56-07:00 summary: pkgutil docs: Link sys constants, add backticks (GH-32356) Co-authored-by: ?ric (cherry picked from commit 63bd72448a5af01206c2a9aec5f1ed1e903f1e12) Co-authored-by: Boris Verkhovskiy files: M Doc/library/pkgutil.rst diff --git a/Doc/library/pkgutil.rst b/Doc/library/pkgutil.rst index 3b17b9a621987..788a02dcb8922 100644 --- a/Doc/library/pkgutil.rst +++ b/Doc/library/pkgutil.rst @@ -26,7 +26,7 @@ support. __path__ = extend_path(__path__, __name__) This will add to the package's ``__path__`` all subdirectories of directories - on ``sys.path`` named after the package. This is useful if one wants to + on :data:`sys.path` named after the package. This is useful if one wants to distribute different parts of a single logical package as multiple directories. @@ -128,9 +128,9 @@ support. Yield :term:`finder` objects for the given module name. - If fullname contains a '.', the finders will be for the package + If fullname contains a ``'.'``, the finders will be for the package containing fullname, otherwise they will be all registered top level - finders (i.e. those on both sys.meta_path and sys.path_hooks). + finders (i.e. those on both :data:`sys.meta_path` and :data:`sys.path_hooks`). If the named module is in a package, that package is imported as a side effect of invoking this function. @@ -145,7 +145,7 @@ support. .. function:: iter_modules(path=None, prefix='') Yields :class:`ModuleInfo` for all submodules on *path*, or, if - *path* is ``None``, all top-level modules on ``sys.path``. + *path* is ``None``, all top-level modules on :data:`sys.path`. *path* should be either ``None`` or a list of paths to look for modules in. From webhook-mailer at python.org Wed Apr 6 21:31:49 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Thu, 07 Apr 2022 01:31:49 -0000 Subject: [Python-checkins] doc: Link to `string.capwords` from `str.title` (GH-20913) Message-ID: https://github.com/python/cpython/commit/b786d9ec52a2c2b0b6627be7fd4a3948c61fbdea commit: b786d9ec52a2c2b0b6627be7fd4a3948c61fbdea branch: main author: Eric Wieser committer: JelleZijlstra date: 2022-04-06T18:31:39-07:00 summary: doc: Link to `string.capwords` from `str.title` (GH-20913) Since `title()` mentions its own short-comings, it should also mention the library function which does not possess them. Co-authored-by: Jelle Zijlstra files: M Doc/library/stdtypes.rst diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst index d6d90cd462c22..f25e8303f94a2 100644 --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -2189,7 +2189,11 @@ expression support in the :mod:`re` module). >>> "they're bill's friends from the UK".title() "They'Re Bill'S Friends From The Uk" - A workaround for apostrophes can be constructed using regular expressions:: + The :func:`string.capwords` function does not have this problem, as it + splits words on spaces only. + + Alternatively, a workaround for apostrophes can be constructed using regular + expressions:: >>> import re >>> def titlecase(s): From webhook-mailer at python.org Wed Apr 6 21:38:44 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Thu, 07 Apr 2022 01:38:44 -0000 Subject: [Python-checkins] pickle docs: Fix typos and improve wording (GH-24776) Message-ID: https://github.com/python/cpython/commit/1d0f08fa46b54f5a9b43a916b66d50b97d56cf36 commit: 1d0f08fa46b54f5a9b43a916b66d50b97d56cf36 branch: main author: G?ry Ogam committer: JelleZijlstra date: 2022-04-06T18:38:35-07:00 summary: pickle docs: Fix typos and improve wording (GH-24776) Co-authored-by: Jelle Zijlstra files: M Doc/library/pickle.rst diff --git a/Doc/library/pickle.rst b/Doc/library/pickle.rst index a8ad5d437aaab..fa14b64027e8f 100644 --- a/Doc/library/pickle.rst +++ b/Doc/library/pickle.rst @@ -147,7 +147,7 @@ to read the pickle produced. earlier versions of Python. * Protocol version 2 was introduced in Python 2.3. It provides much more - efficient pickling of :term:`new-style class`\es. Refer to :pep:`307` for + efficient pickling of :term:`new-style classes `. Refer to :pep:`307` for information about improvements brought by protocol 2. * Protocol version 3 was added in Python 3.0. It has explicit support for @@ -261,7 +261,7 @@ process more convenient: protocol argument is needed. Bytes past the pickled representation of the object are ignored. - Arguments *file*, *fix_imports*, *encoding*, *errors*, *strict* and *buffers* + Arguments *fix_imports*, *encoding*, *errors*, *strict* and *buffers* have the same meaning as in the :class:`Unpickler` constructor. .. versionchanged:: 3.8 @@ -368,7 +368,7 @@ The :mod:`pickle` module exports three classes, :class:`Pickler`, .. versionadded:: 3.3 - .. method:: reducer_override(self, obj) + .. method:: reducer_override(obj) Special reducer that can be defined in :class:`Pickler` subclasses. This method has priority over any reducer in the :attr:`dispatch_table`. It @@ -494,20 +494,18 @@ What can be pickled and unpickled? The following types can be pickled: -* ``None``, ``True``, and ``False`` - -* integers, floating point numbers, complex numbers +* ``None``, ``True``, and ``False``; -* strings, bytes, bytearrays +* integers, floating-point numbers, complex numbers; -* tuples, lists, sets, and dictionaries containing only picklable objects +* strings, bytes, bytearrays; -* functions defined at the top level of a module (using :keyword:`def`, not - :keyword:`lambda`) +* tuples, lists, sets, and dictionaries containing only picklable objects; -* built-in functions defined at the top level of a module +* functions (built-in and user-defined) defined at the top level of a module + (using :keyword:`def`, not :keyword:`lambda`); -* classes that are defined at the top level of a module +* classes defined at the top level of a module; * instances of such classes whose the result of calling :meth:`__getstate__` is picklable (see section :ref:`pickle-inst` for details). @@ -519,14 +517,14 @@ structure may exceed the maximum recursion depth, a :exc:`RecursionError` will b raised in this case. You can carefully raise this limit with :func:`sys.setrecursionlimit`. -Note that functions (built-in and user-defined) are pickled by "fully qualified" -name reference, not by value. [#]_ This means that only the function name is +Note that functions (built-in and user-defined) are pickled by fully qualified +name, not by value. [#]_ This means that only the function name is pickled, along with the name of the module the function is defined in. Neither the function's code, nor any of its function attributes are pickled. Thus the defining module must be importable in the unpickling environment, and the module must contain the named object, otherwise an exception will be raised. [#]_ -Similarly, classes are pickled by named reference, so the same restrictions in +Similarly, classes are pickled by fully qualified name, so the same restrictions in the unpickling environment apply. Note that none of the class's code or data is pickled, so in the following example the class attribute ``attr`` is not restored in the unpickling environment:: @@ -536,7 +534,7 @@ restored in the unpickling environment:: picklestring = pickle.dumps(Foo) -These restrictions are why picklable functions and classes must be defined in +These restrictions are why picklable functions and classes must be defined at the top level of a module. Similarly, when class instances are pickled, their class's code and data are not @@ -568,7 +566,7 @@ implementation of this behaviour:: def save(obj): return (obj.__class__, obj.__dict__) - def load(cls, attributes): + def restore(cls, attributes): obj = cls.__new__(cls) obj.__dict__.update(attributes) return obj @@ -807,14 +805,15 @@ the code :: f = io.BytesIO() p = MyPickler(f) -does the same, but all instances of ``MyPickler`` will by default -share the same dispatch table. The equivalent code using the -:mod:`copyreg` module is :: +does the same but all instances of ``MyPickler`` will by default +share the private dispatch table. On the other hand, the code :: copyreg.pickle(SomeClass, reduce_SomeClass) f = io.BytesIO() p = pickle.Pickler(f) +modifies the global dispatch table shared by all users of the :mod:`copyreg` module. + .. _pickle-state: Handling Stateful Objects @@ -1117,7 +1116,7 @@ Here is an example of an unpickler allowing only few safe classes from the """Helper function analogous to pickle.loads().""" return RestrictedUnpickler(io.BytesIO(s)).load() -A sample usage of our unpickler working has intended:: +A sample usage of our unpickler working as intended:: >>> restricted_loads(pickle.dumps([1, 2, range(15)])) [1, 2, range(0, 15)] @@ -1161,7 +1160,7 @@ For the simplest code, use the :func:`dump` and :func:`load` functions. :: # An arbitrary collection of objects supported by pickle. data = { - 'a': [1, 2.0, 3, 4+6j], + 'a': [1, 2.0, 3+4j], 'b': ("character string", b"byte string"), 'c': {None, True, False} } @@ -1217,6 +1216,6 @@ The following example reads the resulting pickled data. :: operations. .. [#] The limitation on alphanumeric characters is due to the fact - the persistent IDs, in protocol 0, are delimited by the newline + that persistent IDs in protocol 0 are delimited by the newline character. Therefore if any kind of newline characters occurs in - persistent IDs, the resulting pickle will become unreadable. + persistent IDs, the resulting pickled data will become unreadable. From webhook-mailer at python.org Wed Apr 6 21:53:09 2022 From: webhook-mailer at python.org (miss-islington) Date: Thu, 07 Apr 2022 01:53:09 -0000 Subject: [Python-checkins] doc: Link to `string.capwords` from `str.title` (GH-20913) Message-ID: https://github.com/python/cpython/commit/ccac6312b9f9d8209646c85492920962fb5704ba commit: ccac6312b9f9d8209646c85492920962fb5704ba branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-06T18:52:51-07:00 summary: doc: Link to `string.capwords` from `str.title` (GH-20913) Since `title()` mentions its own short-comings, it should also mention the library function which does not possess them. Co-authored-by: Jelle Zijlstra (cherry picked from commit b786d9ec52a2c2b0b6627be7fd4a3948c61fbdea) Co-authored-by: Eric Wieser files: M Doc/library/stdtypes.rst diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst index f60e936089c23..77b2590860e32 100644 --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -2151,7 +2151,11 @@ expression support in the :mod:`re` module). >>> "they're bill's friends from the UK".title() "They'Re Bill'S Friends From The Uk" - A workaround for apostrophes can be constructed using regular expressions:: + The :func:`string.capwords` function does not have this problem, as it + splits words on spaces only. + + Alternatively, a workaround for apostrophes can be constructed using regular + expressions:: >>> import re >>> def titlecase(s): From webhook-mailer at python.org Wed Apr 6 21:57:43 2022 From: webhook-mailer at python.org (miss-islington) Date: Thu, 07 Apr 2022 01:57:43 -0000 Subject: [Python-checkins] doc: Link to `string.capwords` from `str.title` (GH-20913) Message-ID: https://github.com/python/cpython/commit/85f0792b8fdd86b71844a3ae90ccd7b8d7890012 commit: 85f0792b8fdd86b71844a3ae90ccd7b8d7890012 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-06T18:57:38-07:00 summary: doc: Link to `string.capwords` from `str.title` (GH-20913) Since `title()` mentions its own short-comings, it should also mention the library function which does not possess them. Co-authored-by: Jelle Zijlstra (cherry picked from commit b786d9ec52a2c2b0b6627be7fd4a3948c61fbdea) Co-authored-by: Eric Wieser files: M Doc/library/stdtypes.rst diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst index d96fb1f2c9ca6..cf1a638344c8e 100644 --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -2118,7 +2118,11 @@ expression support in the :mod:`re` module). >>> "they're bill's friends from the UK".title() "They'Re Bill'S Friends From The Uk" - A workaround for apostrophes can be constructed using regular expressions:: + The :func:`string.capwords` function does not have this problem, as it + splits words on spaces only. + + Alternatively, a workaround for apostrophes can be constructed using regular + expressions:: >>> import re >>> def titlecase(s): From webhook-mailer at python.org Wed Apr 6 23:00:55 2022 From: webhook-mailer at python.org (rhettinger) Date: Thu, 07 Apr 2022 03:00:55 -0000 Subject: [Python-checkins] Remove micro-optimization that no longer shows a benefit. (GH-32397) Message-ID: https://github.com/python/cpython/commit/5aee46b31ba37d65cdf4d5a96cabb8835c508deb commit: 5aee46b31ba37d65cdf4d5a96cabb8835c508deb branch: main author: Raymond Hettinger committer: rhettinger date: 2022-04-06T22:00:47-05:00 summary: Remove micro-optimization that no longer shows a benefit. (GH-32397) files: M Modules/_collectionsmodule.c diff --git a/Modules/_collectionsmodule.c b/Modules/_collectionsmodule.c index f78e2613fc243..18c762b21754b 100644 --- a/Modules/_collectionsmodule.c +++ b/Modules/_collectionsmodule.c @@ -1536,12 +1536,6 @@ deque_sizeof(dequeobject *deque, void *unused) PyDoc_STRVAR(sizeof_doc, "D.__sizeof__() -- size of D in memory, in bytes"); -static int -deque_bool(dequeobject *deque) -{ - return Py_SIZE(deque) != 0; -} - static PyObject * deque_get_maxlen(dequeobject *deque, void *Py_UNUSED(ignored)) { @@ -1572,20 +1566,6 @@ static PySequenceMethods deque_as_sequence = { (ssizeargfunc)deque_inplace_repeat, /* sq_inplace_repeat */ }; -static PyNumberMethods deque_as_number = { - 0, /* nb_add */ - 0, /* nb_subtract */ - 0, /* nb_multiply */ - 0, /* nb_remainder */ - 0, /* nb_divmod */ - 0, /* nb_power */ - 0, /* nb_negative */ - 0, /* nb_positive */ - 0, /* nb_absolute */ - (inquiry)deque_bool, /* nb_bool */ - 0, /* nb_invert */ - }; - static PyObject *deque_iter(dequeobject *deque); static PyObject *deque_reviter(dequeobject *deque, PyObject *Py_UNUSED(ignored)); PyDoc_STRVAR(reversed_doc, @@ -1650,7 +1630,7 @@ static PyTypeObject deque_type = { 0, /* tp_setattr */ 0, /* tp_as_async */ deque_repr, /* tp_repr */ - &deque_as_number, /* tp_as_number */ + 0, /* tp_as_number */ &deque_as_sequence, /* tp_as_sequence */ 0, /* tp_as_mapping */ PyObject_HashNotImplemented, /* tp_hash */ From webhook-mailer at python.org Thu Apr 7 03:23:19 2022 From: webhook-mailer at python.org (tiran) Date: Thu, 07 Apr 2022 07:23:19 -0000 Subject: [Python-checkins] bpo-40280: Detect missing threading on WASM platforms (GH-32352) Message-ID: https://github.com/python/cpython/commit/2b16a08bc77475917dd5c96417aef4c5210b45ac commit: 2b16a08bc77475917dd5c96417aef4c5210b45ac branch: main author: Christian Heimes committer: tiran date: 2022-04-07T09:22:47+02:00 summary: bpo-40280: Detect missing threading on WASM platforms (GH-32352) Co-authored-by: Brett Cannon files: A Misc/NEWS.d/next/Tests/2022-04-06-10-16-27.bpo-40280.KT5Apg.rst M Lib/distutils/tests/test_build_ext.py M Lib/test/libregrtest/main.py M Lib/test/pickletester.py M Lib/test/support/threading_helper.py M Lib/test/test_bz2.py M Lib/test/test_capi.py M Lib/test/test_context.py M Lib/test/test_decimal.py M Lib/test/test_email/test_email.py M Lib/test/test_enum.py M Lib/test/test_functools.py M Lib/test/test_gc.py M Lib/test/test_hashlib.py M Lib/test/test_import/__init__.py M Lib/test/test_importlib/test_locks.py M Lib/test/test_importlib/test_threaded_import.py M Lib/test/test_io.py M Lib/test/test_itertools.py M Lib/test/test_logging.py M Lib/test/test_queue.py M Lib/test/test_sched.py M Lib/test/test_signal.py M Lib/test/test_support.py M Lib/test/test_sys.py M Lib/test/test_thread.py M Lib/test/test_threadedtempfile.py M Lib/test/test_threading.py M Lib/test/test_threading_local.py M Lib/test/test_threadsignals.py M Lib/test/test_weakref.py M configure M configure.ac diff --git a/Lib/distutils/tests/test_build_ext.py b/Lib/distutils/tests/test_build_ext.py index 460b62f50bc12..031897bd2734f 100644 --- a/Lib/distutils/tests/test_build_ext.py +++ b/Lib/distutils/tests/test_build_ext.py @@ -17,6 +17,7 @@ from test import support from test.support import os_helper from test.support.script_helper import assert_python_ok +from test.support import threading_helper # http://bugs.python.org/issue4373 # Don't load the xx module more than once. @@ -165,6 +166,7 @@ def test_user_site(self): self.assertIn(lib, cmd.rpath) self.assertIn(incl, cmd.include_dirs) + @threading_helper.requires_working_threading() def test_optional_extension(self): # this extension will fail, but let's ignore this failure diff --git a/Lib/test/libregrtest/main.py b/Lib/test/libregrtest/main.py index af58114ed925c..e7e3dde0b0a66 100644 --- a/Lib/test/libregrtest/main.py +++ b/Lib/test/libregrtest/main.py @@ -20,6 +20,7 @@ from test.libregrtest.utils import removepy, count, format_duration, printlist from test import support from test.support import os_helper +from test.support import threading_helper # bpo-38203: Maximum delay in seconds to exit Python (call Py_Finalize()). @@ -676,7 +677,8 @@ def main(self, tests=None, **kwargs): except SystemExit as exc: # bpo-38203: Python can hang at exit in Py_Finalize(), especially # on threading._shutdown() call: put a timeout - faulthandler.dump_traceback_later(EXIT_TIMEOUT, exit=True) + if threading_helper.can_start_thread: + faulthandler.dump_traceback_later(EXIT_TIMEOUT, exit=True) sys.exit(exc.code) diff --git a/Lib/test/pickletester.py b/Lib/test/pickletester.py index 63fa7604fbfc9..d0ea7d0e55e7d 100644 --- a/Lib/test/pickletester.py +++ b/Lib/test/pickletester.py @@ -1380,6 +1380,7 @@ def test_truncated_data(self): self.check_unpickling_error(self.truncated_errors, p) @threading_helper.reap_threads + @threading_helper.requires_working_threading() def test_unpickle_module_race(self): # https://bugs.python.org/issue34572 locker_module = dedent(""" diff --git a/Lib/test/support/threading_helper.py b/Lib/test/support/threading_helper.py index 2ae4577f41f7b..7b636f0ccf097 100644 --- a/Lib/test/support/threading_helper.py +++ b/Lib/test/support/threading_helper.py @@ -4,6 +4,7 @@ import sys import threading import time +import unittest from test import support @@ -210,7 +211,7 @@ def __exit__(self, *exc_info): def _can_start_thread() -> bool: - """Detect if Python can start new threads. + """Detect whether Python can start new threads. Some WebAssembly platforms do not provide a working pthread implementation. Thread support is stubbed and any attempt @@ -234,3 +235,15 @@ def _can_start_thread() -> bool: return True can_start_thread = _can_start_thread() + +def requires_working_threading(*, module=False): + """Skip tests or modules that require working threading. + + Can be used as a function/class decorator or to skip an entire module. + """ + msg = "requires threading support" + if module: + if not can_start_thread: + raise unittest.SkipTest(msg) + else: + return unittest.skipUnless(can_start_thread, msg) diff --git a/Lib/test/test_bz2.py b/Lib/test/test_bz2.py index 9965c1fe2e5f1..c97ed1cea0d11 100644 --- a/Lib/test/test_bz2.py +++ b/Lib/test/test_bz2.py @@ -496,6 +496,7 @@ def testContextProtocol(self): else: self.fail("1/0 didn't raise an exception") + @threading_helper.requires_working_threading() def testThreading(self): # Issue #7205: Using a BZ2File from several threads shouldn't deadlock. data = b"1" * 2**20 diff --git a/Lib/test/test_capi.py b/Lib/test/test_capi.py index 714a2d98e9fb5..3837f801b3c73 100644 --- a/Lib/test/test_capi.py +++ b/Lib/test/test_capi.py @@ -710,6 +710,7 @@ def pendingcalls_wait(self, l, n, context = None): if False and support.verbose: print("(%i)"%(len(l),)) + @threading_helper.requires_working_threading() def test_pendingcalls_threaded(self): #do every callback on a separate thread @@ -840,6 +841,7 @@ def test_module_state_shared_in_global(self): class TestThreadState(unittest.TestCase): @threading_helper.reap_threads + @threading_helper.requires_working_threading() def test_thread_state(self): # some extra thread-state tests driven via _testcapi def target(): diff --git a/Lib/test/test_context.py b/Lib/test/test_context.py index 2d8b63a1f5958..3132cea668cb1 100644 --- a/Lib/test/test_context.py +++ b/Lib/test/test_context.py @@ -6,6 +6,7 @@ import time import unittest import weakref +from test.support import threading_helper try: from _testcapi import hamt @@ -341,6 +342,7 @@ def ctx2_fun(): ctx1.run(ctx1_fun) @isolated_context + @threading_helper.requires_working_threading() def test_context_threads_1(self): cvar = contextvars.ContextVar('cvar') diff --git a/Lib/test/test_decimal.py b/Lib/test/test_decimal.py index b68cfbef23f16..0e7491ecdd45c 100644 --- a/Lib/test/test_decimal.py +++ b/Lib/test/test_decimal.py @@ -39,6 +39,7 @@ run_with_locale, cpython_only, darwin_malloc_err_warning) from test.support.import_helper import import_fresh_module +from test.support import threading_helper from test.support import warnings_helper import random import inspect @@ -1591,6 +1592,8 @@ def thfunc2(cls): for sig in Overflow, Underflow, DivisionByZero, InvalidOperation: cls.assertFalse(thiscontext.flags[sig]) + + at threading_helper.requires_working_threading() class ThreadingTest(unittest.TestCase): '''Unit tests for thread local contexts in Decimal.''' diff --git a/Lib/test/test_email/test_email.py b/Lib/test/test_email/test_email.py index a3ccbbbabfb32..ca9c773bbc559 100644 --- a/Lib/test/test_email/test_email.py +++ b/Lib/test/test_email/test_email.py @@ -3285,6 +3285,7 @@ def test_getaddresses_header_obj(self): addrs = utils.getaddresses([Header('Al Person ')]) self.assertEqual(addrs[0][1], 'aperson at dom.ain') + @threading_helper.requires_working_threading() def test_make_msgid_collisions(self): # Test make_msgid uniqueness, even with multiple threads class MsgidsThread(Thread): diff --git a/Lib/test/test_enum.py b/Lib/test/test_enum.py index f2572b2ac351a..b1b8e82b3859f 100644 --- a/Lib/test/test_enum.py +++ b/Lib/test/test_enum.py @@ -2949,6 +2949,7 @@ class Color(StrMixin, AllMixin, Flag): self.assertEqual(str(Color.BLUE), 'blue') @threading_helper.reap_threads + @threading_helper.requires_working_threading() def test_unique_composite(self): # override __eq__ to be identity only class TestFlag(Flag): @@ -3481,6 +3482,7 @@ class Color(StrMixin, AllMixin, IntFlag): self.assertEqual(str(Color.BLUE), 'blue') @threading_helper.reap_threads + @threading_helper.requires_working_threading() def test_unique_composite(self): # override __eq__ to be identity only class TestFlag(IntFlag): diff --git a/Lib/test/test_functools.py b/Lib/test/test_functools.py index 82e73f46a3fba..e3c065615778f 100644 --- a/Lib/test/test_functools.py +++ b/Lib/test/test_functools.py @@ -1637,6 +1637,7 @@ def f(zomg: 'zomg_annotation'): for attr in self.module.WRAPPER_ASSIGNMENTS: self.assertEqual(getattr(g, attr), getattr(f, attr)) + @threading_helper.requires_working_threading() def test_lru_cache_threaded(self): n, m = 5, 11 def orig(x, y): @@ -1685,6 +1686,7 @@ def clear(): finally: sys.setswitchinterval(orig_si) + @threading_helper.requires_working_threading() def test_lru_cache_threaded2(self): # Simultaneous call with the same arguments n, m = 5, 7 @@ -1712,6 +1714,7 @@ def test(): pause.reset() self.assertEqual(f.cache_info(), (0, (i+1)*n, m*n, i+1)) + @threading_helper.requires_working_threading() def test_lru_cache_threaded3(self): @self.module.lru_cache(maxsize=2) def f(x): @@ -2914,6 +2917,7 @@ def test_cached_attribute_name_differs_from_func_name(self): self.assertEqual(item.get_cost(), 4) self.assertEqual(item.cached_cost, 3) + @threading_helper.requires_working_threading() def test_threaded(self): go = threading.Event() item = CachedCostItemWait(go) diff --git a/Lib/test/test_gc.py b/Lib/test/test_gc.py index c4d4355dec9c6..ce04042679bbc 100644 --- a/Lib/test/test_gc.py +++ b/Lib/test/test_gc.py @@ -365,6 +365,7 @@ def __del__(self): v = {1: v, 2: Ouch()} gc.disable() + @threading_helper.requires_working_threading() def test_trashcan_threads(self): # Issue #13992: trashcan mechanism should be thread-safe NESTING = 60 diff --git a/Lib/test/test_hashlib.py b/Lib/test/test_hashlib.py index d2a92147d5f02..67becdd6d317f 100644 --- a/Lib/test/test_hashlib.py +++ b/Lib/test/test_hashlib.py @@ -915,6 +915,7 @@ def test_gil(self): ) @threading_helper.reap_threads + @threading_helper.requires_working_threading() def test_threaded_hashing(self): # Updating the same hash object from several threads at once # using data chunk sizes containing the same byte sequences. diff --git a/Lib/test/test_import/__init__.py b/Lib/test/test_import/__init__.py index 7cca9f9d60ab6..be88677dc697e 100644 --- a/Lib/test/test_import/__init__.py +++ b/Lib/test/test_import/__init__.py @@ -448,6 +448,7 @@ def test_issue31492(self): with self.assertRaises(AttributeError): os.does_not_exist + @threading_helper.requires_working_threading() def test_concurrency(self): # bpo 38091: this is a hack to slow down the code that calls # has_deadlock(); the logic was itself sometimes deadlocking. diff --git a/Lib/test/test_importlib/test_locks.py b/Lib/test/test_importlib/test_locks.py index 584d013caacad..56d73c496e6bb 100644 --- a/Lib/test/test_importlib/test_locks.py +++ b/Lib/test/test_importlib/test_locks.py @@ -12,6 +12,9 @@ from test import lock_tests +threading_helper.requires_working_threading(module=True) + + class ModuleLockAsRLockTests: locktype = classmethod(lambda cls: cls.LockType("some_lock")) @@ -146,4 +149,4 @@ def setUpModule(): if __name__ == '__main__': - unittets.main() + unittest.main() diff --git a/Lib/test/test_importlib/test_threaded_import.py b/Lib/test/test_importlib/test_threaded_import.py index 76b028eac97bf..cc1d804f35f91 100644 --- a/Lib/test/test_importlib/test_threaded_import.py +++ b/Lib/test/test_importlib/test_threaded_import.py @@ -19,6 +19,8 @@ from test.support.os_helper import (TESTFN, unlink, rmtree) from test.support import script_helper, threading_helper +threading_helper.requires_working_threading(module=True) + def task(N, done, done_tasks, errors): try: # We don't use modulefinder but still import it in order to stress diff --git a/Lib/test/test_io.py b/Lib/test/test_io.py index 67be108d2526f..29fe287550b2d 100644 --- a/Lib/test/test_io.py +++ b/Lib/test/test_io.py @@ -1451,6 +1451,7 @@ def test_read_all(self): self.assertEqual(b"abcdefg", bufio.read()) @support.requires_resource('cpu') + @threading_helper.requires_working_threading() def test_threads(self): try: # Write out many bytes with exactly the same number of 0's, @@ -1825,6 +1826,7 @@ def test_truncate_after_write(self): self.assertEqual(f.tell(), buffer_size + 2) @support.requires_resource('cpu') + @threading_helper.requires_working_threading() def test_threads(self): try: # Write out many bytes from many threads and test they were @@ -1895,6 +1897,7 @@ def bad_write(b): self.assertRaises(OSError, b.close) # exception not swallowed self.assertTrue(b.closed) + @threading_helper.requires_working_threading() def test_slow_close_from_thread(self): # Issue #31976 rawio = self.SlowFlushRawIO() @@ -3287,6 +3290,7 @@ def test_errors_property(self): self.assertEqual(f.errors, "replace") @support.no_tracing + @threading_helper.requires_working_threading() def test_threads_write(self): # Issue6750: concurrent writes could duplicate data event = threading.Event() @@ -4362,9 +4366,11 @@ def run(): else: self.assertFalse(err.strip('.!')) + @threading_helper.requires_working_threading() def test_daemon_threads_shutdown_stdout_deadlock(self): self.check_daemon_threads_shutdown_deadlock('stdout') + @threading_helper.requires_working_threading() def test_daemon_threads_shutdown_stderr_deadlock(self): self.check_daemon_threads_shutdown_deadlock('stderr') diff --git a/Lib/test/test_itertools.py b/Lib/test/test_itertools.py index 3f3f7cb35d0bc..238afbbd883d3 100644 --- a/Lib/test/test_itertools.py +++ b/Lib/test/test_itertools.py @@ -1,6 +1,7 @@ import doctest import unittest from test import support +from test.support import threading_helper from itertools import * import weakref from decimal import Decimal @@ -1533,6 +1534,7 @@ def __next__(self): with self.assertRaisesRegex(RuntimeError, "tee"): next(a) + @threading_helper.requires_working_threading() def test_tee_concurrent(self): start = threading.Event() finish = threading.Event() diff --git a/Lib/test/test_logging.py b/Lib/test/test_logging.py index f6f5977df2a1e..7555f05d1d0cf 100644 --- a/Lib/test/test_logging.py +++ b/Lib/test/test_logging.py @@ -630,6 +630,7 @@ def test_path_objects(self): @unittest.skipIf( support.is_emscripten, "Emscripten cannot fstat unlinked files." ) + @threading_helper.requires_working_threading() def test_race(self): # Issue #14632 refers. def remove_loop(fname, tries): @@ -679,6 +680,7 @@ def remove_loop(fname, tries): # This helps ensure that when fork exists (the important concept) that the # register_at_fork mechanism is also present and used. @support.requires_fork() + @threading_helper.requires_working_threading() def test_post_fork_child_no_deadlock(self): """Ensure child logging locks are not held; bpo-6721 & bpo-36533.""" class _OurHandler(logging.Handler): @@ -1063,6 +1065,7 @@ class TestUnixDatagramServer(TestUDPServer): # - end of server_helper section @support.requires_working_socket() + at threading_helper.requires_working_threading() class SMTPHandlerTest(BaseTest): # bpo-14314, bpo-19665, bpo-34092: don't wait forever TIMEOUT = support.LONG_TIMEOUT @@ -1172,6 +1175,7 @@ def test_flush_on_close(self): # assert that no new lines have been added self.assert_log_lines(lines) # no change + @threading_helper.requires_working_threading() def test_race_between_set_target_and_flush(self): class MockRaceConditionHandler: def __init__(self, mem_hdlr): @@ -1687,6 +1691,7 @@ def test_defaults_do_no_interpolation(self): @support.requires_working_socket() + at threading_helper.requires_working_threading() class SocketHandlerTest(BaseTest): """Test for SocketHandler objects.""" @@ -1802,6 +1807,7 @@ def tearDown(self): os_helper.unlink(self.address) @support.requires_working_socket() + at threading_helper.requires_working_threading() class DatagramHandlerTest(BaseTest): """Test for DatagramHandler.""" @@ -1884,6 +1890,7 @@ def tearDown(self): os_helper.unlink(self.address) @support.requires_working_socket() + at threading_helper.requires_working_threading() class SysLogHandlerTest(BaseTest): """Test for SysLogHandler using UDP.""" @@ -1994,6 +2001,7 @@ def tearDown(self): super(IPv6SysLogHandlerTest, self).tearDown() @support.requires_working_socket() + at threading_helper.requires_working_threading() class HTTPHandlerTest(BaseTest): """Test for HTTPHandler.""" @@ -3575,6 +3583,7 @@ def test_logrecord_class(self): ]) + at threading_helper.requires_working_threading() class QueueHandlerTest(BaseTest): # Do not bother with a logger name group. expected_log_pat = r"^[\w.]+ -> (\w+): (\d+)$" @@ -3684,6 +3693,7 @@ def test_queue_listener_with_multiple_handlers(self): import multiprocessing from unittest.mock import patch + @threading_helper.requires_working_threading() class QueueListenerTest(BaseTest): """ Tests based on patch submitted for issue #27930. Ensure that diff --git a/Lib/test/test_queue.py b/Lib/test/test_queue.py index cfa6003a867da..e3080376a9de5 100644 --- a/Lib/test/test_queue.py +++ b/Lib/test/test_queue.py @@ -87,6 +87,7 @@ def do_exceptional_blocking_test(self,block_func, block_args, trigger_func, self.fail("trigger thread ended but event never set") + at threading_helper.requires_working_threading() class BaseQueueTestMixin(BlockingTestMixin): def setUp(self): self.cum = 0 @@ -289,6 +290,8 @@ class CPriorityQueueTest(PriorityQueueTest, unittest.TestCase): # A Queue subclass that can provoke failure at a moment's notice :) class FailingQueueException(Exception): pass + + at threading_helper.requires_working_threading() class FailingQueueTest(BlockingTestMixin): def setUp(self): @@ -464,6 +467,7 @@ def consume_timeout(self, q, results, sentinel): return results.append(val) + @threading_helper.requires_working_threading() def run_threads(self, n_threads, q, inputs, feed_func, consume_func): results = [] sentinel = None diff --git a/Lib/test/test_sched.py b/Lib/test/test_sched.py index 7ae7baae85e2c..32cc8105bc055 100644 --- a/Lib/test/test_sched.py +++ b/Lib/test/test_sched.py @@ -58,6 +58,7 @@ def test_enterabs(self): scheduler.run() self.assertEqual(l, [0.01, 0.02, 0.03, 0.04, 0.05]) + @threading_helper.requires_working_threading() def test_enter_concurrent(self): q = queue.Queue() fun = q.put @@ -111,6 +112,7 @@ def test_cancel(self): scheduler.run() self.assertEqual(l, [0.02, 0.03, 0.04]) + @threading_helper.requires_working_threading() def test_cancel_concurrent(self): q = queue.Queue() fun = q.put diff --git a/Lib/test/test_signal.py b/Lib/test/test_signal.py index 37b46065e532c..ea13c59ec711e 100644 --- a/Lib/test/test_signal.py +++ b/Lib/test/test_signal.py @@ -14,6 +14,7 @@ from test import support from test.support import os_helper from test.support.script_helper import assert_python_ok, spawn_python +from test.support import threading_helper try: import _testcapi except ImportError: @@ -876,6 +877,7 @@ def handler(signum, frame): @unittest.skipUnless(hasattr(signal, 'pthread_kill'), 'need signal.pthread_kill()') + @threading_helper.requires_working_threading() def test_pthread_kill(self): code = """if 1: import signal @@ -1012,6 +1014,7 @@ def test_sigtimedwait_negative_timeout(self): 'need signal.sigwait()') @unittest.skipUnless(hasattr(signal, 'pthread_sigmask'), 'need signal.pthread_sigmask()') + @threading_helper.requires_working_threading() def test_sigwait_thread(self): # Check that calling sigwait() from a thread doesn't suspend the whole # process. A new interpreter is spawned to avoid problems when mixing @@ -1067,6 +1070,7 @@ def test_pthread_sigmask_valid_signals(self): @unittest.skipUnless(hasattr(signal, 'pthread_sigmask'), 'need signal.pthread_sigmask()') + @threading_helper.requires_working_threading() def test_pthread_sigmask(self): code = """if 1: import signal @@ -1144,6 +1148,7 @@ def read_sigmask(): @unittest.skipUnless(hasattr(signal, 'pthread_kill'), 'need signal.pthread_kill()') + @threading_helper.requires_working_threading() def test_pthread_kill_main_thread(self): # Test that a signal can be sent to the main thread with pthread_kill() # before any other thread has been created (see issue #12392). @@ -1298,6 +1303,7 @@ def handler(signum, frame): @unittest.skipUnless(hasattr(signal, "SIGUSR1"), "test needs SIGUSR1") + @threading_helper.requires_working_threading() def test_stress_modifying_handlers(self): # bpo-43406: race condition between trip_signal() and signal.signal signum = signal.SIGUSR1 diff --git a/Lib/test/test_support.py b/Lib/test/test_support.py index 2cff377cf629e..90e971deb397f 100644 --- a/Lib/test/test_support.py +++ b/Lib/test/test_support.py @@ -123,15 +123,18 @@ def test_forget(self): os_helper.unlink(mod_filename) os_helper.rmtree('__pycache__') + @support.requires_working_socket() def test_HOST(self): s = socket.create_server((socket_helper.HOST, 0)) s.close() + @support.requires_working_socket() def test_find_unused_port(self): port = socket_helper.find_unused_port() s = socket.create_server((socket_helper.HOST, port)) s.close() + @support.requires_working_socket() def test_bind_port(self): s = socket.socket() socket_helper.bind_port(s) diff --git a/Lib/test/test_sys.py b/Lib/test/test_sys.py index 445ff893ff6fc..b70871ff551d9 100644 --- a/Lib/test/test_sys.py +++ b/Lib/test/test_sys.py @@ -401,6 +401,7 @@ def test_getframe(self): # sys._current_frames() is a CPython-only gimmick. @threading_helper.reap_threads + @threading_helper.requires_working_threading() def test_current_frames(self): import threading import traceback @@ -466,6 +467,7 @@ def g456(): t.join() @threading_helper.reap_threads + @threading_helper.requires_working_threading() def test_current_exceptions(self): import threading import traceback @@ -1176,11 +1178,12 @@ class X(Exception): for moduleName in 'builtins', '__main__', 'some_module': with self.subTest(moduleName=moduleName): A.B.X.__module__ = moduleName - with test.support.captured_stderr() as stderr, \ - test.support.swap_attr(sys, 'unraisablehook', - sys.__unraisablehook__): + with test.support.captured_stderr() as stderr, test.support.swap_attr( + sys, 'unraisablehook', sys.__unraisablehook__ + ): expected = self.write_unraisable_exc( - A.B.X(), "msg", "obj"); + A.B.X(), "msg", "obj" + ) report = stderr.getvalue() self.assertIn(A.B.X.__qualname__, report) if moduleName in ['builtins', '__main__']: diff --git a/Lib/test/test_thread.py b/Lib/test/test_thread.py index d55fb731b6df5..ed527e7164fd0 100644 --- a/Lib/test/test_thread.py +++ b/Lib/test/test_thread.py @@ -9,6 +9,8 @@ from test import lock_tests +threading_helper.requires_working_threading(module=True) + NUMTASKS = 10 NUMTRIPS = 3 POLL_SLEEP = 0.010 # seconds = 10 ms diff --git a/Lib/test/test_threadedtempfile.py b/Lib/test/test_threadedtempfile.py index fe63c9e91437b..420fc6ec8be3d 100644 --- a/Lib/test/test_threadedtempfile.py +++ b/Lib/test/test_threadedtempfile.py @@ -21,6 +21,7 @@ import threading from traceback import print_exc +threading_helper.requires_working_threading(module=True) NUM_THREADS = 20 FILES_PER_THREAD = 50 diff --git a/Lib/test/test_threading.py b/Lib/test/test_threading.py index 16c6934c6d432..f7dea136a87c0 100644 --- a/Lib/test/test_threading.py +++ b/Lib/test/test_threading.py @@ -25,6 +25,7 @@ from test import lock_tests from test import support +threading_helper.requires_working_threading(module=True) # Between fork() and exec(), only async-safe functions are allowed (issues # #12316 and #11870), and fork() from a worker thread is known to trigger diff --git a/Lib/test/test_threading_local.py b/Lib/test/test_threading_local.py index 1567c41000bc5..9888e631881fa 100644 --- a/Lib/test/test_threading_local.py +++ b/Lib/test/test_threading_local.py @@ -12,6 +12,9 @@ import _threading_local +threading_helper.requires_working_threading(module=True) + + class Weak(object): pass diff --git a/Lib/test/test_threadsignals.py b/Lib/test/test_threadsignals.py index bac82b8a44511..e0ac18c946398 100644 --- a/Lib/test/test_threadsignals.py +++ b/Lib/test/test_threadsignals.py @@ -36,6 +36,7 @@ def send_signals(): os.kill(process_pid, signal.SIGUSR2) signalled_all.release() + at threading_helper.requires_working_threading() class ThreadSignals(unittest.TestCase): def test_signals(self): diff --git a/Lib/test/test_weakref.py b/Lib/test/test_weakref.py index 46dac0228295d..702bb4495e749 100644 --- a/Lib/test/test_weakref.py +++ b/Lib/test/test_weakref.py @@ -14,6 +14,7 @@ from test import support from test.support import script_helper, ALWAYS_EQ from test.support import gc_collect +from test.support import threading_helper # Used in ReferencesTestCase.test_ref_created_during_del() . ref_from_del = None @@ -1851,6 +1852,7 @@ def test_make_weak_keyed_dict_repr(self): dict = weakref.WeakKeyDictionary() self.assertRegex(repr(dict), '') + @threading_helper.requires_working_threading() def test_threaded_weak_valued_setdefault(self): d = weakref.WeakValueDictionary() with collect_in_thread(): @@ -1859,6 +1861,7 @@ def test_threaded_weak_valued_setdefault(self): self.assertIsNot(x, None) # we never put None in there! del x + @threading_helper.requires_working_threading() def test_threaded_weak_valued_pop(self): d = weakref.WeakValueDictionary() with collect_in_thread(): @@ -1867,6 +1870,7 @@ def test_threaded_weak_valued_pop(self): x = d.pop(10, 10) self.assertIsNot(x, None) # we never put None in there! + @threading_helper.requires_working_threading() def test_threaded_weak_valued_consistency(self): # Issue #28427: old keys should not remove new values from # WeakValueDictionary when collecting from another thread. @@ -1940,21 +1944,25 @@ def pop_and_collect(lst): if exc: raise exc[0] + @threading_helper.requires_working_threading() def test_threaded_weak_key_dict_copy(self): # Issue #35615: Weakref keys or values getting GC'ed during dict # copying should not result in a crash. self.check_threaded_weak_dict_copy(weakref.WeakKeyDictionary, False) + @threading_helper.requires_working_threading() def test_threaded_weak_key_dict_deepcopy(self): # Issue #35615: Weakref keys or values getting GC'ed during dict # copying should not result in a crash. self.check_threaded_weak_dict_copy(weakref.WeakKeyDictionary, True) + @threading_helper.requires_working_threading() def test_threaded_weak_value_dict_copy(self): # Issue #35615: Weakref keys or values getting GC'ed during dict # copying should not result in a crash. self.check_threaded_weak_dict_copy(weakref.WeakValueDictionary, False) + @threading_helper.requires_working_threading() def test_threaded_weak_value_dict_deepcopy(self): # Issue #35615: Weakref keys or values getting GC'ed during dict # copying should not result in a crash. diff --git a/Misc/NEWS.d/next/Tests/2022-04-06-10-16-27.bpo-40280.KT5Apg.rst b/Misc/NEWS.d/next/Tests/2022-04-06-10-16-27.bpo-40280.KT5Apg.rst new file mode 100644 index 0000000000000..9fcb4c9a8b3b0 --- /dev/null +++ b/Misc/NEWS.d/next/Tests/2022-04-06-10-16-27.bpo-40280.KT5Apg.rst @@ -0,0 +1 @@ +Threading tests are now skipped on WASM targets without pthread support. diff --git a/configure b/configure index e10462cf4dbfc..69b12309de578 100755 --- a/configure +++ b/configure @@ -6311,15 +6311,7 @@ esac else - case $ac_sys_system in #( - Emscripten) : - enable_wasm_dynamic_linking=no ;; #( - WASI) : - enable_wasm_dynamic_linking=no ;; #( - *) : - enable_wasm_dynamic_linking=missing - ;; -esac + enable_wasm_dynamic_linking=missing fi diff --git a/configure.ac b/configure.ac index 4f256eeff5d7b..5860595b752c8 100644 --- a/configure.ac +++ b/configure.ac @@ -1122,11 +1122,7 @@ AC_ARG_ENABLE([wasm-dynamic-linking], [AC_MSG_ERROR([--enable-wasm-dynamic-linking only applies to Emscripten and WASI])] ) ], [ - AS_CASE([$ac_sys_system], - [Emscripten], [enable_wasm_dynamic_linking=no], - [WASI], [enable_wasm_dynamic_linking=no], - [enable_wasm_dynamic_linking=missing] - ) + enable_wasm_dynamic_linking=missing ]) AC_MSG_RESULT([$enable_wasm_dynamic_linking]) From webhook-mailer at python.org Thu Apr 7 10:03:14 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Thu, 07 Apr 2022 14:03:14 -0000 Subject: [Python-checkins] c-api docs: There are five fields, not four (GH-32379) Message-ID: https://github.com/python/cpython/commit/4c92427fb85e420404a9bd26347e32acc1bbd3b7 commit: 4c92427fb85e420404a9bd26347e32acc1bbd3b7 branch: main author: Jelle Zijlstra committer: JelleZijlstra date: 2022-04-07T07:02:54-07:00 summary: c-api docs: There are five fields, not four (GH-32379) files: M Doc/c-api/memory.rst diff --git a/Doc/c-api/memory.rst b/Doc/c-api/memory.rst index e81a246cb75cc..987dea40dd674 100644 --- a/Doc/c-api/memory.rst +++ b/Doc/c-api/memory.rst @@ -403,7 +403,7 @@ Customize Memory Allocators .. c:type:: PyMemAllocatorEx Structure used to describe a memory block allocator. The structure has - four fields: + the following fields: +----------------------------------------------------------+---------------------------------------+ | Field | Meaning | From webhook-mailer at python.org Thu Apr 7 10:28:09 2022 From: webhook-mailer at python.org (miss-islington) Date: Thu, 07 Apr 2022 14:28:09 -0000 Subject: [Python-checkins] c-api docs: There are five fields, not four (GH-32379) Message-ID: https://github.com/python/cpython/commit/b4abef229536c80138ec139f01671d38c0b5a4e5 commit: b4abef229536c80138ec139f01671d38c0b5a4e5 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-07T07:27:40-07:00 summary: c-api docs: There are five fields, not four (GH-32379) (cherry picked from commit 4c92427fb85e420404a9bd26347e32acc1bbd3b7) Co-authored-by: Jelle Zijlstra files: M Doc/c-api/memory.rst diff --git a/Doc/c-api/memory.rst b/Doc/c-api/memory.rst index ced7ca79ca590..f17d24bda589c 100644 --- a/Doc/c-api/memory.rst +++ b/Doc/c-api/memory.rst @@ -403,7 +403,7 @@ Customize Memory Allocators .. c:type:: PyMemAllocatorEx Structure used to describe a memory block allocator. The structure has - four fields: + the following fields: +----------------------------------------------------------+---------------------------------------+ | Field | Meaning | From webhook-mailer at python.org Thu Apr 7 10:32:29 2022 From: webhook-mailer at python.org (miss-islington) Date: Thu, 07 Apr 2022 14:32:29 -0000 Subject: [Python-checkins] c-api docs: There are five fields, not four (GH-32379) Message-ID: https://github.com/python/cpython/commit/b124a3913d847149daea071dfafac09f1cb1c472 commit: b124a3913d847149daea071dfafac09f1cb1c472 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-07T07:32:09-07:00 summary: c-api docs: There are five fields, not four (GH-32379) (cherry picked from commit 4c92427fb85e420404a9bd26347e32acc1bbd3b7) Co-authored-by: Jelle Zijlstra files: M Doc/c-api/memory.rst diff --git a/Doc/c-api/memory.rst b/Doc/c-api/memory.rst index 91b901d726e61..624bddcbb8bed 100644 --- a/Doc/c-api/memory.rst +++ b/Doc/c-api/memory.rst @@ -362,7 +362,7 @@ Customize Memory Allocators .. c:type:: PyMemAllocatorEx Structure used to describe a memory block allocator. The structure has - four fields: + the following fields: +----------------------------------------------------------+---------------------------------------+ | Field | Meaning | From webhook-mailer at python.org Thu Apr 7 12:29:33 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Thu, 07 Apr 2022 16:29:33 -0000 Subject: [Python-checkins] ssl docs: Fix typo (GH-32336) Message-ID: https://github.com/python/cpython/commit/9ee2d3a93914776d15ac5cc7c44bb3aaca3e0fe5 commit: 9ee2d3a93914776d15ac5cc7c44bb3aaca3e0fe5 branch: main author: Frederick committer: JelleZijlstra date: 2022-04-07T09:29:23-07:00 summary: ssl docs: Fix typo (GH-32336) files: M Doc/library/ssl.rst diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst index ce9b71f49c64d..04818f824d105 100644 --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -1548,7 +1548,7 @@ to speed up repeated connections from the same clients. string must be the path to a single file in PEM format containing the certificate as well as any number of CA certificates needed to establish the certificate's authenticity. The *keyfile* string, if present, must - point to a file containing the private key in. Otherwise the private + point to a file containing the private key. Otherwise the private key will be taken from *certfile* as well. See the discussion of :ref:`ssl-certificates` for more information on how the certificate is stored in the *certfile*. From webhook-mailer at python.org Thu Apr 7 12:51:51 2022 From: webhook-mailer at python.org (miss-islington) Date: Thu, 07 Apr 2022 16:51:51 -0000 Subject: [Python-checkins] ssl docs: Fix typo (GH-32336) Message-ID: https://github.com/python/cpython/commit/877fd622e81989e6e9cb5401ac4974c9fd5a128a commit: 877fd622e81989e6e9cb5401ac4974c9fd5a128a branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-07T09:51:40-07:00 summary: ssl docs: Fix typo (GH-32336) (cherry picked from commit 9ee2d3a93914776d15ac5cc7c44bb3aaca3e0fe5) Co-authored-by: Frederick files: M Doc/library/ssl.rst diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst index 84abbaf935374..d21d6122e0a80 100644 --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -1548,7 +1548,7 @@ to speed up repeated connections from the same clients. string must be the path to a single file in PEM format containing the certificate as well as any number of CA certificates needed to establish the certificate's authenticity. The *keyfile* string, if present, must - point to a file containing the private key in. Otherwise the private + point to a file containing the private key. Otherwise the private key will be taken from *certfile* as well. See the discussion of :ref:`ssl-certificates` for more information on how the certificate is stored in the *certfile*. From webhook-mailer at python.org Thu Apr 7 13:07:03 2022 From: webhook-mailer at python.org (miss-islington) Date: Thu, 07 Apr 2022 17:07:03 -0000 Subject: [Python-checkins] ssl docs: Fix typo (GH-32336) Message-ID: https://github.com/python/cpython/commit/8b358d414735ab2e9125c1620d370f1523270f4e commit: 8b358d414735ab2e9125c1620d370f1523270f4e branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-07T10:06:54-07:00 summary: ssl docs: Fix typo (GH-32336) (cherry picked from commit 9ee2d3a93914776d15ac5cc7c44bb3aaca3e0fe5) Co-authored-by: Frederick files: M Doc/library/ssl.rst diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst index 0256e04a28c9e..f5446808cbdeb 100644 --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -1494,7 +1494,7 @@ to speed up repeated connections from the same clients. string must be the path to a single file in PEM format containing the certificate as well as any number of CA certificates needed to establish the certificate's authenticity. The *keyfile* string, if present, must - point to a file containing the private key in. Otherwise the private + point to a file containing the private key. Otherwise the private key will be taken from *certfile* as well. See the discussion of :ref:`ssl-certificates` for more information on how the certificate is stored in the *certfile*. From webhook-mailer at python.org Thu Apr 7 14:27:31 2022 From: webhook-mailer at python.org (ned-deily) Date: Thu, 07 Apr 2022 18:27:31 -0000 Subject: [Python-checkins] Doc: Fix spurious comma in the author metadata field (GH-32386) Message-ID: https://github.com/python/cpython/commit/1df4298b62fcc124ba9de861f8dc8239ad72cde2 commit: 1df4298b62fcc124ba9de861f8dc8239ad72cde2 branch: main author: CAM Gerlach committer: ned-deily date: 2022-04-07T14:27:14-04:00 summary: Doc: Fix spurious comma in the author metadata field (GH-32386) Signed-off-by: C.A.M. Gerlach files: M Doc/conf.py diff --git a/Doc/conf.py b/Doc/conf.py index 1aadd961b6ef0..e539da539e655 100644 --- a/Doc/conf.py +++ b/Doc/conf.py @@ -139,7 +139,7 @@ # Grouping the document tree into LaTeX files. List of tuples # (source start file, target name, title, author, document class [howto/manual]). -_stdauthor = r'Guido van Rossum\\and the Python development team' +_stdauthor = 'Guido van Rossum and the Python development team' latex_documents = [ ('c-api/index', 'c-api.tex', 'The Python/C API', _stdauthor, 'manual'), From webhook-mailer at python.org Thu Apr 7 15:27:44 2022 From: webhook-mailer at python.org (brettcannon) Date: Thu, 07 Apr 2022 19:27:44 -0000 Subject: [Python-checkins] Deprecate audioop (GH-32392) Message-ID: https://github.com/python/cpython/commit/87eec70d97b250f820325b4f1b4f781b443b5180 commit: 87eec70d97b250f820325b4f1b4f781b443b5180 branch: main author: Brett Cannon committer: brettcannon date: 2022-04-07T12:27:35-07:00 summary: Deprecate audioop (GH-32392) files: A Misc/NEWS.d/next/Library/2022-04-06-18-01-28.bpo-47061.qoVTR9.rst M Doc/whatsnew/3.11.rst M Lib/aifc.py M Lib/sunau.py M Lib/test/test_aifc.py M Lib/test/test_audioop.py M Lib/test/test_ossaudiodev.py M Lib/test/test_pyclbr.py M Lib/test/test_sunau.py M Lib/test/test_wave.py M Lib/wave.py M Modules/audioop.c diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index 31cdf10f66656..bc4a1953c10c2 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -833,6 +833,7 @@ Deprecated slated for removal in Python 3.13: * :mod:`aifc` + * :mod:`audioop` (Contributed by Brett Cannon in :issue:`47061`.) diff --git a/Lib/aifc.py b/Lib/aifc.py index b5eab9215d6ef..314cfd230d6cb 100644 --- a/Lib/aifc.py +++ b/Lib/aifc.py @@ -451,15 +451,21 @@ def readframes(self, nframes): # def _alaw2lin(self, data): - import audioop + with warnings.catch_warnings(): + warnings.simplefilter('ignore', category=DeprecationWarning) + import audioop return audioop.alaw2lin(data, 2) def _ulaw2lin(self, data): - import audioop + with warnings.catch_warnings(): + warnings.simplefilter('ignore', category=DeprecationWarning) + import audioop return audioop.ulaw2lin(data, 2) def _adpcm2lin(self, data): - import audioop + with warnings.catch_warnings(): + warnings.simplefilter('ignore', category=DeprecationWarning) + import audioop if not hasattr(self, '_adpcmstate'): # first time self._adpcmstate = None @@ -467,7 +473,9 @@ def _adpcm2lin(self, data): return data def _sowt2lin(self, data): - import audioop + with warnings.catch_warnings(): + warnings.simplefilter('ignore', category=DeprecationWarning) + import audioop return audioop.byteswap(data, 2) def _read_comm_chunk(self, chunk): @@ -774,22 +782,30 @@ def close(self): # def _lin2alaw(self, data): - import audioop + with warnings.catch_warnings(): + warnings.simplefilter('ignore', category=DeprecationWarning) + import audioop return audioop.lin2alaw(data, 2) def _lin2ulaw(self, data): - import audioop + with warnings.catch_warnings(): + warnings.simplefilter('ignore', category=DeprecationWarning) + import audioop return audioop.lin2ulaw(data, 2) def _lin2adpcm(self, data): - import audioop + with warnings.catch_warnings(): + warnings.simplefilter('ignore', category=DeprecationWarning) + import audioop if not hasattr(self, '_adpcmstate'): self._adpcmstate = None data, self._adpcmstate = audioop.lin2adpcm(data, 2, self._adpcmstate) return data def _lin2sowt(self, data): - import audioop + with warnings.catch_warnings(): + warnings.simplefilter('ignore', category=DeprecationWarning) + import audioop return audioop.byteswap(data, 2) def _ensure_header_written(self, datasize): diff --git a/Lib/sunau.py b/Lib/sunau.py index 79750a9d23ebd..9b3533d930676 100644 --- a/Lib/sunau.py +++ b/Lib/sunau.py @@ -104,6 +104,7 @@ """ from collections import namedtuple +import warnings _sunau_params = namedtuple('_sunau_params', @@ -275,7 +276,9 @@ def readframes(self, nframes): data = self._file.read(nframes * self._framesize) self._soundpos += len(data) // self._framesize if self._encoding == AUDIO_FILE_ENCODING_MULAW_8: - import audioop + with warnings.catch_warnings(): + warnings.simplefilter('ignore', category=DeprecationWarning) + import audioop data = audioop.ulaw2lin(data, self._sampwidth) return data return None # XXX--not implemented yet @@ -421,7 +424,9 @@ def writeframesraw(self, data): data = memoryview(data).cast('B') self._ensure_header_written() if self._comptype == 'ULAW': - import audioop + with warnings.catch_warnings(): + warnings.simplefilter('ignore', category=DeprecationWarning) + import audioop data = audioop.lin2ulaw(data, self._sampwidth) nframes = len(data) // self._framesize self._file.write(data) diff --git a/Lib/test/test_aifc.py b/Lib/test/test_aifc.py index ad8a7ee053cca..d3863d4915d44 100644 --- a/Lib/test/test_aifc.py +++ b/Lib/test/test_aifc.py @@ -4,13 +4,13 @@ import unittest from unittest import mock from test import audiotests -from audioop import byteswap import io import sys import struct aifc = import_deprecated("aifc") +audioop = import_deprecated("audioop") class AifcTest(audiotests.AudioWriteTests, @@ -124,7 +124,7 @@ class AifcULAWTest(AifcTest, unittest.TestCase): E5040CBC 617C0A3C 08BC0A3C 2C7C0B3C 517C0E3C 8A8410FC B6840EBC 457C0A3C \ """) if sys.byteorder != 'big': - frames = byteswap(frames, 2) + frames = audioop.byteswap(frames, 2) class AifcALAWTest(AifcTest, unittest.TestCase): @@ -145,7 +145,7 @@ class AifcALAWTest(AifcTest, unittest.TestCase): E4800CC0 62000A40 08C00A40 2B000B40 52000E40 8A001180 B6000EC0 46000A40 \ """) if sys.byteorder != 'big': - frames = byteswap(frames, 2) + frames = audioop.byteswap(frames, 2) class AifcMiscTest(unittest.TestCase): diff --git a/Lib/test/test_audioop.py b/Lib/test/test_audioop.py index 9baa62ad45c04..05c0f20e122a4 100644 --- a/Lib/test/test_audioop.py +++ b/Lib/test/test_audioop.py @@ -1,7 +1,10 @@ -import audioop import sys +from test.support import warnings_helper import unittest +audioop = warnings_helper.import_deprecated("audioop") + + def pack(width, data): return b''.join(v.to_bytes(width, sys.byteorder, signed=True) for v in data) diff --git a/Lib/test/test_ossaudiodev.py b/Lib/test/test_ossaudiodev.py index ebce3e9c272f9..37d2d1f5ff441 100644 --- a/Lib/test/test_ossaudiodev.py +++ b/Lib/test/test_ossaudiodev.py @@ -1,16 +1,16 @@ from test import support -from test.support import import_helper +from test.support import import_helper, warnings_helper support.requires('audio') from test.support import findfile ossaudiodev = import_helper.import_module('ossaudiodev') +audioop = warnings_helper.import_deprecated('audioop') import errno import sys import sunau import time -import audioop import unittest # Arggh, AFMT_S16_NE not defined on all platforms -- seems to be a diff --git a/Lib/test/test_pyclbr.py b/Lib/test/test_pyclbr.py index 329acf0c64295..ad7b31aef1ddd 100644 --- a/Lib/test/test_pyclbr.py +++ b/Lib/test/test_pyclbr.py @@ -217,7 +217,6 @@ def test_others(self): cm = self.checkModule # These were once some of the longest modules. - cm('aifc', ignore=('_aifc_params',)) # set with = in module cm('random', ignore=('Random',)) # from _random import Random as CoreGenerator cm('cgi', ignore=('log',)) # set with = in module cm('pickle', ignore=('partial', 'PickleBuffer')) diff --git a/Lib/test/test_sunau.py b/Lib/test/test_sunau.py index 7f1c0a5cbdeda..e65742b69f2e9 100644 --- a/Lib/test/test_sunau.py +++ b/Lib/test/test_sunau.py @@ -1,10 +1,12 @@ import unittest from test import audiotests -from audioop import byteswap import io import struct import sys import sunau +from test.support import warnings_helper + +audioop = warnings_helper.import_deprecated("audioop") class SunauTest(audiotests.AudioWriteTests, @@ -116,7 +118,7 @@ class SunauULAWTest(SunauTest, unittest.TestCase): E5040CBC 617C0A3C 08BC0A3C 2C7C0B3C 517C0E3C 8A8410FC B6840EBC 457C0A3C \ """) if sys.byteorder != 'big': - frames = byteswap(frames, 2) + frames = audioop.byteswap(frames, 2) class SunauLowLevelTest(unittest.TestCase): diff --git a/Lib/test/test_wave.py b/Lib/test/test_wave.py index f85e40b31d010..0cc94e88b437d 100644 --- a/Lib/test/test_wave.py +++ b/Lib/test/test_wave.py @@ -1,7 +1,6 @@ import unittest from test import audiotests from test import support -from audioop import byteswap import io import struct import sys @@ -48,7 +47,7 @@ class WavePCM16Test(WaveTest, unittest.TestCase): E4B50CEB 63440A5A 08CA0A1F 2BBA0B0B 51460E47 8BCB113C B6F50EEA 44150A59 \ """) if sys.byteorder != 'big': - frames = byteswap(frames, 2) + frames = wave._byteswap(frames, 2) class WavePCM24Test(WaveTest, unittest.TestCase): @@ -75,7 +74,7 @@ class WavePCM24Test(WaveTest, unittest.TestCase): 51486F0E44E1 8BCC64113B05 B6F4EC0EEB36 4413170A5B48 \ """) if sys.byteorder != 'big': - frames = byteswap(frames, 3) + frames = wave._byteswap(frames, 3) class WavePCM32Test(WaveTest, unittest.TestCase): @@ -102,7 +101,7 @@ class WavePCM32Test(WaveTest, unittest.TestCase): 51486F800E44E190 8BCC6480113B0580 B6F4EC000EEB3630 441317800A5B48A0 \ """) if sys.byteorder != 'big': - frames = byteswap(frames, 4) + frames = wave._byteswap(frames, 4) class MiscTestCase(unittest.TestCase): diff --git a/Lib/wave.py b/Lib/wave.py index b7071198e6b84..47a233df0a3ab 100644 --- a/Lib/wave.py +++ b/Lib/wave.py @@ -73,7 +73,6 @@ from chunk import Chunk from collections import namedtuple -import audioop import builtins import struct import sys @@ -91,6 +90,16 @@ class Error(Exception): _wave_params = namedtuple('_wave_params', 'nchannels sampwidth framerate nframes comptype compname') +def _byteswap(data, width): + swapped_data = bytearray(len(data)) + + for i in range(0, len(data), width): + for j in range(width): + swapped_data[i + width - 1 - j] = data[i + j] + + return bytes(swapped_data) + + class Wave_read: """Variables used in this class: @@ -241,7 +250,7 @@ def readframes(self, nframes): return b'' data = self._data_chunk.read(nframes * self._framesize) if self._sampwidth != 1 and sys.byteorder == 'big': - data = audioop.byteswap(data, self._sampwidth) + data = _byteswap(data, self._sampwidth) if self._convert and data: data = self._convert(data) self._soundpos = self._soundpos + len(data) // (self._nchannels * self._sampwidth) @@ -428,7 +437,7 @@ def writeframesraw(self, data): if self._convert: data = self._convert(data) if self._sampwidth != 1 and sys.byteorder == 'big': - data = audioop.byteswap(data, self._sampwidth) + data = _byteswap(data, self._sampwidth) self._file.write(data) self._datawritten += len(data) self._nframeswritten = self._nframeswritten + nframes diff --git a/Misc/NEWS.d/next/Library/2022-04-06-18-01-28.bpo-47061.qoVTR9.rst b/Misc/NEWS.d/next/Library/2022-04-06-18-01-28.bpo-47061.qoVTR9.rst new file mode 100644 index 0000000000000..65ffa2e180791 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-06-18-01-28.bpo-47061.qoVTR9.rst @@ -0,0 +1 @@ +Deprecate audioop. diff --git a/Modules/audioop.c b/Modules/audioop.c index 32237ca6177cd..d74e634ac4488 100644 --- a/Modules/audioop.c +++ b/Modules/audioop.c @@ -1975,5 +1975,12 @@ static struct PyModuleDef audioopmodule = { PyMODINIT_FUNC PyInit_audioop(void) { + if (PyErr_WarnEx(PyExc_DeprecationWarning, + "'audioop' is deprecated and slated for removal in " + "Python 3.13", + 7)) { + return NULL; + } + return PyModuleDef_Init(&audioopmodule); } From webhook-mailer at python.org Thu Apr 7 15:31:06 2022 From: webhook-mailer at python.org (brandtbucher) Date: Thu, 07 Apr 2022 19:31:06 -0000 Subject: [Python-checkins] bpo-47177: Replace `f_lasti` with `prev_instr` (GH-32208) Message-ID: https://github.com/python/cpython/commit/ef6a482b0285870c45f39c9b17ed827362b334ae commit: ef6a482b0285870c45f39c9b17ed827362b334ae branch: main author: Brandt Bucher committer: brandtbucher date: 2022-04-07T12:31:01-07:00 summary: bpo-47177: Replace `f_lasti` with `prev_instr` (GH-32208) files: A Misc/NEWS.d/next/Core and Builtins/2022-03-31-01-30-03.bpo-47177.fQqaov.rst M Include/internal/pycore_frame.h M Modules/_tracemalloc.c M Objects/frameobject.c M Objects/genobject.c M Objects/typeobject.c M Python/ceval.c M Python/frame.c M Python/traceback.c M Tools/gdb/libpython.py diff --git a/Include/internal/pycore_frame.h b/Include/internal/pycore_frame.h index 211831a6e497f..49bdc6324ca36 100644 --- a/Include/internal/pycore_frame.h +++ b/Include/internal/pycore_frame.h @@ -39,11 +39,6 @@ enum _frameowner { FRAME_OWNED_BY_FRAME_OBJECT = 2 }; -/* - frame->f_lasti refers to the index of the last instruction, - unless it's -1 in which case next_instr should be first_instr. -*/ - typedef struct _PyInterpreterFrame { PyFunctionObject *f_func; /* Strong reference */ PyObject *f_globals; /* Borrowed reference */ @@ -52,13 +47,20 @@ typedef struct _PyInterpreterFrame { PyCodeObject *f_code; /* Strong reference */ PyFrameObject *frame_obj; /* Strong reference, may be NULL */ struct _PyInterpreterFrame *previous; - int f_lasti; /* Last instruction if called */ + // NOTE: This is not necessarily the last instruction started in the given + // frame. Rather, it is the code unit *prior to* the *next* instruction. For + // example, it may be an inline CACHE entry, an instruction we just jumped + // over, or (in the case of a newly-created frame) a totally invalid value: + _Py_CODEUNIT *prev_instr; int stacktop; /* Offset of TOS from localsplus */ bool is_entry; // Whether this is the "root" frame for the current _PyCFrame. char owner; PyObject *localsplus[1]; } _PyInterpreterFrame; +#define _PyInterpreterFrame_LASTI(IF) \ + ((int)((IF)->prev_instr - _PyCode_CODE((IF)->f_code))) + static inline PyObject **_PyFrame_Stackbase(_PyInterpreterFrame *f) { return f->localsplus + f->f_code->co_nlocalsplus; } @@ -97,7 +99,7 @@ _PyFrame_InitializeSpecials( frame->f_locals = Py_XNewRef(locals); frame->stacktop = nlocalsplus; frame->frame_obj = NULL; - frame->f_lasti = -1; + frame->prev_instr = _PyCode_CODE(frame->f_code) - 1; frame->is_entry = false; frame->owner = FRAME_OWNED_BY_THREAD; } @@ -186,6 +188,7 @@ void _PyThreadState_PopFrame(PyThreadState *tstate, _PyInterpreterFrame *frame); _PyInterpreterFrame * _PyFrame_Push(PyThreadState *tstate, PyFunctionObject *func); +int _PyInterpreterFrame_GetLine(_PyInterpreterFrame *frame); static inline PyGenObject *_PyFrame_GetGenerator(_PyInterpreterFrame *frame) diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-03-31-01-30-03.bpo-47177.fQqaov.rst b/Misc/NEWS.d/next/Core and Builtins/2022-03-31-01-30-03.bpo-47177.fQqaov.rst new file mode 100644 index 0000000000000..01e6c88392458 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2022-03-31-01-30-03.bpo-47177.fQqaov.rst @@ -0,0 +1 @@ +Replace the ``f_lasti`` member of the internal ``_PyInterpreterFrame`` structure with a ``prev_instr`` pointer, which reduces overhead in the main interpreter loop. The?``f_lasti`` attribute of Python-layer frame objects is preserved for backward-compatibility. diff --git a/Modules/_tracemalloc.c b/Modules/_tracemalloc.c index 738d54530c967..ae09869deda70 100644 --- a/Modules/_tracemalloc.c +++ b/Modules/_tracemalloc.c @@ -308,7 +308,7 @@ static void tracemalloc_get_frame(_PyInterpreterFrame *pyframe, frame_t *frame) { frame->filename = &_Py_STR(anon_unknown); - int lineno = PyCode_Addr2Line(pyframe->f_code, pyframe->f_lasti*sizeof(_Py_CODEUNIT)); + int lineno = _PyInterpreterFrame_GetLine(pyframe); if (lineno < 0) { lineno = 0; } diff --git a/Objects/frameobject.c b/Objects/frameobject.c index 6842e62839fd1..07b610717d2a2 100644 --- a/Objects/frameobject.c +++ b/Objects/frameobject.c @@ -39,7 +39,7 @@ PyFrame_GetLineNumber(PyFrameObject *f) return f->f_lineno; } else { - return PyCode_Addr2Line(f->f_frame->f_code, f->f_frame->f_lasti*sizeof(_Py_CODEUNIT)); + return _PyInterpreterFrame_GetLine(f->f_frame); } } @@ -58,10 +58,11 @@ frame_getlineno(PyFrameObject *f, void *closure) static PyObject * frame_getlasti(PyFrameObject *f, void *closure) { - if (f->f_frame->f_lasti < 0) { + int lasti = _PyInterpreterFrame_LASTI(f->f_frame); + if (lasti < 0) { return PyLong_FromLong(-1); } - return PyLong_FromLong(f->f_frame->f_lasti*sizeof(_Py_CODEUNIT)); + return PyLong_FromLong(lasti * sizeof(_Py_CODEUNIT)); } static PyObject * @@ -419,12 +420,11 @@ _PyFrame_GetState(PyFrameObject *frame) } case FRAME_OWNED_BY_THREAD: { - if (frame->f_frame->f_lasti < 0) { + if (_PyInterpreterFrame_LASTI(frame->f_frame) < 0) { return FRAME_CREATED; } - uint8_t *code = (uint8_t *)frame->f_frame->f_code->co_code_adaptive; - int opcode = code[frame->f_frame->f_lasti*sizeof(_Py_CODEUNIT)]; - switch(_PyOpcode_Deopt[opcode]) { + switch (_PyOpcode_Deopt[_Py_OPCODE(*frame->f_frame->prev_instr)]) + { case COPY_FREE_VARS: case MAKE_CELL: case RETURN_GENERATOR: @@ -555,7 +555,7 @@ frame_setlineno(PyFrameObject *f, PyObject* p_new_lineno, void *Py_UNUSED(ignore int64_t best_stack = OVERFLOWED; int best_addr = -1; - int64_t start_stack = stacks[f->f_frame->f_lasti]; + int64_t start_stack = stacks[_PyInterpreterFrame_LASTI(f->f_frame)]; int err = -1; const char *msg = "cannot find bytecode for specified line"; for (int i = 0; i < len; i++) { @@ -598,7 +598,7 @@ frame_setlineno(PyFrameObject *f, PyObject* p_new_lineno, void *Py_UNUSED(ignore } /* Finally set the new lasti and return OK. */ f->f_lineno = 0; - f->f_frame->f_lasti = best_addr; + f->f_frame->prev_instr = _PyCode_CODE(f->f_frame->f_code) + best_addr; return 0; } @@ -880,10 +880,11 @@ _PyFrame_OpAlreadyRan(_PyInterpreterFrame *frame, int opcode, int oparg) // This only works when opcode is a non-quickened form: assert(_PyOpcode_Deopt[opcode] == opcode); int check_oparg = 0; - for (int i = 0; i < frame->f_lasti; i++) { - _Py_CODEUNIT instruction = _PyCode_CODE(frame->f_code)[i]; - int check_opcode = _PyOpcode_Deopt[_Py_OPCODE(instruction)]; - check_oparg |= _Py_OPARG(instruction); + for (_Py_CODEUNIT *instruction = _PyCode_CODE(frame->f_code); + instruction < frame->prev_instr; instruction++) + { + int check_opcode = _PyOpcode_Deopt[_Py_OPCODE(*instruction)]; + check_oparg |= _Py_OPARG(*instruction); if (check_opcode == opcode && check_oparg == oparg) { return 1; } @@ -893,7 +894,7 @@ _PyFrame_OpAlreadyRan(_PyInterpreterFrame *frame, int opcode, int oparg) else { check_oparg = 0; } - i += _PyOpcode_Caches[check_opcode]; + instruction += _PyOpcode_Caches[check_opcode]; } return 0; } @@ -914,8 +915,8 @@ _PyFrame_FastToLocalsWithError(_PyInterpreterFrame *frame) { fast = _PyFrame_GetLocalsArray(frame); // COPY_FREE_VARS has no quickened forms, so no need to use _PyOpcode_Deopt // here: - if (frame->f_lasti < 0 && _Py_OPCODE(_PyCode_CODE(co)[0]) == COPY_FREE_VARS) - { + int lasti = _PyInterpreterFrame_LASTI(frame); + if (lasti < 0 && _Py_OPCODE(_PyCode_CODE(co)[0]) == COPY_FREE_VARS) { /* Free vars have not been initialized -- Do that */ PyCodeObject *co = frame->f_code; PyObject *closure = frame->f_func->func_closure; @@ -926,7 +927,7 @@ _PyFrame_FastToLocalsWithError(_PyInterpreterFrame *frame) { frame->localsplus[offset + i] = o; } // COPY_FREE_VARS doesn't have inline CACHEs, either: - frame->f_lasti = 0; + frame->prev_instr = _PyCode_CODE(frame->f_code); } for (int i = 0; i < co->co_nlocalsplus; i++) { _PyLocals_Kind kind = _PyLocals_GetKind(co->co_localspluskinds, i); diff --git a/Objects/genobject.c b/Objects/genobject.c index cdb2a0f76b085..e58118b2694ff 100644 --- a/Objects/genobject.c +++ b/Objects/genobject.c @@ -352,14 +352,14 @@ _PyGen_yf(PyGenObject *gen) if (gen->gi_frame_state < FRAME_CLEARED) { _PyInterpreterFrame *frame = (_PyInterpreterFrame *)gen->gi_iframe; - if (frame->f_lasti < 1) { + if (gen->gi_frame_state == FRAME_CREATED) { /* Return immediately if the frame didn't start yet. SEND always come after LOAD_CONST: a code object should not start with SEND */ assert(_Py_OPCODE(_PyCode_CODE(gen->gi_code)[0]) != SEND); return NULL; } - _Py_CODEUNIT next = _PyCode_CODE(gen->gi_code)[frame->f_lasti + 1]; + _Py_CODEUNIT next = frame->prev_instr[1]; if (_PyOpcode_Deopt[_Py_OPCODE(next)] != RESUME || _Py_OPARG(next) < 2) { /* Not in a yield from */ @@ -490,13 +490,11 @@ _gen_throw(PyGenObject *gen, int close_on_genexit, // XXX: Performing this jump ourselves is awkward and problematic. // See https://github.com/python/cpython/pull/31968. /* Termination repetition of SEND loop */ - assert(frame->f_lasti >= 0); - _Py_CODEUNIT *code = _PyCode_CODE(gen->gi_code); + assert(_PyInterpreterFrame_LASTI(frame) >= 0); /* Backup to SEND */ - frame->f_lasti--; - assert(_Py_OPCODE(code[frame->f_lasti]) == SEND); - int jump = _Py_OPARG(code[frame->f_lasti]); - frame->f_lasti += jump; + assert(_Py_OPCODE(frame->prev_instr[-1]) == SEND); + int jump = _Py_OPARG(frame->prev_instr[-1]); + frame->prev_instr += jump - 1; if (_PyGen_FetchStopIterationValue(&val) == 0) { ret = gen_send(gen, val); Py_DECREF(val); @@ -1344,9 +1342,8 @@ compute_cr_origin(int origin_depth, _PyInterpreterFrame *current_frame) frame = current_frame; for (int i = 0; i < frame_count; ++i) { PyCodeObject *code = frame->f_code; - PyObject *frameinfo = Py_BuildValue("OiO", - code->co_filename, - PyCode_Addr2Line(frame->f_code, frame->f_lasti*sizeof(_Py_CODEUNIT)), + int line = _PyInterpreterFrame_GetLine(frame); + PyObject *frameinfo = Py_BuildValue("OiO", code->co_filename, line, code->co_name); if (!frameinfo) { Py_DECREF(cr_origin); diff --git a/Objects/typeobject.c b/Objects/typeobject.c index 53e4f0781d65b..64c4bbb5a9326 100644 --- a/Objects/typeobject.c +++ b/Objects/typeobject.c @@ -8980,7 +8980,7 @@ super_init_without_args(_PyInterpreterFrame *cframe, PyCodeObject *co, if (firstarg != NULL && (_PyLocals_GetKind(co->co_localspluskinds, 0) & CO_FAST_CELL)) { // "firstarg" is a cell here unless (very unlikely) super() // was called from the C-API before the first MAKE_CELL op. - if (cframe->f_lasti >= 0) { + if (_PyInterpreterFrame_LASTI(cframe) >= 0) { // MAKE_CELL and COPY_FREE_VARS have no quickened forms, so no need // to use _PyOpcode_Deopt here: assert(_Py_OPCODE(_PyCode_CODE(co)[0]) == MAKE_CELL || diff --git a/Python/ceval.c b/Python/ceval.c index 5384aac5d6e6c..cb739c1572d16 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -58,12 +58,10 @@ static int prtrace(PyThreadState *, PyObject *, const char *); static void lltrace_instruction(_PyInterpreterFrame *frame, int opcode, int oparg) { if (HAS_ARG(opcode)) { - printf("%d: %d, %d\n", - frame->f_lasti, opcode, oparg); + printf("%d: %d, %d\n", _PyInterpreterFrame_LASTI(frame), opcode, oparg); } else { - printf("%d: %d\n", - frame->f_lasti, opcode); + printf("%d: %d\n", _PyInterpreterFrame_LASTI(frame), opcode); } } #endif @@ -1249,14 +1247,13 @@ eval_frame_handle_pending(PyThreadState *tstate) #ifdef Py_STATS #define INSTRUCTION_START(op) \ do { \ - frame->f_lasti = INSTR_OFFSET(); \ - next_instr++; \ + frame->prev_instr = next_instr++; \ OPCODE_EXE_INC(op); \ _py_stats.opcode_stats[lastopcode].pair_count[op]++; \ lastopcode = op; \ } while (0) #else -#define INSTRUCTION_START(op) frame->f_lasti = INSTR_OFFSET(); next_instr++ +#define INSTRUCTION_START(op) (frame->prev_instr = next_instr++) #endif #if USE_COMPUTED_GOTOS @@ -1646,9 +1643,9 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int consts = co->co_consts; \ first_instr = _PyCode_CODE(co); \ } \ - assert(frame->f_lasti >= -1); \ + assert(_PyInterpreterFrame_LASTI(frame) >= -1); \ /* Jump back to the last instruction executed... */ \ - next_instr = first_instr + frame->f_lasti + 1; \ + next_instr = frame->prev_instr + 1; \ stack_pointer = _PyFrame_GetStackPointer(frame); \ /* Set stackdepth to -1. \ Update when returning or calling trace function. \ @@ -2204,7 +2201,8 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int new_frame->localsplus[i] = NULL; } _PyFrame_SetStackPointer(frame, stack_pointer); - frame->f_lasti += INLINE_CACHE_ENTRIES_BINARY_SUBSCR; + JUMPBY(INLINE_CACHE_ENTRIES_BINARY_SUBSCR); + frame->prev_instr = next_instr - 1; new_frame->previous = frame; frame = cframe.current_frame = new_frame; CALL_STAT_INC(inlined_py_calls); @@ -2605,7 +2603,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int if (oparg) { PyObject *lasti = PEEK(oparg + 1); if (PyLong_Check(lasti)) { - frame->f_lasti = PyLong_AsLong(lasti); + frame->prev_instr = first_instr + PyLong_AsLong(lasti); assert(!_PyErr_Occurred(tstate)); } else { @@ -4603,7 +4601,8 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int goto error; } _PyFrame_SetStackPointer(frame, stack_pointer); - frame->f_lasti += INLINE_CACHE_ENTRIES_CALL; + JUMPBY(INLINE_CACHE_ENTRIES_CALL); + frame->prev_instr = next_instr - 1; new_frame->previous = frame; cframe.current_frame = frame = new_frame; CALL_STAT_INC(inlined_py_calls); @@ -4708,7 +4707,8 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int } STACK_SHRINK(2-is_meth); _PyFrame_SetStackPointer(frame, stack_pointer); - frame->f_lasti += INLINE_CACHE_ENTRIES_CALL; + JUMPBY(INLINE_CACHE_ENTRIES_CALL); + frame->prev_instr = next_instr - 1; new_frame->previous = frame; frame = cframe.current_frame = new_frame; goto start_frame; @@ -4748,7 +4748,8 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int } STACK_SHRINK(2-is_meth); _PyFrame_SetStackPointer(frame, stack_pointer); - frame->f_lasti += INLINE_CACHE_ENTRIES_CALL; + JUMPBY(INLINE_CACHE_ENTRIES_CALL); + frame->prev_instr = next_instr - 1; new_frame->previous = frame; frame = cframe.current_frame = new_frame; goto start_frame; @@ -5435,8 +5436,8 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int #endif { if (tstate->tracing == 0) { - int instr_prev = frame->f_lasti; - frame->f_lasti = INSTR_OFFSET(); + int instr_prev = _PyInterpreterFrame_LASTI(frame); + frame->prev_instr = next_instr; TRACING_NEXTOPARG(); switch(opcode) { case COPY_FREE_VARS: @@ -5474,7 +5475,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int goto error; } /* Reload possibly changed frame fields */ - JUMPTO(frame->f_lasti); + next_instr = frame->prev_instr; stack_pointer = _PyFrame_GetStackPointer(frame); frame->stacktop = -1; @@ -5491,10 +5492,8 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int #else default: #endif - fprintf(stderr, - "XXX lineno: %d, opcode: %d\n", - PyCode_Addr2Line(frame->f_code, frame->f_lasti*sizeof(_Py_CODEUNIT)), - opcode); + fprintf(stderr, "XXX lineno: %d, opcode: %d\n", + _PyInterpreterFrame_GetLine(frame), opcode); _PyErr_SetString(tstate, PyExc_SystemError, "unknown opcode"); goto error; @@ -5598,7 +5597,8 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int } PyObject *exc, *val, *tb; if (lasti) { - PyObject *lasti = PyLong_FromLong(frame->f_lasti); + int frame_lasti = _PyInterpreterFrame_LASTI(frame); + PyObject *lasti = PyLong_FromLong(frame_lasti); if (lasti == NULL) { goto exception_unwind; } @@ -6690,9 +6690,10 @@ call_trace(Py_tracefunc func, PyObject *obj, int old_what = tstate->tracing_what; tstate->tracing_what = what; PyThreadState_EnterTracing(tstate); - assert(frame->f_lasti >= 0); + assert(_PyInterpreterFrame_LASTI(frame) >= 0); initialize_trace_info(&tstate->trace_info, frame); - f->f_lineno = _PyCode_CheckLineNumber(frame->f_lasti*sizeof(_Py_CODEUNIT), &tstate->trace_info.bounds); + int addr = _PyInterpreterFrame_LASTI(frame) * sizeof(_Py_CODEUNIT); + f->f_lineno = _PyCode_CheckLineNumber(addr, &tstate->trace_info.bounds); result = func(obj, f, what, arg); f->f_lineno = 0; PyThreadState_LeaveTracing(tstate); @@ -6742,7 +6743,8 @@ maybe_call_line_trace(Py_tracefunc func, PyObject *obj, else { lastline = _PyCode_CheckLineNumber(instr_prev*sizeof(_Py_CODEUNIT), &tstate->trace_info.bounds); } - int line = _PyCode_CheckLineNumber(frame->f_lasti*sizeof(_Py_CODEUNIT), &tstate->trace_info.bounds); + int addr = _PyInterpreterFrame_LASTI(frame) * sizeof(_Py_CODEUNIT); + int line = _PyCode_CheckLineNumber(addr, &tstate->trace_info.bounds); PyFrameObject *f = _PyFrame_GetFrameObject(frame); if (f == NULL) { return -1; @@ -6750,10 +6752,10 @@ maybe_call_line_trace(Py_tracefunc func, PyObject *obj, if (line != -1 && f->f_trace_lines) { /* Trace backward edges (except in 'yield from') or if line number has changed */ int trace = line != lastline || - (frame->f_lasti < instr_prev && - // SEND has no quickened forms, so no need to use _PyOpcode_Deopt - // here: - _Py_OPCODE(_PyCode_CODE(frame->f_code)[frame->f_lasti]) != SEND); + (_PyInterpreterFrame_LASTI(frame) < instr_prev && + // SEND has no quickened forms, so no need to use _PyOpcode_Deopt + // here: + _Py_OPCODE(*frame->prev_instr) != SEND); if (trace) { result = call_trace(func, obj, tstate, frame, PyTrace_LINE, Py_None); } @@ -7685,7 +7687,7 @@ dtrace_function_entry(_PyInterpreterFrame *frame) PyCodeObject *code = frame->f_code; filename = PyUnicode_AsUTF8(code->co_filename); funcname = PyUnicode_AsUTF8(code->co_name); - lineno = PyCode_Addr2Line(frame->f_code, frame->f_lasti*sizeof(_Py_CODEUNIT)); + lineno = _PyInterpreterFrame_GetLine(frame); PyDTrace_FUNCTION_ENTRY(filename, funcname, lineno); } @@ -7700,7 +7702,7 @@ dtrace_function_return(_PyInterpreterFrame *frame) PyCodeObject *code = frame->f_code; filename = PyUnicode_AsUTF8(code->co_filename); funcname = PyUnicode_AsUTF8(code->co_name); - lineno = PyCode_Addr2Line(frame->f_code, frame->f_lasti*sizeof(_Py_CODEUNIT)); + lineno = _PyInterpreterFrame_GetLine(frame); PyDTrace_FUNCTION_RETURN(filename, funcname, lineno); } @@ -7718,11 +7720,12 @@ maybe_dtrace_line(_PyInterpreterFrame *frame, */ initialize_trace_info(trace_info, frame); int lastline = _PyCode_CheckLineNumber(instr_prev*sizeof(_Py_CODEUNIT), &trace_info->bounds); - int line = _PyCode_CheckLineNumber(frame->f_lasti*sizeof(_Py_CODEUNIT), &trace_info->bounds); + int addr = _PyInterpreterFrame_LASTI(frame) * sizeof(_Py_CODEUNIT); + int line = _PyCode_CheckLineNumber(addr, &trace_info->bounds); if (line != -1) { /* Trace backward edges or first instruction of a new line */ - if (frame->f_lasti < instr_prev || - (line != lastline && frame->f_lasti*sizeof(_Py_CODEUNIT) == (unsigned int)trace_info->bounds.ar_start)) + if (_PyInterpreterFrame_LASTI(frame) < instr_prev || + (line != lastline && addr == trace_info->bounds.ar_start)) { co_filename = PyUnicode_AsUTF8(frame->f_code->co_filename); if (!co_filename) { diff --git a/Python/frame.c b/Python/frame.c index 3396ed8d2aeb0..c2da123a2bbc1 100644 --- a/Python/frame.c +++ b/Python/frame.c @@ -127,3 +127,10 @@ _PyFrame_Push(PyThreadState *tstate, PyFunctionObject *func) _PyFrame_InitializeSpecials(new_frame, func, NULL, code->co_nlocalsplus); return new_frame; } + +int +_PyInterpreterFrame_GetLine(_PyInterpreterFrame *frame) +{ + int addr = _PyInterpreterFrame_LASTI(frame) * sizeof(_Py_CODEUNIT); + return PyCode_Addr2Line(frame->f_code, addr); +} diff --git a/Python/traceback.c b/Python/traceback.c index 488c1b17cf51f..3ec0618af99f2 100644 --- a/Python/traceback.c +++ b/Python/traceback.c @@ -238,8 +238,8 @@ _PyTraceBack_FromFrame(PyObject *tb_next, PyFrameObject *frame) { assert(tb_next == NULL || PyTraceBack_Check(tb_next)); assert(frame != NULL); - - return tb_create_raw((PyTracebackObject *)tb_next, frame, frame->f_frame->f_lasti*sizeof(_Py_CODEUNIT), + int addr = _PyInterpreterFrame_LASTI(frame->f_frame) * sizeof(_Py_CODEUNIT); + return tb_create_raw((PyTracebackObject *)tb_next, frame, addr, PyFrame_GetLineNumber(frame)); } @@ -1180,7 +1180,7 @@ dump_frame(int fd, _PyInterpreterFrame *frame) PUTS(fd, "???"); } - int lineno = PyCode_Addr2Line(code, frame->f_lasti*sizeof(_Py_CODEUNIT)); + int lineno = _PyInterpreterFrame_GetLine(frame); PUTS(fd, ", line "); if (lineno >= 0) { _Py_DumpDecimal(fd, (size_t)lineno); diff --git a/Tools/gdb/libpython.py b/Tools/gdb/libpython.py index 00cdcca084e74..4f7a8bca5fd78 100755 --- a/Tools/gdb/libpython.py +++ b/Tools/gdb/libpython.py @@ -1015,7 +1015,10 @@ def _f_nlocalsplus(self): return self._f_special("nlocalsplus", int_from_int) def _f_lasti(self): - return self._f_special("f_lasti", int_from_int) + codeunit_p = gdb.lookup_type("_Py_CODEUNIT").pointer() + prev_instr = self._gdbval["prev_instr"] + first_instr = self._f_code().field("co_code_adaptive").cast(codeunit_p) + return int(prev_instr - first_instr) def is_entry(self): return self._f_special("is_entry", bool) From webhook-mailer at python.org Thu Apr 7 18:21:11 2022 From: webhook-mailer at python.org (zooba) Date: Thu, 07 Apr 2022 22:21:11 -0000 Subject: [Python-checkins] bpo-47103: Copy pgort140.dll into output directory when building PGInstrument on Windows (GH-32083) Message-ID: https://github.com/python/cpython/commit/b0ec17b6d9e0fb61081b6d15a1b2a14b607851b7 commit: b0ec17b6d9e0fb61081b6d15a1b2a14b607851b7 branch: 3.10 author: Steve Dower committer: zooba date: 2022-04-07T23:21:03+01:00 summary: bpo-47103: Copy pgort140.dll into output directory when building PGInstrument on Windows (GH-32083) files: A Misc/NEWS.d/next/Build/2022-03-23-20-01-16.bpo-47103.b4-00F.rst M Lib/test/test_embed.py M PCbuild/python.vcxproj diff --git a/Lib/test/test_embed.py b/Lib/test/test_embed.py index 503d4925c9402..8c343f3721085 100644 --- a/Lib/test/test_embed.py +++ b/Lib/test/test_embed.py @@ -1195,20 +1195,11 @@ def tmpdir_with_python(self): if MS_WINDOWS: # Copy pythonXY.dll (or pythonXY_d.dll) - ver = sys.version_info - dll = f'python{ver.major}{ver.minor}' - dll3 = f'python{ver.major}' - if debug_build(sys.executable): - dll += '_d' - dll3 += '_d' - dll += '.dll' - dll3 += '.dll' - dll = os.path.join(os.path.dirname(self.test_exe), dll) - dll3 = os.path.join(os.path.dirname(self.test_exe), dll3) - dll_copy = os.path.join(tmpdir, os.path.basename(dll)) - dll3_copy = os.path.join(tmpdir, os.path.basename(dll3)) - shutil.copyfile(dll, dll_copy) - shutil.copyfile(dll3, dll3_copy) + import fnmatch + exedir = os.path.dirname(self.test_exe) + for f in os.listdir(exedir): + if fnmatch.fnmatch(f, '*.dll'): + shutil.copyfile(os.path.join(exedir, f), os.path.join(tmpdir, f)) # Copy Python program exec_copy = os.path.join(tmpdir, os.path.basename(self.test_exe)) diff --git a/Misc/NEWS.d/next/Build/2022-03-23-20-01-16.bpo-47103.b4-00F.rst b/Misc/NEWS.d/next/Build/2022-03-23-20-01-16.bpo-47103.b4-00F.rst new file mode 100644 index 0000000000000..c1e01adce0d26 --- /dev/null +++ b/Misc/NEWS.d/next/Build/2022-03-23-20-01-16.bpo-47103.b4-00F.rst @@ -0,0 +1,2 @@ +Windows ``PGInstrument`` builds now copy a required DLL into the output +directory, making it easier to run the profile stage of a PGO build. diff --git a/PCbuild/python.vcxproj b/PCbuild/python.vcxproj index b58945a4d19b6..b6dcf14823546 100644 --- a/PCbuild/python.vcxproj +++ b/PCbuild/python.vcxproj @@ -128,9 +128,6 @@ set PYTHONPATH=$(PySourcePath)Lib - <_PGOPath Condition="$(Configuration) == 'PGInstrument' and $(Platform) == 'Win32'">@set PATH=%PATH%%3B$(VCInstallDir)bin - <_PGOPath Condition="$(Configuration) == 'PGInstrument' and $(Platform) == 'x64'">@set PATH=%PATH%%3B$(VCInstallDir)bin\amd64 - <_PGOPath Condition="$(Configuration) == 'PGInstrument' and $(VC_PGO_RunTime_Dir) != ''">@set PATH=%PATH%%3B$(VC_PGO_RunTime_Dir) <_Content>@rem This script invokes the most recently built Python with all arguments @rem passed through to the interpreter. This file is generated by the @rem build process and any changes *will* be thrown away by the next @@ -140,11 +137,21 @@ set PYTHONPATH=$(PySourcePath)Lib @echo Running $(Configuration)^|$(Platform) interpreter... @setlocal @set PYTHONHOME=$(PySourcePath) -$(_PGOPath) @"$(OutDir)python$(PyDebugExt).exe" %* <_ExistingContent Condition="Exists('$(PySourcePath)python.bat')">$([System.IO.File]::ReadAllText('$(PySourcePath)python.bat')) + + + <_PGORT Include="$(VCToolsInstallDir)bin\Hostx86\x86\pgort140.dll" Condition="$(Platform) == 'Win32'" /> + <_PGORT Include="$(VCToolsInstallDir)bin\Hostx64\x64\pgort140.dll" Condition="$(Platform) == 'x64'" /> + <_PGORT Include="$(VCToolsInstallDir)bin\arm64\pgort140.dll" Condition="$(Platform) == 'ARM64'" /> + + + + + + From webhook-mailer at python.org Thu Apr 7 19:27:30 2022 From: webhook-mailer at python.org (corona10) Date: Thu, 07 Apr 2022 23:27:30 -0000 Subject: [Python-checkins] bpo-47250: Fix refleak from object.__getstate__() (GH-32403) Message-ID: https://github.com/python/cpython/commit/e2d78baed385c349d756e96d8f0def0268fa9c4f commit: e2d78baed385c349d756e96d8f0def0268fa9c4f branch: main author: Dong-hee Na committer: corona10 date: 2022-04-08T08:27:00+09:00 summary: bpo-47250: Fix refleak from object.__getstate__() (GH-32403) Co-authored-by: Brandt Bucher files: M Objects/typeobject.c diff --git a/Objects/typeobject.c b/Objects/typeobject.c index 64c4bbb5a93260..f529e18134025b 100644 --- a/Objects/typeobject.c +++ b/Objects/typeobject.c @@ -5030,7 +5030,6 @@ object_getstate_default(PyObject *obj, int required) name = PyList_GET_ITEM(slotnames, i); Py_INCREF(name); - value = PyObject_GetAttr(obj, name); if (_PyObject_LookupAttr(obj, name, &value) < 0) { Py_DECREF(name); goto error; From webhook-mailer at python.org Thu Apr 7 20:03:07 2022 From: webhook-mailer at python.org (zooba) Date: Fri, 08 Apr 2022 00:03:07 -0000 Subject: [Python-checkins] bpo-47103: Copy pgort140.dll into output directory when building PGInstrument on Windows (GH-32083) Message-ID: https://github.com/python/cpython/commit/80c115385c01f456cdc6550543cf2112ae7a8161 commit: 80c115385c01f456cdc6550543cf2112ae7a8161 branch: 3.9 author: Steve Dower committer: zooba date: 2022-04-08T01:02:58+01:00 summary: bpo-47103: Copy pgort140.dll into output directory when building PGInstrument on Windows (GH-32083) files: A Misc/NEWS.d/next/Build/2022-03-23-20-01-16.bpo-47103.b4-00F.rst M Lib/test/test_embed.py M PCbuild/python.vcxproj diff --git a/Lib/test/test_embed.py b/Lib/test/test_embed.py index 11c8f012896d3..8e5cfa4824b15 100644 --- a/Lib/test/test_embed.py +++ b/Lib/test/test_embed.py @@ -1145,20 +1145,11 @@ def tmpdir_with_python(self): if MS_WINDOWS: # Copy pythonXY.dll (or pythonXY_d.dll) - ver = sys.version_info - dll = f'python{ver.major}{ver.minor}' - dll3 = f'python{ver.major}' - if debug_build(sys.executable): - dll += '_d' - dll3 += '_d' - dll += '.dll' - dll3 += '.dll' - dll = os.path.join(os.path.dirname(self.test_exe), dll) - dll3 = os.path.join(os.path.dirname(self.test_exe), dll3) - dll_copy = os.path.join(tmpdir, os.path.basename(dll)) - dll3_copy = os.path.join(tmpdir, os.path.basename(dll3)) - shutil.copyfile(dll, dll_copy) - shutil.copyfile(dll3, dll3_copy) + import fnmatch + exedir = os.path.dirname(self.test_exe) + for f in os.listdir(exedir): + if fnmatch.fnmatch(f, '*.dll'): + shutil.copyfile(os.path.join(exedir, f), os.path.join(tmpdir, f)) # Copy Python program exec_copy = os.path.join(tmpdir, os.path.basename(self.test_exe)) diff --git a/Misc/NEWS.d/next/Build/2022-03-23-20-01-16.bpo-47103.b4-00F.rst b/Misc/NEWS.d/next/Build/2022-03-23-20-01-16.bpo-47103.b4-00F.rst new file mode 100644 index 0000000000000..c1e01adce0d26 --- /dev/null +++ b/Misc/NEWS.d/next/Build/2022-03-23-20-01-16.bpo-47103.b4-00F.rst @@ -0,0 +1,2 @@ +Windows ``PGInstrument`` builds now copy a required DLL into the output +directory, making it easier to run the profile stage of a PGO build. diff --git a/PCbuild/python.vcxproj b/PCbuild/python.vcxproj index 2094420a8df39..42b27084e04d7 100644 --- a/PCbuild/python.vcxproj +++ b/PCbuild/python.vcxproj @@ -126,9 +126,6 @@ set PYTHONPATH=$(PySourcePath)Lib - <_PGOPath Condition="$(Configuration) == 'PGInstrument' and $(Platform) == 'Win32'">@set PATH=%PATH%%3B$(VCInstallDir)bin - <_PGOPath Condition="$(Configuration) == 'PGInstrument' and $(Platform) == 'x64'">@set PATH=%PATH%%3B$(VCInstallDir)bin\amd64 - <_PGOPath Condition="$(Configuration) == 'PGInstrument' and $(VC_PGO_RunTime_Dir) != ''">@set PATH=%PATH%%3B$(VC_PGO_RunTime_Dir) <_Content>@rem This script invokes the most recently built Python with all arguments @rem passed through to the interpreter. This file is generated by the @rem build process and any changes *will* be thrown away by the next @@ -138,7 +135,6 @@ set PYTHONPATH=$(PySourcePath)Lib @echo Running $(Configuration)^|$(Platform) interpreter... @setlocal @set PYTHONHOME=$(PySourcePath) -$(_PGOPath) @"$(OutDir)python$(PyDebugExt).exe" %* <_ExistingContent Condition="Exists('$(PySourcePath)python.bat')">$([System.IO.File]::ReadAllText('$(PySourcePath)python.bat')) @@ -164,4 +160,15 @@ $(_PGOPath) Overwrite="true" Lines="@(_LicenseFiles->'%(Content)')" /> + + + <_PGORT Include="$(VCToolsInstallDir)bin\Hostx86\x86\pgort140.dll" Condition="$(Platform) == 'Win32'" /> + <_PGORT Include="$(VCToolsInstallDir)bin\Hostx64\x64\pgort140.dll" Condition="$(Platform) == 'x64'" /> + <_PGORT Include="$(VCToolsInstallDir)bin\arm64\pgort140.dll" Condition="$(Platform) == 'ARM64'" /> + + + + + + From webhook-mailer at python.org Thu Apr 7 21:56:31 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Fri, 08 Apr 2022 01:56:31 -0000 Subject: [Python-checkins] pickle docs: Fix typos and improve wording (GH-24776) (GH-32395) Message-ID: https://github.com/python/cpython/commit/89192c46da7b984811ff3bd648f8e827e4ef053c commit: 89192c46da7b984811ff3bd648f8e827e4ef053c branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: JelleZijlstra date: 2022-04-07T18:56:17-07:00 summary: pickle docs: Fix typos and improve wording (GH-24776) (GH-32395) Co-authored-by: Jelle Zijlstra (cherry picked from commit 1d0f08fa46b54f5a9b43a916b66d50b97d56cf36) Co-authored-by: G?ry Ogam files: M Doc/library/pickle.rst diff --git a/Doc/library/pickle.rst b/Doc/library/pickle.rst index be48561ed10ac..f7db0e8415f4c 100644 --- a/Doc/library/pickle.rst +++ b/Doc/library/pickle.rst @@ -147,7 +147,7 @@ to read the pickle produced. earlier versions of Python. * Protocol version 2 was introduced in Python 2.3. It provides much more - efficient pickling of :term:`new-style class`\es. Refer to :pep:`307` for + efficient pickling of :term:`new-style classes `. Refer to :pep:`307` for information about improvements brought by protocol 2. * Protocol version 3 was added in Python 3.0. It has explicit support for @@ -261,7 +261,7 @@ process more convenient: protocol argument is needed. Bytes past the pickled representation of the object are ignored. - Arguments *file*, *fix_imports*, *encoding*, *errors*, *strict* and *buffers* + Arguments *fix_imports*, *encoding*, *errors*, *strict* and *buffers* have the same meaning as in the :class:`Unpickler` constructor. .. versionchanged:: 3.8 @@ -368,7 +368,7 @@ The :mod:`pickle` module exports three classes, :class:`Pickler`, .. versionadded:: 3.3 - .. method:: reducer_override(self, obj) + .. method:: reducer_override(obj) Special reducer that can be defined in :class:`Pickler` subclasses. This method has priority over any reducer in the :attr:`dispatch_table`. It @@ -494,20 +494,18 @@ What can be pickled and unpickled? The following types can be pickled: -* ``None``, ``True``, and ``False`` - -* integers, floating point numbers, complex numbers +* ``None``, ``True``, and ``False``; -* strings, bytes, bytearrays +* integers, floating-point numbers, complex numbers; -* tuples, lists, sets, and dictionaries containing only picklable objects +* strings, bytes, bytearrays; -* functions defined at the top level of a module (using :keyword:`def`, not - :keyword:`lambda`) +* tuples, lists, sets, and dictionaries containing only picklable objects; -* built-in functions defined at the top level of a module +* functions (built-in and user-defined) defined at the top level of a module + (using :keyword:`def`, not :keyword:`lambda`); -* classes that are defined at the top level of a module +* classes defined at the top level of a module; * instances of such classes whose :attr:`~object.__dict__` or the result of calling :meth:`__getstate__` is picklable (see section :ref:`pickle-inst` for @@ -520,14 +518,14 @@ structure may exceed the maximum recursion depth, a :exc:`RecursionError` will b raised in this case. You can carefully raise this limit with :func:`sys.setrecursionlimit`. -Note that functions (built-in and user-defined) are pickled by "fully qualified" -name reference, not by value. [#]_ This means that only the function name is +Note that functions (built-in and user-defined) are pickled by fully qualified +name, not by value. [#]_ This means that only the function name is pickled, along with the name of the module the function is defined in. Neither the function's code, nor any of its function attributes are pickled. Thus the defining module must be importable in the unpickling environment, and the module must contain the named object, otherwise an exception will be raised. [#]_ -Similarly, classes are pickled by named reference, so the same restrictions in +Similarly, classes are pickled by fully qualified name, so the same restrictions in the unpickling environment apply. Note that none of the class's code or data is pickled, so in the following example the class attribute ``attr`` is not restored in the unpickling environment:: @@ -537,7 +535,7 @@ restored in the unpickling environment:: picklestring = pickle.dumps(Foo) -These restrictions are why picklable functions and classes must be defined in +These restrictions are why picklable functions and classes must be defined at the top level of a module. Similarly, when class instances are pickled, their class's code and data are not @@ -569,7 +567,7 @@ implementation of this behaviour:: def save(obj): return (obj.__class__, obj.__dict__) - def load(cls, attributes): + def restore(cls, attributes): obj = cls.__new__(cls) obj.__dict__.update(attributes) return obj @@ -788,14 +786,15 @@ the code :: f = io.BytesIO() p = MyPickler(f) -does the same, but all instances of ``MyPickler`` will by default -share the same dispatch table. The equivalent code using the -:mod:`copyreg` module is :: +does the same but all instances of ``MyPickler`` will by default +share the private dispatch table. On the other hand, the code :: copyreg.pickle(SomeClass, reduce_SomeClass) f = io.BytesIO() p = pickle.Pickler(f) +modifies the global dispatch table shared by all users of the :mod:`copyreg` module. + .. _pickle-state: Handling Stateful Objects @@ -1098,7 +1097,7 @@ Here is an example of an unpickler allowing only few safe classes from the """Helper function analogous to pickle.loads().""" return RestrictedUnpickler(io.BytesIO(s)).load() -A sample usage of our unpickler working has intended:: +A sample usage of our unpickler working as intended:: >>> restricted_loads(pickle.dumps([1, 2, range(15)])) [1, 2, range(0, 15)] @@ -1142,7 +1141,7 @@ For the simplest code, use the :func:`dump` and :func:`load` functions. :: # An arbitrary collection of objects supported by pickle. data = { - 'a': [1, 2.0, 3, 4+6j], + 'a': [1, 2.0, 3+4j], 'b': ("character string", b"byte string"), 'c': {None, True, False} } @@ -1198,6 +1197,6 @@ The following example reads the resulting pickled data. :: operations. .. [#] The limitation on alphanumeric characters is due to the fact - the persistent IDs, in protocol 0, are delimited by the newline + that persistent IDs in protocol 0 are delimited by the newline character. Therefore if any kind of newline characters occurs in - persistent IDs, the resulting pickle will become unreadable. + persistent IDs, the resulting pickled data will become unreadable. From webhook-mailer at python.org Thu Apr 7 21:56:34 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Fri, 08 Apr 2022 01:56:34 -0000 Subject: [Python-checkins] pickle docs: Fix typos and improve wording (GH-24776) (GH-32396) Message-ID: https://github.com/python/cpython/commit/8cd0f274921233ebc645f96c2aaee2ed7a046b12 commit: 8cd0f274921233ebc645f96c2aaee2ed7a046b12 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: JelleZijlstra date: 2022-04-07T18:56:29-07:00 summary: pickle docs: Fix typos and improve wording (GH-24776) (GH-32396) Co-authored-by: Jelle Zijlstra (cherry picked from commit 1d0f08fa46b54f5a9b43a916b66d50b97d56cf36) Co-authored-by: G?ry Ogam files: M Doc/library/pickle.rst diff --git a/Doc/library/pickle.rst b/Doc/library/pickle.rst index be48561ed10ac..f7db0e8415f4c 100644 --- a/Doc/library/pickle.rst +++ b/Doc/library/pickle.rst @@ -147,7 +147,7 @@ to read the pickle produced. earlier versions of Python. * Protocol version 2 was introduced in Python 2.3. It provides much more - efficient pickling of :term:`new-style class`\es. Refer to :pep:`307` for + efficient pickling of :term:`new-style classes `. Refer to :pep:`307` for information about improvements brought by protocol 2. * Protocol version 3 was added in Python 3.0. It has explicit support for @@ -261,7 +261,7 @@ process more convenient: protocol argument is needed. Bytes past the pickled representation of the object are ignored. - Arguments *file*, *fix_imports*, *encoding*, *errors*, *strict* and *buffers* + Arguments *fix_imports*, *encoding*, *errors*, *strict* and *buffers* have the same meaning as in the :class:`Unpickler` constructor. .. versionchanged:: 3.8 @@ -368,7 +368,7 @@ The :mod:`pickle` module exports three classes, :class:`Pickler`, .. versionadded:: 3.3 - .. method:: reducer_override(self, obj) + .. method:: reducer_override(obj) Special reducer that can be defined in :class:`Pickler` subclasses. This method has priority over any reducer in the :attr:`dispatch_table`. It @@ -494,20 +494,18 @@ What can be pickled and unpickled? The following types can be pickled: -* ``None``, ``True``, and ``False`` - -* integers, floating point numbers, complex numbers +* ``None``, ``True``, and ``False``; -* strings, bytes, bytearrays +* integers, floating-point numbers, complex numbers; -* tuples, lists, sets, and dictionaries containing only picklable objects +* strings, bytes, bytearrays; -* functions defined at the top level of a module (using :keyword:`def`, not - :keyword:`lambda`) +* tuples, lists, sets, and dictionaries containing only picklable objects; -* built-in functions defined at the top level of a module +* functions (built-in and user-defined) defined at the top level of a module + (using :keyword:`def`, not :keyword:`lambda`); -* classes that are defined at the top level of a module +* classes defined at the top level of a module; * instances of such classes whose :attr:`~object.__dict__` or the result of calling :meth:`__getstate__` is picklable (see section :ref:`pickle-inst` for @@ -520,14 +518,14 @@ structure may exceed the maximum recursion depth, a :exc:`RecursionError` will b raised in this case. You can carefully raise this limit with :func:`sys.setrecursionlimit`. -Note that functions (built-in and user-defined) are pickled by "fully qualified" -name reference, not by value. [#]_ This means that only the function name is +Note that functions (built-in and user-defined) are pickled by fully qualified +name, not by value. [#]_ This means that only the function name is pickled, along with the name of the module the function is defined in. Neither the function's code, nor any of its function attributes are pickled. Thus the defining module must be importable in the unpickling environment, and the module must contain the named object, otherwise an exception will be raised. [#]_ -Similarly, classes are pickled by named reference, so the same restrictions in +Similarly, classes are pickled by fully qualified name, so the same restrictions in the unpickling environment apply. Note that none of the class's code or data is pickled, so in the following example the class attribute ``attr`` is not restored in the unpickling environment:: @@ -537,7 +535,7 @@ restored in the unpickling environment:: picklestring = pickle.dumps(Foo) -These restrictions are why picklable functions and classes must be defined in +These restrictions are why picklable functions and classes must be defined at the top level of a module. Similarly, when class instances are pickled, their class's code and data are not @@ -569,7 +567,7 @@ implementation of this behaviour:: def save(obj): return (obj.__class__, obj.__dict__) - def load(cls, attributes): + def restore(cls, attributes): obj = cls.__new__(cls) obj.__dict__.update(attributes) return obj @@ -788,14 +786,15 @@ the code :: f = io.BytesIO() p = MyPickler(f) -does the same, but all instances of ``MyPickler`` will by default -share the same dispatch table. The equivalent code using the -:mod:`copyreg` module is :: +does the same but all instances of ``MyPickler`` will by default +share the private dispatch table. On the other hand, the code :: copyreg.pickle(SomeClass, reduce_SomeClass) f = io.BytesIO() p = pickle.Pickler(f) +modifies the global dispatch table shared by all users of the :mod:`copyreg` module. + .. _pickle-state: Handling Stateful Objects @@ -1098,7 +1097,7 @@ Here is an example of an unpickler allowing only few safe classes from the """Helper function analogous to pickle.loads().""" return RestrictedUnpickler(io.BytesIO(s)).load() -A sample usage of our unpickler working has intended:: +A sample usage of our unpickler working as intended:: >>> restricted_loads(pickle.dumps([1, 2, range(15)])) [1, 2, range(0, 15)] @@ -1142,7 +1141,7 @@ For the simplest code, use the :func:`dump` and :func:`load` functions. :: # An arbitrary collection of objects supported by pickle. data = { - 'a': [1, 2.0, 3, 4+6j], + 'a': [1, 2.0, 3+4j], 'b': ("character string", b"byte string"), 'c': {None, True, False} } @@ -1198,6 +1197,6 @@ The following example reads the resulting pickled data. :: operations. .. [#] The limitation on alphanumeric characters is due to the fact - the persistent IDs, in protocol 0, are delimited by the newline + that persistent IDs in protocol 0 are delimited by the newline character. Therefore if any kind of newline characters occurs in - persistent IDs, the resulting pickle will become unreadable. + persistent IDs, the resulting pickled data will become unreadable. From webhook-mailer at python.org Thu Apr 7 22:10:04 2022 From: webhook-mailer at python.org (rhettinger) Date: Fri, 08 Apr 2022 02:10:04 -0000 Subject: [Python-checkins] Clarify that this sentence applies to the above example. GH-32405 Message-ID: https://github.com/python/cpython/commit/f4b328e2bbbcc1096a28e903f07868b425397767 commit: f4b328e2bbbcc1096a28e903f07868b425397767 branch: main author: Raymond Hettinger committer: rhettinger date: 2022-04-07T21:10:00-05:00 summary: Clarify that this sentence applies to the above example. GH-32405 files: M Doc/faq/programming.rst diff --git a/Doc/faq/programming.rst b/Doc/faq/programming.rst index a1adf851bdded..da011ccb29edb 100644 --- a/Doc/faq/programming.rst +++ b/Doc/faq/programming.rst @@ -1951,9 +1951,9 @@ relevant instance attributes are mutable, the *cached_property* approach can't be made to work because it cannot detect changes to the attributes. -The *lru_cache* approach can be made to work, but the class needs to define the -*__eq__* and *__hash__* methods so the cache can detect relevant attribute -updates:: +To make the *lru_cache* approach work when the *station_id* is mutable, +the class needs to define the *__eq__* and *__hash__* methods so that +the cache can detect relevant attribute updates:: class Weather: "Example with a mutable station identifier" From webhook-mailer at python.org Fri Apr 8 07:19:02 2022 From: webhook-mailer at python.org (markshannon) Date: Fri, 08 Apr 2022 11:19:02 -0000 Subject: [Python-checkins] Add new PyFrame_GetLasti C-API function (GH-32413) Message-ID: https://github.com/python/cpython/commit/5b4a4b6f0905c60514528b454af43aeea058b5a2 commit: 5b4a4b6f0905c60514528b454af43aeea058b5a2 branch: main author: Mark Shannon committer: markshannon date: 2022-04-08T12:18:57+01:00 summary: Add new PyFrame_GetLasti C-API function (GH-32413) files: A Misc/NEWS.d/next/C API/2022-04-08-11-29-36.bpo-40421.H0ORmT.rst M Doc/c-api/frame.rst M Doc/whatsnew/3.11.rst M Include/cpython/frameobject.h M Lib/test/test_capi.py M Modules/_testcapimodule.c M Objects/frameobject.c diff --git a/Doc/c-api/frame.rst b/Doc/c-api/frame.rst index f6c682c1e6c1c..68e5dc6daeaec 100644 --- a/Doc/c-api/frame.rst +++ b/Doc/c-api/frame.rst @@ -78,6 +78,17 @@ See also :ref:`Reflection `. .. versionadded:: 3.11 +.. c:function:: int PyFrame_GetLasti(PyFrameObject *frame) + + Get the *frame*'s ``f_lasti`` attribute (:class:`dict`). + + Returns -1 if ``frame.f_lasti`` is ``None``. + + *frame* must not be ``NULL``. + + .. versionadded:: 3.11 + + .. c:function:: PyObject* PyFrame_GetLocals(PyFrameObject *frame) Get the *frame*'s ``f_locals`` attribute (:class:`dict`). diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index bc4a1953c10c2..2da01d8105ac6 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -1121,7 +1121,7 @@ New Features * Add new functions to get frame object attributes: :c:func:`PyFrame_GetBuiltins`, :c:func:`PyFrame_GetGenerator`, - :c:func:`PyFrame_GetGlobals`. + :c:func:`PyFrame_GetGlobals`, :c:func:`PyFrame_GetLasti`. Porting to Python 3.11 ---------------------- @@ -1246,9 +1246,9 @@ Porting to Python 3.11 * ``f_gen``: use :c:func:`PyFrame_GetGenerator`. * ``f_globals``: use :c:func:`PyFrame_GetGlobals`. * ``f_iblock``: removed. - * ``f_lasti``: use ``PyObject_GetAttrString((PyObject*)frame, "f_lasti")``. + * ``f_lasti``: use :c:func:`PyFrame_GetLasti`. Code using ``f_lasti`` with ``PyCode_Addr2Line()`` should use - :c:func:`PyFrame_GetLineNumber` instead. + :c:func:`PyFrame_GetLineNumber` instead; it may be faster. * ``f_lineno``: use :c:func:`PyFrame_GetLineNumber` * ``f_locals``: use :c:func:`PyFrame_GetLocals`. * ``f_stackdepth``: removed. diff --git a/Include/cpython/frameobject.h b/Include/cpython/frameobject.h index ffeb8bd04a46f..01cf6c9b89ae4 100644 --- a/Include/cpython/frameobject.h +++ b/Include/cpython/frameobject.h @@ -29,3 +29,4 @@ PyAPI_FUNC(PyObject *) PyFrame_GetGlobals(PyFrameObject *frame); PyAPI_FUNC(PyObject *) PyFrame_GetBuiltins(PyFrameObject *frame); PyAPI_FUNC(PyObject *) PyFrame_GetGenerator(PyFrameObject *frame); +PyAPI_FUNC(int) PyFrame_GetLasti(PyFrameObject *frame); diff --git a/Lib/test/test_capi.py b/Lib/test/test_capi.py index 3837f801b3c73..40e4774c6b8ed 100644 --- a/Lib/test/test_capi.py +++ b/Lib/test/test_capi.py @@ -1102,6 +1102,7 @@ def test_frame_getters(self): self.assertEqual(frame.f_locals, _testcapi.frame_getlocals(frame)) self.assertIs(frame.f_globals, _testcapi.frame_getglobals(frame)) self.assertIs(frame.f_builtins, _testcapi.frame_getbuiltins(frame)) + self.assertEqual(frame.f_lasti, _testcapi.frame_getlasti(frame)) def test_frame_get_generator(self): gen = self.getgenframe() diff --git a/Misc/NEWS.d/next/C API/2022-04-08-11-29-36.bpo-40421.H0ORmT.rst b/Misc/NEWS.d/next/C API/2022-04-08-11-29-36.bpo-40421.H0ORmT.rst new file mode 100644 index 0000000000000..2e10c2394c933 --- /dev/null +++ b/Misc/NEWS.d/next/C API/2022-04-08-11-29-36.bpo-40421.H0ORmT.rst @@ -0,0 +1,2 @@ +Add ``PyFrame_GetLasti`` C-API function to access frame object's ``lasti`` +attribute safely from C code. diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c index 759656ae1f8a1..13dd29427aa2c 100644 --- a/Modules/_testcapimodule.c +++ b/Modules/_testcapimodule.c @@ -5893,6 +5893,21 @@ frame_getbuiltins(PyObject *self, PyObject *frame) return PyFrame_GetBuiltins((PyFrameObject *)frame); } +static PyObject * +frame_getlasti(PyObject *self, PyObject *frame) +{ + if (!PyFrame_Check(frame)) { + PyErr_SetString(PyExc_TypeError, "argument must be a frame"); + return NULL; + } + int lasti = PyFrame_GetLasti((PyFrameObject *)frame); + if (lasti < 0) { + assert(lasti == -1); + Py_RETURN_NONE; + } + return PyLong_FromLong(lasti); +} + static PyObject *negative_dictoffset(PyObject *, PyObject *); static PyObject *test_buildvalue_issue38913(PyObject *, PyObject *); @@ -6186,6 +6201,7 @@ static PyMethodDef TestMethods[] = { {"frame_getglobals", frame_getglobals, METH_O, NULL}, {"frame_getgenerator", frame_getgenerator, METH_O, NULL}, {"frame_getbuiltins", frame_getbuiltins, METH_O, NULL}, + {"frame_getlasti", frame_getlasti, METH_O, NULL}, {NULL, NULL} /* sentinel */ }; diff --git a/Objects/frameobject.c b/Objects/frameobject.c index 07b610717d2a2..5bb8937009897 100644 --- a/Objects/frameobject.c +++ b/Objects/frameobject.c @@ -880,7 +880,7 @@ _PyFrame_OpAlreadyRan(_PyInterpreterFrame *frame, int opcode, int oparg) // This only works when opcode is a non-quickened form: assert(_PyOpcode_Deopt[opcode] == opcode); int check_oparg = 0; - for (_Py_CODEUNIT *instruction = _PyCode_CODE(frame->f_code); + for (_Py_CODEUNIT *instruction = _PyCode_CODE(frame->f_code); instruction < frame->prev_instr; instruction++) { int check_opcode = _PyOpcode_Deopt[_Py_OPCODE(*instruction)]; @@ -1135,6 +1135,16 @@ PyFrame_GetBuiltins(PyFrameObject *frame) return frame_getbuiltins(frame, NULL); } +int +PyFrame_GetLasti(PyFrameObject *frame) +{ + int lasti = _PyInterpreterFrame_LASTI(frame->f_frame); + if (lasti < 0) { + return -1; + } + return lasti*2; +} + PyObject * PyFrame_GetGenerator(PyFrameObject *frame) { From webhook-mailer at python.org Fri Apr 8 08:35:28 2022 From: webhook-mailer at python.org (encukou) Date: Fri, 08 Apr 2022 12:35:28 -0000 Subject: [Python-checkins] Add feature macro PY_HAVE_THREAD_NATIVE_ID to the stable ABI definition (GH-32365) Message-ID: https://github.com/python/cpython/commit/1c2fddddae5ce27be354b519f8ae51886b4dd9f4 commit: 1c2fddddae5ce27be354b519f8ae51886b4dd9f4 branch: main author: Petr Viktorin committer: encukou date: 2022-04-08T14:35:11+02:00 summary: Add feature macro PY_HAVE_THREAD_NATIVE_ID to the stable ABI definition (GH-32365) files: A Misc/NEWS.d/next/C API/2022-04-06-16-29-14.bpo-47169.wVv2bT.rst M Doc/data/stable_abi.dat M Lib/test/test_stable_abi_ctypes.py M Misc/stable_abi.txt M Tools/scripts/stable_abi.py diff --git a/Doc/data/stable_abi.dat b/Doc/data/stable_abi.dat index 7f42f9c12de71..849a2cfd51f24 100644 --- a/Doc/data/stable_abi.dat +++ b/Doc/data/stable_abi.dat @@ -625,7 +625,7 @@ function,PyThread_free_lock,3.2,, function,PyThread_get_key_value,3.2,, function,PyThread_get_stacksize,3.2,, function,PyThread_get_thread_ident,3.2,, -function,PyThread_get_thread_native_id,3.2,, +function,PyThread_get_thread_native_id,3.2,on platforms with native thread IDs, function,PyThread_init_thread,3.2,, function,PyThread_release_lock,3.2,, function,PyThread_set_key_value,3.2,, diff --git a/Lib/test/test_stable_abi_ctypes.py b/Lib/test/test_stable_abi_ctypes.py index a49235b81c1b0..efd3b1b7cd2d2 100644 --- a/Lib/test/test_stable_abi_ctypes.py +++ b/Lib/test/test_stable_abi_ctypes.py @@ -615,7 +615,6 @@ def test_available_symbols(self): "PyThread_get_key_value", "PyThread_get_stacksize", "PyThread_get_thread_ident", - "PyThread_get_thread_native_id", "PyThread_init_thread", "PyThread_release_lock", "PyThread_set_key_value", diff --git a/Misc/NEWS.d/next/C API/2022-04-06-16-29-14.bpo-47169.wVv2bT.rst b/Misc/NEWS.d/next/C API/2022-04-06-16-29-14.bpo-47169.wVv2bT.rst new file mode 100644 index 0000000000000..66eac052391e5 --- /dev/null +++ b/Misc/NEWS.d/next/C API/2022-04-06-16-29-14.bpo-47169.wVv2bT.rst @@ -0,0 +1,2 @@ +:c:func:`PyThread_get_thread_native_id` is excluded from the stable ABI on +platforms where it doesn't exist (like Solaris). diff --git a/Misc/stable_abi.txt b/Misc/stable_abi.txt index 04d22603cc19b..4864bf319a76f 100644 --- a/Misc/stable_abi.txt +++ b/Misc/stable_abi.txt @@ -1787,6 +1787,7 @@ function PyThread_get_stacksize function PyThread_get_thread_ident added 3.2 function PyThread_get_thread_native_id + ifdef PY_HAVE_THREAD_NATIVE_ID added 3.2 function PyThread_init_thread added 3.2 diff --git a/Tools/scripts/stable_abi.py b/Tools/scripts/stable_abi.py index feca9a2ba5a4c..9b90e344977f4 100755 --- a/Tools/scripts/stable_abi.py +++ b/Tools/scripts/stable_abi.py @@ -49,8 +49,17 @@ 'MS_WINDOWS': 'on Windows', 'HAVE_FORK': 'on platforms with fork()', 'USE_STACKCHECK': 'on platforms with USE_STACKCHECK', + 'PY_HAVE_THREAD_NATIVE_ID': 'on platforms with native thread IDs', } +# To generate the DLL definition, we need to know which feature macros are +# defined on Windows. On all platforms. +# Best way to do that is to hardcode the list (and later test in on Windows). +WINDOWS_IFDEFS = frozenset({ + 'MS_WINDOWS', + 'PY_HAVE_THREAD_NATIVE_ID', +}) + # The stable ABI manifest (Misc/stable_abi.txt) exists only to fill the # following dataclasses. # Feel free to change its syntax (and the `parse_manifest` function) @@ -232,7 +241,7 @@ def sort_key(item): for item in sorted( manifest.select( - {'function'}, include_abi_only=True, ifdef={'MS_WINDOWS'}), + {'function'}, include_abi_only=True, ifdef=WINDOWS_IFDEFS), key=sort_key): write(f'EXPORT_FUNC({item.name})') @@ -240,7 +249,7 @@ def sort_key(item): for item in sorted( manifest.select( - {'data'}, include_abi_only=True, ifdef={'MS_WINDOWS'}), + {'data'}, include_abi_only=True, ifdef=WINDOWS_IFDEFS), key=sort_key): write(f'EXPORT_DATA({item.name})') From webhook-mailer at python.org Fri Apr 8 09:06:25 2022 From: webhook-mailer at python.org (ambv) Date: Fri, 08 Apr 2022 13:06:25 -0000 Subject: [Python-checkins] [3.8] bpo-47138: Fix documentation build by pinning Jinja version to 3.0.3 (GH-32109) Message-ID: https://github.com/python/cpython/commit/d35af52caae844cb4ea0aff06fa3fc5328708af1 commit: d35af52caae844cb4ea0aff06fa3fc5328708af1 branch: 3.8 author: m-aciek committer: ambv date: 2022-04-08T15:06:17+02:00 summary: [3.8] bpo-47138: Fix documentation build by pinning Jinja version to 3.0.3 (GH-32109) Co-authored-by: Ned Deily files: A Misc/NEWS.d/next/Documentation/2022-03-28-12-36-17.bpo-47138.TbLXgV.rst M Doc/requirements.txt diff --git a/Doc/requirements.txt b/Doc/requirements.txt index cb21ed20397b2..aa615e6bf3141 100644 --- a/Doc/requirements.txt +++ b/Doc/requirements.txt @@ -8,6 +8,8 @@ sphinx==2.4.4 # version 2.4.4. It can be removed after bumping Sphinx version to at # least 3.5.4. docutils==0.17.1 +# Jinja version is pinned to a version compatible with Sphinx version 2.4.4. +jinja2==3.0.3 blurb diff --git a/Misc/NEWS.d/next/Documentation/2022-03-28-12-36-17.bpo-47138.TbLXgV.rst b/Misc/NEWS.d/next/Documentation/2022-03-28-12-36-17.bpo-47138.TbLXgV.rst new file mode 100644 index 0000000000000..9dde4de2dc8b5 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2022-03-28-12-36-17.bpo-47138.TbLXgV.rst @@ -0,0 +1 @@ +Pin Jinja to a version compatible with Sphinx version 2.4.4. From webhook-mailer at python.org Fri Apr 8 09:36:28 2022 From: webhook-mailer at python.org (rhettinger) Date: Fri, 08 Apr 2022 13:36:28 -0000 Subject: [Python-checkins] Fix bad grammar and import docstring for split/rsplit (GH-32381) Message-ID: https://github.com/python/cpython/commit/d6fb104690cdeeea04ecbaf5c9bcafc622e03648 commit: d6fb104690cdeeea04ecbaf5c9bcafc622e03648 branch: main author: Raymond Hettinger committer: rhettinger date: 2022-04-08T08:36:20-05:00 summary: Fix bad grammar and import docstring for split/rsplit (GH-32381) files: M Objects/clinic/unicodeobject.c.h M Objects/unicodeobject.c diff --git a/Objects/clinic/unicodeobject.c.h b/Objects/clinic/unicodeobject.c.h index 9ef8ce2e35364..803b5f2353f9e 100644 --- a/Objects/clinic/unicodeobject.c.h +++ b/Objects/clinic/unicodeobject.c.h @@ -858,15 +858,21 @@ PyDoc_STRVAR(unicode_split__doc__, "split($self, /, sep=None, maxsplit=-1)\n" "--\n" "\n" -"Return a list of the words in the string, using sep as the delimiter string.\n" +"Return a list of the substrings in the string, using sep as the separator string.\n" "\n" " sep\n" -" The delimiter according which to split the string.\n" -" None (the default value) means split according to any whitespace,\n" -" and discard empty strings from the result.\n" +" The separator used to split the string.\n" +"\n" +" When set to None (the default value), will split on any whitespace\n" +" character (including \\\\n \\\\r \\\\t \\\\f and spaces) and will discard\n" +" empty strings from the result.\n" " maxsplit\n" -" Maximum number of splits to do.\n" -" -1 (the default value) means no limit."); +" Maximum number of splits (starting from the left).\n" +" -1 (the default value) means no limit.\n" +"\n" +"Note, str.split() is mainly useful for data that has been intentionally\n" +"delimited. With natural text that includes punctuation, consider using\n" +"the regular expression module."); #define UNICODE_SPLIT_METHODDEF \ {"split", (PyCFunction)(void(*)(void))unicode_split, METH_FASTCALL|METH_KEYWORDS, unicode_split__doc__}, @@ -953,17 +959,19 @@ PyDoc_STRVAR(unicode_rsplit__doc__, "rsplit($self, /, sep=None, maxsplit=-1)\n" "--\n" "\n" -"Return a list of the words in the string, using sep as the delimiter string.\n" +"Return a list of the substrings in the string, using sep as the separator string.\n" "\n" " sep\n" -" The delimiter according which to split the string.\n" -" None (the default value) means split according to any whitespace,\n" -" and discard empty strings from the result.\n" +" The separator used to split the string.\n" +"\n" +" When set to None (the default value), will split on any whitespace\n" +" character (including \\\\n \\\\r \\\\t \\\\f and spaces) and will discard\n" +" empty strings from the result.\n" " maxsplit\n" -" Maximum number of splits to do.\n" +" Maximum number of splits (starting from the left).\n" " -1 (the default value) means no limit.\n" "\n" -"Splits are done starting at the end of the string and working to the front."); +"Splitting starts at the end of the string and works to the front."); #define UNICODE_RSPLIT_METHODDEF \ {"rsplit", (PyCFunction)(void(*)(void))unicode_rsplit, METH_FASTCALL|METH_KEYWORDS, unicode_rsplit__doc__}, @@ -1327,4 +1335,4 @@ unicode_new(PyTypeObject *type, PyObject *args, PyObject *kwargs) exit: return return_value; } -/*[clinic end generated code: output=f10cf85d3935b3b7 input=a9049054013a1b77]*/ +/*[clinic end generated code: output=c494bed46209961d input=a9049054013a1b77]*/ diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c index 2d4096397784f..1b2f33212a779 100644 --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -13155,19 +13155,26 @@ PyUnicode_Split(PyObject *s, PyObject *sep, Py_ssize_t maxsplit) str.split as unicode_split sep: object = None - The delimiter according which to split the string. - None (the default value) means split according to any whitespace, - and discard empty strings from the result. + The separator used to split the string. + + When set to None (the default value), will split on any whitespace + character (including \\n \\r \\t \\f and spaces) and will discard + empty strings from the result. maxsplit: Py_ssize_t = -1 - Maximum number of splits to do. + Maximum number of splits (starting from the left). -1 (the default value) means no limit. -Return a list of the words in the string, using sep as the delimiter string. +Return a list of the substrings in the string, using sep as the separator string. + +Note, str.split() is mainly useful for data that has been intentionally +delimited. With natural text that includes punctuation, consider using +the regular expression module. + [clinic start generated code]*/ static PyObject * unicode_split_impl(PyObject *self, PyObject *sep, Py_ssize_t maxsplit) -/*[clinic end generated code: output=3a65b1db356948dc input=606e750488a82359]*/ +/*[clinic end generated code: output=3a65b1db356948dc input=906d953b44efc43b]*/ { if (sep == Py_None) return split(self, NULL, maxsplit); @@ -13338,14 +13345,14 @@ PyUnicode_RSplit(PyObject *s, PyObject *sep, Py_ssize_t maxsplit) /*[clinic input] str.rsplit as unicode_rsplit = str.split -Return a list of the words in the string, using sep as the delimiter string. +Return a list of the substrings in the string, using sep as the separator string. -Splits are done starting at the end of the string and working to the front. +Splitting starts at the end of the string and works to the front. [clinic start generated code]*/ static PyObject * unicode_rsplit_impl(PyObject *self, PyObject *sep, Py_ssize_t maxsplit) -/*[clinic end generated code: output=c2b815c63bcabffc input=12ad4bf57dd35f15]*/ +/*[clinic end generated code: output=c2b815c63bcabffc input=ea78406060fce33c]*/ { if (sep == Py_None) return rsplit(self, NULL, maxsplit); From webhook-mailer at python.org Fri Apr 8 13:06:37 2022 From: webhook-mailer at python.org (rhettinger) Date: Fri, 08 Apr 2022 17:06:37 -0000 Subject: [Python-checkins] Fix bad grammar and import docstring for split/rsplit (GH-32381) (GH-32416) Message-ID: https://github.com/python/cpython/commit/69edc30d2b47fe9b95975b1b66214e7473a9ccf5 commit: 69edc30d2b47fe9b95975b1b66214e7473a9ccf5 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: rhettinger date: 2022-04-08T12:06:19-05:00 summary: Fix bad grammar and import docstring for split/rsplit (GH-32381) (GH-32416) files: M Objects/clinic/unicodeobject.c.h M Objects/unicodeobject.c diff --git a/Objects/clinic/unicodeobject.c.h b/Objects/clinic/unicodeobject.c.h index 9ef8ce2e35364..803b5f2353f9e 100644 --- a/Objects/clinic/unicodeobject.c.h +++ b/Objects/clinic/unicodeobject.c.h @@ -858,15 +858,21 @@ PyDoc_STRVAR(unicode_split__doc__, "split($self, /, sep=None, maxsplit=-1)\n" "--\n" "\n" -"Return a list of the words in the string, using sep as the delimiter string.\n" +"Return a list of the substrings in the string, using sep as the separator string.\n" "\n" " sep\n" -" The delimiter according which to split the string.\n" -" None (the default value) means split according to any whitespace,\n" -" and discard empty strings from the result.\n" +" The separator used to split the string.\n" +"\n" +" When set to None (the default value), will split on any whitespace\n" +" character (including \\\\n \\\\r \\\\t \\\\f and spaces) and will discard\n" +" empty strings from the result.\n" " maxsplit\n" -" Maximum number of splits to do.\n" -" -1 (the default value) means no limit."); +" Maximum number of splits (starting from the left).\n" +" -1 (the default value) means no limit.\n" +"\n" +"Note, str.split() is mainly useful for data that has been intentionally\n" +"delimited. With natural text that includes punctuation, consider using\n" +"the regular expression module."); #define UNICODE_SPLIT_METHODDEF \ {"split", (PyCFunction)(void(*)(void))unicode_split, METH_FASTCALL|METH_KEYWORDS, unicode_split__doc__}, @@ -953,17 +959,19 @@ PyDoc_STRVAR(unicode_rsplit__doc__, "rsplit($self, /, sep=None, maxsplit=-1)\n" "--\n" "\n" -"Return a list of the words in the string, using sep as the delimiter string.\n" +"Return a list of the substrings in the string, using sep as the separator string.\n" "\n" " sep\n" -" The delimiter according which to split the string.\n" -" None (the default value) means split according to any whitespace,\n" -" and discard empty strings from the result.\n" +" The separator used to split the string.\n" +"\n" +" When set to None (the default value), will split on any whitespace\n" +" character (including \\\\n \\\\r \\\\t \\\\f and spaces) and will discard\n" +" empty strings from the result.\n" " maxsplit\n" -" Maximum number of splits to do.\n" +" Maximum number of splits (starting from the left).\n" " -1 (the default value) means no limit.\n" "\n" -"Splits are done starting at the end of the string and working to the front."); +"Splitting starts at the end of the string and works to the front."); #define UNICODE_RSPLIT_METHODDEF \ {"rsplit", (PyCFunction)(void(*)(void))unicode_rsplit, METH_FASTCALL|METH_KEYWORDS, unicode_rsplit__doc__}, @@ -1327,4 +1335,4 @@ unicode_new(PyTypeObject *type, PyObject *args, PyObject *kwargs) exit: return return_value; } -/*[clinic end generated code: output=f10cf85d3935b3b7 input=a9049054013a1b77]*/ +/*[clinic end generated code: output=c494bed46209961d input=a9049054013a1b77]*/ diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c index 377fa6c8e2a28..38a7b3c8b44b9 100644 --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -13478,19 +13478,26 @@ PyUnicode_Split(PyObject *s, PyObject *sep, Py_ssize_t maxsplit) str.split as unicode_split sep: object = None - The delimiter according which to split the string. - None (the default value) means split according to any whitespace, - and discard empty strings from the result. + The separator used to split the string. + + When set to None (the default value), will split on any whitespace + character (including \\n \\r \\t \\f and spaces) and will discard + empty strings from the result. maxsplit: Py_ssize_t = -1 - Maximum number of splits to do. + Maximum number of splits (starting from the left). -1 (the default value) means no limit. -Return a list of the words in the string, using sep as the delimiter string. +Return a list of the substrings in the string, using sep as the separator string. + +Note, str.split() is mainly useful for data that has been intentionally +delimited. With natural text that includes punctuation, consider using +the regular expression module. + [clinic start generated code]*/ static PyObject * unicode_split_impl(PyObject *self, PyObject *sep, Py_ssize_t maxsplit) -/*[clinic end generated code: output=3a65b1db356948dc input=606e750488a82359]*/ +/*[clinic end generated code: output=3a65b1db356948dc input=906d953b44efc43b]*/ { if (sep == Py_None) return split(self, NULL, maxsplit); @@ -13661,14 +13668,14 @@ PyUnicode_RSplit(PyObject *s, PyObject *sep, Py_ssize_t maxsplit) /*[clinic input] str.rsplit as unicode_rsplit = str.split -Return a list of the words in the string, using sep as the delimiter string. +Return a list of the substrings in the string, using sep as the separator string. -Splits are done starting at the end of the string and working to the front. +Splitting starts at the end of the string and works to the front. [clinic start generated code]*/ static PyObject * unicode_rsplit_impl(PyObject *self, PyObject *sep, Py_ssize_t maxsplit) -/*[clinic end generated code: output=c2b815c63bcabffc input=12ad4bf57dd35f15]*/ +/*[clinic end generated code: output=c2b815c63bcabffc input=ea78406060fce33c]*/ { if (sep == Py_None) return rsplit(self, NULL, maxsplit); From webhook-mailer at python.org Fri Apr 8 13:40:57 2022 From: webhook-mailer at python.org (gpshead) Date: Fri, 08 Apr 2022 17:40:57 -0000 Subject: [Python-checkins] bpo-47260: Fix os.closerange() potentially being a no-op in a seccomp sandbox (GH-32418) Message-ID: https://github.com/python/cpython/commit/1c8b3b5d66a629258f1db16939b996264a8b9c37 commit: 1c8b3b5d66a629258f1db16939b996264a8b9c37 branch: main author: Alexey Izbyshev committer: gpshead date: 2022-04-08T10:40:39-07:00 summary: bpo-47260: Fix os.closerange() potentially being a no-op in a seccomp sandbox (GH-32418) _Py_closerange() currently assumes that close_range() closes all file descriptors even if it returns an error (other than ENOSYS). This assumption can be wrong on Linux if a seccomp sandbox denies the underlying syscall, pretending that it returns EPERM or EACCES. In this case _Py_closerange() won't close any descriptors at all, which in the worst case can be a security issue. Fix this by falling back to other methods in case of any close_range() error. Note that fallbacks will not be triggered on any problems with closing individual file descriptors because close_range() is documented to ignore such errors on both Linux[1] and FreeBSD[2]. [1] https://man7.org/linux/man-pages/man2/close_range.2.html [2] https://www.freebsd.org/cgi/man.cgi?query=close_range&sektion=2 files: A Misc/NEWS.d/next/Library/2022-04-08-14-30-53.bpo-47260.TtcNxI.rst M Python/fileutils.c diff --git a/Misc/NEWS.d/next/Library/2022-04-08-14-30-53.bpo-47260.TtcNxI.rst b/Misc/NEWS.d/next/Library/2022-04-08-14-30-53.bpo-47260.TtcNxI.rst new file mode 100644 index 0000000000000..300baa19c279a --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-08-14-30-53.bpo-47260.TtcNxI.rst @@ -0,0 +1,2 @@ +Fix ``os.closerange()`` potentially being a no-op in a Linux seccomp +sandbox. diff --git a/Python/fileutils.c b/Python/fileutils.c index 111d7fa84b188..d1d62dce5daa4 100644 --- a/Python/fileutils.c +++ b/Python/fileutils.c @@ -2624,10 +2624,11 @@ _Py_closerange(int first, int last) first = Py_MAX(first, 0); _Py_BEGIN_SUPPRESS_IPH #ifdef HAVE_CLOSE_RANGE - if (close_range(first, last, 0) == 0 || errno != ENOSYS) { - /* Any errors encountered while closing file descriptors are ignored; - * ENOSYS means no kernel support, though, - * so we'll fallback to the other methods. */ + if (close_range(first, last, 0) == 0) { + /* close_range() ignores errors when it closes file descriptors. + * Possible reasons of an error return are lack of kernel support + * or denial of the underlying syscall by a seccomp sandbox on Linux. + * Fallback to other methods in case of any error. */ } else #endif /* HAVE_CLOSE_RANGE */ From webhook-mailer at python.org Fri Apr 8 14:10:43 2022 From: webhook-mailer at python.org (miss-islington) Date: Fri, 08 Apr 2022 18:10:43 -0000 Subject: [Python-checkins] bpo-47260: Fix os.closerange() potentially being a no-op in a seccomp sandbox (GH-32418) Message-ID: https://github.com/python/cpython/commit/89697f7374ea947ebe8e36131e2d3e21fff6fa1d commit: 89697f7374ea947ebe8e36131e2d3e21fff6fa1d branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-08T11:10:38-07:00 summary: bpo-47260: Fix os.closerange() potentially being a no-op in a seccomp sandbox (GH-32418) _Py_closerange() currently assumes that close_range() closes all file descriptors even if it returns an error (other than ENOSYS). This assumption can be wrong on Linux if a seccomp sandbox denies the underlying syscall, pretending that it returns EPERM or EACCES. In this case _Py_closerange() won't close any descriptors at all, which in the worst case can be a security issue. Fix this by falling back to other methods in case of any close_range() error. Note that fallbacks will not be triggered on any problems with closing individual file descriptors because close_range() is documented to ignore such errors on both Linux[1] and FreeBSD[2]. [1] https://man7.org/linux/man-pages/man2/close_range.2.html [2] https://www.freebsd.org/cgi/man.cgi?query=close_range&sektion=2 (cherry picked from commit 1c8b3b5d66a629258f1db16939b996264a8b9c37) Co-authored-by: Alexey Izbyshev files: A Misc/NEWS.d/next/Library/2022-04-08-14-30-53.bpo-47260.TtcNxI.rst M Python/fileutils.c diff --git a/Misc/NEWS.d/next/Library/2022-04-08-14-30-53.bpo-47260.TtcNxI.rst b/Misc/NEWS.d/next/Library/2022-04-08-14-30-53.bpo-47260.TtcNxI.rst new file mode 100644 index 0000000000000..300baa19c279a --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-08-14-30-53.bpo-47260.TtcNxI.rst @@ -0,0 +1,2 @@ +Fix ``os.closerange()`` potentially being a no-op in a Linux seccomp +sandbox. diff --git a/Python/fileutils.c b/Python/fileutils.c index c3144ee40782e..3b53baa00eeb1 100644 --- a/Python/fileutils.c +++ b/Python/fileutils.c @@ -2395,10 +2395,11 @@ _Py_closerange(int first, int last) first = Py_MAX(first, 0); _Py_BEGIN_SUPPRESS_IPH #ifdef HAVE_CLOSE_RANGE - if (close_range(first, last, 0) == 0 || errno != ENOSYS) { - /* Any errors encountered while closing file descriptors are ignored; - * ENOSYS means no kernel support, though, - * so we'll fallback to the other methods. */ + if (close_range(first, last, 0) == 0) { + /* close_range() ignores errors when it closes file descriptors. + * Possible reasons of an error return are lack of kernel support + * or denial of the underlying syscall by a seccomp sandbox on Linux. + * Fallback to other methods in case of any error. */ } else #endif /* HAVE_CLOSE_RANGE */ From webhook-mailer at python.org Fri Apr 8 20:15:46 2022 From: webhook-mailer at python.org (brettcannon) Date: Sat, 09 Apr 2022 00:15:46 -0000 Subject: [Python-checkins] bpo-47061: deprecate cgi and cgitb (GH-32410) Message-ID: https://github.com/python/cpython/commit/cd29bd13ef1fe18970c5d43b66c545dd03117cb9 commit: cd29bd13ef1fe18970c5d43b66c545dd03117cb9 branch: main author: Brett Cannon committer: brettcannon date: 2022-04-08T17:15:35-07:00 summary: bpo-47061: deprecate cgi and cgitb (GH-32410) Part of PEP 594. files: A Misc/NEWS.d/next/Library/2022-04-07-20-32-47.bpo-47061.TOufgh.rst M Doc/whatsnew/3.11.rst M Lib/cgi.py M Lib/cgitb.py M Lib/distutils/config.py M Lib/test/test_cgi.py M Lib/test/test_cgitb.py M Lib/test/test_httpservers.py diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index 2da01d8105ac6..2758a268e957e 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -834,6 +834,8 @@ Deprecated * :mod:`aifc` * :mod:`audioop` + * :mod:`cgi` + * :mod:`cgitb` (Contributed by Brett Cannon in :issue:`47061`.) diff --git a/Lib/cgi.py b/Lib/cgi.py index 22897a14a9c12..8787567be7c08 100755 --- a/Lib/cgi.py +++ b/Lib/cgi.py @@ -53,6 +53,9 @@ "print_form", "print_directory", "print_arguments", "print_environ_usage"] + +warnings._deprecated(__name__, remove=(3,13)) + # Logging support # =============== diff --git a/Lib/cgitb.py b/Lib/cgitb.py index ec156843099d3..8ce0e833a989a 100644 --- a/Lib/cgitb.py +++ b/Lib/cgitb.py @@ -31,8 +31,12 @@ import time import tokenize import traceback +import warnings from html import escape as html_escape +warnings._deprecated(__name__, remove=(3, 13)) + + def reset(): """Return a string that resets the CGI and browser to a known state.""" return ''' POP_JUMP_IF_TRUE to non-jump def f(a, b, c): return ((a or b) @@ -414,7 +415,7 @@ def f(a, b, c): self.check_lnotab(f) self.assertNotInBytecode(f, 'JUMP_IF_TRUE_OR_POP') self.assertInBytecode(f, 'JUMP_IF_FALSE_OR_POP') - self.assertInBytecode(f, 'POP_JUMP_IF_TRUE') + self.assertInBytecode(f, 'POP_JUMP_FORWARD_IF_TRUE') def test_elim_jump_after_return1(self): # Eliminate dead code: jumps immediately after returns can't be reached diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-04-06-22-50-31.bpo-47120.mbfHs5.rst b/Misc/NEWS.d/next/Core and Builtins/2022-04-06-22-50-31.bpo-47120.mbfHs5.rst new file mode 100644 index 0000000000000..3afe8c2bb6552 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2022-04-06-22-50-31.bpo-47120.mbfHs5.rst @@ -0,0 +1 @@ +Make :opcode:`POP_JUMP_IF_TRUE`, :opcode:`POP_JUMP_IF_FALSE`, :opcode:`POP_JUMP_IF_NONE` and :opcode:`POP_JUMP_IF_NOT_NONE` virtual, mapping to new relative jump opcodes. diff --git a/Objects/frameobject.c b/Objects/frameobject.c index 5bb8937009897..7a4d2fac9382c 100644 --- a/Objects/frameobject.c +++ b/Objects/frameobject.c @@ -206,11 +206,21 @@ mark_stacks(PyCodeObject *code_obj, int len) switch (opcode) { case JUMP_IF_FALSE_OR_POP: case JUMP_IF_TRUE_OR_POP: - case POP_JUMP_IF_FALSE: - case POP_JUMP_IF_TRUE: + case POP_JUMP_FORWARD_IF_FALSE: + case POP_JUMP_BACKWARD_IF_FALSE: + case POP_JUMP_FORWARD_IF_TRUE: + case POP_JUMP_BACKWARD_IF_TRUE: { int64_t target_stack; int j = get_arg(code, i); + if (opcode == POP_JUMP_FORWARD_IF_FALSE || + opcode == POP_JUMP_FORWARD_IF_TRUE) { + j += i + 1; + } + else if (opcode == POP_JUMP_BACKWARD_IF_FALSE || + opcode == POP_JUMP_BACKWARD_IF_TRUE) { + j = i + 1 - j; + } assert(j < len); if (stacks[j] == UNINITIALIZED && j < i) { todo = 1; diff --git a/Python/ceval.c b/Python/ceval.c index f3d1c100d83ae..b46b1ef842939 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -3646,8 +3646,6 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int goto error; } JUMPBY(INLINE_CACHE_ENTRIES_COMPARE_OP); - PREDICT(POP_JUMP_IF_FALSE); - PREDICT(POP_JUMP_IF_TRUE); DISPATCH(); } @@ -3670,7 +3668,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int TARGET(COMPARE_OP_FLOAT_JUMP) { assert(cframe.use_tracing == 0); - // Combined: COMPARE_OP (float ? float) + POP_JUMP_IF_(true/false) + // Combined: COMPARE_OP (float ? float) + POP_JUMP_(direction)_IF_(true/false) _PyCompareOpCache *cache = (_PyCompareOpCache *)next_instr; int when_to_jump_mask = cache->mask; PyObject *right = TOP(); @@ -3688,22 +3686,31 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int STACK_SHRINK(2); Py_DECREF(left); Py_DECREF(right); - assert(opcode == POP_JUMP_IF_TRUE || opcode == POP_JUMP_IF_FALSE); - int jump = (1 << (sign + 1)) & when_to_jump_mask; + assert(opcode == POP_JUMP_FORWARD_IF_FALSE || + opcode == POP_JUMP_BACKWARD_IF_FALSE || + opcode == POP_JUMP_FORWARD_IF_TRUE || + opcode == POP_JUMP_BACKWARD_IF_TRUE); + int jump = (9 << (sign + 1)) & when_to_jump_mask; if (!jump) { next_instr++; - NOTRACE_DISPATCH(); } - else { - JUMPTO(oparg); + else if (jump >= 8) { + assert(opcode == POP_JUMP_BACKWARD_IF_TRUE || + opcode == POP_JUMP_BACKWARD_IF_FALSE); + JUMPBY(1 - oparg); CHECK_EVAL_BREAKER(); - NOTRACE_DISPATCH(); } + else { + assert(opcode == POP_JUMP_FORWARD_IF_TRUE || + opcode == POP_JUMP_FORWARD_IF_FALSE); + JUMPBY(1 + oparg); + } + NOTRACE_DISPATCH(); } TARGET(COMPARE_OP_INT_JUMP) { assert(cframe.use_tracing == 0); - // Combined: COMPARE_OP (int ? int) + POP_JUMP_IF_(true/false) + // Combined: COMPARE_OP (int ? int) + POP_JUMP_(direction)_IF_(true/false) _PyCompareOpCache *cache = (_PyCompareOpCache *)next_instr; int when_to_jump_mask = cache->mask; PyObject *right = TOP(); @@ -3722,24 +3729,33 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int STACK_SHRINK(2); Py_DECREF(left); Py_DECREF(right); - assert(opcode == POP_JUMP_IF_TRUE || opcode == POP_JUMP_IF_FALSE); - int jump = (1 << (sign + 1)) & when_to_jump_mask; + assert(opcode == POP_JUMP_FORWARD_IF_FALSE || + opcode == POP_JUMP_BACKWARD_IF_FALSE || + opcode == POP_JUMP_FORWARD_IF_TRUE || + opcode == POP_JUMP_BACKWARD_IF_TRUE); + int jump = (9 << (sign + 1)) & when_to_jump_mask; if (!jump) { next_instr++; - NOTRACE_DISPATCH(); } - else { - JUMPTO(oparg); + else if (jump >= 8) { + assert(opcode == POP_JUMP_BACKWARD_IF_TRUE || + opcode == POP_JUMP_BACKWARD_IF_FALSE); + JUMPBY(1 - oparg); CHECK_EVAL_BREAKER(); - NOTRACE_DISPATCH(); } + else { + assert(opcode == POP_JUMP_FORWARD_IF_TRUE || + opcode == POP_JUMP_FORWARD_IF_FALSE); + JUMPBY(1 + oparg); + } + NOTRACE_DISPATCH(); } TARGET(COMPARE_OP_STR_JUMP) { assert(cframe.use_tracing == 0); - // Combined: COMPARE_OP (str == str or str != str) + POP_JUMP_IF_(true/false) + // Combined: COMPARE_OP (str == str or str != str) + POP_JUMP_(direction)_IF_(true/false) _PyCompareOpCache *cache = (_PyCompareOpCache *)next_instr; - int invert = cache->mask; + int when_to_jump_mask = cache->mask; PyObject *right = TOP(); PyObject *left = SECOND(); DEOPT_IF(!PyUnicode_CheckExact(left), COMPARE_OP); @@ -3752,22 +3768,31 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int assert(oparg == Py_EQ || oparg == Py_NE); JUMPBY(INLINE_CACHE_ENTRIES_COMPARE_OP); NEXTOPARG(); - assert(opcode == POP_JUMP_IF_TRUE || opcode == POP_JUMP_IF_FALSE); + assert(opcode == POP_JUMP_FORWARD_IF_FALSE || + opcode == POP_JUMP_BACKWARD_IF_FALSE || + opcode == POP_JUMP_FORWARD_IF_TRUE || + opcode == POP_JUMP_BACKWARD_IF_TRUE); STACK_SHRINK(2); Py_DECREF(left); Py_DECREF(right); assert(res == 0 || res == 1); - assert(invert == 0 || invert == 1); - int jump = res ^ invert; + int sign = 1 - res; + int jump = (9 << (sign + 1)) & when_to_jump_mask; if (!jump) { next_instr++; - NOTRACE_DISPATCH(); } - else { - JUMPTO(oparg); + else if (jump >= 8) { + assert(opcode == POP_JUMP_BACKWARD_IF_TRUE || + opcode == POP_JUMP_BACKWARD_IF_FALSE); + JUMPBY(1 - oparg); CHECK_EVAL_BREAKER(); - NOTRACE_DISPATCH(); } + else { + assert(opcode == POP_JUMP_FORWARD_IF_TRUE || + opcode == POP_JUMP_FORWARD_IF_FALSE); + JUMPBY(1 + oparg); + } + NOTRACE_DISPATCH(); } TARGET(IS_OP) { @@ -3779,8 +3804,6 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int SET_TOP(b); Py_DECREF(left); Py_DECREF(right); - PREDICT(POP_JUMP_IF_FALSE); - PREDICT(POP_JUMP_IF_TRUE); DISPATCH(); } @@ -3796,8 +3819,6 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int PyObject *b = (res^oparg) ? Py_True : Py_False; Py_INCREF(b); PUSH(b); - PREDICT(POP_JUMP_IF_FALSE); - PREDICT(POP_JUMP_IF_TRUE); DISPATCH(); } @@ -3915,26 +3936,25 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int JUMP_TO_INSTRUCTION(JUMP_BACKWARD_QUICK); } - TARGET(POP_JUMP_IF_FALSE) { - PREDICTED(POP_JUMP_IF_FALSE); + TARGET(POP_JUMP_BACKWARD_IF_FALSE) { + PREDICTED(POP_JUMP_BACKWARD_IF_FALSE); PyObject *cond = POP(); - int err; if (Py_IsTrue(cond)) { Py_DECREF(cond); DISPATCH(); } if (Py_IsFalse(cond)) { Py_DECREF(cond); - JUMPTO(oparg); + JUMPBY(-oparg); CHECK_EVAL_BREAKER(); DISPATCH(); } - err = PyObject_IsTrue(cond); + int err = PyObject_IsTrue(cond); Py_DECREF(cond); if (err > 0) ; else if (err == 0) { - JUMPTO(oparg); + JUMPBY(-oparg); CHECK_EVAL_BREAKER(); } else @@ -3942,24 +3962,46 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int DISPATCH(); } - TARGET(POP_JUMP_IF_TRUE) { - PREDICTED(POP_JUMP_IF_TRUE); + TARGET(POP_JUMP_FORWARD_IF_FALSE) { + PREDICTED(POP_JUMP_FORWARD_IF_FALSE); + PyObject *cond = POP(); + if (Py_IsTrue(cond)) { + Py_DECREF(cond); + } + else if (Py_IsFalse(cond)) { + Py_DECREF(cond); + JUMPBY(oparg); + } + else { + int err = PyObject_IsTrue(cond); + Py_DECREF(cond); + if (err > 0) + ; + else if (err == 0) { + JUMPBY(oparg); + } + else + goto error; + } + DISPATCH(); + } + + TARGET(POP_JUMP_BACKWARD_IF_TRUE) { PyObject *cond = POP(); - int err; if (Py_IsFalse(cond)) { Py_DECREF(cond); DISPATCH(); } if (Py_IsTrue(cond)) { Py_DECREF(cond); - JUMPTO(oparg); + JUMPBY(-oparg); CHECK_EVAL_BREAKER(); DISPATCH(); } - err = PyObject_IsTrue(cond); + int err = PyObject_IsTrue(cond); Py_DECREF(cond); if (err > 0) { - JUMPTO(oparg); + JUMPBY(-oparg); CHECK_EVAL_BREAKER(); } else if (err == 0) @@ -3969,11 +4011,34 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int DISPATCH(); } - TARGET(POP_JUMP_IF_NOT_NONE) { + TARGET(POP_JUMP_FORWARD_IF_TRUE) { + PyObject *cond = POP(); + if (Py_IsFalse(cond)) { + Py_DECREF(cond); + } + else if (Py_IsTrue(cond)) { + Py_DECREF(cond); + JUMPBY(oparg); + } + else { + int err = PyObject_IsTrue(cond); + Py_DECREF(cond); + if (err > 0) { + JUMPBY(oparg); + } + else if (err == 0) + ; + else + goto error; + } + DISPATCH(); + } + + TARGET(POP_JUMP_BACKWARD_IF_NOT_NONE) { PyObject *value = POP(); if (!Py_IsNone(value)) { Py_DECREF(value); - JUMPTO(oparg); + JUMPBY(-oparg); CHECK_EVAL_BREAKER(); DISPATCH(); } @@ -3981,11 +4046,20 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int DISPATCH(); } - TARGET(POP_JUMP_IF_NONE) { + TARGET(POP_JUMP_FORWARD_IF_NOT_NONE) { + PyObject *value = POP(); + if (!Py_IsNone(value)) { + JUMPBY(oparg); + } + Py_DECREF(value); + DISPATCH(); + } + + TARGET(POP_JUMP_BACKWARD_IF_NONE) { PyObject *value = POP(); if (Py_IsNone(value)) { Py_DECREF(value); - JUMPTO(oparg); + JUMPBY(-oparg); CHECK_EVAL_BREAKER(); DISPATCH(); } @@ -3993,6 +4067,15 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int DISPATCH(); } + TARGET(POP_JUMP_FORWARD_IF_NONE) { + PyObject *value = POP(); + if (Py_IsNone(value)) { + JUMPBY(oparg); + } + Py_DECREF(value); + DISPATCH(); + } + TARGET(JUMP_IF_FALSE_OR_POP) { PyObject *cond = TOP(); int err; @@ -4108,7 +4191,8 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int PyObject *res = match ? Py_True : Py_False; Py_INCREF(res); PUSH(res); - PREDICT(POP_JUMP_IF_FALSE); + PREDICT(POP_JUMP_FORWARD_IF_FALSE); + PREDICT(POP_JUMP_BACKWARD_IF_FALSE); DISPATCH(); } @@ -4118,7 +4202,8 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int PyObject *res = match ? Py_True : Py_False; Py_INCREF(res); PUSH(res); - PREDICT(POP_JUMP_IF_FALSE); + PREDICT(POP_JUMP_FORWARD_IF_FALSE); + PREDICT(POP_JUMP_BACKWARD_IF_FALSE); DISPATCH(); } diff --git a/Python/compile.c b/Python/compile.c index f04ba9ec50f6f..38cf5f362705a 100644 --- a/Python/compile.c +++ b/Python/compile.c @@ -78,8 +78,12 @@ #define POP_BLOCK -4 #define JUMP -5 #define JUMP_NO_INTERRUPT -6 +#define POP_JUMP_IF_FALSE -7 +#define POP_JUMP_IF_TRUE -8 +#define POP_JUMP_IF_NONE -9 +#define POP_JUMP_IF_NOT_NONE -10 -#define MIN_VIRTUAL_OPCODE -6 +#define MIN_VIRTUAL_OPCODE -10 #define MAX_ALLOWED_OPCODE 254 #define IS_WITHIN_OPCODE_RANGE(opcode) \ @@ -87,11 +91,36 @@ #define IS_VIRTUAL_OPCODE(opcode) ((opcode) < 0) +#define IS_VIRTUAL_JUMP_OPCODE(opcode) \ + ((opcode) == JUMP || \ + (opcode) == JUMP_NO_INTERRUPT || \ + (opcode) == POP_JUMP_IF_NONE || \ + (opcode) == POP_JUMP_IF_NOT_NONE || \ + (opcode) == POP_JUMP_IF_FALSE || \ + (opcode) == POP_JUMP_IF_TRUE) + /* opcodes which are not emitted in codegen stage, only by the assembler */ #define IS_ASSEMBLER_OPCODE(opcode) \ ((opcode) == JUMP_FORWARD || \ (opcode) == JUMP_BACKWARD || \ - (opcode) == JUMP_BACKWARD_NO_INTERRUPT) + (opcode) == JUMP_BACKWARD_NO_INTERRUPT || \ + (opcode) == POP_JUMP_FORWARD_IF_NONE || \ + (opcode) == POP_JUMP_BACKWARD_IF_NONE || \ + (opcode) == POP_JUMP_FORWARD_IF_NOT_NONE || \ + (opcode) == POP_JUMP_BACKWARD_IF_NOT_NONE || \ + (opcode) == POP_JUMP_FORWARD_IF_TRUE || \ + (opcode) == POP_JUMP_BACKWARD_IF_TRUE || \ + (opcode) == POP_JUMP_FORWARD_IF_FALSE || \ + (opcode) == POP_JUMP_BACKWARD_IF_FALSE) + + +#define IS_BACKWARDS_JUMP_OPCODE(opcode) \ + ((opcode) == JUMP_BACKWARD || \ + (opcode) == JUMP_BACKWARD_NO_INTERRUPT || \ + (opcode) == POP_JUMP_BACKWARD_IF_NONE || \ + (opcode) == POP_JUMP_BACKWARD_IF_NOT_NONE || \ + (opcode) == POP_JUMP_BACKWARD_IF_TRUE || \ + (opcode) == POP_JUMP_BACKWARD_IF_FALSE) #define IS_TOP_LEVEL_AWAIT(c) ( \ @@ -156,7 +185,7 @@ is_block_push(struct instr *instr) static inline int is_jump(struct instr *i) { - return i->i_opcode == JUMP || + return IS_VIRTUAL_JUMP_OPCODE(i->i_opcode) || is_bit_set_in_table(_PyOpcode_Jump, i->i_opcode); } @@ -1027,10 +1056,18 @@ stack_effect(int opcode, int oparg, int jump) case JUMP_IF_FALSE_OR_POP: return jump ? 0 : -1; - case POP_JUMP_IF_FALSE: - case POP_JUMP_IF_TRUE: + case POP_JUMP_BACKWARD_IF_NONE: + case POP_JUMP_FORWARD_IF_NONE: case POP_JUMP_IF_NONE: + case POP_JUMP_BACKWARD_IF_NOT_NONE: + case POP_JUMP_FORWARD_IF_NOT_NONE: case POP_JUMP_IF_NOT_NONE: + case POP_JUMP_FORWARD_IF_FALSE: + case POP_JUMP_BACKWARD_IF_FALSE: + case POP_JUMP_IF_FALSE: + case POP_JUMP_FORWARD_IF_TRUE: + case POP_JUMP_BACKWARD_IF_TRUE: + case POP_JUMP_IF_TRUE: return -1; case LOAD_GLOBAL: @@ -7609,14 +7646,33 @@ normalize_jumps(struct assembler *a) } struct instr *last = &b->b_instr[b->b_iused-1]; assert(!IS_ASSEMBLER_OPCODE(last->i_opcode)); - if (last->i_opcode == JUMP) { - bool is_forward = last->i_target->b_visited == 0; - last->i_opcode = is_forward ? JUMP_FORWARD : JUMP_BACKWARD; - } - if (last->i_opcode == JUMP_NO_INTERRUPT) { + if (is_jump(last)) { bool is_forward = last->i_target->b_visited == 0; - last->i_opcode = is_forward ? - JUMP_FORWARD : JUMP_BACKWARD_NO_INTERRUPT; + switch(last->i_opcode) { + case JUMP: + last->i_opcode = is_forward ? JUMP_FORWARD : JUMP_BACKWARD; + break; + case JUMP_NO_INTERRUPT: + last->i_opcode = is_forward ? + JUMP_FORWARD : JUMP_BACKWARD_NO_INTERRUPT; + break; + case POP_JUMP_IF_NOT_NONE: + last->i_opcode = is_forward ? + POP_JUMP_FORWARD_IF_NOT_NONE : POP_JUMP_BACKWARD_IF_NOT_NONE; + break; + case POP_JUMP_IF_NONE: + last->i_opcode = is_forward ? + POP_JUMP_FORWARD_IF_NONE : POP_JUMP_BACKWARD_IF_NONE; + break; + case POP_JUMP_IF_FALSE: + last->i_opcode = is_forward ? + POP_JUMP_FORWARD_IF_FALSE : POP_JUMP_BACKWARD_IF_FALSE; + break; + case POP_JUMP_IF_TRUE: + last->i_opcode = is_forward ? + POP_JUMP_FORWARD_IF_TRUE : POP_JUMP_BACKWARD_IF_TRUE; + break; + } } } } @@ -7652,16 +7708,17 @@ assemble_jump_offsets(struct assembler *a, struct compiler *c) instr->i_oparg = instr->i_target->b_offset; if (is_relative_jump(instr)) { if (instr->i_oparg < bsize) { - assert(instr->i_opcode == JUMP_BACKWARD || - instr->i_opcode == JUMP_BACKWARD_NO_INTERRUPT); + assert(IS_BACKWARDS_JUMP_OPCODE(instr->i_opcode)); instr->i_oparg = bsize - instr->i_oparg; } else { - assert(instr->i_opcode != JUMP_BACKWARD); - assert(instr->i_opcode != JUMP_BACKWARD_NO_INTERRUPT); + assert(!IS_BACKWARDS_JUMP_OPCODE(instr->i_opcode)); instr->i_oparg -= bsize; } } + else { + assert(!IS_BACKWARDS_JUMP_OPCODE(instr->i_opcode)); + } if (instr_size(instr) != isize) { extended_arg_recompile = 1; } @@ -8644,7 +8701,7 @@ apply_static_swaps(basicblock *block, int i) static bool jump_thread(struct instr *inst, struct instr *target, int opcode) { - assert(!IS_VIRTUAL_OPCODE(opcode) || opcode == JUMP); + assert(!IS_VIRTUAL_OPCODE(opcode) || IS_VIRTUAL_JUMP_OPCODE(opcode)); assert(is_jump(inst)); assert(is_jump(target)); // bpo-45773: If inst->i_target == target->i_target, then nothing actually diff --git a/Python/opcode_targets.h b/Python/opcode_targets.h index 064aa060c8428..5268c4f2a2775 100644 --- a/Python/opcode_targets.h +++ b/Python/opcode_targets.h @@ -113,8 +113,8 @@ static void *opcode_targets[256] = { &&TARGET_JUMP_IF_FALSE_OR_POP, &&TARGET_JUMP_IF_TRUE_OR_POP, &&TARGET_PRECALL_NO_KW_STR_1, - &&TARGET_POP_JUMP_IF_FALSE, - &&TARGET_POP_JUMP_IF_TRUE, + &&TARGET_POP_JUMP_FORWARD_IF_FALSE, + &&TARGET_POP_JUMP_FORWARD_IF_TRUE, &&TARGET_LOAD_GLOBAL, &&TARGET_IS_OP, &&TARGET_CONTAINS_OP, @@ -127,8 +127,8 @@ static void *opcode_targets[256] = { &&TARGET_STORE_FAST, &&TARGET_DELETE_FAST, &&TARGET_PRECALL_NO_KW_TYPE_1, - &&TARGET_POP_JUMP_IF_NOT_NONE, - &&TARGET_POP_JUMP_IF_NONE, + &&TARGET_POP_JUMP_FORWARD_IF_NOT_NONE, + &&TARGET_POP_JUMP_FORWARD_IF_NONE, &&TARGET_RAISE_VARARGS, &&TARGET_GET_AWAITABLE, &&TARGET_MAKE_FUNCTION, @@ -172,6 +172,10 @@ static void *opcode_targets[256] = { &&TARGET_UNPACK_SEQUENCE_ADAPTIVE, &&TARGET_CALL, &&TARGET_KW_NAMES, + &&TARGET_POP_JUMP_BACKWARD_IF_NOT_NONE, + &&TARGET_POP_JUMP_BACKWARD_IF_NONE, + &&TARGET_POP_JUMP_BACKWARD_IF_FALSE, + &&TARGET_POP_JUMP_BACKWARD_IF_TRUE, &&TARGET_UNPACK_SEQUENCE_LIST, &&TARGET_UNPACK_SEQUENCE_TUPLE, &&TARGET_UNPACK_SEQUENCE_TWO_TUPLE, @@ -250,9 +254,5 @@ static void *opcode_targets[256] = { &&_unknown_opcode, &&_unknown_opcode, &&_unknown_opcode, - &&_unknown_opcode, - &&_unknown_opcode, - &&_unknown_opcode, - &&_unknown_opcode, &&TARGET_DO_TRACING }; diff --git a/Python/specialize.c b/Python/specialize.c index 36b05026489cf..3a8b768549c63 100644 --- a/Python/specialize.c +++ b/Python/specialize.c @@ -1883,7 +1883,10 @@ _Py_Specialize_CompareOp(PyObject *lhs, PyObject *rhs, _Py_CODEUNIT *instr, assert(_PyOpcode_Caches[COMPARE_OP] == INLINE_CACHE_ENTRIES_COMPARE_OP); _PyCompareOpCache *cache = (_PyCompareOpCache *)(instr + 1); int next_opcode = _Py_OPCODE(instr[INLINE_CACHE_ENTRIES_COMPARE_OP + 1]); - if (next_opcode != POP_JUMP_IF_FALSE && next_opcode != POP_JUMP_IF_TRUE) { + if (next_opcode != POP_JUMP_FORWARD_IF_FALSE && + next_opcode != POP_JUMP_BACKWARD_IF_FALSE && + next_opcode != POP_JUMP_FORWARD_IF_TRUE && + next_opcode != POP_JUMP_BACKWARD_IF_TRUE) { // Can't ever combine, so don't don't bother being adaptive (unless // we're collecting stats, where it's more important to get accurate hit // counts for the unadaptive version and each of the different failure @@ -1901,9 +1904,14 @@ _Py_Specialize_CompareOp(PyObject *lhs, PyObject *rhs, _Py_CODEUNIT *instr, } assert(oparg <= Py_GE); int when_to_jump_mask = compare_masks[oparg]; - if (next_opcode == POP_JUMP_IF_FALSE) { + if (next_opcode == POP_JUMP_FORWARD_IF_FALSE || + next_opcode == POP_JUMP_BACKWARD_IF_FALSE) { when_to_jump_mask = (1 | 2 | 4) & ~when_to_jump_mask; } + if (next_opcode == POP_JUMP_BACKWARD_IF_TRUE || + next_opcode == POP_JUMP_BACKWARD_IF_FALSE) { + when_to_jump_mask <<= 3; + } if (Py_TYPE(lhs) != Py_TYPE(rhs)) { SPECIALIZATION_FAIL(COMPARE_OP, compare_op_fail_kind(lhs, rhs)); goto failure; @@ -1931,7 +1939,7 @@ _Py_Specialize_CompareOp(PyObject *lhs, PyObject *rhs, _Py_CODEUNIT *instr, } else { _Py_SET_OPCODE(*instr, COMPARE_OP_STR_JUMP); - cache->mask = (when_to_jump_mask & 2) == 0; + cache->mask = when_to_jump_mask; goto success; } } From webhook-mailer at python.org Mon Apr 11 10:34:35 2022 From: webhook-mailer at python.org (mdickinson) Date: Mon, 11 Apr 2022 14:34:35 -0000 Subject: [Python-checkins] bpo-45995: add "z" format specifer to coerce negative 0 to zero (GH-30049) Message-ID: https://github.com/python/cpython/commit/b0b836b20cb56c225874a4a39ef895f89ab2970f commit: b0b836b20cb56c225874a4a39ef895f89ab2970f branch: main author: John Belmonte committer: mdickinson date: 2022-04-11T15:34:18+01:00 summary: bpo-45995: add "z" format specifer to coerce negative 0 to zero (GH-30049) Add "z" format specifier to coerce negative 0 to zero. See https://github.com/python/cpython/issues/90153 (originally https://bugs.python.org/issue45995) for discussion. This covers `str.format()` and f-strings. Old-style string interpolation is not supported. Co-authored-by: Mark Dickinson files: A Misc/NEWS.d/next/Library/2021-12-14-13-15-41.bpo-45995.Am9pNL.rst M Doc/library/string.rst M Include/internal/pycore_format.h M Include/pystrtod.h M Lib/_pydecimal.py M Lib/pydoc_data/topics.py M Lib/test/test_decimal.py M Lib/test/test_float.py M Lib/test/test_format.py M Lib/test/test_types.py M Modules/_decimal/_decimal.c M Objects/bytesobject.c M Objects/unicodeobject.c M Python/ast_opt.c M Python/formatter_unicode.c M Python/pystrtod.c diff --git a/Doc/library/string.rst b/Doc/library/string.rst index 78bd167bcf579..35e9bc116803f 100644 --- a/Doc/library/string.rst +++ b/Doc/library/string.rst @@ -309,7 +309,7 @@ non-empty format specification typically modifies the result. The general form of a *standard format specifier* is: .. productionlist:: format-spec - format_spec: [[`fill`]`align`][`sign`][#][0][`width`][`grouping_option`][.`precision`][`type`] + format_spec: [[`fill`]`align`][`sign`][z][#][0][`width`][`grouping_option`][.`precision`][`type`] fill: align: "<" | ">" | "=" | "^" sign: "+" | "-" | " " @@ -380,6 +380,15 @@ following: +---------+----------------------------------------------------------+ +.. index:: single: z; in string formatting + +The ``'z'`` option coerces negative zero floating-point values to positive +zero after rounding to the format precision. This option is only valid for +floating-point presentation types. + +.. versionchanged:: 3.11 + Added the ``'z'`` option (see also :pep:`682`). + .. index:: single: # (hash); in string formatting The ``'#'`` option causes the "alternate form" to be used for the diff --git a/Include/internal/pycore_format.h b/Include/internal/pycore_format.h index 1b8d57539ca50..1899609e77ef2 100644 --- a/Include/internal/pycore_format.h +++ b/Include/internal/pycore_format.h @@ -14,12 +14,14 @@ extern "C" { * F_BLANK ' ' * F_ALT '#' * F_ZERO '0' + * F_NO_NEG_0 'z' */ #define F_LJUST (1<<0) #define F_SIGN (1<<1) #define F_BLANK (1<<2) #define F_ALT (1<<3) #define F_ZERO (1<<4) +#define F_NO_NEG_0 (1<<5) #ifdef __cplusplus } diff --git a/Include/pystrtod.h b/Include/pystrtod.h index c1e84de6fe542..fa056d17b6395 100644 --- a/Include/pystrtod.h +++ b/Include/pystrtod.h @@ -32,6 +32,7 @@ PyAPI_FUNC(double) _Py_parse_inf_or_nan(const char *p, char **endptr); #define Py_DTSF_ADD_DOT_0 0x02 /* if the result is an integer add ".0" */ #define Py_DTSF_ALT 0x04 /* "alternate" formatting. it's format_code specific */ +#define Py_DTSF_NO_NEG_0 0x08 /* negative zero result is coerced to 0 */ /* PyOS_double_to_string's "type", if non-NULL, will be set to one of: */ #define Py_DTST_FINITE 0 diff --git a/Lib/_pydecimal.py b/Lib/_pydecimal.py index f6d9ddf42e473..89646fa714c54 100644 --- a/Lib/_pydecimal.py +++ b/Lib/_pydecimal.py @@ -3795,6 +3795,10 @@ def __format__(self, specifier, context=None, _localeconv=None): # represented in fixed point; rescale them to 0e0. if not self and self._exp > 0 and spec['type'] in 'fF%': self = self._rescale(0, rounding) + if not self and spec['no_neg_0'] and self._sign: + adjusted_sign = 0 + else: + adjusted_sign = self._sign # figure out placement of the decimal point leftdigits = self._exp + len(self._int) @@ -3825,7 +3829,7 @@ def __format__(self, specifier, context=None, _localeconv=None): # done with the decimal-specific stuff; hand over the rest # of the formatting to the _format_number function - return _format_number(self._sign, intpart, fracpart, exp, spec) + return _format_number(adjusted_sign, intpart, fracpart, exp, spec) def _dec_from_triple(sign, coefficient, exponent, special=False): """Create a decimal instance directly, without any validation, @@ -6143,7 +6147,7 @@ def _convert_for_comparison(self, other, equality_op=False): # # A format specifier for Decimal looks like: # -# [[fill]align][sign][#][0][minimumwidth][,][.precision][type] +# [[fill]align][sign][z][#][0][minimumwidth][,][.precision][type] _parse_format_specifier_regex = re.compile(r"""\A (?: @@ -6151,6 +6155,7 @@ def _convert_for_comparison(self, other, equality_op=False): (?P[<>=^]) )? (?P[-+ ])? +(?Pz)? (?P\#)? (?P0)? (?P(?!0)\d+)? diff --git a/Lib/pydoc_data/topics.py b/Lib/pydoc_data/topics.py index 9b684c6dfc296..f3c79cab220a1 100644 --- a/Lib/pydoc_data/topics.py +++ b/Lib/pydoc_data/topics.py @@ -6119,7 +6119,7 @@ 'The general form of a *standard format specifier* is:\n' '\n' ' format_spec ::= ' - '[[fill]align][sign][#][0][width][grouping_option][.precision][type]\n' + '[[fill]align][sign][z][#][0][width][grouping_option][.precision][type]\n' ' fill ::= \n' ' align ::= "<" | ">" | "=" | "^"\n' ' sign ::= "+" | "-" | " "\n' @@ -6221,6 +6221,15 @@ ' ' '+-----------+------------------------------------------------------------+\n' '\n' + 'The "\'z\'" option coerces negative zero floating-point ' + 'values to positive\n' + 'zero after rounding to the format precision. This option ' + 'is only valid for\n' + 'floating-point presentation types.\n' + '\n' + 'Changed in version 3.11: Added the "\'z\'" option (see also ' + '**PEP 682**).\n' + '\n' 'The "\'#\'" option causes the ?alternate form? to be used ' 'for the\n' 'conversion. The alternate form is defined differently for ' diff --git a/Lib/test/test_decimal.py b/Lib/test/test_decimal.py index 0e7491ecdd45c..5e77e3c56cbbc 100644 --- a/Lib/test/test_decimal.py +++ b/Lib/test/test_decimal.py @@ -1072,6 +1072,57 @@ def test_formatting(self): (',e', '123456', '1.23456e+5'), (',E', '123456', '1.23456E+5'), + # negative zero: default behavior + ('.1f', '-0', '-0.0'), + ('.1f', '-.0', '-0.0'), + ('.1f', '-.01', '-0.0'), + + # negative zero: z option + ('z.1f', '0.', '0.0'), + ('z6.1f', '0.', ' 0.0'), + ('z6.1f', '-1.', ' -1.0'), + ('z.1f', '-0.', '0.0'), + ('z.1f', '.01', '0.0'), + ('z.1f', '-.01', '0.0'), + ('z.2f', '0.', '0.00'), + ('z.2f', '-0.', '0.00'), + ('z.2f', '.001', '0.00'), + ('z.2f', '-.001', '0.00'), + + ('z.1e', '0.', '0.0e+1'), + ('z.1e', '-0.', '0.0e+1'), + ('z.1E', '0.', '0.0E+1'), + ('z.1E', '-0.', '0.0E+1'), + + ('z.2e', '-0.001', '-1.00e-3'), # tests for mishandled rounding + ('z.2g', '-0.001', '-0.001'), + ('z.2%', '-0.001', '-0.10%'), + + ('zf', '-0.0000', '0.0000'), # non-normalized form is preserved + + ('z.1f', '-00000.000001', '0.0'), + ('z.1f', '-00000.', '0.0'), + ('z.1f', '-.0000000000', '0.0'), + + ('z.2f', '-00000.000001', '0.00'), + ('z.2f', '-00000.', '0.00'), + ('z.2f', '-.0000000000', '0.00'), + + ('z.1f', '.09', '0.1'), + ('z.1f', '-.09', '-0.1'), + + (' z.0f', '-0.', ' 0'), + ('+z.0f', '-0.', '+0'), + ('-z.0f', '-0.', '0'), + (' z.0f', '-1.', '-1'), + ('+z.0f', '-1.', '-1'), + ('-z.0f', '-1.', '-1'), + + ('z>6.1f', '-0.', 'zz-0.0'), + ('z>z6.1f', '-0.', 'zzz0.0'), + ('x>z6.1f', '-0.', 'xxx0.0'), + ('?>z6.1f', '-0.', '???0.0'), # multi-byte fill char + # issue 6850 ('a=-7.0', '0.12345', 'aaaa0.1'), @@ -1086,6 +1137,15 @@ def test_formatting(self): # bytes format argument self.assertRaises(TypeError, Decimal(1).__format__, b'-020') + def test_negative_zero_format_directed_rounding(self): + with self.decimal.localcontext() as ctx: + ctx.rounding = ROUND_CEILING + self.assertEqual(format(self.decimal.Decimal('-0.001'), 'z.2f'), + '0.00') + + def test_negative_zero_bad_format(self): + self.assertRaises(ValueError, format, self.decimal.Decimal('1.23'), 'fz') + def test_n_format(self): Decimal = self.decimal.Decimal diff --git a/Lib/test/test_float.py b/Lib/test/test_float.py index 9cf223f892678..d8c0fe1854eba 100644 --- a/Lib/test/test_float.py +++ b/Lib/test/test_float.py @@ -701,18 +701,16 @@ def test_format(self): # conversion to string should fail self.assertRaises(ValueError, format, 3.0, "s") - # other format specifiers shouldn't work on floats, - # in particular int specifiers - for format_spec in ([chr(x) for x in range(ord('a'), ord('z')+1)] + - [chr(x) for x in range(ord('A'), ord('Z')+1)]): - if not format_spec in 'eEfFgGn%': - self.assertRaises(ValueError, format, 0.0, format_spec) - self.assertRaises(ValueError, format, 1.0, format_spec) - self.assertRaises(ValueError, format, -1.0, format_spec) - self.assertRaises(ValueError, format, 1e100, format_spec) - self.assertRaises(ValueError, format, -1e100, format_spec) - self.assertRaises(ValueError, format, 1e-100, format_spec) - self.assertRaises(ValueError, format, -1e-100, format_spec) + # confirm format options expected to fail on floats, such as integer + # presentation types + for format_spec in 'sbcdoxX': + self.assertRaises(ValueError, format, 0.0, format_spec) + self.assertRaises(ValueError, format, 1.0, format_spec) + self.assertRaises(ValueError, format, -1.0, format_spec) + self.assertRaises(ValueError, format, 1e100, format_spec) + self.assertRaises(ValueError, format, -1e100, format_spec) + self.assertRaises(ValueError, format, 1e-100, format_spec) + self.assertRaises(ValueError, format, -1e-100, format_spec) # issue 3382 self.assertEqual(format(NAN, 'f'), 'nan') diff --git a/Lib/test/test_format.py b/Lib/test/test_format.py index 16d29d1ea3d94..69b0d5f1c5a51 100644 --- a/Lib/test/test_format.py +++ b/Lib/test/test_format.py @@ -546,6 +546,80 @@ def test_unicode_in_error_message(self): with self.assertRaisesRegex(ValueError, str_err): "{a:%????}".format(a='a') + def test_negative_zero(self): + ## default behavior + self.assertEqual(f"{-0.:.1f}", "-0.0") + self.assertEqual(f"{-.01:.1f}", "-0.0") + self.assertEqual(f"{-0:.1f}", "0.0") # integers do not distinguish -0 + + ## z sign option + self.assertEqual(f"{0.:z.1f}", "0.0") + self.assertEqual(f"{0.:z6.1f}", " 0.0") + self.assertEqual(f"{-1.:z6.1f}", " -1.0") + self.assertEqual(f"{-0.:z.1f}", "0.0") + self.assertEqual(f"{.01:z.1f}", "0.0") + self.assertEqual(f"{-0:z.1f}", "0.0") # z is allowed for integer input + self.assertEqual(f"{-.01:z.1f}", "0.0") + self.assertEqual(f"{0.:z.2f}", "0.00") + self.assertEqual(f"{-0.:z.2f}", "0.00") + self.assertEqual(f"{.001:z.2f}", "0.00") + self.assertEqual(f"{-.001:z.2f}", "0.00") + + self.assertEqual(f"{0.:z.1e}", "0.0e+00") + self.assertEqual(f"{-0.:z.1e}", "0.0e+00") + self.assertEqual(f"{0.:z.1E}", "0.0E+00") + self.assertEqual(f"{-0.:z.1E}", "0.0E+00") + + self.assertEqual(f"{-0.001:z.2e}", "-1.00e-03") # tests for mishandled + # rounding + self.assertEqual(f"{-0.001:z.2g}", "-0.001") + self.assertEqual(f"{-0.001:z.2%}", "-0.10%") + + self.assertEqual(f"{-00000.000001:z.1f}", "0.0") + self.assertEqual(f"{-00000.:z.1f}", "0.0") + self.assertEqual(f"{-.0000000000:z.1f}", "0.0") + + self.assertEqual(f"{-00000.000001:z.2f}", "0.00") + self.assertEqual(f"{-00000.:z.2f}", "0.00") + self.assertEqual(f"{-.0000000000:z.2f}", "0.00") + + self.assertEqual(f"{.09:z.1f}", "0.1") + self.assertEqual(f"{-.09:z.1f}", "-0.1") + + self.assertEqual(f"{-0.: z.0f}", " 0") + self.assertEqual(f"{-0.:+z.0f}", "+0") + self.assertEqual(f"{-0.:-z.0f}", "0") + self.assertEqual(f"{-1.: z.0f}", "-1") + self.assertEqual(f"{-1.:+z.0f}", "-1") + self.assertEqual(f"{-1.:-z.0f}", "-1") + + self.assertEqual(f"{0.j:z.1f}", "0.0+0.0j") + self.assertEqual(f"{-0.j:z.1f}", "0.0+0.0j") + self.assertEqual(f"{.01j:z.1f}", "0.0+0.0j") + self.assertEqual(f"{-.01j:z.1f}", "0.0+0.0j") + + self.assertEqual(f"{-0.:z>6.1f}", "zz-0.0") # test fill, esp. 'z' fill + self.assertEqual(f"{-0.:z>z6.1f}", "zzz0.0") + self.assertEqual(f"{-0.:x>z6.1f}", "xxx0.0") + self.assertEqual(f"{-0.:?>z6.1f}", "???0.0") # multi-byte fill char + + def test_specifier_z_error(self): + error_msg = re.compile("Invalid format specifier '.*z.*'") + with self.assertRaisesRegex(ValueError, error_msg): + f"{0:z+f}" # wrong position + with self.assertRaisesRegex(ValueError, error_msg): + f"{0:fz}" # wrong position + + error_msg = re.escape("Negative zero coercion (z) not allowed") + with self.assertRaisesRegex(ValueError, error_msg): + f"{0:zd}" # can't apply to int presentation type + with self.assertRaisesRegex(ValueError, error_msg): + f"{'x':zs}" # can't apply to string + + error_msg = re.escape("unsupported format character 'z'") + with self.assertRaisesRegex(ValueError, error_msg): + "%z.1f" % 0 # not allowed in old style string interpolation + if __name__ == "__main__": unittest.main() diff --git a/Lib/test/test_types.py b/Lib/test/test_types.py index f8b239117f513..42fd4f56235fa 100644 --- a/Lib/test/test_types.py +++ b/Lib/test/test_types.py @@ -524,18 +524,16 @@ def test(f, format_spec, result): self.assertRaises(TypeError, 3.0.__format__, None) self.assertRaises(TypeError, 3.0.__format__, 0) - # other format specifiers shouldn't work on floats, - # in particular int specifiers - for format_spec in ([chr(x) for x in range(ord('a'), ord('z')+1)] + - [chr(x) for x in range(ord('A'), ord('Z')+1)]): - if not format_spec in 'eEfFgGn%': - self.assertRaises(ValueError, format, 0.0, format_spec) - self.assertRaises(ValueError, format, 1.0, format_spec) - self.assertRaises(ValueError, format, -1.0, format_spec) - self.assertRaises(ValueError, format, 1e100, format_spec) - self.assertRaises(ValueError, format, -1e100, format_spec) - self.assertRaises(ValueError, format, 1e-100, format_spec) - self.assertRaises(ValueError, format, -1e-100, format_spec) + # confirm format options expected to fail on floats, such as integer + # presentation types + for format_spec in 'sbcdoxX': + self.assertRaises(ValueError, format, 0.0, format_spec) + self.assertRaises(ValueError, format, 1.0, format_spec) + self.assertRaises(ValueError, format, -1.0, format_spec) + self.assertRaises(ValueError, format, 1e100, format_spec) + self.assertRaises(ValueError, format, -1e100, format_spec) + self.assertRaises(ValueError, format, 1e-100, format_spec) + self.assertRaises(ValueError, format, -1e-100, format_spec) # Alternate float formatting test(1.0, '.0e', '1e+00') diff --git a/Misc/NEWS.d/next/Library/2021-12-14-13-15-41.bpo-45995.Am9pNL.rst b/Misc/NEWS.d/next/Library/2021-12-14-13-15-41.bpo-45995.Am9pNL.rst new file mode 100644 index 0000000000000..dd42bc092c280 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-12-14-13-15-41.bpo-45995.Am9pNL.rst @@ -0,0 +1,3 @@ +Add a "z" option to the string formatting specification that coerces negative +zero floating-point values to positive zero after rounding to the format +precision. Contributed by John Belmonte. diff --git a/Modules/_decimal/_decimal.c b/Modules/_decimal/_decimal.c index 35a115676a71b..4637b8b34c4ce 100644 --- a/Modules/_decimal/_decimal.c +++ b/Modules/_decimal/_decimal.c @@ -3183,6 +3183,56 @@ dotsep_as_utf8(const char *s) return utf8; } +/* copy of libmpdec _mpd_round() */ +static void +_mpd_round(mpd_t *result, const mpd_t *a, mpd_ssize_t prec, + const mpd_context_t *ctx, uint32_t *status) +{ + mpd_ssize_t exp = a->exp + a->digits - prec; + + if (prec <= 0) { + mpd_seterror(result, MPD_Invalid_operation, status); + return; + } + if (mpd_isspecial(a) || mpd_iszero(a)) { + mpd_qcopy(result, a, status); + return; + } + + mpd_qrescale_fmt(result, a, exp, ctx, status); + if (result->digits > prec) { + mpd_qrescale_fmt(result, result, exp+1, ctx, status); + } +} + +/* Locate negative zero "z" option within a UTF-8 format spec string. + * Returns pointer to "z", else NULL. + * The portion of the spec we're working with is [[fill]align][sign][z] */ +static const char * +format_spec_z_search(char const *fmt, Py_ssize_t size) { + char const *pos = fmt; + char const *fmt_end = fmt + size; + /* skip over [[fill]align] (fill may be multi-byte character) */ + pos += 1; + while (pos < fmt_end && *pos & 0x80) { + pos += 1; + } + if (pos < fmt_end && strchr("<>=^", *pos) != NULL) { + pos += 1; + } else { + /* fill not present-- skip over [align] */ + pos = fmt; + if (pos < fmt_end && strchr("<>=^", *pos) != NULL) { + pos += 1; + } + } + /* skip over [sign] */ + if (pos < fmt_end && strchr("+- ", *pos) != NULL) { + pos += 1; + } + return pos < fmt_end && *pos == 'z' ? pos : NULL; +} + static int dict_get_item_string(PyObject *dict, const char *key, PyObject **valueobj, const char **valuestr) { @@ -3220,11 +3270,16 @@ dec_format(PyObject *dec, PyObject *args) PyObject *fmtarg; PyObject *context; mpd_spec_t spec; - char *fmt; + char const *fmt; + char *fmt_copy = NULL; char *decstring = NULL; uint32_t status = 0; int replace_fillchar = 0; + int no_neg_0 = 0; Py_ssize_t size; + mpd_t *mpd = MPD(dec); + mpd_uint_t dt[MPD_MINALLOC_MAX]; + mpd_t tmp = {MPD_STATIC|MPD_STATIC_DATA,0,0,0,MPD_MINALLOC_MAX,dt}; CURRENT_CONTEXT(context); @@ -3233,19 +3288,39 @@ dec_format(PyObject *dec, PyObject *args) } if (PyUnicode_Check(fmtarg)) { - fmt = (char *)PyUnicode_AsUTF8AndSize(fmtarg, &size); + fmt = PyUnicode_AsUTF8AndSize(fmtarg, &size); if (fmt == NULL) { return NULL; } + /* NOTE: If https://github.com/python/cpython/pull/29438 lands, the + * format string manipulation below can be eliminated by enhancing + * the forked mpd_parse_fmt_str(). */ if (size > 0 && fmt[0] == '\0') { /* NUL fill character: must be replaced with a valid UTF-8 char before calling mpd_parse_fmt_str(). */ replace_fillchar = 1; - fmt = dec_strdup(fmt, size); - if (fmt == NULL) { + fmt = fmt_copy = dec_strdup(fmt, size); + if (fmt_copy == NULL) { return NULL; } - fmt[0] = '_'; + fmt_copy[0] = '_'; + } + /* Strip 'z' option, which isn't understood by mpd_parse_fmt_str(). + * NOTE: fmt is always null terminated by PyUnicode_AsUTF8AndSize() */ + char const *z_position = format_spec_z_search(fmt, size); + if (z_position != NULL) { + no_neg_0 = 1; + size_t z_index = z_position - fmt; + if (fmt_copy == NULL) { + fmt = fmt_copy = dec_strdup(fmt, size); + if (fmt_copy == NULL) { + return NULL; + } + } + /* Shift characters (including null terminator) left, + overwriting the 'z' option. */ + memmove(fmt_copy + z_index, fmt_copy + z_index + 1, size - z_index); + size -= 1; } } else { @@ -3311,8 +3386,45 @@ dec_format(PyObject *dec, PyObject *args) } } + if (no_neg_0 && mpd_isnegative(mpd) && !mpd_isspecial(mpd)) { + /* Round into a temporary (carefully mirroring the rounding + of mpd_qformat_spec()), and check if the result is negative zero. + If so, clear the sign and format the resulting positive zero. */ + mpd_ssize_t prec; + mpd_qcopy(&tmp, mpd, &status); + if (spec.prec >= 0) { + switch (spec.type) { + case 'f': + mpd_qrescale(&tmp, &tmp, -spec.prec, CTX(context), &status); + break; + case '%': + tmp.exp += 2; + mpd_qrescale(&tmp, &tmp, -spec.prec, CTX(context), &status); + break; + case 'g': + prec = (spec.prec == 0) ? 1 : spec.prec; + if (tmp.digits > prec) { + _mpd_round(&tmp, &tmp, prec, CTX(context), &status); + } + break; + case 'e': + if (!mpd_iszero(&tmp)) { + _mpd_round(&tmp, &tmp, spec.prec+1, CTX(context), &status); + } + break; + } + } + if (status & MPD_Errors) { + PyErr_SetString(PyExc_ValueError, "unexpected error when rounding"); + goto finish; + } + if (mpd_iszero(&tmp)) { + mpd_set_positive(&tmp); + mpd = &tmp; + } + } - decstring = mpd_qformat_spec(MPD(dec), &spec, CTX(context), &status); + decstring = mpd_qformat_spec(mpd, &spec, CTX(context), &status); if (decstring == NULL) { if (status & MPD_Malloc_error) { PyErr_NoMemory(); @@ -3335,7 +3447,7 @@ dec_format(PyObject *dec, PyObject *args) Py_XDECREF(grouping); Py_XDECREF(sep); Py_XDECREF(dot); - if (replace_fillchar) PyMem_Free(fmt); + if (fmt_copy) PyMem_Free(fmt_copy); if (decstring) mpd_free(decstring); return result; } diff --git a/Objects/bytesobject.c b/Objects/bytesobject.c index 78c42c2c54b5f..d0124c050d1e1 100644 --- a/Objects/bytesobject.c +++ b/Objects/bytesobject.c @@ -415,6 +415,7 @@ formatfloat(PyObject *v, int flags, int prec, int type, PyObject *result; double x; size_t len; + int dtoa_flags = 0; x = PyFloat_AsDouble(v); if (x == -1.0 && PyErr_Occurred()) { @@ -426,8 +427,13 @@ formatfloat(PyObject *v, int flags, int prec, int type, if (prec < 0) prec = 6; - p = PyOS_double_to_string(x, type, prec, - (flags & F_ALT) ? Py_DTSF_ALT : 0, NULL); + if (flags & F_ALT) { + dtoa_flags |= Py_DTSF_ALT; + } + if (flags & F_NO_NEG_0) { + dtoa_flags |= Py_DTSF_NO_NEG_0; + } + p = PyOS_double_to_string(x, type, prec, dtoa_flags, NULL); if (p == NULL) return NULL; @@ -706,6 +712,7 @@ _PyBytes_FormatEx(const char *format, Py_ssize_t format_len, case ' ': flags |= F_BLANK; continue; case '#': flags |= F_ALT; continue; case '0': flags |= F_ZERO; continue; + case 'z': flags |= F_NO_NEG_0; continue; } break; } diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c index 1b2f33212a779..c665f577abaa4 100644 --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -14372,7 +14372,7 @@ formatfloat(PyObject *v, struct unicode_format_arg_t *arg, double x; Py_ssize_t len; int prec; - int dtoa_flags; + int dtoa_flags = 0; x = PyFloat_AsDouble(v); if (x == -1.0 && PyErr_Occurred()) @@ -14383,9 +14383,9 @@ formatfloat(PyObject *v, struct unicode_format_arg_t *arg, prec = 6; if (arg->flags & F_ALT) - dtoa_flags = Py_DTSF_ALT; - else - dtoa_flags = 0; + dtoa_flags |= Py_DTSF_ALT; + if (arg->flags & F_NO_NEG_0) + dtoa_flags |= Py_DTSF_NO_NEG_0; p = PyOS_double_to_string(x, arg->ch, prec, dtoa_flags, NULL); if (p == NULL) return -1; diff --git a/Python/ast_opt.c b/Python/ast_opt.c index 77ed29d0cdddd..b1d807bcf10ae 100644 --- a/Python/ast_opt.c +++ b/Python/ast_opt.c @@ -310,6 +310,7 @@ simple_format_arg_parse(PyObject *fmt, Py_ssize_t *ppos, case ' ': *flags |= F_BLANK; continue; case '#': *flags |= F_ALT; continue; case '0': *flags |= F_ZERO; continue; + case 'z': *flags |= F_NO_NEG_0; continue; } break; } diff --git a/Python/formatter_unicode.c b/Python/formatter_unicode.c index a1e50e20c9d8c..04d37c0be28cd 100644 --- a/Python/formatter_unicode.c +++ b/Python/formatter_unicode.c @@ -130,6 +130,7 @@ typedef struct { Py_UCS4 fill_char; Py_UCS4 align; int alternate; + int no_neg_0; Py_UCS4 sign; Py_ssize_t width; enum LocaleType thousands_separators; @@ -166,6 +167,7 @@ parse_internal_render_format_spec(PyObject *obj, format->fill_char = ' '; format->align = default_align; format->alternate = 0; + format->no_neg_0 = 0; format->sign = '\0'; format->width = -1; format->thousands_separators = LT_NO_LOCALE; @@ -193,6 +195,13 @@ parse_internal_render_format_spec(PyObject *obj, ++pos; } + /* If the next character is z, request coercion of negative 0. + Applies only to floats. */ + if (end-pos >= 1 && READ_spec(pos) == 'z') { + format->no_neg_0 = 1; + ++pos; + } + /* If the next character is #, we're in alternate mode. This only applies to integers. */ if (end-pos >= 1 && READ_spec(pos) == '#') { @@ -779,6 +788,14 @@ format_string_internal(PyObject *value, const InternalFormatSpec *format, goto done; } + /* negative 0 coercion is not allowed on strings */ + if (format->no_neg_0) { + PyErr_SetString(PyExc_ValueError, + "Negative zero coercion (z) not allowed in string format " + "specifier"); + goto done; + } + /* alternate is not allowed on strings */ if (format->alternate) { PyErr_SetString(PyExc_ValueError, @@ -872,6 +889,13 @@ format_long_internal(PyObject *value, const InternalFormatSpec *format, "Precision not allowed in integer format specifier"); goto done; } + /* no negative zero coercion on integers */ + if (format->no_neg_0) { + PyErr_SetString(PyExc_ValueError, + "Negative zero coercion (z) not allowed in integer" + " format specifier"); + goto done; + } /* special case for character formatting */ if (format->type == 'c') { @@ -1049,6 +1073,8 @@ format_float_internal(PyObject *value, if (format->alternate) flags |= Py_DTSF_ALT; + if (format->no_neg_0) + flags |= Py_DTSF_NO_NEG_0; if (type == '\0') { /* Omitted type specifier. Behaves in the same way as repr(x) @@ -1238,6 +1264,8 @@ format_complex_internal(PyObject *value, if (format->alternate) flags |= Py_DTSF_ALT; + if (format->no_neg_0) + flags |= Py_DTSF_NO_NEG_0; if (type == '\0') { /* Omitted type specifier. Should be like str(self). */ diff --git a/Python/pystrtod.c b/Python/pystrtod.c index 1b27f0a3ad36a..d77b846f0403f 100644 --- a/Python/pystrtod.c +++ b/Python/pystrtod.c @@ -916,6 +916,18 @@ char * PyOS_double_to_string(double val, (flags & Py_DTSF_ALT ? "#" : ""), precision, format_code); _PyOS_ascii_formatd(buf, bufsize, format, val, precision); + + if (flags & Py_DTSF_NO_NEG_0 && buf[0] == '-') { + char *buf2 = buf + 1; + while (*buf2 == '0' || *buf2 == '.') { + ++buf2; + } + if (*buf2 == 0 || *buf2 == 'e') { + size_t len = buf2 - buf + strlen(buf2); + assert(buf[len] == 0); + memmove(buf, buf+1, len); + } + } } /* Add sign when requested. It's convenient (esp. when formatting @@ -995,8 +1007,8 @@ static char * format_float_short(double d, char format_code, int mode, int precision, int always_add_sign, int add_dot_0_if_integer, - int use_alt_formatting, const char * const *float_strings, - int *type) + int use_alt_formatting, int no_negative_zero, + const char * const *float_strings, int *type) { char *buf = NULL; char *p = NULL; @@ -1022,6 +1034,11 @@ format_float_short(double d, char format_code, assert(digits_end != NULL && digits_end >= digits); digits_len = digits_end - digits; + if (no_negative_zero && sign == 1 && + (digits_len == 0 || (digits_len == 1 && digits[0] == '0'))) { + sign = 0; + } + if (digits_len && !Py_ISDIGIT(digits[0])) { /* Infinities and nans here; adapt Gay's output, so convert Infinity to inf and NaN to nan, and @@ -1301,6 +1318,7 @@ char * PyOS_double_to_string(double val, flags & Py_DTSF_SIGN, flags & Py_DTSF_ADD_DOT_0, flags & Py_DTSF_ALT, + flags & Py_DTSF_NO_NEG_0, float_strings, type); } #endif // _PY_SHORT_FLOAT_REPR == 1 From webhook-mailer at python.org Mon Apr 11 10:51:37 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Mon, 11 Apr 2022 14:51:37 -0000 Subject: [Python-checkins] bpo-44807: Allow Protocol classes to define __init__ (GH-31628) Message-ID: https://github.com/python/cpython/commit/5f2abae61ec69264b835dcabe2cdabe57b9a990e commit: 5f2abae61ec69264b835dcabe2cdabe57b9a990e branch: main author: Adrian Garcia Badaracco <1755071+adriangb at users.noreply.github.com> committer: JelleZijlstra date: 2022-04-11T07:51:25-07:00 summary: bpo-44807: Allow Protocol classes to define __init__ (GH-31628) Co-authored-by: Jelle Zijlstra files: A Misc/NEWS.d/next/Library/2022-03-02-04-25-58.bpo-44807.gHNC9J.rst M Lib/test/test_typing.py M Lib/typing.py diff --git a/Lib/test/test_typing.py b/Lib/test/test_typing.py index e09f8aa3fb849..b884f7b2cced3 100644 --- a/Lib/test/test_typing.py +++ b/Lib/test/test_typing.py @@ -1598,6 +1598,32 @@ class CG(PG[T]): pass with self.assertRaises(TypeError): CG[int](42) + def test_protocol_defining_init_does_not_get_overridden(self): + # check that P.__init__ doesn't get clobbered + # see https://bugs.python.org/issue44807 + + class P(Protocol): + x: int + def __init__(self, x: int) -> None: + self.x = x + class C: pass + + c = C() + P.__init__(c, 1) + self.assertEqual(c.x, 1) + + def test_concrete_class_inheriting_init_from_protocol(self): + class P(Protocol): + x: int + def __init__(self, x: int) -> None: + self.x = x + + class C(P): pass + + c = C(1) + self.assertIsInstance(c, C) + self.assertEqual(c.x, 1) + def test_cannot_instantiate_abstract(self): @runtime_checkable class P(Protocol): diff --git a/Lib/typing.py b/Lib/typing.py index 26c6b8c278b73..ec8cbbd8b20a3 100644 --- a/Lib/typing.py +++ b/Lib/typing.py @@ -1997,7 +1997,8 @@ def _proto_hook(other): issubclass(base, Generic) and base._is_protocol): raise TypeError('Protocols can only inherit from other' ' protocols, got %r' % base) - cls.__init__ = _no_init_or_replace_init + if cls.__init__ is Protocol.__init__: + cls.__init__ = _no_init_or_replace_init class _AnnotatedAlias(_GenericAlias, _root=True): diff --git a/Misc/NEWS.d/next/Library/2022-03-02-04-25-58.bpo-44807.gHNC9J.rst b/Misc/NEWS.d/next/Library/2022-03-02-04-25-58.bpo-44807.gHNC9J.rst new file mode 100644 index 0000000000000..4757d3420caf8 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-03-02-04-25-58.bpo-44807.gHNC9J.rst @@ -0,0 +1 @@ +:class:`typing.Protocol` no longer silently replaces :meth:`__init__` methods defined on subclasses. Patch by Adrian Garcia Badaracco. From webhook-mailer at python.org Mon Apr 11 11:05:42 2022 From: webhook-mailer at python.org (markshannon) Date: Mon, 11 Apr 2022 15:05:42 -0000 Subject: [Python-checkins] GH-89480: Document motivation, design and implementation of 3.11 frame stack. (GH-32304) Message-ID: https://github.com/python/cpython/commit/f6e43e834c9712d548ce9dd2f7a1d9033e45ce51 commit: f6e43e834c9712d548ce9dd2f7a1d9033e45ce51 branch: main author: Mark Shannon committer: markshannon date: 2022-04-11T16:05:20+01:00 summary: GH-89480: Document motivation, design and implementation of 3.11 frame stack. (GH-32304) files: A Misc/NEWS.d/next/Core and Builtins/2022-04-04-15-12-38.bpo-45317.UDLOt8.rst A Objects/frame_layout.md M Include/internal/pycore_frame.h diff --git a/Include/internal/pycore_frame.h b/Include/internal/pycore_frame.h index 49bdc6324ca36..405afd67d2274 100644 --- a/Include/internal/pycore_frame.h +++ b/Include/internal/pycore_frame.h @@ -7,6 +7,11 @@ extern "C" { #include #include +/* See Objects/frame_layout.md for an explanation of the frame stack + * including explanation of the PyFrameObject and _PyInterpreterFrame + * structs. */ + + struct _frame { PyObject_HEAD PyFrameObject *f_back; /* previous frame, or NULL */ @@ -40,12 +45,14 @@ enum _frameowner { }; typedef struct _PyInterpreterFrame { + /* "Specials" section */ PyFunctionObject *f_func; /* Strong reference */ PyObject *f_globals; /* Borrowed reference */ PyObject *f_builtins; /* Borrowed reference */ PyObject *f_locals; /* Strong reference, may be NULL */ PyCodeObject *f_code; /* Strong reference */ PyFrameObject *frame_obj; /* Strong reference, may be NULL */ + /* Linkage section */ struct _PyInterpreterFrame *previous; // NOTE: This is not necessarily the last instruction started in the given // frame. Rather, it is the code unit *prior to* the *next* instruction. For @@ -55,6 +62,7 @@ typedef struct _PyInterpreterFrame { int stacktop; /* Offset of TOS from localsplus */ bool is_entry; // Whether this is the "root" frame for the current _PyCFrame. char owner; + /* Locals and stack */ PyObject *localsplus[1]; } _PyInterpreterFrame; diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-04-04-15-12-38.bpo-45317.UDLOt8.rst b/Misc/NEWS.d/next/Core and Builtins/2022-04-04-15-12-38.bpo-45317.UDLOt8.rst new file mode 100644 index 0000000000000..dd2da464e2552 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2022-04-04-15-12-38.bpo-45317.UDLOt8.rst @@ -0,0 +1 @@ +Add internal documentation explaining design of new (for 3.11) frame stack. diff --git a/Objects/frame_layout.md b/Objects/frame_layout.md new file mode 100644 index 0000000000000..11688f68e42e3 --- /dev/null +++ b/Objects/frame_layout.md @@ -0,0 +1,122 @@ +# The Frame Stack + +Each call to a Python function has an activation record, +commonly known as a "frame". +Python semantics allows frames to outlive the activation, +so they have (before 3.11) been allocated on the heap. +This is expensive as it requires many allocations and +results in poor locality of reference. + +In 3.11, rather than have these frames scattered about memory, +as happens for heap-allocated objects, frames are allocated +contiguously in a per-thread stack. +This improves performance significantly for two reasons: +* It reduces allocation overhead to a pointer comparison and increment. +* Stack allocated data has the best possible locality and will always be in + CPU cache. + +Generator and coroutines still need heap allocated activation records, but +can be linked into the per-thread stack so as to not impact performance too much. + +## Layout + +Each activation record consists of four conceptual sections: + +* Local variables (including arguments, cells and free variables) +* Evaluation stack +* Specials: The per-frame object references needed by the VM: globals dict, + code object, etc. +* Linkage: Pointer to the previous activation record, stack depth, etc. + +### Layout + +The specials and linkage sections are a fixed size, so are grouped together. + +Each activation record is laid out as: +* Specials and linkage +* Locals +* Stack + +This seems to provide the best performance without excessive complexity. +It needs the interpreter to hold two pointers, a frame pointer and a stack pointer. + +#### Alternative layout + +An alternative layout that was used for part of 3.11 alpha was: + +* Locals +* Specials and linkage +* Stack + +This has the advantage that no copying is required when making a call, +as the arguments on the stack are (usually) already in the correct +location for the parameters. However, it requires the VM to maintain +an extra pointer for the locals, which can hurt performance. + +A variant that only needs the need two pointers is to reverse the numbering +of the locals, so that the last one is numbered `0`, and the first in memory +is numbered `N-1`. +This allows the locals, specials and linkage to accessed from the frame pointer. +We may implement this in the future. + +#### Note: + +> In a contiguous stack, we would need to save one fewer registers, as the +> top of the caller's activation record would be the same at the base of the +> callee's. However, since some activation records are kept on the heap we +> cannot do this. + +### Generators and Coroutines + +Generators and coroutines contain a `_PyInterpreterFrame` +The specials sections contains the following pointers: + +* Globals dict +* Builtins dict +* Locals dict (not the "fast" locals, but the locals for eval and class creation) +* Code object +* Heap allocated `PyFrameObject` for this activation record, if any. +* The function. + +The pointer to the function is not strictly required, but it is cheaper to +store a strong reference to the function and borrowed references to the globals +and builtins, than strong references to both globals and builtins. + +### Frame objects + +When creating a backtrace or when calling `sys._getframe()` the frame becomes +visible to Python code. When this happens a new `PyFrameObject` is created +and a strong reference to it placed in the `frame_obj` field of the specials +section. The `frame_obj` field is initially `NULL`. + +The `PyFrameObject` may outlive a stack-allocated `_PyInterpreterFrame`. +If it does then `_PyInterpreterFrame` is copied into the `PyFrameObject`, +except the evaluation stack which must be empty at this point. +The linkage section is updated to reflect the new location of the frame. + +This mechanism provides the appearance of persistent, heap-allocated +frames for each activation, but with low runtime overhead. + +### Generators and Coroutines + + +Generator objects have a `_PyInterpreterFrame` embedded in them. +This means that creating a generator requires only a single allocation, +reducing allocation overhead and improving locality of reference. +The embedded frame is linked into the per-thread frame when iterated or +awaited. + +If a frame object associated with a generator outlives the generator, then +the embedded `_PyInterpreterFrame` is copied into the frame object. + + +All the above applies to coroutines and async generators as well. + +### Field names + +Many of the fields in `_PyInterpreterFrame` were copied from the 3.10 `PyFrameObject`. +Thus, some of the field names may be a bit misleading. + +For example the `f_globals` field has a `f_` prefix implying it belongs to the +`PyFrameObject` struct, although it belongs to the `_PyInterpreterFrame` struct. +We may rationalize this naming scheme for 3.12. \ No newline at end of file From webhook-mailer at python.org Mon Apr 11 11:10:47 2022 From: webhook-mailer at python.org (ezio-melotti) Date: Mon, 11 Apr 2022 15:10:47 -0000 Subject: [Python-checkins] gh-91423: Remove bugs.python.org from bugs.rst (GH-91425) Message-ID: https://github.com/python/cpython/commit/df81d2892eed3a256eb61ce59304f2173fb0c945 commit: df81d2892eed3a256eb61ce59304f2173fb0c945 branch: main author: roy reznik committer: ezio-melotti date: 2022-04-11T17:10:34+02:00 summary: gh-91423: Remove bugs.python.org from bugs.rst (GH-91425) * Remove bugs.python.org from bugs.rst * Update bugs.rst to the github issue tracker * reflow * Fix a typo and rephrase a sentence. Co-authored-by: Inada Naoki Co-authored-by: Ezio Melotti files: M Doc/bugs.rst diff --git a/Doc/bugs.rst b/Doc/bugs.rst index b3d057797c256..b32d359d02850 100644 --- a/Doc/bugs.rst +++ b/Doc/bugs.rst @@ -40,38 +40,39 @@ though it may take a while to be processed. Using the Python issue tracker ============================== -Bug reports for Python itself should be submitted via the Python Bug Tracker -(https://bugs.python.org/). The bug tracker offers a web form which allows -pertinent information to be entered and submitted to the developers. +Issue reports for Python itself should be submitted via the GitHub issues +tracker (https://github.com/python/cpython/issues). +The GitHub issues tracker offers a web form which allows pertinent information +to be entered and submitted to the developers. The first step in filing a report is to determine whether the problem has already been reported. The advantage in doing so, aside from saving the -developers time, is that you learn what has been done to fix it; it may be that +developers' time, is that you learn what has been done to fix it; it may be that the problem has already been fixed for the next release, or additional information is needed (in which case you are welcome to provide it if you can!). -To do this, search the bug database using the search box on the top of the page. +To do this, search the tracker using the search box at the top of the page. -If the problem you're reporting is not already in the bug tracker, go back to -the Python Bug Tracker and log in. If you don't already have a tracker account, -select the "Register" link or, if you use OpenID, one of the OpenID provider -logos in the sidebar. It is not possible to submit a bug report anonymously. +If the problem you're reporting is not already in the list, log in to GitHub. +If you don't already have a GitHub account, create a new account using the +"Sign up" link. +It is not possible to submit a bug report anonymously. -Being now logged in, you can submit a bug. Select the "Create New" link in the -sidebar to open the bug reporting form. +Being now logged in, you can submit an issue. +Click on the "New issue" button in the top bar to report a new issue. -The submission form has a number of fields. For the "Title" field, enter a -*very* short description of the problem; less than ten words is good. In the -"Type" field, select the type of your problem; also select the "Component" and -"Versions" to which the bug relates. +The submission form has two fields, "Title" and "Comment". + +For the "Title" field, enter a *very* short description of the problem; +less than ten words is good. In the "Comment" field, describe the problem in detail, including what you expected to happen and what did happen. Be sure to include whether any extension modules were involved, and what hardware and software platform you were using (including version information as appropriate). -Each bug report will be assigned to a developer who will determine what needs to -be done to correct the problem. You will receive an update each time action is -taken on the bug. +Each issue report will be reviewed by a developer who will determine what needs to +be done to correct the problem. You will receive an update each time an action is +taken on the issue. .. seealso:: @@ -95,6 +96,6 @@ patching Python in the `Python Developer's Guide`_. If you have questions, the `core-mentorship mailing list`_ is a friendly place to get answers to any and all questions pertaining to the process of fixing issues in Python. -.. _Documentation bugs: https://bugs.python.org/issue?@filter=status&@filter=components&components=4&status=1&@columns=id,activity,title,status&@sort=-activity +.. _Documentation bugs: https://github.com/python/cpython/issues?q=is%3Aissue+is%3Aopen+label%3Adocs .. _Python Developer's Guide: https://devguide.python.org/ .. _core-mentorship mailing list: https://mail.python.org/mailman3/lists/core-mentorship.python.org/ From webhook-mailer at python.org Mon Apr 11 11:59:47 2022 From: webhook-mailer at python.org (ezio-melotti) Date: Mon, 11 Apr 2022 15:59:47 -0000 Subject: [Python-checkins] Update Sphinx bpo role to use redirect URI. (#32342) Message-ID: https://github.com/python/cpython/commit/08cfe079503ffd19d8b7ab324f0fdb1c6b150ca8 commit: 08cfe079503ffd19d8b7ab324f0fdb1c6b150ca8 branch: main author: Ezio Melotti committer: ezio-melotti date: 2022-04-11T17:59:35+02:00 summary: Update Sphinx bpo role to use redirect URI. (#32342) files: M Doc/tools/extensions/pyspecific.py diff --git a/Doc/tools/extensions/pyspecific.py b/Doc/tools/extensions/pyspecific.py index ff522c98d0f62..9fb54bc6041ab 100644 --- a/Doc/tools/extensions/pyspecific.py +++ b/Doc/tools/extensions/pyspecific.py @@ -43,7 +43,7 @@ import suspicious -ISSUE_URI = 'https://bugs.python.org/issue%s' +ISSUE_URI = 'https://bugs.python.org/issue?@action=redirect&bpo=%s' SOURCE_URI = 'https://github.com/python/cpython/tree/main/%s' # monkey-patch reST parser to disable alphabetic and roman enumerated lists From webhook-mailer at python.org Mon Apr 11 12:34:26 2022 From: webhook-mailer at python.org (ambv) Date: Mon, 11 Apr 2022 16:34:26 -0000 Subject: [Python-checkins] Remove dead "Check PRs with 'CLA not signed' label" (#91429) Message-ID: https://github.com/python/cpython/commit/a8abb76af92a5f6883a640a987f9f45b47ec852b commit: a8abb76af92a5f6883a640a987f9f45b47ec852b branch: main author: Oleg Iarygin committer: ambv date: 2022-04-11T18:34:17+02:00 summary: Remove dead "Check PRs with 'CLA not signed' label" (#91429) files: M .github/workflows/stale.yml diff --git a/.github/workflows/stale.yml b/.github/workflows/stale.yml index 25e29e83dff2b..72a7a0ba9c83c 100644 --- a/.github/workflows/stale.yml +++ b/.github/workflows/stale.yml @@ -14,27 +14,13 @@ jobs: runs-on: ubuntu-latest steps: - - name: "Check PRs with 'CLA signed' label" + - name: "Check PRs" uses: actions/stale at v4 with: repo-token: ${{ secrets.GITHUB_TOKEN }} - only-pr-labels: 'CLA signed' stale-pr-message: 'This PR is stale because it has been open for 30 days with no activity.' stale-pr-label: 'stale' days-before-stale: 30 days-before-close: -1 ascending: true operations-per-run: 120 - - - name: "Check PRs with 'CLA not signed' label" - uses: actions/stale at v4 - with: - repo-token: ${{ secrets.GITHUB_TOKEN }} - only-pr-labels: 'CLA not signed' - stale-pr-message: 'This PR is stale because it has been open for 30 days with no activity. If the CLA is not signed within 14 days, it will be closed. See also https://devguide.python.org/pullrequest/#licensing' - stale-pr-label: 'stale' - close-pr-message: 'Closing this stale PR because the CLA is still not signed.' - days-before-stale: 30 - days-before-close: 14 - ascending: true - operations-per-run: 120 From webhook-mailer at python.org Mon Apr 11 16:07:19 2022 From: webhook-mailer at python.org (sweeneyde) Date: Mon, 11 Apr 2022 20:07:19 -0000 Subject: [Python-checkins] gh-91117: Ensure integer mod and pow operations use cached small ints (GH-31843) Message-ID: https://github.com/python/cpython/commit/8be8949116e1812df5e329a18647ebdea7fb9c6c commit: 8be8949116e1812df5e329a18647ebdea7fb9c6c branch: main author: Dennis Sweeney <36520290+sweeneyde at users.noreply.github.com> committer: sweeneyde <36520290+sweeneyde at users.noreply.github.com> date: 2022-04-11T16:07:09-04:00 summary: gh-91117: Ensure integer mod and pow operations use cached small ints (GH-31843) files: A Misc/NEWS.d/next/Core and Builtins/2022-03-13-08-23-17.bpo-46961.SgGCkG.rst M Lib/test/test_long.py M Objects/longobject.c diff --git a/Lib/test/test_long.py b/Lib/test/test_long.py index e68dfb4c542ea..2de4526ff33fd 100644 --- a/Lib/test/test_long.py +++ b/Lib/test/test_long.py @@ -1024,6 +1024,48 @@ def test_small_ints_in_huge_calculation(self): self.assertIs(a + b, 1) self.assertIs(c - a, 1) + @support.cpython_only + def test_pow_uses_cached_small_ints(self): + self.assertIs(pow(10, 3, 998), 2) + self.assertIs(10 ** 3 % 998, 2) + a, p, m = 10, 3, 998 + self.assertIs(a ** p % m, 2) + + self.assertIs(pow(2, 31, 2 ** 31 - 1), 1) + self.assertIs(2 ** 31 % (2 ** 31 - 1), 1) + a, p, m = 2, 31, 2 ** 31 - 1 + self.assertIs(a ** p % m, 1) + + self.assertIs(pow(2, 100, 2**100 - 3), 3) + self.assertIs(2 ** 100 % (2 ** 100 - 3), 3) + a, p, m = 2, 100, 2**100 - 3 + self.assertIs(a ** p % m, 3) + + @support.cpython_only + def test_divmod_uses_cached_small_ints(self): + big = 10 ** 100 + + self.assertIs((big + 1) % big, 1) + self.assertIs((big + 1) // big, 1) + self.assertIs(big // (big // 2), 2) + self.assertIs(big // (big // -4), -4) + + q, r = divmod(2 * big + 3, big) + self.assertIs(q, 2) + self.assertIs(r, 3) + + q, r = divmod(-4 * big + 100, big) + self.assertIs(q, -4) + self.assertIs(r, 100) + + q, r = divmod(3 * (-big) - 1, -big) + self.assertIs(q, 3) + self.assertIs(r, -1) + + q, r = divmod(3 * big - 1, -big) + self.assertIs(q, -3) + self.assertIs(r, -1) + def test_small_ints(self): for i in range(-5, 257): self.assertIs(i, i + 0) diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-03-13-08-23-17.bpo-46961.SgGCkG.rst b/Misc/NEWS.d/next/Core and Builtins/2022-03-13-08-23-17.bpo-46961.SgGCkG.rst new file mode 100644 index 0000000000000..8753377f6b8ed --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2022-03-13-08-23-17.bpo-46961.SgGCkG.rst @@ -0,0 +1 @@ +Integer mod/remainder operations, including the three-argument form of :func:`pow`, now consistently return ints from the global small integer cache when applicable. diff --git a/Objects/longobject.c b/Objects/longobject.c index f85ef241a445d..cc4aef31d743c 100644 --- a/Objects/longobject.c +++ b/Objects/longobject.c @@ -2679,6 +2679,7 @@ long_divrem(PyLongObject *a, PyLongObject *b, } else { z = x_divrem(a, b, prem); + *prem = maybe_small_long(*prem); if (z == NULL) return -1; } @@ -2732,6 +2733,7 @@ long_rem(PyLongObject *a, PyLongObject *b, PyLongObject **prem) else { /* Slow path using divrem. */ Py_XDECREF(x_divrem(a, b, prem)); + *prem = maybe_small_long(*prem); if (*prem == NULL) return -1; } From webhook-mailer at python.org Mon Apr 11 18:02:45 2022 From: webhook-mailer at python.org (brettcannon) Date: Mon, 11 Apr 2022 22:02:45 -0000 Subject: [Python-checkins] gh-47061: Deprecate `chunk` (GH-91419) Message-ID: https://github.com/python/cpython/commit/3869a839d5f14a91978c6158a03c68fac5e938dd commit: 3869a839d5f14a91978c6158a03c68fac5e938dd branch: main author: Brett Cannon committer: brettcannon date: 2022-04-11T15:02:41-07:00 summary: gh-47061: Deprecate `chunk` (GH-91419) files: A Misc/NEWS.d/next/Library/2022-04-10-11-11-33.gh-issue-91217.K82AuH.rst M Doc/whatsnew/3.11.rst M Lib/aifc.py M Lib/chunk.py M Lib/wave.py diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index df0b0a7fbebec..cc358b4ffdb2a 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -850,6 +850,7 @@ Deprecated * :mod:`audioop` * :mod:`cgi` * :mod:`cgitb` + * :mod:`chunk` (Contributed by Brett Cannon in :issue:`47061`.) diff --git a/Lib/aifc.py b/Lib/aifc.py index 314cfd230d6cb..5254987e22bc1 100644 --- a/Lib/aifc.py +++ b/Lib/aifc.py @@ -255,7 +255,9 @@ def _write_float(f, x): _write_ulong(f, himant) _write_ulong(f, lomant) -from chunk import Chunk +with warnings.catch_warnings(): + warnings.simplefilter("ignore", DeprecationWarning) + from chunk import Chunk from collections import namedtuple _aifc_params = namedtuple('_aifc_params', diff --git a/Lib/chunk.py b/Lib/chunk.py index 870c39fe7f503..618781efd11e7 100644 --- a/Lib/chunk.py +++ b/Lib/chunk.py @@ -48,6 +48,10 @@ default is 1, i.e. aligned. """ +import warnings + +warnings._deprecated(__name__, remove=(3, 13)) + class Chunk: def __init__(self, file, align=True, bigendian=True, inclheader=False): import struct diff --git a/Lib/wave.py b/Lib/wave.py index 47a233df0a3ab..9a4557487b6e6 100644 --- a/Lib/wave.py +++ b/Lib/wave.py @@ -71,7 +71,6 @@ is destroyed. """ -from chunk import Chunk from collections import namedtuple import builtins import struct @@ -100,6 +99,119 @@ def _byteswap(data, width): return bytes(swapped_data) +class _Chunk: + def __init__(self, file, align=True, bigendian=True, inclheader=False): + import struct + self.closed = False + self.align = align # whether to align to word (2-byte) boundaries + if bigendian: + strflag = '>' + else: + strflag = '<' + self.file = file + self.chunkname = file.read(4) + if len(self.chunkname) < 4: + raise EOFError + try: + self.chunksize = struct.unpack_from(strflag+'L', file.read(4))[0] + except struct.error: + raise EOFError from None + if inclheader: + self.chunksize = self.chunksize - 8 # subtract header + self.size_read = 0 + try: + self.offset = self.file.tell() + except (AttributeError, OSError): + self.seekable = False + else: + self.seekable = True + + def getname(self): + """Return the name (ID) of the current chunk.""" + return self.chunkname + + def close(self): + if not self.closed: + try: + self.skip() + finally: + self.closed = True + + def seek(self, pos, whence=0): + """Seek to specified position into the chunk. + Default position is 0 (start of chunk). + If the file is not seekable, this will result in an error. + """ + + if self.closed: + raise ValueError("I/O operation on closed file") + if not self.seekable: + raise OSError("cannot seek") + if whence == 1: + pos = pos + self.size_read + elif whence == 2: + pos = pos + self.chunksize + if pos < 0 or pos > self.chunksize: + raise RuntimeError + self.file.seek(self.offset + pos, 0) + self.size_read = pos + + def tell(self): + if self.closed: + raise ValueError("I/O operation on closed file") + return self.size_read + + def read(self, size=-1): + """Read at most size bytes from the chunk. + If size is omitted or negative, read until the end + of the chunk. + """ + + if self.closed: + raise ValueError("I/O operation on closed file") + if self.size_read >= self.chunksize: + return b'' + if size < 0: + size = self.chunksize - self.size_read + if size > self.chunksize - self.size_read: + size = self.chunksize - self.size_read + data = self.file.read(size) + self.size_read = self.size_read + len(data) + if self.size_read == self.chunksize and \ + self.align and \ + (self.chunksize & 1): + dummy = self.file.read(1) + self.size_read = self.size_read + len(dummy) + return data + + def skip(self): + """Skip the rest of the chunk. + If you are not interested in the contents of the chunk, + this method should be called so that the file points to + the start of the next chunk. + """ + + if self.closed: + raise ValueError("I/O operation on closed file") + if self.seekable: + try: + n = self.chunksize - self.size_read + # maybe fix alignment + if self.align and (self.chunksize & 1): + n = n + 1 + self.file.seek(n, 1) + self.size_read = self.size_read + n + return + except OSError: + pass + while self.size_read < self.chunksize: + n = min(8192, self.chunksize - self.size_read) + dummy = self.read(n) + if not dummy: + raise EOFError + + + class Wave_read: """Variables used in this class: @@ -134,7 +246,7 @@ class Wave_read: def initfp(self, file): self._convert = None self._soundpos = 0 - self._file = Chunk(file, bigendian = 0) + self._file = _Chunk(file, bigendian = 0) if self._file.getname() != b'RIFF': raise Error('file does not start with RIFF id') if self._file.read(4) != b'WAVE': @@ -144,7 +256,7 @@ def initfp(self, file): while 1: self._data_seek_needed = 1 try: - chunk = Chunk(self._file, bigendian = 0) + chunk = _Chunk(self._file, bigendian = 0) except EOFError: break chunkname = chunk.getname() diff --git a/Misc/NEWS.d/next/Library/2022-04-10-11-11-33.gh-issue-91217.K82AuH.rst b/Misc/NEWS.d/next/Library/2022-04-10-11-11-33.gh-issue-91217.K82AuH.rst new file mode 100644 index 0000000000000..0181bbcce9cd8 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-10-11-11-33.gh-issue-91217.K82AuH.rst @@ -0,0 +1 @@ +Deprecate the chunk module. From webhook-mailer at python.org Mon Apr 11 18:33:10 2022 From: webhook-mailer at python.org (sweeneyde) Date: Mon, 11 Apr 2022 22:33:10 -0000 Subject: [Python-checkins] gh-91428: Add _PyOpcode_OpName to opcode.h of debug builds (GH-91430) Message-ID: https://github.com/python/cpython/commit/8a35ce3796e92f8a826955753920ca0567dbe343 commit: 8a35ce3796e92f8a826955753920ca0567dbe343 branch: main author: Dennis Sweeney <36520290+sweeneyde at users.noreply.github.com> committer: sweeneyde <36520290+sweeneyde at users.noreply.github.com> date: 2022-04-11T18:33:00-04:00 summary: gh-91428: Add _PyOpcode_OpName to opcode.h of debug builds (GH-91430) files: A Misc/NEWS.d/next/Core and Builtins/2022-04-10-18-47-21.gh-issue-91428.ZewV-M.rst M Include/opcode.h M Tools/scripts/generate_opcode_h.py diff --git a/Include/opcode.h b/Include/opcode.h index 57a72a6e892a8..ca4a18de10716 100644 --- a/Include/opcode.h +++ b/Include/opcode.h @@ -446,6 +446,121 @@ const uint8_t _PyOpcode_Deopt[256] = { #define NB_INPLACE_TRUE_DIVIDE 24 #define NB_INPLACE_XOR 25 +#ifdef Py_DEBUG +static const char *const _PyOpcode_OpName[256] = { + [CACHE] = "CACHE", + [POP_TOP] = "POP_TOP", + [PUSH_NULL] = "PUSH_NULL", + [NOP] = "NOP", + [UNARY_POSITIVE] = "UNARY_POSITIVE", + [UNARY_NEGATIVE] = "UNARY_NEGATIVE", + [UNARY_NOT] = "UNARY_NOT", + [UNARY_INVERT] = "UNARY_INVERT", + [BINARY_SUBSCR] = "BINARY_SUBSCR", + [GET_LEN] = "GET_LEN", + [MATCH_MAPPING] = "MATCH_MAPPING", + [MATCH_SEQUENCE] = "MATCH_SEQUENCE", + [MATCH_KEYS] = "MATCH_KEYS", + [PUSH_EXC_INFO] = "PUSH_EXC_INFO", + [CHECK_EXC_MATCH] = "CHECK_EXC_MATCH", + [CHECK_EG_MATCH] = "CHECK_EG_MATCH", + [WITH_EXCEPT_START] = "WITH_EXCEPT_START", + [GET_AITER] = "GET_AITER", + [GET_ANEXT] = "GET_ANEXT", + [BEFORE_ASYNC_WITH] = "BEFORE_ASYNC_WITH", + [BEFORE_WITH] = "BEFORE_WITH", + [END_ASYNC_FOR] = "END_ASYNC_FOR", + [STORE_SUBSCR] = "STORE_SUBSCR", + [DELETE_SUBSCR] = "DELETE_SUBSCR", + [GET_ITER] = "GET_ITER", + [GET_YIELD_FROM_ITER] = "GET_YIELD_FROM_ITER", + [PRINT_EXPR] = "PRINT_EXPR", + [LOAD_BUILD_CLASS] = "LOAD_BUILD_CLASS", + [LOAD_ASSERTION_ERROR] = "LOAD_ASSERTION_ERROR", + [RETURN_GENERATOR] = "RETURN_GENERATOR", + [LIST_TO_TUPLE] = "LIST_TO_TUPLE", + [RETURN_VALUE] = "RETURN_VALUE", + [IMPORT_STAR] = "IMPORT_STAR", + [SETUP_ANNOTATIONS] = "SETUP_ANNOTATIONS", + [YIELD_VALUE] = "YIELD_VALUE", + [ASYNC_GEN_WRAP] = "ASYNC_GEN_WRAP", + [PREP_RERAISE_STAR] = "PREP_RERAISE_STAR", + [POP_EXCEPT] = "POP_EXCEPT", + [STORE_NAME] = "STORE_NAME", + [DELETE_NAME] = "DELETE_NAME", + [UNPACK_SEQUENCE] = "UNPACK_SEQUENCE", + [FOR_ITER] = "FOR_ITER", + [UNPACK_EX] = "UNPACK_EX", + [STORE_ATTR] = "STORE_ATTR", + [DELETE_ATTR] = "DELETE_ATTR", + [STORE_GLOBAL] = "STORE_GLOBAL", + [DELETE_GLOBAL] = "DELETE_GLOBAL", + [SWAP] = "SWAP", + [LOAD_CONST] = "LOAD_CONST", + [LOAD_NAME] = "LOAD_NAME", + [BUILD_TUPLE] = "BUILD_TUPLE", + [BUILD_LIST] = "BUILD_LIST", + [BUILD_SET] = "BUILD_SET", + [BUILD_MAP] = "BUILD_MAP", + [LOAD_ATTR] = "LOAD_ATTR", + [COMPARE_OP] = "COMPARE_OP", + [IMPORT_NAME] = "IMPORT_NAME", + [IMPORT_FROM] = "IMPORT_FROM", + [JUMP_FORWARD] = "JUMP_FORWARD", + [JUMP_IF_FALSE_OR_POP] = "JUMP_IF_FALSE_OR_POP", + [JUMP_IF_TRUE_OR_POP] = "JUMP_IF_TRUE_OR_POP", + [POP_JUMP_FORWARD_IF_FALSE] = "POP_JUMP_FORWARD_IF_FALSE", + [POP_JUMP_FORWARD_IF_TRUE] = "POP_JUMP_FORWARD_IF_TRUE", + [LOAD_GLOBAL] = "LOAD_GLOBAL", + [IS_OP] = "IS_OP", + [CONTAINS_OP] = "CONTAINS_OP", + [RERAISE] = "RERAISE", + [COPY] = "COPY", + [BINARY_OP] = "BINARY_OP", + [SEND] = "SEND", + [LOAD_FAST] = "LOAD_FAST", + [STORE_FAST] = "STORE_FAST", + [DELETE_FAST] = "DELETE_FAST", + [POP_JUMP_FORWARD_IF_NOT_NONE] = "POP_JUMP_FORWARD_IF_NOT_NONE", + [POP_JUMP_FORWARD_IF_NONE] = "POP_JUMP_FORWARD_IF_NONE", + [RAISE_VARARGS] = "RAISE_VARARGS", + [GET_AWAITABLE] = "GET_AWAITABLE", + [MAKE_FUNCTION] = "MAKE_FUNCTION", + [BUILD_SLICE] = "BUILD_SLICE", + [JUMP_BACKWARD_NO_INTERRUPT] = "JUMP_BACKWARD_NO_INTERRUPT", + [MAKE_CELL] = "MAKE_CELL", + [LOAD_CLOSURE] = "LOAD_CLOSURE", + [LOAD_DEREF] = "LOAD_DEREF", + [STORE_DEREF] = "STORE_DEREF", + [DELETE_DEREF] = "DELETE_DEREF", + [JUMP_BACKWARD] = "JUMP_BACKWARD", + [CALL_FUNCTION_EX] = "CALL_FUNCTION_EX", + [EXTENDED_ARG] = "EXTENDED_ARG", + [LIST_APPEND] = "LIST_APPEND", + [SET_ADD] = "SET_ADD", + [MAP_ADD] = "MAP_ADD", + [LOAD_CLASSDEREF] = "LOAD_CLASSDEREF", + [COPY_FREE_VARS] = "COPY_FREE_VARS", + [RESUME] = "RESUME", + [MATCH_CLASS] = "MATCH_CLASS", + [FORMAT_VALUE] = "FORMAT_VALUE", + [BUILD_CONST_KEY_MAP] = "BUILD_CONST_KEY_MAP", + [BUILD_STRING] = "BUILD_STRING", + [LOAD_METHOD] = "LOAD_METHOD", + [LIST_EXTEND] = "LIST_EXTEND", + [SET_UPDATE] = "SET_UPDATE", + [DICT_MERGE] = "DICT_MERGE", + [DICT_UPDATE] = "DICT_UPDATE", + [PRECALL] = "PRECALL", + [CALL] = "CALL", + [KW_NAMES] = "KW_NAMES", + [POP_JUMP_BACKWARD_IF_NOT_NONE] = "POP_JUMP_BACKWARD_IF_NOT_NONE", + [POP_JUMP_BACKWARD_IF_NONE] = "POP_JUMP_BACKWARD_IF_NONE", + [POP_JUMP_BACKWARD_IF_FALSE] = "POP_JUMP_BACKWARD_IF_FALSE", + [POP_JUMP_BACKWARD_IF_TRUE] = "POP_JUMP_BACKWARD_IF_TRUE", +}; +#endif + #define HAS_ARG(op) ((op) >= HAVE_ARGUMENT) /* Reserve some bytecodes for internal use in the compiler. diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-04-10-18-47-21.gh-issue-91428.ZewV-M.rst b/Misc/NEWS.d/next/Core and Builtins/2022-04-10-18-47-21.gh-issue-91428.ZewV-M.rst new file mode 100644 index 0000000000000..3f17a406f8f57 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2022-04-10-18-47-21.gh-issue-91428.ZewV-M.rst @@ -0,0 +1,5 @@ +Add ``static const char *const _PyOpcode_OpName[256] = {...};`` to +``opcode.h`` for debug builds to assist in debugging the Python interpreter. +It is now more convenient to make various forms of debugging output more +human-readable by including opcode names rather than just the corresponding +decimal digits. diff --git a/Tools/scripts/generate_opcode_h.py b/Tools/scripts/generate_opcode_h.py index 3b79dc6b7359f..6b5cc7d7a7d3d 100644 --- a/Tools/scripts/generate_opcode_h.py +++ b/Tools/scripts/generate_opcode_h.py @@ -108,6 +108,14 @@ def main(opcode_py, outfile='Include/opcode.h'): for i, (op, _) in enumerate(opcode["_nb_ops"]): fobj.write(DEFINE.format(op, i)) + fobj.write("\n") + fobj.write("#ifdef Py_DEBUG\n") + fobj.write("static const char *const _PyOpcode_OpName[256] = {\n") + for name in opmap: + fobj.write(f''' [{name}] = "{name}",\n''') + fobj.write("};\n") + fobj.write("#endif\n") + fobj.write(footer) From webhook-mailer at python.org Mon Apr 11 20:02:27 2022 From: webhook-mailer at python.org (brettcannon) Date: Tue, 12 Apr 2022 00:02:27 -0000 Subject: [Python-checkins] gh-91217: deprecate crypt (GH-91459) Message-ID: https://github.com/python/cpython/commit/f45aa8f304a12990c2ca687f2088f04b07906033 commit: f45aa8f304a12990c2ca687f2088f04b07906033 branch: main author: Brett Cannon committer: brettcannon date: 2022-04-11T17:02:19-07:00 summary: gh-91217: deprecate crypt (GH-91459) files: A Misc/NEWS.d/next/Library/2022-04-11-16-13-26.gh-issue-91217.2rf8rc.rst M Doc/whatsnew/3.11.rst M Lib/crypt.py M Lib/test/test_crypt.py diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index cc358b4ffdb2a..354e2112338db 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -851,6 +851,7 @@ Deprecated * :mod:`cgi` * :mod:`cgitb` * :mod:`chunk` + * :mod:`crypt` (Contributed by Brett Cannon in :issue:`47061`.) diff --git a/Lib/crypt.py b/Lib/crypt.py index 33dbc46bb3e96..46c3de8474bf1 100644 --- a/Lib/crypt.py +++ b/Lib/crypt.py @@ -12,10 +12,14 @@ import errno import string as _string +import warnings from random import SystemRandom as _SystemRandom from collections import namedtuple as _namedtuple +warnings._deprecated(__name__, remove=(3, 13)) + + _saltchars = _string.ascii_letters + _string.digits + './' _sr = _SystemRandom() diff --git a/Lib/test/test_crypt.py b/Lib/test/test_crypt.py index 877c575c5534a..b2a5ce6db0919 100644 --- a/Lib/test/test_crypt.py +++ b/Lib/test/test_crypt.py @@ -1,12 +1,12 @@ import sys import unittest -from test.support import check_sanitizer +from test.support import check_sanitizer, warnings_helper try: if check_sanitizer(address=True, memory=True): raise unittest.SkipTest("The crypt module SEGFAULTs on ASAN/MSAN builds") - import crypt + crypt = warnings_helper.import_deprecated("crypt") IMPORT_ERROR = None except ImportError as ex: if sys.platform != 'win32': diff --git a/Misc/NEWS.d/next/Library/2022-04-11-16-13-26.gh-issue-91217.2rf8rc.rst b/Misc/NEWS.d/next/Library/2022-04-11-16-13-26.gh-issue-91217.2rf8rc.rst new file mode 100644 index 0000000000000..067783f85c28a --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-11-16-13-26.gh-issue-91217.2rf8rc.rst @@ -0,0 +1 @@ +Deprecate the crypt module. From webhook-mailer at python.org Mon Apr 11 20:56:07 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Tue, 12 Apr 2022 00:56:07 -0000 Subject: [Python-checkins] gh-79097: Add support for aggregate window functions in sqlite3 (GH-20903) Message-ID: https://github.com/python/cpython/commit/9ebcece82fe11b87cc3d6e6b4c439aab9e3ab1e6 commit: 9ebcece82fe11b87cc3d6e6b4c439aab9e3ab1e6 branch: main author: Erlend Egeberg Aasland committer: JelleZijlstra date: 2022-04-11T17:55:59-07:00 summary: gh-79097: Add support for aggregate window functions in sqlite3 (GH-20903) files: A Doc/includes/sqlite3/sumintwindow.py A Misc/NEWS.d/next/Library/2020-05-24-23-52-03.bpo-40617.lycF9q.rst M Doc/library/sqlite3.rst M Doc/whatsnew/3.11.rst M Lib/test/test_sqlite3/test_dbapi.py M Lib/test/test_sqlite3/test_userfunctions.py M Modules/_sqlite/clinic/connection.c.h M Modules/_sqlite/connection.c M Modules/_sqlite/module.c M Modules/_sqlite/module.h diff --git a/Doc/includes/sqlite3/sumintwindow.py b/Doc/includes/sqlite3/sumintwindow.py new file mode 100644 index 0000000000000..0e915d6cc6ae6 --- /dev/null +++ b/Doc/includes/sqlite3/sumintwindow.py @@ -0,0 +1,46 @@ +# Example taken from https://www.sqlite.org/windowfunctions.html#udfwinfunc +import sqlite3 + + +class WindowSumInt: + def __init__(self): + self.count = 0 + + def step(self, value): + """Adds a row to the current window.""" + self.count += value + + def value(self): + """Returns the current value of the aggregate.""" + return self.count + + def inverse(self, value): + """Removes a row from the current window.""" + self.count -= value + + def finalize(self): + """Returns the final value of the aggregate. + + Any clean-up actions should be placed here. + """ + return self.count + + +con = sqlite3.connect(":memory:") +cur = con.execute("create table test(x, y)") +values = [ + ("a", 4), + ("b", 5), + ("c", 3), + ("d", 8), + ("e", 1), +] +cur.executemany("insert into test values(?, ?)", values) +con.create_window_function("sumint", 1, WindowSumInt) +cur.execute(""" + select x, sumint(y) over ( + order by x rows between 1 preceding and 1 following + ) as sum_y + from test order by x +""") +print(cur.fetchall()) diff --git a/Doc/library/sqlite3.rst b/Doc/library/sqlite3.rst index 852b68437a265..60dfbefd2e255 100644 --- a/Doc/library/sqlite3.rst +++ b/Doc/library/sqlite3.rst @@ -473,6 +473,35 @@ Connection Objects .. literalinclude:: ../includes/sqlite3/mysumaggr.py + .. method:: create_window_function(name, num_params, aggregate_class, /) + + Creates user-defined aggregate window function *name*. + + *aggregate_class* must implement the following methods: + + * ``step``: adds a row to the current window + * ``value``: returns the current value of the aggregate + * ``inverse``: removes a row from the current window + * ``finalize``: returns the final value of the aggregate + + ``step`` and ``value`` accept *num_params* number of parameters, + unless *num_params* is ``-1``, in which case they may take any number of + arguments. ``finalize`` and ``value`` can return any of the types + supported by SQLite: + :class:`bytes`, :class:`str`, :class:`int`, :class:`float`, and + :const:`None`. Call :meth:`create_window_function` with + *aggregate_class* set to :const:`None` to clear window function *name*. + + Aggregate window functions are supported by SQLite 3.25.0 and higher. + :exc:`NotSupportedError` will be raised if used with older versions. + + .. versionadded:: 3.11 + + Example: + + .. literalinclude:: ../includes/sqlite3/sumintwindow.py + + .. method:: create_collation(name, callable) Creates a collation with the specified *name* and *callable*. The callable will diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index 354e2112338db..d803801f273c5 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -389,6 +389,10 @@ sqlite3 serializing and deserializing databases. (Contributed by Erlend E. Aasland in :issue:`41930`.) +* Add :meth:`~sqlite3.Connection.create_window_function` to + :class:`sqlite3.Connection` for creating aggregate window functions. + (Contributed by Erlend E. Aasland in :issue:`34916`.) + sys --- diff --git a/Lib/test/test_sqlite3/test_dbapi.py b/Lib/test/test_sqlite3/test_dbapi.py index 02482816cb933..2d2e58a3d44f5 100644 --- a/Lib/test/test_sqlite3/test_dbapi.py +++ b/Lib/test/test_sqlite3/test_dbapi.py @@ -1084,6 +1084,8 @@ def test_check_connection_thread(self): if hasattr(sqlite.Connection, "serialize"): fns.append(lambda: self.con.serialize()) fns.append(lambda: self.con.deserialize(b"")) + if sqlite.sqlite_version_info >= (3, 25, 0): + fns.append(lambda: self.con.create_window_function("foo", 0, None)) for fn in fns: with self.subTest(fn=fn): diff --git a/Lib/test/test_sqlite3/test_userfunctions.py b/Lib/test/test_sqlite3/test_userfunctions.py index 9070c9e01b25a..0970b0378ad61 100644 --- a/Lib/test/test_sqlite3/test_userfunctions.py +++ b/Lib/test/test_sqlite3/test_userfunctions.py @@ -27,9 +27,9 @@ import re import sys import unittest -import unittest.mock import sqlite3 as sqlite +from unittest.mock import Mock, patch from test.support import bigmemtest, catch_unraisable_exception, gc_collect from test.test_sqlite3.test_dbapi import cx_limit @@ -393,7 +393,7 @@ def append_result(arg): # indices, which allows testing based on syntax, iso. the query optimizer. @unittest.skipIf(sqlite.sqlite_version_info < (3, 8, 3), "Requires SQLite 3.8.3 or higher") def test_func_non_deterministic(self): - mock = unittest.mock.Mock(return_value=None) + mock = Mock(return_value=None) self.con.create_function("nondeterministic", 0, mock, deterministic=False) if sqlite.sqlite_version_info < (3, 15, 0): self.con.execute("select nondeterministic() = nondeterministic()") @@ -404,7 +404,7 @@ def test_func_non_deterministic(self): @unittest.skipIf(sqlite.sqlite_version_info < (3, 8, 3), "Requires SQLite 3.8.3 or higher") def test_func_deterministic(self): - mock = unittest.mock.Mock(return_value=None) + mock = Mock(return_value=None) self.con.create_function("deterministic", 0, mock, deterministic=True) if sqlite.sqlite_version_info < (3, 15, 0): self.con.execute("select deterministic() = deterministic()") @@ -482,6 +482,164 @@ def test_func_return_illegal_value(self): self.con.execute, "select badreturn()") +class WindowSumInt: + def __init__(self): + self.count = 0 + + def step(self, value): + self.count += value + + def value(self): + return self.count + + def inverse(self, value): + self.count -= value + + def finalize(self): + return self.count + +class BadWindow(Exception): + pass + + + at unittest.skipIf(sqlite.sqlite_version_info < (3, 25, 0), + "Requires SQLite 3.25.0 or newer") +class WindowFunctionTests(unittest.TestCase): + def setUp(self): + self.con = sqlite.connect(":memory:") + self.cur = self.con.cursor() + + # Test case taken from https://www.sqlite.org/windowfunctions.html#udfwinfunc + values = [ + ("a", 4), + ("b", 5), + ("c", 3), + ("d", 8), + ("e", 1), + ] + with self.con: + self.con.execute("create table test(x, y)") + self.con.executemany("insert into test values(?, ?)", values) + self.expected = [ + ("a", 9), + ("b", 12), + ("c", 16), + ("d", 12), + ("e", 9), + ] + self.query = """ + select x, %s(y) over ( + order by x rows between 1 preceding and 1 following + ) as sum_y + from test order by x + """ + self.con.create_window_function("sumint", 1, WindowSumInt) + + def test_win_sum_int(self): + self.cur.execute(self.query % "sumint") + self.assertEqual(self.cur.fetchall(), self.expected) + + def test_win_error_on_create(self): + self.assertRaises(sqlite.ProgrammingError, + self.con.create_window_function, + "shouldfail", -100, WindowSumInt) + + @with_tracebacks(BadWindow) + def test_win_exception_in_method(self): + for meth in "__init__", "step", "value", "inverse": + with self.subTest(meth=meth): + with patch.object(WindowSumInt, meth, side_effect=BadWindow): + name = f"exc_{meth}" + self.con.create_window_function(name, 1, WindowSumInt) + msg = f"'{meth}' method raised error" + with self.assertRaisesRegex(sqlite.OperationalError, msg): + self.cur.execute(self.query % name) + self.cur.fetchall() + + @with_tracebacks(BadWindow) + def test_win_exception_in_finalize(self): + # Note: SQLite does not (as of version 3.38.0) propagate finalize + # callback errors to sqlite3_step(); this implies that OperationalError + # is _not_ raised. + with patch.object(WindowSumInt, "finalize", side_effect=BadWindow): + name = f"exception_in_finalize" + self.con.create_window_function(name, 1, WindowSumInt) + self.cur.execute(self.query % name) + self.cur.fetchall() + + @with_tracebacks(AttributeError) + def test_win_missing_method(self): + class MissingValue: + def step(self, x): pass + def inverse(self, x): pass + def finalize(self): return 42 + + class MissingInverse: + def step(self, x): pass + def value(self): return 42 + def finalize(self): return 42 + + class MissingStep: + def value(self): return 42 + def inverse(self, x): pass + def finalize(self): return 42 + + dataset = ( + ("step", MissingStep), + ("value", MissingValue), + ("inverse", MissingInverse), + ) + for meth, cls in dataset: + with self.subTest(meth=meth, cls=cls): + name = f"exc_{meth}" + self.con.create_window_function(name, 1, cls) + with self.assertRaisesRegex(sqlite.OperationalError, + f"'{meth}' method not defined"): + self.cur.execute(self.query % name) + self.cur.fetchall() + + @with_tracebacks(AttributeError) + def test_win_missing_finalize(self): + # Note: SQLite does not (as of version 3.38.0) propagate finalize + # callback errors to sqlite3_step(); this implies that OperationalError + # is _not_ raised. + class MissingFinalize: + def step(self, x): pass + def value(self): return 42 + def inverse(self, x): pass + + name = "missing_finalize" + self.con.create_window_function(name, 1, MissingFinalize) + self.cur.execute(self.query % name) + self.cur.fetchall() + + def test_win_clear_function(self): + self.con.create_window_function("sumint", 1, None) + self.assertRaises(sqlite.OperationalError, self.cur.execute, + self.query % "sumint") + + def test_win_redefine_function(self): + # Redefine WindowSumInt; adjust the expected results accordingly. + class Redefined(WindowSumInt): + def step(self, value): self.count += value * 2 + def inverse(self, value): self.count -= value * 2 + expected = [(v[0], v[1]*2) for v in self.expected] + + self.con.create_window_function("sumint", 1, Redefined) + self.cur.execute(self.query % "sumint") + self.assertEqual(self.cur.fetchall(), expected) + + def test_win_error_value_return(self): + class ErrorValueReturn: + def __init__(self): pass + def step(self, x): pass + def value(self): return 1 << 65 + + self.con.create_window_function("err_val_ret", 1, ErrorValueReturn) + self.assertRaisesRegex(sqlite.DataError, "string or blob too big", + self.cur.execute, self.query % "err_val_ret") + + class AggregateTests(unittest.TestCase): def setUp(self): self.con = sqlite.connect(":memory:") @@ -527,10 +685,10 @@ def test_aggr_no_step(self): def test_aggr_no_finalize(self): cur = self.con.cursor() - with self.assertRaises(sqlite.OperationalError) as cm: + msg = "user-defined aggregate's 'finalize' method not defined" + with self.assertRaisesRegex(sqlite.OperationalError, msg): cur.execute("select nofinalize(t) from test") val = cur.fetchone()[0] - self.assertEqual(str(cm.exception), "user-defined aggregate's 'finalize' method raised error") @with_tracebacks(ZeroDivisionError, name="AggrExceptionInInit") def test_aggr_exception_in_init(self): diff --git a/Misc/NEWS.d/next/Library/2020-05-24-23-52-03.bpo-40617.lycF9q.rst b/Misc/NEWS.d/next/Library/2020-05-24-23-52-03.bpo-40617.lycF9q.rst new file mode 100644 index 0000000000000..123b49ddb5ac6 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-05-24-23-52-03.bpo-40617.lycF9q.rst @@ -0,0 +1,3 @@ +Add :meth:`~sqlite3.Connection.create_window_function` to +:class:`sqlite3.Connection` for creating aggregate window functions. +Patch by Erlend E. Aasland. diff --git a/Modules/_sqlite/clinic/connection.c.h b/Modules/_sqlite/clinic/connection.c.h index 99ef94ecd71ec..2b933f8522465 100644 --- a/Modules/_sqlite/clinic/connection.c.h +++ b/Modules/_sqlite/clinic/connection.c.h @@ -235,6 +235,53 @@ pysqlite_connection_create_function(pysqlite_Connection *self, PyTypeObject *cls return return_value; } +#if defined(HAVE_WINDOW_FUNCTIONS) + +PyDoc_STRVAR(create_window_function__doc__, +"create_window_function($self, name, num_params, aggregate_class, /)\n" +"--\n" +"\n" +"Creates or redefines an aggregate window function. Non-standard.\n" +"\n" +" name\n" +" The name of the SQL aggregate window function to be created or\n" +" redefined.\n" +" num_params\n" +" The number of arguments the step and inverse methods takes.\n" +" aggregate_class\n" +" A class with step(), finalize(), value(), and inverse() methods.\n" +" Set to None to clear the window function."); + +#define CREATE_WINDOW_FUNCTION_METHODDEF \ + {"create_window_function", (PyCFunction)(void(*)(void))create_window_function, METH_METHOD|METH_FASTCALL|METH_KEYWORDS, create_window_function__doc__}, + +static PyObject * +create_window_function_impl(pysqlite_Connection *self, PyTypeObject *cls, + const char *name, int num_params, + PyObject *aggregate_class); + +static PyObject * +create_window_function(pysqlite_Connection *self, PyTypeObject *cls, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) +{ + PyObject *return_value = NULL; + static const char * const _keywords[] = {"", "", "", NULL}; + static _PyArg_Parser _parser = {"siO:create_window_function", _keywords, 0}; + const char *name; + int num_params; + PyObject *aggregate_class; + + if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser, + &name, &num_params, &aggregate_class)) { + goto exit; + } + return_value = create_window_function_impl(self, cls, name, num_params, aggregate_class); + +exit: + return return_value; +} + +#endif /* defined(HAVE_WINDOW_FUNCTIONS) */ + PyDoc_STRVAR(pysqlite_connection_create_aggregate__doc__, "create_aggregate($self, /, name, n_arg, aggregate_class)\n" "--\n" @@ -975,6 +1022,10 @@ getlimit(pysqlite_Connection *self, PyObject *arg) return return_value; } +#ifndef CREATE_WINDOW_FUNCTION_METHODDEF + #define CREATE_WINDOW_FUNCTION_METHODDEF +#endif /* !defined(CREATE_WINDOW_FUNCTION_METHODDEF) */ + #ifndef PYSQLITE_CONNECTION_ENABLE_LOAD_EXTENSION_METHODDEF #define PYSQLITE_CONNECTION_ENABLE_LOAD_EXTENSION_METHODDEF #endif /* !defined(PYSQLITE_CONNECTION_ENABLE_LOAD_EXTENSION_METHODDEF) */ @@ -990,4 +1041,4 @@ getlimit(pysqlite_Connection *self, PyObject *arg) #ifndef DESERIALIZE_METHODDEF #define DESERIALIZE_METHODDEF #endif /* !defined(DESERIALIZE_METHODDEF) */ -/*[clinic end generated code: output=d965a68f9229a56c input=a9049054013a1b77]*/ +/*[clinic end generated code: output=b9af1b52fda808bf input=a9049054013a1b77]*/ diff --git a/Modules/_sqlite/connection.c b/Modules/_sqlite/connection.c index 9d187cfa99d23..d7c0a9e46161c 100644 --- a/Modules/_sqlite/connection.c +++ b/Modules/_sqlite/connection.c @@ -33,6 +33,10 @@ #define HAVE_TRACE_V2 #endif +#if SQLITE_VERSION_NUMBER >= 3025000 +#define HAVE_WINDOW_FUNCTIONS +#endif + static const char * get_isolation_level(const char *level) { @@ -799,7 +803,7 @@ final_callback(sqlite3_context *context) goto error; } - /* Keep the exception (if any) of the last call to step() */ + // Keep the exception (if any) of the last call to step, value, or inverse PyErr_Fetch(&exception, &value, &tb); callback_context *ctx = (callback_context *)sqlite3_user_data(context); @@ -814,13 +818,20 @@ final_callback(sqlite3_context *context) Py_DECREF(function_result); } if (!ok) { - set_sqlite_error(context, - "user-defined aggregate's 'finalize' method raised error"); - } + int attr_err = PyErr_ExceptionMatches(PyExc_AttributeError); + _PyErr_ChainExceptions(exception, value, tb); - /* Restore the exception (if any) of the last call to step(), - but clear also the current exception if finalize() failed */ - PyErr_Restore(exception, value, tb); + /* Note: contrary to the step, value, and inverse callbacks, SQLite + * does _not_, as of SQLite 3.38.0, propagate errors to sqlite3_step() + * from the finalize callback. This implies that execute*() will not + * raise OperationalError, as it normally would. */ + set_sqlite_error(context, attr_err + ? "user-defined aggregate's 'finalize' method not defined" + : "user-defined aggregate's 'finalize' method raised error"); + } + else { + PyErr_Restore(exception, value, tb); + } error: PyGILState_Release(threadstate); @@ -968,6 +979,159 @@ pysqlite_connection_create_function_impl(pysqlite_Connection *self, Py_RETURN_NONE; } +#ifdef HAVE_WINDOW_FUNCTIONS +/* + * Regarding the 'inverse' aggregate callback: + * This method is only required by window aggregate functions, not + * ordinary aggregate function implementations. It is invoked to remove + * a row from the current window. The function arguments, if any, + * correspond to the row being removed. + */ +static void +inverse_callback(sqlite3_context *context, int argc, sqlite3_value **params) +{ + PyGILState_STATE gilstate = PyGILState_Ensure(); + + callback_context *ctx = (callback_context *)sqlite3_user_data(context); + assert(ctx != NULL); + + int size = sizeof(PyObject *); + PyObject **cls = (PyObject **)sqlite3_aggregate_context(context, size); + assert(cls != NULL); + assert(*cls != NULL); + + PyObject *method = PyObject_GetAttr(*cls, ctx->state->str_inverse); + if (method == NULL) { + set_sqlite_error(context, + "user-defined aggregate's 'inverse' method not defined"); + goto exit; + } + + PyObject *args = _pysqlite_build_py_params(context, argc, params); + if (args == NULL) { + set_sqlite_error(context, + "unable to build arguments for user-defined aggregate's " + "'inverse' method"); + goto exit; + } + + PyObject *res = PyObject_CallObject(method, args); + Py_DECREF(args); + if (res == NULL) { + set_sqlite_error(context, + "user-defined aggregate's 'inverse' method raised error"); + goto exit; + } + Py_DECREF(res); + +exit: + Py_XDECREF(method); + PyGILState_Release(gilstate); +} + +/* + * Regarding the 'value' aggregate callback: + * This method is only required by window aggregate functions, not + * ordinary aggregate function implementations. It is invoked to return + * the current value of the aggregate. + */ +static void +value_callback(sqlite3_context *context) +{ + PyGILState_STATE gilstate = PyGILState_Ensure(); + + callback_context *ctx = (callback_context *)sqlite3_user_data(context); + assert(ctx != NULL); + + int size = sizeof(PyObject *); + PyObject **cls = (PyObject **)sqlite3_aggregate_context(context, size); + assert(cls != NULL); + assert(*cls != NULL); + + PyObject *res = PyObject_CallMethodNoArgs(*cls, ctx->state->str_value); + if (res == NULL) { + int attr_err = PyErr_ExceptionMatches(PyExc_AttributeError); + set_sqlite_error(context, attr_err + ? "user-defined aggregate's 'value' method not defined" + : "user-defined aggregate's 'value' method raised error"); + } + else { + int rc = _pysqlite_set_result(context, res); + Py_DECREF(res); + if (rc < 0) { + set_sqlite_error(context, + "unable to set result from user-defined aggregate's " + "'value' method"); + } + } + + PyGILState_Release(gilstate); +} + +/*[clinic input] +_sqlite3.Connection.create_window_function as create_window_function + + cls: defining_class + name: str + The name of the SQL aggregate window function to be created or + redefined. + num_params: int + The number of arguments the step and inverse methods takes. + aggregate_class: object + A class with step(), finalize(), value(), and inverse() methods. + Set to None to clear the window function. + / + +Creates or redefines an aggregate window function. Non-standard. +[clinic start generated code]*/ + +static PyObject * +create_window_function_impl(pysqlite_Connection *self, PyTypeObject *cls, + const char *name, int num_params, + PyObject *aggregate_class) +/*[clinic end generated code: output=5332cd9464522235 input=46d57a54225b5228]*/ +{ + if (sqlite3_libversion_number() < 3025000) { + PyErr_SetString(self->NotSupportedError, + "create_window_function() requires " + "SQLite 3.25.0 or higher"); + return NULL; + } + + if (!pysqlite_check_thread(self) || !pysqlite_check_connection(self)) { + return NULL; + } + + int flags = SQLITE_UTF8; + int rc; + if (Py_IsNone(aggregate_class)) { + rc = sqlite3_create_window_function(self->db, name, num_params, flags, + 0, 0, 0, 0, 0, 0); + } + else { + callback_context *ctx = create_callback_context(cls, aggregate_class); + if (ctx == NULL) { + return NULL; + } + rc = sqlite3_create_window_function(self->db, name, num_params, flags, + ctx, + &step_callback, + &final_callback, + &value_callback, + &inverse_callback, + &destructor_callback); + } + + if (rc != SQLITE_OK) { + // Errors are not set on the database connection, so we cannot + // use _pysqlite_seterror(). + PyErr_SetString(self->ProgrammingError, sqlite3_errstr(rc)); + return NULL; + } + Py_RETURN_NONE; +} +#endif + /*[clinic input] _sqlite3.Connection.create_aggregate as pysqlite_connection_create_aggregate @@ -2092,6 +2256,7 @@ static PyMethodDef connection_methods[] = { GETLIMIT_METHODDEF SERIALIZE_METHODDEF DESERIALIZE_METHODDEF + CREATE_WINDOW_FUNCTION_METHODDEF {NULL, NULL} }; diff --git a/Modules/_sqlite/module.c b/Modules/_sqlite/module.c index 07f090c4a2614..ffda836d7a3cc 100644 --- a/Modules/_sqlite/module.c +++ b/Modules/_sqlite/module.c @@ -630,8 +630,10 @@ module_clear(PyObject *module) Py_CLEAR(state->str___conform__); Py_CLEAR(state->str_executescript); Py_CLEAR(state->str_finalize); + Py_CLEAR(state->str_inverse); Py_CLEAR(state->str_step); Py_CLEAR(state->str_upper); + Py_CLEAR(state->str_value); return 0; } @@ -717,8 +719,10 @@ module_exec(PyObject *module) ADD_INTERNED(state, __conform__); ADD_INTERNED(state, executescript); ADD_INTERNED(state, finalize); + ADD_INTERNED(state, inverse); ADD_INTERNED(state, step); ADD_INTERNED(state, upper); + ADD_INTERNED(state, value); /* Set error constants */ if (add_error_constants(module) < 0) { diff --git a/Modules/_sqlite/module.h b/Modules/_sqlite/module.h index cca52d1e04b2c..fcea7096924ce 100644 --- a/Modules/_sqlite/module.h +++ b/Modules/_sqlite/module.h @@ -64,8 +64,10 @@ typedef struct { PyObject *str___conform__; PyObject *str_executescript; PyObject *str_finalize; + PyObject *str_inverse; PyObject *str_step; PyObject *str_upper; + PyObject *str_value; } pysqlite_state; extern pysqlite_state pysqlite_global_state; From webhook-mailer at python.org Mon Apr 11 20:58:01 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Tue, 12 Apr 2022 00:58:01 -0000 Subject: [Python-checkins] gh-88513: clarify shutil.copytree's dirs_exist_ok arg (GH-91434) Message-ID: https://github.com/python/cpython/commit/f33e2c87a83917b5139d97fd8ef7cba7223ebef5 commit: f33e2c87a83917b5139d97fd8ef7cba7223ebef5 branch: main author: Jack DeVries committer: JelleZijlstra date: 2022-04-11T17:57:52-07:00 summary: gh-88513: clarify shutil.copytree's dirs_exist_ok arg (GH-91434) * add a paragraph to document this kwarg in detail * update docstring in the source accordingly files: A Misc/NEWS.d/next/Documentation/2022-04-10-20-28-20.bpo-44347.Q1m3DM.rst M Doc/library/shutil.rst M Lib/shutil.py diff --git a/Doc/library/shutil.rst b/Doc/library/shutil.rst index ac271ce9ffbe4..75ffb79d535c8 100644 --- a/Doc/library/shutil.rst +++ b/Doc/library/shutil.rst @@ -230,9 +230,8 @@ Directory and files operations dirs_exist_ok=False) Recursively copy an entire directory tree rooted at *src* to a directory - named *dst* and return the destination directory. *dirs_exist_ok* dictates - whether to raise an exception in case *dst* or any missing parent directory - already exists. + named *dst* and return the destination directory. All intermediate + directories needed to contain *dst* will also be created by default. Permissions and times of directories are copied with :func:`copystat`, individual files are copied using :func:`~shutil.copy2`. @@ -263,8 +262,14 @@ Directory and files operations If *copy_function* is given, it must be a callable that will be used to copy each file. It will be called with the source path and the destination path - ? as arguments. By default, :func:`~shutil.copy2` is used, but any function - ? that supports the same signature (like :func:`~shutil.copy`) can be used. + as arguments. By default, :func:`~shutil.copy2` is used, but any function + that supports the same signature (like :func:`~shutil.copy`) can be used. + + If *dirs_exist_ok* is false (the default) and *dst* already exists, a + :exc:`FileExistsError` is raised. If *dirs_exist_ok* is true, the copying + operation will continue if it encounters existing directories, and files + within the *dst* tree will be overwritten by corresponding files from the + *src* tree. .. audit-event:: shutil.copytree src,dst shutil.copytree @@ -275,7 +280,7 @@ Directory and files operations .. versionchanged:: 3.2 Added the *copy_function* argument to be able to provide a custom copy function. - Added the *ignore_dangling_symlinks* argument to silent dangling symlinks + Added the *ignore_dangling_symlinks* argument to silence dangling symlinks errors when *symlinks* is false. .. versionchanged:: 3.8 diff --git a/Lib/shutil.py b/Lib/shutil.py index 22bd86d569e7e..de82453aa56e1 100644 --- a/Lib/shutil.py +++ b/Lib/shutil.py @@ -518,9 +518,6 @@ def copytree(src, dst, symlinks=False, ignore=None, copy_function=copy2, ignore_dangling_symlinks=False, dirs_exist_ok=False): """Recursively copy a directory tree and return the destination directory. - dirs_exist_ok dictates whether to raise an exception in case dst or any - missing parent directory already exists. - If exception(s) occur, an Error is raised with a list of reasons. If the optional symlinks flag is true, symbolic links in the @@ -551,6 +548,11 @@ def copytree(src, dst, symlinks=False, ignore=None, copy_function=copy2, destination path as arguments. By default, copy2() is used, but any function that supports the same signature (like copy()) can be used. + If dirs_exist_ok is false (the default) and `dst` already exists, a + `FileExistsError` is raised. If `dirs_exist_ok` is true, the copying + operation will continue if it encounters existing directories, and files + within the `dst` tree will be overwritten by corresponding files from the + `src` tree. """ sys.audit("shutil.copytree", src, dst) with os.scandir(src) as itr: diff --git a/Misc/NEWS.d/next/Documentation/2022-04-10-20-28-20.bpo-44347.Q1m3DM.rst b/Misc/NEWS.d/next/Documentation/2022-04-10-20-28-20.bpo-44347.Q1m3DM.rst new file mode 100644 index 0000000000000..27aa5742cd008 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2022-04-10-20-28-20.bpo-44347.Q1m3DM.rst @@ -0,0 +1 @@ +Clarify the meaning of *dirs_exist_ok*, a kwarg of :func:`shutil.copytree`. From webhook-mailer at python.org Tue Apr 12 00:14:51 2022 From: webhook-mailer at python.org (zware) Date: Tue, 12 Apr 2022 04:14:51 -0000 Subject: [Python-checkins] [3.9] GH-89074: Fixed IsolatedAsyncioTestCase from throwing an exception on leaked tasks (GH-27765) (#91471) Message-ID: https://github.com/python/cpython/commit/4cc4fe27890ef73d91c49b1bc0686a50ec704ade commit: 4cc4fe27890ef73d91c49b1bc0686a50ec704ade branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: zware date: 2022-04-11T23:14:43-05:00 summary: [3.9] GH-89074: Fixed IsolatedAsyncioTestCase from throwing an exception on leaked tasks (GH-27765) (#91471) (cherry picked from commit 2cb1a6806c0cefab0c3a40fdd428a89a4392570e) Co-authored-by: Bar Harel files: A Misc/NEWS.d/next/Library/2021-08-14-00-55-16.bpo-44911.uk3hYk.rst M Lib/unittest/async_case.py M Lib/unittest/test/test_async_case.py diff --git a/Lib/unittest/async_case.py b/Lib/unittest/async_case.py index a2980e797ac5b..23231199f9870 100644 --- a/Lib/unittest/async_case.py +++ b/Lib/unittest/async_case.py @@ -134,7 +134,7 @@ def _tearDownAsyncioLoop(self): task.cancel() loop.run_until_complete( - asyncio.gather(*to_cancel, loop=loop, return_exceptions=True)) + asyncio.gather(*to_cancel, return_exceptions=True)) for task in to_cancel: if task.cancelled(): diff --git a/Lib/unittest/test/test_async_case.py b/Lib/unittest/test/test_async_case.py index 2b6e751f3b027..22e2d0864ee04 100644 --- a/Lib/unittest/test/test_async_case.py +++ b/Lib/unittest/test/test_async_case.py @@ -278,6 +278,26 @@ async def test_cancel(self): output = test.run() self.assertFalse(output.wasSuccessful()) + def test_cancellation_hanging_tasks(self): + cancelled = False + class Test(unittest.IsolatedAsyncioTestCase): + async def test_leaking_task(self): + async def coro(): + nonlocal cancelled + try: + await asyncio.sleep(1) + except asyncio.CancelledError: + cancelled = True + raise + + # Leave this running in the background + asyncio.create_task(coro()) + + test = Test("test_leaking_task") + output = test.run() + self.assertTrue(cancelled) + + if __name__ == "__main__": unittest.main() diff --git a/Misc/NEWS.d/next/Library/2021-08-14-00-55-16.bpo-44911.uk3hYk.rst b/Misc/NEWS.d/next/Library/2021-08-14-00-55-16.bpo-44911.uk3hYk.rst new file mode 100644 index 0000000000000..f8aed69a40a3b --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-08-14-00-55-16.bpo-44911.uk3hYk.rst @@ -0,0 +1 @@ +:class:`~unittest.IsolatedAsyncioTestCase` will no longer throw an exception while cancelling leaked tasks. Patch by Bar Harel. \ No newline at end of file From webhook-mailer at python.org Tue Apr 12 09:36:04 2022 From: webhook-mailer at python.org (iritkatriel) Date: Tue, 12 Apr 2022 13:36:04 -0000 Subject: [Python-checkins] gh-91276: make space for longer opcodes in dis output (GH-91444) Message-ID: https://github.com/python/cpython/commit/e44f988b26cffedc8fd74c0424ac1fada695688b commit: e44f988b26cffedc8fd74c0424ac1fada695688b branch: main author: Irit Katriel <1055913+iritkatriel at users.noreply.github.com> committer: iritkatriel <1055913+iritkatriel at users.noreply.github.com> date: 2022-04-12T14:35:56+01:00 summary: gh-91276: make space for longer opcodes in dis output (GH-91444) files: A Misc/NEWS.d/next/Library/2022-04-11-13-07-30.gh-issue-91276.Vttu15.rst M Lib/dis.py M Lib/test/test_dis.py diff --git a/Lib/dis.py b/Lib/dis.py index ac900d743a036..f7b38a82ab13e 100644 --- a/Lib/dis.py +++ b/Lib/dis.py @@ -245,7 +245,7 @@ def show_code(co, *, file=None): _ExceptionTableEntry = collections.namedtuple("_ExceptionTableEntry", "start end target depth lasti") -_OPNAME_WIDTH = 20 +_OPNAME_WIDTH = max(map(len, opmap)) _OPARG_WIDTH = 5 class Instruction(_Instruction): diff --git a/Lib/test/test_dis.py b/Lib/test/test_dis.py index 1703e711d56b7..fbc34a5dbe4ef 100644 --- a/Lib/test/test_dis.py +++ b/Lib/test/test_dis.py @@ -41,48 +41,48 @@ def cm(cls, x): cls.x = x == 1 dis_c_instance_method = """\ -%3d RESUME 0 - -%3d LOAD_FAST 1 (x) - LOAD_CONST 1 (1) - COMPARE_OP 2 (==) - LOAD_FAST 0 (self) - STORE_ATTR 0 (x) - LOAD_CONST 0 (None) +%3d RESUME 0 + +%3d LOAD_FAST 1 (x) + LOAD_CONST 1 (1) + COMPARE_OP 2 (==) + LOAD_FAST 0 (self) + STORE_ATTR 0 (x) + LOAD_CONST 0 (None) RETURN_VALUE """ % (_C.__init__.__code__.co_firstlineno, _C.__init__.__code__.co_firstlineno + 1,) dis_c_instance_method_bytes = """\ - RESUME 0 - LOAD_FAST 1 - LOAD_CONST 1 - COMPARE_OP 2 (==) - LOAD_FAST 0 - STORE_ATTR 0 - LOAD_CONST 0 + RESUME 0 + LOAD_FAST 1 + LOAD_CONST 1 + COMPARE_OP 2 (==) + LOAD_FAST 0 + STORE_ATTR 0 + LOAD_CONST 0 RETURN_VALUE """ dis_c_class_method = """\ -%3d RESUME 0 - -%3d LOAD_FAST 1 (x) - LOAD_CONST 1 (1) - COMPARE_OP 2 (==) - LOAD_FAST 0 (cls) - STORE_ATTR 0 (x) - LOAD_CONST 0 (None) +%3d RESUME 0 + +%3d LOAD_FAST 1 (x) + LOAD_CONST 1 (1) + COMPARE_OP 2 (==) + LOAD_FAST 0 (cls) + STORE_ATTR 0 (x) + LOAD_CONST 0 (None) RETURN_VALUE """ % (_C.cm.__code__.co_firstlineno, _C.cm.__code__.co_firstlineno + 2,) dis_c_static_method = """\ -%3d RESUME 0 +%3d RESUME 0 -%3d LOAD_FAST 0 (x) - LOAD_CONST 1 (1) - COMPARE_OP 2 (==) - STORE_FAST 0 (x) - LOAD_CONST 0 (None) +%3d LOAD_FAST 0 (x) + LOAD_CONST 1 (1) + COMPARE_OP 2 (==) + STORE_FAST 0 (x) + LOAD_CONST 0 (None) RETURN_VALUE """ % (_C.sm.__code__.co_firstlineno, _C.sm.__code__.co_firstlineno + 2,) @@ -103,15 +103,15 @@ def _f(a): return 1 dis_f = """\ -%3d RESUME 0 +%3d RESUME 0 -%3d LOAD_GLOBAL 1 (NULL + print) - LOAD_FAST 0 (a) - PRECALL 1 - CALL 1 +%3d LOAD_GLOBAL 1 (NULL + print) + LOAD_FAST 0 (a) + PRECALL 1 + CALL 1 POP_TOP -%3d LOAD_CONST 1 (1) +%3d LOAD_CONST 1 (1) RETURN_VALUE """ % (_f.__code__.co_firstlineno, _f.__code__.co_firstlineno + 1, @@ -119,13 +119,13 @@ def _f(a): dis_f_co_code = """\ - RESUME 0 - LOAD_GLOBAL 1 - LOAD_FAST 0 - PRECALL 1 - CALL 1 + RESUME 0 + LOAD_GLOBAL 1 + LOAD_FAST 0 + PRECALL 1 + CALL 1 POP_TOP - LOAD_CONST 1 + LOAD_CONST 1 RETURN_VALUE """ @@ -136,22 +136,22 @@ def bug708901(): pass dis_bug708901 = """\ -%3d RESUME 0 +%3d RESUME 0 -%3d LOAD_GLOBAL 1 (NULL + range) - LOAD_CONST 1 (1) +%3d LOAD_GLOBAL 1 (NULL + range) + LOAD_CONST 1 (1) -%3d LOAD_CONST 2 (10) +%3d LOAD_CONST 2 (10) -%3d PRECALL 2 - CALL 2 +%3d PRECALL 2 + CALL 2 GET_ITER - >> FOR_ITER 2 (to 40) - STORE_FAST 0 (res) + >> FOR_ITER 2 (to 40) + STORE_FAST 0 (res) -%3d JUMP_BACKWARD 3 (to 34) +%3d JUMP_BACKWARD 3 (to 34) -%3d >> LOAD_CONST 0 (None) +%3d >> LOAD_CONST 0 (None) RETURN_VALUE """ % (bug708901.__code__.co_firstlineno, bug708901.__code__.co_firstlineno + 1, @@ -167,22 +167,22 @@ def bug1333982(x=[]): pass dis_bug1333982 = """\ -%3d RESUME 0 +%3d RESUME 0 %3d LOAD_ASSERTION_ERROR - LOAD_CONST 2 ( at 0x..., file "%s", line %d>) - MAKE_FUNCTION 0 - LOAD_FAST 0 (x) + LOAD_CONST 2 ( at 0x..., file "%s", line %d>) + MAKE_FUNCTION 0 + LOAD_FAST 0 (x) GET_ITER - PRECALL 0 - CALL 0 + PRECALL 0 + CALL 0 -%3d LOAD_CONST 3 (1) +%3d LOAD_CONST 3 (1) -%3d BINARY_OP 0 (+) - PRECALL 0 - CALL 0 - RAISE_VARARGS 1 +%3d BINARY_OP 0 (+) + PRECALL 0 + CALL 0 + RAISE_VARARGS 1 """ % (bug1333982.__code__.co_firstlineno, bug1333982.__code__.co_firstlineno + 1, __file__, @@ -200,8 +200,8 @@ def bug42562(): dis_bug42562 = """\ - RESUME 0 - LOAD_CONST 0 (None) + RESUME 0 + LOAD_CONST 0 (None) RETURN_VALUE """ @@ -215,10 +215,10 @@ def bug42562(): ]) dis_bug_45757 = """\ - EXTENDED_ARG 1 + EXTENDED_ARG 1 NOP - EXTENDED_ARG 1 - LOAD_CONST 297 + EXTENDED_ARG 1 + LOAD_CONST 297 RETURN_VALUE """ @@ -232,39 +232,39 @@ def bug42562(): dis_bug46724 = """\ - >> EXTENDED_ARG 255 - EXTENDED_ARG 65535 - EXTENDED_ARG 16777215 - JUMP_FORWARD -4 (to 0) + >> EXTENDED_ARG 255 + EXTENDED_ARG 65535 + EXTENDED_ARG 16777215 + JUMP_FORWARD -4 (to 0) """ _BIG_LINENO_FORMAT = """\ - 1 RESUME 0 + 1 RESUME 0 -%3d LOAD_GLOBAL 0 (spam) +%3d LOAD_GLOBAL 0 (spam) POP_TOP - LOAD_CONST 0 (None) + LOAD_CONST 0 (None) RETURN_VALUE """ _BIG_LINENO_FORMAT2 = """\ - 1 RESUME 0 + 1 RESUME 0 -%4d LOAD_GLOBAL 0 (spam) +%4d LOAD_GLOBAL 0 (spam) POP_TOP - LOAD_CONST 0 (None) + LOAD_CONST 0 (None) RETURN_VALUE """ dis_module_expected_results = """\ Disassembly of f: - 4 RESUME 0 - LOAD_CONST 0 (None) + 4 RESUME 0 + LOAD_CONST 0 (None) RETURN_VALUE Disassembly of g: - 5 RESUME 0 - LOAD_CONST 0 (None) + 5 RESUME 0 + LOAD_CONST 0 (None) RETURN_VALUE """ @@ -272,24 +272,24 @@ def bug42562(): expr_str = "x + 1" dis_expr_str = """\ - RESUME 0 + RESUME 0 - 1 LOAD_NAME 0 (x) - LOAD_CONST 0 (1) - BINARY_OP 0 (+) + 1 LOAD_NAME 0 (x) + LOAD_CONST 0 (1) + BINARY_OP 0 (+) RETURN_VALUE """ simple_stmt_str = "x = x + 1" dis_simple_stmt_str = """\ - RESUME 0 + RESUME 0 - 1 LOAD_NAME 0 (x) - LOAD_CONST 0 (1) - BINARY_OP 0 (+) - STORE_NAME 0 (x) - LOAD_CONST 1 (None) + 1 LOAD_NAME 0 (x) + LOAD_CONST 0 (1) + BINARY_OP 0 (+) + STORE_NAME 0 (x) + LOAD_CONST 1 (None) RETURN_VALUE """ @@ -302,36 +302,36 @@ def bug42562(): # leading newline is for a reason (tests lineno) dis_annot_stmt_str = """\ - RESUME 0 + RESUME 0 2 SETUP_ANNOTATIONS - LOAD_CONST 0 (1) - STORE_NAME 0 (x) - LOAD_NAME 1 (int) - LOAD_NAME 2 (__annotations__) - LOAD_CONST 1 ('x') + LOAD_CONST 0 (1) + STORE_NAME 0 (x) + LOAD_NAME 1 (int) + LOAD_NAME 2 (__annotations__) + LOAD_CONST 1 ('x') STORE_SUBSCR 3 PUSH_NULL - LOAD_NAME 3 (fun) - LOAD_CONST 0 (1) - PRECALL 1 - CALL 1 - LOAD_NAME 2 (__annotations__) - LOAD_CONST 2 ('y') + LOAD_NAME 3 (fun) + LOAD_CONST 0 (1) + PRECALL 1 + CALL 1 + LOAD_NAME 2 (__annotations__) + LOAD_CONST 2 ('y') STORE_SUBSCR - 4 LOAD_CONST 0 (1) - LOAD_NAME 4 (lst) + 4 LOAD_CONST 0 (1) + LOAD_NAME 4 (lst) PUSH_NULL - LOAD_NAME 3 (fun) - LOAD_CONST 3 (0) - PRECALL 1 - CALL 1 + LOAD_NAME 3 (fun) + LOAD_CONST 3 (0) + PRECALL 1 + CALL 1 STORE_SUBSCR - LOAD_NAME 1 (int) + LOAD_NAME 1 (int) POP_TOP - LOAD_CONST 4 (None) + LOAD_CONST 4 (None) RETURN_VALUE """ @@ -342,59 +342,59 @@ def bug42562(): # Trailing newline has been deliberately omitted dis_compound_stmt_str = """\ - RESUME 0 + RESUME 0 - 1 LOAD_CONST 0 (0) - STORE_NAME 0 (x) + 1 LOAD_CONST 0 (0) + STORE_NAME 0 (x) 2 NOP - 3 >> LOAD_NAME 0 (x) - LOAD_CONST 1 (1) - BINARY_OP 13 (+=) - STORE_NAME 0 (x) + 3 >> LOAD_NAME 0 (x) + LOAD_CONST 1 (1) + BINARY_OP 13 (+=) + STORE_NAME 0 (x) - 2 JUMP_BACKWARD 6 (to 8) + 2 JUMP_BACKWARD 6 (to 8) """ dis_traceback = """\ -%3d RESUME 0 +%3d RESUME 0 %3d NOP -%3d LOAD_CONST 1 (1) - LOAD_CONST 2 (0) - --> BINARY_OP 11 (/) +%3d LOAD_CONST 1 (1) + LOAD_CONST 2 (0) + --> BINARY_OP 11 (/) POP_TOP -%3d LOAD_FAST 1 (tb) +%3d LOAD_FAST 1 (tb) RETURN_VALUE >> PUSH_EXC_INFO -%3d LOAD_GLOBAL 0 (Exception) +%3d LOAD_GLOBAL 0 (Exception) CHECK_EXC_MATCH - POP_JUMP_FORWARD_IF_FALSE 18 (to 72) - STORE_FAST 0 (e) + POP_JUMP_FORWARD_IF_FALSE 18 (to 72) + STORE_FAST 0 (e) -%3d LOAD_FAST 0 (e) - LOAD_ATTR 1 (__traceback__) - STORE_FAST 1 (tb) +%3d LOAD_FAST 0 (e) + LOAD_ATTR 1 (__traceback__) + STORE_FAST 1 (tb) POP_EXCEPT - LOAD_CONST 0 (None) - STORE_FAST 0 (e) - DELETE_FAST 0 (e) + LOAD_CONST 0 (None) + STORE_FAST 0 (e) + DELETE_FAST 0 (e) -%3d LOAD_FAST 1 (tb) +%3d LOAD_FAST 1 (tb) RETURN_VALUE - >> LOAD_CONST 0 (None) - STORE_FAST 0 (e) - DELETE_FAST 0 (e) - RERAISE 1 + >> LOAD_CONST 0 (None) + STORE_FAST 0 (e) + DELETE_FAST 0 (e) + RERAISE 1 -%3d >> RERAISE 0 - >> COPY 3 +%3d >> RERAISE 0 + >> COPY 3 POP_EXCEPT - RERAISE 1 + RERAISE 1 ExceptionTable: """ % (TRACEBACK_CODE.co_firstlineno, TRACEBACK_CODE.co_firstlineno + 1, @@ -409,22 +409,22 @@ def _fstring(a, b, c, d): return f'{a} {b:4} {c!r} {d!r:4}' dis_fstring = """\ -%3d RESUME 0 - -%3d LOAD_FAST 0 (a) - FORMAT_VALUE 0 - LOAD_CONST 1 (' ') - LOAD_FAST 1 (b) - LOAD_CONST 2 ('4') - FORMAT_VALUE 4 (with format) - LOAD_CONST 1 (' ') - LOAD_FAST 2 (c) - FORMAT_VALUE 2 (repr) - LOAD_CONST 1 (' ') - LOAD_FAST 3 (d) - LOAD_CONST 2 ('4') - FORMAT_VALUE 6 (repr, with format) - BUILD_STRING 7 +%3d RESUME 0 + +%3d LOAD_FAST 0 (a) + FORMAT_VALUE 0 + LOAD_CONST 1 (' ') + LOAD_FAST 1 (b) + LOAD_CONST 2 ('4') + FORMAT_VALUE 4 (with format) + LOAD_CONST 1 (' ') + LOAD_FAST 2 (c) + FORMAT_VALUE 2 (repr) + LOAD_CONST 1 (' ') + LOAD_FAST 3 (d) + LOAD_CONST 2 ('4') + FORMAT_VALUE 6 (repr, with format) + BUILD_STRING 7 RETURN_VALUE """ % (_fstring.__code__.co_firstlineno, _fstring.__code__.co_firstlineno + 1) @@ -441,28 +441,28 @@ def _tryfinallyconst(b): b() dis_tryfinally = """\ -%3d RESUME 0 +%3d RESUME 0 %3d NOP -%3d LOAD_FAST 0 (a) +%3d LOAD_FAST 0 (a) %3d PUSH_NULL - LOAD_FAST 1 (b) - PRECALL 0 - CALL 0 + LOAD_FAST 1 (b) + PRECALL 0 + CALL 0 POP_TOP RETURN_VALUE >> PUSH_EXC_INFO PUSH_NULL - LOAD_FAST 1 (b) - PRECALL 0 - CALL 0 + LOAD_FAST 1 (b) + PRECALL 0 + CALL 0 POP_TOP - RERAISE 0 - >> COPY 3 + RERAISE 0 + >> COPY 3 POP_EXCEPT - RERAISE 1 + RERAISE 1 ExceptionTable: """ % (_tryfinally.__code__.co_firstlineno, _tryfinally.__code__.co_firstlineno + 1, @@ -471,29 +471,29 @@ def _tryfinallyconst(b): ) dis_tryfinallyconst = """\ -%3d RESUME 0 +%3d RESUME 0 %3d NOP %3d NOP %3d PUSH_NULL - LOAD_FAST 0 (b) - PRECALL 0 - CALL 0 + LOAD_FAST 0 (b) + PRECALL 0 + CALL 0 POP_TOP - LOAD_CONST 1 (1) + LOAD_CONST 1 (1) RETURN_VALUE PUSH_EXC_INFO PUSH_NULL - LOAD_FAST 0 (b) - PRECALL 0 - CALL 0 + LOAD_FAST 0 (b) + PRECALL 0 + CALL 0 POP_TOP - RERAISE 0 - >> COPY 3 + RERAISE 0 + >> COPY 3 POP_EXCEPT - RERAISE 1 + RERAISE 1 ExceptionTable: """ % (_tryfinallyconst.__code__.co_firstlineno, _tryfinallyconst.__code__.co_firstlineno + 1, @@ -518,17 +518,17 @@ def foo(x): return foo dis_nested_0 = """\ - MAKE_CELL 0 (y) + MAKE_CELL 0 (y) -%3d RESUME 0 +%3d RESUME 0 -%3d LOAD_CLOSURE 0 (y) - BUILD_TUPLE 1 - LOAD_CONST 1 () - MAKE_FUNCTION 8 (closure) - STORE_FAST 1 (foo) +%3d LOAD_CLOSURE 0 (y) + BUILD_TUPLE 1 + LOAD_CONST 1 () + MAKE_FUNCTION 8 (closure) + STORE_FAST 1 (foo) -%3d LOAD_FAST 1 (foo) +%3d LOAD_FAST 1 (foo) RETURN_VALUE """ % (_h.__code__.co_firstlineno, _h.__code__.co_firstlineno + 1, @@ -539,19 +539,19 @@ def foo(x): dis_nested_1 = """%s Disassembly of : - COPY_FREE_VARS 1 - MAKE_CELL 0 (x) + COPY_FREE_VARS 1 + MAKE_CELL 0 (x) -%3d RESUME 0 +%3d RESUME 0 -%3d LOAD_CLOSURE 0 (x) - BUILD_TUPLE 1 - LOAD_CONST 1 ( at 0x..., file "%s", line %d>) - MAKE_FUNCTION 8 (closure) - LOAD_DEREF 1 (y) +%3d LOAD_CLOSURE 0 (x) + BUILD_TUPLE 1 + LOAD_CONST 1 ( at 0x..., file "%s", line %d>) + MAKE_FUNCTION 8 (closure) + LOAD_DEREF 1 (y) GET_ITER - PRECALL 0 - CALL 0 + PRECALL 0 + CALL 0 RETURN_VALUE """ % (dis_nested_0, __file__, @@ -564,18 +564,18 @@ def foo(x): dis_nested_2 = """%s Disassembly of at 0x..., file "%s", line %d>: - COPY_FREE_VARS 1 - -%3d RESUME 0 - BUILD_LIST 0 - LOAD_FAST 0 (.0) - >> FOR_ITER 7 (to 24) - STORE_FAST 1 (z) - LOAD_DEREF 2 (x) - LOAD_FAST 1 (z) - BINARY_OP 0 (+) - LIST_APPEND 2 - JUMP_BACKWARD 8 (to 8) + COPY_FREE_VARS 1 + +%3d RESUME 0 + BUILD_LIST 0 + LOAD_FAST 0 (.0) + >> FOR_ITER 7 (to 24) + STORE_FAST 1 (z) + LOAD_DEREF 2 (x) + LOAD_FAST 1 (z) + BINARY_OP 0 (+) + LIST_APPEND 2 + JUMP_BACKWARD 8 (to 8) >> RETURN_VALUE """ % (dis_nested_1, __file__, @@ -683,18 +683,6 @@ def test_boundaries(self): def test_widths(self): for opcode, opname in enumerate(dis.opname): - if opname in ('BUILD_MAP_UNPACK_WITH_CALL', - 'BUILD_TUPLE_UNPACK_WITH_CALL', - 'JUMP_BACKWARD_NO_INTERRUPT', - 'POP_JUMP_FORWARD_IF_NONE', - 'POP_JUMP_BACKWARD_IF_NONE', - 'POP_JUMP_FORWARD_IF_NOT_NONE', - 'POP_JUMP_BACKWARD_IF_NOT_NONE', - 'POP_JUMP_FORWARD_IF_TRUE', - 'POP_JUMP_BACKWARD_IF_TRUE', - 'POP_JUMP_FORWARD_IF_FALSE', - 'POP_JUMP_BACKWARD_IF_FALSE'): - continue with self.subTest(opname=opname): width = dis._OPNAME_WIDTH if opcode < dis.HAVE_ARGUMENT: @@ -760,19 +748,19 @@ def func(count): def expected(count, w): s = ['''\ - 1 %*d RESUME 0 + 1 %*d RESUME 0 ''' % (w, 0)] s += ['''\ - %*d LOAD_FAST 0 (x) - %*d LOAD_CONST 1 (1) - %*d BINARY_OP 0 (+) - %*d STORE_FAST 0 (x) + %*d LOAD_FAST 0 (x) + %*d LOAD_CONST 1 (1) + %*d BINARY_OP 0 (+) + %*d STORE_FAST 0 (x) ''' % (w, 10*i + 2, w, 10*i + 4, w, 10*i + 6, w, 10*i + 10) for i in range(count)] s += ['''\ - 3 %*d LOAD_FAST 0 (x) + 3 %*d LOAD_FAST 0 (x) %*d RETURN_VALUE ''' % (w, 10*count + 2, w, 10*count + 4)] s[1] = ' 2' + s[1][3:] diff --git a/Misc/NEWS.d/next/Library/2022-04-11-13-07-30.gh-issue-91276.Vttu15.rst b/Misc/NEWS.d/next/Library/2022-04-11-13-07-30.gh-issue-91276.Vttu15.rst new file mode 100644 index 0000000000000..9a7299f3c6e68 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-11-13-07-30.gh-issue-91276.Vttu15.rst @@ -0,0 +1 @@ +Make space for longer opcodes in :mod:`dis` output. From webhook-mailer at python.org Tue Apr 12 09:47:08 2022 From: webhook-mailer at python.org (corona10) Date: Tue, 12 Apr 2022 13:47:08 -0000 Subject: [Python-checkins] gh-90839: Forward gzip.compress() compresslevel to zlib (gh-31215) Message-ID: https://github.com/python/cpython/commit/943ca5e1d6b72830c530a72426cff094f155d010 commit: 943ca5e1d6b72830c530a72426cff094f155d010 branch: main author: Ilya Leoshkevich committer: corona10 date: 2022-04-12T22:46:40+09:00 summary: gh-90839: Forward gzip.compress() compresslevel to zlib (gh-31215) files: A Misc/NEWS.d/next/Library/2022-03-21-13-50-07.bpo-46681.RRhopn.rst M Lib/gzip.py diff --git a/Lib/gzip.py b/Lib/gzip.py index 6773ea3eef097..5b20e5ba698ee 100644 --- a/Lib/gzip.py +++ b/Lib/gzip.py @@ -587,7 +587,8 @@ def compress(data, compresslevel=_COMPRESS_LEVEL_BEST, *, mtime=None): header = _create_simple_gzip_header(compresslevel, mtime) trailer = struct.pack(" https://github.com/python/cpython/commit/474fdbe9e4a2ff90ef39e8748da644c86a200981 commit: 474fdbe9e4a2ff90ef39e8748da644c86a200981 branch: main author: Serhiy Storchaka committer: serhiy-storchaka date: 2022-04-12T18:34:06+03:00 summary: bpo-47152: Automatically regenerate sre_constants.h (GH-91439) * Move the code for generating Modules/_sre/sre_constants.h from Lib/re/_constants.py into a separate script Tools/scripts/generate_sre_constants.py. * Add target `regen-sre` in the makefile. * Make target `regen-all` depending on `regen-sre`. files: A Misc/NEWS.d/next/Build/2022-04-10-16-33-31.bpo-47152.TLkxKm.rst A Tools/scripts/generate_sre_constants.py M Lib/re/_constants.py M Makefile.pre.in M Modules/_sre/sre_constants.h diff --git a/Lib/re/_constants.py b/Lib/re/_constants.py index 4c7e93e67ec60..5e999dea337d3 100644 --- a/Lib/re/_constants.py +++ b/Lib/re/_constants.py @@ -215,46 +215,3 @@ def _makecodes(names): SRE_INFO_PREFIX = 1 # has prefix SRE_INFO_LITERAL = 2 # entire pattern is literal (given by prefix) SRE_INFO_CHARSET = 4 # pattern starts with character from given set - -if __name__ == "__main__": - def dump(f, d, prefix): - items = sorted(d) - for item in items: - f.write("#define %s_%s %d\n" % (prefix, item, item)) - with open("sre_constants.h", "w") as f: - f.write("""\ -/* - * Secret Labs' Regular Expression Engine - * - * regular expression matching engine - * - * NOTE: This file is generated by Lib/re/_constants.py. If you need - * to change anything in here, edit Lib/re/_constants.py and run it. - * - * Copyright (c) 1997-2001 by Secret Labs AB. All rights reserved. - * - * See the sre.c file for information on usage and redistribution. - */ - -""") - - f.write("#define SRE_MAGIC %d\n" % MAGIC) - - dump(f, OPCODES, "SRE_OP") - dump(f, ATCODES, "SRE") - dump(f, CHCODES, "SRE") - - f.write("#define SRE_FLAG_IGNORECASE %d\n" % SRE_FLAG_IGNORECASE) - f.write("#define SRE_FLAG_LOCALE %d\n" % SRE_FLAG_LOCALE) - f.write("#define SRE_FLAG_MULTILINE %d\n" % SRE_FLAG_MULTILINE) - f.write("#define SRE_FLAG_DOTALL %d\n" % SRE_FLAG_DOTALL) - f.write("#define SRE_FLAG_UNICODE %d\n" % SRE_FLAG_UNICODE) - f.write("#define SRE_FLAG_VERBOSE %d\n" % SRE_FLAG_VERBOSE) - f.write("#define SRE_FLAG_DEBUG %d\n" % SRE_FLAG_DEBUG) - f.write("#define SRE_FLAG_ASCII %d\n" % SRE_FLAG_ASCII) - - f.write("#define SRE_INFO_PREFIX %d\n" % SRE_INFO_PREFIX) - f.write("#define SRE_INFO_LITERAL %d\n" % SRE_INFO_LITERAL) - f.write("#define SRE_INFO_CHARSET %d\n" % SRE_INFO_CHARSET) - - print("done") diff --git a/Makefile.pre.in b/Makefile.pre.in index 22a68a7048792..f803391346bd6 100644 --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -1197,7 +1197,7 @@ regen-limited-abi: all # Regenerate all generated files regen-all: regen-opcode regen-opcode-targets regen-typeslots \ - regen-token regen-ast regen-keyword regen-frozen clinic \ + regen-token regen-ast regen-keyword regen-sre regen-frozen clinic \ regen-pegen-metaparser regen-pegen regen-test-frozenmain \ regen-global-objects @echo @@ -1350,6 +1350,13 @@ regen-stdlib-module-names: build_all Programs/_testembed > $(srcdir)/Python/stdlib_module_names.h.new $(UPDATE_FILE) $(srcdir)/Python/stdlib_module_names.h $(srcdir)/Python/stdlib_module_names.h.new +regen-sre: + # Regenerate Modules/_sre/sre_constants.h from Lib/re/_constants.py + # using Tools/scripts/generate_sre_constants.py + $(PYTHON_FOR_REGEN) $(srcdir)/Tools/scripts/generate_sre_constants.py \ + $(srcdir)/Lib/re/_constants.py \ + $(srcdir)/Modules/_sre/sre_constants.h + Python/compile.o Python/symtable.o Python/ast_unparse.o Python/ast.o Python/future.o: $(srcdir)/Include/internal/pycore_ast.h Python/getplatform.o: $(srcdir)/Python/getplatform.c diff --git a/Misc/NEWS.d/next/Build/2022-04-10-16-33-31.bpo-47152.TLkxKm.rst b/Misc/NEWS.d/next/Build/2022-04-10-16-33-31.bpo-47152.TLkxKm.rst new file mode 100644 index 0000000000000..889353f824c8a --- /dev/null +++ b/Misc/NEWS.d/next/Build/2022-04-10-16-33-31.bpo-47152.TLkxKm.rst @@ -0,0 +1 @@ +Add script and make target for generating ``sre_constants.h``. diff --git a/Modules/_sre/sre_constants.h b/Modules/_sre/sre_constants.h index e53fa392ec585..b1ef27eccc839 100644 --- a/Modules/_sre/sre_constants.h +++ b/Modules/_sre/sre_constants.h @@ -3,8 +3,8 @@ * * regular expression matching engine * - * NOTE: This file is generated by Lib/re/_constants.py. If you need - * to change anything in here, edit Lib/re/_constants.py and run it. + * Auto-generated by Tools/scripts/generate_sre_constants.py from + * Lib/re/_constants.py. * * Copyright (c) 1997-2001 by Secret Labs AB. All rights reserved. * diff --git a/Tools/scripts/generate_sre_constants.py b/Tools/scripts/generate_sre_constants.py new file mode 100755 index 0000000000000..b8f0df9cc23b1 --- /dev/null +++ b/Tools/scripts/generate_sre_constants.py @@ -0,0 +1,62 @@ +#! /usr/bin/env python3 +# This script generates Modules/_sre/sre_constants.h from Lib/re/_constants.py. + + +def update_file(file, content): + try: + with open(file, 'r') as fobj: + if fobj.read() == content: + return False + except (OSError, ValueError): + pass + with open(file, 'w') as fobj: + fobj.write(content) + return True + +sre_constants_header = """\ +/* + * Secret Labs' Regular Expression Engine + * + * regular expression matching engine + * + * Auto-generated by Tools/scripts/generate_sre_constants.py from + * Lib/re/_constants.py. + * + * Copyright (c) 1997-2001 by Secret Labs AB. All rights reserved. + * + * See the sre.c file for information on usage and redistribution. + */ + +""" + +def main(infile='Lib/re/_constants.py', outfile='Modules/_sre/sre_constants.h'): + ns = {} + with open(infile) as fp: + code = fp.read() + exec(code, ns) + + def dump(d, prefix): + items = sorted(d) + for item in items: + yield "#define %s_%s %d\n" % (prefix, item, item) + + def dump2(d, prefix): + items = [(value, name) for name, value in d.items() + if name.startswith(prefix)] + for value, name in sorted(items): + yield "#define %s %d\n" % (name, value) + + content = [sre_constants_header] + content.append("#define SRE_MAGIC %d\n" % ns["MAGIC"]) + content.extend(dump(ns["OPCODES"], "SRE_OP")) + content.extend(dump(ns["ATCODES"], "SRE")) + content.extend(dump(ns["CHCODES"], "SRE")) + content.extend(dump2(ns, "SRE_FLAG_")) + content.extend(dump2(ns, "SRE_INFO_")) + + update_file(outfile, ''.join(content)) + + +if __name__ == '__main__': + import sys + main(*sys.argv[1:]) From webhook-mailer at python.org Tue Apr 12 15:31:06 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Tue, 12 Apr 2022 19:31:06 -0000 Subject: [Python-checkins] gh-91243: Add typing.Required and NotRequired (PEP 655) (GH-32419) Message-ID: https://github.com/python/cpython/commit/ac6c3de03c5bb06a9a463701fb297148f5a5746f commit: ac6c3de03c5bb06a9a463701fb297148f5a5746f branch: main author: Jelle Zijlstra committer: JelleZijlstra date: 2022-04-12T12:31:02-07:00 summary: gh-91243: Add typing.Required and NotRequired (PEP 655) (GH-32419) I talked to @davidfstr and I offered to implement the runtime part of PEP 655 to make sure we can get it in before the feature freeze. We're going to defer the documentation to a separate PR, because it can wait until after the feature freeze. The runtime implementation conveniently already exists in typing-extensions, so I largely copied that. Co-authored-by: David Foster files: A Misc/NEWS.d/next/Library/2022-04-08-08-55-36.bpo-47087.Q5C3EI.rst M Lib/test/_typed_dict_helper.py M Lib/test/test_typing.py M Lib/typing.py diff --git a/Lib/test/_typed_dict_helper.py b/Lib/test/_typed_dict_helper.py index d333db193183e..3328330c995b3 100644 --- a/Lib/test/_typed_dict_helper.py +++ b/Lib/test/_typed_dict_helper.py @@ -6,13 +6,19 @@ class Bar(_typed_dict_helper.Foo, total=False): b: int + +In addition, it uses multiple levels of Annotated to test the interaction +between the __future__ import, Annotated, and Required. """ from __future__ import annotations -from typing import Optional, TypedDict +from typing import Annotated, Optional, Required, TypedDict OptionalIntType = Optional[int] class Foo(TypedDict): a: OptionalIntType + +class VeryAnnotated(TypedDict, total=False): + a: Annotated[Annotated[Annotated[Required[int], "a"], "b"], "c"] diff --git a/Lib/test/test_typing.py b/Lib/test/test_typing.py index b884f7b2cced3..2afa5449bc2e6 100644 --- a/Lib/test/test_typing.py +++ b/Lib/test/test_typing.py @@ -23,7 +23,7 @@ from typing import reveal_type from typing import no_type_check, no_type_check_decorator from typing import Type -from typing import NamedTuple, TypedDict +from typing import NamedTuple, NotRequired, Required, TypedDict from typing import IO, TextIO, BinaryIO from typing import Pattern, Match from typing import Annotated, ForwardRef @@ -3993,6 +3993,26 @@ class Options(TypedDict, total=False): log_level: int log_path: str +class TotalMovie(TypedDict): + title: str + year: NotRequired[int] + +class NontotalMovie(TypedDict, total=False): + title: Required[str] + year: int + +class AnnotatedMovie(TypedDict): + title: Annotated[Required[str], "foobar"] + year: NotRequired[Annotated[int, 2000]] + +class DeeplyAnnotatedMovie(TypedDict): + title: Annotated[Annotated[Required[str], "foobar"], "another level"] + year: NotRequired[Annotated[int, 2000]] + +class WeirdlyQuotedMovie(TypedDict): + title: Annotated['Annotated[Required[str], "foobar"]', "another level"] + year: NotRequired['Annotated[int, 2000]'] + class HasForeignBaseClass(mod_generics_cache.A): some_xrepr: 'XRepr' other_a: 'mod_generics_cache.A' @@ -4280,6 +4300,36 @@ def test_top_level_class_var(self): ): get_type_hints(ann_module6) + def test_get_type_hints_typeddict(self): + self.assertEqual(get_type_hints(TotalMovie), {'title': str, 'year': int}) + self.assertEqual(get_type_hints(TotalMovie, include_extras=True), { + 'title': str, + 'year': NotRequired[int], + }) + + self.assertEqual(get_type_hints(AnnotatedMovie), {'title': str, 'year': int}) + self.assertEqual(get_type_hints(AnnotatedMovie, include_extras=True), { + 'title': Annotated[Required[str], "foobar"], + 'year': NotRequired[Annotated[int, 2000]], + }) + + self.assertEqual(get_type_hints(DeeplyAnnotatedMovie), {'title': str, 'year': int}) + self.assertEqual(get_type_hints(DeeplyAnnotatedMovie, include_extras=True), { + 'title': Annotated[Required[str], "foobar", "another level"], + 'year': NotRequired[Annotated[int, 2000]], + }) + + self.assertEqual(get_type_hints(WeirdlyQuotedMovie), {'title': str, 'year': int}) + self.assertEqual(get_type_hints(WeirdlyQuotedMovie, include_extras=True), { + 'title': Annotated[Required[str], "foobar", "another level"], + 'year': NotRequired[Annotated[int, 2000]], + }) + + self.assertEqual(get_type_hints(_typed_dict_helper.VeryAnnotated), {'a': int}) + self.assertEqual(get_type_hints(_typed_dict_helper.VeryAnnotated, include_extras=True), { + 'a': Annotated[Required[int], "a", "b", "c"] + }) + class GetUtilitiesTestCase(TestCase): def test_get_origin(self): @@ -4305,6 +4355,8 @@ class C(Generic[T]): pass self.assertIs(get_origin(list | str), types.UnionType) self.assertIs(get_origin(P.args), P) self.assertIs(get_origin(P.kwargs), P) + self.assertIs(get_origin(Required[int]), Required) + self.assertIs(get_origin(NotRequired[int]), NotRequired) def test_get_args(self): T = TypeVar('T') @@ -4342,6 +4394,8 @@ class C(Generic[T]): pass self.assertEqual(get_args(Callable[Concatenate[int, P], int]), (Concatenate[int, P], int)) self.assertEqual(get_args(list | str), (list, str)) + self.assertEqual(get_args(Required[int]), (int,)) + self.assertEqual(get_args(NotRequired[int]), (int,)) class CollectionsAbcTests(BaseTestCase): @@ -5299,6 +5353,32 @@ class Cat(Animal): 'voice': str, } + def test_required_notrequired_keys(self): + self.assertEqual(NontotalMovie.__required_keys__, + frozenset({"title"})) + self.assertEqual(NontotalMovie.__optional_keys__, + frozenset({"year"})) + + self.assertEqual(TotalMovie.__required_keys__, + frozenset({"title"})) + self.assertEqual(TotalMovie.__optional_keys__, + frozenset({"year"})) + + self.assertEqual(_typed_dict_helper.VeryAnnotated.__required_keys__, + frozenset()) + self.assertEqual(_typed_dict_helper.VeryAnnotated.__optional_keys__, + frozenset({"a"})) + + self.assertEqual(AnnotatedMovie.__required_keys__, + frozenset({"title"})) + self.assertEqual(AnnotatedMovie.__optional_keys__, + frozenset({"year"})) + + self.assertEqual(WeirdlyQuotedMovie.__required_keys__, + frozenset({"title"})) + self.assertEqual(WeirdlyQuotedMovie.__optional_keys__, + frozenset({"year"})) + def test_multiple_inheritance(self): class One(TypedDict): one: int @@ -5399,6 +5479,98 @@ def test_get_type_hints(self): ) +class RequiredTests(BaseTestCase): + + def test_basics(self): + with self.assertRaises(TypeError): + Required[NotRequired] + with self.assertRaises(TypeError): + Required[int, str] + with self.assertRaises(TypeError): + Required[int][str] + + def test_repr(self): + self.assertEqual(repr(Required), 'typing.Required') + cv = Required[int] + self.assertEqual(repr(cv), 'typing.Required[int]') + cv = Required[Employee] + self.assertEqual(repr(cv), f'typing.Required[{__name__}.Employee]') + + def test_cannot_subclass(self): + with self.assertRaises(TypeError): + class C(type(Required)): + pass + with self.assertRaises(TypeError): + class C(type(Required[int])): + pass + with self.assertRaises(TypeError): + class C(Required): + pass + with self.assertRaises(TypeError): + class C(Required[int]): + pass + + def test_cannot_init(self): + with self.assertRaises(TypeError): + Required() + with self.assertRaises(TypeError): + type(Required)() + with self.assertRaises(TypeError): + type(Required[Optional[int]])() + + def test_no_isinstance(self): + with self.assertRaises(TypeError): + isinstance(1, Required[int]) + with self.assertRaises(TypeError): + issubclass(int, Required) + + +class NotRequiredTests(BaseTestCase): + + def test_basics(self): + with self.assertRaises(TypeError): + NotRequired[Required] + with self.assertRaises(TypeError): + NotRequired[int, str] + with self.assertRaises(TypeError): + NotRequired[int][str] + + def test_repr(self): + self.assertEqual(repr(NotRequired), 'typing.NotRequired') + cv = NotRequired[int] + self.assertEqual(repr(cv), 'typing.NotRequired[int]') + cv = NotRequired[Employee] + self.assertEqual(repr(cv), f'typing.NotRequired[{__name__}.Employee]') + + def test_cannot_subclass(self): + with self.assertRaises(TypeError): + class C(type(NotRequired)): + pass + with self.assertRaises(TypeError): + class C(type(NotRequired[int])): + pass + with self.assertRaises(TypeError): + class C(NotRequired): + pass + with self.assertRaises(TypeError): + class C(NotRequired[int]): + pass + + def test_cannot_init(self): + with self.assertRaises(TypeError): + NotRequired() + with self.assertRaises(TypeError): + type(NotRequired)() + with self.assertRaises(TypeError): + type(NotRequired[Optional[int]])() + + def test_no_isinstance(self): + with self.assertRaises(TypeError): + isinstance(1, NotRequired[int]) + with self.assertRaises(TypeError): + issubclass(int, NotRequired) + + class IOTests(BaseTestCase): def test_io(self): diff --git a/Lib/typing.py b/Lib/typing.py index ec8cbbd8b20a3..46fc72218dceb 100644 --- a/Lib/typing.py +++ b/Lib/typing.py @@ -132,9 +132,11 @@ def _idfunc(_, x): 'no_type_check', 'no_type_check_decorator', 'NoReturn', + 'NotRequired', 'overload', 'ParamSpecArgs', 'ParamSpecKwargs', + 'Required', 'reveal_type', 'runtime_checkable', 'Self', @@ -2262,6 +2264,8 @@ def _strip_annotations(t): """ if isinstance(t, _AnnotatedAlias): return _strip_annotations(t.__origin__) + if hasattr(t, "__origin__") and t.__origin__ in (Required, NotRequired): + return _strip_annotations(t.__args__[0]) if isinstance(t, _GenericAlias): stripped_args = tuple(_strip_annotations(a) for a in t.__args__) if stripped_args == t.__args__: @@ -2786,10 +2790,22 @@ def __new__(cls, name, bases, ns, total=True): optional_keys.update(base.__dict__.get('__optional_keys__', ())) annotations.update(own_annotations) - if total: - required_keys.update(own_annotation_keys) - else: - optional_keys.update(own_annotation_keys) + for annotation_key, annotation_type in own_annotations.items(): + annotation_origin = get_origin(annotation_type) + if annotation_origin is Annotated: + annotation_args = get_args(annotation_type) + if annotation_args: + annotation_type = annotation_args[0] + annotation_origin = get_origin(annotation_type) + + if annotation_origin is Required: + required_keys.add(annotation_key) + elif annotation_origin is NotRequired: + optional_keys.add(annotation_key) + elif total: + required_keys.add(annotation_key) + else: + optional_keys.add(annotation_key) tp_dict.__annotations__ = annotations tp_dict.__required_keys__ = frozenset(required_keys) @@ -2874,6 +2890,45 @@ class body be required. TypedDict.__mro_entries__ = lambda bases: (_TypedDict,) + at _SpecialForm +def Required(self, parameters): + """A special typing construct to mark a key of a total=False TypedDict + as required. For example: + + class Movie(TypedDict, total=False): + title: Required[str] + year: int + + m = Movie( + title='The Matrix', # typechecker error if key is omitted + year=1999, + ) + + There is no runtime checking that a required key is actually provided + when instantiating a related TypedDict. + """ + item = _type_check(parameters, f'{self._name} accepts only a single type.') + return _GenericAlias(self, (item,)) + + + at _SpecialForm +def NotRequired(self, parameters): + """A special typing construct to mark a key of a TypedDict as + potentially missing. For example: + + class Movie(TypedDict): + title: str + year: NotRequired[int] + + m = Movie( + title='The Matrix', # typechecker error if key is omitted + year=1999, + ) + """ + item = _type_check(parameters, f'{self._name} accepts only a single type.') + return _GenericAlias(self, (item,)) + + class NewType: """NewType creates simple unique types with almost zero runtime overhead. NewType(name, tp) is considered a subtype of tp diff --git a/Misc/NEWS.d/next/Library/2022-04-08-08-55-36.bpo-47087.Q5C3EI.rst b/Misc/NEWS.d/next/Library/2022-04-08-08-55-36.bpo-47087.Q5C3EI.rst new file mode 100644 index 0000000000000..3399663214045 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-08-08-55-36.bpo-47087.Q5C3EI.rst @@ -0,0 +1,2 @@ +Implement ``typing.Required`` and ``typing.NotRequired`` (:pep:`655`). Patch +by Jelle Zijlstra. From webhook-mailer at python.org Tue Apr 12 22:58:17 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Wed, 13 Apr 2022 02:58:17 -0000 Subject: [Python-checkins] gh-88513: clarify shutil.copytree's dirs_exist_ok arg (GH-91434) (GH-91465) Message-ID: https://github.com/python/cpython/commit/a8d245a675c11972390d1cfaea4a97151162e6e7 commit: a8d245a675c11972390d1cfaea4a97151162e6e7 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: JelleZijlstra date: 2022-04-12T19:58:02-07:00 summary: gh-88513: clarify shutil.copytree's dirs_exist_ok arg (GH-91434) (GH-91465) * add a paragraph to document this kwarg in detail * update docstring in the source accordingly (cherry picked from commit f33e2c87a83917b5139d97fd8ef7cba7223ebef5) Co-authored-by: Jack DeVries files: A Misc/NEWS.d/next/Documentation/2022-04-10-20-28-20.bpo-44347.Q1m3DM.rst M Doc/library/shutil.rst M Lib/shutil.py diff --git a/Doc/library/shutil.rst b/Doc/library/shutil.rst index 5f71049f918e0..8ce0e80ec8120 100644 --- a/Doc/library/shutil.rst +++ b/Doc/library/shutil.rst @@ -230,9 +230,8 @@ Directory and files operations dirs_exist_ok=False) Recursively copy an entire directory tree rooted at *src* to a directory - named *dst* and return the destination directory. *dirs_exist_ok* dictates - whether to raise an exception in case *dst* or any missing parent directory - already exists. + named *dst* and return the destination directory. All intermediate + directories needed to contain *dst* will also be created by default. Permissions and times of directories are copied with :func:`copystat`, individual files are copied using :func:`~shutil.copy2`. @@ -263,8 +262,14 @@ Directory and files operations If *copy_function* is given, it must be a callable that will be used to copy each file. It will be called with the source path and the destination path - ? as arguments. By default, :func:`~shutil.copy2` is used, but any function - ? that supports the same signature (like :func:`~shutil.copy`) can be used. + as arguments. By default, :func:`~shutil.copy2` is used, but any function + that supports the same signature (like :func:`~shutil.copy`) can be used. + + If *dirs_exist_ok* is false (the default) and *dst* already exists, a + :exc:`FileExistsError` is raised. If *dirs_exist_ok* is true, the copying + operation will continue if it encounters existing directories, and files + within the *dst* tree will be overwritten by corresponding files from the + *src* tree. .. audit-event:: shutil.copytree src,dst shutil.copytree @@ -275,7 +280,7 @@ Directory and files operations .. versionchanged:: 3.2 Added the *copy_function* argument to be able to provide a custom copy function. - Added the *ignore_dangling_symlinks* argument to silent dangling symlinks + Added the *ignore_dangling_symlinks* argument to silence dangling symlinks errors when *symlinks* is false. .. versionchanged:: 3.8 diff --git a/Lib/shutil.py b/Lib/shutil.py index c048cdf9b2cbd..48a60c0d28d07 100644 --- a/Lib/shutil.py +++ b/Lib/shutil.py @@ -526,9 +526,6 @@ def copytree(src, dst, symlinks=False, ignore=None, copy_function=copy2, ignore_dangling_symlinks=False, dirs_exist_ok=False): """Recursively copy a directory tree and return the destination directory. - dirs_exist_ok dictates whether to raise an exception in case dst or any - missing parent directory already exists. - If exception(s) occur, an Error is raised with a list of reasons. If the optional symlinks flag is true, symbolic links in the @@ -559,6 +556,11 @@ def copytree(src, dst, symlinks=False, ignore=None, copy_function=copy2, destination path as arguments. By default, copy2() is used, but any function that supports the same signature (like copy()) can be used. + If dirs_exist_ok is false (the default) and `dst` already exists, a + `FileExistsError` is raised. If `dirs_exist_ok` is true, the copying + operation will continue if it encounters existing directories, and files + within the `dst` tree will be overwritten by corresponding files from the + `src` tree. """ sys.audit("shutil.copytree", src, dst) with os.scandir(src) as itr: diff --git a/Misc/NEWS.d/next/Documentation/2022-04-10-20-28-20.bpo-44347.Q1m3DM.rst b/Misc/NEWS.d/next/Documentation/2022-04-10-20-28-20.bpo-44347.Q1m3DM.rst new file mode 100644 index 0000000000000..27aa5742cd008 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2022-04-10-20-28-20.bpo-44347.Q1m3DM.rst @@ -0,0 +1 @@ +Clarify the meaning of *dirs_exist_ok*, a kwarg of :func:`shutil.copytree`. From webhook-mailer at python.org Tue Apr 12 23:01:06 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Wed, 13 Apr 2022 03:01:06 -0000 Subject: [Python-checkins] gh-91421: Use constant value check during runtime (GH-91422) Message-ID: https://github.com/python/cpython/commit/0859368335d470b9ff33fc53ed9a85ec2654b278 commit: 0859368335d470b9ff33fc53ed9a85ec2654b278 branch: main author: Tobias Stoeckmann committer: JelleZijlstra date: 2022-04-12T20:01:02-07:00 summary: gh-91421: Use constant value check during runtime (GH-91422) The left-hand side expression of the if-check can be converted to a constant by the compiler, but the addition on the right-hand side is performed during runtime. Move the addition from the right-hand side to the left-hand side by turning it into a subtraction there. Since the values are known to be large enough to not turn negative, this is a safe operation. Prevents a very unlikely integer overflow on 32 bit systems. Fixes GH-91421. files: A Misc/NEWS.d/next/Core and Builtins/2022-04-10-22-57-27.gh-issue-91421.dHhv6U.rst M Objects/unicodeobject.c diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-04-10-22-57-27.gh-issue-91421.dHhv6U.rst b/Misc/NEWS.d/next/Core and Builtins/2022-04-10-22-57-27.gh-issue-91421.dHhv6U.rst new file mode 100644 index 0000000000000..898eb0df18d01 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2022-04-10-22-57-27.gh-issue-91421.dHhv6U.rst @@ -0,0 +1 @@ +Fix a potential integer overflow in _Py_DecodeUTF8Ex. diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c index c665f577abaa4..d35a671a81680 100644 --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -5296,7 +5296,7 @@ _Py_DecodeUTF8Ex(const char *s, Py_ssize_t size, wchar_t **wstr, size_t *wlen, /* Note: size will always be longer than the resulting Unicode character count */ - if (PY_SSIZE_T_MAX / (Py_ssize_t)sizeof(wchar_t) < (size + 1)) { + if (PY_SSIZE_T_MAX / (Py_ssize_t)sizeof(wchar_t) - 1 < size) { return -1; } From webhook-mailer at python.org Tue Apr 12 23:09:13 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Wed, 13 Apr 2022 03:09:13 -0000 Subject: [Python-checkins] bpo-43224: Forbid TypeVar substitution with Unpack (GH-32031) Message-ID: https://github.com/python/cpython/commit/15537c51c188a6633248c25d211d5216e673aee3 commit: 15537c51c188a6633248c25d211d5216e673aee3 branch: main author: Serhiy Storchaka committer: JelleZijlstra date: 2022-04-12T20:08:49-07:00 summary: bpo-43224: Forbid TypeVar substitution with Unpack (GH-32031) files: M Lib/test/test_typing.py M Lib/typing.py diff --git a/Lib/test/test_typing.py b/Lib/test/test_typing.py index 2afa5449bc2e6..97fc66a2f748f 100644 --- a/Lib/test/test_typing.py +++ b/Lib/test/test_typing.py @@ -562,6 +562,20 @@ class G(Generic[Unpack[Ts]]): pass self.assertEqual(E[float, str, int, bytes], Tuple[List[float], A[str, int], List[bytes]]) + def test_bad_var_substitution(self): + Ts = TypeVarTuple('Ts') + T = TypeVar('T') + T2 = TypeVar('T2') + class G(Generic[Unpack[Ts]]): pass + + for A in G, Tuple: + B = A[T, Unpack[Ts], str, T2] + with self.assertRaises(TypeError): + B[int, Unpack[Ts]] + C = A[T, Unpack[Ts], str, T2] + with self.assertRaises(TypeError): + C[int, Unpack[Ts], Unpack[Ts]] + def test_repr_is_correct(self): Ts = TypeVarTuple('Ts') self.assertEqual(repr(Ts), 'Ts') diff --git a/Lib/typing.py b/Lib/typing.py index 46fc72218dceb..1b584bea0c3e5 100644 --- a/Lib/typing.py +++ b/Lib/typing.py @@ -965,6 +965,8 @@ def __init__(self, name, *constraints, bound=None, def __typing_subst__(self, arg): msg = "Parameters to generic types must be types." arg = _type_check(arg, msg, is_argument=True) + if (isinstance(arg, _GenericAlias) and arg.__origin__ is Unpack): + raise TypeError(f"{arg} is not valid as type argument") return arg From webhook-mailer at python.org Wed Apr 13 04:07:32 2022 From: webhook-mailer at python.org (vsajip) Date: Wed, 13 Apr 2022 08:07:32 -0000 Subject: [Python-checkins] bpo-43218: Prevent venv creation when the target directory contains a PATH separator. (GH-24530) Message-ID: https://github.com/python/cpython/commit/54f67ad54366f1b9161edca283e19670e065e1b8 commit: 54f67ad54366f1b9161edca283e19670e065e1b8 branch: main author: Dustin Rodrigues committer: vsajip date: 2022-04-13T09:07:10+01:00 summary: bpo-43218: Prevent venv creation when the target directory contains a PATH separator. (GH-24530) files: A Misc/NEWS.d/next/Library/2021-02-14-20-55-53.bpo-43218.VZv2M4.rst M Lib/test/test_venv.py M Lib/venv/__init__.py diff --git a/Lib/test/test_venv.py b/Lib/test/test_venv.py index c91b75493841b..d96cf1e6c7493 100644 --- a/Lib/test/test_venv.py +++ b/Lib/test/test_venv.py @@ -467,6 +467,14 @@ def test_macos_env(self): 'import os; print("__PYVENV_LAUNCHER__" in os.environ)']) self.assertEqual(out.strip(), 'False'.encode()) + def test_pathsep_error(self): + """ + Test that venv creation fails when the target directory contains + the path separator. + """ + rmtree(self.env_dir) + self.assertRaises(ValueError, venv.create, self.env_dir + os.pathsep) + @requireVenvCreate class EnsurePipTest(BaseTest): """Test venv module installation of pip.""" diff --git a/Lib/venv/__init__.py b/Lib/venv/__init__.py index a8640d9163fbe..7bfbadda7b497 100644 --- a/Lib/venv/__init__.py +++ b/Lib/venv/__init__.py @@ -116,6 +116,9 @@ def create_if_needed(d): elif os.path.islink(d) or os.path.isfile(d): raise ValueError('Unable to create directory %r' % d) + if os.pathsep in env_dir: + raise ValueError(f'Refusing to create a venv in {env_dir} because ' + f'it contains the PATH separator {os.pathsep}.') if os.path.exists(env_dir) and self.clear: self.clear_directory(env_dir) context = types.SimpleNamespace() diff --git a/Misc/NEWS.d/next/Library/2021-02-14-20-55-53.bpo-43218.VZv2M4.rst b/Misc/NEWS.d/next/Library/2021-02-14-20-55-53.bpo-43218.VZv2M4.rst new file mode 100644 index 0000000000000..31229c35a2148 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-02-14-20-55-53.bpo-43218.VZv2M4.rst @@ -0,0 +1,2 @@ +Prevent creation of a venv whose path contains the PATH separator. This +could affect the usage of the activate script. Patch by Dustin Rodrigues. From webhook-mailer at python.org Wed Apr 13 09:07:06 2022 From: webhook-mailer at python.org (pablogsal) Date: Wed, 13 Apr 2022 13:07:06 -0000 Subject: [Python-checkins] gh-91502: Add a new API to check if a frame is an entry frame (GH-91503) Message-ID: https://github.com/python/cpython/commit/37a53fb6bd36964d97faad628d16db0b2b62704c commit: 37a53fb6bd36964d97faad628d16db0b2b62704c branch: main author: Pablo Galindo Salgado committer: pablogsal date: 2022-04-13T14:06:56+01:00 summary: gh-91502: Add a new API to check if a frame is an entry frame (GH-91503) files: A Misc/NEWS.d/next/Core and Builtins/2022-04-13-11-15-09.gh-issue-91502.11YXHQ.rst M Include/cpython/frameobject.h M Objects/frameobject.c diff --git a/Include/cpython/frameobject.h b/Include/cpython/frameobject.h index 01cf6c9b89ae4..9cd711e43559a 100644 --- a/Include/cpython/frameobject.h +++ b/Include/cpython/frameobject.h @@ -19,6 +19,16 @@ PyAPI_FUNC(PyFrameObject *) PyFrame_New(PyThreadState *, PyCodeObject *, PyAPI_FUNC(void) PyFrame_LocalsToFast(PyFrameObject *, int); +/* -- Caveat emptor -- + * The concept of entry frames is an implementation detail of the CPython + * interpreter. This API is considered unstable and is provided for the + * convenience of debuggers, profilers and state-inspecting tools. Notice that + * this API can be changed in future minor versions if the underlying frame + * mechanism change or the concept of an 'entry frame' or its semantics becomes + * obsolete or outdated. */ + +PyAPI_FUNC(int) _PyFrame_IsEntryFrame(PyFrameObject *frame); + PyAPI_FUNC(int) PyFrame_FastToLocalsWithError(PyFrameObject *f); PyAPI_FUNC(void) PyFrame_FastToLocals(PyFrameObject *); diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-04-13-11-15-09.gh-issue-91502.11YXHQ.rst b/Misc/NEWS.d/next/Core and Builtins/2022-04-13-11-15-09.gh-issue-91502.11YXHQ.rst new file mode 100644 index 0000000000000..4edff6f005490 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2022-04-13-11-15-09.gh-issue-91502.11YXHQ.rst @@ -0,0 +1,2 @@ +Add a new :c:func:`_PyFrame_IsEntryFrame` API function, to check if a +:c:type:`PyFrameObject` is an entry frame. Patch by Pablo Galindo. diff --git a/Objects/frameobject.c b/Objects/frameobject.c index 7a4d2fac9382c..dc4ef8bcda541 100644 --- a/Objects/frameobject.c +++ b/Objects/frameobject.c @@ -1104,6 +1104,14 @@ PyFrame_LocalsToFast(PyFrameObject *f, int clear) } } + +int _PyFrame_IsEntryFrame(PyFrameObject *frame) +{ + assert(frame != NULL); + return frame->f_frame->is_entry; +} + + PyCodeObject * PyFrame_GetCode(PyFrameObject *frame) { From webhook-mailer at python.org Wed Apr 13 10:42:37 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Wed, 13 Apr 2022 14:42:37 -0000 Subject: [Python-checkins] gh-90971: suppress deprecation warning in `test_lib2to3`(GH-31464) Message-ID: https://github.com/python/cpython/commit/63a032270eba726d36a67e6a33a87284d50e2658 commit: 63a032270eba726d36a67e6a33a87284d50e2658 branch: main author: Nikita Sobolev committer: JelleZijlstra date: 2022-04-13T07:42:25-07:00 summary: gh-90971: suppress deprecation warning in `test_lib2to3`(GH-31464) Fixes GH-90971 Co-authored-by: Jelle Zijlstra Co-authored-by: ?ric files: M Lib/lib2to3/tests/__init__.py M Lib/lib2to3/tests/test_parser.py diff --git a/Lib/lib2to3/tests/__init__.py b/Lib/lib2to3/tests/__init__.py index 54221c7994386..f323c2355b228 100644 --- a/Lib/lib2to3/tests/__init__.py +++ b/Lib/lib2to3/tests/__init__.py @@ -1,8 +1,11 @@ # Author: Collin Winter import os +import warnings from test.support import load_package_tests def load_tests(*args): - return load_package_tests(os.path.dirname(__file__), *args) + with warnings.catch_warnings(): + warnings.filterwarnings('ignore', category=DeprecationWarning, message='lib2to3') + return load_package_tests(os.path.dirname(__file__), *args) diff --git a/Lib/lib2to3/tests/test_parser.py b/Lib/lib2to3/tests/test_parser.py index 74a5787574209..e2dddbec577c0 100644 --- a/Lib/lib2to3/tests/test_parser.py +++ b/Lib/lib2to3/tests/test_parser.py @@ -92,10 +92,8 @@ def test_load_grammar_from_subprocess(self): from lib2to3.pgen2 import driver as pgen2_driver pgen2_driver.load_grammar(%r, save=True, force=True) """ % (grammar_sub_copy,) - msg = ("lib2to3 package is deprecated and may not be able " - "to parse Python 3.10+") cmd = [sys.executable, - f'-Wignore:{msg}:PendingDeprecationWarning', + '-Wignore:lib2to3:DeprecationWarning', '-c', code] subprocess.check_call( cmd, env=sub_env) self.assertTrue(os.path.exists(pickle_sub_name)) From webhook-mailer at python.org Wed Apr 13 10:52:32 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Wed, 13 Apr 2022 14:52:32 -0000 Subject: [Python-checkins] gh-91243: Update authors for Required[] and NotRequired[] implementation (GH-91506) Message-ID: https://github.com/python/cpython/commit/dfbc792a4b1e033d6628eda463c0933aef081bbe commit: dfbc792a4b1e033d6628eda463c0933aef081bbe branch: main author: David Foster committer: JelleZijlstra date: 2022-04-13T07:52:19-07:00 summary: gh-91243: Update authors for Required[] and NotRequired[] implementation (GH-91506) files: M Misc/NEWS.d/next/Library/2022-04-08-08-55-36.bpo-47087.Q5C3EI.rst diff --git a/Misc/NEWS.d/next/Library/2022-04-08-08-55-36.bpo-47087.Q5C3EI.rst b/Misc/NEWS.d/next/Library/2022-04-08-08-55-36.bpo-47087.Q5C3EI.rst index 3399663214045..ca686b7da873b 100644 --- a/Misc/NEWS.d/next/Library/2022-04-08-08-55-36.bpo-47087.Q5C3EI.rst +++ b/Misc/NEWS.d/next/Library/2022-04-08-08-55-36.bpo-47087.Q5C3EI.rst @@ -1,2 +1,2 @@ Implement ``typing.Required`` and ``typing.NotRequired`` (:pep:`655`). Patch -by Jelle Zijlstra. +by David Foster and Jelle Zijlstra. From webhook-mailer at python.org Wed Apr 13 13:47:55 2022 From: webhook-mailer at python.org (warsaw) Date: Wed, 13 Apr 2022 17:47:55 -0000 Subject: [Python-checkins] gh-91217: deprecate imghdr (#91461) Message-ID: https://github.com/python/cpython/commit/3fc57e8f6ff925b561b03c46bcf5bd323782c19c commit: 3fc57e8f6ff925b561b03c46bcf5bd323782c19c branch: main author: Brett Cannon committer: warsaw date: 2022-04-13T10:47:41-07:00 summary: gh-91217: deprecate imghdr (#91461) * Deprecate imghdr Co-authored-by: Hugo van Kemenade * Update Doc/whatsnew/3.11.rst Co-authored-by: Hugo van Kemenade * Inline `imghdr` into `email.mime.image` Co-authored-by: Hugo van Kemenade Co-authored-by: Barry Warsaw files: A Misc/NEWS.d/next/Library/2022-04-11-17-04-38.gh-issue-91217.QVDLOq.rst M Doc/includes/email-mime.py M Doc/library/email.mime.rst M Doc/whatsnew/3.11.rst M Lib/email/mime/image.py M Lib/imghdr.py M Lib/test/test_email/test_email.py M Lib/test/test_imghdr.py diff --git a/Doc/includes/email-mime.py b/Doc/includes/email-mime.py index 6af2be0b08a48..c87db6a064b00 100644 --- a/Doc/includes/email-mime.py +++ b/Doc/includes/email-mime.py @@ -1,9 +1,6 @@ # Import smtplib for the actual sending function import smtplib -# And imghdr to find the types of our images -import imghdr - # Here are the email package modules we'll need from email.message import EmailMessage @@ -22,7 +19,7 @@ with open(file, 'rb') as fp: img_data = fp.read() msg.add_attachment(img_data, maintype='image', - subtype=imghdr.what(None, img_data)) + subtype='jpeg') # Send the email via our own SMTP server. with smtplib.SMTP('localhost') as s: diff --git a/Doc/library/email.mime.rst b/Doc/library/email.mime.rst index f37f6aa28dec7..ab4f7bc54e025 100644 --- a/Doc/library/email.mime.rst +++ b/Doc/library/email.mime.rst @@ -180,11 +180,12 @@ Here are the classes: A subclass of :class:`~email.mime.nonmultipart.MIMENonMultipart`, the :class:`MIMEImage` class is used to create MIME message objects of major type :mimetype:`image`. *_imagedata* is a string containing the raw image data. If - this data can be decoded by the standard Python module :mod:`imghdr`, then the - subtype will be automatically included in the :mailheader:`Content-Type` header. - Otherwise you can explicitly specify the image subtype via the *_subtype* - argument. If the minor type could not be guessed and *_subtype* was not given, - then :exc:`TypeError` is raised. + this data type can be detected (jpeg, png, gif, tiff, rgb, pbm, pgm, ppm, + rast, xbm, bmp, webp, and exr attempted), then the subtype will be + automatically included in the :mailheader:`Content-Type` header. Otherwise + you can explicitly specify the image subtype via the *_subtype* argument. + If the minor type could not be guessed and *_subtype* was not given, then + :exc:`TypeError` is raised. Optional *_encoder* is a callable (i.e. function) which will perform the actual encoding of the image data for transport. This callable takes one argument, diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index d803801f273c5..894ec8a9d0d92 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -856,6 +856,7 @@ Deprecated * :mod:`cgitb` * :mod:`chunk` * :mod:`crypt` + * :mod:`imghdr` (Contributed by Brett Cannon in :issue:`47061`.) diff --git a/Lib/email/mime/image.py b/Lib/email/mime/image.py index 92724643cdeea..fac238c7289fa 100644 --- a/Lib/email/mime/image.py +++ b/Lib/email/mime/image.py @@ -6,13 +6,113 @@ __all__ = ['MIMEImage'] -import imghdr - from email import encoders from email.mime.nonmultipart import MIMENonMultipart - +# Originally from the imghdr module. +def _what(h): + for tf in tests: + if res := tf(h): + return res + else: + return None + +tests = [] + +def _test_jpeg(h): + """JPEG data with JFIF or Exif markers; and raw JPEG""" + if h[6:10] in (b'JFIF', b'Exif'): + return 'jpeg' + elif h[:4] == b'\xff\xd8\xff\xdb': + return 'jpeg' + +tests.append(_test_jpeg) + +def _test_png(h): + if h.startswith(b'\211PNG\r\n\032\n'): + return 'png' + +tests.append(_test_png) + +def _test_gif(h): + """GIF ('87 and '89 variants)""" + if h[:6] in (b'GIF87a', b'GIF89a'): + return 'gif' + +tests.append(_test_gif) + +def _test_tiff(h): + """TIFF (can be in Motorola or Intel byte order)""" + if h[:2] in (b'MM', b'II'): + return 'tiff' + +tests.append(_test_tiff) + +def _test_rgb(h): + """SGI image library""" + if h.startswith(b'\001\332'): + return 'rgb' + +tests.append(_test_rgb) + +def _test_pbm(h): + """PBM (portable bitmap)""" + if len(h) >= 3 and \ + h[0] == ord(b'P') and h[1] in b'14' and h[2] in b' \t\n\r': + return 'pbm' + +tests.append(_test_pbm) + +def _test_pgm(h): + """PGM (portable graymap)""" + if len(h) >= 3 and \ + h[0] == ord(b'P') and h[1] in b'25' and h[2] in b' \t\n\r': + return 'pgm' + +tests.append(_test_pgm) + +def _test_ppm(h): + """PPM (portable pixmap)""" + if len(h) >= 3 and \ + h[0] == ord(b'P') and h[1] in b'36' and h[2] in b' \t\n\r': + return 'ppm' + +tests.append(_test_ppm) + +def _test_rast(h): + """Sun raster file""" + if h.startswith(b'\x59\xA6\x6A\x95'): + return 'rast' + +tests.append(_test_rast) + +def _test_xbm(h): + """X bitmap (X10 or X11)""" + if h.startswith(b'#define '): + return 'xbm' + +tests.append(_test_xbm) + +def _test_bmp(h): + if h.startswith(b'BM'): + return 'bmp' + +tests.append(_test_bmp) + +def _test_webp(h): + if h.startswith(b'RIFF') and h[8:12] == b'WEBP': + return 'webp' + +tests.append(_test_webp) + +def _test_exr(h): + if h.startswith(b'\x76\x2f\x31\x01'): + return 'exr' + +tests.append(_test_exr) + + class MIMEImage(MIMENonMultipart): """Class for generating image/* type MIME documents.""" @@ -20,11 +120,11 @@ def __init__(self, _imagedata, _subtype=None, _encoder=encoders.encode_base64, *, policy=None, **_params): """Create an image/* type MIME document. - _imagedata is a string containing the raw image data. If this data - can be decoded by the standard Python `imghdr' module, then the - subtype will be automatically included in the Content-Type header. - Otherwise, you can specify the specific image subtype via the _subtype - parameter. + _imagedata is a string containing the raw image data. If the data + type can be detected (jpeg, png, gif, tiff, rgb, pbm, pgm, ppm, + rast, xbm, bmp, webp, and exr attempted), then the subtype will be + automatically included in the Content-Type header. Otherwise, you can + specify the specific image subtype via the _subtype parameter. _encoder is a function which will perform the actual encoding for transport of the image data. It takes one argument, which is this @@ -38,9 +138,8 @@ def __init__(self, _imagedata, _subtype=None, header. """ if _subtype is None: - _subtype = imghdr.what(None, _imagedata) - if _subtype is None: - raise TypeError('Could not guess image MIME subtype') + if (_subtype := _what(_imagedata)) is None: + raise TypeError('Could not guess image MIME subtype') MIMENonMultipart.__init__(self, 'image', _subtype, policy=policy, **_params) self.set_payload(_imagedata) diff --git a/Lib/imghdr.py b/Lib/imghdr.py index afcb67772ee9a..6a372e66c7f26 100644 --- a/Lib/imghdr.py +++ b/Lib/imghdr.py @@ -1,9 +1,14 @@ """Recognize image file formats based on their first few bytes.""" from os import PathLike +import warnings __all__ = ["what"] + +warnings._deprecated(__name__, remove=(3, 13)) + + #-------------------------# # Recognize image headers # #-------------------------# diff --git a/Lib/test/test_email/test_email.py b/Lib/test/test_email/test_email.py index ca9c773bbc559..b87dae22de1d2 100644 --- a/Lib/test/test_email/test_email.py +++ b/Lib/test/test_email/test_email.py @@ -7,6 +7,7 @@ import base64 import unittest import textwrap +import warnings from io import StringIO, BytesIO from itertools import chain @@ -1591,7 +1592,6 @@ def test_add_header(self): header='foobar'), missing) - # Test the basic MIMEApplication class class TestMIMEApplication(unittest.TestCase): def test_headers(self): diff --git a/Lib/test/test_imghdr.py b/Lib/test/test_imghdr.py index ca0a0b23c3cf1..208c8eee455e7 100644 --- a/Lib/test/test_imghdr.py +++ b/Lib/test/test_imghdr.py @@ -1,12 +1,13 @@ -import imghdr import io import os import pathlib import unittest import warnings -from test.support import findfile +from test.support import findfile, warnings_helper from test.support.os_helper import TESTFN, unlink +imghdr = warnings_helper.import_deprecated("imghdr") + TEST_FILES = ( ('python.png', 'png'), diff --git a/Misc/NEWS.d/next/Library/2022-04-11-17-04-38.gh-issue-91217.QVDLOq.rst b/Misc/NEWS.d/next/Library/2022-04-11-17-04-38.gh-issue-91217.QVDLOq.rst new file mode 100644 index 0000000000000..3e59c205aae29 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-11-17-04-38.gh-issue-91217.QVDLOq.rst @@ -0,0 +1 @@ +Deprecate the imghdr module. From webhook-mailer at python.org Wed Apr 13 16:45:58 2022 From: webhook-mailer at python.org (iritkatriel) Date: Wed, 13 Apr 2022 20:45:58 -0000 Subject: [Python-checkins] gh-90449: Improve accuracy and readability of exceptions tutorial (GH-31899) Message-ID: https://github.com/python/cpython/commit/04f9658c591f61f0d45ab1a818840a4c4ef82f64 commit: 04f9658c591f61f0d45ab1a818840a4c4ef82f64 branch: main author: Irit Katriel <1055913+iritkatriel at users.noreply.github.com> committer: iritkatriel <1055913+iritkatriel at users.noreply.github.com> date: 2022-04-13T21:45:33+01:00 summary: gh-90449: Improve accuracy and readability of exceptions tutorial (GH-31899) files: M Doc/tutorial/errors.rst diff --git a/Doc/tutorial/errors.rst b/Doc/tutorial/errors.rst index 888740cdd0f19..01fd76283556b 100644 --- a/Doc/tutorial/errors.rst +++ b/Doc/tutorial/errors.rst @@ -147,10 +147,52 @@ For example, the following code will print B, C, D in that order:: Note that if the *except clauses* were reversed (with ``except B`` first), it would have printed B, B, B --- the first matching *except clause* is triggered. -All exceptions inherit from :exc:`BaseException`, and so it can be used to serve -as a wildcard. Use this with extreme caution, since it is easy to mask a real -programming error in this way! It can also be used to print an error message and -then re-raise the exception (allowing a caller to handle the exception as well):: +When an exception occurs, it may have associated values, also known as the +exception's *arguments*. The presence and types of the arguments depend on the +exception type. + +The *except clause* may specify a variable after the exception name. The +variable is bound to the exception instance which typically has an ``args`` +attribute that stores the arguments. For convenience, builtin exception +types define :meth:`__str__` to print all the arguments without explicitly +accessing ``.args``. :: + + >>> try: + ... raise Exception('spam', 'eggs') + ... except Exception as inst: + ... print(type(inst)) # the exception instance + ... print(inst.args) # arguments stored in .args + ... print(inst) # __str__ allows args to be printed directly, + ... # but may be overridden in exception subclasses + ... x, y = inst.args # unpack args + ... print('x =', x) + ... print('y =', y) + ... + + ('spam', 'eggs') + ('spam', 'eggs') + x = spam + y = eggs + +The exception's :meth:`__str__` output is printed as the last part ('detail') +of the message for unhandled exceptions. + +:exc:`BaseException` is the common base class of all exceptions. One of its +subclasses, :exc:`Exception`, is the base class of all the non-fatal exceptions. +Exceptions which are not subclasses of :exc:`Exception` are not typically +handled, because they are used to indicate that the program should terminate. +They include :exc:`SystemExit` which is raised by :meth:`sys.exit` and +:exc:`KeyboardInterrupt` which is raised when a user wishes to interrupt +the program. + +:exc:`Exception` can be used as a wildcard that catches (almost) everything. +However, it is good practice to be as specific as possible with the types +of exceptions that we intend to handle, and to allow any unexpected +exceptions to propagate on. + +The most common pattern for handling :exc:`Exception` is to print or log +the exception and then re-raise it (allowing a caller to handle the +exception as well):: import sys @@ -159,16 +201,13 @@ then re-raise the exception (allowing a caller to handle the exception as well): s = f.readline() i = int(s.strip()) except OSError as err: - print("OS error: {0}".format(err)) + print("OS error:", err) except ValueError: print("Could not convert data to an integer.") - except BaseException as err: + except Exception as err: print(f"Unexpected {err=}, {type(err)=}") raise -Alternatively the last except clause may omit the exception name(s), however the exception -value must then be retrieved with ``sys.exception()``. - The :keyword:`try` ... :keyword:`except` statement has an optional *else clause*, which, when present, must follow all *except clauses*. It is useful for code that must be executed if the *try clause* does not raise an exception. @@ -188,39 +227,8 @@ the :keyword:`try` clause because it avoids accidentally catching an exception that wasn't raised by the code being protected by the :keyword:`!try` ... :keyword:`!except` statement. -When an exception occurs, it may have an associated value, also known as the -exception's *argument*. The presence and type of the argument depend on the -exception type. - -The *except clause* may specify a variable after the exception name. The -variable is bound to an exception instance with the arguments stored in -``instance.args``. For convenience, the exception instance defines -:meth:`__str__` so the arguments can be printed directly without having to -reference ``.args``. One may also instantiate an exception first before -raising it and add any attributes to it as desired. :: - - >>> try: - ... raise Exception('spam', 'eggs') - ... except Exception as inst: - ... print(type(inst)) # the exception instance - ... print(inst.args) # arguments stored in .args - ... print(inst) # __str__ allows args to be printed directly, - ... # but may be overridden in exception subclasses - ... x, y = inst.args # unpack args - ... print('x =', x) - ... print('y =', y) - ... - - ('spam', 'eggs') - ('spam', 'eggs') - x = spam - y = eggs - -If an exception has arguments, they are printed as the last part ('detail') of -the message for unhandled exceptions. - -Exception handlers don't just handle exceptions if they occur immediately in the -*try clause*, but also if they occur inside functions that are called (even +Exception handlers do not handle only exceptions that occur immediately in the +*try clause*, but also those that occur inside functions that are called (even indirectly) in the *try clause*. For example:: >>> def this_fails(): @@ -249,8 +257,9 @@ exception to occur. For example:: The sole argument to :keyword:`raise` indicates the exception to be raised. This must be either an exception instance or an exception class (a class that -derives from :class:`Exception`). If an exception class is passed, it will -be implicitly instantiated by calling its constructor with no arguments:: +derives from :class:`BaseException`, such as :exc:`Exception` or one of its +subclasses). If an exception class is passed, it will be implicitly +instantiated by calling its constructor with no arguments:: raise ValueError # shorthand for 'raise ValueError()' @@ -335,8 +344,7 @@ Most exceptions are defined with names that end in "Error", similar to the naming of the standard exceptions. Many standard modules define their own exceptions to report errors that may -occur in functions they define. More information on classes is presented in -chapter :ref:`tut-classes`. +occur in functions they define. .. _tut-cleanup: From webhook-mailer at python.org Wed Apr 13 18:09:56 2022 From: webhook-mailer at python.org (brandtbucher) Date: Wed, 13 Apr 2022 22:09:56 -0000 Subject: [Python-checkins] Fill holes in internal compiler structs (#91458) Message-ID: https://github.com/python/cpython/commit/5f056acdf28662e773ec5fe27db47ccad0e7671f commit: 5f056acdf28662e773ec5fe27db47ccad0e7671f branch: main author: L. A. F. Pereira committer: brandtbucher date: 2022-04-13T15:09:20-07:00 summary: Fill holes in internal compiler structs (#91458) files: M Python/compile.c diff --git a/Python/compile.c b/Python/compile.c index 38cf5f362705a..718b521858b27 100644 --- a/Python/compile.c +++ b/Python/compile.c @@ -233,19 +233,23 @@ typedef struct basicblock_ { reverse order that the block are allocated. b_list points to the next block, not to be confused with b_next, which is next by control flow. */ struct basicblock_ *b_list; - /* number of instructions used */ - int b_iused; - /* length of instruction array (b_instr) */ - int b_ialloc; + /* Exception stack at start of block, used by assembler to create the exception handling table */ + ExceptStack *b_exceptstack; /* pointer to an array of instructions, initially NULL */ struct instr *b_instr; /* If b_next is non-NULL, it is a pointer to the next block reached by normal control flow. */ struct basicblock_ *b_next; - /* b_return is true if a RETURN_VALUE opcode is inserted. */ - unsigned b_return : 1; + /* number of instructions used */ + int b_iused; + /* length of instruction array (b_instr) */ + int b_ialloc; /* Number of predecssors that a block has. */ int b_predecessors; + /* depth of stack upon entry of block, computed by stackdepth() */ + int b_startdepth; + /* instruction offset for block, computed by assemble_jump_offsets() */ + int b_offset; /* Basic block has no fall through (it ends with a return, raise or jump) */ unsigned b_nofallthrough : 1; /* Basic block is an exception handler that preserves lasti */ @@ -254,12 +258,8 @@ typedef struct basicblock_ { unsigned b_visited : 1; /* Basic block exits scope (it ends with a return or raise) */ unsigned b_exit : 1; - /* depth of stack upon entry of block, computed by stackdepth() */ - int b_startdepth; - /* instruction offset for block, computed by assemble_jump_offsets() */ - int b_offset; - /* Exception stack at start of block, used by assembler to create the exception handling table */ - ExceptStack *b_exceptstack; + /* b_return is true if a RETURN_VALUE opcode is inserted. */ + unsigned b_return : 1; } basicblock; /* fblockinfo tracks the current frame block. @@ -7051,23 +7051,23 @@ compiler_match(struct compiler *c, stmt_ty s) struct assembler { PyObject *a_bytecode; /* bytes containing bytecode */ + PyObject *a_lnotab; /* bytes containing lnotab */ + PyObject *a_enotab; /* bytes containing enotab */ + PyObject *a_cnotab; /* bytes containing cnotab */ + PyObject *a_except_table; /* bytes containing exception table */ + basicblock *a_entry; int a_offset; /* offset into bytecode */ int a_nblocks; /* number of reachable blocks */ - PyObject *a_lnotab; /* bytes containing lnotab */ - PyObject* a_enotab; /* bytes containing enotab */ - PyObject* a_cnotab; /* bytes containing cnotab */ + int a_except_table_off; /* offset into exception table */ int a_lnotab_off; /* offset into lnotab */ int a_enotab_off; /* offset into enotab */ int a_cnotab_off; /* offset into cnotab */ - PyObject *a_except_table; /* bytes containing exception table */ - int a_except_table_off; /* offset into exception table */ int a_prevlineno; /* lineno of last emitted line in line table */ int a_prev_end_lineno; /* end_lineno of last emitted line in line table */ int a_lineno; /* lineno of last emitted instruction */ int a_end_lineno; /* end_lineno of last emitted instruction */ int a_lineno_start; /* bytecode start offset of current lineno */ int a_end_lineno_start; /* bytecode start offset of current end_lineno */ - basicblock *a_entry; }; Py_LOCAL_INLINE(void) From webhook-mailer at python.org Wed Apr 13 21:38:47 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Thu, 14 Apr 2022 01:38:47 -0000 Subject: [Python-checkins] gh-91421: Use constant value check during runtime (GH-91422) (GH-91492) Message-ID: https://github.com/python/cpython/commit/72114c06fd8f105437d671277b0593a378ecc3f4 commit: 72114c06fd8f105437d671277b0593a378ecc3f4 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: JelleZijlstra date: 2022-04-13T18:38:37-07:00 summary: gh-91421: Use constant value check during runtime (GH-91422) (GH-91492) The left-hand side expression of the if-check can be converted to a constant by the compiler, but the addition on the right-hand side is performed during runtime. Move the addition from the right-hand side to the left-hand side by turning it into a subtraction there. Since the values are known to be large enough to not turn negative, this is a safe operation. Prevents a very unlikely integer overflow on 32 bit systems. Fixes GH-91421. (cherry picked from commit 0859368335d470b9ff33fc53ed9a85ec2654b278) Co-authored-by: Tobias Stoeckmann files: A Misc/NEWS.d/next/Core and Builtins/2022-04-10-22-57-27.gh-issue-91421.dHhv6U.rst M Objects/unicodeobject.c diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-04-10-22-57-27.gh-issue-91421.dHhv6U.rst b/Misc/NEWS.d/next/Core and Builtins/2022-04-10-22-57-27.gh-issue-91421.dHhv6U.rst new file mode 100644 index 0000000000000..898eb0df18d01 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2022-04-10-22-57-27.gh-issue-91421.dHhv6U.rst @@ -0,0 +1 @@ +Fix a potential integer overflow in _Py_DecodeUTF8Ex. diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c index 38a7b3c8b44b9..5ddde5054428a 100644 --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -5373,7 +5373,7 @@ _Py_DecodeUTF8Ex(const char *s, Py_ssize_t size, wchar_t **wstr, size_t *wlen, /* Note: size will always be longer than the resulting Unicode character count */ - if (PY_SSIZE_T_MAX / (Py_ssize_t)sizeof(wchar_t) < (size + 1)) { + if (PY_SSIZE_T_MAX / (Py_ssize_t)sizeof(wchar_t) - 1 < size) { return -1; } From webhook-mailer at python.org Wed Apr 13 21:38:59 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Thu, 14 Apr 2022 01:38:59 -0000 Subject: [Python-checkins] gh-91421: Use constant value check during runtime (GH-91422) (GH-91493) Message-ID: https://github.com/python/cpython/commit/edf1a77f239069235f59103cfd8ce7f939c7fd10 commit: edf1a77f239069235f59103cfd8ce7f939c7fd10 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: JelleZijlstra date: 2022-04-13T18:38:55-07:00 summary: gh-91421: Use constant value check during runtime (GH-91422) (GH-91493) The left-hand side expression of the if-check can be converted to a constant by the compiler, but the addition on the right-hand side is performed during runtime. Move the addition from the right-hand side to the left-hand side by turning it into a subtraction there. Since the values are known to be large enough to not turn negative, this is a safe operation. Prevents a very unlikely integer overflow on 32 bit systems. Fixes GH-91421. (cherry picked from commit 0859368335d470b9ff33fc53ed9a85ec2654b278) Co-authored-by: Tobias Stoeckmann files: A Misc/NEWS.d/next/Core and Builtins/2022-04-10-22-57-27.gh-issue-91421.dHhv6U.rst M Objects/unicodeobject.c diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-04-10-22-57-27.gh-issue-91421.dHhv6U.rst b/Misc/NEWS.d/next/Core and Builtins/2022-04-10-22-57-27.gh-issue-91421.dHhv6U.rst new file mode 100644 index 0000000000000..898eb0df18d01 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2022-04-10-22-57-27.gh-issue-91421.dHhv6U.rst @@ -0,0 +1 @@ +Fix a potential integer overflow in _Py_DecodeUTF8Ex. diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c index 7767d140e6c39..369f91342e3ac 100644 --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -5219,7 +5219,7 @@ _Py_DecodeUTF8Ex(const char *s, Py_ssize_t size, wchar_t **wstr, size_t *wlen, /* Note: size will always be longer than the resulting Unicode character count */ - if (PY_SSIZE_T_MAX / (Py_ssize_t)sizeof(wchar_t) < (size + 1)) { + if (PY_SSIZE_T_MAX / (Py_ssize_t)sizeof(wchar_t) - 1 < size) { return -1; } From webhook-mailer at python.org Wed Apr 13 22:19:26 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Thu, 14 Apr 2022 02:19:26 -0000 Subject: [Python-checkins] gh-87497: Document that urllib.request sends headers in camel case (GH-24661) Message-ID: https://github.com/python/cpython/commit/325d6f50357474c7d9fd2475be0e2481f7ae0476 commit: 325d6f50357474c7d9fd2475be0e2481f7ae0476 branch: main author: Alix Lourme committer: JelleZijlstra date: 2022-04-13T19:19:16-07:00 summary: gh-87497: Document that urllib.request sends headers in camel case (GH-24661) Co-authored-by: Jelle Zijlstra files: M Doc/library/urllib.request.rst M Lib/test/test_urllib2_localnet.py diff --git a/Doc/library/urllib.request.rst b/Doc/library/urllib.request.rst index 88e93ba6b002e..a8501ab863968 100644 --- a/Doc/library/urllib.request.rst +++ b/Doc/library/urllib.request.rst @@ -218,6 +218,7 @@ The following classes are provided: (X11; U; Linux i686) Gecko/20071127 Firefox/2.0.0.11"``, while :mod:`urllib`'s default user agent string is ``"Python-urllib/2.6"`` (on Python 2.6). + All header keys are sent in camel case. An appropriate ``Content-Type`` header should be included if the *data* argument is present. If this header has not been provided and *data* diff --git a/Lib/test/test_urllib2_localnet.py b/Lib/test/test_urllib2_localnet.py index 36314312ac0eb..f4729358557c9 100644 --- a/Lib/test/test_urllib2_localnet.py +++ b/Lib/test/test_urllib2_localnet.py @@ -617,6 +617,15 @@ def test_sending_headers(self): pass self.assertEqual(handler.headers_received["Range"], "bytes=20-39") + def test_sending_headers_camel(self): + handler = self.start_server() + req = urllib.request.Request("http://localhost:%s/" % handler.port, + headers={"X-SoMe-hEader": "foobar"}) + with urllib.request.urlopen(req): + pass + self.assertIn("X-Some-Header", handler.headers_received.keys()) + self.assertNotIn("X-SoMe-hEader", handler.headers_received.keys()) + def test_basic(self): handler = self.start_server() with urllib.request.urlopen("http://localhost:%s" % handler.port) as open_url: From webhook-mailer at python.org Wed Apr 13 22:20:42 2022 From: webhook-mailer at python.org (sweeneyde) Date: Thu, 14 Apr 2022 02:20:42 -0000 Subject: [Python-checkins] gh-91266: refactor bytearray strip methods (GH-32096) Message-ID: https://github.com/python/cpython/commit/355cbaadbbdce82b08431999d9de89661324263f commit: 355cbaadbbdce82b08431999d9de89661324263f branch: main author: Pieter Eendebak committer: sweeneyde <36520290+sweeneyde at users.noreply.github.com> date: 2022-04-13T22:20:38-04:00 summary: gh-91266: refactor bytearray strip methods (GH-32096) files: A Misc/NEWS.d/next/Core and Builtins/2022-04-13-07-14-30.gh-issue-91266.6Vkzzt.rst M Objects/bytearrayobject.c diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-04-13-07-14-30.gh-issue-91266.6Vkzzt.rst b/Misc/NEWS.d/next/Core and Builtins/2022-04-13-07-14-30.gh-issue-91266.6Vkzzt.rst new file mode 100644 index 0000000000000..bb1d7dd90f17f --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2022-04-13-07-14-30.gh-issue-91266.6Vkzzt.rst @@ -0,0 +1 @@ +Refactor the ``bytearray`` strip methods ``strip``, ``lstrip`` and ``rstrip`` to use a common implementation. diff --git a/Objects/bytearrayobject.c b/Objects/bytearrayobject.c index f784e0448191f..b9436d9aa5b1a 100644 --- a/Objects/bytearrayobject.c +++ b/Objects/bytearrayobject.c @@ -1845,26 +1845,46 @@ bytearray_remove_impl(PyByteArrayObject *self, int value) Py_RETURN_NONE; } -/* XXX These two helpers could be optimized if argsize == 1 */ +#define LEFTSTRIP 0 +#define RIGHTSTRIP 1 +#define BOTHSTRIP 2 -static Py_ssize_t -lstrip_helper(const char *myptr, Py_ssize_t mysize, - const void *argptr, Py_ssize_t argsize) +static PyObject* +bytearray_strip_impl_helper(PyByteArrayObject* self, PyObject* bytes, int striptype) { - Py_ssize_t i = 0; - while (i < mysize && memchr(argptr, (unsigned char) myptr[i], argsize)) - i++; - return i; -} + Py_ssize_t mysize, byteslen; + const char* myptr; + const char* bytesptr; + Py_buffer vbytes; -static Py_ssize_t -rstrip_helper(const char *myptr, Py_ssize_t mysize, - const void *argptr, Py_ssize_t argsize) -{ - Py_ssize_t i = mysize - 1; - while (i >= 0 && memchr(argptr, (unsigned char) myptr[i], argsize)) - i--; - return i + 1; + if (bytes == Py_None) { + bytesptr = "\t\n\r\f\v "; + byteslen = 6; + } + else { + if (PyObject_GetBuffer(bytes, &vbytes, PyBUF_SIMPLE) != 0) + return NULL; + bytesptr = (const char*)vbytes.buf; + byteslen = vbytes.len; + } + myptr = PyByteArray_AS_STRING(self); + mysize = Py_SIZE(self); + + Py_ssize_t left = 0; + if (striptype != RIGHTSTRIP) { + while (left < mysize && memchr(bytesptr, (unsigned char)myptr[left], byteslen)) + left++; + } + Py_ssize_t right = mysize; + if (striptype != LEFTSTRIP) { + do { + right--; + } while (right >= left && memchr(bytesptr, (unsigned char)myptr[right], byteslen)); + right++; + } + if (bytes != Py_None) + PyBuffer_Release(&vbytes); + return PyByteArray_FromStringAndSize(myptr + left, right - left); } /*[clinic input] @@ -1882,31 +1902,7 @@ static PyObject * bytearray_strip_impl(PyByteArrayObject *self, PyObject *bytes) /*[clinic end generated code: output=760412661a34ad5a input=ef7bb59b09c21d62]*/ { - Py_ssize_t left, right, mysize, byteslen; - char *myptr; - const char *bytesptr; - Py_buffer vbytes; - - if (bytes == Py_None) { - bytesptr = "\t\n\r\f\v "; - byteslen = 6; - } - else { - if (PyObject_GetBuffer(bytes, &vbytes, PyBUF_SIMPLE) != 0) - return NULL; - bytesptr = (const char *) vbytes.buf; - byteslen = vbytes.len; - } - myptr = PyByteArray_AS_STRING(self); - mysize = Py_SIZE(self); - left = lstrip_helper(myptr, mysize, bytesptr, byteslen); - if (left == mysize) - right = left; - else - right = rstrip_helper(myptr, mysize, bytesptr, byteslen); - if (bytes != Py_None) - PyBuffer_Release(&vbytes); - return PyByteArray_FromStringAndSize(myptr + left, right - left); + return bytearray_strip_impl_helper(self, bytes, BOTHSTRIP); } /*[clinic input] @@ -1924,28 +1920,7 @@ static PyObject * bytearray_lstrip_impl(PyByteArrayObject *self, PyObject *bytes) /*[clinic end generated code: output=d005c9d0ab909e66 input=80843f975dd7c480]*/ { - Py_ssize_t left, right, mysize, byteslen; - char *myptr; - const char *bytesptr; - Py_buffer vbytes; - - if (bytes == Py_None) { - bytesptr = "\t\n\r\f\v "; - byteslen = 6; - } - else { - if (PyObject_GetBuffer(bytes, &vbytes, PyBUF_SIMPLE) != 0) - return NULL; - bytesptr = (const char *) vbytes.buf; - byteslen = vbytes.len; - } - myptr = PyByteArray_AS_STRING(self); - mysize = Py_SIZE(self); - left = lstrip_helper(myptr, mysize, bytesptr, byteslen); - right = mysize; - if (bytes != Py_None) - PyBuffer_Release(&vbytes); - return PyByteArray_FromStringAndSize(myptr + left, right - left); + return bytearray_strip_impl_helper(self, bytes, LEFTSTRIP); } /*[clinic input] @@ -1963,27 +1938,7 @@ static PyObject * bytearray_rstrip_impl(PyByteArrayObject *self, PyObject *bytes) /*[clinic end generated code: output=030e2fbd2f7276bd input=e728b994954cfd91]*/ { - Py_ssize_t right, mysize, byteslen; - char *myptr; - const char *bytesptr; - Py_buffer vbytes; - - if (bytes == Py_None) { - bytesptr = "\t\n\r\f\v "; - byteslen = 6; - } - else { - if (PyObject_GetBuffer(bytes, &vbytes, PyBUF_SIMPLE) != 0) - return NULL; - bytesptr = (const char *) vbytes.buf; - byteslen = vbytes.len; - } - myptr = PyByteArray_AS_STRING(self); - mysize = Py_SIZE(self); - right = rstrip_helper(myptr, mysize, bytesptr, byteslen); - if (bytes != Py_None) - PyBuffer_Release(&vbytes); - return PyByteArray_FromStringAndSize(myptr, right); + return bytearray_strip_impl_helper(self, bytes, RIGHTSTRIP); } /*[clinic input] From webhook-mailer at python.org Wed Apr 13 22:46:05 2022 From: webhook-mailer at python.org (miss-islington) Date: Thu, 14 Apr 2022 02:46:05 -0000 Subject: [Python-checkins] gh-87497: Document that urllib.request sends headers in camel case (GH-24661) Message-ID: https://github.com/python/cpython/commit/0ab5e83ff78e5e871768440ab0a7030bc825398f commit: 0ab5e83ff78e5e871768440ab0a7030bc825398f branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-13T19:45:56-07:00 summary: gh-87497: Document that urllib.request sends headers in camel case (GH-24661) Co-authored-by: Jelle Zijlstra (cherry picked from commit 325d6f50357474c7d9fd2475be0e2481f7ae0476) Co-authored-by: Alix Lourme files: M Doc/library/urllib.request.rst M Lib/test/test_urllib2_localnet.py diff --git a/Doc/library/urllib.request.rst b/Doc/library/urllib.request.rst index bfb994e315468..1307143a04f1d 100644 --- a/Doc/library/urllib.request.rst +++ b/Doc/library/urllib.request.rst @@ -213,6 +213,7 @@ The following classes are provided: (X11; U; Linux i686) Gecko/20071127 Firefox/2.0.0.11"``, while :mod:`urllib`'s default user agent string is ``"Python-urllib/2.6"`` (on Python 2.6). + All header keys are sent in camel case. An appropriate ``Content-Type`` header should be included if the *data* argument is present. If this header has not been provided and *data* diff --git a/Lib/test/test_urllib2_localnet.py b/Lib/test/test_urllib2_localnet.py index 74374c08684da..162e356d1f1bb 100644 --- a/Lib/test/test_urllib2_localnet.py +++ b/Lib/test/test_urllib2_localnet.py @@ -613,6 +613,15 @@ def test_sending_headers(self): pass self.assertEqual(handler.headers_received["Range"], "bytes=20-39") + def test_sending_headers_camel(self): + handler = self.start_server() + req = urllib.request.Request("http://localhost:%s/" % handler.port, + headers={"X-SoMe-hEader": "foobar"}) + with urllib.request.urlopen(req): + pass + self.assertIn("X-Some-Header", handler.headers_received.keys()) + self.assertNotIn("X-SoMe-hEader", handler.headers_received.keys()) + def test_basic(self): handler = self.start_server() with urllib.request.urlopen("http://localhost:%s" % handler.port) as open_url: From webhook-mailer at python.org Thu Apr 14 00:15:04 2022 From: webhook-mailer at python.org (vsajip) Date: Thu, 14 Apr 2022 04:15:04 -0000 Subject: [Python-checkins] gh-90326: Remove quotes for logging config (GH-91516) Message-ID: https://github.com/python/cpython/commit/45e8c9d43ff4f53c6b6f9f00d94d02ee36299b36 commit: 45e8c9d43ff4f53c6b6f9f00d94d02ee36299b36 branch: main author: slateny <46876382+slateny at users.noreply.github.com> committer: vsajip date: 2022-04-14T05:15:00+01:00 summary: gh-90326: Remove quotes for logging config (GH-91516) files: M Doc/library/logging.config.rst diff --git a/Doc/library/logging.config.rst b/Doc/library/logging.config.rst index c979961a221c9..2f105ba29441d 100644 --- a/Doc/library/logging.config.rst +++ b/Doc/library/logging.config.rst @@ -833,7 +833,7 @@ Sections which specify formatter configuration are typified by the following. [formatter_form01] format=F1 %(asctime)s %(levelname)s %(message)s datefmt= - style='%' + style=% validate=True class=logging.Formatter From webhook-mailer at python.org Thu Apr 14 01:57:36 2022 From: webhook-mailer at python.org (corona10) Date: Thu, 14 Apr 2022 05:57:36 -0000 Subject: [Python-checkins] gh-90699: Remove usage of _Py_IDENTIFIER from bisect module. (GH-91522) Message-ID: https://github.com/python/cpython/commit/7b87e8af0cb8df0d76e8ab18a9b12affb4526103 commit: 7b87e8af0cb8df0d76e8ab18a9b12affb4526103 branch: main author: Dong-hee Na committer: corona10 date: 2022-04-14T14:57:25+09:00 summary: gh-90699: Remove usage of _Py_IDENTIFIER from bisect module. (GH-91522) files: M Modules/_bisectmodule.c diff --git a/Modules/_bisectmodule.c b/Modules/_bisectmodule.c index f884d94413c0d..c54dc4979d00b 100644 --- a/Modules/_bisectmodule.c +++ b/Modules/_bisectmodule.c @@ -4,7 +4,6 @@ Converted to C by Dmitry Vasiliev (dima at hlabs.spb.ru). */ #define PY_SSIZE_T_CLEAN -#define NEEDS_PY_IDENTIFIER #include "Python.h" /*[clinic input] @@ -14,8 +13,6 @@ module _bisect #include "clinic/_bisectmodule.c.h" -_Py_IDENTIFIER(insert); - static inline Py_ssize_t internal_bisect_right(PyObject *list, PyObject *item, Py_ssize_t lo, Py_ssize_t hi, PyObject* key) @@ -132,7 +129,7 @@ _bisect_insort_right_impl(PyObject *module, PyObject *a, PyObject *x, return NULL; } else { - result = _PyObject_CallMethodId(a, &PyId_insert, "nO", index, x); + result = PyObject_CallMethod(a, "insert", "nO", index, x); if (result == NULL) return NULL; Py_DECREF(result); @@ -258,7 +255,7 @@ _bisect_insort_left_impl(PyObject *module, PyObject *a, PyObject *x, if (PyList_Insert(a, index, x) < 0) return NULL; } else { - result = _PyObject_CallMethodId(a, &PyId_insert, "nO", index, x); + result = PyObject_CallMethod(a, "insert", "nO", index, x); if (result == NULL) return NULL; Py_DECREF(result); From webhook-mailer at python.org Thu Apr 14 03:00:41 2022 From: webhook-mailer at python.org (methane) Date: Thu, 14 Apr 2022 07:00:41 -0000 Subject: [Python-checkins] gh-91156: Fix `encoding="locale"` in UTF-8 mode (GH-70056) Message-ID: https://github.com/python/cpython/commit/13b17e2a0a56d506d62a0bf2774e3deb4cbaeb72 commit: 13b17e2a0a56d506d62a0bf2774e3deb4cbaeb72 branch: main author: Inada Naoki committer: methane date: 2022-04-14T16:00:35+09:00 summary: gh-91156: Fix `encoding="locale"` in UTF-8 mode (GH-70056) files: A Misc/NEWS.d/next/Library/2022-04-10-17-50-18.bpo-47000.JlQkFx.rst M Doc/library/io.rst M Doc/using/windows.rst M Lib/_pyio.py M Lib/locale.py M Lib/test/test_io.py M Modules/_io/_iomodule.c M Modules/_io/clinic/_iomodule.c.h M Modules/_io/clinic/textio.c.h M Modules/_io/textio.c M Tools/c-analyzer/TODO diff --git a/Doc/library/io.rst b/Doc/library/io.rst index 80107d539505c..53dad99fa1dbc 100644 --- a/Doc/library/io.rst +++ b/Doc/library/io.rst @@ -112,7 +112,7 @@ Text Encoding ------------- The default encoding of :class:`TextIOWrapper` and :func:`open` is -locale-specific (:func:`locale.getpreferredencoding(False) `). +locale-specific (:func:`locale.getencoding`). However, many developers forget to specify the encoding when opening text files encoded in UTF-8 (e.g. JSON, TOML, Markdown, etc...) since most Unix @@ -948,8 +948,7 @@ Text I/O :class:`TextIOBase`. *encoding* gives the name of the encoding that the stream will be decoded or - encoded with. It defaults to - :func:`locale.getpreferredencoding(False) `. + encoded with. It defaults to :func:`locale.getencoding()`. ``encoding="locale"`` can be used to specify the current locale's encoding explicitly. See :ref:`io-text-encoding` for more information. diff --git a/Doc/using/windows.rst b/Doc/using/windows.rst index 83eee281d4e5c..88dcb002e2c24 100644 --- a/Doc/using/windows.rst +++ b/Doc/using/windows.rst @@ -618,7 +618,7 @@ UTF-8 mode Windows still uses legacy encodings for the system encoding (the ANSI Code Page). Python uses it for the default encoding of text files (e.g. -:func:`locale.getpreferredencoding`). +:func:`locale.getencoding`). This may cause issues because UTF-8 is widely used on the internet and most Unix systems, including WSL (Windows Subsystem for Linux). diff --git a/Lib/_pyio.py b/Lib/_pyio.py index e3ff59eb1adb1..0f33ed59492e7 100644 --- a/Lib/_pyio.py +++ b/Lib/_pyio.py @@ -1988,7 +1988,7 @@ class TextIOWrapper(TextIOBase): r"""Character and line based layer over a BufferedIOBase object, buffer. encoding gives the name of the encoding that the stream will be - decoded or encoded with. It defaults to locale.getpreferredencoding(False). + decoded or encoded with. It defaults to locale.getencoding(). errors determines the strictness of encoding and decoding (see the codecs.register) and defaults to "strict". @@ -2021,7 +2021,9 @@ def __init__(self, buffer, encoding=None, errors=None, newline=None, self._check_newline(newline) encoding = text_encoding(encoding) - if encoding == "locale": + if encoding == "locale" and sys.platform == "win32": + # On Unix, os.device_encoding() returns "utf-8" instead of locale encoding + # in the UTF-8 mode. So we use os.device_encoding() only on Windows. try: encoding = os.device_encoding(buffer.fileno()) or "locale" except (AttributeError, UnsupportedOperation): @@ -2034,7 +2036,7 @@ def __init__(self, buffer, encoding=None, errors=None, newline=None, # Importing locale may fail if Python is being built encoding = "utf-8" else: - encoding = locale.getpreferredencoding(False) + encoding = locale.getencoding() if not isinstance(encoding, str): raise ValueError("invalid encoding: %r" % encoding) diff --git a/Lib/locale.py b/Lib/locale.py index 496cc803c88f7..170e5eea45b8c 100644 --- a/Lib/locale.py +++ b/Lib/locale.py @@ -557,7 +557,7 @@ def getdefaultlocale(envvars=('LC_ALL', 'LC_CTYPE', 'LANG', 'LANGUAGE')): import warnings warnings.warn( - "Use setlocale(), getpreferredencoding(False) and getlocale() instead", + "Use setlocale(), getencoding() and getlocale() instead", DeprecationWarning, stacklevel=2 ) diff --git a/Lib/test/test_io.py b/Lib/test/test_io.py index 29fe287550b2d..c86251dfe5734 100644 --- a/Lib/test/test_io.py +++ b/Lib/test/test_io.py @@ -2737,6 +2737,7 @@ def test_default_encoding(self): os.environ.update(old_environ) @support.cpython_only + @unittest.skipIf(sys.platform != "win32", "Windows-only test") @unittest.skipIf(sys.flags.utf8_mode, "utf-8 mode is enabled") def test_device_encoding(self): # Issue 15989 diff --git a/Misc/NEWS.d/next/Library/2022-04-10-17-50-18.bpo-47000.JlQkFx.rst b/Misc/NEWS.d/next/Library/2022-04-10-17-50-18.bpo-47000.JlQkFx.rst new file mode 100644 index 0000000000000..77d5b8f33d9a0 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-10-17-50-18.bpo-47000.JlQkFx.rst @@ -0,0 +1,2 @@ +Make :class:`TextIOWrapper` uses locale encoding when ``encoding="locale"`` +is specified even in UTF-8 mode. diff --git a/Modules/_io/_iomodule.c b/Modules/_io/_iomodule.c index 065f5e29c315b..38ef24637b731 100644 --- a/Modules/_io/_iomodule.c +++ b/Modules/_io/_iomodule.c @@ -92,9 +92,9 @@ it already exists), 'x' for creating and writing to a new file, and 'a' for appending (which on some Unix systems, means that all writes append to the end of the file regardless of the current seek position). In text mode, if encoding is not specified the encoding used is platform -dependent: locale.getpreferredencoding(False) is called to get the -current locale encoding. (For reading and writing raw bytes use binary -mode and leave encoding unspecified.) The available modes are: +dependent: locale.getencoding() is called to get the current locale encoding. +(For reading and writing raw bytes use binary mode and leave encoding +unspecified.) The available modes are: ========= =============================================================== Character Meaning @@ -196,7 +196,7 @@ static PyObject * _io_open_impl(PyObject *module, PyObject *file, const char *mode, int buffering, const char *encoding, const char *errors, const char *newline, int closefd, PyObject *opener) -/*[clinic end generated code: output=aefafc4ce2b46dc0 input=1543f4511d2356a5]*/ +/*[clinic end generated code: output=aefafc4ce2b46dc0 input=5bb37f174cb2fb11]*/ { unsigned i; diff --git a/Modules/_io/clinic/_iomodule.c.h b/Modules/_io/clinic/_iomodule.c.h index e4a6b8c42e1d8..1fdbe6835c717 100644 --- a/Modules/_io/clinic/_iomodule.c.h +++ b/Modules/_io/clinic/_iomodule.c.h @@ -22,9 +22,9 @@ PyDoc_STRVAR(_io_open__doc__, "\'a\' for appending (which on some Unix systems, means that all writes\n" "append to the end of the file regardless of the current seek position).\n" "In text mode, if encoding is not specified the encoding used is platform\n" -"dependent: locale.getpreferredencoding(False) is called to get the\n" -"current locale encoding. (For reading and writing raw bytes use binary\n" -"mode and leave encoding unspecified.) The available modes are:\n" +"dependent: locale.getencoding() is called to get the current locale encoding.\n" +"(For reading and writing raw bytes use binary mode and leave encoding\n" +"unspecified.) The available modes are:\n" "\n" "========= ===============================================================\n" "Character Meaning\n" @@ -355,4 +355,4 @@ _io_open_code(PyObject *module, PyObject *const *args, Py_ssize_t nargs, PyObjec exit: return return_value; } -/*[clinic end generated code: output=1a7fd7755c9a9609 input=a9049054013a1b77]*/ +/*[clinic end generated code: output=e562f29e3c2533a6 input=a9049054013a1b77]*/ diff --git a/Modules/_io/clinic/textio.c.h b/Modules/_io/clinic/textio.c.h index 0b047ac0aab4e..7e81eb370fab4 100644 --- a/Modules/_io/clinic/textio.c.h +++ b/Modules/_io/clinic/textio.c.h @@ -146,7 +146,7 @@ PyDoc_STRVAR(_io_TextIOWrapper___init____doc__, "Character and line based layer over a BufferedIOBase object, buffer.\n" "\n" "encoding gives the name of the encoding that the stream will be\n" -"decoded or encoded with. It defaults to locale.getpreferredencoding(False).\n" +"decoded or encoded with. It defaults to locale.getencoding().\n" "\n" "errors determines the strictness of encoding and decoding (see\n" "help(codecs.Codec) or the documentation for codecs.register) and\n" @@ -671,4 +671,4 @@ _io_TextIOWrapper_close(textio *self, PyObject *Py_UNUSED(ignored)) { return _io_TextIOWrapper_close_impl(self); } -/*[clinic end generated code: output=2604c8f3a45b9a03 input=a9049054013a1b77]*/ +/*[clinic end generated code: output=e88abad34e31c0cb input=a9049054013a1b77]*/ diff --git a/Modules/_io/textio.c b/Modules/_io/textio.c index 0e207413257f4..6ba7393c3a6a3 100644 --- a/Modules/_io/textio.c +++ b/Modules/_io/textio.c @@ -1023,7 +1023,7 @@ _io.TextIOWrapper.__init__ Character and line based layer over a BufferedIOBase object, buffer. encoding gives the name of the encoding that the stream will be -decoded or encoded with. It defaults to locale.getpreferredencoding(False). +decoded or encoded with. It defaults to locale.getencoding(). errors determines the strictness of encoding and decoding (see help(codecs.Codec) or the documentation for codecs.register) and @@ -1055,12 +1055,12 @@ _io_TextIOWrapper___init___impl(textio *self, PyObject *buffer, const char *encoding, PyObject *errors, const char *newline, int line_buffering, int write_through) -/*[clinic end generated code: output=72267c0c01032ed2 input=77d8696d1a1f460b]*/ +/*[clinic end generated code: output=72267c0c01032ed2 input=72590963698f289b]*/ { PyObject *raw, *codec_info = NULL; - _PyIO_State *state = NULL; PyObject *res; int r; + int use_locale_encoding = 0; // Use locale encoding even in UTF-8 mode. self->ok = 0; self->detached = 0; @@ -1076,6 +1076,7 @@ _io_TextIOWrapper___init___impl(textio *self, PyObject *buffer, } else if (strcmp(encoding, "locale") == 0) { encoding = NULL; + use_locale_encoding = 1; } if (errors == Py_None) { @@ -1113,10 +1114,15 @@ _io_TextIOWrapper___init___impl(textio *self, PyObject *buffer, self->encodefunc = NULL; self->b2cratio = 0.0; +#ifdef MS_WINDOWS + // os.device_encoding() on Unix is the locale encoding or UTF-8 + // according to UTF-8 Mode. + // Since UTF-8 mode shouldn't affect `encoding="locale"`, we call + // os.device_encoding() only on Windows. if (encoding == NULL) { /* Try os.device_encoding(fileno) */ PyObject *fileno; - state = IO_STATE(); + _PyIO_State *state = IO_STATE(); if (state == NULL) goto error; fileno = PyObject_CallMethodNoArgs(buffer, &_Py_ID(fileno)); @@ -1144,8 +1150,10 @@ _io_TextIOWrapper___init___impl(textio *self, PyObject *buffer, Py_CLEAR(self->encoding); } } +#endif + if (encoding == NULL && self->encoding == NULL) { - if (_PyRuntime.preconfig.utf8_mode) { + if (_PyRuntime.preconfig.utf8_mode && !use_locale_encoding) { _Py_DECLARE_STR(utf_8, "utf-8"); self->encoding = Py_NewRef(&_Py_STR(utf_8)); } diff --git a/Tools/c-analyzer/TODO b/Tools/c-analyzer/TODO index 55338ebc855d0..6683df5993b8c 100644 --- a/Tools/c-analyzer/TODO +++ b/Tools/c-analyzer/TODO @@ -251,7 +251,6 @@ Modules/_io/textio.c:PyId_close _Py_IDENTIFIER( Modules/_io/textio.c:PyId_decode _Py_IDENTIFIER(decode) Modules/_io/textio.c:PyId_fileno _Py_IDENTIFIER(fileno) Modules/_io/textio.c:PyId_flush _Py_IDENTIFIER(flush) -Modules/_io/textio.c:PyId_getpreferredencoding _Py_IDENTIFIER(getpreferredencoding) Modules/_io/textio.c:PyId_isatty _Py_IDENTIFIER(isatty) Modules/_io/textio.c:PyId_mode _Py_IDENTIFIER(mode) Modules/_io/textio.c:PyId_name _Py_IDENTIFIER(name) From webhook-mailer at python.org Thu Apr 14 04:24:06 2022 From: webhook-mailer at python.org (corona10) Date: Thu, 14 Apr 2022 08:24:06 -0000 Subject: [Python-checkins] gh-90879: Fix missing parameter for put_nowait() (GH-91514) Message-ID: https://github.com/python/cpython/commit/0fc3517cf46ec79b4681c31916d4081055a7ed09 commit: 0fc3517cf46ec79b4681c31916d4081055a7ed09 branch: main author: slateny <46876382+slateny at users.noreply.github.com> committer: corona10 date: 2022-04-14T17:23:57+09:00 summary: gh-90879: Fix missing parameter for put_nowait() (GH-91514) files: M Doc/library/queue.rst M Lib/queue.py diff --git a/Doc/library/queue.rst b/Doc/library/queue.rst index cbf27d2bb10d0..540dde9e154e8 100644 --- a/Doc/library/queue.rst +++ b/Doc/library/queue.rst @@ -138,7 +138,7 @@ provide the public methods described below. .. method:: Queue.put_nowait(item) - Equivalent to ``put(item, False)``. + Equivalent to ``put(item, block=False)``. .. method:: Queue.get(block=True, timeout=None) @@ -248,7 +248,7 @@ SimpleQueue Objects .. method:: SimpleQueue.put_nowait(item) - Equivalent to ``put(item)``, provided for compatibility with + Equivalent to ``put(item, block=False)``, provided for compatibility with :meth:`Queue.put_nowait`. diff --git a/Lib/queue.py b/Lib/queue.py index 10dbcbc18ece8..55f50088460f9 100644 --- a/Lib/queue.py +++ b/Lib/queue.py @@ -298,7 +298,7 @@ def get(self, block=True, timeout=None): def put_nowait(self, item): '''Put an item into the queue without blocking. - This is exactly equivalent to `put(item)` and is only provided + This is exactly equivalent to `put(item, block=False)` and is only provided for compatibility with the Queue class. ''' return self.put(item, block=False) From webhook-mailer at python.org Thu Apr 14 05:18:49 2022 From: webhook-mailer at python.org (miss-islington) Date: Thu, 14 Apr 2022 09:18:49 -0000 Subject: [Python-checkins] gh-90879: Fix missing parameter for put_nowait() (GH-91514) Message-ID: https://github.com/python/cpython/commit/52ce75fc1ee31032ce64bee4f01f1c1e70b39c28 commit: 52ce75fc1ee31032ce64bee4f01f1c1e70b39c28 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-14T02:18:31-07:00 summary: gh-90879: Fix missing parameter for put_nowait() (GH-91514) (cherry picked from commit 0fc3517cf46ec79b4681c31916d4081055a7ed09) Co-authored-by: slateny <46876382+slateny at users.noreply.github.com> files: M Doc/library/queue.rst M Lib/queue.py diff --git a/Doc/library/queue.rst b/Doc/library/queue.rst index cbf27d2bb10d0..540dde9e154e8 100644 --- a/Doc/library/queue.rst +++ b/Doc/library/queue.rst @@ -138,7 +138,7 @@ provide the public methods described below. .. method:: Queue.put_nowait(item) - Equivalent to ``put(item, False)``. + Equivalent to ``put(item, block=False)``. .. method:: Queue.get(block=True, timeout=None) @@ -248,7 +248,7 @@ SimpleQueue Objects .. method:: SimpleQueue.put_nowait(item) - Equivalent to ``put(item)``, provided for compatibility with + Equivalent to ``put(item, block=False)``, provided for compatibility with :meth:`Queue.put_nowait`. diff --git a/Lib/queue.py b/Lib/queue.py index 10dbcbc18ece8..55f50088460f9 100644 --- a/Lib/queue.py +++ b/Lib/queue.py @@ -298,7 +298,7 @@ def get(self, block=True, timeout=None): def put_nowait(self, item): '''Put an item into the queue without blocking. - This is exactly equivalent to `put(item)` and is only provided + This is exactly equivalent to `put(item, block=False)` and is only provided for compatibility with the Queue class. ''' return self.put(item, block=False) From webhook-mailer at python.org Thu Apr 14 05:23:26 2022 From: webhook-mailer at python.org (miss-islington) Date: Thu, 14 Apr 2022 09:23:26 -0000 Subject: [Python-checkins] gh-90879: Fix missing parameter for put_nowait() (GH-91514) Message-ID: https://github.com/python/cpython/commit/f3f5d4be7a9a8d058524c4b8e85371862245ae31 commit: f3f5d4be7a9a8d058524c4b8e85371862245ae31 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-14T02:23:15-07:00 summary: gh-90879: Fix missing parameter for put_nowait() (GH-91514) (cherry picked from commit 0fc3517cf46ec79b4681c31916d4081055a7ed09) Co-authored-by: slateny <46876382+slateny at users.noreply.github.com> files: M Doc/library/queue.rst M Lib/queue.py diff --git a/Doc/library/queue.rst b/Doc/library/queue.rst index 0ec5900bef5bb..def39eae8adc5 100644 --- a/Doc/library/queue.rst +++ b/Doc/library/queue.rst @@ -138,7 +138,7 @@ provide the public methods described below. .. method:: Queue.put_nowait(item) - Equivalent to ``put(item, False)``. + Equivalent to ``put(item, block=False)``. .. method:: Queue.get(block=True, timeout=None) @@ -249,7 +249,7 @@ SimpleQueue Objects .. method:: SimpleQueue.put_nowait(item) - Equivalent to ``put(item)``, provided for compatibility with + Equivalent to ``put(item, block=False)``, provided for compatibility with :meth:`Queue.put_nowait`. diff --git a/Lib/queue.py b/Lib/queue.py index 10dbcbc18ece8..55f50088460f9 100644 --- a/Lib/queue.py +++ b/Lib/queue.py @@ -298,7 +298,7 @@ def get(self, block=True, timeout=None): def put_nowait(self, item): '''Put an item into the queue without blocking. - This is exactly equivalent to `put(item)` and is only provided + This is exactly equivalent to `put(item, block=False)` and is only provided for compatibility with the Queue class. ''' return self.put(item, block=False) From webhook-mailer at python.org Thu Apr 14 10:06:20 2022 From: webhook-mailer at python.org (ambv) Date: Thu, 14 Apr 2022 14:06:20 -0000 Subject: [Python-checkins] Add redirects to Misc/NEWS bpo links (#91454) Message-ID: https://github.com/python/cpython/commit/17dbb6bc10ca8a8b602335414c047294f00afcbe commit: 17dbb6bc10ca8a8b602335414c047294f00afcbe branch: main author: Ezio Melotti committer: ambv date: 2022-04-14T16:06:01+02:00 summary: Add redirects to Misc/NEWS bpo links (#91454) files: M Doc/tools/extensions/pyspecific.py diff --git a/Doc/tools/extensions/pyspecific.py b/Doc/tools/extensions/pyspecific.py index 9fb54bc6041ab..80533a97b1189 100644 --- a/Doc/tools/extensions/pyspecific.py +++ b/Doc/tools/extensions/pyspecific.py @@ -433,7 +433,8 @@ def run(self): text = 'The NEWS file is not available.' node = nodes.strong(text, text) return [node] - content = issue_re.sub(r'`bpo-\1 `__', + content = issue_re.sub(r'`bpo-\1 `__', content) content = whatsnew_re.sub(r'\1', content) # remove first 3 lines as they are the main heading From webhook-mailer at python.org Thu Apr 14 10:27:11 2022 From: webhook-mailer at python.org (tiran) Date: Thu, 14 Apr 2022 14:27:11 -0000 Subject: [Python-checkins] gh-91353: Fix void return type handling in ctypes (GH-32246) Message-ID: https://github.com/python/cpython/commit/1b035d9699aaebbe4f1ca096345d538caa07907a commit: 1b035d9699aaebbe4f1ca096345d538caa07907a branch: main author: Hood Chatham committer: tiran date: 2022-04-14T16:27:01+02:00 summary: gh-91353: Fix void return type handling in ctypes (GH-32246) files: A Misc/NEWS.d/next/Core and Builtins/2022-04-03-17-21-04.bpo-47197.Ji_c30.rst M Modules/_ctypes/callbacks.c M Modules/_ctypes/callproc.c diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-04-03-17-21-04.bpo-47197.Ji_c30.rst b/Misc/NEWS.d/next/Core and Builtins/2022-04-03-17-21-04.bpo-47197.Ji_c30.rst new file mode 100644 index 0000000000000..697b07dc95ecc --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2022-04-03-17-21-04.bpo-47197.Ji_c30.rst @@ -0,0 +1,5 @@ +ctypes used to mishandle ``void`` return types, so that for instance a +function declared like ``ctypes.CFUNCTYPE(None, ctypes.c_int)`` would be +called with signature ``int f(int)`` instead of ``void f(int)``. Wasm +targets require function pointers to be called with the correct signatures +so this led to crashes. The problem is now fixed. diff --git a/Modules/_ctypes/callbacks.c b/Modules/_ctypes/callbacks.c index bb59456a7321a..7dd1f99021a7f 100644 --- a/Modules/_ctypes/callbacks.c +++ b/Modules/_ctypes/callbacks.c @@ -399,7 +399,7 @@ CThunkObject *_ctypes_alloc_callback(PyObject *callable, #endif result = ffi_prep_cif(&p->cif, cc, Py_SAFE_DOWNCAST(nargs, Py_ssize_t, int), - _ctypes_get_ffi_type(restype), + p->ffi_restype, &p->atypes[0]); if (result != FFI_OK) { PyErr_Format(PyExc_RuntimeError, diff --git a/Modules/_ctypes/callproc.c b/Modules/_ctypes/callproc.c index 4a6b8ec3ee016..720c4c09f538f 100644 --- a/Modules/_ctypes/callproc.c +++ b/Modules/_ctypes/callproc.c @@ -1209,7 +1209,12 @@ PyObject *_ctypes_callproc(PPROC pProc, } } - rtype = _ctypes_get_ffi_type(restype); + if (restype == Py_None) { + rtype = &ffi_type_void; + } else { + rtype = _ctypes_get_ffi_type(restype); + } + resbuf = alloca(max(rtype->size, sizeof(ffi_arg))); #ifdef _Py_MEMORY_SANITIZER From webhook-mailer at python.org Thu Apr 14 10:27:51 2022 From: webhook-mailer at python.org (tiran) Date: Thu, 14 Apr 2022 14:27:51 -0000 Subject: [Python-checkins] gh-84461: Drop -sWASM, fix building tests for browser (GH-91530) Message-ID: https://github.com/python/cpython/commit/7acedd71dec4e60400c865911e8961dbb49d5597 commit: 7acedd71dec4e60400c865911e8961dbb49d5597 branch: main author: Christian Heimes committer: tiran date: 2022-04-14T16:27:41+02:00 summary: gh-84461: Drop -sWASM, fix building tests for browser (GH-91530) - drop unnecessary ``=1`` suffix from Emscripten flags - drop unnecessary ``-sWASM`` flag for side modules - rename ``build_platform`` to ``build_wasm``. I introduced the target for WASM builds a couple of months ago. - fix ``--enable-test-modules`` for browser builds files: M Makefile.pre.in M configure M configure.ac diff --git a/Makefile.pre.in b/Makefile.pre.in index f803391346bd6..d89886ddc76f1 100644 --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -580,7 +580,7 @@ LIBEXPAT_HEADERS= \ all: @DEF_MAKE_ALL_RULE@ build_all: check-clean-src $(BUILDPYTHON) oldsharedmods sharedmods gdbhooks \ Programs/_testembed python-config -build_platform: check-clean-src $(BUILDPYTHON) platform +build_wasm: check-clean-src $(BUILDPYTHON) platform oldsharedmods python-config # Check that the source is clean when building out of source. check-clean-src: @@ -2475,7 +2475,7 @@ update-config: Python/thread.o: @THREADHEADERS@ $(srcdir)/Python/condvar.h # Declare targets that aren't real files -.PHONY: all build_all sharedmods check-clean-src oldsharedmods test quicktest +.PHONY: all build_all build_wasm sharedmods check-clean-src oldsharedmods test quicktest .PHONY: install altinstall oldsharedinstall bininstall altbininstall .PHONY: maninstall libinstall inclinstall libainstall sharedinstall .PHONY: frameworkinstall frameworkinstallframework frameworkinstallstructure diff --git a/configure b/configure index 96530325f8cb9..26a891ee7fa03 100755 --- a/configure +++ b/configure @@ -7226,8 +7226,8 @@ fi ;; esac -elif test "$ac_sys_system" = "Emscripten"; then - DEF_MAKE_ALL_RULE="build_platform" +elif test "$ac_sys_system" = "Emscripten" -o "$ac_sys_system" = "WASI"; then + DEF_MAKE_ALL_RULE="build_wasm" REQUIRE_PGO="no" DEF_MAKE_RULE="all" else @@ -7791,17 +7791,17 @@ fi case $ac_sys_system/$ac_sys_emscripten_target in #( Emscripten/browser*) : - LDFLAGS_NODIST="$LDFLAGS_NODIST -sALLOW_MEMORY_GROWTH=1" + LDFLAGS_NODIST="$LDFLAGS_NODIST -sALLOW_MEMORY_GROWTH" LINKFORSHARED="--preload-file=\$(WASM_ASSETS_DIR)" if test "x$enable_wasm_dynamic_linking" = xyes; then : - as_fn_append LINKFORSHARED " -sMAIN_MODULE=1" + as_fn_append LINKFORSHARED " -sMAIN_MODULE" fi WASM_ASSETS_DIR=".\$(prefix)" WASM_STDLIB="\$(WASM_ASSETS_DIR)/local/lib/python\$(VERSION)/os.py" if test "$Py_DEBUG" = 'true' -o "$ac_sys_emscripten_target" = "browser-debug"; then - LDFLAGS_NODIST="$LDFLAGS_NODIST -sASSERTIONS=1" + LDFLAGS_NODIST="$LDFLAGS_NODIST -sASSERTIONS" LINKFORSHARED="$LINKFORSHARED -gsource-map --emit-symbol-map" else LINKFORSHARED="$LINKFORSHARED -O2 -g0" @@ -7809,16 +7809,16 @@ fi ;; #( Emscripten/node*) : - LDFLAGS_NODIST="$LDFLAGS_NODIST -sALLOW_MEMORY_GROWTH=1 -sNODERAWFS=1 -sUSE_PTHREADS=1" - LINKFORSHARED="-sPROXY_TO_PTHREAD=1 -sEXIT_RUNTIME=1" + LDFLAGS_NODIST="$LDFLAGS_NODIST -sALLOW_MEMORY_GROWTH -sNODERAWFS -sUSE_PTHREADS" + LINKFORSHARED="-sPROXY_TO_PTHREAD -sEXIT_RUNTIME" if test "x$enable_wasm_dynamic_linking" = xyes; then : - as_fn_append LINKFORSHARED " -sMAIN_MODULE=1" + as_fn_append LINKFORSHARED " -sMAIN_MODULE" fi CFLAGS_NODIST="$CFLAGS_NODIST -pthread" if test "$Py_DEBUG" = 'true' -o "$ac_sys_emscripten_target" = "node-debug"; then - LDFLAGS_NODIST="$LDFLAGS_NODIST -sASSERTIONS=1" + LDFLAGS_NODIST="$LDFLAGS_NODIST -sASSERTIONS" LINKFORSHARED="$LINKFORSHARED -gseparate-dwarf --emit-symbol-map" else LINKFORSHARED="$LINKFORSHARED -O2 -g0" @@ -10567,7 +10567,7 @@ fi;; fi if test "$enable_wasm_dynamic_linking" = "yes" -a "$ac_sys_system" = "Emscripten"; then - BLDSHARED='$(CC) -shared -sSIDE_MODULE=1 -sWASM=1' + BLDSHARED='$(CC) -shared -sSIDE_MODULE=1' fi { $as_echo "$as_me:${as_lineno-$LINENO}: result: $LDSHARED" >&5 @@ -16030,8 +16030,8 @@ $as_echo "yes" >&6; } fi if test "$have_zlib" = "yes" -a "$ac_sys_system" = "Emscripten" -a "$ZLIB_LIBS" = "-lz"; then - ZLIB_CFLAGS="-sUSE_ZLIB=1" - ZLIB_LIBS="-sUSE_ZLIB=1" + ZLIB_CFLAGS="-sUSE_ZLIB" + ZLIB_LIBS="-sUSE_ZLIB" fi if test "x$have_zlib" = xyes; then : @@ -16264,8 +16264,8 @@ $as_echo "yes" >&6; } fi if test "$have_bzip2" = "yes" -a "$ac_sys_system" = "Emscripten" -a "$BZIP2_LIBS" = "-lbz2"; then - BZIP2_CFLAGS="-sUSE_BZIP2=1" - BZIP2_LIBS="-sUSE_BZIP2=1" + BZIP2_CFLAGS="-sUSE_BZIP2" + BZIP2_LIBS="-sUSE_BZIP2" fi @@ -22341,28 +22341,27 @@ $as_echo_n "checking for --disable-test-modules... " >&6; } # Check whether --enable-test-modules was given. if test "${enable_test_modules+set}" = set; then : enableval=$enable_test_modules; + if test "x$enable_test_modules" = xyes; then : + TEST_MODULES=yes +else + TEST_MODULES=no fi -if test "$enable_test_modules" = no; then - TEST_MODULES=no else - case $ac_sys_system/$ac_sys_emscripten_target in #( + + case $ac_sys_system/$ac_sys_emscripten_target in #( Emscripten/browser*) : TEST_MODULES=no ;; #( *) : TEST_MODULES=yes - ;; + ;; esac -fi -if test "x$TEST_MODULES" = xyes; then : - { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 -$as_echo "no" >&6; } -else - { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 -$as_echo "yes" >&6; } fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $TEST_MODULES" >&5 +$as_echo "$TEST_MODULES" >&6; } + diff --git a/configure.ac b/configure.ac index 6a460cd94c243..5c915d1a45cdc 100644 --- a/configure.ac +++ b/configure.ac @@ -1574,10 +1574,10 @@ if test "$Py_OPT" = 'true' ; then ]) ;; esac -elif test "$ac_sys_system" = "Emscripten"; then +elif test "$ac_sys_system" = "Emscripten" -o "$ac_sys_system" = "WASI"; then dnl Emscripten does not support shared extensions yet. Build - dnl "python.[js,html,wasm]", "pybuilddir.txt", and "platform" files. - DEF_MAKE_ALL_RULE="build_platform" + dnl "python.[js,wasm]", "pybuilddir.txt", and "platform" files. + DEF_MAKE_ALL_RULE="build_wasm" REQUIRE_PGO="no" DEF_MAKE_RULE="all" else @@ -1911,30 +1911,30 @@ fi # WASM flags AS_CASE([$ac_sys_system/$ac_sys_emscripten_target], [Emscripten/browser*], [ - LDFLAGS_NODIST="$LDFLAGS_NODIST -sALLOW_MEMORY_GROWTH=1" + LDFLAGS_NODIST="$LDFLAGS_NODIST -sALLOW_MEMORY_GROWTH" LINKFORSHARED="--preload-file=\$(WASM_ASSETS_DIR)" AS_VAR_IF([enable_wasm_dynamic_linking], [yes], [ - AS_VAR_APPEND([LINKFORSHARED], [" -sMAIN_MODULE=1"]) + AS_VAR_APPEND([LINKFORSHARED], [" -sMAIN_MODULE"]) ]) WASM_ASSETS_DIR=".\$(prefix)" WASM_STDLIB="\$(WASM_ASSETS_DIR)/local/lib/python\$(VERSION)/os.py" dnl separate-dwarf does not seem to work in Chrome DevTools Support. if test "$Py_DEBUG" = 'true' -o "$ac_sys_emscripten_target" = "browser-debug"; then - LDFLAGS_NODIST="$LDFLAGS_NODIST -sASSERTIONS=1" + LDFLAGS_NODIST="$LDFLAGS_NODIST -sASSERTIONS" LINKFORSHARED="$LINKFORSHARED -gsource-map --emit-symbol-map" else LINKFORSHARED="$LINKFORSHARED -O2 -g0" fi ], [Emscripten/node*], [ - LDFLAGS_NODIST="$LDFLAGS_NODIST -sALLOW_MEMORY_GROWTH=1 -sNODERAWFS=1 -sUSE_PTHREADS=1" - LINKFORSHARED="-sPROXY_TO_PTHREAD=1 -sEXIT_RUNTIME=1" + LDFLAGS_NODIST="$LDFLAGS_NODIST -sALLOW_MEMORY_GROWTH -sNODERAWFS -sUSE_PTHREADS" + LINKFORSHARED="-sPROXY_TO_PTHREAD -sEXIT_RUNTIME" AS_VAR_IF([enable_wasm_dynamic_linking], [yes], [ - AS_VAR_APPEND([LINKFORSHARED], [" -sMAIN_MODULE=1"]) + AS_VAR_APPEND([LINKFORSHARED], [" -sMAIN_MODULE"]) ]) CFLAGS_NODIST="$CFLAGS_NODIST -pthread" if test "$Py_DEBUG" = 'true' -o "$ac_sys_emscripten_target" = "node-debug"; then - LDFLAGS_NODIST="$LDFLAGS_NODIST -sASSERTIONS=1" + LDFLAGS_NODIST="$LDFLAGS_NODIST -sASSERTIONS" LINKFORSHARED="$LINKFORSHARED -gseparate-dwarf --emit-symbol-map" else LINKFORSHARED="$LINKFORSHARED -O2 -g0" @@ -3009,7 +3009,7 @@ fi dnl Emscripten's emconfigure sets LDSHARED. Set BLDSHARED outside the dnl test -z $LDSHARED block to configure BLDSHARED for side module support. if test "$enable_wasm_dynamic_linking" = "yes" -a "$ac_sys_system" = "Emscripten"; then - BLDSHARED='$(CC) -shared -sSIDE_MODULE=1 -sWASM=1' + BLDSHARED='$(CC) -shared -sSIDE_MODULE=1' fi AC_MSG_RESULT($LDSHARED) @@ -4549,8 +4549,8 @@ PKG_CHECK_MODULES([ZLIB], [zlib >= 1.2.0], [ ]) if test "$have_zlib" = "yes" -a "$ac_sys_system" = "Emscripten" -a "$ZLIB_LIBS" = "-lz"; then - ZLIB_CFLAGS="-sUSE_ZLIB=1" - ZLIB_LIBS="-sUSE_ZLIB=1" + ZLIB_CFLAGS="-sUSE_ZLIB" + ZLIB_LIBS="-sUSE_ZLIB" fi dnl binascii can use zlib for optimized crc32. @@ -4572,8 +4572,8 @@ PKG_CHECK_MODULES([BZIP2], [bzip2], [have_bzip2=yes], [ ]) if test "$have_bzip2" = "yes" -a "$ac_sys_system" = "Emscripten" -a "$BZIP2_LIBS" = "-lbz2"; then - BZIP2_CFLAGS="-sUSE_BZIP2=1" - BZIP2_LIBS="-sUSE_BZIP2=1" + BZIP2_CFLAGS="-sUSE_BZIP2" + BZIP2_LIBS="-sUSE_BZIP2" fi @@ -6557,21 +6557,18 @@ fi], # Check whether to disable test modules. Once set, setup.py will not build # test extension modules and "make install" will not install test suites. -AC_MSG_CHECKING(for --disable-test-modules) -AC_ARG_ENABLE(test-modules, - AS_HELP_STRING([--disable-test-modules], [don't build nor install test modules])) -if test "$enable_test_modules" = no; then - TEST_MODULES=no -else - AS_CASE([$ac_sys_system/$ac_sys_emscripten_target], - [Emscripten/browser*], [TEST_MODULES=no], - [TEST_MODULES=yes] - ) -fi -AS_VAR_IF([TEST_MODULES], [yes], - [AC_MSG_RESULT(no)], [AC_MSG_RESULT(yes)] -) -AC_SUBST(TEST_MODULES) +AC_MSG_CHECKING([for --disable-test-modules]) +AC_ARG_ENABLE([test-modules], + [AS_HELP_STRING([--disable-test-modules], [don't build nor install test modules])], [ + AS_VAR_IF([enable_test_modules], [yes], [TEST_MODULES=yes], [TEST_MODULES=no]) +], [ + AS_CASE([$ac_sys_system/$ac_sys_emscripten_target], + [Emscripten/browser*], [TEST_MODULES=no], + [TEST_MODULES=yes] + ) +]) +AC_MSG_RESULT([$TEST_MODULES]) +AC_SUBST([TEST_MODULES]) AC_DEFUN([PY_STDLIB_MOD_SET_NA], [ m4_foreach([mod], [$@], [ From webhook-mailer at python.org Thu Apr 14 11:01:20 2022 From: webhook-mailer at python.org (ambv) Date: Thu, 14 Apr 2022 15:01:20 -0000 Subject: [Python-checkins] Add redirects to Misc/NEWS bpo links (GH-91454) (GH-91536) Message-ID: https://github.com/python/cpython/commit/35ce2b771bb367bdfe2855afda52f8dc7e211a64 commit: 35ce2b771bb367bdfe2855afda52f8dc7e211a64 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2022-04-14T17:01:15+02:00 summary: Add redirects to Misc/NEWS bpo links (GH-91454) (GH-91536) (cherry picked from commit 17dbb6bc10ca8a8b602335414c047294f00afcbe) Co-authored-by: Ezio Melotti files: M Doc/tools/extensions/pyspecific.py diff --git a/Doc/tools/extensions/pyspecific.py b/Doc/tools/extensions/pyspecific.py index 74de6ffa75559..7c448310f47f6 100644 --- a/Doc/tools/extensions/pyspecific.py +++ b/Doc/tools/extensions/pyspecific.py @@ -395,7 +395,8 @@ def run(self): text = 'The NEWS file is not available.' node = nodes.strong(text, text) return [node] - content = issue_re.sub(r'`bpo-\1 `__', + content = issue_re.sub(r'`bpo-\1 `__', content) content = whatsnew_re.sub(r'\1', content) # remove first 3 lines as they are the main heading From webhook-mailer at python.org Thu Apr 14 11:02:08 2022 From: webhook-mailer at python.org (ambv) Date: Thu, 14 Apr 2022 15:02:08 -0000 Subject: [Python-checkins] Add redirects to Misc/NEWS bpo links (GH-91454) (GH-91535) Message-ID: https://github.com/python/cpython/commit/35fef2711033ce793ec1cb43dfbd95e2d06ab7bb commit: 35fef2711033ce793ec1cb43dfbd95e2d06ab7bb branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2022-04-14T17:02:03+02:00 summary: Add redirects to Misc/NEWS bpo links (GH-91454) (GH-91535) (cherry picked from commit 17dbb6bc10ca8a8b602335414c047294f00afcbe) Co-authored-by: Ezio Melotti files: M Doc/tools/extensions/pyspecific.py diff --git a/Doc/tools/extensions/pyspecific.py b/Doc/tools/extensions/pyspecific.py index 92fc5e7c71cd7..7f505a1ede89a 100644 --- a/Doc/tools/extensions/pyspecific.py +++ b/Doc/tools/extensions/pyspecific.py @@ -433,7 +433,8 @@ def run(self): text = 'The NEWS file is not available.' node = nodes.strong(text, text) return [node] - content = issue_re.sub(r'`bpo-\1 `__', + content = issue_re.sub(r'`bpo-\1 `__', content) content = whatsnew_re.sub(r'\1', content) # remove first 3 lines as they are the main heading From webhook-mailer at python.org Thu Apr 14 14:26:37 2022 From: webhook-mailer at python.org (gpshead) Date: Thu, 14 Apr 2022 18:26:37 -0000 Subject: [Python-checkins] gh-89455: Fix an uninitialized bool in exception print context. (#91466) Message-ID: https://github.com/python/cpython/commit/861974b514bc15ca1a3021f32a2d381c22f1a971 commit: 861974b514bc15ca1a3021f32a2d381c22f1a971 branch: main author: Gregory P. Smith committer: gpshead date: 2022-04-14T11:26:25-07:00 summary: gh-89455: Fix an uninitialized bool in exception print context. (#91466) Fix an uninitialized bool in exception print context. `struct exception_print_context.need_close` was uninitialized. Found by oss-fuzz in a test case running under the undefined behavior sanitizer. https://oss-fuzz.com/testcase-detail/6217746058182656 ``` Python/pythonrun.c:1241:28: runtime error: load of value 253, which is not a valid value for type 'bool' #0 0xbf2203 in print_chained cpython3/Python/pythonrun.c:1241:28 #1 0xbea4bb in print_exception_cause_and_context cpython3/Python/pythonrun.c:1320:19 #2 0xbea4bb in print_exception_recursive cpython3/Python/pythonrun.c:1470:13 #3 0xbe9e39 in _PyErr_Display cpython3/Python/pythonrun.c:1517:9 ``` Pretty obvious what the ommission was upon code inspection. files: A Misc/NEWS.d/next/Core and Builtins/2022-04-11-18-44-19.gh-issue-89455.d0qMYd.rst M Python/pythonrun.c diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-04-11-18-44-19.gh-issue-89455.d0qMYd.rst b/Misc/NEWS.d/next/Core and Builtins/2022-04-11-18-44-19.gh-issue-89455.d0qMYd.rst new file mode 100644 index 0000000000000..e22b4ac44c096 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2022-04-11-18-44-19.gh-issue-89455.d0qMYd.rst @@ -0,0 +1,2 @@ +Fixed an uninitialized bool value in the traceback printing code path that +was introduced by the initial bpo-45292 exception groups work. diff --git a/Python/pythonrun.c b/Python/pythonrun.c index d117b790dfdaa..e086f0f345c22 100644 --- a/Python/pythonrun.c +++ b/Python/pythonrun.c @@ -1504,6 +1504,7 @@ _PyErr_Display(PyObject *file, PyObject *exception, PyObject *value, PyObject *t struct exception_print_context ctx; ctx.file = file; ctx.exception_group_depth = 0; + ctx.need_close = false; ctx.max_group_width = PyErr_MAX_GROUP_WIDTH; ctx.max_group_depth = PyErr_MAX_GROUP_DEPTH; From webhook-mailer at python.org Thu Apr 14 15:50:20 2022 From: webhook-mailer at python.org (brettcannon) Date: Thu, 14 Apr 2022 19:50:20 -0000 Subject: [Python-checkins] gh-91217: deprecate msilib (GH-91515) Message-ID: https://github.com/python/cpython/commit/1b6cd872f440ee99989debc1bdecd8bea5a368bf commit: 1b6cd872f440ee99989debc1bdecd8bea5a368bf branch: main author: Brett Cannon committer: brettcannon date: 2022-04-14T12:50:11-07:00 summary: gh-91217: deprecate msilib (GH-91515) files: A Misc/NEWS.d/next/Library/2022-04-12-19-42-20.gh-issue-91217.b9_Rz9.rst M Doc/whatsnew/3.11.rst M Lib/msilib/__init__.py M Lib/test/test_msilib.py diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index 894ec8a9d0d92..f8e86f6ba349b 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -857,6 +857,7 @@ Deprecated * :mod:`chunk` * :mod:`crypt` * :mod:`imghdr` + * :mod:`msilib` (Contributed by Brett Cannon in :issue:`47061`.) diff --git a/Lib/msilib/__init__.py b/Lib/msilib/__init__.py index 0e85aa28967ea..565bf631abd4e 100644 --- a/Lib/msilib/__init__.py +++ b/Lib/msilib/__init__.py @@ -6,6 +6,9 @@ import re import string import sys +import warnings + +warnings._deprecated(__name__, remove=(3, 13)) AMD64 = "AMD64" in sys.version # Keep msilib.Win64 around to preserve backwards compatibility. diff --git a/Lib/test/test_msilib.py b/Lib/test/test_msilib.py index e29cd4a84c546..db16f62a4c121 100644 --- a/Lib/test/test_msilib.py +++ b/Lib/test/test_msilib.py @@ -3,7 +3,10 @@ import unittest from test.support.import_helper import import_module from test.support.os_helper import TESTFN, unlink -msilib = import_module('msilib') +import warnings +with warnings.catch_warnings(): + warnings.simplefilter("ignore", DeprecationWarning) + msilib = import_module('msilib') import msilib.schema diff --git a/Misc/NEWS.d/next/Library/2022-04-12-19-42-20.gh-issue-91217.b9_Rz9.rst b/Misc/NEWS.d/next/Library/2022-04-12-19-42-20.gh-issue-91217.b9_Rz9.rst new file mode 100644 index 0000000000000..ee1fc22deaef0 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-12-19-42-20.gh-issue-91217.b9_Rz9.rst @@ -0,0 +1 @@ +Deprecate msilib. From webhook-mailer at python.org Thu Apr 14 16:01:03 2022 From: webhook-mailer at python.org (sweeneyde) Date: Thu, 14 Apr 2022 20:01:03 -0000 Subject: [Python-checkins] gh-91428: include specialized opcodes in _PyOpcode_OpName (GH-91467) Message-ID: https://github.com/python/cpython/commit/c9d41bcd68dbe339396523e931904930a87819b9 commit: c9d41bcd68dbe339396523e931904930a87819b9 branch: main author: Dennis Sweeney <36520290+sweeneyde at users.noreply.github.com> committer: sweeneyde <36520290+sweeneyde at users.noreply.github.com> date: 2022-04-14T16:00:58-04:00 summary: gh-91428: include specialized opcodes in _PyOpcode_OpName (GH-91467) files: M Include/opcode.h M Tools/scripts/generate_opcode_h.py diff --git a/Include/opcode.h b/Include/opcode.h index ca4a18de10716..0badf78e32771 100644 --- a/Include/opcode.h +++ b/Include/opcode.h @@ -451,33 +451,85 @@ static const char *const _PyOpcode_OpName[256] = { [CACHE] = "CACHE", [POP_TOP] = "POP_TOP", [PUSH_NULL] = "PUSH_NULL", + [BINARY_OP_ADAPTIVE] = "BINARY_OP_ADAPTIVE", + [BINARY_OP_ADD_FLOAT] = "BINARY_OP_ADD_FLOAT", + [BINARY_OP_ADD_INT] = "BINARY_OP_ADD_INT", + [BINARY_OP_ADD_UNICODE] = "BINARY_OP_ADD_UNICODE", + [BINARY_OP_INPLACE_ADD_UNICODE] = "BINARY_OP_INPLACE_ADD_UNICODE", + [BINARY_OP_MULTIPLY_FLOAT] = "BINARY_OP_MULTIPLY_FLOAT", [NOP] = "NOP", [UNARY_POSITIVE] = "UNARY_POSITIVE", [UNARY_NEGATIVE] = "UNARY_NEGATIVE", [UNARY_NOT] = "UNARY_NOT", + [BINARY_OP_MULTIPLY_INT] = "BINARY_OP_MULTIPLY_INT", + [BINARY_OP_SUBTRACT_FLOAT] = "BINARY_OP_SUBTRACT_FLOAT", [UNARY_INVERT] = "UNARY_INVERT", + [BINARY_OP_SUBTRACT_INT] = "BINARY_OP_SUBTRACT_INT", + [BINARY_SUBSCR_ADAPTIVE] = "BINARY_SUBSCR_ADAPTIVE", + [BINARY_SUBSCR_DICT] = "BINARY_SUBSCR_DICT", + [BINARY_SUBSCR_GETITEM] = "BINARY_SUBSCR_GETITEM", + [BINARY_SUBSCR_LIST_INT] = "BINARY_SUBSCR_LIST_INT", + [BINARY_SUBSCR_TUPLE_INT] = "BINARY_SUBSCR_TUPLE_INT", + [CALL_ADAPTIVE] = "CALL_ADAPTIVE", + [CALL_PY_EXACT_ARGS] = "CALL_PY_EXACT_ARGS", + [CALL_PY_WITH_DEFAULTS] = "CALL_PY_WITH_DEFAULTS", [BINARY_SUBSCR] = "BINARY_SUBSCR", + [COMPARE_OP_ADAPTIVE] = "COMPARE_OP_ADAPTIVE", + [COMPARE_OP_FLOAT_JUMP] = "COMPARE_OP_FLOAT_JUMP", + [COMPARE_OP_INT_JUMP] = "COMPARE_OP_INT_JUMP", + [COMPARE_OP_STR_JUMP] = "COMPARE_OP_STR_JUMP", [GET_LEN] = "GET_LEN", [MATCH_MAPPING] = "MATCH_MAPPING", [MATCH_SEQUENCE] = "MATCH_SEQUENCE", [MATCH_KEYS] = "MATCH_KEYS", + [JUMP_BACKWARD_QUICK] = "JUMP_BACKWARD_QUICK", [PUSH_EXC_INFO] = "PUSH_EXC_INFO", [CHECK_EXC_MATCH] = "CHECK_EXC_MATCH", [CHECK_EG_MATCH] = "CHECK_EG_MATCH", + [LOAD_ATTR_ADAPTIVE] = "LOAD_ATTR_ADAPTIVE", + [LOAD_ATTR_INSTANCE_VALUE] = "LOAD_ATTR_INSTANCE_VALUE", + [LOAD_ATTR_MODULE] = "LOAD_ATTR_MODULE", + [LOAD_ATTR_SLOT] = "LOAD_ATTR_SLOT", + [LOAD_ATTR_WITH_HINT] = "LOAD_ATTR_WITH_HINT", + [LOAD_CONST__LOAD_FAST] = "LOAD_CONST__LOAD_FAST", + [LOAD_FAST__LOAD_CONST] = "LOAD_FAST__LOAD_CONST", + [LOAD_FAST__LOAD_FAST] = "LOAD_FAST__LOAD_FAST", + [LOAD_GLOBAL_ADAPTIVE] = "LOAD_GLOBAL_ADAPTIVE", + [LOAD_GLOBAL_BUILTIN] = "LOAD_GLOBAL_BUILTIN", + [LOAD_GLOBAL_MODULE] = "LOAD_GLOBAL_MODULE", [WITH_EXCEPT_START] = "WITH_EXCEPT_START", [GET_AITER] = "GET_AITER", [GET_ANEXT] = "GET_ANEXT", [BEFORE_ASYNC_WITH] = "BEFORE_ASYNC_WITH", [BEFORE_WITH] = "BEFORE_WITH", [END_ASYNC_FOR] = "END_ASYNC_FOR", + [LOAD_METHOD_ADAPTIVE] = "LOAD_METHOD_ADAPTIVE", + [LOAD_METHOD_CLASS] = "LOAD_METHOD_CLASS", + [LOAD_METHOD_MODULE] = "LOAD_METHOD_MODULE", + [LOAD_METHOD_NO_DICT] = "LOAD_METHOD_NO_DICT", + [LOAD_METHOD_WITH_DICT] = "LOAD_METHOD_WITH_DICT", [STORE_SUBSCR] = "STORE_SUBSCR", [DELETE_SUBSCR] = "DELETE_SUBSCR", + [LOAD_METHOD_WITH_VALUES] = "LOAD_METHOD_WITH_VALUES", + [PRECALL_ADAPTIVE] = "PRECALL_ADAPTIVE", + [PRECALL_BOUND_METHOD] = "PRECALL_BOUND_METHOD", + [PRECALL_BUILTIN_CLASS] = "PRECALL_BUILTIN_CLASS", + [PRECALL_BUILTIN_FAST_WITH_KEYWORDS] = "PRECALL_BUILTIN_FAST_WITH_KEYWORDS", + [PRECALL_METHOD_DESCRIPTOR_FAST_WITH_KEYWORDS] = "PRECALL_METHOD_DESCRIPTOR_FAST_WITH_KEYWORDS", [GET_ITER] = "GET_ITER", [GET_YIELD_FROM_ITER] = "GET_YIELD_FROM_ITER", [PRINT_EXPR] = "PRINT_EXPR", [LOAD_BUILD_CLASS] = "LOAD_BUILD_CLASS", + [PRECALL_NO_KW_BUILTIN_FAST] = "PRECALL_NO_KW_BUILTIN_FAST", + [PRECALL_NO_KW_BUILTIN_O] = "PRECALL_NO_KW_BUILTIN_O", [LOAD_ASSERTION_ERROR] = "LOAD_ASSERTION_ERROR", [RETURN_GENERATOR] = "RETURN_GENERATOR", + [PRECALL_NO_KW_ISINSTANCE] = "PRECALL_NO_KW_ISINSTANCE", + [PRECALL_NO_KW_LEN] = "PRECALL_NO_KW_LEN", + [PRECALL_NO_KW_LIST_APPEND] = "PRECALL_NO_KW_LIST_APPEND", + [PRECALL_NO_KW_METHOD_DESCRIPTOR_FAST] = "PRECALL_NO_KW_METHOD_DESCRIPTOR_FAST", + [PRECALL_NO_KW_METHOD_DESCRIPTOR_NOARGS] = "PRECALL_NO_KW_METHOD_DESCRIPTOR_NOARGS", + [PRECALL_NO_KW_METHOD_DESCRIPTOR_O] = "PRECALL_NO_KW_METHOD_DESCRIPTOR_O", [LIST_TO_TUPLE] = "LIST_TO_TUPLE", [RETURN_VALUE] = "RETURN_VALUE", [IMPORT_STAR] = "IMPORT_STAR", @@ -509,6 +561,7 @@ static const char *const _PyOpcode_OpName[256] = { [JUMP_FORWARD] = "JUMP_FORWARD", [JUMP_IF_FALSE_OR_POP] = "JUMP_IF_FALSE_OR_POP", [JUMP_IF_TRUE_OR_POP] = "JUMP_IF_TRUE_OR_POP", + [PRECALL_NO_KW_STR_1] = "PRECALL_NO_KW_STR_1", [POP_JUMP_FORWARD_IF_FALSE] = "POP_JUMP_FORWARD_IF_FALSE", [POP_JUMP_FORWARD_IF_TRUE] = "POP_JUMP_FORWARD_IF_TRUE", [LOAD_GLOBAL] = "LOAD_GLOBAL", @@ -516,11 +569,13 @@ static const char *const _PyOpcode_OpName[256] = { [CONTAINS_OP] = "CONTAINS_OP", [RERAISE] = "RERAISE", [COPY] = "COPY", + [PRECALL_NO_KW_TUPLE_1] = "PRECALL_NO_KW_TUPLE_1", [BINARY_OP] = "BINARY_OP", [SEND] = "SEND", [LOAD_FAST] = "LOAD_FAST", [STORE_FAST] = "STORE_FAST", [DELETE_FAST] = "DELETE_FAST", + [PRECALL_NO_KW_TYPE_1] = "PRECALL_NO_KW_TYPE_1", [POP_JUMP_FORWARD_IF_NOT_NONE] = "POP_JUMP_FORWARD_IF_NOT_NONE", [POP_JUMP_FORWARD_IF_NONE] = "POP_JUMP_FORWARD_IF_NONE", [RAISE_VARARGS] = "RAISE_VARARGS", @@ -534,30 +589,121 @@ static const char *const _PyOpcode_OpName[256] = { [STORE_DEREF] = "STORE_DEREF", [DELETE_DEREF] = "DELETE_DEREF", [JUMP_BACKWARD] = "JUMP_BACKWARD", + [PRECALL_PYFUNC] = "PRECALL_PYFUNC", [CALL_FUNCTION_EX] = "CALL_FUNCTION_EX", + [RESUME_QUICK] = "RESUME_QUICK", [EXTENDED_ARG] = "EXTENDED_ARG", [LIST_APPEND] = "LIST_APPEND", [SET_ADD] = "SET_ADD", [MAP_ADD] = "MAP_ADD", [LOAD_CLASSDEREF] = "LOAD_CLASSDEREF", [COPY_FREE_VARS] = "COPY_FREE_VARS", + [STORE_ATTR_ADAPTIVE] = "STORE_ATTR_ADAPTIVE", [RESUME] = "RESUME", [MATCH_CLASS] = "MATCH_CLASS", + [STORE_ATTR_INSTANCE_VALUE] = "STORE_ATTR_INSTANCE_VALUE", + [STORE_ATTR_SLOT] = "STORE_ATTR_SLOT", [FORMAT_VALUE] = "FORMAT_VALUE", [BUILD_CONST_KEY_MAP] = "BUILD_CONST_KEY_MAP", [BUILD_STRING] = "BUILD_STRING", + [STORE_ATTR_WITH_HINT] = "STORE_ATTR_WITH_HINT", + [STORE_FAST__LOAD_FAST] = "STORE_FAST__LOAD_FAST", [LOAD_METHOD] = "LOAD_METHOD", + [STORE_FAST__STORE_FAST] = "STORE_FAST__STORE_FAST", [LIST_EXTEND] = "LIST_EXTEND", [SET_UPDATE] = "SET_UPDATE", [DICT_MERGE] = "DICT_MERGE", [DICT_UPDATE] = "DICT_UPDATE", [PRECALL] = "PRECALL", + [STORE_SUBSCR_ADAPTIVE] = "STORE_SUBSCR_ADAPTIVE", + [STORE_SUBSCR_DICT] = "STORE_SUBSCR_DICT", + [STORE_SUBSCR_LIST_INT] = "STORE_SUBSCR_LIST_INT", + [UNPACK_SEQUENCE_ADAPTIVE] = "UNPACK_SEQUENCE_ADAPTIVE", [CALL] = "CALL", [KW_NAMES] = "KW_NAMES", [POP_JUMP_BACKWARD_IF_NOT_NONE] = "POP_JUMP_BACKWARD_IF_NOT_NONE", [POP_JUMP_BACKWARD_IF_NONE] = "POP_JUMP_BACKWARD_IF_NONE", [POP_JUMP_BACKWARD_IF_FALSE] = "POP_JUMP_BACKWARD_IF_FALSE", [POP_JUMP_BACKWARD_IF_TRUE] = "POP_JUMP_BACKWARD_IF_TRUE", + [UNPACK_SEQUENCE_LIST] = "UNPACK_SEQUENCE_LIST", + [UNPACK_SEQUENCE_TUPLE] = "UNPACK_SEQUENCE_TUPLE", + [UNPACK_SEQUENCE_TWO_TUPLE] = "UNPACK_SEQUENCE_TWO_TUPLE", + [180] = "<180>", + [181] = "<181>", + [182] = "<182>", + [183] = "<183>", + [184] = "<184>", + [185] = "<185>", + [186] = "<186>", + [187] = "<187>", + [188] = "<188>", + [189] = "<189>", + [190] = "<190>", + [191] = "<191>", + [192] = "<192>", + [193] = "<193>", + [194] = "<194>", + [195] = "<195>", + [196] = "<196>", + [197] = "<197>", + [198] = "<198>", + [199] = "<199>", + [200] = "<200>", + [201] = "<201>", + [202] = "<202>", + [203] = "<203>", + [204] = "<204>", + [205] = "<205>", + [206] = "<206>", + [207] = "<207>", + [208] = "<208>", + [209] = "<209>", + [210] = "<210>", + [211] = "<211>", + [212] = "<212>", + [213] = "<213>", + [214] = "<214>", + [215] = "<215>", + [216] = "<216>", + [217] = "<217>", + [218] = "<218>", + [219] = "<219>", + [220] = "<220>", + [221] = "<221>", + [222] = "<222>", + [223] = "<223>", + [224] = "<224>", + [225] = "<225>", + [226] = "<226>", + [227] = "<227>", + [228] = "<228>", + [229] = "<229>", + [230] = "<230>", + [231] = "<231>", + [232] = "<232>", + [233] = "<233>", + [234] = "<234>", + [235] = "<235>", + [236] = "<236>", + [237] = "<237>", + [238] = "<238>", + [239] = "<239>", + [240] = "<240>", + [241] = "<241>", + [242] = "<242>", + [243] = "<243>", + [244] = "<244>", + [245] = "<245>", + [246] = "<246>", + [247] = "<247>", + [248] = "<248>", + [249] = "<249>", + [250] = "<250>", + [251] = "<251>", + [252] = "<252>", + [253] = "<253>", + [254] = "<254>", + [DO_TRACING] = "DO_TRACING", }; #endif diff --git a/Tools/scripts/generate_opcode_h.py b/Tools/scripts/generate_opcode_h.py index 6b5cc7d7a7d3d..a13fa778db4ea 100644 --- a/Tools/scripts/generate_opcode_h.py +++ b/Tools/scripts/generate_opcode_h.py @@ -59,8 +59,22 @@ def main(opcode_py, outfile='Include/opcode.h'): hasjabs = opcode['hasjabs'] used = [ False ] * 256 next_op = 1 + for name, op in opmap.items(): used[op] = True + + specialized_opmap = {} + opname_including_specialized = opname.copy() + for name in opcode['_specialized_instructions']: + while used[next_op]: + next_op += 1 + specialized_opmap[name] = next_op + opname_including_specialized[next_op] = name + used[next_op] = True + specialized_opmap['DO_TRACING'] = 255 + opname_including_specialized[255] = 'DO_TRACING' + used[255] = True + with open(outfile, 'w') as fobj: fobj.write(header) for name in opname: @@ -69,12 +83,9 @@ def main(opcode_py, outfile='Include/opcode.h'): if name == 'POP_EXCEPT': # Special entry for HAVE_ARGUMENT fobj.write(DEFINE.format("HAVE_ARGUMENT", opcode["HAVE_ARGUMENT"])) - for name in opcode['_specialized_instructions']: - while used[next_op]: - next_op += 1 - fobj.write(DEFINE.format(name, next_op)) - used[next_op] = True - fobj.write(DEFINE.format('DO_TRACING', 255)) + for name, op in specialized_opmap.items(): + fobj.write(DEFINE.format(name, op)) + fobj.write("\nextern const uint8_t _PyOpcode_Caches[256];\n") fobj.write("\nextern const uint8_t _PyOpcode_Deopt[256];\n") fobj.write("\n#ifdef NEED_OPCODE_TABLES\n") @@ -111,8 +122,10 @@ def main(opcode_py, outfile='Include/opcode.h'): fobj.write("\n") fobj.write("#ifdef Py_DEBUG\n") fobj.write("static const char *const _PyOpcode_OpName[256] = {\n") - for name in opmap: - fobj.write(f''' [{name}] = "{name}",\n''') + for op, name in enumerate(opname_including_specialized): + if name[0] != "<": + op = name + fobj.write(f''' [{op}] = "{name}",\n''') fobj.write("};\n") fobj.write("#endif\n") From webhook-mailer at python.org Thu Apr 14 20:03:03 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Fri, 15 Apr 2022 00:03:03 -0000 Subject: [Python-checkins] gh-69093: Support basic incremental I/O to blobs in `sqlite3` (GH-30680) Message-ID: https://github.com/python/cpython/commit/ee475430d431814cbb6eb5e8a6c0ae51943349d4 commit: ee475430d431814cbb6eb5e8a6c0ae51943349d4 branch: main author: Erlend Egeberg Aasland committer: JelleZijlstra date: 2022-04-14T17:02:56-07:00 summary: gh-69093: Support basic incremental I/O to blobs in `sqlite3` (GH-30680) Authored-by: Aviv Palivoda Co-authored-by: Erlend E. Aasland Co-authored-by: palaviv Co-authored-by: Kumar Aditya <59607654+kumaraditya303 at users.noreply.github.com> Co-authored-by: Jelle Zijlstra files: A Doc/includes/sqlite3/blob.py A Misc/NEWS.d/next/Library/2018-04-18-16-15-55.bpo-24905.jYqjYx.rst A Modules/_sqlite/blob.c A Modules/_sqlite/blob.h A Modules/_sqlite/clinic/blob.c.h M Doc/library/sqlite3.rst M Doc/whatsnew/3.11.rst M Lib/test/test_sqlite3/test_dbapi.py M Modules/_sqlite/clinic/connection.c.h M Modules/_sqlite/connection.c M Modules/_sqlite/connection.h M Modules/_sqlite/module.c M Modules/_sqlite/module.h M PCbuild/_sqlite3.vcxproj M PCbuild/_sqlite3.vcxproj.filters M setup.py diff --git a/Doc/includes/sqlite3/blob.py b/Doc/includes/sqlite3/blob.py new file mode 100644 index 0000000000000..61994fb82dd72 --- /dev/null +++ b/Doc/includes/sqlite3/blob.py @@ -0,0 +1,12 @@ +import sqlite3 + +con = sqlite3.connect(":memory:") +con.execute("create table test(blob_col blob)") +con.execute("insert into test(blob_col) values (zeroblob(10))") + +blob = con.blobopen("test", "blob_col", 1) +blob.write(b"Hello") +blob.write(b"World") +blob.seek(0) +print(blob.read()) # will print b"HelloWorld" +blob.close() diff --git a/Doc/library/sqlite3.rst b/Doc/library/sqlite3.rst index 60dfbefd2e255..d0274fb79744d 100644 --- a/Doc/library/sqlite3.rst +++ b/Doc/library/sqlite3.rst @@ -394,6 +394,20 @@ Connection Objects supplied, this must be a callable returning an instance of :class:`Cursor` or its subclasses. + .. method:: blobopen(table, column, row, /, *, readonly=False, name="main") + + Open a :class:`Blob` handle to the :abbr:`BLOB (Binary Large OBject)` + located in row *row*, column *column*, table *table* of database *name*. + When *readonly* is :const:`True` the blob is opened without write + permissions. + + .. note:: + + The blob size cannot be changed using the :class:`Blob` class. + Use the SQL function ``zeroblob`` to create a blob with a fixed size. + + .. versionadded:: 3.11 + .. method:: commit() This method commits the current transaction. If you don't call this method, @@ -1088,6 +1102,58 @@ Exceptions transactions turned off. It is a subclass of :exc:`DatabaseError`. +.. _sqlite3-blob-objects: + +Blob Objects +------------ + +.. versionadded:: 3.11 + +.. class:: Blob + + A :class:`Blob` instance is a :term:`file-like object` that can read and write + data in an SQLite :abbr:`BLOB (Binary Large OBject)`. Call ``len(blob)`` to + get the size (number of bytes) of the blob. + + .. method:: close() + + Close the blob. + + The blob will be unusable from this point onward. An + :class:`~sqlite3.Error` (or subclass) exception will be raised if any + further operation is attempted with the blob. + + .. method:: read(length=-1, /) + + Read *length* bytes of data from the blob at the current offset position. + If the end of the blob is reached, the data up to + :abbr:`EOF (End of File)` will be returned. When *length* is not + specified, or is negative, :meth:`~Blob.read` will read until the end of + the blob. + + .. method:: write(data, /) + + Write *data* to the blob at the current offset. This function cannot + change the blob length. Writing beyond the end of the blob will raise + :exc:`ValueError`. + + .. method:: tell() + + Return the current access position of the blob. + + .. method:: seek(offset, origin=os.SEEK_SET, /) + + Set the current access position of the blob to *offset*. The *origin* + argument defaults to :data:`os.SEEK_SET` (absolute blob positioning). + Other values for *origin* are :data:`os.SEEK_CUR` (seek relative to the + current position) and :data:`os.SEEK_END` (seek relative to the blob?s + end). + + :class:`Blob` example: + + .. literalinclude:: ../includes/sqlite3/blob.py + + .. _sqlite3-types: SQLite and Python types diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index f8e86f6ba349b..dba554cc834ec 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -393,6 +393,10 @@ sqlite3 :class:`sqlite3.Connection` for creating aggregate window functions. (Contributed by Erlend E. Aasland in :issue:`34916`.) +* Add :meth:`~sqlite3.Connection.blobopen` to :class:`sqlite3.Connection`. + :class:`sqlite3.Blob` allows incremental I/O operations on blobs. + (Contributed by Aviv Palivoda and Erlend E. Aasland in :issue:`24905`) + sys --- diff --git a/Lib/test/test_sqlite3/test_dbapi.py b/Lib/test/test_sqlite3/test_dbapi.py index 2d2e58a3d44f5..faaa3713cb510 100644 --- a/Lib/test/test_sqlite3/test_dbapi.py +++ b/Lib/test/test_sqlite3/test_dbapi.py @@ -33,6 +33,8 @@ check_disallow_instantiation, threading_helper, ) +from _testcapi import INT_MAX +from os import SEEK_SET, SEEK_CUR, SEEK_END from test.support.os_helper import TESTFN, unlink, temp_dir @@ -1041,11 +1043,163 @@ def test_same_query_in_multiple_cursors(self): self.assertEqual(cu.fetchall(), [(1,)]) +class BlobTests(unittest.TestCase): + def setUp(self): + self.cx = sqlite.connect(":memory:") + self.cx.execute("create table test(b blob)") + self.data = b"this blob data string is exactly fifty bytes long!" + self.cx.execute("insert into test(b) values (?)", (self.data,)) + self.blob = self.cx.blobopen("test", "b", 1) + + def tearDown(self): + self.blob.close() + self.cx.close() + + def test_blob_seek_and_tell(self): + self.blob.seek(10) + self.assertEqual(self.blob.tell(), 10) + + self.blob.seek(10, SEEK_SET) + self.assertEqual(self.blob.tell(), 10) + + self.blob.seek(10, SEEK_CUR) + self.assertEqual(self.blob.tell(), 20) + + self.blob.seek(-10, SEEK_END) + self.assertEqual(self.blob.tell(), 40) + + def test_blob_seek_error(self): + msg_oor = "offset out of blob range" + msg_orig = "'origin' should be os.SEEK_SET, os.SEEK_CUR, or os.SEEK_END" + msg_of = "seek offset results in overflow" + + dataset = ( + (ValueError, msg_oor, lambda: self.blob.seek(1000)), + (ValueError, msg_oor, lambda: self.blob.seek(-10)), + (ValueError, msg_orig, lambda: self.blob.seek(10, -1)), + (ValueError, msg_orig, lambda: self.blob.seek(10, 3)), + ) + for exc, msg, fn in dataset: + with self.subTest(exc=exc, msg=msg, fn=fn): + self.assertRaisesRegex(exc, msg, fn) + + # Force overflow errors + self.blob.seek(1, SEEK_SET) + with self.assertRaisesRegex(OverflowError, msg_of): + self.blob.seek(INT_MAX, SEEK_CUR) + with self.assertRaisesRegex(OverflowError, msg_of): + self.blob.seek(INT_MAX, SEEK_END) + + def test_blob_read(self): + buf = self.blob.read() + self.assertEqual(buf, self.data) + + def test_blob_read_oversized(self): + buf = self.blob.read(len(self.data) * 2) + self.assertEqual(buf, self.data) + + def test_blob_read_advance_offset(self): + n = 10 + buf = self.blob.read(n) + self.assertEqual(buf, self.data[:n]) + self.assertEqual(self.blob.tell(), n) + + def test_blob_read_at_offset(self): + self.blob.seek(10) + self.assertEqual(self.blob.read(10), self.data[10:20]) + + def test_blob_read_error_row_changed(self): + self.cx.execute("update test set b='aaaa' where rowid=1") + with self.assertRaises(sqlite.OperationalError): + self.blob.read() + + def test_blob_write(self): + new_data = b"new data".ljust(50) + self.blob.write(new_data) + row = self.cx.execute("select b from test").fetchone() + self.assertEqual(row[0], new_data) + + def test_blob_write_at_offset(self): + new_data = b"c" * 25 + self.blob.seek(25) + self.blob.write(new_data) + row = self.cx.execute("select b from test").fetchone() + self.assertEqual(row[0], self.data[:25] + new_data) + + def test_blob_write_advance_offset(self): + self.blob.write(b"d"*10) + self.assertEqual(self.blob.tell(), 10) + + def test_blob_write_error_length(self): + with self.assertRaisesRegex(ValueError, "data longer than blob"): + self.blob.write(b"a" * 1000) + + def test_blob_write_error_row_changed(self): + self.cx.execute("update test set b='aaaa' where rowid=1") + with self.assertRaises(sqlite.OperationalError): + self.blob.write(b"aaa") + + def test_blob_write_error_readonly(self): + ro_blob = self.cx.blobopen("test", "b", 1, readonly=True) + with self.assertRaisesRegex(sqlite.OperationalError, "readonly"): + ro_blob.write(b"aaa") + ro_blob.close() + + def test_blob_open_error(self): + dataset = ( + (("test", "b", 1), {"name": "notexisting"}), + (("notexisting", "b", 1), {}), + (("test", "notexisting", 1), {}), + (("test", "b", 2), {}), + ) + regex = "no such" + for args, kwds in dataset: + with self.subTest(args=args, kwds=kwds): + with self.assertRaisesRegex(sqlite.OperationalError, regex): + self.cx.blobopen(*args, **kwds) + + def test_blob_sequence_not_supported(self): + with self.assertRaises(TypeError): + self.blob + self.blob + with self.assertRaises(TypeError): + self.blob * 5 + with self.assertRaises(TypeError): + b"a" in self.blob + + def test_blob_closed(self): + with memory_database() as cx: + cx.execute("create table test(b blob)") + cx.execute("insert into test values (zeroblob(100))") + blob = cx.blobopen("test", "b", 1) + blob.close() + + msg = "Cannot operate on a closed blob" + with self.assertRaisesRegex(sqlite.ProgrammingError, msg): + blob.read() + with self.assertRaisesRegex(sqlite.ProgrammingError, msg): + blob.write(b"") + with self.assertRaisesRegex(sqlite.ProgrammingError, msg): + blob.seek(0) + with self.assertRaisesRegex(sqlite.ProgrammingError, msg): + blob.tell() + + def test_blob_closed_db_read(self): + with memory_database() as cx: + cx.execute("create table test(b blob)") + cx.execute("insert into test(b) values (zeroblob(100))") + blob = cx.blobopen("test", "b", 1) + cx.close() + self.assertRaisesRegex(sqlite.ProgrammingError, + "Cannot operate on a closed database", + blob.read) + + class ThreadTests(unittest.TestCase): def setUp(self): self.con = sqlite.connect(":memory:") self.cur = self.con.cursor() - self.cur.execute("create table test(name text)") + self.cur.execute("create table test(name text, b blob)") + self.cur.execute("insert into test values('blob', zeroblob(1))") def tearDown(self): self.cur.close() @@ -1080,6 +1234,7 @@ def test_check_connection_thread(self): lambda: self.con.create_collation("foo", None), lambda: self.con.setlimit(sqlite.SQLITE_LIMIT_LENGTH, -1), lambda: self.con.getlimit(sqlite.SQLITE_LIMIT_LENGTH), + lambda: self.con.blobopen("test", "b", 1), ] if hasattr(sqlite.Connection, "serialize"): fns.append(lambda: self.con.serialize()) diff --git a/Misc/NEWS.d/next/Library/2018-04-18-16-15-55.bpo-24905.jYqjYx.rst b/Misc/NEWS.d/next/Library/2018-04-18-16-15-55.bpo-24905.jYqjYx.rst new file mode 100644 index 0000000000000..0a57f90c12378 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2018-04-18-16-15-55.bpo-24905.jYqjYx.rst @@ -0,0 +1,3 @@ +Add :meth:`~sqlite3.Connection.blobopen` to :class:`sqlite3.Connection`. +:class:`sqlite3.Blob` allows incremental I/O operations on blobs. +Patch by Aviv Palivoda and Erlend E. Aasland. diff --git a/Modules/_sqlite/blob.c b/Modules/_sqlite/blob.c new file mode 100644 index 0000000000000..821295cee813f --- /dev/null +++ b/Modules/_sqlite/blob.c @@ -0,0 +1,351 @@ +#include "blob.h" +#include "util.h" + +#define clinic_state() (pysqlite_get_state_by_type(Py_TYPE(self))) +#include "clinic/blob.c.h" +#undef clinic_state + +/*[clinic input] +module _sqlite3 +class _sqlite3.Blob "pysqlite_Blob *" "clinic_state()->BlobType" +[clinic start generated code]*/ +/*[clinic end generated code: output=da39a3ee5e6b4b0d input=908d3e16a45f8da7]*/ + +static void +close_blob(pysqlite_Blob *self) +{ + if (self->blob) { + sqlite3_blob *blob = self->blob; + self->blob = NULL; + + Py_BEGIN_ALLOW_THREADS + sqlite3_blob_close(blob); + Py_END_ALLOW_THREADS + } +} + +static int +blob_traverse(pysqlite_Blob *self, visitproc visit, void *arg) +{ + Py_VISIT(Py_TYPE(self)); + Py_VISIT(self->connection); + return 0; +} + +static int +blob_clear(pysqlite_Blob *self) +{ + Py_CLEAR(self->connection); + return 0; +} + +static void +blob_dealloc(pysqlite_Blob *self) +{ + PyTypeObject *tp = Py_TYPE(self); + PyObject_GC_UnTrack(self); + + close_blob(self); + + if (self->in_weakreflist != NULL) { + PyObject_ClearWeakRefs((PyObject*)self); + } + tp->tp_clear((PyObject *)self); + tp->tp_free(self); + Py_DECREF(tp); +} + +// Return 1 if the blob object is usable, 0 if not. +static int +check_blob(pysqlite_Blob *self) +{ + if (!pysqlite_check_connection(self->connection) || + !pysqlite_check_thread(self->connection)) { + return 0; + } + if (self->blob == NULL) { + pysqlite_state *state = self->connection->state; + PyErr_SetString(state->ProgrammingError, + "Cannot operate on a closed blob."); + return 0; + } + return 1; +} + + +/*[clinic input] +_sqlite3.Blob.close as blob_close + +Close the blob. +[clinic start generated code]*/ + +static PyObject * +blob_close_impl(pysqlite_Blob *self) +/*[clinic end generated code: output=848accc20a138d1b input=7bc178a402a40bd8]*/ +{ + if (!pysqlite_check_connection(self->connection) || + !pysqlite_check_thread(self->connection)) + { + return NULL; + } + close_blob(self); + Py_RETURN_NONE; +}; + +void +pysqlite_close_all_blobs(pysqlite_Connection *self) +{ + for (int i = 0; i < PyList_GET_SIZE(self->blobs); i++) { + PyObject *weakref = PyList_GET_ITEM(self->blobs, i); + PyObject *blob = PyWeakref_GetObject(weakref); + if (!Py_IsNone(blob)) { + close_blob((pysqlite_Blob *)blob); + } + } +} + +static void +blob_seterror(pysqlite_Blob *self, int rc) +{ + assert(self->connection != NULL); +#if SQLITE_VERSION_NUMBER < 3008008 + // SQLite pre 3.8.8 does not set this blob error on the connection + if (rc == SQLITE_ABORT) { + PyErr_SetString(self->connection->OperationalError, + "Cannot operate on an expired blob handle"); + return; + } +#endif + _pysqlite_seterror(self->connection->state, self->connection->db); +} + +static PyObject * +inner_read(pysqlite_Blob *self, int length, int offset) +{ + PyObject *buffer = PyBytes_FromStringAndSize(NULL, length); + if (buffer == NULL) { + return NULL; + } + + char *raw_buffer = PyBytes_AS_STRING(buffer); + int rc; + Py_BEGIN_ALLOW_THREADS + rc = sqlite3_blob_read(self->blob, raw_buffer, length, offset); + Py_END_ALLOW_THREADS + + if (rc != SQLITE_OK) { + Py_DECREF(buffer); + blob_seterror(self, rc); + return NULL; + } + return buffer; +} + + +/*[clinic input] +_sqlite3.Blob.read as blob_read + + length: int = -1 + Read length in bytes. + / + +Read data at the current offset position. + +If the end of the blob is reached, the data up to end of file will be returned. +When length is not specified, or is negative, Blob.read() will read until the +end of the blob. +[clinic start generated code]*/ + +static PyObject * +blob_read_impl(pysqlite_Blob *self, int length) +/*[clinic end generated code: output=1fc99b2541360dde input=f2e4aa4378837250]*/ +{ + if (!check_blob(self)) { + return NULL; + } + + /* Make sure we never read past "EOB". Also read the rest of the blob if a + * negative length is specified. */ + int blob_len = sqlite3_blob_bytes(self->blob); + int max_read_len = blob_len - self->offset; + if (length < 0 || length > max_read_len) { + length = max_read_len; + } + + PyObject *buffer = inner_read(self, length, self->offset); + if (buffer == NULL) { + return NULL; + } + self->offset += length; + return buffer; +}; + +static int +inner_write(pysqlite_Blob *self, const void *buf, Py_ssize_t len, int offset) +{ + int remaining_len = sqlite3_blob_bytes(self->blob) - self->offset; + if (len > remaining_len) { + PyErr_SetString(PyExc_ValueError, "data longer than blob length"); + return -1; + } + + int rc; + Py_BEGIN_ALLOW_THREADS + rc = sqlite3_blob_write(self->blob, buf, (int)len, offset); + Py_END_ALLOW_THREADS + + if (rc != SQLITE_OK) { + blob_seterror(self, rc); + return -1; + } + return 0; +} + + +/*[clinic input] +_sqlite3.Blob.write as blob_write + + data: Py_buffer + / + +Write data at the current offset. + +This function cannot change the blob length. Writing beyond the end of the +blob will result in an exception being raised. +[clinic start generated code]*/ + +static PyObject * +blob_write_impl(pysqlite_Blob *self, Py_buffer *data) +/*[clinic end generated code: output=b34cf22601b570b2 input=a84712f24a028e6d]*/ +{ + if (!check_blob(self)) { + return NULL; + } + + int rc = inner_write(self, data->buf, data->len, self->offset); + if (rc < 0) { + return NULL; + } + self->offset += (int)data->len; + Py_RETURN_NONE; +} + + +/*[clinic input] +_sqlite3.Blob.seek as blob_seek + + offset: int + origin: int = 0 + / + +Set the current access position to offset. + +The origin argument defaults to os.SEEK_SET (absolute blob positioning). +Other values for origin are os.SEEK_CUR (seek relative to the current position) +and os.SEEK_END (seek relative to the blob's end). +[clinic start generated code]*/ + +static PyObject * +blob_seek_impl(pysqlite_Blob *self, int offset, int origin) +/*[clinic end generated code: output=854c5a0e208547a5 input=5da9a07e55fe6bb6]*/ +{ + if (!check_blob(self)) { + return NULL; + } + + int blob_len = sqlite3_blob_bytes(self->blob); + switch (origin) { + case SEEK_SET: + break; + case SEEK_CUR: + if (offset > INT_MAX - self->offset) { + goto overflow; + } + offset += self->offset; + break; + case SEEK_END: + if (offset > INT_MAX - blob_len) { + goto overflow; + } + offset += blob_len; + break; + default: + PyErr_SetString(PyExc_ValueError, + "'origin' should be os.SEEK_SET, os.SEEK_CUR, or " + "os.SEEK_END"); + return NULL; + } + + if (offset < 0 || offset > blob_len) { + PyErr_SetString(PyExc_ValueError, "offset out of blob range"); + return NULL; + } + + self->offset = offset; + Py_RETURN_NONE; + +overflow: + PyErr_SetString(PyExc_OverflowError, "seek offset results in overflow"); + return NULL; +} + + +/*[clinic input] +_sqlite3.Blob.tell as blob_tell + +Return the current access position for the blob. +[clinic start generated code]*/ + +static PyObject * +blob_tell_impl(pysqlite_Blob *self) +/*[clinic end generated code: output=3d3ba484a90b3a99 input=7e34057aa303612c]*/ +{ + if (!check_blob(self)) { + return NULL; + } + return PyLong_FromLong(self->offset); +} + + +static PyMethodDef blob_methods[] = { + BLOB_CLOSE_METHODDEF + BLOB_READ_METHODDEF + BLOB_SEEK_METHODDEF + BLOB_TELL_METHODDEF + BLOB_WRITE_METHODDEF + {NULL, NULL} +}; + +static struct PyMemberDef blob_members[] = { + {"__weaklistoffset__", T_PYSSIZET, offsetof(pysqlite_Blob, in_weakreflist), READONLY}, + {NULL}, +}; + +static PyType_Slot blob_slots[] = { + {Py_tp_dealloc, blob_dealloc}, + {Py_tp_traverse, blob_traverse}, + {Py_tp_clear, blob_clear}, + {Py_tp_methods, blob_methods}, + {Py_tp_members, blob_members}, + {0, NULL}, +}; + +static PyType_Spec blob_spec = { + .name = MODULE_NAME ".Blob", + .basicsize = sizeof(pysqlite_Blob), + .flags = (Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC | + Py_TPFLAGS_IMMUTABLETYPE), + .slots = blob_slots, +}; + +int +pysqlite_blob_setup_types(PyObject *mod) +{ + PyObject *type = PyType_FromModuleAndSpec(mod, &blob_spec, NULL); + if (type == NULL) { + return -1; + } + pysqlite_state *state = pysqlite_get_state(mod); + state->BlobType = (PyTypeObject *)type; + return 0; +} diff --git a/Modules/_sqlite/blob.h b/Modules/_sqlite/blob.h new file mode 100644 index 0000000000000..418ca03bdb51d --- /dev/null +++ b/Modules/_sqlite/blob.h @@ -0,0 +1,24 @@ +#ifndef PYSQLITE_BLOB_H +#define PYSQLITE_BLOB_H + +#include "Python.h" +#include "sqlite3.h" +#include "connection.h" + +#define BLOB_SEEK_START 0 +#define BLOB_SEEK_CUR 1 +#define BLOB_SEEK_END 2 + +typedef struct { + PyObject_HEAD + pysqlite_Connection *connection; + sqlite3_blob *blob; + int offset; + + PyObject *in_weakreflist; +} pysqlite_Blob; + +int pysqlite_blob_setup_types(PyObject *mod); +void pysqlite_close_all_blobs(pysqlite_Connection *self); + +#endif diff --git a/Modules/_sqlite/clinic/blob.c.h b/Modules/_sqlite/clinic/blob.c.h new file mode 100644 index 0000000000000..30b3e3c194739 --- /dev/null +++ b/Modules/_sqlite/clinic/blob.c.h @@ -0,0 +1,165 @@ +/*[clinic input] +preserve +[clinic start generated code]*/ + +PyDoc_STRVAR(blob_close__doc__, +"close($self, /)\n" +"--\n" +"\n" +"Close the blob."); + +#define BLOB_CLOSE_METHODDEF \ + {"close", (PyCFunction)blob_close, METH_NOARGS, blob_close__doc__}, + +static PyObject * +blob_close_impl(pysqlite_Blob *self); + +static PyObject * +blob_close(pysqlite_Blob *self, PyObject *Py_UNUSED(ignored)) +{ + return blob_close_impl(self); +} + +PyDoc_STRVAR(blob_read__doc__, +"read($self, length=-1, /)\n" +"--\n" +"\n" +"Read data at the current offset position.\n" +"\n" +" length\n" +" Read length in bytes.\n" +"\n" +"If the end of the blob is reached, the data up to end of file will be returned.\n" +"When length is not specified, or is negative, Blob.read() will read until the\n" +"end of the blob."); + +#define BLOB_READ_METHODDEF \ + {"read", (PyCFunction)(void(*)(void))blob_read, METH_FASTCALL, blob_read__doc__}, + +static PyObject * +blob_read_impl(pysqlite_Blob *self, int length); + +static PyObject * +blob_read(pysqlite_Blob *self, PyObject *const *args, Py_ssize_t nargs) +{ + PyObject *return_value = NULL; + int length = -1; + + if (!_PyArg_CheckPositional("read", nargs, 0, 1)) { + goto exit; + } + if (nargs < 1) { + goto skip_optional; + } + length = _PyLong_AsInt(args[0]); + if (length == -1 && PyErr_Occurred()) { + goto exit; + } +skip_optional: + return_value = blob_read_impl(self, length); + +exit: + return return_value; +} + +PyDoc_STRVAR(blob_write__doc__, +"write($self, data, /)\n" +"--\n" +"\n" +"Write data at the current offset.\n" +"\n" +"This function cannot change the blob length. Writing beyond the end of the\n" +"blob will result in an exception being raised."); + +#define BLOB_WRITE_METHODDEF \ + {"write", (PyCFunction)blob_write, METH_O, blob_write__doc__}, + +static PyObject * +blob_write_impl(pysqlite_Blob *self, Py_buffer *data); + +static PyObject * +blob_write(pysqlite_Blob *self, PyObject *arg) +{ + PyObject *return_value = NULL; + Py_buffer data = {NULL, NULL}; + + if (PyObject_GetBuffer(arg, &data, PyBUF_SIMPLE) != 0) { + goto exit; + } + if (!PyBuffer_IsContiguous(&data, 'C')) { + _PyArg_BadArgument("write", "argument", "contiguous buffer", arg); + goto exit; + } + return_value = blob_write_impl(self, &data); + +exit: + /* Cleanup for data */ + if (data.obj) { + PyBuffer_Release(&data); + } + + return return_value; +} + +PyDoc_STRVAR(blob_seek__doc__, +"seek($self, offset, origin=0, /)\n" +"--\n" +"\n" +"Set the current access position to offset.\n" +"\n" +"The origin argument defaults to os.SEEK_SET (absolute blob positioning).\n" +"Other values for origin are os.SEEK_CUR (seek relative to the current position)\n" +"and os.SEEK_END (seek relative to the blob\'s end)."); + +#define BLOB_SEEK_METHODDEF \ + {"seek", (PyCFunction)(void(*)(void))blob_seek, METH_FASTCALL, blob_seek__doc__}, + +static PyObject * +blob_seek_impl(pysqlite_Blob *self, int offset, int origin); + +static PyObject * +blob_seek(pysqlite_Blob *self, PyObject *const *args, Py_ssize_t nargs) +{ + PyObject *return_value = NULL; + int offset; + int origin = 0; + + if (!_PyArg_CheckPositional("seek", nargs, 1, 2)) { + goto exit; + } + offset = _PyLong_AsInt(args[0]); + if (offset == -1 && PyErr_Occurred()) { + goto exit; + } + if (nargs < 2) { + goto skip_optional; + } + origin = _PyLong_AsInt(args[1]); + if (origin == -1 && PyErr_Occurred()) { + goto exit; + } +skip_optional: + return_value = blob_seek_impl(self, offset, origin); + +exit: + return return_value; +} + +PyDoc_STRVAR(blob_tell__doc__, +"tell($self, /)\n" +"--\n" +"\n" +"Return the current access position for the blob."); + +#define BLOB_TELL_METHODDEF \ + {"tell", (PyCFunction)blob_tell, METH_NOARGS, blob_tell__doc__}, + +static PyObject * +blob_tell_impl(pysqlite_Blob *self); + +static PyObject * +blob_tell(pysqlite_Blob *self, PyObject *Py_UNUSED(ignored)) +{ + return blob_tell_impl(self); +} +/*[clinic end generated code: output=d3a02b127f2cfa58 input=a9049054013a1b77]*/ diff --git a/Modules/_sqlite/clinic/connection.c.h b/Modules/_sqlite/clinic/connection.c.h index 2b933f8522465..d4597086f4c9a 100644 --- a/Modules/_sqlite/clinic/connection.c.h +++ b/Modules/_sqlite/clinic/connection.c.h @@ -145,6 +145,110 @@ pysqlite_connection_cursor(pysqlite_Connection *self, PyObject *const *args, Py_ return return_value; } +PyDoc_STRVAR(blobopen__doc__, +"blobopen($self, table, column, row, /, *, readonly=False, name=\'main\')\n" +"--\n" +"\n" +"Open and return a BLOB object.\n" +"\n" +" table\n" +" Table name.\n" +" column\n" +" Column name.\n" +" row\n" +" Row index.\n" +" readonly\n" +" Open the BLOB without write permissions.\n" +" name\n" +" Database name."); + +#define BLOBOPEN_METHODDEF \ + {"blobopen", (PyCFunction)(void(*)(void))blobopen, METH_FASTCALL|METH_KEYWORDS, blobopen__doc__}, + +static PyObject * +blobopen_impl(pysqlite_Connection *self, const char *table, const char *col, + int row, int readonly, const char *name); + +static PyObject * +blobopen(pysqlite_Connection *self, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) +{ + PyObject *return_value = NULL; + static const char * const _keywords[] = {"", "", "", "readonly", "name", NULL}; + static _PyArg_Parser _parser = {NULL, _keywords, "blobopen", 0}; + PyObject *argsbuf[5]; + Py_ssize_t noptargs = nargs + (kwnames ? PyTuple_GET_SIZE(kwnames) : 0) - 3; + const char *table; + const char *col; + int row; + int readonly = 0; + const char *name = "main"; + + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 3, 3, 0, argsbuf); + if (!args) { + goto exit; + } + if (!PyUnicode_Check(args[0])) { + _PyArg_BadArgument("blobopen", "argument 1", "str", args[0]); + goto exit; + } + Py_ssize_t table_length; + table = PyUnicode_AsUTF8AndSize(args[0], &table_length); + if (table == NULL) { + goto exit; + } + if (strlen(table) != (size_t)table_length) { + PyErr_SetString(PyExc_ValueError, "embedded null character"); + goto exit; + } + if (!PyUnicode_Check(args[1])) { + _PyArg_BadArgument("blobopen", "argument 2", "str", args[1]); + goto exit; + } + Py_ssize_t col_length; + col = PyUnicode_AsUTF8AndSize(args[1], &col_length); + if (col == NULL) { + goto exit; + } + if (strlen(col) != (size_t)col_length) { + PyErr_SetString(PyExc_ValueError, "embedded null character"); + goto exit; + } + row = _PyLong_AsInt(args[2]); + if (row == -1 && PyErr_Occurred()) { + goto exit; + } + if (!noptargs) { + goto skip_optional_kwonly; + } + if (args[3]) { + readonly = _PyLong_AsInt(args[3]); + if (readonly == -1 && PyErr_Occurred()) { + goto exit; + } + if (!--noptargs) { + goto skip_optional_kwonly; + } + } + if (!PyUnicode_Check(args[4])) { + _PyArg_BadArgument("blobopen", "argument 'name'", "str", args[4]); + goto exit; + } + Py_ssize_t name_length; + name = PyUnicode_AsUTF8AndSize(args[4], &name_length); + if (name == NULL) { + goto exit; + } + if (strlen(name) != (size_t)name_length) { + PyErr_SetString(PyExc_ValueError, "embedded null character"); + goto exit; + } +skip_optional_kwonly: + return_value = blobopen_impl(self, table, col, row, readonly, name); + +exit: + return return_value; +} + PyDoc_STRVAR(pysqlite_connection_close__doc__, "close($self, /)\n" "--\n" @@ -1041,4 +1145,4 @@ getlimit(pysqlite_Connection *self, PyObject *arg) #ifndef DESERIALIZE_METHODDEF #define DESERIALIZE_METHODDEF #endif /* !defined(DESERIALIZE_METHODDEF) */ -/*[clinic end generated code: output=b9af1b52fda808bf input=a9049054013a1b77]*/ +/*[clinic end generated code: output=be2f526e78fa65b1 input=a9049054013a1b77]*/ diff --git a/Modules/_sqlite/connection.c b/Modules/_sqlite/connection.c index d7c0a9e46161c..85fb128fc7f1c 100644 --- a/Modules/_sqlite/connection.c +++ b/Modules/_sqlite/connection.c @@ -26,6 +26,7 @@ #include "connection.h" #include "statement.h" #include "cursor.h" +#include "blob.h" #include "prepare_protocol.h" #include "util.h" @@ -234,10 +235,17 @@ pysqlite_connection_init_impl(pysqlite_Connection *self, return -1; } - // Create list of weak references to cursors. + /* Create lists of weak references to cursors and blobs */ PyObject *cursors = PyList_New(0); if (cursors == NULL) { - Py_DECREF(statement_cache); + Py_XDECREF(statement_cache); + return -1; + } + + PyObject *blobs = PyList_New(0); + if (blobs == NULL) { + Py_XDECREF(statement_cache); + Py_XDECREF(cursors); return -1; } @@ -250,6 +258,7 @@ pysqlite_connection_init_impl(pysqlite_Connection *self, self->thread_ident = PyThread_get_thread_ident(); self->statement_cache = statement_cache; self->cursors = cursors; + self->blobs = blobs; self->created_cursors = 0; self->row_factory = Py_NewRef(Py_None); self->text_factory = Py_NewRef(&PyUnicode_Type); @@ -291,6 +300,7 @@ connection_traverse(pysqlite_Connection *self, visitproc visit, void *arg) Py_VISIT(Py_TYPE(self)); Py_VISIT(self->statement_cache); Py_VISIT(self->cursors); + Py_VISIT(self->blobs); Py_VISIT(self->row_factory); Py_VISIT(self->text_factory); VISIT_CALLBACK_CONTEXT(self->trace_ctx); @@ -314,6 +324,7 @@ connection_clear(pysqlite_Connection *self) { Py_CLEAR(self->statement_cache); Py_CLEAR(self->cursors); + Py_CLEAR(self->blobs); Py_CLEAR(self->row_factory); Py_CLEAR(self->text_factory); clear_callback_context(self->trace_ctx); @@ -429,6 +440,76 @@ pysqlite_connection_cursor_impl(pysqlite_Connection *self, PyObject *factory) return cursor; } +/*[clinic input] +_sqlite3.Connection.blobopen as blobopen + + table: str + Table name. + column as col: str + Column name. + row: int + Row index. + / + * + readonly: bool(accept={int}) = False + Open the BLOB without write permissions. + name: str = "main" + Database name. + +Open and return a BLOB object. +[clinic start generated code]*/ + +static PyObject * +blobopen_impl(pysqlite_Connection *self, const char *table, const char *col, + int row, int readonly, const char *name) +/*[clinic end generated code: output=0c8e2e58516d0b5c input=1e7052516acfc94d]*/ +{ + if (!pysqlite_check_thread(self) || !pysqlite_check_connection(self)) { + return NULL; + } + + int rc; + sqlite3_blob *blob; + + Py_BEGIN_ALLOW_THREADS + rc = sqlite3_blob_open(self->db, name, table, col, row, !readonly, &blob); + Py_END_ALLOW_THREADS + + if (rc != SQLITE_OK) { + _pysqlite_seterror(self->state, self->db); + return NULL; + } + + pysqlite_Blob *obj = PyObject_GC_New(pysqlite_Blob, self->state->BlobType); + if (obj == NULL) { + goto error; + } + + obj->connection = (pysqlite_Connection *)Py_NewRef(self); + obj->blob = blob; + obj->offset = 0; + obj->in_weakreflist = NULL; + + PyObject_GC_Track(obj); + + // Add our blob to connection blobs list + PyObject *weakref = PyWeakref_NewRef((PyObject *)obj, NULL); + if (weakref == NULL) { + goto error; + } + rc = PyList_Append(self->blobs, weakref); + Py_DECREF(weakref); + if (rc < 0) { + goto error; + } + + return (PyObject *)obj; + +error: + Py_XDECREF(obj); + return NULL; +} + /*[clinic input] _sqlite3.Connection.close as pysqlite_connection_close @@ -451,6 +532,7 @@ pysqlite_connection_close_impl(pysqlite_Connection *self) return NULL; } + pysqlite_close_all_blobs(self); Py_CLEAR(self->statement_cache); connection_close(self); @@ -2257,6 +2339,7 @@ static PyMethodDef connection_methods[] = { SERIALIZE_METHODDEF DESERIALIZE_METHODDEF CREATE_WINDOW_FUNCTION_METHODDEF + BLOBOPEN_METHODDEF {NULL, NULL} }; diff --git a/Modules/_sqlite/connection.h b/Modules/_sqlite/connection.h index 84f1f095cb386..2b946ff3c7369 100644 --- a/Modules/_sqlite/connection.h +++ b/Modules/_sqlite/connection.h @@ -63,8 +63,9 @@ typedef struct PyObject *statement_cache; - /* Lists of weak references to statements and cursors used within this connection */ - PyObject* cursors; + /* Lists of weak references to cursors and blobs used within this connection */ + PyObject *cursors; + PyObject *blobs; /* Counters for how many cursors were created in the connection. May be * reset to 0 at certain intervals */ diff --git a/Modules/_sqlite/module.c b/Modules/_sqlite/module.c index ffda836d7a3cc..d355c2be37a2a 100644 --- a/Modules/_sqlite/module.c +++ b/Modules/_sqlite/module.c @@ -27,6 +27,7 @@ #include "prepare_protocol.h" #include "microprotocols.h" #include "row.h" +#include "blob.h" #if SQLITE_VERSION_NUMBER < 3007015 #error "SQLite 3.7.15 or higher required" @@ -582,6 +583,7 @@ module_traverse(PyObject *module, visitproc visit, void *arg) Py_VISIT(state->Warning); // Types + Py_VISIT(state->BlobType); Py_VISIT(state->ConnectionType); Py_VISIT(state->CursorType); Py_VISIT(state->PrepareProtocolType); @@ -614,6 +616,7 @@ module_clear(PyObject *module) Py_CLEAR(state->Warning); // Types + Py_CLEAR(state->BlobType); Py_CLEAR(state->ConnectionType); Py_CLEAR(state->CursorType); Py_CLEAR(state->PrepareProtocolType); @@ -687,7 +690,8 @@ module_exec(PyObject *module) (pysqlite_cursor_setup_types(module) < 0) || (pysqlite_connection_setup_types(module) < 0) || (pysqlite_statement_setup_types(module) < 0) || - (pysqlite_prepare_protocol_setup_types(module) < 0) + (pysqlite_prepare_protocol_setup_types(module) < 0) || + (pysqlite_blob_setup_types(module) < 0) ) { goto error; } diff --git a/Modules/_sqlite/module.h b/Modules/_sqlite/module.h index fcea7096924ce..7deba22ffec1b 100644 --- a/Modules/_sqlite/module.h +++ b/Modules/_sqlite/module.h @@ -53,6 +53,7 @@ typedef struct { int BaseTypeAdapted; int enable_callback_tracebacks; + PyTypeObject *BlobType; PyTypeObject *ConnectionType; PyTypeObject *CursorType; PyTypeObject *PrepareProtocolType; diff --git a/PCbuild/_sqlite3.vcxproj b/PCbuild/_sqlite3.vcxproj index 9cff43f73e5be..804aa07367a02 100644 --- a/PCbuild/_sqlite3.vcxproj +++ b/PCbuild/_sqlite3.vcxproj @@ -106,6 +106,7 @@ + @@ -116,6 +117,7 @@ + diff --git a/PCbuild/_sqlite3.vcxproj.filters b/PCbuild/_sqlite3.vcxproj.filters index 79fc17b53fb50..f4a265eba7dd8 100644 --- a/PCbuild/_sqlite3.vcxproj.filters +++ b/PCbuild/_sqlite3.vcxproj.filters @@ -36,6 +36,9 @@ Header Files + + Header Files + @@ -62,6 +65,9 @@ Source Files + + Source Files + diff --git a/setup.py b/setup.py index 1e1627ec3927d..60a45cf260f6b 100644 --- a/setup.py +++ b/setup.py @@ -1256,6 +1256,7 @@ def detect_dbm_gdbm(self): def detect_sqlite(self): sources = [ + "_sqlite/blob.c", "_sqlite/connection.c", "_sqlite/cursor.c", "_sqlite/microprotocols.c", From webhook-mailer at python.org Thu Apr 14 20:49:12 2022 From: webhook-mailer at python.org (warsaw) Date: Fri, 15 Apr 2022 00:49:12 -0000 Subject: [Python-checkins] gh-91520: Rewrite imghdr inlining for clarity and completeness (#91521) Message-ID: https://github.com/python/cpython/commit/1fcb39ea64192fc83e7b52f067856bdf977ec2c1 commit: 1fcb39ea64192fc83e7b52f067856bdf977ec2c1 branch: main author: Barry Warsaw committer: warsaw date: 2022-04-14T17:48:59-07:00 summary: gh-91520: Rewrite imghdr inlining for clarity and completeness (#91521) * Rewrite imghdr inlining for clarity and completeness * Move MIMEImage class back closer to the top of the file since it's the important thing. * Use a decorate to mark a given rule function and simplify the rule function names for clarity. * Copy over all the imghdr test data files into the email package's test data directory. This way when imghdr is actually removed, it won't affect the MIMEImage guessing tests. * Rewrite and extend the MIMEImage tests to test for all supported auto-detected MIME image subtypes. * Remove the now redundant PyBanner048.gif data file. * See https://github.com/python/cpython/pull/91461#discussion_r850313336 Co-authored-by: Oleg Iarygin Co-authored-by: Oleg Iarygin files: A Lib/test/test_email/data/python.bmp A Lib/test/test_email/data/python.exr A Lib/test/test_email/data/python.gif A Lib/test/test_email/data/python.jpg A Lib/test/test_email/data/python.pbm A Lib/test/test_email/data/python.pgm A Lib/test/test_email/data/python.png A Lib/test/test_email/data/python.ppm A Lib/test/test_email/data/python.ras A Lib/test/test_email/data/python.sgi A Lib/test/test_email/data/python.tiff A Lib/test/test_email/data/python.webp A Lib/test/test_email/data/python.xbm D Lib/test/test_email/data/PyBanner048.gif M Doc/includes/email-mime.py M Lib/email/mime/image.py M Lib/test/test_email/test_email.py diff --git a/Doc/includes/email-mime.py b/Doc/includes/email-mime.py index c87db6a064b00..34c6bdb60fff2 100644 --- a/Doc/includes/email-mime.py +++ b/Doc/includes/email-mime.py @@ -1,7 +1,7 @@ -# Import smtplib for the actual sending function +# Import smtplib for the actual sending function. import smtplib -# Here are the email package modules we'll need +# Here are the email package modules we'll need. from email.message import EmailMessage # Create the container email message. @@ -13,13 +13,13 @@ msg['To'] = ', '.join(family) msg.preamble = 'You will not see this in a MIME-aware mail reader.\n' -# Open the files in binary mode. Use imghdr to figure out the -# MIME subtype for each specific image. +# Open the files in binary mode. You can also omit the subtype +# if you want MIMEImage to guess it. for file in pngfiles: with open(file, 'rb') as fp: img_data = fp.read() msg.add_attachment(img_data, maintype='image', - subtype='jpeg') + subtype='png') # Send the email via our own SMTP server. with smtplib.SMTP('localhost') as s: diff --git a/Lib/email/mime/image.py b/Lib/email/mime/image.py index fac238c7289fa..e19dea91c0c99 100644 --- a/Lib/email/mime/image.py +++ b/Lib/email/mime/image.py @@ -10,137 +10,143 @@ from email.mime.nonmultipart import MIMENonMultipart +class MIMEImage(MIMENonMultipart): + """Class for generating image/* type MIME documents.""" + + def __init__(self, _imagedata, _subtype=None, + _encoder=encoders.encode_base64, *, policy=None, **_params): + """Create an image/* type MIME document. + + _imagedata is a string containing the raw image data. If the data + type can be detected (jpeg, png, gif, tiff, rgb, pbm, pgm, ppm, + rast, xbm, bmp, webp, and exr attempted), then the subtype will be + automatically included in the Content-Type header. Otherwise, you can + specify the specific image subtype via the _subtype parameter. + + _encoder is a function which will perform the actual encoding for + transport of the image data. It takes one argument, which is this + Image instance. It should use get_payload() and set_payload() to + change the payload to the encoded form. It should also add any + Content-Transfer-Encoding or other headers to the message as + necessary. The default encoding is Base64. + + Any additional keyword arguments are passed to the base class + constructor, which turns them into parameters on the Content-Type + header. + """ + _subtype = _what(_imagedata) if _subtype is None else _subtype + if _subtype is None: + raise TypeError('Could not guess image MIME subtype') + MIMENonMultipart.__init__(self, 'image', _subtype, policy=policy, + **_params) + self.set_payload(_imagedata) + _encoder(self) + + +_rules = [] + + # Originally from the imghdr module. -def _what(h): - for tf in tests: - if res := tf(h): +def _what(data): + for rule in _rules: + if res := rule(data): return res else: return None -tests = [] -def _test_jpeg(h): +def rule(rulefunc): + _rules.append(rulefunc) + return rulefunc + + + at rule +def _jpeg(h): """JPEG data with JFIF or Exif markers; and raw JPEG""" if h[6:10] in (b'JFIF', b'Exif'): return 'jpeg' elif h[:4] == b'\xff\xd8\xff\xdb': return 'jpeg' -tests.append(_test_jpeg) -def _test_png(h): + at rule +def _png(h): if h.startswith(b'\211PNG\r\n\032\n'): return 'png' -tests.append(_test_png) -def _test_gif(h): + at rule +def _gif(h): """GIF ('87 and '89 variants)""" if h[:6] in (b'GIF87a', b'GIF89a'): return 'gif' -tests.append(_test_gif) -def _test_tiff(h): + at rule +def _tiff(h): """TIFF (can be in Motorola or Intel byte order)""" if h[:2] in (b'MM', b'II'): return 'tiff' -tests.append(_test_tiff) -def _test_rgb(h): + at rule +def _rgb(h): """SGI image library""" if h.startswith(b'\001\332'): return 'rgb' -tests.append(_test_rgb) -def _test_pbm(h): + at rule +def _pbm(h): """PBM (portable bitmap)""" if len(h) >= 3 and \ - h[0] == ord(b'P') and h[1] in b'14' and h[2] in b' \t\n\r': + h[0] == ord(b'P') and h[1] in b'14' and h[2] in b' \t\n\r': return 'pbm' -tests.append(_test_pbm) -def _test_pgm(h): + at rule +def _pgm(h): """PGM (portable graymap)""" if len(h) >= 3 and \ - h[0] == ord(b'P') and h[1] in b'25' and h[2] in b' \t\n\r': + h[0] == ord(b'P') and h[1] in b'25' and h[2] in b' \t\n\r': return 'pgm' -tests.append(_test_pgm) -def _test_ppm(h): + at rule +def _ppm(h): """PPM (portable pixmap)""" if len(h) >= 3 and \ - h[0] == ord(b'P') and h[1] in b'36' and h[2] in b' \t\n\r': + h[0] == ord(b'P') and h[1] in b'36' and h[2] in b' \t\n\r': return 'ppm' -tests.append(_test_ppm) -def _test_rast(h): + at rule +def _rast(h): """Sun raster file""" if h.startswith(b'\x59\xA6\x6A\x95'): return 'rast' -tests.append(_test_rast) -def _test_xbm(h): + at rule +def _xbm(h): """X bitmap (X10 or X11)""" if h.startswith(b'#define '): return 'xbm' -tests.append(_test_xbm) -def _test_bmp(h): + at rule +def _bmp(h): if h.startswith(b'BM'): return 'bmp' -tests.append(_test_bmp) -def _test_webp(h): + at rule +def _webp(h): if h.startswith(b'RIFF') and h[8:12] == b'WEBP': return 'webp' -tests.append(_test_webp) -def _test_exr(h): + at rule +def _exr(h): if h.startswith(b'\x76\x2f\x31\x01'): return 'exr' - -tests.append(_test_exr) - - -class MIMEImage(MIMENonMultipart): - """Class for generating image/* type MIME documents.""" - - def __init__(self, _imagedata, _subtype=None, - _encoder=encoders.encode_base64, *, policy=None, **_params): - """Create an image/* type MIME document. - - _imagedata is a string containing the raw image data. If the data - type can be detected (jpeg, png, gif, tiff, rgb, pbm, pgm, ppm, - rast, xbm, bmp, webp, and exr attempted), then the subtype will be - automatically included in the Content-Type header. Otherwise, you can - specify the specific image subtype via the _subtype parameter. - - _encoder is a function which will perform the actual encoding for - transport of the image data. It takes one argument, which is this - Image instance. It should use get_payload() and set_payload() to - change the payload to the encoded form. It should also add any - Content-Transfer-Encoding or other headers to the message as - necessary. The default encoding is Base64. - - Any additional keyword arguments are passed to the base class - constructor, which turns them into parameters on the Content-Type - header. - """ - if _subtype is None: - if (_subtype := _what(_imagedata)) is None: - raise TypeError('Could not guess image MIME subtype') - MIMENonMultipart.__init__(self, 'image', _subtype, policy=policy, - **_params) - self.set_payload(_imagedata) - _encoder(self) diff --git a/Lib/test/test_email/data/PyBanner048.gif b/Lib/test/test_email/data/PyBanner048.gif deleted file mode 100644 index 7e308f542b864..0000000000000 Binary files a/Lib/test/test_email/data/PyBanner048.gif and /dev/null differ diff --git a/Lib/test/test_email/data/python.bmp b/Lib/test/test_email/data/python.bmp new file mode 100644 index 0000000000000..675f95191a45f Binary files /dev/null and b/Lib/test/test_email/data/python.bmp differ diff --git a/Lib/test/test_email/data/python.exr b/Lib/test/test_email/data/python.exr new file mode 100644 index 0000000000000..773c81ee1fb85 Binary files /dev/null and b/Lib/test/test_email/data/python.exr differ diff --git a/Lib/test/test_email/data/python.gif b/Lib/test/test_email/data/python.gif new file mode 100644 index 0000000000000..efa0be3861d79 Binary files /dev/null and b/Lib/test/test_email/data/python.gif differ diff --git a/Lib/test/test_email/data/python.jpg b/Lib/test/test_email/data/python.jpg new file mode 100644 index 0000000000000..21222c09f5a71 Binary files /dev/null and b/Lib/test/test_email/data/python.jpg differ diff --git a/Lib/test/test_email/data/python.pbm b/Lib/test/test_email/data/python.pbm new file mode 100644 index 0000000000000..1848ba7ff064e --- /dev/null +++ b/Lib/test/test_email/data/python.pbm @@ -0,0 +1,3 @@ +P4 +16 16 +????????[??a_?X????????? \ No newline at end of file diff --git a/Lib/test/test_email/data/python.pgm b/Lib/test/test_email/data/python.pgm new file mode 100644 index 0000000000000..8349f2a53a9be Binary files /dev/null and b/Lib/test/test_email/data/python.pgm differ diff --git a/Lib/test/test_email/data/python.png b/Lib/test/test_email/data/python.png new file mode 100644 index 0000000000000..1a987f79fcd24 Binary files /dev/null and b/Lib/test/test_email/data/python.png differ diff --git a/Lib/test/test_email/data/python.ppm b/Lib/test/test_email/data/python.ppm new file mode 100644 index 0000000000000..7d9cdb3215877 Binary files /dev/null and b/Lib/test/test_email/data/python.ppm differ diff --git a/Lib/test/test_email/data/python.ras b/Lib/test/test_email/data/python.ras new file mode 100644 index 0000000000000..130e96f817ed9 Binary files /dev/null and b/Lib/test/test_email/data/python.ras differ diff --git a/Lib/test/test_email/data/python.sgi b/Lib/test/test_email/data/python.sgi new file mode 100644 index 0000000000000..ffe9081c7a5b6 Binary files /dev/null and b/Lib/test/test_email/data/python.sgi differ diff --git a/Lib/test/test_email/data/python.tiff b/Lib/test/test_email/data/python.tiff new file mode 100644 index 0000000000000..39d0bfcec0253 Binary files /dev/null and b/Lib/test/test_email/data/python.tiff differ diff --git a/Lib/test/test_email/data/python.webp b/Lib/test/test_email/data/python.webp new file mode 100644 index 0000000000000..e824ec7fb1c7f Binary files /dev/null and b/Lib/test/test_email/data/python.webp differ diff --git a/Lib/test/test_email/data/python.xbm b/Lib/test/test_email/data/python.xbm new file mode 100644 index 0000000000000..cfbee2e980621 --- /dev/null +++ b/Lib/test/test_email/data/python.xbm @@ -0,0 +1,6 @@ +#define python_width 16 +#define python_height 16 +static char python_bits[] = { + 0xDF, 0xFE, 0x8F, 0xFD, 0x5F, 0xFB, 0xAB, 0xFE, 0xB5, 0x8D, 0xDA, 0x8F, + 0xA5, 0x86, 0xFA, 0x83, 0x1A, 0x80, 0x0D, 0x80, 0x0D, 0x80, 0x0F, 0xE0, + 0x0F, 0xF8, 0x0F, 0xF8, 0x0F, 0xFC, 0xFF, 0xFF, }; diff --git a/Lib/test/test_email/test_email.py b/Lib/test/test_email/test_email.py index b87dae22de1d2..6ead5947acb6f 100644 --- a/Lib/test/test_email/test_email.py +++ b/Lib/test/test_email/test_email.py @@ -798,7 +798,7 @@ def test_unicode_body_defaults_to_utf8_encoding(self): class TestEncoders(unittest.TestCase): def test_EncodersEncode_base64(self): - with openfile('PyBanner048.gif', 'rb') as fp: + with openfile('python.gif', 'rb') as fp: bindata = fp.read() mimed = email.mime.image.MIMEImage(bindata) base64ed = mimed.get_payload() @@ -1555,24 +1555,44 @@ def test_add_header(self): # Test the basic MIMEImage class class TestMIMEImage(unittest.TestCase): - def setUp(self): - with openfile('PyBanner048.gif', 'rb') as fp: + def _make_image(self, ext): + with openfile(f'python.{ext}', 'rb') as fp: self._imgdata = fp.read() self._im = MIMEImage(self._imgdata) def test_guess_minor_type(self): - self.assertEqual(self._im.get_content_type(), 'image/gif') + for ext, subtype in { + 'bmp': None, + 'exr': None, + 'gif': None, + 'jpg': 'jpeg', + 'pbm': None, + 'pgm': None, + 'png': None, + 'ppm': None, + 'ras': 'rast', + 'sgi': 'rgb', + 'tiff': None, + 'webp': None, + 'xbm': None, + }.items(): + self._make_image(ext) + subtype = ext if subtype is None else subtype + self.assertEqual(self._im.get_content_type(), f'image/{subtype}') def test_encoding(self): + self._make_image('gif') payload = self._im.get_payload() self.assertEqual(base64.decodebytes(bytes(payload, 'ascii')), - self._imgdata) + self._imgdata) def test_checkSetMinor(self): + self._make_image('gif') im = MIMEImage(self._imgdata, 'fish') self.assertEqual(im.get_content_type(), 'image/fish') def test_add_header(self): + self._make_image('gif') eq = self.assertEqual self._im.add_header('Content-Disposition', 'attachment', filename='dingusfish.gif') @@ -1747,7 +1767,7 @@ def test_utf8_input_no_charset(self): # Test complicated multipart/* messages class TestMultipart(TestEmailBase): def setUp(self): - with openfile('PyBanner048.gif', 'rb') as fp: + with openfile('python.gif', 'rb') as fp: data = fp.read() container = MIMEBase('multipart', 'mixed', boundary='BOUNDARY') image = MIMEImage(data, name='dingusfish.gif') @@ -3444,7 +3464,7 @@ def test_BytesGenerator_linend_with_non_ascii(self): def test_mime_classes_policy_argument(self): with openfile('audiotest.au', 'rb') as fp: audiodata = fp.read() - with openfile('PyBanner048.gif', 'rb') as fp: + with openfile('python.gif', 'rb') as fp: bindata = fp.read() classes = [ (MIMEApplication, ('',)), From webhook-mailer at python.org Thu Apr 14 21:07:03 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Fri, 15 Apr 2022 01:07:03 -0000 Subject: [Python-checkins] gh-70979: Fix runpy.run_path parameter name in docs (GH-32265) Message-ID: https://github.com/python/cpython/commit/f1e989b04507db6f0adbccb5e1624d81cb217ea8 commit: f1e989b04507db6f0adbccb5e1624d81cb217ea8 branch: main author: Jelle Zijlstra committer: JelleZijlstra date: 2022-04-14T18:06:58-07:00 summary: gh-70979: Fix runpy.run_path parameter name in docs (GH-32265) Noticed while reviewing GH-30729. files: M Doc/library/runpy.rst diff --git a/Doc/library/runpy.rst b/Doc/library/runpy.rst index af35e81a2d452..26a4f1435214e 100644 --- a/Doc/library/runpy.rst +++ b/Doc/library/runpy.rst @@ -93,7 +93,7 @@ The :mod:`runpy` module provides two functions: run this way, as well as ensuring the real module name is always accessible as ``__spec__.name``. -.. function:: run_path(file_path, init_globals=None, run_name=None) +.. function:: run_path(path_name, init_globals=None, run_name=None) .. index:: module: __main__ @@ -140,7 +140,7 @@ The :mod:`runpy` module provides two functions: A number of alterations are also made to the :mod:`sys` module. Firstly, ``sys.path`` may be altered as described above. ``sys.argv[0]`` is updated - with the value of ``file_path`` and ``sys.modules[__name__]`` is updated + with the value of ``path_name`` and ``sys.modules[__name__]`` is updated with a temporary module object for the module being executed. All modifications to items in :mod:`sys` are reverted before the function returns. From webhook-mailer at python.org Thu Apr 14 21:23:27 2022 From: webhook-mailer at python.org (miss-islington) Date: Fri, 15 Apr 2022 01:23:27 -0000 Subject: [Python-checkins] gh-70979: Fix runpy.run_path parameter name in docs (GH-32265) Message-ID: https://github.com/python/cpython/commit/de7b7565219a24422bf34792239192a57c925281 commit: de7b7565219a24422bf34792239192a57c925281 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-14T18:23:20-07:00 summary: gh-70979: Fix runpy.run_path parameter name in docs (GH-32265) Noticed while reviewing GH-30729. (cherry picked from commit f1e989b04507db6f0adbccb5e1624d81cb217ea8) Co-authored-by: Jelle Zijlstra files: M Doc/library/runpy.rst diff --git a/Doc/library/runpy.rst b/Doc/library/runpy.rst index af35e81a2d452..26a4f1435214e 100644 --- a/Doc/library/runpy.rst +++ b/Doc/library/runpy.rst @@ -93,7 +93,7 @@ The :mod:`runpy` module provides two functions: run this way, as well as ensuring the real module name is always accessible as ``__spec__.name``. -.. function:: run_path(file_path, init_globals=None, run_name=None) +.. function:: run_path(path_name, init_globals=None, run_name=None) .. index:: module: __main__ @@ -140,7 +140,7 @@ The :mod:`runpy` module provides two functions: A number of alterations are also made to the :mod:`sys` module. Firstly, ``sys.path`` may be altered as described above. ``sys.argv[0]`` is updated - with the value of ``file_path`` and ``sys.modules[__name__]`` is updated + with the value of ``path_name`` and ``sys.modules[__name__]`` is updated with a temporary module object for the module being executed. All modifications to items in :mod:`sys` are reverted before the function returns. From webhook-mailer at python.org Thu Apr 14 21:27:53 2022 From: webhook-mailer at python.org (miss-islington) Date: Fri, 15 Apr 2022 01:27:53 -0000 Subject: [Python-checkins] gh-70979: Fix runpy.run_path parameter name in docs (GH-32265) Message-ID: https://github.com/python/cpython/commit/b32d12495cc514f05579520875c95f964afc5b6c commit: b32d12495cc514f05579520875c95f964afc5b6c branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-14T18:27:45-07:00 summary: gh-70979: Fix runpy.run_path parameter name in docs (GH-32265) Noticed while reviewing GH-30729. (cherry picked from commit f1e989b04507db6f0adbccb5e1624d81cb217ea8) Co-authored-by: Jelle Zijlstra files: M Doc/library/runpy.rst diff --git a/Doc/library/runpy.rst b/Doc/library/runpy.rst index af35e81a2d452..26a4f1435214e 100644 --- a/Doc/library/runpy.rst +++ b/Doc/library/runpy.rst @@ -93,7 +93,7 @@ The :mod:`runpy` module provides two functions: run this way, as well as ensuring the real module name is always accessible as ``__spec__.name``. -.. function:: run_path(file_path, init_globals=None, run_name=None) +.. function:: run_path(path_name, init_globals=None, run_name=None) .. index:: module: __main__ @@ -140,7 +140,7 @@ The :mod:`runpy` module provides two functions: A number of alterations are also made to the :mod:`sys` module. Firstly, ``sys.path`` may be altered as described above. ``sys.argv[0]`` is updated - with the value of ``file_path`` and ``sys.modules[__name__]`` is updated + with the value of ``path_name`` and ``sys.modules[__name__]`` is updated with a temporary module object for the module being executed. All modifications to items in :mod:`sys` are reverted before the function returns. From webhook-mailer at python.org Thu Apr 14 21:44:09 2022 From: webhook-mailer at python.org (corona10) Date: Fri, 15 Apr 2022 01:44:09 -0000 Subject: [Python-checkins] Remove usage of _Py_IDENTIFIER from unicodedata module. (GH-91532) Message-ID: https://github.com/python/cpython/commit/2bf5f64455c47eca082696291ba96e15e2f2299c commit: 2bf5f64455c47eca082696291ba96e15e2f2299c branch: main author: Dong-hee Na committer: corona10 date: 2022-04-15T10:44:05+09:00 summary: Remove usage of _Py_IDENTIFIER from unicodedata module. (GH-91532) files: M Modules/unicodedata.c diff --git a/Modules/unicodedata.c b/Modules/unicodedata.c index 7369db53d2e0b..64918115283d1 100644 --- a/Modules/unicodedata.c +++ b/Modules/unicodedata.c @@ -15,7 +15,6 @@ #ifndef Py_BUILD_CORE_BUILTIN # define Py_BUILD_CORE_MODULE 1 #endif -#define NEEDS_PY_IDENTIFIER #define PY_SSIZE_T_CLEAN @@ -25,11 +24,6 @@ #include -_Py_IDENTIFIER(NFC); -_Py_IDENTIFIER(NFD); -_Py_IDENTIFIER(NFKC); -_Py_IDENTIFIER(NFKD); - /*[clinic input] module unicodedata class unicodedata.UCD 'PreviousDBVersion *' '' @@ -890,17 +884,17 @@ unicodedata_UCD_is_normalized_impl(PyObject *self, PyObject *form, PyObject *cmp; int match = 0; - if (_PyUnicode_EqualToASCIIId(form, &PyId_NFC)) { + if (PyUnicode_CompareWithASCIIString(form, "NFC") == 0) { nfc = true; } - else if (_PyUnicode_EqualToASCIIId(form, &PyId_NFKC)) { + else if (PyUnicode_CompareWithASCIIString(form, "NFKC") == 0) { nfc = true; k = true; } - else if (_PyUnicode_EqualToASCIIId(form, &PyId_NFD)) { + else if (PyUnicode_CompareWithASCIIString(form, "NFD") == 0) { /* matches default values for `nfc` and `k` */ } - else if (_PyUnicode_EqualToASCIIId(form, &PyId_NFKD)) { + else if (PyUnicode_CompareWithASCIIString(form, "NFKD") == 0) { k = true; } else { @@ -953,7 +947,7 @@ unicodedata_UCD_normalize_impl(PyObject *self, PyObject *form, return input; } - if (_PyUnicode_EqualToASCIIId(form, &PyId_NFC)) { + if (PyUnicode_CompareWithASCIIString(form, "NFC") == 0) { if (is_normalized_quickcheck(self, input, true, false, true) == YES) { Py_INCREF(input); @@ -961,7 +955,7 @@ unicodedata_UCD_normalize_impl(PyObject *self, PyObject *form, } return nfc_nfkc(self, input, 0); } - if (_PyUnicode_EqualToASCIIId(form, &PyId_NFKC)) { + if (PyUnicode_CompareWithASCIIString(form, "NFKC") == 0) { if (is_normalized_quickcheck(self, input, true, true, true) == YES) { Py_INCREF(input); @@ -969,7 +963,7 @@ unicodedata_UCD_normalize_impl(PyObject *self, PyObject *form, } return nfc_nfkc(self, input, 1); } - if (_PyUnicode_EqualToASCIIId(form, &PyId_NFD)) { + if (PyUnicode_CompareWithASCIIString(form, "NFD") == 0) { if (is_normalized_quickcheck(self, input, false, false, true) == YES) { Py_INCREF(input); @@ -977,7 +971,7 @@ unicodedata_UCD_normalize_impl(PyObject *self, PyObject *form, } return nfd_nfkd(self, input, 0); } - if (_PyUnicode_EqualToASCIIId(form, &PyId_NFKD)) { + if (PyUnicode_CompareWithASCIIString(form, "NFKD") == 0) { if (is_normalized_quickcheck(self, input, false, true, true) == YES) { Py_INCREF(input); From webhook-mailer at python.org Thu Apr 14 23:55:15 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Fri, 15 Apr 2022 03:55:15 -0000 Subject: [Python-checkins] bpo-40376: slightly improved the wording for os.getgrouplist (GH-19702) Message-ID: https://github.com/python/cpython/commit/bd26ef5e9e701d2ab3509a49d9351259a3670772 commit: bd26ef5e9e701d2ab3509a49d9351259a3670772 branch: main author: Jens Holzk?mper committer: JelleZijlstra date: 2022-04-14T20:55:07-07:00 summary: bpo-40376: slightly improved the wording for os.getgrouplist (GH-19702) The documentation for os.getgrouplist potentially read like it returned all groups a user belongs to but it potentially doesn't. Co-authored-by: Jelle Zijlstra files: M Doc/library/os.rst diff --git a/Doc/library/os.rst b/Doc/library/os.rst index 60eac2f78572e..254d6e192cdf0 100644 --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -355,7 +355,8 @@ process and user. Return list of group ids that *user* belongs to. If *group* is not in the list, it is included; typically, *group* is specified as the group ID - field from the password record for *user*. + field from the password record for *user*, because that group ID will + otherwise be potentially omitted. .. availability:: Unix. From webhook-mailer at python.org Fri Apr 15 08:23:19 2022 From: webhook-mailer at python.org (ambv) Date: Fri, 15 Apr 2022 12:23:19 -0000 Subject: [Python-checkins] gh-79156: Add start_tls() method to streams API (#91453) Message-ID: https://github.com/python/cpython/commit/6217864fe5f6855f59d608733ce83fd4466e1b8c commit: 6217864fe5f6855f59d608733ce83fd4466e1b8c branch: main author: Oleg Iarygin committer: ambv date: 2022-04-15T14:23:14+02:00 summary: gh-79156: Add start_tls() method to streams API (#91453) The existing event loop `start_tls()` method is not sufficient for connections using the streams API. The existing StreamReader works because the new transport passes received data to the original protocol. The StreamWriter must then write data to the new transport, and the StreamReaderProtocol must be updated to close the new transport correctly. The new StreamWriter `start_tls()` updates itself and the reader protocol to the new SSL transport. Co-authored-by: Ian Good files: A Misc/NEWS.d/next/Library/2019-05-06-23-36-34.bpo-34975.eb49jr.rst M Doc/library/asyncio-stream.rst M Doc/whatsnew/3.11.rst M Lib/asyncio/streams.py M Lib/test/test_asyncio/test_streams.py diff --git a/Doc/library/asyncio-stream.rst b/Doc/library/asyncio-stream.rst index ba534f9903fb4..72355d356f205 100644 --- a/Doc/library/asyncio-stream.rst +++ b/Doc/library/asyncio-stream.rst @@ -295,6 +295,24 @@ StreamWriter be resumed. When there is nothing to wait for, the :meth:`drain` returns immediately. + .. coroutinemethod:: start_tls(sslcontext, \*, server_hostname=None, \ + ssl_handshake_timeout=None) + + Upgrade an existing stream-based connection to TLS. + + Parameters: + + * *sslcontext*: a configured instance of :class:`~ssl.SSLContext`. + + * *server_hostname*: sets or overrides the host name that the target + server's certificate will be matched against. + + * *ssl_handshake_timeout* is the time in seconds to wait for the TLS + handshake to complete before aborting the connection. ``60.0`` seconds + if ``None`` (default). + + .. versionadded:: 3.8 + .. method:: is_closing() Return ``True`` if the stream is closed or in the process of diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index dba554cc834ec..9f7f6f52a8e9e 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -246,6 +246,10 @@ asyncio :meth:`~asyncio.AbstractEventLoop.sock_recvfrom_into`. (Contributed by Alex Gr?nholm in :issue:`46805`.) +* Add :meth:`~asyncio.streams.StreamWriter.start_tls` method for upgrading + existing stream-based connections to TLS. (Contributed by Ian Good in + :issue:`34975`.) + fractions --------- diff --git a/Lib/asyncio/streams.py b/Lib/asyncio/streams.py index 080d8a62cde1e..a568c4e4b295f 100644 --- a/Lib/asyncio/streams.py +++ b/Lib/asyncio/streams.py @@ -217,6 +217,13 @@ def _stream_reader(self): return None return self._stream_reader_wr() + def _replace_writer(self, writer): + loop = self._loop + transport = writer.transport + self._stream_writer = writer + self._transport = transport + self._over_ssl = transport.get_extra_info('sslcontext') is not None + def connection_made(self, transport): if self._reject_connection: context = { @@ -371,6 +378,20 @@ async def drain(self): await sleep(0) await self._protocol._drain_helper() + async def start_tls(self, sslcontext, *, + server_hostname=None, + ssl_handshake_timeout=None): + """Upgrade an existing stream-based connection to TLS.""" + server_side = self._protocol._client_connected_cb is not None + protocol = self._protocol + await self.drain() + new_transport = await self._loop.start_tls( # type: ignore + self._transport, protocol, sslcontext, + server_side=server_side, server_hostname=server_hostname, + ssl_handshake_timeout=ssl_handshake_timeout) + self._transport = new_transport + protocol._replace_writer(self) + class StreamReader: diff --git a/Lib/test/test_asyncio/test_streams.py b/Lib/test/test_asyncio/test_streams.py index 227b2279e172c..a7d17894e1c52 100644 --- a/Lib/test/test_asyncio/test_streams.py +++ b/Lib/test/test_asyncio/test_streams.py @@ -706,6 +706,69 @@ async def client(path): self.assertEqual(messages, []) + @unittest.skipIf(ssl is None, 'No ssl module') + def test_start_tls(self): + + class MyServer: + + def __init__(self, loop): + self.server = None + self.loop = loop + + async def handle_client(self, client_reader, client_writer): + data1 = await client_reader.readline() + client_writer.write(data1) + await client_writer.drain() + assert client_writer.get_extra_info('sslcontext') is None + await client_writer.start_tls( + test_utils.simple_server_sslcontext()) + assert client_writer.get_extra_info('sslcontext') is not None + data2 = await client_reader.readline() + client_writer.write(data2) + await client_writer.drain() + client_writer.close() + await client_writer.wait_closed() + + def start(self): + sock = socket.create_server(('127.0.0.1', 0)) + self.server = self.loop.run_until_complete( + asyncio.start_server(self.handle_client, + sock=sock)) + return sock.getsockname() + + def stop(self): + if self.server is not None: + self.server.close() + self.loop.run_until_complete(self.server.wait_closed()) + self.server = None + + async def client(addr): + reader, writer = await asyncio.open_connection(*addr) + writer.write(b"hello world 1!\n") + await writer.drain() + msgback1 = await reader.readline() + assert writer.get_extra_info('sslcontext') is None + await writer.start_tls(test_utils.simple_client_sslcontext()) + assert writer.get_extra_info('sslcontext') is not None + writer.write(b"hello world 2!\n") + await writer.drain() + msgback2 = await reader.readline() + writer.close() + await writer.wait_closed() + return msgback1, msgback2 + + messages = [] + self.loop.set_exception_handler(lambda loop, ctx: messages.append(ctx)) + + server = MyServer(self.loop) + addr = server.start() + msg1, msg2 = self.loop.run_until_complete(client(addr)) + server.stop() + + self.assertEqual(messages, []) + self.assertEqual(msg1, b"hello world 1!\n") + self.assertEqual(msg2, b"hello world 2!\n") + @unittest.skipIf(sys.platform == 'win32', "Don't have pipes") def test_read_all_from_pipe_reader(self): # See asyncio issue 168. This test is derived from the example diff --git a/Misc/NEWS.d/next/Library/2019-05-06-23-36-34.bpo-34975.eb49jr.rst b/Misc/NEWS.d/next/Library/2019-05-06-23-36-34.bpo-34975.eb49jr.rst new file mode 100644 index 0000000000000..1576269da99ee --- /dev/null +++ b/Misc/NEWS.d/next/Library/2019-05-06-23-36-34.bpo-34975.eb49jr.rst @@ -0,0 +1,3 @@ +Adds a ``start_tls()`` method to :class:`~asyncio.streams.StreamWriter`, +which upgrades the connection with TLS using the given +:class:`~ssl.SSLContext`. From webhook-mailer at python.org Fri Apr 15 09:08:15 2022 From: webhook-mailer at python.org (ambv) Date: Fri, 15 Apr 2022 13:08:15 -0000 Subject: [Python-checkins] bpo-40376: slightly improved the wording for os.getgrouplist (GH-19702) (GH-91557) Message-ID: https://github.com/python/cpython/commit/2c034700f88367df995a1e7bdc9668d99c4986b9 commit: 2c034700f88367df995a1e7bdc9668d99c4986b9 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2022-04-15T15:08:11+02:00 summary: bpo-40376: slightly improved the wording for os.getgrouplist (GH-19702) (GH-91557) The documentation for os.getgrouplist potentially read like it returned all groups a user belongs to but it potentially doesn't. Co-authored-by: Jelle Zijlstra (cherry picked from commit bd26ef5e9e701d2ab3509a49d9351259a3670772) Co-authored-by: Jens Holzk?mper files: M Doc/library/os.rst diff --git a/Doc/library/os.rst b/Doc/library/os.rst index 2f7b37019e4a4..9dc49a2034144 100644 --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -287,7 +287,8 @@ process and user. Return list of group ids that *user* belongs to. If *group* is not in the list, it is included; typically, *group* is specified as the group ID - field from the password record for *user*. + field from the password record for *user*, because that group ID will + otherwise be potentially omitted. .. availability:: Unix. From webhook-mailer at python.org Fri Apr 15 09:30:05 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Fri, 15 Apr 2022 13:30:05 -0000 Subject: [Python-checkins] gh-69093: Expose sqlite3.Blob as a class (GH-91550) Message-ID: https://github.com/python/cpython/commit/b7f83bdd0e16f82288dc3557b7e715bb5c7d96d0 commit: b7f83bdd0e16f82288dc3557b7e715bb5c7d96d0 branch: main author: Jelle Zijlstra committer: JelleZijlstra date: 2022-04-15T06:29:57-07:00 summary: gh-69093: Expose sqlite3.Blob as a class (GH-91550) I noticed this was missing while writing typeshed stubs. It's useful to expose it for use in annotations and for exploration. files: M Lib/test/test_sqlite3/test_dbapi.py M Modules/_sqlite/module.c diff --git a/Lib/test/test_sqlite3/test_dbapi.py b/Lib/test/test_sqlite3/test_dbapi.py index faaa3713cb510..6613d5f0ea4bd 100644 --- a/Lib/test/test_sqlite3/test_dbapi.py +++ b/Lib/test/test_sqlite3/test_dbapi.py @@ -1055,6 +1055,9 @@ def tearDown(self): self.blob.close() self.cx.close() + def test_blob_is_a_blob(self): + self.assertIsInstance(self.blob, sqlite.Blob) + def test_blob_seek_and_tell(self): self.blob.seek(10) self.assertEqual(self.blob.tell(), 10) diff --git a/Modules/_sqlite/module.c b/Modules/_sqlite/module.c index d355c2be37a2a..fbc57c7cc739e 100644 --- a/Modules/_sqlite/module.c +++ b/Modules/_sqlite/module.c @@ -697,6 +697,7 @@ module_exec(PyObject *module) } pysqlite_state *state = pysqlite_get_state(module); + ADD_TYPE(module, state->BlobType); ADD_TYPE(module, state->ConnectionType); ADD_TYPE(module, state->CursorType); ADD_TYPE(module, state->PrepareProtocolType); From webhook-mailer at python.org Fri Apr 15 09:55:17 2022 From: webhook-mailer at python.org (ambv) Date: Fri, 15 Apr 2022 13:55:17 -0000 Subject: [Python-checkins] gh-88513: clarify shutil.copytree's dirs_exist_ok arg (GH-91434) (GH-91464) Message-ID: https://github.com/python/cpython/commit/289f27d06b0d19739b114f6b2f60a6264655b8f6 commit: 289f27d06b0d19739b114f6b2f60a6264655b8f6 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2022-04-15T15:55:13+02:00 summary: gh-88513: clarify shutil.copytree's dirs_exist_ok arg (GH-91434) (GH-91464) * add a paragraph to document this kwarg in detail * update docstring in the source accordingly (cherry picked from commit f33e2c87a83917b5139d97fd8ef7cba7223ebef5) Co-authored-by: Jelle Zijlstra files: A Misc/NEWS.d/next/Documentation/2022-04-10-20-28-20.bpo-44347.Q1m3DM.rst M Doc/library/shutil.rst M Lib/shutil.py diff --git a/Doc/library/shutil.rst b/Doc/library/shutil.rst index 5f71049f918e0..8ce0e80ec8120 100644 --- a/Doc/library/shutil.rst +++ b/Doc/library/shutil.rst @@ -230,9 +230,8 @@ Directory and files operations dirs_exist_ok=False) Recursively copy an entire directory tree rooted at *src* to a directory - named *dst* and return the destination directory. *dirs_exist_ok* dictates - whether to raise an exception in case *dst* or any missing parent directory - already exists. + named *dst* and return the destination directory. All intermediate + directories needed to contain *dst* will also be created by default. Permissions and times of directories are copied with :func:`copystat`, individual files are copied using :func:`~shutil.copy2`. @@ -263,8 +262,14 @@ Directory and files operations If *copy_function* is given, it must be a callable that will be used to copy each file. It will be called with the source path and the destination path - ? as arguments. By default, :func:`~shutil.copy2` is used, but any function - ? that supports the same signature (like :func:`~shutil.copy`) can be used. + as arguments. By default, :func:`~shutil.copy2` is used, but any function + that supports the same signature (like :func:`~shutil.copy`) can be used. + + If *dirs_exist_ok* is false (the default) and *dst* already exists, a + :exc:`FileExistsError` is raised. If *dirs_exist_ok* is true, the copying + operation will continue if it encounters existing directories, and files + within the *dst* tree will be overwritten by corresponding files from the + *src* tree. .. audit-event:: shutil.copytree src,dst shutil.copytree @@ -275,7 +280,7 @@ Directory and files operations .. versionchanged:: 3.2 Added the *copy_function* argument to be able to provide a custom copy function. - Added the *ignore_dangling_symlinks* argument to silent dangling symlinks + Added the *ignore_dangling_symlinks* argument to silence dangling symlinks errors when *symlinks* is false. .. versionchanged:: 3.8 diff --git a/Lib/shutil.py b/Lib/shutil.py index 37bf98df79672..2768bcf6ab815 100644 --- a/Lib/shutil.py +++ b/Lib/shutil.py @@ -516,9 +516,6 @@ def copytree(src, dst, symlinks=False, ignore=None, copy_function=copy2, ignore_dangling_symlinks=False, dirs_exist_ok=False): """Recursively copy a directory tree and return the destination directory. - dirs_exist_ok dictates whether to raise an exception in case dst or any - missing parent directory already exists. - If exception(s) occur, an Error is raised with a list of reasons. If the optional symlinks flag is true, symbolic links in the @@ -549,6 +546,11 @@ def copytree(src, dst, symlinks=False, ignore=None, copy_function=copy2, destination path as arguments. By default, copy2() is used, but any function that supports the same signature (like copy()) can be used. + If dirs_exist_ok is false (the default) and `dst` already exists, a + `FileExistsError` is raised. If `dirs_exist_ok` is true, the copying + operation will continue if it encounters existing directories, and files + within the `dst` tree will be overwritten by corresponding files from the + `src` tree. """ sys.audit("shutil.copytree", src, dst) with os.scandir(src) as itr: diff --git a/Misc/NEWS.d/next/Documentation/2022-04-10-20-28-20.bpo-44347.Q1m3DM.rst b/Misc/NEWS.d/next/Documentation/2022-04-10-20-28-20.bpo-44347.Q1m3DM.rst new file mode 100644 index 0000000000000..27aa5742cd008 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2022-04-10-20-28-20.bpo-44347.Q1m3DM.rst @@ -0,0 +1 @@ +Clarify the meaning of *dirs_exist_ok*, a kwarg of :func:`shutil.copytree`. From webhook-mailer at python.org Fri Apr 15 10:21:43 2022 From: webhook-mailer at python.org (corona10) Date: Fri, 15 Apr 2022 14:21:43 -0000 Subject: [Python-checkins] gh-90699: Use _Py_STR(empty) instead of PyUnicode_New(0, 0) for BUILD_STRING (GH-91476) Message-ID: https://github.com/python/cpython/commit/72965981d1128b3923dad5e850c8cff626ae4dc7 commit: 72965981d1128b3923dad5e850c8cff626ae4dc7 branch: main author: Dennis Sweeney <36520290+sweeneyde at users.noreply.github.com> committer: corona10 date: 2022-04-15T23:21:35+09:00 summary: gh-90699: Use _Py_STR(empty) instead of PyUnicode_New(0, 0) for BUILD_STRING (GH-91476) files: M Python/ceval.c diff --git a/Python/ceval.c b/Python/ceval.c index b46b1ef842939..7891547ffc687 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -3152,12 +3152,8 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int TARGET(BUILD_STRING) { PyObject *str; - PyObject *empty = PyUnicode_New(0, 0); - if (empty == NULL) { - goto error; - } - str = _PyUnicode_JoinArray(empty, stack_pointer - oparg, oparg); - Py_DECREF(empty); + str = _PyUnicode_JoinArray(&_Py_STR(empty), + stack_pointer - oparg, oparg); if (str == NULL) goto error; while (--oparg >= 0) { From webhook-mailer at python.org Fri Apr 15 12:25:12 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Fri, 15 Apr 2022 16:25:12 -0000 Subject: [Python-checkins] gh-69093: Don't allow instantiation of sqlite3.Blob objects (GH-91570) Message-ID: https://github.com/python/cpython/commit/d104f4d21f735693ea93fe65ea4b4e1aa1779343 commit: d104f4d21f735693ea93fe65ea4b4e1aa1779343 branch: main author: Erlend Egeberg Aasland committer: JelleZijlstra date: 2022-04-15T09:25:03-07:00 summary: gh-69093: Don't allow instantiation of sqlite3.Blob objects (GH-91570) files: M Lib/test/test_sqlite3/test_dbapi.py M Modules/_sqlite/blob.c diff --git a/Lib/test/test_sqlite3/test_dbapi.py b/Lib/test/test_sqlite3/test_dbapi.py index 6613d5f0ea4bd..b010813fff7c5 100644 --- a/Lib/test/test_sqlite3/test_dbapi.py +++ b/Lib/test/test_sqlite3/test_dbapi.py @@ -356,6 +356,7 @@ def test_shared_cache_deprecated(self): def test_disallow_instantiation(self): cx = sqlite.connect(":memory:") check_disallow_instantiation(self, type(cx("select 1"))) + check_disallow_instantiation(self, sqlite.Blob) def test_complete_statement(self): self.assertFalse(sqlite.complete_statement("select t")) diff --git a/Modules/_sqlite/blob.c b/Modules/_sqlite/blob.c index 821295cee813f..c4f8be45b2f94 100644 --- a/Modules/_sqlite/blob.c +++ b/Modules/_sqlite/blob.c @@ -334,7 +334,7 @@ static PyType_Spec blob_spec = { .name = MODULE_NAME ".Blob", .basicsize = sizeof(pysqlite_Blob), .flags = (Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC | - Py_TPFLAGS_IMMUTABLETYPE), + Py_TPFLAGS_IMMUTABLETYPE | Py_TPFLAGS_DISALLOW_INSTANTIATION), .slots = blob_slots, }; From webhook-mailer at python.org Fri Apr 15 12:26:49 2022 From: webhook-mailer at python.org (brandtbucher) Date: Fri, 15 Apr 2022 16:26:49 -0000 Subject: [Python-checkins] gh-91404: Use computed gotos and reduce indirection in re (#91495) Message-ID: https://github.com/python/cpython/commit/1b34b5687b20a54cff2158c8660201e7377dec21 commit: 1b34b5687b20a54cff2158c8660201e7377dec21 branch: main author: Brandt Bucher committer: brandtbucher date: 2022-04-15T09:26:44-07:00 summary: gh-91404: Use computed gotos and reduce indirection in re (#91495) files: A Misc/NEWS.d/next/Library/2022-04-12-19-08-13.gh-issue-91404.zjqYHo.rst A Modules/_sre/sre_targets.h M Doc/whatsnew/3.11.rst M Makefile.pre.in M Modules/_sre/sre_lib.h M Tools/scripts/generate_sre_constants.py diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index 9f7f6f52a8e9e..a5a52682b503c 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -520,6 +520,12 @@ Optimizations becomes 272 bytes from 352 bytes on 64bit platform. (Contributed by Inada Naoki in :issue:`46845`.) +* :mod:`re`'s regular expression matching engine has been partially refactored, + and now uses computed gotos (or "threaded code") on supported platforms. As a + result, Python 3.11 executes the `pyperformance regular expression benchmarks + `_ up to 10% + faster than Python 3.10. + Faster CPython ============== diff --git a/Makefile.pre.in b/Makefile.pre.in index d89886ddc76f1..f6c8c72bbc2c3 100644 --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -1351,11 +1351,12 @@ regen-stdlib-module-names: build_all Programs/_testembed $(UPDATE_FILE) $(srcdir)/Python/stdlib_module_names.h $(srcdir)/Python/stdlib_module_names.h.new regen-sre: - # Regenerate Modules/_sre/sre_constants.h from Lib/re/_constants.py - # using Tools/scripts/generate_sre_constants.py + # Regenerate Modules/_sre/sre_constants.h and Modules/_sre/sre_targets.h + # from Lib/re/_constants.py using Tools/scripts/generate_sre_constants.py $(PYTHON_FOR_REGEN) $(srcdir)/Tools/scripts/generate_sre_constants.py \ $(srcdir)/Lib/re/_constants.py \ - $(srcdir)/Modules/_sre/sre_constants.h + $(srcdir)/Modules/_sre/sre_constants.h \ + $(srcdir)/Modules/_sre/sre_targets.h Python/compile.o Python/symtable.o Python/ast_unparse.o Python/ast.o Python/future.o: $(srcdir)/Include/internal/pycore_ast.h diff --git a/Misc/NEWS.d/next/Library/2022-04-12-19-08-13.gh-issue-91404.zjqYHo.rst b/Misc/NEWS.d/next/Library/2022-04-12-19-08-13.gh-issue-91404.zjqYHo.rst new file mode 100644 index 0000000000000..58464fceeba5a --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-12-19-08-13.gh-issue-91404.zjqYHo.rst @@ -0,0 +1,3 @@ +Improve the performance of :mod:`re` matching by using computed gotos (or +"threaded code") on supported platforms and removing expensive pointer +indirections. diff --git a/Modules/_sre/sre_lib.h b/Modules/_sre/sre_lib.h index 34cd0552532f7..3472e65b87ae6 100644 --- a/Modules/_sre/sre_lib.h +++ b/Modules/_sre/sre_lib.h @@ -485,16 +485,20 @@ do { \ #define JUMP_ATOMIC_GROUP 16 #define DO_JUMPX(jumpvalue, jumplabel, nextpattern, toplevel_) \ + ctx->pattern = pattern; \ + ctx->ptr = ptr; \ DATA_ALLOC(SRE(match_context), nextctx); \ nextctx->last_ctx_pos = ctx_pos; \ nextctx->jump = jumpvalue; \ nextctx->pattern = nextpattern; \ nextctx->toplevel = toplevel_; \ + pattern = nextpattern; \ ctx_pos = alloc_pos; \ ctx = nextctx; \ goto entrance; \ jumplabel: \ - while (0) /* gcc doesn't like labels at end of scopes */ \ + pattern = ctx->pattern; \ + ptr = ctx->ptr; #define DO_JUMP(jumpvalue, jumplabel, nextpattern) \ DO_JUMPX(jumpvalue, jumplabel, nextpattern, ctx->toplevel) @@ -517,6 +521,36 @@ typedef struct { int toplevel; } SRE(match_context); +#define MAYBE_CHECK_SIGNALS \ + do { \ + if ((0 == (++sigcount & 0xfff)) && PyErr_CheckSignals()) { \ + RETURN_ERROR(SRE_ERROR_INTERRUPTED); \ + } \ + } while (0) + +#ifdef HAVE_COMPUTED_GOTOS + #ifndef USE_COMPUTED_GOTOS + #define USE_COMPUTED_GOTOS 1 + #endif +#elif defined(USE_COMPUTED_GOTOS) && USE_COMPUTED_GOTOS + #error "Computed gotos are not supported on this compiler." +#else + #undef USE_COMPUTED_GOTOS + #define USE_COMPUTED_GOTOS 0 +#endif + +#if USE_COMPUTED_GOTOS + #define TARGET(OP) TARGET_ ## OP + #define DISPATCH \ + do { \ + MAYBE_CHECK_SIGNALS; \ + goto *sre_targets[*pattern++]; \ + } while (0) +#else + #define TARGET(OP) case OP + #define DISPATCH goto dispatch +#endif + /* check if string matches the given pattern. returns <0 for error, 0 for failure, and 1 for success */ LOCAL(Py_ssize_t) @@ -536,38 +570,44 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) DATA_ALLOC(SRE(match_context), ctx); ctx->last_ctx_pos = -1; ctx->jump = JUMP_NONE; - ctx->pattern = pattern; ctx->toplevel = toplevel; ctx_pos = alloc_pos; +#if USE_COMPUTED_GOTOS +#include "sre_targets.h" +#endif + entrance: - ctx->ptr = (SRE_CHAR *)state->ptr; + ; // Fashion statement. + const SRE_CHAR *ptr = (SRE_CHAR *)state->ptr; - if (ctx->pattern[0] == SRE_OP_INFO) { + if (pattern[0] == SRE_OP_INFO) { /* optimization info block */ /* <1=skip> <2=flags> <3=min> ... */ - if (ctx->pattern[3] && (uintptr_t)(end - ctx->ptr) < ctx->pattern[3]) { + if (pattern[3] && (uintptr_t)(end - ptr) < pattern[3]) { TRACE(("reject (got %zd chars, need %zd)\n", - end - ctx->ptr, (Py_ssize_t) ctx->pattern[3])); + end - ptr, (Py_ssize_t) pattern[3])); RETURN_FAILURE; } - ctx->pattern += ctx->pattern[1] + 1; + pattern += pattern[1] + 1; } - for (;;) { - ++sigcount; - if ((0 == (sigcount & 0xfff)) && PyErr_CheckSignals()) - RETURN_ERROR(SRE_ERROR_INTERRUPTED); - - switch (*ctx->pattern++) { +#if USE_COMPUTED_GOTOS + DISPATCH; +#else +dispatch: + MAYBE_CHECK_SIGNALS; + switch (*pattern++) +#endif + { - case SRE_OP_MARK: + TARGET(SRE_OP_MARK): /* set mark */ /* */ - TRACE(("|%p|%p|MARK %d\n", ctx->pattern, - ctx->ptr, ctx->pattern[0])); - i = ctx->pattern[0]; + TRACE(("|%p|%p|MARK %d\n", pattern, + ptr, pattern[0])); + i = pattern[0]; if (i & 1) state->lastindex = i/2 + 1; if (i > state->lastmark) { @@ -580,210 +620,210 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) state->mark[j++] = NULL; state->lastmark = i; } - state->mark[i] = ctx->ptr; - ctx->pattern++; - break; + state->mark[i] = ptr; + pattern++; + DISPATCH; - case SRE_OP_LITERAL: + TARGET(SRE_OP_LITERAL): /* match literal string */ /* */ - TRACE(("|%p|%p|LITERAL %d\n", ctx->pattern, - ctx->ptr, *ctx->pattern)); - if (ctx->ptr >= end || (SRE_CODE) ctx->ptr[0] != ctx->pattern[0]) + TRACE(("|%p|%p|LITERAL %d\n", pattern, + ptr, *pattern)); + if (ptr >= end || (SRE_CODE) ptr[0] != pattern[0]) RETURN_FAILURE; - ctx->pattern++; - ctx->ptr++; - break; + pattern++; + ptr++; + DISPATCH; - case SRE_OP_NOT_LITERAL: + TARGET(SRE_OP_NOT_LITERAL): /* match anything that is not literal character */ /* */ - TRACE(("|%p|%p|NOT_LITERAL %d\n", ctx->pattern, - ctx->ptr, *ctx->pattern)); - if (ctx->ptr >= end || (SRE_CODE) ctx->ptr[0] == ctx->pattern[0]) + TRACE(("|%p|%p|NOT_LITERAL %d\n", pattern, + ptr, *pattern)); + if (ptr >= end || (SRE_CODE) ptr[0] == pattern[0]) RETURN_FAILURE; - ctx->pattern++; - ctx->ptr++; - break; + pattern++; + ptr++; + DISPATCH; - case SRE_OP_SUCCESS: + TARGET(SRE_OP_SUCCESS): /* end of pattern */ - TRACE(("|%p|%p|SUCCESS\n", ctx->pattern, ctx->ptr)); + TRACE(("|%p|%p|SUCCESS\n", pattern, ptr)); if (ctx->toplevel && - ((state->match_all && ctx->ptr != state->end) || - (state->must_advance && ctx->ptr == state->start))) + ((state->match_all && ptr != state->end) || + (state->must_advance && ptr == state->start))) { RETURN_FAILURE; } - state->ptr = ctx->ptr; + state->ptr = ptr; RETURN_SUCCESS; - case SRE_OP_AT: + TARGET(SRE_OP_AT): /* match at given position */ /* */ - TRACE(("|%p|%p|AT %d\n", ctx->pattern, ctx->ptr, *ctx->pattern)); - if (!SRE(at)(state, ctx->ptr, *ctx->pattern)) + TRACE(("|%p|%p|AT %d\n", pattern, ptr, *pattern)); + if (!SRE(at)(state, ptr, *pattern)) RETURN_FAILURE; - ctx->pattern++; - break; + pattern++; + DISPATCH; - case SRE_OP_CATEGORY: + TARGET(SRE_OP_CATEGORY): /* match at given category */ /* */ - TRACE(("|%p|%p|CATEGORY %d\n", ctx->pattern, - ctx->ptr, *ctx->pattern)); - if (ctx->ptr >= end || !sre_category(ctx->pattern[0], ctx->ptr[0])) + TRACE(("|%p|%p|CATEGORY %d\n", pattern, + ptr, *pattern)); + if (ptr >= end || !sre_category(pattern[0], ptr[0])) RETURN_FAILURE; - ctx->pattern++; - ctx->ptr++; - break; + pattern++; + ptr++; + DISPATCH; - case SRE_OP_ANY: + TARGET(SRE_OP_ANY): /* match anything (except a newline) */ /* */ - TRACE(("|%p|%p|ANY\n", ctx->pattern, ctx->ptr)); - if (ctx->ptr >= end || SRE_IS_LINEBREAK(ctx->ptr[0])) + TRACE(("|%p|%p|ANY\n", pattern, ptr)); + if (ptr >= end || SRE_IS_LINEBREAK(ptr[0])) RETURN_FAILURE; - ctx->ptr++; - break; + ptr++; + DISPATCH; - case SRE_OP_ANY_ALL: + TARGET(SRE_OP_ANY_ALL): /* match anything */ /* */ - TRACE(("|%p|%p|ANY_ALL\n", ctx->pattern, ctx->ptr)); - if (ctx->ptr >= end) + TRACE(("|%p|%p|ANY_ALL\n", pattern, ptr)); + if (ptr >= end) RETURN_FAILURE; - ctx->ptr++; - break; + ptr++; + DISPATCH; - case SRE_OP_IN: + TARGET(SRE_OP_IN): /* match set member (or non_member) */ /* */ - TRACE(("|%p|%p|IN\n", ctx->pattern, ctx->ptr)); - if (ctx->ptr >= end || - !SRE(charset)(state, ctx->pattern + 1, *ctx->ptr)) + TRACE(("|%p|%p|IN\n", pattern, ptr)); + if (ptr >= end || + !SRE(charset)(state, pattern + 1, *ptr)) RETURN_FAILURE; - ctx->pattern += ctx->pattern[0]; - ctx->ptr++; - break; + pattern += pattern[0]; + ptr++; + DISPATCH; - case SRE_OP_LITERAL_IGNORE: + TARGET(SRE_OP_LITERAL_IGNORE): TRACE(("|%p|%p|LITERAL_IGNORE %d\n", - ctx->pattern, ctx->ptr, ctx->pattern[0])); - if (ctx->ptr >= end || - sre_lower_ascii(*ctx->ptr) != *ctx->pattern) + pattern, ptr, pattern[0])); + if (ptr >= end || + sre_lower_ascii(*ptr) != *pattern) RETURN_FAILURE; - ctx->pattern++; - ctx->ptr++; - break; + pattern++; + ptr++; + DISPATCH; - case SRE_OP_LITERAL_UNI_IGNORE: + TARGET(SRE_OP_LITERAL_UNI_IGNORE): TRACE(("|%p|%p|LITERAL_UNI_IGNORE %d\n", - ctx->pattern, ctx->ptr, ctx->pattern[0])); - if (ctx->ptr >= end || - sre_lower_unicode(*ctx->ptr) != *ctx->pattern) + pattern, ptr, pattern[0])); + if (ptr >= end || + sre_lower_unicode(*ptr) != *pattern) RETURN_FAILURE; - ctx->pattern++; - ctx->ptr++; - break; + pattern++; + ptr++; + DISPATCH; - case SRE_OP_LITERAL_LOC_IGNORE: + TARGET(SRE_OP_LITERAL_LOC_IGNORE): TRACE(("|%p|%p|LITERAL_LOC_IGNORE %d\n", - ctx->pattern, ctx->ptr, ctx->pattern[0])); - if (ctx->ptr >= end - || !char_loc_ignore(*ctx->pattern, *ctx->ptr)) + pattern, ptr, pattern[0])); + if (ptr >= end + || !char_loc_ignore(*pattern, *ptr)) RETURN_FAILURE; - ctx->pattern++; - ctx->ptr++; - break; + pattern++; + ptr++; + DISPATCH; - case SRE_OP_NOT_LITERAL_IGNORE: + TARGET(SRE_OP_NOT_LITERAL_IGNORE): TRACE(("|%p|%p|NOT_LITERAL_IGNORE %d\n", - ctx->pattern, ctx->ptr, *ctx->pattern)); - if (ctx->ptr >= end || - sre_lower_ascii(*ctx->ptr) == *ctx->pattern) + pattern, ptr, *pattern)); + if (ptr >= end || + sre_lower_ascii(*ptr) == *pattern) RETURN_FAILURE; - ctx->pattern++; - ctx->ptr++; - break; + pattern++; + ptr++; + DISPATCH; - case SRE_OP_NOT_LITERAL_UNI_IGNORE: + TARGET(SRE_OP_NOT_LITERAL_UNI_IGNORE): TRACE(("|%p|%p|NOT_LITERAL_UNI_IGNORE %d\n", - ctx->pattern, ctx->ptr, *ctx->pattern)); - if (ctx->ptr >= end || - sre_lower_unicode(*ctx->ptr) == *ctx->pattern) + pattern, ptr, *pattern)); + if (ptr >= end || + sre_lower_unicode(*ptr) == *pattern) RETURN_FAILURE; - ctx->pattern++; - ctx->ptr++; - break; + pattern++; + ptr++; + DISPATCH; - case SRE_OP_NOT_LITERAL_LOC_IGNORE: + TARGET(SRE_OP_NOT_LITERAL_LOC_IGNORE): TRACE(("|%p|%p|NOT_LITERAL_LOC_IGNORE %d\n", - ctx->pattern, ctx->ptr, *ctx->pattern)); - if (ctx->ptr >= end - || char_loc_ignore(*ctx->pattern, *ctx->ptr)) + pattern, ptr, *pattern)); + if (ptr >= end + || char_loc_ignore(*pattern, *ptr)) RETURN_FAILURE; - ctx->pattern++; - ctx->ptr++; - break; + pattern++; + ptr++; + DISPATCH; - case SRE_OP_IN_IGNORE: - TRACE(("|%p|%p|IN_IGNORE\n", ctx->pattern, ctx->ptr)); - if (ctx->ptr >= end - || !SRE(charset)(state, ctx->pattern+1, - (SRE_CODE)sre_lower_ascii(*ctx->ptr))) + TARGET(SRE_OP_IN_IGNORE): + TRACE(("|%p|%p|IN_IGNORE\n", pattern, ptr)); + if (ptr >= end + || !SRE(charset)(state, pattern+1, + (SRE_CODE)sre_lower_ascii(*ptr))) RETURN_FAILURE; - ctx->pattern += ctx->pattern[0]; - ctx->ptr++; - break; + pattern += pattern[0]; + ptr++; + DISPATCH; - case SRE_OP_IN_UNI_IGNORE: - TRACE(("|%p|%p|IN_UNI_IGNORE\n", ctx->pattern, ctx->ptr)); - if (ctx->ptr >= end - || !SRE(charset)(state, ctx->pattern+1, - (SRE_CODE)sre_lower_unicode(*ctx->ptr))) + TARGET(SRE_OP_IN_UNI_IGNORE): + TRACE(("|%p|%p|IN_UNI_IGNORE\n", pattern, ptr)); + if (ptr >= end + || !SRE(charset)(state, pattern+1, + (SRE_CODE)sre_lower_unicode(*ptr))) RETURN_FAILURE; - ctx->pattern += ctx->pattern[0]; - ctx->ptr++; - break; + pattern += pattern[0]; + ptr++; + DISPATCH; - case SRE_OP_IN_LOC_IGNORE: - TRACE(("|%p|%p|IN_LOC_IGNORE\n", ctx->pattern, ctx->ptr)); - if (ctx->ptr >= end - || !SRE(charset_loc_ignore)(state, ctx->pattern+1, *ctx->ptr)) + TARGET(SRE_OP_IN_LOC_IGNORE): + TRACE(("|%p|%p|IN_LOC_IGNORE\n", pattern, ptr)); + if (ptr >= end + || !SRE(charset_loc_ignore)(state, pattern+1, *ptr)) RETURN_FAILURE; - ctx->pattern += ctx->pattern[0]; - ctx->ptr++; - break; + pattern += pattern[0]; + ptr++; + DISPATCH; - case SRE_OP_JUMP: - case SRE_OP_INFO: + TARGET(SRE_OP_JUMP): + TARGET(SRE_OP_INFO): /* jump forward */ /* */ - TRACE(("|%p|%p|JUMP %d\n", ctx->pattern, - ctx->ptr, ctx->pattern[0])); - ctx->pattern += ctx->pattern[0]; - break; + TRACE(("|%p|%p|JUMP %d\n", pattern, + ptr, pattern[0])); + pattern += pattern[0]; + DISPATCH; - case SRE_OP_BRANCH: + TARGET(SRE_OP_BRANCH): /* alternation */ /* <0=skip> code ... */ - TRACE(("|%p|%p|BRANCH\n", ctx->pattern, ctx->ptr)); + TRACE(("|%p|%p|BRANCH\n", pattern, ptr)); LASTMARK_SAVE(); if (state->repeat) MARK_PUSH(ctx->lastmark); - for (; ctx->pattern[0]; ctx->pattern += ctx->pattern[0]) { - if (ctx->pattern[1] == SRE_OP_LITERAL && - (ctx->ptr >= end || - (SRE_CODE) *ctx->ptr != ctx->pattern[2])) + for (; pattern[0]; pattern += pattern[0]) { + if (pattern[1] == SRE_OP_LITERAL && + (ptr >= end || + (SRE_CODE) *ptr != pattern[2])) continue; - if (ctx->pattern[1] == SRE_OP_IN && - (ctx->ptr >= end || - !SRE(charset)(state, ctx->pattern + 3, - (SRE_CODE) *ctx->ptr))) + if (pattern[1] == SRE_OP_IN && + (ptr >= end || + !SRE(charset)(state, pattern + 3, + (SRE_CODE) *ptr))) continue; - state->ptr = ctx->ptr; - DO_JUMP(JUMP_BRANCH, jump_branch, ctx->pattern+1); + state->ptr = ptr; + DO_JUMP(JUMP_BRANCH, jump_branch, pattern+1); if (ret) { if (state->repeat) MARK_POP_DISCARD(ctx->lastmark); @@ -798,7 +838,7 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) MARK_POP_DISCARD(ctx->lastmark); RETURN_FAILURE; - case SRE_OP_REPEAT_ONE: + TARGET(SRE_OP_REPEAT_ONE): /* match repeated sequence (maximizing regexp) */ /* this operator only works if the repeated item is @@ -808,34 +848,34 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) /* <1=min> <2=max> item tail */ - TRACE(("|%p|%p|REPEAT_ONE %d %d\n", ctx->pattern, ctx->ptr, - ctx->pattern[1], ctx->pattern[2])); + TRACE(("|%p|%p|REPEAT_ONE %d %d\n", pattern, ptr, + pattern[1], pattern[2])); - if ((Py_ssize_t) ctx->pattern[1] > end - ctx->ptr) + if ((Py_ssize_t) pattern[1] > end - ptr) RETURN_FAILURE; /* cannot match */ - state->ptr = ctx->ptr; + state->ptr = ptr; - ret = SRE(count)(state, ctx->pattern+3, ctx->pattern[2]); + ret = SRE(count)(state, pattern+3, pattern[2]); RETURN_ON_ERROR(ret); DATA_LOOKUP_AT(SRE(match_context), ctx, ctx_pos); ctx->count = ret; - ctx->ptr += ctx->count; + ptr += ctx->count; /* when we arrive here, count contains the number of - matches, and ctx->ptr points to the tail of the target + matches, and ptr points to the tail of the target string. check if the rest of the pattern matches, and backtrack if not. */ - if (ctx->count < (Py_ssize_t) ctx->pattern[1]) + if (ctx->count < (Py_ssize_t) pattern[1]) RETURN_FAILURE; - if (ctx->pattern[ctx->pattern[0]] == SRE_OP_SUCCESS && - ctx->ptr == state->end && - !(ctx->toplevel && state->must_advance && ctx->ptr == state->start)) + if (pattern[pattern[0]] == SRE_OP_SUCCESS && + ptr == state->end && + !(ctx->toplevel && state->must_advance && ptr == state->start)) { /* tail is empty. we're finished */ - state->ptr = ctx->ptr; + state->ptr = ptr; RETURN_SUCCESS; } @@ -843,21 +883,21 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) if (state->repeat) MARK_PUSH(ctx->lastmark); - if (ctx->pattern[ctx->pattern[0]] == SRE_OP_LITERAL) { + if (pattern[pattern[0]] == SRE_OP_LITERAL) { /* tail starts with a literal. skip positions where the rest of the pattern cannot possibly match */ - ctx->u.chr = ctx->pattern[ctx->pattern[0]+1]; + ctx->u.chr = pattern[pattern[0]+1]; for (;;) { - while (ctx->count >= (Py_ssize_t) ctx->pattern[1] && - (ctx->ptr >= end || *ctx->ptr != ctx->u.chr)) { - ctx->ptr--; + while (ctx->count >= (Py_ssize_t) pattern[1] && + (ptr >= end || *ptr != ctx->u.chr)) { + ptr--; ctx->count--; } - if (ctx->count < (Py_ssize_t) ctx->pattern[1]) + if (ctx->count < (Py_ssize_t) pattern[1]) break; - state->ptr = ctx->ptr; + state->ptr = ptr; DO_JUMP(JUMP_REPEAT_ONE_1, jump_repeat_one_1, - ctx->pattern+ctx->pattern[0]); + pattern+pattern[0]); if (ret) { if (state->repeat) MARK_POP_DISCARD(ctx->lastmark); @@ -868,17 +908,17 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) MARK_POP_KEEP(ctx->lastmark); LASTMARK_RESTORE(); - ctx->ptr--; + ptr--; ctx->count--; } if (state->repeat) MARK_POP_DISCARD(ctx->lastmark); } else { /* general case */ - while (ctx->count >= (Py_ssize_t) ctx->pattern[1]) { - state->ptr = ctx->ptr; + while (ctx->count >= (Py_ssize_t) pattern[1]) { + state->ptr = ptr; DO_JUMP(JUMP_REPEAT_ONE_2, jump_repeat_one_2, - ctx->pattern+ctx->pattern[0]); + pattern+pattern[0]); if (ret) { if (state->repeat) MARK_POP_DISCARD(ctx->lastmark); @@ -889,7 +929,7 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) MARK_POP_KEEP(ctx->lastmark); LASTMARK_RESTORE(); - ctx->ptr--; + ptr--; ctx->count--; } if (state->repeat) @@ -897,7 +937,7 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) } RETURN_FAILURE; - case SRE_OP_MIN_REPEAT_ONE: + TARGET(SRE_OP_MIN_REPEAT_ONE): /* match repeated sequence (minimizing regexp) */ /* this operator only works if the repeated item is @@ -907,36 +947,36 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) /* <1=min> <2=max> item tail */ - TRACE(("|%p|%p|MIN_REPEAT_ONE %d %d\n", ctx->pattern, ctx->ptr, - ctx->pattern[1], ctx->pattern[2])); + TRACE(("|%p|%p|MIN_REPEAT_ONE %d %d\n", pattern, ptr, + pattern[1], pattern[2])); - if ((Py_ssize_t) ctx->pattern[1] > end - ctx->ptr) + if ((Py_ssize_t) pattern[1] > end - ptr) RETURN_FAILURE; /* cannot match */ - state->ptr = ctx->ptr; + state->ptr = ptr; - if (ctx->pattern[1] == 0) + if (pattern[1] == 0) ctx->count = 0; else { /* count using pattern min as the maximum */ - ret = SRE(count)(state, ctx->pattern+3, ctx->pattern[1]); + ret = SRE(count)(state, pattern+3, pattern[1]); RETURN_ON_ERROR(ret); DATA_LOOKUP_AT(SRE(match_context), ctx, ctx_pos); - if (ret < (Py_ssize_t) ctx->pattern[1]) + if (ret < (Py_ssize_t) pattern[1]) /* didn't match minimum number of times */ RETURN_FAILURE; /* advance past minimum matches of repeat */ ctx->count = ret; - ctx->ptr += ctx->count; + ptr += ctx->count; } - if (ctx->pattern[ctx->pattern[0]] == SRE_OP_SUCCESS && + if (pattern[pattern[0]] == SRE_OP_SUCCESS && !(ctx->toplevel && - ((state->match_all && ctx->ptr != state->end) || - (state->must_advance && ctx->ptr == state->start)))) + ((state->match_all && ptr != state->end) || + (state->must_advance && ptr == state->start)))) { /* tail is empty. we're finished */ - state->ptr = ctx->ptr; + state->ptr = ptr; RETURN_SUCCESS; } else { @@ -945,11 +985,11 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) if (state->repeat) MARK_PUSH(ctx->lastmark); - while ((Py_ssize_t)ctx->pattern[2] == SRE_MAXREPEAT - || ctx->count <= (Py_ssize_t)ctx->pattern[2]) { - state->ptr = ctx->ptr; + while ((Py_ssize_t)pattern[2] == SRE_MAXREPEAT + || ctx->count <= (Py_ssize_t)pattern[2]) { + state->ptr = ptr; DO_JUMP(JUMP_MIN_REPEAT_ONE,jump_min_repeat_one, - ctx->pattern+ctx->pattern[0]); + pattern+pattern[0]); if (ret) { if (state->repeat) MARK_POP_DISCARD(ctx->lastmark); @@ -960,14 +1000,14 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) MARK_POP_KEEP(ctx->lastmark); LASTMARK_RESTORE(); - state->ptr = ctx->ptr; - ret = SRE(count)(state, ctx->pattern+3, 1); + state->ptr = ptr; + ret = SRE(count)(state, pattern+3, 1); RETURN_ON_ERROR(ret); DATA_LOOKUP_AT(SRE(match_context), ctx, ctx_pos); if (ret == 0) break; assert(ret == 1); - ctx->ptr++; + ptr++; ctx->count++; } if (state->repeat) @@ -975,7 +1015,7 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) } RETURN_FAILURE; - case SRE_OP_POSSESSIVE_REPEAT_ONE: + TARGET(SRE_OP_POSSESSIVE_REPEAT_ONE): /* match repeated sequence (maximizing regexp) without backtracking */ @@ -987,67 +1027,67 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) /* <1=min> <2=max> item tail */ - TRACE(("|%p|%p|POSSESSIVE_REPEAT_ONE %d %d\n", ctx->pattern, - ctx->ptr, ctx->pattern[1], ctx->pattern[2])); + TRACE(("|%p|%p|POSSESSIVE_REPEAT_ONE %d %d\n", pattern, + ptr, pattern[1], pattern[2])); - if (ctx->ptr + ctx->pattern[1] > end) { + if (ptr + pattern[1] > end) { RETURN_FAILURE; /* cannot match */ } - state->ptr = ctx->ptr; + state->ptr = ptr; - ret = SRE(count)(state, ctx->pattern + 3, ctx->pattern[2]); + ret = SRE(count)(state, pattern + 3, pattern[2]); RETURN_ON_ERROR(ret); DATA_LOOKUP_AT(SRE(match_context), ctx, ctx_pos); ctx->count = ret; - ctx->ptr += ctx->count; + ptr += ctx->count; /* when we arrive here, count contains the number of - matches, and ctx->ptr points to the tail of the target + matches, and ptr points to the tail of the target string. check if the rest of the pattern matches, and fail if not. */ /* Test for not enough repetitions in match */ - if (ctx->count < (Py_ssize_t) ctx->pattern[1]) { + if (ctx->count < (Py_ssize_t) pattern[1]) { RETURN_FAILURE; } /* Update the pattern to point to the next op code */ - ctx->pattern += ctx->pattern[0]; + pattern += pattern[0]; /* Let the tail be evaluated separately and consider this match successful. */ - if (*ctx->pattern == SRE_OP_SUCCESS && - ctx->ptr == state->end && - !(ctx->toplevel && state->must_advance && ctx->ptr == state->start)) + if (*pattern == SRE_OP_SUCCESS && + ptr == state->end && + !(ctx->toplevel && state->must_advance && ptr == state->start)) { /* tail is empty. we're finished */ - state->ptr = ctx->ptr; + state->ptr = ptr; RETURN_SUCCESS; } /* Attempt to match the rest of the string */ - break; + DISPATCH; - case SRE_OP_REPEAT: + TARGET(SRE_OP_REPEAT): /* create repeat context. all the hard work is done by the UNTIL operator (MAX_UNTIL, MIN_UNTIL) */ /* <1=min> <2=max> <3=repeat_index> item tail */ - TRACE(("|%p|%p|REPEAT %d %d %d\n", ctx->pattern, ctx->ptr, - ctx->pattern[1], ctx->pattern[2], ctx->pattern[3])); + TRACE(("|%p|%p|REPEAT %d %d %d\n", pattern, ptr, + pattern[1], pattern[2], pattern[3])); /* install repeat context */ - ctx->u.rep = &state->repeats_array[ctx->pattern[3]]; + ctx->u.rep = &state->repeats_array[pattern[3]]; ctx->u.rep->count = -1; - ctx->u.rep->pattern = ctx->pattern; + ctx->u.rep->pattern = pattern; ctx->u.rep->prev = state->repeat; ctx->u.rep->last_ptr = NULL; state->repeat = ctx->u.rep; - state->ptr = ctx->ptr; - DO_JUMP(JUMP_REPEAT, jump_repeat, ctx->pattern+ctx->pattern[0]); + state->ptr = ptr; + DO_JUMP(JUMP_REPEAT, jump_repeat, pattern+pattern[0]); state->repeat = ctx->u.rep->prev; if (ret) { @@ -1056,7 +1096,7 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) } RETURN_FAILURE; - case SRE_OP_MAX_UNTIL: + TARGET(SRE_OP_MAX_UNTIL): /* maximizing repeat */ /* <1=min> <2=max> <3=repeat_index> item tail */ @@ -1068,12 +1108,12 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) if (!ctx->u.rep) RETURN_ERROR(SRE_ERROR_STATE); - state->ptr = ctx->ptr; + state->ptr = ptr; ctx->count = ctx->u.rep->count+1; - TRACE(("|%p|%p|MAX_UNTIL %zd\n", ctx->pattern, - ctx->ptr, ctx->count)); + TRACE(("|%p|%p|MAX_UNTIL %zd\n", pattern, + ptr, ctx->count)); if (ctx->count < (Py_ssize_t) ctx->u.rep->pattern[1]) { /* not enough matches */ @@ -1085,7 +1125,7 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) RETURN_SUCCESS; } ctx->u.rep->count = ctx->count-1; - state->ptr = ctx->ptr; + state->ptr = ptr; RETURN_FAILURE; } @@ -1111,20 +1151,20 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) MARK_POP(ctx->lastmark); LASTMARK_RESTORE(); ctx->u.rep->count = ctx->count-1; - state->ptr = ctx->ptr; + state->ptr = ptr; } /* cannot match more repeated items here. make sure the tail matches */ state->repeat = ctx->u.rep->prev; - DO_JUMP(JUMP_MAX_UNTIL_3, jump_max_until_3, ctx->pattern); + DO_JUMP(JUMP_MAX_UNTIL_3, jump_max_until_3, pattern); state->repeat = ctx->u.rep; // restore repeat before return RETURN_ON_SUCCESS(ret); - state->ptr = ctx->ptr; + state->ptr = ptr; RETURN_FAILURE; - case SRE_OP_MIN_UNTIL: + TARGET(SRE_OP_MIN_UNTIL): /* minimizing repeat */ /* <1=min> <2=max> <3=repeat_index> item tail */ @@ -1133,12 +1173,12 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) if (!ctx->u.rep) RETURN_ERROR(SRE_ERROR_STATE); - state->ptr = ctx->ptr; + state->ptr = ptr; ctx->count = ctx->u.rep->count+1; - TRACE(("|%p|%p|MIN_UNTIL %zd %p\n", ctx->pattern, - ctx->ptr, ctx->count, ctx->u.rep->pattern)); + TRACE(("|%p|%p|MIN_UNTIL %zd %p\n", pattern, + ptr, ctx->count, ctx->u.rep->pattern)); if (ctx->count < (Py_ssize_t) ctx->u.rep->pattern[1]) { /* not enough matches */ @@ -1150,7 +1190,7 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) RETURN_SUCCESS; } ctx->u.rep->count = ctx->count-1; - state->ptr = ctx->ptr; + state->ptr = ptr; RETURN_FAILURE; } @@ -1161,7 +1201,7 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) if (state->repeat) MARK_PUSH(ctx->lastmark); - DO_JUMP(JUMP_MIN_UNTIL_2, jump_min_until_2, ctx->pattern); + DO_JUMP(JUMP_MIN_UNTIL_2, jump_min_until_2, pattern); SRE_REPEAT *repeat_of_tail = state->repeat; state->repeat = ctx->u.rep; // restore repeat before return @@ -1175,7 +1215,7 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) MARK_POP(ctx->lastmark); LASTMARK_RESTORE(); - state->ptr = ctx->ptr; + state->ptr = ptr; if ((ctx->count >= (Py_ssize_t) ctx->u.rep->pattern[2] && ctx->u.rep->pattern[2] != SRE_MAXREPEAT) || @@ -1194,34 +1234,34 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) RETURN_SUCCESS; } ctx->u.rep->count = ctx->count-1; - state->ptr = ctx->ptr; + state->ptr = ptr; RETURN_FAILURE; - case SRE_OP_POSSESSIVE_REPEAT: + TARGET(SRE_OP_POSSESSIVE_REPEAT): /* create possessive repeat contexts. */ /* <1=min> <2=max> pattern tail */ - TRACE(("|%p|%p|POSSESSIVE_REPEAT %d %d\n", ctx->pattern, - ctx->ptr, ctx->pattern[1], ctx->pattern[2])); + TRACE(("|%p|%p|POSSESSIVE_REPEAT %d %d\n", pattern, + ptr, pattern[1], pattern[2])); /* Set the global Input pointer to this context's Input pointer */ - state->ptr = ctx->ptr; + state->ptr = ptr; /* Initialize Count to 0 */ ctx->count = 0; /* Check for minimum required matches. */ - while (ctx->count < (Py_ssize_t)ctx->pattern[1]) { + while (ctx->count < (Py_ssize_t)pattern[1]) { /* not enough matches */ DO_JUMP(JUMP_POSS_REPEAT_1, jump_poss_repeat_1, - &ctx->pattern[3]); + &pattern[3]); if (ret) { RETURN_ON_ERROR(ret); ctx->count++; } else { - state->ptr = ctx->ptr; + state->ptr = ptr; RETURN_FAILURE; } } @@ -1229,13 +1269,13 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) /* Clear the context's Input stream pointer so that it doesn't match the global state so that the while loop can be entered. */ - ctx->ptr = NULL; + ptr = NULL; /* Keep trying to parse the sub-pattern until the end is reached, creating a new context each time. */ - while ((ctx->count < (Py_ssize_t)ctx->pattern[2] || - (Py_ssize_t)ctx->pattern[2] == SRE_MAXREPEAT) && - state->ptr != ctx->ptr) { + while ((ctx->count < (Py_ssize_t)pattern[2] || + (Py_ssize_t)pattern[2] == SRE_MAXREPEAT) && + state->ptr != ptr) { /* Save the Capture Group Marker state into the current Context and back up the current highest number Capture Group marker. */ @@ -1257,12 +1297,12 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) maximum number of matches are counted, and because of this, we could immediately stop at that point and consider this match successful. */ - ctx->ptr = state->ptr; + ptr = state->ptr; /* We have not reached the maximin matches, so try to match once more. */ DO_JUMP(JUMP_POSS_REPEAT_2, jump_poss_repeat_2, - &ctx->pattern[3]); + &pattern[3]); /* Check to see if the last attempted match succeeded. */ @@ -1293,47 +1333,47 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) /* Evaluate Tail */ /* Jump to end of pattern indicated by skip, and then skip the SUCCESS op code that follows it. */ - ctx->pattern += ctx->pattern[0] + 1; - ctx->ptr = state->ptr; - break; + pattern += pattern[0] + 1; + ptr = state->ptr; + DISPATCH; - case SRE_OP_ATOMIC_GROUP: + TARGET(SRE_OP_ATOMIC_GROUP): /* Atomic Group Sub Pattern */ /* pattern tail */ - TRACE(("|%p|%p|ATOMIC_GROUP\n", ctx->pattern, ctx->ptr)); + TRACE(("|%p|%p|ATOMIC_GROUP\n", pattern, ptr)); /* Set the global Input pointer to this context's Input pointer */ - state->ptr = ctx->ptr; + state->ptr = ptr; /* Evaluate the Atomic Group in a new context, terminating when the end of the group, represented by a SUCCESS op code, is reached. */ /* Group Pattern begins at an offset of 1 code. */ DO_JUMP(JUMP_ATOMIC_GROUP, jump_atomic_group, - &ctx->pattern[1]); + &pattern[1]); /* Test Exit Condition */ RETURN_ON_ERROR(ret); if (ret == 0) { /* Atomic Group failed to Match. */ - state->ptr = ctx->ptr; + state->ptr = ptr; RETURN_FAILURE; } /* Evaluate Tail */ /* Jump to end of pattern indicated by skip, and then skip the SUCCESS op code that follows it. */ - ctx->pattern += ctx->pattern[0]; - ctx->ptr = state->ptr; - break; + pattern += pattern[0]; + ptr = state->ptr; + DISPATCH; - case SRE_OP_GROUPREF: + TARGET(SRE_OP_GROUPREF): /* match backreference */ - TRACE(("|%p|%p|GROUPREF %d\n", ctx->pattern, - ctx->ptr, ctx->pattern[0])); - i = ctx->pattern[0]; + TRACE(("|%p|%p|GROUPREF %d\n", pattern, + ptr, pattern[0])); + i = pattern[0]; { Py_ssize_t groupref = i+i; if (groupref >= state->lastmark) { @@ -1344,21 +1384,21 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) if (!p || !e || e < p) RETURN_FAILURE; while (p < e) { - if (ctx->ptr >= end || *ctx->ptr != *p) + if (ptr >= end || *ptr != *p) RETURN_FAILURE; p++; - ctx->ptr++; + ptr++; } } } - ctx->pattern++; - break; + pattern++; + DISPATCH; - case SRE_OP_GROUPREF_IGNORE: + TARGET(SRE_OP_GROUPREF_IGNORE): /* match backreference */ - TRACE(("|%p|%p|GROUPREF_IGNORE %d\n", ctx->pattern, - ctx->ptr, ctx->pattern[0])); - i = ctx->pattern[0]; + TRACE(("|%p|%p|GROUPREF_IGNORE %d\n", pattern, + ptr, pattern[0])); + i = pattern[0]; { Py_ssize_t groupref = i+i; if (groupref >= state->lastmark) { @@ -1369,22 +1409,22 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) if (!p || !e || e < p) RETURN_FAILURE; while (p < e) { - if (ctx->ptr >= end || - sre_lower_ascii(*ctx->ptr) != sre_lower_ascii(*p)) + if (ptr >= end || + sre_lower_ascii(*ptr) != sre_lower_ascii(*p)) RETURN_FAILURE; p++; - ctx->ptr++; + ptr++; } } } - ctx->pattern++; - break; + pattern++; + DISPATCH; - case SRE_OP_GROUPREF_UNI_IGNORE: + TARGET(SRE_OP_GROUPREF_UNI_IGNORE): /* match backreference */ - TRACE(("|%p|%p|GROUPREF_UNI_IGNORE %d\n", ctx->pattern, - ctx->ptr, ctx->pattern[0])); - i = ctx->pattern[0]; + TRACE(("|%p|%p|GROUPREF_UNI_IGNORE %d\n", pattern, + ptr, pattern[0])); + i = pattern[0]; { Py_ssize_t groupref = i+i; if (groupref >= state->lastmark) { @@ -1395,22 +1435,22 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) if (!p || !e || e < p) RETURN_FAILURE; while (p < e) { - if (ctx->ptr >= end || - sre_lower_unicode(*ctx->ptr) != sre_lower_unicode(*p)) + if (ptr >= end || + sre_lower_unicode(*ptr) != sre_lower_unicode(*p)) RETURN_FAILURE; p++; - ctx->ptr++; + ptr++; } } } - ctx->pattern++; - break; + pattern++; + DISPATCH; - case SRE_OP_GROUPREF_LOC_IGNORE: + TARGET(SRE_OP_GROUPREF_LOC_IGNORE): /* match backreference */ - TRACE(("|%p|%p|GROUPREF_LOC_IGNORE %d\n", ctx->pattern, - ctx->ptr, ctx->pattern[0])); - i = ctx->pattern[0]; + TRACE(("|%p|%p|GROUPREF_LOC_IGNORE %d\n", pattern, + ptr, pattern[0])); + i = pattern[0]; { Py_ssize_t groupref = i+i; if (groupref >= state->lastmark) { @@ -1421,64 +1461,64 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) if (!p || !e || e < p) RETURN_FAILURE; while (p < e) { - if (ctx->ptr >= end || - sre_lower_locale(*ctx->ptr) != sre_lower_locale(*p)) + if (ptr >= end || + sre_lower_locale(*ptr) != sre_lower_locale(*p)) RETURN_FAILURE; p++; - ctx->ptr++; + ptr++; } } } - ctx->pattern++; - break; + pattern++; + DISPATCH; - case SRE_OP_GROUPREF_EXISTS: - TRACE(("|%p|%p|GROUPREF_EXISTS %d\n", ctx->pattern, - ctx->ptr, ctx->pattern[0])); + TARGET(SRE_OP_GROUPREF_EXISTS): + TRACE(("|%p|%p|GROUPREF_EXISTS %d\n", pattern, + ptr, pattern[0])); /* codeyes codeno ... */ - i = ctx->pattern[0]; + i = pattern[0]; { Py_ssize_t groupref = i+i; if (groupref >= state->lastmark) { - ctx->pattern += ctx->pattern[1]; - break; + pattern += pattern[1]; + DISPATCH; } else { SRE_CHAR* p = (SRE_CHAR*) state->mark[groupref]; SRE_CHAR* e = (SRE_CHAR*) state->mark[groupref+1]; if (!p || !e || e < p) { - ctx->pattern += ctx->pattern[1]; - break; + pattern += pattern[1]; + DISPATCH; } } } - ctx->pattern += 2; - break; + pattern += 2; + DISPATCH; - case SRE_OP_ASSERT: + TARGET(SRE_OP_ASSERT): /* assert subpattern */ /* */ - TRACE(("|%p|%p|ASSERT %d\n", ctx->pattern, - ctx->ptr, ctx->pattern[1])); - if (ctx->ptr - (SRE_CHAR *)state->beginning < (Py_ssize_t)ctx->pattern[1]) + TRACE(("|%p|%p|ASSERT %d\n", pattern, + ptr, pattern[1])); + if (ptr - (SRE_CHAR *)state->beginning < (Py_ssize_t)pattern[1]) RETURN_FAILURE; - state->ptr = ctx->ptr - ctx->pattern[1]; - DO_JUMP0(JUMP_ASSERT, jump_assert, ctx->pattern+2); + state->ptr = ptr - pattern[1]; + DO_JUMP0(JUMP_ASSERT, jump_assert, pattern+2); RETURN_ON_FAILURE(ret); - ctx->pattern += ctx->pattern[0]; - break; + pattern += pattern[0]; + DISPATCH; - case SRE_OP_ASSERT_NOT: + TARGET(SRE_OP_ASSERT_NOT): /* assert not subpattern */ /* */ - TRACE(("|%p|%p|ASSERT_NOT %d\n", ctx->pattern, - ctx->ptr, ctx->pattern[1])); - if (ctx->ptr - (SRE_CHAR *)state->beginning >= (Py_ssize_t)ctx->pattern[1]) { - state->ptr = ctx->ptr - ctx->pattern[1]; + TRACE(("|%p|%p|ASSERT_NOT %d\n", pattern, + ptr, pattern[1])); + if (ptr - (SRE_CHAR *)state->beginning >= (Py_ssize_t)pattern[1]) { + state->ptr = ptr - pattern[1]; LASTMARK_SAVE(); if (state->repeat) MARK_PUSH(ctx->lastmark); - DO_JUMP0(JUMP_ASSERT_NOT, jump_assert_not, ctx->pattern+2); + DO_JUMP0(JUMP_ASSERT_NOT, jump_assert_not, pattern+2); if (ret) { if (state->repeat) MARK_POP_DISCARD(ctx->lastmark); @@ -1489,19 +1529,29 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) MARK_POP(ctx->lastmark); LASTMARK_RESTORE(); } - ctx->pattern += ctx->pattern[0]; - break; + pattern += pattern[0]; + DISPATCH; - case SRE_OP_FAILURE: + TARGET(SRE_OP_FAILURE): /* immediate failure */ - TRACE(("|%p|%p|FAILURE\n", ctx->pattern, ctx->ptr)); + TRACE(("|%p|%p|FAILURE\n", pattern, ptr)); RETURN_FAILURE; +#if !USE_COMPUTED_GOTOS default: - TRACE(("|%p|%p|UNKNOWN %d\n", ctx->pattern, ctx->ptr, - ctx->pattern[-1])); +#endif + // Also any unused opcodes: + TARGET(SRE_OP_RANGE_UNI_IGNORE): + TARGET(SRE_OP_SUBPATTERN): + TARGET(SRE_OP_RANGE): + TARGET(SRE_OP_NEGATE): + TARGET(SRE_OP_BIGCHARSET): + TARGET(SRE_OP_CHARSET): + TARGET(SRE_OP_CALL): + TRACE(("|%p|%p|UNKNOWN %d\n", pattern, ptr, + pattern[-1])); RETURN_ERROR(SRE_ERROR_ILLEGAL); - } + } exit: @@ -1514,56 +1564,56 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) switch (jump) { case JUMP_MAX_UNTIL_2: - TRACE(("|%p|%p|JUMP_MAX_UNTIL_2\n", ctx->pattern, ctx->ptr)); + TRACE(("|%p|%p|JUMP_MAX_UNTIL_2\n", pattern, ptr)); goto jump_max_until_2; case JUMP_MAX_UNTIL_3: - TRACE(("|%p|%p|JUMP_MAX_UNTIL_3\n", ctx->pattern, ctx->ptr)); + TRACE(("|%p|%p|JUMP_MAX_UNTIL_3\n", pattern, ptr)); goto jump_max_until_3; case JUMP_MIN_UNTIL_2: - TRACE(("|%p|%p|JUMP_MIN_UNTIL_2\n", ctx->pattern, ctx->ptr)); + TRACE(("|%p|%p|JUMP_MIN_UNTIL_2\n", pattern, ptr)); goto jump_min_until_2; case JUMP_MIN_UNTIL_3: - TRACE(("|%p|%p|JUMP_MIN_UNTIL_3\n", ctx->pattern, ctx->ptr)); + TRACE(("|%p|%p|JUMP_MIN_UNTIL_3\n", pattern, ptr)); goto jump_min_until_3; case JUMP_BRANCH: - TRACE(("|%p|%p|JUMP_BRANCH\n", ctx->pattern, ctx->ptr)); + TRACE(("|%p|%p|JUMP_BRANCH\n", pattern, ptr)); goto jump_branch; case JUMP_MAX_UNTIL_1: - TRACE(("|%p|%p|JUMP_MAX_UNTIL_1\n", ctx->pattern, ctx->ptr)); + TRACE(("|%p|%p|JUMP_MAX_UNTIL_1\n", pattern, ptr)); goto jump_max_until_1; case JUMP_MIN_UNTIL_1: - TRACE(("|%p|%p|JUMP_MIN_UNTIL_1\n", ctx->pattern, ctx->ptr)); + TRACE(("|%p|%p|JUMP_MIN_UNTIL_1\n", pattern, ptr)); goto jump_min_until_1; case JUMP_POSS_REPEAT_1: - TRACE(("|%p|%p|JUMP_POSS_REPEAT_1\n", ctx->pattern, ctx->ptr)); + TRACE(("|%p|%p|JUMP_POSS_REPEAT_1\n", pattern, ptr)); goto jump_poss_repeat_1; case JUMP_POSS_REPEAT_2: - TRACE(("|%p|%p|JUMP_POSS_REPEAT_2\n", ctx->pattern, ctx->ptr)); + TRACE(("|%p|%p|JUMP_POSS_REPEAT_2\n", pattern, ptr)); goto jump_poss_repeat_2; case JUMP_REPEAT: - TRACE(("|%p|%p|JUMP_REPEAT\n", ctx->pattern, ctx->ptr)); + TRACE(("|%p|%p|JUMP_REPEAT\n", pattern, ptr)); goto jump_repeat; case JUMP_REPEAT_ONE_1: - TRACE(("|%p|%p|JUMP_REPEAT_ONE_1\n", ctx->pattern, ctx->ptr)); + TRACE(("|%p|%p|JUMP_REPEAT_ONE_1\n", pattern, ptr)); goto jump_repeat_one_1; case JUMP_REPEAT_ONE_2: - TRACE(("|%p|%p|JUMP_REPEAT_ONE_2\n", ctx->pattern, ctx->ptr)); + TRACE(("|%p|%p|JUMP_REPEAT_ONE_2\n", pattern, ptr)); goto jump_repeat_one_2; case JUMP_MIN_REPEAT_ONE: - TRACE(("|%p|%p|JUMP_MIN_REPEAT_ONE\n", ctx->pattern, ctx->ptr)); + TRACE(("|%p|%p|JUMP_MIN_REPEAT_ONE\n", pattern, ptr)); goto jump_min_repeat_one; case JUMP_ATOMIC_GROUP: - TRACE(("|%p|%p|JUMP_ATOMIC_GROUP\n", ctx->pattern, ctx->ptr)); + TRACE(("|%p|%p|JUMP_ATOMIC_GROUP\n", pattern, ptr)); goto jump_atomic_group; case JUMP_ASSERT: - TRACE(("|%p|%p|JUMP_ASSERT\n", ctx->pattern, ctx->ptr)); + TRACE(("|%p|%p|JUMP_ASSERT\n", pattern, ptr)); goto jump_assert; case JUMP_ASSERT_NOT: - TRACE(("|%p|%p|JUMP_ASSERT_NOT\n", ctx->pattern, ctx->ptr)); + TRACE(("|%p|%p|JUMP_ASSERT_NOT\n", pattern, ptr)); goto jump_assert_not; case JUMP_NONE: - TRACE(("|%p|%p|RETURN %zd\n", ctx->pattern, - ctx->ptr, ret)); + TRACE(("|%p|%p|RETURN %zd\n", pattern, + ptr, ret)); break; } diff --git a/Modules/_sre/sre_targets.h b/Modules/_sre/sre_targets.h new file mode 100644 index 0000000000000..389e7d77e949c --- /dev/null +++ b/Modules/_sre/sre_targets.h @@ -0,0 +1,59 @@ +/* + * Secret Labs' Regular Expression Engine + * + * regular expression matching engine + * + * Auto-generated by Tools/scripts/generate_sre_constants.py from + * Lib/re/_constants.py. + * + * Copyright (c) 1997-2001 by Secret Labs AB. All rights reserved. + * + * See the sre.c file for information on usage and redistribution. + */ + +static void *sre_targets[44] = { + &&TARGET_SRE_OP_FAILURE, + &&TARGET_SRE_OP_SUCCESS, + &&TARGET_SRE_OP_ANY, + &&TARGET_SRE_OP_ANY_ALL, + &&TARGET_SRE_OP_ASSERT, + &&TARGET_SRE_OP_ASSERT_NOT, + &&TARGET_SRE_OP_AT, + &&TARGET_SRE_OP_BRANCH, + &&TARGET_SRE_OP_CALL, + &&TARGET_SRE_OP_CATEGORY, + &&TARGET_SRE_OP_CHARSET, + &&TARGET_SRE_OP_BIGCHARSET, + &&TARGET_SRE_OP_GROUPREF, + &&TARGET_SRE_OP_GROUPREF_EXISTS, + &&TARGET_SRE_OP_IN, + &&TARGET_SRE_OP_INFO, + &&TARGET_SRE_OP_JUMP, + &&TARGET_SRE_OP_LITERAL, + &&TARGET_SRE_OP_MARK, + &&TARGET_SRE_OP_MAX_UNTIL, + &&TARGET_SRE_OP_MIN_UNTIL, + &&TARGET_SRE_OP_NOT_LITERAL, + &&TARGET_SRE_OP_NEGATE, + &&TARGET_SRE_OP_RANGE, + &&TARGET_SRE_OP_REPEAT, + &&TARGET_SRE_OP_REPEAT_ONE, + &&TARGET_SRE_OP_SUBPATTERN, + &&TARGET_SRE_OP_MIN_REPEAT_ONE, + &&TARGET_SRE_OP_ATOMIC_GROUP, + &&TARGET_SRE_OP_POSSESSIVE_REPEAT, + &&TARGET_SRE_OP_POSSESSIVE_REPEAT_ONE, + &&TARGET_SRE_OP_GROUPREF_IGNORE, + &&TARGET_SRE_OP_IN_IGNORE, + &&TARGET_SRE_OP_LITERAL_IGNORE, + &&TARGET_SRE_OP_NOT_LITERAL_IGNORE, + &&TARGET_SRE_OP_GROUPREF_LOC_IGNORE, + &&TARGET_SRE_OP_IN_LOC_IGNORE, + &&TARGET_SRE_OP_LITERAL_LOC_IGNORE, + &&TARGET_SRE_OP_NOT_LITERAL_LOC_IGNORE, + &&TARGET_SRE_OP_GROUPREF_UNI_IGNORE, + &&TARGET_SRE_OP_IN_UNI_IGNORE, + &&TARGET_SRE_OP_LITERAL_UNI_IGNORE, + &&TARGET_SRE_OP_NOT_LITERAL_UNI_IGNORE, + &&TARGET_SRE_OP_RANGE_UNI_IGNORE, +}; diff --git a/Tools/scripts/generate_sre_constants.py b/Tools/scripts/generate_sre_constants.py index b8f0df9cc23b1..72715076d29ab 100755 --- a/Tools/scripts/generate_sre_constants.py +++ b/Tools/scripts/generate_sre_constants.py @@ -29,7 +29,11 @@ def update_file(file, content): """ -def main(infile='Lib/re/_constants.py', outfile='Modules/_sre/sre_constants.h'): +def main( + infile="Lib/re/_constants.py", + outfile_constants="Modules/_sre/sre_constants.h", + outfile_targets="Modules/_sre/sre_targets.h", +): ns = {} with open(infile) as fp: code = fp.read() @@ -46,6 +50,11 @@ def dump2(d, prefix): for value, name in sorted(items): yield "#define %s %d\n" % (name, value) + def dump_gotos(d, prefix): + for i, item in enumerate(sorted(d)): + assert i == item + yield f" &&{prefix}_{item},\n" + content = [sre_constants_header] content.append("#define SRE_MAGIC %d\n" % ns["MAGIC"]) content.extend(dump(ns["OPCODES"], "SRE_OP")) @@ -54,7 +63,14 @@ def dump2(d, prefix): content.extend(dump2(ns, "SRE_FLAG_")) content.extend(dump2(ns, "SRE_INFO_")) - update_file(outfile, ''.join(content)) + update_file(outfile_constants, ''.join(content)) + + content = [sre_constants_header] + content.append(f"static void *sre_targets[{len(ns['OPCODES'])}] = {{\n") + content.extend(dump_gotos(ns["OPCODES"], "TARGET_SRE_OP")) + content.append("};\n") + + update_file(outfile_targets, ''.join(content)) if __name__ == '__main__': From webhook-mailer at python.org Fri Apr 15 12:27:43 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Fri, 15 Apr 2022 16:27:43 -0000 Subject: [Python-checkins] gh-69093: improve sqlite3.Connection.blobopen() error handling (GH-91571) Message-ID: https://github.com/python/cpython/commit/c06a4ffe818feddef3b5083d9746a1c0b82c84ab commit: c06a4ffe818feddef3b5083d9746a1c0b82c84ab branch: main author: Erlend Egeberg Aasland committer: JelleZijlstra date: 2022-04-15T09:27:39-07:00 summary: gh-69093: improve sqlite3.Connection.blobopen() error handling (GH-91571) Unless sqlite3_blob_open() returns SQLITE_MISUSE, the error code and message are available on the connection object. This means we have to handle SQLITE_MISUSE error messages explicitly. files: M Modules/_sqlite/connection.c diff --git a/Modules/_sqlite/connection.c b/Modules/_sqlite/connection.c index 85fb128fc7f1c..0028cf72f997a 100644 --- a/Modules/_sqlite/connection.c +++ b/Modules/_sqlite/connection.c @@ -475,7 +475,11 @@ blobopen_impl(pysqlite_Connection *self, const char *table, const char *col, rc = sqlite3_blob_open(self->db, name, table, col, row, !readonly, &blob); Py_END_ALLOW_THREADS - if (rc != SQLITE_OK) { + if (rc == SQLITE_MISUSE) { + PyErr_Format(self->state->InterfaceError, sqlite3_errstr(rc)); + return NULL; + } + else if (rc != SQLITE_OK) { _pysqlite_seterror(self->state, self->db); return NULL; } From webhook-mailer at python.org Fri Apr 15 14:57:54 2022 From: webhook-mailer at python.org (iritkatriel) Date: Fri, 15 Apr 2022 18:57:54 -0000 Subject: [Python-checkins] gh-90501: Add PyErr_GetHandledException and PyErr_SetHandledException (GH-30531) Message-ID: https://github.com/python/cpython/commit/5d421d7342fc0d278c129c05bea7028430e94a4e commit: 5d421d7342fc0d278c129c05bea7028430e94a4e branch: main author: Irit Katriel <1055913+iritkatriel at users.noreply.github.com> committer: iritkatriel <1055913+iritkatriel at users.noreply.github.com> date: 2022-04-15T19:57:47+01:00 summary: gh-90501: Add PyErr_GetHandledException and PyErr_SetHandledException (GH-30531) files: A Misc/NEWS.d/next/C API/2022-01-11-12-52-37.bpo-46343.JQJWhZ.rst M Doc/c-api/exceptions.rst M Doc/data/stable_abi.dat M Doc/library/sys.rst M Doc/whatsnew/3.11.rst M Include/cpython/pyerrors.h M Include/pyerrors.h M Lib/test/test_capi.py M Lib/test/test_stable_abi_ctypes.py M Misc/stable_abi.txt M Modules/_testcapimodule.c M PC/python3dll.c M Python/errors.c diff --git a/Doc/c-api/exceptions.rst b/Doc/c-api/exceptions.rst index a5a93d0ebbf28..7bfeca5958cc4 100644 --- a/Doc/c-api/exceptions.rst +++ b/Doc/c-api/exceptions.rst @@ -460,12 +460,46 @@ Querying the error indicator } -.. c:function:: void PyErr_GetExcInfo(PyObject **ptype, PyObject **pvalue, PyObject **ptraceback) +.. c:function:: PyObject* PyErr_GetHandledException(void) + + Retrieve the active exception instance, as would be returned by :func:`sys.exception`. + This refers to an exception that was *already caught*, not to an exception that was + freshly raised. Returns a new reference to the exception or ``NULL``. + Does not modify the interpreter's exception state. + + .. note:: + + This function is not normally used by code that wants to handle exceptions. + Rather, it can be used when code needs to save and restore the exception + state temporarily. Use :c:func:`PyErr_SetHandledException` to restore or + clear the exception state. + + .. versionadded:: 3.11 - Retrieve the exception info, as known from ``sys.exc_info()``. This refers +.. c:function:: void PyErr_SetHandledException(PyObject *exc) + + Set the active exception, as known from ``sys.exception()``. This refers to an exception that was *already caught*, not to an exception that was - freshly raised. Returns new references for the three objects, any of which - may be ``NULL``. Does not modify the exception info state. + freshly raised. + To clear the exception state, pass ``NULL``. + + .. note:: + + This function is not normally used by code that wants to handle exceptions. + Rather, it can be used when code needs to save and restore the exception + state temporarily. Use :c:func:`PyErr_GetHandledException` to get the exception + state. + + .. versionadded:: 3.11 + +.. c:function:: void PyErr_GetExcInfo(PyObject **ptype, PyObject **pvalue, PyObject **ptraceback) + + Retrieve the old-style representation of the exception info, as known from + :func:`sys.exc_info`. This refers to an exception that was *already caught*, + not to an exception that was freshly raised. Returns new references for the + three objects, any of which may be ``NULL``. Does not modify the exception + info state. This function is kept for backwards compatibility. Prefer using + :c:func:`PyErr_GetHandledException`. .. note:: @@ -483,6 +517,8 @@ Querying the error indicator to an exception that was *already caught*, not to an exception that was freshly raised. This function steals the references of the arguments. To clear the exception state, pass ``NULL`` for all three arguments. + This function is kept for backwards compatibility. Prefer using + :c:func:`PyErr_SetHandledException`. .. note:: diff --git a/Doc/data/stable_abi.dat b/Doc/data/stable_abi.dat index 849a2cfd51f24..5387d0bf983fa 100644 --- a/Doc/data/stable_abi.dat +++ b/Doc/data/stable_abi.dat @@ -137,6 +137,7 @@ function,PyErr_Fetch,3.2,, function,PyErr_Format,3.2,, function,PyErr_FormatV,3.5,, function,PyErr_GetExcInfo,3.7,, +function,PyErr_GetHandledException,3.11,, function,PyErr_GivenExceptionMatches,3.2,, function,PyErr_NewException,3.2,, function,PyErr_NewExceptionWithDoc,3.2,, @@ -159,6 +160,7 @@ function,PyErr_SetFromErrnoWithFilenameObject,3.2,, function,PyErr_SetFromErrnoWithFilenameObjects,3.7,, function,PyErr_SetFromWindowsErr,3.7,on Windows, function,PyErr_SetFromWindowsErrWithFilename,3.7,on Windows, +function,PyErr_SetHandledException,3.11,, function,PyErr_SetImportError,3.7,, function,PyErr_SetImportErrorSubclass,3.6,, function,PyErr_SetInterrupt,3.2,, diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst index 126da31b5bd32..2a8b532b592e8 100644 --- a/Doc/library/sys.rst +++ b/Doc/library/sys.rst @@ -381,19 +381,12 @@ always available. .. function:: exception() - This function returns the exception instance that is currently being - handled. This exception is specific both to the current thread and - to the current stack frame. If the current stack frame is not handling - an exception, the exception is taken from the calling stack frame, or its - caller, and so on until a stack frame is found that is handling an - exception. Here, "handling an exception" is defined as "executing an - except clause." For any stack frame, only the exception being currently - handled is accessible. + This function, when called while an exception handler is executing (such as + an ``except`` or ``except*`` clause), returns the exception instance that + was caught by this handler. When exception handlers are nested within one + another, only the exception handled by the innermost handler is accessible. - .. index:: object: traceback - - If no exception is being handled anywhere on the stack, ``None`` is - returned. + If no exception handler is executing, this function returns ``None``. .. versionadded:: 3.11 diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index a5a52682b503c..b6f47f532ca4c 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -1161,6 +1161,14 @@ New Features :c:func:`PyFrame_GetBuiltins`, :c:func:`PyFrame_GetGenerator`, :c:func:`PyFrame_GetGlobals`, :c:func:`PyFrame_GetLasti`. +* Added two new functions to get and set the active exception instance: + :c:func:`PyErr_GetHandledException` and :c:func:`PyErr_SetHandledException`. + These are alternatives to :c:func:`PyErr_SetExcInfo()` and + :c:func:`PyErr_GetExcInfo()` which work with the legacy 3-tuple + representation of exceptions. + (Contributed by Irit Katriel in :issue:`46343`.) + + Porting to Python 3.11 ---------------------- diff --git a/Include/cpython/pyerrors.h b/Include/cpython/pyerrors.h index 5281fde1f1a54..08630cce8ac90 100644 --- a/Include/cpython/pyerrors.h +++ b/Include/cpython/pyerrors.h @@ -91,6 +91,8 @@ typedef PyOSErrorObject PyWindowsErrorObject; PyAPI_FUNC(void) _PyErr_SetKeyError(PyObject *); PyAPI_FUNC(_PyErr_StackItem*) _PyErr_GetTopmostException(PyThreadState *tstate); +PyAPI_FUNC(PyObject*) _PyErr_GetHandledException(PyThreadState *); +PyAPI_FUNC(void) _PyErr_SetHandledException(PyThreadState *, PyObject *); PyAPI_FUNC(void) _PyErr_GetExcInfo(PyThreadState *, PyObject **, PyObject **, PyObject **); /* Context manipulation (PEP 3134) */ diff --git a/Include/pyerrors.h b/Include/pyerrors.h index 77d791427d492..34e3de3328f41 100644 --- a/Include/pyerrors.h +++ b/Include/pyerrors.h @@ -18,6 +18,10 @@ PyAPI_FUNC(PyObject *) PyErr_Occurred(void); PyAPI_FUNC(void) PyErr_Clear(void); PyAPI_FUNC(void) PyErr_Fetch(PyObject **, PyObject **, PyObject **); PyAPI_FUNC(void) PyErr_Restore(PyObject *, PyObject *, PyObject *); +#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x030b0000 +PyAPI_FUNC(PyObject*) PyErr_GetHandledException(void); +PyAPI_FUNC(void) PyErr_SetHandledException(PyObject *); +#endif #if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x03030000 PyAPI_FUNC(void) PyErr_GetExcInfo(PyObject **, PyObject **, PyObject **); PyAPI_FUNC(void) PyErr_SetExcInfo(PyObject *, PyObject *, PyObject *); diff --git a/Lib/test/test_capi.py b/Lib/test/test_capi.py index 40e4774c6b8ed..eb0edbf5a3a12 100644 --- a/Lib/test/test_capi.py +++ b/Lib/test/test_capi.py @@ -88,6 +88,28 @@ def test_no_FatalError_infinite_loop(self): def test_memoryview_from_NULL_pointer(self): self.assertRaises(ValueError, _testcapi.make_memoryview_from_NULL_pointer) + def test_exception(self): + raised_exception = ValueError("5") + new_exc = TypeError("TEST") + try: + raise raised_exception + except ValueError as e: + orig_sys_exception = sys.exception() + orig_exception = _testcapi.set_exception(new_exc) + new_sys_exception = sys.exception() + new_exception = _testcapi.set_exception(orig_exception) + reset_sys_exception = sys.exception() + + self.assertEqual(orig_exception, e) + + self.assertEqual(orig_exception, raised_exception) + self.assertEqual(orig_sys_exception, orig_exception) + self.assertEqual(reset_sys_exception, orig_exception) + self.assertEqual(new_exception, new_exc) + self.assertEqual(new_sys_exception, new_exception) + else: + self.fail("Exception not raised") + def test_exc_info(self): raised_exception = ValueError("5") new_exc = TypeError("TEST") diff --git a/Lib/test/test_stable_abi_ctypes.py b/Lib/test/test_stable_abi_ctypes.py index efd3b1b7cd2d2..0656ff5581be5 100644 --- a/Lib/test/test_stable_abi_ctypes.py +++ b/Lib/test/test_stable_abi_ctypes.py @@ -152,6 +152,7 @@ def test_available_symbols(self): "PyErr_Format", "PyErr_FormatV", "PyErr_GetExcInfo", + "PyErr_GetHandledException", "PyErr_GivenExceptionMatches", "PyErr_NewException", "PyErr_NewExceptionWithDoc", @@ -168,6 +169,7 @@ def test_available_symbols(self): "PyErr_SetFromErrnoWithFilename", "PyErr_SetFromErrnoWithFilenameObject", "PyErr_SetFromErrnoWithFilenameObjects", + "PyErr_SetHandledException", "PyErr_SetImportError", "PyErr_SetImportErrorSubclass", "PyErr_SetInterrupt", diff --git a/Misc/NEWS.d/next/C API/2022-01-11-12-52-37.bpo-46343.JQJWhZ.rst b/Misc/NEWS.d/next/C API/2022-01-11-12-52-37.bpo-46343.JQJWhZ.rst new file mode 100644 index 0000000000000..1ac8da853c879 --- /dev/null +++ b/Misc/NEWS.d/next/C API/2022-01-11-12-52-37.bpo-46343.JQJWhZ.rst @@ -0,0 +1,5 @@ +Added :c:func:`PyErr_GetHandledException` and +:c:func:`PyErr_SetHandledException` as simpler alternatives to +:c:func:`PyErr_GetExcInfo` and :c:func:`PyErr_SetExcInfo`. + +They are included in the stable ABI. diff --git a/Misc/stable_abi.txt b/Misc/stable_abi.txt index 4864bf319a76f..66777a62c4301 100644 --- a/Misc/stable_abi.txt +++ b/Misc/stable_abi.txt @@ -2253,3 +2253,8 @@ function PyMemoryView_FromBuffer data Py_Version added 3.11 +function PyErr_GetHandledException + added 3.11 +function PyErr_SetHandledException + added 3.11 + diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c index 13dd29427aa2c..71683abebb231 100644 --- a/Modules/_testcapimodule.c +++ b/Modules/_testcapimodule.c @@ -2562,6 +2562,16 @@ set_errno(PyObject *self, PyObject *args) Py_RETURN_NONE; } +static PyObject * +test_set_exception(PyObject *self, PyObject *new_exc) +{ + PyObject *exc = PyErr_GetHandledException(); + assert(PyExceptionInstance_Check(exc) || exc == NULL); + + PyErr_SetHandledException(new_exc); + return exc; +} + static PyObject * test_set_exc_info(PyObject *self, PyObject *args) { @@ -6068,6 +6078,7 @@ static PyMethodDef TestMethods[] = { #endif {"traceback_print", traceback_print, METH_VARARGS}, {"exception_print", exception_print, METH_VARARGS}, + {"set_exception", test_set_exception, METH_O}, {"set_exc_info", test_set_exc_info, METH_VARARGS}, {"argparsing", argparsing, METH_VARARGS}, {"code_newempty", code_newempty, METH_VARARGS}, diff --git a/PC/python3dll.c b/PC/python3dll.c index 70f11dc190554..0aee2aec84726 100755 --- a/PC/python3dll.c +++ b/PC/python3dll.c @@ -196,6 +196,7 @@ EXPORT_FUNC(PyErr_Fetch) EXPORT_FUNC(PyErr_Format) EXPORT_FUNC(PyErr_FormatV) EXPORT_FUNC(PyErr_GetExcInfo) +EXPORT_FUNC(PyErr_GetHandledException) EXPORT_FUNC(PyErr_GivenExceptionMatches) EXPORT_FUNC(PyErr_NewException) EXPORT_FUNC(PyErr_NewExceptionWithDoc) @@ -218,6 +219,7 @@ EXPORT_FUNC(PyErr_SetFromErrnoWithFilenameObject) EXPORT_FUNC(PyErr_SetFromErrnoWithFilenameObjects) EXPORT_FUNC(PyErr_SetFromWindowsErr) EXPORT_FUNC(PyErr_SetFromWindowsErrWithFilename) +EXPORT_FUNC(PyErr_SetHandledException) EXPORT_FUNC(PyErr_SetImportError) EXPORT_FUNC(PyErr_SetImportErrorSubclass) EXPORT_FUNC(PyErr_SetInterrupt) diff --git a/Python/errors.c b/Python/errors.c index e170c9dff2dbb..ce7785855b8e5 100644 --- a/Python/errors.c +++ b/Python/errors.c @@ -499,6 +499,38 @@ _PyErr_GetExcInfo(PyThreadState *tstate, Py_XINCREF(*p_traceback); } +PyObject* +_PyErr_GetHandledException(PyThreadState *tstate) +{ + _PyErr_StackItem *exc_info = _PyErr_GetTopmostException(tstate); + PyObject *exc = exc_info->exc_value; + if (exc == NULL || exc == Py_None) { + return NULL; + } + return Py_NewRef(exc); +} + +PyObject* +PyErr_GetHandledException(void) +{ + PyThreadState *tstate = _PyThreadState_GET(); + return _PyErr_GetHandledException(tstate); +} + +void +_PyErr_SetHandledException(PyThreadState *tstate, PyObject *exc) +{ + PyObject *oldexc = tstate->exc_info->exc_value; + tstate->exc_info->exc_value = Py_XNewRef(exc); + Py_XDECREF(oldexc); +} + +void +PyErr_SetHandledException(PyObject *exc) +{ + PyThreadState *tstate = _PyThreadState_GET(); + _PyErr_SetHandledException(tstate, exc); +} void PyErr_GetExcInfo(PyObject **p_type, PyObject **p_value, PyObject **p_traceback) @@ -510,17 +542,10 @@ PyErr_GetExcInfo(PyObject **p_type, PyObject **p_value, PyObject **p_traceback) void PyErr_SetExcInfo(PyObject *type, PyObject *value, PyObject *traceback) { - PyThreadState *tstate = _PyThreadState_GET(); - - PyObject *oldvalue = tstate->exc_info->exc_value; - - tstate->exc_info->exc_value = value; - + PyErr_SetHandledException(value); /* These args are no longer used, but we still need to steal a ref */ Py_XDECREF(type); Py_XDECREF(traceback); - - Py_XDECREF(oldvalue); } From webhook-mailer at python.org Fri Apr 15 15:19:28 2022 From: webhook-mailer at python.org (iritkatriel) Date: Fri, 15 Apr 2022 19:19:28 -0000 Subject: [Python-checkins] gh-91276: Make JUMP_IF_TRUE_OR_POP/JUMP_IF_FALSE_OR_POP relative (GH-32215) Message-ID: https://github.com/python/cpython/commit/ea2ae026078b328ddeab060940568a4d3bf1b417 commit: ea2ae026078b328ddeab060940568a4d3bf1b417 branch: main author: Irit Katriel <1055913+iritkatriel at users.noreply.github.com> committer: iritkatriel <1055913+iritkatriel at users.noreply.github.com> date: 2022-04-15T20:19:24+01:00 summary: gh-91276: Make JUMP_IF_TRUE_OR_POP/JUMP_IF_FALSE_OR_POP relative (GH-32215) files: A Misc/NEWS.d/next/Core and Builtins/2022-03-31-14-33-48.bpo-47120.6S_uoU.rst M Doc/library/dis.rst M Doc/whatsnew/3.11.rst M Include/opcode.h M Lib/importlib/_bootstrap_external.py M Lib/opcode.py M Python/ceval.c M Python/compile.c diff --git a/Doc/library/dis.rst b/Doc/library/dis.rst index 657778c0f4098..08e6c736d3e3c 100644 --- a/Doc/library/dis.rst +++ b/Doc/library/dis.rst @@ -993,21 +993,26 @@ iterations of the loop. .. versionadded:: 3.11 -.. opcode:: JUMP_IF_TRUE_OR_POP (target) +.. opcode:: JUMP_IF_TRUE_OR_POP (delta) - If TOS is true, sets the bytecode counter to *target* and leaves TOS on the + If TOS is true, increments the bytecode counter by *delta* and leaves TOS on the stack. Otherwise (TOS is false), TOS is popped. .. versionadded:: 3.1 + .. versionchanged:: 3.11 + The oparg is now a relative delta rather than an absolute target. -.. opcode:: JUMP_IF_FALSE_OR_POP (target) +.. opcode:: JUMP_IF_FALSE_OR_POP (delta) - If TOS is false, sets the bytecode counter to *target* and leaves TOS on the + If TOS is false, increments the bytecode counter by *delta* and leaves TOS on the stack. Otherwise (TOS is true), TOS is popped. .. versionadded:: 3.1 + .. versionchanged:: 3.11 + The oparg is now a relative delta rather than an absolute target. + .. opcode:: FOR_ITER (delta) diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index b6f47f532ca4c..ca76efc963719 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -787,6 +787,9 @@ CPython bytecode changes :opcode:`POP_JUMP_FORWARD_IF_NONE` and :opcode:`POP_JUMP_BACKWARD_IF_NONE` opcodes to speed up conditional jumps. +* :opcode:`JUMP_IF_TRUE_OR_POP` and :opcode:`JUMP_IF_FALSE_OR_POP` are now + relative rather than absolute. + Deprecated ========== diff --git a/Include/opcode.h b/Include/opcode.h index 0badf78e32771..8db42380cedd1 100644 --- a/Include/opcode.h +++ b/Include/opcode.h @@ -199,7 +199,7 @@ static const uint32_t _PyOpcode_RelativeJump[8] = { 0U, 0U, 536870912U, - 135020544U, + 135118848U, 4163U, 122880U, 0U, diff --git a/Lib/importlib/_bootstrap_external.py b/Lib/importlib/_bootstrap_external.py index d580e5417387b..4eece8de24c3e 100644 --- a/Lib/importlib/_bootstrap_external.py +++ b/Lib/importlib/_bootstrap_external.py @@ -401,6 +401,7 @@ def _write_atomic(path, data, mode=0o666): # Python 3.11a6 3491 (remove JUMP_IF_NOT_EG_MATCH, add CHECK_EG_MATCH, # add JUMP_BACKWARD_NO_INTERRUPT, make JUMP_NO_INTERRUPT virtual) # Python 3.11a7 3492 (make POP_JUMP_IF_NONE/NOT_NONE/TRUE/FALSE relative) +# Python 3.11a7 3493 (Make JUMP_IF_TRUE_OR_POP/JUMP_IF_FALSE_OR_POP relative) # Python 3.12 will start with magic number 3500 @@ -415,7 +416,8 @@ def _write_atomic(path, data, mode=0o666): # Whenever MAGIC_NUMBER is changed, the ranges in the magic_values array # in PC/launcher.c must also be updated. -MAGIC_NUMBER = (3492).to_bytes(2, 'little') + b'\r\n' +MAGIC_NUMBER = (3493).to_bytes(2, 'little') + b'\r\n' + _RAW_MAGIC_NUMBER = int.from_bytes(MAGIC_NUMBER, 'little') # For import.c _PYCACHE = '__pycache__' diff --git a/Lib/opcode.py b/Lib/opcode.py index ee9effbe540b2..9ee06831c3768 100644 --- a/Lib/opcode.py +++ b/Lib/opcode.py @@ -131,8 +131,8 @@ def jabs_op(name, op, entries=0): name_op('IMPORT_NAME', 108) # Index in name list name_op('IMPORT_FROM', 109) # Index in name list jrel_op('JUMP_FORWARD', 110) # Number of words to skip -jabs_op('JUMP_IF_FALSE_OR_POP', 111) # Target byte offset from beginning of code -jabs_op('JUMP_IF_TRUE_OR_POP', 112) # "" +jrel_op('JUMP_IF_FALSE_OR_POP', 111) # Number of words to skip +jrel_op('JUMP_IF_TRUE_OR_POP', 112) # "" jrel_op('POP_JUMP_FORWARD_IF_FALSE', 114) jrel_op('POP_JUMP_FORWARD_IF_TRUE', 115) name_op('LOAD_GLOBAL', 116, 5) # Index in name list diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-03-31-14-33-48.bpo-47120.6S_uoU.rst b/Misc/NEWS.d/next/Core and Builtins/2022-03-31-14-33-48.bpo-47120.6S_uoU.rst new file mode 100644 index 0000000000000..c87d9843d9157 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2022-03-31-14-33-48.bpo-47120.6S_uoU.rst @@ -0,0 +1 @@ +Make opcodes :opcode:`JUMP_IF_TRUE_OR_POP` and :opcode:`JUMP_IF_FALSE_OR_POP` relative rather than absolute. diff --git a/Python/ceval.c b/Python/ceval.c index 7891547ffc687..66856e5177607 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -4081,7 +4081,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int DISPATCH(); } if (Py_IsFalse(cond)) { - JUMPTO(oparg); + JUMPBY(oparg); DISPATCH(); } err = PyObject_IsTrue(cond); @@ -4090,7 +4090,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int Py_DECREF(cond); } else if (err == 0) - JUMPTO(oparg); + JUMPBY(oparg); else goto error; DISPATCH(); @@ -4105,12 +4105,12 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int DISPATCH(); } if (Py_IsTrue(cond)) { - JUMPTO(oparg); + JUMPBY(oparg); DISPATCH(); } err = PyObject_IsTrue(cond); if (err > 0) { - JUMPTO(oparg); + JUMPBY(oparg); } else if (err == 0) { STACK_SHRINK(1); diff --git a/Python/compile.c b/Python/compile.c index 718b521858b27..3b91566efba29 100644 --- a/Python/compile.c +++ b/Python/compile.c @@ -7672,6 +7672,21 @@ normalize_jumps(struct assembler *a) last->i_opcode = is_forward ? POP_JUMP_FORWARD_IF_TRUE : POP_JUMP_BACKWARD_IF_TRUE; break; + case JUMP_IF_TRUE_OR_POP: + case JUMP_IF_FALSE_OR_POP: + if (!is_forward) { + /* As far as we can tell, the compiler never emits + * these jumps with a backwards target. If/when this + * exception is raised, we have found a use case for + * a backwards version of this jump (or to replace + * it with the sequence (COPY 1, POP_JUMP_IF_T/F, POP) + */ + PyErr_Format(PyExc_SystemError, + "unexpected %s jumping backwards", + last->i_opcode == JUMP_IF_TRUE_OR_POP ? + "JUMP_IF_TRUE_OR_POP" : "JUMP_IF_FALSE_OR_POP"); + } + break; } } } From webhook-mailer at python.org Fri Apr 15 15:33:06 2022 From: webhook-mailer at python.org (brettcannon) Date: Fri, 15 Apr 2022 19:33:06 -0000 Subject: [Python-checkins] gh-91217: deprecate nntplib (GH-91543) Message-ID: https://github.com/python/cpython/commit/c9e231de8551ab6d06c92dfa95033150e52d7f1f commit: c9e231de8551ab6d06c92dfa95033150e52d7f1f branch: main author: Brett Cannon committer: brettcannon date: 2022-04-15T12:32:56-07:00 summary: gh-91217: deprecate nntplib (GH-91543) files: A Misc/NEWS.d/next/Library/2022-04-12-20-19-10.gh-issue-91217.acd4h9.rst M Doc/whatsnew/3.11.rst M Lib/nntplib.py M Lib/test/support/socket_helper.py M Lib/test/test_nntplib.py diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index ca76efc963719..b5b2a7648c55e 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -875,6 +875,7 @@ Deprecated * :mod:`crypt` * :mod:`imghdr` * :mod:`msilib` + * :mod:`nntplib` (Contributed by Brett Cannon in :issue:`47061`.) diff --git a/Lib/nntplib.py b/Lib/nntplib.py index f6e746e7c95c5..dddea059982be 100644 --- a/Lib/nntplib.py +++ b/Lib/nntplib.py @@ -68,6 +68,7 @@ import collections import datetime import sys +import warnings try: import ssl @@ -85,6 +86,8 @@ "decode_header", ] +warnings._deprecated(__name__, remove=(3, 13)) + # maximal line length when calling readline(). This is to prevent # reading arbitrary length lines. RFC 3977 limits NNTP line length to # 512 characters, including CRLF. We have selected 2048 just to be on diff --git a/Lib/test/support/socket_helper.py b/Lib/test/support/socket_helper.py index 0ee7a5d69a1b3..754af181ec922 100644 --- a/Lib/test/support/socket_helper.py +++ b/Lib/test/support/socket_helper.py @@ -5,7 +5,7 @@ import sys from .. import support - +from . import warnings_helper HOST = "localhost" HOSTv4 = "127.0.0.1" @@ -190,7 +190,7 @@ def get_socket_conn_refused_errs(): def transient_internet(resource_name, *, timeout=_NOT_SET, errnos=()): """Return a context manager that raises ResourceDenied when various issues with the internet connection manifest themselves as exceptions.""" - import nntplib + nntplib = warnings_helper.import_deprecated("nntplib") import urllib.error if timeout is _NOT_SET: timeout = support.INTERNET_TIMEOUT diff --git a/Lib/test/test_nntplib.py b/Lib/test/test_nntplib.py index 4f0592188f844..9812c05519351 100644 --- a/Lib/test/test_nntplib.py +++ b/Lib/test/test_nntplib.py @@ -5,13 +5,13 @@ import unittest import functools import contextlib -import nntplib import os.path import re import threading from test import support -from test.support import socket_helper +from test.support import socket_helper, warnings_helper +nntplib = warnings_helper.import_deprecated("nntplib") from nntplib import NNTP, GroupInfo from unittest.mock import patch try: diff --git a/Misc/NEWS.d/next/Library/2022-04-12-20-19-10.gh-issue-91217.acd4h9.rst b/Misc/NEWS.d/next/Library/2022-04-12-20-19-10.gh-issue-91217.acd4h9.rst new file mode 100644 index 0000000000000..4a74b9d085142 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-12-20-19-10.gh-issue-91217.acd4h9.rst @@ -0,0 +1 @@ +Deprecate nntplib. From webhook-mailer at python.org Fri Apr 15 15:59:11 2022 From: webhook-mailer at python.org (miss-islington) Date: Fri, 15 Apr 2022 19:59:11 -0000 Subject: [Python-checkins] gh-91487: Optimize asyncio UDP speed (GH-91488) Message-ID: https://github.com/python/cpython/commit/42fabc3ea767f10989363536eaaa9da32616ab57 commit: 42fabc3ea767f10989363536eaaa9da32616ab57 branch: main author: msoxzw <56633971+msoxzw at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-15T12:59:01-07:00 summary: gh-91487: Optimize asyncio UDP speed (GH-91488) Fix #91487 When transferring a small file, e.g. 256 KiB, the speed of this PR is comparable. However, if a large file, e.g. 65536 KiB, is transferred, asyncio UDP will be over 100 times faster than the original. The speed is presumably significantly faster if a larger file is transferred, e.g. 1048576 KiB. Automerge-Triggered-By: GH:gpshead files: A Misc/NEWS.d/next/Library/2022-04-15-19-34-02.gh-issue-91487.2aqguF.rst M Lib/asyncio/proactor_events.py M Lib/asyncio/selector_events.py diff --git a/Lib/asyncio/proactor_events.py b/Lib/asyncio/proactor_events.py index ff6d08f78eecb..9636c6b4d28fa 100644 --- a/Lib/asyncio/proactor_events.py +++ b/Lib/asyncio/proactor_events.py @@ -459,6 +459,7 @@ def __init__(self, loop, sock, protocol, address=None, waiter=None, extra=None): self._address = address self._empty_waiter = None + self._buffer_size = 0 # We don't need to call _protocol.connection_made() since our base # constructor does it for us. super().__init__(loop, sock, protocol, waiter=waiter, extra=extra) @@ -471,7 +472,7 @@ def _set_extra(self, sock): _set_socket_extra(self, sock) def get_write_buffer_size(self): - return sum(len(data) for data, _ in self._buffer) + return self._buffer_size def abort(self): self._force_close(None) @@ -496,6 +497,7 @@ def sendto(self, data, addr=None): # Ensure that what we buffer is immutable. self._buffer.append((bytes(data), addr)) + self._buffer_size += len(data) if self._write_fut is None: # No current write operations are active, kick one off @@ -522,6 +524,7 @@ def _loop_writing(self, fut=None): return data, addr = self._buffer.popleft() + self._buffer_size -= len(data) if self._address is not None: self._write_fut = self._loop._proactor.send(self._sock, data) diff --git a/Lib/asyncio/selector_events.py b/Lib/asyncio/selector_events.py index e99a50395e7cb..c9bbe2ac01435 100644 --- a/Lib/asyncio/selector_events.py +++ b/Lib/asyncio/selector_events.py @@ -1131,6 +1131,7 @@ def __init__(self, loop, sock, protocol, address=None, waiter=None, extra=None): super().__init__(loop, sock, protocol, extra) self._address = address + self._buffer_size = 0 self._loop.call_soon(self._protocol.connection_made, self) # only start reading when connection_made() has been called self._loop.call_soon(self._add_reader, @@ -1141,7 +1142,7 @@ def __init__(self, loop, sock, protocol, address=None, waiter, None) def get_write_buffer_size(self): - return sum(len(data) for data, _ in self._buffer) + return self._buffer_size def _read_ready(self): if self._conn_lost: @@ -1200,11 +1201,13 @@ def sendto(self, data, addr=None): # Ensure that what we buffer is immutable. self._buffer.append((bytes(data), addr)) + self._buffer_size += len(data) self._maybe_pause_protocol() def _sendto_ready(self): while self._buffer: data, addr = self._buffer.popleft() + self._buffer_size -= len(data) try: if self._extra['peername']: self._sock.send(data) @@ -1212,6 +1215,7 @@ def _sendto_ready(self): self._sock.sendto(data, addr) except (BlockingIOError, InterruptedError): self._buffer.appendleft((data, addr)) # Try again later. + self._buffer_size += len(data) break except OSError as exc: self._protocol.error_received(exc) diff --git a/Misc/NEWS.d/next/Library/2022-04-15-19-34-02.gh-issue-91487.2aqguF.rst b/Misc/NEWS.d/next/Library/2022-04-15-19-34-02.gh-issue-91487.2aqguF.rst new file mode 100644 index 0000000000000..d3c41a7e4cbc1 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-15-19-34-02.gh-issue-91487.2aqguF.rst @@ -0,0 +1 @@ +Optimize asyncio UDP speed, over 100 times faster when transferring a large file. From webhook-mailer at python.org Fri Apr 15 21:18:28 2022 From: webhook-mailer at python.org (ezio-melotti) Date: Sat, 16 Apr 2022 01:18:28 -0000 Subject: [Python-checkins] Add link to documentation translation list (#91560) Message-ID: https://github.com/python/cpython/commit/c4e8a93eb3fa5e5d930cea64f213443242c2588c commit: c4e8a93eb3fa5e5d930cea64f213443242c2588c branch: main author: slateny <46876382+slateny at users.noreply.github.com> committer: ezio-melotti date: 2022-04-16T03:18:10+02:00 summary: Add link to documentation translation list (#91560) files: M Doc/bugs.rst diff --git a/Doc/bugs.rst b/Doc/bugs.rst index b32d359d028501..69d7c27410d56a 100644 --- a/Doc/bugs.rst +++ b/Doc/bugs.rst @@ -35,6 +35,10 @@ though it may take a while to be processed. `Helping with Documentation `_ Comprehensive guide for individuals that are interested in contributing to Python documentation. + `Documentation Translations `_ + A list of GitHub pages for documentation translation and their primary contacts. + + .. _using-the-tracker: Using the Python issue tracker From webhook-mailer at python.org Fri Apr 15 21:21:02 2022 From: webhook-mailer at python.org (ezio-melotti) Date: Sat, 16 Apr 2022 01:21:02 -0000 Subject: [Python-checkins] Add minimal issue templates (#91569) Message-ID: https://github.com/python/cpython/commit/1169b0b3ca1812b97382ee16a2fad4b0e7506eaf commit: 1169b0b3ca1812b97382ee16a2fad4b0e7506eaf branch: main author: Alex Waygood committer: ezio-melotti date: 2022-04-16T03:20:54+02:00 summary: Add minimal issue templates (#91569) * Add minimal issue templates * Wording tweaks * Apply suggestions from code review Co-authored-by: Hugo van Kemenade * Improve words in `security` template * Update bug.md * Update crash.md * Add link to security vulnerability website from first page * Never edit on your phone * Apply suggestions from code review Co-authored-by: Hugo van Kemenade * Update .github/ISSUE_TEMPLATE/config.yml Co-authored-by: Hugo van Kemenade * Apply suggestions from code review Co-authored-by: Hugo van Kemenade * Apply suggestions from code review Co-authored-by: Ezio Melotti Co-authored-by: Erlend Egeberg Aasland * There might not be a traceback if there's a crash Co-authored-by: Ezio Melotti * Update .github/ISSUE_TEMPLATE/config.yml Co-authored-by: Ezio Melotti * Reorder `config.yml` file * Fix Erlend's nits * version -> architecture * Apply suggestions from code review Co-authored-by: Ezio Melotti Co-authored-by: Hugo van Kemenade Co-authored-by: Ezio Melotti Co-authored-by: Erlend Egeberg Aasland files: A .github/ISSUE_TEMPLATE/bug.md A .github/ISSUE_TEMPLATE/config.yml A .github/ISSUE_TEMPLATE/crash.md A .github/ISSUE_TEMPLATE/documentation.md A .github/ISSUE_TEMPLATE/feature.md A .github/ISSUE_TEMPLATE/security.md diff --git a/.github/ISSUE_TEMPLATE/bug.md b/.github/ISSUE_TEMPLATE/bug.md new file mode 100644 index 0000000000000..7bdca2112b287 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug.md @@ -0,0 +1,32 @@ +--- +name: Bug report +about: Submit a bug report +labels: "type-bug" +--- + + + +**Bug report** + +A clear and concise description of what the bug is. +Include a minimal, reproducible example (https://stackoverflow.com/help/minimal-reproducible-example), if possible. + +**Your environment** + + + +- CPython versions tested on: +- Operating system and architecture: + + diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml new file mode 100644 index 0000000000000..547c724113985 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -0,0 +1,10 @@ +contact_links: + - name: "Getting help" + about: "Ask questions about using Python and debugging errors on Discourse." + url: "https://discuss.python.org/c/users/7" + - name: "Proposing new features" + about: "Submit major feature proposal (e.g. syntax changes) to an ideas forum first." + url: "https://discuss.python.org/c/ideas/6" + - name: "Reporting security vulnerabilities" + about: "See here for details on reporting security vulnerabilities in CPython" + url: "https://www.python.org/dev/security/" diff --git a/.github/ISSUE_TEMPLATE/crash.md b/.github/ISSUE_TEMPLATE/crash.md new file mode 100644 index 0000000000000..28d7bfec19381 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/crash.md @@ -0,0 +1,33 @@ +--- +name: Crash report +about: A hard crash of the interpreter, possibly with a core dump +labels: "type-crash" +--- + + + +**Crash report** + +Tell us what happened, ideally including a minimal, reproducible example (https://stackoverflow.com/help/minimal-reproducible-example). + +**Error messages** + +Enter any relevant error message caused by the crash, including a core dump if there is one. + +**Your environment** + + + +- CPython versions tested on: +- Operating system and architecture: + + diff --git a/.github/ISSUE_TEMPLATE/documentation.md b/.github/ISSUE_TEMPLATE/documentation.md new file mode 100644 index 0000000000000..669c92adb4753 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/documentation.md @@ -0,0 +1,9 @@ +--- +name: Documentation +about: Report a problem with the documentation +labels: "docs" +--- + +**Documentation** + +(A clear and concise description of the issue.) diff --git a/.github/ISSUE_TEMPLATE/feature.md b/.github/ISSUE_TEMPLATE/feature.md new file mode 100644 index 0000000000000..75ef03bea675c --- /dev/null +++ b/.github/ISSUE_TEMPLATE/feature.md @@ -0,0 +1,28 @@ +--- +name: Feature or enhancement +about: Submit a proposal for a new CPython feature or enhancement +labels: "type-feature" +--- + +**Feature or enhancement** + +(A clear and concise description of your proposal.) + +**Pitch** + +(Explain why this feature or enhacement should be implemented and how it would be used. + Add examples, if applicable.) + +**Previous discussion** + + + + + \ No newline at end of file diff --git a/.github/ISSUE_TEMPLATE/security.md b/.github/ISSUE_TEMPLATE/security.md new file mode 100644 index 0000000000000..fd45cae2d28d2 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/security.md @@ -0,0 +1,25 @@ +--- +name: Security vulnerability report +about: Submit a report regarding a security vulnerability +labels: "type-security" +--- + + + +**Security vulnerability report** + +A description of what the security vulnerability is. + +**Your environment** + + + +- CPython versions tested on: +- Operating system and architecture: + + From webhook-mailer at python.org Fri Apr 15 22:07:16 2022 From: webhook-mailer at python.org (ezio-melotti) Date: Sat, 16 Apr 2022 02:07:16 -0000 Subject: [Python-checkins] Issue templates: Remove duplicate links to security policy (#91590) Message-ID: https://github.com/python/cpython/commit/4e661cd69164318c1f871faa476c68a04092ddc4 commit: 4e661cd69164318c1f871faa476c68a04092ddc4 branch: main author: Alex Waygood committer: ezio-melotti date: 2022-04-16T04:06:51+02:00 summary: Issue templates: Remove duplicate links to security policy (#91590) * Remove security link from config file * Delete security.md files: D .github/ISSUE_TEMPLATE/security.md M .github/ISSUE_TEMPLATE/config.yml diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml index 547c724113985..75d174307ce16 100644 --- a/.github/ISSUE_TEMPLATE/config.yml +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -5,6 +5,3 @@ contact_links: - name: "Proposing new features" about: "Submit major feature proposal (e.g. syntax changes) to an ideas forum first." url: "https://discuss.python.org/c/ideas/6" - - name: "Reporting security vulnerabilities" - about: "See here for details on reporting security vulnerabilities in CPython" - url: "https://www.python.org/dev/security/" diff --git a/.github/ISSUE_TEMPLATE/security.md b/.github/ISSUE_TEMPLATE/security.md deleted file mode 100644 index fd45cae2d28d2..0000000000000 --- a/.github/ISSUE_TEMPLATE/security.md +++ /dev/null @@ -1,25 +0,0 @@ ---- -name: Security vulnerability report -about: Submit a report regarding a security vulnerability -labels: "type-security" ---- - - - -**Security vulnerability report** - -A description of what the security vulnerability is. - -**Your environment** - - - -- CPython versions tested on: -- Operating system and architecture: - - From webhook-mailer at python.org Fri Apr 15 22:15:11 2022 From: webhook-mailer at python.org (ezio-melotti) Date: Sat, 16 Apr 2022 02:15:11 -0000 Subject: [Python-checkins] Add link to documentation translation list (GH-91560) (#91588) Message-ID: https://github.com/python/cpython/commit/e0fc81ced91113ae3336c688936014cccd089679 commit: e0fc81ced91113ae3336c688936014cccd089679 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ezio-melotti date: 2022-04-16T04:15:03+02:00 summary: Add link to documentation translation list (GH-91560) (#91588) (cherry picked from commit c4e8a93eb3fa5e5d930cea64f213443242c2588c) Co-authored-by: slateny <46876382+slateny at users.noreply.github.com> Co-authored-by: slateny <46876382+slateny at users.noreply.github.com> files: M Doc/bugs.rst diff --git a/Doc/bugs.rst b/Doc/bugs.rst index 192995ab06344..6654a23c060e3 100644 --- a/Doc/bugs.rst +++ b/Doc/bugs.rst @@ -35,6 +35,10 @@ though it may take a while to be processed. `Helping with Documentation `_ Comprehensive guide for individuals that are interested in contributing to Python documentation. + `Documentation Translations `_ + A list of GitHub pages for documentation translation and their primary contacts. + + .. _using-the-tracker: Using the Python issue tracker From webhook-mailer at python.org Fri Apr 15 22:16:09 2022 From: webhook-mailer at python.org (ezio-melotti) Date: Sat, 16 Apr 2022 02:16:09 -0000 Subject: [Python-checkins] Add link to documentation translation list (GH-91560) (#91589) Message-ID: https://github.com/python/cpython/commit/2e1f9693333534e7de73bb8790b12572f83f7129 commit: 2e1f9693333534e7de73bb8790b12572f83f7129 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ezio-melotti date: 2022-04-16T04:16:05+02:00 summary: Add link to documentation translation list (GH-91560) (#91589) (cherry picked from commit c4e8a93eb3fa5e5d930cea64f213443242c2588c) Co-authored-by: slateny <46876382+slateny at users.noreply.github.com> Co-authored-by: slateny <46876382+slateny at users.noreply.github.com> files: M Doc/bugs.rst diff --git a/Doc/bugs.rst b/Doc/bugs.rst index b3d057797c256..0feddeb5793fc 100644 --- a/Doc/bugs.rst +++ b/Doc/bugs.rst @@ -35,6 +35,10 @@ though it may take a while to be processed. `Helping with Documentation `_ Comprehensive guide for individuals that are interested in contributing to Python documentation. + `Documentation Translations `_ + A list of GitHub pages for documentation translation and their primary contacts. + + .. _using-the-tracker: Using the Python issue tracker From webhook-mailer at python.org Sat Apr 16 00:21:17 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Sat, 16 Apr 2022 04:21:17 -0000 Subject: [Python-checkins] gh-69093: Add context manager support to sqlite3.Blob (GH-91562) Message-ID: https://github.com/python/cpython/commit/a8617566759a07c67d14c9b6ed663e32d3b3f5e7 commit: a8617566759a07c67d14c9b6ed663e32d3b3f5e7 branch: main author: Erlend Egeberg Aasland committer: JelleZijlstra date: 2022-04-15T21:21:12-07:00 summary: gh-69093: Add context manager support to sqlite3.Blob (GH-91562) files: A Misc/NEWS.d/next/Library/2022-04-14-00-59-01.gh-issue-69093.bmlMwI.rst M Doc/includes/sqlite3/blob.py M Doc/library/sqlite3.rst M Lib/test/test_sqlite3/test_dbapi.py M Modules/_sqlite/blob.c M Modules/_sqlite/clinic/blob.c.h diff --git a/Doc/includes/sqlite3/blob.py b/Doc/includes/sqlite3/blob.py index 61994fb82dd72a..b3694ad08af46b 100644 --- a/Doc/includes/sqlite3/blob.py +++ b/Doc/includes/sqlite3/blob.py @@ -4,9 +4,13 @@ con.execute("create table test(blob_col blob)") con.execute("insert into test(blob_col) values (zeroblob(10))") -blob = con.blobopen("test", "blob_col", 1) -blob.write(b"Hello") -blob.write(b"World") -blob.seek(0) -print(blob.read()) # will print b"HelloWorld" -blob.close() +# Write to our blob, using two write operations: +with con.blobopen("test", "blob_col", 1) as blob: + blob.write(b"Hello") + blob.write(b"World") + +# Read the contents of our blob +with con.blobopen("test", "blob_col", 1) as blob: + greeting = blob.read() + +print(greeting) # outputs "b'HelloWorld'" diff --git a/Doc/library/sqlite3.rst b/Doc/library/sqlite3.rst index d0274fb79744d4..4838db01669e66 100644 --- a/Doc/library/sqlite3.rst +++ b/Doc/library/sqlite3.rst @@ -1115,6 +1115,11 @@ Blob Objects data in an SQLite :abbr:`BLOB (Binary Large OBject)`. Call ``len(blob)`` to get the size (number of bytes) of the blob. + Use the :class:`Blob` as a :term:`context manager` to ensure that the blob + handle is closed after use. + + .. literalinclude:: ../includes/sqlite3/blob.py + .. method:: close() Close the blob. @@ -1149,10 +1154,6 @@ Blob Objects current position) and :data:`os.SEEK_END` (seek relative to the blob?s end). - :class:`Blob` example: - - .. literalinclude:: ../includes/sqlite3/blob.py - .. _sqlite3-types: diff --git a/Lib/test/test_sqlite3/test_dbapi.py b/Lib/test/test_sqlite3/test_dbapi.py index b010813fff7c58..79dcb3ef8954a0 100644 --- a/Lib/test/test_sqlite3/test_dbapi.py +++ b/Lib/test/test_sqlite3/test_dbapi.py @@ -1170,6 +1170,25 @@ def test_blob_sequence_not_supported(self): with self.assertRaises(TypeError): b"a" in self.blob + def test_blob_context_manager(self): + data = b"a" * 50 + with self.cx.blobopen("test", "b", 1) as blob: + blob.write(data) + actual = self.cx.execute("select b from test").fetchone()[0] + self.assertEqual(actual, data) + + # Check that __exit__ closed the blob + with self.assertRaisesRegex(sqlite.ProgrammingError, "closed blob"): + blob.read() + + def test_blob_context_manager_reraise_exceptions(self): + class DummyException(Exception): + pass + with self.assertRaisesRegex(DummyException, "reraised"): + with self.cx.blobopen("test", "b", 1) as blob: + raise DummyException("reraised") + + def test_blob_closed(self): with memory_database() as cx: cx.execute("create table test(b blob)") @@ -1186,6 +1205,10 @@ def test_blob_closed(self): blob.seek(0) with self.assertRaisesRegex(sqlite.ProgrammingError, msg): blob.tell() + with self.assertRaisesRegex(sqlite.ProgrammingError, msg): + blob.__enter__() + with self.assertRaisesRegex(sqlite.ProgrammingError, msg): + blob.__exit__(None, None, None) def test_blob_closed_db_read(self): with memory_database() as cx: diff --git a/Misc/NEWS.d/next/Library/2022-04-14-00-59-01.gh-issue-69093.bmlMwI.rst b/Misc/NEWS.d/next/Library/2022-04-14-00-59-01.gh-issue-69093.bmlMwI.rst new file mode 100644 index 00000000000000..d45a139b50e821 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-14-00-59-01.gh-issue-69093.bmlMwI.rst @@ -0,0 +1,2 @@ +Add :term:`context manager` support to :class:`sqlite3.Blob`. +Patch by Aviv Palivoda and Erlend E. Aasland. diff --git a/Modules/_sqlite/blob.c b/Modules/_sqlite/blob.c index c4f8be45b2f941..3f766302d62517 100644 --- a/Modules/_sqlite/blob.c +++ b/Modules/_sqlite/blob.c @@ -307,8 +307,51 @@ blob_tell_impl(pysqlite_Blob *self) } +/*[clinic input] +_sqlite3.Blob.__enter__ as blob_enter + +Blob context manager enter. +[clinic start generated code]*/ + +static PyObject * +blob_enter_impl(pysqlite_Blob *self) +/*[clinic end generated code: output=4fd32484b071a6cd input=fe4842c3c582d5a7]*/ +{ + if (!check_blob(self)) { + return NULL; + } + return Py_NewRef(self); +} + + +/*[clinic input] +_sqlite3.Blob.__exit__ as blob_exit + + type: object + val: object + tb: object + / + +Blob context manager exit. +[clinic start generated code]*/ + +static PyObject * +blob_exit_impl(pysqlite_Blob *self, PyObject *type, PyObject *val, + PyObject *tb) +/*[clinic end generated code: output=fc86ceeb2b68c7b2 input=575d9ecea205f35f]*/ +{ + if (!check_blob(self)) { + return NULL; + } + close_blob(self); + Py_RETURN_FALSE; +} + + static PyMethodDef blob_methods[] = { BLOB_CLOSE_METHODDEF + BLOB_ENTER_METHODDEF + BLOB_EXIT_METHODDEF BLOB_READ_METHODDEF BLOB_SEEK_METHODDEF BLOB_TELL_METHODDEF diff --git a/Modules/_sqlite/clinic/blob.c.h b/Modules/_sqlite/clinic/blob.c.h index 30b3e3c194739f..237877a9b37f13 100644 --- a/Modules/_sqlite/clinic/blob.c.h +++ b/Modules/_sqlite/clinic/blob.c.h @@ -162,4 +162,55 @@ blob_tell(pysqlite_Blob *self, PyObject *Py_UNUSED(ignored)) { return blob_tell_impl(self); } -/*[clinic end generated code: output=d3a02b127f2cfa58 input=a9049054013a1b77]*/ + +PyDoc_STRVAR(blob_enter__doc__, +"__enter__($self, /)\n" +"--\n" +"\n" +"Blob context manager enter."); + +#define BLOB_ENTER_METHODDEF \ + {"__enter__", (PyCFunction)blob_enter, METH_NOARGS, blob_enter__doc__}, + +static PyObject * +blob_enter_impl(pysqlite_Blob *self); + +static PyObject * +blob_enter(pysqlite_Blob *self, PyObject *Py_UNUSED(ignored)) +{ + return blob_enter_impl(self); +} + +PyDoc_STRVAR(blob_exit__doc__, +"__exit__($self, type, val, tb, /)\n" +"--\n" +"\n" +"Blob context manager exit."); + +#define BLOB_EXIT_METHODDEF \ + {"__exit__", (PyCFunction)(void(*)(void))blob_exit, METH_FASTCALL, blob_exit__doc__}, + +static PyObject * +blob_exit_impl(pysqlite_Blob *self, PyObject *type, PyObject *val, + PyObject *tb); + +static PyObject * +blob_exit(pysqlite_Blob *self, PyObject *const *args, Py_ssize_t nargs) +{ + PyObject *return_value = NULL; + PyObject *type; + PyObject *val; + PyObject *tb; + + if (!_PyArg_CheckPositional("__exit__", nargs, 3, 3)) { + goto exit; + } + type = args[0]; + val = args[1]; + tb = args[2]; + return_value = blob_exit_impl(self, type, val, tb); + +exit: + return return_value; +} +/*[clinic end generated code: output=ca2400862c18dadb input=a9049054013a1b77]*/ From webhook-mailer at python.org Sat Apr 16 00:23:14 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Sat, 16 Apr 2022 04:23:14 -0000 Subject: [Python-checkins] gh-82849: revise intro to os.path.rst (GH-32232) Message-ID: https://github.com/python/cpython/commit/468314cc8bfdb6fd328cbbbb7d0807728f25e043 commit: 468314cc8bfdb6fd328cbbbb7d0807728f25e043 branch: main author: Jack DeVries committer: JelleZijlstra date: 2022-04-15T21:23:07-07:00 summary: gh-82849: revise intro to os.path.rst (GH-32232) * revise the first paragraph of docs for os.path * add a mention of `os.PathLike` protocol * remove warnings rendered irrelevant by :pep:`383` and :pep:`529` Co-authored-by: Jelle Zijlstra files: A Misc/NEWS.d/next/Documentation/2022-04-01-09-28-31.bpo-38668.j4mrqW.rst M Doc/library/os.path.rst diff --git a/Doc/library/os.path.rst b/Doc/library/os.path.rst index 6b15a113f5450..c201b1460ede3 100644 --- a/Doc/library/os.path.rst +++ b/Doc/library/os.path.rst @@ -11,16 +11,10 @@ -------------- -This module implements some useful functions on pathnames. To read or -write files see :func:`open`, and for accessing the filesystem see the -:mod:`os` module. The path parameters can be passed as either strings, -or bytes. Applications are encouraged to represent file names as -(Unicode) character strings. Unfortunately, some file names may not be -representable as strings on Unix, so applications that need to support -arbitrary file names on Unix should use bytes objects to represent -path names. Vice versa, using bytes objects cannot represent all file -names on Windows (in the standard ``mbcs`` encoding), hence Windows -applications should use string objects to access all files. +This module implements some useful functions on pathnames. To read or write +files see :func:`open`, and for accessing the filesystem see the :mod:`os` +module. The path parameters can be passed as strings, or bytes, or any object +implementing the :class:`os.PathLike` protocol. Unlike a unix shell, Python does not do any *automatic* path expansions. Functions such as :func:`expanduser` and :func:`expandvars` can be invoked @@ -38,7 +32,6 @@ the :mod:`glob` module.) their parameters. The result is an object of the same type, if a path or file name is returned. - .. note:: Since different operating systems have different path name conventions, there diff --git a/Misc/NEWS.d/next/Documentation/2022-04-01-09-28-31.bpo-38668.j4mrqW.rst b/Misc/NEWS.d/next/Documentation/2022-04-01-09-28-31.bpo-38668.j4mrqW.rst new file mode 100644 index 0000000000000..512f0deb3543c --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2022-04-01-09-28-31.bpo-38668.j4mrqW.rst @@ -0,0 +1,3 @@ +Update the introduction to documentation for :mod:`os.path` to remove +warnings that became irrelevant after the implementations of :pep:`383` and +:pep:`529`. From webhook-mailer at python.org Sat Apr 16 00:24:34 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Sat, 16 Apr 2022 04:24:34 -0000 Subject: [Python-checkins] bpo-43224: Add tests for TypeVarTuple substitution in Annotated (GH-31846) Message-ID: https://github.com/python/cpython/commit/f2bc12f0d5297899b57f3fa688b24f3c1d1bee7b commit: f2bc12f0d5297899b57f3fa688b24f3c1d1bee7b branch: main author: Matthew Rahtz committer: JelleZijlstra date: 2022-04-15T21:24:28-07:00 summary: bpo-43224: Add tests for TypeVarTuple substitution in Annotated (GH-31846) files: M Lib/test/test_typing.py M Lib/typing.py diff --git a/Lib/test/test_typing.py b/Lib/test/test_typing.py index 97fc66a2f748f..ffd0592a3b414 100644 --- a/Lib/test/test_typing.py +++ b/Lib/test/test_typing.py @@ -5873,6 +5873,77 @@ def test_subst(self): with self.assertRaises(TypeError): LI[None] + def test_typevar_subst(self): + dec = "a decoration" + Ts = TypeVarTuple('Ts') + T = TypeVar('T') + T1 = TypeVar('T1') + T2 = TypeVar('T2') + + A = Annotated[Tuple[Unpack[Ts]], dec] + self.assertEqual(A[int], Annotated[Tuple[int], dec]) + self.assertEqual(A[str, int], Annotated[Tuple[str, int], dec]) + with self.assertRaises(TypeError): + Annotated[Unpack[Ts], dec] + + B = Annotated[Tuple[T, Unpack[Ts]], dec] + self.assertEqual(B[int], Annotated[Tuple[int], dec]) + self.assertEqual(B[int, str], Annotated[Tuple[int, str], dec]) + self.assertEqual( + B[int, str, float], + Annotated[Tuple[int, str, float], dec] + ) + with self.assertRaises(TypeError): + B[()] + + C = Annotated[Tuple[Unpack[Ts], T], dec] + self.assertEqual(C[int], Annotated[Tuple[int], dec]) + self.assertEqual(C[int, str], Annotated[Tuple[int, str], dec]) + self.assertEqual( + C[int, str, float], + Annotated[Tuple[int, str, float], dec] + ) + with self.assertRaises(TypeError): + C[()] + + D = Annotated[Tuple[T1, Unpack[Ts], T2], dec] + self.assertEqual(D[int, str], Annotated[Tuple[int, str], dec]) + self.assertEqual( + D[int, str, float], + Annotated[Tuple[int, str, float], dec] + ) + self.assertEqual( + D[int, str, bool, float], + Annotated[Tuple[int, str, bool, float], dec] + ) + with self.assertRaises(TypeError): + D[int] + + # Now let's try creating an alias from an alias. + + Ts2 = TypeVarTuple('Ts2') + T3 = TypeVar('T3') + T4 = TypeVar('T4') + + E = D[T3, Unpack[Ts2], T4] + self.assertEqual( + E, + Annotated[Tuple[T3, Unpack[Ts2], T4], dec] + ) + self.assertEqual( + E[int, str], Annotated[Tuple[int, str], dec] + ) + self.assertEqual( + E[int, str, float], + Annotated[Tuple[int, str, float], dec] + ) + self.assertEqual( + E[int, str, bool, float], + Annotated[Tuple[int, str, bool, float], dec] + ) + with self.assertRaises(TypeError): + E[int] + def test_annotated_in_other_types(self): X = List[Annotated[T, 5]] self.assertEqual(X[int], List[Annotated[int, 5]]) diff --git a/Lib/typing.py b/Lib/typing.py index 1b584bea0c3e5..b26adc63cc136 100644 --- a/Lib/typing.py +++ b/Lib/typing.py @@ -2080,6 +2080,17 @@ class Annotated: OptimizedList = Annotated[List[T], runtime.Optimize()] OptimizedList[int] == Annotated[List[int], runtime.Optimize()] + + - Annotated cannot be used with an unpacked TypeVarTuple:: + + Annotated[*Ts, Ann1] # NOT valid + + This would be equivalent to + + Annotated[T1, T2, T3, ..., Ann1] + + where T1, T2 etc. are TypeVars, which would be invalid, because + only one type should be passed to Annotated. """ __slots__ = () @@ -2093,6 +2104,9 @@ def __class_getitem__(cls, params): raise TypeError("Annotated[...] should be used " "with at least two arguments (a type and an " "annotation).") + if _is_unpacked_typevartuple(params[0]): + raise TypeError("Annotated[...] should not be used with an " + "unpacked TypeVarTuple") msg = "Annotated[t, ...]: t must be a type." origin = _type_check(params[0], msg, allow_special_forms=True) metadata = tuple(params[1:]) From webhook-mailer at python.org Sat Apr 16 00:38:26 2022 From: webhook-mailer at python.org (miss-islington) Date: Sat, 16 Apr 2022 04:38:26 -0000 Subject: [Python-checkins] gh-82849: revise intro to os.path.rst (GH-32232) Message-ID: https://github.com/python/cpython/commit/f5542ecf6d340eaaf86f31d90a7a7ff7a99f25a2 commit: f5542ecf6d340eaaf86f31d90a7a7ff7a99f25a2 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-15T21:38:11-07:00 summary: gh-82849: revise intro to os.path.rst (GH-32232) * revise the first paragraph of docs for os.path * add a mention of `os.PathLike` protocol * remove warnings rendered irrelevant by :pep:`383` and :pep:`529` Co-authored-by: Jelle Zijlstra (cherry picked from commit 468314cc8bfdb6fd328cbbbb7d0807728f25e043) Co-authored-by: Jack DeVries files: A Misc/NEWS.d/next/Documentation/2022-04-01-09-28-31.bpo-38668.j4mrqW.rst M Doc/library/os.path.rst diff --git a/Doc/library/os.path.rst b/Doc/library/os.path.rst index 6b15a113f5450..c201b1460ede3 100644 --- a/Doc/library/os.path.rst +++ b/Doc/library/os.path.rst @@ -11,16 +11,10 @@ -------------- -This module implements some useful functions on pathnames. To read or -write files see :func:`open`, and for accessing the filesystem see the -:mod:`os` module. The path parameters can be passed as either strings, -or bytes. Applications are encouraged to represent file names as -(Unicode) character strings. Unfortunately, some file names may not be -representable as strings on Unix, so applications that need to support -arbitrary file names on Unix should use bytes objects to represent -path names. Vice versa, using bytes objects cannot represent all file -names on Windows (in the standard ``mbcs`` encoding), hence Windows -applications should use string objects to access all files. +This module implements some useful functions on pathnames. To read or write +files see :func:`open`, and for accessing the filesystem see the :mod:`os` +module. The path parameters can be passed as strings, or bytes, or any object +implementing the :class:`os.PathLike` protocol. Unlike a unix shell, Python does not do any *automatic* path expansions. Functions such as :func:`expanduser` and :func:`expandvars` can be invoked @@ -38,7 +32,6 @@ the :mod:`glob` module.) their parameters. The result is an object of the same type, if a path or file name is returned. - .. note:: Since different operating systems have different path name conventions, there diff --git a/Misc/NEWS.d/next/Documentation/2022-04-01-09-28-31.bpo-38668.j4mrqW.rst b/Misc/NEWS.d/next/Documentation/2022-04-01-09-28-31.bpo-38668.j4mrqW.rst new file mode 100644 index 0000000000000..512f0deb3543c --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2022-04-01-09-28-31.bpo-38668.j4mrqW.rst @@ -0,0 +1,3 @@ +Update the introduction to documentation for :mod:`os.path` to remove +warnings that became irrelevant after the implementations of :pep:`383` and +:pep:`529`. From webhook-mailer at python.org Sat Apr 16 00:42:31 2022 From: webhook-mailer at python.org (miss-islington) Date: Sat, 16 Apr 2022 04:42:31 -0000 Subject: [Python-checkins] gh-82849: revise intro to os.path.rst (GH-32232) Message-ID: https://github.com/python/cpython/commit/514162b835a86173d63ea83f8fe39b9af32c3fa5 commit: 514162b835a86173d63ea83f8fe39b9af32c3fa5 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-15T21:42:20-07:00 summary: gh-82849: revise intro to os.path.rst (GH-32232) * revise the first paragraph of docs for os.path * add a mention of `os.PathLike` protocol * remove warnings rendered irrelevant by :pep:`383` and :pep:`529` Co-authored-by: Jelle Zijlstra (cherry picked from commit 468314cc8bfdb6fd328cbbbb7d0807728f25e043) Co-authored-by: Jack DeVries files: A Misc/NEWS.d/next/Documentation/2022-04-01-09-28-31.bpo-38668.j4mrqW.rst M Doc/library/os.path.rst diff --git a/Doc/library/os.path.rst b/Doc/library/os.path.rst index 2fb6ea9da8e09..0f91971cd8d07 100644 --- a/Doc/library/os.path.rst +++ b/Doc/library/os.path.rst @@ -11,16 +11,10 @@ -------------- -This module implements some useful functions on pathnames. To read or -write files see :func:`open`, and for accessing the filesystem see the -:mod:`os` module. The path parameters can be passed as either strings, -or bytes. Applications are encouraged to represent file names as -(Unicode) character strings. Unfortunately, some file names may not be -representable as strings on Unix, so applications that need to support -arbitrary file names on Unix should use bytes objects to represent -path names. Vice versa, using bytes objects cannot represent all file -names on Windows (in the standard ``mbcs`` encoding), hence Windows -applications should use string objects to access all files. +This module implements some useful functions on pathnames. To read or write +files see :func:`open`, and for accessing the filesystem see the :mod:`os` +module. The path parameters can be passed as strings, or bytes, or any object +implementing the :class:`os.PathLike` protocol. Unlike a unix shell, Python does not do any *automatic* path expansions. Functions such as :func:`expanduser` and :func:`expandvars` can be invoked @@ -38,7 +32,6 @@ the :mod:`glob` module.) their parameters. The result is an object of the same type, if a path or file name is returned. - .. note:: Since different operating systems have different path name conventions, there diff --git a/Misc/NEWS.d/next/Documentation/2022-04-01-09-28-31.bpo-38668.j4mrqW.rst b/Misc/NEWS.d/next/Documentation/2022-04-01-09-28-31.bpo-38668.j4mrqW.rst new file mode 100644 index 0000000000000..512f0deb3543c --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2022-04-01-09-28-31.bpo-38668.j4mrqW.rst @@ -0,0 +1,3 @@ +Update the introduction to documentation for :mod:`os.path` to remove +warnings that became irrelevant after the implementations of :pep:`383` and +:pep:`529`. From webhook-mailer at python.org Sat Apr 16 09:28:48 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Sat, 16 Apr 2022 13:28:48 -0000 Subject: [Python-checkins] gh-69093: sqlite3 blob doc amendments (GH-91561) Message-ID: https://github.com/python/cpython/commit/95573ade42d1635dd9b69380117cbb47b6790772 commit: 95573ade42d1635dd9b69380117cbb47b6790772 branch: main author: Erlend Egeberg Aasland committer: JelleZijlstra date: 2022-04-16T06:28:38-07:00 summary: gh-69093: sqlite3 blob doc amendments (GH-91561) - document that you cannot open a blob handle in a WITHOUT ROWID table - document the blobopen() positional arguments in the same order as they appear - relocate sqlite3.Blob section files: M Doc/library/sqlite3.rst diff --git a/Doc/library/sqlite3.rst b/Doc/library/sqlite3.rst index 4838db01669e6..5c9e2e868ab63 100644 --- a/Doc/library/sqlite3.rst +++ b/Doc/library/sqlite3.rst @@ -397,9 +397,12 @@ Connection Objects .. method:: blobopen(table, column, row, /, *, readonly=False, name="main") Open a :class:`Blob` handle to the :abbr:`BLOB (Binary Large OBject)` - located in row *row*, column *column*, table *table* of database *name*. + located in table name *table*, column name *column*, and row index *row* + of database *name*. When *readonly* is :const:`True` the blob is opened without write permissions. + Trying to open a blob in a ``WITHOUT ROWID`` table will raise + :exc:`OperationalError`. .. note:: @@ -1044,6 +1047,57 @@ Now we plug :class:`Row` in:: 35.14 +Blob Objects +------------ + +.. versionadded:: 3.11 + +.. class:: Blob + + A :class:`Blob` instance is a :term:`file-like object` that can read and write + data in an SQLite :abbr:`BLOB (Binary Large OBject)`. Call ``len(blob)`` to + get the size (number of bytes) of the blob. + + Use the :class:`Blob` as a :term:`context manager` to ensure that the blob + handle is closed after use. + + .. literalinclude:: ../includes/sqlite3/blob.py + + .. method:: close() + + Close the blob. + + The blob will be unusable from this point onward. An + :class:`~sqlite3.Error` (or subclass) exception will be raised if any + further operation is attempted with the blob. + + .. method:: read(length=-1, /) + + Read *length* bytes of data from the blob at the current offset position. + If the end of the blob is reached, the data up to + :abbr:`EOF (End of File)` will be returned. When *length* is not + specified, or is negative, :meth:`~Blob.read` will read until the end of + the blob. + + .. method:: write(data, /) + + Write *data* to the blob at the current offset. This function cannot + change the blob length. Writing beyond the end of the blob will raise + :exc:`ValueError`. + + .. method:: tell() + + Return the current access position of the blob. + + .. method:: seek(offset, origin=os.SEEK_SET, /) + + Set the current access position of the blob to *offset*. The *origin* + argument defaults to :data:`os.SEEK_SET` (absolute blob positioning). + Other values for *origin* are :data:`os.SEEK_CUR` (seek relative to the + current position) and :data:`os.SEEK_END` (seek relative to the blob?s + end). + + .. _sqlite3-exceptions: Exceptions @@ -1104,57 +1158,6 @@ Exceptions .. _sqlite3-blob-objects: -Blob Objects ------------- - -.. versionadded:: 3.11 - -.. class:: Blob - - A :class:`Blob` instance is a :term:`file-like object` that can read and write - data in an SQLite :abbr:`BLOB (Binary Large OBject)`. Call ``len(blob)`` to - get the size (number of bytes) of the blob. - - Use the :class:`Blob` as a :term:`context manager` to ensure that the blob - handle is closed after use. - - .. literalinclude:: ../includes/sqlite3/blob.py - - .. method:: close() - - Close the blob. - - The blob will be unusable from this point onward. An - :class:`~sqlite3.Error` (or subclass) exception will be raised if any - further operation is attempted with the blob. - - .. method:: read(length=-1, /) - - Read *length* bytes of data from the blob at the current offset position. - If the end of the blob is reached, the data up to - :abbr:`EOF (End of File)` will be returned. When *length* is not - specified, or is negative, :meth:`~Blob.read` will read until the end of - the blob. - - .. method:: write(data, /) - - Write *data* to the blob at the current offset. This function cannot - change the blob length. Writing beyond the end of the blob will raise - :exc:`ValueError`. - - .. method:: tell() - - Return the current access position of the blob. - - .. method:: seek(offset, origin=os.SEEK_SET, /) - - Set the current access position of the blob to *offset*. The *origin* - argument defaults to :data:`os.SEEK_SET` (absolute blob positioning). - Other values for *origin* are :data:`os.SEEK_CUR` (seek relative to the - current position) and :data:`os.SEEK_END` (seek relative to the blob?s - end). - - .. _sqlite3-types: SQLite and Python types From webhook-mailer at python.org Sat Apr 16 10:35:15 2022 From: webhook-mailer at python.org (ezio-melotti) Date: Sat, 16 Apr 2022 14:35:15 -0000 Subject: [Python-checkins] gh-91595: fix the comparison of character and integer by using ord() (#91596) Message-ID: https://github.com/python/cpython/commit/9300b6d72948b94c0924a75ea14c6298156522d0 commit: 9300b6d72948b94c0924a75ea14c6298156522d0 branch: main author: Yu Liu committer: ezio-melotti date: 2022-04-16T16:34:48+02:00 summary: gh-91595: fix the comparison of character and integer by using ord() (#91596) * fix the comparison of character and integer by using ord() * ?? Added by blurb_it. Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> files: A Misc/NEWS.d/next/Library/2022-04-16-05-12-13.gh-issue-91595.CocJBv.rst M Tools/gdb/libpython.py diff --git a/Misc/NEWS.d/next/Library/2022-04-16-05-12-13.gh-issue-91595.CocJBv.rst b/Misc/NEWS.d/next/Library/2022-04-16-05-12-13.gh-issue-91595.CocJBv.rst new file mode 100644 index 0000000000000..637079a6487a4 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-16-05-12-13.gh-issue-91595.CocJBv.rst @@ -0,0 +1 @@ +Fix the comparison of character and integer inside :func:`Tools.gdb.libpython.write_repr`. Patch by Yu Liu. diff --git a/Tools/gdb/libpython.py b/Tools/gdb/libpython.py index 4f7a8bca5fd78..610d13099432c 100755 --- a/Tools/gdb/libpython.py +++ b/Tools/gdb/libpython.py @@ -1418,7 +1418,7 @@ def write_repr(self, out, visited): out.write('\\r') # Map non-printable US ASCII to '\xhh' */ - elif ch < ' ' or ch == 0x7F: + elif ch < ' ' or ord(ch) == 0x7F: out.write('\\x') out.write(hexdigits[(ord(ch) >> 4) & 0x000F]) out.write(hexdigits[ord(ch) & 0x000F]) From webhook-mailer at python.org Sat Apr 16 10:57:15 2022 From: webhook-mailer at python.org (miss-islington) Date: Sat, 16 Apr 2022 14:57:15 -0000 Subject: [Python-checkins] gh-91595: fix the comparison of character and integer by using ord() (GH-91596) Message-ID: https://github.com/python/cpython/commit/c171d757f3892263e163eddd702ae5249308404d commit: c171d757f3892263e163eddd702ae5249308404d branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-16T07:57:07-07:00 summary: gh-91595: fix the comparison of character and integer by using ord() (GH-91596) * fix the comparison of character and integer by using ord() * ?? Added by blurb_it. Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> (cherry picked from commit 9300b6d72948b94c0924a75ea14c6298156522d0) Co-authored-by: Yu Liu files: A Misc/NEWS.d/next/Library/2022-04-16-05-12-13.gh-issue-91595.CocJBv.rst M Tools/gdb/libpython.py diff --git a/Misc/NEWS.d/next/Library/2022-04-16-05-12-13.gh-issue-91595.CocJBv.rst b/Misc/NEWS.d/next/Library/2022-04-16-05-12-13.gh-issue-91595.CocJBv.rst new file mode 100644 index 0000000000000..637079a6487a4 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-16-05-12-13.gh-issue-91595.CocJBv.rst @@ -0,0 +1 @@ +Fix the comparison of character and integer inside :func:`Tools.gdb.libpython.write_repr`. Patch by Yu Liu. diff --git a/Tools/gdb/libpython.py b/Tools/gdb/libpython.py index 45be5ab9bb9b3..cc0cd2fd1f768 100755 --- a/Tools/gdb/libpython.py +++ b/Tools/gdb/libpython.py @@ -1285,7 +1285,7 @@ def write_repr(self, out, visited): out.write('\\r') # Map non-printable US ASCII to '\xhh' */ - elif ch < ' ' or ch == 0x7F: + elif ch < ' ' or ord(ch) == 0x7F: out.write('\\x') out.write(hexdigits[(ord(ch) >> 4) & 0x000F]) out.write(hexdigits[ord(ch) & 0x000F]) From webhook-mailer at python.org Sat Apr 16 10:59:41 2022 From: webhook-mailer at python.org (miss-islington) Date: Sat, 16 Apr 2022 14:59:41 -0000 Subject: [Python-checkins] gh-91595: fix the comparison of character and integer by using ord() (GH-91596) Message-ID: https://github.com/python/cpython/commit/84c279b514141f608cf480905c87d48998e296d1 commit: 84c279b514141f608cf480905c87d48998e296d1 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-16T07:59:32-07:00 summary: gh-91595: fix the comparison of character and integer by using ord() (GH-91596) * fix the comparison of character and integer by using ord() * ?? Added by blurb_it. Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> (cherry picked from commit 9300b6d72948b94c0924a75ea14c6298156522d0) Co-authored-by: Yu Liu files: A Misc/NEWS.d/next/Library/2022-04-16-05-12-13.gh-issue-91595.CocJBv.rst M Tools/gdb/libpython.py diff --git a/Misc/NEWS.d/next/Library/2022-04-16-05-12-13.gh-issue-91595.CocJBv.rst b/Misc/NEWS.d/next/Library/2022-04-16-05-12-13.gh-issue-91595.CocJBv.rst new file mode 100644 index 0000000000000..637079a6487a4 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-16-05-12-13.gh-issue-91595.CocJBv.rst @@ -0,0 +1 @@ +Fix the comparison of character and integer inside :func:`Tools.gdb.libpython.write_repr`. Patch by Yu Liu. diff --git a/Tools/gdb/libpython.py b/Tools/gdb/libpython.py index 270aeb426eb5b..12b519330d8a6 100755 --- a/Tools/gdb/libpython.py +++ b/Tools/gdb/libpython.py @@ -1291,7 +1291,7 @@ def write_repr(self, out, visited): out.write('\\r') # Map non-printable US ASCII to '\xhh' */ - elif ch < ' ' or ch == 0x7F: + elif ch < ' ' or ord(ch) == 0x7F: out.write('\\x') out.write(hexdigits[(ord(ch) >> 4) & 0x000F]) out.write(hexdigits[ord(ch) & 0x000F]) From webhook-mailer at python.org Sat Apr 16 12:01:48 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Sat, 16 Apr 2022 16:01:48 -0000 Subject: [Python-checkins] gh-89263: Add typing.get_overloads (GH-31716) Message-ID: https://github.com/python/cpython/commit/055760ed9e745a3104acbfa8a3b76eb26a72590d commit: 055760ed9e745a3104acbfa8a3b76eb26a72590d branch: main author: Jelle Zijlstra committer: JelleZijlstra date: 2022-04-16T09:01:43-07:00 summary: gh-89263: Add typing.get_overloads (GH-31716) Based on suggestions by Guido van Rossum, Spencer Brown, and Alex Waygood. Co-authored-by: Alex Waygood Co-authored-by: Guido van Rossum Co-authored-by: Ken Jin files: A Misc/NEWS.d/next/Library/2022-03-06-18-15-32.bpo-45100.B_lHu0.rst M Doc/library/typing.rst M Lib/test/test_typing.py M Lib/typing.py diff --git a/Doc/library/typing.rst b/Doc/library/typing.rst index fdd00a277b507..6b2a0934171a2 100644 --- a/Doc/library/typing.rst +++ b/Doc/library/typing.rst @@ -2407,6 +2407,35 @@ Functions and decorators See :pep:`484` for details and comparison with other typing semantics. + .. versionchanged:: 3.11 + Overloaded functions can now be introspected at runtime using + :func:`get_overloads`. + + +.. function:: get_overloads(func) + + Return a sequence of :func:`@overload `-decorated definitions for + *func*. *func* is the function object for the implementation of the + overloaded function. For example, given the definition of ``process`` in + the documentation for :func:`@overload `, + ``get_overloads(process)`` will return a sequence of three function objects + for the three defined overloads. If called on a function with no overloads, + ``get_overloads`` returns an empty sequence. + + ``get_overloads`` can be used for introspecting an overloaded function at + runtime. + + .. versionadded:: 3.11 + + +.. function:: clear_overloads() + + Clear all registered overloads in the internal registry. This can be used + to reclaim the memory used by the registry. + + .. versionadded:: 3.11 + + .. decorator:: final A decorator to indicate to type checkers that the decorated method diff --git a/Lib/test/test_typing.py b/Lib/test/test_typing.py index ffd0592a3b414..d4808474e4fce 100644 --- a/Lib/test/test_typing.py +++ b/Lib/test/test_typing.py @@ -1,5 +1,6 @@ import contextlib import collections +from collections import defaultdict from functools import lru_cache import inspect import pickle @@ -7,9 +8,11 @@ import sys import warnings from unittest import TestCase, main, skipUnless, skip +from unittest.mock import patch from copy import copy, deepcopy from typing import Any, NoReturn, Never, assert_never +from typing import overload, get_overloads, clear_overloads from typing import TypeVar, TypeVarTuple, Unpack, AnyStr from typing import T, KT, VT # Not in __all__. from typing import Union, Optional, Literal @@ -3890,11 +3893,22 @@ def test_or(self): self.assertEqual("x" | X, Union["x", X]) + at lru_cache() +def cached_func(x, y): + return 3 * x + y + + +class MethodHolder: + @classmethod + def clsmethod(cls): ... + @staticmethod + def stmethod(): ... + def method(self): ... + + class OverloadTests(BaseTestCase): def test_overload_fails(self): - from typing import overload - with self.assertRaises(RuntimeError): @overload @@ -3904,8 +3918,6 @@ def blah(): blah() def test_overload_succeeds(self): - from typing import overload - @overload def blah(): pass @@ -3915,6 +3927,58 @@ def blah(): blah() + def set_up_overloads(self): + def blah(): + pass + + overload1 = blah + overload(blah) + + def blah(): + pass + + overload2 = blah + overload(blah) + + def blah(): + pass + + return blah, [overload1, overload2] + + # Make sure we don't clear the global overload registry + @patch("typing._overload_registry", + defaultdict(lambda: defaultdict(dict))) + def test_overload_registry(self): + # The registry starts out empty + self.assertEqual(typing._overload_registry, {}) + + impl, overloads = self.set_up_overloads() + self.assertNotEqual(typing._overload_registry, {}) + self.assertEqual(list(get_overloads(impl)), overloads) + + def some_other_func(): pass + overload(some_other_func) + other_overload = some_other_func + def some_other_func(): pass + self.assertEqual(list(get_overloads(some_other_func)), [other_overload]) + + # Make sure that after we clear all overloads, the registry is + # completely empty. + clear_overloads() + self.assertEqual(typing._overload_registry, {}) + self.assertEqual(get_overloads(impl), []) + + # Querying a function with no overloads shouldn't change the registry. + def the_only_one(): pass + self.assertEqual(get_overloads(the_only_one), []) + self.assertEqual(typing._overload_registry, {}) + + def test_overload_registry_repeated(self): + for _ in range(2): + impl, overloads = self.set_up_overloads() + + self.assertEqual(list(get_overloads(impl)), overloads) + # Definitions needed for features introduced in Python 3.6 diff --git a/Lib/typing.py b/Lib/typing.py index b26adc63cc136..3e0fbdb989155 100644 --- a/Lib/typing.py +++ b/Lib/typing.py @@ -21,6 +21,7 @@ from abc import abstractmethod, ABCMeta import collections +from collections import defaultdict import collections.abc import contextlib import functools @@ -121,9 +122,11 @@ def _idfunc(_, x): 'assert_type', 'assert_never', 'cast', + 'clear_overloads', 'final', 'get_args', 'get_origin', + 'get_overloads', 'get_type_hints', 'is_typeddict', 'LiteralString', @@ -2450,6 +2453,10 @@ def _overload_dummy(*args, **kwds): "by an implementation that is not @overload-ed.") +# {module: {qualname: {firstlineno: func}}} +_overload_registry = defaultdict(functools.partial(defaultdict, dict)) + + def overload(func): """Decorator for overloaded functions/methods. @@ -2475,10 +2482,37 @@ def utf8(value: bytes) -> bytes: ... def utf8(value: str) -> bytes: ... def utf8(value): # implementation goes here + + The overloads for a function can be retrieved at runtime using the + get_overloads() function. """ + # classmethod and staticmethod + f = getattr(func, "__func__", func) + try: + _overload_registry[f.__module__][f.__qualname__][f.__code__.co_firstlineno] = func + except AttributeError: + # Not a normal function; ignore. + pass return _overload_dummy +def get_overloads(func): + """Return all defined overloads for *func* as a sequence.""" + # classmethod and staticmethod + f = getattr(func, "__func__", func) + if f.__module__ not in _overload_registry: + return [] + mod_dict = _overload_registry[f.__module__] + if f.__qualname__ not in mod_dict: + return [] + return list(mod_dict[f.__qualname__].values()) + + +def clear_overloads(): + """Clear all overloads in the registry.""" + _overload_registry.clear() + + def final(f): """A decorator to indicate final methods and final classes. diff --git a/Misc/NEWS.d/next/Library/2022-03-06-18-15-32.bpo-45100.B_lHu0.rst b/Misc/NEWS.d/next/Library/2022-03-06-18-15-32.bpo-45100.B_lHu0.rst new file mode 100644 index 0000000000000..d644557545366 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-03-06-18-15-32.bpo-45100.B_lHu0.rst @@ -0,0 +1,2 @@ +Add :func:`typing.get_overloads` and :func:`typing.clear_overloads`. +Patch by Jelle Zijlstra. From webhook-mailer at python.org Sat Apr 16 13:34:34 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Sat, 16 Apr 2022 17:34:34 -0000 Subject: [Python-checkins] bpo-40676: Use Argument Clinic for csv (where possible) (GH-20200) Message-ID: https://github.com/python/cpython/commit/1adc837bf1d88a110e1d9e3021abc92b7e7dfa8e commit: 1adc837bf1d88a110e1d9e3021abc92b7e7dfa8e branch: main author: Shantanu <12621235+hauntsaninja at users.noreply.github.com> committer: JelleZijlstra date: 2022-04-16T10:34:23-07:00 summary: bpo-40676: Use Argument Clinic for csv (where possible) (GH-20200) files: A Misc/NEWS.d/next/Library/2020-05-19-01-40-51.bpo-40676.yJfq1J.rst A Modules/clinic/_csv.c.h M Modules/_csv.c diff --git a/Misc/NEWS.d/next/Library/2020-05-19-01-40-51.bpo-40676.yJfq1J.rst b/Misc/NEWS.d/next/Library/2020-05-19-01-40-51.bpo-40676.yJfq1J.rst new file mode 100644 index 0000000000000..c67d4dcdb3216 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-05-19-01-40-51.bpo-40676.yJfq1J.rst @@ -0,0 +1,3 @@ +Convert :mod:`csv` to use Argument Clinic for :func:`csv.field_size_limit`, +:func:`csv.get_dialect`, :func:`csv.unregister_dialect` and :func:`csv.list_dialects`. + diff --git a/Modules/_csv.c b/Modules/_csv.c index 991b623d6d6d3..4d68df54a0cb0 100644 --- a/Modules/_csv.c +++ b/Modules/_csv.c @@ -14,6 +14,12 @@ module instead. #include "structmember.h" // PyMemberDef #include +/*[clinic input] +module _csv +[clinic start generated code]*/ +/*[clinic end generated code: output=da39a3ee5e6b4b0d input=385118b71aa43706]*/ + +#include "clinic/_csv.c.h" #define NOT_SET ((Py_UCS4)-1) #define EOL ((Py_UCS4)-2) @@ -1473,8 +1479,18 @@ csv_writer(PyObject *module, PyObject *args, PyObject *keyword_args) /* * DIALECT REGISTRY */ + +/*[clinic input] +_csv.list_dialects + +Return a list of all known dialect names. + + names = csv.list_dialects() +[clinic start generated code]*/ + static PyObject * -csv_list_dialects(PyObject *module, PyObject *args) +_csv_list_dialects_impl(PyObject *module) +/*[clinic end generated code: output=a5b92b215b006a6d input=8953943eb17d98ab]*/ { return PyDict_Keys(get_csv_state(module)->dialects); } @@ -1506,11 +1522,23 @@ csv_register_dialect(PyObject *module, PyObject *args, PyObject *kwargs) Py_RETURN_NONE; } + +/*[clinic input] +_csv.unregister_dialect + + name: object + +Delete the name/dialect mapping associated with a string name. + + csv.unregister_dialect(name) +[clinic start generated code]*/ + static PyObject * -csv_unregister_dialect(PyObject *module, PyObject *name_obj) +_csv_unregister_dialect_impl(PyObject *module, PyObject *name) +/*[clinic end generated code: output=0813ebca6c058df4 input=6b5c1557bf60c7e7]*/ { _csvstate *module_state = get_csv_state(module); - if (PyDict_DelItem(module_state->dialects, name_obj) < 0) { + if (PyDict_DelItem(module_state->dialects, name) < 0) { if (PyErr_ExceptionMatches(PyExc_KeyError)) { PyErr_Format(module_state->error_obj, "unknown dialect"); } @@ -1519,21 +1547,42 @@ csv_unregister_dialect(PyObject *module, PyObject *name_obj) Py_RETURN_NONE; } +/*[clinic input] +_csv.get_dialect + + name: object + +Return the dialect instance associated with name. + + dialect = csv.get_dialect(name) +[clinic start generated code]*/ + static PyObject * -csv_get_dialect(PyObject *module, PyObject *name_obj) +_csv_get_dialect_impl(PyObject *module, PyObject *name) +/*[clinic end generated code: output=aa988cd573bebebb input=edf9ddab32e448fb]*/ { - return get_dialect_from_registry(name_obj, get_csv_state(module)); + return get_dialect_from_registry(name, get_csv_state(module)); } +/*[clinic input] +_csv.field_size_limit + + new_limit: object = NULL + +Sets an upper limit on parsed fields. + + csv.field_size_limit([limit]) + +Returns old limit. If limit is not given, no new limit is set and +the old limit is returned +[clinic start generated code]*/ + static PyObject * -csv_field_size_limit(PyObject *module, PyObject *args) +_csv_field_size_limit_impl(PyObject *module, PyObject *new_limit) +/*[clinic end generated code: output=f2799ecd908e250b input=cec70e9226406435]*/ { - PyObject *new_limit = NULL; _csvstate *module_state = get_csv_state(module); long old_limit = module_state->field_limit; - - if (!PyArg_UnpackTuple(args, "field_size_limit", 0, 1, &new_limit)) - return NULL; if (new_limit != NULL) { if (!PyLong_CheckExact(new_limit)) { PyErr_Format(PyExc_TypeError, @@ -1650,44 +1699,21 @@ PyDoc_STRVAR(csv_writer_doc, "\n" "The \"fileobj\" argument can be any object that supports the file API.\n"); -PyDoc_STRVAR(csv_list_dialects_doc, -"Return a list of all know dialect names.\n" -" names = csv.list_dialects()"); - -PyDoc_STRVAR(csv_get_dialect_doc, -"Return the dialect instance associated with name.\n" -" dialect = csv.get_dialect(name)"); - PyDoc_STRVAR(csv_register_dialect_doc, "Create a mapping from a string name to a dialect class.\n" " dialect = csv.register_dialect(name[, dialect[, **fmtparams]])"); -PyDoc_STRVAR(csv_unregister_dialect_doc, -"Delete the name/dialect mapping associated with a string name.\n" -" csv.unregister_dialect(name)"); - -PyDoc_STRVAR(csv_field_size_limit_doc, -"Sets an upper limit on parsed fields.\n" -" csv.field_size_limit([limit])\n" -"\n" -"Returns old limit. If limit is not given, no new limit is set and\n" -"the old limit is returned"); - static struct PyMethodDef csv_methods[] = { { "reader", (PyCFunction)(void(*)(void))csv_reader, METH_VARARGS | METH_KEYWORDS, csv_reader_doc}, { "writer", (PyCFunction)(void(*)(void))csv_writer, METH_VARARGS | METH_KEYWORDS, csv_writer_doc}, - { "list_dialects", (PyCFunction)csv_list_dialects, - METH_NOARGS, csv_list_dialects_doc}, { "register_dialect", (PyCFunction)(void(*)(void))csv_register_dialect, METH_VARARGS | METH_KEYWORDS, csv_register_dialect_doc}, - { "unregister_dialect", (PyCFunction)csv_unregister_dialect, - METH_O, csv_unregister_dialect_doc}, - { "get_dialect", (PyCFunction)csv_get_dialect, - METH_O, csv_get_dialect_doc}, - { "field_size_limit", (PyCFunction)csv_field_size_limit, - METH_VARARGS, csv_field_size_limit_doc}, + _CSV_LIST_DIALECTS_METHODDEF + _CSV_UNREGISTER_DIALECT_METHODDEF + _CSV_GET_DIALECT_METHODDEF + _CSV_FIELD_SIZE_LIMIT_METHODDEF { NULL, NULL } }; diff --git a/Modules/clinic/_csv.c.h b/Modules/clinic/_csv.c.h new file mode 100644 index 0000000000000..41c0cdde7482a --- /dev/null +++ b/Modules/clinic/_csv.c.h @@ -0,0 +1,134 @@ +/*[clinic input] +preserve +[clinic start generated code]*/ + +PyDoc_STRVAR(_csv_list_dialects__doc__, +"list_dialects($module, /)\n" +"--\n" +"\n" +"Return a list of all known dialect names.\n" +"\n" +" names = csv.list_dialects()"); + +#define _CSV_LIST_DIALECTS_METHODDEF \ + {"list_dialects", (PyCFunction)_csv_list_dialects, METH_NOARGS, _csv_list_dialects__doc__}, + +static PyObject * +_csv_list_dialects_impl(PyObject *module); + +static PyObject * +_csv_list_dialects(PyObject *module, PyObject *Py_UNUSED(ignored)) +{ + return _csv_list_dialects_impl(module); +} + +PyDoc_STRVAR(_csv_unregister_dialect__doc__, +"unregister_dialect($module, /, name)\n" +"--\n" +"\n" +"Delete the name/dialect mapping associated with a string name.\n" +"\n" +" csv.unregister_dialect(name)"); + +#define _CSV_UNREGISTER_DIALECT_METHODDEF \ + {"unregister_dialect", (PyCFunction)(void(*)(void))_csv_unregister_dialect, METH_FASTCALL|METH_KEYWORDS, _csv_unregister_dialect__doc__}, + +static PyObject * +_csv_unregister_dialect_impl(PyObject *module, PyObject *name); + +static PyObject * +_csv_unregister_dialect(PyObject *module, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) +{ + PyObject *return_value = NULL; + static const char * const _keywords[] = {"name", NULL}; + static _PyArg_Parser _parser = {NULL, _keywords, "unregister_dialect", 0}; + PyObject *argsbuf[1]; + PyObject *name; + + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 1, 1, 0, argsbuf); + if (!args) { + goto exit; + } + name = args[0]; + return_value = _csv_unregister_dialect_impl(module, name); + +exit: + return return_value; +} + +PyDoc_STRVAR(_csv_get_dialect__doc__, +"get_dialect($module, /, name)\n" +"--\n" +"\n" +"Return the dialect instance associated with name.\n" +"\n" +" dialect = csv.get_dialect(name)"); + +#define _CSV_GET_DIALECT_METHODDEF \ + {"get_dialect", (PyCFunction)(void(*)(void))_csv_get_dialect, METH_FASTCALL|METH_KEYWORDS, _csv_get_dialect__doc__}, + +static PyObject * +_csv_get_dialect_impl(PyObject *module, PyObject *name); + +static PyObject * +_csv_get_dialect(PyObject *module, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) +{ + PyObject *return_value = NULL; + static const char * const _keywords[] = {"name", NULL}; + static _PyArg_Parser _parser = {NULL, _keywords, "get_dialect", 0}; + PyObject *argsbuf[1]; + PyObject *name; + + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 1, 1, 0, argsbuf); + if (!args) { + goto exit; + } + name = args[0]; + return_value = _csv_get_dialect_impl(module, name); + +exit: + return return_value; +} + +PyDoc_STRVAR(_csv_field_size_limit__doc__, +"field_size_limit($module, /, new_limit=)\n" +"--\n" +"\n" +"Sets an upper limit on parsed fields.\n" +"\n" +" csv.field_size_limit([limit])\n" +"\n" +"Returns old limit. If limit is not given, no new limit is set and\n" +"the old limit is returned"); + +#define _CSV_FIELD_SIZE_LIMIT_METHODDEF \ + {"field_size_limit", (PyCFunction)(void(*)(void))_csv_field_size_limit, METH_FASTCALL|METH_KEYWORDS, _csv_field_size_limit__doc__}, + +static PyObject * +_csv_field_size_limit_impl(PyObject *module, PyObject *new_limit); + +static PyObject * +_csv_field_size_limit(PyObject *module, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) +{ + PyObject *return_value = NULL; + static const char * const _keywords[] = {"new_limit", NULL}; + static _PyArg_Parser _parser = {NULL, _keywords, "field_size_limit", 0}; + PyObject *argsbuf[1]; + Py_ssize_t noptargs = nargs + (kwnames ? PyTuple_GET_SIZE(kwnames) : 0) - 0; + PyObject *new_limit = NULL; + + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 0, 1, 0, argsbuf); + if (!args) { + goto exit; + } + if (!noptargs) { + goto skip_optional_pos; + } + new_limit = args[0]; +skip_optional_pos: + return_value = _csv_field_size_limit_impl(module, new_limit); + +exit: + return return_value; +} +/*[clinic end generated code: output=d51f3c42e9b88802 input=a9049054013a1b77]*/ From webhook-mailer at python.org Sat Apr 16 13:38:16 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Sat, 16 Apr 2022 17:38:16 -0000 Subject: [Python-checkins] gh-86178: Add wsgiref.types (GH-32335) Message-ID: https://github.com/python/cpython/commit/0ddc63b240340a952692b11dfe0810973393ed11 commit: 0ddc63b240340a952692b11dfe0810973393ed11 branch: main author: Sebastian Rittau committer: JelleZijlstra date: 2022-04-16T10:37:58-07:00 summary: gh-86178: Add wsgiref.types (GH-32335) files: A Lib/wsgiref/types.py A Misc/NEWS.d/next/Library/2022-04-05-17-18-13.bpo-42012.zMocQz.rst M Doc/library/wsgiref.rst M Doc/whatsnew/3.11.rst M Lib/wsgiref/__init__.py diff --git a/Doc/library/wsgiref.rst b/Doc/library/wsgiref.rst index e924448b86d7f..0ca7b003bf4dd 100644 --- a/Doc/library/wsgiref.rst +++ b/Doc/library/wsgiref.rst @@ -23,6 +23,7 @@ an existing framework. be used to add WSGI support to a web server or framework. It provides utilities for manipulating WSGI environment variables and response headers, base classes for implementing WSGI servers, a demo HTTP server that serves WSGI applications, +types for static type checking, and a validation tool that checks WSGI servers and applications for conformance to the WSGI specification (:pep:`3333`). @@ -43,7 +44,9 @@ This module provides a variety of utility functions for working with WSGI environments. A WSGI environment is a dictionary containing HTTP request variables as described in :pep:`3333`. All of the functions taking an *environ* parameter expect a WSGI-compliant dictionary to be supplied; please see -:pep:`3333` for a detailed specification. +:pep:`3333` for a detailed specification and +:data:`~wsgiref.types.WSGIEnvironment` for a type alias that can be used +in type annotations. .. function:: guess_scheme(environ) @@ -150,7 +153,9 @@ also provides these miscellaneous utilities: .. class:: FileWrapper(filelike, blksize=8192) - A wrapper to convert a file-like object to an :term:`iterator`. The resulting objects + A concrete implementation of the :class:`wsgiref.types.FileWrapper` + protocol used to convert a file-like object to an :term:`iterator`. + The resulting objects are :term:`iterable`\ s. As the object is iterated over, the optional *blksize* parameter will be repeatedly passed to the *filelike* object's :meth:`read` method to obtain bytestrings to yield. When :meth:`read` @@ -349,7 +354,8 @@ request. (E.g., using the :func:`shift_path_info` function from .. method:: WSGIRequestHandler.get_environ() - Returns a dictionary containing the WSGI environment for a request. The default + Return a :data:`~wsgiref.types.WSGIEnvironment` dictionary for a + request. The default implementation copies the contents of the :class:`WSGIServer` object's :attr:`base_environ` dictionary attribute and then adds various headers derived from the HTTP request. Each call to this method should return a new dictionary @@ -558,13 +564,15 @@ input, output, and error streams. .. method:: BaseHandler.get_stdin() - Return an input stream object suitable for use as the ``wsgi.input`` of the + Return an object compatible with :class:`~wsgiref.types.InputStream` + suitable for use as the ``wsgi.input`` of the request currently being processed. .. method:: BaseHandler.get_stderr() - Return an output stream object suitable for use as the ``wsgi.errors`` of the + Return an object compatible with :class:`~wsgiref.types.ErrorStream` + suitable for use as the ``wsgi.errors`` of the request currently being processed. @@ -703,8 +711,9 @@ input, output, and error streams. .. attribute:: BaseHandler.wsgi_file_wrapper - A ``wsgi.file_wrapper`` factory, or ``None``. The default value of this - attribute is the :class:`wsgiref.util.FileWrapper` class. + A ``wsgi.file_wrapper`` factory, compatible with + :class:`wsgiref.types.FileWrapper`, or ``None``. The default value + of this attribute is the :class:`wsgiref.util.FileWrapper` class. .. method:: BaseHandler.sendfile() @@ -754,6 +763,51 @@ input, output, and error streams. .. versionadded:: 3.2 +:mod:`wsgiref.types` -- WSGI types for static type checking +----------------------------------------------------------- + +.. module:: wsgiref.types + :synopsis: WSGI types for static type checking + + +This module provides various types for static type checking as described +in :pep:`3333`. + +.. versionadded:: 3.11 + + +.. class:: StartResponse() + + A :class:`typing.Protocol` describing `start_response() + `_ + callables (:pep:`3333`). + +.. data:: WSGIEnvironment + + A type alias describing a WSGI environment dictionary. + +.. data:: WSGIApplication + + A type alias describing a WSGI application callable. + +.. class:: InputStream() + + A :class:`typing.Protocol` describing a `WSGI Input Stream + `_. + +.. class:: ErrorStream() + + A :class:`typing.Protocol` describing a `WSGI Error Stream + `_. + +.. class:: FileWrapper() + + A :class:`typing.Protocol` describing a `file wrapper + `_. + See :class:`wsgiref.util.FileWrapper` for a concrete implementation of this + protocol. + + Examples -------- diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index b5b2a7648c55e..19e3d2f92fb94 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -233,6 +233,10 @@ New Modules * A new module, :mod:`tomllib`, was added for parsing TOML. (Contributed by Taneli Hukkinen in :issue:`40059`.) +* :mod:`wsgiref.types`, containing WSGI-specific types for static type + checking, was added. + (Contributed by Sebastian Rittau in :issue:`42012`.) + Improved Modules ================ diff --git a/Lib/wsgiref/__init__.py b/Lib/wsgiref/__init__.py index 1efbba01a306e..59ee48fddec23 100644 --- a/Lib/wsgiref/__init__.py +++ b/Lib/wsgiref/__init__.py @@ -13,6 +13,8 @@ * validate -- validation wrapper that sits between an app and a server to detect errors in either +* types -- collection of WSGI-related types for static type checking + To-Do: * cgi_gateway -- Run WSGI apps under CGI (pending a deployment standard) diff --git a/Lib/wsgiref/types.py b/Lib/wsgiref/types.py new file mode 100644 index 0000000000000..4a519e5682f3c --- /dev/null +++ b/Lib/wsgiref/types.py @@ -0,0 +1,54 @@ +"""WSGI-related types for static type checking""" + +from collections.abc import Callable, Iterable +from types import TracebackType +from typing import Any, Protocol, TypeAlias + +__all__ = [ + "StartResponse", + "WSGIEnvironment", + "WSGIApplication", + "InputStream", + "ErrorStream", + "FileWrapper", +] + +_ExcInfo = tuple[type[BaseException], BaseException, TracebackType] +_OptExcInfo = _ExcInfo | tuple[None, None, None] + +class StartResponse(Protocol): + """start_response() callable as defined in PEP 3333""" + def __call__( + self, + status: str, + headers: list[tuple[str, str]], + exc_info: _OptExcInfo | None = ..., + /, + ) -> Callable[[bytes], object]: ... + +WSGIEnvironment: TypeAlias = dict[str, Any] +WSGIApplication: TypeAlias = Callable[[WSGIEnvironment, StartResponse], + Iterable[bytes]] + +class InputStream(Protocol): + """WSGI input stream as defined in PEP 3333""" + def read(self, size: int = ..., /) -> bytes: ... + def readline(self, size: int = ..., /) -> bytes: ... + def readlines(self, hint: int = ..., /) -> list[bytes]: ... + def __iter__(self) -> Iterable[bytes]: ... + +class ErrorStream(Protocol): + """WSGI error stream as defined in PEP 3333""" + def flush(self) -> object: ... + def write(self, s: str, /) -> object: ... + def writelines(self, seq: list[str], /) -> object: ... + +class _Readable(Protocol): + def read(self, size: int = ..., /) -> bytes: ... + # Optional: def close(self) -> object: ... + +class FileWrapper(Protocol): + """WSGI file wrapper as defined in PEP 3333""" + def __call__( + self, file: _Readable, block_size: int = ..., /, + ) -> Iterable[bytes]: ... diff --git a/Misc/NEWS.d/next/Library/2022-04-05-17-18-13.bpo-42012.zMocQz.rst b/Misc/NEWS.d/next/Library/2022-04-05-17-18-13.bpo-42012.zMocQz.rst new file mode 100644 index 0000000000000..ba84041782563 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-05-17-18-13.bpo-42012.zMocQz.rst @@ -0,0 +1,2 @@ +Add :mod:`wsgiref.types`, containing WSGI-specific types for static type +checking. From webhook-mailer at python.org Sat Apr 16 14:46:47 2022 From: webhook-mailer at python.org (gpshead) Date: Sat, 16 Apr 2022 18:46:47 -0000 Subject: [Python-checkins] gh-91607: Fix several test_concurrent_futures tests to actually test what they claim (#91600) Message-ID: https://github.com/python/cpython/commit/7fa3a5a2197896066e3fe53ee325ac6ab54c3414 commit: 7fa3a5a2197896066e3fe53ee325ac6ab54c3414 branch: main author: Gregory P. Smith committer: gpshead date: 2022-04-16T11:46:33-07:00 summary: gh-91607: Fix several test_concurrent_futures tests to actually test what they claim (#91600) * Fix test_concurrent_futures to actually test what it says. Many ProcessPoolExecutor based tests were ignoring the mp_context and using the default instead. This meant we lacked proper test coverage of all of them. Also removes the old _prime_executor() worker delay seeding code as it appears to have no point and causes 20-30 seconds extra latency on this already long test. It also interfered with some of the refactoring to fix the above to not needlessly create their own executor when setUp has already created an appropriate one. * Don't import the name from multiprocessing directly to avoid confusion. * ?? Added by blurb_it. Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> files: A Misc/NEWS.d/next/Tests/2022-04-16-17-54-05.gh-issue-91607.FnXjtW.rst M Lib/test/test_concurrent_futures.py diff --git a/Lib/test/test_concurrent_futures.py b/Lib/test/test_concurrent_futures.py index 8adba36a387ad..978a748df7fa3 100644 --- a/Lib/test/test_concurrent_futures.py +++ b/Lib/test/test_concurrent_futures.py @@ -26,10 +26,10 @@ PENDING, RUNNING, CANCELLED, CANCELLED_AND_NOTIFIED, FINISHED, Future, BrokenExecutor) from concurrent.futures.process import BrokenProcessPool, _check_system_limits -from multiprocessing import get_context import multiprocessing.process import multiprocessing.util +import multiprocessing as mp if support.check_sanitizer(address=True, memory=True): @@ -130,7 +130,6 @@ def setUp(self): self.executor = self.executor_type( max_workers=self.worker_count, **self.executor_kwargs) - self._prime_executor() def tearDown(self): self.executor.shutdown(wait=True) @@ -144,15 +143,7 @@ def tearDown(self): super().tearDown() def get_context(self): - return get_context(self.ctx) - - def _prime_executor(self): - # Make sure that the executor is ready to do work before running the - # tests. This should reduce the probability of timeouts in the tests. - futures = [self.executor.submit(time.sleep, 0.1) - for _ in range(self.worker_count)] - for f in futures: - f.result() + return mp.get_context(self.ctx) class ThreadPoolMixin(ExecutorMixin): @@ -275,9 +266,6 @@ def test_initializer(self): with self.assertRaises(BrokenExecutor): self.executor.submit(get_init_status) - def _prime_executor(self): - pass - @contextlib.contextmanager def _assert_logged(self, msg): if self.log_queue is not None: @@ -364,14 +352,14 @@ def test_hang_issue12364(self): f.result() def test_cancel_futures(self): - executor = self.executor_type(max_workers=3) - fs = [executor.submit(time.sleep, .1) for _ in range(50)] - executor.shutdown(cancel_futures=True) + assert self.worker_count <= 5, "test needs few workers" + fs = [self.executor.submit(time.sleep, .1) for _ in range(50)] + self.executor.shutdown(cancel_futures=True) # We can't guarantee the exact number of cancellations, but we can - # guarantee that *some* were cancelled. With setting max_workers to 3, - # most of the submitted futures should have been cancelled. + # guarantee that *some* were cancelled. With few workers, many of + # the submitted futures should have been cancelled. cancelled = [fut for fut in fs if fut.cancelled()] - self.assertTrue(len(cancelled) >= 35, msg=f"{len(cancelled)=}") + self.assertGreater(len(cancelled), 20) # Ensure the other futures were able to finish. # Use "not fut.cancelled()" instead of "fut.done()" to include futures @@ -384,33 +372,32 @@ def test_cancel_futures(self): # Similar to the number of cancelled futures, we can't guarantee the # exact number that completed. But, we can guarantee that at least # one finished. - self.assertTrue(len(others) > 0, msg=f"{len(others)=}") + self.assertGreater(len(others), 0) - def test_hang_issue39205(self): + def test_hang_gh83386(self): """shutdown(wait=False) doesn't hang at exit with running futures. - See https://bugs.python.org/issue39205. + See https://github.com/python/cpython/issues/83386. """ if self.executor_type == futures.ProcessPoolExecutor: raise unittest.SkipTest( - "Hangs due to https://bugs.python.org/issue39205") + "Hangs, see https://github.com/python/cpython/issues/83386") rc, out, err = assert_python_ok('-c', """if True: from concurrent.futures import {executor_type} from test.test_concurrent_futures import sleep_and_print if __name__ == "__main__": + if {context!r}: multiprocessing.set_start_method({context!r}) t = {executor_type}(max_workers=3) t.submit(sleep_and_print, 1.0, "apple") t.shutdown(wait=False) - """.format(executor_type=self.executor_type.__name__)) + """.format(executor_type=self.executor_type.__name__, + context=getattr(self, 'ctx', None))) self.assertFalse(err) self.assertEqual(out.strip(), b"apple") class ThreadPoolShutdownTest(ThreadPoolMixin, ExecutorShutdownTest, BaseTestCase): - def _prime_executor(self): - pass - def test_threads_terminate(self): def acquire_lock(lock): lock.acquire() @@ -505,14 +492,11 @@ def test_cancel_futures_wait_false(self): class ProcessPoolShutdownTest(ExecutorShutdownTest): - def _prime_executor(self): - pass - def test_processes_terminate(self): def acquire_lock(lock): lock.acquire() - mp_context = get_context() + mp_context = self.get_context() sem = mp_context.Semaphore(0) for _ in range(3): self.executor.submit(acquire_lock, sem) @@ -526,7 +510,8 @@ def acquire_lock(lock): p.join() def test_context_manager_shutdown(self): - with futures.ProcessPoolExecutor(max_workers=5) as e: + with futures.ProcessPoolExecutor( + max_workers=5, mp_context=self.get_context()) as e: processes = e._processes self.assertEqual(list(e.map(abs, range(-5, 5))), [5, 4, 3, 2, 1, 0, 1, 2, 3, 4]) @@ -535,7 +520,8 @@ def test_context_manager_shutdown(self): p.join() def test_del_shutdown(self): - executor = futures.ProcessPoolExecutor(max_workers=5) + executor = futures.ProcessPoolExecutor( + max_workers=5, mp_context=self.get_context()) res = executor.map(abs, range(-5, 5)) executor_manager_thread = executor._executor_manager_thread processes = executor._processes @@ -558,7 +544,8 @@ def test_del_shutdown(self): def test_shutdown_no_wait(self): # Ensure that the executor cleans up the processes when calling # shutdown with wait=False - executor = futures.ProcessPoolExecutor(max_workers=5) + executor = futures.ProcessPoolExecutor( + max_workers=5, mp_context=self.get_context()) res = executor.map(abs, range(-5, 5)) processes = executor._processes call_queue = executor._call_queue @@ -935,7 +922,7 @@ def submit(pool): pool.submit(submit, pool) for _ in range(50): - with futures.ProcessPoolExecutor(1, mp_context=get_context('fork')) as workers: + with futures.ProcessPoolExecutor(1, mp_context=mp.get_context('fork')) as workers: workers.submit(tuple) @@ -1005,7 +992,7 @@ def test_traceback(self): def test_ressources_gced_in_workers(self): # Ensure that argument for a job are correctly gc-ed after the job # is finished - mgr = get_context(self.ctx).Manager() + mgr = self.get_context().Manager() obj = EventfulGCObj(mgr) future = self.executor.submit(id, obj) future.result() @@ -1021,38 +1008,41 @@ def test_ressources_gced_in_workers(self): mgr.join() def test_saturation(self): - executor = self.executor_type(4) - mp_context = get_context() + executor = self.executor + mp_context = self.get_context() sem = mp_context.Semaphore(0) job_count = 15 * executor._max_workers - try: - for _ in range(job_count): - executor.submit(sem.acquire) - self.assertEqual(len(executor._processes), executor._max_workers) - for _ in range(job_count): - sem.release() - finally: - executor.shutdown() + for _ in range(job_count): + executor.submit(sem.acquire) + self.assertEqual(len(executor._processes), executor._max_workers) + for _ in range(job_count): + sem.release() def test_idle_process_reuse_one(self): - executor = self.executor_type(4) + executor = self.executor + assert executor._max_workers >= 4 executor.submit(mul, 21, 2).result() executor.submit(mul, 6, 7).result() executor.submit(mul, 3, 14).result() self.assertEqual(len(executor._processes), 1) - executor.shutdown() def test_idle_process_reuse_multiple(self): - executor = self.executor_type(4) + executor = self.executor + assert executor._max_workers <= 5 executor.submit(mul, 12, 7).result() executor.submit(mul, 33, 25) executor.submit(mul, 25, 26).result() executor.submit(mul, 18, 29) - self.assertLessEqual(len(executor._processes), 2) + executor.submit(mul, 1, 2).result() + executor.submit(mul, 0, 9) + self.assertLessEqual(len(executor._processes), 3) executor.shutdown() def test_max_tasks_per_child(self): - executor = self.executor_type(1, max_tasks_per_child=3) + # not using self.executor as we need to control construction. + # arguably this could go in another class w/o that mixin. + executor = self.executor_type( + 1, mp_context=self.get_context(), max_tasks_per_child=3) f1 = executor.submit(os.getpid) original_pid = f1.result() # The worker pid remains the same as the worker could be reused @@ -1072,7 +1062,10 @@ def test_max_tasks_per_child(self): executor.shutdown() def test_max_tasks_early_shutdown(self): - executor = self.executor_type(3, max_tasks_per_child=1) + # not using self.executor as we need to control construction. + # arguably this could go in another class w/o that mixin. + executor = self.executor_type( + 3, mp_context=self.get_context(), max_tasks_per_child=1) futures = [] for i in range(6): futures.append(executor.submit(mul, i, i)) @@ -1182,7 +1175,7 @@ def _check_crash(self, error, func, *args, ignore_stderr=False): self.executor.shutdown(wait=True) executor = self.executor_type( - max_workers=2, mp_context=get_context(self.ctx)) + max_workers=2, mp_context=self.get_context()) res = executor.submit(func, *args) if ignore_stderr: @@ -1261,7 +1254,7 @@ def test_shutdown_deadlock(self): # if a worker fails after the shutdown call. self.executor.shutdown(wait=True) with self.executor_type(max_workers=2, - mp_context=get_context(self.ctx)) as executor: + mp_context=self.get_context()) as executor: self.executor = executor # Allow clean up in fail_on_deadlock f = executor.submit(_crash, delay=.1) executor.shutdown(wait=True) @@ -1274,7 +1267,7 @@ def test_shutdown_deadlock_pickle(self): # Reported in bpo-39104. self.executor.shutdown(wait=True) with self.executor_type(max_workers=2, - mp_context=get_context(self.ctx)) as executor: + mp_context=self.get_context()) as executor: self.executor = executor # Allow clean up in fail_on_deadlock # Start the executor and get the executor_manager_thread to collect diff --git a/Misc/NEWS.d/next/Tests/2022-04-16-17-54-05.gh-issue-91607.FnXjtW.rst b/Misc/NEWS.d/next/Tests/2022-04-16-17-54-05.gh-issue-91607.FnXjtW.rst new file mode 100644 index 0000000000000..32839a826a41e --- /dev/null +++ b/Misc/NEWS.d/next/Tests/2022-04-16-17-54-05.gh-issue-91607.FnXjtW.rst @@ -0,0 +1 @@ +Fix ``test_concurrent_futures`` to test the correct multiprocessing start method context in several cases where the test logic mixed this up. From webhook-mailer at python.org Sat Apr 16 15:00:13 2022 From: webhook-mailer at python.org (iritkatriel) Date: Sat, 16 Apr 2022 19:00:13 -0000 Subject: [Python-checkins] gh-89770: Implement PEP-678 - Exception notes (GH-31317) Message-ID: https://github.com/python/cpython/commit/d4c4a76ed1427c947fcbbe692625b3f644cf3aaf commit: d4c4a76ed1427c947fcbbe692625b3f644cf3aaf branch: main author: Irit Katriel <1055913+iritkatriel at users.noreply.github.com> committer: iritkatriel <1055913+iritkatriel at users.noreply.github.com> date: 2022-04-16T19:59:52+01:00 summary: gh-89770: Implement PEP-678 - Exception notes (GH-31317) files: A Misc/NEWS.d/next/Core and Builtins/2022-04-12-11-56-23.gh-issue-91479.-dyGJX.rst M Doc/library/exceptions.rst M Doc/whatsnew/3.11.rst M Include/cpython/pyerrors.h M Include/internal/pycore_global_strings.h M Include/internal/pycore_runtime_init.h M Lib/test/test_exception_group.py M Lib/test/test_exceptions.py M Lib/test/test_traceback.py M Lib/traceback.py M Objects/exceptions.c M Python/pythonrun.c diff --git a/Doc/library/exceptions.rst b/Doc/library/exceptions.rst index 137566e079d20..2eccbd17c482c 100644 --- a/Doc/library/exceptions.rst +++ b/Doc/library/exceptions.rst @@ -126,13 +126,20 @@ The following exceptions are used mostly as base classes for other exceptions. tb = sys.exc_info()[2] raise OtherException(...).with_traceback(tb) - .. attribute:: __note__ + .. method:: add_note(note) - A mutable field which is :const:`None` by default and can be set to a string. - If it is not :const:`None`, it is included in the traceback. This field can - be used to enrich exceptions after they have been caught. + Add the string ``note`` to the exception's notes which appear in the standard + traceback after the exception string. A :exc:`TypeError` is raised if ``note`` + is not a string. - .. versionadded:: 3.11 + .. versionadded:: 3.11 + + .. attribute:: __notes__ + + A list of the notes of this exception, which were added with :meth:`add_note`. + This attribute is created when :meth:`add_note` is called. + + .. versionadded:: 3.11 .. exception:: Exception @@ -907,7 +914,7 @@ their subgroups based on the types of the contained exceptions. The nesting structure of the current exception is preserved in the result, as are the values of its :attr:`message`, :attr:`__traceback__`, - :attr:`__cause__`, :attr:`__context__` and :attr:`__note__` fields. + :attr:`__cause__`, :attr:`__context__` and :attr:`__notes__` fields. Empty nested groups are omitted from the result. The condition is checked for all exceptions in the nested exception group, @@ -924,7 +931,7 @@ their subgroups based on the types of the contained exceptions. Returns an exception group with the same :attr:`message`, :attr:`__traceback__`, :attr:`__cause__`, :attr:`__context__` - and :attr:`__note__` but which wraps the exceptions in ``excs``. + and :attr:`__notes__` but which wraps the exceptions in ``excs``. This method is used by :meth:`subgroup` and :meth:`split`. A subclass needs to override it in order to make :meth:`subgroup` diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index 19e3d2f92fb94..cd452fc537157 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -157,12 +157,15 @@ The :option:`-X` ``no_debug_ranges`` option and the environment variable See :pep:`657` for more details. (Contributed by Pablo Galindo, Batuhan Taskaya and Ammar Askar in :issue:`43950`.) -Exceptions can be enriched with a string ``__note__`` -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Exceptions can be enriched with notes (PEP 678) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The :meth:`add_note` method was added to :exc:`BaseException`. It can be +used to enrich exceptions with context information which is not available +at the time when the exception is raised. The notes added appear in the +default traceback. See :pep:`678` for more details. (Contributed by +Irit Katriel in :issue:`45607`.) -The ``__note__`` field was added to :exc:`BaseException`. It is ``None`` -by default but can be set to a string which is added to the exception's -traceback. (Contributed by Irit Katriel in :issue:`45607`.) Other Language Changes ====================== diff --git a/Include/cpython/pyerrors.h b/Include/cpython/pyerrors.h index 08630cce8ac90..47d80e3242302 100644 --- a/Include/cpython/pyerrors.h +++ b/Include/cpython/pyerrors.h @@ -6,7 +6,7 @@ /* PyException_HEAD defines the initial segment of every exception class. */ #define PyException_HEAD PyObject_HEAD PyObject *dict;\ - PyObject *args; PyObject *note; PyObject *traceback;\ + PyObject *args; PyObject *notes; PyObject *traceback;\ PyObject *context; PyObject *cause;\ char suppress_context; diff --git a/Include/internal/pycore_global_strings.h b/Include/internal/pycore_global_strings.h index 833ff2710a787..77f96cd5ed2f0 100644 --- a/Include/internal/pycore_global_strings.h +++ b/Include/internal/pycore_global_strings.h @@ -155,7 +155,7 @@ struct _Py_global_strings { STRUCT_FOR_ID(__newobj__) STRUCT_FOR_ID(__newobj_ex__) STRUCT_FOR_ID(__next__) - STRUCT_FOR_ID(__note__) + STRUCT_FOR_ID(__notes__) STRUCT_FOR_ID(__or__) STRUCT_FOR_ID(__orig_class__) STRUCT_FOR_ID(__origin__) diff --git a/Include/internal/pycore_runtime_init.h b/Include/internal/pycore_runtime_init.h index fd925b3e060df..371f2d23ad8ec 100644 --- a/Include/internal/pycore_runtime_init.h +++ b/Include/internal/pycore_runtime_init.h @@ -778,7 +778,7 @@ extern "C" { INIT_ID(__newobj__), \ INIT_ID(__newobj_ex__), \ INIT_ID(__next__), \ - INIT_ID(__note__), \ + INIT_ID(__notes__), \ INIT_ID(__or__), \ INIT_ID(__orig_class__), \ INIT_ID(__origin__), \ diff --git a/Lib/test/test_exception_group.py b/Lib/test/test_exception_group.py index 793e8d20de7e3..2cfd8738304d1 100644 --- a/Lib/test/test_exception_group.py +++ b/Lib/test/test_exception_group.py @@ -567,7 +567,9 @@ def leaves(exc): self.assertIs(eg.__cause__, part.__cause__) self.assertIs(eg.__context__, part.__context__) self.assertIs(eg.__traceback__, part.__traceback__) - self.assertIs(eg.__note__, part.__note__) + self.assertEqual( + getattr(eg, '__notes__', None), + getattr(part, '__notes__', None)) def tbs_for_leaf(leaf, eg): for e, tbs in leaf_generator(eg): @@ -632,7 +634,7 @@ def level3(i): try: nested_group() except ExceptionGroup as e: - e.__note__ = f"the note: {id(e)}" + e.add_note(f"the note: {id(e)}") eg = e eg_template = [ @@ -728,6 +730,35 @@ def exc(ex): self.assertMatchesTemplate( rest, ExceptionGroup, [ValueError(1)]) + def test_split_copies_notes(self): + # make sure each exception group after a split has its own __notes__ list + eg = ExceptionGroup("eg", [ValueError(1), TypeError(2)]) + eg.add_note("note1") + eg.add_note("note2") + orig_notes = list(eg.__notes__) + match, rest = eg.split(TypeError) + self.assertEqual(eg.__notes__, orig_notes) + self.assertEqual(match.__notes__, orig_notes) + self.assertEqual(rest.__notes__, orig_notes) + self.assertIsNot(eg.__notes__, match.__notes__) + self.assertIsNot(eg.__notes__, rest.__notes__) + self.assertIsNot(match.__notes__, rest.__notes__) + eg.add_note("eg") + match.add_note("match") + rest.add_note("rest") + self.assertEqual(eg.__notes__, orig_notes + ["eg"]) + self.assertEqual(match.__notes__, orig_notes + ["match"]) + self.assertEqual(rest.__notes__, orig_notes + ["rest"]) + + def test_split_does_not_copy_non_sequence_notes(self): + # __notes__ should be a sequence, which is shallow copied. + # If it is not a sequence, the split parts don't get any notes. + eg = ExceptionGroup("eg", [ValueError(1), TypeError(2)]) + eg.__notes__ = 123 + match, rest = eg.split(TypeError) + self.assertFalse(hasattr(match, '__notes__')) + self.assertFalse(hasattr(rest, '__notes__')) + class NestedExceptionGroupSubclassSplitTest(ExceptionGroupSplitTestBase): diff --git a/Lib/test/test_exceptions.py b/Lib/test/test_exceptions.py index 6dca79efef180..2b5b51934562a 100644 --- a/Lib/test/test_exceptions.py +++ b/Lib/test/test_exceptions.py @@ -547,26 +547,32 @@ def testAttributes(self): 'pickled "%r", attribute "%s' % (e, checkArgName)) - def test_note(self): + def test_notes(self): for e in [BaseException(1), Exception(2), ValueError(3)]: with self.subTest(e=e): - self.assertIsNone(e.__note__) - e.__note__ = "My Note" - self.assertEqual(e.__note__, "My Note") + self.assertFalse(hasattr(e, '__notes__')) + e.add_note("My Note") + self.assertEqual(e.__notes__, ["My Note"]) with self.assertRaises(TypeError): - e.__note__ = 42 - self.assertEqual(e.__note__, "My Note") + e.add_note(42) + self.assertEqual(e.__notes__, ["My Note"]) - e.__note__ = "Your Note" - self.assertEqual(e.__note__, "Your Note") + e.add_note("Your Note") + self.assertEqual(e.__notes__, ["My Note", "Your Note"]) - with self.assertRaises(TypeError): - del e.__note__ - self.assertEqual(e.__note__, "Your Note") + del e.__notes__ + self.assertFalse(hasattr(e, '__notes__')) + + e.add_note("Our Note") + self.assertEqual(e.__notes__, ["Our Note"]) - e.__note__ = None - self.assertIsNone(e.__note__) + e.__notes__ = 42 + self.assertEqual(e.__notes__, 42) + + with self.assertRaises(TypeError): + e.add_note("will not work") + self.assertEqual(e.__notes__, 42) def testWithTraceback(self): try: diff --git a/Lib/test/test_traceback.py b/Lib/test/test_traceback.py index 75d668df64d4c..962322c89ff66 100644 --- a/Lib/test/test_traceback.py +++ b/Lib/test/test_traceback.py @@ -1323,21 +1323,80 @@ def test_syntax_error_various_offsets(self): self.assertEqual(exp, err) def test_exception_with_note(self): - e = ValueError(42) + e = ValueError(123) vanilla = self.get_report(e) - e.__note__ = 'My Note' + e.add_note('My Note') self.assertEqual(self.get_report(e), vanilla + 'My Note\n') - e.__note__ = '' + del e.__notes__ + e.add_note('') self.assertEqual(self.get_report(e), vanilla + '\n') - e.__note__ = 'Your Note' + del e.__notes__ + e.add_note('Your Note') self.assertEqual(self.get_report(e), vanilla + 'Your Note\n') - e.__note__ = None + del e.__notes__ self.assertEqual(self.get_report(e), vanilla) + def test_exception_with_invalid_notes(self): + e = ValueError(123) + vanilla = self.get_report(e) + + # non-sequence __notes__ + class BadThing: + def __str__(self): + return 'bad str' + + def __repr__(self): + return 'bad repr' + + # unprintable, non-sequence __notes__ + class Unprintable: + def __repr__(self): + raise ValueError('bad value') + + e.__notes__ = BadThing() + notes_repr = 'bad repr' + self.assertEqual(self.get_report(e), vanilla + notes_repr) + + e.__notes__ = Unprintable() + err_msg = '<__notes__ repr() failed>' + self.assertEqual(self.get_report(e), vanilla + err_msg) + + # non-string item in the __notes__ sequence + e.__notes__ = [BadThing(), 'Final Note'] + bad_note = 'bad str' + self.assertEqual(self.get_report(e), vanilla + bad_note + '\nFinal Note\n') + + # unprintable, non-string item in the __notes__ sequence + e.__notes__ = [Unprintable(), 'Final Note'] + err_msg = '' + self.assertEqual(self.get_report(e), vanilla + err_msg + '\nFinal Note\n') + + def test_exception_with_note_with_multiple_notes(self): + e = ValueError(42) + vanilla = self.get_report(e) + + e.add_note('Note 1') + e.add_note('Note 2') + e.add_note('Note 3') + + self.assertEqual( + self.get_report(e), + vanilla + 'Note 1\n' + 'Note 2\n' + 'Note 3\n') + + del e.__notes__ + e.add_note('Note 4') + del e.__notes__ + e.add_note('Note 5') + e.add_note('Note 6') + + self.assertEqual( + self.get_report(e), + vanilla + 'Note 5\n' + 'Note 6\n') + def test_exception_qualname(self): class A: class B: @@ -1688,16 +1747,16 @@ def exc(): try: raise ValueError(msg) except ValueError as e: - e.__note__ = f'the {msg}' + e.add_note(f'the {msg}') excs.append(e) raise ExceptionGroup("nested", excs) except ExceptionGroup as e: - e.__note__ = ('>> Multi line note\n' - '>> Because I am such\n' - '>> an important exception.\n' - '>> empty lines work too\n' - '\n' - '(that was an empty line)') + e.add_note(('>> Multi line note\n' + '>> Because I am such\n' + '>> an important exception.\n' + '>> empty lines work too\n' + '\n' + '(that was an empty line)')) raise expected = (f' + Exception Group Traceback (most recent call last):\n' @@ -1733,6 +1792,64 @@ def exc(): report = self.get_report(exc) self.assertEqual(report, expected) + def test_exception_group_with_multiple_notes(self): + def exc(): + try: + excs = [] + for msg in ['bad value', 'terrible value']: + try: + raise ValueError(msg) + except ValueError as e: + e.add_note(f'the {msg}') + e.add_note(f'Goodbye {msg}') + excs.append(e) + raise ExceptionGroup("nested", excs) + except ExceptionGroup as e: + e.add_note(('>> Multi line note\n' + '>> Because I am such\n' + '>> an important exception.\n' + '>> empty lines work too\n' + '\n' + '(that was an empty line)')) + e.add_note('Goodbye!') + raise + + expected = (f' + Exception Group Traceback (most recent call last):\n' + f' | File "{__file__}", line {self.callable_line}, in get_exception\n' + f' | exception_or_callable()\n' + f' | ^^^^^^^^^^^^^^^^^^^^^^^\n' + f' | File "{__file__}", line {exc.__code__.co_firstlineno + 10}, in exc\n' + f' | raise ExceptionGroup("nested", excs)\n' + f' | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n' + f' | ExceptionGroup: nested (2 sub-exceptions)\n' + f' | >> Multi line note\n' + f' | >> Because I am such\n' + f' | >> an important exception.\n' + f' | >> empty lines work too\n' + f' | \n' + f' | (that was an empty line)\n' + f' | Goodbye!\n' + f' +-+---------------- 1 ----------------\n' + f' | Traceback (most recent call last):\n' + f' | File "{__file__}", line {exc.__code__.co_firstlineno + 5}, in exc\n' + f' | raise ValueError(msg)\n' + f' | ^^^^^^^^^^^^^^^^^^^^^\n' + f' | ValueError: bad value\n' + f' | the bad value\n' + f' | Goodbye bad value\n' + f' +---------------- 2 ----------------\n' + f' | Traceback (most recent call last):\n' + f' | File "{__file__}", line {exc.__code__.co_firstlineno + 5}, in exc\n' + f' | raise ValueError(msg)\n' + f' | ^^^^^^^^^^^^^^^^^^^^^\n' + f' | ValueError: terrible value\n' + f' | the terrible value\n' + f' | Goodbye terrible value\n' + f' +------------------------------------\n') + + report = self.get_report(exc) + self.assertEqual(report, expected) + class PyExcReportingTests(BaseExceptionReportingTests, unittest.TestCase): # @@ -2077,32 +2194,32 @@ def some_inner(): [f'{__file__}:{some_inner.__code__.co_firstlineno + 1}']) def test_dropping_frames(self): - def f(): - 1/0 - - def g(): - try: - f() - except: - return sys.exc_info() - - exc_info = g() - - class Skip_G(traceback.StackSummary): - def format_frame_summary(self, frame_summary): - if frame_summary.name == 'g': - return None - return super().format_frame_summary(frame_summary) - - stack = Skip_G.extract( - traceback.walk_tb(exc_info[2])).format() - - self.assertEqual(len(stack), 1) - lno = f.__code__.co_firstlineno + 1 - self.assertEqual( - stack[0], - f' File "{__file__}", line {lno}, in f\n 1/0\n' - ) + def f(): + 1/0 + + def g(): + try: + f() + except: + return sys.exc_info() + + exc_info = g() + + class Skip_G(traceback.StackSummary): + def format_frame_summary(self, frame_summary): + if frame_summary.name == 'g': + return None + return super().format_frame_summary(frame_summary) + + stack = Skip_G.extract( + traceback.walk_tb(exc_info[2])).format() + + self.assertEqual(len(stack), 1) + lno = f.__code__.co_firstlineno + 1 + self.assertEqual( + stack[0], + f' File "{__file__}", line {lno}, in f\n 1/0\n' + ) class TestTracebackException(unittest.TestCase): diff --git a/Lib/traceback.py b/Lib/traceback.py index 05f1fffef0d3b..3afe49d1d8a0e 100644 --- a/Lib/traceback.py +++ b/Lib/traceback.py @@ -1,6 +1,6 @@ """Extract, format and print information about Python stack traces.""" -import collections +import collections.abc import itertools import linecache import sys @@ -163,18 +163,18 @@ def format_exception_only(exc, /, value=_sentinel): # -- not official API but folk probably use these two functions. def _format_final_exc_line(etype, value): - valuestr = _some_str(value) + valuestr = _safe_string(value, 'exception') if value is None or not valuestr: line = "%s\n" % etype else: line = "%s: %s\n" % (etype, valuestr) return line -def _some_str(value): +def _safe_string(value, what, func=str): try: - return str(value) + return func(value) except: - return '' + return f'<{what} {func.__name__}() failed>' # -- @@ -688,8 +688,8 @@ def __init__(self, exc_type, exc_value, exc_traceback, *, limit=None, self.exc_type = exc_type # Capture now to permit freeing resources: only complication is in the # unofficial API _format_final_exc_line - self._str = _some_str(exc_value) - self.__note__ = exc_value.__note__ if exc_value else None + self._str = _safe_string(exc_value, 'exception') + self.__notes__ = getattr(exc_value, '__notes__', None) if exc_type and issubclass(exc_type, SyntaxError): # Handle SyntaxError's specially @@ -822,8 +822,12 @@ def format_exception_only(self): yield _format_final_exc_line(stype, self._str) else: yield from self._format_syntax_error(stype) - if self.__note__ is not None: - yield from [l + '\n' for l in self.__note__.split('\n')] + if isinstance(self.__notes__, collections.abc.Sequence): + for note in self.__notes__: + note = _safe_string(note, 'note') + yield from [l + '\n' for l in note.split('\n')] + elif self.__notes__ is not None: + yield _safe_string(self.__notes__, '__notes__', func=repr) def _format_syntax_error(self, stype): """Format SyntaxError exceptions (internal helper).""" @@ -913,7 +917,7 @@ def format(self, *, chain=True, _ctx=None): # format exception group is_toplevel = (_ctx.exception_group_depth == 0) if is_toplevel: - _ctx.exception_group_depth += 1 + _ctx.exception_group_depth += 1 if exc.stack: yield from _ctx.emit( diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-04-12-11-56-23.gh-issue-91479.-dyGJX.rst b/Misc/NEWS.d/next/Core and Builtins/2022-04-12-11-56-23.gh-issue-91479.-dyGJX.rst new file mode 100644 index 0000000000000..e131e91e75312 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2022-04-12-11-56-23.gh-issue-91479.-dyGJX.rst @@ -0,0 +1 @@ +Replaced the ``__note__`` field of :exc:`BaseException` (added in an earlier version of 3.11) with the final design of :pep:`678`. Namely, :exc:`BaseException` gets an :meth:`add_note` method, and its ``__notes__`` field is created when necessary. diff --git a/Objects/exceptions.c b/Objects/exceptions.c index df10a3c2416e3..b26a0e93af484 100644 --- a/Objects/exceptions.c +++ b/Objects/exceptions.c @@ -47,7 +47,7 @@ BaseException_new(PyTypeObject *type, PyObject *args, PyObject *kwds) return NULL; /* the dict is created on the fly in PyObject_GenericSetAttr */ self->dict = NULL; - self->note = NULL; + self->notes = NULL; self->traceback = self->cause = self->context = NULL; self->suppress_context = 0; @@ -83,7 +83,7 @@ BaseException_clear(PyBaseExceptionObject *self) { Py_CLEAR(self->dict); Py_CLEAR(self->args); - Py_CLEAR(self->note); + Py_CLEAR(self->notes); Py_CLEAR(self->traceback); Py_CLEAR(self->cause); Py_CLEAR(self->context); @@ -108,7 +108,7 @@ BaseException_traverse(PyBaseExceptionObject *self, visitproc visit, void *arg) { Py_VISIT(self->dict); Py_VISIT(self->args); - Py_VISIT(self->note); + Py_VISIT(self->notes); Py_VISIT(self->traceback); Py_VISIT(self->cause); Py_VISIT(self->context); @@ -186,12 +186,62 @@ PyDoc_STRVAR(with_traceback_doc, "Exception.with_traceback(tb) --\n\ set self.__traceback__ to tb and return self."); +static inline PyBaseExceptionObject* +_PyBaseExceptionObject_cast(PyObject *exc) +{ + assert(PyExceptionInstance_Check(exc)); + return (PyBaseExceptionObject *)exc; +} + +static PyObject * +BaseException_add_note(PyObject *self, PyObject *note) +{ + if (!PyUnicode_Check(note)) { + PyErr_Format(PyExc_TypeError, + "note must be a str, not '%s'", + Py_TYPE(note)->tp_name); + return NULL; + } + + if (!PyObject_HasAttr(self, &_Py_ID(__notes__))) { + PyObject *new_notes = PyList_New(0); + if (new_notes == NULL) { + return NULL; + } + if (PyObject_SetAttr(self, &_Py_ID(__notes__), new_notes) < 0) { + Py_DECREF(new_notes); + return NULL; + } + Py_DECREF(new_notes); + } + PyObject *notes = PyObject_GetAttr(self, &_Py_ID(__notes__)); + if (notes == NULL) { + return NULL; + } + if (!PyList_Check(notes)) { + Py_DECREF(notes); + PyErr_SetString(PyExc_TypeError, "Cannot add note: __notes__ is not a list"); + return NULL; + } + if (PyList_Append(notes, note) < 0) { + Py_DECREF(notes); + return NULL; + } + Py_DECREF(notes); + Py_RETURN_NONE; +} + +PyDoc_STRVAR(add_note_doc, +"Exception.add_note(note) --\n\ + add a note to the exception"); static PyMethodDef BaseException_methods[] = { {"__reduce__", (PyCFunction)BaseException_reduce, METH_NOARGS }, {"__setstate__", (PyCFunction)BaseException_setstate, METH_O }, {"with_traceback", (PyCFunction)BaseException_with_traceback, METH_O, with_traceback_doc}, + {"add_note", (PyCFunction)BaseException_add_note, METH_O, + add_note_doc}, {NULL, NULL, 0, NULL}, }; @@ -220,33 +270,6 @@ BaseException_set_args(PyBaseExceptionObject *self, PyObject *val, void *Py_UNUS return 0; } -static PyObject * -BaseException_get_note(PyBaseExceptionObject *self, void *Py_UNUSED(ignored)) -{ - if (self->note == NULL) { - Py_RETURN_NONE; - } - return Py_NewRef(self->note); -} - -static int -BaseException_set_note(PyBaseExceptionObject *self, PyObject *note, - void *Py_UNUSED(ignored)) -{ - if (note == NULL) { - PyErr_SetString(PyExc_TypeError, "__note__ may not be deleted"); - return -1; - } - else if (note != Py_None && !PyUnicode_CheckExact(note)) { - PyErr_SetString(PyExc_TypeError, "__note__ must be a string or None"); - return -1; - } - - Py_INCREF(note); - Py_XSETREF(self->note, note); - return 0; -} - static PyObject * BaseException_get_tb(PyBaseExceptionObject *self, void *Py_UNUSED(ignored)) { @@ -337,7 +360,6 @@ BaseException_set_cause(PyObject *self, PyObject *arg, void *Py_UNUSED(ignored)) static PyGetSetDef BaseException_getset[] = { {"__dict__", PyObject_GenericGetDict, PyObject_GenericSetDict}, {"args", (getter)BaseException_get_args, (setter)BaseException_set_args}, - {"__note__", (getter)BaseException_get_note, (setter)BaseException_set_note}, {"__traceback__", (getter)BaseException_get_tb, (setter)BaseException_set_tb}, {"__context__", BaseException_get_context, BaseException_set_context, PyDoc_STR("exception context")}, @@ -347,14 +369,6 @@ static PyGetSetDef BaseException_getset[] = { }; -static inline PyBaseExceptionObject* -_PyBaseExceptionObject_cast(PyObject *exc) -{ - assert(PyExceptionInstance_Check(exc)); - return (PyBaseExceptionObject *)exc; -} - - PyObject * PyException_GetTraceback(PyObject *self) { @@ -910,9 +924,32 @@ exceptiongroup_subset( PyException_SetContext(eg, PyException_GetContext(orig)); PyException_SetCause(eg, PyException_GetCause(orig)); - PyObject *note = _PyBaseExceptionObject_cast(orig)->note; - Py_XINCREF(note); - _PyBaseExceptionObject_cast(eg)->note = note; + if (PyObject_HasAttr(orig, &_Py_ID(__notes__))) { + PyObject *notes = PyObject_GetAttr(orig, &_Py_ID(__notes__)); + if (notes == NULL) { + goto error; + } + if (PySequence_Check(notes)) { + /* Make a copy so the parts have independent notes lists. */ + PyObject *notes_copy = PySequence_List(notes); + Py_DECREF(notes); + if (notes_copy == NULL) { + goto error; + } + int res = PyObject_SetAttr(eg, &_Py_ID(__notes__), notes_copy); + Py_DECREF(notes_copy); + if (res < 0) { + goto error; + } + } + else { + /* __notes__ is supposed to be a list, and split() is not a + * good place to report earlier user errors, so we just ignore + * notes of non-sequence type. + */ + Py_DECREF(notes); + } + } *result = eg; return 0; @@ -1262,7 +1299,7 @@ is_same_exception_metadata(PyObject *exc1, PyObject *exc2) PyBaseExceptionObject *e1 = (PyBaseExceptionObject *)exc1; PyBaseExceptionObject *e2 = (PyBaseExceptionObject *)exc2; - return (e1->note == e2->note && + return (e1->notes == e2->notes && e1->traceback == e2->traceback && e1->cause == e2->cause && e1->context == e2->context); diff --git a/Python/pythonrun.c b/Python/pythonrun.c index e086f0f345c22..769c34ea161e0 100644 --- a/Python/pythonrun.c +++ b/Python/pythonrun.c @@ -1129,7 +1129,7 @@ print_exception_suggestions(struct exception_print_context *ctx, } static int -print_exception_note(struct exception_print_context *ctx, PyObject *value) +print_exception_notes(struct exception_print_context *ctx, PyObject *value) { PyObject *f = ctx->file; @@ -1137,41 +1137,74 @@ print_exception_note(struct exception_print_context *ctx, PyObject *value) return 0; } - PyObject *note = PyObject_GetAttr(value, &_Py_ID(__note__)); - if (note == NULL) { + if (!PyObject_HasAttr(value, &_Py_ID(__notes__))) { + return 0; + } + PyObject *notes = PyObject_GetAttr(value, &_Py_ID(__notes__)); + if (notes == NULL) { return -1; } - if (!PyUnicode_Check(note)) { - Py_DECREF(note); - return 0; + if (!PySequence_Check(notes)) { + int res = 0; + if (write_indented_margin(ctx, f) < 0) { + res = -1; + } + PyObject *s = PyObject_Repr(notes); + if (s == NULL) { + PyErr_Clear(); + res = PyFile_WriteString("<__notes__ repr() failed>", f); + } + else { + res = PyFile_WriteObject(s, f, Py_PRINT_RAW); + Py_DECREF(s); + } + Py_DECREF(notes); + return res; } + Py_ssize_t num_notes = PySequence_Length(notes); + PyObject *lines = NULL; + for (Py_ssize_t ni = 0; ni < num_notes; ni++) { + PyObject *note = PySequence_GetItem(notes, ni); + PyObject *note_str = PyObject_Str(note); + Py_DECREF(note); - PyObject *lines = PyUnicode_Splitlines(note, 1); - Py_DECREF(note); + if (note_str == NULL) { + PyErr_Clear(); + if (PyFile_WriteString("", f) < 0) { + goto error; + } + } + else { + lines = PyUnicode_Splitlines(note_str, 1); + Py_DECREF(note_str); - if (lines == NULL) { - return -1; - } + if (lines == NULL) { + goto error; + } - Py_ssize_t n = PyList_GET_SIZE(lines); - for (Py_ssize_t i = 0; i < n; i++) { - PyObject *line = PyList_GET_ITEM(lines, i); - assert(PyUnicode_Check(line)); - if (write_indented_margin(ctx, f) < 0) { - goto error; + Py_ssize_t n = PyList_GET_SIZE(lines); + for (Py_ssize_t i = 0; i < n; i++) { + PyObject *line = PyList_GET_ITEM(lines, i); + assert(PyUnicode_Check(line)); + if (write_indented_margin(ctx, f) < 0) { + goto error; + } + if (PyFile_WriteObject(line, f, Py_PRINT_RAW) < 0) { + goto error; + } + } + Py_CLEAR(lines); } - if (PyFile_WriteObject(line, f, Py_PRINT_RAW) < 0) { + if (PyFile_WriteString("\n", f) < 0) { goto error; } } - if (PyFile_WriteString("\n", f) < 0) { - goto error; - } - Py_DECREF(lines); + Py_DECREF(notes); return 0; error: - Py_DECREF(lines); + Py_XDECREF(lines); + Py_DECREF(notes); return -1; } @@ -1206,7 +1239,7 @@ print_exception(struct exception_print_context *ctx, PyObject *value) if (PyFile_WriteString("\n", f) < 0) { goto error; } - if (print_exception_note(ctx, value) < 0) { + if (print_exception_notes(ctx, value) < 0) { goto error; } From webhook-mailer at python.org Sat Apr 16 15:40:11 2022 From: webhook-mailer at python.org (iritkatriel) Date: Sat, 16 Apr 2022 19:40:11 -0000 Subject: [Python-checkins] Fix an out of date comment in compile.c (GH-91615) Message-ID: https://github.com/python/cpython/commit/304f5b63e904c3ab0b746ef7ad58c5c2ff0154a8 commit: 304f5b63e904c3ab0b746ef7ad58c5c2ff0154a8 branch: main author: Irit Katriel <1055913+iritkatriel at users.noreply.github.com> committer: iritkatriel <1055913+iritkatriel at users.noreply.github.com> date: 2022-04-16T20:40:02+01:00 summary: Fix an out of date comment in compile.c (GH-91615) files: M Python/compile.c diff --git a/Python/compile.c b/Python/compile.c index 3b91566efba29..4108b896ade46 100644 --- a/Python/compile.c +++ b/Python/compile.c @@ -71,7 +71,8 @@ /* Pseudo-instructions used in the compiler, - * but turned into NOPs by the assembler. */ + * but turned into NOPs or other instructions + * by the assembler. */ #define SETUP_FINALLY -1 #define SETUP_CLEANUP -2 #define SETUP_WITH -3 From webhook-mailer at python.org Sat Apr 16 16:17:34 2022 From: webhook-mailer at python.org (miss-islington) Date: Sat, 16 Apr 2022 20:17:34 -0000 Subject: [Python-checkins] gh-91217: deprecate nis (GH-91606) Message-ID: https://github.com/python/cpython/commit/9f06ff96ccbda25c447b6a6f94082ea1e7784d96 commit: 9f06ff96ccbda25c447b6a6f94082ea1e7784d96 branch: main author: Brett Cannon committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-16T13:17:30-07:00 summary: gh-91217: deprecate nis (GH-91606) Automerge-Triggered-By: GH:brettcannon files: A Misc/NEWS.d/next/Library/2022-04-16-09-33-14.gh-issue-91217.nt9JFs.rst M Doc/whatsnew/3.11.rst M Lib/test/test_nis.py M Modules/nismodule.c diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index cd452fc537157..bedf80a72fab5 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -882,6 +882,7 @@ Deprecated * :mod:`crypt` * :mod:`imghdr` * :mod:`msilib` + * :mod:`nis` * :mod:`nntplib` (Contributed by Brett Cannon in :issue:`47061`.) diff --git a/Lib/test/test_nis.py b/Lib/test/test_nis.py index a22142f4069ba..797c9023ad34d 100644 --- a/Lib/test/test_nis.py +++ b/Lib/test/test_nis.py @@ -1,10 +1,13 @@ from test import support from test.support import import_helper import unittest +import warnings # Skip test if nis module does not exist. -nis = import_helper.import_module('nis') +with warnings.catch_warnings(): + warnings.simplefilter("ignore", DeprecationWarning) + nis = import_helper.import_module('nis') class NisTests(unittest.TestCase): diff --git a/Misc/NEWS.d/next/Library/2022-04-16-09-33-14.gh-issue-91217.nt9JFs.rst b/Misc/NEWS.d/next/Library/2022-04-16-09-33-14.gh-issue-91217.nt9JFs.rst new file mode 100644 index 0000000000000..fc29034169866 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-16-09-33-14.gh-issue-91217.nt9JFs.rst @@ -0,0 +1 @@ +Deprecate the nis module. diff --git a/Modules/nismodule.c b/Modules/nismodule.c index cdda1a6a2fb7d..948f756935db5 100644 --- a/Modules/nismodule.c +++ b/Modules/nismodule.c @@ -524,5 +524,11 @@ static struct PyModuleDef nismodule = { PyMODINIT_FUNC PyInit_nis(void) { + if (PyErr_WarnEx(PyExc_DeprecationWarning, + "'nis' is deprecated and slated for removal in " + "Python 3.13", + 7)) { + return NULL; + } return PyModuleDef_Init(&nismodule); } From webhook-mailer at python.org Sat Apr 16 16:37:11 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Sat, 16 Apr 2022 20:37:11 -0000 Subject: [Python-checkins] gh-86178: wsgiref.types: Add missing TypeAlias annotations (GH-91608) Message-ID: https://github.com/python/cpython/commit/25af5ea40f286df0a578c738c67ab54faceda90a commit: 25af5ea40f286df0a578c738c67ab54faceda90a branch: main author: Sebastian Rittau committer: JelleZijlstra date: 2022-04-16T13:37:02-07:00 summary: gh-86178: wsgiref.types: Add missing TypeAlias annotations (GH-91608) files: M Lib/wsgiref/types.py diff --git a/Lib/wsgiref/types.py b/Lib/wsgiref/types.py index 4a519e5682f3c..9e74a6c7312b1 100644 --- a/Lib/wsgiref/types.py +++ b/Lib/wsgiref/types.py @@ -13,8 +13,8 @@ "FileWrapper", ] -_ExcInfo = tuple[type[BaseException], BaseException, TracebackType] -_OptExcInfo = _ExcInfo | tuple[None, None, None] +_ExcInfo: TypeAlias = tuple[type[BaseException], BaseException, TracebackType] +_OptExcInfo: TypeAlias = _ExcInfo | tuple[None, None, None] class StartResponse(Protocol): """start_response() callable as defined in PEP 3333""" From webhook-mailer at python.org Sat Apr 16 16:48:15 2022 From: webhook-mailer at python.org (gpshead) Date: Sat, 16 Apr 2022 20:48:15 -0000 Subject: [Python-checkins] [3.10] gh-91607: Fix several test_concurrent_futures tests to actually test what they claim (GH-91600) (#91612) Message-ID: https://github.com/python/cpython/commit/9a458934f7be37c59525350f1f9ecbdcd69903c1 commit: 9a458934f7be37c59525350f1f9ecbdcd69903c1 branch: 3.10 author: Gregory P. Smith committer: gpshead date: 2022-04-16T13:48:11-07:00 summary: [3.10] gh-91607: Fix several test_concurrent_futures tests to actually test what they claim (GH-91600) (#91612) * Fix test_concurrent_futures to actually test what it says. Many ProcessPoolExecutor based tests were ignoring the mp_context and using the default instead. This meant we lacked proper test coverage of all of them. Also removes the old _prime_executor() worker delay seeding code as it appears to have no point and causes 20-30 seconds extra latency on this already long test. It also interfered with some of the refactoring to fix the above to not needlessly create their own executor when setUp has already created an appropriate one. * Don't import the name from multiprocessing directly to avoid confusion. (cherry picked from commit 7fa3a5a2197896066e3fe53ee325ac6ab54c3414) Co-authored-by: Gregory P. Smith files: A Misc/NEWS.d/next/Tests/2022-04-16-17-54-05.gh-issue-91607.FnXjtW.rst M Lib/test/test_concurrent_futures.py diff --git a/Lib/test/test_concurrent_futures.py b/Lib/test/test_concurrent_futures.py index 50fa1f189b174..d3d912892cc2d 100644 --- a/Lib/test/test_concurrent_futures.py +++ b/Lib/test/test_concurrent_futures.py @@ -26,10 +26,10 @@ PENDING, RUNNING, CANCELLED, CANCELLED_AND_NOTIFIED, FINISHED, Future, BrokenExecutor) from concurrent.futures.process import BrokenProcessPool, _check_system_limits -from multiprocessing import get_context import multiprocessing.process import multiprocessing.util +import multiprocessing as mp if support.check_sanitizer(address=True, memory=True): @@ -131,7 +131,6 @@ def setUp(self): self.executor = self.executor_type( max_workers=self.worker_count, **self.executor_kwargs) - self._prime_executor() def tearDown(self): self.executor.shutdown(wait=True) @@ -145,15 +144,7 @@ def tearDown(self): super().tearDown() def get_context(self): - return get_context(self.ctx) - - def _prime_executor(self): - # Make sure that the executor is ready to do work before running the - # tests. This should reduce the probability of timeouts in the tests. - futures = [self.executor.submit(time.sleep, 0.1) - for _ in range(self.worker_count)] - for f in futures: - f.result() + return mp.get_context(self.ctx) class ThreadPoolMixin(ExecutorMixin): @@ -276,9 +267,6 @@ def test_initializer(self): with self.assertRaises(BrokenExecutor): self.executor.submit(get_init_status) - def _prime_executor(self): - pass - @contextlib.contextmanager def _assert_logged(self, msg): if self.log_queue is not None: @@ -365,14 +353,14 @@ def test_hang_issue12364(self): f.result() def test_cancel_futures(self): - executor = self.executor_type(max_workers=3) - fs = [executor.submit(time.sleep, .1) for _ in range(50)] - executor.shutdown(cancel_futures=True) + assert self.worker_count <= 5, "test needs few workers" + fs = [self.executor.submit(time.sleep, .1) for _ in range(50)] + self.executor.shutdown(cancel_futures=True) # We can't guarantee the exact number of cancellations, but we can - # guarantee that *some* were cancelled. With setting max_workers to 3, - # most of the submitted futures should have been cancelled. + # guarantee that *some* were cancelled. With few workers, many of + # the submitted futures should have been cancelled. cancelled = [fut for fut in fs if fut.cancelled()] - self.assertTrue(len(cancelled) >= 35, msg=f"{len(cancelled)=}") + self.assertGreater(len(cancelled), 20) # Ensure the other futures were able to finish. # Use "not fut.cancelled()" instead of "fut.done()" to include futures @@ -385,33 +373,32 @@ def test_cancel_futures(self): # Similar to the number of cancelled futures, we can't guarantee the # exact number that completed. But, we can guarantee that at least # one finished. - self.assertTrue(len(others) > 0, msg=f"{len(others)=}") + self.assertGreater(len(others), 0) - def test_hang_issue39205(self): + def test_hang_gh83386(self): """shutdown(wait=False) doesn't hang at exit with running futures. - See https://bugs.python.org/issue39205. + See https://github.com/python/cpython/issues/83386. """ if self.executor_type == futures.ProcessPoolExecutor: raise unittest.SkipTest( - "Hangs due to https://bugs.python.org/issue39205") + "Hangs, see https://github.com/python/cpython/issues/83386") rc, out, err = assert_python_ok('-c', """if True: from concurrent.futures import {executor_type} from test.test_concurrent_futures import sleep_and_print if __name__ == "__main__": + if {context!r}: multiprocessing.set_start_method({context!r}) t = {executor_type}(max_workers=3) t.submit(sleep_and_print, 1.0, "apple") t.shutdown(wait=False) - """.format(executor_type=self.executor_type.__name__)) + """.format(executor_type=self.executor_type.__name__, + context=getattr(self, 'ctx', None))) self.assertFalse(err) self.assertEqual(out.strip(), b"apple") class ThreadPoolShutdownTest(ThreadPoolMixin, ExecutorShutdownTest, BaseTestCase): - def _prime_executor(self): - pass - def test_threads_terminate(self): def acquire_lock(lock): lock.acquire() @@ -506,14 +493,11 @@ def test_cancel_futures_wait_false(self): class ProcessPoolShutdownTest(ExecutorShutdownTest): - def _prime_executor(self): - pass - def test_processes_terminate(self): def acquire_lock(lock): lock.acquire() - mp_context = get_context() + mp_context = self.get_context() sem = mp_context.Semaphore(0) for _ in range(3): self.executor.submit(acquire_lock, sem) @@ -527,7 +511,8 @@ def acquire_lock(lock): p.join() def test_context_manager_shutdown(self): - with futures.ProcessPoolExecutor(max_workers=5) as e: + with futures.ProcessPoolExecutor( + max_workers=5, mp_context=self.get_context()) as e: processes = e._processes self.assertEqual(list(e.map(abs, range(-5, 5))), [5, 4, 3, 2, 1, 0, 1, 2, 3, 4]) @@ -536,7 +521,8 @@ def test_context_manager_shutdown(self): p.join() def test_del_shutdown(self): - executor = futures.ProcessPoolExecutor(max_workers=5) + executor = futures.ProcessPoolExecutor( + max_workers=5, mp_context=self.get_context()) res = executor.map(abs, range(-5, 5)) executor_manager_thread = executor._executor_manager_thread processes = executor._processes @@ -559,7 +545,8 @@ def test_del_shutdown(self): def test_shutdown_no_wait(self): # Ensure that the executor cleans up the processes when calling # shutdown with wait=False - executor = futures.ProcessPoolExecutor(max_workers=5) + executor = futures.ProcessPoolExecutor( + max_workers=5, mp_context=self.get_context()) res = executor.map(abs, range(-5, 5)) processes = executor._processes call_queue = executor._call_queue @@ -936,7 +923,7 @@ def submit(pool): pool.submit(submit, pool) for _ in range(50): - with futures.ProcessPoolExecutor(1, mp_context=get_context('fork')) as workers: + with futures.ProcessPoolExecutor(1, mp_context=mp.get_context('fork')) as workers: workers.submit(tuple) @@ -1006,7 +993,7 @@ def test_traceback(self): def test_ressources_gced_in_workers(self): # Ensure that argument for a job are correctly gc-ed after the job # is finished - mgr = get_context(self.ctx).Manager() + mgr = self.get_context().Manager() obj = EventfulGCObj(mgr) future = self.executor.submit(id, obj) future.result() @@ -1022,36 +1009,37 @@ def test_ressources_gced_in_workers(self): mgr.join() def test_saturation(self): - executor = self.executor_type(4) - mp_context = get_context() + executor = self.executor + mp_context = self.get_context() sem = mp_context.Semaphore(0) job_count = 15 * executor._max_workers - try: - for _ in range(job_count): - executor.submit(sem.acquire) - self.assertEqual(len(executor._processes), executor._max_workers) - for _ in range(job_count): - sem.release() - finally: - executor.shutdown() + for _ in range(job_count): + executor.submit(sem.acquire) + self.assertEqual(len(executor._processes), executor._max_workers) + for _ in range(job_count): + sem.release() def test_idle_process_reuse_one(self): - executor = self.executor_type(4) + executor = self.executor + assert executor._max_workers >= 4 executor.submit(mul, 21, 2).result() executor.submit(mul, 6, 7).result() executor.submit(mul, 3, 14).result() self.assertEqual(len(executor._processes), 1) - executor.shutdown() def test_idle_process_reuse_multiple(self): - executor = self.executor_type(4) + executor = self.executor + assert executor._max_workers <= 5 executor.submit(mul, 12, 7).result() executor.submit(mul, 33, 25) executor.submit(mul, 25, 26).result() executor.submit(mul, 18, 29) - self.assertLessEqual(len(executor._processes), 2) + executor.submit(mul, 1, 2).result() + executor.submit(mul, 0, 9) + self.assertLessEqual(len(executor._processes), 3) executor.shutdown() + create_executor_tests(ProcessPoolExecutorTest, executor_mixins=(ProcessPoolForkMixin, ProcessPoolForkserverMixin, @@ -1153,7 +1141,7 @@ def _check_crash(self, error, func, *args, ignore_stderr=False): self.executor.shutdown(wait=True) executor = self.executor_type( - max_workers=2, mp_context=get_context(self.ctx)) + max_workers=2, mp_context=self.get_context()) res = executor.submit(func, *args) if ignore_stderr: @@ -1232,7 +1220,7 @@ def test_shutdown_deadlock(self): # if a worker fails after the shutdown call. self.executor.shutdown(wait=True) with self.executor_type(max_workers=2, - mp_context=get_context(self.ctx)) as executor: + mp_context=self.get_context()) as executor: self.executor = executor # Allow clean up in fail_on_deadlock f = executor.submit(_crash, delay=.1) executor.shutdown(wait=True) @@ -1245,7 +1233,7 @@ def test_shutdown_deadlock_pickle(self): # Reported in bpo-39104. self.executor.shutdown(wait=True) with self.executor_type(max_workers=2, - mp_context=get_context(self.ctx)) as executor: + mp_context=self.get_context()) as executor: self.executor = executor # Allow clean up in fail_on_deadlock # Start the executor and get the executor_manager_thread to collect diff --git a/Misc/NEWS.d/next/Tests/2022-04-16-17-54-05.gh-issue-91607.FnXjtW.rst b/Misc/NEWS.d/next/Tests/2022-04-16-17-54-05.gh-issue-91607.FnXjtW.rst new file mode 100644 index 0000000000000..32839a826a41e --- /dev/null +++ b/Misc/NEWS.d/next/Tests/2022-04-16-17-54-05.gh-issue-91607.FnXjtW.rst @@ -0,0 +1 @@ +Fix ``test_concurrent_futures`` to test the correct multiprocessing start method context in several cases where the test logic mixed this up. From webhook-mailer at python.org Sat Apr 16 17:36:39 2022 From: webhook-mailer at python.org (sweeneyde) Date: Sat, 16 Apr 2022 21:36:39 -0000 Subject: [Python-checkins] gh-91462: Make lltrace output human-readable. (GH-91463) Message-ID: https://github.com/python/cpython/commit/8560f4a0f288fec33ba49f85bb872353d631a4dc commit: 8560f4a0f288fec33ba49f85bb872353d631a4dc branch: main author: Dennis Sweeney <36520290+sweeneyde at users.noreply.github.com> committer: sweeneyde <36520290+sweeneyde at users.noreply.github.com> date: 2022-04-16T17:36:29-04:00 summary: gh-91462: Make lltrace output human-readable. (GH-91463) * Transform opcodes into opnames * Print the whole stack at each opcode, and eliminate prtrace output at each (push/pop/stackadj) * Display info about the function at each resume_frame files: A Misc/NEWS.d/next/Core and Builtins/2022-04-12-00-44-14.gh-issue-91462.t8oxyd.rst M Lib/test/test_lltrace.py M Python/ceval.c diff --git a/Lib/test/test_lltrace.py b/Lib/test/test_lltrace.py index 06e33f4c4c2f3..75b377b637b9a 100644 --- a/Lib/test/test_lltrace.py +++ b/Lib/test/test_lltrace.py @@ -1,21 +1,103 @@ -import os +import dis +import sys import textwrap import unittest -from test.support import os_helper +from test.support import os_helper, verbose from test.support.script_helper import assert_python_ok +def example(): + x = [] + for i in range(1): + x.append(i) + x = "this is" + y = "an example" + print(x, y) +Py_DEBUG = hasattr(sys, 'gettotalrefcount') + + at unittest.skipUnless(Py_DEBUG, "lltrace requires Py_DEBUG") class TestLLTrace(unittest.TestCase): + def run_code(self, code): + code = textwrap.dedent(code).strip() + with open(os_helper.TESTFN, 'w', encoding='utf-8') as fd: + self.addCleanup(os_helper.unlink, os_helper.TESTFN) + fd.write(code) + status, stdout, stderr = assert_python_ok(os_helper.TESTFN) + self.assertEqual(stderr, b"") + self.assertEqual(status, 0) + result = stdout.decode('utf-8') + if verbose: + print("\n\n--- code ---") + print(code) + print("\n--- stdout ---") + print(result) + print() + return result + + def test_lltrace(self): + stdout = self.run_code(""" + def dont_trace_1(): + a = "a" + a = 10 * a + def trace_me(): + for i in range(3): + +i + def dont_trace_2(): + x = 42 + y = -x + dont_trace_1() + __ltrace__ = 1 + trace_me() + del __ltrace__ + dont_trace_2() + """) + self.assertIn("GET_ITER", stdout) + self.assertIn("FOR_ITER", stdout) + self.assertIn("UNARY_POSITIVE", stdout) + self.assertIn("POP_TOP", stdout) + self.assertNotIn("BINARY_OP", stdout) + self.assertNotIn("UNARY_NEGATIVE", stdout) + + self.assertIn("'trace_me' in module '__main__'", stdout) + self.assertNotIn("dont_trace_1", stdout) + self.assertNotIn("'dont_trace_2' in module", stdout) + + def test_lltrace_different_module(self): + stdout = self.run_code(""" + from test import test_lltrace + test_lltrace.__ltrace__ = 1 + test_lltrace.example() + """) + self.assertIn("'example' in module 'test.test_lltrace'", stdout) + self.assertIn('LOAD_CONST', stdout) + self.assertIn('FOR_ITER', stdout) + self.assertIn('this is an example', stdout) + + # check that offsets match the output of dis.dis() + instr_map = {i.offset: i for i in dis.get_instructions(example)} + for line in stdout.splitlines(): + offset, colon, opname_oparg = line.partition(":") + if not colon: + continue + offset = int(offset) + opname_oparg = opname_oparg.split() + if len(opname_oparg) == 2: + opname, oparg = opname_oparg + oparg = int(oparg) + else: + (opname,) = opname_oparg + oparg = None + self.assertEqual(instr_map[offset].opname, opname) + self.assertEqual(instr_map[offset].arg, oparg) + def test_lltrace_does_not_crash_on_subscript_operator(self): # If this test fails, it will reproduce a crash reported as # bpo-34113. The crash happened at the command line console of # debug Python builds with __ltrace__ enabled (only possible in console), # when the internal Python stack was negatively adjusted - with open(os_helper.TESTFN, 'w', encoding='utf-8') as fd: - self.addCleanup(os_helper.unlink, os_helper.TESTFN) - fd.write(textwrap.dedent("""\ + stdout = self.run_code(""" import code console = code.InteractiveConsole() @@ -23,9 +105,8 @@ def test_lltrace_does_not_crash_on_subscript_operator(self): console.push('a = [1, 2, 3]') console.push('a[0] = 1') print('unreachable if bug exists') - """)) - - assert_python_ok(os_helper.TESTFN) + """) + self.assertIn("unreachable if bug exists", stdout) if __name__ == "__main__": unittest.main() diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-04-12-00-44-14.gh-issue-91462.t8oxyd.rst b/Misc/NEWS.d/next/Core and Builtins/2022-04-12-00-44-14.gh-issue-91462.t8oxyd.rst new file mode 100644 index 0000000000000..0656b303a4570 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2022-04-12-00-44-14.gh-issue-91462.t8oxyd.rst @@ -0,0 +1 @@ +Make the interpreter's low-level tracing (lltrace) feature output more readable by displaying opcode names (rather than just numbers), and by displaying stack contents before each opcode. diff --git a/Python/ceval.c b/Python/ceval.c index 66856e5177607..d6f11d8f4364a 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -54,15 +54,77 @@ static PyObject * do_call_core( #ifdef LLTRACE static int lltrace; -static int prtrace(PyThreadState *, PyObject *, const char *); -static void lltrace_instruction(_PyInterpreterFrame *frame, int opcode, int oparg) +static void +dump_stack(_PyInterpreterFrame *frame, PyObject **stack_pointer) +{ + PyObject **stack_base = _PyFrame_Stackbase(frame); + PyObject *type, *value, *traceback; + PyErr_Fetch(&type, &value, &traceback); + printf(" stack=["); + for (PyObject **ptr = stack_base; ptr < stack_pointer; ptr++) { + if (ptr != stack_base) { + printf(", "); + } + if (PyObject_Print(*ptr, stdout, 0) != 0) { + PyErr_Clear(); + printf("<%s object at %p>", + Py_TYPE(*ptr)->tp_name, (void *)(*ptr)); + } + } + printf("]\n"); + fflush(stdout); + PyErr_Restore(type, value, traceback); +} + +static void +lltrace_instruction(_PyInterpreterFrame *frame, + PyObject **stack_pointer, + _Py_CODEUNIT *next_instr) { + dump_stack(frame, stack_pointer); + int oparg = _Py_OPARG(*next_instr); + int opcode = _Py_OPCODE(*next_instr); + const char *opname = _PyOpcode_OpName[opcode]; + assert(opname != NULL); + int offset = (int)(next_instr - _PyCode_CODE(frame->f_code)); if (HAS_ARG(opcode)) { - printf("%d: %d, %d\n", _PyInterpreterFrame_LASTI(frame), opcode, oparg); + printf("%d: %s %d\n", offset * 2, opname, oparg); } else { - printf("%d: %d\n", _PyInterpreterFrame_LASTI(frame), opcode); + printf("%d: %s\n", offset * 2, opname); + } + fflush(stdout); +} +static void +lltrace_resume_frame(_PyInterpreterFrame *frame) +{ + PyFunctionObject *f = frame->f_func; + if (f == NULL) { + printf("\nResuming frame."); + return; + } + PyObject *type, *value, *traceback; + PyErr_Fetch(&type, &value, &traceback); + PyObject *name = f->func_qualname; + if (name == NULL) { + name = f->func_name; + } + printf("\nResuming frame"); + if (name) { + printf(" for "); + if (PyObject_Print(name, stdout, 0) < 0) { + PyErr_Clear(); + } } + if (f->func_module) { + printf(" in module "); + if (PyObject_Print(f->func_module, stdout, 0) < 0) { + PyErr_Clear(); + } + } + printf("\n"); + fflush(stdout); + PyErr_Restore(type, value, traceback); } #endif static int call_trace(Py_tracefunc, PyObject *, @@ -1266,7 +1328,8 @@ eval_frame_handle_pending(PyThreadState *tstate) /* PRE_DISPATCH_GOTO() does lltrace if enabled. Normally a no-op */ #ifdef LLTRACE -#define PRE_DISPATCH_GOTO() if (lltrace) { lltrace_instruction(frame, opcode, oparg); } +#define PRE_DISPATCH_GOTO() if (lltrace) { \ + lltrace_instruction(frame, stack_pointer, next_instr); } #else #define PRE_DISPATCH_GOTO() ((void)0) #endif @@ -1375,6 +1438,7 @@ eval_frame_handle_pending(PyThreadState *tstate) /* The stack can grow at most MAXINT deep, as co_nlocals and co_stacksize are ints. */ #define STACK_LEVEL() ((int)(stack_pointer - _PyFrame_Stackbase(frame))) +#define STACK_SIZE() (frame->f_code->co_stacksize) #define EMPTY() (STACK_LEVEL() == 0) #define TOP() (stack_pointer[-1]) #define SECOND() (stack_pointer[-2]) @@ -1387,23 +1451,21 @@ eval_frame_handle_pending(PyThreadState *tstate) #define BASIC_PUSH(v) (*stack_pointer++ = (v)) #define BASIC_POP() (*--stack_pointer) -#ifdef LLTRACE -#define PUSH(v) { (void)(BASIC_PUSH(v), \ - lltrace && prtrace(tstate, TOP(), "push")); \ - assert(STACK_LEVEL() <= frame->f_code->co_stacksize); } -#define POP() ((void)(lltrace && prtrace(tstate, TOP(), "pop")), \ - BASIC_POP()) +#ifdef Py_DEBUG +#define PUSH(v) do { \ + BASIC_PUSH(v); \ + assert(STACK_LEVEL() <= STACK_SIZE()); \ + } while (0) +#define POP() (assert(STACK_LEVEL() > 0), BASIC_POP()) #define STACK_GROW(n) do { \ - assert(n >= 0); \ - (void)(BASIC_STACKADJ(n), \ - lltrace && prtrace(tstate, TOP(), "stackadj")); \ - assert(STACK_LEVEL() <= frame->f_code->co_stacksize); \ + assert(n >= 0); \ + BASIC_STACKADJ(n); \ + assert(STACK_LEVEL() <= STACK_SIZE()); \ } while (0) #define STACK_SHRINK(n) do { \ assert(n >= 0); \ - (void)(lltrace && prtrace(tstate, TOP(), "stackadj")); \ - (void)(BASIC_STACKADJ(-(n))); \ - assert(STACK_LEVEL() <= frame->f_code->co_stacksize); \ + assert(STACK_LEVEL() >= n); \ + BASIC_STACKADJ(-(n)); \ } while (0) #else #define PUSH(v) BASIC_PUSH(v) @@ -1673,6 +1735,9 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int } lltrace = r; } + if (lltrace) { + lltrace_resume_frame(frame); + } #endif #ifdef Py_DEBUG @@ -6663,23 +6728,6 @@ unpack_iterable(PyThreadState *tstate, PyObject *v, return 0; } -#ifdef LLTRACE -static int -prtrace(PyThreadState *tstate, PyObject *v, const char *str) -{ - printf("%s ", str); - PyObject *type, *value, *traceback; - PyErr_Fetch(&type, &value, &traceback); - if (PyObject_Print(v, stdout, 0) != 0) { - /* Don't know what else to do */ - _PyErr_Clear(tstate); - } - printf("\n"); - PyErr_Restore(type, value, traceback); - return 1; -} -#endif - static void call_exc_trace(Py_tracefunc func, PyObject *self, PyThreadState *tstate, From webhook-mailer at python.org Sat Apr 16 18:57:11 2022 From: webhook-mailer at python.org (sweeneyde) Date: Sat, 16 Apr 2022 22:57:11 -0000 Subject: [Python-checkins] gh-78607: Replace __ltrace__ with __lltrace__ (GH-91619) Message-ID: https://github.com/python/cpython/commit/37965d2fb434d8343d5c70fb6a462a16ae7882b8 commit: 37965d2fb434d8343d5c70fb6a462a16ae7882b8 branch: main author: Dennis Sweeney <36520290+sweeneyde at users.noreply.github.com> committer: sweeneyde <36520290+sweeneyde at users.noreply.github.com> date: 2022-04-16T18:57:00-04:00 summary: gh-78607: Replace __ltrace__ with __lltrace__ (GH-91619) files: A Misc/NEWS.d/next/Core and Builtins/2022-04-16-21-54-31.gh-issue-78607._Y7bMm.rst M Doc/using/configure.rst M Include/internal/pycore_global_strings.h M Include/internal/pycore_runtime_init.h M Lib/test/test_lltrace.py M Misc/SpecialBuilds.txt M Python/ceval.c M Tools/c-analyzer/TODO diff --git a/Doc/using/configure.rst b/Doc/using/configure.rst index b46157cc6ad59..2e632d822f9bd 100644 --- a/Doc/using/configure.rst +++ b/Doc/using/configure.rst @@ -273,7 +273,7 @@ Effects of a debug build: * Add :func:`sys.gettotalrefcount` function. * Add :option:`-X showrefcount <-X>` command line option. * Add :envvar:`PYTHONTHREADDEBUG` environment variable. -* Add support for the ``__ltrace__`` variable: enable low-level tracing in the +* Add support for the ``__lltrace__`` variable: enable low-level tracing in the bytecode evaluation loop if the variable is defined. * Install :ref:`debug hooks on memory allocators ` to detect buffer overflow and other memory errors. diff --git a/Include/internal/pycore_global_strings.h b/Include/internal/pycore_global_strings.h index 77f96cd5ed2f0..cc94662256e87 100644 --- a/Include/internal/pycore_global_strings.h +++ b/Include/internal/pycore_global_strings.h @@ -137,10 +137,10 @@ struct _Py_global_strings { STRUCT_FOR_ID(__le__) STRUCT_FOR_ID(__len__) STRUCT_FOR_ID(__length_hint__) + STRUCT_FOR_ID(__lltrace__) STRUCT_FOR_ID(__loader__) STRUCT_FOR_ID(__lshift__) STRUCT_FOR_ID(__lt__) - STRUCT_FOR_ID(__ltrace__) STRUCT_FOR_ID(__main__) STRUCT_FOR_ID(__matmul__) STRUCT_FOR_ID(__missing__) diff --git a/Include/internal/pycore_runtime_init.h b/Include/internal/pycore_runtime_init.h index 371f2d23ad8ec..9a3a9d04324ba 100644 --- a/Include/internal/pycore_runtime_init.h +++ b/Include/internal/pycore_runtime_init.h @@ -760,10 +760,10 @@ extern "C" { INIT_ID(__le__), \ INIT_ID(__len__), \ INIT_ID(__length_hint__), \ + INIT_ID(__lltrace__), \ INIT_ID(__loader__), \ INIT_ID(__lshift__), \ INIT_ID(__lt__), \ - INIT_ID(__ltrace__), \ INIT_ID(__main__), \ INIT_ID(__matmul__), \ INIT_ID(__missing__), \ diff --git a/Lib/test/test_lltrace.py b/Lib/test/test_lltrace.py index 75b377b637b9a..63b65e4b2f821 100644 --- a/Lib/test/test_lltrace.py +++ b/Lib/test/test_lltrace.py @@ -48,9 +48,9 @@ def dont_trace_2(): x = 42 y = -x dont_trace_1() - __ltrace__ = 1 + __lltrace__ = 1 trace_me() - del __ltrace__ + del __lltrace__ dont_trace_2() """) self.assertIn("GET_ITER", stdout) @@ -67,7 +67,7 @@ def dont_trace_2(): def test_lltrace_different_module(self): stdout = self.run_code(""" from test import test_lltrace - test_lltrace.__ltrace__ = 1 + test_lltrace.__lltrace__ = 1 test_lltrace.example() """) self.assertIn("'example' in module 'test.test_lltrace'", stdout) @@ -95,13 +95,13 @@ def test_lltrace_different_module(self): def test_lltrace_does_not_crash_on_subscript_operator(self): # If this test fails, it will reproduce a crash reported as # bpo-34113. The crash happened at the command line console of - # debug Python builds with __ltrace__ enabled (only possible in console), + # debug Python builds with __lltrace__ enabled (only possible in console), # when the internal Python stack was negatively adjusted stdout = self.run_code(""" import code console = code.InteractiveConsole() - console.push('__ltrace__ = 1') + console.push('__lltrace__ = 1') console.push('a = [1, 2, 3]') console.push('a[0] = 1') print('unreachable if bug exists') diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-04-16-21-54-31.gh-issue-78607._Y7bMm.rst b/Misc/NEWS.d/next/Core and Builtins/2022-04-16-21-54-31.gh-issue-78607._Y7bMm.rst new file mode 100644 index 0000000000000..808aedd50981e --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2022-04-16-21-54-31.gh-issue-78607._Y7bMm.rst @@ -0,0 +1 @@ +The LLTRACE special build now looks for the name ``__lltrace__`` defined in module globals, rather than the name ``__ltrace__``, which had been introduced as a typo. diff --git a/Misc/SpecialBuilds.txt b/Misc/SpecialBuilds.txt index a7cee8098e9c0..5609928284d44 100644 --- a/Misc/SpecialBuilds.txt +++ b/Misc/SpecialBuilds.txt @@ -94,7 +94,7 @@ Compile in support for Low Level TRACE-ing of the main interpreter loop. When this preprocessor symbol is defined, before PyEval_EvalFrame executes a frame's code it checks the frame's global namespace for a variable -"__ltrace__". If such a variable is found, mounds of information about what +"__lltrace__". If such a variable is found, mounds of information about what the interpreter is doing are sprayed to stdout, such as every opcode and opcode argument and values pushed onto and popped off the value stack. diff --git a/Python/ceval.c b/Python/ceval.c index d6f11d8f4364a..d358a3134bc63 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -1729,7 +1729,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int #ifdef LLTRACE { - int r = PyDict_Contains(GLOBALS(), &_Py_ID(__ltrace__)); + int r = PyDict_Contains(GLOBALS(), &_Py_ID(__lltrace__)); if (r < 0) { goto exit_unwind; } diff --git a/Tools/c-analyzer/TODO b/Tools/c-analyzer/TODO index 6683df5993b8c..43760369b1980 100644 --- a/Tools/c-analyzer/TODO +++ b/Tools/c-analyzer/TODO @@ -468,7 +468,6 @@ Python/ceval.c:_PyEval_EvalFrameDefault():PyId___annotations__ _Py_IDENTIFIER( Python/ceval.c:_PyEval_EvalFrameDefault():PyId___build_class__ _Py_IDENTIFIER(__build_class__) Python/ceval.c:_PyEval_EvalFrameDefault():PyId___enter__ _Py_IDENTIFIER(__enter__) Python/ceval.c:_PyEval_EvalFrameDefault():PyId___exit__ _Py_IDENTIFIER(__exit__) -Python/ceval.c:_PyEval_EvalFrameDefault():PyId___ltrace__ _Py_IDENTIFIER(__ltrace__) Python/ceval.c:_PyEval_EvalFrameDefault():PyId_displayhook _Py_IDENTIFIER(displayhook) Python/ceval.c:_PyEval_EvalFrameDefault():PyId_send _Py_IDENTIFIER(send) Python/ceval.c:import_all_from():PyId___all__ _Py_IDENTIFIER(__all__) From webhook-mailer at python.org Sun Apr 17 01:47:52 2022 From: webhook-mailer at python.org (gpshead) Date: Sun, 17 Apr 2022 05:47:52 -0000 Subject: [Python-checkins] [3.9] [3.10] gh-91607: Fix several test_concurrent_futures tests to actually test what they claim (GH-91600) (GH-91612) (#91617) Message-ID: https://github.com/python/cpython/commit/2a43afdba94b86e1e11e8d311c66e40b77675500 commit: 2a43afdba94b86e1e11e8d311c66e40b77675500 branch: 3.9 author: Gregory P. Smith committer: gpshead date: 2022-04-16T22:47:41-07:00 summary: [3.9] [3.10] gh-91607: Fix several test_concurrent_futures tests to actually test what they claim (GH-91600) (GH-91612) (#91617) * Fix test_concurrent_futures to actually test what it says. Many ProcessPoolExecutor based tests were ignoring the mp_context and using the default instead. This meant we lacked proper test coverage of all of them. Also removes the old _prime_executor() worker delay seeding code as it appears to have no point and causes 20-30 seconds extra latency on this already long test. It also interfered with some of the refactoring to fix the above to not needlessly create their own executor when setUp has already created an appropriate one. * Don't import the name from multiprocessing directly to avoid confusion. (cherry picked from commit 7fa3a5a2197896066e3fe53ee325ac6ab54c3414) (cherry picked from commit 9a458934f7be37c59525350f1f9ecbdcd69903c1) files: A Misc/NEWS.d/next/Tests/2022-04-16-17-54-05.gh-issue-91607.FnXjtW.rst M Lib/test/test_concurrent_futures.py diff --git a/Lib/test/test_concurrent_futures.py b/Lib/test/test_concurrent_futures.py index d421b02754cf5..81bc0e14261c3 100644 --- a/Lib/test/test_concurrent_futures.py +++ b/Lib/test/test_concurrent_futures.py @@ -26,10 +26,10 @@ PENDING, RUNNING, CANCELLED, CANCELLED_AND_NOTIFIED, FINISHED, Future, BrokenExecutor) from concurrent.futures.process import BrokenProcessPool -from multiprocessing import get_context import multiprocessing.process import multiprocessing.util +import multiprocessing as mp if support.check_sanitizer(address=True, memory=True): @@ -131,7 +131,6 @@ def setUp(self): self.executor = self.executor_type( max_workers=self.worker_count, **self.executor_kwargs) - self._prime_executor() def tearDown(self): self.executor.shutdown(wait=True) @@ -145,15 +144,7 @@ def tearDown(self): super().tearDown() def get_context(self): - return get_context(self.ctx) - - def _prime_executor(self): - # Make sure that the executor is ready to do work before running the - # tests. This should reduce the probability of timeouts in the tests. - futures = [self.executor.submit(time.sleep, 0.1) - for _ in range(self.worker_count)] - for f in futures: - f.result() + return mp.get_context(self.ctx) class ThreadPoolMixin(ExecutorMixin): @@ -261,9 +252,6 @@ def test_initializer(self): with self.assertRaises(BrokenExecutor): self.executor.submit(get_init_status) - def _prime_executor(self): - pass - @contextlib.contextmanager def _assert_logged(self, msg): if self.log_queue is not None: @@ -350,14 +338,14 @@ def test_hang_issue12364(self): f.result() def test_cancel_futures(self): - executor = self.executor_type(max_workers=3) - fs = [executor.submit(time.sleep, .1) for _ in range(50)] - executor.shutdown(cancel_futures=True) + assert self.worker_count <= 5, "test needs few workers" + fs = [self.executor.submit(time.sleep, .1) for _ in range(50)] + self.executor.shutdown(cancel_futures=True) # We can't guarantee the exact number of cancellations, but we can - # guarantee that *some* were cancelled. With setting max_workers to 3, - # most of the submitted futures should have been cancelled. + # guarantee that *some* were cancelled. With few workers, many of + # the submitted futures should have been cancelled. cancelled = [fut for fut in fs if fut.cancelled()] - self.assertTrue(len(cancelled) >= 35, msg=f"{len(cancelled)=}") + self.assertGreater(len(cancelled), 20) # Ensure the other futures were able to finish. # Use "not fut.cancelled()" instead of "fut.done()" to include futures @@ -370,33 +358,32 @@ def test_cancel_futures(self): # Similar to the number of cancelled futures, we can't guarantee the # exact number that completed. But, we can guarantee that at least # one finished. - self.assertTrue(len(others) > 0, msg=f"{len(others)=}") + self.assertGreater(len(others), 0) - def test_hang_issue39205(self): + def test_hang_gh83386(self): """shutdown(wait=False) doesn't hang at exit with running futures. - See https://bugs.python.org/issue39205. + See https://github.com/python/cpython/issues/83386. """ if self.executor_type == futures.ProcessPoolExecutor: raise unittest.SkipTest( - "Hangs due to https://bugs.python.org/issue39205") + "Hangs, see https://github.com/python/cpython/issues/83386") rc, out, err = assert_python_ok('-c', """if True: from concurrent.futures import {executor_type} from test.test_concurrent_futures import sleep_and_print if __name__ == "__main__": + if {context!r}: multiprocessing.set_start_method({context!r}) t = {executor_type}(max_workers=3) t.submit(sleep_and_print, 1.0, "apple") t.shutdown(wait=False) - """.format(executor_type=self.executor_type.__name__)) + """.format(executor_type=self.executor_type.__name__, + context=getattr(self, 'ctx', None))) self.assertFalse(err) self.assertEqual(out.strip(), b"apple") class ThreadPoolShutdownTest(ThreadPoolMixin, ExecutorShutdownTest, BaseTestCase): - def _prime_executor(self): - pass - def test_threads_terminate(self): def acquire_lock(lock): lock.acquire() @@ -491,14 +478,11 @@ def test_cancel_futures_wait_false(self): class ProcessPoolShutdownTest(ExecutorShutdownTest): - def _prime_executor(self): - pass - def test_processes_terminate(self): def acquire_lock(lock): lock.acquire() - mp_context = get_context() + mp_context = self.get_context() sem = mp_context.Semaphore(0) for _ in range(3): self.executor.submit(acquire_lock, sem) @@ -512,7 +496,8 @@ def acquire_lock(lock): p.join() def test_context_manager_shutdown(self): - with futures.ProcessPoolExecutor(max_workers=5) as e: + with futures.ProcessPoolExecutor( + max_workers=5, mp_context=self.get_context()) as e: processes = e._processes self.assertEqual(list(e.map(abs, range(-5, 5))), [5, 4, 3, 2, 1, 0, 1, 2, 3, 4]) @@ -521,7 +506,8 @@ def test_context_manager_shutdown(self): p.join() def test_del_shutdown(self): - executor = futures.ProcessPoolExecutor(max_workers=5) + executor = futures.ProcessPoolExecutor( + max_workers=5, mp_context=self.get_context()) res = executor.map(abs, range(-5, 5)) executor_manager_thread = executor._executor_manager_thread processes = executor._processes @@ -544,7 +530,8 @@ def test_del_shutdown(self): def test_shutdown_no_wait(self): # Ensure that the executor cleans up the processes when calling # shutdown with wait=False - executor = futures.ProcessPoolExecutor(max_workers=5) + executor = futures.ProcessPoolExecutor( + max_workers=5, mp_context=self.get_context()) res = executor.map(abs, range(-5, 5)) processes = executor._processes call_queue = executor._call_queue @@ -921,7 +908,7 @@ def submit(pool): pool.submit(submit, pool) for _ in range(50): - with futures.ProcessPoolExecutor(1, mp_context=get_context('fork')) as workers: + with futures.ProcessPoolExecutor(1, mp_context=mp.get_context('fork')) as workers: workers.submit(tuple) @@ -991,7 +978,7 @@ def test_traceback(self): def test_ressources_gced_in_workers(self): # Ensure that argument for a job are correctly gc-ed after the job # is finished - mgr = get_context(self.ctx).Manager() + mgr = self.get_context().Manager() obj = EventfulGCObj(mgr) future = self.executor.submit(id, obj) future.result() @@ -1007,36 +994,37 @@ def test_ressources_gced_in_workers(self): mgr.join() def test_saturation(self): - executor = self.executor_type(4) - mp_context = get_context() + executor = self.executor + mp_context = self.get_context() sem = mp_context.Semaphore(0) job_count = 15 * executor._max_workers - try: - for _ in range(job_count): - executor.submit(sem.acquire) - self.assertEqual(len(executor._processes), executor._max_workers) - for _ in range(job_count): - sem.release() - finally: - executor.shutdown() + for _ in range(job_count): + executor.submit(sem.acquire) + self.assertEqual(len(executor._processes), executor._max_workers) + for _ in range(job_count): + sem.release() def test_idle_process_reuse_one(self): - executor = self.executor_type(4) + executor = self.executor + assert executor._max_workers >= 4 executor.submit(mul, 21, 2).result() executor.submit(mul, 6, 7).result() executor.submit(mul, 3, 14).result() self.assertEqual(len(executor._processes), 1) - executor.shutdown() def test_idle_process_reuse_multiple(self): - executor = self.executor_type(4) + executor = self.executor + assert executor._max_workers <= 5 executor.submit(mul, 12, 7).result() executor.submit(mul, 33, 25) executor.submit(mul, 25, 26).result() executor.submit(mul, 18, 29) - self.assertLessEqual(len(executor._processes), 2) + executor.submit(mul, 1, 2).result() + executor.submit(mul, 0, 9) + self.assertLessEqual(len(executor._processes), 3) executor.shutdown() + create_executor_tests(ProcessPoolExecutorTest, executor_mixins=(ProcessPoolForkMixin, ProcessPoolForkserverMixin, @@ -1138,7 +1126,7 @@ def _check_crash(self, error, func, *args, ignore_stderr=False): self.executor.shutdown(wait=True) executor = self.executor_type( - max_workers=2, mp_context=get_context(self.ctx)) + max_workers=2, mp_context=self.get_context()) res = executor.submit(func, *args) if ignore_stderr: @@ -1217,7 +1205,7 @@ def test_shutdown_deadlock(self): # if a worker fails after the shutdown call. self.executor.shutdown(wait=True) with self.executor_type(max_workers=2, - mp_context=get_context(self.ctx)) as executor: + mp_context=self.get_context()) as executor: self.executor = executor # Allow clean up in fail_on_deadlock f = executor.submit(_crash, delay=.1) executor.shutdown(wait=True) @@ -1230,7 +1218,7 @@ def test_shutdown_deadlock_pickle(self): # Reported in bpo-39104. self.executor.shutdown(wait=True) with self.executor_type(max_workers=2, - mp_context=get_context(self.ctx)) as executor: + mp_context=self.get_context()) as executor: self.executor = executor # Allow clean up in fail_on_deadlock # Start the executor and get the executor_manager_thread to collect diff --git a/Misc/NEWS.d/next/Tests/2022-04-16-17-54-05.gh-issue-91607.FnXjtW.rst b/Misc/NEWS.d/next/Tests/2022-04-16-17-54-05.gh-issue-91607.FnXjtW.rst new file mode 100644 index 0000000000000..32839a826a41e --- /dev/null +++ b/Misc/NEWS.d/next/Tests/2022-04-16-17-54-05.gh-issue-91607.FnXjtW.rst @@ -0,0 +1 @@ +Fix ``test_concurrent_futures`` to test the correct multiprocessing start method context in several cases where the test logic mixed this up. From webhook-mailer at python.org Sun Apr 17 02:52:57 2022 From: webhook-mailer at python.org (sweeneyde) Date: Sun, 17 Apr 2022 06:52:57 -0000 Subject: [Python-checkins] Fix refleaks in PyErr_SetHandledException (GH-91627) Message-ID: https://github.com/python/cpython/commit/3289209716011b21128f45de69f4a8e9d376c78e commit: 3289209716011b21128f45de69f4a8e9d376c78e branch: main author: Dennis Sweeney <36520290+sweeneyde at users.noreply.github.com> committer: sweeneyde <36520290+sweeneyde at users.noreply.github.com> date: 2022-04-17T02:52:53-04:00 summary: Fix refleaks in PyErr_SetHandledException (GH-91627) files: M Python/errors.c diff --git a/Python/errors.c b/Python/errors.c index ce7785855b8e5..3eb8a5ef04d28 100644 --- a/Python/errors.c +++ b/Python/errors.c @@ -520,9 +520,7 @@ PyErr_GetHandledException(void) void _PyErr_SetHandledException(PyThreadState *tstate, PyObject *exc) { - PyObject *oldexc = tstate->exc_info->exc_value; - tstate->exc_info->exc_value = Py_XNewRef(exc); - Py_XDECREF(oldexc); + Py_XSETREF(tstate->exc_info->exc_value, Py_XNewRef(exc)); } void @@ -543,6 +541,7 @@ void PyErr_SetExcInfo(PyObject *type, PyObject *value, PyObject *traceback) { PyErr_SetHandledException(value); + Py_XDECREF(value); /* These args are no longer used, but we still need to steal a ref */ Py_XDECREF(type); Py_XDECREF(traceback); From webhook-mailer at python.org Sun Apr 17 11:10:31 2022 From: webhook-mailer at python.org (jaraco) Date: Sun, 17 Apr 2022 15:10:31 -0000 Subject: [Python-checkins] bpo-46126: Restore docstrings in importlib.metadata tests. (#32288) Message-ID: https://github.com/python/cpython/commit/67712e71b3f20541326d57a475f4bd369a98d918 commit: 67712e71b3f20541326d57a475f4bd369a98d918 branch: main author: Jason R. Coombs committer: jaraco date: 2022-04-17T11:10:26-04:00 summary: bpo-46126: Restore docstrings in importlib.metadata tests. (#32288) files: M Lib/test/test_importlib/test_main.py M Lib/test/test_importlib/test_metadata_api.py diff --git a/Lib/test/test_importlib/test_main.py b/Lib/test/test_importlib/test_main.py index 2e120f7ac50ac..c80a0e4a5a9f9 100644 --- a/Lib/test/test_importlib/test_main.py +++ b/Lib/test/test_importlib/test_main.py @@ -37,10 +37,12 @@ def test_for_name_does_not_exist(self): Distribution.from_name('does-not-exist') def test_package_not_found_mentions_metadata(self): - # When a package is not found, that could indicate that the - # packgae is not installed or that it is installed without - # metadata. Ensure the exception mentions metadata to help - # guide users toward the cause. See #124. + """ + When a package is not found, that could indicate that the + packgae is not installed or that it is installed without + metadata. Ensure the exception mentions metadata to help + guide users toward the cause. See #124. + """ with self.assertRaises(PackageNotFoundError) as ctx: Distribution.from_name('does-not-exist') @@ -89,8 +91,10 @@ def pkg_with_dashes(site_dir): return 'my-pkg' def test_dashes_in_dist_name_found_as_underscores(self): - # For a package with a dash in the name, the dist-info metadata - # uses underscores in the name. Ensure the metadata loads. + """ + For a package with a dash in the name, the dist-info metadata + uses underscores in the name. Ensure the metadata loads. + """ pkg_name = self.pkg_with_dashes(self.site_dir) assert version(pkg_name) == '1.0' @@ -108,7 +112,9 @@ def pkg_with_mixed_case(site_dir): return 'CherryPy' def test_dist_name_found_as_any_case(self): - # Ensure the metadata loads when queried with any case. + """ + Ensure the metadata loads when queried with any case. + """ pkg_name = self.pkg_with_mixed_case(self.site_dir) assert version(pkg_name) == '1.0' assert version(pkg_name.lower()) == '1.0' @@ -244,11 +250,13 @@ def test_repr(self): assert "'name'" in repr(self.ep) def test_hashable(self): - # EntryPoints should be hashable. + """EntryPoints should be hashable""" hash(self.ep) def test_json_dump(self): - # json should not expect to be able to dump an EntryPoint. + """ + json should not expect to be able to dump an EntryPoint + """ with self.assertRaises(Exception): with warnings.catch_warnings(record=True): json.dumps(self.ep) @@ -260,7 +268,9 @@ def test_attr(self): assert self.ep.attr is None def test_sortable(self): - # EntryPoint objects are sortable, but result is undefined. + """ + EntryPoint objects are sortable, but result is undefined. + """ sorted( [ EntryPoint(name='b', value='val', group='group'), @@ -273,8 +283,10 @@ class FileSystem( fixtures.OnSysPath, fixtures.SiteDir, fixtures.FileBuilder, unittest.TestCase ): def test_unicode_dir_on_sys_path(self): - # Ensure a Unicode subdirectory of a directory on sys.path - # does not crash. + """ + Ensure a Unicode subdirectory of a directory on sys.path + does not crash. + """ fixtures.build_files( {self.unicode_filename(): {}}, prefix=self.site_dir, diff --git a/Lib/test/test_importlib/test_metadata_api.py b/Lib/test/test_importlib/test_metadata_api.py index b3627cbb75bd7..c86bb4df19c5e 100644 --- a/Lib/test/test_importlib/test_metadata_api.py +++ b/Lib/test/test_importlib/test_metadata_api.py @@ -90,8 +90,10 @@ def test_entry_points_distribution(self): self.assertEqual(ep.dist.version, "1.0.0") def test_entry_points_unique_packages(self): - # Entry points should only be exposed for the first package - # on sys.path with a given name. + """ + Entry points should only be exposed for the first package + on sys.path with a given name. + """ alt_site_dir = self.fixtures.enter_context(fixtures.tempdir()) self.fixtures.enter_context(self.add_sys_path(alt_site_dir)) alt_pkg = { @@ -123,9 +125,11 @@ def test_entry_points_missing_group(self): assert entry_points(group='missing') == () def test_entry_points_dict_construction(self): - # Prior versions of entry_points() returned simple lists and - # allowed casting those lists into maps by name using ``dict()``. - # Capture this now deprecated use-case. + """ + Prior versions of entry_points() returned simple lists and + allowed casting those lists into maps by name using ``dict()``. + Capture this now deprecated use-case. + """ with suppress_known_deprecation() as caught: eps = dict(entry_points(group='entries')) @@ -154,9 +158,11 @@ def test_entry_points_by_index(self): assert "Accessing entry points by index is deprecated" in str(expected) def test_entry_points_groups_getitem(self): - # Prior versions of entry_points() returned a dict. Ensure - # that callers using '.__getitem__()' are supported but warned to - # migrate. + """ + Prior versions of entry_points() returned a dict. Ensure + that callers using '.__getitem__()' are supported but warned to + migrate. + """ with suppress_known_deprecation(): entry_points()['entries'] == entry_points(group='entries') @@ -164,9 +170,11 @@ def test_entry_points_groups_getitem(self): entry_points()['missing'] def test_entry_points_groups_get(self): - # Prior versions of entry_points() returned a dict. Ensure - # that callers using '.get()' are supported but warned to - # migrate. + """ + Prior versions of entry_points() returned a dict. Ensure + that callers using '.get()' are supported but warned to + migrate. + """ with suppress_known_deprecation(): entry_points().get('missing', 'default') == 'default' entry_points().get('entries', 'default') == entry_points()['entries'] @@ -313,7 +321,7 @@ def test_find_distributions_specified_path(self): assert any(dist.metadata['Name'] == 'distinfo-pkg' for dist in dists) def test_distribution_at_pathlib(self): - # Demonstrate how to load metadata direct from a directory. + """Demonstrate how to load metadata direct from a directory.""" dist_info_path = self.site_dir / 'distinfo_pkg-1.0.0.dist-info' dist = Distribution.at(dist_info_path) assert dist.version == '1.0.0' From webhook-mailer at python.org Sun Apr 17 11:10:41 2022 From: webhook-mailer at python.org (jaraco) Date: Sun, 17 Apr 2022 15:10:41 -0000 Subject: [Python-checkins] gh-91298: Refine traversable (apply changes from importlib_resources 5.7.1) (#91623) Message-ID: https://github.com/python/cpython/commit/7659681556977fe3a19d9f4c5dd93362b9eae25c commit: 7659681556977fe3a19d9f4c5dd93362b9eae25c branch: main author: Jason R. Coombs committer: jaraco date: 2022-04-17T11:10:36-04:00 summary: gh-91298: Refine traversable (apply changes from importlib_resources 5.7.1) (#91623) * bpo-47142: Refine traversable (apply changes from importlib_resources 5.7.1) * Replace changelog referencing github issue. files: A Misc/NEWS.d/next/Documentation/2022-04-17-03-19-51.gh-issue-91298.NT9qHi.rst M Lib/importlib/resources/abc.py M Lib/importlib/resources/simple.py M Lib/test/test_importlib/update-zips.py diff --git a/Lib/importlib/resources/abc.py b/Lib/importlib/resources/abc.py index e9efdab5ea830..0b7bfdc415829 100644 --- a/Lib/importlib/resources/abc.py +++ b/Lib/importlib/resources/abc.py @@ -1,6 +1,14 @@ import abc -from typing import BinaryIO, Iterable, Text +import io +import os +from typing import Any, BinaryIO, Iterable, Iterator, NoReturn, Text, Optional from typing import runtime_checkable, Protocol +from typing import Union + + +StrPath = Union[str, os.PathLike[str]] + +__all__ = ["ResourceReader", "Traversable", "TraversableResources"] class ResourceReader(metaclass=abc.ABCMeta): @@ -50,22 +58,25 @@ class Traversable(Protocol): """ An object with a subset of pathlib.Path methods suitable for traversing directories and opening files. + + Any exceptions that occur when accessing the backing resource + may propagate unaltered. """ @abc.abstractmethod - def iterdir(self): + def iterdir(self) -> Iterator["Traversable"]: """ Yield Traversable objects in self """ - def read_bytes(self): + def read_bytes(self) -> bytes: """ Read contents of self as bytes """ with self.open('rb') as strm: return strm.read() - def read_text(self, encoding=None): + def read_text(self, encoding: Optional[str] = None) -> str: """ Read contents of self as text """ @@ -85,12 +96,16 @@ def is_file(self) -> bool: """ @abc.abstractmethod - def joinpath(self, child): + def joinpath(self, *descendants: StrPath) -> "Traversable": """ - Return Traversable child in self + Return Traversable resolved with any descendants applied. + + Each descendant should be a path segment relative to self + and each may contain multiple levels separated by + ``posixpath.sep`` (``/``). """ - def __truediv__(self, child): + def __truediv__(self, child: StrPath) -> "Traversable": """ Return Traversable child in self """ @@ -120,17 +135,17 @@ class TraversableResources(ResourceReader): """ @abc.abstractmethod - def files(self): + def files(self) -> "Traversable": """Return a Traversable object for the loaded package.""" - def open_resource(self, resource): + def open_resource(self, resource: StrPath) -> io.BufferedReader: return self.files().joinpath(resource).open('rb') - def resource_path(self, resource): + def resource_path(self, resource: Any) -> NoReturn: raise FileNotFoundError(resource) - def is_resource(self, path): + def is_resource(self, path: StrPath) -> bool: return self.files().joinpath(path).is_file() - def contents(self): + def contents(self) -> Iterator[str]: return (item.name for item in self.files().iterdir()) diff --git a/Lib/importlib/resources/simple.py b/Lib/importlib/resources/simple.py index da073cbdb11e6..d0fbf237762c2 100644 --- a/Lib/importlib/resources/simple.py +++ b/Lib/importlib/resources/simple.py @@ -99,10 +99,19 @@ def iterdir(self): def open(self, *args, **kwargs): raise IsADirectoryError() - def joinpath(self, name): + @staticmethod + def _flatten(compound_names): + for name in compound_names: + yield from name.split('/') + + def joinpath(self, *descendants): + if not descendants: + return self + names = self._flatten(descendants) + target = next(names) return next( - traversable for traversable in self.iterdir() if traversable.name == name - ) + traversable for traversable in self.iterdir() if traversable.name == target + ).joinpath(*names) class TraversableReader(TraversableResources, SimpleReader): diff --git a/Lib/test/test_importlib/update-zips.py b/Lib/test/test_importlib/update-zips.py index 9ef0224ca65ca..231334aa7e38b 100755 --- a/Lib/test/test_importlib/update-zips.py +++ b/Lib/test/test_importlib/update-zips.py @@ -42,7 +42,7 @@ def generate(suffix): def walk(datapath): for dirpath, dirnames, filenames in os.walk(datapath): - with contextlib.suppress(KeyError): + with contextlib.suppress(ValueError): dirnames.remove('__pycache__') for filename in filenames: res = pathlib.Path(dirpath) / filename diff --git a/Misc/NEWS.d/next/Documentation/2022-04-17-03-19-51.gh-issue-91298.NT9qHi.rst b/Misc/NEWS.d/next/Documentation/2022-04-17-03-19-51.gh-issue-91298.NT9qHi.rst new file mode 100644 index 0000000000000..471a7ced6886b --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2022-04-17-03-19-51.gh-issue-91298.NT9qHi.rst @@ -0,0 +1,2 @@ +In ``importlib.resources.abc``, refined the documentation of the Traversable +Protocol, applying changes from importlib_resources 5.7.1. From webhook-mailer at python.org Sun Apr 17 14:04:38 2022 From: webhook-mailer at python.org (sweeneyde) Date: Sun, 17 Apr 2022 18:04:38 -0000 Subject: [Python-checkins] gh-91625: Don't ignore extended args of adaptive opcodes (GH-91626) Message-ID: https://github.com/python/cpython/commit/cec5d858f509ea28a5325b75fd94e2f275866f5f commit: cec5d858f509ea28a5325b75fd94e2f275866f5f branch: main author: Dennis Sweeney <36520290+sweeneyde at users.noreply.github.com> committer: sweeneyde <36520290+sweeneyde at users.noreply.github.com> date: 2022-04-17T14:04:29-04:00 summary: gh-91625: Don't ignore extended args of adaptive opcodes (GH-91626) files: A Misc/NEWS.d/next/Core and Builtins/2022-04-17-02-55-38.gh-issue-91625.80CrC7.rst M Lib/test/test_descr.py M Lib/test/test_dynamic.py M Lib/test/test_unpack.py M Python/ceval.c diff --git a/Lib/test/test_descr.py b/Lib/test/test_descr.py index 5d36cb98792ac..378ff5227e221 100644 --- a/Lib/test/test_descr.py +++ b/Lib/test/test_descr.py @@ -1961,6 +1961,20 @@ def __delitem__(self, key): del a[0:10] self.assertEqual(a.delitem, (slice(0, 10))) + def test_load_attr_extended_arg(self): + # https://github.com/python/cpython/issues/91625 + class Numbers: + def __getattr__(self, attr): + return int(attr.lstrip("_")) + attrs = ", ".join(f"Z._{n:03d}" for n in range(280)) + code = f"def number_attrs(Z):\n return [ {attrs} ]" + ns = {} + exec(code, ns) + number_attrs = ns["number_attrs"] + # Warm up the the function for quickening (PEP 659) + for _ in range(30): + self.assertEqual(number_attrs(Numbers()), list(range(280))) + def test_methods(self): # Testing methods... class C(object): @@ -4435,8 +4449,8 @@ def __getattr__(self, attr): raise RuntimeError(f"Premature access to sys.stdout.{attr}") with redirect_stdout(StdoutGuard()): - with self.assertRaises(RuntimeError): - print("Oops!") + with self.assertRaises(RuntimeError): + print("Oops!") def test_vicious_descriptor_nonsense(self): # Testing vicious_descriptor_nonsense... diff --git a/Lib/test/test_dynamic.py b/Lib/test/test_dynamic.py index 3ae090fd66ae2..3e0fcf4d158f8 100644 --- a/Lib/test/test_dynamic.py +++ b/Lib/test/test_dynamic.py @@ -133,6 +133,18 @@ def test_eval_gives_lambda_custom_globals(self): self.assertEqual(foo(), 7) + def test_load_global_specialization_failure_keeps_oparg(self): + # https://github.com/python/cpython/issues/91625 + class MyGlobals(dict): + def __missing__(self, key): + return int(key.removeprefix("_number_")) + + code = "lambda: " + "+".join(f"_number_{i}" for i in range(1000)) + sum_1000 = eval(code, MyGlobals()) + expected = sum(range(1000)) + # Warm up the the function for quickening (PEP 659) + for _ in range(30): + self.assertEqual(sum_1000(), expected) if __name__ == "__main__": unittest.main() diff --git a/Lib/test/test_unpack.py b/Lib/test/test_unpack.py index 472c8343eb243..f5ca1d455b5c6 100644 --- a/Lib/test/test_unpack.py +++ b/Lib/test/test_unpack.py @@ -151,5 +151,21 @@ def load_tests(loader, tests, pattern): return tests +class TestCornerCases(unittest.TestCase): + def test_extended_oparg_not_ignored(self): + # https://github.com/python/cpython/issues/91625 + target = "(" + "y,"*400 + ")" + code = f"""def unpack_400(x): + {target} = x + return y + """ + ns = {} + exec(code, ns) + unpack_400 = ns["unpack_400"] + # Warm up the the function for quickening (PEP 659) + for _ in range(30): + y = unpack_400(range(400)) + self.assertEqual(y, 399) + if __name__ == "__main__": unittest.main() diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-04-17-02-55-38.gh-issue-91625.80CrC7.rst b/Misc/NEWS.d/next/Core and Builtins/2022-04-17-02-55-38.gh-issue-91625.80CrC7.rst new file mode 100644 index 0000000000000..ea5b57b2b4719 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2022-04-17-02-55-38.gh-issue-91625.80CrC7.rst @@ -0,0 +1 @@ +Fixed a bug in which adaptive opcodes ignored any preceding ``EXTENDED_ARG``\ s on specialization failure. diff --git a/Python/ceval.c b/Python/ceval.c index d358a3134bc63..f523e52fe0116 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -1351,6 +1351,13 @@ eval_frame_handle_pending(PyThreadState *tstate) DISPATCH_GOTO(); \ } +#define NOTRACE_DISPATCH_SAME_OPARG() \ + { \ + opcode = _Py_OPCODE(*next_instr); \ + PRE_DISPATCH_GOTO(); \ + DISPATCH_GOTO(); \ + } + #define CHECK_EVAL_BREAKER() \ _Py_CHECK_EMSCRIPTEN_SIGNALS_PERIODICALLY(); \ if (_Py_atomic_load_relaxed(eval_breaker)) { \ @@ -2158,7 +2165,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int if (_Py_Specialize_BinarySubscr(container, sub, next_instr) < 0) { goto error; } - DISPATCH(); + NOTRACE_DISPATCH_SAME_OPARG(); } else { STAT_INC(BINARY_SUBSCR, deferred); @@ -2323,7 +2330,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int if (_Py_Specialize_StoreSubscr(container, sub, next_instr) < 0) { goto error; } - DISPATCH(); + NOTRACE_DISPATCH_SAME_OPARG(); } else { STAT_INC(STORE_SUBSCR, deferred); @@ -2813,7 +2820,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int PyObject *seq = TOP(); next_instr--; _Py_Specialize_UnpackSequence(seq, next_instr, oparg); - DISPATCH(); + NOTRACE_DISPATCH_SAME_OPARG(); } else { STAT_INC(UNPACK_SEQUENCE, deferred); @@ -3056,7 +3063,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int if (_Py_Specialize_LoadGlobal(GLOBALS(), BUILTINS(), next_instr, name) < 0) { goto error; } - DISPATCH(); + NOTRACE_DISPATCH_SAME_OPARG(); } else { STAT_INC(LOAD_GLOBAL, deferred); @@ -3481,7 +3488,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int if (_Py_Specialize_LoadAttr(owner, next_instr, name) < 0) { goto error; } - DISPATCH(); + NOTRACE_DISPATCH_SAME_OPARG(); } else { STAT_INC(LOAD_ATTR, deferred); @@ -3590,7 +3597,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int if (_Py_Specialize_StoreAttr(owner, next_instr, name) < 0) { goto error; } - DISPATCH(); + NOTRACE_DISPATCH_SAME_OPARG(); } else { STAT_INC(STORE_ATTR, deferred); @@ -3718,7 +3725,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int PyObject *left = SECOND(); next_instr--; _Py_Specialize_CompareOp(left, right, next_instr, oparg); - DISPATCH(); + NOTRACE_DISPATCH_SAME_OPARG(); } else { STAT_INC(COMPARE_OP, deferred); @@ -4523,7 +4530,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int if (_Py_Specialize_LoadMethod(owner, next_instr, name) < 0) { goto error; } - DISPATCH(); + NOTRACE_DISPATCH_SAME_OPARG(); } else { STAT_INC(LOAD_METHOD, deferred); @@ -4797,7 +4804,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int if (err < 0) { goto error; } - DISPATCH(); + NOTRACE_DISPATCH_SAME_OPARG(); } else { STAT_INC(PRECALL, deferred); @@ -4818,7 +4825,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int if (err < 0) { goto error; } - DISPATCH(); + NOTRACE_DISPATCH_SAME_OPARG(); } else { STAT_INC(CALL, deferred); @@ -5545,7 +5552,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int PyObject *rhs = TOP(); next_instr--; _Py_Specialize_BinaryOp(lhs, rhs, next_instr, oparg, &GETLOCAL(0)); - DISPATCH(); + NOTRACE_DISPATCH_SAME_OPARG(); } else { STAT_INC(BINARY_OP, deferred); @@ -5564,11 +5571,9 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int TARGET(EXTENDED_ARG) { assert(oparg); - int oldoparg = oparg; - NEXTOPARG(); - oparg |= oldoparg << 8; - PRE_DISPATCH_GOTO(); - DISPATCH_GOTO(); + oparg <<= 8; + oparg |= _Py_OPARG(*next_instr); + NOTRACE_DISPATCH_SAME_OPARG(); } TARGET(CACHE) { From webhook-mailer at python.org Sun Apr 17 16:53:42 2022 From: webhook-mailer at python.org (zware) Date: Sun, 17 Apr 2022 20:53:42 -0000 Subject: [Python-checkins] Use git attribute macros for CRLF and no-EOL files (GH-30762) Message-ID: https://github.com/python/cpython/commit/0b906ae562fa057162a67d4a5b7ea426f6b9dd23 commit: 0b906ae562fa057162a67d4a5b7ea426f6b9dd23 branch: main author: Erlend Egeberg Aasland committer: zware date: 2022-04-17T15:53:31-05:00 summary: Use git attribute macros for CRLF and no-EOL files (GH-30762) files: M .gitattributes diff --git a/.gitattributes b/.gitattributes index bc513e437e018..d956a45753c80 100644 --- a/.gitattributes +++ b/.gitattributes @@ -22,23 +22,27 @@ Lib/test/sndhdrdata/sndhdr.* binary PC/classicAppCompat.* binary # Text files that should not be subject to eol conversion -Lib/test/cjkencodings/* -text -Lib/test/decimaltestdata/*.decTest -text -Lib/test/test_email/data/*.txt -text -Lib/test/xmltestdata/* -text -Lib/test/coding20731.py -text -Lib/test/test_importlib/data01/* -text -Lib/test/test_importlib/namespacedata01/* -text +[attr]noeol -text + +Lib/test/cjkencodings/* noeol +Lib/test/coding20731.py noeol +Lib/test/decimaltestdata/*.decTest noeol +Lib/test/test_email/data/*.txt noeol +Lib/test/test_importlib/data01/* noeol +Lib/test/test_importlib/namespacedata01/* noeol +Lib/test/xmltestdata/* noeol # CRLF files -*.bat text eol=crlf -*.ps1 text eol=crlf -*.sln text eol=crlf -*.vcxproj* text eol=crlf -*.props text eol=crlf -*.proj text eol=crlf -PCbuild/readme.txt text eol=crlf -PC/readme.txt text eol=crlf +[attr]dos text eol=crlf + +*.bat dos +*.proj dos +*.props dos +*.ps1 dos +*.sln dos +*.vcxproj* dos +PC/readme.txt dos +PCbuild/readme.txt dos # Language aware diff headers # https://tekin.co.uk/2020/10/better-git-diff-output-for-ruby-python-elixir-and-more From webhook-mailer at python.org Sun Apr 17 17:13:03 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Sun, 17 Apr 2022 21:13:03 -0000 Subject: [Python-checkins] gh-90923: Improve sqlite3.Connection.execute* docs (#91643) Message-ID: https://github.com/python/cpython/commit/017f07a229a337e9c17bed8cd1879e0177a8d89d commit: 017f07a229a337e9c17bed8cd1879e0177a8d89d branch: main author: Erlend Egeberg Aasland committer: JelleZijlstra date: 2022-04-17T14:12:55-07:00 summary: gh-90923: Improve sqlite3.Connection.execute* docs (#91643) - Drop 'nonstandard'; it does not add any value - Try to be more concise - Make return value a little more explicit files: M Doc/library/sqlite3.rst diff --git a/Doc/library/sqlite3.rst b/Doc/library/sqlite3.rst index 5c9e2e868ab63..cbe7bb1fb9a0d 100644 --- a/Doc/library/sqlite3.rst +++ b/Doc/library/sqlite3.rst @@ -431,24 +431,21 @@ Connection Objects .. method:: execute(sql[, parameters]) - This is a nonstandard shortcut that creates a cursor object by calling - the :meth:`~Connection.cursor` method, calls the cursor's - :meth:`~Cursor.execute` method with the *parameters* given, and returns - the cursor. + Create a new :class:`Cursor` object and call + :meth:`~Cursor.execute` on it with the given *sql* and *parameters*. + Return the new cursor object. .. method:: executemany(sql[, parameters]) - This is a nonstandard shortcut that creates a cursor object by - calling the :meth:`~Connection.cursor` method, calls the cursor's - :meth:`~Cursor.executemany` method with the *parameters* given, and - returns the cursor. + Create a new :class:`Cursor` object and call + :meth:`~Cursor.executemany` on it with the given *sql* and *parameters*. + Return the new cursor object. .. method:: executescript(sql_script) - This is a nonstandard shortcut that creates a cursor object by - calling the :meth:`~Connection.cursor` method, calls the cursor's - :meth:`~Cursor.executescript` method with the given *sql_script*, and - returns the cursor. + Create a new :class:`Cursor` object and call + :meth:`~Cursor.executescript` on it with the given *sql_script*. + Return the new cursor object. .. method:: create_function(name, num_params, func, *, deterministic=False) From webhook-mailer at python.org Sun Apr 17 17:17:09 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Sun, 17 Apr 2022 21:17:09 -0000 Subject: [Python-checkins] gh-91541: Fix error in example in modules tutorial (#91634) Message-ID: https://github.com/python/cpython/commit/efbc668183400597070356a2df2fbab114a53cb3 commit: efbc668183400597070356a2df2fbab114a53cb3 branch: main author: 180909 <734461790 at qq.com> committer: JelleZijlstra date: 2022-04-17T14:17:04-07:00 summary: gh-91541: Fix error in example in modules tutorial (#91634) files: M Doc/tutorial/modules.rst diff --git a/Doc/tutorial/modules.rst b/Doc/tutorial/modules.rst index 0d1fadadf83a0..524d3c32417b9 100644 --- a/Doc/tutorial/modules.rst +++ b/Doc/tutorial/modules.rst @@ -508,7 +508,7 @@ code:: __all__ = ["echo", "surround", "reverse"] This would mean that ``from sound.effects import *`` would import the three -named submodules of the :mod:`sound` package. +named submodules of the :mod:`sound.effects` package. If ``__all__`` is not defined, the statement ``from sound.effects import *`` does *not* import all submodules from the package :mod:`sound.effects` into the From webhook-mailer at python.org Sun Apr 17 17:20:17 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Sun, 17 Apr 2022 21:20:17 -0000 Subject: [Python-checkins] gh-89885: Improve import example in language reference (#91523) Message-ID: https://github.com/python/cpython/commit/d5a69571f586080af4c29671c47f9c4bc671af7f commit: d5a69571f586080af4c29671c47f9c4bc671af7f branch: main author: slateny <46876382+slateny at users.noreply.github.com> committer: JelleZijlstra date: 2022-04-17T14:20:13-07:00 summary: gh-89885: Improve import example in language reference (#91523) Co-authored-by: Jelle Zijlstra files: M Doc/reference/import.rst diff --git a/Doc/reference/import.rst b/Doc/reference/import.rst index 66737c698ae90..988d41c81c6ab 100644 --- a/Doc/reference/import.rst +++ b/Doc/reference/import.rst @@ -490,21 +490,19 @@ submodule. Let's say you have the following directory structure:: spam/ __init__.py foo.py - bar.py -and ``spam/__init__.py`` has the following lines in it:: +and ``spam/__init__.py`` has the following line in it:: from .foo import Foo - from .bar import Bar -then executing the following puts a name binding to ``foo`` and ``bar`` in the +then executing the following puts name bindings for ``foo`` and ``Foo`` in the ``spam`` module:: >>> import spam >>> spam.foo - >>> spam.bar - + >>> spam.Foo + Given Python's familiar name binding rules this might seem surprising, but it's actually a fundamental feature of the import system. The invariant From webhook-mailer at python.org Sun Apr 17 17:29:33 2022 From: webhook-mailer at python.org (miss-islington) Date: Sun, 17 Apr 2022 21:29:33 -0000 Subject: [Python-checkins] gh-90923: Improve sqlite3.Connection.execute* docs (GH-91643) Message-ID: https://github.com/python/cpython/commit/2bcfe20d9271f077b954afb4c8b45024adc10ce5 commit: 2bcfe20d9271f077b954afb4c8b45024adc10ce5 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-17T14:29:21-07:00 summary: gh-90923: Improve sqlite3.Connection.execute* docs (GH-91643) - Drop 'nonstandard'; it does not add any value - Try to be more concise - Make return value a little more explicit (cherry picked from commit 017f07a229a337e9c17bed8cd1879e0177a8d89d) Co-authored-by: Erlend Egeberg Aasland files: M Doc/library/sqlite3.rst diff --git a/Doc/library/sqlite3.rst b/Doc/library/sqlite3.rst index e7c191b9f61e9..3fac2db174c33 100644 --- a/Doc/library/sqlite3.rst +++ b/Doc/library/sqlite3.rst @@ -378,24 +378,21 @@ Connection Objects .. method:: execute(sql[, parameters]) - This is a nonstandard shortcut that creates a cursor object by calling - the :meth:`~Connection.cursor` method, calls the cursor's - :meth:`~Cursor.execute` method with the *parameters* given, and returns - the cursor. + Create a new :class:`Cursor` object and call + :meth:`~Cursor.execute` on it with the given *sql* and *parameters*. + Return the new cursor object. .. method:: executemany(sql[, parameters]) - This is a nonstandard shortcut that creates a cursor object by - calling the :meth:`~Connection.cursor` method, calls the cursor's - :meth:`~Cursor.executemany` method with the *parameters* given, and - returns the cursor. + Create a new :class:`Cursor` object and call + :meth:`~Cursor.executemany` on it with the given *sql* and *parameters*. + Return the new cursor object. .. method:: executescript(sql_script) - This is a nonstandard shortcut that creates a cursor object by - calling the :meth:`~Connection.cursor` method, calls the cursor's - :meth:`~Cursor.executescript` method with the given *sql_script*, and - returns the cursor. + Create a new :class:`Cursor` object and call + :meth:`~Cursor.executescript` on it with the given *sql_script*. + Return the new cursor object. .. method:: create_function(name, num_params, func, *, deterministic=False) From webhook-mailer at python.org Sun Apr 17 17:31:08 2022 From: webhook-mailer at python.org (miss-islington) Date: Sun, 17 Apr 2022 21:31:08 -0000 Subject: [Python-checkins] gh-91541: Fix error in example in modules tutorial (GH-91634) Message-ID: https://github.com/python/cpython/commit/e5636f3a0eb86932175c26fe602e0f53b6d3cbb9 commit: e5636f3a0eb86932175c26fe602e0f53b6d3cbb9 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-17T14:30:57-07:00 summary: gh-91541: Fix error in example in modules tutorial (GH-91634) (cherry picked from commit efbc668183400597070356a2df2fbab114a53cb3) Co-authored-by: 180909 <734461790 at qq.com> files: M Doc/tutorial/modules.rst diff --git a/Doc/tutorial/modules.rst b/Doc/tutorial/modules.rst index f1d4957e37eb1..47c1f1877d696 100644 --- a/Doc/tutorial/modules.rst +++ b/Doc/tutorial/modules.rst @@ -504,7 +504,7 @@ code:: __all__ = ["echo", "surround", "reverse"] This would mean that ``from sound.effects import *`` would import the three -named submodules of the :mod:`sound` package. +named submodules of the :mod:`sound.effects` package. If ``__all__`` is not defined, the statement ``from sound.effects import *`` does *not* import all submodules from the package :mod:`sound.effects` into the From webhook-mailer at python.org Sun Apr 17 17:34:55 2022 From: webhook-mailer at python.org (miss-islington) Date: Sun, 17 Apr 2022 21:34:55 -0000 Subject: [Python-checkins] gh-89885: Improve import example in language reference (GH-91523) Message-ID: https://github.com/python/cpython/commit/531f66ad629b02d1fc655dc1eb8956ce245dacd6 commit: 531f66ad629b02d1fc655dc1eb8956ce245dacd6 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-17T14:34:41-07:00 summary: gh-89885: Improve import example in language reference (GH-91523) Co-authored-by: Jelle Zijlstra (cherry picked from commit d5a69571f586080af4c29671c47f9c4bc671af7f) Co-authored-by: slateny <46876382+slateny at users.noreply.github.com> files: M Doc/reference/import.rst diff --git a/Doc/reference/import.rst b/Doc/reference/import.rst index c01535d8049df..9c3c86649f555 100644 --- a/Doc/reference/import.rst +++ b/Doc/reference/import.rst @@ -490,21 +490,19 @@ submodule. Let's say you have the following directory structure:: spam/ __init__.py foo.py - bar.py -and ``spam/__init__.py`` has the following lines in it:: +and ``spam/__init__.py`` has the following line in it:: from .foo import Foo - from .bar import Bar -then executing the following puts a name binding to ``foo`` and ``bar`` in the +then executing the following puts name bindings for ``foo`` and ``Foo`` in the ``spam`` module:: >>> import spam >>> spam.foo - >>> spam.bar - + >>> spam.Foo + Given Python's familiar name binding rules this might seem surprising, but it's actually a fundamental feature of the import system. The invariant From webhook-mailer at python.org Sun Apr 17 17:36:36 2022 From: webhook-mailer at python.org (miss-islington) Date: Sun, 17 Apr 2022 21:36:36 -0000 Subject: [Python-checkins] gh-90923: Improve sqlite3.Connection.execute* docs (GH-91643) Message-ID: https://github.com/python/cpython/commit/833ff16430644a7dacafb061b8a5509818c8b0f7 commit: 833ff16430644a7dacafb061b8a5509818c8b0f7 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-17T14:36:32-07:00 summary: gh-90923: Improve sqlite3.Connection.execute* docs (GH-91643) - Drop 'nonstandard'; it does not add any value - Try to be more concise - Make return value a little more explicit (cherry picked from commit 017f07a229a337e9c17bed8cd1879e0177a8d89d) Co-authored-by: Erlend Egeberg Aasland files: M Doc/library/sqlite3.rst diff --git a/Doc/library/sqlite3.rst b/Doc/library/sqlite3.rst index 9e6950652ebc2..d4ba3e2627d8d 100644 --- a/Doc/library/sqlite3.rst +++ b/Doc/library/sqlite3.rst @@ -373,24 +373,21 @@ Connection Objects .. method:: execute(sql[, parameters]) - This is a nonstandard shortcut that creates a cursor object by calling - the :meth:`~Connection.cursor` method, calls the cursor's - :meth:`~Cursor.execute` method with the *parameters* given, and returns - the cursor. + Create a new :class:`Cursor` object and call + :meth:`~Cursor.execute` on it with the given *sql* and *parameters*. + Return the new cursor object. .. method:: executemany(sql[, parameters]) - This is a nonstandard shortcut that creates a cursor object by - calling the :meth:`~Connection.cursor` method, calls the cursor's - :meth:`~Cursor.executemany` method with the *parameters* given, and - returns the cursor. + Create a new :class:`Cursor` object and call + :meth:`~Cursor.executemany` on it with the given *sql* and *parameters*. + Return the new cursor object. .. method:: executescript(sql_script) - This is a nonstandard shortcut that creates a cursor object by - calling the :meth:`~Connection.cursor` method, calls the cursor's - :meth:`~Cursor.executescript` method with the given *sql_script*, and - returns the cursor. + Create a new :class:`Cursor` object and call + :meth:`~Cursor.executescript` on it with the given *sql_script*. + Return the new cursor object. .. method:: create_function(name, num_params, func, *, deterministic=False) From webhook-mailer at python.org Sun Apr 17 17:40:39 2022 From: webhook-mailer at python.org (miss-islington) Date: Sun, 17 Apr 2022 21:40:39 -0000 Subject: [Python-checkins] gh-91541: Fix error in example in modules tutorial (GH-91634) Message-ID: https://github.com/python/cpython/commit/c6c98234552ba0ddc696c17d27eb48cc89d20abf commit: c6c98234552ba0ddc696c17d27eb48cc89d20abf branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-17T14:40:14-07:00 summary: gh-91541: Fix error in example in modules tutorial (GH-91634) (cherry picked from commit efbc668183400597070356a2df2fbab114a53cb3) Co-authored-by: 180909 <734461790 at qq.com> files: M Doc/tutorial/modules.rst diff --git a/Doc/tutorial/modules.rst b/Doc/tutorial/modules.rst index f6250291c8824..afce7f81d226f 100644 --- a/Doc/tutorial/modules.rst +++ b/Doc/tutorial/modules.rst @@ -504,7 +504,7 @@ code:: __all__ = ["echo", "surround", "reverse"] This would mean that ``from sound.effects import *`` would import the three -named submodules of the :mod:`sound` package. +named submodules of the :mod:`sound.effects` package. If ``__all__`` is not defined, the statement ``from sound.effects import *`` does *not* import all submodules from the package :mod:`sound.effects` into the From webhook-mailer at python.org Sun Apr 17 17:51:26 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Sun, 17 Apr 2022 21:51:26 -0000 Subject: [Python-checkins] gh-89885: Improve import example in language reference (GH-91523) (#91649) Message-ID: https://github.com/python/cpython/commit/3537d897e069be9c7bde9a50b205f275791ffdbe commit: 3537d897e069be9c7bde9a50b205f275791ffdbe branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: JelleZijlstra date: 2022-04-17T14:51:19-07:00 summary: gh-89885: Improve import example in language reference (GH-91523) (#91649) Co-authored-by: Jelle Zijlstra (cherry picked from commit d5a69571f586080af4c29671c47f9c4bc671af7f) Co-authored-by: slateny <46876382+slateny at users.noreply.github.com> files: M Doc/reference/import.rst diff --git a/Doc/reference/import.rst b/Doc/reference/import.rst index 2c84a97a26775..2d9802b0f3f65 100644 --- a/Doc/reference/import.rst +++ b/Doc/reference/import.rst @@ -483,21 +483,19 @@ submodule. Let's say you have the following directory structure:: spam/ __init__.py foo.py - bar.py -and ``spam/__init__.py`` has the following lines in it:: +and ``spam/__init__.py`` has the following line in it:: from .foo import Foo - from .bar import Bar -then executing the following puts a name binding to ``foo`` and ``bar`` in the +then executing the following puts name bindings for ``foo`` and ``Foo`` in the ``spam`` module:: >>> import spam >>> spam.foo - >>> spam.bar - + >>> spam.Foo + Given Python's familiar name binding rules this might seem surprising, but it's actually a fundamental feature of the import system. The invariant From webhook-mailer at python.org Sun Apr 17 18:02:44 2022 From: webhook-mailer at python.org (miss-islington) Date: Sun, 17 Apr 2022 22:02:44 -0000 Subject: [Python-checkins] gh-91217: deprecate ossaudiodev (GH-91641) Message-ID: https://github.com/python/cpython/commit/ceea0715df33ae6dc3931670b4b749432d4e9707 commit: ceea0715df33ae6dc3931670b4b749432d4e9707 branch: main author: Brett Cannon committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-17T15:02:36-07:00 summary: gh-91217: deprecate ossaudiodev (GH-91641) Automerge-Triggered-By: GH:brettcannon files: A Misc/NEWS.d/next/Library/2022-04-17-11-56-17.gh-issue-91217.McJre3.rst M Doc/whatsnew/3.11.rst M Lib/test/test_ossaudiodev.py M Modules/ossaudiodev.c diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index bedf80a72fab5..27a5a0fc3d687 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -884,6 +884,7 @@ Deprecated * :mod:`msilib` * :mod:`nis` * :mod:`nntplib` + * :mod:`ossaudiodev` (Contributed by Brett Cannon in :issue:`47061`.) diff --git a/Lib/test/test_ossaudiodev.py b/Lib/test/test_ossaudiodev.py index 37d2d1f5ff441..3275333545882 100644 --- a/Lib/test/test_ossaudiodev.py +++ b/Lib/test/test_ossaudiodev.py @@ -1,10 +1,13 @@ from test import support from test.support import import_helper, warnings_helper +import warnings support.requires('audio') from test.support import findfile -ossaudiodev = import_helper.import_module('ossaudiodev') +with warnings.catch_warnings(): + warnings.simplefilter("ignore", DeprecationWarning) + ossaudiodev = import_helper.import_module('ossaudiodev') audioop = warnings_helper.import_deprecated('audioop') import errno diff --git a/Misc/NEWS.d/next/Library/2022-04-17-11-56-17.gh-issue-91217.McJre3.rst b/Misc/NEWS.d/next/Library/2022-04-17-11-56-17.gh-issue-91217.McJre3.rst new file mode 100644 index 0000000000000..86ad05f963cb9 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-17-11-56-17.gh-issue-91217.McJre3.rst @@ -0,0 +1 @@ +Deprecate the ossaudiodev module. diff --git a/Modules/ossaudiodev.c b/Modules/ossaudiodev.c index 172a6e4297531..3926831e0c0d9 100644 --- a/Modules/ossaudiodev.c +++ b/Modules/ossaudiodev.c @@ -1117,6 +1117,13 @@ PyInit_ossaudiodev(void) { PyObject *m; + if (PyErr_WarnEx(PyExc_DeprecationWarning, + "'ossaudiodev' is deprecated and slated for removal in " + "Python 3.13", + 7)) { + return NULL; + } + if (PyType_Ready(&OSSAudioType) < 0) return NULL; From webhook-mailer at python.org Sun Apr 17 18:38:54 2022 From: webhook-mailer at python.org (warsaw) Date: Sun, 17 Apr 2022 22:38:54 -0000 Subject: [Python-checkins] Remove the ancient Pynche color editor (#91554) Message-ID: https://github.com/python/cpython/commit/7173fd5de0cb259b488828634745c03dbea94e6b commit: 7173fd5de0cb259b488828634745c03dbea94e6b branch: main author: Barry Warsaw committer: warsaw date: 2022-04-17T15:38:44-07:00 summary: Remove the ancient Pynche color editor (#91554) Closes #91551 files: A Misc/NEWS.d/next/Tools-Demos/2022-04-14-18-11-46.gh-issue-91551.l_nNT-.rst D Tools/pynche/ChipViewer.py D Tools/pynche/ColorDB.py D Tools/pynche/DetailsViewer.py D Tools/pynche/ListViewer.py D Tools/pynche/Main.py D Tools/pynche/PyncheWidget.py D Tools/pynche/README D Tools/pynche/StripViewer.py D Tools/pynche/Switchboard.py D Tools/pynche/TextViewer.py D Tools/pynche/TypeinViewer.py D Tools/pynche/X/rgb.txt D Tools/pynche/X/xlicense.txt D Tools/pynche/__init__.py D Tools/pynche/html40colors.txt D Tools/pynche/namedcolors.txt D Tools/pynche/pyColorChooser.py D Tools/pynche/pynche D Tools/pynche/pynche.pyw D Tools/pynche/webcolors.txt D Tools/pynche/websafe.txt M PC/layout/main.py M Tools/README M Tools/msi/tools/tools.wixproj diff --git a/Misc/NEWS.d/next/Tools-Demos/2022-04-14-18-11-46.gh-issue-91551.l_nNT-.rst b/Misc/NEWS.d/next/Tools-Demos/2022-04-14-18-11-46.gh-issue-91551.l_nNT-.rst new file mode 100644 index 0000000000000..f0c1866ac0e3c --- /dev/null +++ b/Misc/NEWS.d/next/Tools-Demos/2022-04-14-18-11-46.gh-issue-91551.l_nNT-.rst @@ -0,0 +1 @@ +Remove the ancient Pynche color editor. diff --git a/PC/layout/main.py b/PC/layout/main.py index fb6f5265859ff..8e69ecc259121 100644 --- a/PC/layout/main.py +++ b/PC/layout/main.py @@ -61,7 +61,7 @@ DATA_DIRS = FileNameSet("data") -TOOLS_DIRS = FileNameSet("scripts", "i18n", "pynche", "demo", "parser") +TOOLS_DIRS = FileNameSet("scripts", "i18n", "demo", "parser") TOOLS_FILES = FileSuffixSet(".py", ".pyw", ".txt") diff --git a/Tools/README b/Tools/README index 1f9d927fb613b..862bba7edb825 100644 --- a/Tools/README +++ b/Tools/README @@ -25,8 +25,6 @@ parser Un-parsing tool to generate code from an AST. peg_generator PEG-based parser generator (pegen) used for new parser. -pynche A Tkinter-based color editor. - scripts A number of useful single-file programs, e.g. tabnanny.py by Tim Peters, which checks for inconsistent mixing of tabs and spaces, and 2to3, which converts Python 2 code diff --git a/Tools/msi/tools/tools.wixproj b/Tools/msi/tools/tools.wixproj index b7fc41ee92e6d..2963048ddb49b 100644 --- a/Tools/msi/tools/tools.wixproj +++ b/Tools/msi/tools/tools.wixproj @@ -14,9 +14,6 @@ - - - - \ No newline at end of file + diff --git a/Tools/pynche/ChipViewer.py b/Tools/pynche/ChipViewer.py deleted file mode 100644 index 78139f8f2da95..0000000000000 --- a/Tools/pynche/ChipViewer.py +++ /dev/null @@ -1,130 +0,0 @@ -"""Chip viewer and widget. - -In the lower left corner of the main Pynche window, you will see two -ChipWidgets, one for the selected color and one for the nearest color. The -selected color is the actual RGB value expressed as an X11 #COLOR name. The -nearest color is the named color from the X11 database that is closest to the -selected color in 3D space. There may be other colors equally close, but the -nearest one is the first one found. - -Clicking on the nearest color chip selects that named color. - -The ChipViewer class includes the entire lower left quandrant; i.e. both the -selected and nearest ChipWidgets. -""" - -from tkinter import * -import ColorDB - - -class ChipWidget: - _WIDTH = 150 - _HEIGHT = 80 - - def __init__(self, - master = None, - width = _WIDTH, - height = _HEIGHT, - text = 'Color', - initialcolor = 'blue', - presscmd = None, - releasecmd = None): - # create the text label - self.__label = Label(master, text=text) - self.__label.grid(row=0, column=0) - # create the color chip, implemented as a frame - self.__chip = Frame(master, relief=RAISED, borderwidth=2, - width=width, - height=height, - background=initialcolor) - self.__chip.grid(row=1, column=0) - # create the color name - self.__namevar = StringVar() - self.__namevar.set(initialcolor) - self.__name = Entry(master, textvariable=self.__namevar, - relief=FLAT, justify=CENTER, state=DISABLED, - font=self.__label['font']) - self.__name.grid(row=2, column=0) - # create the message area - self.__msgvar = StringVar() - self.__name = Entry(master, textvariable=self.__msgvar, - relief=FLAT, justify=CENTER, state=DISABLED, - font=self.__label['font']) - self.__name.grid(row=3, column=0) - # set bindings - if presscmd: - self.__chip.bind('', presscmd) - if releasecmd: - self.__chip.bind('', releasecmd) - - def set_color(self, color): - self.__chip.config(background=color) - - def get_color(self): - return self.__chip['background'] - - def set_name(self, colorname): - self.__namevar.set(colorname) - - def set_message(self, message): - self.__msgvar.set(message) - - def press(self): - self.__chip.configure(relief=SUNKEN) - - def release(self): - self.__chip.configure(relief=RAISED) - - - -class ChipViewer: - def __init__(self, switchboard, master=None): - self.__sb = switchboard - self.__frame = Frame(master, relief=RAISED, borderwidth=1) - self.__frame.grid(row=3, column=0, ipadx=5, sticky='NSEW') - # create the chip that will display the currently selected color - # exactly - self.__sframe = Frame(self.__frame) - self.__sframe.grid(row=0, column=0) - self.__selected = ChipWidget(self.__sframe, text='Selected') - # create the chip that will display the nearest real X11 color - # database color name - self.__nframe = Frame(self.__frame) - self.__nframe.grid(row=0, column=1) - self.__nearest = ChipWidget(self.__nframe, text='Nearest', - presscmd = self.__buttonpress, - releasecmd = self.__buttonrelease) - - def update_yourself(self, red, green, blue): - # Selected always shows the #rrggbb name of the color, nearest always - # shows the name of the nearest color in the database. BAW: should - # an exact match be indicated in some way? - # - # Always use the #rrggbb style to actually set the color, since we may - # not be using X color names (e.g. "web-safe" names) - colordb = self.__sb.colordb() - rgbtuple = (red, green, blue) - rrggbb = ColorDB.triplet_to_rrggbb(rgbtuple) - # find the nearest - nearest = colordb.nearest(red, green, blue) - nearest_tuple = colordb.find_byname(nearest) - nearest_rrggbb = ColorDB.triplet_to_rrggbb(nearest_tuple) - self.__selected.set_color(rrggbb) - self.__nearest.set_color(nearest_rrggbb) - # set the name and messages areas - self.__selected.set_name(rrggbb) - if rrggbb == nearest_rrggbb: - self.__selected.set_message(nearest) - else: - self.__selected.set_message('') - self.__nearest.set_name(nearest_rrggbb) - self.__nearest.set_message(nearest) - - def __buttonpress(self, event=None): - self.__nearest.press() - - def __buttonrelease(self, event=None): - self.__nearest.release() - rrggbb = self.__nearest.get_color() - red, green, blue = ColorDB.rrggbb_to_triplet(rrggbb) - self.__sb.update_views(red, green, blue) diff --git a/Tools/pynche/ColorDB.py b/Tools/pynche/ColorDB.py deleted file mode 100644 index c013a60896908..0000000000000 --- a/Tools/pynche/ColorDB.py +++ /dev/null @@ -1,271 +0,0 @@ -"""Color Database. - -This file contains one class, called ColorDB, and several utility functions. -The class must be instantiated by the get_colordb() function in this file, -passing it a filename to read a database out of. - -The get_colordb() function will try to examine the file to figure out what the -format of the file is. If it can't figure out the file format, or it has -trouble reading the file, None is returned. You can pass get_colordb() an -optional filetype argument. - -Supported file types are: - - X_RGB_TXT -- X Consortium rgb.txt format files. Three columns of numbers - from 0 .. 255 separated by whitespace. Arbitrary trailing - columns used as the color name. - -The utility functions are useful for converting between the various expected -color formats, and for calculating other color values. - -""" - -import sys -import re -from types import * - -class BadColor(Exception): - pass - -DEFAULT_DB = None -SPACE = ' ' -COMMASPACE = ', ' - - - -# generic class -class ColorDB: - def __init__(self, fp): - lineno = 2 - self.__name = fp.name - # Maintain several dictionaries for indexing into the color database. - # Note that while Tk supports RGB intensities of 4, 8, 12, or 16 bits, - # for now we only support 8 bit intensities. At least on OpenWindows, - # all intensities in the /usr/openwin/lib/rgb.txt file are 8-bit - # - # key is (red, green, blue) tuple, value is (name, [aliases]) - self.__byrgb = {} - # key is name, value is (red, green, blue) - self.__byname = {} - # all unique names (non-aliases). built-on demand - self.__allnames = None - for line in fp: - # get this compiled regular expression from derived class - mo = self._re.match(line) - if not mo: - print('Error in', fp.name, ' line', lineno, file=sys.stderr) - lineno += 1 - continue - # extract the red, green, blue, and name - red, green, blue = self._extractrgb(mo) - name = self._extractname(mo) - keyname = name.lower() - # BAW: for now the `name' is just the first named color with the - # rgb values we find. Later, we might want to make the two word - # version the `name', or the CapitalizedVersion, etc. - key = (red, green, blue) - foundname, aliases = self.__byrgb.get(key, (name, [])) - if foundname != name and foundname not in aliases: - aliases.append(name) - self.__byrgb[key] = (foundname, aliases) - # add to byname lookup - self.__byname[keyname] = key - lineno = lineno + 1 - - # override in derived classes - def _extractrgb(self, mo): - return [int(x) for x in mo.group('red', 'green', 'blue')] - - def _extractname(self, mo): - return mo.group('name') - - def filename(self): - return self.__name - - def find_byrgb(self, rgbtuple): - """Return name for rgbtuple""" - try: - return self.__byrgb[rgbtuple] - except KeyError: - raise BadColor(rgbtuple) from None - - def find_byname(self, name): - """Return (red, green, blue) for name""" - name = name.lower() - try: - return self.__byname[name] - except KeyError: - raise BadColor(name) from None - - def nearest(self, red, green, blue): - """Return the name of color nearest (red, green, blue)""" - # BAW: should we use Voronoi diagrams, Delaunay triangulation, or - # octree for speeding up the locating of nearest point? Exhaustive - # search is inefficient, but seems fast enough. - nearest = -1 - nearest_name = '' - for name, aliases in self.__byrgb.values(): - r, g, b = self.__byname[name.lower()] - rdelta = red - r - gdelta = green - g - bdelta = blue - b - distance = rdelta * rdelta + gdelta * gdelta + bdelta * bdelta - if nearest == -1 or distance < nearest: - nearest = distance - nearest_name = name - return nearest_name - - def unique_names(self): - # sorted - if not self.__allnames: - self.__allnames = [] - for name, aliases in self.__byrgb.values(): - self.__allnames.append(name) - self.__allnames.sort(key=str.lower) - return self.__allnames - - def aliases_of(self, red, green, blue): - try: - name, aliases = self.__byrgb[(red, green, blue)] - except KeyError: - raise BadColor((red, green, blue)) from None - return [name] + aliases - - -class RGBColorDB(ColorDB): - _re = re.compile( - r'\s*(?P\d+)\s+(?P\d+)\s+(?P\d+)\s+(?P.*)') - - -class HTML40DB(ColorDB): - _re = re.compile(r'(?P\S+)\s+(?P#[0-9a-fA-F]{6})') - - def _extractrgb(self, mo): - return rrggbb_to_triplet(mo.group('hexrgb')) - -class LightlinkDB(HTML40DB): - _re = re.compile(r'(?P(.+))\s+(?P#[0-9a-fA-F]{6})') - - def _extractname(self, mo): - return mo.group('name').strip() - -class WebsafeDB(ColorDB): - _re = re.compile('(?P#[0-9a-fA-F]{6})') - - def _extractrgb(self, mo): - return rrggbb_to_triplet(mo.group('hexrgb')) - - def _extractname(self, mo): - return mo.group('hexrgb').upper() - - - -# format is a tuple (RE, SCANLINES, CLASS) where RE is a compiled regular -# expression, SCANLINES is the number of header lines to scan, and CLASS is -# the class to instantiate if a match is found - -FILETYPES = [ - (re.compile('Xorg'), RGBColorDB), - (re.compile('XConsortium'), RGBColorDB), - (re.compile('HTML'), HTML40DB), - (re.compile('lightlink'), LightlinkDB), - (re.compile('Websafe'), WebsafeDB), - ] - -def get_colordb(file, filetype=None): - colordb = None - fp = open(file) - try: - line = fp.readline() - if not line: - return None - # try to determine the type of RGB file it is - if filetype is None: - filetypes = FILETYPES - else: - filetypes = [filetype] - for typere, class_ in filetypes: - mo = typere.search(line) - if mo: - break - else: - # no matching type - return None - # we know the type and the class to grok the type, so suck it in - colordb = class_(fp) - finally: - fp.close() - # save a global copy - global DEFAULT_DB - DEFAULT_DB = colordb - return colordb - - - -_namedict = {} - -def rrggbb_to_triplet(color): - """Converts a #rrggbb color to the tuple (red, green, blue).""" - rgbtuple = _namedict.get(color) - if rgbtuple is None: - if color[0] != '#': - raise BadColor(color) - red = color[1:3] - green = color[3:5] - blue = color[5:7] - rgbtuple = int(red, 16), int(green, 16), int(blue, 16) - _namedict[color] = rgbtuple - return rgbtuple - - -_tripdict = {} -def triplet_to_rrggbb(rgbtuple): - """Converts a (red, green, blue) tuple to #rrggbb.""" - global _tripdict - hexname = _tripdict.get(rgbtuple) - if hexname is None: - hexname = '#%02x%02x%02x' % rgbtuple - _tripdict[rgbtuple] = hexname - return hexname - - -def triplet_to_fractional_rgb(rgbtuple): - return [x / 256 for x in rgbtuple] - - -def triplet_to_brightness(rgbtuple): - # return the brightness (grey level) along the scale 0.0==black to - # 1.0==white - r = 0.299 - g = 0.587 - b = 0.114 - return r*rgbtuple[0] + g*rgbtuple[1] + b*rgbtuple[2] - - - -if __name__ == '__main__': - colordb = get_colordb('/usr/openwin/lib/rgb.txt') - if not colordb: - print('No parseable color database found') - sys.exit(1) - # on my system, this color matches exactly - target = 'navy' - red, green, blue = rgbtuple = colordb.find_byname(target) - print(target, ':', red, green, blue, triplet_to_rrggbb(rgbtuple)) - name, aliases = colordb.find_byrgb(rgbtuple) - print('name:', name, 'aliases:', COMMASPACE.join(aliases)) - r, g, b = (1, 1, 128) # nearest to navy - r, g, b = (145, 238, 144) # nearest to lightgreen - r, g, b = (255, 251, 250) # snow - print('finding nearest to', target, '...') - import time - t0 = time.time() - nearest = colordb.nearest(r, g, b) - t1 = time.time() - print('found nearest color', nearest, 'in', t1-t0, 'seconds') - # dump the database - for n in colordb.unique_names(): - r, g, b = colordb.find_byname(n) - aliases = colordb.aliases_of(r, g, b) - print('%20s: (%3d/%3d/%3d) == %s' % (n, r, g, b, - SPACE.join(aliases[1:]))) diff --git a/Tools/pynche/DetailsViewer.py b/Tools/pynche/DetailsViewer.py deleted file mode 100644 index bed11f4f4efdf..0000000000000 --- a/Tools/pynche/DetailsViewer.py +++ /dev/null @@ -1,273 +0,0 @@ -"""DetailsViewer class. - -This class implements a pure input window which allows you to meticulously -edit the current color. You have both mouse control of the color (via the -buttons along the bottom row), and there are keyboard bindings for each of the -increment/decrement buttons. - -The top three check buttons allow you to specify which of the three color -variations are tied together when incrementing and decrementing. Red, green, -and blue are self evident. By tying together red and green, you can modify -the yellow level of the color. By tying together red and blue, you can modify -the magenta level of the color. By tying together green and blue, you can -modify the cyan level, and by tying all three together, you can modify the -grey level. - -The behavior at the boundaries (0 and 255) are defined by the `At boundary' -option menu: - - Stop - When the increment or decrement would send any of the tied variations - out of bounds, the entire delta is discarded. - - Wrap Around - When the increment or decrement would send any of the tied variations - out of bounds, the out of bounds variation is wrapped around to the - other side. Thus if red were at 238 and 25 were added to it, red - would have the value 7. - - Preserve Distance - When the increment or decrement would send any of the tied variations - out of bounds, all tied variations are wrapped as one, so as to - preserve the distance between them. Thus if green and blue were tied, - and green was at 238 while blue was at 223, and an increment of 25 - were applied, green would be at 15 and blue would be at 0. - - Squash - When the increment or decrement would send any of the tied variations - out of bounds, the out of bounds variation is set to the ceiling of - 255 or floor of 0, as appropriate. In this way, all tied variations - are squashed to one edge or the other. - -The following key bindings can be used as accelerators. Note that Pynche can -fall behind if you hold the key down as a key repeat: - -Left arrow == -1 -Right arrow == +1 - -Control + Left == -10 -Control + Right == 10 - -Shift + Left == -25 -Shift + Right == +25 -""" - -from tkinter import * - -STOP = 'Stop' -WRAP = 'Wrap Around' -RATIO = 'Preserve Distance' -GRAV = 'Squash' - -ADDTOVIEW = 'Details Window...' - - -class DetailsViewer: - def __init__(self, switchboard, master=None): - self.__sb = switchboard - optiondb = switchboard.optiondb() - self.__red, self.__green, self.__blue = switchboard.current_rgb() - # GUI - root = self.__root = Toplevel(master, class_='Pynche') - root.protocol('WM_DELETE_WINDOW', self.withdraw) - root.title('Pynche Details Window') - root.iconname('Pynche Details Window') - root.bind('', self.__quit) - root.bind('', self.__quit) - root.bind('', self.withdraw) - root.bind('', self.withdraw) - # accelerators - root.bind('', self.__minus1) - root.bind('', self.__plus1) - root.bind('', self.__minus10) - root.bind('', self.__plus10) - root.bind('', self.__minus25) - root.bind('', self.__plus25) - # - # color ties - frame = self.__frame = Frame(root) - frame.pack(expand=YES, fill=X) - self.__l1 = Label(frame, text='Move Sliders:') - self.__l1.grid(row=1, column=0, sticky=E) - self.__rvar = IntVar() - self.__rvar.set(optiondb.get('RSLIDER', 4)) - self.__radio1 = Checkbutton(frame, text='Red', - variable=self.__rvar, - command=self.__effect, - onvalue=4, offvalue=0) - self.__radio1.grid(row=1, column=1, sticky=W) - self.__gvar = IntVar() - self.__gvar.set(optiondb.get('GSLIDER', 2)) - self.__radio2 = Checkbutton(frame, text='Green', - variable=self.__gvar, - command=self.__effect, - onvalue=2, offvalue=0) - self.__radio2.grid(row=2, column=1, sticky=W) - self.__bvar = IntVar() - self.__bvar.set(optiondb.get('BSLIDER', 1)) - self.__radio3 = Checkbutton(frame, text='Blue', - variable=self.__bvar, - command=self.__effect, - onvalue=1, offvalue=0) - self.__radio3.grid(row=3, column=1, sticky=W) - self.__l2 = Label(frame) - self.__l2.grid(row=4, column=1, sticky=W) - self.__effect() - # - # Boundary behavior - self.__l3 = Label(frame, text='At boundary:') - self.__l3.grid(row=5, column=0, sticky=E) - self.__boundvar = StringVar() - self.__boundvar.set(optiondb.get('ATBOUND', STOP)) - self.__omenu = OptionMenu(frame, self.__boundvar, - STOP, WRAP, RATIO, GRAV) - self.__omenu.grid(row=5, column=1, sticky=W) - self.__omenu.configure(width=17) - # - # Buttons - frame = self.__btnframe = Frame(frame) - frame.grid(row=0, column=0, columnspan=2, sticky='EW') - self.__down25 = Button(frame, text='-25', - command=self.__minus25) - self.__down10 = Button(frame, text='-10', - command=self.__minus10) - self.__down1 = Button(frame, text='-1', - command=self.__minus1) - self.__up1 = Button(frame, text='+1', - command=self.__plus1) - self.__up10 = Button(frame, text='+10', - command=self.__plus10) - self.__up25 = Button(frame, text='+25', - command=self.__plus25) - self.__down25.pack(expand=YES, fill=X, side=LEFT) - self.__down10.pack(expand=YES, fill=X, side=LEFT) - self.__down1.pack(expand=YES, fill=X, side=LEFT) - self.__up1.pack(expand=YES, fill=X, side=LEFT) - self.__up10.pack(expand=YES, fill=X, side=LEFT) - self.__up25.pack(expand=YES, fill=X, side=LEFT) - - def __effect(self, event=None): - tie = self.__rvar.get() + self.__gvar.get() + self.__bvar.get() - if tie in (0, 1, 2, 4): - text = '' - else: - text = '(= %s Level)' % {3: 'Cyan', - 5: 'Magenta', - 6: 'Yellow', - 7: 'Grey'}[tie] - self.__l2.configure(text=text) - - def __quit(self, event=None): - self.__root.quit() - - def withdraw(self, event=None): - self.__root.withdraw() - - def deiconify(self, event=None): - self.__root.deiconify() - - def __minus25(self, event=None): - self.__delta(-25) - - def __minus10(self, event=None): - self.__delta(-10) - - def __minus1(self, event=None): - self.__delta(-1) - - def __plus1(self, event=None): - self.__delta(1) - - def __plus10(self, event=None): - self.__delta(10) - - def __plus25(self, event=None): - self.__delta(25) - - def __delta(self, delta): - tie = [] - if self.__rvar.get(): - red = self.__red + delta - tie.append(red) - else: - red = self.__red - if self.__gvar.get(): - green = self.__green + delta - tie.append(green) - else: - green = self.__green - if self.__bvar.get(): - blue = self.__blue + delta - tie.append(blue) - else: - blue = self.__blue - # now apply at boundary behavior - atbound = self.__boundvar.get() - if atbound == STOP: - if red < 0 or green < 0 or blue < 0 or \ - red > 255 or green > 255 or blue > 255: - # then - red, green, blue = self.__red, self.__green, self.__blue - elif atbound == WRAP or (atbound == RATIO and len(tie) < 2): - if red < 0: - red += 256 - if green < 0: - green += 256 - if blue < 0: - blue += 256 - if red > 255: - red -= 256 - if green > 255: - green -= 256 - if blue > 255: - blue -= 256 - elif atbound == RATIO: - # for when 2 or 3 colors are tied together - dir = 0 - for c in tie: - if c < 0: - dir = -1 - elif c > 255: - dir = 1 - if dir == -1: - delta = max(tie) - if self.__rvar.get(): - red = red + 255 - delta - if self.__gvar.get(): - green = green + 255 - delta - if self.__bvar.get(): - blue = blue + 255 - delta - elif dir == 1: - delta = min(tie) - if self.__rvar.get(): - red = red - delta - if self.__gvar.get(): - green = green - delta - if self.__bvar.get(): - blue = blue - delta - elif atbound == GRAV: - if red < 0: - red = 0 - if green < 0: - green = 0 - if blue < 0: - blue = 0 - if red > 255: - red = 255 - if green > 255: - green = 255 - if blue > 255: - blue = 255 - self.__sb.update_views(red, green, blue) - self.__root.update_idletasks() - - def update_yourself(self, red, green, blue): - self.__red = red - self.__green = green - self.__blue = blue - - def save_options(self, optiondb): - optiondb['RSLIDER'] = self.__rvar.get() - optiondb['GSLIDER'] = self.__gvar.get() - optiondb['BSLIDER'] = self.__bvar.get() - optiondb['ATBOUND'] = self.__boundvar.get() diff --git a/Tools/pynche/ListViewer.py b/Tools/pynche/ListViewer.py deleted file mode 100644 index b18784453c3e0..0000000000000 --- a/Tools/pynche/ListViewer.py +++ /dev/null @@ -1,175 +0,0 @@ -"""ListViewer class. - -This class implements an input/output view on the color model. It lists every -unique color (e.g. unique r/g/b value) found in the color database. Each -color is shown by small swatch and primary color name. Some colors have -aliases -- more than one name for the same r/g/b value. These aliases are -displayed in the small listbox at the bottom of the screen. - -Clicking on a color name or swatch selects that color and updates all other -windows. When a color is selected in a different viewer, the color list is -scrolled to the selected color and it is highlighted. If the selected color -is an r/g/b value without a name, no scrolling occurs. - -You can turn off Update On Click if all you want to see is the alias for a -given name, without selecting the color. -""" - -from tkinter import * -import ColorDB - -ADDTOVIEW = 'Color %List Window...' - -class ListViewer: - def __init__(self, switchboard, master=None): - self.__sb = switchboard - optiondb = switchboard.optiondb() - self.__lastbox = None - self.__dontcenter = 0 - # GUI - root = self.__root = Toplevel(master, class_='Pynche') - root.protocol('WM_DELETE_WINDOW', self.withdraw) - root.title('Pynche Color List') - root.iconname('Pynche Color List') - root.bind('', self.__quit) - root.bind('', self.__quit) - root.bind('', self.withdraw) - root.bind('', self.withdraw) - # - # create the canvas which holds everything, and its scrollbar - # - frame = self.__frame = Frame(root) - frame.pack() - canvas = self.__canvas = Canvas(frame, width=160, height=300, - borderwidth=2, relief=SUNKEN) - self.__scrollbar = Scrollbar(frame) - self.__scrollbar.pack(fill=Y, side=RIGHT) - canvas.pack(fill=BOTH, expand=1) - canvas.configure(yscrollcommand=(self.__scrollbar, 'set')) - self.__scrollbar.configure(command=(canvas, 'yview')) - self.__populate() - # - # Update on click - self.__uoc = BooleanVar() - self.__uoc.set(optiondb.get('UPONCLICK', 1)) - self.__uocbtn = Checkbutton(root, - text='Update on Click', - variable=self.__uoc, - command=self.__toggleupdate) - self.__uocbtn.pack(expand=1, fill=BOTH) - # - # alias list - self.__alabel = Label(root, text='Aliases:') - self.__alabel.pack() - self.__aliases = Listbox(root, height=5, - selectmode=BROWSE) - self.__aliases.pack(expand=1, fill=BOTH) - - def __populate(self): - # - # create all the buttons - colordb = self.__sb.colordb() - canvas = self.__canvas - row = 0 - widest = 0 - bboxes = self.__bboxes = [] - for name in colordb.unique_names(): - exactcolor = ColorDB.triplet_to_rrggbb(colordb.find_byname(name)) - canvas.create_rectangle(5, row*20 + 5, - 20, row*20 + 20, - fill=exactcolor) - textid = canvas.create_text(25, row*20 + 13, - text=name, - anchor=W) - x1, y1, textend, y2 = canvas.bbox(textid) - boxid = canvas.create_rectangle(3, row*20+3, - textend+3, row*20 + 23, - outline='', - tags=(exactcolor, 'all')) - canvas.bind('', self.__onrelease) - bboxes.append(boxid) - if textend+3 > widest: - widest = textend+3 - row += 1 - canvheight = (row-1)*20 + 25 - canvas.config(scrollregion=(0, 0, 150, canvheight)) - for box in bboxes: - x1, y1, x2, y2 = canvas.coords(box) - canvas.coords(box, x1, y1, widest, y2) - - def __onrelease(self, event=None): - canvas = self.__canvas - # find the current box - x = canvas.canvasx(event.x) - y = canvas.canvasy(event.y) - ids = canvas.find_overlapping(x, y, x, y) - for boxid in ids: - if boxid in self.__bboxes: - break - else: -## print 'No box found!' - return - tags = self.__canvas.gettags(boxid) - for t in tags: - if t[0] == '#': - break - else: -## print 'No color tag found!' - return - red, green, blue = ColorDB.rrggbb_to_triplet(t) - self.__dontcenter = 1 - if self.__uoc.get(): - self.__sb.update_views(red, green, blue) - else: - self.update_yourself(red, green, blue) - self.__red, self.__green, self.__blue = red, green, blue - - def __toggleupdate(self, event=None): - if self.__uoc.get(): - self.__sb.update_views(self.__red, self.__green, self.__blue) - - def __quit(self, event=None): - self.__root.quit() - - def withdraw(self, event=None): - self.__root.withdraw() - - def deiconify(self, event=None): - self.__root.deiconify() - - def update_yourself(self, red, green, blue): - canvas = self.__canvas - # turn off the last box - if self.__lastbox: - canvas.itemconfigure(self.__lastbox, outline='') - # turn on the current box - colortag = ColorDB.triplet_to_rrggbb((red, green, blue)) - canvas.itemconfigure(colortag, outline='black') - self.__lastbox = colortag - # fill the aliases - self.__aliases.delete(0, END) - try: - aliases = self.__sb.colordb().aliases_of(red, green, blue)[1:] - except ColorDB.BadColor: - self.__aliases.insert(END, '') - return - if not aliases: - self.__aliases.insert(END, '') - else: - for name in aliases: - self.__aliases.insert(END, name) - # maybe scroll the canvas so that the item is visible - if self.__dontcenter: - self.__dontcenter = 0 - else: - ig, ig, ig, y1 = canvas.coords(colortag) - ig, ig, ig, y2 = canvas.coords(self.__bboxes[-1]) - h = int(canvas['height']) * 0.5 - canvas.yview('moveto', (y1-h) / y2) - - def save_options(self, optiondb): - optiondb['UPONCLICK'] = self.__uoc.get() - - def colordb_changed(self, colordb): - self.__canvas.delete('all') - self.__populate() diff --git a/Tools/pynche/Main.py b/Tools/pynche/Main.py deleted file mode 100644 index 4db560b568179..0000000000000 --- a/Tools/pynche/Main.py +++ /dev/null @@ -1,229 +0,0 @@ -"""Pynche -- The PYthon Natural Color and Hue Editor. - -Contact: %(AUTHNAME)s -Email: %(AUTHEMAIL)s -Version: %(__version__)s - -Pynche is based largely on a similar color editor I wrote years ago for the -SunView window system. That editor was called ICE: the Interactive Color -Editor. I'd always wanted to port the editor to X but didn't feel like -hacking X and C code to do it. Fast forward many years, to where Python + -Tkinter provides such a nice programming environment, with enough power, that -I finally buckled down and implemented it. I changed the name because these -days, too many other systems have the acronym `ICE'. - -This program currently requires Python 2.2 with Tkinter. - -Usage: %(PROGRAM)s [-d file] [-i file] [-X] [-v] [-h] [initialcolor] - -Where: - --database file - -d file - Alternate location of a color database file - - --initfile file - -i file - Alternate location of the initialization file. This file contains a - persistent database of the current Pynche options and color. This - means that Pynche restores its option settings and current color when - it restarts, using this file (unless the -X option is used). The - default is ~/.pynche - - --ignore - -X - Ignore the initialization file when starting up. Pynche will still - write the current option settings to this file when it quits. - - --version - -v - print the version number and exit - - --help - -h - print this message - - initialcolor - initial color, as a color name or #RRGGBB format -""" - -__version__ = '1.4.1' - -import sys -import os -import getopt -import ColorDB - -from PyncheWidget import PyncheWidget -from Switchboard import Switchboard -from StripViewer import StripViewer -from ChipViewer import ChipViewer -from TypeinViewer import TypeinViewer - - - -PROGRAM = sys.argv[0] -AUTHNAME = 'Barry Warsaw' -AUTHEMAIL = 'barry at python.org' - -# Default locations of rgb.txt or other textual color database -RGB_TXT = [ - # Solaris OpenWindows - '/usr/openwin/lib/rgb.txt', - # Linux - '/usr/lib/X11/rgb.txt', - # The X11R6.4 rgb.txt file - os.path.join(sys.path[0], 'X/rgb.txt'), - # add more here - ] - - - -# Do this because PyncheWidget.py wants to get at the interpolated docstring -# too, for its Help menu. -def docstring(): - return __doc__ % globals() - - -def usage(code, msg=''): - print(docstring()) - if msg: - print(msg) - sys.exit(code) - - - -def initial_color(s, colordb): - # function called on every color - def scan_color(s, colordb=colordb): - try: - r, g, b = colordb.find_byname(s) - except ColorDB.BadColor: - try: - r, g, b = ColorDB.rrggbb_to_triplet(s) - except ColorDB.BadColor: - return None, None, None - return r, g, b - # - # First try the passed in color - r, g, b = scan_color(s) - if r is None: - # try the same color with '#' prepended, since some shells require - # this to be escaped, which is a pain - r, g, b = scan_color('#' + s) - if r is None: - print('Bad initial color, using gray50:', s) - r, g, b = scan_color('gray50') - if r is None: - usage(1, 'Cannot find an initial color to use') - # does not return - return r, g, b - - - -def build(master=None, initialcolor=None, initfile=None, ignore=None, - dbfile=None): - # create all output widgets - s = Switchboard(not ignore and initfile) - # defer to the command line chosen color database, falling back to the one - # in the .pynche file. - if dbfile is None: - dbfile = s.optiondb().get('DBFILE') - # find a parseable color database - colordb = None - files = RGB_TXT[:] - if dbfile is None: - dbfile = files.pop() - while colordb is None: - try: - colordb = ColorDB.get_colordb(dbfile) - except (KeyError, IOError): - pass - if colordb is None: - if not files: - break - dbfile = files.pop(0) - if not colordb: - usage(1, 'No color database file found, see the -d option.') - s.set_colordb(colordb) - - # create the application window decorations - app = PyncheWidget(__version__, s, master=master) - w = app.window() - - # these built-in viewers live inside the main Pynche window - s.add_view(StripViewer(s, w)) - s.add_view(ChipViewer(s, w)) - s.add_view(TypeinViewer(s, w)) - - # get the initial color as components and set the color on all views. if - # there was no initial color given on the command line, use the one that's - # stored in the option database - if initialcolor is None: - optiondb = s.optiondb() - red = optiondb.get('RED') - green = optiondb.get('GREEN') - blue = optiondb.get('BLUE') - # but if there wasn't any stored in the database, use grey50 - if red is None or blue is None or green is None: - red, green, blue = initial_color('grey50', colordb) - else: - red, green, blue = initial_color(initialcolor, colordb) - s.update_views(red, green, blue) - return app, s - - -def run(app, s): - try: - app.start() - except KeyboardInterrupt: - pass - - - -def main(): - try: - opts, args = getopt.getopt( - sys.argv[1:], - 'hd:i:Xv', - ['database=', 'initfile=', 'ignore', 'help', 'version']) - except getopt.error as msg: - usage(1, msg) - - if len(args) == 0: - initialcolor = None - elif len(args) == 1: - initialcolor = args[0] - else: - usage(1) - - ignore = False - dbfile = None - initfile = os.path.expanduser('~/.pynche') - for opt, arg in opts: - if opt in ('-h', '--help'): - usage(0) - elif opt in ('-v', '--version'): - print("""\ -Pynche -- The PYthon Natural Color and Hue Editor. -Contact: %(AUTHNAME)s -Email: %(AUTHEMAIL)s -Version: %(__version__)s""" % globals()) - sys.exit(0) - elif opt in ('-d', '--database'): - dbfile = arg - elif opt in ('-X', '--ignore'): - ignore = True - elif opt in ('-i', '--initfile'): - initfile = arg - - app, sb = build(initialcolor=initialcolor, - initfile=initfile, - ignore=ignore, - dbfile=dbfile) - run(app, sb) - sb.save_views() - - - -if __name__ == '__main__': - main() diff --git a/Tools/pynche/PyncheWidget.py b/Tools/pynche/PyncheWidget.py deleted file mode 100644 index ea456e577e12a..0000000000000 --- a/Tools/pynche/PyncheWidget.py +++ /dev/null @@ -1,313 +0,0 @@ -"""Main Pynche (Pythonically Natural Color and Hue Editor) widget. - -This window provides the basic decorations, primarily including the menubar. -It is used to bring up other windows. -""" - -import sys -import os -from tkinter import * -from tkinter import messagebox, filedialog -import ColorDB - -# Milliseconds between interrupt checks -KEEPALIVE_TIMER = 500 - - - -class PyncheWidget: - def __init__(self, version, switchboard, master=None, extrapath=[]): - self.__sb = switchboard - self.__version = version - self.__textwin = None - self.__listwin = None - self.__detailswin = None - self.__helpwin = None - self.__dialogstate = {} - modal = self.__modal = not not master - # If a master was given, we are running as a modal dialog servant to - # some other application. We rearrange our UI in this case (there's - # no File menu and we get `Okay' and `Cancel' buttons), and we do a - # grab_set() to make ourselves modal - if modal: - self.__tkroot = tkroot = Toplevel(master, class_='Pynche') - tkroot.grab_set() - tkroot.withdraw() - else: - # Is there already a default root for Tk, say because we're - # running under Guido's IDE? :-) Two conditions say no, either the - # _default_root is None or it is unset. - tkroot = getattr(tkinter, '_default_root', None) - if not tkroot: - tkroot = Tk(className='Pynche') - self.__tkroot = tkroot - # but this isn't our top level widget, so make it invisible - tkroot.withdraw() - # create the menubar - menubar = self.__menubar = Menu(tkroot) - # - # File menu - # - filemenu = self.__filemenu = Menu(menubar, tearoff=0) - filemenu.add_command(label='Load palette...', - command=self.__load, - underline=0) - if not modal: - filemenu.add_command(label='Quit', - command=self.__quit, - accelerator='Alt-Q', - underline=0) - # - # View menu - # - views = make_view_popups(self.__sb, self.__tkroot, extrapath) - viewmenu = Menu(menubar, tearoff=0) - for v in views: - viewmenu.add_command(label=v.menutext(), - command=v.popup, - underline=v.underline()) - # - # Help menu - # - helpmenu = Menu(menubar, name='help', tearoff=0) - helpmenu.add_command(label='About Pynche...', - command=self.__popup_about, - underline=0) - helpmenu.add_command(label='Help...', - command=self.__popup_usage, - underline=0) - # - # Tie them all together - # - menubar.add_cascade(label='File', - menu=filemenu, - underline=0) - menubar.add_cascade(label='View', - menu=viewmenu, - underline=0) - menubar.add_cascade(label='Help', - menu=helpmenu, - underline=0) - - # now create the top level window - root = self.__root = Toplevel(tkroot, class_='Pynche', menu=menubar) - root.protocol('WM_DELETE_WINDOW', - modal and self.__bell or self.__quit) - root.title('Pynche %s' % version) - root.iconname('Pynche') - # Only bind accelerators for the File->Quit menu item if running as a - # standalone app - if not modal: - root.bind('', self.__quit) - root.bind('', self.__quit) - else: - # We're a modal dialog so we have a new row of buttons - bframe = Frame(root, borderwidth=1, relief=RAISED) - bframe.grid(row=4, column=0, columnspan=2, - sticky='EW', - ipady=5) - okay = Button(bframe, - text='Okay', - command=self.__okay) - okay.pack(side=LEFT, expand=1) - cancel = Button(bframe, - text='Cancel', - command=self.__cancel) - cancel.pack(side=LEFT, expand=1) - - def __quit(self, event=None): - self.__tkroot.quit() - - def __bell(self, event=None): - self.__tkroot.bell() - - def __okay(self, event=None): - self.__sb.withdraw_views() - self.__tkroot.grab_release() - self.__quit() - - def __cancel(self, event=None): - self.__sb.canceled() - self.__okay() - - def __keepalive(self): - # Exercise the Python interpreter regularly so keyboard interrupts get - # through. - self.__tkroot.tk.createtimerhandler(KEEPALIVE_TIMER, self.__keepalive) - - def start(self): - if not self.__modal: - self.__keepalive() - self.__tkroot.mainloop() - - def window(self): - return self.__root - - def __popup_about(self, event=None): - from Main import __version__ - messagebox.showinfo('About Pynche ' + __version__, - '''\ -Pynche %s -The PYthonically Natural -Color and Hue Editor - -For information -contact: Barry A. Warsaw -email: bwarsaw at python.org''' % __version__) - - def __popup_usage(self, event=None): - if not self.__helpwin: - self.__helpwin = Helpwin(self.__root, self.__quit) - self.__helpwin.deiconify() - - def __load(self, event=None): - while 1: - idir, ifile = os.path.split(self.__sb.colordb().filename()) - file = filedialog.askopenfilename( - filetypes=[('Text files', '*.txt'), - ('All files', '*'), - ], - initialdir=idir, - initialfile=ifile) - if not file: - # cancel button - return - try: - colordb = ColorDB.get_colordb(file) - except IOError: - messagebox.showerror('Read error', '''\ -Could not open file for reading: -%s''' % file) - continue - if colordb is None: - messagebox.showerror('Unrecognized color file type', '''\ -Unrecognized color file type in file: -%s''' % file) - continue - break - self.__sb.set_colordb(colordb) - - def withdraw(self): - self.__root.withdraw() - - def deiconify(self): - self.__root.deiconify() - - - -class Helpwin: - def __init__(self, master, quitfunc): - from Main import docstring - self.__root = root = Toplevel(master, class_='Pynche') - root.protocol('WM_DELETE_WINDOW', self.__withdraw) - root.title('Pynche Help Window') - root.iconname('Pynche Help Window') - root.bind('', quitfunc) - root.bind('', quitfunc) - root.bind('', self.__withdraw) - root.bind('', self.__withdraw) - - # more elaborate help is available in the README file - readmefile = os.path.join(sys.path[0], 'README') - try: - fp = None - try: - fp = open(readmefile) - contents = fp.read() - # wax the last page, it contains Emacs cruft - i = contents.rfind('\f') - if i > 0: - contents = contents[:i].rstrip() - finally: - if fp: - fp.close() - except IOError: - sys.stderr.write("Couldn't open Pynche's README, " - 'using docstring instead.\n') - contents = docstring() - - self.__text = text = Text(root, relief=SUNKEN, - width=80, height=24) - self.__text.focus_set() - text.insert(0.0, contents) - scrollbar = Scrollbar(root) - scrollbar.pack(fill=Y, side=RIGHT) - text.pack(fill=BOTH, expand=YES) - text.configure(yscrollcommand=(scrollbar, 'set')) - scrollbar.configure(command=(text, 'yview')) - - def __withdraw(self, event=None): - self.__root.withdraw() - - def deiconify(self): - self.__root.deiconify() - - - -import functools - at functools.total_ordering -class PopupViewer: - def __init__(self, module, name, switchboard, root): - self.__m = module - self.__name = name - self.__sb = switchboard - self.__root = root - self.__menutext = module.ADDTOVIEW - # find the underline character - underline = module.ADDTOVIEW.find('%') - if underline == -1: - underline = 0 - else: - self.__menutext = module.ADDTOVIEW.replace('%', '', 1) - self.__underline = underline - self.__window = None - - def menutext(self): - return self.__menutext - - def underline(self): - return self.__underline - - def popup(self, event=None): - if not self.__window: - # class and module must have the same name - class_ = getattr(self.__m, self.__name) - self.__window = class_(self.__sb, self.__root) - self.__sb.add_view(self.__window) - self.__window.deiconify() - - def __eq__(self, other): - if isinstance(self, PopupViewer): - return self.__menutext == other.__menutext - return NotImplemented - - def __lt__(self, other): - if isinstance(self, PopupViewer): - return self.__menutext < other.__menutext - return NotImplemented - - -def make_view_popups(switchboard, root, extrapath): - viewers = [] - # where we are in the file system - dirs = [os.path.dirname(__file__)] + extrapath - for dir in dirs: - if dir == '': - dir = '.' - for file in os.listdir(dir): - if file[-9:] == 'Viewer.py': - name = file[:-3] - try: - module = __import__(name) - except ImportError: - # Pynche is running from inside a package, so get the - # module using the explicit path. - pkg = __import__('pynche.'+name) - module = getattr(pkg, name) - if hasattr(module, 'ADDTOVIEW') and module.ADDTOVIEW: - # this is an external viewer - v = PopupViewer(module, name, switchboard, root) - viewers.append(v) - # sort alphabetically - viewers.sort() - return viewers diff --git a/Tools/pynche/README b/Tools/pynche/README deleted file mode 100644 index ae8183f2a7148..0000000000000 --- a/Tools/pynche/README +++ /dev/null @@ -1,398 +0,0 @@ -Pynche - The PYthonically Natural Color and Hue Editor - -Contact: Barry A. Warsaw -Email: bwarsaw at python.org -Version: 1.3 - -Introduction - - Pynche is a color editor based largely on a similar program that I - originally wrote back in 1987 for the Sunview window system. That - editor was called ICE, the Interactive Color Editor. I'd always - wanted to port this program to X but didn't feel like hacking X - and C code to do it. Fast forward many years, to where Python + - Tkinter provides such a nice programming environment, with enough - power, that I finally buckled down and re-implemented it. I - changed the name because these days, too many other systems have - the acronym `ICE'. - - Pynche should work with any variant of Python after 1.5.2 - (e.g. 2.0.1 and 2.1.1), using Tk 8.0.x. It's been tested on - Solaris 2.6, Windows NT 4, and various Linux distros. You'll want - to be sure to have at least Tk 8.0.3 for Windows. Also, Pynche is - very colormap intensive, so it doesn't work very well on 8-bit - graphics cards; 24bit+ graphics cards are so cheap these days, - I'll probably never "fix" that. - - Pynche must find a text database of colors names in order to - provide `nearest' color matching. Pynche is distributed with an - rgb.txt file from the X11R6.4 distribution for this reason, along - with other "web related" database (see below). You can use a - different file with the -d option. The file xlicense.txt contains - the license only for rgb.txt and both files are in the X/ - subdirectory. - - Pynche is pronounced: Pin'-chee - - -Running Standalone - - On Unix, start it by running the `pynche' script. On Windows, run - pynche.pyw to inhibit the console window. When run from the - command line, the following options are recognized: - - --database file - -d file - Alternate location of the color database file. Without this - option, the first valid file found will be used (see below). - - --initfile file - -i file - Alternate location of the persistent initialization file. See - the section on Persistency below. - - --ignore - -X - Ignore the persistent initialization file when starting up. - Pynche will still write the current option settings to the - persistent init file when it quits. - - --help - -h - Print the help message. - - initialcolor - a Tk color name or #rrggbb color spec to be used as the - initially selected color. This overrides any color saved in - the persistent init file. Since `#' needs to be escaped in - many shells, it is optional in the spec (e.g. #45dd1f is the - same as 45dd1f). - - -Running as a Modal Dialog - - Pynche can be run as a modal dialog, inside another application, - say as a general color chooser. In fact, Grail 0.6 uses Pynche - and a future version of IDLE may as well. Pynche supports the API - implemented by the Tkinter standard tkColorChooser module, with a - few changes as described below. By importing pyColorChooser from - the Pynche package, you can run - - pyColorChooser.askcolor() - - which will popup Pynche as a modal dialog, and return the selected - color. - - There are some UI differences when running as a modal - vs. standalone. When running as a modal, there is no "Quit" menu - item under the "File" menu. Instead there are "Okay" and "Cancel" - buttons. - - When "Okay" is hit, askcolor() returns the tuple - - ((r, g, b), "name") - - where r, g, and b are red, green, and blue color values - respectively (in the range 0 to 255). "name" will be a color name - from the color database if there is an exact match, otherwise it - will be an X11 color spec of the form "#rrggbb". Note that this - is different than tkColorChooser, which doesn't know anything - about color names. - - askcolor() supports the following optional keyword arguments: - - color - the color to set as the initial selected color - - master[*] - the master window to use as the parent of the modal - dialog. Without this argument, pyColorChooser will create - its own Tkinter.Tk instance as the master. This may not - be what you want. - - databasefile - similar to the --database option, the value must be a - file name - - initfile[*] - similar to the --initfile option, the value must be a - file name - - ignore[*] - similar to the --ignore flag, the value is a boolean - - wantspec - When this is true, the "name" field in the return tuple - will always be a color spec of the form "#rrggbb". It - will not return a color name even if there is a match; - this is so pyColorChooser can exactly match the API of - tkColorChooser. - - [*] these arguments must be specified the first time - askcolor() is used and cannot be changed on subsequent calls. - - -The Colorstrip Window - - The top part of the main Pynche window contains the "variation - strips". Each strip contains a number of "color chips". The - strips always indicate the currently selected color by a highlight - rectangle around the selected color chip, with an arrow pointing - to the chip. Each arrow has an associated number giving you the - color value along the variation's axis. Each variation strip - shows you the colors that are reachable from the selected color by - varying just one axis of the color solid. - - For example, when the selected color is (in Red/Green/Blue - notation) 127/127/127, the Red Variations strip shows you every - color in the range 0/127/127 to 255/127/127. Similarly for the - green and blue axes. You can select any color by clicking on its - chip. This will update the highlight rectangle and the arrow, as - well as other displays in Pynche. - - Click on "Update while dragging" if you want Pynche to update the - selected color while you drag along any variation strip (this will - be a bit slower). Click on "Hexadecimal" to display the arrow - numbers in hex. - - There are also two shortcut buttons in this window, which - auto-select Black (0/0/0) and White (255/255/255). - - -The Proof Window - - In the lower left corner of the main window you see two larger - color chips. The Selected chip shows you a larger version of the - color selected in the variation strips, along with its X11 color - specification. The Nearest chip shows you the closest color in - the X11 database to the selected color, giving its X11 color - specification, and below that, its X11 color name. When the - Selected chip color exactly matches the Nearest chip color, you - will see the color name appear below the color specification for - the Selected chip. - - Clicking on the Nearest color chip selects that color. Color - distance is calculated in the 3D space of the RGB color solid and - if more than one color name is the same distance from the selected - color, the first one found will be chosen. - - Note that there may be more than one X11 color name for the same - RGB value. In that case, the first one found in the text database - is designated the "primary" name, and this is shown under the - Nearest chip. The other names are "aliases" and they are visible - in the Color List Window (see below). - - Both the color specifications and color names are selectable for - copying and pasting into another window. - - -The Type-in Window - - At the lower right of the main window are three entry fields. - Here you can type numeric values for any of the three color axes. - Legal values are between 0 and 255, and these fields do not allow - you to enter illegal values. You must hit Enter or Tab to select - the new color. - - Click on "Update while typing" if you want Pynche to select the - color on every keystroke (well, every one that produces a legal - value!) Click on "Hexadecimal" to display and enter color values - in hex. - - -Other Views - - There are three secondary windows which are not displayed by - default. You can bring these up via the "View" menu on the main - Pynche window. - - -The Text Window - - The "Text Window" allows you to see what effects various colors - have on the standard Tk text widget elements. In the upper part - of the window is a plain Tk text widget and here you can edit the - text, select a region of text, etc. Below this is a button "Track - color changes". When this is turned on, any colors selected in - the other windows will change the text widget element specified in - the radio buttons below. When this is turned off, text widget - elements are not affected by color selection. - - You can choose which element gets changed by color selection by - clicking on one of the radio buttons in the bottom part of this - window. Text foreground and background affect the text in the - upper part of the window. Selection foreground and background - affect the colors of the primary selection which is what you see - when you click the middle button (depending on window system) and - drag it through some text. - - The Insertion is the insertion cursor in the text window, where - new text will be inserted as you type. The insertion cursor only - has a background. - - -The Color List Window - - The "Color List" window shows every named color in the color name - database (this window may take a while to come up). In the upper - part of the window you see a scrolling list of all the color names - in the database, in alphabetical order. Click on any color to - select it. In the bottom part of the window is displayed any - aliases for the selected color (those color names that have the - same RGB value, but were found later in the text database). For - example, find the color "Black" and you'll see that its aliases - are "gray0" and "grey0". - - If the color has no aliases you'll see "" here. If you - just want to see if a color has an alias, and do not want to select a - color when you click on it, turn off "Update on Click". - - Note that the color list is always updated when a color is selected - from the main window. There's no way to turn this feature off. If - the selected color has no matching color name you'll see - "" in the Aliases window. - - -The Details Window - - The "Details" window gives you more control over color selection - than just clicking on a color chip in the main window. The row of - buttons along the top apply the specified increment and decrement - amounts to the selected color. These delta amounts are applied to - the variation strips specified by the check boxes labeled "Move - Sliders". Thus if just Red and Green are selected, hitting -10 - will subtract 10 from the color value along the red and green - variation only. Note the message under the checkboxes; this - indicates the primary color level being changed when more than one - slider is tied together. For example, if Red and Green are - selected, you will be changing the Yellow level of the selected - color. - - The "At Boundary" behavior determines what happens when any color - variation hits either the lower or upper boundaries (0 or 255) as - a result of clicking on the top row buttons: - - Stop - When the increment or decrement would send any of the tied - variations out of bounds, the entire delta is discarded. - - Wrap Around - When the increment or decrement would send any of the tied - variations out of bounds, the out of bounds value is wrapped - around to the other side. Thus if red were at 238 and +25 - were clicked, red would have the value 7. - - Preserve Distance - When the increment or decrement would send any of the tied - variations out of bounds, all tied variations are wrapped as - one, so as to preserve the distance between them. Thus if - green and blue were tied, and green was at 238 while blue was - at 223, and +25 were clicked, green would be at 15 and blue - would be at 0. - - Squash - When the increment or decrement would send any of the tied - variations out of bounds, the out of bounds variation is set - to the ceiling of 255 or floor of 0, as appropriate. In this - way, all tied variations are squashed to one edge or the - other. - - The top row buttons have the following keyboard accelerators: - - -25 == Shift Left Arrow - -10 == Control Left Arrow - -1 == Left Arrow - +1 == Right Arrow - +10 == Control Right Arrow - +25 == Shift Right Arrow - - -Keyboard Accelerators - - Alt-w in any secondary window dismisses the window. In the main - window it exits Pynche (except when running as a modal). - - Alt-q in any window exits Pynche (except when running as a modal). - - -Persistency - - Pynche remembers various settings of options and colors between - invocations, storing these values in a `persistent initialization - file'. The actual location of this file is specified by the - --initfile option (see above), and defaults to ~/.pynche. - - When Pynche exits, it saves these values in the init file, and - re-reads them when it starts up. There is no locking on this - file, so if you run multiple instances of Pynche at a time, you - may clobber the init file. - - The actual options stored include - - - the currently selected color - - - all settings of checkbox and radio button options in all windows - - - the contents of the text window, the current text selection and - insertion point, and all current text widget element color - settings. - - - the name of the color database file (but not its contents) - - You can inhibit Pynche from reading the init file by supplying the - --ignore option on the command line. However, you cannot suppress - the storing of the settings in the init file on Pynche exit. If - you really want to do this, use /dev/null as the init file, using - --initfile. - - -Color Name Database Files - - Pynche uses a color name database file to calculate the nearest - color to the selected color, and to display in the Color List - view. Several files are distributed with Pynche, described - below. By default, the X11 color name database file is selected. - Other files: - - html40colors.txt -- the HTML 4.0 guaranteed color names - - websafe.txt -- the 216 "web-safe" colors that Netscape and MSIE - guarantee will not be dithered. These are specified in #rrggbb - format for both values and names - - webcolors.txt -- The 140 color names that Tim Peters and his - sister say NS and MSIE both understand (with some controversy over - AliceBlue). - - namedcolors.txt -- an alternative set of Netscape colors. - - You can switch between files by choosing "Load palette..." from - the "File" menu. This brings up a standard Tk file dialog. - Choose the file you want and then click "Ok". If Pynche - understands the format in this file, it will load the database and - update the appropriate windows. If not, it will bring up an error - dialog. - - -To Do - - Here's a brief list of things I want to do (some mythical day): - - - Better support for resizing the top level windows - - - More output views, e.g. color solids - - - Have the notion of a `last color selected'; this may require a - new output view - - - Support setting the font in the text view - - - Support distutils setup.py for installation - - I'm open to suggestions! - - - -Local Variables: -indent-tabs-mode: nil -End: diff --git a/Tools/pynche/StripViewer.py b/Tools/pynche/StripViewer.py deleted file mode 100644 index 6914ca94b2bd3..0000000000000 --- a/Tools/pynche/StripViewer.py +++ /dev/null @@ -1,397 +0,0 @@ -"""Strip viewer and related widgets. - -The classes in this file implement the StripViewer shown in the top two thirds -of the main Pynche window. It consists of three StripWidgets which display -the variations in red, green, and blue respectively of the currently selected -r/g/b color value. - -Each StripWidget shows the color variations that are reachable by varying an -axis of the currently selected color. So for example, if the color is - - (R,G,B)=(127,163,196) - -then the Red variations show colors from (0,163,196) to (255,163,196), the -Green variations show colors from (127,0,196) to (127,255,196), and the Blue -variations show colors from (127,163,0) to (127,163,255). - -The selected color is always visible in all three StripWidgets, and in fact -each StripWidget highlights the selected color, and has an arrow pointing to -the selected chip, which includes the value along that particular axis. - -Clicking on any chip in any StripWidget selects that color, and updates all -arrows and other windows. By toggling on Update while dragging, Pynche will -select the color under the cursor while you drag it, but be forewarned that -this can be slow. -""" - -from tkinter import * -import ColorDB - -# Load this script into the Tcl interpreter and call it in -# StripWidget.set_color(). This is about as fast as it can be with the -# current _tkinter.c interface, which doesn't support Tcl Objects. -TCLPROC = '''\ -proc setcolor {canv colors} { - set i 1 - foreach c $colors { - $canv itemconfigure $i -fill $c -outline $c - incr i - } -} -''' - -# Tcl event types -BTNDOWN = 4 -BTNUP = 5 -BTNDRAG = 6 - -SPACE = ' ' - - - -def constant(numchips): - step = 255.0 / (numchips - 1) - start = 0.0 - seq = [] - while numchips > 0: - seq.append(int(start)) - start = start + step - numchips = numchips - 1 - return seq - -# red variations, green+blue = cyan constant -def constant_red_generator(numchips, red, green, blue): - seq = constant(numchips) - return list(zip([red] * numchips, seq, seq)) - -# green variations, red+blue = magenta constant -def constant_green_generator(numchips, red, green, blue): - seq = constant(numchips) - return list(zip(seq, [green] * numchips, seq)) - -# blue variations, red+green = yellow constant -def constant_blue_generator(numchips, red, green, blue): - seq = constant(numchips) - return list(zip(seq, seq, [blue] * numchips)) - -# red variations, green+blue = cyan constant -def constant_cyan_generator(numchips, red, green, blue): - seq = constant(numchips) - return list(zip(seq, [green] * numchips, [blue] * numchips)) - -# green variations, red+blue = magenta constant -def constant_magenta_generator(numchips, red, green, blue): - seq = constant(numchips) - return list(zip([red] * numchips, seq, [blue] * numchips)) - -# blue variations, red+green = yellow constant -def constant_yellow_generator(numchips, red, green, blue): - seq = constant(numchips) - return list(zip([red] * numchips, [green] * numchips, seq)) - - - -class LeftArrow: - _ARROWWIDTH = 30 - _ARROWHEIGHT = 15 - _YOFFSET = 13 - _TEXTYOFFSET = 1 - _TAG = ('leftarrow',) - - def __init__(self, canvas, x): - self._canvas = canvas - self.__arrow, self.__text = self._create(x) - self.move_to(x) - - def _create(self, x): - arrow = self._canvas.create_line( - x, self._ARROWHEIGHT + self._YOFFSET, - x, self._YOFFSET, - x + self._ARROWWIDTH, self._YOFFSET, - arrow='first', - width=3.0, - tags=self._TAG) - text = self._canvas.create_text( - x + self._ARROWWIDTH + 13, - self._ARROWHEIGHT - self._TEXTYOFFSET, - tags=self._TAG, - text='128') - return arrow, text - - def _x(self): - coords = list(self._canvas.coords(self._TAG)) - assert coords - return coords[0] - - def move_to(self, x): - deltax = x - self._x() - self._canvas.move(self._TAG, deltax, 0) - - def set_text(self, text): - self._canvas.itemconfigure(self.__text, text=text) - - -class RightArrow(LeftArrow): - _TAG = ('rightarrow',) - - def _create(self, x): - arrow = self._canvas.create_line( - x, self._YOFFSET, - x + self._ARROWWIDTH, self._YOFFSET, - x + self._ARROWWIDTH, self._ARROWHEIGHT + self._YOFFSET, - arrow='last', - width=3.0, - tags=self._TAG) - text = self._canvas.create_text( - x - self._ARROWWIDTH + 15, # BAW: kludge - self._ARROWHEIGHT - self._TEXTYOFFSET, - justify=RIGHT, - text='128', - tags=self._TAG) - return arrow, text - - def _x(self): - coords = list(self._canvas.coords(self._TAG)) - assert coords - return coords[0] + self._ARROWWIDTH - - - -class StripWidget: - _CHIPHEIGHT = 50 - _CHIPWIDTH = 10 - _NUMCHIPS = 40 - - def __init__(self, switchboard, - master = None, - chipwidth = _CHIPWIDTH, - chipheight = _CHIPHEIGHT, - numchips = _NUMCHIPS, - generator = None, - axis = None, - label = '', - uwdvar = None, - hexvar = None): - # instance variables - self.__generator = generator - self.__axis = axis - self.__numchips = numchips - assert self.__axis in (0, 1, 2) - self.__uwd = uwdvar - self.__hexp = hexvar - # the last chip selected - self.__lastchip = None - self.__sb = switchboard - - canvaswidth = numchips * (chipwidth + 1) - canvasheight = chipheight + 43 # BAW: Kludge - - # create the canvas and pack it - canvas = self.__canvas = Canvas(master, - width=canvaswidth, - height=canvasheight, -## borderwidth=2, -## relief=GROOVE - ) - - canvas.pack() - canvas.bind('', self.__select_chip) - canvas.bind('', self.__select_chip) - canvas.bind('', self.__select_chip) - - # Load a proc into the Tcl interpreter. This is used in the - # set_color() method to speed up setting the chip colors. - canvas.tk.eval(TCLPROC) - - # create the color strip - chips = self.__chips = [] - x = 1 - y = 30 - tags = ('chip',) - for c in range(self.__numchips): - color = 'grey' - canvas.create_rectangle( - x, y, x+chipwidth, y+chipheight, - fill=color, outline=color, - tags=tags) - x = x + chipwidth + 1 # for outline - chips.append(color) - - # create the strip label - self.__label = canvas.create_text( - 3, y + chipheight + 8, - text=label, - anchor=W) - - # create the arrow and text item - chipx = self.__arrow_x(0) - self.__leftarrow = LeftArrow(canvas, chipx) - - chipx = self.__arrow_x(len(chips) - 1) - self.__rightarrow = RightArrow(canvas, chipx) - - def __arrow_x(self, chipnum): - coords = self.__canvas.coords(chipnum+1) - assert coords - x0, y0, x1, y1 = coords - return (x1 + x0) / 2.0 - - # Invoked when one of the chips is clicked. This should just tell the - # switchboard to set the color on all the output components - def __select_chip(self, event=None): - x = event.x - y = event.y - canvas = self.__canvas - chip = canvas.find_overlapping(x, y, x, y) - if chip and (1 <= chip[0] <= self.__numchips): - color = self.__chips[chip[0]-1] - red, green, blue = ColorDB.rrggbb_to_triplet(color) - etype = int(event.type) - if (etype == BTNUP or self.__uwd.get()): - # update everyone - self.__sb.update_views(red, green, blue) - else: - # just track the arrows - self.__trackarrow(chip[0], (red, green, blue)) - - def __trackarrow(self, chip, rgbtuple): - # invert the last chip - if self.__lastchip is not None: - color = self.__canvas.itemcget(self.__lastchip, 'fill') - self.__canvas.itemconfigure(self.__lastchip, outline=color) - self.__lastchip = chip - # get the arrow's text - coloraxis = rgbtuple[self.__axis] - if self.__hexp.get(): - # hex - text = hex(coloraxis) - else: - # decimal - text = repr(coloraxis) - # move the arrow, and set its text - if coloraxis <= 128: - # use the left arrow - self.__leftarrow.set_text(text) - self.__leftarrow.move_to(self.__arrow_x(chip-1)) - self.__rightarrow.move_to(-100) - else: - # use the right arrow - self.__rightarrow.set_text(text) - self.__rightarrow.move_to(self.__arrow_x(chip-1)) - self.__leftarrow.move_to(-100) - # and set the chip's outline - brightness = ColorDB.triplet_to_brightness(rgbtuple) - if brightness <= 128: - outline = 'white' - else: - outline = 'black' - self.__canvas.itemconfigure(chip, outline=outline) - - - def update_yourself(self, red, green, blue): - assert self.__generator - i = 1 - chip = 0 - chips = self.__chips = [] - tk = self.__canvas.tk - # get the red, green, and blue components for all chips - for t in self.__generator(self.__numchips, red, green, blue): - rrggbb = ColorDB.triplet_to_rrggbb(t) - chips.append(rrggbb) - tred, tgreen, tblue = t - if tred <= red and tgreen <= green and tblue <= blue: - chip = i - i = i + 1 - # call the raw tcl script - colors = SPACE.join(chips) - tk.eval('setcolor %s {%s}' % (self.__canvas._w, colors)) - # move the arrows around - self.__trackarrow(chip, (red, green, blue)) - - def set(self, label, generator): - self.__canvas.itemconfigure(self.__label, text=label) - self.__generator = generator - - -class StripViewer: - def __init__(self, switchboard, master=None): - self.__sb = switchboard - optiondb = switchboard.optiondb() - # create a frame inside the master. - frame = Frame(master, relief=RAISED, borderwidth=1) - frame.grid(row=1, column=0, columnspan=2, sticky='NSEW') - # create the options to be used later - uwd = self.__uwdvar = BooleanVar() - uwd.set(optiondb.get('UPWHILEDRAG', 0)) - hexp = self.__hexpvar = BooleanVar() - hexp.set(optiondb.get('HEXSTRIP', 0)) - # create the red, green, blue strips inside their own frame - frame1 = Frame(frame) - frame1.pack(expand=YES, fill=BOTH) - self.__reds = StripWidget(switchboard, frame1, - generator=constant_cyan_generator, - axis=0, - label='Red Variations', - uwdvar=uwd, hexvar=hexp) - - self.__greens = StripWidget(switchboard, frame1, - generator=constant_magenta_generator, - axis=1, - label='Green Variations', - uwdvar=uwd, hexvar=hexp) - - self.__blues = StripWidget(switchboard, frame1, - generator=constant_yellow_generator, - axis=2, - label='Blue Variations', - uwdvar=uwd, hexvar=hexp) - - # create a frame to contain the controls - frame2 = Frame(frame) - frame2.pack(expand=YES, fill=BOTH) - frame2.columnconfigure(0, weight=20) - frame2.columnconfigure(2, weight=20) - - padx = 8 - - # create the black button - blackbtn = Button(frame2, - text='Black', - command=self.__toblack) - blackbtn.grid(row=0, column=0, rowspan=2, sticky=W, padx=padx) - - # create the controls - uwdbtn = Checkbutton(frame2, - text='Update while dragging', - variable=uwd) - uwdbtn.grid(row=0, column=1, sticky=W) - hexbtn = Checkbutton(frame2, - text='Hexadecimal', - variable=hexp, - command=self.__togglehex) - hexbtn.grid(row=1, column=1, sticky=W) - - # create the white button - whitebtn = Button(frame2, - text='White', - command=self.__towhite) - whitebtn.grid(row=0, column=2, rowspan=2, sticky=E, padx=padx) - - def update_yourself(self, red, green, blue): - self.__reds.update_yourself(red, green, blue) - self.__greens.update_yourself(red, green, blue) - self.__blues.update_yourself(red, green, blue) - - def __togglehex(self, event=None): - red, green, blue = self.__sb.current_rgb() - self.update_yourself(red, green, blue) - - def __toblack(self, event=None): - self.__sb.update_views(0, 0, 0) - - def __towhite(self, event=None): - self.__sb.update_views(255, 255, 255) - - def save_options(self, optiondb): - optiondb['UPWHILEDRAG'] = self.__uwdvar.get() - optiondb['HEXSTRIP'] = self.__hexpvar.get() diff --git a/Tools/pynche/Switchboard.py b/Tools/pynche/Switchboard.py deleted file mode 100644 index 013bb01389139..0000000000000 --- a/Tools/pynche/Switchboard.py +++ /dev/null @@ -1,138 +0,0 @@ -"""Switchboard class. - -This class is used to coordinate updates among all Viewers. Every Viewer must -conform to the following interface: - - - it must include a method called update_yourself() which takes three - arguments; the red, green, and blue values of the selected color. - - - When a Viewer selects a color and wishes to update all other Views, it - should call update_views() on the Switchboard object. Note that the - Viewer typically does *not* update itself before calling update_views(), - since this would cause it to get updated twice. - -Optionally, Viewers can also implement: - - - save_options() which takes an optiondb (a dictionary). Store into this - dictionary any values the Viewer wants to save in the persistent - ~/.pynche file. This dictionary is saved using marshal. The namespace - for the keys is ad-hoc; make sure you don't clobber some other Viewer's - keys! - - - withdraw() which takes no arguments. This is called when Pynche is - unmapped. All Viewers should implement this. - - - colordb_changed() which takes a single argument, an instance of - ColorDB. This is called whenever the color name database is changed and - gives a chance for the Viewers to do something on those events. See - ListViewer for details. - -External Viewers are found dynamically. Viewer modules should have names such -as FooViewer.py. If such a named module has a module global variable called -ADDTOVIEW and this variable is true, the Viewer will be added dynamically to -the `View' menu. ADDTOVIEW contains a string which is used as the menu item -to display the Viewer (one kludge: if the string contains a `%', this is used -to indicate that the next character will get an underline in the menu, -otherwise the first character is underlined). - -FooViewer.py should contain a class called FooViewer, and its constructor -should take two arguments, an instance of Switchboard, and optionally a Tk -master window. - -""" - -import sys -import marshal - - - -class Switchboard: - def __init__(self, initfile): - self.__initfile = initfile - self.__colordb = None - self.__optiondb = {} - self.__views = [] - self.__red = 0 - self.__green = 0 - self.__blue = 0 - self.__canceled = 0 - # read the initialization file - fp = None - if initfile: - try: - try: - fp = open(initfile, 'rb') - self.__optiondb = marshal.load(fp) - if not isinstance(self.__optiondb, dict): - print('Problem reading options from file:', initfile, - file=sys.stderr) - self.__optiondb = {} - except (IOError, EOFError, ValueError): - pass - finally: - if fp: - fp.close() - - def add_view(self, view): - self.__views.append(view) - - def update_views(self, red, green, blue): - self.__red = red - self.__green = green - self.__blue = blue - for v in self.__views: - v.update_yourself(red, green, blue) - - def update_views_current(self): - self.update_views(self.__red, self.__green, self.__blue) - - def current_rgb(self): - return self.__red, self.__green, self.__blue - - def colordb(self): - return self.__colordb - - def set_colordb(self, colordb): - self.__colordb = colordb - for v in self.__views: - if hasattr(v, 'colordb_changed'): - v.colordb_changed(colordb) - self.update_views_current() - - def optiondb(self): - return self.__optiondb - - def save_views(self): - # save the current color - self.__optiondb['RED'] = self.__red - self.__optiondb['GREEN'] = self.__green - self.__optiondb['BLUE'] = self.__blue - for v in self.__views: - if hasattr(v, 'save_options'): - v.save_options(self.__optiondb) - # save the name of the file used for the color database. we'll try to - # load this first. - self.__optiondb['DBFILE'] = self.__colordb.filename() - fp = None - try: - try: - fp = open(self.__initfile, 'wb') - except IOError: - print('Cannot write options to file:', \ - self.__initfile, file=sys.stderr) - else: - marshal.dump(self.__optiondb, fp) - finally: - if fp: - fp.close() - - def withdraw_views(self): - for v in self.__views: - if hasattr(v, 'withdraw'): - v.withdraw() - - def canceled(self, flag=1): - self.__canceled = flag - - def canceled_p(self): - return self.__canceled diff --git a/Tools/pynche/TextViewer.py b/Tools/pynche/TextViewer.py deleted file mode 100644 index baa1e62ca7603..0000000000000 --- a/Tools/pynche/TextViewer.py +++ /dev/null @@ -1,188 +0,0 @@ -"""TextViewer class. - -The TextViewer allows you to see how the selected color would affect various -characteristics of a Tk text widget. This is an output viewer only. - -In the top part of the window is a standard text widget with some sample text -in it. You are free to edit this text in any way you want (BAW: allow you to -change font characteristics). If you want changes in other viewers to update -text characteristics, turn on Track color changes. - -To select which characteristic tracks the change, select one of the radio -buttons in the window below. Text foreground and background affect the text -in the window above. The Selection is what you see when you click the middle -button and drag it through some text. The Insertion is the insertion cursor -in the text window (which only has a background). -""" - -from tkinter import * -import ColorDB - -ADDTOVIEW = 'Text Window...' - - - -class TextViewer: - def __init__(self, switchboard, master=None): - self.__sb = switchboard - optiondb = switchboard.optiondb() - root = self.__root = Toplevel(master, class_='Pynche') - root.protocol('WM_DELETE_WINDOW', self.withdraw) - root.title('Pynche Text Window') - root.iconname('Pynche Text Window') - root.bind('', self.__quit) - root.bind('', self.__quit) - root.bind('', self.withdraw) - root.bind('', self.withdraw) - # - # create the text widget - # - self.__text = Text(root, relief=SUNKEN, - background=optiondb.get('TEXTBG', 'black'), - foreground=optiondb.get('TEXTFG', 'white'), - width=35, height=15) - sfg = optiondb.get('TEXT_SFG') - if sfg: - self.__text.configure(selectforeground=sfg) - sbg = optiondb.get('TEXT_SBG') - if sbg: - self.__text.configure(selectbackground=sbg) - ibg = optiondb.get('TEXT_IBG') - if ibg: - self.__text.configure(insertbackground=ibg) - self.__text.pack() - self.__text.insert(0.0, optiondb.get('TEXT', '''\ -Insert some stuff here and play -with the buttons below to see -how the colors interact in -textual displays. - -See how the selection can also -be affected by tickling the buttons -and choosing a color.''')) - insert = optiondb.get('TEXTINS') - if insert: - self.__text.mark_set(INSERT, insert) - try: - start, end = optiondb.get('TEXTSEL', (6.0, END)) - self.__text.tag_add(SEL, start, end) - except ValueError: - # selection wasn't set - pass - self.__text.focus_set() - # - # variables - self.__trackp = BooleanVar() - self.__trackp.set(optiondb.get('TRACKP', 0)) - self.__which = IntVar() - self.__which.set(optiondb.get('WHICH', 0)) - # - # track toggle - self.__t = Checkbutton(root, text='Track color changes', - variable=self.__trackp, - relief=GROOVE, - command=self.__toggletrack) - self.__t.pack(fill=X, expand=YES) - frame = self.__frame = Frame(root) - frame.pack() - # - # labels - self.__labels = [] - row = 2 - for text in ('Text:', 'Selection:', 'Insertion:'): - l = Label(frame, text=text) - l.grid(row=row, column=0, sticky=E) - self.__labels.append(l) - row += 1 - col = 1 - for text in ('Foreground', 'Background'): - l = Label(frame, text=text) - l.grid(row=1, column=col) - self.__labels.append(l) - col += 1 - # - # radios - self.__radios = [] - for col in (1, 2): - for row in (2, 3, 4): - # there is no insertforeground option - if row==4 and col==1: - continue - r = Radiobutton(frame, variable=self.__which, - value=(row-2)*2 + col-1, - command=self.__set_color) - r.grid(row=row, column=col) - self.__radios.append(r) - self.__toggletrack() - - def __quit(self, event=None): - self.__root.quit() - - def withdraw(self, event=None): - self.__root.withdraw() - - def deiconify(self, event=None): - self.__root.deiconify() - - def __forceupdate(self, event=None): - self.__sb.update_views_current() - - def __toggletrack(self, event=None): - if self.__trackp.get(): - state = NORMAL - fg = self.__radios[0]['foreground'] - else: - state = DISABLED - fg = self.__radios[0]['disabledforeground'] - for r in self.__radios: - r.configure(state=state) - for l in self.__labels: - l.configure(foreground=fg) - - def __set_color(self, event=None): - which = self.__which.get() - text = self.__text - if which == 0: - color = text['foreground'] - elif which == 1: - color = text['background'] - elif which == 2: - color = text['selectforeground'] - elif which == 3: - color = text['selectbackground'] - elif which == 5: - color = text['insertbackground'] - try: - red, green, blue = ColorDB.rrggbb_to_triplet(color) - except ColorDB.BadColor: - # must have been a color name - red, green, blue = self.__sb.colordb().find_byname(color) - self.__sb.update_views(red, green, blue) - - def update_yourself(self, red, green, blue): - if self.__trackp.get(): - colorname = ColorDB.triplet_to_rrggbb((red, green, blue)) - which = self.__which.get() - text = self.__text - if which == 0: - text.configure(foreground=colorname) - elif which == 1: - text.configure(background=colorname) - elif which == 2: - text.configure(selectforeground=colorname) - elif which == 3: - text.configure(selectbackground=colorname) - elif which == 5: - text.configure(insertbackground=colorname) - - def save_options(self, optiondb): - optiondb['TRACKP'] = self.__trackp.get() - optiondb['WHICH'] = self.__which.get() - optiondb['TEXT'] = self.__text.get(0.0, 'end - 1c') - optiondb['TEXTSEL'] = self.__text.tag_ranges(SEL)[0:2] - optiondb['TEXTINS'] = self.__text.index(INSERT) - optiondb['TEXTFG'] = self.__text['foreground'] - optiondb['TEXTBG'] = self.__text['background'] - optiondb['TEXT_SFG'] = self.__text['selectforeground'] - optiondb['TEXT_SBG'] = self.__text['selectbackground'] - optiondb['TEXT_IBG'] = self.__text['insertbackground'] diff --git a/Tools/pynche/TypeinViewer.py b/Tools/pynche/TypeinViewer.py deleted file mode 100644 index 2f93e6b44f4ac..0000000000000 --- a/Tools/pynche/TypeinViewer.py +++ /dev/null @@ -1,161 +0,0 @@ -"""TypeinViewer class. - -The TypeinViewer is what you see at the lower right of the main Pynche -widget. It contains three text entry fields, one each for red, green, blue. -Input into these windows is highly constrained; it only allows you to enter -values that are legal for a color axis. This usually means 0-255 for decimal -input and 0x0 - 0xff for hex input. - -You can toggle whether you want to view and input the values in either decimal -or hex by clicking on Hexadecimal. By clicking on Update while typing, the -color selection will be made on every change to the text field. Otherwise, -you must hit Return or Tab to select the color. -""" - -from tkinter import * - - - -class TypeinViewer: - def __init__(self, switchboard, master=None): - # non-gui ivars - self.__sb = switchboard - optiondb = switchboard.optiondb() - self.__hexp = BooleanVar() - self.__hexp.set(optiondb.get('HEXTYPE', 0)) - self.__uwtyping = BooleanVar() - self.__uwtyping.set(optiondb.get('UPWHILETYPE', 0)) - # create the gui - self.__frame = Frame(master, relief=RAISED, borderwidth=1) - self.__frame.grid(row=3, column=1, sticky='NSEW') - # Red - self.__xl = Label(self.__frame, text='Red:') - self.__xl.grid(row=0, column=0, sticky=E) - subframe = Frame(self.__frame) - subframe.grid(row=0, column=1) - self.__xox = Label(subframe, text='0x') - self.__xox.grid(row=0, column=0, sticky=E) - self.__xox['font'] = 'courier' - self.__x = Entry(subframe, width=3) - self.__x.grid(row=0, column=1) - self.__x.bindtags(self.__x.bindtags() + ('Normalize', 'Update')) - self.__x.bind_class('Normalize', '', self.__normalize) - self.__x.bind_class('Update' , '', self.__maybeupdate) - # Green - self.__yl = Label(self.__frame, text='Green:') - self.__yl.grid(row=1, column=0, sticky=E) - subframe = Frame(self.__frame) - subframe.grid(row=1, column=1) - self.__yox = Label(subframe, text='0x') - self.__yox.grid(row=0, column=0, sticky=E) - self.__yox['font'] = 'courier' - self.__y = Entry(subframe, width=3) - self.__y.grid(row=0, column=1) - self.__y.bindtags(self.__y.bindtags() + ('Normalize', 'Update')) - # Blue - self.__zl = Label(self.__frame, text='Blue:') - self.__zl.grid(row=2, column=0, sticky=E) - subframe = Frame(self.__frame) - subframe.grid(row=2, column=1) - self.__zox = Label(subframe, text='0x') - self.__zox.grid(row=0, column=0, sticky=E) - self.__zox['font'] = 'courier' - self.__z = Entry(subframe, width=3) - self.__z.grid(row=0, column=1) - self.__z.bindtags(self.__z.bindtags() + ('Normalize', 'Update')) - # Update while typing? - self.__uwt = Checkbutton(self.__frame, - text='Update while typing', - variable=self.__uwtyping) - self.__uwt.grid(row=3, column=0, columnspan=2, sticky=W) - # Hex/Dec - self.__hex = Checkbutton(self.__frame, - text='Hexadecimal', - variable=self.__hexp, - command=self.__togglehex) - self.__hex.grid(row=4, column=0, columnspan=2, sticky=W) - - def __togglehex(self, event=None): - red, green, blue = self.__sb.current_rgb() - if self.__hexp.get(): - label = '0x' - else: - label = ' ' - self.__xox['text'] = label - self.__yox['text'] = label - self.__zox['text'] = label - self.update_yourself(red, green, blue) - - def __normalize(self, event=None): - ew = event.widget - contents = ew.get() - icursor = ew.index(INSERT) - if contents and contents[0] in 'xX' and self.__hexp.get(): - contents = '0' + contents - # Figure out the contents in the current base. - try: - if self.__hexp.get(): - v = int(contents, 16) - else: - v = int(contents) - except ValueError: - v = None - # If value is not legal, or empty, delete the last character inserted - # and ring the bell. Don't ring the bell if the field is empty (it'll - # just equal zero. - if v is None: - pass - elif v < 0 or v > 255: - i = ew.index(INSERT) - if event.char: - contents = contents[:i-1] + contents[i:] - icursor -= 1 - ew.bell() - elif self.__hexp.get(): - contents = hex(v)[2:] - else: - contents = int(v) - ew.delete(0, END) - ew.insert(0, contents) - ew.icursor(icursor) - - def __maybeupdate(self, event=None): - if self.__uwtyping.get() or event.keysym in ('Return', 'Tab'): - self.__update(event) - - def __update(self, event=None): - redstr = self.__x.get() or '0' - greenstr = self.__y.get() or '0' - bluestr = self.__z.get() or '0' - if self.__hexp.get(): - base = 16 - else: - base = 10 - red, green, blue = [int(x, base) for x in (redstr, greenstr, bluestr)] - self.__sb.update_views(red, green, blue) - - def update_yourself(self, red, green, blue): - if self.__hexp.get(): - sred, sgreen, sblue = [hex(x)[2:] for x in (red, green, blue)] - else: - sred, sgreen, sblue = red, green, blue - x, y, z = self.__x, self.__y, self.__z - xicursor = x.index(INSERT) - yicursor = y.index(INSERT) - zicursor = z.index(INSERT) - x.delete(0, END) - y.delete(0, END) - z.delete(0, END) - x.insert(0, sred) - y.insert(0, sgreen) - z.insert(0, sblue) - x.icursor(xicursor) - y.icursor(yicursor) - z.icursor(zicursor) - - def hexp_var(self): - return self.__hexp - - def save_options(self, optiondb): - optiondb['HEXTYPE'] = self.__hexp.get() - optiondb['UPWHILETYPE'] = self.__uwtyping.get() diff --git a/Tools/pynche/X/rgb.txt b/Tools/pynche/X/rgb.txt deleted file mode 100644 index b11ffd058cb96..0000000000000 --- a/Tools/pynche/X/rgb.txt +++ /dev/null @@ -1,753 +0,0 @@ -! $XConsortium: rgb.txt,v 10.41 94/02/20 18:39:36 rws Exp $ -255 250 250 snow -248 248 255 ghost white -248 248 255 GhostWhite -245 245 245 white smoke -245 245 245 WhiteSmoke -220 220 220 gainsboro -255 250 240 floral white -255 250 240 FloralWhite -253 245 230 old lace -253 245 230 OldLace -250 240 230 linen -250 235 215 antique white -250 235 215 AntiqueWhite -255 239 213 papaya whip -255 239 213 PapayaWhip -255 235 205 blanched almond -255 235 205 BlanchedAlmond -255 228 196 bisque -255 218 185 peach puff -255 218 185 PeachPuff -255 222 173 navajo white -255 222 173 NavajoWhite -255 228 181 moccasin -255 248 220 cornsilk -255 255 240 ivory -255 250 205 lemon chiffon -255 250 205 LemonChiffon -255 245 238 seashell -240 255 240 honeydew -245 255 250 mint cream -245 255 250 MintCream -240 255 255 azure -240 248 255 alice blue -240 248 255 AliceBlue -230 230 250 lavender -255 240 245 lavender blush -255 240 245 LavenderBlush -255 228 225 misty rose -255 228 225 MistyRose -255 255 255 white - 0 0 0 black - 47 79 79 dark slate gray - 47 79 79 DarkSlateGray - 47 79 79 dark slate grey - 47 79 79 DarkSlateGrey -105 105 105 dim gray -105 105 105 DimGray -105 105 105 dim grey -105 105 105 DimGrey -112 128 144 slate gray -112 128 144 SlateGray -112 128 144 slate grey -112 128 144 SlateGrey -119 136 153 light slate gray -119 136 153 LightSlateGray -119 136 153 light slate grey -119 136 153 LightSlateGrey -190 190 190 gray -190 190 190 grey -211 211 211 light grey -211 211 211 LightGrey -211 211 211 light gray -211 211 211 LightGray - 25 25 112 midnight blue - 25 25 112 MidnightBlue - 0 0 128 navy - 0 0 128 navy blue - 0 0 128 NavyBlue -100 149 237 cornflower blue -100 149 237 CornflowerBlue - 72 61 139 dark slate blue - 72 61 139 DarkSlateBlue -106 90 205 slate blue -106 90 205 SlateBlue -123 104 238 medium slate blue -123 104 238 MediumSlateBlue -132 112 255 light slate blue -132 112 255 LightSlateBlue - 0 0 205 medium blue - 0 0 205 MediumBlue - 65 105 225 royal blue - 65 105 225 RoyalBlue - 0 0 255 blue - 30 144 255 dodger blue - 30 144 255 DodgerBlue - 0 191 255 deep sky blue - 0 191 255 DeepSkyBlue -135 206 235 sky blue -135 206 235 SkyBlue -135 206 250 light sky blue -135 206 250 LightSkyBlue - 70 130 180 steel blue - 70 130 180 SteelBlue -176 196 222 light steel blue -176 196 222 LightSteelBlue -173 216 230 light blue -173 216 230 LightBlue -176 224 230 powder blue -176 224 230 PowderBlue -175 238 238 pale turquoise -175 238 238 PaleTurquoise - 0 206 209 dark turquoise - 0 206 209 DarkTurquoise - 72 209 204 medium turquoise - 72 209 204 MediumTurquoise - 64 224 208 turquoise - 0 255 255 cyan -224 255 255 light cyan -224 255 255 LightCyan - 95 158 160 cadet blue - 95 158 160 CadetBlue -102 205 170 medium aquamarine -102 205 170 MediumAquamarine -127 255 212 aquamarine - 0 100 0 dark green - 0 100 0 DarkGreen - 85 107 47 dark olive green - 85 107 47 DarkOliveGreen -143 188 143 dark sea green -143 188 143 DarkSeaGreen - 46 139 87 sea green - 46 139 87 SeaGreen - 60 179 113 medium sea green - 60 179 113 MediumSeaGreen - 32 178 170 light sea green - 32 178 170 LightSeaGreen -152 251 152 pale green -152 251 152 PaleGreen - 0 255 127 spring green - 0 255 127 SpringGreen -124 252 0 lawn green -124 252 0 LawnGreen - 0 255 0 green -127 255 0 chartreuse - 0 250 154 medium spring green - 0 250 154 MediumSpringGreen -173 255 47 green yellow -173 255 47 GreenYellow - 50 205 50 lime green - 50 205 50 LimeGreen -154 205 50 yellow green -154 205 50 YellowGreen - 34 139 34 forest green - 34 139 34 ForestGreen -107 142 35 olive drab -107 142 35 OliveDrab -189 183 107 dark khaki -189 183 107 DarkKhaki -240 230 140 khaki -238 232 170 pale goldenrod -238 232 170 PaleGoldenrod -250 250 210 light goldenrod yellow -250 250 210 LightGoldenrodYellow -255 255 224 light yellow -255 255 224 LightYellow -255 255 0 yellow -255 215 0 gold -238 221 130 light goldenrod -238 221 130 LightGoldenrod -218 165 32 goldenrod -184 134 11 dark goldenrod -184 134 11 DarkGoldenrod -188 143 143 rosy brown -188 143 143 RosyBrown -205 92 92 indian red -205 92 92 IndianRed -139 69 19 saddle brown -139 69 19 SaddleBrown -160 82 45 sienna -205 133 63 peru -222 184 135 burlywood -245 245 220 beige -245 222 179 wheat -244 164 96 sandy brown -244 164 96 SandyBrown -210 180 140 tan -210 105 30 chocolate -178 34 34 firebrick -165 42 42 brown -233 150 122 dark salmon -233 150 122 DarkSalmon -250 128 114 salmon -255 160 122 light salmon -255 160 122 LightSalmon -255 165 0 orange -255 140 0 dark orange -255 140 0 DarkOrange -255 127 80 coral -240 128 128 light coral -240 128 128 LightCoral -255 99 71 tomato -255 69 0 orange red -255 69 0 OrangeRed -255 0 0 red -255 105 180 hot pink -255 105 180 HotPink -255 20 147 deep pink -255 20 147 DeepPink -255 192 203 pink -255 182 193 light pink -255 182 193 LightPink -219 112 147 pale violet red -219 112 147 PaleVioletRed -176 48 96 maroon -199 21 133 medium violet red -199 21 133 MediumVioletRed -208 32 144 violet red -208 32 144 VioletRed -255 0 255 magenta -238 130 238 violet -221 160 221 plum -218 112 214 orchid -186 85 211 medium orchid -186 85 211 MediumOrchid -153 50 204 dark orchid -153 50 204 DarkOrchid -148 0 211 dark violet -148 0 211 DarkViolet -138 43 226 blue violet -138 43 226 BlueViolet -160 32 240 purple -147 112 219 medium purple -147 112 219 MediumPurple -216 191 216 thistle -255 250 250 snow1 -238 233 233 snow2 -205 201 201 snow3 -139 137 137 snow4 -255 245 238 seashell1 -238 229 222 seashell2 -205 197 191 seashell3 -139 134 130 seashell4 -255 239 219 AntiqueWhite1 -238 223 204 AntiqueWhite2 -205 192 176 AntiqueWhite3 -139 131 120 AntiqueWhite4 -255 228 196 bisque1 -238 213 183 bisque2 -205 183 158 bisque3 -139 125 107 bisque4 -255 218 185 PeachPuff1 -238 203 173 PeachPuff2 -205 175 149 PeachPuff3 -139 119 101 PeachPuff4 -255 222 173 NavajoWhite1 -238 207 161 NavajoWhite2 -205 179 139 NavajoWhite3 -139 121 94 NavajoWhite4 -255 250 205 LemonChiffon1 -238 233 191 LemonChiffon2 -205 201 165 LemonChiffon3 -139 137 112 LemonChiffon4 -255 248 220 cornsilk1 -238 232 205 cornsilk2 -205 200 177 cornsilk3 -139 136 120 cornsilk4 -255 255 240 ivory1 -238 238 224 ivory2 -205 205 193 ivory3 -139 139 131 ivory4 -240 255 240 honeydew1 -224 238 224 honeydew2 -193 205 193 honeydew3 -131 139 131 honeydew4 -255 240 245 LavenderBlush1 -238 224 229 LavenderBlush2 -205 193 197 LavenderBlush3 -139 131 134 LavenderBlush4 -255 228 225 MistyRose1 -238 213 210 MistyRose2 -205 183 181 MistyRose3 -139 125 123 MistyRose4 -240 255 255 azure1 -224 238 238 azure2 -193 205 205 azure3 -131 139 139 azure4 -131 111 255 SlateBlue1 -122 103 238 SlateBlue2 -105 89 205 SlateBlue3 - 71 60 139 SlateBlue4 - 72 118 255 RoyalBlue1 - 67 110 238 RoyalBlue2 - 58 95 205 RoyalBlue3 - 39 64 139 RoyalBlue4 - 0 0 255 blue1 - 0 0 238 blue2 - 0 0 205 blue3 - 0 0 139 blue4 - 30 144 255 DodgerBlue1 - 28 134 238 DodgerBlue2 - 24 116 205 DodgerBlue3 - 16 78 139 DodgerBlue4 - 99 184 255 SteelBlue1 - 92 172 238 SteelBlue2 - 79 148 205 SteelBlue3 - 54 100 139 SteelBlue4 - 0 191 255 DeepSkyBlue1 - 0 178 238 DeepSkyBlue2 - 0 154 205 DeepSkyBlue3 - 0 104 139 DeepSkyBlue4 -135 206 255 SkyBlue1 -126 192 238 SkyBlue2 -108 166 205 SkyBlue3 - 74 112 139 SkyBlue4 -176 226 255 LightSkyBlue1 -164 211 238 LightSkyBlue2 -141 182 205 LightSkyBlue3 - 96 123 139 LightSkyBlue4 -198 226 255 SlateGray1 -185 211 238 SlateGray2 -159 182 205 SlateGray3 -108 123 139 SlateGray4 -202 225 255 LightSteelBlue1 -188 210 238 LightSteelBlue2 -162 181 205 LightSteelBlue3 -110 123 139 LightSteelBlue4 -191 239 255 LightBlue1 -178 223 238 LightBlue2 -154 192 205 LightBlue3 -104 131 139 LightBlue4 -224 255 255 LightCyan1 -209 238 238 LightCyan2 -180 205 205 LightCyan3 -122 139 139 LightCyan4 -187 255 255 PaleTurquoise1 -174 238 238 PaleTurquoise2 -150 205 205 PaleTurquoise3 -102 139 139 PaleTurquoise4 -152 245 255 CadetBlue1 -142 229 238 CadetBlue2 -122 197 205 CadetBlue3 - 83 134 139 CadetBlue4 - 0 245 255 turquoise1 - 0 229 238 turquoise2 - 0 197 205 turquoise3 - 0 134 139 turquoise4 - 0 255 255 cyan1 - 0 238 238 cyan2 - 0 205 205 cyan3 - 0 139 139 cyan4 -151 255 255 DarkSlateGray1 -141 238 238 DarkSlateGray2 -121 205 205 DarkSlateGray3 - 82 139 139 DarkSlateGray4 -127 255 212 aquamarine1 -118 238 198 aquamarine2 -102 205 170 aquamarine3 - 69 139 116 aquamarine4 -193 255 193 DarkSeaGreen1 -180 238 180 DarkSeaGreen2 -155 205 155 DarkSeaGreen3 -105 139 105 DarkSeaGreen4 - 84 255 159 SeaGreen1 - 78 238 148 SeaGreen2 - 67 205 128 SeaGreen3 - 46 139 87 SeaGreen4 -154 255 154 PaleGreen1 -144 238 144 PaleGreen2 -124 205 124 PaleGreen3 - 84 139 84 PaleGreen4 - 0 255 127 SpringGreen1 - 0 238 118 SpringGreen2 - 0 205 102 SpringGreen3 - 0 139 69 SpringGreen4 - 0 255 0 green1 - 0 238 0 green2 - 0 205 0 green3 - 0 139 0 green4 -127 255 0 chartreuse1 -118 238 0 chartreuse2 -102 205 0 chartreuse3 - 69 139 0 chartreuse4 -192 255 62 OliveDrab1 -179 238 58 OliveDrab2 -154 205 50 OliveDrab3 -105 139 34 OliveDrab4 -202 255 112 DarkOliveGreen1 -188 238 104 DarkOliveGreen2 -162 205 90 DarkOliveGreen3 -110 139 61 DarkOliveGreen4 -255 246 143 khaki1 -238 230 133 khaki2 -205 198 115 khaki3 -139 134 78 khaki4 -255 236 139 LightGoldenrod1 -238 220 130 LightGoldenrod2 -205 190 112 LightGoldenrod3 -139 129 76 LightGoldenrod4 -255 255 224 LightYellow1 -238 238 209 LightYellow2 -205 205 180 LightYellow3 -139 139 122 LightYellow4 -255 255 0 yellow1 -238 238 0 yellow2 -205 205 0 yellow3 -139 139 0 yellow4 -255 215 0 gold1 -238 201 0 gold2 -205 173 0 gold3 -139 117 0 gold4 -255 193 37 goldenrod1 -238 180 34 goldenrod2 -205 155 29 goldenrod3 -139 105 20 goldenrod4 -255 185 15 DarkGoldenrod1 -238 173 14 DarkGoldenrod2 -205 149 12 DarkGoldenrod3 -139 101 8 DarkGoldenrod4 -255 193 193 RosyBrown1 -238 180 180 RosyBrown2 -205 155 155 RosyBrown3 -139 105 105 RosyBrown4 -255 106 106 IndianRed1 -238 99 99 IndianRed2 -205 85 85 IndianRed3 -139 58 58 IndianRed4 -255 130 71 sienna1 -238 121 66 sienna2 -205 104 57 sienna3 -139 71 38 sienna4 -255 211 155 burlywood1 -238 197 145 burlywood2 -205 170 125 burlywood3 -139 115 85 burlywood4 -255 231 186 wheat1 -238 216 174 wheat2 -205 186 150 wheat3 -139 126 102 wheat4 -255 165 79 tan1 -238 154 73 tan2 -205 133 63 tan3 -139 90 43 tan4 -255 127 36 chocolate1 -238 118 33 chocolate2 -205 102 29 chocolate3 -139 69 19 chocolate4 -255 48 48 firebrick1 -238 44 44 firebrick2 -205 38 38 firebrick3 -139 26 26 firebrick4 -255 64 64 brown1 -238 59 59 brown2 -205 51 51 brown3 -139 35 35 brown4 -255 140 105 salmon1 -238 130 98 salmon2 -205 112 84 salmon3 -139 76 57 salmon4 -255 160 122 LightSalmon1 -238 149 114 LightSalmon2 -205 129 98 LightSalmon3 -139 87 66 LightSalmon4 -255 165 0 orange1 -238 154 0 orange2 -205 133 0 orange3 -139 90 0 orange4 -255 127 0 DarkOrange1 -238 118 0 DarkOrange2 -205 102 0 DarkOrange3 -139 69 0 DarkOrange4 -255 114 86 coral1 -238 106 80 coral2 -205 91 69 coral3 -139 62 47 coral4 -255 99 71 tomato1 -238 92 66 tomato2 -205 79 57 tomato3 -139 54 38 tomato4 -255 69 0 OrangeRed1 -238 64 0 OrangeRed2 -205 55 0 OrangeRed3 -139 37 0 OrangeRed4 -255 0 0 red1 -238 0 0 red2 -205 0 0 red3 -139 0 0 red4 -255 20 147 DeepPink1 -238 18 137 DeepPink2 -205 16 118 DeepPink3 -139 10 80 DeepPink4 -255 110 180 HotPink1 -238 106 167 HotPink2 -205 96 144 HotPink3 -139 58 98 HotPink4 -255 181 197 pink1 -238 169 184 pink2 -205 145 158 pink3 -139 99 108 pink4 -255 174 185 LightPink1 -238 162 173 LightPink2 -205 140 149 LightPink3 -139 95 101 LightPink4 -255 130 171 PaleVioletRed1 -238 121 159 PaleVioletRed2 -205 104 137 PaleVioletRed3 -139 71 93 PaleVioletRed4 -255 52 179 maroon1 -238 48 167 maroon2 -205 41 144 maroon3 -139 28 98 maroon4 -255 62 150 VioletRed1 -238 58 140 VioletRed2 -205 50 120 VioletRed3 -139 34 82 VioletRed4 -255 0 255 magenta1 -238 0 238 magenta2 -205 0 205 magenta3 -139 0 139 magenta4 -255 131 250 orchid1 -238 122 233 orchid2 -205 105 201 orchid3 -139 71 137 orchid4 -255 187 255 plum1 -238 174 238 plum2 -205 150 205 plum3 -139 102 139 plum4 -224 102 255 MediumOrchid1 -209 95 238 MediumOrchid2 -180 82 205 MediumOrchid3 -122 55 139 MediumOrchid4 -191 62 255 DarkOrchid1 -178 58 238 DarkOrchid2 -154 50 205 DarkOrchid3 -104 34 139 DarkOrchid4 -155 48 255 purple1 -145 44 238 purple2 -125 38 205 purple3 - 85 26 139 purple4 -171 130 255 MediumPurple1 -159 121 238 MediumPurple2 -137 104 205 MediumPurple3 - 93 71 139 MediumPurple4 -255 225 255 thistle1 -238 210 238 thistle2 -205 181 205 thistle3 -139 123 139 thistle4 - 0 0 0 gray0 - 0 0 0 grey0 - 3 3 3 gray1 - 3 3 3 grey1 - 5 5 5 gray2 - 5 5 5 grey2 - 8 8 8 gray3 - 8 8 8 grey3 - 10 10 10 gray4 - 10 10 10 grey4 - 13 13 13 gray5 - 13 13 13 grey5 - 15 15 15 gray6 - 15 15 15 grey6 - 18 18 18 gray7 - 18 18 18 grey7 - 20 20 20 gray8 - 20 20 20 grey8 - 23 23 23 gray9 - 23 23 23 grey9 - 26 26 26 gray10 - 26 26 26 grey10 - 28 28 28 gray11 - 28 28 28 grey11 - 31 31 31 gray12 - 31 31 31 grey12 - 33 33 33 gray13 - 33 33 33 grey13 - 36 36 36 gray14 - 36 36 36 grey14 - 38 38 38 gray15 - 38 38 38 grey15 - 41 41 41 gray16 - 41 41 41 grey16 - 43 43 43 gray17 - 43 43 43 grey17 - 46 46 46 gray18 - 46 46 46 grey18 - 48 48 48 gray19 - 48 48 48 grey19 - 51 51 51 gray20 - 51 51 51 grey20 - 54 54 54 gray21 - 54 54 54 grey21 - 56 56 56 gray22 - 56 56 56 grey22 - 59 59 59 gray23 - 59 59 59 grey23 - 61 61 61 gray24 - 61 61 61 grey24 - 64 64 64 gray25 - 64 64 64 grey25 - 66 66 66 gray26 - 66 66 66 grey26 - 69 69 69 gray27 - 69 69 69 grey27 - 71 71 71 gray28 - 71 71 71 grey28 - 74 74 74 gray29 - 74 74 74 grey29 - 77 77 77 gray30 - 77 77 77 grey30 - 79 79 79 gray31 - 79 79 79 grey31 - 82 82 82 gray32 - 82 82 82 grey32 - 84 84 84 gray33 - 84 84 84 grey33 - 87 87 87 gray34 - 87 87 87 grey34 - 89 89 89 gray35 - 89 89 89 grey35 - 92 92 92 gray36 - 92 92 92 grey36 - 94 94 94 gray37 - 94 94 94 grey37 - 97 97 97 gray38 - 97 97 97 grey38 - 99 99 99 gray39 - 99 99 99 grey39 -102 102 102 gray40 -102 102 102 grey40 -105 105 105 gray41 -105 105 105 grey41 -107 107 107 gray42 -107 107 107 grey42 -110 110 110 gray43 -110 110 110 grey43 -112 112 112 gray44 -112 112 112 grey44 -115 115 115 gray45 -115 115 115 grey45 -117 117 117 gray46 -117 117 117 grey46 -120 120 120 gray47 -120 120 120 grey47 -122 122 122 gray48 -122 122 122 grey48 -125 125 125 gray49 -125 125 125 grey49 -127 127 127 gray50 -127 127 127 grey50 -130 130 130 gray51 -130 130 130 grey51 -133 133 133 gray52 -133 133 133 grey52 -135 135 135 gray53 -135 135 135 grey53 -138 138 138 gray54 -138 138 138 grey54 -140 140 140 gray55 -140 140 140 grey55 -143 143 143 gray56 -143 143 143 grey56 -145 145 145 gray57 -145 145 145 grey57 -148 148 148 gray58 -148 148 148 grey58 -150 150 150 gray59 -150 150 150 grey59 -153 153 153 gray60 -153 153 153 grey60 -156 156 156 gray61 -156 156 156 grey61 -158 158 158 gray62 -158 158 158 grey62 -161 161 161 gray63 -161 161 161 grey63 -163 163 163 gray64 -163 163 163 grey64 -166 166 166 gray65 -166 166 166 grey65 -168 168 168 gray66 -168 168 168 grey66 -171 171 171 gray67 -171 171 171 grey67 -173 173 173 gray68 -173 173 173 grey68 -176 176 176 gray69 -176 176 176 grey69 -179 179 179 gray70 -179 179 179 grey70 -181 181 181 gray71 -181 181 181 grey71 -184 184 184 gray72 -184 184 184 grey72 -186 186 186 gray73 -186 186 186 grey73 -189 189 189 gray74 -189 189 189 grey74 -191 191 191 gray75 -191 191 191 grey75 -194 194 194 gray76 -194 194 194 grey76 -196 196 196 gray77 -196 196 196 grey77 -199 199 199 gray78 -199 199 199 grey78 -201 201 201 gray79 -201 201 201 grey79 -204 204 204 gray80 -204 204 204 grey80 -207 207 207 gray81 -207 207 207 grey81 -209 209 209 gray82 -209 209 209 grey82 -212 212 212 gray83 -212 212 212 grey83 -214 214 214 gray84 -214 214 214 grey84 -217 217 217 gray85 -217 217 217 grey85 -219 219 219 gray86 -219 219 219 grey86 -222 222 222 gray87 -222 222 222 grey87 -224 224 224 gray88 -224 224 224 grey88 -227 227 227 gray89 -227 227 227 grey89 -229 229 229 gray90 -229 229 229 grey90 -232 232 232 gray91 -232 232 232 grey91 -235 235 235 gray92 -235 235 235 grey92 -237 237 237 gray93 -237 237 237 grey93 -240 240 240 gray94 -240 240 240 grey94 -242 242 242 gray95 -242 242 242 grey95 -245 245 245 gray96 -245 245 245 grey96 -247 247 247 gray97 -247 247 247 grey97 -250 250 250 gray98 -250 250 250 grey98 -252 252 252 gray99 -252 252 252 grey99 -255 255 255 gray100 -255 255 255 grey100 -169 169 169 dark grey -169 169 169 DarkGrey -169 169 169 dark gray -169 169 169 DarkGray -0 0 139 dark blue -0 0 139 DarkBlue -0 139 139 dark cyan -0 139 139 DarkCyan -139 0 139 dark magenta -139 0 139 DarkMagenta -139 0 0 dark red -139 0 0 DarkRed -144 238 144 light green -144 238 144 LightGreen diff --git a/Tools/pynche/X/xlicense.txt b/Tools/pynche/X/xlicense.txt deleted file mode 100644 index b4471db6753b4..0000000000000 --- a/Tools/pynche/X/xlicense.txt +++ /dev/null @@ -1,29 +0,0 @@ -X Window System License - X11R6.4 - -Copyright (c) 1998 The Open Group - -Permission is hereby granted, free of charge, to any person obtaining -a copy of this software and associated documentation files (the -"Software"), to deal in the Software without restriction, including -without limitation the rights to use, copy, modify, merge, publish, -distribute, sublicense, and/or sell copies of the Software, and to -permit persons to whom the Software is furnished to do so, subject to -the following conditions: - -The above copyright notice and this permission notice shall be -included in all copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, -EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF -MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. -IN NO EVENT SHALL THE OPEN GROUP BE LIABLE FOR ANY CLAIM, DAMAGES OR -OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, -ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR -OTHER DEALINGS IN THE SOFTWARE. - -Except as contained in this notice, the name of The Open Group shall -not be used in advertising or otherwise to promote the sale, use or -other dealings in this Software without prior written authorization -from The Open Group. - -X Window System is a trademark of The Open Group diff --git a/Tools/pynche/__init__.py b/Tools/pynche/__init__.py deleted file mode 100644 index b93054b3ecf3a..0000000000000 --- a/Tools/pynche/__init__.py +++ /dev/null @@ -1 +0,0 @@ -# Dummy file to make this directory a package. diff --git a/Tools/pynche/html40colors.txt b/Tools/pynche/html40colors.txt deleted file mode 100644 index 3eeb0a10a89f9..0000000000000 --- a/Tools/pynche/html40colors.txt +++ /dev/null @@ -1,17 +0,0 @@ -# HTML 4.0 color names -Black #000000 -Silver #c0c0c0 -Gray #808080 -White #ffffff -Maroon #800000 -Red #ff0000 -Purple #800080 -Fuchsia #ff00ff -Green #008000 -Lime #00ff00 -Olive #808000 -Yellow #ffff00 -Navy #000080 -Blue #0000ff -Teal #008080 -Aqua #00ffff diff --git a/Tools/pynche/namedcolors.txt b/Tools/pynche/namedcolors.txt deleted file mode 100644 index 3658e85f25688..0000000000000 --- a/Tools/pynche/namedcolors.txt +++ /dev/null @@ -1,100 +0,0 @@ -# named colors from http://www.lightlink.com/xine/bells/namedcolors.html -White #FFFFFF -Red #FF0000 -Green #00FF00 -Blue #0000FF -Magenta #FF00FF -Cyan #00FFFF -Yellow #FFFF00 -Black #000000 -Aquamarine #70DB93 -Baker's Chocolate #5C3317 -Blue Violet #9F5F9F -Brass #B5A642 -Bright Gold #D9D919 -Brown #A62A2A -Bronze #8C7853 -Bronze II #A67D3D -Cadet Blue #5F9F9F -Cool Copper #D98719 -Copper #B87333 -Coral #FF7F00 -Corn Flower Blue #42426F -Dark Brown #5C4033 -Dark Green #2F4F2F -Dark Green Copper #4A766E -Dark Olive Green #4F4F2F -Dark Orchid #9932CD -Dark Purple #871F78 -Dark Slate Blue #6B238E -Dark Slate Grey #2F4F4F -Dark Tan #97694F -Dark Turquoise #7093DB -Dark Wood #855E42 -Dim Grey #545454 -Dusty Rose #856363 -Feldspar #D19275 -Firebrick #8E2323 -Forest Green #238E23 -Gold #CD7F32 -Goldenrod #DBDB70 -Grey #C0C0C0 -Green Copper #527F76 -Green Yellow #93DB70 -Hunter Green #215E21 -Indian Red #4E2F2F -Khaki #9F9F5F -Light Blue #C0D9D9 -Light Grey #A8A8A8 -Light Steel Blue #8F8FBD -Light Wood #E9C2A6 -Lime Green #32CD32 -Mandarian Orange #E47833 -Maroon #8E236B -Medium Aquamarine #32CD99 -Medium Blue #3232CD -Medium Forest Green #6B8E23 -Medium Goldenrod #EAEAAE -Medium Orchid #9370DB -Medium Sea Green #426F42 -Medium Slate Blue #7F00FF -Medium Spring Green #7FFF00 -Medium Turquoise #70DBDB -Medium Violet Red #DB7093 -Medium Wood #A68064 -Midnight Blue #2F2F4F -Navy Blue #23238E -Neon Blue #4D4DFF -Neon Pink #FF6EC7 -New Midnight Blue #00009C -New Tan #EBC79E -Old Gold #CFB53B -Orange #FF7F00 -Orange Red #FF2400 -Orchid #DB70DB -Pale Green #8FBC8F -Pink #BC8F8F -Plum #EAADEA -Quartz #D9D9F3 -Rich Blue #5959AB -Salmon #6F4242 -Scarlet #8C1717 -Sea Green #238E68 -Semi-Sweet Chocolate #6B4226 -Sienna #8E6B23 -Silver #E6E8FA -Sky Blue #3299CC -Slate Blue #007FFF -Spicy Pink #FF1CAE -Spring Green #00FF7F -Steel Blue #236B8E -Summer Sky #38B0DE -Tan #DB9370 -Thistle #D8BFD8 -Turquoise #ADEAEA -Very Dark Brown #5C4033 -Very Light Grey #CDCDCD -Violet #4F2F4F -Violet Red #CC3299 -Wheat #D8D8BF -Yellow Green #99CC32 diff --git a/Tools/pynche/pyColorChooser.py b/Tools/pynche/pyColorChooser.py deleted file mode 100644 index 3286047a5fc8b..0000000000000 --- a/Tools/pynche/pyColorChooser.py +++ /dev/null @@ -1,125 +0,0 @@ -"""Color chooser implementing (almost) the tkColorColor interface -""" - -import os -import Main -import ColorDB - - - -class Chooser: - """Ask for a color""" - def __init__(self, - master = None, - databasefile = None, - initfile = None, - ignore = None, - wantspec = None): - self.__master = master - self.__databasefile = databasefile - self.__initfile = initfile or os.path.expanduser('~/.pynche') - self.__ignore = ignore - self.__pw = None - self.__wantspec = wantspec - - def show(self, color, options): - # scan for options that can override the ctor options - self.__wantspec = options.get('wantspec', self.__wantspec) - dbfile = options.get('databasefile', self.__databasefile) - # load the database file - colordb = None - if dbfile != self.__databasefile: - colordb = ColorDB.get_colordb(dbfile) - if not self.__master: - from tkinter import Tk - self.__master = Tk() - if not self.__pw: - self.__pw, self.__sb = \ - Main.build(master = self.__master, - initfile = self.__initfile, - ignore = self.__ignore) - else: - self.__pw.deiconify() - # convert color - if colordb: - self.__sb.set_colordb(colordb) - else: - colordb = self.__sb.colordb() - if color: - r, g, b = Main.initial_color(color, colordb) - self.__sb.update_views(r, g, b) - # reset the canceled flag and run it - self.__sb.canceled(0) - Main.run(self.__pw, self.__sb) - rgbtuple = self.__sb.current_rgb() - self.__pw.withdraw() - # check to see if the cancel button was pushed - if self.__sb.canceled_p(): - return None, None - # Try to return the color name from the database if there is an exact - # match, otherwise use the "#rrggbb" spec. BAW: Forget about color - # aliases for now, maybe later we should return these too. - name = None - if not self.__wantspec: - try: - name = colordb.find_byrgb(rgbtuple)[0] - except ColorDB.BadColor: - pass - if name is None: - name = ColorDB.triplet_to_rrggbb(rgbtuple) - return rgbtuple, name - - def save(self): - if self.__sb: - self.__sb.save_views() - - -# convenience stuff -_chooser = None - -def askcolor(color = None, **options): - """Ask for a color""" - global _chooser - if not _chooser: - _chooser = Chooser(**options) - return _chooser.show(color, options) - -def save(): - global _chooser - if _chooser: - _chooser.save() - - -# test stuff -if __name__ == '__main__': - from tkinter import * - - class Tester: - def __init__(self): - self.__root = tk = Tk() - b = Button(tk, text='Choose Color...', command=self.__choose) - b.pack() - self.__l = Label(tk) - self.__l.pack() - q = Button(tk, text='Quit', command=self.__quit) - q.pack() - - def __choose(self, event=None): - rgb, name = askcolor(master=self.__root) - if rgb is None: - text = 'You hit CANCEL!' - else: - r, g, b = rgb - text = 'You picked %s (%3d/%3d/%3d)' % (name, r, g, b) - self.__l.configure(text=text) - - def __quit(self, event=None): - self.__root.quit() - - def run(self): - self.__root.mainloop() - t = Tester() - t.run() - # simpler -## print 'color:', askcolor() -## print 'color:', askcolor() diff --git a/Tools/pynche/pynche b/Tools/pynche/pynche deleted file mode 100755 index 64bf70335db2f..0000000000000 --- a/Tools/pynche/pynche +++ /dev/null @@ -1,7 +0,0 @@ -#! /usr/bin/env python - -"""Run this file under Unix, or when debugging under Windows. -Run the file pynche.pyw under Windows to inhibit the console window. -""" -import Main -Main.main() diff --git a/Tools/pynche/pynche.pyw b/Tools/pynche/pynche.pyw deleted file mode 100755 index 6dfc8fed715fe..0000000000000 --- a/Tools/pynche/pynche.pyw +++ /dev/null @@ -1,7 +0,0 @@ -#! /usr/bin/env python - -"""Run this file under Windows to inhibit the console window. -Run the file pynche.py under Unix or when debugging under Windows. -""" -import Main -Main.main() diff --git a/Tools/pynche/webcolors.txt b/Tools/pynche/webcolors.txt deleted file mode 100644 index f645c1e610642..0000000000000 --- a/Tools/pynche/webcolors.txt +++ /dev/null @@ -1,141 +0,0 @@ -# De-facto NS & MSIE recognized HTML color names -AliceBlue #f0f8ff -AntiqueWhite #faebd7 -Aqua #00ffff -Aquamarine #7fffd4 -Azure #f0ffff -Beige #f5f5dc -Bisque #ffe4c4 -Black #000000 -BlanchedAlmond #ffebcd -Blue #0000ff -BlueViolet #8a2be2 -Brown #a52a2a -BurlyWood #deb887 -CadetBlue #5f9ea0 -Chartreuse #7fff00 -Chocolate #d2691e -Coral #ff7f50 -CornflowerBlue #6495ed -Cornsilk #fff8dc -Crimson #dc143c -Cyan #00ffff -DarkBlue #00008b -DarkCyan #008b8b -DarkGoldenrod #b8860b -DarkGray #a9a9a9 -DarkGreen #006400 -DarkKhaki #bdb76b -DarkMagenta #8b008b -DarkOliveGreen #556b2f -DarkOrange #ff8c00 -DarkOrchid #9932cc -DarkRed #8b0000 -DarkSalmon #e9967a -DarkSeaGreen #8fbc8f -DarkSlateBlue #483d8b -DarkSlateGray #2f4f4f -DarkTurquoise #00ced1 -DarkViolet #9400d3 -DeepPink #ff1493 -DeepSkyBlue #00bfff -DimGray #696969 -DodgerBlue #1e90ff -FireBrick #b22222 -FloralWhite #fffaf0 -ForestGreen #228b22 -Fuchsia #ff00ff -Gainsboro #dcdcdc -GhostWhite #f8f8ff -Gold #ffd700 -Goldenrod #daa520 -Gray #808080 -Green #008000 -GreenYellow #adff2f -Honeydew #f0fff0 -HotPink #ff69b4 -IndianRed #cd5c5c -Indigo #4b0082 -Ivory #fffff0 -Khaki #f0e68c -Lavender #e6e6fa -LavenderBlush #fff0f5 -LawnGreen #7cfc00 -LemonChiffon #fffacd -LightBlue #add8e6 -LightCoral #f08080 -LightCyan #e0ffff -LightGoldenrodYellow #fafad2 -LightGreen #90ee90 -LightGrey #d3d3d3 -LightPink #ffb6c1 -LightSalmon #ffa07a -LightSeaGreen #20b2aa -LightSkyBlue #87cefa -LightSlateGray #778899 -LightSteelBlue #b0c4de -LightYellow #ffffe0 -Lime #00ff00 -LimeGreen #32cd32 -Linen #faf0e6 -Magenta #ff00ff -Maroon #800000 -MediumAquamarine #66cdaa -MediumBlue #0000cd -MediumOrchid #ba55d3 -MediumPurple #9370db -MediumSeaGreen #3cb371 -MediumSlateBlue #7b68ee -MediumSpringGreen #00fa9a -MediumTurquoise #48d1cc -MediumVioletRed #c71585 -MidnightBlue #191970 -MintCream #f5fffa -MistyRose #ffe4e1 -Moccasin #ffe4b5 -NavajoWhite #ffdead -Navy #000080 -OldLace #fdf5e6 -Olive #808000 -OliveDrab #6b8e23 -Orange #ffa500 -OrangeRed #ff4500 -Orchid #da70d6 -PaleGoldenrod #eee8aa -PaleGreen #98fb98 -PaleTurquoise #afeeee -PaleVioletRed #db7093 -PapayaWhip #ffefd5 -PeachPuff #ffdab9 -Peru #cd853f -Pink #ffc0cb -Plum #dda0dd -PowderBlue #b0e0e6 -Purple #800080 -Red #ff0000 -RosyBrown #bc8f8f -RoyalBlue #4169e1 -SaddleBrown #8b4513 -Salmon #fa8072 -SandyBrown #f4a460 -SeaGreen #2e8b57 -Seashell #fff5ee -Sienna #a0522d -Silver #c0c0c0 -SkyBlue #87ceeb -SlateBlue #6a5acd -SlateGray #708090 -Snow #fffafa -SpringGreen #00ff7f -SteelBlue #4682b4 -Tan #d2b48c -Teal #008080 -Thistle #d8bfd8 -Tomato #ff6347 -Turquoise #40e0d0 -Violet #ee82ee -Wheat #f5deb3 -White #ffffff -WhiteSmoke #f5f5f5 -Yellow #ffff00 -YellowGreen #9acd32 diff --git a/Tools/pynche/websafe.txt b/Tools/pynche/websafe.txt deleted file mode 100644 index 70ed51e683243..0000000000000 --- a/Tools/pynche/websafe.txt +++ /dev/null @@ -1,217 +0,0 @@ -# Websafe RGB values -#000000 -#000033 -#000066 -#000099 -#0000cc -#0000ff -#003300 -#003333 -#003366 -#003399 -#0033cc -#0033ff -#006600 -#006633 -#006666 -#006699 -#0066cc -#0066ff -#009900 -#009933 -#009966 -#009999 -#0099cc -#0099ff -#00cc00 -#00cc33 -#00cc66 -#00cc99 -#00cccc -#00ccff -#00ff00 -#00ff33 -#00ff66 -#00ff99 -#00ffcc -#00ffff -#330000 -#330033 -#330066 -#330099 -#3300cc -#3300ff -#333300 -#333333 -#333366 -#333399 -#3333cc -#3333ff -#336600 -#336633 -#336666 -#336699 -#3366cc -#3366ff -#339900 -#339933 -#339966 -#339999 -#3399cc -#3399ff -#33cc00 -#33cc33 -#33cc66 -#33cc99 -#33cccc -#33ccff -#33ff00 -#33ff33 -#33ff66 -#33ff99 -#33ffcc -#33ffff -#660000 -#660033 -#660066 -#660099 -#6600cc -#6600ff -#663300 -#663333 -#663366 -#663399 -#6633cc -#6633ff -#666600 -#666633 -#666666 -#666699 -#6666cc -#6666ff -#669900 -#669933 -#669966 -#669999 -#6699cc -#6699ff -#66cc00 -#66cc33 -#66cc66 -#66cc99 -#66cccc -#66ccff -#66ff00 -#66ff33 -#66ff66 -#66ff99 -#66ffcc -#66ffff -#990000 -#990033 -#990066 -#990099 -#9900cc -#9900ff -#993300 -#993333 -#993366 -#993399 -#9933cc -#9933ff -#996600 -#996633 -#996666 -#996699 -#9966cc -#9966ff -#999900 -#999933 -#999966 -#999999 -#9999cc -#9999ff -#99cc00 -#99cc33 -#99cc66 -#99cc99 -#99cccc -#99ccff -#99ff00 -#99ff33 -#99ff66 -#99ff99 -#99ffcc -#99ffff -#cc0000 -#cc0033 -#cc0066 -#cc0099 -#cc00cc -#cc00ff -#cc3300 -#cc3333 -#cc3366 -#cc3399 -#cc33cc -#cc33ff -#cc6600 -#cc6633 -#cc6666 -#cc6699 -#cc66cc -#cc66ff -#cc9900 -#cc9933 -#cc9966 -#cc9999 -#cc99cc -#cc99ff -#cccc00 -#cccc33 -#cccc66 -#cccc99 -#cccccc -#ccccff -#ccff00 -#ccff33 -#ccff66 -#ccff99 -#ccffcc -#ccffff -#ff0000 -#ff0033 -#ff0066 -#ff0099 -#ff00cc -#ff00ff -#ff3300 -#ff3333 -#ff3366 -#ff3399 -#ff33cc -#ff33ff -#ff6600 -#ff6633 -#ff6666 -#ff6699 -#ff66cc -#ff66ff -#ff9900 -#ff9933 -#ff9966 -#ff9999 -#ff99cc -#ff99ff -#ffcc00 -#ffcc33 -#ffcc66 -#ffcc99 -#ffcccc -#ffccff -#ffff00 -#ffff33 -#ffff66 -#ffff99 -#ffffcc -#ffffff From webhook-mailer at python.org Sun Apr 17 21:53:55 2022 From: webhook-mailer at python.org (facundobatista) Date: Mon, 18 Apr 2022 01:53:55 -0000 Subject: [Python-checkins] gh-85567: Register a cleanup function to close files for FileType objects in argparse (#32257) Message-ID: https://github.com/python/cpython/commit/328dbc051f84bd5fdf61101bb4fa61d85f8b7feb commit: 328dbc051f84bd5fdf61101bb4fa61d85f8b7feb branch: main author: achhina committer: facundobatista date: 2022-04-17T22:53:37-03:00 summary: gh-85567: Register a cleanup function to close files for FileType objects in argparse (#32257) * bpo-41395: Register a cleanup function to close files for FileType objects in argparse * Added import as top level import, and renamed file as fh. files: A Misc/NEWS.d/next/Library/2022-04-02-14-40-53.bpo-41395.Y1ZVvT.rst M Lib/argparse.py M Misc/ACKS diff --git a/Lib/argparse.py b/Lib/argparse.py index 429a72ab7841e..881dfda6d4d93 100644 --- a/Lib/argparse.py +++ b/Lib/argparse.py @@ -84,7 +84,7 @@ 'ZERO_OR_MORE', ] - +import atexit as _atexit import os as _os import re as _re import sys as _sys @@ -1268,8 +1268,12 @@ def __call__(self, string): # all other arguments are used as file names try: - return open(string, self._mode, self._bufsize, self._encoding, - self._errors) + fh = open(string, self._mode, self._bufsize, self._encoding, self._errors) + + # Register cleanup function to close file + _atexit.register(fh.close) + + return fh except OSError as e: args = {'filename': string, 'error': e} message = _("can't open '%(filename)s': %(error)s") diff --git a/Misc/ACKS b/Misc/ACKS index 5e66a2e757adf..a1df84c0d6779 100644 --- a/Misc/ACKS +++ b/Misc/ACKS @@ -313,6 +313,7 @@ Nicolas Chauvat Jerry Chen Michael Chermside Ingrid Cheung +Adam Chhina Terry Chia Albert Chin-A-Young Adal Chiriliuc diff --git a/Misc/NEWS.d/next/Library/2022-04-02-14-40-53.bpo-41395.Y1ZVvT.rst b/Misc/NEWS.d/next/Library/2022-04-02-14-40-53.bpo-41395.Y1ZVvT.rst new file mode 100644 index 0000000000000..5358b0e71715e --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-02-14-40-53.bpo-41395.Y1ZVvT.rst @@ -0,0 +1,3 @@ +FileType objects from argparse may not be closed and lead to +ResourceWarning. Register a file.close function with atexit for FileType +objects to ensure they are closed. Patch Contributed by Adam Chhina. From webhook-mailer at python.org Sun Apr 17 23:39:51 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Mon, 18 Apr 2022 03:39:51 -0000 Subject: [Python-checkins] gh-91118: Fix docstrings that do not honor --without-doc-strings (#31769) Message-ID: https://github.com/python/cpython/commit/a573cb2fec664c645ab744658d7e941d72e1a398 commit: a573cb2fec664c645ab744658d7e941d72e1a398 branch: main author: Oleg Iarygin committer: JelleZijlstra date: 2022-04-17T20:39:32-07:00 summary: gh-91118: Fix docstrings that do not honor --without-doc-strings (#31769) Co-authored-by: ?ric Co-authored-by: Jelle Zijlstra files: A Misc/NEWS.d/next/Core and Builtins/2022-03-08-21-59-57.bpo-46962.UomDfz.rst A Misc/NEWS.d/next/Documentation/2022-03-08-22-10-38.bpo-46962.FIVe9I.rst M Doc/c-api/typeobj.rst M Doc/extending/newtypes_tutorial.rst M Doc/includes/custom.c M Doc/includes/custom2.c M Doc/includes/custom3.c M Doc/includes/custom4.c M Doc/includes/sublist.c M Modules/_ctypes/_ctypes.c M Modules/_ctypes/callbacks.c M Modules/_ctypes/callproc.c M Modules/_ctypes/cfield.c M Modules/_testcapimodule.c M Objects/genericaliasobject.c M Objects/picklebufobject.c M Objects/unionobject.c diff --git a/Doc/c-api/typeobj.rst b/Doc/c-api/typeobj.rst index 4a54139c5d498..ed434d8fd44fb 100644 --- a/Doc/c-api/typeobj.rst +++ b/Doc/c-api/typeobj.rst @@ -2597,7 +2597,7 @@ A basic :ref:`static type `:: PyVarObject_HEAD_INIT(NULL, 0) .tp_name = "mymod.MyObject", .tp_basicsize = sizeof(MyObject), - .tp_doc = "My objects", + .tp_doc = PyDoc_STR("My objects"), .tp_new = myobj_new, .tp_dealloc = (destructor)myobj_dealloc, .tp_repr = (reprfunc)myobj_repr, @@ -2627,7 +2627,7 @@ with a more verbose initializer:: 0, /* tp_setattro */ 0, /* tp_as_buffer */ 0, /* tp_flags */ - "My objects", /* tp_doc */ + PyDoc_STR("My objects"), /* tp_doc */ 0, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ @@ -2660,7 +2660,7 @@ A type that supports weakrefs, instance dicts, and hashing:: PyVarObject_HEAD_INIT(NULL, 0) .tp_name = "mymod.MyObject", .tp_basicsize = sizeof(MyObject), - .tp_doc = "My objects", + .tp_doc = PyDoc_STR("My objects"), .tp_weaklistoffset = offsetof(MyObject, weakreflist), .tp_dictoffset = offsetof(MyObject, inst_dict), .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HAVE_GC, @@ -2688,7 +2688,7 @@ to create instances (e.g. uses a separate factory func) using .tp_name = "mymod.MyStr", .tp_basicsize = sizeof(MyStr), .tp_base = NULL, // set to &PyUnicode_Type in module init - .tp_doc = "my custom str", + .tp_doc = PyDoc_STR("my custom str"), .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_DISALLOW_INSTANTIATION, .tp_repr = (reprfunc)myobj_repr, }; diff --git a/Doc/extending/newtypes_tutorial.rst b/Doc/extending/newtypes_tutorial.rst index 904915306f1f3..34c25d1f6f199 100644 --- a/Doc/extending/newtypes_tutorial.rst +++ b/Doc/extending/newtypes_tutorial.rst @@ -90,7 +90,7 @@ The second bit is the definition of the type object. :: static PyTypeObject CustomType = { PyVarObject_HEAD_INIT(NULL, 0) .tp_name = "custom.Custom", - .tp_doc = "Custom objects", + .tp_doc = PyDoc_STR("Custom objects"), .tp_basicsize = sizeof(CustomObject), .tp_itemsize = 0, .tp_flags = Py_TPFLAGS_DEFAULT, @@ -161,7 +161,7 @@ you will need to OR the corresponding flags. We provide a doc string for the type in :c:member:`~PyTypeObject.tp_doc`. :: - .tp_doc = "Custom objects", + .tp_doc = PyDoc_STR("Custom objects"), To enable object creation, we have to provide a :c:member:`~PyTypeObject.tp_new` handler. This is the equivalent of the Python method :meth:`__new__`, but diff --git a/Doc/includes/custom.c b/Doc/includes/custom.c index f361baf830dd1..26ca754964733 100644 --- a/Doc/includes/custom.c +++ b/Doc/includes/custom.c @@ -9,7 +9,7 @@ typedef struct { static PyTypeObject CustomType = { PyVarObject_HEAD_INIT(NULL, 0) .tp_name = "custom.Custom", - .tp_doc = "Custom objects", + .tp_doc = PyDoc_STR("Custom objects"), .tp_basicsize = sizeof(CustomObject), .tp_itemsize = 0, .tp_flags = Py_TPFLAGS_DEFAULT, diff --git a/Doc/includes/custom2.c b/Doc/includes/custom2.c index 5bacab7a2a971..2a3c59f8f04c3 100644 --- a/Doc/includes/custom2.c +++ b/Doc/includes/custom2.c @@ -98,7 +98,7 @@ static PyMethodDef Custom_methods[] = { static PyTypeObject CustomType = { PyVarObject_HEAD_INIT(NULL, 0) .tp_name = "custom2.Custom", - .tp_doc = "Custom objects", + .tp_doc = PyDoc_STR("Custom objects"), .tp_basicsize = sizeof(CustomObject), .tp_itemsize = 0, .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, diff --git a/Doc/includes/custom3.c b/Doc/includes/custom3.c index 2b7a99ecf96c7..5a47530f0a6b0 100644 --- a/Doc/includes/custom3.c +++ b/Doc/includes/custom3.c @@ -148,7 +148,7 @@ static PyMethodDef Custom_methods[] = { static PyTypeObject CustomType = { PyVarObject_HEAD_INIT(NULL, 0) .tp_name = "custom3.Custom", - .tp_doc = "Custom objects", + .tp_doc = PyDoc_STR("Custom objects"), .tp_basicsize = sizeof(CustomObject), .tp_itemsize = 0, .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, diff --git a/Doc/includes/custom4.c b/Doc/includes/custom4.c index 584992fc5f1a8..c7ee55578488e 100644 --- a/Doc/includes/custom4.c +++ b/Doc/includes/custom4.c @@ -160,7 +160,7 @@ static PyMethodDef Custom_methods[] = { static PyTypeObject CustomType = { PyVarObject_HEAD_INIT(NULL, 0) .tp_name = "custom4.Custom", - .tp_doc = "Custom objects", + .tp_doc = PyDoc_STR("Custom objects"), .tp_basicsize = sizeof(CustomObject), .tp_itemsize = 0, .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HAVE_GC, diff --git a/Doc/includes/sublist.c b/Doc/includes/sublist.c index b2c26e73ebaf7..b36dadf07eae8 100644 --- a/Doc/includes/sublist.c +++ b/Doc/includes/sublist.c @@ -31,7 +31,7 @@ SubList_init(SubListObject *self, PyObject *args, PyObject *kwds) static PyTypeObject SubListType = { PyVarObject_HEAD_INIT(NULL, 0) .tp_name = "sublist.SubList", - .tp_doc = "SubList objects", + .tp_doc = PyDoc_STR("SubList objects"), .tp_basicsize = sizeof(SubListObject), .tp_itemsize = 0, .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-03-08-21-59-57.bpo-46962.UomDfz.rst b/Misc/NEWS.d/next/Core and Builtins/2022-03-08-21-59-57.bpo-46962.UomDfz.rst new file mode 100644 index 0000000000000..395c9b3d8f526 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2022-03-08-21-59-57.bpo-46962.UomDfz.rst @@ -0,0 +1,10 @@ +Classes and functions that unconditionally declared their docstrings +ignoring the `--without-doc-strings` compilation flag no longer do so. + +The classes affected are :class:`ctypes.UnionType`, +:class:`pickle.PickleBuffer`, :class:`testcapi.RecursingInfinitelyError`, +and :class:`types.GenericAlias`. + +The functions affected are 24 methods in :mod:`ctypes`. + +Patch by Oleg Iarygin. diff --git a/Misc/NEWS.d/next/Documentation/2022-03-08-22-10-38.bpo-46962.FIVe9I.rst b/Misc/NEWS.d/next/Documentation/2022-03-08-22-10-38.bpo-46962.FIVe9I.rst new file mode 100644 index 0000000000000..f5b54013bd672 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2022-03-08-22-10-38.bpo-46962.FIVe9I.rst @@ -0,0 +1,4 @@ +All docstrings in code snippets are now wrapped into :func:`PyDoc_STR` to +follow the guideline of `PEP 7's Documentation Strings paragraph +`_. Patch +by Oleg Iarygin. diff --git a/Modules/_ctypes/_ctypes.c b/Modules/_ctypes/_ctypes.c index 0415923694129..99f3d55ff5fc0 100644 --- a/Modules/_ctypes/_ctypes.c +++ b/Modules/_ctypes/_ctypes.c @@ -196,7 +196,7 @@ static PyTypeObject DictRemover_Type = { 0, /* tp_as_buffer */ /* XXX should participate in GC? */ Py_TPFLAGS_DEFAULT, /* tp_flags */ - "deletes a key from a dictionary", /* tp_doc */ + PyDoc_STR("deletes a key from a dictionary"), /* tp_doc */ 0, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ @@ -579,8 +579,8 @@ UnionType_new(PyTypeObject *type, PyObject *args, PyObject *kwds) return StructUnionType_new(type, args, kwds, 0); } -static const char from_address_doc[] = -"C.from_address(integer) -> C instance\naccess a C instance at the specified address"; +PyDoc_STRVAR(from_address_doc, +"C.from_address(integer) -> C instance\naccess a C instance at the specified address"); static PyObject * CDataType_from_address(PyObject *type, PyObject *value) @@ -597,8 +597,8 @@ CDataType_from_address(PyObject *type, PyObject *value) return PyCData_AtAddress(type, buf); } -static const char from_buffer_doc[] = -"C.from_buffer(object, offset=0) -> C instance\ncreate a C instance from a writeable buffer"; +PyDoc_STRVAR(from_buffer_doc, +"C.from_buffer(object, offset=0) -> C instance\ncreate a C instance from a writeable buffer"); static int KeepRef(CDataObject *target, Py_ssize_t index, PyObject *keep); @@ -677,8 +677,8 @@ CDataType_from_buffer(PyObject *type, PyObject *args) return result; } -static const char from_buffer_copy_doc[] = -"C.from_buffer_copy(object, offset=0) -> C instance\ncreate a C instance from a readable buffer"; +PyDoc_STRVAR(from_buffer_copy_doc, +"C.from_buffer_copy(object, offset=0) -> C instance\ncreate a C instance from a readable buffer"); static PyObject * GenericPyCData_new(PyTypeObject *type, PyObject *args, PyObject *kwds); @@ -728,8 +728,8 @@ CDataType_from_buffer_copy(PyObject *type, PyObject *args) return result; } -static const char in_dll_doc[] = -"C.in_dll(dll, name) -> C instance\naccess a C instance in a dll"; +PyDoc_STRVAR(in_dll_doc, +"C.in_dll(dll, name) -> C instance\naccess a C instance in a dll"); static PyObject * CDataType_in_dll(PyObject *type, PyObject *args) @@ -790,8 +790,8 @@ CDataType_in_dll(PyObject *type, PyObject *args) return PyCData_AtAddress(type, address); } -static const char from_param_doc[] = -"Convert a Python object into a function call parameter."; +PyDoc_STRVAR(from_param_doc, +"Convert a Python object into a function call parameter."); static PyObject * CDataType_from_param(PyObject *type, PyObject *value) @@ -945,7 +945,7 @@ PyTypeObject PyCStructType_Type = { PyCStructType_setattro, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HAVE_GC, /* tp_flags */ - "metatype for the CData Objects", /* tp_doc */ + PyDoc_STR("metatype for the CData Objects"), /* tp_doc */ (traverseproc)CDataType_traverse, /* tp_traverse */ (inquiry)CDataType_clear, /* tp_clear */ 0, /* tp_richcompare */ @@ -987,7 +987,7 @@ static PyTypeObject UnionType_Type = { UnionType_setattro, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HAVE_GC, /* tp_flags */ - "metatype for the CData Objects", /* tp_doc */ + PyDoc_STR("metatype for the CData Objects"), /* tp_doc */ (traverseproc)CDataType_traverse, /* tp_traverse */ (inquiry)CDataType_clear, /* tp_clear */ 0, /* tp_richcompare */ @@ -1245,7 +1245,7 @@ PyTypeObject PyCPointerType_Type = { 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HAVE_GC, /* tp_flags */ - "metatype for the Pointer Objects", /* tp_doc */ + PyDoc_STR("metatype for the Pointer Objects"), /* tp_doc */ (traverseproc)CDataType_traverse, /* tp_traverse */ (inquiry)CDataType_clear, /* tp_clear */ 0, /* tp_richcompare */ @@ -1620,7 +1620,7 @@ PyTypeObject PyCArrayType_Type = { 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */ - "metatype for the Array Objects", /* tp_doc */ + PyDoc_STR("metatype for the Array Objects"), /* tp_doc */ 0, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ @@ -2314,7 +2314,7 @@ PyTypeObject PyCSimpleType_Type = { 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */ - "metatype for the PyCSimpleType Objects", /* tp_doc */ + PyDoc_STR("metatype for the PyCSimpleType Objects"), /* tp_doc */ 0, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ @@ -2603,7 +2603,7 @@ PyTypeObject PyCFuncPtrType_Type = { 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HAVE_GC, /* tp_flags */ - "metatype for C function pointers", /* tp_doc */ + PyDoc_STR("metatype for C function pointers"), /* tp_doc */ (traverseproc)CDataType_traverse, /* tp_traverse */ (inquiry)CDataType_clear, /* tp_clear */ 0, /* tp_richcompare */ @@ -2908,7 +2908,7 @@ PyTypeObject PyCData_Type = { 0, /* tp_setattro */ &PyCData_as_buffer, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */ - "XXX to be provided", /* tp_doc */ + PyDoc_STR("XXX to be provided"), /* tp_doc */ (traverseproc)PyCData_traverse, /* tp_traverse */ (inquiry)PyCData_clear, /* tp_clear */ 0, /* tp_richcompare */ @@ -4306,7 +4306,7 @@ PyTypeObject PyCFuncPtr_Type = { 0, /* tp_setattro */ &PyCData_as_buffer, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */ - "Function Pointer", /* tp_doc */ + PyDoc_STR("Function Pointer"), /* tp_doc */ (traverseproc)PyCFuncPtr_traverse, /* tp_traverse */ (inquiry)PyCFuncPtr_clear, /* tp_clear */ 0, /* tp_richcompare */ @@ -4458,7 +4458,7 @@ static PyTypeObject Struct_Type = { 0, /* tp_setattro */ &PyCData_as_buffer, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */ - "Structure base class", /* tp_doc */ + PyDoc_STR("Structure base class"), /* tp_doc */ (traverseproc)PyCData_traverse, /* tp_traverse */ (inquiry)PyCData_clear, /* tp_clear */ 0, /* tp_richcompare */ @@ -4500,7 +4500,7 @@ static PyTypeObject Union_Type = { 0, /* tp_setattro */ &PyCData_as_buffer, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */ - "Union base class", /* tp_doc */ + PyDoc_STR("Union base class"), /* tp_doc */ (traverseproc)PyCData_traverse, /* tp_traverse */ (inquiry)PyCData_clear, /* tp_clear */ 0, /* tp_richcompare */ @@ -4820,7 +4820,7 @@ PyTypeObject PyCArray_Type = { 0, /* tp_setattro */ &PyCData_as_buffer, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */ - "XXX to be provided", /* tp_doc */ + PyDoc_STR("XXX to be provided"), /* tp_doc */ (traverseproc)PyCData_traverse, /* tp_traverse */ (inquiry)PyCData_clear, /* tp_clear */ 0, /* tp_richcompare */ @@ -5039,7 +5039,7 @@ static PyTypeObject Simple_Type = { 0, /* tp_setattro */ &PyCData_as_buffer, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */ - "XXX to be provided", /* tp_doc */ + PyDoc_STR("XXX to be provided"), /* tp_doc */ (traverseproc)PyCData_traverse, /* tp_traverse */ (inquiry)PyCData_clear, /* tp_clear */ 0, /* tp_richcompare */ @@ -5421,7 +5421,7 @@ PyTypeObject PyCPointer_Type = { 0, /* tp_setattro */ &PyCData_as_buffer, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */ - "XXX to be provided", /* tp_doc */ + PyDoc_STR("XXX to be provided"), /* tp_doc */ (traverseproc)PyCData_traverse, /* tp_traverse */ (inquiry)PyCData_clear, /* tp_clear */ 0, /* tp_richcompare */ @@ -5448,12 +5448,12 @@ PyTypeObject PyCPointer_Type = { * Module initialization. */ -static const char module_docs[] = -"Create and manipulate C compatible data types in Python."; +PyDoc_STRVAR(_ctypes__doc__, +"Create and manipulate C compatible data types in Python."); #ifdef MS_WIN32 -static const char comerror_doc[] = "Raised when a COM method call failed."; +PyDoc_STRVAR(comerror_doc, "Raised when a COM method call failed."); int comerror_init(PyObject *self, PyObject *args, PyObject *kwds) @@ -5642,7 +5642,7 @@ wstring_at(const wchar_t *ptr, int size) static struct PyModuleDef _ctypesmodule = { PyModuleDef_HEAD_INIT, .m_name = "_ctypes", - .m_doc = module_docs, + .m_doc = _ctypes__doc__, .m_size = -1, .m_methods = _ctypes_module_methods, }; diff --git a/Modules/_ctypes/callbacks.c b/Modules/_ctypes/callbacks.c index 7dd1f99021a7f..e1e0225f67b34 100644 --- a/Modules/_ctypes/callbacks.c +++ b/Modules/_ctypes/callbacks.c @@ -82,7 +82,7 @@ PyTypeObject PyCThunk_Type = { 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC, /* tp_flags */ - "CThunkObject", /* tp_doc */ + PyDoc_STR("CThunkObject"), /* tp_doc */ CThunkObject_traverse, /* tp_traverse */ CThunkObject_clear, /* tp_clear */ 0, /* tp_richcompare */ diff --git a/Modules/_ctypes/callproc.c b/Modules/_ctypes/callproc.c index 720c4c09f538f..3fab9ad0c1e7b 100644 --- a/Modules/_ctypes/callproc.c +++ b/Modules/_ctypes/callproc.c @@ -1307,11 +1307,11 @@ _parse_voidp(PyObject *obj, void **address) #ifdef MS_WIN32 -static const char format_error_doc[] = +PyDoc_STRVAR(format_error_doc, "FormatError([integer]) -> string\n\ \n\ Convert a win32 error code into a string. If the error code is not\n\ -given, the return value of a call to GetLastError() is used.\n"; +given, the return value of a call to GetLastError() is used.\n"); static PyObject *format_error(PyObject *self, PyObject *args) { PyObject *result; @@ -1331,13 +1331,13 @@ static PyObject *format_error(PyObject *self, PyObject *args) return result; } -static const char load_library_doc[] = +PyDoc_STRVAR(load_library_doc, "LoadLibrary(name, load_flags) -> handle\n\ \n\ Load an executable (usually a DLL), and return a handle to it.\n\ The handle may be used to locate exported functions in this\n\ module. load_flags are as defined for LoadLibraryEx in the\n\ -Windows API.\n"; +Windows API.\n"); static PyObject *load_library(PyObject *self, PyObject *args) { PyObject *nameobj; @@ -1382,10 +1382,10 @@ static PyObject *load_library(PyObject *self, PyObject *args) #endif } -static const char free_library_doc[] = +PyDoc_STRVAR(free_library_doc, "FreeLibrary(handle) -> void\n\ \n\ -Free the handle of an executable previously loaded by LoadLibrary.\n"; +Free the handle of an executable previously loaded by LoadLibrary.\n"); static PyObject *free_library(PyObject *self, PyObject *args) { void *hMod; @@ -1405,8 +1405,8 @@ static PyObject *free_library(PyObject *self, PyObject *args) Py_RETURN_NONE; } -static const char copy_com_pointer_doc[] = -"CopyComPointer(src, dst) -> HRESULT value\n"; +PyDoc_STRVAR(copy_com_pointer_doc, +"CopyComPointer(src, dst) -> HRESULT value\n"); static PyObject * copy_com_pointer(PyObject *self, PyObject *args) @@ -1644,10 +1644,10 @@ call_cdeclfunction(PyObject *self, PyObject *args) /***************************************************************** * functions */ -static const char sizeof_doc[] = +PyDoc_STRVAR(sizeof_doc, "sizeof(C type) -> integer\n" "sizeof(C instance) -> integer\n" -"Return the size in bytes of a C instance"; +"Return the size in bytes of a C instance"); static PyObject * sizeof_func(PyObject *self, PyObject *obj) @@ -1665,10 +1665,10 @@ sizeof_func(PyObject *self, PyObject *obj) return NULL; } -static const char alignment_doc[] = +PyDoc_STRVAR(alignment_doc, "alignment(C type) -> integer\n" "alignment(C instance) -> integer\n" -"Return the alignment requirements of a C instance"; +"Return the alignment requirements of a C instance"); static PyObject * align_func(PyObject *self, PyObject *obj) @@ -1688,10 +1688,10 @@ align_func(PyObject *self, PyObject *obj) return NULL; } -static const char byref_doc[] = +PyDoc_STRVAR(byref_doc, "byref(C instance[, offset=0]) -> byref-object\n" "Return a pointer lookalike to a C instance, only usable\n" -"as function argument"; +"as function argument"); /* * We must return something which can be converted to a parameter, @@ -1732,9 +1732,9 @@ byref(PyObject *self, PyObject *args) return (PyObject *)parg; } -static const char addressof_doc[] = +PyDoc_STRVAR(addressof_doc, "addressof(C instance) -> integer\n" -"Return the address of the C instance internal buffer"; +"Return the address of the C instance internal buffer"); static PyObject * addressof(PyObject *self, PyObject *obj) diff --git a/Modules/_ctypes/cfield.c b/Modules/_ctypes/cfield.c index 3b769f950a459..23c11c98b2afc 100644 --- a/Modules/_ctypes/cfield.c +++ b/Modules/_ctypes/cfield.c @@ -325,7 +325,7 @@ PyTypeObject PyCField_Type = { 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC, /* tp_flags */ - "Structure/Union member", /* tp_doc */ + PyDoc_STR("Structure/Union member"), /* tp_doc */ (traverseproc)PyCField_traverse, /* tp_traverse */ (inquiry)PyCField_clear, /* tp_clear */ 0, /* tp_richcompare */ diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c index 71683abebb231..23af0d36b6ceb 100644 --- a/Modules/_testcapimodule.c +++ b/Modules/_testcapimodule.c @@ -6599,7 +6599,7 @@ static PyTypeObject PyRecursingInfinitelyError_Type = { 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */ - "Instantiating this exception starts infinite recursion.", /* tp_doc */ + PyDoc_STR("Instantiating this exception starts infinite recursion."), /* tp_doc */ 0, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ diff --git a/Objects/genericaliasobject.c b/Objects/genericaliasobject.c index 3f03630005bc5..dc585de4729fc 100644 --- a/Objects/genericaliasobject.c +++ b/Objects/genericaliasobject.c @@ -341,6 +341,11 @@ _Py_subs_parameters(PyObject *self, PyObject *args, PyObject *parameters, PyObje return newargs; } +PyDoc_STRVAR(genericalias__doc__, +"Represent a PEP 585 generic type\n" +"\n" +"E.g. for t = list[int], t.__origin__ is list and t.__args__ is (int,)."); + static PyObject * ga_getitem(PyObject *self, PyObject *item) { @@ -703,14 +708,11 @@ ga_iter(PyObject *self) { // TODO: // - argument clinic? -// - __doc__? // - cache? PyTypeObject Py_GenericAliasType = { PyVarObject_HEAD_INIT(&PyType_Type, 0) .tp_name = "types.GenericAlias", - .tp_doc = "Represent a PEP 585 generic type\n" - "\n" - "E.g. for t = list[int], t.__origin__ is list and t.__args__ is (int,).", + .tp_doc = genericalias__doc__, .tp_basicsize = sizeof(gaobject), .tp_dealloc = ga_dealloc, .tp_repr = ga_repr, diff --git a/Objects/picklebufobject.c b/Objects/picklebufobject.c index a135e5575e28c..aaa852cfbb05b 100644 --- a/Objects/picklebufobject.c +++ b/Objects/picklebufobject.c @@ -206,7 +206,7 @@ static PyMethodDef picklebuf_methods[] = { PyTypeObject PyPickleBuffer_Type = { PyVarObject_HEAD_INIT(NULL, 0) .tp_name = "pickle.PickleBuffer", - .tp_doc = "Wrapper for potentially out-of-band buffers", + .tp_doc = PyDoc_STR("Wrapper for potentially out-of-band buffers"), .tp_basicsize = sizeof(PyPickleBufferObject), .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC, .tp_new = picklebuf_new, diff --git a/Objects/unionobject.c b/Objects/unionobject.c index 72a0a3f2cf8d2..5432c6faa3f90 100644 --- a/Objects/unionobject.c +++ b/Objects/unionobject.c @@ -443,9 +443,9 @@ union_getattro(PyObject *self, PyObject *name) PyTypeObject _PyUnion_Type = { PyVarObject_HEAD_INIT(&PyType_Type, 0) .tp_name = "types.UnionType", - .tp_doc = "Represent a PEP 604 union type\n" + .tp_doc = PyDoc_STR("Represent a PEP 604 union type\n" "\n" - "E.g. for int | str", + "E.g. for int | str"), .tp_basicsize = sizeof(unionobject), .tp_dealloc = unionobject_dealloc, .tp_alloc = PyType_GenericAlloc, From webhook-mailer at python.org Mon Apr 18 00:12:43 2022 From: webhook-mailer at python.org (rhettinger) Date: Mon, 18 Apr 2022 04:12:43 -0000 Subject: [Python-checkins] Update: usage doc for heappushpop (GH-91451) Message-ID: https://github.com/python/cpython/commit/d7d4a0583ff8bd7c5b614490ba22e88da23b5b84 commit: d7d4a0583ff8bd7c5b614490ba22e88da23b5b84 branch: main author: Aditya Aggarwal committer: rhettinger date: 2022-04-17T23:12:33-05:00 summary: Update: usage doc for heappushpop (GH-91451) files: M Lib/heapq.py diff --git a/Lib/heapq.py b/Lib/heapq.py index fabefd87f8bf8..2fd9d1ff4bf82 100644 --- a/Lib/heapq.py +++ b/Lib/heapq.py @@ -12,6 +12,8 @@ item = heappop(heap) # pops the smallest item from the heap item = heap[0] # smallest item on the heap without popping it heapify(x) # transforms list into a heap, in-place, in linear time +item = heappushpop(heap, item) # pushes a new item and then returns + # the smallest item; the heap size is unchanged item = heapreplace(heap, item) # pops and returns smallest item, and adds # new item; the heap size is unchanged From webhook-mailer at python.org Mon Apr 18 00:26:45 2022 From: webhook-mailer at python.org (methane) Date: Mon, 18 Apr 2022 04:26:45 -0000 Subject: [Python-checkins] Remove duplicate explanation (GH-91534) Message-ID: https://github.com/python/cpython/commit/0e6dca01937b62c07cff5b8450b7c74c101b857d commit: 0e6dca01937b62c07cff5b8450b7c74c101b857d branch: main author: Gouvernathor <44340603+Gouvernathor at users.noreply.github.com> committer: methane date: 2022-04-18T13:26:40+09:00 summary: Remove duplicate explanation (GH-91534) files: M Doc/library/stdtypes.rst diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst index f25e8303f94a2..764d83a6800d1 100644 --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -4383,10 +4383,6 @@ then they can be used interchangeably to index the same dictionary entry. (Note however, that since computers store floating-point numbers as approximations it is usually unwise to use them as dictionary keys.) -Dictionaries can be created by placing a comma-separated list of ``key: value`` -pairs within braces, for example: ``{'jack': 4098, 'sjoerd': 4127}`` or ``{4098: -'jack', 4127: 'sjoerd'}``, or by the :class:`dict` constructor. - .. class:: dict(**kwargs) dict(mapping, **kwargs) dict(iterable, **kwargs) From webhook-mailer at python.org Mon Apr 18 00:45:11 2022 From: webhook-mailer at python.org (miss-islington) Date: Mon, 18 Apr 2022 04:45:11 -0000 Subject: [Python-checkins] Remove duplicate explanation (GH-91534) Message-ID: https://github.com/python/cpython/commit/0897a0bf9c75bacf428b8f6f08114fffc9f9c542 commit: 0897a0bf9c75bacf428b8f6f08114fffc9f9c542 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-17T21:44:58-07:00 summary: Remove duplicate explanation (GH-91534) (cherry picked from commit 0e6dca01937b62c07cff5b8450b7c74c101b857d) Co-authored-by: Gouvernathor <44340603+Gouvernathor at users.noreply.github.com> files: M Doc/library/stdtypes.rst diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst index 77b2590860e32..365680d4503ba 100644 --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -4345,10 +4345,6 @@ then they can be used interchangeably to index the same dictionary entry. (Note however, that since computers store floating-point numbers as approximations it is usually unwise to use them as dictionary keys.) -Dictionaries can be created by placing a comma-separated list of ``key: value`` -pairs within braces, for example: ``{'jack': 4098, 'sjoerd': 4127}`` or ``{4098: -'jack', 4127: 'sjoerd'}``, or by the :class:`dict` constructor. - .. class:: dict(**kwargs) dict(mapping, **kwargs) dict(iterable, **kwargs) From webhook-mailer at python.org Mon Apr 18 00:46:28 2022 From: webhook-mailer at python.org (rhettinger) Date: Mon, 18 Apr 2022 04:46:28 -0000 Subject: [Python-checkins] bpo-21150: Add quick link/summary table to the top of argparse documentation (GH-12005) Message-ID: https://github.com/python/cpython/commit/8e76d7e1a9592b24717204ad307b7493d593191f commit: 8e76d7e1a9592b24717204ad307b7493d593191f branch: main author: Susan Su committer: rhettinger date: 2022-04-17T23:46:18-05:00 summary: bpo-21150: Add quick link/summary table to the top of argparse documentation (GH-12005) No work has been done to move this forward. On the theory that perfect is the enemy of good, I'm going to push it and we can make minor edits as needed afterwards. files: A Misc/NEWS.d/next/Documentation/2019-02-24-03-06-59.bpo-21150.Vqv8Yc.rst M Doc/library/argparse.rst diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst index e050d6298b6ff..425409db3cb75 100644 --- a/Doc/library/argparse.rst +++ b/Doc/library/argparse.rst @@ -26,6 +26,75 @@ module also automatically generates help and usage messages and issues errors when users give the program invalid arguments. +Summary +------- + +Core Functionality +^^^^^^^^^^^^^^^^^^ + +The :mod:`argparse` module's support for command-line interfaces is built +from the following: + +The :class:`argparse.ArgumentParser` creates a new :class:`ArgumentParser` +object. Commonly used arguments include prog_, description_, and +formatter_class_. For example, the user can create an instance of +:class:`ArgumentParser` through the following:: + + >>> parser = argparse.ArgumentParser(prog='PROG', description='DESC', + ... formatter_class=argparse.RawDescriptionHelpFormatter) + +The :func:`ArgumentParser.add_argument` is a function that is used +to define how a single command-line argument should be parsed. Commonly used +arguments include `name or flags`_, action_, default_, type_, required_, +and help_. An example of the function :func:`ArgumentParser.add_argument` +is as follows:: + + >>> parser.add_argument('-v', '--verbose', action='store_true', + ... help='Show various debugging information') + + +Basic Usage of :func:`add_argument` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + + +**Name or Flags Type** + +====================== =========================== +Type Example +====================== =========================== +Positional ``'foo'`` +Optional ``'-v'``, ``'--verbose'`` +====================== =========================== + + +**Basic Arguments:** + +====================== =========================================================== ========================================================================================================================= +Name Description Keywords +====================== =========================================================== ========================================================================================================================= +action_ Specifies how an argument should be handled ``'store'``, ``'store_const'``, ``'store_true'``, ``'append'``, ``'append_const'``, ``'count'``, ``'help'``, ``'version'`` +default_ Default value used when an argument is not provided +type_ Automatically converts an argument to the given type :class:`int`, :class:`float`, :class:`bool`, ``argparse.FileType('w')``, ``callable function`` +help_ Help message of an argument +====================== =========================================================== ========================================================================================================================= + + + +**Advanced Arguments:** + +====================== =========================================================== ======================================================================================================================= +Name Description Keywords +====================== =========================================================== ======================================================================================================================= +nargs_ Associates a single action with the number of arguments ``N`` (:class:`int`), ``'?'``, ``'*'``, ``'+'``, ``argparse.REMAINDER`` +const_ Stores constant values of names or flags +choices_ A container that lists the possible values ``['foo', 'bar']``, ``range(1, 10)``, Any object that supports ``in`` operator +required_ Indicates if an optional argument is required or not ``True``, ``False`` +metavar_ An alternative display name for the argument +dest_ Specifies name of attribute to be used in ``parse_args()`` +====================== =========================================================== ======================================================================================================================= + + + Example ------- @@ -196,6 +265,8 @@ ArgumentParser objects The following sections describe how each of these are used. +.. _prog: + prog ^^^^ @@ -293,6 +364,8 @@ The ``%(prog)s`` format specifier is available to fill in the program name in your usage messages. +.. _description: + description ^^^^^^^^^^^ @@ -373,6 +446,8 @@ and one in the child) and raise an error. not be reflected in the child. +.. _formatter_class: + formatter_class ^^^^^^^^^^^^^^^ @@ -716,6 +791,8 @@ The add_argument() method The following sections describe how each of these are used. +.. _name_or_flags: + name or flags ^^^^^^^^^^^^^ @@ -749,6 +826,8 @@ be positional:: PROG: error: the following arguments are required: bar +.. _action: + action ^^^^^^ @@ -884,6 +963,9 @@ An example of a custom action:: For more details, see :class:`Action`. + +.. _nargs: + nargs ^^^^^ @@ -971,6 +1053,8 @@ is determined by the action_. Generally this means a single command-line argume will be consumed and a single item (not a list) will be produced. +.. _const: + const ^^^^^ @@ -997,6 +1081,8 @@ the various :class:`ArgumentParser` actions. The two most common uses of it are ``const=None`` by default, including when ``action='append_const'`` or ``action='store_const'``. +.. _default: + default ^^^^^^^ @@ -1055,6 +1141,8 @@ command-line argument was not present:: Namespace(foo='1') +.. _type: + type ^^^^ @@ -1124,6 +1212,8 @@ For type checkers that simply check against a fixed set of values, consider using the choices_ keyword instead. +.. _choices: + choices ^^^^^^^ @@ -1166,6 +1256,8 @@ from *dest*. This is usually what you want because the user never sees the many choices), just specify an explicit metavar_. +.. _required: + required ^^^^^^^^ @@ -1192,6 +1284,8 @@ present at the command line. *options* to be *optional*, and thus they should be avoided when possible. +.. _help: + help ^^^^ @@ -1247,6 +1341,8 @@ setting the ``help`` value to ``argparse.SUPPRESS``:: -h, --help show this help message and exit +.. _metavar: + metavar ^^^^^^^ @@ -1311,6 +1407,8 @@ arguments:: --foo bar baz +.. _dest: + dest ^^^^ diff --git a/Misc/NEWS.d/next/Documentation/2019-02-24-03-06-59.bpo-21150.Vqv8Yc.rst b/Misc/NEWS.d/next/Documentation/2019-02-24-03-06-59.bpo-21150.Vqv8Yc.rst new file mode 100644 index 0000000000000..d1b5615a12e7f --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2019-02-24-03-06-59.bpo-21150.Vqv8Yc.rst @@ -0,0 +1 @@ +Place summary/quick links table at the top of the documentation for the argparse module to benefit ease of use. \ No newline at end of file From webhook-mailer at python.org Mon Apr 18 00:48:46 2022 From: webhook-mailer at python.org (miss-islington) Date: Mon, 18 Apr 2022 04:48:46 -0000 Subject: [Python-checkins] Remove duplicate explanation (GH-91534) Message-ID: https://github.com/python/cpython/commit/636832fb2a4364ca2014aadecf4c35833eff4495 commit: 636832fb2a4364ca2014aadecf4c35833eff4495 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-17T21:48:36-07:00 summary: Remove duplicate explanation (GH-91534) (cherry picked from commit 0e6dca01937b62c07cff5b8450b7c74c101b857d) Co-authored-by: Gouvernathor <44340603+Gouvernathor at users.noreply.github.com> files: M Doc/library/stdtypes.rst diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst index cf1a638344c8e..1cba75084b04b 100644 --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -4311,10 +4311,6 @@ then they can be used interchangeably to index the same dictionary entry. (Note however, that since computers store floating-point numbers as approximations it is usually unwise to use them as dictionary keys.) -Dictionaries can be created by placing a comma-separated list of ``key: value`` -pairs within braces, for example: ``{'jack': 4098, 'sjoerd': 4127}`` or ``{4098: -'jack', 4127: 'sjoerd'}``, or by the :class:`dict` constructor. - .. class:: dict(**kwargs) dict(mapping, **kwargs) dict(iterable, **kwargs) From webhook-mailer at python.org Mon Apr 18 01:28:15 2022 From: webhook-mailer at python.org (rhettinger) Date: Mon, 18 Apr 2022 05:28:15 -0000 Subject: [Python-checkins] Remove ill-formed an unneeded news entry (GH-91657) Message-ID: https://github.com/python/cpython/commit/4c3b283e83459cf7224bbf353300099eba7a2c1c commit: 4c3b283e83459cf7224bbf353300099eba7a2c1c branch: main author: Raymond Hettinger committer: rhettinger date: 2022-04-18T00:27:41-05:00 summary: Remove ill-formed an unneeded news entry (GH-91657) files: D Misc/NEWS.d/next/Documentation/2019-02-24-03-06-59.bpo-21150.Vqv8Yc.rst diff --git a/Misc/NEWS.d/next/Documentation/2019-02-24-03-06-59.bpo-21150.Vqv8Yc.rst b/Misc/NEWS.d/next/Documentation/2019-02-24-03-06-59.bpo-21150.Vqv8Yc.rst deleted file mode 100644 index d1b5615a12e7f..0000000000000 --- a/Misc/NEWS.d/next/Documentation/2019-02-24-03-06-59.bpo-21150.Vqv8Yc.rst +++ /dev/null @@ -1 +0,0 @@ -Place summary/quick links table at the top of the documentation for the argparse module to benefit ease of use. \ No newline at end of file From webhook-mailer at python.org Mon Apr 18 03:27:14 2022 From: webhook-mailer at python.org (ronaldoussoren) Date: Mon, 18 Apr 2022 07:27:14 -0000 Subject: [Python-checkins] gh-91265: Make old ctypes.macholib tests runned by `python -m test` (gh-32094) Message-ID: https://github.com/python/cpython/commit/804ea2da97af68366cdeceb30d3987f6d9f3e797 commit: 804ea2da97af68366cdeceb30d3987f6d9f3e797 branch: main author: Oleg Iarygin committer: ronaldoussoren date: 2022-04-18T09:27:09+02:00 summary: gh-91265: Make old ctypes.macholib tests runned by `python -m test` (gh-32094) * Move tests from ctypes.macholib.dy* to ctypes.test files: A Misc/NEWS.d/next/Tests/2022-03-24-13-35-01.bpo-47109.FjKQCE.rst M Lib/ctypes/macholib/dyld.py M Lib/ctypes/macholib/dylib.py M Lib/ctypes/macholib/framework.py M Lib/ctypes/test/test_macholib.py diff --git a/Lib/ctypes/macholib/dyld.py b/Lib/ctypes/macholib/dyld.py index 1c3f8fd38b066..583c47daff3ad 100644 --- a/Lib/ctypes/macholib/dyld.py +++ b/Lib/ctypes/macholib/dyld.py @@ -163,11 +163,3 @@ def framework_find(fn, executable_path=None, env=None): raise error finally: error = None - -def test_dyld_find(): - env = {} - assert dyld_find('libSystem.dylib') == '/usr/lib/libSystem.dylib' - assert dyld_find('System.framework/System') == '/System/Library/Frameworks/System.framework/System' - -if __name__ == '__main__': - test_dyld_find() diff --git a/Lib/ctypes/macholib/dylib.py b/Lib/ctypes/macholib/dylib.py index aa107507bd4a9..0ad4cba8da352 100644 --- a/Lib/ctypes/macholib/dylib.py +++ b/Lib/ctypes/macholib/dylib.py @@ -40,24 +40,3 @@ def dylib_info(filename): if not is_dylib: return None return is_dylib.groupdict() - - -def test_dylib_info(): - def d(location=None, name=None, shortname=None, version=None, suffix=None): - return dict( - location=location, - name=name, - shortname=shortname, - version=version, - suffix=suffix - ) - assert dylib_info('completely/invalid') is None - assert dylib_info('completely/invalide_debug') is None - assert dylib_info('P/Foo.dylib') == d('P', 'Foo.dylib', 'Foo') - assert dylib_info('P/Foo_debug.dylib') == d('P', 'Foo_debug.dylib', 'Foo', suffix='debug') - assert dylib_info('P/Foo.A.dylib') == d('P', 'Foo.A.dylib', 'Foo', 'A') - assert dylib_info('P/Foo_debug.A.dylib') == d('P', 'Foo_debug.A.dylib', 'Foo_debug', 'A') - assert dylib_info('P/Foo.A_debug.dylib') == d('P', 'Foo.A_debug.dylib', 'Foo', 'A', 'debug') - -if __name__ == '__main__': - test_dylib_info() diff --git a/Lib/ctypes/macholib/framework.py b/Lib/ctypes/macholib/framework.py index ad6ed554ba0c2..495679fff19d4 100644 --- a/Lib/ctypes/macholib/framework.py +++ b/Lib/ctypes/macholib/framework.py @@ -40,26 +40,3 @@ def framework_info(filename): if not is_framework: return None return is_framework.groupdict() - -def test_framework_info(): - def d(location=None, name=None, shortname=None, version=None, suffix=None): - return dict( - location=location, - name=name, - shortname=shortname, - version=version, - suffix=suffix - ) - assert framework_info('completely/invalid') is None - assert framework_info('completely/invalid/_debug') is None - assert framework_info('P/F.framework') is None - assert framework_info('P/F.framework/_debug') is None - assert framework_info('P/F.framework/F') == d('P', 'F.framework/F', 'F') - assert framework_info('P/F.framework/F_debug') == d('P', 'F.framework/F_debug', 'F', suffix='debug') - assert framework_info('P/F.framework/Versions') is None - assert framework_info('P/F.framework/Versions/A') is None - assert framework_info('P/F.framework/Versions/A/F') == d('P', 'F.framework/Versions/A/F', 'F', 'A') - assert framework_info('P/F.framework/Versions/A/F_debug') == d('P', 'F.framework/Versions/A/F_debug', 'F', 'A', 'debug') - -if __name__ == '__main__': - test_framework_info() diff --git a/Lib/ctypes/test/test_macholib.py b/Lib/ctypes/test/test_macholib.py index a1bac26a7df05..bc75f1a05a8c3 100644 --- a/Lib/ctypes/test/test_macholib.py +++ b/Lib/ctypes/test/test_macholib.py @@ -32,6 +32,8 @@ # -bob from ctypes.macholib.dyld import dyld_find +from ctypes.macholib.dylib import dylib_info +from ctypes.macholib.framework import framework_info def find_lib(name): possible = ['lib'+name+'.dylib', name+'.dylib', name+'.framework/'+name] @@ -42,9 +44,20 @@ def find_lib(name): pass raise ValueError("%s not found" % (name,)) + +def d(location=None, name=None, shortname=None, version=None, suffix=None): + return {'location': location, 'name': name, 'shortname': shortname, + 'version': version, 'suffix': suffix} + + class MachOTest(unittest.TestCase): @unittest.skipUnless(sys.platform == "darwin", 'OSX-specific test') def test_find(self): + self.assertEqual(dyld_find('libSystem.dylib'), + '/usr/lib/libSystem.dylib') + self.assertEqual(dyld_find('System.framework/System'), + '/System/Library/Frameworks/System.framework/System') + # On Mac OS 11, system dylibs are only present in the shared cache, # so symlinks like libpthread.dylib -> libSystem.B.dylib will not # be resolved by dyld_find @@ -62,5 +75,36 @@ def test_find(self): ('/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit', '/System/Library/Frameworks/IOKit.framework/IOKit')) + @unittest.skipUnless(sys.platform == "darwin", 'OSX-specific test') + def test_info(self): + self.assertIsNone(dylib_info('completely/invalid')) + self.assertIsNone(dylib_info('completely/invalide_debug')) + self.assertEqual(dylib_info('P/Foo.dylib'), d('P', 'Foo.dylib', 'Foo')) + self.assertEqual(dylib_info('P/Foo_debug.dylib'), + d('P', 'Foo_debug.dylib', 'Foo', suffix='debug')) + self.assertEqual(dylib_info('P/Foo.A.dylib'), + d('P', 'Foo.A.dylib', 'Foo', 'A')) + self.assertEqual(dylib_info('P/Foo_debug.A.dylib'), + d('P', 'Foo_debug.A.dylib', 'Foo_debug', 'A')) + self.assertEqual(dylib_info('P/Foo.A_debug.dylib'), + d('P', 'Foo.A_debug.dylib', 'Foo', 'A', 'debug')) + + @unittest.skipUnless(sys.platform == "darwin", 'OSX-specific test') + def test_framework_info(self): + self.assertIsNone(framework_info('completely/invalid')) + self.assertIsNone(framework_info('completely/invalid/_debug')) + self.assertIsNone(framework_info('P/F.framework')) + self.assertIsNone(framework_info('P/F.framework/_debug')) + self.assertEqual(framework_info('P/F.framework/F'), + d('P', 'F.framework/F', 'F')) + self.assertEqual(framework_info('P/F.framework/F_debug'), + d('P', 'F.framework/F_debug', 'F', suffix='debug')) + self.assertIsNone(framework_info('P/F.framework/Versions')) + self.assertIsNone(framework_info('P/F.framework/Versions/A')) + self.assertEqual(framework_info('P/F.framework/Versions/A/F'), + d('P', 'F.framework/Versions/A/F', 'F', 'A')) + self.assertEqual(framework_info('P/F.framework/Versions/A/F_debug'), + d('P', 'F.framework/Versions/A/F_debug', 'F', 'A', 'debug')) + if __name__ == "__main__": unittest.main() diff --git a/Misc/NEWS.d/next/Tests/2022-03-24-13-35-01.bpo-47109.FjKQCE.rst b/Misc/NEWS.d/next/Tests/2022-03-24-13-35-01.bpo-47109.FjKQCE.rst new file mode 100644 index 0000000000000..d0a402f3a225d --- /dev/null +++ b/Misc/NEWS.d/next/Tests/2022-03-24-13-35-01.bpo-47109.FjKQCE.rst @@ -0,0 +1,3 @@ +Test for :mod:`ctypes.macholib.dyld`, :mod:`ctypes.macholib.dylib`, and +:mod:`ctypes.macholib.framework` are brought from manual pre-:mod:`unittest` +times to :mod:`ctypes.test` location and structure. Patch by Oleg Iarygin. From webhook-mailer at python.org Mon Apr 18 05:13:14 2022 From: webhook-mailer at python.org (serhiy-storchaka) Date: Mon, 18 Apr 2022 09:13:14 -0000 Subject: [Python-checkins] bpo-46053: Fix OSS audio support on NetBSD (GH-30065) Message-ID: https://github.com/python/cpython/commit/2e7e3c4c109928870c1e33d8af36b78e92895594 commit: 2e7e3c4c109928870c1e33d8af36b78e92895594 branch: main author: Thomas Klausner committer: serhiy-storchaka date: 2022-04-18T12:12:39+03:00 summary: bpo-46053: Fix OSS audio support on NetBSD (GH-30065) files: A Misc/NEWS.d/next/Core and Builtins/2021-12-11-11-36-48.bpo-46045.sfThay.rst A Misc/NEWS.d/next/Library/2022-02-06-12-59-32.bpo-46053.sHFo3S.rst M Modules/ossaudiodev.c M configure M configure.ac diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-12-11-11-36-48.bpo-46045.sfThay.rst b/Misc/NEWS.d/next/Core and Builtins/2021-12-11-11-36-48.bpo-46045.sfThay.rst new file mode 100644 index 0000000000000..97fd1883eb2ab --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2021-12-11-11-36-48.bpo-46045.sfThay.rst @@ -0,0 +1 @@ +Do not use POSIX semaphores on NetBSD diff --git a/Misc/NEWS.d/next/Library/2022-02-06-12-59-32.bpo-46053.sHFo3S.rst b/Misc/NEWS.d/next/Library/2022-02-06-12-59-32.bpo-46053.sHFo3S.rst new file mode 100644 index 0000000000000..ce375885792e8 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-02-06-12-59-32.bpo-46053.sHFo3S.rst @@ -0,0 +1 @@ +Fix OSS audio support on NetBSD. diff --git a/Modules/ossaudiodev.c b/Modules/ossaudiodev.c index 3926831e0c0d9..2f507de46f5dc 100644 --- a/Modules/ossaudiodev.c +++ b/Modules/ossaudiodev.c @@ -1226,16 +1226,36 @@ PyInit_ossaudiodev(void) /* Expose all the ioctl numbers for masochists who like to do this stuff directly. */ +#ifdef SNDCTL_COPR_HALT _EXPORT_INT(m, SNDCTL_COPR_HALT); +#endif +#ifdef SNDCTL_COPR_LOAD _EXPORT_INT(m, SNDCTL_COPR_LOAD); +#endif +#ifdef SNDCTL_COPR_RCODE _EXPORT_INT(m, SNDCTL_COPR_RCODE); +#endif +#ifdef SNDCTL_COPR_RCVMSG _EXPORT_INT(m, SNDCTL_COPR_RCVMSG); +#endif +#ifdef SNDCTL_COPR_RDATA _EXPORT_INT(m, SNDCTL_COPR_RDATA); +#endif +#ifdef SNDCTL_COPR_RESET _EXPORT_INT(m, SNDCTL_COPR_RESET); +#endif +#ifdef SNDCTL_COPR_RUN _EXPORT_INT(m, SNDCTL_COPR_RUN); +#endif +#ifdef SNDCTL_COPR_SENDMSG _EXPORT_INT(m, SNDCTL_COPR_SENDMSG); +#endif +#ifdef SNDCTL_COPR_WCODE _EXPORT_INT(m, SNDCTL_COPR_WCODE); +#endif +#ifdef SNDCTL_COPR_WDATA _EXPORT_INT(m, SNDCTL_COPR_WDATA); +#endif #ifdef SNDCTL_DSP_BIND_CHANNEL _EXPORT_INT(m, SNDCTL_DSP_BIND_CHANNEL); #endif @@ -1278,46 +1298,104 @@ PyInit_ossaudiodev(void) _EXPORT_INT(m, SNDCTL_DSP_STEREO); _EXPORT_INT(m, SNDCTL_DSP_SUBDIVIDE); _EXPORT_INT(m, SNDCTL_DSP_SYNC); +#ifdef SNDCTL_FM_4OP_ENABLE _EXPORT_INT(m, SNDCTL_FM_4OP_ENABLE); +#endif +#ifdef SNDCTL_FM_LOAD_INSTR _EXPORT_INT(m, SNDCTL_FM_LOAD_INSTR); +#endif +#ifdef SNDCTL_MIDI_INFO _EXPORT_INT(m, SNDCTL_MIDI_INFO); +#endif +#ifdef SNDCTL_MIDI_MPUCMD _EXPORT_INT(m, SNDCTL_MIDI_MPUCMD); +#endif +#ifdef SNDCTL_MIDI_MPUMODE _EXPORT_INT(m, SNDCTL_MIDI_MPUMODE); +#endif +#ifdef SNDCTL_MIDI_PRETIME _EXPORT_INT(m, SNDCTL_MIDI_PRETIME); +#endif +#ifdef SNDCTL_SEQ_CTRLRATE _EXPORT_INT(m, SNDCTL_SEQ_CTRLRATE); +#endif +#ifdef SNDCTL_SEQ_GETINCOUNT _EXPORT_INT(m, SNDCTL_SEQ_GETINCOUNT); +#endif +#ifdef SNDCTL_SEQ_GETOUTCOUNT _EXPORT_INT(m, SNDCTL_SEQ_GETOUTCOUNT); +#endif #ifdef SNDCTL_SEQ_GETTIME _EXPORT_INT(m, SNDCTL_SEQ_GETTIME); #endif +#ifdef SNDCTL_SEQ_NRMIDIS _EXPORT_INT(m, SNDCTL_SEQ_NRMIDIS); +#endif +#ifdef SNDCTL_SEQ_NRSYNTHS _EXPORT_INT(m, SNDCTL_SEQ_NRSYNTHS); +#endif +#ifdef SNDCTL_SEQ_OUTOFBAND _EXPORT_INT(m, SNDCTL_SEQ_OUTOFBAND); +#endif +#ifdef SNDCTL_SEQ_PANIC _EXPORT_INT(m, SNDCTL_SEQ_PANIC); +#endif +#ifdef SNDCTL_SEQ_PERCMODE _EXPORT_INT(m, SNDCTL_SEQ_PERCMODE); +#endif +#ifdef SNDCTL_SEQ_RESET _EXPORT_INT(m, SNDCTL_SEQ_RESET); +#endif +#ifdef SNDCTL_SEQ_RESETSAMPLES _EXPORT_INT(m, SNDCTL_SEQ_RESETSAMPLES); +#endif +#ifdef SNDCTL_SEQ_SYNC _EXPORT_INT(m, SNDCTL_SEQ_SYNC); +#endif +#ifdef SNDCTL_SEQ_TESTMIDI _EXPORT_INT(m, SNDCTL_SEQ_TESTMIDI); +#endif +#ifdef SNDCTL_SEQ_THRESHOLD _EXPORT_INT(m, SNDCTL_SEQ_THRESHOLD); +#endif #ifdef SNDCTL_SYNTH_CONTROL _EXPORT_INT(m, SNDCTL_SYNTH_CONTROL); #endif #ifdef SNDCTL_SYNTH_ID _EXPORT_INT(m, SNDCTL_SYNTH_ID); #endif +#ifdef SNDCTL_SYNTH_INFO _EXPORT_INT(m, SNDCTL_SYNTH_INFO); +#endif +#ifdef SNDCTL_SYNTH_MEMAVL _EXPORT_INT(m, SNDCTL_SYNTH_MEMAVL); +#endif #ifdef SNDCTL_SYNTH_REMOVESAMPLE _EXPORT_INT(m, SNDCTL_SYNTH_REMOVESAMPLE); #endif +#ifdef SNDCTL_TMR_CONTINUE _EXPORT_INT(m, SNDCTL_TMR_CONTINUE); +#endif +#ifdef SNDCTL_TMR_METRONOME _EXPORT_INT(m, SNDCTL_TMR_METRONOME); +#endif +#ifdef SNDCTL_TMR_SELECT _EXPORT_INT(m, SNDCTL_TMR_SELECT); +#endif +#ifdef SNDCTL_TMR_SOURCE _EXPORT_INT(m, SNDCTL_TMR_SOURCE); +#endif +#ifdef SNDCTL_TMR_START _EXPORT_INT(m, SNDCTL_TMR_START); +#endif +#ifdef SNDCTL_TMR_STOP _EXPORT_INT(m, SNDCTL_TMR_STOP); +#endif +#ifdef SNDCTL_TMR_TEMPO _EXPORT_INT(m, SNDCTL_TMR_TEMPO); +#endif +#ifdef SNDCTL_TMR_TIMEBASE _EXPORT_INT(m, SNDCTL_TMR_TIMEBASE); +#endif return m; } diff --git a/configure b/configure index 26a891ee7fa03..65495194f4935 100755 --- a/configure +++ b/configure @@ -12181,6 +12181,14 @@ LIBS=$save_LIBS fi +case $ac_sys_system in #( + NetBSD*) : + OSSAUDIODEV_LIBS="-lossaudio" ;; #( + *) : + OSSAUDIODEV_LIBS="" + ;; +esac + pkg_failed=no { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LIBSQLITE3" >&5 @@ -23250,7 +23258,7 @@ fi if test "x$py_cv_module_ossaudiodev" = xyes; then : - + as_fn_append MODULE_BLOCK "MODULE_OSSAUDIODEV_LDFLAGS=$OSSAUDIODEV_LIBS$as_nl" fi if test "$py_cv_module_ossaudiodev" = yes; then diff --git a/configure.ac b/configure.ac index 5c915d1a45cdc..276718aeeb77a 100644 --- a/configure.ac +++ b/configure.ac @@ -3553,6 +3553,12 @@ AS_VAR_IF([have_nis], [yes], [ ]) ]) +dnl On NetBSD, when using OSS audio, you need to link against libossaudio +AS_CASE([$ac_sys_system], + [NetBSD*], [OSSAUDIODEV_LIBS="-lossaudio"], + [OSSAUDIODEV_LIBS=""] +) + dnl Check for SQLite library. Use pkg-config if available. PKG_CHECK_MODULES( [LIBSQLITE3], [sqlite3 >= 3.7.15], [], [ @@ -6786,7 +6792,8 @@ PY_STDLIB_MOD([_socket], dnl platform specific extensions PY_STDLIB_MOD([grp], [], [test "$ac_cv_func_getgrgid" = yes -o "$ac_cv_func_getgrgid_r" = yes]) PY_STDLIB_MOD([ossaudiodev], - [], [test "$ac_cv_header_linux_soundcard_h" = yes -o "$ac_cv_header_sys_soundcard_h" = yes]) + [], [test "$ac_cv_header_linux_soundcard_h" = yes -o "$ac_cv_header_sys_soundcard_h" = yes], + [], [$OSSAUDIODEV_LIBS]) PY_STDLIB_MOD([pwd], [], [test "$ac_cv_func_getpwuid" = yes -o "$ac_cv_func_getpwuid_r" = yes]) PY_STDLIB_MOD([resource], [], [test "$ac_cv_header_sys_resource_h" = yes]) PY_STDLIB_MOD([_scproxy], From webhook-mailer at python.org Mon Apr 18 05:26:36 2022 From: webhook-mailer at python.org (serhiy-storchaka) Date: Mon, 18 Apr 2022 09:26:36 -0000 Subject: [Python-checkins] gh-91575: Update case-insensitive matching in re to the latest Unicode version (GH-91580) Message-ID: https://github.com/python/cpython/commit/1c2fcebf3c5e2ab41d376bb481834445617c8f3c commit: 1c2fcebf3c5e2ab41d376bb481834445617c8f3c branch: main author: Serhiy Storchaka committer: serhiy-storchaka date: 2022-04-18T12:26:30+03:00 summary: gh-91575: Update case-insensitive matching in re to the latest Unicode version (GH-91580) files: A Misc/NEWS.d/next/Library/2022-04-15-18-38-21.gh-issue-91575.fSyAxS.rst M Lib/re/_compiler.py M Lib/test/test_re.py diff --git a/Lib/re/_compiler.py b/Lib/re/_compiler.py index 5b4c4a3f30a40..065f6fbd73244 100644 --- a/Lib/re/_compiler.py +++ b/Lib/re/_compiler.py @@ -57,6 +57,22 @@ (0x3c2, 0x3c3), # ?? # GREEK SMALL LETTER PHI, GREEK PHI SYMBOL (0x3c6, 0x3d5), # ?? + # CYRILLIC SMALL LETTER VE, CYRILLIC SMALL LETTER ROUNDED VE + (0x432, 0x1c80), # ?? + # CYRILLIC SMALL LETTER DE, CYRILLIC SMALL LETTER LONG-LEGGED DE + (0x434, 0x1c81), # ?? + # CYRILLIC SMALL LETTER O, CYRILLIC SMALL LETTER NARROW O + (0x43e, 0x1c82), # ?? + # CYRILLIC SMALL LETTER ES, CYRILLIC SMALL LETTER WIDE ES + (0x441, 0x1c83), # ?? + # CYRILLIC SMALL LETTER TE, CYRILLIC SMALL LETTER TALL TE, CYRILLIC SMALL LETTER THREE-LEGGED TE + (0x442, 0x1c84, 0x1c85), # ??? + # CYRILLIC SMALL LETTER HARD SIGN, CYRILLIC SMALL LETTER TALL HARD SIGN + (0x44a, 0x1c86), # ?? + # CYRILLIC SMALL LETTER YAT, CYRILLIC SMALL LETTER TALL YAT + (0x463, 0x1c87), # ?? + # CYRILLIC SMALL LETTER UNBLENDED UK, CYRILLIC SMALL LETTER MONOGRAPH UK + (0x1c88, 0xa64b), # ?? # LATIN SMALL LETTER S WITH DOT ABOVE, LATIN SMALL LETTER LONG S WITH DOT ABOVE (0x1e61, 0x1e9b), # ?? # LATIN SMALL LIGATURE LONG S T, LATIN SMALL LIGATURE ST @@ -339,11 +355,19 @@ def _optimize_charset(charset, iscased=None, fixup=None, fixes=None): charmap += b'\0' * 0xff00 continue # Character set contains non-BMP character codes. + # For range, all BMP characters in the range are already + # proceeded. if fixup: hascased = True - # There are only two ranges of cased non-BMP characters: - # 10400-1044F (Deseret) and 118A0-118DF (Warang Citi), - # and for both ranges RANGE_UNI_IGNORE works. + # For now, IN_UNI_IGNORE+LITERAL and + # IN_UNI_IGNORE+RANGE_UNI_IGNORE work for all non-BMP + # characters, because two characters (at least one of + # which is not in the BMP) match case-insensitively + # if and only if: + # 1) c1.lower() == c2.lower() + # 2) c1.lower() == c2 or c1.lower().upper() == c2 + # Also, both c.lower() and c.lower().upper() are single + # characters for every non-BMP character. if op is RANGE: op = RANGE_UNI_IGNORE tail.append((op, av)) diff --git a/Lib/test/test_re.py b/Lib/test/test_re.py index 959582e2f1257..a1c27c9bc0f36 100644 --- a/Lib/test/test_re.py +++ b/Lib/test/test_re.py @@ -859,16 +859,30 @@ def test_ignore_case(self): self.assertEqual(re.match(r"((a)\s(abc|a))", "a a", re.I).group(1), "a a") self.assertEqual(re.match(r"((a)\s(abc|a)*)", "a aa", re.I).group(1), "a aa") - assert '\u212a'.lower() == 'k' # '?' + # Two different characters have the same lowercase. + assert 'K'.lower() == '\u212a'.lower() == 'k' # '?' self.assertTrue(re.match(r'K', '\u212a', re.I)) self.assertTrue(re.match(r'k', '\u212a', re.I)) self.assertTrue(re.match(r'\u212a', 'K', re.I)) self.assertTrue(re.match(r'\u212a', 'k', re.I)) - assert '\u017f'.upper() == 'S' # '?' + + # Two different characters have the same uppercase. + assert 's'.upper() == '\u017f'.upper() == 'S' # '?' self.assertTrue(re.match(r'S', '\u017f', re.I)) self.assertTrue(re.match(r's', '\u017f', re.I)) self.assertTrue(re.match(r'\u017f', 'S', re.I)) self.assertTrue(re.match(r'\u017f', 's', re.I)) + + # Two different characters have the same uppercase. Unicode 9.0+. + assert '\u0432'.upper() == '\u1c80'.upper() == '\u0412' # '?', '?', '?' + self.assertTrue(re.match(r'\u0412', '\u0432', re.I)) + self.assertTrue(re.match(r'\u0412', '\u1c80', re.I)) + self.assertTrue(re.match(r'\u0432', '\u0412', re.I)) + self.assertTrue(re.match(r'\u0432', '\u1c80', re.I)) + self.assertTrue(re.match(r'\u1c80', '\u0412', re.I)) + self.assertTrue(re.match(r'\u1c80', '\u0432', re.I)) + + # Two different characters have the same multicharacter uppercase. assert '\ufb05'.upper() == '\ufb06'.upper() == 'ST' # '?', '?' self.assertTrue(re.match(r'\ufb05', '\ufb06', re.I)) self.assertTrue(re.match(r'\ufb06', '\ufb05', re.I)) @@ -882,16 +896,31 @@ def test_ignore_case_set(self): self.assertTrue(re.match(br'[19a]', b'a', re.I)) self.assertTrue(re.match(br'[19a]', b'A', re.I)) self.assertTrue(re.match(br'[19A]', b'a', re.I)) - assert '\u212a'.lower() == 'k' # '?' + + # Two different characters have the same lowercase. + assert 'K'.lower() == '\u212a'.lower() == 'k' # '?' self.assertTrue(re.match(r'[19K]', '\u212a', re.I)) self.assertTrue(re.match(r'[19k]', '\u212a', re.I)) self.assertTrue(re.match(r'[19\u212a]', 'K', re.I)) self.assertTrue(re.match(r'[19\u212a]', 'k', re.I)) - assert '\u017f'.upper() == 'S' # '?' + + # Two different characters have the same uppercase. + assert 's'.upper() == '\u017f'.upper() == 'S' # '?' self.assertTrue(re.match(r'[19S]', '\u017f', re.I)) self.assertTrue(re.match(r'[19s]', '\u017f', re.I)) self.assertTrue(re.match(r'[19\u017f]', 'S', re.I)) self.assertTrue(re.match(r'[19\u017f]', 's', re.I)) + + # Two different characters have the same uppercase. Unicode 9.0+. + assert '\u0432'.upper() == '\u1c80'.upper() == '\u0412' # '?', '?', '?' + self.assertTrue(re.match(r'[19\u0412]', '\u0432', re.I)) + self.assertTrue(re.match(r'[19\u0412]', '\u1c80', re.I)) + self.assertTrue(re.match(r'[19\u0432]', '\u0412', re.I)) + self.assertTrue(re.match(r'[19\u0432]', '\u1c80', re.I)) + self.assertTrue(re.match(r'[19\u1c80]', '\u0412', re.I)) + self.assertTrue(re.match(r'[19\u1c80]', '\u0432', re.I)) + + # Two different characters have the same multicharacter uppercase. assert '\ufb05'.upper() == '\ufb06'.upper() == 'ST' # '?', '?' self.assertTrue(re.match(r'[19\ufb05]', '\ufb06', re.I)) self.assertTrue(re.match(r'[19\ufb06]', '\ufb05', re.I)) @@ -915,16 +944,30 @@ def test_ignore_case_range(self): self.assertTrue(re.match(r'[\U00010400-\U00010427]', '\U00010428', re.I)) self.assertTrue(re.match(r'[\U00010400-\U00010427]', '\U00010400', re.I)) - assert '\u212a'.lower() == 'k' # '?' + # Two different characters have the same lowercase. + assert 'K'.lower() == '\u212a'.lower() == 'k' # '?' self.assertTrue(re.match(r'[J-M]', '\u212a', re.I)) self.assertTrue(re.match(r'[j-m]', '\u212a', re.I)) self.assertTrue(re.match(r'[\u2129-\u212b]', 'K', re.I)) self.assertTrue(re.match(r'[\u2129-\u212b]', 'k', re.I)) - assert '\u017f'.upper() == 'S' # '?' + + # Two different characters have the same uppercase. + assert 's'.upper() == '\u017f'.upper() == 'S' # '?' self.assertTrue(re.match(r'[R-T]', '\u017f', re.I)) self.assertTrue(re.match(r'[r-t]', '\u017f', re.I)) self.assertTrue(re.match(r'[\u017e-\u0180]', 'S', re.I)) self.assertTrue(re.match(r'[\u017e-\u0180]', 's', re.I)) + + # Two different characters have the same uppercase. Unicode 9.0+. + assert '\u0432'.upper() == '\u1c80'.upper() == '\u0412' # '?', '?', '?' + self.assertTrue(re.match(r'[\u0411-\u0413]', '\u0432', re.I)) + self.assertTrue(re.match(r'[\u0411-\u0413]', '\u1c80', re.I)) + self.assertTrue(re.match(r'[\u0431-\u0433]', '\u0412', re.I)) + self.assertTrue(re.match(r'[\u0431-\u0433]', '\u1c80', re.I)) + self.assertTrue(re.match(r'[\u1c80-\u1c82]', '\u0412', re.I)) + self.assertTrue(re.match(r'[\u1c80-\u1c82]', '\u0432', re.I)) + + # Two different characters have the same multicharacter uppercase. assert '\ufb05'.upper() == '\ufb06'.upper() == 'ST' # '?', '?' self.assertTrue(re.match(r'[\ufb04-\ufb05]', '\ufb06', re.I)) self.assertTrue(re.match(r'[\ufb06-\ufb07]', '\ufb05', re.I)) diff --git a/Misc/NEWS.d/next/Library/2022-04-15-18-38-21.gh-issue-91575.fSyAxS.rst b/Misc/NEWS.d/next/Library/2022-04-15-18-38-21.gh-issue-91575.fSyAxS.rst new file mode 100644 index 0000000000000..ba046f2b4d61c --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-15-18-38-21.gh-issue-91575.fSyAxS.rst @@ -0,0 +1,2 @@ +Update case-insensitive matching in the :mod:`re` module to the latest +Unicode version. From webhook-mailer at python.org Mon Apr 18 09:50:47 2022 From: webhook-mailer at python.org (serhiy-storchaka) Date: Mon, 18 Apr 2022 13:50:47 -0000 Subject: [Python-checkins] bpo-47256: Increasing the depth of backtracking in RE (GH-32411) Message-ID: https://github.com/python/cpython/commit/a29f858124bc698f6604716b73306c65b63b5054 commit: a29f858124bc698f6604716b73306c65b63b5054 branch: main author: Ma Lin committer: serhiy-storchaka date: 2022-04-18T16:50:40+03:00 summary: bpo-47256: Increasing the depth of backtracking in RE (GH-32411) Limit the maximum capturing group to 2**30-1 on 64-bit platforms (it was 2**31-1). No change on 32-bit platforms (2**28-1). It allows to reduce the size of SRE(match_context): - On 32 bit platform: 36 bytes, no change. (msvc2022) - On 64 bit platform: 72 bytes -> 56 bytes. (msvc2022/gcc9.4) which leads to increasing the depth of backtracking. files: A Misc/NEWS.d/next/Library/2022-04-16-11-39-59.bpo-47256.1cygyd.rst M Modules/_sre/sre.h M Modules/_sre/sre_lib.h diff --git a/Misc/NEWS.d/next/Library/2022-04-16-11-39-59.bpo-47256.1cygyd.rst b/Misc/NEWS.d/next/Library/2022-04-16-11-39-59.bpo-47256.1cygyd.rst new file mode 100644 index 0000000000000..ac4c52bd7058a --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-16-11-39-59.bpo-47256.1cygyd.rst @@ -0,0 +1,2 @@ +:mod:`re` module, limit the maximum capturing group to 1,073,741,823 in +64-bit build, this increases the depth of backtracking. diff --git a/Modules/_sre/sre.h b/Modules/_sre/sre.h index 129f5595269f5..aff064d343ec4 100644 --- a/Modules/_sre/sre.h +++ b/Modules/_sre/sre.h @@ -18,10 +18,10 @@ #define SRE_CODE Py_UCS4 #if SIZEOF_SIZE_T > 4 # define SRE_MAXREPEAT (~(SRE_CODE)0) -# define SRE_MAXGROUPS ((~(SRE_CODE)0) / 2) +# define SRE_MAXGROUPS ((SRE_CODE)INT32_MAX / 2) #else # define SRE_MAXREPEAT ((SRE_CODE)PY_SSIZE_T_MAX) -# define SRE_MAXGROUPS ((SRE_CODE)PY_SSIZE_T_MAX / SIZEOF_SIZE_T / 2) +# define SRE_MAXGROUPS ((SRE_CODE)PY_SSIZE_T_MAX / SIZEOF_VOID_P / 2) #endif typedef struct { @@ -73,12 +73,12 @@ typedef struct { Py_ssize_t pos, endpos; int isbytes; int charsize; /* character size */ - /* registers */ - Py_ssize_t lastindex; - Py_ssize_t lastmark; - const void** mark; int match_all; int must_advance; + /* marks */ + int lastmark; + int lastindex; + const void** mark; /* dynamically allocated stuff */ char* data_stack; size_t data_stack_size; diff --git a/Modules/_sre/sre_lib.h b/Modules/_sre/sre_lib.h index 3472e65b87ae6..db624aa896d6a 100644 --- a/Modules/_sre/sre_lib.h +++ b/Modules/_sre/sre_lib.h @@ -450,20 +450,23 @@ do { \ #define MARK_PUSH(lastmark) \ do if (lastmark >= 0) { \ - i = lastmark; /* ctx->lastmark may change if reallocated */ \ - DATA_STACK_PUSH(state, state->mark, (i+1)*sizeof(void*)); \ + size_t _marks_size = (lastmark+1) * sizeof(void*); \ + DATA_STACK_PUSH(state, state->mark, _marks_size); \ } while (0) #define MARK_POP(lastmark) \ do if (lastmark >= 0) { \ - DATA_STACK_POP(state, state->mark, (lastmark+1)*sizeof(void*), 1); \ + size_t _marks_size = (lastmark+1) * sizeof(void*); \ + DATA_STACK_POP(state, state->mark, _marks_size, 1); \ } while (0) #define MARK_POP_KEEP(lastmark) \ do if (lastmark >= 0) { \ - DATA_STACK_POP(state, state->mark, (lastmark+1)*sizeof(void*), 0); \ + size_t _marks_size = (lastmark+1) * sizeof(void*); \ + DATA_STACK_POP(state, state->mark, _marks_size, 0); \ } while (0) #define MARK_POP_DISCARD(lastmark) \ do if (lastmark >= 0) { \ - DATA_STACK_POP_DISCARD(state, (lastmark+1)*sizeof(void*)); \ + size_t _marks_size = (lastmark+1) * sizeof(void*); \ + DATA_STACK_POP_DISCARD(state, _marks_size); \ } while (0) #define JUMP_NONE 0 @@ -488,10 +491,10 @@ do { \ ctx->pattern = pattern; \ ctx->ptr = ptr; \ DATA_ALLOC(SRE(match_context), nextctx); \ - nextctx->last_ctx_pos = ctx_pos; \ - nextctx->jump = jumpvalue; \ nextctx->pattern = nextpattern; \ nextctx->toplevel = toplevel_; \ + nextctx->jump = jumpvalue; \ + nextctx->last_ctx_pos = ctx_pos; \ pattern = nextpattern; \ ctx_pos = alloc_pos; \ ctx = nextctx; \ @@ -507,18 +510,18 @@ do { \ DO_JUMPX(jumpvalue, jumplabel, nextpattern, 0) typedef struct { - Py_ssize_t last_ctx_pos; - Py_ssize_t jump; - const SRE_CHAR* ptr; - const SRE_CODE* pattern; Py_ssize_t count; - Py_ssize_t lastmark; - Py_ssize_t lastindex; union { SRE_CODE chr; SRE_REPEAT* rep; } u; + int lastmark; + int lastindex; + const SRE_CODE* pattern; + const SRE_CHAR* ptr; int toplevel; + int jump; + Py_ssize_t last_ctx_pos; } SRE(match_context); #define MAYBE_CHECK_SIGNALS \ @@ -558,8 +561,8 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) { const SRE_CHAR* end = (const SRE_CHAR *)state->end; Py_ssize_t alloc_pos, ctx_pos = -1; - Py_ssize_t i, ret = 0; - Py_ssize_t jump; + Py_ssize_t ret = 0; + int jump; unsigned int sigcount=0; SRE(match_context)* ctx; @@ -607,20 +610,22 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) /* */ TRACE(("|%p|%p|MARK %d\n", pattern, ptr, pattern[0])); - i = pattern[0]; - if (i & 1) - state->lastindex = i/2 + 1; - if (i > state->lastmark) { - /* state->lastmark is the highest valid index in the - state->mark array. If it is increased by more than 1, - the intervening marks must be set to NULL to signal - that these marks have not been encountered. */ - Py_ssize_t j = state->lastmark + 1; - while (j < i) - state->mark[j++] = NULL; - state->lastmark = i; + { + int i = pattern[0]; + if (i & 1) + state->lastindex = i/2 + 1; + if (i > state->lastmark) { + /* state->lastmark is the highest valid index in the + state->mark array. If it is increased by more than 1, + the intervening marks must be set to NULL to signal + that these marks have not been encountered. */ + int j = state->lastmark + 1; + while (j < i) + state->mark[j++] = NULL; + state->lastmark = i; + } + state->mark[i] = ptr; } - state->mark[i] = ptr; pattern++; DISPATCH; @@ -1373,9 +1378,8 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) /* match backreference */ TRACE(("|%p|%p|GROUPREF %d\n", pattern, ptr, pattern[0])); - i = pattern[0]; { - Py_ssize_t groupref = i+i; + int groupref = pattern[0] * 2; if (groupref >= state->lastmark) { RETURN_FAILURE; } else { @@ -1398,9 +1402,8 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) /* match backreference */ TRACE(("|%p|%p|GROUPREF_IGNORE %d\n", pattern, ptr, pattern[0])); - i = pattern[0]; { - Py_ssize_t groupref = i+i; + int groupref = pattern[0] * 2; if (groupref >= state->lastmark) { RETURN_FAILURE; } else { @@ -1424,9 +1427,8 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) /* match backreference */ TRACE(("|%p|%p|GROUPREF_UNI_IGNORE %d\n", pattern, ptr, pattern[0])); - i = pattern[0]; { - Py_ssize_t groupref = i+i; + int groupref = pattern[0] * 2; if (groupref >= state->lastmark) { RETURN_FAILURE; } else { @@ -1450,9 +1452,8 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) /* match backreference */ TRACE(("|%p|%p|GROUPREF_LOC_IGNORE %d\n", pattern, ptr, pattern[0])); - i = pattern[0]; { - Py_ssize_t groupref = i+i; + int groupref = pattern[0] * 2; if (groupref >= state->lastmark) { RETURN_FAILURE; } else { @@ -1476,9 +1477,8 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) TRACE(("|%p|%p|GROUPREF_EXISTS %d\n", pattern, ptr, pattern[0])); /* codeyes codeno ... */ - i = pattern[0]; { - Py_ssize_t groupref = i+i; + int groupref = pattern[0] * 2; if (groupref >= state->lastmark) { pattern += pattern[1]; DISPATCH; From webhook-mailer at python.org Mon Apr 18 10:18:39 2022 From: webhook-mailer at python.org (gvanrossum) Date: Mon, 18 Apr 2022 14:18:39 -0000 Subject: [Python-checkins] gh-91576: Speed up iteration of strings (#91574) Message-ID: https://github.com/python/cpython/commit/8c54c3dacccb12a712acaa48d86a54f9ee9e37b5 commit: 8c54c3dacccb12a712acaa48d86a54f9ee9e37b5 branch: main author: Kumar Aditya <59607654+kumaraditya303 at users.noreply.github.com> committer: gvanrossum date: 2022-04-18T07:18:27-07:00 summary: gh-91576: Speed up iteration of strings (#91574) files: A Misc/NEWS.d/next/Core and Builtins/2022-04-15-16-57-23.gh-issue-91576.adoDj_.rst M Include/internal/pycore_unicodeobject.h M Lib/test/test_unicode.py M Objects/object.c M Objects/unicodeobject.c diff --git a/Include/internal/pycore_unicodeobject.h b/Include/internal/pycore_unicodeobject.h index c7f06051a622f..75b90501db156 100644 --- a/Include/internal/pycore_unicodeobject.h +++ b/Include/internal/pycore_unicodeobject.h @@ -20,6 +20,7 @@ extern void _PyUnicode_Fini(PyInterpreterState *); extern void _PyUnicode_FiniTypes(PyInterpreterState *); extern void _PyStaticUnicode_Dealloc(PyObject *); +extern PyTypeObject _PyUnicodeASCIIIter_Type; /* other API */ diff --git a/Lib/test/test_unicode.py b/Lib/test/test_unicode.py index df7afd5046a56..c98fabf8bc9b5 100644 --- a/Lib/test/test_unicode.py +++ b/Lib/test/test_unicode.py @@ -9,6 +9,7 @@ import codecs import itertools import operator +import pickle import struct import sys import textwrap @@ -185,6 +186,36 @@ def test_iterators(self): self.assertEqual(next(it), "\u3333") self.assertRaises(StopIteration, next, it) + def test_iterators_invocation(self): + cases = [type(iter('abc')), type(iter('?'))] + for cls in cases: + with self.subTest(cls=cls): + self.assertRaises(TypeError, cls) + + def test_iteration(self): + cases = ['abc', '???', "\u1111\u2222\u3333"] + for case in cases: + with self.subTest(string=case): + self.assertEqual(case, "".join(iter(case))) + + def test_exhausted_iterator(self): + cases = ['abc', '???', "\u1111\u2222\u3333"] + for case in cases: + with self.subTest(case=case): + iterator = iter(case) + tuple(iterator) + self.assertRaises(StopIteration, next, iterator) + + def test_pickle_iterator(self): + cases = ['abc', '???', "\u1111\u2222\u3333"] + for case in cases: + with self.subTest(case=case): + for proto in range(pickle.HIGHEST_PROTOCOL + 1): + it = iter(case) + with self.subTest(proto=proto): + pickled = "".join(pickle.loads(pickle.dumps(it, proto))) + self.assertEqual(case, pickled) + def test_count(self): string_tests.CommonTest.test_count(self) # check mixed argument types diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-04-15-16-57-23.gh-issue-91576.adoDj_.rst b/Misc/NEWS.d/next/Core and Builtins/2022-04-15-16-57-23.gh-issue-91576.adoDj_.rst new file mode 100644 index 0000000000000..b792f3e48f29b --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2022-04-15-16-57-23.gh-issue-91576.adoDj_.rst @@ -0,0 +1 @@ +Speed up iteration of ascii strings by 50%. Patch by Kumar Aditya. diff --git a/Objects/object.c b/Objects/object.c index 33dab5ecbf205..fe2d76f578e2a 100644 --- a/Objects/object.c +++ b/Objects/object.c @@ -1936,6 +1936,7 @@ static PyTypeObject* static_types[] = { &_PyNamespace_Type, &_PyNone_Type, &_PyNotImplemented_Type, + &_PyUnicodeASCIIIter_Type, &_PyUnion_Type, &_PyWeakref_CallableProxyType, &_PyWeakref_ProxyType, diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c index d35a671a81680..6b05c37faabfc 100644 --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -15697,7 +15697,7 @@ unicodeiter_traverse(unicodeiterobject *it, visitproc visit, void *arg) static PyObject * unicodeiter_next(unicodeiterobject *it) { - PyObject *seq, *item; + PyObject *seq; assert(it != NULL); seq = it->it_seq; @@ -15709,10 +15709,8 @@ unicodeiter_next(unicodeiterobject *it) int kind = PyUnicode_KIND(seq); const void *data = PyUnicode_DATA(seq); Py_UCS4 chr = PyUnicode_READ(kind, data, it->it_index); - item = PyUnicode_FromOrdinal(chr); - if (item != NULL) - ++it->it_index; - return item; + it->it_index++; + return unicode_char(chr); } it->it_seq = NULL; @@ -15720,6 +15718,29 @@ unicodeiter_next(unicodeiterobject *it) return NULL; } +static PyObject * +unicode_ascii_iter_next(unicodeiterobject *it) +{ + assert(it != NULL); + PyObject *seq = it->it_seq; + if (seq == NULL) { + return NULL; + } + assert(_PyUnicode_CHECK(seq)); + assert(PyUnicode_IS_COMPACT_ASCII(seq)); + if (it->it_index < PyUnicode_GET_LENGTH(seq)) { + const void *data = ((void*)(_PyASCIIObject_CAST(seq) + 1)); + Py_UCS1 chr = (Py_UCS1)PyUnicode_READ(PyUnicode_1BYTE_KIND, + data, it->it_index); + it->it_index++; + PyObject *item = (PyObject*)&_Py_SINGLETON(strings).ascii[chr]; + return Py_NewRef(item); + } + it->it_seq = NULL; + Py_DECREF(seq); + return NULL; +} + static PyObject * unicodeiter_len(unicodeiterobject *it, PyObject *Py_UNUSED(ignored)) { @@ -15808,6 +15829,19 @@ PyTypeObject PyUnicodeIter_Type = { 0, }; +PyTypeObject _PyUnicodeASCIIIter_Type = { + PyVarObject_HEAD_INIT(&PyType_Type, 0) + .tp_name = "str_ascii_iterator", + .tp_basicsize = sizeof(unicodeiterobject), + .tp_dealloc = (destructor)unicodeiter_dealloc, + .tp_getattro = PyObject_GenericGetAttr, + .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC, + .tp_traverse = (traverseproc)unicodeiter_traverse, + .tp_iter = PyObject_SelfIter, + .tp_iternext = (iternextfunc)unicode_ascii_iter_next, + .tp_methods = unicodeiter_methods, +}; + static PyObject * unicode_iter(PyObject *seq) { @@ -15819,7 +15853,12 @@ unicode_iter(PyObject *seq) } if (PyUnicode_READY(seq) == -1) return NULL; - it = PyObject_GC_New(unicodeiterobject, &PyUnicodeIter_Type); + if (PyUnicode_IS_COMPACT_ASCII(seq)) { + it = PyObject_GC_New(unicodeiterobject, &_PyUnicodeASCIIIter_Type); + } + else { + it = PyObject_GC_New(unicodeiterobject, &PyUnicodeIter_Type); + } if (it == NULL) return NULL; it->it_index = 0; From webhook-mailer at python.org Mon Apr 18 10:35:10 2022 From: webhook-mailer at python.org (warsaw) Date: Mon, 18 Apr 2022 14:35:10 -0000 Subject: [Python-checkins] Add some information about where pynche has moved to (#91650) Message-ID: https://github.com/python/cpython/commit/dbd9d75fedfebb215fe3e2b60249ac642cc8abf2 commit: dbd9d75fedfebb215fe3e2b60249ac642cc8abf2 branch: main author: Barry Warsaw committer: warsaw date: 2022-04-18T07:34:53-07:00 summary: Add some information about where pynche has moved to (#91650) files: M Misc/NEWS.d/next/Tools-Demos/2022-04-14-18-11-46.gh-issue-91551.l_nNT-.rst M Tools/README diff --git a/Misc/NEWS.d/next/Tools-Demos/2022-04-14-18-11-46.gh-issue-91551.l_nNT-.rst b/Misc/NEWS.d/next/Tools-Demos/2022-04-14-18-11-46.gh-issue-91551.l_nNT-.rst index f0c1866ac0e3c..95ee25f6e02ce 100644 --- a/Misc/NEWS.d/next/Tools-Demos/2022-04-14-18-11-46.gh-issue-91551.l_nNT-.rst +++ b/Misc/NEWS.d/next/Tools-Demos/2022-04-14-18-11-46.gh-issue-91551.l_nNT-.rst @@ -1 +1 @@ -Remove the ancient Pynche color editor. +Remove the ancient Pynche color editor. It has moved to https://gitlab.com/warsaw/pynche diff --git a/Tools/README b/Tools/README index 862bba7edb825..f8bfbc0479581 100644 --- a/Tools/README +++ b/Tools/README @@ -42,3 +42,5 @@ unittestgui A Tkinter based GUI test runner for unittest, with test (*) A generic benchmark suite is maintained separately at https://github.com/python/performance + +Note: The pynche color editor has moved to https://gitlab.com/warsaw/pynche From webhook-mailer at python.org Mon Apr 18 16:44:12 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Mon, 18 Apr 2022 20:44:12 -0000 Subject: [Python-checkins] gh-91102: Use Argument Clinic for EncodingMap (#31725) Message-ID: https://github.com/python/cpython/commit/2f0fc521f48af0dcec86dfcc8d0bc91eebf974ee commit: 2f0fc521f48af0dcec86dfcc8d0bc91eebf974ee branch: main author: Oleg Iarygin committer: JelleZijlstra date: 2022-04-18T13:43:56-07:00 summary: gh-91102: Use Argument Clinic for EncodingMap (#31725) Co-authored-by: Jelle Zijlstra files: A Misc/NEWS.d/next/Core and Builtins/2022-04-18-07-23-48.gh-issue-91102.vm-6g1.rst M Objects/clinic/unicodeobject.c.h M Objects/unicodeobject.c diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-04-18-07-23-48.gh-issue-91102.vm-6g1.rst b/Misc/NEWS.d/next/Core and Builtins/2022-04-18-07-23-48.gh-issue-91102.vm-6g1.rst new file mode 100644 index 0000000000000..5f3897e6dcd03 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2022-04-18-07-23-48.gh-issue-91102.vm-6g1.rst @@ -0,0 +1 @@ +Use Argument Clinic for :class:`EncodingMap`. Patch by Oleg Iarygin. diff --git a/Objects/clinic/unicodeobject.c.h b/Objects/clinic/unicodeobject.c.h index 803b5f2353f9e..b9892403a2036 100644 --- a/Objects/clinic/unicodeobject.c.h +++ b/Objects/clinic/unicodeobject.c.h @@ -2,6 +2,24 @@ preserve [clinic start generated code]*/ +PyDoc_STRVAR(EncodingMap_size__doc__, +"size($self, /)\n" +"--\n" +"\n" +"Return the size (in bytes) of this object."); + +#define ENCODINGMAP_SIZE_METHODDEF \ + {"size", (PyCFunction)EncodingMap_size, METH_NOARGS, EncodingMap_size__doc__}, + +static PyObject * +EncodingMap_size_impl(struct encoding_map *self); + +static PyObject * +EncodingMap_size(struct encoding_map *self, PyObject *Py_UNUSED(ignored)) +{ + return EncodingMap_size_impl(self); +} + PyDoc_STRVAR(unicode_title__doc__, "title($self, /)\n" "--\n" @@ -1335,4 +1353,4 @@ unicode_new(PyTypeObject *type, PyObject *args, PyObject *kwargs) exit: return return_value; } -/*[clinic end generated code: output=c494bed46209961d input=a9049054013a1b77]*/ +/*[clinic end generated code: output=e8566b060f558f72 input=a9049054013a1b77]*/ diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c index 6b05c37faabfc..d46a52cdb8965 100644 --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -397,6 +397,7 @@ static const unsigned char ascii_linebreak[] = { static int convert_uc(PyObject *obj, void *addr); +struct encoding_map; #include "clinic/unicodeobject.c.h" _Py_error_handler @@ -8331,6 +8332,11 @@ PyUnicode_DecodeCharmap(const char *s, /* Charmap encoding: the lookup table */ +/*[clinic input] +class EncodingMap "struct encoding_map *" "&EncodingMapType" +[clinic start generated code]*/ +/*[clinic end generated code: output=da39a3ee5e6b4b0d input=14e46bbb6c522d22]*/ + struct encoding_map { PyObject_HEAD unsigned char level1[32]; @@ -8338,62 +8344,32 @@ struct encoding_map { unsigned char level23[1]; }; -static PyObject* -encoding_map_size(PyObject *obj, PyObject* args) +/*[clinic input] +EncodingMap.size + +Return the size (in bytes) of this object. +[clinic start generated code]*/ + +static PyObject * +EncodingMap_size_impl(struct encoding_map *self) +/*[clinic end generated code: output=c4c969e4c99342a4 input=004ff13f26bb5366]*/ { - struct encoding_map *map = (struct encoding_map*)obj; - return PyLong_FromLong(sizeof(*map) - 1 + 16*map->count2 + - 128*map->count3); + return PyLong_FromLong((sizeof(*self) - 1) + 16*self->count2 + + 128*self->count3); } static PyMethodDef encoding_map_methods[] = { - {"size", encoding_map_size, METH_NOARGS, - PyDoc_STR("Return the size (in bytes) of this object") }, - { 0 } + ENCODINGMAP_SIZE_METHODDEF + {NULL, NULL} }; static PyTypeObject EncodingMapType = { PyVarObject_HEAD_INIT(NULL, 0) - "EncodingMap", /*tp_name*/ - sizeof(struct encoding_map), /*tp_basicsize*/ - 0, /*tp_itemsize*/ + .tp_name = "EncodingMap", + .tp_basicsize = sizeof(struct encoding_map), /* methods */ - 0, /*tp_dealloc*/ - 0, /*tp_vectorcall_offset*/ - 0, /*tp_getattr*/ - 0, /*tp_setattr*/ - 0, /*tp_as_async*/ - 0, /*tp_repr*/ - 0, /*tp_as_number*/ - 0, /*tp_as_sequence*/ - 0, /*tp_as_mapping*/ - 0, /*tp_hash*/ - 0, /*tp_call*/ - 0, /*tp_str*/ - 0, /*tp_getattro*/ - 0, /*tp_setattro*/ - 0, /*tp_as_buffer*/ - Py_TPFLAGS_DEFAULT, /*tp_flags*/ - 0, /*tp_doc*/ - 0, /*tp_traverse*/ - 0, /*tp_clear*/ - 0, /*tp_richcompare*/ - 0, /*tp_weaklistoffset*/ - 0, /*tp_iter*/ - 0, /*tp_iternext*/ - encoding_map_methods, /*tp_methods*/ - 0, /*tp_members*/ - 0, /*tp_getset*/ - 0, /*tp_base*/ - 0, /*tp_dict*/ - 0, /*tp_descr_get*/ - 0, /*tp_descr_set*/ - 0, /*tp_dictoffset*/ - 0, /*tp_init*/ - 0, /*tp_alloc*/ - 0, /*tp_new*/ - 0, /*tp_free*/ - 0, /*tp_is_gc*/ + .tp_flags = Py_TPFLAGS_DEFAULT, + .tp_methods = encoding_map_methods, }; PyObject* From webhook-mailer at python.org Mon Apr 18 17:01:29 2022 From: webhook-mailer at python.org (warsaw) Date: Mon, 18 Apr 2022 21:01:29 -0000 Subject: [Python-checkins] gh-91670: Removes `SO` config variable in `sysconfig.py` (#91671) Message-ID: https://github.com/python/cpython/commit/0860b26a4f7abeb61aad9f4b96de8e78eed8c12a commit: 0860b26a4f7abeb61aad9f4b96de8e78eed8c12a branch: main author: Bryan Weber committer: warsaw date: 2022-04-18T14:01:18-07:00 summary: gh-91670: Removes `SO` config variable in `sysconfig.py` (#91671) * Removes SO config variable in sysconfig Per @warsaw in https://github.com/python/cpython/issues/63754, this was deprecated in Python 3.4 and was suggested for removal in Python 3.5. * Add NEWS * Update Misc/NEWS.d/next/Library/2022-04-18-15-23-24.gh-issue-91670.6eyChw.rst Co-authored-by: ?ric Co-authored-by: Barry Warsaw Co-authored-by: ?ric files: A Misc/NEWS.d/next/Library/2022-04-18-15-23-24.gh-issue-91670.6eyChw.rst M Lib/distutils/tests/test_sysconfig.py M Lib/sysconfig.py M Lib/test/test_sysconfig.py diff --git a/Lib/distutils/tests/test_sysconfig.py b/Lib/distutils/tests/test_sysconfig.py index 7a88c88f6cdc50..e7d435f412db79 100644 --- a/Lib/distutils/tests/test_sysconfig.py +++ b/Lib/distutils/tests/test_sysconfig.py @@ -227,26 +227,6 @@ def test_sysconfig_compiler_vars(self): self.assertEqual(global_sysconfig.get_config_var('CC'), sysconfig.get_config_var('CC')) - @unittest.skipIf(sysconfig.get_config_var('EXT_SUFFIX') is None, - 'EXT_SUFFIX required for this test') - def test_SO_deprecation(self): - self.assertWarns(DeprecationWarning, - sysconfig.get_config_var, 'SO') - - @unittest.skipIf(sysconfig.get_config_var('EXT_SUFFIX') is None, - 'EXT_SUFFIX required for this test') - def test_SO_value(self): - with check_warnings(('', DeprecationWarning)): - self.assertEqual(sysconfig.get_config_var('SO'), - sysconfig.get_config_var('EXT_SUFFIX')) - - @unittest.skipIf(sysconfig.get_config_var('EXT_SUFFIX') is None, - 'EXT_SUFFIX required for this test') - def test_SO_in_vars(self): - vars = sysconfig.get_config_vars() - self.assertIsNotNone(vars['SO']) - self.assertEqual(vars['SO'], vars['EXT_SUFFIX']) - @requires_subprocess() def test_customize_compiler_before_get_config_vars(self): # Issue #21923: test that a Distribution compiler diff --git a/Lib/sysconfig.py b/Lib/sysconfig.py index 2a01342eda8d6c..e21b7303fecc6b 100644 --- a/Lib/sysconfig.py +++ b/Lib/sysconfig.py @@ -667,10 +667,6 @@ def get_config_vars(*args): _CONFIG_VARS['VPATH'] = sys._vpath if os.name == 'posix': _init_posix(_CONFIG_VARS) - # For backward compatibility, see issue19555 - SO = _CONFIG_VARS.get('EXT_SUFFIX') - if SO is not None: - _CONFIG_VARS['SO'] = SO if _HAS_USER_BASE: # Setting 'userbase' is done below the call to the # init function to enable using 'get_config_var' in @@ -715,9 +711,6 @@ def get_config_var(name): Equivalent to get_config_vars().get(name) """ - if name == 'SO': - import warnings - warnings.warn('SO is deprecated, use EXT_SUFFIX', DeprecationWarning, 2) return get_config_vars().get(name) diff --git a/Lib/test/test_sysconfig.py b/Lib/test/test_sysconfig.py index c7ec78fa4dc814..f2b93706b270a7 100644 --- a/Lib/test/test_sysconfig.py +++ b/Lib/test/test_sysconfig.py @@ -463,19 +463,6 @@ def test_srcdir_independent_of_cwd(self): srcdir2 = sysconfig.get_config_var('srcdir') self.assertEqual(srcdir, srcdir2) - @unittest.skipIf(sysconfig.get_config_var('EXT_SUFFIX') is None, - 'EXT_SUFFIX required for this test') - def test_SO_deprecation(self): - self.assertWarns(DeprecationWarning, - sysconfig.get_config_var, 'SO') - - @unittest.skipIf(sysconfig.get_config_var('EXT_SUFFIX') is None, - 'EXT_SUFFIX required for this test') - def test_SO_value(self): - with check_warnings(('', DeprecationWarning)): - self.assertEqual(sysconfig.get_config_var('SO'), - sysconfig.get_config_var('EXT_SUFFIX')) - @unittest.skipIf(sysconfig.get_config_var('EXT_SUFFIX') is None, 'EXT_SUFFIX required for this test') def test_EXT_SUFFIX_in_vars(self): @@ -483,8 +470,6 @@ def test_EXT_SUFFIX_in_vars(self): if not _imp.extension_suffixes(): self.skipTest("stub loader has no suffixes") vars = sysconfig.get_config_vars() - self.assertIsNotNone(vars['SO']) - self.assertEqual(vars['SO'], vars['EXT_SUFFIX']) self.assertEqual(vars['EXT_SUFFIX'], _imp.extension_suffixes()[0]) @unittest.skipUnless(sys.platform == 'linux' and diff --git a/Misc/NEWS.d/next/Library/2022-04-18-15-23-24.gh-issue-91670.6eyChw.rst b/Misc/NEWS.d/next/Library/2022-04-18-15-23-24.gh-issue-91670.6eyChw.rst new file mode 100644 index 00000000000000..38ba32db081d5f --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-18-15-23-24.gh-issue-91670.6eyChw.rst @@ -0,0 +1 @@ +Remove deprecated ``SO`` config variable in :mod:`sysconfig`. From webhook-mailer at python.org Mon Apr 18 18:15:45 2022 From: webhook-mailer at python.org (iritkatriel) Date: Mon, 18 Apr 2022 22:15:45 -0000 Subject: [Python-checkins] gh-74166: Add option to get all errors from socket.create_connection (GH-91586) Message-ID: https://github.com/python/cpython/commit/39a54ba63850e081a4a5551a773df5b4d5b1d3cd commit: 39a54ba63850e081a4a5551a773df5b4d5b1d3cd branch: main author: Irit Katriel <1055913+iritkatriel at users.noreply.github.com> committer: iritkatriel <1055913+iritkatriel at users.noreply.github.com> date: 2022-04-18T23:15:41+01:00 summary: gh-74166: Add option to get all errors from socket.create_connection (GH-91586) files: A Misc/NEWS.d/next/Library/2022-04-15-20-56-31.gh-issue-74166.70KlvL.rst M Doc/library/socket.rst M Doc/whatsnew/3.11.rst M Lib/socket.py M Lib/test/test_socket.py diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst index f8392ba1254b75..d7a440127ec930 100755 --- a/Doc/library/socket.rst +++ b/Doc/library/socket.rst @@ -660,7 +660,7 @@ The following functions all create :ref:`socket objects `. Windows support added. -.. function:: create_connection(address[, timeout[, source_address]]) +.. function:: create_connection(address[, timeout[, source_address[, all_errors]]]) Connect to a TCP service listening on the internet *address* (a 2-tuple ``(host, port)``), and return the socket object. This is a higher-level @@ -679,9 +679,18 @@ The following functions all create :ref:`socket objects `. socket to bind to as its source address before connecting. If host or port are '' or 0 respectively the OS default behavior will be used. + When a connection cannot be created, an exception is raised. By default, + it is the exception from the last address in the list. If *all_errors* + is ``True``, it is an :exc:`ExceptionGroup` containing the errors of all + attempts. + .. versionchanged:: 3.2 *source_address* was added. + .. versionchanged:: 3.11 + *all_errors* was added. + + .. function:: create_server(address, *, family=AF_INET, backlog=None, reuse_port=False, dualstack_ipv6=False) Convenience function which creates a TCP socket bound to *address* (a 2-tuple diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index 27a5a0fc3d6871..baff6873991acb 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -357,6 +357,10 @@ socket * Add CAN Socket support for NetBSD. (Contributed by Thomas Klausner in :issue:`30512`.) +* :meth:`~socket.create_connection` has an option to raise, in case of + failure to connect, an :exc:`ExceptionGroup` containing all errors + instead of only raising the last error. + (Contributed by Irit Katriel in :issue:`29980`). sqlite3 ------- diff --git a/Lib/socket.py b/Lib/socket.py index ef82c496cb85a4..97362d92f64656 100755 --- a/Lib/socket.py +++ b/Lib/socket.py @@ -806,7 +806,7 @@ def getfqdn(name=''): _GLOBAL_DEFAULT_TIMEOUT = object() def create_connection(address, timeout=_GLOBAL_DEFAULT_TIMEOUT, - source_address=None): + source_address=None, all_errors=False): """Connect to *address* and return the socket object. Convenience function. Connect to *address* (a 2-tuple ``(host, @@ -816,11 +816,13 @@ def create_connection(address, timeout=_GLOBAL_DEFAULT_TIMEOUT, global default timeout setting returned by :func:`getdefaulttimeout` is used. If *source_address* is set it must be a tuple of (host, port) for the socket to bind as a source address before making the connection. - A host of '' or port 0 tells the OS to use the default. + A host of '' or port 0 tells the OS to use the default. When a connection + cannot be created, raises the last error if *all_errors* is False, + and an ExceptionGroup of all errors if *all_errors* is True. """ host, port = address - err = None + exceptions = [] for res in getaddrinfo(host, port, 0, SOCK_STREAM): af, socktype, proto, canonname, sa = res sock = None @@ -832,20 +834,24 @@ def create_connection(address, timeout=_GLOBAL_DEFAULT_TIMEOUT, sock.bind(source_address) sock.connect(sa) # Break explicitly a reference cycle - err = None + exceptions.clear() return sock - except error as _: - err = _ + except error as exc: + if not all_errors: + exceptions.clear() # raise only the last error + exceptions.append(exc) if sock is not None: sock.close() - if err is not None: + if len(exceptions): try: - raise err + if not all_errors: + raise exceptions[0] + raise ExceptionGroup("create_connection failed", exceptions) finally: # Break explicitly a reference cycle - err = None + exceptions = None else: raise error("getaddrinfo returns an empty list") diff --git a/Lib/test/test_socket.py b/Lib/test/test_socket.py index 97cc6260c3c2a5..613363722cf028 100755 --- a/Lib/test/test_socket.py +++ b/Lib/test/test_socket.py @@ -5177,6 +5177,24 @@ def test_create_connection(self): expected_errnos = socket_helper.get_socket_conn_refused_errs() self.assertIn(cm.exception.errno, expected_errnos) + def test_create_connection_all_errors(self): + port = socket_helper.find_unused_port() + try: + socket.create_connection((HOST, port), all_errors=True) + except ExceptionGroup as e: + eg = e + else: + self.fail('expected connection to fail') + + self.assertIsInstance(eg, ExceptionGroup) + for e in eg.exceptions: + self.assertIsInstance(e, OSError) + + addresses = socket.getaddrinfo( + 'localhost', port, 0, socket.SOCK_STREAM) + # assert that we got an exception for each address + self.assertEqual(len(addresses), len(eg.exceptions)) + def test_create_connection_timeout(self): # Issue #9792: create_connection() should not recast timeout errors # as generic socket errors. diff --git a/Misc/NEWS.d/next/Library/2022-04-15-20-56-31.gh-issue-74166.70KlvL.rst b/Misc/NEWS.d/next/Library/2022-04-15-20-56-31.gh-issue-74166.70KlvL.rst new file mode 100644 index 00000000000000..ddd4eb77e7c746 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-15-20-56-31.gh-issue-74166.70KlvL.rst @@ -0,0 +1 @@ +Add option to raise all errors from :meth:`~socket.create_connection` in an :exc:`ExceptionGroup` when it fails to create a connection. The default remains to raise only the last error that had occurred when multiple addresses were tried. From webhook-mailer at python.org Mon Apr 18 22:44:45 2022 From: webhook-mailer at python.org (methane) Date: Tue, 19 Apr 2022 02:44:45 -0000 Subject: [Python-checkins] gh-91526: io: Remove device encoding support from TextIOWrapper (GH-91529) Message-ID: https://github.com/python/cpython/commit/6fdb62b1fa344b9cdf1f221eac83404fb1980822 commit: 6fdb62b1fa344b9cdf1f221eac83404fb1980822 branch: main author: Inada Naoki committer: methane date: 2022-04-19T11:44:36+09:00 summary: gh-91526: io: Remove device encoding support from TextIOWrapper (GH-91529) `TextIOWrapper.__init__()` called `os.device_encoding(file.fileno())` if fileno is 0-2 and encoding=None. But it is very rarely works, and never documented behavior. files: A Misc/NEWS.d/next/Library/2022-04-14-18-06-00.gh-issue-91526.cwfhSB.rst M Lib/_pyio.py M Lib/test/test_io.py M Modules/_io/textio.c diff --git a/Lib/_pyio.py b/Lib/_pyio.py index 0f33ed59492e7..380a7a736c64e 100644 --- a/Lib/_pyio.py +++ b/Lib/_pyio.py @@ -2021,14 +2021,6 @@ def __init__(self, buffer, encoding=None, errors=None, newline=None, self._check_newline(newline) encoding = text_encoding(encoding) - if encoding == "locale" and sys.platform == "win32": - # On Unix, os.device_encoding() returns "utf-8" instead of locale encoding - # in the UTF-8 mode. So we use os.device_encoding() only on Windows. - try: - encoding = os.device_encoding(buffer.fileno()) or "locale" - except (AttributeError, UnsupportedOperation): - pass - if encoding == "locale": try: import locale diff --git a/Lib/test/test_io.py b/Lib/test/test_io.py index c86251dfe5734..45bf81b61f416 100644 --- a/Lib/test/test_io.py +++ b/Lib/test/test_io.py @@ -2736,18 +2736,6 @@ def test_default_encoding(self): os.environ.clear() os.environ.update(old_environ) - @support.cpython_only - @unittest.skipIf(sys.platform != "win32", "Windows-only test") - @unittest.skipIf(sys.flags.utf8_mode, "utf-8 mode is enabled") - def test_device_encoding(self): - # Issue 15989 - import _testcapi - b = self.BytesIO() - b.fileno = lambda: _testcapi.INT_MAX + 1 - self.assertRaises(OverflowError, self.TextIOWrapper, b, encoding="locale") - b.fileno = lambda: _testcapi.UINT_MAX + 1 - self.assertRaises(OverflowError, self.TextIOWrapper, b, encoding="locale") - def test_encoding(self): # Check the encoding attribute is always set, and valid b = self.BytesIO() diff --git a/Misc/NEWS.d/next/Library/2022-04-14-18-06-00.gh-issue-91526.cwfhSB.rst b/Misc/NEWS.d/next/Library/2022-04-14-18-06-00.gh-issue-91526.cwfhSB.rst new file mode 100644 index 0000000000000..a6633421e2b0f --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-14-18-06-00.gh-issue-91526.cwfhSB.rst @@ -0,0 +1,3 @@ +Stop calling ``os.device_encoding(file.fileno())`` in +:class:`TextIOWrapper`. It was complex, never documented, and didn't work +for most cases. (Patch by Inada Naoki.) diff --git a/Modules/_io/textio.c b/Modules/_io/textio.c index 6ba7393c3a6a3..f45a69787383f 100644 --- a/Modules/_io/textio.c +++ b/Modules/_io/textio.c @@ -1060,7 +1060,6 @@ _io_TextIOWrapper___init___impl(textio *self, PyObject *buffer, PyObject *raw, *codec_info = NULL; PyObject *res; int r; - int use_locale_encoding = 0; // Use locale encoding even in UTF-8 mode. self->ok = 0; self->detached = 0; @@ -1074,10 +1073,6 @@ _io_TextIOWrapper___init___impl(textio *self, PyObject *buffer, } } } - else if (strcmp(encoding, "locale") == 0) { - encoding = NULL; - use_locale_encoding = 1; - } if (errors == Py_None) { errors = &_Py_ID(strict); @@ -1114,57 +1109,18 @@ _io_TextIOWrapper___init___impl(textio *self, PyObject *buffer, self->encodefunc = NULL; self->b2cratio = 0.0; -#ifdef MS_WINDOWS - // os.device_encoding() on Unix is the locale encoding or UTF-8 - // according to UTF-8 Mode. - // Since UTF-8 mode shouldn't affect `encoding="locale"`, we call - // os.device_encoding() only on Windows. - if (encoding == NULL) { - /* Try os.device_encoding(fileno) */ - PyObject *fileno; - _PyIO_State *state = IO_STATE(); - if (state == NULL) - goto error; - fileno = PyObject_CallMethodNoArgs(buffer, &_Py_ID(fileno)); - /* Ignore only AttributeError and UnsupportedOperation */ - if (fileno == NULL) { - if (PyErr_ExceptionMatches(PyExc_AttributeError) || - PyErr_ExceptionMatches(state->unsupported_operation)) { - PyErr_Clear(); - } - else { - goto error; - } - } - else { - int fd = _PyLong_AsInt(fileno); - Py_DECREF(fileno); - if (fd == -1 && PyErr_Occurred()) { - goto error; - } - - self->encoding = _Py_device_encoding(fd); - if (self->encoding == NULL) - goto error; - else if (!PyUnicode_Check(self->encoding)) - Py_CLEAR(self->encoding); - } + if (encoding == NULL && _PyRuntime.preconfig.utf8_mode) { + _Py_DECLARE_STR(utf_8, "utf-8"); + self->encoding = Py_NewRef(&_Py_STR(utf_8)); } -#endif - - if (encoding == NULL && self->encoding == NULL) { - if (_PyRuntime.preconfig.utf8_mode && !use_locale_encoding) { - _Py_DECLARE_STR(utf_8, "utf-8"); - self->encoding = Py_NewRef(&_Py_STR(utf_8)); - } - else { - self->encoding = _Py_GetLocaleEncodingObject(); - } + else if (encoding == NULL || (strcmp(encoding, "locale") == 0)) { + self->encoding = _Py_GetLocaleEncodingObject(); if (self->encoding == NULL) { goto error; } assert(PyUnicode_Check(self->encoding)); } + if (self->encoding != NULL) { encoding = PyUnicode_AsUTF8(self->encoding); if (encoding == NULL) From webhook-mailer at python.org Mon Apr 18 22:51:07 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Tue, 19 Apr 2022 02:51:07 -0000 Subject: [Python-checkins] bpo-46014: Add docs regarding `functools.singledispatch` changes in 3.11 (#32282) Message-ID: https://github.com/python/cpython/commit/014eb7fd0242963ac475abb3c1fb9be0714b203f commit: 014eb7fd0242963ac475abb3c1fb9be0714b203f branch: main author: Yurii Karabas <1998uriyyo at gmail.com> committer: JelleZijlstra date: 2022-04-18T19:50:59-07:00 summary: bpo-46014: Add docs regarding `functools.singledispatch` changes in 3.11 (#32282) Co-authored-by: Jelle Zijlstra files: M Doc/library/functools.rst M Doc/whatsnew/3.11.rst diff --git a/Doc/library/functools.rst b/Doc/library/functools.rst index e23946a0a45e7..dd4d76ef67098 100644 --- a/Doc/library/functools.rst +++ b/Doc/library/functools.rst @@ -436,6 +436,23 @@ The :mod:`functools` module defines the following functions: ... for i, elem in enumerate(arg): ... print(i, elem) + :data:`types.UnionType` and :data:`typing.Union` can also be used:: + + >>> @fun.register + ... def _(arg: int | float, verbose=False): + ... if verbose: + ... print("Strength in numbers, eh?", end=" ") + ... print(arg) + ... + >>> from typing import Union + >>> @fun.register + ... def _(arg: Union[list, set], verbose=False): + ... if verbose: + ... print("Enumerate this:") + ... for i, elem in enumerate(arg): + ... print(i, elem) + ... + For code which doesn't use type annotations, the appropriate type argument can be passed explicitly to the decorator itself:: @@ -535,6 +552,10 @@ The :mod:`functools` module defines the following functions: .. versionchanged:: 3.7 The :func:`register` attribute now supports using type annotations. + .. versionchanged:: 3.11 + The :func:`register` attribute now supports :data:`types.UnionType` + and :data:`typing.Union` as type annotations. + .. class:: singledispatchmethod(func) diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index baff6873991ac..6540a255a0ed8 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -267,6 +267,36 @@ fractions that an ``isinstance(some_fraction, typing.SupportsInt)`` check passes. (Contributed by Mark Dickinson in :issue:`44547`.) +functools +--------- + +* :func:`functools.singledispatch` now supports :data:`types.UnionType` + and :data:`typing.Union` as annotations to the dispatch argument.:: + + >>> from functools import singledispatch + >>> @singledispatch + ... def fun(arg, verbose=False): + ... if verbose: + ... print("Let me just say,", end=" ") + ... print(arg) + ... + >>> @fun.register + ... def _(arg: int | float, verbose=False): + ... if verbose: + ... print("Strength in numbers, eh?", end=" ") + ... print(arg) + ... + >>> from typing import Union + >>> @fun.register + ... def _(arg: Union[list, set], verbose=False): + ... if verbose: + ... print("Enumerate this:") + ... for i, elem in enumerate(arg): + ... print(i, elem) + ... + + (Contributed by Yurii Karabas in :issue:`46014`.) + hashlib ------- From webhook-mailer at python.org Mon Apr 18 22:57:01 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Tue, 19 Apr 2022 02:57:01 -0000 Subject: [Python-checkins] gh-91098: Use Argument Clinic for Object/classobject.c to fix docstrings (#31711) Message-ID: https://github.com/python/cpython/commit/7fbc7f61280d301c0ac88b65ee8747fcf19f01f8 commit: 7fbc7f61280d301c0ac88b65ee8747fcf19f01f8 branch: main author: Oleg Iarygin committer: JelleZijlstra date: 2022-04-18T19:56:53-07:00 summary: gh-91098: Use Argument Clinic for Object/classobject.c to fix docstrings (#31711) Closes GH-91098. Co-authored-by: Jelle Zijlstra files: A Misc/NEWS.d/next/Core and Builtins/2022-03-07-11-51-51.bpo-46942.57obVi.rst A Objects/clinic/classobject.c.h M Lib/idlelib/idle_test/test_calltip.py M Objects/classobject.c diff --git a/Lib/idlelib/idle_test/test_calltip.py b/Lib/idlelib/idle_test/test_calltip.py index e8d2bd17cb681..1ccb63b9dbd65 100644 --- a/Lib/idlelib/idle_test/test_calltip.py +++ b/Lib/idlelib/idle_test/test_calltip.py @@ -77,7 +77,9 @@ class SB: __call__ = None tiptest(List.append, '(self, object, /)' + append_doc) tiptest([].append, '(object, /)' + append_doc) - tiptest(types.MethodType, "method(function, instance)") + tiptest(types.MethodType, + '(function, instance, /)\n' + 'Create a bound instance method object.') tiptest(SB(), default_tip) p = re.compile('') diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-03-07-11-51-51.bpo-46942.57obVi.rst b/Misc/NEWS.d/next/Core and Builtins/2022-03-07-11-51-51.bpo-46942.57obVi.rst new file mode 100644 index 0000000000000..930a9018b1d55 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2022-03-07-11-51-51.bpo-46942.57obVi.rst @@ -0,0 +1,2 @@ +Use Argument Clinic for the :class:`types.MethodType` constructor. Patch by +Oleg Iarygin. diff --git a/Objects/classobject.c b/Objects/classobject.c index c75ba572b9624..cf731358e41c4 100644 --- a/Objects/classobject.c +++ b/Objects/classobject.c @@ -7,8 +7,15 @@ #include "pycore_pystate.h" // _PyThreadState_GET() #include "structmember.h" // PyMemberDef +#include "clinic/classobject.c.h" + #define TP_DESCR_GET(t) ((t)->tp_descr_get) +/*[clinic input] +class method "PyMethodObject *" "&PyMethod_Type" +[clinic start generated code]*/ +/*[clinic end generated code: output=da39a3ee5e6b4b0d input=b16e47edf6107c23]*/ + PyObject * PyMethod_Function(PyObject *im) @@ -115,23 +122,26 @@ PyMethod_New(PyObject *func, PyObject *self) return (PyObject *)im; } +/*[clinic input] +method.__reduce__ +[clinic start generated code]*/ + static PyObject * -method_reduce(PyMethodObject *im, PyObject *Py_UNUSED(ignored)) +method___reduce___impl(PyMethodObject *self) +/*[clinic end generated code: output=6c04506d0fa6fdcb input=143a0bf5e96de6e8]*/ { - PyObject *self = PyMethod_GET_SELF(im); - PyObject *func = PyMethod_GET_FUNCTION(im); - PyObject *funcname; - - funcname = PyObject_GetAttr(func, &_Py_ID(__name__)); + PyObject *funcself = PyMethod_GET_SELF(self); + PyObject *func = PyMethod_GET_FUNCTION(self); + PyObject *funcname = PyObject_GetAttr(func, &_Py_ID(__name__)); if (funcname == NULL) { return NULL; } return Py_BuildValue( - "N(ON)", _PyEval_GetBuiltin(&_Py_ID(getattr)), self, funcname); + "N(ON)", _PyEval_GetBuiltin(&_Py_ID(getattr)), funcself, funcname); } static PyMethodDef method_methods[] = { - {"__reduce__", (PyCFunction)method_reduce, METH_NOARGS, NULL}, + METHOD___REDUCE___METHODDEF {NULL, NULL} }; @@ -193,34 +203,32 @@ method_getattro(PyObject *obj, PyObject *name) return PyObject_GetAttr(im->im_func, name); } -PyDoc_STRVAR(method_doc, -"method(function, instance)\n\ -\n\ -Create a bound instance method object."); +/*[clinic input] + at classmethod +method.__new__ as method_new + function: object + instance: object + / + +Create a bound instance method object. +[clinic start generated code]*/ static PyObject * -method_new(PyTypeObject* type, PyObject* args, PyObject *kw) +method_new_impl(PyTypeObject *type, PyObject *function, PyObject *instance) +/*[clinic end generated code: output=d33ef4ebf702e1f7 input=4e32facc3c3108ae]*/ { - PyObject *func; - PyObject *self; - - if (!_PyArg_NoKeywords("method", kw)) - return NULL; - if (!PyArg_UnpackTuple(args, "method", 2, 2, - &func, &self)) - return NULL; - if (!PyCallable_Check(func)) { + if (!PyCallable_Check(function)) { PyErr_SetString(PyExc_TypeError, "first argument must be callable"); return NULL; } - if (self == NULL || self == Py_None) { + if (instance == NULL || instance == Py_None) { PyErr_SetString(PyExc_TypeError, - "self must not be None"); + "instance must not be None"); return NULL; } - return PyMethod_New(func, self); + return PyMethod_New(function, instance); } static void @@ -322,50 +330,37 @@ method_descr_get(PyObject *meth, PyObject *obj, PyObject *cls) PyTypeObject PyMethod_Type = { PyVarObject_HEAD_INIT(&PyType_Type, 0) - "method", - sizeof(PyMethodObject), - 0, - (destructor)method_dealloc, /* tp_dealloc */ - offsetof(PyMethodObject, vectorcall), /* tp_vectorcall_offset */ - 0, /* tp_getattr */ - 0, /* tp_setattr */ - 0, /* tp_as_async */ - (reprfunc)method_repr, /* tp_repr */ - 0, /* tp_as_number */ - 0, /* tp_as_sequence */ - 0, /* tp_as_mapping */ - (hashfunc)method_hash, /* tp_hash */ - PyVectorcall_Call, /* tp_call */ - 0, /* tp_str */ - method_getattro, /* tp_getattro */ - PyObject_GenericSetAttr, /* tp_setattro */ - 0, /* tp_as_buffer */ - Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC | - Py_TPFLAGS_HAVE_VECTORCALL, /* tp_flags */ - method_doc, /* tp_doc */ - (traverseproc)method_traverse, /* tp_traverse */ - 0, /* tp_clear */ - method_richcompare, /* tp_richcompare */ - offsetof(PyMethodObject, im_weakreflist), /* tp_weaklistoffset */ - 0, /* tp_iter */ - 0, /* tp_iternext */ - method_methods, /* tp_methods */ - method_memberlist, /* tp_members */ - method_getset, /* tp_getset */ - 0, /* tp_base */ - 0, /* tp_dict */ - method_descr_get, /* tp_descr_get */ - 0, /* tp_descr_set */ - 0, /* tp_dictoffset */ - 0, /* tp_init */ - 0, /* tp_alloc */ - method_new, /* tp_new */ + .tp_name = "method", + .tp_basicsize = sizeof(PyMethodObject), + .tp_dealloc = (destructor)method_dealloc, + .tp_vectorcall_offset = offsetof(PyMethodObject, vectorcall), + .tp_repr = (reprfunc)method_repr, + .tp_hash = (hashfunc)method_hash, + .tp_call = PyVectorcall_Call, + .tp_getattro = method_getattro, + .tp_setattro = PyObject_GenericSetAttr, + .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC | + Py_TPFLAGS_HAVE_VECTORCALL, + .tp_doc = method_new__doc__, + .tp_traverse = (traverseproc)method_traverse, + .tp_richcompare = method_richcompare, + .tp_weaklistoffset = offsetof(PyMethodObject, im_weakreflist), + .tp_methods = method_methods, + .tp_members = method_memberlist, + .tp_getset = method_getset, + .tp_descr_get = method_descr_get, + .tp_new = method_new, }; /* ------------------------------------------------------------------------ * instance method */ +/*[clinic input] +class instancemethod "PyInstanceMethodObject *" "&PyInstanceMethod_Type" +[clinic start generated code]*/ +/*[clinic end generated code: output=da39a3ee5e6b4b0d input=28c9762a9016f4d2]*/ + PyObject * PyInstanceMethod_New(PyObject *func) { PyInstanceMethodObject *method; @@ -516,67 +511,43 @@ instancemethod_repr(PyObject *self) return result; } -PyDoc_STRVAR(instancemethod_doc, -"instancemethod(function)\n\ -\n\ -Bind a function to a class."); +/*[clinic input] + at classmethod +instancemethod.__new__ as instancemethod_new + function: object + / + +Bind a function to a class. +[clinic start generated code]*/ static PyObject * -instancemethod_new(PyTypeObject* type, PyObject* args, PyObject *kw) +instancemethod_new_impl(PyTypeObject *type, PyObject *function) +/*[clinic end generated code: output=5e0397b2bdb750be input=cfc54e8b973664a8]*/ { - PyObject *func; - - if (!_PyArg_NoKeywords("instancemethod", kw)) - return NULL; - if (!PyArg_UnpackTuple(args, "instancemethod", 1, 1, &func)) - return NULL; - if (!PyCallable_Check(func)) { + if (!PyCallable_Check(function)) { PyErr_SetString(PyExc_TypeError, "first argument must be callable"); return NULL; } - return PyInstanceMethod_New(func); + return PyInstanceMethod_New(function); } PyTypeObject PyInstanceMethod_Type = { PyVarObject_HEAD_INIT(&PyType_Type, 0) - "instancemethod", /* tp_name */ - sizeof(PyInstanceMethodObject), /* tp_basicsize */ - 0, /* tp_itemsize */ - instancemethod_dealloc, /* tp_dealloc */ - 0, /* tp_vectorcall_offset */ - 0, /* tp_getattr */ - 0, /* tp_setattr */ - 0, /* tp_as_async */ - (reprfunc)instancemethod_repr, /* tp_repr */ - 0, /* tp_as_number */ - 0, /* tp_as_sequence */ - 0, /* tp_as_mapping */ - 0, /* tp_hash */ - instancemethod_call, /* tp_call */ - 0, /* tp_str */ - instancemethod_getattro, /* tp_getattro */ - PyObject_GenericSetAttr, /* tp_setattro */ - 0, /* tp_as_buffer */ - Py_TPFLAGS_DEFAULT - | Py_TPFLAGS_HAVE_GC, /* tp_flags */ - instancemethod_doc, /* tp_doc */ - instancemethod_traverse, /* tp_traverse */ - 0, /* tp_clear */ - instancemethod_richcompare, /* tp_richcompare */ - 0, /* tp_weaklistoffset */ - 0, /* tp_iter */ - 0, /* tp_iternext */ - 0, /* tp_methods */ - instancemethod_memberlist, /* tp_members */ - instancemethod_getset, /* tp_getset */ - 0, /* tp_base */ - 0, /* tp_dict */ - instancemethod_descr_get, /* tp_descr_get */ - 0, /* tp_descr_set */ - 0, /* tp_dictoffset */ - 0, /* tp_init */ - 0, /* tp_alloc */ - instancemethod_new, /* tp_new */ + .tp_name = "instancemethod", + .tp_basicsize = sizeof(PyInstanceMethodObject), + .tp_dealloc = instancemethod_dealloc, + .tp_repr = (reprfunc)instancemethod_repr, + .tp_call = instancemethod_call, + .tp_getattro = instancemethod_getattro, + .tp_setattro = PyObject_GenericSetAttr, + .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC, + .tp_doc = instancemethod_new__doc__, + .tp_traverse = instancemethod_traverse, + .tp_richcompare = instancemethod_richcompare, + .tp_members = instancemethod_memberlist, + .tp_getset = instancemethod_getset, + .tp_descr_get = instancemethod_descr_get, + .tp_new = instancemethod_new, }; diff --git a/Objects/clinic/classobject.c.h b/Objects/clinic/classobject.c.h new file mode 100644 index 0000000000000..a4f190015a0d8 --- /dev/null +++ b/Objects/clinic/classobject.c.h @@ -0,0 +1,83 @@ +/*[clinic input] +preserve +[clinic start generated code]*/ + +PyDoc_STRVAR(method___reduce____doc__, +"__reduce__($self, /)\n" +"--\n" +"\n"); + +#define METHOD___REDUCE___METHODDEF \ + {"__reduce__", (PyCFunction)method___reduce__, METH_NOARGS, method___reduce____doc__}, + +static PyObject * +method___reduce___impl(PyMethodObject *self); + +static PyObject * +method___reduce__(PyMethodObject *self, PyObject *Py_UNUSED(ignored)) +{ + return method___reduce___impl(self); +} + +PyDoc_STRVAR(method_new__doc__, +"method(function, instance, /)\n" +"--\n" +"\n" +"Create a bound instance method object."); + +static PyObject * +method_new_impl(PyTypeObject *type, PyObject *function, PyObject *instance); + +static PyObject * +method_new(PyTypeObject *type, PyObject *args, PyObject *kwargs) +{ + PyObject *return_value = NULL; + PyObject *function; + PyObject *instance; + + if ((type == &PyMethod_Type || + type->tp_init == PyMethod_Type.tp_init) && + !_PyArg_NoKeywords("method", kwargs)) { + goto exit; + } + if (!_PyArg_CheckPositional("method", PyTuple_GET_SIZE(args), 2, 2)) { + goto exit; + } + function = PyTuple_GET_ITEM(args, 0); + instance = PyTuple_GET_ITEM(args, 1); + return_value = method_new_impl(type, function, instance); + +exit: + return return_value; +} + +PyDoc_STRVAR(instancemethod_new__doc__, +"instancemethod(function, /)\n" +"--\n" +"\n" +"Bind a function to a class."); + +static PyObject * +instancemethod_new_impl(PyTypeObject *type, PyObject *function); + +static PyObject * +instancemethod_new(PyTypeObject *type, PyObject *args, PyObject *kwargs) +{ + PyObject *return_value = NULL; + PyObject *function; + + if ((type == &PyInstanceMethod_Type || + type->tp_init == PyInstanceMethod_Type.tp_init) && + !_PyArg_NoKeywords("instancemethod", kwargs)) { + goto exit; + } + if (!_PyArg_CheckPositional("instancemethod", PyTuple_GET_SIZE(args), 1, 1)) { + goto exit; + } + function = PyTuple_GET_ITEM(args, 0); + return_value = instancemethod_new_impl(type, function); + +exit: + return return_value; +} +/*[clinic end generated code: output=a230fe125f664416 input=a9049054013a1b77]*/ From webhook-mailer at python.org Tue Apr 19 03:53:26 2022 From: webhook-mailer at python.org (vstinner) Date: Tue, 19 Apr 2022 07:53:26 -0000 Subject: [Python-checkins] bpo-40421: Cleanup PyFrame C API (GH-32417) Message-ID: https://github.com/python/cpython/commit/aa5c0a9f8dd935a7afff07abb5a60d492c40f2cb commit: aa5c0a9f8dd935a7afff07abb5a60d492c40f2cb branch: main author: Victor Stinner committer: vstinner date: 2022-04-19T09:53:10+02:00 summary: bpo-40421: Cleanup PyFrame C API (GH-32417) files: M Doc/c-api/frame.rst M Misc/NEWS.d/next/C API/2022-04-08-11-29-36.bpo-40421.H0ORmT.rst M Objects/frameobject.c diff --git a/Doc/c-api/frame.rst b/Doc/c-api/frame.rst index 68e5dc6daeaec..46ce700abf147 100644 --- a/Doc/c-api/frame.rst +++ b/Doc/c-api/frame.rst @@ -27,8 +27,6 @@ See also :ref:`Reflection `. Return a :term:`strong reference`, or ``NULL`` if *frame* has no outer frame. - *frame* must not be ``NULL``. - .. versionadded:: 3.9 @@ -38,8 +36,6 @@ See also :ref:`Reflection `. Return a :term:`strong reference`. The result cannot be ``NULL``. - *frame* must not be ``NULL``. - .. versionadded:: 3.11 @@ -49,7 +45,7 @@ See also :ref:`Reflection `. Return a :term:`strong reference`. - *frame* must not be ``NULL``. The result (frame code) cannot be ``NULL``. + The result (frame code) cannot be ``NULL``. .. versionadded:: 3.9 @@ -62,8 +58,6 @@ See also :ref:`Reflection `. Return a :term:`strong reference`, or ``NULL``. - *frame* must not be ``NULL``. - .. versionadded:: 3.11 @@ -73,19 +67,15 @@ See also :ref:`Reflection `. Return a :term:`strong reference`. The result cannot be ``NULL``. - *frame* must not be ``NULL``. - .. versionadded:: 3.11 .. c:function:: int PyFrame_GetLasti(PyFrameObject *frame) - Get the *frame*'s ``f_lasti`` attribute (:class:`dict`). + Get the *frame*'s ``f_lasti`` attribute. Returns -1 if ``frame.f_lasti`` is ``None``. - *frame* must not be ``NULL``. - .. versionadded:: 3.11 @@ -95,13 +85,9 @@ See also :ref:`Reflection `. Return a :term:`strong reference`. - *frame* must not be ``NULL``. - .. versionadded:: 3.11 .. c:function:: int PyFrame_GetLineNumber(PyFrameObject *frame) Return the line number that *frame* is currently executing. - - *frame* must not be ``NULL``. diff --git a/Misc/NEWS.d/next/C API/2022-04-08-11-29-36.bpo-40421.H0ORmT.rst b/Misc/NEWS.d/next/C API/2022-04-08-11-29-36.bpo-40421.H0ORmT.rst index 2e10c2394c933..d4a1dbecf2867 100644 --- a/Misc/NEWS.d/next/C API/2022-04-08-11-29-36.bpo-40421.H0ORmT.rst +++ b/Misc/NEWS.d/next/C API/2022-04-08-11-29-36.bpo-40421.H0ORmT.rst @@ -1,2 +1,2 @@ -Add ``PyFrame_GetLasti`` C-API function to access frame object's ``lasti`` +Add ``PyFrame_GetLasti`` C-API function to access frame object's ``f_lasti`` attribute safely from C code. diff --git a/Objects/frameobject.c b/Objects/frameobject.c index dc4ef8bcda541..e65395ee5f20e 100644 --- a/Objects/frameobject.c +++ b/Objects/frameobject.c @@ -1160,7 +1160,7 @@ PyFrame_GetLasti(PyFrameObject *frame) if (lasti < 0) { return -1; } - return lasti*2; + return lasti * sizeof(_Py_CODEUNIT); } PyObject * From webhook-mailer at python.org Tue Apr 19 04:45:24 2022 From: webhook-mailer at python.org (markshannon) Date: Tue, 19 Apr 2022 08:45:24 -0000 Subject: [Python-checkins] gh-90997: bpo-46841: Disassembly of quickened code (GH-32099) Message-ID: https://github.com/python/cpython/commit/e590379197f065f52c8140c0edd7a59360216531 commit: e590379197f065f52c8140c0edd7a59360216531 branch: main author: penguin_wwy <940375606 at qq.com> committer: markshannon date: 2022-04-19T09:45:08+01:00 summary: gh-90997: bpo-46841: Disassembly of quickened code (GH-32099) files: A Misc/NEWS.d/next/Library/2022-03-25-22-18-45.bpo-46841.NUEsXW.rst M Lib/dis.py M Lib/test/test__opcode.py M Lib/test/test_dis.py diff --git a/Lib/dis.py b/Lib/dis.py index f7b38a82ab13e..205e9d8d193f3 100644 --- a/Lib/dis.py +++ b/Lib/dis.py @@ -7,7 +7,7 @@ from opcode import * from opcode import __all__ as _opcodes_all -from opcode import _nb_ops +from opcode import _nb_ops, _inline_cache_entries, _specializations, _specialized_instructions __all__ = ["code_info", "dis", "disassemble", "distb", "disco", "findlinestarts", "findlabels", "show_code", @@ -34,6 +34,18 @@ CACHE = opmap["CACHE"] +_all_opname = list(opname) +_all_opmap = dict(opmap) +_empty_slot = [slot for slot, name in enumerate(_all_opname) if name.startswith("<")] +for spec_op, specialized in zip(_empty_slot, _specialized_instructions): + # fill opname and opmap + _all_opname[spec_op] = specialized + _all_opmap[specialized] = spec_op + +deoptmap = { + specialized: base for base, family in _specializations.items() for specialized in family +} + def _try_compile(source, name): """Attempts to compile the given source, first as an expression and then as a statement if the first approach fails. @@ -47,7 +59,7 @@ def _try_compile(source, name): c = compile(source, name, 'exec') return c -def dis(x=None, *, file=None, depth=None, show_caches=False): +def dis(x=None, *, file=None, depth=None, show_caches=False, adaptive=False): """Disassemble classes, methods, functions, and other compiled objects. With no argument, disassemble the last traceback. @@ -57,7 +69,7 @@ def dis(x=None, *, file=None, depth=None, show_caches=False): in a special attribute. """ if x is None: - distb(file=file, show_caches=show_caches) + distb(file=file, show_caches=show_caches, adaptive=adaptive) return # Extract functions from methods. if hasattr(x, '__func__'): @@ -78,21 +90,21 @@ def dis(x=None, *, file=None, depth=None, show_caches=False): if isinstance(x1, _have_code): print("Disassembly of %s:" % name, file=file) try: - dis(x1, file=file, depth=depth, show_caches=show_caches) + dis(x1, file=file, depth=depth, show_caches=show_caches, adaptive=adaptive) except TypeError as msg: print("Sorry:", msg, file=file) print(file=file) elif hasattr(x, 'co_code'): # Code object - _disassemble_recursive(x, file=file, depth=depth, show_caches=show_caches) + _disassemble_recursive(x, file=file, depth=depth, show_caches=show_caches, adaptive=adaptive) elif isinstance(x, (bytes, bytearray)): # Raw bytecode _disassemble_bytes(x, file=file, show_caches=show_caches) elif isinstance(x, str): # Source code - _disassemble_str(x, file=file, depth=depth, show_caches=show_caches) + _disassemble_str(x, file=file, depth=depth, show_caches=show_caches, adaptive=adaptive) else: raise TypeError("don't know how to disassemble %s objects" % type(x).__name__) -def distb(tb=None, *, file=None, show_caches=False): +def distb(tb=None, *, file=None, show_caches=False, adaptive=False): """Disassemble a traceback (default: last traceback).""" if tb is None: try: @@ -100,7 +112,7 @@ def distb(tb=None, *, file=None, show_caches=False): except AttributeError: raise RuntimeError("no last traceback to disassemble") from None while tb.tb_next: tb = tb.tb_next - disassemble(tb.tb_frame.f_code, tb.tb_lasti, file=file, show_caches=show_caches) + disassemble(tb.tb_frame.f_code, tb.tb_lasti, file=file, show_caches=show_caches, adaptive=adaptive) # The inspect module interrogates this dictionary to build its # list of CO_* constants. It is also used by pretty_flags to @@ -162,6 +174,13 @@ def _get_code_object(x): raise TypeError("don't know how to disassemble %s objects" % type(x).__name__) +def _deoptop(op): + name = _all_opname[op] + return _all_opmap[deoptmap[name]] if name in deoptmap else op + +def _get_code_array(co, adaptive): + return co._co_code_adaptive if adaptive else co.co_code + def code_info(x): """Formatted details of methods, functions, or code.""" return _format_code_info(_get_code_object(x)) @@ -302,7 +321,7 @@ def _disassemble(self, lineno_width=3, mark_as_current=False, offset_width=4): return ' '.join(fields).rstrip() -def get_instructions(x, *, first_line=None, show_caches=False): +def get_instructions(x, *, first_line=None, show_caches=False, adaptive=False): """Iterator for the opcodes in methods, functions or code Generates a series of Instruction named tuples giving the details of @@ -319,7 +338,7 @@ def get_instructions(x, *, first_line=None, show_caches=False): line_offset = first_line - co.co_firstlineno else: line_offset = 0 - return _get_instructions_bytes(co.co_code, + return _get_instructions_bytes(_get_code_array(co, adaptive), co._varname_from_oparg, co.co_names, co.co_consts, linestarts, line_offset, @@ -415,8 +434,13 @@ def _get_instructions_bytes(code, varname_from_oparg=None, for i in range(start, end): labels.add(target) starts_line = None + cache_counter = 0 for offset, op, arg in _unpack_opargs(code): - if not show_caches and op == CACHE: + if cache_counter > 0: + if show_caches: + yield Instruction("CACHE", 0, None, None, '', + offset, None, False, None) + cache_counter -= 1 continue if linestarts is not None: starts_line = linestarts.get(offset, None) @@ -426,61 +450,63 @@ def _get_instructions_bytes(code, varname_from_oparg=None, argval = None argrepr = '' positions = Positions(*next(co_positions, ())) + deop = _deoptop(op) + cache_counter = _inline_cache_entries[deop] if arg is not None: # Set argval to the dereferenced value of the argument when # available, and argrepr to the string representation of argval. # _disassemble_bytes needs the string repr of the # raw name index for LOAD_GLOBAL, LOAD_CONST, etc. argval = arg - if op in hasconst: - argval, argrepr = _get_const_info(op, arg, co_consts) - elif op in hasname: - if op == LOAD_GLOBAL: + if deop in hasconst: + argval, argrepr = _get_const_info(deop, arg, co_consts) + elif deop in hasname: + if deop == LOAD_GLOBAL: argval, argrepr = _get_name_info(arg//2, get_name) if (arg & 1) and argrepr: argrepr = "NULL + " + argrepr else: argval, argrepr = _get_name_info(arg, get_name) - elif op in hasjabs: + elif deop in hasjabs: argval = arg*2 argrepr = "to " + repr(argval) - elif op in hasjrel: - signed_arg = -arg if _is_backward_jump(op) else arg + elif deop in hasjrel: + signed_arg = -arg if _is_backward_jump(deop) else arg argval = offset + 2 + signed_arg*2 argrepr = "to " + repr(argval) - elif op in haslocal or op in hasfree: + elif deop in haslocal or deop in hasfree: argval, argrepr = _get_name_info(arg, varname_from_oparg) - elif op in hascompare: + elif deop in hascompare: argval = cmp_op[arg] argrepr = argval - elif op == FORMAT_VALUE: + elif deop == FORMAT_VALUE: argval, argrepr = FORMAT_VALUE_CONVERTERS[arg & 0x3] argval = (argval, bool(arg & 0x4)) if argval[1]: if argrepr: argrepr += ', ' argrepr += 'with format' - elif op == MAKE_FUNCTION: + elif deop == MAKE_FUNCTION: argrepr = ', '.join(s for i, s in enumerate(MAKE_FUNCTION_FLAGS) if arg & (1< 0: if depth is not None: depth = depth - 1 @@ -489,7 +515,7 @@ def _disassemble_recursive(co, *, file=None, depth=None, show_caches=False): print(file=file) print("Disassembly of %r:" % (x,), file=file) _disassemble_recursive( - x, file=file, depth=depth, show_caches=show_caches + x, file=file, depth=depth, show_caches=show_caches, adaptive=adaptive ) def _disassemble_bytes(code, lasti=-1, varname_from_oparg=None, @@ -548,7 +574,7 @@ def _unpack_opargs(code): extended_arg = 0 for i in range(0, len(code), 2): op = code[i] - if op >= HAVE_ARGUMENT: + if _deoptop(op) >= HAVE_ARGUMENT: arg = code[i+1] | extended_arg extended_arg = (arg << 8) if op == EXTENDED_ARG else 0 # The oparg is stored as a signed integer @@ -641,7 +667,7 @@ class Bytecode: Iterating over this yields the bytecode operations as Instruction instances. """ - def __init__(self, x, *, first_line=None, current_offset=None, show_caches=False): + def __init__(self, x, *, first_line=None, current_offset=None, show_caches=False, adaptive=False): self.codeobj = co = _get_code_object(x) if first_line is None: self.first_line = co.co_firstlineno @@ -654,10 +680,11 @@ def __init__(self, x, *, first_line=None, current_offset=None, show_caches=False self.current_offset = current_offset self.exception_entries = parse_exception_table(co) self.show_caches = show_caches + self.adaptive = adaptive def __iter__(self): co = self.codeobj - return _get_instructions_bytes(co.co_code, + return _get_instructions_bytes(_get_code_array(co, self.adaptive), co._varname_from_oparg, co.co_names, co.co_consts, self._linestarts, @@ -671,12 +698,12 @@ def __repr__(self): self._original_object) @classmethod - def from_traceback(cls, tb, *, show_caches=False): + def from_traceback(cls, tb, *, show_caches=False, adaptive=False): """ Construct a Bytecode from the given traceback """ while tb.tb_next: tb = tb.tb_next return cls( - tb.tb_frame.f_code, current_offset=tb.tb_lasti, show_caches=show_caches + tb.tb_frame.f_code, current_offset=tb.tb_lasti, show_caches=show_caches, adaptive=adaptive ) def info(self): @@ -691,7 +718,7 @@ def dis(self): else: offset = -1 with io.StringIO() as output: - _disassemble_bytes(co.co_code, + _disassemble_bytes(_get_code_array(co, self.adaptive), varname_from_oparg=co._varname_from_oparg, names=co.co_names, co_consts=co.co_consts, linestarts=self._linestarts, diff --git a/Lib/test/test__opcode.py b/Lib/test/test__opcode.py index 7c1c0cfdb069b..2a4c0d2eeb656 100644 --- a/Lib/test/test__opcode.py +++ b/Lib/test/test__opcode.py @@ -18,7 +18,7 @@ def test_stack_effect(self): self.assertRaises(ValueError, stack_effect, dis.opmap['BUILD_SLICE']) self.assertRaises(ValueError, stack_effect, dis.opmap['POP_TOP'], 0) # All defined opcodes - for name, code in dis.opmap.items(): + for name, code in filter(lambda item: item[0] not in dis.deoptmap, dis.opmap.items()): with self.subTest(opname=name): if code < dis.HAVE_ARGUMENT: stack_effect(code) @@ -47,7 +47,7 @@ def test_stack_effect_jump(self): self.assertEqual(stack_effect(JUMP_FORWARD, 0, jump=False), 0) # All defined opcodes has_jump = dis.hasjabs + dis.hasjrel - for name, code in dis.opmap.items(): + for name, code in filter(lambda item: item[0] not in dis.deoptmap, dis.opmap.items()): with self.subTest(opname=name): if code < dis.HAVE_ARGUMENT: common = stack_effect(code) diff --git a/Lib/test/test_dis.py b/Lib/test/test_dis.py index fbc34a5dbe4ef..f560a5556c8b0 100644 --- a/Lib/test/test_dis.py +++ b/Lib/test/test_dis.py @@ -7,7 +7,7 @@ import sys import types import unittest -from test.support import captured_stdout, requires_debug_ranges +from test.support import captured_stdout, requires_debug_ranges, cpython_only from test.support.bytecode_helper import BytecodeTestCase import opcode @@ -583,6 +583,58 @@ def foo(x): _h.__code__.co_firstlineno + 3, ) +def load_test(x, y=0): + a, b = x, y + return a, b + +dis_load_test_quickened_code = """\ +%3d 0 RESUME_QUICK 0 + +%3d 2 LOAD_FAST__LOAD_FAST 0 (x) + 4 LOAD_FAST 1 (y) + 6 STORE_FAST__STORE_FAST 3 (b) + 8 STORE_FAST__LOAD_FAST 2 (a) + +%3d 10 LOAD_FAST__LOAD_FAST 2 (a) + 12 LOAD_FAST 3 (b) + 14 BUILD_TUPLE 2 + 16 RETURN_VALUE +""" % (load_test.__code__.co_firstlineno, + load_test.__code__.co_firstlineno + 1, + load_test.__code__.co_firstlineno + 2) + +def loop_test(): + for i in [1, 2, 3] * 3: + load_test(i) + +dis_loop_test_quickened_code = """\ +%3d 0 RESUME_QUICK 0 + +%3d 2 BUILD_LIST 0 + 4 LOAD_CONST 1 ((1, 2, 3)) + 6 LIST_EXTEND 1 + 8 LOAD_CONST 2 (3) + 10 BINARY_OP_ADAPTIVE 5 (*) + 14 GET_ITER + 16 FOR_ITER 17 (to 52) + 18 STORE_FAST 0 (i) + +%3d 20 LOAD_GLOBAL_MODULE 1 (NULL + load_test) + 32 LOAD_FAST 0 (i) + 34 PRECALL_PYFUNC 1 + 38 CALL_PY_WITH_DEFAULTS 1 + 48 POP_TOP + 50 JUMP_BACKWARD_QUICK 18 (to 16) + +%3d >> 52 LOAD_CONST 0 (None) + 54 RETURN_VALUE +""" % (loop_test.__code__.co_firstlineno, + loop_test.__code__.co_firstlineno + 1, + loop_test.__code__.co_firstlineno + 2, + loop_test.__code__.co_firstlineno + 1,) + +QUICKENING_WARMUP_DELAY = 8 + class DisTestBase(unittest.TestCase): "Common utilities for DisTests and TestDisTraceback" @@ -860,6 +912,93 @@ def check(expected, **kwargs): check(dis_nested_2, depth=None) check(dis_nested_2) + @staticmethod + def code_quicken(f, times=QUICKENING_WARMUP_DELAY): + for _ in range(times): + f() + + @cpython_only + def test_super_instructions(self): + self.code_quicken(lambda: load_test(0, 0)) + got = self.get_disassembly(load_test, adaptive=True) + self.do_disassembly_compare(got, dis_load_test_quickened_code, True) + + @cpython_only + def test_binary_specialize(self): + binary_op_quicken = """\ + 0 RESUME_QUICK 0 + + 1 2 LOAD_NAME 0 (a) + 4 LOAD_NAME 1 (b) + 6 %s + 10 RETURN_VALUE +""" + co_int = compile('a + b', "", "eval") + self.code_quicken(lambda: exec(co_int, {}, {'a': 1, 'b': 2})) + got = self.get_disassembly(co_int, adaptive=True) + self.do_disassembly_compare(got, binary_op_quicken % "BINARY_OP_ADD_INT 0 (+)", True) + + co_unicode = compile('a + b', "", "eval") + self.code_quicken(lambda: exec(co_unicode, {}, {'a': 'a', 'b': 'b'})) + got = self.get_disassembly(co_unicode, adaptive=True) + self.do_disassembly_compare(got, binary_op_quicken % "BINARY_OP_ADD_UNICODE 0 (+)", True) + + binary_subscr_quicken = """\ + 0 RESUME_QUICK 0 + + 1 2 LOAD_NAME 0 (a) + 4 LOAD_CONST 0 (0) + 6 %s + 16 RETURN_VALUE +""" + co_list = compile('a[0]', "", "eval") + self.code_quicken(lambda: exec(co_list, {}, {'a': [0]})) + got = self.get_disassembly(co_list, adaptive=True) + self.do_disassembly_compare(got, binary_subscr_quicken % "BINARY_SUBSCR_LIST_INT", True) + + co_dict = compile('a[0]', "", "eval") + self.code_quicken(lambda: exec(co_dict, {}, {'a': {0: '1'}})) + got = self.get_disassembly(co_dict, adaptive=True) + self.do_disassembly_compare(got, binary_subscr_quicken % "BINARY_SUBSCR_DICT", True) + + @cpython_only + def test_load_attr_specialize(self): + load_attr_quicken = """\ + 0 RESUME_QUICK 0 + + 1 2 LOAD_CONST 0 ('a') + 4 LOAD_ATTR_SLOT 0 (__class__) + 14 RETURN_VALUE +""" + co = compile("'a'.__class__", "", "eval") + self.code_quicken(lambda: exec(co, {}, {})) + got = self.get_disassembly(co, adaptive=True) + self.do_disassembly_compare(got, load_attr_quicken, True) + + @cpython_only + def test_call_specialize(self): + call_quicken = """\ + 0 RESUME_QUICK 0 + + 1 2 PUSH_NULL + 4 LOAD_NAME 0 (str) + 6 LOAD_CONST 0 (1) + 8 PRECALL_NO_KW_STR_1 1 + 12 CALL_ADAPTIVE 1 + 22 RETURN_VALUE +""" + co = compile("str(1)", "", "eval") + self.code_quicken(lambda: exec(co, {}, {})) + got = self.get_disassembly(co, adaptive=True) + self.do_disassembly_compare(got, call_quicken, True) + + @cpython_only + def test_loop_quicken(self): + # Loop can trigger a quicken where the loop is located + self.code_quicken(loop_test, 1) + got = self.get_disassembly(loop_test, adaptive=True) + self.do_disassembly_compare(got, dis_loop_test_quickened_code, True) + class DisWithFileTests(DisTests): diff --git a/Misc/NEWS.d/next/Library/2022-03-25-22-18-45.bpo-46841.NUEsXW.rst b/Misc/NEWS.d/next/Library/2022-03-25-22-18-45.bpo-46841.NUEsXW.rst new file mode 100644 index 0000000000000..0e778047593a7 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-03-25-22-18-45.bpo-46841.NUEsXW.rst @@ -0,0 +1 @@ +Disassembly of quickened code. From webhook-mailer at python.org Tue Apr 19 05:09:14 2022 From: webhook-mailer at python.org (iritkatriel) Date: Tue, 19 Apr 2022 09:09:14 -0000 Subject: [Python-checkins] gh-74166: break cycle by clearing the list instead of dropping its reference (GH-91685) Message-ID: https://github.com/python/cpython/commit/5b37b49ebc7a81e0a3f9b60ef82e804ab9e1badf commit: 5b37b49ebc7a81e0a3f9b60ef82e804ab9e1badf branch: main author: Irit Katriel <1055913+iritkatriel at users.noreply.github.com> committer: iritkatriel <1055913+iritkatriel at users.noreply.github.com> date: 2022-04-19T10:09:10+01:00 summary: gh-74166: break cycle by clearing the list instead of dropping its reference (GH-91685) files: M Lib/socket.py diff --git a/Lib/socket.py b/Lib/socket.py index 97362d92f6465..28d9c89d601aa 100755 --- a/Lib/socket.py +++ b/Lib/socket.py @@ -851,7 +851,7 @@ def create_connection(address, timeout=_GLOBAL_DEFAULT_TIMEOUT, raise ExceptionGroup("create_connection failed", exceptions) finally: # Break explicitly a reference cycle - exceptions = None + exceptions.clear() else: raise error("getaddrinfo returns an empty list") From webhook-mailer at python.org Tue Apr 19 05:15:53 2022 From: webhook-mailer at python.org (methane) Date: Tue, 19 Apr 2022 09:15:53 -0000 Subject: [Python-checkins] Doc: Fix link formatting typo (GH-91659) Message-ID: https://github.com/python/cpython/commit/6a7a8a740e61508cb5a0fcdac8b752b6c9e5d1ea commit: 6a7a8a740e61508cb5a0fcdac8b752b6c9e5d1ea branch: main author: slateny <46876382+slateny at users.noreply.github.com> committer: methane date: 2022-04-19T18:15:46+09:00 summary: Doc: Fix link formatting typo (GH-91659) files: M Doc/library/codecs.rst diff --git a/Doc/library/codecs.rst b/Doc/library/codecs.rst index e6f5954d33b59..26123fd0d0060 100644 --- a/Doc/library/codecs.rst +++ b/Doc/library/codecs.rst @@ -1426,7 +1426,7 @@ Internationalized Domain Names (IDN)). It builds upon the ``punycode`` encoding and :mod:`stringprep`. If you need the IDNA 2008 standard from :rfc:`5891` and :rfc:`5895`, use the -third-party `idna module _`. +third-party `idna module `_. These RFCs together define a protocol to support non-ASCII characters in domain names. A domain name containing non-ASCII characters (such as From webhook-mailer at python.org Tue Apr 19 05:31:55 2022 From: webhook-mailer at python.org (pablogsal) Date: Tue, 19 Apr 2022 09:31:55 -0000 Subject: [Python-checkins] [3.10] gh-91676 gh-91260 unittest.IsolatedAsyncioTestCase no longer leaks its executor (GH-91680) Message-ID: https://github.com/python/cpython/commit/61570ae0bc1507a62bee9d8b9bc2ff7155da7061 commit: 61570ae0bc1507a62bee9d8b9bc2ff7155da7061 branch: 3.10 author: Gregory P. Smith committer: pablogsal date: 2022-04-19T10:31:50+01:00 summary: [3.10] gh-91676 gh-91260 unittest.IsolatedAsyncioTestCase no longer leaks its executor (GH-91680) For things like test_asyncio.test_thread this was causing frequent "environment modified by test" errors as the executor threads had not always stopped running after the test was over. files: A Misc/NEWS.d/next/Library/2022-04-19-04-33-39.gh-issue-91676.ceQBwh.rst M Lib/unittest/async_case.py diff --git a/Lib/unittest/async_case.py b/Lib/unittest/async_case.py index 23231199f9870..d9c694e368255 100644 --- a/Lib/unittest/async_case.py +++ b/Lib/unittest/async_case.py @@ -148,6 +148,8 @@ def _tearDownAsyncioLoop(self): # shutdown asyncgens loop.run_until_complete(loop.shutdown_asyncgens()) finally: + # Prevent our executor environment from leaking to future tests. + loop.run_until_complete(loop.shutdown_default_executor()) asyncio.set_event_loop(None) loop.close() diff --git a/Misc/NEWS.d/next/Library/2022-04-19-04-33-39.gh-issue-91676.ceQBwh.rst b/Misc/NEWS.d/next/Library/2022-04-19-04-33-39.gh-issue-91676.ceQBwh.rst new file mode 100644 index 0000000000000..dfbaef4440e0a --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-19-04-33-39.gh-issue-91676.ceQBwh.rst @@ -0,0 +1,4 @@ +Fix :class:`unittest.IsolatedAsyncioTestCase` to shutdown the per test event +loop executor before returning from its ``run`` method so that a not yet +stopped or garbage collected executor state does not persist beyond the +test. From webhook-mailer at python.org Tue Apr 19 05:33:58 2022 From: webhook-mailer at python.org (miss-islington) Date: Tue, 19 Apr 2022 09:33:58 -0000 Subject: [Python-checkins] Doc: Fix link formatting typo (GH-91659) Message-ID: https://github.com/python/cpython/commit/b865a661e3d4052e5bee17848bc6df83d121be71 commit: b865a661e3d4052e5bee17848bc6df83d121be71 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-19T02:33:44-07:00 summary: Doc: Fix link formatting typo (GH-91659) (cherry picked from commit 6a7a8a740e61508cb5a0fcdac8b752b6c9e5d1ea) Co-authored-by: slateny <46876382+slateny at users.noreply.github.com> files: M Doc/library/codecs.rst diff --git a/Doc/library/codecs.rst b/Doc/library/codecs.rst index 9e2eb63479282..73fbefccd728a 100644 --- a/Doc/library/codecs.rst +++ b/Doc/library/codecs.rst @@ -1423,7 +1423,7 @@ Internationalized Domain Names (IDN)). It builds upon the ``punycode`` encoding and :mod:`stringprep`. If you need the IDNA 2008 standard from :rfc:`5891` and :rfc:`5895`, use the -third-party `idna module _`. +third-party `idna module `_. These RFCs together define a protocol to support non-ASCII characters in domain names. A domain name containing non-ASCII characters (such as From webhook-mailer at python.org Tue Apr 19 05:38:19 2022 From: webhook-mailer at python.org (miss-islington) Date: Tue, 19 Apr 2022 09:38:19 -0000 Subject: [Python-checkins] Doc: Fix link formatting typo (GH-91659) Message-ID: https://github.com/python/cpython/commit/1c27a1a7a501d771ef06e5def7fc474c83b91333 commit: 1c27a1a7a501d771ef06e5def7fc474c83b91333 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-19T02:38:06-07:00 summary: Doc: Fix link formatting typo (GH-91659) (cherry picked from commit 6a7a8a740e61508cb5a0fcdac8b752b6c9e5d1ea) Co-authored-by: slateny <46876382+slateny at users.noreply.github.com> files: M Doc/library/codecs.rst diff --git a/Doc/library/codecs.rst b/Doc/library/codecs.rst index 1a71e5791170d..e8a0a79f8610c 100644 --- a/Doc/library/codecs.rst +++ b/Doc/library/codecs.rst @@ -1420,7 +1420,7 @@ Internationalized Domain Names (IDN)). It builds upon the ``punycode`` encoding and :mod:`stringprep`. If you need the IDNA 2008 standard from :rfc:`5891` and :rfc:`5895`, use the -third-party `idna module _`. +third-party `idna module `_. These RFCs together define a protocol to support non-ASCII characters in domain names. A domain name containing non-ASCII characters (such as From webhook-mailer at python.org Tue Apr 19 09:12:55 2022 From: webhook-mailer at python.org (corona10) Date: Tue, 19 Apr 2022 13:12:55 -0000 Subject: [Python-checkins] gh-90699: Use module state to access insert str object. (GH-91693) Message-ID: https://github.com/python/cpython/commit/16fc5733b737320e43fe3244bf4be4e6c3b794a5 commit: 16fc5733b737320e43fe3244bf4be4e6c3b794a5 branch: main author: Dong-hee Na committer: corona10 date: 2022-04-19T22:12:46+09:00 summary: gh-90699: Use module state to access insert str object. (GH-91693) files: M Modules/_bisectmodule.c diff --git a/Modules/_bisectmodule.c b/Modules/_bisectmodule.c index c54dc4979d00b..19e9cd2d46f10 100644 --- a/Modules/_bisectmodule.c +++ b/Modules/_bisectmodule.c @@ -13,6 +13,18 @@ module _bisect #include "clinic/_bisectmodule.c.h" +typedef struct { + PyObject *str_insert; +} bisect_state; + +static inline bisect_state* +get_bisect_state(PyObject *module) +{ + void *state = PyModule_GetState(module); + assert(state != NULL); + return (bisect_state *)state; +} + static inline Py_ssize_t internal_bisect_right(PyObject *list, PyObject *item, Py_ssize_t lo, Py_ssize_t hi, PyObject* key) @@ -129,7 +141,8 @@ _bisect_insort_right_impl(PyObject *module, PyObject *a, PyObject *x, return NULL; } else { - result = PyObject_CallMethod(a, "insert", "nO", index, x); + bisect_state *state = get_bisect_state(module); + result = _PyObject_CallMethod(a, state->str_insert, "nO", index, x); if (result == NULL) return NULL; Py_DECREF(result); @@ -255,7 +268,8 @@ _bisect_insort_left_impl(PyObject *module, PyObject *a, PyObject *x, if (PyList_Insert(a, index, x) < 0) return NULL; } else { - result = PyObject_CallMethod(a, "insert", "nO", index, x); + bisect_state *state = get_bisect_state(module); + result = _PyObject_CallMethod(a, state->str_insert, "nO", index, x); if (result == NULL) return NULL; Py_DECREF(result); @@ -280,13 +294,45 @@ having to sort the list after each insertion. For long lists of items with\n\ expensive comparison operations, this can be an improvement over the more\n\ common approach.\n"); +static int +bisect_clear(PyObject *module) +{ + bisect_state *state = get_bisect_state(module); + Py_CLEAR(state->str_insert); + return 0; +} + +static void +bisect_free(void *module) +{ + bisect_clear((PyObject *)module); +} + +static int +bisect_modexec(PyObject *m) +{ + bisect_state *state = get_bisect_state(m); + state->str_insert = PyUnicode_InternFromString("insert"); + if (state->str_insert == NULL) { + return -1; + } + return 0; +} + +static PyModuleDef_Slot bisect_slots[] = { + {Py_mod_exec, bisect_modexec}, + {0, NULL} +}; static struct PyModuleDef _bisectmodule = { PyModuleDef_HEAD_INIT, .m_name = "_bisect", + .m_size = sizeof(bisect_state), .m_doc = module_doc, .m_methods = bisect_methods, - .m_size = 0 + .m_slots = bisect_slots, + .m_clear = bisect_clear, + .m_free = bisect_free, }; PyMODINIT_FUNC From webhook-mailer at python.org Tue Apr 19 09:57:09 2022 From: webhook-mailer at python.org (serhiy-storchaka) Date: Tue, 19 Apr 2022 13:57:09 -0000 Subject: [Python-checkins] Add more tests for group names and refs in RE (GH-91695) Message-ID: https://github.com/python/cpython/commit/74070085da5322ac83c954f101f2caa150655be2 commit: 74070085da5322ac83c954f101f2caa150655be2 branch: main author: Serhiy Storchaka committer: serhiy-storchaka date: 2022-04-19T16:56:51+03:00 summary: Add more tests for group names and refs in RE (GH-91695) files: M Lib/test/test_re.py diff --git a/Lib/test/test_re.py b/Lib/test/test_re.py index a1c27c9bc0f36..7bb8bfa8336e8 100644 --- a/Lib/test/test_re.py +++ b/Lib/test/test_re.py @@ -236,6 +236,16 @@ def test_symbolic_groups(self): re.compile(r'(?Px)(?P=a)(?(a)y)') re.compile(r'(?Px)(?P=a1)(?(a1)y)') re.compile(r'(?Px)\1(?(1)y)') + re.compile(b'(?Px)(?P=a1)(?(a1)y)') + # New valid identifiers in Python 3 + re.compile('(?Px)(?P=?)(?(?)y)') + re.compile('(?Px)(?P=???????)(?(???????)y)') + # Support > 100 groups. + pat = '|'.join('x(?P%x)y' % (i, i) for i in range(1, 200 + 1)) + pat = '(?:%s)(?(200)z|t)' % pat + self.assertEqual(re.match(pat, 'xc8yz').span(), (0, 5)) + + def test_symbolic_groups_errors(self): self.checkPatternError(r'(?P)(?P)', "redefinition of group name 'a' as group 2; " "was group 1") @@ -261,16 +271,22 @@ def test_symbolic_groups(self): self.checkPatternError(r'(?(-1))', "bad character in group name '-1'", 3) self.checkPatternError(r'(?(1a))', "bad character in group name '1a'", 3) self.checkPatternError(r'(?(a.))', "bad character in group name 'a.'", 3) - # New valid/invalid identifiers in Python 3 - re.compile('(?Px)(?P=?)(?(?)y)') - re.compile('(?Px)(?P=???????)(?(???????)y)') self.checkPatternError('(?Px)', "bad character in group name '?'", 4) + self.checkPatternError('(?P=?)', "bad character in group name '?'", 4) + self.checkPatternError('(?(?)y)', "bad character in group name '?'", 3) + + def test_symbolic_refs(self): + self.assertEqual(re.sub('(?Px)|(?Py)', r'\g', 'xx'), '') + self.assertEqual(re.sub('(?Px)|(?Py)', r'\2', 'xx'), '') + self.assertEqual(re.sub(b'(?Px)', br'\g', b'xx'), b'xx') + # New valid identifiers in Python 3 + self.assertEqual(re.sub('(?Px)', r'\g', 'xx'), 'xx') + self.assertEqual(re.sub('(?Px)', r'\g', 'xx'), 'xx') # Support > 100 groups. pat = '|'.join('x(?P%x)y' % (i, i) for i in range(1, 200 + 1)) - pat = '(?:%s)(?(200)z|t)' % pat - self.assertEqual(re.match(pat, 'xc8yz').span(), (0, 5)) + self.assertEqual(re.sub(pat, r'\g<200>', 'xc8yzxc8y'), 'c8zc8') - def test_symbolic_refs(self): + def test_symbolic_refs_errors(self): self.checkTemplateError('(?Px)', r'\g, unterminated name', 3) self.checkTemplateError('(?Px)', r'\g<', 'xx', @@ -288,18 +304,14 @@ def test_symbolic_refs(self): 'invalid group reference 2', 1) with self.assertRaisesRegex(IndexError, "unknown group name 'ab'"): re.sub('(?Px)', r'\g', 'xx') - self.assertEqual(re.sub('(?Px)|(?Py)', r'\g', 'xx'), '') - self.assertEqual(re.sub('(?Px)|(?Py)', r'\2', 'xx'), '') self.checkTemplateError('(?Px)', r'\g<-1>', 'xx', "bad character in group name '-1'", 3) - # New valid/invalid identifiers in Python 3 - self.assertEqual(re.sub('(?Px)', r'\g', 'xx'), 'xx') - self.assertEqual(re.sub('(?Px)', r'\g', 'xx'), 'xx') self.checkTemplateError('(?Px)', r'\g', 'xx', "bad character in group name '?'", 3) - # Support > 100 groups. - pat = '|'.join('x(?P%x)y' % (i, i) for i in range(1, 200 + 1)) - self.assertEqual(re.sub(pat, r'\g<200>', 'xc8yzxc8y'), 'c8zc8') + self.checkTemplateError('(?Px)', r'\g', 'xx', + "bad character in group name '?'", 3) + self.checkTemplateError('(?Px)', r'\g', 'xx', + "bad character in group name '?'", 3) def test_re_subn(self): self.assertEqual(re.subn("(?i)b+", "x", "bbbb BBBB"), ('x x', 2)) @@ -561,9 +573,23 @@ def test_re_groupref_exists(self): pat = '(?:%s)(?(200)z)' % pat self.assertEqual(re.match(pat, 'xc8yz').span(), (0, 5)) - self.checkPatternError(r'(?P)(?(0))', 'bad group number', 10) + def test_re_groupref_exists_errors(self): + self.checkPatternError(r'(?P)(?(0)a|b)', 'bad group number', 10) + self.checkPatternError(r'()(?(-1)a|b)', + "bad character in group name '-1'", 5) + self.checkPatternError(r'()(?(?)a|b)', + "bad character in group name '?'", 5) + self.checkPatternError(r'()(?(?)a|b)', + "bad character in group name '?'", 5) + self.checkPatternError(r'()(?(1', + "missing ), unterminated name", 5) + self.checkPatternError(r'()(?(1)a', + "missing ), unterminated subpattern", 2) self.checkPatternError(r'()(?(1)a|b', 'missing ), unterminated subpattern', 2) + self.checkPatternError(r'()(?(1)a|b|c', + 'conditional backref with more than ' + 'two branches', 10) self.checkPatternError(r'()(?(1)a|b|c)', 'conditional backref with more than ' 'two branches', 10) From webhook-mailer at python.org Tue Apr 19 10:27:11 2022 From: webhook-mailer at python.org (vstinner) Date: Tue, 19 Apr 2022 14:27:11 -0000 Subject: [Python-checkins] gh-91231: Add shutdown_timeout to multiprocessing BaseManager (#32112) Message-ID: https://github.com/python/cpython/commit/061a8bf77c80036bed3ef4973fe0c99705c83fc6 commit: 061a8bf77c80036bed3ef4973fe0c99705c83fc6 branch: main author: Victor Stinner committer: vstinner date: 2022-04-19T16:27:00+02:00 summary: gh-91231: Add shutdown_timeout to multiprocessing BaseManager (#32112) Add an optional keyword 'shutdown_timeout' parameter to the multiprocessing.BaseManager constructor. Kill the process if terminate() takes longer than the timeout. Multiprocessing tests pass test.support.SHORT_TIMEOUT to BaseManager.shutdown_timeout. files: A Misc/NEWS.d/next/Library/2022-04-19-15-30-06.gh-issue-91231.AWy4Cs.rst M Doc/library/multiprocessing.rst M Lib/multiprocessing/managers.py M Lib/test/_test_multiprocessing.py diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst index ee40688781690..83aa5cb87f49f 100644 --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -1676,7 +1676,7 @@ Manager processes will be shutdown as soon as they are garbage collected or their parent process exits. The manager classes are defined in the :mod:`multiprocessing.managers` module: -.. class:: BaseManager([address[, authkey]]) +.. class:: BaseManager(address=None, authkey=None, serializer='pickle', ctx=None, *, shutdown_timeout=1.0) Create a BaseManager object. @@ -1691,6 +1691,20 @@ their parent process exits. The manager classes are defined in the *authkey* is ``None`` then ``current_process().authkey`` is used. Otherwise *authkey* is used and it must be a byte string. + *serializer* must be ``'pickle'`` (use :mod:`pickle` serialization) or + ``'xmlrpclib'`` (use :mod:`xmlrpc.client` serialization). + + *ctx* is a context object, or ``None`` (use the current context). See the + :func:`get_context` function. + + *shutdown_timeout* is a timeout in seconds used to wait until the process + used by the manager completes in the :meth:`shutdown` method. If the + shutdown times out, the process is terminated. If terminating the process + also times out, the process is killed. + + .. versionchanged: 3.11 + Added the *shutdown_timeout* parameter. + .. method:: start([initializer[, initargs]]) Start a subprocess to start the manager. If *initializer* is not ``None`` diff --git a/Lib/multiprocessing/managers.py b/Lib/multiprocessing/managers.py index d97381926d47b..3f6479b7e3a69 100644 --- a/Lib/multiprocessing/managers.py +++ b/Lib/multiprocessing/managers.py @@ -497,7 +497,7 @@ class BaseManager(object): _Server = Server def __init__(self, address=None, authkey=None, serializer='pickle', - ctx=None): + ctx=None, *, shutdown_timeout=1.0): if authkey is None: authkey = process.current_process().authkey self._address = address # XXX not final address if eg ('', 0) @@ -507,6 +507,7 @@ def __init__(self, address=None, authkey=None, serializer='pickle', self._serializer = serializer self._Listener, self._Client = listener_client[serializer] self._ctx = ctx or get_context() + self._shutdown_timeout = shutdown_timeout def get_server(self): ''' @@ -570,8 +571,8 @@ def start(self, initializer=None, initargs=()): self._state.value = State.STARTED self.shutdown = util.Finalize( self, type(self)._finalize_manager, - args=(self._process, self._address, self._authkey, - self._state, self._Client), + args=(self._process, self._address, self._authkey, self._state, + self._Client, self._shutdown_timeout), exitpriority=0 ) @@ -656,7 +657,8 @@ def __exit__(self, exc_type, exc_val, exc_tb): self.shutdown() @staticmethod - def _finalize_manager(process, address, authkey, state, _Client): + def _finalize_manager(process, address, authkey, state, _Client, + shutdown_timeout): ''' Shutdown the manager process; will be registered as a finalizer ''' @@ -671,15 +673,17 @@ def _finalize_manager(process, address, authkey, state, _Client): except Exception: pass - process.join(timeout=1.0) + process.join(timeout=shutdown_timeout) if process.is_alive(): util.info('manager still alive') if hasattr(process, 'terminate'): util.info('trying to `terminate()` manager process') process.terminate() - process.join(timeout=0.1) + process.join(timeout=shutdown_timeout) if process.is_alive(): util.info('manager still alive after terminate') + process.kill() + process.join() state.value = State.SHUTDOWN try: diff --git a/Lib/test/_test_multiprocessing.py b/Lib/test/_test_multiprocessing.py index bb73d9e7cc75e..427fc0c47a3ca 100644 --- a/Lib/test/_test_multiprocessing.py +++ b/Lib/test/_test_multiprocessing.py @@ -119,6 +119,9 @@ def _resource_unlink(name, rtype): else: TIMEOUT1, TIMEOUT2, TIMEOUT3 = 0.1, 0.1, 0.1 +# BaseManager.shutdown_timeout +SHUTDOWN_TIMEOUT = support.SHORT_TIMEOUT + HAVE_GETVALUE = not getattr(_multiprocessing, 'HAVE_BROKEN_SEM_GETVALUE', False) @@ -2897,7 +2900,7 @@ class _TestMyManager(BaseTestCase): ALLOWED_TYPES = ('manager',) def test_mymanager(self): - manager = MyManager() + manager = MyManager(shutdown_timeout=SHUTDOWN_TIMEOUT) manager.start() self.common(manager) manager.shutdown() @@ -2908,7 +2911,8 @@ def test_mymanager(self): self.assertIn(manager._process.exitcode, (0, -signal.SIGTERM)) def test_mymanager_context(self): - with MyManager() as manager: + manager = MyManager(shutdown_timeout=SHUTDOWN_TIMEOUT) + with manager: self.common(manager) # bpo-30356: BaseManager._finalize_manager() sends SIGTERM # to the manager process if it takes longer than 1 second to stop, @@ -2916,7 +2920,7 @@ def test_mymanager_context(self): self.assertIn(manager._process.exitcode, (0, -signal.SIGTERM)) def test_mymanager_context_prestarted(self): - manager = MyManager() + manager = MyManager(shutdown_timeout=SHUTDOWN_TIMEOUT) manager.start() with manager: self.common(manager) @@ -2978,8 +2982,8 @@ class _TestRemoteManager(BaseTestCase): @classmethod def _putter(cls, address, authkey): manager = QueueManager2( - address=address, authkey=authkey, serializer=SERIALIZER - ) + address=address, authkey=authkey, serializer=SERIALIZER, + shutdown_timeout=SHUTDOWN_TIMEOUT) manager.connect() queue = manager.get_queue() # Note that xmlrpclib will deserialize object as a list not a tuple @@ -2989,8 +2993,8 @@ def test_remote(self): authkey = os.urandom(32) manager = QueueManager( - address=(socket_helper.HOST, 0), authkey=authkey, serializer=SERIALIZER - ) + address=(socket_helper.HOST, 0), authkey=authkey, serializer=SERIALIZER, + shutdown_timeout=SHUTDOWN_TIMEOUT) manager.start() self.addCleanup(manager.shutdown) @@ -2999,8 +3003,8 @@ def test_remote(self): p.start() manager2 = QueueManager2( - address=manager.address, authkey=authkey, serializer=SERIALIZER - ) + address=manager.address, authkey=authkey, serializer=SERIALIZER, + shutdown_timeout=SHUTDOWN_TIMEOUT) manager2.connect() queue = manager2.get_queue() @@ -3020,7 +3024,8 @@ class _TestManagerRestart(BaseTestCase): @classmethod def _putter(cls, address, authkey): manager = QueueManager( - address=address, authkey=authkey, serializer=SERIALIZER) + address=address, authkey=authkey, serializer=SERIALIZER, + shutdown_timeout=SHUTDOWN_TIMEOUT) manager.connect() queue = manager.get_queue() queue.put('hello world') @@ -3028,7 +3033,8 @@ def _putter(cls, address, authkey): def test_rapid_restart(self): authkey = os.urandom(32) manager = QueueManager( - address=(socket_helper.HOST, 0), authkey=authkey, serializer=SERIALIZER) + address=(socket_helper.HOST, 0), authkey=authkey, + serializer=SERIALIZER, shutdown_timeout=SHUTDOWN_TIMEOUT) try: srvr = manager.get_server() addr = srvr.address @@ -3048,7 +3054,8 @@ def test_rapid_restart(self): manager.shutdown() manager = QueueManager( - address=addr, authkey=authkey, serializer=SERIALIZER) + address=addr, authkey=authkey, serializer=SERIALIZER, + shutdown_timeout=SHUTDOWN_TIMEOUT) try: manager.start() self.addCleanup(manager.shutdown) @@ -3059,7 +3066,8 @@ def test_rapid_restart(self): # (sporadic failure on buildbots) time.sleep(1.0) manager = QueueManager( - address=addr, authkey=authkey, serializer=SERIALIZER) + address=addr, authkey=authkey, serializer=SERIALIZER, + shutdown_timeout=SHUTDOWN_TIMEOUT) if hasattr(manager, "shutdown"): self.addCleanup(manager.shutdown) diff --git a/Misc/NEWS.d/next/Library/2022-04-19-15-30-06.gh-issue-91231.AWy4Cs.rst b/Misc/NEWS.d/next/Library/2022-04-19-15-30-06.gh-issue-91231.AWy4Cs.rst new file mode 100644 index 0000000000000..a61fd8b9e8a8b --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-19-15-30-06.gh-issue-91231.AWy4Cs.rst @@ -0,0 +1,3 @@ +Add an optional keyword *shutdown_timeout* parameter to the +:class:`multiprocessing.BaseManager` constructor. Kill the process if +terminate() takes longer than the timeout. Patch by Victor Stinner. From webhook-mailer at python.org Tue Apr 19 10:33:22 2022 From: webhook-mailer at python.org (miss-islington) Date: Tue, 19 Apr 2022 14:33:22 -0000 Subject: [Python-checkins] Add more tests for group names and refs in RE (GH-91695) Message-ID: https://github.com/python/cpython/commit/c213cccc9bffe1e6612ad19745cef8066d2e1938 commit: c213cccc9bffe1e6612ad19745cef8066d2e1938 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-19T07:33:09-07:00 summary: Add more tests for group names and refs in RE (GH-91695) (cherry picked from commit 74070085da5322ac83c954f101f2caa150655be2) Co-authored-by: Serhiy Storchaka files: M Lib/test/test_re.py diff --git a/Lib/test/test_re.py b/Lib/test/test_re.py index 3d0637817da98..9e5223a125b05 100644 --- a/Lib/test/test_re.py +++ b/Lib/test/test_re.py @@ -218,6 +218,16 @@ def test_symbolic_groups(self): re.compile(r'(?Px)(?P=a)(?(a)y)') re.compile(r'(?Px)(?P=a1)(?(a1)y)') re.compile(r'(?Px)\1(?(1)y)') + re.compile(b'(?Px)(?P=a1)(?(a1)y)') + # New valid identifiers in Python 3 + re.compile('(?Px)(?P=?)(?(?)y)') + re.compile('(?Px)(?P=???????)(?(???????)y)') + # Support > 100 groups. + pat = '|'.join('x(?P%x)y' % (i, i) for i in range(1, 200 + 1)) + pat = '(?:%s)(?(200)z|t)' % pat + self.assertEqual(re.match(pat, 'xc8yz').span(), (0, 5)) + + def test_symbolic_groups_errors(self): self.checkPatternError(r'(?P)(?P)', "redefinition of group name 'a' as group 2; " "was group 1") @@ -243,16 +253,22 @@ def test_symbolic_groups(self): self.checkPatternError(r'(?(-1))', "bad character in group name '-1'", 3) self.checkPatternError(r'(?(1a))', "bad character in group name '1a'", 3) self.checkPatternError(r'(?(a.))', "bad character in group name 'a.'", 3) - # New valid/invalid identifiers in Python 3 - re.compile('(?Px)(?P=?)(?(?)y)') - re.compile('(?Px)(?P=???????)(?(???????)y)') self.checkPatternError('(?Px)', "bad character in group name '?'", 4) + self.checkPatternError('(?P=?)', "bad character in group name '?'", 4) + self.checkPatternError('(?(?)y)', "bad character in group name '?'", 3) + + def test_symbolic_refs(self): + self.assertEqual(re.sub('(?Px)|(?Py)', r'\g', 'xx'), '') + self.assertEqual(re.sub('(?Px)|(?Py)', r'\2', 'xx'), '') + self.assertEqual(re.sub(b'(?Px)', br'\g', b'xx'), b'xx') + # New valid identifiers in Python 3 + self.assertEqual(re.sub('(?Px)', r'\g', 'xx'), 'xx') + self.assertEqual(re.sub('(?Px)', r'\g', 'xx'), 'xx') # Support > 100 groups. pat = '|'.join('x(?P%x)y' % (i, i) for i in range(1, 200 + 1)) - pat = '(?:%s)(?(200)z|t)' % pat - self.assertEqual(re.match(pat, 'xc8yz').span(), (0, 5)) + self.assertEqual(re.sub(pat, r'\g<200>', 'xc8yzxc8y'), 'c8zc8') - def test_symbolic_refs(self): + def test_symbolic_refs_errors(self): self.checkTemplateError('(?Px)', r'\g, unterminated name', 3) self.checkTemplateError('(?Px)', r'\g<', 'xx', @@ -270,18 +286,14 @@ def test_symbolic_refs(self): 'invalid group reference 2', 1) with self.assertRaisesRegex(IndexError, "unknown group name 'ab'"): re.sub('(?Px)', r'\g', 'xx') - self.assertEqual(re.sub('(?Px)|(?Py)', r'\g', 'xx'), '') - self.assertEqual(re.sub('(?Px)|(?Py)', r'\2', 'xx'), '') self.checkTemplateError('(?Px)', r'\g<-1>', 'xx', "bad character in group name '-1'", 3) - # New valid/invalid identifiers in Python 3 - self.assertEqual(re.sub('(?Px)', r'\g', 'xx'), 'xx') - self.assertEqual(re.sub('(?Px)', r'\g', 'xx'), 'xx') self.checkTemplateError('(?Px)', r'\g', 'xx', "bad character in group name '?'", 3) - # Support > 100 groups. - pat = '|'.join('x(?P%x)y' % (i, i) for i in range(1, 200 + 1)) - self.assertEqual(re.sub(pat, r'\g<200>', 'xc8yzxc8y'), 'c8zc8') + self.checkTemplateError('(?Px)', r'\g', 'xx', + "bad character in group name '?'", 3) + self.checkTemplateError('(?Px)', r'\g', 'xx', + "bad character in group name '?'", 3) def test_re_subn(self): self.assertEqual(re.subn("(?i)b+", "x", "bbbb BBBB"), ('x x', 2)) @@ -543,9 +555,23 @@ def test_re_groupref_exists(self): pat = '(?:%s)(?(200)z)' % pat self.assertEqual(re.match(pat, 'xc8yz').span(), (0, 5)) - self.checkPatternError(r'(?P)(?(0))', 'bad group number', 10) + def test_re_groupref_exists_errors(self): + self.checkPatternError(r'(?P)(?(0)a|b)', 'bad group number', 10) + self.checkPatternError(r'()(?(-1)a|b)', + "bad character in group name '-1'", 5) + self.checkPatternError(r'()(?(?)a|b)', + "bad character in group name '?'", 5) + self.checkPatternError(r'()(?(?)a|b)', + "bad character in group name '?'", 5) + self.checkPatternError(r'()(?(1', + "missing ), unterminated name", 5) + self.checkPatternError(r'()(?(1)a', + "missing ), unterminated subpattern", 2) self.checkPatternError(r'()(?(1)a|b', 'missing ), unterminated subpattern', 2) + self.checkPatternError(r'()(?(1)a|b|c', + 'conditional backref with more than ' + 'two branches', 10) self.checkPatternError(r'()(?(1)a|b|c)', 'conditional backref with more than ' 'two branches', 10) From webhook-mailer at python.org Tue Apr 19 10:35:43 2022 From: webhook-mailer at python.org (miss-islington) Date: Tue, 19 Apr 2022 14:35:43 -0000 Subject: [Python-checkins] Add more tests for group names and refs in RE (GH-91695) Message-ID: https://github.com/python/cpython/commit/63af7b3b11d93b6636afb83c91d7eb7d2558ba20 commit: 63af7b3b11d93b6636afb83c91d7eb7d2558ba20 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-19T07:35:21-07:00 summary: Add more tests for group names and refs in RE (GH-91695) (cherry picked from commit 74070085da5322ac83c954f101f2caa150655be2) Co-authored-by: Serhiy Storchaka files: M Lib/test/test_re.py diff --git a/Lib/test/test_re.py b/Lib/test/test_re.py index 48a609b5a0031..56e98b7aedce7 100644 --- a/Lib/test/test_re.py +++ b/Lib/test/test_re.py @@ -217,6 +217,16 @@ def test_symbolic_groups(self): re.compile(r'(?Px)(?P=a)(?(a)y)') re.compile(r'(?Px)(?P=a1)(?(a1)y)') re.compile(r'(?Px)\1(?(1)y)') + re.compile(b'(?Px)(?P=a1)(?(a1)y)') + # New valid identifiers in Python 3 + re.compile('(?Px)(?P=?)(?(?)y)') + re.compile('(?Px)(?P=???????)(?(???????)y)') + # Support > 100 groups. + pat = '|'.join('x(?P%x)y' % (i, i) for i in range(1, 200 + 1)) + pat = '(?:%s)(?(200)z|t)' % pat + self.assertEqual(re.match(pat, 'xc8yz').span(), (0, 5)) + + def test_symbolic_groups_errors(self): self.checkPatternError(r'(?P)(?P)', "redefinition of group name 'a' as group 2; " "was group 1") @@ -242,16 +252,22 @@ def test_symbolic_groups(self): self.checkPatternError(r'(?(-1))', "bad character in group name '-1'", 3) self.checkPatternError(r'(?(1a))', "bad character in group name '1a'", 3) self.checkPatternError(r'(?(a.))', "bad character in group name 'a.'", 3) - # New valid/invalid identifiers in Python 3 - re.compile('(?Px)(?P=?)(?(?)y)') - re.compile('(?Px)(?P=???????)(?(???????)y)') self.checkPatternError('(?Px)', "bad character in group name '?'", 4) + self.checkPatternError('(?P=?)', "bad character in group name '?'", 4) + self.checkPatternError('(?(?)y)', "bad character in group name '?'", 3) + + def test_symbolic_refs(self): + self.assertEqual(re.sub('(?Px)|(?Py)', r'\g', 'xx'), '') + self.assertEqual(re.sub('(?Px)|(?Py)', r'\2', 'xx'), '') + self.assertEqual(re.sub(b'(?Px)', br'\g', b'xx'), b'xx') + # New valid identifiers in Python 3 + self.assertEqual(re.sub('(?Px)', r'\g', 'xx'), 'xx') + self.assertEqual(re.sub('(?Px)', r'\g', 'xx'), 'xx') # Support > 100 groups. pat = '|'.join('x(?P%x)y' % (i, i) for i in range(1, 200 + 1)) - pat = '(?:%s)(?(200)z|t)' % pat - self.assertEqual(re.match(pat, 'xc8yz').span(), (0, 5)) + self.assertEqual(re.sub(pat, r'\g<200>', 'xc8yzxc8y'), 'c8zc8') - def test_symbolic_refs(self): + def test_symbolic_refs_errors(self): self.checkTemplateError('(?Px)', r'\g, unterminated name', 3) self.checkTemplateError('(?Px)', r'\g<', 'xx', @@ -269,18 +285,14 @@ def test_symbolic_refs(self): 'invalid group reference 2', 1) with self.assertRaisesRegex(IndexError, "unknown group name 'ab'"): re.sub('(?Px)', r'\g', 'xx') - self.assertEqual(re.sub('(?Px)|(?Py)', r'\g', 'xx'), '') - self.assertEqual(re.sub('(?Px)|(?Py)', r'\2', 'xx'), '') self.checkTemplateError('(?Px)', r'\g<-1>', 'xx', "bad character in group name '-1'", 3) - # New valid/invalid identifiers in Python 3 - self.assertEqual(re.sub('(?Px)', r'\g', 'xx'), 'xx') - self.assertEqual(re.sub('(?Px)', r'\g', 'xx'), 'xx') self.checkTemplateError('(?Px)', r'\g', 'xx', "bad character in group name '?'", 3) - # Support > 100 groups. - pat = '|'.join('x(?P%x)y' % (i, i) for i in range(1, 200 + 1)) - self.assertEqual(re.sub(pat, r'\g<200>', 'xc8yzxc8y'), 'c8zc8') + self.checkTemplateError('(?Px)', r'\g', 'xx', + "bad character in group name '?'", 3) + self.checkTemplateError('(?Px)', r'\g', 'xx', + "bad character in group name '?'", 3) def test_re_subn(self): self.assertEqual(re.subn("(?i)b+", "x", "bbbb BBBB"), ('x x', 2)) @@ -542,9 +554,23 @@ def test_re_groupref_exists(self): pat = '(?:%s)(?(200)z)' % pat self.assertEqual(re.match(pat, 'xc8yz').span(), (0, 5)) - self.checkPatternError(r'(?P)(?(0))', 'bad group number', 10) + def test_re_groupref_exists_errors(self): + self.checkPatternError(r'(?P)(?(0)a|b)', 'bad group number', 10) + self.checkPatternError(r'()(?(-1)a|b)', + "bad character in group name '-1'", 5) + self.checkPatternError(r'()(?(?)a|b)', + "bad character in group name '?'", 5) + self.checkPatternError(r'()(?(?)a|b)', + "bad character in group name '?'", 5) + self.checkPatternError(r'()(?(1', + "missing ), unterminated name", 5) + self.checkPatternError(r'()(?(1)a', + "missing ), unterminated subpattern", 2) self.checkPatternError(r'()(?(1)a|b', 'missing ), unterminated subpattern', 2) + self.checkPatternError(r'()(?(1)a|b|c', + 'conditional backref with more than ' + 'two branches', 10) self.checkPatternError(r'()(?(1)a|b|c)', 'conditional backref with more than ' 'two branches', 10) From webhook-mailer at python.org Tue Apr 19 10:49:56 2022 From: webhook-mailer at python.org (serhiy-storchaka) Date: Tue, 19 Apr 2022 14:49:56 -0000 Subject: [Python-checkins] gh-91616: re module, fix .fullmatch() mismatch when using Atomic Grouping or Possessive Quantifiers (GH-91681) Message-ID: https://github.com/python/cpython/commit/e4e8895ae36b44994e3f7b018345ba203ce6b55c commit: e4e8895ae36b44994e3f7b018345ba203ce6b55c branch: main author: Ma Lin committer: serhiy-storchaka date: 2022-04-19T17:49:36+03:00 summary: gh-91616: re module, fix .fullmatch() mismatch when using Atomic Grouping or Possessive Quantifiers (GH-91681) These jumps should use DO_JUMP0() instead of DO_JUMP(): - JUMP_POSS_REPEAT_1 - JUMP_POSS_REPEAT_2 - JUMP_ATOMIC_GROUP files: A Misc/NEWS.d/next/Library/2022-04-17-12-27-25.gh-issue-91616.gSQg69.rst M Lib/test/test_pathlib.py M Lib/test/test_re.py M Modules/_sre/sre_lib.h diff --git a/Lib/test/test_pathlib.py b/Lib/test/test_pathlib.py index 66e44479239cf..b8b08bf0ce1bb 100644 --- a/Lib/test/test_pathlib.py +++ b/Lib/test/test_pathlib.py @@ -1388,6 +1388,7 @@ class _BasePathTest(object): # | |-- dirD # | | `-- fileD # | `-- fileC + # | `-- novel.txt # |-- dirE # No permissions # |-- fileA # |-- linkA -> fileA @@ -1412,6 +1413,8 @@ def cleanup(): f.write(b"this is file B\n") with open(join('dirC', 'fileC'), 'wb') as f: f.write(b"this is file C\n") + with open(join('dirC', 'novel.txt'), 'wb') as f: + f.write(b"this is a novel\n") with open(join('dirC', 'dirD', 'fileD'), 'wb') as f: f.write(b"this is file D\n") os.chmod(join('dirE'), 0) @@ -1679,6 +1682,9 @@ def _check(glob, expected): p = P(BASE, "dirC") _check(p.rglob("file*"), ["dirC/fileC", "dirC/dirD/fileD"]) _check(p.rglob("*/*"), ["dirC/dirD/fileD"]) + # gh-91616, a re module regression + _check(p.rglob("*.txt"), ["dirC/novel.txt"]) + _check(p.rglob("*.*"), ["dirC/novel.txt"]) @os_helper.skip_unless_symlink def test_rglob_symlink_loop(self): @@ -1689,7 +1695,8 @@ def test_rglob_symlink_loop(self): expect = {'brokenLink', 'dirA', 'dirA/linkC', 'dirB', 'dirB/fileB', 'dirB/linkD', - 'dirC', 'dirC/dirD', 'dirC/dirD/fileD', 'dirC/fileC', + 'dirC', 'dirC/dirD', 'dirC/dirD/fileD', + 'dirC/fileC', 'dirC/novel.txt', 'dirE', 'fileA', 'linkA', diff --git a/Lib/test/test_re.py b/Lib/test/test_re.py index 7bb8bfa8336e8..781bfd6ea2eda 100644 --- a/Lib/test/test_re.py +++ b/Lib/test/test_re.py @@ -2242,6 +2242,10 @@ def test_fullmatch_possessive_quantifiers(self): self.assertIsNone(re.fullmatch(r'a*+', 'ab')) self.assertIsNone(re.fullmatch(r'a?+', 'ab')) self.assertIsNone(re.fullmatch(r'a{1,3}+', 'ab')) + self.assertTrue(re.fullmatch(r'a++b', 'ab')) + self.assertTrue(re.fullmatch(r'a*+b', 'ab')) + self.assertTrue(re.fullmatch(r'a?+b', 'ab')) + self.assertTrue(re.fullmatch(r'a{1,3}+b', 'ab')) self.assertTrue(re.fullmatch(r'(?:ab)++', 'ab')) self.assertTrue(re.fullmatch(r'(?:ab)*+', 'ab')) @@ -2251,6 +2255,10 @@ def test_fullmatch_possessive_quantifiers(self): self.assertIsNone(re.fullmatch(r'(?:ab)*+', 'abc')) self.assertIsNone(re.fullmatch(r'(?:ab)?+', 'abc')) self.assertIsNone(re.fullmatch(r'(?:ab){1,3}+', 'abc')) + self.assertTrue(re.fullmatch(r'(?:ab)++c', 'abc')) + self.assertTrue(re.fullmatch(r'(?:ab)*+c', 'abc')) + self.assertTrue(re.fullmatch(r'(?:ab)?+c', 'abc')) + self.assertTrue(re.fullmatch(r'(?:ab){1,3}+c', 'abc')) def test_findall_possessive_quantifiers(self): self.assertEqual(re.findall(r'a++', 'aab'), ['aa']) @@ -2286,6 +2294,10 @@ def test_fullmatch_atomic_grouping(self): self.assertIsNone(re.fullmatch(r'(?>a*)', 'ab')) self.assertIsNone(re.fullmatch(r'(?>a?)', 'ab')) self.assertIsNone(re.fullmatch(r'(?>a{1,3})', 'ab')) + self.assertTrue(re.fullmatch(r'(?>a+)b', 'ab')) + self.assertTrue(re.fullmatch(r'(?>a*)b', 'ab')) + self.assertTrue(re.fullmatch(r'(?>a?)b', 'ab')) + self.assertTrue(re.fullmatch(r'(?>a{1,3})b', 'ab')) self.assertTrue(re.fullmatch(r'(?>(?:ab)+)', 'ab')) self.assertTrue(re.fullmatch(r'(?>(?:ab)*)', 'ab')) @@ -2295,6 +2307,10 @@ def test_fullmatch_atomic_grouping(self): self.assertIsNone(re.fullmatch(r'(?>(?:ab)*)', 'abc')) self.assertIsNone(re.fullmatch(r'(?>(?:ab)?)', 'abc')) self.assertIsNone(re.fullmatch(r'(?>(?:ab){1,3})', 'abc')) + self.assertTrue(re.fullmatch(r'(?>(?:ab)+)c', 'abc')) + self.assertTrue(re.fullmatch(r'(?>(?:ab)*)c', 'abc')) + self.assertTrue(re.fullmatch(r'(?>(?:ab)?)c', 'abc')) + self.assertTrue(re.fullmatch(r'(?>(?:ab){1,3})c', 'abc')) def test_findall_atomic_grouping(self): self.assertEqual(re.findall(r'(?>a+)', 'aab'), ['aa']) @@ -2307,6 +2323,10 @@ def test_findall_atomic_grouping(self): self.assertEqual(re.findall(r'(?>(?:ab)?)', 'ababc'), ['ab', 'ab', '', '']) self.assertEqual(re.findall(r'(?>(?:ab){1,3})', 'ababc'), ['abab']) + def test_bug_gh91616(self): + self.assertTrue(re.fullmatch(r'(?s:(?>.*?\.).*)\Z', "a.txt")) # reproducer + self.assertTrue(re.fullmatch(r'(?s:(?=(?P.*?\.))(?P=g0).*)\Z', "a.txt")) + def get_debug_out(pat): with captured_stdout() as out: diff --git a/Misc/NEWS.d/next/Library/2022-04-17-12-27-25.gh-issue-91616.gSQg69.rst b/Misc/NEWS.d/next/Library/2022-04-17-12-27-25.gh-issue-91616.gSQg69.rst new file mode 100644 index 0000000000000..8f147237aed6b --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-17-12-27-25.gh-issue-91616.gSQg69.rst @@ -0,0 +1,2 @@ +:mod:`re` module, fix :meth:`~re.Pattern.fullmatch` mismatch when using Atomic +Grouping or Possessive Quantifiers. diff --git a/Modules/_sre/sre_lib.h b/Modules/_sre/sre_lib.h index db624aa896d6a..efd6fdeccce3f 100644 --- a/Modules/_sre/sre_lib.h +++ b/Modules/_sre/sre_lib.h @@ -1259,8 +1259,8 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) /* Check for minimum required matches. */ while (ctx->count < (Py_ssize_t)pattern[1]) { /* not enough matches */ - DO_JUMP(JUMP_POSS_REPEAT_1, jump_poss_repeat_1, - &pattern[3]); + DO_JUMP0(JUMP_POSS_REPEAT_1, jump_poss_repeat_1, + &pattern[3]); if (ret) { RETURN_ON_ERROR(ret); ctx->count++; @@ -1306,8 +1306,8 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) /* We have not reached the maximin matches, so try to match once more. */ - DO_JUMP(JUMP_POSS_REPEAT_2, jump_poss_repeat_2, - &pattern[3]); + DO_JUMP0(JUMP_POSS_REPEAT_2, jump_poss_repeat_2, + &pattern[3]); /* Check to see if the last attempted match succeeded. */ @@ -1348,15 +1348,15 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) TRACE(("|%p|%p|ATOMIC_GROUP\n", pattern, ptr)); /* Set the global Input pointer to this context's Input - pointer */ + pointer */ state->ptr = ptr; /* Evaluate the Atomic Group in a new context, terminating when the end of the group, represented by a SUCCESS op code, is reached. */ /* Group Pattern begins at an offset of 1 code. */ - DO_JUMP(JUMP_ATOMIC_GROUP, jump_atomic_group, - &pattern[1]); + DO_JUMP0(JUMP_ATOMIC_GROUP, jump_atomic_group, + &pattern[1]); /* Test Exit Condition */ RETURN_ON_ERROR(ret); From webhook-mailer at python.org Tue Apr 19 11:06:11 2022 From: webhook-mailer at python.org (vstinner) Date: Tue, 19 Apr 2022 15:06:11 -0000 Subject: [Python-checkins] gh-91231: multiprocessing BaseManager waits 1.0 second (#91701) Message-ID: https://github.com/python/cpython/commit/a885f10325eb2fc27cd50ace5614666ea688ab66 commit: a885f10325eb2fc27cd50ace5614666ea688ab66 branch: 3.10 author: Victor Stinner committer: vstinner date: 2022-04-19T17:06:00+02:00 summary: gh-91231: multiprocessing BaseManager waits 1.0 second (#91701) Shutting down a multiprocessing BaseManager now waits for 1 second until the process completes, rather than 0.1 second, after the process is terminated. files: M Lib/multiprocessing/managers.py diff --git a/Lib/multiprocessing/managers.py b/Lib/multiprocessing/managers.py index b6b4cdd9ac15e..22292c78b7b5b 100644 --- a/Lib/multiprocessing/managers.py +++ b/Lib/multiprocessing/managers.py @@ -677,7 +677,7 @@ def _finalize_manager(process, address, authkey, state, _Client): if hasattr(process, 'terminate'): util.info('trying to `terminate()` manager process') process.terminate() - process.join(timeout=0.1) + process.join(timeout=1.0) if process.is_alive(): util.info('manager still alive after terminate') From webhook-mailer at python.org Tue Apr 19 11:30:05 2022 From: webhook-mailer at python.org (miss-islington) Date: Tue, 19 Apr 2022 15:30:05 -0000 Subject: [Python-checkins] gh-91231: multiprocessing BaseManager waits 1.0 second (GH-91701) Message-ID: https://github.com/python/cpython/commit/3b6072e7f5ee5d4263a4c84a74b5ca36ef220ed6 commit: 3b6072e7f5ee5d4263a4c84a74b5ca36ef220ed6 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-19T08:29:57-07:00 summary: gh-91231: multiprocessing BaseManager waits 1.0 second (GH-91701) Shutting down a multiprocessing BaseManager now waits for 1 second until the process completes, rather than 0.1 second, after the process is terminated. (cherry picked from commit a885f10325eb2fc27cd50ace5614666ea688ab66) Co-authored-by: Victor Stinner files: M Lib/multiprocessing/managers.py diff --git a/Lib/multiprocessing/managers.py b/Lib/multiprocessing/managers.py index dfa566c6fc386..1f9295619f673 100644 --- a/Lib/multiprocessing/managers.py +++ b/Lib/multiprocessing/managers.py @@ -669,7 +669,7 @@ def _finalize_manager(process, address, authkey, state, _Client): if hasattr(process, 'terminate'): util.info('trying to `terminate()` manager process') process.terminate() - process.join(timeout=0.1) + process.join(timeout=1.0) if process.is_alive(): util.info('manager still alive after terminate') From webhook-mailer at python.org Tue Apr 19 12:37:19 2022 From: webhook-mailer at python.org (zooba) Date: Tue, 19 Apr 2022 16:37:19 -0000 Subject: [Python-checkins] bpo-40859: Update Windows build to use xz-5.2.5 (GH-20622) Message-ID: https://github.com/python/cpython/commit/a059395921e4402c13a860aaa8fc44fea2023aa3 commit: a059395921e4402c13a860aaa8fc44fea2023aa3 branch: main author: Ma Lin committer: zooba date: 2022-04-19T17:36:51+01:00 summary: bpo-40859: Update Windows build to use xz-5.2.5 (GH-20622) files: A Misc/NEWS.d/next/Windows/2020-06-04-10-42-04.bpo-40859.isKSw7.rst M PCbuild/get_externals.bat M PCbuild/liblzma.vcxproj M PCbuild/liblzma.vcxproj.filters M PCbuild/python.props diff --git a/Misc/NEWS.d/next/Windows/2020-06-04-10-42-04.bpo-40859.isKSw7.rst b/Misc/NEWS.d/next/Windows/2020-06-04-10-42-04.bpo-40859.isKSw7.rst new file mode 100644 index 0000000000000..ef4c727ad2866 --- /dev/null +++ b/Misc/NEWS.d/next/Windows/2020-06-04-10-42-04.bpo-40859.isKSw7.rst @@ -0,0 +1 @@ +Update Windows build to use xz-5.2.5 diff --git a/PCbuild/get_externals.bat b/PCbuild/get_externals.bat index 481e06d9fcd06..289b7f505b955 100644 --- a/PCbuild/get_externals.bat +++ b/PCbuild/get_externals.bat @@ -58,7 +58,7 @@ set libraries=%libraries% sqlite-3.38.1.0 if NOT "%IncludeTkinterSrc%"=="false" set libraries=%libraries% tcl-core-8.6.12.1 if NOT "%IncludeTkinterSrc%"=="false" set libraries=%libraries% tk-8.6.12.1 if NOT "%IncludeTkinterSrc%"=="false" set libraries=%libraries% tix-8.4.3.6 -set libraries=%libraries% xz-5.2.2 +set libraries=%libraries% xz-5.2.5 set libraries=%libraries% zlib-1.2.12 for %%e in (%libraries%) do ( diff --git a/PCbuild/liblzma.vcxproj b/PCbuild/liblzma.vcxproj index a6bd59ec0baa3..4dd42ab98a975 100644 --- a/PCbuild/liblzma.vcxproj +++ b/PCbuild/liblzma.vcxproj @@ -92,7 +92,7 @@ WIN32;HAVE_CONFIG_H;_LIB;%(PreprocessorDefinitions) - $(lzmaDir)windows;$(lzmaDir)src/liblzma/common;$(lzmaDir)src/common;$(lzmaDir)src/liblzma/api;$(lzmaDir)src/liblzma/check;$(lzmaDir)src/liblzma/delta;$(lzmaDir)src/liblzma/lz;$(lzmaDir)src/liblzma/lzma;$(lzmaDir)src/liblzma/rangecoder;$(lzmaDir)src/liblzma/simple;%(AdditionalIncludeDirectories) + $(lzmaDir)windows/vs2019;$(lzmaDir)src/liblzma/common;$(lzmaDir)src/common;$(lzmaDir)src/liblzma/api;$(lzmaDir)src/liblzma/check;$(lzmaDir)src/liblzma/delta;$(lzmaDir)src/liblzma/lz;$(lzmaDir)src/liblzma/lzma;$(lzmaDir)src/liblzma/rangecoder;$(lzmaDir)src/liblzma/simple;%(AdditionalIncludeDirectories) 4028;4113;4133;4244;4267;4996;%(DisableSpecificWarnings) @@ -238,7 +238,7 @@ - + diff --git a/PCbuild/liblzma.vcxproj.filters b/PCbuild/liblzma.vcxproj.filters index 3f58351fa9edb..ebe2a7d5fa9e5 100644 --- a/PCbuild/liblzma.vcxproj.filters +++ b/PCbuild/liblzma.vcxproj.filters @@ -428,7 +428,7 @@ Header Files - + Header Files diff --git a/PCbuild/python.props b/PCbuild/python.props index b9211f60d86d4..28975ff88ce77 100644 --- a/PCbuild/python.props +++ b/PCbuild/python.props @@ -63,7 +63,7 @@ $(ExternalsDir)\ $(ExternalsDir)sqlite-3.38.1.0\ $(ExternalsDir)bzip2-1.0.8\ - $(ExternalsDir)xz-5.2.2\ + $(ExternalsDir)xz-5.2.5\ $(ExternalsDir)libffi-3.4.2\ $(ExternalsDir)libffi-3.4.2\$(ArchName)\ $(libffiOutDir)include From webhook-mailer at python.org Tue Apr 19 12:41:02 2022 From: webhook-mailer at python.org (miss-islington) Date: Tue, 19 Apr 2022 16:41:02 -0000 Subject: [Python-checkins] [3.10] gh-91676 gh-91260 unittest.IsolatedAsyncioTestCase no longer leaks its executor (GH-91680) Message-ID: https://github.com/python/cpython/commit/cb3c85d40d2dfc1d518ebd7afd3d82819e38a390 commit: cb3c85d40d2dfc1d518ebd7afd3d82819e38a390 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-19T09:40:52-07:00 summary: [3.10] gh-91676 gh-91260 unittest.IsolatedAsyncioTestCase no longer leaks its executor (GH-91680) For things like test_asyncio.test_thread this was causing frequent "environment modified by test" errors as the executor threads had not always stopped running after the test was over. (cherry picked from commit 61570ae0bc1507a62bee9d8b9bc2ff7155da7061) Co-authored-by: Gregory P. Smith files: A Misc/NEWS.d/next/Library/2022-04-19-04-33-39.gh-issue-91676.ceQBwh.rst M Lib/unittest/async_case.py diff --git a/Lib/unittest/async_case.py b/Lib/unittest/async_case.py index 23231199f9870..d9c694e368255 100644 --- a/Lib/unittest/async_case.py +++ b/Lib/unittest/async_case.py @@ -148,6 +148,8 @@ def _tearDownAsyncioLoop(self): # shutdown asyncgens loop.run_until_complete(loop.shutdown_asyncgens()) finally: + # Prevent our executor environment from leaking to future tests. + loop.run_until_complete(loop.shutdown_default_executor()) asyncio.set_event_loop(None) loop.close() diff --git a/Misc/NEWS.d/next/Library/2022-04-19-04-33-39.gh-issue-91676.ceQBwh.rst b/Misc/NEWS.d/next/Library/2022-04-19-04-33-39.gh-issue-91676.ceQBwh.rst new file mode 100644 index 0000000000000..dfbaef4440e0a --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-19-04-33-39.gh-issue-91676.ceQBwh.rst @@ -0,0 +1,4 @@ +Fix :class:`unittest.IsolatedAsyncioTestCase` to shutdown the per test event +loop executor before returning from its ``run`` method so that a not yet +stopped or garbage collected executor state does not persist beyond the +test. From webhook-mailer at python.org Tue Apr 19 13:16:29 2022 From: webhook-mailer at python.org (iritkatriel) Date: Tue, 19 Apr 2022 17:16:29 -0000 Subject: [Python-checkins] gh-74166: make all_errors keyword-only (GH-91704) Message-ID: https://github.com/python/cpython/commit/3c4380651301f255ef8149b638a1fae205ede575 commit: 3c4380651301f255ef8149b638a1fae205ede575 branch: main author: Irit Katriel <1055913+iritkatriel at users.noreply.github.com> committer: iritkatriel <1055913+iritkatriel at users.noreply.github.com> date: 2022-04-19T18:16:20+01:00 summary: gh-74166: make all_errors keyword-only (GH-91704) files: M Doc/library/socket.rst M Lib/socket.py diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst index d7a440127ec93..cff5a32afb464 100755 --- a/Doc/library/socket.rst +++ b/Doc/library/socket.rst @@ -660,7 +660,7 @@ The following functions all create :ref:`socket objects `. Windows support added. -.. function:: create_connection(address[, timeout[, source_address[, all_errors]]]) +.. function:: create_connection(address, timeout=GLOBAL_DEFAULT, source_address=None, *, all_errors=False) Connect to a TCP service listening on the internet *address* (a 2-tuple ``(host, port)``), and return the socket object. This is a higher-level diff --git a/Lib/socket.py b/Lib/socket.py index 28d9c89d601aa..e08fb620eb1be 100755 --- a/Lib/socket.py +++ b/Lib/socket.py @@ -806,7 +806,7 @@ def getfqdn(name=''): _GLOBAL_DEFAULT_TIMEOUT = object() def create_connection(address, timeout=_GLOBAL_DEFAULT_TIMEOUT, - source_address=None, all_errors=False): + source_address=None, *, all_errors=False): """Connect to *address* and return the socket object. Convenience function. Connect to *address* (a 2-tuple ``(host, From webhook-mailer at python.org Tue Apr 19 13:41:46 2022 From: webhook-mailer at python.org (ericsnowcurrently) Date: Tue, 19 Apr 2022 17:41:46 -0000 Subject: [Python-checkins] bpo-46712: share more global strings in deepfreeze (gh-32152) Message-ID: https://github.com/python/cpython/commit/ab0d35d70dfe0b4c11583f8f735a8cc49b58c58b commit: ab0d35d70dfe0b4c11583f8f735a8cc49b58c58b branch: main author: Kumar Aditya <59607654+kumaraditya303 at users.noreply.github.com> committer: ericsnowcurrently date: 2022-04-19T11:41:36-06:00 summary: bpo-46712: share more global strings in deepfreeze (gh-32152) (for gh-90868) files: M Modules/_io/textio.c M Modules/_pickle.c M Objects/unicodeobject.c M Python/compile.c M Tools/scripts/deepfreeze.py M Tools/scripts/generate_global_objects.py diff --git a/Modules/_io/textio.c b/Modules/_io/textio.c index f45a69787383f..f1cd6d01da859 100644 --- a/Modules/_io/textio.c +++ b/Modules/_io/textio.c @@ -1959,6 +1959,7 @@ _io_TextIOWrapper_read_impl(textio *self, Py_ssize_t n) if (chunks != NULL) { if (result != NULL && PyList_Append(chunks, result) < 0) goto fail; + _Py_DECLARE_STR(empty, ""); Py_XSETREF(result, PyUnicode_Join(&_Py_STR(empty), chunks)); if (result == NULL) goto fail; diff --git a/Modules/_pickle.c b/Modules/_pickle.c index 84f469dee9984..a5595eb10c8f1 100644 --- a/Modules/_pickle.c +++ b/Modules/_pickle.c @@ -1812,7 +1812,7 @@ get_dotted_path(PyObject *obj, PyObject *name) { PyObject *dotted_path; Py_ssize_t i, n; - + _Py_DECLARE_STR(dot, "."); dotted_path = PyUnicode_Split(name, &_Py_STR(dot), -1); if (dotted_path == NULL) return NULL; diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c index d46a52cdb8965..dd0c0221d9325 100644 --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -258,6 +258,7 @@ static int unicode_is_singleton(PyObject *unicode); // Return a borrowed reference to the empty string singleton. static inline PyObject* unicode_get_empty(void) { + _Py_DECLARE_STR(empty, ""); return &_Py_STR(empty); } diff --git a/Python/compile.c b/Python/compile.c index 4108b896ade46..ceaf85298d70f 100644 --- a/Python/compile.c +++ b/Python/compile.c @@ -782,6 +782,7 @@ compiler_set_qualname(struct compiler *c) } if (base != NULL) { + _Py_DECLARE_STR(dot, "."); name = PyUnicode_Concat(base, &_Py_STR(dot)); Py_DECREF(base); if (name == NULL) @@ -3945,6 +3946,7 @@ compiler_from_import(struct compiler *c, stmt_ty s) ADDOP_NAME(c, IMPORT_NAME, s->v.ImportFrom.module, names); } else { + _Py_DECLARE_STR(empty, ""); ADDOP_NAME(c, IMPORT_NAME, &_Py_STR(empty), names); } for (i = 0; i < n; i++) { @@ -4885,6 +4887,7 @@ compiler_joined_str(struct compiler *c, expr_ty e) Py_ssize_t value_count = asdl_seq_LEN(e->v.JoinedStr.values); if (value_count > STACK_USE_GUIDELINE) { + _Py_DECLARE_STR(empty, ""); ADDOP_LOAD_CONST_NEW(c, &_Py_STR(empty)); ADDOP_NAME(c, LOAD_METHOD, &_Py_ID(join), names); ADDOP_I(c, BUILD_LIST, 0); diff --git a/Tools/scripts/deepfreeze.py b/Tools/scripts/deepfreeze.py index dfa4b3a8eeb01..3c48bac2648d0 100644 --- a/Tools/scripts/deepfreeze.py +++ b/Tools/scripts/deepfreeze.py @@ -18,7 +18,7 @@ from generate_global_objects import get_identifiers_and_strings verbose = False -identifiers = get_identifiers_and_strings()[0] +identifiers, strings = get_identifiers_and_strings() def isprintable(b: bytes) -> bool: return all(0x20 <= c < 0x7f for c in b) @@ -168,6 +168,8 @@ def generate_bytes(self, name: str, b: bytes) -> str: return f"& {name}.ob_base.ob_base" def generate_unicode(self, name: str, s: str) -> str: + if s in strings: + return f"&_Py_STR({strings[s]})" if s in identifiers: return f"&_Py_ID({s})" if re.match(r'\A[A-Za-z0-9_]+\Z', s): diff --git a/Tools/scripts/generate_global_objects.py b/Tools/scripts/generate_global_objects.py index 826f4c4c83aec..2180acd7ea94b 100644 --- a/Tools/scripts/generate_global_objects.py +++ b/Tools/scripts/generate_global_objects.py @@ -1,20 +1,13 @@ import contextlib -import glob import io import os.path import re -import sys - __file__ = os.path.abspath(__file__) ROOT = os.path.dirname(os.path.dirname(os.path.dirname(__file__))) INTERNAL = os.path.join(ROOT, 'Include', 'internal') -STRING_LITERALS = { - 'empty': '', - 'dot': '.', -} IGNORED = { 'ACTION', # Python/_warnings.c 'ATTR', # Python/_warnings.c and Objects/funcobject.c @@ -211,7 +204,7 @@ def generate_global_strings(identifiers, strings): printer.write(START) with printer.block('struct _Py_global_strings', ';'): with printer.block('struct', ' literals;'): - for name, literal in sorted(strings.items()): + for literal, name in sorted(strings.items(), key=lambda x: x[1]): printer.write(f'STRUCT_FOR_STR({name}, "{literal}")') outfile.write('\n') with printer.block('struct', ' identifiers;'): @@ -276,7 +269,7 @@ def generate_runtime_init(identifiers, strings): # Global strings. with printer.block('.strings =', ','): with printer.block('.literals =', ','): - for name, literal in sorted(strings.items()): + for literal, name in sorted(strings.items(), key=lambda x: x[1]): printer.write(f'INIT_STR({name}, "{literal}"),') with printer.block('.identifiers =', ','): for name in sorted(identifiers): @@ -297,15 +290,15 @@ def generate_runtime_init(identifiers, strings): def get_identifiers_and_strings() -> 'tuple[set[str], dict[str, str]]': identifiers = set(IDENTIFIERS) - strings = dict(STRING_LITERALS) + strings = {} for name, string, *_ in iter_global_strings(): if string is None: if name not in IGNORED: identifiers.add(name) else: - if name not in strings: - strings[name] = string - elif string != strings[name]: + if string not in strings: + strings[string] = name + elif name != strings[string]: raise ValueError(f'string mismatch for {name!r} ({string!r} != {strings[name]!r}') return identifiers, strings From webhook-mailer at python.org Tue Apr 19 14:02:29 2022 From: webhook-mailer at python.org (markshannon) Date: Tue, 19 Apr 2022 18:02:29 -0000 Subject: [Python-checkins] gh-90667: Add specializations of Py_DECREF when types are known (GH-30872) Message-ID: https://github.com/python/cpython/commit/da6c78584b1f45ce3766bf7f27fb033169715292 commit: da6c78584b1f45ce3766bf7f27fb033169715292 branch: main author: Dennis Sweeney <36520290+sweeneyde at users.noreply.github.com> committer: markshannon date: 2022-04-19T19:02:19+01:00 summary: gh-90667: Add specializations of Py_DECREF when types are known (GH-30872) files: A Misc/NEWS.d/next/Core and Builtins/2022-01-25-05-39-38.bpo-46509.ljrqrc.rst M Include/internal/pycore_floatobject.h M Include/internal/pycore_object.h M Include/internal/pycore_pyerrors.h M Include/internal/pycore_unicodeobject.h M Objects/boolobject.c M Objects/floatobject.c M Objects/longobject.c M Objects/object.c M Objects/tupleobject.c M Objects/unicodeobject.c M Python/bltinmodule.c M Python/ceval.c diff --git a/Include/internal/pycore_floatobject.h b/Include/internal/pycore_floatobject.h index a099f2ebc0fec..8a655543329f3 100644 --- a/Include/internal/pycore_floatobject.h +++ b/Include/internal/pycore_floatobject.h @@ -38,6 +38,8 @@ struct _Py_float_state { #endif }; +void _PyFloat_ExactDealloc(PyObject *op); + PyAPI_FUNC(void) _PyFloat_DebugMallocStats(FILE* out); diff --git a/Include/internal/pycore_object.h b/Include/internal/pycore_object.h index 177b06e2dd4f7..f022f8246989d 100644 --- a/Include/internal/pycore_object.h +++ b/Include/internal/pycore_object.h @@ -14,7 +14,6 @@ extern "C" { #include "pycore_pystate.h" // _PyInterpreterState_GET() #include "pycore_runtime.h" // _PyRuntime - #define _PyObject_IMMORTAL_INIT(type) \ { \ .ob_refcnt = 999999999, \ @@ -26,6 +25,42 @@ extern "C" { .ob_size = size, \ } +PyAPI_FUNC(void) _Py_NO_RETURN _Py_FatalRefcountErrorFunc( + const char *func, + const char *message); + +#define _Py_FatalRefcountError(message) _Py_FatalRefcountErrorFunc(__func__, message) + +static inline void +_Py_DECREF_SPECIALIZED(PyObject *op, const destructor destruct) +{ +#ifdef Py_REF_DEBUG + _Py_RefTotal--; +#endif + if (--op->ob_refcnt != 0) { + assert(op->ob_refcnt > 0); + } + else { +#ifdef Py_TRACE_REFS + _Py_ForgetReference(op); +#endif + destruct(op); + } +} + +static inline void +_Py_DECREF_NO_DEALLOC(PyObject *op) +{ +#ifdef Py_REF_DEBUG + _Py_RefTotal--; +#endif + op->ob_refcnt--; +#ifdef Py_DEBUG + if (op->ob_refcnt <= 0) { + _Py_FatalRefcountError("Expected a positive remaining refcount"); + } +#endif +} PyAPI_FUNC(int) _PyType_CheckConsistency(PyTypeObject *type); PyAPI_FUNC(int) _PyDict_CheckConsistency(PyObject *mp, int check_content); diff --git a/Include/internal/pycore_pyerrors.h b/Include/internal/pycore_pyerrors.h index e3c445ba5d926..66f37942ef916 100644 --- a/Include/internal/pycore_pyerrors.h +++ b/Include/internal/pycore_pyerrors.h @@ -100,13 +100,6 @@ extern PyObject* _Py_Offer_Suggestions(PyObject* exception); PyAPI_FUNC(Py_ssize_t) _Py_UTF8_Edit_Cost(PyObject *str_a, PyObject *str_b, Py_ssize_t max_cost); -PyAPI_FUNC(void) _Py_NO_RETURN _Py_FatalRefcountErrorFunc( - const char *func, - const char *message); - -#define _Py_FatalRefcountError(message) _Py_FatalRefcountErrorFunc(__func__, message) - - #ifdef __cplusplus } #endif diff --git a/Include/internal/pycore_unicodeobject.h b/Include/internal/pycore_unicodeobject.h index 75b90501db156..4bee2419fbd98 100644 --- a/Include/internal/pycore_unicodeobject.h +++ b/Include/internal/pycore_unicodeobject.h @@ -10,6 +10,7 @@ extern "C" { #include "pycore_fileutils.h" // _Py_error_handler +void _PyUnicode_ExactDealloc(PyObject *op); /* runtime lifecycle */ diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-01-25-05-39-38.bpo-46509.ljrqrc.rst b/Misc/NEWS.d/next/Core and Builtins/2022-01-25-05-39-38.bpo-46509.ljrqrc.rst new file mode 100644 index 0000000000000..e19ce0a243c6c --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2022-01-25-05-39-38.bpo-46509.ljrqrc.rst @@ -0,0 +1 @@ +Add type-specialized versions of the ``Py_DECREF()``, and use them for ``float``, ``int``, ``str``, ``bool``, and ``None`` to avoid pointer-chasing at runtime where types are known at C compile time. diff --git a/Objects/boolobject.c b/Objects/boolobject.c index d86958aff9ccb..ff7218760ab36 100644 --- a/Objects/boolobject.c +++ b/Objects/boolobject.c @@ -1,8 +1,8 @@ /* Boolean type, a subtype of int */ #include "Python.h" +#include "pycore_object.h" // _Py_FatalRefcountError() #include "pycore_runtime.h" // _Py_ID() -#include "pycore_pyerrors.h" // _Py_FatalRefcountError() /* We define bool_repr to return "False" or "True" */ diff --git a/Objects/floatobject.c b/Objects/floatobject.c index 736ddc95d6836..a5774b9e30066 100644 --- a/Objects/floatobject.c +++ b/Objects/floatobject.c @@ -238,28 +238,41 @@ PyFloat_FromString(PyObject *v) return result; } -static void -float_dealloc(PyFloatObject *op) +void +_PyFloat_ExactDealloc(PyObject *obj) { + assert(PyFloat_CheckExact(obj)); + PyFloatObject *op = (PyFloatObject *)obj; #if PyFloat_MAXFREELIST > 0 - if (PyFloat_CheckExact(op)) { - struct _Py_float_state *state = get_float_state(); + struct _Py_float_state *state = get_float_state(); #ifdef Py_DEBUG - // float_dealloc() must not be called after _PyFloat_Fini() - assert(state->numfree != -1); + // float_dealloc() must not be called after _PyFloat_Fini() + assert(state->numfree != -1); #endif - if (state->numfree >= PyFloat_MAXFREELIST) { - PyObject_Free(op); - return; - } - state->numfree++; - Py_SET_TYPE(op, (PyTypeObject *)state->free_list); - state->free_list = op; + if (state->numfree >= PyFloat_MAXFREELIST) { + PyObject_Free(op); + return; + } + state->numfree++; + Py_SET_TYPE(op, (PyTypeObject *)state->free_list); + state->free_list = op; +#else + PyObject_Free(op); +#endif +} + +static void +float_dealloc(PyObject *op) +{ + assert(PyFloat_Check(op)); +#if PyFloat_MAXFREELIST > 0 + if (PyFloat_CheckExact(op)) { + _PyFloat_ExactDealloc(op); } else #endif { - Py_TYPE(op)->tp_free((PyObject *)op); + Py_TYPE(op)->tp_free(op); } } diff --git a/Objects/longobject.c b/Objects/longobject.c index cc4aef31d743c..c104dcc14f986 100644 --- a/Objects/longobject.c +++ b/Objects/longobject.c @@ -36,7 +36,15 @@ medium_value(PyLongObject *x) #define IS_SMALL_INT(ival) (-_PY_NSMALLNEGINTS <= (ival) && (ival) < _PY_NSMALLPOSINTS) #define IS_SMALL_UINT(ival) ((ival) < _PY_NSMALLPOSINTS) -static inline int is_medium_int(stwodigits x) +static inline void +_Py_DECREF_INT(PyLongObject *op) +{ + assert(PyLong_CheckExact(op)); + _Py_DECREF_SPECIALIZED((PyObject *)op, PyObject_Free); +} + +static inline int +is_medium_int(stwodigits x) { /* Take care that we are comparing unsigned values. */ twodigits x_plus_mask = ((twodigits)x) + PyLong_MASK; @@ -58,7 +66,7 @@ maybe_small_long(PyLongObject *v) if (v && IS_MEDIUM_VALUE(v)) { stwodigits ival = medium_value(v); if (IS_SMALL_INT(ival)) { - Py_DECREF(v); + _Py_DECREF_INT(v); return (PyLongObject *)get_small_int((sdigit)ival); } } @@ -1856,7 +1864,7 @@ long_to_decimal_string_internal(PyObject *aa, #undef WRITE_DIGITS #undef WRITE_UNICODE_DIGITS - Py_DECREF(scratch); + _Py_DECREF_INT(scratch); if (writer) { writer->pos += strlen; } @@ -3561,15 +3569,15 @@ k_mul(PyLongObject *a, PyLongObject *b) */ i = Py_SIZE(ret) - shift; /* # digits after shift */ (void)v_isub(ret->ob_digit + shift, i, t2->ob_digit, Py_SIZE(t2)); - Py_DECREF(t2); + _Py_DECREF_INT(t2); (void)v_isub(ret->ob_digit + shift, i, t1->ob_digit, Py_SIZE(t1)); - Py_DECREF(t1); + _Py_DECREF_INT(t1); /* 6. t3 <- (ah+al)(bh+bl), and add into result. */ if ((t1 = x_add(ah, al)) == NULL) goto fail; - Py_DECREF(ah); - Py_DECREF(al); + _Py_DECREF_INT(ah); + _Py_DECREF_INT(al); ah = al = NULL; if (a == b) { @@ -3580,13 +3588,13 @@ k_mul(PyLongObject *a, PyLongObject *b) Py_DECREF(t1); goto fail; } - Py_DECREF(bh); - Py_DECREF(bl); + _Py_DECREF_INT(bh); + _Py_DECREF_INT(bl); bh = bl = NULL; t3 = k_mul(t1, t2); - Py_DECREF(t1); - Py_DECREF(t2); + _Py_DECREF_INT(t1); + _Py_DECREF_INT(t2); if (t3 == NULL) goto fail; assert(Py_SIZE(t3) >= 0); @@ -3594,7 +3602,7 @@ k_mul(PyLongObject *a, PyLongObject *b) * See the (*) comment after this function. */ (void)v_iadd(ret->ob_digit + shift, i, t3->ob_digit, Py_SIZE(t3)); - Py_DECREF(t3); + _Py_DECREF_INT(t3); return long_normalize(ret); @@ -3699,13 +3707,13 @@ k_lopsided_mul(PyLongObject *a, PyLongObject *b) /* Add into result. */ (void)v_iadd(ret->ob_digit + nbdone, Py_SIZE(ret) - nbdone, product->ob_digit, Py_SIZE(product)); - Py_DECREF(product); + _Py_DECREF_INT(product); bsize -= nbtouse; nbdone += nbtouse; } - Py_DECREF(bslice); + _Py_DECREF_INT(bslice); return long_normalize(ret); fail: @@ -5993,7 +6001,7 @@ PyTypeObject PyLong_Type = { 0, /* tp_init */ 0, /* tp_alloc */ long_new, /* tp_new */ - PyObject_Del, /* tp_free */ + PyObject_Free, /* tp_free */ }; static PyTypeObject Int_InfoType; diff --git a/Objects/object.c b/Objects/object.c index fe2d76f578e2a..8adb5065c2af4 100644 --- a/Objects/object.c +++ b/Objects/object.c @@ -9,7 +9,7 @@ #include "pycore_floatobject.h" // _PyFloat_DebugMallocStats() #include "pycore_initconfig.h" // _PyStatus_EXCEPTION() #include "pycore_namespace.h" // _PyNamespace_Type -#include "pycore_object.h" // _PyType_CheckConsistency() +#include "pycore_object.h" // _PyType_CheckConsistency(), _Py_FatalRefcountError() #include "pycore_pyerrors.h" // _PyErr_Occurred() #include "pycore_pymem.h" // _PyMem_IsPtrFreed() #include "pycore_pystate.h" // _PyThreadState_GET() diff --git a/Objects/tupleobject.c b/Objects/tupleobject.c index 306a9112b0600..f21d4da245965 100644 --- a/Objects/tupleobject.c +++ b/Objects/tupleobject.c @@ -5,8 +5,7 @@ #include "pycore_abstract.h" // _PyIndex_Check() #include "pycore_gc.h" // _PyObject_GC_IS_TRACKED() #include "pycore_initconfig.h" // _PyStatus_OK() -#include "pycore_object.h" // _PyObject_GC_TRACK() -#include "pycore_pyerrors.h" // _Py_FatalRefcountError() +#include "pycore_object.h" // _PyObject_GC_TRACK(), _Py_FatalRefcountError() /*[clinic input] class tuple "PyTupleObject *" "&PyTuple_Type" diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c index dd0c0221d9325..7768f66c95655 100644 --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -48,9 +48,8 @@ OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. #include "pycore_initconfig.h" // _PyStatus_OK() #include "pycore_interp.h" // PyInterpreterState.fs_codec #include "pycore_long.h" // _PyLong_FormatWriter() -#include "pycore_object.h" // _PyObject_GC_TRACK() +#include "pycore_object.h" // _PyObject_GC_TRACK(), _Py_FatalRefcountError() #include "pycore_pathconfig.h" // _Py_DumpPathConfig() -#include "pycore_pyerrors.h" // _Py_FatalRefcountError() #include "pycore_pylifecycle.h" // _Py_SetFileSystemEncoding() #include "pycore_pystate.h" // _PyInterpreterState_GET() #include "pycore_ucnhash.h" // _PyUnicode_Name_CAPI @@ -15369,6 +15368,13 @@ unicode_subtype_new(PyTypeObject *type, PyObject *unicode) return NULL; } +void +_PyUnicode_ExactDealloc(PyObject *op) +{ + assert(PyUnicode_CheckExact(op)); + unicode_dealloc(op); +} + PyDoc_STRVAR(unicode_doc, "str(object='') -> str\n\ str(bytes_or_buffer[, encoding[, errors]]) -> str\n\ diff --git a/Python/bltinmodule.c b/Python/bltinmodule.c index 9cfecc521d90c..84ebb680e0b8f 100644 --- a/Python/bltinmodule.c +++ b/Python/bltinmodule.c @@ -2511,7 +2511,7 @@ builtin_sum_impl(PyObject *module, PyObject *iterable, PyObject *start) } if (PyFloat_CheckExact(item)) { f_result += PyFloat_AS_DOUBLE(item); - Py_DECREF(item); + _Py_DECREF_SPECIALIZED(item, _PyFloat_ExactDealloc); continue; } if (PyLong_Check(item)) { diff --git a/Python/ceval.c b/Python/ceval.c index f523e52fe0116..45754ffbe7cb4 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -1977,8 +1977,8 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int STAT_INC(BINARY_OP, hit); PyObject *prod = _PyLong_Multiply((PyLongObject *)left, (PyLongObject *)right); SET_SECOND(prod); - Py_DECREF(right); - Py_DECREF(left); + _Py_DECREF_SPECIALIZED(right, PyObject_Free); + _Py_DECREF_SPECIALIZED(left, PyObject_Free); STACK_SHRINK(1); if (prod == NULL) { goto error; @@ -1998,8 +1998,8 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int ((PyFloatObject *)right)->ob_fval; PyObject *prod = PyFloat_FromDouble(dprod); SET_SECOND(prod); - Py_DECREF(right); - Py_DECREF(left); + _Py_DECREF_SPECIALIZED(right, _PyFloat_ExactDealloc); + _Py_DECREF_SPECIALIZED(left, _PyFloat_ExactDealloc); STACK_SHRINK(1); if (prod == NULL) { goto error; @@ -2017,8 +2017,8 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int STAT_INC(BINARY_OP, hit); PyObject *sub = _PyLong_Subtract((PyLongObject *)left, (PyLongObject *)right); SET_SECOND(sub); - Py_DECREF(right); - Py_DECREF(left); + _Py_DECREF_SPECIALIZED(right, PyObject_Free); + _Py_DECREF_SPECIALIZED(left, PyObject_Free); STACK_SHRINK(1); if (sub == NULL) { goto error; @@ -2037,8 +2037,8 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int double dsub = ((PyFloatObject *)left)->ob_fval - ((PyFloatObject *)right)->ob_fval; PyObject *sub = PyFloat_FromDouble(dsub); SET_SECOND(sub); - Py_DECREF(right); - Py_DECREF(left); + _Py_DECREF_SPECIALIZED(right, _PyFloat_ExactDealloc); + _Py_DECREF_SPECIALIZED(left, _PyFloat_ExactDealloc); STACK_SHRINK(1); if (sub == NULL) { goto error; @@ -2057,8 +2057,8 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int PyObject *res = PyUnicode_Concat(left, right); STACK_SHRINK(1); SET_TOP(res); - Py_DECREF(left); - Py_DECREF(right); + _Py_DECREF_SPECIALIZED(left, _PyUnicode_ExactDealloc); + _Py_DECREF_SPECIALIZED(right, _PyUnicode_ExactDealloc); if (TOP() == NULL) { goto error; } @@ -2090,10 +2090,10 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int * that the string is safe to mutate. */ assert(Py_REFCNT(left) >= 2); - Py_DECREF(left); // XXX never need to dealloc + _Py_DECREF_NO_DEALLOC(left); STACK_SHRINK(2); PyUnicode_Append(target_local, right); - Py_DECREF(right); + _Py_DECREF_SPECIALIZED(right, _PyUnicode_ExactDealloc); if (*target_local == NULL) { goto error; } @@ -2113,8 +2113,8 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int ((PyFloatObject *)right)->ob_fval; PyObject *sum = PyFloat_FromDouble(dsum); SET_SECOND(sum); - Py_DECREF(right); - Py_DECREF(left); + _Py_DECREF_SPECIALIZED(right, _PyFloat_ExactDealloc); + _Py_DECREF_SPECIALIZED(left, _PyFloat_ExactDealloc); STACK_SHRINK(1); if (sum == NULL) { goto error; @@ -2132,8 +2132,8 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int STAT_INC(BINARY_OP, hit); PyObject *sum = _PyLong_Add((PyLongObject *)left, (PyLongObject *)right); SET_SECOND(sum); - Py_DECREF(right); - Py_DECREF(left); + _Py_DECREF_SPECIALIZED(right, PyObject_Free); + _Py_DECREF_SPECIALIZED(left, PyObject_Free); STACK_SHRINK(1); if (sum == NULL) { goto error; @@ -2192,7 +2192,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int assert(res != NULL); Py_INCREF(res); STACK_SHRINK(1); - Py_DECREF(sub); + _Py_DECREF_SPECIALIZED(sub, PyObject_Free); SET_TOP(res); Py_DECREF(list); JUMPBY(INLINE_CACHE_ENTRIES_BINARY_SUBSCR); @@ -2217,7 +2217,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int assert(res != NULL); Py_INCREF(res); STACK_SHRINK(1); - Py_DECREF(sub); + _Py_DECREF_SPECIALIZED(sub, PyObject_Free); SET_TOP(res); Py_DECREF(tuple); JUMPBY(INLINE_CACHE_ENTRIES_BINARY_SUBSCR); @@ -2359,7 +2359,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int STACK_SHRINK(3); assert(old_value != NULL); Py_DECREF(old_value); - Py_DECREF(sub); + _Py_DECREF_SPECIALIZED(sub, PyObject_Free); Py_DECREF(list); JUMPBY(INLINE_CACHE_ENTRIES_STORE_SUBSCR); NOTRACE_DISPATCH(); @@ -3752,8 +3752,8 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int JUMPBY(INLINE_CACHE_ENTRIES_COMPARE_OP); NEXTOPARG(); STACK_SHRINK(2); - Py_DECREF(left); - Py_DECREF(right); + _Py_DECREF_SPECIALIZED(left, _PyFloat_ExactDealloc); + _Py_DECREF_SPECIALIZED(right, _PyFloat_ExactDealloc); assert(opcode == POP_JUMP_FORWARD_IF_FALSE || opcode == POP_JUMP_BACKWARD_IF_FALSE || opcode == POP_JUMP_FORWARD_IF_TRUE || @@ -3795,8 +3795,8 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int JUMPBY(INLINE_CACHE_ENTRIES_COMPARE_OP); NEXTOPARG(); STACK_SHRINK(2); - Py_DECREF(left); - Py_DECREF(right); + _Py_DECREF_SPECIALIZED(left, PyObject_Free); + _Py_DECREF_SPECIALIZED(right, PyObject_Free); assert(opcode == POP_JUMP_FORWARD_IF_FALSE || opcode == POP_JUMP_BACKWARD_IF_FALSE || opcode == POP_JUMP_FORWARD_IF_TRUE || @@ -3841,8 +3841,8 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int opcode == POP_JUMP_FORWARD_IF_TRUE || opcode == POP_JUMP_BACKWARD_IF_TRUE); STACK_SHRINK(2); - Py_DECREF(left); - Py_DECREF(right); + _Py_DECREF_SPECIALIZED(left, _PyUnicode_ExactDealloc); + _Py_DECREF_SPECIALIZED(right, _PyUnicode_ExactDealloc); assert(res == 0 || res == 1); int sign = 1 - res; int jump = (9 << (sign + 1)) & when_to_jump_mask; @@ -4008,11 +4008,11 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int PREDICTED(POP_JUMP_BACKWARD_IF_FALSE); PyObject *cond = POP(); if (Py_IsTrue(cond)) { - Py_DECREF(cond); + _Py_DECREF_NO_DEALLOC(cond); DISPATCH(); } if (Py_IsFalse(cond)) { - Py_DECREF(cond); + _Py_DECREF_NO_DEALLOC(cond); JUMPBY(-oparg); CHECK_EVAL_BREAKER(); DISPATCH(); @@ -4034,10 +4034,10 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int PREDICTED(POP_JUMP_FORWARD_IF_FALSE); PyObject *cond = POP(); if (Py_IsTrue(cond)) { - Py_DECREF(cond); + _Py_DECREF_NO_DEALLOC(cond); } else if (Py_IsFalse(cond)) { - Py_DECREF(cond); + _Py_DECREF_NO_DEALLOC(cond); JUMPBY(oparg); } else { @@ -4057,11 +4057,11 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int TARGET(POP_JUMP_BACKWARD_IF_TRUE) { PyObject *cond = POP(); if (Py_IsFalse(cond)) { - Py_DECREF(cond); + _Py_DECREF_NO_DEALLOC(cond); DISPATCH(); } if (Py_IsTrue(cond)) { - Py_DECREF(cond); + _Py_DECREF_NO_DEALLOC(cond); JUMPBY(-oparg); CHECK_EVAL_BREAKER(); DISPATCH(); @@ -4082,10 +4082,10 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int TARGET(POP_JUMP_FORWARD_IF_TRUE) { PyObject *cond = POP(); if (Py_IsFalse(cond)) { - Py_DECREF(cond); + _Py_DECREF_NO_DEALLOC(cond); } else if (Py_IsTrue(cond)) { - Py_DECREF(cond); + _Py_DECREF_NO_DEALLOC(cond); JUMPBY(oparg); } else { @@ -4110,7 +4110,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int CHECK_EVAL_BREAKER(); DISPATCH(); } - Py_DECREF(value); + _Py_DECREF_NO_DEALLOC(value); DISPATCH(); } @@ -4126,21 +4126,25 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int TARGET(POP_JUMP_BACKWARD_IF_NONE) { PyObject *value = POP(); if (Py_IsNone(value)) { - Py_DECREF(value); + _Py_DECREF_NO_DEALLOC(value); JUMPBY(-oparg); CHECK_EVAL_BREAKER(); - DISPATCH(); } - Py_DECREF(value); + else { + Py_DECREF(value); + } DISPATCH(); } TARGET(POP_JUMP_FORWARD_IF_NONE) { PyObject *value = POP(); if (Py_IsNone(value)) { + _Py_DECREF_NO_DEALLOC(value); JUMPBY(oparg); } - Py_DECREF(value); + else { + Py_DECREF(value); + } DISPATCH(); } @@ -4149,7 +4153,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int int err; if (Py_IsTrue(cond)) { STACK_SHRINK(1); - Py_DECREF(cond); + _Py_DECREF_NO_DEALLOC(cond); DISPATCH(); } if (Py_IsFalse(cond)) { @@ -4173,7 +4177,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int int err; if (Py_IsFalse(cond)) { STACK_SHRINK(1); - Py_DECREF(cond); + _Py_DECREF_NO_DEALLOC(cond); DISPATCH(); } if (Py_IsTrue(cond)) { From webhook-mailer at python.org Tue Apr 19 15:58:38 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Tue, 19 Apr 2022 19:58:38 -0000 Subject: [Python-checkins] [3.9] gh-91118: Fix docstrings that do not honor --without-doc-strings (GH-31769) (#91664) Message-ID: https://github.com/python/cpython/commit/312e16fb7be927521b453f427a28535f772a771c commit: 312e16fb7be927521b453f427a28535f772a771c branch: 3.9 author: Oleg Iarygin committer: JelleZijlstra date: 2022-04-19T12:58:34-07:00 summary: [3.9] gh-91118: Fix docstrings that do not honor --without-doc-strings (GH-31769) (#91664) Co-authored-by: ?ric Co-authored-by: Jelle Zijlstra (cherry picked from commit a573cb2fec664c645ab744658d7e941d72e1a398) files: A Misc/NEWS.d/next/Core and Builtins/2022-03-08-21-59-57.bpo-46962.UomDfz.rst A Misc/NEWS.d/next/Documentation/2022-03-08-22-10-38.bpo-46962.FIVe9I.rst M Doc/c-api/typeobj.rst M Doc/extending/newtypes_tutorial.rst M Doc/includes/custom.c M Doc/includes/custom2.c M Doc/includes/custom3.c M Doc/includes/custom4.c M Doc/includes/sublist.c M Modules/_ctypes/_ctypes.c M Modules/_ctypes/callbacks.c M Modules/_ctypes/callproc.c M Modules/_ctypes/cfield.c M Modules/_testcapimodule.c M Objects/genericaliasobject.c M Objects/picklebufobject.c diff --git a/Doc/c-api/typeobj.rst b/Doc/c-api/typeobj.rst index d58a53b1d69fb..d49f3ffa0a509 100644 --- a/Doc/c-api/typeobj.rst +++ b/Doc/c-api/typeobj.rst @@ -2482,7 +2482,7 @@ A basic static type:: PyVarObject_HEAD_INIT(NULL, 0) .tp_name = "mymod.MyObject", .tp_basicsize = sizeof(MyObject), - .tp_doc = "My objects", + .tp_doc = PyDoc_STR("My objects"), .tp_new = myobj_new, .tp_dealloc = (destructor)myobj_dealloc, .tp_repr = (reprfunc)myobj_repr, @@ -2512,7 +2512,7 @@ with a more verbose initializer:: 0, /* tp_setattro */ 0, /* tp_as_buffer */ 0, /* tp_flags */ - "My objects", /* tp_doc */ + PyDoc_STR("My objects"), /* tp_doc */ 0, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ @@ -2545,7 +2545,7 @@ A type that supports weakrefs, instance dicts, and hashing:: PyVarObject_HEAD_INIT(NULL, 0) .tp_name = "mymod.MyObject", .tp_basicsize = sizeof(MyObject), - .tp_doc = "My objects", + .tp_doc = PyDoc_STR("My objects"), .tp_weaklistoffset = offsetof(MyObject, weakreflist), .tp_dictoffset = offsetof(MyObject, inst_dict), .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HAVE_GC, @@ -2572,7 +2572,7 @@ to create instances (e.g. uses a separate factory func):: .tp_name = "mymod.MyStr", .tp_basicsize = sizeof(MyStr), .tp_base = NULL, // set to &PyUnicode_Type in module init - .tp_doc = "my custom str", + .tp_doc = PyDoc_STR("my custom str"), .tp_flags = Py_TPFLAGS_DEFAULT, .tp_new = NULL, .tp_repr = (reprfunc)myobj_repr, diff --git a/Doc/extending/newtypes_tutorial.rst b/Doc/extending/newtypes_tutorial.rst index 4b5eb69ba7701..7771e20520b1e 100644 --- a/Doc/extending/newtypes_tutorial.rst +++ b/Doc/extending/newtypes_tutorial.rst @@ -89,7 +89,7 @@ The second bit is the definition of the type object. :: static PyTypeObject CustomType = { PyVarObject_HEAD_INIT(NULL, 0) .tp_name = "custom.Custom", - .tp_doc = "Custom objects", + .tp_doc = PyDoc_STR("Custom objects"), .tp_basicsize = sizeof(CustomObject), .tp_itemsize = 0, .tp_flags = Py_TPFLAGS_DEFAULT, @@ -160,7 +160,7 @@ you will need to OR the corresponding flags. We provide a doc string for the type in :c:member:`~PyTypeObject.tp_doc`. :: - .tp_doc = "Custom objects", + .tp_doc = PyDoc_STR("Custom objects"), To enable object creation, we have to provide a :c:member:`~PyTypeObject.tp_new` handler. This is the equivalent of the Python method :meth:`__new__`, but diff --git a/Doc/includes/custom.c b/Doc/includes/custom.c index f361baf830dd1..26ca754964733 100644 --- a/Doc/includes/custom.c +++ b/Doc/includes/custom.c @@ -9,7 +9,7 @@ typedef struct { static PyTypeObject CustomType = { PyVarObject_HEAD_INIT(NULL, 0) .tp_name = "custom.Custom", - .tp_doc = "Custom objects", + .tp_doc = PyDoc_STR("Custom objects"), .tp_basicsize = sizeof(CustomObject), .tp_itemsize = 0, .tp_flags = Py_TPFLAGS_DEFAULT, diff --git a/Doc/includes/custom2.c b/Doc/includes/custom2.c index 5bacab7a2a971..2a3c59f8f04c3 100644 --- a/Doc/includes/custom2.c +++ b/Doc/includes/custom2.c @@ -98,7 +98,7 @@ static PyMethodDef Custom_methods[] = { static PyTypeObject CustomType = { PyVarObject_HEAD_INIT(NULL, 0) .tp_name = "custom2.Custom", - .tp_doc = "Custom objects", + .tp_doc = PyDoc_STR("Custom objects"), .tp_basicsize = sizeof(CustomObject), .tp_itemsize = 0, .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, diff --git a/Doc/includes/custom3.c b/Doc/includes/custom3.c index 2b7a99ecf96c7..5a47530f0a6b0 100644 --- a/Doc/includes/custom3.c +++ b/Doc/includes/custom3.c @@ -148,7 +148,7 @@ static PyMethodDef Custom_methods[] = { static PyTypeObject CustomType = { PyVarObject_HEAD_INIT(NULL, 0) .tp_name = "custom3.Custom", - .tp_doc = "Custom objects", + .tp_doc = PyDoc_STR("Custom objects"), .tp_basicsize = sizeof(CustomObject), .tp_itemsize = 0, .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, diff --git a/Doc/includes/custom4.c b/Doc/includes/custom4.c index 584992fc5f1a8..c7ee55578488e 100644 --- a/Doc/includes/custom4.c +++ b/Doc/includes/custom4.c @@ -160,7 +160,7 @@ static PyMethodDef Custom_methods[] = { static PyTypeObject CustomType = { PyVarObject_HEAD_INIT(NULL, 0) .tp_name = "custom4.Custom", - .tp_doc = "Custom objects", + .tp_doc = PyDoc_STR("Custom objects"), .tp_basicsize = sizeof(CustomObject), .tp_itemsize = 0, .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HAVE_GC, diff --git a/Doc/includes/sublist.c b/Doc/includes/sublist.c index b2c26e73ebaf7..b36dadf07eae8 100644 --- a/Doc/includes/sublist.c +++ b/Doc/includes/sublist.c @@ -31,7 +31,7 @@ SubList_init(SubListObject *self, PyObject *args, PyObject *kwds) static PyTypeObject SubListType = { PyVarObject_HEAD_INIT(NULL, 0) .tp_name = "sublist.SubList", - .tp_doc = "SubList objects", + .tp_doc = PyDoc_STR("SubList objects"), .tp_basicsize = sizeof(SubListObject), .tp_itemsize = 0, .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-03-08-21-59-57.bpo-46962.UomDfz.rst b/Misc/NEWS.d/next/Core and Builtins/2022-03-08-21-59-57.bpo-46962.UomDfz.rst new file mode 100644 index 0000000000000..98f19260a7ed2 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2022-03-08-21-59-57.bpo-46962.UomDfz.rst @@ -0,0 +1,9 @@ +Classes and functions that unconditionally declared their docstrings +ignoring the `--without-doc-strings` compilation flag no longer do so. + +The classes affected are :class:`pickle.PickleBuffer`, +:class:`testcapi.RecursingInfinitelyError`, and :class:`types.GenericAlias`. + +The functions affected are 24 methods in :mod:`ctypes`. + +Patch by Oleg Iarygin. diff --git a/Misc/NEWS.d/next/Documentation/2022-03-08-22-10-38.bpo-46962.FIVe9I.rst b/Misc/NEWS.d/next/Documentation/2022-03-08-22-10-38.bpo-46962.FIVe9I.rst new file mode 100644 index 0000000000000..f5b54013bd672 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2022-03-08-22-10-38.bpo-46962.FIVe9I.rst @@ -0,0 +1,4 @@ +All docstrings in code snippets are now wrapped into :func:`PyDoc_STR` to +follow the guideline of `PEP 7's Documentation Strings paragraph +`_. Patch +by Oleg Iarygin. diff --git a/Modules/_ctypes/_ctypes.c b/Modules/_ctypes/_ctypes.c index ceae67ebb1612..0ffc8858de641 100644 --- a/Modules/_ctypes/_ctypes.c +++ b/Modules/_ctypes/_ctypes.c @@ -181,7 +181,7 @@ static PyTypeObject DictRemover_Type = { 0, /* tp_as_buffer */ /* XXX should participate in GC? */ Py_TPFLAGS_DEFAULT, /* tp_flags */ - "deletes a key from a dictionary", /* tp_doc */ + PyDoc_STR("deletes a key from a dictionary"), /* tp_doc */ 0, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ @@ -563,8 +563,8 @@ UnionType_new(PyTypeObject *type, PyObject *args, PyObject *kwds) return StructUnionType_new(type, args, kwds, 0); } -static const char from_address_doc[] = -"C.from_address(integer) -> C instance\naccess a C instance at the specified address"; +PyDoc_STRVAR(from_address_doc, +"C.from_address(integer) -> C instance\naccess a C instance at the specified address"); static PyObject * CDataType_from_address(PyObject *type, PyObject *value) @@ -581,8 +581,8 @@ CDataType_from_address(PyObject *type, PyObject *value) return PyCData_AtAddress(type, buf); } -static const char from_buffer_doc[] = -"C.from_buffer(object, offset=0) -> C instance\ncreate a C instance from a writeable buffer"; +PyDoc_STRVAR(from_buffer_doc, +"C.from_buffer(object, offset=0) -> C instance\ncreate a C instance from a writeable buffer"); static int KeepRef(CDataObject *target, Py_ssize_t index, PyObject *keep); @@ -661,8 +661,8 @@ CDataType_from_buffer(PyObject *type, PyObject *args) return result; } -static const char from_buffer_copy_doc[] = -"C.from_buffer_copy(object, offset=0) -> C instance\ncreate a C instance from a readable buffer"; +PyDoc_STRVAR(from_buffer_copy_doc, +"C.from_buffer_copy(object, offset=0) -> C instance\ncreate a C instance from a readable buffer"); static PyObject * GenericPyCData_new(PyTypeObject *type, PyObject *args, PyObject *kwds); @@ -712,8 +712,8 @@ CDataType_from_buffer_copy(PyObject *type, PyObject *args) return result; } -static const char in_dll_doc[] = -"C.in_dll(dll, name) -> C instance\naccess a C instance in a dll"; +PyDoc_STRVAR(in_dll_doc, +"C.in_dll(dll, name) -> C instance\naccess a C instance in a dll"); static PyObject * CDataType_in_dll(PyObject *type, PyObject *args) @@ -774,8 +774,8 @@ CDataType_in_dll(PyObject *type, PyObject *args) return PyCData_AtAddress(type, address); } -static const char from_param_doc[] = -"Convert a Python object into a function call parameter."; +PyDoc_STRVAR(from_param_doc, +"Convert a Python object into a function call parameter."); static PyObject * CDataType_from_param(PyObject *type, PyObject *value) @@ -929,7 +929,7 @@ PyTypeObject PyCStructType_Type = { PyCStructType_setattro, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HAVE_GC, /* tp_flags */ - "metatype for the CData Objects", /* tp_doc */ + PyDoc_STR("metatype for the CData Objects"), /* tp_doc */ (traverseproc)CDataType_traverse, /* tp_traverse */ (inquiry)CDataType_clear, /* tp_clear */ 0, /* tp_richcompare */ @@ -971,7 +971,7 @@ static PyTypeObject UnionType_Type = { UnionType_setattro, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HAVE_GC, /* tp_flags */ - "metatype for the CData Objects", /* tp_doc */ + PyDoc_STR("metatype for the CData Objects"), /* tp_doc */ (traverseproc)CDataType_traverse, /* tp_traverse */ (inquiry)CDataType_clear, /* tp_clear */ 0, /* tp_richcompare */ @@ -1229,7 +1229,7 @@ PyTypeObject PyCPointerType_Type = { 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HAVE_GC, /* tp_flags */ - "metatype for the Pointer Objects", /* tp_doc */ + PyDoc_STR("metatype for the Pointer Objects"), /* tp_doc */ (traverseproc)CDataType_traverse, /* tp_traverse */ (inquiry)CDataType_clear, /* tp_clear */ 0, /* tp_richcompare */ @@ -1651,7 +1651,7 @@ PyTypeObject PyCArrayType_Type = { 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */ - "metatype for the Array Objects", /* tp_doc */ + PyDoc_STR("metatype for the Array Objects"), /* tp_doc */ 0, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ @@ -2345,7 +2345,7 @@ PyTypeObject PyCSimpleType_Type = { 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */ - "metatype for the PyCSimpleType Objects", /* tp_doc */ + PyDoc_STR("metatype for the PyCSimpleType Objects"), /* tp_doc */ 0, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ @@ -2627,7 +2627,7 @@ PyTypeObject PyCFuncPtrType_Type = { 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HAVE_GC, /* tp_flags */ - "metatype for C function pointers", /* tp_doc */ + PyDoc_STR("metatype for C function pointers"), /* tp_doc */ (traverseproc)CDataType_traverse, /* tp_traverse */ (inquiry)CDataType_clear, /* tp_clear */ 0, /* tp_richcompare */ @@ -2932,7 +2932,7 @@ PyTypeObject PyCData_Type = { 0, /* tp_setattro */ &PyCData_as_buffer, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */ - "XXX to be provided", /* tp_doc */ + PyDoc_STR("XXX to be provided"), /* tp_doc */ (traverseproc)PyCData_traverse, /* tp_traverse */ (inquiry)PyCData_clear, /* tp_clear */ 0, /* tp_richcompare */ @@ -4327,7 +4327,7 @@ PyTypeObject PyCFuncPtr_Type = { 0, /* tp_setattro */ &PyCData_as_buffer, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */ - "Function Pointer", /* tp_doc */ + PyDoc_STR("Function Pointer"), /* tp_doc */ (traverseproc)PyCFuncPtr_traverse, /* tp_traverse */ (inquiry)PyCFuncPtr_clear, /* tp_clear */ 0, /* tp_richcompare */ @@ -4481,7 +4481,7 @@ static PyTypeObject Struct_Type = { 0, /* tp_setattro */ &PyCData_as_buffer, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */ - "Structure base class", /* tp_doc */ + PyDoc_STR("Structure base class"), /* tp_doc */ (traverseproc)PyCData_traverse, /* tp_traverse */ (inquiry)PyCData_clear, /* tp_clear */ 0, /* tp_richcompare */ @@ -4523,7 +4523,7 @@ static PyTypeObject Union_Type = { 0, /* tp_setattro */ &PyCData_as_buffer, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */ - "Union base class", /* tp_doc */ + PyDoc_STR("Union base class"), /* tp_doc */ (traverseproc)PyCData_traverse, /* tp_traverse */ (inquiry)PyCData_clear, /* tp_clear */ 0, /* tp_richcompare */ @@ -4845,7 +4845,7 @@ PyTypeObject PyCArray_Type = { 0, /* tp_setattro */ &PyCData_as_buffer, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */ - "XXX to be provided", /* tp_doc */ + PyDoc_STR("XXX to be provided"), /* tp_doc */ (traverseproc)PyCData_traverse, /* tp_traverse */ (inquiry)PyCData_clear, /* tp_clear */ 0, /* tp_richcompare */ @@ -5064,7 +5064,7 @@ static PyTypeObject Simple_Type = { 0, /* tp_setattro */ &PyCData_as_buffer, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */ - "XXX to be provided", /* tp_doc */ + PyDoc_STR("XXX to be provided"), /* tp_doc */ (traverseproc)PyCData_traverse, /* tp_traverse */ (inquiry)PyCData_clear, /* tp_clear */ 0, /* tp_richcompare */ @@ -5448,7 +5448,7 @@ PyTypeObject PyCPointer_Type = { 0, /* tp_setattro */ &PyCData_as_buffer, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */ - "XXX to be provided", /* tp_doc */ + PyDoc_STR("XXX to be provided"), /* tp_doc */ (traverseproc)PyCData_traverse, /* tp_traverse */ (inquiry)PyCData_clear, /* tp_clear */ 0, /* tp_richcompare */ @@ -5475,12 +5475,12 @@ PyTypeObject PyCPointer_Type = { * Module initialization. */ -static const char module_docs[] = -"Create and manipulate C compatible data types in Python."; +PyDoc_STRVAR(module_docs, +"Create and manipulate C compatible data types in Python."); #ifdef MS_WIN32 -static const char comerror_doc[] = "Raised when a COM method call failed."; +PyDoc_STRVAR(comerror_doc, "Raised when a COM method call failed."); int comerror_init(PyObject *self, PyObject *args, PyObject *kwds) diff --git a/Modules/_ctypes/callbacks.c b/Modules/_ctypes/callbacks.c index e6e101829a699..dbe6f29fc8171 100644 --- a/Modules/_ctypes/callbacks.c +++ b/Modules/_ctypes/callbacks.c @@ -65,7 +65,7 @@ PyTypeObject PyCThunk_Type = { 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC, /* tp_flags */ - "CThunkObject", /* tp_doc */ + PyDoc_STR("CThunkObject"), /* tp_doc */ CThunkObject_traverse, /* tp_traverse */ CThunkObject_clear, /* tp_clear */ 0, /* tp_richcompare */ diff --git a/Modules/_ctypes/callproc.c b/Modules/_ctypes/callproc.c index 9aff890afe9de..b83f7364de0a1 100644 --- a/Modules/_ctypes/callproc.c +++ b/Modules/_ctypes/callproc.c @@ -1316,11 +1316,11 @@ _parse_voidp(PyObject *obj, void **address) #ifdef MS_WIN32 -static const char format_error_doc[] = +PyDoc_STRVAR(format_error_doc, "FormatError([integer]) -> string\n\ \n\ Convert a win32 error code into a string. If the error code is not\n\ -given, the return value of a call to GetLastError() is used.\n"; +given, the return value of a call to GetLastError() is used.\n"); static PyObject *format_error(PyObject *self, PyObject *args) { PyObject *result; @@ -1340,13 +1340,13 @@ static PyObject *format_error(PyObject *self, PyObject *args) return result; } -static const char load_library_doc[] = +PyDoc_STRVAR(load_library_doc, "LoadLibrary(name, load_flags) -> handle\n\ \n\ Load an executable (usually a DLL), and return a handle to it.\n\ The handle may be used to locate exported functions in this\n\ module. load_flags are as defined for LoadLibraryEx in the\n\ -Windows API.\n"; +Windows API.\n"); static PyObject *load_library(PyObject *self, PyObject *args) { const WCHAR *name; @@ -1394,10 +1394,10 @@ _Py_COMP_DIAG_POP #endif } -static const char free_library_doc[] = +PyDoc_STRVAR(free_library_doc, "FreeLibrary(handle) -> void\n\ \n\ -Free the handle of an executable previously loaded by LoadLibrary.\n"; +Free the handle of an executable previously loaded by LoadLibrary.\n"); static PyObject *free_library(PyObject *self, PyObject *args) { void *hMod; @@ -1417,8 +1417,8 @@ static PyObject *free_library(PyObject *self, PyObject *args) Py_RETURN_NONE; } -static const char copy_com_pointer_doc[] = -"CopyComPointer(src, dst) -> HRESULT value\n"; +PyDoc_STRVAR(copy_com_pointer_doc, +"CopyComPointer(src, dst) -> HRESULT value\n"); static PyObject * copy_com_pointer(PyObject *self, PyObject *args) @@ -1656,10 +1656,10 @@ call_cdeclfunction(PyObject *self, PyObject *args) /***************************************************************** * functions */ -static const char sizeof_doc[] = +PyDoc_STRVAR(sizeof_doc, "sizeof(C type) -> integer\n" "sizeof(C instance) -> integer\n" -"Return the size in bytes of a C instance"; +"Return the size in bytes of a C instance"); static PyObject * sizeof_func(PyObject *self, PyObject *obj) @@ -1677,10 +1677,10 @@ sizeof_func(PyObject *self, PyObject *obj) return NULL; } -static const char alignment_doc[] = +PyDoc_STRVAR(alignment_doc, "alignment(C type) -> integer\n" "alignment(C instance) -> integer\n" -"Return the alignment requirements of a C instance"; +"Return the alignment requirements of a C instance"); static PyObject * align_func(PyObject *self, PyObject *obj) @@ -1700,10 +1700,10 @@ align_func(PyObject *self, PyObject *obj) return NULL; } -static const char byref_doc[] = +PyDoc_STRVAR(byref_doc, "byref(C instance[, offset=0]) -> byref-object\n" "Return a pointer lookalike to a C instance, only usable\n" -"as function argument"; +"as function argument"); /* * We must return something which can be converted to a parameter, @@ -1744,9 +1744,9 @@ byref(PyObject *self, PyObject *args) return (PyObject *)parg; } -static const char addressof_doc[] = +PyDoc_STRVAR(addressof_doc, "addressof(C instance) -> integer\n" -"Return the address of the C instance internal buffer"; +"Return the address of the C instance internal buffer"); static PyObject * addressof(PyObject *self, PyObject *obj) diff --git a/Modules/_ctypes/cfield.c b/Modules/_ctypes/cfield.c index 06b98a6d66fc3..8582af9c3a209 100644 --- a/Modules/_ctypes/cfield.c +++ b/Modules/_ctypes/cfield.c @@ -320,7 +320,7 @@ PyTypeObject PyCField_Type = { 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC, /* tp_flags */ - "Structure/Union member", /* tp_doc */ + PyDoc_STR("Structure/Union member"), /* tp_doc */ (traverseproc)PyCField_traverse, /* tp_traverse */ (inquiry)PyCField_clear, /* tp_clear */ 0, /* tp_richcompare */ diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c index ad1b07454355e..90841270e4738 100644 --- a/Modules/_testcapimodule.c +++ b/Modules/_testcapimodule.c @@ -5990,7 +5990,7 @@ static PyTypeObject PyRecursingInfinitelyError_Type = { 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */ - "Instantiating this exception starts infinite recursion.", /* tp_doc */ + PyDoc_STR("Instantiating this exception starts infinite recursion."), /* tp_doc */ 0, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ diff --git a/Objects/genericaliasobject.c b/Objects/genericaliasobject.c index acbb01cfef92c..36dfbe43cd88c 100644 --- a/Objects/genericaliasobject.c +++ b/Objects/genericaliasobject.c @@ -293,6 +293,11 @@ subs_tvars(PyObject *obj, PyObject *params, PyObject **argitems) return obj; } +PyDoc_STRVAR(genericalias_doc, +"Represent a PEP 585 generic type\n" +"\n" +"E.g. for t = list[int], t.__origin__ is list and t.__args__ is (int,)."); + static PyObject * ga_getitem(PyObject *self, PyObject *item) { @@ -612,14 +617,11 @@ ga_new(PyTypeObject *type, PyObject *args, PyObject *kwds) // TODO: // - argument clinic? -// - __doc__? // - cache? PyTypeObject Py_GenericAliasType = { PyVarObject_HEAD_INIT(&PyType_Type, 0) .tp_name = "types.GenericAlias", - .tp_doc = "Represent a PEP 585 generic type\n" - "\n" - "E.g. for t = list[int], t.__origin__ is list and t.__args__ is (int,).", + .tp_doc = genericalias_doc, .tp_basicsize = sizeof(gaobject), .tp_dealloc = ga_dealloc, .tp_repr = ga_repr, diff --git a/Objects/picklebufobject.c b/Objects/picklebufobject.c index a135e5575e28c..aaa852cfbb05b 100644 --- a/Objects/picklebufobject.c +++ b/Objects/picklebufobject.c @@ -206,7 +206,7 @@ static PyMethodDef picklebuf_methods[] = { PyTypeObject PyPickleBuffer_Type = { PyVarObject_HEAD_INIT(NULL, 0) .tp_name = "pickle.PickleBuffer", - .tp_doc = "Wrapper for potentially out-of-band buffers", + .tp_doc = PyDoc_STR("Wrapper for potentially out-of-band buffers"), .tp_basicsize = sizeof(PyPickleBufferObject), .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC, .tp_new = picklebuf_new, From webhook-mailer at python.org Tue Apr 19 16:01:13 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Tue, 19 Apr 2022 20:01:13 -0000 Subject: [Python-checkins] [3.10] gh-91118: Fix docstrings that do not honor --without-doc-strings (GH-31769) (#91662) Message-ID: https://github.com/python/cpython/commit/e7e8a9fa4f7347bc82ea83ed82d447f447dfe325 commit: e7e8a9fa4f7347bc82ea83ed82d447f447dfe325 branch: 3.10 author: Oleg Iarygin committer: JelleZijlstra date: 2022-04-19T13:01:09-07:00 summary: [3.10] gh-91118: Fix docstrings that do not honor --without-doc-strings (GH-31769) (#91662) Co-authored-by: ?ric Co-authored-by: Jelle Zijlstra (cherry picked from commit a573cb2fec664c645ab744658d7e941d72e1a398) Co-authored-by: Oleg Iarygin files: A Misc/NEWS.d/next/Core and Builtins/2022-03-08-21-59-57.bpo-46962.UomDfz.rst A Misc/NEWS.d/next/Documentation/2022-03-08-22-10-38.bpo-46962.FIVe9I.rst M Doc/c-api/typeobj.rst M Doc/extending/newtypes_tutorial.rst M Doc/includes/custom.c M Doc/includes/custom2.c M Doc/includes/custom3.c M Doc/includes/custom4.c M Doc/includes/sublist.c M Modules/_ctypes/_ctypes.c M Modules/_ctypes/callbacks.c M Modules/_ctypes/callproc.c M Modules/_ctypes/cfield.c M Modules/_testcapimodule.c M Objects/genericaliasobject.c M Objects/picklebufobject.c M Objects/unionobject.c diff --git a/Doc/c-api/typeobj.rst b/Doc/c-api/typeobj.rst index 93492200ea44e..d24eabe7e8d55 100644 --- a/Doc/c-api/typeobj.rst +++ b/Doc/c-api/typeobj.rst @@ -2596,7 +2596,7 @@ A basic :ref:`static type `:: PyVarObject_HEAD_INIT(NULL, 0) .tp_name = "mymod.MyObject", .tp_basicsize = sizeof(MyObject), - .tp_doc = "My objects", + .tp_doc = PyDoc_STR("My objects"), .tp_new = myobj_new, .tp_dealloc = (destructor)myobj_dealloc, .tp_repr = (reprfunc)myobj_repr, @@ -2626,7 +2626,7 @@ with a more verbose initializer:: 0, /* tp_setattro */ 0, /* tp_as_buffer */ 0, /* tp_flags */ - "My objects", /* tp_doc */ + PyDoc_STR("My objects"), /* tp_doc */ 0, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ @@ -2659,7 +2659,7 @@ A type that supports weakrefs, instance dicts, and hashing:: PyVarObject_HEAD_INIT(NULL, 0) .tp_name = "mymod.MyObject", .tp_basicsize = sizeof(MyObject), - .tp_doc = "My objects", + .tp_doc = PyDoc_STR("My objects"), .tp_weaklistoffset = offsetof(MyObject, weakreflist), .tp_dictoffset = offsetof(MyObject, inst_dict), .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HAVE_GC, @@ -2687,7 +2687,7 @@ to create instances (e.g. uses a separate factory func) using .tp_name = "mymod.MyStr", .tp_basicsize = sizeof(MyStr), .tp_base = NULL, // set to &PyUnicode_Type in module init - .tp_doc = "my custom str", + .tp_doc = PyDoc_STR("my custom str"), .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_DISALLOW_INSTANTIATION, .tp_repr = (reprfunc)myobj_repr, }; diff --git a/Doc/extending/newtypes_tutorial.rst b/Doc/extending/newtypes_tutorial.rst index 904915306f1f3..34c25d1f6f199 100644 --- a/Doc/extending/newtypes_tutorial.rst +++ b/Doc/extending/newtypes_tutorial.rst @@ -90,7 +90,7 @@ The second bit is the definition of the type object. :: static PyTypeObject CustomType = { PyVarObject_HEAD_INIT(NULL, 0) .tp_name = "custom.Custom", - .tp_doc = "Custom objects", + .tp_doc = PyDoc_STR("Custom objects"), .tp_basicsize = sizeof(CustomObject), .tp_itemsize = 0, .tp_flags = Py_TPFLAGS_DEFAULT, @@ -161,7 +161,7 @@ you will need to OR the corresponding flags. We provide a doc string for the type in :c:member:`~PyTypeObject.tp_doc`. :: - .tp_doc = "Custom objects", + .tp_doc = PyDoc_STR("Custom objects"), To enable object creation, we have to provide a :c:member:`~PyTypeObject.tp_new` handler. This is the equivalent of the Python method :meth:`__new__`, but diff --git a/Doc/includes/custom.c b/Doc/includes/custom.c index f361baf830dd1..26ca754964733 100644 --- a/Doc/includes/custom.c +++ b/Doc/includes/custom.c @@ -9,7 +9,7 @@ typedef struct { static PyTypeObject CustomType = { PyVarObject_HEAD_INIT(NULL, 0) .tp_name = "custom.Custom", - .tp_doc = "Custom objects", + .tp_doc = PyDoc_STR("Custom objects"), .tp_basicsize = sizeof(CustomObject), .tp_itemsize = 0, .tp_flags = Py_TPFLAGS_DEFAULT, diff --git a/Doc/includes/custom2.c b/Doc/includes/custom2.c index 5bacab7a2a971..2a3c59f8f04c3 100644 --- a/Doc/includes/custom2.c +++ b/Doc/includes/custom2.c @@ -98,7 +98,7 @@ static PyMethodDef Custom_methods[] = { static PyTypeObject CustomType = { PyVarObject_HEAD_INIT(NULL, 0) .tp_name = "custom2.Custom", - .tp_doc = "Custom objects", + .tp_doc = PyDoc_STR("Custom objects"), .tp_basicsize = sizeof(CustomObject), .tp_itemsize = 0, .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, diff --git a/Doc/includes/custom3.c b/Doc/includes/custom3.c index 2b7a99ecf96c7..5a47530f0a6b0 100644 --- a/Doc/includes/custom3.c +++ b/Doc/includes/custom3.c @@ -148,7 +148,7 @@ static PyMethodDef Custom_methods[] = { static PyTypeObject CustomType = { PyVarObject_HEAD_INIT(NULL, 0) .tp_name = "custom3.Custom", - .tp_doc = "Custom objects", + .tp_doc = PyDoc_STR("Custom objects"), .tp_basicsize = sizeof(CustomObject), .tp_itemsize = 0, .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, diff --git a/Doc/includes/custom4.c b/Doc/includes/custom4.c index 584992fc5f1a8..c7ee55578488e 100644 --- a/Doc/includes/custom4.c +++ b/Doc/includes/custom4.c @@ -160,7 +160,7 @@ static PyMethodDef Custom_methods[] = { static PyTypeObject CustomType = { PyVarObject_HEAD_INIT(NULL, 0) .tp_name = "custom4.Custom", - .tp_doc = "Custom objects", + .tp_doc = PyDoc_STR("Custom objects"), .tp_basicsize = sizeof(CustomObject), .tp_itemsize = 0, .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HAVE_GC, diff --git a/Doc/includes/sublist.c b/Doc/includes/sublist.c index b2c26e73ebaf7..b36dadf07eae8 100644 --- a/Doc/includes/sublist.c +++ b/Doc/includes/sublist.c @@ -31,7 +31,7 @@ SubList_init(SubListObject *self, PyObject *args, PyObject *kwds) static PyTypeObject SubListType = { PyVarObject_HEAD_INIT(NULL, 0) .tp_name = "sublist.SubList", - .tp_doc = "SubList objects", + .tp_doc = PyDoc_STR("SubList objects"), .tp_basicsize = sizeof(SubListObject), .tp_itemsize = 0, .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-03-08-21-59-57.bpo-46962.UomDfz.rst b/Misc/NEWS.d/next/Core and Builtins/2022-03-08-21-59-57.bpo-46962.UomDfz.rst new file mode 100644 index 0000000000000..395c9b3d8f526 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2022-03-08-21-59-57.bpo-46962.UomDfz.rst @@ -0,0 +1,10 @@ +Classes and functions that unconditionally declared their docstrings +ignoring the `--without-doc-strings` compilation flag no longer do so. + +The classes affected are :class:`ctypes.UnionType`, +:class:`pickle.PickleBuffer`, :class:`testcapi.RecursingInfinitelyError`, +and :class:`types.GenericAlias`. + +The functions affected are 24 methods in :mod:`ctypes`. + +Patch by Oleg Iarygin. diff --git a/Misc/NEWS.d/next/Documentation/2022-03-08-22-10-38.bpo-46962.FIVe9I.rst b/Misc/NEWS.d/next/Documentation/2022-03-08-22-10-38.bpo-46962.FIVe9I.rst new file mode 100644 index 0000000000000..f5b54013bd672 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2022-03-08-22-10-38.bpo-46962.FIVe9I.rst @@ -0,0 +1,4 @@ +All docstrings in code snippets are now wrapped into :func:`PyDoc_STR` to +follow the guideline of `PEP 7's Documentation Strings paragraph +`_. Patch +by Oleg Iarygin. diff --git a/Modules/_ctypes/_ctypes.c b/Modules/_ctypes/_ctypes.c index 5f8a723f6373a..84378c40357b0 100644 --- a/Modules/_ctypes/_ctypes.c +++ b/Modules/_ctypes/_ctypes.c @@ -187,7 +187,7 @@ static PyTypeObject DictRemover_Type = { 0, /* tp_as_buffer */ /* XXX should participate in GC? */ Py_TPFLAGS_DEFAULT, /* tp_flags */ - "deletes a key from a dictionary", /* tp_doc */ + PyDoc_STR("deletes a key from a dictionary"), /* tp_doc */ 0, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ @@ -570,8 +570,8 @@ UnionType_new(PyTypeObject *type, PyObject *args, PyObject *kwds) return StructUnionType_new(type, args, kwds, 0); } -static const char from_address_doc[] = -"C.from_address(integer) -> C instance\naccess a C instance at the specified address"; +PyDoc_STRVAR(from_address_doc, +"C.from_address(integer) -> C instance\naccess a C instance at the specified address"); static PyObject * CDataType_from_address(PyObject *type, PyObject *value) @@ -588,8 +588,8 @@ CDataType_from_address(PyObject *type, PyObject *value) return PyCData_AtAddress(type, buf); } -static const char from_buffer_doc[] = -"C.from_buffer(object, offset=0) -> C instance\ncreate a C instance from a writeable buffer"; +PyDoc_STRVAR(from_buffer_doc, +"C.from_buffer(object, offset=0) -> C instance\ncreate a C instance from a writeable buffer"); static int KeepRef(CDataObject *target, Py_ssize_t index, PyObject *keep); @@ -668,8 +668,8 @@ CDataType_from_buffer(PyObject *type, PyObject *args) return result; } -static const char from_buffer_copy_doc[] = -"C.from_buffer_copy(object, offset=0) -> C instance\ncreate a C instance from a readable buffer"; +PyDoc_STRVAR(from_buffer_copy_doc, +"C.from_buffer_copy(object, offset=0) -> C instance\ncreate a C instance from a readable buffer"); static PyObject * GenericPyCData_new(PyTypeObject *type, PyObject *args, PyObject *kwds); @@ -719,8 +719,8 @@ CDataType_from_buffer_copy(PyObject *type, PyObject *args) return result; } -static const char in_dll_doc[] = -"C.in_dll(dll, name) -> C instance\naccess a C instance in a dll"; +PyDoc_STRVAR(in_dll_doc, +"C.in_dll(dll, name) -> C instance\naccess a C instance in a dll"); static PyObject * CDataType_in_dll(PyObject *type, PyObject *args) @@ -781,8 +781,8 @@ CDataType_in_dll(PyObject *type, PyObject *args) return PyCData_AtAddress(type, address); } -static const char from_param_doc[] = -"Convert a Python object into a function call parameter."; +PyDoc_STRVAR(from_param_doc, +"Convert a Python object into a function call parameter."); static PyObject * CDataType_from_param(PyObject *type, PyObject *value) @@ -936,7 +936,7 @@ PyTypeObject PyCStructType_Type = { PyCStructType_setattro, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HAVE_GC, /* tp_flags */ - "metatype for the CData Objects", /* tp_doc */ + PyDoc_STR("metatype for the CData Objects"), /* tp_doc */ (traverseproc)CDataType_traverse, /* tp_traverse */ (inquiry)CDataType_clear, /* tp_clear */ 0, /* tp_richcompare */ @@ -978,7 +978,7 @@ static PyTypeObject UnionType_Type = { UnionType_setattro, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HAVE_GC, /* tp_flags */ - "metatype for the CData Objects", /* tp_doc */ + PyDoc_STR("metatype for the CData Objects"), /* tp_doc */ (traverseproc)CDataType_traverse, /* tp_traverse */ (inquiry)CDataType_clear, /* tp_clear */ 0, /* tp_richcompare */ @@ -1236,7 +1236,7 @@ PyTypeObject PyCPointerType_Type = { 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HAVE_GC, /* tp_flags */ - "metatype for the Pointer Objects", /* tp_doc */ + PyDoc_STR("metatype for the Pointer Objects"), /* tp_doc */ (traverseproc)CDataType_traverse, /* tp_traverse */ (inquiry)CDataType_clear, /* tp_clear */ 0, /* tp_richcompare */ @@ -1648,7 +1648,7 @@ PyTypeObject PyCArrayType_Type = { 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */ - "metatype for the Array Objects", /* tp_doc */ + PyDoc_STR("metatype for the Array Objects"), /* tp_doc */ 0, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ @@ -2342,7 +2342,7 @@ PyTypeObject PyCSimpleType_Type = { 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */ - "metatype for the PyCSimpleType Objects", /* tp_doc */ + PyDoc_STR("metatype for the PyCSimpleType Objects"), /* tp_doc */ 0, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ @@ -2624,7 +2624,7 @@ PyTypeObject PyCFuncPtrType_Type = { 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HAVE_GC, /* tp_flags */ - "metatype for C function pointers", /* tp_doc */ + PyDoc_STR("metatype for C function pointers"), /* tp_doc */ (traverseproc)CDataType_traverse, /* tp_traverse */ (inquiry)CDataType_clear, /* tp_clear */ 0, /* tp_richcompare */ @@ -2929,7 +2929,7 @@ PyTypeObject PyCData_Type = { 0, /* tp_setattro */ &PyCData_as_buffer, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */ - "XXX to be provided", /* tp_doc */ + PyDoc_STR("XXX to be provided"), /* tp_doc */ (traverseproc)PyCData_traverse, /* tp_traverse */ (inquiry)PyCData_clear, /* tp_clear */ 0, /* tp_richcompare */ @@ -4327,7 +4327,7 @@ PyTypeObject PyCFuncPtr_Type = { 0, /* tp_setattro */ &PyCData_as_buffer, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */ - "Function Pointer", /* tp_doc */ + PyDoc_STR("Function Pointer"), /* tp_doc */ (traverseproc)PyCFuncPtr_traverse, /* tp_traverse */ (inquiry)PyCFuncPtr_clear, /* tp_clear */ 0, /* tp_richcompare */ @@ -4479,7 +4479,7 @@ static PyTypeObject Struct_Type = { 0, /* tp_setattro */ &PyCData_as_buffer, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */ - "Structure base class", /* tp_doc */ + PyDoc_STR("Structure base class"), /* tp_doc */ (traverseproc)PyCData_traverse, /* tp_traverse */ (inquiry)PyCData_clear, /* tp_clear */ 0, /* tp_richcompare */ @@ -4521,7 +4521,7 @@ static PyTypeObject Union_Type = { 0, /* tp_setattro */ &PyCData_as_buffer, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */ - "Union base class", /* tp_doc */ + PyDoc_STR("Union base class"), /* tp_doc */ (traverseproc)PyCData_traverse, /* tp_traverse */ (inquiry)PyCData_clear, /* tp_clear */ 0, /* tp_richcompare */ @@ -4841,7 +4841,7 @@ PyTypeObject PyCArray_Type = { 0, /* tp_setattro */ &PyCData_as_buffer, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */ - "XXX to be provided", /* tp_doc */ + PyDoc_STR("XXX to be provided"), /* tp_doc */ (traverseproc)PyCData_traverse, /* tp_traverse */ (inquiry)PyCData_clear, /* tp_clear */ 0, /* tp_richcompare */ @@ -5060,7 +5060,7 @@ static PyTypeObject Simple_Type = { 0, /* tp_setattro */ &PyCData_as_buffer, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */ - "XXX to be provided", /* tp_doc */ + PyDoc_STR("XXX to be provided"), /* tp_doc */ (traverseproc)PyCData_traverse, /* tp_traverse */ (inquiry)PyCData_clear, /* tp_clear */ 0, /* tp_richcompare */ @@ -5442,7 +5442,7 @@ PyTypeObject PyCPointer_Type = { 0, /* tp_setattro */ &PyCData_as_buffer, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */ - "XXX to be provided", /* tp_doc */ + PyDoc_STR("XXX to be provided"), /* tp_doc */ (traverseproc)PyCData_traverse, /* tp_traverse */ (inquiry)PyCData_clear, /* tp_clear */ 0, /* tp_richcompare */ @@ -5469,12 +5469,12 @@ PyTypeObject PyCPointer_Type = { * Module initialization. */ -static const char module_docs[] = -"Create and manipulate C compatible data types in Python."; +PyDoc_STRVAR(_ctypes__doc__, +"Create and manipulate C compatible data types in Python."); #ifdef MS_WIN32 -static const char comerror_doc[] = "Raised when a COM method call failed."; +PyDoc_STRVAR(comerror_doc, "Raised when a COM method call failed."); int comerror_init(PyObject *self, PyObject *args, PyObject *kwds) @@ -5663,7 +5663,7 @@ wstring_at(const wchar_t *ptr, int size) static struct PyModuleDef _ctypesmodule = { PyModuleDef_HEAD_INIT, .m_name = "_ctypes", - .m_doc = module_docs, + .m_doc = _ctypes__doc__, .m_size = -1, .m_methods = _ctypes_module_methods, }; diff --git a/Modules/_ctypes/callbacks.c b/Modules/_ctypes/callbacks.c index 5a4d1c543f100..c0ff7891c9a47 100644 --- a/Modules/_ctypes/callbacks.c +++ b/Modules/_ctypes/callbacks.c @@ -65,7 +65,7 @@ PyTypeObject PyCThunk_Type = { 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC, /* tp_flags */ - "CThunkObject", /* tp_doc */ + PyDoc_STR("CThunkObject"), /* tp_doc */ CThunkObject_traverse, /* tp_traverse */ CThunkObject_clear, /* tp_clear */ 0, /* tp_richcompare */ diff --git a/Modules/_ctypes/callproc.c b/Modules/_ctypes/callproc.c index ddf289e3e8f60..48694760dbedf 100644 --- a/Modules/_ctypes/callproc.c +++ b/Modules/_ctypes/callproc.c @@ -1312,11 +1312,11 @@ _parse_voidp(PyObject *obj, void **address) #ifdef MS_WIN32 -static const char format_error_doc[] = +PyDoc_STRVAR(format_error_doc, "FormatError([integer]) -> string\n\ \n\ Convert a win32 error code into a string. If the error code is not\n\ -given, the return value of a call to GetLastError() is used.\n"; +given, the return value of a call to GetLastError() is used.\n"); static PyObject *format_error(PyObject *self, PyObject *args) { PyObject *result; @@ -1336,13 +1336,13 @@ static PyObject *format_error(PyObject *self, PyObject *args) return result; } -static const char load_library_doc[] = +PyDoc_STRVAR(load_library_doc, "LoadLibrary(name, load_flags) -> handle\n\ \n\ Load an executable (usually a DLL), and return a handle to it.\n\ The handle may be used to locate exported functions in this\n\ module. load_flags are as defined for LoadLibraryEx in the\n\ -Windows API.\n"; +Windows API.\n"); static PyObject *load_library(PyObject *self, PyObject *args) { PyObject *nameobj; @@ -1387,10 +1387,10 @@ static PyObject *load_library(PyObject *self, PyObject *args) #endif } -static const char free_library_doc[] = +PyDoc_STRVAR(free_library_doc, "FreeLibrary(handle) -> void\n\ \n\ -Free the handle of an executable previously loaded by LoadLibrary.\n"; +Free the handle of an executable previously loaded by LoadLibrary.\n"); static PyObject *free_library(PyObject *self, PyObject *args) { void *hMod; @@ -1410,8 +1410,8 @@ static PyObject *free_library(PyObject *self, PyObject *args) Py_RETURN_NONE; } -static const char copy_com_pointer_doc[] = -"CopyComPointer(src, dst) -> HRESULT value\n"; +PyDoc_STRVAR(copy_com_pointer_doc, +"CopyComPointer(src, dst) -> HRESULT value\n"); static PyObject * copy_com_pointer(PyObject *self, PyObject *args) @@ -1649,10 +1649,10 @@ call_cdeclfunction(PyObject *self, PyObject *args) /***************************************************************** * functions */ -static const char sizeof_doc[] = +PyDoc_STRVAR(sizeof_doc, "sizeof(C type) -> integer\n" "sizeof(C instance) -> integer\n" -"Return the size in bytes of a C instance"; +"Return the size in bytes of a C instance"); static PyObject * sizeof_func(PyObject *self, PyObject *obj) @@ -1670,10 +1670,10 @@ sizeof_func(PyObject *self, PyObject *obj) return NULL; } -static const char alignment_doc[] = +PyDoc_STRVAR(alignment_doc, "alignment(C type) -> integer\n" "alignment(C instance) -> integer\n" -"Return the alignment requirements of a C instance"; +"Return the alignment requirements of a C instance"); static PyObject * align_func(PyObject *self, PyObject *obj) @@ -1693,10 +1693,10 @@ align_func(PyObject *self, PyObject *obj) return NULL; } -static const char byref_doc[] = +PyDoc_STRVAR(byref_doc, "byref(C instance[, offset=0]) -> byref-object\n" "Return a pointer lookalike to a C instance, only usable\n" -"as function argument"; +"as function argument"); /* * We must return something which can be converted to a parameter, @@ -1737,9 +1737,9 @@ byref(PyObject *self, PyObject *args) return (PyObject *)parg; } -static const char addressof_doc[] = +PyDoc_STRVAR(addressof_doc, "addressof(C instance) -> integer\n" -"Return the address of the C instance internal buffer"; +"Return the address of the C instance internal buffer"); static PyObject * addressof(PyObject *self, PyObject *obj) diff --git a/Modules/_ctypes/cfield.c b/Modules/_ctypes/cfield.c index ec6feca8b0f80..534ec94400128 100644 --- a/Modules/_ctypes/cfield.c +++ b/Modules/_ctypes/cfield.c @@ -318,7 +318,7 @@ PyTypeObject PyCField_Type = { 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC, /* tp_flags */ - "Structure/Union member", /* tp_doc */ + PyDoc_STR("Structure/Union member"), /* tp_doc */ (traverseproc)PyCField_traverse, /* tp_traverse */ (inquiry)PyCField_clear, /* tp_clear */ 0, /* tp_richcompare */ diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c index c5c942820b66f..aaf29ad011544 100644 --- a/Modules/_testcapimodule.c +++ b/Modules/_testcapimodule.c @@ -6287,7 +6287,7 @@ static PyTypeObject PyRecursingInfinitelyError_Type = { 0, /* tp_setattro */ 0, /* tp_as_buffer */ Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */ - "Instantiating this exception starts infinite recursion.", /* tp_doc */ + PyDoc_STR("Instantiating this exception starts infinite recursion."), /* tp_doc */ 0, /* tp_traverse */ 0, /* tp_clear */ 0, /* tp_richcompare */ diff --git a/Objects/genericaliasobject.c b/Objects/genericaliasobject.c index dbe5d89b73962..f52bc974f4d81 100644 --- a/Objects/genericaliasobject.c +++ b/Objects/genericaliasobject.c @@ -349,6 +349,11 @@ _Py_subs_parameters(PyObject *self, PyObject *args, PyObject *parameters, PyObje return newargs; } +PyDoc_STRVAR(genericalias__doc__, +"Represent a PEP 585 generic type\n" +"\n" +"E.g. for t = list[int], t.__origin__ is list and t.__args__ is (int,)."); + static PyObject * ga_getitem(PyObject *self, PyObject *item) { @@ -628,14 +633,11 @@ static PyNumberMethods ga_as_number = { // TODO: // - argument clinic? -// - __doc__? // - cache? PyTypeObject Py_GenericAliasType = { PyVarObject_HEAD_INIT(&PyType_Type, 0) .tp_name = "types.GenericAlias", - .tp_doc = "Represent a PEP 585 generic type\n" - "\n" - "E.g. for t = list[int], t.__origin__ is list and t.__args__ is (int,).", + .tp_doc = genericalias__doc__, .tp_basicsize = sizeof(gaobject), .tp_dealloc = ga_dealloc, .tp_repr = ga_repr, diff --git a/Objects/picklebufobject.c b/Objects/picklebufobject.c index a135e5575e28c..aaa852cfbb05b 100644 --- a/Objects/picklebufobject.c +++ b/Objects/picklebufobject.c @@ -206,7 +206,7 @@ static PyMethodDef picklebuf_methods[] = { PyTypeObject PyPickleBuffer_Type = { PyVarObject_HEAD_INIT(NULL, 0) .tp_name = "pickle.PickleBuffer", - .tp_doc = "Wrapper for potentially out-of-band buffers", + .tp_doc = PyDoc_STR("Wrapper for potentially out-of-band buffers"), .tp_basicsize = sizeof(PyPickleBufferObject), .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC, .tp_new = picklebuf_new, diff --git a/Objects/unionobject.c b/Objects/unionobject.c index 80c70389ab30d..6d8bb02142192 100644 --- a/Objects/unionobject.c +++ b/Objects/unionobject.c @@ -447,9 +447,9 @@ union_getattro(PyObject *self, PyObject *name) PyTypeObject _PyUnion_Type = { PyVarObject_HEAD_INIT(&PyType_Type, 0) .tp_name = "types.UnionType", - .tp_doc = "Represent a PEP 604 union type\n" + .tp_doc = PyDoc_STR("Represent a PEP 604 union type\n" "\n" - "E.g. for int | str", + "E.g. for int | str"), .tp_basicsize = sizeof(unionobject), .tp_dealloc = unionobject_dealloc, .tp_alloc = PyType_GenericAlloc, From webhook-mailer at python.org Tue Apr 19 16:04:03 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Tue, 19 Apr 2022 20:04:03 -0000 Subject: [Python-checkins] build(deps): bump actions/setup-python from 2 to 3 (#31630) Message-ID: https://github.com/python/cpython/commit/74e319239b0a2a5ef8bc27670f4f533ee701d57f commit: 74e319239b0a2a5ef8bc27670f4f533ee701d57f branch: main author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> committer: JelleZijlstra date: 2022-04-19T13:03:58-07:00 summary: build(deps): bump actions/setup-python from 2 to 3 (#31630) Bumps [actions/setup-python](https://github.com/actions/setup-python) from 2 to 3. - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](https://github.com/actions/setup-python/compare/v2...v3) --- updated-dependencies: - dependency-name: actions/setup-python dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jelle Zijlstra files: M .github/workflows/build.yml diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index c34706bb27992..1ddbc30705457 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -61,7 +61,7 @@ jobs: if: needs.check_source.outputs.run_tests == 'true' steps: - uses: actions/checkout at v2 - - uses: actions/setup-python at v2 + - uses: actions/setup-python at v3 - name: Install Dependencies run: sudo ./.github/workflows/posix-deps-apt.sh - name: Add ccache to PATH From webhook-mailer at python.org Tue Apr 19 16:08:12 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Tue, 19 Apr 2022 20:08:12 -0000 Subject: [Python-checkins] bpo-46075: Store localhost cookies in CookieJar (#30108) Message-ID: https://github.com/python/cpython/commit/b6d5e3c3c92374598b327918c8e66e748a1a92a0 commit: b6d5e3c3c92374598b327918c8e66e748a1a92a0 branch: main author: Nick committer: JelleZijlstra date: 2022-04-19T13:08:06-07:00 summary: bpo-46075: Store localhost cookies in CookieJar (#30108) Co-authored-by: Jelle Zijlstra files: A Misc/NEWS.d/next/Library/2021-12-14-21-19-04.bpo-46075.KDtcU-.rst M Lib/http/cookiejar.py M Lib/test/test_http_cookiejar.py diff --git a/Lib/http/cookiejar.py b/Lib/http/cookiejar.py index ee433c0f78f4a..136a1f16ffe63 100644 --- a/Lib/http/cookiejar.py +++ b/Lib/http/cookiejar.py @@ -1043,12 +1043,13 @@ def set_ok_domain(self, cookie, request): else: undotted_domain = domain embedded_dots = (undotted_domain.find(".") >= 0) - if not embedded_dots and domain != ".local": + if not embedded_dots and not erhn.endswith(".local"): _debug(" non-local domain %s contains no embedded dot", domain) return False if cookie.version == 0: - if (not erhn.endswith(domain) and + if (not (erhn.endswith(domain) or + erhn.endswith(f"{undotted_domain}.local")) and (not erhn.startswith(".") and not ("."+erhn).endswith(domain))): _debug(" effective request-host %s (even with added " diff --git a/Lib/test/test_http_cookiejar.py b/Lib/test/test_http_cookiejar.py index 9450104d0b9a7..126ce4fc83f0d 100644 --- a/Lib/test/test_http_cookiejar.py +++ b/Lib/test/test_http_cookiejar.py @@ -920,6 +920,48 @@ def test_two_component_domain_ns(self): ## self.assertEqual(len(c), 2) self.assertEqual(len(c), 4) + def test_localhost_domain(self): + c = CookieJar() + + interact_netscape(c, "http://localhost", "foo=bar; domain=localhost;") + + self.assertEqual(len(c), 1) + + def test_localhost_domain_contents(self): + c = CookieJar() + + interact_netscape(c, "http://localhost", "foo=bar; domain=localhost;") + + self.assertEqual(c._cookies[".localhost"]["/"]["foo"].value, "bar") + + def test_localhost_domain_contents_2(self): + c = CookieJar() + + interact_netscape(c, "http://localhost", "foo=bar;") + + self.assertEqual(c._cookies["localhost.local"]["/"]["foo"].value, "bar") + + def test_evil_nonlocal_domain(self): + c = CookieJar() + + interact_netscape(c, "http://evil.com", "foo=bar; domain=.localhost") + + self.assertEqual(len(c), 0) + + def test_evil_local_domain(self): + c = CookieJar() + + interact_netscape(c, "http://localhost", "foo=bar; domain=.evil.com") + + self.assertEqual(len(c), 0) + + def test_evil_local_domain_2(self): + c = CookieJar() + + interact_netscape(c, "http://localhost", "foo=bar; domain=.someother.local") + + self.assertEqual(len(c), 0) + def test_two_component_domain_rfc2965(self): pol = DefaultCookiePolicy(rfc2965=True) c = CookieJar(pol) diff --git a/Misc/NEWS.d/next/Library/2021-12-14-21-19-04.bpo-46075.KDtcU-.rst b/Misc/NEWS.d/next/Library/2021-12-14-21-19-04.bpo-46075.KDtcU-.rst new file mode 100644 index 0000000000000..e01319300be86 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-12-14-21-19-04.bpo-46075.KDtcU-.rst @@ -0,0 +1 @@ +``CookieJar`` with ``DefaultCookiePolicy`` now can process cookies from localhost with domain=localhost explicitly specified in Set-Cookie header. From webhook-mailer at python.org Tue Apr 19 16:41:48 2022 From: webhook-mailer at python.org (miss-islington) Date: Tue, 19 Apr 2022 20:41:48 -0000 Subject: [Python-checkins] build(deps): bump actions/setup-python from 2 to 3 (GH-31630) Message-ID: https://github.com/python/cpython/commit/68caef8f8e4c127c188a741fbebdf4cf077b0f93 commit: 68caef8f8e4c127c188a741fbebdf4cf077b0f93 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-19T13:41:39-07:00 summary: build(deps): bump actions/setup-python from 2 to 3 (GH-31630) Bumps [actions/setup-python](https://github.com/actions/setup-python) from 2 to 3. - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](https://github.com/actions/setup-python/compare/v2...v3) --- updated-dependencies: - dependency-name: actions/setup-python dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jelle Zijlstra (cherry picked from commit 74e319239b0a2a5ef8bc27670f4f533ee701d57f) Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> files: M .github/workflows/build.yml diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 8bd506c9f3135..43b4f76937657 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -82,7 +82,7 @@ jobs: if: needs.check_source.outputs.run_tests == 'true' steps: - uses: actions/checkout at v2 - - uses: actions/setup-python at v2 + - uses: actions/setup-python at v3 - name: Install Dependencies run: sudo ./.github/workflows/posix-deps-apt.sh - name: Add ccache to PATH From webhook-mailer at python.org Tue Apr 19 16:41:55 2022 From: webhook-mailer at python.org (miss-islington) Date: Tue, 19 Apr 2022 20:41:55 -0000 Subject: [Python-checkins] build(deps): bump actions/setup-python from 2 to 3 (GH-31630) Message-ID: https://github.com/python/cpython/commit/6cf86fc9c1b6096ae9a72b6f3731f8af1563fe07 commit: 6cf86fc9c1b6096ae9a72b6f3731f8af1563fe07 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-19T13:41:51-07:00 summary: build(deps): bump actions/setup-python from 2 to 3 (GH-31630) Bumps [actions/setup-python](https://github.com/actions/setup-python) from 2 to 3. - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](https://github.com/actions/setup-python/compare/v2...v3) --- updated-dependencies: - dependency-name: actions/setup-python dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jelle Zijlstra (cherry picked from commit 74e319239b0a2a5ef8bc27670f4f533ee701d57f) Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> files: M .github/workflows/build.yml diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 596f4d88e26e9..61aefdcc2367b 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -80,7 +80,7 @@ jobs: if: needs.check_source.outputs.run_tests == 'true' steps: - uses: actions/checkout at v2 - - uses: actions/setup-python at v2 + - uses: actions/setup-python at v3 - name: Install Dependencies run: sudo ./.github/workflows/posix-deps-apt.sh - name: Add ccache to PATH From webhook-mailer at python.org Tue Apr 19 21:20:11 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Wed, 20 Apr 2022 01:20:11 -0000 Subject: [Python-checkins] gh-87497: Document that urllib.request sends headers in camel case (GH-24661) (#91517) Message-ID: https://github.com/python/cpython/commit/e08d32381d749bc4f3e4302e067076fa8c4e0661 commit: e08d32381d749bc4f3e4302e067076fa8c4e0661 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: JelleZijlstra date: 2022-04-19T18:20:03-07:00 summary: gh-87497: Document that urllib.request sends headers in camel case (GH-24661) (#91517) Co-authored-by: Jelle Zijlstra (cherry picked from commit 325d6f50357474c7d9fd2475be0e2481f7ae0476) Co-authored-by: Alix Lourme files: M Doc/library/urllib.request.rst M Lib/test/test_urllib2_localnet.py diff --git a/Doc/library/urllib.request.rst b/Doc/library/urllib.request.rst index 659a3632ac9be..9573683dd0594 100644 --- a/Doc/library/urllib.request.rst +++ b/Doc/library/urllib.request.rst @@ -218,6 +218,7 @@ The following classes are provided: (X11; U; Linux i686) Gecko/20071127 Firefox/2.0.0.11"``, while :mod:`urllib`'s default user agent string is ``"Python-urllib/2.6"`` (on Python 2.6). + All header keys are sent in camel case. An appropriate ``Content-Type`` header should be included if the *data* argument is present. If this header has not been provided and *data* diff --git a/Lib/test/test_urllib2_localnet.py b/Lib/test/test_urllib2_localnet.py index 1b2baf2f366b5..fe2ff81a24e69 100644 --- a/Lib/test/test_urllib2_localnet.py +++ b/Lib/test/test_urllib2_localnet.py @@ -614,6 +614,15 @@ def test_sending_headers(self): pass self.assertEqual(handler.headers_received["Range"], "bytes=20-39") + def test_sending_headers_camel(self): + handler = self.start_server() + req = urllib.request.Request("http://localhost:%s/" % handler.port, + headers={"X-SoMe-hEader": "foobar"}) + with urllib.request.urlopen(req): + pass + self.assertIn("X-Some-Header", handler.headers_received.keys()) + self.assertNotIn("X-SoMe-hEader", handler.headers_received.keys()) + def test_basic(self): handler = self.start_server() with urllib.request.urlopen("http://localhost:%s" % handler.port) as open_url: From webhook-mailer at python.org Tue Apr 19 21:20:29 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Wed, 20 Apr 2022 01:20:29 -0000 Subject: [Python-checkins] Fix awkward sentence in signal docs (#91508) Message-ID: https://github.com/python/cpython/commit/326ae71f1d93c12100150baa1173ea7ce7a96ea0 commit: 326ae71f1d93c12100150baa1173ea7ce7a96ea0 branch: main author: AJ Jordan committer: JelleZijlstra date: 2022-04-19T18:20:25-07:00 summary: Fix awkward sentence in signal docs (#91508) Co-authored-by: Jelle Zijlstra files: M Doc/library/signal.rst diff --git a/Doc/library/signal.rst b/Doc/library/signal.rst index c276b52ca4d20..fdc9846f6642a 100644 --- a/Doc/library/signal.rst +++ b/Doc/library/signal.rst @@ -711,10 +711,11 @@ case, wrap your entry point to catch this exception as follows:: if __name__ == '__main__': main() -Do not set :const:`SIGPIPE`'s disposition to :const:`SIG_DFL` -in order to avoid :exc:`BrokenPipeError`. Doing that would cause -your program to exit unexpectedly also whenever any socket connection -is interrupted while your program is still writing to it. +Do not set :const:`SIGPIPE`'s disposition to :const:`SIG_DFL` in +order to avoid :exc:`BrokenPipeError`. Doing that would cause +your program to exit unexpectedly whenever any socket +connection is interrupted while your program is still writing to +it. .. _handlers-and-exceptions: From webhook-mailer at python.org Tue Apr 19 21:38:27 2022 From: webhook-mailer at python.org (miss-islington) Date: Wed, 20 Apr 2022 01:38:27 -0000 Subject: [Python-checkins] Fix awkward sentence in signal docs (GH-91508) Message-ID: https://github.com/python/cpython/commit/f92aa4fde1728ed47dd6dc64206dcdcabd81d1fc commit: f92aa4fde1728ed47dd6dc64206dcdcabd81d1fc branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-19T18:38:15-07:00 summary: Fix awkward sentence in signal docs (GH-91508) Co-authored-by: Jelle Zijlstra (cherry picked from commit 326ae71f1d93c12100150baa1173ea7ce7a96ea0) Co-authored-by: AJ Jordan files: M Doc/library/signal.rst diff --git a/Doc/library/signal.rst b/Doc/library/signal.rst index 11bdfc8e82831..0aa5996b89423 100644 --- a/Doc/library/signal.rst +++ b/Doc/library/signal.rst @@ -676,10 +676,11 @@ case, wrap your entry point to catch this exception as follows:: if __name__ == '__main__': main() -Do not set :const:`SIGPIPE`'s disposition to :const:`SIG_DFL` -in order to avoid :exc:`BrokenPipeError`. Doing that would cause -your program to exit unexpectedly also whenever any socket connection -is interrupted while your program is still writing to it. +Do not set :const:`SIGPIPE`'s disposition to :const:`SIG_DFL` in +order to avoid :exc:`BrokenPipeError`. Doing that would cause +your program to exit unexpectedly whenever any socket +connection is interrupted while your program is still writing to +it. .. _handlers-and-exceptions: From webhook-mailer at python.org Tue Apr 19 21:40:38 2022 From: webhook-mailer at python.org (miss-islington) Date: Wed, 20 Apr 2022 01:40:38 -0000 Subject: [Python-checkins] Fix awkward sentence in signal docs (GH-91508) Message-ID: https://github.com/python/cpython/commit/3fa9191cfc3e778192862ab6bea341b398d6a83f commit: 3fa9191cfc3e778192862ab6bea341b398d6a83f branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-19T18:40:28-07:00 summary: Fix awkward sentence in signal docs (GH-91508) Co-authored-by: Jelle Zijlstra (cherry picked from commit 326ae71f1d93c12100150baa1173ea7ce7a96ea0) Co-authored-by: AJ Jordan files: M Doc/library/signal.rst diff --git a/Doc/library/signal.rst b/Doc/library/signal.rst index faea9d42a3fbd..1a419f42d04fb 100644 --- a/Doc/library/signal.rst +++ b/Doc/library/signal.rst @@ -676,10 +676,11 @@ case, wrap your entry point to catch this exception as follows:: if __name__ == '__main__': main() -Do not set :const:`SIGPIPE`'s disposition to :const:`SIG_DFL` -in order to avoid :exc:`BrokenPipeError`. Doing that would cause -your program to exit unexpectedly also whenever any socket connection -is interrupted while your program is still writing to it. +Do not set :const:`SIGPIPE`'s disposition to :const:`SIG_DFL` in +order to avoid :exc:`BrokenPipeError`. Doing that would cause +your program to exit unexpectedly whenever any socket +connection is interrupted while your program is still writing to +it. .. _handlers-and-exceptions: From webhook-mailer at python.org Tue Apr 19 23:50:13 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Wed, 20 Apr 2022 03:50:13 -0000 Subject: [Python-checkins] build(deps): bump actions/checkout from 2 to 3 (#32226) Message-ID: https://github.com/python/cpython/commit/1ba63e3a9bb176e5ad8e8dd744b9d2b9d588e275 commit: 1ba63e3a9bb176e5ad8e8dd744b9d2b9d588e275 branch: main author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> committer: JelleZijlstra date: 2022-04-19T20:50:07-07:00 summary: build(deps): bump actions/checkout from 2 to 3 (#32226) Bumps [actions/checkout](https://github.com/actions/checkout) from 2 to 3. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](https://github.com/actions/checkout/compare/v2...v3) --- updated-dependencies: - dependency-name: actions/checkout dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> files: M .github/workflows/build.yml M .github/workflows/build_msi.yml M .github/workflows/doc.yml diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 1ddbc30705457..785a7e9b683b3 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -28,7 +28,7 @@ jobs: run_tests: ${{ steps.check.outputs.run_tests }} run_ssl_tests: ${{ steps.check.outputs.run_ssl_tests }} steps: - - uses: actions/checkout at v2 + - uses: actions/checkout at v3 - name: Check for source changes id: check run: | @@ -60,7 +60,7 @@ jobs: needs: check_source if: needs.check_source.outputs.run_tests == 'true' steps: - - uses: actions/checkout at v2 + - uses: actions/checkout at v3 - uses: actions/setup-python at v3 - name: Install Dependencies run: sudo ./.github/workflows/posix-deps-apt.sh @@ -114,7 +114,7 @@ jobs: env: IncludeUwp: 'true' steps: - - uses: actions/checkout at v2 + - uses: actions/checkout at v3 - name: Build CPython run: .\PCbuild\build.bat -e -d -p Win32 timeout-minutes: 30 @@ -131,7 +131,7 @@ jobs: env: IncludeUwp: 'true' steps: - - uses: actions/checkout at v2 + - uses: actions/checkout at v3 - name: Register MSVC problem matcher run: echo "::add-matcher::.github/problem-matchers/msvc.json" - name: Build CPython @@ -150,7 +150,7 @@ jobs: env: PYTHONSTRICTEXTENSIONBUILD: 1 steps: - - uses: actions/checkout at v2 + - uses: actions/checkout at v3 - name: Prepare homebrew environment variables run: | echo "LDFLAGS=-L$(brew --prefix tcl-tk)/lib" >> $GITHUB_ENV @@ -173,7 +173,7 @@ jobs: OPENSSL_VER: 1.1.1n PYTHONSTRICTEXTENSIONBUILD: 1 steps: - - uses: actions/checkout at v2 + - uses: actions/checkout at v3 - name: Register gcc problem matcher run: echo "::add-matcher::.github/problem-matchers/gcc.json" - name: Install Dependencies @@ -236,7 +236,7 @@ jobs: OPENSSL_DIR: ${{ github.workspace }}/multissl/openssl/${{ matrix.openssl_ver }} LD_LIBRARY_PATH: ${{ github.workspace }}/multissl/openssl/${{ matrix.openssl_ver }}/lib steps: - - uses: actions/checkout at v2 + - uses: actions/checkout at v3 - name: Register gcc problem matcher run: echo "::add-matcher::.github/problem-matchers/gcc.json" - name: Install Dependencies @@ -280,7 +280,7 @@ jobs: PYTHONSTRICTEXTENSIONBUILD: 1 ASAN_OPTIONS: detect_leaks=0:allocator_may_return_null=1:handle_segv=0 steps: - - uses: actions/checkout at v2 + - uses: actions/checkout at v3 - name: Register gcc problem matcher run: echo "::add-matcher::.github/problem-matchers/gcc.json" - name: Install Dependencies diff --git a/.github/workflows/build_msi.yml b/.github/workflows/build_msi.yml index 6e989b2ab9886..cba1e51ef27d3 100644 --- a/.github/workflows/build_msi.yml +++ b/.github/workflows/build_msi.yml @@ -26,7 +26,7 @@ jobs: name: 'Windows (x86) Installer' runs-on: windows-latest steps: - - uses: actions/checkout at v2 + - uses: actions/checkout at v3 - name: Build CPython installer run: .\Tools\msi\build.bat -x86 @@ -34,7 +34,7 @@ jobs: name: 'Windows (x64) Installer' runs-on: windows-latest steps: - - uses: actions/checkout at v2 + - uses: actions/checkout at v3 - name: Build CPython installer run: .\Tools\msi\build.bat -x64 @@ -42,6 +42,6 @@ jobs: name: 'Windows (ARM64) Installer' runs-on: windows-latest steps: - - uses: actions/checkout at v2 + - uses: actions/checkout at v3 - name: Build CPython installer run: .\Tools\msi\build.bat -arm64 diff --git a/.github/workflows/doc.yml b/.github/workflows/doc.yml index dcc5181a99356..f96a150970ec7 100644 --- a/.github/workflows/doc.yml +++ b/.github/workflows/doc.yml @@ -27,7 +27,7 @@ jobs: name: 'Docs' runs-on: ubuntu-latest steps: - - uses: actions/checkout at v2 + - uses: actions/checkout at v3 - name: Register Sphinx problem matcher run: echo "::add-matcher::.github/problem-matchers/sphinx.json" - name: 'Install Dependencies' From webhook-mailer at python.org Tue Apr 19 23:59:59 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Wed, 20 Apr 2022 03:59:59 -0000 Subject: [Python-checkins] Add link to sys.path in os lib (#91679) Message-ID: https://github.com/python/cpython/commit/692aea6f3823df48b7fc267ba0aa1ccc45ac606d commit: 692aea6f3823df48b7fc267ba0aa1ccc45ac606d branch: main author: slateny <46876382+slateny at users.noreply.github.com> committer: JelleZijlstra date: 2022-04-19T20:59:48-07:00 summary: Add link to sys.path in os lib (#91679) files: M Doc/library/os.rst diff --git a/Doc/library/os.rst b/Doc/library/os.rst index 254d6e192cdf0..c22bf56a9f2cd 100644 --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -3556,8 +3556,8 @@ to be ignored. Add a path to the DLL search path. This search path is used when resolving dependencies for imported - extension modules (the module itself is resolved through sys.path), - and also by :mod:`ctypes`. + extension modules (the module itself is resolved through + :data:`sys.path`), and also by :mod:`ctypes`. Remove the directory by calling **close()** on the returned object or using it in a :keyword:`with` statement. From webhook-mailer at python.org Wed Apr 20 00:17:41 2022 From: webhook-mailer at python.org (miss-islington) Date: Wed, 20 Apr 2022 04:17:41 -0000 Subject: [Python-checkins] Add link to sys.path in os lib (GH-91679) Message-ID: https://github.com/python/cpython/commit/8299e24d4caa392664648fc5c65f4e56a40d6d46 commit: 8299e24d4caa392664648fc5c65f4e56a40d6d46 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-19T21:17:34-07:00 summary: Add link to sys.path in os lib (GH-91679) (cherry picked from commit 692aea6f3823df48b7fc267ba0aa1ccc45ac606d) Co-authored-by: slateny <46876382+slateny at users.noreply.github.com> files: M Doc/library/os.rst diff --git a/Doc/library/os.rst b/Doc/library/os.rst index 203995c80fd32..dbd3c968dd35a 100644 --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -3546,8 +3546,8 @@ to be ignored. Add a path to the DLL search path. This search path is used when resolving dependencies for imported - extension modules (the module itself is resolved through sys.path), - and also by :mod:`ctypes`. + extension modules (the module itself is resolved through + :data:`sys.path`), and also by :mod:`ctypes`. Remove the directory by calling **close()** on the returned object or using it in a :keyword:`with` statement. From webhook-mailer at python.org Wed Apr 20 00:17:59 2022 From: webhook-mailer at python.org (miss-islington) Date: Wed, 20 Apr 2022 04:17:59 -0000 Subject: [Python-checkins] Add link to sys.path in os lib (GH-91679) Message-ID: https://github.com/python/cpython/commit/6fd77aab4cb702be3bdd60c87f7bf0ee034f1062 commit: 6fd77aab4cb702be3bdd60c87f7bf0ee034f1062 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-19T21:17:55-07:00 summary: Add link to sys.path in os lib (GH-91679) (cherry picked from commit 692aea6f3823df48b7fc267ba0aa1ccc45ac606d) Co-authored-by: slateny <46876382+slateny at users.noreply.github.com> files: M Doc/library/os.rst diff --git a/Doc/library/os.rst b/Doc/library/os.rst index 9dc49a2034144..8e553d404100f 100644 --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -3319,8 +3319,8 @@ to be ignored. Add a path to the DLL search path. This search path is used when resolving dependencies for imported - extension modules (the module itself is resolved through sys.path), - and also by :mod:`ctypes`. + extension modules (the module itself is resolved through + :data:`sys.path`), and also by :mod:`ctypes`. Remove the directory by calling **close()** on the returned object or using it in a :keyword:`with` statement. From webhook-mailer at python.org Wed Apr 20 01:06:41 2022 From: webhook-mailer at python.org (methane) Date: Wed, 20 Apr 2022 05:06:41 -0000 Subject: [Python-checkins] gh-91020: Add `PyBytes_Type.tp_alloc` for subclass (GH-91686) Message-ID: https://github.com/python/cpython/commit/4d2403fd5017324515016a8fad8545935a5c81ef commit: 4d2403fd5017324515016a8fad8545935a5c81ef branch: main author: Inada Naoki committer: methane date: 2022-04-20T14:06:29+09:00 summary: gh-91020: Add `PyBytes_Type.tp_alloc` for subclass (GH-91686) files: A Misc/NEWS.d/next/C API/2022-04-19-17-05-39.gh-issue-91020.BVJ8F3.rst M Objects/bytesobject.c diff --git a/Misc/NEWS.d/next/C API/2022-04-19-17-05-39.gh-issue-91020.BVJ8F3.rst b/Misc/NEWS.d/next/C API/2022-04-19-17-05-39.gh-issue-91020.BVJ8F3.rst new file mode 100644 index 0000000000000..1572c961de34c --- /dev/null +++ b/Misc/NEWS.d/next/C API/2022-04-19-17-05-39.gh-issue-91020.BVJ8F3.rst @@ -0,0 +1,2 @@ +Add ``PyBytes_Type.tp_alloc`` to initialize ``PyBytesObject.ob_shash`` for +bytes subclasses. diff --git a/Objects/bytesobject.c b/Objects/bytesobject.c index d0124c050d1e1..510a836e6b152 100644 --- a/Objects/bytesobject.c +++ b/Objects/bytesobject.c @@ -2861,6 +2861,25 @@ PyBytes_FromObject(PyObject *x) return NULL; } +/* This allocator is needed for subclasses don't want to use __new__. + * See https://github.com/python/cpython/issues/91020#issuecomment-1096793239 + * + * This allocator will be removed when ob_shash is removed. + */ +static PyObject * +bytes_alloc(PyTypeObject *self, Py_ssize_t nitems) +{ + PyBytesObject *obj = (PyBytesObject*)PyType_GenericAlloc(self, nitems); + if (obj == NULL) { + return NULL; + } +_Py_COMP_DIAG_PUSH +_Py_COMP_DIAG_IGNORE_DEPR_DECLS + obj->ob_shash = -1; +_Py_COMP_DIAG_POP + return (PyObject*)obj; +} + static PyObject * bytes_subtype_new(PyTypeObject *type, PyObject *tmp) { @@ -2937,7 +2956,7 @@ PyTypeObject PyBytes_Type = { 0, /* tp_descr_set */ 0, /* tp_dictoffset */ 0, /* tp_init */ - 0, /* tp_alloc */ + bytes_alloc, /* tp_alloc */ bytes_new, /* tp_new */ PyObject_Del, /* tp_free */ }; From webhook-mailer at python.org Wed Apr 20 01:57:48 2022 From: webhook-mailer at python.org (Fidget-Spinner) Date: Wed, 20 Apr 2022 05:57:48 -0000 Subject: [Python-checkins] Fix whitespace/indentation issues in test_sys (GH-32369) (GH-32372) Message-ID: https://github.com/python/cpython/commit/942ea19cf956a2352bd82cc43fb4c1723ba72867 commit: 942ea19cf956a2352bd82cc43fb4c1723ba72867 branch: 3.10 author: Ken Jin committer: Fidget-Spinner date: 2022-04-20T13:57:30+08:00 summary: Fix whitespace/indentation issues in test_sys (GH-32369) (GH-32372) files: M Lib/test/test_sys.py diff --git a/Lib/test/test_sys.py b/Lib/test/test_sys.py index 35da72c432b0a..a0458092553b4 100644 --- a/Lib/test/test_sys.py +++ b/Lib/test/test_sys.py @@ -1078,8 +1078,8 @@ class X(Exception): with test.support.captured_stderr() as stderr, \ test.support.swap_attr(sys, 'unraisablehook', sys.__unraisablehook__): - expected = self.write_unraisable_exc( - A.B.X(), "msg", "obj"); + expected = self.write_unraisable_exc( + A.B.X(), "msg", "obj"); report = stderr.getvalue() testName = 'test_original_unraisablehook_exception_qualname' self.assertIn(f"{testName}..A.B.X", report) From webhook-mailer at python.org Wed Apr 20 02:21:58 2022 From: webhook-mailer at python.org (rhettinger) Date: Wed, 20 Apr 2022 06:21:58 -0000 Subject: [Python-checkins] Clean-up the argparse docs quick links table (GH-91726) Message-ID: https://github.com/python/cpython/commit/26f2e688b84b4c41c3b94b81b747f5776d19a52e commit: 26f2e688b84b4c41c3b94b81b747f5776d19a52e branch: main author: Raymond Hettinger committer: rhettinger date: 2022-04-20T01:21:54-05:00 summary: Clean-up the argparse docs quick links table (GH-91726) files: M Doc/library/argparse.rst diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst index 425409db3cb75..7c206370f5432 100644 --- a/Doc/library/argparse.rst +++ b/Doc/library/argparse.rst @@ -26,73 +26,51 @@ module also automatically generates help and usage messages and issues errors when users give the program invalid arguments. -Summary -------- - Core Functionality -^^^^^^^^^^^^^^^^^^ +------------------ The :mod:`argparse` module's support for command-line interfaces is built -from the following: - -The :class:`argparse.ArgumentParser` creates a new :class:`ArgumentParser` -object. Commonly used arguments include prog_, description_, and -formatter_class_. For example, the user can create an instance of -:class:`ArgumentParser` through the following:: - - >>> parser = argparse.ArgumentParser(prog='PROG', description='DESC', - ... formatter_class=argparse.RawDescriptionHelpFormatter) - -The :func:`ArgumentParser.add_argument` is a function that is used -to define how a single command-line argument should be parsed. Commonly used -arguments include `name or flags`_, action_, default_, type_, required_, -and help_. An example of the function :func:`ArgumentParser.add_argument` -is as follows:: - - >>> parser.add_argument('-v', '--verbose', action='store_true', - ... help='Show various debugging information') +around an instance of :class:`argparse.ArgumentParser`. It is a container for +argument specifications and has options that apply the parser as whole:: + parser = argparse.ArgumentParser( + prog = 'ProgramName', + description = 'What the program does', + epilog = 'Text at the bottom of help') -Basic Usage of :func:`add_argument` -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +The :func:`ArgumentParser.add_argument` function attaches individual argument +specifications to the parser. It supports positional arguments, options that +accept values, and on/off flags:: + parser.add_argument('filename') # positional argument + parser.add_argument('-c', '--count') # option that takes value + parser.add_argument('-v', '--verbose', + action='store_true') # on/off flag -**Name or Flags Type** +The :func:`ArgumentParser.parse_args` function runs the parser and puts +the extracted data in a :class:`argparse.Namespace` object:: -====================== =========================== -Type Example -====================== =========================== -Positional ``'foo'`` -Optional ``'-v'``, ``'--verbose'`` -====================== =========================== + args = parser.parse_args() + print(args.filename, args.count, args.verbose) -**Basic Arguments:** +Quick Links for add_argument() +------------------------------ -====================== =========================================================== ========================================================================================================================= -Name Description Keywords -====================== =========================================================== ========================================================================================================================= -action_ Specifies how an argument should be handled ``'store'``, ``'store_const'``, ``'store_true'``, ``'append'``, ``'append_const'``, ``'count'``, ``'help'``, ``'version'`` +====================== =========================================================== ========================================================================================================================== +Name Description Values +====================== =========================================================== ========================================================================================================================== +action_ Specify how an argument should be handled ``'store'``, ``'store_const'``, ``'store_true'``, ``'append'``, ``'append_const'``, ``'count'``, ``'help'``, ``'version'`` +choices_ Limit values to specific set of choices ``['foo', 'bar']``, ``range(1, 10)``, or an object that supports ``in`` operator +const_ Store a constant value default_ Default value used when an argument is not provided -type_ Automatically converts an argument to the given type :class:`int`, :class:`float`, :class:`bool`, ``argparse.FileType('w')``, ``callable function`` -help_ Help message of an argument -====================== =========================================================== ========================================================================================================================= - - - -**Advanced Arguments:** - -====================== =========================================================== ======================================================================================================================= -Name Description Keywords -====================== =========================================================== ======================================================================================================================= -nargs_ Associates a single action with the number of arguments ``N`` (:class:`int`), ``'?'``, ``'*'``, ``'+'``, ``argparse.REMAINDER`` -const_ Stores constant values of names or flags -choices_ A container that lists the possible values ``['foo', 'bar']``, ``range(1, 10)``, Any object that supports ``in`` operator -required_ Indicates if an optional argument is required or not ``True``, ``False`` -metavar_ An alternative display name for the argument -dest_ Specifies name of attribute to be used in ``parse_args()`` -====================== =========================================================== ======================================================================================================================= - +dest_ Specify the attribute name in result namespace +help_ Help message for an argument +metavar_ Alternate display name for the argument as shown in help +nargs_ Number of times the argument can be used ``N`` (:class:`int`), ``'?'``, ``'*'``, ``'+'``, ``argparse.REMAINDER`` +required_ Indicate whether an optional argument is required or not ``True``, ``False`` +type_ Automatically convert an argument to the given type :class:`int`, :class:`float`, ``argparse.FileType('w')``, or any callable function +====================== =========================================================== ========================================================================================================================== Example From webhook-mailer at python.org Wed Apr 20 04:24:57 2022 From: webhook-mailer at python.org (methane) Date: Wed, 20 Apr 2022 08:24:57 -0000 Subject: [Python-checkins] bpo-30718: Add information about text buffering (GH-32351) Message-ID: https://github.com/python/cpython/commit/5101d97d0b13425ccc5ed37abfabb07701db81fe commit: 5101d97d0b13425ccc5ed37abfabb07701db81fe branch: main author: slateny <46876382+slateny at users.noreply.github.com> committer: methane date: 2022-04-20T17:24:53+09:00 summary: bpo-30718: Add information about text buffering (GH-32351) files: M Doc/library/functions.rst diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst index eaa4d482ce3fc..e6fd0bb5eeef9 100644 --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -1165,7 +1165,11 @@ are always available. They are listed here in alphabetical order. *buffering* is an optional integer used to set the buffering policy. Pass 0 to switch buffering off (only allowed in binary mode), 1 to select line buffering (only usable in text mode), and an integer > 1 to indicate the size - in bytes of a fixed-size chunk buffer. When no *buffering* argument is + in bytes of a fixed-size chunk buffer. Note that specifying a buffer size this + way applies for binary buffered I/O, but ``TextIOWrapper`` (i.e., files opened + with ``mode='r+'``) would have another buffering. To disable buffering in + ``TextIOWrapper``, consider using the ``write_through`` flag for + :func:`io.TextIOWrapper.reconfigure`. When no *buffering* argument is given, the default buffering policy works as follows: * Binary files are buffered in fixed-size chunks; the size of the buffer is From webhook-mailer at python.org Wed Apr 20 05:00:41 2022 From: webhook-mailer at python.org (miss-islington) Date: Wed, 20 Apr 2022 09:00:41 -0000 Subject: [Python-checkins] bpo-30718: Add information about text buffering (GH-32351) Message-ID: https://github.com/python/cpython/commit/33c6feb811326e6f6d3487ed88df8dc66c552edc commit: 33c6feb811326e6f6d3487ed88df8dc66c552edc branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-20T02:00:32-07:00 summary: bpo-30718: Add information about text buffering (GH-32351) (cherry picked from commit 5101d97d0b13425ccc5ed37abfabb07701db81fe) Co-authored-by: slateny <46876382+slateny at users.noreply.github.com> files: M Doc/library/functions.rst diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst index 9a9c707dc177a..937e00bf994e6 100644 --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -1113,7 +1113,11 @@ are always available. They are listed here in alphabetical order. *buffering* is an optional integer used to set the buffering policy. Pass 0 to switch buffering off (only allowed in binary mode), 1 to select line buffering (only usable in text mode), and an integer > 1 to indicate the size - in bytes of a fixed-size chunk buffer. When no *buffering* argument is + in bytes of a fixed-size chunk buffer. Note that specifying a buffer size this + way applies for binary buffered I/O, but ``TextIOWrapper`` (i.e., files opened + with ``mode='r+'``) would have another buffering. To disable buffering in + ``TextIOWrapper``, consider using the ``write_through`` flag for + :func:`io.TextIOWrapper.reconfigure`. When no *buffering* argument is given, the default buffering policy works as follows: * Binary files are buffered in fixed-size chunks; the size of the buffer is From webhook-mailer at python.org Wed Apr 20 05:00:47 2022 From: webhook-mailer at python.org (miss-islington) Date: Wed, 20 Apr 2022 09:00:47 -0000 Subject: [Python-checkins] bpo-30718: Add information about text buffering (GH-32351) Message-ID: https://github.com/python/cpython/commit/88bbc5dd7af2efa74989f23689438d5113521798 commit: 88bbc5dd7af2efa74989f23689438d5113521798 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-20T02:00:43-07:00 summary: bpo-30718: Add information about text buffering (GH-32351) (cherry picked from commit 5101d97d0b13425ccc5ed37abfabb07701db81fe) Co-authored-by: slateny <46876382+slateny at users.noreply.github.com> files: M Doc/library/functions.rst diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst index 95231363a5dc2..42fbf4e6f969c 100644 --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -1168,7 +1168,11 @@ are always available. They are listed here in alphabetical order. *buffering* is an optional integer used to set the buffering policy. Pass 0 to switch buffering off (only allowed in binary mode), 1 to select line buffering (only usable in text mode), and an integer > 1 to indicate the size - in bytes of a fixed-size chunk buffer. When no *buffering* argument is + in bytes of a fixed-size chunk buffer. Note that specifying a buffer size this + way applies for binary buffered I/O, but ``TextIOWrapper`` (i.e., files opened + with ``mode='r+'``) would have another buffering. To disable buffering in + ``TextIOWrapper``, consider using the ``write_through`` flag for + :func:`io.TextIOWrapper.reconfigure`. When no *buffering* argument is given, the default buffering policy works as follows: * Binary files are buffered in fixed-size chunks; the size of the buffer is From webhook-mailer at python.org Wed Apr 20 06:46:13 2022 From: webhook-mailer at python.org (serhiy-storchaka) Date: Wed, 20 Apr 2022 10:46:13 -0000 Subject: [Python-checkins] gh-91734: Fix ossaudio support on Solaris (GH-91735) Message-ID: https://github.com/python/cpython/commit/4420faf273e9e2d03226a9375e1e04a336230c84 commit: 4420faf273e9e2d03226a9375e1e04a336230c84 branch: main author: Jakub Kul?k committer: serhiy-storchaka date: 2022-04-20T13:46:08+03:00 summary: gh-91734: Fix ossaudio support on Solaris (GH-91735) files: A Misc/NEWS.d/next/Library/2022-04-20-09-49-33.gh-issue-91734.4Dj4Gy.rst M Modules/ossaudiodev.c diff --git a/Misc/NEWS.d/next/Library/2022-04-20-09-49-33.gh-issue-91734.4Dj4Gy.rst b/Misc/NEWS.d/next/Library/2022-04-20-09-49-33.gh-issue-91734.4Dj4Gy.rst new file mode 100644 index 0000000000000..47d9e0dea458a --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-20-09-49-33.gh-issue-91734.4Dj4Gy.rst @@ -0,0 +1 @@ +Fix OSS audio support on Solaris. diff --git a/Modules/ossaudiodev.c b/Modules/ossaudiodev.c index 2f507de46f5dc..0568b6dc44103 100644 --- a/Modules/ossaudiodev.c +++ b/Modules/ossaudiodev.c @@ -1277,8 +1277,12 @@ PyInit_ossaudiodev(void) _EXPORT_INT(m, SNDCTL_DSP_GETSPDIF); #endif _EXPORT_INT(m, SNDCTL_DSP_GETTRIGGER); +#ifdef SNDCTL_DSP_MAPINBUF _EXPORT_INT(m, SNDCTL_DSP_MAPINBUF); +#endif +#ifdef SNDCTL_DSP_MAPOUTBUF _EXPORT_INT(m, SNDCTL_DSP_MAPOUTBUF); +#endif _EXPORT_INT(m, SNDCTL_DSP_NONBLOCK); _EXPORT_INT(m, SNDCTL_DSP_POST); #ifdef SNDCTL_DSP_PROFILE From webhook-mailer at python.org Wed Apr 20 07:06:46 2022 From: webhook-mailer at python.org (miss-islington) Date: Wed, 20 Apr 2022 11:06:46 -0000 Subject: [Python-checkins] gh-91734: Fix ossaudio support on Solaris (GH-91735) Message-ID: https://github.com/python/cpython/commit/923ef87c77d4e0ea54fa03eae6928b42b9094e8d commit: 923ef87c77d4e0ea54fa03eae6928b42b9094e8d branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-20T04:06:29-07:00 summary: gh-91734: Fix ossaudio support on Solaris (GH-91735) (cherry picked from commit 4420faf273e9e2d03226a9375e1e04a336230c84) Co-authored-by: Jakub Kul?k files: A Misc/NEWS.d/next/Library/2022-04-20-09-49-33.gh-issue-91734.4Dj4Gy.rst M Modules/ossaudiodev.c diff --git a/Misc/NEWS.d/next/Library/2022-04-20-09-49-33.gh-issue-91734.4Dj4Gy.rst b/Misc/NEWS.d/next/Library/2022-04-20-09-49-33.gh-issue-91734.4Dj4Gy.rst new file mode 100644 index 0000000000000..47d9e0dea458a --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-20-09-49-33.gh-issue-91734.4Dj4Gy.rst @@ -0,0 +1 @@ +Fix OSS audio support on Solaris. diff --git a/Modules/ossaudiodev.c b/Modules/ossaudiodev.c index 4f2d9cb8b7c9c..b22bd42dcaa6f 100644 --- a/Modules/ossaudiodev.c +++ b/Modules/ossaudiodev.c @@ -1243,8 +1243,12 @@ PyInit_ossaudiodev(void) _EXPORT_INT(m, SNDCTL_DSP_GETSPDIF); #endif _EXPORT_INT(m, SNDCTL_DSP_GETTRIGGER); +#ifdef SNDCTL_DSP_MAPINBUF _EXPORT_INT(m, SNDCTL_DSP_MAPINBUF); +#endif +#ifdef SNDCTL_DSP_MAPOUTBUF _EXPORT_INT(m, SNDCTL_DSP_MAPOUTBUF); +#endif _EXPORT_INT(m, SNDCTL_DSP_NONBLOCK); _EXPORT_INT(m, SNDCTL_DSP_POST); #ifdef SNDCTL_DSP_PROFILE From webhook-mailer at python.org Wed Apr 20 07:09:06 2022 From: webhook-mailer at python.org (miss-islington) Date: Wed, 20 Apr 2022 11:09:06 -0000 Subject: [Python-checkins] gh-91734: Fix ossaudio support on Solaris (GH-91735) Message-ID: https://github.com/python/cpython/commit/56c2d0809703eda9d2f4ab5600c054b73a5a2460 commit: 56c2d0809703eda9d2f4ab5600c054b73a5a2460 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-20T04:09:01-07:00 summary: gh-91734: Fix ossaudio support on Solaris (GH-91735) (cherry picked from commit 4420faf273e9e2d03226a9375e1e04a336230c84) Co-authored-by: Jakub Kul?k files: A Misc/NEWS.d/next/Library/2022-04-20-09-49-33.gh-issue-91734.4Dj4Gy.rst M Modules/ossaudiodev.c diff --git a/Misc/NEWS.d/next/Library/2022-04-20-09-49-33.gh-issue-91734.4Dj4Gy.rst b/Misc/NEWS.d/next/Library/2022-04-20-09-49-33.gh-issue-91734.4Dj4Gy.rst new file mode 100644 index 0000000000000..47d9e0dea458a --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-20-09-49-33.gh-issue-91734.4Dj4Gy.rst @@ -0,0 +1 @@ +Fix OSS audio support on Solaris. diff --git a/Modules/ossaudiodev.c b/Modules/ossaudiodev.c index 2a1ac10814a69..fb57764b6aab4 100644 --- a/Modules/ossaudiodev.c +++ b/Modules/ossaudiodev.c @@ -1243,8 +1243,12 @@ PyInit_ossaudiodev(void) _EXPORT_INT(m, SNDCTL_DSP_GETSPDIF); #endif _EXPORT_INT(m, SNDCTL_DSP_GETTRIGGER); +#ifdef SNDCTL_DSP_MAPINBUF _EXPORT_INT(m, SNDCTL_DSP_MAPINBUF); +#endif +#ifdef SNDCTL_DSP_MAPOUTBUF _EXPORT_INT(m, SNDCTL_DSP_MAPOUTBUF); +#endif _EXPORT_INT(m, SNDCTL_DSP_NONBLOCK); _EXPORT_INT(m, SNDCTL_DSP_POST); #ifdef SNDCTL_DSP_PROFILE From webhook-mailer at python.org Wed Apr 20 07:19:14 2022 From: webhook-mailer at python.org (vstinner) Date: Wed, 20 Apr 2022 11:19:14 -0000 Subject: [Python-checkins] gh-91731: Build Python with -std=c11 (#91733) Message-ID: https://github.com/python/cpython/commit/aaeea78b0f8f4c8dfab5cd4eb6fb4c5fe1ee21e2 commit: aaeea78b0f8f4c8dfab5cd4eb6fb4c5fe1ee21e2 branch: main author: Victor Stinner committer: vstinner date: 2022-04-20T13:19:05+02:00 summary: gh-91731: Build Python with -std=c11 (#91733) Python is now built with "-std=c11" compiler option, rather than "-std=c99". files: A Misc/NEWS.d/next/Build/2022-04-20-11-14-51.gh-issue-91731.zRoPcJ.rst M configure M configure.ac diff --git a/Misc/NEWS.d/next/Build/2022-04-20-11-14-51.gh-issue-91731.zRoPcJ.rst b/Misc/NEWS.d/next/Build/2022-04-20-11-14-51.gh-issue-91731.zRoPcJ.rst new file mode 100644 index 0000000000000..92a1f52cbce4c --- /dev/null +++ b/Misc/NEWS.d/next/Build/2022-04-20-11-14-51.gh-issue-91731.zRoPcJ.rst @@ -0,0 +1,2 @@ +Python is now built with ``-std=c11`` compiler option, rather than +``-std=c99``. Patch by Victor Stinner. diff --git a/configure b/configure index 65495194f4935..94adc6fdf015e 100755 --- a/configure +++ b/configure @@ -7871,7 +7871,7 @@ UNIVERSAL_ARCH_FLAGS= # tweak BASECFLAGS based on compiler and platform case $GCC in yes) - CFLAGS_NODIST="$CFLAGS_NODIST -std=c99" + CFLAGS_NODIST="$CFLAGS_NODIST -std=c11" diff --git a/configure.ac b/configure.ac index 276718aeeb77a..9b60b98d1434e 100644 --- a/configure.ac +++ b/configure.ac @@ -1983,7 +1983,7 @@ AC_DEFUN([PY_CHECK_CC_WARNING], [ # tweak BASECFLAGS based on compiler and platform case $GCC in yes) - CFLAGS_NODIST="$CFLAGS_NODIST -std=c99" + CFLAGS_NODIST="$CFLAGS_NODIST -std=c11" PY_CHECK_CC_WARNING([enable], [extra], [if we can add -Wextra]) AS_VAR_IF([ac_cv_enable_extra_warning], [yes], From webhook-mailer at python.org Wed Apr 20 08:43:27 2022 From: webhook-mailer at python.org (iritkatriel) Date: Wed, 20 Apr 2022 12:43:27 -0000 Subject: [Python-checkins] gh-89770: [PEP-678] add exception notes to tutorial (GH-30441) Message-ID: https://github.com/python/cpython/commit/f92bcfe6dea7d52dcc7077585d8818e8a0209604 commit: f92bcfe6dea7d52dcc7077585d8818e8a0209604 branch: main author: Irit Katriel <1055913+iritkatriel at users.noreply.github.com> committer: iritkatriel <1055913+iritkatriel at users.noreply.github.com> date: 2022-04-20T13:43:10+01:00 summary: gh-89770: [PEP-678] add exception notes to tutorial (GH-30441) files: M Doc/library/traceback.rst M Doc/tutorial/errors.rst diff --git a/Doc/library/traceback.rst b/Doc/library/traceback.rst index df4a38c955511..796309c6cf0bb 100644 --- a/Doc/library/traceback.rst +++ b/Doc/library/traceback.rst @@ -236,6 +236,14 @@ capture data for later printing in a lightweight fashion. The ``__suppress_context__`` value from the original exception. + .. attribute:: __notes__ + + The ``__notes__`` value from the original exception, or ``None`` + if the exception does not have any notes. If it is not ``None`` + is it formatted in the traceback after the exception string. + + .. versionadded:: 3.11 + .. attribute:: stack A :class:`StackSummary` representing the traceback. diff --git a/Doc/tutorial/errors.rst b/Doc/tutorial/errors.rst index 01fd76283556b..57919e3bad132 100644 --- a/Doc/tutorial/errors.rst +++ b/Doc/tutorial/errors.rst @@ -559,3 +559,67 @@ the following pattern:: ... raise ExceptionGroup("Test Failures", excs) ... + +Enriching Exceptions with Notes +=============================== + +When an exception is created in order to be raised, it is usually initialized +with information that describes the error that has occurred. There are cases +where it is useful to add information after the exception was caught. For this +purpose, exceptions have a method ``add_note(note)`` that accepts a string and +adds it to the exception's notes list. The standard traceback rendering +includes all notes, in the order they were added, after the exception. :: + + >>> try: + ... raise TypeError('bad type') + ... except Exception as e: + ... e.add_note('Add some information') + ... e.add_note('Add some more information') + ... raise + ... + Traceback (most recent call last): + File "", line 2, in + TypeError: bad type + Add some information + Add some more information + >>> + +For example, when collecting exceptions into an exception group, we may want +to add context information for the individual errors. In the following each +exception in the group has a note indicating when this error has occurred. :: + + >>> def f(): + ... raise OSError('operation failed') + ... + >>> excs = [] + >>> for i in range(3): + ... try: + ... f() + ... except Exception as e: + ... e.add_note(f'Happened in Iteration {i+1}') + ... excs.append(e) + ... + >>> raise ExceptionGroup('We have some problems', excs) + + Exception Group Traceback (most recent call last): + | File "", line 1, in + | ExceptionGroup: We have some problems (3 sub-exceptions) + +-+---------------- 1 ---------------- + | Traceback (most recent call last): + | File "", line 3, in + | File "", line 2, in f + | OSError: operation failed + | Happened in Iteration 1 + +---------------- 2 ---------------- + | Traceback (most recent call last): + | File "", line 3, in + | File "", line 2, in f + | OSError: operation failed + | Happened in Iteration 2 + +---------------- 3 ---------------- + | Traceback (most recent call last): + | File "", line 3, in + | File "", line 2, in f + | OSError: operation failed + | Happened in Iteration 3 + +------------------------------------ + >>> From webhook-mailer at python.org Wed Apr 20 09:30:41 2022 From: webhook-mailer at python.org (zooba) Date: Wed, 20 Apr 2022 13:30:41 -0000 Subject: [Python-checkins] bpo-40859: Update Windows build to use xz-5.2.5 (GH-20622) Message-ID: https://github.com/python/cpython/commit/73af4b0264aeb866536b01a132f8c26a5808d64d commit: 73af4b0264aeb866536b01a132f8c26a5808d64d branch: 3.10 author: Steve Dower committer: zooba date: 2022-04-20T14:30:16+01:00 summary: bpo-40859: Update Windows build to use xz-5.2.5 (GH-20622) Co-authored-by: Ma Lin files: A Misc/NEWS.d/next/Windows/2020-06-04-10-42-04.bpo-40859.isKSw7.rst M PCbuild/get_externals.bat M PCbuild/liblzma.vcxproj M PCbuild/liblzma.vcxproj.filters M PCbuild/python.props diff --git a/Misc/NEWS.d/next/Windows/2020-06-04-10-42-04.bpo-40859.isKSw7.rst b/Misc/NEWS.d/next/Windows/2020-06-04-10-42-04.bpo-40859.isKSw7.rst new file mode 100644 index 0000000000000..ef4c727ad2866 --- /dev/null +++ b/Misc/NEWS.d/next/Windows/2020-06-04-10-42-04.bpo-40859.isKSw7.rst @@ -0,0 +1 @@ +Update Windows build to use xz-5.2.5 diff --git a/PCbuild/get_externals.bat b/PCbuild/get_externals.bat index dc5c909de7441..ee79addd44d02 100644 --- a/PCbuild/get_externals.bat +++ b/PCbuild/get_externals.bat @@ -58,7 +58,7 @@ set libraries=%libraries% sqlite-3.37.2.0 if NOT "%IncludeTkinterSrc%"=="false" set libraries=%libraries% tcl-core-8.6.12.0 if NOT "%IncludeTkinterSrc%"=="false" set libraries=%libraries% tk-8.6.12.0 if NOT "%IncludeTkinterSrc%"=="false" set libraries=%libraries% tix-8.4.3.6 -set libraries=%libraries% xz-5.2.2 +set libraries=%libraries% xz-5.2.5 set libraries=%libraries% zlib-1.2.12 for %%e in (%libraries%) do ( diff --git a/PCbuild/liblzma.vcxproj b/PCbuild/liblzma.vcxproj index a6bd59ec0baa3..4dd42ab98a975 100644 --- a/PCbuild/liblzma.vcxproj +++ b/PCbuild/liblzma.vcxproj @@ -92,7 +92,7 @@ WIN32;HAVE_CONFIG_H;_LIB;%(PreprocessorDefinitions) - $(lzmaDir)windows;$(lzmaDir)src/liblzma/common;$(lzmaDir)src/common;$(lzmaDir)src/liblzma/api;$(lzmaDir)src/liblzma/check;$(lzmaDir)src/liblzma/delta;$(lzmaDir)src/liblzma/lz;$(lzmaDir)src/liblzma/lzma;$(lzmaDir)src/liblzma/rangecoder;$(lzmaDir)src/liblzma/simple;%(AdditionalIncludeDirectories) + $(lzmaDir)windows/vs2019;$(lzmaDir)src/liblzma/common;$(lzmaDir)src/common;$(lzmaDir)src/liblzma/api;$(lzmaDir)src/liblzma/check;$(lzmaDir)src/liblzma/delta;$(lzmaDir)src/liblzma/lz;$(lzmaDir)src/liblzma/lzma;$(lzmaDir)src/liblzma/rangecoder;$(lzmaDir)src/liblzma/simple;%(AdditionalIncludeDirectories) 4028;4113;4133;4244;4267;4996;%(DisableSpecificWarnings) @@ -238,7 +238,7 @@ - + diff --git a/PCbuild/liblzma.vcxproj.filters b/PCbuild/liblzma.vcxproj.filters index 3f58351fa9edb..ebe2a7d5fa9e5 100644 --- a/PCbuild/liblzma.vcxproj.filters +++ b/PCbuild/liblzma.vcxproj.filters @@ -428,7 +428,7 @@ Header Files - + Header Files diff --git a/PCbuild/python.props b/PCbuild/python.props index 7b42037b7cdf8..c451429da2c82 100644 --- a/PCbuild/python.props +++ b/PCbuild/python.props @@ -59,7 +59,7 @@ $(ExternalsDir)\ $(ExternalsDir)sqlite-3.37.2.0\ $(ExternalsDir)bzip2-1.0.8\ - $(ExternalsDir)xz-5.2.2\ + $(ExternalsDir)xz-5.2.5\ $(ExternalsDir)libffi-3.3.0\ $(ExternalsDir)libffi-3.3.0\$(ArchName)\ $(libffiOutDir)include From webhook-mailer at python.org Wed Apr 20 10:11:09 2022 From: webhook-mailer at python.org (corona10) Date: Wed, 20 Apr 2022 14:11:09 -0000 Subject: [Python-checkins] gh-91632: Fix generic_alias_iterator to be finalized at exit. (GH-91727) Message-ID: https://github.com/python/cpython/commit/f571c26fc1805790cacb73e70cbb0291204d41e7 commit: f571c26fc1805790cacb73e70cbb0291204d41e7 branch: main author: Dong-hee Na committer: corona10 date: 2022-04-20T23:10:41+09:00 summary: gh-91632: Fix generic_alias_iterator to be finalized at exit. (GH-91727) files: A Misc/NEWS.d/next/Core and Builtins/2022-04-20-14-43-37.gh-issue-91632.cvUhsZ.rst M Objects/genericaliasobject.c M Objects/object.c diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-04-20-14-43-37.gh-issue-91632.cvUhsZ.rst b/Misc/NEWS.d/next/Core and Builtins/2022-04-20-14-43-37.gh-issue-91632.cvUhsZ.rst new file mode 100644 index 0000000000000..e7837828901fc --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2022-04-20-14-43-37.gh-issue-91632.cvUhsZ.rst @@ -0,0 +1 @@ +Fix a minor memory leak at exit: release the memory of the :class:`generic_alias_iterator` type. Patch by Dong-hee Na. diff --git a/Objects/genericaliasobject.c b/Objects/genericaliasobject.c index dc585de4729fc..7059c4018303e 100644 --- a/Objects/genericaliasobject.c +++ b/Objects/genericaliasobject.c @@ -683,7 +683,9 @@ ga_iter_clear(PyObject *self) { return 0; } -static PyTypeObject Py_GenericAliasIterType = { +// gh-91632: _Py_GenericAliasIterType is exported to be cleared +// in _PyTypes_FiniTypes. +PyTypeObject _Py_GenericAliasIterType = { PyVarObject_HEAD_INIT(&PyType_Type, 0) .tp_name = "generic_alias_iterator", .tp_basicsize = sizeof(gaiterobject), @@ -697,7 +699,7 @@ static PyTypeObject Py_GenericAliasIterType = { static PyObject * ga_iter(PyObject *self) { - gaiterobject *gi = PyObject_GC_New(gaiterobject, &Py_GenericAliasIterType); + gaiterobject *gi = PyObject_GC_New(gaiterobject, &_Py_GenericAliasIterType); if (gi == NULL) { return NULL; } diff --git a/Objects/object.c b/Objects/object.c index 8adb5065c2af4..29880f6003bb1 100644 --- a/Objects/object.c +++ b/Objects/object.c @@ -1834,6 +1834,7 @@ _PyTypes_InitState(PyInterpreterState *interp) #ifdef MS_WINDOWS extern PyTypeObject PyHKEY_Type; #endif +extern PyTypeObject _Py_GenericAliasIterType; static PyTypeObject* static_types[] = { // The two most important base types: must be initialized first and @@ -1923,6 +1924,7 @@ static PyTypeObject* static_types[] = { &_PyAsyncGenWrappedValue_Type, &_PyContextTokenMissing_Type, &_PyCoroWrapper_Type, + &_Py_GenericAliasIterType, &_PyHamtItems_Type, &_PyHamtKeys_Type, &_PyHamtValues_Type, From webhook-mailer at python.org Wed Apr 20 10:40:26 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Wed, 20 Apr 2022 14:40:26 -0000 Subject: [Python-checkins] [3.9] build(deps): bump actions/checkout from 2 to 3 (GH-32226) (#91722) Message-ID: https://github.com/python/cpython/commit/c25a0b4a92171754518a99bf9e27c60396668f59 commit: c25a0b4a92171754518a99bf9e27c60396668f59 branch: 3.9 author: Jelle Zijlstra committer: JelleZijlstra date: 2022-04-20T07:40:17-07:00 summary: [3.9] build(deps): bump actions/checkout from 2 to 3 (GH-32226) (#91722) Bumps [actions/checkout](https://github.com/actions/checkout) from 2 to 3. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](https://github.com/actions/checkout/compare/v2...v3) --- updated-dependencies: - dependency-name: actions/checkout dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>. (cherry picked from commit 1ba63e3a9bb176e5ad8e8dd744b9d2b9d588e275) files: M .github/workflows/build.yml M .github/workflows/build_msi.yml M .github/workflows/doc.yml diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 61aefdcc2367b..13181919efe07 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -25,7 +25,7 @@ jobs: run_tests: ${{ steps.check.outputs.run_tests }} run_ssl_tests: ${{ steps.check.outputs.run_ssl_tests }} steps: - - uses: actions/checkout at v2 + - uses: actions/checkout at v3 - name: Check for source changes id: check run: | @@ -79,7 +79,7 @@ jobs: needs: check_source if: needs.check_source.outputs.run_tests == 'true' steps: - - uses: actions/checkout at v2 + - uses: actions/checkout at v3 - uses: actions/setup-python at v3 - name: Install Dependencies run: sudo ./.github/workflows/posix-deps-apt.sh @@ -119,7 +119,7 @@ jobs: needs: check_source if: needs.check_source.outputs.run_tests == 'true' steps: - - uses: actions/checkout at v2 + - uses: actions/checkout at v3 - name: Build CPython run: .\PCbuild\build.bat -e -p Win32 - name: Display build info @@ -133,7 +133,7 @@ jobs: needs: check_source if: needs.check_source.outputs.run_tests == 'true' steps: - - uses: actions/checkout at v2 + - uses: actions/checkout at v3 - name: Build CPython run: .\PCbuild\build.bat -e -p x64 - name: Display build info @@ -149,7 +149,7 @@ jobs: env: PYTHONSTRICTEXTENSIONBUILD: 1 steps: - - uses: actions/checkout at v2 + - uses: actions/checkout at v3 - name: Configure CPython run: ./configure --with-pydebug --with-openssl=/usr/local/opt/openssl --prefix=/opt/python-dev - name: Build CPython @@ -168,7 +168,7 @@ jobs: OPENSSL_VER: 1.1.1n PYTHONSTRICTEXTENSIONBUILD: 1 steps: - - uses: actions/checkout at v2 + - uses: actions/checkout at v3 - name: Install Dependencies run: sudo ./.github/workflows/posix-deps-apt.sh - name: Configure OpenSSL env vars @@ -214,7 +214,7 @@ jobs: OPENSSL_DIR: ${{ github.workspace }}/multissl/openssl/${{ matrix.openssl_ver }} LD_LIBRARY_PATH: ${{ github.workspace }}/multissl/openssl/${{ matrix.openssl_ver }}/lib steps: - - uses: actions/checkout at v2 + - uses: actions/checkout at v3 - name: Install Dependencies run: sudo ./.github/workflows/posix-deps-apt.sh - name: Configure OpenSSL env vars diff --git a/.github/workflows/build_msi.yml b/.github/workflows/build_msi.yml index 3664ccb32e70e..6de374e5d5ab7 100644 --- a/.github/workflows/build_msi.yml +++ b/.github/workflows/build_msi.yml @@ -25,7 +25,7 @@ jobs: name: 'Windows (x86) Installer' runs-on: windows-latest steps: - - uses: actions/checkout at v2 + - uses: actions/checkout at v3 - name: Build CPython installer run: .\Tools\msi\build.bat -x86 @@ -33,6 +33,6 @@ jobs: name: 'Windows (x64) Installer' runs-on: windows-latest steps: - - uses: actions/checkout at v2 + - uses: actions/checkout at v3 - name: Build CPython installer run: .\Tools\msi\build.bat -x64 diff --git a/.github/workflows/doc.yml b/.github/workflows/doc.yml index 18efd7f328c25..537478a2850f4 100644 --- a/.github/workflows/doc.yml +++ b/.github/workflows/doc.yml @@ -24,7 +24,7 @@ jobs: name: 'Docs' runs-on: ubuntu-latest steps: - - uses: actions/checkout at v2 + - uses: actions/checkout at v3 - name: 'Install Dependencies' run: sudo ./.github/workflows/posix-deps-apt.sh && sudo apt-get install wamerican - name: 'Configure CPython' From webhook-mailer at python.org Wed Apr 20 10:40:42 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Wed, 20 Apr 2022 14:40:42 -0000 Subject: [Python-checkins] [3.10] build(deps): bump actions/checkout from 2 to 3 (GH-32226) (#91723) Message-ID: https://github.com/python/cpython/commit/c33524e68b853b8b313536af1b7092efd53e5d15 commit: c33524e68b853b8b313536af1b7092efd53e5d15 branch: 3.10 author: Jelle Zijlstra committer: JelleZijlstra date: 2022-04-20T07:40:36-07:00 summary: [3.10] build(deps): bump actions/checkout from 2 to 3 (GH-32226) (#91723) Bumps [actions/checkout](https://github.com/actions/checkout) from 2 to 3. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](https://github.com/actions/checkout/compare/v2...v3) --- updated-dependencies: - dependency-name: actions/checkout dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>. (cherry picked from commit 1ba63e3a9bb176e5ad8e8dd744b9d2b9d588e275) files: M .github/workflows/build.yml M .github/workflows/build_msi.yml M .github/workflows/doc.yml diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 43b4f76937657..ce300ebe59bfa 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -27,7 +27,7 @@ jobs: run_tests: ${{ steps.check.outputs.run_tests }} run_ssl_tests: ${{ steps.check.outputs.run_ssl_tests }} steps: - - uses: actions/checkout at v2 + - uses: actions/checkout at v3 - name: Check for source changes id: check run: | @@ -81,7 +81,7 @@ jobs: needs: check_source if: needs.check_source.outputs.run_tests == 'true' steps: - - uses: actions/checkout at v2 + - uses: actions/checkout at v3 - uses: actions/setup-python at v3 - name: Install Dependencies run: sudo ./.github/workflows/posix-deps-apt.sh @@ -127,7 +127,7 @@ jobs: env: IncludeUwp: 'true' steps: - - uses: actions/checkout at v2 + - uses: actions/checkout at v3 - name: Build CPython run: .\PCbuild\build.bat -e -p Win32 - name: Display build info @@ -143,7 +143,7 @@ jobs: env: IncludeUwp: 'true' steps: - - uses: actions/checkout at v2 + - uses: actions/checkout at v3 - name: Register MSVC problem matcher run: echo "::add-matcher::.github/problem-matchers/msvc.json" - name: Build CPython @@ -161,7 +161,7 @@ jobs: env: PYTHONSTRICTEXTENSIONBUILD: 1 steps: - - uses: actions/checkout at v2 + - uses: actions/checkout at v3 - name: Prepare homebrew environment variables run: | echo "LDFLAGS=-L$(brew --prefix tcl-tk)/lib" >> $GITHUB_ENV @@ -184,7 +184,7 @@ jobs: OPENSSL_VER: 1.1.1n PYTHONSTRICTEXTENSIONBUILD: 1 steps: - - uses: actions/checkout at v2 + - uses: actions/checkout at v3 - name: Register gcc problem matcher run: echo "::add-matcher::.github/problem-matchers/gcc.json" - name: Install Dependencies @@ -232,7 +232,7 @@ jobs: OPENSSL_DIR: ${{ github.workspace }}/multissl/openssl/${{ matrix.openssl_ver }} LD_LIBRARY_PATH: ${{ github.workspace }}/multissl/openssl/${{ matrix.openssl_ver }}/lib steps: - - uses: actions/checkout at v2 + - uses: actions/checkout at v3 - name: Register gcc problem matcher run: echo "::add-matcher::.github/problem-matchers/gcc.json" - name: Install Dependencies diff --git a/.github/workflows/build_msi.yml b/.github/workflows/build_msi.yml index 28ce06f3cf4e0..201098a635799 100644 --- a/.github/workflows/build_msi.yml +++ b/.github/workflows/build_msi.yml @@ -25,7 +25,7 @@ jobs: name: 'Windows (x86) Installer' runs-on: windows-latest steps: - - uses: actions/checkout at v2 + - uses: actions/checkout at v3 - name: Build CPython installer run: .\Tools\msi\build.bat -x86 @@ -33,6 +33,6 @@ jobs: name: 'Windows (x64) Installer' runs-on: windows-latest steps: - - uses: actions/checkout at v2 + - uses: actions/checkout at v3 - name: Build CPython installer run: .\Tools\msi\build.bat -x64 diff --git a/.github/workflows/doc.yml b/.github/workflows/doc.yml index 4426238b9d9fa..970f5af2d9bed 100644 --- a/.github/workflows/doc.yml +++ b/.github/workflows/doc.yml @@ -25,7 +25,7 @@ jobs: name: 'Docs' runs-on: ubuntu-latest steps: - - uses: actions/checkout at v2 + - uses: actions/checkout at v3 - name: Register Sphinx problem matcher run: echo "::add-matcher::.github/problem-matchers/sphinx.json" - name: 'Install Dependencies' From webhook-mailer at python.org Wed Apr 20 10:53:26 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Wed, 20 Apr 2022 14:53:26 -0000 Subject: [Python-checkins] build(deps): bump actions/cache from 2.1.7 to 3.0.1 (#32228) Message-ID: https://github.com/python/cpython/commit/3ace1034b8202bc7034e15f34561725934f04ff6 commit: 3ace1034b8202bc7034e15f34561725934f04ff6 branch: main author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> committer: JelleZijlstra date: 2022-04-20T07:53:08-07:00 summary: build(deps): bump actions/cache from 2.1.7 to 3.0.1 (#32228) Bumps [actions/cache](https://github.com/actions/cache) from 2.1.7 to 3.0.1. - [Release notes](https://github.com/actions/cache/releases) - [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md) - [Commits](https://github.com/actions/cache/compare/v2.1.7...v3.0.1) --- updated-dependencies: - dependency-name: actions/cache dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jelle Zijlstra files: M .github/workflows/build.yml diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 785a7e9b683b3..8956c2cfc4e70 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -185,7 +185,7 @@ jobs: echo "LD_LIBRARY_PATH=${GITHUB_WORKSPACE}/multissl/openssl/${OPENSSL_VER}/lib" >> $GITHUB_ENV - name: 'Restore OpenSSL build' id: cache-openssl - uses: actions/cache at v2.1.7 + uses: actions/cache at v3.0.1 with: path: ./multissl/openssl/${{ env.OPENSSL_VER }} key: ${{ runner.os }}-multissl-openssl-${{ env.OPENSSL_VER }} @@ -248,7 +248,7 @@ jobs: echo "LD_LIBRARY_PATH=${GITHUB_WORKSPACE}/multissl/openssl/${OPENSSL_VER}/lib" >> $GITHUB_ENV - name: 'Restore OpenSSL build' id: cache-openssl - uses: actions/cache at v2.1.7 + uses: actions/cache at v3.0.1 with: path: ./multissl/openssl/${{ env.OPENSSL_VER }} key: ${{ runner.os }}-multissl-openssl-${{ env.OPENSSL_VER }} @@ -292,7 +292,7 @@ jobs: echo "LD_LIBRARY_PATH=${GITHUB_WORKSPACE}/multissl/openssl/${OPENSSL_VER}/lib" >> $GITHUB_ENV - name: 'Restore OpenSSL build' id: cache-openssl - uses: actions/cache at v2.1.7 + uses: actions/cache at v3.0.1 with: path: ./multissl/openssl/${{ env.OPENSSL_VER }} key: ${{ runner.os }}-multissl-openssl-${{ env.OPENSSL_VER }} From webhook-mailer at python.org Wed Apr 20 11:05:01 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Wed, 20 Apr 2022 15:05:01 -0000 Subject: [Python-checkins] build(deps): bump actions/upload-artifact from 2.3.1 to 3 (#32227) Message-ID: https://github.com/python/cpython/commit/b8812c9ca3f6864a233574001e16f1b9e92daf6e commit: b8812c9ca3f6864a233574001e16f1b9e92daf6e branch: main author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> committer: JelleZijlstra date: 2022-04-20T08:04:56-07:00 summary: build(deps): bump actions/upload-artifact from 2.3.1 to 3 (#32227) Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 2.3.1 to 3. - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](https://github.com/actions/upload-artifact/compare/v2.3.1...v3) --- updated-dependencies: - dependency-name: actions/upload-artifact dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jelle Zijlstra files: M .github/workflows/doc.yml diff --git a/.github/workflows/doc.yml b/.github/workflows/doc.yml index f96a150970ec7..3ed66e74b8954 100644 --- a/.github/workflows/doc.yml +++ b/.github/workflows/doc.yml @@ -48,7 +48,7 @@ jobs: - name: 'Build HTML documentation' run: make -C Doc/ PYTHON=../python SPHINXOPTS="-q -W --keep-going -j4" html - name: 'Upload' - uses: actions/upload-artifact at v2.3.1 + uses: actions/upload-artifact at v3 with: name: doc-html path: Doc/build/html From webhook-mailer at python.org Wed Apr 20 11:15:58 2022 From: webhook-mailer at python.org (markshannon) Date: Wed, 20 Apr 2022 15:15:58 -0000 Subject: [Python-checkins] Cast to (destructor) to fix compiler warnings (GH-91711) Message-ID: https://github.com/python/cpython/commit/d7d7e6c00778d234fda2c2229faccec5e8f48643 commit: d7d7e6c00778d234fda2c2229faccec5e8f48643 branch: main author: Dennis Sweeney <36520290+sweeneyde at users.noreply.github.com> committer: markshannon date: 2022-04-20T16:15:45+01:00 summary: Cast to (destructor) to fix compiler warnings (GH-91711) files: M Objects/longobject.c M Python/ceval.c diff --git a/Objects/longobject.c b/Objects/longobject.c index c104dcc14f986..660824f90659f 100644 --- a/Objects/longobject.c +++ b/Objects/longobject.c @@ -40,7 +40,7 @@ static inline void _Py_DECREF_INT(PyLongObject *op) { assert(PyLong_CheckExact(op)); - _Py_DECREF_SPECIALIZED((PyObject *)op, PyObject_Free); + _Py_DECREF_SPECIALIZED((PyObject *)op, (destructor)PyObject_Free); } static inline int diff --git a/Python/ceval.c b/Python/ceval.c index 45754ffbe7cb4..12b0aef774e36 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -1977,8 +1977,8 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int STAT_INC(BINARY_OP, hit); PyObject *prod = _PyLong_Multiply((PyLongObject *)left, (PyLongObject *)right); SET_SECOND(prod); - _Py_DECREF_SPECIALIZED(right, PyObject_Free); - _Py_DECREF_SPECIALIZED(left, PyObject_Free); + _Py_DECREF_SPECIALIZED(right, (destructor)PyObject_Free); + _Py_DECREF_SPECIALIZED(left, (destructor)PyObject_Free); STACK_SHRINK(1); if (prod == NULL) { goto error; @@ -2017,8 +2017,8 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int STAT_INC(BINARY_OP, hit); PyObject *sub = _PyLong_Subtract((PyLongObject *)left, (PyLongObject *)right); SET_SECOND(sub); - _Py_DECREF_SPECIALIZED(right, PyObject_Free); - _Py_DECREF_SPECIALIZED(left, PyObject_Free); + _Py_DECREF_SPECIALIZED(right, (destructor)PyObject_Free); + _Py_DECREF_SPECIALIZED(left, (destructor)PyObject_Free); STACK_SHRINK(1); if (sub == NULL) { goto error; @@ -2132,8 +2132,8 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int STAT_INC(BINARY_OP, hit); PyObject *sum = _PyLong_Add((PyLongObject *)left, (PyLongObject *)right); SET_SECOND(sum); - _Py_DECREF_SPECIALIZED(right, PyObject_Free); - _Py_DECREF_SPECIALIZED(left, PyObject_Free); + _Py_DECREF_SPECIALIZED(right, (destructor)PyObject_Free); + _Py_DECREF_SPECIALIZED(left, (destructor)PyObject_Free); STACK_SHRINK(1); if (sum == NULL) { goto error; @@ -2192,7 +2192,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int assert(res != NULL); Py_INCREF(res); STACK_SHRINK(1); - _Py_DECREF_SPECIALIZED(sub, PyObject_Free); + _Py_DECREF_SPECIALIZED(sub, (destructor)PyObject_Free); SET_TOP(res); Py_DECREF(list); JUMPBY(INLINE_CACHE_ENTRIES_BINARY_SUBSCR); @@ -2217,7 +2217,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int assert(res != NULL); Py_INCREF(res); STACK_SHRINK(1); - _Py_DECREF_SPECIALIZED(sub, PyObject_Free); + _Py_DECREF_SPECIALIZED(sub, (destructor)PyObject_Free); SET_TOP(res); Py_DECREF(tuple); JUMPBY(INLINE_CACHE_ENTRIES_BINARY_SUBSCR); @@ -2359,7 +2359,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int STACK_SHRINK(3); assert(old_value != NULL); Py_DECREF(old_value); - _Py_DECREF_SPECIALIZED(sub, PyObject_Free); + _Py_DECREF_SPECIALIZED(sub, (destructor)PyObject_Free); Py_DECREF(list); JUMPBY(INLINE_CACHE_ENTRIES_STORE_SUBSCR); NOTRACE_DISPATCH(); @@ -3795,8 +3795,8 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int JUMPBY(INLINE_CACHE_ENTRIES_COMPARE_OP); NEXTOPARG(); STACK_SHRINK(2); - _Py_DECREF_SPECIALIZED(left, PyObject_Free); - _Py_DECREF_SPECIALIZED(right, PyObject_Free); + _Py_DECREF_SPECIALIZED(left, (destructor)PyObject_Free); + _Py_DECREF_SPECIALIZED(right, (destructor)PyObject_Free); assert(opcode == POP_JUMP_FORWARD_IF_FALSE || opcode == POP_JUMP_BACKWARD_IF_FALSE || opcode == POP_JUMP_FORWARD_IF_TRUE || From webhook-mailer at python.org Wed Apr 20 12:23:49 2022 From: webhook-mailer at python.org (vstinner) Date: Wed, 20 Apr 2022 16:23:49 -0000 Subject: [Python-checkins] bpo-23747: Enhance platform doc, document default behavior (GH-31462) Message-ID: https://github.com/python/cpython/commit/ad3ca17ff5cd63f907430073b52be27695674148 commit: ad3ca17ff5cd63f907430073b52be27695674148 branch: main author: slateny <46876382+slateny at users.noreply.github.com> committer: vstinner date: 2022-04-20T18:23:40+02:00 summary: bpo-23747: Enhance platform doc, document default behavior (GH-31462) files: M Doc/library/platform.rst diff --git a/Doc/library/platform.rst b/Doc/library/platform.rst index a0eece6c4d8aa..346063d66afab 100644 --- a/Doc/library/platform.rst +++ b/Doc/library/platform.rst @@ -139,7 +139,7 @@ Cross Platform .. function:: release() - Returns the system's release, e.g. ``'2.2.0'`` or ``'NT'`` An empty string is + Returns the system's release, e.g. ``'2.2.0'`` or ``'NT'``. An empty string is returned if the value cannot be determined. @@ -176,7 +176,7 @@ Cross Platform Entries which cannot be determined are set to ``''``. .. versionchanged:: 3.3 - Result changed from a tuple to a namedtuple. + Result changed from a tuple to a :func:`~collections.namedtuple`. Java Platform @@ -201,7 +201,9 @@ Windows Platform Get additional version information from the Windows Registry and return a tuple ``(release, version, csd, ptype)`` referring to OS release, version number, - CSD level (service pack) and OS type (multi/single processor). + CSD level (service pack) and OS type (multi/single processor). Values which + cannot be determined are set to the defaults given as parameters (which all + default to an empty string). As a hint: *ptype* is ``'Uniprocessor Free'`` on single processor NT machines and ``'Multiprocessor Free'`` on multi processor machines. The *'Free'* refers @@ -211,9 +213,9 @@ Windows Platform .. function:: win32_edition() - Returns a string representing the current Windows edition. Possible - values include but are not limited to ``'Enterprise'``, ``'IoTUAP'``, - ``'ServerStandard'``, and ``'nanoserver'``. + Returns a string representing the current Windows edition, or ``None`` if the + value cannot be determined. Possible values include but are not limited to + ``'Enterprise'``, ``'IoTUAP'``, ``'ServerStandard'``, and ``'nanoserver'``. .. versionadded:: 3.8 From webhook-mailer at python.org Wed Apr 20 12:41:24 2022 From: webhook-mailer at python.org (miss-islington) Date: Wed, 20 Apr 2022 16:41:24 -0000 Subject: [Python-checkins] bpo-23747: Enhance platform doc, document default behavior (GH-31462) Message-ID: https://github.com/python/cpython/commit/886b22c4c3c5324bcebb79402c92d18ebc4224b6 commit: 886b22c4c3c5324bcebb79402c92d18ebc4224b6 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-20T09:41:12-07:00 summary: bpo-23747: Enhance platform doc, document default behavior (GH-31462) (cherry picked from commit ad3ca17ff5cd63f907430073b52be27695674148) Co-authored-by: slateny <46876382+slateny at users.noreply.github.com> files: M Doc/library/platform.rst diff --git a/Doc/library/platform.rst b/Doc/library/platform.rst index a0eece6c4d8aa..346063d66afab 100644 --- a/Doc/library/platform.rst +++ b/Doc/library/platform.rst @@ -139,7 +139,7 @@ Cross Platform .. function:: release() - Returns the system's release, e.g. ``'2.2.0'`` or ``'NT'`` An empty string is + Returns the system's release, e.g. ``'2.2.0'`` or ``'NT'``. An empty string is returned if the value cannot be determined. @@ -176,7 +176,7 @@ Cross Platform Entries which cannot be determined are set to ``''``. .. versionchanged:: 3.3 - Result changed from a tuple to a namedtuple. + Result changed from a tuple to a :func:`~collections.namedtuple`. Java Platform @@ -201,7 +201,9 @@ Windows Platform Get additional version information from the Windows Registry and return a tuple ``(release, version, csd, ptype)`` referring to OS release, version number, - CSD level (service pack) and OS type (multi/single processor). + CSD level (service pack) and OS type (multi/single processor). Values which + cannot be determined are set to the defaults given as parameters (which all + default to an empty string). As a hint: *ptype* is ``'Uniprocessor Free'`` on single processor NT machines and ``'Multiprocessor Free'`` on multi processor machines. The *'Free'* refers @@ -211,9 +213,9 @@ Windows Platform .. function:: win32_edition() - Returns a string representing the current Windows edition. Possible - values include but are not limited to ``'Enterprise'``, ``'IoTUAP'``, - ``'ServerStandard'``, and ``'nanoserver'``. + Returns a string representing the current Windows edition, or ``None`` if the + value cannot be determined. Possible values include but are not limited to + ``'Enterprise'``, ``'IoTUAP'``, ``'ServerStandard'``, and ``'nanoserver'``. .. versionadded:: 3.8 From webhook-mailer at python.org Wed Apr 20 12:48:09 2022 From: webhook-mailer at python.org (miss-islington) Date: Wed, 20 Apr 2022 16:48:09 -0000 Subject: [Python-checkins] bpo-23747: Enhance platform doc, document default behavior (GH-31462) Message-ID: https://github.com/python/cpython/commit/342b4b0648284a24015be411ea6e849dff5f23b2 commit: 342b4b0648284a24015be411ea6e849dff5f23b2 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-20T09:48:04-07:00 summary: bpo-23747: Enhance platform doc, document default behavior (GH-31462) (cherry picked from commit ad3ca17ff5cd63f907430073b52be27695674148) Co-authored-by: slateny <46876382+slateny at users.noreply.github.com> files: M Doc/library/platform.rst diff --git a/Doc/library/platform.rst b/Doc/library/platform.rst index d87205c3d4af8..722a5b59e3e7e 100644 --- a/Doc/library/platform.rst +++ b/Doc/library/platform.rst @@ -139,7 +139,7 @@ Cross Platform .. function:: release() - Returns the system's release, e.g. ``'2.2.0'`` or ``'NT'`` An empty string is + Returns the system's release, e.g. ``'2.2.0'`` or ``'NT'``. An empty string is returned if the value cannot be determined. @@ -176,7 +176,7 @@ Cross Platform Entries which cannot be determined are set to ``''``. .. versionchanged:: 3.3 - Result changed from a tuple to a namedtuple. + Result changed from a tuple to a :func:`~collections.namedtuple`. Java Platform @@ -201,7 +201,9 @@ Windows Platform Get additional version information from the Windows Registry and return a tuple ``(release, version, csd, ptype)`` referring to OS release, version number, - CSD level (service pack) and OS type (multi/single processor). + CSD level (service pack) and OS type (multi/single processor). Values which + cannot be determined are set to the defaults given as parameters (which all + default to an empty string). As a hint: *ptype* is ``'Uniprocessor Free'`` on single processor NT machines and ``'Multiprocessor Free'`` on multi processor machines. The *'Free'* refers @@ -211,9 +213,9 @@ Windows Platform .. function:: win32_edition() - Returns a string representing the current Windows edition. Possible - values include but are not limited to ``'Enterprise'``, ``'IoTUAP'``, - ``'ServerStandard'``, and ``'nanoserver'``. + Returns a string representing the current Windows edition, or ``None`` if the + value cannot be determined. Possible values include but are not limited to + ``'Enterprise'``, ``'IoTUAP'``, ``'ServerStandard'``, and ``'nanoserver'``. .. versionadded:: 3.8 From webhook-mailer at python.org Wed Apr 20 13:26:49 2022 From: webhook-mailer at python.org (vstinner) Date: Wed, 20 Apr 2022 17:26:49 -0000 Subject: [Python-checkins] gh-91731: Replace Py_BUILD_ASSERT() with static_assert() (#91730) Message-ID: https://github.com/python/cpython/commit/7cdaf87ec50f76c934ba651256484c4624b84ef2 commit: 7cdaf87ec50f76c934ba651256484c4624b84ef2 branch: main author: Victor Stinner committer: vstinner date: 2022-04-20T19:26:40+02:00 summary: gh-91731: Replace Py_BUILD_ASSERT() with static_assert() (#91730) Python 3.11 now uses C11 standard which adds static_assert() to . * In pytime.c, replace Py_BUILD_ASSERT() with preprocessor checks on SIZEOF_TIME_T with #error. * On macOS, py_mach_timebase_info() now accepts timebase members with the same size than _PyTime_t. * py_get_monotonic_clock() now saturates GetTickCount64() to _PyTime_MAX: GetTickCount64() is unsigned, whereas _PyTime_t is signed. files: M Modules/_datetimemodule.c M Modules/_pickle.c M Modules/_testcapimodule.c M Modules/posixmodule.c M Modules/pyexpat.c M Objects/longobject.c M Objects/sliceobject.c M Python/fileutils.c M Python/initconfig.c M Python/pytime.c diff --git a/Modules/_datetimemodule.c b/Modules/_datetimemodule.c index fc766c3c94360..a35c72395ec95 100644 --- a/Modules/_datetimemodule.c +++ b/Modules/_datetimemodule.c @@ -6637,19 +6637,19 @@ _datetime_exec(PyObject *module) /* A 4-year cycle has an extra leap day over what we'd get from * pasting together 4 single years. */ - Py_BUILD_ASSERT(DI4Y == 4 * 365 + 1); + static_assert(DI4Y == 4 * 365 + 1, "DI4Y"); assert(DI4Y == days_before_year(4+1)); /* Similarly, a 400-year cycle has an extra leap day over what we'd * get from pasting together 4 100-year cycles. */ - Py_BUILD_ASSERT(DI400Y == 4 * DI100Y + 1); + static_assert(DI400Y == 4 * DI100Y + 1, "DI400Y"); assert(DI400Y == days_before_year(400+1)); /* OTOH, a 100-year cycle has one fewer leap day than we'd get from * pasting together 25 4-year cycles. */ - Py_BUILD_ASSERT(DI100Y == 25 * DI4Y - 1); + static_assert(DI100Y == 25 * DI4Y - 1, "DI100Y"); assert(DI100Y == days_before_year(100+1)); us_per_ms = PyLong_FromLong(1000); diff --git a/Modules/_pickle.c b/Modules/_pickle.c index a5595eb10c8f1..7a8c9ed7243c3 100644 --- a/Modules/_pickle.c +++ b/Modules/_pickle.c @@ -964,7 +964,7 @@ _write_size64(char *out, size_t value) { size_t i; - Py_BUILD_ASSERT(sizeof(size_t) <= 8); + static_assert(sizeof(size_t) <= 8, "size_t is larger than 64-bit"); for (i = 0; i < sizeof(size_t); i++) { out[i] = (unsigned char)((value >> (8 * i)) & 0xff); diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c index 23af0d36b6ceb..6bd73e8f45379 100644 --- a/Modules/_testcapimodule.c +++ b/Modules/_testcapimodule.c @@ -5157,7 +5157,8 @@ dict_get_version(PyObject *self, PyObject *args) version = dict->ma_version_tag; - Py_BUILD_ASSERT(sizeof(unsigned long long) >= sizeof(version)); + static_assert(sizeof(unsigned long long) >= sizeof(version), + "version is larger than unsigned long long"); return PyLong_FromUnsignedLongLong((unsigned long long)version); } diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c index 700cbd2617ad8..345ed710248f9 100644 --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -2381,7 +2381,8 @@ _pystat_fromstructstat(PyObject *module, STRUCT_STAT *st) return NULL; PyStructSequence_SET_ITEM(v, 0, PyLong_FromLong((long)st->st_mode)); - Py_BUILD_ASSERT(sizeof(unsigned long long) >= sizeof(st->st_ino)); + static_assert(sizeof(unsigned long long) >= sizeof(st->st_ino), + "stat.st_ino is larger than unsigned long long"); PyStructSequence_SET_ITEM(v, 1, PyLong_FromUnsignedLongLong(st->st_ino)); #ifdef MS_WINDOWS PyStructSequence_SET_ITEM(v, 2, PyLong_FromUnsignedLong(st->st_dev)); @@ -2396,7 +2397,8 @@ _pystat_fromstructstat(PyObject *module, STRUCT_STAT *st) PyStructSequence_SET_ITEM(v, 4, _PyLong_FromUid(st->st_uid)); PyStructSequence_SET_ITEM(v, 5, _PyLong_FromGid(st->st_gid)); #endif - Py_BUILD_ASSERT(sizeof(long long) >= sizeof(st->st_size)); + static_assert(sizeof(long long) >= sizeof(st->st_size), + "stat.st_size is larger than long long"); PyStructSequence_SET_ITEM(v, 6, PyLong_FromLongLong(st->st_size)); #if defined(HAVE_STAT_TV_NSEC) @@ -13761,10 +13763,12 @@ _Py_COMP_DIAG_POP self->win32_file_index = stat.st_ino; self->got_file_index = 1; } - Py_BUILD_ASSERT(sizeof(unsigned long long) >= sizeof(self->win32_file_index)); + static_assert(sizeof(unsigned long long) >= sizeof(self->win32_file_index), + "DirEntry.win32_file_index is larger than unsigned long long"); return PyLong_FromUnsignedLongLong(self->win32_file_index); #else /* POSIX */ - Py_BUILD_ASSERT(sizeof(unsigned long long) >= sizeof(self->d_ino)); + static_assert(sizeof(unsigned long long) >= sizeof(self->d_ino), + "DirEntry.d_ino is larger than unsigned long long"); return PyLong_FromUnsignedLongLong(self->d_ino); #endif } diff --git a/Modules/pyexpat.c b/Modules/pyexpat.c index 7a26fe24e7592..ad8148adae98b 100644 --- a/Modules/pyexpat.c +++ b/Modules/pyexpat.c @@ -745,6 +745,8 @@ pyexpat_xmlparser_Parse_impl(xmlparseobject *self, PyTypeObject *cls, slen = view.len; } + static_assert(MAX_CHUNK_SIZE <= INT_MAX, + "MAX_CHUNK_SIZE is larger than INT_MAX"); while (slen > MAX_CHUNK_SIZE) { rc = XML_Parse(self->itself, s, MAX_CHUNK_SIZE, 0); if (!rc) @@ -752,7 +754,7 @@ pyexpat_xmlparser_Parse_impl(xmlparseobject *self, PyTypeObject *cls, s += MAX_CHUNK_SIZE; slen -= MAX_CHUNK_SIZE; } - Py_BUILD_ASSERT(MAX_CHUNK_SIZE <= INT_MAX); + assert(slen <= INT_MAX); rc = XML_Parse(self->itself, s, (int)slen, isfinal); diff --git a/Objects/longobject.c b/Objects/longobject.c index 660824f90659f..e805dac1209e7 100644 --- a/Objects/longobject.c +++ b/Objects/longobject.c @@ -771,7 +771,10 @@ _PyLong_Sign(PyObject *vv) static int bit_length_digit(digit x) { - Py_BUILD_ASSERT(PyLong_SHIFT <= sizeof(unsigned long) * 8); + // digit can be larger than unsigned long, but only PyLong_SHIFT bits + // of it will be ever used. + static_assert(PyLong_SHIFT <= sizeof(unsigned long) * 8, + "digit is larger than unsigned long"); return _Py_bit_length((unsigned long)x); } @@ -5647,7 +5650,7 @@ popcount_digit(digit d) { // digit can be larger than uint32_t, but only PyLong_SHIFT bits // of it will be ever used. - Py_BUILD_ASSERT(PyLong_SHIFT <= 32); + static_assert(PyLong_SHIFT <= 32, "digit is larger than uint32_t"); return _Py_popcount32((uint32_t)d); } diff --git a/Objects/sliceobject.c b/Objects/sliceobject.c index 22fb7c61c354f..713829da574d5 100644 --- a/Objects/sliceobject.c +++ b/Objects/sliceobject.c @@ -206,7 +206,8 @@ PySlice_Unpack(PyObject *_r, PySliceObject *r = (PySliceObject*)_r; /* this is harder to get right than you might think */ - Py_BUILD_ASSERT(PY_SSIZE_T_MIN + 1 <= -PY_SSIZE_T_MAX); + static_assert(PY_SSIZE_T_MIN + 1 <= -PY_SSIZE_T_MAX, + "-PY_SSIZE_T_MAX < PY_SSIZE_T_MIN + 1"); if (r->step == Py_None) { *step = 1; diff --git a/Python/fileutils.c b/Python/fileutils.c index 582c6bafd801a..4f7f8944a72da 100644 --- a/Python/fileutils.c +++ b/Python/fileutils.c @@ -956,7 +956,7 @@ static wchar_t * _Py_ConvertWCharForm(const wchar_t *source, Py_ssize_t size, const char *tocode, const char *fromcode) { - Py_BUILD_ASSERT(sizeof(wchar_t) == 4); + static_assert(sizeof(wchar_t) == 4, "wchar_t must be 32-bit"); /* Ensure we won't overflow the size. */ if (size > (PY_SSIZE_T_MAX / (Py_ssize_t)sizeof(wchar_t))) { diff --git a/Python/initconfig.c b/Python/initconfig.c index d2e74f5878a51..729f7f393a39b 100644 --- a/Python/initconfig.c +++ b/Python/initconfig.c @@ -1507,9 +1507,11 @@ config_get_xoption_value(const PyConfig *config, wchar_t *name) static PyStatus config_init_hash_seed(PyConfig *config) { + static_assert(sizeof(_Py_HashSecret_t) == sizeof(_Py_HashSecret.uc), + "_Py_HashSecret_t has wrong size"); + const char *seed_text = config_get_env(config, "PYTHONHASHSEED"); - Py_BUILD_ASSERT(sizeof(_Py_HashSecret_t) == sizeof(_Py_HashSecret.uc)); /* Convert a text seed to a numeric one */ if (seed_text && strcmp(seed_text, "random") != 0) { const char *endptr = seed_text; diff --git a/Python/pytime.c b/Python/pytime.c index aff20c6e5f1d4..f49a25bf7bce7 100644 --- a/Python/pytime.c +++ b/Python/pytime.c @@ -162,12 +162,11 @@ time_t _PyLong_AsTime_t(PyObject *obj) { #if SIZEOF_TIME_T == SIZEOF_LONG_LONG - long long val; - val = PyLong_AsLongLong(obj); + long long val = PyLong_AsLongLong(obj); +#elif SIZEOF_TIME_T <= SIZEOF_LONG + long val = PyLong_AsLong(obj); #else - long val; - Py_BUILD_ASSERT(sizeof(time_t) <= sizeof(long)); - val = PyLong_AsLong(obj); +# error "unsupported time_t size" #endif if (val == -1 && PyErr_Occurred()) { if (PyErr_ExceptionMatches(PyExc_OverflowError)) { @@ -184,9 +183,10 @@ _PyLong_FromTime_t(time_t t) { #if SIZEOF_TIME_T == SIZEOF_LONG_LONG return PyLong_FromLongLong((long long)t); -#else - Py_BUILD_ASSERT(sizeof(time_t) <= sizeof(long)); +#elif SIZEOF_TIME_T <= SIZEOF_LONG return PyLong_FromLong((long)t); +#else +# error "unsupported time_t size" #endif } @@ -386,10 +386,10 @@ _PyTime_t _PyTime_FromSeconds(int seconds) { /* ensure that integer overflow cannot happen, int type should have 32 - bits, whereas _PyTime_t type has at least 64 bits (SEC_TO_MS takes 30 + bits, whereas _PyTime_t type has at least 64 bits (SEC_TO_NS takes 30 bits). */ - Py_BUILD_ASSERT(INT_MAX <= _PyTime_MAX / SEC_TO_NS); - Py_BUILD_ASSERT(INT_MIN >= _PyTime_MIN / SEC_TO_NS); + static_assert(INT_MAX <= _PyTime_MAX / SEC_TO_NS, "_PyTime_t overflow"); + static_assert(INT_MIN >= _PyTime_MIN / SEC_TO_NS, "_PyTime_t underflow"); _PyTime_t t = (_PyTime_t)seconds; assert((t >= 0 && t <= _PyTime_MAX / SEC_TO_NS) @@ -416,7 +416,8 @@ _PyTime_FromNanosecondsObject(_PyTime_t *tp, PyObject *obj) return -1; } - Py_BUILD_ASSERT(sizeof(long long) == sizeof(_PyTime_t)); + static_assert(sizeof(long long) == sizeof(_PyTime_t), + "_PyTime_t is not long long"); long long nsec = PyLong_AsLongLong(obj); if (nsec == -1 && PyErr_Occurred()) { if (PyErr_ExceptionMatches(PyExc_OverflowError)) { @@ -437,7 +438,8 @@ pytime_fromtimespec(_PyTime_t *tp, struct timespec *ts, int raise_exc) { _PyTime_t t, tv_nsec; - Py_BUILD_ASSERT(sizeof(ts->tv_sec) <= sizeof(_PyTime_t)); + static_assert(sizeof(ts->tv_sec) <= sizeof(_PyTime_t), + "timespec.tv_sec is larger than _PyTime_t"); t = (_PyTime_t)ts->tv_sec; int res1 = pytime_mul(&t, SEC_TO_NS); @@ -466,7 +468,8 @@ _PyTime_FromTimespec(_PyTime_t *tp, struct timespec *ts) static int pytime_fromtimeval(_PyTime_t *tp, struct timeval *tv, int raise_exc) { - Py_BUILD_ASSERT(sizeof(tv->tv_sec) <= sizeof(_PyTime_t)); + static_assert(sizeof(tv->tv_sec) <= sizeof(_PyTime_t), + "timeval.tv_sec is larger than _PyTime_t"); _PyTime_t t = (_PyTime_t)tv->tv_sec; int res1 = pytime_mul(&t, SEC_TO_NS); @@ -537,7 +540,8 @@ pytime_from_object(_PyTime_t *tp, PyObject *obj, _PyTime_round_t round, return -1; } - Py_BUILD_ASSERT(sizeof(long long) <= sizeof(_PyTime_t)); + static_assert(sizeof(long long) <= sizeof(_PyTime_t), + "_PyTime_t is smaller than long long"); _PyTime_t ns = (_PyTime_t)sec; if (pytime_mul(&ns, unit_to_ns) < 0) { pytime_overflow(); @@ -589,7 +593,8 @@ PyObject * _PyTime_AsNanosecondsObject(_PyTime_t t) { _PyTime_t ns = pytime_as_nanoseconds(t); - Py_BUILD_ASSERT(sizeof(long long) >= sizeof(_PyTime_t)); + static_assert(sizeof(long long) >= sizeof(_PyTime_t), + "_PyTime_t is larger than long long"); return PyLong_FromLongLong((long long)ns); } @@ -984,15 +989,17 @@ py_mach_timebase_info(_PyTime_t *pnumer, _PyTime_t *pdenom, int raise) _PyTime_t. In practice, timebase uses uint32_t, so casting cannot overflow. At the end, only make sure that the type is uint32_t (_PyTime_t is 64-bit long). */ - Py_BUILD_ASSERT(sizeof(timebase.numer) < sizeof(_PyTime_t)); - Py_BUILD_ASSERT(sizeof(timebase.denom) < sizeof(_PyTime_t)); + static_assert(sizeof(timebase.numer) <= sizeof(_PyTime_t), + "timebase.numer is larger than _PyTime_t"); + static_assert(sizeof(timebase.denom) <= sizeof(_PyTime_t), + "timebase.denom is larger than _PyTime_t"); - /* Make sure that (ticks * timebase.numer) cannot overflow in - _PyTime_MulDiv(), with ticks < timebase.denom. + /* Make sure that _PyTime_MulDiv(ticks, timebase_numer, timebase_denom) + cannot overflow. Known time bases: - * always (1, 1) on Intel + * (1, 1) on Intel * (1000000000, 33333335) or (1000000000, 25000000) on PowerPC None of these time bases can overflow with 64-bit _PyTime_t, but @@ -1019,8 +1026,17 @@ py_get_monotonic_clock(_PyTime_t *tp, _Py_clock_info_t *info, int raise_exc) #if defined(MS_WINDOWS) ULONGLONG ticks = GetTickCount64(); - Py_BUILD_ASSERT(sizeof(ticks) <= sizeof(_PyTime_t)); - _PyTime_t t = (_PyTime_t)ticks; + static_assert(sizeof(ticks) <= sizeof(_PyTime_t), + "ULONGLONG is larger than _PyTime_t"); + _PyTime_t t; + if (ticks <= (ULONGLONG)_PyTime_MAX) { + t = (_PyTime_t)ticks; + } + else { + // GetTickCount64() maximum is larger than _PyTime_t maximum: + // ULONGLONG is unsigned, whereas _PyTime_t is signed. + t = _PyTime_MAX; + } int res = pytime_mul(&t, MS_TO_NS); *tp = t; @@ -1211,7 +1227,8 @@ py_get_win_perf_counter(_PyTime_t *tp, _Py_clock_info_t *info, int raise_exc) /* Make sure that casting LONGLONG to _PyTime_t cannot overflow, both types are signed */ _PyTime_t ticks; - Py_BUILD_ASSERT(sizeof(ticksll) <= sizeof(ticks)); + static_assert(sizeof(ticksll) <= sizeof(ticks), + "LONGLONG is larger than _PyTime_t"); ticks = (_PyTime_t)ticksll; _PyTime_t ns = _PyTime_MulDiv(ticks, SEC_TO_NS, (_PyTime_t)frequency); From webhook-mailer at python.org Wed Apr 20 15:18:36 2022 From: webhook-mailer at python.org (Mariatta) Date: Wed, 20 Apr 2022 19:18:36 -0000 Subject: [Python-checkins] Fix typo in GH Issue template (GH-91759) Message-ID: https://github.com/python/cpython/commit/d608a0126eca0e04e03e2a7c333b26394d657f2f commit: d608a0126eca0e04e03e2a7c333b26394d657f2f branch: main author: Mariatta Wijaya committer: Mariatta date: 2022-04-20T12:18:30-07:00 summary: Fix typo in GH Issue template (GH-91759) Typo in the word enhancement. files: M .github/ISSUE_TEMPLATE/feature.md diff --git a/.github/ISSUE_TEMPLATE/feature.md b/.github/ISSUE_TEMPLATE/feature.md index 75ef03bea675c..635ea43545f49 100644 --- a/.github/ISSUE_TEMPLATE/feature.md +++ b/.github/ISSUE_TEMPLATE/feature.md @@ -10,7 +10,7 @@ labels: "type-feature" **Pitch** -(Explain why this feature or enhacement should be implemented and how it would be used. +(Explain why this feature or enhancement should be implemented and how it would be used. Add examples, if applicable.) **Previous discussion** @@ -25,4 +25,4 @@ labels: "type-feature" \ No newline at end of file +--> From webhook-mailer at python.org Wed Apr 20 17:03:10 2022 From: webhook-mailer at python.org (rhettinger) Date: Wed, 20 Apr 2022 21:03:10 -0000 Subject: [Python-checkins] Minor improvements to grammar and markup. (GH-91762) Message-ID: https://github.com/python/cpython/commit/25e35742ce247d10abf2f59d9fef1d88e4a46fc3 commit: 25e35742ce247d10abf2f59d9fef1d88e4a46fc3 branch: main author: Raymond Hettinger committer: rhettinger date: 2022-04-20T16:02:47-05:00 summary: Minor improvements to grammar and markup. (GH-91762) files: M Doc/library/argparse.rst diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst index 7c206370f5432..17570efb27081 100644 --- a/Doc/library/argparse.rst +++ b/Doc/library/argparse.rst @@ -34,20 +34,20 @@ around an instance of :class:`argparse.ArgumentParser`. It is a container for argument specifications and has options that apply the parser as whole:: parser = argparse.ArgumentParser( - prog = 'ProgramName', - description = 'What the program does', - epilog = 'Text at the bottom of help') + prog = 'ProgramName', + description = 'What the program does', + epilog = 'Text at the bottom of help') -The :func:`ArgumentParser.add_argument` function attaches individual argument +The :meth:`ArgumentParser.add_argument` method attaches individual argument specifications to the parser. It supports positional arguments, options that accept values, and on/off flags:: parser.add_argument('filename') # positional argument - parser.add_argument('-c', '--count') # option that takes value + parser.add_argument('-c', '--count') # option that takes a value parser.add_argument('-v', '--verbose', action='store_true') # on/off flag -The :func:`ArgumentParser.parse_args` function runs the parser and puts +The :meth:`ArgumentParser.parse_args` method runs the parser and places the extracted data in a :class:`argparse.Namespace` object:: args = parser.parse_args() @@ -61,15 +61,15 @@ Quick Links for add_argument() Name Description Values ====================== =========================================================== ========================================================================================================================== action_ Specify how an argument should be handled ``'store'``, ``'store_const'``, ``'store_true'``, ``'append'``, ``'append_const'``, ``'count'``, ``'help'``, ``'version'`` -choices_ Limit values to specific set of choices ``['foo', 'bar']``, ``range(1, 10)``, or an object that supports ``in`` operator +choices_ Limit values to a specific set of choices ``['foo', 'bar']``, ``range(1, 10)``, or :class:`~collections.abc.Container` instance const_ Store a constant value -default_ Default value used when an argument is not provided -dest_ Specify the attribute name in result namespace +default_ Default value used when an argument is not provided Defaults to *None* +dest_ Specify the attribute name used in the result namespace help_ Help message for an argument metavar_ Alternate display name for the argument as shown in help -nargs_ Number of times the argument can be used ``N`` (:class:`int`), ``'?'``, ``'*'``, ``'+'``, ``argparse.REMAINDER`` -required_ Indicate whether an optional argument is required or not ``True``, ``False`` -type_ Automatically convert an argument to the given type :class:`int`, :class:`float`, ``argparse.FileType('w')``, or any callable function +nargs_ Number of times the argument can be used :class:`int`, ``'?'``, ``'*'``, ``'+'``, or ``argparse.REMAINDER`` +required_ Indicate whether an argument is required or optional ``True`` or ``False`` +type_ Automatically convert an argument to the given type :class:`int`, :class:`float`, ``argparse.FileType('w')``, or callable function ====================== =========================================================== ========================================================================================================================== From webhook-mailer at python.org Wed Apr 20 17:46:50 2022 From: webhook-mailer at python.org (miss-islington) Date: Wed, 20 Apr 2022 21:46:50 -0000 Subject: [Python-checkins] bpo-40859: Update Windows build to use xz-5.2.5 (GH-20622) Message-ID: https://github.com/python/cpython/commit/1fce689c2acf4df371c97851c3a82f54f656ffff commit: 1fce689c2acf4df371c97851c3a82f54f656ffff branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-20T14:46:14-07:00 summary: bpo-40859: Update Windows build to use xz-5.2.5 (GH-20622) Co-authored-by: Ma Lin (cherry picked from commit 73af4b0264aeb866536b01a132f8c26a5808d64d) Co-authored-by: Steve Dower files: A Misc/NEWS.d/next/Windows/2020-06-04-10-42-04.bpo-40859.isKSw7.rst M PCbuild/get_externals.bat M PCbuild/liblzma.vcxproj M PCbuild/liblzma.vcxproj.filters M PCbuild/python.props diff --git a/Misc/NEWS.d/next/Windows/2020-06-04-10-42-04.bpo-40859.isKSw7.rst b/Misc/NEWS.d/next/Windows/2020-06-04-10-42-04.bpo-40859.isKSw7.rst new file mode 100644 index 0000000000000..ef4c727ad2866 --- /dev/null +++ b/Misc/NEWS.d/next/Windows/2020-06-04-10-42-04.bpo-40859.isKSw7.rst @@ -0,0 +1 @@ +Update Windows build to use xz-5.2.5 diff --git a/PCbuild/get_externals.bat b/PCbuild/get_externals.bat index dc5c909de7441..ee79addd44d02 100644 --- a/PCbuild/get_externals.bat +++ b/PCbuild/get_externals.bat @@ -58,7 +58,7 @@ set libraries=%libraries% sqlite-3.37.2.0 if NOT "%IncludeTkinterSrc%"=="false" set libraries=%libraries% tcl-core-8.6.12.0 if NOT "%IncludeTkinterSrc%"=="false" set libraries=%libraries% tk-8.6.12.0 if NOT "%IncludeTkinterSrc%"=="false" set libraries=%libraries% tix-8.4.3.6 -set libraries=%libraries% xz-5.2.2 +set libraries=%libraries% xz-5.2.5 set libraries=%libraries% zlib-1.2.12 for %%e in (%libraries%) do ( diff --git a/PCbuild/liblzma.vcxproj b/PCbuild/liblzma.vcxproj index a6bd59ec0baa3..4dd42ab98a975 100644 --- a/PCbuild/liblzma.vcxproj +++ b/PCbuild/liblzma.vcxproj @@ -92,7 +92,7 @@ WIN32;HAVE_CONFIG_H;_LIB;%(PreprocessorDefinitions) - $(lzmaDir)windows;$(lzmaDir)src/liblzma/common;$(lzmaDir)src/common;$(lzmaDir)src/liblzma/api;$(lzmaDir)src/liblzma/check;$(lzmaDir)src/liblzma/delta;$(lzmaDir)src/liblzma/lz;$(lzmaDir)src/liblzma/lzma;$(lzmaDir)src/liblzma/rangecoder;$(lzmaDir)src/liblzma/simple;%(AdditionalIncludeDirectories) + $(lzmaDir)windows/vs2019;$(lzmaDir)src/liblzma/common;$(lzmaDir)src/common;$(lzmaDir)src/liblzma/api;$(lzmaDir)src/liblzma/check;$(lzmaDir)src/liblzma/delta;$(lzmaDir)src/liblzma/lz;$(lzmaDir)src/liblzma/lzma;$(lzmaDir)src/liblzma/rangecoder;$(lzmaDir)src/liblzma/simple;%(AdditionalIncludeDirectories) 4028;4113;4133;4244;4267;4996;%(DisableSpecificWarnings) @@ -238,7 +238,7 @@ - + diff --git a/PCbuild/liblzma.vcxproj.filters b/PCbuild/liblzma.vcxproj.filters index 3f58351fa9edb..ebe2a7d5fa9e5 100644 --- a/PCbuild/liblzma.vcxproj.filters +++ b/PCbuild/liblzma.vcxproj.filters @@ -428,7 +428,7 @@ Header Files - + Header Files diff --git a/PCbuild/python.props b/PCbuild/python.props index 7b42037b7cdf8..c451429da2c82 100644 --- a/PCbuild/python.props +++ b/PCbuild/python.props @@ -59,7 +59,7 @@ $(ExternalsDir)\ $(ExternalsDir)sqlite-3.37.2.0\ $(ExternalsDir)bzip2-1.0.8\ - $(ExternalsDir)xz-5.2.2\ + $(ExternalsDir)xz-5.2.5\ $(ExternalsDir)libffi-3.3.0\ $(ExternalsDir)libffi-3.3.0\$(ArchName)\ $(libffiOutDir)include From webhook-mailer at python.org Wed Apr 20 20:51:26 2022 From: webhook-mailer at python.org (vstinner) Date: Thu, 21 Apr 2022 00:51:26 -0000 Subject: [Python-checkins] gh-89653: PEP 670: Convert unicodeobject.h macros to functions (#91705) Message-ID: https://github.com/python/cpython/commit/4e52c66f6940a63ef7a62faf8ce32a980ac5aa2c commit: 4e52c66f6940a63ef7a62faf8ce32a980ac5aa2c branch: main author: Victor Stinner committer: vstinner date: 2022-04-21T02:51:17+02:00 summary: gh-89653: PEP 670: Convert unicodeobject.h macros to functions (#91705) Convert unicodeobject.h macros to static inline functions: * PyUnicode_CHECK_INTERNED() * PyUnicode_DATA(), _PyUnicode_COMPACT_DATA(), _PyUnicode_NONCOMPACT_DATA() * PyUnicode_GET_LENGTH() * PyUnicode_IS_ASCII() * PyUnicode_IS_COMPACT() * PyUnicode_IS_COMPACT_ASCII() * PyUnicode_IS_READY() Reorder functions to declare functions before their first usage. Static inline functions are wrapped by macros which casts arguments with _PyObject_CAST() to prevent introducing new compiler warnings when passing "const PyObject*". files: M Include/cpython/unicodeobject.h diff --git a/Include/cpython/unicodeobject.h b/Include/cpython/unicodeobject.h index 69e4abfb5c442..9a993a1c7ff0c 100644 --- a/Include/cpython/unicodeobject.h +++ b/Include/cpython/unicodeobject.h @@ -287,37 +287,80 @@ PyAPI_FUNC(int) _PyUnicode_CheckConsistency( #define SSTATE_INTERNED_IMMORTAL 2 /* Use only if you know it's a string */ -#define PyUnicode_CHECK_INTERNED(op) \ - (_PyASCIIObject_CAST(op)->state.interned) +static inline unsigned int PyUnicode_CHECK_INTERNED(PyObject *op) { + return _PyASCIIObject_CAST(op)->state.interned; +} +#define PyUnicode_CHECK_INTERNED(op) PyUnicode_CHECK_INTERNED(_PyObject_CAST(op)) + +/* Fast check to determine whether an object is ready. Equivalent to: + PyUnicode_IS_COMPACT(op) || _PyUnicodeObject_CAST(op)->data.any */ +static inline unsigned int PyUnicode_IS_READY(PyObject *op) { + return _PyASCIIObject_CAST(op)->state.ready; +} +#define PyUnicode_IS_READY(op) PyUnicode_IS_READY(_PyObject_CAST(op)) /* Return true if the string contains only ASCII characters, or 0 if not. The string may be compact (PyUnicode_IS_COMPACT_ASCII) or not, but must be ready. */ -#define PyUnicode_IS_ASCII(op) \ - (assert(PyUnicode_IS_READY(op)), \ - _PyASCIIObject_CAST(op)->state.ascii) +static inline unsigned int PyUnicode_IS_ASCII(PyObject *op) { + assert(PyUnicode_IS_READY(op)); + return _PyASCIIObject_CAST(op)->state.ascii; +} +#define PyUnicode_IS_ASCII(op) PyUnicode_IS_ASCII(_PyObject_CAST(op)) /* Return true if the string is compact or 0 if not. No type checks or Ready calls are performed. */ -#define PyUnicode_IS_COMPACT(op) \ - (_PyASCIIObject_CAST(op)->state.compact) +static inline unsigned int PyUnicode_IS_COMPACT(PyObject *op) { + return _PyASCIIObject_CAST(op)->state.compact; +} +#define PyUnicode_IS_COMPACT(op) PyUnicode_IS_COMPACT(_PyObject_CAST(op)) /* Return true if the string is a compact ASCII string (use PyASCIIObject structure), or 0 if not. No type checks or Ready calls are performed. */ -#define PyUnicode_IS_COMPACT_ASCII(op) \ - (_PyASCIIObject_CAST(op)->state.ascii && PyUnicode_IS_COMPACT(op)) +static inline int PyUnicode_IS_COMPACT_ASCII(PyObject *op) { + return (_PyASCIIObject_CAST(op)->state.ascii && PyUnicode_IS_COMPACT(op)); +} +#define PyUnicode_IS_COMPACT_ASCII(op) PyUnicode_IS_COMPACT_ASCII(_PyObject_CAST(op)) enum PyUnicode_Kind { /* String contains only wstr byte characters. This is only possible when the string was created with a legacy API and _PyUnicode_Ready() has not been called yet. */ PyUnicode_WCHAR_KIND = 0, -/* Return values of the PyUnicode_KIND() macro: */ +/* Return values of the PyUnicode_KIND() function: */ PyUnicode_1BYTE_KIND = 1, PyUnicode_2BYTE_KIND = 2, PyUnicode_4BYTE_KIND = 4 }; +/* Return one of the PyUnicode_*_KIND values defined above. */ +#define PyUnicode_KIND(op) \ + (assert(PyUnicode_Check(op)), \ + assert(PyUnicode_IS_READY(op)), \ + ((PyASCIIObject *)(op))->state.kind) + +/* Return a void pointer to the raw unicode buffer. */ +static inline void* _PyUnicode_COMPACT_DATA(PyObject *op) { + if (PyUnicode_IS_ASCII(op)) { + return (void*)(_PyASCIIObject_CAST(op) + 1); + } + return (void*)(_PyCompactUnicodeObject_CAST(op) + 1); +} + +static inline void* _PyUnicode_NONCOMPACT_DATA(PyObject *op) { + void *data = _PyUnicodeObject_CAST(op)->data.any; + assert(data != NULL); + return data; +} + +static inline void* PyUnicode_DATA(PyObject *op) { + if (PyUnicode_IS_COMPACT(op)) { + return _PyUnicode_COMPACT_DATA(op); + } + return _PyUnicode_NONCOMPACT_DATA(op); +} +#define PyUnicode_DATA(op) PyUnicode_DATA(_PyObject_CAST(op)) + /* Return pointers to the canonical representation cast to unsigned char, Py_UCS2, or Py_UCS4 for direct character access. No checks are performed, use PyUnicode_KIND() before to ensure @@ -327,24 +370,14 @@ enum PyUnicode_Kind { #define PyUnicode_2BYTE_DATA(op) ((Py_UCS2*)PyUnicode_DATA(op)) #define PyUnicode_4BYTE_DATA(op) ((Py_UCS4*)PyUnicode_DATA(op)) -/* Return one of the PyUnicode_*_KIND values defined above. */ -#define PyUnicode_KIND(op) \ - (assert(PyUnicode_IS_READY(op)), \ - _PyASCIIObject_CAST(op)->state.kind) - -/* Return a void pointer to the raw unicode buffer. */ -#define _PyUnicode_COMPACT_DATA(op) \ - (PyUnicode_IS_ASCII(op) ? \ - ((void*)(_PyASCIIObject_CAST(op) + 1)) : \ - ((void*)(_PyCompactUnicodeObject_CAST(op) + 1))) - -#define _PyUnicode_NONCOMPACT_DATA(op) \ - (assert(_PyUnicodeObject_CAST(op)->data.any), \ - (_PyUnicodeObject_CAST(op)->data.any)) - -#define PyUnicode_DATA(op) \ - (PyUnicode_IS_COMPACT(op) ? _PyUnicode_COMPACT_DATA(op) : \ - _PyUnicode_NONCOMPACT_DATA(op)) +/* Returns the length of the unicode string. The caller has to make sure that + the string has it's canonical representation set before calling + this function. Call PyUnicode_(FAST_)Ready to ensure that. */ +static inline Py_ssize_t PyUnicode_GET_LENGTH(PyObject *op) { + assert(PyUnicode_IS_READY(op)); + return _PyASCIIObject_CAST(op)->length; +} +#define PyUnicode_GET_LENGTH(op) PyUnicode_GET_LENGTH(_PyObject_CAST(op)) /* In the access macros below, "kind" may be evaluated more than once. All other macro parameters are evaluated exactly once, so it is safe @@ -400,19 +433,6 @@ enum PyUnicode_Kind { ) \ )) -/* Returns the length of the unicode string. The caller has to make sure that - the string has it's canonical representation set before calling - this macro. Call PyUnicode_(FAST_)Ready to ensure that. */ -#define PyUnicode_GET_LENGTH(op) \ - (assert(PyUnicode_IS_READY(op)), \ - _PyASCIIObject_CAST(op)->length) - - -/* Fast check to determine whether an object is ready. Equivalent to - PyUnicode_IS_COMPACT(op) || _PyUnicodeObject_CAST(op)->data.any */ - -#define PyUnicode_IS_READY(op) (_PyASCIIObject_CAST(op)->state.ready) - /* PyUnicode_READY() does less work than _PyUnicode_Ready() in the best case. If the canonical representation is not yet set, it will still call _PyUnicode_Ready(). From webhook-mailer at python.org Wed Apr 20 21:11:00 2022 From: webhook-mailer at python.org (vstinner) Date: Thu, 21 Apr 2022 01:11:00 -0000 Subject: [Python-checkins] Revert "gh-85567: Register a cleanup function to close files for FileType objects in argparse (#32257)" (#91771) Message-ID: https://github.com/python/cpython/commit/c77953b23e4b864129edf7983eaa6a0d22414ec6 commit: c77953b23e4b864129edf7983eaa6a0d22414ec6 branch: main author: Victor Stinner committer: vstinner date: 2022-04-21T03:10:51+02:00 summary: Revert "gh-85567: Register a cleanup function to close files for FileType objects in argparse (#32257)" (#91771) This reverts commit 328dbc051f84bd5fdf61101bb4fa61d85f8b7feb. files: D Misc/NEWS.d/next/Library/2022-04-02-14-40-53.bpo-41395.Y1ZVvT.rst M Lib/argparse.py M Misc/ACKS diff --git a/Lib/argparse.py b/Lib/argparse.py index 881dfda6d4d932..429a72ab7841e7 100644 --- a/Lib/argparse.py +++ b/Lib/argparse.py @@ -84,7 +84,7 @@ 'ZERO_OR_MORE', ] -import atexit as _atexit + import os as _os import re as _re import sys as _sys @@ -1268,12 +1268,8 @@ def __call__(self, string): # all other arguments are used as file names try: - fh = open(string, self._mode, self._bufsize, self._encoding, self._errors) - - # Register cleanup function to close file - _atexit.register(fh.close) - - return fh + return open(string, self._mode, self._bufsize, self._encoding, + self._errors) except OSError as e: args = {'filename': string, 'error': e} message = _("can't open '%(filename)s': %(error)s") diff --git a/Misc/ACKS b/Misc/ACKS index a1df84c0d6779e..5e66a2e757adf6 100644 --- a/Misc/ACKS +++ b/Misc/ACKS @@ -313,7 +313,6 @@ Nicolas Chauvat Jerry Chen Michael Chermside Ingrid Cheung -Adam Chhina Terry Chia Albert Chin-A-Young Adal Chiriliuc diff --git a/Misc/NEWS.d/next/Library/2022-04-02-14-40-53.bpo-41395.Y1ZVvT.rst b/Misc/NEWS.d/next/Library/2022-04-02-14-40-53.bpo-41395.Y1ZVvT.rst deleted file mode 100644 index 5358b0e71715e2..00000000000000 --- a/Misc/NEWS.d/next/Library/2022-04-02-14-40-53.bpo-41395.Y1ZVvT.rst +++ /dev/null @@ -1,3 +0,0 @@ -FileType objects from argparse may not be closed and lead to -ResourceWarning. Register a file.close function with atexit for FileType -objects to ensure they are closed. Patch Contributed by Adam Chhina. From webhook-mailer at python.org Wed Apr 20 21:15:30 2022 From: webhook-mailer at python.org (vstinner) Date: Thu, 21 Apr 2022 01:15:30 -0000 Subject: [Python-checkins] gh-90623: signal.raise_signal() calls PyErr_CheckSignals() (#91756) Message-ID: https://github.com/python/cpython/commit/031f1e6040d1bb0454bd6da417fd9e38aad4fc89 commit: 031f1e6040d1bb0454bd6da417fd9e38aad4fc89 branch: main author: Victor Stinner committer: vstinner date: 2022-04-21T03:14:57+02:00 summary: gh-90623: signal.raise_signal() calls PyErr_CheckSignals() (#91756) signal.raise_signal() and os.kill() now call PyErr_CheckSignals() to check immediately for pending signals. files: A Misc/NEWS.d/next/Library/2022-04-20-18-47-27.gh-issue-90623.5fROpX.rst M Modules/posixmodule.c M Modules/signalmodule.c diff --git a/Misc/NEWS.d/next/Library/2022-04-20-18-47-27.gh-issue-90623.5fROpX.rst b/Misc/NEWS.d/next/Library/2022-04-20-18-47-27.gh-issue-90623.5fROpX.rst new file mode 100644 index 00000000000000..566cf35c328689 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-20-18-47-27.gh-issue-90623.5fROpX.rst @@ -0,0 +1,2 @@ +:func:`signal.raise_signal` and :func:`os.kill` now check immediately for +pending signals. Patch by Victor Stinner. diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c index 345ed710248f9b..a9132a78994ce4 100644 --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -7929,8 +7929,17 @@ os_kill_impl(PyObject *module, pid_t pid, Py_ssize_t signal) return NULL; } #ifndef MS_WINDOWS - if (kill(pid, (int)signal) == -1) + if (kill(pid, (int)signal) == -1) { return posix_error(); + } + + // Check immediately if the signal was sent to the current process. + // Don't micro-optimize pid == getpid(), since PyErr_SetString() check + // is cheap. + if (PyErr_CheckSignals()) { + return NULL; + } + Py_RETURN_NONE; #else /* !MS_WINDOWS */ PyObject *result; diff --git a/Modules/signalmodule.c b/Modules/signalmodule.c index 1ee5c669df015f..02c58ff538ecdc 100644 --- a/Modules/signalmodule.c +++ b/Modules/signalmodule.c @@ -481,6 +481,13 @@ signal_raise_signal_impl(PyObject *module, int signalnum) if (err) { return PyErr_SetFromErrno(PyExc_OSError); } + + // If the current thread can handle signals, handle immediately + // the raised signal. + if (PyErr_CheckSignals()) { + return NULL; + } + Py_RETURN_NONE; } From webhook-mailer at python.org Wed Apr 20 21:45:29 2022 From: webhook-mailer at python.org (rhettinger) Date: Thu, 21 Apr 2022 01:45:29 -0000 Subject: [Python-checkins] bpo-42066: CookieJar cookies should not be sorted (GH-22745) Message-ID: https://github.com/python/cpython/commit/615b24c80b0bbbc7b70aa1e9c28f9c4463c8c1ed commit: 615b24c80b0bbbc7b70aa1e9c28f9c4463c8c1ed branch: main author: Iman Kermani <55282537+IKermani at users.noreply.github.com> committer: rhettinger date: 2022-04-20T20:45:24-05:00 summary: bpo-42066: CookieJar cookies should not be sorted (GH-22745) files: A Misc/NEWS.d/next/Library/2020-10-19-08-50-41.bpo-42066.DsB-R6.rst M Lib/http/cookiejar.py M Lib/test/test_http_cookiejar.py diff --git a/Lib/http/cookiejar.py b/Lib/http/cookiejar.py index 136a1f16ffe63d..f19a366496a21a 100644 --- a/Lib/http/cookiejar.py +++ b/Lib/http/cookiejar.py @@ -1224,14 +1224,9 @@ def path_return_ok(self, path, request): _debug(" %s does not path-match %s", req_path, path) return False -def vals_sorted_by_key(adict): - keys = sorted(adict.keys()) - return map(adict.get, keys) - def deepvalues(mapping): - """Iterates over nested mapping, depth-first, in sorted order by key.""" - values = vals_sorted_by_key(mapping) - for obj in values: + """Iterates over nested mapping, depth-first""" + for obj in list(mapping.values()): mapping = False try: obj.items diff --git a/Lib/test/test_http_cookiejar.py b/Lib/test/test_http_cookiejar.py index 126ce4fc83f0d1..ad2b121fdaf5a1 100644 --- a/Lib/test/test_http_cookiejar.py +++ b/Lib/test/test_http_cookiejar.py @@ -1293,11 +1293,11 @@ def test_Cookie_iterator(self): r'port="90,100, 80,8080"; ' r'max-age=100; Comment = "Just kidding! (\"|\\\\) "') - versions = [1, 1, 1, 0, 1] - names = ["bang", "foo", "foo", "spam", "foo"] - domains = [".sol.no", "blah.spam.org", "www.acme.com", - "www.acme.com", "www.acme.com"] - paths = ["/", "/", "/", "/blah", "/blah/"] + versions = [1, 0, 1, 1, 1] + names = ["foo", "spam", "foo", "foo", "bang"] + domains = ["blah.spam.org", "www.acme.com", "www.acme.com", + "www.acme.com", ".sol.no"] + paths = ["/", "/blah", "/blah/", "/", "/"] for i in range(4): i = 0 diff --git a/Misc/NEWS.d/next/Library/2020-10-19-08-50-41.bpo-42066.DsB-R6.rst b/Misc/NEWS.d/next/Library/2020-10-19-08-50-41.bpo-42066.DsB-R6.rst new file mode 100644 index 00000000000000..f3de85461fbc0d --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-10-19-08-50-41.bpo-42066.DsB-R6.rst @@ -0,0 +1,2 @@ +Fix cookies getting sorted in :func:`CookieJar.__iter__` which is an extra behavior and not mentioned in RFC 2965 or Netscape cookie protocol. +Now the cookies in ``CookieJar`` follows the order of the ``Set-Cookie`` header. Patch by Iman Kermani. From webhook-mailer at python.org Wed Apr 20 21:53:23 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Thu, 21 Apr 2022 01:53:23 -0000 Subject: [Python-checkins] [3.10] build(deps): bump actions/cache from 2.1.7 to 3.0.1 (GH-32228) (#91746) Message-ID: https://github.com/python/cpython/commit/d8d5db5f1024f2f902c489be74d4ae713c59c748 commit: d8d5db5f1024f2f902c489be74d4ae713c59c748 branch: 3.10 author: Jelle Zijlstra committer: JelleZijlstra date: 2022-04-20T18:53:05-07:00 summary: [3.10] build(deps): bump actions/cache from 2.1.7 to 3.0.1 (GH-32228) (#91746) Bumps [actions/cache](https://github.com/actions/cache) from 2.1.7 to 3.0.1. - [Release notes](https://github.com/actions/cache/releases) - [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md) - [Commits](https://github.com/actions/cache/compare/v2.1.7...v3.0.1) --- updated-dependencies: - dependency-name: actions/cache dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jelle Zijlstra . (cherry picked from commit 3ace1034b8202bc7034e15f34561725934f04ff6) files: M .github/workflows/build.yml diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index ce300ebe59bfa..adbe2fe1d248c 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -196,7 +196,7 @@ jobs: echo "LD_LIBRARY_PATH=${GITHUB_WORKSPACE}/multissl/openssl/${OPENSSL_VER}/lib" >> $GITHUB_ENV - name: 'Restore OpenSSL build' id: cache-openssl - uses: actions/cache at v2.1.4 + uses: actions/cache at v3.0.1 with: path: ./multissl/openssl/${{ env.OPENSSL_VER }} key: ${{ runner.os }}-multissl-openssl-${{ env.OPENSSL_VER }} @@ -244,7 +244,7 @@ jobs: echo "LD_LIBRARY_PATH=${GITHUB_WORKSPACE}/multissl/openssl/${OPENSSL_VER}/lib" >> $GITHUB_ENV - name: 'Restore OpenSSL build' id: cache-openssl - uses: actions/cache at v2.1.4 + uses: actions/cache at v3.0.1 with: path: ./multissl/openssl/${{ env.OPENSSL_VER }} key: ${{ runner.os }}-multissl-openssl-${{ env.OPENSSL_VER }} From webhook-mailer at python.org Wed Apr 20 21:53:23 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Thu, 21 Apr 2022 01:53:23 -0000 Subject: [Python-checkins] [3.9] build(deps): bump actions/cache from 2.1.7 to 3.0.1 (GH-32228) (#91747) Message-ID: https://github.com/python/cpython/commit/f07d9baea65e6724820ab409e9fd38fd5d07c876 commit: f07d9baea65e6724820ab409e9fd38fd5d07c876 branch: 3.9 author: Jelle Zijlstra committer: JelleZijlstra date: 2022-04-20T18:53:17-07:00 summary: [3.9] build(deps): bump actions/cache from 2.1.7 to 3.0.1 (GH-32228) (#91747) Bumps [actions/cache](https://github.com/actions/cache) from 2.1.7 to 3.0.1. - [Release notes](https://github.com/actions/cache/releases) - [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md) - [Commits](https://github.com/actions/cache/compare/v2.1.7...v3.0.1) --- updated-dependencies: - dependency-name: actions/cache dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jelle Zijlstra . (cherry picked from commit 3ace1034b8202bc7034e15f34561725934f04ff6) files: M .github/workflows/build.yml diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 13181919efe07..7a3f7b35a612c 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -178,7 +178,7 @@ jobs: echo "LD_LIBRARY_PATH=${GITHUB_WORKSPACE}/multissl/openssl/${OPENSSL_VER}/lib" >> $GITHUB_ENV - name: 'Restore OpenSSL build' id: cache-openssl - uses: actions/cache at v2.1.3 + uses: actions/cache at v3.0.1 with: path: ./multissl/openssl/${{ env.OPENSSL_VER }} key: ${{ runner.os }}-multissl-openssl-${{ env.OPENSSL_VER }} @@ -224,7 +224,7 @@ jobs: echo "LD_LIBRARY_PATH=${GITHUB_WORKSPACE}/multissl/openssl/${OPENSSL_VER}/lib" >> $GITHUB_ENV - name: 'Restore OpenSSL build' id: cache-openssl - uses: actions/cache at v2.1.4 + uses: actions/cache at v3.0.1 with: path: ./multissl/openssl/${{ env.OPENSSL_VER }} key: ${{ runner.os }}-multissl-openssl-${{ env.OPENSSL_VER }} From webhook-mailer at python.org Wed Apr 20 21:53:35 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Thu, 21 Apr 2022 01:53:35 -0000 Subject: [Python-checkins] [3.10] build(deps): bump actions/upload-artifact from 2.3.1 to 3 (GH-32227) (#91748) Message-ID: https://github.com/python/cpython/commit/9f8b9a3506fefeba01a18eda0e06c037393be4bf commit: 9f8b9a3506fefeba01a18eda0e06c037393be4bf branch: 3.10 author: Jelle Zijlstra committer: JelleZijlstra date: 2022-04-20T18:53:31-07:00 summary: [3.10] build(deps): bump actions/upload-artifact from 2.3.1 to 3 (GH-32227) (#91748) Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 2.3.1 to 3. - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](https://github.com/actions/upload-artifact/compare/v2.3.1...v3) --- updated-dependencies: - dependency-name: actions/upload-artifact dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jelle Zijlstra . (cherry picked from commit b8812c9ca3f6864a233574001e16f1b9e92daf6e) files: M .github/workflows/doc.yml diff --git a/.github/workflows/doc.yml b/.github/workflows/doc.yml index 970f5af2d9bed..a57552b07905f 100644 --- a/.github/workflows/doc.yml +++ b/.github/workflows/doc.yml @@ -39,7 +39,7 @@ jobs: - name: 'Build documentation' run: xvfb-run make -C Doc/ PYTHON=../python SPHINXOPTS="-q -W --keep-going -j4" doctest html suspicious - name: 'Upload' - uses: actions/upload-artifact at v2.2.2 + uses: actions/upload-artifact at v3 with: name: doc-html path: Doc/build/html From webhook-mailer at python.org Wed Apr 20 21:53:47 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Thu, 21 Apr 2022 01:53:47 -0000 Subject: [Python-checkins] [3.9] build(deps): bump actions/upload-artifact from 2.3.1 to 3 (GH-32227) (#91749) Message-ID: https://github.com/python/cpython/commit/2f75d43f1e8954df53ba4a7ad950ed37352a75fc commit: 2f75d43f1e8954df53ba4a7ad950ed37352a75fc branch: 3.9 author: Jelle Zijlstra committer: JelleZijlstra date: 2022-04-20T18:53:43-07:00 summary: [3.9] build(deps): bump actions/upload-artifact from 2.3.1 to 3 (GH-32227) (#91749) Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 2.3.1 to 3. - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](https://github.com/actions/upload-artifact/compare/v2.3.1...v3) --- updated-dependencies: - dependency-name: actions/upload-artifact dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jelle Zijlstra . (cherry picked from commit b8812c9ca3f6864a233574001e16f1b9e92daf6e) files: M .github/workflows/doc.yml diff --git a/.github/workflows/doc.yml b/.github/workflows/doc.yml index 537478a2850f4..81e68d6a823d6 100644 --- a/.github/workflows/doc.yml +++ b/.github/workflows/doc.yml @@ -36,7 +36,7 @@ jobs: - name: 'Build documentation' run: xvfb-run make -C Doc/ PYTHON=../python SPHINXOPTS="-q -W -j4" doctest suspicious html - name: 'Upload' - uses: actions/upload-artifact at v2.2.2 + uses: actions/upload-artifact at v3 with: name: doc-html path: Doc/build/html From webhook-mailer at python.org Thu Apr 21 02:06:39 2022 From: webhook-mailer at python.org (sweeneyde) Date: Thu, 21 Apr 2022 06:06:39 -0000 Subject: [Python-checkins] gh-91636: Don't clear required fields of function objects (GH-91651) Message-ID: https://github.com/python/cpython/commit/f2b4e458b3327130e46edb4efe8e1847de09efc5 commit: f2b4e458b3327130e46edb4efe8e1847de09efc5 branch: main author: Dennis Sweeney <36520290+sweeneyde at users.noreply.github.com> committer: sweeneyde <36520290+sweeneyde at users.noreply.github.com> date: 2022-04-21T02:06:35-04:00 summary: gh-91636: Don't clear required fields of function objects (GH-91651) files: A Misc/NEWS.d/next/Core and Builtins/2022-04-18-02-45-40.gh-issue-91636.6DFdy_.rst M Lib/test/test_gc.py M Objects/funcobject.c diff --git a/Lib/test/test_gc.py b/Lib/test/test_gc.py index ce04042679bbc..dbbd67b4fc88a 100644 --- a/Lib/test/test_gc.py +++ b/Lib/test/test_gc.py @@ -227,6 +227,73 @@ def test_function(self): del d self.assertEqual(gc.collect(), 2) + def test_function_tp_clear_leaves_consistent_state(self): + # https://github.com/python/cpython/issues/91636 + code = """if 1: + + import gc + import weakref + + class LateFin: + __slots__ = ('ref',) + + def __del__(self): + + # 8. Now `latefin`'s finalizer is called. Here we + # obtain a reference to `func`, which is currently + # undergoing `tp_clear`. + global func + func = self.ref() + + class Cyclic(tuple): + __slots__ = () + + # 4. The finalizers of all garbage objects are called. In + # this case this is only us as `func` doesn't have a + # finalizer. + def __del__(self): + + # 5. Create a weakref to `func` now. If we had created + # it earlier, it would have been cleared by the + # garbage collector before calling the finalizers. + self[1].ref = weakref.ref(self[0]) + + # 6. Drop the global reference to `latefin`. The only + # remaining reference is the one we have. + global latefin + del latefin + + # 7. Now `func` is `tp_clear`-ed. This drops the last + # reference to `Cyclic`, which gets `tp_dealloc`-ed. + # This drops the last reference to `latefin`. + + latefin = LateFin() + def func(): + pass + cyc = tuple.__new__(Cyclic, (func, latefin)) + + # 1. Create a reference cycle of `cyc` and `func`. + func.__module__ = cyc + + # 2. Make the cycle unreachable, but keep the global reference + # to `latefin` so that it isn't detected as garbage. This + # way its finalizer will not be called immediately. + del func, cyc + + # 3. Invoke garbage collection, + # which will find `cyc` and `func` as garbage. + gc.collect() + + # 9. Previously, this would crash because `func_qualname` + # had been NULL-ed out by func_clear(). + print(f"{func=}") + """ + # We're mostly just checking that this doesn't crash. + rc, stdout, stderr = assert_python_ok("-c", code) + self.assertEqual(rc, 0) + self.assertRegex(stdout, rb"""\A\s*func=\s*\Z""") + self.assertFalse(stderr) + @refcount_test def test_frame(self): def f(): diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-04-18-02-45-40.gh-issue-91636.6DFdy_.rst b/Misc/NEWS.d/next/Core and Builtins/2022-04-18-02-45-40.gh-issue-91636.6DFdy_.rst new file mode 100644 index 0000000000000..663339bafb79e --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2022-04-18-02-45-40.gh-issue-91636.6DFdy_.rst @@ -0,0 +1 @@ +Fixed a crash in a garbage-collection edge-case, in which a ``PyFunction_Type.tp_clear`` function could leave a python function object in an inconsistent state. diff --git a/Objects/funcobject.c b/Objects/funcobject.c index deacfd55dd286..1e0cfb7efb479 100644 --- a/Objects/funcobject.c +++ b/Objects/funcobject.c @@ -678,11 +678,8 @@ static int func_clear(PyFunctionObject *op) { op->func_version = 0; - Py_CLEAR(op->func_code); Py_CLEAR(op->func_globals); Py_CLEAR(op->func_builtins); - Py_CLEAR(op->func_name); - Py_CLEAR(op->func_qualname); Py_CLEAR(op->func_module); Py_CLEAR(op->func_defaults); Py_CLEAR(op->func_kwdefaults); @@ -690,6 +687,13 @@ func_clear(PyFunctionObject *op) Py_CLEAR(op->func_dict); Py_CLEAR(op->func_closure); Py_CLEAR(op->func_annotations); + // Don't Py_CLEAR(op->func_code), since code is always required + // to be non-NULL. Similarly, name and qualname shouldn't be NULL. + // However, name and qualname could be str subclasses, so they + // could have reference cycles. The solution is to replace them + // with a genuinely immutable string. + Py_SETREF(op->func_name, Py_NewRef(&_Py_STR(empty))); + Py_SETREF(op->func_qualname, Py_NewRef(&_Py_STR(empty))); return 0; } @@ -701,6 +705,10 @@ func_dealloc(PyFunctionObject *op) PyObject_ClearWeakRefs((PyObject *) op); } (void)func_clear(op); + // These aren't cleared by func_clear(). + Py_DECREF(op->func_code); + Py_DECREF(op->func_name); + Py_DECREF(op->func_qualname); PyObject_GC_Del(op); } From webhook-mailer at python.org Thu Apr 21 10:01:54 2022 From: webhook-mailer at python.org (vstinner) Date: Thu, 21 Apr 2022 14:01:54 -0000 Subject: [Python-checkins] gh-89653: PEP 670: Convert PyWeakref_GET_OBJECT() to function (#91785) Message-ID: https://github.com/python/cpython/commit/1a2b282f201073d5153e737568c833af6f1b349e commit: 1a2b282f201073d5153e737568c833af6f1b349e branch: main author: Victor Stinner committer: vstinner date: 2022-04-21T16:01:47+02:00 summary: gh-89653: PEP 670: Convert PyWeakref_GET_OBJECT() to function (#91785) Convert the PyWeakref_GET_OBJECT() macro to a static inline function. Add an assertion to check the argument with PyWeakref_Check(). Add a macro converting the argument to PyObject* to prevent emitting new compiler warning. files: M Include/cpython/weakrefobject.h diff --git a/Include/cpython/weakrefobject.h b/Include/cpython/weakrefobject.h index 3623071cdb044..1078fffc2e0a5 100644 --- a/Include/cpython/weakrefobject.h +++ b/Include/cpython/weakrefobject.h @@ -36,13 +36,19 @@ PyAPI_FUNC(Py_ssize_t) _PyWeakref_GetWeakrefCount(PyWeakReference *head); PyAPI_FUNC(void) _PyWeakref_ClearRef(PyWeakReference *self); -/* Explanation for the Py_REFCNT() check: when a weakref's target is part - of a long chain of deallocations which triggers the trashcan mechanism, - clearing the weakrefs can be delayed long after the target's refcount - has dropped to zero. In the meantime, code accessing the weakref will - be able to "see" the target object even though it is supposed to be - unreachable. See issue #16602. */ -#define PyWeakref_GET_OBJECT(ref) \ - (Py_REFCNT(((PyWeakReference *)(ref))->wr_object) > 0 \ - ? ((PyWeakReference *)(ref))->wr_object \ - : Py_None) +static inline PyObject* PyWeakref_GET_OBJECT(PyObject *ref_obj) { + assert(PyWeakref_Check(ref_obj)); + PyWeakReference *ref = (PyWeakReference *)ref_obj; + PyObject *obj = ref->wr_object; + // Explanation for the Py_REFCNT() check: when a weakref's target is part + // of a long chain of deallocations which triggers the trashcan mechanism, + // clearing the weakrefs can be delayed long after the target's refcount + // has dropped to zero. In the meantime, code accessing the weakref will + // be able to "see" the target object even though it is supposed to be + // unreachable. See issue gh-60806. + if (Py_REFCNT(obj) > 0) { + return obj; + } + return Py_None; +} +#define PyWeakref_GET_OBJECT(ref) PyWeakref_GET_OBJECT(_PyObject_CAST(ref)) From webhook-mailer at python.org Thu Apr 21 10:40:44 2022 From: webhook-mailer at python.org (vstinner) Date: Thu, 21 Apr 2022 14:40:44 -0000 Subject: [Python-checkins] gh-91782: Define static_assert() macro on FreeBSD (#91787) Message-ID: https://github.com/python/cpython/commit/1b184c84082530f35c593cb7728752e258371c30 commit: 1b184c84082530f35c593cb7728752e258371c30 branch: main author: Victor Stinner committer: vstinner date: 2022-04-21T16:40:34+02:00 summary: gh-91782: Define static_assert() macro on FreeBSD (#91787) On FreeBSD, if the static_assert() macro is not defined, define it in Python until supports C11: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=255290 files: M Include/pymacro.h diff --git a/Include/pymacro.h b/Include/pymacro.h index 2728496976de7..71d6714afd112 100644 --- a/Include/pymacro.h +++ b/Include/pymacro.h @@ -1,6 +1,15 @@ #ifndef Py_PYMACRO_H #define Py_PYMACRO_H +// gh-91782: On FreeBSD 12, if the _POSIX_C_SOURCE and _XOPEN_SOURCE macros are +// defined, disables C11 support and does not define +// the static_assert() macro. Define the static_assert() macro in Python until +// suports C11: +// https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=255290 +#if defined(__FreeBSD__) && !defined(static_assert) +# define static_assert _Static_assert +#endif + /* Minimum value between x and y */ #define Py_MIN(x, y) (((x) > (y)) ? (y) : (x)) From webhook-mailer at python.org Thu Apr 21 10:53:08 2022 From: webhook-mailer at python.org (vstinner) Date: Thu, 21 Apr 2022 14:53:08 -0000 Subject: [Python-checkins] gh-89653: PEP 670: Convert tuple macros to functions (#91786) Message-ID: https://github.com/python/cpython/commit/2a5f171759a31597032cfe52646929e6f8727243 commit: 2a5f171759a31597032cfe52646929e6f8727243 branch: main author: Victor Stinner committer: vstinner date: 2022-04-21T16:52:54+02:00 summary: gh-89653: PEP 670: Convert tuple macros to functions (#91786) Convert macros to static inline functions: * PyTuple_GET_SIZE() * PyTuple_SET_ITEM() * PyList_GET_SIZE() * PyList_SET_ITEM() Add a macro converting arguments to PyTupleObject*, PyListObject* or PyObject* to prevent emitting new compiler warnings. According to PEP 670, PyTuple_GET_ITEM() and PyList_GET_ITEM() are left as macros. files: M Include/cpython/listobject.h M Include/cpython/tupleobject.h diff --git a/Include/cpython/listobject.h b/Include/cpython/listobject.h index 51687d866ce97..0cd69216a4c4c 100644 --- a/Include/cpython/listobject.h +++ b/Include/cpython/listobject.h @@ -24,11 +24,21 @@ typedef struct { PyAPI_FUNC(PyObject *) _PyList_Extend(PyListObject *, PyObject *); PyAPI_FUNC(void) _PyList_DebugMallocStats(FILE *out); -/* Macro, trading safety for speed */ - /* Cast argument to PyListObject* type. */ #define _PyList_CAST(op) (assert(PyList_Check(op)), (PyListObject *)(op)) -#define PyList_GET_ITEM(op, i) (_PyList_CAST(op)->ob_item[i]) -#define PyList_SET_ITEM(op, i, v) _Py_RVALUE(_PyList_CAST(op)->ob_item[i] = (v)) -#define PyList_GET_SIZE(op) Py_SIZE(_PyList_CAST(op)) +// Macros and static inline functions, trading safety for speed + +static inline Py_ssize_t PyList_GET_SIZE(PyListObject *op) { + return Py_SIZE(op); +} +#define PyList_GET_SIZE(op) PyList_GET_SIZE(_PyList_CAST(op)) + +#define PyList_GET_ITEM(op, index) (_PyList_CAST(op)->ob_item[index]) + +static inline void +PyList_SET_ITEM(PyListObject *op, Py_ssize_t index, PyObject *value) { + op->ob_item[index] = value; +} +#define PyList_SET_ITEM(op, index, value) \ + PyList_SET_ITEM(_PyList_CAST(op), index, _PyObject_CAST(value)) diff --git a/Include/cpython/tupleobject.h b/Include/cpython/tupleobject.h index fc37c4e6de1ca..fd15ecd1c772f 100644 --- a/Include/cpython/tupleobject.h +++ b/Include/cpython/tupleobject.h @@ -13,16 +13,24 @@ typedef struct { PyAPI_FUNC(int) _PyTuple_Resize(PyObject **, Py_ssize_t); PyAPI_FUNC(void) _PyTuple_MaybeUntrack(PyObject *); -/* Macros trading safety for speed */ - /* Cast argument to PyTupleObject* type. */ #define _PyTuple_CAST(op) (assert(PyTuple_Check(op)), (PyTupleObject *)(op)) -#define PyTuple_GET_SIZE(op) Py_SIZE(_PyTuple_CAST(op)) +// Macros and static inline functions, trading safety for speed + +static inline Py_ssize_t PyTuple_GET_SIZE(PyTupleObject *op) { + return Py_SIZE(op); +} +#define PyTuple_GET_SIZE(op) PyTuple_GET_SIZE(_PyTuple_CAST(op)) -#define PyTuple_GET_ITEM(op, i) (_PyTuple_CAST(op)->ob_item[i]) +#define PyTuple_GET_ITEM(op, index) (_PyTuple_CAST(op)->ob_item[index]) -/* Macro, *only* to be used to fill in brand new tuples */ -#define PyTuple_SET_ITEM(op, i, v) _Py_RVALUE(_PyTuple_CAST(op)->ob_item[i] = (v)) +/* Function *only* to be used to fill in brand new tuples */ +static inline void +PyTuple_SET_ITEM(PyTupleObject *op, Py_ssize_t index, PyObject *value) { + op->ob_item[index] = value; +} +#define PyTuple_SET_ITEM(op, index, value) \ + PyTuple_SET_ITEM(_PyTuple_CAST(op), index, _PyObject_CAST(value)) PyAPI_FUNC(void) _PyTuple_DebugMallocStats(FILE *out); From webhook-mailer at python.org Thu Apr 21 11:10:48 2022 From: webhook-mailer at python.org (markshannon) Date: Thu, 21 Apr 2022 15:10:48 -0000 Subject: [Python-checkins] GH-88116: Use a compact format to represent end line and column offsets. (GH-91666) Message-ID: https://github.com/python/cpython/commit/944fffee8916cb94321fa33cd3a43f4108717746 commit: 944fffee8916cb94321fa33cd3a43f4108717746 branch: main author: Mark Shannon committer: markshannon date: 2022-04-21T16:10:37+01:00 summary: GH-88116: Use a compact format to represent end line and column offsets. (GH-91666) * Stores all location info in linetable to conform to PEP 626. * Remove column table from code objects. * Remove end-line table from code objects. * Document new location table format files: A Misc/NEWS.d/next/Core and Builtins/2022-04-18-15-22-56.bpo-43950.qrTvWL.rst A Objects/locations.md M Include/cpython/code.h M Include/internal/pycore_code.h M Lib/importlib/_bootstrap_external.py M Lib/test/test_code.py M Lib/test/test_compile.py M Lib/test/test_dis.py M Lib/test/test_exceptions.py M Lib/test/test_marshal.py M Lib/test/test_traceback.py M Objects/clinic/codeobject.c.h M Objects/codeobject.c M Objects/frameobject.c M Programs/test_frozenmain.h M Python/compile.c M Python/marshal.c M Tools/gdb/libpython.py M Tools/scripts/deepfreeze.py M Tools/scripts/umarshal.py diff --git a/Include/cpython/code.h b/Include/cpython/code.h index 6dc2290ffeb5e..be3b10bba724b 100644 --- a/Include/cpython/code.h +++ b/Include/cpython/code.h @@ -86,15 +86,7 @@ typedef uint16_t _Py_CODEUNIT; PyObject *co_filename; /* unicode (where it was loaded from) */ \ PyObject *co_name; /* unicode (name, for reference) */ \ PyObject *co_qualname; /* unicode (qualname, for reference) */ \ - PyObject *co_linetable; /* bytes (encoding addr<->lineno mapping) \ - See Objects/lnotab_notes.txt for details. \ - */ \ - PyObject *co_endlinetable; /* bytes object that holds end lineno for \ - instructions separated across different \ - lines */ \ - PyObject *co_columntable; /* bytes object that holds start/end column \ - offset each instruction */ \ - \ + PyObject *co_linetable; /* bytes object that holds location info */ \ PyObject *co_weakreflist; /* to support weakrefs to code objects */ \ /* Scratch space for extra data relating to the code object. \ Type is a void* to keep the format private in codeobject.c to force \ @@ -153,13 +145,13 @@ PyAPI_FUNC(PyCodeObject *) PyCode_New( int, int, int, int, int, PyObject *, PyObject *, PyObject *, PyObject *, PyObject *, PyObject *, PyObject *, PyObject *, PyObject *, int, PyObject *, - PyObject *, PyObject *, PyObject *); + PyObject *); PyAPI_FUNC(PyCodeObject *) PyCode_NewWithPosOnlyArgs( int, int, int, int, int, int, PyObject *, PyObject *, PyObject *, PyObject *, PyObject *, PyObject *, PyObject *, PyObject *, PyObject *, int, PyObject *, - PyObject *, PyObject *, PyObject *); + PyObject *); /* same as struct above */ /* Creates a new empty code object with the specified source location. */ @@ -176,8 +168,8 @@ PyAPI_FUNC(int) PyCode_Addr2Location(PyCodeObject *, int, int *, int *, int *, i /* for internal use only */ struct _opaque { int computed_line; - const char *lo_next; - const char *limit; + const uint8_t *lo_next; + const uint8_t *limit; }; typedef struct _line_offsets { @@ -210,6 +202,20 @@ PyAPI_FUNC(int) _PyCode_GetExtra(PyObject *code, Py_ssize_t index, PyAPI_FUNC(int) _PyCode_SetExtra(PyObject *code, Py_ssize_t index, void *extra); + +typedef enum _PyCodeLocationInfoKind { + /* short forms are 0 to 9 */ + PY_CODE_LOCATION_INFO_SHORT0 = 0, + /* one lineforms are 10 to 12 */ + PY_CODE_LOCATION_INFO_ONE_LINE0 = 10, + PY_CODE_LOCATION_INFO_ONE_LINE1 = 11, + PY_CODE_LOCATION_INFO_ONE_LINE2 = 12, + + PY_CODE_LOCATION_INFO_NO_COLUMNS = 13, + PY_CODE_LOCATION_INFO_LONG = 14, + PY_CODE_LOCATION_INFO_NONE = 15 +} _PyCodeLocationInfoKind; + #ifdef __cplusplus } #endif diff --git a/Include/internal/pycore_code.h b/Include/internal/pycore_code.h index 8c868bcd5b5cb..3059db465e7d2 100644 --- a/Include/internal/pycore_code.h +++ b/Include/internal/pycore_code.h @@ -176,8 +176,6 @@ struct _PyCodeConstructor { PyObject *code; int firstlineno; PyObject *linetable; - PyObject *endlinetable; - PyObject *columntable; /* used by the code */ PyObject *consts; @@ -221,21 +219,10 @@ extern PyObject* _PyCode_GetCellvars(PyCodeObject *); extern PyObject* _PyCode_GetFreevars(PyCodeObject *); extern PyObject* _PyCode_GetCode(PyCodeObject *); -/* Return the ending source code line number from a bytecode index. */ -extern int _PyCode_Addr2EndLine(PyCodeObject *, int); - -/* Return the ending source code line number from a bytecode index. */ -extern int _PyCode_Addr2EndLine(PyCodeObject *, int); -/* Return the starting source code column offset from a bytecode index. */ -extern int _PyCode_Addr2Offset(PyCodeObject *, int); -/* Return the ending source code column offset from a bytecode index. */ -extern int _PyCode_Addr2EndOffset(PyCodeObject *, int); - /** API for initializing the line number tables. */ extern int _PyCode_InitAddressRange(PyCodeObject* co, PyCodeAddressRange *bounds); -extern int _PyCode_InitEndAddressRange(PyCodeObject* co, PyCodeAddressRange* bounds); -/** Out of process API for initializing the line number table. */ +/** Out of process API for initializing the location table. */ extern void _PyLineTable_InitAddressRange( const char *linetable, Py_ssize_t length, @@ -445,6 +432,40 @@ read_obj(uint16_t *p) return (PyObject *)val; } +static inline int +write_varint(uint8_t *ptr, unsigned int val) +{ + int written = 1; + while (val >= 64) { + *ptr++ = 64 | (val & 63); + val >>= 6; + written++; + } + *ptr = val; + return written; +} + +static inline int +write_signed_varint(uint8_t *ptr, int val) +{ + if (val < 0) { + val = ((-val)<<1) | 1; + } + else { + val = val << 1; + } + return write_varint(ptr, val); +} + +static inline int +write_location_entry_start(uint8_t *ptr, int code, int length) +{ + assert((code & 15) == code); + *ptr = 128 | (code << 3) | (length - 1); + return 1; +} + + #ifdef __cplusplus } #endif diff --git a/Lib/importlib/_bootstrap_external.py b/Lib/importlib/_bootstrap_external.py index 4eece8de24c3e..71e1e24b51e22 100644 --- a/Lib/importlib/_bootstrap_external.py +++ b/Lib/importlib/_bootstrap_external.py @@ -402,7 +402,7 @@ def _write_atomic(path, data, mode=0o666): # add JUMP_BACKWARD_NO_INTERRUPT, make JUMP_NO_INTERRUPT virtual) # Python 3.11a7 3492 (make POP_JUMP_IF_NONE/NOT_NONE/TRUE/FALSE relative) # Python 3.11a7 3493 (Make JUMP_IF_TRUE_OR_POP/JUMP_IF_FALSE_OR_POP relative) - +# Python 3.11a7 3494 (New location info table) # Python 3.12 will start with magic number 3500 diff --git a/Lib/test/test_code.py b/Lib/test/test_code.py index 872f7283fc504..a37ebd27dc388 100644 --- a/Lib/test/test_code.py +++ b/Lib/test/test_code.py @@ -230,9 +230,7 @@ def func(): pass co.co_name, co.co_qualname, co.co_firstlineno, - co.co_lnotab, - co.co_endlinetable, - co.co_columntable, + co.co_linetable, co.co_exceptiontable, co.co_freevars, co.co_cellvars) @@ -273,8 +271,6 @@ def func2(): ("co_filename", "newfilename"), ("co_name", "newname"), ("co_linetable", code2.co_linetable), - ("co_endlinetable", code2.co_endlinetable), - ("co_columntable", code2.co_columntable), ): with self.subTest(attr=attr, value=value): new_code = code.replace(**{attr: value}) @@ -311,9 +307,7 @@ def func(): co.co_name, co.co_qualname, co.co_firstlineno, - co.co_lnotab, - co.co_endlinetable, - co.co_columntable, + co.co_linetable, co.co_exceptiontable, co.co_freevars, co.co_cellvars, @@ -391,14 +385,17 @@ def test_co_positions_artificial_instructions(self): ) def test_endline_and_columntable_none_when_no_debug_ranges(self): - # Make sure that if `-X no_debug_ranges` is used, the endlinetable and - # columntable are None. + # Make sure that if `-X no_debug_ranges` is used, there is + # minimal debug info code = textwrap.dedent(""" def f(): pass - assert f.__code__.co_endlinetable is None - assert f.__code__.co_columntable is None + positions = f.__code__.co_positions() + for line, end_line, column, end_column in positions: + assert line == end_line + assert column is None + assert end_column is None """) assert_python_ok('-X', 'no_debug_ranges', '-c', code) @@ -408,8 +405,11 @@ def test_endline_and_columntable_none_when_no_debug_ranges_env(self): def f(): pass - assert f.__code__.co_endlinetable is None - assert f.__code__.co_columntable is None + positions = f.__code__.co_positions() + for line, end_line, column, end_column in positions: + assert line == end_line + assert column is None + assert end_column is None """) assert_python_ok('-c', code, PYTHONNODEBUGRANGES='1') @@ -421,35 +421,10 @@ def func(): x = 1 new_code = func.__code__.replace(co_linetable=b'') positions = new_code.co_positions() - next(positions) # Skip RESUME at start for line, end_line, column, end_column in positions: self.assertIsNone(line) self.assertEqual(end_line, new_code.co_firstlineno + 1) - @requires_debug_ranges() - def test_co_positions_empty_endlinetable(self): - def func(): - x = 1 - new_code = func.__code__.replace(co_endlinetable=b'') - positions = new_code.co_positions() - next(positions) # Skip RESUME at start - for line, end_line, column, end_column in positions: - self.assertEqual(line, new_code.co_firstlineno + 1) - self.assertIsNone(end_line) - - @requires_debug_ranges() - def test_co_positions_empty_columntable(self): - def func(): - x = 1 - new_code = func.__code__.replace(co_columntable=b'') - positions = new_code.co_positions() - next(positions) # Skip RESUME at start - for line, end_line, column, end_column in positions: - self.assertEqual(line, new_code.co_firstlineno + 1) - self.assertEqual(end_line, new_code.co_firstlineno + 1) - self.assertIsNone(column) - self.assertIsNone(end_column) - def isinterned(s): return s is sys.intern(('_' + s + '_')[1:-1]) @@ -527,6 +502,122 @@ def callback(code): self.assertFalse(bool(coderef())) self.assertTrue(self.called) +# Python implementation of location table parsing algorithm +def read(it): + return next(it) + +def read_varint(it): + b = read(it) + val = b & 63; + shift = 0; + while b & 64: + b = read(it) + shift += 6 + val |= (b&63) << shift + return val + +def read_signed_varint(it): + uval = read_varint(it) + if uval & 1: + return -(uval >> 1) + else: + return uval >> 1 + +def parse_location_table(code): + line = code.co_firstlineno + it = iter(code.co_linetable) + while True: + try: + first_byte = read(it) + except StopIteration: + return + code = (first_byte >> 3) & 15 + length = (first_byte & 7) + 1 + if code == 15: + yield (code, length, None, None, None, None) + elif code == 14: + line_delta = read_signed_varint(it) + line += line_delta + end_line = line + read_varint(it) + col = read_varint(it) + if col == 0: + col = None + else: + col -= 1 + end_col = read_varint(it) + if end_col == 0: + end_col = None + else: + end_col -= 1 + yield (code, length, line, end_line, col, end_col) + elif code == 13: # No column + line_delta = read_signed_varint(it) + line += line_delta + yield (code, length, line, line, None, None) + elif code in (10, 11, 12): # new line + line_delta = code - 10 + line += line_delta + column = read(it) + end_column = read(it) + yield (code, length, line, line, column, end_column) + else: + assert (0 <= code < 10) + second_byte = read(it) + column = code << 3 | (second_byte >> 4) + yield (code, length, line, line, column, column + (second_byte & 15)) + +def positions_from_location_table(code): + for _, length, line, end_line, col, end_col in parse_location_table(code): + for _ in range(length): + yield (line, end_line, col, end_col) + +def misshappen(): + """ + + + + + + """ + x = ( + + + 4 + + + + + y + + ) + y = ( + a + + + b + + + + d + ) + return q if ( + + x + + ) else p + + +class CodeLocationTest(unittest.TestCase): + + def check_positions(self, func): + pos1 = list(func.__code__.co_positions()) + pos2 = list(positions_from_location_table(func.__code__)) + for l1, l2 in zip(pos1, pos2): + self.assertEqual(l1, l2) + self.assertEqual(len(pos1), len(pos2)) + + + def test_positions(self): + self.check_positions(parse_location_table) + self.check_positions(misshappen) + if check_impl_detail(cpython=True) and ctypes is not None: py = ctypes.pythonapi diff --git a/Lib/test/test_compile.py b/Lib/test/test_compile.py index a4e80805d3e5c..5a9c618786f4e 100644 --- a/Lib/test/test_compile.py +++ b/Lib/test/test_compile.py @@ -158,7 +158,9 @@ def test_leading_newlines(self): s256 = "".join(["\n"] * 256 + ["spam"]) co = compile(s256, 'fn', 'exec') self.assertEqual(co.co_firstlineno, 1) - self.assertEqual(list(co.co_lines()), [(0, 2, None), (2, 10, 257)]) + lines = list(co.co_lines()) + self.assertEqual(lines[0][2], None) + self.assertEqual(lines[1][2], 257) def test_literals_with_leading_zeroes(self): for arg in ["077787", "0xj", "0x.", "0e", "090000000000000", @@ -892,12 +894,19 @@ def no_code2(): with self.subTest(func=func): code = func.__code__ lines = list(code.co_lines()) - self.assertEqual(len(lines), 1) start, end, line = lines[0] self.assertEqual(start, 0) - self.assertEqual(end, len(code.co_code)) self.assertEqual(line, code.co_firstlineno) + def get_code_lines(self, code): + last_line = -2 + res = [] + for _, _, line in code.co_lines(): + if line is not None and line != last_line: + res.append(line - code.co_firstlineno) + last_line = line + return res + def test_lineno_attribute(self): def load_attr(): return ( @@ -939,9 +948,7 @@ def aug_store_attr(): for func, lines in zip(funcs, func_lines, strict=True): with self.subTest(func=func): - code_lines = [ line-func.__code__.co_firstlineno - for (_, _, line) in func.__code__.co_lines() - if line is not None ] + code_lines = self.get_code_lines(func.__code__) self.assertEqual(lines, code_lines) def test_line_number_genexp(self): @@ -952,11 +959,10 @@ def return_genexp(): x in y) - genexp_lines = [1, 3, 1] + genexp_lines = [0, 2, 0] genexp_code = return_genexp.__code__.co_consts[1] - code_lines = [ None if line is None else line-return_genexp.__code__.co_firstlineno - for (_, _, line) in genexp_code.co_lines() ] + code_lines = self.get_code_lines(genexp_code) self.assertEqual(genexp_lines, code_lines) def test_line_number_implicit_return_after_async_for(self): @@ -966,8 +972,7 @@ async def test(aseq): body expected_lines = [0, 1, 2, 1] - code_lines = [ None if line is None else line-test.__code__.co_firstlineno - for (_, _, line) in test.__code__.co_lines() ] + code_lines = self.get_code_lines(test.__code__) self.assertEqual(expected_lines, code_lines) def test_big_dict_literal(self): @@ -1112,14 +1117,14 @@ def test_multiline_expression(self): line=1, end_line=3, column=0, end_column=1) def test_very_long_line_end_offset(self): - # Make sure we get None for when the column offset is too large to - # store in a byte. + # Make sure we get the correct column offset for offsets + # too large to store in a byte. long_string = "a" * 1000 snippet = f"g('{long_string}')" compiled_code, _ = self.check_positions_against_ast(snippet) self.assertOpcodeSourcePositionIs(compiled_code, 'CALL', - line=1, end_line=1, column=None, end_column=None) + line=1, end_line=1, column=0, end_column=1005) def test_complex_single_line_expression(self): snippet = "a - b @ (c * x['key'] + 23)" diff --git a/Lib/test/test_dis.py b/Lib/test/test_dis.py index f560a5556c8b0..0bd589d5cfb6f 100644 --- a/Lib/test/test_dis.py +++ b/Lib/test/test_dis.py @@ -196,7 +196,7 @@ def bug42562(): # Set line number for 'pass' to None -bug42562.__code__ = bug42562.__code__.replace(co_linetable=b'\x04\x80') +bug42562.__code__ = bug42562.__code__.replace(co_linetable=b'\xf8') dis_bug42562 = """\ @@ -1560,32 +1560,19 @@ def test_co_positions(self): @requires_debug_ranges() def test_co_positions_missing_info(self): code = compile('x, y, z', '', 'exec') - code_without_column_table = code.replace(co_columntable=b'') - actual = dis.get_instructions(code_without_column_table) + code_without_location_table = code.replace(co_linetable=b'') + actual = dis.get_instructions(code_without_location_table) for instruction in actual: with self.subTest(instruction=instruction): positions = instruction.positions self.assertEqual(len(positions), 4) if instruction.opname == "RESUME": continue - self.assertEqual(positions.lineno, 1) - self.assertEqual(positions.end_lineno, 1) + self.assertIsNone(positions.lineno) + self.assertIsNone(positions.end_lineno) self.assertIsNone(positions.col_offset) self.assertIsNone(positions.end_col_offset) - code_without_endline_table = code.replace(co_endlinetable=b'') - actual = dis.get_instructions(code_without_endline_table) - for instruction in actual: - with self.subTest(instruction=instruction): - positions = instruction.positions - self.assertEqual(len(positions), 4) - if instruction.opname == "RESUME": - continue - self.assertEqual(positions.lineno, 1) - self.assertIsNone(positions.end_lineno) - self.assertIsNotNone(positions.col_offset) - self.assertIsNotNone(positions.end_col_offset) - # get_instructions has its own tests above, so can rely on it to validate # the object oriented API class BytecodeTests(InstructionTestCase, DisTestBase): diff --git a/Lib/test/test_exceptions.py b/Lib/test/test_exceptions.py index 2b5b51934562a..ff1a02821a58f 100644 --- a/Lib/test/test_exceptions.py +++ b/Lib/test/test_exceptions.py @@ -2658,7 +2658,7 @@ def test_missing_lineno_shows_as_none(self): def f(): 1/0 self.lineno_after_raise(f, 1) - f.__code__ = f.__code__.replace(co_linetable=b'\x04\x80\xff\x80') + f.__code__ = f.__code__.replace(co_linetable=b'\xf8\xf8\xf8\xf9\xf8\xf8\xf8') self.lineno_after_raise(f, None) def test_lineno_after_raise_in_with_exit(self): diff --git a/Lib/test/test_marshal.py b/Lib/test/test_marshal.py index 8d55382b195a1..aae86cc257d7e 100644 --- a/Lib/test/test_marshal.py +++ b/Lib/test/test_marshal.py @@ -129,18 +129,18 @@ def test_different_filenames(self): self.assertEqual(co2.co_filename, "f2") @requires_debug_ranges() - def test_no_columntable_and_endlinetable_with_no_debug_ranges(self): + def test_minimal_linetable_with_no_debug_ranges(self): # Make sure when demarshalling objects with `-X no_debug_ranges` - # that the columntable and endlinetable are None. + # that the columns are None. co = ExceptionTestCase.test_exceptions.__code__ code = textwrap.dedent(""" import sys import marshal with open(sys.argv[1], 'rb') as f: co = marshal.load(f) - - assert co.co_endlinetable is None - assert co.co_columntable is None + positions = list(co.co_positions()) + assert positions[0][2] is None + assert positions[0][3] is None """) try: diff --git a/Lib/test/test_traceback.py b/Lib/test/test_traceback.py index 962322c89ff66..ed5ddf95069e3 100644 --- a/Lib/test/test_traceback.py +++ b/Lib/test/test_traceback.py @@ -615,6 +615,7 @@ def test_traceback_very_long_line(self): ' ^^^^^^^^^^\n' f' File "{TESTFN}", line {lineno_f}, in \n' f' {source}\n' + f' {"^"*len(source)}\n' ) self.assertEqual(result_lines, expected_error.splitlines()) diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-04-18-15-22-56.bpo-43950.qrTvWL.rst b/Misc/NEWS.d/next/Core and Builtins/2022-04-18-15-22-56.bpo-43950.qrTvWL.rst new file mode 100644 index 0000000000000..c8bfa5914b027 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2022-04-18-15-22-56.bpo-43950.qrTvWL.rst @@ -0,0 +1,2 @@ +Use a single compact table for line starts, ends and column offsets. Reduces +memory consumption for location info by half diff --git a/Objects/clinic/codeobject.c.h b/Objects/clinic/codeobject.c.h index 272bcd6ea17b2..41c5c2e1170f8 100644 --- a/Objects/clinic/codeobject.c.h +++ b/Objects/clinic/codeobject.c.h @@ -5,8 +5,8 @@ preserve PyDoc_STRVAR(code_new__doc__, "code(argcount, posonlyargcount, kwonlyargcount, nlocals, stacksize,\n" " flags, codestring, constants, names, varnames, filename, name,\n" -" qualname, firstlineno, linetable, endlinetable, columntable,\n" -" exceptiontable, freevars=(), cellvars=(), /)\n" +" qualname, firstlineno, linetable, exceptiontable, freevars=(),\n" +" cellvars=(), /)\n" "--\n" "\n" "Create a code object. Not for the faint of heart."); @@ -17,7 +17,6 @@ code_new_impl(PyTypeObject *type, int argcount, int posonlyargcount, PyObject *code, PyObject *consts, PyObject *names, PyObject *varnames, PyObject *filename, PyObject *name, PyObject *qualname, int firstlineno, PyObject *linetable, - PyObject *endlinetable, PyObject *columntable, PyObject *exceptiontable, PyObject *freevars, PyObject *cellvars); @@ -40,8 +39,6 @@ code_new(PyTypeObject *type, PyObject *args, PyObject *kwargs) PyObject *qualname; int firstlineno; PyObject *linetable; - PyObject *endlinetable; - PyObject *columntable; PyObject *exceptiontable; PyObject *freevars = NULL; PyObject *cellvars = NULL; @@ -51,7 +48,7 @@ code_new(PyTypeObject *type, PyObject *args, PyObject *kwargs) !_PyArg_NoKeywords("code", kwargs)) { goto exit; } - if (!_PyArg_CheckPositional("code", PyTuple_GET_SIZE(args), 18, 20)) { + if (!_PyArg_CheckPositional("code", PyTuple_GET_SIZE(args), 16, 18)) { goto exit; } argcount = _PyLong_AsInt(PyTuple_GET_ITEM(args, 0)); @@ -131,31 +128,29 @@ code_new(PyTypeObject *type, PyObject *args, PyObject *kwargs) goto exit; } linetable = PyTuple_GET_ITEM(args, 14); - endlinetable = PyTuple_GET_ITEM(args, 15); - columntable = PyTuple_GET_ITEM(args, 16); - if (!PyBytes_Check(PyTuple_GET_ITEM(args, 17))) { - _PyArg_BadArgument("code", "argument 18", "bytes", PyTuple_GET_ITEM(args, 17)); + if (!PyBytes_Check(PyTuple_GET_ITEM(args, 15))) { + _PyArg_BadArgument("code", "argument 16", "bytes", PyTuple_GET_ITEM(args, 15)); goto exit; } - exceptiontable = PyTuple_GET_ITEM(args, 17); - if (PyTuple_GET_SIZE(args) < 19) { + exceptiontable = PyTuple_GET_ITEM(args, 15); + if (PyTuple_GET_SIZE(args) < 17) { goto skip_optional; } - if (!PyTuple_Check(PyTuple_GET_ITEM(args, 18))) { - _PyArg_BadArgument("code", "argument 19", "tuple", PyTuple_GET_ITEM(args, 18)); + if (!PyTuple_Check(PyTuple_GET_ITEM(args, 16))) { + _PyArg_BadArgument("code", "argument 17", "tuple", PyTuple_GET_ITEM(args, 16)); goto exit; } - freevars = PyTuple_GET_ITEM(args, 18); - if (PyTuple_GET_SIZE(args) < 20) { + freevars = PyTuple_GET_ITEM(args, 16); + if (PyTuple_GET_SIZE(args) < 18) { goto skip_optional; } - if (!PyTuple_Check(PyTuple_GET_ITEM(args, 19))) { - _PyArg_BadArgument("code", "argument 20", "tuple", PyTuple_GET_ITEM(args, 19)); + if (!PyTuple_Check(PyTuple_GET_ITEM(args, 17))) { + _PyArg_BadArgument("code", "argument 18", "tuple", PyTuple_GET_ITEM(args, 17)); goto exit; } - cellvars = PyTuple_GET_ITEM(args, 19); + cellvars = PyTuple_GET_ITEM(args, 17); skip_optional: - return_value = code_new_impl(type, argcount, posonlyargcount, kwonlyargcount, nlocals, stacksize, flags, code, consts, names, varnames, filename, name, qualname, firstlineno, linetable, endlinetable, columntable, exceptiontable, freevars, cellvars); + return_value = code_new_impl(type, argcount, posonlyargcount, kwonlyargcount, nlocals, stacksize, flags, code, consts, names, varnames, filename, name, qualname, firstlineno, linetable, exceptiontable, freevars, cellvars); exit: return return_value; @@ -167,8 +162,7 @@ PyDoc_STRVAR(code_replace__doc__, " co_flags=-1, co_firstlineno=-1, co_code=None, co_consts=None,\n" " co_names=None, co_varnames=None, co_freevars=None,\n" " co_cellvars=None, co_filename=None, co_name=None,\n" -" co_qualname=None, co_linetable=None, co_endlinetable=None,\n" -" co_columntable=None, co_exceptiontable=None)\n" +" co_qualname=None, co_linetable=None, co_exceptiontable=None)\n" "--\n" "\n" "Return a copy of the code object with new values for the specified fields."); @@ -185,16 +179,16 @@ code_replace_impl(PyCodeObject *self, int co_argcount, PyObject *co_varnames, PyObject *co_freevars, PyObject *co_cellvars, PyObject *co_filename, PyObject *co_name, PyObject *co_qualname, - PyBytesObject *co_linetable, PyObject *co_endlinetable, - PyObject *co_columntable, PyBytesObject *co_exceptiontable); + PyBytesObject *co_linetable, + PyBytesObject *co_exceptiontable); static PyObject * code_replace(PyCodeObject *self, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) { PyObject *return_value = NULL; - static const char * const _keywords[] = {"co_argcount", "co_posonlyargcount", "co_kwonlyargcount", "co_nlocals", "co_stacksize", "co_flags", "co_firstlineno", "co_code", "co_consts", "co_names", "co_varnames", "co_freevars", "co_cellvars", "co_filename", "co_name", "co_qualname", "co_linetable", "co_endlinetable", "co_columntable", "co_exceptiontable", NULL}; + static const char * const _keywords[] = {"co_argcount", "co_posonlyargcount", "co_kwonlyargcount", "co_nlocals", "co_stacksize", "co_flags", "co_firstlineno", "co_code", "co_consts", "co_names", "co_varnames", "co_freevars", "co_cellvars", "co_filename", "co_name", "co_qualname", "co_linetable", "co_exceptiontable", NULL}; static _PyArg_Parser _parser = {NULL, _keywords, "replace", 0}; - PyObject *argsbuf[20]; + PyObject *argsbuf[18]; Py_ssize_t noptargs = nargs + (kwnames ? PyTuple_GET_SIZE(kwnames) : 0) - 0; int co_argcount = self->co_argcount; int co_posonlyargcount = self->co_posonlyargcount; @@ -213,8 +207,6 @@ code_replace(PyCodeObject *self, PyObject *const *args, Py_ssize_t nargs, PyObje PyObject *co_name = self->co_name; PyObject *co_qualname = self->co_qualname; PyBytesObject *co_linetable = (PyBytesObject *)self->co_linetable; - PyObject *co_endlinetable = self->co_endlinetable; - PyObject *co_columntable = self->co_columntable; PyBytesObject *co_exceptiontable = (PyBytesObject *)self->co_exceptiontable; args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 0, 0, 0, argsbuf); @@ -396,25 +388,13 @@ code_replace(PyCodeObject *self, PyObject *const *args, Py_ssize_t nargs, PyObje goto skip_optional_kwonly; } } - if (args[17]) { - co_endlinetable = args[17]; - if (!--noptargs) { - goto skip_optional_kwonly; - } - } - if (args[18]) { - co_columntable = args[18]; - if (!--noptargs) { - goto skip_optional_kwonly; - } - } - if (!PyBytes_Check(args[19])) { - _PyArg_BadArgument("replace", "argument 'co_exceptiontable'", "bytes", args[19]); + if (!PyBytes_Check(args[17])) { + _PyArg_BadArgument("replace", "argument 'co_exceptiontable'", "bytes", args[17]); goto exit; } - co_exceptiontable = (PyBytesObject *)args[19]; + co_exceptiontable = (PyBytesObject *)args[17]; skip_optional_kwonly: - return_value = code_replace_impl(self, co_argcount, co_posonlyargcount, co_kwonlyargcount, co_nlocals, co_stacksize, co_flags, co_firstlineno, co_code, co_consts, co_names, co_varnames, co_freevars, co_cellvars, co_filename, co_name, co_qualname, co_linetable, co_endlinetable, co_columntable, co_exceptiontable); + return_value = code_replace_impl(self, co_argcount, co_posonlyargcount, co_kwonlyargcount, co_nlocals, co_stacksize, co_flags, co_firstlineno, co_code, co_consts, co_names, co_varnames, co_freevars, co_cellvars, co_filename, co_name, co_qualname, co_linetable, co_exceptiontable); exit: return return_value; @@ -456,4 +436,4 @@ code__varname_from_oparg(PyCodeObject *self, PyObject *const *args, Py_ssize_t n exit: return return_value; } -/*[clinic end generated code: output=b1b83a70ffc5b7cd input=a9049054013a1b77]*/ +/*[clinic end generated code: output=ebfeec29d2cff674 input=a9049054013a1b77]*/ diff --git a/Objects/codeobject.c b/Objects/codeobject.c index e872b398e08c8..9a57815882756 100644 --- a/Objects/codeobject.c +++ b/Objects/codeobject.c @@ -244,10 +244,6 @@ _PyCode_Validate(struct _PyCodeConstructor *con) con->qualname == NULL || !PyUnicode_Check(con->qualname) || con->filename == NULL || !PyUnicode_Check(con->filename) || con->linetable == NULL || !PyBytes_Check(con->linetable) || - con->endlinetable == NULL || - (con->endlinetable != Py_None && !PyBytes_Check(con->endlinetable)) || - con->columntable == NULL || - (con->columntable != Py_None && !PyBytes_Check(con->columntable)) || con->exceptiontable == NULL || !PyBytes_Check(con->exceptiontable) ) { PyErr_BadInternalCall(); @@ -307,10 +303,6 @@ init_code(PyCodeObject *co, struct _PyCodeConstructor *con) co->co_firstlineno = con->firstlineno; Py_INCREF(con->linetable); co->co_linetable = con->linetable; - Py_INCREF(con->endlinetable); - co->co_endlinetable = con->endlinetable; - Py_INCREF(con->columntable); - co->co_columntable = con->columntable; Py_INCREF(con->consts); co->co_consts = con->consts; @@ -347,6 +339,97 @@ init_code(PyCodeObject *co, struct _PyCodeConstructor *con) PyBytes_GET_SIZE(con->code)); } +static int +scan_varint(const uint8_t *ptr) +{ + int read = *ptr++; + int val = read & 63; + int shift = 0; + while (read & 64) { + read = *ptr++; + shift += 6; + val |= (read & 63) << shift; + } + return val; +} + +static int +scan_signed_varint(const uint8_t *ptr) +{ + int uval = scan_varint(ptr); + if (uval & 1) { + return -(int)(uval >> 1); + } + else { + return uval >> 1; + } +} + +static int +get_line_delta(const uint8_t *ptr) +{ + int code = ((*ptr) >> 3) & 15; + switch (code) { + case PY_CODE_LOCATION_INFO_NONE: + return 0; + case PY_CODE_LOCATION_INFO_NO_COLUMNS: + case PY_CODE_LOCATION_INFO_LONG: + return scan_signed_varint(ptr+1); + case PY_CODE_LOCATION_INFO_ONE_LINE0: + return 0; + case PY_CODE_LOCATION_INFO_ONE_LINE1: + return 1; + case PY_CODE_LOCATION_INFO_ONE_LINE2: + return 2; + default: + /* Same line */ + return 0; + } +} + +static PyObject * +remove_column_info(PyObject *locations) +{ + int offset = 0; + const uint8_t *data = (const uint8_t *)PyBytes_AS_STRING(locations); + PyObject *res = PyBytes_FromStringAndSize(NULL, 32); + if (res == NULL) { + PyErr_NoMemory(); + return NULL; + } + uint8_t *output = (uint8_t *)PyBytes_AS_STRING(res); + while (offset < PyBytes_GET_SIZE(locations)) { + Py_ssize_t write_offset = output - (uint8_t *)PyBytes_AS_STRING(res); + if (write_offset + 16 >= PyBytes_GET_SIZE(res)) { + if (_PyBytes_Resize(&res, PyBytes_GET_SIZE(res) * 2) < 0) { + return NULL; + } + output = (uint8_t *)PyBytes_AS_STRING(res) + write_offset; + } + int code = (data[offset] >> 3) & 15; + if (code == PY_CODE_LOCATION_INFO_NONE) { + *output++ = data[offset]; + } + else { + int blength = (data[offset] & 7)+1; + output += write_location_entry_start( + output, PY_CODE_LOCATION_INFO_NO_COLUMNS, blength); + int ldelta = get_line_delta(&data[offset]); + output += write_signed_varint(output, ldelta); + } + offset++; + while (offset < PyBytes_GET_SIZE(locations) && + (data[offset] & 128) == 0) { + offset++; + } + } + Py_ssize_t write_offset = output - (uint8_t *)PyBytes_AS_STRING(res); + if (_PyBytes_Resize(&res, write_offset)) { + return NULL; + } + return res; +} + /* The caller is responsible for ensuring that the given data is valid. */ PyCodeObject * @@ -373,21 +456,26 @@ _PyCode_New(struct _PyCodeConstructor *con) return NULL; } - // Discard the endlinetable and columntable if we are opted out of debug + PyObject *replacement_locations = NULL; + // Compact the linetable if we are opted out of debug // ranges. if (!_Py_GetConfig()->code_debug_ranges) { - con->endlinetable = Py_None; - con->columntable = Py_None; + replacement_locations = remove_column_info(con->linetable); + if (replacement_locations == NULL) { + return NULL; + } + con->linetable = replacement_locations; } Py_ssize_t size = PyBytes_GET_SIZE(con->code) / sizeof(_Py_CODEUNIT); PyCodeObject *co = PyObject_NewVar(PyCodeObject, &PyCode_Type, size); if (co == NULL) { + Py_XDECREF(replacement_locations); PyErr_NoMemory(); return NULL; } init_code(co, con); - + Py_XDECREF(replacement_locations); return co; } @@ -403,8 +491,8 @@ PyCode_NewWithPosOnlyArgs(int argcount, int posonlyargcount, int kwonlyargcount, PyObject *varnames, PyObject *freevars, PyObject *cellvars, PyObject *filename, PyObject *name, PyObject *qualname, int firstlineno, - PyObject *linetable, PyObject *endlinetable, - PyObject *columntable, PyObject *exceptiontable) + PyObject *linetable, + PyObject *exceptiontable) { PyCodeObject *co = NULL; PyObject *localsplusnames = NULL; @@ -482,8 +570,6 @@ PyCode_NewWithPosOnlyArgs(int argcount, int posonlyargcount, int kwonlyargcount, .code = code, .firstlineno = firstlineno, .linetable = linetable, - .endlinetable = endlinetable, - .columntable = columntable, .consts = consts, .names = names, @@ -528,14 +614,16 @@ PyCode_New(int argcount, int kwonlyargcount, PyObject *code, PyObject *consts, PyObject *names, PyObject *varnames, PyObject *freevars, PyObject *cellvars, PyObject *filename, PyObject *name, PyObject *qualname, - int firstlineno, PyObject *linetable, PyObject *endlinetable, - PyObject *columntable, PyObject *exceptiontable) + int firstlineno, + PyObject *linetable, + PyObject *exceptiontable) { return PyCode_NewWithPosOnlyArgs(argcount, 0, kwonlyargcount, nlocals, stacksize, flags, code, consts, names, varnames, freevars, cellvars, filename, - name, qualname, firstlineno, linetable, - endlinetable, columntable, exceptiontable); + name, qualname, firstlineno, + linetable, + exceptiontable); } PyCodeObject * @@ -567,8 +655,6 @@ PyCode_NewEmpty(const char *filename, const char *funcname, int firstlineno) .code = emptystring, .firstlineno = firstlineno, .linetable = emptystring, - .endlinetable = emptystring, - .columntable = emptystring, .consts = nulltuple, .names = nulltuple, .localsplusnames = nulltuple, @@ -605,68 +691,10 @@ PyCode_Addr2Line(PyCodeObject *co, int addrq) return _PyCode_CheckLineNumber(addrq, &bounds); } -int -PyCode_Addr2Location(PyCodeObject *co, int addrq, - int *start_line, int *start_column, - int *end_line, int *end_column) -{ - *start_line = PyCode_Addr2Line(co, addrq); - *start_column = _PyCode_Addr2Offset(co, addrq); - *end_line = _PyCode_Addr2EndLine(co, addrq); - *end_column = _PyCode_Addr2EndOffset(co, addrq); - return 1; -} - -int -_PyCode_Addr2EndLine(PyCodeObject* co, int addrq) -{ - if (addrq < 0) { - return co->co_firstlineno; - } - else if (co->co_endlinetable == Py_None) { - return -1; - } - - assert(addrq >= 0 && addrq < _PyCode_NBYTES(co)); - PyCodeAddressRange bounds; - _PyCode_InitEndAddressRange(co, &bounds); - return _PyCode_CheckLineNumber(addrq, &bounds); -} - -int -_PyCode_Addr2Offset(PyCodeObject* co, int addrq) -{ - if (co->co_columntable == Py_None || addrq < 0) { - return -1; - } - addrq /= sizeof(_Py_CODEUNIT); - if (addrq*2 >= PyBytes_GET_SIZE(co->co_columntable)) { - return -1; - } - - unsigned char* bytes = (unsigned char*)PyBytes_AS_STRING(co->co_columntable); - return bytes[addrq*2] - 1; -} - -int -_PyCode_Addr2EndOffset(PyCodeObject* co, int addrq) -{ - if (co->co_columntable == Py_None || addrq < 0) { - return -1; - } - addrq /= sizeof(_Py_CODEUNIT); - if (addrq*2+1 >= PyBytes_GET_SIZE(co->co_columntable)) { - return -1; - } - - unsigned char* bytes = (unsigned char*)PyBytes_AS_STRING(co->co_columntable); - return bytes[addrq*2+1] - 1; -} - void _PyLineTable_InitAddressRange(const char *linetable, Py_ssize_t length, int firstlineno, PyCodeAddressRange *range) { - range->opaque.lo_next = linetable; + range->opaque.lo_next = (const uint8_t *)linetable; range->opaque.limit = range->opaque.lo_next + length; range->ar_start = -1; range->ar_end = 0; @@ -677,21 +705,13 @@ _PyLineTable_InitAddressRange(const char *linetable, Py_ssize_t length, int firs int _PyCode_InitAddressRange(PyCodeObject* co, PyCodeAddressRange *bounds) { + assert(co->co_linetable != NULL); const char *linetable = PyBytes_AS_STRING(co->co_linetable); Py_ssize_t length = PyBytes_GET_SIZE(co->co_linetable); _PyLineTable_InitAddressRange(linetable, length, co->co_firstlineno, bounds); return bounds->ar_line; } -int -_PyCode_InitEndAddressRange(PyCodeObject* co, PyCodeAddressRange* bounds) -{ - char* linetable = PyBytes_AS_STRING(co->co_endlinetable); - Py_ssize_t length = PyBytes_GET_SIZE(co->co_endlinetable); - _PyLineTable_InitAddressRange(linetable, length, co->co_firstlineno, bounds); - return bounds->ar_line; -} - /* Update *bounds to describe the first and one-past-the-last instructions in the same line as lasti. Return the number of that line, or -1 if lasti is out of bounds. */ int @@ -710,43 +730,182 @@ _PyCode_CheckLineNumber(int lasti, PyCodeAddressRange *bounds) return bounds->ar_line; } +static int +is_no_line_marker(uint8_t b) +{ + return (b >> 3) == 0x1f; +} + + +#define ASSERT_VALID_BOUNDS(bounds) \ + assert(bounds->opaque.lo_next <= bounds->opaque.limit && \ + (bounds->ar_line == -1 || bounds->ar_line == bounds->opaque.computed_line) && \ + (bounds->opaque.lo_next == bounds->opaque.limit || \ + (*bounds->opaque.lo_next) & 128)) + +static int +next_code_delta(PyCodeAddressRange *bounds) +{ + assert((*bounds->opaque.lo_next) & 128); + return (((*bounds->opaque.lo_next) & 7) + 1) * sizeof(_Py_CODEUNIT); +} + +static int +previous_code_delta(PyCodeAddressRange *bounds) +{ + const uint8_t *ptr = bounds->opaque.lo_next-1; + while (((*ptr) & 128) == 0) { + ptr--; + } + return (((*ptr) & 7) + 1) * sizeof(_Py_CODEUNIT); +} + +static int +read_byte(PyCodeAddressRange *bounds) +{ + return *bounds->opaque.lo_next++; +} + +static int +read_varint(PyCodeAddressRange *bounds) +{ + int read = read_byte(bounds); + int val = read & 63; + int shift = 0; + while (read & 64) { + read = read_byte(bounds); + shift += 6; + val |= (read & 63) << shift; + } + return val; +} + +static int +read_signed_varint(PyCodeAddressRange *bounds) +{ + int uval = read_varint(bounds); + if (uval & 1) { + return -(int)(uval >> 1); + } + else { + return uval >> 1; + } +} + static void retreat(PyCodeAddressRange *bounds) { - int ldelta = ((signed char *)bounds->opaque.lo_next)[-1]; - if (ldelta == -128) { - ldelta = 0; - } - bounds->opaque.computed_line -= ldelta; - bounds->opaque.lo_next -= 2; + ASSERT_VALID_BOUNDS(bounds); + assert(bounds->ar_start > 0); + do { + bounds->opaque.lo_next--; + } while (((*bounds->opaque.lo_next) & 128) == 0); + bounds->opaque.computed_line -= get_line_delta(bounds->opaque.lo_next); bounds->ar_end = bounds->ar_start; - bounds->ar_start -= ((unsigned char *)bounds->opaque.lo_next)[-2]; - ldelta = ((signed char *)bounds->opaque.lo_next)[-1]; - if (ldelta == -128) { + bounds->ar_start -= previous_code_delta(bounds); + if (is_no_line_marker(bounds->opaque.lo_next[-1])) { bounds->ar_line = -1; } else { bounds->ar_line = bounds->opaque.computed_line; } + ASSERT_VALID_BOUNDS(bounds); } static void advance(PyCodeAddressRange *bounds) { - bounds->ar_start = bounds->ar_end; - int delta = ((unsigned char *)bounds->opaque.lo_next)[0]; - bounds->ar_end += delta; - int ldelta = ((signed char *)bounds->opaque.lo_next)[1]; - bounds->opaque.lo_next += 2; - if (ldelta == -128) { + ASSERT_VALID_BOUNDS(bounds); + bounds->opaque.computed_line += get_line_delta(bounds->opaque.lo_next); + if (is_no_line_marker(*bounds->opaque.lo_next)) { bounds->ar_line = -1; } else { - bounds->opaque.computed_line += ldelta; bounds->ar_line = bounds->opaque.computed_line; } + bounds->ar_start = bounds->ar_end; + bounds->ar_end += next_code_delta(bounds); + do { + bounds->opaque.lo_next++; + } while (bounds->opaque.lo_next < bounds->opaque.limit && + ((*bounds->opaque.lo_next) & 128) == 0); + ASSERT_VALID_BOUNDS(bounds); } +static void +advance_with_locations(PyCodeAddressRange *bounds, int *endline, int *column, int *endcolumn) +{ + ASSERT_VALID_BOUNDS(bounds); + int first_byte = read_byte(bounds); + int code = (first_byte >> 3) & 15; + bounds->ar_start = bounds->ar_end; + bounds->ar_end = bounds->ar_start + ((first_byte & 7) + 1) * sizeof(_Py_CODEUNIT); + switch(code) { + case PY_CODE_LOCATION_INFO_NONE: + bounds->ar_line = *endline = -1; + *column = *endcolumn = -1; + break; + case PY_CODE_LOCATION_INFO_LONG: + { + bounds->opaque.computed_line += read_signed_varint(bounds); + bounds->ar_line = bounds->opaque.computed_line; + *endline = bounds->ar_line + read_varint(bounds); + *column = read_varint(bounds)-1; + *endcolumn = read_varint(bounds)-1; + break; + } + case PY_CODE_LOCATION_INFO_NO_COLUMNS: + { + /* No column */ + bounds->opaque.computed_line += read_signed_varint(bounds); + *endline = bounds->ar_line = bounds->opaque.computed_line; + *column = *endcolumn = -1; + break; + } + case PY_CODE_LOCATION_INFO_ONE_LINE0: + case PY_CODE_LOCATION_INFO_ONE_LINE1: + case PY_CODE_LOCATION_INFO_ONE_LINE2: + { + /* one line form */ + int line_delta = code - 10; + bounds->opaque.computed_line += line_delta; + *endline = bounds->ar_line = bounds->opaque.computed_line; + *column = read_byte(bounds); + *endcolumn = read_byte(bounds); + break; + } + default: + { + /* Short forms */ + int second_byte = read_byte(bounds); + assert((second_byte & 128) == 0); + *endline = bounds->ar_line = bounds->opaque.computed_line; + *column = code << 3 | (second_byte >> 4); + *endcolumn = *column + (second_byte & 15); + } + } + ASSERT_VALID_BOUNDS(bounds); +} +int +PyCode_Addr2Location(PyCodeObject *co, int addrq, + int *start_line, int *start_column, + int *end_line, int *end_column) +{ + if (addrq < 0) { + *start_line = *end_line = co->co_firstlineno; + *start_column = *end_column = 0; + } + assert(addrq >= 0 && addrq < _PyCode_NBYTES(co)); + PyCodeAddressRange bounds; + _PyCode_InitAddressRange(co, &bounds); + _PyCode_CheckLineNumber(addrq, &bounds); + retreat(&bounds); + advance_with_locations(&bounds, end_line, start_column, end_column); + *start_line = bounds.ar_line; + return 1; +} + + static inline int at_end(PyCodeAddressRange *bounds) { return bounds->opaque.lo_next >= bounds->opaque.limit; @@ -759,10 +918,7 @@ _PyLineTable_PreviousAddressRange(PyCodeAddressRange *range) return 0; } retreat(range); - while (range->ar_start == range->ar_end) { - assert(range->ar_start > 0); - retreat(range); - } + assert(range->ar_end > range->ar_start); return 1; } @@ -773,13 +929,37 @@ _PyLineTable_NextAddressRange(PyCodeAddressRange *range) return 0; } advance(range); - while (range->ar_start == range->ar_end) { - assert(!at_end(range)); - advance(range); - } + assert(range->ar_end > range->ar_start); return 1; } +int +_PyLineTable_StartsLine(PyCodeAddressRange *range) +{ + if (range->ar_start <= 0) { + return 0; + } + const uint8_t *ptr = range->opaque.lo_next; + do { + ptr--; + } while (((*ptr) & 128) == 0); + int code = ((*ptr)>> 3) & 15; + switch(code) { + case PY_CODE_LOCATION_INFO_LONG: + return 0; + case PY_CODE_LOCATION_INFO_NO_COLUMNS: + case PY_CODE_LOCATION_INFO_NONE: + return ptr[1] != 0; + case PY_CODE_LOCATION_INFO_ONE_LINE0: + return 0; + case PY_CODE_LOCATION_INFO_ONE_LINE1: + case PY_CODE_LOCATION_INFO_ONE_LINE2: + return 1; + default: + return 0; + } +} + static int emit_pair(PyObject **bytes, int *offset, int a, int b) { @@ -856,7 +1036,6 @@ typedef struct { PyObject_HEAD PyCodeObject *li_code; PyCodeAddressRange li_line; - char *li_end; } lineiterator; @@ -962,7 +1141,11 @@ new_linesiterator(PyCodeObject *code) typedef struct { PyObject_HEAD PyCodeObject* pi_code; + PyCodeAddressRange pi_range; int pi_offset; + int pi_endline; + int pi_column; + int pi_endcolumn; } positionsiterator; static void @@ -983,22 +1166,19 @@ _source_offset_converter(int* value) { static PyObject* positionsiter_next(positionsiterator* pi) { - if (pi->pi_offset >= _PyCode_NBYTES(pi->pi_code)) { - return NULL; - } - - int start_line, start_col, end_line, end_col; - if (!PyCode_Addr2Location(pi->pi_code, pi->pi_offset, &start_line, - &start_col, &end_line, &end_col)) { - return NULL; + if (pi->pi_offset >= pi->pi_range.ar_end) { + assert(pi->pi_offset == pi->pi_range.ar_end); + if (at_end(&pi->pi_range)) { + return NULL; + } + advance_with_locations(&pi->pi_range, &pi->pi_endline, &pi->pi_column, &pi->pi_endcolumn); } - pi->pi_offset += 2; return Py_BuildValue("(O&O&O&O&)", - _source_offset_converter, &start_line, - _source_offset_converter, &end_line, - _source_offset_converter, &start_col, - _source_offset_converter, &end_col); + _source_offset_converter, &pi->pi_range.ar_line, + _source_offset_converter, &pi->pi_endline, + _source_offset_converter, &pi->pi_column, + _source_offset_converter, &pi->pi_endcolumn); } static PyTypeObject PositionsIterator = { @@ -1053,7 +1233,8 @@ code_positionsiterator(PyCodeObject* code, PyObject* Py_UNUSED(args)) } Py_INCREF(code); pi->pi_code = code; - pi->pi_offset = 0; + _PyCode_InitAddressRange(code, &pi->pi_range); + pi->pi_offset = pi->pi_range.ar_end; return (PyObject*)pi; } @@ -1203,8 +1384,6 @@ code.__new__ as code_new qualname: unicode firstlineno: int linetable: object(subclass_of="&PyBytes_Type") - endlinetable: object - columntable: object exceptiontable: object(subclass_of="&PyBytes_Type") freevars: object(subclass_of="&PyTuple_Type", c_default="NULL") = () cellvars: object(subclass_of="&PyTuple_Type", c_default="NULL") = () @@ -1219,10 +1398,9 @@ code_new_impl(PyTypeObject *type, int argcount, int posonlyargcount, PyObject *code, PyObject *consts, PyObject *names, PyObject *varnames, PyObject *filename, PyObject *name, PyObject *qualname, int firstlineno, PyObject *linetable, - PyObject *endlinetable, PyObject *columntable, PyObject *exceptiontable, PyObject *freevars, PyObject *cellvars) -/*[clinic end generated code: output=e1d2086aa8da7c08 input=a06cd92369134063]*/ +/*[clinic end generated code: output=069fa20d299f9dda input=e31da3c41ad8064a]*/ { PyObject *co = NULL; PyObject *ournames = NULL; @@ -1263,17 +1441,6 @@ code_new_impl(PyTypeObject *type, int argcount, int posonlyargcount, goto cleanup; } - if (!Py_IsNone(endlinetable) && !PyBytes_Check(endlinetable)) { - PyErr_SetString(PyExc_ValueError, - "code: endlinetable must be None or bytes"); - goto cleanup; - } - if (!Py_IsNone(columntable) && !PyBytes_Check(columntable)) { - PyErr_SetString(PyExc_ValueError, - "code: columntable must be None or bytes"); - goto cleanup; - } - ournames = validate_and_copy_tuple(names); if (ournames == NULL) goto cleanup; @@ -1300,8 +1467,8 @@ code_new_impl(PyTypeObject *type, int argcount, int posonlyargcount, ourvarnames, ourfreevars, ourcellvars, filename, name, qualname, firstlineno, - linetable, endlinetable, - columntable, exceptiontable + linetable, + exceptiontable ); cleanup: Py_XDECREF(ournames); @@ -1337,8 +1504,6 @@ code_dealloc(PyCodeObject *co) Py_XDECREF(co->co_name); Py_XDECREF(co->co_qualname); Py_XDECREF(co->co_linetable); - Py_XDECREF(co->co_endlinetable); - Py_XDECREF(co->co_columntable); Py_XDECREF(co->co_exceptiontable); if (co->co_weakreflist != NULL) { PyObject_ClearWeakRefs((PyObject*)co); @@ -1488,8 +1653,6 @@ static PyMemberDef code_memberlist[] = { {"co_qualname", T_OBJECT, OFF(co_qualname), READONLY}, {"co_firstlineno", T_INT, OFF(co_firstlineno), READONLY}, {"co_linetable", T_OBJECT, OFF(co_linetable), READONLY}, - {"co_endlinetable", T_OBJECT, OFF(co_endlinetable), READONLY}, - {"co_columntable", T_OBJECT, OFF(co_columntable), READONLY}, {"co_exceptiontable", T_OBJECT, OFF(co_exceptiontable), READONLY}, {NULL} /* Sentinel */ }; @@ -1585,8 +1748,6 @@ code.replace co_name: unicode(c_default="self->co_name") = None co_qualname: unicode(c_default="self->co_qualname") = None co_linetable: PyBytesObject(c_default="(PyBytesObject *)self->co_linetable") = None - co_endlinetable: object(c_default="self->co_endlinetable") = None - co_columntable: object(c_default="self->co_columntable") = None co_exceptiontable: PyBytesObject(c_default="(PyBytesObject *)self->co_exceptiontable") = None Return a copy of the code object with new values for the specified fields. @@ -1601,9 +1762,9 @@ code_replace_impl(PyCodeObject *self, int co_argcount, PyObject *co_varnames, PyObject *co_freevars, PyObject *co_cellvars, PyObject *co_filename, PyObject *co_name, PyObject *co_qualname, - PyBytesObject *co_linetable, PyObject *co_endlinetable, - PyObject *co_columntable, PyBytesObject *co_exceptiontable) -/*[clinic end generated code: output=f046bf0be3bab91f input=78dbe204dbd06c2f]*/ + PyBytesObject *co_linetable, + PyBytesObject *co_exceptiontable) +/*[clinic end generated code: output=b6cd9988391d5711 input=f6f68e03571f8d7c]*/ { #define CHECK_INT_ARG(ARG) \ if (ARG < 0) { \ @@ -1664,24 +1825,12 @@ code_replace_impl(PyCodeObject *self, int co_argcount, co_freevars = freevars; } - if (!Py_IsNone(co_endlinetable) && !PyBytes_Check(co_endlinetable)) { - PyErr_SetString(PyExc_ValueError, - "co_endlinetable must be None or bytes"); - goto error; - } - if (!Py_IsNone(co_columntable) && !PyBytes_Check(co_columntable)) { - PyErr_SetString(PyExc_ValueError, - "co_columntable must be None or bytes"); - goto error; - } - co = PyCode_NewWithPosOnlyArgs( co_argcount, co_posonlyargcount, co_kwonlyargcount, co_nlocals, co_stacksize, co_flags, (PyObject*)co_code, co_consts, co_names, co_varnames, co_freevars, co_cellvars, co_filename, co_name, - co_qualname, co_firstlineno, (PyObject*)co_linetable, - (PyObject*)co_endlinetable, (PyObject*)co_columntable, - (PyObject*)co_exceptiontable); + co_qualname, co_firstlineno, + (PyObject*)co_linetable, (PyObject*)co_exceptiontable); error: Py_XDECREF(code); diff --git a/Objects/frameobject.c b/Objects/frameobject.c index e65395ee5f20e..7278ca147490f 100644 --- a/Objects/frameobject.c +++ b/Objects/frameobject.c @@ -378,6 +378,7 @@ marklines(PyCodeObject *code, int len) PyCodeAddressRange bounds; _PyCode_InitAddressRange(code, &bounds); assert (bounds.ar_end == 0); + int last_line = -1; int *linestarts = PyMem_New(int, len); if (linestarts == NULL) { @@ -389,7 +390,10 @@ marklines(PyCodeObject *code, int len) while (_PyLineTable_NextAddressRange(&bounds)) { assert(bounds.ar_start / (int)sizeof(_Py_CODEUNIT) < len); - linestarts[bounds.ar_start / sizeof(_Py_CODEUNIT)] = bounds.ar_line; + if (bounds.ar_line != last_line && bounds.ar_line != -1) { + linestarts[bounds.ar_start / sizeof(_Py_CODEUNIT)] = bounds.ar_line; + last_line = bounds.ar_line; + } } return linestarts; } diff --git a/Objects/locations.md b/Objects/locations.md new file mode 100644 index 0000000000000..18a338a95978a --- /dev/null +++ b/Objects/locations.md @@ -0,0 +1,69 @@ +# Locations table + +For versions up to 3.10 see ./lnotab_notes.txt + +In version 3.11 the `co_linetable` bytes object of code objects contains a compact representation of the positions returned by the `co_positions()` iterator. + +The `co_linetable` consists of a sequence of location entries. +Each entry starts with a byte with the most significant bit set, followed by zero or more bytes with most significant bit unset. + +Each entry contains the following information: +* The number of code units covered by this entry (length) +* The start line +* The end line +* The start column +* The end column + +The first byte has the following format: + +Bit 7 | Bits 3-6 | Bits 0-2 + ---- | ---- | ---- + 1 | Code | Length (in code units) - 1 + +The codes are enumerated in the `_PyCodeLocationInfoKind` enum. + +## Variable length integer encodings + +Integers are often encoded using a variable length integer encoding + +### Unsigned integers (varint) + +Unsigned integers are encoded in 6 bit chunks, least significant first. +Each chunk but the last has bit 6 set. +For example: + +* 63 is encoded as `0x3f` +* 200 is encoded as `0x48`, `0x03` + +### Signed integers (svarint) + +Signed integers are encoded by converting them to unsigned integers, using the following function: +```Python +def convert(s): + if s < 0: + return ((-s)<<1) | 1 + else: + return (s<<1) +``` + +## Location entries + +The meaning of the codes and the following bytes are as follows: + +Code | Meaning | Start line | End line | Start column | End column + ---- | ---- | ---- | ---- | ---- | ---- + 0-9 | Short form | ? 0 | ? 0 | See below | See below + 10-12 | One line form | ? (code - 10) | ? 0 | unsigned byte | unsigned byte + 13 | No column info | ? svarint | ? 0 | None | None + 14 | Long form | ? svarint | ? varint | varint | varint + 15 | No location | None | None | None | None + +The ? means the value is encoded as a delta from another value: +* Start line: Delta from the previous start line, or `co_firstlineno` for the first entry. +* End line: Delta from the start line + +### The short forms + +Codes 0-9 are the short forms. The short form consists of two bytes, the second byte holding additional column information. The code is the start column divided by 8 (and rounded down). +* Start column: `(code*8) + ((second_byte>>4)&7)` +* End column: `start_column + (second_byte&15)` diff --git a/Programs/test_frozenmain.h b/Programs/test_frozenmain.h index a3c09529116cc..3034927d7a12b 100644 --- a/Programs/test_frozenmain.h +++ b/Programs/test_frozenmain.h @@ -28,19 +28,15 @@ unsigned char M_test_frozenmain[] = { 107,101,121,169,0,243,0,0,0,0,250,18,116,101,115,116, 95,102,114,111,122,101,110,109,97,105,110,46,112,121,250,8, 60,109,111,100,117,108,101,62,114,11,0,0,0,1,0,0, - 0,115,18,0,0,0,2,128,8,3,8,1,22,2,34,1, - 42,1,8,1,48,7,4,249,115,20,0,0,0,2,128,8, - 3,8,1,22,2,34,1,42,1,2,7,4,1,2,249,52, - 7,115,176,0,0,0,0,0,1,11,1,11,1,11,1,11, - 1,25,1,25,1,25,1,25,1,6,1,6,7,27,1,28, - 1,28,1,28,1,28,1,28,1,28,1,28,1,28,1,6, - 1,6,7,17,19,22,19,27,19,27,19,27,19,27,19,27, - 1,28,1,28,1,28,1,28,1,28,1,28,1,28,1,28, - 10,39,10,27,10,39,10,39,10,39,10,39,10,39,10,41, - 10,41,10,41,10,41,10,41,10,41,10,41,42,50,10,51, - 10,51,10,51,10,51,10,51,1,7,12,2,1,42,1,42, - 5,8,5,10,5,10,11,41,21,24,11,41,11,41,28,34, - 35,38,28,39,28,39,28,39,28,39,28,39,11,41,11,41, - 5,42,5,42,5,42,5,42,5,42,5,42,5,42,5,42, - 5,42,1,42,1,42,114,9,0,0,0, + 0,115,152,0,0,0,248,240,6,0,1,11,128,10,128,10, + 128,10,216,0,24,208,0,24,208,0,24,208,0,24,224,0, + 5,128,5,208,6,26,209,0,27,212,0,27,208,0,27,216, + 0,5,128,5,128,106,144,35,148,40,209,0,27,212,0,27, + 208,0,27,216,9,38,208,9,26,212,9,38,209,9,40,212, + 9,40,168,24,212,9,50,128,6,240,2,6,12,2,240,0, + 7,1,42,240,0,7,1,42,128,67,240,14,0,5,10,128, + 69,208,10,40,144,67,208,10,40,208,10,40,152,54,160,35, + 156,59,208,10,40,208,10,40,209,4,41,212,4,41,208,4, + 41,208,4,41,240,15,7,1,42,240,0,7,1,42,114,9, + 0,0,0, }; diff --git a/Python/compile.c b/Python/compile.c index ceaf85298d70f..d66ee17293ac6 100644 --- a/Python/compile.c +++ b/Python/compile.c @@ -7053,25 +7053,23 @@ compiler_match(struct compiler *c, stmt_ty s) XXX must handle implicit jumps from one block to next */ + struct assembler { PyObject *a_bytecode; /* bytes containing bytecode */ - PyObject *a_lnotab; /* bytes containing lnotab */ - PyObject *a_enotab; /* bytes containing enotab */ - PyObject *a_cnotab; /* bytes containing cnotab */ PyObject *a_except_table; /* bytes containing exception table */ basicblock *a_entry; int a_offset; /* offset into bytecode */ int a_nblocks; /* number of reachable blocks */ int a_except_table_off; /* offset into exception table */ - int a_lnotab_off; /* offset into lnotab */ - int a_enotab_off; /* offset into enotab */ - int a_cnotab_off; /* offset into cnotab */ int a_prevlineno; /* lineno of last emitted line in line table */ int a_prev_end_lineno; /* end_lineno of last emitted line in line table */ int a_lineno; /* lineno of last emitted instruction */ int a_end_lineno; /* end_lineno of last emitted instruction */ int a_lineno_start; /* bytecode start offset of current lineno */ int a_end_lineno_start; /* bytecode start offset of current end_lineno */ + /* Location Info */ + PyObject* a_linetable; /* bytes containing location info */ + int a_location_off; /* offset of last written location info frame */ }; Py_LOCAL_INLINE(void) @@ -7169,25 +7167,15 @@ assemble_init(struct assembler *a, int nblocks, int firstlineno) memset(a, 0, sizeof(struct assembler)); a->a_prevlineno = a->a_lineno = firstlineno; a->a_prev_end_lineno = a->a_end_lineno = firstlineno; - a->a_lnotab = NULL; - a->a_enotab = NULL; - a->a_cnotab = NULL; - a->a_cnotab_off = 0; + a->a_linetable = NULL; + a->a_location_off = 0; a->a_except_table = NULL; a->a_bytecode = PyBytes_FromStringAndSize(NULL, DEFAULT_CODE_SIZE); if (a->a_bytecode == NULL) { goto error; } - a->a_lnotab = PyBytes_FromStringAndSize(NULL, DEFAULT_LNOTAB_SIZE); - if (a->a_lnotab == NULL) { - goto error; - } - a->a_enotab = PyBytes_FromStringAndSize(NULL, DEFAULT_LNOTAB_SIZE); - if (a->a_enotab == NULL) { - goto error; - } - a->a_cnotab = PyBytes_FromStringAndSize(NULL, DEFAULT_CNOTAB_SIZE); - if (a->a_cnotab == NULL) { + a->a_linetable = PyBytes_FromStringAndSize(NULL, DEFAULT_CNOTAB_SIZE); + if (a->a_linetable == NULL) { goto error; } a->a_except_table = PyBytes_FromStringAndSize(NULL, DEFAULT_LNOTAB_SIZE); @@ -7201,9 +7189,7 @@ assemble_init(struct assembler *a, int nblocks, int firstlineno) return 1; error: Py_XDECREF(a->a_bytecode); - Py_XDECREF(a->a_lnotab); - Py_XDECREF(a->a_enotab); - Py_XDECREF(a->a_cnotab); + Py_XDECREF(a->a_linetable); Py_XDECREF(a->a_except_table); return 0; } @@ -7212,9 +7198,7 @@ static void assemble_free(struct assembler *a) { Py_XDECREF(a->a_bytecode); - Py_XDECREF(a->a_lnotab); - Py_XDECREF(a->a_enotab); - Py_XDECREF(a->a_cnotab); + Py_XDECREF(a->a_linetable); Py_XDECREF(a->a_except_table); } @@ -7230,25 +7214,6 @@ blocksize(basicblock *b) return size; } -static int -assemble_emit_table_pair(struct assembler* a, PyObject** table, int* offset, - int left, int right) -{ - Py_ssize_t len = PyBytes_GET_SIZE(*table); - if (*offset + 2 >= len) { - if (_PyBytes_Resize(table, len * 2) < 0) - return 0; - } - unsigned char* table_entry = (unsigned char*)PyBytes_AS_STRING(*table); - - table_entry += *offset; - *offset += 2; - - *table_entry++ = left; - *table_entry++ = right; - return 1; -} - static basicblock * push_except_block(ExceptStack *stack, struct instr *setup) { assert(is_block_push(setup)); @@ -7492,118 +7457,153 @@ assemble_exception_table(struct assembler *a) return 1; } -/* Appends a range to the end of the line number table. See - * Objects/lnotab_notes.txt for the description of the line number table. */ +/* Code location emitting code. See locations.md for a description of the format. */ -static int -assemble_line_range(struct assembler* a, int current, PyObject** table, - int* prev, int* start, int* offset) +#define MSB 0x80 + +static void +write_location_byte(struct assembler* a, int val) { - int ldelta, bdelta; - bdelta = (a->a_offset - *start) * sizeof(_Py_CODEUNIT); - if (bdelta == 0) { - return 1; - } - if (current < 0) { - ldelta = -128; - } - else { - ldelta = current - *prev; - *prev = current; - while (ldelta > 127) { - if (!assemble_emit_table_pair(a, table, offset, 0, 127)) { - return 0; - } - ldelta -= 127; - } - while (ldelta < -127) { - if (!assemble_emit_table_pair(a, table, offset, 0, -127)) { - return 0; - } - ldelta += 127; - } - } - assert(-128 <= ldelta && ldelta < 128); - while (bdelta > 254) { - if (!assemble_emit_table_pair(a, table, offset, 254, ldelta)) { - return 0; - } - ldelta = current < 0 ? -128 : 0; - bdelta -= 254; - } - if (!assemble_emit_table_pair(a, table, offset, bdelta, ldelta)) { - return 0; - } - *start = a->a_offset; - return 1; + PyBytes_AS_STRING(a->a_linetable)[a->a_location_off] = val&255; + a->a_location_off++; } -static int -assemble_start_line_range(struct assembler* a) { - return assemble_line_range(a, a->a_lineno, &a->a_lnotab, - &a->a_prevlineno, &a->a_lineno_start, &a->a_lnotab_off); + +static uint8_t * +location_pointer(struct assembler* a) +{ + return (uint8_t *)PyBytes_AS_STRING(a->a_linetable) + + a->a_location_off; } -static int -assemble_end_line_range(struct assembler* a) { - return assemble_line_range(a, a->a_end_lineno, &a->a_enotab, - &a->a_prev_end_lineno, &a->a_end_lineno_start, &a->a_enotab_off); +static void +write_location_first_byte(struct assembler* a, int code, int length) +{ + a->a_location_off += write_location_entry_start( + location_pointer(a), code, length); } -static int -assemble_lnotab(struct assembler* a, struct instr* i) +static void +write_location_varint(struct assembler* a, unsigned int val) { - if (i->i_lineno == a->a_lineno) { - return 1; - } - if (!assemble_start_line_range(a)) { - return 0; - } - a->a_lineno = i->i_lineno; - return 1; + uint8_t *ptr = location_pointer(a); + a->a_location_off += write_varint(ptr, val); +} + + +static void +write_location_signed_varint(struct assembler* a, int val) +{ + uint8_t *ptr = location_pointer(a); + a->a_location_off += write_signed_varint(ptr, val); +} + +static void +write_location_info_short_form(struct assembler* a, int length, int column, int end_column) +{ + assert(length > 0 && length <= 8); + int column_low_bits = column & 7; + int column_group = column >> 3; + assert(column < 80); + assert(end_column - column < 16); + write_location_first_byte(a, PY_CODE_LOCATION_INFO_SHORT0 + column_group, length); + write_location_byte(a, (column_low_bits << 4) | (end_column - column)); +} + +static void +write_location_info_oneline_form(struct assembler* a, int length, int line_delta, int column, int end_column) +{ + assert(length > 0 && length <= 8); + assert(line_delta >= 0 && line_delta < 3); + assert(column < 128); + assert(end_column < 128); + write_location_first_byte(a, PY_CODE_LOCATION_INFO_ONE_LINE0 + line_delta, length); + write_location_byte(a, column); + write_location_byte(a, end_column); +} + +static void +write_location_info_long_form(struct assembler* a, struct instr* i, int length) +{ + assert(length > 0 && length <= 8); + write_location_first_byte(a, PY_CODE_LOCATION_INFO_LONG, length); + write_location_signed_varint(a, i->i_lineno - a->a_lineno); + assert(i->i_end_lineno >= i->i_lineno); + write_location_varint(a, i->i_end_lineno - i->i_lineno); + write_location_varint(a, i->i_col_offset+1); + write_location_varint(a, i->i_end_col_offset+1); } +static void +write_location_info_none(struct assembler* a, int length) +{ + write_location_first_byte(a, PY_CODE_LOCATION_INFO_NONE, length); +} + +static void +write_location_info_no_column(struct assembler* a, int length, int line_delta) +{ + write_location_first_byte(a, PY_CODE_LOCATION_INFO_NO_COLUMNS, length); + write_location_signed_varint(a, line_delta); +} + +#define THEORETICAL_MAX_ENTRY_SIZE 25 /* 1 + 6 + 6 + 6 + 6 */ + static int -assemble_enotab(struct assembler* a, struct instr* i) +write_location_info_entry(struct assembler* a, struct instr* i, int isize) { - if (i->i_end_lineno == a->a_end_lineno) { + Py_ssize_t len = PyBytes_GET_SIZE(a->a_linetable); + if (a->a_location_off + THEORETICAL_MAX_ENTRY_SIZE >= len) { + assert(len > THEORETICAL_MAX_ENTRY_SIZE); + if (_PyBytes_Resize(&a->a_linetable, len*2) < 0) { + return 0; + } + } + if (i->i_lineno < 0) { + write_location_info_none(a, isize); return 1; } - if (!assemble_end_line_range(a)) { - return 0; + int line_delta = i->i_lineno - a->a_lineno; + int column = i->i_col_offset; + int end_column = i->i_end_col_offset; + assert(column >= -1); + assert(end_column >= -1); + if (column < 0 || end_column < 0) { + if (i->i_end_lineno == i->i_lineno || i->i_end_lineno == -1) { + write_location_info_no_column(a, isize, line_delta); + a->a_lineno = i->i_lineno; + return 1; + } } - a->a_end_lineno = i->i_end_lineno; + else if (i->i_end_lineno == i->i_lineno) { + if (line_delta == 0 && column < 80 && end_column - column < 16) { + write_location_info_short_form(a, isize, column, end_column); + return 1; + } + if (line_delta >= 0 && line_delta < 3 && column < 128 && end_column < 128) { + write_location_info_oneline_form(a, isize, line_delta, column, end_column); + a->a_lineno = i->i_lineno; + return 1; + } + } + write_location_info_long_form(a, i, isize); + a->a_lineno = i->i_lineno; return 1; } static int -assemble_cnotab(struct assembler* a, struct instr* i, int instr_size) +assemble_emit_location(struct assembler* a, struct instr* i) { - Py_ssize_t len = PyBytes_GET_SIZE(a->a_cnotab); - int difference = instr_size * 2; - if (a->a_cnotab_off + difference >= len) { - if (_PyBytes_Resize(&a->a_cnotab, difference + (len * 2)) < 0) { + int isize = instr_size(i); + while (isize > 8) { + if (!write_location_info_entry(a, i, 8)) { return 0; } + isize -= 8; } - - unsigned char* cnotab = (unsigned char*)PyBytes_AS_STRING(a->a_cnotab); - cnotab += a->a_cnotab_off; - a->a_cnotab_off += difference; - - for (int j = 0; j < instr_size; j++) { - if (i->i_col_offset > 255 || i->i_end_col_offset > 255) { - *cnotab++ = 0; - *cnotab++ = 0; - continue; - } - *cnotab++ = i->i_col_offset + 1; - *cnotab++ = i->i_end_col_offset + 1; - } - return 1; + return write_location_info_entry(a, i, isize); } - /* assemble_emit() Extend the bytecode with a new instruction. Update lnotab if necessary. @@ -7616,15 +7616,6 @@ assemble_emit(struct assembler *a, struct instr *i) _Py_CODEUNIT *code; int size = instr_size(i); - if (i->i_lineno && !assemble_lnotab(a, i)) { - return 0; - } - if (!assemble_enotab(a, i)) { - return 0; - } - if (!assemble_cnotab(a, i, size)) { - return 0; - } if (a->a_offset + size >= len / (int)sizeof(_Py_CODEUNIT)) { if (len > PY_SSIZE_T_MAX / 2) return 0; @@ -7976,9 +7967,7 @@ makecode(struct compiler *c, struct assembler *a, PyObject *constslist, .code = a->a_bytecode, .firstlineno = c->u->u_firstlineno, - .linetable = a->a_lnotab, - .endlinetable = a->a_enotab, - .columntable = a->a_cnotab, + .linetable = a->a_linetable, .consts = consts, .names = names, @@ -8413,6 +8402,14 @@ assemble(struct compiler *c, int addNone) goto error; } + /* Emit location info */ + a.a_lineno = c->u->u_firstlineno; + for(b = entryblock; b != NULL; b = b->b_next) { + for (j = 0; j < b->b_iused; j++) + if (!assemble_emit_location(&a, &b->b_instr[j])) + goto error; + } + if (!assemble_exception_table(&a)) { goto error; } @@ -8422,30 +8419,14 @@ assemble(struct compiler *c, int addNone) if (!merge_const_one(c, &a.a_except_table)) { goto error; } - if (!assemble_start_line_range(&a)) { - return 0; - } - if (_PyBytes_Resize(&a.a_lnotab, a.a_lnotab_off) < 0) { - goto error; - } - if (!merge_const_one(c, &a.a_lnotab)) { - goto error; - } - if (!assemble_end_line_range(&a)) { - return 0; - } - if (_PyBytes_Resize(&a.a_enotab, a.a_enotab_off) < 0) { - goto error; - } - if (!merge_const_one(c, &a.a_enotab)) { - goto error; - } - if (_PyBytes_Resize(&a.a_cnotab, a.a_cnotab_off) < 0) { + + if (_PyBytes_Resize(&a.a_linetable, a.a_location_off) < 0) { goto error; } - if (!merge_const_one(c, &a.a_cnotab)) { + if (!merge_const_one(c, &a.a_linetable)) { goto error; } + if (_PyBytes_Resize(&a.a_bytecode, a.a_offset * sizeof(_Py_CODEUNIT)) < 0) { goto error; } diff --git a/Python/marshal.c b/Python/marshal.c index 19abcc8ffe4b7..bbe67e3379fd9 100644 --- a/Python/marshal.c +++ b/Python/marshal.c @@ -564,8 +564,6 @@ w_complex_object(PyObject *v, char flag, WFILE *p) w_object(co->co_qualname, p); w_long(co->co_firstlineno, p); w_object(co->co_linetable, p); - w_object(co->co_endlinetable, p); - w_object(co->co_columntable, p); w_object(co->co_exceptiontable, p); Py_DECREF(co_code); } @@ -1357,9 +1355,7 @@ r_object(RFILE *p) PyObject *name = NULL; PyObject *qualname = NULL; int firstlineno; - PyObject *linetable = NULL; - PyObject* endlinetable = NULL; - PyObject* columntable = NULL; + PyObject* linetable = NULL; PyObject *exceptiontable = NULL; idx = r_ref_reserve(flag, p); @@ -1415,12 +1411,6 @@ r_object(RFILE *p) linetable = r_object(p); if (linetable == NULL) goto code_error; - endlinetable = r_object(p); - if (endlinetable == NULL) - goto code_error; - columntable = r_object(p); - if (columntable == NULL) - goto code_error; exceptiontable = r_object(p); if (exceptiontable == NULL) goto code_error; @@ -1434,8 +1424,6 @@ r_object(RFILE *p) .code = code, .firstlineno = firstlineno, .linetable = linetable, - .endlinetable = endlinetable, - .columntable = columntable, .consts = consts, .names = names, @@ -1473,8 +1461,6 @@ r_object(RFILE *p) Py_XDECREF(name); Py_XDECREF(qualname); Py_XDECREF(linetable); - Py_XDECREF(endlinetable); - Py_XDECREF(columntable); Py_XDECREF(exceptiontable); } retval = v; diff --git a/Tools/gdb/libpython.py b/Tools/gdb/libpython.py index 610d13099432c..857e52f00a06e 100755 --- a/Tools/gdb/libpython.py +++ b/Tools/gdb/libpython.py @@ -634,6 +634,63 @@ def proxyval(self, visited): else: return BuiltInMethodProxy(ml_name, pyop_m_self) +# Python implementation of location table parsing algorithm +def read(it): + return ord(next(it)) + +def read_varint(it): + b = read(it) + val = b & 63; + shift = 0; + while b & 64: + b = read(it) + shift += 6 + val |= (b&63) << shift + return val + +def read_signed_varint(it): + uval = read_varint(it) + if uval & 1: + return -(uval >> 1) + else: + return uval >> 1 + +def parse_location_table(firstlineno, linetable): + line = firstlineno + addr = 0 + it = iter(linetable) + while True: + try: + first_byte = read(it) + except StopIteration: + return + code = (first_byte >> 3) & 15 + length = (first_byte & 7) + 1 + end_addr = addr + length + if code == 15: + yield addr, end_addr, None + addr = end_addr + continue + elif code == 14: # Long form + line_delta = read_signed_varint(it) + line += line_delta + end_line = line + read_varint(it) + col = read_varint(it) + end_col = read_varint(it) + elif code == 13: # No column + line_delta = read_signed_varint(it) + line += line_delta + elif code in (10, 11, 12): # new line + line_delta = code - 10 + line += line_delta + column = read(it) + end_column = read(it) + else: + assert (0 <= code < 10) + second_byte = read(it) + column = code << 3 | (second_byte >> 4) + yield addr, end_addr, line + addr = end_addr class PyCodeObjectPtr(PyObjectPtr): """ @@ -658,18 +715,9 @@ def addr2line(self, addrq): if addrq < 0: return lineno addr = 0 - for addr_incr, line_incr in zip(co_linetable[::2], co_linetable[1::2]): - if addr_incr == 255: - break - addr += ord(addr_incr) - line_delta = ord(line_incr) - if line_delta == 128: - line_delta = 0 - elif line_delta > 128: - line_delta -= 256 - lineno += line_delta - if addr > addrq: - return lineno + for addr, end_addr, line in parse_location_table(lineno, co_linetable): + if addr <= addrq and end_addr > addrq: + return line assert False, "Unreachable" @@ -1082,8 +1130,8 @@ def current_line_num(self): if self.is_optimized_out(): return None try: - return self.co.addr2line(self.f_lasti*2) - except Exception: + return self.co.addr2line(self.f_lasti) + except Exception as ex: # bpo-34989: addr2line() is a complex function, it can fail in many # ways. For example, it fails with a TypeError on "FakeRepr" if # gdb fails to load debug symbols. Use a catch-all "except diff --git a/Tools/scripts/deepfreeze.py b/Tools/scripts/deepfreeze.py index 3c48bac2648d0..5ee6c2f58e599 100644 --- a/Tools/scripts/deepfreeze.py +++ b/Tools/scripts/deepfreeze.py @@ -240,8 +240,6 @@ def generate_code(self, name: str, code: types.CodeType) -> str: co_name = self.generate(name + "_name", code.co_name) co_qualname = self.generate(name + "_qualname", code.co_qualname) co_linetable = self.generate(name + "_linetable", code.co_linetable) - co_endlinetable = self.generate(name + "_endlinetable", code.co_endlinetable) - co_columntable = self.generate(name + "_columntable", code.co_columntable) co_exceptiontable = self.generate(name + "_exceptiontable", code.co_exceptiontable) # These fields are not directly accessible localsplusnames, localspluskinds = get_localsplus(code) @@ -280,8 +278,6 @@ def generate_code(self, name: str, code: types.CodeType) -> str: self.write(f".co_name = {co_name},") self.write(f".co_qualname = {co_qualname},") self.write(f".co_linetable = {co_linetable},") - self.write(f".co_endlinetable = {co_endlinetable},") - self.write(f".co_columntable = {co_columntable},") self.write(f".co_code_adaptive = {co_code_adaptive},") name_as_code = f"(PyCodeObject *)&{name}" self.deallocs.append(f"_PyStaticCode_Dealloc({name_as_code});") diff --git a/Tools/scripts/umarshal.py b/Tools/scripts/umarshal.py index 2eaaa7ce2d95b..f61570cbaff75 100644 --- a/Tools/scripts/umarshal.py +++ b/Tools/scripts/umarshal.py @@ -289,8 +289,6 @@ def R_REF(obj: Any) -> Any: retval.co_qualname = self.r_object() retval.co_firstlineno = self.r_long() retval.co_linetable = self.r_object() - retval.co_endlinetable = self.r_object() - retval.co_columntable = self.r_object() retval.co_exceptiontable = self.r_object() return retval elif type == Type.REF: From webhook-mailer at python.org Thu Apr 21 12:03:31 2022 From: webhook-mailer at python.org (encukou) Date: Thu, 21 Apr 2022 16:03:31 -0000 Subject: [Python-checkins] bpo-47169: Export PyOS_CheckStack on Windows (GH-32414) Message-ID: https://github.com/python/cpython/commit/ac4ffd3be2049591701fee9f054191cce5ab9762 commit: ac4ffd3be2049591701fee9f054191cce5ab9762 branch: main author: Petr Viktorin committer: encukou date: 2022-04-21T18:03:25+02:00 summary: bpo-47169: Export PyOS_CheckStack on Windows (GH-32414) files: A Misc/NEWS.d/next/C API/2022-04-06-16-54-39.bpo-47169.EGzX4B.rst M PC/python3dll.c M Tools/scripts/stable_abi.py diff --git a/Misc/NEWS.d/next/C API/2022-04-06-16-54-39.bpo-47169.EGzX4B.rst b/Misc/NEWS.d/next/C API/2022-04-06-16-54-39.bpo-47169.EGzX4B.rst new file mode 100644 index 0000000000000..df2889161193c --- /dev/null +++ b/Misc/NEWS.d/next/C API/2022-04-06-16-54-39.bpo-47169.EGzX4B.rst @@ -0,0 +1 @@ +:c:func:`PyOS_CheckStack` is now exported in the Stable ABI on Windows. diff --git a/PC/python3dll.c b/PC/python3dll.c index 0aee2aec84726..aabc1e83868e8 100755 --- a/PC/python3dll.c +++ b/PC/python3dll.c @@ -484,6 +484,7 @@ EXPORT_FUNC(PyObject_SetItem) EXPORT_FUNC(PyObject_Size) EXPORT_FUNC(PyObject_Str) EXPORT_FUNC(PyObject_Type) +EXPORT_FUNC(PyOS_CheckStack) EXPORT_FUNC(PyOS_double_to_string) EXPORT_FUNC(PyOS_FSPath) EXPORT_FUNC(PyOS_getsig) diff --git a/Tools/scripts/stable_abi.py b/Tools/scripts/stable_abi.py index 9b90e344977f4..7376a4649ca05 100755 --- a/Tools/scripts/stable_abi.py +++ b/Tools/scripts/stable_abi.py @@ -58,6 +58,7 @@ WINDOWS_IFDEFS = frozenset({ 'MS_WINDOWS', 'PY_HAVE_THREAD_NATIVE_ID', + 'USE_STACKCHECK', }) # The stable ABI manifest (Misc/stable_abi.txt) exists only to fill the From webhook-mailer at python.org Thu Apr 21 13:14:10 2022 From: webhook-mailer at python.org (markshannon) Date: Thu, 21 Apr 2022 17:14:10 -0000 Subject: [Python-checkins] GH-88756: Update docs for PEP 523 eval function type. (GH-91788) Message-ID: https://github.com/python/cpython/commit/5974827c71d884bb3cc58f07a9eaefafe0cbaa6e commit: 5974827c71d884bb3cc58f07a9eaefafe0cbaa6e branch: main author: Mark Shannon committer: markshannon date: 2022-04-21T18:14:01+01:00 summary: GH-88756: Update docs for PEP 523 eval function type. (GH-91788) files: M Doc/c-api/init.rst M Doc/whatsnew/3.11.rst diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst index 3fda9c3af4d2a..9d38fb47d10d6 100644 --- a/Doc/c-api/init.rst +++ b/Doc/c-api/init.rst @@ -1228,7 +1228,7 @@ All of the following functions must be called after :c:func:`Py_Initialize`. .. versionadded:: 3.8 -.. c:type:: PyObject* (*_PyFrameEvalFunction)(PyThreadState *tstate, PyFrameObject *frame, int throwflag) +.. c:type:: PyObject* (*_PyFrameEvalFunction)(PyThreadState *tstate, _PyInterpreterFrame *frame, int throwflag) Type of a frame evaluation function. @@ -1238,6 +1238,9 @@ All of the following functions must be called after :c:func:`Py_Initialize`. .. versionchanged:: 3.9 The function now takes a *tstate* parameter. + .. versionchanged:: 3.11 + The *frame* parameter changed from ``PyFrameObject*`` to ``_PyInterpreterFrame*``. + .. c:function:: _PyFrameEvalFunction _PyInterpreterState_GetEvalFrameFunc(PyInterpreterState *interp) Get the frame evaluation function. diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index 6540a255a0ed8..8d74c9bbebad8 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -1155,6 +1155,10 @@ C API Changes be used for ``size``. (Contributed by Kumar Aditya in :issue:`46608`.) +* :c:func:`_PyFrameEvalFunction` now takes ``_PyInterpreterFrame*`` + as its second parameter, instead of ``PyFrameObject*``. + See :pep:`523` for more details of how to use this function pointer type. + New Features ------------ From webhook-mailer at python.org Thu Apr 21 14:08:46 2022 From: webhook-mailer at python.org (markshannon) Date: Thu, 21 Apr 2022 18:08:46 -0000 Subject: [Python-checkins] GH-88116: Document that PyCodeNew is dangerous, and make PyCode_NewEmpty less dangerous. (GH-91790) Message-ID: https://github.com/python/cpython/commit/d44815cabc0a8d9932df2fa95cb374eadddb7c17 commit: d44815cabc0a8d9932df2fa95cb374eadddb7c17 branch: main author: Mark Shannon committer: markshannon date: 2022-04-21T19:08:36+01:00 summary: GH-88116: Document that PyCodeNew is dangerous, and make PyCode_NewEmpty less dangerous. (GH-91790) files: M Doc/c-api/code.rst M Doc/whatsnew/3.11.rst M Lib/test/test_code.py M Objects/codeobject.c diff --git a/Doc/c-api/code.rst b/Doc/c-api/code.rst index 840b8426dbbdc..407d8b481be4a 100644 --- a/Doc/c-api/code.rst +++ b/Doc/c-api/code.rst @@ -33,24 +33,33 @@ bound into a function. Return the number of free variables in *co*. -.. c:function:: PyCodeObject* PyCode_New(int argcount, int kwonlyargcount, int nlocals, int stacksize, int flags, PyObject *code, PyObject *consts, PyObject *names, PyObject *varnames, PyObject *freevars, PyObject *cellvars, PyObject *filename, PyObject *name, int firstlineno, PyObject *lnotab) +.. c:function:: PyCodeObject* PyCode_New(int argcount, int kwonlyargcount, int nlocals, int stacksize, int flags, PyObject *code, PyObject *consts, PyObject *names, PyObject *varnames, PyObject *freevars, PyObject *cellvars, PyObject *filename, PyObject *name, int firstlineno, PyObject *linetable, PyObject *exceptiontable) Return a new code object. If you need a dummy code object to create a frame, use :c:func:`PyCode_NewEmpty` instead. Calling :c:func:`PyCode_New` directly - can bind you to a precise Python version since the definition of the bytecode - changes often. + will bind you to a precise Python version since the definition of the bytecode + changes often. The many arguments of this function are inter-dependent in complex + ways, meaning that subtle changes to values are likely to result in incorrect + execution or VM crashes. Use this function only with extreme care. -.. c:function:: PyCodeObject* PyCode_NewWithPosOnlyArgs(int argcount, int posonlyargcount, int kwonlyargcount, int nlocals, int stacksize, int flags, PyObject *code, PyObject *consts, PyObject *names, PyObject *varnames, PyObject *freevars, PyObject *cellvars, PyObject *filename, PyObject *name, int firstlineno, PyObject *lnotab) + .. versionchanged:: 3.11 + Added ``exceptiontable`` parameter. + +.. c:function:: PyCodeObject* PyCode_NewWithPosOnlyArgs(int argcount, int posonlyargcount, int kwonlyargcount, int nlocals, int stacksize, int flags, PyObject *code, PyObject *consts, PyObject *names, PyObject *varnames, PyObject *freevars, PyObject *cellvars, PyObject *filename, PyObject *name, int firstlineno, PyObject *linetable, PyObject *exceptiontable) Similar to :c:func:`PyCode_New`, but with an extra "posonlyargcount" for positional-only arguments. + The same caveats that apply to ``PyCode_New`` also apply to this function. .. versionadded:: 3.8 + .. versionchanged:: 3.11 + Added ``exceptiontable`` parameter. + .. c:function:: PyCodeObject* PyCode_NewEmpty(const char *filename, const char *funcname, int firstlineno) Return a new empty code object with the specified filename, - function name, and first line number. It is illegal to - :func:`exec` or :func:`eval` the resulting code object. + function name, and first line number. The resulting code + object will raise an ``Exception`` if executed. .. c:function:: int PyCode_Addr2Line(PyCodeObject *co, int byte_offset) diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index 8d74c9bbebad8..c3a8a7e42a110 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -1159,6 +1159,12 @@ C API Changes as its second parameter, instead of ``PyFrameObject*``. See :pep:`523` for more details of how to use this function pointer type. +* :c:func:`PyCode_New` and :c:func:`PyCode_NewWithPosOnlyArgs` now take + an additional ``exception_table`` argument. + Using these functions should be avoided, if at all possible. + To get a custom code object: create a code object using the compiler, + then get a modified version with the ``replace`` method. + New Features ------------ diff --git a/Lib/test/test_code.py b/Lib/test/test_code.py index a37ebd27dc388..1bb138e7f3243 100644 --- a/Lib/test/test_code.py +++ b/Lib/test/test_code.py @@ -176,6 +176,9 @@ def test_newempty(self): self.assertEqual(co.co_filename, "filename") self.assertEqual(co.co_name, "funcname") self.assertEqual(co.co_firstlineno, 15) + #Empty code object should raise, but not crash the VM + with self.assertRaises(Exception): + exec(co) @cpython_only def test_closure_injection(self): diff --git a/Objects/codeobject.c b/Objects/codeobject.c index 9a57815882756..4fc4b8fec68a2 100644 --- a/Objects/codeobject.c +++ b/Objects/codeobject.c @@ -626,12 +626,20 @@ PyCode_New(int argcount, int kwonlyargcount, exceptiontable); } +static const char assert0[4] = { + LOAD_ASSERTION_ERROR, + 0, + RAISE_VARARGS, + 1 +}; + PyCodeObject * PyCode_NewEmpty(const char *filename, const char *funcname, int firstlineno) { PyObject *nulltuple = NULL; PyObject *filename_ob = NULL; PyObject *funcname_ob = NULL; + PyObject *code_ob = NULL; PyCodeObject *result = NULL; nulltuple = PyTuple_New(0); @@ -646,13 +654,17 @@ PyCode_NewEmpty(const char *filename, const char *funcname, int firstlineno) if (filename_ob == NULL) { goto failed; } + code_ob = PyBytes_FromStringAndSize(assert0, 4); + if (code_ob == NULL) { + goto failed; + } #define emptystring (PyObject *)&_Py_SINGLETON(bytes_empty) struct _PyCodeConstructor con = { .filename = filename_ob, .name = funcname_ob, .qualname = funcname_ob, - .code = emptystring, + .code = code_ob, .firstlineno = firstlineno, .linetable = emptystring, .consts = nulltuple, @@ -660,6 +672,7 @@ PyCode_NewEmpty(const char *filename, const char *funcname, int firstlineno) .localsplusnames = nulltuple, .localspluskinds = emptystring, .exceptiontable = emptystring, + .stacksize = 1, }; result = _PyCode_New(&con); @@ -667,6 +680,7 @@ PyCode_NewEmpty(const char *filename, const char *funcname, int firstlineno) Py_XDECREF(nulltuple); Py_XDECREF(funcname_ob); Py_XDECREF(filename_ob); + Py_XDECREF(code_ob); return result; } From webhook-mailer at python.org Thu Apr 21 14:54:12 2022 From: webhook-mailer at python.org (gvanrossum) Date: Thu, 21 Apr 2022 18:54:12 -0000 Subject: [Python-checkins] GH-91719: Make MSVC generate somewhat faster switch code (#91718) Message-ID: https://github.com/python/cpython/commit/f8dc6186d1857a19edd182277a9d78e6d6cc3787 commit: f8dc6186d1857a19edd182277a9d78e6d6cc3787 branch: main author: Guido van Rossum committer: gvanrossum date: 2022-04-21T11:53:57-07:00 summary: GH-91719: Make MSVC generate somewhat faster switch code (#91718) Apparently a switch on an 8-bit quantity where all cases are present generates a more efficient jump (doing only one indexed memory load instead of two). So we make opcode and use_tracing uint8_t, and generate a macro full of extra `case NNN:` lines for all unused opcodes. See https://github.com/faster-cpython/ideas/issues/321#issuecomment-1103263673 files: M Include/cpython/pystate.h M Include/opcode.h M Python/ceval.c M Tools/scripts/generate_opcode_h.py diff --git a/Include/cpython/pystate.h b/Include/cpython/pystate.h index 1af21a2c947d9..2bd46067cbbe1 100644 --- a/Include/cpython/pystate.h +++ b/Include/cpython/pystate.h @@ -46,7 +46,7 @@ typedef struct _PyCFrame { * discipline and make sure that instances of this struct cannot * accessed outside of their lifetime. */ - int use_tracing; + uint8_t use_tracing; // 0 or 255 (or'ed into opcode, hence 8-bit type) /* Pointer to the currently executing frame (it can be NULL) */ struct _PyInterpreterFrame *current_frame; struct _PyCFrame *previous; diff --git a/Include/opcode.h b/Include/opcode.h index 8db42380cedd1..89f56e27ed437 100644 --- a/Include/opcode.h +++ b/Include/opcode.h @@ -707,6 +707,84 @@ static const char *const _PyOpcode_OpName[256] = { }; #endif +#define EXTRA_CASES \ + case 180: \ + case 181: \ + case 182: \ + case 183: \ + case 184: \ + case 185: \ + case 186: \ + case 187: \ + case 188: \ + case 189: \ + case 190: \ + case 191: \ + case 192: \ + case 193: \ + case 194: \ + case 195: \ + case 196: \ + case 197: \ + case 198: \ + case 199: \ + case 200: \ + case 201: \ + case 202: \ + case 203: \ + case 204: \ + case 205: \ + case 206: \ + case 207: \ + case 208: \ + case 209: \ + case 210: \ + case 211: \ + case 212: \ + case 213: \ + case 214: \ + case 215: \ + case 216: \ + case 217: \ + case 218: \ + case 219: \ + case 220: \ + case 221: \ + case 222: \ + case 223: \ + case 224: \ + case 225: \ + case 226: \ + case 227: \ + case 228: \ + case 229: \ + case 230: \ + case 231: \ + case 232: \ + case 233: \ + case 234: \ + case 235: \ + case 236: \ + case 237: \ + case 238: \ + case 239: \ + case 240: \ + case 241: \ + case 242: \ + case 243: \ + case 244: \ + case 245: \ + case 246: \ + case 247: \ + case 248: \ + case 249: \ + case 250: \ + case 251: \ + case 252: \ + case 253: \ + case 254: \ + ; + #define HAS_ARG(op) ((op) >= HAVE_ARGUMENT) /* Reserve some bytecodes for internal use in the compiler. diff --git a/Python/ceval.c b/Python/ceval.c index 12b0aef774e36..a80583f5abd12 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -1662,7 +1662,9 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int #ifdef Py_STATS int lastopcode = 0; #endif - int opcode; /* Current opcode */ + // opcode is an 8-bit value to improve the code generated by MSVC + // for the big switch below (in combination with the EXTRA_CASES macro). + uint8_t opcode; /* Current opcode */ int oparg; /* Current opcode argument, if any */ _Py_atomic_int * const eval_breaker = &tstate->interp->ceval.eval_breaker; @@ -5645,7 +5647,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int #if USE_COMPUTED_GOTOS _unknown_opcode: #else - default: + EXTRA_CASES // From opcode.h, a 'case' for each unused opcode #endif fprintf(stderr, "XXX lineno: %d, opcode: %d\n", _PyInterpreterFrame_GetLine(frame), opcode); diff --git a/Tools/scripts/generate_opcode_h.py b/Tools/scripts/generate_opcode_h.py index a13fa778db4ea..1b45020835dd4 100644 --- a/Tools/scripts/generate_opcode_h.py +++ b/Tools/scripts/generate_opcode_h.py @@ -129,6 +129,13 @@ def main(opcode_py, outfile='Include/opcode.h'): fobj.write("};\n") fobj.write("#endif\n") + fobj.write("\n") + fobj.write("#define EXTRA_CASES \\\n") + for i, flag in enumerate(used): + if not flag: + fobj.write(f" case {i}: \\\n") + fobj.write(" ;\n") + fobj.write(footer) From webhook-mailer at python.org Thu Apr 21 15:53:29 2022 From: webhook-mailer at python.org (vstinner) Date: Thu, 21 Apr 2022 19:53:29 -0000 Subject: [Python-checkins] gh-89653: PEP 670: Convert unicodeobject.h macros to functions (#91773) Message-ID: https://github.com/python/cpython/commit/128d6241176a879ea9b6e34ea67f1d113d22a1b8 commit: 128d6241176a879ea9b6e34ea67f1d113d22a1b8 branch: main author: Victor Stinner committer: vstinner date: 2022-04-21T21:53:18+02:00 summary: gh-89653: PEP 670: Convert unicodeobject.h macros to functions (#91773) Convert unicodeobject.h macros to static inline functions: * PyUnicode_MAX_CHAR_VALUE() * PyUnicode_READ() * PyUnicode_READY() * PyUnicode_READ_CHAR() * PyUnicode_WRITE() Move PyUnicode_READY() after _PyUnicode_Ready(), since it uses _PyUnicode_Ready(). Static inline functions are wrapped by macros which casts arguments with _PyObject_CAST() and casts 'kind' arguments to "unsigned int" to prevent introducing new compiler warnings when passing "const PyObject*". files: M Include/cpython/unicodeobject.h diff --git a/Include/cpython/unicodeobject.h b/Include/cpython/unicodeobject.h index 9a993a1c7ff0c..c1aebce90129c 100644 --- a/Include/cpython/unicodeobject.h +++ b/Include/cpython/unicodeobject.h @@ -45,7 +45,7 @@ #define Py_UNICODE_ISALPHA(ch) _PyUnicode_IsAlpha(ch) #define Py_UNICODE_ISALNUM(ch) \ - (Py_UNICODE_ISALPHA(ch) || \ + (Py_UNICODE_ISALPHA(ch) || \ Py_UNICODE_ISDECIMAL(ch) || \ Py_UNICODE_ISDIGIT(ch) || \ Py_UNICODE_ISNUMERIC(ch)) @@ -379,80 +379,84 @@ static inline Py_ssize_t PyUnicode_GET_LENGTH(PyObject *op) { } #define PyUnicode_GET_LENGTH(op) PyUnicode_GET_LENGTH(_PyObject_CAST(op)) -/* In the access macros below, "kind" may be evaluated more than once. - All other macro parameters are evaluated exactly once, so it is safe - to put side effects into them (such as increasing the index). */ - -/* Write into the canonical representation, this macro does not do any sanity +/* Write into the canonical representation, this function does not do any sanity checks and is intended for usage in loops. The caller should cache the - kind and data pointers obtained from other macro calls. + kind and data pointers obtained from other function calls. index is the index in the string (starts at 0) and value is the new code point value which should be written to that location. */ +static inline void PyUnicode_WRITE(unsigned int kind, void *data, + Py_ssize_t index, Py_UCS4 value) +{ + if (kind == PyUnicode_1BYTE_KIND) { + ((Py_UCS1 *)data)[index] = (Py_UCS1)value; + } + else if (kind == PyUnicode_2BYTE_KIND) { + ((Py_UCS2 *)data)[index] = (Py_UCS2)value; + } + else { + assert(kind == PyUnicode_4BYTE_KIND); + ((Py_UCS4 *)data)[index] = value; + } +} #define PyUnicode_WRITE(kind, data, index, value) \ - do { \ - switch ((kind)) { \ - case PyUnicode_1BYTE_KIND: { \ - ((Py_UCS1 *)(data))[(index)] = (Py_UCS1)(value); \ - break; \ - } \ - case PyUnicode_2BYTE_KIND: { \ - ((Py_UCS2 *)(data))[(index)] = (Py_UCS2)(value); \ - break; \ - } \ - default: { \ - assert((kind) == PyUnicode_4BYTE_KIND); \ - ((Py_UCS4 *)(data))[(index)] = (Py_UCS4)(value); \ - } \ - } \ - } while (0) + PyUnicode_WRITE((unsigned int)(kind), (void*)(data), (index), (Py_UCS4)(value)) /* Read a code point from the string's canonical representation. No checks or ready calls are performed. */ +static inline Py_UCS4 PyUnicode_READ(unsigned int kind, + const void *data, Py_ssize_t index) +{ + if (kind == PyUnicode_1BYTE_KIND) { + return ((const Py_UCS1 *)data)[index]; + } + if (kind == PyUnicode_2BYTE_KIND) { + return ((const Py_UCS2 *)data)[index]; + } + return ((const Py_UCS4 *)data)[index]; +} #define PyUnicode_READ(kind, data, index) \ - ((Py_UCS4) \ - ((kind) == PyUnicode_1BYTE_KIND ? \ - ((const Py_UCS1 *)(data))[(index)] : \ - ((kind) == PyUnicode_2BYTE_KIND ? \ - ((const Py_UCS2 *)(data))[(index)] : \ - ((const Py_UCS4 *)(data))[(index)] \ - ) \ - )) + PyUnicode_READ((unsigned int)(kind), (const void*)(data), (index)) /* PyUnicode_READ_CHAR() is less efficient than PyUnicode_READ() because it calls PyUnicode_KIND() and might call it twice. For single reads, use PyUnicode_READ_CHAR, for multiple consecutive reads callers should cache kind and use PyUnicode_READ instead. */ +static inline Py_UCS4 PyUnicode_READ_CHAR(PyObject *unicode, Py_ssize_t index) +{ + assert(PyUnicode_IS_READY(unicode)); + unsigned int kind = PyUnicode_KIND(unicode); + if (kind == PyUnicode_1BYTE_KIND) { + return PyUnicode_1BYTE_DATA(unicode)[index]; + } + if (kind == PyUnicode_2BYTE_KIND) { + return PyUnicode_2BYTE_DATA(unicode)[index]; + } + return PyUnicode_4BYTE_DATA(unicode)[index]; +} #define PyUnicode_READ_CHAR(unicode, index) \ - (assert(PyUnicode_IS_READY(unicode)), \ - (Py_UCS4) \ - (PyUnicode_KIND((unicode)) == PyUnicode_1BYTE_KIND ? \ - ((const Py_UCS1 *)(PyUnicode_DATA((unicode))))[(index)] : \ - (PyUnicode_KIND((unicode)) == PyUnicode_2BYTE_KIND ? \ - ((const Py_UCS2 *)(PyUnicode_DATA((unicode))))[(index)] : \ - ((const Py_UCS4 *)(PyUnicode_DATA((unicode))))[(index)] \ - ) \ - )) - -/* PyUnicode_READY() does less work than _PyUnicode_Ready() in the best - case. If the canonical representation is not yet set, it will still call - _PyUnicode_Ready(). - Returns 0 on success and -1 on errors. */ -#define PyUnicode_READY(op) \ - ((PyUnicode_IS_READY(op) ? \ - 0 : _PyUnicode_Ready(_PyObject_CAST(op)))) + PyUnicode_READ_CHAR(_PyObject_CAST(unicode), (index)) /* Return a maximum character value which is suitable for creating another string based on op. This is always an approximation but more efficient than iterating over the string. */ +static inline Py_UCS4 PyUnicode_MAX_CHAR_VALUE(PyObject *op) +{ + assert(PyUnicode_IS_READY(op)); + if (PyUnicode_IS_ASCII(op)) { + return 0x7fU; + } + + unsigned int kind = PyUnicode_KIND(op); + if (kind == PyUnicode_1BYTE_KIND) { + return 0xffU; + } + if (kind == PyUnicode_2BYTE_KIND) { + return 0xffffU; + } + return 0x10ffffU; +} #define PyUnicode_MAX_CHAR_VALUE(op) \ - (assert(PyUnicode_IS_READY(op)), \ - (PyUnicode_IS_ASCII(op) ? \ - (0x7f) : \ - (PyUnicode_KIND(op) == PyUnicode_1BYTE_KIND ? \ - (0xffU) : \ - (PyUnicode_KIND(op) == PyUnicode_2BYTE_KIND ? \ - (0xffffU) : \ - (0x10ffffU))))) + PyUnicode_MAX_CHAR_VALUE(_PyObject_CAST(op)) Py_DEPRECATED(3.3) static inline Py_ssize_t PyUnicode_WSTR_LENGTH(PyObject *op) { @@ -479,12 +483,25 @@ PyAPI_FUNC(PyObject*) PyUnicode_New( objects which were created using the old API to the new flexible format introduced with PEP 393. - Don't call this function directly, use the public PyUnicode_READY() macro + Don't call this function directly, use the public PyUnicode_READY() function instead. */ PyAPI_FUNC(int) _PyUnicode_Ready( PyObject *unicode /* Unicode object */ ); +/* PyUnicode_READY() does less work than _PyUnicode_Ready() in the best + case. If the canonical representation is not yet set, it will still call + _PyUnicode_Ready(). + Returns 0 on success and -1 on errors. */ +static inline int PyUnicode_READY(PyObject *op) +{ + if (PyUnicode_IS_READY(op)) { + return 0; + } + return _PyUnicode_Ready(op); +} +#define PyUnicode_READY(op) PyUnicode_READY(_PyObject_CAST(op)) + /* Get a copy of a Unicode string. */ PyAPI_FUNC(PyObject*) _PyUnicode_Copy( PyObject *unicode From webhook-mailer at python.org Thu Apr 21 16:07:24 2022 From: webhook-mailer at python.org (vstinner) Date: Thu, 21 Apr 2022 20:07:24 -0000 Subject: [Python-checkins] gh-91768: C API no longer use "const PyObject*" type (#91769) Message-ID: https://github.com/python/cpython/commit/eaa85cb22fa2d9e7cd31c2eac29a56cd3a8f2f65 commit: eaa85cb22fa2d9e7cd31c2eac29a56cd3a8f2f65 branch: main author: Victor Stinner committer: vstinner date: 2022-04-21T22:07:19+02:00 summary: gh-91768: C API no longer use "const PyObject*" type (#91769) Py_REFCNT(), Py_TYPE(), Py_SIZE() and Py_IS_TYPE() functions argument type is now "PyObject*", rather than "const PyObject*". * Replace also "const PyObject*" with "PyObject*" in functions: * _Py_strhex_impl() * _Py_strhex_with_sep() * _Py_strhex_bytes_with_sep() * Remove _PyObject_CAST_CONST() and _PyVarObject_CAST_CONST() macros. * Py_IS_TYPE() can now use Py_TYPE() in its implementation. files: A Misc/NEWS.d/next/C API/2022-04-21-01-48-22.gh-issue-91768.x_aKzv.rst M Doc/c-api/structures.rst M Include/internal/pycore_strhex.h M Include/object.h M Python/pystrhex.c diff --git a/Doc/c-api/structures.rst b/Doc/c-api/structures.rst index aa74f6cce1ac7..ff5ecf24072c1 100644 --- a/Doc/c-api/structures.rst +++ b/Doc/c-api/structures.rst @@ -62,14 +62,14 @@ the definition of all other Python objects. See documentation of :c:type:`PyVarObject` above. -.. c:function:: int Py_Is(const PyObject *x, const PyObject *y) +.. c:function:: int Py_Is(PyObject *x, PyObject *y) Test if the *x* object is the *y* object, the same as ``x is y`` in Python. .. versionadded:: 3.10 -.. c:function:: int Py_IsNone(const PyObject *x) +.. c:function:: int Py_IsNone(PyObject *x) Test if an object is the ``None`` singleton, the same as ``x is None`` in Python. @@ -77,7 +77,7 @@ the definition of all other Python objects. .. versionadded:: 3.10 -.. c:function:: int Py_IsTrue(const PyObject *x) +.. c:function:: int Py_IsTrue(PyObject *x) Test if an object is the ``True`` singleton, the same as ``x is True`` in Python. @@ -85,7 +85,7 @@ the definition of all other Python objects. .. versionadded:: 3.10 -.. c:function:: int Py_IsFalse(const PyObject *x) +.. c:function:: int Py_IsFalse(PyObject *x) Test if an object is the ``False`` singleton, the same as ``x is False`` in Python. @@ -93,7 +93,7 @@ the definition of all other Python objects. .. versionadded:: 3.10 -.. c:function:: PyTypeObject* Py_TYPE(const PyObject *o) +.. c:function:: PyTypeObject* Py_TYPE(PyObject *o) Get the type of the Python object *o*. @@ -103,6 +103,7 @@ the definition of all other Python objects. .. versionchanged:: 3.11 :c:func:`Py_TYPE()` is changed to an inline static function. + The parameter type is no longer :c:type:`const PyObject*`. .. c:function:: int Py_IS_TYPE(PyObject *o, PyTypeObject *type) @@ -120,12 +121,15 @@ the definition of all other Python objects. .. versionadded:: 3.9 -.. c:function:: Py_ssize_t Py_REFCNT(const PyObject *o) +.. c:function:: Py_ssize_t Py_REFCNT(PyObject *o) Get the reference count of the Python object *o*. Use the :c:func:`Py_SET_REFCNT()` function to set an object reference count. + .. versionchanged:: 3.11 + The parameter type is no longer :c:type:`const PyObject*`. + .. versionchanged:: 3.10 :c:func:`Py_REFCNT()` is changed to the inline static function. @@ -137,7 +141,7 @@ the definition of all other Python objects. .. versionadded:: 3.9 -.. c:function:: Py_ssize_t Py_SIZE(const PyVarObject *o) +.. c:function:: Py_ssize_t Py_SIZE(PyVarObject *o) Get the size of the Python object *o*. @@ -145,6 +149,7 @@ the definition of all other Python objects. .. versionchanged:: 3.11 :c:func:`Py_SIZE()` is changed to an inline static function. + The parameter type is no longer :c:type:`const PyVarObject*`. .. c:function:: void Py_SET_SIZE(PyVarObject *o, Py_ssize_t size) diff --git a/Include/internal/pycore_strhex.h b/Include/internal/pycore_strhex.h index 1633671da0f4c..f427b4d695bd2 100644 --- a/Include/internal/pycore_strhex.h +++ b/Include/internal/pycore_strhex.h @@ -22,12 +22,12 @@ PyAPI_FUNC(PyObject*) _Py_strhex_bytes( PyAPI_FUNC(PyObject*) _Py_strhex_with_sep( const char* argbuf, const Py_ssize_t arglen, - const PyObject* sep, + PyObject* sep, const int bytes_per_group); PyAPI_FUNC(PyObject*) _Py_strhex_bytes_with_sep( const char* argbuf, const Py_ssize_t arglen, - const PyObject* sep, + PyObject* sep, const int bytes_per_group); #ifdef __cplusplus diff --git a/Include/object.h b/Include/object.h index 0b4b55ea1ded9..8a45ddbf6057e 100644 --- a/Include/object.h +++ b/Include/object.h @@ -105,7 +105,6 @@ struct _object { /* Cast argument to PyObject* type. */ #define _PyObject_CAST(op) ((PyObject*)(op)) -#define _PyObject_CAST_CONST(op) ((const PyObject*)(op)) typedef struct { PyObject ob_base; @@ -114,7 +113,6 @@ typedef struct { /* Cast argument to PyVarObject* type. */ #define _PyVarObject_CAST(op) ((PyVarObject*)(op)) -#define _PyVarObject_CAST_CONST(op) ((const PyVarObject*)(op)) // Test if the 'x' object is the 'y' object, the same as "x is y" in Python. @@ -122,31 +120,29 @@ PyAPI_FUNC(int) Py_Is(PyObject *x, PyObject *y); #define Py_Is(x, y) ((x) == (y)) -static inline Py_ssize_t Py_REFCNT(const PyObject *ob) { +static inline Py_ssize_t Py_REFCNT(PyObject *ob) { return ob->ob_refcnt; } -#define Py_REFCNT(ob) Py_REFCNT(_PyObject_CAST_CONST(ob)) +#define Py_REFCNT(ob) Py_REFCNT(_PyObject_CAST(ob)) // bpo-39573: The Py_SET_TYPE() function must be used to set an object type. -static inline PyTypeObject* Py_TYPE(const PyObject *ob) { +static inline PyTypeObject* Py_TYPE(PyObject *ob) { return ob->ob_type; } -#define Py_TYPE(ob) Py_TYPE(_PyObject_CAST_CONST(ob)) +#define Py_TYPE(ob) Py_TYPE(_PyObject_CAST(ob)) // bpo-39573: The Py_SET_SIZE() function must be used to set an object size. -static inline Py_ssize_t Py_SIZE(const PyVarObject *ob) { +static inline Py_ssize_t Py_SIZE(PyVarObject *ob) { return ob->ob_size; } -#define Py_SIZE(ob) Py_SIZE(_PyVarObject_CAST_CONST(ob)) +#define Py_SIZE(ob) Py_SIZE(_PyVarObject_CAST(ob)) -static inline int Py_IS_TYPE(const PyObject *ob, const PyTypeObject *type) { - // bpo-44378: Don't use Py_TYPE() since Py_TYPE() requires a non-const - // object. - return ob->ob_type == type; +static inline int Py_IS_TYPE(PyObject *ob, PyTypeObject *type) { + return Py_TYPE(ob) == type; } -#define Py_IS_TYPE(ob, type) Py_IS_TYPE(_PyObject_CAST_CONST(ob), type) +#define Py_IS_TYPE(ob, type) Py_IS_TYPE(_PyObject_CAST(ob), type) static inline void Py_SET_REFCNT(PyObject *ob, Py_ssize_t refcnt) { diff --git a/Misc/NEWS.d/next/C API/2022-04-21-01-48-22.gh-issue-91768.x_aKzv.rst b/Misc/NEWS.d/next/C API/2022-04-21-01-48-22.gh-issue-91768.x_aKzv.rst new file mode 100644 index 0000000000000..434230693211c --- /dev/null +++ b/Misc/NEWS.d/next/C API/2022-04-21-01-48-22.gh-issue-91768.x_aKzv.rst @@ -0,0 +1,3 @@ +:c:func:`Py_REFCNT`, :c:func:`Py_TYPE`, :c:func:`Py_SIZE` and +:c:func:`Py_IS_TYPE` functions argument type is now ``PyObject*``, rather +than ``const PyObject*``. Patch by Victor Stinner. diff --git a/Python/pystrhex.c b/Python/pystrhex.c index 880af44ea0e94..e4f06d766390e 100644 --- a/Python/pystrhex.c +++ b/Python/pystrhex.c @@ -5,7 +5,7 @@ #include // abs() static PyObject *_Py_strhex_impl(const char* argbuf, const Py_ssize_t arglen, - const PyObject* sep, int bytes_per_sep_group, + PyObject* sep, int bytes_per_sep_group, const int return_bytes) { assert(arglen >= 0); @@ -152,21 +152,23 @@ PyObject * _Py_strhex(const char* argbuf, const Py_ssize_t arglen) /* Same as above but returns a bytes() instead of str() to avoid the * need to decode the str() when bytes are needed. */ -PyObject * _Py_strhex_bytes(const char* argbuf, const Py_ssize_t arglen) +PyObject* _Py_strhex_bytes(const char* argbuf, const Py_ssize_t arglen) { return _Py_strhex_impl(argbuf, arglen, NULL, 0, 1); } /* These variants include support for a separator between every N bytes: */ -PyObject * _Py_strhex_with_sep(const char* argbuf, const Py_ssize_t arglen, const PyObject* sep, const int bytes_per_group) +PyObject* _Py_strhex_with_sep(const char* argbuf, const Py_ssize_t arglen, + PyObject* sep, const int bytes_per_group) { return _Py_strhex_impl(argbuf, arglen, sep, bytes_per_group, 0); } /* Same as above but returns a bytes() instead of str() to avoid the * need to decode the str() when bytes are needed. */ -PyObject * _Py_strhex_bytes_with_sep(const char* argbuf, const Py_ssize_t arglen, const PyObject* sep, const int bytes_per_group) +PyObject* _Py_strhex_bytes_with_sep(const char* argbuf, const Py_ssize_t arglen, + PyObject* sep, const int bytes_per_group) { return _Py_strhex_impl(argbuf, arglen, sep, bytes_per_group, 1); } From webhook-mailer at python.org Thu Apr 21 16:44:33 2022 From: webhook-mailer at python.org (vstinner) Date: Thu, 21 Apr 2022 20:44:33 -0000 Subject: [Python-checkins] gh-89653: PEP 670: Group deprecated API in unicodeobject.h (#91796) Message-ID: https://github.com/python/cpython/commit/c1474fa6c6496e2bf0fd341ccd735ab2fd6d7764 commit: c1474fa6c6496e2bf0fd341ccd735ab2fd6d7764 branch: main author: Victor Stinner committer: vstinner date: 2022-04-21T22:44:23+02:00 summary: gh-89653: PEP 670: Group deprecated API in unicodeobject.h (#91796) files: M Include/cpython/unicodeobject.h diff --git a/Include/cpython/unicodeobject.h b/Include/cpython/unicodeobject.h index c1aebce90129c..0897e66c56d07 100644 --- a/Include/cpython/unicodeobject.h +++ b/Include/cpython/unicodeobject.h @@ -243,40 +243,6 @@ PyAPI_FUNC(int) _PyUnicode_CheckConsistency( (assert(PyUnicode_Check(op)), (PyUnicodeObject*)(op)) -/* Fast access macros */ - -/* Returns the deprecated Py_UNICODE representation's size in code units - (this includes surrogate pairs as 2 units). - If the Py_UNICODE representation is not available, it will be computed - on request. Use PyUnicode_GET_LENGTH() for the length in code points. */ - -/* Py_DEPRECATED(3.3) */ -#define PyUnicode_GET_SIZE(op) \ - (_PyASCIIObject_CAST(op)->wstr ? \ - PyUnicode_WSTR_LENGTH(op) : \ - ((void)PyUnicode_AsUnicode(_PyObject_CAST(op)),\ - assert(_PyASCIIObject_CAST(op)->wstr), \ - PyUnicode_WSTR_LENGTH(op))) - -/* Py_DEPRECATED(3.3) */ -#define PyUnicode_GET_DATA_SIZE(op) \ - (PyUnicode_GET_SIZE(op) * Py_UNICODE_SIZE) - -/* Alias for PyUnicode_AsUnicode(). This will create a wchar_t/Py_UNICODE - representation on demand. Using this macro is very inefficient now, - try to port your code to use the new PyUnicode_*BYTE_DATA() macros or - use PyUnicode_WRITE() and PyUnicode_READ(). */ - -/* Py_DEPRECATED(3.3) */ -#define PyUnicode_AS_UNICODE(op) \ - (_PyASCIIObject_CAST(op)->wstr ? _PyASCIIObject_CAST(op)->wstr : \ - PyUnicode_AsUnicode(_PyObject_CAST(op))) - -/* Py_DEPRECATED(3.3) */ -#define PyUnicode_AS_DATA(op) \ - ((const char *)(PyUnicode_AS_UNICODE(op))) - - /* --- Flexible String Representation Helper Macros (PEP 393) -------------- */ /* Values for PyASCIIObject.state: */ @@ -458,14 +424,6 @@ static inline Py_UCS4 PyUnicode_MAX_CHAR_VALUE(PyObject *op) #define PyUnicode_MAX_CHAR_VALUE(op) \ PyUnicode_MAX_CHAR_VALUE(_PyObject_CAST(op)) -Py_DEPRECATED(3.3) -static inline Py_ssize_t PyUnicode_WSTR_LENGTH(PyObject *op) { - return PyUnicode_IS_COMPACT_ASCII(op) ? - _PyASCIIObject_CAST(op)->length : - _PyCompactUnicodeObject_CAST(op)->wstr_length; -} -#define PyUnicode_WSTR_LENGTH(op) PyUnicode_WSTR_LENGTH(_PyObject_CAST(op)) - /* === Public API ========================================================= */ /* --- Plain Py_UNICODE --------------------------------------------------- */ @@ -568,20 +526,6 @@ PyAPI_FUNC(void) _PyUnicode_FastFill( Py_UCS4 fill_char ); -/* Create a Unicode Object from the Py_UNICODE buffer u of the given - size. - - u may be NULL which causes the contents to be undefined. It is the - user's responsibility to fill in the needed data afterwards. Note - that modifying the Unicode object contents after construction is - only allowed if u was set to NULL. - - The buffer is copied into the new object. */ -Py_DEPRECATED(3.3) PyAPI_FUNC(PyObject*) PyUnicode_FromUnicode( - const Py_UNICODE *u, /* Unicode buffer */ - Py_ssize_t size /* size of buffer */ - ); - /* Create a new string from a buffer of Py_UCS1, Py_UCS2 or Py_UCS4 characters. Scan the string to find the maximum character. */ PyAPI_FUNC(PyObject*) PyUnicode_FromKindAndData( @@ -602,6 +546,22 @@ PyAPI_FUNC(Py_UCS4) _PyUnicode_FindMaxChar ( Py_ssize_t start, Py_ssize_t end); +/* --- Legacy deprecated API ---------------------------------------------- */ + +/* Create a Unicode Object from the Py_UNICODE buffer u of the given + size. + + u may be NULL which causes the contents to be undefined. It is the + user's responsibility to fill in the needed data afterwards. Note + that modifying the Unicode object contents after construction is + only allowed if u was set to NULL. + + The buffer is copied into the new object. */ +Py_DEPRECATED(3.3) PyAPI_FUNC(PyObject*) PyUnicode_FromUnicode( + const Py_UNICODE *u, /* Unicode buffer */ + Py_ssize_t size /* size of buffer */ + ); + /* Return a read-only pointer to the Unicode object's internal Py_UNICODE buffer. If the wchar_t/Py_UNICODE representation is not yet available, this @@ -627,6 +587,48 @@ Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE *) PyUnicode_AsUnicodeAndSize( ); +/* Fast access macros */ + +Py_DEPRECATED(3.3) +static inline Py_ssize_t PyUnicode_WSTR_LENGTH(PyObject *op) { + return PyUnicode_IS_COMPACT_ASCII(op) ? + _PyASCIIObject_CAST(op)->length : + _PyCompactUnicodeObject_CAST(op)->wstr_length; +} +#define PyUnicode_WSTR_LENGTH(op) PyUnicode_WSTR_LENGTH(_PyObject_CAST(op)) + +/* Returns the deprecated Py_UNICODE representation's size in code units + (this includes surrogate pairs as 2 units). + If the Py_UNICODE representation is not available, it will be computed + on request. Use PyUnicode_GET_LENGTH() for the length in code points. */ + +/* Py_DEPRECATED(3.3) */ +#define PyUnicode_GET_SIZE(op) \ + (_PyASCIIObject_CAST(op)->wstr ? \ + PyUnicode_WSTR_LENGTH(op) : \ + ((void)PyUnicode_AsUnicode(_PyObject_CAST(op)),\ + assert(_PyASCIIObject_CAST(op)->wstr), \ + PyUnicode_WSTR_LENGTH(op))) + +/* Py_DEPRECATED(3.3) */ +#define PyUnicode_GET_DATA_SIZE(op) \ + (PyUnicode_GET_SIZE(op) * Py_UNICODE_SIZE) + +/* Alias for PyUnicode_AsUnicode(). This will create a wchar_t/Py_UNICODE + representation on demand. Using this macro is very inefficient now, + try to port your code to use the new PyUnicode_*BYTE_DATA() macros or + use PyUnicode_WRITE() and PyUnicode_READ(). */ + +/* Py_DEPRECATED(3.3) */ +#define PyUnicode_AS_UNICODE(op) \ + (_PyASCIIObject_CAST(op)->wstr ? _PyASCIIObject_CAST(op)->wstr : \ + PyUnicode_AsUnicode(_PyObject_CAST(op))) + +/* Py_DEPRECATED(3.3) */ +#define PyUnicode_AS_DATA(op) \ + ((const char *)(PyUnicode_AS_UNICODE(op))) + + /* --- _PyUnicodeWriter API ----------------------------------------------- */ typedef struct { From webhook-mailer at python.org Thu Apr 21 17:00:33 2022 From: webhook-mailer at python.org (vstinner) Date: Thu, 21 Apr 2022 21:00:33 -0000 Subject: [Python-checkins] gh-79315: Add Include/cpython/modsupport.h header (#91797) Message-ID: https://github.com/python/cpython/commit/6f9addb5ba0729e4bd2a1919d02e55c25382e26e commit: 6f9addb5ba0729e4bd2a1919d02e55c25382e26e branch: main author: Victor Stinner committer: vstinner date: 2022-04-21T23:00:26+02:00 summary: gh-79315: Add Include/cpython/modsupport.h header (#91797) files: A Include/cpython/modsupport.h M Include/modsupport.h M Makefile.pre.in M PCbuild/pythoncore.vcxproj M PCbuild/pythoncore.vcxproj.filters diff --git a/Include/cpython/modsupport.h b/Include/cpython/modsupport.h new file mode 100644 index 0000000000000..769eb52bf6e3a --- /dev/null +++ b/Include/cpython/modsupport.h @@ -0,0 +1,107 @@ +#ifndef Py_CPYTHON_MODSUPPORT_H +# error "this header file must not be included directly" +#endif + +/* If PY_SSIZE_T_CLEAN is defined, each functions treats #-specifier + to mean Py_ssize_t */ +#ifdef PY_SSIZE_T_CLEAN +#define _Py_VaBuildStack _Py_VaBuildStack_SizeT +#else +PyAPI_FUNC(PyObject *) _Py_VaBuildValue_SizeT(const char *, va_list); +PyAPI_FUNC(PyObject **) _Py_VaBuildStack_SizeT( + PyObject **small_stack, + Py_ssize_t small_stack_len, + const char *format, + va_list va, + Py_ssize_t *p_nargs); +#endif + +PyAPI_FUNC(int) _PyArg_UnpackStack( + PyObject *const *args, + Py_ssize_t nargs, + const char *name, + Py_ssize_t min, + Py_ssize_t max, + ...); + +PyAPI_FUNC(int) _PyArg_NoKeywords(const char *funcname, PyObject *kwargs); +PyAPI_FUNC(int) _PyArg_NoKwnames(const char *funcname, PyObject *kwnames); +PyAPI_FUNC(int) _PyArg_NoPositional(const char *funcname, PyObject *args); +#define _PyArg_NoKeywords(funcname, kwargs) \ + ((kwargs) == NULL || _PyArg_NoKeywords((funcname), (kwargs))) +#define _PyArg_NoKwnames(funcname, kwnames) \ + ((kwnames) == NULL || _PyArg_NoKwnames((funcname), (kwnames))) +#define _PyArg_NoPositional(funcname, args) \ + ((args) == NULL || _PyArg_NoPositional((funcname), (args))) + +PyAPI_FUNC(void) _PyArg_BadArgument(const char *, const char *, const char *, PyObject *); +PyAPI_FUNC(int) _PyArg_CheckPositional(const char *, Py_ssize_t, + Py_ssize_t, Py_ssize_t); +#define _PyArg_CheckPositional(funcname, nargs, min, max) \ + ((!ANY_VARARGS(max) && (min) <= (nargs) && (nargs) <= (max)) \ + || _PyArg_CheckPositional((funcname), (nargs), (min), (max))) + +PyAPI_FUNC(PyObject **) _Py_VaBuildStack( + PyObject **small_stack, + Py_ssize_t small_stack_len, + const char *format, + va_list va, + Py_ssize_t *p_nargs); + +typedef struct _PyArg_Parser { + const char *format; + const char * const *keywords; + const char *fname; + const char *custom_msg; + int pos; /* number of positional-only arguments */ + int min; /* minimal number of arguments */ + int max; /* maximal number of positional arguments */ + PyObject *kwtuple; /* tuple of keyword parameter names */ + struct _PyArg_Parser *next; +} _PyArg_Parser; + +#ifdef PY_SSIZE_T_CLEAN +#define _PyArg_ParseTupleAndKeywordsFast _PyArg_ParseTupleAndKeywordsFast_SizeT +#define _PyArg_ParseStack _PyArg_ParseStack_SizeT +#define _PyArg_ParseStackAndKeywords _PyArg_ParseStackAndKeywords_SizeT +#define _PyArg_VaParseTupleAndKeywordsFast _PyArg_VaParseTupleAndKeywordsFast_SizeT +#endif + +PyAPI_FUNC(int) _PyArg_ParseTupleAndKeywordsFast(PyObject *, PyObject *, + struct _PyArg_Parser *, ...); +PyAPI_FUNC(int) _PyArg_ParseStack( + PyObject *const *args, + Py_ssize_t nargs, + const char *format, + ...); +PyAPI_FUNC(int) _PyArg_ParseStackAndKeywords( + PyObject *const *args, + Py_ssize_t nargs, + PyObject *kwnames, + struct _PyArg_Parser *, + ...); +PyAPI_FUNC(int) _PyArg_VaParseTupleAndKeywordsFast(PyObject *, PyObject *, + struct _PyArg_Parser *, va_list); +PyAPI_FUNC(PyObject * const *) _PyArg_UnpackKeywords( + PyObject *const *args, Py_ssize_t nargs, + PyObject *kwargs, PyObject *kwnames, + struct _PyArg_Parser *parser, + int minpos, int maxpos, int minkw, + PyObject **buf); + +PyAPI_FUNC(PyObject * const *) _PyArg_UnpackKeywordsWithVararg( + PyObject *const *args, Py_ssize_t nargs, + PyObject *kwargs, PyObject *kwnames, + struct _PyArg_Parser *parser, + int minpos, int maxpos, int minkw, + int vararg, PyObject **buf); + +#define _PyArg_UnpackKeywords(args, nargs, kwargs, kwnames, parser, minpos, maxpos, minkw, buf) \ + (((minkw) == 0 && (kwargs) == NULL && (kwnames) == NULL && \ + (minpos) <= (nargs) && (nargs) <= (maxpos) && args != NULL) ? (args) : \ + _PyArg_UnpackKeywords((args), (nargs), (kwargs), (kwnames), (parser), \ + (minpos), (maxpos), (minkw), (buf))) + +PyAPI_FUNC(PyObject *) _PyModule_CreateInitialized(PyModuleDef*, int apiver); + +PyAPI_DATA(const char *) _Py_PackageContext; diff --git a/Include/modsupport.h b/Include/modsupport.h index 3cfefb413ca55..0e96a5c988846 100644 --- a/Include/modsupport.h +++ b/Include/modsupport.h @@ -19,19 +19,6 @@ extern "C" { #define PyArg_VaParseTupleAndKeywords _PyArg_VaParseTupleAndKeywords_SizeT #define Py_BuildValue _Py_BuildValue_SizeT #define Py_VaBuildValue _Py_VaBuildValue_SizeT -#ifndef Py_LIMITED_API -#define _Py_VaBuildStack _Py_VaBuildStack_SizeT -#endif -#else -#ifndef Py_LIMITED_API -PyAPI_FUNC(PyObject *) _Py_VaBuildValue_SizeT(const char *, va_list); -PyAPI_FUNC(PyObject **) _Py_VaBuildStack_SizeT( - PyObject **small_stack, - Py_ssize_t small_stack_len, - const char *format, - va_list va, - Py_ssize_t *p_nargs); -#endif /* !Py_LIMITED_API */ #endif /* Due to a glitch in 3.2, the _SizeT versions weren't exported from the DLL. */ @@ -51,97 +38,8 @@ PyAPI_FUNC(PyObject *) _Py_BuildValue_SizeT(const char *, ...); #define ANY_VARARGS(n) (n == PY_SSIZE_T_MAX) -#ifndef Py_LIMITED_API -PyAPI_FUNC(int) _PyArg_UnpackStack( - PyObject *const *args, - Py_ssize_t nargs, - const char *name, - Py_ssize_t min, - Py_ssize_t max, - ...); - -PyAPI_FUNC(int) _PyArg_NoKeywords(const char *funcname, PyObject *kwargs); -PyAPI_FUNC(int) _PyArg_NoKwnames(const char *funcname, PyObject *kwnames); -PyAPI_FUNC(int) _PyArg_NoPositional(const char *funcname, PyObject *args); -#define _PyArg_NoKeywords(funcname, kwargs) \ - ((kwargs) == NULL || _PyArg_NoKeywords((funcname), (kwargs))) -#define _PyArg_NoKwnames(funcname, kwnames) \ - ((kwnames) == NULL || _PyArg_NoKwnames((funcname), (kwnames))) -#define _PyArg_NoPositional(funcname, args) \ - ((args) == NULL || _PyArg_NoPositional((funcname), (args))) - -PyAPI_FUNC(void) _PyArg_BadArgument(const char *, const char *, const char *, PyObject *); -PyAPI_FUNC(int) _PyArg_CheckPositional(const char *, Py_ssize_t, - Py_ssize_t, Py_ssize_t); -#define _PyArg_CheckPositional(funcname, nargs, min, max) \ - ((!ANY_VARARGS(max) && (min) <= (nargs) && (nargs) <= (max)) \ - || _PyArg_CheckPositional((funcname), (nargs), (min), (max))) - -#endif PyAPI_FUNC(PyObject *) Py_VaBuildValue(const char *, va_list); -#ifndef Py_LIMITED_API -PyAPI_FUNC(PyObject **) _Py_VaBuildStack( - PyObject **small_stack, - Py_ssize_t small_stack_len, - const char *format, - va_list va, - Py_ssize_t *p_nargs); -#endif - -#ifndef Py_LIMITED_API -typedef struct _PyArg_Parser { - const char *format; - const char * const *keywords; - const char *fname; - const char *custom_msg; - int pos; /* number of positional-only arguments */ - int min; /* minimal number of arguments */ - int max; /* maximal number of positional arguments */ - PyObject *kwtuple; /* tuple of keyword parameter names */ - struct _PyArg_Parser *next; -} _PyArg_Parser; -#ifdef PY_SSIZE_T_CLEAN -#define _PyArg_ParseTupleAndKeywordsFast _PyArg_ParseTupleAndKeywordsFast_SizeT -#define _PyArg_ParseStack _PyArg_ParseStack_SizeT -#define _PyArg_ParseStackAndKeywords _PyArg_ParseStackAndKeywords_SizeT -#define _PyArg_VaParseTupleAndKeywordsFast _PyArg_VaParseTupleAndKeywordsFast_SizeT -#endif -PyAPI_FUNC(int) _PyArg_ParseTupleAndKeywordsFast(PyObject *, PyObject *, - struct _PyArg_Parser *, ...); -PyAPI_FUNC(int) _PyArg_ParseStack( - PyObject *const *args, - Py_ssize_t nargs, - const char *format, - ...); -PyAPI_FUNC(int) _PyArg_ParseStackAndKeywords( - PyObject *const *args, - Py_ssize_t nargs, - PyObject *kwnames, - struct _PyArg_Parser *, - ...); -PyAPI_FUNC(int) _PyArg_VaParseTupleAndKeywordsFast(PyObject *, PyObject *, - struct _PyArg_Parser *, va_list); -PyAPI_FUNC(PyObject * const *) _PyArg_UnpackKeywords( - PyObject *const *args, Py_ssize_t nargs, - PyObject *kwargs, PyObject *kwnames, - struct _PyArg_Parser *parser, - int minpos, int maxpos, int minkw, - PyObject **buf); - -PyAPI_FUNC(PyObject * const *) _PyArg_UnpackKeywordsWithVararg( - PyObject *const *args, Py_ssize_t nargs, - PyObject *kwargs, PyObject *kwnames, - struct _PyArg_Parser *parser, - int minpos, int maxpos, int minkw, - int vararg, PyObject **buf); - -#define _PyArg_UnpackKeywords(args, nargs, kwargs, kwnames, parser, minpos, maxpos, minkw, buf) \ - (((minkw) == 0 && (kwargs) == NULL && (kwnames) == NULL && \ - (minpos) <= (nargs) && (nargs) <= (maxpos) && args != NULL) ? (args) : \ - _PyArg_UnpackKeywords((args), (nargs), (kwargs), (kwnames), (parser), \ - (minpos), (maxpos), (minkw), (buf))) -#endif /* Py_LIMITED_API */ // Add an attribute with name 'name' and value 'obj' to the module 'mod. // On success, return 0 on success. @@ -154,10 +52,12 @@ PyAPI_FUNC(int) PyModule_AddObject(PyObject *mod, const char *, PyObject *value) PyAPI_FUNC(int) PyModule_AddIntConstant(PyObject *, const char *, long); PyAPI_FUNC(int) PyModule_AddStringConstant(PyObject *, const char *, const char *); + #if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x03090000 /* New in 3.9 */ PyAPI_FUNC(int) PyModule_AddType(PyObject *module, PyTypeObject *type); #endif /* Py_LIMITED_API */ + #define PyModule_AddIntMacro(m, c) PyModule_AddIntConstant(m, #c, c) #define PyModule_AddStringMacro(m, c) PyModule_AddStringConstant(m, #c, c) @@ -231,9 +131,6 @@ PyAPI_FUNC(int) PyModule_ExecDef(PyObject *module, PyModuleDef *def); #endif PyAPI_FUNC(PyObject *) PyModule_Create2(PyModuleDef*, int apiver); -#ifndef Py_LIMITED_API -PyAPI_FUNC(PyObject *) _PyModule_CreateInitialized(PyModuleDef*, int apiver); -#endif #ifdef Py_LIMITED_API #define PyModule_Create(module) \ @@ -256,10 +153,13 @@ PyAPI_FUNC(PyObject *) PyModule_FromDefAndSpec2(PyModuleDef *def, #define PyModule_FromDefAndSpec(module, spec) \ PyModule_FromDefAndSpec2(module, spec, PYTHON_API_VERSION) #endif /* Py_LIMITED_API */ + #endif /* New in 3.5 */ #ifndef Py_LIMITED_API -PyAPI_DATA(const char *) _Py_PackageContext; +# define Py_CPYTHON_MODSUPPORT_H +# include "cpython/modsupport.h" +# undef Py_CPYTHON_MODSUPPORT_H #endif #ifdef __cplusplus diff --git a/Makefile.pre.in b/Makefile.pre.in index f6c8c72bbc2c3..991d69815b2da 100644 --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -1546,6 +1546,7 @@ PYTHON_HEADERS= \ $(srcdir)/Include/cpython/longintrepr.h \ $(srcdir)/Include/cpython/longobject.h \ $(srcdir)/Include/cpython/methodobject.h \ + $(srcdir)/Include/cpython/modsupport.h \ $(srcdir)/Include/cpython/object.h \ $(srcdir)/Include/cpython/objimpl.h \ $(srcdir)/Include/cpython/odictobject.h \ diff --git a/PCbuild/pythoncore.vcxproj b/PCbuild/pythoncore.vcxproj index cbd3936860120..189510a10c6ea 100644 --- a/PCbuild/pythoncore.vcxproj +++ b/PCbuild/pythoncore.vcxproj @@ -156,6 +156,7 @@ + diff --git a/PCbuild/pythoncore.vcxproj.filters b/PCbuild/pythoncore.vcxproj.filters index 70af305222170..1ed52a776ca36 100644 --- a/PCbuild/pythoncore.vcxproj.filters +++ b/PCbuild/pythoncore.vcxproj.filters @@ -402,6 +402,9 @@ Include\cpython + + Include\cpython + Include\cpython From webhook-mailer at python.org Thu Apr 21 17:00:47 2022 From: webhook-mailer at python.org (vstinner) Date: Thu, 21 Apr 2022 21:00:47 -0000 Subject: [Python-checkins] gh-79315: Add Include/cpython/pythread.h header (#91798) Message-ID: https://github.com/python/cpython/commit/8a4e519e7822d17ce062f55ba68e13bfeaf450de commit: 8a4e519e7822d17ce062f55ba68e13bfeaf450de branch: main author: Victor Stinner committer: vstinner date: 2022-04-21T23:00:42+02:00 summary: gh-79315: Add Include/cpython/pythread.h header (#91798) files: A Include/cpython/pythread.h M Include/pythread.h M Makefile.pre.in M PCbuild/pythoncore.vcxproj M PCbuild/pythoncore.vcxproj.filters diff --git a/Include/cpython/pythread.h b/Include/cpython/pythread.h new file mode 100644 index 0000000000000..1fd86a6a90f9a --- /dev/null +++ b/Include/cpython/pythread.h @@ -0,0 +1,39 @@ +#ifndef Py_CPYTHON_PYTHREAD_H +# error "this header file must not be included directly" +#endif + +#define PYTHREAD_INVALID_THREAD_ID ((unsigned long)-1) + +#ifdef HAVE_FORK +/* Private function to reinitialize a lock at fork in the child process. + Reset the lock to the unlocked state. + Return 0 on success, return -1 on error. */ +PyAPI_FUNC(int) _PyThread_at_fork_reinit(PyThread_type_lock *lock); +#endif /* HAVE_FORK */ + +#ifdef HAVE_PTHREAD_H + /* Darwin needs pthread.h to know type name the pthread_key_t. */ +# include +# define NATIVE_TSS_KEY_T pthread_key_t +#elif defined(NT_THREADS) + /* In Windows, native TSS key type is DWORD, + but hardcode the unsigned long to avoid errors for include directive. + */ +# define NATIVE_TSS_KEY_T unsigned long +#else +# error "Require native threads. See https://bugs.python.org/issue31370" +#endif + +/* When Py_LIMITED_API is not defined, the type layout of Py_tss_t is + exposed to allow static allocation in the API clients. Even in this case, + you must handle TSS keys through API functions due to compatibility. +*/ +struct _Py_tss_t { + int _is_initialized; + NATIVE_TSS_KEY_T _key; +}; + +#undef NATIVE_TSS_KEY_T + +/* When static allocation, you must initialize with Py_tss_NEEDS_INIT. */ +#define Py_tss_NEEDS_INIT {0} diff --git a/Include/pythread.h b/Include/pythread.h index 034e660551531..a48329085fac5 100644 --- a/Include/pythread.h +++ b/Include/pythread.h @@ -1,4 +1,3 @@ - #ifndef Py_PYTHREAD_H #define Py_PYTHREAD_H @@ -16,10 +15,6 @@ typedef enum PyLockStatus { PY_LOCK_INTR } PyLockStatus; -#ifndef Py_LIMITED_API -#define PYTHREAD_INVALID_THREAD_ID ((unsigned long)-1) -#endif - PyAPI_FUNC(void) PyThread_init_thread(void); PyAPI_FUNC(unsigned long) PyThread_start_new_thread(void (*)(void *), void *); PyAPI_FUNC(void) _Py_NO_RETURN PyThread_exit_thread(void); @@ -36,15 +31,6 @@ PyAPI_FUNC(int) PyThread_acquire_lock(PyThread_type_lock, int); #define WAIT_LOCK 1 #define NOWAIT_LOCK 0 -#ifndef Py_LIMITED_API -#ifdef HAVE_FORK -/* Private function to reinitialize a lock at fork in the child process. - Reset the lock to the unlocked state. - Return 0 on success, return -1 on error. */ -PyAPI_FUNC(int) _PyThread_at_fork_reinit(PyThread_type_lock *lock); -#endif /* HAVE_FORK */ -#endif /* !Py_LIMITED_API */ - /* PY_TIMEOUT_T is the integral type used to specify timeouts when waiting on a lock (see PyThread_acquire_lock_timed() below). PY_TIMEOUT_MAX is the highest usable value (in microseconds) of that @@ -124,35 +110,6 @@ Py_DEPRECATED(3.7) PyAPI_FUNC(void) PyThread_ReInitTLS(void); typedef struct _Py_tss_t Py_tss_t; /* opaque */ -#ifndef Py_LIMITED_API -#ifdef HAVE_PTHREAD_H - /* Darwin needs pthread.h to know type name the pthread_key_t. */ -# include -# define NATIVE_TSS_KEY_T pthread_key_t -#elif defined(NT_THREADS) - /* In Windows, native TSS key type is DWORD, - but hardcode the unsigned long to avoid errors for include directive. - */ -# define NATIVE_TSS_KEY_T unsigned long -#else -# error "Require native threads. See https://bugs.python.org/issue31370" -#endif - -/* When Py_LIMITED_API is not defined, the type layout of Py_tss_t is - exposed to allow static allocation in the API clients. Even in this case, - you must handle TSS keys through API functions due to compatibility. -*/ -struct _Py_tss_t { - int _is_initialized; - NATIVE_TSS_KEY_T _key; -}; - -#undef NATIVE_TSS_KEY_T - -/* When static allocation, you must initialize with Py_tss_NEEDS_INIT. */ -#define Py_tss_NEEDS_INIT {0} -#endif /* !Py_LIMITED_API */ - PyAPI_FUNC(Py_tss_t *) PyThread_tss_alloc(void); PyAPI_FUNC(void) PyThread_tss_free(Py_tss_t *key); @@ -164,8 +121,13 @@ PyAPI_FUNC(int) PyThread_tss_set(Py_tss_t *key, void *value); PyAPI_FUNC(void *) PyThread_tss_get(Py_tss_t *key); #endif /* New in 3.7 */ +#ifndef Py_LIMITED_API +# define Py_CPYTHON_PYTHREAD_H +# include "cpython/pythread.h" +# undef Py_CPYTHON_PYTHREAD_H +#endif + #ifdef __cplusplus } #endif - #endif /* !Py_PYTHREAD_H */ diff --git a/Makefile.pre.in b/Makefile.pre.in index 991d69815b2da..04a371ddff72f 100644 --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -1559,6 +1559,7 @@ PYTHON_HEADERS= \ $(srcdir)/Include/cpython/pymem.h \ $(srcdir)/Include/cpython/pystate.h \ $(srcdir)/Include/cpython/pythonrun.h \ + $(srcdir)/Include/cpython/pythread.h \ $(srcdir)/Include/cpython/pytime.h \ $(srcdir)/Include/cpython/setobject.h \ $(srcdir)/Include/cpython/sysmodule.h \ diff --git a/PCbuild/pythoncore.vcxproj b/PCbuild/pythoncore.vcxproj index 189510a10c6ea..78bbec1e104e4 100644 --- a/PCbuild/pythoncore.vcxproj +++ b/PCbuild/pythoncore.vcxproj @@ -171,6 +171,7 @@ + diff --git a/PCbuild/pythoncore.vcxproj.filters b/PCbuild/pythoncore.vcxproj.filters index 1ed52a776ca36..0a9d5454ba957 100644 --- a/PCbuild/pythoncore.vcxproj.filters +++ b/PCbuild/pythoncore.vcxproj.filters @@ -459,6 +459,9 @@ Include\cpython + + Include\cpython + Include\cpython From webhook-mailer at python.org Thu Apr 21 17:04:09 2022 From: webhook-mailer at python.org (vstinner) Date: Thu, 21 Apr 2022 21:04:09 -0000 Subject: [Python-checkins] gh-89373: _Py_Dealloc() checks tp_dealloc exception (#32357) Message-ID: https://github.com/python/cpython/commit/364ed9409269fb321dc4eafdea677c09a4bc0d8d commit: 364ed9409269fb321dc4eafdea677c09a4bc0d8d branch: main author: Victor Stinner committer: vstinner date: 2022-04-21T23:04:01+02:00 summary: gh-89373: _Py_Dealloc() checks tp_dealloc exception (#32357) If Python is built in debug mode, _Py_Dealloc() now ensures that the tp_dealloc function leaves the current exception unchanged. files: A Misc/NEWS.d/next/Core and Builtins/2022-04-21-16-15-24.gh-issue-89373.A1jgLx.rst M Doc/using/configure.rst M Lib/test/test_capi.py M Objects/object.c diff --git a/Doc/using/configure.rst b/Doc/using/configure.rst index 2e632d822f9bd..e7efd2bbbc0a5 100644 --- a/Doc/using/configure.rst +++ b/Doc/using/configure.rst @@ -288,6 +288,7 @@ Effects of a debug build: to detect usage of uninitialized objects. * Ensure that functions which can clear or replace the current exception are not called with an exception raised. + * Check that deallocator functions don't change the current exception. * The garbage collector (:func:`gc.collect` function) runs some basic checks on objects consistency. * The :c:macro:`Py_SAFE_DOWNCAST()` macro checks for integer underflow and diff --git a/Lib/test/test_capi.py b/Lib/test/test_capi.py index eb0edbf5a3a12..5f5d3512d1a0e 100644 --- a/Lib/test/test_capi.py +++ b/Lib/test/test_capi.py @@ -516,7 +516,12 @@ def __del__(self): del subclass_instance # Test that setting __class__ modified the reference counts of the types - self.assertEqual(type_refcnt - 1, B.refcnt_in_del) + if Py_DEBUG: + # gh-89373: In debug mode, _Py_Dealloc() keeps a strong reference + # to the type while calling tp_dealloc() + self.assertEqual(type_refcnt, B.refcnt_in_del) + else: + self.assertEqual(type_refcnt - 1, B.refcnt_in_del) self.assertEqual(new_type_refcnt + 1, A.refcnt_in_del) # Test that the original type already has decreased its refcnt @@ -581,7 +586,12 @@ def test_c_subclass_of_heap_ctype_with_del_modifying_dunder_class_only_decrefs_o del subclass_instance # Test that setting __class__ modified the reference counts of the types - self.assertEqual(type_refcnt - 1, _testcapi.HeapCTypeSubclassWithFinalizer.refcnt_in_del) + if Py_DEBUG: + # gh-89373: In debug mode, _Py_Dealloc() keeps a strong reference + # to the type while calling tp_dealloc() + self.assertEqual(type_refcnt, _testcapi.HeapCTypeSubclassWithFinalizer.refcnt_in_del) + else: + self.assertEqual(type_refcnt - 1, _testcapi.HeapCTypeSubclassWithFinalizer.refcnt_in_del) self.assertEqual(new_type_refcnt + 1, _testcapi.HeapCTypeSubclass.refcnt_in_del) # Test that the original type already has decreased its refcnt diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-04-21-16-15-24.gh-issue-89373.A1jgLx.rst b/Misc/NEWS.d/next/Core and Builtins/2022-04-21-16-15-24.gh-issue-89373.A1jgLx.rst new file mode 100644 index 0000000000000..56434f7e79669 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2022-04-21-16-15-24.gh-issue-89373.A1jgLx.rst @@ -0,0 +1,2 @@ +If Python is built in debug mode, Python now ensures that deallocator +functions leave the current exception unchanged. Patch by Victor Stinner. diff --git a/Objects/object.c b/Objects/object.c index 29880f6003bb1..1144719c313e2 100644 --- a/Objects/object.c +++ b/Objects/object.c @@ -2354,11 +2354,45 @@ _PyObject_AssertFailed(PyObject *obj, const char *expr, const char *msg, void _Py_Dealloc(PyObject *op) { - destructor dealloc = Py_TYPE(op)->tp_dealloc; + PyTypeObject *type = Py_TYPE(op); + destructor dealloc = type->tp_dealloc; +#ifdef Py_DEBUG + PyThreadState *tstate = _PyThreadState_GET(); + PyObject *old_exc_type = tstate->curexc_type; + // Keep the old exception type alive to prevent undefined behavior + // on (tstate->curexc_type != old_exc_type) below + Py_XINCREF(old_exc_type); + // Make sure that type->tp_name remains valid + Py_INCREF(type); +#endif + #ifdef Py_TRACE_REFS _Py_ForgetReference(op); #endif (*dealloc)(op); + +#ifdef Py_DEBUG + // gh-89373: The tp_dealloc function must leave the current exception + // unchanged. + if (tstate->curexc_type != old_exc_type) { + const char *err; + if (old_exc_type == NULL) { + err = "Deallocator of type '%s' raised an exception"; + } + else if (tstate->curexc_type == NULL) { + err = "Deallocator of type '%s' cleared the current exception"; + } + else { + // It can happen if dealloc() normalized the current exception. + // A deallocator function must not change the current exception, + // not even normalize it. + err = "Deallocator of type '%s' overrode the current exception"; + } + _Py_FatalErrorFormat(__func__, err, type->tp_name); + } + Py_XDECREF(old_exc_type); + Py_DECREF(type); +#endif } From webhook-mailer at python.org Thu Apr 21 17:07:22 2022 From: webhook-mailer at python.org (vstinner) Date: Thu, 21 Apr 2022 21:07:22 -0000 Subject: [Python-checkins] gh-89653: PEP 670: Convert unicodeobject.h macros to functions (#91799) Message-ID: https://github.com/python/cpython/commit/636ad7b47e5e1997b6e6979342fe11ae284fc1c7 commit: 636ad7b47e5e1997b6e6979342fe11ae284fc1c7 branch: main author: Victor Stinner committer: vstinner date: 2022-04-21T23:07:13+02:00 summary: gh-89653: PEP 670: Convert unicodeobject.h macros to functions (#91799) Convert unicodeobject.h macros to static inline functions: * PyUnicode_AS_DATA() * PyUnicode_AS_UNICODE() * PyUnicode_GET_DATA_SIZE() * PyUnicode_GET_SIZE() Static inline functions are wrapped by macros which casts arguments with _PyObject_CAST() to prevent introducing new compiler warnings when passing "const PyObject*". files: M Include/cpython/unicodeobject.h diff --git a/Include/cpython/unicodeobject.h b/Include/cpython/unicodeobject.h index 0897e66c56d07..8cc9885756037 100644 --- a/Include/cpython/unicodeobject.h +++ b/Include/cpython/unicodeobject.h @@ -590,10 +590,14 @@ Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE *) PyUnicode_AsUnicodeAndSize( /* Fast access macros */ Py_DEPRECATED(3.3) -static inline Py_ssize_t PyUnicode_WSTR_LENGTH(PyObject *op) { - return PyUnicode_IS_COMPACT_ASCII(op) ? - _PyASCIIObject_CAST(op)->length : - _PyCompactUnicodeObject_CAST(op)->wstr_length; +static inline Py_ssize_t PyUnicode_WSTR_LENGTH(PyObject *op) +{ + if (PyUnicode_IS_COMPACT_ASCII(op)) { + return _PyASCIIObject_CAST(op)->length; + } + else { + return _PyCompactUnicodeObject_CAST(op)->wstr_length; + } } #define PyUnicode_WSTR_LENGTH(op) PyUnicode_WSTR_LENGTH(_PyObject_CAST(op)) @@ -603,16 +607,25 @@ static inline Py_ssize_t PyUnicode_WSTR_LENGTH(PyObject *op) { on request. Use PyUnicode_GET_LENGTH() for the length in code points. */ /* Py_DEPRECATED(3.3) */ -#define PyUnicode_GET_SIZE(op) \ - (_PyASCIIObject_CAST(op)->wstr ? \ - PyUnicode_WSTR_LENGTH(op) : \ - ((void)PyUnicode_AsUnicode(_PyObject_CAST(op)),\ - assert(_PyASCIIObject_CAST(op)->wstr), \ - PyUnicode_WSTR_LENGTH(op))) +static inline Py_ssize_t PyUnicode_GET_SIZE(PyObject *op) +{ + _Py_COMP_DIAG_PUSH + _Py_COMP_DIAG_IGNORE_DEPR_DECLS + if (_PyASCIIObject_CAST(op)->wstr == NULL) { + (void)PyUnicode_AsUnicode(op); + assert(_PyASCIIObject_CAST(op)->wstr != NULL); + } + return PyUnicode_WSTR_LENGTH(op); + _Py_COMP_DIAG_POP +} +#define PyUnicode_GET_SIZE(op) PyUnicode_GET_SIZE(_PyObject_CAST(op)) -/* Py_DEPRECATED(3.3) */ -#define PyUnicode_GET_DATA_SIZE(op) \ - (PyUnicode_GET_SIZE(op) * Py_UNICODE_SIZE) + /* Py_DEPRECATED(3.3) */ + static inline Py_ssize_t PyUnicode_GET_DATA_SIZE(PyObject *op) +{ + return PyUnicode_GET_SIZE(op) * Py_UNICODE_SIZE; +} +#define PyUnicode_GET_DATA_SIZE(op) PyUnicode_GET_DATA_SIZE(_PyObject_CAST(op)) /* Alias for PyUnicode_AsUnicode(). This will create a wchar_t/Py_UNICODE representation on demand. Using this macro is very inefficient now, @@ -620,13 +633,26 @@ static inline Py_ssize_t PyUnicode_WSTR_LENGTH(PyObject *op) { use PyUnicode_WRITE() and PyUnicode_READ(). */ /* Py_DEPRECATED(3.3) */ -#define PyUnicode_AS_UNICODE(op) \ - (_PyASCIIObject_CAST(op)->wstr ? _PyASCIIObject_CAST(op)->wstr : \ - PyUnicode_AsUnicode(_PyObject_CAST(op))) +static inline Py_UNICODE* PyUnicode_AS_UNICODE(PyObject *op) +{ + wchar_t *wstr = _PyASCIIObject_CAST(op)->wstr; + if (wstr != NULL) { + return wstr; + } + + _Py_COMP_DIAG_PUSH + _Py_COMP_DIAG_IGNORE_DEPR_DECLS + return PyUnicode_AsUnicode(op); + _Py_COMP_DIAG_POP +} +#define PyUnicode_AS_UNICODE(op) PyUnicode_AS_UNICODE(_PyObject_CAST(op)) /* Py_DEPRECATED(3.3) */ -#define PyUnicode_AS_DATA(op) \ - ((const char *)(PyUnicode_AS_UNICODE(op))) +static inline const char* PyUnicode_AS_DATA(PyObject *op) +{ + return (const char *)PyUnicode_AS_UNICODE(op); +} +#define PyUnicode_AS_DATA(op) PyUnicode_AS_DATA(_PyObject_CAST(op)) /* --- _PyUnicodeWriter API ----------------------------------------------- */ From webhook-mailer at python.org Thu Apr 21 17:26:29 2022 From: webhook-mailer at python.org (vstinner) Date: Thu, 21 Apr 2022 21:26:29 -0000 Subject: [Python-checkins] gh-89653: Add assertions to unicodeobject.h functions (#91800) Message-ID: https://github.com/python/cpython/commit/efe7fd4170bb809ed46cac35c6a9007d5b794e7e commit: efe7fd4170bb809ed46cac35c6a9007d5b794e7e branch: main author: Victor Stinner committer: vstinner date: 2022-04-21T23:26:25+02:00 summary: gh-89653: Add assertions to unicodeobject.h functions (#91800) files: M Include/cpython/unicodeobject.h diff --git a/Include/cpython/unicodeobject.h b/Include/cpython/unicodeobject.h index 8cc9885756037..5d050fd0b7563 100644 --- a/Include/cpython/unicodeobject.h +++ b/Include/cpython/unicodeobject.h @@ -314,6 +314,7 @@ static inline void* _PyUnicode_COMPACT_DATA(PyObject *op) { } static inline void* _PyUnicode_NONCOMPACT_DATA(PyObject *op) { + assert(!PyUnicode_IS_COMPACT(op)); void *data = _PyUnicodeObject_CAST(op)->data.any; assert(data != NULL); return data; @@ -354,13 +355,16 @@ static inline void PyUnicode_WRITE(unsigned int kind, void *data, Py_ssize_t index, Py_UCS4 value) { if (kind == PyUnicode_1BYTE_KIND) { + assert(value <= 0xffU); ((Py_UCS1 *)data)[index] = (Py_UCS1)value; } else if (kind == PyUnicode_2BYTE_KIND) { + assert(value <= 0xffffU); ((Py_UCS2 *)data)[index] = (Py_UCS2)value; } else { assert(kind == PyUnicode_4BYTE_KIND); + assert(value <= 0x10ffffU); ((Py_UCS4 *)data)[index] = value; } } @@ -378,6 +382,7 @@ static inline Py_UCS4 PyUnicode_READ(unsigned int kind, if (kind == PyUnicode_2BYTE_KIND) { return ((const Py_UCS2 *)data)[index]; } + assert(kind == PyUnicode_4BYTE_KIND); return ((const Py_UCS4 *)data)[index]; } #define PyUnicode_READ(kind, data, index) \ @@ -397,6 +402,7 @@ static inline Py_UCS4 PyUnicode_READ_CHAR(PyObject *unicode, Py_ssize_t index) if (kind == PyUnicode_2BYTE_KIND) { return PyUnicode_2BYTE_DATA(unicode)[index]; } + assert(kind == PyUnicode_4BYTE_KIND); return PyUnicode_4BYTE_DATA(unicode)[index]; } #define PyUnicode_READ_CHAR(unicode, index) \ @@ -419,6 +425,7 @@ static inline Py_UCS4 PyUnicode_MAX_CHAR_VALUE(PyObject *op) if (kind == PyUnicode_2BYTE_KIND) { return 0xffffU; } + assert(kind == PyUnicode_4BYTE_KIND); return 0x10ffffU; } #define PyUnicode_MAX_CHAR_VALUE(op) \ From webhook-mailer at python.org Thu Apr 21 21:39:39 2022 From: webhook-mailer at python.org (methane) Date: Fri, 22 Apr 2022 01:39:39 -0000 Subject: [Python-checkins] gh-91156: Use `locale.getencoding()` instead of getpreferredencoding (GH-91732) Message-ID: https://github.com/python/cpython/commit/1317b70f89606bd14597116b7ab68a968ea6c017 commit: 1317b70f89606bd14597116b7ab68a968ea6c017 branch: main author: Inada Naoki committer: methane date: 2022-04-22T10:39:24+09:00 summary: gh-91156: Use `locale.getencoding()` instead of getpreferredencoding (GH-91732) Co-authored-by: Victor Stinner files: M Doc/howto/curses.rst M Doc/library/csv.rst M Doc/library/curses.rst M Doc/library/functions.rst M Doc/library/os.rst M Lib/test/libregrtest/main.py M Lib/test/pythoninfo.py M Lib/test/support/__init__.py M Lib/test/test__locale.py M Lib/test/test_builtin.py M Lib/test/test_cmd_line.py M Lib/test/test_io.py M Lib/test/test_locale.py M Lib/test/test_mimetypes.py diff --git a/Doc/howto/curses.rst b/Doc/howto/curses.rst index c0149ffff3771..26c4ece5ae6df 100644 --- a/Doc/howto/curses.rst +++ b/Doc/howto/curses.rst @@ -299,8 +299,7 @@ The :meth:`~curses.window.addstr` method takes a Python string or bytestring as the value to be displayed. The contents of bytestrings are sent to the terminal as-is. Strings are encoded to bytes using the value of the window's :attr:`encoding` attribute; this defaults to -the default system encoding as returned by -:func:`locale.getpreferredencoding`. +the default system encoding as returned by :func:`locale.getencoding`. The :meth:`~curses.window.addch` methods take a character, which can be either a string of length 1, a bytestring of length 1, or an integer. diff --git a/Doc/library/csv.rst b/Doc/library/csv.rst index 3a7817cfdfad8..9dec7240d9c50 100644 --- a/Doc/library/csv.rst +++ b/Doc/library/csv.rst @@ -542,7 +542,7 @@ The corresponding simplest possible writing example is:: Since :func:`open` is used to open a CSV file for reading, the file will by default be decoded into unicode using the system default -encoding (see :func:`locale.getpreferredencoding`). To decode a file +encoding (see :func:`locale.getencoding`). To decode a file using a different encoding, use the ``encoding`` argument of open:: import csv diff --git a/Doc/library/curses.rst b/Doc/library/curses.rst index 37e822c0e2b20..a7cc495277801 100644 --- a/Doc/library/curses.rst +++ b/Doc/library/curses.rst @@ -27,20 +27,6 @@ Linux and the BSD variants of Unix. Whenever the documentation mentions a *character string* it can be specified as a Unicode string or a byte string. -.. note:: - - Since version 5.4, the ncurses library decides how to interpret non-ASCII data - using the ``nl_langinfo`` function. That means that you have to call - :func:`locale.setlocale` in the application and encode Unicode strings - using one of the system's available encodings. This example uses the - system's default encoding:: - - import locale - locale.setlocale(locale.LC_ALL, '') - code = locale.getpreferredencoding() - - Then use *code* as the encoding for :meth:`str.encode` calls. - .. seealso:: Module :mod:`curses.ascii` @@ -923,8 +909,8 @@ the following methods and attributes: Encoding used to encode method arguments (Unicode strings and characters). The encoding attribute is inherited from the parent window when a subwindow - is created, for example with :meth:`window.subwin`. By default, the locale - encoding is used (see :func:`locale.getpreferredencoding`). + is created, for example with :meth:`window.subwin`. + By default, current locale encoding is used (see :func:`locale.getencoding`). .. versionadded:: 3.3 diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst index e6fd0bb5eeef9..f3b8e40babbd8 100644 --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -1123,8 +1123,8 @@ are always available. They are listed here in alphabetical order. (which on *some* Unix systems, means that *all* writes append to the end of the file regardless of the current seek position). In text mode, if *encoding* is not specified the encoding used is platform-dependent: - ``locale.getpreferredencoding(False)`` is called to get the current locale - encoding. (For reading and writing raw bytes use binary mode and leave + :func:`locale.getencoding()` is called to get the current locale encoding. + (For reading and writing raw bytes use binary mode and leave *encoding* unspecified.) The available modes are: .. _filemodes: @@ -1183,10 +1183,9 @@ are always available. They are listed here in alphabetical order. *encoding* is the name of the encoding used to decode or encode the file. This should only be used in text mode. The default encoding is platform - dependent (whatever :func:`locale.getpreferredencoding` returns), but any - :term:`text encoding` supported by Python - can be used. See the :mod:`codecs` module for - the list of supported encodings. + dependent (whatever :func:`locale.getencoding` returns), but any + :term:`text encoding` supported by Python can be used. + See the :mod:`codecs` module for the list of supported encodings. *errors* is an optional string that specifies how encoding and decoding errors are to be handled?this cannot be used in binary mode. diff --git a/Doc/library/os.rst b/Doc/library/os.rst index c22bf56a9f2cd..471890e74c8e5 100644 --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -105,15 +105,15 @@ of the UTF-8 encoding: * Use UTF-8 as the :term:`filesystem encoding `. -* :func:`sys.getfilesystemencoding()` returns ``'UTF-8'``. -* :func:`locale.getpreferredencoding()` returns ``'UTF-8'`` (the *do_setlocale* +* :func:`sys.getfilesystemencoding()` returns ``'utf-8'``. +* :func:`locale.getpreferredencoding()` returns ``'utf-8'`` (the *do_setlocale* argument has no effect). * :data:`sys.stdin`, :data:`sys.stdout`, and :data:`sys.stderr` all use UTF-8 as their text encoding, with the ``surrogateescape`` :ref:`error handler ` being enabled for :data:`sys.stdin` and :data:`sys.stdout` (:data:`sys.stderr` continues to use ``backslashreplace`` as it does in the default locale-aware mode) -* On Unix, :func:`os.device_encoding` returns ``'UTF-8'`` rather than the +* On Unix, :func:`os.device_encoding` returns ``'utf-8'`` rather than the device encoding. Note that the standard stream settings in UTF-8 mode can be overridden by diff --git a/Lib/test/libregrtest/main.py b/Lib/test/libregrtest/main.py index e7e3dde0b0a66..0cacccfc0b5e3 100644 --- a/Lib/test/libregrtest/main.py +++ b/Lib/test/libregrtest/main.py @@ -482,8 +482,7 @@ def display_header(self): if cpu_count: print("== CPU count:", cpu_count) print("== encodings: locale=%s, FS=%s" - % (locale.getpreferredencoding(False), - sys.getfilesystemencoding())) + % (locale.getencoding(), sys.getfilesystemencoding())) def get_tests_result(self): result = [] diff --git a/Lib/test/pythoninfo.py b/Lib/test/pythoninfo.py index b00830c279e87..39301e6397aab 100644 --- a/Lib/test/pythoninfo.py +++ b/Lib/test/pythoninfo.py @@ -155,7 +155,7 @@ def collect_platform(info_add): def collect_locale(info_add): import locale - info_add('locale.encoding', locale.getpreferredencoding(False)) + info_add('locale.getencoding', locale.getencoding()) def collect_builtins(info_add): diff --git a/Lib/test/support/__init__.py b/Lib/test/support/__init__.py index c5666d66f4782..3b2f33979db9a 100644 --- a/Lib/test/support/__init__.py +++ b/Lib/test/support/__init__.py @@ -1445,7 +1445,7 @@ def skip_if_buggy_ucrt_strfptime(test): global _buggy_ucrt if _buggy_ucrt is None: if(sys.platform == 'win32' and - locale.getpreferredencoding(False) == 'cp65001' and + locale.getencoding() == 'cp65001' and time.localtime().tm_zone == ''): _buggy_ucrt = True else: diff --git a/Lib/test/test__locale.py b/Lib/test/test__locale.py index e25c92c2c82c5..b3bc54cd55104 100644 --- a/Lib/test/test__locale.py +++ b/Lib/test/test__locale.py @@ -43,7 +43,7 @@ def setUpModule(): locale.setlocale(locale.LC_ALL, loc) except Error: continue - encoding = locale.getpreferredencoding(False) + encoding = locale.getencoding() try: localeconv() except Exception as err: diff --git a/Lib/test/test_builtin.py b/Lib/test/test_builtin.py index a601a524d6eb7..29039230201ac 100644 --- a/Lib/test/test_builtin.py +++ b/Lib/test/test_builtin.py @@ -1204,7 +1204,7 @@ def test_open_default_encoding(self): del os.environ[key] self.write_testfile() - current_locale_encoding = locale.getpreferredencoding(False) + current_locale_encoding = locale.getencoding() with warnings.catch_warnings(): warnings.simplefilter("ignore", EncodingWarning) fp = open(TESTFN, 'w') diff --git a/Lib/test/test_cmd_line.py b/Lib/test/test_cmd_line.py index 84eab71f97701..e8f1964c2a40d 100644 --- a/Lib/test/test_cmd_line.py +++ b/Lib/test/test_cmd_line.py @@ -216,7 +216,7 @@ def test_undecodable_code(self): code = ( b'import locale; ' b'print(ascii("' + undecodable + b'"), ' - b'locale.getpreferredencoding())') + b'locale.getencoding())') p = subprocess.Popen( [sys.executable, "-c", code], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, diff --git a/Lib/test/test_io.py b/Lib/test/test_io.py index 45bf81b61f416..5528c461e58ae 100644 --- a/Lib/test/test_io.py +++ b/Lib/test/test_io.py @@ -2726,7 +2726,7 @@ def test_default_encoding(self): if key in os.environ: del os.environ[key] - current_locale_encoding = locale.getpreferredencoding(False) + current_locale_encoding = locale.getencoding() b = self.BytesIO() with warnings.catch_warnings(): warnings.simplefilter("ignore", EncodingWarning) diff --git a/Lib/test/test_locale.py b/Lib/test/test_locale.py index 774b0fcd33344..5cb6edc52d777 100644 --- a/Lib/test/test_locale.py +++ b/Lib/test/test_locale.py @@ -363,7 +363,7 @@ class TestEnUSCollation(BaseLocalizedTest, TestCollation): locale_type = locale.LC_ALL def setUp(self): - enc = codecs.lookup(locale.getpreferredencoding(False) or 'ascii').name + enc = codecs.lookup(locale.getencoding() or 'ascii').name if enc not in ('utf-8', 'iso8859-1', 'cp1252'): raise unittest.SkipTest('encoding not suitable') if enc != 'iso8859-1' and (sys.platform == 'darwin' or is_android or @@ -533,6 +533,14 @@ def test_defaults_UTF8(self): if orig_getlocale is not None: _locale._getdefaultlocale = orig_getlocale + def test_getencoding(self): + # Invoke getencoding to make sure it does not cause exceptions. + enc = locale.getencoding() + self.assertIsInstance(enc, str) + self.assertNotEqual(enc, "") + # make sure it is valid + codecs.lookup(enc) + def test_getpreferredencoding(self): # Invoke getpreferredencoding to make sure it does not cause exceptions. enc = locale.getpreferredencoding() diff --git a/Lib/test/test_mimetypes.py b/Lib/test/test_mimetypes.py index 3477b18376a4f..f2b103693a9b2 100644 --- a/Lib/test/test_mimetypes.py +++ b/Lib/test/test_mimetypes.py @@ -1,5 +1,4 @@ import io -import locale import mimetypes import pathlib import sys @@ -33,7 +32,7 @@ def tearDownModule(): class MimeTypesTestCase(unittest.TestCase): def setUp(self): self.db = mimetypes.MimeTypes() - + def test_case_sensitivity(self): eq = self.assertEqual eq(self.db.guess_type("foobar.HTML"), self.db.guess_type("foobar.html")) @@ -145,11 +144,6 @@ def test_guess_all_types(self): self.assertNotIn('.no-such-ext', all) def test_encoding(self): - getpreferredencoding = locale.getpreferredencoding - self.addCleanup(setattr, locale, 'getpreferredencoding', - getpreferredencoding) - locale.getpreferredencoding = lambda: 'ascii' - filename = support.findfile("mime.types") mimes = mimetypes.MimeTypes([filename]) exts = mimes.guess_all_extensions('application/vnd.geocube+xml', From webhook-mailer at python.org Thu Apr 21 21:45:21 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Fri, 22 Apr 2022 01:45:21 -0000 Subject: [Python-checkins] gh-69093: Add indexing and slicing support to sqlite3.Blob (#91599) Message-ID: https://github.com/python/cpython/commit/29afb7d2efed6ee48a67dafdc1a1f34dd60153cf commit: 29afb7d2efed6ee48a67dafdc1a1f34dd60153cf branch: main author: Erlend Egeberg Aasland committer: JelleZijlstra date: 2022-04-21T18:45:16-07:00 summary: gh-69093: Add indexing and slicing support to sqlite3.Blob (#91599) Authored-by: Aviv Palivoda Co-authored-by: Erlend E. Aasland files: A Misc/NEWS.d/next/Library/2022-04-14-01-00-31.gh-issue-69093.bmlMwI.rst M Doc/includes/sqlite3/blob.py M Doc/library/sqlite3.rst M Lib/test/test_sqlite3/test_dbapi.py M Modules/_sqlite/blob.c diff --git a/Doc/includes/sqlite3/blob.py b/Doc/includes/sqlite3/blob.py index b3694ad08af46..d947059b3ae64 100644 --- a/Doc/includes/sqlite3/blob.py +++ b/Doc/includes/sqlite3/blob.py @@ -2,15 +2,18 @@ con = sqlite3.connect(":memory:") con.execute("create table test(blob_col blob)") -con.execute("insert into test(blob_col) values (zeroblob(10))") +con.execute("insert into test(blob_col) values (zeroblob(13))") # Write to our blob, using two write operations: with con.blobopen("test", "blob_col", 1) as blob: - blob.write(b"Hello") - blob.write(b"World") + blob.write(b"hello, ") + blob.write(b"world.") + # Modify the first and last bytes of our blob + blob[0] = b"H" + blob[-1] = b"!" # Read the contents of our blob with con.blobopen("test", "blob_col", 1) as blob: greeting = blob.read() -print(greeting) # outputs "b'HelloWorld'" +print(greeting) # outputs "b'Hello, world!'" diff --git a/Doc/library/sqlite3.rst b/Doc/library/sqlite3.rst index cbe7bb1fb9a0d..69e77e922a9ab 100644 --- a/Doc/library/sqlite3.rst +++ b/Doc/library/sqlite3.rst @@ -1051,9 +1051,10 @@ Blob Objects .. class:: Blob - A :class:`Blob` instance is a :term:`file-like object` that can read and write - data in an SQLite :abbr:`BLOB (Binary Large OBject)`. Call ``len(blob)`` to - get the size (number of bytes) of the blob. + A :class:`Blob` instance is a :term:`file-like object` + that can read and write data in an SQLite :abbr:`BLOB (Binary Large OBject)`. + Call :func:`len(blob) ` to get the size (number of bytes) of the blob. + Use indices and :term:`slices ` for direct access to the blob data. Use the :class:`Blob` as a :term:`context manager` to ensure that the blob handle is closed after use. diff --git a/Lib/test/test_sqlite3/test_dbapi.py b/Lib/test/test_sqlite3/test_dbapi.py index 79dcb3ef8954a..8bfdce2bbe92e 100644 --- a/Lib/test/test_sqlite3/test_dbapi.py +++ b/Lib/test/test_sqlite3/test_dbapi.py @@ -33,7 +33,7 @@ check_disallow_instantiation, threading_helper, ) -from _testcapi import INT_MAX +from _testcapi import INT_MAX, ULLONG_MAX from os import SEEK_SET, SEEK_CUR, SEEK_END from test.support.os_helper import TESTFN, unlink, temp_dir @@ -1138,6 +1138,13 @@ def test_blob_write_error_length(self): with self.assertRaisesRegex(ValueError, "data longer than blob"): self.blob.write(b"a" * 1000) + self.blob.seek(0, SEEK_SET) + n = len(self.blob) + self.blob.write(b"a" * (n-1)) + self.blob.write(b"a") + with self.assertRaisesRegex(ValueError, "data longer than blob"): + self.blob.write(b"a") + def test_blob_write_error_row_changed(self): self.cx.execute("update test set b='aaaa' where rowid=1") with self.assertRaises(sqlite.OperationalError): @@ -1162,12 +1169,127 @@ def test_blob_open_error(self): with self.assertRaisesRegex(sqlite.OperationalError, regex): self.cx.blobopen(*args, **kwds) + def test_blob_length(self): + self.assertEqual(len(self.blob), 50) + + def test_blob_get_item(self): + self.assertEqual(self.blob[5], b"b") + self.assertEqual(self.blob[6], b"l") + self.assertEqual(self.blob[7], b"o") + self.assertEqual(self.blob[8], b"b") + self.assertEqual(self.blob[-1], b"!") + + def test_blob_set_item(self): + self.blob[0] = b"b" + expected = b"b" + self.data[1:] + actual = self.cx.execute("select b from test").fetchone()[0] + self.assertEqual(actual, expected) + + def test_blob_set_item_with_offset(self): + self.blob.seek(0, SEEK_END) + self.assertEqual(self.blob.read(), b"") # verify that we're at EOB + self.blob[0] = b"T" + self.blob[-1] = b"." + self.blob.seek(0, SEEK_SET) + expected = b"This blob data string is exactly fifty bytes long." + self.assertEqual(self.blob.read(), expected) + + def test_blob_set_buffer_object(self): + from array import array + self.blob[0] = memoryview(b"1") + self.assertEqual(self.blob[0], b"1") + + self.blob[1] = bytearray(b"2") + self.assertEqual(self.blob[1], b"2") + + self.blob[2] = array("b", [4]) + self.assertEqual(self.blob[2], b"\x04") + + self.blob[0:5] = memoryview(b"12345") + self.assertEqual(self.blob[0:5], b"12345") + + self.blob[0:5] = bytearray(b"23456") + self.assertEqual(self.blob[0:5], b"23456") + + self.blob[0:5] = array("b", [1, 2, 3, 4, 5]) + self.assertEqual(self.blob[0:5], b"\x01\x02\x03\x04\x05") + + def test_blob_set_item_negative_index(self): + self.blob[-1] = b"z" + self.assertEqual(self.blob[-1], b"z") + + def test_blob_get_slice(self): + self.assertEqual(self.blob[5:14], b"blob data") + + def test_blob_get_empty_slice(self): + self.assertEqual(self.blob[5:5], b"") + + def test_blob_get_slice_negative_index(self): + self.assertEqual(self.blob[5:-5], self.data[5:-5]) + + def test_blob_get_slice_with_skip(self): + self.assertEqual(self.blob[0:10:2], b"ti lb") + + def test_blob_set_slice(self): + self.blob[0:5] = b"12345" + expected = b"12345" + self.data[5:] + actual = self.cx.execute("select b from test").fetchone()[0] + self.assertEqual(actual, expected) + + def test_blob_set_empty_slice(self): + self.blob[0:0] = b"" + self.assertEqual(self.blob[:], self.data) + + def test_blob_set_slice_with_skip(self): + self.blob[0:10:2] = b"12345" + actual = self.cx.execute("select b from test").fetchone()[0] + expected = b"1h2s3b4o5 " + self.data[10:] + self.assertEqual(actual, expected) + + def test_blob_mapping_invalid_index_type(self): + msg = "indices must be integers" + with self.assertRaisesRegex(TypeError, msg): + self.blob[5:5.5] + with self.assertRaisesRegex(TypeError, msg): + self.blob[1.5] + with self.assertRaisesRegex(TypeError, msg): + self.blob["a"] = b"b" + + def test_blob_get_item_error(self): + dataset = [len(self.blob), 105, -105] + for idx in dataset: + with self.subTest(idx=idx): + with self.assertRaisesRegex(IndexError, "index out of range"): + self.blob[idx] + with self.assertRaisesRegex(IndexError, "cannot fit 'int'"): + self.blob[ULLONG_MAX] + + def test_blob_set_item_error(self): + with self.assertRaisesRegex(ValueError, "must be a single byte"): + self.blob[0] = b"multiple" + with self.assertRaisesRegex(TypeError, "doesn't support.*deletion"): + del self.blob[0] + with self.assertRaisesRegex(IndexError, "Blob index out of range"): + self.blob[1000] = b"a" + + def test_blob_set_slice_error(self): + with self.assertRaisesRegex(IndexError, "wrong size"): + self.blob[5:10] = b"a" + with self.assertRaisesRegex(IndexError, "wrong size"): + self.blob[5:10] = b"a" * 1000 + with self.assertRaisesRegex(TypeError, "doesn't support.*deletion"): + del self.blob[5:10] + with self.assertRaisesRegex(ValueError, "step cannot be zero"): + self.blob[5:10:0] = b"12345" + with self.assertRaises(BufferError): + self.blob[5:10] = memoryview(b"abcde")[::2] + def test_blob_sequence_not_supported(self): - with self.assertRaises(TypeError): + with self.assertRaisesRegex(TypeError, "unsupported operand"): self.blob + self.blob - with self.assertRaises(TypeError): + with self.assertRaisesRegex(TypeError, "unsupported operand"): self.blob * 5 - with self.assertRaises(TypeError): + with self.assertRaisesRegex(TypeError, "is not iterable"): b"a" in self.blob def test_blob_context_manager(self): @@ -1209,6 +1331,14 @@ def test_blob_closed(self): blob.__enter__() with self.assertRaisesRegex(sqlite.ProgrammingError, msg): blob.__exit__(None, None, None) + with self.assertRaisesRegex(sqlite.ProgrammingError, msg): + len(blob) + with self.assertRaisesRegex(sqlite.ProgrammingError, msg): + blob[0] + with self.assertRaisesRegex(sqlite.ProgrammingError, msg): + blob[0:1] + with self.assertRaisesRegex(sqlite.ProgrammingError, msg): + blob[0] = b"" def test_blob_closed_db_read(self): with memory_database() as cx: diff --git a/Misc/NEWS.d/next/Library/2022-04-14-01-00-31.gh-issue-69093.bmlMwI.rst b/Misc/NEWS.d/next/Library/2022-04-14-01-00-31.gh-issue-69093.bmlMwI.rst new file mode 100644 index 0000000000000..4bb8531beeacd --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-14-01-00-31.gh-issue-69093.bmlMwI.rst @@ -0,0 +1,2 @@ +Add indexing and slicing support to :class:`sqlite3.Blob`. Patch by Aviv Palivoda +and Erlend E. Aasland. diff --git a/Modules/_sqlite/blob.c b/Modules/_sqlite/blob.c index 3f766302d6251..0c57ff8ca4252 100644 --- a/Modules/_sqlite/blob.c +++ b/Modules/_sqlite/blob.c @@ -120,8 +120,11 @@ blob_seterror(pysqlite_Blob *self, int rc) } static PyObject * -inner_read(pysqlite_Blob *self, int length, int offset) +inner_read(pysqlite_Blob *self, Py_ssize_t length, Py_ssize_t offset) { + assert(length <= sqlite3_blob_bytes(self->blob)); + assert(offset <= sqlite3_blob_bytes(self->blob)); + PyObject *buffer = PyBytes_FromStringAndSize(NULL, length); if (buffer == NULL) { return NULL; @@ -130,7 +133,7 @@ inner_read(pysqlite_Blob *self, int length, int offset) char *raw_buffer = PyBytes_AS_STRING(buffer); int rc; Py_BEGIN_ALLOW_THREADS - rc = sqlite3_blob_read(self->blob, raw_buffer, length, offset); + rc = sqlite3_blob_read(self->blob, raw_buffer, (int)length, (int)offset); Py_END_ALLOW_THREADS if (rc != SQLITE_OK) { @@ -181,17 +184,20 @@ blob_read_impl(pysqlite_Blob *self, int length) }; static int -inner_write(pysqlite_Blob *self, const void *buf, Py_ssize_t len, int offset) +inner_write(pysqlite_Blob *self, const void *buf, Py_ssize_t len, + Py_ssize_t offset) { - int remaining_len = sqlite3_blob_bytes(self->blob) - self->offset; + Py_ssize_t blob_len = sqlite3_blob_bytes(self->blob); + Py_ssize_t remaining_len = blob_len - offset; if (len > remaining_len) { PyErr_SetString(PyExc_ValueError, "data longer than blob length"); return -1; } + assert(offset <= blob_len); int rc; Py_BEGIN_ALLOW_THREADS - rc = sqlite3_blob_write(self->blob, buf, (int)len, offset); + rc = sqlite3_blob_write(self->blob, buf, (int)len, (int)offset); Py_END_ALLOW_THREADS if (rc != SQLITE_OK) { @@ -347,6 +353,192 @@ blob_exit_impl(pysqlite_Blob *self, PyObject *type, PyObject *val, Py_RETURN_FALSE; } +static Py_ssize_t +blob_length(pysqlite_Blob *self) +{ + if (!check_blob(self)) { + return -1; + } + return sqlite3_blob_bytes(self->blob); +}; + +static Py_ssize_t +get_subscript_index(pysqlite_Blob *self, PyObject *item) +{ + Py_ssize_t i = PyNumber_AsSsize_t(item, PyExc_IndexError); + if (i == -1 && PyErr_Occurred()) { + return -1; + } + int blob_len = sqlite3_blob_bytes(self->blob); + if (i < 0) { + i += blob_len; + } + if (i < 0 || i >= blob_len) { + PyErr_SetString(PyExc_IndexError, "Blob index out of range"); + return -1; + } + return i; +} + +static PyObject * +subscript_index(pysqlite_Blob *self, PyObject *item) +{ + Py_ssize_t i = get_subscript_index(self, item); + if (i < 0) { + return NULL; + } + return inner_read(self, 1, i); +} + +static int +get_slice_info(pysqlite_Blob *self, PyObject *item, Py_ssize_t *start, + Py_ssize_t *stop, Py_ssize_t *step, Py_ssize_t *slicelen) +{ + if (PySlice_Unpack(item, start, stop, step) < 0) { + return -1; + } + int len = sqlite3_blob_bytes(self->blob); + *slicelen = PySlice_AdjustIndices(len, start, stop, *step); + return 0; +} + +static PyObject * +subscript_slice(pysqlite_Blob *self, PyObject *item) +{ + Py_ssize_t start, stop, step, len; + if (get_slice_info(self, item, &start, &stop, &step, &len) < 0) { + return NULL; + } + + if (step == 1) { + return inner_read(self, len, start); + } + PyObject *blob = inner_read(self, stop - start, start); + if (blob == NULL) { + return NULL; + } + PyObject *result = PyBytes_FromStringAndSize(NULL, len); + if (result != NULL) { + char *blob_buf = PyBytes_AS_STRING(blob); + char *res_buf = PyBytes_AS_STRING(result); + for (Py_ssize_t i = 0, j = 0; i < len; i++, j += step) { + res_buf[i] = blob_buf[j]; + } + Py_DECREF(blob); + } + return result; +} + +static PyObject * +blob_subscript(pysqlite_Blob *self, PyObject *item) +{ + if (!check_blob(self)) { + return NULL; + } + + if (PyIndex_Check(item)) { + return subscript_index(self, item); + } + if (PySlice_Check(item)) { + return subscript_slice(self, item); + } + + PyErr_SetString(PyExc_TypeError, "Blob indices must be integers"); + return NULL; +} + +static int +ass_subscript_index(pysqlite_Blob *self, PyObject *item, PyObject *value) +{ + if (value == NULL) { + PyErr_SetString(PyExc_TypeError, + "Blob doesn't support item deletion"); + return -1; + } + Py_ssize_t i = get_subscript_index(self, item); + if (i < 0) { + return -1; + } + + Py_buffer vbuf; + if (PyObject_GetBuffer(value, &vbuf, PyBUF_SIMPLE) < 0) { + return -1; + } + int rc = -1; + if (vbuf.len != 1) { + PyErr_SetString(PyExc_ValueError, "Blob assignment must be a single byte"); + } + else { + rc = inner_write(self, (const char *)vbuf.buf, 1, i); + } + PyBuffer_Release(&vbuf); + return rc; +} + +static int +ass_subscript_slice(pysqlite_Blob *self, PyObject *item, PyObject *value) +{ + if (value == NULL) { + PyErr_SetString(PyExc_TypeError, + "Blob doesn't support slice deletion"); + return -1; + } + + Py_ssize_t start, stop, step, len; + if (get_slice_info(self, item, &start, &stop, &step, &len) < 0) { + return -1; + } + + if (len == 0) { + return 0; + } + + Py_buffer vbuf; + if (PyObject_GetBuffer(value, &vbuf, PyBUF_SIMPLE) < 0) { + return -1; + } + + int rc = -1; + if (vbuf.len != len) { + PyErr_SetString(PyExc_IndexError, + "Blob slice assignment is wrong size"); + } + else if (step == 1) { + rc = inner_write(self, vbuf.buf, len, start); + } + else { + PyObject *blob_bytes = inner_read(self, stop - start, start); + if (blob_bytes != NULL) { + char *blob_buf = PyBytes_AS_STRING(blob_bytes); + for (Py_ssize_t i = 0, j = 0; i < len; i++, j += step) { + blob_buf[j] = ((char *)vbuf.buf)[i]; + } + rc = inner_write(self, blob_buf, stop - start, start); + Py_DECREF(blob_bytes); + } + } + PyBuffer_Release(&vbuf); + return rc; +} + +static int +blob_ass_subscript(pysqlite_Blob *self, PyObject *item, PyObject *value) +{ + if (!check_blob(self)) { + return -1; + } + + if (PyIndex_Check(item)) { + return ass_subscript_index(self, item, value); + } + if (PySlice_Check(item)) { + return ass_subscript_slice(self, item, value); + } + + PyErr_SetString(PyExc_TypeError, "Blob indices must be integers"); + return -1; +} + static PyMethodDef blob_methods[] = { BLOB_CLOSE_METHODDEF @@ -370,6 +562,11 @@ static PyType_Slot blob_slots[] = { {Py_tp_clear, blob_clear}, {Py_tp_methods, blob_methods}, {Py_tp_members, blob_members}, + + // Mapping protocol + {Py_mp_length, blob_length}, + {Py_mp_subscript, blob_subscript}, + {Py_mp_ass_subscript, blob_ass_subscript}, {0, NULL}, }; From webhook-mailer at python.org Thu Apr 21 22:28:43 2022 From: webhook-mailer at python.org (brettcannon) Date: Fri, 22 Apr 2022 02:28:43 -0000 Subject: [Python-checkins] gh-91217: deprecate-pipes (GH-91779) Message-ID: https://github.com/python/cpython/commit/2551a6c92f247eed8121b14f151acd95432dff9e commit: 2551a6c92f247eed8121b14f151acd95432dff9e branch: main author: Brett Cannon committer: brettcannon date: 2022-04-21T19:28:34-07:00 summary: gh-91217: deprecate-pipes (GH-91779) files: A Misc/NEWS.d/next/Library/2022-04-17-12-07-50.gh-issue-91217.TIvrsq.rst M Doc/whatsnew/3.11.rst M Lib/pipes.py M Lib/test/test_pipes.py diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index c3a8a7e42a110..75ea70b7a1832 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -919,6 +919,7 @@ Deprecated * :mod:`nis` * :mod:`nntplib` * :mod:`ossaudiodev` + * :mod:`pipes` (Contributed by Brett Cannon in :issue:`47061`.) diff --git a/Lib/pipes.py b/Lib/pipes.py index 8cc74b0f1f781..61d63b48d3e4c 100644 --- a/Lib/pipes.py +++ b/Lib/pipes.py @@ -60,10 +60,13 @@ import re import os import tempfile +import warnings # we import the quote function rather than the module for backward compat # (quote used to be an undocumented but used function in pipes) from shlex import quote +warnings._deprecated(__name__, remove=(3, 13)) + __all__ = ["Template"] # Conversion step kinds diff --git a/Lib/test/test_pipes.py b/Lib/test/test_pipes.py index 6335e7cbe09c4..09e21153ec858 100644 --- a/Lib/test/test_pipes.py +++ b/Lib/test/test_pipes.py @@ -1,10 +1,12 @@ -import pipes import os import string import unittest import shutil from test.support import reap_children, unix_shell from test.support.os_helper import TESTFN, unlink +from test.support.warnings_helper import import_deprecated + +pipes = import_deprecated("pipes") if os.name != 'posix': diff --git a/Misc/NEWS.d/next/Library/2022-04-17-12-07-50.gh-issue-91217.TIvrsq.rst b/Misc/NEWS.d/next/Library/2022-04-17-12-07-50.gh-issue-91217.TIvrsq.rst new file mode 100644 index 0000000000000..fc8ed5775bbb2 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-17-12-07-50.gh-issue-91217.TIvrsq.rst @@ -0,0 +1 @@ +Deprecate the 'pipes' module. From webhook-mailer at python.org Fri Apr 22 00:23:04 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Fri, 22 Apr 2022 04:23:04 -0000 Subject: [Python-checkins] bpo-43224: Implement pickling of TypeVarTuples (#32119) Message-ID: https://github.com/python/cpython/commit/5e130a8da4e4f13444ec20dfe88a3e2e070005ca commit: 5e130a8da4e4f13444ec20dfe88a3e2e070005ca branch: main author: Matthew Rahtz committer: JelleZijlstra date: 2022-04-21T21:22:53-07:00 summary: bpo-43224: Implement pickling of TypeVarTuples (#32119) Co-authored-by: Jelle Zijlstra files: M Lib/test/test_typing.py M Lib/typing.py diff --git a/Lib/test/test_typing.py b/Lib/test/test_typing.py index d4808474e4fce..1fd99a0f806cf 100644 --- a/Lib/test/test_typing.py +++ b/Lib/test/test_typing.py @@ -1,7 +1,7 @@ import contextlib import collections from collections import defaultdict -from functools import lru_cache +from functools import lru_cache, wraps import inspect import pickle import re @@ -70,6 +70,18 @@ def clear_caches(self): f() +def all_pickle_protocols(test_func): + """Runs `test_func` with various values for `proto` argument.""" + + @wraps(test_func) + def wrapper(self): + for proto in range(pickle.HIGHEST_PROTOCOL + 1): + with self.subTest(pickle_proto=proto): + test_func(self, proto=proto) + + return wrapper + + class Employee: pass @@ -911,6 +923,48 @@ class C(Generic[Unpack[Ts]]): pass self.assertNotEqual(C[Unpack[Ts1]], C[Unpack[Ts2]]) +class TypeVarTuplePicklingTests(BaseTestCase): + # These are slightly awkward tests to run, because TypeVarTuples are only + # picklable if defined in the global scope. We therefore need to push + # various things defined in these tests into the global scope with `global` + # statements at the start of each test. + + @all_pickle_protocols + def test_pickling_then_unpickling_results_in_same_identity(self, proto): + global Ts1 # See explanation at start of class. + Ts1 = TypeVarTuple('Ts1') + Ts2 = pickle.loads(pickle.dumps(Ts1, proto)) + self.assertIs(Ts1, Ts2) + + @all_pickle_protocols + def test_pickling_then_unpickling_unpacked_results_in_same_identity(self, proto): + global Ts # See explanation at start of class. + Ts = TypeVarTuple('Ts') + unpacked1 = Unpack[Ts] + unpacked2 = pickle.loads(pickle.dumps(unpacked1, proto)) + self.assertIs(unpacked1, unpacked2) + + @all_pickle_protocols + def test_pickling_then_unpickling_tuple_with_typevartuple_equality( + self, proto + ): + global T, Ts # See explanation at start of class. + T = TypeVar('T') + Ts = TypeVarTuple('Ts') + + a1 = Tuple[Unpack[Ts]] + a2 = pickle.loads(pickle.dumps(a1, proto)) + self.assertEqual(a1, a2) + + a1 = Tuple[T, Unpack[Ts]] + a2 = pickle.loads(pickle.dumps(a1, proto)) + self.assertEqual(a1, a2) + + a1 = Tuple[int, Unpack[Ts]] + a2 = pickle.loads(pickle.dumps(a1, proto)) + self.assertEqual(a1, a2) + + class UnionTests(BaseTestCase): def test_basics(self): diff --git a/Lib/typing.py b/Lib/typing.py index 3e0fbdb989155..a6f4fa96afb50 100644 --- a/Lib/typing.py +++ b/Lib/typing.py @@ -867,6 +867,13 @@ def _is_typevar_like(x: Any) -> bool: return isinstance(x, (TypeVar, ParamSpec)) or _is_unpacked_typevartuple(x) +class _PickleUsingNameMixin: + """Mixin enabling pickling based on self.__name__.""" + + def __reduce__(self): + return self.__name__ + + class _BoundVarianceMixin: """Mixin giving __init__ bound and variance arguments. @@ -903,11 +910,9 @@ def __repr__(self): prefix = '~' return prefix + self.__name__ - def __reduce__(self): - return self.__name__ - -class TypeVar(_Final, _Immutable, _BoundVarianceMixin, _root=True): +class TypeVar(_Final, _Immutable, _BoundVarianceMixin, _PickleUsingNameMixin, + _root=True): """Type variable. Usage:: @@ -973,7 +978,7 @@ def __typing_subst__(self, arg): return arg -class TypeVarTuple(_Final, _Immutable, _root=True): +class TypeVarTuple(_Final, _Immutable, _PickleUsingNameMixin, _root=True): """Type variable tuple. Usage: @@ -994,11 +999,18 @@ class C(Generic[*Ts]): ... C[()] # Even this is fine For more details, see PEP 646. + + Note that only TypeVarTuples defined in global scope can be pickled. """ def __init__(self, name): self.__name__ = name + # Used for pickling. + def_mod = _caller() + if def_mod != 'typing': + self.__module__ = def_mod + def __iter__(self): yield Unpack[self] @@ -1057,7 +1069,8 @@ def __eq__(self, other): return self.__origin__ == other.__origin__ -class ParamSpec(_Final, _Immutable, _BoundVarianceMixin, _root=True): +class ParamSpec(_Final, _Immutable, _BoundVarianceMixin, _PickleUsingNameMixin, + _root=True): """Parameter specification variable. Usage:: From webhook-mailer at python.org Fri Apr 22 00:27:20 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Fri, 22 Apr 2022 04:27:20 -0000 Subject: [Python-checkins] gh-91291: Accept attributes as keyword arguments in decimal.localcontext (#32242) Message-ID: https://github.com/python/cpython/commit/bcf14ae4336fced718c00edc34b9191c2b48525a commit: bcf14ae4336fced718c00edc34b9191c2b48525a branch: main author: Sam Ezeh committer: JelleZijlstra date: 2022-04-21T21:27:15-07:00 summary: gh-91291: Accept attributes as keyword arguments in decimal.localcontext (#32242) Co-authored-by: Jelle Zijlstra files: A Misc/NEWS.d/next/Library/2022-04-01-21-44-00.bpo-47135.TvkKB-.rst M Doc/library/decimal.rst M Lib/_pydecimal.py M Lib/test/test_decimal.py M Modules/_decimal/_decimal.c M Modules/_decimal/docstrings.h diff --git a/Doc/library/decimal.rst b/Doc/library/decimal.rst index e759c5cf23b9e..2ad84f20b5560 100644 --- a/Doc/library/decimal.rst +++ b/Doc/library/decimal.rst @@ -925,12 +925,13 @@ Each thread has its own current context which is accessed or changed using the You can also use the :keyword:`with` statement and the :func:`localcontext` function to temporarily change the active context. -.. function:: localcontext(ctx=None) +.. function:: localcontext(ctx=None, \*\*kwargs) Return a context manager that will set the current context for the active thread to a copy of *ctx* on entry to the with-statement and restore the previous context when exiting the with-statement. If no context is specified, a copy of the - current context is used. + current context is used. The *kwargs* argument is used to set the attributes + of the new context. For example, the following code sets the current decimal precision to 42 places, performs a calculation, and then automatically restores the previous context:: @@ -942,6 +943,21 @@ function to temporarily change the active context. s = calculate_something() s = +s # Round the final result back to the default precision + Using keyword arguments, the code would be the following:: + + from decimal import localcontext + + with localcontext(prec=42) as ctx: + s = calculate_something() + s = +s + + Raises :exc:`TypeError` if *kwargs* supplies an attribute that :class:`Context` doesn't + support. Raises either :exc:`TypeError` or :exc:`ValueError` if *kwargs* supplies an + invalid value for an attribute. + + .. versionchanged:: 3.11 + :meth:`localcontext` now supports setting context attributes through the use of keyword arguments. + New contexts can also be created using the :class:`Context` constructor described below. In addition, the module provides three pre-made contexts: diff --git a/Lib/_pydecimal.py b/Lib/_pydecimal.py index 89646fa714c54..f9d6c9901f1f3 100644 --- a/Lib/_pydecimal.py +++ b/Lib/_pydecimal.py @@ -441,6 +441,10 @@ class FloatOperation(DecimalException, TypeError): _current_context_var = contextvars.ContextVar('decimal_context') +_context_attributes = frozenset( + ['prec', 'Emin', 'Emax', 'capitals', 'clamp', 'rounding', 'flags', 'traps'] +) + def getcontext(): """Returns this thread's context. @@ -464,7 +468,7 @@ def setcontext(context): del contextvars # Don't contaminate the namespace -def localcontext(ctx=None): +def localcontext(ctx=None, **kwargs): """Return a context manager for a copy of the supplied context Uses a copy of the current context if no context is specified @@ -500,8 +504,14 @@ def sin(x): >>> print(getcontext().prec) 28 """ - if ctx is None: ctx = getcontext() - return _ContextManager(ctx) + if ctx is None: + ctx = getcontext() + ctx_manager = _ContextManager(ctx) + for key, value in kwargs.items(): + if key not in _context_attributes: + raise TypeError(f"'{key}' is an invalid keyword argument for this function") + setattr(ctx_manager.new_context, key, value) + return ctx_manager ##### Decimal class ####################################################### diff --git a/Lib/test/test_decimal.py b/Lib/test/test_decimal.py index 5e77e3c56cbbc..96f8f7f32c454 100644 --- a/Lib/test/test_decimal.py +++ b/Lib/test/test_decimal.py @@ -3665,6 +3665,40 @@ def test_localcontextarg(self): self.assertIsNot(new_ctx, set_ctx, 'did not copy the context') self.assertIs(set_ctx, enter_ctx, '__enter__ returned wrong context') + def test_localcontext_kwargs(self): + with self.decimal.localcontext( + prec=10, rounding=ROUND_HALF_DOWN, + Emin=-20, Emax=20, capitals=0, + clamp=1 + ) as ctx: + self.assertEqual(ctx.prec, 10) + self.assertEqual(ctx.rounding, self.decimal.ROUND_HALF_DOWN) + self.assertEqual(ctx.Emin, -20) + self.assertEqual(ctx.Emax, 20) + self.assertEqual(ctx.capitals, 0) + self.assertEqual(ctx.clamp, 1) + + self.assertRaises(TypeError, self.decimal.localcontext, precision=10) + + self.assertRaises(ValueError, self.decimal.localcontext, Emin=1) + self.assertRaises(ValueError, self.decimal.localcontext, Emax=-1) + self.assertRaises(ValueError, self.decimal.localcontext, capitals=2) + self.assertRaises(ValueError, self.decimal.localcontext, clamp=2) + + self.assertRaises(TypeError, self.decimal.localcontext, rounding="") + self.assertRaises(TypeError, self.decimal.localcontext, rounding=1) + + self.assertRaises(TypeError, self.decimal.localcontext, flags="") + self.assertRaises(TypeError, self.decimal.localcontext, traps="") + self.assertRaises(TypeError, self.decimal.localcontext, Emin="") + self.assertRaises(TypeError, self.decimal.localcontext, Emax="") + + def test_local_context_kwargs_does_not_overwrite_existing_argument(self): + ctx = self.decimal.getcontext() + ctx.prec = 28 + with self.decimal.localcontext(prec=10) as ctx2: + self.assertEqual(ctx.prec, 28) + def test_nested_with_statements(self): # Use a copy of the supplied context in the block Decimal = self.decimal.Decimal diff --git a/Misc/NEWS.d/next/Library/2022-04-01-21-44-00.bpo-47135.TvkKB-.rst b/Misc/NEWS.d/next/Library/2022-04-01-21-44-00.bpo-47135.TvkKB-.rst new file mode 100644 index 0000000000000..2323c22c007e9 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-01-21-44-00.bpo-47135.TvkKB-.rst @@ -0,0 +1 @@ +:meth:`decimal.localcontext` now accepts context attributes via keyword arguments diff --git a/Modules/_decimal/_decimal.c b/Modules/_decimal/_decimal.c index 4637b8b34c4ce..8c08847328bc0 100644 --- a/Modules/_decimal/_decimal.c +++ b/Modules/_decimal/_decimal.c @@ -1156,6 +1156,67 @@ context_setattr(PyObject *self, PyObject *name, PyObject *value) return PyObject_GenericSetAttr(self, name, value); } +static int +context_setattrs(PyObject *self, PyObject *prec, PyObject *rounding, + PyObject *emin, PyObject *emax, PyObject *capitals, + PyObject *clamp, PyObject *status, PyObject *traps) { + + int ret; + if (prec != Py_None && context_setprec(self, prec, NULL) < 0) { + return -1; + } + if (rounding != Py_None && context_setround(self, rounding, NULL) < 0) { + return -1; + } + if (emin != Py_None && context_setemin(self, emin, NULL) < 0) { + return -1; + } + if (emax != Py_None && context_setemax(self, emax, NULL) < 0) { + return -1; + } + if (capitals != Py_None && context_setcapitals(self, capitals, NULL) < 0) { + return -1; + } + if (clamp != Py_None && context_setclamp(self, clamp, NULL) < 0) { + return -1; + } + + if (traps != Py_None) { + if (PyList_Check(traps)) { + ret = context_settraps_list(self, traps); + } +#ifdef EXTRA_FUNCTIONALITY + else if (PyLong_Check(traps)) { + ret = context_settraps(self, traps, NULL); + } +#endif + else { + ret = context_settraps_dict(self, traps); + } + if (ret < 0) { + return ret; + } + } + if (status != Py_None) { + if (PyList_Check(status)) { + ret = context_setstatus_list(self, status); + } +#ifdef EXTRA_FUNCTIONALITY + else if (PyLong_Check(status)) { + ret = context_setstatus(self, status, NULL); + } +#endif + else { + ret = context_setstatus_dict(self, status); + } + if (ret < 0) { + return ret; + } + } + + return 0; +} + static PyObject * context_clear_traps(PyObject *self, PyObject *dummy UNUSED) { @@ -1255,7 +1316,6 @@ context_init(PyObject *self, PyObject *args, PyObject *kwds) PyObject *clamp = Py_None; PyObject *status = Py_None; PyObject *traps = Py_None; - int ret; assert(PyTuple_Check(args)); @@ -1267,59 +1327,11 @@ context_init(PyObject *self, PyObject *args, PyObject *kwds) return -1; } - if (prec != Py_None && context_setprec(self, prec, NULL) < 0) { - return -1; - } - if (rounding != Py_None && context_setround(self, rounding, NULL) < 0) { - return -1; - } - if (emin != Py_None && context_setemin(self, emin, NULL) < 0) { - return -1; - } - if (emax != Py_None && context_setemax(self, emax, NULL) < 0) { - return -1; - } - if (capitals != Py_None && context_setcapitals(self, capitals, NULL) < 0) { - return -1; - } - if (clamp != Py_None && context_setclamp(self, clamp, NULL) < 0) { - return -1; - } - - if (traps != Py_None) { - if (PyList_Check(traps)) { - ret = context_settraps_list(self, traps); - } -#ifdef EXTRA_FUNCTIONALITY - else if (PyLong_Check(traps)) { - ret = context_settraps(self, traps, NULL); - } -#endif - else { - ret = context_settraps_dict(self, traps); - } - if (ret < 0) { - return ret; - } - } - if (status != Py_None) { - if (PyList_Check(status)) { - ret = context_setstatus_list(self, status); - } -#ifdef EXTRA_FUNCTIONALITY - else if (PyLong_Check(status)) { - ret = context_setstatus(self, status, NULL); - } -#endif - else { - ret = context_setstatus_dict(self, status); - } - if (ret < 0) { - return ret; - } - } - - return 0; + return context_setattrs( + self, prec, rounding, + emin, emax, capitals, + clamp, status, traps + ); } static PyObject * @@ -1721,13 +1733,28 @@ PyDec_SetCurrentContext(PyObject *self UNUSED, PyObject *v) static PyObject * ctxmanager_new(PyTypeObject *type UNUSED, PyObject *args, PyObject *kwds) { - static char *kwlist[] = {"ctx", NULL}; + static char *kwlist[] = { + "ctx", "prec", "rounding", + "Emin", "Emax", "capitals", + "clamp", "flags", "traps", + NULL + }; PyDecContextManagerObject *self; PyObject *local = Py_None; PyObject *global; + PyObject *prec = Py_None; + PyObject *rounding = Py_None; + PyObject *Emin = Py_None; + PyObject *Emax = Py_None; + PyObject *capitals = Py_None; + PyObject *clamp = Py_None; + PyObject *flags = Py_None; + PyObject *traps = Py_None; + CURRENT_CONTEXT(global); - if (!PyArg_ParseTupleAndKeywords(args, kwds, "|O", kwlist, &local)) { + if (!PyArg_ParseTupleAndKeywords(args, kwds, "|OOOOOOOOO", kwlist, &local, + &prec, &rounding, &Emin, &Emax, &capitals, &clamp, &flags, &traps)) { return NULL; } if (local == Py_None) { @@ -1754,6 +1781,17 @@ ctxmanager_new(PyTypeObject *type UNUSED, PyObject *args, PyObject *kwds) self->global = global; Py_INCREF(self->global); + int ret = context_setattrs( + self->local, prec, rounding, + Emin, Emax, capitals, + clamp, flags, traps + ); + + if (ret < 0) { + Py_DECREF(self); + return NULL; + } + return (PyObject *)self; } diff --git a/Modules/_decimal/docstrings.h b/Modules/_decimal/docstrings.h index f7fd6e7952998..a1823cdd32b74 100644 --- a/Modules/_decimal/docstrings.h +++ b/Modules/_decimal/docstrings.h @@ -30,7 +30,7 @@ Set a new default context.\n\ \n"); PyDoc_STRVAR(doc_localcontext, -"localcontext($module, /, ctx=None)\n--\n\n\ +"localcontext($module, /, ctx=None, **kwargs)\n--\n\n\ Return a context manager that will set the default context to a copy of ctx\n\ on entry to the with-statement and restore the previous default context when\n\ exiting the with-statement. If no context is specified, a copy of the current\n\ From webhook-mailer at python.org Fri Apr 22 06:45:21 2022 From: webhook-mailer at python.org (encukou) Date: Fri, 22 Apr 2022 10:45:21 -0000 Subject: [Python-checkins] Docs: Clarify availability of PyOS_CheckStack (GH-91816) Message-ID: https://github.com/python/cpython/commit/82ec638ab706577c043056a57e2a2322b43ef94a commit: 82ec638ab706577c043056a57e2a2322b43ef94a branch: main author: Petr Viktorin committer: encukou date: 2022-04-22T12:44:43+02:00 summary: Docs: Clarify availability of PyOS_CheckStack (GH-91816) files: M Doc/c-api/sys.rst M Include/pythonrun.h diff --git a/Doc/c-api/sys.rst b/Doc/c-api/sys.rst index de94e3f0180f0..66216eee6d06e 100644 --- a/Doc/c-api/sys.rst +++ b/Doc/c-api/sys.rst @@ -96,9 +96,9 @@ Operating System Utilities Return true when the interpreter runs out of stack space. This is a reliable check, but is only available when :const:`USE_STACKCHECK` is defined (currently - on Windows using the Microsoft Visual C++ compiler). :const:`USE_STACKCHECK` - will be defined automatically; you should never change the definition in your - own code. + on certain versions of Windows using the Microsoft Visual C++ compiler). + :const:`USE_STACKCHECK` will be defined automatically; you should never + change the definition in your own code. .. c:function:: PyOS_sighandler_t PyOS_getsig(int i) diff --git a/Include/pythonrun.h b/Include/pythonrun.h index 02715775581c6..1b208b734ab1b 100644 --- a/Include/pythonrun.h +++ b/Include/pythonrun.h @@ -24,6 +24,7 @@ PyAPI_DATA(int) (*PyOS_InputHook)(void); #if defined(WIN32) && !defined(MS_WIN64) && !defined(_M_ARM) && defined(_MSC_VER) && _MSC_VER >= 1300 /* Enable stack checking under Microsoft C */ +// When changing the platforms, ensure PyOS_CheckStack() docs are still correct #define USE_STACKCHECK #endif From webhook-mailer at python.org Fri Apr 22 07:05:57 2022 From: webhook-mailer at python.org (vstinner) Date: Fri, 22 Apr 2022 11:05:57 -0000 Subject: [Python-checkins] gh-80527: Deprecate PEP 623 Unicode functions (#91801) Message-ID: https://github.com/python/cpython/commit/9e146bbb7e1c0d872817ac63c60454a201b50039 commit: 9e146bbb7e1c0d872817ac63c60454a201b50039 branch: main author: Victor Stinner committer: vstinner date: 2022-04-22T13:05:36+02:00 summary: gh-80527: Deprecate PEP 623 Unicode functions (#91801) Deprecate functions: * PyUnicode_AS_DATA() * PyUnicode_AS_UNICODE() * PyUnicode_GET_DATA_SIZE() * PyUnicode_GET_SIZE() Previously, these functions were macros and so it wasn't possible to decorate them with Py_DEPRECATED(). files: A Misc/NEWS.d/next/C API/2022-04-21-23-11-35.gh-issue-80527.Cx-95G.rst M Include/cpython/unicodeobject.h diff --git a/Include/cpython/unicodeobject.h b/Include/cpython/unicodeobject.h index 5d050fd0b7563..992588c21c6ee 100644 --- a/Include/cpython/unicodeobject.h +++ b/Include/cpython/unicodeobject.h @@ -613,7 +613,7 @@ static inline Py_ssize_t PyUnicode_WSTR_LENGTH(PyObject *op) If the Py_UNICODE representation is not available, it will be computed on request. Use PyUnicode_GET_LENGTH() for the length in code points. */ -/* Py_DEPRECATED(3.3) */ +Py_DEPRECATED(3.3) static inline Py_ssize_t PyUnicode_GET_SIZE(PyObject *op) { _Py_COMP_DIAG_PUSH @@ -627,10 +627,13 @@ static inline Py_ssize_t PyUnicode_GET_SIZE(PyObject *op) } #define PyUnicode_GET_SIZE(op) PyUnicode_GET_SIZE(_PyObject_CAST(op)) - /* Py_DEPRECATED(3.3) */ - static inline Py_ssize_t PyUnicode_GET_DATA_SIZE(PyObject *op) +Py_DEPRECATED(3.3) +static inline Py_ssize_t PyUnicode_GET_DATA_SIZE(PyObject *op) { + _Py_COMP_DIAG_PUSH + _Py_COMP_DIAG_IGNORE_DEPR_DECLS return PyUnicode_GET_SIZE(op) * Py_UNICODE_SIZE; + _Py_COMP_DIAG_POP } #define PyUnicode_GET_DATA_SIZE(op) PyUnicode_GET_DATA_SIZE(_PyObject_CAST(op)) @@ -639,7 +642,7 @@ static inline Py_ssize_t PyUnicode_GET_SIZE(PyObject *op) try to port your code to use the new PyUnicode_*BYTE_DATA() macros or use PyUnicode_WRITE() and PyUnicode_READ(). */ -/* Py_DEPRECATED(3.3) */ +Py_DEPRECATED(3.3) static inline Py_UNICODE* PyUnicode_AS_UNICODE(PyObject *op) { wchar_t *wstr = _PyASCIIObject_CAST(op)->wstr; @@ -654,10 +657,13 @@ static inline Py_UNICODE* PyUnicode_AS_UNICODE(PyObject *op) } #define PyUnicode_AS_UNICODE(op) PyUnicode_AS_UNICODE(_PyObject_CAST(op)) -/* Py_DEPRECATED(3.3) */ +Py_DEPRECATED(3.3) static inline const char* PyUnicode_AS_DATA(PyObject *op) { + _Py_COMP_DIAG_PUSH + _Py_COMP_DIAG_IGNORE_DEPR_DECLS return (const char *)PyUnicode_AS_UNICODE(op); + _Py_COMP_DIAG_POP } #define PyUnicode_AS_DATA(op) PyUnicode_AS_DATA(_PyObject_CAST(op)) diff --git a/Misc/NEWS.d/next/C API/2022-04-21-23-11-35.gh-issue-80527.Cx-95G.rst b/Misc/NEWS.d/next/C API/2022-04-21-23-11-35.gh-issue-80527.Cx-95G.rst new file mode 100644 index 0000000000000..45e92f1b96759 --- /dev/null +++ b/Misc/NEWS.d/next/C API/2022-04-21-23-11-35.gh-issue-80527.Cx-95G.rst @@ -0,0 +1,3 @@ +Mark functions as deprecated by :pep:`623`: :c:func:`PyUnicode_AS_DATA`, +:c:func:`PyUnicode_AS_UNICODE`, :c:func:`PyUnicode_GET_DATA_SIZE`, +:c:func:`PyUnicode_GET_SIZE`. Patch by Victor Stinner. From webhook-mailer at python.org Fri Apr 22 08:59:27 2022 From: webhook-mailer at python.org (vstinner) Date: Fri, 22 Apr 2022 12:59:27 -0000 Subject: [Python-checkins] gh-89653: PEP 670: Amend docs (GH-91813) Message-ID: https://github.com/python/cpython/commit/121806160962d4e96b5e874a173d515531ae994b commit: 121806160962d4e96b5e874a173d515531ae994b branch: main author: Erlend Egeberg Aasland committer: vstinner date: 2022-04-22T14:59:18+02:00 summary: gh-89653: PEP 670: Amend docs (GH-91813) files: M Doc/c-api/list.rst M Doc/c-api/tuple.rst M Doc/c-api/unicode.rst M Doc/c-api/weakref.rst diff --git a/Doc/c-api/list.rst b/Doc/c-api/list.rst index f338e2ae06689..f9e65354a259f 100644 --- a/Doc/c-api/list.rst +++ b/Doc/c-api/list.rst @@ -53,7 +53,7 @@ List Objects .. c:function:: Py_ssize_t PyList_GET_SIZE(PyObject *list) - Macro form of :c:func:`PyList_Size` without error checking. + Similar to :c:func:`PyList_Size`, but without error checking. .. c:function:: PyObject* PyList_GetItem(PyObject *list, Py_ssize_t index) @@ -66,7 +66,7 @@ List Objects .. c:function:: PyObject* PyList_GET_ITEM(PyObject *list, Py_ssize_t i) - Macro form of :c:func:`PyList_GetItem` without error checking. + Similar to :c:func:`PyList_GetItem`, but without error checking. .. c:function:: int PyList_SetItem(PyObject *list, Py_ssize_t index, PyObject *item) diff --git a/Doc/c-api/tuple.rst b/Doc/c-api/tuple.rst index 6919e61022788..9b85522600d4e 100644 --- a/Doc/c-api/tuple.rst +++ b/Doc/c-api/tuple.rst @@ -91,7 +91,7 @@ Tuple Objects .. note:: - This macro "steals" a reference to *o*, and, unlike + This function "steals" a reference to *o*, and, unlike :c:func:`PyTuple_SetItem`, does *not* discard a reference to any item that is being replaced; any reference in the tuple at position *pos* will be leaked. @@ -215,7 +215,8 @@ type. .. c:function:: void PyStructSequence_SET_ITEM(PyObject *p, Py_ssize_t *pos, PyObject *o) - Macro equivalent of :c:func:`PyStructSequence_SetItem`. + Similar to :c:func:`PyStructSequence_SetItem`, but implemented as a static + inlined function. .. note:: diff --git a/Doc/c-api/unicode.rst b/Doc/c-api/unicode.rst index 11aa751cdad15..00faac5b69abd 100644 --- a/Doc/c-api/unicode.rst +++ b/Doc/c-api/unicode.rst @@ -84,8 +84,8 @@ Python: is exposed to Python code as ``str``. -The following APIs are really C macros and can be used to do fast checks and to -access internal read-only data of Unicode objects: +The following APIs are C macros and static inlined functions for fast checks and +access to internal read-only data of Unicode objects: .. c:function:: int PyUnicode_Check(PyObject *o) @@ -168,20 +168,21 @@ access internal read-only data of Unicode objects: .. versionadded:: 3.3 -.. c:function:: void PyUnicode_WRITE(int kind, void *data, Py_ssize_t index, \ - Py_UCS4 value) +.. c:function:: void PyUnicode_WRITE(unsigned int kind, void *data, \ + Py_ssize_t index, Py_UCS4 value) Write into a canonical representation *data* (as obtained with - :c:func:`PyUnicode_DATA`). This macro does not do any sanity checks and is + :c:func:`PyUnicode_DATA`). This function performs no sanity checks, and is intended for usage in loops. The caller should cache the *kind* value and - *data* pointer as obtained from other macro calls. *index* is the index in + *data* pointer as obtained from other calls. *index* is the index in the string (starts at 0) and *value* is the new code point value which should be written to that location. .. versionadded:: 3.3 -.. c:function:: Py_UCS4 PyUnicode_READ(int kind, void *data, Py_ssize_t index) +.. c:function:: Py_UCS4 PyUnicode_READ(unsigned int kind, void *data, \ + Py_ssize_t index) Read a code point from a canonical representation *data* (as obtained with :c:func:`PyUnicode_DATA`). No checks or ready calls are performed. @@ -198,7 +199,7 @@ access internal read-only data of Unicode objects: .. versionadded:: 3.3 -.. c:macro:: PyUnicode_MAX_CHAR_VALUE(o) +.. c:function:: Py_UCS4 PyUnicode_MAX_CHAR_VALUE(PyObject *o) Return the maximum code point that is suitable for creating another string based on *o*, which must be in the "canonical" representation. This is @@ -239,7 +240,7 @@ access internal read-only data of Unicode objects: a Unicode object (not checked). .. versionchanged:: 3.3 - This macro is now inefficient -- because in many cases the + This function is now inefficient -- because in many cases the :c:type:`Py_UNICODE` representation does not exist and needs to be created -- and can fail (return ``NULL`` with an exception set). Try to port the code to use the new :c:func:`PyUnicode_nBYTE_DATA` macros or use @@ -642,8 +643,8 @@ APIs: .. c:function:: Py_UCS4 PyUnicode_ReadChar(PyObject *unicode, Py_ssize_t index) Read a character from a string. This function checks that *unicode* is a - Unicode object and the index is not out of bounds, in contrast to the macro - version :c:func:`PyUnicode_READ_CHAR`. + Unicode object and the index is not out of bounds, in contrast to + :c:func:`PyUnicode_READ_CHAR`, which performs no error checking. .. versionadded:: 3.3 diff --git a/Doc/c-api/weakref.rst b/Doc/c-api/weakref.rst index 98ebe711adaeb..7b32e17a23972 100644 --- a/Doc/c-api/weakref.rst +++ b/Doc/c-api/weakref.rst @@ -66,5 +66,4 @@ as much as it can. .. c:function:: PyObject* PyWeakref_GET_OBJECT(PyObject *ref) - Similar to :c:func:`PyWeakref_GetObject`, but implemented as a macro that does no - error checking. + Similar to :c:func:`PyWeakref_GetObject`, but does no error checking. From webhook-mailer at python.org Fri Apr 22 10:28:50 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Fri, 22 Apr 2022 14:28:50 -0000 Subject: [Python-checkins] gh-91764: Mark version typing.Unpack and LiteralString were added in (#91817) Message-ID: https://github.com/python/cpython/commit/9fe82d0b70d9d90ca78d6bacdc2e5eb2b3278e23 commit: 9fe82d0b70d9d90ca78d6bacdc2e5eb2b3278e23 branch: main author: Dominic Davis-Foster committer: JelleZijlstra date: 2022-04-22T07:28:39-07:00 summary: gh-91764: Mark version typing.Unpack and LiteralString were added in (#91817) Closes GH-91764 files: M Doc/library/typing.rst diff --git a/Doc/library/typing.rst b/Doc/library/typing.rst index 6b2a0934171a2..2f62193e65c29 100644 --- a/Doc/library/typing.rst +++ b/Doc/library/typing.rst @@ -614,6 +614,8 @@ These can be used as types in annotations and do not support ``[]``. that generate type checker errors could be vulnerable to an SQL injection attack. + .. versionadded:: 3.11 + .. data:: Never The `bottom type `_, @@ -1383,6 +1385,8 @@ These are not used in annotations. They are building blocks for creating generic tup: tuple[*Ts] # Syntax error on Python <= 3.10! tup: tuple[Unpack[Ts]] # Semantically equivalent, and backwards-compatible + .. versionadded:: 3.11 + .. class:: ParamSpec(name, *, bound=None, covariant=False, contravariant=False) Parameter specification variable. A specialized version of From webhook-mailer at python.org Fri Apr 22 10:46:45 2022 From: webhook-mailer at python.org (gvanrossum) Date: Fri, 22 Apr 2022 14:46:45 -0000 Subject: [Python-checkins] gh-89279: In ceval.c, redefine some macros for speed (#32387) Message-ID: https://github.com/python/cpython/commit/2f233fceae9a0c5e66e439bc0169b36547ba47c3 commit: 2f233fceae9a0c5e66e439bc0169b36547ba47c3 branch: main author: Guido van Rossum committer: gvanrossum date: 2022-04-22T07:46:26-07:00 summary: gh-89279: In ceval.c, redefine some macros for speed (#32387) Macros Py_DECREF, Py_XDECREF, Py_IS_TYPE, _Py_atomic_load_32bit_impl and _Py_DECREF_SPECIALIZED are redefined as macros that completely replace the inline functions of the same name. These three came out in the top four of functions that (in MSVC) somehow weren't inlined. Co-authored-by: Victor Stinner Co-authored-by: Dennis Sweeney <36520290+sweeneyde at users.noreply.github.com> files: A Misc/NEWS.d/next/Core and Builtins/2022-04-13-22-03-04.gh-issue-89279.-jAVxZ.rst M Python/ceval.c diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-04-13-22-03-04.gh-issue-89279.-jAVxZ.rst b/Misc/NEWS.d/next/Core and Builtins/2022-04-13-22-03-04.gh-issue-89279.-jAVxZ.rst new file mode 100644 index 0000000000000..66b1c884ec450 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2022-04-13-22-03-04.gh-issue-89279.-jAVxZ.rst @@ -0,0 +1 @@ +Improve interpreter performance on Windows by inlining a few specific macros. diff --git a/Python/ceval.c b/Python/ceval.c index a80583f5abd12..90068bc51a7f6 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -44,6 +44,55 @@ # error "ceval.c must be build with Py_BUILD_CORE define for best performance" #endif +#ifndef Py_DEBUG +// GH-89279: The MSVC compiler does not inline these static inline functions +// in PGO build in _PyEval_EvalFrameDefault(), because this function is over +// the limit of PGO, and that limit cannot be configured. +// Define them as macros to make sure that they are always inlined by the +// preprocessor. + +#undef Py_DECREF +#define Py_DECREF(arg) \ + do { \ + PyObject *op = _PyObject_CAST(arg); \ + if (--op->ob_refcnt == 0) { \ + destructor dealloc = Py_TYPE(op)->tp_dealloc; \ + (*dealloc)(op); \ + } \ + } while (0) + +#undef Py_XDECREF +#define Py_XDECREF(arg) \ + do { \ + PyObject *xop = _PyObject_CAST(arg); \ + if (xop != NULL) { \ + Py_DECREF(xop); \ + } \ + } while (0) + +#undef Py_IS_TYPE +#define Py_IS_TYPE(ob, type) \ + (_PyObject_CAST(ob)->ob_type == (type)) + +#undef _Py_DECREF_SPECIALIZED +#define _Py_DECREF_SPECIALIZED(arg, dealloc) \ + do { \ + PyObject *op = _PyObject_CAST(arg); \ + if (--op->ob_refcnt == 0) { \ + destructor d = (destructor)(dealloc); \ + d(op); \ + } \ + } while (0) +#endif + +// GH-89279: Similar to above, force inlining by using a macro. +#if defined(_MSC_VER) && SIZEOF_INT == 4 +#define _Py_atomic_load_relaxed_int32(ATOMIC_VAL) (assert(sizeof((ATOMIC_VAL)->_value) == 4), *((volatile int*)&((ATOMIC_VAL)->_value))) +#else +#define _Py_atomic_load_relaxed_int32(ATOMIC_VAL) _Py_atomic_load_relaxed(ATOMIC_VAL) +#endif + + /* Forward declarations */ static PyObject *trace_call_function( PyThreadState *tstate, PyObject *callable, PyObject **stack, @@ -192,10 +241,10 @@ COMPUTE_EVAL_BREAKER(PyInterpreterState *interp, struct _ceval_state *ceval2) { _Py_atomic_store_relaxed(&ceval2->eval_breaker, - _Py_atomic_load_relaxed(&ceval2->gil_drop_request) - | (_Py_atomic_load_relaxed(&ceval->signals_pending) + _Py_atomic_load_relaxed_int32(&ceval2->gil_drop_request) + | (_Py_atomic_load_relaxed_int32(&ceval->signals_pending) && _Py_ThreadCanHandleSignals(interp)) - | (_Py_atomic_load_relaxed(&ceval2->pending.calls_to_do) + | (_Py_atomic_load_relaxed_int32(&ceval2->pending.calls_to_do) && _Py_ThreadCanHandlePendingCalls()) | ceval2->pending.async_exc); } @@ -740,7 +789,7 @@ _Py_FinishPendingCalls(PyThreadState *tstate) struct _pending_calls *pending = &tstate->interp->ceval.pending; - if (!_Py_atomic_load_relaxed(&(pending->calls_to_do))) { + if (!_Py_atomic_load_relaxed_int32(&(pending->calls_to_do))) { return; } @@ -1187,7 +1236,7 @@ eval_frame_handle_pending(PyThreadState *tstate) struct _ceval_runtime_state *ceval = &runtime->ceval; /* Pending signals */ - if (_Py_atomic_load_relaxed(&ceval->signals_pending)) { + if (_Py_atomic_load_relaxed_int32(&ceval->signals_pending)) { if (handle_signals(tstate) != 0) { return -1; } @@ -1195,14 +1244,14 @@ eval_frame_handle_pending(PyThreadState *tstate) /* Pending calls */ struct _ceval_state *ceval2 = &tstate->interp->ceval; - if (_Py_atomic_load_relaxed(&ceval2->pending.calls_to_do)) { + if (_Py_atomic_load_relaxed_int32(&ceval2->pending.calls_to_do)) { if (make_pending_calls(tstate->interp) != 0) { return -1; } } /* GIL drop request */ - if (_Py_atomic_load_relaxed(&ceval2->gil_drop_request)) { + if (_Py_atomic_load_relaxed_int32(&ceval2->gil_drop_request)) { /* Give another thread a chance */ if (_PyThreadState_Swap(&runtime->gilstate, NULL) != tstate) { Py_FatalError("tstate mix-up"); @@ -1360,7 +1409,7 @@ eval_frame_handle_pending(PyThreadState *tstate) #define CHECK_EVAL_BREAKER() \ _Py_CHECK_EMSCRIPTEN_SIGNALS_PERIODICALLY(); \ - if (_Py_atomic_load_relaxed(eval_breaker)) { \ + if (_Py_atomic_load_relaxed_int32(eval_breaker)) { \ goto handle_eval_breaker; \ } @@ -1640,10 +1689,8 @@ typedef struct { PyObject *kwnames; } CallShape; -static inline bool -is_method(PyObject **stack_pointer, int args) { - return PEEK(args+2) != NULL; -} +// GH-89279: Must be a macro to be sure it's inlined by MSVC. +#define is_method(stack_pointer, args) (PEEK((args)+2) != NULL) #define KWNAMES_LEN() \ (call_shape.kwnames == NULL ? 0 : ((int)PyTuple_GET_SIZE(call_shape.kwnames))) @@ -1796,7 +1843,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int PREDICTED(RESUME_QUICK); assert(tstate->cframe == &cframe); assert(frame == cframe.current_frame); - if (_Py_atomic_load_relaxed(eval_breaker) && oparg < 2) { + if (_Py_atomic_load_relaxed_int32(eval_breaker) && oparg < 2) { goto handle_eval_breaker; } DISPATCH(); From webhook-mailer at python.org Fri Apr 22 11:35:32 2022 From: webhook-mailer at python.org (serhiy-storchaka) Date: Fri, 22 Apr 2022 15:35:32 -0000 Subject: [Python-checkins] gh-90568: Fix exception type for \N with a named sequence in RE (GH-91665) Message-ID: https://github.com/python/cpython/commit/6ccfa31421393910b52936e0447625db06f2a655 commit: 6ccfa31421393910b52936e0447625db06f2a655 branch: main author: Serhiy Storchaka committer: serhiy-storchaka date: 2022-04-22T18:35:28+03:00 summary: gh-90568: Fix exception type for \N with a named sequence in RE (GH-91665) re.error is now raised instead of TypeError. files: A Misc/NEWS.d/next/Library/2022-04-18-16-31-33.gh-issue-90568.9kiU7o.rst M Lib/re/_parser.py M Lib/test/test_re.py diff --git a/Lib/re/_parser.py b/Lib/re/_parser.py index f191f809a1491..6588862493077 100644 --- a/Lib/re/_parser.py +++ b/Lib/re/_parser.py @@ -333,7 +333,7 @@ def _class_escape(source, escape): charname = source.getuntil('}', 'character name') try: c = ord(unicodedata.lookup(charname)) - except KeyError: + except (KeyError, TypeError): raise source.error("undefined character name %r" % charname, len(charname) + len(r'\N{}')) from None return LITERAL, c @@ -393,7 +393,7 @@ def _escape(source, escape, state): charname = source.getuntil('}', 'character name') try: c = ord(unicodedata.lookup(charname)) - except KeyError: + except (KeyError, TypeError): raise source.error("undefined character name %r" % charname, len(charname) + len(r'\N{}')) from None return LITERAL, c diff --git a/Lib/test/test_re.py b/Lib/test/test_re.py index 781bfd6ea2eda..2d3fef8589e2a 100644 --- a/Lib/test/test_re.py +++ b/Lib/test/test_re.py @@ -772,6 +772,10 @@ def test_named_unicode_escapes(self): "undefined character name 'SPAM'", 0) self.checkPatternError(r'[\N{SPAM}]', "undefined character name 'SPAM'", 1) + self.checkPatternError(r'\N{KEYCAP NUMBER SIGN}', + "undefined character name 'KEYCAP NUMBER SIGN'", 0) + self.checkPatternError(r'[\N{KEYCAP NUMBER SIGN}]', + "undefined character name 'KEYCAP NUMBER SIGN'", 1) self.checkPatternError(br'\N{LESS-THAN SIGN}', r'bad escape \N', 0) self.checkPatternError(br'[\N{LESS-THAN SIGN}]', r'bad escape \N', 1) diff --git a/Misc/NEWS.d/next/Library/2022-04-18-16-31-33.gh-issue-90568.9kiU7o.rst b/Misc/NEWS.d/next/Library/2022-04-18-16-31-33.gh-issue-90568.9kiU7o.rst new file mode 100644 index 0000000000000..4411c715830e2 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-18-16-31-33.gh-issue-90568.9kiU7o.rst @@ -0,0 +1,3 @@ +Parsing ``\N`` escapes of Unicode Named Character Sequences in a +:mod:`regular expression ` raises now :exc:`re.error` instead of +``TypeError``. From webhook-mailer at python.org Fri Apr 22 12:53:15 2022 From: webhook-mailer at python.org (serhiy-storchaka) Date: Fri, 22 Apr 2022 16:53:15 -0000 Subject: [Python-checkins] gh-91700: Validate the group number in conditional expression in RE (GH-91702) Message-ID: https://github.com/python/cpython/commit/48ec61a89a959071206549819448405c2cea61b0 commit: 48ec61a89a959071206549819448405c2cea61b0 branch: main author: Serhiy Storchaka committer: serhiy-storchaka date: 2022-04-22T19:53:10+03:00 summary: gh-91700: Validate the group number in conditional expression in RE (GH-91702) In expression (?(group)...) an appropriate re.error is now raised if the group number refers to not defined group. Previously it raised RuntimeError: invalid SRE code. files: A Misc/NEWS.d/next/Library/2022-04-19-17-30-17.gh-issue-91700.MRJi6m.rst M Lib/re/_parser.py M Lib/test/test_re.py diff --git a/Lib/re/_parser.py b/Lib/re/_parser.py index 6588862493077..60ec3e8ba8bd5 100644 --- a/Lib/re/_parser.py +++ b/Lib/re/_parser.py @@ -77,6 +77,7 @@ def __init__(self): self.groupdict = {} self.groupwidths = [None] # group 0 self.lookbehindgroups = None + self.grouprefpos = {} @property def groups(self): return len(self.groupwidths) @@ -795,6 +796,10 @@ def _parse(source, state, verbose, nested, first=False): if condgroup >= MAXGROUPS: msg = "invalid group reference %d" % condgroup raise source.error(msg, len(condname) + 1) + if condgroup not in state.grouprefpos: + state.grouprefpos[condgroup] = ( + source.tell() - len(condname) - 1 + ) state.checklookbehindgroup(condgroup, source) item_yes = _parse(source, state, verbose, nested + 1) if source.match("|"): @@ -975,6 +980,11 @@ def parse(str, flags=0, state=None): assert source.next == ")" raise source.error("unbalanced parenthesis") + for g in p.state.grouprefpos: + if g >= p.state.groups: + msg = "invalid group reference %d" % g + raise error(msg, str, p.state.grouprefpos[g]) + if flags & SRE_FLAG_DEBUG: p.dump() diff --git a/Lib/test/test_re.py b/Lib/test/test_re.py index 2d3fef8589e2a..700275063f0f1 100644 --- a/Lib/test/test_re.py +++ b/Lib/test/test_re.py @@ -593,6 +593,8 @@ def test_re_groupref_exists_errors(self): self.checkPatternError(r'()(?(1)a|b|c)', 'conditional backref with more than ' 'two branches', 10) + self.checkPatternError(r'()(?(2)a)', + "invalid group reference 2", 5) def test_re_groupref_overflow(self): from re._constants import MAXGROUPS diff --git a/Misc/NEWS.d/next/Library/2022-04-19-17-30-17.gh-issue-91700.MRJi6m.rst b/Misc/NEWS.d/next/Library/2022-04-19-17-30-17.gh-issue-91700.MRJi6m.rst new file mode 100644 index 0000000000000..73b106869697b --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-19-17-30-17.gh-issue-91700.MRJi6m.rst @@ -0,0 +1,4 @@ +Compilation of regular expression containing a conditional expression +``(?(group)...)`` now raises an appropriate :exc:`re.error` if the group +number refers to not defined group. Previously an internal RuntimeError was +raised. From webhook-mailer at python.org Fri Apr 22 14:08:59 2022 From: webhook-mailer at python.org (serhiy-storchaka) Date: Fri, 22 Apr 2022 18:08:59 -0000 Subject: [Python-checkins] [3.10] gh-90568: Fix exception type for \N with a named sequence in RE (GH-91665) (GH-91830) Message-ID: https://github.com/python/cpython/commit/9c18d783c38fca57a63b61aa778d8a8d18945d95 commit: 9c18d783c38fca57a63b61aa778d8a8d18945d95 branch: 3.10 author: Serhiy Storchaka committer: serhiy-storchaka date: 2022-04-22T21:08:49+03:00 summary: [3.10] gh-90568: Fix exception type for \N with a named sequence in RE (GH-91665) (GH-91830) re.error is now raised instead of TypeError. (cherry picked from commit 6ccfa31421393910b52936e0447625db06f2a655) files: A Misc/NEWS.d/next/Library/2022-04-18-16-31-33.gh-issue-90568.9kiU7o.rst M Lib/sre_parse.py M Lib/test/test_re.py diff --git a/Lib/sre_parse.py b/Lib/sre_parse.py index 53706676e9f7b..d3ff196032b30 100644 --- a/Lib/sre_parse.py +++ b/Lib/sre_parse.py @@ -330,7 +330,7 @@ def _class_escape(source, escape): charname = source.getuntil('}', 'character name') try: c = ord(unicodedata.lookup(charname)) - except KeyError: + except (KeyError, TypeError): raise source.error("undefined character name %r" % charname, len(charname) + len(r'\N{}')) return LITERAL, c @@ -390,7 +390,7 @@ def _escape(source, escape, state): charname = source.getuntil('}', 'character name') try: c = ord(unicodedata.lookup(charname)) - except KeyError: + except (KeyError, TypeError): raise source.error("undefined character name %r" % charname, len(charname) + len(r'\N{}')) return LITERAL, c diff --git a/Lib/test/test_re.py b/Lib/test/test_re.py index 9e5223a125b05..305ec8eef326b 100644 --- a/Lib/test/test_re.py +++ b/Lib/test/test_re.py @@ -754,6 +754,10 @@ def test_named_unicode_escapes(self): "undefined character name 'SPAM'", 0) self.checkPatternError(r'[\N{SPAM}]', "undefined character name 'SPAM'", 1) + self.checkPatternError(r'\N{KEYCAP NUMBER SIGN}', + "undefined character name 'KEYCAP NUMBER SIGN'", 0) + self.checkPatternError(r'[\N{KEYCAP NUMBER SIGN}]', + "undefined character name 'KEYCAP NUMBER SIGN'", 1) self.checkPatternError(br'\N{LESS-THAN SIGN}', r'bad escape \N', 0) self.checkPatternError(br'[\N{LESS-THAN SIGN}]', r'bad escape \N', 1) diff --git a/Misc/NEWS.d/next/Library/2022-04-18-16-31-33.gh-issue-90568.9kiU7o.rst b/Misc/NEWS.d/next/Library/2022-04-18-16-31-33.gh-issue-90568.9kiU7o.rst new file mode 100644 index 0000000000000..4411c715830e2 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-18-16-31-33.gh-issue-90568.9kiU7o.rst @@ -0,0 +1,3 @@ +Parsing ``\N`` escapes of Unicode Named Character Sequences in a +:mod:`regular expression ` raises now :exc:`re.error` instead of +``TypeError``. From webhook-mailer at python.org Fri Apr 22 14:09:35 2022 From: webhook-mailer at python.org (serhiy-storchaka) Date: Fri, 22 Apr 2022 18:09:35 -0000 Subject: [Python-checkins] [3.10] gh-91700: Validate the group number in conditional expression in RE (GH-91702) (GH-91831) Message-ID: https://github.com/python/cpython/commit/080781cd49b13da4a73db87b6f5e0c7aeec83e92 commit: 080781cd49b13da4a73db87b6f5e0c7aeec83e92 branch: 3.10 author: Serhiy Storchaka committer: serhiy-storchaka date: 2022-04-22T21:09:30+03:00 summary: [3.10] gh-91700: Validate the group number in conditional expression in RE (GH-91702) (GH-91831) In expression (?(group)...) an appropriate re.error is now raised if the group number refers to not defined group. Previously it raised RuntimeError: invalid SRE code. (cherry picked from commit 48ec61a89a959071206549819448405c2cea61b0) files: A Misc/NEWS.d/next/Library/2022-04-19-17-30-17.gh-issue-91700.MRJi6m.rst M Lib/sre_parse.py M Lib/test/test_re.py diff --git a/Lib/sre_parse.py b/Lib/sre_parse.py index d3ff196032b30..20a602501191f 100644 --- a/Lib/sre_parse.py +++ b/Lib/sre_parse.py @@ -78,6 +78,7 @@ def __init__(self): self.groupdict = {} self.groupwidths = [None] # group 0 self.lookbehindgroups = None + self.grouprefpos = {} @property def groups(self): return len(self.groupwidths) @@ -786,6 +787,10 @@ def _parse(source, state, verbose, nested, first=False): if condgroup >= MAXGROUPS: msg = "invalid group reference %d" % condgroup raise source.error(msg, len(condname) + 1) + if condgroup not in state.grouprefpos: + state.grouprefpos[condgroup] = ( + source.tell() - len(condname) - 1 + ) state.checklookbehindgroup(condgroup, source) item_yes = _parse(source, state, verbose, nested + 1) if source.match("|"): @@ -963,6 +968,11 @@ def parse(str, flags=0, state=None): assert source.next == ")" raise source.error("unbalanced parenthesis") + for g in p.state.grouprefpos: + if g >= p.state.groups: + msg = "invalid group reference %d" % g + raise error(msg, str, p.state.grouprefpos[g]) + if flags & SRE_FLAG_DEBUG: p.dump() diff --git a/Lib/test/test_re.py b/Lib/test/test_re.py index 305ec8eef326b..887ab781fc9d4 100644 --- a/Lib/test/test_re.py +++ b/Lib/test/test_re.py @@ -575,6 +575,8 @@ def test_re_groupref_exists_errors(self): self.checkPatternError(r'()(?(1)a|b|c)', 'conditional backref with more than ' 'two branches', 10) + self.checkPatternError(r'()(?(2)a)', + "invalid group reference 2", 5) def test_re_groupref_overflow(self): from sre_constants import MAXGROUPS diff --git a/Misc/NEWS.d/next/Library/2022-04-19-17-30-17.gh-issue-91700.MRJi6m.rst b/Misc/NEWS.d/next/Library/2022-04-19-17-30-17.gh-issue-91700.MRJi6m.rst new file mode 100644 index 0000000000000..73b106869697b --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-19-17-30-17.gh-issue-91700.MRJi6m.rst @@ -0,0 +1,4 @@ +Compilation of regular expression containing a conditional expression +``(?(group)...)`` now raises an appropriate :exc:`re.error` if the group +number refers to not defined group. Previously an internal RuntimeError was +raised. From webhook-mailer at python.org Fri Apr 22 14:35:13 2022 From: webhook-mailer at python.org (serhiy-storchaka) Date: Fri, 22 Apr 2022 18:35:13 -0000 Subject: [Python-checkins] [3.9] gh-90568: Fix exception type for \N with a named sequence in RE (GH-91665) (GH-91830) (GH-91834) Message-ID: https://github.com/python/cpython/commit/97d14e1dfb9347bc8ef581055b2f70cd03e5f622 commit: 97d14e1dfb9347bc8ef581055b2f70cd03e5f622 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: serhiy-storchaka date: 2022-04-22T21:34:31+03:00 summary: [3.9] gh-90568: Fix exception type for \N with a named sequence in RE (GH-91665) (GH-91830) (GH-91834) re.error is now raised instead of TypeError. (cherry picked from commit 6ccfa31421393910b52936e0447625db06f2a655) (cherry picked from commit 9c18d783c38fca57a63b61aa778d8a8d18945d95) Co-authored-by: Serhiy Storchaka files: A Misc/NEWS.d/next/Library/2022-04-18-16-31-33.gh-issue-90568.9kiU7o.rst M Lib/sre_parse.py M Lib/test/test_re.py diff --git a/Lib/sre_parse.py b/Lib/sre_parse.py index 53706676e9f7b..d3ff196032b30 100644 --- a/Lib/sre_parse.py +++ b/Lib/sre_parse.py @@ -330,7 +330,7 @@ def _class_escape(source, escape): charname = source.getuntil('}', 'character name') try: c = ord(unicodedata.lookup(charname)) - except KeyError: + except (KeyError, TypeError): raise source.error("undefined character name %r" % charname, len(charname) + len(r'\N{}')) return LITERAL, c @@ -390,7 +390,7 @@ def _escape(source, escape, state): charname = source.getuntil('}', 'character name') try: c = ord(unicodedata.lookup(charname)) - except KeyError: + except (KeyError, TypeError): raise source.error("undefined character name %r" % charname, len(charname) + len(r'\N{}')) return LITERAL, c diff --git a/Lib/test/test_re.py b/Lib/test/test_re.py index 56e98b7aedce7..007064093c4d1 100644 --- a/Lib/test/test_re.py +++ b/Lib/test/test_re.py @@ -753,6 +753,10 @@ def test_named_unicode_escapes(self): "undefined character name 'SPAM'", 0) self.checkPatternError(r'[\N{SPAM}]', "undefined character name 'SPAM'", 1) + self.checkPatternError(r'\N{KEYCAP NUMBER SIGN}', + "undefined character name 'KEYCAP NUMBER SIGN'", 0) + self.checkPatternError(r'[\N{KEYCAP NUMBER SIGN}]', + "undefined character name 'KEYCAP NUMBER SIGN'", 1) self.checkPatternError(br'\N{LESS-THAN SIGN}', r'bad escape \N', 0) self.checkPatternError(br'[\N{LESS-THAN SIGN}]', r'bad escape \N', 1) diff --git a/Misc/NEWS.d/next/Library/2022-04-18-16-31-33.gh-issue-90568.9kiU7o.rst b/Misc/NEWS.d/next/Library/2022-04-18-16-31-33.gh-issue-90568.9kiU7o.rst new file mode 100644 index 0000000000000..4411c715830e2 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-18-16-31-33.gh-issue-90568.9kiU7o.rst @@ -0,0 +1,3 @@ +Parsing ``\N`` escapes of Unicode Named Character Sequences in a +:mod:`regular expression ` raises now :exc:`re.error` instead of +``TypeError``. From webhook-mailer at python.org Fri Apr 22 14:38:00 2022 From: webhook-mailer at python.org (serhiy-storchaka) Date: Fri, 22 Apr 2022 18:38:00 -0000 Subject: [Python-checkins] gh-91575: Add a script for generating data for case-insensitive matching in re (GH-91660) Message-ID: https://github.com/python/cpython/commit/f912cc0e413f667a8cc257a41775272bc641b0d8 commit: f912cc0e413f667a8cc257a41775272bc641b0d8 branch: main author: Serhiy Storchaka committer: serhiy-storchaka date: 2022-04-22T21:37:46+03:00 summary: gh-91575: Add a script for generating data for case-insensitive matching in re (GH-91660) Also test that all extra cases are in BMP. files: A Lib/re/_casefix.py A Misc/NEWS.d/next/Tools-Demos/2022-04-18-12-52-16.gh-issue-91575.fK1TEh.rst A Tools/scripts/generate_re_casefix.py M Lib/re/_compiler.py M Makefile.pre.in diff --git a/Lib/re/_casefix.py b/Lib/re/_casefix.py new file mode 100644 index 0000000000000..06507d08bee02 --- /dev/null +++ b/Lib/re/_casefix.py @@ -0,0 +1,106 @@ +# Auto-generated by Tools/scripts/generate_re_casefix.py. + +# Maps the code of lowercased character to codes of different lowercased +# characters which have the same uppercase. +_EXTRA_CASES = { + # LATIN SMALL LETTER I: LATIN SMALL LETTER DOTLESS I + 0x0069: (0x0131,), # 'i': '?' + # LATIN SMALL LETTER S: LATIN SMALL LETTER LONG S + 0x0073: (0x017f,), # 's': '?' + # MICRO SIGN: GREEK SMALL LETTER MU + 0x00b5: (0x03bc,), # '?': '?' + # LATIN SMALL LETTER DOTLESS I: LATIN SMALL LETTER I + 0x0131: (0x0069,), # '?': 'i' + # LATIN SMALL LETTER LONG S: LATIN SMALL LETTER S + 0x017f: (0x0073,), # '?': 's' + # COMBINING GREEK YPOGEGRAMMENI: GREEK SMALL LETTER IOTA, GREEK PROSGEGRAMMENI + 0x0345: (0x03b9, 0x1fbe), # '\u0345': '??' + # GREEK SMALL LETTER IOTA WITH DIALYTIKA AND TONOS: GREEK SMALL LETTER IOTA WITH DIALYTIKA AND OXIA + 0x0390: (0x1fd3,), # '?': '?' + # GREEK SMALL LETTER UPSILON WITH DIALYTIKA AND TONOS: GREEK SMALL LETTER UPSILON WITH DIALYTIKA AND OXIA + 0x03b0: (0x1fe3,), # '?': '?' + # GREEK SMALL LETTER BETA: GREEK BETA SYMBOL + 0x03b2: (0x03d0,), # '?': '?' + # GREEK SMALL LETTER EPSILON: GREEK LUNATE EPSILON SYMBOL + 0x03b5: (0x03f5,), # '?': '?' + # GREEK SMALL LETTER THETA: GREEK THETA SYMBOL + 0x03b8: (0x03d1,), # '?': '?' + # GREEK SMALL LETTER IOTA: COMBINING GREEK YPOGEGRAMMENI, GREEK PROSGEGRAMMENI + 0x03b9: (0x0345, 0x1fbe), # '?': '\u0345?' + # GREEK SMALL LETTER KAPPA: GREEK KAPPA SYMBOL + 0x03ba: (0x03f0,), # '?': '?' + # GREEK SMALL LETTER MU: MICRO SIGN + 0x03bc: (0x00b5,), # '?': '?' + # GREEK SMALL LETTER PI: GREEK PI SYMBOL + 0x03c0: (0x03d6,), # '?': '?' + # GREEK SMALL LETTER RHO: GREEK RHO SYMBOL + 0x03c1: (0x03f1,), # '?': '?' + # GREEK SMALL LETTER FINAL SIGMA: GREEK SMALL LETTER SIGMA + 0x03c2: (0x03c3,), # '?': '?' + # GREEK SMALL LETTER SIGMA: GREEK SMALL LETTER FINAL SIGMA + 0x03c3: (0x03c2,), # '?': '?' + # GREEK SMALL LETTER PHI: GREEK PHI SYMBOL + 0x03c6: (0x03d5,), # '?': '?' + # GREEK BETA SYMBOL: GREEK SMALL LETTER BETA + 0x03d0: (0x03b2,), # '?': '?' + # GREEK THETA SYMBOL: GREEK SMALL LETTER THETA + 0x03d1: (0x03b8,), # '?': '?' + # GREEK PHI SYMBOL: GREEK SMALL LETTER PHI + 0x03d5: (0x03c6,), # '?': '?' + # GREEK PI SYMBOL: GREEK SMALL LETTER PI + 0x03d6: (0x03c0,), # '?': '?' + # GREEK KAPPA SYMBOL: GREEK SMALL LETTER KAPPA + 0x03f0: (0x03ba,), # '?': '?' + # GREEK RHO SYMBOL: GREEK SMALL LETTER RHO + 0x03f1: (0x03c1,), # '?': '?' + # GREEK LUNATE EPSILON SYMBOL: GREEK SMALL LETTER EPSILON + 0x03f5: (0x03b5,), # '?': '?' + # CYRILLIC SMALL LETTER VE: CYRILLIC SMALL LETTER ROUNDED VE + 0x0432: (0x1c80,), # '?': '?' + # CYRILLIC SMALL LETTER DE: CYRILLIC SMALL LETTER LONG-LEGGED DE + 0x0434: (0x1c81,), # '?': '?' + # CYRILLIC SMALL LETTER O: CYRILLIC SMALL LETTER NARROW O + 0x043e: (0x1c82,), # '?': '?' + # CYRILLIC SMALL LETTER ES: CYRILLIC SMALL LETTER WIDE ES + 0x0441: (0x1c83,), # '?': '?' + # CYRILLIC SMALL LETTER TE: CYRILLIC SMALL LETTER TALL TE, CYRILLIC SMALL LETTER THREE-LEGGED TE + 0x0442: (0x1c84, 0x1c85), # '?': '??' + # CYRILLIC SMALL LETTER HARD SIGN: CYRILLIC SMALL LETTER TALL HARD SIGN + 0x044a: (0x1c86,), # '?': '?' + # CYRILLIC SMALL LETTER YAT: CYRILLIC SMALL LETTER TALL YAT + 0x0463: (0x1c87,), # '?': '?' + # CYRILLIC SMALL LETTER ROUNDED VE: CYRILLIC SMALL LETTER VE + 0x1c80: (0x0432,), # '?': '?' + # CYRILLIC SMALL LETTER LONG-LEGGED DE: CYRILLIC SMALL LETTER DE + 0x1c81: (0x0434,), # '?': '?' + # CYRILLIC SMALL LETTER NARROW O: CYRILLIC SMALL LETTER O + 0x1c82: (0x043e,), # '?': '?' + # CYRILLIC SMALL LETTER WIDE ES: CYRILLIC SMALL LETTER ES + 0x1c83: (0x0441,), # '?': '?' + # CYRILLIC SMALL LETTER TALL TE: CYRILLIC SMALL LETTER TE, CYRILLIC SMALL LETTER THREE-LEGGED TE + 0x1c84: (0x0442, 0x1c85), # '?': '??' + # CYRILLIC SMALL LETTER THREE-LEGGED TE: CYRILLIC SMALL LETTER TE, CYRILLIC SMALL LETTER TALL TE + 0x1c85: (0x0442, 0x1c84), # '?': '??' + # CYRILLIC SMALL LETTER TALL HARD SIGN: CYRILLIC SMALL LETTER HARD SIGN + 0x1c86: (0x044a,), # '?': '?' + # CYRILLIC SMALL LETTER TALL YAT: CYRILLIC SMALL LETTER YAT + 0x1c87: (0x0463,), # '?': '?' + # CYRILLIC SMALL LETTER UNBLENDED UK: CYRILLIC SMALL LETTER MONOGRAPH UK + 0x1c88: (0xa64b,), # '?': '?' + # LATIN SMALL LETTER S WITH DOT ABOVE: LATIN SMALL LETTER LONG S WITH DOT ABOVE + 0x1e61: (0x1e9b,), # '?': '?' + # LATIN SMALL LETTER LONG S WITH DOT ABOVE: LATIN SMALL LETTER S WITH DOT ABOVE + 0x1e9b: (0x1e61,), # '?': '?' + # GREEK PROSGEGRAMMENI: COMBINING GREEK YPOGEGRAMMENI, GREEK SMALL LETTER IOTA + 0x1fbe: (0x0345, 0x03b9), # '?': '\u0345?' + # GREEK SMALL LETTER IOTA WITH DIALYTIKA AND OXIA: GREEK SMALL LETTER IOTA WITH DIALYTIKA AND TONOS + 0x1fd3: (0x0390,), # '?': '?' + # GREEK SMALL LETTER UPSILON WITH DIALYTIKA AND OXIA: GREEK SMALL LETTER UPSILON WITH DIALYTIKA AND TONOS + 0x1fe3: (0x03b0,), # '?': '?' + # CYRILLIC SMALL LETTER MONOGRAPH UK: CYRILLIC SMALL LETTER UNBLENDED UK + 0xa64b: (0x1c88,), # '?': '?' + # LATIN SMALL LIGATURE LONG S T: LATIN SMALL LIGATURE ST + 0xfb05: (0xfb06,), # '?': '?' + # LATIN SMALL LIGATURE ST: LATIN SMALL LIGATURE LONG S T + 0xfb06: (0xfb05,), # '?': '?' +} diff --git a/Lib/re/_compiler.py b/Lib/re/_compiler.py index 065f6fbd73244..f621d04af123d 100644 --- a/Lib/re/_compiler.py +++ b/Lib/re/_compiler.py @@ -13,6 +13,7 @@ import _sre from . import _parser from ._constants import * +from ._casefix import _EXTRA_CASES assert _sre.MAGIC == MAGIC, "SRE module mismatch" @@ -27,62 +28,6 @@ POSSESSIVE_REPEAT: (POSSESSIVE_REPEAT, SUCCESS, POSSESSIVE_REPEAT_ONE), } -# Sets of lowercase characters which have the same uppercase. -_equivalences = ( - # LATIN SMALL LETTER I, LATIN SMALL LETTER DOTLESS I - (0x69, 0x131), # i? - # LATIN SMALL LETTER S, LATIN SMALL LETTER LONG S - (0x73, 0x17f), # s? - # MICRO SIGN, GREEK SMALL LETTER MU - (0xb5, 0x3bc), # ?? - # COMBINING GREEK YPOGEGRAMMENI, GREEK SMALL LETTER IOTA, GREEK PROSGEGRAMMENI - (0x345, 0x3b9, 0x1fbe), # \u0345?? - # GREEK SMALL LETTER IOTA WITH DIALYTIKA AND TONOS, GREEK SMALL LETTER IOTA WITH DIALYTIKA AND OXIA - (0x390, 0x1fd3), # ?? - # GREEK SMALL LETTER UPSILON WITH DIALYTIKA AND TONOS, GREEK SMALL LETTER UPSILON WITH DIALYTIKA AND OXIA - (0x3b0, 0x1fe3), # ?? - # GREEK SMALL LETTER BETA, GREEK BETA SYMBOL - (0x3b2, 0x3d0), # ?? - # GREEK SMALL LETTER EPSILON, GREEK LUNATE EPSILON SYMBOL - (0x3b5, 0x3f5), # ?? - # GREEK SMALL LETTER THETA, GREEK THETA SYMBOL - (0x3b8, 0x3d1), # ?? - # GREEK SMALL LETTER KAPPA, GREEK KAPPA SYMBOL - (0x3ba, 0x3f0), # ?? - # GREEK SMALL LETTER PI, GREEK PI SYMBOL - (0x3c0, 0x3d6), # ?? - # GREEK SMALL LETTER RHO, GREEK RHO SYMBOL - (0x3c1, 0x3f1), # ?? - # GREEK SMALL LETTER FINAL SIGMA, GREEK SMALL LETTER SIGMA - (0x3c2, 0x3c3), # ?? - # GREEK SMALL LETTER PHI, GREEK PHI SYMBOL - (0x3c6, 0x3d5), # ?? - # CYRILLIC SMALL LETTER VE, CYRILLIC SMALL LETTER ROUNDED VE - (0x432, 0x1c80), # ?? - # CYRILLIC SMALL LETTER DE, CYRILLIC SMALL LETTER LONG-LEGGED DE - (0x434, 0x1c81), # ?? - # CYRILLIC SMALL LETTER O, CYRILLIC SMALL LETTER NARROW O - (0x43e, 0x1c82), # ?? - # CYRILLIC SMALL LETTER ES, CYRILLIC SMALL LETTER WIDE ES - (0x441, 0x1c83), # ?? - # CYRILLIC SMALL LETTER TE, CYRILLIC SMALL LETTER TALL TE, CYRILLIC SMALL LETTER THREE-LEGGED TE - (0x442, 0x1c84, 0x1c85), # ??? - # CYRILLIC SMALL LETTER HARD SIGN, CYRILLIC SMALL LETTER TALL HARD SIGN - (0x44a, 0x1c86), # ?? - # CYRILLIC SMALL LETTER YAT, CYRILLIC SMALL LETTER TALL YAT - (0x463, 0x1c87), # ?? - # CYRILLIC SMALL LETTER UNBLENDED UK, CYRILLIC SMALL LETTER MONOGRAPH UK - (0x1c88, 0xa64b), # ?? - # LATIN SMALL LETTER S WITH DOT ABOVE, LATIN SMALL LETTER LONG S WITH DOT ABOVE - (0x1e61, 0x1e9b), # ?? - # LATIN SMALL LIGATURE LONG S T, LATIN SMALL LIGATURE ST - (0xfb05, 0xfb06), # ?? -) - -# Maps the lowercase code to lowercase codes which have the same uppercase. -_ignorecase_fixes = {i: tuple(j for j in t if i != j) - for t in _equivalences for i in t} - class _CompileData: __slots__ = ('code', 'repeat_count') def __init__(self): @@ -111,7 +56,7 @@ def _compile(data, pattern, flags): if flags & SRE_FLAG_UNICODE: iscased = _sre.unicode_iscased tolower = _sre.unicode_tolower - fixes = _ignorecase_fixes + fixes = _EXTRA_CASES else: iscased = _sre.ascii_iscased tolower = _sre.ascii_tolower diff --git a/Makefile.pre.in b/Makefile.pre.in index 04a371ddff72f..d9f821dd14e17 100644 --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -948,6 +948,12 @@ regen-test-frozenmain: $(BUILDPYTHON) # using Programs/freeze_test_frozenmain.py $(RUNSHARED) ./$(BUILDPYTHON) $(srcdir)/Programs/freeze_test_frozenmain.py Programs/test_frozenmain.h +.PHONY: regen-re +regen-re: $(BUILDPYTHON) + # Regenerate Lib/re/_casefix.py + # using Tools/scripts/generate_re_casefix.py + $(RUNSHARED) ./$(BUILDPYTHON) $(srcdir)/Tools/scripts/generate_re_casefix.py $(srcdir)/Lib/re/_casefix.py + Programs/_testembed: Programs/_testembed.o $(LINK_PYTHON_DEPS) $(LINKCC) $(PY_CORE_LDFLAGS) $(LINKFORSHARED) -o $@ Programs/_testembed.o $(LINK_PYTHON_OBJS) $(LIBS) $(MODLIBS) $(SYSLIBS) diff --git a/Misc/NEWS.d/next/Tools-Demos/2022-04-18-12-52-16.gh-issue-91575.fK1TEh.rst b/Misc/NEWS.d/next/Tools-Demos/2022-04-18-12-52-16.gh-issue-91575.fK1TEh.rst new file mode 100644 index 0000000000000..3ed34226e070e --- /dev/null +++ b/Misc/NEWS.d/next/Tools-Demos/2022-04-18-12-52-16.gh-issue-91575.fK1TEh.rst @@ -0,0 +1,3 @@ +Add script ``Tools/scripts/generate_re_casefix.py`` and the make target +``regen-re`` for generating additional data for case-insensitive matching +according to the current Unicode version. diff --git a/Tools/scripts/generate_re_casefix.py b/Tools/scripts/generate_re_casefix.py new file mode 100755 index 0000000000000..00b048b5d716c --- /dev/null +++ b/Tools/scripts/generate_re_casefix.py @@ -0,0 +1,95 @@ +#! /usr/bin/env python3 +# This script generates Lib/re/_casefix.py. + +import collections +import re +import sys +import unicodedata + +def update_file(file, content): + try: + with open(file, 'r', encoding='utf-8') as fobj: + if fobj.read() == content: + return False + except (OSError, ValueError): + pass + with open(file, 'w', encoding='utf-8') as fobj: + fobj.write(content) + return True + +re_casefix_template = """\ +# Auto-generated by Tools/scripts/generate_re_casefix.py. + +# Maps the code of lowercased character to codes of different lowercased +# characters which have the same uppercase. +_EXTRA_CASES = { +%s +} +""" + +def uname(i): + return unicodedata.name(chr(i), r'U+%04X' % i) + +class hexint(int): + def __repr__(self): + return '%#06x' % self + +def alpha(i): + c = chr(i) + return c if c.isalpha() else ascii(c)[1:-1] + + +def main(outfile='Lib/re/_casefix.py'): + # Find sets of characters which have the same uppercase. + equivalent_chars = collections.defaultdict(str) + for c in map(chr, range(sys.maxunicode + 1)): + equivalent_chars[c.upper()] += c + equivalent_chars = [t for t in equivalent_chars.values() if len(t) > 1] + + # List of codes of lowercased characters which have the same uppercase. + equivalent_lower_codes = [sorted(t) + for s in equivalent_chars + for t in [set(ord(c.lower()) for c in s)] + if len(t) > 1] + + bad_codes = [] + for t in equivalent_lower_codes: + for i in t: + if i > 0xffff: + bad_codes.extend(t) + try: + bad_codes.append(ord(chr(i).upper())) + except (ValueError, TypeError): + pass + break + if bad_codes: + print('Case-insensitive matching may not work correctly for character:', + file=sys.stderr) + for i in sorted(bad_codes): + print(" '%s' (U+%04x, %s)" % (alpha(i), i, uname(i)), + file=sys.stderr) + sys.exit(1) + + mapping = {i: tuple(j for j in t if i != j) + for t in equivalent_lower_codes + for i in t} + + items = [] + for i, t in sorted(mapping.items()): + items.append(' # %s: %s' % ( + uname(i), + ', '.join(map(uname, t)), + )) + items.append(" %r: %r, # '%s': '%s'" % ( + hexint(i), + tuple(map(hexint, t)), + alpha(i), + ''.join(map(alpha, t)), + )) + + update_file(outfile, re_casefix_template % '\n'.join(items)) + + +if __name__ == '__main__': + import sys + main(*sys.argv[1:]) From webhook-mailer at python.org Fri Apr 22 14:44:31 2022 From: webhook-mailer at python.org (serhiy-storchaka) Date: Fri, 22 Apr 2022 18:44:31 -0000 Subject: [Python-checkins] [3.10] gh-91575: Update case-insensitive matching in re to the latest Unicode version (GH-91580). (GH-91661) Message-ID: https://github.com/python/cpython/commit/1748816e80b23744667e239b49b477c0e283d201 commit: 1748816e80b23744667e239b49b477c0e283d201 branch: 3.10 author: Serhiy Storchaka committer: serhiy-storchaka date: 2022-04-22T21:44:05+03:00 summary: [3.10] gh-91575: Update case-insensitive matching in re to the latest Unicode version (GH-91580). (GH-91661) (cherry picked from commit 1c2fcebf3c5e2ab41d376bb481834445617c8f3c) files: A Misc/NEWS.d/next/Library/2022-04-15-18-38-21.gh-issue-91575.fSyAxS.rst M Lib/sre_compile.py M Lib/test/test_re.py diff --git a/Lib/sre_compile.py b/Lib/sre_compile.py index c6398bfb83a57..aed752d11d2e5 100644 --- a/Lib/sre_compile.py +++ b/Lib/sre_compile.py @@ -52,6 +52,22 @@ (0x3c2, 0x3c3), # ?? # GREEK SMALL LETTER PHI, GREEK PHI SYMBOL (0x3c6, 0x3d5), # ?? + # CYRILLIC SMALL LETTER VE, CYRILLIC SMALL LETTER ROUNDED VE + (0x432, 0x1c80), # ?? + # CYRILLIC SMALL LETTER DE, CYRILLIC SMALL LETTER LONG-LEGGED DE + (0x434, 0x1c81), # ?? + # CYRILLIC SMALL LETTER O, CYRILLIC SMALL LETTER NARROW O + (0x43e, 0x1c82), # ?? + # CYRILLIC SMALL LETTER ES, CYRILLIC SMALL LETTER WIDE ES + (0x441, 0x1c83), # ?? + # CYRILLIC SMALL LETTER TE, CYRILLIC SMALL LETTER TALL TE, CYRILLIC SMALL LETTER THREE-LEGGED TE + (0x442, 0x1c84, 0x1c85), # ??? + # CYRILLIC SMALL LETTER HARD SIGN, CYRILLIC SMALL LETTER TALL HARD SIGN + (0x44a, 0x1c86), # ?? + # CYRILLIC SMALL LETTER YAT, CYRILLIC SMALL LETTER TALL YAT + (0x463, 0x1c87), # ?? + # CYRILLIC SMALL LETTER UNBLENDED UK, CYRILLIC SMALL LETTER MONOGRAPH UK + (0x1c88, 0xa64b), # ?? # LATIN SMALL LETTER S WITH DOT ABOVE, LATIN SMALL LETTER LONG S WITH DOT ABOVE (0x1e61, 0x1e9b), # ?? # LATIN SMALL LIGATURE LONG S T, LATIN SMALL LIGATURE ST @@ -320,11 +336,19 @@ def _optimize_charset(charset, iscased=None, fixup=None, fixes=None): charmap += b'\0' * 0xff00 continue # Character set contains non-BMP character codes. + # For range, all BMP characters in the range are already + # proceeded. if fixup: hascased = True - # There are only two ranges of cased non-BMP characters: - # 10400-1044F (Deseret) and 118A0-118DF (Warang Citi), - # and for both ranges RANGE_UNI_IGNORE works. + # For now, IN_UNI_IGNORE+LITERAL and + # IN_UNI_IGNORE+RANGE_UNI_IGNORE work for all non-BMP + # characters, because two characters (at least one of + # which is not in the BMP) match case-insensitively + # if and only if: + # 1) c1.lower() == c2.lower() + # 2) c1.lower() == c2 or c1.lower().upper() == c2 + # Also, both c.lower() and c.lower().upper() are single + # characters for every non-BMP character. if op is RANGE: op = RANGE_UNI_IGNORE tail.append((op, av)) diff --git a/Lib/test/test_re.py b/Lib/test/test_re.py index 887ab781fc9d4..3d6623b596ef9 100644 --- a/Lib/test/test_re.py +++ b/Lib/test/test_re.py @@ -873,16 +873,30 @@ def test_ignore_case(self): self.assertEqual(re.match(r"((a)\s(abc|a))", "a a", re.I).group(1), "a a") self.assertEqual(re.match(r"((a)\s(abc|a)*)", "a aa", re.I).group(1), "a aa") - assert '\u212a'.lower() == 'k' # '?' + # Two different characters have the same lowercase. + assert 'K'.lower() == '\u212a'.lower() == 'k' # '?' self.assertTrue(re.match(r'K', '\u212a', re.I)) self.assertTrue(re.match(r'k', '\u212a', re.I)) self.assertTrue(re.match(r'\u212a', 'K', re.I)) self.assertTrue(re.match(r'\u212a', 'k', re.I)) - assert '\u017f'.upper() == 'S' # '?' + + # Two different characters have the same uppercase. + assert 's'.upper() == '\u017f'.upper() == 'S' # '?' self.assertTrue(re.match(r'S', '\u017f', re.I)) self.assertTrue(re.match(r's', '\u017f', re.I)) self.assertTrue(re.match(r'\u017f', 'S', re.I)) self.assertTrue(re.match(r'\u017f', 's', re.I)) + + # Two different characters have the same uppercase. Unicode 9.0+. + assert '\u0432'.upper() == '\u1c80'.upper() == '\u0412' # '?', '?', '?' + self.assertTrue(re.match(r'\u0412', '\u0432', re.I)) + self.assertTrue(re.match(r'\u0412', '\u1c80', re.I)) + self.assertTrue(re.match(r'\u0432', '\u0412', re.I)) + self.assertTrue(re.match(r'\u0432', '\u1c80', re.I)) + self.assertTrue(re.match(r'\u1c80', '\u0412', re.I)) + self.assertTrue(re.match(r'\u1c80', '\u0432', re.I)) + + # Two different characters have the same multicharacter uppercase. assert '\ufb05'.upper() == '\ufb06'.upper() == 'ST' # '?', '?' self.assertTrue(re.match(r'\ufb05', '\ufb06', re.I)) self.assertTrue(re.match(r'\ufb06', '\ufb05', re.I)) @@ -896,16 +910,31 @@ def test_ignore_case_set(self): self.assertTrue(re.match(br'[19a]', b'a', re.I)) self.assertTrue(re.match(br'[19a]', b'A', re.I)) self.assertTrue(re.match(br'[19A]', b'a', re.I)) - assert '\u212a'.lower() == 'k' # '?' + + # Two different characters have the same lowercase. + assert 'K'.lower() == '\u212a'.lower() == 'k' # '?' self.assertTrue(re.match(r'[19K]', '\u212a', re.I)) self.assertTrue(re.match(r'[19k]', '\u212a', re.I)) self.assertTrue(re.match(r'[19\u212a]', 'K', re.I)) self.assertTrue(re.match(r'[19\u212a]', 'k', re.I)) - assert '\u017f'.upper() == 'S' # '?' + + # Two different characters have the same uppercase. + assert 's'.upper() == '\u017f'.upper() == 'S' # '?' self.assertTrue(re.match(r'[19S]', '\u017f', re.I)) self.assertTrue(re.match(r'[19s]', '\u017f', re.I)) self.assertTrue(re.match(r'[19\u017f]', 'S', re.I)) self.assertTrue(re.match(r'[19\u017f]', 's', re.I)) + + # Two different characters have the same uppercase. Unicode 9.0+. + assert '\u0432'.upper() == '\u1c80'.upper() == '\u0412' # '?', '?', '?' + self.assertTrue(re.match(r'[19\u0412]', '\u0432', re.I)) + self.assertTrue(re.match(r'[19\u0412]', '\u1c80', re.I)) + self.assertTrue(re.match(r'[19\u0432]', '\u0412', re.I)) + self.assertTrue(re.match(r'[19\u0432]', '\u1c80', re.I)) + self.assertTrue(re.match(r'[19\u1c80]', '\u0412', re.I)) + self.assertTrue(re.match(r'[19\u1c80]', '\u0432', re.I)) + + # Two different characters have the same multicharacter uppercase. assert '\ufb05'.upper() == '\ufb06'.upper() == 'ST' # '?', '?' self.assertTrue(re.match(r'[19\ufb05]', '\ufb06', re.I)) self.assertTrue(re.match(r'[19\ufb06]', '\ufb05', re.I)) @@ -929,16 +958,30 @@ def test_ignore_case_range(self): self.assertTrue(re.match(r'[\U00010400-\U00010427]', '\U00010428', re.I)) self.assertTrue(re.match(r'[\U00010400-\U00010427]', '\U00010400', re.I)) - assert '\u212a'.lower() == 'k' # '?' + # Two different characters have the same lowercase. + assert 'K'.lower() == '\u212a'.lower() == 'k' # '?' self.assertTrue(re.match(r'[J-M]', '\u212a', re.I)) self.assertTrue(re.match(r'[j-m]', '\u212a', re.I)) self.assertTrue(re.match(r'[\u2129-\u212b]', 'K', re.I)) self.assertTrue(re.match(r'[\u2129-\u212b]', 'k', re.I)) - assert '\u017f'.upper() == 'S' # '?' + + # Two different characters have the same uppercase. + assert 's'.upper() == '\u017f'.upper() == 'S' # '?' self.assertTrue(re.match(r'[R-T]', '\u017f', re.I)) self.assertTrue(re.match(r'[r-t]', '\u017f', re.I)) self.assertTrue(re.match(r'[\u017e-\u0180]', 'S', re.I)) self.assertTrue(re.match(r'[\u017e-\u0180]', 's', re.I)) + + # Two different characters have the same uppercase. Unicode 9.0+. + assert '\u0432'.upper() == '\u1c80'.upper() == '\u0412' # '?', '?', '?' + self.assertTrue(re.match(r'[\u0411-\u0413]', '\u0432', re.I)) + self.assertTrue(re.match(r'[\u0411-\u0413]', '\u1c80', re.I)) + self.assertTrue(re.match(r'[\u0431-\u0433]', '\u0412', re.I)) + self.assertTrue(re.match(r'[\u0431-\u0433]', '\u1c80', re.I)) + self.assertTrue(re.match(r'[\u1c80-\u1c82]', '\u0412', re.I)) + self.assertTrue(re.match(r'[\u1c80-\u1c82]', '\u0432', re.I)) + + # Two different characters have the same multicharacter uppercase. assert '\ufb05'.upper() == '\ufb06'.upper() == 'ST' # '?', '?' self.assertTrue(re.match(r'[\ufb04-\ufb05]', '\ufb06', re.I)) self.assertTrue(re.match(r'[\ufb06-\ufb07]', '\ufb05', re.I)) diff --git a/Misc/NEWS.d/next/Library/2022-04-15-18-38-21.gh-issue-91575.fSyAxS.rst b/Misc/NEWS.d/next/Library/2022-04-15-18-38-21.gh-issue-91575.fSyAxS.rst new file mode 100644 index 0000000000000..ba046f2b4d61c --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-15-18-38-21.gh-issue-91575.fSyAxS.rst @@ -0,0 +1,2 @@ +Update case-insensitive matching in the :mod:`re` module to the latest +Unicode version. From webhook-mailer at python.org Fri Apr 22 15:02:30 2022 From: webhook-mailer at python.org (serhiy-storchaka) Date: Fri, 22 Apr 2022 19:02:30 -0000 Subject: [Python-checkins] [3.9] gh-91700: Validate the group number in conditional expression in RE (GH-91702) (GH-91831) (GH-91836) Message-ID: https://github.com/python/cpython/commit/76ff68666fa946c44d16173ff52a127d3a9785db commit: 76ff68666fa946c44d16173ff52a127d3a9785db branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: serhiy-storchaka date: 2022-04-22T22:02:20+03:00 summary: [3.9] gh-91700: Validate the group number in conditional expression in RE (GH-91702) (GH-91831) (GH-91836) In expression (?(group)...) an appropriate re.error is now raised if the group number refers to not defined group. Previously it raised RuntimeError: invalid SRE code. (cherry picked from commit 48ec61a89a959071206549819448405c2cea61b0) (cherry picked from commit 080781cd49b13da4a73db87b6f5e0c7aeec83e92) Co-authored-by: Serhiy Storchaka files: A Misc/NEWS.d/next/Library/2022-04-19-17-30-17.gh-issue-91700.MRJi6m.rst M Lib/sre_parse.py M Lib/test/test_re.py diff --git a/Lib/sre_parse.py b/Lib/sre_parse.py index d3ff196032b30..20a602501191f 100644 --- a/Lib/sre_parse.py +++ b/Lib/sre_parse.py @@ -78,6 +78,7 @@ def __init__(self): self.groupdict = {} self.groupwidths = [None] # group 0 self.lookbehindgroups = None + self.grouprefpos = {} @property def groups(self): return len(self.groupwidths) @@ -786,6 +787,10 @@ def _parse(source, state, verbose, nested, first=False): if condgroup >= MAXGROUPS: msg = "invalid group reference %d" % condgroup raise source.error(msg, len(condname) + 1) + if condgroup not in state.grouprefpos: + state.grouprefpos[condgroup] = ( + source.tell() - len(condname) - 1 + ) state.checklookbehindgroup(condgroup, source) item_yes = _parse(source, state, verbose, nested + 1) if source.match("|"): @@ -963,6 +968,11 @@ def parse(str, flags=0, state=None): assert source.next == ")" raise source.error("unbalanced parenthesis") + for g in p.state.grouprefpos: + if g >= p.state.groups: + msg = "invalid group reference %d" % g + raise error(msg, str, p.state.grouprefpos[g]) + if flags & SRE_FLAG_DEBUG: p.dump() diff --git a/Lib/test/test_re.py b/Lib/test/test_re.py index 007064093c4d1..8eeddd638f1fe 100644 --- a/Lib/test/test_re.py +++ b/Lib/test/test_re.py @@ -574,6 +574,8 @@ def test_re_groupref_exists_errors(self): self.checkPatternError(r'()(?(1)a|b|c)', 'conditional backref with more than ' 'two branches', 10) + self.checkPatternError(r'()(?(2)a)', + "invalid group reference 2", 5) def test_re_groupref_overflow(self): from sre_constants import MAXGROUPS diff --git a/Misc/NEWS.d/next/Library/2022-04-19-17-30-17.gh-issue-91700.MRJi6m.rst b/Misc/NEWS.d/next/Library/2022-04-19-17-30-17.gh-issue-91700.MRJi6m.rst new file mode 100644 index 0000000000000..73b106869697b --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-19-17-30-17.gh-issue-91700.MRJi6m.rst @@ -0,0 +1,4 @@ +Compilation of regular expression containing a conditional expression +``(?(group)...)`` now raises an appropriate :exc:`re.error` if the group +number refers to not defined group. Previously an internal RuntimeError was +raised. From webhook-mailer at python.org Fri Apr 22 15:03:01 2022 From: webhook-mailer at python.org (serhiy-storchaka) Date: Fri, 22 Apr 2022 19:03:01 -0000 Subject: [Python-checkins] [3.9] gh-91575: Update case-insensitive matching in re to the latest Unicode version (GH-91580). (GH-91661) (GH-91837) Message-ID: https://github.com/python/cpython/commit/537dabc8694e13b5159c1d15d70f2e3a8c69da26 commit: 537dabc8694e13b5159c1d15d70f2e3a8c69da26 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: serhiy-storchaka date: 2022-04-22T22:02:56+03:00 summary: [3.9] gh-91575: Update case-insensitive matching in re to the latest Unicode version (GH-91580). (GH-91661) (GH-91837) (cherry picked from commit 1c2fcebf3c5e2ab41d376bb481834445617c8f3c) (cherry picked from commit 1748816e80b23744667e239b49b477c0e283d201) Co-authored-by: Serhiy Storchaka files: A Misc/NEWS.d/next/Library/2022-04-15-18-38-21.gh-issue-91575.fSyAxS.rst M Lib/sre_compile.py M Lib/test/test_re.py diff --git a/Lib/sre_compile.py b/Lib/sre_compile.py index c6398bfb83a57..aed752d11d2e5 100644 --- a/Lib/sre_compile.py +++ b/Lib/sre_compile.py @@ -52,6 +52,22 @@ (0x3c2, 0x3c3), # ?? # GREEK SMALL LETTER PHI, GREEK PHI SYMBOL (0x3c6, 0x3d5), # ?? + # CYRILLIC SMALL LETTER VE, CYRILLIC SMALL LETTER ROUNDED VE + (0x432, 0x1c80), # ?? + # CYRILLIC SMALL LETTER DE, CYRILLIC SMALL LETTER LONG-LEGGED DE + (0x434, 0x1c81), # ?? + # CYRILLIC SMALL LETTER O, CYRILLIC SMALL LETTER NARROW O + (0x43e, 0x1c82), # ?? + # CYRILLIC SMALL LETTER ES, CYRILLIC SMALL LETTER WIDE ES + (0x441, 0x1c83), # ?? + # CYRILLIC SMALL LETTER TE, CYRILLIC SMALL LETTER TALL TE, CYRILLIC SMALL LETTER THREE-LEGGED TE + (0x442, 0x1c84, 0x1c85), # ??? + # CYRILLIC SMALL LETTER HARD SIGN, CYRILLIC SMALL LETTER TALL HARD SIGN + (0x44a, 0x1c86), # ?? + # CYRILLIC SMALL LETTER YAT, CYRILLIC SMALL LETTER TALL YAT + (0x463, 0x1c87), # ?? + # CYRILLIC SMALL LETTER UNBLENDED UK, CYRILLIC SMALL LETTER MONOGRAPH UK + (0x1c88, 0xa64b), # ?? # LATIN SMALL LETTER S WITH DOT ABOVE, LATIN SMALL LETTER LONG S WITH DOT ABOVE (0x1e61, 0x1e9b), # ?? # LATIN SMALL LIGATURE LONG S T, LATIN SMALL LIGATURE ST @@ -320,11 +336,19 @@ def _optimize_charset(charset, iscased=None, fixup=None, fixes=None): charmap += b'\0' * 0xff00 continue # Character set contains non-BMP character codes. + # For range, all BMP characters in the range are already + # proceeded. if fixup: hascased = True - # There are only two ranges of cased non-BMP characters: - # 10400-1044F (Deseret) and 118A0-118DF (Warang Citi), - # and for both ranges RANGE_UNI_IGNORE works. + # For now, IN_UNI_IGNORE+LITERAL and + # IN_UNI_IGNORE+RANGE_UNI_IGNORE work for all non-BMP + # characters, because two characters (at least one of + # which is not in the BMP) match case-insensitively + # if and only if: + # 1) c1.lower() == c2.lower() + # 2) c1.lower() == c2 or c1.lower().upper() == c2 + # Also, both c.lower() and c.lower().upper() are single + # characters for every non-BMP character. if op is RANGE: op = RANGE_UNI_IGNORE tail.append((op, av)) diff --git a/Lib/test/test_re.py b/Lib/test/test_re.py index 8eeddd638f1fe..ca48b9b11f2a8 100644 --- a/Lib/test/test_re.py +++ b/Lib/test/test_re.py @@ -872,16 +872,30 @@ def test_ignore_case(self): self.assertEqual(re.match(r"((a)\s(abc|a))", "a a", re.I).group(1), "a a") self.assertEqual(re.match(r"((a)\s(abc|a)*)", "a aa", re.I).group(1), "a aa") - assert '\u212a'.lower() == 'k' # '?' + # Two different characters have the same lowercase. + assert 'K'.lower() == '\u212a'.lower() == 'k' # '?' self.assertTrue(re.match(r'K', '\u212a', re.I)) self.assertTrue(re.match(r'k', '\u212a', re.I)) self.assertTrue(re.match(r'\u212a', 'K', re.I)) self.assertTrue(re.match(r'\u212a', 'k', re.I)) - assert '\u017f'.upper() == 'S' # '?' + + # Two different characters have the same uppercase. + assert 's'.upper() == '\u017f'.upper() == 'S' # '?' self.assertTrue(re.match(r'S', '\u017f', re.I)) self.assertTrue(re.match(r's', '\u017f', re.I)) self.assertTrue(re.match(r'\u017f', 'S', re.I)) self.assertTrue(re.match(r'\u017f', 's', re.I)) + + # Two different characters have the same uppercase. Unicode 9.0+. + assert '\u0432'.upper() == '\u1c80'.upper() == '\u0412' # '?', '?', '?' + self.assertTrue(re.match(r'\u0412', '\u0432', re.I)) + self.assertTrue(re.match(r'\u0412', '\u1c80', re.I)) + self.assertTrue(re.match(r'\u0432', '\u0412', re.I)) + self.assertTrue(re.match(r'\u0432', '\u1c80', re.I)) + self.assertTrue(re.match(r'\u1c80', '\u0412', re.I)) + self.assertTrue(re.match(r'\u1c80', '\u0432', re.I)) + + # Two different characters have the same multicharacter uppercase. assert '\ufb05'.upper() == '\ufb06'.upper() == 'ST' # '?', '?' self.assertTrue(re.match(r'\ufb05', '\ufb06', re.I)) self.assertTrue(re.match(r'\ufb06', '\ufb05', re.I)) @@ -895,16 +909,31 @@ def test_ignore_case_set(self): self.assertTrue(re.match(br'[19a]', b'a', re.I)) self.assertTrue(re.match(br'[19a]', b'A', re.I)) self.assertTrue(re.match(br'[19A]', b'a', re.I)) - assert '\u212a'.lower() == 'k' # '?' + + # Two different characters have the same lowercase. + assert 'K'.lower() == '\u212a'.lower() == 'k' # '?' self.assertTrue(re.match(r'[19K]', '\u212a', re.I)) self.assertTrue(re.match(r'[19k]', '\u212a', re.I)) self.assertTrue(re.match(r'[19\u212a]', 'K', re.I)) self.assertTrue(re.match(r'[19\u212a]', 'k', re.I)) - assert '\u017f'.upper() == 'S' # '?' + + # Two different characters have the same uppercase. + assert 's'.upper() == '\u017f'.upper() == 'S' # '?' self.assertTrue(re.match(r'[19S]', '\u017f', re.I)) self.assertTrue(re.match(r'[19s]', '\u017f', re.I)) self.assertTrue(re.match(r'[19\u017f]', 'S', re.I)) self.assertTrue(re.match(r'[19\u017f]', 's', re.I)) + + # Two different characters have the same uppercase. Unicode 9.0+. + assert '\u0432'.upper() == '\u1c80'.upper() == '\u0412' # '?', '?', '?' + self.assertTrue(re.match(r'[19\u0412]', '\u0432', re.I)) + self.assertTrue(re.match(r'[19\u0412]', '\u1c80', re.I)) + self.assertTrue(re.match(r'[19\u0432]', '\u0412', re.I)) + self.assertTrue(re.match(r'[19\u0432]', '\u1c80', re.I)) + self.assertTrue(re.match(r'[19\u1c80]', '\u0412', re.I)) + self.assertTrue(re.match(r'[19\u1c80]', '\u0432', re.I)) + + # Two different characters have the same multicharacter uppercase. assert '\ufb05'.upper() == '\ufb06'.upper() == 'ST' # '?', '?' self.assertTrue(re.match(r'[19\ufb05]', '\ufb06', re.I)) self.assertTrue(re.match(r'[19\ufb06]', '\ufb05', re.I)) @@ -928,16 +957,30 @@ def test_ignore_case_range(self): self.assertTrue(re.match(r'[\U00010400-\U00010427]', '\U00010428', re.I)) self.assertTrue(re.match(r'[\U00010400-\U00010427]', '\U00010400', re.I)) - assert '\u212a'.lower() == 'k' # '?' + # Two different characters have the same lowercase. + assert 'K'.lower() == '\u212a'.lower() == 'k' # '?' self.assertTrue(re.match(r'[J-M]', '\u212a', re.I)) self.assertTrue(re.match(r'[j-m]', '\u212a', re.I)) self.assertTrue(re.match(r'[\u2129-\u212b]', 'K', re.I)) self.assertTrue(re.match(r'[\u2129-\u212b]', 'k', re.I)) - assert '\u017f'.upper() == 'S' # '?' + + # Two different characters have the same uppercase. + assert 's'.upper() == '\u017f'.upper() == 'S' # '?' self.assertTrue(re.match(r'[R-T]', '\u017f', re.I)) self.assertTrue(re.match(r'[r-t]', '\u017f', re.I)) self.assertTrue(re.match(r'[\u017e-\u0180]', 'S', re.I)) self.assertTrue(re.match(r'[\u017e-\u0180]', 's', re.I)) + + # Two different characters have the same uppercase. Unicode 9.0+. + assert '\u0432'.upper() == '\u1c80'.upper() == '\u0412' # '?', '?', '?' + self.assertTrue(re.match(r'[\u0411-\u0413]', '\u0432', re.I)) + self.assertTrue(re.match(r'[\u0411-\u0413]', '\u1c80', re.I)) + self.assertTrue(re.match(r'[\u0431-\u0433]', '\u0412', re.I)) + self.assertTrue(re.match(r'[\u0431-\u0433]', '\u1c80', re.I)) + self.assertTrue(re.match(r'[\u1c80-\u1c82]', '\u0412', re.I)) + self.assertTrue(re.match(r'[\u1c80-\u1c82]', '\u0432', re.I)) + + # Two different characters have the same multicharacter uppercase. assert '\ufb05'.upper() == '\ufb06'.upper() == 'ST' # '?', '?' self.assertTrue(re.match(r'[\ufb04-\ufb05]', '\ufb06', re.I)) self.assertTrue(re.match(r'[\ufb06-\ufb07]', '\ufb05', re.I)) diff --git a/Misc/NEWS.d/next/Library/2022-04-15-18-38-21.gh-issue-91575.fSyAxS.rst b/Misc/NEWS.d/next/Library/2022-04-15-18-38-21.gh-issue-91575.fSyAxS.rst new file mode 100644 index 0000000000000..ba046f2b4d61c --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-15-18-38-21.gh-issue-91575.fSyAxS.rst @@ -0,0 +1,2 @@ +Update case-insensitive matching in the :mod:`re` module to the latest +Unicode version. From webhook-mailer at python.org Fri Apr 22 18:47:18 2022 From: webhook-mailer at python.org (brettcannon) Date: Fri, 22 Apr 2022 22:47:18 -0000 Subject: [Python-checkins] bpo-46720: Add support for path-like objects to multiprocessing.set_executable for Windows (GH-31279) Message-ID: https://github.com/python/cpython/commit/5576ddbbbc9c1d7a7819abc961e5d604ae0f7dd7 commit: 5576ddbbbc9c1d7a7819abc961e5d604ae0f7dd7 branch: main author: G?ry Ogam committer: brettcannon date: 2022-04-22T15:47:09-07:00 summary: bpo-46720: Add support for path-like objects to multiprocessing.set_executable for Windows (GH-31279) This bring the API to be on a par with Unix-like systems. files: A Misc/NEWS.d/next/Library/2022-02-11-23-11-35.bpo-46720.nY8spB.rst M Doc/library/multiprocessing.rst M Lib/multiprocessing/spawn.py M Lib/multiprocessing/util.py M Lib/test/_test_multiprocessing.py diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst index 83aa5cb87f49f..70802ee1fdecb 100644 --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -1075,6 +1075,9 @@ Miscellaneous .. versionchanged:: 3.4 Now supported on Unix when the ``'spawn'`` start method is used. + .. versionchanged:: 3.11 + Accepts a :term:`path-like object`. + .. function:: set_start_method(method) Set the method which should be used to start child processes. diff --git a/Lib/multiprocessing/spawn.py b/Lib/multiprocessing/spawn.py index 7cc129e261076..09f8a229d7ccc 100644 --- a/Lib/multiprocessing/spawn.py +++ b/Lib/multiprocessing/spawn.py @@ -33,18 +33,21 @@ WINEXE = getattr(sys, 'frozen', False) WINSERVICE = sys.executable.lower().endswith("pythonservice.exe") -if WINSERVICE: - _python_exe = os.path.join(sys.exec_prefix, 'python.exe') -else: - _python_exe = sys.executable - def set_executable(exe): global _python_exe - _python_exe = exe + if sys.platform == 'win32': + _python_exe = os.fsdecode(exe) + else: + _python_exe = os.fsencode(exe) def get_executable(): return _python_exe +if WINSERVICE: + set_executable(os.path.join(sys.exec_prefix, 'python.exe')) +else: + set_executable(sys.executable) + # # # @@ -86,7 +89,8 @@ def get_command_line(**kwds): prog = 'from multiprocessing.spawn import spawn_main; spawn_main(%s)' prog %= ', '.join('%s=%r' % item for item in kwds.items()) opts = util._args_from_interpreter_flags() - return [_python_exe] + opts + ['-c', prog, '--multiprocessing-fork'] + exe = get_executable() + return [exe] + opts + ['-c', prog, '--multiprocessing-fork'] def spawn_main(pipe_handle, parent_pid=None, tracker_fd=None): diff --git a/Lib/multiprocessing/util.py b/Lib/multiprocessing/util.py index a4683339820f5..abbc4c5e6088b 100644 --- a/Lib/multiprocessing/util.py +++ b/Lib/multiprocessing/util.py @@ -450,7 +450,7 @@ def spawnv_passfds(path, args, passfds): errpipe_read, errpipe_write = os.pipe() try: return _posixsubprocess.fork_exec( - args, [os.fsencode(path)], True, passfds, None, None, + args, [path], True, passfds, None, None, -1, -1, -1, -1, -1, -1, errpipe_read, errpipe_write, False, False, None, None, None, -1, None) finally: diff --git a/Lib/test/_test_multiprocessing.py b/Lib/test/_test_multiprocessing.py index 427fc0c47a3ca..67bb17c0ede36 100644 --- a/Lib/test/_test_multiprocessing.py +++ b/Lib/test/_test_multiprocessing.py @@ -20,6 +20,7 @@ import subprocess import struct import operator +import pathlib import pickle import weakref import warnings @@ -256,6 +257,21 @@ def test_current(self): self.assertEqual(current.ident, os.getpid()) self.assertEqual(current.exitcode, None) + def test_set_executable(self): + if self.TYPE == 'threads': + self.skipTest(f'test not appropriate for {self.TYPE}') + paths = [ + sys.executable, # str + sys.executable.encode(), # bytes + pathlib.Path(sys.executable) # os.PathLike + ] + for path in paths: + self.set_executable(path) + p = self.Process() + p.start() + p.join() + self.assertEqual(p.exitcode, 0) + def test_args_argument(self): # bpo-45735: Using list or tuple as *args* in constructor could # achieve the same effect. @@ -5787,6 +5803,7 @@ class ProcessesMixin(BaseMixin): current_process = staticmethod(multiprocessing.current_process) parent_process = staticmethod(multiprocessing.parent_process) active_children = staticmethod(multiprocessing.active_children) + set_executable = staticmethod(multiprocessing.set_executable) Pool = staticmethod(multiprocessing.Pool) Pipe = staticmethod(multiprocessing.Pipe) Queue = staticmethod(multiprocessing.Queue) diff --git a/Misc/NEWS.d/next/Library/2022-02-11-23-11-35.bpo-46720.nY8spB.rst b/Misc/NEWS.d/next/Library/2022-02-11-23-11-35.bpo-46720.nY8spB.rst new file mode 100644 index 0000000000000..70d5e5ef34333 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-02-11-23-11-35.bpo-46720.nY8spB.rst @@ -0,0 +1,2 @@ +Add support for path-like objects to :func:`multiprocessing.set_executable` for +Windows to be on a par with Unix-like systems. Patch by G?ry Ogam. From webhook-mailer at python.org Fri Apr 22 18:48:07 2022 From: webhook-mailer at python.org (brettcannon) Date: Fri, 22 Apr 2022 22:48:07 -0000 Subject: [Python-checkins] gh-91217: deprecate-sndhdr (#91806) Message-ID: https://github.com/python/cpython/commit/e7929cba169349776e78ff86143b24d0122b2cdd commit: e7929cba169349776e78ff86143b24d0122b2cdd branch: main author: Brett Cannon committer: brettcannon date: 2022-04-22T15:48:03-07:00 summary: gh-91217: deprecate-sndhdr (#91806) Also inline necessary functionality from `sndhdr` into `email.mime.audio` for `MIMEAudio`. Co-authored-by: Hugo van Kemenade files: A Misc/NEWS.d/next/Library/2022-04-17-12-27-46.gh-issue-91217.tNDWtK.rst M Doc/library/email.mime.rst M Doc/whatsnew/3.11.rst M Lib/email/mime/audio.py M Lib/sndhdr.py M Lib/test/test_sndhdr.py diff --git a/Doc/library/email.mime.rst b/Doc/library/email.mime.rst index ab4f7bc54e025..3fe5fe88a0946 100644 --- a/Doc/library/email.mime.rst +++ b/Doc/library/email.mime.rst @@ -146,7 +146,7 @@ Here are the classes: A subclass of :class:`~email.mime.nonmultipart.MIMENonMultipart`, the :class:`MIMEAudio` class is used to create MIME message objects of major type :mimetype:`audio`. *_audiodata* is a string containing the raw audio data. If - this data can be decoded by the standard Python module :mod:`sndhdr`, then the + this data can be decoded as au, wav, aiff, or aifc, then the subtype will be automatically included in the :mailheader:`Content-Type` header. Otherwise you can explicitly specify the audio subtype via the *_subtype* argument. If the minor type could not be guessed and *_subtype* was not given, diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index 75ea70b7a1832..653d32aaf540b 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -920,6 +920,7 @@ Deprecated * :mod:`nntplib` * :mod:`ossaudiodev` * :mod:`pipes` + * :mod:`sndhdr` (Contributed by Brett Cannon in :issue:`47061`.) diff --git a/Lib/email/mime/audio.py b/Lib/email/mime/audio.py index 4bcd7b224a862..e859c2e8a2b6c 100644 --- a/Lib/email/mime/audio.py +++ b/Lib/email/mime/audio.py @@ -6,19 +6,43 @@ __all__ = ['MIMEAudio'] -import sndhdr - from io import BytesIO from email import encoders from email.mime.nonmultipart import MIMENonMultipart - -_sndhdr_MIMEmap = {'au' : 'basic', - 'wav' :'x-wav', - 'aiff':'x-aiff', - 'aifc':'x-aiff', - } +_tests = [] + +def _test_aifc_aiff(h, f): + if not h.startswith(b'FORM'): + return None + if h[8:12] in {b'AIFC', b'AIFF'}: + return 'x-aiff' + else: + return None + +_tests.append(_test_aifc_aiff) + + +def _test_au(h, f): + if h.startswith(b'.snd'): + return 'basic' + else: + return None + +_tests.append(_test_au) + + +def _test_wav(h, f): + import wave + # 'RIFF' 'WAVE' 'fmt ' + if not h.startswith(b'RIFF') or h[8:12] != b'WAVE' or h[12:16] != b'fmt ': + return None + else: + return "x-wav" + +_tests.append(_test_wav) + # There are others in sndhdr that don't have MIME types. :( # Additional ones to be added to sndhdr? midi, mp3, realaudio, wma?? @@ -31,14 +55,14 @@ def _whatsnd(data): """ hdr = data[:512] fakefile = BytesIO(hdr) - for testfn in sndhdr.tests: + for testfn in _tests: res = testfn(hdr, fakefile) if res is not None: - return _sndhdr_MIMEmap.get(res[0]) - return None + return res + else: + return None - class MIMEAudio(MIMENonMultipart): """Class for generating audio/* MIME documents.""" @@ -47,7 +71,7 @@ def __init__(self, _audiodata, _subtype=None, """Create an audio/* type MIME document. _audiodata is a string containing the raw audio data. If this data - can be decoded by the standard Python `sndhdr' module, then the + can be decoded as au, wav, aiff, or aifc, then the subtype will be automatically included in the Content-Type header. Otherwise, you can specify the specific audio subtype via the _subtype parameter. If _subtype is not given, and no subtype can be diff --git a/Lib/sndhdr.py b/Lib/sndhdr.py index a63b6fd20220c..98a783448239a 100644 --- a/Lib/sndhdr.py +++ b/Lib/sndhdr.py @@ -27,13 +27,16 @@ explicitly given directories. """ +import warnings + +warnings._deprecated(__name__, remove=(3, 13)) + # The file structure is top-down except that the test program and its # subroutine come last. __all__ = ['what', 'whathdr'] from collections import namedtuple -import warnings SndHeaders = namedtuple('SndHeaders', 'filetype framerate nchannels nframes sampwidth') diff --git a/Lib/test/test_sndhdr.py b/Lib/test/test_sndhdr.py index 426417c038208..4d97437f9072c 100644 --- a/Lib/test/test_sndhdr.py +++ b/Lib/test/test_sndhdr.py @@ -1,7 +1,10 @@ -import sndhdr import pickle import unittest from test.support import findfile +from test.support import warnings_helper + +sndhdr = warnings_helper.import_deprecated("sndhdr") + class TestFormats(unittest.TestCase): def test_data(self): diff --git a/Misc/NEWS.d/next/Library/2022-04-17-12-27-46.gh-issue-91217.tNDWtK.rst b/Misc/NEWS.d/next/Library/2022-04-17-12-27-46.gh-issue-91217.tNDWtK.rst new file mode 100644 index 0000000000000..4ee6ba284682f --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-17-12-27-46.gh-issue-91217.tNDWtK.rst @@ -0,0 +1,2 @@ +Deprecate the sndhdr module, as well as inline needed functionality for +``email.mime.MIMEAudio``. From webhook-mailer at python.org Fri Apr 22 19:02:00 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Fri, 22 Apr 2022 23:02:00 -0000 Subject: [Python-checkins] gh-91547: Remove "Undocumented modules" page (#91682) Message-ID: https://github.com/python/cpython/commit/254aaa7981d8773658fb14795da5dec888c95f93 commit: 254aaa7981d8773658fb14795da5dec888c95f93 branch: main author: Shantanu <12621235+hauntsaninja at users.noreply.github.com> committer: JelleZijlstra date: 2022-04-22T16:01:52-07:00 summary: gh-91547: Remove "Undocumented modules" page (#91682) files: A Misc/NEWS.d/next/Documentation/2022-04-19-20-16-00.gh-issue-91547.LsNWER.rst D Doc/library/undoc.rst M Doc/library/index.rst diff --git a/Doc/library/index.rst b/Doc/library/index.rst index db8f0d9bdfb8f..7d2002b37df12 100644 --- a/Doc/library/index.rst +++ b/Doc/library/index.rst @@ -74,5 +74,4 @@ the `Python Package Index `_. windows.rst unix.rst superseded.rst - undoc.rst security_warnings.rst diff --git a/Doc/library/undoc.rst b/Doc/library/undoc.rst deleted file mode 100644 index 2444080d6b9d9..0000000000000 --- a/Doc/library/undoc.rst +++ /dev/null @@ -1,26 +0,0 @@ -.. _undoc: - -******************** -Undocumented Modules -******************** - -Here's a quick listing of modules that are currently undocumented, but that -should be documented. Feel free to contribute documentation for them! (Send -via email to docs at python.org.) - -The idea and original contents for this chapter were taken from a posting by -Fredrik Lundh; the specific contents of this chapter have been substantially -revised. - - -Platform specific modules -========================= - -These modules are used to implement the :mod:`os.path` module, and are not -documented beyond this mention. There's little need to document these. - -:mod:`ntpath` - --- Implementation of :mod:`os.path` on Win32 and Win64 platforms. - -:mod:`posixpath` - --- Implementation of :mod:`os.path` on POSIX. diff --git a/Misc/NEWS.d/next/Documentation/2022-04-19-20-16-00.gh-issue-91547.LsNWER.rst b/Misc/NEWS.d/next/Documentation/2022-04-19-20-16-00.gh-issue-91547.LsNWER.rst new file mode 100644 index 0000000000000..95b34cb2fac1f --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2022-04-19-20-16-00.gh-issue-91547.LsNWER.rst @@ -0,0 +1 @@ +Remove "Undocumented modules" page. From webhook-mailer at python.org Fri Apr 22 19:33:49 2022 From: webhook-mailer at python.org (miss-islington) Date: Fri, 22 Apr 2022 23:33:49 -0000 Subject: [Python-checkins] gh-91547: Remove "Undocumented modules" page (GH-91682) Message-ID: https://github.com/python/cpython/commit/42a158b627732a897437fd5153cf23f7da18a10b commit: 42a158b627732a897437fd5153cf23f7da18a10b branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-22T16:33:38-07:00 summary: gh-91547: Remove "Undocumented modules" page (GH-91682) (cherry picked from commit 254aaa7981d8773658fb14795da5dec888c95f93) Co-authored-by: Shantanu <12621235+hauntsaninja at users.noreply.github.com> files: A Misc/NEWS.d/next/Documentation/2022-04-19-20-16-00.gh-issue-91547.LsNWER.rst D Doc/library/undoc.rst M Doc/library/index.rst diff --git a/Doc/library/index.rst b/Doc/library/index.rst index 0fd6e4c0d0b01..5007bacbab1e6 100644 --- a/Doc/library/index.rst +++ b/Doc/library/index.rst @@ -75,5 +75,4 @@ the `Python Package Index `_. windows.rst unix.rst superseded.rst - undoc.rst security_warnings.rst diff --git a/Doc/library/undoc.rst b/Doc/library/undoc.rst deleted file mode 100644 index 2444080d6b9d9..0000000000000 --- a/Doc/library/undoc.rst +++ /dev/null @@ -1,26 +0,0 @@ -.. _undoc: - -******************** -Undocumented Modules -******************** - -Here's a quick listing of modules that are currently undocumented, but that -should be documented. Feel free to contribute documentation for them! (Send -via email to docs at python.org.) - -The idea and original contents for this chapter were taken from a posting by -Fredrik Lundh; the specific contents of this chapter have been substantially -revised. - - -Platform specific modules -========================= - -These modules are used to implement the :mod:`os.path` module, and are not -documented beyond this mention. There's little need to document these. - -:mod:`ntpath` - --- Implementation of :mod:`os.path` on Win32 and Win64 platforms. - -:mod:`posixpath` - --- Implementation of :mod:`os.path` on POSIX. diff --git a/Misc/NEWS.d/next/Documentation/2022-04-19-20-16-00.gh-issue-91547.LsNWER.rst b/Misc/NEWS.d/next/Documentation/2022-04-19-20-16-00.gh-issue-91547.LsNWER.rst new file mode 100644 index 0000000000000..95b34cb2fac1f --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2022-04-19-20-16-00.gh-issue-91547.LsNWER.rst @@ -0,0 +1 @@ +Remove "Undocumented modules" page. From webhook-mailer at python.org Fri Apr 22 19:33:50 2022 From: webhook-mailer at python.org (miss-islington) Date: Fri, 22 Apr 2022 23:33:50 -0000 Subject: [Python-checkins] gh-91547: Remove "Undocumented modules" page (GH-91682) Message-ID: https://github.com/python/cpython/commit/7ade77709b5628594f6f2bedcd29f9e90337406a commit: 7ade77709b5628594f6f2bedcd29f9e90337406a branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-22T16:33:46-07:00 summary: gh-91547: Remove "Undocumented modules" page (GH-91682) (cherry picked from commit 254aaa7981d8773658fb14795da5dec888c95f93) Co-authored-by: Shantanu <12621235+hauntsaninja at users.noreply.github.com> files: A Misc/NEWS.d/next/Documentation/2022-04-19-20-16-00.gh-issue-91547.LsNWER.rst D Doc/library/undoc.rst M Doc/library/index.rst diff --git a/Doc/library/index.rst b/Doc/library/index.rst index db8f0d9bdfb8f..7d2002b37df12 100644 --- a/Doc/library/index.rst +++ b/Doc/library/index.rst @@ -74,5 +74,4 @@ the `Python Package Index `_. windows.rst unix.rst superseded.rst - undoc.rst security_warnings.rst diff --git a/Doc/library/undoc.rst b/Doc/library/undoc.rst deleted file mode 100644 index 2444080d6b9d9..0000000000000 --- a/Doc/library/undoc.rst +++ /dev/null @@ -1,26 +0,0 @@ -.. _undoc: - -******************** -Undocumented Modules -******************** - -Here's a quick listing of modules that are currently undocumented, but that -should be documented. Feel free to contribute documentation for them! (Send -via email to docs at python.org.) - -The idea and original contents for this chapter were taken from a posting by -Fredrik Lundh; the specific contents of this chapter have been substantially -revised. - - -Platform specific modules -========================= - -These modules are used to implement the :mod:`os.path` module, and are not -documented beyond this mention. There's little need to document these. - -:mod:`ntpath` - --- Implementation of :mod:`os.path` on Win32 and Win64 platforms. - -:mod:`posixpath` - --- Implementation of :mod:`os.path` on POSIX. diff --git a/Misc/NEWS.d/next/Documentation/2022-04-19-20-16-00.gh-issue-91547.LsNWER.rst b/Misc/NEWS.d/next/Documentation/2022-04-19-20-16-00.gh-issue-91547.LsNWER.rst new file mode 100644 index 0000000000000..95b34cb2fac1f --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2022-04-19-20-16-00.gh-issue-91547.LsNWER.rst @@ -0,0 +1 @@ +Remove "Undocumented modules" page. From webhook-mailer at python.org Fri Apr 22 20:30:56 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Sat, 23 Apr 2022 00:30:56 -0000 Subject: [Python-checkins] gh-85864: Mark positional-only args in io docs (#91683) Message-ID: https://github.com/python/cpython/commit/a3f2cf3ced378db2569df4e7389ec1f79c85d55c commit: a3f2cf3ced378db2569df4e7389ec1f79c85d55c branch: main author: slateny <46876382+slateny at users.noreply.github.com> committer: JelleZijlstra date: 2022-04-22T17:30:52-07:00 summary: gh-85864: Mark positional-only args in io docs (#91683) files: M Doc/library/io.rst diff --git a/Doc/library/io.rst b/Doc/library/io.rst index 53dad99fa1dbc..82757539c62ac 100644 --- a/Doc/library/io.rst +++ b/Doc/library/io.rst @@ -193,7 +193,7 @@ High-level Module Interface .. versionadded:: 3.8 -.. function:: text_encoding(encoding, stacklevel=2) +.. function:: text_encoding(encoding, stacklevel=2, /) This is a helper function for callables that use :func:`open` or :class:`TextIOWrapper` and have an ``encoding=None`` parameter. @@ -380,7 +380,7 @@ I/O Base Classes Return ``True`` if the stream can be read from. If ``False``, :meth:`read` will raise :exc:`OSError`. - .. method:: readline(size=-1) + .. method:: readline(size=-1, /) Read and return one line from the stream. If *size* is specified, at most *size* bytes will be read. @@ -389,7 +389,7 @@ I/O Base Classes the *newline* argument to :func:`open` can be used to select the line terminator(s) recognized. - .. method:: readlines(hint=-1) + .. method:: readlines(hint=-1, /) Read and return a list of lines from the stream. *hint* can be specified to control the number of lines read: no more lines will be read if the @@ -475,7 +475,7 @@ I/O Base Classes :class:`RawIOBase` provides these methods in addition to those from :class:`IOBase`: - .. method:: read(size=-1) + .. method:: read(size=-1, /) Read up to *size* bytes from the object and return them. As a convenience, if *size* is unspecified or -1, all bytes until EOF are returned. @@ -585,7 +585,7 @@ I/O Base Classes If *size* is ``-1`` (the default), an arbitrary number of bytes are returned (more than zero unless EOF is reached). - .. method:: readinto(b) + .. method:: readinto(b, /) Read bytes into a pre-allocated, writable :term:`bytes-like object` *b* and return the number of bytes read. @@ -597,7 +597,7 @@ I/O Base Classes A :exc:`BlockingIOError` is raised if the underlying raw stream is in non blocking-mode, and has no data available at the moment. - .. method:: readinto1(b) + .. method:: readinto1(b, /) Read bytes into a pre-allocated, writable :term:`bytes-like object` *b*, using at most one call to @@ -727,7 +727,7 @@ than raw I/O does. Return :class:`bytes` containing the entire contents of the buffer. - .. method:: read1([size]) + .. method:: read1([size], /) In :class:`BytesIO`, this is the same as :meth:`~BufferedIOBase.read`. @@ -805,7 +805,7 @@ than raw I/O does. Force bytes held in the buffer into the raw stream. A :exc:`BlockingIOError` should be raised if the raw stream blocks. - .. method:: write(b) + .. method:: write(b, /) Write the :term:`bytes-like object`, *b*, and return the number of bytes written. When in non-blocking mode, a @@ -828,7 +828,7 @@ than raw I/O does. are guaranteed to be implemented. -.. class:: BufferedRWPair(reader, writer, buffer_size=DEFAULT_BUFFER_SIZE) +.. class:: BufferedRWPair(reader, writer, buffer_size=DEFAULT_BUFFER_SIZE, /) A buffered binary stream providing higher-level access to two non seekable :class:`RawIOBase` raw binary streams---one readable, the other writeable. @@ -895,7 +895,7 @@ Text I/O .. versionadded:: 3.1 - .. method:: read(size=-1) + .. method:: read(size=-1, /) Read and return at most *size* characters from the stream as a single :class:`str`. If *size* is negative or ``None``, reads until EOF. From webhook-mailer at python.org Fri Apr 22 20:48:21 2022 From: webhook-mailer at python.org (miss-islington) Date: Sat, 23 Apr 2022 00:48:21 -0000 Subject: [Python-checkins] gh-85864: Mark positional-only args in io docs (GH-91683) Message-ID: https://github.com/python/cpython/commit/0482ed7d072c9b4d55479e719d95d6e41df7b4aa commit: 0482ed7d072c9b4d55479e719d95d6e41df7b4aa branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-22T17:48:13-07:00 summary: gh-85864: Mark positional-only args in io docs (GH-91683) (cherry picked from commit a3f2cf3ced378db2569df4e7389ec1f79c85d55c) Co-authored-by: slateny <46876382+slateny at users.noreply.github.com> files: M Doc/library/io.rst diff --git a/Doc/library/io.rst b/Doc/library/io.rst index d5123348195bd..b566d0c42499d 100644 --- a/Doc/library/io.rst +++ b/Doc/library/io.rst @@ -193,7 +193,7 @@ High-level Module Interface .. versionadded:: 3.8 -.. function:: text_encoding(encoding, stacklevel=2) +.. function:: text_encoding(encoding, stacklevel=2, /) This is a helper function for callables that use :func:`open` or :class:`TextIOWrapper` and have an ``encoding=None`` parameter. @@ -375,7 +375,7 @@ I/O Base Classes Return ``True`` if the stream can be read from. If ``False``, :meth:`read` will raise :exc:`OSError`. - .. method:: readline(size=-1) + .. method:: readline(size=-1, /) Read and return one line from the stream. If *size* is specified, at most *size* bytes will be read. @@ -384,7 +384,7 @@ I/O Base Classes the *newline* argument to :func:`open` can be used to select the line terminator(s) recognized. - .. method:: readlines(hint=-1) + .. method:: readlines(hint=-1, /) Read and return a list of lines from the stream. *hint* can be specified to control the number of lines read: no more lines will be read if the @@ -470,7 +470,7 @@ I/O Base Classes :class:`RawIOBase` provides these methods in addition to those from :class:`IOBase`: - .. method:: read(size=-1) + .. method:: read(size=-1, /) Read up to *size* bytes from the object and return them. As a convenience, if *size* is unspecified or -1, all bytes until EOF are returned. @@ -580,7 +580,7 @@ I/O Base Classes If *size* is ``-1`` (the default), an arbitrary number of bytes are returned (more than zero unless EOF is reached). - .. method:: readinto(b) + .. method:: readinto(b, /) Read bytes into a pre-allocated, writable :term:`bytes-like object` *b* and return the number of bytes read. @@ -592,7 +592,7 @@ I/O Base Classes A :exc:`BlockingIOError` is raised if the underlying raw stream is in non blocking-mode, and has no data available at the moment. - .. method:: readinto1(b) + .. method:: readinto1(b, /) Read bytes into a pre-allocated, writable :term:`bytes-like object` *b*, using at most one call to @@ -722,7 +722,7 @@ than raw I/O does. Return :class:`bytes` containing the entire contents of the buffer. - .. method:: read1([size]) + .. method:: read1([size], /) In :class:`BytesIO`, this is the same as :meth:`~BufferedIOBase.read`. @@ -800,7 +800,7 @@ than raw I/O does. Force bytes held in the buffer into the raw stream. A :exc:`BlockingIOError` should be raised if the raw stream blocks. - .. method:: write(b) + .. method:: write(b, /) Write the :term:`bytes-like object`, *b*, and return the number of bytes written. When in non-blocking mode, a @@ -823,7 +823,7 @@ than raw I/O does. are guaranteed to be implemented. -.. class:: BufferedRWPair(reader, writer, buffer_size=DEFAULT_BUFFER_SIZE) +.. class:: BufferedRWPair(reader, writer, buffer_size=DEFAULT_BUFFER_SIZE, /) A buffered binary stream providing higher-level access to two non seekable :class:`RawIOBase` raw binary streams---one readable, the other writeable. @@ -890,7 +890,7 @@ Text I/O .. versionadded:: 3.1 - .. method:: read(size=-1) + .. method:: read(size=-1, /) Read and return at most *size* characters from the stream as a single :class:`str`. If *size* is negative or ``None``, reads until EOF. From webhook-mailer at python.org Fri Apr 22 22:17:16 2022 From: webhook-mailer at python.org (pablogsal) Date: Sat, 23 Apr 2022 02:17:16 -0000 Subject: [Python-checkins] gh-88116: Enhance the inspect frame APIs to use the extended position information (GH-91531) Message-ID: https://github.com/python/cpython/commit/0daa99f68b7b9f02b37a2f34508f33ae66d95fc4 commit: 0daa99f68b7b9f02b37a2f34508f33ae66d95fc4 branch: main author: Pablo Galindo Salgado committer: pablogsal date: 2022-04-23T03:16:48+01:00 summary: gh-88116: Enhance the inspect frame APIs to use the extended position information (GH-91531) files: A Misc/NEWS.d/next/Library/2022-04-14-13-11-37.gh-issue-88116.j_SybE.rst M Doc/library/inspect.rst M Doc/whatsnew/3.11.rst M Lib/inspect.py M Lib/test/test_inspect.py diff --git a/Doc/library/inspect.rst b/Doc/library/inspect.rst index 8ee2c070cccf5..575b3088900e1 100644 --- a/Doc/library/inspect.rst +++ b/Doc/library/inspect.rst @@ -1163,17 +1163,85 @@ Classes and functions The interpreter stack --------------------- -When the following functions return "frame records," each record is a -:term:`named tuple` -``FrameInfo(frame, filename, lineno, function, code_context, index)``. -The tuple contains the frame object, the filename, the line number of the -current line, -the function name, a list of lines of context from the source code, and the -index of the current line within that list. +Some of the following functions return +:class:`FrameInfo` objects. For backwards compatibility these objects allow +tuple-like operations on all attributes except ``positions``. This behavior +is considered deprecated and may be removed in the future. + +.. class:: FrameInfo + + .. attribute:: frame + + The :ref:`frame object ` that the record corresponds to. + + .. attribute:: filename + + The file name associated with the code being executed by the frame this record + corresponds to. + + .. attribute:: lineno + + The line number of the current line associated with the code being + executed by the frame this record corresponds to. + + .. attribute:: function + + The function name that is being executed by the frame this record corresponds to. + + .. attribute:: code_context + + A list of lines of context from the source code that's being executed by the frame + this record corresponds to. + + .. attribute:: index + + The index of the current line being executed in the :attr:`code_context` list. + + .. attribute:: positions + + A :class:`dis.Positions` object containing the start line number, end line + number, start column offset, and end column offset associated with the + instruction being executed by the frame this record corresponds to. .. versionchanged:: 3.5 Return a named tuple instead of a tuple. +.. versionchanged:: 3.11 + Changed the return object from a named tuple to a regular object (that is + backwards compatible with the previous named tuple). + +.. class:: Traceback + + .. attribute:: filename + + The file name associated with the code being executed by the frame this traceback + corresponds to. + + .. attribute:: lineno + + The line number of the current line associated with the code being + executed by the frame this traceback corresponds to. + + .. attribute:: function + + The function name that is being executed by the frame this traceback corresponds to. + + .. attribute:: code_context + + A list of lines of context from the source code that's being executed by the frame + this traceback corresponds to. + + .. attribute:: index + + The index of the current line being executed in the :attr:`code_context` list. + + .. attribute:: positions + + A :class:`dis.Positions` object containing the start line number, end + line number, start column offset, and end column offset associated with + the instruction being executed by the frame this traceback corresponds + to. + .. note:: Keeping references to frame objects, as found in the first element of the frame @@ -1207,35 +1275,41 @@ line. .. function:: getframeinfo(frame, context=1) - Get information about a frame or traceback object. A :term:`named tuple` - ``Traceback(filename, lineno, function, code_context, index)`` is returned. + Get information about a frame or traceback object. A :class:`Traceback` object + is returned. + .. versionchanged:: 3.11 + A :class:`Traceback` object is returned instead of a named tuple. .. function:: getouterframes(frame, context=1) - Get a list of frame records for a frame and all outer frames. These frames - represent the calls that lead to the creation of *frame*. The first entry in the - returned list represents *frame*; the last entry represents the outermost call - on *frame*'s stack. + Get a list of :class:`FrameInfo` objects for a frame and all outer frames. + These frames represent the calls that lead to the creation of *frame*. The + first entry in the returned list represents *frame*; the last entry + represents the outermost call on *frame*'s stack. .. versionchanged:: 3.5 A list of :term:`named tuples ` ``FrameInfo(frame, filename, lineno, function, code_context, index)`` is returned. + .. versionchanged:: 3.11 + A list of :class:`FrameInfo` objects is returned. .. function:: getinnerframes(traceback, context=1) - Get a list of frame records for a traceback's frame and all inner frames. These - frames represent calls made as a consequence of *frame*. The first entry in the - list represents *traceback*; the last entry represents where the exception was - raised. + Get a list of :class:`FrameInfo` objects for a traceback's frame and all + inner frames. These frames represent calls made as a consequence of *frame*. + The first entry in the list represents *traceback*; the last entry represents + where the exception was raised. .. versionchanged:: 3.5 A list of :term:`named tuples ` ``FrameInfo(frame, filename, lineno, function, code_context, index)`` is returned. + .. versionchanged:: 3.11 + A list of :class:`FrameInfo` objects is returned. .. function:: currentframe() @@ -1251,28 +1325,32 @@ line. .. function:: stack(context=1) - Return a list of frame records for the caller's stack. The first entry in the - returned list represents the caller; the last entry represents the outermost - call on the stack. + Return a list of :class:`FrameInfo` objects for the caller's stack. The + first entry in the returned list represents the caller; the last entry + represents the outermost call on the stack. .. versionchanged:: 3.5 A list of :term:`named tuples ` ``FrameInfo(frame, filename, lineno, function, code_context, index)`` is returned. + .. versionchanged:: 3.11 + A list of :class:`FrameInfo` objects is returned. .. function:: trace(context=1) - Return a list of frame records for the stack between the current frame and the - frame in which an exception currently being handled was raised in. The first - entry in the list represents the caller; the last entry represents where the - exception was raised. + Return a list of :class:`FrameInfo` objects for the stack between the current + frame and the frame in which an exception currently being handled was raised + in. The first entry in the list represents the caller; the last entry + represents where the exception was raised. .. versionchanged:: 3.5 A list of :term:`named tuples ` ``FrameInfo(frame, filename, lineno, function, code_context, index)`` is returned. + .. versionchanged:: 3.11 + A list of :class:`FrameInfo` objects is returned. Fetching attributes statically ------------------------------ diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index 653d32aaf540b..5b53fbe0dda6f 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -326,6 +326,14 @@ inspect * Add :func:`inspect.ismethodwrapper` for checking if the type of an object is a :class:`~types.MethodWrapperType`. (Contributed by Hakan ?elik in :issue:`29418`.) +* Change the frame-related functions in the :mod:`inspect` module to return a + regular object (that is backwards compatible with the old tuple-like + interface) that include the extended :pep:`657` position information (end + line number, column and end column). The affected functions are: + :func:`inspect.getframeinfo`, :func:`inspect.getouterframes`, :func:`inspect.getinnerframes`, + :func:`inspect.stack` and :func:`inspect.trace`. (Contributed by Pablo Galindo in + :issue:`88116`) + locale ------ diff --git a/Lib/inspect.py b/Lib/inspect.py index 9c1283ab3734b..5bc9c04b22e23 100644 --- a/Lib/inspect.py +++ b/Lib/inspect.py @@ -1638,7 +1638,30 @@ def getclosurevars(func): # -------------------------------------------------- stack frame extraction -Traceback = namedtuple('Traceback', 'filename lineno function code_context index') +_Traceback = namedtuple('_Traceback', 'filename lineno function code_context index') + +class Traceback(_Traceback): + def __new__(cls, filename, lineno, function, code_context, index, *, positions=None): + instance = super().__new__(cls, filename, lineno, function, code_context, index) + instance.positions = positions + return instance + + def __repr__(self): + return ('Traceback(filename={!r}, lineno={!r}, function={!r}, ' + 'code_context={!r}, index={!r}, positions={!r})'.format( + self.filename, self.lineno, self.function, self.code_context, + self.index, self.positions)) + +def _get_code_position_from_tb(tb): + code, instruction_index = tb.tb_frame.f_code, tb.tb_lasti + return _get_code_position(code, instruction_index) + +def _get_code_position(code, instruction_index): + if instruction_index < 0: + return (None, None, None, None) + positions_gen = code.co_positions() + # The nth entry in code.co_positions() corresponds to instruction (2*n)th since Python 3.10+ + return next(itertools.islice(positions_gen, instruction_index // 2, None)) def getframeinfo(frame, context=1): """Get information about a frame or traceback object. @@ -1649,10 +1672,20 @@ def getframeinfo(frame, context=1): The optional second argument specifies the number of lines of context to return, which are centered around the current line.""" if istraceback(frame): + positions = _get_code_position_from_tb(frame) lineno = frame.tb_lineno frame = frame.tb_frame else: lineno = frame.f_lineno + positions = _get_code_position(frame.f_code, frame.f_lasti) + + if positions[0] is None: + frame, *positions = (frame, lineno, *positions[1:]) + else: + frame, *positions = (frame, *positions) + + lineno = positions[0] + if not isframe(frame): raise TypeError('{!r} is not a frame or traceback object'.format(frame)) @@ -1670,14 +1703,26 @@ def getframeinfo(frame, context=1): else: lines = index = None - return Traceback(filename, lineno, frame.f_code.co_name, lines, index) + return Traceback(filename, lineno, frame.f_code.co_name, lines, + index, positions=dis.Positions(*positions)) def getlineno(frame): """Get the line number from a frame object, allowing for optimization.""" # FrameType.f_lineno is now a descriptor that grovels co_lnotab return frame.f_lineno -FrameInfo = namedtuple('FrameInfo', ('frame',) + Traceback._fields) +_FrameInfo = namedtuple('_FrameInfo', ('frame',) + Traceback._fields) +class FrameInfo(_FrameInfo): + def __new__(cls, frame, filename, lineno, function, code_context, index, *, positions=None): + instance = super().__new__(cls, frame, filename, lineno, function, code_context, index) + instance.positions = positions + return instance + + def __repr__(self): + return ('FrameInfo(frame={!r}, filename={!r}, lineno={!r}, function={!r}, ' + 'code_context={!r}, index={!r}, positions={!r})'.format( + self.frame, self.filename, self.lineno, self.function, + self.code_context, self.index, self.positions)) def getouterframes(frame, context=1): """Get a list of records for a frame and all higher (calling) frames. @@ -1686,8 +1731,9 @@ def getouterframes(frame, context=1): name, a list of lines of context, and index within the context.""" framelist = [] while frame: - frameinfo = (frame,) + getframeinfo(frame, context) - framelist.append(FrameInfo(*frameinfo)) + traceback_info = getframeinfo(frame, context) + frameinfo = (frame,) + traceback_info + framelist.append(FrameInfo(*frameinfo, positions=traceback_info.positions)) frame = frame.f_back return framelist @@ -1698,8 +1744,9 @@ def getinnerframes(tb, context=1): name, a list of lines of context, and index within the context.""" framelist = [] while tb: - frameinfo = (tb.tb_frame,) + getframeinfo(tb, context) - framelist.append(FrameInfo(*frameinfo)) + traceback_info = getframeinfo(tb, context) + frameinfo = (tb.tb_frame,) + traceback_info + framelist.append(FrameInfo(*frameinfo, positions=traceback_info.positions)) tb = tb.tb_next return framelist diff --git a/Lib/test/test_inspect.py b/Lib/test/test_inspect.py index 9e3c77056d70a..115e97b77e079 100644 --- a/Lib/test/test_inspect.py +++ b/Lib/test/test_inspect.py @@ -7,6 +7,7 @@ import io import linecache import os +import dis from os.path import normcase import _pickle import pickle @@ -361,14 +362,23 @@ def test_abuse_done(self): def test_stack(self): self.assertTrue(len(mod.st) >= 5) - self.assertEqual(revise(*mod.st[0][1:]), + frame1, frame2, frame3, frame4, *_ = mod.st + frameinfo = revise(*frame1[1:]) + self.assertEqual(frameinfo, (modfile, 16, 'eggs', [' st = inspect.stack()\n'], 0)) - self.assertEqual(revise(*mod.st[1][1:]), + self.assertEqual(frame1.positions, dis.Positions(16, 16, 9, 24)) + frameinfo = revise(*frame2[1:]) + self.assertEqual(frameinfo, (modfile, 9, 'spam', [' eggs(b + d, c + f)\n'], 0)) - self.assertEqual(revise(*mod.st[2][1:]), + self.assertEqual(frame2.positions, dis.Positions(9, 9, 4, 22)) + frameinfo = revise(*frame3[1:]) + self.assertEqual(frameinfo, (modfile, 43, 'argue', [' spam(a, b, c)\n'], 0)) - self.assertEqual(revise(*mod.st[3][1:]), + self.assertEqual(frame3.positions, dis.Positions(43, 43, 12, 25)) + frameinfo = revise(*frame4[1:]) + self.assertEqual(frameinfo, (modfile, 39, 'abuse', [' self.argue(a, b, c)\n'], 0)) + self.assertEqual(frame4.positions, dis.Positions(39, 39, 8, 27)) # Test named tuple fields record = mod.st[0] self.assertIs(record.frame, mod.fr) @@ -380,12 +390,16 @@ def test_stack(self): def test_trace(self): self.assertEqual(len(git.tr), 3) - self.assertEqual(revise(*git.tr[0][1:]), + frame1, frame2, frame3, = git.tr + self.assertEqual(revise(*frame1[1:]), (modfile, 43, 'argue', [' spam(a, b, c)\n'], 0)) - self.assertEqual(revise(*git.tr[1][1:]), + self.assertEqual(frame1.positions, dis.Positions(43, 43, 12, 25)) + self.assertEqual(revise(*frame2[1:]), (modfile, 9, 'spam', [' eggs(b + d, c + f)\n'], 0)) - self.assertEqual(revise(*git.tr[2][1:]), + self.assertEqual(frame2.positions, dis.Positions(9, 9, 4, 22)) + self.assertEqual(revise(*frame3[1:]), (modfile, 18, 'eggs', [' q = y / 0\n'], 0)) + self.assertEqual(frame3.positions, dis.Positions(18, 18, 8, 13)) def test_frame(self): args, varargs, varkw, locals = inspect.getargvalues(mod.fr) diff --git a/Misc/NEWS.d/next/Library/2022-04-14-13-11-37.gh-issue-88116.j_SybE.rst b/Misc/NEWS.d/next/Library/2022-04-14-13-11-37.gh-issue-88116.j_SybE.rst new file mode 100644 index 0000000000000..1b7bd6929e959 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-14-13-11-37.gh-issue-88116.j_SybE.rst @@ -0,0 +1,8 @@ +Change the frame-related functions in the :mod:`inspect` module to return a +regular object (that is backwards compatible with the old tuple-like interface) +that include the extended :pep:`657` position information (end line number, +column and end column). The affected functions are: :func:`inspect.getframeinfo`, +:func:`inspect.getouterframes`, :func:`inspect.getinnerframes`, :func:`inspect.stack` and +:func:`inspect.trace`. Patch by Pablo Galindo. + + From webhook-mailer at python.org Sat Apr 23 03:52:26 2022 From: webhook-mailer at python.org (tiran) Date: Sat, 23 Apr 2022 07:52:26 -0000 Subject: [Python-checkins] gh-84461: Add sys._emscripten_info, improve docs and build (gh-91781) Message-ID: https://github.com/python/cpython/commit/9b5ca5405e5a2786b5b3acc0de578f80f8dc9e36 commit: 9b5ca5405e5a2786b5b3acc0de578f80f8dc9e36 branch: main author: Christian Heimes committer: tiran date: 2022-04-23T09:52:16+02:00 summary: gh-84461: Add sys._emscripten_info, improve docs and build (gh-91781) files: A Misc/NEWS.d/next/Library/2022-04-21-11-57-23.gh-issue-84461.S7dbt4.rst M Doc/library/sys.rst M Lib/test/support/threading_helper.py M Lib/test/test_sys.py M Python/sysmodule.c M Tools/wasm/README.md M configure M configure.ac diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst index 2a8b532b592e8..f43344887e421 100644 --- a/Doc/library/sys.rst +++ b/Doc/library/sys.rst @@ -314,6 +314,35 @@ always available. yourself to control bytecode file generation. +.. data:: _emscripten_info + + A :term:`named tuple` holding information about the environment on the + *wasm32-emscripten* platform. The named tuple is provisional and may change + in the future. + + .. tabularcolumns:: |l|L| + + +-----------------------------+----------------------------------------------+ + | Attribute | Explanation | + +=============================+==============================================+ + | :const:`emscripten_version` | Emscripten version as tuple of ints | + | | (major, minor, micro), e.g. ``(3, 1, 8)``. | + +-----------------------------+----------------------------------------------+ + | :const:`runtime` | Runtime string, e.g. browser user agent, | + | | ``'Node.js v14.18.2'``, or ``'UNKNOWN'``. | + +-----------------------------+----------------------------------------------+ + | :const:`pthreads` | ``True`` if Python is compiled with | + | | Emscripten pthreads support. | + +-----------------------------+----------------------------------------------+ + | :const:`shared_memory` | ``True`` if Python is compiled with shared | + | | memory support. | + +-----------------------------+----------------------------------------------+ + + .. availability:: WebAssembly Emscripten platform (*wasm32-emscripten*). + + .. versionadded:: 3.11 + + .. data:: pycache_prefix If this is set (not ``None``), Python will write bytecode-cache ``.pyc`` diff --git a/Lib/test/support/threading_helper.py b/Lib/test/support/threading_helper.py index 7b636f0ccf097..26cbc6f4d2439 100644 --- a/Lib/test/support/threading_helper.py +++ b/Lib/test/support/threading_helper.py @@ -222,12 +222,7 @@ def _can_start_thread() -> bool: support (-s USE_PTHREADS / __EMSCRIPTEN_PTHREADS__). """ if sys.platform == "emscripten": - try: - _thread.start_new_thread(lambda: None, ()) - except RuntimeError: - return False - else: - return True + return sys._emscripten_info.pthreads elif sys.platform == "wasi": return False else: diff --git a/Lib/test/test_sys.py b/Lib/test/test_sys.py index b70871ff551d9..bbf01b6a39011 100644 --- a/Lib/test/test_sys.py +++ b/Lib/test/test_sys.py @@ -629,6 +629,14 @@ def test_thread_info(self): self.assertIn(info.name, ('nt', 'pthread', 'solaris', None)) self.assertIn(info.lock, ('semaphore', 'mutex+cond', None)) + @unittest.skipUnless(support.is_emscripten, "only available on Emscripten") + def test_emscripten_info(self): + self.assertEqual(len(sys._emscripten_info), 4) + self.assertIsInstance(sys._emscripten_info.emscripten_version, tuple) + self.assertIsInstance(sys._emscripten_info.runtime, (str, type(None))) + self.assertIsInstance(sys._emscripten_info.pthreads, bool) + self.assertIsInstance(sys._emscripten_info.shared_memory, bool) + def test_43581(self): # Can't use sys.stdout, as this is a StringIO object when # the test runs under regrtest. diff --git a/Misc/NEWS.d/next/Library/2022-04-21-11-57-23.gh-issue-84461.S7dbt4.rst b/Misc/NEWS.d/next/Library/2022-04-21-11-57-23.gh-issue-84461.S7dbt4.rst new file mode 100644 index 0000000000000..08448d7d7ce21 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-21-11-57-23.gh-issue-84461.S7dbt4.rst @@ -0,0 +1,2 @@ +Add provisional :data:`sys._emscripten_info` named tuple with build-time and +run-time information about Emscripten platform. diff --git a/Python/sysmodule.c b/Python/sysmodule.c index ac44b803b23f5..d5a62fc12b668 100644 --- a/Python/sysmodule.c +++ b/Python/sysmodule.c @@ -48,6 +48,10 @@ extern void *PyWin_DLLhModule; extern const char *PyWin_DLLVersionString; #endif +#ifdef __EMSCRIPTEN__ +#include +#endif + /*[clinic input] module sys [clinic start generated code]*/ @@ -2686,6 +2690,107 @@ make_impl_info(PyObject *version_info) return NULL; } +#ifdef __EMSCRIPTEN__ + +PyDoc_STRVAR(emscripten_info__doc__, +"sys._emscripten_info\n\ +\n\ +WebAssembly Emscripten platform information."); + +static PyTypeObject *EmscriptenInfoType; + +static PyStructSequence_Field emscripten_info_fields[] = { + {"emscripten_version", "Emscripten version (major, minor, micro)"}, + {"runtime", "Runtime (Node.JS version, browser user agent)"}, + {"pthreads", "pthread support"}, + {"shared_memory", "shared memory support"}, + {0} +}; + +static PyStructSequence_Desc emscripten_info_desc = { + "sys._emscripten_info", /* name */ + emscripten_info__doc__ , /* doc */ + emscripten_info_fields, /* fields */ + 4 +}; + +EM_JS(char *, _Py_emscripten_runtime, (void), { + var info; + if (typeof navigator == 'object') { + info = navigator.userAgent; + } else if (typeof process == 'object') { + info = "Node.js ".concat(process.version) + } else { + info = "UNKNOWN" + } + var len = lengthBytesUTF8(info) + 1; + var res = _malloc(len); + stringToUTF8(info, res, len); + return res; +}); + +static PyObject * +make_emscripten_info(void) +{ + PyObject *emscripten_info = NULL; + PyObject *version = NULL; + char *ua; + int pos = 0; + + emscripten_info = PyStructSequence_New(EmscriptenInfoType); + if (emscripten_info == NULL) { + return NULL; + } + + version = Py_BuildValue("(iii)", + __EMSCRIPTEN_major__, __EMSCRIPTEN_minor__, __EMSCRIPTEN_tiny__); + if (version == NULL) { + goto error; + } + PyStructSequence_SET_ITEM(emscripten_info, pos++, version); + + ua = _Py_emscripten_runtime(); + if (ua != NULL) { + PyObject *oua = PyUnicode_DecodeUTF8(ua, strlen(ua), "strict"); + free(ua); + if (oua == NULL) { + goto error; + } + PyStructSequence_SET_ITEM(emscripten_info, pos++, oua); + } else { + Py_INCREF(Py_None); + PyStructSequence_SET_ITEM(emscripten_info, pos++, Py_None); + } + +#define SetBoolItem(flag) \ + PyStructSequence_SET_ITEM(emscripten_info, pos++, PyBool_FromLong(flag)) + +#ifdef __EMSCRIPTEN_PTHREADS__ + SetBoolItem(1); +#else + SetBoolItem(0); +#endif + +#ifdef __EMSCRIPTEN_SHARED_MEMORY__ + SetBoolItem(1); +#else + SetBoolItem(0); +#endif + +#undef SetBoolItem + + if (PyErr_Occurred()) { + goto error; + } + return emscripten_info; + + error: + Py_CLEAR(emscripten_info); + return NULL; +} + +#endif // __EMSCRIPTEN__ + static struct PyModuleDef sysmodule = { PyModuleDef_HEAD_INIT, "sys", @@ -2821,6 +2926,16 @@ _PySys_InitCore(PyThreadState *tstate, PyObject *sysdict) } } +#ifdef __EMSCRIPTEN__ + if (EmscriptenInfoType == NULL) { + EmscriptenInfoType = PyStructSequence_NewType(&emscripten_info_desc); + if (EmscriptenInfoType == NULL) { + goto type_init_failed; + } + } + SET_SYS("_emscripten_info", make_emscripten_info()); +#endif + /* adding sys.path_hooks and sys.path_importer_cache */ SET_SYS("meta_path", PyList_New(0)); SET_SYS("path_importer_cache", PyDict_New()); @@ -3066,6 +3181,9 @@ _PySys_Fini(PyInterpreterState *interp) #endif _PyStructSequence_FiniType(&Hash_InfoType); _PyStructSequence_FiniType(&AsyncGenHooksType); +#ifdef __EMSCRIPTEN__ + Py_CLEAR(EmscriptenInfoType); +#endif } } diff --git a/Tools/wasm/README.md b/Tools/wasm/README.md index fa99703acfc5b..9d3680a53d28e 100644 --- a/Tools/wasm/README.md +++ b/Tools/wasm/README.md @@ -4,11 +4,17 @@ This directory contains configuration and helpers to facilitate cross compilation of CPython to WebAssembly (WASM). For now we support -*wasm32-emscripten* builds for modern browser and for *Node.js*. It's not -possible to build for *wasm32-wasi* out-of-the-box yet. +*wasm32-emscripten* builds for modern browser and for *Node.js*. WASI +(*wasm32-wasi*) is work-in-progress ## wasm32-emscripten build +For now the build system has two target flavors. The ``Emscripten/browser`` +target (``--with-emscripten-target=browser``) is optimized for browsers. +It comes with a reduced and preloaded stdlib without tests and threading +support. The ``Emscripten/node`` target has threading enabled and can +access the file system directly. + Cross compiling to the wasm32-emscripten platform needs the [Emscripten](https://emscripten.org/) SDK and a build Python interpreter. Emscripten 3.1.8 or newer are recommended. All commands below are relative @@ -76,7 +82,7 @@ and header files with debug builds. ### Cross compile to wasm32-emscripten for node -``` +```shell mkdir -p builddir/emscripten-node pushd builddir/emscripten-node @@ -91,7 +97,7 @@ emmake make -j$(nproc) popd ``` -``` +```shell node --experimental-wasm-threads --experimental-wasm-bulk-memory builddir/emscripten-node/python.js ``` @@ -150,9 +156,9 @@ functions. - Most stdlib modules with a dependency on external libraries are missing, e.g. ``ctypes``, ``readline``, ``sqlite3``, ``ssl``, and more. - Shared extension modules are not implemented yet. All extension modules - are statically linked into the main binary. - The experimental configure option ``--enable-wasm-dynamic-linking`` enables - dynamic extensions. + are statically linked into the main binary. The experimental configure + option ``--enable-wasm-dynamic-linking`` enables dynamic extensions + supports. It's currently known to crash in combination with threading. - glibc extensions for date and time formatting are not available. - ``locales`` module is affected by musl libc issues, [bpo-46390](https://bugs.python.org/issue46390). @@ -167,8 +173,10 @@ functions. distutils, multiprocessing, dbm, tests and similar modules are not shipped. All other modules are bundled as pre-compiled ``pyc`` files. -- Threading is not supported. +- Threading is disabled. - In-memory file system (MEMFS) is not persistent and limited. +- Test modules are disabled by default. Use ``--enable-test-modules`` build + test modules like ``_testcapi``. ## wasm32-emscripten in node @@ -205,11 +213,17 @@ AddType application/wasm wasm ``` +# WASI (wasm32-wasi) + +WASI builds require [WASI SDK](https://github.com/WebAssembly/wasi-sdk) and +currently [wasix](https://github.com/singlestore-labs/wasix) for POSIX +compatibility stubs. + # Detect WebAssembly builds ## Python code -```# python +```python import os, sys if sys.platform == "emscripten": @@ -222,7 +236,36 @@ if os.name == "posix": # Windows does not provide os.uname(). machine = os.uname().machine if machine.startswith("wasm"): - # WebAssembly (wasm32 or wasm64) + # WebAssembly (wasm32, wasm64 in the future) +``` + +```python +>>> import os, sys +>>> os.uname() +posix.uname_result(sysname='Emscripten', nodename='emscripten', release='1.0', version='#1', machine='wasm32') +>>> os.name +'posix' +>>> sys.platform +'emscripten' +>>> sys._emscripten_info +sys._emscripten_info( + emscripten_version=(3, 1, 8), + runtime='Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:99.0) Gecko/20100101 Firefox/99.0', + pthreads=False, + shared_memory=False +) +>>> sys._emscripten_info +sys._emscripten_info(emscripten_version=(3, 1, 8), runtime='Node.js v14.18.2', pthreads=True, shared_memory=True) +``` + +```python +>>> import os, sys +>>> os.uname() +posix.uname_result(sysname='wasi', nodename='(none)', release='0.0.0', version='0.0.0', machine='wasm32') +>>> os.name +'posix' +>>> sys.platform +'wasi' ``` ## C code @@ -231,7 +274,7 @@ Emscripten SDK and WASI SDK define several built-in macros. You can dump a full list of built-ins with ``emcc -dM -E - < /dev/null`` and ``/path/to/wasi-sdk/bin/clang -dM -E - < /dev/null``. -```# C +```C #ifdef __EMSCRIPTEN__ // Python on Emscripten #endif diff --git a/configure b/configure index 94adc6fdf015e..25e767be82fa4 100755 --- a/configure +++ b/configure @@ -10513,6 +10513,9 @@ then BLDSHARED="$LDSHARED" fi ;; + Emscripten|WASI) + LDSHARED='$(CC) -shared' + LDCXXSHARED='$(CXX) -shared';; Linux*|GNU*|QNX*|VxWorks*|Haiku*) LDSHARED='$(CC) -shared' LDCXXSHARED='$(CXX) -shared';; @@ -22374,14 +22377,14 @@ $as_echo "$TEST_MODULES" >&6; } # stdlib not available -case $ac_sys_system/$ac_sys_emscripten_target in #( - AIX/*) : +case $ac_sys_system in #( + AIX) : py_cv_module__scproxy=n/a py_cv_module_spwd=n/a ;; #( - VxWorks*/*) : + VxWorks*) : py_cv_module__scproxy=n/a @@ -22389,35 +22392,34 @@ case $ac_sys_system/$ac_sys_emscripten_target in #( py_cv_module_termios=n/a py_cv_module_grp=n/a ;; #( - Darwin/*) : + Darwin) : py_cv_module_ossaudiodev=n/a py_cv_module_spwd=n/a ;; #( - CYGWIN*/*) : + CYGWIN*) : py_cv_module__scproxy=n/a py_cv_module_nis=n/a ;; #( - QNX*/*) : + QNX*) : py_cv_module__scproxy=n/a py_cv_module_nis=n/a ;; #( - FreeBSD*/*) : + FreeBSD*) : py_cv_module__scproxy=n/a py_cv_module_spwd=n/a ;; #( - Emscripten/browser*) : + Emscripten|WASI) : - py_cv_module__ctypes=n/a py_cv_module__curses=n/a py_cv_module__curses_panel=n/a py_cv_module__dbm=n/a @@ -22428,64 +22430,34 @@ case $ac_sys_system/$ac_sys_emscripten_target in #( py_cv_module__scproxy=n/a py_cv_module__tkinter=n/a py_cv_module__xxsubinterpreters=n/a - py_cv_module_fcntl=n/a py_cv_module_grp=n/a py_cv_module_nis=n/a py_cv_module_ossaudiodev=n/a py_cv_module_pwd=n/a - py_cv_module_resource=n/a - py_cv_module_readline=n/a py_cv_module_spwd=n/a py_cv_module_syslog=n/a - py_cv_module_termios=n/a py_cv_module_=n/a - ;; #( - Emscripten/node*) : + case $ac_sys_system/$ac_sys_emscripten_target in #( + Emscripten/browser*) : - py_cv_module__ctypes=n/a - py_cv_module__curses=n/a - py_cv_module__curses_panel=n/a - py_cv_module__dbm=n/a - py_cv_module__gdbm=n/a - py_cv_module__multiprocessing=n/a - py_cv_module__posixshmem=n/a - py_cv_module__posixsubprocess=n/a - py_cv_module__scproxy=n/a - py_cv_module__tkinter=n/a - py_cv_module__xxsubinterpreters=n/a - py_cv_module_grp=n/a - py_cv_module_nis=n/a - py_cv_module_ossaudiodev=n/a - py_cv_module_pwd=n/a - py_cv_module_spwd=n/a - py_cv_module_syslog=n/a + py_cv_module_fcntl=n/a + py_cv_module_resource=n/a + py_cv_module_readline=n/a + py_cv_module_termios=n/a py_cv_module_=n/a - ;; #( + ;; #( + Emscripten/node*) : + ;; #( WASI/*) : - - - py_cv_module__ctypes=n/a - py_cv_module__ctypes_test=n/a - py_cv_module__curses=n/a - py_cv_module__curses_panel=n/a - py_cv_module__dbm=n/a - py_cv_module__gdbm=n/a - py_cv_module__scproxy=n/a - py_cv_module__tkinter=n/a - py_cv_module__xxsubinterpreters=n/a - py_cv_module_grp=n/a - py_cv_module_nis=n/a - py_cv_module_ossaudiodev=n/a - py_cv_module_pwd=n/a - py_cv_module_spwd=n/a - py_cv_module_syslog=n/a - py_cv_module_=n/a - + ;; #( + *) : + ;; +esac ;; #( *) : diff --git a/configure.ac b/configure.ac index 9b60b98d1434e..82211f4ee1d68 100644 --- a/configure.ac +++ b/configure.ac @@ -2954,6 +2954,9 @@ then BLDSHARED="$LDSHARED" fi ;; + Emscripten|WASI) + LDSHARED='$(CC) -shared' + LDCXXSHARED='$(CXX) -shared';; Linux*|GNU*|QNX*|VxWorks*|Haiku*) LDSHARED='$(CC) -shared' LDCXXSHARED='$(CXX) -shared';; @@ -6585,43 +6588,19 @@ AC_DEFUN([PY_STDLIB_MOD_SET_NA], [ dnl Modules that are not available on some platforms dnl AIX has shadow passwords, but access is not via getspent() dnl VxWorks does not provide crypt() function -AS_CASE([$ac_sys_system/$ac_sys_emscripten_target], - [AIX/*], [PY_STDLIB_MOD_SET_NA([_scproxy], [spwd])], - [VxWorks*/*], [PY_STDLIB_MOD_SET_NA([_scproxy], [_crypt], [termios], [grp])], - [Darwin/*], [PY_STDLIB_MOD_SET_NA([ossaudiodev], [spwd])], - [CYGWIN*/*], [PY_STDLIB_MOD_SET_NA([_scproxy], [nis])], - [QNX*/*], [PY_STDLIB_MOD_SET_NA([_scproxy], [nis])], - [FreeBSD*/*], [PY_STDLIB_MOD_SET_NA([_scproxy], [spwd])], - [Emscripten/browser*], [ - PY_STDLIB_MOD_SET_NA( - [_ctypes], - [_curses], - [_curses_panel], - [_dbm], - [_gdbm], - [_multiprocessing], - [_posixshmem], - [_posixsubprocess], - [_scproxy], - [_tkinter], - [_xxsubinterpreters], - [fcntl], - [grp], - [nis], - [ossaudiodev], - [pwd], - [resource], - [readline], - [spwd], - [syslog], - [termios], - ) - ], - dnl Some modules like _posixsubprocess do not work. We build them anyway - dnl so imports in tests do not fail. - [Emscripten/node*], [ +AS_CASE([$ac_sys_system], + [AIX], [PY_STDLIB_MOD_SET_NA([_scproxy], [spwd])], + [VxWorks*], [PY_STDLIB_MOD_SET_NA([_scproxy], [_crypt], [termios], [grp])], + [Darwin], [PY_STDLIB_MOD_SET_NA([ossaudiodev], [spwd])], + [CYGWIN*], [PY_STDLIB_MOD_SET_NA([_scproxy], [nis])], + [QNX*], [PY_STDLIB_MOD_SET_NA([_scproxy], [nis])], + [FreeBSD*], [PY_STDLIB_MOD_SET_NA([_scproxy], [spwd])], + [Emscripten|WASI], [ + dnl subprocess and multiprocessing are not supported (no fork syscall). + dnl curses and tkinter user interface are not available. + dnl dbm and gdbm aren't available, too. + dnl Emscripten and WASI provide only stubs for pwd, grp APIs. PY_STDLIB_MOD_SET_NA( - [_ctypes], [_curses], [_curses_panel], [_dbm], @@ -6639,24 +6618,18 @@ AS_CASE([$ac_sys_system/$ac_sys_emscripten_target], [spwd], [syslog], ) - ], - [WASI/*], [ - PY_STDLIB_MOD_SET_NA( - [_ctypes], - [_ctypes_test], - [_curses], - [_curses_panel], - [_dbm], - [_gdbm], - [_scproxy], - [_tkinter], - [_xxsubinterpreters], - [grp], - [nis], - [ossaudiodev], - [pwd], - [spwd], - [syslog], + AS_CASE([$ac_sys_system/$ac_sys_emscripten_target], + [Emscripten/browser*], [ + dnl These modules are not particularly useful in browsers. + PY_STDLIB_MOD_SET_NA( + [fcntl], + [resource], + [readline], + [termios], + ) + ], + [Emscripten/node*], [], + [WASI/*], [] ) ], [PY_STDLIB_MOD_SET_NA([_scproxy])] From webhook-mailer at python.org Sat Apr 23 05:49:37 2022 From: webhook-mailer at python.org (serhiy-storchaka) Date: Sat, 23 Apr 2022 09:49:37 -0000 Subject: [Python-checkins] Add more tests for inline flag "x" and re.VERBOSE (GH-91854) Message-ID: https://github.com/python/cpython/commit/6b45076bd62407103433daea8acf085a99e6cb7e commit: 6b45076bd62407103433daea8acf085a99e6cb7e branch: main author: Serhiy Storchaka committer: serhiy-storchaka date: 2022-04-23T12:49:06+03:00 summary: RE: Add more tests for inline flag "x" and re.VERBOSE (GH-91854) files: M Lib/test/test_re.py diff --git a/Lib/test/test_re.py b/Lib/test/test_re.py index 700275063f0f1..91971d8126c98 100644 --- a/Lib/test/test_re.py +++ b/Lib/test/test_re.py @@ -1660,11 +1660,6 @@ def test_scoped_flags(self): self.assertIsNone(re.match(r'(?i:(?-i:a)b)', 'Ab')) self.assertTrue(re.match(r'(?i:(?-i:a)b)', 'aB')) - self.assertTrue(re.match(r'(?x: a) b', 'a b')) - self.assertIsNone(re.match(r'(?x: a) b', ' a b')) - self.assertTrue(re.match(r'(?-x: a) b', ' ab', re.VERBOSE)) - self.assertIsNone(re.match(r'(?-x: a) b', 'ab', re.VERBOSE)) - self.assertTrue(re.match(r'\w(?a:\W)\w', '\xe0\xe0\xe0')) self.assertTrue(re.match(r'(?a:\W(?u:\w)\W)', '\xe0\xe0\xe0')) self.assertTrue(re.match(r'\W(?u:\w)\W', '\xe0\xe0\xe0', re.ASCII)) @@ -1690,6 +1685,33 @@ def test_scoped_flags(self): self.checkPatternError(r'(?i+', 'missing -, : or )', 3) self.checkPatternError(r'(?iz', 'unknown flag', 3) + def test_ignore_spaces(self): + for space in " \t\n\r\v\f": + self.assertTrue(re.fullmatch(space + 'a', 'a', re.VERBOSE)) + for space in b" ", b"\t", b"\n", b"\r", b"\v", b"\f": + self.assertTrue(re.fullmatch(space + b'a', b'a', re.VERBOSE)) + self.assertTrue(re.fullmatch('(?x) a', 'a')) + self.assertTrue(re.fullmatch(' (?x) a', 'a', re.VERBOSE)) + self.assertTrue(re.fullmatch('(?x) (?x) a', 'a')) + self.assertTrue(re.fullmatch(' a(?x: b) c', ' ab c')) + self.assertTrue(re.fullmatch(' a(?-x: b) c', 'a bc', re.VERBOSE)) + self.assertTrue(re.fullmatch('(?x) a(?-x: b) c', 'a bc')) + self.assertTrue(re.fullmatch('(?x) a| b', 'a')) + self.assertTrue(re.fullmatch('(?x) a| b', 'b')) + + def test_comments(self): + self.assertTrue(re.fullmatch('#x\na', 'a', re.VERBOSE)) + self.assertTrue(re.fullmatch(b'#x\na', b'a', re.VERBOSE)) + self.assertTrue(re.fullmatch('(?x)#x\na', 'a')) + self.assertTrue(re.fullmatch('#x\n(?x)#y\na', 'a', re.VERBOSE)) + self.assertTrue(re.fullmatch('(?x)#x\n(?x)#y\na', 'a')) + self.assertTrue(re.fullmatch('#x\na(?x:#y\nb)#z\nc', '#x\nab#z\nc')) + self.assertTrue(re.fullmatch('#x\na(?-x:#y\nb)#z\nc', 'a#y\nbc', + re.VERBOSE)) + self.assertTrue(re.fullmatch('(?x)#x\na(?-x:#y\nb)#z\nc', 'a#y\nbc')) + self.assertTrue(re.fullmatch('(?x)#x\na|#y\nb', 'a')) + self.assertTrue(re.fullmatch('(?x)#x\na|#y\nb', 'b')) + def test_bug_6509(self): # Replacement strings of both types must parse properly. # all strings From webhook-mailer at python.org Sat Apr 23 05:50:46 2022 From: webhook-mailer at python.org (serhiy-storchaka) Date: Sat, 23 Apr 2022 09:50:46 -0000 Subject: [Python-checkins] gh-91308: Simplify parsing inline flag "x" (verbose) (GH-91855) Message-ID: https://github.com/python/cpython/commit/130a8c386bc1a8de49eadf784fa178869ce01f37 commit: 130a8c386bc1a8de49eadf784fa178869ce01f37 branch: main author: Serhiy Storchaka committer: serhiy-storchaka date: 2022-04-23T12:50:42+03:00 summary: gh-91308: Simplify parsing inline flag "x" (verbose) (GH-91855) files: M Lib/re/_parser.py diff --git a/Lib/re/_parser.py b/Lib/re/_parser.py index 60ec3e8ba8bd5..b864bf2b108a6 100644 --- a/Lib/re/_parser.py +++ b/Lib/re/_parser.py @@ -67,9 +67,6 @@ TYPE_FLAGS = SRE_FLAG_ASCII | SRE_FLAG_LOCALE | SRE_FLAG_UNICODE GLOBAL_FLAGS = SRE_FLAG_DEBUG -class Verbose(Exception): - pass - class State: # keeps track of state for parsing def __init__(self): @@ -448,6 +445,8 @@ def _parse_sub(source, state, verbose, nested): not nested and not items)) if not sourcematch("|"): break + if not nested: + verbose = state.flags & SRE_FLAG_VERBOSE if len(items) == 1: return items[0] @@ -826,8 +825,7 @@ def _parse(source, state, verbose, nested, first=False): raise source.error('global flags not at the start ' 'of the expression', source.tell() - start) - if (state.flags & SRE_FLAG_VERBOSE) and not verbose: - raise Verbose + verbose = state.flags & SRE_FLAG_VERBOSE continue add_flags, del_flags = flags @@ -963,17 +961,7 @@ def parse(str, flags=0, state=None): state.flags = flags state.str = str - try: - p = _parse_sub(source, state, flags & SRE_FLAG_VERBOSE, 0) - except Verbose: - # the VERBOSE flag was switched on inside the pattern. to be - # on the safe side, we'll parse the whole thing again... - state = State() - state.flags = flags | SRE_FLAG_VERBOSE - state.str = str - source.seek(0) - p = _parse_sub(source, state, True, 0) - + p = _parse_sub(source, state, flags & SRE_FLAG_VERBOSE, 0) p.state.flags = fix_flags(str, p.state.flags) if source.next is not None: From webhook-mailer at python.org Sat Apr 23 06:17:04 2022 From: webhook-mailer at python.org (miss-islington) Date: Sat, 23 Apr 2022 10:17:04 -0000 Subject: [Python-checkins] Add more tests for inline flag "x" and re.VERBOSE (GH-91854) Message-ID: https://github.com/python/cpython/commit/4165702ee89d2b90531f36e04374cfeb2016f1fd commit: 4165702ee89d2b90531f36e04374cfeb2016f1fd branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-23T03:16:50-07:00 summary: RE: Add more tests for inline flag "x" and re.VERBOSE (GH-91854) (cherry picked from commit 6b45076bd62407103433daea8acf085a99e6cb7e) Co-authored-by: Serhiy Storchaka files: M Lib/test/test_re.py diff --git a/Lib/test/test_re.py b/Lib/test/test_re.py index ca48b9b11f2a8..c779057bc1dd5 100644 --- a/Lib/test/test_re.py +++ b/Lib/test/test_re.py @@ -1682,11 +1682,6 @@ def test_scoped_flags(self): self.assertIsNone(re.match(r'(?i:(?-i:a)b)', 'Ab')) self.assertTrue(re.match(r'(?i:(?-i:a)b)', 'aB')) - self.assertTrue(re.match(r'(?x: a) b', 'a b')) - self.assertIsNone(re.match(r'(?x: a) b', ' a b')) - self.assertTrue(re.match(r'(?-x: a) b', ' ab', re.VERBOSE)) - self.assertIsNone(re.match(r'(?-x: a) b', 'ab', re.VERBOSE)) - self.assertTrue(re.match(r'\w(?a:\W)\w', '\xe0\xe0\xe0')) self.assertTrue(re.match(r'(?a:\W(?u:\w)\W)', '\xe0\xe0\xe0')) self.assertTrue(re.match(r'\W(?u:\w)\W', '\xe0\xe0\xe0', re.ASCII)) @@ -1712,6 +1707,33 @@ def test_scoped_flags(self): self.checkPatternError(r'(?i+', 'missing -, : or )', 3) self.checkPatternError(r'(?iz', 'unknown flag', 3) + def test_ignore_spaces(self): + for space in " \t\n\r\v\f": + self.assertTrue(re.fullmatch(space + 'a', 'a', re.VERBOSE)) + for space in b" ", b"\t", b"\n", b"\r", b"\v", b"\f": + self.assertTrue(re.fullmatch(space + b'a', b'a', re.VERBOSE)) + self.assertTrue(re.fullmatch('(?x) a', 'a')) + self.assertTrue(re.fullmatch(' (?x) a', 'a', re.VERBOSE)) + self.assertTrue(re.fullmatch('(?x) (?x) a', 'a')) + self.assertTrue(re.fullmatch(' a(?x: b) c', ' ab c')) + self.assertTrue(re.fullmatch(' a(?-x: b) c', 'a bc', re.VERBOSE)) + self.assertTrue(re.fullmatch('(?x) a(?-x: b) c', 'a bc')) + self.assertTrue(re.fullmatch('(?x) a| b', 'a')) + self.assertTrue(re.fullmatch('(?x) a| b', 'b')) + + def test_comments(self): + self.assertTrue(re.fullmatch('#x\na', 'a', re.VERBOSE)) + self.assertTrue(re.fullmatch(b'#x\na', b'a', re.VERBOSE)) + self.assertTrue(re.fullmatch('(?x)#x\na', 'a')) + self.assertTrue(re.fullmatch('#x\n(?x)#y\na', 'a', re.VERBOSE)) + self.assertTrue(re.fullmatch('(?x)#x\n(?x)#y\na', 'a')) + self.assertTrue(re.fullmatch('#x\na(?x:#y\nb)#z\nc', '#x\nab#z\nc')) + self.assertTrue(re.fullmatch('#x\na(?-x:#y\nb)#z\nc', 'a#y\nbc', + re.VERBOSE)) + self.assertTrue(re.fullmatch('(?x)#x\na(?-x:#y\nb)#z\nc', 'a#y\nbc')) + self.assertTrue(re.fullmatch('(?x)#x\na|#y\nb', 'a')) + self.assertTrue(re.fullmatch('(?x)#x\na|#y\nb', 'b')) + def test_bug_6509(self): # Replacement strings of both types must parse properly. # all strings From webhook-mailer at python.org Sat Apr 23 06:17:04 2022 From: webhook-mailer at python.org (miss-islington) Date: Sat, 23 Apr 2022 10:17:04 -0000 Subject: [Python-checkins] Add more tests for inline flag "x" and re.VERBOSE (GH-91854) Message-ID: https://github.com/python/cpython/commit/c7e6bfd1500588e5cede4590d3a721d67f915867 commit: c7e6bfd1500588e5cede4590d3a721d67f915867 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-23T03:16:36-07:00 summary: RE: Add more tests for inline flag "x" and re.VERBOSE (GH-91854) (cherry picked from commit 6b45076bd62407103433daea8acf085a99e6cb7e) Co-authored-by: Serhiy Storchaka files: M Lib/test/test_re.py diff --git a/Lib/test/test_re.py b/Lib/test/test_re.py index 3d6623b596ef9..62bfc3a7aa43b 100644 --- a/Lib/test/test_re.py +++ b/Lib/test/test_re.py @@ -1683,11 +1683,6 @@ def test_scoped_flags(self): self.assertIsNone(re.match(r'(?i:(?-i:a)b)', 'Ab')) self.assertTrue(re.match(r'(?i:(?-i:a)b)', 'aB')) - self.assertTrue(re.match(r'(?x: a) b', 'a b')) - self.assertIsNone(re.match(r'(?x: a) b', ' a b')) - self.assertTrue(re.match(r'(?-x: a) b', ' ab', re.VERBOSE)) - self.assertIsNone(re.match(r'(?-x: a) b', 'ab', re.VERBOSE)) - self.assertTrue(re.match(r'\w(?a:\W)\w', '\xe0\xe0\xe0')) self.assertTrue(re.match(r'(?a:\W(?u:\w)\W)', '\xe0\xe0\xe0')) self.assertTrue(re.match(r'\W(?u:\w)\W', '\xe0\xe0\xe0', re.ASCII)) @@ -1713,6 +1708,33 @@ def test_scoped_flags(self): self.checkPatternError(r'(?i+', 'missing -, : or )', 3) self.checkPatternError(r'(?iz', 'unknown flag', 3) + def test_ignore_spaces(self): + for space in " \t\n\r\v\f": + self.assertTrue(re.fullmatch(space + 'a', 'a', re.VERBOSE)) + for space in b" ", b"\t", b"\n", b"\r", b"\v", b"\f": + self.assertTrue(re.fullmatch(space + b'a', b'a', re.VERBOSE)) + self.assertTrue(re.fullmatch('(?x) a', 'a')) + self.assertTrue(re.fullmatch(' (?x) a', 'a', re.VERBOSE)) + self.assertTrue(re.fullmatch('(?x) (?x) a', 'a')) + self.assertTrue(re.fullmatch(' a(?x: b) c', ' ab c')) + self.assertTrue(re.fullmatch(' a(?-x: b) c', 'a bc', re.VERBOSE)) + self.assertTrue(re.fullmatch('(?x) a(?-x: b) c', 'a bc')) + self.assertTrue(re.fullmatch('(?x) a| b', 'a')) + self.assertTrue(re.fullmatch('(?x) a| b', 'b')) + + def test_comments(self): + self.assertTrue(re.fullmatch('#x\na', 'a', re.VERBOSE)) + self.assertTrue(re.fullmatch(b'#x\na', b'a', re.VERBOSE)) + self.assertTrue(re.fullmatch('(?x)#x\na', 'a')) + self.assertTrue(re.fullmatch('#x\n(?x)#y\na', 'a', re.VERBOSE)) + self.assertTrue(re.fullmatch('(?x)#x\n(?x)#y\na', 'a')) + self.assertTrue(re.fullmatch('#x\na(?x:#y\nb)#z\nc', '#x\nab#z\nc')) + self.assertTrue(re.fullmatch('#x\na(?-x:#y\nb)#z\nc', 'a#y\nbc', + re.VERBOSE)) + self.assertTrue(re.fullmatch('(?x)#x\na(?-x:#y\nb)#z\nc', 'a#y\nbc')) + self.assertTrue(re.fullmatch('(?x)#x\na|#y\nb', 'a')) + self.assertTrue(re.fullmatch('(?x)#x\na|#y\nb', 'b')) + def test_bug_6509(self): # Replacement strings of both types must parse properly. # all strings From webhook-mailer at python.org Sat Apr 23 08:59:51 2022 From: webhook-mailer at python.org (tiran) Date: Sat, 23 Apr 2022 12:59:51 -0000 Subject: [Python-checkins] gh-84461: Add --enable-wasm-pthreads and more file systems (GH-91820) Message-ID: https://github.com/python/cpython/commit/92c1037afc28d9d22e43b275c5e8fae41729ec1c commit: 92c1037afc28d9d22e43b275c5e8fae41729ec1c branch: main author: Christian Heimes committer: tiran date: 2022-04-23T14:59:33+02:00 summary: gh-84461: Add --enable-wasm-pthreads and more file systems (GH-91820) files: A Misc/NEWS.d/next/Library/2022-04-22-13-01-20.gh-issue-84461.rsCiTH.rst M Doc/using/configure.rst M Tools/wasm/README.md M configure M configure.ac diff --git a/Doc/using/configure.rst b/Doc/using/configure.rst index e7efd2bbbc0a5..057efa3bd077c 100644 --- a/Doc/using/configure.rst +++ b/Doc/using/configure.rst @@ -164,6 +164,12 @@ WebAssemby Options .. versionadded:: 3.11 +.. cmdoption:: --enable-wasm-pthreads + + Turn on pthreads support for WASM. + + .. versionadded:: 3.11 + Install Options --------------- diff --git a/Misc/NEWS.d/next/Library/2022-04-22-13-01-20.gh-issue-84461.rsCiTH.rst b/Misc/NEWS.d/next/Library/2022-04-22-13-01-20.gh-issue-84461.rsCiTH.rst new file mode 100644 index 0000000000000..f1b8dc85ba82e --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-22-13-01-20.gh-issue-84461.rsCiTH.rst @@ -0,0 +1,3 @@ +Add :option:`--enable-wasm-pthreads` to enable pthreads support for WASM +builds. ``Emscripten/node`` no longer has threading enabled by default. +Include additional file systems. diff --git a/Tools/wasm/README.md b/Tools/wasm/README.md index 9d3680a53d28e..83806f0581ace 100644 --- a/Tools/wasm/README.md +++ b/Tools/wasm/README.md @@ -121,18 +121,27 @@ functions. - The ``select`` module is limited. ``select.select()`` crashes the runtime due to lack of exectfd support. -## processes, threads, signals +## processes, signals - Processes are not supported. System calls like fork, popen, and subprocess fail with ``ENOSYS`` or ``ENOSUP``. - Signal support is limited. ``signal.alarm``, ``itimer``, ``sigaction`` are not available or do not work correctly. ``SIGTERM`` exits the runtime. - Keyboard interrupt (CTRL+C) handling is not implemented yet. -- Browser builds cannot start new threads. Node's web workers consume - extra file descriptors. - Resource-related functions like ``os.nice`` and most functions of the ``resource`` module are not available. +## threading + +- Threading is disabled by default. The ``configure`` option + ``--enable-wasm-pthreads`` adds compiler flag ``-pthread`` and + linker flags ``-sUSE_PTHREADS -sPROXY_TO_PTHREAD``. +- pthread support requires WASM threads and SharedArrayBuffer (bulk memory). + The Node.JS runtime keeps a pool of web workers around. Each web worker + uses several file descriptors (eventfd, epoll, pipe). +- It's not advised to enable threading when building for browsers or with + dynamic linking support; there are performance and stability issues. + ## file system - Most user, group, and permission related function and modules are not @@ -173,20 +182,16 @@ functions. distutils, multiprocessing, dbm, tests and similar modules are not shipped. All other modules are bundled as pre-compiled ``pyc`` files. -- Threading is disabled. - In-memory file system (MEMFS) is not persistent and limited. - Test modules are disabled by default. Use ``--enable-test-modules`` build test modules like ``_testcapi``. ## wasm32-emscripten in node -Node builds use ``NODERAWFS``, ``USE_PTHREADS`` and ``PROXY_TO_PTHREAD`` -linker options. +Node builds use ``NODERAWFS``. -- Node RawFS allows direct access to the host file system. -- pthread support requires WASM threads and SharedArrayBuffer (bulk memory). - The runtime keeps a pool of web workers around. Each web worker uses - several file descriptors (eventfd, epoll, pipe). +- Node RawFS allows direct access to the host file system without need to + perform ``FS.mount()`` call. # Hosting Python WASM builds diff --git a/configure b/configure index 25e767be82fa4..f161e4ebed94b 100755 --- a/configure +++ b/configure @@ -1018,6 +1018,7 @@ enable_framework with_cxx_main with_emscripten_target enable_wasm_dynamic_linking +enable_wasm_pthreads with_suffix enable_shared with_static_libpython @@ -1734,6 +1735,8 @@ Optional Features: --enable-wasm-dynamic-linking Enable dynamic linking support for WebAssembly (default is no) + --enable-wasm-pthreads Enable pthread emulation for WebAssembly (default is + no) --enable-shared enable building a shared Python library (default is no) --enable-profiling enable C-level code profiling with gprof (default is @@ -6318,6 +6321,30 @@ fi { $as_echo "$as_me:${as_lineno-$LINENO}: result: $enable_wasm_dynamic_linking" >&5 $as_echo "$enable_wasm_dynamic_linking" >&6; } +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for --enable-wasm-pthreads" >&5 +$as_echo_n "checking for --enable-wasm-pthreads... " >&6; } +# Check whether --enable-wasm-pthreads was given. +if test "${enable_wasm_pthreads+set}" = set; then : + enableval=$enable_wasm_pthreads; + case $ac_sys_system in #( + Emscripten) : + ;; #( + WASI) : + as_fn_error $? "WASI threading is not implemented yet." "$LINENO" 5 ;; #( + *) : + as_fn_error $? "--enable-wasm-pthreads only applies to Emscripten and WASI" "$LINENO" 5 + ;; +esac + +else + + enable_wasm_pthreads=missing + +fi + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $enable_wasm_pthreads" >&5 +$as_echo "$enable_wasm_pthreads" >&6; } + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for --with-suffix" >&5 $as_echo_n "checking for --with-suffix... " >&6; } @@ -7788,43 +7815,70 @@ then fi # WASM flags -case $ac_sys_system/$ac_sys_emscripten_target in #( - Emscripten/browser*) : +case $ac_sys_system in #( + Emscripten) : + + if test "x$Py_DEBUG" = xyes; then : + wasm_debug=yes +else + wasm_debug=no +fi + + as_fn_append LDFLAGS_NODIST " -sALLOW_MEMORY_GROWTH -sTOTAL_MEMORY=20971520" + + as_fn_append LDFLAGS_NODIST " -sFORCE_FILESYSTEM -lidbfs.js -lnodefs.js -lproxyfs.js -lworkerfs.js" - LDFLAGS_NODIST="$LDFLAGS_NODIST -sALLOW_MEMORY_GROWTH" - LINKFORSHARED="--preload-file=\$(WASM_ASSETS_DIR)" if test "x$enable_wasm_dynamic_linking" = xyes; then : as_fn_append LINKFORSHARED " -sMAIN_MODULE" fi - WASM_ASSETS_DIR=".\$(prefix)" - WASM_STDLIB="\$(WASM_ASSETS_DIR)/local/lib/python\$(VERSION)/os.py" - if test "$Py_DEBUG" = 'true' -o "$ac_sys_emscripten_target" = "browser-debug"; then - LDFLAGS_NODIST="$LDFLAGS_NODIST -sASSERTIONS" - LINKFORSHARED="$LINKFORSHARED -gsource-map --emit-symbol-map" - else - LINKFORSHARED="$LINKFORSHARED -O2 -g0" - fi - ;; #( - Emscripten/node*) : - LDFLAGS_NODIST="$LDFLAGS_NODIST -sALLOW_MEMORY_GROWTH -sNODERAWFS -sUSE_PTHREADS" - LINKFORSHARED="-sPROXY_TO_PTHREAD -sEXIT_RUNTIME" - if test "x$enable_wasm_dynamic_linking" = xyes; then : + if test "x$enable_wasm_pthreads" = xyes; then : - as_fn_append LINKFORSHARED " -sMAIN_MODULE" + as_fn_append CFLAGS_NODIST " -pthread" + as_fn_append LDFLAGS_NODIST " -sUSE_PTHREADS" + as_fn_append LINKFORSHARED " -sPROXY_TO_PTHREAD" + +fi + + case $ac_sys_emscripten_target in #( + browser*) : + + if test "x$ac_sys_emscripten_target" = xbrowser-debug; then : + wasm_debug=yes +fi + as_fn_append LINKFORSHARED " --preload-file=\$(WASM_ASSETS_DIR)" + WASM_ASSETS_DIR=".\$(prefix)" + WASM_STDLIB="\$(WASM_ASSETS_DIR)/local/lib/python\$(VERSION)/os.py" + WASM_LINKFORSHARED_DEBUG="-gsource-map --emit-symbol-map" + ;; #( + node*) : + + if test "x$ac_sys_emscripten_target" = xnode-debug; then : + wasm_debug=yes +fi + as_fn_append LDFLAGS_NODIST " -sALLOW_MEMORY_GROWTH -sNODERAWFS" + as_fn_append LINKFORSHARED " -sEXIT_RUNTIME" + WASM_LINKFORSHARED_DEBUG="-gseparate-dwarf --emit-symbol-map" + + ;; #( + *) : + ;; +esac + + if test "x$wasm_debug" = xyes; then : + + as_fn_append LDFLAGS_NODIST " -sASSERTIONS" + as_fn_append LINKFORSHARED " $WASM_LINKFORSHARED_DEBUG" + +else + + as_fn_append LINKFORSHARED " -O2 -g0" fi - CFLAGS_NODIST="$CFLAGS_NODIST -pthread" - if test "$Py_DEBUG" = 'true' -o "$ac_sys_emscripten_target" = "node-debug"; then - LDFLAGS_NODIST="$LDFLAGS_NODIST -sASSERTIONS" - LINKFORSHARED="$LINKFORSHARED -gseparate-dwarf --emit-symbol-map" - else - LINKFORSHARED="$LINKFORSHARED -O2 -g0" - fi ;; #( - WASI/*) : + WASI) : $as_echo "#define _WASI_EMULATED_SIGNAL 1" >>confdefs.h diff --git a/configure.ac b/configure.ac index 82211f4ee1d68..86839547e2683 100644 --- a/configure.ac +++ b/configure.ac @@ -1126,6 +1126,21 @@ AC_ARG_ENABLE([wasm-dynamic-linking], ]) AC_MSG_RESULT([$enable_wasm_dynamic_linking]) +AC_MSG_CHECKING([for --enable-wasm-pthreads]) +AC_ARG_ENABLE([wasm-pthreads], + [AS_HELP_STRING([--enable-wasm-pthreads], + [Enable pthread emulation for WebAssembly (default is no)])], +[ + AS_CASE([$ac_sys_system], + [Emscripten], [], + [WASI], [AC_MSG_ERROR([WASI threading is not implemented yet.])], + [AC_MSG_ERROR([--enable-wasm-pthreads only applies to Emscripten and WASI])] + ) +], [ + enable_wasm_pthreads=missing +]) +AC_MSG_RESULT([$enable_wasm_pthreads]) + AC_MSG_CHECKING([for --with-suffix]) AC_ARG_WITH([suffix], [AS_HELP_STRING([--with-suffix=SUFFIX], [set executable suffix to SUFFIX (default is empty, yes is mapped to '.exe')])], @@ -1909,38 +1924,53 @@ then fi # WASM flags -AS_CASE([$ac_sys_system/$ac_sys_emscripten_target], - [Emscripten/browser*], [ - LDFLAGS_NODIST="$LDFLAGS_NODIST -sALLOW_MEMORY_GROWTH" - LINKFORSHARED="--preload-file=\$(WASM_ASSETS_DIR)" +AS_CASE([$ac_sys_system], + [Emscripten], [ + dnl build with WASM debug info if either Py_DEBUG is set or the target is + dnl node-debug or browser-debug. + AS_VAR_IF([Py_DEBUG], [yes], [wasm_debug=yes], [wasm_debug=no]) + + dnl Start with 20 MB and allow to grow + AS_VAR_APPEND([LDFLAGS_NODIST], [" -sALLOW_MEMORY_GROWTH -sTOTAL_MEMORY=20971520"]) + + dnl Include file system support + AS_VAR_APPEND([LDFLAGS_NODIST], [" -sFORCE_FILESYSTEM -lidbfs.js -lnodefs.js -lproxyfs.js -lworkerfs.js"]) + AS_VAR_IF([enable_wasm_dynamic_linking], [yes], [ AS_VAR_APPEND([LINKFORSHARED], [" -sMAIN_MODULE"]) ]) - WASM_ASSETS_DIR=".\$(prefix)" - WASM_STDLIB="\$(WASM_ASSETS_DIR)/local/lib/python\$(VERSION)/os.py" - dnl separate-dwarf does not seem to work in Chrome DevTools Support. - if test "$Py_DEBUG" = 'true' -o "$ac_sys_emscripten_target" = "browser-debug"; then - LDFLAGS_NODIST="$LDFLAGS_NODIST -sASSERTIONS" - LINKFORSHARED="$LINKFORSHARED -gsource-map --emit-symbol-map" - else - LINKFORSHARED="$LINKFORSHARED -O2 -g0" - fi - ], - [Emscripten/node*], [ - LDFLAGS_NODIST="$LDFLAGS_NODIST -sALLOW_MEMORY_GROWTH -sNODERAWFS -sUSE_PTHREADS" - LINKFORSHARED="-sPROXY_TO_PTHREAD -sEXIT_RUNTIME" - AS_VAR_IF([enable_wasm_dynamic_linking], [yes], [ - AS_VAR_APPEND([LINKFORSHARED], [" -sMAIN_MODULE"]) + + AS_VAR_IF([enable_wasm_pthreads], [yes], [ + AS_VAR_APPEND([CFLAGS_NODIST], [" -pthread"]) + AS_VAR_APPEND([LDFLAGS_NODIST], [" -sUSE_PTHREADS"]) + AS_VAR_APPEND([LINKFORSHARED], [" -sPROXY_TO_PTHREAD"]) + ]) + + AS_CASE([$ac_sys_emscripten_target], + [browser*], [ + AS_VAR_IF([ac_sys_emscripten_target], [browser-debug], [wasm_debug=yes]) + AS_VAR_APPEND([LINKFORSHARED], [" --preload-file=\$(WASM_ASSETS_DIR)"]) + WASM_ASSETS_DIR=".\$(prefix)" + WASM_STDLIB="\$(WASM_ASSETS_DIR)/local/lib/python\$(VERSION)/os.py" + dnl separate-dwarf does not seem to work in Chrome DevTools Support. + WASM_LINKFORSHARED_DEBUG="-gsource-map --emit-symbol-map" + ], + [node*], [ + AS_VAR_IF([ac_sys_emscripten_target], [node-debug], [wasm_debug=yes]) + AS_VAR_APPEND([LDFLAGS_NODIST], [" -sALLOW_MEMORY_GROWTH -sNODERAWFS"]) + AS_VAR_APPEND([LINKFORSHARED], [" -sEXIT_RUNTIME"]) + WASM_LINKFORSHARED_DEBUG="-gseparate-dwarf --emit-symbol-map" + ] + ) + + AS_VAR_IF([wasm_debug], [yes], [ + AS_VAR_APPEND([LDFLAGS_NODIST], [" -sASSERTIONS"]) + AS_VAR_APPEND([LINKFORSHARED], [" $WASM_LINKFORSHARED_DEBUG"]) + ], [ + AS_VAR_APPEND([LINKFORSHARED], [" -O2 -g0"]) ]) - CFLAGS_NODIST="$CFLAGS_NODIST -pthread" - if test "$Py_DEBUG" = 'true' -o "$ac_sys_emscripten_target" = "node-debug"; then - LDFLAGS_NODIST="$LDFLAGS_NODIST -sASSERTIONS" - LINKFORSHARED="$LINKFORSHARED -gseparate-dwarf --emit-symbol-map" - else - LINKFORSHARED="$LINKFORSHARED -O2 -g0" - fi ], - [WASI/*], [ + [WASI], [ AC_DEFINE([_WASI_EMULATED_SIGNAL], [1], [Define to 1 if you want to emulate signals on WASI]) AC_DEFINE([_WASI_EMULATED_GETPID], [1], [Define to 1 if you want to emulate getpid() on WASI]) AC_DEFINE([_WASI_EMULATED_PROCESS_CLOCKS], [1], [Define to 1 if you want to emulate process clocks on WASI]) From webhook-mailer at python.org Sat Apr 23 11:49:27 2022 From: webhook-mailer at python.org (serhiy-storchaka) Date: Sat, 23 Apr 2022 15:49:27 -0000 Subject: [Python-checkins] Pre-split the list of opcode names (GH-91859) Message-ID: https://github.com/python/cpython/commit/28890427c58d30f1041b36859733159475c67496 commit: 28890427c58d30f1041b36859733159475c67496 branch: main author: Serhiy Storchaka committer: serhiy-storchaka date: 2022-04-23T18:49:23+03:00 summary: RE: Pre-split the list of opcode names (GH-91859) 1. It makes them interned. 2. It allows to add comments to individual opcodes. files: M Lib/re/_constants.py diff --git a/Lib/re/_constants.py b/Lib/re/_constants.py index 5e999dea337d3..aa1a590290fd9 100644 --- a/Lib/re/_constants.py +++ b/Lib/re/_constants.py @@ -64,88 +64,89 @@ def __repr__(self): MAXREPEAT = _NamedIntConstant(MAXREPEAT, 'MAXREPEAT') -def _makecodes(names): - names = names.strip().split() +def _makecodes(*names): items = [_NamedIntConstant(i, name) for i, name in enumerate(names)] globals().update({item.name: item for item in items}) return items # operators -# failure=0 success=1 (just because it looks better that way :-) -OPCODES = _makecodes(""" - FAILURE SUCCESS - - ANY ANY_ALL - ASSERT ASSERT_NOT - AT - BRANCH - CALL - CATEGORY - CHARSET BIGCHARSET - GROUPREF GROUPREF_EXISTS - IN - INFO - JUMP - LITERAL - MARK - MAX_UNTIL - MIN_UNTIL - NOT_LITERAL - NEGATE - RANGE - REPEAT - REPEAT_ONE - SUBPATTERN - MIN_REPEAT_ONE - ATOMIC_GROUP - POSSESSIVE_REPEAT - POSSESSIVE_REPEAT_ONE - - GROUPREF_IGNORE - IN_IGNORE - LITERAL_IGNORE - NOT_LITERAL_IGNORE - - GROUPREF_LOC_IGNORE - IN_LOC_IGNORE - LITERAL_LOC_IGNORE - NOT_LITERAL_LOC_IGNORE - - GROUPREF_UNI_IGNORE - IN_UNI_IGNORE - LITERAL_UNI_IGNORE - NOT_LITERAL_UNI_IGNORE - RANGE_UNI_IGNORE - - MIN_REPEAT MAX_REPEAT -""") +OPCODES = _makecodes( + # failure=0 success=1 (just because it looks better that way :-) + 'FAILURE', 'SUCCESS', + + 'ANY', 'ANY_ALL', + 'ASSERT', 'ASSERT_NOT', + 'AT', + 'BRANCH', + 'CALL', + 'CATEGORY', + 'CHARSET', 'BIGCHARSET', + 'GROUPREF', 'GROUPREF_EXISTS', + 'IN', + 'INFO', + 'JUMP', + 'LITERAL', + 'MARK', + 'MAX_UNTIL', + 'MIN_UNTIL', + 'NOT_LITERAL', + 'NEGATE', + 'RANGE', + 'REPEAT', + 'REPEAT_ONE', + 'SUBPATTERN', + 'MIN_REPEAT_ONE', + 'ATOMIC_GROUP', + 'POSSESSIVE_REPEAT', + 'POSSESSIVE_REPEAT_ONE', + + 'GROUPREF_IGNORE', + 'IN_IGNORE', + 'LITERAL_IGNORE', + 'NOT_LITERAL_IGNORE', + + 'GROUPREF_LOC_IGNORE', + 'IN_LOC_IGNORE', + 'LITERAL_LOC_IGNORE', + 'NOT_LITERAL_LOC_IGNORE', + + 'GROUPREF_UNI_IGNORE', + 'IN_UNI_IGNORE', + 'LITERAL_UNI_IGNORE', + 'NOT_LITERAL_UNI_IGNORE', + 'RANGE_UNI_IGNORE', + + # The following opcodes are only occurred in the parser output, + # but not in the compiled code. + 'MIN_REPEAT', 'MAX_REPEAT', +) del OPCODES[-2:] # remove MIN_REPEAT and MAX_REPEAT # positions -ATCODES = _makecodes(""" - AT_BEGINNING AT_BEGINNING_LINE AT_BEGINNING_STRING - AT_BOUNDARY AT_NON_BOUNDARY - AT_END AT_END_LINE AT_END_STRING +ATCODES = _makecodes( + 'AT_BEGINNING', 'AT_BEGINNING_LINE', 'AT_BEGINNING_STRING', + 'AT_BOUNDARY', 'AT_NON_BOUNDARY', + 'AT_END', 'AT_END_LINE', 'AT_END_STRING', - AT_LOC_BOUNDARY AT_LOC_NON_BOUNDARY + 'AT_LOC_BOUNDARY', 'AT_LOC_NON_BOUNDARY', - AT_UNI_BOUNDARY AT_UNI_NON_BOUNDARY -""") + 'AT_UNI_BOUNDARY', 'AT_UNI_NON_BOUNDARY', +) # categories -CHCODES = _makecodes(""" - CATEGORY_DIGIT CATEGORY_NOT_DIGIT - CATEGORY_SPACE CATEGORY_NOT_SPACE - CATEGORY_WORD CATEGORY_NOT_WORD - CATEGORY_LINEBREAK CATEGORY_NOT_LINEBREAK - - CATEGORY_LOC_WORD CATEGORY_LOC_NOT_WORD - - CATEGORY_UNI_DIGIT CATEGORY_UNI_NOT_DIGIT - CATEGORY_UNI_SPACE CATEGORY_UNI_NOT_SPACE - CATEGORY_UNI_WORD CATEGORY_UNI_NOT_WORD - CATEGORY_UNI_LINEBREAK CATEGORY_UNI_NOT_LINEBREAK -""") +CHCODES = _makecodes( + 'CATEGORY_DIGIT', 'CATEGORY_NOT_DIGIT', + 'CATEGORY_SPACE', 'CATEGORY_NOT_SPACE', + 'CATEGORY_WORD', 'CATEGORY_NOT_WORD', + 'CATEGORY_LINEBREAK', 'CATEGORY_NOT_LINEBREAK', + + 'CATEGORY_LOC_WORD', 'CATEGORY_LOC_NOT_WORD', + + 'CATEGORY_UNI_DIGIT', 'CATEGORY_UNI_NOT_DIGIT', + 'CATEGORY_UNI_SPACE', 'CATEGORY_UNI_NOT_SPACE', + 'CATEGORY_UNI_WORD', 'CATEGORY_UNI_NOT_WORD', + 'CATEGORY_UNI_LINEBREAK', 'CATEGORY_UNI_NOT_LINEBREAK', +) # replacement operations for "ignore case" mode From webhook-mailer at python.org Sat Apr 23 13:30:04 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Sat, 23 Apr 2022 17:30:04 -0000 Subject: [Python-checkins] [3.9] gh-85864: Mark positional-only args in io docs (GH-91683). (#91848) Message-ID: https://github.com/python/cpython/commit/e4f5bff14018ac79409f57c7a9ecbe9b1ffd4975 commit: e4f5bff14018ac79409f57c7a9ecbe9b1ffd4975 branch: 3.9 author: Jelle Zijlstra committer: JelleZijlstra date: 2022-04-23T10:29:45-07:00 summary: [3.9] gh-85864: Mark positional-only args in io docs (GH-91683). (#91848) (cherry picked from commit a3f2cf3ced378db2569df4e7389ec1f79c85d55c) Co-authored-by: slateny <46876382+slateny at users.noreply.github.com> files: M Doc/library/io.rst diff --git a/Doc/library/io.rst b/Doc/library/io.rst index 9f62dd21afbea..28ad8dfd13126 100644 --- a/Doc/library/io.rst +++ b/Doc/library/io.rst @@ -299,7 +299,7 @@ I/O Base Classes Return ``True`` if the stream can be read from. If ``False``, :meth:`read` will raise :exc:`OSError`. - .. method:: readline(size=-1) + .. method:: readline(size=-1, /) Read and return one line from the stream. If *size* is specified, at most *size* bytes will be read. @@ -308,7 +308,7 @@ I/O Base Classes the *newline* argument to :func:`open` can be used to select the line terminator(s) recognized. - .. method:: readlines(hint=-1) + .. method:: readlines(hint=-1, /) Read and return a list of lines from the stream. *hint* can be specified to control the number of lines read: no more lines will be read if the @@ -394,7 +394,7 @@ I/O Base Classes :class:`RawIOBase` provides these methods in addition to those from :class:`IOBase`: - .. method:: read(size=-1) + .. method:: read(size=-1, /) Read up to *size* bytes from the object and return them. As a convenience, if *size* is unspecified or -1, all bytes until EOF are returned. @@ -504,7 +504,7 @@ I/O Base Classes If *size* is ``-1`` (the default), an arbitrary number of bytes are returned (more than zero unless EOF is reached). - .. method:: readinto(b) + .. method:: readinto(b, /) Read bytes into a pre-allocated, writable :term:`bytes-like object` *b* and return the number of bytes read. @@ -516,7 +516,7 @@ I/O Base Classes A :exc:`BlockingIOError` is raised if the underlying raw stream is in non blocking-mode, and has no data available at the moment. - .. method:: readinto1(b) + .. method:: readinto1(b, /) Read bytes into a pre-allocated, writable :term:`bytes-like object` *b*, using at most one call to @@ -646,7 +646,7 @@ than raw I/O does. Return :class:`bytes` containing the entire contents of the buffer. - .. method:: read1([size]) + .. method:: read1([size], /) In :class:`BytesIO`, this is the same as :meth:`~BufferedIOBase.read`. @@ -724,7 +724,7 @@ than raw I/O does. Force bytes held in the buffer into the raw stream. A :exc:`BlockingIOError` should be raised if the raw stream blocks. - .. method:: write(b) + .. method:: write(b, /) Write the :term:`bytes-like object`, *b*, and return the number of bytes written. When in non-blocking mode, a @@ -747,7 +747,7 @@ than raw I/O does. are guaranteed to be implemented. -.. class:: BufferedRWPair(reader, writer, buffer_size=DEFAULT_BUFFER_SIZE) +.. class:: BufferedRWPair(reader, writer, buffer_size=DEFAULT_BUFFER_SIZE, /) A buffered binary stream providing higher-level access to two non seekable :class:`RawIOBase` raw binary streams---one readable, the other writeable. @@ -814,7 +814,7 @@ Text I/O .. versionadded:: 3.1 - .. method:: read(size=-1) + .. method:: read(size=-1, /) Read and return at most *size* characters from the stream as a single :class:`str`. If *size* is negative or ``None``, reads until EOF. From webhook-mailer at python.org Sat Apr 23 17:48:21 2022 From: webhook-mailer at python.org (brettcannon) Date: Sat, 23 Apr 2022 21:48:21 -0000 Subject: [Python-checkins] gh-91217: deprecate spwd (#91846) Message-ID: https://github.com/python/cpython/commit/692e9078a10b268530f8da7d3095cfb05c48435b commit: 692e9078a10b268530f8da7d3095cfb05c48435b branch: main author: Brett Cannon committer: brettcannon date: 2022-04-23T14:48:17-07:00 summary: gh-91217: deprecate spwd (#91846) Co-authored-by: Hugo van Kemenade files: A Misc/NEWS.d/next/Library/2022-04-17-12-32-40.gh-issue-91217.ms49Rg.rst M Doc/whatsnew/3.11.rst M Lib/test/test_spwd.py M Modules/spwdmodule.c diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index 5b53fbe0dda6f..34742515168c3 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -929,6 +929,7 @@ Deprecated * :mod:`ossaudiodev` * :mod:`pipes` * :mod:`sndhdr` + * :mod:`spwd` (Contributed by Brett Cannon in :issue:`47061`.) diff --git a/Lib/test/test_spwd.py b/Lib/test/test_spwd.py index a143acc659ef6..50766c25482a6 100644 --- a/Lib/test/test_spwd.py +++ b/Lib/test/test_spwd.py @@ -1,9 +1,12 @@ import os import unittest from test.support import import_helper +import warnings -spwd = import_helper.import_module('spwd') +with warnings.catch_warnings(): + warnings.simplefilter("ignore", DeprecationWarning) + spwd = import_helper.import_module('spwd') @unittest.skipUnless(hasattr(os, 'geteuid') and os.geteuid() == 0, diff --git a/Misc/NEWS.d/next/Library/2022-04-17-12-32-40.gh-issue-91217.ms49Rg.rst b/Misc/NEWS.d/next/Library/2022-04-17-12-32-40.gh-issue-91217.ms49Rg.rst new file mode 100644 index 0000000000000..9655522d0eb46 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-17-12-32-40.gh-issue-91217.ms49Rg.rst @@ -0,0 +1 @@ +Deprecate the spwd module. diff --git a/Modules/spwdmodule.c b/Modules/spwdmodule.c index acea30679bf5e..42123c93b5936 100644 --- a/Modules/spwdmodule.c +++ b/Modules/spwdmodule.c @@ -256,5 +256,12 @@ static struct PyModuleDef spwdmodule = { PyMODINIT_FUNC PyInit_spwd(void) { + if (PyErr_WarnEx(PyExc_DeprecationWarning, + "'spwd' is deprecated and slated for removal in " + "Python 3.13", + 7)) { + return NULL; + } + return PyModuleDef_Init(&spwdmodule); } From webhook-mailer at python.org Sat Apr 23 20:55:33 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Sun, 24 Apr 2022 00:55:33 -0000 Subject: [Python-checkins] gh-91230: Concise catch_warnings with simplefilter (#91435) Message-ID: https://github.com/python/cpython/commit/b4e048411f4c62ad7343bca32c307f0bf5ef74b4 commit: b4e048411f4c62ad7343bca32c307f0bf5ef74b4 branch: main author: Zac Hatfield-Dodds committer: JelleZijlstra date: 2022-04-23T17:55:22-07:00 summary: gh-91230: Concise catch_warnings with simplefilter (#91435) files: A Misc/NEWS.d/next/Library/2022-04-10-17-12-23.gh-issue-91230.T1d_fG.rst M Doc/library/warnings.rst M Doc/whatsnew/3.11.rst M Lib/test/test_warnings/__init__.py M Lib/warnings.py diff --git a/Doc/library/warnings.rst b/Doc/library/warnings.rst index 289b28229e1a0..f7a1f70833b7f 100644 --- a/Doc/library/warnings.rst +++ b/Doc/library/warnings.rst @@ -491,7 +491,7 @@ Available Functions Available Context Managers -------------------------- -.. class:: catch_warnings(*, record=False, module=None) +.. class:: catch_warnings(*, record=False, module=None, action=None, category=Warning, lineno=0, append=False) A context manager that copies and, upon exit, restores the warnings filter and the :func:`showwarning` function. @@ -507,6 +507,10 @@ Available Context Managers protected. This argument exists primarily for testing the :mod:`warnings` module itself. + If the *action* argument is not ``None``, the remaining arguments are + passed to :func:`simplefilter` as if it were called immediately on + entering the context. + .. note:: The :class:`catch_warnings` manager works by replacing and @@ -514,3 +518,7 @@ Available Context Managers :func:`showwarning` function and internal list of filter specifications. This means the context manager is modifying global state and therefore is not thread-safe. + + .. versionchanged:: 3.11 + + Added the *action*, *category*, *lineno*, and *append* parameters. diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index 34742515168c3..43d8e6b1cbf41 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -529,6 +529,13 @@ venv Third party code that also creates new virtual environments should do the same. (Contributed by Miro Hron?ok in :issue:`45413`.) +warnings +-------- + +* :func:`warnings.catch_warnings` now accepts arguments for :func:`warnings.simplefilter`, + providing a more concise way to locally ignore warnings or convert them to errors. + (Contributed by Zac Hatfield-Dodds in :issue:`47074`.) + zipfile ------- diff --git a/Lib/test/test_warnings/__init__.py b/Lib/test/test_warnings/__init__.py index f7f931130714c..0f960b82bfaeb 100644 --- a/Lib/test/test_warnings/__init__.py +++ b/Lib/test/test_warnings/__init__.py @@ -373,6 +373,25 @@ def test_append_duplicate(self): "appended duplicate changed order of filters" ) + def test_catchwarnings_with_simplefilter_ignore(self): + with original_warnings.catch_warnings(module=self.module): + self.module.resetwarnings() + self.module.simplefilter("error") + with self.module.catch_warnings( + module=self.module, action="ignore" + ): + self.module.warn("This will be ignored") + + def test_catchwarnings_with_simplefilter_error(self): + with original_warnings.catch_warnings(module=self.module): + self.module.resetwarnings() + with self.module.catch_warnings( + module=self.module, action="error", category=FutureWarning + ): + self.module.warn("Other types of warnings are not errors") + self.assertRaises(FutureWarning, + self.module.warn, FutureWarning("msg")) + class CFilterTests(FilterTests, unittest.TestCase): module = c_warnings diff --git a/Lib/warnings.py b/Lib/warnings.py index 887ca6ecd1a72..7d8c4400127f7 100644 --- a/Lib/warnings.py +++ b/Lib/warnings.py @@ -432,9 +432,13 @@ class catch_warnings(object): named 'warnings' and imported under that name. This argument is only useful when testing the warnings module itself. + If the 'action' argument is not None, the remaining arguments are passed + to warnings.simplefilter() as if it were called immediately on entering the + context. """ - def __init__(self, *, record=False, module=None): + def __init__(self, *, record=False, module=None, + action=None, category=Warning, lineno=0, append=False): """Specify whether to record warnings and if an alternative module should be used other than sys.modules['warnings']. @@ -445,6 +449,10 @@ def __init__(self, *, record=False, module=None): self._record = record self._module = sys.modules['warnings'] if module is None else module self._entered = False + if action is None: + self._filter = None + else: + self._filter = (action, category, lineno, append) def __repr__(self): args = [] @@ -464,6 +472,8 @@ def __enter__(self): self._module._filters_mutated() self._showwarning = self._module.showwarning self._showwarnmsg_impl = self._module._showwarnmsg_impl + if self._filter is not None: + simplefilter(*self._filter) if self._record: log = [] self._module._showwarnmsg_impl = log.append diff --git a/Misc/NEWS.d/next/Library/2022-04-10-17-12-23.gh-issue-91230.T1d_fG.rst b/Misc/NEWS.d/next/Library/2022-04-10-17-12-23.gh-issue-91230.T1d_fG.rst new file mode 100644 index 0000000000000..1efc7afca4eb2 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-10-17-12-23.gh-issue-91230.T1d_fG.rst @@ -0,0 +1,3 @@ +:func:`warnings.catch_warnings` now accepts arguments for +:func:`warnings.simplefilter`, providing a more concise way to +locally ignore warnings or convert them to errors. From webhook-mailer at python.org Sun Apr 24 03:24:10 2022 From: webhook-mailer at python.org (serhiy-storchaka) Date: Sun, 24 Apr 2022 07:24:10 -0000 Subject: [Python-checkins] Simplify testing the warning filename (GH-91868) Message-ID: https://github.com/python/cpython/commit/090721721b373c50544d297b56c217cf15992cbe commit: 090721721b373c50544d297b56c217cf15992cbe branch: main author: Serhiy Storchaka committer: serhiy-storchaka date: 2022-04-24T10:23:59+03:00 summary: Simplify testing the warning filename (GH-91868) The context manager result has the "filename" attribute. files: M Lib/test/test_asyncio/test_events.py M Lib/test/test_asyncio/test_futures.py M Lib/test/test_asyncio/test_streams.py M Lib/test/test_asyncio/test_tasks.py M Lib/test/test_re.py M Lib/unittest/test/test_async_case.py M Lib/unittest/test/test_case.py M Lib/unittest/test/test_loader.py diff --git a/Lib/test/test_asyncio/test_events.py b/Lib/test/test_asyncio/test_events.py index a30867e28029f..05d9107b28e2a 100644 --- a/Lib/test/test_asyncio/test_events.py +++ b/Lib/test/test_asyncio/test_events.py @@ -2712,12 +2712,12 @@ def get_event_loop(self): with self.assertWarns(DeprecationWarning) as cm: with self.assertRaises(TestError): asyncio.get_event_loop() - self.assertEqual(cm.warnings[0].filename, __file__) + self.assertEqual(cm.filename, __file__) asyncio.set_event_loop(None) with self.assertWarns(DeprecationWarning) as cm: with self.assertRaises(TestError): asyncio.get_event_loop() - self.assertEqual(cm.warnings[0].filename, __file__) + self.assertEqual(cm.filename, __file__) with self.assertRaisesRegex(RuntimeError, 'no running'): asyncio.get_running_loop() @@ -2734,13 +2734,13 @@ async def func(): with self.assertWarns(DeprecationWarning) as cm: with self.assertRaises(TestError): asyncio.get_event_loop() - self.assertEqual(cm.warnings[0].filename, __file__) + self.assertEqual(cm.filename, __file__) asyncio.set_event_loop(None) with self.assertWarns(DeprecationWarning) as cm: with self.assertRaises(TestError): asyncio.get_event_loop() - self.assertEqual(cm.warnings[0].filename, __file__) + self.assertEqual(cm.filename, __file__) finally: asyncio.set_event_loop_policy(old_policy) @@ -2762,12 +2762,12 @@ def test_get_event_loop_returns_running_loop2(self): with self.assertWarns(DeprecationWarning) as cm: loop2 = asyncio.get_event_loop() self.addCleanup(loop2.close) - self.assertEqual(cm.warnings[0].filename, __file__) + self.assertEqual(cm.filename, __file__) asyncio.set_event_loop(None) with self.assertWarns(DeprecationWarning) as cm: with self.assertRaisesRegex(RuntimeError, 'no current'): asyncio.get_event_loop() - self.assertEqual(cm.warnings[0].filename, __file__) + self.assertEqual(cm.filename, __file__) with self.assertRaisesRegex(RuntimeError, 'no running'): asyncio.get_running_loop() @@ -2783,13 +2783,13 @@ async def func(): asyncio.set_event_loop(loop) with self.assertWarns(DeprecationWarning) as cm: self.assertIs(asyncio.get_event_loop(), loop) - self.assertEqual(cm.warnings[0].filename, __file__) + self.assertEqual(cm.filename, __file__) asyncio.set_event_loop(None) with self.assertWarns(DeprecationWarning) as cm: with self.assertRaisesRegex(RuntimeError, 'no current'): asyncio.get_event_loop() - self.assertEqual(cm.warnings[0].filename, __file__) + self.assertEqual(cm.filename, __file__) finally: asyncio.set_event_loop_policy(old_policy) diff --git a/Lib/test/test_asyncio/test_futures.py b/Lib/test/test_asyncio/test_futures.py index cf677f6a95115..f4a46ec90a16f 100644 --- a/Lib/test/test_asyncio/test_futures.py +++ b/Lib/test/test_asyncio/test_futures.py @@ -148,7 +148,7 @@ def test_constructor_without_loop(self): with self.assertWarns(DeprecationWarning) as cm: with self.assertRaisesRegex(RuntimeError, 'There is no current event loop'): self._new_future() - self.assertEqual(cm.warnings[0].filename, __file__) + self.assertEqual(cm.filename, __file__) def test_constructor_use_running_loop(self): async def test(): @@ -163,7 +163,7 @@ def test_constructor_use_global_loop(self): self.addCleanup(asyncio.set_event_loop, None) with self.assertWarns(DeprecationWarning) as cm: f = self._new_future() - self.assertEqual(cm.warnings[0].filename, __file__) + self.assertEqual(cm.filename, __file__) self.assertIs(f._loop, self.loop) self.assertIs(f.get_loop(), self.loop) @@ -510,7 +510,7 @@ def run(arg): with self.assertWarns(DeprecationWarning) as cm: with self.assertRaises(RuntimeError): asyncio.wrap_future(f1) - self.assertEqual(cm.warnings[0].filename, __file__) + self.assertEqual(cm.filename, __file__) ex.shutdown(wait=True) def test_wrap_future_use_running_loop(self): @@ -534,7 +534,7 @@ def run(arg): f1 = ex.submit(run, 'oi') with self.assertWarns(DeprecationWarning) as cm: f2 = asyncio.wrap_future(f1) - self.assertEqual(cm.warnings[0].filename, __file__) + self.assertEqual(cm.filename, __file__) self.assertIs(self.loop, f2._loop) ex.shutdown(wait=True) diff --git a/Lib/test/test_asyncio/test_streams.py b/Lib/test/test_asyncio/test_streams.py index a7d17894e1c52..098a0da344d0f 100644 --- a/Lib/test/test_asyncio/test_streams.py +++ b/Lib/test/test_asyncio/test_streams.py @@ -813,7 +813,7 @@ def test_streamreader_constructor_without_loop(self): with self.assertWarns(DeprecationWarning) as cm: with self.assertRaisesRegex(RuntimeError, 'There is no current event loop'): asyncio.StreamReader() - self.assertEqual(cm.warnings[0].filename, __file__) + self.assertEqual(cm.filename, __file__) def test_streamreader_constructor_use_running_loop(self): # asyncio issue #184: Ensure that StreamReaderProtocol constructor @@ -832,7 +832,7 @@ def test_streamreader_constructor_use_global_loop(self): asyncio.set_event_loop(self.loop) with self.assertWarns(DeprecationWarning) as cm: reader = asyncio.StreamReader() - self.assertEqual(cm.warnings[0].filename, __file__) + self.assertEqual(cm.filename, __file__) self.assertIs(reader._loop, self.loop) @@ -841,7 +841,7 @@ def test_streamreaderprotocol_constructor_without_loop(self): with self.assertWarns(DeprecationWarning) as cm: with self.assertRaisesRegex(RuntimeError, 'There is no current event loop'): asyncio.StreamReaderProtocol(reader) - self.assertEqual(cm.warnings[0].filename, __file__) + self.assertEqual(cm.filename, __file__) def test_streamreaderprotocol_constructor_use_running_loop(self): # asyncio issue #184: Ensure that StreamReaderProtocol constructor @@ -861,7 +861,7 @@ def test_streamreaderprotocol_constructor_use_global_loop(self): reader = mock.Mock() with self.assertWarns(DeprecationWarning) as cm: protocol = asyncio.StreamReaderProtocol(reader) - self.assertEqual(cm.warnings[0].filename, __file__) + self.assertEqual(cm.filename, __file__) self.assertIs(protocol._loop, self.loop) def test_drain_raises(self): diff --git a/Lib/test/test_asyncio/test_tasks.py b/Lib/test/test_asyncio/test_tasks.py index 80afb27351362..6458859db2d12 100644 --- a/Lib/test/test_asyncio/test_tasks.py +++ b/Lib/test/test_asyncio/test_tasks.py @@ -211,7 +211,7 @@ async def notmuch(): with self.assertWarns(DeprecationWarning) as cm: with self.assertRaisesRegex(RuntimeError, 'There is no current event loop'): asyncio.ensure_future(a) - self.assertEqual(cm.warnings[0].filename, __file__) + self.assertEqual(cm.filename, __file__) async def test(): return asyncio.ensure_future(notmuch()) @@ -226,7 +226,7 @@ async def test(): self.addCleanup(asyncio.set_event_loop, None) with self.assertWarns(DeprecationWarning) as cm: t = asyncio.ensure_future(notmuch()) - self.assertEqual(cm.warnings[0].filename, __file__) + self.assertEqual(cm.filename, __file__) self.assertIs(t._loop, self.loop) self.loop.run_until_complete(t) self.assertTrue(t.done()) @@ -1449,7 +1449,7 @@ async def coro(): with self.assertWarns(DeprecationWarning) as cm: with self.assertRaisesRegex(RuntimeError, 'There is no current event loop'): list(futs) - self.assertEqual(cm.warnings[0].filename, __file__) + self.assertEqual(cm.filename, __file__) def test_as_completed_coroutine_use_running_loop(self): loop = self.new_test_loop() @@ -1881,7 +1881,7 @@ async def coro(): with self.assertWarns(DeprecationWarning) as cm: with self.assertRaisesRegex(RuntimeError, 'There is no current event loop'): asyncio.shield(inner) - self.assertEqual(cm.warnings[0].filename, __file__) + self.assertEqual(cm.filename, __file__) def test_shield_coroutine_use_running_loop(self): async def coro(): @@ -1903,7 +1903,7 @@ async def coro(): self.addCleanup(asyncio.set_event_loop, None) with self.assertWarns(DeprecationWarning) as cm: outer = asyncio.shield(coro()) - self.assertEqual(cm.warnings[0].filename, __file__) + self.assertEqual(cm.filename, __file__) self.assertEqual(outer._loop, self.loop) res = self.loop.run_until_complete(outer) self.assertEqual(res, 42) @@ -2911,7 +2911,7 @@ def test_constructor_empty_sequence_without_loop(self): with self.assertWarns(DeprecationWarning) as cm: with self.assertRaises(RuntimeError): asyncio.gather() - self.assertEqual(cm.warnings[0].filename, __file__) + self.assertEqual(cm.filename, __file__) def test_constructor_empty_sequence_use_running_loop(self): async def gather(): @@ -2929,7 +2929,7 @@ def test_constructor_empty_sequence_use_global_loop(self): self.addCleanup(asyncio.set_event_loop, None) with self.assertWarns(DeprecationWarning) as cm: fut = asyncio.gather() - self.assertEqual(cm.warnings[0].filename, __file__) + self.assertEqual(cm.filename, __file__) self.assertIsInstance(fut, asyncio.Future) self.assertIs(fut._loop, self.one_loop) self._run_loop(self.one_loop) @@ -3020,7 +3020,7 @@ async def coro(): with self.assertWarns(DeprecationWarning) as cm: with self.assertRaises(RuntimeError): asyncio.gather(gen1, gen2) - self.assertEqual(cm.warnings[0].filename, __file__) + self.assertEqual(cm.filename, __file__) def test_constructor_use_running_loop(self): async def coro(): @@ -3043,7 +3043,7 @@ async def coro(): gen2 = coro() with self.assertWarns(DeprecationWarning) as cm: fut = asyncio.gather(gen1, gen2) - self.assertEqual(cm.warnings[0].filename, __file__) + self.assertEqual(cm.filename, __file__) self.assertIs(fut._loop, self.other_loop) self.other_loop.run_until_complete(fut) diff --git a/Lib/test/test_re.py b/Lib/test/test_re.py index 91971d8126c98..a4c2f1f3e4ba3 100644 --- a/Lib/test/test_re.py +++ b/Lib/test/test_re.py @@ -2608,11 +2608,11 @@ def test_deprecated_modules(self): for name in deprecated: with self.subTest(module=name): sys.modules.pop(name, None) - with self.assertWarns(DeprecationWarning) as cm: + with self.assertWarns(DeprecationWarning) as w: __import__(name) - self.assertEqual(str(cm.warnings[0].message), + self.assertEqual(str(w.warning), f"module {name!r} is deprecated") - self.assertEqual(cm.warnings[0].filename, __file__) + self.assertEqual(w.filename, __file__) self.assertIn(name, sys.modules) mod = sys.modules[name] self.assertEqual(mod.__name__, name) diff --git a/Lib/unittest/test/test_async_case.py b/Lib/unittest/test/test_async_case.py index a48140829cc21..1b910a44eea0d 100644 --- a/Lib/unittest/test/test_async_case.py +++ b/Lib/unittest/test/test_async_case.py @@ -259,15 +259,15 @@ async def test2(self): with self.assertWarns(DeprecationWarning) as w: Test('test1').run() - self.assertIn('It is deprecated to return a value!=None', str(w.warnings[0].message)) - self.assertIn('test1', str(w.warnings[0].message)) - self.assertEqual(w.warnings[0].filename, __file__) + self.assertIn('It is deprecated to return a value!=None', str(w.warning)) + self.assertIn('test1', str(w.warning)) + self.assertEqual(w.filename, __file__) with self.assertWarns(DeprecationWarning) as w: Test('test2').run() - self.assertIn('It is deprecated to return a value!=None', str(w.warnings[0].message)) - self.assertIn('test2', str(w.warnings[0].message)) - self.assertEqual(w.warnings[0].filename, __file__) + self.assertIn('It is deprecated to return a value!=None', str(w.warning)) + self.assertIn('test2', str(w.warning)) + self.assertEqual(w.filename, __file__) def test_cleanups_interleave_order(self): events = [] diff --git a/Lib/unittest/test/test_case.py b/Lib/unittest/test/test_case.py index f6cb9977d06d4..374a255255566 100644 --- a/Lib/unittest/test/test_case.py +++ b/Lib/unittest/test/test_case.py @@ -316,15 +316,15 @@ def test2(self): with self.assertWarns(DeprecationWarning) as w: Foo('test1').run() - self.assertIn('It is deprecated to return a value!=None', str(w.warnings[0].message)) - self.assertIn('test1', str(w.warnings[0].message)) - self.assertEqual(w.warnings[0].filename, __file__) + self.assertIn('It is deprecated to return a value!=None', str(w.warning)) + self.assertIn('test1', str(w.warning)) + self.assertEqual(w.filename, __file__) with self.assertWarns(DeprecationWarning) as w: Foo('test2').run() - self.assertIn('It is deprecated to return a value!=None', str(w.warnings[0].message)) - self.assertIn('test2', str(w.warnings[0].message)) - self.assertEqual(w.warnings[0].filename, __file__) + self.assertIn('It is deprecated to return a value!=None', str(w.warning)) + self.assertIn('test2', str(w.warning)) + self.assertEqual(w.filename, __file__) def _check_call_order__subtests(self, result, events, expected_events): class Foo(Test.LoggingTestCase): diff --git a/Lib/unittest/test/test_loader.py b/Lib/unittest/test/test_loader.py index 90e208182719b..de2268cda9068 100644 --- a/Lib/unittest/test/test_loader.py +++ b/Lib/unittest/test/test_loader.py @@ -1609,7 +1609,7 @@ def test_getTestCaseNames(self): tests = unittest.getTestCaseNames(self.MyTestCase, prefix='check', sortUsing=self.reverse_three_way_cmp, testNamePatterns=None) - self.assertEqual(w.warnings[0].filename, __file__) + self.assertEqual(w.filename, __file__) self.assertEqual(tests, ['check_2', 'check_1']) def test_makeSuite(self): @@ -1617,7 +1617,7 @@ def test_makeSuite(self): suite = unittest.makeSuite(self.MyTestCase, prefix='check', sortUsing=self.reverse_three_way_cmp, suiteClass=self.MyTestSuite) - self.assertEqual(w.warnings[0].filename, __file__) + self.assertEqual(w.filename, __file__) self.assertIsInstance(suite, self.MyTestSuite) expected = self.MyTestSuite([self.MyTestCase('check_2'), self.MyTestCase('check_1')]) @@ -1631,7 +1631,7 @@ def test_findTestCases(self): suite = unittest.findTestCases(m, prefix='check', sortUsing=self.reverse_three_way_cmp, suiteClass=self.MyTestSuite) - self.assertEqual(w.warnings[0].filename, __file__) + self.assertEqual(w.filename, __file__) self.assertIsInstance(suite, self.MyTestSuite) expected = [self.MyTestSuite([self.MyTestCase('check_2'), self.MyTestCase('check_1')])] From webhook-mailer at python.org Sun Apr 24 17:07:55 2022 From: webhook-mailer at python.org (ezio-melotti) Date: Sun, 24 Apr 2022 21:07:55 -0000 Subject: [Python-checkins] [3.10] Update Sphinx bpo role to use redirect URI. (#91890) Message-ID: https://github.com/python/cpython/commit/dc31334ab1b43225b65358fa361b46c22918b400 commit: dc31334ab1b43225b65358fa361b46c22918b400 branch: 3.10 author: Ezio Melotti committer: ezio-melotti date: 2022-04-24T23:07:51+02:00 summary: [3.10] Update Sphinx bpo role to use redirect URI. (#91890) * Update Sphinx bpo role to use redirect URI. (GH-32342) * [3.10] Update Sphinx bpo role to use redirect URI. (GH-32342). (cherry picked from commit 08cfe079503ffd19d8b7ab324f0fdb1c6b150ca8) Co-authored-by: Ezio Melotti * Fix whitespace. files: M Doc/tools/extensions/pyspecific.py diff --git a/Doc/tools/extensions/pyspecific.py b/Doc/tools/extensions/pyspecific.py index 7f505a1ede89a..cce364c0e6245 100644 --- a/Doc/tools/extensions/pyspecific.py +++ b/Doc/tools/extensions/pyspecific.py @@ -43,7 +43,7 @@ import suspicious -ISSUE_URI = 'https://bugs.python.org/issue%s' +ISSUE_URI = 'https://bugs.python.org/issue?@action=redirect&bpo=%s' SOURCE_URI = 'https://github.com/python/cpython/tree/3.10/%s' # monkey-patch reST parser to disable alphabetic and roman enumerated lists From webhook-mailer at python.org Sun Apr 24 17:13:21 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Sun, 24 Apr 2022 21:13:21 -0000 Subject: [Python-checkins] gh-91491: What's New in 3.11 section for typing PEPs (#91721) Message-ID: https://github.com/python/cpython/commit/dd3cf124a033a2e3bb975be53db40213457afd51 commit: dd3cf124a033a2e3bb975be53db40213457afd51 branch: main author: Jelle Zijlstra committer: JelleZijlstra date: 2022-04-24T14:13:15-07:00 summary: gh-91491: What's New in 3.11 section for typing PEPs (#91721) Other aspects of typing aren't covered yet; I'll do that in a separate PR. files: M Doc/whatsnew/3.11.rst diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index 43d8e6b1cbf41..ba85b7896b2c0 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -74,8 +74,11 @@ New syntax features: New typing features: -* :pep:`673`: ``Self`` Type. - (Contributed by James Hilton-Balfe and Pradeep Kumar in :issue:`30924`.) +* :pep:`646`: Variadic generics. +* :pep:`655`: Marking individual TypedDict items as required or potentially-missing. +* :pep:`673`: ``Self`` type. +* :pep:`675`: Arbitrary literal string type. + New Features ============ @@ -167,6 +170,134 @@ default traceback. See :pep:`678` for more details. (Contributed by Irit Katriel in :issue:`45607`.) +.. _new-feat-related-type-hints-311: + +New Features Related to Type Hints +================================== + +This section covers major changes affecting :pep:`484` type hints and +the :mod:`typing` module. + +PEP 646: Variadic generics +-------------------------- + +:pep:`484` introduced :data:`~typing.TypeVar`, enabling creation +of generics parameterised with a single type. :pep:`646` introduces +:data:`~typing.TypeVarTuple`, enabling parameterisation +with an *arbitrary* number of types. In other words, +a :data:`~typing.TypeVarTuple` is a *variadic* type variable, +enabling *variadic* generics. This enables a wide variety of use cases. +In particular, it allows the type of array-like structures +in numerical computing libraries such as NumPy and TensorFlow to be +parameterised with the array *shape*. Static type checkers will now +be able to catch shape-related bugs in code that uses these libraries. + +See :pep:`646` for more details. + +(Contributed by Matthew Rahtz in :issue:`43224`, with contributions by +Serhiy Storchaka and Jelle Zijlstra. PEP written by Mark Mendoza, Matthew +Rahtz, Pradeep Kumar Srinivasan, and Vincent Siles.) + +PEP 655: Marking individual ``TypedDict`` items as required or not-required +--------------------------------------------------------------------------- + +:data:`~typing.Required` and :data:`~typing.NotRequired` provide a +straightforward way to mark whether individual items in a +:data:`~typing.TypedDict` must be present. Previously this was only possible +using inheritance. + +Fields are still required by default, unless the ``total=False`` +parameter is set. +For example, the following specifies a dictionary with one required and +one not-required key:: + + class Movie(TypedDict): + title: str + year: NotRequired[int] + + m1: Movie = {"title": "Black Panther", "year": 2018} # ok + m2: Movie = {"title": "Star Wars"} # ok (year is not required) + m3: Movie = {"year": 2022} # error (missing required field title) + +The following definition is equivalent:: + + class Movie(TypedDict, total=False): + title: Required[str] + year: int + +See :pep:`655` for more details. + +(Contributed by David Foster and Jelle Zijlstra in :issue:`47087`. PEP +written by David Foster.) + +PEP 673: ``Self`` type +---------------------- + +The new :data:`~typing.Self` annotation provides a simple and intuitive +way to annotate methods that return an instance of their class. This +behaves the same as the :data:`~typing.TypeVar`-based approach specified +in :pep:`484` but is more concise and easier to follow. + +Common use cases include alternative constructors provided as classmethods +and :meth:`~object.__enter__` methods that return ``self``:: + + class MyLock: + def __enter__(self) -> Self: + self.lock() + return self + + ... + + class MyInt: + @classmethod + def fromhex(cls, s: str) -> Self: + return cls(int(s, 16)) + + ... + +:data:`~typing.Self` can also be used to annotate method parameters +or attributes of the same type as their enclosing class. + +See :pep:`673` for more details. + +(Contributed by James Hilton-Balfe in :issue:`46534`. PEP written by +Pradeep Kumar Srinivasan and James Hilton-Balfe.) + +PEP 675: Arbitrary literal string type +-------------------------------------- + +The new :data:`~typing.LiteralString` annotation may be used to indicate +that a function parameter can be of any literal string type. This allows +a function to accept arbitrary literal string types, as well as strings +created from other literal strings. Type checkers can then +enforce that sensitive functions, such as those that execute SQL +statements or shell commands, are called only with static arguments, +providing protection against injection attacks. + +For example, a SQL query function could be annotated as follows:: + + def run_query(sql: LiteralString) -> ... + ... + + def caller( + arbitrary_string: str, + query_string: LiteralString, + table_name: LiteralString, + ) -> None: + run_query("SELECT * FROM students") # ok + run_query(query_string) # ok + run_query("SELECT * FROM " + table_name) # ok + run_query(arbitrary_string) # type checker error + run_query( # type checker error + f"SELECT * FROM students WHERE name = {arbitrary_string}" + ) + +See :pep:`675` for more details. + +(Contributed by Jelle Zijlstra in :issue:`47088`. PEP written by Pradeep +Kumar Srinivasan and Graham Bleaney.) + + Other Language Changes ====================== From webhook-mailer at python.org Sun Apr 24 17:17:59 2022 From: webhook-mailer at python.org (ezio-melotti) Date: Sun, 24 Apr 2022 21:17:59 -0000 Subject: [Python-checkins] [3.9] Update Sphinx bpo role to use redirect URI. (#91891) Message-ID: https://github.com/python/cpython/commit/7f897b96262874c5d55bbdc60a53236b3b4021cd commit: 7f897b96262874c5d55bbdc60a53236b3b4021cd branch: 3.9 author: Ezio Melotti committer: ezio-melotti date: 2022-04-24T23:17:48+02:00 summary: [3.9] Update Sphinx bpo role to use redirect URI. (#91891) * Update Sphinx bpo role to use redirect URI. (GH-32342) * [3.9] Update Sphinx bpo role to use redirect URI. (GH-32342). (cherry picked from commit 08cfe079503ffd19d8b7ab324f0fdb1c6b150ca8) Co-authored-by: Ezio Melotti files: M Doc/tools/extensions/pyspecific.py diff --git a/Doc/tools/extensions/pyspecific.py b/Doc/tools/extensions/pyspecific.py index 7c448310f47f6..452366a178926 100644 --- a/Doc/tools/extensions/pyspecific.py +++ b/Doc/tools/extensions/pyspecific.py @@ -43,7 +43,7 @@ import suspicious -ISSUE_URI = 'https://bugs.python.org/issue%s' +ISSUE_URI = 'https://bugs.python.org/issue?@action=redirect&bpo=%s' SOURCE_URI = 'https://github.com/python/cpython/tree/3.9/%s' # monkey-patch reST parser to disable alphabetic and roman enumerated lists From webhook-mailer at python.org Sun Apr 24 18:03:06 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Sun, 24 Apr 2022 22:03:06 -0000 Subject: [Python-checkins] gh-91821: Make decimal test succeed consistently (#91825) Message-ID: https://github.com/python/cpython/commit/e93d2fbddaee329b2e8f405cbb5aec44ca221de6 commit: e93d2fbddaee329b2e8f405cbb5aec44ca221de6 branch: main author: Jelle Zijlstra committer: JelleZijlstra date: 2022-04-24T15:02:57-07:00 summary: gh-91821: Make decimal test succeed consistently (#91825) The test relies on precision being set to 9, but some ways of invoking this test leave it set to 28 instead. I don't know exactly how it happens, but setting the precision directly should make the behavior consistent. files: A Misc/NEWS.d/next/Library/2022-04-22-08-25-18.gh-issue-91821.XwMkj0.rst M Lib/test/test_decimal.py diff --git a/Lib/test/test_decimal.py b/Lib/test/test_decimal.py index 96f8f7f32c454..f7a47c86a3fe8 100644 --- a/Lib/test/test_decimal.py +++ b/Lib/test/test_decimal.py @@ -5520,6 +5520,7 @@ def test_from_tuple(self): with localcontext() as c: + c.prec = 9 c.traps[InvalidOperation] = True c.traps[Overflow] = True c.traps[Underflow] = True diff --git a/Misc/NEWS.d/next/Library/2022-04-22-08-25-18.gh-issue-91821.XwMkj0.rst b/Misc/NEWS.d/next/Library/2022-04-22-08-25-18.gh-issue-91821.XwMkj0.rst new file mode 100644 index 0000000000000..9d7f9d48c4c99 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-22-08-25-18.gh-issue-91821.XwMkj0.rst @@ -0,0 +1 @@ +Fix unstable ``test_from_tuple`` test in ``test_decimal.py``. From webhook-mailer at python.org Sun Apr 24 18:50:17 2022 From: webhook-mailer at python.org (warsaw) Date: Sun, 24 Apr 2022 22:50:17 -0000 Subject: [Python-checkins] Rewrite audio.py to jive with image.py (#91886) Message-ID: https://github.com/python/cpython/commit/440332072706c5e422e6c54a2ec0ebb88e09c85c commit: 440332072706c5e422e6c54a2ec0ebb88e09c85c branch: main author: Barry Warsaw committer: warsaw date: 2022-04-24T15:50:07-07:00 summary: Rewrite audio.py to jive with image.py (#91886) Similar to the rewrite of email/mime/image.py and associated test after the deprecation of imghdr.py, thisrewrites email/mime/audio.py and associated tests after the deprecation of sndhdr.py. Closes #91885 files: A Lib/test/test_email/data/sndhdr.aifc A Lib/test/test_email/data/sndhdr.aiff A Lib/test/test_email/data/sndhdr.au A Lib/test/test_email/data/sndhdr.wav D Lib/test/test_email/data/audiotest.au M Lib/email/mime/audio.py M Lib/test/test_email/test_email.py diff --git a/Lib/email/mime/audio.py b/Lib/email/mime/audio.py index e859c2e8a2b6c..8815f5c5ec06d 100644 --- a/Lib/email/mime/audio.py +++ b/Lib/email/mime/audio.py @@ -11,58 +11,6 @@ from email.mime.nonmultipart import MIMENonMultipart -_tests = [] - -def _test_aifc_aiff(h, f): - if not h.startswith(b'FORM'): - return None - if h[8:12] in {b'AIFC', b'AIFF'}: - return 'x-aiff' - else: - return None - -_tests.append(_test_aifc_aiff) - - -def _test_au(h, f): - if h.startswith(b'.snd'): - return 'basic' - else: - return None - -_tests.append(_test_au) - - -def _test_wav(h, f): - import wave - # 'RIFF' 'WAVE' 'fmt ' - if not h.startswith(b'RIFF') or h[8:12] != b'WAVE' or h[12:16] != b'fmt ': - return None - else: - return "x-wav" - -_tests.append(_test_wav) - - -# There are others in sndhdr that don't have MIME types. :( -# Additional ones to be added to sndhdr? midi, mp3, realaudio, wma?? -def _whatsnd(data): - """Try to identify a sound file type. - - sndhdr.what() has a pretty cruddy interface, unfortunately. This is why - we re-do it here. It would be easier to reverse engineer the Unix 'file' - command and use the standard 'magic' file, as shipped with a modern Unix. - """ - hdr = data[:512] - fakefile = BytesIO(hdr) - for testfn in _tests: - res = testfn(hdr, fakefile) - if res is not None: - return res - else: - return None - - class MIMEAudio(MIMENonMultipart): """Class for generating audio/* MIME documents.""" @@ -89,10 +37,64 @@ def __init__(self, _audiodata, _subtype=None, header. """ if _subtype is None: - _subtype = _whatsnd(_audiodata) + _subtype = _what(_audiodata) if _subtype is None: raise TypeError('Could not find audio MIME subtype') MIMENonMultipart.__init__(self, 'audio', _subtype, policy=policy, **_params) self.set_payload(_audiodata) _encoder(self) + + +_rules = [] + + +# Originally from the sndhdr module. +# +# There are others in sndhdr that don't have MIME types. :( +# Additional ones to be added to sndhdr? midi, mp3, realaudio, wma?? +def _what(data): + # Try to identify a sound file type. + # + # sndhdr.what() had a pretty cruddy interface, unfortunately. This is why + # we re-do it here. It would be easier to reverse engineer the Unix 'file' + # command and use the standard 'magic' file, as shipped with a modern Unix. + hdr = data[:512] + fakefile = BytesIO(hdr) + for testfn in _rules: + if res := testfn(hdr, fakefile): + return res + else: + return None + + +def rule(rulefunc): + _rules.append(rulefunc) + return rulefunc + + + at rule +def _aiff(h, f): + if not h.startswith(b'FORM'): + return None + if h[8:12] in {b'AIFC', b'AIFF'}: + return 'x-aiff' + else: + return None + + + at rule +def _au(h, f): + if h.startswith(b'.snd'): + return 'basic' + else: + return None + + + at rule +def _wav(h, f): + # 'RIFF' 'WAVE' 'fmt ' + if not h.startswith(b'RIFF') or h[8:12] != b'WAVE' or h[12:16] != b'fmt ': + return None + else: + return "x-wav" diff --git a/Lib/test/test_email/data/sndhdr.aifc b/Lib/test/test_email/data/sndhdr.aifc new file mode 100644 index 0000000000000..8aae4e730bdaf Binary files /dev/null and b/Lib/test/test_email/data/sndhdr.aifc differ diff --git a/Lib/test/test_email/data/sndhdr.aiff b/Lib/test/test_email/data/sndhdr.aiff new file mode 100644 index 0000000000000..8c279a762f1c7 Binary files /dev/null and b/Lib/test/test_email/data/sndhdr.aiff differ diff --git a/Lib/test/test_email/data/audiotest.au b/Lib/test/test_email/data/sndhdr.au similarity index 100% rename from Lib/test/test_email/data/audiotest.au rename to Lib/test/test_email/data/sndhdr.au diff --git a/Lib/test/test_email/data/sndhdr.wav b/Lib/test/test_email/data/sndhdr.wav new file mode 100644 index 0000000000000..0dca36739cde3 Binary files /dev/null and b/Lib/test/test_email/data/sndhdr.wav differ diff --git a/Lib/test/test_email/test_email.py b/Lib/test/test_email/test_email.py index 6ead5947acb6f..933aa4cbc1959 100644 --- a/Lib/test/test_email/test_email.py +++ b/Lib/test/test_email/test_email.py @@ -1515,37 +1515,49 @@ def test_multipart_with_bad_bytes_in_cte(self): # Test the basic MIMEAudio class class TestMIMEAudio(unittest.TestCase): - def setUp(self): - with openfile('audiotest.au', 'rb') as fp: + def _make_audio(self, ext): + with openfile(f'sndhdr.{ext}', 'rb') as fp: self._audiodata = fp.read() self._au = MIMEAudio(self._audiodata) def test_guess_minor_type(self): - self.assertEqual(self._au.get_content_type(), 'audio/basic') + for ext, subtype in { + 'aifc': 'x-aiff', + 'aiff': 'x-aiff', + 'wav': 'x-wav', + 'au': 'basic', + }.items(): + self._make_audio(ext) + subtype = ext if subtype is None else subtype + self.assertEqual(self._au.get_content_type(), f'audio/{subtype}') def test_encoding(self): + self._make_audio('au') payload = self._au.get_payload() self.assertEqual(base64.decodebytes(bytes(payload, 'ascii')), - self._audiodata) + self._audiodata) def test_checkSetMinor(self): + self._make_audio('au') au = MIMEAudio(self._audiodata, 'fish') self.assertEqual(au.get_content_type(), 'audio/fish') def test_add_header(self): + self._make_audio('au') eq = self.assertEqual self._au.add_header('Content-Disposition', 'attachment', - filename='audiotest.au') + filename='sndhdr.au') eq(self._au['content-disposition'], - 'attachment; filename="audiotest.au"') + 'attachment; filename="sndhdr.au"') eq(self._au.get_params(header='content-disposition'), - [('attachment', ''), ('filename', 'audiotest.au')]) + [('attachment', ''), ('filename', 'sndhdr.au')]) eq(self._au.get_param('filename', header='content-disposition'), - 'audiotest.au') + 'sndhdr.au') missing = [] eq(self._au.get_param('attachment', header='content-disposition'), '') - self.assertIs(self._au.get_param('foo', failobj=missing, - header='content-disposition'), missing) + self.assertIs(self._au.get_param( + 'foo', failobj=missing, + header='content-disposition'), missing) # Try some missing stuff self.assertIs(self._au.get_param('foobar', missing), missing) self.assertIs(self._au.get_param('attachment', missing, @@ -3462,7 +3474,7 @@ def test_BytesGenerator_linend_with_non_ascii(self): self.assertEqual(s.getvalue(), msgtxt) def test_mime_classes_policy_argument(self): - with openfile('audiotest.au', 'rb') as fp: + with openfile('sndhdr.au', 'rb') as fp: audiodata = fp.read() with openfile('python.gif', 'rb') as fp: bindata = fp.read() From webhook-mailer at python.org Mon Apr 25 06:06:04 2022 From: webhook-mailer at python.org (ezio-melotti) Date: Mon, 25 Apr 2022 10:06:04 -0000 Subject: [Python-checkins] gh-91888: add a `:gh:` role to the documentation (#91889) Message-ID: https://github.com/python/cpython/commit/f7641a2ffec243e5f600028a84debe9028a9ee44 commit: f7641a2ffec243e5f600028a84debe9028a9ee44 branch: main author: Ezio Melotti committer: ezio-melotti date: 2022-04-25T12:05:54+02:00 summary: gh-91888: add a `:gh:` role to the documentation (#91889) * Add a new :gh:`...` role for GitHub issues. * Fix a GitHub id to use the :gh: role. * Add Misc/NEWS entry. * Refactoring and rephrasing. Co-authored-by: Hugo van Kemenade files: A Misc/NEWS.d/next/Documentation/2022-04-24-22-09-31.gh-issue-91888.kTjJLx.rst M Doc/tools/extensions/pyspecific.py M Doc/whatsnew/3.11.rst diff --git a/Doc/tools/extensions/pyspecific.py b/Doc/tools/extensions/pyspecific.py index 80533a97b1189..12b98f49f69f5 100644 --- a/Doc/tools/extensions/pyspecific.py +++ b/Doc/tools/extensions/pyspecific.py @@ -44,6 +44,7 @@ ISSUE_URI = 'https://bugs.python.org/issue?@action=redirect&bpo=%s' +GH_ISSUE_URI = 'https://github.com/python/cpython/issues/%s' SOURCE_URI = 'https://github.com/python/cpython/tree/main/%s' # monkey-patch reST parser to disable alphabetic and roman enumerated lists @@ -58,11 +59,33 @@ def issue_role(typ, rawtext, text, lineno, inliner, options={}, content=[]): issue = utils.unescape(text) + # sanity check: there are no bpo issues within these two values + if 47261 < int(issue) < 400000: + msg = inliner.reporter.error(f'The BPO ID {text!r} seems too high -- ' + 'use :gh:`...` for GitHub IDs', line=lineno) + prb = inliner.problematic(rawtext, rawtext, msg) + return [prb], [msg] text = 'bpo-' + issue refnode = nodes.reference(text, text, refuri=ISSUE_URI % issue) return [refnode], [] +# Support for marking up and linking to GitHub issues + +def gh_issue_role(typ, rawtext, text, lineno, inliner, options={}, content=[]): + issue = utils.unescape(text) + # sanity check: all GitHub issues have ID >= 32426 + # even though some of them are also valid BPO IDs + if int(issue) < 32426: + msg = inliner.reporter.error(f'The GitHub ID {text!r} seems too low -- ' + 'use :issue:`...` for BPO IDs', line=lineno) + prb = inliner.problematic(rawtext, rawtext, msg) + return [prb], [msg] + text = 'gh-' + issue + refnode = nodes.reference(text, text, refuri=GH_ISSUE_URI % issue) + return [refnode], [] + + # Support for linking to Python source files easily def source_role(typ, rawtext, text, lineno, inliner, options={}, content=[]): @@ -615,6 +638,7 @@ def process_audit_events(app, doctree, fromdocname): def setup(app): app.add_role('issue', issue_role) + app.add_role('gh', gh_issue_role) app.add_role('source', source_role) app.add_directive('impl-detail', ImplementationDetail) app.add_directive('availability', Availability) diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index ba85b7896b2c0..ebaa1e993e276 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -463,7 +463,7 @@ inspect line number, column and end column). The affected functions are: :func:`inspect.getframeinfo`, :func:`inspect.getouterframes`, :func:`inspect.getinnerframes`, :func:`inspect.stack` and :func:`inspect.trace`. (Contributed by Pablo Galindo in - :issue:`88116`) + :gh:`88116`) locale ------ diff --git a/Misc/NEWS.d/next/Documentation/2022-04-24-22-09-31.gh-issue-91888.kTjJLx.rst b/Misc/NEWS.d/next/Documentation/2022-04-24-22-09-31.gh-issue-91888.kTjJLx.rst new file mode 100644 index 0000000000000..9194be9dbf92d --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2022-04-24-22-09-31.gh-issue-91888.kTjJLx.rst @@ -0,0 +1 @@ +Add a new `gh` role to the documentation to link to GitHub issues. From webhook-mailer at python.org Mon Apr 25 06:58:40 2022 From: webhook-mailer at python.org (tiran) Date: Mon, 25 Apr 2022 10:58:40 -0000 Subject: [Python-checkins] gh-84461: Include _emscripten_info in pythoninfo output (GH-91907) Message-ID: https://github.com/python/cpython/commit/9ff2f12c876289a7192fd1672ed08a1876a51e17 commit: 9ff2f12c876289a7192fd1672ed08a1876a51e17 branch: main author: Christian Heimes committer: tiran date: 2022-04-25T12:58:25+02:00 summary: gh-84461: Include _emscripten_info in pythoninfo output (GH-91907) files: M Lib/test/pythoninfo.py diff --git a/Lib/test/pythoninfo.py b/Lib/test/pythoninfo.py index 39301e6397aab..28549a645b0f5 100644 --- a/Lib/test/pythoninfo.py +++ b/Lib/test/pythoninfo.py @@ -79,6 +79,7 @@ def call_func(info_add, name, mod, func_name, *, formatter=None): def collect_sys(info_add): attributes = ( + '_emscripten_info', '_framework', 'abiflags', 'api_version', From webhook-mailer at python.org Mon Apr 25 09:40:34 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Mon, 25 Apr 2022 13:40:34 -0000 Subject: [Python-checkins] gh-90633: Improve error and docs for typing.assert_never (#91720) Message-ID: https://github.com/python/cpython/commit/93d280141c369fd1906569ff605148b6e22f6a43 commit: 93d280141c369fd1906569ff605148b6e22f6a43 branch: main author: Jelle Zijlstra committer: JelleZijlstra date: 2022-04-25T06:40:18-07:00 summary: gh-90633: Improve error and docs for typing.assert_never (#91720) Closes #90633 Co-authored-by: Alex Waygood files: A Misc/NEWS.d/next/Library/2022-04-19-19-50-10.gh-issue-90633.Youov0.rst M Doc/library/typing.rst M Lib/test/test_typing.py M Lib/typing.py diff --git a/Doc/library/typing.rst b/Doc/library/typing.rst index 2f62193e65c29..4635da7579ac7 100644 --- a/Doc/library/typing.rst +++ b/Doc/library/typing.rst @@ -2345,11 +2345,25 @@ Functions and decorators case _ as unreachable: assert_never(unreachable) + Here, the annotations allow the type checker to infer that the + last case can never execute, because ``arg`` is either + an :class:`int` or a :class:`str`, and both options are covered by + earlier cases. If a type checker finds that a call to ``assert_never()`` is - reachable, it will emit an error. + reachable, it will emit an error. For example, if the type annotation + for ``arg`` was instead ``int | str | float``, the type checker would + emit an error pointing out that ``unreachable`` is of type :class:`float`. + For a call to ``assert_never`` to succeed, the inferred type of + the argument passed in must be the bottom type, :data:`Never`, and nothing + else. At runtime, this throws an exception when called. + .. seealso:: + `Unreachable Code and Exhaustiveness Checking + _` has more + information about exhaustiveness checking with static typing. + .. versionadded:: 3.11 .. function:: reveal_type(obj) diff --git a/Lib/test/test_typing.py b/Lib/test/test_typing.py index 1fd99a0f806cf..fce340f27d9ac 100644 --- a/Lib/test/test_typing.py +++ b/Lib/test/test_typing.py @@ -223,6 +223,19 @@ def test_exception(self): with self.assertRaises(AssertionError): assert_never(None) + value = "some value" + with self.assertRaisesRegex(AssertionError, value): + assert_never(value) + + # Make sure a huge value doesn't get printed in its entirety + huge_value = "a" * 10000 + with self.assertRaises(AssertionError) as cm: + assert_never(huge_value) + self.assertLess( + len(cm.exception.args[0]), + typing._ASSERT_NEVER_REPR_MAX_LENGTH * 2, + ) + class SelfTests(BaseTestCase): def test_equality(self): diff --git a/Lib/typing.py b/Lib/typing.py index a6f4fa96afb50..9d8149c3f5332 100644 --- a/Lib/typing.py +++ b/Lib/typing.py @@ -2382,6 +2382,9 @@ class Film(TypedDict): return isinstance(tp, _TypedDictMeta) +_ASSERT_NEVER_REPR_MAX_LENGTH = 100 + + def assert_never(arg: Never, /) -> Never: """Statically assert that a line of code is unreachable. @@ -2402,7 +2405,10 @@ def int_or_str(arg: int | str) -> None: At runtime, this throws an exception when called. """ - raise AssertionError("Expected code to be unreachable") + value = repr(arg) + if len(value) > _ASSERT_NEVER_REPR_MAX_LENGTH: + value = value[:_ASSERT_NEVER_REPR_MAX_LENGTH] + '...' + raise AssertionError(f"Expected code to be unreachable, but got: {value}") def no_type_check(arg): diff --git a/Misc/NEWS.d/next/Library/2022-04-19-19-50-10.gh-issue-90633.Youov0.rst b/Misc/NEWS.d/next/Library/2022-04-19-19-50-10.gh-issue-90633.Youov0.rst new file mode 100644 index 0000000000000..d86c2d3ff5de4 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-19-19-50-10.gh-issue-90633.Youov0.rst @@ -0,0 +1,2 @@ +Include the passed value in the exception thrown by +:func:`typing.assert_never`. Patch by Jelle Zijlstra. From webhook-mailer at python.org Mon Apr 25 10:35:56 2022 From: webhook-mailer at python.org (serhiy-storchaka) Date: Mon, 25 Apr 2022 14:35:56 -0000 Subject: [Python-checkins] gh-91904: Fix setting envvar PYTHONREGRTEST_UNICODE_GUARD (GH-91905) Message-ID: https://github.com/python/cpython/commit/54d068adfbf2b822bcbf90dac9b3f6684cec0f99 commit: 54d068adfbf2b822bcbf90dac9b3f6684cec0f99 branch: main author: Serhiy Storchaka committer: serhiy-storchaka date: 2022-04-25T17:35:14+03:00 summary: gh-91904: Fix setting envvar PYTHONREGRTEST_UNICODE_GUARD (GH-91905) It always failed on non-UTF-8 locale and prevented running regrtests. files: A Misc/NEWS.d/next/Tests/2022-04-25-11-16-36.gh-issue-91904.13Uvrz.rst M Lib/test/libregrtest/setup.py M Lib/test/test_regrtest.py diff --git a/Lib/test/libregrtest/setup.py b/Lib/test/libregrtest/setup.py index 4ffd15478f643..cfc1b82d2785c 100644 --- a/Lib/test/libregrtest/setup.py +++ b/Lib/test/libregrtest/setup.py @@ -5,6 +5,7 @@ import sys import unittest from test import support +from test.support.os_helper import TESTFN_UNDECODABLE, FS_NONASCII try: import gc except ImportError: @@ -104,10 +105,10 @@ def _test_audit_hook(name, args): # Ensure there's a non-ASCII character in env vars at all times to force # tests consider this case. See BPO-44647 for details. - os.environ.setdefault( - UNICODE_GUARD_ENV, - "\N{SMILING FACE WITH SUNGLASSES}", - ) + if TESTFN_UNDECODABLE and os.supports_bytes_environ: + os.environb.setdefault(UNICODE_GUARD_ENV.encode(), TESTFN_UNDECODABLE) + elif FS_NONASCII: + os.environ.setdefault(UNICODE_GUARD_ENV, FS_NONASCII) def replace_stdout(): diff --git a/Lib/test/test_regrtest.py b/Lib/test/test_regrtest.py index babc8a690877a..15e2f89ee20c1 100644 --- a/Lib/test/test_regrtest.py +++ b/Lib/test/test_regrtest.py @@ -1339,7 +1339,7 @@ def test_print_warning(self): def test_unicode_guard_env(self): guard = os.environ.get(setup.UNICODE_GUARD_ENV) self.assertIsNotNone(guard, f"{setup.UNICODE_GUARD_ENV} not set") - if guard != "\N{SMILING FACE WITH SUNGLASSES}": + if guard.isascii(): # Skip to signify that the env var value was changed by the user; # possibly to something ASCII to work around Unicode issues. self.skipTest("Modified guard") diff --git a/Misc/NEWS.d/next/Tests/2022-04-25-11-16-36.gh-issue-91904.13Uvrz.rst b/Misc/NEWS.d/next/Tests/2022-04-25-11-16-36.gh-issue-91904.13Uvrz.rst new file mode 100644 index 0000000000000..31ddfc312866b --- /dev/null +++ b/Misc/NEWS.d/next/Tests/2022-04-25-11-16-36.gh-issue-91904.13Uvrz.rst @@ -0,0 +1,2 @@ +Fix initialization of :envvar:`PYTHONREGRTEST_UNICODE_GUARD` which prevented +running regression tests on non-UTF-8 locale. From webhook-mailer at python.org Mon Apr 25 11:04:05 2022 From: webhook-mailer at python.org (miss-islington) Date: Mon, 25 Apr 2022 15:04:05 -0000 Subject: [Python-checkins] gh-91904: Fix setting envvar PYTHONREGRTEST_UNICODE_GUARD (GH-91905) Message-ID: https://github.com/python/cpython/commit/971343eb569a3418aa9a0bad9b638cccf1470ef8 commit: 971343eb569a3418aa9a0bad9b638cccf1470ef8 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-25T08:03:47-07:00 summary: gh-91904: Fix setting envvar PYTHONREGRTEST_UNICODE_GUARD (GH-91905) It always failed on non-UTF-8 locale and prevented running regrtests. (cherry picked from commit 54d068adfbf2b822bcbf90dac9b3f6684cec0f99) Co-authored-by: Serhiy Storchaka files: A Misc/NEWS.d/next/Tests/2022-04-25-11-16-36.gh-issue-91904.13Uvrz.rst M Lib/test/libregrtest/setup.py M Lib/test/test_regrtest.py diff --git a/Lib/test/libregrtest/setup.py b/Lib/test/libregrtest/setup.py index b79175944d2bd..0bfd644bb3179 100644 --- a/Lib/test/libregrtest/setup.py +++ b/Lib/test/libregrtest/setup.py @@ -5,6 +5,7 @@ import sys import unittest from test import support +from test.support.os_helper import TESTFN_UNDECODABLE, FS_NONASCII try: import gc except ImportError: @@ -105,10 +106,10 @@ def _test_audit_hook(name, args): # Ensure there's a non-ASCII character in env vars at all times to force # tests consider this case. See BPO-44647 for details. - os.environ.setdefault( - UNICODE_GUARD_ENV, - "\N{SMILING FACE WITH SUNGLASSES}", - ) + if TESTFN_UNDECODABLE and os.supports_bytes_environ: + os.environb.setdefault(UNICODE_GUARD_ENV.encode(), TESTFN_UNDECODABLE) + elif FS_NONASCII: + os.environ.setdefault(UNICODE_GUARD_ENV, FS_NONASCII) def replace_stdout(): diff --git a/Lib/test/test_regrtest.py b/Lib/test/test_regrtest.py index 3780feeda30e1..62e6c28280e61 100644 --- a/Lib/test/test_regrtest.py +++ b/Lib/test/test_regrtest.py @@ -1303,7 +1303,7 @@ def test_threading_excepthook(self): def test_unicode_guard_env(self): guard = os.environ.get(setup.UNICODE_GUARD_ENV) self.assertIsNotNone(guard, f"{setup.UNICODE_GUARD_ENV} not set") - if guard != "\N{SMILING FACE WITH SUNGLASSES}": + if guard.isascii(): # Skip to signify that the env var value was changed by the user; # possibly to something ASCII to work around Unicode issues. self.skipTest("Modified guard") diff --git a/Misc/NEWS.d/next/Tests/2022-04-25-11-16-36.gh-issue-91904.13Uvrz.rst b/Misc/NEWS.d/next/Tests/2022-04-25-11-16-36.gh-issue-91904.13Uvrz.rst new file mode 100644 index 0000000000000..31ddfc312866b --- /dev/null +++ b/Misc/NEWS.d/next/Tests/2022-04-25-11-16-36.gh-issue-91904.13Uvrz.rst @@ -0,0 +1,2 @@ +Fix initialization of :envvar:`PYTHONREGRTEST_UNICODE_GUARD` which prevented +running regression tests on non-UTF-8 locale. From webhook-mailer at python.org Mon Apr 25 11:56:31 2022 From: webhook-mailer at python.org (gvanrossum) Date: Mon, 25 Apr 2022 15:56:31 -0000 Subject: [Python-checkins] gh-91880: add try/except around `signal.signal` (#91881) Message-ID: https://github.com/python/cpython/commit/1cd8c29dace2dc6b91503803113fea4288ca842b commit: 1cd8c29dace2dc6b91503803113fea4288ca842b branch: main author: David Hewitt <1939362+davidhewitt at users.noreply.github.com> committer: gvanrossum date: 2022-04-25T08:56:20-07:00 summary: gh-91880: add try/except around `signal.signal` (#91881) Fixes gh-91880. files: M Lib/asyncio/runners.py M Lib/test/test_asyncio/test_runners.py diff --git a/Lib/asyncio/runners.py b/Lib/asyncio/runners.py index 2bb9ca331fd68..d274576b10134 100644 --- a/Lib/asyncio/runners.py +++ b/Lib/asyncio/runners.py @@ -100,7 +100,13 @@ def run(self, coro, *, context=None): and signal.getsignal(signal.SIGINT) is signal.default_int_handler ): sigint_handler = functools.partial(self._on_sigint, main_task=task) - signal.signal(signal.SIGINT, sigint_handler) + try: + signal.signal(signal.SIGINT, sigint_handler) + except ValueError: + # `signal.signal` may throw if `threading.main_thread` does + # not support signals (e.g. embedded interpreter with signals + # not registered - see gh-91880) + signal_handler = None else: sigint_handler = None diff --git a/Lib/test/test_asyncio/test_runners.py b/Lib/test/test_asyncio/test_runners.py index 42aa07a0e089d..0c2092147842c 100644 --- a/Lib/test/test_asyncio/test_runners.py +++ b/Lib/test/test_asyncio/test_runners.py @@ -3,10 +3,12 @@ import contextvars import gc import re +import signal import threading import unittest from unittest import mock +from unittest.mock import patch from test.test_asyncio import utils as test_utils @@ -374,6 +376,23 @@ async def coro(): with asyncio.Runner() as runner: with self.assertRaises(asyncio.CancelledError): runner.run(coro()) + + def test_signal_install_not_supported_ok(self): + # signal.signal() can throw if the "main thread" doensn't have signals enabled + assert threading.current_thread() is threading.main_thread() + + async def coro(): + pass + + with asyncio.Runner() as runner: + with patch.object( + signal, + "signal", + side_effect=ValueError( + "signal only works in main thread of the main interpreter" + ) + ): + runner.run(coro()) if __name__ == '__main__': From webhook-mailer at python.org Mon Apr 25 18:11:39 2022 From: webhook-mailer at python.org (vstinner) Date: Mon, 25 Apr 2022 22:11:39 -0000 Subject: [Python-checkins] gh-89653: PEP 670: Functions don't cast pointers (#91697) Message-ID: https://github.com/python/cpython/commit/61381d7da1233849b280706f11dbcae4deed949d commit: 61381d7da1233849b280706f11dbcae4deed949d branch: main author: Victor Stinner committer: vstinner date: 2022-04-26T00:11:34+02:00 summary: gh-89653: PEP 670: Functions don't cast pointers (#91697) In the limited C API version 3.11 and newer, the following functions no longer cast their object pointer argument with _PyObject_CAST() or _PyObject_CAST_CONST(): * Py_REFCNT(), Py_TYPE(), Py_SIZE() * Py_SET_REFCNT(), Py_SET_TYPE(), Py_SET_SIZE() * Py_IS_TYPE() * Py_INCREF(), Py_DECREF() * Py_XINCREF(), Py_XDECREF() * Py_NewRef(), Py_XNewRef() * PyObject_TypeCheck() * PyType_Check() * PyType_CheckExact() Split Py_DECREF() implementation in 3 versions to make the code more readable. Update the xxlimited.c extension, which uses the limited C API version 3.11, to pass PyObject* to these functions. files: M Include/cpython/unicodeobject.h M Include/object.h M Modules/xxlimited.c diff --git a/Include/cpython/unicodeobject.h b/Include/cpython/unicodeobject.h index 992588c21c6ee..2712c583270c6 100644 --- a/Include/cpython/unicodeobject.h +++ b/Include/cpython/unicodeobject.h @@ -256,14 +256,18 @@ PyAPI_FUNC(int) _PyUnicode_CheckConsistency( static inline unsigned int PyUnicode_CHECK_INTERNED(PyObject *op) { return _PyASCIIObject_CAST(op)->state.interned; } -#define PyUnicode_CHECK_INTERNED(op) PyUnicode_CHECK_INTERNED(_PyObject_CAST(op)) +#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 < 0x030b0000 +# define PyUnicode_CHECK_INTERNED(op) PyUnicode_CHECK_INTERNED(_PyObject_CAST(op)) +#endif /* Fast check to determine whether an object is ready. Equivalent to: PyUnicode_IS_COMPACT(op) || _PyUnicodeObject_CAST(op)->data.any */ static inline unsigned int PyUnicode_IS_READY(PyObject *op) { return _PyASCIIObject_CAST(op)->state.ready; } -#define PyUnicode_IS_READY(op) PyUnicode_IS_READY(_PyObject_CAST(op)) +#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 < 0x030b0000 +# define PyUnicode_IS_READY(op) PyUnicode_IS_READY(_PyObject_CAST(op)) +#endif /* Return true if the string contains only ASCII characters, or 0 if not. The string may be compact (PyUnicode_IS_COMPACT_ASCII) or not, but must be @@ -272,21 +276,27 @@ static inline unsigned int PyUnicode_IS_ASCII(PyObject *op) { assert(PyUnicode_IS_READY(op)); return _PyASCIIObject_CAST(op)->state.ascii; } -#define PyUnicode_IS_ASCII(op) PyUnicode_IS_ASCII(_PyObject_CAST(op)) +#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 < 0x030b0000 +# define PyUnicode_IS_ASCII(op) PyUnicode_IS_ASCII(_PyObject_CAST(op)) +#endif /* Return true if the string is compact or 0 if not. No type checks or Ready calls are performed. */ static inline unsigned int PyUnicode_IS_COMPACT(PyObject *op) { return _PyASCIIObject_CAST(op)->state.compact; } -#define PyUnicode_IS_COMPACT(op) PyUnicode_IS_COMPACT(_PyObject_CAST(op)) +#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 < 0x030b0000 +# define PyUnicode_IS_COMPACT(op) PyUnicode_IS_COMPACT(_PyObject_CAST(op)) +#endif /* Return true if the string is a compact ASCII string (use PyASCIIObject structure), or 0 if not. No type checks or Ready calls are performed. */ static inline int PyUnicode_IS_COMPACT_ASCII(PyObject *op) { return (_PyASCIIObject_CAST(op)->state.ascii && PyUnicode_IS_COMPACT(op)); } -#define PyUnicode_IS_COMPACT_ASCII(op) PyUnicode_IS_COMPACT_ASCII(_PyObject_CAST(op)) +#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 < 0x030b0000 +# define PyUnicode_IS_COMPACT_ASCII(op) PyUnicode_IS_COMPACT_ASCII(_PyObject_CAST(op)) +#endif enum PyUnicode_Kind { /* String contains only wstr byte characters. This is only possible @@ -326,7 +336,9 @@ static inline void* PyUnicode_DATA(PyObject *op) { } return _PyUnicode_NONCOMPACT_DATA(op); } -#define PyUnicode_DATA(op) PyUnicode_DATA(_PyObject_CAST(op)) +#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 < 0x030b0000 +# define PyUnicode_DATA(op) PyUnicode_DATA(_PyObject_CAST(op)) +#endif /* Return pointers to the canonical representation cast to unsigned char, Py_UCS2, or Py_UCS4 for direct character access. @@ -344,7 +356,9 @@ static inline Py_ssize_t PyUnicode_GET_LENGTH(PyObject *op) { assert(PyUnicode_IS_READY(op)); return _PyASCIIObject_CAST(op)->length; } -#define PyUnicode_GET_LENGTH(op) PyUnicode_GET_LENGTH(_PyObject_CAST(op)) +#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 < 0x030b0000 +# define PyUnicode_GET_LENGTH(op) PyUnicode_GET_LENGTH(_PyObject_CAST(op)) +#endif /* Write into the canonical representation, this function does not do any sanity checks and is intended for usage in loops. The caller should cache the @@ -405,8 +419,10 @@ static inline Py_UCS4 PyUnicode_READ_CHAR(PyObject *unicode, Py_ssize_t index) assert(kind == PyUnicode_4BYTE_KIND); return PyUnicode_4BYTE_DATA(unicode)[index]; } -#define PyUnicode_READ_CHAR(unicode, index) \ - PyUnicode_READ_CHAR(_PyObject_CAST(unicode), (index)) +#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 < 0x030b0000 +# define PyUnicode_READ_CHAR(unicode, index) \ + PyUnicode_READ_CHAR(_PyObject_CAST(unicode), (index)) +#endif /* Return a maximum character value which is suitable for creating another string based on op. This is always an approximation but more efficient @@ -428,8 +444,10 @@ static inline Py_UCS4 PyUnicode_MAX_CHAR_VALUE(PyObject *op) assert(kind == PyUnicode_4BYTE_KIND); return 0x10ffffU; } -#define PyUnicode_MAX_CHAR_VALUE(op) \ - PyUnicode_MAX_CHAR_VALUE(_PyObject_CAST(op)) +#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 < 0x030b0000 +# define PyUnicode_MAX_CHAR_VALUE(op) \ + PyUnicode_MAX_CHAR_VALUE(_PyObject_CAST(op)) +#endif /* === Public API ========================================================= */ @@ -465,7 +483,9 @@ static inline int PyUnicode_READY(PyObject *op) } return _PyUnicode_Ready(op); } -#define PyUnicode_READY(op) PyUnicode_READY(_PyObject_CAST(op)) +#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 < 0x030b0000 +# define PyUnicode_READY(op) PyUnicode_READY(_PyObject_CAST(op)) +#endif /* Get a copy of a Unicode string. */ PyAPI_FUNC(PyObject*) _PyUnicode_Copy( @@ -606,7 +626,9 @@ static inline Py_ssize_t PyUnicode_WSTR_LENGTH(PyObject *op) return _PyCompactUnicodeObject_CAST(op)->wstr_length; } } -#define PyUnicode_WSTR_LENGTH(op) PyUnicode_WSTR_LENGTH(_PyObject_CAST(op)) +#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 < 0x030b0000 +# define PyUnicode_WSTR_LENGTH(op) PyUnicode_WSTR_LENGTH(_PyObject_CAST(op)) +#endif /* Returns the deprecated Py_UNICODE representation's size in code units (this includes surrogate pairs as 2 units). @@ -625,7 +647,9 @@ static inline Py_ssize_t PyUnicode_GET_SIZE(PyObject *op) return PyUnicode_WSTR_LENGTH(op); _Py_COMP_DIAG_POP } -#define PyUnicode_GET_SIZE(op) PyUnicode_GET_SIZE(_PyObject_CAST(op)) +#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 < 0x030b0000 +# define PyUnicode_GET_SIZE(op) PyUnicode_GET_SIZE(_PyObject_CAST(op)) +#endif Py_DEPRECATED(3.3) static inline Py_ssize_t PyUnicode_GET_DATA_SIZE(PyObject *op) @@ -635,7 +659,9 @@ static inline Py_ssize_t PyUnicode_GET_DATA_SIZE(PyObject *op) return PyUnicode_GET_SIZE(op) * Py_UNICODE_SIZE; _Py_COMP_DIAG_POP } -#define PyUnicode_GET_DATA_SIZE(op) PyUnicode_GET_DATA_SIZE(_PyObject_CAST(op)) +#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 < 0x030b0000 +# define PyUnicode_GET_DATA_SIZE(op) PyUnicode_GET_DATA_SIZE(_PyObject_CAST(op)) +#endif /* Alias for PyUnicode_AsUnicode(). This will create a wchar_t/Py_UNICODE representation on demand. Using this macro is very inefficient now, @@ -655,7 +681,9 @@ static inline Py_UNICODE* PyUnicode_AS_UNICODE(PyObject *op) return PyUnicode_AsUnicode(op); _Py_COMP_DIAG_POP } -#define PyUnicode_AS_UNICODE(op) PyUnicode_AS_UNICODE(_PyObject_CAST(op)) +#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 < 0x030b0000 +# define PyUnicode_AS_UNICODE(op) PyUnicode_AS_UNICODE(_PyObject_CAST(op)) +#endif Py_DEPRECATED(3.3) static inline const char* PyUnicode_AS_DATA(PyObject *op) @@ -665,7 +693,9 @@ static inline const char* PyUnicode_AS_DATA(PyObject *op) return (const char *)PyUnicode_AS_UNICODE(op); _Py_COMP_DIAG_POP } -#define PyUnicode_AS_DATA(op) PyUnicode_AS_DATA(_PyObject_CAST(op)) +#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 < 0x030b0000 +# define PyUnicode_AS_DATA(op) PyUnicode_AS_DATA(_PyObject_CAST(op)) +#endif /* --- _PyUnicodeWriter API ----------------------------------------------- */ diff --git a/Include/object.h b/Include/object.h index 8a45ddbf6057e..5622305f49f78 100644 --- a/Include/object.h +++ b/Include/object.h @@ -123,44 +123,58 @@ PyAPI_FUNC(int) Py_Is(PyObject *x, PyObject *y); static inline Py_ssize_t Py_REFCNT(PyObject *ob) { return ob->ob_refcnt; } -#define Py_REFCNT(ob) Py_REFCNT(_PyObject_CAST(ob)) +#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 < 0x030b0000 +# define Py_REFCNT(ob) Py_REFCNT(_PyObject_CAST(ob)) +#endif // bpo-39573: The Py_SET_TYPE() function must be used to set an object type. static inline PyTypeObject* Py_TYPE(PyObject *ob) { return ob->ob_type; } -#define Py_TYPE(ob) Py_TYPE(_PyObject_CAST(ob)) +#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 < 0x030b0000 +# define Py_TYPE(ob) Py_TYPE(_PyObject_CAST(ob)) +#endif // bpo-39573: The Py_SET_SIZE() function must be used to set an object size. static inline Py_ssize_t Py_SIZE(PyVarObject *ob) { return ob->ob_size; } -#define Py_SIZE(ob) Py_SIZE(_PyVarObject_CAST(ob)) +#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 < 0x030b0000 +# define Py_SIZE(ob) Py_SIZE(_PyVarObject_CAST(ob)) +#endif static inline int Py_IS_TYPE(PyObject *ob, PyTypeObject *type) { return Py_TYPE(ob) == type; } -#define Py_IS_TYPE(ob, type) Py_IS_TYPE(_PyObject_CAST(ob), type) +#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 < 0x030b0000 +# define Py_IS_TYPE(ob, type) Py_IS_TYPE(_PyObject_CAST(ob), type) +#endif static inline void Py_SET_REFCNT(PyObject *ob, Py_ssize_t refcnt) { ob->ob_refcnt = refcnt; } -#define Py_SET_REFCNT(ob, refcnt) Py_SET_REFCNT(_PyObject_CAST(ob), refcnt) +#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 < 0x030b0000 +# define Py_SET_REFCNT(ob, refcnt) Py_SET_REFCNT(_PyObject_CAST(ob), refcnt) +#endif static inline void Py_SET_TYPE(PyObject *ob, PyTypeObject *type) { ob->ob_type = type; } -#define Py_SET_TYPE(ob, type) Py_SET_TYPE(_PyObject_CAST(ob), type) +#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 < 0x030b0000 +# define Py_SET_TYPE(ob, type) Py_SET_TYPE(_PyObject_CAST(ob), type) +#endif static inline void Py_SET_SIZE(PyVarObject *ob, Py_ssize_t size) { ob->ob_size = size; } -#define Py_SET_SIZE(ob, size) Py_SET_SIZE(_PyVarObject_CAST(ob), size) +#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 < 0x030b0000 +# define Py_SET_SIZE(ob, size) Py_SET_SIZE(_PyVarObject_CAST(ob), size) +#endif /* @@ -247,7 +261,9 @@ PyAPI_FUNC(int) PyType_IsSubtype(PyTypeObject *, PyTypeObject *); static inline int PyObject_TypeCheck(PyObject *ob, PyTypeObject *type) { return Py_IS_TYPE(ob, type) || PyType_IsSubtype(Py_TYPE(ob), type); } -#define PyObject_TypeCheck(ob, type) PyObject_TypeCheck(_PyObject_CAST(ob), type) +#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 < 0x030b0000 +# define PyObject_TypeCheck(ob, type) PyObject_TypeCheck(_PyObject_CAST(ob), type) +#endif PyAPI_DATA(PyTypeObject) PyType_Type; /* built-in 'type' */ PyAPI_DATA(PyTypeObject) PyBaseObject_Type; /* built-in 'object' */ @@ -485,39 +501,43 @@ static inline void Py_INCREF(PyObject *op) op->ob_refcnt++; #endif } -#define Py_INCREF(op) Py_INCREF(_PyObject_CAST(op)) - -static inline void Py_DECREF( -#if defined(Py_REF_DEBUG) && !(defined(Py_LIMITED_API) && Py_LIMITED_API+0 >= 0x030A0000) - const char *filename, int lineno, +#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 < 0x030b0000 +# define Py_INCREF(op) Py_INCREF(_PyObject_CAST(op)) #endif - PyObject *op) -{ + + #if defined(Py_REF_DEBUG) && defined(Py_LIMITED_API) && Py_LIMITED_API+0 >= 0x030A0000 - // Stable ABI for Python 3.10 built in debug mode. +// Stable ABI for limited C API version 3.10 of Python debug build +static inline void Py_DECREF(PyObject *op) { _Py_DecRef(op); -#else - // Non-limited C API and limited C API for Python 3.9 and older access - // directly PyObject.ob_refcnt. -#ifdef Py_REF_DEBUG +} +#define Py_DECREF(op) Py_DECREF(_PyObject_CAST(op)) + +#elif defined(Py_REF_DEBUG) +static inline void Py_DECREF(const char *filename, int lineno, PyObject *op) +{ _Py_RefTotal--; -#endif if (--op->ob_refcnt != 0) { -#ifdef Py_REF_DEBUG if (op->ob_refcnt < 0) { _Py_NegativeRefcount(filename, lineno, op); } -#endif } else { _Py_Dealloc(op); } -#endif } -#if defined(Py_REF_DEBUG) && !(defined(Py_LIMITED_API) && Py_LIMITED_API+0 >= 0x030A0000) -# define Py_DECREF(op) Py_DECREF(__FILE__, __LINE__, _PyObject_CAST(op)) +#define Py_DECREF(op) Py_DECREF(__FILE__, __LINE__, _PyObject_CAST(op)) + #else -# define Py_DECREF(op) Py_DECREF(_PyObject_CAST(op)) +static inline void Py_DECREF(PyObject *op) +{ + // Non-limited C API and limited C API for Python 3.9 and older access + // directly PyObject.ob_refcnt. + if (--op->ob_refcnt == 0) { + _Py_Dealloc(op); + } +} +#define Py_DECREF(op) Py_DECREF(_PyObject_CAST(op)) #endif @@ -571,8 +591,9 @@ static inline void Py_XINCREF(PyObject *op) Py_INCREF(op); } } - -#define Py_XINCREF(op) Py_XINCREF(_PyObject_CAST(op)) +#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 < 0x030b0000 +# define Py_XINCREF(op) Py_XINCREF(_PyObject_CAST(op)) +#endif static inline void Py_XDECREF(PyObject *op) { @@ -580,8 +601,9 @@ static inline void Py_XDECREF(PyObject *op) Py_DECREF(op); } } - -#define Py_XDECREF(op) Py_XDECREF(_PyObject_CAST(op)) +#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 < 0x030b0000 +# define Py_XDECREF(op) Py_XDECREF(_PyObject_CAST(op)) +#endif // Create a new strong reference to an object: // increment the reference count of the object and return the object. @@ -605,8 +627,13 @@ static inline PyObject* _Py_XNewRef(PyObject *obj) // Py_NewRef() and Py_XNewRef() are exported as functions for the stable ABI. // Names overridden with macros by static inline functions for best // performances. -#define Py_NewRef(obj) _Py_NewRef(_PyObject_CAST(obj)) -#define Py_XNewRef(obj) _Py_XNewRef(_PyObject_CAST(obj)) +#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 < 0x030b0000 +# define Py_NewRef(obj) _Py_NewRef(_PyObject_CAST(obj)) +# define Py_XNewRef(obj) _Py_XNewRef(_PyObject_CAST(obj)) +#else +# define Py_NewRef(obj) _Py_NewRef(obj) +# define Py_XNewRef(obj) _Py_XNewRef(obj) +#endif /* @@ -749,14 +776,18 @@ PyType_HasFeature(PyTypeObject *type, unsigned long feature) static inline int PyType_Check(PyObject *op) { return PyType_FastSubclass(Py_TYPE(op), Py_TPFLAGS_TYPE_SUBCLASS); } -#define PyType_Check(op) PyType_Check(_PyObject_CAST(op)) +#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 < 0x030b0000 +# define PyType_Check(op) PyType_Check(_PyObject_CAST(op)) +#endif #define _PyType_CAST(op) (assert(PyType_Check(op)), (PyTypeObject*)(op)) static inline int PyType_CheckExact(PyObject *op) { return Py_IS_TYPE(op, &PyType_Type); } -#define PyType_CheckExact(op) PyType_CheckExact(_PyObject_CAST(op)) +#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 < 0x030b0000 +# define PyType_CheckExact(op) PyType_CheckExact(_PyObject_CAST(op)) +#endif #ifdef __cplusplus } diff --git a/Modules/xxlimited.c b/Modules/xxlimited.c index 16d1b8311c62c..c36bf5ce8e1f7 100644 --- a/Modules/xxlimited.c +++ b/Modules/xxlimited.c @@ -110,12 +110,13 @@ newXxoObject(PyObject *module) /* Xxo finalization */ static int -Xxo_traverse(XxoObject *self, visitproc visit, void *arg) +Xxo_traverse(PyObject *self_obj, visitproc visit, void *arg) { // Visit the type - Py_VISIT(Py_TYPE(self)); + Py_VISIT(Py_TYPE(self_obj)); // Visit the attribute dict + XxoObject *self = (XxoObject *)self_obj; Py_VISIT(self->x_attr); return 0; } @@ -128,13 +129,14 @@ Xxo_clear(XxoObject *self) } static void -Xxo_finalize(XxoObject *self) +Xxo_finalize(PyObject *self_obj) { + XxoObject *self = (XxoObject *)self_obj; Py_CLEAR(self->x_attr); } static void -Xxo_dealloc(XxoObject *self) +Xxo_dealloc(PyObject *self) { Xxo_finalize(self); PyTypeObject *tp = Py_TYPE(self); From webhook-mailer at python.org Mon Apr 25 18:13:40 2022 From: webhook-mailer at python.org (vstinner) Date: Mon, 25 Apr 2022 22:13:40 -0000 Subject: [Python-checkins] gh-64783: Fix signal.NSIG value on FreeBSD (#91929) Message-ID: https://github.com/python/cpython/commit/20cc69528677b3e5191139d1cb587531f4893b55 commit: 20cc69528677b3e5191139d1cb587531f4893b55 branch: main author: Victor Stinner committer: vstinner date: 2022-04-26T00:13:31+02:00 summary: gh-64783: Fix signal.NSIG value on FreeBSD (#91929) Fix signal.NSIG value on FreeBSD to accept signal numbers greater than 32, like signal.SIGRTMIN and signal.SIGRTMAX. * Add Py_NSIG constant. * Add pycore_signal.h internal header file. * _Py_Sigset_Converter() now includes the range of valid signals in the error message. files: A Include/internal/pycore_signal.h A Misc/NEWS.d/next/Library/2022-04-25-18-30-20.gh-issue-64783.HFtERN.rst M Doc/library/signal.rst M Include/internal/pycore_pylifecycle.h M Lib/test/test_signal.py M Makefile.pre.in M Modules/faulthandler.c M Modules/posixmodule.c M Modules/signalmodule.c M PCbuild/pythoncore.vcxproj M PCbuild/pythoncore.vcxproj.filters diff --git a/Doc/library/signal.rst b/Doc/library/signal.rst index fdc9846f6642a..678411d4f1705 100644 --- a/Doc/library/signal.rst +++ b/Doc/library/signal.rst @@ -266,6 +266,7 @@ The variables defined in the :mod:`signal` module are: .. data:: NSIG One more than the number of the highest signal number. + Use :func:`valid_signals` to get valid signal numbers. .. data:: ITIMER_REAL diff --git a/Include/internal/pycore_pylifecycle.h b/Include/internal/pycore_pylifecycle.h index 295505f1f3735..b4718b8ade235 100644 --- a/Include/internal/pycore_pylifecycle.h +++ b/Include/internal/pycore_pylifecycle.h @@ -8,24 +8,8 @@ extern "C" { # error "this header requires Py_BUILD_CORE define" #endif -#ifdef HAVE_SIGNAL_H -#include -#endif - #include "pycore_runtime.h" // _PyRuntimeState -#ifndef NSIG -# if defined(_NSIG) -# define NSIG _NSIG /* For BSD/SysV */ -# elif defined(_SIGMAX) -# define NSIG (_SIGMAX + 1) /* For QNX */ -# elif defined(SIGMAX) -# define NSIG (SIGMAX + 1) /* For djgpp */ -# else -# define NSIG 64 /* Use a reasonable default value */ -# endif -#endif - /* Forward declarations */ struct _PyArgv; struct pyruntimestate; diff --git a/Include/internal/pycore_signal.h b/Include/internal/pycore_signal.h new file mode 100644 index 0000000000000..b921dd170e9f6 --- /dev/null +++ b/Include/internal/pycore_signal.h @@ -0,0 +1,35 @@ +// Define Py_NSIG constant for signal handling. + +#ifndef Py_INTERNAL_SIGNAL_H +#define Py_INTERNAL_SIGNAL_H +#ifdef __cplusplus +extern "C" { +#endif + +#ifndef Py_BUILD_CORE +# error "this header requires Py_BUILD_CORE define" +#endif + +#include // NSIG + +#ifdef _SIG_MAXSIG + // gh-91145: On FreeBSD, defines NSIG as 32: it doesn't include + // realtime signals: [SIGRTMIN,SIGRTMAX]. Use _SIG_MAXSIG instead. For + // example on x86-64 FreeBSD 13, SIGRTMAX is 126 and _SIG_MAXSIG is 128. +# define Py_NSIG _SIG_MAXSIG +#elif defined(NSIG) +# define Py_NSIG NSIG +#elif defined(_NSIG) +# define Py_NSIG _NSIG // BSD/SysV +#elif defined(_SIGMAX) +# define Py_NSIG (_SIGMAX + 1) // QNX +#elif defined(SIGMAX) +# define Py_NSIG (SIGMAX + 1) // djgpp +#else +# define Py_NSIG 64 // Use a reasonable default value +#endif + +#ifdef __cplusplus +} +#endif +#endif // !Py_INTERNAL_SIGNAL_H diff --git a/Lib/test/test_signal.py b/Lib/test/test_signal.py index ea13c59ec711e..d38992db7b84d 100644 --- a/Lib/test/test_signal.py +++ b/Lib/test/test_signal.py @@ -116,6 +116,16 @@ def test_valid_signals(self): self.assertNotIn(signal.NSIG, s) self.assertLess(len(s), signal.NSIG) + # gh-91145: Make sure that all SIGxxx constants exposed by the Python + # signal module have a number in the [0; signal.NSIG-1] range. + for name in dir(signal): + if not name.startswith("SIG"): + continue + with self.subTest(name=name): + signum = getattr(signal, name) + self.assertGreaterEqual(signum, 0) + self.assertLess(signum, signal.NSIG) + @unittest.skipUnless(sys.executable, "sys.executable required.") @support.requires_subprocess() def test_keyboard_interrupt_exit_code(self): diff --git a/Makefile.pre.in b/Makefile.pre.in index d9f821dd14e17..3952f5b542b43 100644 --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -1622,6 +1622,7 @@ PYTHON_HEADERS= \ $(srcdir)/Include/internal/pycore_pystate.h \ $(srcdir)/Include/internal/pycore_runtime.h \ $(srcdir)/Include/internal/pycore_runtime_init.h \ + $(srcdir)/Include/internal/pycore_signal.h \ $(srcdir)/Include/internal/pycore_sliceobject.h \ $(srcdir)/Include/internal/pycore_strhex.h \ $(srcdir)/Include/internal/pycore_structseq.h \ diff --git a/Misc/NEWS.d/next/Library/2022-04-25-18-30-20.gh-issue-64783.HFtERN.rst b/Misc/NEWS.d/next/Library/2022-04-25-18-30-20.gh-issue-64783.HFtERN.rst new file mode 100644 index 0000000000000..41814a6eb76da --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-25-18-30-20.gh-issue-64783.HFtERN.rst @@ -0,0 +1,3 @@ +Fix :data:`signal.NSIG` value on FreeBSD to accept signal numbers greater than +32, like :data:`signal.SIGRTMIN` and :data:`signal.SIGRTMAX`. Patch by Victor +Stinner. diff --git a/Modules/faulthandler.c b/Modules/faulthandler.c index 91f96665a4a30..4f709edb479a2 100644 --- a/Modules/faulthandler.c +++ b/Modules/faulthandler.c @@ -2,6 +2,7 @@ #include "pycore_initconfig.h" // _PyStatus_ERR #include "pycore_pyerrors.h" // _Py_DumpExtensionModules #include "pycore_pystate.h" // _PyThreadState_GET() +#include "pycore_signal.h" // Py_NSIG #include "pycore_traceback.h" // _Py_DumpTracebackThreads #include "frameobject.h" @@ -115,19 +116,6 @@ typedef struct { static user_signal_t *user_signals; -/* the following macros come from Python: Modules/signalmodule.c */ -#ifndef NSIG -# if defined(_NSIG) -# define NSIG _NSIG /* For BSD/SysV */ -# elif defined(_SIGMAX) -# define NSIG (_SIGMAX + 1) /* For QNX */ -# elif defined(SIGMAX) -# define NSIG (SIGMAX + 1) /* For djgpp */ -# else -# define NSIG 64 /* Use a reasonable default value */ -# endif -#endif - static void faulthandler_user(int signum); #endif /* FAULTHANDLER_USER */ @@ -896,7 +884,7 @@ check_signum(int signum) return 0; } } - if (signum < 1 || NSIG <= signum) { + if (signum < 1 || Py_NSIG <= signum) { PyErr_SetString(PyExc_ValueError, "signal number out of range"); return 0; } @@ -935,7 +923,7 @@ faulthandler_register_py(PyObject *self, return NULL; if (user_signals == NULL) { - user_signals = PyMem_Calloc(NSIG, sizeof(user_signal_t)); + user_signals = PyMem_Calloc(Py_NSIG, sizeof(user_signal_t)); if (user_signals == NULL) return PyErr_NoMemory(); } @@ -1215,7 +1203,7 @@ faulthandler_traverse(PyObject *module, visitproc visit, void *arg) Py_VISIT(thread.file); #ifdef FAULTHANDLER_USER if (user_signals != NULL) { - for (size_t signum=0; signum < NSIG; signum++) + for (size_t signum=0; signum < Py_NSIG; signum++) Py_VISIT(user_signals[signum].file); } #endif @@ -1416,7 +1404,7 @@ void _PyFaulthandler_Fini(void) #ifdef FAULTHANDLER_USER /* user */ if (user_signals != NULL) { - for (size_t signum=0; signum < NSIG; signum++) { + for (size_t signum=0; signum < Py_NSIG; signum++) { faulthandler_unregister(&user_signals[signum], signum); } PyMem_Free(user_signals); diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c index a9132a78994ce..a2ea5079969e9 100644 --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -29,6 +29,7 @@ #include "pycore_moduleobject.h" // _PyModule_GetState() #include "pycore_object.h" // _PyObject_LookupSpecial() #include "pycore_pystate.h" // _PyInterpreterState_GET() +#include "pycore_signal.h" // Py_NSIG #include "structmember.h" // PyMemberDef #ifndef MS_WINDOWS @@ -1503,10 +1504,11 @@ _Py_Sigset_Converter(PyObject *obj, void *addr) while ((item = PyIter_Next(iterator)) != NULL) { signum = PyLong_AsLongAndOverflow(item, &overflow); Py_DECREF(item); - if (signum <= 0 || signum >= NSIG) { + if (signum <= 0 || signum >= Py_NSIG) { if (overflow || signum != -1 || !PyErr_Occurred()) { PyErr_Format(PyExc_ValueError, - "signal number %ld out of range", signum); + "signal number %ld out of range [1; %i]", + signum, Py_NSIG - 1); } goto error; } diff --git a/Modules/signalmodule.c b/Modules/signalmodule.c index 02c58ff538ecd..b602338e4f91e 100644 --- a/Modules/signalmodule.c +++ b/Modules/signalmodule.c @@ -7,13 +7,13 @@ #include "pycore_atomic.h" // _Py_atomic_int #include "pycore_call.h" // _PyObject_Call() #include "pycore_ceval.h" // _PyEval_SignalReceived() +#include "pycore_emscripten_signal.h" // _Py_CHECK_EMSCRIPTEN_SIGNALS #include "pycore_fileutils.h" // _Py_BEGIN_SUPPRESS_IPH #include "pycore_frame.h" // _PyInterpreterFrame #include "pycore_moduleobject.h" // _PyModule_GetState() #include "pycore_pyerrors.h" // _PyErr_SetString() -#include "pycore_pylifecycle.h" // NSIG #include "pycore_pystate.h" // _PyThreadState_GET() -#include "pycore_emscripten_signal.h" // _Py_CHECK_EMSCRIPTEN_SIGNALS +#include "pycore_signal.h" // Py_NSIG #ifndef MS_WINDOWS # include "posixmodule.h" @@ -106,7 +106,7 @@ static volatile struct { * (even though it would probably be otherwise, anyway). */ _Py_atomic_address func; -} Handlers[NSIG]; +} Handlers[Py_NSIG]; #ifdef MS_WINDOWS #define INVALID_FD ((SOCKET_T)-1) @@ -542,7 +542,7 @@ signal_signal_impl(PyObject *module, int signalnum, PyObject *handler) "of the main interpreter"); return NULL; } - if (signalnum < 1 || signalnum >= NSIG) { + if (signalnum < 1 || signalnum >= Py_NSIG) { _PyErr_SetString(tstate, PyExc_ValueError, "signal number out of range"); return NULL; @@ -601,7 +601,7 @@ signal_getsignal_impl(PyObject *module, int signalnum) /*[clinic end generated code: output=35b3e0e796fd555e input=ac23a00f19dfa509]*/ { PyObject *old_handler; - if (signalnum < 1 || signalnum >= NSIG) { + if (signalnum < 1 || signalnum >= Py_NSIG) { PyErr_SetString(PyExc_ValueError, "signal number out of range"); return NULL; @@ -634,7 +634,7 @@ signal_strsignal_impl(PyObject *module, int signalnum) { const char *res; - if (signalnum < 1 || signalnum >= NSIG) { + if (signalnum < 1 || signalnum >= Py_NSIG) { PyErr_SetString(PyExc_ValueError, "signal number out of range"); return NULL; @@ -712,7 +712,7 @@ static PyObject * signal_siginterrupt_impl(PyObject *module, int signalnum, int flag) /*[clinic end generated code: output=063816243d85dd19 input=4160acacca3e2099]*/ { - if (signalnum < 1 || signalnum >= NSIG) { + if (signalnum < 1 || signalnum >= Py_NSIG) { PyErr_SetString(PyExc_ValueError, "signal number out of range"); return NULL; @@ -964,7 +964,7 @@ sigset_to_set(sigset_t mask) if (result == NULL) return NULL; - for (sig = 1; sig < NSIG; sig++) { + for (sig = 1; sig < Py_NSIG; sig++) { if (sigismember(&mask, sig) != 1) continue; @@ -1439,13 +1439,15 @@ the first is the signal number, the second is the interrupted stack frame."); static int signal_add_constants(PyObject *module) { + if (PyModule_AddIntConstant(module, "NSIG", Py_NSIG) < 0) { + return -1; + } + #define ADD_INT_MACRO(macro) \ if (PyModule_AddIntConstant(module, #macro, macro) < 0) { \ return -1; \ } - ADD_INT_MACRO(NSIG); - // SIG_xxx pthread_sigmask() constants #ifdef SIG_BLOCK ADD_INT_MACRO(SIG_BLOCK); @@ -1605,7 +1607,7 @@ static int signal_get_set_handlers(signal_state_t *state, PyObject *mod_dict) { // Get signal handlers - for (int signum = 1; signum < NSIG; signum++) { + for (int signum = 1; signum < Py_NSIG; signum++) { void (*c_handler)(int) = PyOS_getsig(signum); PyObject *func; if (c_handler == SIG_DFL) { @@ -1762,7 +1764,7 @@ _PySignal_Fini(void) signal_state_t *state = &signal_global_state; // Restore default signals and clear handlers - for (int signum = 1; signum < NSIG; signum++) { + for (int signum = 1; signum < Py_NSIG; signum++) { PyObject *func = get_handler(signum); _Py_atomic_store_relaxed(&Handlers[signum].tripped, 0); set_handler(signum, NULL); @@ -1828,7 +1830,7 @@ _PyErr_CheckSignalsTstate(PyThreadState *tstate) _PyInterpreterFrame *frame = tstate->cframe->current_frame; signal_state_t *state = &signal_global_state; - for (int i = 1; i < NSIG; i++) { + for (int i = 1; i < Py_NSIG; i++) { if (!_Py_atomic_load_relaxed(&Handlers[i].tripped)) { continue; } @@ -1905,7 +1907,7 @@ _PyErr_CheckSignals(void) int PyErr_SetInterruptEx(int signum) { - if (signum < 1 || signum >= NSIG) { + if (signum < 1 || signum >= Py_NSIG) { return -1; } @@ -1995,7 +1997,7 @@ _PySignal_Init(int install_signal_handlers) } #endif - for (int signum = 1; signum < NSIG; signum++) { + for (int signum = 1; signum < Py_NSIG; signum++) { _Py_atomic_store_relaxed(&Handlers[signum].tripped, 0); } @@ -2045,7 +2047,7 @@ _clear_pending_signals(void) } _Py_atomic_store(&is_tripped, 0); - for (int i = 1; i < NSIG; ++i) { + for (int i = 1; i < Py_NSIG; ++i) { _Py_atomic_store_relaxed(&Handlers[i].tripped, 0); } } diff --git a/PCbuild/pythoncore.vcxproj b/PCbuild/pythoncore.vcxproj index 78bbec1e104e4..3ce116d2babb0 100644 --- a/PCbuild/pythoncore.vcxproj +++ b/PCbuild/pythoncore.vcxproj @@ -238,6 +238,7 @@ + diff --git a/PCbuild/pythoncore.vcxproj.filters b/PCbuild/pythoncore.vcxproj.filters index 0a9d5454ba957..542d551045686 100644 --- a/PCbuild/pythoncore.vcxproj.filters +++ b/PCbuild/pythoncore.vcxproj.filters @@ -618,6 +618,9 @@ Include\internal + + Include\internal + Include\internal From webhook-mailer at python.org Mon Apr 25 18:14:34 2022 From: webhook-mailer at python.org (vstinner) Date: Mon, 25 Apr 2022 22:14:34 -0000 Subject: [Python-checkins] gh-91719: Add pycore_opcode.h internal header file (#91906) Message-ID: https://github.com/python/cpython/commit/64a54e511debaac6d3a6e53685824fce435c440c commit: 64a54e511debaac6d3a6e53685824fce435c440c branch: main author: Victor Stinner committer: vstinner date: 2022-04-26T00:14:30+02:00 summary: gh-91719: Add pycore_opcode.h internal header file (#91906) Move the following API from Include/opcode.h (public C API) to a new Include/internal/pycore_opcode.h header file (internal C API): * EXTRA_CASES * _PyOpcode_Caches * _PyOpcode_Deopt * _PyOpcode_Jump * _PyOpcode_OpName * _PyOpcode_RelativeJump files: A Include/internal/pycore_opcode.h M Include/opcode.h M Makefile.pre.in M Objects/codeobject.c M Objects/frameobject.c M Objects/genobject.c M PCbuild/regen.targets M Python/ceval.c M Python/compile.c M Python/specialize.c M Tools/scripts/generate_opcode_h.py diff --git a/Include/internal/pycore_opcode.h b/Include/internal/pycore_opcode.h new file mode 100644 index 0000000000000..46c9986b500c8 --- /dev/null +++ b/Include/internal/pycore_opcode.h @@ -0,0 +1,581 @@ +// Auto-generated by Tools/scripts/generate_opcode_h.py from Lib/opcode.py + +#ifndef Py_INTERNAL_OPCODE_H +#define Py_INTERNAL_OPCODE_H +#ifdef __cplusplus +extern "C" { +#endif + +#ifndef Py_BUILD_CORE +# error "this header requires Py_BUILD_CORE define" +#endif + +#include "opcode.h" + +extern const uint8_t _PyOpcode_Caches[256]; + +extern const uint8_t _PyOpcode_Deopt[256]; + +#ifdef NEED_OPCODE_TABLES +static const uint32_t _PyOpcode_RelativeJump[8] = { + 0U, + 0U, + 536870912U, + 135118848U, + 4163U, + 122880U, + 0U, + 0U, +}; +static const uint32_t _PyOpcode_Jump[8] = { + 0U, + 0U, + 536870912U, + 135118848U, + 4163U, + 122880U, + 0U, + 0U, +}; + +const uint8_t _PyOpcode_Caches[256] = { + [BINARY_SUBSCR] = 4, + [STORE_SUBSCR] = 1, + [UNPACK_SEQUENCE] = 1, + [STORE_ATTR] = 4, + [LOAD_ATTR] = 4, + [COMPARE_OP] = 2, + [LOAD_GLOBAL] = 5, + [BINARY_OP] = 1, + [LOAD_METHOD] = 10, + [PRECALL] = 1, + [CALL] = 4, +}; + +const uint8_t _PyOpcode_Deopt[256] = { + [ASYNC_GEN_WRAP] = ASYNC_GEN_WRAP, + [BEFORE_ASYNC_WITH] = BEFORE_ASYNC_WITH, + [BEFORE_WITH] = BEFORE_WITH, + [BINARY_OP] = BINARY_OP, + [BINARY_OP_ADAPTIVE] = BINARY_OP, + [BINARY_OP_ADD_FLOAT] = BINARY_OP, + [BINARY_OP_ADD_INT] = BINARY_OP, + [BINARY_OP_ADD_UNICODE] = BINARY_OP, + [BINARY_OP_INPLACE_ADD_UNICODE] = BINARY_OP, + [BINARY_OP_MULTIPLY_FLOAT] = BINARY_OP, + [BINARY_OP_MULTIPLY_INT] = BINARY_OP, + [BINARY_OP_SUBTRACT_FLOAT] = BINARY_OP, + [BINARY_OP_SUBTRACT_INT] = BINARY_OP, + [BINARY_SUBSCR] = BINARY_SUBSCR, + [BINARY_SUBSCR_ADAPTIVE] = BINARY_SUBSCR, + [BINARY_SUBSCR_DICT] = BINARY_SUBSCR, + [BINARY_SUBSCR_GETITEM] = BINARY_SUBSCR, + [BINARY_SUBSCR_LIST_INT] = BINARY_SUBSCR, + [BINARY_SUBSCR_TUPLE_INT] = BINARY_SUBSCR, + [BUILD_CONST_KEY_MAP] = BUILD_CONST_KEY_MAP, + [BUILD_LIST] = BUILD_LIST, + [BUILD_MAP] = BUILD_MAP, + [BUILD_SET] = BUILD_SET, + [BUILD_SLICE] = BUILD_SLICE, + [BUILD_STRING] = BUILD_STRING, + [BUILD_TUPLE] = BUILD_TUPLE, + [CACHE] = CACHE, + [CALL] = CALL, + [CALL_ADAPTIVE] = CALL, + [CALL_FUNCTION_EX] = CALL_FUNCTION_EX, + [CALL_PY_EXACT_ARGS] = CALL, + [CALL_PY_WITH_DEFAULTS] = CALL, + [CHECK_EG_MATCH] = CHECK_EG_MATCH, + [CHECK_EXC_MATCH] = CHECK_EXC_MATCH, + [COMPARE_OP] = COMPARE_OP, + [COMPARE_OP_ADAPTIVE] = COMPARE_OP, + [COMPARE_OP_FLOAT_JUMP] = COMPARE_OP, + [COMPARE_OP_INT_JUMP] = COMPARE_OP, + [COMPARE_OP_STR_JUMP] = COMPARE_OP, + [CONTAINS_OP] = CONTAINS_OP, + [COPY] = COPY, + [COPY_FREE_VARS] = COPY_FREE_VARS, + [DELETE_ATTR] = DELETE_ATTR, + [DELETE_DEREF] = DELETE_DEREF, + [DELETE_FAST] = DELETE_FAST, + [DELETE_GLOBAL] = DELETE_GLOBAL, + [DELETE_NAME] = DELETE_NAME, + [DELETE_SUBSCR] = DELETE_SUBSCR, + [DICT_MERGE] = DICT_MERGE, + [DICT_UPDATE] = DICT_UPDATE, + [END_ASYNC_FOR] = END_ASYNC_FOR, + [EXTENDED_ARG] = EXTENDED_ARG, + [FORMAT_VALUE] = FORMAT_VALUE, + [FOR_ITER] = FOR_ITER, + [GET_AITER] = GET_AITER, + [GET_ANEXT] = GET_ANEXT, + [GET_AWAITABLE] = GET_AWAITABLE, + [GET_ITER] = GET_ITER, + [GET_LEN] = GET_LEN, + [GET_YIELD_FROM_ITER] = GET_YIELD_FROM_ITER, + [IMPORT_FROM] = IMPORT_FROM, + [IMPORT_NAME] = IMPORT_NAME, + [IMPORT_STAR] = IMPORT_STAR, + [IS_OP] = IS_OP, + [JUMP_BACKWARD] = JUMP_BACKWARD, + [JUMP_BACKWARD_NO_INTERRUPT] = JUMP_BACKWARD_NO_INTERRUPT, + [JUMP_BACKWARD_QUICK] = JUMP_BACKWARD, + [JUMP_FORWARD] = JUMP_FORWARD, + [JUMP_IF_FALSE_OR_POP] = JUMP_IF_FALSE_OR_POP, + [JUMP_IF_TRUE_OR_POP] = JUMP_IF_TRUE_OR_POP, + [KW_NAMES] = KW_NAMES, + [LIST_APPEND] = LIST_APPEND, + [LIST_EXTEND] = LIST_EXTEND, + [LIST_TO_TUPLE] = LIST_TO_TUPLE, + [LOAD_ASSERTION_ERROR] = LOAD_ASSERTION_ERROR, + [LOAD_ATTR] = LOAD_ATTR, + [LOAD_ATTR_ADAPTIVE] = LOAD_ATTR, + [LOAD_ATTR_INSTANCE_VALUE] = LOAD_ATTR, + [LOAD_ATTR_MODULE] = LOAD_ATTR, + [LOAD_ATTR_SLOT] = LOAD_ATTR, + [LOAD_ATTR_WITH_HINT] = LOAD_ATTR, + [LOAD_BUILD_CLASS] = LOAD_BUILD_CLASS, + [LOAD_CLASSDEREF] = LOAD_CLASSDEREF, + [LOAD_CLOSURE] = LOAD_CLOSURE, + [LOAD_CONST] = LOAD_CONST, + [LOAD_CONST__LOAD_FAST] = LOAD_CONST, + [LOAD_DEREF] = LOAD_DEREF, + [LOAD_FAST] = LOAD_FAST, + [LOAD_FAST__LOAD_CONST] = LOAD_FAST, + [LOAD_FAST__LOAD_FAST] = LOAD_FAST, + [LOAD_GLOBAL] = LOAD_GLOBAL, + [LOAD_GLOBAL_ADAPTIVE] = LOAD_GLOBAL, + [LOAD_GLOBAL_BUILTIN] = LOAD_GLOBAL, + [LOAD_GLOBAL_MODULE] = LOAD_GLOBAL, + [LOAD_METHOD] = LOAD_METHOD, + [LOAD_METHOD_ADAPTIVE] = LOAD_METHOD, + [LOAD_METHOD_CLASS] = LOAD_METHOD, + [LOAD_METHOD_MODULE] = LOAD_METHOD, + [LOAD_METHOD_NO_DICT] = LOAD_METHOD, + [LOAD_METHOD_WITH_DICT] = LOAD_METHOD, + [LOAD_METHOD_WITH_VALUES] = LOAD_METHOD, + [LOAD_NAME] = LOAD_NAME, + [MAKE_CELL] = MAKE_CELL, + [MAKE_FUNCTION] = MAKE_FUNCTION, + [MAP_ADD] = MAP_ADD, + [MATCH_CLASS] = MATCH_CLASS, + [MATCH_KEYS] = MATCH_KEYS, + [MATCH_MAPPING] = MATCH_MAPPING, + [MATCH_SEQUENCE] = MATCH_SEQUENCE, + [NOP] = NOP, + [POP_EXCEPT] = POP_EXCEPT, + [POP_JUMP_BACKWARD_IF_FALSE] = POP_JUMP_BACKWARD_IF_FALSE, + [POP_JUMP_BACKWARD_IF_NONE] = POP_JUMP_BACKWARD_IF_NONE, + [POP_JUMP_BACKWARD_IF_NOT_NONE] = POP_JUMP_BACKWARD_IF_NOT_NONE, + [POP_JUMP_BACKWARD_IF_TRUE] = POP_JUMP_BACKWARD_IF_TRUE, + [POP_JUMP_FORWARD_IF_FALSE] = POP_JUMP_FORWARD_IF_FALSE, + [POP_JUMP_FORWARD_IF_NONE] = POP_JUMP_FORWARD_IF_NONE, + [POP_JUMP_FORWARD_IF_NOT_NONE] = POP_JUMP_FORWARD_IF_NOT_NONE, + [POP_JUMP_FORWARD_IF_TRUE] = POP_JUMP_FORWARD_IF_TRUE, + [POP_TOP] = POP_TOP, + [PRECALL] = PRECALL, + [PRECALL_ADAPTIVE] = PRECALL, + [PRECALL_BOUND_METHOD] = PRECALL, + [PRECALL_BUILTIN_CLASS] = PRECALL, + [PRECALL_BUILTIN_FAST_WITH_KEYWORDS] = PRECALL, + [PRECALL_METHOD_DESCRIPTOR_FAST_WITH_KEYWORDS] = PRECALL, + [PRECALL_NO_KW_BUILTIN_FAST] = PRECALL, + [PRECALL_NO_KW_BUILTIN_O] = PRECALL, + [PRECALL_NO_KW_ISINSTANCE] = PRECALL, + [PRECALL_NO_KW_LEN] = PRECALL, + [PRECALL_NO_KW_LIST_APPEND] = PRECALL, + [PRECALL_NO_KW_METHOD_DESCRIPTOR_FAST] = PRECALL, + [PRECALL_NO_KW_METHOD_DESCRIPTOR_NOARGS] = PRECALL, + [PRECALL_NO_KW_METHOD_DESCRIPTOR_O] = PRECALL, + [PRECALL_NO_KW_STR_1] = PRECALL, + [PRECALL_NO_KW_TUPLE_1] = PRECALL, + [PRECALL_NO_KW_TYPE_1] = PRECALL, + [PRECALL_PYFUNC] = PRECALL, + [PREP_RERAISE_STAR] = PREP_RERAISE_STAR, + [PRINT_EXPR] = PRINT_EXPR, + [PUSH_EXC_INFO] = PUSH_EXC_INFO, + [PUSH_NULL] = PUSH_NULL, + [RAISE_VARARGS] = RAISE_VARARGS, + [RERAISE] = RERAISE, + [RESUME] = RESUME, + [RESUME_QUICK] = RESUME, + [RETURN_GENERATOR] = RETURN_GENERATOR, + [RETURN_VALUE] = RETURN_VALUE, + [SEND] = SEND, + [SETUP_ANNOTATIONS] = SETUP_ANNOTATIONS, + [SET_ADD] = SET_ADD, + [SET_UPDATE] = SET_UPDATE, + [STORE_ATTR] = STORE_ATTR, + [STORE_ATTR_ADAPTIVE] = STORE_ATTR, + [STORE_ATTR_INSTANCE_VALUE] = STORE_ATTR, + [STORE_ATTR_SLOT] = STORE_ATTR, + [STORE_ATTR_WITH_HINT] = STORE_ATTR, + [STORE_DEREF] = STORE_DEREF, + [STORE_FAST] = STORE_FAST, + [STORE_FAST__LOAD_FAST] = STORE_FAST, + [STORE_FAST__STORE_FAST] = STORE_FAST, + [STORE_GLOBAL] = STORE_GLOBAL, + [STORE_NAME] = STORE_NAME, + [STORE_SUBSCR] = STORE_SUBSCR, + [STORE_SUBSCR_ADAPTIVE] = STORE_SUBSCR, + [STORE_SUBSCR_DICT] = STORE_SUBSCR, + [STORE_SUBSCR_LIST_INT] = STORE_SUBSCR, + [SWAP] = SWAP, + [UNARY_INVERT] = UNARY_INVERT, + [UNARY_NEGATIVE] = UNARY_NEGATIVE, + [UNARY_NOT] = UNARY_NOT, + [UNARY_POSITIVE] = UNARY_POSITIVE, + [UNPACK_EX] = UNPACK_EX, + [UNPACK_SEQUENCE] = UNPACK_SEQUENCE, + [UNPACK_SEQUENCE_ADAPTIVE] = UNPACK_SEQUENCE, + [UNPACK_SEQUENCE_LIST] = UNPACK_SEQUENCE, + [UNPACK_SEQUENCE_TUPLE] = UNPACK_SEQUENCE, + [UNPACK_SEQUENCE_TWO_TUPLE] = UNPACK_SEQUENCE, + [WITH_EXCEPT_START] = WITH_EXCEPT_START, + [YIELD_VALUE] = YIELD_VALUE, +}; +#endif // NEED_OPCODE_TABLES + +#ifdef Py_DEBUG +static const char *const _PyOpcode_OpName[256] = { + [CACHE] = "CACHE", + [POP_TOP] = "POP_TOP", + [PUSH_NULL] = "PUSH_NULL", + [BINARY_OP_ADAPTIVE] = "BINARY_OP_ADAPTIVE", + [BINARY_OP_ADD_FLOAT] = "BINARY_OP_ADD_FLOAT", + [BINARY_OP_ADD_INT] = "BINARY_OP_ADD_INT", + [BINARY_OP_ADD_UNICODE] = "BINARY_OP_ADD_UNICODE", + [BINARY_OP_INPLACE_ADD_UNICODE] = "BINARY_OP_INPLACE_ADD_UNICODE", + [BINARY_OP_MULTIPLY_FLOAT] = "BINARY_OP_MULTIPLY_FLOAT", + [NOP] = "NOP", + [UNARY_POSITIVE] = "UNARY_POSITIVE", + [UNARY_NEGATIVE] = "UNARY_NEGATIVE", + [UNARY_NOT] = "UNARY_NOT", + [BINARY_OP_MULTIPLY_INT] = "BINARY_OP_MULTIPLY_INT", + [BINARY_OP_SUBTRACT_FLOAT] = "BINARY_OP_SUBTRACT_FLOAT", + [UNARY_INVERT] = "UNARY_INVERT", + [BINARY_OP_SUBTRACT_INT] = "BINARY_OP_SUBTRACT_INT", + [BINARY_SUBSCR_ADAPTIVE] = "BINARY_SUBSCR_ADAPTIVE", + [BINARY_SUBSCR_DICT] = "BINARY_SUBSCR_DICT", + [BINARY_SUBSCR_GETITEM] = "BINARY_SUBSCR_GETITEM", + [BINARY_SUBSCR_LIST_INT] = "BINARY_SUBSCR_LIST_INT", + [BINARY_SUBSCR_TUPLE_INT] = "BINARY_SUBSCR_TUPLE_INT", + [CALL_ADAPTIVE] = "CALL_ADAPTIVE", + [CALL_PY_EXACT_ARGS] = "CALL_PY_EXACT_ARGS", + [CALL_PY_WITH_DEFAULTS] = "CALL_PY_WITH_DEFAULTS", + [BINARY_SUBSCR] = "BINARY_SUBSCR", + [COMPARE_OP_ADAPTIVE] = "COMPARE_OP_ADAPTIVE", + [COMPARE_OP_FLOAT_JUMP] = "COMPARE_OP_FLOAT_JUMP", + [COMPARE_OP_INT_JUMP] = "COMPARE_OP_INT_JUMP", + [COMPARE_OP_STR_JUMP] = "COMPARE_OP_STR_JUMP", + [GET_LEN] = "GET_LEN", + [MATCH_MAPPING] = "MATCH_MAPPING", + [MATCH_SEQUENCE] = "MATCH_SEQUENCE", + [MATCH_KEYS] = "MATCH_KEYS", + [JUMP_BACKWARD_QUICK] = "JUMP_BACKWARD_QUICK", + [PUSH_EXC_INFO] = "PUSH_EXC_INFO", + [CHECK_EXC_MATCH] = "CHECK_EXC_MATCH", + [CHECK_EG_MATCH] = "CHECK_EG_MATCH", + [LOAD_ATTR_ADAPTIVE] = "LOAD_ATTR_ADAPTIVE", + [LOAD_ATTR_INSTANCE_VALUE] = "LOAD_ATTR_INSTANCE_VALUE", + [LOAD_ATTR_MODULE] = "LOAD_ATTR_MODULE", + [LOAD_ATTR_SLOT] = "LOAD_ATTR_SLOT", + [LOAD_ATTR_WITH_HINT] = "LOAD_ATTR_WITH_HINT", + [LOAD_CONST__LOAD_FAST] = "LOAD_CONST__LOAD_FAST", + [LOAD_FAST__LOAD_CONST] = "LOAD_FAST__LOAD_CONST", + [LOAD_FAST__LOAD_FAST] = "LOAD_FAST__LOAD_FAST", + [LOAD_GLOBAL_ADAPTIVE] = "LOAD_GLOBAL_ADAPTIVE", + [LOAD_GLOBAL_BUILTIN] = "LOAD_GLOBAL_BUILTIN", + [LOAD_GLOBAL_MODULE] = "LOAD_GLOBAL_MODULE", + [WITH_EXCEPT_START] = "WITH_EXCEPT_START", + [GET_AITER] = "GET_AITER", + [GET_ANEXT] = "GET_ANEXT", + [BEFORE_ASYNC_WITH] = "BEFORE_ASYNC_WITH", + [BEFORE_WITH] = "BEFORE_WITH", + [END_ASYNC_FOR] = "END_ASYNC_FOR", + [LOAD_METHOD_ADAPTIVE] = "LOAD_METHOD_ADAPTIVE", + [LOAD_METHOD_CLASS] = "LOAD_METHOD_CLASS", + [LOAD_METHOD_MODULE] = "LOAD_METHOD_MODULE", + [LOAD_METHOD_NO_DICT] = "LOAD_METHOD_NO_DICT", + [LOAD_METHOD_WITH_DICT] = "LOAD_METHOD_WITH_DICT", + [STORE_SUBSCR] = "STORE_SUBSCR", + [DELETE_SUBSCR] = "DELETE_SUBSCR", + [LOAD_METHOD_WITH_VALUES] = "LOAD_METHOD_WITH_VALUES", + [PRECALL_ADAPTIVE] = "PRECALL_ADAPTIVE", + [PRECALL_BOUND_METHOD] = "PRECALL_BOUND_METHOD", + [PRECALL_BUILTIN_CLASS] = "PRECALL_BUILTIN_CLASS", + [PRECALL_BUILTIN_FAST_WITH_KEYWORDS] = "PRECALL_BUILTIN_FAST_WITH_KEYWORDS", + [PRECALL_METHOD_DESCRIPTOR_FAST_WITH_KEYWORDS] = "PRECALL_METHOD_DESCRIPTOR_FAST_WITH_KEYWORDS", + [GET_ITER] = "GET_ITER", + [GET_YIELD_FROM_ITER] = "GET_YIELD_FROM_ITER", + [PRINT_EXPR] = "PRINT_EXPR", + [LOAD_BUILD_CLASS] = "LOAD_BUILD_CLASS", + [PRECALL_NO_KW_BUILTIN_FAST] = "PRECALL_NO_KW_BUILTIN_FAST", + [PRECALL_NO_KW_BUILTIN_O] = "PRECALL_NO_KW_BUILTIN_O", + [LOAD_ASSERTION_ERROR] = "LOAD_ASSERTION_ERROR", + [RETURN_GENERATOR] = "RETURN_GENERATOR", + [PRECALL_NO_KW_ISINSTANCE] = "PRECALL_NO_KW_ISINSTANCE", + [PRECALL_NO_KW_LEN] = "PRECALL_NO_KW_LEN", + [PRECALL_NO_KW_LIST_APPEND] = "PRECALL_NO_KW_LIST_APPEND", + [PRECALL_NO_KW_METHOD_DESCRIPTOR_FAST] = "PRECALL_NO_KW_METHOD_DESCRIPTOR_FAST", + [PRECALL_NO_KW_METHOD_DESCRIPTOR_NOARGS] = "PRECALL_NO_KW_METHOD_DESCRIPTOR_NOARGS", + [PRECALL_NO_KW_METHOD_DESCRIPTOR_O] = "PRECALL_NO_KW_METHOD_DESCRIPTOR_O", + [LIST_TO_TUPLE] = "LIST_TO_TUPLE", + [RETURN_VALUE] = "RETURN_VALUE", + [IMPORT_STAR] = "IMPORT_STAR", + [SETUP_ANNOTATIONS] = "SETUP_ANNOTATIONS", + [YIELD_VALUE] = "YIELD_VALUE", + [ASYNC_GEN_WRAP] = "ASYNC_GEN_WRAP", + [PREP_RERAISE_STAR] = "PREP_RERAISE_STAR", + [POP_EXCEPT] = "POP_EXCEPT", + [STORE_NAME] = "STORE_NAME", + [DELETE_NAME] = "DELETE_NAME", + [UNPACK_SEQUENCE] = "UNPACK_SEQUENCE", + [FOR_ITER] = "FOR_ITER", + [UNPACK_EX] = "UNPACK_EX", + [STORE_ATTR] = "STORE_ATTR", + [DELETE_ATTR] = "DELETE_ATTR", + [STORE_GLOBAL] = "STORE_GLOBAL", + [DELETE_GLOBAL] = "DELETE_GLOBAL", + [SWAP] = "SWAP", + [LOAD_CONST] = "LOAD_CONST", + [LOAD_NAME] = "LOAD_NAME", + [BUILD_TUPLE] = "BUILD_TUPLE", + [BUILD_LIST] = "BUILD_LIST", + [BUILD_SET] = "BUILD_SET", + [BUILD_MAP] = "BUILD_MAP", + [LOAD_ATTR] = "LOAD_ATTR", + [COMPARE_OP] = "COMPARE_OP", + [IMPORT_NAME] = "IMPORT_NAME", + [IMPORT_FROM] = "IMPORT_FROM", + [JUMP_FORWARD] = "JUMP_FORWARD", + [JUMP_IF_FALSE_OR_POP] = "JUMP_IF_FALSE_OR_POP", + [JUMP_IF_TRUE_OR_POP] = "JUMP_IF_TRUE_OR_POP", + [PRECALL_NO_KW_STR_1] = "PRECALL_NO_KW_STR_1", + [POP_JUMP_FORWARD_IF_FALSE] = "POP_JUMP_FORWARD_IF_FALSE", + [POP_JUMP_FORWARD_IF_TRUE] = "POP_JUMP_FORWARD_IF_TRUE", + [LOAD_GLOBAL] = "LOAD_GLOBAL", + [IS_OP] = "IS_OP", + [CONTAINS_OP] = "CONTAINS_OP", + [RERAISE] = "RERAISE", + [COPY] = "COPY", + [PRECALL_NO_KW_TUPLE_1] = "PRECALL_NO_KW_TUPLE_1", + [BINARY_OP] = "BINARY_OP", + [SEND] = "SEND", + [LOAD_FAST] = "LOAD_FAST", + [STORE_FAST] = "STORE_FAST", + [DELETE_FAST] = "DELETE_FAST", + [PRECALL_NO_KW_TYPE_1] = "PRECALL_NO_KW_TYPE_1", + [POP_JUMP_FORWARD_IF_NOT_NONE] = "POP_JUMP_FORWARD_IF_NOT_NONE", + [POP_JUMP_FORWARD_IF_NONE] = "POP_JUMP_FORWARD_IF_NONE", + [RAISE_VARARGS] = "RAISE_VARARGS", + [GET_AWAITABLE] = "GET_AWAITABLE", + [MAKE_FUNCTION] = "MAKE_FUNCTION", + [BUILD_SLICE] = "BUILD_SLICE", + [JUMP_BACKWARD_NO_INTERRUPT] = "JUMP_BACKWARD_NO_INTERRUPT", + [MAKE_CELL] = "MAKE_CELL", + [LOAD_CLOSURE] = "LOAD_CLOSURE", + [LOAD_DEREF] = "LOAD_DEREF", + [STORE_DEREF] = "STORE_DEREF", + [DELETE_DEREF] = "DELETE_DEREF", + [JUMP_BACKWARD] = "JUMP_BACKWARD", + [PRECALL_PYFUNC] = "PRECALL_PYFUNC", + [CALL_FUNCTION_EX] = "CALL_FUNCTION_EX", + [RESUME_QUICK] = "RESUME_QUICK", + [EXTENDED_ARG] = "EXTENDED_ARG", + [LIST_APPEND] = "LIST_APPEND", + [SET_ADD] = "SET_ADD", + [MAP_ADD] = "MAP_ADD", + [LOAD_CLASSDEREF] = "LOAD_CLASSDEREF", + [COPY_FREE_VARS] = "COPY_FREE_VARS", + [STORE_ATTR_ADAPTIVE] = "STORE_ATTR_ADAPTIVE", + [RESUME] = "RESUME", + [MATCH_CLASS] = "MATCH_CLASS", + [STORE_ATTR_INSTANCE_VALUE] = "STORE_ATTR_INSTANCE_VALUE", + [STORE_ATTR_SLOT] = "STORE_ATTR_SLOT", + [FORMAT_VALUE] = "FORMAT_VALUE", + [BUILD_CONST_KEY_MAP] = "BUILD_CONST_KEY_MAP", + [BUILD_STRING] = "BUILD_STRING", + [STORE_ATTR_WITH_HINT] = "STORE_ATTR_WITH_HINT", + [STORE_FAST__LOAD_FAST] = "STORE_FAST__LOAD_FAST", + [LOAD_METHOD] = "LOAD_METHOD", + [STORE_FAST__STORE_FAST] = "STORE_FAST__STORE_FAST", + [LIST_EXTEND] = "LIST_EXTEND", + [SET_UPDATE] = "SET_UPDATE", + [DICT_MERGE] = "DICT_MERGE", + [DICT_UPDATE] = "DICT_UPDATE", + [PRECALL] = "PRECALL", + [STORE_SUBSCR_ADAPTIVE] = "STORE_SUBSCR_ADAPTIVE", + [STORE_SUBSCR_DICT] = "STORE_SUBSCR_DICT", + [STORE_SUBSCR_LIST_INT] = "STORE_SUBSCR_LIST_INT", + [UNPACK_SEQUENCE_ADAPTIVE] = "UNPACK_SEQUENCE_ADAPTIVE", + [CALL] = "CALL", + [KW_NAMES] = "KW_NAMES", + [POP_JUMP_BACKWARD_IF_NOT_NONE] = "POP_JUMP_BACKWARD_IF_NOT_NONE", + [POP_JUMP_BACKWARD_IF_NONE] = "POP_JUMP_BACKWARD_IF_NONE", + [POP_JUMP_BACKWARD_IF_FALSE] = "POP_JUMP_BACKWARD_IF_FALSE", + [POP_JUMP_BACKWARD_IF_TRUE] = "POP_JUMP_BACKWARD_IF_TRUE", + [UNPACK_SEQUENCE_LIST] = "UNPACK_SEQUENCE_LIST", + [UNPACK_SEQUENCE_TUPLE] = "UNPACK_SEQUENCE_TUPLE", + [UNPACK_SEQUENCE_TWO_TUPLE] = "UNPACK_SEQUENCE_TWO_TUPLE", + [180] = "<180>", + [181] = "<181>", + [182] = "<182>", + [183] = "<183>", + [184] = "<184>", + [185] = "<185>", + [186] = "<186>", + [187] = "<187>", + [188] = "<188>", + [189] = "<189>", + [190] = "<190>", + [191] = "<191>", + [192] = "<192>", + [193] = "<193>", + [194] = "<194>", + [195] = "<195>", + [196] = "<196>", + [197] = "<197>", + [198] = "<198>", + [199] = "<199>", + [200] = "<200>", + [201] = "<201>", + [202] = "<202>", + [203] = "<203>", + [204] = "<204>", + [205] = "<205>", + [206] = "<206>", + [207] = "<207>", + [208] = "<208>", + [209] = "<209>", + [210] = "<210>", + [211] = "<211>", + [212] = "<212>", + [213] = "<213>", + [214] = "<214>", + [215] = "<215>", + [216] = "<216>", + [217] = "<217>", + [218] = "<218>", + [219] = "<219>", + [220] = "<220>", + [221] = "<221>", + [222] = "<222>", + [223] = "<223>", + [224] = "<224>", + [225] = "<225>", + [226] = "<226>", + [227] = "<227>", + [228] = "<228>", + [229] = "<229>", + [230] = "<230>", + [231] = "<231>", + [232] = "<232>", + [233] = "<233>", + [234] = "<234>", + [235] = "<235>", + [236] = "<236>", + [237] = "<237>", + [238] = "<238>", + [239] = "<239>", + [240] = "<240>", + [241] = "<241>", + [242] = "<242>", + [243] = "<243>", + [244] = "<244>", + [245] = "<245>", + [246] = "<246>", + [247] = "<247>", + [248] = "<248>", + [249] = "<249>", + [250] = "<250>", + [251] = "<251>", + [252] = "<252>", + [253] = "<253>", + [254] = "<254>", + [DO_TRACING] = "DO_TRACING", +}; +#endif + +#define EXTRA_CASES \ + case 180: \ + case 181: \ + case 182: \ + case 183: \ + case 184: \ + case 185: \ + case 186: \ + case 187: \ + case 188: \ + case 189: \ + case 190: \ + case 191: \ + case 192: \ + case 193: \ + case 194: \ + case 195: \ + case 196: \ + case 197: \ + case 198: \ + case 199: \ + case 200: \ + case 201: \ + case 202: \ + case 203: \ + case 204: \ + case 205: \ + case 206: \ + case 207: \ + case 208: \ + case 209: \ + case 210: \ + case 211: \ + case 212: \ + case 213: \ + case 214: \ + case 215: \ + case 216: \ + case 217: \ + case 218: \ + case 219: \ + case 220: \ + case 221: \ + case 222: \ + case 223: \ + case 224: \ + case 225: \ + case 226: \ + case 227: \ + case 228: \ + case 229: \ + case 230: \ + case 231: \ + case 232: \ + case 233: \ + case 234: \ + case 235: \ + case 236: \ + case 237: \ + case 238: \ + case 239: \ + case 240: \ + case 241: \ + case 242: \ + case 243: \ + case 244: \ + case 245: \ + case 246: \ + case 247: \ + case 248: \ + case 249: \ + case 250: \ + case 251: \ + case 252: \ + case 253: \ + case 254: \ + ; + +#ifdef __cplusplus +} +#endif +#endif // !Py_INTERNAL_OPCODE_H diff --git a/Include/opcode.h b/Include/opcode.h index 89f56e27ed437..399847d4ba050 100644 --- a/Include/opcode.h +++ b/Include/opcode.h @@ -1,4 +1,5 @@ -/* Auto-generated by Tools/scripts/generate_opcode_h.py from Lib/opcode.py */ +// Auto-generated by Tools/scripts/generate_opcode_h.py from Lib/opcode.py + #ifndef Py_OPCODE_H #define Py_OPCODE_H #ifdef __cplusplus @@ -190,230 +191,6 @@ extern "C" { #define UNPACK_SEQUENCE_TWO_TUPLE 179 #define DO_TRACING 255 -extern const uint8_t _PyOpcode_Caches[256]; - -extern const uint8_t _PyOpcode_Deopt[256]; - -#ifdef NEED_OPCODE_TABLES -static const uint32_t _PyOpcode_RelativeJump[8] = { - 0U, - 0U, - 536870912U, - 135118848U, - 4163U, - 122880U, - 0U, - 0U, -}; -static const uint32_t _PyOpcode_Jump[8] = { - 0U, - 0U, - 536870912U, - 135118848U, - 4163U, - 122880U, - 0U, - 0U, -}; - -const uint8_t _PyOpcode_Caches[256] = { - [BINARY_SUBSCR] = 4, - [STORE_SUBSCR] = 1, - [UNPACK_SEQUENCE] = 1, - [STORE_ATTR] = 4, - [LOAD_ATTR] = 4, - [COMPARE_OP] = 2, - [LOAD_GLOBAL] = 5, - [BINARY_OP] = 1, - [LOAD_METHOD] = 10, - [PRECALL] = 1, - [CALL] = 4, -}; - -const uint8_t _PyOpcode_Deopt[256] = { - [ASYNC_GEN_WRAP] = ASYNC_GEN_WRAP, - [BEFORE_ASYNC_WITH] = BEFORE_ASYNC_WITH, - [BEFORE_WITH] = BEFORE_WITH, - [BINARY_OP] = BINARY_OP, - [BINARY_OP_ADAPTIVE] = BINARY_OP, - [BINARY_OP_ADD_FLOAT] = BINARY_OP, - [BINARY_OP_ADD_INT] = BINARY_OP, - [BINARY_OP_ADD_UNICODE] = BINARY_OP, - [BINARY_OP_INPLACE_ADD_UNICODE] = BINARY_OP, - [BINARY_OP_MULTIPLY_FLOAT] = BINARY_OP, - [BINARY_OP_MULTIPLY_INT] = BINARY_OP, - [BINARY_OP_SUBTRACT_FLOAT] = BINARY_OP, - [BINARY_OP_SUBTRACT_INT] = BINARY_OP, - [BINARY_SUBSCR] = BINARY_SUBSCR, - [BINARY_SUBSCR_ADAPTIVE] = BINARY_SUBSCR, - [BINARY_SUBSCR_DICT] = BINARY_SUBSCR, - [BINARY_SUBSCR_GETITEM] = BINARY_SUBSCR, - [BINARY_SUBSCR_LIST_INT] = BINARY_SUBSCR, - [BINARY_SUBSCR_TUPLE_INT] = BINARY_SUBSCR, - [BUILD_CONST_KEY_MAP] = BUILD_CONST_KEY_MAP, - [BUILD_LIST] = BUILD_LIST, - [BUILD_MAP] = BUILD_MAP, - [BUILD_SET] = BUILD_SET, - [BUILD_SLICE] = BUILD_SLICE, - [BUILD_STRING] = BUILD_STRING, - [BUILD_TUPLE] = BUILD_TUPLE, - [CACHE] = CACHE, - [CALL] = CALL, - [CALL_ADAPTIVE] = CALL, - [CALL_FUNCTION_EX] = CALL_FUNCTION_EX, - [CALL_PY_EXACT_ARGS] = CALL, - [CALL_PY_WITH_DEFAULTS] = CALL, - [CHECK_EG_MATCH] = CHECK_EG_MATCH, - [CHECK_EXC_MATCH] = CHECK_EXC_MATCH, - [COMPARE_OP] = COMPARE_OP, - [COMPARE_OP_ADAPTIVE] = COMPARE_OP, - [COMPARE_OP_FLOAT_JUMP] = COMPARE_OP, - [COMPARE_OP_INT_JUMP] = COMPARE_OP, - [COMPARE_OP_STR_JUMP] = COMPARE_OP, - [CONTAINS_OP] = CONTAINS_OP, - [COPY] = COPY, - [COPY_FREE_VARS] = COPY_FREE_VARS, - [DELETE_ATTR] = DELETE_ATTR, - [DELETE_DEREF] = DELETE_DEREF, - [DELETE_FAST] = DELETE_FAST, - [DELETE_GLOBAL] = DELETE_GLOBAL, - [DELETE_NAME] = DELETE_NAME, - [DELETE_SUBSCR] = DELETE_SUBSCR, - [DICT_MERGE] = DICT_MERGE, - [DICT_UPDATE] = DICT_UPDATE, - [END_ASYNC_FOR] = END_ASYNC_FOR, - [EXTENDED_ARG] = EXTENDED_ARG, - [FORMAT_VALUE] = FORMAT_VALUE, - [FOR_ITER] = FOR_ITER, - [GET_AITER] = GET_AITER, - [GET_ANEXT] = GET_ANEXT, - [GET_AWAITABLE] = GET_AWAITABLE, - [GET_ITER] = GET_ITER, - [GET_LEN] = GET_LEN, - [GET_YIELD_FROM_ITER] = GET_YIELD_FROM_ITER, - [IMPORT_FROM] = IMPORT_FROM, - [IMPORT_NAME] = IMPORT_NAME, - [IMPORT_STAR] = IMPORT_STAR, - [IS_OP] = IS_OP, - [JUMP_BACKWARD] = JUMP_BACKWARD, - [JUMP_BACKWARD_NO_INTERRUPT] = JUMP_BACKWARD_NO_INTERRUPT, - [JUMP_BACKWARD_QUICK] = JUMP_BACKWARD, - [JUMP_FORWARD] = JUMP_FORWARD, - [JUMP_IF_FALSE_OR_POP] = JUMP_IF_FALSE_OR_POP, - [JUMP_IF_TRUE_OR_POP] = JUMP_IF_TRUE_OR_POP, - [KW_NAMES] = KW_NAMES, - [LIST_APPEND] = LIST_APPEND, - [LIST_EXTEND] = LIST_EXTEND, - [LIST_TO_TUPLE] = LIST_TO_TUPLE, - [LOAD_ASSERTION_ERROR] = LOAD_ASSERTION_ERROR, - [LOAD_ATTR] = LOAD_ATTR, - [LOAD_ATTR_ADAPTIVE] = LOAD_ATTR, - [LOAD_ATTR_INSTANCE_VALUE] = LOAD_ATTR, - [LOAD_ATTR_MODULE] = LOAD_ATTR, - [LOAD_ATTR_SLOT] = LOAD_ATTR, - [LOAD_ATTR_WITH_HINT] = LOAD_ATTR, - [LOAD_BUILD_CLASS] = LOAD_BUILD_CLASS, - [LOAD_CLASSDEREF] = LOAD_CLASSDEREF, - [LOAD_CLOSURE] = LOAD_CLOSURE, - [LOAD_CONST] = LOAD_CONST, - [LOAD_CONST__LOAD_FAST] = LOAD_CONST, - [LOAD_DEREF] = LOAD_DEREF, - [LOAD_FAST] = LOAD_FAST, - [LOAD_FAST__LOAD_CONST] = LOAD_FAST, - [LOAD_FAST__LOAD_FAST] = LOAD_FAST, - [LOAD_GLOBAL] = LOAD_GLOBAL, - [LOAD_GLOBAL_ADAPTIVE] = LOAD_GLOBAL, - [LOAD_GLOBAL_BUILTIN] = LOAD_GLOBAL, - [LOAD_GLOBAL_MODULE] = LOAD_GLOBAL, - [LOAD_METHOD] = LOAD_METHOD, - [LOAD_METHOD_ADAPTIVE] = LOAD_METHOD, - [LOAD_METHOD_CLASS] = LOAD_METHOD, - [LOAD_METHOD_MODULE] = LOAD_METHOD, - [LOAD_METHOD_NO_DICT] = LOAD_METHOD, - [LOAD_METHOD_WITH_DICT] = LOAD_METHOD, - [LOAD_METHOD_WITH_VALUES] = LOAD_METHOD, - [LOAD_NAME] = LOAD_NAME, - [MAKE_CELL] = MAKE_CELL, - [MAKE_FUNCTION] = MAKE_FUNCTION, - [MAP_ADD] = MAP_ADD, - [MATCH_CLASS] = MATCH_CLASS, - [MATCH_KEYS] = MATCH_KEYS, - [MATCH_MAPPING] = MATCH_MAPPING, - [MATCH_SEQUENCE] = MATCH_SEQUENCE, - [NOP] = NOP, - [POP_EXCEPT] = POP_EXCEPT, - [POP_JUMP_BACKWARD_IF_FALSE] = POP_JUMP_BACKWARD_IF_FALSE, - [POP_JUMP_BACKWARD_IF_NONE] = POP_JUMP_BACKWARD_IF_NONE, - [POP_JUMP_BACKWARD_IF_NOT_NONE] = POP_JUMP_BACKWARD_IF_NOT_NONE, - [POP_JUMP_BACKWARD_IF_TRUE] = POP_JUMP_BACKWARD_IF_TRUE, - [POP_JUMP_FORWARD_IF_FALSE] = POP_JUMP_FORWARD_IF_FALSE, - [POP_JUMP_FORWARD_IF_NONE] = POP_JUMP_FORWARD_IF_NONE, - [POP_JUMP_FORWARD_IF_NOT_NONE] = POP_JUMP_FORWARD_IF_NOT_NONE, - [POP_JUMP_FORWARD_IF_TRUE] = POP_JUMP_FORWARD_IF_TRUE, - [POP_TOP] = POP_TOP, - [PRECALL] = PRECALL, - [PRECALL_ADAPTIVE] = PRECALL, - [PRECALL_BOUND_METHOD] = PRECALL, - [PRECALL_BUILTIN_CLASS] = PRECALL, - [PRECALL_BUILTIN_FAST_WITH_KEYWORDS] = PRECALL, - [PRECALL_METHOD_DESCRIPTOR_FAST_WITH_KEYWORDS] = PRECALL, - [PRECALL_NO_KW_BUILTIN_FAST] = PRECALL, - [PRECALL_NO_KW_BUILTIN_O] = PRECALL, - [PRECALL_NO_KW_ISINSTANCE] = PRECALL, - [PRECALL_NO_KW_LEN] = PRECALL, - [PRECALL_NO_KW_LIST_APPEND] = PRECALL, - [PRECALL_NO_KW_METHOD_DESCRIPTOR_FAST] = PRECALL, - [PRECALL_NO_KW_METHOD_DESCRIPTOR_NOARGS] = PRECALL, - [PRECALL_NO_KW_METHOD_DESCRIPTOR_O] = PRECALL, - [PRECALL_NO_KW_STR_1] = PRECALL, - [PRECALL_NO_KW_TUPLE_1] = PRECALL, - [PRECALL_NO_KW_TYPE_1] = PRECALL, - [PRECALL_PYFUNC] = PRECALL, - [PREP_RERAISE_STAR] = PREP_RERAISE_STAR, - [PRINT_EXPR] = PRINT_EXPR, - [PUSH_EXC_INFO] = PUSH_EXC_INFO, - [PUSH_NULL] = PUSH_NULL, - [RAISE_VARARGS] = RAISE_VARARGS, - [RERAISE] = RERAISE, - [RESUME] = RESUME, - [RESUME_QUICK] = RESUME, - [RETURN_GENERATOR] = RETURN_GENERATOR, - [RETURN_VALUE] = RETURN_VALUE, - [SEND] = SEND, - [SETUP_ANNOTATIONS] = SETUP_ANNOTATIONS, - [SET_ADD] = SET_ADD, - [SET_UPDATE] = SET_UPDATE, - [STORE_ATTR] = STORE_ATTR, - [STORE_ATTR_ADAPTIVE] = STORE_ATTR, - [STORE_ATTR_INSTANCE_VALUE] = STORE_ATTR, - [STORE_ATTR_SLOT] = STORE_ATTR, - [STORE_ATTR_WITH_HINT] = STORE_ATTR, - [STORE_DEREF] = STORE_DEREF, - [STORE_FAST] = STORE_FAST, - [STORE_FAST__LOAD_FAST] = STORE_FAST, - [STORE_FAST__STORE_FAST] = STORE_FAST, - [STORE_GLOBAL] = STORE_GLOBAL, - [STORE_NAME] = STORE_NAME, - [STORE_SUBSCR] = STORE_SUBSCR, - [STORE_SUBSCR_ADAPTIVE] = STORE_SUBSCR, - [STORE_SUBSCR_DICT] = STORE_SUBSCR, - [STORE_SUBSCR_LIST_INT] = STORE_SUBSCR, - [SWAP] = SWAP, - [UNARY_INVERT] = UNARY_INVERT, - [UNARY_NEGATIVE] = UNARY_NEGATIVE, - [UNARY_NOT] = UNARY_NOT, - [UNARY_POSITIVE] = UNARY_POSITIVE, - [UNPACK_EX] = UNPACK_EX, - [UNPACK_SEQUENCE] = UNPACK_SEQUENCE, - [UNPACK_SEQUENCE_ADAPTIVE] = UNPACK_SEQUENCE, - [UNPACK_SEQUENCE_LIST] = UNPACK_SEQUENCE, - [UNPACK_SEQUENCE_TUPLE] = UNPACK_SEQUENCE, - [UNPACK_SEQUENCE_TWO_TUPLE] = UNPACK_SEQUENCE, - [WITH_EXCEPT_START] = WITH_EXCEPT_START, - [YIELD_VALUE] = YIELD_VALUE, -}; -#endif /* OPCODE_TABLES */ - #define HAS_CONST(op) (false\ || ((op) == 100) \ || ((op) == 172) \ @@ -446,345 +223,6 @@ const uint8_t _PyOpcode_Deopt[256] = { #define NB_INPLACE_TRUE_DIVIDE 24 #define NB_INPLACE_XOR 25 -#ifdef Py_DEBUG -static const char *const _PyOpcode_OpName[256] = { - [CACHE] = "CACHE", - [POP_TOP] = "POP_TOP", - [PUSH_NULL] = "PUSH_NULL", - [BINARY_OP_ADAPTIVE] = "BINARY_OP_ADAPTIVE", - [BINARY_OP_ADD_FLOAT] = "BINARY_OP_ADD_FLOAT", - [BINARY_OP_ADD_INT] = "BINARY_OP_ADD_INT", - [BINARY_OP_ADD_UNICODE] = "BINARY_OP_ADD_UNICODE", - [BINARY_OP_INPLACE_ADD_UNICODE] = "BINARY_OP_INPLACE_ADD_UNICODE", - [BINARY_OP_MULTIPLY_FLOAT] = "BINARY_OP_MULTIPLY_FLOAT", - [NOP] = "NOP", - [UNARY_POSITIVE] = "UNARY_POSITIVE", - [UNARY_NEGATIVE] = "UNARY_NEGATIVE", - [UNARY_NOT] = "UNARY_NOT", - [BINARY_OP_MULTIPLY_INT] = "BINARY_OP_MULTIPLY_INT", - [BINARY_OP_SUBTRACT_FLOAT] = "BINARY_OP_SUBTRACT_FLOAT", - [UNARY_INVERT] = "UNARY_INVERT", - [BINARY_OP_SUBTRACT_INT] = "BINARY_OP_SUBTRACT_INT", - [BINARY_SUBSCR_ADAPTIVE] = "BINARY_SUBSCR_ADAPTIVE", - [BINARY_SUBSCR_DICT] = "BINARY_SUBSCR_DICT", - [BINARY_SUBSCR_GETITEM] = "BINARY_SUBSCR_GETITEM", - [BINARY_SUBSCR_LIST_INT] = "BINARY_SUBSCR_LIST_INT", - [BINARY_SUBSCR_TUPLE_INT] = "BINARY_SUBSCR_TUPLE_INT", - [CALL_ADAPTIVE] = "CALL_ADAPTIVE", - [CALL_PY_EXACT_ARGS] = "CALL_PY_EXACT_ARGS", - [CALL_PY_WITH_DEFAULTS] = "CALL_PY_WITH_DEFAULTS", - [BINARY_SUBSCR] = "BINARY_SUBSCR", - [COMPARE_OP_ADAPTIVE] = "COMPARE_OP_ADAPTIVE", - [COMPARE_OP_FLOAT_JUMP] = "COMPARE_OP_FLOAT_JUMP", - [COMPARE_OP_INT_JUMP] = "COMPARE_OP_INT_JUMP", - [COMPARE_OP_STR_JUMP] = "COMPARE_OP_STR_JUMP", - [GET_LEN] = "GET_LEN", - [MATCH_MAPPING] = "MATCH_MAPPING", - [MATCH_SEQUENCE] = "MATCH_SEQUENCE", - [MATCH_KEYS] = "MATCH_KEYS", - [JUMP_BACKWARD_QUICK] = "JUMP_BACKWARD_QUICK", - [PUSH_EXC_INFO] = "PUSH_EXC_INFO", - [CHECK_EXC_MATCH] = "CHECK_EXC_MATCH", - [CHECK_EG_MATCH] = "CHECK_EG_MATCH", - [LOAD_ATTR_ADAPTIVE] = "LOAD_ATTR_ADAPTIVE", - [LOAD_ATTR_INSTANCE_VALUE] = "LOAD_ATTR_INSTANCE_VALUE", - [LOAD_ATTR_MODULE] = "LOAD_ATTR_MODULE", - [LOAD_ATTR_SLOT] = "LOAD_ATTR_SLOT", - [LOAD_ATTR_WITH_HINT] = "LOAD_ATTR_WITH_HINT", - [LOAD_CONST__LOAD_FAST] = "LOAD_CONST__LOAD_FAST", - [LOAD_FAST__LOAD_CONST] = "LOAD_FAST__LOAD_CONST", - [LOAD_FAST__LOAD_FAST] = "LOAD_FAST__LOAD_FAST", - [LOAD_GLOBAL_ADAPTIVE] = "LOAD_GLOBAL_ADAPTIVE", - [LOAD_GLOBAL_BUILTIN] = "LOAD_GLOBAL_BUILTIN", - [LOAD_GLOBAL_MODULE] = "LOAD_GLOBAL_MODULE", - [WITH_EXCEPT_START] = "WITH_EXCEPT_START", - [GET_AITER] = "GET_AITER", - [GET_ANEXT] = "GET_ANEXT", - [BEFORE_ASYNC_WITH] = "BEFORE_ASYNC_WITH", - [BEFORE_WITH] = "BEFORE_WITH", - [END_ASYNC_FOR] = "END_ASYNC_FOR", - [LOAD_METHOD_ADAPTIVE] = "LOAD_METHOD_ADAPTIVE", - [LOAD_METHOD_CLASS] = "LOAD_METHOD_CLASS", - [LOAD_METHOD_MODULE] = "LOAD_METHOD_MODULE", - [LOAD_METHOD_NO_DICT] = "LOAD_METHOD_NO_DICT", - [LOAD_METHOD_WITH_DICT] = "LOAD_METHOD_WITH_DICT", - [STORE_SUBSCR] = "STORE_SUBSCR", - [DELETE_SUBSCR] = "DELETE_SUBSCR", - [LOAD_METHOD_WITH_VALUES] = "LOAD_METHOD_WITH_VALUES", - [PRECALL_ADAPTIVE] = "PRECALL_ADAPTIVE", - [PRECALL_BOUND_METHOD] = "PRECALL_BOUND_METHOD", - [PRECALL_BUILTIN_CLASS] = "PRECALL_BUILTIN_CLASS", - [PRECALL_BUILTIN_FAST_WITH_KEYWORDS] = "PRECALL_BUILTIN_FAST_WITH_KEYWORDS", - [PRECALL_METHOD_DESCRIPTOR_FAST_WITH_KEYWORDS] = "PRECALL_METHOD_DESCRIPTOR_FAST_WITH_KEYWORDS", - [GET_ITER] = "GET_ITER", - [GET_YIELD_FROM_ITER] = "GET_YIELD_FROM_ITER", - [PRINT_EXPR] = "PRINT_EXPR", - [LOAD_BUILD_CLASS] = "LOAD_BUILD_CLASS", - [PRECALL_NO_KW_BUILTIN_FAST] = "PRECALL_NO_KW_BUILTIN_FAST", - [PRECALL_NO_KW_BUILTIN_O] = "PRECALL_NO_KW_BUILTIN_O", - [LOAD_ASSERTION_ERROR] = "LOAD_ASSERTION_ERROR", - [RETURN_GENERATOR] = "RETURN_GENERATOR", - [PRECALL_NO_KW_ISINSTANCE] = "PRECALL_NO_KW_ISINSTANCE", - [PRECALL_NO_KW_LEN] = "PRECALL_NO_KW_LEN", - [PRECALL_NO_KW_LIST_APPEND] = "PRECALL_NO_KW_LIST_APPEND", - [PRECALL_NO_KW_METHOD_DESCRIPTOR_FAST] = "PRECALL_NO_KW_METHOD_DESCRIPTOR_FAST", - [PRECALL_NO_KW_METHOD_DESCRIPTOR_NOARGS] = "PRECALL_NO_KW_METHOD_DESCRIPTOR_NOARGS", - [PRECALL_NO_KW_METHOD_DESCRIPTOR_O] = "PRECALL_NO_KW_METHOD_DESCRIPTOR_O", - [LIST_TO_TUPLE] = "LIST_TO_TUPLE", - [RETURN_VALUE] = "RETURN_VALUE", - [IMPORT_STAR] = "IMPORT_STAR", - [SETUP_ANNOTATIONS] = "SETUP_ANNOTATIONS", - [YIELD_VALUE] = "YIELD_VALUE", - [ASYNC_GEN_WRAP] = "ASYNC_GEN_WRAP", - [PREP_RERAISE_STAR] = "PREP_RERAISE_STAR", - [POP_EXCEPT] = "POP_EXCEPT", - [STORE_NAME] = "STORE_NAME", - [DELETE_NAME] = "DELETE_NAME", - [UNPACK_SEQUENCE] = "UNPACK_SEQUENCE", - [FOR_ITER] = "FOR_ITER", - [UNPACK_EX] = "UNPACK_EX", - [STORE_ATTR] = "STORE_ATTR", - [DELETE_ATTR] = "DELETE_ATTR", - [STORE_GLOBAL] = "STORE_GLOBAL", - [DELETE_GLOBAL] = "DELETE_GLOBAL", - [SWAP] = "SWAP", - [LOAD_CONST] = "LOAD_CONST", - [LOAD_NAME] = "LOAD_NAME", - [BUILD_TUPLE] = "BUILD_TUPLE", - [BUILD_LIST] = "BUILD_LIST", - [BUILD_SET] = "BUILD_SET", - [BUILD_MAP] = "BUILD_MAP", - [LOAD_ATTR] = "LOAD_ATTR", - [COMPARE_OP] = "COMPARE_OP", - [IMPORT_NAME] = "IMPORT_NAME", - [IMPORT_FROM] = "IMPORT_FROM", - [JUMP_FORWARD] = "JUMP_FORWARD", - [JUMP_IF_FALSE_OR_POP] = "JUMP_IF_FALSE_OR_POP", - [JUMP_IF_TRUE_OR_POP] = "JUMP_IF_TRUE_OR_POP", - [PRECALL_NO_KW_STR_1] = "PRECALL_NO_KW_STR_1", - [POP_JUMP_FORWARD_IF_FALSE] = "POP_JUMP_FORWARD_IF_FALSE", - [POP_JUMP_FORWARD_IF_TRUE] = "POP_JUMP_FORWARD_IF_TRUE", - [LOAD_GLOBAL] = "LOAD_GLOBAL", - [IS_OP] = "IS_OP", - [CONTAINS_OP] = "CONTAINS_OP", - [RERAISE] = "RERAISE", - [COPY] = "COPY", - [PRECALL_NO_KW_TUPLE_1] = "PRECALL_NO_KW_TUPLE_1", - [BINARY_OP] = "BINARY_OP", - [SEND] = "SEND", - [LOAD_FAST] = "LOAD_FAST", - [STORE_FAST] = "STORE_FAST", - [DELETE_FAST] = "DELETE_FAST", - [PRECALL_NO_KW_TYPE_1] = "PRECALL_NO_KW_TYPE_1", - [POP_JUMP_FORWARD_IF_NOT_NONE] = "POP_JUMP_FORWARD_IF_NOT_NONE", - [POP_JUMP_FORWARD_IF_NONE] = "POP_JUMP_FORWARD_IF_NONE", - [RAISE_VARARGS] = "RAISE_VARARGS", - [GET_AWAITABLE] = "GET_AWAITABLE", - [MAKE_FUNCTION] = "MAKE_FUNCTION", - [BUILD_SLICE] = "BUILD_SLICE", - [JUMP_BACKWARD_NO_INTERRUPT] = "JUMP_BACKWARD_NO_INTERRUPT", - [MAKE_CELL] = "MAKE_CELL", - [LOAD_CLOSURE] = "LOAD_CLOSURE", - [LOAD_DEREF] = "LOAD_DEREF", - [STORE_DEREF] = "STORE_DEREF", - [DELETE_DEREF] = "DELETE_DEREF", - [JUMP_BACKWARD] = "JUMP_BACKWARD", - [PRECALL_PYFUNC] = "PRECALL_PYFUNC", - [CALL_FUNCTION_EX] = "CALL_FUNCTION_EX", - [RESUME_QUICK] = "RESUME_QUICK", - [EXTENDED_ARG] = "EXTENDED_ARG", - [LIST_APPEND] = "LIST_APPEND", - [SET_ADD] = "SET_ADD", - [MAP_ADD] = "MAP_ADD", - [LOAD_CLASSDEREF] = "LOAD_CLASSDEREF", - [COPY_FREE_VARS] = "COPY_FREE_VARS", - [STORE_ATTR_ADAPTIVE] = "STORE_ATTR_ADAPTIVE", - [RESUME] = "RESUME", - [MATCH_CLASS] = "MATCH_CLASS", - [STORE_ATTR_INSTANCE_VALUE] = "STORE_ATTR_INSTANCE_VALUE", - [STORE_ATTR_SLOT] = "STORE_ATTR_SLOT", - [FORMAT_VALUE] = "FORMAT_VALUE", - [BUILD_CONST_KEY_MAP] = "BUILD_CONST_KEY_MAP", - [BUILD_STRING] = "BUILD_STRING", - [STORE_ATTR_WITH_HINT] = "STORE_ATTR_WITH_HINT", - [STORE_FAST__LOAD_FAST] = "STORE_FAST__LOAD_FAST", - [LOAD_METHOD] = "LOAD_METHOD", - [STORE_FAST__STORE_FAST] = "STORE_FAST__STORE_FAST", - [LIST_EXTEND] = "LIST_EXTEND", - [SET_UPDATE] = "SET_UPDATE", - [DICT_MERGE] = "DICT_MERGE", - [DICT_UPDATE] = "DICT_UPDATE", - [PRECALL] = "PRECALL", - [STORE_SUBSCR_ADAPTIVE] = "STORE_SUBSCR_ADAPTIVE", - [STORE_SUBSCR_DICT] = "STORE_SUBSCR_DICT", - [STORE_SUBSCR_LIST_INT] = "STORE_SUBSCR_LIST_INT", - [UNPACK_SEQUENCE_ADAPTIVE] = "UNPACK_SEQUENCE_ADAPTIVE", - [CALL] = "CALL", - [KW_NAMES] = "KW_NAMES", - [POP_JUMP_BACKWARD_IF_NOT_NONE] = "POP_JUMP_BACKWARD_IF_NOT_NONE", - [POP_JUMP_BACKWARD_IF_NONE] = "POP_JUMP_BACKWARD_IF_NONE", - [POP_JUMP_BACKWARD_IF_FALSE] = "POP_JUMP_BACKWARD_IF_FALSE", - [POP_JUMP_BACKWARD_IF_TRUE] = "POP_JUMP_BACKWARD_IF_TRUE", - [UNPACK_SEQUENCE_LIST] = "UNPACK_SEQUENCE_LIST", - [UNPACK_SEQUENCE_TUPLE] = "UNPACK_SEQUENCE_TUPLE", - [UNPACK_SEQUENCE_TWO_TUPLE] = "UNPACK_SEQUENCE_TWO_TUPLE", - [180] = "<180>", - [181] = "<181>", - [182] = "<182>", - [183] = "<183>", - [184] = "<184>", - [185] = "<185>", - [186] = "<186>", - [187] = "<187>", - [188] = "<188>", - [189] = "<189>", - [190] = "<190>", - [191] = "<191>", - [192] = "<192>", - [193] = "<193>", - [194] = "<194>", - [195] = "<195>", - [196] = "<196>", - [197] = "<197>", - [198] = "<198>", - [199] = "<199>", - [200] = "<200>", - [201] = "<201>", - [202] = "<202>", - [203] = "<203>", - [204] = "<204>", - [205] = "<205>", - [206] = "<206>", - [207] = "<207>", - [208] = "<208>", - [209] = "<209>", - [210] = "<210>", - [211] = "<211>", - [212] = "<212>", - [213] = "<213>", - [214] = "<214>", - [215] = "<215>", - [216] = "<216>", - [217] = "<217>", - [218] = "<218>", - [219] = "<219>", - [220] = "<220>", - [221] = "<221>", - [222] = "<222>", - [223] = "<223>", - [224] = "<224>", - [225] = "<225>", - [226] = "<226>", - [227] = "<227>", - [228] = "<228>", - [229] = "<229>", - [230] = "<230>", - [231] = "<231>", - [232] = "<232>", - [233] = "<233>", - [234] = "<234>", - [235] = "<235>", - [236] = "<236>", - [237] = "<237>", - [238] = "<238>", - [239] = "<239>", - [240] = "<240>", - [241] = "<241>", - [242] = "<242>", - [243] = "<243>", - [244] = "<244>", - [245] = "<245>", - [246] = "<246>", - [247] = "<247>", - [248] = "<248>", - [249] = "<249>", - [250] = "<250>", - [251] = "<251>", - [252] = "<252>", - [253] = "<253>", - [254] = "<254>", - [DO_TRACING] = "DO_TRACING", -}; -#endif - -#define EXTRA_CASES \ - case 180: \ - case 181: \ - case 182: \ - case 183: \ - case 184: \ - case 185: \ - case 186: \ - case 187: \ - case 188: \ - case 189: \ - case 190: \ - case 191: \ - case 192: \ - case 193: \ - case 194: \ - case 195: \ - case 196: \ - case 197: \ - case 198: \ - case 199: \ - case 200: \ - case 201: \ - case 202: \ - case 203: \ - case 204: \ - case 205: \ - case 206: \ - case 207: \ - case 208: \ - case 209: \ - case 210: \ - case 211: \ - case 212: \ - case 213: \ - case 214: \ - case 215: \ - case 216: \ - case 217: \ - case 218: \ - case 219: \ - case 220: \ - case 221: \ - case 222: \ - case 223: \ - case 224: \ - case 225: \ - case 226: \ - case 227: \ - case 228: \ - case 229: \ - case 230: \ - case 231: \ - case 232: \ - case 233: \ - case 234: \ - case 235: \ - case 236: \ - case 237: \ - case 238: \ - case 239: \ - case 240: \ - case 241: \ - case 242: \ - case 243: \ - case 244: \ - case 245: \ - case 246: \ - case 247: \ - case 248: \ - case 249: \ - case 250: \ - case 251: \ - case 252: \ - case 253: \ - case 254: \ - ; - #define HAS_ARG(op) ((op) >= HAVE_ARGUMENT) /* Reserve some bytecodes for internal use in the compiler. diff --git a/Makefile.pre.in b/Makefile.pre.in index 3952f5b542b43..f3bccf4fec7ec 100644 --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -1311,8 +1311,10 @@ regen-opcode: # using Tools/scripts/generate_opcode_h.py $(PYTHON_FOR_REGEN) $(srcdir)/Tools/scripts/generate_opcode_h.py \ $(srcdir)/Lib/opcode.py \ - $(srcdir)/Include/opcode.h.new + $(srcdir)/Include/opcode.h.new \ + $(srcdir)/Include/internal/pycore_opcode.h.new $(UPDATE_FILE) $(srcdir)/Include/opcode.h $(srcdir)/Include/opcode.h.new + $(UPDATE_FILE) $(srcdir)/Include/internal/pycore_opcode.h $(srcdir)/Include/internal/pycore_opcode.h.new .PHONY: regen-token regen-token: diff --git a/Objects/codeobject.c b/Objects/codeobject.c index 4fc4b8fec68a2..e3e4ca159a632 100644 --- a/Objects/codeobject.c +++ b/Objects/codeobject.c @@ -5,6 +5,7 @@ #include "structmember.h" // PyMemberDef #include "pycore_code.h" // _PyCodeConstructor #include "pycore_interp.h" // PyInterpreterState.co_extra_freefuncs +#include "pycore_opcode.h" // _PyOpcode_Deopt #include "pycore_pystate.h" // _PyInterpreterState_GET() #include "pycore_tuple.h" // _PyTuple_ITEMS() #include "clinic/codeobject.c.h" diff --git a/Objects/frameobject.c b/Objects/frameobject.c index 7278ca147490f..56c4fceb6b893 100644 --- a/Objects/frameobject.c +++ b/Objects/frameobject.c @@ -2,10 +2,11 @@ #include "Python.h" #include "pycore_ceval.h" // _PyEval_BuiltinsFromGlobals() -#include "pycore_moduleobject.h" // _PyModule_GetDict() -#include "pycore_object.h" // _PyObject_GC_UNTRACK() #include "pycore_code.h" // CO_FAST_LOCAL, etc. #include "pycore_function.h" // _PyFunction_FromConstructor() +#include "pycore_moduleobject.h" // _PyModule_GetDict() +#include "pycore_object.h" // _PyObject_GC_UNTRACK() +#include "pycore_opcode.h" // _PyOpcode_Caches #include "frameobject.h" // PyFrameObject #include "pycore_frame.h" diff --git a/Objects/genobject.c b/Objects/genobject.c index e58118b2694ff..7920a10c7fb78 100644 --- a/Objects/genobject.c +++ b/Objects/genobject.c @@ -3,12 +3,12 @@ #include "Python.h" #include "pycore_call.h" // _PyObject_CallNoArgs() #include "pycore_ceval.h" // _PyEval_EvalFrame() +#include "pycore_frame.h" // _PyInterpreterFrame #include "pycore_genobject.h" // struct _Py_async_gen_state #include "pycore_object.h" // _PyObject_GC_UNTRACK() +#include "pycore_opcode.h" // _PyOpcode_Deopt #include "pycore_pyerrors.h" // _PyErr_ClearExcState() #include "pycore_pystate.h" // _PyThreadState_GET() -#include "pycore_frame.h" // _PyInterpreterFrame -#include "frameobject.h" // PyFrameObject #include "structmember.h" // PyMemberDef #include "opcode.h" // SEND diff --git a/PCbuild/regen.targets b/PCbuild/regen.targets index a49d97190ce20..24b5ced1de0e0 100644 --- a/PCbuild/regen.targets +++ b/PCbuild/regen.targets @@ -14,7 +14,7 @@ -C <_OpcodeSources Include="$(PySourcePath)Tools\scripts\generate_opcode_h.py;$(PySourcePath)Lib\opcode.py" /> - <_OpcodeOutputs Include="$(PySourcePath)Include\opcode.h;$(PySourcePath)Python\opcode_targets.h" /> + <_OpcodeOutputs Include="$(PySourcePath)Include\opcode.h;$(PySourcePath)Include\internal\pycore_opcode.h;$(PySourcePath)Python\opcode_targets.h" /> <_TokenSources Include="$(PySourcePath)Grammar\Tokens" /> <_TokenOutputs Include="$(PySourcePath)Doc\library\token-list.inc"> rst @@ -59,7 +59,7 @@ Inputs="@(_OpcodeSources)" Outputs="@(_OpcodeOutputs)" DependsOnTargets="FindPythonForBuild"> - diff --git a/Python/ceval.c b/Python/ceval.c index 90068bc51a7f6..6e7a2483112d7 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -15,6 +15,7 @@ #include "pycore_long.h" // _PyLong_GetZero() #include "pycore_object.h" // _PyObject_GC_TRACK() #include "pycore_moduleobject.h" // PyModuleObject +#include "pycore_opcode.h" // EXTRA_CASES #include "pycore_pyerrors.h" // _PyErr_Fetch() #include "pycore_pylifecycle.h" // _PyErr_Print() #include "pycore_pymem.h" // _PyMem_IsPtrFreed() diff --git a/Python/compile.c b/Python/compile.c index d66ee17293ac6..42b011ce93ef7 100644 --- a/Python/compile.c +++ b/Python/compile.c @@ -23,17 +23,18 @@ #include +// Need _PyOpcode_RelativeJump of pycore_opcode.h +#define NEED_OPCODE_TABLES + #include "Python.h" #include "pycore_ast.h" // _PyAST_GetDocString() -#include "pycore_compile.h" // _PyFuture_FromAST() #include "pycore_code.h" // _PyCode_New() -#include "pycore_pymem.h" // _PyMem_IsPtrFreed() +#include "pycore_compile.h" // _PyFuture_FromAST() #include "pycore_long.h" // _PyLong_GetZero() +#include "pycore_opcode.h" // _PyOpcode_Caches +#include "pycore_pymem.h" // _PyMem_IsPtrFreed() #include "pycore_symtable.h" // PySTEntryObject -#define NEED_OPCODE_TABLES -#include "opcode.h" // EXTENDED_ARG - #define DEFAULT_BLOCK_SIZE 16 #define DEFAULT_CODE_SIZE 128 diff --git a/Python/specialize.c b/Python/specialize.c index 3a8b768549c63..9449ac117979d 100644 --- a/Python/specialize.c +++ b/Python/specialize.c @@ -6,7 +6,7 @@ #include "pycore_long.h" #include "pycore_moduleobject.h" #include "pycore_object.h" -#include "opcode.h" +#include "pycore_opcode.h" // _PyOpcode_Caches #include "structmember.h" // struct PyMemberDef, T_OFFSET_EX #include // rand() diff --git a/Tools/scripts/generate_opcode_h.py b/Tools/scripts/generate_opcode_h.py index 1b45020835dd4..6a04297879f2c 100644 --- a/Tools/scripts/generate_opcode_h.py +++ b/Tools/scripts/generate_opcode_h.py @@ -3,12 +3,16 @@ import sys import tokenize -header = """ -/* Auto-generated by Tools/scripts/generate_opcode_h.py from Lib/opcode.py */ +SCRIPT_NAME = "Tools/scripts/generate_opcode_h.py" +PYTHON_OPCODE = "Lib/opcode.py" + +header = f""" +// Auto-generated by {SCRIPT_NAME} from {PYTHON_OPCODE} + #ifndef Py_OPCODE_H #define Py_OPCODE_H #ifdef __cplusplus -extern "C" { +extern "C" {{ #endif @@ -28,6 +32,29 @@ #endif /* !Py_OPCODE_H */ """ +internal_header = f""" +// Auto-generated by {SCRIPT_NAME} from {PYTHON_OPCODE} + +#ifndef Py_INTERNAL_OPCODE_H +#define Py_INTERNAL_OPCODE_H +#ifdef __cplusplus +extern "C" {{ +#endif + +#ifndef Py_BUILD_CORE +# error "this header requires Py_BUILD_CORE define" +#endif + +#include "opcode.h" +""".lstrip() + +internal_footer = """ +#ifdef __cplusplus +} +#endif +#endif // !Py_INTERNAL_OPCODE_H +""" + DEFINE = "#define {:<38} {:>3}\n" UINT32_MASK = (1<<32)-1 @@ -43,7 +70,7 @@ def write_int_array_from_ops(name, ops, out): assert bits == 0 out.write(f"}};\n") -def main(opcode_py, outfile='Include/opcode.h'): +def main(opcode_py, outfile='Include/opcode.h', internaloutfile='Include/internal/pycore_opcode.h'): opcode = {} if hasattr(tokenize, 'open'): fp = tokenize.open(opcode_py) # Python 3.2+ @@ -75,8 +102,10 @@ def main(opcode_py, outfile='Include/opcode.h'): opname_including_specialized[255] = 'DO_TRACING' used[255] = True - with open(outfile, 'w') as fobj: + with (open(outfile, 'w') as fobj, open(internaloutfile, 'w') as iobj): fobj.write(header) + iobj.write(internal_header) + for name in opname: if name in opmap: fobj.write(DEFINE.format(name, opmap[name])) @@ -86,28 +115,29 @@ def main(opcode_py, outfile='Include/opcode.h'): for name, op in specialized_opmap.items(): fobj.write(DEFINE.format(name, op)) - fobj.write("\nextern const uint8_t _PyOpcode_Caches[256];\n") - fobj.write("\nextern const uint8_t _PyOpcode_Deopt[256];\n") - fobj.write("\n#ifdef NEED_OPCODE_TABLES\n") - write_int_array_from_ops("_PyOpcode_RelativeJump", opcode['hasjrel'], fobj) - write_int_array_from_ops("_PyOpcode_Jump", opcode['hasjrel'] + opcode['hasjabs'], fobj) + iobj.write("\nextern const uint8_t _PyOpcode_Caches[256];\n") + iobj.write("\nextern const uint8_t _PyOpcode_Deopt[256];\n") + iobj.write("\n#ifdef NEED_OPCODE_TABLES\n") + write_int_array_from_ops("_PyOpcode_RelativeJump", opcode['hasjrel'], iobj) + write_int_array_from_ops("_PyOpcode_Jump", opcode['hasjrel'] + opcode['hasjabs'], iobj) - fobj.write("\nconst uint8_t _PyOpcode_Caches[256] = {\n") + iobj.write("\nconst uint8_t _PyOpcode_Caches[256] = {\n") for i, entries in enumerate(opcode["_inline_cache_entries"]): if entries: - fobj.write(f" [{opname[i]}] = {entries},\n") - fobj.write("};\n") + iobj.write(f" [{opname[i]}] = {entries},\n") + iobj.write("};\n") + deoptcodes = {} for basic in opmap: deoptcodes[basic] = basic for basic, family in opcode["_specializations"].items(): for specialized in family: deoptcodes[specialized] = basic - fobj.write("\nconst uint8_t _PyOpcode_Deopt[256] = {\n") + iobj.write("\nconst uint8_t _PyOpcode_Deopt[256] = {\n") for opt, deopt in sorted(deoptcodes.items()): - fobj.write(f" [{opt}] = {deopt},\n") - fobj.write("};\n") - fobj.write("#endif /* OPCODE_TABLES */\n") + iobj.write(f" [{opt}] = {deopt},\n") + iobj.write("};\n") + iobj.write("#endif // NEED_OPCODE_TABLES\n") fobj.write("\n") fobj.write("#define HAS_CONST(op) (false\\") @@ -119,28 +149,29 @@ def main(opcode_py, outfile='Include/opcode.h'): for i, (op, _) in enumerate(opcode["_nb_ops"]): fobj.write(DEFINE.format(op, i)) - fobj.write("\n") - fobj.write("#ifdef Py_DEBUG\n") - fobj.write("static const char *const _PyOpcode_OpName[256] = {\n") + iobj.write("\n") + iobj.write("#ifdef Py_DEBUG\n") + iobj.write("static const char *const _PyOpcode_OpName[256] = {\n") for op, name in enumerate(opname_including_specialized): if name[0] != "<": op = name - fobj.write(f''' [{op}] = "{name}",\n''') - fobj.write("};\n") - fobj.write("#endif\n") + iobj.write(f''' [{op}] = "{name}",\n''') + iobj.write("};\n") + iobj.write("#endif\n") - fobj.write("\n") - fobj.write("#define EXTRA_CASES \\\n") + iobj.write("\n") + iobj.write("#define EXTRA_CASES \\\n") for i, flag in enumerate(used): if not flag: - fobj.write(f" case {i}: \\\n") - fobj.write(" ;\n") + iobj.write(f" case {i}: \\\n") + iobj.write(" ;\n") fobj.write(footer) + iobj.write(internal_footer) print(f"{outfile} regenerated from {opcode_py}") if __name__ == '__main__': - main(sys.argv[1], sys.argv[2]) + main(sys.argv[1], sys.argv[2], sys.argv[3]) From webhook-mailer at python.org Mon Apr 25 18:34:10 2022 From: webhook-mailer at python.org (zooba) Date: Mon, 25 Apr 2022 22:34:10 -0000 Subject: [Python-checkins] bpo-46907: Update Windows installer to SQLite 3.38.2 (GH-32147) Message-ID: https://github.com/python/cpython/commit/eddd07f840c9a4ab0ee05ce56d98caac0f072cef commit: eddd07f840c9a4ab0ee05ce56d98caac0f072cef branch: main author: Mariusz Felisiak committer: zooba date: 2022-04-25T23:33:45+01:00 summary: bpo-46907: Update Windows installer to SQLite 3.38.2 (GH-32147) files: A Misc/NEWS.d/next/Windows/2022-03-28-07-01-31.bpo-46907.Ou3G6Z.rst M PCbuild/get_externals.bat M PCbuild/python.props M PCbuild/readme.txt diff --git a/Misc/NEWS.d/next/Windows/2022-03-28-07-01-31.bpo-46907.Ou3G6Z.rst b/Misc/NEWS.d/next/Windows/2022-03-28-07-01-31.bpo-46907.Ou3G6Z.rst new file mode 100644 index 0000000000000..36be10057e497 --- /dev/null +++ b/Misc/NEWS.d/next/Windows/2022-03-28-07-01-31.bpo-46907.Ou3G6Z.rst @@ -0,0 +1 @@ +Update Windows installer to use SQLite 3.38.2. diff --git a/PCbuild/get_externals.bat b/PCbuild/get_externals.bat index 289b7f505b955..35646edb2e9e8 100644 --- a/PCbuild/get_externals.bat +++ b/PCbuild/get_externals.bat @@ -54,7 +54,7 @@ set libraries= set libraries=%libraries% bzip2-1.0.8 if NOT "%IncludeLibffiSrc%"=="false" set libraries=%libraries% libffi-3.4.2 if NOT "%IncludeSSLSrc%"=="false" set libraries=%libraries% openssl-1.1.1n -set libraries=%libraries% sqlite-3.38.1.0 +set libraries=%libraries% sqlite-3.38.2.0 if NOT "%IncludeTkinterSrc%"=="false" set libraries=%libraries% tcl-core-8.6.12.1 if NOT "%IncludeTkinterSrc%"=="false" set libraries=%libraries% tk-8.6.12.1 if NOT "%IncludeTkinterSrc%"=="false" set libraries=%libraries% tix-8.4.3.6 diff --git a/PCbuild/python.props b/PCbuild/python.props index 28975ff88ce77..baa6526378a75 100644 --- a/PCbuild/python.props +++ b/PCbuild/python.props @@ -61,7 +61,7 @@ $(EXTERNALS_DIR) $([System.IO.Path]::GetFullPath(`$(PySourcePath)externals`)) $(ExternalsDir)\ - $(ExternalsDir)sqlite-3.38.1.0\ + $(ExternalsDir)sqlite-3.38.2.0\ $(ExternalsDir)bzip2-1.0.8\ $(ExternalsDir)xz-5.2.5\ $(ExternalsDir)libffi-3.4.2\ diff --git a/PCbuild/readme.txt b/PCbuild/readme.txt index 24bce197389e0..4c99b867c52bd 100644 --- a/PCbuild/readme.txt +++ b/PCbuild/readme.txt @@ -189,7 +189,7 @@ _ssl again when building. _sqlite3 - Wraps SQLite 3.38.1, which is itself built by sqlite3.vcxproj + Wraps SQLite 3.38.2, which is itself built by sqlite3.vcxproj Homepage: http://www.sqlite.org/ _tkinter From webhook-mailer at python.org Mon Apr 25 19:19:46 2022 From: webhook-mailer at python.org (gpshead) Date: Mon, 25 Apr 2022 23:19:46 -0000 Subject: [Python-checkins] gh-91401: Add a failsafe way to disable vfork. (#91490) Message-ID: https://github.com/python/cpython/commit/cd5726fe674eaff442510eeb6c75628858be9e9f commit: cd5726fe674eaff442510eeb6c75628858be9e9f branch: main author: Gregory P. Smith committer: gpshead date: 2022-04-25T16:19:39-07:00 summary: gh-91401: Add a failsafe way to disable vfork. (#91490) Just in case there is ever an issue with _posixsubprocess's use of vfork() due to the complexity of using it properly and potential directions that Linux platforms where it defaults to on could take, this adds a failsafe so that users can disable its use entirely by setting a global flag. No known reason to disable it exists. But it'd be a shame to encounter one and not be able to use CPython without patching and rebuilding it. See the linked issue for some discussion on reasoning. Also documents the existing way to disable posix_spawn. files: A Misc/NEWS.d/next/Library/2022-04-25-21-33-48.gh-issue-91401._Jo4Bu.rst M Doc/library/subprocess.rst M Lib/multiprocessing/util.py M Lib/subprocess.py M Lib/test/test_capi.py M Lib/test/test_subprocess.py M Modules/_posixsubprocess.c diff --git a/Doc/library/subprocess.rst b/Doc/library/subprocess.rst index ab9f1d88a0fc26..fca55496492685 100644 --- a/Doc/library/subprocess.rst +++ b/Doc/library/subprocess.rst @@ -1529,3 +1529,39 @@ runtime): :mod:`shlex` Module which provides function to parse and escape command lines. + + +.. _disable_vfork: +.. _disable_posix_spawn: + +Disabling use of ``vfork()`` or ``posix_spawn()`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +On Linux, :mod:`subprocess` defaults to using the ``vfork()`` system call +internally when it is safe to do so rather than ``fork()``. This greatly +improves performance. + +If you ever encounter a presumed highly-unusual situation where you need to +prevent ``vfork()`` from being used by Python, you can set the +:attr:`subprocess._USE_VFORK` attribute to a false value. + + subprocess._USE_VFORK = False # See CPython issue gh-NNNNNN. + +Setting this has no impact on use of ``posix_spawn()`` which could use +``vfork()`` internally within its libc implementation. There is a similar +:attr:`subprocess._USE_POSIX_SPAWN` attribute if you need to prevent use of +that. + + subprocess._USE_POSIX_SPAWN = False # See CPython issue gh-NNNNNN. + +It is safe to set these to false on any Python version. They will have no +effect on older versions when unsupported. Do not assume the attributes are +available to read. Despite their names, a true value does not indicate that the +corresponding function will be used, only that that it may be. + +Please file issues any time you have to use these private knobs with a way to +reproduce the issue you were seeing. Link to that issue from a comment in your +code. + +.. versionadded:: 3.8 ``_USE_POSIX_SPAWN`` +.. versionadded:: 3.11 ``_USE_VFORK`` diff --git a/Lib/multiprocessing/util.py b/Lib/multiprocessing/util.py index abbc4c5e6088b2..790955e7b71fae 100644 --- a/Lib/multiprocessing/util.py +++ b/Lib/multiprocessing/util.py @@ -446,13 +446,15 @@ def _flush_std_streams(): def spawnv_passfds(path, args, passfds): import _posixsubprocess + import subprocess passfds = tuple(sorted(map(int, passfds))) errpipe_read, errpipe_write = os.pipe() try: return _posixsubprocess.fork_exec( args, [path], True, passfds, None, None, -1, -1, -1, -1, -1, -1, errpipe_read, errpipe_write, - False, False, None, None, None, -1, None) + False, False, None, None, None, -1, None, + subprocess._USE_VFORK) finally: os.close(errpipe_read) os.close(errpipe_write) diff --git a/Lib/subprocess.py b/Lib/subprocess.py index b58c5784283902..a5fa152715c145 100644 --- a/Lib/subprocess.py +++ b/Lib/subprocess.py @@ -702,7 +702,10 @@ def _use_posix_spawn(): return False +# These are primarily fail-safe knobs for negatives. A True value does not +# guarantee the given libc/syscall API will be used. _USE_POSIX_SPAWN = _use_posix_spawn() +_USE_VFORK = True class Popen: @@ -1792,7 +1795,7 @@ def _execute_child(self, args, executable, preexec_fn, close_fds, errpipe_read, errpipe_write, restore_signals, start_new_session, gid, gids, uid, umask, - preexec_fn) + preexec_fn, _USE_VFORK) self._child_created = True finally: # be sure the FD is closed no matter what diff --git a/Lib/test/test_capi.py b/Lib/test/test_capi.py index 5f5d3512d1a0e3..1aed9b71c51aa3 100644 --- a/Lib/test/test_capi.py +++ b/Lib/test/test_capi.py @@ -140,7 +140,7 @@ class Z(object): def __len__(self): return 1 self.assertRaises(TypeError, _posixsubprocess.fork_exec, - 1,Z(),3,(1, 2),5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21) + 1,Z(),3,(1, 2),5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22) # Issue #15736: overflow in _PySequence_BytesToCharpArray() class Z(object): def __len__(self): @@ -148,7 +148,7 @@ def __len__(self): def __getitem__(self, i): return b'x' self.assertRaises(MemoryError, _posixsubprocess.fork_exec, - 1,Z(),3,(1, 2),5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21) + 1,Z(),3,(1, 2),5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22) @unittest.skipUnless(_posixsubprocess, '_posixsubprocess required for this test.') def test_subprocess_fork_exec(self): @@ -158,7 +158,7 @@ def __len__(self): # Issue #15738: crash in subprocess_fork_exec() self.assertRaises(TypeError, _posixsubprocess.fork_exec, - Z(),[b'1'],3,(1, 2),5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21) + Z(),[b'1'],3,(1, 2),5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22) @unittest.skipIf(MISSING_C_DOCSTRINGS, "Signature information for builtins requires docstrings") diff --git a/Lib/test/test_subprocess.py b/Lib/test/test_subprocess.py index 8603b9888bb984..99b5947e54be68 100644 --- a/Lib/test/test_subprocess.py +++ b/Lib/test/test_subprocess.py @@ -1543,6 +1543,22 @@ def test_class_getitems(self): self.assertIsInstance(subprocess.Popen[bytes], types.GenericAlias) self.assertIsInstance(subprocess.CompletedProcess[str], types.GenericAlias) + @unittest.skipIf(not sysconfig.get_config_var("HAVE_VFORK"), + "vfork() not enabled by configure.") + @mock.patch("subprocess._fork_exec") + def test__use_vfork(self, mock_fork_exec): + self.assertTrue(subprocess._USE_VFORK) # The default value regardless. + mock_fork_exec.side_effect = RuntimeError("just testing args") + with self.assertRaises(RuntimeError): + subprocess.run([sys.executable, "-c", "pass"]) + mock_fork_exec.assert_called_once() + self.assertTrue(mock_fork_exec.call_args.args[-1]) + with mock.patch.object(subprocess, '_USE_VFORK', False): + with self.assertRaises(RuntimeError): + subprocess.run([sys.executable, "-c", "pass"]) + self.assertFalse(mock_fork_exec.call_args_list[-1].args[-1]) + + class RunFuncTestCase(BaseTestCase): def run_python(self, code, **kwargs): """Run Python code in a subprocess using subprocess.run""" @@ -3107,7 +3123,7 @@ def test_fork_exec(self): 1, 2, 3, 4, True, True, False, [], 0, -1, - func) + func, False) # Attempt to prevent # "TypeError: fork_exec() takes exactly N arguments (M given)" # from passing the test. More refactoring to have us start @@ -3156,7 +3172,7 @@ def __int__(self): 1, 2, 3, 4, True, True, None, None, None, -1, - None) + None, "no vfork") self.assertIn('fds_to_keep', str(c.exception)) finally: if not gc_enabled: diff --git a/Misc/NEWS.d/next/Library/2022-04-25-21-33-48.gh-issue-91401._Jo4Bu.rst b/Misc/NEWS.d/next/Library/2022-04-25-21-33-48.gh-issue-91401._Jo4Bu.rst new file mode 100644 index 00000000000000..7584710af30ff8 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-25-21-33-48.gh-issue-91401._Jo4Bu.rst @@ -0,0 +1,2 @@ +Provide a way to disable :mod:`subprocess` use of ``vfork()`` just in case +it is ever needed and document the existing mechanism for ``posix_spawn()``. diff --git a/Modules/_posixsubprocess.c b/Modules/_posixsubprocess.c index 440c7c5b335994..2440609e31bcc2 100644 --- a/Modules/_posixsubprocess.c +++ b/Modules/_posixsubprocess.c @@ -751,9 +751,10 @@ subprocess_fork_exec(PyObject *module, PyObject *args) Py_ssize_t arg_num, num_groups = 0; int need_after_fork = 0; int saved_errno = 0; + int allow_vfork; if (!PyArg_ParseTuple( - args, "OOpO!OOiiiiiiiiiiOOOiO:fork_exec", + args, "OOpO!OOiiiiiiiiiiOOOiOp:fork_exec", &process_args, &executable_list, &close_fds, &PyTuple_Type, &py_fds_to_keep, &cwd_obj, &env_list, @@ -761,7 +762,7 @@ subprocess_fork_exec(PyObject *module, PyObject *args) &errread, &errwrite, &errpipe_read, &errpipe_write, &restore_signals, &call_setsid, &gid_object, &groups_list, &uid_object, &child_umask, - &preexec_fn)) + &preexec_fn, &allow_vfork)) return NULL; if ((preexec_fn != Py_None) && @@ -940,7 +941,7 @@ subprocess_fork_exec(PyObject *module, PyObject *args) #ifdef VFORK_USABLE /* Use vfork() only if it's safe. See the comment above child_exec(). */ sigset_t old_sigs; - if (preexec_fn == Py_None && + if (preexec_fn == Py_None && allow_vfork && !call_setuid && !call_setgid && !call_setgroups) { /* Block all signals to ensure that no signal handlers are run in the * child process while it shares memory with us. Note that signals From webhook-mailer at python.org Mon Apr 25 19:26:57 2022 From: webhook-mailer at python.org (brettcannon) Date: Mon, 25 Apr 2022 23:26:57 -0000 Subject: [Python-checkins] gh-91217: deprecate sunau (GH-91866) Message-ID: https://github.com/python/cpython/commit/d174ebe91ebc9f7388a22cc81cdc5f7be8bb8c9b commit: d174ebe91ebc9f7388a22cc81cdc5f7be8bb8c9b branch: main author: Brett Cannon committer: brettcannon date: 2022-04-25T16:26:43-07:00 summary: gh-91217: deprecate sunau (GH-91866) files: A Misc/NEWS.d/next/Library/2022-04-17-12-38-31.gh-issue-91217.55714p.rst M Doc/whatsnew/3.11.rst M Lib/sunau.py M Lib/test/test_ossaudiodev.py M Lib/test/test_sunau.py diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index ebaa1e993e276..8f0d3c7c7099a 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -1068,6 +1068,7 @@ Deprecated * :mod:`pipes` * :mod:`sndhdr` * :mod:`spwd` + * :mod:`sunau` (Contributed by Brett Cannon in :issue:`47061`.) diff --git a/Lib/sunau.py b/Lib/sunau.py index 9b3533d930676..94c42f1582c02 100644 --- a/Lib/sunau.py +++ b/Lib/sunau.py @@ -106,6 +106,8 @@ from collections import namedtuple import warnings +warnings._deprecated(__name__, remove=(3, 13)) + _sunau_params = namedtuple('_sunau_params', 'nchannels sampwidth framerate nframes comptype compname') diff --git a/Lib/test/test_ossaudiodev.py b/Lib/test/test_ossaudiodev.py index 3275333545882..2cc641a49b63a 100644 --- a/Lib/test/test_ossaudiodev.py +++ b/Lib/test/test_ossaudiodev.py @@ -9,10 +9,10 @@ warnings.simplefilter("ignore", DeprecationWarning) ossaudiodev = import_helper.import_module('ossaudiodev') audioop = warnings_helper.import_deprecated('audioop') +sunau = warnings_helper.import_deprecated('sunau') import errno import sys -import sunau import time import unittest diff --git a/Lib/test/test_sunau.py b/Lib/test/test_sunau.py index e65742b69f2e9..40408b01eda9a 100644 --- a/Lib/test/test_sunau.py +++ b/Lib/test/test_sunau.py @@ -3,9 +3,9 @@ import io import struct import sys -import sunau from test.support import warnings_helper +sunau = warnings_helper.import_deprecated("sunau") audioop = warnings_helper.import_deprecated("audioop") diff --git a/Misc/NEWS.d/next/Library/2022-04-17-12-38-31.gh-issue-91217.55714p.rst b/Misc/NEWS.d/next/Library/2022-04-17-12-38-31.gh-issue-91217.55714p.rst new file mode 100644 index 0000000000000..4cebfc4222d4c --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-17-12-38-31.gh-issue-91217.55714p.rst @@ -0,0 +1 @@ +Deprecate the sunau module. From webhook-mailer at python.org Mon Apr 25 20:51:33 2022 From: webhook-mailer at python.org (ezio-melotti) Date: Tue, 26 Apr 2022 00:51:33 -0000 Subject: [Python-checkins] [3.10] gh-91888: add a :gh: role to the documentation (GH-91889) (#91934) Message-ID: https://github.com/python/cpython/commit/4120bbae4930cfd74649118e0bcc2fb5dd32b1fa commit: 4120bbae4930cfd74649118e0bcc2fb5dd32b1fa branch: 3.10 author: Ezio Melotti committer: ezio-melotti date: 2022-04-26T02:51:28+02:00 summary: [3.10] gh-91888: add a :gh: role to the documentation (GH-91889) (#91934) * gh-91888: Add a :gh: role to the documentation (GH-91889). * [3.10] gh-91888: add a `:gh:` role to the documentation (GH-91889) * Add a new :gh:`...` role for GitHub issues. * Fix a GitHub id to use the :gh: role. * Add Misc/NEWS entry. * Refactoring and rephrasing. Co-authored-by: Hugo van Kemenade . (cherry picked from commit f7641a2ffec243e5f600028a84debe9028a9ee44) Co-authored-by: Ezio Melotti files: A Misc/NEWS.d/next/Documentation/2022-04-24-22-09-31.gh-issue-91888.kTjJLx.rst M Doc/tools/extensions/pyspecific.py diff --git a/Doc/tools/extensions/pyspecific.py b/Doc/tools/extensions/pyspecific.py index cce364c0e6245..cbb7638627ef1 100644 --- a/Doc/tools/extensions/pyspecific.py +++ b/Doc/tools/extensions/pyspecific.py @@ -44,6 +44,7 @@ ISSUE_URI = 'https://bugs.python.org/issue?@action=redirect&bpo=%s' +GH_ISSUE_URI = 'https://github.com/python/cpython/issues/%s' SOURCE_URI = 'https://github.com/python/cpython/tree/3.10/%s' # monkey-patch reST parser to disable alphabetic and roman enumerated lists @@ -58,11 +59,33 @@ def issue_role(typ, rawtext, text, lineno, inliner, options={}, content=[]): issue = utils.unescape(text) + # sanity check: there are no bpo issues within these two values + if 47261 < int(issue) < 400000: + msg = inliner.reporter.error(f'The BPO ID {text!r} seems too high -- ' + 'use :gh:`...` for GitHub IDs', line=lineno) + prb = inliner.problematic(rawtext, rawtext, msg) + return [prb], [msg] text = 'bpo-' + issue refnode = nodes.reference(text, text, refuri=ISSUE_URI % issue) return [refnode], [] +# Support for marking up and linking to GitHub issues + +def gh_issue_role(typ, rawtext, text, lineno, inliner, options={}, content=[]): + issue = utils.unescape(text) + # sanity check: all GitHub issues have ID >= 32426 + # even though some of them are also valid BPO IDs + if int(issue) < 32426: + msg = inliner.reporter.error(f'The GitHub ID {text!r} seems too low -- ' + 'use :issue:`...` for BPO IDs', line=lineno) + prb = inliner.problematic(rawtext, rawtext, msg) + return [prb], [msg] + text = 'gh-' + issue + refnode = nodes.reference(text, text, refuri=GH_ISSUE_URI % issue) + return [refnode], [] + + # Support for linking to Python source files easily def source_role(typ, rawtext, text, lineno, inliner, options={}, content=[]): @@ -615,6 +638,7 @@ def process_audit_events(app, doctree, fromdocname): def setup(app): app.add_role('issue', issue_role) + app.add_role('gh', gh_issue_role) app.add_role('source', source_role) app.add_directive('impl-detail', ImplementationDetail) app.add_directive('availability', Availability) diff --git a/Misc/NEWS.d/next/Documentation/2022-04-24-22-09-31.gh-issue-91888.kTjJLx.rst b/Misc/NEWS.d/next/Documentation/2022-04-24-22-09-31.gh-issue-91888.kTjJLx.rst new file mode 100644 index 0000000000000..9194be9dbf92d --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2022-04-24-22-09-31.gh-issue-91888.kTjJLx.rst @@ -0,0 +1 @@ +Add a new `gh` role to the documentation to link to GitHub issues. From webhook-mailer at python.org Mon Apr 25 21:16:31 2022 From: webhook-mailer at python.org (ezio-melotti) Date: Tue, 26 Apr 2022 01:16:31 -0000 Subject: [Python-checkins] [3.9] gh-91888: add a `:gh:` role to the documentation (GH-91889) (#91935) Message-ID: https://github.com/python/cpython/commit/38be31e61e092b07243dad1622ef61177c12ed55 commit: 38be31e61e092b07243dad1622ef61177c12ed55 branch: 3.9 author: Ezio Melotti committer: ezio-melotti date: 2022-04-26T03:16:24+02:00 summary: [3.9] gh-91888: add a `:gh:` role to the documentation (GH-91889) (#91935) * gh-91888: Add a :gh: role to the documentation (GH-91889). * [3.9] gh-91888: add a `:gh:` role to the documentation (GH-91889) * Add a new :gh:`...` role for GitHub issues. * Fix a GitHub id to use the :gh: role. * Add Misc/NEWS entry. * Refactoring and rephrasing. Co-authored-by: Hugo van Kemenade . (cherry picked from commit f7641a2ffec243e5f600028a84debe9028a9ee44) Co-authored-by: Ezio Melotti files: A Misc/NEWS.d/next/Documentation/2022-04-24-22-09-31.gh-issue-91888.kTjJLx.rst M Doc/tools/extensions/pyspecific.py diff --git a/Doc/tools/extensions/pyspecific.py b/Doc/tools/extensions/pyspecific.py index 452366a178926..6c819866976b8 100644 --- a/Doc/tools/extensions/pyspecific.py +++ b/Doc/tools/extensions/pyspecific.py @@ -44,6 +44,7 @@ ISSUE_URI = 'https://bugs.python.org/issue?@action=redirect&bpo=%s' +GH_ISSUE_URI = 'https://github.com/python/cpython/issues/%s' SOURCE_URI = 'https://github.com/python/cpython/tree/3.9/%s' # monkey-patch reST parser to disable alphabetic and roman enumerated lists @@ -58,11 +59,33 @@ def issue_role(typ, rawtext, text, lineno, inliner, options={}, content=[]): issue = utils.unescape(text) + # sanity check: there are no bpo issues within these two values + if 47261 < int(issue) < 400000: + msg = inliner.reporter.error(f'The BPO ID {text!r} seems too high -- ' + 'use :gh:`...` for GitHub IDs', line=lineno) + prb = inliner.problematic(rawtext, rawtext, msg) + return [prb], [msg] text = 'bpo-' + issue refnode = nodes.reference(text, text, refuri=ISSUE_URI % issue) return [refnode], [] +# Support for marking up and linking to GitHub issues + +def gh_issue_role(typ, rawtext, text, lineno, inliner, options={}, content=[]): + issue = utils.unescape(text) + # sanity check: all GitHub issues have ID >= 32426 + # even though some of them are also valid BPO IDs + if int(issue) < 32426: + msg = inliner.reporter.error(f'The GitHub ID {text!r} seems too low -- ' + 'use :issue:`...` for BPO IDs', line=lineno) + prb = inliner.problematic(rawtext, rawtext, msg) + return [prb], [msg] + text = 'gh-' + issue + refnode = nodes.reference(text, text, refuri=GH_ISSUE_URI % issue) + return [refnode], [] + + # Support for linking to Python source files easily def source_role(typ, rawtext, text, lineno, inliner, options={}, content=[]): @@ -577,6 +600,7 @@ def process_audit_events(app, doctree, fromdocname): def setup(app): app.add_role('issue', issue_role) + app.add_role('gh', gh_issue_role) app.add_role('source', source_role) app.add_directive('impl-detail', ImplementationDetail) app.add_directive('availability', Availability) diff --git a/Misc/NEWS.d/next/Documentation/2022-04-24-22-09-31.gh-issue-91888.kTjJLx.rst b/Misc/NEWS.d/next/Documentation/2022-04-24-22-09-31.gh-issue-91888.kTjJLx.rst new file mode 100644 index 0000000000000..9194be9dbf92d --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2022-04-24-22-09-31.gh-issue-91888.kTjJLx.rst @@ -0,0 +1 @@ +Add a new `gh` role to the documentation to link to GitHub issues. From webhook-mailer at python.org Mon Apr 25 23:30:55 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Tue, 26 Apr 2022 03:30:55 -0000 Subject: [Python-checkins] gh-91860: Add typing.dataclass_transform (PEP 681) (#91861) Message-ID: https://github.com/python/cpython/commit/5397b5afc1f594dc0ba3b7743351d595e637bf24 commit: 5397b5afc1f594dc0ba3b7743351d595e637bf24 branch: main author: Jelle Zijlstra committer: JelleZijlstra date: 2022-04-25T20:30:25-07:00 summary: gh-91860: Add typing.dataclass_transform (PEP 681) (#91861) Copied from typing-extensions (python/typing#1054, python/typing#1120). Documentation is intentionally omitted, so we can focus on getting the runtime part in before the feature freeze. files: A Misc/NEWS.d/next/Library/2022-04-23-08-06-36.gh-issue-91860.ityDjK.rst M Lib/test/test_typing.py M Lib/typing.py diff --git a/Lib/test/test_typing.py b/Lib/test/test_typing.py index fce340f27d9ac..a904b7a790c04 100644 --- a/Lib/test/test_typing.py +++ b/Lib/test/test_typing.py @@ -24,6 +24,7 @@ from typing import get_origin, get_args from typing import is_typeddict from typing import reveal_type +from typing import dataclass_transform from typing import no_type_check, no_type_check_decorator from typing import Type from typing import NamedTuple, NotRequired, Required, TypedDict @@ -6607,6 +6608,91 @@ def test_reveal_type(self): self.assertEqual(stderr.getvalue(), "Runtime type is 'object'\n") +class DataclassTransformTests(BaseTestCase): + def test_decorator(self): + def create_model(*, frozen: bool = False, kw_only: bool = True): + return lambda cls: cls + + decorated = dataclass_transform(kw_only_default=True, order_default=False)(create_model) + + class CustomerModel: + id: int + + self.assertIs(decorated, create_model) + self.assertEqual( + decorated.__dataclass_transform__, + { + "eq_default": True, + "order_default": False, + "kw_only_default": True, + "field_specifiers": (), + "kwargs": {}, + } + ) + self.assertIs( + decorated(frozen=True, kw_only=False)(CustomerModel), + CustomerModel + ) + + def test_base_class(self): + class ModelBase: + def __init_subclass__(cls, *, frozen: bool = False): ... + + Decorated = dataclass_transform( + eq_default=True, + order_default=True, + # Arbitrary unrecognized kwargs are accepted at runtime. + make_everything_awesome=True, + )(ModelBase) + + class CustomerModel(Decorated, frozen=True): + id: int + + self.assertIs(Decorated, ModelBase) + self.assertEqual( + Decorated.__dataclass_transform__, + { + "eq_default": True, + "order_default": True, + "kw_only_default": False, + "field_specifiers": (), + "kwargs": {"make_everything_awesome": True}, + } + ) + self.assertIsSubclass(CustomerModel, Decorated) + + def test_metaclass(self): + class Field: ... + + class ModelMeta(type): + def __new__( + cls, name, bases, namespace, *, init: bool = True, + ): + return super().__new__(cls, name, bases, namespace) + + Decorated = dataclass_transform( + order_default=True, field_specifiers=(Field,) + )(ModelMeta) + + class ModelBase(metaclass=Decorated): ... + + class CustomerModel(ModelBase, init=False): + id: int + + self.assertIs(Decorated, ModelMeta) + self.assertEqual( + Decorated.__dataclass_transform__, + { + "eq_default": True, + "order_default": True, + "kw_only_default": False, + "field_specifiers": (Field,), + "kwargs": {}, + } + ) + self.assertIsInstance(CustomerModel, Decorated) + + class AllTests(BaseTestCase): """Tests for __all__.""" diff --git a/Lib/typing.py b/Lib/typing.py index 9d8149c3f5332..f4d4fa4d6713c 100644 --- a/Lib/typing.py +++ b/Lib/typing.py @@ -123,6 +123,7 @@ def _idfunc(_, x): 'assert_never', 'cast', 'clear_overloads', + 'dataclass_transform', 'final', 'get_args', 'get_origin', @@ -3271,3 +3272,81 @@ def reveal_type(obj: T, /) -> T: """ print(f"Runtime type is {type(obj).__name__!r}", file=sys.stderr) return obj + + +def dataclass_transform( + *, + eq_default: bool = True, + order_default: bool = False, + kw_only_default: bool = False, + field_specifiers: tuple[type[Any] | Callable[..., Any], ...] = (), + **kwargs: Any, +) -> Callable[[T], T]: + """Decorator that marks a function, class, or metaclass as providing + dataclass-like behavior. + + Example usage with a decorator function: + + _T = TypeVar("_T") + + @dataclass_transform() + def create_model(cls: type[_T]) -> type[_T]: + ... + return cls + + @create_model + class CustomerModel: + id: int + name: str + + On a base class: + + @dataclass_transform() + class ModelBase: ... + + class CustomerModel(ModelBase): + id: int + name: str + + On a metaclass: + + @dataclass_transform() + class ModelMeta(type): ... + + class ModelBase(metaclass=ModelMeta): ... + + class CustomerModel(ModelBase): + id: int + name: str + + Each of the ``CustomerModel`` classes defined in this example will now + behave similarly to a dataclass created with the ``@dataclasses.dataclass`` + decorator. For example, the type checker will synthesize an ``__init__`` + method. + + The arguments to this decorator can be used to customize this behavior: + - ``eq_default`` indicates whether the ``eq`` parameter is assumed to be + True or False if it is omitted by the caller. + - ``order_default`` indicates whether the ``order`` parameter is + assumed to be True or False if it is omitted by the caller. + - ``kw_only_default`` indicates whether the ``kw_only`` parameter is + assumed to be True or False if it is omitted by the caller. + - ``field_specifiers`` specifies a static list of supported classes + or functions that describe fields, similar to ``dataclasses.field()``. + + At runtime, this decorator records its arguments in the + ``__dataclass_transform__`` attribute on the decorated object. + It has no other runtime effect. + + See PEP 681 for more details. + """ + def decorator(cls_or_fn): + cls_or_fn.__dataclass_transform__ = { + "eq_default": eq_default, + "order_default": order_default, + "kw_only_default": kw_only_default, + "field_specifiers": field_specifiers, + "kwargs": kwargs, + } + return cls_or_fn + return decorator diff --git a/Misc/NEWS.d/next/Library/2022-04-23-08-06-36.gh-issue-91860.ityDjK.rst b/Misc/NEWS.d/next/Library/2022-04-23-08-06-36.gh-issue-91860.ityDjK.rst new file mode 100644 index 0000000000000..d5e81c99887ff --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-23-08-06-36.gh-issue-91860.ityDjK.rst @@ -0,0 +1,2 @@ +Add :func:`typing.dataclass_transform`, implementing :pep:`681`. Patch by +Jelle Zijlstra. From webhook-mailer at python.org Tue Apr 26 00:58:45 2022 From: webhook-mailer at python.org (serhiy-storchaka) Date: Tue, 26 Apr 2022 04:58:45 -0000 Subject: [Python-checkins] gh-91915: Fix test_netrc on non-UTF-8 locale (GH-91918) Message-ID: https://github.com/python/cpython/commit/36306cf7862097318a3fef74224075cc4cf37229 commit: 36306cf7862097318a3fef74224075cc4cf37229 branch: main author: Serhiy Storchaka committer: serhiy-storchaka date: 2022-04-26T07:58:41+03:00 summary: gh-91915: Fix test_netrc on non-UTF-8 locale (GH-91918) files: M Lib/test/test_netrc.py diff --git a/Lib/test/test_netrc.py b/Lib/test/test_netrc.py index a6b4bc47a32c1..3cca1e8ff1ac1 100644 --- a/Lib/test/test_netrc.py +++ b/Lib/test/test_netrc.py @@ -10,7 +10,7 @@ def make_nrc(self, test_data): mode = 'w' if sys.platform != 'cygwin': mode += 't' - with open(temp_filename, mode) as fp: + with open(temp_filename, mode, encoding="utf-8") as fp: fp.write(test_data) try: nrc = netrc.netrc(temp_filename) From webhook-mailer at python.org Tue Apr 26 01:00:00 2022 From: webhook-mailer at python.org (serhiy-storchaka) Date: Tue, 26 Apr 2022 05:00:00 -0000 Subject: [Python-checkins] gh-91914: Fix test_curses on non-UTF-8 locale (GH-91919) Message-ID: https://github.com/python/cpython/commit/f41c16bf512778fca4bfabca887c4c303cc21896 commit: f41c16bf512778fca4bfabca887c4c303cc21896 branch: main author: Serhiy Storchaka committer: serhiy-storchaka date: 2022-04-26T07:59:56+03:00 summary: gh-91914: Fix test_curses on non-UTF-8 locale (GH-91919) files: M Lib/test/test_curses.py diff --git a/Lib/test/test_curses.py b/Lib/test/test_curses.py index d3c152c42cf62..b550f4af555ce 100644 --- a/Lib/test/test_curses.py +++ b/Lib/test/test_curses.py @@ -268,7 +268,12 @@ def test_output_character(self): stdscr.echochar(b'A') stdscr.echochar(65) with self.assertRaises((UnicodeEncodeError, OverflowError)): - stdscr.echochar('\u20ac') + # Unicode is not fully supported yet, but at least it does + # not crash. + # It is supposed to fail because either the character is + # not encodable with the current encoding, or it is encoded to + # a multibyte sequence. + stdscr.echochar('\u0114') stdscr.echochar('A', curses.A_BOLD) self.assertIs(stdscr.is_wintouched(), False) From webhook-mailer at python.org Tue Apr 26 01:00:51 2022 From: webhook-mailer at python.org (serhiy-storchaka) Date: Tue, 26 Apr 2022 05:00:51 -0000 Subject: [Python-checkins] gh-91916: Fix test_runpy on non-UTF-8 locale (GH-91920) Message-ID: https://github.com/python/cpython/commit/a568585069174cec35ce26cdf4d4862c634d9f6d commit: a568585069174cec35ce26cdf4d4862c634d9f6d branch: main author: Serhiy Storchaka committer: serhiy-storchaka date: 2022-04-26T08:00:47+03:00 summary: gh-91916: Fix test_runpy on non-UTF-8 locale (GH-91920) If use a non-builtin codec, partially implemented in Python (e.g. ISO-8859-15), a new RecursionError (with empty error message) can be raised while handle a RecursionError. Testing for error message was needed to distinguish a recursion error from arbitrary RuntimeError. After introducing RecursionError, it became unnecessary. files: M Lib/test/test_runpy.py diff --git a/Lib/test/test_runpy.py b/Lib/test/test_runpy.py index 80e695a5f3f79..93e1ab556ae92 100644 --- a/Lib/test/test_runpy.py +++ b/Lib/test/test_runpy.py @@ -740,8 +740,7 @@ def test_main_recursion_error(self): "runpy.run_path(%r)\n") % dummy_dir script_name = self._make_test_script(script_dir, mod_name, source) zip_name, fname = make_zip_script(script_dir, 'test_zip', script_name) - msg = "recursion depth exceeded" - self.assertRaisesRegex(RecursionError, msg, run_path, zip_name) + self.assertRaises(RecursionError, run_path, zip_name) def test_encoding(self): with temp_dir() as script_dir: From webhook-mailer at python.org Tue Apr 26 01:01:37 2022 From: webhook-mailer at python.org (serhiy-storchaka) Date: Tue, 26 Apr 2022 05:01:37 -0000 Subject: [Python-checkins] gh-91917: Fix test_zipfile on non-UTF-8 locale (GH-91921) Message-ID: https://github.com/python/cpython/commit/4153f2cbcb41a1a9057bfba28d5f65d48ea39283 commit: 4153f2cbcb41a1a9057bfba28d5f65d48ea39283 branch: main author: Serhiy Storchaka committer: serhiy-storchaka date: 2022-04-26T08:01:33+03:00 summary: gh-91917: Fix test_zipfile on non-UTF-8 locale (GH-91921) Skip the extraction test if file names are not encodable. files: M Lib/test/test_zipfile.py diff --git a/Lib/test/test_zipfile.py b/Lib/test/test_zipfile.py index 17111b3a40fef..848bf4f76d453 100644 --- a/Lib/test/test_zipfile.py +++ b/Lib/test/test_zipfile.py @@ -3391,8 +3391,19 @@ def test_cli_with_metadata_encoding(self): for name in self.file_names: self.assertIn(name, listing) + def test_cli_with_metadata_encoding_extract(self): os.mkdir(TESTFN2) self.addCleanup(rmtree, TESTFN2) + # Depending on locale, extracted file names can be not encodable + # with the filesystem encoding. + for fn in self.file_names: + try: + os.stat(os.path.join(TESTFN2, fn)) + except OSError: + pass + except UnicodeEncodeError: + self.skipTest(f'cannot encode file name {fn!r}') + zipfile.main(["--metadata-encoding=shift_jis", "-e", TESTFN, TESTFN2]) listing = os.listdir(TESTFN2) for name in self.file_names: From webhook-mailer at python.org Tue Apr 26 01:27:22 2022 From: webhook-mailer at python.org (miss-islington) Date: Tue, 26 Apr 2022 05:27:22 -0000 Subject: [Python-checkins] gh-91916: Fix test_runpy on non-UTF-8 locale (GH-91920) Message-ID: https://github.com/python/cpython/commit/79712c9d2ec7a338b87d1a8b31a8404191e83823 commit: 79712c9d2ec7a338b87d1a8b31a8404191e83823 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-25T22:26:47-07:00 summary: gh-91916: Fix test_runpy on non-UTF-8 locale (GH-91920) If use a non-builtin codec, partially implemented in Python (e.g. ISO-8859-15), a new RecursionError (with empty error message) can be raised while handle a RecursionError. Testing for error message was needed to distinguish a recursion error from arbitrary RuntimeError. After introducing RecursionError, it became unnecessary. (cherry picked from commit a568585069174cec35ce26cdf4d4862c634d9f6d) Co-authored-by: Serhiy Storchaka files: M Lib/test/test_runpy.py diff --git a/Lib/test/test_runpy.py b/Lib/test/test_runpy.py index 2954dfedc7e42..e6d36825f15a1 100644 --- a/Lib/test/test_runpy.py +++ b/Lib/test/test_runpy.py @@ -740,8 +740,7 @@ def test_main_recursion_error(self): "runpy.run_path(%r)\n") % dummy_dir script_name = self._make_test_script(script_dir, mod_name, source) zip_name, fname = make_zip_script(script_dir, 'test_zip', script_name) - msg = "recursion depth exceeded" - self.assertRaisesRegex(RecursionError, msg, run_path, zip_name) + self.assertRaises(RecursionError, run_path, zip_name) def test_encoding(self): with temp_dir() as script_dir: From webhook-mailer at python.org Tue Apr 26 01:45:28 2022 From: webhook-mailer at python.org (miss-islington) Date: Tue, 26 Apr 2022 05:45:28 -0000 Subject: [Python-checkins] gh-91916: Fix test_runpy on non-UTF-8 locale (GH-91920) Message-ID: https://github.com/python/cpython/commit/653a66668df818aafb76f968247cb02d2f9cd405 commit: 653a66668df818aafb76f968247cb02d2f9cd405 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-25T22:45:24-07:00 summary: gh-91916: Fix test_runpy on non-UTF-8 locale (GH-91920) If use a non-builtin codec, partially implemented in Python (e.g. ISO-8859-15), a new RecursionError (with empty error message) can be raised while handle a RecursionError. Testing for error message was needed to distinguish a recursion error from arbitrary RuntimeError. After introducing RecursionError, it became unnecessary. (cherry picked from commit a568585069174cec35ce26cdf4d4862c634d9f6d) Co-authored-by: Serhiy Storchaka files: M Lib/test/test_runpy.py diff --git a/Lib/test/test_runpy.py b/Lib/test/test_runpy.py index 4aaf75b1bceb6..c734f0d6ce2d6 100644 --- a/Lib/test/test_runpy.py +++ b/Lib/test/test_runpy.py @@ -740,8 +740,7 @@ def test_main_recursion_error(self): "runpy.run_path(%r)\n") % dummy_dir script_name = self._make_test_script(script_dir, mod_name, source) zip_name, fname = make_zip_script(script_dir, 'test_zip', script_name) - msg = "recursion depth exceeded" - self.assertRaisesRegex(RecursionError, msg, run_path, zip_name) + self.assertRaises(RecursionError, run_path, zip_name) def test_encoding(self): with temp_dir() as script_dir: From webhook-mailer at python.org Tue Apr 26 01:51:31 2022 From: webhook-mailer at python.org (miss-islington) Date: Tue, 26 Apr 2022 05:51:31 -0000 Subject: [Python-checkins] gh-91914: Fix test_curses on non-UTF-8 locale (GH-91919) Message-ID: https://github.com/python/cpython/commit/f4252dfb8edea01a892be8d29d3a0747e381eb4e commit: f4252dfb8edea01a892be8d29d3a0747e381eb4e branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-25T22:51:27-07:00 summary: gh-91914: Fix test_curses on non-UTF-8 locale (GH-91919) (cherry picked from commit f41c16bf512778fca4bfabca887c4c303cc21896) Co-authored-by: Serhiy Storchaka files: M Lib/test/test_curses.py diff --git a/Lib/test/test_curses.py b/Lib/test/test_curses.py index 4bdc2379fd1ed..48b1d53f804bb 100644 --- a/Lib/test/test_curses.py +++ b/Lib/test/test_curses.py @@ -266,7 +266,12 @@ def test_output_character(self): stdscr.echochar(b'A') stdscr.echochar(65) with self.assertRaises((UnicodeEncodeError, OverflowError)): - stdscr.echochar('\u20ac') + # Unicode is not fully supported yet, but at least it does + # not crash. + # It is supposed to fail because either the character is + # not encodable with the current encoding, or it is encoded to + # a multibyte sequence. + stdscr.echochar('\u0114') stdscr.echochar('A', curses.A_BOLD) self.assertIs(stdscr.is_wintouched(), False) From webhook-mailer at python.org Tue Apr 26 01:58:13 2022 From: webhook-mailer at python.org (miss-islington) Date: Tue, 26 Apr 2022 05:58:13 -0000 Subject: [Python-checkins] gh-91914: Fix test_curses on non-UTF-8 locale (GH-91919) Message-ID: https://github.com/python/cpython/commit/3e219d312321a1dbb3a475607dfcaaa5925fed79 commit: 3e219d312321a1dbb3a475607dfcaaa5925fed79 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-25T22:57:57-07:00 summary: gh-91914: Fix test_curses on non-UTF-8 locale (GH-91919) (cherry picked from commit f41c16bf512778fca4bfabca887c4c303cc21896) Co-authored-by: Serhiy Storchaka files: M Lib/test/test_curses.py diff --git a/Lib/test/test_curses.py b/Lib/test/test_curses.py index d3c152c42cf62..b550f4af555ce 100644 --- a/Lib/test/test_curses.py +++ b/Lib/test/test_curses.py @@ -268,7 +268,12 @@ def test_output_character(self): stdscr.echochar(b'A') stdscr.echochar(65) with self.assertRaises((UnicodeEncodeError, OverflowError)): - stdscr.echochar('\u20ac') + # Unicode is not fully supported yet, but at least it does + # not crash. + # It is supposed to fail because either the character is + # not encodable with the current encoding, or it is encoded to + # a multibyte sequence. + stdscr.echochar('\u0114') stdscr.echochar('A', curses.A_BOLD) self.assertIs(stdscr.is_wintouched(), False) From webhook-mailer at python.org Tue Apr 26 03:50:55 2022 From: webhook-mailer at python.org (serhiy-storchaka) Date: Tue, 26 Apr 2022 07:50:55 -0000 Subject: [Python-checkins] [3.10] gh-91915: Fix test_netrc on non-UTF-8 locale (GH-91918). (GH-91946) Message-ID: https://github.com/python/cpython/commit/dbe666d39880352360cb07787d47237448196bf4 commit: dbe666d39880352360cb07787d47237448196bf4 branch: 3.10 author: Serhiy Storchaka committer: serhiy-storchaka date: 2022-04-26T10:50:22+03:00 summary: [3.10] gh-91915: Fix test_netrc on non-UTF-8 locale (GH-91918). (GH-91946) (cherry picked from commit 36306cf7862097318a3fef74224075cc4cf37229) Co-authored-by: Serhiy Storchaka files: M Lib/test/test_netrc.py diff --git a/Lib/test/test_netrc.py b/Lib/test/test_netrc.py index 90ef5cd363b3f..8c76bc701e494 100644 --- a/Lib/test/test_netrc.py +++ b/Lib/test/test_netrc.py @@ -11,7 +11,7 @@ def make_nrc(self, test_data): if sys.platform != 'cygwin': mode += 't' temp_fd, temp_filename = tempfile.mkstemp() - with os.fdopen(temp_fd, mode=mode) as fp: + with os.fdopen(temp_fd, mode=mode, encoding="utf-8") as fp: fp.write(test_data) self.addCleanup(os.unlink, temp_filename) return netrc.netrc(temp_filename) From webhook-mailer at python.org Tue Apr 26 13:45:19 2022 From: webhook-mailer at python.org (brettcannon) Date: Tue, 26 Apr 2022 17:45:19 -0000 Subject: [Python-checkins] gh-91217: deprecate telnetlib (GH-91958) Message-ID: https://github.com/python/cpython/commit/1af871eeee4a01cc21a6960d087e9ab8ce4c9f4d commit: 1af871eeee4a01cc21a6960d087e9ab8ce4c9f4d branch: main author: Brett Cannon committer: brettcannon date: 2022-04-26T10:45:08-07:00 summary: gh-91217: deprecate telnetlib (GH-91958) files: A Misc/NEWS.d/next/Library/2022-04-17-12-41-52.gh-issue-91217.3wnHSX.rst M Doc/whatsnew/3.11.rst M Lib/telnetlib.py M Lib/test/test_telnetlib.py diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index 8f0d3c7c7099a..b812658d5e91e 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -1069,6 +1069,7 @@ Deprecated * :mod:`sndhdr` * :mod:`spwd` * :mod:`sunau` + * :mod:`telnetlib` (Contributed by Brett Cannon in :issue:`47061`.) diff --git a/Lib/telnetlib.py b/Lib/telnetlib.py index ae88ea594746f..62d636129853a 100644 --- a/Lib/telnetlib.py +++ b/Lib/telnetlib.py @@ -37,6 +37,9 @@ import socket import selectors from time import monotonic as _time +import warnings + +warnings._deprecated(__name__, remove=(3, 13)) __all__ = ["Telnet"] diff --git a/Lib/test/test_telnetlib.py b/Lib/test/test_telnetlib.py index b50df1459d1f4..a9cade2ee466f 100644 --- a/Lib/test/test_telnetlib.py +++ b/Lib/test/test_telnetlib.py @@ -1,15 +1,16 @@ import socket import selectors -import telnetlib import threading import contextlib from test import support -from test.support import socket_helper +from test.support import socket_helper, warnings_helper import unittest support.requires_working_socket(module=True) +telnetlib = warnings_helper.import_deprecated('telnetlib') + HOST = socket_helper.HOST def server(evt, serv): diff --git a/Misc/NEWS.d/next/Library/2022-04-17-12-41-52.gh-issue-91217.3wnHSX.rst b/Misc/NEWS.d/next/Library/2022-04-17-12-41-52.gh-issue-91217.3wnHSX.rst new file mode 100644 index 0000000000000..6c4acc7924332 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-17-12-41-52.gh-issue-91217.3wnHSX.rst @@ -0,0 +1 @@ +Deprecate the telnetlib module. From webhook-mailer at python.org Tue Apr 26 14:07:33 2022 From: webhook-mailer at python.org (serhiy-storchaka) Date: Tue, 26 Apr 2022 18:07:33 -0000 Subject: [Python-checkins] gh-91870: Remove unsupported SRE opcode CALL (GH-91872) Message-ID: https://github.com/python/cpython/commit/f703c96cf08a0d65e718e7acfb043cbc49812a22 commit: f703c96cf08a0d65e718e7acfb043cbc49812a22 branch: main author: Serhiy Storchaka committer: serhiy-storchaka date: 2022-04-26T21:07:25+03:00 summary: gh-91870: Remove unsupported SRE opcode CALL (GH-91872) It was initially added to support atomic groups, but that support was never fully implemented, and CALL was only left in the compiler, but not interpreter and parser. ATOMIC_GROUP is now used to support atomic groups. files: M Lib/re/_compiler.py M Lib/re/_constants.py M Lib/re/_parser.py M Modules/_sre/sre_constants.h M Modules/_sre/sre_lib.h M Modules/_sre/sre_targets.h diff --git a/Lib/re/_compiler.py b/Lib/re/_compiler.py index f621d04af123d..63d82025505b7 100644 --- a/Lib/re/_compiler.py +++ b/Lib/re/_compiler.py @@ -164,12 +164,6 @@ def _compile(data, pattern, flags): _compile(data, av[1], flags) emit(SUCCESS) code[skip] = _len(code) - skip - elif op is CALL: - emit(op) - skip = _len(code); emit(0) - _compile(data, av, flags) - emit(SUCCESS) - code[skip] = _len(code) - skip elif op is AT: emit(op) if flags & SRE_FLAG_MULTILINE: diff --git a/Lib/re/_constants.py b/Lib/re/_constants.py index aa1a590290fd9..c45ce409e21b8 100644 --- a/Lib/re/_constants.py +++ b/Lib/re/_constants.py @@ -13,7 +13,7 @@ # update when constants are added or removed -MAGIC = 20220402 +MAGIC = 20220423 from _sre import MAXREPEAT, MAXGROUPS @@ -78,7 +78,6 @@ def _makecodes(*names): 'ASSERT', 'ASSERT_NOT', 'AT', 'BRANCH', - 'CALL', 'CATEGORY', 'CHARSET', 'BIGCHARSET', 'GROUPREF', 'GROUPREF_EXISTS', diff --git a/Lib/re/_parser.py b/Lib/re/_parser.py index b864bf2b108a6..933d51589f46c 100644 --- a/Lib/re/_parser.py +++ b/Lib/re/_parser.py @@ -183,10 +183,6 @@ def getwidth(self): j = max(j, h) lo = lo + i hi = hi + j - elif op is CALL: - i, j = av.getwidth() - lo = lo + i - hi = hi + j elif op is ATOMIC_GROUP: i, j = av.getwidth() lo = lo + i diff --git a/Modules/_sre/sre_constants.h b/Modules/_sre/sre_constants.h index b1ef27eccc839..d5de650b7025a 100644 --- a/Modules/_sre/sre_constants.h +++ b/Modules/_sre/sre_constants.h @@ -11,7 +11,7 @@ * See the sre.c file for information on usage and redistribution. */ -#define SRE_MAGIC 20220402 +#define SRE_MAGIC 20220423 #define SRE_OP_FAILURE 0 #define SRE_OP_SUCCESS 1 #define SRE_OP_ANY 2 @@ -20,42 +20,41 @@ #define SRE_OP_ASSERT_NOT 5 #define SRE_OP_AT 6 #define SRE_OP_BRANCH 7 -#define SRE_OP_CALL 8 -#define SRE_OP_CATEGORY 9 -#define SRE_OP_CHARSET 10 -#define SRE_OP_BIGCHARSET 11 -#define SRE_OP_GROUPREF 12 -#define SRE_OP_GROUPREF_EXISTS 13 -#define SRE_OP_IN 14 -#define SRE_OP_INFO 15 -#define SRE_OP_JUMP 16 -#define SRE_OP_LITERAL 17 -#define SRE_OP_MARK 18 -#define SRE_OP_MAX_UNTIL 19 -#define SRE_OP_MIN_UNTIL 20 -#define SRE_OP_NOT_LITERAL 21 -#define SRE_OP_NEGATE 22 -#define SRE_OP_RANGE 23 -#define SRE_OP_REPEAT 24 -#define SRE_OP_REPEAT_ONE 25 -#define SRE_OP_SUBPATTERN 26 -#define SRE_OP_MIN_REPEAT_ONE 27 -#define SRE_OP_ATOMIC_GROUP 28 -#define SRE_OP_POSSESSIVE_REPEAT 29 -#define SRE_OP_POSSESSIVE_REPEAT_ONE 30 -#define SRE_OP_GROUPREF_IGNORE 31 -#define SRE_OP_IN_IGNORE 32 -#define SRE_OP_LITERAL_IGNORE 33 -#define SRE_OP_NOT_LITERAL_IGNORE 34 -#define SRE_OP_GROUPREF_LOC_IGNORE 35 -#define SRE_OP_IN_LOC_IGNORE 36 -#define SRE_OP_LITERAL_LOC_IGNORE 37 -#define SRE_OP_NOT_LITERAL_LOC_IGNORE 38 -#define SRE_OP_GROUPREF_UNI_IGNORE 39 -#define SRE_OP_IN_UNI_IGNORE 40 -#define SRE_OP_LITERAL_UNI_IGNORE 41 -#define SRE_OP_NOT_LITERAL_UNI_IGNORE 42 -#define SRE_OP_RANGE_UNI_IGNORE 43 +#define SRE_OP_CATEGORY 8 +#define SRE_OP_CHARSET 9 +#define SRE_OP_BIGCHARSET 10 +#define SRE_OP_GROUPREF 11 +#define SRE_OP_GROUPREF_EXISTS 12 +#define SRE_OP_IN 13 +#define SRE_OP_INFO 14 +#define SRE_OP_JUMP 15 +#define SRE_OP_LITERAL 16 +#define SRE_OP_MARK 17 +#define SRE_OP_MAX_UNTIL 18 +#define SRE_OP_MIN_UNTIL 19 +#define SRE_OP_NOT_LITERAL 20 +#define SRE_OP_NEGATE 21 +#define SRE_OP_RANGE 22 +#define SRE_OP_REPEAT 23 +#define SRE_OP_REPEAT_ONE 24 +#define SRE_OP_SUBPATTERN 25 +#define SRE_OP_MIN_REPEAT_ONE 26 +#define SRE_OP_ATOMIC_GROUP 27 +#define SRE_OP_POSSESSIVE_REPEAT 28 +#define SRE_OP_POSSESSIVE_REPEAT_ONE 29 +#define SRE_OP_GROUPREF_IGNORE 30 +#define SRE_OP_IN_IGNORE 31 +#define SRE_OP_LITERAL_IGNORE 32 +#define SRE_OP_NOT_LITERAL_IGNORE 33 +#define SRE_OP_GROUPREF_LOC_IGNORE 34 +#define SRE_OP_IN_LOC_IGNORE 35 +#define SRE_OP_LITERAL_LOC_IGNORE 36 +#define SRE_OP_NOT_LITERAL_LOC_IGNORE 37 +#define SRE_OP_GROUPREF_UNI_IGNORE 38 +#define SRE_OP_IN_UNI_IGNORE 39 +#define SRE_OP_LITERAL_UNI_IGNORE 40 +#define SRE_OP_NOT_LITERAL_UNI_IGNORE 41 +#define SRE_OP_RANGE_UNI_IGNORE 42 #define SRE_AT_BEGINNING 0 #define SRE_AT_BEGINNING_LINE 1 #define SRE_AT_BEGINNING_STRING 2 diff --git a/Modules/_sre/sre_lib.h b/Modules/_sre/sre_lib.h index efd6fdeccce3f..1e5b50170ae76 100644 --- a/Modules/_sre/sre_lib.h +++ b/Modules/_sre/sre_lib.h @@ -1547,7 +1547,6 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) TARGET(SRE_OP_NEGATE): TARGET(SRE_OP_BIGCHARSET): TARGET(SRE_OP_CHARSET): - TARGET(SRE_OP_CALL): TRACE(("|%p|%p|UNKNOWN %d\n", pattern, ptr, pattern[-1])); RETURN_ERROR(SRE_ERROR_ILLEGAL); diff --git a/Modules/_sre/sre_targets.h b/Modules/_sre/sre_targets.h index 389e7d77e949c..25b6edd436bb7 100644 --- a/Modules/_sre/sre_targets.h +++ b/Modules/_sre/sre_targets.h @@ -11,7 +11,7 @@ * See the sre.c file for information on usage and redistribution. */ -static void *sre_targets[44] = { +static void *sre_targets[43] = { &&TARGET_SRE_OP_FAILURE, &&TARGET_SRE_OP_SUCCESS, &&TARGET_SRE_OP_ANY, @@ -20,7 +20,6 @@ static void *sre_targets[44] = { &&TARGET_SRE_OP_ASSERT_NOT, &&TARGET_SRE_OP_AT, &&TARGET_SRE_OP_BRANCH, - &&TARGET_SRE_OP_CALL, &&TARGET_SRE_OP_CATEGORY, &&TARGET_SRE_OP_CHARSET, &&TARGET_SRE_OP_BIGCHARSET, From webhook-mailer at python.org Tue Apr 26 15:12:11 2022 From: webhook-mailer at python.org (miss-islington) Date: Tue, 26 Apr 2022 19:12:11 -0000 Subject: [Python-checkins] bpo-21761: Clarify __file__/__cached__ in import reference (GH-31565) Message-ID: https://github.com/python/cpython/commit/b87f7f2c213225cbe5c4bd8f8a83883e0824c7d0 commit: b87f7f2c213225cbe5c4bd8f8a83883e0824c7d0 branch: main author: slateny <46876382+slateny at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-26T12:12:02-07:00 summary: bpo-21761: Clarify __file__/__cached__ in import reference (GH-31565) Automerge-Triggered-By: GH:brettcannon files: M Doc/reference/import.rst diff --git a/Doc/reference/import.rst b/Doc/reference/import.rst index 988d41c81c6ab..3a41403c6476b 100644 --- a/Doc/reference/import.rst +++ b/Doc/reference/import.rst @@ -610,15 +610,16 @@ the module. import system may opt to leave it unset if it has no semantic meaning (e.g. a module loaded from a database). - If ``__file__`` is set, it may also be appropriate to set the - ``__cached__`` attribute which is the path to any compiled version of + If ``__file__`` is set then the ``__cached__`` attribute might also + be set, which is the path to any compiled version of the code (e.g. byte-compiled file). The file does not need to exist to set this attribute; the path can simply point to where the compiled file would exist (see :pep:`3147`). - It is also appropriate to set ``__cached__`` when ``__file__`` is not + Note that ``__cached__`` may be set even if ``__file__`` is not set. However, that scenario is quite atypical. Ultimately, the - loader is what makes use of ``__file__`` and/or ``__cached__``. So + loader is what makes use of the module spec provided by the finder + (from which ``__file__`` and ``__cached__`` are derived). So if a loader can load from a cached module but otherwise does not load from a file, that atypical scenario may be appropriate. From webhook-mailer at python.org Tue Apr 26 16:44:01 2022 From: webhook-mailer at python.org (vstinner) Date: Tue, 26 Apr 2022 20:44:01 -0000 Subject: [Python-checkins] gh-68966: Deprecate the mailcap module (#91951) Message-ID: https://github.com/python/cpython/commit/80de0273c0caf8bae19787bb00255eb3fb2a2d0c commit: 80de0273c0caf8bae19787bb00255eb3fb2a2d0c branch: main author: Victor Stinner committer: vstinner date: 2022-04-26T22:43:50+02:00 summary: gh-68966: Deprecate the mailcap module (#91951) files: A Misc/NEWS.d/next/Library/2022-04-26-09-09-07.gh-issue-68966.roapI2.rst M Doc/library/mailcap.rst M Doc/library/netdata.rst M Doc/library/superseded.rst M Doc/whatsnew/3.11.rst M Lib/mailcap.py M Lib/test/test_mailcap.py diff --git a/Doc/library/mailcap.rst b/Doc/library/mailcap.rst index 7749b7dd45ef4..416b181f45a77 100644 --- a/Doc/library/mailcap.rst +++ b/Doc/library/mailcap.rst @@ -3,9 +3,14 @@ .. module:: mailcap :synopsis: Mailcap file handling. + :deprecated: **Source code:** :source:`Lib/mailcap.py` +.. deprecated:: 3.11 + The :mod:`mailcap` module is deprecated. See :pep:`594` for the rationale + and the :mod:`mimetypes` module for an alternative. + -------------- Mailcap files are used to configure how MIME-aware applications such as mail diff --git a/Doc/library/netdata.rst b/Doc/library/netdata.rst index 8955e859ab634..1541e2a544459 100644 --- a/Doc/library/netdata.rst +++ b/Doc/library/netdata.rst @@ -13,7 +13,6 @@ on the internet. email.rst json.rst - mailcap.rst mailbox.rst mimetypes.rst base64.rst diff --git a/Doc/library/superseded.rst b/Doc/library/superseded.rst index e3f9b0d37fe10..b38f16691f6ea 100644 --- a/Doc/library/superseded.rst +++ b/Doc/library/superseded.rst @@ -20,9 +20,10 @@ backwards compatibility. They have been superseded by other modules. crypt.rst imghdr.rst imp.rst + mailcap.rst msilib.rst - nntplib.rst nis.rst + nntplib.rst optparse.rst ossaudiodev.rst pipes.rst diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index b812658d5e91e..aa0a51b4375f1 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -1061,6 +1061,7 @@ Deprecated * :mod:`chunk` * :mod:`crypt` * :mod:`imghdr` + * :mod:`mailcap` * :mod:`msilib` * :mod:`nis` * :mod:`nntplib` @@ -1071,7 +1072,8 @@ Deprecated * :mod:`sunau` * :mod:`telnetlib` - (Contributed by Brett Cannon in :issue:`47061`.) + (Contributed by Brett Cannon in :issue:`47061` and Victor Stinner in + :gh:`68966`.) Removed diff --git a/Lib/mailcap.py b/Lib/mailcap.py index ae416a8e9fb27..856b6a55475f3 100644 --- a/Lib/mailcap.py +++ b/Lib/mailcap.py @@ -6,6 +6,12 @@ __all__ = ["getcaps","findmatch"] +_DEPRECATION_MSG = ('The {name} module is deprecated and will be removed in ' + 'Python {remove}. See the mimetypes module for an ' + 'alternative.') +warnings._deprecated(__name__, _DEPRECATION_MSG, remove=(3, 13)) + + def lineno_sort_key(entry): # Sort in ascending order, with unspecified entries at the end if 'lineno' in entry: diff --git a/Lib/test/test_mailcap.py b/Lib/test/test_mailcap.py index ef9cad498a75c..97a8fac6e074a 100644 --- a/Lib/test/test_mailcap.py +++ b/Lib/test/test_mailcap.py @@ -1,10 +1,15 @@ -import mailcap -import os import copy +import os +import sys import test.support -from test.support import os_helper import unittest -import sys +import warnings +from test.support import os_helper +from test.support import warnings_helper + + +mailcap = warnings_helper.import_deprecated('mailcap') + # Location of mailcap file MAILCAPFILE = test.support.findfile("mailcap.txt") diff --git a/Misc/NEWS.d/next/Library/2022-04-26-09-09-07.gh-issue-68966.roapI2.rst b/Misc/NEWS.d/next/Library/2022-04-26-09-09-07.gh-issue-68966.roapI2.rst new file mode 100644 index 0000000000000..5c9ffbf09f005 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-26-09-09-07.gh-issue-68966.roapI2.rst @@ -0,0 +1,3 @@ +The :mod:`mailcap` module is now deprecated and will be removed in Python 3.13. +See :pep:`594` for the rationale and the :mod:`mimetypes` module for an +alternative. Patch by Victor Stinner. From webhook-mailer at python.org Wed Apr 27 02:31:31 2022 From: webhook-mailer at python.org (serhiy-storchaka) Date: Wed, 27 Apr 2022 06:31:31 -0000 Subject: [Python-checkins] Fix missing `f` prefix on f-strings (GH-91910) Message-ID: https://github.com/python/cpython/commit/f882d33778ee2625ab32d90e28edb6878fb8af93 commit: f882d33778ee2625ab32d90e28edb6878fb8af93 branch: main author: Alexander Shadchin committer: serhiy-storchaka date: 2022-04-27T09:30:54+03:00 summary: Fix missing `f` prefix on f-strings (GH-91910) files: A Misc/NEWS.d/next/Library/2022-04-25-14-18-01.gh-issue-91910.kY-JR0.rst M Lib/asyncio/base_events.py M Lib/multiprocessing/util.py diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py index 5eea1658df8f6..fa00bf9a2ca09 100644 --- a/Lib/asyncio/base_events.py +++ b/Lib/asyncio/base_events.py @@ -888,7 +888,7 @@ async def _sock_sendfile_native(self, sock, file, offset, count): # non-mmap files even if sendfile is supported by OS raise exceptions.SendfileNotAvailableError( f"syscall sendfile is not available for socket {sock!r} " - "and file {file!r} combination") + f"and file {file!r} combination") async def _sock_sendfile_fallback(self, sock, file, offset, count): if offset: diff --git a/Lib/multiprocessing/util.py b/Lib/multiprocessing/util.py index 790955e7b71fa..6ad4632fccb9e 100644 --- a/Lib/multiprocessing/util.py +++ b/Lib/multiprocessing/util.py @@ -120,7 +120,7 @@ def is_abstract_socket_namespace(address): return address[0] == 0 elif isinstance(address, str): return address[0] == "\0" - raise TypeError('address type of {address!r} unrecognized') + raise TypeError(f'address type of {address!r} unrecognized') abstract_sockets_supported = _platform_supports_abstract_sockets() diff --git a/Misc/NEWS.d/next/Library/2022-04-25-14-18-01.gh-issue-91910.kY-JR0.rst b/Misc/NEWS.d/next/Library/2022-04-25-14-18-01.gh-issue-91910.kY-JR0.rst new file mode 100644 index 0000000000000..f41f357ddfcc3 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-25-14-18-01.gh-issue-91910.kY-JR0.rst @@ -0,0 +1 @@ +Add missing f prefix to f-strings in error messages from the :mod:`multiprocessing` and :mod:`asyncio` modules. From webhook-mailer at python.org Wed Apr 27 03:01:47 2022 From: webhook-mailer at python.org (miss-islington) Date: Wed, 27 Apr 2022 07:01:47 -0000 Subject: [Python-checkins] Fix missing `f` prefix on f-strings (GH-91910) Message-ID: https://github.com/python/cpython/commit/081e95165e2482c80ec05ff3aa4ddad103e64b61 commit: 081e95165e2482c80ec05ff3aa4ddad103e64b61 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-27T00:01:11-07:00 summary: Fix missing `f` prefix on f-strings (GH-91910) (cherry picked from commit f882d33778ee2625ab32d90e28edb6878fb8af93) Co-authored-by: Alexander Shadchin files: A Misc/NEWS.d/next/Library/2022-04-25-14-18-01.gh-issue-91910.kY-JR0.rst M Lib/asyncio/base_events.py M Lib/multiprocessing/util.py diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py index 7a14e5e139f02..4356bfae01798 100644 --- a/Lib/asyncio/base_events.py +++ b/Lib/asyncio/base_events.py @@ -885,7 +885,7 @@ async def _sock_sendfile_native(self, sock, file, offset, count): # non-mmap files even if sendfile is supported by OS raise exceptions.SendfileNotAvailableError( f"syscall sendfile is not available for socket {sock!r} " - "and file {file!r} combination") + f"and file {file!r} combination") async def _sock_sendfile_fallback(self, sock, file, offset, count): if offset: diff --git a/Lib/multiprocessing/util.py b/Lib/multiprocessing/util.py index 21f2a7ebe2500..697b35a0eb09d 100644 --- a/Lib/multiprocessing/util.py +++ b/Lib/multiprocessing/util.py @@ -120,7 +120,7 @@ def is_abstract_socket_namespace(address): return address[0] == 0 elif isinstance(address, str): return address[0] == "\0" - raise TypeError('address type of {address!r} unrecognized') + raise TypeError(f'address type of {address!r} unrecognized') abstract_sockets_supported = _platform_supports_abstract_sockets() diff --git a/Misc/NEWS.d/next/Library/2022-04-25-14-18-01.gh-issue-91910.kY-JR0.rst b/Misc/NEWS.d/next/Library/2022-04-25-14-18-01.gh-issue-91910.kY-JR0.rst new file mode 100644 index 0000000000000..f41f357ddfcc3 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-25-14-18-01.gh-issue-91910.kY-JR0.rst @@ -0,0 +1 @@ +Add missing f prefix to f-strings in error messages from the :mod:`multiprocessing` and :mod:`asyncio` modules. From webhook-mailer at python.org Wed Apr 27 03:08:16 2022 From: webhook-mailer at python.org (miss-islington) Date: Wed, 27 Apr 2022 07:08:16 -0000 Subject: [Python-checkins] Fix missing `f` prefix on f-strings (GH-91910) Message-ID: https://github.com/python/cpython/commit/280749a8fdfa9b42e249b7e09d2b44607fff228d commit: 280749a8fdfa9b42e249b7e09d2b44607fff228d branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-27T00:08:05-07:00 summary: Fix missing `f` prefix on f-strings (GH-91910) (cherry picked from commit f882d33778ee2625ab32d90e28edb6878fb8af93) Co-authored-by: Alexander Shadchin files: A Misc/NEWS.d/next/Library/2022-04-25-14-18-01.gh-issue-91910.kY-JR0.rst M Lib/asyncio/base_events.py M Lib/multiprocessing/util.py diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py index 952da11064fef..ea10399b94131 100644 --- a/Lib/asyncio/base_events.py +++ b/Lib/asyncio/base_events.py @@ -884,7 +884,7 @@ async def _sock_sendfile_native(self, sock, file, offset, count): # non-mmap files even if sendfile is supported by OS raise exceptions.SendfileNotAvailableError( f"syscall sendfile is not available for socket {sock!r} " - "and file {file!r} combination") + f"and file {file!r} combination") async def _sock_sendfile_fallback(self, sock, file, offset, count): if offset: diff --git a/Lib/multiprocessing/util.py b/Lib/multiprocessing/util.py index a4683339820f5..9e07a4e93e597 100644 --- a/Lib/multiprocessing/util.py +++ b/Lib/multiprocessing/util.py @@ -120,7 +120,7 @@ def is_abstract_socket_namespace(address): return address[0] == 0 elif isinstance(address, str): return address[0] == "\0" - raise TypeError('address type of {address!r} unrecognized') + raise TypeError(f'address type of {address!r} unrecognized') abstract_sockets_supported = _platform_supports_abstract_sockets() diff --git a/Misc/NEWS.d/next/Library/2022-04-25-14-18-01.gh-issue-91910.kY-JR0.rst b/Misc/NEWS.d/next/Library/2022-04-25-14-18-01.gh-issue-91910.kY-JR0.rst new file mode 100644 index 0000000000000..f41f357ddfcc3 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-25-14-18-01.gh-issue-91910.kY-JR0.rst @@ -0,0 +1 @@ +Add missing f prefix to f-strings in error messages from the :mod:`multiprocessing` and :mod:`asyncio` modules. From webhook-mailer at python.org Wed Apr 27 04:41:41 2022 From: webhook-mailer at python.org (vstinner) Date: Wed, 27 Apr 2022 08:41:41 -0000 Subject: [Python-checkins] gh-91320: Add _Py_reinterpret_cast() macro (#91959) Message-ID: https://github.com/python/cpython/commit/29e2245ad5d28b565fe0d0d667d283708c6e832c commit: 29e2245ad5d28b565fe0d0d667d283708c6e832c branch: main author: Victor Stinner committer: vstinner date: 2022-04-27T10:40:57+02:00 summary: gh-91320: Add _Py_reinterpret_cast() macro (#91959) Fix C++ compiler warnings about "old-style cast" (g++ -Wold-style-cast) in the Python C API. Use C++ reinterpret_cast<> and static_cast<> casts when the Python C API is used in C++. Example of fixed warning: Include/object.h:107:43: error: use of old-style cast to ?PyObject*? {aka ?struct _object*?} [-Werror=old-style-cast] #define _PyObject_CAST(op) ((PyObject*)(op)) Add _Py_reinterpret_cast() and _Py_static_cast() macros. files: A Misc/NEWS.d/next/C API/2022-04-26-16-51-31.gh-issue-91320.QDHmTv.rst M Include/cpython/abstract.h M Include/cpython/listobject.h M Include/cpython/methodobject.h M Include/cpython/tupleobject.h M Include/cpython/unicodeobject.h M Include/methodobject.h M Include/object.h M Include/objimpl.h M Include/pyport.h diff --git a/Include/cpython/abstract.h b/Include/cpython/abstract.h index bdc0c49cd4c18..42f766da99116 100644 --- a/Include/cpython/abstract.h +++ b/Include/cpython/abstract.h @@ -50,7 +50,8 @@ PyAPI_FUNC(PyObject *) _PyObject_MakeTpCall( PyObject *const *args, Py_ssize_t nargs, PyObject *keywords); -#define PY_VECTORCALL_ARGUMENTS_OFFSET ((size_t)1 << (8 * sizeof(size_t) - 1)) +#define PY_VECTORCALL_ARGUMENTS_OFFSET \ + (_Py_static_cast(size_t, 1) << (8 * sizeof(size_t) - 1)) static inline Py_ssize_t PyVectorcall_NARGS(size_t n) diff --git a/Include/cpython/listobject.h b/Include/cpython/listobject.h index 0cd69216a4c4c..298bfcbf1438e 100644 --- a/Include/cpython/listobject.h +++ b/Include/cpython/listobject.h @@ -25,7 +25,8 @@ PyAPI_FUNC(PyObject *) _PyList_Extend(PyListObject *, PyObject *); PyAPI_FUNC(void) _PyList_DebugMallocStats(FILE *out); /* Cast argument to PyListObject* type. */ -#define _PyList_CAST(op) (assert(PyList_Check(op)), (PyListObject *)(op)) +#define _PyList_CAST(op) \ + (assert(PyList_Check(op)), _Py_reinterpret_cast(PyListObject*, (op))) // Macros and static inline functions, trading safety for speed diff --git a/Include/cpython/methodobject.h b/Include/cpython/methodobject.h index 46d177793fc4c..68cecfe05ba1e 100644 --- a/Include/cpython/methodobject.h +++ b/Include/cpython/methodobject.h @@ -8,9 +8,11 @@ PyAPI_DATA(PyTypeObject) PyCMethod_Type; #define PyCMethod_Check(op) PyObject_TypeCheck(op, &PyCMethod_Type) #define _PyCFunctionObject_CAST(func) \ - (assert(PyCFunction_Check(func)), (PyCFunctionObject *)(func)) + (assert(PyCFunction_Check(func)), \ + _Py_reinterpret_cast(PyCFunctionObject*, (func))) #define _PyCMethodObject_CAST(func) \ - (assert(PyCMethod_Check(func)), (PyCMethodObject *)(func)) + (assert(PyCMethod_Check(func)), \ + _Py_reinterpret_cast(PyCMethodObject*, (func))) /* Macros for direct access to these values. Type checks are *not* done, so use with care. */ diff --git a/Include/cpython/tupleobject.h b/Include/cpython/tupleobject.h index fd15ecd1c772f..8089e2409ce60 100644 --- a/Include/cpython/tupleobject.h +++ b/Include/cpython/tupleobject.h @@ -14,7 +14,8 @@ PyAPI_FUNC(int) _PyTuple_Resize(PyObject **, Py_ssize_t); PyAPI_FUNC(void) _PyTuple_MaybeUntrack(PyObject *); /* Cast argument to PyTupleObject* type. */ -#define _PyTuple_CAST(op) (assert(PyTuple_Check(op)), (PyTupleObject *)(op)) +#define _PyTuple_CAST(op) \ + (assert(PyTuple_Check(op)), _Py_reinterpret_cast(PyTupleObject*, (op))) // Macros and static inline functions, trading safety for speed diff --git a/Include/cpython/unicodeobject.h b/Include/cpython/unicodeobject.h index 2712c583270c6..0397f12a8260b 100644 --- a/Include/cpython/unicodeobject.h +++ b/Include/cpython/unicodeobject.h @@ -236,11 +236,14 @@ PyAPI_FUNC(int) _PyUnicode_CheckConsistency( #define _PyASCIIObject_CAST(op) \ - (assert(PyUnicode_Check(op)), (PyASCIIObject*)(op)) + (assert(PyUnicode_Check(op)), \ + _Py_reinterpret_cast(PyASCIIObject*, (op))) #define _PyCompactUnicodeObject_CAST(op) \ - (assert(PyUnicode_Check(op)), (PyCompactUnicodeObject*)(op)) + (assert(PyUnicode_Check(op)), \ + _Py_reinterpret_cast(PyCompactUnicodeObject*, (op))) #define _PyUnicodeObject_CAST(op) \ - (assert(PyUnicode_Check(op)), (PyUnicodeObject*)(op)) + (assert(PyUnicode_Check(op)), \ + _Py_reinterpret_cast(PyUnicodeObject*, (op))) /* --- Flexible String Representation Helper Macros (PEP 393) -------------- */ diff --git a/Include/methodobject.h b/Include/methodobject.h index 959e77512200c..2046ab948de05 100644 --- a/Include/methodobject.h +++ b/Include/methodobject.h @@ -42,7 +42,9 @@ typedef PyObject *(*PyCMethod)(PyObject *, PyTypeObject *, PyObject *const *, // used to prevent a compiler warning. If the function has a single parameter, // it triggers an undefined behavior when Python calls it with 2 parameters // (bpo-33012). -#define _PyCFunction_CAST(func) ((PyCFunction)(void(*)(void))(func)) +#define _PyCFunction_CAST(func) \ + _Py_reinterpret_cast(PyCFunction, \ + _Py_reinterpret_cast(void(*)(void), (func))) PyAPI_FUNC(PyCFunction) PyCFunction_GetFunction(PyObject *); PyAPI_FUNC(PyObject *) PyCFunction_GetSelf(PyObject *); diff --git a/Include/object.h b/Include/object.h index 5622305f49f78..c51f12b65624d 100644 --- a/Include/object.h +++ b/Include/object.h @@ -104,7 +104,7 @@ struct _object { }; /* Cast argument to PyObject* type. */ -#define _PyObject_CAST(op) ((PyObject*)(op)) +#define _PyObject_CAST(op) _Py_reinterpret_cast(PyObject*, (op)) typedef struct { PyObject ob_base; @@ -112,7 +112,7 @@ typedef struct { } PyVarObject; /* Cast argument to PyVarObject* type. */ -#define _PyVarObject_CAST(op) ((PyVarObject*)(op)) +#define _PyVarObject_CAST(op) _Py_reinterpret_cast(PyVarObject*, (op)) // Test if the 'x' object is the 'y' object, the same as "x is y" in Python. @@ -780,7 +780,8 @@ static inline int PyType_Check(PyObject *op) { # define PyType_Check(op) PyType_Check(_PyObject_CAST(op)) #endif -#define _PyType_CAST(op) (assert(PyType_Check(op)), (PyTypeObject*)(op)) +#define _PyType_CAST(op) \ + (assert(PyType_Check(op)), _Py_reinterpret_cast(PyTypeObject*, (op))) static inline int PyType_CheckExact(PyObject *op) { return Py_IS_TYPE(op, &PyType_Type); diff --git a/Include/objimpl.h b/Include/objimpl.h index 9b98c112ac2cc..94e03045f882a 100644 --- a/Include/objimpl.h +++ b/Include/objimpl.h @@ -182,9 +182,9 @@ PyAPI_FUNC(void) PyObject_GC_UnTrack(void *); PyAPI_FUNC(void) PyObject_GC_Del(void *); #define PyObject_GC_New(type, typeobj) \ - ( (type *) _PyObject_GC_New(typeobj) ) + _Py_reinterpret_cast(type*, _PyObject_GC_New(typeobj)) #define PyObject_GC_NewVar(type, typeobj, n) \ - ( (type *) _PyObject_GC_NewVar((typeobj), (n)) ) + _Py_reinterpret_cast(type*, _PyObject_GC_NewVar((typeobj), (n))) PyAPI_FUNC(int) PyObject_GC_IsTracked(PyObject *); PyAPI_FUNC(int) PyObject_GC_IsFinalized(PyObject *); diff --git a/Include/pyport.h b/Include/pyport.h index 855c382a61ee5..5102f7487d21c 100644 --- a/Include/pyport.h +++ b/Include/pyport.h @@ -14,6 +14,16 @@ #endif +// Macro to use C++ static_cast<> and reinterpret_cast<> in the Python C API +#ifdef __cplusplus +# define _Py_static_cast(type, expr) static_cast(expr) +# define _Py_reinterpret_cast(type, expr) reinterpret_cast(expr) +#else +# define _Py_static_cast(type, expr) ((type)(expr)) +# define _Py_reinterpret_cast(type, expr) ((type)(expr)) +#endif + + /* Defines to build Python and its standard library: * * - Py_BUILD_CORE: Build Python core. Give access to Python internals, but @@ -295,10 +305,11 @@ extern "C" { * VALUE may be evaluated more than once. */ #ifdef Py_DEBUG -#define Py_SAFE_DOWNCAST(VALUE, WIDE, NARROW) \ - (assert((WIDE)(NARROW)(VALUE) == (VALUE)), (NARROW)(VALUE)) +# define Py_SAFE_DOWNCAST(VALUE, WIDE, NARROW) \ + (assert((WIDE)(NARROW)(VALUE) == (VALUE)), (NARROW)(VALUE)) #else -#define Py_SAFE_DOWNCAST(VALUE, WIDE, NARROW) (NARROW)(VALUE) +# define Py_SAFE_DOWNCAST(VALUE, WIDE, NARROW) \ + _Py_reinterpret_cast(NARROW, (VALUE)) #endif diff --git a/Misc/NEWS.d/next/C API/2022-04-26-16-51-31.gh-issue-91320.QDHmTv.rst b/Misc/NEWS.d/next/C API/2022-04-26-16-51-31.gh-issue-91320.QDHmTv.rst new file mode 100644 index 0000000000000..07e27acd4f63b --- /dev/null +++ b/Misc/NEWS.d/next/C API/2022-04-26-16-51-31.gh-issue-91320.QDHmTv.rst @@ -0,0 +1,3 @@ +Fix C++ compiler warnings about "old-style cast" (``g++ -Wold-style-cast``) in +the Python C API. Use C++ ``reinterpret_cast<>`` and ``static_cast<>`` casts +when the Python C API is used in C++. Patch by Victor Stinner. From webhook-mailer at python.org Wed Apr 27 05:40:09 2022 From: webhook-mailer at python.org (vstinner) Date: Wed, 27 Apr 2022 09:40:09 -0000 Subject: [Python-checkins] gh-68966: Document mailcap deprecation in Python 3.11 (#91971) Message-ID: https://github.com/python/cpython/commit/a36d97e3f1daeb431e1c5bc8ab83daca93b747b0 commit: a36d97e3f1daeb431e1c5bc8ab83daca93b747b0 branch: 3.10 author: Victor Stinner committer: vstinner date: 2022-04-27T11:39:51+02:00 summary: gh-68966: Document mailcap deprecation in Python 3.11 (#91971) (cherry picked from commit 80de0273c0caf8bae19787bb00255eb3fb2a2d0c) files: M Doc/library/mailcap.rst M Doc/library/netdata.rst M Doc/library/superseded.rst diff --git a/Doc/library/mailcap.rst b/Doc/library/mailcap.rst index 7749b7dd45ef4..416b181f45a77 100644 --- a/Doc/library/mailcap.rst +++ b/Doc/library/mailcap.rst @@ -3,9 +3,14 @@ .. module:: mailcap :synopsis: Mailcap file handling. + :deprecated: **Source code:** :source:`Lib/mailcap.py` +.. deprecated:: 3.11 + The :mod:`mailcap` module is deprecated. See :pep:`594` for the rationale + and the :mod:`mimetypes` module for an alternative. + -------------- Mailcap files are used to configure how MIME-aware applications such as mail diff --git a/Doc/library/netdata.rst b/Doc/library/netdata.rst index 0e6ebd3b02ddd..1cf717332731c 100644 --- a/Doc/library/netdata.rst +++ b/Doc/library/netdata.rst @@ -13,7 +13,6 @@ on the internet. email.rst json.rst - mailcap.rst mailbox.rst mimetypes.rst base64.rst diff --git a/Doc/library/superseded.rst b/Doc/library/superseded.rst index e3f9b0d37fe10..b38f16691f6ea 100644 --- a/Doc/library/superseded.rst +++ b/Doc/library/superseded.rst @@ -20,9 +20,10 @@ backwards compatibility. They have been superseded by other modules. crypt.rst imghdr.rst imp.rst + mailcap.rst msilib.rst - nntplib.rst nis.rst + nntplib.rst optparse.rst ossaudiodev.rst pipes.rst From webhook-mailer at python.org Wed Apr 27 05:59:53 2022 From: webhook-mailer at python.org (miss-islington) Date: Wed, 27 Apr 2022 09:59:53 -0000 Subject: [Python-checkins] gh-68966: Document mailcap deprecation in Python 3.11 (GH-91971) Message-ID: https://github.com/python/cpython/commit/24ce12366f95d8fa8a3dc609daa59579819481bd commit: 24ce12366f95d8fa8a3dc609daa59579819481bd branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-27T02:59:32-07:00 summary: gh-68966: Document mailcap deprecation in Python 3.11 (GH-91971) (cherry picked from commit 80de0273c0caf8bae19787bb00255eb3fb2a2d0c) (cherry picked from commit a36d97e3f1daeb431e1c5bc8ab83daca93b747b0) Co-authored-by: Victor Stinner files: M Doc/library/mailcap.rst M Doc/library/netdata.rst M Doc/library/superseded.rst diff --git a/Doc/library/mailcap.rst b/Doc/library/mailcap.rst index bf9639bdaca50..cd5ab0b32f49c 100644 --- a/Doc/library/mailcap.rst +++ b/Doc/library/mailcap.rst @@ -3,9 +3,14 @@ .. module:: mailcap :synopsis: Mailcap file handling. + :deprecated: **Source code:** :source:`Lib/mailcap.py` +.. deprecated:: 3.11 + The :mod:`mailcap` module is deprecated. See :pep:`594` for the rationale + and the :mod:`mimetypes` module for an alternative. + -------------- Mailcap files are used to configure how MIME-aware applications such as mail diff --git a/Doc/library/netdata.rst b/Doc/library/netdata.rst index 8a3ad68753a53..2520036c2fa51 100644 --- a/Doc/library/netdata.rst +++ b/Doc/library/netdata.rst @@ -13,7 +13,6 @@ on the Internet. email.rst json.rst - mailcap.rst mailbox.rst mimetypes.rst base64.rst diff --git a/Doc/library/superseded.rst b/Doc/library/superseded.rst index e3f9b0d37fe10..b38f16691f6ea 100644 --- a/Doc/library/superseded.rst +++ b/Doc/library/superseded.rst @@ -20,9 +20,10 @@ backwards compatibility. They have been superseded by other modules. crypt.rst imghdr.rst imp.rst + mailcap.rst msilib.rst - nntplib.rst nis.rst + nntplib.rst optparse.rst ossaudiodev.rst pipes.rst From webhook-mailer at python.org Wed Apr 27 06:46:31 2022 From: webhook-mailer at python.org (vstinner) Date: Wed, 27 Apr 2022 10:46:31 -0000 Subject: [Python-checkins] gh-91719: Mark pycore_opcode.h as generated in .gitattributes (#91976) Message-ID: https://github.com/python/cpython/commit/b733708ca32afb5742d921f0b2cad39f4741af50 commit: b733708ca32afb5742d921f0b2cad39f4741af50 branch: main author: Dennis Sweeney <36520290+sweeneyde at users.noreply.github.com> committer: vstinner date: 2022-04-27T12:45:40+02:00 summary: gh-91719: Mark pycore_opcode.h as generated in .gitattributes (#91976) files: M .gitattributes diff --git a/.gitattributes b/.gitattributes index d956a45753c80..6da587a4c6ea5 100644 --- a/.gitattributes +++ b/.gitattributes @@ -68,6 +68,7 @@ Doc/data/stable_abi.dat generated Doc/library/token-list.inc generated Include/internal/pycore_ast.h generated Include/internal/pycore_ast_state.h generated +Include/internal/pycore_opcode.h generated Include/opcode.h generated Include/token.h generated Lib/keyword.py generated From webhook-mailer at python.org Wed Apr 27 08:47:40 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Wed, 27 Apr 2022 12:47:40 -0000 Subject: [Python-checkins] gh-91498: socket: Add TCP_CONNECTION_INFO on macOS (#69256) Message-ID: https://github.com/python/cpython/commit/ad9f817eeb2d2d36834e7bad2264ad0c0de1d1c4 commit: ad9f817eeb2d2d36834e7bad2264ad0c0de1d1c4 branch: main author: David CARLIER committer: JelleZijlstra date: 2022-04-27T06:47:17-06:00 summary: gh-91498: socket: Add TCP_CONNECTION_INFO on macOS (#69256) Fixes GH-91498 files: A Misc/NEWS.d/next/Library/2022-04-10-08-39-44.bpo-91498.8oII92.rst M Doc/library/socket.rst M Modules/socketmodule.c diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst index cff5a32afb464..3b1912cff88a4 100755 --- a/Doc/library/socket.rst +++ b/Doc/library/socket.rst @@ -388,6 +388,10 @@ Constants Added ``TCP_KEEPALIVE``. On MacOS this constant can be used in the same way that ``TCP_KEEPIDLE`` is used on Linux. + .. versionchanged:: 3.11 + Added ``TCP_CONNECTION_INFO``. On MacOS this constant can be used in the + same way that ``TCP_INFO`` is used on Linux and BSD. + .. data:: AF_CAN PF_CAN SOL_CAN_* diff --git a/Misc/NEWS.d/next/Library/2022-04-10-08-39-44.bpo-91498.8oII92.rst b/Misc/NEWS.d/next/Library/2022-04-10-08-39-44.bpo-91498.8oII92.rst new file mode 100644 index 0000000000000..df3b81fd11683 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-10-08-39-44.bpo-91498.8oII92.rst @@ -0,0 +1 @@ +Add the ``TCP_CONNECTION_INFO`` option (available on macOS) to :mod:`socket`. diff --git a/Modules/socketmodule.c b/Modules/socketmodule.c index c7bc10b5dbbbc..9ecabaf973074 100644 --- a/Modules/socketmodule.c +++ b/Modules/socketmodule.c @@ -8213,6 +8213,9 @@ PyInit__socket(void) #ifdef TCP_INFO PyModule_AddIntMacro(m, TCP_INFO); #endif +#ifdef TCP_CONNECTION_INFO + PyModule_AddIntMacro(m, TCP_CONNECTION_INFO); +#endif #ifdef TCP_QUICKACK PyModule_AddIntMacro(m, TCP_QUICKACK); #endif From webhook-mailer at python.org Wed Apr 27 12:16:24 2022 From: webhook-mailer at python.org (serhiy-storchaka) Date: Wed, 27 Apr 2022 16:16:24 -0000 Subject: [Python-checkins] gh-91810: Expand ElementTree.write() tests to use non-ASCII data (GH-91989) Message-ID: https://github.com/python/cpython/commit/f60b4c3d74f241775f80affe60dcba6448634fe3 commit: f60b4c3d74f241775f80affe60dcba6448634fe3 branch: main author: Serhiy Storchaka committer: serhiy-storchaka date: 2022-04-27T19:16:20+03:00 summary: gh-91810: Expand ElementTree.write() tests to use non-ASCII data (GH-91989) files: M Lib/test/test_xml_etree.py diff --git a/Lib/test/test_xml_etree.py b/Lib/test/test_xml_etree.py index 60a41506d8795..db25eaba1278f 100644 --- a/Lib/test/test_xml_etree.py +++ b/Lib/test/test_xml_etree.py @@ -130,6 +130,9 @@ def newtest(*args, **kwargs): return newtest return decorator +def convlinesep(data): + return data.replace(b'\n', os.linesep.encode()) + class ModuleTest(unittest.TestCase): def test_sanity(self): @@ -3713,32 +3716,92 @@ def test_encoding(self): def test_write_to_filename(self): self.addCleanup(os_helper.unlink, TESTFN) - tree = ET.ElementTree(ET.XML('''''')) + tree = ET.ElementTree(ET.XML('''\xf8''')) tree.write(TESTFN) with open(TESTFN, 'rb') as f: - self.assertEqual(f.read(), b'''''') + self.assertEqual(f.read(), b'''ø''') + + def test_write_to_filename_with_encoding(self): + self.addCleanup(os_helper.unlink, TESTFN) + tree = ET.ElementTree(ET.XML('''\xf8''')) + tree.write(TESTFN, encoding='utf-8') + with open(TESTFN, 'rb') as f: + self.assertEqual(f.read(), b'''\xc3\xb8''') + + tree.write(TESTFN, encoding='ISO-8859-1') + with open(TESTFN, 'rb') as f: + self.assertEqual(f.read(), convlinesep( + b'''\n''' + b'''\xf8''')) + + def test_write_to_filename_as_unicode(self): + self.addCleanup(os_helper.unlink, TESTFN) + with open(TESTFN, 'w') as f: + encoding = f.encoding + os_helper.unlink(TESTFN) + + try: + '\xf8'.encode(encoding) + except UnicodeEncodeError: + self.skipTest(f'default file encoding {encoding} not supported') + + tree = ET.ElementTree(ET.XML('''\xf8''')) + tree.write(TESTFN, encoding='unicode') + with open(TESTFN, 'rb') as f: + data = f.read() + expected = "\xf8".encode(encoding, 'xmlcharrefreplace') + self.assertEqual(data, expected) def test_write_to_text_file(self): self.addCleanup(os_helper.unlink, TESTFN) - tree = ET.ElementTree(ET.XML('''''')) + tree = ET.ElementTree(ET.XML('''\xf8''')) with open(TESTFN, 'w', encoding='utf-8') as f: tree.write(f, encoding='unicode') self.assertFalse(f.closed) with open(TESTFN, 'rb') as f: - self.assertEqual(f.read(), b'''''') + self.assertEqual(f.read(), b'''\xc3\xb8''') + + with open(TESTFN, 'w', encoding='ascii', errors='xmlcharrefreplace') as f: + tree.write(f, encoding='unicode') + self.assertFalse(f.closed) + with open(TESTFN, 'rb') as f: + self.assertEqual(f.read(), b'''ø''') + + with open(TESTFN, 'w', encoding='ISO-8859-1') as f: + tree.write(f, encoding='unicode') + self.assertFalse(f.closed) + with open(TESTFN, 'rb') as f: + self.assertEqual(f.read(), b'''\xf8''') def test_write_to_binary_file(self): self.addCleanup(os_helper.unlink, TESTFN) - tree = ET.ElementTree(ET.XML('''''')) + tree = ET.ElementTree(ET.XML('''\xf8''')) with open(TESTFN, 'wb') as f: tree.write(f) self.assertFalse(f.closed) with open(TESTFN, 'rb') as f: - self.assertEqual(f.read(), b'''''') + self.assertEqual(f.read(), b'''ø''') + + def test_write_to_binary_file_with_encoding(self): + self.addCleanup(os_helper.unlink, TESTFN) + tree = ET.ElementTree(ET.XML('''\xf8''')) + with open(TESTFN, 'wb') as f: + tree.write(f, encoding='utf-8') + self.assertFalse(f.closed) + with open(TESTFN, 'rb') as f: + self.assertEqual(f.read(), b'''\xc3\xb8''') + + with open(TESTFN, 'wb') as f: + tree.write(f, encoding='ISO-8859-1') + self.assertFalse(f.closed) + with open(TESTFN, 'rb') as f: + self.assertEqual(f.read(), + b'''\n''' + b'''\xf8''') def test_write_to_binary_file_with_bom(self): self.addCleanup(os_helper.unlink, TESTFN) - tree = ET.ElementTree(ET.XML('''''')) + tree = ET.ElementTree(ET.XML('''\xf8''')) # test BOM writing to buffered file with open(TESTFN, 'wb') as f: tree.write(f, encoding='utf-16') @@ -3746,7 +3809,7 @@ def test_write_to_binary_file_with_bom(self): with open(TESTFN, 'rb') as f: self.assertEqual(f.read(), '''\n''' - ''''''.encode("utf-16")) + '''\xf8'''.encode("utf-16")) # test BOM writing to non-buffered file with open(TESTFN, 'wb', buffering=0) as f: tree.write(f, encoding='utf-16') @@ -3754,7 +3817,7 @@ def test_write_to_binary_file_with_bom(self): with open(TESTFN, 'rb') as f: self.assertEqual(f.read(), '''\n''' - ''''''.encode("utf-16")) + '''\xf8'''.encode("utf-16")) def test_read_from_stringio(self): tree = ET.ElementTree() @@ -3763,10 +3826,10 @@ def test_read_from_stringio(self): self.assertEqual(tree.getroot().tag, 'site') def test_write_to_stringio(self): - tree = ET.ElementTree(ET.XML('''''')) + tree = ET.ElementTree(ET.XML('''\xf8''')) stream = io.StringIO() tree.write(stream, encoding='unicode') - self.assertEqual(stream.getvalue(), '''''') + self.assertEqual(stream.getvalue(), '''\xf8''') def test_read_from_bytesio(self): tree = ET.ElementTree() @@ -3775,10 +3838,10 @@ def test_read_from_bytesio(self): self.assertEqual(tree.getroot().tag, 'site') def test_write_to_bytesio(self): - tree = ET.ElementTree(ET.XML('''''')) + tree = ET.ElementTree(ET.XML('''\xf8''')) raw = io.BytesIO() tree.write(raw) - self.assertEqual(raw.getvalue(), b'''''') + self.assertEqual(raw.getvalue(), b'''ø''') class dummy: pass @@ -3792,12 +3855,12 @@ def test_read_from_user_text_reader(self): self.assertEqual(tree.getroot().tag, 'site') def test_write_to_user_text_writer(self): - tree = ET.ElementTree(ET.XML('''''')) + tree = ET.ElementTree(ET.XML('''\xf8''')) stream = io.StringIO() writer = self.dummy() writer.write = stream.write tree.write(writer, encoding='unicode') - self.assertEqual(stream.getvalue(), '''''') + self.assertEqual(stream.getvalue(), '''\xf8''') def test_read_from_user_binary_reader(self): raw = io.BytesIO(b'''''') @@ -3809,12 +3872,12 @@ def test_read_from_user_binary_reader(self): tree = ET.ElementTree() def test_write_to_user_binary_writer(self): - tree = ET.ElementTree(ET.XML('''''')) + tree = ET.ElementTree(ET.XML('''\xf8''')) raw = io.BytesIO() writer = self.dummy() writer.write = raw.write tree.write(writer) - self.assertEqual(raw.getvalue(), b'''''') + self.assertEqual(raw.getvalue(), b'''ø''') def test_write_to_user_binary_writer_with_bom(self): tree = ET.ElementTree(ET.XML('''''')) From webhook-mailer at python.org Wed Apr 27 12:50:36 2022 From: webhook-mailer at python.org (miss-islington) Date: Wed, 27 Apr 2022 16:50:36 -0000 Subject: [Python-checkins] gh-91810: Expand ElementTree.write() tests to use non-ASCII data (GH-91989) Message-ID: https://github.com/python/cpython/commit/777d478d12573e126b7c0b29be493aa1048cdb7e commit: 777d478d12573e126b7c0b29be493aa1048cdb7e branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-27T09:50:25-07:00 summary: gh-91810: Expand ElementTree.write() tests to use non-ASCII data (GH-91989) (cherry picked from commit f60b4c3d74f241775f80affe60dcba6448634fe3) Co-authored-by: Serhiy Storchaka files: M Lib/test/test_xml_etree.py diff --git a/Lib/test/test_xml_etree.py b/Lib/test/test_xml_etree.py index 5a34d848b3944..cb6d1697f1da8 100644 --- a/Lib/test/test_xml_etree.py +++ b/Lib/test/test_xml_etree.py @@ -130,6 +130,9 @@ def newtest(*args, **kwargs): return newtest return decorator +def convlinesep(data): + return data.replace(b'\n', os.linesep.encode()) + class ModuleTest(unittest.TestCase): def test_sanity(self): @@ -3713,32 +3716,92 @@ def test_encoding(self): def test_write_to_filename(self): self.addCleanup(os_helper.unlink, TESTFN) - tree = ET.ElementTree(ET.XML('''''')) + tree = ET.ElementTree(ET.XML('''\xf8''')) tree.write(TESTFN) with open(TESTFN, 'rb') as f: - self.assertEqual(f.read(), b'''''') + self.assertEqual(f.read(), b'''ø''') + + def test_write_to_filename_with_encoding(self): + self.addCleanup(os_helper.unlink, TESTFN) + tree = ET.ElementTree(ET.XML('''\xf8''')) + tree.write(TESTFN, encoding='utf-8') + with open(TESTFN, 'rb') as f: + self.assertEqual(f.read(), b'''\xc3\xb8''') + + tree.write(TESTFN, encoding='ISO-8859-1') + with open(TESTFN, 'rb') as f: + self.assertEqual(f.read(), convlinesep( + b'''\n''' + b'''\xf8''')) + + def test_write_to_filename_as_unicode(self): + self.addCleanup(os_helper.unlink, TESTFN) + with open(TESTFN, 'w') as f: + encoding = f.encoding + os_helper.unlink(TESTFN) + + try: + '\xf8'.encode(encoding) + except UnicodeEncodeError: + self.skipTest(f'default file encoding {encoding} not supported') + + tree = ET.ElementTree(ET.XML('''\xf8''')) + tree.write(TESTFN, encoding='unicode') + with open(TESTFN, 'rb') as f: + data = f.read() + expected = "\xf8".encode(encoding, 'xmlcharrefreplace') + self.assertEqual(data, expected) def test_write_to_text_file(self): self.addCleanup(os_helper.unlink, TESTFN) - tree = ET.ElementTree(ET.XML('''''')) + tree = ET.ElementTree(ET.XML('''\xf8''')) with open(TESTFN, 'w', encoding='utf-8') as f: tree.write(f, encoding='unicode') self.assertFalse(f.closed) with open(TESTFN, 'rb') as f: - self.assertEqual(f.read(), b'''''') + self.assertEqual(f.read(), b'''\xc3\xb8''') + + with open(TESTFN, 'w', encoding='ascii', errors='xmlcharrefreplace') as f: + tree.write(f, encoding='unicode') + self.assertFalse(f.closed) + with open(TESTFN, 'rb') as f: + self.assertEqual(f.read(), b'''ø''') + + with open(TESTFN, 'w', encoding='ISO-8859-1') as f: + tree.write(f, encoding='unicode') + self.assertFalse(f.closed) + with open(TESTFN, 'rb') as f: + self.assertEqual(f.read(), b'''\xf8''') def test_write_to_binary_file(self): self.addCleanup(os_helper.unlink, TESTFN) - tree = ET.ElementTree(ET.XML('''''')) + tree = ET.ElementTree(ET.XML('''\xf8''')) with open(TESTFN, 'wb') as f: tree.write(f) self.assertFalse(f.closed) with open(TESTFN, 'rb') as f: - self.assertEqual(f.read(), b'''''') + self.assertEqual(f.read(), b'''ø''') + + def test_write_to_binary_file_with_encoding(self): + self.addCleanup(os_helper.unlink, TESTFN) + tree = ET.ElementTree(ET.XML('''\xf8''')) + with open(TESTFN, 'wb') as f: + tree.write(f, encoding='utf-8') + self.assertFalse(f.closed) + with open(TESTFN, 'rb') as f: + self.assertEqual(f.read(), b'''\xc3\xb8''') + + with open(TESTFN, 'wb') as f: + tree.write(f, encoding='ISO-8859-1') + self.assertFalse(f.closed) + with open(TESTFN, 'rb') as f: + self.assertEqual(f.read(), + b'''\n''' + b'''\xf8''') def test_write_to_binary_file_with_bom(self): self.addCleanup(os_helper.unlink, TESTFN) - tree = ET.ElementTree(ET.XML('''''')) + tree = ET.ElementTree(ET.XML('''\xf8''')) # test BOM writing to buffered file with open(TESTFN, 'wb') as f: tree.write(f, encoding='utf-16') @@ -3746,7 +3809,7 @@ def test_write_to_binary_file_with_bom(self): with open(TESTFN, 'rb') as f: self.assertEqual(f.read(), '''\n''' - ''''''.encode("utf-16")) + '''\xf8'''.encode("utf-16")) # test BOM writing to non-buffered file with open(TESTFN, 'wb', buffering=0) as f: tree.write(f, encoding='utf-16') @@ -3754,7 +3817,7 @@ def test_write_to_binary_file_with_bom(self): with open(TESTFN, 'rb') as f: self.assertEqual(f.read(), '''\n''' - ''''''.encode("utf-16")) + '''\xf8'''.encode("utf-16")) def test_read_from_stringio(self): tree = ET.ElementTree() @@ -3763,10 +3826,10 @@ def test_read_from_stringio(self): self.assertEqual(tree.getroot().tag, 'site') def test_write_to_stringio(self): - tree = ET.ElementTree(ET.XML('''''')) + tree = ET.ElementTree(ET.XML('''\xf8''')) stream = io.StringIO() tree.write(stream, encoding='unicode') - self.assertEqual(stream.getvalue(), '''''') + self.assertEqual(stream.getvalue(), '''\xf8''') def test_read_from_bytesio(self): tree = ET.ElementTree() @@ -3775,10 +3838,10 @@ def test_read_from_bytesio(self): self.assertEqual(tree.getroot().tag, 'site') def test_write_to_bytesio(self): - tree = ET.ElementTree(ET.XML('''''')) + tree = ET.ElementTree(ET.XML('''\xf8''')) raw = io.BytesIO() tree.write(raw) - self.assertEqual(raw.getvalue(), b'''''') + self.assertEqual(raw.getvalue(), b'''ø''') class dummy: pass @@ -3792,12 +3855,12 @@ def test_read_from_user_text_reader(self): self.assertEqual(tree.getroot().tag, 'site') def test_write_to_user_text_writer(self): - tree = ET.ElementTree(ET.XML('''''')) + tree = ET.ElementTree(ET.XML('''\xf8''')) stream = io.StringIO() writer = self.dummy() writer.write = stream.write tree.write(writer, encoding='unicode') - self.assertEqual(stream.getvalue(), '''''') + self.assertEqual(stream.getvalue(), '''\xf8''') def test_read_from_user_binary_reader(self): raw = io.BytesIO(b'''''') @@ -3809,12 +3872,12 @@ def test_read_from_user_binary_reader(self): tree = ET.ElementTree() def test_write_to_user_binary_writer(self): - tree = ET.ElementTree(ET.XML('''''')) + tree = ET.ElementTree(ET.XML('''\xf8''')) raw = io.BytesIO() writer = self.dummy() writer.write = raw.write tree.write(writer) - self.assertEqual(raw.getvalue(), b'''''') + self.assertEqual(raw.getvalue(), b'''ø''') def test_write_to_user_binary_writer_with_bom(self): tree = ET.ElementTree(ET.XML('''''')) From webhook-mailer at python.org Wed Apr 27 13:14:09 2022 From: webhook-mailer at python.org (serhiy-storchaka) Date: Wed, 27 Apr 2022 17:14:09 -0000 Subject: [Python-checkins] [3.9] gh-91810: Expand ElementTree.write() tests to use non-ASCII data (GH-91989). (GH-91994) Message-ID: https://github.com/python/cpython/commit/993bb1632bafa62971cbb9bc3d836351a4046d87 commit: 993bb1632bafa62971cbb9bc3d836351a4046d87 branch: 3.9 author: Serhiy Storchaka committer: serhiy-storchaka date: 2022-04-27T20:14:05+03:00 summary: [3.9] gh-91810: Expand ElementTree.write() tests to use non-ASCII data (GH-91989). (GH-91994) (cherry picked from commit f60b4c3d74f241775f80affe60dcba6448634fe3) files: M Lib/test/test_xml_etree.py diff --git a/Lib/test/test_xml_etree.py b/Lib/test/test_xml_etree.py index 0f45fc71ce773..22f14a2b4d633 100644 --- a/Lib/test/test_xml_etree.py +++ b/Lib/test/test_xml_etree.py @@ -112,6 +112,9 @@ def newtest(*args, **kwargs): return newtest return decorator +def convlinesep(data): + return data.replace(b'\n', os.linesep.encode()) + class ModuleTest(unittest.TestCase): def test_sanity(self): @@ -3654,32 +3657,92 @@ def test_encoding(self): def test_write_to_filename(self): self.addCleanup(support.unlink, TESTFN) - tree = ET.ElementTree(ET.XML('''''')) + tree = ET.ElementTree(ET.XML('''\xf8''')) tree.write(TESTFN) with open(TESTFN, 'rb') as f: - self.assertEqual(f.read(), b'''''') + self.assertEqual(f.read(), b'''ø''') + + def test_write_to_filename_with_encoding(self): + self.addCleanup(support.unlink, TESTFN) + tree = ET.ElementTree(ET.XML('''\xf8''')) + tree.write(TESTFN, encoding='utf-8') + with open(TESTFN, 'rb') as f: + self.assertEqual(f.read(), b'''\xc3\xb8''') + + tree.write(TESTFN, encoding='ISO-8859-1') + with open(TESTFN, 'rb') as f: + self.assertEqual(f.read(), convlinesep( + b'''\n''' + b'''\xf8''')) + + def test_write_to_filename_as_unicode(self): + self.addCleanup(support.unlink, TESTFN) + with open(TESTFN, 'w') as f: + encoding = f.encoding + support.unlink(TESTFN) + + try: + '\xf8'.encode(encoding) + except UnicodeEncodeError: + self.skipTest(f'default file encoding {encoding} not supported') + + tree = ET.ElementTree(ET.XML('''\xf8''')) + tree.write(TESTFN, encoding='unicode') + with open(TESTFN, 'rb') as f: + data = f.read() + expected = "\xf8".encode(encoding, 'xmlcharrefreplace') + self.assertEqual(data, expected) def test_write_to_text_file(self): self.addCleanup(support.unlink, TESTFN) - tree = ET.ElementTree(ET.XML('''''')) + tree = ET.ElementTree(ET.XML('''\xf8''')) with open(TESTFN, 'w', encoding='utf-8') as f: tree.write(f, encoding='unicode') self.assertFalse(f.closed) with open(TESTFN, 'rb') as f: - self.assertEqual(f.read(), b'''''') + self.assertEqual(f.read(), b'''\xc3\xb8''') + + with open(TESTFN, 'w', encoding='ascii', errors='xmlcharrefreplace') as f: + tree.write(f, encoding='unicode') + self.assertFalse(f.closed) + with open(TESTFN, 'rb') as f: + self.assertEqual(f.read(), b'''ø''') + + with open(TESTFN, 'w', encoding='ISO-8859-1') as f: + tree.write(f, encoding='unicode') + self.assertFalse(f.closed) + with open(TESTFN, 'rb') as f: + self.assertEqual(f.read(), b'''\xf8''') def test_write_to_binary_file(self): self.addCleanup(support.unlink, TESTFN) - tree = ET.ElementTree(ET.XML('''''')) + tree = ET.ElementTree(ET.XML('''\xf8''')) with open(TESTFN, 'wb') as f: tree.write(f) self.assertFalse(f.closed) with open(TESTFN, 'rb') as f: - self.assertEqual(f.read(), b'''''') + self.assertEqual(f.read(), b'''ø''') + + def test_write_to_binary_file_with_encoding(self): + self.addCleanup(support.unlink, TESTFN) + tree = ET.ElementTree(ET.XML('''\xf8''')) + with open(TESTFN, 'wb') as f: + tree.write(f, encoding='utf-8') + self.assertFalse(f.closed) + with open(TESTFN, 'rb') as f: + self.assertEqual(f.read(), b'''\xc3\xb8''') + + with open(TESTFN, 'wb') as f: + tree.write(f, encoding='ISO-8859-1') + self.assertFalse(f.closed) + with open(TESTFN, 'rb') as f: + self.assertEqual(f.read(), + b'''\n''' + b'''\xf8''') def test_write_to_binary_file_with_bom(self): self.addCleanup(support.unlink, TESTFN) - tree = ET.ElementTree(ET.XML('''''')) + tree = ET.ElementTree(ET.XML('''\xf8''')) # test BOM writing to buffered file with open(TESTFN, 'wb') as f: tree.write(f, encoding='utf-16') @@ -3687,7 +3750,7 @@ def test_write_to_binary_file_with_bom(self): with open(TESTFN, 'rb') as f: self.assertEqual(f.read(), '''\n''' - ''''''.encode("utf-16")) + '''\xf8'''.encode("utf-16")) # test BOM writing to non-buffered file with open(TESTFN, 'wb', buffering=0) as f: tree.write(f, encoding='utf-16') @@ -3695,7 +3758,7 @@ def test_write_to_binary_file_with_bom(self): with open(TESTFN, 'rb') as f: self.assertEqual(f.read(), '''\n''' - ''''''.encode("utf-16")) + '''\xf8'''.encode("utf-16")) def test_read_from_stringio(self): tree = ET.ElementTree() @@ -3704,10 +3767,10 @@ def test_read_from_stringio(self): self.assertEqual(tree.getroot().tag, 'site') def test_write_to_stringio(self): - tree = ET.ElementTree(ET.XML('''''')) + tree = ET.ElementTree(ET.XML('''\xf8''')) stream = io.StringIO() tree.write(stream, encoding='unicode') - self.assertEqual(stream.getvalue(), '''''') + self.assertEqual(stream.getvalue(), '''\xf8''') def test_read_from_bytesio(self): tree = ET.ElementTree() @@ -3716,10 +3779,10 @@ def test_read_from_bytesio(self): self.assertEqual(tree.getroot().tag, 'site') def test_write_to_bytesio(self): - tree = ET.ElementTree(ET.XML('''''')) + tree = ET.ElementTree(ET.XML('''\xf8''')) raw = io.BytesIO() tree.write(raw) - self.assertEqual(raw.getvalue(), b'''''') + self.assertEqual(raw.getvalue(), b'''ø''') class dummy: pass @@ -3733,12 +3796,12 @@ def test_read_from_user_text_reader(self): self.assertEqual(tree.getroot().tag, 'site') def test_write_to_user_text_writer(self): - tree = ET.ElementTree(ET.XML('''''')) + tree = ET.ElementTree(ET.XML('''\xf8''')) stream = io.StringIO() writer = self.dummy() writer.write = stream.write tree.write(writer, encoding='unicode') - self.assertEqual(stream.getvalue(), '''''') + self.assertEqual(stream.getvalue(), '''\xf8''') def test_read_from_user_binary_reader(self): raw = io.BytesIO(b'''''') @@ -3750,12 +3813,12 @@ def test_read_from_user_binary_reader(self): tree = ET.ElementTree() def test_write_to_user_binary_writer(self): - tree = ET.ElementTree(ET.XML('''''')) + tree = ET.ElementTree(ET.XML('''\xf8''')) raw = io.BytesIO() writer = self.dummy() writer.write = raw.write tree.write(writer) - self.assertEqual(raw.getvalue(), b'''''') + self.assertEqual(raw.getvalue(), b'''ø''') def test_write_to_user_binary_writer_with_bom(self): tree = ET.ElementTree(ET.XML('''''')) From webhook-mailer at python.org Wed Apr 27 13:15:19 2022 From: webhook-mailer at python.org (serhiy-storchaka) Date: Wed, 27 Apr 2022 17:15:19 -0000 Subject: [Python-checkins] gh-87999: Change warning type for numeric literal followed by keyword (GH-91980) Message-ID: https://github.com/python/cpython/commit/43a8bf1ea43127aa0d4d05f9db74827899808266 commit: 43a8bf1ea43127aa0d4d05f9db74827899808266 branch: main author: Serhiy Storchaka committer: serhiy-storchaka date: 2022-04-27T20:15:14+03:00 summary: gh-87999: Change warning type for numeric literal followed by keyword (GH-91980) The warning emitted by the Python parser for a numeric literal immediately followed by keyword has been changed from deprecation warning to syntax warning. files: A Misc/NEWS.d/next/Core and Builtins/2022-04-27-10-36-43.gh-issue-87999.YSPHfO.rst M Lib/test/test_grammar.py M Parser/tokenizer.c diff --git a/Lib/test/test_grammar.py b/Lib/test/test_grammar.py index 2e29cdb39301d..da88519862712 100644 --- a/Lib/test/test_grammar.py +++ b/Lib/test/test_grammar.py @@ -104,6 +104,7 @@ class TokenTests(unittest.TestCase): from test.support import check_syntax_error + from test.support.warnings_helper import check_syntax_warning def test_backslash(self): # Backslash means line continuation: @@ -178,7 +179,7 @@ def test_floats(self): def test_float_exponent_tokenization(self): # See issue 21642. with warnings.catch_warnings(): - warnings.simplefilter('ignore', DeprecationWarning) + warnings.simplefilter('ignore', SyntaxWarning) self.assertEqual(eval("1 if 1else 0"), 1) self.assertEqual(eval("1 if 0else 0"), 0) self.assertRaises(SyntaxError, eval, "0 if 1Else 0") @@ -218,12 +219,13 @@ def check(test, error=False): with self.subTest(expr=test): if error: with warnings.catch_warnings(record=True) as w: - with self.assertRaises(SyntaxError): + with self.assertRaisesRegex(SyntaxError, + r'invalid \w+ literal'): compile(test, "", "eval") self.assertEqual(w, []) else: - with self.assertWarns(DeprecationWarning): - compile(test, "", "eval") + self.check_syntax_warning(test, + errtext=r'invalid \w+ literal') for num in "0xf", "0o7", "0b1", "9", "0", "1.", "1e3", "1j": compile(num, "", "eval") @@ -231,15 +233,22 @@ def check(test, error=False): check(f"{num}or x", error=(num == "0")) check(f"{num}in x") check(f"{num}not in x") - with warnings.catch_warnings(): - warnings.filterwarnings('ignore', '"is" with a literal', - SyntaxWarning) - check(f"{num}is x") check(f"{num}if x else y") check(f"x if {num}else y", error=(num == "0xf")) check(f"[{num}for x in ()]") check(f"{num}spam", error=True) + with warnings.catch_warnings(): + warnings.filterwarnings('ignore', '"is" with a literal', + SyntaxWarning) + with self.assertWarnsRegex(SyntaxWarning, + r'invalid \w+ literal'): + compile(f"{num}is x", "", "eval") + warnings.simplefilter('error', SyntaxWarning) + with self.assertRaisesRegex(SyntaxError, + r'invalid \w+ literal'): + compile(f"{num}is x", "", "eval") + check("[0x1ffor x in ()]") check("[0x1for x in ()]") check("[0xfor x in ()]") diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-04-27-10-36-43.gh-issue-87999.YSPHfO.rst b/Misc/NEWS.d/next/Core and Builtins/2022-04-27-10-36-43.gh-issue-87999.YSPHfO.rst new file mode 100644 index 0000000000000..b0ad9b5a11c12 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2022-04-27-10-36-43.gh-issue-87999.YSPHfO.rst @@ -0,0 +1,3 @@ +The warning emitted by the Python parser for a numeric literal immediately +followed by keyword has been changed from deprecation warning to syntax +warning. diff --git a/Parser/tokenizer.c b/Parser/tokenizer.c index db84e2e92e167..c450aa8e4636c 100644 --- a/Parser/tokenizer.c +++ b/Parser/tokenizer.c @@ -1139,7 +1139,7 @@ indenterror(struct tok_state *tok) } static int -parser_warn(struct tok_state *tok, const char *format, ...) +parser_warn(struct tok_state *tok, PyObject *category, const char *format, ...) { PyObject *errmsg; va_list vargs; @@ -1154,9 +1154,9 @@ parser_warn(struct tok_state *tok, const char *format, ...) goto error; } - if (PyErr_WarnExplicitObject(PyExc_DeprecationWarning, errmsg, tok->filename, + if (PyErr_WarnExplicitObject(category, errmsg, tok->filename, tok->lineno, NULL, NULL) < 0) { - if (PyErr_ExceptionMatches(PyExc_DeprecationWarning)) { + if (PyErr_ExceptionMatches(category)) { /* Replace the DeprecationWarning exception with a SyntaxError to get a more accurate error report */ PyErr_Clear(); @@ -1234,7 +1234,9 @@ verify_end_of_number(struct tok_state *tok, int c, const char *kind) } if (r) { tok_backup(tok, c); - if (parser_warn(tok, "invalid %s literal", kind)) { + if (parser_warn(tok, PyExc_SyntaxWarning, + "invalid %s literal", kind)) + { return 0; } tok_nextc(tok); From webhook-mailer at python.org Wed Apr 27 17:29:16 2022 From: webhook-mailer at python.org (orsenthil) Date: Wed, 27 Apr 2022 21:29:16 -0000 Subject: [Python-checkins] Correct method name typo (#91970) Message-ID: https://github.com/python/cpython/commit/c6b84a727c9299f24edbab4105ce47e9f2bae199 commit: c6b84a727c9299f24edbab4105ce47e9f2bae199 branch: main author: Simon de Vlieger committer: orsenthil date: 2022-04-27T15:28:56-06:00 summary: Correct method name typo (#91970) files: M Lib/difflib.py diff --git a/Lib/difflib.py b/Lib/difflib.py index afd8a0c7c5b61..ba0b256969ebf 100644 --- a/Lib/difflib.py +++ b/Lib/difflib.py @@ -837,7 +837,7 @@ def compare(self, a, b): Each sequence must contain individual single-line strings ending with newlines. Such sequences can be obtained from the `readlines()` method of file-like objects. The delta generated also consists of newline- - terminated strings, ready to be printed as-is via the writeline() + terminated strings, ready to be printed as-is via the writelines() method of a file-like object. Example: From webhook-mailer at python.org Wed Apr 27 18:03:24 2022 From: webhook-mailer at python.org (miss-islington) Date: Wed, 27 Apr 2022 22:03:24 -0000 Subject: [Python-checkins] gh-84459: Make wording more specific for Path.replace (GH-91853) Message-ID: https://github.com/python/cpython/commit/161dff7e10eeb7eaf6d418b91e993aaf84770a5c commit: 161dff7e10eeb7eaf6d418b91e993aaf84770a5c branch: main author: slateny <46876382+slateny at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-27T15:03:03-07:00 summary: gh-84459: Make wording more specific for Path.replace (GH-91853) #84459 files: M Doc/library/os.rst M Doc/library/pathlib.rst diff --git a/Doc/library/os.rst b/Doc/library/os.rst index 471890e74c8e5..d42259b2548f6 100644 --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -2348,7 +2348,7 @@ features: .. function:: replace(src, dst, *, src_dir_fd=None, dst_dir_fd=None) - Rename the file or directory *src* to *dst*. If *dst* is a directory, + Rename the file or directory *src* to *dst*. If *dst* is a non-empty directory, :exc:`OSError` will be raised. If *dst* exists and is a file, it will be replaced silently if the user has permission. The operation may fail if *src* and *dst* are on different filesystems. If successful, diff --git a/Doc/library/pathlib.rst b/Doc/library/pathlib.rst index 1e7bc315471e2..01e9cfb93e391 100644 --- a/Doc/library/pathlib.rst +++ b/Doc/library/pathlib.rst @@ -1042,7 +1042,7 @@ call fails (for example because the path doesn't exist). Rename this file or directory to the given *target*, and return a new Path instance pointing to *target*. If *target* points to an existing file or - directory, it will be unconditionally replaced. + empty directory, it will be unconditionally replaced. The target path may be absolute or relative. Relative paths are interpreted relative to the current working directory, *not* the directory of the Path From webhook-mailer at python.org Wed Apr 27 18:18:12 2022 From: webhook-mailer at python.org (miss-islington) Date: Wed, 27 Apr 2022 22:18:12 -0000 Subject: [Python-checkins] gh-84459: Make wording more specific for Path.replace (GH-91853) Message-ID: https://github.com/python/cpython/commit/86e4bdaf493d3d5e0ded31d170eafd2a15a57f8f commit: 86e4bdaf493d3d5e0ded31d170eafd2a15a57f8f branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-27T15:18:06-07:00 summary: gh-84459: Make wording more specific for Path.replace (GH-91853) GH-84459 (cherry picked from commit 161dff7e10eeb7eaf6d418b91e993aaf84770a5c) Co-authored-by: slateny <46876382+slateny at users.noreply.github.com> files: M Doc/library/os.rst M Doc/library/pathlib.rst diff --git a/Doc/library/os.rst b/Doc/library/os.rst index dbd3c968dd35a..b81574b7eaa46 100644 --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -2338,7 +2338,7 @@ features: .. function:: replace(src, dst, *, src_dir_fd=None, dst_dir_fd=None) - Rename the file or directory *src* to *dst*. If *dst* is a directory, + Rename the file or directory *src* to *dst*. If *dst* is a non-empty directory, :exc:`OSError` will be raised. If *dst* exists and is a file, it will be replaced silently if the user has permission. The operation may fail if *src* and *dst* are on different filesystems. If successful, diff --git a/Doc/library/pathlib.rst b/Doc/library/pathlib.rst index fedea34bcd0c4..f036d8119243a 100644 --- a/Doc/library/pathlib.rst +++ b/Doc/library/pathlib.rst @@ -1042,7 +1042,7 @@ call fails (for example because the path doesn't exist). Rename this file or directory to the given *target*, and return a new Path instance pointing to *target*. If *target* points to an existing file or - directory, it will be unconditionally replaced. + empty directory, it will be unconditionally replaced. The target path may be absolute or relative. Relative paths are interpreted relative to the current working directory, *not* the directory of the Path From webhook-mailer at python.org Wed Apr 27 18:19:54 2022 From: webhook-mailer at python.org (miss-islington) Date: Wed, 27 Apr 2022 22:19:54 -0000 Subject: [Python-checkins] Correct method name typo (GH-91970) Message-ID: https://github.com/python/cpython/commit/e2514cb77fcbdb5c733ae2532802a78e7cb08e79 commit: e2514cb77fcbdb5c733ae2532802a78e7cb08e79 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-27T15:19:43-07:00 summary: Correct method name typo (GH-91970) (cherry picked from commit c6b84a727c9299f24edbab4105ce47e9f2bae199) Co-authored-by: Simon de Vlieger files: M Lib/difflib.py diff --git a/Lib/difflib.py b/Lib/difflib.py index afd8a0c7c5b61..ba0b256969ebf 100644 --- a/Lib/difflib.py +++ b/Lib/difflib.py @@ -837,7 +837,7 @@ def compare(self, a, b): Each sequence must contain individual single-line strings ending with newlines. Such sequences can be obtained from the `readlines()` method of file-like objects. The delta generated also consists of newline- - terminated strings, ready to be printed as-is via the writeline() + terminated strings, ready to be printed as-is via the writelines() method of a file-like object. Example: From webhook-mailer at python.org Wed Apr 27 18:20:57 2022 From: webhook-mailer at python.org (miss-islington) Date: Wed, 27 Apr 2022 22:20:57 -0000 Subject: [Python-checkins] gh-84459: Make wording more specific for Path.replace (GH-91853) Message-ID: https://github.com/python/cpython/commit/35a4eb299a535cf25a8c48a4a90550a45e898305 commit: 35a4eb299a535cf25a8c48a4a90550a45e898305 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-27T15:20:53-07:00 summary: gh-84459: Make wording more specific for Path.replace (GH-91853) GH-84459 (cherry picked from commit 161dff7e10eeb7eaf6d418b91e993aaf84770a5c) Co-authored-by: slateny <46876382+slateny at users.noreply.github.com> files: M Doc/library/os.rst M Doc/library/pathlib.rst diff --git a/Doc/library/os.rst b/Doc/library/os.rst index 8e553d404100f..f8985a1f3ab9d 100644 --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -2207,7 +2207,7 @@ features: .. function:: replace(src, dst, *, src_dir_fd=None, dst_dir_fd=None) - Rename the file or directory *src* to *dst*. If *dst* is a directory, + Rename the file or directory *src* to *dst*. If *dst* is a non-empty directory, :exc:`OSError` will be raised. If *dst* exists and is a file, it will be replaced silently if the user has permission. The operation may fail if *src* and *dst* are on different filesystems. If successful, diff --git a/Doc/library/pathlib.rst b/Doc/library/pathlib.rst index 3a41b5454b864..30611abcb9450 100644 --- a/Doc/library/pathlib.rst +++ b/Doc/library/pathlib.rst @@ -1028,7 +1028,7 @@ call fails (for example because the path doesn't exist). Rename this file or directory to the given *target*, and return a new Path instance pointing to *target*. If *target* points to an existing file or - directory, it will be unconditionally replaced. + empty directory, it will be unconditionally replaced. The target path may be absolute or relative. Relative paths are interpreted relative to the current working directory, *not* the directory of the Path From webhook-mailer at python.org Wed Apr 27 18:25:18 2022 From: webhook-mailer at python.org (orsenthil) Date: Wed, 27 Apr 2022 22:25:18 -0000 Subject: [Python-checkins] Add note that headers added via urllib.request.add_header are added to redirected requests (#30708) Message-ID: https://github.com/python/cpython/commit/f348154c8f8a9c254503306c59d6779d4d09b3a9 commit: f348154c8f8a9c254503306c59d6779d4d09b3a9 branch: main author: Ashwin Ramaswami committer: orsenthil date: 2022-04-27T16:25:03-06:00 summary: Add note that headers added via urllib.request.add_header are added to redirected requests (#30708) files: M Doc/library/urllib.request.rst diff --git a/Doc/library/urllib.request.rst b/Doc/library/urllib.request.rst index a8501ab863968..3823981b37c40 100644 --- a/Doc/library/urllib.request.rst +++ b/Doc/library/urllib.request.rst @@ -547,7 +547,8 @@ request. name, and later calls will overwrite previous calls in case the *key* collides. Currently, this is no loss of HTTP functionality, since all headers which have meaning when used more than once have a (header-specific) way of gaining the - same functionality using only one header. + same functionality using only one header. Note that headers added using + this method are also added to redirected requests. .. method:: Request.add_unredirected_header(key, header) From webhook-mailer at python.org Wed Apr 27 18:26:52 2022 From: webhook-mailer at python.org (orsenthil) Date: Wed, 27 Apr 2022 22:26:52 -0000 Subject: [Python-checkins] Add note that headers added via urllib.request.add_header are added to redirected requests (GH-30708) (#92004) Message-ID: https://github.com/python/cpython/commit/e25799d27d049237849c471b25db3b128b1bfa08 commit: e25799d27d049237849c471b25db3b128b1bfa08 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: orsenthil date: 2022-04-27T16:26:42-06:00 summary: Add note that headers added via urllib.request.add_header are added to redirected requests (GH-30708) (#92004) (cherry picked from commit f348154c8f8a9c254503306c59d6779d4d09b3a9) Co-authored-by: Ashwin Ramaswami Co-authored-by: Ashwin Ramaswami files: M Doc/library/urllib.request.rst diff --git a/Doc/library/urllib.request.rst b/Doc/library/urllib.request.rst index 9573683dd0594..39d1a277aa6c7 100644 --- a/Doc/library/urllib.request.rst +++ b/Doc/library/urllib.request.rst @@ -547,7 +547,8 @@ request. name, and later calls will overwrite previous calls in case the *key* collides. Currently, this is no loss of HTTP functionality, since all headers which have meaning when used more than once have a (header-specific) way of gaining the - same functionality using only one header. + same functionality using only one header. Note that headers added using + this method are also added to redirected requests. .. method:: Request.add_unredirected_header(key, header) From webhook-mailer at python.org Wed Apr 27 18:28:21 2022 From: webhook-mailer at python.org (orsenthil) Date: Wed, 27 Apr 2022 22:28:21 -0000 Subject: [Python-checkins] Add note that headers added via urllib.request.add_header are added to redirected requests (GH-30708) (#92005) Message-ID: https://github.com/python/cpython/commit/0e16105af69ec33e1215e0543f26a33bc8d10a16 commit: 0e16105af69ec33e1215e0543f26a33bc8d10a16 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: orsenthil date: 2022-04-27T16:28:10-06:00 summary: Add note that headers added via urllib.request.add_header are added to redirected requests (GH-30708) (#92005) (cherry picked from commit f348154c8f8a9c254503306c59d6779d4d09b3a9) Co-authored-by: Ashwin Ramaswami Co-authored-by: Ashwin Ramaswami files: M Doc/library/urllib.request.rst diff --git a/Doc/library/urllib.request.rst b/Doc/library/urllib.request.rst index 1307143a04f1d..6f6523d8afe9f 100644 --- a/Doc/library/urllib.request.rst +++ b/Doc/library/urllib.request.rst @@ -542,7 +542,8 @@ request. name, and later calls will overwrite previous calls in case the *key* collides. Currently, this is no loss of HTTP functionality, since all headers which have meaning when used more than once have a (header-specific) way of gaining the - same functionality using only one header. + same functionality using only one header. Note that headers added using + this method are also added to redirected requests. .. method:: Request.add_unredirected_header(key, header) From webhook-mailer at python.org Wed Apr 27 23:26:41 2022 From: webhook-mailer at python.org (miss-islington) Date: Thu, 28 Apr 2022 03:26:41 -0000 Subject: [Python-checkins] gh-91217: deprecate uu (GH-92009) Message-ID: https://github.com/python/cpython/commit/407c3afe1986f4c43cb0e68e28b90da30eebd738 commit: 407c3afe1986f4c43cb0e68e28b90da30eebd738 branch: main author: Brett Cannon committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-27T20:26:33-07:00 summary: gh-91217: deprecate uu (GH-92009) Automerge-Triggered-By: GH:brettcannon files: A Misc/NEWS.d/next/Library/2022-04-21-21-04-08.gh-issue-91217.BZVEki.rst M Doc/whatsnew/3.11.rst M Lib/email/message.py M Lib/test/test_uu.py M Lib/uu.py diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index aa0a51b4375f1..b2b98747d31a1 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -1071,6 +1071,7 @@ Deprecated * :mod:`spwd` * :mod:`sunau` * :mod:`telnetlib` + * :mod:`uu` (Contributed by Brett Cannon in :issue:`47061` and Victor Stinner in :gh:`68966`.) diff --git a/Lib/email/message.py b/Lib/email/message.py index 6752ce0fa1382..65fda507251ce 100644 --- a/Lib/email/message.py +++ b/Lib/email/message.py @@ -6,8 +6,8 @@ __all__ = ['Message', 'EmailMessage'] +import binascii import re -import uu import quopri from io import BytesIO, StringIO @@ -35,7 +35,7 @@ def _splitparam(param): if not sep: return a.strip(), None return a.strip(), b.strip() - + def _formatparam(param, value=None, quote=True): """Convenience function to format and return a key=value pair. @@ -101,7 +101,37 @@ def _unquotevalue(value): return utils.unquote(value) - +def _decode_uu(encoded): + """Decode uuencoded data.""" + decoded_lines = [] + encoded_lines_iter = iter(encoded.splitlines()) + for line in encoded_lines_iter: + if line.startswith(b"begin "): + mode, _, path = line.removeprefix(b"begin ").partition(b" ") + try: + int(mode, base=8) + except ValueError: + continue + else: + break + else: + raise ValueError("`begin` line not found") + for line in encoded_lines_iter: + if not line: + raise ValueError("Truncated input") + elif line.strip(b' \t\r\n\f') == b'end': + break + try: + decoded_line = binascii.a2b_uu(line) + except binascii.Error: + # Workaround for broken uuencoders by /Fredrik Lundh + nbytes = (((line[0]-32) & 63) * 4 + 5) // 3 + decoded_line = binascii.a2b_uu(line[:nbytes]) + decoded_lines.append(decoded_line) + + return b''.join(decoded_lines) + + class Message: """Basic message object. @@ -288,13 +318,10 @@ def get_payload(self, i=None, decode=False): self.policy.handle_defect(self, defect) return value elif cte in ('x-uuencode', 'uuencode', 'uue', 'x-uue'): - in_file = BytesIO(bpayload) - out_file = BytesIO() try: - uu.decode(in_file, out_file, quiet=True) - return out_file.getvalue() - except uu.Error: - # Some decoding problem + return _decode_uu(bpayload) + except ValueError: + # Some decoding problem. return bpayload if isinstance(payload, str): return bpayload diff --git a/Lib/test/test_uu.py b/Lib/test/test_uu.py index 753b31eef0d3b..316a04af1cdaa 100644 --- a/Lib/test/test_uu.py +++ b/Lib/test/test_uu.py @@ -4,12 +4,13 @@ """ import unittest -from test.support import os_helper +from test.support import os_helper, warnings_helper + +uu = warnings_helper.import_deprecated("uu") import os import stat import sys -import uu import io plaintext = b"The symbols on top of your keyboard are !@#$%^&*()_+|~\n" diff --git a/Lib/uu.py b/Lib/uu.py index 9f1f37f1a6410..6f8805d8c5d0c 100755 --- a/Lib/uu.py +++ b/Lib/uu.py @@ -33,6 +33,9 @@ import binascii import os import sys +import warnings + +warnings._deprecated(__name__, remove=(3, 13)) __all__ = ["Error", "encode", "decode"] diff --git a/Misc/NEWS.d/next/Library/2022-04-21-21-04-08.gh-issue-91217.BZVEki.rst b/Misc/NEWS.d/next/Library/2022-04-21-21-04-08.gh-issue-91217.BZVEki.rst new file mode 100644 index 0000000000000..ef5b5c2125584 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-21-21-04-08.gh-issue-91217.BZVEki.rst @@ -0,0 +1 @@ +Deprecate the uu module. From webhook-mailer at python.org Thu Apr 28 00:36:53 2022 From: webhook-mailer at python.org (markshannon) Date: Thu, 28 Apr 2022 04:36:53 -0000 Subject: [Python-checkins] gh-91869: Fix tracing of specialized instructions with extended args (GH-91945) Message-ID: https://github.com/python/cpython/commit/37c6db60f9ac62b8a80bf04a8146274756ee0da0 commit: 37c6db60f9ac62b8a80bf04a8146274756ee0da0 branch: main author: Dennis Sweeney <36520290+sweeneyde at users.noreply.github.com> committer: markshannon date: 2022-04-27T22:36:34-06:00 summary: gh-91869: Fix tracing of specialized instructions with extended args (GH-91945) files: A Misc/NEWS.d/next/Core and Builtins/2022-04-26-05-05-32.gh-issue-91869.ELbTXl.rst M Include/internal/pycore_opcode.h M Include/opcode.h M Lib/importlib/_bootstrap_external.py M Lib/opcode.py M Lib/test/test_sys_settrace.py M Python/ceval.c M Python/compile.c M Python/opcode_targets.h diff --git a/Include/internal/pycore_opcode.h b/Include/internal/pycore_opcode.h index 46c9986b500c8..eadcba1add0ae 100644 --- a/Include/internal/pycore_opcode.h +++ b/Include/internal/pycore_opcode.h @@ -105,6 +105,7 @@ const uint8_t _PyOpcode_Deopt[256] = { [DICT_UPDATE] = DICT_UPDATE, [END_ASYNC_FOR] = END_ASYNC_FOR, [EXTENDED_ARG] = EXTENDED_ARG, + [EXTENDED_ARG_QUICK] = EXTENDED_ARG, [FORMAT_VALUE] = FORMAT_VALUE, [FOR_ITER] = FOR_ITER, [GET_AITER] = GET_AITER, @@ -272,10 +273,11 @@ static const char *const _PyOpcode_OpName[256] = { [MATCH_MAPPING] = "MATCH_MAPPING", [MATCH_SEQUENCE] = "MATCH_SEQUENCE", [MATCH_KEYS] = "MATCH_KEYS", - [JUMP_BACKWARD_QUICK] = "JUMP_BACKWARD_QUICK", + [EXTENDED_ARG_QUICK] = "EXTENDED_ARG_QUICK", [PUSH_EXC_INFO] = "PUSH_EXC_INFO", [CHECK_EXC_MATCH] = "CHECK_EXC_MATCH", [CHECK_EG_MATCH] = "CHECK_EG_MATCH", + [JUMP_BACKWARD_QUICK] = "JUMP_BACKWARD_QUICK", [LOAD_ATTR_ADAPTIVE] = "LOAD_ATTR_ADAPTIVE", [LOAD_ATTR_INSTANCE_VALUE] = "LOAD_ATTR_INSTANCE_VALUE", [LOAD_ATTR_MODULE] = "LOAD_ATTR_MODULE", @@ -286,40 +288,39 @@ static const char *const _PyOpcode_OpName[256] = { [LOAD_FAST__LOAD_FAST] = "LOAD_FAST__LOAD_FAST", [LOAD_GLOBAL_ADAPTIVE] = "LOAD_GLOBAL_ADAPTIVE", [LOAD_GLOBAL_BUILTIN] = "LOAD_GLOBAL_BUILTIN", - [LOAD_GLOBAL_MODULE] = "LOAD_GLOBAL_MODULE", [WITH_EXCEPT_START] = "WITH_EXCEPT_START", [GET_AITER] = "GET_AITER", [GET_ANEXT] = "GET_ANEXT", [BEFORE_ASYNC_WITH] = "BEFORE_ASYNC_WITH", [BEFORE_WITH] = "BEFORE_WITH", [END_ASYNC_FOR] = "END_ASYNC_FOR", + [LOAD_GLOBAL_MODULE] = "LOAD_GLOBAL_MODULE", [LOAD_METHOD_ADAPTIVE] = "LOAD_METHOD_ADAPTIVE", [LOAD_METHOD_CLASS] = "LOAD_METHOD_CLASS", [LOAD_METHOD_MODULE] = "LOAD_METHOD_MODULE", [LOAD_METHOD_NO_DICT] = "LOAD_METHOD_NO_DICT", - [LOAD_METHOD_WITH_DICT] = "LOAD_METHOD_WITH_DICT", [STORE_SUBSCR] = "STORE_SUBSCR", [DELETE_SUBSCR] = "DELETE_SUBSCR", + [LOAD_METHOD_WITH_DICT] = "LOAD_METHOD_WITH_DICT", [LOAD_METHOD_WITH_VALUES] = "LOAD_METHOD_WITH_VALUES", [PRECALL_ADAPTIVE] = "PRECALL_ADAPTIVE", [PRECALL_BOUND_METHOD] = "PRECALL_BOUND_METHOD", [PRECALL_BUILTIN_CLASS] = "PRECALL_BUILTIN_CLASS", [PRECALL_BUILTIN_FAST_WITH_KEYWORDS] = "PRECALL_BUILTIN_FAST_WITH_KEYWORDS", - [PRECALL_METHOD_DESCRIPTOR_FAST_WITH_KEYWORDS] = "PRECALL_METHOD_DESCRIPTOR_FAST_WITH_KEYWORDS", [GET_ITER] = "GET_ITER", [GET_YIELD_FROM_ITER] = "GET_YIELD_FROM_ITER", [PRINT_EXPR] = "PRINT_EXPR", [LOAD_BUILD_CLASS] = "LOAD_BUILD_CLASS", + [PRECALL_METHOD_DESCRIPTOR_FAST_WITH_KEYWORDS] = "PRECALL_METHOD_DESCRIPTOR_FAST_WITH_KEYWORDS", [PRECALL_NO_KW_BUILTIN_FAST] = "PRECALL_NO_KW_BUILTIN_FAST", - [PRECALL_NO_KW_BUILTIN_O] = "PRECALL_NO_KW_BUILTIN_O", [LOAD_ASSERTION_ERROR] = "LOAD_ASSERTION_ERROR", [RETURN_GENERATOR] = "RETURN_GENERATOR", + [PRECALL_NO_KW_BUILTIN_O] = "PRECALL_NO_KW_BUILTIN_O", [PRECALL_NO_KW_ISINSTANCE] = "PRECALL_NO_KW_ISINSTANCE", [PRECALL_NO_KW_LEN] = "PRECALL_NO_KW_LEN", [PRECALL_NO_KW_LIST_APPEND] = "PRECALL_NO_KW_LIST_APPEND", [PRECALL_NO_KW_METHOD_DESCRIPTOR_FAST] = "PRECALL_NO_KW_METHOD_DESCRIPTOR_FAST", [PRECALL_NO_KW_METHOD_DESCRIPTOR_NOARGS] = "PRECALL_NO_KW_METHOD_DESCRIPTOR_NOARGS", - [PRECALL_NO_KW_METHOD_DESCRIPTOR_O] = "PRECALL_NO_KW_METHOD_DESCRIPTOR_O", [LIST_TO_TUPLE] = "LIST_TO_TUPLE", [RETURN_VALUE] = "RETURN_VALUE", [IMPORT_STAR] = "IMPORT_STAR", @@ -351,7 +352,7 @@ static const char *const _PyOpcode_OpName[256] = { [JUMP_FORWARD] = "JUMP_FORWARD", [JUMP_IF_FALSE_OR_POP] = "JUMP_IF_FALSE_OR_POP", [JUMP_IF_TRUE_OR_POP] = "JUMP_IF_TRUE_OR_POP", - [PRECALL_NO_KW_STR_1] = "PRECALL_NO_KW_STR_1", + [PRECALL_NO_KW_METHOD_DESCRIPTOR_O] = "PRECALL_NO_KW_METHOD_DESCRIPTOR_O", [POP_JUMP_FORWARD_IF_FALSE] = "POP_JUMP_FORWARD_IF_FALSE", [POP_JUMP_FORWARD_IF_TRUE] = "POP_JUMP_FORWARD_IF_TRUE", [LOAD_GLOBAL] = "LOAD_GLOBAL", @@ -359,13 +360,13 @@ static const char *const _PyOpcode_OpName[256] = { [CONTAINS_OP] = "CONTAINS_OP", [RERAISE] = "RERAISE", [COPY] = "COPY", - [PRECALL_NO_KW_TUPLE_1] = "PRECALL_NO_KW_TUPLE_1", + [PRECALL_NO_KW_STR_1] = "PRECALL_NO_KW_STR_1", [BINARY_OP] = "BINARY_OP", [SEND] = "SEND", [LOAD_FAST] = "LOAD_FAST", [STORE_FAST] = "STORE_FAST", [DELETE_FAST] = "DELETE_FAST", - [PRECALL_NO_KW_TYPE_1] = "PRECALL_NO_KW_TYPE_1", + [PRECALL_NO_KW_TUPLE_1] = "PRECALL_NO_KW_TUPLE_1", [POP_JUMP_FORWARD_IF_NOT_NONE] = "POP_JUMP_FORWARD_IF_NOT_NONE", [POP_JUMP_FORWARD_IF_NONE] = "POP_JUMP_FORWARD_IF_NONE", [RAISE_VARARGS] = "RAISE_VARARGS", @@ -379,46 +380,46 @@ static const char *const _PyOpcode_OpName[256] = { [STORE_DEREF] = "STORE_DEREF", [DELETE_DEREF] = "DELETE_DEREF", [JUMP_BACKWARD] = "JUMP_BACKWARD", - [PRECALL_PYFUNC] = "PRECALL_PYFUNC", + [PRECALL_NO_KW_TYPE_1] = "PRECALL_NO_KW_TYPE_1", [CALL_FUNCTION_EX] = "CALL_FUNCTION_EX", - [RESUME_QUICK] = "RESUME_QUICK", + [PRECALL_PYFUNC] = "PRECALL_PYFUNC", [EXTENDED_ARG] = "EXTENDED_ARG", [LIST_APPEND] = "LIST_APPEND", [SET_ADD] = "SET_ADD", [MAP_ADD] = "MAP_ADD", [LOAD_CLASSDEREF] = "LOAD_CLASSDEREF", [COPY_FREE_VARS] = "COPY_FREE_VARS", - [STORE_ATTR_ADAPTIVE] = "STORE_ATTR_ADAPTIVE", + [RESUME_QUICK] = "RESUME_QUICK", [RESUME] = "RESUME", [MATCH_CLASS] = "MATCH_CLASS", + [STORE_ATTR_ADAPTIVE] = "STORE_ATTR_ADAPTIVE", [STORE_ATTR_INSTANCE_VALUE] = "STORE_ATTR_INSTANCE_VALUE", - [STORE_ATTR_SLOT] = "STORE_ATTR_SLOT", [FORMAT_VALUE] = "FORMAT_VALUE", [BUILD_CONST_KEY_MAP] = "BUILD_CONST_KEY_MAP", [BUILD_STRING] = "BUILD_STRING", + [STORE_ATTR_SLOT] = "STORE_ATTR_SLOT", [STORE_ATTR_WITH_HINT] = "STORE_ATTR_WITH_HINT", - [STORE_FAST__LOAD_FAST] = "STORE_FAST__LOAD_FAST", [LOAD_METHOD] = "LOAD_METHOD", - [STORE_FAST__STORE_FAST] = "STORE_FAST__STORE_FAST", + [STORE_FAST__LOAD_FAST] = "STORE_FAST__LOAD_FAST", [LIST_EXTEND] = "LIST_EXTEND", [SET_UPDATE] = "SET_UPDATE", [DICT_MERGE] = "DICT_MERGE", [DICT_UPDATE] = "DICT_UPDATE", [PRECALL] = "PRECALL", + [STORE_FAST__STORE_FAST] = "STORE_FAST__STORE_FAST", [STORE_SUBSCR_ADAPTIVE] = "STORE_SUBSCR_ADAPTIVE", [STORE_SUBSCR_DICT] = "STORE_SUBSCR_DICT", [STORE_SUBSCR_LIST_INT] = "STORE_SUBSCR_LIST_INT", - [UNPACK_SEQUENCE_ADAPTIVE] = "UNPACK_SEQUENCE_ADAPTIVE", [CALL] = "CALL", [KW_NAMES] = "KW_NAMES", [POP_JUMP_BACKWARD_IF_NOT_NONE] = "POP_JUMP_BACKWARD_IF_NOT_NONE", [POP_JUMP_BACKWARD_IF_NONE] = "POP_JUMP_BACKWARD_IF_NONE", [POP_JUMP_BACKWARD_IF_FALSE] = "POP_JUMP_BACKWARD_IF_FALSE", [POP_JUMP_BACKWARD_IF_TRUE] = "POP_JUMP_BACKWARD_IF_TRUE", + [UNPACK_SEQUENCE_ADAPTIVE] = "UNPACK_SEQUENCE_ADAPTIVE", [UNPACK_SEQUENCE_LIST] = "UNPACK_SEQUENCE_LIST", [UNPACK_SEQUENCE_TUPLE] = "UNPACK_SEQUENCE_TUPLE", [UNPACK_SEQUENCE_TWO_TUPLE] = "UNPACK_SEQUENCE_TWO_TUPLE", - [180] = "<180>", [181] = "<181>", [182] = "<182>", [183] = "<183>", @@ -498,7 +499,6 @@ static const char *const _PyOpcode_OpName[256] = { #endif #define EXTRA_CASES \ - case 180: \ case 181: \ case 182: \ case 183: \ diff --git a/Include/opcode.h b/Include/opcode.h index 399847d4ba050..084d34b8c73cd 100644 --- a/Include/opcode.h +++ b/Include/opcode.h @@ -140,55 +140,56 @@ extern "C" { #define COMPARE_OP_FLOAT_JUMP 27 #define COMPARE_OP_INT_JUMP 28 #define COMPARE_OP_STR_JUMP 29 -#define JUMP_BACKWARD_QUICK 34 -#define LOAD_ATTR_ADAPTIVE 38 -#define LOAD_ATTR_INSTANCE_VALUE 39 -#define LOAD_ATTR_MODULE 40 -#define LOAD_ATTR_SLOT 41 -#define LOAD_ATTR_WITH_HINT 42 -#define LOAD_CONST__LOAD_FAST 43 -#define LOAD_FAST__LOAD_CONST 44 -#define LOAD_FAST__LOAD_FAST 45 -#define LOAD_GLOBAL_ADAPTIVE 46 -#define LOAD_GLOBAL_BUILTIN 47 -#define LOAD_GLOBAL_MODULE 48 -#define LOAD_METHOD_ADAPTIVE 55 -#define LOAD_METHOD_CLASS 56 -#define LOAD_METHOD_MODULE 57 -#define LOAD_METHOD_NO_DICT 58 -#define LOAD_METHOD_WITH_DICT 59 -#define LOAD_METHOD_WITH_VALUES 62 -#define PRECALL_ADAPTIVE 63 -#define PRECALL_BOUND_METHOD 64 -#define PRECALL_BUILTIN_CLASS 65 -#define PRECALL_BUILTIN_FAST_WITH_KEYWORDS 66 -#define PRECALL_METHOD_DESCRIPTOR_FAST_WITH_KEYWORDS 67 -#define PRECALL_NO_KW_BUILTIN_FAST 72 -#define PRECALL_NO_KW_BUILTIN_O 73 -#define PRECALL_NO_KW_ISINSTANCE 76 -#define PRECALL_NO_KW_LEN 77 -#define PRECALL_NO_KW_LIST_APPEND 78 -#define PRECALL_NO_KW_METHOD_DESCRIPTOR_FAST 79 -#define PRECALL_NO_KW_METHOD_DESCRIPTOR_NOARGS 80 -#define PRECALL_NO_KW_METHOD_DESCRIPTOR_O 81 -#define PRECALL_NO_KW_STR_1 113 -#define PRECALL_NO_KW_TUPLE_1 121 -#define PRECALL_NO_KW_TYPE_1 127 -#define PRECALL_PYFUNC 141 -#define RESUME_QUICK 143 -#define STORE_ATTR_ADAPTIVE 150 -#define STORE_ATTR_INSTANCE_VALUE 153 -#define STORE_ATTR_SLOT 154 -#define STORE_ATTR_WITH_HINT 158 -#define STORE_FAST__LOAD_FAST 159 -#define STORE_FAST__STORE_FAST 161 -#define STORE_SUBSCR_ADAPTIVE 167 -#define STORE_SUBSCR_DICT 168 -#define STORE_SUBSCR_LIST_INT 169 -#define UNPACK_SEQUENCE_ADAPTIVE 170 -#define UNPACK_SEQUENCE_LIST 177 -#define UNPACK_SEQUENCE_TUPLE 178 -#define UNPACK_SEQUENCE_TWO_TUPLE 179 +#define EXTENDED_ARG_QUICK 34 +#define JUMP_BACKWARD_QUICK 38 +#define LOAD_ATTR_ADAPTIVE 39 +#define LOAD_ATTR_INSTANCE_VALUE 40 +#define LOAD_ATTR_MODULE 41 +#define LOAD_ATTR_SLOT 42 +#define LOAD_ATTR_WITH_HINT 43 +#define LOAD_CONST__LOAD_FAST 44 +#define LOAD_FAST__LOAD_CONST 45 +#define LOAD_FAST__LOAD_FAST 46 +#define LOAD_GLOBAL_ADAPTIVE 47 +#define LOAD_GLOBAL_BUILTIN 48 +#define LOAD_GLOBAL_MODULE 55 +#define LOAD_METHOD_ADAPTIVE 56 +#define LOAD_METHOD_CLASS 57 +#define LOAD_METHOD_MODULE 58 +#define LOAD_METHOD_NO_DICT 59 +#define LOAD_METHOD_WITH_DICT 62 +#define LOAD_METHOD_WITH_VALUES 63 +#define PRECALL_ADAPTIVE 64 +#define PRECALL_BOUND_METHOD 65 +#define PRECALL_BUILTIN_CLASS 66 +#define PRECALL_BUILTIN_FAST_WITH_KEYWORDS 67 +#define PRECALL_METHOD_DESCRIPTOR_FAST_WITH_KEYWORDS 72 +#define PRECALL_NO_KW_BUILTIN_FAST 73 +#define PRECALL_NO_KW_BUILTIN_O 76 +#define PRECALL_NO_KW_ISINSTANCE 77 +#define PRECALL_NO_KW_LEN 78 +#define PRECALL_NO_KW_LIST_APPEND 79 +#define PRECALL_NO_KW_METHOD_DESCRIPTOR_FAST 80 +#define PRECALL_NO_KW_METHOD_DESCRIPTOR_NOARGS 81 +#define PRECALL_NO_KW_METHOD_DESCRIPTOR_O 113 +#define PRECALL_NO_KW_STR_1 121 +#define PRECALL_NO_KW_TUPLE_1 127 +#define PRECALL_NO_KW_TYPE_1 141 +#define PRECALL_PYFUNC 143 +#define RESUME_QUICK 150 +#define STORE_ATTR_ADAPTIVE 153 +#define STORE_ATTR_INSTANCE_VALUE 154 +#define STORE_ATTR_SLOT 158 +#define STORE_ATTR_WITH_HINT 159 +#define STORE_FAST__LOAD_FAST 161 +#define STORE_FAST__STORE_FAST 167 +#define STORE_SUBSCR_ADAPTIVE 168 +#define STORE_SUBSCR_DICT 169 +#define STORE_SUBSCR_LIST_INT 170 +#define UNPACK_SEQUENCE_ADAPTIVE 177 +#define UNPACK_SEQUENCE_LIST 178 +#define UNPACK_SEQUENCE_TUPLE 179 +#define UNPACK_SEQUENCE_TWO_TUPLE 180 #define DO_TRACING 255 #define HAS_CONST(op) (false\ diff --git a/Lib/importlib/_bootstrap_external.py b/Lib/importlib/_bootstrap_external.py index 71e1e24b51e22..5f67226adfc52 100644 --- a/Lib/importlib/_bootstrap_external.py +++ b/Lib/importlib/_bootstrap_external.py @@ -416,7 +416,7 @@ def _write_atomic(path, data, mode=0o666): # Whenever MAGIC_NUMBER is changed, the ranges in the magic_values array # in PC/launcher.c must also be updated. -MAGIC_NUMBER = (3493).to_bytes(2, 'little') + b'\r\n' +MAGIC_NUMBER = (3494).to_bytes(2, 'little') + b'\r\n' _RAW_MAGIC_NUMBER = int.from_bytes(MAGIC_NUMBER, 'little') # For import.c diff --git a/Lib/opcode.py b/Lib/opcode.py index 9ee06831c3768..6c3862707c747 100644 --- a/Lib/opcode.py +++ b/Lib/opcode.py @@ -264,6 +264,9 @@ def jabs_op(name, op, entries=0): "COMPARE_OP_INT_JUMP", "COMPARE_OP_STR_JUMP", ], + "EXTENDED_ARG": [ + "EXTENDED_ARG_QUICK", + ], "JUMP_BACKWARD": [ "JUMP_BACKWARD_QUICK", ], diff --git a/Lib/test/test_sys_settrace.py b/Lib/test/test_sys_settrace.py index 85d6bdf7221e3..b1c8f6f80af83 100644 --- a/Lib/test/test_sys_settrace.py +++ b/Lib/test/test_sys_settrace.py @@ -2432,5 +2432,47 @@ def gen(): output.append(5) +class TestExtendedArgs(unittest.TestCase): + + def setUp(self): + self.addCleanup(sys.settrace, sys.gettrace()) + sys.settrace(None) + + def count_traces(self, func): + # warmup + for _ in range(20): + func() + + counts = {"call": 0, "line": 0, "return": 0} + def trace(frame, event, arg): + counts[event] += 1 + return trace + + sys.settrace(trace) + func() + sys.settrace(None) + + return counts + + def test_trace_unpack_long_sequence(self): + ns = {} + code = "def f():\n (" + "y,\n "*300 + ") = range(300)" + exec(code, ns) + counts = self.count_traces(ns["f"]) + self.assertEqual(counts, {'call': 1, 'line': 301, 'return': 1}) + + def test_trace_lots_of_globals(self): + code = """if 1: + def f(): + return ( + {} + ) + """.format("\n+\n".join(f"var{i}\n" for i in range(1000))) + ns = {f"var{i}": i for i in range(1000)} + exec(code, ns) + counts = self.count_traces(ns["f"]) + self.assertEqual(counts, {'call': 1, 'line': 2000, 'return': 1}) + + if __name__ == "__main__": unittest.main() diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-04-26-05-05-32.gh-issue-91869.ELbTXl.rst b/Misc/NEWS.d/next/Core and Builtins/2022-04-26-05-05-32.gh-issue-91869.ELbTXl.rst new file mode 100644 index 0000000000000..05b84be23483b --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2022-04-26-05-05-32.gh-issue-91869.ELbTXl.rst @@ -0,0 +1 @@ +Fix an issue where specialized opcodes with extended arguments could produce incorrect tracing output or lead to assertion failures. diff --git a/Python/ceval.c b/Python/ceval.c index 6e7a2483112d7..1d2c6432d062f 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -5624,6 +5624,15 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int } TARGET(EXTENDED_ARG) { + assert(oparg); + oparg <<= 8; + oparg |= _Py_OPARG(*next_instr); + opcode = _PyOpcode_Deopt[_Py_OPCODE(*next_instr)]; + PRE_DISPATCH_GOTO(); + DISPATCH_GOTO(); + } + + TARGET(EXTENDED_ARG_QUICK) { assert(oparg); oparg <<= 8; oparg |= _Py_OPARG(*next_instr); diff --git a/Python/compile.c b/Python/compile.c index 42b011ce93ef7..10d6307a48406 100644 --- a/Python/compile.c +++ b/Python/compile.c @@ -211,13 +211,13 @@ write_instr(_Py_CODEUNIT *codestr, struct instr *instruction, int ilen) int caches = _PyOpcode_Caches[opcode]; switch (ilen - caches) { case 4: - *codestr++ = _Py_MAKECODEUNIT(EXTENDED_ARG, (oparg >> 24) & 0xFF); + *codestr++ = _Py_MAKECODEUNIT(EXTENDED_ARG_QUICK, (oparg >> 24) & 0xFF); /* fall through */ case 3: - *codestr++ = _Py_MAKECODEUNIT(EXTENDED_ARG, (oparg >> 16) & 0xFF); + *codestr++ = _Py_MAKECODEUNIT(EXTENDED_ARG_QUICK, (oparg >> 16) & 0xFF); /* fall through */ case 2: - *codestr++ = _Py_MAKECODEUNIT(EXTENDED_ARG, (oparg >> 8) & 0xFF); + *codestr++ = _Py_MAKECODEUNIT(EXTENDED_ARG_QUICK, (oparg >> 8) & 0xFF); /* fall through */ case 1: *codestr++ = _Py_MAKECODEUNIT(opcode, oparg & 0xFF); @@ -8254,6 +8254,7 @@ fix_cell_offsets(struct compiler *c, basicblock *entryblock, int *fixedmap) struct instr *inst = &b->b_instr[i]; // This is called before extended args are generated. assert(inst->i_opcode != EXTENDED_ARG); + assert(inst->i_opcode != EXTENDED_ARG_QUICK); int oldoffset = inst->i_oparg; switch(inst->i_opcode) { case MAKE_CELL: diff --git a/Python/opcode_targets.h b/Python/opcode_targets.h index 5268c4f2a2775..d37c132624718 100644 --- a/Python/opcode_targets.h +++ b/Python/opcode_targets.h @@ -33,10 +33,11 @@ static void *opcode_targets[256] = { &&TARGET_MATCH_MAPPING, &&TARGET_MATCH_SEQUENCE, &&TARGET_MATCH_KEYS, - &&TARGET_JUMP_BACKWARD_QUICK, + &&TARGET_EXTENDED_ARG_QUICK, &&TARGET_PUSH_EXC_INFO, &&TARGET_CHECK_EXC_MATCH, &&TARGET_CHECK_EG_MATCH, + &&TARGET_JUMP_BACKWARD_QUICK, &&TARGET_LOAD_ATTR_ADAPTIVE, &&TARGET_LOAD_ATTR_INSTANCE_VALUE, &&TARGET_LOAD_ATTR_MODULE, @@ -47,40 +48,39 @@ static void *opcode_targets[256] = { &&TARGET_LOAD_FAST__LOAD_FAST, &&TARGET_LOAD_GLOBAL_ADAPTIVE, &&TARGET_LOAD_GLOBAL_BUILTIN, - &&TARGET_LOAD_GLOBAL_MODULE, &&TARGET_WITH_EXCEPT_START, &&TARGET_GET_AITER, &&TARGET_GET_ANEXT, &&TARGET_BEFORE_ASYNC_WITH, &&TARGET_BEFORE_WITH, &&TARGET_END_ASYNC_FOR, + &&TARGET_LOAD_GLOBAL_MODULE, &&TARGET_LOAD_METHOD_ADAPTIVE, &&TARGET_LOAD_METHOD_CLASS, &&TARGET_LOAD_METHOD_MODULE, &&TARGET_LOAD_METHOD_NO_DICT, - &&TARGET_LOAD_METHOD_WITH_DICT, &&TARGET_STORE_SUBSCR, &&TARGET_DELETE_SUBSCR, + &&TARGET_LOAD_METHOD_WITH_DICT, &&TARGET_LOAD_METHOD_WITH_VALUES, &&TARGET_PRECALL_ADAPTIVE, &&TARGET_PRECALL_BOUND_METHOD, &&TARGET_PRECALL_BUILTIN_CLASS, &&TARGET_PRECALL_BUILTIN_FAST_WITH_KEYWORDS, - &&TARGET_PRECALL_METHOD_DESCRIPTOR_FAST_WITH_KEYWORDS, &&TARGET_GET_ITER, &&TARGET_GET_YIELD_FROM_ITER, &&TARGET_PRINT_EXPR, &&TARGET_LOAD_BUILD_CLASS, + &&TARGET_PRECALL_METHOD_DESCRIPTOR_FAST_WITH_KEYWORDS, &&TARGET_PRECALL_NO_KW_BUILTIN_FAST, - &&TARGET_PRECALL_NO_KW_BUILTIN_O, &&TARGET_LOAD_ASSERTION_ERROR, &&TARGET_RETURN_GENERATOR, + &&TARGET_PRECALL_NO_KW_BUILTIN_O, &&TARGET_PRECALL_NO_KW_ISINSTANCE, &&TARGET_PRECALL_NO_KW_LEN, &&TARGET_PRECALL_NO_KW_LIST_APPEND, &&TARGET_PRECALL_NO_KW_METHOD_DESCRIPTOR_FAST, &&TARGET_PRECALL_NO_KW_METHOD_DESCRIPTOR_NOARGS, - &&TARGET_PRECALL_NO_KW_METHOD_DESCRIPTOR_O, &&TARGET_LIST_TO_TUPLE, &&TARGET_RETURN_VALUE, &&TARGET_IMPORT_STAR, @@ -112,7 +112,7 @@ static void *opcode_targets[256] = { &&TARGET_JUMP_FORWARD, &&TARGET_JUMP_IF_FALSE_OR_POP, &&TARGET_JUMP_IF_TRUE_OR_POP, - &&TARGET_PRECALL_NO_KW_STR_1, + &&TARGET_PRECALL_NO_KW_METHOD_DESCRIPTOR_O, &&TARGET_POP_JUMP_FORWARD_IF_FALSE, &&TARGET_POP_JUMP_FORWARD_IF_TRUE, &&TARGET_LOAD_GLOBAL, @@ -120,13 +120,13 @@ static void *opcode_targets[256] = { &&TARGET_CONTAINS_OP, &&TARGET_RERAISE, &&TARGET_COPY, - &&TARGET_PRECALL_NO_KW_TUPLE_1, + &&TARGET_PRECALL_NO_KW_STR_1, &&TARGET_BINARY_OP, &&TARGET_SEND, &&TARGET_LOAD_FAST, &&TARGET_STORE_FAST, &&TARGET_DELETE_FAST, - &&TARGET_PRECALL_NO_KW_TYPE_1, + &&TARGET_PRECALL_NO_KW_TUPLE_1, &&TARGET_POP_JUMP_FORWARD_IF_NOT_NONE, &&TARGET_POP_JUMP_FORWARD_IF_NONE, &&TARGET_RAISE_VARARGS, @@ -140,42 +140,43 @@ static void *opcode_targets[256] = { &&TARGET_STORE_DEREF, &&TARGET_DELETE_DEREF, &&TARGET_JUMP_BACKWARD, - &&TARGET_PRECALL_PYFUNC, + &&TARGET_PRECALL_NO_KW_TYPE_1, &&TARGET_CALL_FUNCTION_EX, - &&TARGET_RESUME_QUICK, + &&TARGET_PRECALL_PYFUNC, &&TARGET_EXTENDED_ARG, &&TARGET_LIST_APPEND, &&TARGET_SET_ADD, &&TARGET_MAP_ADD, &&TARGET_LOAD_CLASSDEREF, &&TARGET_COPY_FREE_VARS, - &&TARGET_STORE_ATTR_ADAPTIVE, + &&TARGET_RESUME_QUICK, &&TARGET_RESUME, &&TARGET_MATCH_CLASS, + &&TARGET_STORE_ATTR_ADAPTIVE, &&TARGET_STORE_ATTR_INSTANCE_VALUE, - &&TARGET_STORE_ATTR_SLOT, &&TARGET_FORMAT_VALUE, &&TARGET_BUILD_CONST_KEY_MAP, &&TARGET_BUILD_STRING, + &&TARGET_STORE_ATTR_SLOT, &&TARGET_STORE_ATTR_WITH_HINT, - &&TARGET_STORE_FAST__LOAD_FAST, &&TARGET_LOAD_METHOD, - &&TARGET_STORE_FAST__STORE_FAST, + &&TARGET_STORE_FAST__LOAD_FAST, &&TARGET_LIST_EXTEND, &&TARGET_SET_UPDATE, &&TARGET_DICT_MERGE, &&TARGET_DICT_UPDATE, &&TARGET_PRECALL, + &&TARGET_STORE_FAST__STORE_FAST, &&TARGET_STORE_SUBSCR_ADAPTIVE, &&TARGET_STORE_SUBSCR_DICT, &&TARGET_STORE_SUBSCR_LIST_INT, - &&TARGET_UNPACK_SEQUENCE_ADAPTIVE, &&TARGET_CALL, &&TARGET_KW_NAMES, &&TARGET_POP_JUMP_BACKWARD_IF_NOT_NONE, &&TARGET_POP_JUMP_BACKWARD_IF_NONE, &&TARGET_POP_JUMP_BACKWARD_IF_FALSE, &&TARGET_POP_JUMP_BACKWARD_IF_TRUE, + &&TARGET_UNPACK_SEQUENCE_ADAPTIVE, &&TARGET_UNPACK_SEQUENCE_LIST, &&TARGET_UNPACK_SEQUENCE_TUPLE, &&TARGET_UNPACK_SEQUENCE_TWO_TUPLE, @@ -253,6 +254,5 @@ static void *opcode_targets[256] = { &&_unknown_opcode, &&_unknown_opcode, &&_unknown_opcode, - &&_unknown_opcode, &&TARGET_DO_TRACING }; From webhook-mailer at python.org Thu Apr 28 06:26:22 2022 From: webhook-mailer at python.org (serhiy-storchaka) Date: Thu, 28 Apr 2022 10:26:22 -0000 Subject: [Python-checkins] gh-91603: Speed up operator "|" for UnionType (GH-91955) Message-ID: https://github.com/python/cpython/commit/cd1fbbc81761dc26ce6daf724d57d48e965e5817 commit: cd1fbbc81761dc26ce6daf724d57d48e965e5817 branch: main author: Serhiy Storchaka committer: serhiy-storchaka date: 2022-04-28T13:25:33+03:00 summary: gh-91603: Speed up operator "|" for UnionType (GH-91955) Reduce the complexity from O((M+N)^2) to O(M*N), where M and N are the length of __args__ for both operands (1 for operand which is not a UnionType). As a consequence, the complexity of parameter substitution in UnionType has been reduced from O(N^3) to O(N^2). Co-authored-by: Yurii Karabas <1998uriyyo at gmail.com> files: A Misc/NEWS.d/next/Core and Builtins/2022-04-23-22-08-34.gh-issue-91603.GcWEkK.rst M Objects/unionobject.c diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-04-23-22-08-34.gh-issue-91603.GcWEkK.rst b/Misc/NEWS.d/next/Core and Builtins/2022-04-23-22-08-34.gh-issue-91603.GcWEkK.rst new file mode 100644 index 0000000000000..c12ab72a6d672 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2022-04-23-22-08-34.gh-issue-91603.GcWEkK.rst @@ -0,0 +1,2 @@ +Speed up :class:`types.UnionType` instantiation. Based on patch provided by Yurii +Karabas. diff --git a/Objects/unionobject.c b/Objects/unionobject.c index 5432c6faa3f90..36b032c0c5c12 100644 --- a/Objects/unionobject.c +++ b/Objects/unionobject.c @@ -137,93 +137,80 @@ union_richcompare(PyObject *a, PyObject *b, int op) return result; } -static PyObject* -flatten_args(PyObject* args) +static int +is_same(PyObject *left, PyObject *right) +{ + int is_ga = _PyGenericAlias_Check(left) && _PyGenericAlias_Check(right); + return is_ga ? PyObject_RichCompareBool(left, right, Py_EQ) : left == right; +} + +static int +contains(PyObject **items, Py_ssize_t size, PyObject *obj) { - Py_ssize_t arg_length = PyTuple_GET_SIZE(args); - Py_ssize_t total_args = 0; - // Get number of total args once it's flattened. - for (Py_ssize_t i = 0; i < arg_length; i++) { - PyObject *arg = PyTuple_GET_ITEM(args, i); - if (_PyUnion_Check(arg)) { - total_args += PyTuple_GET_SIZE(((unionobject*) arg)->args); - } else { - total_args++; + for (int i = 0; i < size; i++) { + int is_duplicate = is_same(items[i], obj); + if (is_duplicate) { // -1 or 1 + return is_duplicate; } } - // Create new tuple of flattened args. - PyObject *flattened_args = PyTuple_New(total_args); - if (flattened_args == NULL) { - return NULL; - } + return 0; +} + +static PyObject * +merge(PyObject **items1, Py_ssize_t size1, + PyObject **items2, Py_ssize_t size2) +{ + PyObject *tuple = NULL; Py_ssize_t pos = 0; - for (Py_ssize_t i = 0; i < arg_length; i++) { - PyObject *arg = PyTuple_GET_ITEM(args, i); - if (_PyUnion_Check(arg)) { - PyObject* nested_args = ((unionobject*)arg)->args; - Py_ssize_t nested_arg_length = PyTuple_GET_SIZE(nested_args); - for (Py_ssize_t j = 0; j < nested_arg_length; j++) { - PyObject* nested_arg = PyTuple_GET_ITEM(nested_args, j); - Py_INCREF(nested_arg); - PyTuple_SET_ITEM(flattened_args, pos, nested_arg); - pos++; + + for (int i = 0; i < size2; i++) { + PyObject *arg = items2[i]; + int is_duplicate = contains(items1, size1, arg); + if (is_duplicate < 0) { + Py_XDECREF(tuple); + return NULL; + } + if (is_duplicate) { + continue; + } + + if (tuple == NULL) { + tuple = PyTuple_New(size1 + size2 - i); + if (tuple == NULL) { + return NULL; } - } else { - if (arg == Py_None) { - arg = (PyObject *)&_PyNone_Type; + for (; pos < size1; pos++) { + PyObject *a = items1[pos]; + Py_INCREF(a); + PyTuple_SET_ITEM(tuple, pos, a); } - Py_INCREF(arg); - PyTuple_SET_ITEM(flattened_args, pos, arg); - pos++; } + Py_INCREF(arg); + PyTuple_SET_ITEM(tuple, pos, arg); + pos++; + } + + if (tuple) { + (void) _PyTuple_Resize(&tuple, pos); } - assert(pos == total_args); - return flattened_args; + return tuple; } -static PyObject* -dedup_and_flatten_args(PyObject* args) +static PyObject ** +get_types(PyObject **obj, Py_ssize_t *size) { - args = flatten_args(args); - if (args == NULL) { - return NULL; + if (*obj == Py_None) { + *obj = (PyObject *)&_PyNone_Type; } - Py_ssize_t arg_length = PyTuple_GET_SIZE(args); - PyObject *new_args = PyTuple_New(arg_length); - if (new_args == NULL) { - Py_DECREF(args); - return NULL; + if (_PyUnion_Check(*obj)) { + PyObject *args = ((unionobject *) *obj)->args; + *size = PyTuple_GET_SIZE(args); + return &PyTuple_GET_ITEM(args, 0); } - // Add unique elements to an array. - Py_ssize_t added_items = 0; - for (Py_ssize_t i = 0; i < arg_length; i++) { - int is_duplicate = 0; - PyObject* i_element = PyTuple_GET_ITEM(args, i); - for (Py_ssize_t j = 0; j < added_items; j++) { - PyObject* j_element = PyTuple_GET_ITEM(new_args, j); - int is_ga = _PyGenericAlias_Check(i_element) && - _PyGenericAlias_Check(j_element); - // RichCompare to also deduplicate GenericAlias types (slower) - is_duplicate = is_ga ? PyObject_RichCompareBool(i_element, j_element, Py_EQ) - : i_element == j_element; - // Should only happen if RichCompare fails - if (is_duplicate < 0) { - Py_DECREF(args); - Py_DECREF(new_args); - return NULL; - } - if (is_duplicate) - break; - } - if (!is_duplicate) { - Py_INCREF(i_element); - PyTuple_SET_ITEM(new_args, added_items, i_element); - added_items++; - } + else { + *size = 1; + return obj; } - Py_DECREF(args); - _PyTuple_Resize(&new_args, added_items); - return new_args; } static int @@ -242,9 +229,16 @@ _Py_union_type_or(PyObject* self, PyObject* other) Py_RETURN_NOTIMPLEMENTED; } - PyObject *tuple = PyTuple_Pack(2, self, other); + Py_ssize_t size1, size2; + PyObject **items1 = get_types(&self, &size1); + PyObject **items2 = get_types(&other, &size2); + PyObject *tuple = merge(items1, size1, items2, size2); if (tuple == NULL) { - return NULL; + if (PyErr_Occurred()) { + return NULL; + } + Py_INCREF(self); + return self; } PyObject *new_union = make_union(tuple); @@ -468,23 +462,12 @@ make_union(PyObject *args) { assert(PyTuple_CheckExact(args)); - args = dedup_and_flatten_args(args); - if (args == NULL) { - return NULL; - } - if (PyTuple_GET_SIZE(args) == 1) { - PyObject *result1 = PyTuple_GET_ITEM(args, 0); - Py_INCREF(result1); - Py_DECREF(args); - return result1; - } - unionobject *result = PyObject_GC_New(unionobject, &_PyUnion_Type); if (result == NULL) { - Py_DECREF(args); return NULL; } + Py_INCREF(args); result->parameters = NULL; result->args = args; _PyObject_GC_TRACK(result); From webhook-mailer at python.org Thu Apr 28 07:30:10 2022 From: webhook-mailer at python.org (serhiy-storchaka) Date: Thu, 28 Apr 2022 11:30:10 -0000 Subject: [Python-checkins] gh-91984: Fix trailing spaces in multiline test strings in test_argparse (GH-91986) Message-ID: https://github.com/python/cpython/commit/88dd22795925a8f47d38737ca8067eeb3f0a13aa commit: 88dd22795925a8f47d38737ca8067eeb3f0a13aa branch: main author: Abhigyan Bose committer: serhiy-storchaka date: 2022-04-28T14:29:24+03:00 summary: gh-91984: Fix trailing spaces in multiline test strings in test_argparse (GH-91986) files: A Misc/NEWS.d/next/Library/2022-04-27-18-30-00.gh-issue-91984.LxAB11.rst M Lib/test/test_argparse.py diff --git a/Lib/test/test_argparse.py b/Lib/test/test_argparse.py index 1f03b7fb24261..fd107faee4765 100644 --- a/Lib/test/test_argparse.py +++ b/Lib/test/test_argparse.py @@ -2265,8 +2265,7 @@ def test_help_blank(self): main description positional arguments: - foo - + foo \n options: -h, --help show this help message and exit ''')) @@ -2282,8 +2281,7 @@ def test_help_blank(self): main description positional arguments: - {} - + {} \n options: -h, --help show this help message and exit ''')) diff --git a/Misc/NEWS.d/next/Library/2022-04-27-18-30-00.gh-issue-91984.LxAB11.rst b/Misc/NEWS.d/next/Library/2022-04-27-18-30-00.gh-issue-91984.LxAB11.rst new file mode 100644 index 0000000000000..82c2907bf4daa --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-27-18-30-00.gh-issue-91984.LxAB11.rst @@ -0,0 +1 @@ +Modified test strings in test_argparse.py to not contain trailing spaces before end of line. From webhook-mailer at python.org Thu Apr 28 08:02:58 2022 From: webhook-mailer at python.org (vstinner) Date: Thu, 28 Apr 2022 12:02:58 -0000 Subject: [Python-checkins] gh-89479: Export _Py_GetSpecializationStats() internal function (#92011) Message-ID: https://github.com/python/cpython/commit/87c6cf9aa7cebac3c544067e690c2ad7ce18b772 commit: 87c6cf9aa7cebac3c544067e690c2ad7ce18b772 branch: main author: Victor Stinner committer: vstinner date: 2022-04-28T14:02:45+02:00 summary: gh-89479: Export _Py_GetSpecializationStats() internal function (#92011) When Python is built with "./configure --enable-pystats" (if the Py_STATS macro is defined), the _Py_GetSpecializationStats() function must be exported, since it's used by the _opcode extension which is built as a shared library. files: M Doc/using/configure.rst M Include/internal/pycore_code.h diff --git a/Doc/using/configure.rst b/Doc/using/configure.rst index 057efa3bd077c..2321d24663f68 100644 --- a/Doc/using/configure.rst +++ b/Doc/using/configure.rst @@ -139,7 +139,7 @@ General Options The statistics will be dumped to a arbitrary (probably unique) file in ``/tmp/py_stats/``, or ``C:\temp\py_stats\`` on Windows. - Use ``Tools//summarize_stats.py`` to read the stats. + Use ``Tools/scripts/summarize_stats.py`` to read the stats. .. versionadded:: 3.11 diff --git a/Include/internal/pycore_code.h b/Include/internal/pycore_code.h index 3059db465e7d2..8a599c4246cfc 100644 --- a/Include/internal/pycore_code.h +++ b/Include/internal/pycore_code.h @@ -316,7 +316,8 @@ extern PyStats _py_stats; extern void _Py_PrintSpecializationStats(int to_file); -extern PyObject* _Py_GetSpecializationStats(void); +// Used by the _opcode extension which is built as a shared library +PyAPI_FUNC(PyObject*) _Py_GetSpecializationStats(void); #else #define STAT_INC(opname, name) ((void)0) @@ -324,7 +325,7 @@ extern PyObject* _Py_GetSpecializationStats(void); #define OPCODE_EXE_INC(opname) ((void)0) #define CALL_STAT_INC(name) ((void)0) #define OBJECT_STAT_INC(name) ((void)0) -#endif +#endif // !Py_STATS // Cache values are only valid in memory, so use native endianness. #ifdef WORDS_BIGENDIAN From webhook-mailer at python.org Thu Apr 28 08:14:27 2022 From: webhook-mailer at python.org (miss-islington) Date: Thu, 28 Apr 2022 12:14:27 -0000 Subject: [Python-checkins] gh-84461: Add HOSTRUNNER for program to run Python executable (GH-91931) Message-ID: https://github.com/python/cpython/commit/d1de10784d9678d14759d0d03216431d1090e60e commit: d1de10784d9678d14759d0d03216431d1090e60e branch: main author: Ethan Smith committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-28T05:14:19-07:00 summary: gh-84461: Add HOSTRUNNER for program to run Python executable (GH-91931) `HOSTRUNNER` is a program which can be used to run `BUILDPYTHON` for the host platform (for example, `python.js` requires `node`). Also change depedencies from `build_all` to `all` so that targets which can't build everything (e.g. WASM) can still run `buildbottest` and `pythoninfo`. cc @tiran files: M Makefile.pre.in M configure M configure.ac diff --git a/Makefile.pre.in b/Makefile.pre.in index f3bccf4fec7ec..c12305aed6a11 100644 --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -285,6 +285,8 @@ LIBOBJS= @LIBOBJS@ PYTHON= python$(EXE) BUILDPYTHON= python$(BUILDEXE) +HOSTRUNNER= @HOSTRUNNER@ + PYTHON_FOR_REGEN?=@PYTHON_FOR_REGEN@ UPDATE_FILE=$(PYTHON_FOR_REGEN) $(srcdir)/Tools/scripts/update_file.py PYTHON_FOR_BUILD=@PYTHON_FOR_BUILD@ @@ -1350,7 +1352,7 @@ regen-keyword: $(UPDATE_FILE) $(srcdir)/Lib/keyword.py $(srcdir)/Lib/keyword.py.new .PHONY: regen-stdlib-module-names -regen-stdlib-module-names: build_all Programs/_testembed +regen-stdlib-module-names: all Programs/_testembed # Regenerate Python/stdlib_module_names.h # using Tools/scripts/generate_stdlib_module_names.py $(RUNSHARED) ./$(BUILDPYTHON) \ @@ -1648,7 +1650,7 @@ $(LIBRARY_OBJS) $(MODOBJS) Programs/python.o: $(PYTHON_HEADERS) ###################################################################### TESTOPTS= $(EXTRATESTOPTS) -TESTPYTHON= $(RUNSHARED) ./$(BUILDPYTHON) $(TESTPYTHONOPTS) +TESTPYTHON= $(RUNSHARED) $(PYTHON_FOR_BUILD) $(TESTPYTHONOPTS) TESTRUNNER= $(TESTPYTHON) $(srcdir)/Tools/scripts/run_tests.py TESTTIMEOUT= 1200 @@ -1656,7 +1658,7 @@ TESTTIMEOUT= 1200 # Remove "test_python_*" directories of previous failed test jobs. # Pass TESTOPTS options because it can contain --tempdir option. -cleantest: build_all +cleantest: all $(TESTRUNNER) $(TESTOPTS) --cleanup # Run a basic set of regression tests. @@ -1691,14 +1693,14 @@ testuniversal: @DEF_MAKE_RULE@ platform # Like testall, but with only one pass and without multiple processes. # Run an optional script to include information about the build environment. -buildbottest: build_all platform +buildbottest: all platform - at if which pybuildbot.identify >/dev/null 2>&1; then \ pybuildbot.identify "CC='$(CC)'" "CXX='$(CXX)'"; \ fi $(TESTRUNNER) -j 1 -u all -W --slowest --fail-env-changed --timeout=$(TESTTIMEOUT) $(TESTOPTS) -pythoninfo: build_all - $(RUNSHARED) ./$(BUILDPYTHON) -m test.pythoninfo +pythoninfo: all + $(RUNSHARED) $(HOSTRUNNER) ./$(BUILDPYTHON) -m test.pythoninfo QUICKTESTOPTS= $(TESTOPTS) -x test_subprocess test_io test_lib2to3 \ test_multibytecodec test_urllib2_localnet test_itertools \ @@ -1711,10 +1713,10 @@ quicktest: @DEF_MAKE_RULE@ platform # SSL tests .PHONY: multisslcompile multissltest -multisslcompile: build_all +multisslcompile: all $(RUNSHARED) ./$(BUILDPYTHON) $(srcdir)/Tools/ssl/multissltests.py --steps=modules -multissltest: build_all +multissltest: all $(RUNSHARED) ./$(BUILDPYTHON) $(srcdir)/Tools/ssl/multissltests.py install: @FRAMEWORKINSTALLFIRST@ commoninstall bininstall maninstall @FRAMEWORKINSTALLLAST@ @@ -1986,7 +1988,7 @@ TESTSUBDIRS= ctypes/test \ unittest/test unittest/test/testmock TEST_MODULES=@TEST_MODULES@ -libinstall: build_all $(srcdir)/Modules/xxmodule.c +libinstall: all $(srcdir)/Modules/xxmodule.c @for i in $(SCRIPTDIR) $(LIBDEST); \ do \ if test ! -d $(DESTDIR)$$i; then \ diff --git a/configure b/configure index f161e4ebed94b..2c47a1e85c145 100755 --- a/configure +++ b/configure @@ -888,6 +888,7 @@ AR LINK_PYTHON_OBJS LINK_PYTHON_DEPS LIBRARY_DEPS +HOSTRUNNER STATIC_LIBPYTHON GNULD EXPORTSFROM @@ -1076,6 +1077,7 @@ LDFLAGS LIBS CPPFLAGS CPP +HOSTRUNNER PROFILE_TASK LIBUUID_CFLAGS LIBUUID_LIBS @@ -1878,6 +1880,7 @@ Some influential environment variables: CPPFLAGS (Objective) C/C++ preprocessor flags, e.g. -I if you have headers in a nonstandard directory CPP C preprocessor + HOSTRUNNER Program to run CPython for the host platform PROFILE_TASK Python args for PGO generation task LIBUUID_CFLAGS @@ -6671,6 +6674,33 @@ if test "$cross_compiling" = yes; then RUNSHARED= fi + +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking HOSTRUNNER" >&5 +$as_echo_n "checking HOSTRUNNER... " >&6; } +if test -z "$HOSTRUNNER" +then + case $ac_sys_system/$ac_sys_emscripten_target in #( + Emscripten/node*) : + + if test "x$enable_wasm_pthreads" = xyes; then : + + HOSTRUNNER='node --experimental-wasm-threads --experimental-wasm-bulk-memory' + +else + + HOSTRUNNER='node' + +fi + ;; #( + *) : + HOSTRUNNER='' + ;; +esac +fi + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $HOSTRUNNER" >&5 +$as_echo "$HOSTRUNNER" >&6; } + { $as_echo "$as_me:${as_lineno-$LINENO}: result: $LDLIBRARY" >&5 $as_echo "$LDLIBRARY" >&6; } diff --git a/configure.ac b/configure.ac index 86839547e2683..62da321a98af1 100644 --- a/configure.ac +++ b/configure.ac @@ -1412,6 +1412,24 @@ if test "$cross_compiling" = yes; then RUNSHARED= fi +AC_ARG_VAR([HOSTRUNNER], [Program to run CPython for the host platform]) +AC_MSG_CHECKING([HOSTRUNNER]) +if test -z "$HOSTRUNNER" +then + AS_CASE([$ac_sys_system/$ac_sys_emscripten_target], + [Emscripten/node*], [ + AS_VAR_IF([enable_wasm_pthreads], [yes], [ + HOSTRUNNER='node --experimental-wasm-threads --experimental-wasm-bulk-memory' + ], [ + HOSTRUNNER='node' + ]) + ], + [HOSTRUNNER=''] + ) +fi +AC_SUBST([HOSTRUNNER]) +AC_MSG_RESULT([$HOSTRUNNER]) + AC_MSG_RESULT($LDLIBRARY) # LIBRARY_DEPS, LINK_PYTHON_OBJS and LINK_PYTHON_DEPS variable From webhook-mailer at python.org Thu Apr 28 10:30:39 2022 From: webhook-mailer at python.org (encukou) Date: Thu, 28 Apr 2022 14:30:39 -0000 Subject: [Python-checkins] gh-91324: List feature macros in the stable ABI manifest, improve tests (GH-32415) Message-ID: https://github.com/python/cpython/commit/6dcbc08c95cce4630b3bfb53bdb74e2523795555 commit: 6dcbc08c95cce4630b3bfb53bdb74e2523795555 branch: main author: Petr Viktorin committer: encukou date: 2022-04-28T16:30:28+02:00 summary: gh-91324: List feature macros in the stable ABI manifest, improve tests (GH-32415) files: A Modules/_testcapi_feature_macros.inc M Lib/test/test_stable_abi_ctypes.py M Misc/stable_abi.txt M Modules/_testcapimodule.c M PC/python3dll.c M Tools/scripts/stable_abi.py diff --git a/Lib/test/test_stable_abi_ctypes.py b/Lib/test/test_stable_abi_ctypes.py index 0656ff5581be5..311e216e5066c 100644 --- a/Lib/test/test_stable_abi_ctypes.py +++ b/Lib/test/test_stable_abi_ctypes.py @@ -4,17 +4,35 @@ """Test that all symbols of the Stable ABI are accessible using ctypes """ +import sys import unittest from test.support.import_helper import import_module +from _testcapi import get_feature_macros +feature_macros = get_feature_macros() ctypes_test = import_module('ctypes') class TestStableABIAvailability(unittest.TestCase): def test_available_symbols(self): + for symbol_name in SYMBOL_NAMES: with self.subTest(symbol_name): ctypes_test.pythonapi[symbol_name] + def test_feature_macros(self): + self.assertEqual(set(get_feature_macros()), EXPECTED_IFDEFS) + + # The feature macros for Windows are used in creating the DLL + # definition, so they must be known on all platforms. + # If we are on Windows, we check that the hardcoded data matches + # the reality. + @unittest.skipIf(sys.platform != "win32", "Windows specific test") + def test_windows_feature_macros(self): + for name, value in WINDOWS_IFDEFS.items(): + if value != 'maybe': + with self.subTest(name): + self.assertEqual(feature_macros[name], value) + SYMBOL_NAMES = ( "PyAIter_Check", @@ -855,3 +873,41 @@ def test_available_symbols(self): "_Py_TrueStruct", "_Py_VaBuildValue_SizeT", ) +if feature_macros['MS_WINDOWS']: + SYMBOL_NAMES += ( + 'PyErr_SetExcFromWindowsErr', + 'PyErr_SetExcFromWindowsErrWithFilename', + 'PyErr_SetExcFromWindowsErrWithFilenameObject', + 'PyErr_SetExcFromWindowsErrWithFilenameObjects', + 'PyErr_SetFromWindowsErr', + 'PyErr_SetFromWindowsErrWithFilename', + 'PyExc_WindowsError', + 'PyUnicode_AsMBCSString', + 'PyUnicode_DecodeCodePageStateful', + 'PyUnicode_DecodeMBCS', + 'PyUnicode_DecodeMBCSStateful', + 'PyUnicode_EncodeCodePage', + ) +if feature_macros['HAVE_FORK']: + SYMBOL_NAMES += ( + 'PyOS_AfterFork', + 'PyOS_AfterFork_Child', + 'PyOS_AfterFork_Parent', + 'PyOS_BeforeFork', + ) +if feature_macros['USE_STACKCHECK']: + SYMBOL_NAMES += ( + 'PyOS_CheckStack', + ) +if feature_macros['PY_HAVE_THREAD_NATIVE_ID']: + SYMBOL_NAMES += ( + 'PyThread_get_thread_native_id', + ) +if feature_macros['Py_REF_DEBUG']: + SYMBOL_NAMES += ( + '_Py_NegativeRefcount', + '_Py_RefTotal', + ) + +EXPECTED_IFDEFS = set(['HAVE_FORK', 'MS_WINDOWS', 'PY_HAVE_THREAD_NATIVE_ID', 'Py_REF_DEBUG', 'USE_STACKCHECK']) +WINDOWS_IFDEFS = {'MS_WINDOWS': True, 'HAVE_FORK': False, 'USE_STACKCHECK': 'maybe', 'PY_HAVE_THREAD_NATIVE_ID': True, 'Py_REF_DEBUG': 'maybe'} diff --git a/Misc/stable_abi.txt b/Misc/stable_abi.txt index 66777a62c4301..9b1c87ea8f4f1 100644 --- a/Misc/stable_abi.txt +++ b/Misc/stable_abi.txt @@ -29,6 +29,8 @@ # value may change. # - typedef: A C typedef which is used in other definitions in the limited API. # Its size/layout/signature must not change. +# - ifdef: A feature macro: other items may be conditional on whether the macro +# is defined or not. # Each top-level item can have details defined below it: # - added: The version in which the item was added to the stable ABI. @@ -41,6 +43,10 @@ # of the stable ABI. # - a combination of the above (functions that were called by macros that # were public in the past) +# - doc: for `ifdef`, the blurb added in documentation +# - windows: for `ifdef`, this macro is defined on Windows. (This info is used +# to generate the DLL manifest and needs to be available on all platforms.) +# `maybe` marks macros defined on some but not all Windows builds. # For structs, one of the following must be set: # - opaque: The struct name is available in the Limited API, but its members @@ -59,6 +65,24 @@ # https://docs.python.org/3/c-api/stable.html#stable +# Feature macros for optional functionality: + +ifdef MS_WINDOWS + doc on Windows + windows +ifdef HAVE_FORK + doc on platforms with fork() +ifdef USE_STACKCHECK + doc on platforms with USE_STACKCHECK + windows maybe +ifdef PY_HAVE_THREAD_NATIVE_ID + doc on platforms with native thread IDs + windows +ifdef Py_REF_DEBUG + doc when Python is compiled in debug mode (with Py_REF_DEBUG) + windows maybe + + # Mentioned in PEP 384: struct PyObject diff --git a/Modules/_testcapi_feature_macros.inc b/Modules/_testcapi_feature_macros.inc new file mode 100644 index 0000000000000..b1763b57d913c --- /dev/null +++ b/Modules/_testcapi_feature_macros.inc @@ -0,0 +1,49 @@ +// Generated by Tools/scripts/stable_abi.py + +// Add an entry in dict `result` for each Stable ABI feature macro. + +#ifdef HAVE_FORK + res = PyDict_SetItemString(result, "HAVE_FORK", Py_True); +#else + res = PyDict_SetItemString(result, "HAVE_FORK", Py_False); +#endif +if (res) { + Py_DECREF(result); return NULL; +} + +#ifdef MS_WINDOWS + res = PyDict_SetItemString(result, "MS_WINDOWS", Py_True); +#else + res = PyDict_SetItemString(result, "MS_WINDOWS", Py_False); +#endif +if (res) { + Py_DECREF(result); return NULL; +} + +#ifdef PY_HAVE_THREAD_NATIVE_ID + res = PyDict_SetItemString(result, "PY_HAVE_THREAD_NATIVE_ID", Py_True); +#else + res = PyDict_SetItemString(result, "PY_HAVE_THREAD_NATIVE_ID", Py_False); +#endif +if (res) { + Py_DECREF(result); return NULL; +} + +#ifdef Py_REF_DEBUG + res = PyDict_SetItemString(result, "Py_REF_DEBUG", Py_True); +#else + res = PyDict_SetItemString(result, "Py_REF_DEBUG", Py_False); +#endif +if (res) { + Py_DECREF(result); return NULL; +} + +#ifdef USE_STACKCHECK + res = PyDict_SetItemString(result, "USE_STACKCHECK", Py_True); +#else + res = PyDict_SetItemString(result, "USE_STACKCHECK", Py_False); +#endif +if (res) { + Py_DECREF(result); return NULL; +} + diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c index 6bd73e8f45379..9073f33e226bd 100644 --- a/Modules/_testcapimodule.c +++ b/Modules/_testcapimodule.c @@ -5919,6 +5919,18 @@ frame_getlasti(PyObject *self, PyObject *frame) return PyLong_FromLong(lasti); } +static PyObject * +get_feature_macros(PyObject *self, PyObject *Py_UNUSED(args)) +{ + PyObject *result = PyDict_New(); + if (!result) { + return NULL; + } + int res; +#include "_testcapi_feature_macros.inc" + return result; +} + static PyObject *negative_dictoffset(PyObject *, PyObject *); static PyObject *test_buildvalue_issue38913(PyObject *, PyObject *); @@ -6214,6 +6226,7 @@ static PyMethodDef TestMethods[] = { {"frame_getgenerator", frame_getgenerator, METH_O, NULL}, {"frame_getbuiltins", frame_getbuiltins, METH_O, NULL}, {"frame_getlasti", frame_getlasti, METH_O, NULL}, + {"get_feature_macros", get_feature_macros, METH_NOARGS, NULL}, {NULL, NULL} /* sentinel */ }; diff --git a/PC/python3dll.c b/PC/python3dll.c index aabc1e83868e8..50e7a9607bec9 100755 --- a/PC/python3dll.c +++ b/PC/python3dll.c @@ -19,6 +19,7 @@ EXPORT_FUNC(_Py_CheckRecursiveCall) EXPORT_FUNC(_Py_Dealloc) EXPORT_FUNC(_Py_DecRef) EXPORT_FUNC(_Py_IncRef) +EXPORT_FUNC(_Py_NegativeRefcount) EXPORT_FUNC(_Py_VaBuildValue_SizeT) EXPORT_FUNC(_PyArg_Parse_SizeT) EXPORT_FUNC(_PyArg_ParseTuple_SizeT) @@ -730,6 +731,7 @@ EXPORT_DATA(_Py_EllipsisObject) EXPORT_DATA(_Py_FalseStruct) EXPORT_DATA(_Py_NoneStruct) EXPORT_DATA(_Py_NotImplementedStruct) +EXPORT_DATA(_Py_RefTotal) EXPORT_DATA(_Py_SwappedOp) EXPORT_DATA(_Py_TrueStruct) EXPORT_DATA(_PyWeakref_CallableProxyType) diff --git a/Tools/scripts/stable_abi.py b/Tools/scripts/stable_abi.py index 7376a4649ca05..54075248c7ea2 100755 --- a/Tools/scripts/stable_abi.py +++ b/Tools/scripts/stable_abi.py @@ -45,21 +45,6 @@ MACOS = (sys.platform == "darwin") UNIXY = MACOS or (sys.platform == "linux") # XXX should this be "not Windows"? -IFDEF_DOC_NOTES = { - 'MS_WINDOWS': 'on Windows', - 'HAVE_FORK': 'on platforms with fork()', - 'USE_STACKCHECK': 'on platforms with USE_STACKCHECK', - 'PY_HAVE_THREAD_NATIVE_ID': 'on platforms with native thread IDs', -} - -# To generate the DLL definition, we need to know which feature macros are -# defined on Windows. On all platforms. -# Best way to do that is to hardcode the list (and later test in on Windows). -WINDOWS_IFDEFS = frozenset({ - 'MS_WINDOWS', - 'PY_HAVE_THREAD_NATIVE_ID', - 'USE_STACKCHECK', -}) # The stable ABI manifest (Misc/stable_abi.txt) exists only to fill the # following dataclasses. @@ -130,9 +115,11 @@ class ABIItem: ifdef: str = None struct_abi_kind: str = None members: list = None + doc: str = None + windows: bool = False KINDS = frozenset({ - 'struct', 'function', 'macro', 'data', 'const', 'typedef', + 'struct', 'function', 'macro', 'data', 'const', 'typedef', 'ifdef', }) def dump(self, indent=0): @@ -171,8 +158,8 @@ def raise_error(msg): levels.pop() parent = levels[-1][0] entry = None - if kind in ABIItem.KINDS: - if parent.kind not in {'manifest'}: + if parent.kind == 'manifest': + if kind not in kind in ABIItem.KINDS: raise_error(f'{kind} cannot go in {parent.kind}') entry = ABIItem(kind, content) parent.add(entry) @@ -193,10 +180,29 @@ def raise_error(msg): parent.struct_abi_kind = kind if kind == 'members': parent.members = content.split() + elif kind in {'doc'}: + if parent.kind not in {'ifdef'}: + raise_error(f'{kind} cannot go in {parent.kind}') + parent.doc = content + elif kind in {'windows'}: + if parent.kind not in {'ifdef'}: + raise_error(f'{kind} cannot go in {parent.kind}') + if not content: + parent.windows = True + elif content == 'maybe': + parent.windows = content + else: + raise_error(f'Unexpected: {content}') else: raise_error(f"unknown kind {kind!r}") # When adding more, update the comment in stable_abi.txt. levels.append((entry, level)) + + ifdef_names = {i.name for i in manifest.select({'ifdef'})} + for item in manifest.contents.values(): + if item.ifdef and item.ifdef not in ifdef_names: + raise ValueError(f'{item.name} uses undeclared ifdef {item.ifdef}') + return manifest # The tool can run individual "actions". @@ -240,9 +246,12 @@ def gen_python3dll(manifest, args, outfile): def sort_key(item): return item.name.lower() + windows_ifdefs = { + item.name for item in manifest.select({'ifdef'}) if item.windows + } for item in sorted( manifest.select( - {'function'}, include_abi_only=True, ifdef=WINDOWS_IFDEFS), + {'function'}, include_abi_only=True, ifdef=windows_ifdefs), key=sort_key): write(f'EXPORT_FUNC({item.name})') @@ -250,7 +259,7 @@ def sort_key(item): for item in sorted( manifest.select( - {'data'}, include_abi_only=True, ifdef=WINDOWS_IFDEFS), + {'data'}, include_abi_only=True, ifdef=windows_ifdefs), key=sort_key): write(f'EXPORT_DATA({item.name})') @@ -273,7 +282,7 @@ def gen_doc_annotations(manifest, args, outfile): writer.writeheader() for item in manifest.select(REST_ROLES.keys(), include_abi_only=False): if item.ifdef: - ifdef_note = IFDEF_DOC_NOTES[item.ifdef] + ifdef_note = manifest.contents[item.ifdef].doc else: ifdef_note = None writer.writerow({ @@ -298,23 +307,42 @@ def gen_ctypes_test(manifest, args, outfile): """Test that all symbols of the Stable ABI are accessible using ctypes """ + import sys import unittest from test.support.import_helper import import_module + from _testcapi import get_feature_macros + feature_macros = get_feature_macros() ctypes_test = import_module('ctypes') class TestStableABIAvailability(unittest.TestCase): def test_available_symbols(self): + for symbol_name in SYMBOL_NAMES: with self.subTest(symbol_name): ctypes_test.pythonapi[symbol_name] + def test_feature_macros(self): + self.assertEqual(set(get_feature_macros()), EXPECTED_IFDEFS) + + # The feature macros for Windows are used in creating the DLL + # definition, so they must be known on all platforms. + # If we are on Windows, we check that the hardcoded data matches + # the reality. + @unittest.skipIf(sys.platform != "win32", "Windows specific test") + def test_windows_feature_macros(self): + for name, value in WINDOWS_IFDEFS.items(): + if value != 'maybe': + with self.subTest(name): + self.assertEqual(feature_macros[name], value) + SYMBOL_NAMES = ( ''')) items = manifest.select( {'function', 'data'}, include_abi_only=True, - ifdef=set()) + ) + ifdef_items = {} for item in items: if item.name in ( # Some symbols aren't exported on all platforms. @@ -322,8 +350,45 @@ def test_available_symbols(self): 'PyModule_Create2', 'PyModule_FromDefAndSpec2', ): continue - write(f' "{item.name}",') + if item.ifdef: + ifdef_items.setdefault(item.ifdef, []).append(item.name) + else: + write(f' "{item.name}",') write(")") + for ifdef, names in ifdef_items.items(): + write(f"if feature_macros[{ifdef!r}]:") + write(f" SYMBOL_NAMES += (") + for name in names: + write(f" {name!r},") + write(" )") + write("") + write(f"EXPECTED_IFDEFS = set({sorted(ifdef_items)})") + + windows_ifdef_values = { + name: manifest.contents[name].windows for name in ifdef_items + } + write(f"WINDOWS_IFDEFS = {windows_ifdef_values}") + + + at generator("testcapi_feature_macros", 'Modules/_testcapi_feature_macros.inc') +def gen_testcapi_feature_macros(manifest, args, outfile): + """Generate/check the stable ABI list for documentation annotations""" + write = partial(print, file=outfile) + write('// Generated by Tools/scripts/stable_abi.py') + write() + write('// Add an entry in dict `result` for each Stable ABI feature macro.') + write() + for macro in manifest.select({'ifdef'}): + name = macro.name + write(f'#ifdef {name}') + write(f' res = PyDict_SetItemString(result, "{name}", Py_True);') + write('#else') + write(f' res = PyDict_SetItemString(result, "{name}", Py_False);') + write('#endif') + write('if (res) {') + write(' Py_DECREF(result); return NULL;') + write('}') + write() def generate_or_check(manifest, args, path, func): From webhook-mailer at python.org Thu Apr 28 10:50:52 2022 From: webhook-mailer at python.org (miss-islington) Date: Thu, 28 Apr 2022 14:50:52 -0000 Subject: [Python-checkins] gh-91832: Add 'required' attr to argparse.Action repr (GH-91841) Message-ID: https://github.com/python/cpython/commit/4ed3900041c688a02dca1eb3323083d720dd0d93 commit: 4ed3900041c688a02dca1eb3323083d720dd0d93 branch: main author: Abhigyan Bose committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-28T07:50:27-07:00 summary: gh-91832: Add 'required' attr to argparse.Action repr (GH-91841) # Adding 'required' to names in Lib.argparse.Action gh-91832: Added 'required' to the list `names` in `Lib.argparse.Action`. Changed constant strings that test the Action object. Automerge-Triggered-By: GH:merwok files: A Misc/NEWS.d/next/Library/2022-04-23-03-24-00.gh-issue-91832.TyLi65.rst M Lib/argparse.py M Lib/test/test_argparse.py diff --git a/Lib/argparse.py b/Lib/argparse.py index 429a72ab7841e..668e1d416231f 100644 --- a/Lib/argparse.py +++ b/Lib/argparse.py @@ -850,6 +850,7 @@ def _get_kwargs(self): 'default', 'type', 'choices', + 'required', 'help', 'metavar', ] diff --git a/Lib/test/test_argparse.py b/Lib/test/test_argparse.py index fd107faee4765..5777cb50f7e53 100644 --- a/Lib/test/test_argparse.py +++ b/Lib/test/test_argparse.py @@ -4900,12 +4900,13 @@ def test_optional(self): nargs='+', default=42, choices=[1, 2, 3], + required=False, help='HELP', metavar='METAVAR') string = ( "Action(option_strings=['--foo', '-a', '-b'], dest='b', " "nargs='+', const=None, default=42, type='int', " - "choices=[1, 2, 3], help='HELP', metavar='METAVAR')") + "choices=[1, 2, 3], required=False, help='HELP', metavar='METAVAR')") self.assertStringEqual(option, string) def test_argument(self): @@ -4916,12 +4917,13 @@ def test_argument(self): nargs='?', default=2.5, choices=[0.5, 1.5, 2.5], + required=True, help='H HH H', metavar='MV MV MV') string = ( "Action(option_strings=[], dest='x', nargs='?', " "const=None, default=2.5, type=%r, choices=[0.5, 1.5, 2.5], " - "help='H HH H', metavar='MV MV MV')" % float) + "required=True, help='H HH H', metavar='MV MV MV')" % float) self.assertStringEqual(argument, string) def test_namespace(self): diff --git a/Misc/NEWS.d/next/Library/2022-04-23-03-24-00.gh-issue-91832.TyLi65.rst b/Misc/NEWS.d/next/Library/2022-04-23-03-24-00.gh-issue-91832.TyLi65.rst new file mode 100644 index 0000000000000..0ebf773546558 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-23-03-24-00.gh-issue-91832.TyLi65.rst @@ -0,0 +1 @@ +Add ``required`` attribute to :class:`argparse.Action` repr output. From webhook-mailer at python.org Thu Apr 28 11:19:18 2022 From: webhook-mailer at python.org (miss-islington) Date: Thu, 28 Apr 2022 15:19:18 -0000 Subject: [Python-checkins] gh-91832: Add 'required' attr to argparse.Action repr (GH-91841) Message-ID: https://github.com/python/cpython/commit/fa87c362e1861aa1092c1dcd409b00ae25db7874 commit: fa87c362e1861aa1092c1dcd409b00ae25db7874 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-28T08:19:07-07:00 summary: gh-91832: Add 'required' attr to argparse.Action repr (GH-91841) GH- Adding 'required' to names in Lib.argparse.Action gh-91832: Added 'required' to the list `names` in `Lib.argparse.Action`. Changed constant strings that test the Action object. Automerge-Triggered-By: GH:merwok (cherry picked from commit 4ed3900041c688a02dca1eb3323083d720dd0d93) Co-authored-by: Abhigyan Bose files: A Misc/NEWS.d/next/Library/2022-04-23-03-24-00.gh-issue-91832.TyLi65.rst M Lib/argparse.py M Lib/test/test_argparse.py diff --git a/Lib/argparse.py b/Lib/argparse.py index c46bb302711fc..b9c22051f69d7 100644 --- a/Lib/argparse.py +++ b/Lib/argparse.py @@ -848,6 +848,7 @@ def _get_kwargs(self): 'default', 'type', 'choices', + 'required', 'help', 'metavar', ] diff --git a/Lib/test/test_argparse.py b/Lib/test/test_argparse.py index dcc392faf2310..2d1cec13c22f3 100644 --- a/Lib/test/test_argparse.py +++ b/Lib/test/test_argparse.py @@ -4873,12 +4873,13 @@ def test_optional(self): nargs='+', default=42, choices=[1, 2, 3], + required=False, help='HELP', metavar='METAVAR') string = ( "Action(option_strings=['--foo', '-a', '-b'], dest='b', " "nargs='+', const=None, default=42, type='int', " - "choices=[1, 2, 3], help='HELP', metavar='METAVAR')") + "choices=[1, 2, 3], required=False, help='HELP', metavar='METAVAR')") self.assertStringEqual(option, string) def test_argument(self): @@ -4889,12 +4890,13 @@ def test_argument(self): nargs='?', default=2.5, choices=[0.5, 1.5, 2.5], + required=True, help='H HH H', metavar='MV MV MV') string = ( "Action(option_strings=[], dest='x', nargs='?', " "const=None, default=2.5, type=%r, choices=[0.5, 1.5, 2.5], " - "help='H HH H', metavar='MV MV MV')" % float) + "required=True, help='H HH H', metavar='MV MV MV')" % float) self.assertStringEqual(argument, string) def test_namespace(self): diff --git a/Misc/NEWS.d/next/Library/2022-04-23-03-24-00.gh-issue-91832.TyLi65.rst b/Misc/NEWS.d/next/Library/2022-04-23-03-24-00.gh-issue-91832.TyLi65.rst new file mode 100644 index 0000000000000..0ebf773546558 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-23-03-24-00.gh-issue-91832.TyLi65.rst @@ -0,0 +1 @@ +Add ``required`` attribute to :class:`argparse.Action` repr output. From webhook-mailer at python.org Thu Apr 28 11:24:45 2022 From: webhook-mailer at python.org (Fidget-Spinner) Date: Thu, 28 Apr 2022 15:24:45 -0000 Subject: [Python-checkins] gh-91603: Speed up isinstance/issubclass on union types (GH-91631) Message-ID: https://github.com/python/cpython/commit/0ef8d921f5c6945aa8f386e472c4110b81ac773d commit: 0ef8d921f5c6945aa8f386e472c4110b81ac773d branch: main author: Yurii Karabas <1998uriyyo at gmail.com> committer: Fidget-Spinner date: 2022-04-28T23:24:19+08:00 summary: gh-91603: Speed up isinstance/issubclass on union types (GH-91631) Co-authored-by: Jelle Zijlstra Co-authored-by: Shantanu <12621235+hauntsaninja at users.noreply.github.com> files: A Misc/NEWS.d/next/Core and Builtins/2022-04-17-11-03-45.gh-issue-91603.hYw1Lv.rst M Doc/library/functions.rst M Include/internal/pycore_unionobject.h M Lib/test/test_isinstance.py M Lib/test/test_types.py M Objects/abstract.c M Objects/unionobject.c diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst index f3b8e40babbd8..394281462ded5 100644 --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -905,7 +905,8 @@ are always available. They are listed here in alphabetical order. tuples) or a :ref:`types-union` of multiple types, return ``True`` if *object* is an instance of any of the types. If *classinfo* is not a type or tuple of types and such tuples, - a :exc:`TypeError` exception is raised. + a :exc:`TypeError` exception is raised. :exc:`TypeError` may not be + raised for an invalid type if an earlier check succeeds. .. versionchanged:: 3.10 *classinfo* can be a :ref:`types-union`. diff --git a/Include/internal/pycore_unionobject.h b/Include/internal/pycore_unionobject.h index 9962f57610387..a9ed5651a410e 100644 --- a/Include/internal/pycore_unionobject.h +++ b/Include/internal/pycore_unionobject.h @@ -15,6 +15,7 @@ extern PyObject *_Py_union_type_or(PyObject *, PyObject *); #define _PyGenericAlias_Check(op) PyObject_TypeCheck(op, &Py_GenericAliasType) extern PyObject *_Py_subs_parameters(PyObject *, PyObject *, PyObject *, PyObject *); extern PyObject *_Py_make_parameters(PyObject *); +extern PyObject *_Py_union_args(PyObject *self); #ifdef __cplusplus } diff --git a/Lib/test/test_isinstance.py b/Lib/test/test_isinstance.py index 9d37cff990338..a0974640bc114 100644 --- a/Lib/test/test_isinstance.py +++ b/Lib/test/test_isinstance.py @@ -225,7 +225,7 @@ def test_isinstance_with_or_union(self): with self.assertRaises(TypeError): isinstance(2, list[int] | int) with self.assertRaises(TypeError): - isinstance(2, int | str | list[int] | float) + isinstance(2, float | str | list[int] | int) diff --git a/Lib/test/test_types.py b/Lib/test/test_types.py index 42fd4f56235fa..cde9dadc5e97f 100644 --- a/Lib/test/test_types.py +++ b/Lib/test/test_types.py @@ -951,9 +951,9 @@ def __eq__(self, other): with self.assertRaises(ZeroDivisionError): list[int] | list[bt] - union_ga = (int | list[str], int | collections.abc.Callable[..., str], - int | d) - # Raise error when isinstance(type, type | genericalias) + union_ga = (list[str] | int, collections.abc.Callable[..., str] | int, + d | int) + # Raise error when isinstance(type, genericalias | type) for type_ in union_ga: with self.subTest(f"check isinstance/issubclass is invalid for {type_}"): with self.assertRaises(TypeError): diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-04-17-11-03-45.gh-issue-91603.hYw1Lv.rst b/Misc/NEWS.d/next/Core and Builtins/2022-04-17-11-03-45.gh-issue-91603.hYw1Lv.rst new file mode 100644 index 0000000000000..957bd5ea09b58 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2022-04-17-11-03-45.gh-issue-91603.hYw1Lv.rst @@ -0,0 +1,2 @@ +Speed up :func:`isinstance` and :func:`issubclass` checks for :class:`types.UnionType`. +Patch by Yurii Karabas. diff --git a/Objects/abstract.c b/Objects/abstract.c index 79f5a5f760f8e..cfb0edcab0e5d 100644 --- a/Objects/abstract.c +++ b/Objects/abstract.c @@ -2625,6 +2625,10 @@ object_recursive_isinstance(PyThreadState *tstate, PyObject *inst, PyObject *cls return object_isinstance(inst, cls); } + if (_PyUnion_Check(cls)) { + cls = _Py_union_args(cls); + } + if (PyTuple_Check(cls)) { /* Not a general sequence -- that opens up the road to recursion and stack overflow. */ @@ -2714,6 +2718,10 @@ object_issubclass(PyThreadState *tstate, PyObject *derived, PyObject *cls) return recursive_issubclass(derived, cls); } + if (_PyUnion_Check(cls)) { + cls = _Py_union_args(cls); + } + if (PyTuple_Check(cls)) { if (_Py_EnterRecursiveCall(tstate, " in __subclasscheck__")) { diff --git a/Objects/unionobject.c b/Objects/unionobject.c index 36b032c0c5c12..5eee27c08fa7e 100644 --- a/Objects/unionobject.c +++ b/Objects/unionobject.c @@ -48,73 +48,6 @@ union_hash(PyObject *self) return hash; } -static int -is_generic_alias_in_args(PyObject *args) -{ - Py_ssize_t nargs = PyTuple_GET_SIZE(args); - for (Py_ssize_t iarg = 0; iarg < nargs; iarg++) { - PyObject *arg = PyTuple_GET_ITEM(args, iarg); - if (_PyGenericAlias_Check(arg)) { - return 0; - } - } - return 1; -} - -static PyObject * -union_instancecheck(PyObject *self, PyObject *instance) -{ - unionobject *alias = (unionobject *) self; - Py_ssize_t nargs = PyTuple_GET_SIZE(alias->args); - if (!is_generic_alias_in_args(alias->args)) { - PyErr_SetString(PyExc_TypeError, - "isinstance() argument 2 cannot contain a parameterized generic"); - return NULL; - } - for (Py_ssize_t iarg = 0; iarg < nargs; iarg++) { - PyObject *arg = PyTuple_GET_ITEM(alias->args, iarg); - if (PyType_Check(arg)) { - int res = PyObject_IsInstance(instance, arg); - if (res < 0) { - return NULL; - } - if (res) { - Py_RETURN_TRUE; - } - } - } - Py_RETURN_FALSE; -} - -static PyObject * -union_subclasscheck(PyObject *self, PyObject *instance) -{ - if (!PyType_Check(instance)) { - PyErr_SetString(PyExc_TypeError, "issubclass() arg 1 must be a class"); - return NULL; - } - unionobject *alias = (unionobject *)self; - if (!is_generic_alias_in_args(alias->args)) { - PyErr_SetString(PyExc_TypeError, - "issubclass() argument 2 cannot contain a parameterized generic"); - return NULL; - } - Py_ssize_t nargs = PyTuple_GET_SIZE(alias->args); - for (Py_ssize_t iarg = 0; iarg < nargs; iarg++) { - PyObject *arg = PyTuple_GET_ITEM(alias->args, iarg); - if (PyType_Check(arg)) { - int res = PyObject_IsSubclass(instance, arg); - if (res < 0) { - return NULL; - } - if (res) { - Py_RETURN_TRUE; - } - } - } - Py_RETURN_FALSE; -} - static PyObject * union_richcompare(PyObject *a, PyObject *b, int op) { @@ -342,12 +275,6 @@ static PyMemberDef union_members[] = { {0} }; -static PyMethodDef union_methods[] = { - {"__instancecheck__", union_instancecheck, METH_O}, - {"__subclasscheck__", union_subclasscheck, METH_O}, - {0}}; - - static PyObject * union_getitem(PyObject *self, PyObject *item) { @@ -434,6 +361,13 @@ union_getattro(PyObject *self, PyObject *name) return PyObject_GenericGetAttr(self, name); } +PyObject * +_Py_union_args(PyObject *self) +{ + assert(_PyUnion_Check(self)); + return ((unionobject *) self)->args; +} + PyTypeObject _PyUnion_Type = { PyVarObject_HEAD_INIT(&PyType_Type, 0) .tp_name = "types.UnionType", @@ -449,7 +383,6 @@ PyTypeObject _PyUnion_Type = { .tp_hash = union_hash, .tp_getattro = union_getattro, .tp_members = union_members, - .tp_methods = union_methods, .tp_richcompare = union_richcompare, .tp_as_mapping = &union_as_mapping, .tp_as_number = &union_as_number, From webhook-mailer at python.org Thu Apr 28 11:27:27 2022 From: webhook-mailer at python.org (miss-islington) Date: Thu, 28 Apr 2022 15:27:27 -0000 Subject: [Python-checkins] gh-91832: Add 'required' attr to argparse.Action repr (GH-91841) Message-ID: https://github.com/python/cpython/commit/11652ceccf1dbf9dd332ad52ac9bd41b4adff274 commit: 11652ceccf1dbf9dd332ad52ac9bd41b4adff274 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-28T08:27:17-07:00 summary: gh-91832: Add 'required' attr to argparse.Action repr (GH-91841) GH- Adding 'required' to names in Lib.argparse.Action gh-91832: Added 'required' to the list `names` in `Lib.argparse.Action`. Changed constant strings that test the Action object. Automerge-Triggered-By: GH:merwok (cherry picked from commit 4ed3900041c688a02dca1eb3323083d720dd0d93) Co-authored-by: Abhigyan Bose files: A Misc/NEWS.d/next/Library/2022-04-23-03-24-00.gh-issue-91832.TyLi65.rst M Lib/argparse.py M Lib/test/test_argparse.py diff --git a/Lib/argparse.py b/Lib/argparse.py index 2c0dd853a5bdd..9be18488abe5b 100644 --- a/Lib/argparse.py +++ b/Lib/argparse.py @@ -848,6 +848,7 @@ def _get_kwargs(self): 'default', 'type', 'choices', + 'required', 'help', 'metavar', ] diff --git a/Lib/test/test_argparse.py b/Lib/test/test_argparse.py index d4426bab5e09a..0b237ab5b9302 100644 --- a/Lib/test/test_argparse.py +++ b/Lib/test/test_argparse.py @@ -4873,12 +4873,13 @@ def test_optional(self): nargs='+', default=42, choices=[1, 2, 3], + required=False, help='HELP', metavar='METAVAR') string = ( "Action(option_strings=['--foo', '-a', '-b'], dest='b', " "nargs='+', const=None, default=42, type='int', " - "choices=[1, 2, 3], help='HELP', metavar='METAVAR')") + "choices=[1, 2, 3], required=False, help='HELP', metavar='METAVAR')") self.assertStringEqual(option, string) def test_argument(self): @@ -4889,12 +4890,13 @@ def test_argument(self): nargs='?', default=2.5, choices=[0.5, 1.5, 2.5], + required=True, help='H HH H', metavar='MV MV MV') string = ( "Action(option_strings=[], dest='x', nargs='?', " "const=None, default=2.5, type=%r, choices=[0.5, 1.5, 2.5], " - "help='H HH H', metavar='MV MV MV')" % float) + "required=True, help='H HH H', metavar='MV MV MV')" % float) self.assertStringEqual(argument, string) def test_namespace(self): diff --git a/Misc/NEWS.d/next/Library/2022-04-23-03-24-00.gh-issue-91832.TyLi65.rst b/Misc/NEWS.d/next/Library/2022-04-23-03-24-00.gh-issue-91832.TyLi65.rst new file mode 100644 index 0000000000000..0ebf773546558 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-23-03-24-00.gh-issue-91832.TyLi65.rst @@ -0,0 +1 @@ +Add ``required`` attribute to :class:`argparse.Action` repr output. From webhook-mailer at python.org Thu Apr 28 15:45:35 2022 From: webhook-mailer at python.org (brettcannon) Date: Thu, 28 Apr 2022 19:45:35 -0000 Subject: [Python-checkins] bpo-22276: Change pathlib.Path.glob not to ignore trailing path separator (GH-10349) Message-ID: https://github.com/python/cpython/commit/ea2f5bcda1a392804487e6883be89fbad38a01a5 commit: ea2f5bcda1a392804487e6883be89fbad38a01a5 branch: main author: Eisuke Kawashima committer: brettcannon date: 2022-04-28T12:45:03-07:00 summary: bpo-22276: Change pathlib.Path.glob not to ignore trailing path separator (GH-10349) Now pathlib.Path.glob() **only** matches directories when the pattern ends in a path separator. Co-authored-by: Serhiy Storchaka files: A Misc/NEWS.d/next/Library/2018-11-11-04-41-11.bpo-22276.Tt19TW.rst M Doc/library/pathlib.rst M Doc/whatsnew/3.11.rst M Lib/pathlib.py M Lib/test/test_pathlib.py diff --git a/Doc/library/pathlib.rst b/Doc/library/pathlib.rst index 01e9cfb93e391..ab26e2f1719fe 100644 --- a/Doc/library/pathlib.rst +++ b/Doc/library/pathlib.rst @@ -815,6 +815,9 @@ call fails (for example because the path doesn't exist). .. audit-event:: pathlib.Path.glob self,pattern pathlib.Path.glob + .. versionchanged:: 3.11 + Return only directories if *pattern* ends with a pathname components + separator (:data:`~os.sep` or :data:`~os.altsep`). .. method:: Path.group() @@ -1104,6 +1107,9 @@ call fails (for example because the path doesn't exist). .. audit-event:: pathlib.Path.rglob self,pattern pathlib.Path.rglob + .. versionchanged:: 3.11 + Return only directories if *pattern* ends with a pathname components + separator (:data:`~os.sep` or :data:`~os.altsep`). .. method:: Path.rmdir() diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index b2b98747d31a1..5f1f995a0fe2f 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -506,6 +506,15 @@ os instead of ``CryptGenRandom()`` which is deprecated. (Contributed by Dong-hee Na in :issue:`44611`.) + +pathlib +------- + +* :meth:`~pathlib.Path.glob` and :meth:`~pathlib.Path.rglob` return only + directories if *pattern* ends with a pathname components separator: + :data:`~os.sep` or :data:`~os.altsep`. + (Contributed by Eisuke Kawasima in :issue:`22276` and :issue:`33392`.) + re -- diff --git a/Lib/pathlib.py b/Lib/pathlib.py index 4763ab54f6ba8..1f098fe6bd5f3 100644 --- a/Lib/pathlib.py +++ b/Lib/pathlib.py @@ -281,6 +281,8 @@ def make_uri(self, path): def _make_selector(pattern_parts, flavour): pat = pattern_parts[0] child_parts = pattern_parts[1:] + if not pat: + return _TerminatingSelector() if pat == '**': cls = _RecursiveWildcardSelector elif '**' in pat: @@ -943,6 +945,8 @@ def glob(self, pattern): drv, root, pattern_parts = self._flavour.parse_parts((pattern,)) if drv or root: raise NotImplementedError("Non-relative patterns are unsupported") + if pattern[-1] in (self._flavour.sep, self._flavour.altsep): + pattern_parts.append('') selector = _make_selector(tuple(pattern_parts), self._flavour) for p in selector.select_from(self): yield p @@ -956,6 +960,8 @@ def rglob(self, pattern): drv, root, pattern_parts = self._flavour.parse_parts((pattern,)) if drv or root: raise NotImplementedError("Non-relative patterns are unsupported") + if pattern[-1] in (self._flavour.sep, self._flavour.altsep): + pattern_parts.append('') selector = _make_selector(("**",) + tuple(pattern_parts), self._flavour) for p in selector.select_from(self): yield p diff --git a/Lib/test/test_pathlib.py b/Lib/test/test_pathlib.py index b8b08bf0ce1bb..6737068c0ff6d 100644 --- a/Lib/test/test_pathlib.py +++ b/Lib/test/test_pathlib.py @@ -1662,6 +1662,11 @@ def _check(glob, expected): else: _check(p.glob("*/fileB"), ['dirB/fileB', 'linkB/fileB']) + if not os_helper.can_symlink(): + _check(p.glob("*/"), ["dirA", "dirB", "dirC", "dirE"]) + else: + _check(p.glob("*/"), ["dirA", "dirB", "dirC", "dirE", "linkB"]) + def test_rglob_common(self): def _check(glob, expected): self.assertEqual(set(glob), { P(BASE, q) for q in expected }) @@ -1679,6 +1684,16 @@ def _check(glob, expected): "linkB/fileB", "dirA/linkC/fileB"]) _check(p.rglob("file*"), ["fileA", "dirB/fileB", "dirC/fileC", "dirC/dirD/fileD"]) + if not os_helper.can_symlink(): + _check(p.rglob("*/"), [ + "dirA", "dirB", "dirC", "dirC/dirD", "dirE", + ]) + else: + _check(p.rglob("*/"), [ + "dirA", "dirA/linkC", "dirB", "dirB/linkD", "dirC", + "dirC/dirD", "dirE", "linkB", + ]) + p = P(BASE, "dirC") _check(p.rglob("file*"), ["dirC/fileC", "dirC/dirD/fileD"]) _check(p.rglob("*/*"), ["dirC/dirD/fileD"]) @@ -2704,6 +2719,7 @@ def test_glob(self): P = self.cls p = P(BASE) self.assertEqual(set(p.glob("FILEa")), { P(BASE, "fileA") }) + self.assertEqual(set(p.glob("*a\\")), { P(BASE, "dirA") }) self.assertEqual(set(p.glob("F*a")), { P(BASE, "fileA") }) self.assertEqual(set(map(str, p.glob("FILEa"))), {f"{p}\\FILEa"}) self.assertEqual(set(map(str, p.glob("F*a"))), {f"{p}\\fileA"}) @@ -2712,6 +2728,7 @@ def test_rglob(self): P = self.cls p = P(BASE, "dirC") self.assertEqual(set(p.rglob("FILEd")), { P(BASE, "dirC/dirD/fileD") }) + self.assertEqual(set(p.rglob("*\\")), { P(BASE, "dirC/dirD") }) self.assertEqual(set(map(str, p.rglob("FILEd"))), {f"{p}\\dirD\\FILEd"}) def test_expanduser(self): diff --git a/Misc/NEWS.d/next/Library/2018-11-11-04-41-11.bpo-22276.Tt19TW.rst b/Misc/NEWS.d/next/Library/2018-11-11-04-41-11.bpo-22276.Tt19TW.rst new file mode 100644 index 0000000000000..357c3f4ae0e6a --- /dev/null +++ b/Misc/NEWS.d/next/Library/2018-11-11-04-41-11.bpo-22276.Tt19TW.rst @@ -0,0 +1,4 @@ +:class:`~pathlib.Path` methods :meth:`~pathlib.Path.glob` and :meth:`~pathlib.Path.rglob` return only +directories if *pattern* ends with a pathname components separator +(``/`` or :data:`~os.sep`). +Patch by Eisuke Kawashima. From webhook-mailer at python.org Fri Apr 29 00:22:30 2022 From: webhook-mailer at python.org (Fidget-Spinner) Date: Fri, 29 Apr 2022 04:22:30 -0000 Subject: [Python-checkins] gh-91873: Summarise deprecations in typing at the top level (GH-91864) Message-ID: https://github.com/python/cpython/commit/64113a4ba801126028505c50a7383f3e9df29573 commit: 64113a4ba801126028505c50a7383f3e9df29573 branch: main author: Ken Jin committer: Fidget-Spinner date: 2022-04-29T12:22:25+08:00 summary: gh-91873: Summarise deprecations in typing at the top level (GH-91864) files: M Doc/library/typing.rst diff --git a/Doc/library/typing.rst b/Doc/library/typing.rst index 4635da7579ac7..d4a6f32aa36d2 100644 --- a/Doc/library/typing.rst +++ b/Doc/library/typing.rst @@ -36,6 +36,10 @@ New features are frequently added to the ``typing`` module. The `typing_extensions `_ package provides backports of these new features to older versions of Python. +For a summary of deprecated features and a deprecation timeline, please see +`Deprecation Timeline of Major Features`_. + + .. _relevant-peps: Relevant PEPs @@ -2644,3 +2648,20 @@ Constant (see :pep:`563`). .. versionadded:: 3.5.2 + +Deprecation Timeline of Major Features +====================================== + +Certain features in ``typing`` are deprecated and may be removed in a future +version of Python. The following table summarizes major deprecations for your +convenience. This is subject to change, and not all deprecations are listed. + ++----------------------------------+---------------+-------------------+----------------+ +| Feature | Deprecated in | Projected removal | PEP/issue | ++==================================+===============+===================+================+ +| ``typing.io`` and ``typing.re`` | 3.8 | 3.12 | :issue:`38291` | +| submodules | | | | ++----------------------------------+---------------+-------------------+----------------+ +| ``typing`` versions of standard | 3.9 | Undecided | :pep:`585` | +| collections | | | | ++----------------------------------+---------------+-------------------+----------------+ From webhook-mailer at python.org Fri Apr 29 09:47:36 2022 From: webhook-mailer at python.org (encukou) Date: Fri, 29 Apr 2022 13:47:36 -0000 Subject: [Python-checkins] bpo-36329: Remove 'make -C Doc serve' in favour of 'make -C Doc htmlview' (GH-32354) Message-ID: https://github.com/python/cpython/commit/89c6b2b8f615a1c1827a92c4582c213b1a5027fb commit: 89c6b2b8f615a1c1827a92c4582c213b1a5027fb branch: main author: Hugo van Kemenade committer: encukou date: 2022-04-29T15:47:26+02:00 summary: bpo-36329: Remove 'make -C Doc serve' in favour of 'make -C Doc htmlview' (GH-32354) Also updated `make -C htmlview` so it used a full path with `file://`, because the original didn't open the page (macOS). For example: ```sh cd Doc # Doesn't open anything: python3 -c "import webbrowser; webbrowser.open('build/html/index.html')" # Opens the docs page e.g. file:///Users/hugo/github/cpython/Doc/build/html/index.html : python3 -c "import os, webbrowser; webbrowser.open('file://' + os.path.realpath('build/html/index.html'))" ``` https://bugs.python.org/issue36329 files: A Misc/NEWS.d/next/Documentation/2022-04-06-11-53-41.bpo-36329.EVtAtK.rst D Tools/scripts/serve.py M Doc/Makefile M Doc/library/wsgiref.rst M Doc/make.bat M Tools/scripts/README diff --git a/Doc/Makefile b/Doc/Makefile index 61a7ce0d0981f..3a3417bf99af3 100644 --- a/Doc/Makefile +++ b/Doc/Makefile @@ -13,7 +13,6 @@ PAPER = SOURCES = DISTVERSION = $(shell $(PYTHON) tools/extensions/patchlevel.py) SPHINXERRORHANDLING = -W -SERVE_PORT = # Internal variables. PAPEROPT_a4 = -D latex_elements.papersize=a4paper @@ -45,7 +44,6 @@ help: @echo " dist to create a \"dist\" directory with archived docs for download" @echo " suspicious to check for suspicious markup in output text" @echo " check to run a check for frequent markup errors" - @echo " serve to serve the documentation on the localhost (8000)" build: -mkdir -p build @@ -141,7 +139,7 @@ pydoc-topics: build "cp build/pydoc-topics/topics.py ../Lib/pydoc_data/topics.py" htmlview: html - $(PYTHON) -c "import webbrowser; webbrowser.open('build/html/index.html')" + $(PYTHON) -c "import os, webbrowser; webbrowser.open('file://' + os.path.realpath('build/html/index.html'))" clean: clean-venv -rm -rf build/* @@ -219,7 +217,7 @@ check: $(SPHINXLINT) ../Misc/NEWS.d/next/ serve: - $(PYTHON) ../Tools/scripts/serve.py build/html $(SERVE_PORT) + @echo "The serve target was removed, use htmlview instead (see bpo-36329)" # Targets for daily automated doc build # By default, Sphinx only rebuilds pages where the page content has changed. diff --git a/Doc/library/wsgiref.rst b/Doc/library/wsgiref.rst index 0ca7b003bf4dd..6a2d47891b473 100644 --- a/Doc/library/wsgiref.rst +++ b/Doc/library/wsgiref.rst @@ -813,30 +813,76 @@ Examples This is a working "Hello World" WSGI application:: + """ + Every WSGI application must have an application object - a callable + object that accepts two arguments. For that purpose, we're going to + use a function (note that you're not limited to a function, you can + use a class for example). The first argument passed to the function + is a dictionary containing CGI-style environment variables and the + second variable is the callable object. + """ from wsgiref.simple_server import make_server - # Every WSGI application must have an application object - a callable - # object that accepts two arguments. For that purpose, we're going to - # use a function (note that you're not limited to a function, you can - # use a class for example). The first argument passed to the function - # is a dictionary containing CGI-style environment variables and the - # second variable is the callable object. + def hello_world_app(environ, start_response): - status = '200 OK' # HTTP Status - headers = [('Content-type', 'text/plain; charset=utf-8')] # HTTP Headers + status = "200 OK" # HTTP Status + headers = [("Content-type", "text/plain; charset=utf-8")] # HTTP Headers start_response(status, headers) # The returned object is going to be printed return [b"Hello World"] - with make_server('', 8000, hello_world_app) as httpd: + with make_server("", 8000, hello_world_app) as httpd: print("Serving on port 8000...") # Serve until process is killed httpd.serve_forever() + Example of a WSGI application serving the current directory, accept optional -directory and port number (default: 8000) on the command line: +directory and port number (default: 8000) on the command line:: + + """ + Small wsgiref based web server. Takes a path to serve from and an + optional port number (defaults to 8000), then tries to serve files. + MIME types are guessed from the file names, 404 errors are raised + if the file is not found. + """ + import mimetypes + import os + import sys + from wsgiref import simple_server, util + + + def app(environ, respond): + # Get the file name and MIME type + fn = os.path.join(path, environ["PATH_INFO"][1:]) + if "." not in fn.split(os.path.sep)[-1]: + fn = os.path.join(fn, "index.html") + mime_type = mimetypes.guess_type(fn)[0] + + # Return 200 OK if file exists, otherwise 404 Not Found + if os.path.exists(fn): + respond("200 OK", [("Content-Type", mime_type)]) + return util.FileWrapper(open(fn, "rb")) + else: + respond("404 Not Found", [("Content-Type", "text/plain")]) + return [b"not found"] + + + if __name__ == "__main__": + # Get the path and port from command-line arguments + path = sys.argv[1] if len(sys.argv) > 1 else os.getcwd() + port = int(sys.argv[2]) if len(sys.argv) > 2 else 8000 + + # Make and start the server until control-c + httpd = simple_server.make_server("", port, app) + print(f"Serving {path} on port {port}, control-C to stop") + try: + httpd.serve_forever() + except KeyboardInterrupt: + print("Shutting down.") + httpd.server_close() + -.. literalinclude:: ../../Tools/scripts/serve.py diff --git a/Doc/make.bat b/Doc/make.bat index f3e9b44d61d4a..d9a7aa4ca7fa6 100644 --- a/Doc/make.bat +++ b/Doc/make.bat @@ -111,7 +111,7 @@ echo. Provided by Sphinx: echo. html, htmlhelp, latex, text echo. suspicious, linkcheck, changes, doctest echo. Provided by this script: -echo. clean, check, serve, htmlview +echo. clean, check, htmlview echo. echo.All arguments past the first one are passed through to sphinx-build as echo.filenames to build or are ignored. See README.rst in this directory or @@ -184,7 +184,7 @@ cmd /S /C "%SPHINXLINT% -i tools" goto end :serve -cmd /S /C "%PYTHON% ..\Tools\scripts\serve.py "%BUILDDIR%\html"" +echo.The serve target was removed, use htmlview instead (see bpo-36329) goto end :end diff --git a/Misc/NEWS.d/next/Documentation/2022-04-06-11-53-41.bpo-36329.EVtAtK.rst b/Misc/NEWS.d/next/Documentation/2022-04-06-11-53-41.bpo-36329.EVtAtK.rst new file mode 100644 index 0000000000000..67398de51aed2 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2022-04-06-11-53-41.bpo-36329.EVtAtK.rst @@ -0,0 +1 @@ +Remove 'make -C Doc serve' in favour of 'make -C Doc htmlview' diff --git a/Tools/scripts/README b/Tools/scripts/README index ba0f662c45f0c..c1d66731ba649 100644 --- a/Tools/scripts/README +++ b/Tools/scripts/README @@ -57,7 +57,6 @@ reindent.py Change .py files to use 4-space indents reindent-rst.py Fix-up reStructuredText file whitespace rgrep.py Reverse grep through a file (useful for big logfiles) run_tests.py Run the test suite with more sensible default options -serve.py Small wsgiref-based web server, used in make serve in Doc stable_abi.py Stable ABI checks and file generators. suff.py Sort a list of files by suffix texi2html.py Convert GNU texinfo files into HTML diff --git a/Tools/scripts/serve.py b/Tools/scripts/serve.py deleted file mode 100755 index 7ac9c10507832..0000000000000 --- a/Tools/scripts/serve.py +++ /dev/null @@ -1,36 +0,0 @@ -#!/usr/bin/env python3 -''' -Small wsgiref based web server. Takes a path to serve from and an -optional port number (defaults to 8000), then tries to serve files. -Mime types are guessed from the file names, 404 errors are raised -if the file is not found. Used for the make serve target in Doc. -''' -import sys -import os -import mimetypes -from wsgiref import simple_server, util - -def app(environ, respond): - - fn = os.path.join(path, environ['PATH_INFO'][1:]) - if '.' not in fn.split(os.path.sep)[-1]: - fn = os.path.join(fn, 'index.html') - type = mimetypes.guess_type(fn)[0] - - if os.path.exists(fn): - respond('200 OK', [('Content-Type', type)]) - return util.FileWrapper(open(fn, "rb")) - else: - respond('404 Not Found', [('Content-Type', 'text/plain')]) - return [b'not found'] - -if __name__ == '__main__': - path = sys.argv[1] if len(sys.argv) > 1 else os.getcwd() - port = int(sys.argv[2]) if len(sys.argv) > 2 else 8000 - httpd = simple_server.make_server('', port, app) - print("Serving {} on port {}, control-C to stop".format(path, port)) - try: - httpd.serve_forever() - except KeyboardInterrupt: - print("Shutting down.") - httpd.server_close() From webhook-mailer at python.org Fri Apr 29 10:18:18 2022 From: webhook-mailer at python.org (encukou) Date: Fri, 29 Apr 2022 14:18:18 -0000 Subject: [Python-checkins] gh-91324: Convert the stable ABI manifest to TOML (GH-92026) Message-ID: https://github.com/python/cpython/commit/83bce8ef14ee5546cb3d60f398f0cbb04bd3d9df commit: 83bce8ef14ee5546cb3d60f398f0cbb04bd3d9df branch: main author: Petr Viktorin committer: encukou date: 2022-04-29T16:18:08+02:00 summary: gh-91324: Convert the stable ABI manifest to TOML (GH-92026) files: A Misc/stable_abi.toml D Misc/stable_abi.txt M Lib/test/test_stable_abi_ctypes.py M Makefile.pre.in M Tools/scripts/stable_abi.py diff --git a/Lib/test/test_stable_abi_ctypes.py b/Lib/test/test_stable_abi_ctypes.py index 311e216e5066c8..18c85061ca0893 100644 --- a/Lib/test/test_stable_abi_ctypes.py +++ b/Lib/test/test_stable_abi_ctypes.py @@ -20,7 +20,8 @@ def test_available_symbols(self): ctypes_test.pythonapi[symbol_name] def test_feature_macros(self): - self.assertEqual(set(get_feature_macros()), EXPECTED_IFDEFS) + self.assertEqual( + set(get_feature_macros()), EXPECTED_FEATURE_MACROS) # The feature macros for Windows are used in creating the DLL # definition, so they must be known on all platforms. @@ -28,7 +29,7 @@ def test_feature_macros(self): # the reality. @unittest.skipIf(sys.platform != "win32", "Windows specific test") def test_windows_feature_macros(self): - for name, value in WINDOWS_IFDEFS.items(): + for name, value in WINDOWS_FEATURE_MACROS.items(): if value != 'maybe': with self.subTest(name): self.assertEqual(feature_macros[name], value) @@ -909,5 +910,13 @@ def test_windows_feature_macros(self): '_Py_RefTotal', ) -EXPECTED_IFDEFS = set(['HAVE_FORK', 'MS_WINDOWS', 'PY_HAVE_THREAD_NATIVE_ID', 'Py_REF_DEBUG', 'USE_STACKCHECK']) -WINDOWS_IFDEFS = {'MS_WINDOWS': True, 'HAVE_FORK': False, 'USE_STACKCHECK': 'maybe', 'PY_HAVE_THREAD_NATIVE_ID': True, 'Py_REF_DEBUG': 'maybe'} +EXPECTED_FEATURE_MACROS = set(['HAVE_FORK', + 'MS_WINDOWS', + 'PY_HAVE_THREAD_NATIVE_ID', + 'Py_REF_DEBUG', + 'USE_STACKCHECK']) +WINDOWS_FEATURE_MACROS = {'HAVE_FORK': False, + 'MS_WINDOWS': True, + 'PY_HAVE_THREAD_NATIVE_ID': True, + 'Py_REF_DEBUG': 'maybe', + 'USE_STACKCHECK': 'maybe'} diff --git a/Makefile.pre.in b/Makefile.pre.in index c12305aed6a118..e45d4fe3ecb6ea 100644 --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -1199,7 +1199,7 @@ regen-global-objects: $(srcdir)/Tools/scripts/generate_global_objects.py # ABI regen-limited-abi: all - $(RUNSHARED) ./$(BUILDPYTHON) $(srcdir)/Tools/scripts/stable_abi.py --generate-all $(srcdir)/Misc/stable_abi.txt + $(RUNSHARED) ./$(BUILDPYTHON) $(srcdir)/Tools/scripts/stable_abi.py --generate-all $(srcdir)/Misc/stable_abi.toml ############################################################################ # Regenerate all generated files @@ -2476,7 +2476,7 @@ patchcheck: @DEF_MAKE_RULE@ $(RUNSHARED) ./$(BUILDPYTHON) $(srcdir)/Tools/scripts/patchcheck.py check-limited-abi: all - $(RUNSHARED) ./$(BUILDPYTHON) $(srcdir)/Tools/scripts/stable_abi.py --all $(srcdir)/Misc/stable_abi.txt + $(RUNSHARED) ./$(BUILDPYTHON) $(srcdir)/Tools/scripts/stable_abi.py --all $(srcdir)/Misc/stable_abi.toml .PHONY: update-config update-config: diff --git a/Misc/stable_abi.toml b/Misc/stable_abi.toml new file mode 100644 index 00000000000000..e34bfcd0b61733 --- /dev/null +++ b/Misc/stable_abi.toml @@ -0,0 +1,2275 @@ +# This file lists the contents of the Limited API and Stable ABI. +# Please append new items at the end. + +# The syntax of this file is not fixed. +# It is designed to be read only by Tools/stable_abi.py, which can change +# without notice. + +# For the history of the stable ABI prior to this file, +# see the history of PC/python3dll.c and before that, PC/python3.def, +# and PC/pythonXYstub.def + + +# The current format is TOML. + +# There are these kinds of top-level "items": +# - struct: A C struct. Currently this file does not distinguish between: +# - opaque structs, which the Limited API only handles via pointers +# (so these can change at any time) +# - structs where only certain members are part of the stable ABI (e.g. +# PyObject) +# - structs which must not be changed at all (e.g. PyType_Slot, which is +# fully defined and used in arrays) +# - function: A function that must be kept available (and exported, i.e. not +# converted to a macro). +# - const: A simple value, defined with `#define`. +# - macro: A preprocessor macro more complex than a simple `const` value. +# - data: An exported object, which must continue to be available but its exact +# value may change. +# - typedef: A C typedef which is used in other definitions in the limited API. +# Its size/layout/signature must not change. +# - feature_macro: Other items may be conditional on whether this macro +# is defined or not. + +# Each top-level item can have details defined for it: +# - added: The version in which the item was added to the stable ABI. +# - ifdef: The item is only available if the given feature_macro is defined. +# - abi_only: If present, the item is not part of the Limited API, but it *is* +# part of the stable ABI. The item will not show up in user-facing docs. +# Typically used for: +# - private functions called by public macros, e.g. _Py_BuildValue_SizeT +# - items that were part of the limited API in the past, and must remain part +# of the stable ABI. +# - a combination of the above (functions that were called by macros that +# were public in the past) +# - doc: for `feature_macro`, the blurb added in documentation +# - windows: for `feature_macro`, this macro is defined on Windows. +# (This info is used to generate the DLL manifest and needs to be available +# on all platforms.) + +# Removing items from this file is generally not allowed, and additions should +# be considered with that in mind. See the devguide for exact rules: +# https://devguide.python.org/c-api/#limited-api + +# User-facing docs are at: +# https://docs.python.org/3/c-api/stable.html#stable + + +# Feature macros for optional functionality: + +[feature_macro.MS_WINDOWS] + doc = 'on Windows' + windows = true +[feature_macro.HAVE_FORK] + doc = 'on platforms with fork()' +[feature_macro.USE_STACKCHECK] + doc = 'on platforms with USE_STACKCHECK' + windows = 'maybe' +[feature_macro.PY_HAVE_THREAD_NATIVE_ID] + doc = 'on platforms with native thread IDs' + windows = true +[feature_macro.Py_REF_DEBUG] + doc = 'when Python is compiled in debug mode (with Py_REF_DEBUG)' + windows = 'maybe' + + +# Mentioned in PEP 384: + +[struct.PyObject] + added = '3.2' + members = ['ob_refcnt', 'ob_type'] + struct_abi_kind = 'members' +[struct.PyVarObject] + added = '3.2' + members = ['ob_base', 'ob_size'] + struct_abi_kind = 'members' +[struct.PyMethodDef] + added = '3.2' + struct_abi_kind = 'full-abi' +[struct.PyMemberDef] + added = '3.2' + struct_abi_kind = 'full-abi' +[struct.PyGetSetDef] + added = '3.2' + struct_abi_kind = 'full-abi' +[struct.PyModuleDef_Base] + added = '3.2' + struct_abi_kind = 'full-abi' +[struct.PyModuleDef] + added = '3.2' + struct_abi_kind = 'full-abi' +[struct.PyStructSequence_Field] + added = '3.2' + struct_abi_kind = 'full-abi' +[struct.PyStructSequence_Desc] + added = '3.2' + struct_abi_kind = 'full-abi' +[struct.PyType_Slot] + added = '3.2' + struct_abi_kind = 'full-abi' +[struct.PyType_Spec] + added = '3.2' + struct_abi_kind = 'full-abi' +[struct.PyThreadState] + added = '3.2' + struct_abi_kind = 'opaque' +[struct.PyInterpreterState] + added = '3.2' + struct_abi_kind = 'opaque' +[struct.PyFrameObject] + added = '3.2' + struct_abi_kind = 'opaque' +[struct.symtable] + added = '3.2' + struct_abi_kind = 'opaque' +[struct.PyWeakReference] + added = '3.2' + struct_abi_kind = 'opaque' +[struct.PyLongObject] + added = '3.2' + struct_abi_kind = 'opaque' +[struct.PyTypeObject] + added = '3.2' + struct_abi_kind = 'opaque' + +[function.PyType_FromSpec] + added = '3.2' + +[const.Py_tp_dealloc] + added = '3.2' +[const.Py_tp_getattr] + added = '3.2' +[const.Py_tp_setattr] + added = '3.2' +[const.Py_tp_repr] + added = '3.2' +[const.Py_tp_hash] + added = '3.2' +[const.Py_tp_call] + added = '3.2' +[const.Py_tp_str] + added = '3.2' +[const.Py_tp_getattro] + added = '3.2' +[const.Py_tp_setattro] + added = '3.2' +[const.Py_tp_doc] + added = '3.2' +[const.Py_tp_traverse] + added = '3.2' +[const.Py_tp_clear] + added = '3.2' +[const.Py_tp_richcompare] + added = '3.2' +[const.Py_tp_iter] + added = '3.2' +[const.Py_tp_iternext] + added = '3.2' +[const.Py_tp_methods] + added = '3.2' +[const.Py_tp_base] + added = '3.2' +[const.Py_tp_descr_get] + added = '3.2' +[const.Py_tp_descr_set] + added = '3.2' +[const.Py_tp_init] + added = '3.2' +[const.Py_tp_alloc] + added = '3.2' +[const.Py_tp_new] + added = '3.2' +[const.Py_tp_members] + added = '3.2' +[const.Py_tp_getset] + added = '3.2' +[const.Py_tp_free] + added = '3.2' +[const.Py_tp_is_gc] + added = '3.2' +[const.Py_tp_bases] + added = '3.2' +[const.Py_tp_del] + added = '3.2' +[const.Py_nb_add] + added = '3.2' +[const.Py_nb_subtract] + added = '3.2' +[const.Py_nb_multiply] + added = '3.2' +[const.Py_nb_remainder] + added = '3.2' +[const.Py_nb_divmod] + added = '3.2' +[const.Py_nb_power] + added = '3.2' +[const.Py_nb_negative] + added = '3.2' +[const.Py_nb_positive] + added = '3.2' +[const.Py_nb_absolute] + added = '3.2' +[const.Py_nb_bool] + added = '3.2' +[const.Py_nb_invert] + added = '3.2' +[const.Py_nb_lshift] + added = '3.2' +[const.Py_nb_rshift] + added = '3.2' +[const.Py_nb_and] + added = '3.2' +[const.Py_nb_xor] + added = '3.2' +[const.Py_nb_or] + added = '3.2' +[const.Py_nb_int] + added = '3.2' +[const.Py_nb_float] + added = '3.2' +[const.Py_nb_inplace_add] + added = '3.2' +[const.Py_nb_inplace_subtract] + added = '3.2' +[const.Py_nb_inplace_multiply] + added = '3.2' +[const.Py_nb_inplace_remainder] + added = '3.2' +[const.Py_nb_inplace_power] + added = '3.2' +[const.Py_nb_inplace_lshift] + added = '3.2' +[const.Py_nb_inplace_rshift] + added = '3.2' +[const.Py_nb_inplace_and] + added = '3.2' +[const.Py_nb_inplace_xor] + added = '3.2' +[const.Py_nb_inplace_or] + added = '3.2' +[const.Py_nb_floor_divide] + added = '3.2' +[const.Py_nb_true_divide] + added = '3.2' +[const.Py_nb_inplace_floor_divide] + added = '3.2' +[const.Py_nb_inplace_true_divide] + added = '3.2' +[const.Py_nb_index] + added = '3.2' +[const.Py_sq_length] + added = '3.2' +[const.Py_sq_concat] + added = '3.2' +[const.Py_sq_repeat] + added = '3.2' +[const.Py_sq_item] + added = '3.2' +[const.Py_sq_ass_item] + added = '3.2' +[const.Py_sq_contains] + added = '3.2' +[const.Py_sq_inplace_concat] + added = '3.2' +[const.Py_sq_inplace_repeat] + added = '3.2' +[const.Py_mp_length] + added = '3.2' +[const.Py_mp_subscript] + added = '3.2' +[const.Py_mp_ass_subscript] + added = '3.2' + +[typedef.Py_uintptr_t] + added = '3.2' +[typedef.Py_intptr_t] + added = '3.2' +[typedef.Py_ssize_t] + added = '3.2' +[typedef.unaryfunc] + added = '3.2' +[typedef.binaryfunc] + added = '3.2' +[typedef.ternaryfunc] + added = '3.2' +[typedef.inquiry] + added = '3.2' +[typedef.lenfunc] + added = '3.2' +[typedef.ssizeargfunc] + added = '3.2' +[typedef.ssizessizeargfunc] + added = '3.2' +[typedef.ssizeobjargproc] + added = '3.2' +[typedef.ssizessizeobjargproc] + added = '3.2' +[typedef.objobjargproc] + added = '3.2' +[typedef.objobjproc] + added = '3.2' +[typedef.visitproc] + added = '3.2' +[typedef.traverseproc] + added = '3.2' +[typedef.destructor] + added = '3.2' +[typedef.getattrfunc] + added = '3.2' +[typedef.getattrofunc] + added = '3.2' +[typedef.setattrfunc] + added = '3.2' +[typedef.setattrofunc] + added = '3.2' +[typedef.reprfunc] + added = '3.2' +[typedef.hashfunc] + added = '3.2' +[typedef.richcmpfunc] + added = '3.2' +[typedef.getiterfunc] + added = '3.2' +[typedef.iternextfunc] + added = '3.2' +[typedef.descrgetfunc] + added = '3.2' +[typedef.descrsetfunc] + added = '3.2' +[typedef.initproc] + added = '3.2' +[typedef.newfunc] + added = '3.2' +[typedef.allocfunc] + added = '3.2' +[typedef.PyCFunction] + added = '3.2' +[typedef.PyCFunctionWithKeywords] + added = '3.2' +[typedef.PyCapsule_Destructor] + added = '3.2' +[typedef.getter] + added = '3.2' +[typedef.setter] + added = '3.2' +[typedef.PyOS_sighandler_t] + added = '3.2' +[typedef.PyGILState_STATE] + added = '3.2' +[typedef.Py_UCS4] + added = '3.2' + +[macro.Py_BEGIN_ALLOW_THREADS] + added = '3.2' +[macro.Py_BLOCK_THREADS] + added = '3.2' +[macro.Py_UNBLOCK_THREADS] + added = '3.2' +[macro.Py_END_ALLOW_THREADS] + added = '3.2' + +# The following were added in PC/python3.def in the initial stable ABI commit, +# 4d0d471a8031de90a2b1ce99c4ac4780e60b3bc9, +# and later amendments in 3.2: +# 0d012f284be829c6217f60523db0e1671b7db9d9 +# c83bc3c1fbed14d27a5de3032e24d2cf006a7c4b + +[function.PyArg_Parse] + added = '3.2' +[function.PyArg_ParseTuple] + added = '3.2' +[function.PyArg_ParseTupleAndKeywords] + added = '3.2' +[function.PyArg_UnpackTuple] + added = '3.2' +[function.PyArg_VaParse] + added = '3.2' +[function.PyArg_VaParseTupleAndKeywords] + added = '3.2' +[function.PyArg_ValidateKeywordArguments] + added = '3.2' +[data.PyBaseObject_Type] + added = '3.2' +[function.PyBool_FromLong] + added = '3.2' +[data.PyBool_Type] + added = '3.2' +[data.PyByteArrayIter_Type] + added = '3.2' +[function.PyByteArray_AsString] + added = '3.2' +[function.PyByteArray_Concat] + added = '3.2' +[function.PyByteArray_FromObject] + added = '3.2' +[function.PyByteArray_FromStringAndSize] + added = '3.2' +[function.PyByteArray_Resize] + added = '3.2' +[function.PyByteArray_Size] + added = '3.2' +[data.PyByteArray_Type] + added = '3.2' +[data.PyBytesIter_Type] + added = '3.2' +[function.PyBytes_AsString] + added = '3.2' +[function.PyBytes_AsStringAndSize] + added = '3.2' +[function.PyBytes_Concat] + added = '3.2' +[function.PyBytes_ConcatAndDel] + added = '3.2' +[function.PyBytes_DecodeEscape] + added = '3.2' +[function.PyBytes_FromFormat] + added = '3.2' +[function.PyBytes_FromFormatV] + added = '3.2' +[function.PyBytes_FromObject] + added = '3.2' +[function.PyBytes_FromString] + added = '3.2' +[function.PyBytes_FromStringAndSize] + added = '3.2' +[function.PyBytes_Repr] + added = '3.2' +[function.PyBytes_Size] + added = '3.2' +[data.PyBytes_Type] + added = '3.2' +[function.PyCFunction_Call] + added = '3.2' +[function.PyCFunction_GetFlags] + added = '3.2' +[function.PyCFunction_GetFunction] + added = '3.2' +[function.PyCFunction_GetSelf] + added = '3.2' +[function.PyCFunction_NewEx] + added = '3.2' +[data.PyCFunction_Type] + added = '3.2' +[function.PyCallIter_New] + added = '3.2' +[data.PyCallIter_Type] + added = '3.2' +[function.PyCallable_Check] + added = '3.2' +[function.PyCapsule_GetContext] + added = '3.2' +[function.PyCapsule_GetDestructor] + added = '3.2' +[function.PyCapsule_GetName] + added = '3.2' +[function.PyCapsule_GetPointer] + added = '3.2' +[function.PyCapsule_Import] + added = '3.2' +[function.PyCapsule_IsValid] + added = '3.2' +[function.PyCapsule_New] + added = '3.2' +[function.PyCapsule_SetContext] + added = '3.2' +[function.PyCapsule_SetDestructor] + added = '3.2' +[function.PyCapsule_SetName] + added = '3.2' +[function.PyCapsule_SetPointer] + added = '3.2' +[data.PyCapsule_Type] + added = '3.2' +[data.PyClassMethodDescr_Type] + added = '3.2' +[function.PyCodec_BackslashReplaceErrors] + added = '3.2' +[function.PyCodec_Decode] + added = '3.2' +[function.PyCodec_Decoder] + added = '3.2' +[function.PyCodec_Encode] + added = '3.2' +[function.PyCodec_Encoder] + added = '3.2' +[function.PyCodec_IgnoreErrors] + added = '3.2' +[function.PyCodec_IncrementalDecoder] + added = '3.2' +[function.PyCodec_IncrementalEncoder] + added = '3.2' +[function.PyCodec_KnownEncoding] + added = '3.2' +[function.PyCodec_LookupError] + added = '3.2' +[function.PyCodec_Register] + added = '3.2' +[function.PyCodec_RegisterError] + added = '3.2' +[function.PyCodec_ReplaceErrors] + added = '3.2' +[function.PyCodec_StreamReader] + added = '3.2' +[function.PyCodec_StreamWriter] + added = '3.2' +[function.PyCodec_StrictErrors] + added = '3.2' +[function.PyCodec_XMLCharRefReplaceErrors] + added = '3.2' +[function.PyComplex_FromDoubles] + added = '3.2' +[function.PyComplex_ImagAsDouble] + added = '3.2' +[function.PyComplex_RealAsDouble] + added = '3.2' +[data.PyComplex_Type] + added = '3.2' +[function.PyDescr_NewClassMethod] + added = '3.2' +[function.PyDescr_NewGetSet] + added = '3.2' +[function.PyDescr_NewMember] + added = '3.2' +[function.PyDescr_NewMethod] + added = '3.2' +[data.PyDictItems_Type] + added = '3.2' +[data.PyDictIterItem_Type] + added = '3.2' +[data.PyDictIterKey_Type] + added = '3.2' +[data.PyDictIterValue_Type] + added = '3.2' +[data.PyDictKeys_Type] + added = '3.2' +[function.PyDictProxy_New] + added = '3.2' +[data.PyDictProxy_Type] + added = '3.2' +[data.PyDictValues_Type] + added = '3.2' +[function.PyDict_Clear] + added = '3.2' +[function.PyDict_Contains] + added = '3.2' +[function.PyDict_Copy] + added = '3.2' +[function.PyDict_DelItem] + added = '3.2' +[function.PyDict_DelItemString] + added = '3.2' +[function.PyDict_GetItem] + added = '3.2' +[function.PyDict_GetItemString] + added = '3.2' +[function.PyDict_GetItemWithError] + added = '3.2' +[function.PyDict_Items] + added = '3.2' +[function.PyDict_Keys] + added = '3.2' +[function.PyDict_Merge] + added = '3.2' +[function.PyDict_MergeFromSeq2] + added = '3.2' +[function.PyDict_New] + added = '3.2' +[function.PyDict_Next] + added = '3.2' +[function.PyDict_SetItem] + added = '3.2' +[function.PyDict_SetItemString] + added = '3.2' +[function.PyDict_Size] + added = '3.2' +[data.PyDict_Type] + added = '3.2' +[function.PyDict_Update] + added = '3.2' +[function.PyDict_Values] + added = '3.2' +[data.PyEllipsis_Type] + added = '3.2' +[data.PyEnum_Type] + added = '3.2' +[function.PyErr_BadArgument] + added = '3.2' +[function.PyErr_BadInternalCall] + added = '3.2' +[function.PyErr_CheckSignals] + added = '3.2' +[function.PyErr_Clear] + added = '3.2' +[function.PyErr_Display] + added = '3.2' +[function.PyErr_ExceptionMatches] + added = '3.2' +[function.PyErr_Fetch] + added = '3.2' +[function.PyErr_Format] + added = '3.2' +[function.PyErr_GivenExceptionMatches] + added = '3.2' +[function.PyErr_NewException] + added = '3.2' +[function.PyErr_NewExceptionWithDoc] + added = '3.2' +[function.PyErr_NoMemory] + added = '3.2' +[function.PyErr_NormalizeException] + added = '3.2' +[function.PyErr_Occurred] + added = '3.2' +[function.PyErr_Print] + added = '3.2' +[function.PyErr_PrintEx] + added = '3.2' +[function.PyErr_ProgramText] + added = '3.2' +[function.PyErr_Restore] + added = '3.2' +[function.PyErr_SetFromErrno] + added = '3.2' +[function.PyErr_SetFromErrnoWithFilename] + added = '3.2' +[function.PyErr_SetFromErrnoWithFilenameObject] + added = '3.2' +[function.PyErr_SetInterrupt] + added = '3.2' +[function.PyErr_SetNone] + added = '3.2' +[function.PyErr_SetObject] + added = '3.2' +[function.PyErr_SetString] + added = '3.2' +[function.PyErr_SyntaxLocation] + added = '3.2' +[function.PyErr_WarnEx] + added = '3.2' +[function.PyErr_WarnExplicit] + added = '3.2' +[function.PyErr_WarnFormat] + added = '3.2' +[function.PyErr_WriteUnraisable] + added = '3.2' +[function.PyEval_AcquireLock] + added = '3.2' +[function.PyEval_AcquireThread] + added = '3.2' +[function.PyEval_CallFunction] + added = '3.2' +[function.PyEval_CallMethod] + added = '3.2' +[function.PyEval_CallObjectWithKeywords] + added = '3.2' +[function.PyEval_EvalCode] + added = '3.2' +[function.PyEval_EvalCodeEx] + added = '3.2' +[function.PyEval_EvalFrame] + added = '3.2' +[function.PyEval_EvalFrameEx] + added = '3.2' +[function.PyEval_GetBuiltins] + added = '3.2' +[function.PyEval_GetFrame] + added = '3.2' +[function.PyEval_GetFuncDesc] + added = '3.2' +[function.PyEval_GetFuncName] + added = '3.2' +[function.PyEval_GetGlobals] + added = '3.2' +[function.PyEval_GetLocals] + added = '3.2' +[function.PyEval_InitThreads] + added = '3.2' +[function.PyEval_ReleaseLock] + added = '3.2' +[function.PyEval_ReleaseThread] + added = '3.2' +[function.PyEval_RestoreThread] + added = '3.2' +[function.PyEval_SaveThread] + added = '3.2' +[function.PyEval_ThreadsInitialized] + added = '3.2' +[data.PyExc_ArithmeticError] + added = '3.2' +[data.PyExc_AssertionError] + added = '3.2' +[data.PyExc_AttributeError] + added = '3.2' +[data.PyExc_BaseException] + added = '3.2' +[data.PyExc_BaseExceptionGroup] + added = '3.11' +[data.PyExc_BufferError] + added = '3.2' +[data.PyExc_BytesWarning] + added = '3.2' +[data.PyExc_DeprecationWarning] + added = '3.2' +[data.PyExc_EOFError] + added = '3.2' +[data.PyExc_EnvironmentError] + added = '3.2' +[data.PyExc_Exception] + added = '3.2' +[data.PyExc_FloatingPointError] + added = '3.2' +[data.PyExc_FutureWarning] + added = '3.2' +[data.PyExc_GeneratorExit] + added = '3.2' +[data.PyExc_IOError] + added = '3.2' +[data.PyExc_ImportError] + added = '3.2' +[data.PyExc_ImportWarning] + added = '3.2' +[data.PyExc_IndentationError] + added = '3.2' +[data.PyExc_IndexError] + added = '3.2' +[data.PyExc_KeyError] + added = '3.2' +[data.PyExc_KeyboardInterrupt] + added = '3.2' +[data.PyExc_LookupError] + added = '3.2' +[data.PyExc_MemoryError] + added = '3.2' +[data.PyExc_NameError] + added = '3.2' +[data.PyExc_NotImplementedError] + added = '3.2' +[data.PyExc_OSError] + added = '3.2' +[data.PyExc_OverflowError] + added = '3.2' +[data.PyExc_PendingDeprecationWarning] + added = '3.2' +[data.PyExc_ReferenceError] + added = '3.2' +[data.PyExc_RuntimeError] + added = '3.2' +[data.PyExc_RuntimeWarning] + added = '3.2' +[data.PyExc_StopIteration] + added = '3.2' +[data.PyExc_SyntaxError] + added = '3.2' +[data.PyExc_SyntaxWarning] + added = '3.2' +[data.PyExc_SystemError] + added = '3.2' +[data.PyExc_SystemExit] + added = '3.2' +[data.PyExc_TabError] + added = '3.2' +[data.PyExc_TypeError] + added = '3.2' +[data.PyExc_UnboundLocalError] + added = '3.2' +[data.PyExc_UnicodeDecodeError] + added = '3.2' +[data.PyExc_UnicodeEncodeError] + added = '3.2' +[data.PyExc_UnicodeError] + added = '3.2' +[data.PyExc_UnicodeTranslateError] + added = '3.2' +[data.PyExc_UnicodeWarning] + added = '3.2' +[data.PyExc_UserWarning] + added = '3.2' +[data.PyExc_ValueError] + added = '3.2' +[data.PyExc_Warning] + added = '3.2' +[data.PyExc_ZeroDivisionError] + added = '3.2' +[function.PyException_GetCause] + added = '3.2' +[function.PyException_GetContext] + added = '3.2' +[function.PyException_GetTraceback] + added = '3.2' +[function.PyException_SetCause] + added = '3.2' +[function.PyException_SetContext] + added = '3.2' +[function.PyException_SetTraceback] + added = '3.2' +[function.PyFile_FromFd] + added = '3.2' +[function.PyFile_GetLine] + added = '3.2' +[function.PyFile_WriteObject] + added = '3.2' +[function.PyFile_WriteString] + added = '3.2' +[data.PyFilter_Type] + added = '3.2' +[function.PyFloat_AsDouble] + added = '3.2' +[function.PyFloat_FromDouble] + added = '3.2' +[function.PyFloat_FromString] + added = '3.2' +[function.PyFloat_GetInfo] + added = '3.2' +[function.PyFloat_GetMax] + added = '3.2' +[function.PyFloat_GetMin] + added = '3.2' +[data.PyFloat_Type] + added = '3.2' +[function.PyFrozenSet_New] + added = '3.2' +[data.PyFrozenSet_Type] + added = '3.2' +[function.PyGC_Collect] + added = '3.2' +[function.PyGILState_Ensure] + added = '3.2' +[function.PyGILState_GetThisThreadState] + added = '3.2' +[function.PyGILState_Release] + added = '3.2' +[data.PyGetSetDescr_Type] + added = '3.2' +[function.PyImport_AddModule] + added = '3.2' +[function.PyImport_AppendInittab] + added = '3.2' +[function.PyImport_ExecCodeModule] + added = '3.2' +[function.PyImport_ExecCodeModuleEx] + added = '3.2' +[function.PyImport_ExecCodeModuleWithPathnames] + added = '3.2' +[function.PyImport_GetImporter] + added = '3.2' +[function.PyImport_GetMagicNumber] + added = '3.2' +[function.PyImport_GetMagicTag] + added = '3.2' +[function.PyImport_GetModuleDict] + added = '3.2' +[function.PyImport_Import] + added = '3.2' +[function.PyImport_ImportFrozenModule] + added = '3.2' +[function.PyImport_ImportModule] + added = '3.2' +[function.PyImport_ImportModuleLevel] + added = '3.2' +[function.PyImport_ImportModuleNoBlock] + added = '3.2' +[function.PyImport_ReloadModule] + added = '3.2' +[function.PyInterpreterState_Clear] + added = '3.2' +[function.PyInterpreterState_Delete] + added = '3.2' +[function.PyInterpreterState_New] + added = '3.2' +[function.PyIter_Next] + added = '3.2' +[data.PyListIter_Type] + added = '3.2' +[data.PyListRevIter_Type] + added = '3.2' +[function.PyList_Append] + added = '3.2' +[function.PyList_AsTuple] + added = '3.2' +[function.PyList_GetItem] + added = '3.2' +[function.PyList_GetSlice] + added = '3.2' +[function.PyList_Insert] + added = '3.2' +[function.PyList_New] + added = '3.2' +[function.PyList_Reverse] + added = '3.2' +[function.PyList_SetItem] + added = '3.2' +[function.PyList_SetSlice] + added = '3.2' +[function.PyList_Size] + added = '3.2' +[function.PyList_Sort] + added = '3.2' +[data.PyList_Type] + added = '3.2' +[data.PyLongRangeIter_Type] + added = '3.2' +[function.PyLong_AsDouble] + added = '3.2' +[function.PyLong_AsLong] + added = '3.2' +[function.PyLong_AsLongAndOverflow] + added = '3.2' +[function.PyLong_AsLongLong] + added = '3.2' +[function.PyLong_AsLongLongAndOverflow] + added = '3.2' +[function.PyLong_AsSize_t] + added = '3.2' +[function.PyLong_AsSsize_t] + added = '3.2' +[function.PyLong_AsUnsignedLong] + added = '3.2' +[function.PyLong_AsUnsignedLongLong] + added = '3.2' +[function.PyLong_AsUnsignedLongLongMask] + added = '3.2' +[function.PyLong_AsUnsignedLongMask] + added = '3.2' +[function.PyLong_AsVoidPtr] + added = '3.2' +[function.PyLong_FromDouble] + added = '3.2' +[function.PyLong_FromLong] + added = '3.2' +[function.PyLong_FromLongLong] + added = '3.2' +[function.PyLong_FromSize_t] + added = '3.2' +[function.PyLong_FromSsize_t] + added = '3.2' +[function.PyLong_FromString] + added = '3.2' +[function.PyLong_FromUnsignedLong] + added = '3.2' +[function.PyLong_FromUnsignedLongLong] + added = '3.2' +[function.PyLong_FromVoidPtr] + added = '3.2' +[function.PyLong_GetInfo] + added = '3.2' +[data.PyLong_Type] + added = '3.2' +[data.PyMap_Type] + added = '3.2' +[function.PyMapping_Check] + added = '3.2' +[function.PyMapping_GetItemString] + added = '3.2' +[function.PyMapping_HasKey] + added = '3.2' +[function.PyMapping_HasKeyString] + added = '3.2' +[function.PyMapping_Items] + added = '3.2' +[function.PyMapping_Keys] + added = '3.2' +[function.PyMapping_Length] + added = '3.2' +[function.PyMapping_SetItemString] + added = '3.2' +[function.PyMapping_Size] + added = '3.2' +[function.PyMapping_Values] + added = '3.2' +[function.PyMem_Free] + added = '3.2' +[function.PyMem_Malloc] + added = '3.2' +[function.PyMem_Realloc] + added = '3.2' +[data.PyMemberDescr_Type] + added = '3.2' +[function.PyMemoryView_FromObject] + added = '3.2' +[function.PyMemoryView_GetContiguous] + added = '3.2' +[data.PyMemoryView_Type] + added = '3.2' +[data.PyMethodDescr_Type] + added = '3.2' +[function.PyModule_AddIntConstant] + added = '3.2' +[function.PyModule_AddObject] + added = '3.2' +[function.PyModule_AddStringConstant] + added = '3.2' +[function.PyModule_Create2] + added = '3.2' +[function.PyModule_GetDef] + added = '3.2' +[function.PyModule_GetDict] + added = '3.2' +[function.PyModule_GetFilename] + added = '3.2' +[function.PyModule_GetFilenameObject] + added = '3.2' +[function.PyModule_GetName] + added = '3.2' +[function.PyModule_GetState] + added = '3.2' +[function.PyModule_New] + added = '3.2' +[data.PyModule_Type] + added = '3.2' +[function.PyNumber_Absolute] + added = '3.2' +[function.PyNumber_Add] + added = '3.2' +[function.PyNumber_And] + added = '3.2' +[function.PyNumber_AsSsize_t] + added = '3.2' +[function.PyNumber_Check] + added = '3.2' +[function.PyNumber_Divmod] + added = '3.2' +[function.PyNumber_Float] + added = '3.2' +[function.PyNumber_FloorDivide] + added = '3.2' +[function.PyNumber_InPlaceAdd] + added = '3.2' +[function.PyNumber_InPlaceAnd] + added = '3.2' +[function.PyNumber_InPlaceFloorDivide] + added = '3.2' +[function.PyNumber_InPlaceLshift] + added = '3.2' +[function.PyNumber_InPlaceMultiply] + added = '3.2' +[function.PyNumber_InPlaceOr] + added = '3.2' +[function.PyNumber_InPlacePower] + added = '3.2' +[function.PyNumber_InPlaceRemainder] + added = '3.2' +[function.PyNumber_InPlaceRshift] + added = '3.2' +[function.PyNumber_InPlaceSubtract] + added = '3.2' +[function.PyNumber_InPlaceTrueDivide] + added = '3.2' +[function.PyNumber_InPlaceXor] + added = '3.2' +[function.PyNumber_Index] + added = '3.2' +[function.PyNumber_Invert] + added = '3.2' +[function.PyNumber_Long] + added = '3.2' +[function.PyNumber_Lshift] + added = '3.2' +[function.PyNumber_Multiply] + added = '3.2' +[function.PyNumber_Negative] + added = '3.2' +[function.PyNumber_Or] + added = '3.2' +[function.PyNumber_Positive] + added = '3.2' +[function.PyNumber_Power] + added = '3.2' +[function.PyNumber_Remainder] + added = '3.2' +[function.PyNumber_Rshift] + added = '3.2' +[function.PyNumber_Subtract] + added = '3.2' +[function.PyNumber_ToBase] + added = '3.2' +[function.PyNumber_TrueDivide] + added = '3.2' +[function.PyNumber_Xor] + added = '3.2' +[function.PyOS_AfterFork] + added = '3.2' + ifdef = 'HAVE_FORK' +[data.PyOS_InputHook] + added = '3.2' +[function.PyOS_InterruptOccurred] + added = '3.2' +[function.PyOS_double_to_string] + added = '3.2' +[function.PyOS_getsig] + added = '3.2' +[function.PyOS_mystricmp] + added = '3.2' +[function.PyOS_mystrnicmp] + added = '3.2' +[function.PyOS_setsig] + added = '3.2' +[function.PyOS_snprintf] + added = '3.2' +[function.PyOS_string_to_double] + added = '3.2' +[function.PyOS_strtol] + added = '3.2' +[function.PyOS_strtoul] + added = '3.2' +[function.PyOS_vsnprintf] + added = '3.2' +[function.PyObject_ASCII] + added = '3.2' +[function.PyObject_AsFileDescriptor] + added = '3.2' +[function.PyObject_Bytes] + added = '3.2' +[function.PyObject_Call] + added = '3.2' +[function.PyObject_CallFunction] + added = '3.2' +[function.PyObject_CallFunctionObjArgs] + added = '3.2' +[function.PyObject_CallMethod] + added = '3.2' +[function.PyObject_CallMethodObjArgs] + added = '3.2' +[function.PyObject_CallObject] + added = '3.2' +[function.PyObject_ClearWeakRefs] + added = '3.2' +[function.PyObject_DelItem] + added = '3.2' +[function.PyObject_DelItemString] + added = '3.2' +[function.PyObject_Dir] + added = '3.2' +[function.PyObject_Format] + added = '3.2' +[function.PyObject_Free] + added = '3.2' +[function.PyObject_GC_Del] + added = '3.2' +[function.PyObject_GC_Track] + added = '3.2' +[function.PyObject_GC_UnTrack] + added = '3.2' +[function.PyObject_GenericGetAttr] + added = '3.2' +[function.PyObject_GenericSetAttr] + added = '3.2' +[function.PyObject_GetAttr] + added = '3.2' +[function.PyObject_GetAttrString] + added = '3.2' +[function.PyObject_GetItem] + added = '3.2' +[function.PyObject_GetIter] + added = '3.2' +[function.PyObject_HasAttr] + added = '3.2' +[function.PyObject_HasAttrString] + added = '3.2' +[function.PyObject_Hash] + added = '3.2' +[function.PyObject_HashNotImplemented] + added = '3.2' +[function.PyObject_Init] + added = '3.2' +[function.PyObject_InitVar] + added = '3.2' +[function.PyObject_IsInstance] + added = '3.2' +[function.PyObject_IsSubclass] + added = '3.2' +[function.PyObject_IsTrue] + added = '3.2' +[function.PyObject_Length] + added = '3.2' +[function.PyObject_Malloc] + added = '3.2' +[function.PyObject_Not] + added = '3.2' +[function.PyObject_Realloc] + added = '3.2' +[function.PyObject_Repr] + added = '3.2' +[function.PyObject_RichCompare] + added = '3.2' +[function.PyObject_RichCompareBool] + added = '3.2' +[function.PyObject_SelfIter] + added = '3.2' +[function.PyObject_SetAttr] + added = '3.2' +[function.PyObject_SetAttrString] + added = '3.2' +[function.PyObject_SetItem] + added = '3.2' +[function.PyObject_Size] + added = '3.2' +[function.PyObject_Str] + added = '3.2' +[function.PyObject_Type] + added = '3.2' +[data.PyProperty_Type] + added = '3.2' +[data.PyRangeIter_Type] + added = '3.2' +[data.PyRange_Type] + added = '3.2' +[data.PyReversed_Type] + added = '3.2' +[function.PySeqIter_New] + added = '3.2' +[data.PySeqIter_Type] + added = '3.2' +[function.PySequence_Check] + added = '3.2' +[function.PySequence_Concat] + added = '3.2' +[function.PySequence_Contains] + added = '3.2' +[function.PySequence_Count] + added = '3.2' +[function.PySequence_DelItem] + added = '3.2' +[function.PySequence_DelSlice] + added = '3.2' +[function.PySequence_Fast] + added = '3.2' +[function.PySequence_GetItem] + added = '3.2' +[function.PySequence_GetSlice] + added = '3.2' +[function.PySequence_In] + added = '3.2' +[function.PySequence_InPlaceConcat] + added = '3.2' +[function.PySequence_InPlaceRepeat] + added = '3.2' +[function.PySequence_Index] + added = '3.2' +[function.PySequence_Length] + added = '3.2' +[function.PySequence_List] + added = '3.2' +[function.PySequence_Repeat] + added = '3.2' +[function.PySequence_SetItem] + added = '3.2' +[function.PySequence_SetSlice] + added = '3.2' +[function.PySequence_Size] + added = '3.2' +[function.PySequence_Tuple] + added = '3.2' +[data.PySetIter_Type] + added = '3.2' +[function.PySet_Add] + added = '3.2' +[function.PySet_Clear] + added = '3.2' +[function.PySet_Contains] + added = '3.2' +[function.PySet_Discard] + added = '3.2' +[function.PySet_New] + added = '3.2' +[function.PySet_Pop] + added = '3.2' +[function.PySet_Size] + added = '3.2' +[data.PySet_Type] + added = '3.2' +[function.PySlice_GetIndices] + added = '3.2' +[function.PySlice_GetIndicesEx] + added = '3.2' +[function.PySlice_New] + added = '3.2' +[data.PySlice_Type] + added = '3.2' +[function.PyState_FindModule] + added = '3.2' +[function.PyStructSequence_GetItem] + added = '3.2' +[function.PyStructSequence_New] + added = '3.2' +[function.PyStructSequence_NewType] + added = '3.2' +[function.PyStructSequence_SetItem] + added = '3.2' +[data.PySuper_Type] + added = '3.2' +[function.PySys_AddWarnOption] + added = '3.2' +[function.PySys_AddWarnOptionUnicode] + added = '3.2' +[function.PySys_FormatStderr] + added = '3.2' +[function.PySys_FormatStdout] + added = '3.2' +[function.PySys_GetObject] + added = '3.2' +[function.PySys_HasWarnOptions] + added = '3.2' +[function.PySys_ResetWarnOptions] + added = '3.2' +[function.PySys_SetArgv] + added = '3.2' +[function.PySys_SetArgvEx] + added = '3.2' +[function.PySys_SetObject] + added = '3.2' +[function.PySys_SetPath] + added = '3.2' +[function.PySys_WriteStderr] + added = '3.2' +[function.PySys_WriteStdout] + added = '3.2' +[function.PyThreadState_Clear] + added = '3.2' +[function.PyThreadState_Delete] + added = '3.2' +[function.PyThreadState_DeleteCurrent] + added = '3.2' + abi_only = true +[function.PyThreadState_Get] + added = '3.2' +[function.PyThreadState_GetDict] + added = '3.2' +[function.PyThreadState_New] + added = '3.2' +[function.PyThreadState_SetAsyncExc] + added = '3.2' +[function.PyThreadState_Swap] + added = '3.2' +[function.PyTraceBack_Here] + added = '3.2' +[function.PyTraceBack_Print] + added = '3.2' +[data.PyTraceBack_Type] + added = '3.2' +[data.PyTupleIter_Type] + added = '3.2' +[function.PyTuple_GetItem] + added = '3.2' +[function.PyTuple_GetSlice] + added = '3.2' +[function.PyTuple_New] + added = '3.2' +[function.PyTuple_Pack] + added = '3.2' +[function.PyTuple_SetItem] + added = '3.2' +[function.PyTuple_Size] + added = '3.2' +[data.PyTuple_Type] + added = '3.2' +[function.PyType_ClearCache] + added = '3.2' +[function.PyType_GenericAlloc] + added = '3.2' +[function.PyType_GenericNew] + added = '3.2' +[function.PyType_GetFlags] + added = '3.2' +[function.PyType_IsSubtype] + added = '3.2' +[function.PyType_Modified] + added = '3.2' +[function.PyType_Ready] + added = '3.2' +[data.PyType_Type] + added = '3.2' +[function.PyUnicodeDecodeError_Create] + added = '3.2' +[function.PyUnicodeDecodeError_GetEncoding] + added = '3.2' +[function.PyUnicodeDecodeError_GetEnd] + added = '3.2' +[function.PyUnicodeDecodeError_GetObject] + added = '3.2' +[function.PyUnicodeDecodeError_GetReason] + added = '3.2' +[function.PyUnicodeDecodeError_GetStart] + added = '3.2' +[function.PyUnicodeDecodeError_SetEnd] + added = '3.2' +[function.PyUnicodeDecodeError_SetReason] + added = '3.2' +[function.PyUnicodeDecodeError_SetStart] + added = '3.2' +[function.PyUnicodeEncodeError_GetEncoding] + added = '3.2' +[function.PyUnicodeEncodeError_GetEnd] + added = '3.2' +[function.PyUnicodeEncodeError_GetObject] + added = '3.2' +[function.PyUnicodeEncodeError_GetReason] + added = '3.2' +[function.PyUnicodeEncodeError_GetStart] + added = '3.2' +[function.PyUnicodeEncodeError_SetEnd] + added = '3.2' +[function.PyUnicodeEncodeError_SetReason] + added = '3.2' +[function.PyUnicodeEncodeError_SetStart] + added = '3.2' +[data.PyUnicodeIter_Type] + added = '3.2' +[function.PyUnicodeTranslateError_GetEnd] + added = '3.2' +[function.PyUnicodeTranslateError_GetObject] + added = '3.2' +[function.PyUnicodeTranslateError_GetReason] + added = '3.2' +[function.PyUnicodeTranslateError_GetStart] + added = '3.2' +[function.PyUnicodeTranslateError_SetEnd] + added = '3.2' +[function.PyUnicodeTranslateError_SetReason] + added = '3.2' +[function.PyUnicodeTranslateError_SetStart] + added = '3.2' +[function.PyUnicode_Append] + added = '3.2' +[function.PyUnicode_AppendAndDel] + added = '3.2' +[function.PyUnicode_AsASCIIString] + added = '3.2' +[function.PyUnicode_AsCharmapString] + added = '3.2' +[function.PyUnicode_AsDecodedObject] + added = '3.2' +[function.PyUnicode_AsDecodedUnicode] + added = '3.2' +[function.PyUnicode_AsEncodedObject] + added = '3.2' +[function.PyUnicode_AsEncodedString] + added = '3.2' +[function.PyUnicode_AsEncodedUnicode] + added = '3.2' +[function.PyUnicode_AsLatin1String] + added = '3.2' +[function.PyUnicode_AsRawUnicodeEscapeString] + added = '3.2' +[function.PyUnicode_AsUTF16String] + added = '3.2' +[function.PyUnicode_AsUTF32String] + added = '3.2' +[function.PyUnicode_AsUTF8String] + added = '3.2' +[function.PyUnicode_AsUnicodeEscapeString] + added = '3.2' +[function.PyUnicode_AsWideChar] + added = '3.2' +[function.PyUnicode_Compare] + added = '3.2' +[function.PyUnicode_Concat] + added = '3.2' +[function.PyUnicode_Contains] + added = '3.2' +[function.PyUnicode_Count] + added = '3.2' +[function.PyUnicode_Decode] + added = '3.2' +[function.PyUnicode_DecodeASCII] + added = '3.2' +[function.PyUnicode_DecodeCharmap] + added = '3.2' +[function.PyUnicode_DecodeFSDefault] + added = '3.2' +[function.PyUnicode_DecodeFSDefaultAndSize] + added = '3.2' +[function.PyUnicode_DecodeLatin1] + added = '3.2' +[function.PyUnicode_DecodeRawUnicodeEscape] + added = '3.2' +[function.PyUnicode_DecodeUTF16] + added = '3.2' +[function.PyUnicode_DecodeUTF16Stateful] + added = '3.2' +[function.PyUnicode_DecodeUTF32] + added = '3.2' +[function.PyUnicode_DecodeUTF32Stateful] + added = '3.2' +[function.PyUnicode_DecodeUTF8] + added = '3.2' +[function.PyUnicode_DecodeUTF8Stateful] + added = '3.2' +[function.PyUnicode_DecodeUnicodeEscape] + added = '3.2' +[function.PyUnicode_FSConverter] + added = '3.2' +[function.PyUnicode_FSDecoder] + added = '3.2' +[function.PyUnicode_Find] + added = '3.2' +[function.PyUnicode_Format] + added = '3.2' +[function.PyUnicode_FromEncodedObject] + added = '3.2' +[function.PyUnicode_FromFormat] + added = '3.2' +[function.PyUnicode_FromFormatV] + added = '3.2' +[function.PyUnicode_FromObject] + added = '3.2' +[function.PyUnicode_FromOrdinal] + added = '3.2' +[function.PyUnicode_FromString] + added = '3.2' +[function.PyUnicode_FromStringAndSize] + added = '3.2' +[function.PyUnicode_FromWideChar] + added = '3.2' +[function.PyUnicode_GetDefaultEncoding] + added = '3.2' +[function.PyUnicode_GetSize] + added = '3.2' +[function.PyUnicode_IsIdentifier] + added = '3.2' +[function.PyUnicode_Join] + added = '3.2' +[function.PyUnicode_Partition] + added = '3.2' +[function.PyUnicode_RPartition] + added = '3.2' +[function.PyUnicode_RSplit] + added = '3.2' +[function.PyUnicode_Replace] + added = '3.2' +[function.PyUnicode_Resize] + added = '3.2' +[function.PyUnicode_RichCompare] + added = '3.2' +[function.PyUnicode_Split] + added = '3.2' +[function.PyUnicode_Splitlines] + added = '3.2' +[function.PyUnicode_Tailmatch] + added = '3.2' +[function.PyUnicode_Translate] + added = '3.2' +[function.PyUnicode_BuildEncodingMap] + added = '3.2' +[function.PyUnicode_CompareWithASCIIString] + added = '3.2' +[function.PyUnicode_DecodeUTF7] + added = '3.2' +[function.PyUnicode_DecodeUTF7Stateful] + added = '3.2' +[function.PyUnicode_EncodeFSDefault] + added = '3.2' +[function.PyUnicode_InternFromString] + added = '3.2' +[function.PyUnicode_InternImmortal] + added = '3.2' +[function.PyUnicode_InternInPlace] + added = '3.2' +[data.PyUnicode_Type] + added = '3.2' +[function.PyWeakref_GetObject] + added = '3.2' +[function.PyWeakref_NewProxy] + added = '3.2' +[function.PyWeakref_NewRef] + added = '3.2' +[data.PyWrapperDescr_Type] + added = '3.2' +[function.PyWrapper_New] + added = '3.2' +[data.PyZip_Type] + added = '3.2' +[function.Py_AddPendingCall] + added = '3.2' +[function.Py_AtExit] + added = '3.2' +[function.Py_BuildValue] + added = '3.2' +[function.Py_CompileString] + added = '3.2' +[function.Py_DecRef] + added = '3.2' +[function.Py_EndInterpreter] + added = '3.2' +[function.Py_Exit] + added = '3.2' +[function.Py_FatalError] + added = '3.2' +[data.Py_FileSystemDefaultEncoding] + added = '3.2' +[function.Py_Finalize] + added = '3.2' +[function.Py_GetBuildInfo] + added = '3.2' +[function.Py_GetCompiler] + added = '3.2' +[function.Py_GetCopyright] + added = '3.2' +[function.Py_GetExecPrefix] + added = '3.2' +[function.Py_GetPath] + added = '3.2' +[function.Py_GetPlatform] + added = '3.2' +[function.Py_GetPrefix] + added = '3.2' +[function.Py_GetProgramFullPath] + added = '3.2' +[function.Py_GetProgramName] + added = '3.2' +[function.Py_GetPythonHome] + added = '3.2' +[function.Py_GetRecursionLimit] + added = '3.2' +[function.Py_GetVersion] + added = '3.2' +[data.Py_HasFileSystemDefaultEncoding] + added = '3.2' +[function.Py_IncRef] + added = '3.2' +[function.Py_Initialize] + added = '3.2' +[function.Py_InitializeEx] + added = '3.2' +[function.Py_IsInitialized] + added = '3.2' +[function.Py_Main] + added = '3.2' +[function.Py_MakePendingCalls] + added = '3.2' +[function.Py_NewInterpreter] + added = '3.2' +[function.Py_ReprEnter] + added = '3.2' +[function.Py_ReprLeave] + added = '3.2' +[function.Py_SetProgramName] + added = '3.2' +[function.Py_SetPythonHome] + added = '3.2' +[function.Py_SetRecursionLimit] + added = '3.2' +[function.Py_VaBuildValue] + added = '3.2' + +[function._PyErr_BadInternalCall] + added = '3.2' + abi_only = true +[function._PyObject_CallFunction_SizeT] + added = '3.2' + abi_only = true +[function._PyObject_CallMethod_SizeT] + added = '3.2' + abi_only = true +[function._PyObject_GC_New] + added = '3.2' + abi_only = true +[function._PyObject_GC_NewVar] + added = '3.2' + abi_only = true +[function._PyObject_GC_Resize] + added = '3.2' + abi_only = true +[function._PyObject_New] + added = '3.2' + abi_only = true +[function._PyObject_NewVar] + added = '3.2' + abi_only = true +[function._PyState_AddModule] + added = '3.2' + abi_only = true +[function._PyThreadState_Init] + added = '3.2' + abi_only = true +[function._PyThreadState_Prealloc] + added = '3.2' + abi_only = true +[data._PyWeakref_CallableProxyType] + added = '3.2' + abi_only = true +[data._PyWeakref_ProxyType] + added = '3.2' + abi_only = true +[data._PyWeakref_RefType] + added = '3.2' + abi_only = true +[function._Py_BuildValue_SizeT] + added = '3.2' + abi_only = true +[function._Py_CheckRecursiveCall] + added = '3.2' + abi_only = true +[function._Py_Dealloc] + added = '3.2' + abi_only = true +[data._Py_EllipsisObject] + added = '3.2' + abi_only = true +[data._Py_FalseStruct] + added = '3.2' + abi_only = true +[data._Py_NoneStruct] + added = '3.2' + abi_only = true +[data._Py_NotImplementedStruct] + added = '3.2' + abi_only = true +[data._Py_SwappedOp] + added = '3.2' + abi_only = true +[data._Py_TrueStruct] + added = '3.2' + abi_only = true +[function._Py_VaBuildValue_SizeT] + added = '3.2' + abi_only = true + +# Old buffer protocol support (deprecated) + +[function.PyObject_AsCharBuffer] + added = '3.2' +[function.PyObject_AsReadBuffer] + added = '3.2' +[function.PyObject_AsWriteBuffer] + added = '3.2' +[function.PyObject_CheckReadBuffer] + added = '3.2' + +# Flags are implicitly part of the ABI: + +[const.Py_TPFLAGS_DEFAULT] + added = '3.2' +[const.Py_TPFLAGS_BASETYPE] + added = '3.2' +[const.Py_TPFLAGS_HAVE_GC] + added = '3.2' + +[const.METH_VARARGS] + added = '3.2' +[const.METH_NOARGS] + added = '3.2' +[const.METH_O] + added = '3.2' +[const.METH_CLASS] + added = '3.2' +[const.METH_STATIC] + added = '3.2' +[const.METH_COEXIST] + added = '3.2' +# METH_STACKLESS is undocumented +# METH_FASTCALL is not part of limited API. + +# The following are defined in private headers, but historically +# they were exported as part of the stable ABI. +[function.PyMarshal_ReadObjectFromString] + added = '3.2' + abi_only = true +[function.PyMarshal_WriteObjectToString] + added = '3.2' + abi_only = true +[function.PyMember_GetOne] + added = '3.2' + abi_only = true +[function.PyMember_SetOne] + added = '3.2' + abi_only = true + +# TLS api is deprecated; superseded by TSS API + +[function.PyThread_ReInitTLS] + added = '3.2' +[function.PyThread_create_key] + added = '3.2' +[function.PyThread_delete_key] + added = '3.2' +[function.PyThread_set_key_value] + added = '3.2' +[function.PyThread_get_key_value] + added = '3.2' +[function.PyThread_delete_key_value] + added = '3.2' +[function.PyThread_acquire_lock] + added = '3.2' +[function.PyThread_acquire_lock_timed] + added = '3.2' +[function.PyThread_allocate_lock] + added = '3.2' +[function.PyThread_exit_thread] + added = '3.2' +[function.PyThread_free_lock] + added = '3.2' +[function.PyThread_get_stacksize] + added = '3.2' +[function.PyThread_get_thread_ident] + added = '3.2' +[function.PyThread_get_thread_native_id] + added = '3.2' + ifdef = 'PY_HAVE_THREAD_NATIVE_ID' +[function.PyThread_init_thread] + added = '3.2' +[function.PyThread_release_lock] + added = '3.2' +[function.PyThread_set_stacksize] + added = '3.2' +[function.PyThread_start_new_thread] + added = '3.2' + +# The following were added in PC/python3.def in Python 3.3: +# 7800f75827b1be557be16f3b18f5170fbf9fae08 +# 9c56409d3353b8cd4cfc19e0467bbe23fd34fc92 +# 75aeaa9b18667219bbacbc58ba6efecccef9dfbd + +[function.PyState_AddModule] + added = '3.3' +[function.PyState_RemoveModule] + added = '3.3' +[function.PyType_FromSpecWithBases] + added = '3.3' +[function._PyArg_Parse_SizeT] + added = '3.3' + abi_only = true +[function._PyArg_ParseTuple_SizeT] + added = '3.3' + abi_only = true +[function._PyArg_ParseTupleAndKeywords_SizeT] + added = '3.3' + abi_only = true +[function._PyArg_VaParse_SizeT] + added = '3.3' + abi_only = true +[function._PyArg_VaParseTupleAndKeywords_SizeT] + added = '3.3' + abi_only = true +[function.PyThread_GetInfo] + added = '3.3' + +# The following were added in PC/python3.def in Python 3.4: +# 3ba3a3ee56c142e93d6bbe20ff6bf939212a30f0 + +[function.PyCFunction_New] + added = '3.4' +[function.PyType_GetSlot] + added = '3.4' + +# The following were added in PC/python3.def in Python 3.5: +# 11d7b1423fc44d764eba7065ea5eba58ed748b21 +# f3b73ad51da3097d7915796fdc62608b1ab90c0a + +[function.PyErr_FormatV] + added = '3.5' +[function.PyModuleDef_Init] + added = '3.5' +[data.PyModuleDef_Type] + added = '3.5' + +# New slots in 3.5: +# d51374ed78a3e3145911a16cdf3b9b84b3ba7d15 - Matrix multiplication (PEP 465) +# 7544508f0245173bff5866aa1598c8f6cce1fc5f - Async iterators (PEP 492) +# 0969a9f8abcf98bb43ea77b1dd050426adcfb4f7 - tp_finalize + +[const.Py_nb_matrix_multiply] + added = '3.5' +[const.Py_nb_inplace_matrix_multiply] + added = '3.5' +[const.Py_am_await] + added = '3.5' +[const.Py_am_aiter] + added = '3.5' +[const.Py_am_anext] + added = '3.5' +[const.Py_tp_finalize] + added = '3.5' + +# The following were added in PC/python3.def in Python 3.6: + +[function.Py_FinalizeEx] + added = '3.6' + +[function.PyOS_FSPath] + added = '3.6' +[function.PyErr_ResourceWarning] + added = '3.6' +[function.PyErr_SetImportErrorSubclass] + added = '3.6' +[data.PyExc_ModuleNotFoundError] + added = '3.6' + +# The following were added in PC/python3.def in Python 3.6.1 and 3.5.3/3.5.4: + +[function.PyCodec_NameReplaceErrors] + added = '3.7' # (and 3.6.1 and 3.5.3) +[function.PyErr_GetExcInfo] + added = '3.7' # (and 3.6.1 and 3.5.3) +[function.PyErr_SetExcInfo] + added = '3.7' # (and 3.6.1 and 3.5.3) +[function.PyErr_SetFromErrnoWithFilenameObjects] + added = '3.7' # (and 3.6.1 and 3.5.3) +[function.PyErr_SetImportError] + added = '3.7' # (and 3.6.1 and 3.5.3) +[function.PyErr_SyntaxLocationEx] + added = '3.7' # (and 3.6.1 and 3.5.3) +[data.PyExc_BlockingIOError] + added = '3.7' # (and 3.6.1 and 3.5.3) +[data.PyExc_BrokenPipeError] + added = '3.7' # (and 3.6.1 and 3.5.3) +[data.PyExc_ChildProcessError] + added = '3.7' # (and 3.6.1 and 3.5.3) +[data.PyExc_ConnectionAbortedError] + added = '3.7' # (and 3.6.1 and 3.5.3) +[data.PyExc_ConnectionError] + added = '3.7' # (and 3.6.1 and 3.5.3) +[data.PyExc_ConnectionRefusedError] + added = '3.7' # (and 3.6.1 and 3.5.3) +[data.PyExc_ConnectionResetError] + added = '3.7' # (and 3.6.1 and 3.5.3) +[data.PyExc_FileExistsError] + added = '3.7' # (and 3.6.1 and 3.5.3) +[data.PyExc_FileNotFoundError] + added = '3.7' # (and 3.6.1 and 3.5.3) +[data.PyExc_InterruptedError] + added = '3.7' # (and 3.6.1 and 3.5.3) +[data.PyExc_IsADirectoryError] + added = '3.7' # (and 3.6.1 and 3.5.3) +[data.PyExc_NotADirectoryError] + added = '3.7' # (and 3.6.1 and 3.5.3) +[data.PyExc_PermissionError] + added = '3.7' # (and 3.6.1 and 3.5.3) +[data.PyExc_ProcessLookupError] + added = '3.7' # (and 3.6.1 and 3.5.3) +[data.PyExc_RecursionError] + added = '3.7' # (and 3.6.1 and 3.5.3) +[data.PyExc_ResourceWarning] + added = '3.7' # (and 3.6.1 and 3.5.3) +[data.PyExc_StopAsyncIteration] + added = '3.7' # (and 3.6.1 and 3.5.3) +[data.PyExc_TimeoutError] + added = '3.7' # (and 3.6.1 and 3.5.3) +[function.PyImport_AddModuleObject] + added = '3.7' # (and 3.6.1 and 3.5.3) +[function.PyImport_ExecCodeModuleObject] + added = '3.7' # (and 3.6.1 and 3.5.3) +[function.PyImport_ImportFrozenModuleObject] + added = '3.7' # (and 3.6.1 and 3.5.3) +[function.PyImport_ImportModuleLevelObject] + added = '3.7' # (and 3.6.1 and 3.5.3) +[function.PyMem_Calloc] + added = '3.7' # (and 3.6.1 and 3.5.3) +[function.PyMemoryView_FromMemory] + added = '3.7' # (and 3.6.1 and 3.5.3) +[function.PyModule_AddFunctions] + added = '3.7' # (and 3.6.1 and 3.5.3) +[function.PyModule_ExecDef] + added = '3.7' # (and 3.6.1 and 3.5.3) +[function.PyModule_FromDefAndSpec2] + added = '3.7' # (and 3.6.1 and 3.5.3) +[function.PyModule_GetNameObject] + added = '3.7' # (and 3.6.1 and 3.5.3) +[function.PyModule_NewObject] + added = '3.7' # (and 3.6.1 and 3.5.3) +[function.PyModule_SetDocString] + added = '3.7' # (and 3.6.1 and 3.5.3) +[function.PyNumber_InPlaceMatrixMultiply] + added = '3.7' # (and 3.6.1 and 3.5.3) +[function.PyNumber_MatrixMultiply] + added = '3.7' # (and 3.6.1 and 3.5.3) +[function.PyObject_Calloc] + added = '3.7' # (and 3.6.1 and 3.5.3) +[function.PyObject_GenericSetDict] + added = '3.7' # (and 3.6.1 and 3.5.3) +[function.PySys_AddXOption] + added = '3.7' # (and 3.6.1 and 3.5.3) +[function.PySys_GetXOptions] + added = '3.7' # (and 3.6.1 and 3.5.3) +[function.PyUnicode_AsUCS4] + added = '3.7' # (and 3.6.1 and 3.5.3) +[function.PyUnicode_AsUCS4Copy] + added = '3.7' # (and 3.6.1 and 3.5.3) +[function.PyUnicode_AsWideCharString] + added = '3.7' # (and 3.6.1 and 3.5.3) +[function.PyUnicode_DecodeLocale] + added = '3.7' # (and 3.6.1 and 3.5.3) +[function.PyUnicode_DecodeLocaleAndSize] + added = '3.7' # (and 3.6.1 and 3.5.3) +[function.PyUnicode_EncodeLocale] + added = '3.7' # (and 3.6.1 and 3.5.3) +[function.PyUnicode_FindChar] + added = '3.7' # (and 3.6.1 and 3.5.3) +[function.PyUnicode_GetLength] + added = '3.7' # (and 3.6.1 and 3.5.3) +[function.PyUnicode_ReadChar] + added = '3.7' # (and 3.6.1 and 3.5.3) +[function.PyUnicode_Substring] + added = '3.7' # (and 3.6.1 and 3.5.3) +[function.PyUnicode_WriteChar] + added = '3.7' # (and 3.6.1 and 3.5.3) +[function.Py_DecodeLocale] + added = '3.7' # (and 3.6.1 and 3.5.3) +[function.Py_EncodeLocale] + added = '3.7' # (and 3.6.1 and 3.5.3) +[function.Py_SetPath] + added = '3.7' # (and 3.6.1 and 3.5.3) +[function.PyErr_SetExcFromWindowsErr] + added = '3.7' # (and 3.6.1 and 3.5.3) + ifdef = 'MS_WINDOWS' +[function.PyErr_SetExcFromWindowsErrWithFilename] + added = '3.7' # (and 3.6.1 and 3.5.3) + ifdef = 'MS_WINDOWS' +[function.PyErr_SetExcFromWindowsErrWithFilenameObject] + added = '3.7' # (and 3.6.1 and 3.5.3) + ifdef = 'MS_WINDOWS' +[function.PyErr_SetExcFromWindowsErrWithFilenameObjects] + added = '3.7' # (and 3.6.1 and 3.5.3) + ifdef = 'MS_WINDOWS' +[function.PyErr_SetFromWindowsErr] + added = '3.7' # (and 3.6.1 and 3.5.3) + ifdef = 'MS_WINDOWS' +[function.PyErr_SetFromWindowsErrWithFilename] + added = '3.7' # (and 3.6.1 and 3.5.3) + ifdef = 'MS_WINDOWS' +[data.PyExc_WindowsError] + added = '3.7' # (and 3.6.1 and 3.5.3) + ifdef = 'MS_WINDOWS' +[function.PyOS_CheckStack] + added = '3.7' # (and 3.6.1 and 3.5.3) + ifdef = 'USE_STACKCHECK' +[function.PyUnicode_AsMBCSString] + added = '3.7' # (and 3.6.1 and 3.5.3) + ifdef = 'MS_WINDOWS' +[function.PyUnicode_DecodeCodePageStateful] + added = '3.7' # (and 3.6.1 and 3.5.3) + ifdef = 'MS_WINDOWS' +[function.PyUnicode_DecodeMBCS] + added = '3.7' # (and 3.6.1 and 3.5.3) + ifdef = 'MS_WINDOWS' +[function.PyUnicode_DecodeMBCSStateful] + added = '3.7' # (and 3.6.1 and 3.5.3) + ifdef = 'MS_WINDOWS' +[function.PyUnicode_EncodeCodePage] + added = '3.7' # (and 3.6.1 and 3.5.3) + ifdef = 'MS_WINDOWS' + +# 3.5.4: +[function.PySlice_AdjustIndices] + added = '3.7' # (and 3.6.1 and 3.5.4) +[function.PySlice_Unpack] + added = '3.7' # (and 3.6.1 and 3.5.4) + +# The following were added in PC/python3.def in Python 3.7: + +[function.PyInterpreterState_GetID] + added = '3.7' +[function.PyThread_tss_alloc] + added = '3.7' +[function.PyThread_tss_create] + added = '3.7' +[function.PyThread_tss_delete] + added = '3.7' +[function.PyThread_tss_free] + added = '3.7' +[function.PyThread_tss_get] + added = '3.7' +[function.PyThread_tss_is_created] + added = '3.7' +[function.PyThread_tss_set] + added = '3.7' +[function.PyOS_BeforeFork] + added = '3.7' + ifdef = 'HAVE_FORK' +[function.PyOS_AfterFork_Parent] + added = '3.7' + ifdef = 'HAVE_FORK' +[function.PyOS_AfterFork_Child] + added = '3.7' + ifdef = 'HAVE_FORK' + +# New method flags in 3.7 (PEP 590): + +[const.METH_FASTCALL] + added = '3.7' +[const.METH_METHOD] + added = '3.7' + +# The following were added in PC/python3.def in Python 3.8: + +[function.PyImport_GetModule] + added = '3.8' +[data.Py_UTF8Mode] + added = '3.8' +[function.PyExceptionClass_Name] + added = '3.8' +[function.PyIndex_Check] + added = '3.8' +[function.PyIter_Check] + added = '3.8' +[data.PyDictRevIterItem_Type] + added = '3.8' +[data.PyDictRevIterKey_Type] + added = '3.8' +[data.PyDictRevIterValue_Type] + added = '3.8' +[function.PyInterpreterState_GetDict] + added = '3.8' +[function.Py_BytesMain] + added = '3.8' + +# New type flag (PEP 590): + +[const.Py_TPFLAGS_METHOD_DESCRIPTOR] + added = '3.8' + +# The following were added in PC/python3.def in Python 3.9: + +[function.Py_EnterRecursiveCall] + added = '3.9' +[function.Py_LeaveRecursiveCall] + added = '3.9' +[function.Py_GenericAlias] + added = '3.9' +[data.Py_GenericAliasType] + added = '3.9' +[function.PyCMethod_New] + added = '3.9' +[function.PyInterpreterState_Get] + added = '3.9' +[function.PyObject_GC_IsFinalized] + added = '3.9' +[function.PyObject_GC_IsTracked] + added = '3.9' + +# The following were added in PC/python3.def in Python 3.10: + +[function.Py_GetArgcArgv] + added = '3.10' + abi_only = true +[function.PyIter_Send] + added = '3.10' +[function.PyUnicode_AsUTF8AndSize] + added = '3.10' +[function.PyObject_GenericGetDict] + added = '3.10' +[function.Py_NewRef] + added = '3.10' +[function.Py_XNewRef] + added = '3.10' +[function.PyModule_AddType] + added = '3.10' +[function.PyType_FromModuleAndSpec] + added = '3.10' +[function.PyType_GetModule] + added = '3.10' +[function.PyType_GetModuleState] + added = '3.10' +[function.PyFrame_GetLineNumber] + added = '3.10' +[function.PyFrame_GetCode] + added = '3.10' +[function.PyObject_CallNoArgs] + added = '3.10' +[function.PyThreadState_GetFrame] + added = '3.10' +[function.PyThreadState_GetID] + added = '3.10' +[function.PyThreadState_GetInterpreter] + added = '3.10' +[function.PyModule_AddObjectRef] + added = '3.10' +[data.Py_FileSystemDefaultEncodeErrors] + added = '3.10' +[function.PyCodec_Unregister] + added = '3.10' +[function.PyErr_SetInterruptEx] + added = '3.10' +[function.Py_Is] + added = '3.10' +[function.Py_IsTrue] + added = '3.10' +[function.Py_IsFalse] + added = '3.10' +[function.Py_IsNone] + added = '3.10' +[function._Py_IncRef] + added = '3.10' + abi_only = true +[function._Py_DecRef] + added = '3.10' + abi_only = true +[function.PyAIter_Check] + added = '3.10' +[function.PyObject_GetAIter] + added = '3.10' +[data.PyExc_EncodingWarning] + added = '3.10' + +# Support for Stable ABI in debug builds + +[data._Py_RefTotal] + added = '3.10' + ifdef = 'Py_REF_DEBUG' + abi_only = true +[function._Py_NegativeRefcount] + added = '3.10' + ifdef = 'Py_REF_DEBUG' + abi_only = true + +# New slots in 3.10: + +[const.Py_am_send] + added = '3.10' + + +# New GC control functions in Py3.10 (https://bugs.python.org/issue28254) + +[function.PyGC_Disable] + added = '3.10' +[function.PyGC_Enable] + added = '3.10' +[function.PyGC_IsEnabled] + added = '3.10' + +# Add new C API in Python 3.11 + +[function.PyType_GetName] + added = '3.11' +[function.PyType_GetQualName] + added = '3.11' +[data.PyStructSequence_UnnamedField] + added = '3.11' + +# Add stable Py_buffer API in Python 3.11 (https://bugs.python.org/issue45459) +[struct.Py_buffer] + added = '3.11' + struct_abi_kind = 'full-abi' +[function.PyObject_CheckBuffer] + added = '3.11' +[function.PyObject_GetBuffer] + added = '3.11' +[function.PyBuffer_GetPointer] + added = '3.11' +[function.PyBuffer_SizeFromFormat] + added = '3.11' +[function.PyBuffer_ToContiguous] + added = '3.11' +[function.PyBuffer_FromContiguous] + added = '3.11' +[function.PyObject_CopyData] + added = '3.11' +[function.PyBuffer_IsContiguous] + added = '3.11' +[function.PyBuffer_FillContiguousStrides] + added = '3.11' +[function.PyBuffer_FillInfo] + added = '3.11' +[function.PyBuffer_Release] + added = '3.11' +[function.PyMemoryView_FromBuffer] + added = '3.11' + +# (Detailed comments aren't really needed for further entries: from here on +# we can use version control logs.) + +[data.Py_Version] + added = '3.11' +[function.PyErr_GetHandledException] + added = '3.11' +[function.PyErr_SetHandledException] + added = '3.11' diff --git a/Misc/stable_abi.txt b/Misc/stable_abi.txt deleted file mode 100644 index 9b1c87ea8f4f17..00000000000000 --- a/Misc/stable_abi.txt +++ /dev/null @@ -1,2284 +0,0 @@ -# This file lists the contents of the Limited API and Stable ABI. -# Please append new items at the end. - -# The syntax of this file is not fixed. -# It is designed to be read only by Tools/stable_abi.py, which can change -# without notice. - -# For the history of the stable ABI prior to this file, -# see the history of PC/python3dll.c and before that, PC/python3.def, -# and PC/pythonXYstub.def - - -# The current format is a simple line-based one with significant indentation. -# Anything after a hash is a comment. - -# There are these kinds of top-level "items": -# - struct: A C struct. Currently this file does not distinguish between: -# - opaque structs, which the Limited API only handles via pointers -# (so these can change at any time) -# - structs where only certain members are part of the stable ABI (e.g. -# PyObject) -# - structs which must not be changed at all (e.g. PyType_Slot, which is -# fully defined and used in arrays) -# - function: A function that must be kept available (and exported, i.e. not -# converted to a macro). -# - const: A simple value, defined with `#define`. -# - macro: A preprocessor macro more complex than a simple `const` value. -# - data: An exported object, which must continue to be available but its exact -# value may change. -# - typedef: A C typedef which is used in other definitions in the limited API. -# Its size/layout/signature must not change. -# - ifdef: A feature macro: other items may be conditional on whether the macro -# is defined or not. - -# Each top-level item can have details defined below it: -# - added: The version in which the item was added to the stable ABI. -# - ifdef: A feature macro: the item is only available if this macro is defined -# - abi_only: If present, the item is not part of the Limited API, but it *is* -# part of the stable ABI. The item will not show up in user-facing docs. -# Typically used for: -# - private functions called by public macros, e.g. _Py_BuildValue_SizeT -# - items that were part of the limited API in the past, and must remain part -# of the stable ABI. -# - a combination of the above (functions that were called by macros that -# were public in the past) -# - doc: for `ifdef`, the blurb added in documentation -# - windows: for `ifdef`, this macro is defined on Windows. (This info is used -# to generate the DLL manifest and needs to be available on all platforms.) -# `maybe` marks macros defined on some but not all Windows builds. - -# For structs, one of the following must be set: -# - opaque: The struct name is available in the Limited API, but its members -# are not. Users must manipulate it via pointers. -# - members: Space-separated list of members which are part of the -# Limited API and Stable ABI. -# Members that aren't listed are not accessible to applications. -# - full-abi: The entire struct -- all its members and its size -- is part of -# the Stable ABI, and must not change. - -# Removing items from this file is generally not allowed, and additions should -# be considered with that in mind. See the devguide for exact rules: -# https://devguide.python.org/c-api/#limited-api - -# User-facing docs are at: -# https://docs.python.org/3/c-api/stable.html#stable - - -# Feature macros for optional functionality: - -ifdef MS_WINDOWS - doc on Windows - windows -ifdef HAVE_FORK - doc on platforms with fork() -ifdef USE_STACKCHECK - doc on platforms with USE_STACKCHECK - windows maybe -ifdef PY_HAVE_THREAD_NATIVE_ID - doc on platforms with native thread IDs - windows -ifdef Py_REF_DEBUG - doc when Python is compiled in debug mode (with Py_REF_DEBUG) - windows maybe - - -# Mentioned in PEP 384: - -struct PyObject - added 3.2 - members ob_refcnt ob_type -struct PyVarObject - added 3.2 - members ob_base ob_size -struct PyMethodDef - added 3.2 - full-abi -struct PyMemberDef - added 3.2 - full-abi -struct PyGetSetDef - added 3.2 - full-abi -struct PyModuleDef_Base - added 3.2 - full-abi -struct PyModuleDef - added 3.2 - full-abi -struct PyStructSequence_Field - added 3.2 - full-abi -struct PyStructSequence_Desc - added 3.2 - full-abi -struct PyType_Slot - added 3.2 - full-abi -struct PyType_Spec - added 3.2 - full-abi -struct PyThreadState - added 3.2 - opaque -struct PyInterpreterState - added 3.2 - opaque -struct PyFrameObject - added 3.2 - opaque -struct symtable - added 3.2 - opaque -struct PyWeakReference - added 3.2 - opaque -struct PyLongObject - added 3.2 - opaque -struct PyTypeObject - added 3.2 - opaque - -function PyType_FromSpec - added 3.2 - -const Py_tp_dealloc - added 3.2 -const Py_tp_getattr - added 3.2 -const Py_tp_setattr - added 3.2 -const Py_tp_repr - added 3.2 -const Py_tp_hash - added 3.2 -const Py_tp_call - added 3.2 -const Py_tp_str - added 3.2 -const Py_tp_getattro - added 3.2 -const Py_tp_setattro - added 3.2 -const Py_tp_doc - added 3.2 -const Py_tp_traverse - added 3.2 -const Py_tp_clear - added 3.2 -const Py_tp_richcompare - added 3.2 -const Py_tp_iter - added 3.2 -const Py_tp_iternext - added 3.2 -const Py_tp_methods - added 3.2 -const Py_tp_base - added 3.2 -const Py_tp_descr_get - added 3.2 -const Py_tp_descr_set - added 3.2 -const Py_tp_init - added 3.2 -const Py_tp_alloc - added 3.2 -const Py_tp_new - added 3.2 -const Py_tp_members - added 3.2 -const Py_tp_getset - added 3.2 -const Py_tp_free - added 3.2 -const Py_tp_is_gc - added 3.2 -const Py_tp_bases - added 3.2 -const Py_tp_del - added 3.2 -const Py_nb_add - added 3.2 -const Py_nb_subtract - added 3.2 -const Py_nb_multiply - added 3.2 -const Py_nb_remainder - added 3.2 -const Py_nb_divmod - added 3.2 -const Py_nb_power - added 3.2 -const Py_nb_negative - added 3.2 -const Py_nb_positive - added 3.2 -const Py_nb_absolute - added 3.2 -const Py_nb_bool - added 3.2 -const Py_nb_invert - added 3.2 -const Py_nb_lshift - added 3.2 -const Py_nb_rshift - added 3.2 -const Py_nb_and - added 3.2 -const Py_nb_xor - added 3.2 -const Py_nb_or - added 3.2 -const Py_nb_int - added 3.2 -const Py_nb_float - added 3.2 -const Py_nb_inplace_add - added 3.2 -const Py_nb_inplace_subtract - added 3.2 -const Py_nb_inplace_multiply - added 3.2 -const Py_nb_inplace_remainder - added 3.2 -const Py_nb_inplace_power - added 3.2 -const Py_nb_inplace_lshift - added 3.2 -const Py_nb_inplace_rshift - added 3.2 -const Py_nb_inplace_and - added 3.2 -const Py_nb_inplace_xor - added 3.2 -const Py_nb_inplace_or - added 3.2 -const Py_nb_floor_divide - added 3.2 -const Py_nb_true_divide - added 3.2 -const Py_nb_inplace_floor_divide - added 3.2 -const Py_nb_inplace_true_divide - added 3.2 -const Py_nb_index - added 3.2 -const Py_sq_length - added 3.2 -const Py_sq_concat - added 3.2 -const Py_sq_repeat - added 3.2 -const Py_sq_item - added 3.2 -const Py_sq_ass_item - added 3.2 -const Py_sq_contains - added 3.2 -const Py_sq_inplace_concat - added 3.2 -const Py_sq_inplace_repeat - added 3.2 -const Py_mp_length - added 3.2 -const Py_mp_subscript - added 3.2 -const Py_mp_ass_subscript - added 3.2 - -typedef Py_uintptr_t - added 3.2 -typedef Py_intptr_t - added 3.2 -typedef Py_ssize_t - added 3.2 -typedef unaryfunc - added 3.2 -typedef binaryfunc - added 3.2 -typedef ternaryfunc - added 3.2 -typedef inquiry - added 3.2 -typedef lenfunc - added 3.2 -typedef ssizeargfunc - added 3.2 -typedef ssizessizeargfunc - added 3.2 -typedef ssizeobjargproc - added 3.2 -typedef ssizessizeobjargproc - added 3.2 -typedef objobjargproc - added 3.2 -typedef objobjproc - added 3.2 -typedef visitproc - added 3.2 -typedef traverseproc - added 3.2 -typedef destructor - added 3.2 -typedef getattrfunc - added 3.2 -typedef getattrofunc - added 3.2 -typedef setattrfunc - added 3.2 -typedef setattrofunc - added 3.2 -typedef reprfunc - added 3.2 -typedef hashfunc - added 3.2 -typedef richcmpfunc - added 3.2 -typedef getiterfunc - added 3.2 -typedef iternextfunc - added 3.2 -typedef descrgetfunc - added 3.2 -typedef descrsetfunc - added 3.2 -typedef initproc - added 3.2 -typedef newfunc - added 3.2 -typedef allocfunc - added 3.2 -typedef PyCFunction - added 3.2 -typedef PyCFunctionWithKeywords - added 3.2 -typedef PyCapsule_Destructor - added 3.2 -typedef getter - added 3.2 -typedef setter - added 3.2 -typedef PyOS_sighandler_t - added 3.2 -typedef PyGILState_STATE - added 3.2 -typedef Py_UCS4 - added 3.2 - -macro Py_BEGIN_ALLOW_THREADS - added 3.2 -macro Py_BLOCK_THREADS - added 3.2 -macro Py_UNBLOCK_THREADS - added 3.2 -macro Py_END_ALLOW_THREADS - added 3.2 - -# The following were added in PC/python3.def in the initial stable ABI commit, -# 4d0d471a8031de90a2b1ce99c4ac4780e60b3bc9, -# and later amendments in 3.2: -# 0d012f284be829c6217f60523db0e1671b7db9d9 -# c83bc3c1fbed14d27a5de3032e24d2cf006a7c4b - -function PyArg_Parse - added 3.2 -function PyArg_ParseTuple - added 3.2 -function PyArg_ParseTupleAndKeywords - added 3.2 -function PyArg_UnpackTuple - added 3.2 -function PyArg_VaParse - added 3.2 -function PyArg_VaParseTupleAndKeywords - added 3.2 -function PyArg_ValidateKeywordArguments - added 3.2 -data PyBaseObject_Type - added 3.2 -function PyBool_FromLong - added 3.2 -data PyBool_Type - added 3.2 -data PyByteArrayIter_Type - added 3.2 -function PyByteArray_AsString - added 3.2 -function PyByteArray_Concat - added 3.2 -function PyByteArray_FromObject - added 3.2 -function PyByteArray_FromStringAndSize - added 3.2 -function PyByteArray_Resize - added 3.2 -function PyByteArray_Size - added 3.2 -data PyByteArray_Type - added 3.2 -data PyBytesIter_Type - added 3.2 -function PyBytes_AsString - added 3.2 -function PyBytes_AsStringAndSize - added 3.2 -function PyBytes_Concat - added 3.2 -function PyBytes_ConcatAndDel - added 3.2 -function PyBytes_DecodeEscape - added 3.2 -function PyBytes_FromFormat - added 3.2 -function PyBytes_FromFormatV - added 3.2 -function PyBytes_FromObject - added 3.2 -function PyBytes_FromString - added 3.2 -function PyBytes_FromStringAndSize - added 3.2 -function PyBytes_Repr - added 3.2 -function PyBytes_Size - added 3.2 -data PyBytes_Type - added 3.2 -function PyCFunction_Call - added 3.2 -function PyCFunction_GetFlags - added 3.2 -function PyCFunction_GetFunction - added 3.2 -function PyCFunction_GetSelf - added 3.2 -function PyCFunction_NewEx - added 3.2 -data PyCFunction_Type - added 3.2 -function PyCallIter_New - added 3.2 -data PyCallIter_Type - added 3.2 -function PyCallable_Check - added 3.2 -function PyCapsule_GetContext - added 3.2 -function PyCapsule_GetDestructor - added 3.2 -function PyCapsule_GetName - added 3.2 -function PyCapsule_GetPointer - added 3.2 -function PyCapsule_Import - added 3.2 -function PyCapsule_IsValid - added 3.2 -function PyCapsule_New - added 3.2 -function PyCapsule_SetContext - added 3.2 -function PyCapsule_SetDestructor - added 3.2 -function PyCapsule_SetName - added 3.2 -function PyCapsule_SetPointer - added 3.2 -data PyCapsule_Type - added 3.2 -data PyClassMethodDescr_Type - added 3.2 -function PyCodec_BackslashReplaceErrors - added 3.2 -function PyCodec_Decode - added 3.2 -function PyCodec_Decoder - added 3.2 -function PyCodec_Encode - added 3.2 -function PyCodec_Encoder - added 3.2 -function PyCodec_IgnoreErrors - added 3.2 -function PyCodec_IncrementalDecoder - added 3.2 -function PyCodec_IncrementalEncoder - added 3.2 -function PyCodec_KnownEncoding - added 3.2 -function PyCodec_LookupError - added 3.2 -function PyCodec_Register - added 3.2 -function PyCodec_RegisterError - added 3.2 -function PyCodec_ReplaceErrors - added 3.2 -function PyCodec_StreamReader - added 3.2 -function PyCodec_StreamWriter - added 3.2 -function PyCodec_StrictErrors - added 3.2 -function PyCodec_XMLCharRefReplaceErrors - added 3.2 -function PyComplex_FromDoubles - added 3.2 -function PyComplex_ImagAsDouble - added 3.2 -function PyComplex_RealAsDouble - added 3.2 -data PyComplex_Type - added 3.2 -function PyDescr_NewClassMethod - added 3.2 -function PyDescr_NewGetSet - added 3.2 -function PyDescr_NewMember - added 3.2 -function PyDescr_NewMethod - added 3.2 -data PyDictItems_Type - added 3.2 -data PyDictIterItem_Type - added 3.2 -data PyDictIterKey_Type - added 3.2 -data PyDictIterValue_Type - added 3.2 -data PyDictKeys_Type - added 3.2 -function PyDictProxy_New - added 3.2 -data PyDictProxy_Type - added 3.2 -data PyDictValues_Type - added 3.2 -function PyDict_Clear - added 3.2 -function PyDict_Contains - added 3.2 -function PyDict_Copy - added 3.2 -function PyDict_DelItem - added 3.2 -function PyDict_DelItemString - added 3.2 -function PyDict_GetItem - added 3.2 -function PyDict_GetItemString - added 3.2 -function PyDict_GetItemWithError - added 3.2 -function PyDict_Items - added 3.2 -function PyDict_Keys - added 3.2 -function PyDict_Merge - added 3.2 -function PyDict_MergeFromSeq2 - added 3.2 -function PyDict_New - added 3.2 -function PyDict_Next - added 3.2 -function PyDict_SetItem - added 3.2 -function PyDict_SetItemString - added 3.2 -function PyDict_Size - added 3.2 -data PyDict_Type - added 3.2 -function PyDict_Update - added 3.2 -function PyDict_Values - added 3.2 -data PyEllipsis_Type - added 3.2 -data PyEnum_Type - added 3.2 -function PyErr_BadArgument - added 3.2 -function PyErr_BadInternalCall - added 3.2 -function PyErr_CheckSignals - added 3.2 -function PyErr_Clear - added 3.2 -function PyErr_Display - added 3.2 -function PyErr_ExceptionMatches - added 3.2 -function PyErr_Fetch - added 3.2 -function PyErr_Format - added 3.2 -function PyErr_GivenExceptionMatches - added 3.2 -function PyErr_NewException - added 3.2 -function PyErr_NewExceptionWithDoc - added 3.2 -function PyErr_NoMemory - added 3.2 -function PyErr_NormalizeException - added 3.2 -function PyErr_Occurred - added 3.2 -function PyErr_Print - added 3.2 -function PyErr_PrintEx - added 3.2 -function PyErr_ProgramText - added 3.2 -function PyErr_Restore - added 3.2 -function PyErr_SetFromErrno - added 3.2 -function PyErr_SetFromErrnoWithFilename - added 3.2 -function PyErr_SetFromErrnoWithFilenameObject - added 3.2 -function PyErr_SetInterrupt - added 3.2 -function PyErr_SetNone - added 3.2 -function PyErr_SetObject - added 3.2 -function PyErr_SetString - added 3.2 -function PyErr_SyntaxLocation - added 3.2 -function PyErr_WarnEx - added 3.2 -function PyErr_WarnExplicit - added 3.2 -function PyErr_WarnFormat - added 3.2 -function PyErr_WriteUnraisable - added 3.2 -function PyEval_AcquireLock - added 3.2 -function PyEval_AcquireThread - added 3.2 -function PyEval_CallFunction - added 3.2 -function PyEval_CallMethod - added 3.2 -function PyEval_CallObjectWithKeywords - added 3.2 -function PyEval_EvalCode - added 3.2 -function PyEval_EvalCodeEx - added 3.2 -function PyEval_EvalFrame - added 3.2 -function PyEval_EvalFrameEx - added 3.2 -function PyEval_GetBuiltins - added 3.2 -function PyEval_GetFrame - added 3.2 -function PyEval_GetFuncDesc - added 3.2 -function PyEval_GetFuncName - added 3.2 -function PyEval_GetGlobals - added 3.2 -function PyEval_GetLocals - added 3.2 -function PyEval_InitThreads - added 3.2 -function PyEval_ReleaseLock - added 3.2 -function PyEval_ReleaseThread - added 3.2 -function PyEval_RestoreThread - added 3.2 -function PyEval_SaveThread - added 3.2 -function PyEval_ThreadsInitialized - added 3.2 -data PyExc_ArithmeticError - added 3.2 -data PyExc_AssertionError - added 3.2 -data PyExc_AttributeError - added 3.2 -data PyExc_BaseException - added 3.2 -data PyExc_BaseExceptionGroup - added 3.11 -data PyExc_BufferError - added 3.2 -data PyExc_BytesWarning - added 3.2 -data PyExc_DeprecationWarning - added 3.2 -data PyExc_EOFError - added 3.2 -data PyExc_EnvironmentError - added 3.2 -data PyExc_Exception - added 3.2 -data PyExc_FloatingPointError - added 3.2 -data PyExc_FutureWarning - added 3.2 -data PyExc_GeneratorExit - added 3.2 -data PyExc_IOError - added 3.2 -data PyExc_ImportError - added 3.2 -data PyExc_ImportWarning - added 3.2 -data PyExc_IndentationError - added 3.2 -data PyExc_IndexError - added 3.2 -data PyExc_KeyError - added 3.2 -data PyExc_KeyboardInterrupt - added 3.2 -data PyExc_LookupError - added 3.2 -data PyExc_MemoryError - added 3.2 -data PyExc_NameError - added 3.2 -data PyExc_NotImplementedError - added 3.2 -data PyExc_OSError - added 3.2 -data PyExc_OverflowError - added 3.2 -data PyExc_PendingDeprecationWarning - added 3.2 -data PyExc_ReferenceError - added 3.2 -data PyExc_RuntimeError - added 3.2 -data PyExc_RuntimeWarning - added 3.2 -data PyExc_StopIteration - added 3.2 -data PyExc_SyntaxError - added 3.2 -data PyExc_SyntaxWarning - added 3.2 -data PyExc_SystemError - added 3.2 -data PyExc_SystemExit - added 3.2 -data PyExc_TabError - added 3.2 -data PyExc_TypeError - added 3.2 -data PyExc_UnboundLocalError - added 3.2 -data PyExc_UnicodeDecodeError - added 3.2 -data PyExc_UnicodeEncodeError - added 3.2 -data PyExc_UnicodeError - added 3.2 -data PyExc_UnicodeTranslateError - added 3.2 -data PyExc_UnicodeWarning - added 3.2 -data PyExc_UserWarning - added 3.2 -data PyExc_ValueError - added 3.2 -data PyExc_Warning - added 3.2 -data PyExc_ZeroDivisionError - added 3.2 -function PyException_GetCause - added 3.2 -function PyException_GetContext - added 3.2 -function PyException_GetTraceback - added 3.2 -function PyException_SetCause - added 3.2 -function PyException_SetContext - added 3.2 -function PyException_SetTraceback - added 3.2 -function PyFile_FromFd - added 3.2 -function PyFile_GetLine - added 3.2 -function PyFile_WriteObject - added 3.2 -function PyFile_WriteString - added 3.2 -data PyFilter_Type - added 3.2 -function PyFloat_AsDouble - added 3.2 -function PyFloat_FromDouble - added 3.2 -function PyFloat_FromString - added 3.2 -function PyFloat_GetInfo - added 3.2 -function PyFloat_GetMax - added 3.2 -function PyFloat_GetMin - added 3.2 -data PyFloat_Type - added 3.2 -function PyFrozenSet_New - added 3.2 -data PyFrozenSet_Type - added 3.2 -function PyGC_Collect - added 3.2 -function PyGILState_Ensure - added 3.2 -function PyGILState_GetThisThreadState - added 3.2 -function PyGILState_Release - added 3.2 -data PyGetSetDescr_Type - added 3.2 -function PyImport_AddModule - added 3.2 -function PyImport_AppendInittab - added 3.2 -function PyImport_ExecCodeModule - added 3.2 -function PyImport_ExecCodeModuleEx - added 3.2 -function PyImport_ExecCodeModuleWithPathnames - added 3.2 -function PyImport_GetImporter - added 3.2 -function PyImport_GetMagicNumber - added 3.2 -function PyImport_GetMagicTag - added 3.2 -function PyImport_GetModuleDict - added 3.2 -function PyImport_Import - added 3.2 -function PyImport_ImportFrozenModule - added 3.2 -function PyImport_ImportModule - added 3.2 -function PyImport_ImportModuleLevel - added 3.2 -function PyImport_ImportModuleNoBlock - added 3.2 -function PyImport_ReloadModule - added 3.2 -function PyInterpreterState_Clear - added 3.2 -function PyInterpreterState_Delete - added 3.2 -function PyInterpreterState_New - added 3.2 -function PyIter_Next - added 3.2 -data PyListIter_Type - added 3.2 -data PyListRevIter_Type - added 3.2 -function PyList_Append - added 3.2 -function PyList_AsTuple - added 3.2 -function PyList_GetItem - added 3.2 -function PyList_GetSlice - added 3.2 -function PyList_Insert - added 3.2 -function PyList_New - added 3.2 -function PyList_Reverse - added 3.2 -function PyList_SetItem - added 3.2 -function PyList_SetSlice - added 3.2 -function PyList_Size - added 3.2 -function PyList_Sort - added 3.2 -data PyList_Type - added 3.2 -data PyLongRangeIter_Type - added 3.2 -function PyLong_AsDouble - added 3.2 -function PyLong_AsLong - added 3.2 -function PyLong_AsLongAndOverflow - added 3.2 -function PyLong_AsLongLong - added 3.2 -function PyLong_AsLongLongAndOverflow - added 3.2 -function PyLong_AsSize_t - added 3.2 -function PyLong_AsSsize_t - added 3.2 -function PyLong_AsUnsignedLong - added 3.2 -function PyLong_AsUnsignedLongLong - added 3.2 -function PyLong_AsUnsignedLongLongMask - added 3.2 -function PyLong_AsUnsignedLongMask - added 3.2 -function PyLong_AsVoidPtr - added 3.2 -function PyLong_FromDouble - added 3.2 -function PyLong_FromLong - added 3.2 -function PyLong_FromLongLong - added 3.2 -function PyLong_FromSize_t - added 3.2 -function PyLong_FromSsize_t - added 3.2 -function PyLong_FromString - added 3.2 -function PyLong_FromUnsignedLong - added 3.2 -function PyLong_FromUnsignedLongLong - added 3.2 -function PyLong_FromVoidPtr - added 3.2 -function PyLong_GetInfo - added 3.2 -data PyLong_Type - added 3.2 -data PyMap_Type - added 3.2 -function PyMapping_Check - added 3.2 -function PyMapping_GetItemString - added 3.2 -function PyMapping_HasKey - added 3.2 -function PyMapping_HasKeyString - added 3.2 -function PyMapping_Items - added 3.2 -function PyMapping_Keys - added 3.2 -function PyMapping_Length - added 3.2 -function PyMapping_SetItemString - added 3.2 -function PyMapping_Size - added 3.2 -function PyMapping_Values - added 3.2 -function PyMem_Free - added 3.2 -function PyMem_Malloc - added 3.2 -function PyMem_Realloc - added 3.2 -data PyMemberDescr_Type - added 3.2 -function PyMemoryView_FromObject - added 3.2 -function PyMemoryView_GetContiguous - added 3.2 -data PyMemoryView_Type - added 3.2 -data PyMethodDescr_Type - added 3.2 -function PyModule_AddIntConstant - added 3.2 -function PyModule_AddObject - added 3.2 -function PyModule_AddStringConstant - added 3.2 -function PyModule_Create2 - added 3.2 -function PyModule_GetDef - added 3.2 -function PyModule_GetDict - added 3.2 -function PyModule_GetFilename - added 3.2 -function PyModule_GetFilenameObject - added 3.2 -function PyModule_GetName - added 3.2 -function PyModule_GetState - added 3.2 -function PyModule_New - added 3.2 -data PyModule_Type - added 3.2 -function PyNumber_Absolute - added 3.2 -function PyNumber_Add - added 3.2 -function PyNumber_And - added 3.2 -function PyNumber_AsSsize_t - added 3.2 -function PyNumber_Check - added 3.2 -function PyNumber_Divmod - added 3.2 -function PyNumber_Float - added 3.2 -function PyNumber_FloorDivide - added 3.2 -function PyNumber_InPlaceAdd - added 3.2 -function PyNumber_InPlaceAnd - added 3.2 -function PyNumber_InPlaceFloorDivide - added 3.2 -function PyNumber_InPlaceLshift - added 3.2 -function PyNumber_InPlaceMultiply - added 3.2 -function PyNumber_InPlaceOr - added 3.2 -function PyNumber_InPlacePower - added 3.2 -function PyNumber_InPlaceRemainder - added 3.2 -function PyNumber_InPlaceRshift - added 3.2 -function PyNumber_InPlaceSubtract - added 3.2 -function PyNumber_InPlaceTrueDivide - added 3.2 -function PyNumber_InPlaceXor - added 3.2 -function PyNumber_Index - added 3.2 -function PyNumber_Invert - added 3.2 -function PyNumber_Long - added 3.2 -function PyNumber_Lshift - added 3.2 -function PyNumber_Multiply - added 3.2 -function PyNumber_Negative - added 3.2 -function PyNumber_Or - added 3.2 -function PyNumber_Positive - added 3.2 -function PyNumber_Power - added 3.2 -function PyNumber_Remainder - added 3.2 -function PyNumber_Rshift - added 3.2 -function PyNumber_Subtract - added 3.2 -function PyNumber_ToBase - added 3.2 -function PyNumber_TrueDivide - added 3.2 -function PyNumber_Xor - added 3.2 -function PyOS_AfterFork - added 3.2 - ifdef HAVE_FORK -data PyOS_InputHook - added 3.2 -function PyOS_InterruptOccurred - added 3.2 -function PyOS_double_to_string - added 3.2 -function PyOS_getsig - added 3.2 -function PyOS_mystricmp - added 3.2 -function PyOS_mystrnicmp - added 3.2 -function PyOS_setsig - added 3.2 -function PyOS_snprintf - added 3.2 -function PyOS_string_to_double - added 3.2 -function PyOS_strtol - added 3.2 -function PyOS_strtoul - added 3.2 -function PyOS_vsnprintf - added 3.2 -function PyObject_ASCII - added 3.2 -function PyObject_AsFileDescriptor - added 3.2 -function PyObject_Bytes - added 3.2 -function PyObject_Call - added 3.2 -function PyObject_CallFunction - added 3.2 -function PyObject_CallFunctionObjArgs - added 3.2 -function PyObject_CallMethod - added 3.2 -function PyObject_CallMethodObjArgs - added 3.2 -function PyObject_CallObject - added 3.2 -function PyObject_ClearWeakRefs - added 3.2 -function PyObject_DelItem - added 3.2 -function PyObject_DelItemString - added 3.2 -function PyObject_Dir - added 3.2 -function PyObject_Format - added 3.2 -function PyObject_Free - added 3.2 -function PyObject_GC_Del - added 3.2 -function PyObject_GC_Track - added 3.2 -function PyObject_GC_UnTrack - added 3.2 -function PyObject_GenericGetAttr - added 3.2 -function PyObject_GenericSetAttr - added 3.2 -function PyObject_GetAttr - added 3.2 -function PyObject_GetAttrString - added 3.2 -function PyObject_GetItem - added 3.2 -function PyObject_GetIter - added 3.2 -function PyObject_HasAttr - added 3.2 -function PyObject_HasAttrString - added 3.2 -function PyObject_Hash - added 3.2 -function PyObject_HashNotImplemented - added 3.2 -function PyObject_Init - added 3.2 -function PyObject_InitVar - added 3.2 -function PyObject_IsInstance - added 3.2 -function PyObject_IsSubclass - added 3.2 -function PyObject_IsTrue - added 3.2 -function PyObject_Length - added 3.2 -function PyObject_Malloc - added 3.2 -function PyObject_Not - added 3.2 -function PyObject_Realloc - added 3.2 -function PyObject_Repr - added 3.2 -function PyObject_RichCompare - added 3.2 -function PyObject_RichCompareBool - added 3.2 -function PyObject_SelfIter - added 3.2 -function PyObject_SetAttr - added 3.2 -function PyObject_SetAttrString - added 3.2 -function PyObject_SetItem - added 3.2 -function PyObject_Size - added 3.2 -function PyObject_Str - added 3.2 -function PyObject_Type - added 3.2 -data PyProperty_Type - added 3.2 -data PyRangeIter_Type - added 3.2 -data PyRange_Type - added 3.2 -data PyReversed_Type - added 3.2 -function PySeqIter_New - added 3.2 -data PySeqIter_Type - added 3.2 -function PySequence_Check - added 3.2 -function PySequence_Concat - added 3.2 -function PySequence_Contains - added 3.2 -function PySequence_Count - added 3.2 -function PySequence_DelItem - added 3.2 -function PySequence_DelSlice - added 3.2 -function PySequence_Fast - added 3.2 -function PySequence_GetItem - added 3.2 -function PySequence_GetSlice - added 3.2 -function PySequence_In - added 3.2 -function PySequence_InPlaceConcat - added 3.2 -function PySequence_InPlaceRepeat - added 3.2 -function PySequence_Index - added 3.2 -function PySequence_Length - added 3.2 -function PySequence_List - added 3.2 -function PySequence_Repeat - added 3.2 -function PySequence_SetItem - added 3.2 -function PySequence_SetSlice - added 3.2 -function PySequence_Size - added 3.2 -function PySequence_Tuple - added 3.2 -data PySetIter_Type - added 3.2 -function PySet_Add - added 3.2 -function PySet_Clear - added 3.2 -function PySet_Contains - added 3.2 -function PySet_Discard - added 3.2 -function PySet_New - added 3.2 -function PySet_Pop - added 3.2 -function PySet_Size - added 3.2 -data PySet_Type - added 3.2 -function PySlice_GetIndices - added 3.2 -function PySlice_GetIndicesEx - added 3.2 -function PySlice_New - added 3.2 -data PySlice_Type - added 3.2 -function PyState_FindModule - added 3.2 -function PyStructSequence_GetItem - added 3.2 -function PyStructSequence_New - added 3.2 -function PyStructSequence_NewType - added 3.2 -function PyStructSequence_SetItem - added 3.2 -data PySuper_Type - added 3.2 -function PySys_AddWarnOption - added 3.2 -function PySys_AddWarnOptionUnicode - added 3.2 -function PySys_FormatStderr - added 3.2 -function PySys_FormatStdout - added 3.2 -function PySys_GetObject - added 3.2 -function PySys_HasWarnOptions - added 3.2 -function PySys_ResetWarnOptions - added 3.2 -function PySys_SetArgv - added 3.2 -function PySys_SetArgvEx - added 3.2 -function PySys_SetObject - added 3.2 -function PySys_SetPath - added 3.2 -function PySys_WriteStderr - added 3.2 -function PySys_WriteStdout - added 3.2 -function PyThreadState_Clear - added 3.2 -function PyThreadState_Delete - added 3.2 -function PyThreadState_DeleteCurrent - added 3.2 - abi_only -function PyThreadState_Get - added 3.2 -function PyThreadState_GetDict - added 3.2 -function PyThreadState_New - added 3.2 -function PyThreadState_SetAsyncExc - added 3.2 -function PyThreadState_Swap - added 3.2 -function PyTraceBack_Here - added 3.2 -function PyTraceBack_Print - added 3.2 -data PyTraceBack_Type - added 3.2 -data PyTupleIter_Type - added 3.2 -function PyTuple_GetItem - added 3.2 -function PyTuple_GetSlice - added 3.2 -function PyTuple_New - added 3.2 -function PyTuple_Pack - added 3.2 -function PyTuple_SetItem - added 3.2 -function PyTuple_Size - added 3.2 -data PyTuple_Type - added 3.2 -function PyType_ClearCache - added 3.2 -function PyType_GenericAlloc - added 3.2 -function PyType_GenericNew - added 3.2 -function PyType_GetFlags - added 3.2 -function PyType_IsSubtype - added 3.2 -function PyType_Modified - added 3.2 -function PyType_Ready - added 3.2 -data PyType_Type - added 3.2 -function PyUnicodeDecodeError_Create - added 3.2 -function PyUnicodeDecodeError_GetEncoding - added 3.2 -function PyUnicodeDecodeError_GetEnd - added 3.2 -function PyUnicodeDecodeError_GetObject - added 3.2 -function PyUnicodeDecodeError_GetReason - added 3.2 -function PyUnicodeDecodeError_GetStart - added 3.2 -function PyUnicodeDecodeError_SetEnd - added 3.2 -function PyUnicodeDecodeError_SetReason - added 3.2 -function PyUnicodeDecodeError_SetStart - added 3.2 -function PyUnicodeEncodeError_GetEncoding - added 3.2 -function PyUnicodeEncodeError_GetEnd - added 3.2 -function PyUnicodeEncodeError_GetObject - added 3.2 -function PyUnicodeEncodeError_GetReason - added 3.2 -function PyUnicodeEncodeError_GetStart - added 3.2 -function PyUnicodeEncodeError_SetEnd - added 3.2 -function PyUnicodeEncodeError_SetReason - added 3.2 -function PyUnicodeEncodeError_SetStart - added 3.2 -data PyUnicodeIter_Type - added 3.2 -function PyUnicodeTranslateError_GetEnd - added 3.2 -function PyUnicodeTranslateError_GetObject - added 3.2 -function PyUnicodeTranslateError_GetReason - added 3.2 -function PyUnicodeTranslateError_GetStart - added 3.2 -function PyUnicodeTranslateError_SetEnd - added 3.2 -function PyUnicodeTranslateError_SetReason - added 3.2 -function PyUnicodeTranslateError_SetStart - added 3.2 -function PyUnicode_Append - added 3.2 -function PyUnicode_AppendAndDel - added 3.2 -function PyUnicode_AsASCIIString - added 3.2 -function PyUnicode_AsCharmapString - added 3.2 -function PyUnicode_AsDecodedObject - added 3.2 -function PyUnicode_AsDecodedUnicode - added 3.2 -function PyUnicode_AsEncodedObject - added 3.2 -function PyUnicode_AsEncodedString - added 3.2 -function PyUnicode_AsEncodedUnicode - added 3.2 -function PyUnicode_AsLatin1String - added 3.2 -function PyUnicode_AsRawUnicodeEscapeString - added 3.2 -function PyUnicode_AsUTF16String - added 3.2 -function PyUnicode_AsUTF32String - added 3.2 -function PyUnicode_AsUTF8String - added 3.2 -function PyUnicode_AsUnicodeEscapeString - added 3.2 -function PyUnicode_AsWideChar - added 3.2 -function PyUnicode_Compare - added 3.2 -function PyUnicode_Concat - added 3.2 -function PyUnicode_Contains - added 3.2 -function PyUnicode_Count - added 3.2 -function PyUnicode_Decode - added 3.2 -function PyUnicode_DecodeASCII - added 3.2 -function PyUnicode_DecodeCharmap - added 3.2 -function PyUnicode_DecodeFSDefault - added 3.2 -function PyUnicode_DecodeFSDefaultAndSize - added 3.2 -function PyUnicode_DecodeLatin1 - added 3.2 -function PyUnicode_DecodeRawUnicodeEscape - added 3.2 -function PyUnicode_DecodeUTF16 - added 3.2 -function PyUnicode_DecodeUTF16Stateful - added 3.2 -function PyUnicode_DecodeUTF32 - added 3.2 -function PyUnicode_DecodeUTF32Stateful - added 3.2 -function PyUnicode_DecodeUTF8 - added 3.2 -function PyUnicode_DecodeUTF8Stateful - added 3.2 -function PyUnicode_DecodeUnicodeEscape - added 3.2 -function PyUnicode_FSConverter - added 3.2 -function PyUnicode_FSDecoder - added 3.2 -function PyUnicode_Find - added 3.2 -function PyUnicode_Format - added 3.2 -function PyUnicode_FromEncodedObject - added 3.2 -function PyUnicode_FromFormat - added 3.2 -function PyUnicode_FromFormatV - added 3.2 -function PyUnicode_FromObject - added 3.2 -function PyUnicode_FromOrdinal - added 3.2 -function PyUnicode_FromString - added 3.2 -function PyUnicode_FromStringAndSize - added 3.2 -function PyUnicode_FromWideChar - added 3.2 -function PyUnicode_GetDefaultEncoding - added 3.2 -function PyUnicode_GetSize - added 3.2 -function PyUnicode_IsIdentifier - added 3.2 -function PyUnicode_Join - added 3.2 -function PyUnicode_Partition - added 3.2 -function PyUnicode_RPartition - added 3.2 -function PyUnicode_RSplit - added 3.2 -function PyUnicode_Replace - added 3.2 -function PyUnicode_Resize - added 3.2 -function PyUnicode_RichCompare - added 3.2 -function PyUnicode_Split - added 3.2 -function PyUnicode_Splitlines - added 3.2 -function PyUnicode_Tailmatch - added 3.2 -function PyUnicode_Translate - added 3.2 -function PyUnicode_BuildEncodingMap - added 3.2 -function PyUnicode_CompareWithASCIIString - added 3.2 -function PyUnicode_DecodeUTF7 - added 3.2 -function PyUnicode_DecodeUTF7Stateful - added 3.2 -function PyUnicode_EncodeFSDefault - added 3.2 -function PyUnicode_InternFromString - added 3.2 -function PyUnicode_InternImmortal - added 3.2 -function PyUnicode_InternInPlace - added 3.2 -data PyUnicode_Type - added 3.2 -function PyWeakref_GetObject - added 3.2 -function PyWeakref_NewProxy - added 3.2 -function PyWeakref_NewRef - added 3.2 -data PyWrapperDescr_Type - added 3.2 -function PyWrapper_New - added 3.2 -data PyZip_Type - added 3.2 -function Py_AddPendingCall - added 3.2 -function Py_AtExit - added 3.2 -function Py_BuildValue - added 3.2 -function Py_CompileString - added 3.2 -function Py_DecRef - added 3.2 -function Py_EndInterpreter - added 3.2 -function Py_Exit - added 3.2 -function Py_FatalError - added 3.2 -data Py_FileSystemDefaultEncoding - added 3.2 -function Py_Finalize - added 3.2 -function Py_GetBuildInfo - added 3.2 -function Py_GetCompiler - added 3.2 -function Py_GetCopyright - added 3.2 -function Py_GetExecPrefix - added 3.2 -function Py_GetPath - added 3.2 -function Py_GetPlatform - added 3.2 -function Py_GetPrefix - added 3.2 -function Py_GetProgramFullPath - added 3.2 -function Py_GetProgramName - added 3.2 -function Py_GetPythonHome - added 3.2 -function Py_GetRecursionLimit - added 3.2 -function Py_GetVersion - added 3.2 -data Py_HasFileSystemDefaultEncoding - added 3.2 -function Py_IncRef - added 3.2 -function Py_Initialize - added 3.2 -function Py_InitializeEx - added 3.2 -function Py_IsInitialized - added 3.2 -function Py_Main - added 3.2 -function Py_MakePendingCalls - added 3.2 -function Py_NewInterpreter - added 3.2 -function Py_ReprEnter - added 3.2 -function Py_ReprLeave - added 3.2 -function Py_SetProgramName - added 3.2 -function Py_SetPythonHome - added 3.2 -function Py_SetRecursionLimit - added 3.2 -function Py_VaBuildValue - added 3.2 - -function _PyErr_BadInternalCall - added 3.2 - abi_only -function _PyObject_CallFunction_SizeT - added 3.2 - abi_only -function _PyObject_CallMethod_SizeT - added 3.2 - abi_only -function _PyObject_GC_New - added 3.2 - abi_only -function _PyObject_GC_NewVar - added 3.2 - abi_only -function _PyObject_GC_Resize - added 3.2 - abi_only -function _PyObject_New - added 3.2 - abi_only -function _PyObject_NewVar - added 3.2 - abi_only -function _PyState_AddModule - added 3.2 - abi_only -function _PyThreadState_Init - added 3.2 - abi_only -function _PyThreadState_Prealloc - added 3.2 - abi_only -data _PyWeakref_CallableProxyType - added 3.2 - abi_only -data _PyWeakref_ProxyType - added 3.2 - abi_only -data _PyWeakref_RefType - added 3.2 - abi_only -function _Py_BuildValue_SizeT - added 3.2 - abi_only -function _Py_CheckRecursiveCall - added 3.2 - abi_only -function _Py_Dealloc - added 3.2 - abi_only -data _Py_EllipsisObject - added 3.2 - abi_only -data _Py_FalseStruct - added 3.2 - abi_only -data _Py_NoneStruct - added 3.2 - abi_only -data _Py_NotImplementedStruct - added 3.2 - abi_only -data _Py_SwappedOp - added 3.2 - abi_only -data _Py_TrueStruct - added 3.2 - abi_only -function _Py_VaBuildValue_SizeT - added 3.2 - abi_only - -# Old buffer protocol support (deprecated) - -function PyObject_AsCharBuffer - added 3.2 -function PyObject_AsReadBuffer - added 3.2 -function PyObject_AsWriteBuffer - added 3.2 -function PyObject_CheckReadBuffer - added 3.2 - -# Flags are implicitly part of the ABI: - -const Py_TPFLAGS_DEFAULT - added 3.2 -const Py_TPFLAGS_BASETYPE - added 3.2 -const Py_TPFLAGS_HAVE_GC - added 3.2 - -const METH_VARARGS - added 3.2 -const METH_NOARGS - added 3.2 -const METH_O - added 3.2 -const METH_CLASS - added 3.2 -const METH_STATIC - added 3.2 -const METH_COEXIST - added 3.2 -# METH_STACKLESS is undocumented -# METH_FASTCALL is not part of limited API. - -# The following are defined in private headers, but historically -# they were exported as part of the stable ABI. -function PyMarshal_ReadObjectFromString - added 3.2 - abi_only -function PyMarshal_WriteObjectToString - added 3.2 - abi_only -function PyMember_GetOne - added 3.2 - abi_only -function PyMember_SetOne - added 3.2 - abi_only - -# TLS api is deprecated; superseded by TSS API - -function PyThread_ReInitTLS - added 3.2 -function PyThread_create_key - added 3.2 -function PyThread_delete_key - added 3.2 -function PyThread_set_key_value - added 3.2 -function PyThread_get_key_value - added 3.2 -function PyThread_delete_key_value - added 3.2 -function PyThread_acquire_lock - added 3.2 -function PyThread_acquire_lock_timed - added 3.2 -function PyThread_allocate_lock - added 3.2 -function PyThread_exit_thread - added 3.2 -function PyThread_free_lock - added 3.2 -function PyThread_get_stacksize - added 3.2 -function PyThread_get_thread_ident - added 3.2 -function PyThread_get_thread_native_id - ifdef PY_HAVE_THREAD_NATIVE_ID - added 3.2 -function PyThread_init_thread - added 3.2 -function PyThread_release_lock - added 3.2 -function PyThread_set_stacksize - added 3.2 -function PyThread_start_new_thread - added 3.2 - -# The following were added in PC/python3.def in Python 3.3: -# 7800f75827b1be557be16f3b18f5170fbf9fae08 -# 9c56409d3353b8cd4cfc19e0467bbe23fd34fc92 -# 75aeaa9b18667219bbacbc58ba6efecccef9dfbd - -function PyState_AddModule - added 3.3 -function PyState_RemoveModule - added 3.3 -function PyType_FromSpecWithBases - added 3.3 -function _PyArg_Parse_SizeT - added 3.3 - abi_only -function _PyArg_ParseTuple_SizeT - added 3.3 - abi_only -function _PyArg_ParseTupleAndKeywords_SizeT - added 3.3 - abi_only -function _PyArg_VaParse_SizeT - added 3.3 - abi_only -function _PyArg_VaParseTupleAndKeywords_SizeT - added 3.3 - abi_only -function PyThread_GetInfo - added 3.3 - -# The following were added in PC/python3.def in Python 3.4: -# 3ba3a3ee56c142e93d6bbe20ff6bf939212a30f0 - -function PyCFunction_New - added 3.4 -function PyType_GetSlot - added 3.4 - -# The following were added in PC/python3.def in Python 3.5: -# 11d7b1423fc44d764eba7065ea5eba58ed748b21 -# f3b73ad51da3097d7915796fdc62608b1ab90c0a - -function PyErr_FormatV - added 3.5 -function PyModuleDef_Init - added 3.5 -data PyModuleDef_Type - added 3.5 - -# New slots in 3.5: -# d51374ed78a3e3145911a16cdf3b9b84b3ba7d15 - Matrix multiplication (PEP 465) -# 7544508f0245173bff5866aa1598c8f6cce1fc5f - Async iterators (PEP 492) -# 0969a9f8abcf98bb43ea77b1dd050426adcfb4f7 - tp_finalize - -const Py_nb_matrix_multiply - added 3.5 -const Py_nb_inplace_matrix_multiply - added 3.5 -const Py_am_await - added 3.5 -const Py_am_aiter - added 3.5 -const Py_am_anext - added 3.5 -const Py_tp_finalize - added 3.5 - -# The following were added in PC/python3.def in Python 3.6: - -function Py_FinalizeEx - added 3.6 - -function PyOS_FSPath - added 3.6 -function PyErr_ResourceWarning - added 3.6 -function PyErr_SetImportErrorSubclass - added 3.6 -data PyExc_ModuleNotFoundError - added 3.6 - -# The following were added in PC/python3.def in Python 3.6.1 and 3.5.3/3.5.4: - -function PyCodec_NameReplaceErrors - added 3.7 # (and 3.6.1 and 3.5.3) -function PyErr_GetExcInfo - added 3.7 # (and 3.6.1 and 3.5.3) -function PyErr_SetExcInfo - added 3.7 # (and 3.6.1 and 3.5.3) -function PyErr_SetFromErrnoWithFilenameObjects - added 3.7 # (and 3.6.1 and 3.5.3) -function PyErr_SetImportError - added 3.7 # (and 3.6.1 and 3.5.3) -function PyErr_SyntaxLocationEx - added 3.7 # (and 3.6.1 and 3.5.3) -data PyExc_BlockingIOError - added 3.7 # (and 3.6.1 and 3.5.3) -data PyExc_BrokenPipeError - added 3.7 # (and 3.6.1 and 3.5.3) -data PyExc_ChildProcessError - added 3.7 # (and 3.6.1 and 3.5.3) -data PyExc_ConnectionAbortedError - added 3.7 # (and 3.6.1 and 3.5.3) -data PyExc_ConnectionError - added 3.7 # (and 3.6.1 and 3.5.3) -data PyExc_ConnectionRefusedError - added 3.7 # (and 3.6.1 and 3.5.3) -data PyExc_ConnectionResetError - added 3.7 # (and 3.6.1 and 3.5.3) -data PyExc_FileExistsError - added 3.7 # (and 3.6.1 and 3.5.3) -data PyExc_FileNotFoundError - added 3.7 # (and 3.6.1 and 3.5.3) -data PyExc_InterruptedError - added 3.7 # (and 3.6.1 and 3.5.3) -data PyExc_IsADirectoryError - added 3.7 # (and 3.6.1 and 3.5.3) -data PyExc_NotADirectoryError - added 3.7 # (and 3.6.1 and 3.5.3) -data PyExc_PermissionError - added 3.7 # (and 3.6.1 and 3.5.3) -data PyExc_ProcessLookupError - added 3.7 # (and 3.6.1 and 3.5.3) -data PyExc_RecursionError - added 3.7 # (and 3.6.1 and 3.5.3) -data PyExc_ResourceWarning - added 3.7 # (and 3.6.1 and 3.5.3) -data PyExc_StopAsyncIteration - added 3.7 # (and 3.6.1 and 3.5.3) -data PyExc_TimeoutError - added 3.7 # (and 3.6.1 and 3.5.3) -function PyImport_AddModuleObject - added 3.7 # (and 3.6.1 and 3.5.3) -function PyImport_ExecCodeModuleObject - added 3.7 # (and 3.6.1 and 3.5.3) -function PyImport_ImportFrozenModuleObject - added 3.7 # (and 3.6.1 and 3.5.3) -function PyImport_ImportModuleLevelObject - added 3.7 # (and 3.6.1 and 3.5.3) -function PyMem_Calloc - added 3.7 # (and 3.6.1 and 3.5.3) -function PyMemoryView_FromMemory - added 3.7 # (and 3.6.1 and 3.5.3) -function PyModule_AddFunctions - added 3.7 # (and 3.6.1 and 3.5.3) -function PyModule_ExecDef - added 3.7 # (and 3.6.1 and 3.5.3) -function PyModule_FromDefAndSpec2 - added 3.7 # (and 3.6.1 and 3.5.3) -function PyModule_GetNameObject - added 3.7 # (and 3.6.1 and 3.5.3) -function PyModule_NewObject - added 3.7 # (and 3.6.1 and 3.5.3) -function PyModule_SetDocString - added 3.7 # (and 3.6.1 and 3.5.3) -function PyNumber_InPlaceMatrixMultiply - added 3.7 # (and 3.6.1 and 3.5.3) -function PyNumber_MatrixMultiply - added 3.7 # (and 3.6.1 and 3.5.3) -function PyObject_Calloc - added 3.7 # (and 3.6.1 and 3.5.3) -function PyObject_GenericSetDict - added 3.7 # (and 3.6.1 and 3.5.3) -function PySys_AddXOption - added 3.7 # (and 3.6.1 and 3.5.3) -function PySys_GetXOptions - added 3.7 # (and 3.6.1 and 3.5.3) -function PyUnicode_AsUCS4 - added 3.7 # (and 3.6.1 and 3.5.3) -function PyUnicode_AsUCS4Copy - added 3.7 # (and 3.6.1 and 3.5.3) -function PyUnicode_AsWideCharString - added 3.7 # (and 3.6.1 and 3.5.3) -function PyUnicode_DecodeLocale - added 3.7 # (and 3.6.1 and 3.5.3) -function PyUnicode_DecodeLocaleAndSize - added 3.7 # (and 3.6.1 and 3.5.3) -function PyUnicode_EncodeLocale - added 3.7 # (and 3.6.1 and 3.5.3) -function PyUnicode_FindChar - added 3.7 # (and 3.6.1 and 3.5.3) -function PyUnicode_GetLength - added 3.7 # (and 3.6.1 and 3.5.3) -function PyUnicode_ReadChar - added 3.7 # (and 3.6.1 and 3.5.3) -function PyUnicode_Substring - added 3.7 # (and 3.6.1 and 3.5.3) -function PyUnicode_WriteChar - added 3.7 # (and 3.6.1 and 3.5.3) -function Py_DecodeLocale - added 3.7 # (and 3.6.1 and 3.5.3) -function Py_EncodeLocale - added 3.7 # (and 3.6.1 and 3.5.3) -function Py_SetPath - added 3.7 # (and 3.6.1 and 3.5.3) -function PyErr_SetExcFromWindowsErr - added 3.7 # (and 3.6.1 and 3.5.3) - ifdef MS_WINDOWS -function PyErr_SetExcFromWindowsErrWithFilename - added 3.7 # (and 3.6.1 and 3.5.3) - ifdef MS_WINDOWS -function PyErr_SetExcFromWindowsErrWithFilenameObject - added 3.7 # (and 3.6.1 and 3.5.3) - ifdef MS_WINDOWS -function PyErr_SetExcFromWindowsErrWithFilenameObjects - added 3.7 # (and 3.6.1 and 3.5.3) - ifdef MS_WINDOWS -function PyErr_SetFromWindowsErr - added 3.7 # (and 3.6.1 and 3.5.3) - ifdef MS_WINDOWS -function PyErr_SetFromWindowsErrWithFilename - added 3.7 # (and 3.6.1 and 3.5.3) - ifdef MS_WINDOWS -data PyExc_WindowsError - added 3.7 # (and 3.6.1 and 3.5.3) - ifdef MS_WINDOWS -function PyOS_CheckStack - added 3.7 # (and 3.6.1 and 3.5.3) - ifdef USE_STACKCHECK -function PyUnicode_AsMBCSString - added 3.7 # (and 3.6.1 and 3.5.3) - ifdef MS_WINDOWS -function PyUnicode_DecodeCodePageStateful - added 3.7 # (and 3.6.1 and 3.5.3) - ifdef MS_WINDOWS -function PyUnicode_DecodeMBCS - added 3.7 # (and 3.6.1 and 3.5.3) - ifdef MS_WINDOWS -function PyUnicode_DecodeMBCSStateful - added 3.7 # (and 3.6.1 and 3.5.3) - ifdef MS_WINDOWS -function PyUnicode_EncodeCodePage - added 3.7 # (and 3.6.1 and 3.5.3) - ifdef MS_WINDOWS - -# 3.5.4: -function PySlice_AdjustIndices - added 3.7 # (and 3.6.1 and 3.5.4) -function PySlice_Unpack - added 3.7 # (and 3.6.1 and 3.5.4) - -# The following were added in PC/python3.def in Python 3.7: - -function PyInterpreterState_GetID - added 3.7 -function PyThread_tss_alloc - added 3.7 -function PyThread_tss_create - added 3.7 -function PyThread_tss_delete - added 3.7 -function PyThread_tss_free - added 3.7 -function PyThread_tss_get - added 3.7 -function PyThread_tss_is_created - added 3.7 -function PyThread_tss_set - added 3.7 -function PyOS_BeforeFork - added 3.7 - ifdef HAVE_FORK -function PyOS_AfterFork_Parent - added 3.7 - ifdef HAVE_FORK -function PyOS_AfterFork_Child - added 3.7 - ifdef HAVE_FORK - -# New method flags in 3.7 (PEP 590): - -const METH_FASTCALL - added 3.7 -const METH_METHOD - added 3.7 - -# The following were added in PC/python3.def in Python 3.8: - -function PyImport_GetModule - added 3.8 -data Py_UTF8Mode - added 3.8 -function PyExceptionClass_Name - added 3.8 -function PyIndex_Check - added 3.8 -function PyIter_Check - added 3.8 -data PyDictRevIterItem_Type - added 3.8 -data PyDictRevIterKey_Type - added 3.8 -data PyDictRevIterValue_Type - added 3.8 -function PyInterpreterState_GetDict - added 3.8 -function Py_BytesMain - added 3.8 - -# New type flag (PEP 590): - -const Py_TPFLAGS_METHOD_DESCRIPTOR - added 3.8 - -# The following were added in PC/python3.def in Python 3.9: - -function Py_EnterRecursiveCall - added 3.9 -function Py_LeaveRecursiveCall - added 3.9 -function Py_GenericAlias - added 3.9 -data Py_GenericAliasType - added 3.9 -function PyCMethod_New - added 3.9 # Windows: 3.10 & 3.9.2 -- https://bugs.python.org/issue43155 -function PyInterpreterState_Get - added 3.9 -function PyObject_GC_IsFinalized - added 3.9 -function PyObject_GC_IsTracked - added 3.9 - -# The following were added in PC/python3.def in Python 3.10: - -function Py_GetArgcArgv - added 3.10 - abi_only -function PyIter_Send - added 3.10 -function PyUnicode_AsUTF8AndSize - added 3.10 -function PyObject_GenericGetDict - added 3.10 -function Py_NewRef - added 3.10 -function Py_XNewRef - added 3.10 -function PyModule_AddType - added 3.10 -function PyType_FromModuleAndSpec - added 3.10 -function PyType_GetModule - added 3.10 -function PyType_GetModuleState - added 3.10 -function PyFrame_GetLineNumber - added 3.10 -function PyFrame_GetCode - added 3.10 -function PyObject_CallNoArgs - added 3.10 -function PyThreadState_GetFrame - added 3.10 -function PyThreadState_GetID - added 3.10 -function PyThreadState_GetInterpreter - added 3.10 -function PyModule_AddObjectRef - added 3.10 -data Py_FileSystemDefaultEncodeErrors - added 3.10 -function PyCodec_Unregister - added 3.10 -function PyErr_SetInterruptEx - added 3.10 -function Py_Is - added 3.10 -function Py_IsTrue - added 3.10 -function Py_IsFalse - added 3.10 -function Py_IsNone - added 3.10 -function _Py_IncRef - added 3.10 - abi_only -function _Py_DecRef - added 3.10 - abi_only -function PyAIter_Check - added 3.10 -function PyObject_GetAIter - added 3.10 -data PyExc_EncodingWarning - added 3.10 - -# Support for Stable ABI in debug builds - -data _Py_RefTotal - added 3.10 - abi_only - ifdef Py_REF_DEBUG -function _Py_NegativeRefcount - added 3.10 - abi_only - ifdef Py_REF_DEBUG - -# New slots in 3.10: - -const Py_am_send - added 3.10 - - -# New GC control functions in Py3.10 (https://bugs.python.org/issue28254) - -function PyGC_Disable - added 3.10 -function PyGC_Enable - added 3.10 -function PyGC_IsEnabled - added 3.10 - -# Add new C API in Python 3.11 - -function PyType_GetName - added 3.11 -function PyType_GetQualName - added 3.11 -data PyStructSequence_UnnamedField - added 3.11 - -# Add stable Py_buffer API in Python 3.11 (https://bugs.python.org/issue45459) -struct Py_buffer - added 3.11 - full-abi -function PyObject_CheckBuffer - added 3.11 -function PyObject_GetBuffer - added 3.11 -function PyBuffer_GetPointer - added 3.11 -function PyBuffer_SizeFromFormat - added 3.11 -function PyBuffer_ToContiguous - added 3.11 -function PyBuffer_FromContiguous - added 3.11 -function PyObject_CopyData - added 3.11 -function PyBuffer_IsContiguous - added 3.11 -function PyBuffer_FillContiguousStrides - added 3.11 -function PyBuffer_FillInfo - added 3.11 -function PyBuffer_Release - added 3.11 -function PyMemoryView_FromBuffer - added 3.11 - -# (Detailed comments aren't really needed for further entries: from here on -# we can use version control logs.) - -data Py_Version - added 3.11 -function PyErr_GetHandledException - added 3.11 -function PyErr_SetHandledException - added 3.11 - diff --git a/Tools/scripts/stable_abi.py b/Tools/scripts/stable_abi.py index 54075248c7ea21..f5a9f8d2dd617b 100755 --- a/Tools/scripts/stable_abi.py +++ b/Tools/scripts/stable_abi.py @@ -14,8 +14,10 @@ import sysconfig import argparse import textwrap +import tomllib import difflib import shutil +import pprint import sys import os import os.path @@ -46,17 +48,15 @@ UNIXY = MACOS or (sys.platform == "linux") # XXX should this be "not Windows"? -# The stable ABI manifest (Misc/stable_abi.txt) exists only to fill the +# The stable ABI manifest (Misc/stable_abi.toml) exists only to fill the # following dataclasses. # Feel free to change its syntax (and the `parse_manifest` function) # to better serve that purpose (while keeping it human-readable). - at dataclasses.dataclass class Manifest: """Collection of `ABIItem`s forming the stable ABI/limited API.""" - - kind = 'manifest' - contents: dict = dataclasses.field(default_factory=dict) + def __init__(self): + self.contents = dict() def add(self, item): if item.name in self.contents: @@ -65,14 +65,6 @@ def add(self, item): raise ValueError(f'duplicate ABI item {item.name}') self.contents[item.name] = item - @property - def feature_defines(self): - """Return all feature defines which affect what's available - - These are e.g. HAVE_FORK and MS_WINDOWS. - """ - return set(item.ifdef for item in self.contents.values()) - {None} - def select(self, kinds, *, include_abi_only=True, ifdef=None): """Yield selected items of the manifest @@ -81,7 +73,7 @@ def select(self, kinds, *, include_abi_only=True, ifdef=None): stable ABI. If False, include only items from the limited API (i.e. items people should use today) - ifdef: set of feature defines (e.g. {'HAVE_FORK', 'MS_WINDOWS'}). + ifdef: set of feature macros (e.g. {'HAVE_FORK', 'MS_WINDOWS'}). If None (default), items are not filtered by this. (This is different from the empty set, which filters out all such conditional items.) @@ -99,109 +91,74 @@ def select(self, kinds, *, include_abi_only=True, ifdef=None): def dump(self): """Yield lines to recreate the manifest file (sans comments/newlines)""" - # Recursive in preparation for struct member & function argument nodes for item in self.contents.values(): - yield from item.dump(indent=0) - + fields = dataclasses.fields(item) + yield f"[{item.kind}.{item.name}]" + for field in fields: + if field.name in {'name', 'value', 'kind'}: + continue + value = getattr(item, field.name) + if value == field.default: + pass + elif value is True: + yield f" {field.name} = true" + elif value: + yield f" {field.name} = {value!r}" + + +itemclasses = {} +def itemclass(kind): + """Register the decorated class in `itemclasses`""" + def decorator(cls): + itemclasses[kind] = cls + return cls + return decorator + + at itemclass('function') + at itemclass('macro') + at itemclass('data') + at itemclass('const') + at itemclass('typedef') @dataclasses.dataclass class ABIItem: """Information on one item (function, macro, struct, etc.)""" - kind: str name: str + kind: str added: str = None - contents: list = dataclasses.field(default_factory=list) abi_only: bool = False ifdef: str = None - struct_abi_kind: str = None - members: list = None - doc: str = None + + at itemclass('feature_macro') + at dataclasses.dataclass(kw_only=True) +class FeatureMacro(ABIItem): + name: str + doc: str windows: bool = False + abi_only: bool = True - KINDS = frozenset({ - 'struct', 'function', 'macro', 'data', 'const', 'typedef', 'ifdef', - }) + at itemclass('struct') + at dataclasses.dataclass(kw_only=True) +class Struct(ABIItem): + struct_abi_kind: str + members: list = None - def dump(self, indent=0): - yield f"{' ' * indent}{self.kind} {self.name}" - if self.added: - yield f"{' ' * (indent+1)}added {self.added}" - if self.ifdef: - yield f"{' ' * (indent+1)}ifdef {self.ifdef}" - if self.abi_only: - yield f"{' ' * (indent+1)}abi_only" def parse_manifest(file): """Parse the given file (iterable of lines) to a Manifest""" - LINE_RE = re.compile('(?P[ ]*)(?P[^ ]+)[ ]*(?P.*)') manifest = Manifest() - # parents of currently processed line, each with its indentation level - levels = [(manifest, -1)] + data = tomllib.load(file) - def raise_error(msg): - raise SyntaxError(f'line {lineno}: {msg}') - - for lineno, line in enumerate(file, start=1): - line, sep, comment = line.partition('#') - line = line.rstrip() - if not line: - continue - match = LINE_RE.fullmatch(line) - if not match: - raise_error(f'invalid syntax: {line}') - level = len(match['indent']) - kind = match['kind'] - content = match['content'] - while level <= levels[-1][1]: - levels.pop() - parent = levels[-1][0] - entry = None - if parent.kind == 'manifest': - if kind not in kind in ABIItem.KINDS: - raise_error(f'{kind} cannot go in {parent.kind}') - entry = ABIItem(kind, content) - parent.add(entry) - elif kind in {'added', 'ifdef'}: - if parent.kind not in ABIItem.KINDS: - raise_error(f'{kind} cannot go in {parent.kind}') - setattr(parent, kind, content) - elif kind in {'abi_only'}: - if parent.kind not in {'function', 'data'}: - raise_error(f'{kind} cannot go in {parent.kind}') - parent.abi_only = True - elif kind in {'members', 'full-abi', 'opaque'}: - if parent.kind not in {'struct'}: - raise_error(f'{kind} cannot go in {parent.kind}') - if prev := getattr(parent, 'struct_abi_kind', None): - raise_error( - f'{parent.name} already has {prev}, cannot add {kind}') - parent.struct_abi_kind = kind - if kind == 'members': - parent.members = content.split() - elif kind in {'doc'}: - if parent.kind not in {'ifdef'}: - raise_error(f'{kind} cannot go in {parent.kind}') - parent.doc = content - elif kind in {'windows'}: - if parent.kind not in {'ifdef'}: - raise_error(f'{kind} cannot go in {parent.kind}') - if not content: - parent.windows = True - elif content == 'maybe': - parent.windows = content - else: - raise_error(f'Unexpected: {content}') - else: - raise_error(f"unknown kind {kind!r}") - # When adding more, update the comment in stable_abi.txt. - levels.append((entry, level)) - - ifdef_names = {i.name for i in manifest.select({'ifdef'})} - for item in manifest.contents.values(): - if item.ifdef and item.ifdef not in ifdef_names: - raise ValueError(f'{item.name} uses undeclared ifdef {item.ifdef}') + for kind, itemclass in itemclasses.items(): + for name, item_data in data[kind].items(): + try: + item = itemclass(name=name, kind=kind, **item_data) + manifest.add(item) + except BaseException as exc: + exc.add_note(f'in {kind} {name}') + raise return manifest @@ -246,12 +203,14 @@ def gen_python3dll(manifest, args, outfile): def sort_key(item): return item.name.lower() - windows_ifdefs = { - item.name for item in manifest.select({'ifdef'}) if item.windows + windows_feature_macros = { + item.name for item in manifest.select({'feature_macro'}) if item.windows } for item in sorted( manifest.select( - {'function'}, include_abi_only=True, ifdef=windows_ifdefs), + {'function'}, + include_abi_only=True, + ifdef=windows_feature_macros), key=sort_key): write(f'EXPORT_FUNC({item.name})') @@ -259,7 +218,9 @@ def sort_key(item): for item in sorted( manifest.select( - {'data'}, include_abi_only=True, ifdef=windows_ifdefs), + {'data'}, + include_abi_only=True, + ifdef=windows_feature_macros), key=sort_key): write(f'EXPORT_DATA({item.name})') @@ -285,17 +246,20 @@ def gen_doc_annotations(manifest, args, outfile): ifdef_note = manifest.contents[item.ifdef].doc else: ifdef_note = None - writer.writerow({ + row = { 'role': REST_ROLES[item.kind], 'name': item.name, 'added': item.added, - 'ifdef_note': ifdef_note, - 'struct_abi_kind': item.struct_abi_kind}) - for member_name in item.members or (): - writer.writerow({ - 'role': 'member', - 'name': f'{item.name}.{member_name}', - 'added': item.added}) + 'ifdef_note': ifdef_note} + rows = [row] + if item.kind == 'struct': + row['struct_abi_kind'] = item.struct_abi_kind + for member_name in item.members or (): + rows.append({ + 'role': 'member', + 'name': f'{item.name}.{member_name}', + 'added': item.added}) + writer.writerows(rows) @generator("ctypes_test", 'Lib/test/test_stable_abi_ctypes.py') def gen_ctypes_test(manifest, args, outfile): @@ -323,7 +287,8 @@ def test_available_symbols(self): ctypes_test.pythonapi[symbol_name] def test_feature_macros(self): - self.assertEqual(set(get_feature_macros()), EXPECTED_IFDEFS) + self.assertEqual( + set(get_feature_macros()), EXPECTED_FEATURE_MACROS) # The feature macros for Windows are used in creating the DLL # definition, so they must be known on all platforms. @@ -331,7 +296,7 @@ def test_feature_macros(self): # the reality. @unittest.skipIf(sys.platform != "win32", "Windows specific test") def test_windows_feature_macros(self): - for name, value in WINDOWS_IFDEFS.items(): + for name, value in WINDOWS_FEATURE_MACROS.items(): if value != 'maybe': with self.subTest(name): self.assertEqual(feature_macros[name], value) @@ -342,7 +307,7 @@ def test_windows_feature_macros(self): {'function', 'data'}, include_abi_only=True, ) - ifdef_items = {} + optional_items = {} for item in items: if item.name in ( # Some symbols aren't exported on all platforms. @@ -351,23 +316,23 @@ def test_windows_feature_macros(self): ): continue if item.ifdef: - ifdef_items.setdefault(item.ifdef, []).append(item.name) + optional_items.setdefault(item.ifdef, []).append(item.name) else: write(f' "{item.name}",') write(")") - for ifdef, names in ifdef_items.items(): + for ifdef, names in optional_items.items(): write(f"if feature_macros[{ifdef!r}]:") write(f" SYMBOL_NAMES += (") for name in names: write(f" {name!r},") write(" )") write("") - write(f"EXPECTED_IFDEFS = set({sorted(ifdef_items)})") + feature_macros = list(manifest.select({'feature_macro'})) + feature_names = sorted(m.name for m in feature_macros) + write(f"EXPECTED_FEATURE_MACROS = set({pprint.pformat(feature_names)})") - windows_ifdef_values = { - name: manifest.contents[name].windows for name in ifdef_items - } - write(f"WINDOWS_IFDEFS = {windows_ifdef_values}") + windows_feature_macros = {m.name: m.windows for m in feature_macros} + write(f"WINDOWS_FEATURE_MACROS = {pprint.pformat(windows_feature_macros)}") @generator("testcapi_feature_macros", 'Modules/_testcapi_feature_macros.inc') @@ -378,7 +343,7 @@ def gen_testcapi_feature_macros(manifest, args, outfile): write() write('// Add an entry in dict `result` for each Stable ABI feature macro.') write() - for macro in manifest.select({'ifdef'}): + for macro in manifest.select({'feature_macro'}): name = macro.name write(f'#ifdef {name}') write(f' res = PyDict_SetItemString(result, "{name}", Py_True);') @@ -425,7 +390,8 @@ def do_unixy_check(manifest, args): # Get all macros first: we'll need feature macros like HAVE_FORK and # MS_WINDOWS for everything else present_macros = gcc_get_limited_api_macros(['Include/Python.h']) - feature_defines = manifest.feature_defines & present_macros + feature_macros = set(m.name for m in manifest.select({'feature_macro'})) + feature_macros &= present_macros # Check that we have all needed macros expected_macros = set( @@ -438,7 +404,7 @@ def do_unixy_check(manifest, args): + 'with Py_LIMITED_API:') expected_symbols = set(item.name for item in manifest.select( - {'function', 'data'}, include_abi_only=True, ifdef=feature_defines, + {'function', 'data'}, include_abi_only=True, ifdef=feature_macros, )) # Check the static library (*.a) @@ -458,7 +424,7 @@ def do_unixy_check(manifest, args): # Check definitions in the header files expected_defs = set(item.name for item in manifest.select( - {'function', 'data'}, include_abi_only=False, ifdef=feature_defines, + {'function', 'data'}, include_abi_only=False, ifdef=feature_macros, )) found_defs = gcc_get_limited_api_definitions(['Include/Python.h']) missing_defs = expected_defs - found_defs @@ -635,6 +601,28 @@ def check_private_names(manifest): f'`{name}` is private (underscore-prefixed) and should be ' + 'removed from the stable ABI list or or marked `abi_only`') +def check_dump(manifest, filename): + """Check that manifest.dump() corresponds to the data. + + Mainly useful when debugging this script. + """ + dumped = tomllib.loads('\n'.join(manifest.dump())) + with filename.open('rb') as file: + from_file = tomllib.load(file) + if dumped != from_file: + print(f'Dump differs from loaded data!', file=sys.stderr) + diff = difflib.unified_diff( + pprint.pformat(dumped).splitlines(), + pprint.pformat(from_file).splitlines(), + '', str(filename), + lineterm='', + ) + for line in diff: + print(line, file=sys.stderr) + return False + else: + return True + def main(): parser = argparse.ArgumentParser( description=__doc__, @@ -696,7 +684,16 @@ def main(): run_all_generators = True args.unixy_check = True - with args.file.open() as file: + try: + file = args.file.open('rb') + except FileNotFoundError as err: + if args.file.suffix == '.txt': + # Provide a better error message + suggestion = args.file.with_suffix('.toml') + raise FileNotFoundError( + f'{args.file} not found. Did you mean {suggestion} ?') from err + raise + with file: manifest = parse_manifest(file) check_private_names(manifest) @@ -709,7 +706,7 @@ def main(): if args.dump: for line in manifest.dump(): print(line) - results['dump'] = True + results['dump'] = check_dump(manifest, args.file) for gen in generators: filename = getattr(args, gen.var_name) From webhook-mailer at python.org Fri Apr 29 14:22:56 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Fri, 29 Apr 2022 18:22:56 -0000 Subject: [Python-checkins] bpo-26792: Improve docstrings of runpy module run_functions (#30729) Message-ID: https://github.com/python/cpython/commit/117836f123a1c65d9ba50401822b883f11f0a347 commit: 117836f123a1c65d9ba50401822b883f11f0a347 branch: main author: Humbled Drugman committer: JelleZijlstra date: 2022-04-29T12:22:46-06:00 summary: bpo-26792: Improve docstrings of runpy module run_functions (#30729) Co-authored-by: Jelle Zijlstra Co-authored-by: Kumar Aditya <59607654+kumaraditya303 at users.noreply.github.com> files: A Misc/NEWS.d/next/Documentation/2022-01-23-20-44-53.bpo-26792.dQ1v1W.rst M Lib/runpy.py diff --git a/Lib/runpy.py b/Lib/runpy.py index 949fd5939ba2c..54fc136d4074f 100644 --- a/Lib/runpy.py +++ b/Lib/runpy.py @@ -200,9 +200,24 @@ def _run_module_as_main(mod_name, alter_argv=True): def run_module(mod_name, init_globals=None, run_name=None, alter_sys=False): - """Execute a module's code without importing it + """Execute a module's code without importing it. - Returns the resulting top level namespace dictionary + mod_name -- an absolute module name or package name. + + Optional arguments: + init_globals -- dictionary used to pre-populate the module?s + globals dictionary before the code is executed. + + run_name -- if not None, this will be used for setting __name__; + otherwise, __name__ will be set to mod_name + '__main__' if the + named module is a package and to just mod_name otherwise. + + alter_sys -- if True, sys.argv[0] is updated with the value of + __file__ and sys.modules[__name__] is updated with a temporary + module object for the module being executed. Both are + restored to their original values before the function returns. + + Returns the resulting module globals dictionary. """ mod_name, mod_spec, code = _get_module_details(mod_name) if run_name is None: @@ -245,14 +260,19 @@ def _get_code_from_file(run_name, fname): return code, fname def run_path(path_name, init_globals=None, run_name=None): - """Execute code located at the specified filesystem location + """Execute code located at the specified filesystem location. + + path_name -- filesystem location of a Python script, zipfile, + or directory containing a top level __main__.py script. + + Optional arguments: + init_globals -- dictionary used to pre-populate the module?s + globals dictionary before the code is executed. - Returns the resulting top level namespace dictionary + run_name -- if not None, this will be used to set __name__; + otherwise, '' will be used for __name__. - The file path may refer directly to a Python script (i.e. - one that could be directly executed with execfile) or else - it may refer to a zipfile or directory containing a top - level __main__.py script. + Returns the resulting module globals dictionary. """ if run_name is None: run_name = "" diff --git a/Misc/NEWS.d/next/Documentation/2022-01-23-20-44-53.bpo-26792.dQ1v1W.rst b/Misc/NEWS.d/next/Documentation/2022-01-23-20-44-53.bpo-26792.dQ1v1W.rst new file mode 100644 index 0000000000000..64a3956447601 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2022-01-23-20-44-53.bpo-26792.dQ1v1W.rst @@ -0,0 +1,2 @@ +Improve the docstrings of :func:`runpy.run_module` and :func:`runpy.run_path`. +Original patch by Andrew Brezovsky. From webhook-mailer at python.org Fri Apr 29 14:45:52 2022 From: webhook-mailer at python.org (miss-islington) Date: Fri, 29 Apr 2022 18:45:52 -0000 Subject: [Python-checkins] bpo-26792: Improve docstrings of runpy module run_functions (GH-30729) Message-ID: https://github.com/python/cpython/commit/7bd4411b90e9f4b376754be23b5f6b86bf4e3c5f commit: 7bd4411b90e9f4b376754be23b5f6b86bf4e3c5f branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-29T11:45:43-07:00 summary: bpo-26792: Improve docstrings of runpy module run_functions (GH-30729) Co-authored-by: Jelle Zijlstra Co-authored-by: Kumar Aditya <59607654+kumaraditya303 at users.noreply.github.com> (cherry picked from commit 117836f123a1c65d9ba50401822b883f11f0a347) Co-authored-by: Humbled Drugman files: A Misc/NEWS.d/next/Documentation/2022-01-23-20-44-53.bpo-26792.dQ1v1W.rst M Lib/runpy.py diff --git a/Lib/runpy.py b/Lib/runpy.py index caba121426238..c7d3d8caad161 100644 --- a/Lib/runpy.py +++ b/Lib/runpy.py @@ -198,9 +198,24 @@ def _run_module_as_main(mod_name, alter_argv=True): def run_module(mod_name, init_globals=None, run_name=None, alter_sys=False): - """Execute a module's code without importing it + """Execute a module's code without importing it. - Returns the resulting top level namespace dictionary + mod_name -- an absolute module name or package name. + + Optional arguments: + init_globals -- dictionary used to pre-populate the module?s + globals dictionary before the code is executed. + + run_name -- if not None, this will be used for setting __name__; + otherwise, __name__ will be set to mod_name + '__main__' if the + named module is a package and to just mod_name otherwise. + + alter_sys -- if True, sys.argv[0] is updated with the value of + __file__ and sys.modules[__name__] is updated with a temporary + module object for the module being executed. Both are + restored to their original values before the function returns. + + Returns the resulting module globals dictionary. """ mod_name, mod_spec, code = _get_module_details(mod_name) if run_name is None: @@ -243,14 +258,19 @@ def _get_code_from_file(run_name, fname): return code, fname def run_path(path_name, init_globals=None, run_name=None): - """Execute code located at the specified filesystem location + """Execute code located at the specified filesystem location. + + path_name -- filesystem location of a Python script, zipfile, + or directory containing a top level __main__.py script. + + Optional arguments: + init_globals -- dictionary used to pre-populate the module?s + globals dictionary before the code is executed. - Returns the resulting top level namespace dictionary + run_name -- if not None, this will be used to set __name__; + otherwise, '' will be used for __name__. - The file path may refer directly to a Python script (i.e. - one that could be directly executed with execfile) or else - it may refer to a zipfile or directory containing a top - level __main__.py script. + Returns the resulting module globals dictionary. """ if run_name is None: run_name = "" diff --git a/Misc/NEWS.d/next/Documentation/2022-01-23-20-44-53.bpo-26792.dQ1v1W.rst b/Misc/NEWS.d/next/Documentation/2022-01-23-20-44-53.bpo-26792.dQ1v1W.rst new file mode 100644 index 0000000000000..64a3956447601 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2022-01-23-20-44-53.bpo-26792.dQ1v1W.rst @@ -0,0 +1,2 @@ +Improve the docstrings of :func:`runpy.run_module` and :func:`runpy.run_path`. +Original patch by Andrew Brezovsky. From webhook-mailer at python.org Fri Apr 29 14:46:51 2022 From: webhook-mailer at python.org (miss-islington) Date: Fri, 29 Apr 2022 18:46:51 -0000 Subject: [Python-checkins] bpo-26792: Improve docstrings of runpy module run_functions (GH-30729) Message-ID: https://github.com/python/cpython/commit/36de20d0f9191c74048474afc989296665362f10 commit: 36de20d0f9191c74048474afc989296665362f10 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-29T11:46:47-07:00 summary: bpo-26792: Improve docstrings of runpy module run_functions (GH-30729) Co-authored-by: Jelle Zijlstra Co-authored-by: Kumar Aditya <59607654+kumaraditya303 at users.noreply.github.com> (cherry picked from commit 117836f123a1c65d9ba50401822b883f11f0a347) Co-authored-by: Humbled Drugman files: A Misc/NEWS.d/next/Documentation/2022-01-23-20-44-53.bpo-26792.dQ1v1W.rst M Lib/runpy.py diff --git a/Lib/runpy.py b/Lib/runpy.py index 7e1e1ac5dde2d..c602bbe3f7d63 100644 --- a/Lib/runpy.py +++ b/Lib/runpy.py @@ -199,9 +199,24 @@ def _run_module_as_main(mod_name, alter_argv=True): def run_module(mod_name, init_globals=None, run_name=None, alter_sys=False): - """Execute a module's code without importing it + """Execute a module's code without importing it. - Returns the resulting top level namespace dictionary + mod_name -- an absolute module name or package name. + + Optional arguments: + init_globals -- dictionary used to pre-populate the module?s + globals dictionary before the code is executed. + + run_name -- if not None, this will be used for setting __name__; + otherwise, __name__ will be set to mod_name + '__main__' if the + named module is a package and to just mod_name otherwise. + + alter_sys -- if True, sys.argv[0] is updated with the value of + __file__ and sys.modules[__name__] is updated with a temporary + module object for the module being executed. Both are + restored to their original values before the function returns. + + Returns the resulting module globals dictionary. """ mod_name, mod_spec, code = _get_module_details(mod_name) if run_name is None: @@ -243,14 +258,19 @@ def _get_code_from_file(run_name, fname): return code, fname def run_path(path_name, init_globals=None, run_name=None): - """Execute code located at the specified filesystem location + """Execute code located at the specified filesystem location. + + path_name -- filesystem location of a Python script, zipfile, + or directory containing a top level __main__.py script. + + Optional arguments: + init_globals -- dictionary used to pre-populate the module?s + globals dictionary before the code is executed. - Returns the resulting top level namespace dictionary + run_name -- if not None, this will be used to set __name__; + otherwise, '' will be used for __name__. - The file path may refer directly to a Python script (i.e. - one that could be directly executed with execfile) or else - it may refer to a zipfile or directory containing a top - level __main__.py script. + Returns the resulting module globals dictionary. """ if run_name is None: run_name = "" diff --git a/Misc/NEWS.d/next/Documentation/2022-01-23-20-44-53.bpo-26792.dQ1v1W.rst b/Misc/NEWS.d/next/Documentation/2022-01-23-20-44-53.bpo-26792.dQ1v1W.rst new file mode 100644 index 0000000000000..64a3956447601 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2022-01-23-20-44-53.bpo-26792.dQ1v1W.rst @@ -0,0 +1,2 @@ +Improve the docstrings of :func:`runpy.run_module` and :func:`runpy.run_path`. +Original patch by Andrew Brezovsky. From webhook-mailer at python.org Fri Apr 29 16:03:26 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Fri, 29 Apr 2022 20:03:26 -0000 Subject: [Python-checkins] gh-87390: Add tests demonstrating current type variable substitution behaviour (#32341) Message-ID: https://github.com/python/cpython/commit/f6656163de483003697d510031827b7512056d55 commit: f6656163de483003697d510031827b7512056d55 branch: main author: Matthew Rahtz committer: JelleZijlstra date: 2022-04-29T14:03:21-06:00 summary: gh-87390: Add tests demonstrating current type variable substitution behaviour (#32341) files: M Lib/test/test_typing.py diff --git a/Lib/test/test_typing.py b/Lib/test/test_typing.py index a904b7a790c04..dab549b17e149 100644 --- a/Lib/test/test_typing.py +++ b/Lib/test/test_typing.py @@ -3,6 +3,7 @@ from collections import defaultdict from functools import lru_cache, wraps import inspect +import itertools import pickle import re import sys @@ -476,6 +477,392 @@ def test_bad_var_substitution(self): list[T][arg] +def template_replace(templates: list[str], replacements: dict[str, list[str]]) -> list[tuple[str]]: + """Renders templates with possible combinations of replacements. + + Example 1: Suppose that: + templates = ["dog_breed are awesome", "dog_breed are cool"] + replacements = ["dog_breed": ["Huskies", "Beagles"]] + Then we would return: + [ + ("Huskies are awesome", "Huskies are cool"), + ("Beagles are awesome", "Beagles are cool") + ] + + Example 2: Suppose that: + templates = ["Huskies are word1 but also word2"] + replacements = {"word1": ["playful", "cute"], + "word2": ["feisty", "tiring"]} + Then we would return: + [ + ("Huskies are playful but also feisty"), + ("Huskies are playful but also tiring"), + ("Huskies are cute but also feisty"), + ("Huskies are cute but also tiring") + ] + + Note that if any of the replacements do not occur in any template: + templates = ["Huskies are word1", "Beagles!"] + replacements = {"word1": ["playful", "cute"], + "word2": ["feisty", "tiring"]} + Then we do not generate duplicates, returning: + [ + ("Huskies are playful", "Beagles!"), + ("Huskies are cute", "Beagles!") + ] + """ + # First, build a structure like: + # [ + # [("word1", "playful"), ("word1", "cute")], + # [("word2", "feisty"), ("word2", "tiring")] + # ] + replacement_combos = [] + for original, possible_replacements in replacements.items(): + original_replacement_tuples = [] + for replacement in possible_replacements: + original_replacement_tuples.append((original, replacement)) + replacement_combos.append(original_replacement_tuples) + + # Second, generate rendered templates, including possible duplicates. + rendered_templates = [] + for replacement_combo in itertools.product(*replacement_combos): + # replacement_combo would be e.g. + # [("word1", "playful"), ("word2", "feisty")] + templates_with_replacements = [] + for template in templates: + for original, replacement in replacement_combo: + template = template.replace(original, replacement) + templates_with_replacements.append(template) + rendered_templates.append(tuple(templates_with_replacements)) + + # Finally, remove the duplicates (but keep the order). + rendered_templates_no_duplicates = [] + for x in rendered_templates: + # Inefficient, but should be fine for our purposes. + if x not in rendered_templates_no_duplicates: + rendered_templates_no_duplicates.append(x) + + return rendered_templates_no_duplicates + + +class TemplateReplacementTests(BaseTestCase): + + def test_two_templates_two_replacements_yields_correct_renders(self): + actual = template_replace( + templates=["Cats are word1", "Dogs are word2"], + replacements={ + "word1": ["small", "cute"], + "word2": ["big", "fluffy"], + }, + ) + expected = [ + ("Cats are small", "Dogs are big"), + ("Cats are small", "Dogs are fluffy"), + ("Cats are cute", "Dogs are big"), + ("Cats are cute", "Dogs are fluffy"), + ] + self.assertEqual(actual, expected) + + def test_no_duplicates_if_replacement_not_in_templates(self): + actual = template_replace( + templates=["Cats are word1", "Dogs!"], + replacements={ + "word1": ["small", "cute"], + "word2": ["big", "fluffy"], + }, + ) + expected = [ + ("Cats are small", "Dogs!"), + ("Cats are cute", "Dogs!"), + ] + self.assertEqual(actual, expected) + + +class GenericAliasSubstitutionTests(BaseTestCase): + """Tests for type variable substitution in generic aliases. + + Note that the expected results here are tentative, based on a + still-being-worked-out spec for what we allow at runtime (given that + implementation of *full* substitution logic at runtime would add too much + complexity to typing.py). This spec is currently being discussed at + https://github.com/python/cpython/issues/91162. + """ + + def test_one_parameter(self): + T = TypeVar('T') + Ts = TypeVarTuple('Ts') + + class C(Generic[T]): pass + + generics = ['C', 'list', 'List'] + tuple_types = ['tuple', 'Tuple'] + + tests = [ + # Alias # Args # Expected result + ('generic[T]', '[()]', 'TypeError'), + ('generic[T]', '[int]', 'generic[int]'), + ('generic[T]', '[int, str]', 'TypeError'), + ('generic[T]', '[tuple_type[int, ...]]', 'generic[tuple_type[int, ...]]'), + # Should raise TypeError: a) according to the tentative spec, + # unpacked types cannot be used as arguments to aliases that expect + # a fixed number of arguments; b) it's equivalent to generic[()]. + ('generic[T]', '[*tuple[()]]', 'generic[*tuple[()]]'), + ('generic[T]', '[*Tuple[()]]', 'TypeError'), + # Should raise TypeError according to the tentative spec: unpacked + # types cannot be used as arguments to aliases that expect a fixed + # number of arguments. + ('generic[T]', '[*tuple[int]]', 'generic[*tuple[int]]'), + ('generic[T]', '[*Tuple[int]]', 'TypeError'), + # Ditto. + ('generic[T]', '[*tuple[int, str]]', 'generic[*tuple[int, str]]'), + ('generic[T]', '[*Tuple[int, str]]', 'TypeError'), + # Ditto. + ('generic[T]', '[*tuple[int, ...]]', 'generic[*tuple[int, ...]]'), + ('generic[T]', '[*Tuple[int, ...]]', 'TypeError'), + ('generic[T]', '[*Ts]', 'TypeError'), + ('generic[T]', '[T, *Ts]', 'TypeError'), + ('generic[T]', '[*Ts, T]', 'TypeError'), + # Raises TypeError because C is not variadic. + # (If C _were_ variadic, it'd be fine.) + ('C[T, *tuple_type[int, ...]]', '[int]', 'TypeError'), + # Should definitely raise TypeError: list only takes one argument. + ('list[T, *tuple_type[int, ...]]', '[int]', 'list[int, *tuple_type[int, ...]]'), + ('List[T, *tuple_type[int, ...]]', '[int]', 'TypeError'), + ] + + for alias_template, args_template, expected_template in tests: + rendered_templates = template_replace( + templates=[alias_template, args_template, expected_template], + replacements={'generic': generics, 'tuple_type': tuple_types} + ) + for alias_str, args_str, expected_str in rendered_templates: + with self.subTest(alias=alias_str, args=args_str, expected=expected_str): + if expected_str == 'TypeError': + with self.assertRaises(TypeError): + eval(alias_str + args_str) + else: + self.assertEqual( + eval(alias_str + args_str), + eval(expected_str) + ) + + + def test_two_parameters(self): + T1 = TypeVar('T1') + T2 = TypeVar('T2') + Ts = TypeVarTuple('Ts') + + class C(Generic[T1, T2]): pass + + generics = ['C', 'dict', 'Dict'] + tuple_types = ['tuple', 'Tuple'] + + tests = [ + # Alias # Args # Expected result + ('generic[T1, T2]', '[()]', 'TypeError'), + ('generic[T1, T2]', '[int]', 'TypeError'), + ('generic[T1, T2]', '[int, str]', 'generic[int, str]'), + ('generic[T1, T2]', '[int, str, bool]', 'TypeError'), + ('generic[T1, T2]', '[*tuple_type[int]]', 'TypeError'), + ('generic[T1, T2]', '[*tuple_type[int, str]]', 'TypeError'), + ('generic[T1, T2]', '[*tuple_type[int, str, bool]]', 'TypeError'), + + # Should raise TypeError according to the tentative spec: unpacked + # types cannot be used as arguments to aliases that expect a fixed + # number of arguments. + ('generic[T1, T2]', '[*tuple[int, str], *tuple[float, bool]]', 'generic[*tuple[int, str], *tuple[float, bool]]'), + ('generic[T1, T2]', '[*Tuple[int, str], *Tuple[float, bool]]', 'TypeError'), + + ('generic[T1, T2]', '[tuple_type[int, ...]]', 'TypeError'), + ('generic[T1, T2]', '[tuple_type[int, ...], tuple_type[str, ...]]', 'generic[tuple_type[int, ...], tuple_type[str, ...]]'), + ('generic[T1, T2]', '[*tuple_type[int, ...]]', 'TypeError'), + + # Ditto. + ('generic[T1, T2]', '[*tuple[int, ...], *tuple[str, ...]]', 'generic[*tuple[int, ...], *tuple[str, ...]]'), + ('generic[T1, T2]', '[*Tuple[int, ...], *Tuple[str, ...]]', 'TypeError'), + + ('generic[T1, T2]', '[*Ts]', 'TypeError'), + ('generic[T1, T2]', '[T, *Ts]', 'TypeError'), + ('generic[T1, T2]', '[*Ts, T]', 'TypeError'), + # Should raise TypeError according to the tentative spec: unpacked + # types cannot be used as arguments to generics that expect a fixed + # number of arguments. + # (None of the things in `generics` were defined using *Ts.) + ('generic[T1, *tuple_type[int, ...]]', '[str]', 'generic[str, *tuple_type[int, ...]]'), + ] + + for alias_template, args_template, expected_template in tests: + rendered_templates = template_replace( + templates=[alias_template, args_template, expected_template], + replacements={'generic': generics, 'tuple_type': tuple_types} + ) + for alias_str, args_str, expected_str in rendered_templates: + with self.subTest(alias=alias_str, args=args_str, expected=expected_str): + if expected_str == 'TypeError': + with self.assertRaises(TypeError): + eval(alias_str + args_str) + else: + self.assertEqual( + eval(alias_str + args_str), + eval(expected_str) + ) + + def test_three_parameters(self): + T1 = TypeVar('T1') + T2 = TypeVar('T2') + T3 = TypeVar('T3') + + class C(Generic[T1, T2, T3]): pass + + generics = ['C'] + tuple_types = ['tuple', 'Tuple'] + + tests = [ + # Alias # Args # Expected result + ('generic[T1, bool, T2]', '[int, str]', 'generic[int, bool, str]'), + ('generic[T1, bool, T2]', '[*tuple_type[int, str]]', 'TypeError'), + ] + + for alias_template, args_template, expected_template in tests: + rendered_templates = template_replace( + templates=[alias_template, args_template, expected_template], + replacements={'generic': generics, 'tuple_type': tuple_types} + ) + for alias_str, args_str, expected_str in rendered_templates: + with self.subTest(alias=alias_str, args=args_str, expected=expected_str): + if expected_str == 'TypeError': + with self.assertRaises(TypeError): + eval(alias_str + args_str) + else: + self.assertEqual( + eval(alias_str + args_str), + eval(expected_str) + ) + + def test_variadic_parameters(self): + T1 = TypeVar('T1') + T2 = TypeVar('T2') + Ts = TypeVarTuple('Ts') + + class C(Generic[*Ts]): pass + + generics = ['C', 'tuple', 'Tuple'] + tuple_types = ['tuple', 'Tuple'] + + # The majority of these have three separate cases for C, tuple and + # Tuple because tuple currently behaves differently. + tests = [ + # Alias # Args # Expected result + ('C[*Ts]', '[()]', 'C[()]'), + ('tuple[*Ts]', '[()]', 'TypeError'), # Should be tuple[()] + ('Tuple[*Ts]', '[()]', 'Tuple[()]'), + + ('C[*Ts]', '[int]', 'C[int]'), + ('tuple[*Ts]', '[int]', 'tuple[(int,),]'), # Should be tuple[int] + ('Tuple[*Ts]', '[int]', 'Tuple[int]'), + + ('C[*Ts]', '[int, str]', 'C[int, str]'), + ('tuple[*Ts]', '[int, str]', 'TypeError'), # Should be tuple[int, str] + ('Tuple[*Ts]', '[int, str]', 'Tuple[int, str]'), + + ('C[*Ts]', '[*tuple_type[int]]', 'C[*tuple_type[int]]'), # Should be C[int] + ('tuple[*Ts]', '[*tuple_type[int]]', 'tuple[(*tuple_type[int],),]'), # Should be tuple[int] + ('Tuple[*Ts]', '[*tuple_type[int]]', 'Tuple[*tuple_type[int]]'), # Should be Tuple[int] + + ('C[*Ts]', '[*tuple_type[*Ts]]', 'C[*tuple_type[*Ts]]'), # Should be C[*Ts] + ('tuple[*Ts]', '[*tuple_type[*Ts]]', 'tuple[(*tuple_type[*Ts],),]'), # Should be tuple[*Ts] + ('Tuple[*Ts]', '[*tuple_type[*Ts]]', 'Tuple[*tuple_type[*Ts]]'), # Should be Tuple[*Ts] + + ('C[*Ts]', '[*tuple_type[int, str]]', 'C[*tuple_type[int, str]]'), # Should be C[int, str] + ('tuple[*Ts]', '[*tuple_type[int, str]]', 'tuple[(*tuple_type[int, str],),]'), # Should be tuple[int, str] + ('Tuple[*Ts]', '[*tuple_type[int, str]]', 'Tuple[*tuple_type[int, str]]'), # Should be Tuple[int, str] + + ('C[*Ts]', '[tuple_type[int, ...]]', 'C[tuple_type[int, ...]]'), + ('tuple[*Ts]', '[tuple_type[int, ...]]', 'tuple[(tuple_type[int, ...],),]'), # Should be tuple[tuple_type[int, ...]] + ('Tuple[*Ts]', '[tuple_type[int, ...]]', 'Tuple[tuple_type[int, ...]]'), + + ('C[*Ts]', '[tuple_type[int, ...], tuple_type[str, ...]]', 'C[tuple_type[int, ...], tuple_type[str, ...]]'), + ('tuple[*Ts]', '[tuple_type[int, ...], tuple_type[str, ...]]', 'TypeError'), # Should be tuple[tuple_type[int, ...], tuple_type[str, ...]] + ('Tuple[*Ts]', '[tuple_type[int, ...], tuple_type[str, ...]]', 'Tuple[tuple_type[int, ...], tuple_type[str, ...]]'), + + ('C[*Ts]', '[*tuple_type[int, ...]]', 'C[*tuple_type[int, ...]]'), + ('tuple[*Ts]', '[*tuple_type[int, ...]]', 'tuple[(*tuple_type[int, ...],),]'), # Should be tuple[*tuple_type[int, ...]] + ('Tuple[*Ts]', '[*tuple_type[int, ...]]', 'Tuple[*tuple_type[int, ...]]'), + + # Technically, multiple unpackings are forbidden by PEP 646, but we + # choose to be less restrictive at runtime, to allow folks room + # to experiment. So all three of these should be valid. + ('C[*Ts]', '[*tuple_type[int, ...], *tuple_type[str, ...]]', 'C[*tuple_type[int, ...], *tuple_type[str, ...]]'), + # Should be tuple[*tuple_type[int, ...], *tuple_type[str, ...]], to match the other two. + ('tuple[*Ts]', '[*tuple_type[int, ...], *tuple_type[str, ...]]', 'TypeError'), + ('Tuple[*Ts]', '[*tuple_type[int, ...], *tuple_type[str, ...]]', 'Tuple[*tuple_type[int, ...], *tuple_type[str, ...]]'), + + ('C[*Ts]', '[*Ts]', 'C[*Ts]'), + ('tuple[*Ts]', '[*Ts]', 'tuple[(*Ts,),]'), # Should be tuple[*Ts] + ('Tuple[*Ts]', '[*Ts]', 'Tuple[*Ts]'), + + ('C[*Ts]', '[T, *Ts]', 'C[T, *Ts]'), + ('tuple[*Ts]', '[T, *Ts]', 'TypeError'), # Should be tuple[T, *Ts] + ('Tuple[*Ts]', '[T, *Ts]', 'Tuple[T, *Ts]'), + + ('C[*Ts]', '[*Ts, T]', 'C[*Ts, T]'), + ('tuple[*Ts]', '[*Ts, T]', 'TypeError'), # Should be tuple[*Ts, T] + ('Tuple[*Ts]', '[*Ts, T]', 'Tuple[*Ts, T]'), + + ('C[T, *Ts]', '[int]', 'C[int]'), + ('tuple[T, *Ts]', '[int]', 'TypeError'), # Should be tuple[int] + ('Tuple[T, *Ts]', '[int]', 'Tuple[int]'), + + ('C[T, *Ts]', '[int, str]', 'C[int, str]'), + ('tuple[T, *Ts]', '[int, str]', 'tuple[int, (str,)]'), # Should be tuple[int, str] + ('Tuple[T, *Ts]', '[int, str]', 'Tuple[int, str]'), + + ('C[T, *Ts]', '[int, str, bool]', 'C[int, str, bool]'), + ('tuple[T, *Ts]', '[int, str, bool]', 'TypeError'), # Should be tuple[int, str, bool] + ('Tuple[T, *Ts]', '[int, str, bool]', 'Tuple[int, str, bool]'), + + ('C[T, *Ts]', '[*tuple[int, ...]]', 'C[*tuple[int, ...]]'), # Should be C[int, *tuple[int, ...]] + ('C[T, *Ts]', '[*Tuple[int, ...]]', 'TypeError'), # Ditto + ('tuple[T, *Ts]', '[*tuple_type[int, ...]]', 'TypeError'), # Should be tuple[int, *tuple[int, ...]] + ('Tuple[T, *Ts]', '[*tuple[int, ...]]', 'Tuple[*tuple[int, ...]]'), # Ditto + ('Tuple[T, *Ts]', '[*Tuple[int, ...]]', 'TypeError'), # Ditto + + ('C[*Ts, T]', '[int]', 'C[int]'), + ('tuple[*Ts, T]', '[int]', 'TypeError'), # Should be tuple[int] + ('Tuple[*Ts, T]', '[int]', 'Tuple[int]'), + + ('C[*Ts, T]', '[int, str]', 'C[int, str]'), + ('tuple[*Ts, T]', '[int, str]', 'tuple[(int,), str]'), # Should be tuple[int, str] + ('Tuple[*Ts, T]', '[int, str]', 'Tuple[int, str]'), + + ('C[*Ts, T]', '[int, str, bool]', 'C[int, str, bool]'), + ('tuple[*Ts, T]', '[int, str, bool]', 'TypeError'), # Should be tuple[int, str, bool] + ('Tuple[*Ts, T]', '[int, str, bool]', 'Tuple[int, str, bool]'), + + ('generic[T, *tuple_type[int, ...]]', '[str]', 'generic[str, *tuple_type[int, ...]]'), + ('generic[T1, T2, *tuple_type[int, ...]]', '[str, bool]', 'generic[str, bool, *tuple_type[int, ...]]'), + ('generic[T1, *tuple_type[int, ...], T2]', '[str, bool]', 'generic[str, *tuple_type[int, ...], bool]'), + ('generic[T1, *tuple_type[int, ...], T2]', '[str, bool, float]', 'TypeError'), + ] + + for alias_template, args_template, expected_template in tests: + rendered_templates = template_replace( + templates=[alias_template, args_template, expected_template], + replacements={'generic': generics, 'tuple_type': tuple_types} + ) + for alias_str, args_str, expected_str in rendered_templates: + with self.subTest(alias=alias_str, args=args_str, expected=expected_str): + if expected_str == 'TypeError': + with self.assertRaises(TypeError): + eval(alias_str + args_str) + else: + self.assertEqual( + eval(alias_str + args_str), + eval(expected_str) + ) + + + class UnpackTests(BaseTestCase): def test_accepts_single_type(self): From webhook-mailer at python.org Fri Apr 29 17:01:52 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Fri, 29 Apr 2022 21:01:52 -0000 Subject: [Python-checkins] bpo-44791: Accept ellipsis as the last argument of typing.Concatenate (#30969) Message-ID: https://github.com/python/cpython/commit/81120b6754c67a8f1f00cbc3af6963c0e1945911 commit: 81120b6754c67a8f1f00cbc3af6963c0e1945911 branch: main author: Serhiy Storchaka committer: JelleZijlstra date: 2022-04-29T15:01:40-06:00 summary: bpo-44791: Accept ellipsis as the last argument of typing.Concatenate (#30969) files: A Misc/NEWS.d/next/Library/2022-01-27-14-46-15.bpo-44791.tR1JFG.rst M Doc/library/typing.rst M Lib/test/test_typing.py M Lib/typing.py diff --git a/Doc/library/typing.rst b/Doc/library/typing.rst index d4a6f32aa36d2..0de380551c9fb 100644 --- a/Doc/library/typing.rst +++ b/Doc/library/typing.rst @@ -852,7 +852,8 @@ These can be used as types in annotations using ``[]``, each having a unique syn callable. Usage is in the form ``Concatenate[Arg1Type, Arg2Type, ..., ParamSpecVariable]``. ``Concatenate`` is currently only valid when used as the first argument to a :data:`Callable`. - The last parameter to ``Concatenate`` must be a :class:`ParamSpec`. + The last parameter to ``Concatenate`` must be a :class:`ParamSpec` or + ellipsis (``...``). For example, to annotate a decorator ``with_lock`` which provides a :class:`threading.Lock` to the decorated function, ``Concatenate`` can be diff --git a/Lib/test/test_typing.py b/Lib/test/test_typing.py index dab549b17e149..929f0df960ae0 100644 --- a/Lib/test/test_typing.py +++ b/Lib/test/test_typing.py @@ -1755,8 +1755,7 @@ def test_concatenate(self): self.assertEqual(C[[], int], Callable[[int], int]) self.assertEqual(C[Concatenate[str, P2], int], Callable[Concatenate[int, str, P2], int]) - with self.assertRaises(TypeError): - C[..., int] + self.assertEqual(C[..., int], Callable[Concatenate[int, ...], int]) C = Callable[Concatenate[int, P], int] self.assertEqual(repr(C), @@ -1767,8 +1766,7 @@ def test_concatenate(self): self.assertEqual(C[[]], Callable[[int], int]) self.assertEqual(C[Concatenate[str, P2]], Callable[Concatenate[int, str, P2], int]) - with self.assertRaises(TypeError): - C[...] + self.assertEqual(C[...], Callable[Concatenate[int, ...], int]) def test_errors(self): Callable = self.Callable @@ -6739,8 +6737,7 @@ def test_var_substitution(self): self.assertEqual(C[int, []], (int,)) self.assertEqual(C[int, Concatenate[str, P2]], Concatenate[int, str, P2]) - with self.assertRaises(TypeError): - C[int, ...] + self.assertEqual(C[int, ...], Concatenate[int, ...]) C = Concatenate[int, P] self.assertEqual(C[P2], Concatenate[int, P2]) @@ -6748,8 +6745,7 @@ def test_var_substitution(self): self.assertEqual(C[str, float], (int, str, float)) self.assertEqual(C[[]], (int,)) self.assertEqual(C[Concatenate[str, P2]], Concatenate[int, str, P2]) - with self.assertRaises(TypeError): - C[...] + self.assertEqual(C[...], Concatenate[int, ...]) class TypeGuardTests(BaseTestCase): def test_basics(self): diff --git a/Lib/typing.py b/Lib/typing.py index f4d4fa4d6713c..b250f2992dfff 100644 --- a/Lib/typing.py +++ b/Lib/typing.py @@ -714,9 +714,9 @@ def Concatenate(self, parameters): raise TypeError("Cannot take a Concatenate of no types.") if not isinstance(parameters, tuple): parameters = (parameters,) - if not isinstance(parameters[-1], ParamSpec): + if not (parameters[-1] is ... or isinstance(parameters[-1], ParamSpec)): raise TypeError("The last parameter to Concatenate should be a " - "ParamSpec variable.") + "ParamSpec variable or ellipsis.") msg = "Concatenate[arg, ...]: each arg must be a type." parameters = (*(_type_check(p, msg) for p in parameters[:-1]), parameters[-1]) return _ConcatenateGenericAlias(self, parameters, @@ -1641,9 +1641,6 @@ def copy_with(self, params): return (*params[:-1], *params[-1]) if isinstance(params[-1], _ConcatenateGenericAlias): params = (*params[:-1], *params[-1].__args__) - elif not isinstance(params[-1], ParamSpec): - raise TypeError("The last parameter to Concatenate should be a " - "ParamSpec variable.") return super().copy_with(params) diff --git a/Misc/NEWS.d/next/Library/2022-01-27-14-46-15.bpo-44791.tR1JFG.rst b/Misc/NEWS.d/next/Library/2022-01-27-14-46-15.bpo-44791.tR1JFG.rst new file mode 100644 index 0000000000000..31c6dcc058e94 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-01-27-14-46-15.bpo-44791.tR1JFG.rst @@ -0,0 +1 @@ +Accept ellipsis as the last argument of :data:`typing.Concatenate`. From webhook-mailer at python.org Fri Apr 29 18:08:21 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Fri, 29 Apr 2022 22:08:21 -0000 Subject: [Python-checkins] sorting howto: Add clarification on < using __lt__ (#92010) Message-ID: https://github.com/python/cpython/commit/53ca774497fde7c5fcf3a84813ea42f95f75c639 commit: 53ca774497fde7c5fcf3a84813ea42f95f75c639 branch: main author: slateny <46876382+slateny at users.noreply.github.com> committer: JelleZijlstra date: 2022-04-29T16:08:07-06:00 summary: sorting howto: Add clarification on < using __lt__ (#92010) files: M Doc/howto/sorting.rst diff --git a/Doc/howto/sorting.rst b/Doc/howto/sorting.rst index 37328c82dff27..32b47711f8570 100644 --- a/Doc/howto/sorting.rst +++ b/Doc/howto/sorting.rst @@ -325,7 +325,7 @@ Odd and Ends >>> standard_way [('red', 1), ('red', 2), ('blue', 1), ('blue', 2)] -* The sort routines are guaranteed to use :meth:`__lt__` when making comparisons +* The sort routines use ``<`` when making comparisons between two objects. So, it is easy to add a standard sort order to a class by defining an :meth:`__lt__` method: @@ -335,6 +335,9 @@ Odd and Ends >>> sorted(student_objects) [('dave', 'B', 10), ('jane', 'B', 12), ('john', 'A', 15)] + However, note that ``<`` can fall back to using :meth:`__gt__` if + :meth:`__lt__` is not implemented (see :func:`object.__lt__`). + * Key functions need not depend directly on the objects being sorted. A key function can also access external resources. For instance, if the student grades are stored in a dictionary, they can be used to sort a separate list of student From webhook-mailer at python.org Fri Apr 29 18:25:42 2022 From: webhook-mailer at python.org (miss-islington) Date: Fri, 29 Apr 2022 22:25:42 -0000 Subject: [Python-checkins] sorting howto: Add clarification on < using __lt__ (GH-92010) Message-ID: https://github.com/python/cpython/commit/7149b21c2e990d1b641bc71c6ffc3cb153d0500d commit: 7149b21c2e990d1b641bc71c6ffc3cb153d0500d branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-29T15:25:31-07:00 summary: sorting howto: Add clarification on < using __lt__ (GH-92010) (cherry picked from commit 53ca774497fde7c5fcf3a84813ea42f95f75c639) Co-authored-by: slateny <46876382+slateny at users.noreply.github.com> files: M Doc/howto/sorting.rst diff --git a/Doc/howto/sorting.rst b/Doc/howto/sorting.rst index 37328c82dff27..32b47711f8570 100644 --- a/Doc/howto/sorting.rst +++ b/Doc/howto/sorting.rst @@ -325,7 +325,7 @@ Odd and Ends >>> standard_way [('red', 1), ('red', 2), ('blue', 1), ('blue', 2)] -* The sort routines are guaranteed to use :meth:`__lt__` when making comparisons +* The sort routines use ``<`` when making comparisons between two objects. So, it is easy to add a standard sort order to a class by defining an :meth:`__lt__` method: @@ -335,6 +335,9 @@ Odd and Ends >>> sorted(student_objects) [('dave', 'B', 10), ('jane', 'B', 12), ('john', 'A', 15)] + However, note that ``<`` can fall back to using :meth:`__gt__` if + :meth:`__lt__` is not implemented (see :func:`object.__lt__`). + * Key functions need not depend directly on the objects being sorted. A key function can also access external resources. For instance, if the student grades are stored in a dictionary, they can be used to sort a separate list of student From webhook-mailer at python.org Fri Apr 29 18:26:56 2022 From: webhook-mailer at python.org (miss-islington) Date: Fri, 29 Apr 2022 22:26:56 -0000 Subject: [Python-checkins] sorting howto: Add clarification on < using __lt__ (GH-92010) Message-ID: https://github.com/python/cpython/commit/869a89433899950f925d177bacd8fdd43affe827 commit: 869a89433899950f925d177bacd8fdd43affe827 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-29T15:26:51-07:00 summary: sorting howto: Add clarification on < using __lt__ (GH-92010) (cherry picked from commit 53ca774497fde7c5fcf3a84813ea42f95f75c639) Co-authored-by: slateny <46876382+slateny at users.noreply.github.com> files: M Doc/howto/sorting.rst diff --git a/Doc/howto/sorting.rst b/Doc/howto/sorting.rst index a8efe65353d6e..be017cfba35ff 100644 --- a/Doc/howto/sorting.rst +++ b/Doc/howto/sorting.rst @@ -287,7 +287,7 @@ Odd and Ends >>> standard_way [('red', 1), ('red', 2), ('blue', 1), ('blue', 2)] -* The sort routines are guaranteed to use :meth:`__lt__` when making comparisons +* The sort routines use ``<`` when making comparisons between two objects. So, it is easy to add a standard sort order to a class by defining an :meth:`__lt__` method:: @@ -295,6 +295,9 @@ Odd and Ends >>> sorted(student_objects) [('dave', 'B', 10), ('jane', 'B', 12), ('john', 'A', 15)] + However, note that ``<`` can fall back to using :meth:`__gt__` if + :meth:`__lt__` is not implemented (see :func:`object.__lt__`). + * Key functions need not depend directly on the objects being sorted. A key function can also access external resources. For instance, if the student grades are stored in a dictionary, they can be used to sort a separate list of student From webhook-mailer at python.org Fri Apr 29 18:58:45 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Fri, 29 Apr 2022 22:58:45 -0000 Subject: [Python-checkins] Fix indentation for typing.Unpack docs (#92058) Message-ID: https://github.com/python/cpython/commit/7861a9e3276cf9bf2bfa56d86f29ef06b2456f87 commit: 7861a9e3276cf9bf2bfa56d86f29ef06b2456f87 branch: main author: Jelle Zijlstra committer: JelleZijlstra date: 2022-04-29T16:58:40-06:00 summary: Fix indentation for typing.Unpack docs (#92058) https://docs.python.org/3.11/library/typing.html#typing.Unpack files: M Doc/library/typing.rst diff --git a/Doc/library/typing.rst b/Doc/library/typing.rst index 0de380551c9fb..0f31b90b7610e 100644 --- a/Doc/library/typing.rst +++ b/Doc/library/typing.rst @@ -1368,27 +1368,27 @@ These are not used in annotations. They are building blocks for creating generic .. data:: Unpack - A typing operator that conceptually marks an object as having been - unpacked. For example, using the unpack operator ``*`` on a - :class:`type variable tuple ` is equivalent to using ``Unpack`` - to mark the type variable tuple as having been unpacked:: - - Ts = TypeVarTuple('Ts') - tup: tuple[*Ts] - # Effectively does: - tup: tuple[Unpack[Ts]] - - In fact, ``Unpack`` can be used interchangeably with ``*`` in the context - of types. You might see ``Unpack`` being used explicitly in older versions - of Python, where ``*`` couldn't be used in certain places:: - - # In older versions of Python, TypeVarTuple and Unpack - # are located in the `typing_extensions` backports package. - from typing_extensions import TypeVarTuple, Unpack - - Ts = TypeVarTuple('Ts') - tup: tuple[*Ts] # Syntax error on Python <= 3.10! - tup: tuple[Unpack[Ts]] # Semantically equivalent, and backwards-compatible + A typing operator that conceptually marks an object as having been + unpacked. For example, using the unpack operator ``*`` on a + :class:`type variable tuple ` is equivalent to using ``Unpack`` + to mark the type variable tuple as having been unpacked:: + + Ts = TypeVarTuple('Ts') + tup: tuple[*Ts] + # Effectively does: + tup: tuple[Unpack[Ts]] + + In fact, ``Unpack`` can be used interchangeably with ``*`` in the context + of types. You might see ``Unpack`` being used explicitly in older versions + of Python, where ``*`` couldn't be used in certain places:: + + # In older versions of Python, TypeVarTuple and Unpack + # are located in the `typing_extensions` backports package. + from typing_extensions import TypeVarTuple, Unpack + + Ts = TypeVarTuple('Ts') + tup: tuple[*Ts] # Syntax error on Python <= 3.10! + tup: tuple[Unpack[Ts]] # Semantically equivalent, and backwards-compatible .. versionadded:: 3.11 From webhook-mailer at python.org Fri Apr 29 21:22:33 2022 From: webhook-mailer at python.org (miss-islington) Date: Sat, 30 Apr 2022 01:22:33 -0000 Subject: [Python-checkins] gh-91217: deprecate xdrlib (GH-92066) Message-ID: https://github.com/python/cpython/commit/c2b579741dc36f4b7da00d4361d651544996d8f4 commit: c2b579741dc36f4b7da00d4361d651544996d8f4 branch: main author: Brett Cannon committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-29T18:22:10-07:00 summary: gh-91217: deprecate xdrlib (GH-92066) Automerge-Triggered-By: GH:brettcannon files: A Misc/NEWS.d/next/Library/2022-04-21-21-06-54.gh-issue-91217.2cVma_.rst M Doc/whatsnew/3.11.rst M Lib/test/test_xdrlib.py M Lib/xdrlib.py diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index 5f1f995a0fe2f..3ba35a8620a56 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -1081,6 +1081,7 @@ Deprecated * :mod:`sunau` * :mod:`telnetlib` * :mod:`uu` + * :mod:`xdrlib` (Contributed by Brett Cannon in :issue:`47061` and Victor Stinner in :gh:`68966`.) diff --git a/Lib/test/test_xdrlib.py b/Lib/test/test_xdrlib.py index 3df5f264ced46..226b70ae87b0b 100644 --- a/Lib/test/test_xdrlib.py +++ b/Lib/test/test_xdrlib.py @@ -1,6 +1,8 @@ import unittest +from test.support import warnings_helper + +xdrlib = warnings_helper.import_deprecated("xdrlib") -import xdrlib class XDRTest(unittest.TestCase): diff --git a/Lib/xdrlib.py b/Lib/xdrlib.py index d6e1aeb527266..b56ffa59b73dc 100644 --- a/Lib/xdrlib.py +++ b/Lib/xdrlib.py @@ -7,6 +7,9 @@ import struct from io import BytesIO from functools import wraps +import warnings + +warnings._deprecated(__name__, remove=(3, 13)) __all__ = ["Error", "Packer", "Unpacker", "ConversionError"] diff --git a/Misc/NEWS.d/next/Library/2022-04-21-21-06-54.gh-issue-91217.2cVma_.rst b/Misc/NEWS.d/next/Library/2022-04-21-21-06-54.gh-issue-91217.2cVma_.rst new file mode 100644 index 0000000000000..14f2c4d06491b --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-21-21-06-54.gh-issue-91217.2cVma_.rst @@ -0,0 +1 @@ +Deprecate the xdrlib module. From webhook-mailer at python.org Fri Apr 29 23:12:51 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Sat, 30 Apr 2022 03:12:51 -0000 Subject: [Python-checkins] gh-91491: Add several typing features to What's New (#92060) Message-ID: https://github.com/python/cpython/commit/d0064a1e72c6709ede3f2ac1db109c9208d42eb0 commit: d0064a1e72c6709ede3f2ac1db109c9208d42eb0 branch: main author: Shantanu <12621235+hauntsaninja at users.noreply.github.com> committer: JelleZijlstra date: 2022-04-29T21:12:45-06:00 summary: gh-91491: Add several typing features to What's New (#92060) This gets all the major items in #91491. However, I didn't get around to adding what's new entries for the large clump of changes in the last bullet point in the issue. Co-authored-by: Jelle Zijlstra files: M Doc/whatsnew/3.11.rst diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index 3ba35a8620a56..fdd35ce14317e 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -297,6 +297,38 @@ See :pep:`675` for more details. (Contributed by Jelle Zijlstra in :issue:`47088`. PEP written by Pradeep Kumar Srinivasan and Graham Bleaney.) +PEP 681: Data Class Transforms +------------------------------ + +The new :data:`~typing.dataclass_transform` annotation may be used to +decorate a function that is itself a decorator, a class, or a metaclass. +The presence of ``@dataclass_transform()`` tells a static type checker that the +decorated function, class, or metaclass performs runtime "magic" that +transforms a class, endowing it with dataclass-like behaviors. + +For example:: + + # The ``create_model`` decorator is defined by a library. + @typing.dataclass_transform() + def create_model(cls: Type[_T]) -> Type[_T]: + cls.__init__ = ... + cls.__eq__ = ... + cls.__ne__ = ... + return cls + + # The ``create_model`` decorator can now be used to create new model + # classes, like this: + @create_model + class CustomerModel: + id: int + name: str + + c = CustomerModel(id=327, name="John Smith") + +See :pep:`681` for more details. + +(Contributed by Jelle Zijlstra in :gh:`91860`. PEP written by +Erik De Bonte and Eric Traut.) Other Language Changes ====================== @@ -649,6 +681,39 @@ time it had a resolution of 1 millisecond (10\ :sup:`-3` seconds). (Contributed by Benjamin Sz?ke, Dong-hee Na, Eryk Sun and Victor Stinner in :issue:`21302` and :issue:`45429`.) +typing +------ + +For major changes, see :ref:`new-feat-related-type-hints-311`. + +* Add :func:`typing.assert_never` and :class:`typing.Never`. + :func:`typing.assert_never` is useful for asking a type checker to confirm + that a line of code is not reachable. At runtime, it raises an + :exc:`AssertionError`. + (Contributed by Jelle Zijlstra in :gh:`90633`.) + +* Add :func:`typing.reveal_type`. This is useful for asking a type checker + what type it has inferred for a given expression. At runtime it prints + the type of the received value. + (Contributed by Jelle Zijlstra in :gh:`90572`.) + +* Add :func:`typing.assert_type`. This is useful for asking a type checker + to confirm that the type it has inferred for a given expression matches + the given type. At runtime it simply returns the received value. + (Contributed by Jelle Zijlstra in :gh:`90638`.) + +* Allow subclassing of :class:`typing.Any`. This is useful for avoiding + type checker errors related to highly dynamic class, such as mocks. + (Contributed by Shantanu Jain in :gh:`91154`.) + +* The :func:`typing.final` decorator now sets the ``__final__`` attributed on + the decorated object. + (Contributed by Jelle Zijlstra in :gh:`90500`.) + +* The :func:`typing.get_overloads` function can be used for introspecting + the overloads of a function. :func:`typing.clear_overloads` can be used + to clear all registered overloads of a function. + (Contributed by Jelle Zijlstra in :gh:`89263`.) unicodedata ----------- From webhook-mailer at python.org Fri Apr 29 23:13:25 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Sat, 30 Apr 2022 03:13:25 -0000 Subject: [Python-checkins] Rephrase typing.assert_never docs (#92061) Message-ID: https://github.com/python/cpython/commit/2f7952c35b8fb2487cb9749081d2c0c604d41107 commit: 2f7952c35b8fb2487cb9749081d2c0c604d41107 branch: main author: Shantanu <12621235+hauntsaninja at users.noreply.github.com> committer: JelleZijlstra date: 2022-04-29T21:13:21-06:00 summary: Rephrase typing.assert_never docs (#92061) This change is similar to that in #32069 files: M Doc/library/typing.rst diff --git a/Doc/library/typing.rst b/Doc/library/typing.rst index 0f31b90b7610e..426cbf12b8c92 100644 --- a/Doc/library/typing.rst +++ b/Doc/library/typing.rst @@ -2337,7 +2337,7 @@ Functions and decorators .. function:: assert_never(arg, /) - Assert to the type checker that a line of code is unreachable. + Ask a static type checker to confirm that a line of code is unreachable. Example:: @@ -2358,7 +2358,7 @@ Functions and decorators reachable, it will emit an error. For example, if the type annotation for ``arg`` was instead ``int | str | float``, the type checker would emit an error pointing out that ``unreachable`` is of type :class:`float`. - For a call to ``assert_never`` to succeed, the inferred type of + For a call to ``assert_never`` to pass type checking, the inferred type of the argument passed in must be the bottom type, :data:`Never`, and nothing else. From webhook-mailer at python.org Fri Apr 29 23:18:47 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Sat, 30 Apr 2022 03:18:47 -0000 Subject: [Python-checkins] gh-92064: Fix global variable name collision in test_typing (#92067) Message-ID: https://github.com/python/cpython/commit/a29aa76a3ff8e5a4ee85961bb37a163f6b75a034 commit: a29aa76a3ff8e5a4ee85961bb37a163f6b75a034 branch: main author: Dennis Sweeney <36520290+sweeneyde at users.noreply.github.com> committer: JelleZijlstra date: 2022-04-29T21:18:38-06:00 summary: gh-92064: Fix global variable name collision in test_typing (#92067) Fixes #92064 files: M Lib/test/test_typing.py diff --git a/Lib/test/test_typing.py b/Lib/test/test_typing.py index 929f0df960ae0..412d117c238da 100644 --- a/Lib/test/test_typing.py +++ b/Lib/test/test_typing.py @@ -1332,16 +1332,16 @@ class TypeVarTuplePicklingTests(BaseTestCase): @all_pickle_protocols def test_pickling_then_unpickling_results_in_same_identity(self, proto): - global Ts1 # See explanation at start of class. - Ts1 = TypeVarTuple('Ts1') - Ts2 = pickle.loads(pickle.dumps(Ts1, proto)) - self.assertIs(Ts1, Ts2) + global global_Ts1 # See explanation at start of class. + global_Ts1 = TypeVarTuple('global_Ts1') + global_Ts2 = pickle.loads(pickle.dumps(global_Ts1, proto)) + self.assertIs(global_Ts1, global_Ts2) @all_pickle_protocols def test_pickling_then_unpickling_unpacked_results_in_same_identity(self, proto): - global Ts # See explanation at start of class. - Ts = TypeVarTuple('Ts') - unpacked1 = Unpack[Ts] + global global_Ts # See explanation at start of class. + global_Ts = TypeVarTuple('global_Ts') + unpacked1 = Unpack[global_Ts] unpacked2 = pickle.loads(pickle.dumps(unpacked1, proto)) self.assertIs(unpacked1, unpacked2) @@ -1349,19 +1349,19 @@ def test_pickling_then_unpickling_unpacked_results_in_same_identity(self, proto) def test_pickling_then_unpickling_tuple_with_typevartuple_equality( self, proto ): - global T, Ts # See explanation at start of class. - T = TypeVar('T') - Ts = TypeVarTuple('Ts') + global global_T, global_Ts # See explanation at start of class. + global_T = TypeVar('global_T') + global_Ts = TypeVarTuple('global_Ts') - a1 = Tuple[Unpack[Ts]] + a1 = Tuple[Unpack[global_Ts]] a2 = pickle.loads(pickle.dumps(a1, proto)) self.assertEqual(a1, a2) - a1 = Tuple[T, Unpack[Ts]] + a1 = Tuple[T, Unpack[global_Ts]] a2 = pickle.loads(pickle.dumps(a1, proto)) self.assertEqual(a1, a2) - a1 = Tuple[int, Unpack[Ts]] + a1 = Tuple[int, Unpack[global_Ts]] a2 = pickle.loads(pickle.dumps(a1, proto)) self.assertEqual(a1, a2) From webhook-mailer at python.org Sat Apr 30 01:22:50 2022 From: webhook-mailer at python.org (serhiy-storchaka) Date: Sat, 30 Apr 2022 05:22:50 -0000 Subject: [Python-checkins] bpo-43224: Implement substitution of unpacked TypeVarTuple in C (GH-31828) Message-ID: https://github.com/python/cpython/commit/e8c2f72b94ae5dfba50c0f2e2c06b7ee975c8acb commit: e8c2f72b94ae5dfba50c0f2e2c06b7ee975c8acb branch: main author: Serhiy Storchaka committer: serhiy-storchaka date: 2022-04-30T08:22:46+03:00 summary: bpo-43224: Implement substitution of unpacked TypeVarTuple in C (GH-31828) Co-authored-by: Matthew Rahtz files: M Include/internal/pycore_global_strings.h M Include/internal/pycore_runtime_init.h M Lib/test/test_typing.py M Lib/typing.py M Objects/genericaliasobject.c diff --git a/Include/internal/pycore_global_strings.h b/Include/internal/pycore_global_strings.h index cc94662256e87..28fffa95071a0 100644 --- a/Include/internal/pycore_global_strings.h +++ b/Include/internal/pycore_global_strings.h @@ -202,6 +202,7 @@ struct _Py_global_strings { STRUCT_FOR_ID(__truediv__) STRUCT_FOR_ID(__trunc__) STRUCT_FOR_ID(__typing_subst__) + STRUCT_FOR_ID(__typing_unpacked__) STRUCT_FOR_ID(__warningregistry__) STRUCT_FOR_ID(__weakref__) STRUCT_FOR_ID(__xor__) diff --git a/Include/internal/pycore_runtime_init.h b/Include/internal/pycore_runtime_init.h index 9a3a9d04324ba..941badfc8cb6a 100644 --- a/Include/internal/pycore_runtime_init.h +++ b/Include/internal/pycore_runtime_init.h @@ -825,6 +825,7 @@ extern "C" { INIT_ID(__truediv__), \ INIT_ID(__trunc__), \ INIT_ID(__typing_subst__), \ + INIT_ID(__typing_unpacked__), \ INIT_ID(__warningregistry__), \ INIT_ID(__weakref__), \ INIT_ID(__xor__), \ diff --git a/Lib/test/test_typing.py b/Lib/test/test_typing.py index 412d117c238da..88be2850acb1c 100644 --- a/Lib/test/test_typing.py +++ b/Lib/test/test_typing.py @@ -754,89 +754,89 @@ class C(Generic[*Ts]): pass tests = [ # Alias # Args # Expected result ('C[*Ts]', '[()]', 'C[()]'), - ('tuple[*Ts]', '[()]', 'TypeError'), # Should be tuple[()] + ('tuple[*Ts]', '[()]', 'tuple[()]'), ('Tuple[*Ts]', '[()]', 'Tuple[()]'), ('C[*Ts]', '[int]', 'C[int]'), - ('tuple[*Ts]', '[int]', 'tuple[(int,),]'), # Should be tuple[int] + ('tuple[*Ts]', '[int]', 'tuple[int]'), ('Tuple[*Ts]', '[int]', 'Tuple[int]'), ('C[*Ts]', '[int, str]', 'C[int, str]'), - ('tuple[*Ts]', '[int, str]', 'TypeError'), # Should be tuple[int, str] + ('tuple[*Ts]', '[int, str]', 'tuple[int, str]'), ('Tuple[*Ts]', '[int, str]', 'Tuple[int, str]'), ('C[*Ts]', '[*tuple_type[int]]', 'C[*tuple_type[int]]'), # Should be C[int] - ('tuple[*Ts]', '[*tuple_type[int]]', 'tuple[(*tuple_type[int],),]'), # Should be tuple[int] + ('tuple[*Ts]', '[*tuple_type[int]]', 'tuple[*tuple_type[int]]'), # Should be tuple[int] ('Tuple[*Ts]', '[*tuple_type[int]]', 'Tuple[*tuple_type[int]]'), # Should be Tuple[int] ('C[*Ts]', '[*tuple_type[*Ts]]', 'C[*tuple_type[*Ts]]'), # Should be C[*Ts] - ('tuple[*Ts]', '[*tuple_type[*Ts]]', 'tuple[(*tuple_type[*Ts],),]'), # Should be tuple[*Ts] + ('tuple[*Ts]', '[*tuple_type[*Ts]]', 'tuple[*tuple_type[*Ts]]'), # Should be tuple[*Ts] ('Tuple[*Ts]', '[*tuple_type[*Ts]]', 'Tuple[*tuple_type[*Ts]]'), # Should be Tuple[*Ts] ('C[*Ts]', '[*tuple_type[int, str]]', 'C[*tuple_type[int, str]]'), # Should be C[int, str] - ('tuple[*Ts]', '[*tuple_type[int, str]]', 'tuple[(*tuple_type[int, str],),]'), # Should be tuple[int, str] + ('tuple[*Ts]', '[*tuple_type[int, str]]', 'tuple[*tuple_type[int, str]]'), # Should be tuple[int, str] ('Tuple[*Ts]', '[*tuple_type[int, str]]', 'Tuple[*tuple_type[int, str]]'), # Should be Tuple[int, str] ('C[*Ts]', '[tuple_type[int, ...]]', 'C[tuple_type[int, ...]]'), - ('tuple[*Ts]', '[tuple_type[int, ...]]', 'tuple[(tuple_type[int, ...],),]'), # Should be tuple[tuple_type[int, ...]] + ('tuple[*Ts]', '[tuple_type[int, ...]]', 'tuple[tuple_type[int, ...]]'), ('Tuple[*Ts]', '[tuple_type[int, ...]]', 'Tuple[tuple_type[int, ...]]'), ('C[*Ts]', '[tuple_type[int, ...], tuple_type[str, ...]]', 'C[tuple_type[int, ...], tuple_type[str, ...]]'), - ('tuple[*Ts]', '[tuple_type[int, ...], tuple_type[str, ...]]', 'TypeError'), # Should be tuple[tuple_type[int, ...], tuple_type[str, ...]] + ('tuple[*Ts]', '[tuple_type[int, ...], tuple_type[str, ...]]', 'tuple[tuple_type[int, ...], tuple_type[str, ...]]'), ('Tuple[*Ts]', '[tuple_type[int, ...], tuple_type[str, ...]]', 'Tuple[tuple_type[int, ...], tuple_type[str, ...]]'), ('C[*Ts]', '[*tuple_type[int, ...]]', 'C[*tuple_type[int, ...]]'), - ('tuple[*Ts]', '[*tuple_type[int, ...]]', 'tuple[(*tuple_type[int, ...],),]'), # Should be tuple[*tuple_type[int, ...]] + ('tuple[*Ts]', '[*tuple_type[int, ...]]', 'tuple[*tuple_type[int, ...]]'), ('Tuple[*Ts]', '[*tuple_type[int, ...]]', 'Tuple[*tuple_type[int, ...]]'), # Technically, multiple unpackings are forbidden by PEP 646, but we # choose to be less restrictive at runtime, to allow folks room # to experiment. So all three of these should be valid. ('C[*Ts]', '[*tuple_type[int, ...], *tuple_type[str, ...]]', 'C[*tuple_type[int, ...], *tuple_type[str, ...]]'), - # Should be tuple[*tuple_type[int, ...], *tuple_type[str, ...]], to match the other two. - ('tuple[*Ts]', '[*tuple_type[int, ...], *tuple_type[str, ...]]', 'TypeError'), + ('tuple[*Ts]', '[*tuple_type[int, ...], *tuple_type[str, ...]]', 'tuple[*tuple_type[int, ...], *tuple_type[str, ...]]'), ('Tuple[*Ts]', '[*tuple_type[int, ...], *tuple_type[str, ...]]', 'Tuple[*tuple_type[int, ...], *tuple_type[str, ...]]'), ('C[*Ts]', '[*Ts]', 'C[*Ts]'), - ('tuple[*Ts]', '[*Ts]', 'tuple[(*Ts,),]'), # Should be tuple[*Ts] + ('tuple[*Ts]', '[*Ts]', 'tuple[*Ts]'), ('Tuple[*Ts]', '[*Ts]', 'Tuple[*Ts]'), ('C[*Ts]', '[T, *Ts]', 'C[T, *Ts]'), - ('tuple[*Ts]', '[T, *Ts]', 'TypeError'), # Should be tuple[T, *Ts] + ('tuple[*Ts]', '[T, *Ts]', 'tuple[T, *Ts]'), ('Tuple[*Ts]', '[T, *Ts]', 'Tuple[T, *Ts]'), ('C[*Ts]', '[*Ts, T]', 'C[*Ts, T]'), - ('tuple[*Ts]', '[*Ts, T]', 'TypeError'), # Should be tuple[*Ts, T] + ('tuple[*Ts]', '[*Ts, T]', 'tuple[*Ts, T]'), ('Tuple[*Ts]', '[*Ts, T]', 'Tuple[*Ts, T]'), ('C[T, *Ts]', '[int]', 'C[int]'), - ('tuple[T, *Ts]', '[int]', 'TypeError'), # Should be tuple[int] + ('tuple[T, *Ts]', '[int]', 'tuple[int]'), ('Tuple[T, *Ts]', '[int]', 'Tuple[int]'), ('C[T, *Ts]', '[int, str]', 'C[int, str]'), - ('tuple[T, *Ts]', '[int, str]', 'tuple[int, (str,)]'), # Should be tuple[int, str] + ('tuple[T, *Ts]', '[int, str]', 'tuple[int, str]'), ('Tuple[T, *Ts]', '[int, str]', 'Tuple[int, str]'), ('C[T, *Ts]', '[int, str, bool]', 'C[int, str, bool]'), - ('tuple[T, *Ts]', '[int, str, bool]', 'TypeError'), # Should be tuple[int, str, bool] + ('tuple[T, *Ts]', '[int, str, bool]', 'tuple[int, str, bool]'), ('Tuple[T, *Ts]', '[int, str, bool]', 'Tuple[int, str, bool]'), ('C[T, *Ts]', '[*tuple[int, ...]]', 'C[*tuple[int, ...]]'), # Should be C[int, *tuple[int, ...]] ('C[T, *Ts]', '[*Tuple[int, ...]]', 'TypeError'), # Ditto - ('tuple[T, *Ts]', '[*tuple_type[int, ...]]', 'TypeError'), # Should be tuple[int, *tuple[int, ...]] - ('Tuple[T, *Ts]', '[*tuple[int, ...]]', 'Tuple[*tuple[int, ...]]'), # Ditto - ('Tuple[T, *Ts]', '[*Tuple[int, ...]]', 'TypeError'), # Ditto + ('tuple[T, *Ts]', '[*tuple[int, ...]]', 'tuple[*tuple[int, ...]]'), # Should be tuple[int, *tuple[int, ...]] + ('tuple[T, *Ts]', '[*Tuple[int, ...]]', 'TypeError'), # Should be tuple[int, *Tuple[int, ...]] + ('Tuple[T, *Ts]', '[*tuple[int, ...]]', 'Tuple[*tuple[int, ...]]'), # Should be Tuple[int, *tuple[int, ...]] + ('Tuple[T, *Ts]', '[*Tuple[int, ...]]', 'TypeError'), # Should be Tuple[int, *Tuple[int, ...]] ('C[*Ts, T]', '[int]', 'C[int]'), - ('tuple[*Ts, T]', '[int]', 'TypeError'), # Should be tuple[int] + ('tuple[*Ts, T]', '[int]', 'tuple[int]'), ('Tuple[*Ts, T]', '[int]', 'Tuple[int]'), ('C[*Ts, T]', '[int, str]', 'C[int, str]'), - ('tuple[*Ts, T]', '[int, str]', 'tuple[(int,), str]'), # Should be tuple[int, str] + ('tuple[*Ts, T]', '[int, str]', 'tuple[int, str]'), ('Tuple[*Ts, T]', '[int, str]', 'Tuple[int, str]'), ('C[*Ts, T]', '[int, str, bool]', 'C[int, str, bool]'), - ('tuple[*Ts, T]', '[int, str, bool]', 'TypeError'), # Should be tuple[int, str, bool] + ('tuple[*Ts, T]', '[int, str, bool]', 'tuple[int, str, bool]'), ('Tuple[*Ts, T]', '[int, str, bool]', 'Tuple[int, str, bool]'), ('generic[T, *tuple_type[int, ...]]', '[str]', 'generic[str, *tuple_type[int, ...]]'), @@ -945,7 +945,7 @@ def test_var_substitution(self): T2 = TypeVar('T2') class G(Generic[Unpack[Ts]]): pass - for A in G, Tuple: + for A in G, Tuple, tuple: B = A[Unpack[Ts]] self.assertEqual(B[()], A[()]) self.assertEqual(B[float], A[float]) @@ -984,6 +984,21 @@ def test_bad_var_substitution(self): T2 = TypeVar('T2') class G(Generic[Unpack[Ts]]): pass + for A in G, Tuple, tuple: + B = A[Ts] + with self.assertRaises(TypeError): + B[int, str] + + C = A[T, T2] + with self.assertRaises(TypeError): + C[Unpack[Ts]] + + def test_repr_is_correct(self): + Ts = TypeVarTuple('Ts') + T = TypeVar('T') + T2 = TypeVar('T2') + class G(Generic[Unpack[Ts]]): pass + for A in G, Tuple: B = A[T, Unpack[Ts], str, T2] with self.assertRaises(TypeError): diff --git a/Lib/typing.py b/Lib/typing.py index b250f2992dfff..35deb32220eb8 100644 --- a/Lib/typing.py +++ b/Lib/typing.py @@ -1019,7 +1019,7 @@ def __repr__(self): return self.__name__ def __typing_subst__(self, arg): - raise AssertionError + raise TypeError("Substitution of bare TypeVarTuple is not supported") class ParamSpecArgs(_Final, _Immutable, _root=True): @@ -1686,11 +1686,15 @@ def __repr__(self): return '*' + repr(self.__args__[0]) def __getitem__(self, args): - if (len(self.__parameters__) == 1 and - isinstance(self.__parameters__[0], TypeVarTuple)): + if self.__typing_unpacked__(): return args return super().__getitem__(args) + def __typing_unpacked__(self): + # If x is Unpack[tuple[...]], __parameters__ will be empty. + return bool(self.__parameters__ and + isinstance(self.__parameters__[0], TypeVarTuple)) + class Generic: """Abstract base class for generic types. diff --git a/Objects/genericaliasobject.c b/Objects/genericaliasobject.c index 7059c4018303e..7b689912dffc1 100644 --- a/Objects/genericaliasobject.c +++ b/Objects/genericaliasobject.c @@ -190,6 +190,23 @@ tuple_add(PyObject *self, Py_ssize_t len, PyObject *item) return 0; } +static Py_ssize_t +tuple_extend(PyObject **dst, Py_ssize_t dstindex, + PyObject **src, Py_ssize_t count) +{ + assert(count >= 0); + if (_PyTuple_Resize(dst, PyTuple_GET_SIZE(*dst) + count - 1) != 0) { + return -1; + } + assert(dstindex + count <= PyTuple_GET_SIZE(*dst)); + for (Py_ssize_t i = 0; i < count; ++i) { + PyObject *item = src[i]; + Py_INCREF(item); + PyTuple_SET_ITEM(*dst, dstindex + i, item); + } + return dstindex + count; +} + PyObject * _Py_make_parameters(PyObject *args) { @@ -251,7 +268,8 @@ _Py_make_parameters(PyObject *args) If obj doesn't have a __parameters__ attribute or that's not a non-empty tuple, return a new reference to obj. */ static PyObject * -subs_tvars(PyObject *obj, PyObject *params, PyObject **argitems) +subs_tvars(PyObject *obj, PyObject *params, + PyObject **argitems, Py_ssize_t nargs, Py_ssize_t varparam) { PyObject *subparams; if (_PyObject_LookupAttr(obj, &_Py_ID(__parameters__), &subparams) < 0) { @@ -265,14 +283,27 @@ subs_tvars(PyObject *obj, PyObject *params, PyObject **argitems) Py_DECREF(subparams); return NULL; } - for (Py_ssize_t i = 0; i < nsubargs; ++i) { + for (Py_ssize_t i = 0, j = 0; i < nsubargs; ++i) { PyObject *arg = PyTuple_GET_ITEM(subparams, i); Py_ssize_t iparam = tuple_index(params, nparams, arg); - if (iparam >= 0) { - arg = argitems[iparam]; + if (iparam == varparam) { + j = tuple_extend(&subargs, j, + argitems + iparam, nargs - nparams + 1); + if (j < 0) { + return NULL; + } + } + else { + if (iparam >= 0) { + if (iparam > varparam) { + iparam += nargs - nsubargs; + } + arg = argitems[iparam]; + } + Py_INCREF(arg); + PyTuple_SET_ITEM(subargs, j, arg); + j++; } - Py_INCREF(arg); - PyTuple_SET_ITEM(subargs, i, arg); } obj = PyObject_GetItem(obj, subargs); @@ -286,6 +317,23 @@ subs_tvars(PyObject *obj, PyObject *params, PyObject **argitems) return obj; } +static int +_is_unpacked_typevartuple(PyObject *arg) +{ + PyObject *meth; + int res = _PyObject_LookupAttr(arg, &_Py_ID(__typing_unpacked__), &meth); + if (res > 0) { + PyObject *tmp = PyObject_CallNoArgs(meth); + Py_DECREF(meth); + if (tmp == NULL) { + return -1; + } + res = PyObject_IsTrue(tmp); + Py_DECREF(tmp); + } + return res; +} + PyObject * _Py_subs_parameters(PyObject *self, PyObject *args, PyObject *parameters, PyObject *item) { @@ -298,11 +346,27 @@ _Py_subs_parameters(PyObject *self, PyObject *args, PyObject *parameters, PyObje int is_tuple = PyTuple_Check(item); Py_ssize_t nitems = is_tuple ? PyTuple_GET_SIZE(item) : 1; PyObject **argitems = is_tuple ? &PyTuple_GET_ITEM(item, 0) : &item; - if (nitems != nparams) { - return PyErr_Format(PyExc_TypeError, - "Too %s arguments for %R", - nitems > nparams ? "many" : "few", - self); + Py_ssize_t varparam = 0; + for (; varparam < nparams; varparam++) { + PyObject *param = PyTuple_GET_ITEM(parameters, varparam); + if (Py_TYPE(param)->tp_iter) { // TypeVarTuple + break; + } + } + if (varparam < nparams) { + if (nitems < nparams - 1) { + return PyErr_Format(PyExc_TypeError, + "Too few arguments for %R", + self); + } + } + else { + if (nitems != nparams) { + return PyErr_Format(PyExc_TypeError, + "Too %s arguments for %R", + nitems > nparams ? "many" : "few", + self); + } } /* Replace all type variables (specified by parameters) with corresponding values specified by argitems. @@ -315,8 +379,13 @@ _Py_subs_parameters(PyObject *self, PyObject *args, PyObject *parameters, PyObje if (newargs == NULL) { return NULL; } - for (Py_ssize_t iarg = 0; iarg < nargs; iarg++) { + for (Py_ssize_t iarg = 0, jarg = 0; iarg < nargs; iarg++) { PyObject *arg = PyTuple_GET_ITEM(args, iarg); + int unpack = _is_unpacked_typevartuple(arg); + if (unpack < 0) { + Py_DECREF(newargs); + return NULL; + } PyObject *subst; if (_PyObject_LookupAttr(arg, &_Py_ID(__typing_subst__), &subst) < 0) { Py_DECREF(newargs); @@ -325,17 +394,38 @@ _Py_subs_parameters(PyObject *self, PyObject *args, PyObject *parameters, PyObje if (subst) { Py_ssize_t iparam = tuple_index(parameters, nparams, arg); assert(iparam >= 0); + if (iparam == varparam) { + Py_DECREF(subst); + Py_DECREF(newargs); + PyErr_SetString(PyExc_TypeError, + "Substitution of bare TypeVarTuple is not supported"); + return NULL; + } + if (iparam > varparam) { + iparam += nitems - nparams; + } arg = PyObject_CallOneArg(subst, argitems[iparam]); Py_DECREF(subst); } else { - arg = subs_tvars(arg, parameters, argitems); + arg = subs_tvars(arg, parameters, argitems, nitems, varparam); } if (arg == NULL) { Py_DECREF(newargs); return NULL; } - PyTuple_SET_ITEM(newargs, iarg, arg); + if (unpack) { + jarg = tuple_extend(&newargs, jarg, + &PyTuple_GET_ITEM(arg, 0), PyTuple_GET_SIZE(arg)); + Py_DECREF(arg); + if (jarg < 0) { + return NULL; + } + } + else { + PyTuple_SET_ITEM(newargs, jarg, arg); + jarg++; + } } return newargs; From webhook-mailer at python.org Sat Apr 30 01:24:10 2022 From: webhook-mailer at python.org (miss-islington) Date: Sat, 30 Apr 2022 05:24:10 -0000 Subject: [Python-checkins] gh-91880 - fix typo (GH-92069) Message-ID: https://github.com/python/cpython/commit/c7b7f12b8609f932a23a9bc96a5de7cd9ecd5723 commit: c7b7f12b8609f932a23a9bc96a5de7cd9ecd5723 branch: main author: David Hewitt <1939362+davidhewitt at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-29T22:23:54-07:00 summary: gh-91880 - fix typo (GH-92069) https://github.com/python/cpython/issues/91880#issuecomment-1113914241 - With thanks to @MojoVampire for spotting this. Automerge-Triggered-By: GH:gvanrossum files: M Lib/asyncio/runners.py diff --git a/Lib/asyncio/runners.py b/Lib/asyncio/runners.py index d274576b10134..065691bed9928 100644 --- a/Lib/asyncio/runners.py +++ b/Lib/asyncio/runners.py @@ -106,7 +106,7 @@ def run(self, coro, *, context=None): # `signal.signal` may throw if `threading.main_thread` does # not support signals (e.g. embedded interpreter with signals # not registered - see gh-91880) - signal_handler = None + sigint_handler = None else: sigint_handler = None From webhook-mailer at python.org Sat Apr 30 02:53:40 2022 From: webhook-mailer at python.org (methane) Date: Sat, 30 Apr 2022 06:53:40 -0000 Subject: [Python-checkins] gh-91954: Emit EncodingWarning from locale and subprocess (GH-91977) Message-ID: https://github.com/python/cpython/commit/354ace8b07e7d445fd2de713b6af1271536adce0 commit: 354ace8b07e7d445fd2de713b6af1271536adce0 branch: main author: Inada Naoki committer: methane date: 2022-04-30T15:53:29+09:00 summary: gh-91954: Emit EncodingWarning from locale and subprocess (GH-91977) locale.getpreferredencoding() and subprocess.Popen() emit EncodingWarning files: A Misc/NEWS.d/next/Library/2022-04-27-13-30-26.gh-issue-91954.cC7ga_.rst M Doc/library/subprocess.rst M Lib/locale.py M Lib/subprocess.py M Lib/test/test_subprocess.py diff --git a/Doc/library/subprocess.rst b/Doc/library/subprocess.rst index fca5549649268..6a334acc5a17c 100644 --- a/Doc/library/subprocess.rst +++ b/Doc/library/subprocess.rst @@ -619,7 +619,7 @@ functions. If *encoding* or *errors* are specified, or *text* is true, the file objects *stdin*, *stdout* and *stderr* are opened in text mode with the specified - encoding and *errors*, as described above in :ref:`frequently-used-arguments`. + *encoding* and *errors*, as described above in :ref:`frequently-used-arguments`. The *universal_newlines* argument is equivalent to *text* and is provided for backwards compatibility. By default, file objects are opened in binary mode. @@ -1445,12 +1445,13 @@ This module also provides the following legacy functions from the 2.x none of the guarantees described above regarding security and exception handling consistency are valid for these functions. -.. function:: getstatusoutput(cmd) +.. function:: getstatusoutput(cmd, *, encoding=None, errors=None) Return ``(exitcode, output)`` of executing *cmd* in a shell. Execute the string *cmd* in a shell with :meth:`Popen.check_output` and - return a 2-tuple ``(exitcode, output)``. The locale encoding is used; + return a 2-tuple ``(exitcode, output)``. + *encoding* and *errors* are used to decode output; see the notes on :ref:`frequently-used-arguments` for more details. A trailing newline is stripped from the output. @@ -1475,8 +1476,10 @@ handling consistency are valid for these functions. as it did in Python 3.3.3 and earlier. exitcode has the same value as :attr:`~Popen.returncode`. + .. versionadded:: 3.11 + Added *encoding* and *errors* arguments. -.. function:: getoutput(cmd) +.. function:: getoutput(cmd, *, encoding=None, errors=None) Return output (stdout and stderr) of executing *cmd* in a shell. @@ -1491,6 +1494,9 @@ handling consistency are valid for these functions. .. versionchanged:: 3.3.4 Windows support added + .. versionadded:: 3.11 + Added *encoding* and *errors* arguments. + Notes ----- diff --git a/Lib/locale.py b/Lib/locale.py index 170e5eea45b8c..25eb75ac65a32 100644 --- a/Lib/locale.py +++ b/Lib/locale.py @@ -655,6 +655,11 @@ def getencoding(): except NameError: def getpreferredencoding(do_setlocale=True): """Return the charset that the user is likely using.""" + if sys.flags.warn_default_encoding: + import warnings + warnings.warn( + "UTF-8 Mode affects locale.getpreferredencoding(). Consider locale.getencoding() instead.", + EncodingWarning, 2) if sys.flags.utf8_mode: return 'utf-8' return getencoding() @@ -663,6 +668,12 @@ def getpreferredencoding(do_setlocale=True): def getpreferredencoding(do_setlocale=True): """Return the charset that the user is likely using, according to the system configuration.""" + + if sys.flags.warn_default_encoding: + import warnings + warnings.warn( + "UTF-8 Mode affects locale.getpreferredencoding(). Consider locale.getencoding() instead.", + EncodingWarning, 2) if sys.flags.utf8_mode: return 'utf-8' diff --git a/Lib/subprocess.py b/Lib/subprocess.py index a5fa152715c14..968cfc14ddf91 100644 --- a/Lib/subprocess.py +++ b/Lib/subprocess.py @@ -43,6 +43,7 @@ import builtins import errno import io +import locale import os import time import signal @@ -344,6 +345,26 @@ def _args_from_interpreter_flags(): return args +def _text_encoding(): + # Return default text encoding and emit EncodingWarning if + # sys.flags.warn_default_encoding is true. + if sys.flags.warn_default_encoding: + f = sys._getframe() + filename = f.f_code.co_filename + stacklevel = 2 + while f := f.f_back: + if f.f_code.co_filename != filename: + break + stacklevel += 1 + warnings.warn("'encoding' argument not specified.", + EncodingWarning, stacklevel) + + if sys.flags.utf8_mode: + return "utf-8" + else: + return locale.getencoding() + + def call(*popenargs, timeout=None, **kwargs): """Run command with arguments. Wait for command to complete or timeout, then return the returncode attribute. @@ -610,7 +631,7 @@ def list2cmdline(seq): # Various tools for executing commands and looking at their output and status. # -def getstatusoutput(cmd): +def getstatusoutput(cmd, *, encoding=None, errors=None): """Return (exitcode, output) of executing cmd in a shell. Execute the string 'cmd' in a shell with 'check_output' and @@ -632,7 +653,8 @@ def getstatusoutput(cmd): (-15, '') """ try: - data = check_output(cmd, shell=True, text=True, stderr=STDOUT) + data = check_output(cmd, shell=True, text=True, stderr=STDOUT, + encoding=encoding, errors=errors) exitcode = 0 except CalledProcessError as ex: data = ex.output @@ -641,7 +663,7 @@ def getstatusoutput(cmd): data = data[:-1] return exitcode, data -def getoutput(cmd): +def getoutput(cmd, *, encoding=None, errors=None): """Return output (stdout or stderr) of executing cmd in a shell. Like getstatusoutput(), except the exit status is ignored and the return @@ -651,7 +673,8 @@ def getoutput(cmd): >>> subprocess.getoutput('ls /bin/ls') '/bin/ls' """ - return getstatusoutput(cmd)[1] + return getstatusoutput(cmd, encoding=encoding, errors=errors)[1] + def _use_posix_spawn(): @@ -858,13 +881,8 @@ def __init__(self, args, bufsize=-1, executable=None, errread = msvcrt.open_osfhandle(errread.Detach(), 0) self.text_mode = encoding or errors or text or universal_newlines - - # PEP 597: We suppress the EncodingWarning in subprocess module - # for now (at Python 3.10), because we focus on files for now. - # This will be changed to encoding = io.text_encoding(encoding) - # in the future. if self.text_mode and encoding is None: - self.encoding = encoding = "locale" + self.encoding = encoding = _text_encoding() # How long to resume waiting on a child after the first ^C. # There is no right value for this. The purpose is to be polite diff --git a/Lib/test/test_subprocess.py b/Lib/test/test_subprocess.py index 99b5947e54be6..5814a6d924e12 100644 --- a/Lib/test/test_subprocess.py +++ b/Lib/test/test_subprocess.py @@ -1733,6 +1733,20 @@ def test_run_with_shell_timeout_and_capture_output(self): msg="TimeoutExpired was delayed! Bad traceback:\n```\n" f"{stacks}```") + def test_encoding_warning(self): + code = textwrap.dedent("""\ + from subprocess import * + args = ["echo", "hello"] + run(args, text=True) + check_output(args, text=True) + """) + cp = subprocess.run([sys.executable, "-Xwarn_default_encoding", "-c", code], + capture_output=True) + lines = cp.stderr.splitlines() + self.assertEqual(len(lines), 2) + self.assertTrue(lines[0].startswith(b":3: EncodingWarning: ")) + self.assertTrue(lines[1].startswith(b":4: EncodingWarning: ")) + def _get_test_grp_name(): for name_group in ('staff', 'nogroup', 'grp', 'nobody', 'nfsnobody'): diff --git a/Misc/NEWS.d/next/Library/2022-04-27-13-30-26.gh-issue-91954.cC7ga_.rst b/Misc/NEWS.d/next/Library/2022-04-27-13-30-26.gh-issue-91954.cC7ga_.rst new file mode 100644 index 0000000000000..b63db25f32a4f --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-27-13-30-26.gh-issue-91954.cC7ga_.rst @@ -0,0 +1,2 @@ +Add *encoding* and *errors* arguments to :func:`subprocess.getoutput` and +:func:`subprocess.getstatusoutput`. From webhook-mailer at python.org Sat Apr 30 06:03:27 2022 From: webhook-mailer at python.org (serhiy-storchaka) Date: Sat, 30 Apr 2022 10:03:27 -0000 Subject: [Python-checkins] gh-92049: Forbid pickling constants re._constants.SUCCESS etc (GH-92070) Message-ID: https://github.com/python/cpython/commit/6d0d547033e295f91f05030322acfbb0e280fc1f commit: 6d0d547033e295f91f05030322acfbb0e280fc1f branch: main author: Serhiy Storchaka committer: serhiy-storchaka date: 2022-04-30T13:03:23+03:00 summary: gh-92049: Forbid pickling constants re._constants.SUCCESS etc (GH-92070) Previously, pickling did not fail, but the result could not be unpickled. files: A Misc/NEWS.d/next/Library/2022-04-30-10-53-10.gh-issue-92049.5SEKoh.rst M Lib/re/_constants.py diff --git a/Lib/re/_constants.py b/Lib/re/_constants.py index c45ce409e21b8..71204d903b322 100644 --- a/Lib/re/_constants.py +++ b/Lib/re/_constants.py @@ -62,6 +62,8 @@ def __new__(cls, value, name): def __repr__(self): return self.name + __reduce__ = None + MAXREPEAT = _NamedIntConstant(MAXREPEAT, 'MAXREPEAT') def _makecodes(*names): diff --git a/Misc/NEWS.d/next/Library/2022-04-30-10-53-10.gh-issue-92049.5SEKoh.rst b/Misc/NEWS.d/next/Library/2022-04-30-10-53-10.gh-issue-92049.5SEKoh.rst new file mode 100644 index 0000000000000..cad4621c65096 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-30-10-53-10.gh-issue-92049.5SEKoh.rst @@ -0,0 +1,2 @@ +Forbid pickling constants ``re._constants.SUCCESS`` etc. Previously, +pickling did not fail, but the result could not be unpickled. From webhook-mailer at python.org Sat Apr 30 06:13:50 2022 From: webhook-mailer at python.org (serhiy-storchaka) Date: Sat, 30 Apr 2022 10:13:50 -0000 Subject: [Python-checkins] gh-91760: Deprecate group names and numbers which will be invalid in future (GH-91794) Message-ID: https://github.com/python/cpython/commit/19dca041212f9f58ee11833bff3f8c157d4fd3e8 commit: 19dca041212f9f58ee11833bff3f8c157d4fd3e8 branch: main author: Serhiy Storchaka committer: serhiy-storchaka date: 2022-04-30T13:13:46+03:00 summary: gh-91760: Deprecate group names and numbers which will be invalid in future (GH-91794) Only sequence of ASCII digits will be accepted as a numerical reference. The group name in bytes patterns and replacement strings could only contain ASCII letters and digits and underscore. files: A Misc/NEWS.d/next/Library/2022-04-21-19-46-03.gh-issue-91760.zDtv1E.rst M Doc/library/re.rst M Doc/whatsnew/3.11.rst M Lib/re/_parser.py M Lib/test/test_re.py diff --git a/Doc/library/re.rst b/Doc/library/re.rst index 89de9286ace79..3cd9f252fee6f 100644 --- a/Doc/library/re.rst +++ b/Doc/library/re.rst @@ -417,6 +417,9 @@ The special characters are: | | * ``\1`` | +---------------------------------------+----------------------------------+ + .. deprecated:: 3.11 + Group names containing non-ASCII characters in bytes patterns. + .. index:: single: (?P=; in regular expressions ``(?P=name)`` @@ -486,6 +489,9 @@ The special characters are: will match with ``''`` as well as ``'user at host.com'``, but not with ``''``. + .. deprecated:: 3.11 + Group *id* containing anything except ASCII digits. + The special sequences consist of ``'\'`` and a character from the list below. If the ordinary character is not an ASCII digit or an ASCII letter, then the @@ -995,6 +1001,10 @@ form. Empty matches for the pattern are replaced when adjacent to a previous non-empty match. + .. deprecated:: 3.11 + Group *id* containing anything except ASCII digits. + Group names containing non-ASCII characters in bytes replacement strings. + .. function:: subn(pattern, repl, string, count=0, flags=0) diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index fdd35ce14317e..1a692f2fe7f60 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -1151,6 +1151,14 @@ Deprecated (Contributed by Brett Cannon in :issue:`47061` and Victor Stinner in :gh:`68966`.) +* More strict rules will be applied now applied for numerical group references + and group names in regular expressions in future Python versions. + Only sequence of ASCII digits will be now accepted as a numerical reference. + The group name in bytes patterns and replacement strings could only + contain ASCII letters and digits and underscore. + For now, a deprecation warning is raised for such syntax. + (Contributed by Serhiy Storchaka in :gh:`91760`.) + Removed ======= diff --git a/Lib/re/_parser.py b/Lib/re/_parser.py index 933d51589f46c..a393c508d86e5 100644 --- a/Lib/re/_parser.py +++ b/Lib/re/_parser.py @@ -287,8 +287,22 @@ def seek(self, index): self.__next() def error(self, msg, offset=0): + if not self.istext: + msg = msg.encode('ascii', 'backslashreplace').decode('ascii') return error(msg, self.string, self.tell() - offset) + def checkgroupname(self, name, offset, nested): + if not name.isidentifier(): + msg = "bad character in group name %r" % name + raise self.error(msg, len(name) + offset) + if not (self.istext or name.isascii()): + import warnings + warnings.warn( + "bad character in group name %a at position %d" % + (name, self.tell() - len(name) - offset), + DeprecationWarning, stacklevel=nested + 7 + ) + def _class_escape(source, escape): # handle escape code inside character class code = ESCAPES.get(escape) @@ -703,15 +717,11 @@ def _parse(source, state, verbose, nested, first=False): if sourcematch("<"): # named group: skip forward to end of name name = source.getuntil(">", "group name") - if not name.isidentifier(): - msg = "bad character in group name %r" % name - raise source.error(msg, len(name) + 1) + source.checkgroupname(name, 1, nested) elif sourcematch("="): # named backreference name = source.getuntil(")", "group name") - if not name.isidentifier(): - msg = "bad character in group name %r" % name - raise source.error(msg, len(name) + 1) + source.checkgroupname(name, 1, nested) gid = state.groupdict.get(name) if gid is None: msg = "unknown group name %r" % name @@ -773,6 +783,7 @@ def _parse(source, state, verbose, nested, first=False): # conditional backreference group condname = source.getuntil(")", "group name") if condname.isidentifier(): + source.checkgroupname(condname, 1, nested) condgroup = state.groupdict.get(condname) if condgroup is None: msg = "unknown group name %r" % condname @@ -795,6 +806,14 @@ def _parse(source, state, verbose, nested, first=False): state.grouprefpos[condgroup] = ( source.tell() - len(condname) - 1 ) + if not (condname.isdecimal() and condname.isascii()): + import warnings + warnings.warn( + "bad character in group name %s at position %d" % + (repr(condname) if source.istext else ascii(condname), + source.tell() - len(condname) - 1), + DeprecationWarning, stacklevel=nested + 6 + ) state.checklookbehindgroup(condgroup, source) item_yes = _parse(source, state, verbose, nested + 1) if source.match("|"): @@ -1000,11 +1019,11 @@ def addgroup(index, pos): # group c = this[1] if c == "g": - name = "" if not s.match("<"): raise s.error("missing <") name = s.getuntil(">", "group name") if name.isidentifier(): + s.checkgroupname(name, 1, -1) try: index = groupindex[name] except KeyError: @@ -1020,6 +1039,14 @@ def addgroup(index, pos): if index >= MAXGROUPS: raise s.error("invalid group reference %d" % index, len(name) + 1) + if not (name.isdecimal() and name.isascii()): + import warnings + warnings.warn( + "bad character in group name %s at position %d" % + (repr(name) if s.istext else ascii(name), + s.tell() - len(name) - 1), + DeprecationWarning, stacklevel=5 + ) addgroup(index, len(name) + 1) elif c == "0": if s.next in OCTDIGITS: diff --git a/Lib/test/test_re.py b/Lib/test/test_re.py index a4c2f1f3e4ba3..c1014753802c9 100644 --- a/Lib/test/test_re.py +++ b/Lib/test/test_re.py @@ -135,6 +135,7 @@ def test_basic_re_sub(self): self.assertEqual(re.sub('(?Px)', r'\g\g<1>', 'xx'), 'xxxx') self.assertEqual(re.sub('(?Px)', r'\g\g', 'xx'), 'xxxx') self.assertEqual(re.sub('(?Px)', r'\g<1>\g<1>', 'xx'), 'xxxx') + self.assertEqual(re.sub('()x', r'\g<0>\g<0>', 'xx'), 'xxxx') self.assertEqual(re.sub('a', r'\t\n\v\r\f\a\b', 'a'), '\t\n\v\r\f\a\b') self.assertEqual(re.sub('a', '\t\n\v\r\f\a\b', 'a'), '\t\n\v\r\f\a\b') @@ -274,6 +275,21 @@ def test_symbolic_groups_errors(self): self.checkPatternError('(?Px)', "bad character in group name '?'", 4) self.checkPatternError('(?P=?)', "bad character in group name '?'", 4) self.checkPatternError('(?(?)y)', "bad character in group name '?'", 3) + with self.assertWarnsRegex(DeprecationWarning, + r"bad character in group name '\\xc2\\xb5' " + r"at position 4") as w: + re.compile(b'(?P<\xc2\xb5>x)') + self.assertEqual(w.filename, __file__) + with self.assertWarnsRegex(DeprecationWarning, + r"bad character in group name '\\xc2\\xb5' " + r"at position 4"): + self.checkPatternError(b'(?P=\xc2\xb5)', + r"unknown group name '\xc2\xb5'", 4) + with self.assertWarnsRegex(DeprecationWarning, + r"bad character in group name '\\xc2\\xb5' " + r"at position 3"): + self.checkPatternError(b'(?(\xc2\xb5)y)', + r"unknown group name '\xc2\xb5'", 3) def test_symbolic_refs(self): self.assertEqual(re.sub('(?Px)|(?Py)', r'\g', 'xx'), '') @@ -306,12 +322,35 @@ def test_symbolic_refs_errors(self): re.sub('(?Px)', r'\g', 'xx') self.checkTemplateError('(?Px)', r'\g<-1>', 'xx', "bad character in group name '-1'", 3) + with self.assertWarnsRegex(DeprecationWarning, + r"bad character in group name '\+1' " + r"at position 3") as w: + re.sub('(?Px)', r'\g<+1>', 'xx') + self.assertEqual(w.filename, __file__) + with self.assertWarnsRegex(DeprecationWarning, + r"bad character in group name '1_0' " + r"at position 3"): + re.sub('()'*10, r'\g<1_0>', 'xx') + with self.assertWarnsRegex(DeprecationWarning, + r"bad character in group name ' 1 ' " + r"at position 3"): + re.sub('(?Px)', r'\g< 1 >', 'xx') self.checkTemplateError('(?Px)', r'\g', 'xx', "bad character in group name '?'", 3) + with self.assertWarnsRegex(DeprecationWarning, + r"bad character in group name '\\xc2\\xb5' " + r"at position 3") as w: + with self.assertRaisesRegex(IndexError, "unknown group name '\xc2\xb5'"): + re.sub(b'(?Px)', b'\\g<\xc2\xb5>', b'xx') + self.assertEqual(w.filename, __file__) self.checkTemplateError('(?Px)', r'\g', 'xx', "bad character in group name '?'", 3) self.checkTemplateError('(?Px)', r'\g', 'xx', "bad character in group name '?'", 3) + with self.assertWarnsRegex(DeprecationWarning, + r"bad character in group name '?' " + r"at position 3"): + re.sub('(?Px)', r'\g', 'xx') def test_re_subn(self): self.assertEqual(re.subn("(?i)b+", "x", "bbbb BBBB"), ('x x', 2)) @@ -577,10 +616,27 @@ def test_re_groupref_exists_errors(self): self.checkPatternError(r'(?P)(?(0)a|b)', 'bad group number', 10) self.checkPatternError(r'()(?(-1)a|b)', "bad character in group name '-1'", 5) + with self.assertWarnsRegex(DeprecationWarning, + r"bad character in group name '\+1' " + r"at position 5") as w: + re.compile(r'()(?(+1)a|b)') + self.assertEqual(w.filename, __file__) + with self.assertWarnsRegex(DeprecationWarning, + r"bad character in group name '1_0' " + r"at position 23"): + re.compile(r'()'*10 + r'(?(1_0)a|b)') + with self.assertWarnsRegex(DeprecationWarning, + r"bad character in group name ' 1 ' " + r"at position 5"): + re.compile(r'()(?( 1 )a|b)') self.checkPatternError(r'()(?(?)a|b)', "bad character in group name '?'", 5) self.checkPatternError(r'()(?(?)a|b)', "bad character in group name '?'", 5) + with self.assertWarnsRegex(DeprecationWarning, + r"bad character in group name '?' " + r"at position 5"): + re.compile(r'()(?(?)a|b)') self.checkPatternError(r'()(?(1', "missing ), unterminated name", 5) self.checkPatternError(r'()(?(1)a', diff --git a/Misc/NEWS.d/next/Library/2022-04-21-19-46-03.gh-issue-91760.zDtv1E.rst b/Misc/NEWS.d/next/Library/2022-04-21-19-46-03.gh-issue-91760.zDtv1E.rst new file mode 100644 index 0000000000000..0bddbbe093144 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-21-19-46-03.gh-issue-91760.zDtv1E.rst @@ -0,0 +1,4 @@ +More strict rules will be applied for numerical group references and group +names in regular expressions. For now, a deprecation warning is emitted for +group references and group names which will be errors in future Python +versions. From webhook-mailer at python.org Sat Apr 30 06:15:09 2022 From: webhook-mailer at python.org (serhiy-storchaka) Date: Sat, 30 Apr 2022 10:15:09 -0000 Subject: [Python-checkins] gh-91583: AC: Fix regression for functions with defining_class (GH-91739) Message-ID: https://github.com/python/cpython/commit/a055dac0b45031878a8196a8735522de018491e3 commit: a055dac0b45031878a8196a8735522de018491e3 branch: main author: Serhiy Storchaka committer: serhiy-storchaka date: 2022-04-30T13:15:02+03:00 summary: gh-91583: AC: Fix regression for functions with defining_class (GH-91739) Argument Clinic now generates the same efficient code as before adding the defining_class parameter. files: A Misc/NEWS.d/next/Tools-Demos/2022-04-20-14-26-14.gh-issue-91583.200qI0.rst M Modules/_sqlite/clinic/connection.c.h M Modules/_sre/clinic/sre.c.h M Modules/cjkcodecs/clinic/multibytecodec.c.h M Modules/clinic/_curses_panel.c.h M Modules/clinic/_dbmmodule.c.h M Modules/clinic/_gdbmmodule.c.h M Modules/clinic/_lsprof.c.h M Modules/clinic/_queuemodule.c.h M Modules/clinic/_testmultiphase.c.h M Modules/clinic/arraymodule.c.h M Modules/clinic/md5module.c.h M Modules/clinic/posixmodule.c.h M Modules/clinic/pyexpat.c.h M Modules/clinic/sha1module.c.h M Modules/clinic/sha256module.c.h M Modules/clinic/sha512module.c.h M Modules/clinic/zlibmodule.c.h M Tools/clinic/clinic.py diff --git a/Misc/NEWS.d/next/Tools-Demos/2022-04-20-14-26-14.gh-issue-91583.200qI0.rst b/Misc/NEWS.d/next/Tools-Demos/2022-04-20-14-26-14.gh-issue-91583.200qI0.rst new file mode 100644 index 0000000000000..bdfa71100f95a --- /dev/null +++ b/Misc/NEWS.d/next/Tools-Demos/2022-04-20-14-26-14.gh-issue-91583.200qI0.rst @@ -0,0 +1,2 @@ +Fix regression in the code generated by Argument Clinic for functions with +the ``defining_class`` parameter. diff --git a/Modules/_sqlite/clinic/connection.c.h b/Modules/_sqlite/clinic/connection.c.h index d4597086f4c9a..b012c4d4bbfa3 100644 --- a/Modules/_sqlite/clinic/connection.c.h +++ b/Modules/_sqlite/clinic/connection.c.h @@ -323,16 +323,44 @@ pysqlite_connection_create_function(pysqlite_Connection *self, PyTypeObject *cls { PyObject *return_value = NULL; static const char * const _keywords[] = {"name", "narg", "func", "deterministic", NULL}; - static _PyArg_Parser _parser = {"siO|$p:create_function", _keywords, 0}; + static _PyArg_Parser _parser = {NULL, _keywords, "create_function", 0}; + PyObject *argsbuf[4]; + Py_ssize_t noptargs = nargs + (kwnames ? PyTuple_GET_SIZE(kwnames) : 0) - 3; const char *name; int narg; PyObject *func; int deterministic = 0; - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser, - &name, &narg, &func, &deterministic)) { + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 3, 3, 0, argsbuf); + if (!args) { + goto exit; + } + if (!PyUnicode_Check(args[0])) { + _PyArg_BadArgument("create_function", "argument 'name'", "str", args[0]); + goto exit; + } + Py_ssize_t name_length; + name = PyUnicode_AsUTF8AndSize(args[0], &name_length); + if (name == NULL) { + goto exit; + } + if (strlen(name) != (size_t)name_length) { + PyErr_SetString(PyExc_ValueError, "embedded null character"); + goto exit; + } + narg = _PyLong_AsInt(args[1]); + if (narg == -1 && PyErr_Occurred()) { + goto exit; + } + func = args[2]; + if (!noptargs) { + goto skip_optional_kwonly; + } + deterministic = PyObject_IsTrue(args[3]); + if (deterministic < 0) { goto exit; } +skip_optional_kwonly: return_value = pysqlite_connection_create_function_impl(self, cls, name, narg, func, deterministic); exit: @@ -369,15 +397,34 @@ create_window_function(pysqlite_Connection *self, PyTypeObject *cls, PyObject *c { PyObject *return_value = NULL; static const char * const _keywords[] = {"", "", "", NULL}; - static _PyArg_Parser _parser = {"siO:create_window_function", _keywords, 0}; + static _PyArg_Parser _parser = {NULL, _keywords, "create_window_function", 0}; + PyObject *argsbuf[3]; const char *name; int num_params; PyObject *aggregate_class; - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser, - &name, &num_params, &aggregate_class)) { + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 3, 3, 0, argsbuf); + if (!args) { + goto exit; + } + if (!PyUnicode_Check(args[0])) { + _PyArg_BadArgument("create_window_function", "argument 1", "str", args[0]); + goto exit; + } + Py_ssize_t name_length; + name = PyUnicode_AsUTF8AndSize(args[0], &name_length); + if (name == NULL) { + goto exit; + } + if (strlen(name) != (size_t)name_length) { + PyErr_SetString(PyExc_ValueError, "embedded null character"); goto exit; } + num_params = _PyLong_AsInt(args[1]); + if (num_params == -1 && PyErr_Occurred()) { + goto exit; + } + aggregate_class = args[2]; return_value = create_window_function_impl(self, cls, name, num_params, aggregate_class); exit: @@ -406,15 +453,34 @@ pysqlite_connection_create_aggregate(pysqlite_Connection *self, PyTypeObject *cl { PyObject *return_value = NULL; static const char * const _keywords[] = {"name", "n_arg", "aggregate_class", NULL}; - static _PyArg_Parser _parser = {"siO:create_aggregate", _keywords, 0}; + static _PyArg_Parser _parser = {NULL, _keywords, "create_aggregate", 0}; + PyObject *argsbuf[3]; const char *name; int n_arg; PyObject *aggregate_class; - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser, - &name, &n_arg, &aggregate_class)) { + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 3, 3, 0, argsbuf); + if (!args) { + goto exit; + } + if (!PyUnicode_Check(args[0])) { + _PyArg_BadArgument("create_aggregate", "argument 'name'", "str", args[0]); + goto exit; + } + Py_ssize_t name_length; + name = PyUnicode_AsUTF8AndSize(args[0], &name_length); + if (name == NULL) { + goto exit; + } + if (strlen(name) != (size_t)name_length) { + PyErr_SetString(PyExc_ValueError, "embedded null character"); + goto exit; + } + n_arg = _PyLong_AsInt(args[1]); + if (n_arg == -1 && PyErr_Occurred()) { goto exit; } + aggregate_class = args[2]; return_value = pysqlite_connection_create_aggregate_impl(self, cls, name, n_arg, aggregate_class); exit: @@ -440,13 +506,15 @@ pysqlite_connection_set_authorizer(pysqlite_Connection *self, PyTypeObject *cls, { PyObject *return_value = NULL; static const char * const _keywords[] = {"authorizer_callback", NULL}; - static _PyArg_Parser _parser = {"O:set_authorizer", _keywords, 0}; + static _PyArg_Parser _parser = {NULL, _keywords, "set_authorizer", 0}; + PyObject *argsbuf[1]; PyObject *callable; - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser, - &callable)) { + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 1, 1, 0, argsbuf); + if (!args) { goto exit; } + callable = args[0]; return_value = pysqlite_connection_set_authorizer_impl(self, cls, callable); exit: @@ -472,12 +540,18 @@ pysqlite_connection_set_progress_handler(pysqlite_Connection *self, PyTypeObject { PyObject *return_value = NULL; static const char * const _keywords[] = {"progress_handler", "n", NULL}; - static _PyArg_Parser _parser = {"Oi:set_progress_handler", _keywords, 0}; + static _PyArg_Parser _parser = {NULL, _keywords, "set_progress_handler", 0}; + PyObject *argsbuf[2]; PyObject *callable; int n; - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser, - &callable, &n)) { + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 2, 2, 0, argsbuf); + if (!args) { + goto exit; + } + callable = args[0]; + n = _PyLong_AsInt(args[1]); + if (n == -1 && PyErr_Occurred()) { goto exit; } return_value = pysqlite_connection_set_progress_handler_impl(self, cls, callable, n); @@ -505,13 +579,15 @@ pysqlite_connection_set_trace_callback(pysqlite_Connection *self, PyTypeObject * { PyObject *return_value = NULL; static const char * const _keywords[] = {"trace_callback", NULL}; - static _PyArg_Parser _parser = {"O:set_trace_callback", _keywords, 0}; + static _PyArg_Parser _parser = {NULL, _keywords, "set_trace_callback", 0}; + PyObject *argsbuf[1]; PyObject *callable; - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser, - &callable)) { + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 1, 1, 0, argsbuf); + if (!args) { goto exit; } + callable = args[0]; return_value = pysqlite_connection_set_trace_callback_impl(self, cls, callable); exit: @@ -830,14 +906,29 @@ pysqlite_connection_create_collation(pysqlite_Connection *self, PyTypeObject *cl { PyObject *return_value = NULL; static const char * const _keywords[] = {"", "", NULL}; - static _PyArg_Parser _parser = {"sO:create_collation", _keywords, 0}; + static _PyArg_Parser _parser = {NULL, _keywords, "create_collation", 0}; + PyObject *argsbuf[2]; const char *name; PyObject *callable; - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser, - &name, &callable)) { + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 2, 2, 0, argsbuf); + if (!args) { + goto exit; + } + if (!PyUnicode_Check(args[0])) { + _PyArg_BadArgument("create_collation", "argument 1", "str", args[0]); + goto exit; + } + Py_ssize_t name_length; + name = PyUnicode_AsUTF8AndSize(args[0], &name_length); + if (name == NULL) { + goto exit; + } + if (strlen(name) != (size_t)name_length) { + PyErr_SetString(PyExc_ValueError, "embedded null character"); goto exit; } + callable = args[1]; return_value = pysqlite_connection_create_collation_impl(self, cls, name, callable); exit: @@ -1145,4 +1236,4 @@ getlimit(pysqlite_Connection *self, PyObject *arg) #ifndef DESERIALIZE_METHODDEF #define DESERIALIZE_METHODDEF #endif /* !defined(DESERIALIZE_METHODDEF) */ -/*[clinic end generated code: output=be2f526e78fa65b1 input=a9049054013a1b77]*/ +/*[clinic end generated code: output=6c9432a9fe0a4f5e input=a9049054013a1b77]*/ diff --git a/Modules/_sre/clinic/sre.c.h b/Modules/_sre/clinic/sre.c.h index 34cbe21f14071..c1b247ef7e9dd 100644 --- a/Modules/_sre/clinic/sre.c.h +++ b/Modules/_sre/clinic/sre.c.h @@ -176,15 +176,51 @@ _sre_SRE_Pattern_match(PatternObject *self, PyTypeObject *cls, PyObject *const * { PyObject *return_value = NULL; static const char * const _keywords[] = {"string", "pos", "endpos", NULL}; - static _PyArg_Parser _parser = {"O|nn:match", _keywords, 0}; + static _PyArg_Parser _parser = {NULL, _keywords, "match", 0}; + PyObject *argsbuf[3]; + Py_ssize_t noptargs = nargs + (kwnames ? PyTuple_GET_SIZE(kwnames) : 0) - 1; PyObject *string; Py_ssize_t pos = 0; Py_ssize_t endpos = PY_SSIZE_T_MAX; - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser, - &string, &pos, &endpos)) { + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 1, 3, 0, argsbuf); + if (!args) { goto exit; } + string = args[0]; + if (!noptargs) { + goto skip_optional_pos; + } + if (args[1]) { + { + Py_ssize_t ival = -1; + PyObject *iobj = _PyNumber_Index(args[1]); + if (iobj != NULL) { + ival = PyLong_AsSsize_t(iobj); + Py_DECREF(iobj); + } + if (ival == -1 && PyErr_Occurred()) { + goto exit; + } + pos = ival; + } + if (!--noptargs) { + goto skip_optional_pos; + } + } + { + Py_ssize_t ival = -1; + PyObject *iobj = _PyNumber_Index(args[2]); + if (iobj != NULL) { + ival = PyLong_AsSsize_t(iobj); + Py_DECREF(iobj); + } + if (ival == -1 && PyErr_Occurred()) { + goto exit; + } + endpos = ival; + } +skip_optional_pos: return_value = _sre_SRE_Pattern_match_impl(self, cls, string, pos, endpos); exit: @@ -210,15 +246,51 @@ _sre_SRE_Pattern_fullmatch(PatternObject *self, PyTypeObject *cls, PyObject *con { PyObject *return_value = NULL; static const char * const _keywords[] = {"string", "pos", "endpos", NULL}; - static _PyArg_Parser _parser = {"O|nn:fullmatch", _keywords, 0}; + static _PyArg_Parser _parser = {NULL, _keywords, "fullmatch", 0}; + PyObject *argsbuf[3]; + Py_ssize_t noptargs = nargs + (kwnames ? PyTuple_GET_SIZE(kwnames) : 0) - 1; PyObject *string; Py_ssize_t pos = 0; Py_ssize_t endpos = PY_SSIZE_T_MAX; - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser, - &string, &pos, &endpos)) { + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 1, 3, 0, argsbuf); + if (!args) { goto exit; } + string = args[0]; + if (!noptargs) { + goto skip_optional_pos; + } + if (args[1]) { + { + Py_ssize_t ival = -1; + PyObject *iobj = _PyNumber_Index(args[1]); + if (iobj != NULL) { + ival = PyLong_AsSsize_t(iobj); + Py_DECREF(iobj); + } + if (ival == -1 && PyErr_Occurred()) { + goto exit; + } + pos = ival; + } + if (!--noptargs) { + goto skip_optional_pos; + } + } + { + Py_ssize_t ival = -1; + PyObject *iobj = _PyNumber_Index(args[2]); + if (iobj != NULL) { + ival = PyLong_AsSsize_t(iobj); + Py_DECREF(iobj); + } + if (ival == -1 && PyErr_Occurred()) { + goto exit; + } + endpos = ival; + } +skip_optional_pos: return_value = _sre_SRE_Pattern_fullmatch_impl(self, cls, string, pos, endpos); exit: @@ -246,15 +318,51 @@ _sre_SRE_Pattern_search(PatternObject *self, PyTypeObject *cls, PyObject *const { PyObject *return_value = NULL; static const char * const _keywords[] = {"string", "pos", "endpos", NULL}; - static _PyArg_Parser _parser = {"O|nn:search", _keywords, 0}; + static _PyArg_Parser _parser = {NULL, _keywords, "search", 0}; + PyObject *argsbuf[3]; + Py_ssize_t noptargs = nargs + (kwnames ? PyTuple_GET_SIZE(kwnames) : 0) - 1; PyObject *string; Py_ssize_t pos = 0; Py_ssize_t endpos = PY_SSIZE_T_MAX; - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser, - &string, &pos, &endpos)) { + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 1, 3, 0, argsbuf); + if (!args) { goto exit; } + string = args[0]; + if (!noptargs) { + goto skip_optional_pos; + } + if (args[1]) { + { + Py_ssize_t ival = -1; + PyObject *iobj = _PyNumber_Index(args[1]); + if (iobj != NULL) { + ival = PyLong_AsSsize_t(iobj); + Py_DECREF(iobj); + } + if (ival == -1 && PyErr_Occurred()) { + goto exit; + } + pos = ival; + } + if (!--noptargs) { + goto skip_optional_pos; + } + } + { + Py_ssize_t ival = -1; + PyObject *iobj = _PyNumber_Index(args[2]); + if (iobj != NULL) { + ival = PyLong_AsSsize_t(iobj); + Py_DECREF(iobj); + } + if (ival == -1 && PyErr_Occurred()) { + goto exit; + } + endpos = ival; + } +skip_optional_pos: return_value = _sre_SRE_Pattern_search_impl(self, cls, string, pos, endpos); exit: @@ -351,15 +459,51 @@ _sre_SRE_Pattern_finditer(PatternObject *self, PyTypeObject *cls, PyObject *cons { PyObject *return_value = NULL; static const char * const _keywords[] = {"string", "pos", "endpos", NULL}; - static _PyArg_Parser _parser = {"O|nn:finditer", _keywords, 0}; + static _PyArg_Parser _parser = {NULL, _keywords, "finditer", 0}; + PyObject *argsbuf[3]; + Py_ssize_t noptargs = nargs + (kwnames ? PyTuple_GET_SIZE(kwnames) : 0) - 1; PyObject *string; Py_ssize_t pos = 0; Py_ssize_t endpos = PY_SSIZE_T_MAX; - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser, - &string, &pos, &endpos)) { + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 1, 3, 0, argsbuf); + if (!args) { goto exit; } + string = args[0]; + if (!noptargs) { + goto skip_optional_pos; + } + if (args[1]) { + { + Py_ssize_t ival = -1; + PyObject *iobj = _PyNumber_Index(args[1]); + if (iobj != NULL) { + ival = PyLong_AsSsize_t(iobj); + Py_DECREF(iobj); + } + if (ival == -1 && PyErr_Occurred()) { + goto exit; + } + pos = ival; + } + if (!--noptargs) { + goto skip_optional_pos; + } + } + { + Py_ssize_t ival = -1; + PyObject *iobj = _PyNumber_Index(args[2]); + if (iobj != NULL) { + ival = PyLong_AsSsize_t(iobj); + Py_DECREF(iobj); + } + if (ival == -1 && PyErr_Occurred()) { + goto exit; + } + endpos = ival; + } +skip_optional_pos: return_value = _sre_SRE_Pattern_finditer_impl(self, cls, string, pos, endpos); exit: @@ -384,15 +528,51 @@ _sre_SRE_Pattern_scanner(PatternObject *self, PyTypeObject *cls, PyObject *const { PyObject *return_value = NULL; static const char * const _keywords[] = {"string", "pos", "endpos", NULL}; - static _PyArg_Parser _parser = {"O|nn:scanner", _keywords, 0}; + static _PyArg_Parser _parser = {NULL, _keywords, "scanner", 0}; + PyObject *argsbuf[3]; + Py_ssize_t noptargs = nargs + (kwnames ? PyTuple_GET_SIZE(kwnames) : 0) - 1; PyObject *string; Py_ssize_t pos = 0; Py_ssize_t endpos = PY_SSIZE_T_MAX; - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser, - &string, &pos, &endpos)) { + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 1, 3, 0, argsbuf); + if (!args) { goto exit; } + string = args[0]; + if (!noptargs) { + goto skip_optional_pos; + } + if (args[1]) { + { + Py_ssize_t ival = -1; + PyObject *iobj = _PyNumber_Index(args[1]); + if (iobj != NULL) { + ival = PyLong_AsSsize_t(iobj); + Py_DECREF(iobj); + } + if (ival == -1 && PyErr_Occurred()) { + goto exit; + } + pos = ival; + } + if (!--noptargs) { + goto skip_optional_pos; + } + } + { + Py_ssize_t ival = -1; + PyObject *iobj = _PyNumber_Index(args[2]); + if (iobj != NULL) { + ival = PyLong_AsSsize_t(iobj); + Py_DECREF(iobj); + } + if (ival == -1 && PyErr_Occurred()) { + goto exit; + } + endpos = ival; + } +skip_optional_pos: return_value = _sre_SRE_Pattern_scanner_impl(self, cls, string, pos, endpos); exit: @@ -468,15 +648,35 @@ _sre_SRE_Pattern_sub(PatternObject *self, PyTypeObject *cls, PyObject *const *ar { PyObject *return_value = NULL; static const char * const _keywords[] = {"repl", "string", "count", NULL}; - static _PyArg_Parser _parser = {"OO|n:sub", _keywords, 0}; + static _PyArg_Parser _parser = {NULL, _keywords, "sub", 0}; + PyObject *argsbuf[3]; + Py_ssize_t noptargs = nargs + (kwnames ? PyTuple_GET_SIZE(kwnames) : 0) - 2; PyObject *repl; PyObject *string; Py_ssize_t count = 0; - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser, - &repl, &string, &count)) { + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 2, 3, 0, argsbuf); + if (!args) { goto exit; } + repl = args[0]; + string = args[1]; + if (!noptargs) { + goto skip_optional_pos; + } + { + Py_ssize_t ival = -1; + PyObject *iobj = _PyNumber_Index(args[2]); + if (iobj != NULL) { + ival = PyLong_AsSsize_t(iobj); + Py_DECREF(iobj); + } + if (ival == -1 && PyErr_Occurred()) { + goto exit; + } + count = ival; + } +skip_optional_pos: return_value = _sre_SRE_Pattern_sub_impl(self, cls, repl, string, count); exit: @@ -502,15 +702,35 @@ _sre_SRE_Pattern_subn(PatternObject *self, PyTypeObject *cls, PyObject *const *a { PyObject *return_value = NULL; static const char * const _keywords[] = {"repl", "string", "count", NULL}; - static _PyArg_Parser _parser = {"OO|n:subn", _keywords, 0}; + static _PyArg_Parser _parser = {NULL, _keywords, "subn", 0}; + PyObject *argsbuf[3]; + Py_ssize_t noptargs = nargs + (kwnames ? PyTuple_GET_SIZE(kwnames) : 0) - 2; PyObject *repl; PyObject *string; Py_ssize_t count = 0; - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser, - &repl, &string, &count)) { + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 2, 3, 0, argsbuf); + if (!args) { goto exit; } + repl = args[0]; + string = args[1]; + if (!noptargs) { + goto skip_optional_pos; + } + { + Py_ssize_t ival = -1; + PyObject *iobj = _PyNumber_Index(args[2]); + if (iobj != NULL) { + ival = PyLong_AsSsize_t(iobj); + Py_DECREF(iobj); + } + if (ival == -1 && PyErr_Occurred()) { + goto exit; + } + count = ival; + } +skip_optional_pos: return_value = _sre_SRE_Pattern_subn_impl(self, cls, repl, string, count); exit: @@ -882,18 +1102,11 @@ _sre_SRE_Scanner_match_impl(ScannerObject *self, PyTypeObject *cls); static PyObject * _sre_SRE_Scanner_match(ScannerObject *self, PyTypeObject *cls, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) { - PyObject *return_value = NULL; - static const char * const _keywords[] = { NULL}; - static _PyArg_Parser _parser = {":match", _keywords, 0}; - - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser - )) { - goto exit; + if (nargs) { + PyErr_SetString(PyExc_TypeError, "match() takes no arguments"); + return NULL; } - return_value = _sre_SRE_Scanner_match_impl(self, cls); - -exit: - return return_value; + return _sre_SRE_Scanner_match_impl(self, cls); } PyDoc_STRVAR(_sre_SRE_Scanner_search__doc__, @@ -910,17 +1123,10 @@ _sre_SRE_Scanner_search_impl(ScannerObject *self, PyTypeObject *cls); static PyObject * _sre_SRE_Scanner_search(ScannerObject *self, PyTypeObject *cls, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) { - PyObject *return_value = NULL; - static const char * const _keywords[] = { NULL}; - static _PyArg_Parser _parser = {":search", _keywords, 0}; - - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser - )) { - goto exit; + if (nargs) { + PyErr_SetString(PyExc_TypeError, "search() takes no arguments"); + return NULL; } - return_value = _sre_SRE_Scanner_search_impl(self, cls); - -exit: - return return_value; + return _sre_SRE_Scanner_search_impl(self, cls); } -/*[clinic end generated code: output=9d7510a57a157a38 input=a9049054013a1b77]*/ +/*[clinic end generated code: output=8d8200b2177cd667 input=a9049054013a1b77]*/ diff --git a/Modules/cjkcodecs/clinic/multibytecodec.c.h b/Modules/cjkcodecs/clinic/multibytecodec.c.h index 1dfb9a1224827..60515245490f7 100644 --- a/Modules/cjkcodecs/clinic/multibytecodec.c.h +++ b/Modules/cjkcodecs/clinic/multibytecodec.c.h @@ -493,13 +493,15 @@ _multibytecodec_MultibyteStreamWriter_write(MultibyteStreamWriterObject *self, P { PyObject *return_value = NULL; static const char * const _keywords[] = {"", NULL}; - static _PyArg_Parser _parser = {"O:write", _keywords, 0}; + static _PyArg_Parser _parser = {NULL, _keywords, "write", 0}; + PyObject *argsbuf[1]; PyObject *strobj; - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser, - &strobj)) { + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 1, 1, 0, argsbuf); + if (!args) { goto exit; } + strobj = args[0]; return_value = _multibytecodec_MultibyteStreamWriter_write_impl(self, cls, strobj); exit: @@ -524,13 +526,15 @@ _multibytecodec_MultibyteStreamWriter_writelines(MultibyteStreamWriterObject *se { PyObject *return_value = NULL; static const char * const _keywords[] = {"", NULL}; - static _PyArg_Parser _parser = {"O:writelines", _keywords, 0}; + static _PyArg_Parser _parser = {NULL, _keywords, "writelines", 0}; + PyObject *argsbuf[1]; PyObject *lines; - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser, - &lines)) { + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 1, 1, 0, argsbuf); + if (!args) { goto exit; } + lines = args[0]; return_value = _multibytecodec_MultibyteStreamWriter_writelines_impl(self, cls, lines); exit: @@ -552,18 +556,11 @@ _multibytecodec_MultibyteStreamWriter_reset_impl(MultibyteStreamWriterObject *se static PyObject * _multibytecodec_MultibyteStreamWriter_reset(MultibyteStreamWriterObject *self, PyTypeObject *cls, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) { - PyObject *return_value = NULL; - static const char * const _keywords[] = { NULL}; - static _PyArg_Parser _parser = {":reset", _keywords, 0}; - - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser - )) { - goto exit; + if (nargs) { + PyErr_SetString(PyExc_TypeError, "reset() takes no arguments"); + return NULL; } - return_value = _multibytecodec_MultibyteStreamWriter_reset_impl(self, cls); - -exit: - return return_value; + return _multibytecodec_MultibyteStreamWriter_reset_impl(self, cls); } PyDoc_STRVAR(_multibytecodec___create_codec__doc__, @@ -573,4 +570,4 @@ PyDoc_STRVAR(_multibytecodec___create_codec__doc__, #define _MULTIBYTECODEC___CREATE_CODEC_METHODDEF \ {"__create_codec", (PyCFunction)_multibytecodec___create_codec, METH_O, _multibytecodec___create_codec__doc__}, -/*[clinic end generated code: output=8813c05077580bda input=a9049054013a1b77]*/ +/*[clinic end generated code: output=0ee00c9f6883afe9 input=a9049054013a1b77]*/ diff --git a/Modules/clinic/_curses_panel.c.h b/Modules/clinic/_curses_panel.c.h index 45898070b1f54..e4642b58332de 100644 --- a/Modules/clinic/_curses_panel.c.h +++ b/Modules/clinic/_curses_panel.c.h @@ -17,18 +17,11 @@ _curses_panel_panel_bottom_impl(PyCursesPanelObject *self, PyTypeObject *cls); static PyObject * _curses_panel_panel_bottom(PyCursesPanelObject *self, PyTypeObject *cls, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) { - PyObject *return_value = NULL; - static const char * const _keywords[] = { NULL}; - static _PyArg_Parser _parser = {":bottom", _keywords, 0}; - - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser - )) { - goto exit; + if (nargs) { + PyErr_SetString(PyExc_TypeError, "bottom() takes no arguments"); + return NULL; } - return_value = _curses_panel_panel_bottom_impl(self, cls); - -exit: - return return_value; + return _curses_panel_panel_bottom_impl(self, cls); } PyDoc_STRVAR(_curses_panel_panel_hide__doc__, @@ -48,18 +41,11 @@ _curses_panel_panel_hide_impl(PyCursesPanelObject *self, PyTypeObject *cls); static PyObject * _curses_panel_panel_hide(PyCursesPanelObject *self, PyTypeObject *cls, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) { - PyObject *return_value = NULL; - static const char * const _keywords[] = { NULL}; - static _PyArg_Parser _parser = {":hide", _keywords, 0}; - - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser - )) { - goto exit; + if (nargs) { + PyErr_SetString(PyExc_TypeError, "hide() takes no arguments"); + return NULL; } - return_value = _curses_panel_panel_hide_impl(self, cls); - -exit: - return return_value; + return _curses_panel_panel_hide_impl(self, cls); } PyDoc_STRVAR(_curses_panel_panel_show__doc__, @@ -77,18 +63,11 @@ _curses_panel_panel_show_impl(PyCursesPanelObject *self, PyTypeObject *cls); static PyObject * _curses_panel_panel_show(PyCursesPanelObject *self, PyTypeObject *cls, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) { - PyObject *return_value = NULL; - static const char * const _keywords[] = { NULL}; - static _PyArg_Parser _parser = {":show", _keywords, 0}; - - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser - )) { - goto exit; + if (nargs) { + PyErr_SetString(PyExc_TypeError, "show() takes no arguments"); + return NULL; } - return_value = _curses_panel_panel_show_impl(self, cls); - -exit: - return return_value; + return _curses_panel_panel_show_impl(self, cls); } PyDoc_STRVAR(_curses_panel_panel_top__doc__, @@ -106,18 +85,11 @@ _curses_panel_panel_top_impl(PyCursesPanelObject *self, PyTypeObject *cls); static PyObject * _curses_panel_panel_top(PyCursesPanelObject *self, PyTypeObject *cls, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) { - PyObject *return_value = NULL; - static const char * const _keywords[] = { NULL}; - static _PyArg_Parser _parser = {":top", _keywords, 0}; - - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser - )) { - goto exit; + if (nargs) { + PyErr_SetString(PyExc_TypeError, "top() takes no arguments"); + return NULL; } - return_value = _curses_panel_panel_top_impl(self, cls); - -exit: - return return_value; + return _curses_panel_panel_top_impl(self, cls); } PyDoc_STRVAR(_curses_panel_panel_above__doc__, @@ -192,12 +164,21 @@ _curses_panel_panel_move(PyCursesPanelObject *self, PyTypeObject *cls, PyObject { PyObject *return_value = NULL; static const char * const _keywords[] = {"", "", NULL}; - static _PyArg_Parser _parser = {"ii:move", _keywords, 0}; + static _PyArg_Parser _parser = {NULL, _keywords, "move", 0}; + PyObject *argsbuf[2]; int y; int x; - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser, - &y, &x)) { + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 2, 2, 0, argsbuf); + if (!args) { + goto exit; + } + y = _PyLong_AsInt(args[0]); + if (y == -1 && PyErr_Occurred()) { + goto exit; + } + x = _PyLong_AsInt(args[1]); + if (x == -1 && PyErr_Occurred()) { goto exit; } return_value = _curses_panel_panel_move_impl(self, cls, y, x); @@ -243,13 +224,19 @@ _curses_panel_panel_replace(PyCursesPanelObject *self, PyTypeObject *cls, PyObje { PyObject *return_value = NULL; static const char * const _keywords[] = {"", NULL}; - static _PyArg_Parser _parser = {"O!:replace", _keywords, 0}; + static _PyArg_Parser _parser = {NULL, _keywords, "replace", 0}; + PyObject *argsbuf[1]; PyCursesWindowObject *win; - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser, - &PyCursesWindow_Type, &win)) { + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 1, 1, 0, argsbuf); + if (!args) { + goto exit; + } + if (!PyObject_TypeCheck(args[0], &PyCursesWindow_Type)) { + _PyArg_BadArgument("replace", "argument 1", (&PyCursesWindow_Type)->tp_name, args[0]); goto exit; } + win = (PyCursesWindowObject *)args[0]; return_value = _curses_panel_panel_replace_impl(self, cls, win); exit: @@ -274,13 +261,15 @@ _curses_panel_panel_set_userptr(PyCursesPanelObject *self, PyTypeObject *cls, Py { PyObject *return_value = NULL; static const char * const _keywords[] = {"", NULL}; - static _PyArg_Parser _parser = {"O:set_userptr", _keywords, 0}; + static _PyArg_Parser _parser = {NULL, _keywords, "set_userptr", 0}; + PyObject *argsbuf[1]; PyObject *obj; - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser, - &obj)) { + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 1, 1, 0, argsbuf); + if (!args) { goto exit; } + obj = args[0]; return_value = _curses_panel_panel_set_userptr_impl(self, cls, obj); exit: @@ -303,18 +292,11 @@ _curses_panel_panel_userptr_impl(PyCursesPanelObject *self, static PyObject * _curses_panel_panel_userptr(PyCursesPanelObject *self, PyTypeObject *cls, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) { - PyObject *return_value = NULL; - static const char * const _keywords[] = { NULL}; - static _PyArg_Parser _parser = {":userptr", _keywords, 0}; - - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser - )) { - goto exit; + if (nargs) { + PyErr_SetString(PyExc_TypeError, "userptr() takes no arguments"); + return NULL; } - return_value = _curses_panel_panel_userptr_impl(self, cls); - -exit: - return return_value; + return _curses_panel_panel_userptr_impl(self, cls); } PyDoc_STRVAR(_curses_panel_bottom_panel__doc__, @@ -401,4 +383,4 @@ _curses_panel_update_panels(PyObject *module, PyObject *Py_UNUSED(ignored)) { return _curses_panel_update_panels_impl(module); } -/*[clinic end generated code: output=3081ef24e5560cb0 input=a9049054013a1b77]*/ +/*[clinic end generated code: output=c552457e8067bb0a input=a9049054013a1b77]*/ diff --git a/Modules/clinic/_dbmmodule.c.h b/Modules/clinic/_dbmmodule.c.h index f0b8220e7fee2..a8b41607221f6 100644 --- a/Modules/clinic/_dbmmodule.c.h +++ b/Modules/clinic/_dbmmodule.c.h @@ -35,18 +35,11 @@ _dbm_dbm_keys_impl(dbmobject *self, PyTypeObject *cls); static PyObject * _dbm_dbm_keys(dbmobject *self, PyTypeObject *cls, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) { - PyObject *return_value = NULL; - static const char * const _keywords[] = { NULL}; - static _PyArg_Parser _parser = {":keys", _keywords, 0}; - - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser - )) { - goto exit; + if (nargs) { + PyErr_SetString(PyExc_TypeError, "keys() takes no arguments"); + return NULL; } - return_value = _dbm_dbm_keys_impl(self, cls); - -exit: - return return_value; + return _dbm_dbm_keys_impl(self, cls); } PyDoc_STRVAR(_dbm_dbm_get__doc__, @@ -179,4 +172,4 @@ dbmopen(PyObject *module, PyObject *const *args, Py_ssize_t nargs) exit: return return_value; } -/*[clinic end generated code: output=32ef6c0f8f2d3db9 input=a9049054013a1b77]*/ +/*[clinic end generated code: output=492be70729515fe3 input=a9049054013a1b77]*/ diff --git a/Modules/clinic/_gdbmmodule.c.h b/Modules/clinic/_gdbmmodule.c.h index a40e80d8e1a7b..ced48b99ed325 100644 --- a/Modules/clinic/_gdbmmodule.c.h +++ b/Modules/clinic/_gdbmmodule.c.h @@ -104,18 +104,11 @@ _gdbm_gdbm_keys_impl(gdbmobject *self, PyTypeObject *cls); static PyObject * _gdbm_gdbm_keys(gdbmobject *self, PyTypeObject *cls, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) { - PyObject *return_value = NULL; - static const char * const _keywords[] = { NULL}; - static _PyArg_Parser _parser = {":keys", _keywords, 0}; - - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser - )) { - goto exit; + if (nargs) { + PyErr_SetString(PyExc_TypeError, "keys() takes no arguments"); + return NULL; } - return_value = _gdbm_gdbm_keys_impl(self, cls); - -exit: - return return_value; + return _gdbm_gdbm_keys_impl(self, cls); } PyDoc_STRVAR(_gdbm_gdbm_firstkey__doc__, @@ -137,18 +130,11 @@ _gdbm_gdbm_firstkey_impl(gdbmobject *self, PyTypeObject *cls); static PyObject * _gdbm_gdbm_firstkey(gdbmobject *self, PyTypeObject *cls, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) { - PyObject *return_value = NULL; - static const char * const _keywords[] = { NULL}; - static _PyArg_Parser _parser = {":firstkey", _keywords, 0}; - - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser - )) { - goto exit; + if (nargs) { + PyErr_SetString(PyExc_TypeError, "firstkey() takes no arguments"); + return NULL; } - return_value = _gdbm_gdbm_firstkey_impl(self, cls); - -exit: - return return_value; + return _gdbm_gdbm_firstkey_impl(self, cls); } PyDoc_STRVAR(_gdbm_gdbm_nextkey__doc__, @@ -212,18 +198,11 @@ _gdbm_gdbm_reorganize_impl(gdbmobject *self, PyTypeObject *cls); static PyObject * _gdbm_gdbm_reorganize(gdbmobject *self, PyTypeObject *cls, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) { - PyObject *return_value = NULL; - static const char * const _keywords[] = { NULL}; - static _PyArg_Parser _parser = {":reorganize", _keywords, 0}; - - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser - )) { - goto exit; + if (nargs) { + PyErr_SetString(PyExc_TypeError, "reorganize() takes no arguments"); + return NULL; } - return_value = _gdbm_gdbm_reorganize_impl(self, cls); - -exit: - return return_value; + return _gdbm_gdbm_reorganize_impl(self, cls); } PyDoc_STRVAR(_gdbm_gdbm_sync__doc__, @@ -244,18 +223,11 @@ _gdbm_gdbm_sync_impl(gdbmobject *self, PyTypeObject *cls); static PyObject * _gdbm_gdbm_sync(gdbmobject *self, PyTypeObject *cls, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) { - PyObject *return_value = NULL; - static const char * const _keywords[] = { NULL}; - static _PyArg_Parser _parser = {":sync", _keywords, 0}; - - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser - )) { - goto exit; + if (nargs) { + PyErr_SetString(PyExc_TypeError, "sync() takes no arguments"); + return NULL; } - return_value = _gdbm_gdbm_sync_impl(self, cls); - -exit: - return return_value; + return _gdbm_gdbm_sync_impl(self, cls); } PyDoc_STRVAR(dbmopen__doc__, @@ -333,4 +305,4 @@ dbmopen(PyObject *module, PyObject *const *args, Py_ssize_t nargs) exit: return return_value; } -/*[clinic end generated code: output=63c507f93d84a3a4 input=a9049054013a1b77]*/ +/*[clinic end generated code: output=fe7a0812eb560b23 input=a9049054013a1b77]*/ diff --git a/Modules/clinic/_lsprof.c.h b/Modules/clinic/_lsprof.c.h index 5d9c209eab856..b0adb746974ab 100644 --- a/Modules/clinic/_lsprof.c.h +++ b/Modules/clinic/_lsprof.c.h @@ -39,17 +39,10 @@ _lsprof_Profiler_getstats_impl(ProfilerObject *self, PyTypeObject *cls); static PyObject * _lsprof_Profiler_getstats(ProfilerObject *self, PyTypeObject *cls, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) { - PyObject *return_value = NULL; - static const char * const _keywords[] = { NULL}; - static _PyArg_Parser _parser = {":getstats", _keywords, 0}; - - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser - )) { - goto exit; + if (nargs) { + PyErr_SetString(PyExc_TypeError, "getstats() takes no arguments"); + return NULL; } - return_value = _lsprof_Profiler_getstats_impl(self, cls); - -exit: - return return_value; + return _lsprof_Profiler_getstats_impl(self, cls); } -/*[clinic end generated code: output=b4727cfebecdd22d input=a9049054013a1b77]*/ +/*[clinic end generated code: output=57c7b6b0b8666429 input=a9049054013a1b77]*/ diff --git a/Modules/clinic/_queuemodule.c.h b/Modules/clinic/_queuemodule.c.h index 22d2e992b6398..056a8ef52e00f 100644 --- a/Modules/clinic/_queuemodule.c.h +++ b/Modules/clinic/_queuemodule.c.h @@ -146,14 +146,30 @@ _queue_SimpleQueue_get(simplequeueobject *self, PyTypeObject *cls, PyObject *con { PyObject *return_value = NULL; static const char * const _keywords[] = {"block", "timeout", NULL}; - static _PyArg_Parser _parser = {"|pO:get", _keywords, 0}; + static _PyArg_Parser _parser = {NULL, _keywords, "get", 0}; + PyObject *argsbuf[2]; + Py_ssize_t noptargs = nargs + (kwnames ? PyTuple_GET_SIZE(kwnames) : 0) - 0; int block = 1; PyObject *timeout_obj = Py_None; - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser, - &block, &timeout_obj)) { + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 0, 2, 0, argsbuf); + if (!args) { goto exit; } + if (!noptargs) { + goto skip_optional_pos; + } + if (args[0]) { + block = PyObject_IsTrue(args[0]); + if (block < 0) { + goto exit; + } + if (!--noptargs) { + goto skip_optional_pos; + } + } + timeout_obj = args[1]; +skip_optional_pos: return_value = _queue_SimpleQueue_get_impl(self, cls, block, timeout_obj); exit: @@ -179,18 +195,11 @@ _queue_SimpleQueue_get_nowait_impl(simplequeueobject *self, static PyObject * _queue_SimpleQueue_get_nowait(simplequeueobject *self, PyTypeObject *cls, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) { - PyObject *return_value = NULL; - static const char * const _keywords[] = { NULL}; - static _PyArg_Parser _parser = {":get_nowait", _keywords, 0}; - - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser - )) { - goto exit; + if (nargs) { + PyErr_SetString(PyExc_TypeError, "get_nowait() takes no arguments"); + return NULL; } - return_value = _queue_SimpleQueue_get_nowait_impl(self, cls); - -exit: - return return_value; + return _queue_SimpleQueue_get_nowait_impl(self, cls); } PyDoc_STRVAR(_queue_SimpleQueue_empty__doc__, @@ -248,4 +257,4 @@ _queue_SimpleQueue_qsize(simplequeueobject *self, PyObject *Py_UNUSED(ignored)) exit: return return_value; } -/*[clinic end generated code: output=acfaf0191d8935db input=a9049054013a1b77]*/ +/*[clinic end generated code: output=36301c405a858f39 input=a9049054013a1b77]*/ diff --git a/Modules/clinic/_testmultiphase.c.h b/Modules/clinic/_testmultiphase.c.h index b8ee93c6e19ea..753670934bf81 100644 --- a/Modules/clinic/_testmultiphase.c.h +++ b/Modules/clinic/_testmultiphase.c.h @@ -21,18 +21,11 @@ _testmultiphase_StateAccessType_get_defining_module_impl(StateAccessTypeObject * static PyObject * _testmultiphase_StateAccessType_get_defining_module(StateAccessTypeObject *self, PyTypeObject *cls, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) { - PyObject *return_value = NULL; - static const char * const _keywords[] = { NULL}; - static _PyArg_Parser _parser = {":get_defining_module", _keywords, 0}; - - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser - )) { - goto exit; + if (nargs) { + PyErr_SetString(PyExc_TypeError, "get_defining_module() takes no arguments"); + return NULL; } - return_value = _testmultiphase_StateAccessType_get_defining_module_impl(self, cls); - -exit: - return return_value; + return _testmultiphase_StateAccessType_get_defining_module_impl(self, cls); } PyDoc_STRVAR(_testmultiphase_StateAccessType_getmodulebydef_bad_def__doc__, @@ -51,18 +44,11 @@ _testmultiphase_StateAccessType_getmodulebydef_bad_def_impl(StateAccessTypeObjec static PyObject * _testmultiphase_StateAccessType_getmodulebydef_bad_def(StateAccessTypeObject *self, PyTypeObject *cls, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) { - PyObject *return_value = NULL; - static const char * const _keywords[] = { NULL}; - static _PyArg_Parser _parser = {":getmodulebydef_bad_def", _keywords, 0}; - - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser - )) { - goto exit; + if (nargs) { + PyErr_SetString(PyExc_TypeError, "getmodulebydef_bad_def() takes no arguments"); + return NULL; } - return_value = _testmultiphase_StateAccessType_getmodulebydef_bad_def_impl(self, cls); - -exit: - return return_value; + return _testmultiphase_StateAccessType_getmodulebydef_bad_def_impl(self, cls); } PyDoc_STRVAR(_testmultiphase_StateAccessType_increment_count_clinic__doc__, @@ -88,14 +74,37 @@ _testmultiphase_StateAccessType_increment_count_clinic(StateAccessTypeObject *se { PyObject *return_value = NULL; static const char * const _keywords[] = {"n", "twice", NULL}; - static _PyArg_Parser _parser = {"|i$p:increment_count_clinic", _keywords, 0}; + static _PyArg_Parser _parser = {NULL, _keywords, "increment_count_clinic", 0}; + PyObject *argsbuf[2]; + Py_ssize_t noptargs = nargs + (kwnames ? PyTuple_GET_SIZE(kwnames) : 0) - 0; int n = 1; int twice = 0; - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser, - &n, &twice)) { + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 0, 1, 0, argsbuf); + if (!args) { goto exit; } + if (!noptargs) { + goto skip_optional_pos; + } + if (args[0]) { + n = _PyLong_AsInt(args[0]); + if (n == -1 && PyErr_Occurred()) { + goto exit; + } + if (!--noptargs) { + goto skip_optional_pos; + } + } +skip_optional_pos: + if (!noptargs) { + goto skip_optional_kwonly; + } + twice = PyObject_IsTrue(args[1]); + if (twice < 0) { + goto exit; + } +skip_optional_kwonly: return_value = _testmultiphase_StateAccessType_increment_count_clinic_impl(self, cls, n, twice); exit: @@ -118,17 +127,10 @@ _testmultiphase_StateAccessType_get_count_impl(StateAccessTypeObject *self, static PyObject * _testmultiphase_StateAccessType_get_count(StateAccessTypeObject *self, PyTypeObject *cls, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) { - PyObject *return_value = NULL; - static const char * const _keywords[] = { NULL}; - static _PyArg_Parser _parser = {":get_count", _keywords, 0}; - - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser - )) { - goto exit; + if (nargs) { + PyErr_SetString(PyExc_TypeError, "get_count() takes no arguments"); + return NULL; } - return_value = _testmultiphase_StateAccessType_get_count_impl(self, cls); - -exit: - return return_value; + return _testmultiphase_StateAccessType_get_count_impl(self, cls); } -/*[clinic end generated code: output=e8d074b4e6437438 input=a9049054013a1b77]*/ +/*[clinic end generated code: output=b41a400ef84f50af input=a9049054013a1b77]*/ diff --git a/Modules/clinic/arraymodule.c.h b/Modules/clinic/arraymodule.c.h index 583ff28bf3287..030c6e7e1a487 100644 --- a/Modules/clinic/arraymodule.c.h +++ b/Modules/clinic/arraymodule.c.h @@ -155,13 +155,15 @@ array_array_extend(arrayobject *self, PyTypeObject *cls, PyObject *const *args, { PyObject *return_value = NULL; static const char * const _keywords[] = {"", NULL}; - static _PyArg_Parser _parser = {"O:extend", _keywords, 0}; + static _PyArg_Parser _parser = {NULL, _keywords, "extend", 0}; + PyObject *argsbuf[1]; PyObject *bb; - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser, - &bb)) { + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 1, 1, 0, argsbuf); + if (!args) { goto exit; } + bb = args[0]; return_value = array_array_extend_impl(self, cls, bb); exit: @@ -296,14 +298,28 @@ array_array_fromfile(arrayobject *self, PyTypeObject *cls, PyObject *const *args { PyObject *return_value = NULL; static const char * const _keywords[] = {"", "", NULL}; - static _PyArg_Parser _parser = {"On:fromfile", _keywords, 0}; + static _PyArg_Parser _parser = {NULL, _keywords, "fromfile", 0}; + PyObject *argsbuf[2]; PyObject *f; Py_ssize_t n; - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser, - &f, &n)) { + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 2, 2, 0, argsbuf); + if (!args) { goto exit; } + f = args[0]; + { + Py_ssize_t ival = -1; + PyObject *iobj = _PyNumber_Index(args[1]); + if (iobj != NULL) { + ival = PyLong_AsSsize_t(iobj); + Py_DECREF(iobj); + } + if (ival == -1 && PyErr_Occurred()) { + goto exit; + } + n = ival; + } return_value = array_array_fromfile_impl(self, cls, f, n); exit: @@ -327,13 +343,15 @@ array_array_tofile(arrayobject *self, PyTypeObject *cls, PyObject *const *args, { PyObject *return_value = NULL; static const char * const _keywords[] = {"", NULL}; - static _PyArg_Parser _parser = {"O:tofile", _keywords, 0}; + static _PyArg_Parser _parser = {NULL, _keywords, "tofile", 0}; + PyObject *argsbuf[1]; PyObject *f; - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser, - &f)) { + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 1, 1, 0, argsbuf); + if (!args) { goto exit; } + f = args[0]; return_value = array_array_tofile_impl(self, cls, f); exit: @@ -567,13 +585,15 @@ array_array___reduce_ex__(arrayobject *self, PyTypeObject *cls, PyObject *const { PyObject *return_value = NULL; static const char * const _keywords[] = {"", NULL}; - static _PyArg_Parser _parser = {"O:__reduce_ex__", _keywords, 0}; + static _PyArg_Parser _parser = {NULL, _keywords, "__reduce_ex__", 0}; + PyObject *argsbuf[1]; PyObject *value; - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser, - &value)) { + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 1, 1, 0, argsbuf); + if (!args) { goto exit; } + value = args[0]; return_value = array_array___reduce_ex___impl(self, cls, value); exit: @@ -595,18 +615,11 @@ array_arrayiterator___reduce___impl(arrayiterobject *self, PyTypeObject *cls); static PyObject * array_arrayiterator___reduce__(arrayiterobject *self, PyTypeObject *cls, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) { - PyObject *return_value = NULL; - static const char * const _keywords[] = { NULL}; - static _PyArg_Parser _parser = {":__reduce__", _keywords, 0}; - - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser - )) { - goto exit; + if (nargs) { + PyErr_SetString(PyExc_TypeError, "__reduce__() takes no arguments"); + return NULL; } - return_value = array_arrayiterator___reduce___impl(self, cls); - -exit: - return return_value; + return array_arrayiterator___reduce___impl(self, cls); } PyDoc_STRVAR(array_arrayiterator___setstate____doc__, @@ -617,4 +630,4 @@ PyDoc_STRVAR(array_arrayiterator___setstate____doc__, #define ARRAY_ARRAYITERATOR___SETSTATE___METHODDEF \ {"__setstate__", (PyCFunction)array_arrayiterator___setstate__, METH_O, array_arrayiterator___setstate____doc__}, -/*[clinic end generated code: output=1db6decd8492bf91 input=a9049054013a1b77]*/ +/*[clinic end generated code: output=7f48d1691fa27442 input=a9049054013a1b77]*/ diff --git a/Modules/clinic/md5module.c.h b/Modules/clinic/md5module.c.h index 4762f2800d4b8..0ff09b0050a56 100644 --- a/Modules/clinic/md5module.c.h +++ b/Modules/clinic/md5module.c.h @@ -17,18 +17,11 @@ MD5Type_copy_impl(MD5object *self, PyTypeObject *cls); static PyObject * MD5Type_copy(MD5object *self, PyTypeObject *cls, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) { - PyObject *return_value = NULL; - static const char * const _keywords[] = { NULL}; - static _PyArg_Parser _parser = {":copy", _keywords, 0}; - - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser - )) { - goto exit; + if (nargs) { + PyErr_SetString(PyExc_TypeError, "copy() takes no arguments"); + return NULL; } - return_value = MD5Type_copy_impl(self, cls); - -exit: - return return_value; + return MD5Type_copy_impl(self, cls); } PyDoc_STRVAR(MD5Type_digest__doc__, @@ -126,4 +119,4 @@ _md5_md5(PyObject *module, PyObject *const *args, Py_ssize_t nargs, PyObject *kw exit: return return_value; } -/*[clinic end generated code: output=53ff7f22dbaaea36 input=a9049054013a1b77]*/ +/*[clinic end generated code: output=3061297a669c645c input=a9049054013a1b77]*/ diff --git a/Modules/clinic/posixmodule.c.h b/Modules/clinic/posixmodule.c.h index 282a5410f7020..352dadc9b5627 100644 --- a/Modules/clinic/posixmodule.c.h +++ b/Modules/clinic/posixmodule.c.h @@ -8290,12 +8290,10 @@ static PyObject * os_DirEntry_is_symlink(DirEntry *self, PyTypeObject *defining_class, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) { PyObject *return_value = NULL; - static const char * const _keywords[] = { NULL}; - static _PyArg_Parser _parser = {":is_symlink", _keywords, 0}; int _return_value; - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser - )) { + if (nargs) { + PyErr_SetString(PyExc_TypeError, "is_symlink() takes no arguments"); goto exit; } _return_value = os_DirEntry_is_symlink_impl(self, defining_class); @@ -8326,13 +8324,23 @@ os_DirEntry_stat(DirEntry *self, PyTypeObject *defining_class, PyObject *const * { PyObject *return_value = NULL; static const char * const _keywords[] = {"follow_symlinks", NULL}; - static _PyArg_Parser _parser = {"|$p:stat", _keywords, 0}; + static _PyArg_Parser _parser = {NULL, _keywords, "stat", 0}; + PyObject *argsbuf[1]; + Py_ssize_t noptargs = nargs + (kwnames ? PyTuple_GET_SIZE(kwnames) : 0) - 0; int follow_symlinks = 1; - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser, - &follow_symlinks)) { + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 0, 0, 0, argsbuf); + if (!args) { goto exit; } + if (!noptargs) { + goto skip_optional_kwonly; + } + follow_symlinks = PyObject_IsTrue(args[0]); + if (follow_symlinks < 0) { + goto exit; + } +skip_optional_kwonly: return_value = os_DirEntry_stat_impl(self, defining_class, follow_symlinks); exit: @@ -8357,14 +8365,24 @@ os_DirEntry_is_dir(DirEntry *self, PyTypeObject *defining_class, PyObject *const { PyObject *return_value = NULL; static const char * const _keywords[] = {"follow_symlinks", NULL}; - static _PyArg_Parser _parser = {"|$p:is_dir", _keywords, 0}; + static _PyArg_Parser _parser = {NULL, _keywords, "is_dir", 0}; + PyObject *argsbuf[1]; + Py_ssize_t noptargs = nargs + (kwnames ? PyTuple_GET_SIZE(kwnames) : 0) - 0; int follow_symlinks = 1; int _return_value; - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser, - &follow_symlinks)) { + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 0, 0, 0, argsbuf); + if (!args) { goto exit; } + if (!noptargs) { + goto skip_optional_kwonly; + } + follow_symlinks = PyObject_IsTrue(args[0]); + if (follow_symlinks < 0) { + goto exit; + } +skip_optional_kwonly: _return_value = os_DirEntry_is_dir_impl(self, defining_class, follow_symlinks); if ((_return_value == -1) && PyErr_Occurred()) { goto exit; @@ -8393,14 +8411,24 @@ os_DirEntry_is_file(DirEntry *self, PyTypeObject *defining_class, PyObject *cons { PyObject *return_value = NULL; static const char * const _keywords[] = {"follow_symlinks", NULL}; - static _PyArg_Parser _parser = {"|$p:is_file", _keywords, 0}; + static _PyArg_Parser _parser = {NULL, _keywords, "is_file", 0}; + PyObject *argsbuf[1]; + Py_ssize_t noptargs = nargs + (kwnames ? PyTuple_GET_SIZE(kwnames) : 0) - 0; int follow_symlinks = 1; int _return_value; - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser, - &follow_symlinks)) { + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 0, 0, 0, argsbuf); + if (!args) { goto exit; } + if (!noptargs) { + goto skip_optional_kwonly; + } + follow_symlinks = PyObject_IsTrue(args[0]); + if (follow_symlinks < 0) { + goto exit; + } +skip_optional_kwonly: _return_value = os_DirEntry_is_file_impl(self, defining_class, follow_symlinks); if ((_return_value == -1) && PyErr_Occurred()) { goto exit; @@ -9303,4 +9331,4 @@ os_waitstatus_to_exitcode(PyObject *module, PyObject *const *args, Py_ssize_t na #ifndef OS_WAITSTATUS_TO_EXITCODE_METHODDEF #define OS_WAITSTATUS_TO_EXITCODE_METHODDEF #endif /* !defined(OS_WAITSTATUS_TO_EXITCODE_METHODDEF) */ -/*[clinic end generated code: output=d95ba7b0b9c52685 input=a9049054013a1b77]*/ +/*[clinic end generated code: output=c8c5b148b96068b4 input=a9049054013a1b77]*/ diff --git a/Modules/clinic/pyexpat.c.h b/Modules/clinic/pyexpat.c.h index 7c56d6a8b2591..bee2ee66950a5 100644 --- a/Modules/clinic/pyexpat.c.h +++ b/Modules/clinic/pyexpat.c.h @@ -22,14 +22,24 @@ pyexpat_xmlparser_Parse(xmlparseobject *self, PyTypeObject *cls, PyObject *const { PyObject *return_value = NULL; static const char * const _keywords[] = {"", "", NULL}; - static _PyArg_Parser _parser = {"O|i:Parse", _keywords, 0}; + static _PyArg_Parser _parser = {NULL, _keywords, "Parse", 0}; + PyObject *argsbuf[2]; PyObject *data; int isfinal = 0; - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser, - &data, &isfinal)) { + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 1, 2, 0, argsbuf); + if (!args) { + goto exit; + } + data = args[0]; + if (nargs < 2) { + goto skip_optional_posonly; + } + isfinal = _PyLong_AsInt(args[1]); + if (isfinal == -1 && PyErr_Occurred()) { goto exit; } +skip_optional_posonly: return_value = pyexpat_xmlparser_Parse_impl(self, cls, data, isfinal); exit: @@ -54,13 +64,15 @@ pyexpat_xmlparser_ParseFile(xmlparseobject *self, PyTypeObject *cls, PyObject *c { PyObject *return_value = NULL; static const char * const _keywords[] = {"", NULL}; - static _PyArg_Parser _parser = {"O:ParseFile", _keywords, 0}; + static _PyArg_Parser _parser = {NULL, _keywords, "ParseFile", 0}; + PyObject *argsbuf[1]; PyObject *file; - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser, - &file)) { + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 1, 1, 0, argsbuf); + if (!args) { goto exit; } + file = args[0]; return_value = pyexpat_xmlparser_ParseFile_impl(self, cls, file); exit: @@ -164,14 +176,50 @@ pyexpat_xmlparser_ExternalEntityParserCreate(xmlparseobject *self, PyTypeObject { PyObject *return_value = NULL; static const char * const _keywords[] = {"", "", NULL}; - static _PyArg_Parser _parser = {"z|s:ExternalEntityParserCreate", _keywords, 0}; + static _PyArg_Parser _parser = {NULL, _keywords, "ExternalEntityParserCreate", 0}; + PyObject *argsbuf[2]; const char *context; const char *encoding = NULL; - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser, - &context, &encoding)) { + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 1, 2, 0, argsbuf); + if (!args) { + goto exit; + } + if (args[0] == Py_None) { + context = NULL; + } + else if (PyUnicode_Check(args[0])) { + Py_ssize_t context_length; + context = PyUnicode_AsUTF8AndSize(args[0], &context_length); + if (context == NULL) { + goto exit; + } + if (strlen(context) != (size_t)context_length) { + PyErr_SetString(PyExc_ValueError, "embedded null character"); + goto exit; + } + } + else { + _PyArg_BadArgument("ExternalEntityParserCreate", "argument 1", "str or None", args[0]); + goto exit; + } + if (nargs < 2) { + goto skip_optional_posonly; + } + if (!PyUnicode_Check(args[1])) { + _PyArg_BadArgument("ExternalEntityParserCreate", "argument 2", "str", args[1]); goto exit; } + Py_ssize_t encoding_length; + encoding = PyUnicode_AsUTF8AndSize(args[1], &encoding_length); + if (encoding == NULL) { + goto exit; + } + if (strlen(encoding) != (size_t)encoding_length) { + PyErr_SetString(PyExc_ValueError, "embedded null character"); + goto exit; + } +skip_optional_posonly: return_value = pyexpat_xmlparser_ExternalEntityParserCreate_impl(self, cls, context, encoding); exit: @@ -235,13 +283,22 @@ pyexpat_xmlparser_UseForeignDTD(xmlparseobject *self, PyTypeObject *cls, PyObjec { PyObject *return_value = NULL; static const char * const _keywords[] = {"", NULL}; - static _PyArg_Parser _parser = {"|p:UseForeignDTD", _keywords, 0}; + static _PyArg_Parser _parser = {NULL, _keywords, "UseForeignDTD", 0}; + PyObject *argsbuf[1]; int flag = 1; - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser, - &flag)) { + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 0, 1, 0, argsbuf); + if (!args) { + goto exit; + } + if (nargs < 1) { + goto skip_optional_posonly; + } + flag = PyObject_IsTrue(args[0]); + if (flag < 0) { goto exit; } +skip_optional_posonly: return_value = pyexpat_xmlparser_UseForeignDTD_impl(self, cls, flag); exit: @@ -368,4 +425,4 @@ pyexpat_ErrorString(PyObject *module, PyObject *arg) #ifndef PYEXPAT_XMLPARSER_USEFOREIGNDTD_METHODDEF #define PYEXPAT_XMLPARSER_USEFOREIGNDTD_METHODDEF #endif /* !defined(PYEXPAT_XMLPARSER_USEFOREIGNDTD_METHODDEF) */ -/*[clinic end generated code: output=612b9d6a17a679a7 input=a9049054013a1b77]*/ +/*[clinic end generated code: output=5d60049d385d5d56 input=a9049054013a1b77]*/ diff --git a/Modules/clinic/sha1module.c.h b/Modules/clinic/sha1module.c.h index 3a3ab58c1233c..d9a2e150f6d41 100644 --- a/Modules/clinic/sha1module.c.h +++ b/Modules/clinic/sha1module.c.h @@ -17,18 +17,11 @@ SHA1Type_copy_impl(SHA1object *self, PyTypeObject *cls); static PyObject * SHA1Type_copy(SHA1object *self, PyTypeObject *cls, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) { - PyObject *return_value = NULL; - static const char * const _keywords[] = { NULL}; - static _PyArg_Parser _parser = {":copy", _keywords, 0}; - - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser - )) { - goto exit; + if (nargs) { + PyErr_SetString(PyExc_TypeError, "copy() takes no arguments"); + return NULL; } - return_value = SHA1Type_copy_impl(self, cls); - -exit: - return return_value; + return SHA1Type_copy_impl(self, cls); } PyDoc_STRVAR(SHA1Type_digest__doc__, @@ -126,4 +119,4 @@ _sha1_sha1(PyObject *module, PyObject *const *args, Py_ssize_t nargs, PyObject * exit: return return_value; } -/*[clinic end generated code: output=abf1ab2545cea5a2 input=a9049054013a1b77]*/ +/*[clinic end generated code: output=93ced3c8f8fa4f21 input=a9049054013a1b77]*/ diff --git a/Modules/clinic/sha256module.c.h b/Modules/clinic/sha256module.c.h index 89205c4f14f4e..619fab1a6fe2f 100644 --- a/Modules/clinic/sha256module.c.h +++ b/Modules/clinic/sha256module.c.h @@ -17,18 +17,11 @@ SHA256Type_copy_impl(SHAobject *self, PyTypeObject *cls); static PyObject * SHA256Type_copy(SHAobject *self, PyTypeObject *cls, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) { - PyObject *return_value = NULL; - static const char * const _keywords[] = { NULL}; - static _PyArg_Parser _parser = {":copy", _keywords, 0}; - - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser - )) { - goto exit; + if (nargs) { + PyErr_SetString(PyExc_TypeError, "copy() takes no arguments"); + return NULL; } - return_value = SHA256Type_copy_impl(self, cls); - -exit: - return return_value; + return SHA256Type_copy_impl(self, cls); } PyDoc_STRVAR(SHA256Type_digest__doc__, @@ -177,4 +170,4 @@ _sha256_sha224(PyObject *module, PyObject *const *args, Py_ssize_t nargs, PyObje exit: return return_value; } -/*[clinic end generated code: output=b7283f75c9d08f30 input=a9049054013a1b77]*/ +/*[clinic end generated code: output=4f9fe3ca546b0c58 input=a9049054013a1b77]*/ diff --git a/Modules/clinic/sha512module.c.h b/Modules/clinic/sha512module.c.h index f1192d74f9a1a..fada35eff2261 100644 --- a/Modules/clinic/sha512module.c.h +++ b/Modules/clinic/sha512module.c.h @@ -17,18 +17,11 @@ SHA512Type_copy_impl(SHAobject *self, PyTypeObject *cls); static PyObject * SHA512Type_copy(SHAobject *self, PyTypeObject *cls, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) { - PyObject *return_value = NULL; - static const char * const _keywords[] = { NULL}; - static _PyArg_Parser _parser = {":copy", _keywords, 0}; - - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser - )) { - goto exit; + if (nargs) { + PyErr_SetString(PyExc_TypeError, "copy() takes no arguments"); + return NULL; } - return_value = SHA512Type_copy_impl(self, cls); - -exit: - return return_value; + return SHA512Type_copy_impl(self, cls); } PyDoc_STRVAR(SHA512Type_digest__doc__, @@ -177,4 +170,4 @@ _sha512_sha384(PyObject *module, PyObject *const *args, Py_ssize_t nargs, PyObje exit: return return_value; } -/*[clinic end generated code: output=9ff9f11937fabf35 input=a9049054013a1b77]*/ +/*[clinic end generated code: output=26d2fe27b9673ac2 input=a9049054013a1b77]*/ diff --git a/Modules/clinic/zlibmodule.c.h b/Modules/clinic/zlibmodule.c.h index 71136d31db0e0..deadf872d0cf7 100644 --- a/Modules/clinic/zlibmodule.c.h +++ b/Modules/clinic/zlibmodule.c.h @@ -352,11 +352,19 @@ zlib_Compress_compress(compobject *self, PyTypeObject *cls, PyObject *const *arg { PyObject *return_value = NULL; static const char * const _keywords[] = {"", NULL}; - static _PyArg_Parser _parser = {"y*:compress", _keywords, 0}; + static _PyArg_Parser _parser = {NULL, _keywords, "compress", 0}; + PyObject *argsbuf[1]; Py_buffer data = {NULL, NULL}; - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser, - &data)) { + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 1, 1, 0, argsbuf); + if (!args) { + goto exit; + } + if (PyObject_GetBuffer(args[0], &data, PyBUF_SIMPLE) != 0) { + goto exit; + } + if (!PyBuffer_IsContiguous(&data, 'C')) { + _PyArg_BadArgument("compress", "argument 1", "contiguous buffer", args[0]); goto exit; } return_value = zlib_Compress_compress_impl(self, cls, &data); @@ -399,14 +407,39 @@ zlib_Decompress_decompress(compobject *self, PyTypeObject *cls, PyObject *const { PyObject *return_value = NULL; static const char * const _keywords[] = {"", "max_length", NULL}; - static _PyArg_Parser _parser = {"y*|n:decompress", _keywords, 0}; + static _PyArg_Parser _parser = {NULL, _keywords, "decompress", 0}; + PyObject *argsbuf[2]; + Py_ssize_t noptargs = nargs + (kwnames ? PyTuple_GET_SIZE(kwnames) : 0) - 1; Py_buffer data = {NULL, NULL}; Py_ssize_t max_length = 0; - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser, - &data, &max_length)) { + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 1, 2, 0, argsbuf); + if (!args) { + goto exit; + } + if (PyObject_GetBuffer(args[0], &data, PyBUF_SIMPLE) != 0) { + goto exit; + } + if (!PyBuffer_IsContiguous(&data, 'C')) { + _PyArg_BadArgument("decompress", "argument 1", "contiguous buffer", args[0]); goto exit; } + if (!noptargs) { + goto skip_optional_pos; + } + { + Py_ssize_t ival = -1; + PyObject *iobj = _PyNumber_Index(args[1]); + if (iobj != NULL) { + ival = PyLong_AsSsize_t(iobj); + Py_DECREF(iobj); + } + if (ival == -1 && PyErr_Occurred()) { + goto exit; + } + max_length = ival; + } +skip_optional_pos: return_value = zlib_Decompress_decompress_impl(self, cls, &data, max_length); exit: @@ -441,13 +474,22 @@ zlib_Compress_flush(compobject *self, PyTypeObject *cls, PyObject *const *args, { PyObject *return_value = NULL; static const char * const _keywords[] = {"", NULL}; - static _PyArg_Parser _parser = {"|i:flush", _keywords, 0}; + static _PyArg_Parser _parser = {NULL, _keywords, "flush", 0}; + PyObject *argsbuf[1]; int mode = Z_FINISH; - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser, - &mode)) { + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 0, 1, 0, argsbuf); + if (!args) { + goto exit; + } + if (nargs < 1) { + goto skip_optional_posonly; + } + mode = _PyLong_AsInt(args[0]); + if (mode == -1 && PyErr_Occurred()) { goto exit; } +skip_optional_posonly: return_value = zlib_Compress_flush_impl(self, cls, mode); exit: @@ -471,18 +513,11 @@ zlib_Compress_copy_impl(compobject *self, PyTypeObject *cls); static PyObject * zlib_Compress_copy(compobject *self, PyTypeObject *cls, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) { - PyObject *return_value = NULL; - static const char * const _keywords[] = { NULL}; - static _PyArg_Parser _parser = {":copy", _keywords, 0}; - - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser - )) { - goto exit; + if (nargs) { + PyErr_SetString(PyExc_TypeError, "copy() takes no arguments"); + return NULL; } - return_value = zlib_Compress_copy_impl(self, cls); - -exit: - return return_value; + return zlib_Compress_copy_impl(self, cls); } #endif /* defined(HAVE_ZLIB_COPY) */ @@ -503,18 +538,11 @@ zlib_Compress___copy___impl(compobject *self, PyTypeObject *cls); static PyObject * zlib_Compress___copy__(compobject *self, PyTypeObject *cls, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) { - PyObject *return_value = NULL; - static const char * const _keywords[] = { NULL}; - static _PyArg_Parser _parser = {":__copy__", _keywords, 0}; - - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser - )) { - goto exit; + if (nargs) { + PyErr_SetString(PyExc_TypeError, "__copy__() takes no arguments"); + return NULL; } - return_value = zlib_Compress___copy___impl(self, cls); - -exit: - return return_value; + return zlib_Compress___copy___impl(self, cls); } #endif /* defined(HAVE_ZLIB_COPY) */ @@ -538,13 +566,15 @@ zlib_Compress___deepcopy__(compobject *self, PyTypeObject *cls, PyObject *const { PyObject *return_value = NULL; static const char * const _keywords[] = {"", NULL}; - static _PyArg_Parser _parser = {"O:__deepcopy__", _keywords, 0}; + static _PyArg_Parser _parser = {NULL, _keywords, "__deepcopy__", 0}; + PyObject *argsbuf[1]; PyObject *memo; - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser, - &memo)) { + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 1, 1, 0, argsbuf); + if (!args) { goto exit; } + memo = args[0]; return_value = zlib_Compress___deepcopy___impl(self, cls, memo); exit: @@ -570,18 +600,11 @@ zlib_Decompress_copy_impl(compobject *self, PyTypeObject *cls); static PyObject * zlib_Decompress_copy(compobject *self, PyTypeObject *cls, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) { - PyObject *return_value = NULL; - static const char * const _keywords[] = { NULL}; - static _PyArg_Parser _parser = {":copy", _keywords, 0}; - - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser - )) { - goto exit; + if (nargs) { + PyErr_SetString(PyExc_TypeError, "copy() takes no arguments"); + return NULL; } - return_value = zlib_Decompress_copy_impl(self, cls); - -exit: - return return_value; + return zlib_Decompress_copy_impl(self, cls); } #endif /* defined(HAVE_ZLIB_COPY) */ @@ -602,18 +625,11 @@ zlib_Decompress___copy___impl(compobject *self, PyTypeObject *cls); static PyObject * zlib_Decompress___copy__(compobject *self, PyTypeObject *cls, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) { - PyObject *return_value = NULL; - static const char * const _keywords[] = { NULL}; - static _PyArg_Parser _parser = {":__copy__", _keywords, 0}; - - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser - )) { - goto exit; + if (nargs) { + PyErr_SetString(PyExc_TypeError, "__copy__() takes no arguments"); + return NULL; } - return_value = zlib_Decompress___copy___impl(self, cls); - -exit: - return return_value; + return zlib_Decompress___copy___impl(self, cls); } #endif /* defined(HAVE_ZLIB_COPY) */ @@ -637,13 +653,15 @@ zlib_Decompress___deepcopy__(compobject *self, PyTypeObject *cls, PyObject *cons { PyObject *return_value = NULL; static const char * const _keywords[] = {"", NULL}; - static _PyArg_Parser _parser = {"O:__deepcopy__", _keywords, 0}; + static _PyArg_Parser _parser = {NULL, _keywords, "__deepcopy__", 0}; + PyObject *argsbuf[1]; PyObject *memo; - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser, - &memo)) { + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 1, 1, 0, argsbuf); + if (!args) { goto exit; } + memo = args[0]; return_value = zlib_Decompress___deepcopy___impl(self, cls, memo); exit: @@ -673,13 +691,30 @@ zlib_Decompress_flush(compobject *self, PyTypeObject *cls, PyObject *const *args { PyObject *return_value = NULL; static const char * const _keywords[] = {"", NULL}; - static _PyArg_Parser _parser = {"|n:flush", _keywords, 0}; + static _PyArg_Parser _parser = {NULL, _keywords, "flush", 0}; + PyObject *argsbuf[1]; Py_ssize_t length = DEF_BUF_SIZE; - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser, - &length)) { + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 0, 1, 0, argsbuf); + if (!args) { goto exit; } + if (nargs < 1) { + goto skip_optional_posonly; + } + { + Py_ssize_t ival = -1; + PyObject *iobj = _PyNumber_Index(args[0]); + if (iobj != NULL) { + ival = PyLong_AsSsize_t(iobj); + Py_DECREF(iobj); + } + if (ival == -1 && PyErr_Occurred()) { + goto exit; + } + length = ival; + } +skip_optional_posonly: return_value = zlib_Decompress_flush_impl(self, cls, length); exit: @@ -820,4 +855,4 @@ zlib_crc32(PyObject *module, PyObject *const *args, Py_ssize_t nargs) #ifndef ZLIB_DECOMPRESS___DEEPCOPY___METHODDEF #define ZLIB_DECOMPRESS___DEEPCOPY___METHODDEF #endif /* !defined(ZLIB_DECOMPRESS___DEEPCOPY___METHODDEF) */ -/*[clinic end generated code: output=f009d194565416d1 input=a9049054013a1b77]*/ +/*[clinic end generated code: output=bb1e27d8b2198a86 input=a9049054013a1b77]*/ diff --git a/Tools/clinic/clinic.py b/Tools/clinic/clinic.py index 14252b2514ea8..f660474da94a0 100755 --- a/Tools/clinic/clinic.py +++ b/Tools/clinic/clinic.py @@ -639,6 +639,10 @@ def output_templates(self, f): assert parameters assert isinstance(parameters[0].converter, self_converter) del parameters[0] + requires_defining_class = False + if parameters and isinstance(parameters[0].converter, defining_class_converter): + requires_defining_class = True + del parameters[0] converters = [p.converter for p in parameters] has_option_groups = parameters and (parameters[0].group or parameters[-1].group) @@ -667,10 +671,6 @@ def output_templates(self, f): if not p.is_optional(): min_pos = i - requires_defining_class = any( - isinstance(p.converter, defining_class_converter) - for p in parameters) - meth_o = (len(parameters) == 1 and parameters[0].is_positional_only() and not converters[0].is_optional() and @@ -773,24 +773,40 @@ def parser_body(prototype, *fields, declarations=''): return linear_format(output(), parser_declarations=declarations) if not parameters: - # no parameters, METH_NOARGS + if not requires_defining_class: + # no parameters, METH_NOARGS + flags = "METH_NOARGS" - flags = "METH_NOARGS" + parser_prototype = normalize_snippet(""" + static PyObject * + {c_basename}({self_type}{self_name}, PyObject *Py_UNUSED(ignored)) + """) + parser_code = [] - parser_prototype = normalize_snippet(""" - static PyObject * - {c_basename}({self_type}{self_name}, PyObject *Py_UNUSED(ignored)) - """) - parser_definition = parser_prototype + else: + assert not new_or_init - if default_return_converter: - parser_definition = parser_prototype + '\n' + normalize_snippet(""" - {{ - return {c_basename}_impl({impl_arguments}); + flags = "METH_METHOD|METH_FASTCALL|METH_KEYWORDS" + + parser_prototype = parser_prototype_def_class + return_error = ('return NULL;' if default_return_converter + else 'goto exit;') + parser_code = [normalize_snippet(""" + if (nargs) {{ + PyErr_SetString(PyExc_TypeError, "{name}() takes no arguments"); + %s }} - """) + """ % return_error, indent=4)] + + if default_return_converter: + parser_definition = '\n'.join([ + parser_prototype, + '{{', + *parser_code, + ' return {c_basename}_impl({impl_arguments});', + '}}']) else: - parser_definition = parser_body(parser_prototype) + parser_definition = parser_body(parser_prototype, *parser_code) elif meth_o: flags = "METH_O" @@ -993,6 +1009,9 @@ def parser_body(prototype, *fields, declarations=''): add_label = None for i, p in enumerate(parameters): + if isinstance(p.converter, defining_class_converter): + raise ValueError("defining_class should be the first " + "parameter (after self)") displayname = p.get_displayname(i+1) parsearg = p.converter.parse_arg(argname_fmt % i, displayname) if parsearg is None: From webhook-mailer at python.org Sat Apr 30 06:16:35 2022 From: webhook-mailer at python.org (serhiy-storchaka) Date: Sat, 30 Apr 2022 10:16:35 -0000 Subject: [Python-checkins] gh-81548: Deprecate octal escape sequences with value larger than 0o377 (GH-91668) Message-ID: https://github.com/python/cpython/commit/3483299a24e41a7f2e958369cb3573d7c2253e33 commit: 3483299a24e41a7f2e958369cb3573d7c2253e33 branch: main author: Serhiy Storchaka committer: serhiy-storchaka date: 2022-04-30T13:16:27+03:00 summary: gh-81548: Deprecate octal escape sequences with value larger than 0o377 (GH-91668) files: A Misc/NEWS.d/next/Core and Builtins/2022-04-18-20-25-01.gh-issue-81548.n3VYgp.rst M Doc/reference/lexical_analysis.rst M Doc/whatsnew/3.11.rst M Lib/test/test_codecs.py M Lib/test/test_string_literals.py M Objects/bytesobject.c M Objects/unicodeobject.c M Parser/string_parser.c diff --git a/Doc/reference/lexical_analysis.rst b/Doc/reference/lexical_analysis.rst index dba1a9dd2e15a..0e64a056a6eb2 100644 --- a/Doc/reference/lexical_analysis.rst +++ b/Doc/reference/lexical_analysis.rst @@ -596,6 +596,11 @@ Notes: (1) As in Standard C, up to three octal digits are accepted. + .. versionchanged:: 3.11 + Octal escapes with value larger than ``0o377`` produce a :exc:`DeprecationWarning`. + In a future Python version they will be a :exc:`SyntaxWarning` and + eventually a :exc:`SyntaxError`. + (2) Unlike in Standard C, exactly two hex digits are required. diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index 1a692f2fe7f60..59c8dd6e44732 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -1055,6 +1055,12 @@ CPython bytecode changes Deprecated ========== +* Octal escapes with value larger than ``0o377`` now produce + a :exc:`DeprecationWarning`. + In a future Python version they will be a :exc:`SyntaxWarning` and + eventually a :exc:`SyntaxError`. + (Contributed by Serhiy Storchaka in :issue:`81548`.) + * The :mod:`lib2to3` package and ``2to3`` tool are now deprecated and may not be able to parse Python 3.10 or newer. See the :pep:`617` (New PEG parser for CPython). (Contributed by Victor Stinner in :issue:`40360`.) diff --git a/Lib/test/test_codecs.py b/Lib/test/test_codecs.py index 5853e08819715..42c600dcb00de 100644 --- a/Lib/test/test_codecs.py +++ b/Lib/test/test_codecs.py @@ -1193,7 +1193,6 @@ def test_escape(self): check(br"[\418]", b"[!8]") check(br"[\101]", b"[A]") check(br"[\1010]", b"[A0]") - check(br"[\501]", b"[A]") check(br"[\x41]", b"[A]") check(br"[\x410]", b"[A0]") for i in range(97, 123): @@ -1209,6 +1208,9 @@ def test_escape(self): check(br"\9", b"\\9") with self.assertWarns(DeprecationWarning): check(b"\\\xfa", b"\\\xfa") + for i in range(0o400, 0o1000): + with self.assertWarns(DeprecationWarning): + check(rb'\%o' % i, bytes([i & 0o377])) def test_errors(self): decode = codecs.escape_decode @@ -2435,6 +2437,9 @@ def test_escape_decode(self): check(br"\9", "\\9") with self.assertWarns(DeprecationWarning): check(b"\\\xfa", "\\\xfa") + for i in range(0o400, 0o1000): + with self.assertWarns(DeprecationWarning): + check(rb'\%o' % i, chr(i)) def test_decode_errors(self): decode = codecs.unicode_escape_decode diff --git a/Lib/test/test_string_literals.py b/Lib/test/test_string_literals.py index 7231970acf19d..3a3830bcb6e96 100644 --- a/Lib/test/test_string_literals.py +++ b/Lib/test/test_string_literals.py @@ -116,6 +116,7 @@ def test_eval_str_invalid_escape(self): warnings.simplefilter('always', category=DeprecationWarning) eval("'''\n\\z'''") self.assertEqual(len(w), 1) + self.assertEqual(str(w[0].message), r"invalid escape sequence '\z'") self.assertEqual(w[0].filename, '') self.assertEqual(w[0].lineno, 1) @@ -125,6 +126,32 @@ def test_eval_str_invalid_escape(self): eval("'''\n\\z'''") exc = cm.exception self.assertEqual(w, []) + self.assertEqual(exc.msg, r"invalid escape sequence '\z'") + self.assertEqual(exc.filename, '') + self.assertEqual(exc.lineno, 1) + self.assertEqual(exc.offset, 1) + + def test_eval_str_invalid_octal_escape(self): + for i in range(0o400, 0o1000): + with self.assertWarns(DeprecationWarning): + self.assertEqual(eval(r"'\%o'" % i), chr(i)) + + with warnings.catch_warnings(record=True) as w: + warnings.simplefilter('always', category=DeprecationWarning) + eval("'''\n\\407'''") + self.assertEqual(len(w), 1) + self.assertEqual(str(w[0].message), + r"invalid octal escape sequence '\407'") + self.assertEqual(w[0].filename, '') + self.assertEqual(w[0].lineno, 1) + + with warnings.catch_warnings(record=True) as w: + warnings.simplefilter('error', category=DeprecationWarning) + with self.assertRaises(SyntaxError) as cm: + eval("'''\n\\407'''") + exc = cm.exception + self.assertEqual(w, []) + self.assertEqual(exc.msg, r"invalid octal escape sequence '\407'") self.assertEqual(exc.filename, '') self.assertEqual(exc.lineno, 1) self.assertEqual(exc.offset, 1) @@ -166,6 +193,7 @@ def test_eval_bytes_invalid_escape(self): warnings.simplefilter('always', category=DeprecationWarning) eval("b'''\n\\z'''") self.assertEqual(len(w), 1) + self.assertEqual(str(w[0].message), r"invalid escape sequence '\z'") self.assertEqual(w[0].filename, '') self.assertEqual(w[0].lineno, 1) @@ -175,6 +203,31 @@ def test_eval_bytes_invalid_escape(self): eval("b'''\n\\z'''") exc = cm.exception self.assertEqual(w, []) + self.assertEqual(exc.msg, r"invalid escape sequence '\z'") + self.assertEqual(exc.filename, '') + self.assertEqual(exc.lineno, 1) + + def test_eval_bytes_invalid_octal_escape(self): + for i in range(0o400, 0o1000): + with self.assertWarns(DeprecationWarning): + self.assertEqual(eval(r"b'\%o'" % i), bytes([i & 0o377])) + + with warnings.catch_warnings(record=True) as w: + warnings.simplefilter('always', category=DeprecationWarning) + eval("b'''\n\\407'''") + self.assertEqual(len(w), 1) + self.assertEqual(str(w[0].message), + r"invalid octal escape sequence '\407'") + self.assertEqual(w[0].filename, '') + self.assertEqual(w[0].lineno, 1) + + with warnings.catch_warnings(record=True) as w: + warnings.simplefilter('error', category=DeprecationWarning) + with self.assertRaises(SyntaxError) as cm: + eval("b'''\n\\407'''") + exc = cm.exception + self.assertEqual(w, []) + self.assertEqual(exc.msg, r"invalid octal escape sequence '\407'") self.assertEqual(exc.filename, '') self.assertEqual(exc.lineno, 1) diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-04-18-20-25-01.gh-issue-81548.n3VYgp.rst b/Misc/NEWS.d/next/Core and Builtins/2022-04-18-20-25-01.gh-issue-81548.n3VYgp.rst new file mode 100644 index 0000000000000..56b1fd63b71c0 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2022-04-18-20-25-01.gh-issue-81548.n3VYgp.rst @@ -0,0 +1,3 @@ +Octal escapes with value larger than ``0o377`` now produce a +:exc:`DeprecationWarning`. In a future Python version they will be a +:exc:`SyntaxWarning` and eventually a :exc:`SyntaxError`. diff --git a/Objects/bytesobject.c b/Objects/bytesobject.c index 510a836e6b152..b5066d0986c7b 100644 --- a/Objects/bytesobject.c +++ b/Objects/bytesobject.c @@ -1113,6 +1113,12 @@ PyObject *_PyBytes_DecodeEscape(const char *s, if (s < end && '0' <= *s && *s <= '7') c = (c<<3) + *s++ - '0'; } + if (c > 0377) { + if (*first_invalid_escape == NULL) { + *first_invalid_escape = s-3; /* Back up 3 chars, since we've + already incremented s. */ + } + } *p++ = c; break; case 'x': @@ -1179,11 +1185,24 @@ PyObject *PyBytes_DecodeEscape(const char *s, if (result == NULL) return NULL; if (first_invalid_escape != NULL) { - if (PyErr_WarnFormat(PyExc_DeprecationWarning, 1, - "invalid escape sequence '\\%c'", - (unsigned char)*first_invalid_escape) < 0) { - Py_DECREF(result); - return NULL; + unsigned char c = *first_invalid_escape; + if ('4' <= c && c <= '7') { + if (PyErr_WarnFormat(PyExc_DeprecationWarning, 1, + "invalid octal escape sequence '\\%.3s'", + first_invalid_escape) < 0) + { + Py_DECREF(result); + return NULL; + } + } + else { + if (PyErr_WarnFormat(PyExc_DeprecationWarning, 1, + "invalid escape sequence '\\%c'", + c) < 0) + { + Py_DECREF(result); + return NULL; + } } } return result; diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c index 7768f66c95655..493307556529b 100644 --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -6404,6 +6404,12 @@ _PyUnicode_DecodeUnicodeEscapeInternal(const char *s, ch = (ch<<3) + *s++ - '0'; } } + if (ch > 0377) { + if (*first_invalid_escape == NULL) { + *first_invalid_escape = s-3; /* Back up 3 chars, since we've + already incremented s. */ + } + } WRITE_CHAR(ch); continue; @@ -6554,11 +6560,24 @@ _PyUnicode_DecodeUnicodeEscapeStateful(const char *s, if (result == NULL) return NULL; if (first_invalid_escape != NULL) { - if (PyErr_WarnFormat(PyExc_DeprecationWarning, 1, - "invalid escape sequence '\\%c'", - (unsigned char)*first_invalid_escape) < 0) { - Py_DECREF(result); - return NULL; + unsigned char c = *first_invalid_escape; + if ('4' <= c && c <= '7') { + if (PyErr_WarnFormat(PyExc_DeprecationWarning, 1, + "invalid octal escape sequence '\\%.3s'", + first_invalid_escape) < 0) + { + Py_DECREF(result); + return NULL; + } + } + else { + if (PyErr_WarnFormat(PyExc_DeprecationWarning, 1, + "invalid escape sequence '\\%c'", + c) < 0) + { + Py_DECREF(result); + return NULL; + } } } return result; diff --git a/Parser/string_parser.c b/Parser/string_parser.c index 65ddd46874b20..9c12d8ca101d0 100644 --- a/Parser/string_parser.c +++ b/Parser/string_parser.c @@ -9,10 +9,15 @@ //// STRING HANDLING FUNCTIONS //// static int -warn_invalid_escape_sequence(Parser *p, unsigned char first_invalid_escape_char, Token *t) +warn_invalid_escape_sequence(Parser *p, const char *first_invalid_escape, Token *t) { + unsigned char c = *first_invalid_escape; + int octal = ('4' <= c && c <= '7'); PyObject *msg = - PyUnicode_FromFormat("invalid escape sequence '\\%c'", first_invalid_escape_char); + octal + ? PyUnicode_FromFormat("invalid octal escape sequence '\\%.3s'", + first_invalid_escape) + : PyUnicode_FromFormat("invalid escape sequence '\\%c'", c); if (msg == NULL) { return -1; } @@ -27,7 +32,13 @@ warn_invalid_escape_sequence(Parser *p, unsigned char first_invalid_escape_char, since _PyPegen_raise_error uses p->tokens[p->fill - 1] for the error location, if p->known_err_token is not set. */ p->known_err_token = t; - RAISE_SYNTAX_ERROR("invalid escape sequence '\\%c'", first_invalid_escape_char); + if (octal) { + RAISE_SYNTAX_ERROR("invalid octal escape sequence '\\%.3s'", + first_invalid_escape); + } + else { + RAISE_SYNTAX_ERROR("invalid escape sequence '\\%c'", c); + } } Py_DECREF(msg); return -1; @@ -118,7 +129,7 @@ decode_unicode_with_escapes(Parser *parser, const char *s, size_t len, Token *t) v = _PyUnicode_DecodeUnicodeEscapeInternal(s, len, NULL, NULL, &first_invalid_escape); if (v != NULL && first_invalid_escape != NULL) { - if (warn_invalid_escape_sequence(parser, *first_invalid_escape, t) < 0) { + if (warn_invalid_escape_sequence(parser, first_invalid_escape, t) < 0) { /* We have not decref u before because first_invalid_escape points inside u. */ Py_XDECREF(u); @@ -140,7 +151,7 @@ decode_bytes_with_escapes(Parser *p, const char *s, Py_ssize_t len, Token *t) } if (first_invalid_escape != NULL) { - if (warn_invalid_escape_sequence(p, *first_invalid_escape, t) < 0) { + if (warn_invalid_escape_sequence(p, first_invalid_escape, t) < 0) { Py_DECREF(result); return NULL; } @@ -357,7 +368,7 @@ fstring_compile_expr(Parser *p, const char *expr_start, const char *expr_end, break; } } - + if (s == expr_end) { if (*expr_end == '!' || *expr_end == ':' || *expr_end == '=') { RAISE_SYNTAX_ERROR("f-string: expression required before '%c'", *expr_end); @@ -465,7 +476,7 @@ fstring_find_literal(Parser *p, const char **str, const char *end, int raw, decode_unicode_with_escapes(). */ continue; } - if (ch == '{' && warn_invalid_escape_sequence(p, ch, t) < 0) { + if (ch == '{' && warn_invalid_escape_sequence(p, s-1, t) < 0) { return -1; } } From webhook-mailer at python.org Sat Apr 30 06:17:28 2022 From: webhook-mailer at python.org (serhiy-storchaka) Date: Sat, 30 Apr 2022 10:17:28 -0000 Subject: [Python-checkins] bpo-43323: Fix UnicodeEncodeError in the email module (GH-32137) Message-ID: https://github.com/python/cpython/commit/e91dee87edcf6dee5dd78053004d76e5f05456d4 commit: e91dee87edcf6dee5dd78053004d76e5f05456d4 branch: main author: Serhiy Storchaka committer: serhiy-storchaka date: 2022-04-30T13:17:23+03:00 summary: bpo-43323: Fix UnicodeEncodeError in the email module (GH-32137) It was raised if the charset itself contains characters not encodable in UTF-8 (in particular \udcxx characters representing non-decodable bytes in the source). files: A Misc/NEWS.d/next/Library/2022-03-27-12-40-16.bpo-43323.9mFPuI.rst M Lib/email/_encoded_words.py M Lib/email/_header_value_parser.py M Lib/test/test_email/test__encoded_words.py M Lib/test/test_email/test_email.py M Lib/test/test_email/test_headerregistry.py diff --git a/Lib/email/_encoded_words.py b/Lib/email/_encoded_words.py index 295ae7eb21237..6795a606de037 100644 --- a/Lib/email/_encoded_words.py +++ b/Lib/email/_encoded_words.py @@ -179,15 +179,15 @@ def decode(ew): # Turn the CTE decoded bytes into unicode. try: string = bstring.decode(charset) - except UnicodeError: + except UnicodeDecodeError: defects.append(errors.UndecodableBytesDefect("Encoded word " - "contains bytes not decodable using {} charset".format(charset))) + f"contains bytes not decodable using {charset!r} charset")) string = bstring.decode(charset, 'surrogateescape') - except LookupError: + except (LookupError, UnicodeEncodeError): string = bstring.decode('ascii', 'surrogateescape') if charset.lower() != 'unknown-8bit': - defects.append(errors.CharsetError("Unknown charset {} " - "in encoded word; decoded as unknown bytes".format(charset))) + defects.append(errors.CharsetError(f"Unknown charset {charset!r} " + f"in encoded word; decoded as unknown bytes")) return string, charset, lang, defects diff --git a/Lib/email/_header_value_parser.py b/Lib/email/_header_value_parser.py index 51d355fbb0abc..8a8fb8bc42a95 100644 --- a/Lib/email/_header_value_parser.py +++ b/Lib/email/_header_value_parser.py @@ -781,7 +781,7 @@ def params(self): else: try: value = value.decode(charset, 'surrogateescape') - except LookupError: + except (LookupError, UnicodeEncodeError): # XXX: there should really be a custom defect for # unknown character set to make it easy to find, # because otherwise unknown charset is a silent diff --git a/Lib/test/test_email/test__encoded_words.py b/Lib/test/test_email/test__encoded_words.py index 0b8b1de3359aa..1713962f94cae 100644 --- a/Lib/test/test_email/test__encoded_words.py +++ b/Lib/test/test_email/test__encoded_words.py @@ -130,6 +130,13 @@ def test_unknown_charset(self): # XXX Should this be a new Defect instead? defects = [errors.CharsetError]) + def test_invalid_character_in_charset(self): + self._test('=?utf-8\udce2\udc80\udc9d?q?foo=ACbar?=', + b'foo\xacbar'.decode('ascii', 'surrogateescape'), + charset = 'utf-8\udce2\udc80\udc9d', + # XXX Should this be a new Defect instead? + defects = [errors.CharsetError]) + def test_q_nonascii(self): self._test('=?utf-8?q?=C3=89ric?=', '?ric', diff --git a/Lib/test/test_email/test_email.py b/Lib/test/test_email/test_email.py index 933aa4cbc1959..69f883a3673f2 100644 --- a/Lib/test/test_email/test_email.py +++ b/Lib/test/test_email/test_email.py @@ -5356,6 +5356,15 @@ def test_rfc2231_unknown_encoding(self): Content-Transfer-Encoding: 8bit Content-Disposition: inline; filename*=X-UNKNOWN''myfile.txt +""" + msg = email.message_from_string(m) + self.assertEqual(msg.get_filename(), 'myfile.txt') + + def test_rfc2231_bad_character_in_encoding(self): + m = """\ +Content-Transfer-Encoding: 8bit +Content-Disposition: inline; filename*=utf-8\udce2\udc80\udc9d''myfile.txt + """ msg = email.message_from_string(m) self.assertEqual(msg.get_filename(), 'myfile.txt') diff --git a/Lib/test/test_email/test_headerregistry.py b/Lib/test/test_email/test_headerregistry.py index 59fcd932e0ec4..25347ef13c214 100644 --- a/Lib/test/test_email/test_headerregistry.py +++ b/Lib/test/test_email/test_headerregistry.py @@ -714,6 +714,18 @@ def content_type_as_value(self, " charset*=unknown-8bit''utf-8%E2%80%9D\n", ), + 'rfc2231_nonascii_in_charset_of_charset_parameter_value': ( + "text/plain; charset*=utf-8?''utf-8%E2%80%9D", + 'text/plain', + 'text', + 'plain', + {'charset': 'utf-8?'}, + [], + 'text/plain; charset="utf-8?"', + "Content-Type: text/plain;" + " charset*=utf-8''utf-8%E2%80%9D\n", + ), + 'rfc2231_encoded_then_unencoded_segments': ( ('application/x-foo;' '\tname*0*="us-ascii\'en-us\'My";' diff --git a/Misc/NEWS.d/next/Library/2022-03-27-12-40-16.bpo-43323.9mFPuI.rst b/Misc/NEWS.d/next/Library/2022-03-27-12-40-16.bpo-43323.9mFPuI.rst new file mode 100644 index 0000000000000..98d73101d3ee5 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-03-27-12-40-16.bpo-43323.9mFPuI.rst @@ -0,0 +1,2 @@ +Fix errors in the :mod:`email` module if the charset itself contains +undecodable/unencodable characters. From webhook-mailer at python.org Sat Apr 30 08:31:38 2022 From: webhook-mailer at python.org (miss-islington) Date: Sat, 30 Apr 2022 12:31:38 -0000 Subject: [Python-checkins] bpo-43323: Fix UnicodeEncodeError in the email module (GH-32137) Message-ID: https://github.com/python/cpython/commit/19a079690c29cd6a9f3fc3cb6aeb9f5a11c491e3 commit: 19a079690c29cd6a9f3fc3cb6aeb9f5a11c491e3 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-30T05:31:28-07:00 summary: bpo-43323: Fix UnicodeEncodeError in the email module (GH-32137) It was raised if the charset itself contains characters not encodable in UTF-8 (in particular \udcxx characters representing non-decodable bytes in the source). (cherry picked from commit e91dee87edcf6dee5dd78053004d76e5f05456d4) Co-authored-by: Serhiy Storchaka files: A Misc/NEWS.d/next/Library/2022-03-27-12-40-16.bpo-43323.9mFPuI.rst M Lib/email/_encoded_words.py M Lib/email/_header_value_parser.py M Lib/test/test_email/test__encoded_words.py M Lib/test/test_email/test_email.py M Lib/test/test_email/test_headerregistry.py diff --git a/Lib/email/_encoded_words.py b/Lib/email/_encoded_words.py index 295ae7eb21237..6795a606de037 100644 --- a/Lib/email/_encoded_words.py +++ b/Lib/email/_encoded_words.py @@ -179,15 +179,15 @@ def decode(ew): # Turn the CTE decoded bytes into unicode. try: string = bstring.decode(charset) - except UnicodeError: + except UnicodeDecodeError: defects.append(errors.UndecodableBytesDefect("Encoded word " - "contains bytes not decodable using {} charset".format(charset))) + f"contains bytes not decodable using {charset!r} charset")) string = bstring.decode(charset, 'surrogateescape') - except LookupError: + except (LookupError, UnicodeEncodeError): string = bstring.decode('ascii', 'surrogateescape') if charset.lower() != 'unknown-8bit': - defects.append(errors.CharsetError("Unknown charset {} " - "in encoded word; decoded as unknown bytes".format(charset))) + defects.append(errors.CharsetError(f"Unknown charset {charset!r} " + f"in encoded word; decoded as unknown bytes")) return string, charset, lang, defects diff --git a/Lib/email/_header_value_parser.py b/Lib/email/_header_value_parser.py index 51d355fbb0abc..8a8fb8bc42a95 100644 --- a/Lib/email/_header_value_parser.py +++ b/Lib/email/_header_value_parser.py @@ -781,7 +781,7 @@ def params(self): else: try: value = value.decode(charset, 'surrogateescape') - except LookupError: + except (LookupError, UnicodeEncodeError): # XXX: there should really be a custom defect for # unknown character set to make it easy to find, # because otherwise unknown charset is a silent diff --git a/Lib/test/test_email/test__encoded_words.py b/Lib/test/test_email/test__encoded_words.py index 0b8b1de3359aa..1713962f94cae 100644 --- a/Lib/test/test_email/test__encoded_words.py +++ b/Lib/test/test_email/test__encoded_words.py @@ -130,6 +130,13 @@ def test_unknown_charset(self): # XXX Should this be a new Defect instead? defects = [errors.CharsetError]) + def test_invalid_character_in_charset(self): + self._test('=?utf-8\udce2\udc80\udc9d?q?foo=ACbar?=', + b'foo\xacbar'.decode('ascii', 'surrogateescape'), + charset = 'utf-8\udce2\udc80\udc9d', + # XXX Should this be a new Defect instead? + defects = [errors.CharsetError]) + def test_q_nonascii(self): self._test('=?utf-8?q?=C3=89ric?=', '?ric', diff --git a/Lib/test/test_email/test_email.py b/Lib/test/test_email/test_email.py index a3ccbbbabfb32..af0ea03fedf0c 100644 --- a/Lib/test/test_email/test_email.py +++ b/Lib/test/test_email/test_email.py @@ -5323,6 +5323,15 @@ def test_rfc2231_unknown_encoding(self): Content-Transfer-Encoding: 8bit Content-Disposition: inline; filename*=X-UNKNOWN''myfile.txt +""" + msg = email.message_from_string(m) + self.assertEqual(msg.get_filename(), 'myfile.txt') + + def test_rfc2231_bad_character_in_encoding(self): + m = """\ +Content-Transfer-Encoding: 8bit +Content-Disposition: inline; filename*=utf-8\udce2\udc80\udc9d''myfile.txt + """ msg = email.message_from_string(m) self.assertEqual(msg.get_filename(), 'myfile.txt') diff --git a/Lib/test/test_email/test_headerregistry.py b/Lib/test/test_email/test_headerregistry.py index 59fcd932e0ec4..25347ef13c214 100644 --- a/Lib/test/test_email/test_headerregistry.py +++ b/Lib/test/test_email/test_headerregistry.py @@ -714,6 +714,18 @@ def content_type_as_value(self, " charset*=unknown-8bit''utf-8%E2%80%9D\n", ), + 'rfc2231_nonascii_in_charset_of_charset_parameter_value': ( + "text/plain; charset*=utf-8?''utf-8%E2%80%9D", + 'text/plain', + 'text', + 'plain', + {'charset': 'utf-8?'}, + [], + 'text/plain; charset="utf-8?"', + "Content-Type: text/plain;" + " charset*=utf-8''utf-8%E2%80%9D\n", + ), + 'rfc2231_encoded_then_unencoded_segments': ( ('application/x-foo;' '\tname*0*="us-ascii\'en-us\'My";' diff --git a/Misc/NEWS.d/next/Library/2022-03-27-12-40-16.bpo-43323.9mFPuI.rst b/Misc/NEWS.d/next/Library/2022-03-27-12-40-16.bpo-43323.9mFPuI.rst new file mode 100644 index 0000000000000..98d73101d3ee5 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-03-27-12-40-16.bpo-43323.9mFPuI.rst @@ -0,0 +1,2 @@ +Fix errors in the :mod:`email` module if the charset itself contains +undecodable/unencodable characters. From webhook-mailer at python.org Sat Apr 30 08:31:41 2022 From: webhook-mailer at python.org (miss-islington) Date: Sat, 30 Apr 2022 12:31:41 -0000 Subject: [Python-checkins] bpo-43323: Fix UnicodeEncodeError in the email module (GH-32137) Message-ID: https://github.com/python/cpython/commit/3d0a5f73f5436eac1c6c0b4b6210e3b3518dcc83 commit: 3d0a5f73f5436eac1c6c0b4b6210e3b3518dcc83 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-30T05:31:37-07:00 summary: bpo-43323: Fix UnicodeEncodeError in the email module (GH-32137) It was raised if the charset itself contains characters not encodable in UTF-8 (in particular \udcxx characters representing non-decodable bytes in the source). (cherry picked from commit e91dee87edcf6dee5dd78053004d76e5f05456d4) Co-authored-by: Serhiy Storchaka files: A Misc/NEWS.d/next/Library/2022-03-27-12-40-16.bpo-43323.9mFPuI.rst M Lib/email/_encoded_words.py M Lib/email/_header_value_parser.py M Lib/test/test_email/test__encoded_words.py M Lib/test/test_email/test_email.py M Lib/test/test_email/test_headerregistry.py diff --git a/Lib/email/_encoded_words.py b/Lib/email/_encoded_words.py index 295ae7eb21237..6795a606de037 100644 --- a/Lib/email/_encoded_words.py +++ b/Lib/email/_encoded_words.py @@ -179,15 +179,15 @@ def decode(ew): # Turn the CTE decoded bytes into unicode. try: string = bstring.decode(charset) - except UnicodeError: + except UnicodeDecodeError: defects.append(errors.UndecodableBytesDefect("Encoded word " - "contains bytes not decodable using {} charset".format(charset))) + f"contains bytes not decodable using {charset!r} charset")) string = bstring.decode(charset, 'surrogateescape') - except LookupError: + except (LookupError, UnicodeEncodeError): string = bstring.decode('ascii', 'surrogateescape') if charset.lower() != 'unknown-8bit': - defects.append(errors.CharsetError("Unknown charset {} " - "in encoded word; decoded as unknown bytes".format(charset))) + defects.append(errors.CharsetError(f"Unknown charset {charset!r} " + f"in encoded word; decoded as unknown bytes")) return string, charset, lang, defects diff --git a/Lib/email/_header_value_parser.py b/Lib/email/_header_value_parser.py index 51d355fbb0abc..8a8fb8bc42a95 100644 --- a/Lib/email/_header_value_parser.py +++ b/Lib/email/_header_value_parser.py @@ -781,7 +781,7 @@ def params(self): else: try: value = value.decode(charset, 'surrogateescape') - except LookupError: + except (LookupError, UnicodeEncodeError): # XXX: there should really be a custom defect for # unknown character set to make it easy to find, # because otherwise unknown charset is a silent diff --git a/Lib/test/test_email/test__encoded_words.py b/Lib/test/test_email/test__encoded_words.py index 0b8b1de3359aa..1713962f94cae 100644 --- a/Lib/test/test_email/test__encoded_words.py +++ b/Lib/test/test_email/test__encoded_words.py @@ -130,6 +130,13 @@ def test_unknown_charset(self): # XXX Should this be a new Defect instead? defects = [errors.CharsetError]) + def test_invalid_character_in_charset(self): + self._test('=?utf-8\udce2\udc80\udc9d?q?foo=ACbar?=', + b'foo\xacbar'.decode('ascii', 'surrogateescape'), + charset = 'utf-8\udce2\udc80\udc9d', + # XXX Should this be a new Defect instead? + defects = [errors.CharsetError]) + def test_q_nonascii(self): self._test('=?utf-8?q?=C3=89ric?=', '?ric', diff --git a/Lib/test/test_email/test_email.py b/Lib/test/test_email/test_email.py index 489cd05be4dd5..761ea90b78f15 100644 --- a/Lib/test/test_email/test_email.py +++ b/Lib/test/test_email/test_email.py @@ -5323,6 +5323,15 @@ def test_rfc2231_unknown_encoding(self): Content-Transfer-Encoding: 8bit Content-Disposition: inline; filename*=X-UNKNOWN''myfile.txt +""" + msg = email.message_from_string(m) + self.assertEqual(msg.get_filename(), 'myfile.txt') + + def test_rfc2231_bad_character_in_encoding(self): + m = """\ +Content-Transfer-Encoding: 8bit +Content-Disposition: inline; filename*=utf-8\udce2\udc80\udc9d''myfile.txt + """ msg = email.message_from_string(m) self.assertEqual(msg.get_filename(), 'myfile.txt') diff --git a/Lib/test/test_email/test_headerregistry.py b/Lib/test/test_email/test_headerregistry.py index 68bbc9561c4af..9a512fdb9d778 100644 --- a/Lib/test/test_email/test_headerregistry.py +++ b/Lib/test/test_email/test_headerregistry.py @@ -698,6 +698,18 @@ def content_type_as_value(self, " charset*=unknown-8bit''utf-8%E2%80%9D\n", ), + 'rfc2231_nonascii_in_charset_of_charset_parameter_value': ( + "text/plain; charset*=utf-8?''utf-8%E2%80%9D", + 'text/plain', + 'text', + 'plain', + {'charset': 'utf-8?'}, + [], + 'text/plain; charset="utf-8?"', + "Content-Type: text/plain;" + " charset*=utf-8''utf-8%E2%80%9D\n", + ), + 'rfc2231_encoded_then_unencoded_segments': ( ('application/x-foo;' '\tname*0*="us-ascii\'en-us\'My";' diff --git a/Misc/NEWS.d/next/Library/2022-03-27-12-40-16.bpo-43323.9mFPuI.rst b/Misc/NEWS.d/next/Library/2022-03-27-12-40-16.bpo-43323.9mFPuI.rst new file mode 100644 index 0000000000000..98d73101d3ee5 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-03-27-12-40-16.bpo-43323.9mFPuI.rst @@ -0,0 +1,2 @@ +Fix errors in the :mod:`email` module if the charset itself contains +undecodable/unencodable characters. From webhook-mailer at python.org Sat Apr 30 08:33:44 2022 From: webhook-mailer at python.org (serhiy-storchaka) Date: Sat, 30 Apr 2022 12:33:44 -0000 Subject: [Python-checkins] [3.10] gh-92049: Forbid pickling constants re._constants.SUCCESS etc (GH-92070) (GH-92073) Message-ID: https://github.com/python/cpython/commit/e8ff3c92f69b475aa20ba7c08efccbc329f9b42e commit: e8ff3c92f69b475aa20ba7c08efccbc329f9b42e branch: 3.10 author: Serhiy Storchaka committer: serhiy-storchaka date: 2022-04-30T15:33:39+03:00 summary: [3.10] gh-92049: Forbid pickling constants re._constants.SUCCESS etc (GH-92070) (GH-92073) Previously, pickling did not fail, but the result could not be unpickled. (cherry picked from commit 6d0d547033e295f91f05030322acfbb0e280fc1f) files: A Misc/NEWS.d/next/Library/2022-04-30-10-53-10.gh-issue-92049.5SEKoh.rst M Lib/sre_constants.py diff --git a/Lib/sre_constants.py b/Lib/sre_constants.py index 8e613cb3fa5dc..db3ca51e8306a 100644 --- a/Lib/sre_constants.py +++ b/Lib/sre_constants.py @@ -62,6 +62,8 @@ def __new__(cls, value, name): def __repr__(self): return self.name + __reduce__ = None + MAXREPEAT = _NamedIntConstant(MAXREPEAT, 'MAXREPEAT') def _makecodes(names): diff --git a/Misc/NEWS.d/next/Library/2022-04-30-10-53-10.gh-issue-92049.5SEKoh.rst b/Misc/NEWS.d/next/Library/2022-04-30-10-53-10.gh-issue-92049.5SEKoh.rst new file mode 100644 index 0000000000000..cad4621c65096 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-30-10-53-10.gh-issue-92049.5SEKoh.rst @@ -0,0 +1,2 @@ +Forbid pickling constants ``re._constants.SUCCESS`` etc. Previously, +pickling did not fail, but the result could not be unpickled. From webhook-mailer at python.org Sat Apr 30 11:02:05 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Sat, 30 Apr 2022 15:02:05 -0000 Subject: [Python-checkins] gh-92019: Make sqlite3.Blob indexing conform with the norm (#92020) Message-ID: https://github.com/python/cpython/commit/9ea9078ec715ffc92c10c61321f3b1184fa3cac9 commit: 9ea9078ec715ffc92c10c61321f3b1184fa3cac9 branch: main author: Erlend Egeberg Aasland committer: JelleZijlstra date: 2022-04-30T09:01:37-06:00 summary: gh-92019: Make sqlite3.Blob indexing conform with the norm (#92020) - get index now returns an int - set index now requires an int in range(0, 256) Resolves #92019 files: M Doc/includes/sqlite3/blob.py M Lib/test/test_sqlite3/test_dbapi.py M Modules/_sqlite/blob.c diff --git a/Doc/includes/sqlite3/blob.py b/Doc/includes/sqlite3/blob.py index d947059b3ae64..ff58d6c352b65 100644 --- a/Doc/includes/sqlite3/blob.py +++ b/Doc/includes/sqlite3/blob.py @@ -9,8 +9,8 @@ blob.write(b"hello, ") blob.write(b"world.") # Modify the first and last bytes of our blob - blob[0] = b"H" - blob[-1] = b"!" + blob[0] = ord("H") + blob[-1] = ord("!") # Read the contents of our blob with con.blobopen("test", "blob_col", 1) as blob: diff --git a/Lib/test/test_sqlite3/test_dbapi.py b/Lib/test/test_sqlite3/test_dbapi.py index 8bfdce2bbe92e..7e675a91b5d1f 100644 --- a/Lib/test/test_sqlite3/test_dbapi.py +++ b/Lib/test/test_sqlite3/test_dbapi.py @@ -1173,14 +1173,14 @@ def test_blob_length(self): self.assertEqual(len(self.blob), 50) def test_blob_get_item(self): - self.assertEqual(self.blob[5], b"b") - self.assertEqual(self.blob[6], b"l") - self.assertEqual(self.blob[7], b"o") - self.assertEqual(self.blob[8], b"b") - self.assertEqual(self.blob[-1], b"!") + self.assertEqual(self.blob[5], ord("b")) + self.assertEqual(self.blob[6], ord("l")) + self.assertEqual(self.blob[7], ord("o")) + self.assertEqual(self.blob[8], ord("b")) + self.assertEqual(self.blob[-1], ord("!")) def test_blob_set_item(self): - self.blob[0] = b"b" + self.blob[0] = ord("b") expected = b"b" + self.data[1:] actual = self.cx.execute("select b from test").fetchone()[0] self.assertEqual(actual, expected) @@ -1188,23 +1188,14 @@ def test_blob_set_item(self): def test_blob_set_item_with_offset(self): self.blob.seek(0, SEEK_END) self.assertEqual(self.blob.read(), b"") # verify that we're at EOB - self.blob[0] = b"T" - self.blob[-1] = b"." + self.blob[0] = ord("T") + self.blob[-1] = ord(".") self.blob.seek(0, SEEK_SET) expected = b"This blob data string is exactly fifty bytes long." self.assertEqual(self.blob.read(), expected) - def test_blob_set_buffer_object(self): + def test_blob_set_slice_buffer_object(self): from array import array - self.blob[0] = memoryview(b"1") - self.assertEqual(self.blob[0], b"1") - - self.blob[1] = bytearray(b"2") - self.assertEqual(self.blob[1], b"2") - - self.blob[2] = array("b", [4]) - self.assertEqual(self.blob[2], b"\x04") - self.blob[0:5] = memoryview(b"12345") self.assertEqual(self.blob[0:5], b"12345") @@ -1215,8 +1206,8 @@ def test_blob_set_buffer_object(self): self.assertEqual(self.blob[0:5], b"\x01\x02\x03\x04\x05") def test_blob_set_item_negative_index(self): - self.blob[-1] = b"z" - self.assertEqual(self.blob[-1], b"z") + self.blob[-1] = 255 + self.assertEqual(self.blob[-1], 255) def test_blob_get_slice(self): self.assertEqual(self.blob[5:14], b"blob data") @@ -1264,13 +1255,29 @@ def test_blob_get_item_error(self): with self.assertRaisesRegex(IndexError, "cannot fit 'int'"): self.blob[ULLONG_MAX] + # Provoke read error + self.cx.execute("update test set b='aaaa' where rowid=1") + with self.assertRaises(sqlite.OperationalError): + self.blob[0] + def test_blob_set_item_error(self): - with self.assertRaisesRegex(ValueError, "must be a single byte"): + with self.assertRaisesRegex(TypeError, "cannot be interpreted"): self.blob[0] = b"multiple" + with self.assertRaisesRegex(TypeError, "cannot be interpreted"): + self.blob[0] = b"1" + with self.assertRaisesRegex(TypeError, "cannot be interpreted"): + self.blob[0] = bytearray(b"1") with self.assertRaisesRegex(TypeError, "doesn't support.*deletion"): del self.blob[0] with self.assertRaisesRegex(IndexError, "Blob index out of range"): - self.blob[1000] = b"a" + self.blob[1000] = 0 + with self.assertRaisesRegex(ValueError, "must be in range"): + self.blob[0] = -1 + with self.assertRaisesRegex(ValueError, "must be in range"): + self.blob[0] = 256 + # Overflow errors are overridden with ValueError + with self.assertRaisesRegex(ValueError, "must be in range"): + self.blob[0] = 2**65 def test_blob_set_slice_error(self): with self.assertRaisesRegex(IndexError, "wrong size"): diff --git a/Modules/_sqlite/blob.c b/Modules/_sqlite/blob.c index 0c57ff8ca4252..76d261baf00f3 100644 --- a/Modules/_sqlite/blob.c +++ b/Modules/_sqlite/blob.c @@ -120,10 +120,26 @@ blob_seterror(pysqlite_Blob *self, int rc) } static PyObject * -inner_read(pysqlite_Blob *self, Py_ssize_t length, Py_ssize_t offset) +read_single(pysqlite_Blob *self, Py_ssize_t offset) +{ + unsigned char buf = 0; + int rc; + Py_BEGIN_ALLOW_THREADS + rc = sqlite3_blob_read(self->blob, (void *)&buf, 1, (int)offset); + Py_END_ALLOW_THREADS + + if (rc != SQLITE_OK) { + blob_seterror(self, rc); + return NULL; + } + return PyLong_FromUnsignedLong((unsigned long)buf); +} + +static PyObject * +read_multiple(pysqlite_Blob *self, Py_ssize_t length, Py_ssize_t offset) { assert(length <= sqlite3_blob_bytes(self->blob)); - assert(offset <= sqlite3_blob_bytes(self->blob)); + assert(offset < sqlite3_blob_bytes(self->blob)); PyObject *buffer = PyBytes_FromStringAndSize(NULL, length); if (buffer == NULL) { @@ -175,7 +191,12 @@ blob_read_impl(pysqlite_Blob *self, int length) length = max_read_len; } - PyObject *buffer = inner_read(self, length, self->offset); + assert(length >= 0); + if (length == 0) { + return PyBytes_FromStringAndSize(NULL, 0); + } + + PyObject *buffer = read_multiple(self, length, self->offset); if (buffer == NULL) { return NULL; } @@ -387,7 +408,7 @@ subscript_index(pysqlite_Blob *self, PyObject *item) if (i < 0) { return NULL; } - return inner_read(self, 1, i); + return read_single(self, i); } static int @@ -411,9 +432,9 @@ subscript_slice(pysqlite_Blob *self, PyObject *item) } if (step == 1) { - return inner_read(self, len, start); + return read_multiple(self, len, start); } - PyObject *blob = inner_read(self, stop - start, start); + PyObject *blob = read_multiple(self, stop - start, start); if (blob == NULL) { return NULL; } @@ -455,24 +476,29 @@ ass_subscript_index(pysqlite_Blob *self, PyObject *item, PyObject *value) "Blob doesn't support item deletion"); return -1; } + if (!PyLong_Check(value)) { + PyErr_Format(PyExc_TypeError, + "'%s' object cannot be interpreted as an integer", + Py_TYPE(value)->tp_name); + return -1; + } Py_ssize_t i = get_subscript_index(self, item); if (i < 0) { return -1; } - Py_buffer vbuf; - if (PyObject_GetBuffer(value, &vbuf, PyBUF_SIMPLE) < 0) { - return -1; + long val = PyLong_AsLong(value); + if (val == -1 && PyErr_Occurred()) { + PyErr_Clear(); + val = -1; } - int rc = -1; - if (vbuf.len != 1) { - PyErr_SetString(PyExc_ValueError, "Blob assignment must be a single byte"); - } - else { - rc = inner_write(self, (const char *)vbuf.buf, 1, i); + if (val < 0 || val > 255) { + PyErr_SetString(PyExc_ValueError, "byte must be in range(0, 256)"); + return -1; } - PyBuffer_Release(&vbuf); - return rc; + // Downcast to avoid endianness problems. + unsigned char byte = (unsigned char)val; + return inner_write(self, (const void *)&byte, 1, i); } static int @@ -507,7 +533,7 @@ ass_subscript_slice(pysqlite_Blob *self, PyObject *item, PyObject *value) rc = inner_write(self, vbuf.buf, len, start); } else { - PyObject *blob_bytes = inner_read(self, stop - start, start); + PyObject *blob_bytes = read_multiple(self, stop - start, start); if (blob_bytes != NULL) { char *blob_buf = PyBytes_AS_STRING(blob_bytes); for (Py_ssize_t i = 0, j = 0; i < len; i++, j += step) { From webhook-mailer at python.org Sat Apr 30 11:22:44 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Sat, 30 Apr 2022 15:22:44 -0000 Subject: [Python-checkins] gh-85864: io docs: Add missing position-only parameters (#91950) Message-ID: https://github.com/python/cpython/commit/3a8e2b6e65fea1252477f6e29a384fa9a492ed06 commit: 3a8e2b6e65fea1252477f6e29a384fa9a492ed06 branch: main author: slateny <46876382+slateny at users.noreply.github.com> committer: JelleZijlstra date: 2022-04-30T09:22:33-06:00 summary: gh-85864: io docs: Add missing position-only parameters (#91950) files: M Doc/library/io.rst diff --git a/Doc/library/io.rst b/Doc/library/io.rst index 82757539c62ac..ecd8c8043351d 100644 --- a/Doc/library/io.rst +++ b/Doc/library/io.rst @@ -401,7 +401,7 @@ I/O Base Classes Note that it's already possible to iterate on file objects using ``for line in file: ...`` without calling ``file.readlines()``. - .. method:: seek(offset, whence=SEEK_SET) + .. method:: seek(offset, whence=SEEK_SET, /) Change the stream position to the given byte *offset*. *offset* is interpreted relative to the position indicated by *whence*. The default @@ -433,7 +433,7 @@ I/O Base Classes Return the current stream position. - .. method:: truncate(size=None) + .. method:: truncate(size=None, /) Resize the stream to the given *size* in bytes (or the current position if *size* is not specified). The current stream position isn't changed. @@ -450,7 +450,7 @@ I/O Base Classes Return ``True`` if the stream supports writing. If ``False``, :meth:`write` and :meth:`truncate` will raise :exc:`OSError`. - .. method:: writelines(lines) + .. method:: writelines(lines, /) Write a list of lines to the stream. Line separators are not added, so it is usual for each of the lines provided to have a line separator at the @@ -494,7 +494,7 @@ I/O Base Classes Read and return all the bytes from the stream until EOF, using multiple calls to the stream if necessary. - .. method:: readinto(b) + .. method:: readinto(b, /) Read bytes into a pre-allocated, writable :term:`bytes-like object` *b*, and return the @@ -502,7 +502,7 @@ I/O Base Classes If the object is in non-blocking mode and no bytes are available, ``None`` is returned. - .. method:: write(b) + .. method:: write(b, /) Write the given :term:`bytes-like object`, *b*, to the underlying raw stream, and return the number of @@ -559,7 +559,7 @@ I/O Base Classes .. versionadded:: 3.1 - .. method:: read(size=-1) + .. method:: read(size=-1, /) Read and return up to *size* bytes. If the argument is omitted, ``None``, or negative, data is read and returned until EOF is reached. An empty @@ -574,7 +574,7 @@ I/O Base Classes A :exc:`BlockingIOError` is raised if the underlying raw stream is in non blocking-mode, and has no data available at the moment. - .. method:: read1([size]) + .. method:: read1(size=-1, /) Read and return up to *size* bytes, with at most one call to the underlying raw stream's :meth:`~RawIOBase.read` (or @@ -609,7 +609,7 @@ I/O Base Classes .. versionadded:: 3.5 - .. method:: write(b) + .. method:: write(b, /) Write the given :term:`bytes-like object`, *b*, and return the number of bytes written (always equal to the length of *b* in bytes, since if @@ -692,7 +692,7 @@ Buffered Streams Buffered I/O streams provide a higher-level interface to an I/O device than raw I/O does. -.. class:: BytesIO([initial_bytes]) +.. class:: BytesIO(initial_bytes=b'') A binary stream using an in-memory bytes buffer. It inherits :class:`BufferedIOBase`. The buffer is discarded when the @@ -727,14 +727,14 @@ than raw I/O does. Return :class:`bytes` containing the entire contents of the buffer. - .. method:: read1([size], /) + .. method:: read1(size=-1, /) In :class:`BytesIO`, this is the same as :meth:`~BufferedIOBase.read`. .. versionchanged:: 3.7 The *size* argument is now optional. - .. method:: readinto1(b) + .. method:: readinto1(b, /) In :class:`BytesIO`, this is the same as :meth:`~BufferedIOBase.readinto`. @@ -757,18 +757,18 @@ than raw I/O does. :class:`BufferedReader` provides or overrides these methods in addition to those from :class:`BufferedIOBase` and :class:`IOBase`: - .. method:: peek([size]) + .. method:: peek(size=0, /) Return bytes from the stream without advancing the position. At most one single read on the raw stream is done to satisfy the call. The number of bytes returned may be less or more than requested. - .. method:: read([size]) + .. method:: read(size=-1, /) Read and return *size* bytes, or if *size* is not given or negative, until EOF or if the read call would block in non-blocking mode. - .. method:: read1([size]) + .. method:: read1(size=-1, /) Read and return up to *size* bytes with only one call on the raw stream. If at least one byte is buffered, only buffered bytes are returned. @@ -900,14 +900,14 @@ Text I/O Read and return at most *size* characters from the stream as a single :class:`str`. If *size* is negative or ``None``, reads until EOF. - .. method:: readline(size=-1) + .. method:: readline(size=-1, /) Read until newline or EOF and return a single ``str``. If the stream is already at EOF, an empty string is returned. If *size* is specified, at most *size* characters will be read. - .. method:: seek(offset, whence=SEEK_SET) + .. method:: seek(offset, whence=SEEK_SET, /) Change the stream position to the given *offset*. Behaviour depends on the *whence* parameter. The default value for *whence* is @@ -934,7 +934,7 @@ Text I/O does not usually represent a number of bytes in the underlying binary storage. - .. method:: write(s) + .. method:: write(s, /) Write the string *s* to the stream and return the number of characters written. From webhook-mailer at python.org Sat Apr 30 11:39:45 2022 From: webhook-mailer at python.org (miss-islington) Date: Sat, 30 Apr 2022 15:39:45 -0000 Subject: [Python-checkins] gh-85864: io docs: Add missing position-only parameters (GH-91950) Message-ID: https://github.com/python/cpython/commit/5ed9556cfbcfddc64e5e928b0f6ff1908553fcae commit: 5ed9556cfbcfddc64e5e928b0f6ff1908553fcae branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-30T08:39:29-07:00 summary: gh-85864: io docs: Add missing position-only parameters (GH-91950) (cherry picked from commit 3a8e2b6e65fea1252477f6e29a384fa9a492ed06) Co-authored-by: slateny <46876382+slateny at users.noreply.github.com> files: M Doc/library/io.rst diff --git a/Doc/library/io.rst b/Doc/library/io.rst index b566d0c42499d..c125ec2aa2e67 100644 --- a/Doc/library/io.rst +++ b/Doc/library/io.rst @@ -396,7 +396,7 @@ I/O Base Classes Note that it's already possible to iterate on file objects using ``for line in file: ...`` without calling ``file.readlines()``. - .. method:: seek(offset, whence=SEEK_SET) + .. method:: seek(offset, whence=SEEK_SET, /) Change the stream position to the given byte *offset*. *offset* is interpreted relative to the position indicated by *whence*. The default @@ -428,7 +428,7 @@ I/O Base Classes Return the current stream position. - .. method:: truncate(size=None) + .. method:: truncate(size=None, /) Resize the stream to the given *size* in bytes (or the current position if *size* is not specified). The current stream position isn't changed. @@ -445,7 +445,7 @@ I/O Base Classes Return ``True`` if the stream supports writing. If ``False``, :meth:`write` and :meth:`truncate` will raise :exc:`OSError`. - .. method:: writelines(lines) + .. method:: writelines(lines, /) Write a list of lines to the stream. Line separators are not added, so it is usual for each of the lines provided to have a line separator at the @@ -489,7 +489,7 @@ I/O Base Classes Read and return all the bytes from the stream until EOF, using multiple calls to the stream if necessary. - .. method:: readinto(b) + .. method:: readinto(b, /) Read bytes into a pre-allocated, writable :term:`bytes-like object` *b*, and return the @@ -497,7 +497,7 @@ I/O Base Classes If the object is in non-blocking mode and no bytes are available, ``None`` is returned. - .. method:: write(b) + .. method:: write(b, /) Write the given :term:`bytes-like object`, *b*, to the underlying raw stream, and return the number of @@ -554,7 +554,7 @@ I/O Base Classes .. versionadded:: 3.1 - .. method:: read(size=-1) + .. method:: read(size=-1, /) Read and return up to *size* bytes. If the argument is omitted, ``None``, or negative, data is read and returned until EOF is reached. An empty @@ -569,7 +569,7 @@ I/O Base Classes A :exc:`BlockingIOError` is raised if the underlying raw stream is in non blocking-mode, and has no data available at the moment. - .. method:: read1([size]) + .. method:: read1(size=-1, /) Read and return up to *size* bytes, with at most one call to the underlying raw stream's :meth:`~RawIOBase.read` (or @@ -604,7 +604,7 @@ I/O Base Classes .. versionadded:: 3.5 - .. method:: write(b) + .. method:: write(b, /) Write the given :term:`bytes-like object`, *b*, and return the number of bytes written (always equal to the length of *b* in bytes, since if @@ -687,7 +687,7 @@ Buffered Streams Buffered I/O streams provide a higher-level interface to an I/O device than raw I/O does. -.. class:: BytesIO([initial_bytes]) +.. class:: BytesIO(initial_bytes=b'') A binary stream using an in-memory bytes buffer. It inherits :class:`BufferedIOBase`. The buffer is discarded when the @@ -722,14 +722,14 @@ than raw I/O does. Return :class:`bytes` containing the entire contents of the buffer. - .. method:: read1([size], /) + .. method:: read1(size=-1, /) In :class:`BytesIO`, this is the same as :meth:`~BufferedIOBase.read`. .. versionchanged:: 3.7 The *size* argument is now optional. - .. method:: readinto1(b) + .. method:: readinto1(b, /) In :class:`BytesIO`, this is the same as :meth:`~BufferedIOBase.readinto`. @@ -752,18 +752,18 @@ than raw I/O does. :class:`BufferedReader` provides or overrides these methods in addition to those from :class:`BufferedIOBase` and :class:`IOBase`: - .. method:: peek([size]) + .. method:: peek(size=0, /) Return bytes from the stream without advancing the position. At most one single read on the raw stream is done to satisfy the call. The number of bytes returned may be less or more than requested. - .. method:: read([size]) + .. method:: read(size=-1, /) Read and return *size* bytes, or if *size* is not given or negative, until EOF or if the read call would block in non-blocking mode. - .. method:: read1([size]) + .. method:: read1(size=-1, /) Read and return up to *size* bytes with only one call on the raw stream. If at least one byte is buffered, only buffered bytes are returned. @@ -895,14 +895,14 @@ Text I/O Read and return at most *size* characters from the stream as a single :class:`str`. If *size* is negative or ``None``, reads until EOF. - .. method:: readline(size=-1) + .. method:: readline(size=-1, /) Read until newline or EOF and return a single ``str``. If the stream is already at EOF, an empty string is returned. If *size* is specified, at most *size* characters will be read. - .. method:: seek(offset, whence=SEEK_SET) + .. method:: seek(offset, whence=SEEK_SET, /) Change the stream position to the given *offset*. Behaviour depends on the *whence* parameter. The default value for *whence* is @@ -929,7 +929,7 @@ Text I/O does not usually represent a number of bytes in the underlying binary storage. - .. method:: write(s) + .. method:: write(s, /) Write the string *s* to the stream and return the number of characters written. From webhook-mailer at python.org Sat Apr 30 11:43:05 2022 From: webhook-mailer at python.org (miss-islington) Date: Sat, 30 Apr 2022 15:43:05 -0000 Subject: [Python-checkins] gh-85864: io docs: Add missing position-only parameters (GH-91950) Message-ID: https://github.com/python/cpython/commit/2c859e3d589b2a47b0b5267e77a4f80034f3058d commit: 2c859e3d589b2a47b0b5267e77a4f80034f3058d branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-30T08:42:55-07:00 summary: gh-85864: io docs: Add missing position-only parameters (GH-91950) (cherry picked from commit 3a8e2b6e65fea1252477f6e29a384fa9a492ed06) Co-authored-by: slateny <46876382+slateny at users.noreply.github.com> files: M Doc/library/io.rst diff --git a/Doc/library/io.rst b/Doc/library/io.rst index 28ad8dfd13126..ea57aae71a235 100644 --- a/Doc/library/io.rst +++ b/Doc/library/io.rst @@ -320,7 +320,7 @@ I/O Base Classes Note that it's already possible to iterate on file objects using ``for line in file: ...`` without calling ``file.readlines()``. - .. method:: seek(offset, whence=SEEK_SET) + .. method:: seek(offset, whence=SEEK_SET, /) Change the stream position to the given byte *offset*. *offset* is interpreted relative to the position indicated by *whence*. The default @@ -352,7 +352,7 @@ I/O Base Classes Return the current stream position. - .. method:: truncate(size=None) + .. method:: truncate(size=None, /) Resize the stream to the given *size* in bytes (or the current position if *size* is not specified). The current stream position isn't changed. @@ -369,7 +369,7 @@ I/O Base Classes Return ``True`` if the stream supports writing. If ``False``, :meth:`write` and :meth:`truncate` will raise :exc:`OSError`. - .. method:: writelines(lines) + .. method:: writelines(lines, /) Write a list of lines to the stream. Line separators are not added, so it is usual for each of the lines provided to have a line separator at the @@ -413,7 +413,7 @@ I/O Base Classes Read and return all the bytes from the stream until EOF, using multiple calls to the stream if necessary. - .. method:: readinto(b) + .. method:: readinto(b, /) Read bytes into a pre-allocated, writable :term:`bytes-like object` *b*, and return the @@ -421,7 +421,7 @@ I/O Base Classes If the object is in non-blocking mode and no bytes are available, ``None`` is returned. - .. method:: write(b) + .. method:: write(b, /) Write the given :term:`bytes-like object`, *b*, to the underlying raw stream, and return the number of @@ -478,7 +478,7 @@ I/O Base Classes .. versionadded:: 3.1 - .. method:: read(size=-1) + .. method:: read(size=-1, /) Read and return up to *size* bytes. If the argument is omitted, ``None``, or negative, data is read and returned until EOF is reached. An empty @@ -493,7 +493,7 @@ I/O Base Classes A :exc:`BlockingIOError` is raised if the underlying raw stream is in non blocking-mode, and has no data available at the moment. - .. method:: read1([size]) + .. method:: read1(size=-1, /) Read and return up to *size* bytes, with at most one call to the underlying raw stream's :meth:`~RawIOBase.read` (or @@ -528,7 +528,7 @@ I/O Base Classes .. versionadded:: 3.5 - .. method:: write(b) + .. method:: write(b, /) Write the given :term:`bytes-like object`, *b*, and return the number of bytes written (always equal to the length of *b* in bytes, since if @@ -611,7 +611,7 @@ Buffered Streams Buffered I/O streams provide a higher-level interface to an I/O device than raw I/O does. -.. class:: BytesIO([initial_bytes]) +.. class:: BytesIO(initial_bytes=b'') A binary stream using an in-memory bytes buffer. It inherits :class:`BufferedIOBase`. The buffer is discarded when the @@ -646,14 +646,14 @@ than raw I/O does. Return :class:`bytes` containing the entire contents of the buffer. - .. method:: read1([size], /) + .. method:: read1(size=-1, /) In :class:`BytesIO`, this is the same as :meth:`~BufferedIOBase.read`. .. versionchanged:: 3.7 The *size* argument is now optional. - .. method:: readinto1(b) + .. method:: readinto1(b, /) In :class:`BytesIO`, this is the same as :meth:`~BufferedIOBase.readinto`. @@ -676,18 +676,18 @@ than raw I/O does. :class:`BufferedReader` provides or overrides these methods in addition to those from :class:`BufferedIOBase` and :class:`IOBase`: - .. method:: peek([size]) + .. method:: peek(size=0, /) Return bytes from the stream without advancing the position. At most one single read on the raw stream is done to satisfy the call. The number of bytes returned may be less or more than requested. - .. method:: read([size]) + .. method:: read(size=-1, /) Read and return *size* bytes, or if *size* is not given or negative, until EOF or if the read call would block in non-blocking mode. - .. method:: read1([size]) + .. method:: read1(size=-1, /) Read and return up to *size* bytes with only one call on the raw stream. If at least one byte is buffered, only buffered bytes are returned. @@ -819,14 +819,14 @@ Text I/O Read and return at most *size* characters from the stream as a single :class:`str`. If *size* is negative or ``None``, reads until EOF. - .. method:: readline(size=-1) + .. method:: readline(size=-1, /) Read until newline or EOF and return a single ``str``. If the stream is already at EOF, an empty string is returned. If *size* is specified, at most *size* characters will be read. - .. method:: seek(offset, whence=SEEK_SET) + .. method:: seek(offset, whence=SEEK_SET, /) Change the stream position to the given *offset*. Behaviour depends on the *whence* parameter. The default value for *whence* is @@ -853,7 +853,7 @@ Text I/O does not usually represent a number of bytes in the underlying binary storage. - .. method:: write(s) + .. method:: write(s, /) Write the string *s* to the stream and return the number of characters written. From webhook-mailer at python.org Sat Apr 30 14:36:01 2022 From: webhook-mailer at python.org (markshannon) Date: Sat, 30 Apr 2022 18:36:01 -0000 Subject: [Python-checkins] gh-92063: Enforce types in specialized PRECALL opcodes (GH-92068) Message-ID: https://github.com/python/cpython/commit/868b1afa05c7847e6ffdd1808916719b511b4bcc commit: 868b1afa05c7847e6ffdd1808916719b511b4bcc branch: main author: Dennis Sweeney <36520290+sweeneyde at users.noreply.github.com> committer: markshannon date: 2022-04-30T12:35:33-06:00 summary: gh-92063: Enforce types in specialized PRECALL opcodes (GH-92068) * Check the types of PRECALL_METHOD_DESCRIPTOR_FAST_WITH_KEYWORDS * fix PRECALL_NO_KW_METHOD_DESCRIPTOR_NOARGS as well * fix PRECALL_NO_KW_METHOD_DESCRIPTOR_O * fix PRECALL_NO_KW_METHOD_DESCRIPTOR_FAST files: A Misc/NEWS.d/next/Core and Builtins/2022-04-30-04-26-01.gh-issue-92063.vHnhf6.rst M Lib/test/test_descr.py M Python/ceval.c diff --git a/Lib/test/test_descr.py b/Lib/test/test_descr.py index 378ff5227e221..48d43d7af85d9 100644 --- a/Lib/test/test_descr.py +++ b/Lib/test/test_descr.py @@ -4726,6 +4726,33 @@ class FakeStr: with self.assertRaises(TypeError): str.__add__(fake_str, "abc") + def test_specialized_method_calls_check_types(self): + # https://github.com/python/cpython/issues/92063 + class Thing: + pass + thing = Thing() + for i in range(20): + with self.assertRaises(TypeError): + # PRECALL_METHOD_DESCRIPTOR_FAST_WITH_KEYWORDS + list.sort(thing) + for i in range(20): + with self.assertRaises(TypeError): + # PRECALL_METHOD_DESCRIPTOR_FAST_WITH_KEYWORDS + str.split(thing) + for i in range(20): + with self.assertRaises(TypeError): + # PRECALL_NO_KW_METHOD_DESCRIPTOR_NOARGS + str.upper(thing) + for i in range(20): + with self.assertRaises(TypeError): + # PRECALL_NO_KW_METHOD_DESCRIPTOR_FAST + str.strip(thing) + from collections import deque + for i in range(20): + with self.assertRaises(TypeError): + # PRECALL_NO_KW_METHOD_DESCRIPTOR_O + deque.append(thing, thing) + def test_repr_as_str(self): # Issue #11603: crash or infinite loop when rebinding __str__ as # __repr__. diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-04-30-04-26-01.gh-issue-92063.vHnhf6.rst b/Misc/NEWS.d/next/Core and Builtins/2022-04-30-04-26-01.gh-issue-92063.vHnhf6.rst new file mode 100644 index 0000000000000..d737ccc24481f --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2022-04-30-04-26-01.gh-issue-92063.vHnhf6.rst @@ -0,0 +1,2 @@ +The ``PRECALL_METHOD_DESCRIPTOR_FAST_WITH_KEYWORDS`` instruction +now ensures methods are called only on objects of the correct type. diff --git a/Python/ceval.c b/Python/ceval.c index 1d2c6432d062f..f3329b5d9d454 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -5254,11 +5254,15 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int assert(call_shape.kwnames == NULL); int is_meth = is_method(stack_pointer, oparg); int total_args = oparg + is_meth; - PyObject *callable = PEEK(total_args + 1); + PyMethodDescrObject *callable = + (PyMethodDescrObject *)PEEK(total_args + 1); DEOPT_IF(total_args != 2, PRECALL); DEOPT_IF(!Py_IS_TYPE(callable, &PyMethodDescr_Type), PRECALL); - PyMethodDef *meth = ((PyMethodDescrObject *)callable)->d_method; + PyMethodDef *meth = callable->d_method; DEOPT_IF(meth->ml_flags != METH_O, PRECALL); + PyObject *arg = TOP(); + PyObject *self = SECOND(); + DEOPT_IF(!Py_IS_TYPE(self, callable->d_common.d_type), PRECALL); STAT_INC(PRECALL, hit); SKIP_CALL(); PyCFunction cfunc = meth->ml_meth; @@ -5267,8 +5271,6 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int if (_Py_EnterRecursiveCall(tstate, " while calling a Python object")) { goto error; } - PyObject *arg = TOP(); - PyObject *self = SECOND(); PyObject *res = cfunc(self, arg); _Py_LeaveRecursiveCall(tstate); assert((res != NULL) ^ (_PyErr_Occurred(tstate) != NULL)); @@ -5287,17 +5289,22 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int TARGET(PRECALL_METHOD_DESCRIPTOR_FAST_WITH_KEYWORDS) { int is_meth = is_method(stack_pointer, oparg); int total_args = oparg + is_meth; - PyObject *callable = PEEK(total_args + 1); + PyMethodDescrObject *callable = + (PyMethodDescrObject *)PEEK(total_args + 1); DEOPT_IF(!Py_IS_TYPE(callable, &PyMethodDescr_Type), PRECALL); - PyMethodDef *meth = ((PyMethodDescrObject *)callable)->d_method; + PyMethodDef *meth = callable->d_method; DEOPT_IF(meth->ml_flags != (METH_FASTCALL|METH_KEYWORDS), PRECALL); + PyTypeObject *d_type = callable->d_common.d_type; + PyObject *self = PEEK(total_args); + DEOPT_IF(!Py_IS_TYPE(self, d_type), PRECALL); STAT_INC(PRECALL, hit); SKIP_CALL(); int nargs = total_args-1; STACK_SHRINK(nargs); - _PyCFunctionFastWithKeywords cfunc = (_PyCFunctionFastWithKeywords)(void(*)(void))meth->ml_meth; - PyObject *self = TOP(); - PyObject *res = cfunc(self, stack_pointer, nargs - KWNAMES_LEN(), call_shape.kwnames); + _PyCFunctionFastWithKeywords cfunc = + (_PyCFunctionFastWithKeywords)(void(*)(void))meth->ml_meth; + PyObject *res = cfunc(self, stack_pointer, nargs - KWNAMES_LEN(), + call_shape.kwnames); assert((res != NULL) ^ (_PyErr_Occurred(tstate) != NULL)); call_shape.kwnames = NULL; @@ -5322,9 +5329,11 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int int is_meth = is_method(stack_pointer, oparg); int total_args = oparg + is_meth; DEOPT_IF(total_args != 1, PRECALL); - PyObject *callable = SECOND(); + PyMethodDescrObject *callable = (PyMethodDescrObject *)SECOND(); DEOPT_IF(!Py_IS_TYPE(callable, &PyMethodDescr_Type), PRECALL); - PyMethodDef *meth = ((PyMethodDescrObject *)callable)->d_method; + PyMethodDef *meth = callable->d_method; + PyObject *self = TOP(); + DEOPT_IF(!Py_IS_TYPE(self, callable->d_common.d_type), PRECALL); DEOPT_IF(meth->ml_flags != METH_NOARGS, PRECALL); STAT_INC(PRECALL, hit); SKIP_CALL(); @@ -5334,7 +5343,6 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int if (_Py_EnterRecursiveCall(tstate, " while calling a Python object")) { goto error; } - PyObject *self = TOP(); PyObject *res = cfunc(self, NULL); _Py_LeaveRecursiveCall(tstate); assert((res != NULL) ^ (_PyErr_Occurred(tstate) != NULL)); @@ -5353,17 +5361,20 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int assert(call_shape.kwnames == NULL); int is_meth = is_method(stack_pointer, oparg); int total_args = oparg + is_meth; - PyObject *callable = PEEK(total_args + 1); + PyMethodDescrObject *callable = + (PyMethodDescrObject *)PEEK(total_args + 1); /* Builtin METH_FASTCALL methods, without keywords */ DEOPT_IF(!Py_IS_TYPE(callable, &PyMethodDescr_Type), PRECALL); - PyMethodDef *meth = ((PyMethodDescrObject *)callable)->d_method; + PyMethodDef *meth = callable->d_method; DEOPT_IF(meth->ml_flags != METH_FASTCALL, PRECALL); + PyObject *self = PEEK(total_args); + DEOPT_IF(!Py_IS_TYPE(self, callable->d_common.d_type), PRECALL); STAT_INC(PRECALL, hit); SKIP_CALL(); - _PyCFunctionFast cfunc = (_PyCFunctionFast)(void(*)(void))meth->ml_meth; + _PyCFunctionFast cfunc = + (_PyCFunctionFast)(void(*)(void))meth->ml_meth; int nargs = total_args-1; STACK_SHRINK(nargs); - PyObject *self = TOP(); PyObject *res = cfunc(self, stack_pointer, nargs); assert((res != NULL) ^ (_PyErr_Occurred(tstate) != NULL)); /* Clear the stack of the arguments. */ From webhook-mailer at python.org Sat Apr 30 15:54:11 2022 From: webhook-mailer at python.org (sweeneyde) Date: Sat, 30 Apr 2022 19:54:11 -0000 Subject: [Python-checkins] Convert :issue: to :gh: in whatsnew (GH-92089) Message-ID: https://github.com/python/cpython/commit/b660d0931e06d4b84b4e03cbc328b58f18bad3ca commit: b660d0931e06d4b84b4e03cbc328b58f18bad3ca branch: main author: Dennis Sweeney <36520290+sweeneyde at users.noreply.github.com> committer: sweeneyde <36520290+sweeneyde at users.noreply.github.com> date: 2022-04-30T15:53:59-04:00 summary: Convert :issue: to :gh: in whatsnew (GH-92089) files: M Doc/whatsnew/3.11.rst diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index 59c8dd6e44732..0a8ba1e8843e0 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -1059,7 +1059,7 @@ Deprecated a :exc:`DeprecationWarning`. In a future Python version they will be a :exc:`SyntaxWarning` and eventually a :exc:`SyntaxError`. - (Contributed by Serhiy Storchaka in :issue:`81548`.) + (Contributed by Serhiy Storchaka in :gh:`81548`.) * The :mod:`lib2to3` package and ``2to3`` tool are now deprecated and may not be able to parse Python 3.10 or newer. See the :pep:`617` (New PEG parser for From webhook-mailer at python.org Sat Apr 30 17:24:06 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Sat, 30 Apr 2022 21:24:06 -0000 Subject: [Python-checkins] gh-87192: Update wording for fcntl 'Changed in' (#91658) Message-ID: https://github.com/python/cpython/commit/d7eb1ffbe8f913693e4c9ffa1b32edccac987ab6 commit: d7eb1ffbe8f913693e4c9ffa1b32edccac987ab6 branch: main author: slateny <46876382+slateny at users.noreply.github.com> committer: JelleZijlstra date: 2022-04-30T15:24:01-06:00 summary: gh-87192: Update wording for fcntl 'Changed in' (#91658) files: M Doc/library/fcntl.rst diff --git a/Doc/library/fcntl.rst b/Doc/library/fcntl.rst index 846c8d7ba96da..1ecd552fbd02c 100644 --- a/Doc/library/fcntl.rst +++ b/Doc/library/fcntl.rst @@ -37,7 +37,8 @@ descriptor. On macOS, the fcntl module exposes the ``F_GETPATH`` constant, which obtains the path of a file from a file descriptor. On Linux(>=3.15), the fcntl module exposes the ``F_OFD_GETLK``, ``F_OFD_SETLK`` - and ``F_OFD_SETLKW`` constants, which working with open file description locks. + and ``F_OFD_SETLKW`` constants, which are used when working with open file + description locks. .. versionchanged:: 3.10 On Linux >= 2.6.11, the fcntl module exposes the ``F_GETPIPE_SZ`` and From webhook-mailer at python.org Sat Apr 30 17:32:14 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Sat, 30 Apr 2022 21:32:14 -0000 Subject: [Python-checkins] gh-87801: Add run() to subprocess.CalledProcessError description (#91628) Message-ID: https://github.com/python/cpython/commit/567be058b403db9689af45bf831d4c732c8b1105 commit: 567be058b403db9689af45bf831d4c732c8b1105 branch: main author: slateny <46876382+slateny at users.noreply.github.com> committer: JelleZijlstra date: 2022-04-30T15:32:00-06:00 summary: gh-87801: Add run() to subprocess.CalledProcessError description (#91628) files: M Doc/library/subprocess.rst diff --git a/Doc/library/subprocess.rst b/Doc/library/subprocess.rst index 6a334acc5a17c..1e619a51e216a 100644 --- a/Doc/library/subprocess.rst +++ b/Doc/library/subprocess.rst @@ -214,7 +214,9 @@ compatibility with older versions, see the :ref:`call-function-trio` section. .. exception:: CalledProcessError Subclass of :exc:`SubprocessError`, raised when a process run by - :func:`check_call` or :func:`check_output` returns a non-zero exit status. + :func:`check_call`, :func:`check_output`, or :func:`run` (with ``check=True``) + returns a non-zero exit status. + .. attribute:: returncode From webhook-mailer at python.org Sat Apr 30 17:44:12 2022 From: webhook-mailer at python.org (miss-islington) Date: Sat, 30 Apr 2022 21:44:12 -0000 Subject: [Python-checkins] gh-87192: Update wording for fcntl 'Changed in' (GH-91658) Message-ID: https://github.com/python/cpython/commit/cfcda0acf27e40eab38c1c7c542362d872f6b403 commit: cfcda0acf27e40eab38c1c7c542362d872f6b403 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-30T14:44:08-07:00 summary: gh-87192: Update wording for fcntl 'Changed in' (GH-91658) (cherry picked from commit d7eb1ffbe8f913693e4c9ffa1b32edccac987ab6) Co-authored-by: slateny <46876382+slateny at users.noreply.github.com> files: M Doc/library/fcntl.rst diff --git a/Doc/library/fcntl.rst b/Doc/library/fcntl.rst index 9d8021150c42f..d9b579fc47d3d 100644 --- a/Doc/library/fcntl.rst +++ b/Doc/library/fcntl.rst @@ -37,7 +37,8 @@ descriptor. On macOS, the fcntl module exposes the ``F_GETPATH`` constant, which obtains the path of a file from a file descriptor. On Linux(>=3.15), the fcntl module exposes the ``F_OFD_GETLK``, ``F_OFD_SETLK`` - and ``F_OFD_SETLKW`` constants, which working with open file description locks. + and ``F_OFD_SETLKW`` constants, which are used when working with open file + description locks. .. versionchanged:: 3.10 On Linux >= 2.6.11, the fcntl module exposes the ``F_GETPIPE_SZ`` and From webhook-mailer at python.org Sat Apr 30 17:45:06 2022 From: webhook-mailer at python.org (miss-islington) Date: Sat, 30 Apr 2022 21:45:06 -0000 Subject: [Python-checkins] gh-87192: Update wording for fcntl 'Changed in' (GH-91658) Message-ID: https://github.com/python/cpython/commit/2830543247219427f46fa36a873eaa4b37813fa1 commit: 2830543247219427f46fa36a873eaa4b37813fa1 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-30T14:45:02-07:00 summary: gh-87192: Update wording for fcntl 'Changed in' (GH-91658) (cherry picked from commit d7eb1ffbe8f913693e4c9ffa1b32edccac987ab6) Co-authored-by: slateny <46876382+slateny at users.noreply.github.com> files: M Doc/library/fcntl.rst diff --git a/Doc/library/fcntl.rst b/Doc/library/fcntl.rst index 07a15d27216e9..f1c071ab991f7 100644 --- a/Doc/library/fcntl.rst +++ b/Doc/library/fcntl.rst @@ -37,7 +37,8 @@ descriptor. On macOS, the fcntl module exposes the ``F_GETPATH`` constant, which obtains the path of a file from a file descriptor. On Linux(>=3.15), the fcntl module exposes the ``F_OFD_GETLK``, ``F_OFD_SETLK`` - and ``F_OFD_SETLKW`` constants, which working with open file description locks. + and ``F_OFD_SETLKW`` constants, which are used when working with open file + description locks. The module defines the following functions: From webhook-mailer at python.org Sat Apr 30 17:51:58 2022 From: webhook-mailer at python.org (miss-islington) Date: Sat, 30 Apr 2022 21:51:58 -0000 Subject: [Python-checkins] gh-87801: Add run() to subprocess.CalledProcessError description (GH-91628) Message-ID: https://github.com/python/cpython/commit/335aae195093d89d11f4379d36c663233abead61 commit: 335aae195093d89d11f4379d36c663233abead61 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-30T14:51:45-07:00 summary: gh-87801: Add run() to subprocess.CalledProcessError description (GH-91628) (cherry picked from commit 567be058b403db9689af45bf831d4c732c8b1105) Co-authored-by: slateny <46876382+slateny at users.noreply.github.com> files: M Doc/library/subprocess.rst diff --git a/Doc/library/subprocess.rst b/Doc/library/subprocess.rst index ab9f1d88a0fc2..4d3312077bbd0 100644 --- a/Doc/library/subprocess.rst +++ b/Doc/library/subprocess.rst @@ -214,7 +214,9 @@ compatibility with older versions, see the :ref:`call-function-trio` section. .. exception:: CalledProcessError Subclass of :exc:`SubprocessError`, raised when a process run by - :func:`check_call` or :func:`check_output` returns a non-zero exit status. + :func:`check_call`, :func:`check_output`, or :func:`run` (with ``check=True``) + returns a non-zero exit status. + .. attribute:: returncode From webhook-mailer at python.org Sat Apr 30 17:52:17 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Sat, 30 Apr 2022 21:52:17 -0000 Subject: [Python-checkins] gh-91611: Use example.com for documentation, not mydomain.com (#91613) Message-ID: https://github.com/python/cpython/commit/ea392467829d6e93f824bde8eb87bdb31d9e4c62 commit: ea392467829d6e93f824bde8eb87bdb31d9e4c62 branch: main author: Motoki Naruse committer: JelleZijlstra date: 2022-04-30T15:52:13-06:00 summary: gh-91611: Use example.com for documentation, not mydomain.com (#91613) example.com is reserved by the IANA as special-use domain name for documentation purposes. The domain names are used widely in books, tutorials, sample network configurations, and generally as examples for the use of domain name. On the other hand, mydomain.com is real Domain Name Registration service. files: M Doc/library/secrets.rst M Doc/library/ssl.rst diff --git a/Doc/library/secrets.rst b/Doc/library/secrets.rst index afa8e2d385fa4..c22da727b55c9 100644 --- a/Doc/library/secrets.rst +++ b/Doc/library/secrets.rst @@ -193,7 +193,7 @@ suitable for password recovery applications: .. testcode:: import secrets - url = 'https://mydomain.com/reset=' + secrets.token_urlsafe() + url = 'https://example.com/reset=' + secrets.token_urlsafe() diff --git a/Doc/library/ssl.rst b/Doc/library/ssl.rst index 04818f824d105..54bd335282699 100644 --- a/Doc/library/ssl.rst +++ b/Doc/library/ssl.rst @@ -2357,7 +2357,7 @@ waiting for clients to connect:: context.load_cert_chain(certfile="mycertfile", keyfile="mykeyfile") bindsocket = socket.socket() - bindsocket.bind(('myaddr.mydomain.com', 10023)) + bindsocket.bind(('myaddr.example.com', 10023)) bindsocket.listen(5) When a client connects, you'll call :meth:`accept` on the socket to get the From webhook-mailer at python.org Sat Apr 30 17:55:07 2022 From: webhook-mailer at python.org (miss-islington) Date: Sat, 30 Apr 2022 21:55:07 -0000 Subject: [Python-checkins] gh-87801: Add run() to subprocess.CalledProcessError description (GH-91628) Message-ID: https://github.com/python/cpython/commit/2e84e97656cf4929bc887b9a212a12138382e807 commit: 2e84e97656cf4929bc887b9a212a12138382e807 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-30T14:54:58-07:00 summary: gh-87801: Add run() to subprocess.CalledProcessError description (GH-91628) (cherry picked from commit 567be058b403db9689af45bf831d4c732c8b1105) Co-authored-by: slateny <46876382+slateny at users.noreply.github.com> files: M Doc/library/subprocess.rst diff --git a/Doc/library/subprocess.rst b/Doc/library/subprocess.rst index f9e9f63f2151e..45eff7ffcc350 100644 --- a/Doc/library/subprocess.rst +++ b/Doc/library/subprocess.rst @@ -214,7 +214,9 @@ compatibility with older versions, see the :ref:`call-function-trio` section. .. exception:: CalledProcessError Subclass of :exc:`SubprocessError`, raised when a process run by - :func:`check_call` or :func:`check_output` returns a non-zero exit status. + :func:`check_call`, :func:`check_output`, or :func:`run` (with ``check=True``) + returns a non-zero exit status. + .. attribute:: returncode From webhook-mailer at python.org Sat Apr 30 18:10:01 2022 From: webhook-mailer at python.org (JelleZijlstra) Date: Sat, 30 Apr 2022 22:10:01 -0000 Subject: [Python-checkins] Fix typo in asyncio-extending.rst (#91609) Message-ID: https://github.com/python/cpython/commit/2b97d7f6dfb025c6ab8b522c26f07c621707731c commit: 2b97d7f6dfb025c6ab8b522c26f07c621707731c branch: main author: Yurii Karabas <1998uriyyo at gmail.com> committer: JelleZijlstra date: 2022-04-30T16:09:48-06:00 summary: Fix typo in asyncio-extending.rst (#91609) files: M Doc/library/asyncio-extending.rst diff --git a/Doc/library/asyncio-extending.rst b/Doc/library/asyncio-extending.rst index 215d215bb14fe..acbaa6f7faf74 100644 --- a/Doc/library/asyncio-extending.rst +++ b/Doc/library/asyncio-extending.rst @@ -63,7 +63,7 @@ For this purpose the following, *private* constructors are listed: *context* argument is added. -.. method:: Tasl._check_future(future) +.. method:: Task._check_future(future) Return ``True`` if *future* is attached to the same loop as the task, ``False`` otherwise. From webhook-mailer at python.org Sat Apr 30 18:12:43 2022 From: webhook-mailer at python.org (miss-islington) Date: Sat, 30 Apr 2022 22:12:43 -0000 Subject: [Python-checkins] gh-85757: Change wording from nested to inner (GH-91811) Message-ID: https://github.com/python/cpython/commit/efb87b1090a21e1be8f5f4928349eb742fd17f4c commit: efb87b1090a21e1be8f5f4928349eb742fd17f4c branch: main author: slateny <46876382+slateny at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-30T15:12:33-07:00 summary: gh-85757: Change wording from nested to inner (GH-91811) #85757 https://docs.python.org/3/tutorial/datastructures.html#nested-list-comprehensions I do think this is clearer, but I wonder if 'nested' should be kept though to get the terminology out there more often. So perhaps it could be something like 'inner (nested) listcomp' or 'nested (inner) listcomp' despite sounding a bit redundant Automerge-Triggered-By: GH:rhettinger files: M Doc/tutorial/datastructures.rst diff --git a/Doc/tutorial/datastructures.rst b/Doc/tutorial/datastructures.rst index 927a6722ca251..f847ee325e0d7 100644 --- a/Doc/tutorial/datastructures.rst +++ b/Doc/tutorial/datastructures.rst @@ -303,7 +303,7 @@ The following list comprehension will transpose rows and columns:: >>> [[row[i] for row in matrix] for i in range(4)] [[1, 5, 9], [2, 6, 10], [3, 7, 11], [4, 8, 12]] -As we saw in the previous section, the nested listcomp is evaluated in +As we saw in the previous section, the inner list comprehension is evaluated in the context of the :keyword:`for` that follows it, so this example is equivalent to:: From webhook-mailer at python.org Sat Apr 30 18:14:42 2022 From: webhook-mailer at python.org (miss-islington) Date: Sat, 30 Apr 2022 22:14:42 -0000 Subject: [Python-checkins] gh-89253: Add 3.10 whatsnew section for itertools.pairwise (GH-91563) Message-ID: https://github.com/python/cpython/commit/5dcfb916c765d825b2e8372281d71d32316b41cf commit: 5dcfb916c765d825b2e8372281d71d32316b41cf branch: main author: slateny <46876382+slateny at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-30T15:14:38-07:00 summary: gh-89253: Add 3.10 whatsnew section for itertools.pairwise (GH-91563) #89253 [`pairwise()`](https://docs.python.org/3/library/itertools.html#itertools.pairwise) already has the 'new in python3.10' Automerge-Triggered-By: GH:rhettinger files: M Doc/whatsnew/3.10.rst diff --git a/Doc/whatsnew/3.10.rst b/Doc/whatsnew/3.10.rst index dd01c88d8ae66..f93523a4c50a8 100644 --- a/Doc/whatsnew/3.10.rst +++ b/Doc/whatsnew/3.10.rst @@ -1223,6 +1223,12 @@ now call :func:`inspect.get_annotations` to retrieve annotations. This means also now un-stringize stringized annotations. (Contributed by Larry Hastings in :issue:`43817`.) +itertools +--------- + +Add :func:`itertools.pairwise()`. +(Contributed by Raymond Hettinger in :issue:`38200`.) + linecache --------- From webhook-mailer at python.org Sat Apr 30 18:20:38 2022 From: webhook-mailer at python.org (miss-islington) Date: Sat, 30 Apr 2022 22:20:38 -0000 Subject: [Python-checkins] Improving readability of argparse.rst (GH-91751) Message-ID: https://github.com/python/cpython/commit/238aa6253bcd89a4eff50513cd72c07054e47d8e commit: 238aa6253bcd89a4eff50513cd72c07054e47d8e branch: main author: Antonio Andrade committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-30T15:20:10-07:00 summary: Improving readability of argparse.rst (GH-91751) Fixed few typos and enhanced few paragraphs for your review and consideration. Trivial contribution towards continuous improvement, so no issue was raised. Automerge-Triggered-By: GH:rhettinger files: M Doc/library/argparse.rst diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst index 17570efb27081..5d269cc4eac4a 100644 --- a/Doc/library/argparse.rst +++ b/Doc/library/argparse.rst @@ -22,8 +22,8 @@ The :mod:`argparse` module makes it easy to write user-friendly command-line interfaces. The program defines what arguments it requires, and :mod:`argparse` will figure out how to parse those out of :data:`sys.argv`. The :mod:`argparse` -module also automatically generates help and usage messages and issues errors -when users give the program invalid arguments. +module also automatically generates help and usage messages. The module +will also issue errors when users give the program invalid arguments. Core Functionality @@ -91,8 +91,8 @@ produces either the sum or the max:: args = parser.parse_args() print(args.accumulate(args.integers)) -Assuming the Python code above is saved into a file called ``prog.py``, it can -be run at the command line and provides useful help messages: +Assuming the above Python code is saved into a file called ``prog.py``, it can +be run at the command line and it provides useful help messages: .. code-block:: shell-session @@ -119,7 +119,7 @@ the command-line integers: $ python prog.py 1 2 3 4 --sum 10 -If invalid arguments are passed in, it will issue an error: +If invalid arguments are passed in, an error will be displayed: .. code-block:: shell-session @@ -159,7 +159,7 @@ used when :meth:`~ArgumentParser.parse_args` is called. For example:: Later, calling :meth:`~ArgumentParser.parse_args` will return an object with two attributes, ``integers`` and ``accumulate``. The ``integers`` attribute -will be a list of one or more ints, and the ``accumulate`` attribute will be +will be a list of one or more integers, and the ``accumulate`` attribute will be either the :func:`sum` function, if ``--sum`` was specified at the command line, or the :func:`max` function if it was not. @@ -555,7 +555,7 @@ disallowed. fromfile_prefix_chars ^^^^^^^^^^^^^^^^^^^^^ -Sometimes, for example when dealing with a particularly long argument lists, it +Sometimes, when dealing with a particularly long argument lists, it may make sense to keep the list of arguments in a file rather than typing it out at the command line. If the ``fromfile_prefix_chars=`` argument is given to the :class:`ArgumentParser` constructor, then arguments that start with any of the @@ -1228,7 +1228,7 @@ Any container can be passed as the *choices* value, so :class:`list` objects, Use of :class:`enum.Enum` is not recommended because it is difficult to control its appearance in usage, help, and error messages. -Formatted choices overrides the default *metavar* which is normally derived +Formatted choices override the default *metavar* which is normally derived from *dest*. This is usually what you want because the user never sees the *dest* parameter. If this display isn't desirable (perhaps because there are many choices), just specify an explicit metavar_. From webhook-mailer at python.org Sat Apr 30 18:29:22 2022 From: webhook-mailer at python.org (miss-islington) Date: Sat, 30 Apr 2022 22:29:22 -0000 Subject: [Python-checkins] gh-89253: Add 3.10 whatsnew section for itertools.pairwise (GH-91563) Message-ID: https://github.com/python/cpython/commit/8b6e5d38636807850e6b9579a98117b71b116e97 commit: 8b6e5d38636807850e6b9579a98117b71b116e97 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2022-04-30T15:29:09-07:00 summary: gh-89253: Add 3.10 whatsnew section for itertools.pairwise (GH-91563) GH-89253 [`pairwise()`](https://docs.python.org/3/library/itertools.htmlGH-itertools.pairwise) already has the 'new in python3.10' Automerge-Triggered-By: GH:rhettinger (cherry picked from commit 5dcfb916c765d825b2e8372281d71d32316b41cf) Co-authored-by: slateny <46876382+slateny at users.noreply.github.com> files: M Doc/whatsnew/3.10.rst diff --git a/Doc/whatsnew/3.10.rst b/Doc/whatsnew/3.10.rst index 5788e32da1c92..55e7d62b54159 100644 --- a/Doc/whatsnew/3.10.rst +++ b/Doc/whatsnew/3.10.rst @@ -1212,6 +1212,12 @@ now call :func:`inspect.get_annotations` to retrieve annotations. This means also now un-stringize stringized annotations. (Contributed by Larry Hastings in :issue:`43817`.) +itertools +--------- + +Add :func:`itertools.pairwise()`. +(Contributed by Raymond Hettinger in :issue:`38200`.) + linecache --------- From webhook-mailer at python.org Sat Apr 30 20:38:23 2022 From: webhook-mailer at python.org (sweeneyde) Date: Sun, 01 May 2022 00:38:23 -0000 Subject: [Python-checkins] gh-91954: Use shell=True in test_subprocess.test_encoding_warning (GH-92090) Message-ID: https://github.com/python/cpython/commit/b9636180b32ea483f12564657ba1c3eb5b6d73d8 commit: b9636180b32ea483f12564657ba1c3eb5b6d73d8 branch: main author: Dennis Sweeney <36520290+sweeneyde at users.noreply.github.com> committer: sweeneyde <36520290+sweeneyde at users.noreply.github.com> date: 2022-04-30T20:38:19-04:00 summary: gh-91954: Use shell=True in test_subprocess.test_encoding_warning (GH-92090) files: M Lib/test/test_subprocess.py diff --git a/Lib/test/test_subprocess.py b/Lib/test/test_subprocess.py index 5814a6d924e12..0764120952ebd 100644 --- a/Lib/test/test_subprocess.py +++ b/Lib/test/test_subprocess.py @@ -1736,16 +1736,15 @@ def test_run_with_shell_timeout_and_capture_output(self): def test_encoding_warning(self): code = textwrap.dedent("""\ from subprocess import * - args = ["echo", "hello"] - run(args, text=True) - check_output(args, text=True) + run("echo hello", shell=True, text=True) + check_output("echo hello", shell=True, text=True) """) cp = subprocess.run([sys.executable, "-Xwarn_default_encoding", "-c", code], capture_output=True) lines = cp.stderr.splitlines() - self.assertEqual(len(lines), 2) - self.assertTrue(lines[0].startswith(b":3: EncodingWarning: ")) - self.assertTrue(lines[1].startswith(b":4: EncodingWarning: ")) + self.assertEqual(len(lines), 2, lines) + self.assertTrue(lines[0].startswith(b":2: EncodingWarning: ")) + self.assertTrue(lines[1].startswith(b":3: EncodingWarning: ")) def _get_test_grp_name(): From webhook-mailer at python.org Sat Apr 30 21:44:52 2022 From: webhook-mailer at python.org (methane) Date: Sun, 01 May 2022 01:44:52 -0000 Subject: [Python-checkins] gh-91952: Make TextIOWrapper.reconfigure() supports "locale" encoding (GH-91982) Message-ID: https://github.com/python/cpython/commit/0729b31a8b9ac35ef9b79fdc5aed22cccec9ba16 commit: 0729b31a8b9ac35ef9b79fdc5aed22cccec9ba16 branch: main author: Inada Naoki committer: methane date: 2022-05-01T10:44:14+09:00 summary: gh-91952: Make TextIOWrapper.reconfigure() supports "locale" encoding (GH-91982) files: A Misc/NEWS.d/next/Library/2022-04-27-18-04-24.gh-issue-91952.9A4RXx.rst M Doc/library/io.rst M Lib/_pyio.py M Modules/_io/textio.c diff --git a/Doc/library/io.rst b/Doc/library/io.rst index ecd8c8043351d..d8e7b1621e2ea 100644 --- a/Doc/library/io.rst +++ b/Doc/library/io.rst @@ -1038,6 +1038,9 @@ Text I/O .. versionadded:: 3.7 + .. versionchanged:: 3.11 + The method supports ``encoding="locale"`` option. + .. class:: StringIO(initial_value='', newline='\\n') diff --git a/Lib/_pyio.py b/Lib/_pyio.py index 380a7a736c64e..0f647eed99d81 100644 --- a/Lib/_pyio.py +++ b/Lib/_pyio.py @@ -2161,6 +2161,8 @@ def reconfigure(self, *, else: if not isinstance(encoding, str): raise TypeError("invalid encoding: %r" % encoding) + if encoding == "locale": + encoding = locale.getencoding() if newline is Ellipsis: newline = self._readnl diff --git a/Misc/NEWS.d/next/Library/2022-04-27-18-04-24.gh-issue-91952.9A4RXx.rst b/Misc/NEWS.d/next/Library/2022-04-27-18-04-24.gh-issue-91952.9A4RXx.rst new file mode 100644 index 0000000000000..a0b48d16fe866 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-04-27-18-04-24.gh-issue-91952.9A4RXx.rst @@ -0,0 +1 @@ +Add ``encoding="locale"`` support to :meth:`TextIOWrapper.reconfigure`. diff --git a/Modules/_io/textio.c b/Modules/_io/textio.c index f1cd6d01da859..3cbaca3ef4606 100644 --- a/Modules/_io/textio.c +++ b/Modules/_io/textio.c @@ -1248,8 +1248,16 @@ textiowrapper_change_encoding(textio *self, PyObject *encoding, errors = self->errors; } } - else if (errors == Py_None) { - errors = &_Py_ID(strict); + else { + if (_PyUnicode_EqualToASCIIString(encoding, "locale")) { + encoding = _Py_GetLocaleEncodingObject(); + if (encoding == NULL) { + return -1; + } + } + if (errors == Py_None) { + errors = &_Py_ID(strict); + } } const char *c_errors = PyUnicode_AsUTF8(errors);