[Python-Dev] cpython: Implement PEP 380 - 'yield from' (closes #11682)
Georg Brandl
g.brandl at gmx.net
Fri Jan 13 16:17:09 CET 2012
Caution, long review ahead.
On 01/13/2012 12:43 PM, nick.coghlan wrote:
> http://hg.python.org/cpython/rev/d64ac9ab4cd0
> changeset: 74356:d64ac9ab4cd0
> user: Nick Coghlan <ncoghlan at gmail.com>
> date: Fri Jan 13 21:43:40 2012 +1000
> summary:
> Implement PEP 380 - 'yield from' (closes #11682)
> diff --git a/Doc/reference/expressions.rst b/Doc/reference/expressions.rst
> --- a/Doc/reference/expressions.rst
> +++ b/Doc/reference/expressions.rst
> @@ -318,7 +318,7 @@
There should probably be a "versionadded" somewhere on this page.
> .. productionlist::
> yield_atom: "(" `yield_expression` ")"
> - yield_expression: "yield" [`expression_list`]
> + yield_expression: "yield" [`expression_list` | "from" `expression`]
>
> The :keyword:`yield` expression is only used when defining a generator function,
> and can only be used in the body of a function definition. Using a
> @@ -336,7 +336,10 @@
> the generator's methods, the function can proceed exactly as if the
> :keyword:`yield` expression was just another external call. The value of the
> :keyword:`yield` expression after resuming depends on the method which resumed
> -the execution.
> +the execution. If :meth:`__next__` is used (typically via either a
> +:keyword:`for` or the :func:`next` builtin) then the result is :const:`None`,
> +otherwise, if :meth:`send` is used, then the result will be the value passed
> +in to that method.
>
> .. index:: single: coroutine
>
> @@ -346,12 +349,29 @@
> where should the execution continue after it yields; the control is always
> transferred to the generator's caller.
>
> -The :keyword:`yield` statement is allowed in the :keyword:`try` clause of a
> +:keyword:`yield` expressions are allowed in the :keyword:`try` clause of a
> :keyword:`try` ... :keyword:`finally` construct. If the generator is not
> resumed before it is finalized (by reaching a zero reference count or by being
> garbage collected), the generator-iterator's :meth:`close` method will be
> called, allowing any pending :keyword:`finally` clauses to execute.
>
> +When ``yield from expression`` is used, it treats the supplied expression as
> +a subiterator. All values produced by that subiterator are passed directly
> +to the caller of the current generator's methods. Any values passed in with
> +:meth:`send` and any exceptions passed in with :meth:`throw` are passed to
> +the underlying iterator if it has the appropriate methods. If this is not the
> +case, then :meth:`send` will raise :exc:`AttributeError` or :exc:`TypeError`,
> +while :meth:`throw` will just raise the passed in exception immediately.
> +
> +When the underlying iterator is complete, the :attr:`~StopIteration.value`
> +attribute of the raised :exc:`StopIteration` instance becomes the value of
> +the yield expression. It can be either set explicitly when raising
> +:exc:`StopIteration`, or automatically when the sub-iterator is a generator
> +(by returning a value from the sub-generator).
> +
> +The parentheses can be omitted when the :keyword:`yield` expression is the
> +sole expression on the right hand side of an assignment statement.
> +
> .. index:: object: generator
>
> The following generator's methods can be used to control the execution of a
> @@ -444,6 +464,10 @@
> The proposal to enhance the API and syntax of generators, making them
> usable as simple coroutines.
>
> + :pep:`0380` - Syntax for Delegating to a Subgenerator
> + The proposal to introduce the :token:`yield_from` syntax, making delegation
> + to sub-generators easy.
> +
>
> .. _primaries:
>
> PEP 3155: Qualified name for classes and functions
> ==================================================
>
> @@ -208,7 +224,6 @@
> how they might be accessible from the global scope.
>
> Example with (non-bound) methods::
> -
> >>> class C:
> ... def meth(self):
> ... pass
This looks like a spurious (and syntax-breaking) change.
> diff --git a/Grammar/Grammar b/Grammar/Grammar
> --- a/Grammar/Grammar
> +++ b/Grammar/Grammar
> @@ -121,7 +121,7 @@
> |'**' test)
> # The reason that keywords are test nodes instead of NAME is that using NAME
> # results in an ambiguity. ast.c makes sure it's a NAME.
> -argument: test [comp_for] | test '=' test # Really [keyword '='] test
> +argument: (test) [comp_for] | test '=' test # Really [keyword '='] test
This looks like a change without effect?
> diff --git a/Include/genobject.h b/Include/genobject.h
> --- a/Include/genobject.h
> +++ b/Include/genobject.h
> @@ -11,20 +11,20 @@
> struct _frame; /* Avoid including frameobject.h */
>
> typedef struct {
> - PyObject_HEAD
> - /* The gi_ prefix is intended to remind of generator-iterator. */
> + PyObject_HEAD
> + /* The gi_ prefix is intended to remind of generator-iterator. */
>
> - /* Note: gi_frame can be NULL if the generator is "finished" */
> - struct _frame *gi_frame;
> + /* Note: gi_frame can be NULL if the generator is "finished" */
> + struct _frame *gi_frame;
>
> - /* True if generator is being executed. */
> - int gi_running;
> + /* True if generator is being executed. */
> + int gi_running;
>
> - /* The code object backing the generator */
> - PyObject *gi_code;
> + /* The code object backing the generator */
> + PyObject *gi_code;
>
> - /* List of weak reference. */
> - PyObject *gi_weakreflist;
> + /* List of weak reference. */
> + PyObject *gi_weakreflist;
> } PyGenObject;
While these change tabs into spaces, it should be 4 spaces, not 8.
> @@ -34,6 +34,7 @@
>
> PyAPI_FUNC(PyObject *) PyGen_New(struct _frame *);
> PyAPI_FUNC(int) PyGen_NeedsFinalizing(PyGenObject *);
> +PyAPI_FUNC(int) PyGen_FetchStopIterationValue(PyObject **);
>
> #ifdef __cplusplus
> }
Does this API need to be public? If yes, it needs to be documented.
> diff --git a/Include/opcode.h b/Include/opcode.h
> --- a/Include/opcode.h
> +++ b/Include/opcode.h
> @@ -7,116 +7,117 @@
>
> /* Instruction opcodes for compiled code */
>
> -#define POP_TOP 1
> -#define ROT_TWO 2
> -#define ROT_THREE 3
> -#define DUP_TOP 4
> +#define POP_TOP 1
> +#define ROT_TWO 2
> +#define ROT_THREE 3
> +#define DUP_TOP 4
> #define DUP_TOP_TWO 5
> -#define NOP 9
> +#define NOP 9
>
> -#define UNARY_POSITIVE 10
> -#define UNARY_NEGATIVE 11
> -#define UNARY_NOT 12
> +#define UNARY_POSITIVE 10
> +#define UNARY_NEGATIVE 11
> +#define UNARY_NOT 12
>
> -#define UNARY_INVERT 15
> +#define UNARY_INVERT 15
>
> -#define BINARY_POWER 19
> +#define BINARY_POWER 19
>
> -#define BINARY_MULTIPLY 20
> +#define BINARY_MULTIPLY 20
>
> -#define BINARY_MODULO 22
> -#define BINARY_ADD 23
> -#define BINARY_SUBTRACT 24
> -#define BINARY_SUBSCR 25
> +#define BINARY_MODULO 22
> +#define BINARY_ADD 23
> +#define BINARY_SUBTRACT 24
> +#define BINARY_SUBSCR 25
> #define BINARY_FLOOR_DIVIDE 26
> #define BINARY_TRUE_DIVIDE 27
> #define INPLACE_FLOOR_DIVIDE 28
> #define INPLACE_TRUE_DIVIDE 29
>
> -#define STORE_MAP 54
> -#define INPLACE_ADD 55
> -#define INPLACE_SUBTRACT 56
> -#define INPLACE_MULTIPLY 57
> +#define STORE_MAP 54
> +#define INPLACE_ADD 55
> +#define INPLACE_SUBTRACT 56
> +#define INPLACE_MULTIPLY 57
>
> -#define INPLACE_MODULO 59
> -#define STORE_SUBSCR 60
> -#define DELETE_SUBSCR 61
> +#define INPLACE_MODULO 59
> +#define STORE_SUBSCR 60
> +#define DELETE_SUBSCR 61
>
> -#define BINARY_LSHIFT 62
> -#define BINARY_RSHIFT 63
> -#define BINARY_AND 64
> -#define BINARY_XOR 65
> -#define BINARY_OR 66
> -#define INPLACE_POWER 67
> -#define GET_ITER 68
> -#define STORE_LOCALS 69
> -#define PRINT_EXPR 70
> +#define BINARY_LSHIFT 62
> +#define BINARY_RSHIFT 63
> +#define BINARY_AND 64
> +#define BINARY_XOR 65
> +#define BINARY_OR 66
> +#define INPLACE_POWER 67
> +#define GET_ITER 68
> +#define STORE_LOCALS 69
> +#define PRINT_EXPR 70
> #define LOAD_BUILD_CLASS 71
> +#define YIELD_FROM 72
>
> -#define INPLACE_LSHIFT 75
> -#define INPLACE_RSHIFT 76
> -#define INPLACE_AND 77
> -#define INPLACE_XOR 78
> -#define INPLACE_OR 79
> -#define BREAK_LOOP 80
> +#define INPLACE_LSHIFT 75
> +#define INPLACE_RSHIFT 76
> +#define INPLACE_AND 77
> +#define INPLACE_XOR 78
> +#define INPLACE_OR 79
> +#define BREAK_LOOP 80
> #define WITH_CLEANUP 81
>
> -#define RETURN_VALUE 83
> -#define IMPORT_STAR 84
> +#define RETURN_VALUE 83
> +#define IMPORT_STAR 84
>
> -#define YIELD_VALUE 86
> -#define POP_BLOCK 87
> -#define END_FINALLY 88
> -#define POP_EXCEPT 89
> +#define YIELD_VALUE 86
> +#define POP_BLOCK 87
> +#define END_FINALLY 88
> +#define POP_EXCEPT 89
>
> -#define HAVE_ARGUMENT 90 /* Opcodes from here have an argument: */
> +#define HAVE_ARGUMENT 90 /* Opcodes from here have an argument: */
>
> -#define STORE_NAME 90 /* Index in name list */
> -#define DELETE_NAME 91 /* "" */
> -#define UNPACK_SEQUENCE 92 /* Number of sequence items */
> -#define FOR_ITER 93
> +#define STORE_NAME 90 /* Index in name list */
> +#define DELETE_NAME 91 /* "" */
> +#define UNPACK_SEQUENCE 92 /* Number of sequence items */
> +#define FOR_ITER 93
> #define UNPACK_EX 94 /* Num items before variable part +
> (Num items after variable part << 8) */
>
> -#define STORE_ATTR 95 /* Index in name list */
> -#define DELETE_ATTR 96 /* "" */
> -#define STORE_GLOBAL 97 /* "" */
> -#define DELETE_GLOBAL 98 /* "" */
> +#define STORE_ATTR 95 /* Index in name list */
> +#define DELETE_ATTR 96 /* "" */
> +#define STORE_GLOBAL 97 /* "" */
> +#define DELETE_GLOBAL 98 /* "" */
>
> -#define LOAD_CONST 100 /* Index in const list */
> -#define LOAD_NAME 101 /* Index in name list */
> -#define BUILD_TUPLE 102 /* Number of tuple items */
> -#define BUILD_LIST 103 /* Number of list items */
> -#define BUILD_SET 104 /* Number of set items */
> -#define BUILD_MAP 105 /* Always zero for now */
> -#define LOAD_ATTR 106 /* Index in name list */
> -#define COMPARE_OP 107 /* Comparison operator */
> -#define IMPORT_NAME 108 /* Index in name list */
> -#define IMPORT_FROM 109 /* Index in name list */
> +#define LOAD_CONST 100 /* Index in const list */
> +#define LOAD_NAME 101 /* Index in name list */
> +#define BUILD_TUPLE 102 /* Number of tuple items */
> +#define BUILD_LIST 103 /* Number of list items */
> +#define BUILD_SET 104 /* Number of set items */
> +#define BUILD_MAP 105 /* Always zero for now */
> +#define LOAD_ATTR 106 /* Index in name list */
> +#define COMPARE_OP 107 /* Comparison operator */
> +#define IMPORT_NAME 108 /* Index in name list */
> +#define IMPORT_FROM 109 /* Index in name list */
>
> -#define JUMP_FORWARD 110 /* Number of bytes to skip */
> -#define JUMP_IF_FALSE_OR_POP 111 /* Target byte offset from beginning of code */
> -#define JUMP_IF_TRUE_OR_POP 112 /* "" */
> -#define JUMP_ABSOLUTE 113 /* "" */
> -#define POP_JUMP_IF_FALSE 114 /* "" */
> -#define POP_JUMP_IF_TRUE 115 /* "" */
> +#define JUMP_FORWARD 110 /* Number of bytes to skip */
> +#define JUMP_IF_FALSE_OR_POP 111 /* Target byte offset from beginning of code */
> +#define JUMP_IF_TRUE_OR_POP 112 /* "" */
> +#define JUMP_ABSOLUTE 113 /* "" */
> +#define POP_JUMP_IF_FALSE 114 /* "" */
> +#define POP_JUMP_IF_TRUE 115 /* "" */
>
> -#define LOAD_GLOBAL 116 /* Index in name list */
> +#define LOAD_GLOBAL 116 /* Index in name list */
>
> -#define CONTINUE_LOOP 119 /* Start of loop (absolute) */
> -#define SETUP_LOOP 120 /* Target address (relative) */
> -#define SETUP_EXCEPT 121 /* "" */
> -#define SETUP_FINALLY 122 /* "" */
> +#define CONTINUE_LOOP 119 /* Start of loop (absolute) */
> +#define SETUP_LOOP 120 /* Target address (relative) */
> +#define SETUP_EXCEPT 121 /* "" */
> +#define SETUP_FINALLY 122 /* "" */
>
> -#define LOAD_FAST 124 /* Local variable number */
> -#define STORE_FAST 125 /* Local variable number */
> -#define DELETE_FAST 126 /* Local variable number */
> +#define LOAD_FAST 124 /* Local variable number */
> +#define STORE_FAST 125 /* Local variable number */
> +#define DELETE_FAST 126 /* Local variable number */
>
> -#define RAISE_VARARGS 130 /* Number of raise arguments (1, 2 or 3) */
> +#define RAISE_VARARGS 130 /* Number of raise arguments (1, 2 or 3) */
> /* CALL_FUNCTION_XXX opcodes defined below depend on this definition */
> -#define CALL_FUNCTION 131 /* #args + (#kwargs<<8) */
> -#define MAKE_FUNCTION 132 /* #defaults + #kwdefaults<<8 + #annotations<<16 */
> -#define BUILD_SLICE 133 /* Number of items */
> +#define CALL_FUNCTION 131 /* #args + (#kwargs<<8) */
> +#define MAKE_FUNCTION 132 /* #defaults + #kwdefaults<<8 + #annotations<<16 */
> +#define BUILD_SLICE 133 /* Number of items */
>
> #define MAKE_CLOSURE 134 /* same as MAKE_FUNCTION */
> #define LOAD_CLOSURE 135 /* Load free variable from closure */
Not sure putting these and all the other cosmetic changes into an already
big patch is such a good idea...
> diff --git a/Include/pyerrors.h b/Include/pyerrors.h
> --- a/Include/pyerrors.h
> +++ b/Include/pyerrors.h
> @@ -51,6 +51,11 @@
> Py_ssize_t written; /* only for BlockingIOError, -1 otherwise */
> } PyOSErrorObject;
>
> +typedef struct {
> + PyException_HEAD
> + PyObject *value;
> +} PyStopIterationObject;
> +
> /* Compatibility typedefs */
> typedef PyOSErrorObject PyEnvironmentErrorObject;
> #ifdef MS_WINDOWS
> @@ -380,6 +385,8 @@
> const char *reason /* UTF-8 encoded string */
> );
>
> +/* create a StopIteration exception with the given value */
> +PyAPI_FUNC(PyObject *) PyStopIteration_Create(PyObject *);
About this API see below.
> diff --git a/Objects/abstract.c b/Objects/abstract.c
> --- a/Objects/abstract.c
> +++ b/Objects/abstract.c
> @@ -2267,7 +2267,6 @@
>
> func = PyObject_GetAttrString(o, name);
> if (func == NULL) {
> - PyErr_SetString(PyExc_AttributeError, name);
> return 0;
> }
>
> @@ -2311,7 +2310,6 @@
>
> func = PyObject_GetAttrString(o, name);
> if (func == NULL) {
> - PyErr_SetString(PyExc_AttributeError, name);
> return 0;
> }
> va_start(va, format);
These two changes also look suspiciously unrelated?
> +PyObject *
> +PyStopIteration_Create(PyObject *value)
> +{
> + return PyObject_CallFunctionObjArgs(PyExc_StopIteration, value, NULL);
> +}
I think this function is rather questionable. It is only used once at all.
If kept, it should rather be named _PyE{rr,xc}_CreateStopIteration. But since
it's so trivial, it should be removed altogether.
> diff --git a/Objects/genobject.c b/Objects/genobject.c
> --- a/Objects/genobject.c
> +++ b/Objects/genobject.c
> @@ -5,6 +5,9 @@
> #include "structmember.h"
> #include "opcode.h"
>
> +static PyObject *gen_close(PyGenObject *gen, PyObject *args);
> +static void gen_undelegate(PyGenObject *gen);
> +
> static int
> gen_traverse(PyGenObject *gen, visitproc visit, void *arg)
> {
> @@ -90,12 +93,18 @@
>
> /* If the generator just returned (as opposed to yielding), signal
> * that the generator is exhausted. */
> - if (result == Py_None && f->f_stacktop == NULL) {
> - Py_DECREF(result);
> - result = NULL;
> - /* Set exception if not called by gen_iternext() */
> - if (arg)
> + if (result && f->f_stacktop == NULL) {
> + if (result == Py_None) {
> + /* Delay exception instantiation if we can */
> PyErr_SetNone(PyExc_StopIteration);
> + } else {
> + PyObject *e = PyStopIteration_Create(result);
> + if (e != NULL) {
> + PyErr_SetObject(PyExc_StopIteration, e);
> + Py_DECREF(e);
> + }
Wouldn't PyErr_SetObject(PyExc_StopIteration, value) suffice here
anyway?
> +/*
> + * If StopIteration exception is set, fetches its 'value'
> + * attribute if any, otherwise sets pvalue to None.
> + *
> + * Returns 0 if no exception or StopIteration is set.
> + * If any other exception is set, returns -1 and leaves
> + * pvalue unchanged.
> + */
> +
> +int
> +PyGen_FetchStopIterationValue(PyObject **pvalue) {
> + PyObject *et, *ev, *tb;
> + PyObject *value = NULL;
> +
> + if (PyErr_ExceptionMatches(PyExc_StopIteration)) {
> + PyErr_Fetch(&et, &ev, &tb);
> + Py_XDECREF(et);
> + Py_XDECREF(tb);
> + if (ev) {
> + value = ((PyStopIterationObject *)ev)->value;
> + Py_DECREF(ev);
> + }
PyErr_Fetch without PyErr_Restore clears the exception, that should be
mentioned in the docstring.
Georg
More information about the Python-Dev
mailing list