[Python-ideas] PEP 479: Change StopIteration handling inside generators

[Chris Angelico]

On Mon, Nov 17, 2014 at 11:03 AM, Steven D'Aprano <steve at pearwood.info> wrote:
> On Mon, Nov 17, 2014 at 11:05:01AM +1300, Greg Ewing wrote:
>> Chris Angelico wrote:
>> >if you call on someone else's generator,
>> >and that someone hasn't applied the __future__ directive, you'll be in
>> >the current situation of not being able to distinguish 'return' from
>> >'raise StopIteration'. But for your own generators, you can guarantee
>> >that they're distinct.
>>
>> This suggests that the use of a __future__ directive is not
>> really appropriate, since it can affect code outside of the
>> module with the __future__ directive in it.
>
> I don't see how that is different from any other __future__ directive.
> They are all per-module, and if you gain access to an object from
> another module, it will behave as specified in the module that created
> it, not the module that imported it. How is this different?

Well, let's see. For feature in sorted(__future__.all_feature_names):

absolute_import: Affects implementation of a keyword
barry_as_FLUFL: Not entirely sure what this one actually accomplishes. :)
division: Changes the meaning of one operator.
generators: Introduces a keyword
nested_scopes: Alters the compilation of source to byte-code(?)
print_function: Removes a keyword
unicode_literals: Alters the type used for literals
with_statement: Introduces a keyword

Apart from the joke, it seems that every __future__ directive is there
to affect the compilation, not execution, of its module: that is, once
a module has been compiled to .pyc, it shouldn't matter whether it
used __future__ or not. Regardless of unicode_literals, you can create
bytes literals with b'asdf' and unicode literals with u'asdf'. I'm not
entirely sure about division (can you call on true-division without
the future directive?), but in any case, it's all done at compilation
time, as can be seen interactively:

Python 2.7.3 (default, Mar 13 2014, 11:03:55)
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> def division1(x,y):
...     return x/y, x//y, x%y
...
>>> from __future__ import division
>>> def division2(x,y):
...     return x/y, x//y, x%y
...
>>> import dis
>>> dis.dis(division1)
  2           0 LOAD_FAST                0 (x)
              3 LOAD_FAST                1 (y)
              6 BINARY_DIVIDE
              7 LOAD_FAST                0 (x)
             10 LOAD_FAST                1 (y)
             13 BINARY_FLOOR_DIVIDE
             14 LOAD_FAST                0 (x)
             17 LOAD_FAST                1 (y)
             20 BINARY_MODULO
             21 BUILD_TUPLE              3
             24 RETURN_VALUE
>>> dis.dis(division2)
  2           0 LOAD_FAST                0 (x)
              3 LOAD_FAST                1 (y)
              6 BINARY_TRUE_DIVIDE
              7 LOAD_FAST                0 (x)
             10 LOAD_FAST                1 (y)
             13 BINARY_FLOOR_DIVIDE
             14 LOAD_FAST                0 (x)
             17 LOAD_FAST                1 (y)
             20 BINARY_MODULO
             21 BUILD_TUPLE              3
             24 RETURN_VALUE

So to make this consistent with all other __future__ directives, there
would need to be some kind of safe way to define this: perhaps an
attribute on the generator object. Something like this:

>>> def gen(x):
...     yield x
...     raise StopIteration(42)
...
>>> g=gen(123)
>>> list(g)
[123]
>>> gen.__distinguish_returns__ = True
>>> g=gen(123)
>>> list(g)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in gen
StopIteration: 42

The attribute on the function would be what affects behaviour; the
__future__ directive applies that attribute to all generator functions
in its module (including genexprs). Once the __future__ directive
becomes automatic, the attribute can and will be dropped - any code
which interrogates it MUST be prepared to stop interrogating it once
the feature applies to all modules.

Does that sound reasonable? Should it be added to the PEP?

ChrisA