[Python-ideas] Change how Generator Expressions handle StopIteration

[Akira Li]

Guido van Rossum <guido at python.org> writes:

> On Sun, Nov 2, 2014 at 8:02 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
>> However, I also see potentially significant backwards compatibility
>> problems when it comes to helper functions that throw StopIteration to
>> terminate the calling generator - there would likely need to be some
>> kind of thread local state and a helper "iterexit()" builtin and
>> "PyIter_Exit()" C API to call instead of raising StopIteration
>> directly.
>>
>
> That's what I was afraid of. Can you point me to an example of code that
> depends on this that isn't trivial like Andrew Barnert's ensure_positive()
> example? I think that particular example, and the category it represents,
> are excessive cleverness that abuse the feature under discussion -- but you
> sound like you have helpers for context managers that couldn't be easily
> dismissed like that.
>
>

The pure Python implementation of itertools.groupby() provided in its
docs [1] uses next() as the helper function that terminates the calling
generator by raising StopIteration

[1]: https://docs.python.org/3/library/itertools.html#itertools.groupby

Here's a simplified example:

  from functools import partial
  
  def groupby(iterable):
      """
      >>> ' '.join(k for k, g in groupby('AAAABBBCCDAABBB'))
      'A B C D A B'
      >>> ' '.join(''.join(g) for k, g in groupby('AAAABBBCCDAABBB'))
      'AAAA BBB CC D AA BBB'
      """
      next_value = partial(next, iter(iterable))
      def yield_same(group_value_): # generate group values
          nonlocal value
          while value == group_value_:
              yield value
              value = next_value() # exit on StopIteration
  
      group_value = value = object()
      while True:
          while value == group_value: # discard unconsumed values
              value = next_value() # exit on StopIteration
  
          group_value = value
          yield group_value, yield_same(group_value)
  
The alternative is to return a sentinel from next():

  def groupby_done(iterable):
      done = object() # sentinel
      next_value = partial(next, iter(iterable), done)
      def yield_same(group_value_): # generate group values
          nonlocal value
          while value == group_value_:
              yield value
              value = next_value()
              if value is done:
                  return
  
      group_value = value = object()
      while value is not done:
          while value == group_value: # discard unconsumed values
              value = next_value()
              if value is done:
                  return
          group_value = value
          yield group_value, yield_same(group_value)
  
The first code example exploits the fact that `next(it)` continues to
raise StopIteration on subsequent calls.  The second code example has to
propagate up the stack the termination condition manually (`while True`
is replace with `while value is not done`).


--
Akira