[Python-ideas] Saving state in list/generator comprehension

[Andrew Carter]

Forgive me for any problems in this e-mail as I'm new to this mailing list.

I thought it might be nice to be able to somehow save a state in
list/generator comprehensions,
a side effect of this (although not the intended goal) is it would make
reduce feasible in a clean manner as the final result would just be the
state.

One mechanism I can think of is to overload the with/as keyword for use
inside of list/generator comprehensions, and using the previous result as
state
I believe the change to the grammar in python3k would be

comp_iter : comp_for | comp_if | comp_with
comp_with: 'with' testlist 'as' testlist

So something in the form of
  [expr for i in iterable with initializer as accumulator]
would resolve to something like
  result = []
  accumulator = initializer
  for i in iterable:
    accumulator = expr
    result.append(accumulator)
  return result

For instance reduce could be defined as (assuming all 3 arguments are
required)
  reduce = lambda function, iterable, initializer : ([initializer] +
[function(accumulator, i) for i in iterable with initializer as
accumulator])[-1]
Breaking this down, the "with initializer as accumulator" statement means
that when the list comprehension begins accumulator=initializer,
then after each iteration, accumulator = function(accumulator, i), so with
the function f, list [i1,i2,i3,...], and initial value i0, the resulting
list of
"[function(accumulator, i) for i in iterable with initializer as
accumulator]" would be [f(i0,i1), f(f(i0,i1),i2),
f(f(f(i0,i1),i2),i3),...], or in left associative infix form with
the f = "+" operator, [i0+i1,i0+i1+i2,i0+i1+i2+i3,...].
Consing (effectively) initializer to the beginning of the list ensures
clean behavior for empty lists, and indexing [-1] gets the last element
which is really the only
element that matters.

Consider a slightly more complex example of a Fibonacci generator, one
might define it as follows,
  def fibs():
    a, b = 1, 0
    while True:
      a, b = b, a + b
      yield b

Using the with statement, it would require two generator comprehensions
  fibs = lambda : (b for a,b in (b, a+b for i in itertools.cycle((None,))
with a,b = 0,1))
The inner generator comprehension
  (b, a+b for i in itertools.repeat(None) with a,b = 0,1)
creates an infinite generator of tuples which are
consecutive Fibonacci numbers, the outer list comprehension strips off the
unneeded "state".

Some of the pros of doing it this way is that because with/as are already
keywords in python backwards compatibility shouldn't be an issue,
but if one is just mapping with state then an extra list/generator
comprehension block is needed to strip the state from the intermediate list.

I apologize if similar ideas have already been discussed.
-Andrew Carter

p.s. Is there a built-in way to get the last element from a generator
(perhaps even with a default) a quick google search did not reveal one?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20120610/c1fee3a2/attachment.html>