[Python-ideas] Reduce/fold and scan with generator expressions and comprehensions

[Steven D'Aprano]

On Sun, Oct 23, 2016 at 02:10:42PM -0200, Danilo J. S. Bellini wrote:
> >
> > Ah, that makes SIX existing solutions. Do we need a seventh?
> 
> It might have dozens of solutions, perhaps an infinity of solutions.
> Brainfuck and assembly can be used, or even turing machine instructions...

No, those are *implementations*. You can implement your solution in any 
language you like. For integration with Python, any of C, Fortran, Rust, 
Julia, Cython and (of course) pure Python are proven to work well. Using 
Turing Machine instructions requires some sort of TM compiler... good luck with 
that.

But you cut out the most important part of my post. You've given lots of 
existing solutions. Why aren't they satisfactory? Even if somebody wants 
to write in a functional style, the reduce() solution you show seems 
perfectly clean and conventional to anyone used to functional code:

from functools import reduce
reduce(lambda prev, x: abs(prev - x), [3, 4, 5], 2)

returns 2. What is wrong with this solution? That is the obvious 
solution for somebody looking for a functional style: something called 
reduce or fold. And there's no harm in needing to import reduce. Not 
every function has to be a built-in.


Whereas your suggestion needs TWO new features: new syntax:

(abs(prev - x) for x in [3, 4, 5] from prev = 2)

plus a new built-in function last() which extracts the final value from 
an iterator. That means you will risk encouraging people to wastefully 
generate a large container of unneeded and unwanted intermediate values:

last([abs(prev - x) for x in range(100000) from prev = 2])

which will generate a list 100000 items long just to extract the final 
one. reduce() is better, that is exactly what reduce() is designed for.


> But there should be one, and preferably only one, OBVIOUS way to do it.
> Readability counts.

Right. And the obvious way is the imperative approach (only this time I 
will use a better variable name):

py> result = 2
py> for x in [3, 4, 5]:
...     result = abs(result - x)
...
py> result
2


For those who think in functional programming terms, reduce() is the 
obvious way.



Also, I feel that your proposal could have been explained better. I felt 
overloaded by the sheer mass of different alternatives, and mislead by 
your use of the name "prev" for something that I see now on more careful 
reading is *not* the previous value of the loop variable (as I first 
understood) but the running calculation result.

In fairness I am sick and if I were well I may have been able to keep 
this straight in my head, but calling the variable "prev" is actively 
misleading. I was mislead, and (I think) Chris who just suggested this 
was similar to the SQL "lag" function may have been mislead as well. (Or 
perhaps he was just mislead by me, in which case, sorry Chris!)


-- 
Steve