Pre-pep discussion material: in-place equivalents to map and filter

[Steven D'Aprano]

On Thursday 03 November 2016 17:56, arthurhavlicek at gmail.com wrote:

[...]
> Python features a powerful and fast way to create lists through
> comprehensions. Because of their ease of use and efficiency through native
> implementation, they are an advantageous alternative to map, filter, and
> more. However, when used in replacement for an in-place version of these
> functions, they are sub-optimal, and Python offer no alternative.


Of course Python offers an alternative: the basic for loop.


for i, x in enumerate(alist):
    alist[i] = func(x)


Not enough of a one-liner for you? Write a helper function:

def inplace_map(func, alist):
    for i, x in enumerate(alist):
        alist[i] = func(x)


In practice, you may even find that except for the most enormous lists (so big 
that memory becomes an issue, so we're talking tens of millions of items) it 
will probably be faster to use a slice and a comprehension:

alist[:] = [func(x) for x in alist]


[...]
> Notice the concerns of OPs in his comments to replies in this one:
> http://stackoverflow.com/questions/3000461/python-map-in-place

What I saw was the OP's *premature optimization*:

"I wanted to use map for performance gains"

"I figured map would be quicker than the list comprehension way"

Although in fairness he does also say:

"I'm working on a big list, and the times we're talking about are seconds in 
difference, which clearly matters"

I'm not necessarily sure that seconds always matters -- if your code takes 90 
seconds, or 96 seconds, who is going to even notice? But let's assume he's 
right and a few seconds difference makes a real difference.

He has three obvious alternatives to waiting until Python 3.7 (the earliest 
such a new feature can be added):

- modify the list in place with a for-loop;
- slice assignment using map;
- slice assignment using list comprehension.


> 1 - Code readability and reduced redundancy
> 
> lst = [ item for item in lst if predicate(item) ]
> lst = [ f(item) for item in lst ]
> 
> Both these expressions feature redundancy, lst occurs twice and item at least
> twice. 

That's not what redundancy means. "Redundancy" doesn't refer to the re-use of 
any arbitrary token. It means doing the same thing twice in two different 
places.


> Additionally, the readability is hurt, because one has to dive through
> the semantics of the comprehension to truely understand I am filtering the
> list or remapping its values.

Nonsense. It is perfectly readable because it is explicit about what is being 
done, unlike some magic method that you have to read the docs to understand 
what it does.

A list comprehension or for-loop is more general and can be combined so you can 
do both:

alist[:] = [func(x) for x in alist if condition(x)]



> Map and filter, although they are more explicit, 

*Less* explicit.

To most people, "map" means the thing that you follow when you are travelling 
in unfamiliar territory, and "filter" means the paper doohickey you put in your 
coffee machine to keep the ground up coffee from ending up in your cup.


> also feature redundancy.
> They look OK with functional predicate:
> 
> lst = map (f, lst)
> lst = filter (predicate, lst)
> 
> But are less elegant when using an expression, than one has to convert
> through a lambda:
> 
> lst = map (lambda x: x*5, lst)
> lst = filter (lambda x: x%3 == 1, lst)

And that's why we have list comprehensions.


> And perform especially bad in CPython compared to a comprehension.

I doubt that.



> 2 - Efficiency
> 
> A language support for these operations to be made in-place could improve the
> efficiency of this operations through reduced use of memory.

*shrug*

Saving memory sometimes costs time.


> I would propose this syntax. (TODO: find appropriate keywords I guess):
> 
> lst.map x: x*5
> lst.filter x: x%3 == 1

I think the chances of Guido accepting new syntax for something as trivial as 
this with three existing solutions is absolutely zero.

I think the chances of Guido accepting new list/dict methods for in place map 
and/or filter is a tiny bit higher than zero.


> The reasonning for the need of a language-level approach is the need for an
> efficient implementation that would support giving an arbitrary expression
> and not only a function. 

We already have three of those: for-loops, list comprehensions, and map.




-- 
Steven
299792.458 km/s — not just a good idea, it’s the law!