Explanation of this Python language feature? [x for x in x for x in x] (to flatten a nested list)

[Steven D'Aprano]

On Mon, 24 Mar 2014 14:47:11 -0500, Mark H Harris wrote:

> On 3/24/14 4:49 AM, Steven D'Aprano wrote:
>> There's no doubt that lambda is less-often useful than is the def
>> statement. But not only is it still useful, but there is a continual
>> stream of people asking for Python to make it *more useful* by allowing
>> the body of a lambda to be a full block, not just a single statement.
>> Guido has stated (although I cannot remember where, so cannot paste a
>> link) that he accepts that in principle, so long as somebody comes up
>> with acceptable syntax.
> 
>     Now, this will seem strange to you, but if lambda were expanded to
> include an entire block, well now, I would know two things:
>    1) python has commitment to lambda

Python has a commitment to lambda. lambda is not going anywhere. There 
was some idea chatter five or six years ago about dropping it from the 
language. It didn't get dropped, so now it is as much a part of the 
language as for-loops, print, classes and import.


>    and 2) lambda would become WAY more useful...

Every one agrees that being able to include a full suite of statements in 
an expression form would be useful. Unfortunately, due to the constraint 
of Python's syntax, it seems to be impossible. People have looked for 20+ 
years, and nobody has come up with an acceptable syntax.


[...]
>>> Lambda is a problem of confusion for scientists and other
>>> mathematicians (amateur and otherwise) {snip}
>>{snip}
>> But nor should the confusion of somebody who doesn't know the language
>> count for much.
> 
>     This is the main disagreement between the two of us on this topic.
> It should count for TONS; whole metric TONs. Because someone who
> understands python primarily, and after five years of coding (say me)
> somebody tells (me) to use lambda in a key clause so that my sort will
> work, or work better, or work "their" way; and I have no idea why I need
> this nameless function (after I find out that "that" is what it is!)

Then you don't understand *sorting*. If you don't understand why you need 
a key function, the problem is not the syntax used to create the key 
function, but your understanding of sort. Time to remove sorting from 
Python, yes? No?

It's not lambda that is giving you trouble. That's just syntax. There is 
no conceptual difference between:

    data.sort(key=lambda item: *magic goes on here*)

and

    def key_func(item):
        *magic goes on here*

    data.sort(key=key_func)


If the first confuses you, so will the second. If you are capable of 
learning what the first means, learning the second is no harder.


> Its confusing, and please contrast between complicated. I don't mind
> complicated, or extensive, or even hard, but when something is
> needlessly confusing, then why?

I don't know why you insist it is confusing. Maybe you're just no good at 
seeing the correspondence between 

    lambda x: expression

and

    def func(x):
        return expression


>> Python's ints don't work exactly the same as ints do in some other
>> languages, nor do Python's strings, nor is Python's object model
>> precisely the same as that of some other languages. I think that it
>> would be a ridiculous idea to discard int, str and the entire object
>> system just because some people are surprised that Python doesn't
>> behave exactly like some other language. I don't think lambda is any
>> different.
> 
>     Your logic is faulty here.  Conceptually python's lambda doesn't
> work as expected, from contexts that define lambda

Python defines lambda too. Why should Python have to bend over to suit 
the definition used by other languages? They don't own the concept of 
anonymous functions. Anonymous functions are found in all sorts of 
languages, with different semantics and different syntax:

Pike:  lambda(typ1 para1, typ2, para2, ...) { ... };
Perl:  sub { my ($a, $b) = @_; ... }
Smalltalk:  [:a :b| ... ]
Haskell:  \a b -> ...
Erlang: fun(a, b) -> ... end
Dylan: method(a, b) ... end method


to mention just a few. Ruby even has at least three ways of spelling 
lambda, one of which even uses the word lambda:

    {|a, b| ... } 
    lambda {|a, b| ...}
    proc {|a, b| ...}


> and where lambda in functional programming is VERY important.

And? Nobody is saying that they have to stop using the term lambda. 
Certainly I'm not. I wouldn't be so rude.


> For one thing, why would we
> want to "spell out" the word "lambda".    ( \x y -> x + y ) a b ) If
> we're going to use lambda, then use it.

So you don't like the syntax of Python's lambda. That's your prerogative. 
But frankly, one thing I dislike about Haskell is that it is too terse. 
Why would you use lambda without spelling out explicitly that you are 
doing so?


>> Other languages -- not even Lisp -- don't get to be the sole decider as
>> to what counts as an anonymous function. Python's lambda may be
>> limited, but it is a perfectly fine lambda for what it is intended to
>> do.
> 
>     Is that the same as George W. Bush's "decider" ?

I have no idea why you are mocking my use of the word "decider". It's a 
perfectly legitimate word, a standard term in English for someone who 
decides. The Oxford dictionary dates it back to sometime between 1670 and 
1699, so it is hardly a neologism or one of Junior Bush's malapropisms.

Let me put it another way. Functional programming languages do not get to 
be the sole decision-maker of what counts as a legitimate syntax or 
semantics for anonymous functions.


>     I agree, and so does the link I referenced, for what it was designed
> to do, its powerful; esp if one needs to embed a function in an
> expression;  that is what lambda is for.

Right.

 
>>> Its also confusing to sophisticated users of all stripes who may not
>>> be aware of "lambda" at all.
>>
>> If they're not aware of lambda, how are they confused by it?
> 
>     See above.  They are recommended its use (maybe on this list) and
> they just don't get it; because its unnecessarily confusing.

Then they are aware of lambda.

Confusing in what way? Is it the keyword used? You don't need to know the 
background of the term to learn to use it. If non-English speakers can 
memorise that you use "class" to define classes, English speakers can 
stop being so precious and memorise that "lambda" is used to define a 
function in an expression.

'lambda' is just one of those jargon words, like 'tuple', 'iterate', 
'boolean', 'closure', ... just learn it, and you'll be fine. If you want 
to investigate the background of the word, go right ahead, but it won't 
make you a better programmer to know that "boolean" is named after George 
Boole, and I still have no idea what the origin of "tuple" is. (And after 
15+ years I still want to spell it "turple".)


[...]
> Programming should be a
> liberal art available to everyone in the set of all "normal" educated
> people. 

Why should it be? Programming is no more a liberal art than painting is 
an offshoot of calculus.

Most "normal" (normal as defined by whom?) educated people have no 
interest, no desire, and no aptitude for programming. Most *programmers* 
have no aptitude for the rigor of programming -- the world is full of 
lousy programmers churning out their 40 lines of buggy VB/PHP/Java/C code 
per day but aren't any good at it:

http://www.yacoset.com/Home/signs-that-you-re-a-bad-programmer
http://haacked.com/archive/2007/02/27/Why_Cant_Programmers._Read.aspx/

Programming is a skill, like writing iambic pentameter. Should liberal 
arts courses ban the use of iambic pentameter by poets because some 
people find it confusing and can't count syllables or tell the difference 
between stressed and unstressed? (I know I can't. My wife despairs that I 
am so useless at anything like poetry.)


> Some developers (themselves perhaps with degrees Ph.D. in
> mathematics or computer science) forget about the fact that "normal"
> educated people want to leverage their computer (maybe with python) for
> problem solution without having to become a professional computer
> scientist. I am advocating for those people.

So you say. But I think you are actually being terribly condescending and 
elitist at the same time. Consider:

(1) People who just want to get the job done, without learning a bunch of 
theory, *won't care* how their sort key function is written. They're 
looking for a recipe that they can copy and paste, and whether you write 
it like this:

    data.sort(key=lambda item: item[1])

or like this:

    from operator import itemgetter
    data.sort(key=itemgetter(1))

or like this:

    def sort_key(item):
        return item[1]
    data.sort(key=sort_key)


*they won't care*. In fact, they'll probably prefer the first version, 
with lambda, because it is fewer lines to copy.

And this is okay. Sometimes you've got more important things to do than 
understand how every line of your code works, especially for throw-away 
code, you just want to use it. It is very elitist to insist that features 
that some people don't understand have to go. Do you understand how 
import works? I bet you don't. It's so complicated that there are 
probably only one or two core developers who really understand it in all 
it's glory. You just use it: "import this", and it just works. lambda is 
the same.

Sometimes programming is just a means to an end and the programmer just 
wants an answer now.


(2) You're concluding that just because somebody is confused by lambda, 
they would be better off if lambda was removed. That's horribly 
patronising. Have you considered that maybe they will take that as an 
opportunity to *learn something new*? That they'll learn how to use 
lambda, they may even investigate the theory behind it?

I am especially aggravated by your attack on lambda because I was that 
person. I had never heard of lambda calculus. When I first learned 
Python, lambda's syntax troubled me for a while. I had to keep looking it 
up, until suddenly the penny dropped that the argument list is exactly 
the same as the argument list of def. Replace "def name" with "lambda", 
drop the brackets, and use a single expression without "return", and 
Joan's your Auntie.

But I learned it, and it got me interested in where the name came from, 
and I discovered the lambda calculus and decided that while I have every 
admiration for the people who fly that high into the rarefied air of 
theory, that's not an area I have much interest in. In any case, I am a 
better programmer -- or at least a more educated one -- precisely because 
Python didn't dumb down, it didn't patronise me, it offered an 
"advanced" (huh! not really) feature that I otherwise would never have 
been exposed to.

Yes, Python could remove lambda, and take that opportunity away from 
others. Python would be a lesser language for it, and Python programmers 
will be lesser programmers for it.

Python programmers are some of most widely read programmers around, 
compared to the average C or PHP or VB coder. That's because they are 
exposed to so many different concepts in the language. Not everyone cares 
to learn about them, and that's okay, but they aren't Python's core 
audience.


>> You want to remove lambda because it causes confusion. Okay, for the
>> sake of the argument I will agree that lambda is confusing. Do you know
>> what else is confusing?
> 
>     I don't want to remove anything at this point. I am only suggesting
> that over time the python community might evaluate whether the lambda
> (map, filter, reduce) thing is a language benefit. If so, manet in
> aeternum.  Elif, deprecate it and over time phase it out completely. Its
> a community decision of course, and I'm NOT demanding anything.

This discussion was held five years ago. And lambda is still here.


>> Threading is confusing. So is multiprocessing. So are databases.
>> Unicode is confusing. First-class functions are confusing. Recursion is
>> confusing. Closures are confusing. List comprehensions are confusing.
>> What the hell are trampolines? I'm certainly confused by them.
> 
>     Argument by analogy almost never works. Please don't confuse my word
> "confusing" with your interpretation "complicated". Actually, I do not
> find any of those above mentioned as "confusing," 

Good for you! I mean it, I'm not being sarcastic. But I *know* people 
find all those things confusing, because (1) either I found, or still do, 
find them confusing, or (2) because I've seen people confused by them. 
When I say "confusing", I mean it. I don't mean complicated.


> while I do admit they
> are complicated--even extremely complicated. An intelligent person
> educated in the liberal arts should be able to sit down over time and
> pick-up the concepts in a new area--without--being confused by the
> material or presentation. Lambda is confusing NOT because its
> complicated (because its not) but because 1) it does not work like other
> lambda concepts in functional programming, 

Do you really believe that people educated in the liberal arts are 
exposed to the lambda calculus? Most MATHEMATICIANS aren't exposed to 
lambda calculus.


> and 2) because the syntax
> does not lend itself easily to immediate interpretation (by normal
> people) nor does it lend itself easily to explanation even in the best
> effort.

I think that's just nonsense. Take a one-line function:

    def func(args):
        return stuff

Remove the "def func", the brackets, and the "return". Move it all to one 
line, and stick the magic word "lambda" at the front:

    lambda args: stuff


And you're done. That's all there is to it.

Your insistence that lambda is confusing is awfully condescending. People 
are not as dumb as you insist, and they are perfectly capable of learning 
lambda without a comp sci degree. Like any technical jargon, there is 
vocabulary and meaning to learn, but the concept is no more difficult 
than ordinary def functions.



-- 
Steven D'Aprano
http://import-that.dreamwidth.org/