Explanation of this Python language feature? [x for x in x for x in x] (to flatten a nested list)

[Steven D'Aprano]

On Mon, 24 Mar 2014 00:52:52 -0500, Mark H Harris wrote:

> On 3/22/14 4:46 AM, Steven D'Aprano wrote:
>> On Fri, 21 Mar 2014 23:51:38 -0500, Mark H Harris wrote:
>>
>>> Lambda is a problem, if only because it causes confusion. What's the
>>> problem?  Glad you asked. The constructs DO NOT work the way most
>>> people would expect them to, having limited knowledge of python!
> 
>     One of the best links for understanding what is wrong with lambda is
> here, from Guido, (by the way I agree totally with his assessment, there
> is no point really in writing it out again):
> 
>    http://www.artima.com/weblogs/viewpost.jsp?thread=98196

It's only one paragraph. Here it is:

    Why drop lambda? Most Python users are unfamiliar with Lisp or
    Scheme, so the name is confusing; also, there is a widespread
    misunderstanding that lambda can do things that a nested function
    can't -- I still recall Laura Creighton's Aha!-erlebnis after I
    showed her there was no difference! Even with a better name, I think
    having the two choices side-by-side just requires programmers to
    think about making a choice that's irrelevant for their program; not
    having the choice streamlines the thought process. Also, once map(),
    filter() and reduce() are gone, there aren't a whole lot of places
    where you really need to write very short local functions; Tkinter
    callbacks come to mind, but I find that more often than not the
    callbacks should be methods of some state-carrying object anyway (the
    exception being toy programs). 


None of map(), filter() or reduce() are gone. reduce() has been pushed 
out into a module, but map() and filter() have actually been enhanced. 
Any suggestion that lambda is obsolete flounders on that fact.

There's no doubt that lambda is less-often useful than is the def 
statement. But not only is it still useful, but there is a continual 
stream of people asking for Python to make it *more useful* by allowing 
the body of a lambda to be a full block, not just a single statement. 
Guido has stated (although I cannot remember where, so cannot paste a 
link) that he accepts that in principle, so long as somebody comes up 
with acceptable syntax.

In Ruby, anonymous blocks are used *all the time*. Python is not Ruby, 
and never will be, but a lightweight way to generate anonymous functions 
in expressions is one difference between a modern, powerful language, and 
an archaic or toy under-powered language. I'm too lazy to do this the 
right way, but there are at least 99 mentions of "lambda" in the 3.3 
standard library:

steve at runes:/usr/local/lib/python3.3$ grep lambda *.py | wc -l 
99


It's used in calendar, cgitb, configparser, difflib, decimal, functools, 
inspect, os, pydoc and others. Python is a programming language aimed at 
programmers, not a toy. It should include tools that programmers use, 
like threads, closures, first-class functions. And lambda.


>> Why is that a problem? Would you consider it a problem that people who
>> don't understand BASIC can't understand BASIC? ("I don't get the
>> difference between GOTO and GOSUB. {snip}
>>
>> The problem is ignorance, not the calculator, and not lambda.
> 
>     You're argument, based on analogy, is also a red herring, as well a
> straw man. 

Please stop misusing "straw man". It has a specific meaning:

https://yourlogicalfallacyis.com/strawman


> Lambda is a problem of confusion for scientists and other
> mathematicians (amateur and otherwise) who may be confused because
> python's lambda does not do precisely what they might expect from other
> functional programming languages, nor does it match precisely with the
> lambda calculus. 

Yes, you've already said that. And my response remains the same: users of 
a language should be expected to learn the language, at least to the 
point that they can "get by". One doesn't need ten thousand hours 
experience and to be an expert. But nor should the confusion of somebody 
who doesn't know the language count for much.

Python's ints don't work exactly the same as ints do in some other 
languages, nor do Python's strings, nor is Python's object model 
precisely the same as that of some other languages. I think that it would 
be a ridiculous idea to discard int, str and the entire object system 
just because some people are surprised that Python doesn't behave exactly 
like some other language. I don't think lambda is any different.

The same argument applies to *any other language*. Ocaml is not precisely 
the same as Haskell, Java not precisely the same as C++, Ruby is not 
precisely the same as Javascript. Should Ruby discard their anonymous 
blocks because they don't work "precisely" the same as Haskell? Or 
perhaps Haskell should throw the towel in because it doesn't behave 
"precisely" the same as Ruby? 

Of course not -- new languages come into existence precisely so that they 
will be different from what is already there. Who is to say that there 
aren't people who like Python's lambda *because* it is different from the 
anonymous functions in other languages? Or because it is less abstract 
than the lambda calculus?

Other languages -- not even Lisp -- don't get to be the sole decider as 
to what counts as an anonymous function. Python's lambda may be limited, 
but it is a perfectly fine lambda for what it is intended to do.


> Its also confusing to sophisticated users of all
> stripes who may not be aware of "lambda" at all.

If they're not aware of lambda, how are they confused by it?


>> You've told us that "most people" (which people? dentists? lawyers?
>> illiterate tribesmen from the Amazon? professional programmers?) are
>> surprised by lamba's behaviour, but you haven't actually told us what
>> behaviour is surprising, or what "most people" expect lambda to do.
>> Perhaps you ought to?
> 
>     I love your rhetoric, actually, but red herrings all . . .

No, not really. I don't understand who your intended audience for Python 
is. If I were to try to guess, I suspect that you want to dumb Python 
down until it is like Dartmouth BASIC circa 1980 except without the line 
numbers and with a slightly different syntax. Perhaps I'm wrong, I 
certainly hope I'm wrong, but that's the impression I got from your 
comments on the python-ideas list. 

Python is a programming language. People who program in Python are 
programmers. It doesn't matter whether they are school children, or 
scientists, or farmhands, once they start writing programs, they are 
programming, which makes them a programmer. They might not be paid for 
it, but they are still programmers. Who should be using Python, if not 
programmers?


>     Here is a link that advocates lambda (map, filter, and reduce) as a
> very powerful construct for doing various things:  I agree:
> 
> http://www.secnetix.de/olli/Python/lambda_functions.hawk
> 
>     Its a great link, but the important sentence for this discussion is
> this unique quote, about "lambda,"
> 
>  >"This is not exactly the same as lambda in functional
>> programming languages, but it is a very powerful concept . . ."
> 
>     People who understand lambda from functional languages must hassle
> with the fact that lambda is different in python, and people who do not
> understand functional programming languages (nor the lambda calculus)
> are confused by the syntax, also just what it does.

You want to remove lambda because it causes confusion. Okay, for the sake 
of the argument I will agree that lambda is confusing. Do you know what 
else is confusing?

Threading is confusing. So is multiprocessing. So are databases. Unicode 
is confusing. First-class functions are confusing. Recursion is 
confusing. Closures are confusing. List comprehensions are confusing. 
What the hell are trampolines? I'm certainly confused by them.

If we remove lambda because it is confusing, shouldn't we also remove all 
these other confusing things? Where shall we stop?

Should we stop only when Python is a confusion-free, utterly useless toy 
language? I don't think so. Or should we stop *before* going down that 
path? You want to single out lambda. I don't think we should.


>> Python is not a pure functional language, but you can write functional
>> code in it. If you want a pure functional language, you know where to
>> find one.
> 
>     Yes, that's obvious; but you're missing the point. Python is not a
> functional language, and implying that it can be used as one is
> misleading at best (maybe a lie at worst) just because it has a
> construct for generating a dynamic anonymous function.

I didn't say that Python can be used as a functional language. I said you 
can write functional code in Python. And that is so obviously true that I 
cannot believe that you are disputing it.

Functions are first-class values. You can pass them around as arguments 
to other functions. You can wrap them to make closures, or compose them. 
You can return functions from other functions. You can, if you wish, 
eschew holding state and write pure functions with no side-effects and no 
state. You can even eschew local variables (well, mostly) inside your 
functions and do (nearly) everything by composing other functions. You 
can write pipelines of iterators, generators or co-routines.

It may be hard to write an *entire* application using nothing but 
functional style in Python, but using functional style throughout the 
application is so easy and simple that most people don't even realise 
that they are writing in a functional style when they do so.


>> (I wonder whether, say, Haskell has the people coming along and
>> demanding that it should become more like Python?)
> 
>     Another straw man.
> 
>     Well, number one, I'm not demanding anything. Number two, everyone
> who uses Haskell (for whatever reason) knows well from the start that
> its a pure functional programming language. That is the advertisement,
> and that is the expectation. No one expects anything different.

And everyone who uses Python ought to know that Python is not. That is 
the advertisement, and that ought to be the expectation.

Anonymous functions are not the sole preserve of purely functional 
languages, any more than objects are the sole preserve of purely OOP 
languages.


>>> I can see uses for python's lambda.  But, honestly, I think python
>>> could deprecate its use and in five years just remove it from the
>>> language; along with filter, map, and reduce !
>>
>> Python could deprecate many things. It would make Python a worse
>> language.
> 
>     How so?  Read Guido's argument above. Another way to answer this
> question is that I have been programming with Python for almost a decade
> and I've not used lambda. In fact, I have gone out of my way to NOT use
> lambda because I am fully aware that the BDFL hates it.

In other words, you have had situations where you *could have* used 
lambda, perhaps even *should have* used lambda, but you intentionally 
went *out of your way* (i.e. made the job harder than it needed to be) to 
avoid it, just to slavishly ape the BDFL.

Thank you for just proving my point for me. Your code would have been 
better, or at least easier, with lambda. Removing it would make the 
language worse, not better.


> Reduce is no
> longer in the standard library (although you can import it) 

You mean it is no longer in the builtin namespace. It is still in the 
standard library.


> and there
> are equally good ways to do what lambda was designed for without the
> hassle nor confusion.

Incorrect. lambda was designed to allow you to create anonymous functions 
in an expression, not as a statement. There is *no other way* to do that 
in Python. Even if you come up with some weird hack involving exec or 
FunctionType and code objects, it won't be *equally good*. It will be a 
broken, third-rate imitation of lambda.


>> By the way, are you aware that lambda is *identical* to def except for
>> three things?
>>
>> - lambda is an expression, not a statement like def;
>>
>> - the body of a function created with lambda is limited to a
>>    single expression, not a block;
>>
>> - the function object created with lambda has a generic name.
> 
>     Absolutely, thank you for pointing those out !  You forgot that
> lambdas can be used in expressions while def functions may not... 

Umm, read my list again. It's the first item.


> but
> you really have proved my point.  There is so little difference between
> the two that (for the sake of clarity and reducing confusion) it might
> be good to just remove the lambda thing altogether.  Just say'in.

The *number* of differences may be small, but the *consequences* of those 
differences are enormous.


>> So unless the surprising behaviour about lambda that you're about to
>> tell us about relates to one of those three things, I *guarantee* that
>> functions created with def have the same surprising behaviour.
> 
>     No, the surprising behavior is that python lambda does not work
> exactly the way that lambda works in functional programming languages

Then neither does def.

Would you like to discard def as well? No? Why not?


> (see the two links provided) the confusion it creates for "normal" users
> does not warrant its inclusion in the language.
> 
>     Having said that, I must admit that I've "played" with lambda
> extensively and find it interesting from a comp sci standpoint, but,
> again, for normal users I don't think there is a really good value add
> for having it in the language.




-- 
Steven