Multi-dimensional list initialization

[Steven D'Aprano]

On Tue, 06 Nov 2012 14:41:24 -0800, Andrew Robinson wrote:

> Yes.  But this isn't going to cost any more time than figuring out
> whether or not the list multiplication is going to cause quirks, itself.
>  Human psychology *tends* (it's a FAQ!) to automatically assume the
> purpose of the list multiplication is to pre-allocate memory for the
> equivalent (using lists) of a multi-dimensional array.  Note the OP even
> said "4d array".

I'm not entirely sure what your point is here. The OP screwed up -- he 
didn't generate a 4-dimensional array. He generated a 2-dimensional 
array. If his intuition about the number of dimensions is so poor, why 
should his intuition about list multiplication be treated as sacrosanct?

As they say, the only truly intuitive interface is the nipple. There are 
many places where people's intuition about programming fail. And many 
places where Fred's intuition is the opposite of Barney's intuition.

Even more exciting, there are places where people's intuition is 
*inconsistent*, where they expect a line of code to behave differently 
depending on their intention, rather than on the code. And intuition is 
often sub-optimal: e.g. isn't it intuitively obvious that "42" + 1 should 
give 43? (Unless it is intuitively obvious that it should give 421.)

So while I prefer intuitively obvious behaviour where possible, it is not 
the holy grail, and I am quite happy to give it up.


> The OP's original construction was simple, elegant, easy to read and
> very commonly done by newbies learning the language because it's
> *intuitive*.  His second try was still intuitive, but less easy to read,
> and not as elegant.

Yes. And list multiplication is one of those areas where intuition is 
suboptimal -- it produces a worse outcome overall, even if one minor use-
case gets a better outcome.

I'm not disputing that [[0]*n]*m is intuitively obvious and easy. I'm 
disputing that this matters. Python would be worse off if list 
multiplication behaved intuitively.

An analogy: the intuitively obvious thing to do with a screw is to bang 
it in with a hammer. It's long, thin, has a point at the end, and a flat 
head that just screams "hit me". But if you do the intuitive thing, your 
carpentry will be *much worse* than the alternatives -- a hammered in 
screw holds much less strongly than either a nail or a screwed in screw. 
The surface area available for gripping is about 2% compared to a nail 
and about 0.01% compared to a screw used correctly.

Having list multiplication copy has consequences beyond 2D arrays. Those 
consequences make the intuitive behaviour you are requesting a negative 
rather than a positive. If that means that newbie programmers have to 
learn not to hammer screws in, so be it. It might be harder, slower, and 
less elegant to drill a pilot hole and then screw the screw in, but the 
overall result is better.


>> * Consistency of semantics is better than a plethora of special
>>    cases. Python has a very simple and useful rule: objects should not
>>    be copied unless explicitly requested to be copied. This is much
>>    better than having to remember whether this operation or that
>>    operation makes a copy. The answer is consistent:
> 
> Bull.  Even in the last thread I noted the range() object produces
> special cases.
>  >>> range(0,5)[1]
> 1
>  >>> range(0,5)[1:3]
> range(1, 3)

What's the special case here? What do you think is copied?

You take a slice of a tuple, you get a new tuple.

You take a slice of a list, you get a new list.

You take a slice of a range object, you get a new range object.

I'm honestly not getting what you think is inconsistent about this.



> The principle involved is that it gives you what you *usually* want;

Who is the "you" that decides what "you" usually want? And how do they 
know what is "usual"?

Two-dimensional arrays in Python using lists are quite rare. Anyone who 
is doing serious numeric work where they need 2D arrays is using numpy, 
not lists. There are millions of people using Python, so it's hardly 
surprising that once or twice a year some newbie trips over this. But 
it's not something that people tend to trip over again and again and 
again, like C's "assignment is an expression" misfeature.


> I read some of the documentation on why Python 3 chose to implement it
> this way.

What documentation is this? Because this is a design decision that goes 
all the way back to at least Python 1.5:

[steve at ando ~]$ python1.5
Python 1.5.2 (#1, Aug 27 2012, 09:09:18)  [GCC 4.1.2 20080704 (Red Hat 
4.1.2-52)] on linux2
Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam
>>> x = [[0]*5]*3
>>> x[0][1] = 99
>>> x
[[0, 99, 0, 0, 0], [0, 99, 0, 0, 0], [0, 99, 0, 0, 0]]


So I expect the design decision for Python 3 was "we made the right 
decision before, there's no need to change it".



>>    (pardon me for belabouring the point here)
>>
>>      Q: Does [0]*10 make ten copies of the integer object? A: No, list
>>      multiplication doesn't make copies of elements.
>
> Neither would my idea for the vast majority of things on your first
> list.

Um, yes? The point is that "vast majority" is not "everything". Hence, 
your suggested behaviour is inconsistent.



>      Q: What about [[]]*10?
>      A: No, the elements are never copied.
> 
> YES! For the obvious reason that such a construction is making mutable
> lists that the user wants to populate later.  If they *didn't* want to
> populate them later, they ought to have used tuples -- which take less
> overhead.  Who even does this thing you are suggesting?!

Who knows? Who cares? Nobody does:

n -= n

instead of just n=0, but that doesn't mean that we should give it some 
sort of special meaning different from n -= m. If it turns out that the 
definition of list multiplication is such that NOBODY, EVER, uses [[]]*n, 
that is *still* not a good reason for special-casing it. All it means is 
that this will be a less-obscure example of the billions of things which 
can be done in Python but nobody wants to.

You have quoted from the Zen of Python a few times in this post. Perhaps 
you missed one of the most critical ones?

Special cases aren't special enough to break the rules.

There are perfectly good ways to generate a 2D array out of lists, and 
even better reasons not to use lists for that in the first place. (Numpy 
arrays are much better suited for serious work.)


>>      Q: What about other mutable objects like sets or dicts? A: No, the
>>      elements are never copied.
>
> They aren't list multiplication compatible in any event! It's a total
> nonsense objection.

I'm afraid you've just lost an awful lot of credibility there.

py> x = [{}]*5
py> x
[{}, {}, {}, {}, {}]
py> x[0]['key'] = 1
py> x
[{'key': 1}, {'key': 1}, {'key': 1}, {'key': 1}, {'key': 1}]

And similarly for any other mutable object.

If you don't understand that lists can contain other mutable objects 
apart from lists, then you really shouldn't be discussing this issue.


>> Your proposal throws away consistency for a trivial benefit on a rare
>> use- case, and replaces it with a bunch of special cases:
>
> RARE!!!! You are NUTS!!!!

Yes, rare. I base that on about 15 years of Python coding and many 
thousands (tens of thousands?) of hours on Python forums like this one. 
What's your opinion based on?

List multiplication is rare enough, but when it is used, it is usually 
used to generate a 1D array like this:

values = [None]*n  # or 0 is another popular starting value

Using it twice to generate a 2D array is even rarer.


>>      Q: How about if I use delegation to proxy a list? A: Oh no, they
>>      definitely won't be copied.
>
> Give an example usage of why someone would want to do this.  Then we can
> discuss it.

Proxying objects is hardly a rare scenario. Delegation is less common 
since you can subclass built-ins, but it is still used. It is a standard 
design pattern.


>> Losing consistency in favour of saving a few characters for something
>> as uncommon as list multiplication is a poor tradeoff. That's why this
>> proposal has been rejected again and again and again every time it has
>> been suggested.
>
> Please link to the objection being proposed to the developers, and their
> reasoning for rejecting it.
> I think you are exaggerating.

Python is a twenty year old language. Do you really think this is the 
first time somebody has noticed it?

It's hard to search for discussions on the dev list, because the obvious 
search terms bring up many false positives. But here are a couple of bug 
reports closed as "won't fix":

http://bugs.python.org/issue1408
http://bugs.python.org/issue12597

I suspect it is long past time for a PEP so this can be rejected once and 
for all.


>> List multiplication [x]*n is conceptually equivalent to: <snip>
>> This is nice and simple and efficient.
> No it isn't efficient. It's *slow* when done as in your example.

Well of course it is slow*er* when you move it from low-level C to high 
level Python, but it is still fast.

>> Copying other objects is slow and inefficient. Keeping list
>> multiplication consistent, and fast, is MUCH more important than making
>> it work as expected for the rare case of 2D arrays:
>
> I don't think so -- again, look at range(); it was made to work
> inconsistent for a "common" case.

You mentioned range before, but it isn't clear to me what you think is 
inconsistent about it.


> Besides, 2D arrays are *not* rare and people *have* to copy internals of
> them very often.

So you say.


> The copy speed will be the same or *faster*, and the typing less -- and
> the psychological mistakes *less*, the elegance more.

You think that it is *faster* to copy a list than to make a new pointer 
to it? Your credibility is not looking too good here.


> It's hardly going to confuse anyone to say that lists are copied with
> list multiplication, but the elements are not.

Well, that confuses me. What about a list where the elements are lists? 
Are they copied?

What about other mutable objects? Are they copied?

What about mutable objects which are uncopyable, like file objects?


> Every time someone passes a list to a function, they *know* that the
> list is passed by value -- and the elements are passed by reference.

And there goes the last of your credibility. *You* might "know" this, but 
that doesn't make it so.

Python does not use either call-by-value or call-by-reference, and it 
certainly doesn't use different calling conventions for different 
arguments or parts of arguments. Everything is passed using the same 
calling convention. Start here:

http://mail.python.org/pipermail/tutor/2010-December/080505.html


> People in Python are USED to lists being "the" way to weird behavior
> that other languages don't do.

Python's calling behaviour is identical to that used by languages 
including Java (excluding unboxed primitives) and Ruby, to mention only 
two.

You're starting to shout and yell, so perhaps it's best if I finish this 
here.


-- 
Steven