Behavior of += (was Re: [Python-Dev] Customization docs)

[Chris Barker]

John Roth wrote:
> And I disagree that assignment implements two distinct operations.
> Assignment always rebinds. Always.

Yes, but += is NOT Always an assignment operator, it is sometimes an
assignment operator.

> In the examples cited, it
> attempted to rebind into a tuple, because that's what was asked
> for when the left hand side had a subscript.

What was asked for was to mutate an element of a tuple, which is
perfectly legal in other cases:

>>> t[1].extend([3])
>>> t
([0], [1, 3], [2])

in fact, list += list is equivalent to list += list, as far as I can tell.
 
> The reason that the examples given at the top of the thread fail
> in such strange ways is simply that '+=' does not check whether
> the assignment is possible before doing the addition (or concatination,
> in the specific cases cited.)

It shouldn't do the assignment at all, as it isn't necessary, but if
that is too difficult, it should certainly check.
 
> To see this, you could try the first example, using explicit '+' and
> '='. I suspect it would fail in exactly the same way, with exactly
> the same side effect.

No need to suspect:
>>> t = ([0],[1],[2])
>>> t[1] = t[1] + [3]
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
TypeError: object doesn't support item assignment
>>> t
([0], [1], [2])

So, it fails, but with no side effect, as it should.

Huaiyu Zhu wrote:
> Great explanation!  The problem stems from the fact that the current +=
> operator implements two semantically distinct operations, chosen according
> to some properties of the operands.

exactly, and it wa kind of a weird decision. Personally, I feel the only
compelling reason to add The += (and friends) operator was to support
in-place addition, most notably for Numeric arrays. Frankly it ought to
just be disallowed for non-mutable types, but all the folks that want it
for integers would scream. to be honest, I've kind of lost track of why
it is a good idea for numbers to be immutable, but that's another topic!

> This is not unlike the situation of the
> division operator with distinct behaviors on floats and ints.

I suppose that is a pretty good parallel, but not quite the same, as the
behavior of += is much more similar for the different types: most of the
time it doesn't matter if the object is changes in pace or not, the main
reason I want it is for performance with Numeric arrays. Of course,
being able to change a mutable object in place inside a tuple is nice
too...oops, that doesn't work!

> I'd imagine that at some future time there would be a desire to split += to
> two distinct operators, one for true in-place operator and one for the fake
> one currently implemented for immutables.  That would remove the spurious
> exceptions for mutable members of immutable structures.

Do we really need a new notation? is a = a + b so bad? In fact, are the
current semantics of += s bad, or should this bug just be fixed?

> That's not unlike the difficulty involved with the division operator change.
> Maybe it's better to change earlier?

If there is going to be a change, the sooner the better!!

Gustavo Cordova wrote:
> The thing is that += MUST work also for immutable objects,
> as well as mutables.

That's arguable, but it how it is done now...

> So, it does a rebind of the final
> value obtained --which in a list or dictionary, being mutable,
> is the same as the initial reference, but in the case of
> immutable objects, it's a different one than before--.

couldn't the re-bind happen only if required?
 
>From the reference docs:

These methods are called to implement the augmented arithmetic
operations (+=, -=, *=, /=, %=, **=, <<=, >>=, &=, =, |=). These methods
should attempt to do the operation in-place (modifying self) and return
the result (which could be, but does not have to be, self). If a
specific method is not defined, the augmented operation falls back to
the normal methods. For instance, to evaluate the expression x+=y, where
x is an instance of a class that has an __iadd__() method, x.__iadd__(y)
is called. If x is an instance of a class that does not define a
__iadd() method, x.__add__(y) and y.__radd__(x) are considered, as with
the evaluation of x+y. 

So, if you simply do not define a set of __i***__ methods for
non-mutable types, then the interpreter would do "the right thing", and
re-binding would not have to happen for mutable types. I no so little
about Python byte code generating and interpreting, that I'm sure I
missing something, but it certainly looks like it could be done.

> So, it's not actually a bug, but an artifact of the hoops
> Python needs to jump through in order to maintain total
> dynamism in data types.

It certainly is a bug. All bugs are artifacts of something! IF not a
bug, I suppose a MAJOR wart.

> Instead of this:
> 
>    a = ([1],[2],[3])
>    a[0] += [3]
> 
> do this:
> 
>    a = ([1],[2],[3])
>    t = a[0]
>    t += [3]

why not just:
a = ([1],[2],[3])
try:
	a[0] += [3]
except TypeError:
	pass

Because that would work too. And you don't think it's a bug?

> It's not *totally* equivalent, because it's missing a
> single, last step:
> 
>    a[0] = t
> 
> *that* is what's raising the exception, which is what
> the += operator is trying to do in you're original version.

Right, but it shouldn't be, clear and simple. += means "add in place, if
possible" it is possible, there should be no error.

I'm in no position to figure out how this should be solved, but it
should be. I'm dumbfounded that folks seem to think that this is not a problem!

me:     Doctor, it hurts when I do this.
Doctor: Well, then don't do that!

By the way, the += and friends operators also cause similar confusion
with name scopes in functions, += is assumed to be a re-bind, when it
isn't necessarily intended to:

>>> l = [1,2,3]
>>> def test():
...     l += [4]
... 
>>> test()
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "<stdin>", line 2, in test
UnboundLocalError: local variable 'l' referenced before assignment

but:

>>> def test2():
...     l.extend([4])

>>> test2()
>>> l
[1, 2, 3, 5]

Works just fine. Perhaps  Huaiyu is right, we need different operators.
However, then the operator for incrementing a non-mutable would just be
a synonym for a = a + b, which looks like unnecessary syntactical sugar
to me.

-Chris


-- 
Christopher Barker, Ph.D.
Oceanographer
                                    		
NOAA/OR&R/HAZMAT         (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov