PATCH: Augmented assignment

[Thomas Wouters]

As promised, here is my patch to add augmented assignment (+=, -= and
family) to Python. It's still Work In Progress, and probably requires a
rewrite of some magnitude before Guido will accept it.

The patch can be found here:
http://www.xs4all.nl/~thomas/augmented_assign.diff

What it does:

It adds support for 'in-place' binary operands, using the same syntax as C,
Perl and many more languages. In short, the new syntax

x += 1

is the same as 

x = x + 1

for immutable types. For mutable types, it's more like

x.add(1)

if the mutable type defines it like that. For classes, it tries to call
'instance.__add_ad__', which should return the resulting object -- which
could be self, or a new object, to simulate mutablility and immutability,
respectively. If the class does not define __add_ad_, regular __add__ is
attempted, and if that fails, __radd__ on the right hand side of the
expression.

The patch implements these operands (and their overloaders):

 +=	(__add_ad__)
 -= 	(__sub_ad__)
 *= 	(__mul_ad__)
 /= 	(__div_ad__)
 %= 	(__mod_ad__)
 **= 	(__pow_ad__)
 |= 	(__or_ad__)
 &= 	(__and_ad__)
 ^= 	(__xor_ad__)
 >>= 	(__rshift_ad__)
 <<= 	(__lshift_ad__)

This patch is based on Michael Hudsons' patch that added augmented
assignment only to builtin-types (and only the syntax, not the in-place
modification), and comments from Michael, Guido and Tim. It's quite an
improvement over Michaels' patch in that it adds overloading, and
inplace-operator-bytecodes, so that the new code is actually 'thread safe'.
I'd like to think that this patch is also easier to understand, because
there is no need to manipulate the stack in strange ways to get the right
operands in the right spot for the next operation ;)

The 'ad' in __add_ad__ is what Guido proposed, and I have to say I dont
particularly like it. Making it '__iadd__' or perhaps '__inplace_add__'
seems more obvious, and more in keeping with '__radd__'. But perhaps Guido
meant something entirely different from this patch.

BUGS:
I have tested the patch, by running all the tests in the test suite (which
includes an aug_assignment test) and by running PySol with the new python,
and it hasn't found any killing bugs (at least, not in this patch ;)

However, I'm fairly certain I screwed up the INCREF/DECREF count in a couple
of places, so it's quite probable that this patch makes Python the Exxon
Valdez of bytecode interpreters.

Also, the builtin mutable types do not (and can not) actually do in-place
modification, currently; Python falls back to the normal 'add' operations,
so 

x = [1,2,3]
x += [1,2,3]

and 

x = [1,2,3]
x *= 2

both work exactly as expected, but they create new objects, instead of
modifying x. See the TODO list.


TODO:
- add in-place variants of binaryfunctions to the PyNumberMethods struct, so
  builtin and extention types can provide them

- supply in-place binary functions for the builtin types

- Add proper prototypes to the proper places to avoid warnings

- Add support for the new opcodes to Lib/dis.py

- Add support for the new Grammar to parsermodule.so

- eradicate bugs

- Make Grammar more readable by splitting 'exprlist' and ajust compile.c
  accordingly. (Wanted to do that right away, but couldn't be bothered with
  figuring out how to modify compile.c to understand the extra branch, at
  that moment ;)

- write JPython version (I haven't looked at this yet, but I doubt I know
  enough Java. Volunteers greatly appreciated ! :-)

- write documentation (See JPython ;)

- Add support for in-place operators to the std. library, where it makes
  sense.

PROGRAMMER NOTES:

To implement this, I've had to add support for 2-argument bytecodes. (It was
either that, or add 11 times as many opcodes (a total of 99, that is), or
create a PyObject only intended for passing an argument to a bytecode -- not
very nice. I didn't want to risk pushing an int onto the
object-pointer-stack by pretending it is a pointer.) 2-argument bytecodes
may come in handy in other situations, I do think, but I'm not saying this
is the Best Solution.

I've also added 9 new opcodes:

/* Opcodes for augmented assignment (+= and family) */
/* Slice/subscr opcodes take the specific operation as argument */
#define GETSET_SLICE    150     /* Augmented operation to perform on stack */
/* Also uses 151-153 */
#define GETSET_SUBSCR   155     /* "" */

#define HAVE_2ARGUMENT  156

/* vrbl-assignment take two arguments: operation to perform, and index into
   name list */
#define GETSET_NAME     156     /* arg1: operation, arg2: index in namelist */
#define GETSET_ATTR     157     /* "" */
#define GETSET_GLOBAL   158     /* "" */
#define GETSET_FAST     159     /* arg1: operation, arg2: local vrbl number */

The GETSET_ name is only the best I could come up with while writing it,
perhaps something else would be more obvious :P

The patch also adds these functions to the 'number protocol' API:

PyObject * PyNumber_InPlaceAdd ( PyObject *o1, PyObject *o2 );
PyObject * PyNumber_InPlaceSubtract ( PyObject *o1, PyObject *o2 );
PyObject * PyNumber_InPlaceMultiply ( PyObject *o1, PyObject *o2 );
PyObject * PyNumber_InPlaceDivide ( PyObject *o1, PyObject *o2 );
PyObject * PyNumber_InPlaceRemainder ( PyObject *o1, PyObject *o2 );
PyObject * PyNumber_InPlacePower ( PyObject *o1, PyObject *o2 );
PyObject * PyNumber_InPlaceLshift ( PyObject *o1, PyObject *o2 );
PyObject * PyNumber_InPlaceRshift ( PyObject *o1, PyObject *o2 );
PyObject * PyNumber_InPlaceAnd ( PyObject *o1, PyObject *o2 );
PyObject * PyNumber_InPlaceXor ( PyObject *o1, PyObject *o2 );
PyObject * PyNumber_InPlaceOr ( PyObject *o1,  PyObject *o2 );

Which are InPlace versions of the normal functions.

QUESTIONS

Having written it this far, I'm getting a bit stuck. Perhaps a more
experienced python-patcher can give me a hint on these things:

- Is it safe to add new members to the PyNumberMethods struct ? How is
binary compatibility achieved ?

- Should the patch for 2-argument opcodes be sent in seperately ? Or should
I forget about 2-argument opcodes and use the object stack instead ?

- Is it likely this patch gets accepted as it is (but finished), or is there
something grossly wrong with it ? The purpose of this exercize, extreme fun
though it is, really is to get the patch accepted, so if I'm on the wrong
track, please let me know ;-P

The last two issues are not exactly urgent, as I wont submit the patch 'for
real' until I've added the first four entries in the TODO list at least
(unless someone influential suggests I do it anyway), but if I have to
change the entire approach, the sooner the better ;)

Unforuntately, while I was writing this mail, my Palm Vx *finally* arrived,
so I'm not very likely to spend the weekend on this patch ;-) But, for the
adventurous, the curious and the innovative, here is the URL to the patch,
v0.3. Thanx to Michael Hudson for providing the hints on how to proceed, and
Tim for explaining what Guido *really* meant ;-)

http://www.xs4all.nl/~thomas/augmented_assign.diff

-- 
Thomas Wouters <thomas at xs4all.net>

Hi! I'm a .signature virus! copy me into your .signature file to help me spread!