Pass-by-reference : Could a C#-like approach work in Python?

[Stephen Horne]

I have in the past suggested that Python functions could use some kind
of modifier token for parameters to indicate pass-by-reference, so
that immutable items in particular could be modified (or more
accurately replaced) by the function, with that modification affecting
what the caller sees. In case I'm explaining this badly, here is how
Python currently works...

>>> def Inc(p) :
...   p += 1
...
>>> x = 1
>>> Inc(x)
>>> x
1

... and here is (roughly) what I'd like to be able to do...

>>> def Inc(ref p) :
...   p += 1
...
>>> x = 1
>>> Inc(x)
>>> x
2


There are good reasons why this is not supported in Python at present,
which have been explained to me in the past. However, I've been
learning C# just recently, and I've just been struck by a blinding
flash of the obvious. C# has a rather similar issue - some types are
'value types' (as opposed to 'reference types') and comparable to
immutables in Python in many ways, yet there is still a need (on
occasion) to pass them by reference. While C# has static typing and
could have done exactly the same thing as Pascal or as C++, it
actually did something different.

The C# solution uses a modifier keyword, but that keyword must be
specified in both the function definition *and* the call. For
instance...

...
public static void Inc (ref int p)
{
  p++;
}
...
int i = 1;

Inc (ref i);
...

The rationale for this in C#, I assume, is mostly about clarity and
maintainability - the caller can't be surprised when value type
parameters are changed in a function as he had to explicitly allow it.
It also allows both a by-reference and a by-value version of the same
function to be defined using overloading (though that seems much less
useful to me).

However, it occurred to me that this approach could also be used to
solve the main implementation problem with adding a call-by-reference
scheme to Python.

In Python, the compiler cannot adapt the calling convention to
different types of functions because it cannot refer back to the
declaration of the function at compile time - the function being
called is not always knowable until run-time (e.g. using a variable
can be assigned a function object, lambda can create anonymous
functions, etc). At first sight, it appears that supporting
call-by-reference would require a big run-time overhead as the
run-time engine queried the function object to determine which
parameters to pass by reference and to adapt the call appropriately.

Because the C# approach indicates which parameters are by reference in
the call, however, this overhead can be avoided - the compiler doesn't
need to refer to information in the declaration to determine which
parameters to pass by reference.

Basically, when a function or method is declared with 'ref' modifiers
on some parameters, Python could use a variant of the function type
which supports a new call mechanism and knows which parameters need to
be passed by reference. Similarly, when a call uses 'ref' modifiers,
Python can do a consistency check with the function type being called
and use the new call mechanism.

Performance with functions that don't use call-by-reference would be
unaffected...

1.  If a call with ref-modifiers is attempted on a non-ref function,
    this would be rejected in the same way as any attempted call
    on a non-callable object - the required behaviour wouldn't exist
    for conventional function objects.

2.  If a call without ref-modifiers is attempted on a function with
    ref modifiers, it would again be rejected as the new function
    type wouldn't support the required behaviour for conventional
    calls.

3.  A conventional call to a conventional function would operate
    exactly as it does now.

In short, only calls using call-by-reference would incur any overhead
(to validate that the correct parameters are by-reference or not, and
to implement the new calling convention).

Even that validation in the call overhead could probably be avoided
(by having a large number of function-class variants), leaving only
the implementation of the calling convention as a problem.

I would then be able to write...

>>> def Inc(ref p) :
...   p += 1
...
>>> x = 1
>>> Inc(ref x)
>>> x
2

Obviously the implementation would not be trivial, but i think this
would be a very useful feature.

Any thoughts?