PEP thought experiment: Unix style exec for function/method calls

[Carl Banks]

Michael wrote:
> Suppose we could do the same for a python function - suppose we could
> call the python function but either /without/ creating a new stack
> frame or /replacing/ the current stack frame with the new one.

I'm confused about what you mean.  I'm guessing by "not creating a new
stack frame" you wish to execute the bytecode from a different function
in the current stack frame, correct?  As for replacing the current
stack frame, do you want to replace the whole stack, down to the
__main__ module entry point, or just replace the top frame?  Your
comments and examples seem to be inconsistent about this.  I'm going to
assume you just want to replace or reuse the top frame in my replies.


[snip]
> Since os.exec* exists and "exec" already  exists in python, I need to
> differentiate what I mean by a unix style exec from python. So for
> convenience I'll call it "cexe".
>
> Now, suppose I have:
>     ----------
>     def set_name():
>         name = raw_input("Enter your name! > ")
>         cexe greet()
>
>     def greet():
>         print "hello", name
>
>     cexe set_name()
>     print "We don't reach here"
>     ----------
>
> This would execute, ask for the user's name, say hello to them and then
> exit - not reaching the final print "We don't reach here" statement.

Only if you were to replace the whole stack.  If you only replace or
reuse the top frame, I would think greet would exit and execution would
resume right after the point from which set_name was called.  Or am I
misunderstanding what you want?


> Let's ignore for the moment that this example sucks (and is a good example
> of the danger of this as a language feature), what I want to do here is
> use this to explain the meaning of "cexe".
>
> There's two cases to consider:
>   cexe some_func_noargs()
>
>     This transfers execution to the function that would normally be called
>     if I simply called without using "cexe" some_func_noargs() . However,
>     unlike a function call, we're /replacing/ the current thread of
>     execution with the thread of execution in some_func_noargs(), rather
>     than stacking the current location, in order to come back to later.
>
>     ie, in the above this could also be viewed as "call without creating a
>     new return point" or "call without bothering to create a new stack
>     frame".
>
>     It's this last point why in the above example "name" leaks between the
>     two function calls - due to it being used as a cexe call.

I doubt this would be possible without a major change in how functions
work in Python.  The most obvious problem is that functions refer to
local variables by an index, not by name.  If you were to execute the
bytecode of one function in the stack frame of another, Bad Things
would happen.


> Case 2:
>   given...
>      def some_func_withargs(colour,tone, *listopts, **dictopts)
>
>   consider...
>      cexe some_func_withargs(foo,bar, *argv, **argd)
>
>      This would be much the same as the previous case, except in the new
>      execution point, the name colour & tone map to the values foo & bar had
>      in the original context, whilst listopts and dictopts map the values
>      that argv & argd had in the original content)

This is a lot more doable, but I'm not sure a new statement is needed.
It seems like it wouldn't be terribly impossible to optimize the return
statement to do this (in fact there's a certain situation where it's
common to do just that, in other languages; more on that later).  If
function A were to return the result of calling function B, the A could
possibly pop itself off the stack before calling B (but after
evaluating B's arguments).  B would then return directly to A's caller
in place of A.  Essentially, B has replaced A on the stack.  Biggest
problem is you'd have to know whether an object is a Python function at
compile time--though that may not be so hard in the future.

(Now, if you're thinking that you'll replace the whole stack and not
just the top frame, Python already has the machinery to do this if you
can tolerate wrapping up your main function to catch an exception and
dispatching on it.  For example:

    class _Cexe(Exception):
        def __init__(self,func,*args,**kwargs):
            self.func = func
            self.args = args
            self.kwargs = kwargs

    def cexe(func,*args,**kwargs):
        raise _Cexe(func,*args,**kwargs)

    def main_wrapper(*args,**kwargs):
        func = main
        while True:
            try:
                func(*args,**kwargs)
                return
            except _Cexe,e:
                func = e.func
                args = e.args
                kwargs = e.kwargs

As an added bonus, you can do this higher up the stack if you want.  A
cexe statement wouldn't improve this too much; it wouldn't even make it
more efficent.  You can't just reset the stack pointer; you have to
clean things up.  The only upside to cexe I can think of is it could
help reduce latency by delaying cleanup until garbage collection, but
it wouldn't help overall efficiency too much.)


[snip a lot]
> It also struck me that any sufficiently interesting idea is likely to
> have already been implemented, though perhaps not looking quite like the
> above, so I thought I'd ask the questions:
>
>   * Has anyone tried this sort of thing?

Yes.  There is a major use case for what you seem to be proposing,
namely tail-recursive functions.    A tail-recursive function is a
recursive function that immediately returns the result of calling
itself.  A tail-recursive function never needs to keep its state around
once it's called itself, and it's very common in functional languages,
so it's an easy and common optimization.

However, what you seem to be proposing is a generalization of that.  It
seems like it's possible, theoretically, to generalize this so that it
works when you return the result of any function call, recursive or
not.  Theoretically.  It wouldn't surprise me if functional languages
like Haskell do this already.

>   * Has anyone tried simply not creating a new stack frame when doing
>     a function call in python? (or perhaps replacing the current one with
>     a new one)

Doubt it.

>   * Has anyone else tried modelling the unix system exec function in
>     python? If so what did you find?
>
>   * Since I can't find anything in the archives, I'm presuming my
>     searching abilities are bust today - can anyone suggest any better
>     search terms or threads to look at?

Maybe look to see how tail-recursive optimization in languages such as
Scheme work, and whether it can be generalized.

>   * Am I mad? :)

Yep. :)


Carl Banks