Re-executing the code object from a class 'declaration'

[Carlos Ribeiro]

On Wed, 6 Oct 2004 14:06:23 +0200, Alex Martelli <aleaxit at yahoo.com> wrote:
> Urk.  Ahem, I mean, interesting solution.  I _think_ it poses certain,
> ahem, problems, such as infinite recursion if a call to re_execute is in
> the same code object as that where the 'class MyClass' is, for example
> -- because basically you ARE re-executing the whole kaboodle.  You're
> not just re-executing the 'class' statement, but everything that comes
> before or after it in the same containing codeobject.  Your tests in an
> interactive interpreter are tricky that way, because in that special
> environment a code object is a tiny little thing.  But stick this into a
> module and place a dis.dis(dct['my_code']) in the metaclass's __new__
> and you'll see that basically MyClass.re_execute isn't very different
> from a reload of the module.  Isn't that a _bit_ too much in general?

(repeating to myself: dis is your friend. dis is your friend).

Well, it seems that I'm running out of options. I've checked dis, and
done a few more tests. I found a way to find the correct code object
for the _body_ of the class statement (or at least I think so), but
this does not solve the problem, as it will be shown below:

-------------
import inspect
import dis
import time
import types

class Meta(type):
    def __new__(klass, name, bases, dct):
        caller = inspect.stack(0)[1]
        caller_code     = caller[0].f_code
        caller_filename = caller[1]
        caller_lineno   = caller[2]
        print "Caller info: ", caller_filename, caller_lineno, caller_code
        print dis.dis(caller[0].f_code)
        for code in caller_code.co_consts:
            if isinstance(code, types.CodeType):
                if (code.co_filename == caller_filename) and \
                   (code.co_firstlineno == caller_lineno):
                    print "found!", code.co_name
                    dct['my_code'] = code
        return type(name, bases, dct)

class MyClass(object):
    __metaclass__ = Meta
    creation_time = time.ctime()
    print "I'm inside my class definition!"
    def re_execute(klass):
        exec klass.my_code
        return MyClass
    re_execute = classmethod(re_execute)
-------------

The code checks the filename and the line number to find the correct
code object. It could also check the class name, just to make sure
that everything matches. It isn't needed, as far as I know. But the
problem is that the code block that is found refers to the class
_body_ only. Checking dis.dis() helps to understand it:

 27          64 LOAD_CONST               4 ('MyClass')
             67 BUILD_TUPLE              0
             70 LOAD_CONST               5 (<code object MyClass at
01209E60, file "c:\work\help-on-c-l-py\re_exec_class.py", line 27>)
             73 MAKE_FUNCTION            0
             76 CALL_FUNCTION            0
             79 BUILD_CLASS         
             80 STORE_NAME               7 (MyClass)

There is not a single code block that does all the tasks related to
the class statement, and only it. In fact, there are two different
code objects to choose from, and neither really solves the problem.
One is the module object; as you've pointed out, it's too generic and
executes too much stuff, not just the class statement. The other one
is the <code object MyClass at 01209E60> referred to in the dis()
output above, that is the _body_ of the class statement. Calling it
doesn't call the metaclass, and neither does the name binding 'magic':

>>> exec MyClass.my_code
I'm inside my class definition!
# it does not print any of the debug messages in the metaclass, and
it's obvious why

It also doesn't take into account that the class may have base
classes. For example, changing the statement to 'class
MyClass(object)', the compiled code looks like this:

 27          64 LOAD_CONST               4 ('MyClass')
             67 LOAD_NAME                7 (object)
             70 BUILD_TUPLE              1

>From this point, few options are left. What I need to execute is in
fact a part of the module code object -- just one line of it, that's
the code that wraps the entire class statement. Besides giving up
(which is a reasonable option, given the constraints), what is left
are some horrible hacks. (I'm don't feel like discussing it here; it's
like Mordor Black Speech, if shall not be uttered in a public forum
:-).

BTW, the two code objects represent exactly the two alternative
solutions that can be done using only pure Python code:

1) reload the entire module (which is nearly equivalent to calling the
module code object, with the difference that it doesn't reload and
recompile the actual source code)

2) enclose the class statement inside a def, and run the def to obtain
the class template:

def BuildTemplate(<template parameters>):
    class MyTemplate(...):
        ...
    return MyTemplate

For my application, I'm tending towards (2), because it allows for a
better modularization of the code. But I'm sorry, in a way, because it
breaks the expect semantics in a way -- it's a function that returns a
class. It's valid, but some may regard it as 'ugly' and 'unpythonic'.

> > 3) I thought that the same technique could be made to work without a
> > metaclass, but I couldn't make it work. I haven't invested much time
> > to understand why -- it just works as implemented above. (It seems
> > that it does not bind correctly to the original name as it does if you
> > use the metaclass)
> 
> It appears to me, but I haven't tested this, that putting the inspect
> introspection in the classbody itself should be a sounder technique, as
> it should be able to find itself and stash itself away more reliably
> than just based on name (which is the best the metaclass can do).  The
> name binding issue is separate, however.

I've tried it yesterday, and it won't work, for some still unknown
reason. I could find the code object, but when I ran it, it would not
bind to the name in the same way as the metaclass would do. But it
doesn't matter at this point.
 
> > [1] Think about it as a templating mechanism, similar to the C++ one,
> > but for Python; this mechanism generates new classes that can be
> > instantiated as many times as needed. It's not the same thing as to
> > customize a instance during its initialization.
> 
> I agree, there's one metalevel of difference between instantiating a
> class to make an instance, and instantiating a metaclass to make a
> class.


-- 
Carlos Ribeiro
Consultoria em Projetos
blog: http://rascunhosrotos.blogspot.com
blog: http://pythonnotes.blogspot.com
mail: carribeiro at gmail.com
mail: carribeiro at yahoo.com