[Edu-sig] Reloading code (was Re: OLPC: first thoughts)

[Paul D. Fernhout]

"Oh, what a giveaway! Did you hear that? Did you hear that, eh? That's 
what I'm on about!" :-)

This xreload module Guido supplies is similar to the approach I used for 
Jython in my earlier link.
http://www.arcknowledge.com/gmane.comp.lang.jython.user/2006-01/msg00023.html
If you look at that thread, I acknowledge Michael Spencer as having 
supplied some of the more sophisticated reload logic which is similar to 
what Guido has posted here (not sure where Michael got it from of course, 
nor where Guido got xreload from). See Michael's first related post here:
http://www.arcknowledge.com/gmane.comp.lang.jython.user/2006-01/msg00016.html

Notice xreload's update is much smaller than the one I made with Michael's 
additional inspiration and ideas:
http://www.arcknowledge.com/gmane.comp.lang.jython.user/2006-01/msg00023.html
In some ways, xreload's is better by including a general "Provide a hook 
for updating" concept. However xreload just ignores what to do with top 
level objects like lists or dicts -- xreload apparently throws away the 
updates to them. Notice it is not recursive -- for example does it handle 
nested classes? I'm not sure of all the reasons for the other differences 
-- it was tricky code I admit I never fully understood how to write 
thoroughly, mucking about with Python/Jython internals I only hazily 
understood. And even for the reload code I supplied, it is not always 
clear what to do with changes to top level objects -- do you want to 
update them or replace them? If you do update them, how deeply do you want 
the update to go?

Notice xreload still has sections with comments like "Not something we 
recognize, just give up" (that's throwing away changes to top level lists 
and dicts and singletons and such), or "Cop-out: if the type changed, give 
up", and it can not handle in general the issue of side effects I 
mentioned (e.g. is a singleton instance recreated?). And of course you 
can't use it to restart an exception after making a change (though it is 
perhaps a piece of that puzzle). It does however deal with updating 
pointers to existing function as far as I can tell (so previously set GUI 
callbacks will still call the old code) -- and does it in the same way the 
previous code linked to did (e.g. oldfunc.func_code = newfunc.func_code).

So, the xreload module Guido supplies is nifty piece of code to have 
(assuming Python licensed?). Using it properly will make you more 
productive. However, it still requires building hooks into your 
application explicitly to call it for each module of interest; the Jython 
version I supplied includes a brief GUI control window listing all modules 
which could be made to work similarly under wx or tk.

Still, elegant as it is (including the reload hook), even xreload does not 
handle all the important cases -- just (forgive me :-) the *easy* ones.

When Guido supplies an xrestart.py :-) python 2.5 code module that lets 
you restart exceptions after having modified one of the functions in the 
stack trace, then I will admit it is something I have never seen before in 
Python and be suitably impressed. Is it even possible without modifying 
the VM? :-)

There will remain the semantic problem of what to do with top level 
objects, but as I said, one can avoid that somewhat by just never 
expecting to reload an entire module at a time -- so using the core of 
these ideas to reload a method or class (which is what reload does, as it 
does not update other globals than code related ones).

And then, a next bigger step is getting both xreload and xrestart (or 
related code) into general use into common Python IDEs like IDLE or PyDev.
Otherwise, how is a beginner ever going to understand why you would want 
this xreload piece of code, or how it works, or when to use it? It's one 
thing to have a bit of code that suggest how something can be done -- it 
is another to weave an idea throughout a community and a set of 
application implementations. And the prevalence of those ideas of fine 
grained changes and fixing code using the debugger and then restarting is 
a big reason Smalltalk coders are more productive than Python coders, all 
other things being equal (which of course they are not given licensing, 
availability, libraries, community, and so on, which is why I use Python 
instead of Smalltalk. Can't stop pining for the fjords, I guess. :-).

One impressive thing about the Python design which I liked from all this 
was how Python separates the notion of execution code from a pointer 
reference. That is what makes all these reloading tricks possible. And my 
hat goes off to Guido for having included the extra level of indirection 
which makes this feasible. I can hope that generality and late bindingness 
might also make possible restarting an exception without VM changes.

--Paul Fernhout

Guido van Rossum wrote:
> On 2/24/07, Paul D. Fernhout <pdfernhout at kurtz-fernhout.com> wrote:
>>To step back for a minute, the fundamental problem here is that for
>>whatever reason a programmer wants to modify just one method of an already
>>loaded Python class (which came from a textual module which was loaded
>>already), save the change somewhere so it can be reloaded later
>>(overwriting part of the textual module?), and also have the program start
>>using the new behavior for existing instances without any other side
>>effects arising from recompiling this one change. In practice, this is
>>trivial to do in almost any Smalltalk system; it is hard if not impossible
>>to do in any widely used Python IDE or program (even when a Python shell
>>is embedded).
>
> # xreload.py.
> 
> """Alternative to reload().
> 
> This works by executing the module in a scratch namespace, and then
> patching classes, methods and functions.  This avoids the need to
> patch instances.  New objects are copied into the target namespace.
> """
> 
> import imp
> import sys
> import types
> 
> 
> def xreload(mod):
>     """Reload a module in place, updating classes, methods and functions.
> 
>     Args:
>       mod: a module object
> 
>     Returns:
>       The (updated) input object itself.
>     """
>     # Get the module name, e.g. 'foo.bar.whatever'
>     modname = mod.__name__
>     # Get the module namespace (dict) early; this is part of the type check
>     modns = mod.__dict__
>     # Parse it into package name and module name, e.g. 'foo.bar' and 'whatever'
>     i = modname.rfind(".")
>     if i >= 0:
>         pkgname, modname = modname[:i], modname[i+1:]
>     else:
>         pkgname = None
>     # Compute the search path
>     if pkgname:
>         # We're not reloading the package, only the module in it
>         pkg = sys.modules[pkgname]
>         path = pkg.__path__  # Search inside the package
>     else:
>         # Search the top-level module path
>         pkg  = None
>         path = None  # Make find_module() uses the default search path
>     # Find the module; may raise ImportError
>     (stream, filename, (suffix, mode, kind)) = imp.find_module(modname, path)
>     # Turn it into a code object
>     try:
>         # Is it Python source code or byte code read from a file?
>         # XXX Could handle frozen modules, zip-import modules
>         if kind not in (imp.PY_COMPILED, imp.PY_SOURCE):
>             # Fall back to built-in reload()
>             return reload(mod)
>         if kind == imp.PY_SOURCE:
>             source = stream.read()
>             code = compile(source, filename, "exec")
>         else:
>             code = marshal.load(stream)
>     finally:
>         if stream:
>             stream.close()
>     # Execute the code im a temporary namespace; if this fails, no changes
>     tmpns = {}
>     exec(code, tmpns)
>     # Now we get to the hard part
>     oldnames = set(modns)
>     newnames = set(tmpns)
>     # Add newly introduced names
>     for name in newnames - oldnames:
>         modns[name] = tmpns[name]
>     # Delete names that are no longer current
>     for name in oldnames - newnames - set(["__name__"]):
>         del modns[name]
>     # Now update the rest in place
>     for name in oldnames & newnames:
>         modns[name] = _update(modns[name], tmpns[name])
>     # Done!
>     return mod
> 
> 
> def _update(oldobj, newobj):
>     """Update oldobj, if possible in place, with newobj.
> 
>     If oldobj is immutable, this simply returns newobj.
> 
>     Args:
>       oldobj: the object to be updated
>       newobj: the object used as the source for the update
> 
>     Returns:
>       either oldobj, updated in place, or newobj.
>     """
>     if type(oldobj) is not type(newobj):
>         # Cop-out: if the type changed, give up
>         return newobj
>     if hasattr(newobj, "__reload_update__"):
>         # Provide a hook for updating
>         return newobj.__reload_update__(oldobj)
>     if isinstance(newobj, types.ClassType):
>         return _update_class(oldobj, newobj)
>     if isinstance(newobj, types.FunctionType):
>         return _update_function(oldobj, newobj)
>     if isinstance(newobj, types.MethodType):
>         return _update_method(oldobj, newobj)
>     # XXX Support class methods, static methods, other decorators
>     # Not something we recognize, just give up
>     return newobj
> 
> 
> def _update_function(oldfunc, newfunc):
>     """Update a function object."""
>     oldfunc.__doc__ = newfunc.__doc__
>     oldfunc.__dict__.update(newfunc.__dict__)
>     oldfunc.func_code = newfunc.func_code
>     oldfunc.func_defaults = newfunc.func_defaults
>     # XXX What else?
>     return oldfunc
> 
> 
> def _update_method(oldmeth, newmeth):
>     """Update a method object."""
>     # XXX What if im_func is not a function?
>     _update_function(oldmeth.im_func, newmeth.im_func)
>     return oldmeth
> 
> 
> def _update_class(oldclass, newclass):
>     """Update a class object."""
>     # XXX What about __slots__?
>     olddict = oldclass.__dict__
>     newdict = newclass.__dict__
>     oldnames = set(olddict)
>     newnames = set(newdict)
>     for name in newnames - oldnames:
>         setattr(oldclass, name, newdict[name])
>     for name in oldnames - newnames:
>         delattr(oldclass, name)
>     for name in oldnames & newnames - set(["__dict__", "__doc__"]):
>         setattr(oldclass, name,  newdict[name])
>     return oldclass
>