[Tutor] The Best Way to go About with Self Modifying Code/Code Generation?

[Steven D'Aprano]

Aaron Tp wrote:
> 
> Hey all, I have a question on using self-modifying code/code generation in Python;
> namely how to do it. 

I know others have already said not to do this, and to be honest I was going 
to say that same thing, but I have changed my mind. Buggrit, this is Python, 
where we are all consenting adults and you're allowed to shoot yourself in the 
foot if you want.

"Writing self-modifying code" actually covers a large range of techniques, 
ranging from the simple and straight-forward use of conventional factory 
functions, to the completely crazy self-modifying source code. There's a lot 
to learn. Rather than try to cover it all, I'll sketch out the various things 
you should learn about before you can make self-modifying code.

Obviously basic Python skills, how to read and write files, import modules, 
etc. That's the boring part, and I'm going to assume you know all that. If you 
don't, do a few tutorials and then come back.

Still here? Cool! On to the hard part. But first, beware:

     HERE BE DRAGONS


As fans of the British fantasy author Terry Pratchett all know, there are two 
types of dragons, swamp dragons and Draco nobilis, the noble dragon.

Swamp, or common, dragons are small, mostly harmless, with a very finicky 
digestive system and a regrettable tendency to blow up, but with a little bit 
of care and attention that can be occasionally useful.

The noble dragon, on the other hand, are huge, cunning, cruel, malicious, and 
anything but harmless. They don't so much blow up as silently fly down out of 
the sky and vapourize you with white-hot flame. They are best avoided by 
anyone other than experts. Actually, even experts should avoid them.

Writing self-modifying code is like dealing with dragons. And like dealing 
with dragons, you should start off with the fairly harmless variety before 
moving up to the more tricky type.

So start with this simple example of self-modifying code:

http://code.activestate.com/recipes/68429-ring-buffer/

This is probably as harmless and simple as self-modifying code can get: you 
have an instance which dynamically changes it's own class as it goes, so as to 
change its behaviour. Other languages might choke on this, but for Python, 
this counts as trivial.

If this example confuses you, if it's too hard for you to understand how it 
works, then you're not ready to try dragon taming. Learn some more Python 
first before you get into self-modifying code. We can help you with that.


Other simple techniques to start with include:

- dynamic code generation with factory functions and closures

- shadowing globals, built-ins or modules with your own code

- rebinding methods, functions or classes at runtime

- monkey-patching builtin functions

- functions or methods which rebind *themselves* at runtime


Feel free to ask for help on any of these. Or google on key phrases such as 
"monkey-patching", "shadowing builtins", "factory function", "closure".


So far, this all counts as the easy, conservative side of self-modifying code. 
Factory functions are even staid and boring. Shadowing is sometimes useful. 
Even monkey-patching, which is what those craz-ee Ruby hackers do, is almost 
respectable. Respectable like drinking two bottles of gin before going 
fox-hunting -- everybody knows you shouldn't do it, but if you survive the 
first time, you start to think nothing could possibly go wrong.

Be warned that "easy" is relative -- these can actually be fairly advanced 
techniques, and are tricky to get right and difficult to debug when they go 
wrong. And things *will* go wrong. Remember the warnings from the creator of 
the Unix operating system, Brian Kernighan:

     "Everyone knows that debugging is twice as hard as writing a
     program in the first place. So if you’re as clever as you can
     be when you write it, how will you ever debug it?"

     "Debugging is twice as hard as writing the code in the first
     place. Therefore, if you write the code as cleverly as possible,
     you are, by definition, not smart enough to debug it."


Okay. You've got your handgun, you're ready to shoot your own foot.
Once you've learned to tame the swamp dragons of runtime rebinding, 
monkey-patching and dynamic code generation, you're ready to move up to the 
next, slightly more radical, stage: eval and exec.

Here are some links to get you started:

http://lucumr.pocoo.org/2011/2/1/exec-in-python/
http://nedbatchelder.com/blog/201206/eval_really_is_dangerous.html
http://code.activestate.com/recipes/500261-named-tuples/


Still got all your toes? Excellent. Now you *might* be ready to go after the 
*really* big dragons:

- AST manipulation

http://www.dalkescientific.com/writings/diary/archive/2010/02/22/instrumenting_the_ast.html
http://docs.python.org/library/ast.html


- byte code hacks

http://nedbatchelder.com/blog/200804/wicked_hack_python_bytecode_tracing.html
http://wiki.python.org/moin/ByteplayDoc
http://code.activestate.com/recipes/498242/
http://www.voidspace.org.uk/python/weblog/arch_d7_2006_11_18.shtml#e553


- source code self-modification

http://discuss.fogcreek.com/joelonsoftware/default.asp?cmd=show&ixPost=44953
http://thedailywtf.com/Articles/Self-Modifying-VBA.aspx

And now you are ready for Mel.

http://www.catb.org/jargon/html/story-of-mel.html


Have fun!



-- 
Steven