Running autogenerated code in another python instance

[Tom Anderson]

On Thu, 3 Nov 2005, Paul Cochrane wrote:

> On Wed, 02 Nov 2005 06:33:28 +0000, Bengt Richter wrote:
>
>> On Wed, 2 Nov 2005 06:08:22 +0000 (UTC), Paul Cochrane <cochrane at shake56.esscc.uq.edu.au> wrote:
>>
>>> I've got an application that I'm writing that autogenerates python 
>>> code which I then execute with exec().  I know that this is not the 
>>> best way to run things, and I'm not 100% sure as to what I really 
>>> should do.  I've had a look through Programming Python and the Python 
>>> Cookbook, which have given me ideas, but nothing has gelled yet, so I 
>>> thought I'd put the question to the community.  But first, let me be a 
>>> little more detailed in what I want to do:

Paul, this is a rather interesting problem. There are two aspects to it, 
which i believe are probably separable: getting instructions from the 
client to the server, and getting data back from the server to the client. 
The former is more complex, i think, and what's attracted the attention so 
far.

The first thing i'd say is that, while eval/exec is definitely a code 
smell, that doesn't mean it's never the right solution. If you need to be 
able to express complex things, python code might well be the best way to 
do it, and the best way to evaluate python code is eval/exec.

>> It's a little hard to tell without knowing more about your user input 
>> (command language?) syntax that is translated to or feeds the process 
>> that "autogenerates python code".
>
> It's basically just a command language I guess.

Hang on - the stuff that the user writes is what you're calling "pyvisi 
code", is that right? That doesn't look like 'just a command language', 
that looks like python, using a library you've written. Or is there 
another language, the "just a command language", on top of that?

And what you call "vtk-python code" - this is python again, but using the 
renderer's native library, right?

And you generate the vtk-python from the pyvisi-python by executing the 
pyvisi-python, there being (pluggable renderer-specific) logic in the guts 
of your pyvisi classes to emit the vtk-python code, right? You're not 
parsing anything?

>> There are lots of easy things you could do without generating and exec-ing
>> python code per se.
>
> I'd love to know of other options.  I like the idea of generating the 
> code one would have to write for a particular renderer so that if the 
> user wanted to, they could use the autogenerated code to form the basis 
> of a specific visualisation script they could then hack themselves.

If you want vtk-python code as an intermediate, i think you're stuck with 
eval/exec [1].

> One of the main ideas of the module is to distill the common visualisation
> tasks down to a simple set of commands, and then let the interface work out
> how to actually implement that.

Okay. There's a classic design pattern called Interpreter that applies 
here. This is one of the more complex patterns, and one that's rather 
poorly explained in the Gang of Four book, so it's not well-known.

Basically, the idea is that you provide classes which make it possible for 
a program to build structures encoding a series of instructions - 
essentially, you define a language whose concrete syntax is objects, not 
text - then you write code which takes such structures and carries out the 
instructions encoded in them - an interpreter, in other words.

For example, here's a very simple example for doing basic arithmetic:

# the language

class expression(object):
 	pass

class constant(expression):
 	def __init__(self, value):
 		self.value = value

class unary(expression):
 	def __init__(self, op, arg):
 		self.op = op
 		self.arg = arg

class binary(expression):
 	def __init__(self, op, arg_l, arg_r):
 		self.op = op
 		self.arg_l = arg_l
 		self.arg_r = arg_r

# the interpreter

UNARY_OPS = {
 	"-": lambda x: -x,
 	"|": lambda x: abs(x) # apologies for abnormal syntax
}

BINARY_OPS = {
 	"+": lambda l, r: l + r,
 	"-": lambda l, r: l - r,
 	"*": lambda l, r: l * r,
 	"/": lambda l, r: l / r,
}

def evaluate(expr):
 	if isinstance(expr, constant):
 		return expr.value
 	elif isinstance(expr, unary):
 		op = UNARY_OPS[expr.op]
 		arg = evaluate(expr.arg)
 		return op(arg)
 	elif isinstance(expr, binary):
 		op = BINARY_OPS[expr.op]
 		arg_l = evaluate(expr.arg_l)
 		arg_r = evaluate(expr.arg_r)
 		return op(arg_l, arg_r)
 	else:
 		raise Exception, "unknown expression type: " + str(type(expr))

# a quick demo

expr = binary("-",
 	binary("*",
 		constant(2.0),
 		constant(3.0)),
 	unary("-",
 		binary("/",
 			constant(4.0),
 			constant(5.0))))

print evaluate(expr)

This is by no means a useful or well-designed bit of code, and there are 
several things that could have been done differently (bare vs wrapped 
constants, operations defined by a symbol vs expression subtypes for each 
operation, etc), but i hope it gets the idea across - representing a 
language using an object graph, which lets you write programs that can 
speak that language.

Your code is already doing something a bit like this - you build scene 
graphs, then call render on them to get them to do something. Instead of 
that, you'd pass the whole scene to a renderer object, which would do the 
rendering (directly, rather than by generating code). The point is that 
the renderer could be in another process, provided you have a way to move 
the scene graph from one process to another - the pickle module, for 
example, or a custom serialisation format if you feel like reinventing the 
wheel.

An approach like this has a natural solution to your second problem, too - 
the evaluator function can return objects, which again can just be pickled 
and sent over the network.

tom

[1] Okay, so there is a way to do this without ever actually creating 
python code. You're not going to like this.

You need to apply the interpreter pattern to python itself. Well, a 
simplified subset of it. Looking at your generated vtk-python code, you 
basically do the following things:

- call methods with variables and literals as arguments
 	- throwing away the result
 	- or keeping it in a variable
- do for loops over ranges of integers

To make things a bit simpler, i'm going to add:

- getting attributes
- subscripting arrays
- return a value

You also need to do some arithmetic, i think; i leave that as an exercise 
for the reader.

So we need a language like:

class statement(object):
 	pass

class invoke(statement):
 	def __init__(self, var, target, method, args):
 		self.var = var # name of variable for result; None to throw away
 		self.target = target
 		self.method = method
 		self.args = args

class get(statement):
 	def __init__(self, var, target, field):
 		self.var = var
 		self.target = target
 		self.field = field

class subscript(statement):
 	def __init__(self, var, target, index):
 		self.var = var
 		self.target = target
 		self.index = index

class forloop(statement):
 	def __init__(self, var, limit, body):
 		self.var = var
 		self.limit = limit
 		self.body = body # tuple of statements

def return_(statement):
 	def __init__(self, value):
 		self.value = value

With which we can write a script like:

vtk_script = [
 	invoke("_plot", "vtk", "vtkXYPlotActor", ()),
 	invoke("_renderer", "vtk", "vtkRenderer", ()),
 	invoke("_renderWindow", "vtk", "vtkRenderWindow", ()),
 	invoke(None, "_renderWindow", "AddRenderer", ("_renderer")),
 	invoke(None, "_renderWindow", "SetSize", (640, 480)),
 	invoke(None, "_renderer", "SetBackground", (1, 1, 1)),
 	invoke("_x", None, "array", ([0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0],)),
 	get("vtk_vtkDataArray", "vtk", "vtkDataArray"),
 	get("vtk_VTK_FLOAT", "vtk", "VTK_FLOAT"),
 	invoke("_xData", "vtk_vtkDataArray", "CreateDataArray", ("vtk_VTK_FLOAT",)),
 	invoke("_x_len", None, "len", ("_x",)),
 	invoke(None, "_xData", "SetNumberOfTuples", ("_x_len",)),
 	invoke("_y0", None, "array", ([0.0, 1.0, 4.0, 9.0, 16.0, 25.0, 36.0, 49.0, 64.0, 81.0],)),
 	invoke("_y0Data", "vtk_vtkDataArray", "CreateDataArray", ("vtk_VTK_FLOAT",)),
 	invoke("_y0_len", None, "len", ("_y0",)),
 	invoke(None, "_y0Data", "SetNumberOfTuples", ("_y0_len",)),
 	forloop("i", "_x_len", (
 		subscript("_x_i", "_x", "i"),
 		invoke(None, "_xData", "SetTuple1", ("i", "_x_i")),)),
 	# etc
]

Or rather, your pyvisi classes can generate structures like this, exactly 
as they currently generate code.

Code to convert this into python source is trivial, so i'll gloss over 
that. The interpreter looks like this:

def isiterable(x):
 	return hasattr(x, "__iter__")

def execute(script, vars=None):
 	if (vars == None):
 		vars = {}
 		# initialise variables with 'vtk' and anything else you need
 	def decode_arg(arg):
 		if (isinstance(arg, str)):
 			return vars[arg]
 		elif (isiterable(arg)):
 			return map(decode_arg, arg)
 		else:
 			return arg
 	for stmt in script:
 		if (isinstance(stmt, invoke)):
 			target = vars[stmt.target]
 			method = getattr(target, stmt.method)
 			args = decode_arg(stmt.args)
 			result = method(*args)
 			if (stmt.var != None):
 				vars[stmt.var] = result
 		elif (isinstance(stmt, get)):
 			target = vars[stmt.target]
 			vars[stmt.var] = getattr(target, stmt.field)
 		elif (isinstance(stmt, subscript)):
 			target = vars[stmt.target]
 			vars[stmt.var] = getattr(target, decode_arg(stmt.index))
 		elif (isinstance(stmt, forloop)):
 			var = stmt.var
 			limit = decode_arg(stmt.limit)
 			body = stmt.body
 			for i in range(limit):
 				vars[var] = i
 				execute(body, vars)
 		elif (isinstance(stmt, return_)):
 			return vars[stmt.value]
 			# nb won't work from inside a for loop!
 			# you can use an exception to handle returns properly

Note that i haven't tested this, i've just written it off the top of my 
head, so it probably won't work, but maybe you get the idea.

To be honest, i'd go with exec.

-- 
Fitter, Happier, More Productive.