Limits of Metaprogramming

[Wilson]

On 4 Aug, 14:47, Tomasz Rola <rto... at ceti.com.pl> wrote:
> On Mon, 4 Aug 2008, Wilson wrote:
> > Hi all,
>
> Howdy,
>
> I am not sure if my remarks will be of any use for you, but here it goes.
>
> > I have an interesting problem that I'm hoping can be solved with
> > metaprogramming, but I don't know how far Python supports code
> > generation (and I don't know if I'm taking the correct approach
> > either... hence why I'm asking on this group):
>
> > I'd like to write a program that writes/manipulates a statemachine. My
> > idea was that I would define states in a .py file along with their
> > transitions and other metadata. I could then write a suite of programs
> > that would manipulate these states in interesting ways, such as
> > generate diagrams and documentation, test suites, do verifications and
> > also edit them. These programs would import the .py file and use
> > introspection upon the module's classes (states would be modelled as
> > classes). For example, a documentation tool could use graphviz to draw
> > the statediagram with the transition table and docstring...
>
> Maybe you have actually two problems.
>
> One problem is processing source code into something useful, but
> not executable. So it does not necessarily need metaprogramming to be
> solved, just some "code comprehension" (like get a python grammar,
> lex/yacc and read a lot of docs how to use them).

I do want my resultant code to be executable; I want to be able to use
"import" to get programmatical access to the classes and structures it
contains, such that they can be manipulated, added to, exercised and
drawn (using graphviz) via introspection... After any manipulation,
i'd like to be able to flatten them back into a .py file (as source
such that they are readable as Python code).

> > My problem is that I don't know if it's possible to edit these states
> > and then write them back to .py. Firstly, if my editing tool was to
> > create a new state, I would want to create the class (using type) and
> > attach it to the imported state module somehow. I can't find
> > information on whether this is possible. Secondly, I'd like to
> > manipulate the class and then use the inspect module to get the source
> > and write it back to file. Now I'm pretty sure the inspect module will
> > only write the source code as found in the original .py file. Is there
> > a way to generate the code for a class or other python objects in a
> > generic way? Or will I have to bite the bullet and write to custom
> > file and generate code from that (like glade with XML)? Or is there
> > another solution with templates that I'm over looking?
>
> Well, "everything" is doable in a Turing-complete language :-). However,
> this one problem - code generation and using it on the fly - may be better
> solved with some other language, perhaps?
>
> Not that I want to dissuade you (or anyone else) from using Python, but
> last time I checked the state of things (was a good couple of years ago, I
> wanted to solve some math-related problem with metaprogramming in Python)
> I decided I would do better if I learned Scheme. At least in Scheme,
> program structure is so simple, it is just a list (or a sequence of them),
> so manipulating code is IMHO much easier.
>
> I mean, even if in theory every language in Turing-complete class is equal
> to each other (one can do the same kind of stuff in one as in the other),
> in practice they are not equal at all. So doing everything in one ultimate
> language of choice is not very wise from my point of view.
>
> It does not mean you should immediately go to
>
> http://en.wikipedia.org/wiki/Scheme_(programming_language)
>
> but you may do so, and then make an informed choice by yourself. To be
> better informed, you may also google for more material - introductions,
> opinions, comparisons and some first-hand experience.
>
> On the other hand, writing "metafiles" and interpreters for them - ugh...
> :-). I have read somewhere, that some programmers resist using Scheme
> (or some other Lisp) for solving their problem and end up with writing
> their own Lisp-like (language|file format) and interpreter(s) for it. So
> you better watch yourself. If this effort is going to be something more
> permament than "done and go", then you will probably find yourself in the
> situation of reinventing the wheel, as your project goes more and more
> complex. If this is going to happen, you might use existing wheels and
> save some time for thinking on something better.

I appreciate these comments, and I know the quote that you are
refering to although I can't find it now... Something like:
" Every sufficiently large application has a poor/incomplete
implementation of LISP embedded within it ". I've looked at LISP
before and do appreciate its elegance, but Python has a beauty of its
own in its pragmatism, standard libraries and community.  So I'll
choose to stick with it.

> > Having the classes in memory, editing their attributes, adding methods
> > and then serialising them to code has an elegance to it in my mind and
> > would hopefully simplify things. The result would be directly
> > executable by the interpreter itself and editable by python
> > developers. But I fear that I may need an intermediary notation/format
> > for storage. Any advice/hints would be very useful.
>
> Yes, there is something elegant in this way of thinking, at least for me.
>
> As I (indirectly) said above, I am not up to date with recent Python
> abilities (I still use Python for a lot of things, I just do not overuse
> it for things, that it seems to be not so good at). I hope someone knows
> better and steps in to prove me wrong - it is always good to learn
> something new. Also, if you have just one problem, then learning a new
> language because of it is a bit too much (or maybe not, if you enjoy it
> :-) ). But if you plan doing similar or more difficult things in the
> future, then knowing Scheme may be good for you. Even if you choose to not
> use it, you will be aware of what can be done with it so this knowledge
> can provide some ready to use templates.

I've been through quite a few of the SICP lectures and again, do
appreciate its elegance and they have changed the way I program.

> And yes, there are also some other languages in Lisp family, but I think
> Scheme is best choice if you don't know any of them already. It is quite
> small, it is well defined (I mean, it has some real specification,
> instead of being "specified by its implementation") and there is a lot of
> info about it on the net that could be easily understood. And, last but
> not least, it has metaprogramming included with nice bunch of additional
> stuff. For a start, PLT's DrScheme looks nice (IMHO - yes, there are
> other nice looking Scheme implementations but this one is probably best
> fitted for a beginner):

> http://www.plt-scheme.org/
>
> Regards,
> Tomasz Rola

Thanks for your comments. But I believe my solution may lie in using a
template language such as cheetah. Too much is already invested in
Python!

Best Regards,
Paul