[Python-Dev] C AST to Python discussion

[Greg Ewing]

Brett Cannon wrote:
> One protects us from ending up with an unusable AST since
> the seralization can keep the original AST around and if the version
> passed back in from Python code is junk it can be tossed and the
> original version used.

I don't understand why this is an issue. If Python code
produces junk and tries to use it as an AST, then it's
buggy and deserves what it gets. All the AST compiler
should be responsible for is to try not to crash the
interpreter under those conditions. But that's true
whatever method is used for passing ASTs from Python
to the compiler.


   The PyObjects branch most likely won't have
> this since the actual AST will most likely be passed to Python code. 
> But there is performance issues with all of this seralization compared
> to a simple Pyobject pointer into Pythonland.  Jeremy supports the
> serialization option.  I am personally indifferent while leaning
> towards the serialization.
> 
> Then there is the API.  First we need to decide if AST modification is
> allowed or not.  It has been argued on my blog by someone (see
> http://sayspy.blogspot.com/2006/02/possibilities-of-ast.html for the
> entry on this whole topic which highly mirrors this email) that Guido
> won't okay AST transformations since it can lead to control flow
> changes behind the scenes.  I say that is fine as long as knowing that
> AST transformations are occurring are sufficiently obvious.  I say
> allow transformations.
> 
> Once that is settled, I see three places for possible access to the
> AST.  One is the command line like -m.  Totally obvious to the user as
> long as they are not just working off of the .pyc files.  Next is
> something like sys.ast_transformations that is a list of functions
> that are passed in the AST (and return a new version if modifications
> are allowed).  This could allow chaining of AST transformations by
> feeding the next function with another one.  Next is per-object AST
> access.  This could get expensive since if we don't keep a copy of the
> AST with the code objects (which we probably shouldn't since that is
> wasted memory if the AST is not used a lot) we will need to read the
> code a second time to get the AST regenerated.
> 
> I personally think we should choose an initial global access API to
> the AST as a starting API.  I like the sys.ast_transformations idea
> since it is simple and gives enough access that whether read-only or
> read-write is allowed something like PyChecker can get the access it
> needs.  It also allows for simple Python scripts that can install the
> desired functions and then compile or check the passed-in files. 
> Obviously write accesss would be needed for optimization stuff (such
> as if the peepholer was rewritten in Python and used by default), but
> we can also expose this later if we want.
> 
> In terms of 2.5, I think we really need to settle on the fate of the
> ast-objects branch.  If we can get the very basic API for exposing the
> AST to Python code in 2.5 that would be great, but I don't view that
> as critical as choosing on the final AST implementation style since
> wasting work on a version that will disappear would just plain suck. 
> It would be great to resolve this before the PyCon sprints since a
> good chunk of the AST-caring folk will be there for at least part of
> the time.
> 
> -Brett
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/greg.ewing%40canterbury.ac.nz