From bckfnn@worldonline.dk Thu May 23 13:07:44 2002 From: bckfnn@worldonline.dk (Finn Bock) Date: Thu, 23 May 2002 14:07:44 +0200 Subject: [Compiler-sig] More jython progress. Message-ID: <3CECDB90.3080901@worldonline.dk> Hi, I now got a version of jythonc running with the new AST. As always jythonc (the java sourcecode generator) is about twice as complicated as the java bytecode generator. Sigh. In jythonc, Jim Hugunin had added an additional abstraction layer on top of the anonymous SimpleNode ast tree that jython have used so far. With the new AST we no longer need the abstraction layer. Without that layer the code in jythonc became (IMO) a lot simpler and easier to follow. The new AST also means that the codegen in our two generators are far more similar than before. A huge win because we always have to implement new language features in both generators. IOW I'm very happy with the new AST and with the use of asdl as a way of describing the AST. Good work Jeremy. Since jythonc is written in python I would like to work towards an agreement on what the public python API to the AST should be like. What package/module should contain the AST node classes? In jython we have a scope analyzer that analyze a module and return symboltables for module, class and function levels. I guess CPython must also have some such thing. Should the compiler package have a public API to such scope analyzer? Similar we have future analyzer. Should we define a public API to this? What should be the public names of the AST node classes? I have used the name of the Constructors directly but I had to change the name of the 'except' production (to 'excepthandler'). Each field in a constructor is obviously a class instance attribute. Each asdl constructor becomes a java class. That means new attributes can't be added dynamicly. A strong limitation compared to a python class instance. It also means that the __XXX__ attributes can't be changed or depended on (such as __class__ and __name__). The sequences in the asdl becomes java arrays. That means that the elements in a sequence can be changed but the length of the sequence can't. I think we should ban all attempts to modify the sequences. Are these restrictions unacceptabe for the potential users of the AST tree? I would like to reach some type of agreement soon. regards, finn From bckfnn@worldonline.dk Thu May 23 13:28:16 2002 From: bckfnn@worldonline.dk (Finn Bock) Date: Thu, 23 May 2002 14:28:16 +0200 Subject: [Compiler-sig] Very experimental python AST tree and unpickling support. Message-ID: <3CECE060.5030007@worldonline.dk> Hi, Jeremy suggested that the CPython view of the AST tree could be implemented by pickling the C tree and unpickling a python tree. I have made such a python tree, mostly in order to reach agreement on how the python API of the tree should look like. You can find my work here: http://www.jython.org/ast/ Pythonast.py contains the tree classes including unpickle support and asdl_py.py is used to create the Pythonast.py. I have not given much though to the pickling format, it is just quick and dirty and I'm sure it is full of bugs. The only pickling code I have is located in the java tree so from jython I have create the 'os.py.pckl' which can be unpickled as a 'Module' like this: import Pythonast as ast print ast.unpickle(open("os.py.pckl", "rb")) Thoughs, comments and feedback welcome. regards, finn From ecn@metaslash.com Thu May 23 14:25:51 2002 From: ecn@metaslash.com (Eric C. Newton) Date: Thu, 23 May 2002 09:25:51 -0400 Subject: [Compiler-sig] More jython progress. In-Reply-To: <3CECDB90.3080901@worldonline.dk>; from bckfnn@worldonline.dk on Thu, May 23, 2002 at 02:07:44PM +0200 References: <3CECDB90.3080901@worldonline.dk> Message-ID: <20020523092551.A22805@ecn> > What package/module should contain the AST node classes? > What should be the public names of the AST node classes? I have used > the name of the Constructors directly but I had to change the name > of the 'except' production (to 'excepthandler'). Doesn't matter to me. > In jython we have a scope analyzer that analyze a module and return > symboltables for module, class and function levels. I guess CPython must > also have some such thing. The python version of the compiler has such a pass: compiler.symbols > Should the compiler package have a public API to such scope > analyzer? That would be helpful. > Similar we have future analyzer. Should we define a public API to this? What does the future analyzer do? > Each asdl constructor becomes a java class. That means new > attributes can't be added dynamicly. A strong limitation compared to > a python class instance. It also means that the __XXX__ attributes > can't be changed or depended on (such as __class__ and __name__). As long as isinstance() works, I can live with that. Right now the __class__ name is used for the generic visit() implementation, but that can be changed. > The sequences in the asdl becomes java arrays. That means that the > elements in a sequence can be changed but the length of the sequence > can't. I think we should ban all attempts to modify the sequences. I'm modifying the tree structure in only one case in PyChecker2, and I think I can live without it. I'm adding attributes to the tree _everywhere_. As PyChecker2 grinds over the tree, it adds discovered information to the tree (things like type information and cross-references). I can probably live without it by tracking the information somewhere else. -Eric From bckfnn@worldonline.dk Thu May 23 18:13:32 2002 From: bckfnn@worldonline.dk (Finn Bock) Date: Thu, 23 May 2002 19:13:32 +0200 Subject: [Compiler-sig] More jython progress. References: <3CECDB90.3080901@worldonline.dk> <20020523092551.A22805@ecn> Message-ID: <3CED233C.7040405@worldonline.dk> [Eric C. Newton] >>In jython we have a scope analyzer that analyze a module and return >>symboltables for module, class and function levels. I guess CPython must >>also have some such thing. > > The python version of the compiler has such a pass: compiler.symbols I didn't know that. Thank you for the info. >>Similar we have future analyzer. Should we define a public API to this? > > What does the future analyzer do? It simply return an object with properties 'nested_scope' and 'division' (in 2.1). It indicate which 'from __future__ import xxx' statement that was discovered in the module. >>The sequences in the asdl becomes java arrays. That means that the >>elements in a sequence can be changed but the length of the sequence >>can't. I think we should ban all attempts to modify the sequences. > > I'm modifying the tree structure in only one case in PyChecker2, and I > think I can live without it. The tree can still be modified, it is just the sequences that can't. It is perfectly valid to assign a new sequence (list or tuple) to a sequence attribute. I just want us to ban appending (and deleting) nodes to a sequence. > I'm adding attributes to the tree _everywhere_. As PyChecker2 grinds > over the tree, it adds discovered information to the tree (things like > type information and cross-references). I can probably live without > it by tracking the information somewhere else. Yeah, but you want to live without the ability to add new class instance attributes? In the old SimpleNode AST tree jython had over time added several attributes to the nodes (such as scope info). So it obviously is usefull. If we decide that the AST node must support attributes then I can also add it to jython. I can simple do a kind of deepcopy of the tree that returns real class instances in place of the java instances. It will just make the AST tree building from python a little slower. I don't think such dynamic attributes will survice across several compiler passes. If f.ex a peephole optimizer is implemented in C the tree must be pickled, optimized and unpickled. All dynamic attributes will be lost. Thank you for your comments. regards, finn From ecn@metaslash.com Fri May 24 02:19:12 2002 From: ecn@metaslash.com (Eric C. Newton) Date: Thu, 23 May 2002 21:19:12 -0400 Subject: [Compiler-sig] More jython progress. In-Reply-To: <3CED233C.7040405@worldonline.dk>; from bckfnn@worldonline.dk on Thu, May 23, 2002 at 07:13:32PM +0200 References: <3CECDB90.3080901@worldonline.dk> <20020523092551.A22805@ecn> <3CED233C.7040405@worldonline.dk> Message-ID: <20020523211912.A27767@ecn> > Yeah, but you want to live without the ability to add new class instance > attributes? Perhaps I'm just getting hung up on definitions. I was translating "class instance attributes" as "object attributes" since an "instance of a class" is called an "object". Now that I think about it, it could mean "attributes which are stored in classes". If I can add attributes to the objects which make up the tree, then great! I don't put any attributes in a class. > If we decide that the AST node must support attributes then I can also > add it to jython. I can simple do a kind of deepcopy of the tree that > returns real class instances in place of the java instances. It will > just make the AST tree building from python a little slower. Slower is not better. I prefer to work in python, even if I have a few more rules. If it gets slower, I'll be forced to go to C. I tried to work with the C version of the AST for a few hours. It wasn't pleasant. -Eric From bckfnn@worldonline.dk Fri May 24 12:22:14 2002 From: bckfnn@worldonline.dk (Finn Bock) Date: Fri, 24 May 2002 13:22:14 +0200 Subject: [Compiler-sig] More jython progress. References: <3CECDB90.3080901@worldonline.dk> <20020523092551.A22805@ecn> <3CED233C.7040405@worldonline.dk> <20020523211912.A27767@ecn> Message-ID: <3CEE2266.9010405@worldonline.dk> [Eric C. Newton] >>Yeah, but you want to live without the ability to add new class instance >>attributes? > > Perhaps I'm just getting hung up on definitions. > > I was translating "class instance attributes" as "object attributes" > since an "instance of a class" is called an "object". Now that I > think about it, it could mean "attributes which are stored in > classes". I meant it as "object attributes". We can't add new attributes to any of the nodes in the AST tree. I think that the correct python term isn't "object" (everything is an object, even classes) but "class instance". I got that from: http://www.python.org/doc/current/ref/types.html but maybe I'm wrong. >>If we decide that the AST node must support attributes then I can also >>add it to jython. I can simple do a kind of deepcopy of the tree that >>returns real class instances in place of the java instances. It will >>just make the AST tree building from python a little slower. > > Slower is not better. I prefer to work in python, even if I have a > few more rules. If it gets slower, I'll be forced to go to C. The "little slower" I was talking about would only be for jython and I don't think that the "little slower" speed would be much different from the speed of the pickling phase suggested by Jeremy. regards, finn From bckfnn@worldonline.dk Thu May 30 18:09:48 2002 From: bckfnn@worldonline.dk (Finn Bock) Date: Thu, 30 May 2002 19:09:48 +0200 Subject: [Compiler-sig] Jython progress. Message-ID: <3CF65CDC.9040104@worldonline.dk> Hi, Just to let you know, the AST patch for jython has now been integrated into jython CVS. We can still make changes to the AST, at least until we release jython-2.2, but I would like you to remember that for jython, the AST is now out of the sandbox. Obviously I hope that the new AST will be included in CPython-2.3. If it isn't, we only get a huge improvement in the internal jython codes. If the AST gets ready for 2.3 we will also get great compatibility with CPython. regards, finn