From hpk at trillke.net Fri Aug 1 13:03:50 2003 From: hpk at trillke.net (holger krekel) Date: Fri, 1 Aug 2003 13:03:50 +0200 Subject: [pypy-dev] Python parsers... In-Reply-To: ; from jriehl@cs.uchicago.edu on Wed, Jul 30, 2003 at 06:20:23PM -0500 References: <20030731003137.F32350@prim.han.de> Message-ID: <20030801130350.T32350@prim.han.de> Hi Jonathan, [Jonathan David Riehl Wed, Jul 30, 2003 at 06:20:23PM -0500] > On Thu, 31 Jul 2003, holger krekel wrote: > > > [Jeremy Hylton Wed, Jul 30, 2003 at 06:09:41PM -0400] > > > > > > Are you interested in integrating it into the compiler package? The Python > > > CVS trunk is now wide open for development, and the compiler package could > > > definitely use some tidying up. > > ... > > note, that we mirror the Python-CVS trunk into our svn-repository. > > This makes it easy to have a compiler-package within pypy and merge > > back and fro with CPython's compiler package. > > Heh, this would make three repositories for my code. I already have it > integrated into my Basil CVS tree. :) I guess the more the merrier! For starters, just mail me your prefered username and password and i set up an account and help you through occuring problems. I have repeatedly merged an experimental CPython-fork from Python-2.2 to Python-2.3 from our svn-CPython-mirror without any problems. > > However, i am not sure how many design goals a pypy-compiler and > > cpython-compiler will share in the longer run. E.g. we probably > > want the pypy-compiler to be configurable at run time, e.g. > > > > from pypy import compiler > > compiler.set_minimal() # only compile very basic constructs > > compiler.allow_listcomprehension() > > compiler.allow_classes() > > > > # set a different way to compile try/except/finally constructs [*] > > compiler.set_exception_compiler(MyExcCompiler) > > > > compiler.compile(testsource) # ... > > > > and CPython's compiler package probably doesn't have such > > design goals. But then again, it wouldn't hurt if CPython's > > compiler package could do such things :-) > > > > This is cool stuff Holger. Could you give me pointers as to where I > should head with this stuff? I've had pretty firm ideas in the past, but > I'm not sure how they'd be received. After reading about your GRAD/Path and Basil i think that you know much more about compiler & parser technology than me :-) Nevertheless, here are my *wishes* to a *compiler* package as i currently see it fit for PyPy. - separate "aspects" of compiling the python language out into classes so that they can be easily subclassed/replaced. - completly dynamic approach to bytecodes. Each "aspect" adds/references bytecodes including their implementation. Actually the aspect classes should generate the bytecode as a sequence of arbitrary Bytecode instances, i guess. - the list of bytecode instances might be converted into a Std-CPython code object or into an enriched PyPy-one (or into Jython/Java-code?) - i wonder if the enriched PyPy code objects could just be a pickle of of the 'internal compiler result' so that the PyPy-interpreter at runtime calls the 'code.create_frame' function passing in an ObjSpace and wrapped args/kwargs. and then the frame.eval-method execute the code object. This model might also fit with gatewaying to C or other languages. - there seems to be a chicken-and-egg problem with the implementation of a Bytecode instance 'B': it needs to be compiled, too. The closure of all Bytecode instances needed for this specific 'B' implementation should obviously not contain itself. Maybe bytecode implementations should be compiled to a different (restricted) set of Bytecodes alltogether? I haven't thought about how parser & Grammars kick into this picture but ... > For example, using the parse tree of > the input grammar, I either have or could easily create a tool that would > build a base tree walker class and then developers could derive custom > behavior from there. ... this sounds like an excellent starting point. From an experimentator's/implementor's POV it would probably be nice to put all aspects of providing a grammar providing a compile for the constructs in the grammar providing the implementations of the compiled bytecodes into one simple unit you can pass to the parser/compiler-package. I am unsure how decomposable a grammar like CPython's is. > Perhaps the compiler would not be stateful (in your > example you use several method/function calls to set compiler state) so > much as a module that exposes multiple tree walker classes. Sure, i didn't mean to suggest an actual API but just the functionality. > A lot of this technology is key to the Basil project as I was going to > build a control flow and data flow analysis toolset for C/C++/Python. Here Armin or others need to chime in. The whole abstract interpretation/annoated object space thing is still somewhat opaque to me. > FYI, as I stated above, the code (or most of it, anyway) is the Basil SF > CVS repository (basil.sf.net) if anyone wants a sneak peak (see the > basil.lang.python & basil.parsing packages.) The actual Python parser is > still hidden in a test of the PyPgen stuff and is built at run time, but > I should get around to pretty printing it and packaging it soon. "The SourceForge.net Website is currently down for maintenance." cheers, holger P.S.: If i used (PyPy) terminology you don't know or deem dubious please feel free to ask/correct me. So far we have carried pretty ok through all the vagueness but questioning and clarifying concepts can't hurt. From nathanh at zu.com Fri Aug 1 23:11:10 2003 From: nathanh at zu.com (Nathan Heagy) Date: Fri, 1 Aug 2003 15:11:10 -0600 Subject: [pypy-dev] another website update (design-wise) In-Reply-To: <20030724214647.R32350@prim.han.de> Message-ID: Seems more that reasonable to expect the url of the site to be somewhere in a message talking about it. On Thursday, July 24, 2003, at 01:46 PM, holger krekel wrote: > hello everybody, > > just a short notice that another cleanup-update to the web-page just > took > place. it's still far from perfect but now the layout has been > unified to not use tables (yay!). Btw, the diffs for webpage updates > are not sent to pypy-svn (unless somebody screems for those). > > At least all layout of the page elements should now be pretty stable > through all the applications we use (some cgi-scripts, roundup, > moin, mailman). > > I think it's a good time to cleanup the doc-hierarchy because > it is now nicely visible from the web :-) > > cheers, > > holger > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > > -- Nathan Heagy phone:306.653.4747 fax:306.653.4774 http://www.zu.com On June 2nd, zu.com communications was named Employer of the Year at the 2003 Canadian New Media Awards. From nathanh at zu.com Fri Aug 1 23:14:38 2003 From: nathanh at zu.com (Nathan Heagy) Date: Fri, 1 Aug 2003 15:14:38 -0600 Subject: [pypy-dev] another website update (design-wise) In-Reply-To: <20030724214647.R32350@prim.han.de> Message-ID: <28F42F08-C465-11D7-9DED-00039385F5E6@zu.com> There's a typo on the first page "beeing" should be "being" Is there a place to log bugs with the web site? Are any graphics needed? On Thursday, July 24, 2003, at 01:46 PM, holger krekel wrote: > hello everybody, > > just a short notice that another cleanup-update to the web-page just > took > place. it's still far from perfect but now the layout has been > unified to not use tables (yay!). Btw, the diffs for webpage updates > are not sent to pypy-svn (unless somebody screems for those). > > At least all layout of the page elements should now be pretty stable > through all the applications we use (some cgi-scripts, roundup, > moin, mailman). > > I think it's a good time to cleanup the doc-hierarchy because > it is now nicely visible from the web :-) > > cheers, > > holger > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > > -- Nathan Heagy phone:306.653.4747 fax:306.653.4774 http://www.zu.com On June 2nd, zu.com communications was named Employer of the Year at the 2003 Canadian New Media Awards. From hpk at trillke.net Sat Aug 2 10:56:35 2003 From: hpk at trillke.net (holger krekel) Date: Sat, 2 Aug 2003 10:56:35 +0200 Subject: [pypy-dev] another website update (design-wise) In-Reply-To: ; from nathanh@zu.com on Fri, Aug 01, 2003 at 03:11:10PM -0600 References: <20030724214647.R32350@prim.han.de> Message-ID: <20030802105634.Z32350@prim.han.de> [Nathan Heagy Fri, Aug 01, 2003 at 03:11:10PM -0600] > Seems more that reasonable to expect the url of the site to be > somewhere in a message talking about it. ok, http://codespeak.net/pypy If you find bugs with the web-page you can use our 'roundup' instance which is an issue-tracker. You can submit them either via a mail to pypy-issues at codespeak.net with a telling subject line (and [nosy=hpk] appended to it so that i get them - if they are website issues). Or you can go do http://codespeak.net/pypy/issues register a username there and create a new issue through the web. hth, holger From arigo at tunes.org Mon Aug 4 10:48:27 2003 From: arigo at tunes.org (Armin Rigo) Date: Mon, 4 Aug 2003 10:48:27 +0200 Subject: [pypy-dev] opcodes bundled in a class? In-Reply-To: <20030728142617.L32350@prim.han.de> References: <20030728142617.L32350@prim.han.de> Message-ID: <20030804084827.GA20817@magma.unil.ch> Hello Holger, On Mon, Jul 28, 2003 at 02:26:18PM +0200, holger krekel wrote: > with the builtinrefactor branch i am at the point where i want > to eliminate the 'appfile' concept alltogether in favour of > intermingling app-level and interp-level code directly. > (...) > However, is anybody against putting the opcodes/helpers in a class? As we discussed the other day on #pypy, there are in my opinion two different issues in your mail: how to get rid of the *_app.py files (including how to expose interpreter structures to app-level), and whether the opcodes should be moved into a class. Currently the second issue is (also) motivated by the fact that the code in the builtinrefactor branch can only mix app-level code as methods of an instance that have a 'space' attribute -- hence the push to make opcodes into a class. (Let me discuss the *_app issue in the next mail.) Independently of this problem I guess it is a good idea to make a class with the opcodes, instead of just a module like it is now. I'd even say that it should simply be a subclass of PyFrame, with PyFrame being an abstract class that has the logic to load and dispatch opcodes from a bytecode string but not the actual implementation of individual opcodes. In my point of view *code objects* are interpreter-level classes which are not tied to a particular object space (they are "lexical objects", i.e. essentially the same as the source code they represent); then a *function object* is a code object bound to a particular environment, i.e. with an object space, with default arguments, maybe with a closure for nested scopes, and so on. The function object, when called, creates a *frame object* and execute it. In PyPy there is only one function type, but several code object types: standard Python code object (app-level source code), built-in (i.e. implemented at interpreter-level), and possibly others like "using my own special opcodes". In this point of view, you can only call a function object (not a code object), but the function object needs to call a method build_frame() on the code object to create the appropriate kind of frame object. The resulting frame object should then have an eval() method to be actually run. I'm not sure where exactly the ExecutionContext comes into play; its role is to store the frames in the frame stack, but in Python built-in functions do not register frames there. Maybe it should be the role of the specific PyFrame class (implementing an app-level frame) to register and unregister itself into the frame stack. And what about generators ? I guess it would be nice if built-in functions could act as generators as well. A bientot, Armin. From arigo at tunes.org Mon Aug 4 11:13:06 2003 From: arigo at tunes.org (Armin Rigo) Date: Mon, 4 Aug 2003 11:13:06 +0200 Subject: [pypy-dev] opcodes bundled in a class? In-Reply-To: <20030728142617.L32350@prim.han.de> References: <20030728142617.L32350@prim.han.de> Message-ID: <20030804091306.GB20817@magma.unil.ch> Hello again, On Mon, Jul 28, 2003 at 02:26:18PM +0200, holger krekel wrote: > with the builtinrefactor branch i am at the point where i want > to eliminate the 'appfile' concept alltogether in favour of > intermingling app-level and interp-level code directly. Some issues we discussed (but couldn't conclude) over IRC with Holger about the details of: def app_f(x): return x+1 # to be run at app-level, i.e. interpreted by PyPy f = app2interp(app_f) ** Currently, this only works if app_f and f are methods of a class, whose instances must have a 'space' attribute. You can then call 'self.f(w_x)', where w_x is a wrapped object, and it automatically interprets app_f at app-level in the object space given by 'self.space'. My objection to this (beside the fact that it doesn't work for global functions) is that you must tie 'self' to a particular object space in the first place, which is not conceptually required for all interpreter-level objects. For example, I tend to think about code objects as "literal objects" representing some source code independently of any particular object space. So if we later want to run several object spaces in a single PyPy instance it will probably come in the way at some point. The alternative is to ask that an interpreter-level call should *always* specify 'space' explicitely (but still there is no 'space' argument in 'app_f', because it is implicit for all app-level code). So in the above example we would not change the definition of 'app_f(x)' or 'f', but call it with 'self.f(space, w_x)'. ** Visibility issues. Should anything that starts with 'app_' be visible at app-level ? It means that even "internal" helpers would become visible to user code (at least if the user can obtain a reference to the internal object). I tend to think that it is not a problem; in Python itself everything is visible by default (who uses the trick to make attributes and methods private to a class?). If really we want to hide an app-level helper, we can use name tricks: def hidden_app_f(x): return x+1 f = app2interp(hidden_app_f) This is invisible because it doesn't begin with 'app_', but still it is app-level code because of the app2interp() wrapper. ** app2interp() wrappers vs. attribute magic: it would be easy (with a __getattr__ or with metaclasses) to avoid the use of app2interp altogether: for any attribute 'app_foo' there would automatically be a corresponding interpreter-level attribute 'foo = app2interp(app_foo)'. Against this idea are arguments ranging from "this is magic" to "explicit is better than implicit". Also, an explicit call to app2interp() is the perfect place to give additional information if needed, e.g. "the argument 'x' must be an integer". This is something that we will need anyway at least for the converse wrapper interp2app(): class X: def f(self, space, w_obj, i): ... app_f = interp2app(f, typeof_i=int) (just tentative syntax) publishing an app-level method 'f' that can be called with two arguments (three including 'self'), saying that the interp-level implementation expects the last argument to be an unwrapped integer (thus the app-level call 'self.f("foo", "bar")' would give a TypeError). A bientot, Armin. From hpk at trillke.net Mon Aug 4 12:08:02 2003 From: hpk at trillke.net (holger krekel) Date: Mon, 4 Aug 2003 12:08:02 +0200 Subject: [pypy-dev] opcodes bundled in a class? In-Reply-To: <20030804084827.GA20817@magma.unil.ch>; from arigo@tunes.org on Mon, Aug 04, 2003 at 10:48:27AM +0200 References: <20030728142617.L32350@prim.han.de> <20030804084827.GA20817@magma.unil.ch> Message-ID: <20030804120802.N32350@prim.han.de> Hi Armin, [Armin Rigo Mon, Aug 04, 2003 at 10:48:27AM +0200] > On Mon, Jul 28, 2003 at 02:26:18PM +0200, holger krekel wrote: > > with the builtinrefactor branch i am at the point where i want > > to eliminate the 'appfile' concept alltogether in favour of > > intermingling app-level and interp-level code directly. > > > (...) > > However, is anybody against putting the opcodes/helpers in a class? > > As we discussed the other day on #pypy, there are in my opinion two different > issues in your mail: how to get rid of the *_app.py files (including how to > expose interpreter structures to app-level), and whether the opcodes should be > moved into a class. Currently the second issue is (also) motivated by the fact > that the code in the builtinrefactor branch can only mix app-level code as > methods of an instance that have a 'space' attribute -- hence the push to make > opcodes into a class. (Let me discuss the *_app issue in the next mail.) Please let me clarify the concepts behind the current builtinrefactor branch. The main point of the refactoring was to channel all calls to app-level code mainly through two classes: ScopedCode: a code object with a global and closure scope. you can execute ala c = ScopedCode(space, cpycodeobj, w_globals, closure_w) c.eval_frame(w_locals) this is currently needed in many places. Maybe it should be renamed to 'InterpretedCode' or so. InterpretedFunction: a function which will be interpreted at app-level. it derives from ScopedCode and you can use it ala: f = InterpretedFunction(space, cpyfunc, w_globals, closure_w) f.create_frame(w_args, w_kwargs) executioncontext.eval_frame(f) In fact there is a third in-between one, InterpretedFunctionFromCode, which provides initialization from a cpy-code object rather than from a cpy-function object. Therefore, the current builtinrefactor is not fundamentally requiring a 'space' attribute and the 'app2interp' definition is trivial to change to your suggestion (passing a space explicitely into app2interp-converted functions): class app2interp: def __init__(self, appfunc): self.appfunc = appfunc def __call__(self, space, w_args, w_kwargs): f = InterpretedFunction(space, self.appfunc) return f.eval_frame(w_args, w_kwargs) with which you have from def app_f(x): return x+1 f = app2interp(app_f) the desired explicit f(space, w_args, w_kwargs) interface. IOW, app2interp is just a thin wrapper around the lower-level machinery which was the real target of my refactoring efforts and reduced the LOC's of the interpreter-directory by 10% (more to follow). > Independently of this problem I guess it is a good idea to make a class with > the opcodes, instead of just a module like it is now. I'd even say that it > should simply be a subclass of PyFrame, with PyFrame being an abstract class > that has the logic to load and dispatch opcodes from a bytecode string but not > the actual implementation of individual opcodes. Yes, something like this. Also see the other compiler-discussion, lately. (i am currently a bit in a rush ...). cheers, holger From arigo at tunes.org Mon Aug 4 15:05:44 2003 From: arigo at tunes.org (Armin Rigo) Date: Mon, 4 Aug 2003 15:05:44 +0200 Subject: [pypy-dev] Python parsers... In-Reply-To: <20030801130350.T32350@prim.han.de> References: <20030731003137.F32350@prim.han.de> <20030801130350.T32350@prim.han.de> Message-ID: <20030804130544.GC20817@magma.unil.ch> Hello, On Fri, Aug 01, 2003 at 01:03:50PM +0200, holger krekel wrote: > - there seems to be a chicken-and-egg problem with the implementation > of a Bytecode instance 'B': it needs to be compiled, too. I think we should not worry about this problem. Currently we need CPython for bootstrapping. When we can create (using CPython) a stand-alone PyPy that can interpret regular Python code, then we can simply switch from CPython to that PyPy to bootstrap the next versions of PyPy. > > A lot of this technology is key to the Basil project as I was going to > > build a control flow and data flow analysis toolset for C/C++/Python. > > Here Armin or others need to chime in. The whole abstract > interpretation/annoated object space thing is still somewhat opaque to me. There is certainly room for both approaches to code analysis: the syntax one (a parse tree walker is cool), and the semantic one, which is the purpose of the annotated object space (abstract interpretation). The two approaches complete each other. The first one is suited to high-level analysis of the code, close to the source; we can reason about the constructs used in the source. It is good to check global properties of the code, or to do control flow analysis. The second one is closer to the actual execution, and I'd say it is more suited for things like type inference or data flow analysis because we follow step by step what the interpreter would do with real data, at a low level. In other words, abstract interpretation is excellent for questions like "what would the interpreter do at run-time if...". It cannot answer questions about the source. Our "annotation object space" tries to do type inference, and I thought we could then use another object space to translate the code to low-level C code using the information previously inferred. Now that I think about it it may be even nicer to combine the semantic analysis of the type inference with a *syntactic* translator to keep the higher-level "feeling" of the source code in C; not that it is really needed in our particular case, but I guess some would like it (e.g. for a Pyrex-like project). A bient?t, Armin. From arigo at tunes.org Mon Aug 4 15:18:44 2003 From: arigo at tunes.org (Armin Rigo) Date: Mon, 4 Aug 2003 15:18:44 +0200 Subject: [pypy-dev] opcodes bundled in a class? In-Reply-To: <20030804120802.N32350@prim.han.de> References: <20030728142617.L32350@prim.han.de> <20030804084827.GA20817@magma.unil.ch> <20030804120802.N32350@prim.han.de> Message-ID: <20030804131844.GD20817@magma.unil.ch> Hello Holger, On Mon, Aug 04, 2003 at 12:08:02PM +0200, holger krekel wrote: > Please let me clarify the concepts behind the current builtinrefactor > branch. The main point of the refactoring was to channel all calls to > app-level code mainly through two classes: > > ScopedCode: a code object with a global and closure scope. > (...) > InterpretedFunction: > a function which will be interpreted at app-level. > it derives from ScopedCode and you can use it ala: > (...) I still find this confusing. I believe that the distinction between code objects and functions in pretty fundamental in Python. A code object has no closure whatsoever (reference to global, to default args...). It is a lexical object. It can be stored in .pyc files or created by compile() out of pure text. In some other programming languages, Python's code objects are blocks of code (maybe whole function bodies, maybe smaller); and Python's function objects are what they call "closures", and as with Python's "def" statement they are created by capturing a block of code and some environment. (Python's "closure" only means the tuple of nested-scope variables going from the parent to the child function; in its broader sense "closure" refers to the whole captured environment, which includes in Python the globals and the default arguments). Armin From hpk at trillke.net Mon Aug 4 19:47:27 2003 From: hpk at trillke.net (holger krekel) Date: Mon, 4 Aug 2003 19:47:27 +0200 Subject: [pypy-dev] opcodes bundled in a class? In-Reply-To: <20030804131844.GD20817@magma.unil.ch>; from arigo@tunes.org on Mon, Aug 04, 2003 at 03:18:44PM +0200 References: <20030728142617.L32350@prim.han.de> <20030804084827.GA20817@magma.unil.ch> <20030804120802.N32350@prim.han.de> <20030804131844.GD20817@magma.unil.ch> Message-ID: <20030804194727.U32350@prim.han.de> [Armin Rigo Mon, Aug 04, 2003 at 03:18:44PM +0200] > Hello Holger, > > On Mon, Aug 04, 2003 at 12:08:02PM +0200, holger krekel wrote: > > Please let me clarify the concepts behind the current builtinrefactor > > branch. The main point of the refactoring was to channel all calls to > > app-level code mainly through two classes: > > > > ScopedCode: a code object with a global and closure scope. > > (...) > > InterpretedFunction: > > a function which will be interpreted at app-level. > > it derives from ScopedCode and you can use it ala: > > (...) > > I still find this confusing. I believe that the distinction between code > objects and functions in pretty fundamental in Python. A code object has no > closure whatsoever (reference to global, to default args...). right before you interpret it you must place it into a context of global(, closure) and local scopes. > It is a lexical object. 'ScopedCode' was not meant to be such a plain code object. It's rather wrapping a plain lexical code object (the cpy one) and tying it into a context of scopes. Maybe the name 'ScopedCode' is not perfect but without this concept you have a lot of boilerplate at various places dealing with code objects and scopes. To me it seemed that the steps 1. tying a code object to certain scopes 2. creating the frame / possibly parsing arguments into locals 3. executing the frame are a useful separation of concerns. cheers, holger From hpk at trillke.net Mon Aug 4 20:57:54 2003 From: hpk at trillke.net (holger krekel) Date: Mon, 4 Aug 2003 20:57:54 +0200 Subject: [pypy-dev] next sprint time Message-ID: <20030804205754.W32350@prim.han.de> Hello, on the last sprint in Belgium we discussed the next sprint place and time. The idea was End of September in Berlin. Because we couldn't find the right place in Berlin, yet, i asked someone in Amsterdam and he is now also checking if it's possible there. At least one person from the US wants to attend so Amsterdam wouldn'be the worst choice in that respect. However, even more important than the place is to find the right time. From the last discussions i take it that end of september/beginning of October is the earliest possible date. Also for quite a few people it's important to know the date as long in advance as possible. So i try to set up some reasanable choices: 22th-28th of September 29th-5th of October 5th-12th of October Please, if you are interested to come at any of the above the dates (or at all of them) please send me a private mail. I'll post the results by the end of the week and would pick the date where most people can come. From Jum and Armin i already have some information (and Armin seems to be able to come to any of the above but also wants to know early). If you have a better suggestion how to fix the next sprint time feel free to chime in. cheers, Holger -- From lac at strakt.com Mon Aug 4 23:56:54 2003 From: lac at strakt.com (Laura Creighton) Date: Mon, 4 Aug 2003 23:56:54 +0200 Subject: [pypy-dev] update Message-ID: <200308042156.h74LuslP016832@ratthing-b246.strakt.com> Jacob and I have managed to catch something, so far unknown, which has made us very sick, the sort where getting out of bed and making dinner is an accomplishment. We've been like this for more than a week now, for me more than two, and our brains have turned into potato salad. So we are very quiet, because most of the time, we are sleeping. back to bed, Laura From tismer at tismer.com Tue Aug 5 02:14:09 2003 From: tismer at tismer.com (Christian Tismer) Date: Tue, 05 Aug 2003 02:14:09 +0200 Subject: [pypy-dev] opcodes bundled in a class? In-Reply-To: <20030804084827.GA20817@magma.unil.ch> References: <20030728142617.L32350@prim.han.de> <20030804084827.GA20817@magma.unil.ch> Message-ID: <3F2EF6D1.8050109@tismer.com> Armin Rigo wrote: ... [stuff agreed] > In this point of view, you can only call a function object (not a code > object), but the function object needs to call a method build_frame() on the > code object to create the appropriate kind of frame object. The resulting > frame object should then have an eval() method to be actually run. I'm not > sure where exactly the ExecutionContext comes into play; its role is to store > the frames in the frame stack, but in Python built-in functions do not > register frames there. Maybe it should be the role of the specific PyFrame > class (implementing an app-level frame) to register and unregister itself into > the frame stack. And what about generators ? I guess it would be nice if > built-in functions could act as generators as well. Maybe I've read not enough of context, but if I only can call a function object, how are then classes and complete modules executed? In standard Python, a frame is created for a code object, together with parameters and additional info, which may, but needn't come from a function object (speaking of closures). Why not stick with this? ciao - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From arigo at tunes.org Tue Aug 5 09:00:30 2003 From: arigo at tunes.org (Armin Rigo) Date: Tue, 5 Aug 2003 00:00:30 -0700 (PDT) Subject: [pypy-dev] opcodes bundled in a class? In-Reply-To: <20030804194727.U32350@prim.han.de>; from hpk@trillke.net on Mon, Aug 04, 2003 at 07:47:27PM +0200 References: <20030728142617.L32350@prim.han.de> <20030804084827.GA20817@magma.unil.ch> <20030804120802.N32350@prim.han.de> <20030804131844.GD20817@magma.unil.ch> <20030804194727.U32350@prim.han.de> Message-ID: <20030805070030.ABDA01DD8D@bespin.org> Hello Holger, On Mon, Aug 04, 2003 at 07:47:27PM +0200, holger krekel wrote: > 'ScopedCode' was not meant to be such a plain code object. It's rather wrapping > a plain lexical code object (the cpy one) and tying it into a context of scopes. Then it is a function object... > Maybe the name 'ScopedCode' is not perfect but without this concept you > have a lot of boilerplate at various places dealing with code objects > and scopes. To me it seemed that the steps > > 1. tying a code object to certain scopes > 2. creating the frame / possibly parsing arguments into locals > 3. executing the frame > > are a useful separation of concerns. Indeed. As far as I understand it, (1.) was the reason why Python has separate notions of code objects and function objects. It is even clearer in CPython, where function objects are not built-in functions but really nothing more than a code object tied to a certain environment. It is essentially just a question of terminology so far, but then if 'ScopedCode' is actually a function object (which I guess you knew from the start, but I never actually understood that), you should consider that we thought about defining only a single function type in PyPy and varying just the code object types to reflect the different kinds of implementation. In gateway.py, on the other hand, you have created a whole hierarchy of subclasses of 'ScopedCode' (some with 'Function' in their name -- I really should have understood it earlier). All these classes' instances have a 'cpycode' attributes, but some subclasses expect it to be a code object with an interpretable bytecode. (BTW these ones have the 'cpycode' attribute repeated under another name, 'func_code'.) When we talked about having only one function type, the goal was to have a hierarchy similar to the one in gateway.py but with code objects instead. The (single) function class would only have the issues that do not depend on the kind of code object used for the implementation, mainly argument parsing. A bientot, Armin. From arigo at tunes.org Tue Aug 5 10:43:34 2003 From: arigo at tunes.org (Armin Rigo) Date: Tue, 5 Aug 2003 10:43:34 +0200 Subject: [pypy-dev] opcodes bundled in a class? In-Reply-To: <3F2EF6D1.8050109@tismer.com> References: <20030728142617.L32350@prim.han.de> <20030804084827.GA20817@magma.unil.ch> <3F2EF6D1.8050109@tismer.com> Message-ID: <20030805084334.GA19347@magma.unil.ch> Hello Christian, On Tue, Aug 05, 2003 at 02:14:09AM +0200, Christian Tismer wrote: > Maybe I've read not enough of context, but if I only can call > a function object, how are then classes and complete > modules executed? Point taken. In Python a code object can be executed either with PyEval_EvalCode(), which takes few arguments, or PyEval_EvalCodeEx(), which has tons of arguments. > In standard Python, a frame is created for a code object, together > with parameters and additional info, which may, but needn't > come from a function object (speaking of closures). > Why not stick with this? I suggest then the following interface: 1. code objects have a create_frame(w_globals) that creates a new empty frame; 2. function objects have a parse_args(frame, w_args, w_kwargs) that does the argument decoding dance and fills the given frame; 3. frame objects can be executed with eval(). So if we have no particular arguments to give to the code object, as in the case of class bodies and modules, we only do steps 1 and 3. The step 2 replaces the PyEval_EvalCodeEx() functionality by avoiding the need for a routine with hundreds of arguments (these arguments are exactly the fields of the function object). A bientot, Armin. From hpk at trillke.net Tue Aug 5 11:08:13 2003 From: hpk at trillke.net (holger krekel) Date: Tue, 5 Aug 2003 11:08:13 +0200 Subject: [pypy-dev] opcodes bundled in a class? In-Reply-To: <20030805070030.ABDA01DD8D@bespin.org>; from arigo@tunes.org on Tue, Aug 05, 2003 at 12:00:30AM -0700 References: <20030728142617.L32350@prim.han.de> <20030804084827.GA20817@magma.unil.ch> <20030804120802.N32350@prim.han.de> <20030804131844.GD20817@magma.unil.ch> <20030804194727.U32350@prim.han.de> <20030805070030.ABDA01DD8D@bespin.org> Message-ID: <20030805110813.X32350@prim.han.de> [Armin Rigo Tue, Aug 05, 2003 at 12:00:30AM -0700] > Hello Holger, > > On Mon, Aug 04, 2003 at 07:47:27PM +0200, holger krekel wrote: > > 'ScopedCode' was not meant to be such a plain code object. It's rather wrapping > > a plain lexical code object (the cpy one) and tying it into a context of scopes. > > Then it is a function object... really only if you pass arguments. There are code objects that are simply executed in a context of scopes - no argument parsing stuff whatsoever. > > Maybe the name 'ScopedCode' is not perfect but without this concept you > > have a lot of boilerplate at various places dealing with code objects > > and scopes. To me it seemed that the steps > > > > 1. tying a code object to certain scopes > > 2. creating the frame / possibly parsing arguments into locals > > 3. executing the frame > > > > are a useful separation of concerns. > > Indeed. As far as I understand it, (1.) was the reason why Python has separate > notions of code objects and function objects. It is even clearer in CPython, > where function objects are not built-in functions but really nothing more than > a code object tied to a certain environment. Hmmm. i think we have two cases for interpretation: 1. code objects, tied to a scope, executed (modules, exec statements ...) 2. function objects, tied to a scope, parsing arguments, executed/interpreted the design in gateway.py was meant to fit this model and because they share functionality 'InterpretedFunction' derives from 'ScopedCode'. Of course we could declare 1. as beeing a degenerated function (with no argument parsing) but somehow i felt that 'scoped code' objects are the real "one" building block. I wouldn't think of 'module code' execution as executing a 'function'. However, Armin, please feel free to just modify the branch so that it more suits your view. I don't think that we are far away from each other [*]. The real challenge probably is how to handle globals: how do app-level functions/data see each other? I would (try) approach this in a static way (rather than the very dynamic way i understood from our IRC discussions), e.g. construct a w_globals for every module and put all app name-bindings in there once and for all. For builtin-modules the globals should be on the class instance. Again, feel free to design this differently. cheers, holger [*] please note that 'InterpretedMethod' is just a hack for unittest_w because all the app-level unit-tests still have this (currently) useless 'self' argument. I didn't want to change all the tests unless there is a app/global/visibility concept. From arigo at tunes.org Sat Aug 9 14:51:46 2003 From: arigo at tunes.org (Armin Rigo) Date: Sat, 9 Aug 2003 14:51:46 +0200 Subject: [pypy-dev] opcodes bundled in a class? In-Reply-To: <20030805110813.X32350@prim.han.de> References: <20030728142617.L32350@prim.han.de> <20030804084827.GA20817@magma.unil.ch> <20030804120802.N32350@prim.han.de> <20030804131844.GD20817@magma.unil.ch> <20030804194727.U32350@prim.han.de> <20030805070030.ABDA01DD8D@bespin.org> <20030805110813.X32350@prim.han.de> Message-ID: <20030809125146.GA3068@magma.unil.ch> Hello Holger, Ok, I am working on refactoring things to our "almost-common" view :-) On Tue, Aug 05, 2003 at 11:08:13AM +0200, holger krekel wrote: > (...) > really only if you pass arguments. There are code objects that are > simply executed in a context of scopes - no argument parsing stuff > whatsoever. Yes, you are right here. I am now putting argument parsing code in a new Function class, where it seems to belong. Statements like 'exec code' should not create a function object at all, nor involve argument parsing. We have a potential difference with CPython here, however, because argument parsing is always done in CPython: e.g. >>> def f(x): pass ... >>> exec f.func_code Traceback (most recent call last): File "", line 1, in ? TypeError: f() takes exactly 1 argument (0 given) In the model corresponding to the refactoring, the 'exec' statement would work slightly differently. All frame objects now have getxxx() and setxxx() accessors to read or write data in various ways, including setlocaldict(d) and setlocalvar(index, value). The former sets the locals dictionary (and calls locals2fast if needed, to keep the fast variables in sync), which the latter directly sets the indexth local variable. The argument parsing stuff in Function calls setlocalvar() to initialize the frame, while the 'exec' statement calls setlocaldict(). It means that the following would work in PyPy but not in CPython : >>> def f(x): pass ... >>> exec f.func_code in {'x': 5} I cannot see a real problem in allowing this to work in PyPy, as it is somehow a more regular behavior; in CPython it explicitely doesn't work because of the fast locals optimization. Note that by extension I expect the following behavior in PyPy: >>> def f(x): ... print x ... >>> exec f.func_code in {} Traceback (most recent call last): File "", line 1, in ? File "", line 2, in f UnboundLocalError: local variable 'x' referenced before assignment Note that the error only shows up on the 'print x' statement. Anyone has a reason to forbid this ? The 'exec' statement seems quite low-level anyway, and I don't expect it to be normally used for code objects from 'func_code' attributes. Armin From tmeka at gmx.net Mon Aug 11 01:26:53 2003 From: tmeka at gmx.net (Tomek Meka) Date: 11 Aug 2003 01:26:53 +0200 Subject: [pypy-dev] Sprint in Berlin, 31.9. - 6/7.10.2003 Message-ID: <1060558014.1131.38.camel@tomek> Hello everybody! Now we have it! Thanks to the DG 2 FIRM (http://www.dg2.de/) we have 2 rooms, each 25 sqm, in the centre of Berlin, Ackerstr. 3a (Mitte). There are many cafes there, a university cafeteria is about ten minutes walk from there. There is also a cheap, clean hotel close-by (about 16 euros per night). I'll post you more details this week. It is also possible, that we (Christian and I) find free accomodation by our friends. Holger, could we arrange a wiki-page, so that everybody, who is coming to the sprint, could post up, when he/she is arriving, what is the flight number, phone number (mobile)? Bye for today! Tomek From hpk at trillke.net Mon Aug 11 08:52:08 2003 From: hpk at trillke.net (holger krekel) Date: Mon, 11 Aug 2003 08:52:08 +0200 Subject: [pypy-dev] Sprint in Berlin, 31.9. - 6/7.10.2003 In-Reply-To: <1060558014.1131.38.camel@tomek>; from tmeka@gmx.net on Mon, Aug 11, 2003 at 01:26:53AM +0200 References: <1060558014.1131.38.camel@tomek> Message-ID: <20030811085208.M32350@prim.han.de> Hello everybody and hello Tomek, [Tomek Meka Mon, Aug 11, 2003 at 01:26:53AM +0200] > Now we have it! Thanks to the DG 2 FIRM (http://www.dg2.de/) we have 2 > rooms, each 25 sqm, in the centre of Berlin, Ackerstr. 3a (Mitte). There > are many cafes there, a university cafeteria is about ten minutes walk > from there. There is also a cheap, clean hotel close-by (about 16 euros > per night). I'll post you more details this week. Cool. > It is also possible, > that we (Christian and I) find free accomodation by our friends. Holger, > could we arrange a wiki-page, so that everybody, who is coming to the > sprint, could post up, when he/she is arriving, what is the flight > number, phone number (mobile)? Btw, the sprint is supposed to take place from 29th of September till 4th of October 2003 and it's always nice to arrive a day earlier (on a sunday) so we can start monday morning. Here is a wiki page where all participants can fill in their details or add themselves: http://codespeak.net/moin/pypy/moin.cgi/SprintAttendants I already filled in the names of the people who signaled me that they want to come (6 people so far). Besides we have our traditional pypy-sprint list where organisational details will be communicated. You can subscribe here if you are interested to participate in the Berlin Sprint: http://codespeak.net/mailman/listinfo/pypy-sprint By the end of this week, I will unsubscribe anybody from the sprint-mailing list until i get notice or see the name in the wiki-page. I suggest, that beginning of next week we make a wider announcement once Tomek and Christian have sorted out the details and we have more info on the website. As to what might be the themes of the sprint: integrating parser and compiler technology into pypy and translating PyPy into Pyrex or C. Plus lots of other tasks :-) > Bye for today! > Tomek cheers, holger From roccomoretti at netscape.net Sun Aug 24 03:00:58 2003 From: roccomoretti at netscape.net (Rocco Moretti) Date: Sat, 23 Aug 2003 21:00:58 -0400 Subject: [pypy-dev] "Unwrap" Concidered Harmful Message-ID: <193BFA2F.746B29ED.9ADE5C6A@netscape.net> Maybe I overstate the subject line, but there are serious limitations on the .unwrap() concept. The limitation stems from the "blackbox" concept for object spaces. In general, we cannot be sure that unwrap will give a valid object. It may work for a direct representation of Python objects (such as trivial and standard object spaces), but when we start to use object space representations that do not exactly mirror Python semantics, unwrap isn't guarranteed to work. For example, if an annotation-like object space represents obects as type & bool (for truth tracking, for example), upon calling wrap(), it discards all information about value. So there is no way we can say that s.unwrap(s.wrap(x)) == x. Even if an object space keeps some kind of info on the wrapped object such that it can be unwrapped, there is no guarrantee that something valid will remain after any sort of operations on it, as a different object space may induce a different or multiple path(s) through the code. This is probably tractable within the object space itself, as any sort of vagarities in the wrapping can be compensated for on a space by space basis. However, if we wish to have a general code evaluation framework for multiple object spaces, the code evaluation code has to be general. At the moment there are a number of places in the code evaluation framework which call unwrap(). - I think it is mostly for unwrapping code objects. The reason I bring this up is I think that this is the reason the general builtin-module import code I posted earlier crashes for the annotation object space. The code attemped to unwrap attributes of the sys module, assuming that normal Python sematics are valid. This doesn't hold for annotation object space ... hence the crash. So if we wish to be general in the execution framework, we need to limit the use of unwrap() to uses which are guarranteed to be semantically valid by the obect space interface. (At least for the subset of object spaces we wish to use for a particular code execution framework.) A small but significant correllary to this is that the semantics of the operations within a general object space should not be assumed to be Python-like. That is, operations on object space obects should be performed for object space effects, not for execution level effects. This poses significant problems for the application space helpers to compute execution level relevant results. I'm not sure how to address this yet. -Rocco P.S. I appologize if this has been discused before on IRC. Has any progress been made in creating an archive of IRC logs? I get the impression a fair number of significant points have been made on IRC, and it would be best to archive them such that others not present can benefit from them as well. __________________________________________________________________ McAfee VirusScan Online from the Netscape Network. Comprehensive protection for your entire computer. Get your free trial today! http://channels.netscape.com/ns/computing/mcafee/index.jsp?promo=393397 Get AOL Instant Messenger 5.1 free of charge. Download Now! http://aim.aol.com/aimnew/Aim/register.adp?promo=380455