From florian.proff.schulze at gmx.net Sat Oct 4 22:34:25 2003 From: florian.proff.schulze at gmx.net (Florian Schulze) Date: Sat, 04 Oct 2003 22:34:25 +0200 Subject: [pypy-dev] Sprint results? Message-ID: Hi! How well did the sprint work out? I have seen that there is some pyrex code generation now and there are tests, but what where the results in this area during the sprint? Just a very short mail with some information would be grately appreciated. Regards, Florian Schulze From lac at strakt.com Sun Oct 5 16:52:36 2003 From: lac at strakt.com (Laura Creighton) Date: Sun, 05 Oct 2003 16:52:36 +0200 Subject: [pypy-dev] Sprint results? In-Reply-To: Message from Florian Schulze of "Sat, 04 Oct 2003 22:34:25 +0200." References: Message-ID: <200310051452.h95Eqah9003617@ratthing-b246.strakt.com> In a message of Sat, 04 Oct 2003 22:34:25 +0200, Florian Schulze writes: >Hi! > >How well did the sprint work out? > >I have seen that there is some pyrex code generation now and there are >tests, but what where the results in this area during the sprint? > >Just a very short mail with some information would be grately appreciated >. > >Regards, >Florian Schulze > >_______________________________________________ >pypy-dev at codespeak.net >http://codespeak.net/mailman/listinfo/pypy-dev The sprint went well. Before I got there, they got an inference engine working. It is now inferring things correctly. Its very pretty. Then I went and spoiled the party by making everybody work on funding issues. Nicolas Chauvat of Logilab and Alistair Burt of DKFI showed up and gave us the benefit of their wisdom having done EU funding things before. We are working on getting them in the project. This is exciting from the point of view of 'getting this funded' but it isn't technically all that interesting. I don't know about the rest of the gang, but I am dead tired. I will leave technical update to somebody more qualified, but wanted to let you know that yes, this is working. We will have another Sprint. Amsterdam looks good this time. Laura -- stuck in a maze of twisty little EU intellectual property pronouncements, all different :-) From hpk at trillke.net Sun Oct 5 18:36:32 2003 From: hpk at trillke.net (holger krekel) Date: Sun, 5 Oct 2003 18:36:32 +0200 Subject: infos from the berlin-sprint (was Re: [pypy-dev] Sprint results?) In-Reply-To: ; from florian.proff.schulze@gmx.net on Sat, Oct 04, 2003 at 10:34:25PM +0200 References: Message-ID: <20031005183632.B9501@prim.han.de> Hi Florian, [Florian Schulze Sat, Oct 04, 2003 at 10:34:25PM +0200] > Hi! > > How well did the sprint work out? > > I have seen that there is some pyrex code generation now and there are > tests, but what where the results in this area during the sprint? > > Just a very short mail with some information would be grately appreciated. Here is my take. Other mileages may vary so excuse me if i miss anything. On Monday morning we made a few design decision which led to the implementation of the following abstractions in the next two days: - a new FlowObjSpace which does abstract interpretation plus some very nice tricks (which we came up with during a long-winded discussion in a restaurant :-) to construct a FunctionGraph. This functiongraph (fully) represents the abstract or symbolic execution of a function. e.g. for this function: def while_func(i): total = 0 while i > 0: total = total + i i = i - 1 return total the following graph is generated (shown here in an slightly optimized version): http://codespeak.net/~hpk/while_func.ps - 'make_dot.py' takes this flowmodel and generates a nice graphical graph from it (see above urls) - the pyrex-translator also takes this objectmodel (in flowmodel.py) and generates Pyrex-Code from it. The generated code looks pretty low-level but this is expected as we eventually want to generate C or assembly directly. For the above function the following pyrex-source code is generated (again with some easy optimizations applied): def while_func(object v413): v419, v420 = v413, 0 cinline "Label1:" v422 = v419 > 0 if v422: v424 = v420 + v419 v425 = v419 - 1 v419, v420 = v425, v424 cinline "goto Label1;" else: return v420 btw, the 'cinline' statement is a hack to pyrex and allows to insert arbitrary C-code. An objectspace cannot really identify loops and so we need "goto". We consider goto to be useful unless you have to type and understand them manually :-) - translator/annotation.py also takes the flowmodel and applies a new technique for type inference: it uses space-operations to note 'assertions' about variables and relaxes those assertions during analysis of the flowgraph. IOW we didn't come up with yet another type-system (which is the classical approach) but reuse the notion of "space-operations" which we had from the beginning of the project. Btw, Armin thinks that this type-inference algorithm is worth a scientific paper but more about this either later and/or from him. - we adapted Jonathan David Riehl's Python-Parser (written completly in python using its own "rex"-approach) and adapted it so that it will be a drop-in replacement for CPython's current parser (living the boring life of a C-extension). Actually Jonathan's larger 'basil' project is now in the codespeak-repository and we can easily link it into PyPy or branch off it if neccessary. So alltogether the Flowgraph/Functiongraph/flowmodel (there is no completly fixed terminology yet) is the central point for several independent algorithms that - if combined - eventually produce typed C-code. To sum it up there are the following abstractions: interpreter interpreting bytecode, dispatching operations on objects to objectspace implementing operations on boxed objects stdobjspace a concrete space implementing python's standard type system flowobjspace a conrete space performing abstract/symbolic interpretation and producing a (bytecode-indepedent) flowmodel of execution annotator analysing the flowmodel to infer types. genpyrex taking the (annotated) flowmodel to generate pyrex-code pyrex translating into an C-extension As a consequence the former Ann(otation)Space has been ripped apart (partly into flowobjspace) and is gone now. Long live the flowspace. A really nice property of the above abstractions is that they allow development and testing *independently* from one another which was of invaluable help. Thanks here go to Greg Ewing for Pyrex and sorry for the evil cinline-hack :-) Anybody interested in helping with the next steps might look into the TODO file in the pypy-root directory. We also have discussed yesterday evening a refactored flowmodel which we want to employ soon. Big thanks go to Tomek Meka and Christian Tismer for organizing the sprint and Stephan Diehl and Dinu Gherman for their help in various organizational areas. And especially to Jonathan David Riehl who made it from Chicago. We hope he can stay with us more often. And here is a (hopefully complete) list of people who attended and made all of the above possible: Armin Rigo Christian Tismer Dinu Gherman Guenter Jantzen Jonathan David Riehl Samuele Pedroni Stephan Diehl Tomek Meka and shame on me if i forgot anyone (i am tired ...) And of course many many thanks to Laura Creighton (AB Strakt), Nicolas Chauvat (Logilab) and Alistair Burt (DFKI) who tried hard to work with us on EU-funding-issues. Actually we came up with a nice technical 2-year plan but a lot of business issues still need to be resolved and fixed. Let's hope that the EU-funding effort is as successful as our coding sprints this year has been. Ah yes, the next sprint we hope to do mid-december probably in Amsterdam. If all goes well (some more people helping between the sprints that is :-) we might even do a first public release with PyPy prototypically running as a C-extension to CPython. That's it for now from me. (sprinters: Please correct/fix any issues i misrepresented) cheers, holger From hpk at trillke.net Sun Oct 5 18:46:45 2003 From: hpk at trillke.net (holger krekel) Date: Sun, 5 Oct 2003 18:46:45 +0200 Subject: [pypy-dev] checking out PyPy changed Message-ID: <20031005184645.C9501@prim.han.de> hello again, just a short but important info. The berlin sprint hacks resulted in a few external packages we are using now (sometimes slightly adapted for our evil purposes). This means that you absolutely need to checkout using this url: http://codespeak.net/svn/pypy/trunk/src and NOT .../src/pypy which wouldn't give you the external packages that PyPy currently depends on. These packages are bound into the pypy-project via the notion of 'svn:externals'. this is a property you can stick to a subversion-controled directory. When doing 'svn up' or 'svn co' on such a directory the external items will be fetched and integrated into your working-copy. Here is the current list of packages we use: Pyrex slightly hacked version of Pyrex-0.8.2 Plex the Lexer that Pyrex uses vpath a package that helps with filename/file-manipulation Soon there will also be an external to a subtree of Jonathan D. Riehls Python-Parser (which is part of Basil now living in http://codespeak.net/svn/basil). This will allow us to do the complete tokenize/parser/compile cycle with plain python packages. cheers, holger From florian.proff.schulze at gmx.net Sun Oct 5 21:21:59 2003 From: florian.proff.schulze at gmx.net (Florian Schulze) Date: Sun, 05 Oct 2003 21:21:59 +0200 Subject: [pypy-dev] Re: infos from the berlin-sprint (was Re: Sprint results?) References: <20031005183632.B9501@prim.han.de> Message-ID: On Sun, 5 Oct 2003 18:36:32 +0200, holger krekel wrote: > Hi Florian, Hi! First: Thanks to Laura and Holger for the replies! > So alltogether the Flowgraph/Functiongraph/flowmodel (there is no > completly fixed terminology yet) is the central point for several > independent algorithms that - if combined - eventually produce typed > C-code. > > To sum it up there are the following abstractions: > > interpreter interpreting bytecode, dispatching operations on objects > to > > objectspace implementing operations on boxed objects > > stdobjspace a concrete space implementing python's standard type > system > > flowobjspace a conrete space performing abstract/symbolic > interpretation and > producing a (bytecode-indepedent) flowmodel of execution > > annotator analysing the flowmodel to infer types. > > genpyrex taking the (annotated) flowmodel to generate pyrex-code > > pyrex translating into an C-extension Just a few questions about this: As far as I can tell there is no script yet, which just takes a python file and generates Pyrex code, correct? What would be needed? - a way to get functions from a python module - build_flow from the flow object space which takes those functions and generates a graph - GenPyrex from genpyrex.py which takes that graph and emits a pyrex function I tried out the following with a normal python interpreter in the pypy/src directory: >>> import pypy.objspace.flow >>> space = pypy.objspace.flow.Space() >>> import pypy.translator.genpyrex >>> def f(): ... i = 0 ... while i < 10: ... i = i + 1 ... return i ... >>> print pypy.translator.genpyrex.GenPyrex(space.build_flow(f)).emitcode() def f(): v21 = 0 cinline "Label2:" v22 = v21 < 10 if v22: v24 = v21 + 1 v21 = v24 cinline "goto Label2;" else: v27, v28 = v21, v22 v29 = v27 return v29 Which is quite impressive in my opinion. Regards, Florian From hpk at trillke.net Sun Oct 5 22:05:29 2003 From: hpk at trillke.net (holger krekel) Date: Sun, 5 Oct 2003 22:05:29 +0200 Subject: [pypy-dev] Re: infos from the berlin-sprint (was Re: Sprint results?) In-Reply-To: ; from florian.proff.schulze@gmx.net on Sun, Oct 05, 2003 at 09:21:59PM +0200 References: <20031005183632.B9501@prim.han.de> Message-ID: <20031005220529.D9501@prim.han.de> [Florian Schulze Sun, Oct 05, 2003 at 09:21:59PM +0200] > Just a few questions about this: As far as I can tell there is no script > yet, which just takes a python file and generates Pyrex code, correct? > What would be needed? > - a way to get functions from a python module > - build_flow from the flow object space which takes those functions and > generates a graph > - GenPyrex from genpyrex.py which takes that graph and emits a pyrex > function Maybe i should have pointed out that we are not at the point where pypy takes any (RPython) python function, let alone unrestricted python. > I tried out the following with a normal python interpreter in the pypy/src > directory: > > >>> import pypy.objspace.flow > >>> space = pypy.objspace.flow.Space() > >>> import pypy.translator.genpyrex > >>> def f(): > ... i = 0 > ... while i < 10: > ... i = i + 1 > ... return i > ... it is interesting to then continue with: >>> flow = space.build_flow(f) >>> genpyx = pypy.translator.genpyrex.GenPyrex(flow) >>> genpyx.annotate([]) >>> print genpyx.emitcode() def f(): cdef int i_v4 cdef int i_v8 cdef int i_v1 cdef int i_v7 cdef int i_v2 cdef int i_v9 i_v1 = 0 cinline "Label2:" i_v2 = i_v1 < 10 if i_v2: i_v4 = i_v1 + 1 i_v1 = i_v4 cinline "goto Label2;" else: i_v7, i_v8 = i_v1, i_v2 i_v9 = i_v7 return i_v9 which means you also get the type information. cheers, holger From arigo at tunes.org Tue Oct 7 15:01:07 2003 From: arigo at tunes.org (Armin Rigo) Date: Tue, 7 Oct 2003 14:01:07 +0100 Subject: [pypy-dev] svn with HTTP proxy Message-ID: <20031007130107.GA23916@vicky.ecs.soton.ac.uk> Hello everybody, In case someone has the same problems than me with svn because of a broken HTTP proxy that doesn't understand the extended webdav commands, here is a fix (I also need input as to whether this fix is "clean" enough). Edit your ~/.subversion/servers files to contain: [groups] codespeak = codespeak.net [codespeak] http-proxy-host = codespeak.net http-proxy-port = 8080 This redirects all requests to codespeak.net to the port 8080. The fix is kind of a hack, because codespeak.net:8080 is not really an HTTP proxy server but an HTTP server on its own, but it seems to work perfectly. With this hack I no longer have to use the svn url http://codespeak.net:8080/svn/. The problem with it was that it only worked for the "internal" part of the repository; external items are explicitly referenced with an url of the form http://codespeak.net/svn. The above hack thus forces svn to redirect *all* connections to codespeak.net to port 8080. A bientot, Armin. From tinuviel at sparcs.kaist.ac.kr Tue Oct 7 18:07:26 2003 From: tinuviel at sparcs.kaist.ac.kr (Seo Sanghyeon) Date: Wed, 8 Oct 2003 01:07:26 +0900 Subject: [pypy-dev] Killing newlines in pyx source Message-ID: <20031007160726.GA20195@sparcs.kaist.ac.kr> Hello, I'm now playing with translator stuffs from the sprint. I tried to read emitted Pyrex code, and found that many annoying empty lines are generated after def line. (They are for type declarations I guess?) My fix is attached. -------------- next part -------------- Index: pypy/translator/genpyrex.py =================================================================== --- pypy/translator/genpyrex.py (revision 1605) +++ pypy/translator/genpyrex.py (working copy) @@ -144,7 +144,9 @@ #self.putline("# %r" % self.annotations) for var in self.variablelocations: if var not in fun.startblock.input_args: - self.putline(self._vardecl(var)) + decl = self._vardecl(var) + if decl: + self.putline(decl) self.indent -= 1 self.lines.extend(functionbodylines) From hpk at trillke.net Tue Oct 7 18:16:35 2003 From: hpk at trillke.net (holger krekel) Date: Tue, 7 Oct 2003 18:16:35 +0200 Subject: [pypy-dev] Killing newlines in pyx source In-Reply-To: <20031007160726.GA20195@sparcs.kaist.ac.kr>; from tinuviel@sparcs.kaist.ac.kr on Wed, Oct 08, 2003 at 01:07:26AM +0900 References: <20031007160726.GA20195@sparcs.kaist.ac.kr> Message-ID: <20031007181635.X9501@prim.han.de> Hi Seo, [Seo Sanghyeon Wed, Oct 08, 2003 at 01:07:26AM +0900] > Hello, > > I'm now playing with translator stuffs from the sprint. I tried > to read emitted Pyrex code, and found that many annoying empty > lines are generated after def line. (They are for type declarations > I guess?) > > My fix is attached. thanks. applied. holger From hpk at trillke.net Fri Oct 10 11:41:46 2003 From: hpk at trillke.net (holger krekel) Date: Fri, 10 Oct 2003 11:41:46 +0200 Subject: [pypy-dev] recent lisp support / cross-language Message-ID: <20031010114146.U9501@prim.han.de> hello pypy, i think it's nice that we start to have clisp support. but there are no unit-tests and 'translator/gencl.py' adds more code to keep in sync with our model. So without tests i wouldn't like this code to exist. I guess we need some boilerplate that allows starting lisp-programs from commandline with instructions which functions to call and parsing the results. the unittests could then do the same what they do for pyrex: checking that the function at least computes correctly. btw, I can imagine a cross-language sprint aimed at producing multiple backends (C, Pyrex, CLISP, whatever) or multiple frontends (prolog...). That would certainly help to get the wider language research/developer community get interested. A dumb side-question: can clisp-source be annotated with types or is it just another VHLL :-) ? cheers, holger P.S.: if Seo wants to maintain the lisp area for the time beeing i am for giving him commit privs. From mwh at python.net Fri Oct 10 12:25:48 2003 From: mwh at python.net (Michael Hudson) Date: Fri, 10 Oct 2003 11:25:48 +0100 Subject: [pypy-dev] Re: recent lisp support / cross-language References: <20031010114146.U9501@prim.han.de> Message-ID: <2mwubdv177.fsf@starship.python.net> holger krekel writes: > hello pypy, > > i think it's nice that we start to have clisp support. but there > are no unit-tests and 'translator/gencl.py' adds more code to keep > in sync with our model. So without tests i wouldn't like this code > to exist. Yeah, perhaps it shouldn't have been checked in. But it's not very much code, and it's quite neat. Oh, and don't say "clisp": that's the name of an *implementation* of Common Lisp. If you can't be bothered to type "Common Lisp" out, "CL" is a better abbreviation. > I guess we need some boilerplate that allows starting lisp-programs > from commandline with instructions which functions to call and > parsing the results. Unfortunately, the CL world being what it is, this depends on which implementation you have installed... we could demand you set an env var appropriately. > the unittests could then do the same what they do for pyrex: > checking that the function at least computes correctly. > > btw, I can imagine a cross-language sprint aimed at producing multiple > backends (C, Pyrex, CLISP, whatever) or multiple frontends (prolog...). > That would certainly help to get the wider language research/developer > community get interested. Yep! Count me in for PPC assembler :-) > A dumb side-question: can clisp-source be annotated with types > or is it just another VHLL :-) ? Yeah, CL has type declarations. Whether they make any difference is (spot a pattern?) implementation dependent, but they can make a huge difference in, e.g., CMUCL. > P.S.: if Seo wants to maintain the lisp area for the time beeing i am > for giving him commit privs. Me too! Cheers, mwh -- That's why the smartest companies use Common Lisp, but lie about it so all their competitors think Lisp is slow and C++ is fast. (This rumor has, however, gotten a little out of hand. :) -- Erik Naggum, comp.lang.lisp From gherman at darwin.in-berlin.de Fri Oct 10 12:42:06 2003 From: gherman at darwin.in-berlin.de (Dinu Gherman) Date: Fri, 10 Oct 2003 12:42:06 +0200 Subject: [pypy-dev] Xerox Parc software papers Message-ID: <6461E3D3-FB0E-11D7-A9D6-00039345C610@darwin.in-berlin.de> Hi, I just found this interesting repository of papers by Xerox Parc members, quite a few about issues more or less relevant to PyPy: http://www2.parc.com/csl/groups/sda/publications.shtml Regards, Dinu -- Dinu C. Gherman ...................................................................... "It was wonderful to find America, but it would have been more wonder- ful to miss it." (Mark Twain) From Alexandre.Fayolle at logilab.fr Fri Oct 10 13:03:55 2003 From: Alexandre.Fayolle at logilab.fr (Alexandre Fayolle) Date: Fri, 10 Oct 2003 13:03:55 +0200 Subject: [pypy-dev] gnu lightning Message-ID: <20031010110354.GQ23519@calvin> Hello Your probably aware of this already, but then again, maybe not. I've just stumbled across the gnu lightning project http://www.gnu.org/software/lightning/lightning.html "GNU lightning is a library that generates assembly language code at run-time; it is very fast, making it ideal for Just-In-Time compilers, and it abstracts over the target CPU, as it exposes to the clients a standardized RISC instruction set inspired by the MIPS and SPARC chips." And I thought this may be something quite useful for pypy or for psyco. -- Alexandre Fayolle LOGILAB, Paris (France). http://www.logilab.com http://www.logilab.fr http://www.logilab.org D?veloppement logiciel avanc? - Intelligence Artificielle - Formations From hpk at trillke.net Fri Oct 10 13:09:56 2003 From: hpk at trillke.net (holger krekel) Date: Fri, 10 Oct 2003 13:09:56 +0200 Subject: [pypy-dev] gnu lightning In-Reply-To: <20031010110354.GQ23519@calvin>; from Alexandre.Fayolle@logilab.fr on Fri, Oct 10, 2003 at 01:03:55PM +0200 References: <20031010110354.GQ23519@calvin> Message-ID: <20031010130955.Y9501@prim.han.de> [Alexandre Fayolle Fri, Oct 10, 2003 at 01:03:55PM +0200] > Hello > > Your probably aware of this already, but then again, maybe not. I've > just stumbled across the gnu lightning project > http://www.gnu.org/software/lightning/lightning.html > > "GNU lightning is a library that generates assembly language code at > run-time; it is very fast, making it ideal for Just-In-Time compilers, > and it abstracts over the target CPU, as it exposes to the clients a > standardized RISC instruction set inspired by the MIPS and SPARC chips." > > And I thought this may be something quite useful for pypy or for psyco. maybe, but is there any more information? the page doesn't seem to contain useful links. holger From mwh at python.net Fri Oct 10 13:21:35 2003 From: mwh at python.net (Michael Hudson) Date: Fri, 10 Oct 2003 12:21:35 +0100 Subject: [pypy-dev] Re: gnu lightning References: <20031010110354.GQ23519@calvin> Message-ID: <2msmm1uym8.fsf@starship.python.net> Alexandre Fayolle writes: > Your probably aware of this already, but then again, maybe not. I've > just stumbled across the gnu lightning project > http://www.gnu.org/software/lightning/lightning.html I know Armin knows about this and doesn't (didn't?) consider it suitable for psyco, but I must admit I don't remember why. Cheers, mwh -- ARTHUR: Why should he want to know where his towel is? FORD: Everybody should know where his towel is. ARTHUR: I think your head's come undone. -- The Hitch-Hikers Guide to the Galaxy, Episode 7 From lac at strakt.com Fri Oct 10 13:25:29 2003 From: lac at strakt.com (Laura Creighton) Date: Fri, 10 Oct 2003 13:25:29 +0200 Subject: [pypy-dev] recent lisp support / cross-language In-Reply-To: Message from holger krekel of "Fri, 10 Oct 2003 11:41:46 +0200." <20031010114146.U9501@prim.han.de> References: <20031010114146.U9501@prim.han.de> Message-ID: <200310101125.h9ABPTh9030878@ratthing-b246.strakt.com> In a message of Fri, 10 Oct 2003 11:41:46 +0200, holger krekel writes: >hello pypy, > >i think it's nice that we start to have clisp support. but there >are no unit-tests and 'translator/gencl.py' adds more code to keep >in sync with our model. So without tests i wouldn't like this code >to exist. I guess we need some boilerplate that allows starting >lisp-programs from commandline with instructions which functions to >call and parsing the results. the unittests could then do the same >what they do for pyrex: checking that the function at least computes >correctly. > >btw, I can imagine a cross-language sprint aimed at producing multiple >backends (C, Pyrex, CLISP, whatever) or multiple frontends (prolog...). >That would certainly help to get the wider language research/developer >community get interested. > >A dumb side-question: can clisp-source be annotated with types >or is it just another VHLL :-) ? > >cheers, > > holger > > >P.S.: if Seo wants to maintain the lisp area for the time beeing i am > for giving him commit privs. +1 Laura From Alexandre.Fayolle at logilab.fr Fri Oct 10 13:56:36 2003 From: Alexandre.Fayolle at logilab.fr (Alexandre Fayolle) Date: Fri, 10 Oct 2003 13:56:36 +0200 Subject: [pypy-dev] gnu lightning In-Reply-To: <20031010130955.Y9501@prim.han.de> References: <20031010110354.GQ23519@calvin> <20031010130955.Y9501@prim.han.de> Message-ID: <20031010115635.GR23519@calvin> On Fri, Oct 10, 2003 at 01:09:56PM +0200, holger krekel wrote: > [Alexandre Fayolle Fri, Oct 10, 2003 at 01:03:55PM +0200] > > Hello > > > > Your probably aware of this already, but then again, maybe not. I've > > just stumbled across the gnu lightning project > > http://www.gnu.org/software/lightning/lightning.html > > > > "GNU lightning is a library that generates assembly language code at > > run-time; it is very fast, making it ideal for Just-In-Time compilers, > > and it abstracts over the target CPU, as it exposes to the clients a > > standardized RISC instruction set inspired by the MIPS and SPARC chips." > > > > And I thought this may be something quite useful for pypy or for psyco. > > maybe, but is there any more information? the page doesn't seem to > contain useful links. There's a link to the FTP, but no tarball in there apparently because of the attack against this server last summer. You can download it from Debian's repository though, through http://packages.debian.org/unstable/devel/lightning.html (or "apt-get install lightning" if you're running testing or unstable) -- Alexandre Fayolle LOGILAB, Paris (France). http://www.logilab.com http://www.logilab.fr http://www.logilab.org D?veloppement logiciel avanc? - Intelligence Artificielle - Formations From pedronis at bluewin.ch Fri Oct 10 14:09:02 2003 From: pedronis at bluewin.ch (Samuele Pedroni) Date: Fri, 10 Oct 2003 14:09:02 +0200 Subject: [pypy-dev] Re: gnu lightning In-Reply-To: <2msmm1uym8.fsf@starship.python.net> References: <20031010110354.GQ23519@calvin> Message-ID: <5.2.1.1.0.20031010140724.00ad3008@pop.bluewin.ch> At 12:21 10.10.2003 +0100, Michael Hudson wrote: >Alexandre Fayolle writes: > > > Your probably aware of this already, but then again, maybe not. I've > > just stumbled across the gnu lightning project > > http://www.gnu.org/software/lightning/lightning.html > >I know Armin knows about this and doesn't (didn't?) consider it >suitable for psyco, but I must admit I don't remember why. Obviously Armin can/will answer himself. If I recollect correctly it was something about Psyco emitting fragments at a time instead of whole functions (?) From Alexandre.Fayolle at logilab.fr Fri Oct 10 14:11:27 2003 From: Alexandre.Fayolle at logilab.fr (Alexandre Fayolle) Date: Fri, 10 Oct 2003 14:11:27 +0200 Subject: [pypy-dev] Re: gnu lightning In-Reply-To: <2msmm1uym8.fsf@starship.python.net> References: <20031010110354.GQ23519@calvin> <2msmm1uym8.fsf@starship.python.net> Message-ID: <20031010121127.GS23519@calvin> On Fri, Oct 10, 2003 at 12:21:35PM +0100, Michael Hudson wrote: > Alexandre Fayolle writes: > > > Your probably aware of this already, but then again, maybe not. I've > > just stumbled across the gnu lightning project > > http://www.gnu.org/software/lightning/lightning.html > > I know Armin knows about this and doesn't (didn't?) consider it > suitable for psyco, but I must admit I don't remember why. Maybe licensing problems, since lightning is GPLed. -- Alexandre Fayolle LOGILAB, Paris (France). http://www.logilab.com http://www.logilab.fr http://www.logilab.org D?veloppement logiciel avanc? - Intelligence Artificielle - Formations From arigo at tunes.org Fri Oct 10 14:26:21 2003 From: arigo at tunes.org (Armin Rigo) Date: Fri, 10 Oct 2003 13:26:21 +0100 Subject: [pypy-dev] Re: gnu lightning In-Reply-To: <5.2.1.1.0.20031010140724.00ad3008@pop.bluewin.ch> References: <20031010110354.GQ23519@calvin> <5.2.1.1.0.20031010140724.00ad3008@pop.bluewin.ch> Message-ID: <20031010122621.GA9372@vicky.ecs.soton.ac.uk> Hello, On Fri, Oct 10, 2003 at 02:09:02PM +0200, Samuele Pedroni wrote: > >> just stumbled across the gnu lightning project > > > >I know Armin knows about this and doesn't (didn't?) consider it > >suitable for psyco, but I must admit I don't remember why. > > Obviously Armin can/will answer himself. If I recollect correctly it was > something about Psyco emitting fragments at a time instead of whole > functions (?) Yes, this was the main reason: GNU Lightning and most other similar libraries are oriented around the concept of function. They are unsuitable for Psyco because I really need fragments. (The licensing issue was another problem with GNU Lightning.) For PyPy the situation might be a bit different. I am sure there are use cases in which we'd like to emit code a bit faster than by calling an external C compiler, but still don't need essentially finer-grained control. The problem might still be that we'll probably need compiler-like optimizations before we can emit good machine code, and GNU Lightning doesn't provide that. Actually I think that writing machine code won't be the most difficult problem. It should be a nice exercice to design a declarative way to describe instruction encodings in Python. Then we can have an assembler, a disassembler, and more for free. I've actually experimented with something like that in an earlier attempt at Psyco. (XXX link) Armin From tismer at tismer.com Fri Oct 10 15:21:41 2003 From: tismer at tismer.com (Christian Tismer) Date: Fri, 10 Oct 2003 15:21:41 +0200 Subject: [pypy-dev] Re: recent lisp support / cross-language In-Reply-To: <2mwubdv177.fsf@starship.python.net> References: <20031010114146.U9501@prim.han.de> <2mwubdv177.fsf@starship.python.net> Message-ID: <3F86B265.1020403@tismer.com> Michael Hudson wrote: > holger krekel writes: > > >>hello pypy, >> >>i think it's nice that we start to have clisp support. but there >>are no unit-tests and 'translator/gencl.py' adds more code to keep >>in sync with our model. So without tests i wouldn't like this code >>to exist. > > > Yeah, perhaps it shouldn't have been checked in. But it's not very > much code, and it's quite neat. I think we should check in everything that sounds good, but put it into some ugly "untested" folder. :-) > Oh, and don't say "clisp": that's the name of an *implementation* of > Common Lisp. If you can't be bothered to type "Common Lisp" out, "CL" > is a better abbreviation. If possible, please, can we remove CL and turn the stuff into Scheme? I always liked that so much better... ... >>btw, I can imagine a cross-language sprint aimed at producing multiple >>backends (C, Pyrex, CLISP, whatever) or multiple frontends (prolog...). >>That would certainly help to get the wider language research/developer >>community get interested. > > > Yep! Count me in for PPC assembler :-) Yep! Count me in for X86 assembler via Forth :-) ... >>P.S.: if Seo wants to maintain the lisp area for the time beeing i am >> for giving him commit privs. > > > Me too! Sure. ciao - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From mwh at python.net Fri Oct 10 15:29:23 2003 From: mwh at python.net (Michael Hudson) Date: Fri, 10 Oct 2003 14:29:23 +0100 Subject: [pypy-dev] Re: recent lisp support / cross-language References: <20031010114146.U9501@prim.han.de> <2mwubdv177.fsf@starship.python.net><3F86B265.1020403@tismer.com> Message-ID: <2mk77dusp8.fsf@starship.python.net> Christian Tismer writes: >> Oh, and don't say "clisp": that's the name of an *implementation* of >> Common Lisp. If you can't be bothered to type "Common Lisp" out, "CL" >> is a better abbreviation. > > If possible, please, can we remove CL and turn the > stuff into Scheme? I always liked that so much better... Not if I have anything to do with it! Scheme is a toy language, at least as defined by R5RS. I have no objection to generating scheme *as well* (and it might be an idea to target a specific implementation, or at least mandate a few SFRIs). Cheers, mwh -- Sufficiently advanced political correctness is indistinguishable from irony. -- Erik Naggum From tismer at tismer.com Fri Oct 10 15:47:19 2003 From: tismer at tismer.com (Christian Tismer) Date: Fri, 10 Oct 2003 15:47:19 +0200 Subject: [pypy-dev] Re: recent lisp support / cross-language In-Reply-To: <2mk77dusp8.fsf@starship.python.net> References: <20031010114146.U9501@prim.han.de> <2mwubdv177.fsf@starship.python.net><3F86B265.1020403@tismer.com> <2mk77dusp8.fsf@starship.python.net> Message-ID: <3F86B867.5050609@tismer.com> Michael Hudson wrote: > Christian Tismer writes: ... >>If possible, please, can we remove CL and turn the >>stuff into Scheme? I always liked that so much better... > > > Not if I have anything to do with it! Scheme is a toy language, at > least as defined by R5RS. Oh! I didn't realize that. I don't write much Lisp, but used Scheme a little, many years ago, and thought it were "the better Lisp" meanwhile. Sorry for my ignorance. ciao - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From tinuviel at sparcs.kaist.ac.kr Sat Oct 11 19:42:54 2003 From: tinuviel at sparcs.kaist.ac.kr (Seo Sanghyeon) Date: Sun, 12 Oct 2003 02:42:54 +0900 Subject: [pypy-dev] Unpacking sequence Message-ID: <20031011174254.GA23295@sparcs.kaist.ac.kr> ...miserably fails with FlowObjSpace. I ran pypy/translator/translator.py and tried, ---- def boom(): a, b = 0, 1 t = Translator(boom) ---- Traceback (most recent call last): File "", line 1, in ? File "translator.py", line 52, in __init__ self.flowgraph = space.build_flow(func) File "/home/tinuviel/svn/src/pypy/objspace/flow/objspace.py", line 60, in build_flow ec.build_flow() File "/home/tinuviel/svn/src/pypy/objspace/flow/flowcontext.py", line 186, in build_flow w_result = frame.eval(self) File "/home/tinuviel/svn/src/pypy/interpreter/pyframe.py", line 49, in eval self.dispatch() File "/home/tinuviel/svn/src/pypy/interpreter/pyopcode.py", line 48, in dispatch fn(self, oparg) File "/home/tinuviel/svn/src/pypy/interpreter/pyopcode.py", line 419, in UNPACK_SEQUENCE raise OperationError(f.space.w_ValueError, f.space.wrap(str(e))) AttributeError: FlowObjSpace instance has no attribute 'w_ValueError' I don't know how to fix this at all. I know everybody is busy for funding. Should I add this one to PyPy issue tracker? Any idea from those who reading pypy-dev but not involved in funding? sanxiyn From arigo at tunes.org Sat Oct 11 20:18:53 2003 From: arigo at tunes.org (Armin Rigo) Date: Sat, 11 Oct 2003 19:18:53 +0100 Subject: [pypy-dev] Unpacking sequence In-Reply-To: <20031011174254.GA23295@sparcs.kaist.ac.kr> References: <20031011174254.GA23295@sparcs.kaist.ac.kr> Message-ID: <20031011181853.GA24724@vicky.ecs.soton.ac.uk> Hello Seo, On Sun, Oct 12, 2003 at 02:42:54AM +0900, Seo Sanghyeon wrote: > def boom(): > a, b = 0, 1 Ah. The issue is a bit complicated by the fact that we currently don't handle exception. There is an exception hidden there because we also don't remember that (0, 1) is actually a tuple of two elements, so it could be the case that there is the wrong number of values to unpack. I guess that besides remining us that we haven't exceptions yet, it shows us that we need to be a bit more subtle even in the control flow analysis and not only remember either "it's a variable" or "it's a constant". The same problem will appear for try: finally: blocks, for example, where we might want to remember that the saved stack unrolled is "either a StopUnrolling or a SApplicationException instance". In other words some kind of type analysis is still needed in the control flow. Hopefully, however, we can avoid the former problems of AnnObjSpace by only considering immutable objects. Armin From florian.proff.schulze at gmx.net Mon Oct 13 11:47:08 2003 From: florian.proff.schulze at gmx.net (Florian Schulze) Date: Mon, 13 Oct 2003 11:47:08 +0200 Subject: [pypy-dev] Ints vs Longs Message-ID: I just wondered what happens when ints overflow. This question also applies to Psyco. Is a long returned? How much of a speed penalty does occure due to this? Regards, Florian From mwh at python.net Mon Oct 13 11:58:14 2003 From: mwh at python.net (Michael Hudson) Date: Mon, 13 Oct 2003 10:58:14 +0100 Subject: [pypy-dev] Re: Ints vs Longs References: Message-ID: <2mr81htq6h.fsf@starship.python.net> Florian Schulze writes: > I just wondered what happens when ints overflow. In PyPy, um, I'm not sure :-) We don't have a long type in the StdObjSpace yet, so I guess you'll get an overflow error. > This question also applies to Psyco. Psyco correctly detects overflow and returns a long. For mulitplication, at least, this is easier in machine code than C... > Is a long returned? How much of a speed penalty does occure due to > this? I don't know. It's probably quite substantial. Cheers, mwh -- If you give someone Fortran, he has Fortran. If you give someone Lisp, he has any language he pleases. -- Guy L. Steele Jr, quoted by David Rush in comp.lang.scheme.scsh From gherman at darwin.in-berlin.de Mon Oct 13 12:35:34 2003 From: gherman at darwin.in-berlin.de (Dinu Gherman) Date: Mon, 13 Oct 2003 12:35:34 +0200 Subject: [pypy-dev] Pyrex 0.9 Message-ID: Hi, I just saw that Pyrex was updated to 0.9, and given Holgers exten- sions I wonder if they will make it into Pyrex itself, anytime soon...? Dinu PS: http://www.cosc.canterbury.ac.nz/~greg/python/Pyrex/ -- Dinu C. Gherman ...................................................................... "The whole point of brainwashing, is that those being brainwashed don't know it." (Graham Haley) From hpk at trillke.net Mon Oct 13 12:38:37 2003 From: hpk at trillke.net (holger krekel) Date: Mon, 13 Oct 2003 12:38:37 +0200 Subject: [pypy-dev] Pyrex 0.9 In-Reply-To: ; from gherman@darwin.in-berlin.de on Mon, Oct 13, 2003 at 12:35:34PM +0200 References: Message-ID: <20031013123837.Z9501@prim.han.de> [Dinu Gherman Mon, Oct 13, 2003 at 12:35:34PM +0200] > Hi, > > I just saw that Pyrex was updated to 0.9, and given Holgers exten- > sions I wonder if they will make it into Pyrex itself, anytime > soon...? I wouldn't think so and haven't tried. The changes from 0.8.2 to 0.9 should be meaningless for our purposes. It's easy to port my patch to 0.9 though. cheers, holger From florian.proff.schulze at gmx.net Mon Oct 13 12:51:26 2003 From: florian.proff.schulze at gmx.net (Florian Schulze) Date: Mon, 13 Oct 2003 12:51:26 +0200 Subject: [pypy-dev] Re: Ints vs Longs References: <2mr81htq6h.fsf@starship.python.net> Message-ID: On Mon, 13 Oct 2003 10:58:14 +0100, Michael Hudson wrote: > Florian Schulze writes: > >> I just wondered what happens when ints overflow. > > In PyPy, um, I'm not sure :-) We don't have a long type in the > StdObjSpace yet, so I guess you'll get an overflow error. > >> This question also applies to Psyco. > > Psyco correctly detects overflow and returns a long. > > For mulitplication, at least, this is easier in machine code than C... That's because there is an overflow flag ;) >> Is a long returned? How much of a speed penalty does occure due to >> this? > > I don't know. It's probably quite substantial. Hmm, now that you mentioned that Psyco uses machinecode, I think the penalty is not *that* high. It depends on how well the processor handles the branching, I guess. Florian From mwh at python.net Mon Oct 13 12:58:33 2003 From: mwh at python.net (Michael Hudson) Date: Mon, 13 Oct 2003 11:58:33 +0100 Subject: [pypy-dev] Re: Ints vs Longs References: <2mr81htq6h.fsf@starship.python.net> Message-ID: <2mn0c5tndy.fsf@starship.python.net> Florian Schulze writes: > On Mon, 13 Oct 2003 10:58:14 +0100, Michael Hudson wrote: > >> Florian Schulze writes: >> >>> I just wondered what happens when ints overflow. >> >> In PyPy, um, I'm not sure :-) We don't have a long type in the >> StdObjSpace yet, so I guess you'll get an overflow error. >> >>> This question also applies to Psyco. >> >> Psyco correctly detects overflow and returns a long. >> >> For mulitplication, at least, this is easier in machine code than C... > > That's because there is an overflow flag ;) Exactly :-) >>> Is a long returned? How much of a speed penalty does occure due to >>> this? >> >> I don't know. It's probably quite substantial. > > Hmm, now that you mentioned that Psyco uses machinecode, I think the > penalty is not *that* high. It depends on how well the processor > handles the branching, I guess. Ah, I thought you were asking about how much overhead occurred when there *was* an overflow. That could be substantial, as I don't think psyco knows very much about longs (could be wrong, though). The overhead of checking for overflow is, as you say, probably not that high. I don't know enough about the x86 architecture to say more. Cheers, mwh -- Or if you happen to be resigned to the size of your trouser snake and would rather not be reminded of it, training a shared classifier to reject penis-enlargement spam stops Barry from getting the help he so desperately needs. -- Tim Peters, c.l.python From roccomoretti at hotpop.com Mon Oct 13 15:53:21 2003 From: roccomoretti at hotpop.com (Rocco Moretti) Date: Mon, 13 Oct 2003 08:53:21 -0500 Subject: [pypy-dev] fixeol In-Reply-To: <635DFF1F.4296E41D.9ADE5C6A@netscape.net> References: <635DFF1F.4296E41D.9ADE5C6A@netscape.net> Message-ID: <3F8AAE51.6060005@hotpop.com> Sorry for brining up old news, but my Netscape e-mail wasn't letting me log in for more than about 2 seconds ... so I changed free email providers. I've had a backlog of messages to work through ... >>I'm still a bit worried that if fixeol actually changes content, as would be >>the case if it is run by someone on another platform than the one used to >>create it, then svn will record it as completely changed (ideal source for >>conflicts). The only solution I could imagine would be to make sure that >>people specify files to be text when they first add it to the repository. >>Somehow the default "files-are-binary" of SVN creates different, new kind of >>problems than the default "files-are-text" of CVS. Maybe a no-default >>you-must-specify-it-explicitely wouldn't have been such a bad idea. >> >> > >we could enforce this on a pre-commit hook and maybe other things >like e.g. that REST-files compile without errors. The latter would make >a checkin of documents slower but i think it's worth it IMO. > One option would be to sell our friends at subversion on the concept of repository level tagging. I would imagine that it is common to want *all* *.py, *.txt, *.c and *.h files to be handled as eol-native files throughout the repository. One way repository level tagging could work is that on the svn server is a config file that applies globally to all files and directories in the repository, setting specific property tags upon file/directory addition. For example: ['*.txt', '*.py']: svn:eol-style = native would cause any text or python files to have the svn:eol-style property be set to native. Though generally applied, it could apply to all tags and file designations. PyPy svn admins: Does this seem like a reasonable feature to request? Or is it already availible? -Rocco From roccomoretti at hotpop.com Mon Oct 13 15:53:51 2003 From: roccomoretti at hotpop.com (Rocco Moretti) Date: Mon, 13 Oct 2003 08:53:51 -0500 Subject: [pypy-dev] "Unwrap" Concidered Harmful In-Reply-To: <3805E442.3C34E604.9ADE5C6A@netscape.net> References: <3805E442.3C34E604.9ADE5C6A@netscape.net> Message-ID: <3F8AAE6F.5070303@hotpop.com> Armin Rigo wrote: >On Sat, Sep 13, 2003 at 02:24:51PM -0400, Rocco Moretti wrote: > > >>space.is_true() is an example of such a limited use function. In my mind, >>is_true() exists for a single purpose - to determine if a wrapped >>object is true so that the interpreter can determine if a >>JUMP_IF_TRUE type branch should be taken. Any other use would be >>considered a "hack". >> >> > >Right. However there are a few other cases in which the interpreter may be >interested in "probing" information about a wrapped object. Here are two I can >think of: > > * getting an integer. We need this at a few points, e.g. to know the exact >length of a tuple (which is needed to interpret something like "f(*args)"). > > * getting a (short) string. Again there is an example in arguments decoding, >to interpret "f(**kw)" we need to manipulate the keywords. > >It may be possible to remove all these uses of unwrap (e.g. the length of a >tuple can be obtained by iterating over it and counting), but I'd say that it >would only make matters more obscure. Instead, just as we have "is_true()" we >could do with a few other methods to get the integer value of an object, or >its string value. > I'd argue that the interpreter level length of an argument tuple should be determined by a specific function, and not by unwrapping the application level length. We may want to implement some object space where the application level length is different than the length at which the intepreter should view it. For example, an application level length may be "between 4 and 7, but most likely 5", but for purpouses of interpreter level execution we want it to be treated as 7. -Rocco From bsammy_61 at yahoo.com Mon Oct 13 16:23:26 2003 From: bsammy_61 at yahoo.com (bsnowbird@juno.com) Date: Mon, 13 Oct 2003 16:23:26 +0200 Subject: [pypy-dev] Re: Hi Message-ID: <20031013142616.A1C595A442@thoth.codespeak.net> An HTML attachment was scrubbed... URL: From tismer at tismer.com Mon Oct 13 18:22:24 2003 From: tismer at tismer.com (Christian Tismer) Date: Mon, 13 Oct 2003 18:22:24 +0200 Subject: [pypy-dev] fixeol In-Reply-To: <3F8AAE51.6060005@hotpop.com> References: <635DFF1F.4296E41D.9ADE5C6A@netscape.net> <3F8AAE51.6060005@hotpop.com> Message-ID: <3F8AD140.1080306@tismer.com> Rocco Moretti wrote: ... > For example: > > ['*.txt', '*.py']: > svn:eol-style = native > > would cause any text or python files to have the svn:eol-style property > be set to native. Though generally applied, it could apply to all tags > and file designations. > > PyPy svn admins: Does this seem like a reasonable feature to request? Or > is it already availible? For some reason, this didn't work from the beginning, sure we wanted that and will fix it. -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From hpk at trillke.net Tue Oct 14 11:13:17 2003 From: hpk at trillke.net (holger krekel) Date: Tue, 14 Oct 2003 11:13:17 +0200 Subject: [pypy-dev] early sprint planning - amsterdam Message-ID: <20031014111317.G9501@prim.han.de> hello pypy-devers, hello current pypy-sprint subscribers, just wanted to gather some more feedback for the next (fifth!) PyPy-sprint. It is currently planned to take place at "Vrije Universiteit in Amsterdam", 14th-21th Dec 2003. thanks to Etienne Posthumus. There was already some agreement in Berlin about this proposal, and Armin, Michael and me told me afterwars it's likely they will come. I guess the main goal will be to complete C code generation (probably still using Pyrex as an intermediate layer). It would also be nice to work on some fun frontends to PyPy like one with pygame that visualizes interactions between interpreter and objectspace. But we will certainly need to discuss which main goals the next sprint should have. The sprint should be a rather pure coding sprint, no funding strings attached :-) short after this mail i will unsubscribe everybody (except armin, michael and me) from the pypy-sprint list. So please SUBSCRIBE yourself ASAP if you are *interested* to come to Amsterdam: http://codespeak.net/mailman/listinfo/pypy-sprint A real announcement will supposedly go soon to pypy-dev, c.l.py and python-dev. We want to fix especially the date early so that people can book cheap flights. If you want to come but really have no way at all of financing it please contact me. We will try to help but can't promise anything. cheers, holger From jacob at strakt.com Tue Oct 14 11:59:25 2003 From: jacob at strakt.com (Jacob =?iso-8859-1?q?Hall=E9n?=) Date: Tue, 14 Oct 2003 11:59:25 +0200 Subject: [pypy-dev] Re: [pypy-sprint] early sprint planning - amsterdam In-Reply-To: <20031014111317.G9501@prim.han.de> References: <20031014111317.G9501@prim.han.de> Message-ID: <200310141000.h9EA0BeS002934@theraft.strakt.com> On Tuesday 14 October 2003 11.13, holger krekel wrote: > hello pypy-devers, hello current pypy-sprint subscribers, > > just wanted to gather some more feedback for the next (fifth!) > PyPy-sprint. It is currently planned to take place at > > "Vrije Universiteit in Amsterdam", 14th-21th Dec 2003. > > thanks to Etienne Posthumus. There was already some agreement in > Berlin about this proposal, and Armin, Michael and me told me > afterwars it's likely they will come. Laura and I plan to be there. I hope we can get Bea to come and learn how we work. > I guess the main goal will be to complete C code generation > (probably still using Pyrex as an intermediate layer). It would > also be nice to work on some fun frontends to PyPy like one > with pygame that visualizes interactions between interpreter > and objectspace. But we will certainly need to discuss which > main goals the next sprint should have. The sprint should > be a rather pure coding sprint, no funding strings attached :-) I think we need to keep the funding issue open. If the EU moves fast (we'd all be stunned, wouldn't we), we may have to handle funding issues at the sprint. I'd really prefer not to, but I'm keeping an open mind about it. Cheers Jacob From arigo at tunes.org Tue Oct 14 12:41:45 2003 From: arigo at tunes.org (Armin Rigo) Date: Tue, 14 Oct 2003 11:41:45 +0100 Subject: [pypy-dev] "Unwrap" Concidered Harmful In-Reply-To: <3F8AAE6F.5070303@hotpop.com> References: <3805E442.3C34E604.9ADE5C6A@netscape.net> <3F8AAE6F.5070303@hotpop.com> Message-ID: <20031014104145.GA15118@vicky.ecs.soton.ac.uk> Hello Rocco, On Mon, Oct 13, 2003 at 08:53:51AM -0500, Rocco Moretti wrote: > be determined by a specific function, and not by unwrapping the > application level length. We may want to implement some object space > where the application level length is different than the length at which > the intepreter should view it. For example, an application level length > may be "between 4 and 7, but most likely 5", but for purpouses of > interpreter level execution we want it to be treated as 7. I don't really see how this could be useful, but even if it were I assume that it can still be handled by the object space in a call like space.unwrap_int(space.len(w_list)). I would rather try to keep the number of special operations as low as possible. BTW there is a nice parallel to make between unwrap_int() and is_true(), which both extract some kind of interpreter-level information out of an application-level object. We could probably find a more coherent naming scheme. A bientot, Armin From jeremy at alum.mit.edu Wed Oct 15 02:21:45 2003 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Tue, 14 Oct 2003 20:21:45 -0400 Subject: [pypy-dev] Re: recent lisp support / cross-language In-Reply-To: <3F86B867.5050609@tismer.com> References: <20031010114146.U9501@prim.han.de> <2mwubdv177.fsf@starship.python.net><3F86B265.1020403@tismer.com> <2mk77dusp8.fsf@starship.python.net> <3F86B867.5050609@tismer.com> Message-ID: <1066177305.5141.95.camel@localhost.localdomain> On Fri, 2003-10-10 at 09:47, Christian Tismer wrote: > > Not if I have anything to do with it! Scheme is a toy language, at > > least as defined by R5RS. > > Oh! I didn't realize that. I don't write much Lisp, but > used Scheme a little, many years ago, and thought > it were "the better Lisp" meanwhile. > Sorry for my ignorance. He's being oblique. R5RS doesn't define any of the libraries or extra features you might consider essential for day-to-day programming. There are certainly mature, useful Scheme implementations around -- interpreters and compilers. And the SRFIs define a bunch of standard interfaces for libraries. Another slant is that Perl is to Common Lisp as Python is to Scheme. CL has got every feature that anyone thought would be useful. Scheme has got a minimal set of features that everyone agreed were necessary. Jeremy From lac at strakt.com Wed Oct 15 10:16:34 2003 From: lac at strakt.com (Laura Creighton) Date: Wed, 15 Oct 2003 10:16:34 +0200 Subject: [pypy-dev] Maybe Ian Bicking wants our test framework. Message-ID: <200310150816.h9F8GYTA006465@ratthing-b246.strakt.com> http://blog.colorstudy.com/ianb/weblog/2003/10/10.html#P11 From hpk at trillke.net Wed Oct 15 10:34:23 2003 From: hpk at trillke.net (holger krekel) Date: Wed, 15 Oct 2003 10:34:23 +0200 Subject: [pypy-dev] Maybe Ian Bicking wants our test framework. In-Reply-To: <200310150816.h9F8GYTA006465@ratthing-b246.strakt.com>; from lac@strakt.com on Wed, Oct 15, 2003 at 10:16:34AM +0200 References: <200310150816.h9F8GYTA006465@ratthing-b246.strakt.com> Message-ID: <20031015103423.S9501@prim.han.de> [Laura Creighton Wed, Oct 15, 2003 at 10:16:34AM +0200] > http://blog.colorstudy.com/ianb/weblog/2003/10/10.html#P11 interesting thread. But our test-"framework" is really just a hack hardly serving our own purposes. OTOH, our use cases for driving tests are somewhat different from Zope3, twisted and all the other projects who wrote their own. For one, we have interpreter-level and app-level tests, further parametrized by Object Spaces. Also we are using tools (such as dot, clisp, ...) which may sometimes not be there and there is no clean way to skip those tests. And i'd like to have a "TestConsole" that allows me to repetively rerun failing tests until the implementation is fixed, IOW support test-driven development directly. Plus i'd like pluggable report modules (producing html for automated tests on the web-page etc.) and, and and. unittest.py and most of the above mentioned replacements don't seem to be designed for all these uses. I think it makes sense to tackle a rewrite of unit-test making all those use-cases easy and pythonic maybe not only for PyPy. I am not sure how much of the current std/unittest.py API/internals we want to inherit, though. cheers, holger From gherman at darwin.in-berlin.de Wed Oct 15 15:30:54 2003 From: gherman at darwin.in-berlin.de (Dinu Gherman) Date: Wed, 15 Oct 2003 15:30:54 +0200 Subject: [pypy-dev] Maybe Ian Bicking wants our test framework. In-Reply-To: <200310150816.h9F8GYTA006465@ratthing-b246.strakt.com> Message-ID: Laura Creighton: > http://blog.colorstudy.com/ianb/weblog/2003/10/10.html#P11 As the Brits are saying: "interesting". If he complains about the Python unittest package needing too much "boilerplate" code he should, perhaps, have some fun using CppUnit! :-) Dinu -- Dinu C. Gherman - http://python.net/~gherman ...................................................................... "The whole point of brainwashing, is that those being brainwashed don't know it." (Graham Haley) From jriehl at cs.uchicago.edu Thu Oct 16 03:15:40 2003 From: jriehl at cs.uchicago.edu (Jonathan David Riehl) Date: Wed, 15 Oct 2003 20:15:40 -0500 (CDT) Subject: [pypy-dev] LL3 abstract. Message-ID: Hi all, I've attatched my first draft of a presentation abstract that I'll be submitting to LL3. This proposal is due this Friday (the 17th), so if you are interested in what I'm going to say in Boston, you might want to check this out right now and get any feedback to me ASAP. I'd especially like feedback from people that have already given presentations on PyPy or are doing development on the translator (the whole sprint was a blur to me - I'm going to need help remembering what I was witness to!) In case you don't remember what LL3 is, Dinu forwarded a call for presentations late last month, and you can look at the website at http://ll3.ai.mit.edu/. If anyone else in pypy-dev is attending, I should be there whether my presentation is accepted or not, and we should discuss hanging out. Thanks! -Jon -------------- next part -------------- TALK ABSTRACT: "Yet Another Python Implementation? - Some recent results from the PyPy project." Jonathan Riehl In January of 2003, Armin Rigo, Christian Tismer and Holger Krekel began a project with the intention of providing a new implementation of Python written in the Python language. Named PyPy, the project's goal is to provide a self-hosting Python that allows both ease of maintenance and flexibility for language implementors. In the last year, the PyPy team has developed a proof of concept implementation of Python in Python, and has recently taken steps toward bootstrapping PyPy into running outside of the current C implementation of Python, commonly referred to as CPython. One of the more attractive goals in the bootstrapping process is to have PyPy generate a native executable that fully replaces the CPython interpreter. This step would allow Pypy development to proceed fully independent of CPython development. Furthermore, a Python implementation hosted by CPython suffers performance penalties by nesting a VM implementation inside another VM. Therefore, an obvious step in this strategy is to build a translator from Python source into either a statically built language (such as C) or directly into a machine language of the target architecture (such as a JIT compiler). One of the more interesting aspects of the system design is that both translation methods may be readily added to PyPy. My presentation will have two primary goals: First, I will introduce the PyPy project and its architecture, providing a high level overview of the system design. Second, I will present some interesting results obtained during a recent development workshop (commonly referred to as a sprint in Python parlance). Specifically, I will discuss how PyPy is now capable of a limited Python to C translation, and contrast this with previous work done by several other groups in the Python community. Finally, I will discuss some future directions being investigated by the development team, including a possibly novel approach to type inference based on logic programming. (Estimated time: 10 mins. on background, 10-15 mins. on translation and possibly type inference, 5-10 mins. on questions.) From bob at redivi.com Thu Oct 16 03:21:39 2003 From: bob at redivi.com (Bob Ippolito) Date: Wed, 15 Oct 2003 21:21:39 -0400 Subject: [pypy-dev] LL3 abstract. In-Reply-To: Message-ID: <17F8A57E-FF77-11D7-9E91-000A95686CD8@redivi.com> On Wednesday, Oct 15, 2003, at 21:15 America/New_York, Jonathan David Riehl wrote: > I've attatched my first draft of a presentation abstract that I'll > be submitting to LL3. This proposal is due this Friday (the 17th), so > if > you are interested in what I'm going to say in Boston, you might want > to > check this out right now and get any feedback to me ASAP. I'd > especially > like feedback from people that have already given presentations on > PyPy or > are doing development on the translator (the whole sprint was a blur > to me > - I'm going to need help remembering what I was witness to!) > In case you don't remember what LL3 is, Dinu forwarded a call for > presentations late last month, and you can look at the website at > http://ll3.ai.mit.edu/. If anyone else in pypy-dev is attending, I > should > be there whether my presentation is accepted or not, and we should > discuss > hanging out. Sounds fun, count me in.. I'm in NYC, so it shouldn't be hard to catch a train up there. -bob From hpk at trillke.net Thu Oct 16 09:28:59 2003 From: hpk at trillke.net (holger krekel) Date: Thu, 16 Oct 2003 09:28:59 +0200 Subject: [pypy-dev] LL3 abstract. In-Reply-To: ; from jriehl@cs.uchicago.edu on Wed, Oct 15, 2003 at 08:15:40PM -0500 References: Message-ID: <20031016092859.A9501@prim.han.de> Hey Jonathan, [Jonathan David Riehl Wed, Oct 15, 2003 at 08:15:40PM -0500] > I've attatched my first draft of a presentation abstract that I'll > be submitting to LL3. This proposal is due this Friday (the 17th), so if > you are interested in what I'm going to say in Boston, you might want to > check this out right now and get any feedback to me ASAP. proposal .. oh no^h^h^h^h^ great, i was just developing a lost feeling without any proposal in sight. For your abstract (sounds fine) you might want to strip some stuff out of our objectives and implementation chapter (feel free at ripping!) we just submitted to the EU: http://codespeak.net/pypy/index.cgi?doc/funding/B1.0_objectives.html http://codespeak.net/pypy/index.cgi?doc/funding/B6.0_detailed_implementation.html > I'd especially like feedback from people that have already given presentations > on PyPy or are doing development on the translator (the whole sprint was a blur to me > - I'm going to need help remembering what I was witness to!) You get used to this feeling by the time beeing :-) > In case you don't remember what LL3 is, Dinu forwarded a call for > presentations late last month, and you can look at the website at > http://ll3.ai.mit.edu/. If anyone else in pypy-dev is attending, I should > be there whether my presentation is accepted or not, and we should discuss > hanging out. sounds good but as it only a one-day meeting i guess not many europeans will be interested to come. but who knows. and let me know if i can help, holger From tismer at tismer.com Thu Oct 16 14:40:16 2003 From: tismer at tismer.com (Christian Tismer) Date: Thu, 16 Oct 2003 14:40:16 +0200 Subject: [pypy-dev] PyPy on FPGA? In-Reply-To: <087d01c393ca$68913710$c36ff181@idi.ntnu.no> References: <20030919113514.GA2007@hex.no> <3F6B4986.4030101@tismer.com> <20030923141644.GA3869@hex.no> <3F737C8A.6080908@tismer.com> <20031009193549.GD19585@hex.no> <3F8D91AD.4040605@tismer.com> <087d01c393ca$68913710$c36ff181@idi.ntnu.no> Message-ID: <3F8E91B0.6060303@tismer.com> Amund Tveit wrote: ... > btw, regarding the PyPy project. Are there anyone of you working on a > FPGA-port of the PyPy core? It would certainly be interesting to port the > core to a VHDLish language and make it run on low-power FPGA-chips, in > particular for potential mobile device adaption of Python. That comment > could potentially be something to put in your EU funding proposal? :-) We were of course considering PyPy for embedded systems, different hardware platforms, and we certainly will implement PyPy for different environments. See section B.6.7.11 of the proposal: "Embed in Specialized Hardware", page 69 of http://www.codespeak.net/svn/pypy/trunk/doc/funding/proposal/part_b.pdf I don't know if this will involve FPGA in the first place. But this is anyway just the beginning. By the end of this 2 year project, PyPy is either dead, or everybody will rush at us and have us port the software to whatsoever. cheers - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From tinuviel at sparcs.kaist.ac.kr Thu Oct 16 15:04:22 2003 From: tinuviel at sparcs.kaist.ac.kr (Seo Sanghyeon) Date: Thu, 16 Oct 2003 22:04:22 +0900 Subject: [pypy-dev] IRC log: is list.append a RPython op? Message-ID: <20031016130422.GA6474@sparcs.kaist.ac.kr> I asked Armin whether list.append is a RPython op, and had a brief chat on the topic. Armin expressed his opinion, and agreed to post IRC log to this list. Any comment? -------------- next part -------------- #pypy log Am I correct in that translators are supposed to translate StdObjSpace? yes but not multimethod.py if that's what you are afraid of :-) I started mapping list.append to vector-push-extend, etc, and realized that it should be done by translating listobject.list_append__List_ANY. (is my understanding correct?) no Eh, you mean, "if it does the same thing, using host-functionality is fine"? (eh, of course, it must be fine...?) the RPython operations should be translated directly as you did the list_append__List_ANY functions operate at another level Then I ask, is list.append RPython op? ok, this one hasn't received a definitive answer yet If it is RPython op, then _ins1, _list_resize, etc in listobject just doesn't make sense at all. i think we should have "list + list" but not list.append in fact, for Lisp i think we should not use vectors with that extra "current size" field i mean, arrays but not fill poitners Adjustable array, but not vector-push? in other words only "simple arrays" in the Lisp sense, which are more or less just C arrays Ah, understood. yes, adjustable would be nice for list.extend or list += the idea is that we want to avoid the complex logic that would be needed in C to support efficient appends So... make-array for malloc(), adjust-array for realloc(), etc? exactly So list.append is apparently not a RPython op. so far i'd say it is not, though it may be implemented stupidly as list += [item] From tvassila at siac.com Thu Oct 16 15:29:55 2003 From: tvassila at siac.com (Thanos Vassilakis) Date: Thu, 16 Oct 2003 09:29:55 -0400 Subject: [pypy-dev] PyPy on FPGA? Message-ID: I have a Python on FPGA project going that is partly financed by a FPGA chip manufacture. It is early days and would be very interested in any other such projects. thanos ----------------------------------------- This message and its attachments may contain privileged and confidential information. If you are not the intended recipient(s), you are prohibited from printing, forwarding, saving or copying this email. If you have received this e-mail in error, please immediately notify the sender and delete this e-mail and its attachments from your computer. From tinuviel at sparcs.kaist.ac.kr Thu Oct 16 15:48:15 2003 From: tinuviel at sparcs.kaist.ac.kr (Seo Sanghyeon) Date: Thu, 16 Oct 2003 22:48:15 +0900 Subject: [pypy-dev] Re: IRC log: is list.append a RPython op? In-Reply-To: <20031016130422.GA6474@sparcs.kaist.ac.kr> References: <20031016130422.GA6474@sparcs.kaist.ac.kr> Message-ID: <20031016134815.GA12112@sparcs.kaist.ac.kr> Christian joined, and a bit more discussion on what RPython should be. Using Flow Graph Transformation is suggested. pypy-dev summary would be nice, but I think #pypy summary would be as nice. :-) -------------- next part -------------- #pypy log I agree that list.append is better not an RPython op. are you still sure that list+list is in RPython? I think we required that list lengths are known when lists are created. stackless: yes i think it is realloc() + memcpy() list += list is realloc() + a single memcpy() both look all right but this would mean that we virtually support append. we also said that [blah for blah in stuff] cannot have an if clause since we need to know list length. yes but we don't *really* support append, only *virtually* because when you write list += [item] you know you are doing a funny thing list stuff: ok, so we allow it but don't encourage it. Unless we have a situation where the compiler can figure out that it can optimize a series of such operations. Is that possible? i guess we'll need something along these lines for [x for y in z], yes * sanxiyn tries dis.dis. * sanxiyn confirmed [ x for y in z ] uses LOAD_ATTR "append" internally. well, allowing list+[item] gives a realloc, which we wanted to avoid in [x for x in y]. Consequently, we also could allow [x for x in y if z] and have an inefficient realloc case there, too. no i think all syntax/operations should be either implemented efficiently or disallowed (or at least give a warning, say) btw., looking at the current list implementation, it isn't inefficient at all to realloc like they do. ah? nice does it still make sense for CPython to have this over-allocation logic for lists then? I think the effort is linear. oops, I was referring to the overallocation as *the* trick to make it efficient. ah i thought you meant the PyMem_Realloc implementation they double list length (or something less) when it is needed, this give linear behavior. but here we're talking about the RPython lists for which we may or may not want to implement complex overallocation no, sure, I forgot. Wrong level of thinking, again. Wrong level, I think. IMO RPython shouldn't do that. [x for y in z] in RPython produces a control flow graph with calls to the append method in a simple loop i believe we can detect this case by looking at the flow graph simpler is better. more simpler is more better. Yes, and knows the length in advance. yes, we can do it with flow graph transformation only changing it into a graph that says "l = [None]*len(z)" and then loops this sounds like a criterion for defining what RPython is. Can we do it with FGT, then it's fine. yes So, list-int multiplication translates to allocation? yes. and maybe only list-of-size-1 with int multiplication And it translates to malloc + memset? so we basically avoid any operation and just use the syntax to express the missign "array[n]" declaration. :-) sanxiyn: yes yes all very nice. got a little clearer picture. i think flow graph transforms can be used extensively, replacing bits with simpler bits using primitive operations like "malloc_and_set" explicitely RPython replaces typing info by using things the right way. Programming by gestures From Nicolas.Chauvat at logilab.fr Fri Oct 17 17:26:23 2003 From: Nicolas.Chauvat at logilab.fr (Nicolas Chauvat) Date: Fri, 17 Oct 2003 17:26:23 +0200 Subject: [pypy-dev] Maybe Ian Bicking wants our test framework. In-Reply-To: <20031015103423.S9501@prim.han.de> References: <200310150816.h9F8GYTA006465@ratthing-b246.strakt.com> <20031015103423.S9501@prim.han.de> Message-ID: <20031017152622.GA30188@logilab.fr> > For one, we have interpreter-level and app-level tests, further > parametrized by Object Spaces. Also we are using tools (such as > dot, clisp, ...) which may sometimes not be there and there is > no clean way to skip those tests. And i'd like to have a > "TestConsole" that allows me to repetively rerun failing tests > until the implementation is fixed, IOW support test-driven development > directly. Plus i'd like pluggable report modules (producing > html for automated tests on the web-page etc.) and, and and. > > unittest.py and most of the above mentioned replacements don't > seem to be designed for all these uses. I think it makes sense > to tackle a rewrite of unit-test making all those use-cases > easy and pythonic maybe not only for PyPy. I am not sure how > much of the current std/unittest.py API/internals we want to > inherit, though. We happen to have written our own encapsulation to unittest in order to get our automated testing framework up and running at Logilab. I'd be interested in such a rewrite and/or addition to std/unittest which is already pretty good as it is. -- Nicolas Chauvat http://www.logilab.com - "Mais o? est donc Ornicar ?" - LOGILAB, Paris (France) From lac at strakt.com Fri Oct 17 22:36:02 2003 From: lac at strakt.com (Laura Creighton) Date: Fri, 17 Oct 2003 22:36:02 +0200 Subject: [pypy-dev] Improved Unit Test Framework Message-ID: <200310172036.h9HKa2qo015631@ratthing-b246.strakt.com> I couldn't remember what Peter had told me about his testing framework, so I asked him. Now he is curious as to what we want to do differently. Laura ------- Forwarded Message Return-Path: peter at engcorp.com To: Laura Creighton Subject: Re: Unit testing References: <200310171413.h9HEDlLH014913 at ratthing-b246.strakt.com> Laura Creighton wrote: > > DO I remember it correctly that you have extensively hacked up the > unittest module to build a nicer framework? and if so, is what you > did freely available? PyPy wants a nicer framework ... Actually, we carefully avoided touching "unittest.py" at all, as we were new to this game. Instead we've added a wrapper ('ktest') around it which provides a few features we find useful (but only under Win/DOS). - - type "unit xxxx" to run unit tests for module xxxx, expecting to find them at ./unit/xxxx_unit.py - - "unit -a" runs all tests in all ./unit subdirectories, recursively - - "unit -r" randomizes the order of the individual TestCases (but not yet the individual test methods) to help catch test-order dependencies - - all output from tests go to log files as xxxx_unit.log for later inspection - - no graphical abilities whatsoever; no pretty green/red bar (I regret the latter part) - - automatic inclusion of .pth files found in ./unit subfolders, and automatic inclusion of parent folder in the sys.path (all tests run with the ./unit folder as current directory) - - automatically use Python 2.0 or 2.2 based on flag file in ./unit folder as we have not yet migrated entirely from 2.0 - - probably a few other bits I seriously doubt that what we have would be of much use to anyone as they would likely have to adopt a completely different approach to testing than they currently use. The area where we have actually excelled, instead of where we've stuck to the entirely non-fancy console/DOS interface with limited features, is in our auxiliary test frameworks. One of them, PyJTF, has just been put up on Sourceforge (http://pyjtf.sourceforge.net/). It allows test-driven development of Javascript code for the IE browser (again, limited to Win32 :-( ) using Python and COM and Twisted to great effect. The other, as yet unreleased, incorporates a Python-based HC12 CPU simulator to allow test-driven development of embedded system code in C. Tests in Python, code in C, fairly simple interface and amazing performance (one set of 98 tests takes only 4 seconds to run). As this one matures, I hope to release it as well. If the kunit features are remotely interesting, I can forward the code and its own tests. Feel free to post any part or all of this to the PyPy mailing list, or elsewhere, as you see fit. :-) - -Peter Hansen ------- End of Forwarded Message From hpk at trillke.net Sat Oct 18 01:55:36 2003 From: hpk at trillke.net (holger krekel) Date: Sat, 18 Oct 2003 01:55:36 +0200 Subject: [pypy-dev] Improved Unit Test Framework In-Reply-To: <200310172036.h9HKa2qo015631@ratthing-b246.strakt.com>; from lac@strakt.com on Fri, Oct 17, 2003 at 10:36:02PM +0200 References: <200310172036.h9HKa2qo015631@ratthing-b246.strakt.com> Message-ID: <20031018015536.M20543@prim.han.de> [Laura Creighton Fri, Oct 17, 2003 at 10:36:02PM +0200] > I couldn't remember what Peter had told me about his testing framework, > so I asked him. Now he is curious as to what we want to do differently. Apparently, almost every larger python project writes its own unittest-workaround/extension with often similar features. It's sure interesting to condense a list of useful features from all the different approaches. However, PyPy requires e.g. to sometimes take over the execution of a test method because it needs to be run at app-level with a specific objspace. Also some methods can/should be run on multiple objectspaces. Moreover, i guess that at some point we want to rerun tests with our generated (C-version) PyPy, etc.pp So until someone ports over their unittest-framework and shows how this can all nicely work out i remain unconvinced that we can just reuse another unittest-wrapper :-) After all, i don't think that writing a more flexible (unit)test module suiting our purposes is that hard. We already have quite a bit of it and can rip from unittest.py. cheers, holger > (Peters description skipped but appreciated) From lac at strakt.com Sat Oct 18 09:28:04 2003 From: lac at strakt.com (Laura Creighton) Date: Sat, 18 Oct 2003 09:28:04 +0200 Subject: [pypy-dev] Improved Unit Test Framework In-Reply-To: Message from holger krekel of "Sat, 18 Oct 2003 01:55:36 +0200." <20031018015536.M20543@prim.han.de> References: <200310172036.h9HKa2qo015631@ratthing-b246.strakt.com> <20031018015536.M20543@prim.han.de> Message-ID: <200310180728.h9I7S4I1016949@ratthing-b246.strakt.com> In a message of Sat, 18 Oct 2003 01:55:36 +0200, holger krekel writes: >Apparently, almost every larger python project writes its own >unittest-workaround/extension with often similar features. It's sure >interesting to condense a list of useful features from all the different >approaches. >After all, i don't think that writing a more flexible (unit)test module >suiting our purposes is that hard. We already have quite a bit of it >and can rip from unittest.py. > >cheers, > > holger It might be nice to get a hold of other people and see what they would like to have in a unittest module. That way, even if we don't write it that way ourselves, we can make sure that it will be easy for them to add their heart's desire later. Laura From anthony at interlink.com.au Sat Oct 18 09:52:40 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Sat, 18 Oct 2003 17:52:40 +1000 Subject: [pypy-dev] Improved Unit Test Framework In-Reply-To: <200310180728.h9I7S4I1016949@ratthing-b246.strakt.com> Message-ID: <200310180752.h9I7qfXM011037@localhost.localdomain> >>> Laura Creighton wrote > It might be nice to get a hold of other people and see what they > would like to have in a unittest module. That way, even if we > don't write it that way ourselves, we can make sure that it will > be easy for them to add their heart's desire later. It's probably worth breaking this into two categories: - improvements to unittest itself - a test runner (e.g. Zope3's test.py, twisted's trial) The latter is what's _really_ needed - it'd be nice if something could be done for Python 2.4. Anthony -- Anthony Baxter It's never too late to have a happy childhood. From gherman at darwin.in-berlin.de Sat Oct 18 10:14:35 2003 From: gherman at darwin.in-berlin.de (Dinu Gherman) Date: Sat, 18 Oct 2003 10:14:35 +0200 Subject: [pypy-dev] Improved Unit Test Framework In-Reply-To: <20031018015536.M20543@prim.han.de> Message-ID: <1BF8E0D2-0143-11D8-917E-00039345C610@darwin.in-berlin.de> holger krekel: > Apparently, almost every larger python project writes its own > unittest-workaround/extension with often similar features. It's sure > interesting to condense a list of useful features from all the > different > approaches. There seems to be an increasing number complaints about the unittest package, which I have difficulties to follow. PyUnit is a framework, large parts of which you can adapt to your own needs. When I started using it I found that, like with any good framework, it takes a while to grasp all degrees of freedom you really have. When people write their own wrappers and extensions it doesn't mean the framework is bad! It means you can do it! As Holger basically says, I also suggest, whoever wants PyUnit to do something it cannot do without extensions or maybe even at all, come up with a descrip- tion of desired features on the PyUnit mailing list! Write a few XP- like user stories and let Steve Purcell know about it. Holger, I'm sure we can setup a Wiki page for special PyPy testing demands? I wrote a Cocoa GUI frontent to PyUnit a while ago which I should probably run over the PyPy testsuite and see what happens... ;-) If you're on a Mac see the link below to do the same. Dinu -- Dinu C. Gherman - http://python.net/~gherman/Pycotine.html ...................................................................... "Political satire became obsolete when Henry Kissinger was awarded the Nobel Peace Prize." (Tom Lehrer) From hpk at trillke.net Sun Oct 19 13:06:54 2003 From: hpk at trillke.net (holger krekel) Date: Sun, 19 Oct 2003 13:06:54 +0200 Subject: [pypy-dev] Improved Unit Test Framework In-Reply-To: <200310180752.h9I7qfXM011037@localhost.localdomain>; from anthony@interlink.com.au on Sat, Oct 18, 2003 at 05:52:40PM +1000 References: <200310180728.h9I7S4I1016949@ratthing-b246.strakt.com> <200310180752.h9I7qfXM011037@localhost.localdomain> Message-ID: <20031019130654.Q20543@prim.han.de> [Anthony Baxter Sat, Oct 18, 2003 at 05:52:40PM +1000] > > >>> Laura Creighton wrote > > It might be nice to get a hold of other people and see what they > > would like to have in a unittest module. That way, even if we > > don't write it that way ourselves, we can make sure that it will > > be easy for them to add their heart's desire later. > > It's probably worth breaking this into two categories: > > - improvements to unittest itself > - a test runner (e.g. Zope3's test.py, twisted's trial) > > The latter is what's _really_ needed - it'd be nice if something could be > done for Python 2.4. Yes, the distinction makes sense. ASFAIK most testframeworks provide a much better testrunner but don't care that much about the actual unittest machinery (because it's hard to change, anyway :-). As a side note, i think that the distinction between unit-tests and functional tests is not always practical. We certainly do a lot of more-or-less functional testing in PyPy. Actually, a new testing module shouldn't need to worry about unit<->function tests too much. cheers, holger From anthony at interlink.com.au Sun Oct 19 15:31:41 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Sun, 19 Oct 2003 23:31:41 +1000 Subject: [pypy-dev] Improved Unit Test Framework In-Reply-To: <20031019130654.Q20543@prim.han.de> Message-ID: <200310191331.h9JDVhgF013915@localhost.localdomain> >>> holger krekel wrote > Yes, the distinction makes sense. ASFAIK most testframeworks provide a > much better testrunner but don't care that much about the actual > unittest machinery (because it's hard to change, anyway :-). As a side > note, i think that the distinction between unit-tests and functional > tests is not always practical. We certainly do a lot of more-or-less > functional testing in PyPy. Actually, a new testing module shouldn't > need to worry about unit<->function tests too much. The Zope3 testrunner distinguishes between them - the unit tests live in subdirectories called 'test', while the functional tests live in 'ftest'. The main problem with separating them in this way is that people forget to run the functional tests. On the other hand, functional tests probably need to derive from a different TestCase class, that defines a bunch of extra 'setup' (not setUp) methods. For instance, in quetzalcoatl, our ETL tool, functional tests have a bunch of machinery for establishing database connections and the like. You don't want/need these in the unit tests. Anthony -- Anthony Baxter It's never too late to have a happy childhood. From peter at engcorp.com Sun Oct 19 15:14:43 2003 From: peter at engcorp.com (Peter Hansen) Date: Sun, 19 Oct 2003 09:14:43 -0400 Subject: [pypy-dev] Improved Unit Test Framework References: <200310180728.h9I7S4I1016949@ratthing-b246.strakt.com> <200310180752.h9I7qfXM011037@localhost.localdomain> <20031019130654.Q20543@prim.han.de> Message-ID: <3F928E43.2E0469DE@engcorp.com> holger krekel wrote: > > [Anthony Baxter Sat, Oct 18, 2003 at 05:52:40PM +1000] > > > > >>> Laura Creighton wrote > > > It might be nice to get a hold of other people and see what they > > > would like to have in a unittest module. That way, even if we > > > don't write it that way ourselves, we can make sure that it will > > > be easy for them to add their heart's desire later. > > > > It's probably worth breaking this into two categories: > > > > - improvements to unittest itself Just what would those be? I still haven't heard a single useful improvement. There must be some (at least, that's obviously what a lot of people feel), but what are they? > > - a test runner (e.g. Zope3's test.py, twisted's trial) This was the area where we focused as well. We needed the recursive invocation feature, the change in structure (tests in ./unit and ./accept subfolders rather than in same folder as source), the automatic handling of .pth files on a per-project basis, etc. unittest.py was fine for us as-is, and we're even using some bits exposed in its own TestRunner. In the XP or TDD mailing list, "Phlip" reported on a system whereby merely by saving his files, his test suite would automatically be re-run and the results displayed in the test runner's window, and he could double-click on the errors and it would jump him directly back in the editor to the failing line. > > The latter is what's _really_ needed - it'd be nice if something could be > > done for Python 2.4. > > Yes, the distinction makes sense. ASFAIK most testframeworks provide a > much better testrunner but don't care that much about the actual > unittest machinery (because it's hard to change, anyway :-). As a side > note, i think that the distinction between unit-tests and functional > tests is not always practical. We certainly do a lot of more-or-less > functional testing in PyPy. Actually, a new testing module shouldn't > need to worry about unit<->function tests too much. Agreed on that. Although we separate the two for various reasons into unit and acceptance test folders, in some cases its arguable which is which and in any case the testing framework doesn't distinguish at all. For us, the only thing that really distinguishes an acceptance test, other than the inherent level of abstraction of the test, is the degree of mocking and "borging" that we have to do. (At Kaval, a "borg" object is like a mock object in that it simulates the real thing, but in the case of the borg object it is a fake shell around the real code of something the program-under-test must talk to, such as a server. Rather than run the server code on a real (say, Linux) server during the test, we will make a "borg" version that thinks it's on a real server but is just running in a separate process on the Win32 test machine.) -Peter From anthony at interlink.com.au Sun Oct 19 15:42:40 2003 From: anthony at interlink.com.au (Anthony Baxter) Date: Sun, 19 Oct 2003 23:42:40 +1000 Subject: [pypy-dev] Improved Unit Test Framework In-Reply-To: <3F928E43.2E0469DE@engcorp.com> Message-ID: <200310191342.h9JDgfdx014141@localhost.localdomain> >>> Peter Hansen wrote > > > [me] > > > - improvements to unittest itself > > Just what would those be? I still haven't heard a single useful > improvement. There must be some (at least, that's obviously what > a lot of people feel), but what are they? The unittest improvements I have in my local tree are all pretty minor - I've made the output of an assertRaises failure more meaningful, that sort of thing. At some point I'll go through them and figure out which ones should be in the stdlib, and submit as an SF patch. While I'm not _entirely_ happy with the standard unittest module (it seems to me like I spend a lot of time writing boilerplate code - this is something python's supposed to make better...), it's hard for me to point to a solution that is obviously "better". Anthony -- Anthony Baxter It's never too late to have a happy childhood. From hpk at trillke.net Sun Oct 19 19:23:06 2003 From: hpk at trillke.net (holger krekel) Date: Sun, 19 Oct 2003 19:23:06 +0200 Subject: [pypy-dev] Improved Unit Test Framework In-Reply-To: <3F928E43.2E0469DE@engcorp.com>; from peter@engcorp.com on Sun, Oct 19, 2003 at 09:14:43AM -0400 References: <200310180728.h9I7S4I1016949@ratthing-b246.strakt.com> <200310180752.h9I7qfXM011037@localhost.localdomain> <20031019130654.Q20543@prim.han.de> <3F928E43.2E0469DE@engcorp.com> Message-ID: <20031019192306.T20543@prim.han.de> [Peter Hansen Sun, Oct 19, 2003 at 09:14:43AM -0400] > holger krekel wrote: > > > > [Anthony Baxter Sat, Oct 18, 2003 at 05:52:40PM +1000] > > > > > > >>> Laura Creighton wrote > > > > It might be nice to get a hold of other people and see what they > > > > would like to have in a unittest module. That way, even if we > > > > don't write it that way ourselves, we can make sure that it will > > > > be easy for them to add their heart's desire later. > > > > > > It's probably worth breaking this into two categories: > > > > > > - improvements to unittest itself > > Just what would those be? I still haven't heard a single useful > improvement. There must be some (at least, that's obviously what > a lot of people feel), but what are they? ok, here a few possible improvements for starters (i have a lot more in mind :-) - beeing able to provide a function that executes a test-method (today you have to replicate all the boilerplate in TestCase.__call__ etc.pp.) - beeing able to raise e.g. TestSkip Error to indicate that a test should not be run with the current setup. (You can't alway tell that offhand in setUp). - allow providing extra options to the cmdline-interface - plan for/do other user-interfaces (web-driven, pygame, whatever :-) - provide means (a TestConsole) to easily get a list of failed tests (interactively), keep the testing-process alive and repetitively redo the failed tests. Or save off the info to some file and reload it on rerunning the test. cheers, holger From peter at engcorp.com Sun Oct 19 20:09:42 2003 From: peter at engcorp.com (Peter Hansen) Date: Sun, 19 Oct 2003 14:09:42 -0400 Subject: [pypy-dev] Improved Unit Test Framework Message-ID: <3F92D366.E5B1BA0A@engcorp.com> (Sorry, Holger... I first sent this to you directly by accident, rather than just to the mailing list.) holger krekel wrote: > > [Peter Hansen Sun, Oct 19, 2003 at 09:14:43AM -0400] > > holger krekel wrote: > > > > - improvements to unittest itself > > > > Just what would those be? I still haven't heard a single useful > > improvement. There must be some (at least, that's obviously what > > a lot of people feel), but what are they? > > ok, here a few possible improvements for starters (i have a lot more in > mind :-) > > - beeing able to provide a function that executes a test-method > (today you have to replicate all the boilerplate in TestCase.__call__ > etc.pp.) Why do you need to do that? Or maybe I should say, can't you already do that? If you have a TestCase, I think you just call .run or something on it, same as when you specify a single class and (optionally) a test method name on the command line. I frequently run a single test using our framework wrapper, which just stole code from the existing runner in unittest. > - beeing able to raise e.g. TestSkip Error to indicate that a test > should not be run with the current setup. (You can't alway tell > that offhand in setUp). What does this do that raise TestSkipError (after defining it) would do not do? Continue running other tests? Seems like a property of the test runner, not the unittest side of things (assuming a new test runner, as we are by making the distinction into two categories, I think). > - allow providing extra options to the cmdline-interface > > - plan for/do other user-interfaces (web-driven, pygame, whatever :-) > > - provide means (a TestConsole) to easily get a list of failed tests > (interactively), keep the testing-process alive and repetitively redo > the failed tests. Or save off the info to some file and reload it > on rerunning the test. These are all properties of the test runner, aren't they? The existing unittest has its own version of this stuff, but I got the impression you were already accepting that a different runner would be needed. -Peter -------------- next part -------------- An embedded message was scrubbed... From: Peter Hansen Subject: Re: [pypy-dev] Improved Unit Test Framework Date: Sun, 19 Oct 2003 14:07:42 -0400 Size: 2656 URL: From hpk at trillke.net Sun Oct 19 23:43:26 2003 From: hpk at trillke.net (holger krekel) Date: Sun, 19 Oct 2003 23:43:26 +0200 Subject: [pypy-dev] Improved Unit Test Framework In-Reply-To: <3F92D366.E5B1BA0A@engcorp.com>; from peter@engcorp.com on Sun, Oct 19, 2003 at 02:09:42PM -0400 References: <3F92D366.E5B1BA0A@engcorp.com> Message-ID: <20031019234326.U20543@prim.han.de> Hi Peter, [Peter Hansen Sun, Oct 19, 2003 at 02:07:42PM -0400] > holger krekel wrote: > > > > [Peter Hansen Sun, Oct 19, 2003 at 09:14:43AM -0400] > > > holger krekel wrote: > > > > > - improvements to unittest itself > > > > > > Just what would those be? I still haven't heard a single useful > > > improvement. There must be some (at least, that's obviously what > > > a lot of people feel), but what are they? > > > > ok, here a few possible improvements for starters (i have a lot more in > > mind :-) > > > > - beeing able to provide a function that executes a test-method > > (today you have to replicate all the boilerplate in TestCase.__call__ > > etc.pp.) > > Why do you need to do that? Or maybe I should say, can't you already > do that? If you have a TestCase, I think you just call .run or something > on it, same as when you specify a single class and (optionally) a test > method name on the command line. I frequently run a single test using > our framework wrapper, which just stole code from the existing runner > in unittest. I don't mean selecting but executing a test. I am really just interested in having a TestCase hook like this: def execute_test(self, callable): """ execute a testmethod """ callable() # default implementation Maybe it's special to PyPy but we could use this for our application-level tests. There are two basic levels at which a test can possibly run: - application level: such a test is executed indirectly using the PyPy interpreter (which itself requires an Object Space to operate on objects). - interpreter level: those are executed at the same level as how the interpreter sees the world: as wrapped objects which only an Object Spaces knows how to access and manipulate. Application level tests are faster and easier to write. For driving them the above hook would probably help. > > - beeing able to raise e.g. TestSkip Error to indicate that a test > > should not be run with the current setup. (You can't alway tell > > that offhand in setUp). > > What does this do that raise TestSkipError (after defining it) > would do not do? Continue running other tests? Seems like a > property of the test runner, not the unittest side of things > (assuming a new test runner, as we are by making the distinction > into two categories, I think). OK, we could pack this to the TestRunner which resides also in unittest.py, anyway ... > > - allow providing extra options to the cmdline-interface > > > > - plan for/do other user-interfaces (web-driven, pygame, whatever :-) > > > > - provide means (a TestConsole) to easily get a list of failed tests > > (interactively), keep the testing-process alive and repetitively redo > > the failed tests. Or save off the info to some file and reload it > > on rerunning the test. > > These are all properties of the test runner, aren't they? The > existing unittest has its own version of this stuff, but I got > the impression you were already accepting that a different runner > would be needed. In some strict sense maybe only TestCase does not belong to the the "frontend", which then includes TestRunner, TestProgram, main, TestLoader, TestResult etc.pp. However, my impression is that the mechanics in unittest.py are not really flexible enough to drive our development in a sufficiently nice way. And when doing test-driven development why not have *very* nice tools (instead of OK-tools) to support it? cheers, holger From tinuviel at sparcs.kaist.ac.kr Fri Oct 24 05:05:54 2003 From: tinuviel at sparcs.kaist.ac.kr (Seo Sanghyeon) Date: Fri, 24 Oct 2003 12:05:54 +0900 Subject: [pypy-dev] Does RPython have slice? Message-ID: <20031024030554.GA15108@sparcs.kaist.ac.kr> I was playing with translator frontend, writing random snippets. And I discovered FlowObjSpace currently lacks newslice interface. Does RPython have slice? I checked StdObjSpace implementation. slicefind.sh is located at SVN repo user/sanxiyn/script. Slice is currently used only in stringojbect, namely split, strip, count, startswith, endswith, and repr. (Two matches in listobject doesn't need to be RPython.) It wouldn't be hard to rewrite them without slice. I remember a discussion on IRC whose log I posted for a while ago: in RPython, op should be able to be implemented efficiently, or be disallowed. And flowgraph transformation was suggested. Perhaps slice to sans-slice transformation should be written, and translators may or may not use the transformation? Seo Sanghyeon From arigo at tunes.org Fri Oct 24 13:54:43 2003 From: arigo at tunes.org (Armin Rigo) Date: Fri, 24 Oct 2003 12:54:43 +0100 Subject: [pypy-dev] Does RPython have slice? In-Reply-To: <20031024030554.GA15108@sparcs.kaist.ac.kr> References: <20031024030554.GA15108@sparcs.kaist.ac.kr> Message-ID: <20031024115443.GA9430@vicky.ecs.soton.ac.uk> Hello Seo, On Fri, Oct 24, 2003 at 12:05:54PM +0900, Seo Sanghyeon wrote: > Does RPython have slice? I am not too familiar with std/stringobject.py, I can't say how this code should be translated or if it shouldn't have used slices in the first place. In general I think that slicing from a list can be regarded as the operation of malloc()ating a new, smaller list and memcpy()ing data around, which is fine as long as you are aware of this. Depending on the code generators, it may be useful as you suggest to have general graph transformations that change unsupported constructs into reasonably efficient lower-level supported ones. I can imagine that (using type inference) we could change newslice/getitem pairs into, say, malloc followed by an explicit loop of getitem/setitem. BTW (in case you're not already aware of this) the Lisp generator will have to be fixed at some point, because it cannot simply translate "add" into "+". This only works if type inference says the arguments are integers. A bientot, Armin. From arigo at tunes.org Sun Oct 26 18:05:40 2003 From: arigo at tunes.org (Armin Rigo) Date: Sun, 26 Oct 2003 17:05:40 +0000 Subject: [pypy-dev] Refactored flow model Message-ID: <20031026170540.GA16738@vicky.ecs.soton.ac.uk> Hello everybody, A bunch of commits from yesterday: Holger and I rewrote the classes that we use to represent control flow graphs. We had agreed during the whole Berlin sprint that the many XxxBranch classes weren't very practical to use, and we discussed an alternative at the end of the sprint. That's what we just implemented. There is also a much nicer dot-generator now. The graphs are becoming quite comprehensible now, much shorter, with nice colors, and all :-) Try running the tests with '-v' (either individual tests or the test_all.py script); if you installed dot, it will leave Postscript files in /tmp/usession-*/*.ps, with both the non-simplified and the simplified graph showing on the same page. A bientot, Armin. From arigo at tunes.org Sun Oct 26 18:10:21 2003 From: arigo at tunes.org (Armin Rigo) Date: Sun, 26 Oct 2003 17:10:21 +0000 Subject: [pypy-dev] PyPy: sprint and news Message-ID: <20031026171021.GB16738@vicky.ecs.soton.ac.uk> PyPy Sprint announcement & news from the project ================================================ We are coming close to a first experimental release of PyPy, a more flexible Python implementation written in Python. The sprint to make this happen will take place in Amsterdam, a city know to be reachable by cheap flights :-) This is 1) the announcement for the sprint; 2) news about the current state of PyPy; 3) some words about a proposal we recently submitted to the European Union. Amsterdam Sprint Details ------------------------ The Sprint will take place from the 14th of December to the 21st of December at the "Vrije Universiteit in Amsterdam", 14th-21th Dec 2003. thanks to Etienne Posthumus, who helps us to organize the event. The main goal will be a complete C translation of PyPy, probably still using a hacked Pyrex-variant as an intermediate layer and using CPython's runtime. We also plan to work on some fun frontends to PyPy like one based on pygame or a web browser to visualize interactions between interpreter and objectspace. If you want to participate on the sprint, please subscribe here http://codespeak.net/mailman/listinfo/pypy-sprint and list yourself on this wiki page http://codespeak.net/moin/pypy/moin.cgi/AmsterdamSprint where you will also find more information as the sprint date approaches. If you are just interested but don't know if you come then only subscribe to the mailing list. State of the PyPy project -------------------------- PyPy works pretty well but still on top of CPython. The double interpretation penalty makes it - as expected - incredibly slow :-) In the Berlin sprint we have thus started to work on the "translation" part, i.e. how this code should be translated into C. We can now translate simple functions to C-like code including some type annotations. For convenience, we are reusing a modified subset of Pyrex to generate the low-level C code. Thanks to Seo (who joined the project from south-korea) we also have a lisp-backend to fuel the endless c.l.py threads about python versus lisp :-) The goal of the next sprint is to complete this work so that we can translate the complete PyPy source into a huge Pyrex module, and then a big CPython extension module. True, the result is not independent from CPython, as it runs reusing its runtime environment. But it's probably an interesting enough state to make a public release from. The translation is done by generating a control flow of functions by means of abstract interpretation. IOW, we run the PyPy interpreter with a custom object space ("flowobjspace") which generates a control flow graph (including the elementary operations) which is then used to generate low-level code for backends. We also have preliminary type inference on the graphs, which can be used by the Pyrex generator to emit some C type declarations. Writing transformations and analysis of these graphs and displaying them with GraphViz's 'dot' is great fun! We certainly have a greater need than ever for graphical interactive tools to see, understand and debug all these graph manipulations and run tests of them. Currently it is a bit difficult to write a test that checks that a transformed graph "looks right"! What we expect from the Amsterdam sprint is thus: - a big not-too-slow "cpypy.so" extension module for CPython, where at least integer arithmetic is done efficiently - interactive tools to display and debug and test PyPy, visualizing control flow, call-graphs and state models. - improving and rewriting our testing tools to give us more control over the testing process, and to allow more interactive testing sessions. Other interesting News ---------------------- Before mid October, we also had a quite different Sprint. It was an approximately 10-day effort towards submitting a proposal to the EU. If it is accepted we will have resources to fund some developers working full- or parttime on the project. However, our "sprint driven development" will continue to play the central role for development of PyPy. There are especially two technical sections of the proposal which you might find interesting to read: "Scientific and technological objectives": http://codespeak.net/pypy/index.cgi?doc/funding/B1.0 "Detailed implementation plan" http://codespeak.net/pypy/index.cgi?doc/funding/B6.0 Maybe you want to read the whole proposal for other reasons, too, like making a EU project of your own or competing with us. Actually, with our sprints there is usually a lot of room for cooperation :-) Anyway, here is the PDF-url: http://codespeak.net/svn/pypy/trunk/doc/funding/proposal/part_b.pdf Everybody who thinks that he/she could help on the project is invited to join! Btw, the latest discussions about our sprint goals usually take place on the pypy-dev list: http://codespeak.net/mailman/listinfo/pypy-dev have fun, Armin & Holger From hpk at trillke.net Mon Oct 27 17:15:27 2003 From: hpk at trillke.net (holger krekel) Date: Mon, 27 Oct 2003 17:15:27 +0100 Subject: [pypy-dev] subversion repo will be upgraded **now** Message-ID: <20031027171527.F31080@prim.han.de> hello, we will be upgrading the subversion server libraries and need to dump the old database and restore it to the new format. This will take some time so expect some downtime regarding svn-accesses. The rest of apache will probably continue to run as usual, though. We'll reply to this posting if everything is stable again. We are pretty sure already, that you will need to run svn clients with at least version 0.30 (we'll also post some results for windows and linux clients which we will test the new server with). cheers, Holger & Jum From hpk at trillke.net Mon Oct 27 20:15:37 2003 From: hpk at trillke.net (holger krekel) Date: Mon, 27 Oct 2003 20:15:37 +0100 Subject: [pypy-dev] subversion repo will be upgraded **now** In-Reply-To: <20031027171527.F31080@prim.han.de>; from hpk@trillke.net on Mon, Oct 27, 2003 at 05:15:27PM +0100 References: <20031027171527.F31080@prim.han.de> Message-ID: <20031027201537.H31080@prim.han.de> [holger krekel Mon, Oct 27, 2003 at 05:15:27PM +0100] > hello, > > we will be upgrading the subversion server libraries and need to dump > the old database and restore it to the new format. This will take some > time so expect some downtime regarding svn-accesses. The rest of apache will > probably continue to run as usual, though. We'll reply to this posting > if everything is stable again. > > We are pretty sure already, that you will need to run svn clients with > at least version 0.30 (we'll also post some results for windows and > linux clients which we will test the new server with). OK, the update is complete, we are now running svn 0.32.1. Note that it is *highly recommended* to update your svn-client. See our usual page: http://codespeak.net/pypy/index.cgi?doc/devel/howtosvn.html Until subversion 1.0 is reached (probably end of this year) compatibility on the network layer is only guaranteed across two minor releases. For example, you should still be able to use a 0.31.x client with a 0.32.1 server. Although older clients may work in a lot of cases you may get strange errors in certain situations. It's also interesting to note that the new 'svn blame' command (similar to 'cvs annotate') does not always work on all PyPy files. We suspect this is so because when Armin and Holger merged the builtinrefactor branch to the trunk they actually swapped the 'trunk' and the 'branch' path to keep a linear history. However, we will communicate with the svn developers to resolve the issue. have fun, Holger & Jum From stephan.diehl at gmx.net Tue Oct 28 09:53:14 2003 From: stephan.diehl at gmx.net (Stephan Diehl) Date: Tue, 28 Oct 2003 09:53:14 +0100 Subject: [pypy-dev] Interesting project Message-ID: <200310280953.14726.stephan.diehl@gmx.net> Hi, I've just stumbled over this site: http://pliant.cx/ At a first glance, it looks like an operating system (on top of Linux) written in a python alike language (indentation is significant, so it MUST be similar to Python :-) Don't know, if this can be usefull, or not. Cheers Stephan From arigo at tunes.org Tue Oct 28 12:29:44 2003 From: arigo at tunes.org (Armin Rigo) Date: Tue, 28 Oct 2003 11:29:44 +0000 Subject: [pypy-dev] Interesting project In-Reply-To: <200310280953.14726.stephan.diehl@gmx.net> References: <200310280953.14726.stephan.diehl@gmx.net> Message-ID: <20031028112944.GA21770@vicky.ecs.soton.ac.uk> Hello Stephan, On Tue, Oct 28, 2003 at 09:53:14AM +0100, Stephan Diehl wrote: > At a first glance, it looks like an operating system (on top of Linux) > written in a python alike language (indentation is significant, so it MUST > be similar to Python :-) I know about it, it is a cool project because the language is to some extent self-defined -- you can access the parser and compiler from a module and change anything. However there are aspects I don't like. For one thing the language is defined first by a C core, on top of which a first layer adds more syntax, on top of which yet another layer adds more (sounds cool but it isn't very useful for the language user who still cannot modify these layers easily from his program). Another example: a library can access the syntax tree containing calls to its exported functions, which is nice for example to replace "(x**y)%z" with a "pow(x,y,z)" automatically. But it is a bit of syntactic magic, because if you write (the Pliant equivalent to): a = x ** y b = a % z then it won't notice it. Also note that there are actually a number of languages in which indentation is significant (e.g. O'Caml) but as in Pliant it is actually an "abbreviation" to mean something else with braces -- and you can use braces if you prefer. Python is the only language I know in which there are really no braces (or equivalent). A bientot, Armin. From tismer at tismer.com Tue Oct 28 12:40:20 2003 From: tismer at tismer.com (Christian Tismer) Date: Tue, 28 Oct 2003 12:40:20 +0100 Subject: [pypy-dev] LLVM - did we know this? Message-ID: <3F9E55A4.20504@tismer.com> Hi Friends, Richard Emslie just pushed my nose into something interesting. I don't remember, did we talk about this before? """ Anyways, I was wondering if anyone in pypy has investigated LLVM (http://llvm.cs.uiuc.edu/). Might be totally off the ball, but looks interesting nonetheless. Also look as if it has funding behind it which you guys could jump on. """ ciao - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From arigo at tunes.org Tue Oct 28 14:31:16 2003 From: arigo at tunes.org (Armin Rigo) Date: Tue, 28 Oct 2003 13:31:16 +0000 Subject: [pypy-dev] LLVM - did we know this? In-Reply-To: <3F9E55A4.20504@tismer.com> References: <3F9E55A4.20504@tismer.com> Message-ID: <20031028133116.GA27431@vicky.ecs.soton.ac.uk> Hello Christian, On Tue, Oct 28, 2003 at 12:40:20PM +0100, Christian Tismer wrote: > Richard Emslie just pushed my nose into something > interesting. I don't remember, did we talk about > this before? Not that I know of. This looks very interesting, indeed. Their low-level instruction set is at precisely the same level as our control flow graphs. It looks like the ideal target. I don't know how good their compilers already are, but we should definitely try to emit LLVM code in addition to C. And of course LLVM would be really better than C for runtime code generation. Writing PowerPC code generators for LLVM might be a good way to support Psyco-style optimizations for PyPy (and maybe even getting paid by the University of Illinois or its sponsors to do so :-) Armin From pedronis at bluewin.ch Tue Oct 28 15:49:41 2003 From: pedronis at bluewin.ch (Samuele Pedroni) Date: Tue, 28 Oct 2003 15:49:41 +0100 Subject: [pypy-dev] LLVM - did we know this? In-Reply-To: <3F9E55A4.20504@tismer.com> Message-ID: <5.2.1.1.0.20031028154426.0289bac8@pop.bluewin.ch> At 12:40 28.10.2003 +0100, Christian Tismer wrote: >Hi Friends, > >Richard Emslie just pushed my nose into something >interesting. I don't remember, did we talk about >this before? > >""" >Anyways, I was wondering if anyone in pypy has investigated LLVM >(http://llvm.cs.uiuc.edu/). Might be totally off the ball, but looks >interesting nonetheless. Also look as if it has funding behind it which >you guys could jump on. >""" it seems not to provide direct GC support? From arigo at tunes.org Tue Oct 28 16:05:56 2003 From: arigo at tunes.org (Armin Rigo) Date: Tue, 28 Oct 2003 15:05:56 +0000 Subject: [pypy-dev] LLVM - did we know this? In-Reply-To: <5.2.1.1.0.20031028154426.0289bac8@pop.bluewin.ch> References: <3F9E55A4.20504@tismer.com> <5.2.1.1.0.20031028154426.0289bac8@pop.bluewin.ch> Message-ID: <20031028150556.GA21765@vicky.ecs.soton.ac.uk> Hello Samuele, On Tue, Oct 28, 2003 at 03:49:41PM +0100, Samuele Pedroni wrote: > it seems not to provide direct GC support? LLVM is not like a JVM or the .NET thing, it's really an assembly language with some (optional) typing information. You can write non-type-safe programs in LLVM. You can write your own memory management in LLVM too. They don't impose any memory management policy. Armin From pedronis at bluewin.ch Tue Oct 28 16:27:39 2003 From: pedronis at bluewin.ch (Samuele Pedroni) Date: Tue, 28 Oct 2003 16:27:39 +0100 Subject: [pypy-dev] LLVM - did we know this? In-Reply-To: <20031028150556.GA21765@vicky.ecs.soton.ac.uk> References: <5.2.1.1.0.20031028154426.0289bac8@pop.bluewin.ch> <3F9E55A4.20504@tismer.com> <5.2.1.1.0.20031028154426.0289bac8@pop.bluewin.ch> Message-ID: <5.2.1.1.0.20031028162410.0287d308@pop.bluewin.ch> At 15:05 28.10.2003 +0000, Armin Rigo wrote: >Hello Samuele, > >On Tue, Oct 28, 2003 at 03:49:41PM +0100, Samuele Pedroni wrote: > > it seems not to provide direct GC support? > >LLVM is not like a JVM or the .NET thing, it's really an assembly language >with some (optional) typing information. You can write non-type-safe programs >in LLVM. You can write your own memory management in LLVM too. They don't >impose any memory management policy. I got that, but can you access, manipulate or walk the stack for example, in an native arch. indipendent way? From arigo at tunes.org Tue Oct 28 16:32:33 2003 From: arigo at tunes.org (Armin Rigo) Date: Tue, 28 Oct 2003 15:32:33 +0000 Subject: [pypy-dev] LLVM - did we know this? In-Reply-To: <5.2.1.1.0.20031028162410.0287d308@pop.bluewin.ch> References: <5.2.1.1.0.20031028154426.0289bac8@pop.bluewin.ch> <3F9E55A4.20504@tismer.com> <5.2.1.1.0.20031028154426.0289bac8@pop.bluewin.ch> <5.2.1.1.0.20031028162410.0287d308@pop.bluewin.ch> Message-ID: <20031028153233.GB21765@vicky.ecs.soton.ac.uk> Hello Samuele, On Tue, Oct 28, 2003 at 04:27:39PM +0100, Samuele Pedroni wrote: > >> it seems not to provide direct GC support? > > > >LLVM is not like a JVM or the .NET thing, it's really an assembly language > > I got that, but can you access, manipulate or walk the stack for example, > in an native arch. indipendent way? Ok, sorry. I don't know that. Armin From arigo at tunes.org Tue Oct 28 16:33:15 2003 From: arigo at tunes.org (Armin Rigo) Date: Tue, 28 Oct 2003 15:33:15 +0000 Subject: [pypy-dev] LLVM - did we know this? In-Reply-To: <3F9E55A4.20504@tismer.com> References: <3F9E55A4.20504@tismer.com> Message-ID: <20031028153314.GC21765@vicky.ecs.soton.ac.uk> Hello PyPy, from short IRC discussions on the IRC channels and evaluating the link that Christian provided (thanks, thanks, thanks!) it might make sense to send off the following mail to the LLVM developers. Comments? Holger, Armin --- proposed mail ---- Hello Chris, Luckily your project (LLVM) came to our attention today. Especially your message here http://mail.cs.uiuc.edu/pipermail/llvmdev/2003-October/000501.html along with the goals and documentation about the project made a great impression on us. This seems like a perfect match for our PyPy project which we started earlier this year. It is a reimplementation of Python within Python (as a good example of a higher level language), aimed at a more flexible, configurable and performant runtime system. To produce low level machine code the architecture currently has four stages: 1) compiling source code into byte code 2) performing symbolic/abstract interpretation and generating an intermediate flow-graph representation of the program. 3) (optionally) optimizing the graph 4) generating low-level C code When we learned about LLVM, the link was immediately obvious: the flow-graph representation is exactly like LLVM code. So please excuse us, if this mail is a bit over-enthusiatic :-) LLVM is also a perfect match because it can either agressively compile statically, or quickly generate code at runtime, which will be essential to port to PyPy the just-in-time compilation techniques prototyped in Psyco (http://psyco.sourceforge.net). Maybe a bit more background what we did or are: - we are an international group of individuals collaborating on our free time mostly. We are very involved with open source communities and especially the Python communities. - During the course of four one-week meetings (which we call development "sprints") we have done a rather complete interpreter and can generate some working C or Lisp code already. - we very recently submitted a funding proposal to the European Union: http://codespeak.net/svn/pypy/trunk/doc/funding/proposal/part_b.pdf and you may find these two chapters particularly interesting: http://codespeak.net/pypy/index.cgi?doc/funding/B1.0_objectives http://codespeak.net/pypy/index.cgi?doc/funding/B6.0_detailed However, we'd be interested in collaborating with your project on (hopefully) several levels. We will certainly consider contributing a PowerPC backend. On another level, some of the PyPy core developers are actually also involved with the 'codespeak' site which aims to connect interesting open source projects and provide new collaborative services. http://codespeak.net Also we are extensively using subversion which is a very interesting (and stable) alternative to cvs. For more info about subversion, see http://svnbook.red-bean.com/book.html#svn-ch-1-sect-1 but we are also interested in innovative and collaborative ways of software development, like "sprints". So if you need any help with setting up some public infrastructure the codespeak guys will certainly welcome you. Feel free to forward this mail to the LLVM mailing list, btw. We are just interested in getting some first contact and enter a productive discussion and - who knows - some interesting collaboration! A bientot, Armin Rigo, Holger Krekel From bob at redivi.com Tue Oct 28 16:53:10 2003 From: bob at redivi.com (Bob Ippolito) Date: Tue, 28 Oct 2003 10:53:10 -0500 Subject: [pypy-dev] LLVM - did we know this? In-Reply-To: <5.2.1.1.0.20031028162410.0287d308@pop.bluewin.ch> References: <5.2.1.1.0.20031028154426.0289bac8@pop.bluewin.ch> <3F9E55A4.20504@tismer.com> <5.2.1.1.0.20031028154426.0289bac8@pop.bluewin.ch> <5.2.1.1.0.20031028162410.0287d308@pop.bluewin.ch> Message-ID: On Oct 28, 2003, at 10:27 AM, Samuele Pedroni wrote: > At 15:05 28.10.2003 +0000, Armin Rigo wrote: >> Hello Samuele, >> >> On Tue, Oct 28, 2003 at 03:49:41PM +0100, Samuele Pedroni wrote: >> > it seems not to provide direct GC support? >> >> LLVM is not like a JVM or the .NET thing, it's really an assembly >> language >> with some (optional) typing information. You can write non-type-safe >> programs >> in LLVM. You can write your own memory management in LLVM too. They >> don't >> impose any memory management policy. > > I got that, but can you access, manipulate or walk the stack for > example, in an native arch. indipendent way? Well, it's all C++ code (at a glance, I didn't see any assembly anywhere, at least not in separate files) and it doesn't use libffi.. so I would probably guess that support for this might be limited. -bob From arigo at tunes.org Tue Oct 28 18:18:55 2003 From: arigo at tunes.org (Armin Rigo) Date: Tue, 28 Oct 2003 17:18:55 +0000 Subject: [pypy-dev] LLVM - did we know this? In-Reply-To: <20031028153314.GC21765@vicky.ecs.soton.ac.uk> References: <3F9E55A4.20504@tismer.com> <20031028153314.GC21765@vicky.ecs.soton.ac.uk> Message-ID: <20031028171855.GD21765@vicky.ecs.soton.ac.uk> Hello, The previous e-mail prompted some discussion which I logged in a new directory, "doc/irclog", which also contains a log about the annotations as used by the translator's type inference. A bientot, Armin. From hpk at trillke.net Wed Oct 29 14:01:21 2003 From: hpk at trillke.net (holger krekel) Date: Wed, 29 Oct 2003 14:01:21 +0100 Subject: [pypy-dev] LLVM - did we know this? In-Reply-To: <3F9E55A4.20504@tismer.com>; from tismer@tismer.com on Tue, Oct 28, 2003 at 12:40:20PM +0100 References: <3F9E55A4.20504@tismer.com> Message-ID: <20031029140121.Z31080@prim.han.de> Hello PyPy, Second try from Armin and me after IRC discussions ... --- proposed mail ---- Hello Chris, we have been investigating your project and the good documentation and are very impressed. If we understood your goals correctly this seems like a good match for our ongoing and active PyPy project, a reimplementation of the Python language in Python. We'll definitely try using llvm as our low-level backend. But actually we contact you now to ask you if you'd be interested in some bidirectional collaboration. Maybe a bit more background what we did or are: - we are an open international group of individuals collaborating on our free time mostly. We are very involved with research, open source communities and especially the Python communities. - During the course of four one-week meetings (which we call development "sprints") we have done a rather complete interpreter and can translate parts of it to C or Lisp code already (using a control-flow representation which is actually very similar to LLVM code, hence our enthusiasm!) - we very recently submitted a funding proposal to the European Union: http://codespeak.net/svn/pypy/trunk/doc/funding/proposal/part_b.pdf and you may find these two chapters particularly interesting: http://codespeak.net/pypy/index.cgi?doc/funding/B1.0_objectives http://codespeak.net/pypy/index.cgi?doc/funding/B6.0_detailed On the technical level we are interested to know about and maybe collaborate with efforts to support very-high-level language features in LLVM (e.g. walking the stack, for garbage collection), fine-grained runtime code generation (generating code only one basic block at a time), and possibly also contribute a PowerPC back-end, and Python bindings for LLVM. On another level, some of the PyPy core developers are actually also involved with the 'codespeak' site which aims at connecting interesting open source projects and provide new collaborative development services. The PyPy project is extensively using subversion which is a very interesting (and stable) alternative to cvs. So if you need any help with setting up some publically accessible infrastructure the codespeak guys will certainly welcome you. Feel free to forward this mail to the LLVM mailing list, btw. We are just interested in getting some first contact and enter a productive discussion and - who knows - some interesting collaboration! a bientot, Armin Rigo, Holger Krekel From hpk at trillke.net Fri Oct 31 20:04:47 2003 From: hpk at trillke.net (holger krekel) Date: Fri, 31 Oct 2003 20:04:47 +0100 Subject: [pypy-dev] Re: [LLVMdev] Re: LLVM and PyPy In-Reply-To: ; from sabre@nondot.org on Fri, Oct 31, 2003 at 10:58:45AM -0600 References: <20031031111827.GB15289@vicky.ecs.soton.ac.uk> Message-ID: <20031031200447.T31080@prim.han.de> Hi Chris, [Chris Lattner Fri, Oct 31, 2003 at 10:58:45AM -0600] > On Fri, 31 Oct 2003, Armin Rigo & Holger Krekel wrote: > > > Hello Chris, > > > > We have been investigating your project and the good documentation > > and are very impressed. If we understood your goals correctly > > this seems like a good match for our ongoing and active PyPy project, > > a reimplementation of the Python language in Python. > > Cool. We are all big fans of Python here. :) That's good because we might want to recode some LLVM functionalities in Python :-) > > We'll definitely try using LLVM as our low-level backend. > > But actually we contact you now to ask you if you'd be interested > > in some bidirectional collaboration. > > I've read up a bit on PyPy, and it looks like LLVM could be a nice way to > get the JIT type interface that you would like. Also, making use of the > LLVM optimizer can make your staticly generated code nice and fast. :) Yes, but we also would want to dynamically emit and execute LLVM code. But a static translation is indeed our first goal :-) > We've tossed around the idea of creating C bindings > for LLVM, which would make interfacing from other languages easier than > going directly to the C++ API, but we just haven't had a chance to yet. > Maybe you guys would be interested in helping with that project? Thinking some more about it, we would probably try to translate our PyPy implementation into LLVM-code and also generate some glue-LLVM-code which allows us to programmatically drive LLVM from Python. Is LLVM able to "drive" itself? I mean can the LLVM-low-level object code generate LLVM-low-level object code and then execute it? This would fit nicely with PyPy because we are running ourselves (in 'abstract interpretation' mode) in order to generate a low-level representation of ourselves. This low-level representation is already close to LLVM's low-level view. So if the LLVM-code gets executed (beeing a python interpreter) it should be able to just-in-time-compile new LLVM code and execute it. With our architecture, for such a JIT we could reuse a good part of the code we already have for generating our low-level representation. It's a rather self-referential thing (also see our logo: http://codespeak.net/pypy/ :-). > > On another level, some of the PyPy core developers are actually also > > involved with the 'codespeak' site which aims at connecting interesting > > open source projects and provide new collaborative development services. > > The PyPy project is extensively using subversion which is a very interesting > > (and stable) alternative to cvs. So if you need any help with setting > > up some publically accessible infrastructure the codespeak guys will > > certainly welcome you. > > At this point, we're working like crazy to get important features > implemented in LLVM. We certainly acknowledge that CVS has severe > deficiencies, but in the near future we'll probably stay with it. > Perhaps after SVN 1.0 comes out... :) then we may want to mirror your cvs repo to subversion :-) The reason is that we want to provide consistent versions of all the libraries/modules/projects we use. And subversion makes this rather easy (if the other project is svn-controled, too), e.g. you can say 'i want to follow the HEAD version of LLVM in this branch' or 'i want to use this stable version of LLVM for my own stable-release'. Then you can just issue 'svn up' and you will have the desired versions on your working-copy. However, i can understand that you don't want to consider subversion right now and will stop advertising now :-) cheers, holger From sabre at nondot.org Fri Oct 31 21:04:01 2003 From: sabre at nondot.org (Chris Lattner) Date: Fri, 31 Oct 2003 14:04:01 -0600 (CST) Subject: [pypy-dev] Re: [LLVMdev] Re: LLVM and PyPy In-Reply-To: <20031031200447.T31080@prim.han.de> Message-ID: > > Cool. We are all big fans of Python here. :) > > That's good because we might want to recode some LLVM functionalities > in Python :-) As long as it makes sense. Needless duplication of effort is never a good idea... > > I've read up a bit on PyPy, and it looks like LLVM could be a nice way to > > get the JIT type interface that you would like. Also, making use of the > > LLVM optimizer can make your staticly generated code nice and fast. :) > > Yes, but we also would want to dynamically emit and execute LLVM code. > But a static translation is indeed our first goal :-) Of course. We can do both. In fact, we can even emit C code, which will be useful initially if you're work on PowerPC machines. > Thinking some more about it, we would probably try to translate our PyPy > implementation into LLVM-code and also generate some glue-LLVM-code > which allows us to programmatically drive LLVM from Python. Is LLVM > able to "drive" itself? I mean can the LLVM-low-level object code > generate LLVM-low-level object code and then execute it? Yes, this should certainly be possible. Kindof like what the Jalapeno/Jikes JVM does with Java. The point about the C bindings is that they will allow a nice interface between the parts written in python, and the parts written in C++. It doesn't make sense for you to rewrite all of LLVM in python, especially since the interface to build the LLVM is pretty clean. > This would fit nicely with PyPy because we are running ourselves (in > 'abstract interpretation' mode) in order to generate a low-level > representation of ourselves. This low-level representation is already > close to LLVM's low-level view. So if the LLVM-code gets executed > (beeing a python interpreter) it should be able to just-in-time-compile > new LLVM code and execute it. With our architecture, for such a JIT we Makes a lot of sense. > > At this point, we're working like crazy to get important features > > implemented in LLVM. We certainly acknowledge that CVS has severe > > deficiencies, but in the near future we'll probably stay with it. > > Perhaps after SVN 1.0 comes out... :) > > then we may want to mirror your cvs repo to subversion :-) That is obviously no problem. :) > The reason is that we want to provide consistent versions of all > the libraries/modules/projects we use. And subversion makes Makes sense. If it is publically accessible and stable, perhaps we can add information about it on the LLVM pages for others who would prefer to work with SVN... -Chris -- http://llvm.cs.uiuc.edu/ http://www.nondot.org/~sabre/Projects/ From arigo at tunes.org Fri Oct 31 21:48:40 2003 From: arigo at tunes.org (Armin Rigo) Date: Fri, 31 Oct 2003 20:48:40 +0000 Subject: [pypy-dev] Re: LLVM and PyPy In-Reply-To: References: <20031031111827.GB15289@vicky.ecs.soton.ac.uk> Message-ID: <20031031204840.GA7164@vicky.ecs.soton.ac.uk> Hello Chris, On Fri, Oct 31, 2003 at 10:58:45AM -0600, Chris Lattner wrote: > These are definitely features that we plan to add, but just haven't gotten > to yet. In particular, Alkis is working on a Java front-end, which will > require similar features. In the beginning, we will probably just use a > conservative collector, eventually adding support for precise GC. Great! > We already have the capability of doing function-at-a-time code > generation: what is basic-block at a time generation used for? How do you > do global optimizations like register allocation? It is central to Psyco, the Python just-in-time specializer (http://psyco.sourceforge.net) whose techniques we plan to integrate with PyPy. Unlike other environments like Self, which collects execution profiles during interpretation and use them to recompile whole functions, Psyco has no interpretation stage: it directly emits a basic block and run it; the values found at run-time trigger the compilation of more basic blocks, which are run, and so on. So each function's machine code is a dynamic network of basic blocks which are various specialized versions of a bit of the original function. This network is not statically known, in particular because basic blocks often have a "switch" exit based on some value or type collected at run-time. Every new value encountered at this point triggers the compilation of a new switch case jumping to a new basic block. We will also certainly consider Self-style recompilations, as they allow more agressive optimizations. (Register allocation in Psyco is done using a simple round-robin scheme; code generation is very fast.) > That would be great! We've tossed around the idea of creating C bindings > for LLVM, which would make interfacing from other languages easier than > going directly to the C++ API, but we just haven't had a chance to yet. > Maybe you guys would be interested in helping with that project? Well, as the C++ API is nice and clean it is probably simpler to bind it directly to Python. We would probably go for Boost-Python, which makes C++ objects directly accessible to Python. But nothing is sure about this; maybe driving LLVM from LLVM code is closer to our needs. Is there a specific interface to do that? Is it possible to extract from LLVM the required code only, and link it with the final executable? In my experience, there are a few limitations of C that require explicit assembly code, like building calls dynamically (i.e. the caller's equivalent of varargs). A bientot, Armin.