From fijall at gmail.com Sun Mar 1 16:50:57 2009 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sun, 1 Mar 2009 16:50:57 +0100 Subject: [pypy-dev] pypy translation error :\ In-Reply-To: <34f4097d0902280009p17e54432ked63e3542b271539@mail.gmail.com> References: <34f4097d0902280009p17e54432ked63e3542b271539@mail.gmail.com> Message-ID: <693bc9ab0903010750g40fc9df6w57a6a43dfb19e3df@mail.gmail.com> thanks for the report, should be fixed by now. On Sat, Feb 28, 2009 at 9:09 AM, Jurgis Pralgauskis wrote: > Hello, > > maybe smb could give a hint, why: > http://files.akl.lt/users/jurgis/etc/pypy_translation_error.txt > > ps.: I acted according to instructions on > http://blog.sandbox.lt/en/WSGI%20and%20PyPy%20sandbox > my env: python 2.5.2, Ubuntu 8.04 > > > Thanks in advance > -- > Jurgis Pralgauskis > Don't worry, be happy and make things better ;) > http://sagemath.visiems.lt > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From santagada at gmail.com Mon Mar 2 16:00:03 2009 From: santagada at gmail.com (Leonardo Santagada) Date: Mon, 2 Mar 2009 12:00:03 -0300 Subject: [pypy-dev] Parsing in PyPy (and runicode) In-Reply-To: <49A86AEB.6080902@temporal-wave.com> References: <6CA22A60-3D3F-44A2-A121-23B0CA8FCC12@gmail.com> <4dab5f760902270642y526129e3s2fcee99eff45579b@mail.gmail.com> <200902271919.03717.jacob@openend.se> <693E7D78-65B6-4680-A34D-E4999915B172@gmail.com> <49A86AEB.6080902@temporal-wave.com> Message-ID: <6A9BE8C6-1FF1-4970-B319-59BBFBD0F571@gmail.com> On Feb 27, 2009, at 7:36 PM, Jim Idle wrote: > Leonardo Santagada wrote: >> >> On Feb 27, 2009, at 3:19 PM, Jacob Hall?n wrote: >> >>> fredagen den 27 februari 2009 skrev Frank Wierzbicki: >>>> On Thu, Feb 26, 2009 at 2:55 PM, Leonardo Santagada >>> > >>> >>> Andrew Dalke, who is very thorough in his investigation of >>> software, has >>> written som interesting things about his experience with ANTLR as >>> well as >>> some other parsing projects. In short, he likes ANTLR as a tool, >>> but in his >>> application, it is considerably slower than some other alternatives. >>> He also has something called python4ply, which is a ready, MIT >>> licensed >>> parser for Python. >>> >>> You can find his articles on >>> http://www.dalkescientific.com/writings/diary/archive/ >> >> >> The problem he might be having is with the python backend for >> ANTLR, wich neither us (we are going to have to create a rpython >> one) nor cpython (which would use a c89 one) would have. but this >> is just a guess as I have had no time to read his article yet, > All I can find is info about using the Python backend. > Unfortunately, the Python backend is very slow. There was some > discussion between the Python runtime author and Guido about why - I > can't re-quote as it was private email, but basically the runtime > and the generated code are method-call heavy, which isn't a good > idea in Python. Also, as string handling and other things are not > particularly quick, it is hard to get Python to perform when running > a parser using a design that wasn't specifically tailored for > Python. After all, Python wasn't really aimed at writing things like > lexers and parsers and is much better at things in other domains. > All that said, I think that the Python runtime will get better as > there will be more expansive choices for backend runtime authors in > the future. Then again, is the speed of the parser going to be a > factor? Not much I think. Well I did take a look at both the support code and the generated code (and a fast look at the runtime). The support code in java is really crazy, at least for me I didn't get most of it (I should read the docs later). Now the generated code seems to be following the java backend so far as being almost RPython. there is a problem that RPython doesn't have sets so it will take some time to make it work... but I think it is doable. Somehow my generated code has "pass" before every block of code on both parsers and lexers, the reason for that is still a mistery for me (do anyone knows why?). Maybe in this weekend I will have more time to look/work more seriosly on this. > The Java runtime is a lot quicker that the Python runtime and unless > there are Python translation units with 25,000 lines (there will be > somewhere ;-), performance would not be a factor. > > When I wrote the C runtime however, I did not make a blind copy of > the Java runtime, hence the performance is akin to hand written > code. For instance the GNU C parser written for ANTLR and running > with the C runtime, is almost the same speed as the GNU C parser > itself. This is great, but is the code C89 or do you use something from a newer standard? Because the only way to have a shot at using it with cpython would be if it was C89... though using the same parser in jython and pypy would be cool enough. > None of this would help (in terms of ANTLR) if you want a Python > parser that runs in Python of course :-) Well knowing that the Java one is quick is a good indication that the rpython one can be quick also. Thanks for all the info and for the quick response, -- Leonardo Santagada santagada at gmail.com From arigo at tunes.org Mon Mar 2 16:47:14 2009 From: arigo at tunes.org (Armin Rigo) Date: Mon, 2 Mar 2009 16:47:14 +0100 Subject: [pypy-dev] Leysin Winter Sprint Message-ID: <20090302154714.GA21804@code0.codespeak.net> Hi all! ===================================================================== PyPy Leysin Winter Sprint (14-21th April 2009) ===================================================================== .. image:: http://www.ermina.ch/002.JPG The next PyPy sprint will be in Leysin, Switzerland, for the sixth time. This sprint will take place immediately after Easter. This is a fully public sprint: newcomers and topics other than those proposed below are welcome. ------------------------------ Goals and topics of the sprint ------------------------------ * The overall idea of the sprint is to continue working on making PyPy ready for general use. There are a few tasks left in there. In parallel, we will continue the work on the JIT, if there is general interest. And as usual, we are ready to add any other task -- please mention on the mailing list what you would like to work on; the list of task is not really fixed. * And as usual, the main side goal is to have fun in winter sports :-) We can take a day off for ski until Sunday, the 19th; afterwards, the installations close. (There was quite a lot of snow this winter, so there should be some left even though it's relatively late in the season.) ----------------------- Location & Accomodation ----------------------- Leysin, Switzerland, "same place as before". Let me refresh your memory: both the sprint venue and the lodging will be in a very spacious pair of chalets built specifically for bed & breakfast: http://www.ermina.ch/. The place has a good ADSL Internet connexion with wireless installed. You can of course arrange your own lodging anywhere (so long as you are in Leysin, you cannot be more than a 15 minutes walk away from the sprint venue), but I definitely recommend lodging there too -- you won't find a better view anywhere else (though you probably won't get much worse ones easily, either :-) Please *confirm* that you are coming so that we can adjust the reservations as appropriate. The rate so far has been around 60 CHF a night all included in 2-person rooms, with breakfast. There are larger rooms too (less expensive) and maybe the possibility to get a single room if you really want to. Please register by svn: http://codespeak.net/svn/pypy/extradoc/sprintinfo/leysin-winter-2009/people.txt or on the pypy-sprint mailing list if you do not yet have check-in rights: http://codespeak.net/mailman/listinfo/pypy-sprint You need a Swiss-to-(insert country here) power adapter. There will be some Swiss-to-EU adapters around -- bring a EU-format power strip if you have one. ----------- Exact times ----------- Officially, 14-21 April 2009. The idea is that the 14th should be the arrival day, so we won't work too much until the 15th. ----------------------------------------------------------------------- Armin From fwierzbicki at gmail.com Mon Mar 2 21:00:19 2009 From: fwierzbicki at gmail.com (Frank Wierzbicki) Date: Mon, 2 Mar 2009 15:00:19 -0500 Subject: [pypy-dev] Parsing in PyPy (and runicode) In-Reply-To: <49AC0F89.7040503@temporal-wave.com> References: <6CA22A60-3D3F-44A2-A121-23B0CA8FCC12@gmail.com> <4dab5f760902270642y526129e3s2fcee99eff45579b@mail.gmail.com> <200902271919.03717.jacob@openend.se> <693E7D78-65B6-4680-A34D-E4999915B172@gmail.com> <49A86AEB.6080902@temporal-wave.com> <6A9BE8C6-1FF1-4970-B319-59BBFBD0F571@gmail.com> <49AC0F89.7040503@temporal-wave.com> Message-ID: <4dab5f760903021200j11c31023r3a768ec98d3c0994@mail.gmail.com> On Mon, Mar 2, 2009 at 11:55 AM, Jim Idle wrote: >> Well knowing that the Java one is quick is a good indication that the >> rpython one can be quick also. > > Yes - I think so. Perhaps one way is to ignore the current runtime, which > has been written more for functionality than performance as far as I can > see, and write an RPython runtime that is strictly aimed at performance and > RPython semantics etc. It is not a huge project to do this, but it is some > work of course. Indeed the Java parser has not been optimized for performance (yet) -- it was written with compatibility as the #1 concern. In particular I'm sure I've lost a bit of performance in order to support _ast.py and ast.py. -Frank From santagada at gmail.com Wed Mar 4 02:38:40 2009 From: santagada at gmail.com (Leonardo Santagada) Date: Tue, 3 Mar 2009 22:38:40 -0300 Subject: [pypy-dev] Compiling RSDL and ExternalCompilationInfo Message-ID: I've tried running some RSDL examples in my computer and they are not working, but if I try to compile the code by hand it works (without the framework and includes it does complain the same as translation on pypy): Apfelstrudel:~/projects/pypy/pypy/rlib/rsdl/macosx-sdl-main santagada$ gcc -I/Library/Frameworks/SDL.framework/Headers -framework Cocoa - framework SDL SDLMain.m Undefined symbols: "_SDL_main", referenced from: -[SDLMain applicationDidFinishLaunching:] in ccCdkKBZ.o ld: symbol(s) not found collect2: ld returned 1 exit status The error is because the supporting code does not have a main (I think). But when I try to translate something with pypy I get this other error on this paste http://paste.pocoo.org/show/106398/ wich ends with: pypy.translator.platform.CompilationError: Apfelstrudel:~/projects/pypy/pypy/rlib/rsdl santagada$ Using py.test to run the example it stops saying I don't have sdl installed (the error defined on the check_sdl_installation in eci.py. The eci.py file should be defining it but besides Version/A/Headers (which I also tried changing on my WC) appears to be trying to define what I typed to gcc. What could be going on? -- Leonardo Santagada santagada at gmail.com From pedronis at openend.se Thu Mar 5 23:57:26 2009 From: pedronis at openend.se (Samuele Pedroni) Date: Thu, 05 Mar 2009 23:57:26 +0100 Subject: [pypy-dev] cleaned builds with lots of failures, issues with nightly translations Message-ID: <49B058D6.70206@openend.se> I cleaned up the builds with lots of failures that happened on 3rd. Btw build files are stored with names starting with build numbers in the builder directories (named like the builders) under /home/buildmaster/pypy on codespeak. It seems that some our nightly translations by buildbot are timing out now, it may be legitimate and then we need to up the timeouts or something wrong may be going on, I don't have time to investigate that right now. Samuele From holger at merlinux.eu Fri Mar 6 10:22:52 2009 From: holger at merlinux.eu (holger krekel) Date: Fri, 6 Mar 2009 10:22:52 +0100 Subject: [pypy-dev] cleaned builds with lots of failures, issues with nightly translations In-Reply-To: <49B058D6.70206@openend.se> References: <49B058D6.70206@openend.se> Message-ID: <20090306092252.GZ4576@trillke.net> Hi Samuele, On Thu, Mar 05, 2009 at 23:57 +0100, Samuele Pedroni wrote: > I cleaned up the builds with lots of failures that happened on 3rd. great, thanks. > Btw build files are stored with names starting with build numbers in the > builder directories (named like the builders) under > /home/buildmaster/pypy on codespeak. ok. what's the best way to find out the svn revision those build numbers belong to? > It seems that some our nightly translations by buildbot are timing out > now, it may be legitimate and then we need to up the timeouts or > something wrong may be going on, > I don't have time to investigate that right now. > at least http://tinyurl.com/dcbn8v indicates that our lib-python tests ran against a recent pypy-c, with only three failures out of 250 cpython regression test files. didn'T look further. holger -- Metaprogramming, Python, Testing: http://tetamap.wordpress.com Python, PyPy, pytest contracting: http://merlinux.eu From jurgis.pralgauskis at gmail.com Sun Mar 8 00:24:27 2009 From: jurgis.pralgauskis at gmail.com (Jurgis Pralgauskis) Date: Sun, 8 Mar 2009 01:24:27 +0200 Subject: [pypy-dev] sandboxed exec function with context passing? Message-ID: <34f4097d0903071524n16ebbfet98f87a61a7bb48ff@mail.gmail.com> I would like to use sandboxed pypy for online step by step tutorials... 1) I need ability to pass context then. 2) and also would be nice to get result as variable not just like output. my experiments are based on explanations from http://blog.sandbox.lt/en/WSGI%20and%20PyPy%20sandbox maybe my question seems too trivial, but I looked around pypy_interact code, its class inherits from 2 calsses I followed one of them and went till sandlib.py handle_until_return() but here didn't see anything apparent and felt like lost... could smb direct me ? :) or just tell the functions which would act like usual eval/exec (just sandboxed) Thanks in advance -- Jurgis Pralgauskis From arigo at tunes.org Mon Mar 9 17:46:50 2009 From: arigo at tunes.org (Armin Rigo) Date: Mon, 9 Mar 2009 17:46:50 +0100 Subject: [pypy-dev] sandboxed exec function with context passing? In-Reply-To: <34f4097d0903071524n16ebbfet98f87a61a7bb48ff@mail.gmail.com> References: <34f4097d0903071524n16ebbfet98f87a61a7bb48ff@mail.gmail.com> Message-ID: <20090309164650.GA5434@code0.codespeak.net> Hi Jurgis, On Sun, Mar 08, 2009 at 01:24:27AM +0200, Jurgis Pralgauskis wrote: > I would like to use sandboxed pypy for online step by step tutorials... > 1) I need ability to pass context then. > 2) and also would be nice to get result as variable not just like output. There is no way any security mechanism can allow arbitrary objects to be passed in or out, as this would create a security hole. In our case, it is an extreme version of this constrain. The subprocess is really a different process, and you communicate with it using only the (sandboxed) I/O functions. That means that you must encode everything as I/O operations (and maybe build nicer abstractions to use on top of that, but these would have to use the low-level I/O method of exchanging information). There is nothing like that built so far, but you can easily use e.g. the marshal module: marshal the data outside, and send it to the sandboxed process via a pseudo "file read", and then unmarshal it there. And the same in the other direction with a "file write". A bientot, Armin. From anto.cuni at gmail.com Tue Mar 10 15:48:26 2009 From: anto.cuni at gmail.com (Antonio Cuni) Date: Tue, 10 Mar 2009 15:48:26 +0100 Subject: [pypy-dev] Leysin Winter Sprint In-Reply-To: <20090302154714.GA21804@code0.codespeak.net> References: <20090302154714.GA21804@code0.codespeak.net> Message-ID: <49B67DBA.3060702@gmail.com> Armin Rigo wrote: > Please *confirm* that you are coming so that we can adjust the reservations > as appropriate. The rate so far has been around 60 CHF a night all included > in 2-person rooms, with breakfast. There are larger rooms too (less > expensive) and maybe the possibility to get a single room if you really want > to. is 60 CHF per-person or per-room? Anyone interested in sharing one room? ciao, Anto From lac at openend.se Wed Mar 11 15:01:34 2009 From: lac at openend.se (Laura Creighton) Date: Wed, 11 Mar 2009 15:01:34 +0100 Subject: [pypy-dev] RunSnakeRun Message-ID: <200903111401.n2BE1YA9004344@theraft.openend.se> I'd never heard of this before, and thought it looked cool. http://www.vrplumber.com/programming/runsnakerun/ Laura From arigo at tunes.org Wed Mar 11 15:10:59 2009 From: arigo at tunes.org (Armin Rigo) Date: Wed, 11 Mar 2009 15:10:59 +0100 Subject: [pypy-dev] Leysin Winter Sprint In-Reply-To: <49B67DBA.3060702@gmail.com> References: <20090302154714.GA21804@code0.codespeak.net> <49B67DBA.3060702@gmail.com> Message-ID: <20090311141059.GA535@code0.codespeak.net> Hi Anto, On Tue, Mar 10, 2009 at 03:48:26PM +0100, Antonio Cuni wrote: > is 60 CHF per-person or per-room? It's the price per person. A bientot, Armin. From irving at naml.us Fri Mar 13 01:57:21 2009 From: irving at naml.us (Geoffrey Irving) Date: Thu, 12 Mar 2009 20:57:21 -0400 Subject: [pypy-dev] pypy for typed languages? Message-ID: <7f9d599f0903121757k37421a77kaa707214d0ce291c@mail.gmail.com> Hello, I've come to the conclusion that JIT compilation is essential to avoid abstract penalties even for typed languages. The one sentence summary is that the C++ inline keyword is exactly the opposite of what you want: in a JIT language you can have a "flatten" function that recursively inlines everything that a function calls (including closures). Thus, I'm curious about whether pypy could be practically applied to a typed front end language, say with the following basic set of features: 1. Packed storage such as C structs and arrays. 2. A Hindley Milner polymorphic type system with type classes. In particular, what restrictions does pypy impose on storage layout? For example, would it be able to handle dynamically-typed homogeneous lists, represented as a single pointer to a type object and an array of structs required to be of that type? Also, if pypy can target LLVM, how do you imagine the two JIT systems cooperating (if at all)? Thanks! Geoffrey From william.leslie.ttg at gmail.com Fri Mar 13 03:17:18 2009 From: william.leslie.ttg at gmail.com (William Leslie) Date: Fri, 13 Mar 2009 13:17:18 +1100 Subject: [pypy-dev] pypy for typed languages? In-Reply-To: <7f9d599f0903121757k37421a77kaa707214d0ce291c@mail.gmail.com> References: <7f9d599f0903121757k37421a77kaa707214d0ce291c@mail.gmail.com> Message-ID: (sorry Geoffrey for the missend) 2009/3/13 Geoffrey Irving : > Hello, Hi! and welcome. > I'm curious about whether pypy could be practically applied to a > typed front end language, say with the following basic set of > features: > > 1. Packed storage such as C structs and arrays. >From my limited understanding this shouldn't be too difficult. Packed storage is provided at app level, for example, by the ctypes and rawffi modules, and at interpreter level by using lltypes directly (which is what the JIT operates on). You could evaluate defstruct by manipulating lltypes. Class definitions could possibly use rpython.rclass (maybe with a few modifications if classes have strict layout requirements too). > 2. A Hindley Milner polymorphic type system with type classes. I'll leave it to someone with a better knowledge of how the JIT takes advantage of types to answer this. William Leslie From arigo at tunes.org Fri Mar 13 12:13:02 2009 From: arigo at tunes.org (Armin Rigo) Date: Fri, 13 Mar 2009 12:13:02 +0100 Subject: [pypy-dev] pypy for typed languages? In-Reply-To: <7f9d599f0903121757k37421a77kaa707214d0ce291c@mail.gmail.com> References: <7f9d599f0903121757k37421a77kaa707214d0ce291c@mail.gmail.com> Message-ID: <20090313111302.GA27894@code0.codespeak.net> Hi Geoffrey, On Thu, Mar 12, 2009 at 08:57:21PM -0400, Geoffrey Irving wrote: > I've come to the conclusion that JIT compilation is essential to avoid > abstract penalties even for typed languages. Indeed: our approach to JIT compilation was first tried with machine code (see Dynamo/DynamoRio) and then with Java. > Thus, I'm curious about whether pypy could be practically applied to a > typed front end language, say with the following basic set of > features: I suppose you can try. For a direct application you need a language for which it makes sense to write an interpreter in RPython. > In particular, what restrictions does pypy impose on storage layout? > For example, would it be able to handle dynamically-typed homogeneous > lists, represented as a single pointer to a type object and an array > of structs required to be of that type? RPython doesn't have support for this. If you go directly to the lltype type system, then it's possible -- but our current JIT generator doesn't support it. And also, if you write an interpreter using the lltype type system directly, you loose the abstraction level of writing an interpreter in RPython in the first place -- e.g. it would not be translatable any more to the ootype type system, thus to Java or CLI. So all in all, PyPy could probably be subverted to do what you want, but it's not really meant to (and we are a bit unlikely to give much support, as far as I can see). > Also, if pypy can target LLVM, how do you imagine the two JIT systems > cooperating (if at all)? That's a longer-term goal, but I can easily imagine that the code produced by our JIT could be sent to LLVM at runtime for further optimizations. A bientot, Armin. From irving at naml.us Fri Mar 13 15:04:41 2009 From: irving at naml.us (Geoffrey Irving) Date: Fri, 13 Mar 2009 10:04:41 -0400 Subject: [pypy-dev] pypy for typed languages? In-Reply-To: <20090313111302.GA27894@code0.codespeak.net> References: <7f9d599f0903121757k37421a77kaa707214d0ce291c@mail.gmail.com> <20090313111302.GA27894@code0.codespeak.net> Message-ID: <7f9d599f0903130704g74e62e68lad65d735c85eb9e6@mail.gmail.com> On Fri, Mar 13, 2009 at 7:13 AM, Armin Rigo wrote: >> In particular, what restrictions does pypy impose on storage layout? >> For example, would it be able to handle dynamically-typed homogeneous >> lists, represented as a single pointer to a type object and an array >> of structs required to be of that type? > > RPython doesn't have support for this. ?If you go directly to the lltype > type system, then it's possible -- but our current JIT generator doesn't > support it. ?And also, if you write an interpreter using the lltype type > system directly, you loose the abstraction level of writing an > interpreter in RPython in the first place -- e.g. it would not be > translatable any more to the ootype type system, thus to Java or CLI. > > So all in all, PyPy could probably be subverted to do what you want, but > it's not really meant to (and we are a bit unlikely to give much > support, as far as I can see). Thanks for the reply. It seems like targeting LLVM directly is the way to go for such projects for now. Geoffrey From filippo at esaurito.net Fri Mar 13 16:00:40 2009 From: filippo at esaurito.net (Filippo Giunchedi) Date: Fri, 13 Mar 2009 16:00:40 +0100 Subject: [pypy-dev] distributed version control mirror of trunk? Message-ID: <20090313150040.GB4625@clamp.esaurito.net> Hi, regarding the switch to a DVCS discussed in http://codespeak.net/pipermail/pypy-dev/2008q3/004816.html I was wondering if there's a (one way) mirror of SVN trunk/branches to a DVCS such as git or if such mirror is planned or wished. thanks in advance, filippo -- Filippo Giunchedi - http://esaurito.net - 0x6B79D401 I never forget a face, but in your case I'll be glad to make an exception. -- Groucho Marx -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: Digital signature URL: From fijall at gmail.com Fri Mar 13 16:19:28 2009 From: fijall at gmail.com (Maciej Fijalkowski) Date: Fri, 13 Mar 2009 16:19:28 +0100 Subject: [pypy-dev] pypy for typed languages? In-Reply-To: <7f9d599f0903130704g74e62e68lad65d735c85eb9e6@mail.gmail.com> References: <7f9d599f0903121757k37421a77kaa707214d0ce291c@mail.gmail.com> <20090313111302.GA27894@code0.codespeak.net> <7f9d599f0903130704g74e62e68lad65d735c85eb9e6@mail.gmail.com> Message-ID: <693bc9ab0903130819y1a686216h8cdada85fca59d17@mail.gmail.com> On Fri, Mar 13, 2009 at 3:04 PM, Geoffrey Irving wrote: > On Fri, Mar 13, 2009 at 7:13 AM, Armin Rigo wrote: >>> In particular, what restrictions does pypy impose on storage layout? >>> For example, would it be able to handle dynamically-typed homogeneous >>> lists, represented as a single pointer to a type object and an array >>> of structs required to be of that type? >> >> RPython doesn't have support for this. ?If you go directly to the lltype >> type system, then it's possible -- but our current JIT generator doesn't >> support it. ?And also, if you write an interpreter using the lltype type >> system directly, you loose the abstraction level of writing an >> interpreter in RPython in the first place -- e.g. it would not be >> translatable any more to the ootype type system, thus to Java or CLI. >> >> So all in all, PyPy could probably be subverted to do what you want, but >> it's not really meant to (and we are a bit unlikely to give much >> support, as far as I can see). > > Thanks for the reply. ?It seems like targeting LLVM directly is the > way to go for such projects for now. > I don't know, do you have any example? From irving at naml.us Fri Mar 13 16:48:39 2009 From: irving at naml.us (Geoffrey Irving) Date: Fri, 13 Mar 2009 11:48:39 -0400 Subject: [pypy-dev] pypy for typed languages? In-Reply-To: <693bc9ab0903130819y1a686216h8cdada85fca59d17@mail.gmail.com> References: <7f9d599f0903121757k37421a77kaa707214d0ce291c@mail.gmail.com> <20090313111302.GA27894@code0.codespeak.net> <7f9d599f0903130704g74e62e68lad65d735c85eb9e6@mail.gmail.com> <693bc9ab0903130819y1a686216h8cdada85fca59d17@mail.gmail.com> Message-ID: <7f9d599f0903130848m282b371fk231e4b17bb9cff34@mail.gmail.com> On Fri, Mar 13, 2009 at 11:19 AM, Maciej Fijalkowski wrote: > On Fri, Mar 13, 2009 at 3:04 PM, Geoffrey Irving wrote: >> On Fri, Mar 13, 2009 at 7:13 AM, Armin Rigo wrote: >>>> In particular, what restrictions does pypy impose on storage layout? >>>> For example, would it be able to handle dynamically-typed homogeneous >>>> lists, represented as a single pointer to a type object and an array >>>> of structs required to be of that type? >>> >>> RPython doesn't have support for this. ?If you go directly to the lltype >>> type system, then it's possible -- but our current JIT generator doesn't >>> support it. ?And also, if you write an interpreter using the lltype type >>> system directly, you loose the abstraction level of writing an >>> interpreter in RPython in the first place -- e.g. it would not be >>> translatable any more to the ootype type system, thus to Java or CLI. >>> >>> So all in all, PyPy could probably be subverted to do what you want, but >>> it's not really meant to (and we are a bit unlikely to give much >>> support, as far as I can see). >> >> Thanks for the reply. ?It seems like targeting LLVM directly is the >> way to go for such projects for now. > > I don't know, do you have any example? At this point I'm just in the curiosity stage, so I don't have a concrete example. Roughly, I'm imagining a language based on System CT [1] or the like, which is somewhat like duck typing for static languages with homogeneous data structures. Since all function can be overloaded, at runtime functions would pass around large dictionaries of overloads, which would need to be optimized away via JIT for reasonable speed. A JIT would have an easier job on a System CT-like language than with python or java, due to the guarantee of homogeneity. For example, in the sum function def sum(container): sum = zero for elem in container: sum = sum + elem return sum a language with homogeneous containers knows that "+" is constant throughout the loop even if it is impossible to know which constant it is in advance. Geoffrey [1] http://homepages.dcc.ufmg.br/~camarao/CT From santagada at gmail.com Fri Mar 13 17:40:34 2009 From: santagada at gmail.com (Leonardo Santagada) Date: Fri, 13 Mar 2009 13:40:34 -0300 Subject: [pypy-dev] distributed version control mirror of trunk? In-Reply-To: <20090313150040.GB4625@clamp.esaurito.net> References: <20090313150040.GB4625@clamp.esaurito.net> Message-ID: <3CE783A0-2E3D-471A-9521-C3271302977B@gmail.com> On Mar 13, 2009, at 12:00 PM, Filippo Giunchedi wrote: > Hi, > regarding the switch to a DVCS discussed in > http://codespeak.net/pipermail/pypy-dev/2008q3/004816.html I was > wondering if > there's a (one way) mirror of SVN trunk/branches to a DVCS such as > git or if > such mirror is planned or wished. I think michael hudson did have something on launchpad at one time, but I don't think it is updated anymore. I wish there was a two way mirror to hg, but I do so little work on pypy that my vote doesnt count :) (and I really don't mind svn much). -- Leonardo Santagada santagada at gmail.com From p.giarrusso at gmail.com Sat Mar 14 05:28:32 2009 From: p.giarrusso at gmail.com (Paolo Giarrusso) Date: Sat, 14 Mar 2009 05:28:32 +0100 Subject: [pypy-dev] pypy for typed languages? In-Reply-To: <7f9d599f0903130848m282b371fk231e4b17bb9cff34@mail.gmail.com> References: <7f9d599f0903121757k37421a77kaa707214d0ce291c@mail.gmail.com> <20090313111302.GA27894@code0.codespeak.net> <7f9d599f0903130704g74e62e68lad65d735c85eb9e6@mail.gmail.com> <693bc9ab0903130819y1a686216h8cdada85fca59d17@mail.gmail.com> <7f9d599f0903130848m282b371fk231e4b17bb9cff34@mail.gmail.com> Message-ID: On Fri, Mar 13, 2009 at 16:48, Geoffrey Irving wrote: > On Fri, Mar 13, 2009 at 11:19 AM, Maciej Fijalkowski wrote: >> On Fri, Mar 13, 2009 at 3:04 PM, Geoffrey Irving wrote: >>> On Fri, Mar 13, 2009 at 7:13 AM, Armin Rigo wrote: >>>>> In particular, what restrictions does pypy impose on storage layout? >>>>> For example, would it be able to handle dynamically-typed homogeneous >>>>> lists, represented as a single pointer to a type object and an array >>>>> of structs required to be of that type? >>>> >>>> RPython doesn't have support for this. ?If you go directly to the lltype >>>> type system, then it's possible -- but our current JIT generator doesn't >>>> support it. ?And also, if you write an interpreter using the lltype type >>>> system directly, you loose the abstraction level of writing an >>>> interpreter in RPython in the first place -- e.g. it would not be >>>> translatable any more to the ootype type system, thus to Java or CLI. >>>> >>>> So all in all, PyPy could probably be subverted to do what you want, but >>>> it's not really meant to (and we are a bit unlikely to give much >>>> support, as far as I can see). >>> >>> Thanks for the reply. ?It seems like targeting LLVM directly is the >>> way to go for such projects for now. >> >> I don't know, do you have any example? > > At this point I'm just in the curiosity stage, so I don't have a > concrete example. ?Roughly, I'm imagining a language based on System > CT [1] or the like, which is somewhat like duck typing for static > languages with homogeneous data structures. ?Since all function can be > overloaded, at runtime functions would pass around large dictionaries > of overloads, which would need to be optimized away via JIT for > reasonable speed. What is your plan for optimizing those? Are you already planning to use inline caching for that (what I describe below)? > A JIT would have an easier job on a System CT-like language than with > python or java, due to the guarantee of homogeneity. ?For example, in > the sum function > > def sum(container): > ? ?sum = zero > ? ?for elem in container: > ? ? ? ?sum = sum + elem > ? ?return sum > > a language with homogeneous containers knows that "+" is constant > throughout the loop even if it is impossible to know which constant it > is in advance. That can be useful, but in modern JIT that's maybe not the key point, although you can surely make also use of that. Most JIT have a heuristic solution which optimizes late binding anyway, without semantic restrictions, through inline caching or aggressive inlining - around 90% percent of call sites always call the same actual method (i.e. they are monomorphic), so code similar to: if (dest->type == ACertainType) ACertainType::method(args); //Early bound call, possibly inlined. else dest->method(args); //Late bound call can be generated by the JIT. This avoids the indirect jump in most cases. I'm omitting various details needed to make it actually go fast, but they're described in the literature. For that case, it'd be likely better to inline that function into the caller to get monomorphic call sites (and by any standard that function should be small enough for that). Having a constant '+' doesn't help so much if it can have any value. You save a complete method lookup over a hierarchy, in the Python case (or in Java with inteface methods), but you still have a call through a function pointer, which can be quite slow; applying the above technique Implementing the above idea in generated JVM bytecode made a Jython-like JIT compiler for Python, written by another student at the course, be much faster than CPython (they reported speedups of at least 3x to 5x, with much room for improvements). I think also PyPy does something similar for Python, but only through a hash table. I'd love to see how much inline caching would help on top of that, when I'll have time to start working on this. Regards -- Paolo Giarrusso From arigo at tunes.org Sat Mar 14 18:52:19 2009 From: arigo at tunes.org (Armin Rigo) Date: Sat, 14 Mar 2009 18:52:19 +0100 Subject: [pypy-dev] pypy for typed languages? In-Reply-To: References: <7f9d599f0903121757k37421a77kaa707214d0ce291c@mail.gmail.com> <20090313111302.GA27894@code0.codespeak.net> <7f9d599f0903130704g74e62e68lad65d735c85eb9e6@mail.gmail.com> <693bc9ab0903130819y1a686216h8cdada85fca59d17@mail.gmail.com> <7f9d599f0903130848m282b371fk231e4b17bb9cff34@mail.gmail.com> Message-ID: <20090314175219.GA12036@code0.codespeak.net> Hi Paolo, On Sat, Mar 14, 2009 at 05:28:32AM +0100, Paolo Giarrusso wrote: > I think also PyPy does something similar for Python, but only through > a hash table. I'd love to see how much inline caching would help on > top of that, when I'll have time to start working on this. No. PyPy does something very different, more similar to trace-based JITs (goggle :-) That does not involve hash tables (I don't know what you are referring to here). A bientot, Armin. From steve at shrogers.com Tue Mar 17 03:36:15 2009 From: steve at shrogers.com (Steven H. Rogers) Date: Mon, 16 Mar 2009 20:36:15 -0600 Subject: [pypy-dev] Speeding Up SimPy Message-ID: <49BF0C9F.7030004@shrogers.com> SimPy (http://simpy.sf.net) is a very nice discrete event simulation package written in Python. It is, however, rather slow for large models and I'm looking at ways to speed it up. SimPy should be a good test case for PyPy and when the JIT stabilizes, PyPy may provide faster execution times. As we'd like to see some immediate benefits, I've tried Psyco which provides a worthwhile 45% improvement on one model and I'll look at modifying SimPy to work better with it. SimPy makes heavy use of generators, so I'm interested in the support for them planned for the next release. How close is Psyco generator support to being useful? Greenlets look like they could be a viable alternative to generators for simulating concurrency. How expensive are greenlets compared to generators? Regards, Steve From richard.m.tew at gmail.com Tue Mar 17 04:18:53 2009 From: richard.m.tew at gmail.com (Richard Tew) Date: Mon, 16 Mar 2009 23:18:53 -0400 Subject: [pypy-dev] Speeding Up SimPy In-Reply-To: <49BF0C9F.7030004@shrogers.com> References: <49BF0C9F.7030004@shrogers.com> Message-ID: <952d92df0903162018u314e2c9sddb7bff4044265d4@mail.gmail.com> On Mon, Mar 16, 2009 at 10:36 PM, Steven H. Rogers wrote: > Greenlets look like they could be a viable alternative to generators for > simulating concurrency. ?How expensive are greenlets compared to generators? I haven't heard of anyone who has benchmarked the difference. I'd be interested in seeing some results myself. I could handwave based on indirect reports, for what it's worth, as it's easier than actually benchmarking it myself :-) The Concurrence framework supports running on top of greenlets and on top of Stackless Python, which greenlets are a spin off of. It claims that there's only a 10-25% difference in performance in Stackless' favour [1]. Stackless Python and a generator based solution were compared in a recent benchmark [2] with generators being around 10 times slower than Stackless. Given that both reports are more or less accurate, I expect greenlets to be considerably faster than generators. Cheers, Richard. [1] http://opensource.hyves.org/concurrence/index.html [2] http://entitycrisis.blogspot.com/2009/03/concurrent-scaling-benchmarks.html From cumber at netspace.net.au Tue Mar 17 10:03:42 2009 From: cumber at netspace.net.au (Ben Mellor) Date: Tue, 17 Mar 2009 20:03:42 +1100 Subject: [pypy-dev] RPython I/O Message-ID: <20090317200342.1426a1ec@intyalie> Hi Just wondering, what's the right way to do IO in an RPython program? sys.stdin/sys.stdout don't work, neither does opening files using open(), and raw_input doesn't work. All I can get is print. I can't find anything about this in the docs, and I haven't been able to understand how the PyPy interpreter is doing it from looking at the source code so far. I've been playing around with using the PyPy framework to implement a small interpreter, and this is currently blocking me from translating it. -- Ben Mellor From fijall at gmail.com Tue Mar 17 10:08:11 2009 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 17 Mar 2009 10:08:11 +0100 Subject: [pypy-dev] RPython I/O In-Reply-To: <20090317200342.1426a1ec@intyalie> References: <20090317200342.1426a1ec@intyalie> Message-ID: <693bc9ab0903170208o322aaff7gcf57b559ee04e5cc@mail.gmail.com> There are two official ways: 1. use directly os.read/os.open/os.write, rather tedious if you ask me 2. use interfaces that are present in pypy/rlib/streamio.py. I think only documentation for that is in tests unfortunately :( It has interface similar to python's streams, but adapted a bit to accomodate rpythonism. Cheers, fijal On Tue, Mar 17, 2009 at 10:03 AM, Ben Mellor wrote: > Hi > > Just wondering, what's the right way to do IO in an RPython program? > > sys.stdin/sys.stdout don't work, neither does opening files using open(), and > raw_input doesn't work. All I can get is print. I can't find anything about > this in the docs, and I haven't been able to understand how the PyPy > interpreter is doing it from looking at the source code so far. > > I've been playing around with using the PyPy framework to implement a small > interpreter, and this is currently blocking me from translating it. > > -- Ben Mellor > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From steve at shrogers.com Tue Mar 17 13:55:45 2009 From: steve at shrogers.com (Steven H. Rogers) Date: Tue, 17 Mar 2009 06:55:45 -0600 Subject: [pypy-dev] Speeding Up SimPy In-Reply-To: <952d92df0903162018u314e2c9sddb7bff4044265d4@mail.gmail.com> References: <49BF0C9F.7030004@shrogers.com> <952d92df0903162018u314e2c9sddb7bff4044265d4@mail.gmail.com> Message-ID: <49BF9DD1.20304@shrogers.com> Richard Tew wrote: > On Mon, Mar 16, 2009 at 10:36 PM, Steven H. Rogers wrote: > >> Greenlets look like they could be a viable alternative to generators for >> simulating concurrency. How expensive are greenlets compared to generators? >> > > I haven't heard of anyone who has benchmarked the difference. I'd be > interested in seeing some results myself. > > I could handwave based on indirect reports, for what it's worth, as > it's easier than actually benchmarking it myself :-) ... > Thanks Richard. A little hand waving from someone with your experience provides good insight. I'll start work on some benchmarks. Does anyone know how well greenlets and Psyco play together? Regards, Steve From jacob at openend.se Tue Mar 17 19:50:35 2009 From: jacob at openend.se (Jacob =?utf-8?q?Hall=C3=A9n?=) Date: Tue, 17 Mar 2009 19:50:35 +0100 Subject: [pypy-dev] Processor simulators Message-ID: <200903171950.35840.jacob@openend.se> While catching up on todays events on the IRC channel, it struck me that we are getting to a point where using a simulator for the processor would be very useful. Once upon a time I did a lot of development of embedded systems, and using a simulator was a really nice way of timing and debugging your code. You could also do neat tricks if you had the source code of the simulator, recreating rare events and generally break the notion of your hardware being an unchangeable constant. I did a quick Goggle search and found that there are plenty of simulators for the X86 architecture, with several being fully Open Sourced. For instance, at http://www.ptlsim.org/, there is a simulator that can run code natively on the machine and then jump into and out of simulation mode. This allows for precise timing of your selected pieces of code and the possibility of tracing exactly what happens at the machine level, with cache misses and whatnot. I think it might be a useful tool for some of the current backend work for the JIT, and for future optimizations. Jacob From cumber at netspace.net.au Wed Mar 18 07:45:33 2009 From: cumber at netspace.net.au (Ben Mellor) Date: Wed, 18 Mar 2009 17:45:33 +1100 Subject: [pypy-dev] RPython I/O In-Reply-To: <693bc9ab0903170208o322aaff7gcf57b559ee04e5cc@mail.gmail.com> References: <20090317200342.1426a1ec@intyalie> <693bc9ab0903170208o322aaff7gcf57b559ee04e5cc@mail.gmail.com> Message-ID: <20090318174533.583f749e@intyalie> Thanks! Got it working now. If I do: stdin_fd = sys.stdin.fileno() during import time and then use that with os.read (either directly or indirectly through streams), will it also be translated correctly for the other non-C backends? > There are two official ways: > > 1. use directly os.read/os.open/os.write, rather tedious if you ask me > 2. use interfaces that are present in pypy/rlib/streamio.py. I think > only documentation for that is in tests unfortunately :( > > It has interface similar to python's streams, but adapted a bit to > accomodate rpythonism. > > Cheers, > fijal > > On Tue, Mar 17, 2009 at 10:03 AM, Ben Mellor wrote: > > Hi > > > > Just wondering, what's the right way to do IO in an RPython program? > > > > sys.stdin/sys.stdout don't work, neither does opening files using open(), > > and raw_input doesn't work. All I can get is print. I can't find anything > > about this in the docs, and I haven't been able to understand how the PyPy > > interpreter is doing it from looking at the source code so far. > > > > I've been playing around with using the PyPy framework to implement a small > > interpreter, and this is currently blocking me from translating it. > > > > -- Ben Mellor > > _______________________________________________ > > pypy-dev at codespeak.net > > http://codespeak.net/mailman/listinfo/pypy-dev > > From lac at openend.se Mon Mar 23 10:14:50 2009 From: lac at openend.se (Laura Creighton) Date: Mon, 23 Mar 2009 10:14:50 +0100 Subject: [pypy-dev] EuroSciPy -- Place to find people who might like to try out the JIT? Message-ID: <200903230914.n2N9Eo5a021471@theraft.openend.se> just a thought, Laura ------- Forwarded Message Return-Path: python-announce-list-bounces+lac=openend.se at python.org Delivery-Date: Mon Mar 23 09:32:52 2009 Date: Sun, 22 Mar 2009 20:59:56 +0100 From: =?ISO-8859-15?Q?Mike_M=FCller?= Organization: Python Academy User-Agent: Thunderbird 2.0.0.19 (Windows/20081209) MIME-Version: 1.0 To: python-announce-list at python.org Subject: EuroSciPy 2009, Leipzig, Germany, July 25-26 EuroSciPy 2009 ============== We're pleased to announce the EuroSciPy 2009 Conference to be held in Leipzig, Germany on July 25-26, 2009. http://www.euroscipy.org This is the second conference after the successful conference last year. Again, EuroSciPy will be a venue for the European community of users of the Python programming language in science. Call for Participation - ---------------------- If you are a scientist using Python for your computational work, we'd love to have you formally present your results, methods or experiences. To apply to present a talk at this year's EuroSciPy, please submit an abstract of your talk as a PDF, MS Word or plain text file to mmueller at python-academy dot de. The deadline for abstract submission is April 30, 2009. Papers and/or presentation slides are acceptable and are due by June 15, 2009. Presentations will be allotted 30 minutes. Registration - ------------ Registration is open. The registration fee is 100.00 ? for early registrants and will increase to 150.00 ? for late registration after June 15, 2009. Registration will include breakfast, snacks and lunch for Saturday and Sunday. Please register here: http://www.euroscipy.org/presentations/index.html Important Dates - --------------- March 21 Registration opens April 30 Abstract submission deadline May 15 Acceptance of presentations May 30 Announcement of conference program June 15 Early bird registration deadline June 15 Paper/slides submission deadline July 20 - 24 Pre-Conference courses July 25/26 Conference Venue - ----- mediencampus Poetenweg 28 04155 Leipzig Germany See http://www.euroscipy.org/venue.html for details. Help Welcome - ------------ You like to help make the EuroSciPy 2009 a success? Here are some ways you can get involved: * attend the conference * submit an abstract for a presentation * give a lightning talk * make EuroSciPy known: - write about it on your website - in your blog - talk to friends about it - post to local e-mail lists - post to related forums - spread flyers and posters in your institution - make entries in relevant event calendars - anything you can think of * inform potential sponsors about the event * become a sponsor If you're interested in volunteering to help organize things or have some other idea that can help the conference, please email us at mmueller at python-academy dot de. Sponsorship - ----------- Do you like to sponsor the conference? There are several options available: http://www.euroscipy.org/sponsors/become_a_sponsor.html Pre-Conference Courses - ---------------------- Would you like to learn Python or about some of the most used scientific libraries in Python? Then the "Python Summer Course" [1] might be for you. There are two parts to this course: * a two-day course "Introduction to Python" [2] for people with programming experience in other languages and * a three-day course "Python for Scientists and Engineers" [3] that introduces some of the most used Python tools for scientists and engineers such as NumPy, PyTables, and matplotlib Both courses can be booked individually [4]. Of course, you can attend the courses without registering for EuroSciPy. [1] http://www.python-academy.com/courses/python_summer_course.html [2] http://www.python-academy.com/courses/python_course_programmers.html [3] http://www.python-academy.com/courses/python_course_scientists.html [4] http://www.python-academy.com/courses/dates.html - -- http://mail.python.org/mailman/listinfo/python-announce-list Support the Python Software Foundation: http://www.python.org/psf/donations.html ------- End of Forwarded Message From p.giarrusso at gmail.com Mon Mar 23 11:46:29 2009 From: p.giarrusso at gmail.com (Paolo Giarrusso) Date: Mon, 23 Mar 2009 11:46:29 +0100 Subject: [pypy-dev] pypy for typed languages? In-Reply-To: <20090314175219.GA12036@code0.codespeak.net> References: <7f9d599f0903121757k37421a77kaa707214d0ce291c@mail.gmail.com> <20090313111302.GA27894@code0.codespeak.net> <7f9d599f0903130704g74e62e68lad65d735c85eb9e6@mail.gmail.com> <693bc9ab0903130819y1a686216h8cdada85fca59d17@mail.gmail.com> <7f9d599f0903130848m282b371fk231e4b17bb9cff34@mail.gmail.com> <20090314175219.GA12036@code0.codespeak.net> Message-ID: Hi Armin, On Sat, Mar 14, 2009 at 18:52, Armin Rigo wrote: > Hi Paolo, > On Sat, Mar 14, 2009 at 05:28:32AM +0100, Paolo Giarrusso wrote: >> I think also PyPy does something similar for Python, but only through >> a hash table. I'd love to see how much inline caching would help on >> top of that, when I'll have time to start working on this. > No. ?PyPy does something very different, more similar to trace-based > JITs (goggle :-) Ah OK, sorry for that. > That does not involve hash tables (I don't know what > you are referring to here). I was thinking about the method cache in the interpreter; I'm not yet familiar with the approaches you used for JIT compilation (neither the one of PyPy 1.0 nor the one being prototyped right now), but I know the general purpose "get a JIT from an interpreter", and I thought that having a matching behaviour implied using the same lookup algorithm in JITted code. I understand that's no more true, with the ideas of trace-based JITs. Greetings, -- Paolo Giarrusso From vincent.legoll at gmail.com Tue Mar 24 11:24:05 2009 From: vincent.legoll at gmail.com (Vincent Legoll) Date: Tue, 24 Mar 2009 11:24:05 +0100 Subject: [pypy-dev] [OT] PyPy/python newb - need dbm other than dumbdbm anddbm Message-ID: <4727185d0903240324p475a6d6fv9c0ffbd4e700fc1f@mail.gmail.com> Hello, have you tried tailor (http://progetti.arstecnica.it/tailor) I've had good experience with it converting perforce to svn, a branch with about 220 K changesets on mid-end workstation HW (4 GB Ram / Dual-proc dual core Xeon 1.8 GHz) I can't remeber the total time it took, but it was in the 2-4 days range. tailor keeps state of where it is in the process of migration, so it can restart at the same place in case a problem needs to be manually solved. The process was more IO bound than CPU bound, I don't know how well it handles cvs though. -- Vincent Legoll From arigo at tunes.org Tue Mar 24 11:48:16 2009 From: arigo at tunes.org (Armin Rigo) Date: Tue, 24 Mar 2009 11:48:16 +0100 Subject: [pypy-dev] pypy for typed languages? In-Reply-To: References: <7f9d599f0903121757k37421a77kaa707214d0ce291c@mail.gmail.com> <20090313111302.GA27894@code0.codespeak.net> <7f9d599f0903130704g74e62e68lad65d735c85eb9e6@mail.gmail.com> <693bc9ab0903130819y1a686216h8cdada85fca59d17@mail.gmail.com> <7f9d599f0903130848m282b371fk231e4b17bb9cff34@mail.gmail.com> <20090314175219.GA12036@code0.codespeak.net> Message-ID: <20090324104816.GA1990@code0.codespeak.net> Hi Paolo, On Mon, Mar 23, 2009 at 11:46:29AM +0100, Paolo Giarrusso wrote: > I was thinking about the method cache in the interpreter; I'm not yet > familiar with the approaches you used for JIT compilation (neither the > one of PyPy 1.0 nor the one being prototyped right now), but I know > the general purpose "get a JIT from an interpreter", and I thought > that having a matching behaviour implied using the same lookup > algorithm in JITted code. I understand that's no more true, with the > ideas of trace-based JITs. Ah, sorry. I may have misread your original post. The fact is that both the PyPy 1.0 JIT and the current prototype JIT (and all other prototypes inbetween) are similar, in that they are derived automatically from the interpreter. If this interpreter happens to contain a hash table lookup, as the Python interpreter of PyPy optionally does, then the JIT will contain it too. So we are left with experimenting with which interpreter features are useful for the JIT too, and which ones give bad results. This experimentation is both easy and not quite doable right now because we are still developping the JIT generation basics. A bientot, Armin. From igor_trindade at yahoo.com.br Tue Mar 24 16:08:28 2009 From: igor_trindade at yahoo.com.br (Igor Trindade Oliveira) Date: Tue, 24 Mar 2009 11:08:28 -0400 Subject: [pypy-dev] GSoC proposal - C++ binding Message-ID: <200903241108.29227.igor_trindade@yahoo.com.br> Hi guys, i was talking with some pypy guys about work in this summer in GSoC with C++ bindings to Pypy and i wrote a proposal and i like that you review it. Igor PROPOSAL: Problem Pypy are becoming one of the most used Python interpreter because it is easy and flexible, but we still have a big problem, it does not have any binding to any Graphical User Interface(GUI) and we have the second problem, many GUIs are wrote in C++(like Qt or WxWidgets) and Pypy can not access C++ code because it does not have any C API, it just can access C code using ctypes. Solution Many solutions had been argued but the main solution is use Reflex[2],which is developed at CERN (which has an incredible amount of C++ libraries). It is not mainly intended for writing Python bindings for C++ libraries but instead provides reflection capabilities for C++. The idea is that for every C++ shared library, an additional shared library is produced, which allows together with Reflex to introspect properties of C++ classes, methods, etc. at runtime. These facilities are then used for writing a small generic CPython extension module, that allows CPython to use any C++ library for which this reflection information was generated [3]. This approach is a bit similar to the ctypes module, apart from the fact that ctypes does not use any reflection information, but the user has to specify the data structures that occur in the C code herself. This makes it sometimes rather burdensome to write cross-platform library bindings. For PyPy the approach seems rather fitting: We would need to implement only the generic extension module(a C language interface to Reflex) and could then use any number of C++ libraries. Goals The main goal of this project is to do a C reflex interface and make an easy API using ctypes to RPython communicate with that interface. More specific, the goals of the project are: 1.create an interface to reflex in C; 2.Use RPython ctypes to communicate with it; 3.create an API to RPython ctypes have an easy way to communicate with C reflex. 4.Begin a binding to Qt; Timeline The Google Summer of Code is subdivided in two parts. First part plans: 1.work in reflex to create a C interface for it. 2.Begin work on RPython ctypes; Second part plans: 1.create an API to RPython ctypes communicate with C reflex; 2.Begin Qt binding; Bio My name is Igor Trindade Oliveira, I am a Undergraduate Computer Engineering student in Brazil, in the fourth year. I work also in KDE project in KDE-PIM projects[1] and have experience with microcontrollers like 8051, MSP430, Mips r4000 and Arm 9. And with KDE I have very experience with C++ code. [1] http://techbase.kde.org/Projects/PIM/Akonadi/Testing [2] http://root.cern.ch/drupal/content/reflex [3] http://morepypy.blogspot.com/2008/10/sprint-discussions-c-library- bindings.html From ndbecker2 at gmail.com Tue Mar 24 18:18:14 2009 From: ndbecker2 at gmail.com (Neal Becker) Date: Tue, 24 Mar 2009 13:18:14 -0400 Subject: [pypy-dev] GSoC proposal - C++ binding References: <200903241108.29227.igor_trindade@yahoo.com.br> Message-ID: Igor Trindade Oliveira wrote: > Hi guys, > > i was talking with some pypy guys about work in this summer in GSoC with > C++ bindings to Pypy and i wrote a proposal and i like that you review it. > > Igor > > PROPOSAL: > > Problem > > Pypy are becoming one of the most used Python interpreter because it is > easy and flexible, but we still have a big problem, it does not have any > binding to any Graphical User Interface(GUI) and we have the second > problem, many GUIs are wrote in C++(like Qt or WxWidgets) and Pypy can not > access C++ code because it does not have any C API, it just can access C > code using ctypes. > > Solution > > Many solutions had been argued but the main solution is use > Reflex[2],which is developed at CERN (which has an incredible amount of > C++ libraries). It is not mainly intended for writing Python bindings for > C++ libraries but instead provides reflection capabilities for C++. The > idea is that for every C++ shared library, an additional shared library is > produced, which allows together with Reflex to introspect properties of > C++ classes, methods, etc. at runtime. These facilities are then used for > writing a small generic CPython extension module, that allows CPython to > use any C++ library for which this reflection information was generated > [3]. This approach is a bit similar to the ctypes module, apart from the > fact that ctypes does not use any reflection information, but the user has > to specify the data structures that occur in the C code herself. This > makes it sometimes rather burdensome to write cross-platform library > bindings. For PyPy the approach seems rather fitting: We would need to > implement only the generic extension module(a C language interface to > Reflex) and could then use any number of C++ libraries. > ... I wonder if py++ would help? From santagada at gmail.com Tue Mar 24 19:13:25 2009 From: santagada at gmail.com (Leonardo Santagada) Date: Tue, 24 Mar 2009 15:13:25 -0300 Subject: [pypy-dev] GSoC proposal - C++ binding In-Reply-To: References: <200903241108.29227.igor_trindade@yahoo.com.br> Message-ID: <68532EF2-A9AC-439D-AC3B-85EC03EC6545@gmail.com> On Mar 24, 2009, at 2:18 PM, Neal Becker wrote: > Igor Trindade Oliveira wrote: > >> Hi guys, >> >> i was talking with some pypy guys about work in this summer in GSoC >> with >> C++ bindings to Pypy and i wrote a proposal and i like that you >> review it. >> >> Igor >> >> PROPOSAL: >> >> Problem >> >> Pypy are becoming one of the most used Python interpreter because >> it is >> easy and flexible, but we still have a big problem, it does not >> have any >> binding to any Graphical User Interface(GUI) and we have the second >> problem, many GUIs are wrote in C++(like Qt or WxWidgets) and Pypy >> can not >> access C++ code because it does not have any C API, it just can >> access C >> code using ctypes. >> >> Solution >> >> Many solutions had been argued but the main solution is use >> Reflex[2],which is developed at CERN (which has an incredible >> amount of >> C++ libraries). It is not mainly intended for writing Python >> bindings for >> C++ libraries but instead provides reflection capabilities for C++. >> The >> idea is that for every C++ shared library, an additional shared >> library is >> produced, which allows together with Reflex to introspect >> properties of >> C++ classes, methods, etc. at runtime. These facilities are then >> used for >> writing a small generic CPython extension module, that allows >> CPython to >> use any C++ library for which this reflection information was >> generated >> [3]. This approach is a bit similar to the ctypes module, apart >> from the >> fact that ctypes does not use any reflection information, but the >> user has >> to specify the data structures that occur in the C code herself. This >> makes it sometimes rather burdensome to write cross-platform library >> bindings. For PyPy the approach seems rather fitting: We would need >> to >> implement only the generic extension module(a C language interface to >> Reflex) and could then use any number of C++ libraries. >> > ... > I wonder if py++ would help? When people were looking at what would be best to use they considered py++ (and boost.python). One of the reasons to not use it is that if I remember correctly py++ uses gcc-xml and it needs the source of the C++ library before generating some cpython module that needs to be compiled also. One of the good points of Reflex is that after the generation of the reflection library you don't need the source of the package neither a compiler installed, and reflex supports c++ also (which would be interesting if we want to convince OS people of including reflex modules of c++ libraries). Could Reflex be made to generate a reflex module of c packages also, so you don't have to declare them again in ctypes in python and all languages that support libffi? That would be cool. -- Leonardo Santagada santagada at gmail.com From igor_trindade at yahoo.com.br Tue Mar 24 20:27:27 2009 From: igor_trindade at yahoo.com.br (Igor Trindade Oliveira) Date: Tue, 24 Mar 2009 15:27:27 -0400 Subject: [pypy-dev] GSoC proposal - C++ binding In-Reply-To: <68532EF2-A9AC-439D-AC3B-85EC03EC6545@gmail.com> References: <200903241108.29227.igor_trindade@yahoo.com.br> <68532EF2-A9AC-439D-AC3B-85EC03EC6545@gmail.com> Message-ID: <200903241527.27794.igor_trindade@yahoo.com.br> On Tuesday 24 March 2009 14:13:25 Leonardo Santagada wrote: > On Mar 24, 2009, at 2:18 PM, Neal Becker wrote: > > Igor Trindade Oliveira wrote: > >> Hi guys, > >> > >> i was talking with some pypy guys about work in this summer in GSoC > >> with > >> C++ bindings to Pypy and i wrote a proposal and i like that you > >> review it. > >> > >> Igor > >> > >> PROPOSAL: > >> > >> Problem > >> > >> Pypy are becoming one of the most used Python interpreter because > >> it is > >> easy and flexible, but we still have a big problem, it does not > >> have any > >> binding to any Graphical User Interface(GUI) and we have the second > >> problem, many GUIs are wrote in C++(like Qt or WxWidgets) and Pypy > >> can not > >> access C++ code because it does not have any C API, it just can > >> access C > >> code using ctypes. > >> > >> Solution > >> > >> Many solutions had been argued but the main solution is use > >> Reflex[2],which is developed at CERN (which has an incredible > >> amount of > >> C++ libraries). It is not mainly intended for writing Python > >> bindings for > >> C++ libraries but instead provides reflection capabilities for C++. > >> The > >> idea is that for every C++ shared library, an additional shared > >> library is > >> produced, which allows together with Reflex to introspect > >> properties of > >> C++ classes, methods, etc. at runtime. These facilities are then > >> used for > >> writing a small generic CPython extension module, that allows > >> CPython to > >> use any C++ library for which this reflection information was > >> generated > >> [3]. This approach is a bit similar to the ctypes module, apart > >> from the > >> fact that ctypes does not use any reflection information, but the > >> user has > >> to specify the data structures that occur in the C code herself. This > >> makes it sometimes rather burdensome to write cross-platform library > >> bindings. For PyPy the approach seems rather fitting: We would need > >> to > >> implement only the generic extension module(a C language interface to > >> Reflex) and could then use any number of C++ libraries. > > > > ... > > I wonder if py++ would help? > > When people were looking at what would be best to use they considered > py++ (and boost.python). > > One of the reasons to not use it is that if I remember correctly py++ > uses gcc-xml and it needs the source of the C++ library before > generating some cpython module that needs to be compiled also. One of > the good points of Reflex is that after the generation of the > reflection library you don't need the source of the package neither a > compiler installed, and reflex supports c++ also (which would be > interesting if we want to convince OS people of including reflex > modules of c++ libraries). > > Could Reflex be made to generate a reflex module of c packages also, > so you don't have to declare them again in ctypes in python and all > languages that support libffi? That would be cool. right now i dont know but i can check it, but i believe that it just works with C++ packages. Igor Trindade Oliveira > > -- > Leonardo Santagada > santagada at gmail.com > > > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev From chris.nicholls at some.ox.ac.uk Wed Mar 25 14:14:26 2009 From: chris.nicholls at some.ox.ac.uk (Chris Nicholls) Date: Wed, 25 Mar 2009 13:14:26 +0000 Subject: [pypy-dev] Google Summer of code Message-ID: <198166e70903250614t297d1d65hca5a7c797ccded3b@mail.gmail.com> Hello, I'm a student interested in taking part in Google summer of code with pypy. I've been following pypy for a while now and this seems like a good opportunity to get involved, Is there a list of ideas anywhere? I'm not particularly sure on where to start... I found last years list herebut I expect its quite out of date. Chris -------------- next part -------------- An HTML attachment was scrubbed... URL: From tismer at stackless.com Thu Mar 26 01:27:33 2009 From: tismer at stackless.com (Christian Tismer) Date: Thu, 26 Mar 2009 01:27:33 +0100 Subject: [pypy-dev] http://code.google.com/p/unladen-swallow/wiki/ProjectPlan Message-ID: <49CACBF5.4020904@stackless.com> Hi friends, please have a look at this. http://code.google.com/p/unladen-swallow/wiki/ProjectPlan is this YAIPAP ? Yet Another Ignorant Python Acceleration Project I see them mentioning H?lzle and Self, and I see a reference to Psyco where they want to steal from, but PyPy does not exist. IMHO, this is totally GAGA - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From cumber at netspace.net.au Thu Mar 26 01:41:27 2009 From: cumber at netspace.net.au (Ben Mellor) Date: Thu, 26 Mar 2009 11:41:27 +1100 Subject: [pypy-dev] RPython rendering of % operator Message-ID: <20090326114127.147ff5ac@intyalie> Does anyone know why lambda x, y: x % y is being converted to a flow graph as: v2 = int_mod(x_0, y_0) v1 = int_add(v2, [int_mul([int_and([cast_bool_to_int([int_le([int_xor(x_0, y_0)], (0))])], [cast_bool_to_int([int_ne(v2, (0))])])], y_0)]) (v1 is returned) What's wrong with just v1 = int_mod(x_0, y_0)? -- Ben From santagada at gmail.com Thu Mar 26 02:42:32 2009 From: santagada at gmail.com (Leonardo Santagada) Date: Wed, 25 Mar 2009 22:42:32 -0300 Subject: [pypy-dev] http://code.google.com/p/unladen-swallow/wiki/ProjectPlan In-Reply-To: <49CACBF5.4020904@stackless.com> References: <49CACBF5.4020904@stackless.com> Message-ID: <9B54C41A-EF31-4BB1-927F-6D02B88A7B41@gmail.com> On Mar 25, 2009, at 9:27 PM, Christian Tismer wrote: > Hi friends, > > please have a look at this. > http://code.google.com/p/unladen-swallow/wiki/ProjectPlan > > is this YAIPAP ? > Yet Another Ignorant Python Acceleration Project > > I see them mentioning H?lzle and Self, and I see a reference to > Psyco where they want to steal from, but PyPy does not exist. > > IMHO, this is totally GAGA - chris I was reading it earlier, the simpler ideas, like making pickle faster and most of q1 deliverables seems nice, and could really help python right now, but those seems more like the things Need For Speed sprint was doing. Not the LLVM-JIT, new GC, eliminate the GIL seems unrealistic, even the pace of the project seems to be counting on tons of new developers joining the project after the Q1 milestone. -- Leonardo Santagada santagada at gmail.com From amauryfa at gmail.com Thu Mar 26 09:36:00 2009 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Thu, 26 Mar 2009 09:36:00 +0100 Subject: [pypy-dev] RPython rendering of % operator In-Reply-To: <20090326114127.147ff5ac@intyalie> References: <20090326114127.147ff5ac@intyalie> Message-ID: Hello, On Thu, Mar 26, 2009 at 01:41, Ben Mellor wrote: > Does anyone know why > > lambda x, y: x % y > > is being converted to a flow graph as: > > v2 = int_mod(x_0, y_0) > v1 = int_add(v2, [int_mul([int_and([cast_bool_to_int([int_le([int_xor(x_0, > y_0)], (0))])], [cast_bool_to_int([int_ne(v2, (0))])])], y_0)]) > > (v1 is returned) > > > What's wrong with just v1 = int_mod(x_0, y_0)? You are looking at the graph of the function: pypy.rpython.lltypesystem.opimpl.op_int_mod The difference is certainly because C and python handle modulus differently when one argument is negative: (-5 % 3) returns -2 in C, but 1 with python. Now, I don't know whether RPython should absolutely implement python semantics here. RPython does not check for overflow for example. -- Amaury Forgeot d'Arc From cfbolz at gmx.de Thu Mar 26 09:51:39 2009 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Thu, 26 Mar 2009 09:51:39 +0100 Subject: [pypy-dev] RPython rendering of % operator In-Reply-To: References: <20090326114127.147ff5ac@intyalie> Message-ID: <49CB421B.4050600@gmx.de> Amaury Forgeot d'Arc wrote: > Hello, > > On Thu, Mar 26, 2009 at 01:41, Ben Mellor wrote: >> Does anyone know why >> >> lambda x, y: x % y >> >> is being converted to a flow graph as: >> >> v2 = int_mod(x_0, y_0) >> v1 = int_add(v2, [int_mul([int_and([cast_bool_to_int([int_le([int_xor(x_0, >> y_0)], (0))])], [cast_bool_to_int([int_ne(v2, (0))])])], y_0)]) >> >> (v1 is returned) >> >> >> What's wrong with just v1 = int_mod(x_0, y_0)? > > You are looking at the graph of the function: > pypy.rpython.lltypesystem.opimpl.op_int_mod > > The difference is certainly because C and python handle modulus > differently when one argument is negative: > (-5 % 3) returns -2 in C, but 1 with python. > > Now, I don't know whether RPython should absolutely implement python > semantics here. > RPython does not check for overflow for example. The semantics of % in RPython _are_ the same as in Python. It's just that the semantics of int_mod aren't that of Python, therefore % must be mapped to something more complex. The reason why int_mod has the C semantics is that it makes the life of all backends easier. Cheers, Carl Friedrich From fijall at gmail.com Thu Mar 26 13:05:35 2009 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 26 Mar 2009 13:05:35 +0100 Subject: [pypy-dev] Google Summer of code In-Reply-To: <198166e70903250614t297d1d65hca5a7c797ccded3b@mail.gmail.com> References: <198166e70903250614t297d1d65hca5a7c797ccded3b@mail.gmail.com> Message-ID: <693bc9ab0903260505p3d081950m33b21b5878df8bb9@mail.gmail.com> Hi Chris. There is no up-to-date list (we were too busy with other stuff and conferences etc.), but you can surely come to #pypy on IRC and ask a bit. It depends what are you interested in of course. Cheers, fijal 2009/3/25 Chris Nicholls : > Hello, > > I'm a student interested in taking part in Google summer of code with pypy. > I've been following pypy for a while now and this seems like a good > opportunity to get involved, Is there a list of ideas anywhere? I'm not > particularly sure on where to start... I found last years list here but I > expect its quite out of date. > > Chris > > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From p.giarrusso at gmail.com Mon Mar 30 01:28:31 2009 From: p.giarrusso at gmail.com (Paolo Giarrusso) Date: Mon, 30 Mar 2009 00:28:31 +0100 Subject: [pypy-dev] http://code.google.com/p/unladen-swallow/wiki/ProjectPlan In-Reply-To: <9B54C41A-EF31-4BB1-927F-6D02B88A7B41@gmail.com> References: <49CACBF5.4020904@stackless.com> <9B54C41A-EF31-4BB1-927F-6D02B88A7B41@gmail.com> Message-ID: On Thu, Mar 26, 2009 at 02:42, Leonardo Santagada wrote: > > On Mar 25, 2009, at 9:27 PM, Christian Tismer wrote: > > > Hi friends, > > > > please have a look at this. > > http://code.google.com/p/unladen-swallow/wiki/ProjectPlan > > > > is this YAIPAP ? > > Yet Another Ignorant Python Acceleration Project > > > > I see them mentioning H?lzle and Self, and I see a reference to > > Psyco where they want to steal from, but PyPy does not exist. > > > > IMHO, this is totally GAGA ?- chris First, I suggest you have a look at the FAQ - when I did, I discovered the developers are full-time Google engineers. Having said that, the Google V8 group does not seem to be involved, and that's quite stupid. However, I studied with Lars Bak and the base ideas he had are the same. Missing knowledge of PyPy is also stupid obviously, but I wonder why nobody proposed "hey, let's tell them"; fighting another project. They'd sure benefit, for instance, from CALL_METHOD and CALL_LIKELY_BUILTIN ideas (they mention among their purposes fixing the performance problems addressed by CALL_LIKELY_BUILTIN, that's why I mention these). Also, they're already started with releasing. Have a look at their benchmarks: http://code.google.com/p/unladen-swallow/wiki/Releases Did you look at that because declaring it GAGA? > I was reading it earlier, the simpler ideas, like making pickle faster > and most of q1 deliverables seems nice, and could really help python > right now, but those seems more like the things Need For Speed sprint > was doing. I don't know about Need For Sprint, but my first guess is that this maybe happens because, as pointed out various ways, CPython has lots of "obvious" deficiencies. > Not the LLVM-JIT, new GC, eliminate the GIL seems > unrealistic, even the pace of the project seems to be counting on tons > of new developers joining the project after the Q1 milestone. Well, the milestones seem crazy but they did achieve something notable in their Q1 deliverable, but most of the ideas seem correct. "Eliminate the GIL" is not hard by itself, the problem (IMHO, no hard numbers for that) is that it's impossible with refcounting (because of the cost of atomic refcount manipulation). The author of the old "free threading" patch mentioned only another problem, that is locking around name lookups in mutable name dictionaries (for looking up object members, globals, etc.), which can also be approached (and I think refcount manipulation is a bigger issue). Regards -- Paolo Giarrusso From tismer at stackless.com Mon Mar 30 13:48:39 2009 From: tismer at stackless.com (Christian Tismer) Date: Mon, 30 Mar 2009 13:48:39 +0200 Subject: [pypy-dev] http://code.google.com/p/unladen-swallow/wiki/ProjectPlan In-Reply-To: References: <49CACBF5.4020904@stackless.com> <9B54C41A-EF31-4BB1-927F-6D02B88A7B41@gmail.com> Message-ID: <49D0B197.5070509@stackless.com> On 3/30/09 1:28 AM, Paolo Giarrusso wrote: > On Thu, Mar 26, 2009 at 02:42, Leonardo Santagada wrote: >> On Mar 25, 2009, at 9:27 PM, Christian Tismer wrote: >> >>> Hi friends, >>> >>> please have a look at this. >>> http://code.google.com/p/unladen-swallow/wiki/ProjectPlan >>> >>> is this YAIPAP ? >>> Yet Another Ignorant Python Acceleration Project >>> >>> I see them mentioning H?lzle and Self, and I see a reference to >>> Psyco where they want to steal from, but PyPy does not exist. >>> >>> IMHO, this is totally GAGA - chris > > First, I suggest you have a look at the FAQ - when I did, I discovered > the developers are full-time Google engineers. Having said that, the > Google V8 group does not seem to be involved, and that's quite stupid. > However, I studied with Lars Bak and the base ideas he had are the > same. Interesting. Yes, I read the FAQ, but I don't know the people. > Missing knowledge of PyPy is also stupid obviously, but I wonder why > nobody proposed "hey, let's tell them"; fighting another project. > They'd sure benefit, for instance, from CALL_METHOD and > CALL_LIKELY_BUILTIN ideas (they mention among their purposes fixing > the performance problems addressed by CALL_LIKELY_BUILTIN, that's why > I mention these). Well, we could have contacted them. But (I'm speaking for myself) if they have decided to totally ignore a project like PyPy, then I think it is for a reason. Not knowing about PyPy means to close more than two eyes. Therefore I see no point in approaching them. > Also, they're already started with releasing. Have a look at their benchmarks: > http://code.google.com/p/unladen-swallow/wiki/Releases > > Did you look at that because declaring it GAGA? My first perception of the project was a bit distorted. I saw this as an attempt to replace PyPy with something better, and it seemed GAGA to me to do that ignoring prior work. Now that I realize that the project has much smaller goals, it becomes more realistic. The Q1 goals are relatively doable without doubt. The current achievements speedwise remind me of the anxient Python2C project. It showed the typical acceleration by a factor of around 2, which is what you can expect when eliminating the interpreter loop. >> I was reading it earlier, the simpler ideas, like making pickle faster >> and most of q1 deliverables seems nice, and could really help python >> right now, but those seems more like the things Need For Speed sprint >> was doing. Yes, Need for Speed did small, doable things. >> Not the LLVM-JIT, new GC, eliminate the GIL seems >> unrealistic, even the pace of the project seems to be counting on tons >> of new developers joining the project after the Q1 milestone. > > Well, the milestones seem crazy but they did achieve something notable > in their Q1 deliverable, but most of the ideas seem correct. Q1 is fine, but it does by no means give any proof that the next milestones can be achieved. > "Eliminate the GIL" is not hard by itself, the problem (IMHO, no hard > numbers for that) is that it's impossible with refcounting (because of > the cost of atomic refcount manipulation). The author of the old "free > threading" patch mentioned only another problem, that is locking > around name lookups in mutable name dictionaries (for looking up > object members, globals, etc.), which can also be approached (and I > think refcount manipulation is a bigger issue). As I remember that patch, the overhead was around 40%, mostly because of reference counting. I guess nobody actually goes this path, because it is such a waste, compared to multiple processes, and doing it "right" (where I'm referring to some Java achievements I heard of) is pretty much of a total re-write of Python. I'm pretty much wondering if the latter makes sense, given the existence of PyPy. cheers - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From p.giarrusso at gmail.com Mon Mar 30 20:38:31 2009 From: p.giarrusso at gmail.com (Paolo Giarrusso) Date: Mon, 30 Mar 2009 20:38:31 +0200 Subject: [pypy-dev] http://code.google.com/p/unladen-swallow/wiki/ProjectPlan In-Reply-To: <49D0B197.5070509@stackless.com> References: <49CACBF5.4020904@stackless.com> <9B54C41A-EF31-4BB1-927F-6D02B88A7B41@gmail.com> <49D0B197.5070509@stackless.com> Message-ID: On Mon, Mar 30, 2009 at 13:48, Christian Tismer wrote: > On 3/30/09 1:28 AM, Paolo Giarrusso wrote: >> >> On Thu, Mar 26, 2009 at 02:42, Leonardo Santagada >> ?wrote: >>> >>> On Mar 25, 2009, at 9:27 PM, Christian Tismer wrote: >>> >>>> Hi friends, >>>> >>>> please have a look at this. >>>> http://code.google.com/p/unladen-swallow/wiki/ProjectPlan >>>> >>>> is this YAIPAP ? >>>> Yet Another Ignorant Python Acceleration Project >>>> >>>> I see them mentioning H?lzle and Self, and I see a reference to >>>> Psyco where they want to steal from, but PyPy does not exist. >>>> >>>> IMHO, this is totally GAGA ?- chris >> >> First, I suggest you have a look at the FAQ - when I did, I discovered >> the developers are full-time Google engineers. Having said that, the >> Google V8 group does not seem to be involved, and that's quite stupid. >> However, I studied with Lars Bak and the base ideas he had are the >> same. > > Interesting. Yes, I read the FAQ, but I don't know the people. > >> Missing knowledge of PyPy is also stupid obviously, but I wonder why >> nobody proposed "hey, let's tell them"; fighting another project. >> They'd sure benefit, for instance, from CALL_METHOD and >> CALL_LIKELY_BUILTIN ideas (they mention among their purposes fixing >> the performance problems addressed by CALL_LIKELY_BUILTIN, that's why >> I mention these). > > Well, we could have contacted them. > But (I'm speaking for myself) if they have decided to totally > ignore a project like PyPy, then I think it is for a reason. > Not knowing about PyPy means to close more than two eyes. > Therefore I see no point in approaching them. Well, there is a mention of this project on the PyPy blog (just noticed), and they talk in term of "friendly competition". Since they seem to know about PyPy, I wonder why it's not listed on the website. But it's anyway by far incomplete (lots of interesting questions weren't answered by those pages). >> Also, they're already started with releasing. Have a look at their >> benchmarks: >> http://code.google.com/p/unladen-swallow/wiki/Releases >> >> Did you look at that because declaring it GAGA? > > My first perception of the project was a bit distorted. > I saw this as an attempt to replace PyPy with something better, > and it seemed GAGA to me to do that ignoring prior work. > > Now that I realize that the project has much smaller > goals, it becomes more realistic. > > The Q1 goals are relatively doable without doubt. The current > achievements speedwise remind me of the anxient Python2C project. > It showed the typical acceleration by a factor of around 2, which > is what you can expect when eliminating the interpreter loop. > >>> I was reading it earlier, the simpler ideas, like making pickle faster >>> and most of q1 deliverables seems nice, and could really help python >>> right now, but those seems more like the things Need For Speed sprint >>> was doing. > > Yes, Need for Speed did small, doable things. > >>> Not the LLVM-JIT, new GC, eliminate the GIL seems >>> unrealistic, even the pace of the project seems to be counting on tons >>> of new developers joining the project after the Q1 milestone. >> >> Well, the milestones seem crazy but they did achieve something notable >> in their Q1 deliverable, but most of the ideas seem correct. > > Q1 is fine, but it does by no means give any proof that the next > milestones can be achieved. > >> "Eliminate the GIL" is not hard by itself, the problem (IMHO, no hard >> numbers for that) is that it's impossible with refcounting (because of >> the cost of atomic refcount manipulation). The author of the old "free >> threading" patch mentioned only another problem, that is locking >> around name lookups in mutable name dictionaries (for looking up >> object members, globals, etc.), which can also be approached (and I >> think refcount manipulation is a bigger issue). > > As I remember that patch, the overhead was around 40%, mostly because > of reference counting. You're right, but I couldn't find someone stating that it was due to reference counting, on the MLs or on FAQs. The mention of dictionaries (which are probably also a problem) was from the author of the patch, Greg Stein: http://mail.python.org/pipermail/python-dev/2001-August/017099.html > I guess nobody actually goes this path, > because it is such a waste, compared to multiple processes, and doing > it "right" (where I'm referring to some Java achievements I heard of) > is pretty much of a total re-write of Python. Well, converting each and every C module from refcounting to garbage collection is _not_ a total rewrite of Python. I can understand why you claim it has the same costs. But eXtreme Programming on one side, and the daily activity of the Linux kernel on another side, show that given enough manpower such things can be done. I mean, why CPython doesn't allow having multiple interpreters in the same process? Because it would take time to move all statics into the thread state. Well, much bigger changes do take place in the Linux kernel much more than once per release. Having had experience working there, I see either a severe lack of manpower, or a lack of trust into what developers can do. And even if I say "enough manpower", it still doesn't amount at all to a complete rewrite. For a human programmer, the conversion is mostly mechanical: remove (or make conditional) refcount stuff, then in all functions, register all pointers with the GC (or, in C++, turn them into handles). If you don't do that over night, you'll get bugs only in very particular situations. It's obviously boring though, but programming is not always fun. > I'm pretty much wondering if the latter makes sense, given the > existence of PyPy. Because PyPy still needs lots of time, because the PyPy idea _is_ research, and because if they meet their deadlines (I'm not sure if it's conceivably possible) Google will save computing power much earlier. Especially because their C API will be more compatible with the current one. Well, they actually claim full compatibility, but this point _is_ IMHO crazy, and they'll notice when they'll have to remove refcounting. About the "PyPy is research" statement, just a quote: http://morepypy.blogspot.com/2008/10/sprint-discussions-jit-generator.html "Partial evaluation (the basis for our JIT generator) is a 30 years old technique that was always just promising and never really successful, so the fact that we think we can solve its problems in a few years is very much hubris anyway :-). On the positive side, we think that we now know these problems much better than ever before and that we have a plan that has a chance to succeed." I didn't read the post about tracing JITs yet, so I don't know if that still holds true (I mean, if this is still about partial evaluation). Anyway, at most it's research approaching completion and production status, but IMHO nobody imagines getting production stability by the end of 2009 (that's IMHO though, I might very well be wrong, and I hope so). Regards -- Paolo Giarrusso From lac at openend.se Tue Mar 31 18:37:57 2009 From: lac at openend.se (Laura Creighton) Date: Tue, 31 Mar 2009 18:37:57 +0200 Subject: [pypy-dev] the next time we get frustrated with buildbot Message-ID: <200903311637.n2VGbvFX009404@theraft.openend.se> Something Open Source that somebody says is better.... I'd never heard of it before ... Laura - ------- Forwarded Message Return-Path: testing-in-python-bounces at lists.idyll.org Delivery-Date: Tue Mar 31 18:16:27 2009 Return-Path: From: Kumar McMillan To: titus at idyll.org Cc:tip at lists.idyll.org Subject: Re: [TIP] Everybody wants a pony! X-BeenThere: testing-in-python at lists.idyll.org On Tue, Mar 31, 2009 at 9:32 AM, C. Titus Brown wrote: > Hi all, > > one thing that became clear at PyCon this year is that everyone wants a > pony. ?Preferably a pink pony. > > (You had to be there.) > > Anyway, along the lines of a pony, I am proposing to build or extend a > replacement for buildbot that will serve those of you who need a simpler > and more easily-installable continuous interation framework. ?I have a > few basic ideas, but I'd love to hear what everyone else has to say. I hate to be "that guy;" I love Python, really I do, but what I love more is when a great open source tool with great documentation and a community of maintainers solves a problem better than any other tool. It makes the decision process easy. It's called Hudson: https://hudson.dev.java.net/ Yes, it is written in Java but Java is ubiquitous and Hudson is light years beyond Buildbot. Anyone (i.e. non-programmers) can configure builds and workflows very easily. If not obvious from the documentation, you can build and test anything, it doesn't have to be Java. At Leapfrog we have converted all of our Python and Ruby buildbots to Hudson (oh did I say that out loud?) and have never been happier or more efficient in how we do continuous integration. You don't need anything special in your tests to use Hudson but you get some extra details (that buildbot never provided) if you use xunit style test output. There are several nose plugins for this like Nosexunit -- the androgynous plugin. Example: nosetests --with-nosexunit Titus, if you *must* spend your precious spare time (your hourly rate is what now? priceless?) on this then at the very least make a Python clone of Hudson :) But seriously, give it a fair try and I think you'll like it. It has a plugin system too. I assume one could write plugins using Jython even. > > Roughly speaking, I'd like to have pony-build do the following, relative > to buildbot: > > ?- "push" results from the client back to the server over an RPC > ? connection, rather than have the server control the client. > ? (That is, base it on a polling model rather than a remote control > ? model.) > > ?- Be trivially installable on clients (a single "easy_install" that > ? works, even for Steve Holden). > > ?- A separable and flexible reporting server to support RPC queries and > ? programmable workflows. > > ?- A separable and flexible scheduling/config server to support build > ? requests and request builds on specific servers. > > ?- "Anonymous" push of results, so that Joe Blow users can build, > ? compile, test, and send the results back to the server for your > ? examination, without breaking a sweat. > > ?- Push of metadata to the reporting server for e.g. communication of > ? coverage data. > > ?- distutils/setuptools and configure/make support. > > A lot of these ideas come from comparing DART with buildbot, and having > good (and bad) experiences with both. > > We do have a prototype up and running and I'll try to get a screencast > up within a week or two. > > Your thoughts welcome! > > cheers, > --titus > -- > C. Titus Brown, ctb at msu.edu > > _______________________________________________ > testing-in-python mailing list > testing-in-python at lists.idyll.org > http://lists.idyll.org/listinfo/testing-in-python > _______________________________________________ testing-in-python mailing list testing-in-python at lists.idyll.org http://lists.idyll.org/listinfo/testing-in-python ------- End of Forwarded Message From jbaker at zyasoft.com Tue Mar 31 20:36:48 2009 From: jbaker at zyasoft.com (Jim Baker) Date: Tue, 31 Mar 2009 12:36:48 -0600 Subject: [pypy-dev] the next time we get frustrated with buildbot In-Reply-To: <200903311637.n2VGbvFX009404@theraft.openend.se> References: <200903311637.n2VGbvFX009404@theraft.openend.se> Message-ID: Hudson works very well for Jython, so I'd certainly recommend it. Our version of regrtest has some minor modifications to support JUnit XML, see there, test.test_support, and test.junit_xml for this code. In particular, by breaking out our test cases we can more readily observe a change in time of a specific failing case, so it's quite useful. You can see it in action here: http://bob.underboss.org:8080/job/jython/ On Tue, Mar 31, 2009 at 10:37 AM, Laura Creighton wrote: > > Something Open Source that somebody says is better.... > I'd never heard of it before ... > > Laura > > - ------- Forwarded Message > > Return-Path: testing-in-python-bounces at lists.idyll.org > Delivery-Date: Tue Mar 31 18:16:27 2009 > Return-Path: > From: Kumar McMillan > To: titus at idyll.org > Cc:tip at lists.idyll.org > Subject: Re: [TIP] Everybody wants a pony! > X-BeenThere: testing-in-python at lists.idyll.org > > On Tue, Mar 31, 2009 at 9:32 AM, C. Titus Brown wrote: > > Hi all, > > > > one thing that became clear at PyCon this year is that everyone wants a > > pony. Preferably a pink pony. > > > > (You had to be there.) > > > > Anyway, along the lines of a pony, I am proposing to build or extend a > > replacement for buildbot that will serve those of you who need a simpler > > and more easily-installable continuous interation framework. I have a > > few basic ideas, but I'd love to hear what everyone else has to say. > > I hate to be "that guy;" I love Python, really I do, but what I love > more is when a great open source tool with great documentation and a > community of maintainers solves a problem better than any other tool. > It makes the decision process easy. > > It's called Hudson: > https://hudson.dev.java.net/ > > Yes, it is written in Java but Java is ubiquitous and Hudson is light > years beyond Buildbot. Anyone (i.e. non-programmers) can configure > builds and workflows very easily. > > If not obvious from the documentation, you can build and test > anything, it doesn't have to be Java. At Leapfrog we have converted > all of our Python and Ruby buildbots to Hudson (oh did I say that out > loud?) and have never been happier or more efficient in how we do > continuous integration. You don't need anything special in your tests > to use Hudson but you get some extra details (that buildbot never > provided) if you use xunit style test output. There are several nose > plugins for this like Nosexunit -- the androgynous plugin. Example: > nosetests --with-nosexunit > > Titus, if you *must* spend your precious spare time (your hourly rate > is what now? priceless?) on this then at the very least make a Python > clone of Hudson :) But seriously, give it a fair try and I think > you'll like it. > > It has a plugin system too. I assume one could write plugins using Jython > even. > > > > > > Roughly speaking, I'd like to have pony-build do the following, relative > > to buildbot: > > > > - "push" results from the client back to the server over an RPC > > connection, rather than have the server control the client. > > (That is, base it on a polling model rather than a remote control > > model.) > > > > - Be trivially installable on clients (a single "easy_install" that > > works, even for Steve Holden). > > > > - A separable and flexible reporting server to support RPC queries and > > programmable workflows. > > > > - A separable and flexible scheduling/config server to support build > > requests and request builds on specific servers. > > > > - "Anonymous" push of results, so that Joe Blow users can build, > > compile, test, and send the results back to the server for your > > examination, without breaking a sweat. > > > > - Push of metadata to the reporting server for e.g. communication of > > coverage data. > > > > - distutils/setuptools and configure/make support. > > > > A lot of these ideas come from comparing DART with buildbot, and having > > good (and bad) experiences with both. > > > > We do have a prototype up and running and I'll try to get a screencast > > up within a week or two. > > > > Your thoughts welcome! > > > > cheers, > > --titus > > -- > > C. Titus Brown, ctb at msu.edu > > > > _______________________________________________ > > testing-in-python mailing list > > testing-in-python at lists.idyll.org > > http://lists.idyll.org/listinfo/testing-in-python > > > > _______________________________________________ > testing-in-python mailing list > testing-in-python at lists.idyll.org > http://lists.idyll.org/listinfo/testing-in-python > > ------- End of Forwarded Message > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > -- Jim Baker jbaker at zyasoft.com -------------- next part -------------- An HTML attachment was scrubbed... URL: