From cfbolz at gmx.de Tue Aug 2 22:25:55 2005 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Tue, 02 Aug 2005 22:25:55 +0200 Subject: [pypy-dev] day 7: winding down Message-ID: <42EFD6D3.50400@gmx.de> Hi PyPy-Dev! Some quit notes about Sunday, the 7th and last day of the sprint. In general it was sort of a lazy day where we reveled in the success of the previous night ;-). The day was started with an extensive breakfast in the sun. After that Alex (from the Trillke-Gut) brought bows and arrows outside and gave us a short introduction to archery. We spent some time trying it (and lost four arrows in the proccess). - Holger and Armin worked on refactoring and improving the compiling infrastructure which acquired quite a lot of cruft over time. The goal was to be able to use the compiled PyPy as a stand-alone executable and not just a extension module to CPython (note that this extension module is independent from CPython, we use CPython only to call the entry point). - Samuele and Carl Friedrich managed to write a test for the wrong exception behaviour. Additionally they fixed a wrong graph transformation where we ended up with an empty block with exception handling. This gets now optimized away (the non-exceptional link is taken). - Richard stole tests from genc and made most of them work on genllvm. He added support for unicode and implemented missing mathematical operations. Additionally he implemented arrays of void (which are basically ints because they still have a length field) which occur in strange situation (like a list of Nones). - CF with the occasional advice from Samuele worked on implementing lltypes on top of the memory simulator he wrote the day before. In the afternoon we went to an art exhibition of an artist from Turkey who had stayed on Trillke-Gut for a few weeks and who Holger knew. We stayed there quite some time since there was a barbecue and we decided to have dinner there. Afterwards we went to meet Christian in the Galileo cafe and discussed what to do until the Heidelberg sprint: the general idea is to start cleaning some dark corners that had turned strange. The plan is to have a clean base for the sprint in Heidelberg, for the planned release there and especially for phase 2. In the evening we cleaned up the room we sprinted in. Christian tried to integrate the interplevel marshal implementation he'd written during the last days. This turned up a problem with the annotator related to bound methods. Samuele came up with a minimal test but decided that it would take too long to fix. Cheers, Carl Friedrich From tismer at stackless.com Wed Aug 3 05:45:25 2005 From: tismer at stackless.com (Christian Tismer) Date: Wed, 03 Aug 2005 05:45:25 +0200 Subject: [pypy-dev] PyPy broken! What's up? Message-ID: <42F03DD5.2060807@stackless.com> Hi Friends, in Hildesheim, I struggled for many hours to get my new marshal module to compile and integrate. Finally, after removing all problematic things where the annotator or the rtyper complained about, I figured out that the current PyPy SVN version did not compile at all. Today, I tried again, after I brought marshal some quality levels further. Still, PyPy does not translate, neither on Windows ( with my patches of course) nor on Debian Linux. The property to be translatable to a C binary is much more valuable than it is treated. I would really like to see some notation of that, like tagging a PyPy version as translatable. I think we are lacking something, here. Due to this fact, I will now carelessly check marshal in, without disabling it. This has cost Samuele and me way too many hours, although I figured out that PyPy was not translatable at all, at that time. (Yes, we had many bugs and removed them). Please, I want a principal decision about this. Either breaking PyPy must be forbidden, or check-ins must be tagged whether they do compile or not. Random results about this are not acceptable for me, because this takes very much time. - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From hpk at trillke.net Wed Aug 3 10:07:56 2005 From: hpk at trillke.net (holger krekel) Date: Wed, 3 Aug 2005 10:07:56 +0200 Subject: [pypy-dev] PyPy broken! What's up? In-Reply-To: <42F03DD5.2060807@stackless.com> References: <42F03DD5.2060807@stackless.com> Message-ID: <20050803080755.GN11996@solar.trillke.net> Hi Chris, On Wed, Aug 03, 2005 at 05:45 +0200, Christian Tismer wrote: > in Hildesheim, I struggled for many hours to get my new marshal > module to compile and integrate. Finally, after removing all > problematic things where the annotator or the rtyper complained > about, I figured out that the current PyPy SVN version did > not compile at all. > > Today, I tried again, after I brought marshal some quality levels > further. Still, PyPy does not translate, neither on Windows ( with > my patches of course) nor on Debian Linux. I tried to translate yesterday afternoon (on linux) and it worked. From looking at the commits today i guess that the integration of the _sre module triggered some problems (which i haven't analysed yet). If i leave out the _sre module but leave the marshal included then it fails as well. If i leave out the _sre and marshal module and inhibit importing of the marshal multimethod in std/model.py then PyPy translates. So i committed changes along these lines to make the default trunk translate again: Revision 15538 should be translateable (on linux). > The property to be translatable to a C binary is much more > valuable than it is treated. I would really like to see some > notation of that, like tagging a PyPy version as translatable. > I think we are lacking something, here. Well, i guess you know that we have the "working from snapshot" scheme which we used up until the hildesheim2 sprint? On the sprint we discussed whether to continue to work from such a snapshot or whether to stay with the trunk. We decided that we want to try working from the trunk, especially since annotation is/was not producing any SomeObjects and is relatively fast. Keeping a snapshot is extra work but maybe we have to go back using it. > Due to this fact, I will now carelessly check marshal in, > without disabling it. This has cost Samuele and me way too many > hours, although I figured out that PyPy was not translatable > at all, at that time. (Yes, we had many bugs and removed them). Sorry, I am not completely sure what you refer to. That PyPy was not translateable in an unmodified-trunk version on sunday? > Please, I want a principal decision about this. Either breaking > PyPy must be forbidden, or check-ins must be tagged whether they > do compile or not. Random results about this are not acceptable > for me, because this takes very much time. We all want robust translation. But it's irritating that you seem to take the position of a frustrated PyPy customer although you are part of the very developing group that tries to bring PyPy to an alpha stage. holger From tismer at stackless.com Wed Aug 3 14:47:05 2005 From: tismer at stackless.com (Christian Tismer) Date: Wed, 03 Aug 2005 14:47:05 +0200 Subject: [pypy-dev] PyPy broken! What's up? In-Reply-To: <20050803080755.GN11996@solar.trillke.net> References: <42F03DD5.2060807@stackless.com> <20050803080755.GN11996@solar.trillke.net> Message-ID: <42F0BCC9.5080003@stackless.com> Hi Holger > We all want robust translation. But it's irritating that > you seem to take the position of a frustrated PyPy customer > although you are part of the very developing group that tries > to bring PyPy to an alpha stage. Of course I was frustrated. I never got PyPy to compile on Windows, yet, and I need to get my code in, no time to continue right now. Well, the hope was to not to have to disable marshal, for that I wanted to use a translatable version. Found out way too late that the version I got on Sunday evening was already broken. I generally feel liable for my own bugs and want to fix them. Right now, I can't even tell what my bugs are. ciao - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From hpk at trillke.net Wed Aug 3 20:11:47 2005 From: hpk at trillke.net (holger krekel) Date: Wed, 3 Aug 2005 20:11:47 +0200 Subject: [pypy-dev] pypy-sync meeting 4th August 1pm Message-ID: <20050803181147.GH11996@solar.trillke.net> Hi pypy-dev, we've had three so called 'pypy-sync' meetings already on the last thursdays. The minutes are to be found under http://codespeak.net/svn/pypy/extradoc/minute However, they have so far been only announced on internal channels and only some of the minutes went to pypy-dev. If i remember correctly. This post is to announce the pypy-sync meeting for tomorrow (thursday 4th August) 1pm. The meeting is open for all active developers and closed to lurkers (which can read the minutes afterwards). The duration is capped at 30 minutes. We did manage to conclude always in time so far. The agenda always has a fixed part and a variable part with "topics of the week". I guess we somewhat expect from EU funded developers to attend the meeting (or provide info off-line) but for all others it's a complete matter of their free choice. So what is the purpose of the pypy-sync meeting? It's an XP-style meeting that serves to synchronize development work and let everybody know who is working on what. It also serves as a decision board of the PyPy developers. If discussions last too long and decisions cannot be reached they are delegated to a sub-group or get postponed. Here is the agenda (and if you come, please try to come in time, there is no waiting and attendants are expected to not walk away during the 30 minutes). So below is the the agenda for this week's meeting. Developers wanting a topic on the list need to contact me until wednesday afternoon. This procedure is repeated every week. cheers, holger ============================================= pypy-sync developer meeting 4th August 2005 ============================================= Regular Topics ==================== - roll call. - activity reports (3 prepared lines of info). All Attendees submit activity reports in the form of three lines prepared before the meeting: LAST: work area1, work area2, ... NEXT: work area7, issue... BLOCKERS: the unresolved XXX blocked my work on issueX - resolve conflicts/blockers here we try to resolve any conflicts/blocks Topics of the week (brief discussion + quick decisions) ========================================================= regularly doing more closed sprints? ---------------------------------------- In a private conversation Bea asked Holger how the developer group thinks of the hildesheim2-sprint type (which was similar to the Pre-EP sprint in Gotenburg). A question is: do we think we want to regularly do sprints where we only invite developers who are at least somewhat up to speed with PyPy? Should sprints around conferences always be completely open and some of the sprints in between then possibly more closed? Note that Bea thinks of organizing a discussion about this at the upcoming Heidelberg sprint. 0.6.2 release: dropping the idea and just going for 0.7 ... ---------------------------------------------------------------------------- It has basically been decided by the hildesheim2 sprint group that Heidelberg should tackle a 0.7 self-contained PyPy version. (a previous pypy-sync meeting delegated this decision there,btw). It seems that we then probably want to drop the idea of a 0.6.2 release. But the release goal for 0.6.2, namely the compliancy goal of again reaching 90% compliancy for our 2.4.1 based tree, should then be shifted to 0.7. We should conciously decide this. Plans for fixing compliancy bugs ------------------------------------ It has been discussed by the hildesheim2 sprint group that we want to have a text file in lib-python/ that lists the problems with failing test modules so that we can share our analysis and have a lightweight process of fixing and tracking those issues. Moreover, Holger suggests (or even reminds?) to mandate adding small appropriate tests below pypy/ when fixing compliancy tests instead of just relying on the sometimes very slow-to-run compliancy tests. From hpk at trillke.net Wed Aug 3 20:42:59 2005 From: hpk at trillke.net (holger krekel) Date: Wed, 3 Aug 2005 20:42:59 +0200 Subject: [pypy-dev] pypy-sync meeting 4th August 1pm In-Reply-To: <20050803181147.GH11996@solar.trillke.net> References: <20050803181147.GH11996@solar.trillke.net> Message-ID: <20050803184259.GL11996@solar.trillke.net> On Wed, Aug 03, 2005 at 20:11 +0200, holger krekel wrote: > This post is to announce the pypy-sync meeting for tomorrow > (thursday 4th August) 1pm. I should probably have mentioned that the 30 minute meeting takes place on #pypy-sync on irc.freenode.net. holger From hpk at trillke.net Thu Aug 4 10:02:02 2005 From: hpk at trillke.net (holger krekel) Date: Thu, 4 Aug 2005 10:02:02 +0200 Subject: [pypy-dev] winding down and getting interactive In-Reply-To: <42EFD6D3.50400@gmx.de> References: <42EFD6D3.50400@gmx.de> Message-ID: <20050804080202.GR11996@solar.trillke.net> Hi all, On Tue, Aug 02, 2005 at 22:25 +0200, Carl Friedrich Bolz wrote: > Hi PyPy-Dev! > > Some quit notes about Sunday, the 7th and last day of the sprint. In > general it was sort of a lazy day where we reveled in the success of the > previous night ;-). You can also see some views on this nicely described "winding-down" in my finalized picture set found here: http://codespeak.net/~hpk/hildesheim2-sprint-www/ Moreover, yesterday Samuele managed to fix one remaining problem (see https://codespeak.net/issue/pypy-dev/issue106) that was preventing us from executing "import code ; code.interact()" with the translated PyPy: a self-contained interactive prompt! Of course, we are still missing lots of builtin modules. And of course, it's slow because lots of stuff is running at applevel and we are not tackling optimizations yet. So at least on linux platforms you may have success checking out the latest dist, going to pypy/translator/goal and executing 'sh runtranslate.sh'. This should after around 30 minutes result in a PyPy extension module (it's not using any CPython stuff, it's just still an extension module for historic integration reasons) and 'app_example.py' should get executed. This should result in the mentioned interactive prompt. You can also go to the extension module found in the latest /tmp/usession-NUM directory, and execute: python import testing_1 testing_1.entry_point("/path/to/app_example.py") or specify some other file containing a small test program. cheers & have fun, holger From hpk at trillke.net Thu Aug 4 20:47:31 2005 From: hpk at trillke.net (holger krekel) Date: Thu, 4 Aug 2005 20:47:31 +0200 Subject: [pypy-dev] pypy-sync meeting 4th August 1pm In-Reply-To: <20050803181147.GH11996@solar.trillke.net> References: <20050803181147.GH11996@solar.trillke.net> Message-ID: <20050804184731.GY11996@solar.trillke.net> On Wed, Aug 03, 2005 at 20:11 +0200, holger krekel wrote: > Hi pypy-dev, > > we've had three so called 'pypy-sync' meetings already > on the last thursdays. The minutes are to be found > under http://codespeak.net/svn/pypy/extradoc/minute here are the minutes of today's pypy-sync meeting: http://codespeak.net/pypy/index.cgi?extradoc/minute/pypy-sync-08-04-2005.html or find it pasted below as a text version. (It's still possible that the pypy-sync get correction commits by someone, though). cheers, holger ============================================= pypy-sync developer meeting 4th August 2005 ============================================= Attendees:: Samuele Pedroni, Adrien Di Mascio, Carl Friedrich Bolz, Armin Rigo, Samuele Pedroni, Anders Lehmann, Niklaus Heidimann, Eric van Riet Paap, Holger Krekel (minutes/moderation) with pre info:: Richard Emslie, Ludovic Aubry, Regular Topics ==================== - roll call. holger opens the meeting. - activity reports (3 prepared lines of info). All Attendees submitted activity reports (see `IRC-Log`_ at the end and 'LAST/NEXT/BLOCKERS' entries in particular) - resolve conflicts/blockers No direct conflicts were discovered. Compliancy work was discussed under its own topic. Topics of the week =================== "regular" more closed sprints? --------------------------------- In a private conversation Bea asked Holger how the developer group thinks of the hildesheim2-sprint type (which was similar to the Pre-EP sprint in Gotenburg). A question is: do we think we want to regularly do sprints where we only invite developers who are at least half up to speed with PyPy? Should sprints around conferences always be completely open and some of the sprints in between then possibly more closed? The group agrees that the actual issue is not so much about being "closed" per se but about the active developers being able to pair among themselves. Also it is agreed that interested developers, especially ones close to the sprint venue, should not be excluded even from restricted sprints. Around conferences we always want to invite and prepare for developers completely new to PyPy, whereas between we'd sometime like to restrict the sprints to active developers. It is also clear that we don't want to go against the open source spirit of allowing interested people to participate. 0.6.2 release: dropping the idea and just going for 0.7 ... ---------------------------------------------------------------------------- It has basically been decided by the hildesheim2 sprint group that Heidelberg should tackle a 0.7 self-contained PyPy version. It seems that we then probably want to drop the idea of a 0.6.2 release. But the release goal for 0.6.2, namely the compliancy goal of again reaching 90% compliancy for our 2.4.1 based tree, should then be shifted to 0.7. The attendants quickly agreed that we drop the 0.6.2 release and go for 0.7 directly, including the 0.6.2 release goals. fixing compliancy bugs ------------------------------------ It has been decided by the hildesheim2 sprint group that we want to have a text file in lib-python that lists the problems with failing test modules so that we can share analysis. After a short discussion we reach agreement on a) we add progress information regarding single compliancy tests to lib-python/failure_list.txt. If anyone works longer on a compliancy test then it should be noted in the file to avoid duplication of work. b) fixing a compliancy test should usually be accompanied by adding a small test in our pypy/ hierarchy. Only if the test is too obscure to be reproduced nicely it is ok to not add such a test. c) we may need to prepare a bit that we count the failing/suceeding compliancy tests more correctly. This can probably still be done at the heidelberg sprint if neccessary. Closing ------------------ Holger closes the meeting in time at 13:30pm. .. _`IRC-log`: Here is the full IRC log:: Aug 04 12:54:29 --> You are now talking on #pypy-sync Aug 04 12:58:42 should we start? (we're 6 minutes behind already) Aug 04 12:58:53 ups, my clock says 12:58 Aug 04 12:59:00 sorry Aug 04 12:59:04 then sure, let's start right ahead Aug 04 12:59:06 and hi Aug 04 12:59:09 hi! Aug 04 12:59:11 hi Aug 04 12:59:16 hi :-) Aug 04 12:59:16 hi Aug 04 12:59:17 Hi Aug 04 12:59:20 hi as well Aug 04 12:59:36 ok, let's go, here is the agenda: Aug 04 12:59:44 - roll call. Aug 04 12:59:44 - activity reports (3 prepared lines of info). Aug 04 12:59:44 - resolve conflicts/blockers Aug 04 12:59:44 * Topics of the week * Aug 04 12:59:44 - regularly more closed sprints? Aug 04 12:59:44 - 0.6.2 release: dropping the idea and just going for 0.7?! Aug 04 12:59:44 - fixing compliancy bugs Aug 04 12:59:44 - (re)assignments Aug 04 13:00:04 (i will be pretending, my clock is right, so it's exactly 13:00 now :-) Aug 04 13:00:11 activity reports, i supposed the following order: Aug 04 13:00:11 :-) Aug 04 13:00:22 rxe,hpk,ericvrp2,nik,adim,arigo,aleale,cfbolz,pedronis,adim Aug 04 13:00:28 --- You are now known as rxe Aug 04 13:00:36 LAST: llvm work with Eric (pbc, operations, bpnn, richards, readability of source Aug 04 13:00:36 NEXT: would be nice to translate pypy itself - not too far off - but will probably be busy wit Aug 04 13:00:36 h other stuff Aug 04 13:00:36 BLOCKERS: none Aug 04 13:00:41 --- You are now known as hpk Aug 04 13:00:50 LAST: exception handling in llvm backend Aug 04 13:00:51 NEXT: operator exceptions (divzero,etc) Aug 04 13:00:51 BLOCKERS: a lot of XXX in llvm backend, might need to refactor first Aug 04 13:00:55 LAST: hildesheim2-sprint + after-work/reporting, small fixes + cleanup here and there Aug 04 13:00:55 NEXT: codespeak-migration, (maybe) improve test-report/translation tools Aug 04 13:00:55 BLOCKERS: time pressure regarding migration Aug 04 13:01:02 LAST: integrating _sre and array Aug 04 13:01:10 NEXT: app-level optimizations for _sre, some array improvements. after that (= next week) i'm probably be free for work on general compliance. Aug 04 13:01:14 BLOCKERS: - Aug 04 13:01:27 LAST: gave a course Aug 04 13:01:27 NEXT: continue astbuilder (only since today) Aug 04 13:01:27 BLOCKERS: no really blockers, byt small problems for some isntructions Aug 04 13:01:32 --- adim is now known as luda1 Aug 04 13:01:39 LAST: vacation Aug 04 13:01:39 NEXT: vacation Aug 04 13:01:39 BLOCKERS: none Aug 04 13:01:45 --- luda1 is now known as adim Aug 04 13:01:52 LAST: fixing bugs in translate_pypy; getting a stand-alone executable instead of an ext module Aug 04 13:01:52 NEXT: clean-ups (e.g. the Translator class); work on compliance tests; more bugs shown by translate_pypy Aug 04 13:01:52 BLOCKERS: - Aug 04 13:01:56 prev: Compliance tests, getting settled in at DFKI Aug 04 13:01:58 next: Compliance tests Aug 04 13:01:59 blockers: none Aug 04 13:02:07 LAST: Hildesheim sprint, lltypesimulation on top of memory simulator, looking ways to integrate lltypesimulation with llinterp Aug 04 13:02:07 NEXT: integrating lltypesimulation into lltinterp, enhancing llinterp to be able to use simulated GC (finding roots) Aug 04 13:02:07 BLOCKERS: none Aug 04 13:02:30 Last: sprint, fix obscure(tm) intermittent annotator problem, fix what prevented "import code" to succeed Aug 04 13:02:32 Next: work on open issues Aug 04 13:02:33 Issues: when we try very clever approaches we should try to think of and add sanity checks Aug 04 13:03:00 ok, that's activity reports, there are no immediate blockers ... Aug 04 13:03:15 however, we should talk a bit about how to approach compliancy under its own topic Aug 04 13:03:46 so on to: regularly doing more closed sprints? Aug 04 13:03:56 Bea mailed me and there were discussions here and there Aug 04 13:04:06 what do people think about having two types of sprints Aug 04 13:04:22 like one completely open and more "closed"? Aug 04 13:04:49 how closed would "closed" be? Aug 04 13:05:03 I think I only have tried the "closed" one Aug 04 13:05:29 cfbolz: i'd suggest to define closed as "enough room for the active developers to pair with each other" Aug 04 13:05:56 because i think that is really the issue, the more newbies or not-up-to-speed people the less the active developers can pair with each other Aug 04 13:06:01 sounds good :-) Aug 04 13:06:08 yes Aug 04 13:06:28 hildesheim was very productive Aug 04 13:07:04 any other agreeing/disagreeing opinions on this? Aug 04 13:07:48 Keeping sprints open around conferences seems a good idea, but having more "closed" ones around "deadlines" should be possible too Aug 04 13:07:56 I think that it goes somewhat against the opensource idea Aug 04 13:08:19 not really. depends on definition of "active developers" Aug 04 13:08:21 I would suggest at least 1-2 days for people that might live nearby and want to get the feeling Aug 04 13:08:33 ericvrp2: i see your point Aug 04 13:09:03 for example, i am not worried about heidelberg, because there are quite a lot of people registered that know PyPy reasonably well Aug 04 13:09:28 so if we had 2-4 newbies showing up, that should be quite ok Aug 04 13:09:54 so by your definition it would even be a closed sprint :-) Aug 04 13:10:04 cfbolz: psst! :-) Aug 04 13:11:07 no, I agree. Open would mean something like after EP where really lots of people showed up rather spontaneously Aug 04 13:11:08 ok, i would like to summarize that the general opinion is that closed sprints are ok, but we want to be careful to a) not do them to often b) not harshly exclude interested developers c) look that we don't go against the open-source spirit Aug 04 13:11:19 makes sense to always allow nearby people to show up. We should only recommend to other people to fly to an "open" sprint if they have to fly anyway Aug 04 13:11:37 seems good Aug 04 13:12:01 fine! Aug 04 13:12:11 ok for me. Aug 04 13:12:46 nik: does it also make sense to you? Aug 04 13:13:08 yep, fine with me Aug 04 13:13:32 pedronis, aleale: i presume you don't object either (until you do) Aug 04 13:13:50 correct Aug 04 13:14:06 * hpk 's clock is 13:14 now Aug 04 13:14:20 next topic: 0.6.2 release: dropping the idea and just going for 0.7 ... Aug 04 13:14:27 it's probably uncontroversial Aug 04 13:14:37 but i want to follow up on earlier discussions/decisions Aug 04 13:14:54 i guess that we don't go for any 0.6 release anymore and directly tackle 0.7 at heidelberg Aug 04 13:15:07 everybody fine with this? Aug 04 13:15:11 yes Aug 04 13:15:16 what would have been the focus of 0.6.2? Aug 04 13:15:29 switch to CPython 2.4.1 Aug 04 13:15:42 i see Aug 04 13:15:44 nik: compliance, move to 2.4.1 and the parser running on top of cpython both 2.4 and 2.3 Aug 04 13:16:09 those goals would be shifted to 0.7 Aug 04 13:16:32 agreed Aug 04 13:16:38 yes Aug 04 13:16:50 ok Aug 04 13:16:54 ok with this Aug 04 13:17:00 I'm fine with that Aug 04 13:17:08 great, then the next topic (we still have 13 minutes according to my clock) Aug 04 13:17:18 Plans for fixing compliancy bugs Aug 04 13:17:49 there now is a first incomplete text file failure_list.txt in lib-p?ython Aug 04 13:18:25 the question is a bit a) how we distribute the tasks/analysis Aug 04 13:18:43 b) how we tackle fixing compliancy issues (adding tests or not?) Aug 04 13:19:06 c) do we urgently need to count differently then by failing test file? Aug 04 13:19:22 a) I propose that whoever starts seriously looking at a test, says so in the failure_list.txt Aug 04 13:19:52 and then he follows up there too when he found the problem, and finally says if/when the problem is fixed or if he can't fix it Aug 04 13:20:38 b) let's add tests to our own suite unless they are really too obscure Aug 04 13:20:57 c) I personally don't care too much about counting tests differently Aug 04 13:21:13 arigo: but the EU might :-) Aug 04 13:22:01 the revision xxx before the 2.4.1 switch passed 90% of tests Aug 04 13:22:01 cfbolz: but counting a whole test file just containing one failure as a complete failure is certainly "good for the EU" :-) Aug 04 13:22:13 do they mandate a figure, like 95% compliancy? Aug 04 13:22:20 nik: we promised 90% compliancy Aug 04 13:22:23 hpk: of course :-) Aug 04 13:23:03 arigo: i agree to a) and b) but regarding c): i think we should prepare for counting differently before heidelberg - just in case Aug 04 13:23:14 Lets wait until Heidelberg and see if it is needed Aug 04 13:23:55 hpk: ok Aug 04 13:24:00 yes, but armin's point is valid as well: no matter how we count, it's annoying that a lot of tests fail now that passed once Aug 04 13:24:15 cfbolz: i am not even sure about that: 2.4.1 has maybe just added tests? Aug 04 13:24:36 maybe Aug 04 13:24:40 that may be true Aug 04 13:24:44 i noticed several times Aug 04 13:24:49 that stuff isn't tested in CPython Aug 04 13:24:49 counting differently sound alot like early optimization to me Aug 04 13:25:26 ericvrp2: well, we are really counting wrong currently Aug 04 13:26:02 ericvrp2: consider one test file with 10000 tests all failing, and one test file with 1 test passing -> compkliancy 50% Aug 04 13:26:52 although most of the time it's really the other way round right now Aug 04 13:26:54 hpk: I agree with you, but changing that now (unless 5 minutes work) doesn't sound very productive Aug 04 13:26:59 hpk: fixing the counting should be easy to do at Heidelberg if needed Aug 04 13:27:21 hpk: I think that time is better spent trying to fix the tests themselves, for now Aug 04 13:27:51 ok, fine by me (but i still want to to look _a bit_ into the test reporting, anyway, so i might give that a look as well) Aug 04 13:28:07 sure Aug 04 13:28:28 there are at least curerntly encoding-problems which prevent test reports Aug 04 13:28:35 one suggestion: it would help if the whole compliancy suite was run daily on some server Aug 04 13:28:40 and the results checked in Aug 04 13:28:54 nik: yes, i had that on codespeak.net but stopped it when the server was getting unstable Aug 04 13:29:08 ok, i think we have a conclusion on this Aug 04 13:29:10 hpk: I can do that here, if you give me your script Aug 04 13:29:20 cfbolz: ok, will tell you off-this-meeting Aug 04 13:29:27 ok Aug 04 13:29:36 * hpk is about to close the meeting Aug 04 13:30:02 thanks all for coming, and see you on the neighbour channel :-) Aug 04 13:30:05 :-) Aug 04 13:30:08 bye Aug 04 13:30:17 bye Aug 04 13:30:28 Bye Aug 04 13:30:36 bye Aug 04 13:30:40 byebye Aug 04 13:30:47 bye Aug 04 13:31:06 <-- arigo (~arigo at arigo.sustaining.supporter.pdpc) has left #pypy-sync Aug 04 13:31:14 <-- cfbolz (~test at zenon.physi.uni-heidelberg.de) has left #pypy-sync ("Leaving") Aug 04 13:31:36 <-- nik (~chatzilla at 253.71.77.83.cust.bluewin.ch) has left #pypy-sync Aug 04 13:31:45 <-- aleale (~chatzilla at clogs.dfki.uni-sb.de) has left #pypy-sync Aug 04 13:32:16 <-- adim (~adim at logilab.net2.nerim.net) has left #pypy-sync From tismer at stackless.com Sat Aug 6 12:56:22 2005 From: tismer at stackless.com (Christian Tismer) Date: Sat, 06 Aug 2005 12:56:22 +0200 Subject: [pypy-dev] enabling marshal Message-ID: <42F49756.9070900@stackless.com> Some small issues which came up when I enabled marshal: ------------------------------------------------------- a dict that never gets a value does not create an emptydict, but rtyper crashes. This happens easily, when code is disabled, temporally. os.write does not know that it returns a nonnegative int. I think we should have a way to express this instead of putting assertions. A funny observation on stringobject: The RTyper did not know how to handle mul_str_times. That should imply that marshal pulled this code in, what I don't understand, because I thought all multimethods would be instantiated. I had to initialize the buffer list in the exception case. But later on, I removed this hack and it still worked. Later it was needed, again, so I finally kept it with XXX. Strange interaction, must have been a side effect. I would love to use rpython dicts of integers for caching ids of objects. Is this hard, or bad? Should I use a real dict, instead? Addendum: I don't really see why we have more than one dict implementation at all. In a way, they all share the same basic layout. How hard would it be to factor this out and create the dicts we need, regardless if we are doing rtyper dicts or applevel dicts? It could be possible to do this with specialization. Translation works, but it is a huge batch run, very unpythonic so far. I would like to get into a direction, where we start to isolate modules for seperate compilation. One possible candidate would be marshal. We would need of a way to spell that something is not here, yet, like a stub for marshal's exported methods that raises or redirects to an applevel implementation. I'm anyway very impressed how great this all works, folks! -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From Ben.Young at risk.sungard.com Tue Aug 9 10:32:02 2005 From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com) Date: Tue, 9 Aug 2005 09:32:02 +0100 Subject: [pypy-dev] Build and test failures In-Reply-To: <20050731192412.GA1510@code1.codespeak.net> Message-ID: Thats great news. Congratuations to all of you! Cheers, Ben Armin Rigo wrote on 31/07/2005 20:24:12: > Hi Ben, > > On Fri, Jul 22, 2005 at 09:18:56AM +0100, Ben.Young at risk.sungard.com wrote: > > So is a family of classes the set of any classes that can be assigned to a > > particular instance variable, even if they dont share a common base? > > > > e.g: > > > > class A: > > pass > > > > class B: > > pass > > > > foo = somebool ? A() : B() > > > > Would it be simpler to require that all instances must at least be > > annotated to a common base, and fail to annotate if a not-base-defined > > method is called on the instance. Or is that really what you are already > > doing!? > > Indeed, we track each class that can be in a particular variable, > instead of (at this point) falling back to a common class, requiring > that it exists. While this would not be absolutely necessary for > classes, we need the notion of "set of values" anyway for functions and > for prebuilt constant instances. > > > Do function PBCs all have to have the same signature, or can it vary? > > It can vary to some extent. We have code in rpython/normalizecalls.py > that analyzes calls from a single place that can go to a family of > functions, and that "normalizes" the functions -- i.e. modify their > signature and insert conversions at the beginning of the functions if > needed, so that they all get exactly the same signature. > > > Anyway, thanks for the reply! As illuminating as ever. I guess with the > > rtyping, the first flush of exciting coding has gone and you are down to > > the grinding out bugs stage. Are you pleased with the way things have gone > > or are there any chages you would make to the architecture now? > > Given that we succeeded in translating the whole of PyPy yesterday -- > sorry for the long delay again in answering -- I guess that yes, I'm > pretty much pleased in general with the way everything has gone :-) > > > P.S os.ftruncate doesn't exist on windows I'm afraid, so most of the tests > > fail at the moment > > Thanks for the report. It still fails on Windows, indeed... > > > A bientot, > > Armin > From Ben.Young at risk.sungard.com Wed Aug 10 11:06:08 2005 From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com) Date: Wed, 10 Aug 2005 10:06:08 +0100 Subject: [pypy-dev] Compilation errors Message-ID: Hi everyone, Is anyone else getting compilation errors on any generated c program at the moment? This is happening on Windows, and has been going on for a couple of days. The errors are: - - - - - - - - - - - test_set_attr: recorded stdout - - - - - - - - - - - - testing_1.c^M c:\Documents and Settings\YoungB\dist\pypy\translator\c\src\ll_os.h(92) : warning C4013: 'getcwd' undefined; assuming extern returning int^M c:\Documents and Settings\YoungB\dist\pypy\translator\c\src\ll_os.h(92) : warning C4047: '=' : 'char *' differs in levels of indirectio n from 'int'^M c:\Documents and Settings\YoungB\dist\pypy\translator\c\src\ll_os.h(160) : warning C4244: 'return' : conversion from '__int64' to 'long ', possible loss of data^M c:\Documents and Settings\YoungB\dist\pypy\translator\c\src\ll_strtod.h(26) : error C2143: syntax error : missing ';' before 'type'^M c:\Documents and Settings\YoungB\dist\pypy\translator\c\src\ll_strtod.h(34) : error C2143: syntax error : missing ';' before 'type'^M c:\Documents and Settings\YoungB\dist\pypy\translator\c\src\ll_strtod.h(35) : error C2065: 's' : undeclared identifier^M c:\Documents and Settings\YoungB\dist\pypy\translator\c\src\ll_strtod.h(35) : warning C4047: 'function' : 'char *' differs in levels of indirection from 'int'^M c:\Documents and Settings\YoungB\dist\pypy\translator\c\src\ll_strtod.h(36) : warning C4047: 'function' : 'char *' differs in levels of indirection from 'int'^M c:\Documents and Settings\YoungB\dist\pypy\translator\c\src\ll_strtod.h(37) : warning C4047: 'function' : 'char *' differs in levels of indirection from 'int'^M c:\Documents and Settings\YoungB\dist\pypy\translator\c\src\ll_strtod.h(38) : warning C4047: 'function' : 'char *' differs in levels of indirection from 'int'^M c:\Documents and Settings\YoungB\dist\pypy\translator\c\src\ll_strtod.h(39) : warning C4047: 'function' : 'char *' differs in levels of indirection from 'int'^M c:\Documents and Settings\YoungB\dist\pypy\translator\c\src\ll_strtod.h(40) : warning C4047: 'function' : 'char *' differs in levels of indirection from 'int'^M c:\Documents and Settings\YoungB\dist\pypy\translator\c\src\ll_strtod.h(42) : error C2065: 'buf_size' : undeclared identifier^M c:\Documents and Settings\YoungB\dist\pypy\translator\c\src\ll_strtod.h(42) : warning C4047: '=' : 'char *' differs in levels of indire ction from 'int'^M c:\Documents and Settings\YoungB\dist\pypy\translator\c\src\ll_strtod.h(43) : warning C4047: 'function' : 'const char *' differs in lev els of indirection from 'int'^M c:\Documents and Settings\YoungB\dist\pypy\translator\c\src\ll_strtod.h(45) : warning C4022: 'free' : pointer mismatch for actual param eter 1^M c:\Documents and Settings\YoungB\dist\pypy\translator\c\src\ll_strtod.h(48) : warning C4047: '==' : 'char *' differs in levels of indir ection from 'int'^M c:\Documents and Settings\YoungB\dist\pypy\translator\c\src\ll_strtod.h(53) : warning C4047: 'function' : 'const char *' differs in lev els of indirection from 'int'^M c:\Documents and Settings\YoungB\dist\pypy\translator\c\src\ll_strtod.h(60) : error C2143: syntax error : missing '{' before '*'^M c:\Documents and Settings\YoungB\dist\pypy\translator\c\src\ll_strtod.h(104) : warning C4133: 'return' : incompatible types - from 'RPy String *' to 'int *'^M Not sure exactly what's causing the error, as visual studio never does a file "traceback". Cheers, Ben From hpk at trillke.net Wed Aug 10 20:15:32 2005 From: hpk at trillke.net (holger krekel) Date: Wed, 10 Aug 2005 20:15:32 +0200 Subject: [pypy-dev] pypy-sync meeting 11th Aug 1pm Message-ID: <20050810181532.GP11996@solar.trillke.net> hi pypy-dev, tomorrow is the usual thursday-1pm active developers meeting at #pypy-sync on freenode. See http://codespeak.net/svn/pypy/extradoc/minute for minutes of previous meetings and below for the agenda of this one. If you can't come you may send me or someone else your activity infos by mail and we'll proxy them accordingly. cheers, holger ============================================= pypy-sync developer meeting 11th August 2005 ============================================= Regular Topics ==================== - roll call. - activity reports (3 prepared lines of info). - resolve conflicts/blockers Topics of the week =================== re / array status ------------------ Niklaus is going to drop a few sentences on the status of his current re/array work and maybe open problems (if ever he has any :-). llvm status --------------------- If Eric is there it would be great to hear a bit about the status and progress of the LLVM backend. GC and threading --------------------------------- Two important aspects of the translated PyPy version regard Garbage Collection and Threading. We planned for having both GC and threading implemented as translation aspects. While Carl is working on GC (during his SoC project) we have no translation-aspect code yet regarding threading integration. This topic serves to to stir some first common discussion regarding threading and GC aspects. On friday the so called technical board meets on the GC/thread-related deliverables in the EU project. It would thus be good if pypy-sync is fine with delegating decisions about strategies and coding plans on GC/threading to the technical board. FYI: codespeak migration status ----------------------------------- The migration of codespeak.net got postponed because the target machine's network connectivity is not satisfying yet (latency and dropped packets problems). However, commits are now mirrored to the new machine which is basically ready to take over in case the current machine gets problems. It's possible that the services get migrated soon without prior announcements (unless people really think it's neccessary to pre announce it). From arigo at tunes.org Wed Aug 10 22:47:22 2005 From: arigo at tunes.org (Armin Rigo) Date: Wed, 10 Aug 2005 22:47:22 +0200 Subject: [pypy-dev] Towards Milestone 1 Message-ID: <20050810204722.GB25561@code1.codespeak.net> Hi all, Here are some generally interesting comparisons between the current status of translation in PyPy and what we set as our Milestone 1. This is getting closer, both in term of work remaining to be done (good!) and deadline promized to the EU (hurry!). (For the latter, I am refering to http://codespeak.net/svn/pypy/funding/negotiations/part_b_2004_12_11.pdf pp 80-81.) Near the end of the Hildesheim sprint we acheived the first self-contained translated PyPy. Shortly thereafter we could produce a standalone executable instead of an extension module for CPython. In this respect we have reached an important part of our first milestone. Note that this is still in heavy development; I'm not sure the current trunk successfully translates. It did, take my word :-) (BTW I propose that the pypy-translation-snapshot always contains the latest revision that is known to translate; we should have a script to automate taking a new snapshot.) There are some pieces missing from PyPy itself, most notably some C extension modules of CPython and a few specific features (zip-imports, weakrefs, ...). I will not discuss these here. The subject of this e-mail is to see what is missing translation-wise: * first it would be cool if LLVM could also translate PyPy. This seems to be very close as well! (Eric should tell us more about it after tomorrow's pypy-sync meeting) * memory management: the C back-end uses refcounting; LLVM uses the Boehm GC. It would be quite easy to have a flag to the C back-end to use Boehm as well instead of refcounting. Carl Friedrich is working on a more general GC framework; at the moment it is unclear (at least to me) when and how easily we will be able to use it to add custom GCs to our back-ends. (Carl should tell us more about it after tomorrow's pypy-sync meeting) * threading: implementing this requires a mixture of source-code-level changes and help from the translation process, depending on the approach taken. At the moment we have no threads at all. There are two threading models that are relatively easy to implement by now, and more to think about: 1) the Global Interpreter Lock (GIL), as in CPython. Only one thread interprets Python bytecodes at a time. All other threads are either blocked waiting for the GIL, or doing I/O. In CPython, around each I/O function call, there are hand-coded lines to release and re-acquire the GIL. In PyPy we can insert these lines automatically at translation time, whenever we call one of the hand-coded C functions in pypy/translator/c/src/ll_*.h. The source code of PyPy needs a minor extension so that every 10 or 100 bytecodes it calls a special function "now is a good time to release the GIL to give the other threads a chance to run". Should be fairly straightforward. 2) full Stackless. As long as some rather inefficient solution is good enough, this is not so difficult. We can modify the C back-end to generate functions differently, so that no C function calls any other C function directly. Instead, there is a short "main loop", along the lines of while (1) { next_fn = state->continuation_fn; state = next_fn(state); } Each generated function returns a new 'state' structure whose 'continuation_fn' member contains the function to call next. The 'state' structure also contains arbitrary data like the arguments we want to send to the next function. The net result is that the C stack is no longer used. Then we can have "tasklets" which each record a 'state' to run next, and switching to another tasklet is done by a special C function that returns the 'state' of the other tasklet. Getting the basics done should not be too difficult -- although it's an open door to endless involved optimization hacks, as Christian knows :-) 3) there are also other less well-thought ideas for threading, mostly along the lines of per-object locking. Here too, it should be possible to get a not-too-bad result without having to insert locks everywhere by hand. For example, we could reuse the proxy object space mechanism for a LockingObjSpace which first acquires the lock of each object involved in each space operation. * finally, let's consider the translation process itself. It is flexible in principle, but it's not really designed as a framework with a well-defined API or hooks to plug into. However, it is possible to hack here and there to change various translation aspects. This is, to some extent, the whole idea of PyPy's flexibility: a framework with hooks and APIs allows only so much experimentation; sooner or later the ability to directly code things differently is more powerful. Nevertheless I guess that some refactoring would help to make localized translation aspects easier to change, e.g. by providing more "policy" objects to control the process, or (for OOP enthusiasts) by designing the classes with subclassing in mind. For the short-term future, we have to draw priorities with this EU Milestone 1 in mind. We promized "hooks into the internals to alter translation aspects"; in my opinion (debate welcome!) spending time on this right now is not a really good idea. Better spend it on the more concrete issues of GC and threading, which in some sense already prove that we have enough hooks to do some variations. In particular, the Stackless version of PyPy should be an excellent and -- I believe -- quite reachable result. It's more than a minor variation: it's a completely different kind of generated code. It would definitely show that our translation process is capable of producing more than just a dummy translation -- which is the whole point. A bientot, Armin. From jacob at strakt.com Thu Aug 11 10:47:28 2005 From: jacob at strakt.com (Jacob =?iso-8859-1?q?Hall=E9n?=) Date: Thu, 11 Aug 2005 10:47:28 +0200 Subject: [pypy-dev] Towards Milestone 1 In-Reply-To: <20050810204722.GB25561@code1.codespeak.net> References: <20050810204722.GB25561@code1.codespeak.net> Message-ID: <200508111047.28428.jacob@strakt.com> onsdagen den 10 augusti 2005 22.47 skrev Armin Rigo: > * threading: implementing this requires a mixture of source-code-level > changes and help from the translation process, depending on the > approach taken. At the moment we have no threads at all. There are > two threading models that are relatively easy to implement by now, and > more to think about: ... > 3) there are also other less well-thought ideas for threading, mostly > along the lines of per-object locking. Here too, it should be > possible to get a not-too-bad result without having to insert locks > everywhere by hand. For example, we could reuse the proxy object > space mechanism for a LockingObjSpace which first acquires the lock > of each object involved in each space operation. > > * finally, let's consider the translation process itself. It is > flexible in principle, but it's not really designed as a framework > with a well-defined API or hooks to plug into. However, it is > possible to hack here and there to change various translation aspects. > This is, to some extent, the whole idea of PyPy's flexibility: a > framework with hooks and APIs allows only so much experimentation; > sooner or later the ability to directly code things differently is > more powerful. Nevertheless I guess that some refactoring would help > to make localized translation aspects easier to change, e.g. by providing > more "policy" objects to control the process, or (for OOP enthusiasts) > by designing the classes with subclassing in mind. A loose idea that I have had would be to have one object space per thread. There would be several different possibilities for how to access an object that isn't in your own object space; one would be that the creating thread would continue to own the object. Another one would be to marshal the object and transfer it. A third one would be to transfer shared object to a shared object space. Jacob From tismer at stackless.com Thu Aug 11 14:03:48 2005 From: tismer at stackless.com (Christian Tismer) Date: Thu, 11 Aug 2005 14:03:48 +0200 Subject: [pypy-dev] Towards Milestone 1 In-Reply-To: <200508111047.28428.jacob@strakt.com> References: <20050810204722.GB25561@code1.codespeak.net> <200508111047.28428.jacob@strakt.com> Message-ID: <42FB3EA4.40702@stackless.com> Jacob Hall?n wrote: > A loose idea that I have had would be to have one object space per thread. > There would be several different possibilities for how to access an object > that isn't in your own object space; one would be that the creating thread > would continue to own the object. Another one would be to marshal the object > and transfer it. A third one would be to transfer shared object to a shared > object space. In extent to what Armin suggested, I think this makes very much sense, too. It is different from what CPython does, and it requires no GIL. CPython makes the "mistake"to allow for threading, while making everything world visible. This requires the GIL or every object to be protected, which costs > 30 % of performance. Going the opposite way makes communication between threads less easy, but doesn't affect the default behavior. Notabene that this is not far away to do threading without threads, but just to spawn new processes. I think we should try these all and later let the user design what he needs in her particular application. ciao - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From arigo at tunes.org Thu Aug 11 17:36:00 2005 From: arigo at tunes.org (Armin Rigo) Date: Thu, 11 Aug 2005 17:36:00 +0200 Subject: [pypy-dev] Towards Milestone 1 In-Reply-To: <42FB3EA4.40702@stackless.com> References: <20050810204722.GB25561@code1.codespeak.net> <200508111047.28428.jacob@strakt.com> <42FB3EA4.40702@stackless.com> Message-ID: <20050811153600.GA4998@code1.codespeak.net> Hi Christian, On Thu, Aug 11, 2005 at 02:03:48PM +0200, Christian Tismer wrote: > >A loose idea that I have had would be to have one object space per thread. > > Notabene that this is not far away to do threading without threads, > but just to spawn new processes. Yes, this is close to a multiprocess approach (as we can do today with multiple CPython processes). An alternative might be to not marshal the objects to pass them around; they can be "virtually" passed around but stay where they are physically in memory. This is better because we can have container objects that point to objects from other spaces, that themselves can point to objects from yet other spaces, etc. Note that this reintroduces some locking issues but they could have a much better performance impact (e.g. only along the boundaries between spaces). Finally, note that in practice it is unclear if several object space instances are really the best way to implement it; depending on the situation it could also be implemented as a single object space, with only conceptual boundaries, implemented with proxy objects (similar to the lazy objects of the thunk object space). I know that multiple object spaces was a hot idea at the beginning but it's still unclear how to use this idea... A bientot, Armin. From tismer at stackless.com Sun Aug 14 17:00:19 2005 From: tismer at stackless.com (Christian Tismer) Date: Sun, 14 Aug 2005 17:00:19 +0200 Subject: [pypy-dev] Re: [pypy-svn] r16054 - pypy/dist/pypy/translator/llvm2/test In-Reply-To: <20050813224005.BD17A27B6A@code1.codespeak.net> References: <20050813224005.BD17A27B6A@code1.codespeak.net> Message-ID: <42FF5C83.1010803@stackless.com> rxe at codespeak.net wrote: > ASIDE - any pointers to any docs relating to acronyms used in the > transformation would be much appreciated. Before I read the docstring I > assumed it was doing the opposite of what it does. Sorry for my ignorance in > this matter. This article is quite useful: http://www.cl.cam.ac.uk/~jds31/research/ssi.html ciao - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From tismer at stackless.com Mon Aug 15 02:40:21 2005 From: tismer at stackless.com (Christian Tismer) Date: Mon, 15 Aug 2005 02:40:21 +0200 Subject: [pypy-dev] Re: [pypy-svn] r16070 - pypy/dist/pypy/objspace/std In-Reply-To: <20050814193303.A486627B6D@code1.codespeak.net> References: <20050814193303.A486627B6D@code1.codespeak.net> Message-ID: <42FFE475.9000200@stackless.com> ericvrp at codespeak.net wrote: > Author: ericvrp > Date: Sun Aug 14 21:33:03 2005 > New Revision: 16070 > > Modified: > pypy/dist/pypy/objspace/std/strutil.py > Log: > Better to use math.log10(float) consistently. > Backends (at least llvm) will have no math.log10(int) implementation. That's correct in principle, but not the reasoning: The code is applevel and therefore allowed to be more sloppy. No backend will ever see it. The actual version is interp_string_to_float, and that had the correction. ciao - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From tismer at stackless.com Mon Aug 15 02:50:44 2005 From: tismer at stackless.com (Christian Tismer) Date: Mon, 15 Aug 2005 02:50:44 +0200 Subject: [pypy-dev] math.log0 / interactive shell Message-ID: <42FFE6E4.5060502@stackless.com> Just a funny observation: in interpreted PyPy, math.log10(1L << 10000) works just fine. In compiled PyPy, I get an overflow error. Can this be related to some Exception problems? Other problem: When the compiled pypy shell starts, and any exception occours, it takes about 10 minutes(!) until you get the exception. After that, it is as low or fast as other command prompts. This was one reason why I looked into geninterp and gateway, becasue I expected that something would be compiled late, but I didn't find it. Which doesn't mean anything :-) ciao - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From hpk at trillke.net Mon Aug 15 10:07:49 2005 From: hpk at trillke.net (holger krekel) Date: Mon, 15 Aug 2005 10:07:49 +0200 Subject: [pypy-dev] math.log0 / interactive shell In-Reply-To: <42FFE6E4.5060502@stackless.com> References: <42FFE6E4.5060502@stackless.com> Message-ID: <20050815080749.GF666@solar.trillke.net> On Mon, Aug 15, 2005 at 02:50 +0200, Christian Tismer wrote: > When the compiled pypy shell starts, and any exception > occours, it takes about 10 minutes(!) until you get > the exception. probably because the (applevel) compiler has to deal with importing traceback at applevel and then running the traceback printing logic (possibly involving reading files which again live at applevel). So maybe it's not too suprising? cheers, holger From tismer at stackless.com Mon Aug 15 10:18:27 2005 From: tismer at stackless.com (Christian Tismer) Date: Mon, 15 Aug 2005 10:18:27 +0200 Subject: [pypy-dev] math.log0 / interactive shell In-Reply-To: <20050815080749.GF666@solar.trillke.net> References: <42FFE6E4.5060502@stackless.com> <20050815080749.GF666@solar.trillke.net> Message-ID: <43004FD3.6020906@stackless.com> holger krekel wrote: > On Mon, Aug 15, 2005 at 02:50 +0200, Christian Tismer wrote: > >>When the compiled pypy shell starts, and any exception >>occours, it takes about 10 minutes(!) until you get >>the exception. > > > probably because the (applevel) compiler has to > deal with importing traceback at applevel and then > running the traceback printing logic (possibly > involving reading files which again live at applevel). > So maybe it's not too suprising? It is, still. We have the marshal module, and you might have recognized how startup time goes from several minutes to fractions of seconds, when it is importing code. Why should this not work in the exception case? That's exactly the question: what do I miss to cache? -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From arigo at tunes.org Mon Aug 15 10:30:28 2005 From: arigo at tunes.org (Armin Rigo) Date: Mon, 15 Aug 2005 10:30:28 +0200 Subject: [pypy-dev] Re: genc to genllvm In-Reply-To: References: Message-ID: <20050815083027.GA779@code1.codespeak.net> Hi Richard! (CC to pypy-dev) On Sun, Aug 14, 2005 at 12:29:51AM +0100, Richard Emslie wrote: > Remember we talked about this at the end of the last sprint: > > def test_cast_to_int(): > def casting(v): > return int(ord(chr(v))) > ... being what caused the seg fault in the first few iterations of > genc'ing and pypy IIRC. Well I went hunting for the test again and didnt > find it (so the above test is mine, along with misunderstandings :-). It's in translator/c/test/test_typed.py: test_ord_returns_a_positive. > What I dont understand is that if rpy_string chars are supposed to be > signed so why were we expecting this to work? The Char type is supposed to represent characters; a character is not necessarily the same thing as a number. Only in C there is a confusion between them. In Python for example it doesn't really make sense to ask whether a character is signed or not. All that matters is the order between characters, and the ord() function. The only important thing is that these work as expected. So we decided a bit arbitrarily to use the C 'char' type for characters. > (*) actually - wouldnt rpy_string with unsigned chars make more sense? > Guess would involve more casting to and from external functions though. Exactly. Using 'char' directly is more natural in C, and putting some casts at two places (comparison and ord()) is rather easy. Note that some C compilers think that 'char' is signed and some think that it is unsigned, as far as I know. > Why is it wrong when we do a cast from an int to char (which overflows > and becomes negative) then the cast to an int which doesnt overflow > and has the same value. A cast from int to char is meant to represent the function chr(). The reverse is for the function ord(). It's not just arbitrary casts -- you don't have casts in Python or RPython -- it's really about these two functions. The goal is that after translation code like assert ord(chr(x)) == x continues to work for any x in range(256). The direct numeric value of the character chr(x) is not relevant because we cannot inspect it in Python/RPython; we have to use ord(). A bientot, Armin From eric at vanrietpaap.nl Mon Aug 15 10:35:16 2005 From: eric at vanrietpaap.nl (Eric van Riet Paap) Date: Mon, 15 Aug 2005 10:35:16 +0200 Subject: [pypy-dev] Re: [pypy-svn] r16070 - pypy/dist/pypy/objspace/std In-Reply-To: <42FFE475.9000200@stackless.com> References: <20050814193303.A486627B6D@code1.codespeak.net> <42FFE475.9000200@stackless.com> Message-ID: <430053C4.5060703@vanrietpaap.nl> comments inline: Christian Tismer wrote: > ericvrp at codespeak.net wrote: > >> Author: ericvrp >> Date: Sun Aug 14 21:33:03 2005 >> New Revision: 16070 >> >> Modified: >> pypy/dist/pypy/objspace/std/strutil.py >> Log: >> Better to use math.log10(float) consistently. Backends (at least >> llvm) will have no math.log10(int) implementation. > > > That's correct in principle, but not the reasoning: > The code is applevel and therefore allowed to be more sloppy. > No backend will ever see it. Because this is a function with suggested_primitive tag attached, the backends will have to implement them. If we do not use the parameter types consistently, the backends will have to implement multiple versions. The fact that this is applevel does not play a role here, as far as I can see. cheers Eric > The actual version is interp_string_to_float, and that had the > correction. > > ciao - chris From arigo at tunes.org Mon Aug 15 10:51:18 2005 From: arigo at tunes.org (Armin Rigo) Date: Mon, 15 Aug 2005 10:51:18 +0200 Subject: [pypy-dev] Re: [pypy-svn] r16070 - pypy/dist/pypy/objspace/std In-Reply-To: <430053C4.5060703@vanrietpaap.nl> References: <20050814193303.A486627B6D@code1.codespeak.net> <42FFE475.9000200@stackless.com> <430053C4.5060703@vanrietpaap.nl> Message-ID: <20050815085118.GA1330@code1.codespeak.net> Hi Eric, On Mon, Aug 15, 2005 at 10:35:16AM +0200, Eric van Riet Paap wrote: > Because this is a function with suggested_primitive tag attached, the > backends will have to implement them. > If we do not use the parameter types consistently, the backends will > have to implement multiple versions. > The fact that this is applevel does not play a role here, as far as I > can see. Precisely not: app-level code is normal full Python code; the back-end never sees it. This 'math.log10()' is from our mixed module math, not from the back-end. Armin. From eric at vanrietpaap.nl Mon Aug 15 10:54:01 2005 From: eric at vanrietpaap.nl (Eric van Riet Paap) Date: Mon, 15 Aug 2005 10:54:01 +0200 Subject: [pypy-dev] Re: [pypy-svn] r16070 - pypy/dist/pypy/objspace/std In-Reply-To: <20050815085118.GA1330@code1.codespeak.net> References: <20050814193303.A486627B6D@code1.codespeak.net> <42FFE475.9000200@stackless.com> <430053C4.5060703@vanrietpaap.nl> <20050815085118.GA1330@code1.codespeak.net> Message-ID: <43005829.6090807@vanrietpaap.nl> Armin Rigo wrote: >Hi Eric, > >On Mon, Aug 15, 2005 at 10:35:16AM +0200, Eric van Riet Paap wrote: > > >>Because this is a function with suggested_primitive tag attached, the >>backends will have to implement them. >>If we do not use the parameter types consistently, the backends will >>have to implement multiple versions. >>The fact that this is applevel does not play a role here, as far as I >>can see. >> >> > >Precisely not: app-level code is normal full Python code; the back-end >never sees it. This 'math.log10()' is from our mixed module math, not >from the back-end. > > I am sure you are right! But why then did I get a call to ll_math_log10(int 2) in the backend? Eric > >Armin. >_______________________________________________ >pypy-dev at codespeak.net >http://codespeak.net/mailman/listinfo/pypy-dev > > From arigo at tunes.org Mon Aug 15 10:54:07 2005 From: arigo at tunes.org (Armin Rigo) Date: Mon, 15 Aug 2005 10:54:07 +0200 Subject: [pypy-dev] math.log0 / interactive shell In-Reply-To: <43004FD3.6020906@stackless.com> References: <42FFE6E4.5060502@stackless.com> <20050815080749.GF666@solar.trillke.net> <43004FD3.6020906@stackless.com> Message-ID: <20050815085407.GB1330@code1.codespeak.net> Hi Christian, On Mon, Aug 15, 2005 at 10:18:27AM +0200, Christian Tismer wrote: > >>When the compiled pypy shell starts, and any exception > >>occours, it takes about 10 minutes(!) until you get > >>the exception. I have no clue either. Even importing a few modules shouldn't take so long. We should try to run this under gdb and see where is goes loafing around... A bientot, Armin From arigo at tunes.org Mon Aug 15 10:59:38 2005 From: arigo at tunes.org (Armin Rigo) Date: Mon, 15 Aug 2005 10:59:38 +0200 Subject: [pypy-dev] Re: [pypy-svn] r16070 - pypy/dist/pypy/objspace/std In-Reply-To: <43005829.6090807@vanrietpaap.nl> References: <20050814193303.A486627B6D@code1.codespeak.net> <42FFE475.9000200@stackless.com> <430053C4.5060703@vanrietpaap.nl> <20050815085118.GA1330@code1.codespeak.net> <43005829.6090807@vanrietpaap.nl> Message-ID: <20050815085938.GC1330@code1.codespeak.net> Hi Eric, On Mon, Aug 15, 2005 at 10:54:01AM +0200, Eric van Riet Paap wrote: > But why then did I get a call to ll_math_log10(int 2) in the backend? Ah, oups. Correction: math.log10 is supposed to be allowed at interp-level. I see that we use it to implement the app-level math.log10(). If I remember the reasoning correctly, there was no point in trying not to use log10() at interp-level because back-ends typically provide this function directly (e.g. C has it in math.h). So actually it's expected that you get ll_math_log10 in LLVM... Sorry for the confusion, Armin From eric at vanrietpaap.nl Mon Aug 15 11:08:51 2005 From: eric at vanrietpaap.nl (Eric van Riet Paap) Date: Mon, 15 Aug 2005 11:08:51 +0200 Subject: [pypy-dev] Re: [pypy-svn] r16070 - pypy/dist/pypy/objspace/std In-Reply-To: <20050815085938.GC1330@code1.codespeak.net> References: <20050814193303.A486627B6D@code1.codespeak.net> <42FFE475.9000200@stackless.com> <430053C4.5060703@vanrietpaap.nl> <20050815085118.GA1330@code1.codespeak.net> <43005829.6090807@vanrietpaap.nl> <20050815085938.GC1330@code1.codespeak.net> Message-ID: <43005BA3.6070208@vanrietpaap.nl> Armin Rigo wrote: >Hi Eric, > >On Mon, Aug 15, 2005 at 10:54:01AM +0200, Eric van Riet Paap wrote: > > >>But why then did I get a call to ll_math_log10(int 2) in the backend? >> >> > >Ah, oups. Correction: math.log10 is supposed to be allowed at >interp-level. I see that we use it to implement the app-level >math.log10(). If I remember the reasoning correctly, there was no point >in trying not to use log10() at interp-level because back-ends typically >provide this function directly (e.g. C has it in math.h). > >So actually it's expected that you get ll_math_log10 in LLVM... > > Yes, and therefor the invocation needs to be math.log10(double). Or probably better would be that pypy not marks math.log10 as a primitive function, but math.log10(double) and provide the correct cast from int to double?!? Eric > >Sorry for the confusion, > >Armin >_______________________________________________ >pypy-dev at codespeak.net >http://codespeak.net/mailman/listinfo/pypy-dev > > From Ben.Young at risk.sungard.com Mon Aug 15 12:16:32 2005 From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com) Date: Mon, 15 Aug 2005 11:16:32 +0100 Subject: [pypy-dev] m.lib Message-ID: Hi, Does anyone know what m.lib is, as alot of the executables produced seem to want to link with it, and I don't have it! (This is on Windows, usgin cygwin) Cheers, Ben From tismer at stackless.com Mon Aug 15 14:44:59 2005 From: tismer at stackless.com (Christian Tismer) Date: Mon, 15 Aug 2005 14:44:59 +0200 Subject: [pypy-dev] m.lib In-Reply-To: References: Message-ID: <43008E4B.6070403@stackless.com> Ben.Young at risk.sungard.com wrote: > Hi, > > Does anyone know what m.lib is, as alot of the executables produced seem > to want to link with it, and I don't have it! (This is on Windows, usgin > cygwin) I got the same problem. It appears to be a glitch with distutils, which to my knowledge were not made for production of standalone executables. At least it works when producing an extension module. I'm currently building a VC project for building PyPy and will check the dependencies. ciao - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From arigo at tunes.org Mon Aug 15 22:43:44 2005 From: arigo at tunes.org (Armin Rigo) Date: Mon, 15 Aug 2005 22:43:44 +0200 Subject: [pypy-dev] m.lib In-Reply-To: <43008E4B.6070403@stackless.com> References: <43008E4B.6070403@stackless.com> Message-ID: <20050815204344.GA8361@code1.codespeak.net> Hi, On Mon, Aug 15, 2005 at 02:44:59PM +0200, Christian Tismer wrote: > >Does anyone know what m.lib is, as alot of the executables produced seem > >to want to link with it, and I don't have it! (This is on Windows, usgin > >cygwin) > > I got the same problem. It appears to be a glitch with > distutils, which to my knowledge were not made for > production of standalone executables. At least it works > when producing an extension module. It's the math lib, needed for on Linux. In translator/tool/cbuild.py, we just add 'm' to the list of libraries to compile a standalone program. I guess we need to be more subtle here... Armin From arigo at tunes.org Mon Aug 15 23:51:33 2005 From: arigo at tunes.org (Armin Rigo) Date: Mon, 15 Aug 2005 23:51:33 +0200 Subject: [pypy-dev] Re: [pypy-svn] r16013 - in pypy/dist/pypy/interpreter/pyparser: . test In-Reply-To: <20050812161151.45DFB27B64@code1.codespeak.net> References: <20050812161151.45DFB27B64@code1.codespeak.net> Message-ID: <20050815215133.GA9072@code1.codespeak.net> Hi Adrien, On Fri, Aug 12, 2005 at 06:11:51PM +0200, adim at codespeak.net wrote: > - don't use eval() builtin anymore in eval_number() and eval_string(). The > replacement implementation is incomplete though, so it needs to be > finished (esp. for raw/unicode strings, and 0x-like numbers) We have done this code during the last sprint, "en passant". Sorry, I intended to mention it here but forgot. We needed it because the app-level ast transformer from the stdlib also uses eval() to get literals, but this causes an infinite recursion (eval() at app-level causes the compiler to be called, which uses the ast transformer again). The code is: * for string literals: interpreter/pyparser/parsestring.parsestr() * for numeric literals: lib/_stablecompiler/transformer.decode_numeric_literal() The latter is not RPython but it's not too long either. Getting these right was a bit involved so I guess there is no point in duplicating the effort... A bientot, Armin From tismer at stackless.com Tue Aug 16 11:38:34 2005 From: tismer at stackless.com (Christian Tismer) Date: Tue, 16 Aug 2005 11:38:34 +0200 Subject: [pypy-dev] m.lib In-Reply-To: <20050815204344.GA8361@code1.codespeak.net> References: <43008E4B.6070403@stackless.com> <20050815204344.GA8361@code1.codespeak.net> Message-ID: <4301B41A.2010002@stackless.com> Armin Rigo wrote: > Hi, > > On Mon, Aug 15, 2005 at 02:44:59PM +0200, Christian Tismer wrote: > >>>Does anyone know what m.lib is, as alot of the executables produced seem >>>to want to link with it, and I don't have it! (This is on Windows, usgin >>>cygwin) >> >>I got the same problem. It appears to be a glitch with >>distutils, which to my knowledge were not made for >>production of standalone executables. At least it works >>when producing an extension module. > > > It's the math lib, needed for on Linux. In > translator/tool/cbuild.py, we just add 'm' to the list of libraries to > compile a standalone program. I guess we need to be more subtle here... Did that. We can now batch-build on Windows. ciao - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From Ben.Young at risk.sungard.com Tue Aug 16 12:44:44 2005 From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com) Date: Tue, 16 Aug 2005 11:44:44 +0100 Subject: [pypy-dev] Fw: Patches Message-ID: Hi Everyone, I was wondering if anyone would mind applying this patch that causes a few more tests to pass on windows. After this has been applied I've got the number of test failures (on windows) down to 14. Cheers, Ben From Ben.Young at risk.sungard.com Tue Aug 16 12:48:08 2005 From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com) Date: Tue, 16 Aug 2005 11:48:08 +0100 Subject: [pypy-dev] Fw: Patches In-Reply-To: Message-ID: Not sure if the patch actually got through. Here it is directly: Index: module/unicodedata/test/test_unicodedata.py =================================================================== --- module/unicodedata/test/test_unicodedata.py (revision 16097) +++ module/unicodedata/test/test_unicodedata.py (working copy) @@ -42,9 +42,12 @@ def test_cjk(self): import unicodedata + import sys for first, last in ((0x3400, 0x4DB5), (0x4E00, 0x9FA5), # 9FBB in Unicode 4.1 (0x20000, 0x2A6D6)): + if last + 1 > sys.maxunicode: + continue # Test at and inside the boundary for i in (first, first + 1, last - 1, last): charname = 'CJK UNIFIED IDEOGRAPH-%X'%i Index: module/__builtin__/test/impsubdir/compiled/x.pyc =================================================================== Cannot display: file marked as a binary type. svn:mime-type = application/octet-stream Index: lib/test2/test_file_extra.py =================================================================== --- lib/test2/test_file_extra.py (revision 16097) +++ lib/test2/test_file_extra.py (working copy) @@ -27,7 +27,7 @@ def test_repr(self): r = repr(self.file) assert r.find('open file') >= 0 - assert r.find(self.file.name) >= 0 + assert r.find(repr(self.file.name)) >= 0 assert r.find(self.file.mode) >= 0 class TestFdFile(TestFile): Index: translator/llvm2/test/test_lltype.py =================================================================== --- translator/llvm2/test/test_lltype.py (revision 16097) +++ translator/llvm2/test/test_lltype.py (working copy) @@ -227,9 +227,9 @@ floats.f1 = 1.25 floats.f2 = 10000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000.252984 floats.f3 = float(29050000000000000000000000000000000000000000000000000000000000000000) - floats.f4 = float("inf") - floats.f5 = float("nan") - f = float("nan") + #floats.f4 = float("inf") + #floats.f5 = float("nan") + #f = float("nan") def floats_fn(): res = floats.f1 == 1.25 res += floats.f2 > 1e100 pypy-dev-bounces at codespeak.net wrote on 16/08/2005 11:44:44: > Hi Everyone, > > I was wondering if anyone would mind applying this patch that causes a few > more tests to pass on windows. After this has been applied I've got the > number of test failures (on windows) down to 14. > > > > Cheers, > Ben_______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev From arigo at tunes.org Tue Aug 16 21:31:04 2005 From: arigo at tunes.org (Armin Rigo) Date: Tue, 16 Aug 2005 21:31:04 +0200 Subject: [pypy-dev] m.lib In-Reply-To: <4301B41A.2010002@stackless.com> References: <43008E4B.6070403@stackless.com> <20050815204344.GA8361@code1.codespeak.net> <4301B41A.2010002@stackless.com> Message-ID: <20050816193104.GA21659@code1.codespeak.net> Hi Christian, On Tue, Aug 16, 2005 at 11:38:34AM +0200, Christian Tismer wrote: > Did that. We can now batch-build on Windows. Great! Armin From arigo at tunes.org Tue Aug 16 21:33:47 2005 From: arigo at tunes.org (Armin Rigo) Date: Tue, 16 Aug 2005 21:33:47 +0200 Subject: [pypy-dev] Re: [pypy-svn] r16070 - pypy/dist/pypy/objspace/std In-Reply-To: <43005BA3.6070208@vanrietpaap.nl> References: <20050814193303.A486627B6D@code1.codespeak.net> <42FFE475.9000200@stackless.com> <430053C4.5060703@vanrietpaap.nl> <20050815085118.GA1330@code1.codespeak.net> <43005829.6090807@vanrietpaap.nl> <20050815085938.GC1330@code1.codespeak.net> <43005BA3.6070208@vanrietpaap.nl> Message-ID: <20050816193347.GB21659@code1.codespeak.net> Hi Eric, On Mon, Aug 15, 2005 at 11:08:51AM +0200, Eric van Riet Paap wrote: > Or probably better would be that pypy not marks math.log10 as a > primitive function, > but math.log10(double) and provide the correct cast from int to double?!? Indeed. Our external function table only specifies the return type, because the annotator can usually find the correct type for the argument based on the usage. But this int/float confusion shows a limitation of this idea. For now, it will do, as we can use log10(2.0) instead of log10(2) in the source, as you did. For the future I guess we should put this information in the table explicitly (and complain at annotation-time if the function is called with something that doesn't match at all). A bientot, Armin. From rxe at ukshells.co.uk Wed Aug 17 17:03:53 2005 From: rxe at ukshells.co.uk (Richard Emslie) Date: Wed, 17 Aug 2005 16:03:53 +0100 (BST) Subject: [pypy-dev] Fw: Patches In-Reply-To: References: Message-ID: Hi Ben, An updated version is now in svn that should work on windows - but isnt tested yet. Thanks for reporting it. (thanks to mwh for helping fix it) Cheers, Richard On Tue, 16 Aug 2005 Ben.Young at risk.sungard.com wrote: > Index: module/unicodedata/test/test_unicodedata.py > Index: translator/llvm2/test/test_lltype.py > =================================================================== > --- translator/llvm2/test/test_lltype.py (revision 16097) > +++ translator/llvm2/test/test_lltype.py (working copy) > @@ -227,9 +227,9 @@ > floats.f1 = 1.25 > floats.f2 = > 10000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000.252984 > floats.f3 = > float(29050000000000000000000000000000000000000000000000000000000000000000) > - floats.f4 = float("inf") > - floats.f5 = float("nan") > - f = float("nan") > + #floats.f4 = float("inf") > + #floats.f5 = float("nan") > + #f = float("nan") > def floats_fn(): > res = floats.f1 == 1.25 > res += floats.f2 > 1e100 > > > pypy-dev-bounces at codespeak.net wrote on 16/08/2005 11:44:44: > >> Hi Everyone, >> >> I was wondering if anyone would mind applying this patch that causes a > few >> more tests to pass on windows. After this has been applied I've got the >> number of test failures (on windows) down to 14. >> >> >> >> Cheers, >> Ben_______________________________________________ >> pypy-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/pypy-dev > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > > From eric at vanrietpaap.nl Wed Aug 17 22:49:27 2005 From: eric at vanrietpaap.nl (Eric van Riet Paap) Date: Wed, 17 Aug 2005 22:49:27 +0200 Subject: [pypy-dev] genllvm standalone announcement Message-ID: <200508172249.30938.eric@vanrietpaap.nl> We are proud to announce that since this afternoon the llvm backend can also create a standalone executable of the PyPy interpreter! The current version even runs up to and including the InteractiveConsole, though some things are obviously not working yet. The llvm backend would not have been possible without all the people contributing to PyPy. Carl Friedrich did an amazing amount for groundwork during the first half of this year. Holger helped out with the architecture of the current llvm backend and Richard and I worked on it for the last two months. If you would like to run the current version on x86 (linux), the executable can be downloaded from http://codespeak.net/~ericvrp/download/entry_point You will see some debug information and sometimes segfaults because not all external functions are (fully) implemented and neither are some other features. At the Heidelberg sprint hopefully the remaining functionality will be implemented. So for PyPy 0.7 we will have two fully self-contained executables. Who would have thought that 2 months ago... enjoy, Eric From hpk at trillke.net Thu Aug 18 02:21:12 2005 From: hpk at trillke.net (holger krekel) Date: Thu, 18 Aug 2005 02:21:12 +0200 Subject: [pypy-dev] pypy-sync 18th August Message-ID: <20050818002112.GW666@solar.trillke.net> Hi folks, as usually there is the thursday #pypy-sync meeting at 1pm. This weeks topic is planning the heidelberg sprint in some more detail. cheers, holger From hpk at trillke.net Tue Aug 23 11:27:43 2005 From: hpk at trillke.net (holger krekel) Date: Tue, 23 Aug 2005 11:27:43 +0200 Subject: [pypy-dev] internal website changes Message-ID: <20050823092743.GU666@solar.trillke.net> Hi pypy-dev, today i'd like to internally restructure the PyPy website and the documentation in particular. It seems sensible to do this before the next release that we are currently working on at the Heidelberg sprint. The goal of the new scheme is to get rid of the dynamic '.cgi' script handling and instead be able to generate the website by going into pypy/doc and issueing 'py.test' which will then generate static web pages that you can preview locally. It also checks referential integrity and with '-R' you can check remote URLs from the documentation. As a result, the URLs will change but i'll try to redirect the current URLs to the new location as good as possible. If you notice website-related problems later in the afternoon then please mail me directly instead of spamming pypy-dev. The new scheme also should also enable us to do per-version documentation of PyPy which will probably (hopefully? :-) become an issue some time soon. cheers, holger From Ben.Young at risk.sungard.com Wed Aug 24 10:17:21 2005 From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com) Date: Wed, 24 Aug 2005 09:17:21 +0100 Subject: [pypy-dev] Link failures on windows Message-ID: Hi Everyone, Did someone change the compilation chain so that on windows the modules don't link against Python any more? At the moment, all the genc tests fail with errors like: testing_11.obj : error LNK2019: unresolved external symbol _PyErr_SetString referenced in function __RPyConvertExceptionToCPython testing_11.obj : error LNK2019: unresolved external symbol _PyErr_SetNone referenced in function __RPyConvertExceptionToCPython testing_11.obj : error LNK2019: unresolved external symbol _PyClass_IsSubclass referenced in function __RPyConvertExceptionToCPython testing_11.obj : error LNK2019: unresolved external symbol _PyExc_Exception referenced in function __RPyConvertExceptionToCPython testing_11.obj : error LNK2019: unresolved external symbol _PyClass_Type referenced in function __RPyConvertExceptionToCPython testing_11.obj : error LNK2019: unresolved external symbol _PyDict_GetItemString referenced in function __RPyConvertExceptionToCPython testing_11.obj : error LNK2019: unresolved external symbol _PyEval_GetBuiltins referenced in function __RPyConvertExceptionToCPython testing_11.obj : error LNK2019: unresolved external symbol _PyMethod_New referenced in function _gencfunc_descr_get ... Cheers, Ben From cfbolz at gmx.de Thu Aug 25 11:50:48 2005 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Thu, 25 Aug 2005 11:50:48 +0200 Subject: [pypy-dev] Heidelberg Sprint Report Message-ID: <430D9478.9060608@gmx.de> Hi PyPy-Dev! Some notes about what's going on in Heidelberg: People arrived on Saturday and Sunday but we started hacking on Monday morning. First we had to solve all sorts of network problems because the computing center of the University of Heidelberg imposes a kind of restrictive setup. After this was done we had a planning meeting, distributed the work and started. Monday: - Samuele and Carl worked on refactoring the parts of genc that are responsible for the use of refcounting in the translation to make it possible to compile in a different GC. It turned out that refcounting is kind of deeply embedded into genc so it took a lot of time to introduce sensible hooks that other GCs could use. - Armin and Richard started to change genc so that it can handle the new external objects that Armin had to introduce to implement threading in PyPy. For now we have a simple GIL but it is not really deeply implanted in the interpreter so we should be able to change that later. After two days of hacking the were finished. Despite that it is still not possible to translate PyPy with threading because we are missing dictionaries with int keys on the RPython level. - Holger started preparing the release that is planned to happen during the sprint. Then he refactored our website so that all the websites are now generated statically and use a saner URL scheme. He then spent quite some time to make redirections from the old URLs work. See this mail for details: http://codespeak.net/pipermail/pypy-dev/2005q3/002285.html - Anders L. and Jacob worked on compliancy: they fixed various failing tests that are in some ways related to unicode. - Ludovic and Nik worked on making more parts of our compiler RPython. - Laura continued working on a docstring exctraction tool that she started to work on at the Gothenburg sprint. The tool takes the docstrings of types out of CPython and puts them in the relevant places of the PyPy codebase. With this we now have the correct docstrings for the builtin types. - Arre and Christian worked on compliancy. Christian feeled that the compliancy tests were taking to long on the interpreted PyPy so he tried making them run on the translated PyPy. This also involved problems since the compiler of the translated PyPy runs on applevel which means that running scripts involves a big startup overhead before it actually gets faster. Therefore he tried to introduce a hack so that the compiled PyPy uses CPython to translate its stuff and then run it. In the meantime Arre looked at the tests and started to reclassify core tests: he identified some tests that are in core but really shouldn't be there and the other way round. Tuesday: - Samuele, Carl, Richard, Holger and Armin continued to work on the stuff they started on Monday. - Nik worked on his 'Summer of Code'-project: he tried to move more parts of _sre to interplevel and make them annotatable. Additionally he worked on compliancy and fixed some tests. - Arre, Anders L., Jacob worked on compliancy Wednesday: This day was supposed to be devoted entirely to compliancy work. Some exceptions were allowed, though. - Richard and Eric (who arrived on Tuesday afternoon) started working on genllvm to bring it into a shape where it can translate PyPy better (they had already reached the translation goal a few days ago -- great work guys!). They kept finding some small bugs in LLVM. Still some external functions are missing as well and their goal was not to write them by hand but autogenerate them out of C code that already exists for genc. - Carl and Samuele used the garbage collection hooks they had introduced on Monday and Tuesday to make the Boehm garbage collector usable for genc as well (genllvm already uses it as their default (and only) collector). This was compared to factoring out refcounting relatively easy and was finished after a few hours and started working on compliancy afterwards. - Ludovic continued working on the compiler - Basically everybody else worked on compliancy tests. A lot of tests were recategorized to be non-core tests. Quite some failures are related to compiler issues. In the afternoon Holger, Samuele and later Armin started to dive into strange bugs of the compiler module that are related to nested scopes and that made some tests fail. These are the parts that I remember, corrections/additions welcome. Cheers, Carl Friedrich From Ben.Young at risk.sungard.com Fri Aug 26 13:10:35 2005 From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com) Date: Fri, 26 Aug 2005 12:10:35 +0100 Subject: [pypy-dev] Re: [pypy-svn] r16598 - pypy/dist/lib-python/modified-2.4.1 In-Reply-To: <20050826110700.BF5F627B45@code1.codespeak.net> Message-ID: Wasn't this discussed on Python-Dev recently? I think CPython has the same problem. Cheers, Ben pypy-svn-bounces at codespeak.net wrote on 26/08/2005 12:07:00: > Author: cfbolz > Date: Fri Aug 26 13:07:00 2005 > New Revision: 16598 > > Added: > pypy/dist/lib-python/modified-2.4.1/linecache.py > - copied, changed from r16593, pypy/dist/lib-python/2.4.1/linecache.py > Log: > For now, use Richard's idea to optimize linecache.py until we figure out why > readlines() is so slow in 'U'-mode files. > > _______________________________________________ > pypy-svn mailing list > pypy-svn at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-svn > From Ben.Young at risk.sungard.com Fri Aug 26 13:39:05 2005 From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com) Date: Fri, 26 Aug 2005 12:39:05 +0100 Subject: [pypy-dev] Release Message-ID: I know you are making a release soon, so I thought I'd send you this patch which fixes a couple of test issues. With this patch I get 14 test failures still, so I will attach the test log as well. Cheers, Ben -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: results.txt URL: From Ben.Young at risk.sungard.com Fri Aug 26 13:41:11 2005 From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com) Date: Fri, 26 Aug 2005 12:41:11 +0100 Subject: [pypy-dev] Release In-Reply-To: Message-ID: Oops, meant to attach diff file as well! pypy-dev-bounces at codespeak.net wrote on 26/08/2005 12:39:05: > I know you are making a release soon, so I thought I'd send you this patch > which fixes a couple of test issues. With this patch I get 14 test > failures still, so I will attach the test log as well. > > > > Cheers, > Ben[attachment "results.txt" deleted by Ben Young/Infinity] > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: diff.txt URL: From Ben.Young at risk.sungard.com Fri Aug 26 14:04:21 2005 From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com) Date: Fri, 26 Aug 2005 13:04:21 +0100 Subject: [pypy-dev] PyPy-dev Message-ID: Hi PyPy'rs, Are the sprinters reading pypy-dev at the moment? Cheers, Ben From Ben.Young at risk.sungard.com Fri Aug 26 14:11:29 2005 From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com) Date: Fri, 26 Aug 2005 13:11:29 +0100 Subject: [pypy-dev] PyPy-dev In-Reply-To: Message-ID: Thats alright! I was just checking I wasn't sending messages into the void! Cheers, Ben Beatrice During wrote on 26/08/2005 13:08:36: > Hi there > > On Fri, 26 Aug 2005 Ben.Young at risk.sungard.com wrote: > > > Hi PyPy'rs, > > > > Are the sprinters reading pypy-dev at the moment? > > > > Cheers, > > Ben > > People are quite caught up with the release and didnt react to your email > until now - sorry about that. They are looking at your patch right now so > you will probably hear something soon.. > > Cheers > > Bea > Beatrice D?ring Change Maker > Tel: 031- 7750940 J?rntorget 3 > Mobil: 0734- 22 89 06 413 04 G?teborg > E-post: bea at changemaker.nu www.changemaker.nu > > "Alla dessa m?sten och alldaglighet. > Allt detta som binder v?r verklighet > i bojor av skam och rep utav tv?ng. > Alla dessa kedjor som binder v?r s?ng. > Jag skall slita dem alla i sm?, sm? stycken > och m?jligtvis av resterna g?ra mig smycken." > - hemlig From lac at strakt.com Fri Aug 26 14:27:25 2005 From: lac at strakt.com (Laura Creighton) Date: Fri, 26 Aug 2005 14:27:25 +0200 Subject: [pypy-dev] PyPy-dev In-Reply-To: Message from Ben.Young@risk.sungard.com of "Fri, 26 Aug 2005 13:04:21 BST." References: Message-ID: <200508261227.j7QCRPpj032090@ratthing-b246.strakt.com> In a message of Fri, 26 Aug 2005 13:04:21 BST, Ben.Young at risk.sungard.com write s: >Hi PyPy'rs, > >Are the sprinters reading pypy-dev at the moment? > >Cheers, >Ben Some of us. And Arre (Anders Christr?m) is looking at your patch now. Laura From hpk at trillke.net Sun Aug 28 17:53:22 2005 From: hpk at trillke.net (holger krekel) Date: Sun, 28 Aug 2005 17:53:22 +0200 Subject: [pypy-dev] pypy-0.7.0 is out! Message-ID: <20050828155322.GM666@solar.trillke.net> pypy-0.7.0: first PyPy-generated Python Implementations ============================================================== What was once just an idea between a few people discussing on some nested mailing list thread and in a pub became reality ... the PyPy development team is happy to announce its first public release of a fully translatable self contained Python implementation. The 0.7 release showcases the results of our efforts in the last few months since the 0.6 preview release which have been partially funded by the European Union: - whole program type inference on our Python Interpreter implementation with full translation to two different machine-level targets: C and LLVM - a translation choice of using a refcounting or Boehm garbage collectors - the ability to translate with or without thread support - very complete language-level compliancy with CPython 2.4.1 What is PyPy (about)? ------------------------------------------------ PyPy is a MIT-licensed research-oriented reimplementation of Python written in Python itself, flexible and easy to experiment with. It translates itself to lower level languages. Our goals are to target a large variety of platforms, small and large, by providing a compilation toolsuite that can produce custom Python versions. Platform, Memory and Threading models are to become aspects of the translation process - as opposed to encoding low level details into a language implementation itself. Eventually, dynamic optimization techniques - implemented as another translation aspect - should become robust against language changes. Note that PyPy is mainly a research and development project and does not by itself focus on getting a production-ready Python implementation although we do hope and expect it to become a viable contender in that area sometime next year. Where to start? ----------------------------- Getting started: http://codespeak.net/pypy/dist/pypy/doc/getting-started.html PyPy Documentation: http://codespeak.net/pypy/dist/pypy/doc/ PyPy Homepage: http://codespeak.net/pypy/ The interpreter and object model implementations shipped with the 0.7 version can run on their own and implement the core language features of Python as of CPython 2.4. However, we still do not recommend using PyPy for anything else than for education, playing or research purposes. Ongoing work and near term goals --------------------------------- PyPy has been developed during approximately 15 coding sprints across Europe and the US. It continues to be a very dynamically and incrementally evolving project with many one-week meetings to follow. You are invited to consider coming to the next such meeting in Paris mid October 2005 where we intend to plan and head for an even more intense phase of the project involving building a JIT-Compiler and enabling unique features not found in other Python language implementations. PyPy has been a community effort from the start and it would not have got that far without the coding and feedback support from numerous people. Please feel free to give feedback and raise questions. contact points: http://codespeak.net/pypy/dist/pypy/doc/contact.html contributor list: http://codespeak.net/pypy/dist/pypy/doc/contributor.html have fun, the pypy team, of which here is a partial snapshot of mainly involved persons: Armin Rigo, Samuele Pedroni, Holger Krekel, Christian Tismer, Carl Friedrich Bolz, Michael Hudson, Eric van Riet Paap, Richard Emslie, Anders Chrigstroem, Anders Lehmann, Ludovic Aubry, Adrien Di Mascio, Niklaus Haldimann, Jacob Hallen, Bea During, Laura Creighton, and many contributors ... PyPy development and activities happen as an open source project and with the support of a consortium partially funded by a two year European Union IST research grant. Here is a list of the full partners of that consortium: Heinrich-Heine University (Germany), AB Strakt (Sweden) merlinux GmbH (Germany), tismerysoft GmbH(Germany) Logilab Paris (France), DFKI GmbH (Germany) ChangeMaker (Sweden), Impara (Germany) From arigo at tunes.org Sun Aug 28 19:11:16 2005 From: arigo at tunes.org (Armin Rigo) Date: Sun, 28 Aug 2005 19:11:16 +0200 Subject: [pypy-dev] Re: [pypy-svn] r16598 - pypy/dist/lib-python/modified-2.4.1 In-Reply-To: References: <20050826110700.BF5F627B45@code1.codespeak.net> Message-ID: <20050828171116.GA12276@code1.codespeak.net> Hi Ben, On Fri, Aug 26, 2005 at 12:10:35PM +0100, Ben.Young at risk.sungard.com wrote: > Wasn't this discussed on Python-Dev recently? I think CPython has the same > problem. > pypy/dist/lib-python/2.4.1/linecache.py > > Log: > > For now, use Richard's idea to optimize linecache.py until we figure out > > why readlines() is so slow in 'U'-mode files. I can't tell. I would be surprized if the problem is exactly the same: it seems that (at that point) if we replace f.readlines() with f.read().splitlines(True), things are seriously faster. This points to a problem in our lib/_sio.py which is exposed by our lib/_file.py. Meanwhile, Richard removed this linecache.py hack after he figured out one thing that killed performance: string slicing. Its implementation in stringobject.py was very old and completely broken performance-wise. A bientot, Armin From amp at singingwizard.org Mon Aug 29 00:15:58 2005 From: amp at singingwizard.org (Arthur Peters) Date: Sun, 28 Aug 2005 17:15:58 -0500 Subject: [pypy-dev] pypy-0.7.0 is out! In-Reply-To: <20050828155322.GM666@solar.trillke.net> Message-ID: I tried translating pypy on my AMD64 laptop and it translated fine, compilation did finally run the machine out of ram. The machine was somewhat usable during the transation as the python process was only using ~700 of 784MB of RAM (everything constantly swapping of course). It got a bit more rocky during the C code generation when the usage slowly went up by another 50MB. But I quit trying to code with it in the background and let it continue. After a while I checked and it had started to compile this continued, very slowly, for long while longer at one point the cc1 process was using over 800MB. The lesson here is that it isn't really posible to translate it on a 64-bit machine with a normal amount of RAM, yet. Also I got a bunch of warnings from the C compiler: "warning: cast to pointer from integer of different size". This sounds to my like the code assumes that the int is the same size as the pointer and it isn't on my machine. And I suspect the result would not have been runable because of it. The end of the story is that I was running out of RAM and swap and I tried to add a swap file and the kernel killed the cc1 process. BTW, The machine has in excess of 900MB of swap and my gnome session only uses a couple hundred MB, so the compilation must have use an /huge/ amount of memory to overflow the well over 1GB of available VM. After the cc1 process was killed I was able to switch back to the X vt and the graph viewer window had poped up and seemed to work. I figure you know or could have guessed this would happen, but I thought I'd send a report. Hope it is useful. -Arthur PS: BTW, I'm willing to try it again if it would help. This time I will kill everything unnessisary and use 2-3GB of swap ;-) From arigo at tunes.org Mon Aug 29 12:02:49 2005 From: arigo at tunes.org (Armin Rigo) Date: Mon, 29 Aug 2005 12:02:49 +0200 Subject: [pypy-dev] pypy-0.7.0 is out! In-Reply-To: References: <20050828155322.GM666@solar.trillke.net> Message-ID: <20050829100249.GA24581@code1.codespeak.net> Hi Arthur, On Sun, Aug 28, 2005 at 05:15:58PM -0500, Arthur Peters wrote: > I tried translating pypy on my AMD64 laptop and it translated fine, > compilation did finally run the machine out of ram. As it got as far as generating the .c source, you can try aborting the translation at this point and compiling the testing_1.c file manually with no translate_pypy.py left in the background. This also has the advantage that you can control which options you pass to gcc, so you can disable advanced optimizations. E.g. (fix the paths, of course): $ cd /tmp/usession-NN/testing_1/ $ gcc -c testing_1.c -o testing_1.o -I/usr/include/python2.4 \ -I/path/to/pypy/translator/c $ gcc -lpthread -lm testing_1.o -o pypy-c Armin From amp at singingwizard.org Mon Aug 29 14:26:24 2005 From: amp at singingwizard.org (Arthur Peters) Date: Mon, 29 Aug 2005 07:26:24 -0500 Subject: [pypy-dev] pypy-0.7.0 is out! In-Reply-To: <43124CAD.2030808@gmx.de> Message-ID: Thanks for your replies. I had forgoten -t-lowmem. I unloaded X and a few other things and tried it again. It translated without swapping. The python translator process finished holding 704M. I didn't notice it going any higher (but it may have). It used about 28 minutes of CPU time. I used the options "-t-lowmem -text" (-text because I'm not running under X). The cc1 process used in excess of 800M again. Would it be posible to emit a set of files instead of a single file? Then compile them seperately. Spliting it into 10 files would speed up compilation alot because it wouldn't swap. pypy-c starts and seems to work but is VERY slow. It spends a lot of time consuming 100% CPU and not accessing the disk. It also seems to be constantly allocating more RAM, also. Is this the issue related to slow module loading or something else? (help(theading) took almost 40 minutes, and failed with missing popen) Also it takes a couple seconds to respond after I hit enter no matter what the command is (even things like "x = 1"). I think implementing readline-like features for the interactive prompt would be very nice. At the very least a working backspace and command history. I tried llvm but it failed will downloading from codespeak.net/pypy/llvm-gcc.cgi (Strange problem for a compiler) because the ethernet cable had come loose from the machine. Overall I am very impressed and I can't wait for some new features to come out of this. The JIT will be neat, but I'm also interested in the thunk object space as a way of implementing a threading model in which each thread makes requests from other threads. The requests return immediately, but the returned object is a special object that is automatically replaced with the actual result when it is available. It seems like a thread safe version of the thunk object space would do quite nice for implementing this. (I learn about this idea from reading about Eifel but I've never actually programmed with it) Just some thoughts of my own. Not a request. Once the whole thing gets a bit more stable I'll start playing with implementing this kinda thing myself. (assuming I have some free time ;-) Good work, -Arthur PS: I'll remember the "compile it by hand later" trick, but it wasn't needed this time. On 8/28/2005, "Carl Friedrich Bolz" wrote: >Hi Arthur! > >Arthur Peters wrote: >> I tried translating pypy on my AMD64 laptop and it translated fine, >> compilation did finally run the machine out of ram. > >Damn. > >[snip] > >> PS: BTW, I'm willing to try it again if it would help. This time I will >> kill everything unnessisary and use 2-3GB of swap ;-) > >I don't know if you did that already but you might want to use -t-lowmem > as an option to translate_pypy which should reduce memory usage quite >a bit. > >Thanks for trying this out! > >Cheers, > >Carl Friedrich From hpk at trillke.net Mon Aug 29 16:19:18 2005 From: hpk at trillke.net (holger krekel) Date: Mon, 29 Aug 2005 16:19:18 +0200 Subject: [pypy-dev] pypy-0.7.0 is out! In-Reply-To: References: <43124CAD.2030808@gmx.de> Message-ID: <20050829141918.GS666@solar.trillke.net> Hello Arthur, On Mon, Aug 29, 2005 at 07:26 -0500, Arthur Peters wrote: > pypy-c starts and seems to work but is VERY slow. It spends a lot of time > consuming 100% CPU and not accessing the disk. It also seems to be > constantly allocating more RAM, also. Is this the issue related to slow > module loading or something else? (help(theading) took almost 40 > minutes, and failed with missing popen) Also it takes a couple seconds > to respond after I hit enter no matter what the command is (even things > like "x = 1"). The main issue with the interactive command line being slow is that the bytecode compiler is running completely at application level. You will note that if you issue e.g. "x=range(10000)" that it takes the same time as "x=range(1000)". The time is mostly consumed by compiling the expression. The actual bytecode interpretation is reasonably fast. One major area for the current pypy code base is getting the compiler (and ST->AST transformation) to be statically translateable. This should tremendously increase interactive performance and get a better feel. > I think implementing readline-like features for the interactive prompt > would be very nice. At the very least a working backspace and command > history. Yes, i agree. But our current translation driving needs some serious cleanup before we can tackle to having flexible ways to be able to optionally link and interface to libraries like readline. > I tried llvm but it failed will downloading from > codespeak.net/pypy/llvm-gcc.cgi (Strange problem for a compiler) because > the ethernet cable had come loose from the machine. Problem is that we require LLVM-cvs and the gcc-frontend in particular and that is hard to install. Therefore we do the indirection via codespeak which has a recent version installed and does the actual compilation. Not elegant and a bit suprising but convenient nevertheless. > Overall I am very impressed and I can't wait for some new features to > come out of this. The JIT will be neat, but I'm also interested in the > thunk object space as a way of implementing a threading model in which > each thread makes requests from other threads. The requests return > immediately, but the returned object is a special object that is > automatically replaced with the actual result when it is available. It > seems like a thread safe version of the thunk object space would do > quite nice for implementing this. (I learn about this idea from reading > about Eifel but I've never actually programmed with it) makes all sense to me and i am interested in this as well. Certainly a nice sprint topic at some point ... cheers, holger From Ben.Young at risk.sungard.com Tue Aug 30 11:31:43 2005 From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com) Date: Tue, 30 Aug 2005 10:31:43 +0100 Subject: [pypy-dev] Release Message-ID: Congratulations everyone one the release! It looks really good! So what's the next priority? Speed or more customisability (or both!)? Cheers, Ben From hpk at trillke.net Tue Aug 30 11:56:01 2005 From: hpk at trillke.net (holger krekel) Date: Tue, 30 Aug 2005 11:56:01 +0200 Subject: some next steps (was: Re: [pypy-dev] Release) In-Reply-To: References: Message-ID: <20050830095601.GL666@solar.trillke.net> Hi Ben, hi all, On Tue, Aug 30, 2005 at 10:31 +0100, Ben.Young at risk.sungard.com wrote: > Congratulations everyone one the release! It looks really good! thanks, also for your constant support! > So what's the next priority? Speed or more customisability (or both!)? we had a brief discussion at the end of the sprint and apart from working on the bytecode compiler (which makes the interactive speed appear so slow) we intend to cleanup translation driving and various other areas before heading off to the next phases of the project. Also we currently plan the next sprint in Paris (10th-17th October) which we should announce soon. It's quite likely we are discussing/starting on the next efforts there regarding JIT compilation and massive multithreading and what not. There also is the ongoing effort of integrating Carl Friedrich's GC code into the actual translated PyPy and improving flexilibity around threading, completing some crucial external functions (like os.listdir) and whatnot. Personally, i hope i will find some time to seriously improve the testing framework on various levels. With PyPy, we begin to have lots of options and variants in testing our own code base, the standard python library's tests as well as testing translation targets and variants. I'd like to implement an approach that allows completely peer-driven testing and sending of reports to a central site where they can be queried according to os/processor/python. I intend to implement this in a PyPy neutral manner so that the numerous other users of py.test can reuse our efforts for their projects. Additionally, i'd like to have tests become interactively distributable to multiple machines (listed via ssh-account login information) from a single (possibly modified) working copy. Also, for the EU side of things some of us will need to invest time into reporting and writing papers. We intend to keep as much of that work reusable on the website as we have no inclination to just produce dead paper. Last but not least we are still looking for sprint places end of this and the whole next year. There appear to be possibilities in Istanbul (Turkey), Bern (Switzerland) and Romania but none of these are concrete at this point. It would also already be good to know if there is interest in doing a PyPy sprint at Pycon US in the next year. cheers, holger From Ben.Young at risk.sungard.com Tue Aug 30 13:02:55 2005 From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com) Date: Tue, 30 Aug 2005 12:02:55 +0100 Subject: some next steps (was: Re: [pypy-dev] Release) In-Reply-To: <20050830095601.GL666@solar.trillke.net> Message-ID: Thanks for the reply Holger hpk at trillke.net (holger krekel) wrote on 30/08/2005 10:56:01: > Hi Ben, hi all, > > On Tue, Aug 30, 2005 at 10:31 +0100, Ben.Young at risk.sungard.com wrote: > > Congratulations everyone one the release! It looks really good! > > thanks, also for your constant support! > > > So what's the next priority? Speed or more customisability (or both!)? > > we had a brief discussion at the end of the sprint and apart > from working on the bytecode compiler (which makes the interactive > speed appear so slow) we intend to cleanup translation driving > and various other areas before heading off to the next phases > of the project. Also we currently plan the next sprint in Paris > (10th-17th October) which we should announce soon. It's quite > likely we are discussing/starting on the next efforts there > regarding JIT compilation and massive multithreading and > what not. > > There also is the ongoing effort of integrating Carl Friedrich's > GC code into the actual translated PyPy and improving flexilibity > around threading, completing some crucial external functions > (like os.listdir) and whatnot. > Will there eventually be a way for existing c extension modules to talk to the generated pypy? Or will people have to reimplement their extensions (perhaps using a c-types style notation). I guess the hard bit is making it cross-backend compatiable (for instance the way ironpython/jython can both automatically see the platform objects) > Personally, i hope i will find some time to seriously improve > the testing framework on various levels. With PyPy, we begin to > have lots of options and variants in testing our own code > base, the standard python library's tests as well as testing > translation targets and variants. I'd like to implement an > approach that allows completely peer-driven testing and > sending of reports to a central site where they can be queried > according to os/processor/python. I intend to implement this > in a PyPy neutral manner so that the numerous other users of > py.test can reuse our efforts for their projects. Additionally, > i'd like to have tests become interactively distributable > to multiple machines (listed via ssh-account login information) > from a single (possibly modified) working copy. > Have you come up with any solutions to make the annotation/translation process a bit less fragile, as it appears a small fix somewhere in the code can accidently produce huge amounts of confusion in the annotator. Perhaps some "checkpoints" in places in the code, where if an object doesn't have a particular annotation then we stop at that point? > Also, for the EU side of things some of us will need to invest > time into reporting and writing papers. We intend to keep > as much of that work reusable on the website as we have no > inclination to just produce dead paper. > > Last but not least we are still looking for sprint places end > of this and the whole next year. There appear to be > possibilities in Istanbul (Turkey), Bern (Switzerland) and > Romania but none of these are concrete at this point. It would > also already be good to know if there is interest in doing a PyPy > sprint at Pycon US in the next year. Thanks for your patience in my incessant questioning! Cheers, Ben > > cheers, > > holger > From gherman at darwin.in-berlin.de Tue Aug 30 14:45:25 2005 From: gherman at darwin.in-berlin.de (Dinu Gherman) Date: Tue, 30 Aug 2005 14:45:25 +0200 Subject: some next steps (was: Re: [pypy-dev] Release) In-Reply-To: <20050830095601.GL666@solar.trillke.net> References: <20050830095601.GL666@solar.trillke.net> Message-ID: holger krekel: > Personally, i hope i will find some time to seriously improve > the testing framework on various levels. With PyPy, we begin to > have lots of options and variants in testing our own code > base, the standard python library's tests as well as testing > translation targets and variants. Being a fan of testing I'd like to suggest conducting some compara- tive tests between CPython and PyPy, as well. At least I find stuff like the following pretty "interesting". It's about using re for splitting strings at very large substrings of some minimum length (something I just used for processing AIFF audio files, the code here is slightly simpler): Python 2.4 (#1, Feb 7 2005, 21:41:21) [GCC 3.3 20030304 (Apple Computer, Inc. build 1640)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> >>> import re >>> n = 'o' >>> l = int(1e5) >>> inp = "012" + n*l + "abc" + n*l + "xyz" >>> exp = ["012", "abc", "xyz"] >>> res = re.split(n+'{%d,%d}'%(l, l), inp) >>> exp == res False vs. PyPy 0.7.0 in StdObjSpace on top of Python 2.4 (startupttime: 7.99 secs) >>> >>>> import re >>>> n = 'o' >>>> l = int(1e5) >>>> inp = "012" + n*l + "abc" + n*l + "xyz" >>>> exp = ["012", "abc", "xyz"] >>>> res = re.split(n+'{%d,%d}'%(l, l), inp) >>>> exp == res True There could be workarounds for this particular case, but the point is that PyPy can be "correct" in places where CPython is not (here prob- ably because of limitations of the re machinery). And because they'd fail you would not expect to find such test cases in the "normal" test suites... In a way it's like saying "Look ma, I might be still bloody slow, but eventually I'm getting to the right place!" Dinu From rodsenra at gpr.com.br Tue Aug 30 15:04:29 2005 From: rodsenra at gpr.com.br (Rodrigo Dias Arruda Senra) Date: Tue, 30 Aug 2005 10:04:29 -0300 Subject: some next steps (was: Re: [pypy-dev] Release) In-Reply-To: References: <20050830095601.GL666@solar.trillke.net> Message-ID: <20050830100429.63047376@localhost.localdomain> On Tue, 30 Aug 2005 14:45:25 +0200 Dinu Gherman wrote: > Being a fan of testing I'd like to suggest conducting some compara- > tive tests between CPython and PyPy, as well. At least I find stuff > like the following pretty "interesting". It's about using re for > splitting strings at very large substrings of some minimum length > (something I just used for processing AIFF audio files, the code > here is slightly simpler): > > Python 2.4 (#1, Feb 7 2005, 21:41:21) > [GCC 3.3 20030304 (Apple Computer, Inc. build 1640)] on darwin > Type "help", "copyright", "credits" or "license" for more > information. > >>> > >>> import re > >>> n = 'o' > >>> l = int(1e5) > >>> inp = "012" + n*l + "abc" + n*l + "xyz" > >>> exp = ["012", "abc", "xyz"] > >>> res = re.split(n+'{%d,%d}'%(l, l), inp) > >>> exp == res > False Dinu, that scared me deeply! So I stopped everything and tryied it. Python 2.3.5 (#1, Aug 11 2005, 10:10:19) [GCC 3.3.5 (Debian 1:3.3.5-8ubuntu2)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import re >>> n = 'o' >>> l = int(1e5) >>> inp = "012" + n*l + "abc" + n*l + "xyz" >>> exp = ["012", "abc", "xyz"] >>> res = re.split(n+'{%d,%d}'%(l, l), inp) >>> exp == res False Python 2.4.1 (#2, Mar 30 2005, 21:51:10) [GCC 3.3.5 (Debian 1:3.3.5-8ubuntu2)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import re >>> n = 'o' >>> l = int(1e5) >>> inp = "012" + n*l + "abc" + n*l + "xyz" >>> exp = ["012", "abc", "xyz"] >>> res = re.split(n+'{%d,%d}'%(l, l), inp) >>> exp == res True So, this seems to be a bug fixed in CPython after 2.4 > There could be workarounds for this particular case, but the point is > that PyPy can be "correct" in places where CPython is not (here prob- > ably because of limitations of the re machinery). And because they'd > fail you would not expect to find such test cases in the "normal" > test suites... I'm a *big fan* of the pypy-team and pypy-itself. But I do not think __this particular case__ is fair enough to advertise PyPy getting it wright where CPython got it wrong. best regards, Rod Senra rsenra _at_ acm.org From gherman at darwin.in-berlin.de Tue Aug 30 17:13:49 2005 From: gherman at darwin.in-berlin.de (Dinu Gherman) Date: Tue, 30 Aug 2005 17:13:49 +0200 Subject: some next steps (was: Re: [pypy-dev] Release) In-Reply-To: <20050830100429.63047376@localhost.localdomain> References: <20050830095601.GL666@solar.trillke.net> <20050830100429.63047376@localhost.localdomain> Message-ID: <9a238f03cda7e06828f788cab67e2344@darwin.in-berlin.de> Rodrigo Dias Arruda Senra: > So, this seems to be a bug fixed in CPython after 2.4 Phew - good news! ;-) > I'm a *big fan* of the pypy-team and pypy-itself. But I do not think > __this particular case__ is fair enough to advertise PyPy getting it > wright where CPython got it wrong. Quite true! In fact, both should implement the same language! And from that point of view it sounds strange to suggest com- parative tests, except maybe for finding implementation buglets like this one, maybe especially in the standard library... Regards, Dinu From mwh at python.net Tue Aug 30 20:11:48 2005 From: mwh at python.net (Michael Hudson) Date: Tue, 30 Aug 2005 19:11:48 +0100 Subject: [pypy-dev] Re: some next steps References: <20050830095601.GL666@solar.trillke.net> <20050830100429.63047376@localhost.localdomain> <9a238f03cda7e06828f788cab67e2344@darwin.in-berlin.de> Message-ID: <2mk6i3pa57.fsf@starship.python.net> Dinu Gherman writes: > Rodrigo Dias Arruda Senra: > >> So, this seems to be a bug fixed in CPython after 2.4 > > Phew - good news! ;-) > >> I'm a *big fan* of the pypy-team and pypy-itself. But I do not think >> __this particular case__ is fair enough to advertise PyPy getting it >> wright where CPython got it wrong. > > Quite true! In fact, both should implement the same language! > And from that point of view it sounds strange to suggest com- > parative tests, except maybe for finding implementation buglets > like this one, maybe especially in the standard library... Implementing PyPy has found more than one strange bug in the CPython implementation, to be sure... Cheers, mwh -- > Well, as an American citizen I hope that the EU tells the MPAA > and RIAA to shove it where the Sun don't shine. Actually they already did. Only first they bent over and dropped their trousers. -- Shmuel (Seymour J.) Metz & Toni Lassila, asr From sanxiyn at gmail.com Wed Aug 31 03:49:42 2005 From: sanxiyn at gmail.com (Sanghyeon Seo) Date: Wed, 31 Aug 2005 10:49:42 +0900 Subject: some next steps (was: Re: [pypy-dev] Release) In-Reply-To: <20050830095601.GL666@solar.trillke.net> References: <20050830095601.GL666@solar.trillke.net> Message-ID: <5b024817050830184949246ac5@mail.gmail.com> On 8/30/05, holger krekel wrote: > > Personally, i hope i will find some time to seriously improve > the testing framework on various levels. With PyPy, we begin to > have lots of options and variants in testing our own code > base, the standard python library's tests as well as testing > translation targets and variants. I'd like to implement an > approach that allows completely peer-driven testing and > sending of reports to a central site where they can be queried > according to os/processor/python. I intend to implement this > in a PyPy neutral manner so that the numerous other users of > py.test can reuse our efforts for their projects. Additionally, > i'd like to have tests become interactively distributable > to multiple machines (listed via ssh-account login information) > from a single (possibly modified) working copy. This reminds me of BuildBot: http://buildbot.sourceforge.net/ . Does it look relevant? Seo Sanghyeon From hpk at trillke.net Wed Aug 31 08:14:42 2005 From: hpk at trillke.net (holger krekel) Date: Wed, 31 Aug 2005 08:14:42 +0200 Subject: some next steps (was: Re: [pypy-dev] Release) In-Reply-To: <5b024817050830184949246ac5@mail.gmail.com> References: <20050830095601.GL666@solar.trillke.net> <5b024817050830184949246ac5@mail.gmail.com> Message-ID: <20050831061442.GS666@solar.trillke.net> Hey Seo! On Wed, Aug 31, 2005 at 10:49 +0900, Sanghyeon Seo wrote: > On 8/30/05, holger krekel wrote: > > > > Personally, i hope i will find some time to seriously improve > > the testing framework on various levels. With PyPy, we begin to > > have lots of options and variants in testing our own code > > base, the standard python library's tests as well as testing > > translation targets and variants. I'd like to implement an > > approach that allows completely peer-driven testing and > > sending of reports to a central site where they can be queried > > according to os/processor/python. I intend to implement this > > in a PyPy neutral manner so that the numerous other users of > > py.test can reuse our efforts for their projects. Additionally, > > i'd like to have tests become interactively distributable > > to multiple machines (listed via ssh-account login information) > > from a single (possibly modified) working copy. > > This reminds me of BuildBot: http://buildbot.sourceforge.net/ . Does it look > relevant? I know of buildbot but i think it has a different focus. It works with a central installation and it targets more general build processes whereas we would probably focus on detailed python testing and have it peer-driven so that everyone can contribute to gather information (which does obviously not exclude having servers which do it on a regular basis via cron or are triggered by svn-notification emails). cheers, holger From Ben.Young at risk.sungard.com Wed Aug 31 11:25:05 2005 From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com) Date: Wed, 31 Aug 2005 10:25:05 +0100 Subject: [pypy-dev] irc logs dead? Message-ID: Hi Everyone, The archived irc log server appears to be dead at the moment (and has been for a couple of days). Does anyone know anything about this, or is it out of your hands? Cheers, Ben From cfbolz at gmx.de Wed Aug 31 12:24:09 2005 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Wed, 31 Aug 2005 12:24:09 +0200 (MEST) Subject: =?ISO-8859-1?Q?Re:_[pypy-dev]_irc_logs_dead=3F?= References: Message-ID: <7524.1125483849@www32.gmx.net> Hi Ben, > The archived irc log server appears to be dead at the moment (and has been > for a couple of days). Does anyone know anything about this, or is it out > of your hands? Christian told me that the hoster of nimrod.terra-link had some sort of bandwidth problem, therefore the site won't be reachable before the 1st of September. The bots is still active though, we just can't reach the results. Cheers, Carl Friedrich From hpk at trillke.net Wed Aug 31 20:29:45 2005 From: hpk at trillke.net (holger krekel) Date: Wed, 31 Aug 2005 20:29:45 +0200 Subject: [pypy-dev] pypy-sync back again thursday 1pm Message-ID: <20050831182945.GF666@solar.trillke.net> Hi again, so the heidelberg sprint wears off and just when you thought things were getting relaxed here is the next pypy-sync meeting invitation :-) ============================================= pypy-sync developer meeting 1st September ============================================= Time & location: 1pm (30 minutes) at #pypy-sync Regular Topics ==================== - activity reports (3 prepared lines of info, LAST/NEXT/BLOCKERS). - resolve conflicts/blockers Annotated Topics of the week ================================= Recap of discussed development areas until October ------------------------------------------------------- At the meeting at the heidelberg sprint we listed a few areas that deserve developing focus in the next time: - bringing the compiler to good state, Arre and Samuele will try to help and discuss with Ludovic. The idea is to have the compiler effort almost done before the October Paris sprint and get rid of the various temporary hacks we have around. - cleanups in various areas should be tackled, in particular - the translation driving (translate_pypy.py and friends) and - improved test/reporting support. Anything else that we want to list explicitely? Like lifting the number of Pystones? :-) Paris Sprint announcement ----------------------------------------------- We need to check what the status of the Paris sprint announcement is as it should go out this week. Topics as discussed at the end of the heidelberg sprint: - threading and GC - refactoring/translation features - start on JIT/optimizations/stackless Let's reconfirm/extend this and then make sure that we have it out by the weekend. Preparation/Moderation Next pypy-sync meeting -------------------------------------------------- There is the idea to rotate pypy-sync meeting preparation and moderation between developers. So who would like to do some preparation and the moderation next time? It would be great if we had at least 3 people who shift between them unless we just want to do alphabetical order :-) From hpk at trillke.net Wed Aug 31 20:45:04 2005 From: hpk at trillke.net (holger krekel) Date: Wed, 31 Aug 2005 20:45:04 +0200 Subject: [pypy-dev] refactoring Module discovery/implementation? Message-ID: <20050831184504.GG666@solar.trillke.net> Hi pypy-dev, I'd like to suggest heading towards making PyPy's mixed C-ported modules more self-contained and not have it's implementation and initilization spread over several directories and code areas. IMO it's good that CPython has basically one C file that describes a Module's implementation. With Python we could make it a package so that we can still separate information like interp-level, app-level and low-level glue code but keep things more together and have it picked up via some more plugin-based scheme. The package scheme (probably based on the current mixed modules approach) should have a runtime interface and try to stay indepedent from the inner PyPy developments like where exactly things at low level are located. With "runtime interface" i mean that PyPy would automatically discover all modules in a given plugin/module directory by importing and querying them to get information the provided module, a description (docstring) and e.g. which ll-functions are provided and special annotations and what not. Oh, and "py.py --listmodules" should provide a full view of available modules (and modules included by the current defaults) so that you can pass that to --usemodules="x,y,z". This involves a bit of API design that we haven't done much for our internal stuff yet but i think it's about time to provide some nicer APIs and separate things. What do you think? A sprint topic? First write documentation and then implement? cheers, holger From bea at netg.se Fri Aug 26 14:08:36 2005 From: bea at netg.se (Beatrice During) Date: Fri, 26 Aug 2005 14:08:36 +0200 (CEST) Subject: [pypy-dev] PyPy-dev In-Reply-To: References: Message-ID: Hi there On Fri, 26 Aug 2005 Ben.Young at risk.sungard.com wrote: > Hi PyPy'rs, > > Are the sprinters reading pypy-dev at the moment? > > Cheers, > Ben People are quite caught up with the release and didnt react to your email until now - sorry about that. They are looking at your patch right now so you will probably hear something soon.. Cheers Bea Beatrice D?ring Change Maker Tel: 031- 7750940 J?rntorget 3 Mobil: 0734- 22 89 06 413 04 G?teborg E-post: bea at changemaker.nu www.changemaker.nu "Alla dessa m?sten och alldaglighet. Allt detta som binder v?r verklighet i bojor av skam och rep utav tv?ng. Alla dessa kedjor som binder v?r s?ng. Jag skall slita dem alla i sm?, sm? stycken och m?jligtvis av resterna g?ra mig smycken." - hemlig