From arigo at tunes.org Thu Sep 1 11:41:26 2005 From: arigo at tunes.org (Armin Rigo) Date: Thu, 1 Sep 2005 11:41:26 +0200 Subject: [pypy-dev] refactoring Module discovery/implementation? In-Reply-To: <20050831184504.GG666@solar.trillke.net> References: <20050831184504.GG666@solar.trillke.net> Message-ID: <20050901094126.GA1217@code1.codespeak.net> Hi Holger, On Wed, Aug 31, 2005 at 08:45:04PM +0200, holger krekel wrote: > I'd like to suggest heading towards making PyPy's mixed > C-ported modules more self-contained and not have it's > implementation and initilization spread over several > directories and code areas. The 'thread' module is pretty much self-contained in this respect (I tried to keep it this way as much as possible), so I would like to hear more specifically what you think is still missing. The structure is: pypy/module/thread/ __init__.py interface app_*.py app-level stuff os_*.py interp-level stuff ('os' means using os threads) test/ tests for the above rpython/ exttable.py table of external functions (guides both the annotator and the rtyper) ll_thread.py the low-level functions (similar to rpython/module/ll_*.py) test/ rpython-level tests of external functions There is nothing at all specific to the thread module in pypy/annotator or in pypy/rpython. However, for now, there is some C support code in pypy/translator/c/src/*.h; I didn't know exactly if it is a good idea to put even that in the pypy/module/thread package. Maybe it is. A bientot, Armin. From lac at strakt.com Fri Sep 2 20:07:38 2005 From: lac at strakt.com (Laura Creighton) Date: Fri, 2 Sep 2005 20:07:38 +0200 Subject: [pypy-dev] PyCON draft CFP has a proposed submission deadline of Oct 31. Message-ID: <200509021807.j82I7c78007186@theraft.strakt.com> Just so nobody is surprised. See: http://mail.python.org/mailman/private/pycon-organizers/2005-September/004149.html Laura From hpk at trillke.net Fri Sep 2 20:24:18 2005 From: hpk at trillke.net (holger krekel) Date: Fri, 2 Sep 2005 20:24:18 +0200 Subject: [pypy-dev] PyCON draft CFP has a proposed submission deadline of Oct 31. In-Reply-To: <200509021807.j82I7c78007186@theraft.strakt.com> References: <200509021807.j82I7c78007186@theraft.strakt.com> Message-ID: <20050902182418.GU666@solar.trillke.net> On Fri, Sep 02, 2005 at 20:07 +0200, Laura Creighton wrote: > Just so nobody is surprised. > See: http://mail.python.org/mailman/private/pycon-organizers/2005-September/004149.html you are pointing to a private mailing list archive. Is the information already meant for publishing? holger From lac at strakt.com Fri Sep 2 20:32:51 2005 From: lac at strakt.com (Laura Creighton) Date: Fri, 02 Sep 2005 20:32:51 +0200 Subject: [pypy-dev] PyCON draft CFP has a proposed submission deadline of Oct 31. In-Reply-To: Message from hpk@trillke.net (holger krekel) of "Fri, 02 Sep 2005 20:24:18 +0200." <20050902182418.GU666@solar.trillke.net> References: <200509021807.j82I7c78007186@theraft.strakt.com> <20050902182418.GU666@solar.trillke.net> Message-ID: <200509021832.j82IWpDJ007879@theraft.strakt.com> In a message of Fri, 02 Sep 2005 20:24:18 +0200, holger krekel writes: >On Fri, Sep 02, 2005 at 20:07 +0200, Laura Creighton wrote: >> Just so nobody is surprised. >> See: http://mail.python.org/mailman/private/pycon-organizers/2005-Septe >mber/004149.html > >you are pointing to a private mailing list archive. >Is the information already meant for publishing? > > holger Not yet. Want me to jump up and down and say 'Oct 31 is too soon?'? Otherwise I can easily see it taking them a week or 2 to decide on the wording of a bunch of other things and still keep the Oct 31 deadline. Laura From hpk at trillke.net Fri Sep 2 20:43:45 2005 From: hpk at trillke.net (holger krekel) Date: Fri, 2 Sep 2005 20:43:45 +0200 Subject: [pypy-dev] PyCON draft CFP has a proposed submission deadline of Oct 31. In-Reply-To: <200509021832.j82IWpDJ007879@theraft.strakt.com> References: <200509021807.j82I7c78007186@theraft.strakt.com> <20050902182418.GU666@solar.trillke.net> <200509021832.j82IWpDJ007879@theraft.strakt.com> Message-ID: <20050902184345.GW666@solar.trillke.net> On Fri, Sep 02, 2005 at 20:32 +0200, Laura Creighton wrote: > In a message of Fri, 02 Sep 2005 20:24:18 +0200, holger krekel writes: > >On Fri, Sep 02, 2005 at 20:07 +0200, Laura Creighton wrote: > >> Just so nobody is surprised. > >> See: http://mail.python.org/mailman/private/pycon-organizers/2005-Septe > >mber/004149.html > > > >you are pointing to a private mailing list archive. > >Is the information already meant for publishing? > > Not yet. Want me to jump up and down and say ... I don't know the background nor the reasoning for the date setting and i don't have an opinion on this. holger From hpk at trillke.net Fri Sep 2 23:42:18 2005 From: hpk at trillke.net (holger krekel) Date: Fri, 2 Sep 2005 23:42:18 +0200 Subject: [pypy-dev] refactoring Module discovery/implementation? In-Reply-To: <20050901094126.GA1217@code1.codespeak.net> References: <20050831184504.GG666@solar.trillke.net> <20050901094126.GA1217@code1.codespeak.net> Message-ID: <20050902214218.GB666@solar.trillke.net> Hi Armin, On Thu, Sep 01, 2005 at 11:41 +0200, Armin Rigo wrote: > On Wed, Aug 31, 2005 at 08:45:04PM +0200, holger krekel wrote: > > I'd like to suggest heading towards making PyPy's mixed > > C-ported modules more self-contained and not have it's > > implementation and initilization spread over several > > directories and code areas. > > The 'thread' module is pretty much self-contained in this respect (I > tried to keep it this way as much as possible), so I would like to hear > more specifically what you think is still missing. The structure is: > > pypy/module/thread/ > __init__.py interface > app_*.py app-level stuff > os_*.py interp-level stuff ('os' means using os threads) > test/ tests for the above > rpython/ > exttable.py table of external functions (guides both the > annotator and the rtyper) > ll_thread.py the low-level functions (similar to > rpython/module/ll_*.py) > test/ rpython-level tests of external functions That looks good indeed but we are not using similar structures for most of the other mixed modules. Moreover, i am more interested in the "runtime" interface and decoupling module implementations from PyPy internals. For example, we could import all module/X's at root level and query the package (or plain module, even) for all neccessary info like lldefs, annotation info for external functions, etc. And the module files themselves should use a resonably stable and documented way to access the neccessary lltypes and friends. The idea is a bit to allow more people to work on porting C-modules conveniently and rather self-containedly (with good error messages for annotations that conflict with provided info in the exttable for example). cheers, holger From hpk at trillke.net Mon Sep 5 08:03:47 2005 From: hpk at trillke.net (holger krekel) Date: Mon, 5 Sep 2005 08:03:47 +0200 Subject: [pypy-dev] Re: [pypy-svn] r17215 - pypy/dist/pypy/rpython In-Reply-To: <20050905003250.BF42C27B47@code1.codespeak.net> References: <20050905003250.BF42C27B47@code1.codespeak.net> Message-ID: <20050905060347.GK666@solar.trillke.net> Hi Christian, hi all, On Mon, Sep 05, 2005 at 02:32 +0200, tismer at codespeak.net wrote: > Modified: pypy/dist/pypy/rpython/rlist.py > ============================================================================== > --- pypy/dist/pypy/rpython/rlist.py (original) > +++ pypy/dist/pypy/rpython/rlist.py Mon Sep 5 02:32:49 2005 > @@ -393,8 +393,10 @@ > _ll_list_resize(l, length+1) > i = length > items = l.items > + i1 = i+1 > while i > 0: > - items[i] = items[i+1] > + items[i] = items[i1] > + i1 = i > i -= 1 > items[0] = newitem this and this ... > @@ -403,8 +405,10 @@ > _ll_list_resize(l, length+1) > items = l.items > i = length > + i1 = i+1 > while i > index: > - items[i] = items[i+1] > + items[i] = items[i1] > + i1 = i > i -= 1 > items[i] = newitem ... are such optimizations measurably effective at all? I would think that such low-level details are handled by the backend or, if not, that we could write a generic transformation that does that for us. Or am i missing something? cheers, holger From marius at pov.lt Mon Sep 5 11:50:43 2005 From: marius at pov.lt (Marius Gedminas) Date: Mon, 5 Sep 2005 12:50:43 +0300 Subject: some next steps (was: Re: [pypy-dev] Release) In-Reply-To: <20050831061442.GS666@solar.trillke.net> References: <20050830095601.GL666@solar.trillke.net> <5b024817050830184949246ac5@mail.gmail.com> <20050831061442.GS666@solar.trillke.net> Message-ID: <20050905095043.GB25287@fridge.pov.lt> On Wed, Aug 31, 2005 at 08:14:42AM +0200, holger krekel wrote: > On Wed, Aug 31, 2005 at 10:49 +0900, Sanghyeon Seo wrote: > > On 8/30/05, holger krekel wrote: > > > Personally, i hope i will find some time to seriously improve > > > the testing framework on various levels. With PyPy, we begin to > > > have lots of options and variants in testing our own code > > > base, the standard python library's tests as well as testing > > > translation targets and variants. I'd like to implement an > > > approach that allows completely peer-driven testing and > > > sending of reports to a central site where they can be queried > > > according to os/processor/python. I intend to implement this > > > in a PyPy neutral manner so that the numerous other users of > > > py.test can reuse our efforts for their projects. > > > Additionally, > > > i'd like to have tests become interactively distributable > > > to multiple machines (listed via ssh-account login information) > > > from a single (possibly modified) working copy. Yay! I've been hacking on something like this recently. > > This reminds me of BuildBot: http://buildbot.sourceforge.net/ . Does it look > > relevant? > > I know of buildbot but i think it has a different focus. > It works with a central installation and it targets more general > build processes whereas we would probably focus on detailed python > testing and have it peer-driven so that everyone can contribute to > gather information (which does obviously not exclude having servers > which do it on a regular basis via cron or are triggered by > svn-notification emails). Buildbot is a nice tool if you want to run a test suite automatically (perhaps on several machines with different hardware/software configurations) after every svn checkin. There's one master server that collects and displays results, watches for checkins and tells slaves to start the build when something changes. Anyone can run a buildbot slave on their own machine, if you give them a username and password for connecting to the master. Or do you want something that is more ad-hoc (e.g. a developer downloads pypy, runs the test suite, and sends the test log by email)? Marius Gedminas -- I code in vi because I don't want to learn another OS. :) -- Robert Love From ludal at logilab.fr Mon Sep 5 12:15:46 2005 From: ludal at logilab.fr (Ludovic Aubry) Date: Mon, 05 Sep 2005 12:15:46 +0200 Subject: [pypy-dev] Re: [pypy-svn] r17217 - pypy/dist/pypy/interpreter/astcompiler In-Reply-To: <20050905083024.33A5027B46@code1.codespeak.net> References: <20050905083024.33A5027B46@code1.codespeak.net> Message-ID: <1125915346.5471.5.camel@octans.logilab.fr> Please make sure you modify astgen.py at the same time. ast.py is generated by astgen.py which is a modified version of the original CPython generator. On Mon, 2005-09-05 at 10:30 +0200, pedronis at codespeak.net wrote: > Author: pedronis > Date: Mon Sep 5 10:30:22 2005 > New Revision: 17217 > > Modified: > pypy/dist/pypy/interpreter/astcompiler/ast.py > Log: > some getChildNodes were still returning tuples > > > > Modified: pypy/dist/pypy/interpreter/astcompiler/ast.py > ============================================================================== > --- pypy/dist/pypy/interpreter/astcompiler/ast.py (original) > +++ pypy/dist/pypy/interpreter/astcompiler/ast.py Mon Sep 5 10:30:22 2005 > @@ -332,7 +332,7 @@ > return () > > def getChildNodes(self): > - return () > + return [] > > def __repr__(self): > return "Break()" > @@ -449,7 +449,7 @@ > return () > > def getChildNodes(self): > - return () > + return [] > > def __repr__(self): > return "Continue()" > @@ -538,7 +538,7 @@ > return () > > def getChildNodes(self): > - return () > + return [] > > def __repr__(self): > return "Ellipsis()" > @@ -1107,7 +1107,7 @@ > return () > > def getChildNodes(self): > - return () > + return [] > > def __repr__(self): > return "NoneConst()" > @@ -1176,7 +1176,7 @@ > return () > > def getChildNodes(self): > - return () > + return [] > > def __repr__(self): > return "Pass()" > _______________________________________________ > pypy-svn mailing list > pypy-svn at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-svn From ludal at logilab.fr Mon Sep 5 17:00:30 2005 From: ludal at logilab.fr (Ludovic Aubry) Date: Mon, 05 Sep 2005 17:00:30 +0200 Subject: [pypy-dev] ANN: PyPy Sprint in Paris October 10th to 16th Message-ID: <1125932431.6101.10.camel@octans.logilab.fr> Here is the official sprint announcement You can also find it here : http://codespeak.net/pypy/extradoc/sprintinfo/paris-2005-sprint.html PyPy Sprint in Paris 10th - 16th October 2005 ========================================================== The next PyPy sprint will take place in Paris at Logilab's office in France from the 10th to the 16th of October 2005 (both days fully included). To learn more about the new PyPy Python implementation look here: http://codespeak.net/pypy Goals and topics of the sprint ------------------------------ Now that pypy-0.7.0_ has been released, it is time for refactoring and planning of the next features. For these reasons the currently scheduled main topics of this sprint will be: - threading and GC - refactoring and translation features - discussing/experimenting towards JIT_ and `continuation-passing`_ style translation Moreover, we are open to work in other areas that participants are interested in. Please raise topics on the mailinglist_. Location & Accomodation ------------------------ The sprint will be held in Logilab's offices in Paris. The exact address is:: 10 rue Louis Vicat 75015 Paris We'll provide WLAN connection and there is a small kitchen to make tea and coffee. `See here`_ for a map of the area nearby. There are two metro lines and three bus lines that stop close to Logilab's: - Metro line 12 : stop "Porte de Versailles" - Metro line 13 : stop "Malakoff plateau de Vanves" - Bus 89 or 58 : stop "Carrefour Albert Legris" - Bus PC1 : stop "Porte de la Plaine" Those are more or less within 5 minutes walk. There is a huge number of hotels_ in Paris, all of which are unfortunately rather expensive. Here is another site for `hotels en France`_. It should be no problem to reach the sprint venue from anywhere in Paris using public transportation. However it would be better to find accomodation in the southern districts (14th, 15th) or in the nearby towns (Malakoff, Vanves) not far from Paris. Another good choice is to find an hotel near one of the metro line above. (avoid long bus trips as it takes more time.) Looking at the map here: hotels_, the area you should look at is between (3) and (6), (5) is still ok. As an alternative you could also try to group (4 or 5 people) and try to rent a furnished appartment for the week. Almost all announces from French websites come from `Lodgis agency`_. They are rather expensive but most of the time it can be cheaper than hotels. For the more adventurous you can look on the french site pap_ which lists rental ads. (select appartements, d??partement=75, arrondissements=15,14,13) For transportation we strongly recommend the public transportation (metro and buses). There are weekly tickets (monday to sunday) from 15.70 euros (Zone 1-2). You need a picture (ID format) and ask for a *Carte Orange*. Don't get the tourist's cards which are more expensive and valid only for 5 days. Also note that zone 1-2 won't pay for your ticket from the airport and is valid only (mostly) inside Paris. If you arrive by plane this will be most probably at Roissy airport and from there you can take the RER line B to Paris for about 8 euros. .. _`See here`: http://www.logilab.com/contact .. _hotels: http://www.hotel-paris.net/index-hotels-English.htm .. _`hotels en France`: http://www.0800paris-hotels.com/ .. _pap: http://www.pap.fr/immobilier/offres/offre-location-vacances.asp .. _`Lodgis agency`: http://www.lodgis.com/ Exact times ----------- The PyPy sprint is held Monday 10th October - Sunday 16th October 2005. Hours will be from 09:00 until 20:00. It's a good idea to arrive a day before the sprint starts and leave a day later. In the middle of the sprint there usually is a break day and it's usually ok to take half-days off if you feel like it. Network, Food ------------- There are plenty of restaurants and sandwich places around and a big supermarket for those who would rather make their own food. You will probably need a wireless network card to access the network but we can provide ethernet cables for those who don't have one. Registration etc. ----------------- Please subscribe to the `PyPy sprint mailing list`_, introduce yourself and post a note that you want to come. Feel free to ask any questions there! There also is a separate `Paris people`_ page tracking who is already thought to come. If you have commit rights on codespeak then you can modify yourself a checkout of http://codespeak.net/svn/pypy/extradoc/sprintinfo/paris-2005-people.txt .. _`Paris people`: http://codespeak.net/pypy/index.cgi?extradoc/sprintinfo/paris-people.html .. _`mailinglist`: .. _`PyPy sprint mailing list`: http://codespeak.net/mailman/listinfo/pypy-sprint .. _`pypy-0.7.0`: http://codespeak.net/pypy/dist/pypy/doc/release-0.7.0.html .. _JIT: http://en.wikipedia.org/wiki/Just-in-time_compilation .. _`continuation-passing`: http://en.wikipedia.org/wiki/Continuation_passing_style From amp at singingwizard.org Tue Sep 6 14:09:40 2005 From: amp at singingwizard.org (Arthur Peters) Date: Tue, 06 Sep 2005 07:09:40 -0500 Subject: [pypy-dev] Very long compilation on pypy Message-ID: I successfully compiled pypy a couple days ago. Then I took the source file and tried to compile it with optimizations. That didn't go as well. It's been compiling for the last 2 days (process time is 2098minutes at the moment) and has not finished. The command line was: gcc -O2 -c testing_1.c -o testing_1.o -I/usr/include/python2.4 \ -I ~/source/pypy-dist/pypy/translator/c/ -I /usr/include/gc/ I just aborted it because it was making the system slow. Has anyone else experienced this? Does -O2 trigger a bug in GCC with the large source file? or does it really take 3-4 days to compile it? Just thought I'd bring it up. I'm using the braohm GC. -Arthur From arigo at tunes.org Tue Sep 6 16:12:41 2005 From: arigo at tunes.org (Armin Rigo) Date: Tue, 6 Sep 2005 16:12:41 +0200 Subject: [pypy-dev] [hatcliff@cis.ksu.edu: [PEPM 2006] Call For Papers] Message-ID: <20050906141241.GE13491@code1.codespeak.net> Hi, A conference where I'd probably like to go, early 2006: ----- Forwarded message from John Hatcliff ----- Date: Wed, 27 Jul 2005 16:53:01 -0500 From: John Hatcliff To: arigo at tunes.org Subject: [PEPM 2006] Call For Papers Reply-To: hatcliff at cis.ksu.edu Hi Armin, I wanted to send you a personal email to encourage you to consider submitting a paper and attending PEPM 2006 -- to held in conjunction with POPL 2006 in Charleston, South Carolina (USA) on January 9-10, 2006. As an author of a PEPM 2004 paper, I am sure that you are interested in seeing PEPM continue as a successful meeting in which researchers interested in a variety of forms of semantics-based program manipulation meet to present results and exchange ideas that drive future research. Frank Tip and I, serving as program co-chairs for PEPM 2006, are working hard to put together a strong and diverse program committee and workshop format that we hope will lead to an exciting meeting. We've included the PEPM 2006 Call For Papers below and you can find additional information on the PEPM web-page (http://www.cis.ksu.edu/santos/pepm06). As you can see from the CFP and the program committee we are aiming to continue to broaden PEPM to consider a variety of forms of semantics-based program manipulation including refactoring, rule-based transformations, slicing, model-driven code generation and transformations. In closing, Frank and I would like to encourage you again to considering submitting to PEPM 2006 and to help us in our publicity efforts by forwarding the PEPM CFP below to any colleagues who might be interested in PEPM 2006. If you have any questions about suitable topics/content for PEPM submissions, please do not hesitate to email the PC Chairs directly. Best regards, john ==================================================================== C A L L F O R P A P E R S === P E P M 2006 === ACM SIGPLAN 2006 Workshop on Partial Evaluation and Program Manipulation (Affiliated with POPL 2006) http://www.cis.ksu.edu/santos/pepm06 January 9-10, 2006 Charleston, South Carolina The PEPM Symposium/Workshop series aims to bring together researchers and practitioners working in the areas of program manipulation, partial evaluation, and program generation. PEPM focuses on techniques, supporting theory, tools, and applications of the analysis and manipulation of programs. The 2006 PEPM workshop will be based on a broad interpretation of semantics-based program manipulation. This year, a concerted effort will be made to expand the scope of PEPM significantly beyond the traditionally covered areas of partial evaluation and specialization and include practical applications of program transformations such as refactoring tools, and practical implementation techniques such as rule-based transformation systems. In addition, the scope of PEPM will be broadened to cover manipulation and transformations of program and system representations such as structural and semantic models that occur in the context of model-driven development. In order to reach out to practitioners, a separate category of tool demonstration papers will be solicited. Topics of interest for PEPM'06 include, but are not limited to: * Program and model manipulation techniques such as transformations driven by rules, patterns, or analyses, partial evaluation, specialization, slicing, symbolic execution, refactoring, aspect weaving, decompilation, and obfuscation. * Program analysis techniques that are used to drive program/model manipulation such as abstract interpretation, static analysis, binding-time analysis, dynamic analysis, constraint solving, and type systems. * Analysis and transformation for programs/models with advanced features such as objects, generics, ownership types, aspects, reflection, XML type systems, component frameworks, and middleware. * Techniques that treat programs/models as data objects including meta-programming, generative programming, model-driven program generation and transformation. * Application of the above techniques including experimental studies, engineering needed for scalability, and benchmarking in a wide variety of domains including source code manipulation, domain-specific language implementations, scientific computing, middleware frameworks and infrastructure needed for distributed and web-based applications. We especially encourage papers that break new ground including descriptions of how program/model manipulation tools can be integrated into realistic software development processes, descriptions of robust tools capable of effectively handling realistic applications, and new areas of application such as rapidly evolving systems, distributed and web-based programming including middleware manipulation, model-driven development, and on-the-fly program adaptation driven by run-time or statistical analysis. Submission Categories, Guidelines, and Proceedings: Regular Research Papers must not exceed 10 pages in ACM Proceedings style. Tool demonstration papers must not exceed 4 pages in ACM Proceedings style, and authors will be expected to present a live demonstration of the described tool at the workshop. Suggested topics, evaluation criteria, and writing guidelines for both research tool demonstration papers will be made available on the PEPM'06 Web-site. Papers should be submitted electronically via the workshop web site. We plan to publish the workshop proceedings in ACM SIGPLAN Notices (with full papers appearing in the ACM Digital Library) and selected papers will be invited for a journal special issue dedicated to PEPM'06. Important Dates: Submission........: October 7, 2005 Apia, 11:59pm, Samoan time (firm deadline, no extensions) Notification......: November 18, 2005 Camera-Ready Paper: December 16, 2005. Workshop co-Chairs: John Hatcliff, Kansas State University, USA (hatcliff at cis.ksu.edu) Frank Tip, IBM T.J. Watson Research Center, USA (ftip at us.ibm.com) PEPM 2006 Program Committee: Krzysztof Czarnecki University of Waterloo Gary Daugherty Rockwell Collins Advanced Technology Center Tom Dean Queen's University Mangala Gowri Nanda IBM India John Hatcliff (co-chair) Kansas State University Nevin Heintze Agere Systems Jaakko J?rvi Texas A & M University Jens Krinke University of Hagen Shriram Krishnamurthi Brown University Julia Lawall University of Copenhagen (DIKU) Oege de Moor Oxford University Germ?n Puebla Technical University of Madrid Peter Sestoft Royal Veterinary and Agricultural University (Denmark) Gregor Snelting University of Passau Frank Tip (co-chair) IBM T.J. Watson Research Center Eelco Visser Universiteit Utrecht -- ---------------------------------------------------------------------------- John Hatcliff Phone: 785-532-6350 Professor Fax..: 785-532-7353 Department of Computing and Information Sciences Email: hatcliff at cis.ksu.edu 234 Nichols Hall WWW..: http://www.cis.ksu.edu/~hatcliff Kansas State University Manhattan, KS 66506 ---------------------------------------------------------------------------- ----- End forwarded message ----- From 2005a at usenet.alexanderweb.de Tue Sep 6 16:16:35 2005 From: 2005a at usenet.alexanderweb.de (Alexander Schremmer) Date: Tue, 6 Sep 2005 16:16:35 +0200 Subject: [pypy-dev] Re: Very long compilation on pypy References: Message-ID: On Tue, 06 Sep 2005 07:09:40 -0500, Arthur Peters wrote: > Does it really take 3-4 days to compile it? How many MB of RAM do you have? Did gcc fit into memory while compiling? Kind regards, Alexander From amp at singingwizard.org Wed Sep 7 00:26:31 2005 From: amp at singingwizard.org (Arthur Peters) Date: Tue, 06 Sep 2005 17:26:31 -0500 Subject: [pypy-dev] Re: Very long compilation on pypy In-Reply-To: Message-ID: Yes, it used ~430MB VM (I'm remembering this, so could be wrong); ~360MB of the was in RAM. My system has 512MB of RAM. It didn't swap and it constantly used ~100% CPU. -Arthur PS: Alexander sorry about the duplicate. On 9/6/2005, "Alexander Schremmer" <2005a at usenet.alexanderweb.de> wrote: >On Tue, 06 Sep 2005 07:09:40 -0500, Arthur Peters wrote: > >> Does it really take 3-4 days to compile it? > >How many MB of RAM do you have? Did gcc fit into memory while compiling? > >Kind regards, >Alexander > >_______________________________________________ >pypy-dev at codespeak.net >http://codespeak.net/mailman/listinfo/pypy-dev From tismer at stackless.com Wed Sep 7 03:01:40 2005 From: tismer at stackless.com (Christian Tismer) Date: Wed, 07 Sep 2005 03:01:40 +0200 Subject: [pypy-dev] Re: [pypy-svn] r17215 - pypy/dist/pypy/rpython In-Reply-To: <20050905060347.GK666@solar.trillke.net> References: <20050905003250.BF42C27B47@code1.codespeak.net> <20050905060347.GK666@solar.trillke.net> Message-ID: <431E3BF4.5070802@stackless.com> holger krekel wrote: > Hi Christian, hi all, ... > ... are such optimizations measurably effective at all? > I would think that such low-level details are > handled by the backend or, if not, that we could > write a generic transformation that does that for us. > Or am i missing something? Not really. I think I tend to overdoing. But in a sense, since this is very low-level code, I like to see just one operation on a line, not hiding ops in expressions. I agree that this was not necessary, I just liked it better. This is not where my time was spent, that was a very different matter :-) -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From tismer at stackless.com Wed Sep 7 03:07:03 2005 From: tismer at stackless.com (Christian Tismer) Date: Wed, 07 Sep 2005 03:07:03 +0200 Subject: [pypy-dev] Re: Very long compilation on pypy In-Reply-To: References: Message-ID: <431E3D37.5070000@stackless.com> Arthur Peters wrote: > Yes, it used ~430MB VM (I'm remembering this, so could be wrong); ~360MB > of the was in RAM. My system has 512MB of RAM. It didn't swap and it > constantly used ~100% CPU. Nah, I (sorry) doubt this a little bit. 512 MB of RAM is definately not enough when you try to compile PyPy without extra options. You can get the memory requirements below 400 MB by using -t-lowmem. At least this is my experience on a Windows laptop. Did you see gcc in the task list, or was it still generating the C code? -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From amp at singingwizard.org Wed Sep 7 02:26:45 2005 From: amp at singingwizard.org (Arthur Peters) Date: Tue, 06 Sep 2005 19:26:45 -0500 Subject: [pypy-dev] Re: Very long compilation on pypy In-Reply-To: <431E3D37.5070000@stackless.com> Message-ID: I generated the source using the -c option on translate_pypy (I did use -t-lowmem as well). I then compiled the generated C code by hand after having exited the translate_pypy process. I did it once successfully without -O of any kind, then I had problems with a compile using -O2. All the information was about a cc1 process (it's the compiler process gcc uses). The info was gathered using top. I am quite certain the machine was not swapping (I would have heard it. I listen for disk noise.) and that the process was using 100% CPU (which it could not have been doing if it were swapping). Also the 2098 was CPU time not realtime. So time to swap data from/to disk would not have been counted. I must not have included enough information to make it clear what I did. I'm sorry. -Arthur On 9/6/2005, "Christian Tismer" wrote: >Arthur Peters wrote: > >> Yes, it used ~430MB VM (I'm remembering this, so could be wrong); ~360MB >> of the was in RAM. My system has 512MB of RAM. It didn't swap and it >> constantly used ~100% CPU. > >Nah, I (sorry) doubt this a little bit. >512 MB of RAM is definately not enough when you try to compile >PyPy without extra options. >You can get the memory requirements below 400 MB by using -t-lowmem. > >At least this is my experience on a Windows laptop. > >Did you see gcc in the task list, or was it still generating the C code? > >-- >Christian Tismer :^) >tismerysoft GmbH : Have a break! Take a ride on Python's >Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ >14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ >work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 >PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 > whom do you want to sponsor today? http://www.stackless.com/ From ludal at logilab.fr Wed Sep 7 11:31:59 2005 From: ludal at logilab.fr (Ludovic Aubry) Date: Wed, 07 Sep 2005 11:31:59 +0200 Subject: [pypy-dev] Re: Very long compilation on pypy In-Reply-To: References: Message-ID: <1126085519.5470.4.camel@octans.logilab.fr> Hello, If you are using a recent gcc version (>= 3.3 or 3.4) you could try to use : -O2 -fno-unit-at-a-time Here's a sample of the documentation : """Parse the whole compilation unit before starting to produce code. This allows some extra optimizations to take place but consumes more memory (in general).""" I haven't tried it (doing it right now) but it should improve things. regards, On Tue, 2005-09-06 at 19:26 -0500, Arthur Peters wrote: > > I am quite certain the machine was not swapping (I would have heard > it. I > listen for disk noise.) and that the process was using 100% CPU (which > it could not have been doing if it were swapping). Also the 2098 was > CPU > time not realtime. So time to swap data from/to disk would not have > been > counted. From arigo at tunes.org Wed Sep 7 17:50:48 2005 From: arigo at tunes.org (Armin Rigo) Date: Wed, 7 Sep 2005 17:50:48 +0200 Subject: [pypy-dev] next pypy-sync meeting Thursday 1pm Message-ID: <20050907155048.GA28722@code1.codespeak.net> Hi all, Next #pypy-sync meeting on freenode.net is tomorrow Thursday at 1pm as usual. Regular topics -------------- * activity reports (3 prepared lines of info) * resolve conflicts/blockers Topics of the week ------------------ Documentation and reports: status, assignments: summary of the draft for the reports we still have to write (eu-tracking/deliverable/D05-reports-planning.txt); who would like to help on which one. Setting up another PyPy server: I (Armin) will shortly put on-line a PyPy server at the HHU, for any computation-intensive task, with logins for any PyPy developer. Should we have tests and/or translate_pypy run automatically on this machine? See you tomorrow, Armin From seberino at spawar.navy.mil Wed Sep 7 20:51:21 2005 From: seberino at spawar.navy.mil (seberino at spawar.navy.mil) Date: Wed, 7 Sep 2005 11:51:21 -0700 Subject: [pypy-dev] *FREE on-site* PyPy helper in exchange for PyPy MENTORING!?.... Message-ID: <20050907185121.GA26649@spawar.navy.mil> There is a slight chance I may be able to convince US Navy to let me visit Europe for a few months to learn about cutting edge European software technologies. This may be the chance I've been looking for to talk about PyPy source with gurus *in person* to come up to speed fast on it and be a real contributor to PyPy project. Any suggestions are desperately needed.... ------------------ What can I give back to encourage someone to mentor me? (Something that involved learning the source really well like helping comment the code and document the source is an idea.) Who would be a good person to mentor me who can help me understand the source code really well? Where are most PyPy developers working? Perhaps multiple PyPy gurus are located in similar location? Are there any conferences and meetings I should make sure to attend? What time frame is a good idea? Any other thoughts? Thanks! Chris Seberino ------------------ _______________________________________ Christian Seberino, Ph.D. SPAWAR Systems Center San Diego Code 2872 49258 Mills Street, Room 158 San Diego, CA 92152-5385 U.S.A. Phone: (619) 553-9973 Fax : (619) 553-6521 Email: seberino at spawar.navy.mil _______________________________________ From bea at netg.se Wed Sep 7 22:46:42 2005 From: bea at netg.se (Beatrice During) Date: Wed, 7 Sep 2005 22:46:42 +0200 (CEST) Subject: [pypy-dev] *FREE on-site* PyPy helper in exchange for PyPy MENTORING!?.... In-Reply-To: <20050907185121.GA26649@spawar.navy.mil> References: <20050907185121.GA26649@spawar.navy.mil> Message-ID: Hi there On Wed, 7 Sep 2005 seberino at spawar.navy.mil wrote: > There is a slight chance I may be able to convince US Navy to let me > visit Europe for a few months to learn about cutting edge European > software technologies. > > This may be the chance I've been looking for > to talk about PyPy source with gurus *in person* to come up to speed > fast on it and be a real contributor to PyPy project. Well - thanks for being so interested in PyPy! I will try to answer some of your questions - I am sure some of the "gurus" will fill in as well. > Any suggestions are desperately needed.... > > ------------------ > > What can I give back to encourage someone to mentor me? (Something > that involved learning the source really well like helping comment > the code and document the source is an idea.) We are currently putting a lot of effort into documentation on several key aspects of the latest 0.7 release. Im sure that this documentation will be a good way to start identifying where you have more questions. Then pypy-dev is the right place to post them. Give back? Hmm - we always appreciate feedback on tests as well as documentation! Your question about mentoring is a bit tricky - it would depend on what kind of questions you have after reading the updated documentation (which will be ready by the end of September). > Who would be a good person to mentor me who can help me understand the > source code really well? Well - key "gurus" are Holger Krekel, Armin Rigo, Samuele Pedronis, Christian Tismer, Michael Hudson and they are somewhat busy with writing documentation ;-) We also have lots of other very skilled and experienced developers contributing on pypy-dev - your questions will be answered but I think it could be tricky to "assign" a mentor.... > Where are most PyPy developers working? Perhaps > multiple PyPy gurus are located in similar location? Well - Gothenburg in Sweden is growing into some kind of PyPy center with Samuele Pedronis, Armin Rigo and Anders Chrigstr?m working (mostly) from there. We also have Holger Krekel, Carl Friedrich Bolz and Christian Tismer in Germany (Hildesheim etc) > Are there any conferences and meetings I should make sure to attend? You are welcome to our sprints, unfortunately most of them are held in Europe. We sprint every 6th week (we aim for that at least) and sprinting is a great way to dive into the codebase relatively quickly (via tutorials, "gurus" and pairprogramming. Next sprint is in Paris (10-17th October and we probably are having a sprint in early December (location not decided yet). Please join our mailinglist pypy-sprint at codespeak.net for ongoing information. Conferences - mostly in Europe. We sprinted at PyCon this year and are currently discussing wether we will sprint there next year. But some of us are definitely showing up - maybe that would be a good place to meet? Europython is also a sure bet if you can come over that is. We are currently planning our attendance to conferences the upcoming year and we will post this info on codespeak and to pypydev during this month. > What time frame is a good idea? > Any other thoughts? Just a thank you for your interest! > Thanks! > > Chris Seberino Cheers Beatrice D?ring/assistant project manager PyPy ( a non developer BTW ;-) From cfbolz at gmx.de Wed Sep 7 23:37:17 2005 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Wed, 07 Sep 2005 23:37:17 +0200 Subject: [pypy-dev] *FREE on-site* PyPy helper in exchange for PyPy MENTORING!?.... In-Reply-To: <20050907185121.GA26649@spawar.navy.mil> References: <20050907185121.GA26649@spawar.navy.mil> Message-ID: <431F5D8D.9070002@gmx.de> Hi! seberino at spawar.navy.mil wrote: > There is a slight chance I may be able to convince US Navy to let me > visit Europe for a few months to learn about cutting edge European > software technologies. Cool! > This may be the chance I've been looking for > to talk about PyPy source with gurus *in person* to come up to speed > fast on it and be a real contributor to PyPy project. Contributions are of course very welcome! I guess the most immediate step would be to read through the documentation and ask any question you might have (here or on the IRC channel). > Any suggestions are desperately needed.... > > ------------------ > > What can I give back to encourage someone to mentor me? (Something > that involved learning the source really well like helping comment > the code and document the source is an idea.) It certainly won't be a problem finding work for you :-) > Who would be a good person to mentor me who can help me understand the > source code really well? > > Where are most PyPy developers working? The paid PyPy developers are working in Gothenburg: Armin, Anders C., Samuele (Sweden); Hildesheim: Holger, Berlin: Christian, Saarbruecken: Anders L., Heidelberg: me (from October on) (Germany) and Paris: Ludovic, Adrien. > Perhaps multiple PyPy gurus are located in similar location? > Are there any conferences and meetings I should make sure to attend? Perhaps the easiest way to come up to speed with PyPy would be to come to one (or several :-) )of our coding sprints. The next one will be in Paris in October. Most of the developers attend these sprints and quite some new contributors were introduced to PyPy by coming to a sprint. > Any other thoughts? Can't really say much about the mentoring idea: I guess one of the problems would be that the person mentoring you would be distracted from his own work. On the other hand the work you would do could probably make up for this. From my own experience it is very easy to get all the advice you need from the gurus by hanging out on IRC and bothering them :-). Although a sprint is of course more effective. Cheers, Carl Friedrich From hpk at trillke.net Thu Sep 8 00:01:49 2005 From: hpk at trillke.net (holger krekel) Date: Thu, 8 Sep 2005 00:01:49 +0200 Subject: [pypy-dev] *FREE on-site* PyPy helper in exchange for PyPy MENTORING!?.... In-Reply-To: <20050907185121.GA26649@spawar.navy.mil> References: <20050907185121.GA26649@spawar.navy.mil> Message-ID: <20050907220149.GD666@solar.trillke.net> Hi Chris, On Wed, Sep 07, 2005 at 11:51 -0700, seberino at spawar.navy.mil wrote: > There is a slight chance I may be able to convince US Navy to let me > visit Europe for a few months to learn about cutting edge European > software technologies. The european union will be pleased to hear that the US Navy is taking interest :-) > This may be the chance I've been looking for > to talk about PyPy source with gurus *in person* to come up to speed > fast on it and be a real contributor to PyPy project. Guruness depends on the area of interest. It would actually be interesting and helpful to know a bit of your background to recommend the right kind of guruness. For "how the hell this development process works" you might ask Bea, for example. For understanding 'specialize:ctr_location' it's probably a good idea to ask Samuele or Armin. And so on. The project starts to reach a size where only very few people (if any) can know everything. > Any suggestions are desperately needed.... Let's wait with desperation until we have encountered "cosmic ray" annotation problems in connection with stackless JIT compiler experiments. More or less the topics of the next PyPy sprint in Paris :-) > ------------------ > > What can I give back to encourage someone to mentor me? (Something > that involved learning the source really well like helping comment > the code and document the source is an idea.) Sure. Also there are a number of dark corners (tm) that need clenaups/rewrites or (re)thinking further. Oh, actually documenting the very existence of dark corners has been suggested in Heidelberg recently :-) > Who would be a good person to mentor me who can help me understand the > source code really well? A lot of people probably. Depends on your area of interest and what you mean by "really well". Do you have a solid Python programming background and e.g. know a bit about Python code and function objects, and how the CPython virtual machine works? You may consider parts of PyPy as documentation about how CPython actually works on a more internal level, btw. > Where are most PyPy developers working? Perhaps > multiple PyPy gurus are located in similar location? We are working from multiple places mostly in Europe and usually meet for 6-8 days "sprint" meetings. > Are there any conferences and meetings I should make sure to attend? Visiting the US or Europe Python conferences certainly makes sense. Attending our sprints probably makes most sense. All a bit depending on your background and interests. Our sprints around conferences are usually most newbie friendly, whereas our "we must get things done in three weeks time" meetings are less newcomer friendly. > What time frame is a good idea? I am not sure i understand the question. Time frame for what exactly? > Any other thoughts? Have you already considered coming to a PyPy sprint, e.g. the next one in Paris 10th-16th October? cheers, holger From Ben.Young at risk.sungard.com Wed Sep 7 16:51:12 2005 From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com) Date: Wed, 7 Sep 2005 15:51:12 +0100 Subject: [pypy-dev] Current failures Message-ID: Hi PyPy'rs, Thought I had better let you know about my current test failures, as these tests appear to have been failing for about a week or more. Here's the log: Two of the failures appear to be windows math functions doing something slightly different to the CPython equivs. The last two failures are link errors which is weird, as you would think the same would be happening on Linux? Cheers, Ben -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: results.txt URL: From arigo at tunes.org Thu Sep 8 12:09:26 2005 From: arigo at tunes.org (Armin Rigo) Date: Thu, 8 Sep 2005 12:09:26 +0200 Subject: [pypy-dev] Re: [pypy-svn] r17301 - pypy/dist/pypy/objspace/std In-Reply-To: <20050906201234.B682727B5E@code1.codespeak.net> References: <20050906201234.B682727B5E@code1.codespeak.net> Message-ID: <20050908100926.GA7109@code1.codespeak.net> Hi Christian, On Tue, Sep 06, 2005 at 10:12:34PM +0200, tismer at codespeak.net wrote: > (check-in of listobject.py) > turned all range loops into regular ones, since ranges support is no good, yet. This is the typical kind of changes that we should not do. It makes the code harder to read just for a premature optimization. We know anyway that we need complete support for "for in range" loops, rather sooner than later, and Carl is progressing in this direction. A bientot, Armin From arigo at tunes.org Thu Sep 8 17:43:30 2005 From: arigo at tunes.org (Armin Rigo) Date: Thu, 8 Sep 2005 17:43:30 +0200 Subject: [pypy-dev] *FREE on-site* PyPy helper in exchange for PyPy MENTORING!?.... In-Reply-To: <20050907185121.GA26649@spawar.navy.mil> References: <20050907185121.GA26649@spawar.navy.mil> Message-ID: <20050908154330.GA8251@code1.codespeak.net> Hi, On Wed, Sep 07, 2005 at 11:51:21AM -0700, seberino at spawar.navy.mil wrote: > There is a slight chance I may be able to convince US Navy to let me > visit Europe for a few months to learn about cutting edge European > software technologies. Great! > This may be the chance I've been looking for > to talk about PyPy source with gurus *in person* to come up to speed > fast on it and be a real contributor to PyPy project. > > What can I give back to encourage someone to mentor me? Just to add to the answers already given, this is definitely a group project, in the sense that you will probably not find one of us ready to give mentoring single-handed -- or more to the point, thinking that he should necessarily do so. We try to focus this effort in writing good documentation and keeping the #pypy channel alive -- we are basically working on #pypy between the sprints. Feel free to drop by (very approximately, you have a good chance to find us between 11 and 19 UTC). The sprints themselves are the most productive meetings, and the best place for learning as well; most people go beyond the basics by attenting sprints. So I can only mention again the Paris sprint :-) If you want to plan for a longer trip, then there is probably no preferred part of the year, with the exception of EuroPython which is in the summer. Feel free to introduce yourself (Python background etc.) here or on #pypy! A bientot, Armin. From arigo at tunes.org Thu Sep 8 18:53:40 2005 From: arigo at tunes.org (Armin Rigo) Date: Thu, 8 Sep 2005 18:53:40 +0200 Subject: [pypy-dev] Current failures Message-ID: <20050908165340.GA11962@code1.codespeak.net> Hi Ben, Thanks for tirelessly reporting Windows failures :-) Which exact Windows are you using, BTW? The link failures should be resolved. They are due to the unsetenv() built-in missing on Windows, but still present in os.py -- faked, implemented in pure Python as os.setenv(xxx, ''). This caused all parts of the translation to assume that there was an unsetenv() built-in, but the C include files at the end didn't actually define it because CPython doesn't define HAVE_UNSETENV. The rounding errors are more tricky. We could of course just use an "is almost equal" comparison, but still we are a bit clueless (though Christian is digging). Armin From tismer at stackless.com Thu Sep 8 22:01:55 2005 From: tismer at stackless.com (Christian Tismer) Date: Thu, 08 Sep 2005 22:01:55 +0200 Subject: [pypy-dev] Current failures In-Reply-To: <20050908165340.GA11962@code1.codespeak.net> References: <20050908165340.GA11962@code1.codespeak.net> Message-ID: <432098B3.3040202@stackless.com> Armin Rigo wrote: ... > The rounding errors are more tricky. We could of course just use an "is > almost equal" comparison, but still we are a bit clueless (though > Christian is digging). digged quite deeply. Temporarily used an "almost equal",but then figured out that we also don't get an exception on log(-1.0). Then I used the /Op option for precision math, and things worked. There are still two errors left, which I don't understand. one is mallocs, frees = module.malloc_counters() assert mallocs == frees gives 3 == 1 on windows in the context of testing test_start_new_thread The other one is def test_environ(): import py gw = py.execnet.PopenGateway() chan = gw.remote_exec(py.code.Source(test_src)) res = chan.receive() E assert res > assert False I'm not sure if execnet is mature enough to blame windows, but I could have put time there, too, if I had some. -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From Ben.Young at risk.sungard.com Fri Sep 9 09:47:26 2005 From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com) Date: Fri, 9 Sep 2005 08:47:26 +0100 Subject: [pypy-dev] Current failures In-Reply-To: <20050908165340.GA11962@code1.codespeak.net> Message-ID: pypy-dev-bounces at codespeak.net wrote on 08/09/2005 17:53:40: > Hi Ben, > > Thanks for tirelessly reporting Windows failures :-) > Which exact Windows are you using, BTW? > That's alright :) I wish I could do more to actually fix the issues, but by the time I get home from work a can barely stand to even look at a computer if you know what I mean! I'll stop reporting errors / questions if it gets too tiresome for you all. For your information i'm running PyPy on cygwin on XP Pro, which probably exercises a slightly different set of issues to straight windows. P.S Do you have any plans to make annotation/translation more robust, as it seems it is very easy to introduce SomeObjectness without knowing, and even of annotation succeeds, successful translation is not guaranteed (or am I talking rubbish?) Cheers, Ben > The link failures should be resolved. They are due to the unsetenv() > built-in missing on Windows, but still present in os.py -- faked, > implemented in pure Python as os.setenv(xxx, ''). This caused all parts > of the translation to assume that there was an unsetenv() built-in, but > the C include files at the end didn't actually define it because CPython > doesn't define HAVE_UNSETENV. > > The rounding errors are more tricky. We could of course just use an "is > almost equal" comparison, but still we are a bit clueless (though > Christian is digging). > > > Armin > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > > From hpk at trillke.net Fri Sep 9 10:14:47 2005 From: hpk at trillke.net (holger krekel) Date: Fri, 9 Sep 2005 10:14:47 +0200 Subject: [pypy-dev] Current failures In-Reply-To: References: <20050908165340.GA11962@code1.codespeak.net> Message-ID: <20050909081447.GY666@solar.trillke.net> Hi Ben, On Fri, Sep 09, 2005 at 08:47 +0100, Ben.Young at risk.sungard.com wrote: > P.S Do you have any plans to make annotation/translation more robust, as > it seems it is very easy to introduce SomeObjectness without knowing, and > even of annotation succeeds, successful translation is not guaranteed (or > am I talking rubbish?) We don't have an explicit strategy. I guess, for one, it's a matter of educating as many people as possible about things to avoid. Second, it's a matter of having enough tests and example translation targets where we notice failures as early as possibly. Third, it's a matter of refactoring and improving error reporting regarding the annotation and specialiazing phases. All three points are being tackled at the moment but there is way to go, especially when it comes to recognize Windows problems early on. Maybe we could use a "test fixing" or "bug" day where we specifically tackle random test and translation failures. cheers, holger From Ben.Young at risk.sungard.com Fri Sep 9 10:29:41 2005 From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com) Date: Fri, 9 Sep 2005 09:29:41 +0100 Subject: [pypy-dev] Current failures In-Reply-To: <20050908165340.GA11962@code1.codespeak.net> Message-ID: Hi Everyone, Further to my previous mail, I just read the pypy logs for last night, and I wanted to say well done for fixing such a delicate problem. I'm sorry for causing you all so much work. Cheers, Ben pypy-dev-bounces at codespeak.net wrote on 08/09/2005 17:53:40: > Hi Ben, > > Thanks for tirelessly reporting Windows failures :-) > Which exact Windows are you using, BTW? > > The link failures should be resolved. They are due to the unsetenv() > built-in missing on Windows, but still present in os.py -- faked, > implemented in pure Python as os.setenv(xxx, ''). This caused all parts > of the translation to assume that there was an unsetenv() built-in, but > the C include files at the end didn't actually define it because CPython > doesn't define HAVE_UNSETENV. > > The rounding errors are more tricky. We could of course just use an "is > almost equal" comparison, but still we are a bit clueless (though > Christian is digging). > > > Armin > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > > From tismer at stackless.com Fri Sep 9 17:29:07 2005 From: tismer at stackless.com (Christian Tismer) Date: Fri, 09 Sep 2005 17:29:07 +0200 Subject: [pypy-dev] Current failures In-Reply-To: References: Message-ID: <4321AA43.6070102@stackless.com> Ben.Young at risk.sungard.com wrote: > Hi Everyone, > > Further to my previous mail, I just read the pypy logs for last night, and > I wanted to say well done for fixing such a delicate problem. I'm sorry > for causing you all so much work. Please keep naggin', this is very helpful! ciao - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From tismer at stackless.com Sat Sep 10 00:20:25 2005 From: tismer at stackless.com (Christian Tismer) Date: Sat, 10 Sep 2005 00:20:25 +0200 Subject: [pypy-dev] Re: [pypy-svn] r17433 - in pypy/dist/pypy: annotation rpython rpython/test In-Reply-To: <20050909194927.B2FF027B82@code1.codespeak.net> References: <20050909194927.B2FF027B82@code1.codespeak.net> Message-ID: <43220AA9.7080805@stackless.com> arigo at codespeak.net wrote: ... > Support for the iterkeys(), itervalues() and iteritems() methods on RPython > dicts. All three kinds of iterators are implementated with the same > SomeIterator and StrDictIteratorRepr classes, using a 'variant' field to > distinguish (and the dummy function trick in rdict.py to get the > specialization). > > All xxxIteratorRepr classes now inherit from a common IteratorRepr, which > implements the common functionality of returning the iterator itself when the > 'iter' operation is applied. Congratulations! I'm very impressed how quickly you could hammer out this nice solution. And fine that the dummy function approach stays useful. -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From gaiacrtn at free.fr Sun Sep 11 00:46:32 2005 From: gaiacrtn at free.fr (Baptiste Lepilleur) Date: Sun, 11 Sep 2005 00:46:32 +0200 Subject: [pypy-dev] Translating code to LLVM Message-ID: <008001c5b659$7e50db60$0300a8c0@anabelle> Following the getting started instruction, I'm trying to get the LLVM translation pgcd example below. I'm on Windows XP using LLVM cvs build with VC++ 7.1. --- >>> t = Translator(test.my_gcd) >>> a = t.annotate([int, int]) >>> t.specialize() >>> print t.llvm() <... really huge amount of LLVM code ...> >>> f = t.llvmcompile() >>> f.pypy_my_gcd_wrapper(15, 10) --- But when executing (with pypy 0.7) >>> f = t.llvmcompile() The following error occurs. Traceback (most recent call last): File "", line 1, in ? File "E:\prg\py\App\pypy-0.7\pypy\translator\translator.py", line 296, in llvmcompile return genllvm.genllvm(self, really_compile=really_compile, standalone=standalone, optimize=opti mize, exe_name=exe_name) [...] truncated File "E:\prg\py\App\pypy-0.7\py\process\cmdexec.py", line 107, in win32_exec_cmd raise ExecutionFailed(status, status, cmd, out, err) py.__.process.cmdexec.ExecutionFailed: ExecutionFailed: 1 llvm-as < my_gcd_1.ll | opt -verify -lowe rsetjmp -funcresolve -raiseallocs -simplifycfg -mem2reg -globalopt -globaldce -ipconstprop -deadarge lim -instcombine -simplifycfg -prune-eh -inline -simplify-libcalls -argpromotion -raise -tailduplica te -simplifycfg -scalarrepl -instcombine -break-crit-edges -condprop -tailcallelim -simplifycfg -rea ssociate -loopsimplify -licm -instcombine -indvars -loop-unroll -instcombine -load-vn -gcse -sccp -i nstcombine -break-crit-edges -condprop -dse -mergereturn -adce -simplifycfg -deadtypeelim -constmerg e -verify -f -o my_gcd_1.bc opt: Invalid Top Level Block Length! Type:1, Size:49557 (Vers=0, Pos=8) With PyPy SVN head, I get a more direct error: >>> print t.llvm() [...] truncated Traceback (most recent call last): File "", line 1, in ? File "E:\prg\py\App\pypy\pypy\translator\translator.py", line 201, in llvm filename = gen.gen_llvm_source() File "E:\prg\py\App\pypy\pypy\translator\llvm\genllvm.py", line 179, in gen_llvm_source codewriter.append(self.gcpolicy.declarations()) AttributeError: 'NoneType' object has no attribute 'declarations' Does anyone know what I'm doing wrong ? On another topic, where do the generated files go ? I tried to find the files generated by the c and llvm translation but couldn't find them... Baptiste. From sabre at nondot.org Sun Sep 11 06:11:06 2005 From: sabre at nondot.org (Chris Lattner) Date: Sat, 10 Sep 2005 23:11:06 -0500 (CDT) Subject: [pypy-dev] Translating code to LLVM In-Reply-To: <008001c5b659$7e50db60$0300a8c0@anabelle> References: <008001c5b659$7e50db60$0300a8c0@anabelle> Message-ID: On Sun, 11 Sep 2005, Baptiste Lepilleur wrote: > Following the getting started instruction, I'm trying to get the LLVM > translation pgcd example below. > > I'm on Windows XP using LLVM cvs build with VC++ 7.1. ... > raise ExecutionFailed(status, status, cmd, out, err) > py.__.process.cmdexec.ExecutionFailed: ExecutionFailed: 1 llvm-as < > my_gcd_1.ll | opt -verify -lowe > rsetjmp -funcresolve -raiseallocs -simplifycfg -mem2reg -globalopt -globaldce > -ipconstprop -deadarge > lim -instcombine -simplifycfg -prune-eh -inline -simplify-libcalls > -argpromotion -raise -tailduplica > te -simplifycfg -scalarrepl -instcombine -break-crit-edges -condprop > -tailcallelim -simplifycfg -rea > ssociate -loopsimplify -licm -instcombine -indvars -loop-unroll -instcombine > -load-vn -gcse -sccp -i > nstcombine -break-crit-edges -condprop -dse -mergereturn -adce -simplifycfg > -deadtypeelim -constmerg > e -verify -f -o my_gcd_1.bc > opt: Invalid Top Level Block Length! Type:1, Size:49557 (Vers=0, Pos=8) This sounds like an LLVM problem: the assembler is producing an invalid bytecode file. Either the .ll file is broken (in which case the assembler should have rejected it) or the bytecode writer is doing something wrong. In any case, if you gzip up the source and email it to me, I'd be happy to look at it. -Chris > With PyPy SVN head, I get a more direct error: >>>> print t.llvm() > [...] truncated > Traceback (most recent call last): > File "", line 1, in ? > File "E:\prg\py\App\pypy\pypy\translator\translator.py", line 201, in llvm > filename = gen.gen_llvm_source() > File "E:\prg\py\App\pypy\pypy\translator\llvm\genllvm.py", line 179, in > gen_llvm_source > codewriter.append(self.gcpolicy.declarations()) > AttributeError: 'NoneType' object has no attribute 'declarations' > > Does anyone know what I'm doing wrong ? > > On another topic, where do the generated files go ? I tried to find the files > generated by the c and llvm translation but couldn't find them... > > Baptiste. > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > -Chris -- http://nondot.org/sabre/ http://llvm.org/ From gaiacrtn at free.fr Sun Sep 11 10:05:45 2005 From: gaiacrtn at free.fr (Baptiste Lepilleur) Date: Sun, 11 Sep 2005 10:05:45 +0200 Subject: [pypy-dev] Translating code to LLVM References: <008001c5b659$7e50db60$0300a8c0@anabelle> Message-ID: <008e01c5b6a7$9d0aede0$0300a8c0@anabelle> Chris Lattner wrote: > On Sun, 11 Sep 2005, Baptiste Lepilleur wrote: > >> Following the getting started instruction, I'm trying to get the LLVM >> translation pgcd example below. >> >> I'm on Windows XP using LLVM cvs build with VC++ 7.1. > ... > >> ... >> opt: Invalid Top Level Block Length! Type:1, Size:49557 (Vers=0, >> Pos=8) > > This sounds like an LLVM problem: the assembler is producing an > invalid bytecode file. Either the .ll file is broken (in which case > the assembler should have rejected it) or the bytecode writer is doing > something wrong. In any case, if you gzip up the source and email it > to me, I'd be happy to look at it. I would be happy to do so, but what is the location of the generated .ll ? > -Chris > >> With PyPy SVN head, I get a more direct error: >>>>> print t.llvm() >> [...] truncated >> Traceback (most recent call last): >> File "", line 1, in ? >> File "E:\prg\py\App\pypy\pypy\translator\translator.py", line 201, >> in llvm filename = gen.gen_llvm_source() >> File "E:\prg\py\App\pypy\pypy\translator\llvm\genllvm.py", line 179, >> in gen_llvm_source >> codewriter.append(self.gcpolicy.declarations()) >> AttributeError: 'NoneType' object has no attribute 'declarations' >> >> Does anyone know what I'm doing wrong ? >> >> On another topic, where do the generated files go ? I tried to find >> the files generated by the c and llvm translation but couldn't find >> them... >> >> Baptiste. >> _______________________________________________ >> pypy-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/pypy-dev >> > > -Chris From eric at vanrietpaap.nl Sun Sep 11 18:53:07 2005 From: eric at vanrietpaap.nl (Eric van Riet Paap) Date: Sun, 11 Sep 2005 18:53:07 +0200 Subject: [pypy-dev] Translating code to LLVM In-Reply-To: <008e01c5b6a7$9d0aede0$0300a8c0@anabelle> References: <008001c5b659$7e50db60$0300a8c0@anabelle> <008e01c5b6a7$9d0aede0$0300a8c0@anabelle> Message-ID: <432460F3.80509@vanrietpaap.nl> Hello, We not yet tested the LLVM on Windows, but I believe the files will be stored in the \temp\usession-* directories. The current SVN head is not compatible with the current translate_pypy . I would suggest you stick with pypy0.7 for testing the basic functionality. I think it is also a good idea to run the unittests in pypy/translator/llvm/test first to see if things work. cheers Eric Baptiste Lepilleur schreef: > Chris Lattner wrote: > >> On Sun, 11 Sep 2005, Baptiste Lepilleur wrote: >> >>> Following the getting started instruction, I'm trying to get the LLVM >>> translation pgcd example below. >>> >>> I'm on Windows XP using LLVM cvs build with VC++ 7.1. >> >> ... >> >>> ... >>> opt: Invalid Top Level Block Length! Type:1, Size:49557 (Vers=0, >>> Pos=8) >> >> >> This sounds like an LLVM problem: the assembler is producing an >> invalid bytecode file. Either the .ll file is broken (in which case >> the assembler should have rejected it) or the bytecode writer is doing >> something wrong. In any case, if you gzip up the source and email it >> to me, I'd be happy to look at it. > > > I would be happy to do so, but what is the location of the generated > .ll ? > >> -Chris >> >>> With PyPy SVN head, I get a more direct error: >>> >>>>>> print t.llvm() >>>>> >>> [...] truncated >>> Traceback (most recent call last): >>> File "", line 1, in ? >>> File "E:\prg\py\App\pypy\pypy\translator\translator.py", line 201, >>> in llvm filename = gen.gen_llvm_source() >>> File "E:\prg\py\App\pypy\pypy\translator\llvm\genllvm.py", line 179, >>> in gen_llvm_source >>> codewriter.append(self.gcpolicy.declarations()) >>> AttributeError: 'NoneType' object has no attribute 'declarations' >>> >>> Does anyone know what I'm doing wrong ? >>> >>> On another topic, where do the generated files go ? I tried to find >>> the files generated by the c and llvm translation but couldn't find >>> them... >>> Baptiste. >>> _______________________________________________ >>> pypy-dev at codespeak.net >>> http://codespeak.net/mailman/listinfo/pypy-dev >>> >> >> -Chris > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev From tismer at stackless.com Mon Sep 12 03:29:19 2005 From: tismer at stackless.com (Christian Tismer) Date: Mon, 12 Sep 2005 03:29:19 +0200 Subject: [pypy-dev] funny observation with range Message-ID: <4324D9EF.2050501@stackless.com> Hi Armin, when I added a few tests for the new variable step variant, I observed the following behavior of ranges: When a variable is conditionally assigned two different ranges, the ranges are kept, unless the steps are different. Well, after all this is not too surprizing, since we are compiling the step constant in. But I thing the action taken is not so pleasant: The range is turned into a list in this case! I guess what we want is either special-casing the respective blocks, or at least map this to the variable step case, if this is possible. I would like to understand why the annotator thinks this must become a list, and how I can convince it to take a different action. I think, using the variable variant would be adequate in this case. A test_range_funny was added to test_rrange.py, not really meant as a test, but to explain that. Please let me know what you think. cheers - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From pedronis at strakt.com Mon Sep 12 04:19:50 2005 From: pedronis at strakt.com (Samuele Pedroni) Date: Mon, 12 Sep 2005 04:19:50 +0200 Subject: [pypy-dev] funny observation with range In-Reply-To: <4324D9EF.2050501@stackless.com> References: <4324D9EF.2050501@stackless.com> Message-ID: <4324E5C6.9050009@strakt.com> Christian Tismer wrote: > Hi Armin, > > when I added a few tests for the new variable step variant, > I observed the following behavior of ranges: > > When a variable is conditionally assigned two different ranges, > the ranges are kept, unless the steps are different. > Well, after all this is not too surprizing, since we are compiling > the step constant in. > But I thing the action taken is not so pleasant: > The range is turned into a list in this case! > > I guess what we want is either special-casing the respective blocks, > or at least map this to the variable step case, if this is possible. > > I would like to understand why the annotator thinks this must > become a list, and how I can convince it to take a different action. > I think, using the variable variant would be adequate in this case. you changed what happens on range calls annotation-wise, but not the behavior of what should occur when types are mixed, this is specified by the union operation which for list/ranges basically goes through the ListItem.merge, which still has the behavior that mixing different-step ranges unifies to list: if other.range_step != self.range_step: self.range_step = None in listdef.py . If I have understood your changes you want 0 in this case unless one of the range_step is already None meaning list. Notice that if you change this you probably need to change rtype_builtin_range because this assumes that it can use partially the information in its arguments instead of about the result to decide what to produce. > A test_range_funny was added to test_rrange.py, not really > meant as a test, but to explain that. > > Please let me know what you think. > > cheers - chris > From tismer at stackless.com Mon Sep 12 08:13:33 2005 From: tismer at stackless.com (Christian Tismer) Date: Mon, 12 Sep 2005 08:13:33 +0200 Subject: [pypy-dev] funny observation with range In-Reply-To: <4324E5C6.9050009@strakt.com> References: <4324D9EF.2050501@stackless.com> <4324E5C6.9050009@strakt.com> Message-ID: <43251C8D.7050406@stackless.com> Samuele Pedroni wrote: > Christian Tismer wrote: ... >> I would like to understand why the annotator thinks this must >> become a list, and how I can convince it to take a different action. >> I think, using the variable variant would be adequate in this case. > > you changed what happens on range calls annotation-wise, but not the > behavior of what should occur when types are mixed, this is specified by > the union operation which for list/ranges basically goes through the > ListItem.merge, which still has the behavior that mixing different-step > ranges unifies to list: > > if other.range_step != self.range_step: > self.range_step = None > > in listdef.py . If I have understood your changes you want 0 in this > case unless one of the range_step is already None meaning list. > > Notice that if you change this you probably need to change > rtype_builtin_range because this assumes that it can use partially the > information in its arguments instead of about the result to decide > what to produce. Ah, thank you very much, that was the missing spot. No magic at all, but more places to look into. I will add something to cope with this. Although I don't believe it is worthwhile to do in this case, I guess it would be much more difficult to specialize on step? thanks again - ciao - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From Amaury.Forgeotdarc at Ubitrade.Com Tue Sep 13 14:17:47 2005 From: Amaury.Forgeotdarc at Ubitrade.Com (Amaury Forgeot D Arc) Date: Tue, 13 Sep 2005 14:17:47 +0200 Subject: [pypy-dev] unionof and dictionaries Message-ID: Hello, While playing with the annotator/translator, I found a strange test case. The following code works: from pypy.annotation.model import * from pypy.annotation.dictdef import DictDef def test_dictdef(): def1 = DictDef(None, SomeInteger(), SomeInteger()) def2 = DictDef(None, SomeInteger(), SomeInteger()) dic1 = SomeDict(def1) dic2 = SomeDict(def2) assert not dic1 == dic2 assert not dic1.contains(dic2) x = unionof(dic1, dic2) assert dic1 == dic2 # !!!!!!! assert dic1.contains(dic2) # !!!!!!! Is the "unionof" function expected to have side-effects? Are the first two tests even correct? I would expect that two dicts defined the same way would compare equal. -- Amaury Forgeot d'Arc Ubix Development www.ubitrade.com From eric at vanrietpaap.nl Tue Sep 13 17:32:57 2005 From: eric at vanrietpaap.nl (Eric van Riet Paap) Date: Tue, 13 Sep 2005 17:32:57 +0200 Subject: [pypy-dev] Re: PyPy - Physical Persons In-Reply-To: <4326C5D3.7080906@dfki.de> References: <4326C5D3.7080906@dfki.de> Message-ID: <200509131732.57981.eric@vanrietpaap.nl> Hello Stephan, Will do that, assuming passport copies will be equally fine. But perhaps you can first tell us a little bit more about the current situation. From what I read on pypy-funding the whole PP accession seems uncertain. regards Eric On Tuesday 13 September 2005 14:28, you wrote: > Dear All, > > The PO needs see copies of your ID cards. Can you please fax a good > copie (both sides)? > > Thank you and best regards > Stephan From bea at netg.se Tue Sep 13 17:52:25 2005 From: bea at netg.se (Beatrice During) Date: Tue, 13 Sep 2005 17:52:25 +0200 (CEST) Subject: [pypy-dev] Re: PyPy - Physical Persons In-Reply-To: <200509131732.57981.eric@vanrietpaap.nl> References: <4326C5D3.7080906@dfki.de> <200509131732.57981.eric@vanrietpaap.nl> Message-ID: Hi there On Tue, 13 Sep 2005, Eric van Riet Paap wrote: > Hello Stephan, > > Will do that, assuming passport copies will be equally fine. > But perhaps you can first tell us a little bit more about the current > situation. From what I read on pypy-funding the whole PP accession seems > uncertain. > > regards > Eric Maybe I can try to answer that. The entire structure of physical persons in Pypy doesnt really fit the usual profile of consortium and partners. Even physical persons are full partners and thus need to participate fully in the consortium - being liable and all, participating in our consortium meetings, admin etc. This we didnt want for pypydev people contributing to the project so we created an Association Agreement (AA)in which we specify that our physical persons have specific limitations on responsibilities and liabilities. This is a complement to the consortium agreement that all partners sign. The PO said - when his admin staff started to look at the 2 amendment - that our AA wasnt "doable" - all partners are full partners in FP6. What I did (and sent to pypy-funding) was to find an old email from him where we validated the entire procedure around PP:s (in feb). In that email he states that it is an internal matter to the consortium whether there are certain limitations in responsibilities and liabilities, it has noting to do with rules in FP6. So Stephan and I talked and I think the best strategy is to push this with the PO and remind him that he is in conflict with data he himself provided us while we were working out this procedure. (This was also validated with the german EU office and their lawyer). So Stephan is to call the PO (I think/hope he has reached him today) and to reiterate our procedure, with this old email from him as "artillery". If they still persist we need to have a meeting with them in Brussels pronto, this is even more critical than the two amendments in pipeline. This has to do with our entire strategy and organisation of the consortium and the connection to the pypy community. Please rest assured that we will do _a lot_ and will go to great lengths to get this solved with the Commission as fast as possible. I am very sorry that this matter is being dragged out like this - you must believe us though that we thought that our procedure was validated and confirmed. We are very grateful for you and others who have continued, on private funding, to join our sprints and contributed huge amounts of work. Pypy wouldnt be where it is today if it where not for you and others - contributing despite the EU bureacracy. Cheers Bea Beatrice D?ring Change Maker Tel: 031- 7750940 J?rntorget 3 Mobil: 0734- 22 89 06 413 04 G?teborg E-post: bea at changemaker.nu www.changemaker.nu "Alla dessa m?sten och alldaglighet. Allt detta som binder v?r verklighet i bojor av skam och rep utav tv?ng. Alla dessa kedjor som binder v?r s?ng. Jag skall slita dem alla i sm?, sm? stycken och m?jligtvis av resterna g?ra mig smycken." - hemlig From arigo at tunes.org Wed Sep 14 14:58:37 2005 From: arigo at tunes.org (Armin Rigo) Date: Wed, 14 Sep 2005 14:58:37 +0200 Subject: [pypy-dev] unionof and dictionaries In-Reply-To: References: Message-ID: <20050914125837.GA27665@code1.codespeak.net> Hi Amaury, On Tue, Sep 13, 2005 at 02:17:47PM +0200, Amaury Forgeot D Arc wrote: > While playing with the annotator/translator, I found > a strange test case. The test as you reported it shows the intended behavior of DictDefs; what did you expect, i.e. what are you trying to achieve? Dicts and lists are a delicate part of the annotator because they are mutable objects. The annotator's goal here is to figure out what kind of objects will *ever* be put in the dict. It's the purpose of the DictDefs to record (as a side-effect) what is done with the dicts. > def test_dictdef(): > def1 = DictDef(None, SomeInteger(), SomeInteger()) > def2 = DictDef(None, SomeInteger(), SomeInteger()) > dic1 = SomeDict(def1) > dic2 = SomeDict(def2) > assert not dic1 == dic2 The two dicts are unrelated; the annotator thinks they come from different creation points. Until proven otherwise the two SomeDict()s represent two unrelated sets of possible run-time dictionaries. > x = unionof(dic1, dic2) This means that the two dicts actually go into the same place, so that the two dicts are now known to be just two different creation points for what is essentially the same kind of dict. So unionof() has the side-effect of making def1 and def2 complete synonyms. Now the two SomeDict()s stand for the same possibly larger set of run-time dictionaries. A bientot, Armin. From Amaury.Forgeotdarc at Ubitrade.Com Wed Sep 14 19:09:44 2005 From: Amaury.Forgeotdarc at Ubitrade.Com (Amaury Forgeot D Arc) Date: Wed, 14 Sep 2005 19:09:44 +0200 Subject: [pypy-dev] unionof and dictionaries Message-ID: Hello Armin, > The test as you reported it shows the intended behavior of DictDefs; > what did you expect, i.e. what are you trying to achieve? Yesterday (before r17525) there was an assertion about the contains() method when used on SomeIterators. I tried to see what was going on, and why a SomeIterator did not contain itself. In the pdb session, I tried some unionof() statements, and was very confused by the non-reproducible results... > The two dicts are unrelated; the annotator thinks they come from > different creation points. Until proven otherwise the two SomeDict()s > represent two unrelated sets of possible run-time dictionaries. This makes sense, and is what I had in mind. > > > x = unionof(dic1, dic2) > > This means that the two dicts actually go into the same place, so that > the two dicts are now known to be just two different creation points for > what is essentially the same kind of dict. So unionof() has the > side-effect of making def1 and def2 complete synonyms. Now the two > SomeDict()s stand for the same possibly larger set of run-time > dictionaries. Ah, I did not see the "no_side_effects_in_union" trick. Now I understand that union() is meant to have the side-effect you describe, and that contains() has no side-effect and is safe to use in assertions. Thanks for your patient explanations. -- Amaury Forgeot d'Arc Ubix Development www.ubitrade.com From tismer at stackless.com Thu Sep 15 01:36:44 2005 From: tismer at stackless.com (Christian Tismer) Date: Thu, 15 Sep 2005 01:36:44 +0200 Subject: [pypy-dev] next pypy-sync meeting 20050915 Message-ID: <4328B40C.2060809@stackless.com> Hi all, Next #pypy-sync meeting on freenode.net is tomorrow Thursday at 1pm as usual. Regular topics -------------- * activity reports (3 prepared lines of info) * resolve conflicts/blockers Here is my draft proposal of Topics of the week ------------------ A quick point about IRC logging I (Chris) have set up an eggdrop server on tismerysoft.de. This server logs #pypy, #pypy-funding (password), #pypy-sync, #pypy-tb. Let me know tomorrow if it should log something else. To make this service more reliable, I would like to add a second server. I could use another one of mine, but this makes no sense in using the same network. I'd like to ask which other server to use, maybe codespeak, maybe Armin's new server. Anyway, I'd like to take responsibility and the rights to set that up, since I did it before. Participation on the CCC congress in Berlin There is a CCC congress in Berlin 2005?12?27 to 2005?12?30. See http://wiki.chaostreff.ch/index.php/Main_Page They told us on IRC that they are interested in talks from us. This appeared to me to be quite unspecific, they seem to be interested in scientific talks. For PyPy, I anyway thing that it would be good not to emphasize too much on Python itself, but the general aspectsof translation which are possible to tackle with out framework. Se also Armin's draft in pypy/doc. We should collect who wants to give a talk or just attend this congress. We also might talk about how we manage claims about giving a talk. Practice has been to say it somewhere at some time often enough, but I'm not sure if this is a pleasant strategy for everybody to continue with. I would like to ask people who would like to attend in any way to inform me early, because I probably can provide help in finding locals who can share a room. Current optimization activities and allocation of work There are currently several ongoings on optimization, where I can't remember that they were assigned officially in any way. Practice seems to be to justpick an issue which has a good chance to provide a 200 % speedup and then go for it. I am personally not really happy with this and would like to ask about opinions. My proposal is to have a more informal way to decide about activities, by issuing a short IRC meeting and giving everybody a chance to ask for participation. We should also be open to discuss reasons for an individual's rejection. This might be painful, but setting facts by just doing things is a worse practice IMHO. Other activities, follow-up on sync meeting of last week As a conclusion from last week's meeting, docs and reports appear to reduce to a meeting of Holger, Carl, Samuele and Armin one day before the Paris sprint. I would like to ask again, if this was meant this way. There is quite some time left until Paris. Can we talk again about what we want to work on in the meantime, if it is not just optimization? -- The next topic is related, maybe this is a merger. Making money out of PyPy Jacob asked me on the last sprint, if it is possible to make money out of PyPy right now. I passed this around once, but would like to insist. I am personally interested to make PyPy move away a bit from its purely academical status and to think a little bit about how we can create practical applications in the near future, which allow to grow marketable services for our members, in order to reduce PyPy's dependency from the EU sponsoring. Please propose additions and changes, I just listed what immediately came to my mind. I'm also happy if you say some topics are too big and should be moved to another meeting. See you tomorrow at 1 pm. sincerely -- chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From michal at sabren.com Thu Sep 15 04:17:12 2005 From: michal at sabren.com (Michal Wallace) Date: Wed, 14 Sep 2005 22:17:12 -0400 (EDT) Subject: [pypy-dev] next pypy-sync meeting 20050915 In-Reply-To: <4328B40C.2060809@stackless.com> References: <4328B40C.2060809@stackless.com> Message-ID: On Thu, 15 Sep 2005, Christian Tismer wrote: > Making money out of PyPy > > Jacob asked me on the last sprint, if it is possible to make > money out of PyPy right now. I passed this around once, but would > like to insist. I am personally interested to make PyPy move > away a bit from its purely academical status and to think a little > bit about how we can create practical applications in the near future, > which allow to grow marketable services for our members, in order > to reduce PyPy's dependency from the EU sponsoring. Hey all, I've only been reading these updates, and I haven't actually played with PyPy in while, so I could be way off here. But I do run a business and know a little bit about marketing, and I head up another python compiler project (pirate) so I know a little bit about the market demand for these things... :) Basically, there are really that many applications of this technology in its current state. The stated goal for this project is just to make a faster version of python, but can you really charge for that given that pypy is already open source? One option could be to find people who need python to run fast and get them to pay you to handle the cases that apply to *their project* up front, but I really don't think this is a viable market. It's much cheaper for the client to just write a C extension or port the whole thing to C. Another option is that you could use your technology created in pypy to create new *frontends* for pypy, and sell the service of taking legacy code in other languages, and translating that into call trees, and turning it magically into python or lisp or c... This seems like it could be a much more lucrative service, but it's a whole new research project and it takes you away from your core focus on python. (On the other hand, a company that did this would have a massive incentive to sponsor pypy development... Maybe there's a company that already does this kind of thing with some other technology... Like, say, people just rewriting things by hand? Another idea has to do with the sprints. I already told Holger this a while back, but I think you guys have one of the most impressive project management styles around, and the sprint idea seems like a great amount of fun. What if you capitalized on *that*? A week in europe to work on an open source project? That would be an awesome vacation! And to get to learn about compilers and python along the way? What I'm saying is you could market the sprints as a sort of training package for big companies interested in trying out python. They send their developers and pay you guys to train them. Even if the companies don't care about python, their developers *might* and that could be a great reward. Wasn't there a post just like this from someone in the army a while back? Market demand. :) Heck, do your next sprint as a partnership with these guys: http://www.geekcruises.com/ Anyway, it might be a crazy idea, but if you could get it working, it would capitalize on what you're already doing rather than force you to come up with some other devlepoment project off to the side... Sincerely, Michal J Wallace Sabren Enterprises, Inc. ------------------------------------- contact: michal at sabren.com hosting: http://www.cornerhost.com/ my site: http://www.withoutane.com/ ------------------------------------- From hpk at trillke.net Thu Sep 15 08:58:24 2005 From: hpk at trillke.net (holger krekel) Date: Thu, 15 Sep 2005 08:58:24 +0200 Subject: [pypy-dev] next pypy-sync meeting 20050915 In-Reply-To: References: <4328B40C.2060809@stackless.com> Message-ID: <20050915065824.GO22844@solar.trillke.net> Hi Michal, On Wed, Sep 14, 2005 at 22:17 -0400, Michal Wallace wrote: > On Thu, 15 Sep 2005, Christian Tismer wrote: > > > Making money out of PyPy > > > > Jacob asked me on the last sprint, if it is possible to make > > money out of PyPy right now. I passed this around once, but would > > like to insist. I am personally interested to make PyPy move > > away a bit from its purely academical status and to think a little > > bit about how we can create practical applications in the near future, > > which allow to grow marketable services for our members, in order > > to reduce PyPy's dependency from the EU sponsoring. > > Hey all, > > I've only been reading these updates, and > I haven't actually played with PyPy in > while, so I could be way off here. But I > do run a business and know a little bit > about marketing, and I head up another > python compiler project (pirate) so I > know a little bit about the market > demand for these things... :) > > > Basically, there are really that many > applications of this technology in its > current state. The stated goal for this > project is just to make a faster version of > python, but can you really charge for that > given that pypy is already open source? PyPy is not primarily - at least not only - about making Python faster although many people like to view it this way :-) PyPy is a lot about flexibility to produce custom implementations of Python for use in particular environments. Regarding persistence, security (e.g. sandboxing) and deployment there is a lot that we can possiblly do with PyPy implementations. I am sure that such possibilities will become clearer next year. > One option could be to find people who > need python to run fast and get them to > pay you to handle the cases that apply > to *their project* up front, but I really > don't think this is a viable market. It's > much cheaper for the client to just write > a C extension or port the whole thing to C. > > Another option is that you could use your > technology created in pypy to create new > *frontends* for pypy, and sell the service > of taking legacy code in other languages, > and translating that into call trees, and > turning it magically into python or lisp or c... > This seems like it could be a much more > lucrative service, but it's a whole new > research project and it takes you away > from your core focus on python. (On the > other hand, a company that did this would > have a massive incentive to sponsor pypy > development... Maybe there's a company that > already does this kind of thing with some > other technology... Like, say, people just > rewriting things by hand? A build tool is part of the current EU project and of our plans. > Another idea has to do with the sprints. > I already told Holger this a while back, > but I think you guys have one of the most > impressive project management styles around, > and the sprint idea seems like a great > amount of fun. > > What if you capitalized on *that*? > > A week in europe to work on an open source > project? That would be an awesome vacation! > And to get to learn about compilers and python > along the way? We are doing this regularly :-) > What I'm saying is you could market the > sprints as a sort of training package for > big companies interested in trying out python. > They send their developers and pay you guys > to train them. Even if the companies don't > care about python, their developers *might* > and that could be a great reward. > > Wasn't there a post just like this from > someone in the army a while back? > > Market demand. :) :-) > Heck, do your next sprint as a partnership with > these guys: > > http://www.geekcruises.com/ > > Anyway, it might be a crazy idea, but if you > could get it working, it would capitalize > on what you're already doing rather than > force you to come up with some other devlepoment > project off to the side... There are many possibilities but only so much time and focus to pursue them. Bea also reported quite some interest from other projects and entities in the EU context. We got invitations to Afrika and other places and i guess we pursue them :-) Thanks for your kind advice, holger From bea at netg.se Thu Sep 15 10:16:56 2005 From: bea at netg.se (Beatrice During) Date: Thu, 15 Sep 2005 10:16:56 +0200 (CEST) Subject: [pypy-dev] next pypy-sync meeting 20050915 In-Reply-To: References: <4328B40C.2060809@stackless.com> Message-ID: Hi there Just a short comment to your email Michael (thanks for your thoughts onj this matter BTW ;-): One of the current major discussions about how to capitalize on Open Source as a business model is just that - learning/training. So I think you are on to something there Michael - both regarding the technical aspects and the project management/developer process aspects. Sprinting combines them both ;-) Cheers Bea On Wed, 14 Sep 2005, Michal Wallace wrote: > On Thu, 15 Sep 2005, Christian Tismer wrote: > >> Making money out of PyPy >> >> Jacob asked me on the last sprint, if it is possible to make >> money out of PyPy right now. I passed this around once, but would >> like to insist. I am personally interested to make PyPy move >> away a bit from its purely academical status and to think a little >> bit about how we can create practical applications in the near future, >> which allow to grow marketable services for our members, in order >> to reduce PyPy's dependency from the EU sponsoring. > > Hey all, > > I've only been reading these updates, and > I haven't actually played with PyPy in > while, so I could be way off here. But I > do run a business and know a little bit > about marketing, and I head up another > python compiler project (pirate) so I > know a little bit about the market > demand for these things... :) > > > Basically, there are really that many > applications of this technology in its > current state. The stated goal for this > project is just to make a faster version of > python, but can you really charge for that > given that pypy is already open source? > > One option could be to find people who > need python to run fast and get them to > pay you to handle the cases that apply > to *their project* up front, but I really > don't think this is a viable market. It's > much cheaper for the client to just write > a C extension or port the whole thing to C. > > Another option is that you could use your > technology created in pypy to create new > *frontends* for pypy, and sell the service > of taking legacy code in other languages, > and translating that into call trees, and > turning it magically into python or lisp or c... > This seems like it could be a much more > lucrative service, but it's a whole new > research project and it takes you away > from your core focus on python. (On the > other hand, a company that did this would > have a massive incentive to sponsor pypy > development... Maybe there's a company that > already does this kind of thing with some > other technology... Like, say, people just > rewriting things by hand? > > Another idea has to do with the sprints. > I already told Holger this a while back, > but I think you guys have one of the most > impressive project management styles around, > and the sprint idea seems like a great > amount of fun. > > What if you capitalized on *that*? > > A week in europe to work on an open source > project? That would be an awesome vacation! > And to get to learn about compilers and python > along the way? > > What I'm saying is you could market the > sprints as a sort of training package for > big companies interested in trying out python. > They send their developers and pay you guys > to train them. Even if the companies don't > care about python, their developers *might* > and that could be a great reward. > > Wasn't there a post just like this from > someone in the army a while back? > > Market demand. :) > > Heck, do your next sprint as a partnership with > these guys: > > http://www.geekcruises.com/ > > Anyway, it might be a crazy idea, but if you > could get it working, it would capitalize > on what you're already doing rather than > force you to come up with some other devlepoment > project off to the side... > > Sincerely, > > Michal J Wallace > Sabren Enterprises, Inc. > ------------------------------------- > contact: michal at sabren.com > hosting: http://www.cornerhost.com/ > my site: http://www.withoutane.com/ > ------------------------------------- > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > Beatrice D?ring Change Maker Tel: 031- 7750940 J?rntorget 3 Mobil: 0734- 22 89 06 413 04 G?teborg E-post: bea at changemaker.nu www.changemaker.nu "Alla dessa m?sten och alldaglighet. Allt detta som binder v?r verklighet i bojor av skam och rep utav tv?ng. Alla dessa kedjor som binder v?r s?ng. Jag skall slita dem alla i sm?, sm? stycken och m?jligtvis av resterna g?ra mig smycken." - hemlig From mwh at python.net Thu Sep 15 11:04:09 2005 From: mwh at python.net (Michael Hudson) Date: Thu, 15 Sep 2005 10:04:09 +0100 Subject: [pypy-dev] Re: next pypy-sync meeting 20050915 References: <4328B40C.2060809@stackless.com> Message-ID: <2m8xxyn1me.fsf@starship.python.net> Christian Tismer writes: > Hi all, > > Next #pypy-sync meeting on freenode.net is tomorrow Thursday at 1pm as > usual. I'll try to remember when it starts, this time... > Regular topics > -------------- > > * activity reports (3 prepared lines of info) > * resolve conflicts/blockers > > Here is my draft proposal of > > Topics of the week > ------------------ > > A quick point about IRC logging > > I (Chris) have set up an eggdrop server on tismerysoft.de. > This server logs > #pypy, #pypy-funding (password), #pypy-sync, #pypy-tb. > Let me know tomorrow if it should log something else. > To make this service more reliable, I would like to add > a second server. I could use another one of mine, but this > makes no sense in using the same network. > I'd like to ask which other server to use, maybe codespeak, > maybe Armin's new server. Anyway, I'd like to take responsibility > and the rights to set that up, since I did it before. The starship? I think you already have an account there :) > Participation on the CCC congress in Berlin Is the event in German? I guess from the fact that the website it, the event probably is too... > Current optimization activities and allocation of work > > There are currently several ongoings on optimization, where I > can't remember that they were assigned officially in any way. > Practice seems to be to justpick an issue which has a good chance > to provide a 200 % speedup and then go for it. I am personally > not really happy with this and would like to ask about opinions. > My proposal is to have a more informal way to decide about > activities, by issuing a short IRC meeting and giving everybody > a chance to ask for participation. We should also be open to > discuss reasons for an individual's rejection. This might be > painful, but setting facts by just doing things is a worse > practice IMHO. It seems to me that rather than saying "inlining will make us faster" or "string interning will make us faster" or whatever, some effort needs to go into finding out why we are slow. > Making money out of PyPy > > Jacob asked me on the last sprint, if it is possible to make > money out of PyPy right now. Well, my uninformed opinion would be "no, not right now". > I passed this around once, but would like to insist. I am > personally interested to make PyPy move away a bit from its > purely academical status and to think a little bit about how we > can create practical applications in the near future, which allow > to grow marketable services for our members, in order to reduce > PyPy's dependency from the EU sponsoring. Lots of people are interested (though maybe they shouldn't be) in a non-GIL implementation of Python. Offering different concurrency models is a really big selling point of PyPy, I think. Cheers, mwh -- That one is easily explained away as massively intricate conspiracy, though. -- Chris Klein, alt.sysadmin.recovery From lac at strakt.com Thu Sep 15 11:49:02 2005 From: lac at strakt.com (Laura Creighton) Date: Thu, 15 Sep 2005 11:49:02 +0200 Subject: [pypy-dev] Re: next pypy-sync meeting 20050915 In-Reply-To: Message from Michael Hudson of "Thu, 15 Sep 2005 10:04:09 BST." <2m8xxyn1me.fsf@starship.python.net> References: <4328B40C.2060809@stackless.com> <2m8xxyn1me.fsf@starship.python.net> Message-ID: <200509150949.j8F9n2c3025608@theraft.strakt.com> In a message of Thu, 15 Sep 2005 10:04:09 BST, Michael Hudson writes: >Christian Tismer writes: >Is the event in German? I guess from the fact that the website it, >the event probably is too... No, there are talks in all languages. Lots of German ones, 'natch. re: making money -- lightweight threading model, anyone? Laura From faassen at infrae.com Thu Sep 15 12:42:29 2005 From: faassen at infrae.com (Martijn Faassen) Date: Thu, 15 Sep 2005 12:42:29 +0200 Subject: [pypy-dev] a faster Python not a primary goal of PyPy? In-Reply-To: <20050915065824.GO22844@solar.trillke.net> References: <4328B40C.2060809@stackless.com> <20050915065824.GO22844@solar.trillke.net> Message-ID: <43295015.2060109@infrae.com> holger krekel wrote: [snip] > PyPy is not primarily - at least not only - about making > Python faster although many people like to view it this way :-) This correction is a bit confusing... It is not surprising that people view "a faster Python" as a primary goal of your project, as that's the message that's been spread repeatedly by you all over the years: "The PyPy project aims at producing a flexible and fast Python implementation. The guiding idea is to translate a Python-level description of the Python language itself to lower level languages. Rumors have it that the secret goal is being faster-than-C which is nonsense, isn't it?" and "In the next step of the project, we will generate C code or machine code from the source of Pypy, thereby reducing the speed penalty. Later in the project, we will introduce optimisation steps that should make Pypy run faster than CPython." and "Psyco is a Python extension module which can massively speed up the execution of any Python code. ... The future of Psyco now lies in the PyPy project, which according to plan will provide a good base for a Python interpreter with better and well-integrated Psyco-like techniques as soon as 2006." I certainly hope that your "not primarily - at least not only" doesn't indicate that the goal of making Python faster is slowly becoming less important to you all, as I'd very much like a faster Python, and more flexibility of implementation is, while of tremendous personal interest to me, not one of much business interest in the foreseeable future, which means it's unlikely I'll get to play with PyPy much unless it offers performance benefits. I realize that of course that flexibility has always been a primary goal of PyPy as well; obviously speed isn't the *only* goal of the project, but the flexibility goal now dominates and a faster Python is indeed not of primary importance to the project, I would suggest you revise the previous statements, which make it rather likely people will get this wrong impression. Regards, Martijn From jacob at strakt.com Thu Sep 15 12:52:55 2005 From: jacob at strakt.com (Jacob =?iso-8859-1?q?Hall=E9n?=) Date: Thu, 15 Sep 2005 12:52:55 +0200 Subject: [pypy-dev] a faster Python not a primary goal of PyPy? In-Reply-To: <43295015.2060109@infrae.com> References: <4328B40C.2060809@stackless.com> <20050915065824.GO22844@solar.trillke.net> <43295015.2060109@infrae.com> Message-ID: <200509151252.55437.jacob@strakt.com> torsdagen den 15 september 2005 12.42 skrev Martijn Faassen: > holger krekel wrote: > [snip] > > > PyPy is not primarily - at least not only - about making > > Python faster although many people like to view it this way :-) > > This correction is a bit confusing... It is not surprising that people > view "a faster Python" as a primary goal of your project, as that's the > message that's been spread repeatedly by you all over the years: I think you misunderstand Holger. Making Python faster is _a_ goal with Pypy, but it is not _the_ goal. The most important goals, as I see it are to produce something that is flexible and which has a clear implementation. However, this becomes an academic exercise if we produce something that is slower than competing implementations of Python. We want to build something that produces actual benefits in real world applications. Jacob From tismer at stackless.com Thu Sep 15 12:57:42 2005 From: tismer at stackless.com (Christian Tismer) Date: Thu, 15 Sep 2005 12:57:42 +0200 Subject: [pypy-dev] a faster Python not a primary goal of PyPy? In-Reply-To: <200509151252.55437.jacob@strakt.com> References: <4328B40C.2060809@stackless.com> <20050915065824.GO22844@solar.trillke.net> <43295015.2060109@infrae.com> <200509151252.55437.jacob@strakt.com> Message-ID: <432953A6.6050103@stackless.com> Jacob Hall?n wrote: > torsdagen den 15 september 2005 12.42 skrev Martijn Faassen: > >>holger krekel wrote: >>[snip] >> >> >>>PyPy is not primarily - at least not only - about making >>>Python faster although many people like to view it this way :-) >> >>This correction is a bit confusing... It is not surprising that people >>view "a faster Python" as a primary goal of your project, as that's the >>message that's been spread repeatedly by you all over the years: > > > I think you misunderstand Holger. Making Python faster is _a_ goal with Pypy, > but it is not _the_ goal. The most important goals, as I see it are to > produce something that is flexible and which has a clear implementation. > However, this becomes an academic exercise if we produce something that is > slower than competing implementations of Python. We want to build something > that produces actual benefits in real world applications. That's the point. Not prioritizing speed is fine, but it does not allow us to be remarkably slower. Being of competitive speed and then more flexible is a must. If we are slower, the market will not consider us at all. ciao - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From faassen at infrae.com Thu Sep 15 14:57:11 2005 From: faassen at infrae.com (Martijn Faassen) Date: Thu, 15 Sep 2005 14:57:11 +0200 Subject: [pypy-dev] a faster Python not a primary goal of PyPy? In-Reply-To: <200509151252.55437.jacob@strakt.com> References: <4328B40C.2060809@stackless.com> <20050915065824.GO22844@solar.trillke.net> <43295015.2060109@infrae.com> <200509151252.55437.jacob@strakt.com> Message-ID: <43296FA7.2010807@infrae.com> Jacob Hall?n wrote: > torsdagen den 15 september 2005 12.42 skrev Martijn Faassen: > >> holger krekel wrote: [snip] >> >>> PyPy is not primarily - at least not only - about making Python >>> faster although many people like to view it this way :-) >> >> This correction is a bit confusing... It is not surprising that >> people view "a faster Python" as a primary goal of your project, as >> that's the message that's been spread repeatedly by you all over >> the years: > > I think you misunderstand Holger. I'm not sure how to understand Holger. He said PyPy is "not primarily - at least not only" about making a faster Python, and was in the context of correcting someone. I figured that perhaps the PyPy project is not bringing the message out as well as possible, whatever this message is. > Making Python faster is _a_ goal with Pypy, but it is not _the_ goal. > The most important goals, as I see it are to produce something that > is flexible and which has a clear implementation. However, this > becomes an academic exercise if we produce something that is slower > than competing implementations of Python. We want to build something > that produces actual benefits in real world applications. So, is making a faster Python a *primary* goal of PyPy or not? I understand that the way to go about this is to produce a clear, flexible implementation, and that this is also cool, and also a primary goal by itself, but is making a faster Python *also* a primary goal or not? If it's a primary goal, why the correction? If not, the message you all are putting across might need some clarification, as you're certainly implying to me in the texts I quoted that it is a primary goal. Regards, Martijn From faassen at infrae.com Thu Sep 15 15:00:32 2005 From: faassen at infrae.com (Martijn Faassen) Date: Thu, 15 Sep 2005 15:00:32 +0200 Subject: [pypy-dev] a faster Python not a primary goal of PyPy? In-Reply-To: <432953A6.6050103@stackless.com> References: <4328B40C.2060809@stackless.com> <20050915065824.GO22844@solar.trillke.net> <43295015.2060109@infrae.com> <200509151252.55437.jacob@strakt.com> <432953A6.6050103@stackless.com> Message-ID: <43297070.4020004@infrae.com> Christian Tismer wrote: [snip] > That's the point. Not prioritizing speed is fine, but it > does not allow us to be remarkably slower. Being of competitive > speed and then more flexible is a must. If we are slower, the > market will not consider us at all. That's true, and that's a useful clarification. It's still different from the impression you're giving out, where you're strongly implying at least to me that speed will be significantly *faster* eventually with PyPy. Then again, it's forgivable that some hype is used to attract interest to the project. Regards, Martijn From cfbolz at gmx.de Thu Sep 15 15:29:45 2005 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Thu, 15 Sep 2005 15:29:45 +0200 Subject: [pypy-dev] a faster Python not a primary goal of PyPy? In-Reply-To: <43295015.2060109@infrae.com> References: <4328B40C.2060809@stackless.com> <20050915065824.GO22844@solar.trillke.net> <43295015.2060109@infrae.com> Message-ID: <43297749.8080809@gmx.de> Hi Martijn, since I am in a very nitpicking mode... Martijn Faassen wrote: > holger krekel wrote: > [snip] >> PyPy is not primarily - at least not only - about making >> Python faster although many people like to view it this way :-) > > This correction is a bit confusing... It is not surprising that people > view "a faster Python" as a primary goal of your project, as that's the > message that's been spread repeatedly by you all over the years: > > "The PyPy project aims at producing a flexible and fast Python > implementation. The guiding idea is to translate a Python-level > description of the Python language itself to lower level languages. > Rumors have it that the secret goal is being faster-than-C which is > nonsense, isn't it?" Note that it says "fexible and fast": fexible comes first. And we explicitely say that the faster-than-C goal is nonsense. > and > > "In the next step of the project, we will generate C code or machine > code from the source of Pypy, thereby reducing the speed penalty. > "reducing the speed penalty" does not imply being very fast. It just means being not so slow. > Later in the project, we will introduce optimisation steps that should > make Pypy run faster than CPython." > > and > > "Psyco is a Python extension module which can massively speed up the > execution of any Python code. > ... > The future of Psyco now lies in the PyPy project, which according to > plan will provide a good base for a Python interpreter with better and > well-integrated Psyco-like techniques as soon as 2006." "a good base" does not necessarily mean as fast as Psyco is now. > I certainly hope that your "not primarily - at least not only" doesn't > indicate that the goal of making Python faster is slowly becoming less > important to you all, as I'd very much like a faster Python, and more > flexibility of implementation is, while of tremendous personal interest > to me, not one of much business interest in the foreseeable future, > which means it's unlikely I'll get to play with PyPy much unless it > offers performance benefits. > > I realize that of course that flexibility has always been a primary goal > of PyPy as well; obviously speed isn't the *only* goal of the project, > but the flexibility goal now dominates and a faster Python is indeed not > of primary importance to the project, I would suggest you revise the > previous statements, which make it rather likely people will get this > wrong impression. > Holger's statement is very important. At the moment most people have the notion that PyPy is mainly about speed. But the really exciting fact about PyPy is its flexibility, therefore it makes sense to advertise the flexibility goals a bit (even if that means diminishing the speed goal in one single mail to pypy-dev). Of course, achieving high speed is a worthwhile goal but this goal can be reached _because of_ the flexibility of PyPy (not the other way round). :-) Cheers, Carl Friedrich From faassen at infrae.com Thu Sep 15 16:46:33 2005 From: faassen at infrae.com (Martijn Faassen) Date: Thu, 15 Sep 2005 16:46:33 +0200 Subject: [pypy-dev] a faster Python not a primary goal of PyPy? In-Reply-To: <43297749.8080809@gmx.de> References: <4328B40C.2060809@stackless.com> <20050915065824.GO22844@solar.trillke.net> <43295015.2060109@infrae.com> <43297749.8080809@gmx.de> Message-ID: <43298949.5000508@infrae.com> Hey, Carl Friedrich Bolz wrote: > since I am in a very nitpicking mode... I'm nitpicking too, but even in the first announcement of PyPy, before it even had this name, the claim that the thing was going to be faster than CPython was made: As Armin Rigo of PSYCO fame takes part in the effort, we are confident that MinimalPython will eventually run faster than today's CPython. > Holger's statement is very important. At the moment most people have > the notion that PyPy is mainly about speed. What I am saying is that it's due to the PyPy project's communications that this notion has spread. To correct someone who picked up on this notion is therefore a bit confusing. > But the really exciting fact about PyPy is its flexibility, therefore > it makes sense to advertise the flexibility goals a bit (even if that > means diminishing the speed goal in one single mail to pypy-dev). > Of course, achieving high speed is a worthwhile goal but this goal > can be reached _because of_ the flexibility of PyPy (not the other > way round). I know that, and I'll stop nitpicking now. I just thought it was a bit weird to start correcting an impression in people that seems a legitimate interpretation of previous communication of this project. Then again, an emphasis on flexibility is justified. What is exciting about PyPy seems to depend on ones perspective too. To the PyPy hackers themselves, the coolest thing is the flexibility allowing all sorts of fun experimentations with syntaxes and semantics and ways Python code is executed. To the wider world outside Python language hackers however the *results* of this flexibility of PyPy is what is truly interesting. The flexibility by itself is not interesting at all from that perspective. Since the topic of this thread was making money out of PyPy, this is a relevant thing to talk about. For non Python core hackers, the biggest potential gains in PyPy that I heard about so far would be: * increased performance * making Python better in an increasingly concurrent world (threading/GIL/stackless, etc). That ties into increased performance too, in multi processing environments. Regards, Martijn From tismer at stackless.com Thu Sep 15 17:16:03 2005 From: tismer at stackless.com (Christian Tismer) Date: Thu, 15 Sep 2005 17:16:03 +0200 Subject: [pypy-dev] a faster Python not a primary goal of PyPy? In-Reply-To: <43297070.4020004@infrae.com> References: <4328B40C.2060809@stackless.com> <20050915065824.GO22844@solar.trillke.net> <43295015.2060109@infrae.com> <200509151252.55437.jacob@strakt.com> <432953A6.6050103@stackless.com> <43297070.4020004@infrae.com> Message-ID: <43299033.5080103@stackless.com> Martijn Faassen wrote: > Christian Tismer wrote: > [snip] > >> That's the point. Not prioritizing speed is fine, but it >> does not allow us to be remarkably slower. Being of competitive >> speed and then more flexible is a must. If we are slower, the >> market will not consider us at all. > > > That's true, and that's a useful clarification. It's still different > from the impression you're giving out, where you're strongly implying at > least to me that speed will be significantly *faster* eventually with > PyPy. Then again, it's forgivable that some hype is used to attract > interest to the project. I thik I'm the worst speed junkie in the project, and I'm holding my breath a little until I'm convinced that we can be fast. ATM I'm quite vague about that, because of the complexity of crucial parts of our system, and the fact that I don't understand how we are going to translate that into speedy code without very high level optimization techniques. I simply can't tell whether we can reach this without major hand-rewriting or whether we can gather the missing speed factor by applying JIT technology. I always thought that the initial big step of creating an initial binary of reasonable speed would be an easy, first step. But it turned out to be a huge step which is by far not in the shape it should be. This amount of my misestimation makes me nervous and stops me from praising PyPy's speed all too loudly. On the other hand, we have extremely efficient results on the low level side. RPython programs have been shown to become possibly very fast. This is an area where I have no doubt making promises, and that's why I proposed to put some more effort there and at least provide a tool to produce fast extension modules from RPython, soon. This is also the reason why I want to make RPython more complete and easier to use. For sure we never have put any hype onto PyPy just to make it more attractive. We learned that flexibility and speed are not trivial to marry, and we have proven our claims about flexibility, but unfortunately not yet on speed. I would really hate to retract this claim. ATM, there is simply no prognosis possible. cheers - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From tismer at stackless.com Thu Sep 15 17:29:25 2005 From: tismer at stackless.com (Christian Tismer) Date: Thu, 15 Sep 2005 17:29:25 +0200 Subject: [pypy-dev] a faster Python not a primary goal of PyPy? In-Reply-To: <43297749.8080809@gmx.de> References: <4328B40C.2060809@stackless.com> <20050915065824.GO22844@solar.trillke.net> <43295015.2060109@infrae.com> <43297749.8080809@gmx.de> Message-ID: <43299355.2080507@stackless.com> Carl Friedrich Bolz wrote: > Hi Martijn, > > since I am in a very nitpicking mode... > > Martijn Faassen wrote: ... >> "The PyPy project aims at producing a flexible and fast Python >> implementation. The guiding idea is to translate a Python-level >> description of the Python language itself to lower level languages. >> Rumors have it that the secret goal is being faster-than-C which is >> nonsense, isn't it?" > > > Note that it says "fexible and fast": fexible comes first. And we > explicitely say that the faster-than-C goal is nonsense. I don't think that order is relevant in a mathematical "and". For me it means "flexible without being slow" and "fast without being inflexible". This is the matter of optimization. We have seen examples of both. Python is a quite fast, inflexible implementation. PyPy is a very flexible but yet slow implementation. The proof of concept, that we can have both at the same time, is still not there. And I don't support the nonsense part at all. Psyco has proven that it can make CPython much faster under circumstances. The same factor can be at least expected for PyPy. But if PyPy does not jump over its current factor 40-50 hurdle, this acceleration might still not outperform CPython. ... > Holger's statement is very important. At the moment most people have the > notion that PyPy is mainly about speed. But the really exciting fact > about PyPy is its flexibility, therefore it makes sense to advertise the > flexibility goals a bit (even if that means diminishing the speed goal > in one single mail to pypy-dev). Of course, achieving high speed is a > worthwhile goal but this goal can be reached _because of_ the > flexibility of PyPy (not the other way round). The impression of mostly being about speed might also come from the fact that speed is the thing that PyPy lacks at most, hurtingly. And it is clear if a project is started in the most flexible way possible from the ground, with no consideration of speed at all. We will need to find yet another stone of the wise to morph the code reasonably. As said, I very much hope it, while there is currently no indicator that we are close to it. We have to prove that last point about flexibility. The gap lies in the needed transformations, which need to be invented. cheers - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From njriley at uiuc.edu Thu Sep 15 18:18:15 2005 From: njriley at uiuc.edu (Nicholas Riley) Date: Thu, 15 Sep 2005 11:18:15 -0500 Subject: [pypy-dev] Re: next pypy-sync meeting 20050915 In-Reply-To: <2m8xxyn1me.fsf@starship.python.net> References: <4328B40C.2060809@stackless.com> <2m8xxyn1me.fsf@starship.python.net> Message-ID: <20050915161815.GA41521@uiuc.edu> On Thu, Sep 15, 2005 at 10:04:09AM +0100, Michael Hudson wrote: > Lots of people are interested (though maybe they shouldn't be) in a > non-GIL implementation of Python. Offering different concurrency > models is a really big selling point of PyPy, I think. This is what I'm working on (though from an academic context :), and I've been in somewhat of a holding pattern recently - doing some small experiments with CPython, reading and following mailing lists/IRC, trying to figure out when PyPy is finished enough that I can start doing work. Of course, if there's something I can do to help PyPy get to this point, I'll help. Has anyone written anything about the (at least near-term) plans for PyPy in the area of concurrency/locking? -- Nicholas Riley | From michal at sabren.com Thu Sep 15 19:36:41 2005 From: michal at sabren.com (Michal Wallace) Date: Thu, 15 Sep 2005 13:36:41 -0400 (EDT) Subject: [pypy-dev] features, benefits, and recruiting help In-Reply-To: <20050915065824.GO22844@solar.trillke.net> References: <4328B40C.2060809@stackless.com> <20050915065824.GO22844@solar.trillke.net> Message-ID: On Thu, 15 Sep 2005, holger krekel wrote: > On Wed, Sep 14, 2005 at 22:17 -0400, Michal Wallace wrote: > > ... The stated goal for this > > project is just to make a faster version of > > python, but can you really charge for that > > given that pypy is already open source? > > PyPy is not primarily - at least not only - about making > Python faster although many people like to view it this way :-) Hi Holger, What I said is that the stated goal is to make a faster version of python. I actually looked it up before I wrote that: """ The PyPy project aims at producing a flexible and fast Python implementation. The guiding idea is to translate a Python-level description of the Python language itself to lower level languages. Rumors have it that the secret goal is being faster-than-C which is nonsense, isn't it? more... """ -- http://codespeak.net/pypy/dist/pypy/doc/news.html As Martijn Faassen said, it's not surprising that people have the impression that pypy is all about speed. I think part of the problem is that people understand the word "fast". What that means is obvious. It's like python, only it runs faster! "Flexible" doesn't mean anything to me. Python is already flexible, so what does that mean? Well, you've got these different backends. Maybe that's what it means. And the people who made stackless and psyco and green threads are on the team, so maybe it's about those things. Who knows? The site doesn't say so if we want to figure out the status of those things, we have to think about it. > PyPy is a lot about flexibility to produce custom implementations > of Python for use in particular environments. Regarding persistence, Persistence. What does that mean? Is that like Squeak, where you can save the state of the whole interpreter as a single image? If so, that's pretty neat! It ought to be right up front on the site. > security (e.g. sandboxing) Also neat. So another benefit is that you can make programs that run other programs safely? I know some guys from twisted like that stuff, because they want to make games where people can run their own scripts. > and deployment I don't know what this means at all. > there is a lot that we can possiblly do with PyPy implementations. I > am sure that such possibilities will become clearer next year. Cool. I guess I was thinking more about what you could do to make money *now*, given the current state of the codebase... Sounds like one big part of that would be telling people more about the potential so they can say "I want that! I'll pay to make it happen!" :) Another thing I suspect is part of pypy but isn't really right up front is the whole continuations thing. Yes, CPS dos make an appearance int he top news article on the news page, but that'll just get buried as you post more news. Basically "flexible" is a feature, and not a very well defined one. Features don't sell well. What sells are benefits, and "fast" is a benefit. So if it's not the primary benefit, it's time to fix your message. :) > > Another option is that you could use your > > technology created in pypy to create new > > *frontends* for pypy, and sell the service > > of taking legacy code in other languages, > > and translating that into call trees, and > > turning it magically into python or lisp or c... > > This seems like it could be a much more > > lucrative service, but it's a whole new > > research project and it takes you away > > from your core focus on python. (On the > > other hand, a company that did this would > > have a massive incentive to sponsor pypy > > development... Maybe there's a company that > > already does this kind of thing with some > > other technology... Like, say, people just > > rewriting things by hand? > > A build tool is part of the current EU project > and of our plans. I don't understand. What I'm saying is that since you can translate python->lisp or python->python, then maybe you can use the same technology to translate cobol->python or cobol->java or whatever using the cool internal technology that pypy has. This would be a major research project and well outside the bounds of the pypy project. However, if I recall, there are already companies that do this kind of thing and pypy's internals might be of use to them and their clients. That's all I was saying. I'm not sure what build tools has to do with that. ;) > > Another idea has to do with the sprints. > > I already told Holger this a while back, > > but I think you guys have one of the most > > impressive project management styles around, > > and the sprint idea seems like a great > > amount of fun. > > > > What if you capitalized on *that*? > > > > A week in europe to work on an open source > > project? That would be an awesome vacation! > > And to get to learn about compilers and python > > along the way? > > We are doing this regularly :-) You have sprints regularly, but are you marketing them to companies as a reward or training for top employees? Last I heard, at least some of the sprints were limited to experienced PyPy developers, right? Obviously you don't want 500 newbies running around, but... > > Anyway, it might be a crazy idea, but if you > > could get it working, it would capitalize > > on what you're already doing rather than > > force you to come up with some other devlepoment > > project off to the side... > > There are many possibilities but only so much > time and focus to pursue them. Bea also reported > quite some interest from other projects and entities > in the EU context. We got invitations to Afrika and > other places and i guess we pursue them :-) Well, now that I hear the talk that there are other benefits than speed, I'd say you'd have a much better chance of making money if you capitalized on that. Also, all of this is about the top line, or revenue revenue. What about the bottom line? The bottom line is your profit: income minus expenses. PyPy development is very expensive. As I said, I love the idea of the sprints. I think they're great. They're also expensive. I don't know for sure that they're expensive to host, but they're certainly expensive to attend - especially for people who live on another continent. Lowering the cost of participating in pypy won't put any money in your pockets, but if the goal of the money is to have more man hours put into developing pypy (and I don't know what the goal is, but that is my guess) then there are many other ways you can get more smart people working on it. For example, I clicked on "issues" on the pypy site. Turns out these are not just issues, but also a sort of "to-do" list. Are any of these things that a newbie could do without going to a sprint? Maybe with some guidance. For example, to take an issue at random... #118 : "pickling of ll flowgraphs" I'm sure to you guys, pickling of ll flowgraphs makes a lot of sense. But to me as an outsider, I'm not sure what it means. Does ll mean llvm or low level? And what prevents you from pickling it? Can you write a test case that would pass if pickling ll flowgraphs worked? To an outsider like me, PyPy works by genius-level black magic. So if I wanted to help out and get ll flowgraph pickles working, I'd have a huge learning curve. I also don't know if it's really important or not. So basically if I want to help out with some code over the weekend, I look at the task here and I see a huge learning curve and then after all that work, I don't know if the task is really important or not. So what I'm saying is that it's difficult for people to pick up pypy. The impression is that you have to wait for a sprint and then go to europe if you want to help out. So I imagine that recruiting help is very expensive in terms of time. The point, then, is that one way to reach your goals without needing people to give you more money is to make it easier for people to give you their *time*. I'm working on a similar problem for pirate and at my own company, and I'd be willing to help with making the to-do list more approachable to newbies if you guys are interested. Sincerely, Michal J Wallace Sabren Enterprises, Inc. ------------------------------------- contact: michal at sabren.com hosting: http://www.cornerhost.com/ my site: http://www.withoutane.com/ ------------------------------------- From michal at sabren.com Thu Sep 15 19:42:12 2005 From: michal at sabren.com (Michal Wallace) Date: Thu, 15 Sep 2005 13:42:12 -0400 (EDT) Subject: [pypy-dev] next pypy-sync meeting 20050915 In-Reply-To: References: <4328B40C.2060809@stackless.com> Message-ID: On Thu, 15 Sep 2005, Beatrice During wrote: > Just a short comment to your email Michael (thanks for your thoughts > onj this matter BTW ;-): :) > One of the current major discussions about how to capitalize > on Open Source as a business model is just that - learning/training. > > So I think you are on to something there Michael - both regarding the > technical aspects and the project management/developer process aspects. > Sprinting combines them both ;-) Hi Bea, I recently did a "sprint" in the form of the pyweek game challenge. We hung out in IRC for most of the week and made a game. (Or at least a game engine, before we ran out of time!) To me it was an exercise in project management but it was also a recruiting tool: I wanted to see what it was like to work with certain people before I spent the money to actually hire them. It worked well, and I plan to continue using that model to screen potential hires in the future... Maybe a company with more cash in the bank could use pypy in a similar way? Sincerely, Michal J Wallace Sabren Enterprises, Inc. ------------------------------------- contact: michal at sabren.com hosting: http://www.cornerhost.com/ my site: http://www.withoutane.com/ ------------------------------------- From arigo at tunes.org Thu Sep 15 19:44:59 2005 From: arigo at tunes.org (Armin Rigo) Date: Thu, 15 Sep 2005 19:44:59 +0200 Subject: [pypy-dev] unionof and dictionaries In-Reply-To: References: Message-ID: <20050915174459.GA15169@code1.codespeak.net> Hi Amaury, On Wed, Sep 14, 2005 at 07:09:44PM +0200, Amaury Forgeot D Arc wrote: > Ah, I did not see the "no_side_effects_in_union" trick. Now I > understand that union() is meant to have the side-effect you > describe, and that contains() has no side-effect and is safe > to use in assertions. Right, contains() is based on union_of() but with a trick to disable side-effects. Armin From arigo at tunes.org Thu Sep 15 20:07:16 2005 From: arigo at tunes.org (Armin Rigo) Date: Thu, 15 Sep 2005 20:07:16 +0200 Subject: [pypy-dev] Re: next pypy-sync meeting 20050915 In-Reply-To: <20050915161815.GA41521@uiuc.edu> References: <4328B40C.2060809@stackless.com> <2m8xxyn1me.fsf@starship.python.net> <20050915161815.GA41521@uiuc.edu> Message-ID: <20050915180716.GB15169@code1.codespeak.net> Hi Nicholas, On Thu, Sep 15, 2005 at 11:18:15AM -0500, Nicholas Riley wrote: > > non-GIL implementation of Python. > > This is what I'm working on (though from an academic context :), and > I've been in somewhat of a holding pattern recently - doing some small > experiments with CPython, reading and following mailing lists/IRC, > trying to figure out when PyPy is finished enough that I can start > doing work. > > Of course, if there's something I can do to help PyPy get to this > point, I'll help. Has anyone written anything about the (at least > near-term) plans for PyPy in the area of concurrency/locking? Interesting. We should be about to consider alternate locking strategies right now. The current status is: we use GIL locking, and didn't even finish that -- the GIL is not released around system calls. Considering other locking strategies, I think the following one is possible. It would incur a small general run-time overhead, but it's of course all PyPy is about (whatever opinions might be heard around here :-) : avoiding as much as possible hard-wired design decisions and leaving compromizes to the choice of the user (or at least, for now, the person who translates and compiles PyPy). The naive locking idea is to put a cheap lock on each object, and then each space operation would first acquire the lock on its arguments and release it when it finishes. Recursive calls to the interpreter (i.e. calls to space operations that call back app-level Python code) would also release all the locks held by that thread. The locks should be re-entrants, i.e. not block if already held by the same thread. These locks can probably be optimized using some assumptions about our use case: we could for example acquire more locks when entering a space operation, but delay all releasing to the end of the bytecode (or even to the end of the next 'checkinterval' bytecodes), at which point we would bulk-release all locks held by the current thread. The above scheme is rather easy to implement as a proxy object space, doing the locking and then calling the standard object space. Other ideas are possible too; I'd be interested to hear about yours, what you've been playing with? A bientot, Armin. From bea at netg.se Thu Sep 15 20:46:28 2005 From: bea at netg.se (Beatrice During) Date: Thu, 15 Sep 2005 20:46:28 +0200 (CEST) Subject: [pypy-dev] a faster Python not a primary goal of PyPy? In-Reply-To: <43298949.5000508@infrae.com> References: <4328B40C.2060809@stackless.com> <20050915065824.GO22844@solar.trillke.net> <43295015.2060109@infrae.com> <43297749.8080809@gmx.de> <43298949.5000508@infrae.com> Message-ID: Hi there Just a short comment: The fact that we have different perspectives and views on what has been communicated about the project goal indicates to me that there is/or was an unclear communication about this ;-) Thanks for raising attention to this Martijn. Cheers Bea On Thu, 15 Sep 2005, Martijn Faassen wrote: > Hey, > > Carl Friedrich Bolz wrote: >> since I am in a very nitpicking mode... > > I'm nitpicking too, but even in the first announcement of PyPy, before > it even had this name, the claim that the thing was going to be faster > than CPython was made: > > As Armin Rigo of PSYCO fame takes part in the effort, > we are confident that MinimalPython will eventually > run faster than today's CPython. > >> Holger's statement is very important. At the moment most people have >> the notion that PyPy is mainly about speed. > > What I am saying is that it's due to the PyPy project's communications > that this notion has spread. To correct someone who picked up on this > notion is therefore a bit confusing. > >> But the really exciting fact about PyPy is its flexibility, therefore >> it makes sense to advertise the flexibility goals a bit (even if that >> means diminishing the speed goal in one single mail to pypy-dev). > >> Of course, achieving high speed is a worthwhile goal but this goal >> can be reached _because of_ the flexibility of PyPy (not the other >> way round). > > I know that, and I'll stop nitpicking now. I just thought it was a bit > weird to start correcting an impression in people that seems a > legitimate interpretation of previous communication of this project. > Then again, an emphasis on flexibility is justified. > > What is exciting about PyPy seems to depend on ones perspective too. To the > PyPy hackers themselves, the coolest thing is the flexibility allowing all > sorts of fun experimentations with syntaxes and semantics and ways Python > code is executed. > > To the wider world outside Python language hackers however the *results* > of this flexibility of PyPy is what is truly interesting. The > flexibility by itself is not interesting at all from that perspective. > Since the topic of this thread was making money out of PyPy, this is a > relevant thing to talk about. > > For non Python core hackers, the biggest potential gains in PyPy that I > heard about so far would be: > > * increased performance > > * making Python better in an increasingly concurrent world > (threading/GIL/stackless, etc). That ties into increased performance > too, in multi processing environments. > > Regards, > > Martijn > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > Beatrice D?ring Change Maker Tel: 031- 7750940 J?rntorget 3 Mobil: 0734- 22 89 06 413 04 G?teborg E-post: bea at changemaker.nu www.changemaker.nu "Alla dessa m?sten och alldaglighet. Allt detta som binder v?r verklighet i bojor av skam och rep utav tv?ng. Alla dessa kedjor som binder v?r s?ng. Jag skall slita dem alla i sm?, sm? stycken och m?jligtvis av resterna g?ra mig smycken." - hemlig From lac at strakt.com Thu Sep 15 20:59:08 2005 From: lac at strakt.com (Laura Creighton) Date: Thu, 15 Sep 2005 20:59:08 +0200 Subject: [pypy-dev] features, benefits, and recruiting help In-Reply-To: Message from Michal Wallace of "Thu, 15 Sep 2005 13:36:41 EDT." References: <4328B40C.2060809@stackless.com> <20050915065824.GO22844@solar.trillke.net> Message-ID: <200509151859.j8FIx8eD021247@theraft.strakt.com> In a message of Thu, 15 Sep 2005 13:36:41 EDT, Michal Wallace writes: >"Flexible" doesn't mean anything to me. Python is >already flexible, so what does that mean? Well, you've >got these different backends. Maybe that's what it >means. And the people who made stackless and psyco >and green threads are on the team, so maybe it's >about those things. Who knows? The site doesn't say >so if we want to figure out the status of those >things, we have to think about it. >From our point of view, Python is not flexible. You are stuck with some 'reasonable compromises' at the language-implementor's-decision level. If you have tons of memory, the best algorithms to use are different from those that you use if you are an embedded system and have none, and neither of those are about right for a typical laptop. A truly flexible system would let you code up your own 'search the dictionary' algorithm and stick it right inside the language where it belongs, and see whether things work faster. It is interesting -- the challenge for PyPy does not really seem to be 'to get faster than CPython'. Instead, it is 'to get almost as fast as CPython.' Once PyPy is fast enough to use commercially, everybody who has an idea for a faster implementation of library function X, given data that looks like Y, will be free to code it up into its own objectspace and see what happens. And there are a lot of cool algorithms out there. The problem will be getting reasonable performance at all. There is overhead associated with having this much flexibility. Laura From bea at netg.se Thu Sep 15 20:58:16 2005 From: bea at netg.se (Beatrice During) Date: Thu, 15 Sep 2005 20:58:16 +0200 (CEST) Subject: [pypy-dev] next pypy-sync meeting 20050915 In-Reply-To: References: <4328B40C.2060809@stackless.com> Message-ID: Hi there On Thu, 15 Sep 2005, Michal Wallace wrote: > On Thu, 15 Sep 2005, Beatrice During wrote: > >> Just a short comment to your email Michael (thanks for your thoughts >> onj this matter BTW ;-): > > :) > >> One of the current major discussions about how to capitalize >> on Open Source as a business model is just that - learning/training. >> >> So I think you are on to something there Michael - both regarding the >> technical aspects and the project management/developer process aspects. >> Sprinting combines them both ;-) > > Hi Bea, > > I recently did a "sprint" in the form of > the pyweek game challenge. We hung out in > IRC for most of the week and made a game. > (Or at least a game engine, before we ran > out of time!) > > To me it was an exercise in project > management but it was also a recruiting > tool: I wanted to see what it was like to > work with certain people before I spent > the money to actually hire them. > > It worked well, and I plan to continue > using that model to screen potential > hires in the future... Maybe a company > with more cash in the bank could use > pypy in a similar way? > > Sincerely, > > Michal J Wallace > Sabren Enterprises, Inc. What a great application - do you mind if I spread the word about this..? Cheers Bea Beatrice D?ring Change Maker Tel: 031- 7750940 J?rntorget 3 Mobil: 0734- 22 89 06 413 04 G?teborg E-post: bea at changemaker.nu www.changemaker.nu "Alla dessa m?sten och alldaglighet. Allt detta som binder v?r verklighet i bojor av skam och rep utav tv?ng. Alla dessa kedjor som binder v?r s?ng. Jag skall slita dem alla i sm?, sm? stycken och m?jligtvis av resterna g?ra mig smycken." - hemlig From hpk at trillke.net Thu Sep 15 21:59:14 2005 From: hpk at trillke.net (holger krekel) Date: Thu, 15 Sep 2005 21:59:14 +0200 Subject: [pypy-dev] next pypy-sync meeting 20050915 In-Reply-To: References: <4328B40C.2060809@stackless.com> Message-ID: <20050915195914.GB10613@solar.trillke.net> Hi Bea, hi Michal, On Thu, Sep 15, 2005 at 20:58 +0200, Beatrice During wrote: > On Thu, 15 Sep 2005, Michal Wallace wrote: > >I recently did a "sprint" in the form of > >the pyweek game challenge. We hung out in > >IRC for most of the week and made a game. > >(Or at least a game engine, before we ran > >out of time!) > > > >To me it was an exercise in project > >management but it was also a recruiting > >tool: I wanted to see what it was like to > >work with certain people before I spent > >the money to actually hire them. > > > >It worked well, and I plan to continue > >using that model to screen potential > >hires in the future... Maybe a company > >with more cash in the bank could use > >pypy in a similar way? > > > What a great application - do you mind if I spread the word about this..? I'd rather like to think that it's a nice side effect of sprints that you can get to know people who are interesting to hire. But the main reason for that is that you meet people who are doing things out of their own interest. If you start proposing or inviting to sprints for hiring purposes they might easily become more of a "working contest" or exam situation and loose much of the fun that you get from collaborating out of free interest. cheers, holger From michal at sabren.com Thu Sep 15 22:19:36 2005 From: michal at sabren.com (Michal Wallace) Date: Thu, 15 Sep 2005 16:19:36 -0400 (EDT) Subject: [pypy-dev] features, benefits, and recruiting help In-Reply-To: <200509151859.j8FIx8eD021247@theraft.strakt.com> References: <4328B40C.2060809@stackless.com> <20050915065824.GO22844@solar.trillke.net> <200509151859.j8FIx8eD021247@theraft.strakt.com> Message-ID: On Thu, 15 Sep 2005, Laura Creighton wrote: > From our point of view, Python is not flexible. You are stuck with > some 'reasonable compromises' at the language-implementor's-decision > level. If you have tons of memory, the best algorithms to use are > different from those that you use if you are an embedded system and > have none, and neither of those are about right for a typical > laptop. A truly flexible system would let you code up your own > 'search the dictionary' algorithm and stick it right inside the > language where it belongs, and see whether things work faster. > > It is interesting -- the challenge for PyPy does not really seem to > be 'to get faster than CPython'. Instead, it is 'to get almost as > fast as CPython.' Once PyPy is fast enough to use commercially, > everybody who has an idea for a faster implementation of library > function X, given data that looks like Y, will be free to code it up > into its own objectspace and see what happens. And there are a lot > of cool algorithms out there. > > The problem will be getting reasonable performance at all. There is > overhead associated with having this much flexibility. Hi Laura, Thanks for explaining this. So this is another great benefit that ought to be part of the marketing message. :) In my mind I've been picturing pypy as a compiler. Python goes in, C or lisp code goes out. But what I think you're saying here is that since pypy is written in python, it can compile itself and produce a variety of custom virtual machines? So, for example, jython and ironpython and pirate could all eventually just be implemented as backends for pypy? And we could finally get a decent python interpreter running on a palm pilot? Because if so, I can think of a whole list of companies that might be interested in adding funding to the mix: microsoft (ha?) novell (they fund mono) sun or ibm (for the jvm) nokia (for their series 60 phone) palm (for palm pilot) sony or ibm (for the cell processor) Basically, these are all groups that want a lot of developers to use their technology. Sony, for example is coming out with their playstation 3 based on this new cell chip. Video games take years to program. They're usually written in C. If you can show sony that by supporting pypy they can have a fast-as-c language custom taylored to their architecture, they might just go for it. The've already announced plans for a PS3 with linux installed, so they must have some experience funding open source projects. Similarly, palm has devoted resources to the eclipse project to create a standard IDE for developing palm pilot applications. I don't even think you have to market a product here. You can just market the potential benefits of the research itself to the right people, show them how it would benefit them, and show them the care and professionalism with which you're already running this project (so they know you'll actually deliver). And of course when I say you, I mean anybody on the list who was interested in getting more funding for this project. :) Sincerely, Michal J Wallace Sabren Enterprises, Inc. ------------------------------------- contact: michal at sabren.com hosting: http://www.cornerhost.com/ my site: http://www.withoutane.com/ ------------------------------------- From hpk at trillke.net Thu Sep 15 22:21:12 2005 From: hpk at trillke.net (holger krekel) Date: Thu, 15 Sep 2005 22:21:12 +0200 Subject: [pypy-dev] Re: features, benefits, and recruiting help In-Reply-To: References: <4328B40C.2060809@stackless.com> <20050915065824.GO22844@solar.trillke.net> Message-ID: <20050915202112.GC10613@solar.trillke.net> Hi Michal, there is a lot in your mail that i'd like to make some comments on. But I'd rather wait until we meet at some sprint to discuss some of that :-) On Thu, Sep 15, 2005 at 13:36 -0400, Michal Wallace wrote: > On Thu, 15 Sep 2005, holger krekel wrote: > Lowering the cost of participating in pypy won't > put any money in your pockets, but if the goal of > the money is to have more man hours put into developing > pypy (and I don't know what the goal is, but that is > my guess) then there are many other ways you can get > more smart people working on it. It's also a matter of good timing to bring more people into the project. For example, the Paris sprint is a quite good opportunity because we are heading for new stuff there where people can get involved and being helped in a more focused way. > For example, I clicked on "issues" on the pypy site. > Turns out these are not just issues, but also a sort > of "to-do" list. Are any of these things that a newbie > could do without going to a sprint? The default issue tracker view separates issues into "easy", "medium" and "hard" to tackle and displays the easiest ones first. > Maybe with some guidance. For example, to take an issue > at random... > > #118 : "pickling of ll flowgraphs" That's a 'medium' issue and thus not really suited for newcomers. > I'm sure to you guys, pickling of ll flowgraphs makes > a lot of sense. But to me as an outsider, I'm not sure > what it means. Does ll mean llvm or low level? And what > prevents you from pickling it? Can you write a test case > that would pass if pickling ll flowgraphs worked? I agree that it's good to try to describe issues assuming as few pre-knowledge as possible. > To an outsider like me, PyPy works by genius-level > black magic. So if I wanted to help out and get > ll flowgraph pickles working, I'd have a huge > learning curve. I also don't know if it's really > important or not. OK. But aren't there also a number of issues where it's not so hard to guess about their usefulness? > ... > So what I'm saying is that it's difficult for people > to pick up pypy. The impression is that you have to > wait for a sprint and then go to europe if you want > to help out. So I imagine that recruiting help is > very expensive in terms of time. > > The point, then, is that one way to reach your goals > without needing people to give you more money is to > make it easier for people to give you their *time*. Sure. We actively mentor (or at least try to) new people and i hold that our documentation is not the worst. The fact is that writing a full Python interpreter implementation and a full compiler are - by themselves - not trivial tasks. Add some challenging goals (including making the EU project part of the project work) to that and you have current PyPy :-) > I'm working on a similar problem for pirate and > at my own company, and I'd be willing to help > with making the to-do list more approachable to > newbies if you guys are interested. you are welcome! Going over the to-do list and adding useful information (maybe grabbed from someone at IRC) could certainly help. cheers, holger From jacob at strakt.com Thu Sep 15 23:31:45 2005 From: jacob at strakt.com (Jacob =?iso-8859-1?q?Hall=E9n?=) Date: Thu, 15 Sep 2005 23:31:45 +0200 Subject: [pypy-dev] features, benefits, and recruiting help In-Reply-To: References: <4328B40C.2060809@stackless.com> <200509151859.j8FIx8eD021247@theraft.strakt.com> Message-ID: <200509152331.45535.jacob@strakt.com> torsdagen den 15 september 2005 22.19 skrev Michal Wallace: > Thanks for explaining this. So this is another > great benefit that ought to be part of the > marketing message. :) It is mentioned in the architecture overview. We still don't have a "key benefits" marketing page, simply because it is still too hard to tell what the key benefits are. The ideas are there, but we may have to sacrifice some of them to attain some other goals. We think it is too early to make firm commitments at this point in time. Pypy is a reseach project and we still have 14 months to go before we need to show a useable product to the EU. Being an agile project, we have no Big Design Up Front. We are convinced that we can attain the goal of being as fast as CPython, but the path to get there has not been laid out in detail. We have hopes of becoming faster than CPython, at least for some specific problems and platforms, but again, we currently don't know the exact route of getting there, or even which problems we will be good at solving. Our faith in being able to solve the problems is built on the expertise and previous accomplishments of our project members. > > In my mind I've been picturing pypy as a compiler. > Python goes in, C or lisp code goes out. This is where quite a few people stumble. The end product of the Pypy project is essentially an executable, that is able to interpret Python, just like the regular Python interpreter, which is implemented in C. This executable is generated from source code which is written in Python, or to be more correct, a subset of Python which we call RPython. This subset forbids some of the very dynamic usages of Python, making the problem of compiling to efficient static code somewhat simpler. This means that a very large part of the work in Pypy is geared towards generating this single interpreter. All the work put into this may be reusable, if people are willing to code extension modules using RPython, which could then be translated into C/machine code/Lisp/whatever. However, this requires that we freeze the specification of RPython, document it, and build tools that help check that no illegal Python constructs are used. Currently, RPython is a convention, not a tightly specified programming language. Quite likely, it may remain a convention as it may be impossible to make exhaustive compile time checks of all constructs. > But what I think you're saying here is that since > pypy is written in python, it can compile itself > and produce a variety of custom virtual machines? No, because of its modular design, it can produce a variety of custom virtual machines. This would be possible in other environments as well, but the high level of abstraction in Python makes the task easier. > So, for example, jython and ironpython and pirate > could all eventually just be implemented as > backends for pypy? With a hand-wavy interpretation of what you are saying that would be true. With a relatively modest amount of work (compared to building things from scratch), it would be possible to implement the Java VM, the .NET and the Parrot VM as Pypy execution models. It is still too early to say to what extent the optimisations we are doing for the Python VM would be applicable to the other VMs. Probably quite a few, but certainly not all. > > And we could finally get a decent python > interpreter running on a palm pilot? Possibly. One part of the EU project is to test Pypy on an embedded device. However, this may require using the modular structure to strip out unneeded functionality. If you don't need floats, you can easily disable them and get a smaller interpreter. > Because if so, I can think of a whole list of > companies that might be interested in adding > funding to the mix: > > microsoft (ha?) > novell (they fund mono) > sun or ibm (for the jvm) > nokia (for their series 60 phone) > palm (for palm pilot) > sony or ibm (for the cell processor) > > Basically, these are all groups that want > a lot of developers to use their technology. We are still 200 times slower than CPython and we still use quite a lot of memory. Approaching the big players requires that we are further on the way, or that we have some isolated aspect that would make sense for them to invest in. (We are still too much of Py in the Sky ;-)) > Sony, for example is coming out with their > playstation 3 based on this new cell chip. > Video games take years to program. They're > usually written in C. If you can show sony > that by supporting pypy they can have a > fast-as-c language custom taylored to their > architecture, they might just go for it. > The've already announced plans for a PS3 with > linux installed, so they must have some > experience funding open source projects. > > Similarly, palm has devoted resources to > the eclipse project to create a standard > IDE for developing palm pilot applications. > > I don't even think you have to market a product > here. You can just market the potential > benefits of the research itself to the right > people, show them how it would benefit them, > and show them the care and professionalism > with which you're already running this > project (so they know you'll actually deliver). > > And of course when I say you, I mean > anybody on the list who was interested > in getting more funding for this project. :) Thanks for your comments Michal, they offered some ideas for how to proceed, when we have some more promising results. Jacob Hall?n From bokr at oz.net Fri Sep 16 03:46:30 2005 From: bokr at oz.net (Bengt Richter) Date: Thu, 15 Sep 2005 18:46:30 -0700 Subject: [pypy-dev] a faster Python not a primary goal of PyPy? In-Reply-To: <43299033.5080103@stackless.com> References: <43297070.4020004@infrae.com> <4328B40C.2060809@stackless.com> <20050915065824.GO22844@solar.trillke.net> <43295015.2060109@infrae.com> <200509151252.55437.jacob@strakt.com> <432953A6.6050103@stackless.com> <43297070.4020004@infrae.com> Message-ID: <5.0.2.1.1.20050915154827.00b3d760@mail.oz.net> Hello Christian and pypy team ;-) At 17:16 2005-09-15 +0200, you (Christian Tismer) wrote: >Martijn Faassen wrote: > >>Christian Tismer wrote: >>[snip] >> >>>That's the point. Not prioritizing speed is fine, but it >>>does not allow us to be remarkably slower. Being of competitive >>>speed and then more flexible is a must. If we are slower, the >>>market will not consider us at all. >> >>That's true, and that's a useful clarification. It's still different from the impression you're giving out, where you're strongly implying at least to me that speed will be significantly *faster* eventually with PyPy. Then again, it's forgivable that some hype is used to attract interest to the project. > >I thik I'm the worst speed junkie in the project, and I'm holding >my breath a little until I'm convinced that we can be fast. >ATM I'm quite vague about that, because of the complexity of >crucial parts of our system, and the fact that I don't understand >how we are going to translate that into speedy code without >very high level optimization techniques. I simply can't tell >whether we can reach this without major hand-rewriting or whether >we can gather the missing speed factor by applying JIT technology. >I always thought that the initial big step of creating an initial >binary of reasonable speed would be an easy, first step. > >But it turned out to be a huge step which is by far not in the shape >it should be. This amount of my misestimation makes me nervous and >stops me from praising PyPy's speed all too loudly. > >On the other hand, we have extremely efficient results on the low >level side. RPython programs have been shown to become possibly >very fast. This is an area where I have no doubt making promises, >and that's why I proposed to put some more effort there and at least >provide a tool to produce fast extension modules from RPython, soon. >This is also the reason why I want to make RPython more complete >and easier to use. > >For sure we never have put any hype onto PyPy just to make it more >attractive. We learned that flexibility and speed are not trivial >to marry, and we have proven our claims about flexibility, but >unfortunately not yet on speed. I would really hate to retract >this claim. ATM, there is simply no prognosis possible. Well, keep the faith ;-) A couple of comments: 1) ISTM that "speed" needs to be split at least into speed of translation and speed of the resulting code/interpreter pair, though interactively one experiences the combined effect. (BTW, by that '/' I mean to highlight the fact that output code could be raw machine code packaged for OS linking/loading and where the "interpreter" is a CPU, vs. byte code for a VM such as found in cpython or jython etc. Also it could be pointed out that the target pair could eventually be on another platform entirely, such as a cell phone or game device, or I suppose a vending machine control system etc.) 1a) Re speed of translation: I note that translation has multiple phases/passes, so I wonder if this would be amenable to pipelining, so as to be able to take automatic advantage of multiple processors, either in the same box, or cooperating via fast network links if there were command line options for such delegation? Could one pickle state and pass it on to the next phase? Could you factor the passes as separate programs to connect in a unix-os-level pipeline? In which case I guess parallel make utilities could possibly come into play. Just thoughts... 1b) (1a) Brings the thought that the same interpreter will have a lot of unused baggage for any particular use, if it is a monolith. Which again brings up the translation-time/platform vs run-time/platform distinctions. I.e. space may be as important as speed in the target run-time environment, but I may be willing to crunch overnight translating on a cluster to make something that will fit and run acceptably on the target platform. BTW, is the issue of ROM-ability entirely low-level, or is it some kind of concern higher up? 1c) Will the different components of an interpreter be directable to dlls/sos and be able to load lazily/be-excluded-if-not-needed/whatever e.g. for good interactive startup, but also be configurable to load monolithically? IWT this is make-file stuff, but a makefile can't select pieces if they were welded together by the way they were developed, so I am wondering how that aspect is happening. 2) ISTM that you guys will be the first to notice what meta-info will be useful to pass into the translator(s) to achieve various results, so I wonder what analogies to e.g., C's "volatile" declaration you may want, and what vehicle you will want for representing and transmitting such information. Pragmas? more -*- cookies -*- ? If you come up with something really good, isn't it likely regular python will adopt it, as well as vice versa? The same goes for language syntax changes that don't hide behind a '#' -- IOW, will you guys wind up with some recommendations for language changes? I don't think you should be burdened with writing PEPs and haggling about syntactic sugar, but IWT you guys will discover optimization obstacles that no one else is in a position to see (or maybe even comprehend ;-), but that you could communicate by saying, "If we had a way to say this about such-and-so in python, we could optimize such-and-so much better" and that could help the language evolve in a win-win way. Is there a way to get a peek into your ongoing thoughts without asking you to hang all your unfinished paintings in the window when you go home from the studio each night? 2a) ISTM there will need to be module-specific meta-info, not just -O or -OO etc command line options. And since module translation occurs primarily at import time, there is the opportunity to pass dynamically determined meta-info to the import process as well, not just letting it see meta-info embedded in the source file or refer to interpreter startup command line parameter info. (I guess you could also package meta-info in various config files, or consolidate such info in one file for members of a package, etc. but I wonder if it's not dangerous to go further down the path of tangling abstract hierarchies with platform specific file system properties). So I'm wondering, if you see useful mods/variations to the semantics of import, or other python aspects, how would such ideas get back to python, so the appropriate language PEPs can get under way (though not prematurely) ? 3) I'm wondering how much of pypy translation technology could be factored out to apply to any language translation, and whether there could be a family of porting tool spinoffs eventually. I guess I rambled, as usual ;-) Anyhow, I am sure EU pypy funding is well spent. I hope they will renew their commitment when the time comes. Best wishes to you all. Regards, Bengt Richter From Ben.Young at risk.sungard.com Mon Sep 19 11:04:54 2005 From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com) Date: Mon, 19 Sep 2005 10:04:54 +0100 Subject: [pypy-dev] Current test_all on windows Message-ID: Hi Everyone, It really looks like your optimization work is progressing! At this rate you'll be faster than CPython by Christmas. I just had a couple of questions from looking over the current changes. Is there a plan to write an API for graph transformations that stops you from performing invalid changes? That would be really useful considering the amount and subtlety of the changes that are being made. Also, do you really need to do inlining to do malloc removal? If you annotate whether a function takes any ownership over an argument (would need to propagate up from the leaf functions) then you can tell whether you can just stack allocate the argument and just pass it in with & (Or is that the plan anyway?). Now as I was told to continue my nagging emails here is a latest list of test failures on Windows. They have all been failing for a week or so (I think). Interestingly the "inf" one only started failing around the time you set more compliant calculations on for Windows Cheers, Ben -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: results.txt URL: From mchermside at ingdirect.com Mon Sep 19 14:57:44 2005 From: mchermside at ingdirect.com (Chermside, Michael) Date: Mon, 19 Sep 2005 08:57:44 -0400 Subject: [pypy-dev] next pypy-sync meeting 20050915 Message-ID: <0CFFADBB825C6249A26FDF11C1772AE10132A321@ingdexj1.ingdirect.com> Michal Wallace writes: > I think you guys have one of the most > impressive project management styles around, > and the sprint idea seems like a great > amount of fun. > > What if you capitalized on *that*? > > A week in europe to work on an open source > project? That would be an awesome vacation! > And to get to learn about compilers and python > along the way? > > What I'm saying is you could market the > sprints as a sort of training package for > big companies interested in trying out python. > They send their developers and pay you guys > to train them. Even if the companies don't > care about python, their developers *might* > and that could be a great reward. [...] > Heck, do your next sprint as a partnership with > these guys: > > http://www.geekcruises.com/ > > Anyway, it might be a crazy idea Definitely a crazy idea. But a GOOD sort of crazy. I REALLY like this idea -- now, I'll be honest and admit that I don't think it would actually WORK (in the sense of making money). It's hard enough to get people to volunteer to come to sprints to work on an open source project (although PyPy is doing QUITE well at it), but getting them to PAY to do so is really stretching credulity. But I LOVE the creativity of this idea... maybe there IS some way to make use of it. ***************************************************************************** This email may contain confidential or privileged information. If you believe you have received the message in error, please notify the sender and delete the message without copying or disclosing it. ***************************************************************************** From mwh at python.net Mon Sep 19 22:01:02 2005 From: mwh at python.net (Michael Hudson) Date: Mon, 19 Sep 2005 21:01:02 +0100 Subject: [pypy-dev] Re: [pypy-svn] r17659 - pypy/dist/pypy/objspace/std In-Reply-To: <20050919141025.B52F427B68@code1.codespeak.net> (adim@codespeak.net's message of "Mon, 19 Sep 2005 16:10:25 +0200 (CEST)") References: <20050919141025.B52F427B68@code1.codespeak.net> Message-ID: <2mr7bkltdt.fsf@starship.python.net> adim at codespeak.net writes: > Author: adim > Date: Mon Sep 19 16:10:24 2005 > New Revision: 17659 > > Modified: > pypy/dist/pypy/objspace/std/intobject.py > Log: > fixed error messages to match CPython's > > > Modified: pypy/dist/pypy/objspace/std/intobject.py > ============================================================================== > --- pypy/dist/pypy/objspace/std/intobject.py (original) > +++ pypy/dist/pypy/objspace/std/intobject.py Mon Sep 19 16:10:24 2005 > @@ -147,7 +147,7 @@ > z = ovfcheck(x // y) > except ZeroDivisionError: > raise OperationError(space.w_ZeroDivisionError, > - space.wrap("integer division by zero")) > + space.wrap("integer division or modulo by zero")) > except OverflowError: > raise FailedToImplement(space.w_OverflowError, > space.wrap("integer division")) This kind of thing bothers me a bit. We're actively making our error messages _worse_, because CPython's are worse. I'm all for conformance, but... Cheers, mwh -- I don't have any special knowledge of all this. In fact, I made all the above up, in the hope that it corresponds to reality. -- Mark Carroll, ucam.chat From tjreedy at udel.edu Tue Sep 20 01:34:29 2005 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 19 Sep 2005 19:34:29 -0400 Subject: [pypy-dev] Re: [pypy-svn] r17659 - pypy/dist/pypy/objspace/std References: <20050919141025.B52F427B68@code1.codespeak.net> <2mr7bkltdt.fsf@starship.python.net> Message-ID: "Michael Hudson" wrote in message news:2mr7bkltdt.fsf at starship.python.net... > This kind of thing bothers me a bit. We're actively making our error > messages _worse_, because CPython's are worse. I'm all for > conformance, but... It would be better, I think, to offer patches to improve CPython's error messages. They are not it's best feature ;-) TJR From tismer at stackless.com Tue Sep 20 03:40:41 2005 From: tismer at stackless.com (Christian Tismer) Date: Tue, 20 Sep 2005 03:40:41 +0200 Subject: [pypy-dev] Current test_all on windows In-Reply-To: References: Message-ID: <432F6899.6070308@stackless.com> Hola Ben, > It really looks like your optimization work is progressing! At this rate > you'll be faster than CPython by Christmas. I just had a couple of > questions from looking over the current changes. Yes, but with no idea of the kind of courve we are sliding along, I would not want to fix the year of this Christmas. :-) > Is there a plan to write an API for graph transformations that stops you > from performing invalid changes? That would be really useful considering > the amount and subtlety of the changes that are being made. No idea. All changes are supported by adding tests, and when the tests are not enough, I'm yelling out loud if my exe doesn't work. > Also, do you really need to do inlining to do malloc removal? If you > annotate whether a function takes any ownership over an argument (would > need to propagate up from the leaf functions) then you can tell whether > you can just stack allocate the argument and just pass it in with & (Or is > that the plan anyway?). This is an alternative to the current concept, that is considered. Inlining was mostly done to have a cheap way to see what saving the mallocs can do. In fact, saving malloc and inlining work together, but are not really related and should be better handled seperately at some point. ATM we get very much inlining which is not desirable. Ideally, we would run some simulation to find the true candidates, and I'm personally even pushing to do the opposite: Move common code which is not frequently used into artificial functions, and re-unify specializations which are not worth it, performance-wise. > Now as I was told to continue my nagging emails here is a latest list of > test failures on Windows. They have all been failing for a week or so (I > think). Interestingly the "inf" one only started failing around the time > you set more compliant calculations on for Windows :-) ok, ok, I was lazy and busy with timings. If nobody else takes it, who is the windows head here? yes, keep naggin', thanks so much -- chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From pedronis at strakt.com Tue Sep 20 14:25:26 2005 From: pedronis at strakt.com (Samuele Pedroni) Date: Tue, 20 Sep 2005 14:25:26 +0200 Subject: [pypy-dev] Re: [pypy-svn] r17675 - pypy/dist/pypy/translator/goal In-Reply-To: <20050920024046.B527C27B80@code1.codespeak.net> References: <20050920024046.B527C27B80@code1.codespeak.net> Message-ID: <432FFFB6.1030607@strakt.com> tismer at codespeak.net wrote: > Author: tismer > Date: Tue Sep 20 04:40:45 2005 > New Revision: 17675 > > Modified: > pypy/dist/pypy/translator/goal/bench-windows.py > Log: > executable abs.richards abs.pystone rel.rich rel.pystone > pypy-c-17439 35165 ms 668.586 42.4 61.1 > pypy-c-17600 26388 ms 900.481 31.8 45.4 > pypy-c-17634 20108 ms 1017.720 24.2 40.1 > pypy-c-17649 22662 ms 1035.910 27.3 39.4 > pypy-c-17674-nolowmem 15817 ms 1981.470 19.1 20.6 > pypy-c-17674-t-lowmem 16834 ms 1274.650 20.3 32.1 > python 2.3.3 830 ms 40861.400 1.0 1.0 > > 17649 was with explicit fixed stack. > Changes after 17634 we not included. > 17674 has an outrageous effect. I cannot really > find out what it was. Did Armin do the fixed stack > patch already? Probably not. Was it Samuele's avoiding > of duplicate zeroing? Really just that? I think so, and > this is incredible. no, there's the effect of 17660 in there too > Even more incredible is the fact that not using using > t-lowmem accelerates pystone so much. This is an indicator > that we missed something used in pystone that still contains > applevel code. I can't believe it, will find it tomorrow. > > I looked at this, there's range, which is not used just for the main loop but in at least in one of the Procs. > Modified: pypy/dist/pypy/translator/goal/bench-windows.py > ============================================================================== > --- pypy/dist/pypy/translator/goal/bench-windows.py (original) > +++ pypy/dist/pypy/translator/goal/bench-windows.py Tue Sep 20 04:40:45 2005 > @@ -3,20 +3,26 @@ > # where a couple of .exe files is expected. > > current_result = """ > -executable abs.richards abs.pystone rel.richards rel.pystone > -pypy-c-17439 35180 ms 661.339 41.9 59.7 > -pypy-c-17512 46007 ms 6 59.205 54.8 59.9 > -pypy-c-17516 37944 ms 704.839 45.2 56.0 > -pypy-c-17545-intern 34309 ms 764.987 40.8 51.6 > -pypy-c-17572 36061 ms 736.094 42.9 53.7 > -pypy-c-17600 26348 ms 901.957 31.4 43.8 > -pypy-c-17623-32_4 24734 ms 970.845 29.4 40.7 > -pypy-c-17634 20088 ms 1018.240 23.9 38.8 > -pypy-c-17649 22902 ms 1018.300 27.3 38.8 > -python 2.3.3 840 ms 39500.600 1.0 1.0 > +executable abs.richards abs.pystone rel.rich rel.pystone > +pypy-c-17439 35165 ms 668.586 42.4 61.1 > +pypy-c-17600 26388 ms 900.481 31.8 45.4 > +pypy-c-17634 20108 ms 1017.720 24.2 40.1 > +pypy-c-17649 22662 ms 1035.910 27.3 39.4 > +pypy-c-17674-nolowmem 15817 ms 1981.470 19.1 20.6 > +pypy-c-17674-t-lowmem 16834 ms 1274.650 20.3 32.1 > +python 2.3.3 830 ms 40861.400 1.0 1.0 > > 17649 was with explicit fixed stack. > Changes after 17634 we not included. > +17674 has an outrageous effect. I cannot really > +find out what it was. Did Armin do the fixed stack > +patch already? Probably not. Was it Samuele's avoiding > +of duplicate zeroing? Really just that? I think so, and > +this is incredible. > +Even more incredible is the fact that not using using > +t-lowmem accelerates pystone so much. This is an indicator > +that we missed something used in pystone that still contains > +applevel code. I can't believe it, will find it tomorrow. > """ > > import os, sys > @@ -48,7 +54,7 @@ > print res > return res > > -def run_richards(executable='python', n=10): > +def run_richards(executable='python', n=20): > argstr = RICHARDS_CMD % n > txt = run_cmd('%s -c "%s"' % (executable, argstr)) > res = get_result(txt, RICHARDS_PATTERN) > @@ -60,18 +66,10 @@ > exes.sort() > return exes > > -LAYOUT = ''' > -executable abs.richards abs.pystone rel.richards rel.pystone > -pypy-c-17439 40929 ms 637.274 47.8 56.6 > -pypy-c-17512 46105 ms 658.1 53.9 54.8 > -pypy-current 33937 ms 698.415 39.6 51.7 > -python 2.3.3 856 ms 36081.6 1.0 1.0 > -''' > - > HEADLINE = '''\ > -executable abs.richards abs.pystone rel.richards rel.pystone''' > +executable abs.richards abs.pystone rel.rich rel.pystone''' > FMT = '''\ > -%-20s ''' + '%5d ms %9.3f ' + '%5.1f %5.1f' > +%-27s ''' + '%5d ms %9.3f ' + '%5.1f %5.1f' > > def main(): > print 'getting the richards reference' > @@ -81,7 +79,7 @@ > res = [] > for exe in get_executables(): > exename = os.path.splitext(exe)[0] > - res.append( (exename, run_richards(exe, 1), run_pystone(exe, 2000)) ) > + res.append( (exename, run_richards(exe, 2), run_pystone(exe, 20000)) ) > res.append( ('python %s' % sys.version.split()[0], ref_rich, ref_stone) ) > print HEADLINE > for exe, rich, stone in res: > _______________________________________________ > pypy-svn mailing list > pypy-svn at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-svn From lac at strakt.com Tue Sep 20 22:17:35 2005 From: lac at strakt.com (Laura Creighton) Date: Tue, 20 Sep 2005 22:17:35 +0200 Subject: [pypy-dev] OSCON Europe Oct 17-20 Amsterdam Message-ID: <200509202017.j8KKHZrC027498@theraft.strakt.com> http://conferences.oreillynet.com/eurooscon/ Anybody going? Laura From hpk at trillke.net Tue Sep 20 22:38:25 2005 From: hpk at trillke.net (holger krekel) Date: Tue, 20 Sep 2005 22:38:25 +0200 Subject: [pypy-dev] OSCON Europe Oct 17-20 Amsterdam In-Reply-To: <200509202017.j8KKHZrC027498@theraft.strakt.com> References: <200509202017.j8KKHZrC027498@theraft.strakt.com> Message-ID: <20050920203825.GC10613@solar.trillke.net> On Tue, Sep 20, 2005 at 22:17 +0200, Laura Creighton wrote: > http://conferences.oreillynet.com/eurooscon/ > > Anybody going? Unfortunately, not from the pypy group as far as i know. I am still considering it but it's actually not really fitting with my time planning (being right after the Paris sprint etc.pp.). holger From lac at strakt.com Tue Sep 20 23:11:02 2005 From: lac at strakt.com (Laura Creighton) Date: Tue, 20 Sep 2005 23:11:02 +0200 Subject: [pypy-dev] OSCON Europe Oct 17-20 Amsterdam In-Reply-To: Message from hpk@trillke.net (holger krekel) of "Tue, 20 Sep 2005 22:38:25 +0200." <20050920203825.GC10613@solar.trillke.net> References: <200509202017.j8KKHZrC027498@theraft.strakt.com> <20050920203825.GC10613@solar.trillke.net> Message-ID: <200509202111.j8KLB2Tk029978@theraft.strakt.com> In a message of Tue, 20 Sep 2005 22:38:25 +0200, holger krekel writes: >On Tue, Sep 20, 2005 at 22:17 +0200, Laura Creighton wrote: >> http://conferences.oreillynet.com/eurooscon/ >> >> Anybody going? > >Unfortunately, not from the pypy group as far as i know. >I am still considering it but it's actually not really fitting >with my time planning (being right after the Paris sprint >etc.pp.). > > holger yes, me too. Alex will be there, and he was hoping to see some of us. But as far as I know, that won't work. He cannot come to the Sprint, since he will be busy celebrating his 50th birthday in Bologna with freinds and family -- I suppose not for the whole week, but still ... Laura From ms at cerenity.org Wed Sep 21 02:15:27 2005 From: ms at cerenity.org (Michael Sparks) Date: Wed, 21 Sep 2005 01:15:27 +0100 Subject: [pypy-dev] Re: OSCON Europe Oct 17-20 Amsterdam References: <200509202017.j8KKHZrC027498@theraft.strakt.com> Message-ID: Laura Creighton wrote: > http://conferences.oreillynet.com/eurooscon/ > > Anybody going? I am - I'm talking about our toys there. (Though we've got quite a few new toys since Python UK & Europython :) If anyone wants to go, I've been given some speakers discount codes which are either 35% off or 25% off depending on who asks when. (The 35% ones have a limit, whereas the 25% ones don't) BTW, I tried posting something a few days back via gmane regarding some comments about the status of generators in RPython, and what barriers there are to their translation right now. It was a lot longer than that, but that's the real gist of it :-) Regards, Michael. -- http://kamaelia.sf.net/ Any similarity between this post and reality is purely co-incidental From serendipity-soft at get2net.dk Wed Sep 21 18:31:38 2005 From: serendipity-soft at get2net.dk (Anders Lehmann) Date: Wed, 21 Sep 2005 18:31:38 +0200 Subject: [pypy-dev] (no subject) Message-ID: <7749BD5D-9CEE-410D-87A4-4787F847D335@get2net.dk> Hi all, Next #pypy-sync meeting on freenode.net is tomorrow Thursday at 1pm as usual. Regular topics -------------- * activity reports (3 prepared lines of info) * resolve conflicts/blockers Here is my draft proposal of Topics of the week ------------------ translate_pypy_new What more do want to do with this ? A pypy scrap-book ? Should we have a special place to collect press coverage, quotes, conference presentation, etc There remains about two weeks before the Paris Sprint, are there other areas that need refactoring which could be done before Paris ? Please propose additions and changes. See you tomorrow at 1 pm. Kind regards Anders (ale) From serendipity-soft at get2net.dk Wed Sep 21 22:25:44 2005 From: serendipity-soft at get2net.dk (Anders Lehmann) Date: Wed, 21 Sep 2005 22:25:44 +0200 Subject: [pypy-dev] (no subject) Should have been : "Next pypy-sync meeting tomorrow" In-Reply-To: <7749BD5D-9CEE-410D-87A4-4787F847D335@get2net.dk> References: <7749BD5D-9CEE-410D-87A4-4787F847D335@get2net.dk> Message-ID: <136DF178-F1BF-4282-83CF-C3625FF82D37@get2net.dk> Den 21/09/2005 kl. 18.31 skrev Anders Lehmann: > > Hi all, > > Next #pypy-sync meeting on freenode.net is tomorrow Thursday at 1pm as > usual. > > Regular topics > -------------- > > * activity reports (3 prepared lines of info) > * resolve conflicts/blockers > > Here is my draft proposal of > > Topics of the week > ------------------ > > translate_pypy_new > > What more do want to do with this ? > > A pypy scrap-book ? > > Should we have a special place to collect press coverage, > quotes, conference presentation, etc > > There remains about two weeks before the Paris Sprint, are there > other areas that need refactoring which could be done before Paris ? > > > > Please propose additions and changes. > > See you tomorrow at 1 pm. > > Kind regards > > Anders (ale) > > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From michal at sabren.com Wed Sep 21 22:51:20 2005 From: michal at sabren.com (Michal Wallace) Date: Wed, 21 Sep 2005 16:51:20 -0400 (EDT) Subject: [pypy-dev] (no subject) Should have been : "Next pypy-sync meeting tomorrow" In-Reply-To: <136DF178-F1BF-4282-83CF-C3625FF82D37@get2net.dk> References: <7749BD5D-9CEE-410D-87A4-4787F847D335@get2net.dk> <136DF178-F1BF-4282-83CF-C3625FF82D37@get2net.dk> Message-ID: On Wed, 21 Sep 2005, Anders Lehmann wrote: > > Next #pypy-sync meeting on freenode.net is tomorrow Thursday at > > 1pm as usual. Which timezone? :) Sincerely, Michal J Wallace Sabren Enterprises, Inc. ------------------------------------- contact: michal at sabren.com hosting: http://www.cornerhost.com/ my site: http://www.withoutane.com/ ------------------------------------- From serendipity-soft at get2net.dk Thu Sep 22 09:57:28 2005 From: serendipity-soft at get2net.dk (Anders Lehmann) Date: Thu, 22 Sep 2005 09:57:28 +0200 Subject: [pypy-dev] (no subject) Should have been : "Next pypy-sync meeting tomorrow" In-Reply-To: References: <7749BD5D-9CEE-410D-87A4-4787F847D335@get2net.dk> <136DF178-F1BF-4282-83CF-C3625FF82D37@get2net.dk> Message-ID: CET GMT+2 Den 21/09/2005 kl. 22.51 skrev Michal Wallace: > On Wed, 21 Sep 2005, Anders Lehmann wrote: > > >>> Next #pypy-sync meeting on freenode.net is tomorrow Thursday at >>> 1pm as usual. >>> > > Which timezone? :) > > Sincerely, > > Michal J Wallace > Sabren Enterprises, Inc. > ------------------------------------- > contact: michal at sabren.com > hosting: http://www.cornerhost.com/ > my site: http://www.withoutane.com/ > ------------------------------------- > > From 2005a at usenet.alexanderweb.de Thu Sep 22 12:20:55 2005 From: 2005a at usenet.alexanderweb.de (Alexander Schremmer) Date: Thu, 22 Sep 2005 12:20:55 +0200 Subject: [pypy-dev] LowLevelDatabase instance has no attribute 'pyobjmaker' Message-ID: Hi, translating this target: def entry_point(argv): print "Test" return 0 def target(geninterp=True): return entry_point, None fails with: ... File "/home/contest/xoraxax/pypy-dist/pypy/translator/c/database.py", line 181, in complete self.get(value) File "/home/contest/xoraxax/pypy-dist/pypy/translator/c/database.py", line 123, in get node = self.getcontainernode(obj._obj) File "/home/contest/xoraxax/pypy-dist/pypy/translator/c/database.py", line 101, in getcontainernode node = nodefactory(self, T, container) File "/home/contest/xoraxax/pypy-dist/pypy/translator/c/node.py", line 579, in __init__ self.name = db.pyobjmaker.computenameof(obj.value) AttributeError: LowLevelDatabase instance has no attribute 'pyobjmaker' Looks like a bug to me. Or is print not supported in rpython? Kind regards, Alexander From Ben.Young at risk.sungard.com Thu Sep 22 13:47:14 2005 From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com) Date: Thu, 22 Sep 2005 12:47:14 +0100 Subject: [pypy-dev] Crash! Message-ID: Hi Everyone, Current checkout, compiled pypy (windows as usual): $ ./pypy-c.exe debug: entry point starting debug: argv -> c:\Documents and Settings\YoungB\dist\pypy\translator\goal\pypy-c.exe debug: importing code debug: calling code.interact() Python 2.4.1 (pypy 0.7.1 build) on win32 Type "help", "copyright", "credits" or "license" for more information. (InteractiveConsole) >>>> fe -> Segfault Appears to be a problem throwing exceptions. On another point, I notice that when lists of pointers are resized lots of incref/decrefs happen when all that really needs to happen is a move operation (or even a memcpy of the whole block)? Also, could all lists of pointers share the same c representation, with casts from void*? Cheers, Ben From Ben.Young at risk.sungard.com Fri Sep 23 12:11:01 2005 From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com) Date: Fri, 23 Sep 2005 11:11:01 +0100 Subject: [pypy-dev] Re: [pypy-svn] r17783 - in pypy/dist/pypy/objspace/flow: . test In-Reply-To: <20050923100405.4E4D627B93@code1.codespeak.net> Message-ID: Hi Armin, Just a quick question. Is mutating a list while iterating over it rpython? (Also was wondering if you noticed the mail I sent yesterday to pypy-dev?) Cheers, Ben P.S Do you think there is ever going to be a sprint in London sometime as I would love to be able to come to one one day! I guess it's down to someone to organise a room etc. pypy-svn-bounces at codespeak.net wrote on 23/09/2005 11:04:05: > Author: arigo > Date: Fri Sep 23 12:04:04 2005 > New Revision: 17783 > > Modified: > pypy/dist/pypy/objspace/flow/model.py > pypy/dist/pypy/objspace/flow/test/test_model.py > Log: > Further simplification of traverse(), the full flexibility of which > is never used anywhere and the speed of which is crucial during > translation. > > > Modified: pypy/dist/pypy/objspace/flow/model.py > ============================================================================== > --- pypy/dist/pypy/objspace/flow/model.py (original) > +++ pypy/dist/pypy/objspace/flow/model.py Fri Sep 23 12:04:04 2005 > @@ -344,52 +344,65 @@ > #_________________________________________________________ > # a visitor for easy traversal of the above model > > -import inspect # for getmro > +##import inspect # for getmro > > -class traverse: > +##class traverse: > > - def __init__(self, visitor, functiongraph): > - """ send the visitor over all (reachable) nodes. > - the visitor needs to have either callable attributes > 'visit_typename' > - or otherwise is callable itself. > - """ > - self.visitor = visitor > - self.visitor_cache = {} > - self.seen = {} > - self.visit(functiongraph) > - > - def visit(self, node): > - if id(node) in self.seen: > - return > +## def __init__(self, visitor, functiongraph): > +## """ send the visitor over all (reachable) nodes. > +## the visitor needs to have either callable attributes > 'visit_typename' > +## or otherwise is callable itself. > +## """ > +## self.visitor = visitor > +## self.visitor_cache = {} > +## self.seen = {} > +## self.visit(functiongraph) > + > +## def visit(self, node): > +## if id(node) in self.seen: > +## return > + > +## # do the visit > +## cls = node.__class__ > +## try: > +## consume = self.visitor_cache[cls] > +## except KeyError: > +## for subclass in inspect.getmro(cls): > +## consume = getattr(self.visitor, "visit_" + > subclass.__name__, None) > +## if consume: > +## break > +## else: > +## consume = getattr(self.visitor, 'visit', self.visitor) > + > +## assert callable(consume), "visitor not found for > %r on %r" % (cls, self.visitor) > + > +## self.visitor_cache[cls] = consume > + > +## self.seen[id(node)] = consume(node) > + > +## # recurse > +## if isinstance(node, Block): > +## for obj in node.exits: > +## self.visit(obj) > +## elif isinstance(node, Link): > +## self.visit(node.target) > +## elif isinstance(node, FunctionGraph): > +## self.visit(node.startblock) > +## else: > +## raise ValueError, "could not dispatch %r" % cls > + > +def traverse(visit, functiongraph): > + pending = [functiongraph.startblock] > + seen = {id(functiongraph.startblock): True} > + for block in pending: > + visit(block) > + for link in block.exits: > + visit(link) > + targetid = id(link.target) > + if targetid not in seen: > + pending.append(link.target) > + seen[targetid] = True > > - # do the visit > - cls = node.__class__ > - try: > - consume = self.visitor_cache[cls] > - except KeyError: > - for subclass in inspect.getmro(cls): > - consume = getattr(self.visitor, "visit_" + > subclass.__name__, None) > - if consume: > - break > - else: > - consume = getattr(self.visitor, 'visit', self.visitor) > - > - assert callable(consume), "visitor not found for %r > on %r" % (cls, self.visitor) > - > - self.visitor_cache[cls] = consume > - > - self.seen[id(node)] = consume(node) > - > - # recurse > - if isinstance(node, Block): > - for obj in node.exits: > - self.visit(obj) > - elif isinstance(node, Link): > - self.visit(node.target) > - elif isinstance(node, FunctionGraph): > - self.visit(node.startblock) > - else: > - raise ValueError, "could not dispatch %r" % cls > > def flatten(funcgraph): > l = [] > > Modified: pypy/dist/pypy/objspace/flow/test/test_model.py > ============================================================================== > --- pypy/dist/pypy/objspace/flow/test/test_model.py (original) > +++ pypy/dist/pypy/objspace/flow/test/test_model.py Fri Sep 23 12:04:04 2005 > @@ -28,27 +28,27 @@ > graph = self.getflow(self.simplefunc) > assert all_operations(graph) == {'add': 1} > > - def test_class(self): > - graph = self.getflow(self.simplefunc) > +## def test_class(self): > +## graph = self.getflow(self.simplefunc) > > - class MyVisitor: > - def __init__(self): > - self.blocks = [] > - self.links = [] > - > - def visit_FunctionGraph(self, graph): > - self.graph = graph > - def visit_Block(self, block): > - self.blocks.append(block) > - def visit_Link(self, link): > - self.links.append(link) > - > - v = MyVisitor() > - traverse(v, graph) > - #assert len(v.blocks) == 2 > - #assert len(v.links) == 1 > - assert v.graph == graph > - assert v.links[0] == graph.startblock.exits[0] > +## class MyVisitor: > +## def __init__(self): > +## self.blocks = [] > +## self.links = [] > + > +## def visit_FunctionGraph(self, graph): > +## self.graph = graph > +## def visit_Block(self, block): > +## self.blocks.append(block) > +## def visit_Link(self, link): > +## self.links.append(link) > + > +## v = MyVisitor() > +## traverse(v, graph) > +## #assert len(v.blocks) == 2 > +## #assert len(v.links) == 1 > +## assert v.graph == graph > +## assert v.links[0] == graph.startblock.exits[0] > > ## def test_partial_class(self): > ## graph = self.getflow(self.simplefunc) > _______________________________________________ > pypy-svn mailing list > pypy-svn at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-svn > From lac at strakt.com Fri Sep 23 13:33:14 2005 From: lac at strakt.com (Laura Creighton) Date: Fri, 23 Sep 2005 13:33:14 +0200 Subject: [pypy-dev] Re: [pypy-svn] r17783 - in pypy/dist/pypy/objspace/flow: . test In-Reply-To: Message from Ben.Young@risk.sungard.com of "Fri, 23 Sep 2005 11:11:01 BST." References: Message-ID: <200509231133.j8NBXEDc031823@theraft.strakt.com> In a message of Fri, 23 Sep 2005 11:11:01 BST, Ben.Young at risk.sungard.com writes: >Hi Armin, > >Just a quick question. Is mutating a list while iterating over it rpython >? > >(Also was wondering if you noticed the mail I sent yesterday to pypy-dev? >) > >Cheers, >Ben > >P.S Do you think there is ever going to be a sprint in London sometime as > >I would love to be able to come to one one day! I guess it's down to >someone to organise a room etc. Wonder if the BBC would let us use their place. Michael? Laura From Ben.Young at risk.sungard.com Fri Sep 23 14:05:13 2005 From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com) Date: Fri, 23 Sep 2005 13:05:13 +0100 Subject: [pypy-dev] Fw: Crash! Message-ID: Further to my previous mail, current check seems to crash much more often now: $ ./pypy-c.exe debug: entry point starting debug: argv -> c:\Documents and Settings\YoungB\dist\pypy\translator\goal\pypy-c.exe debug: importing code debug: calling code.interact() Python 2.4.1 (pypy 0.7.1 build) on win32 Type "help", "copyright", "credits" or "license" for more information. (InteractiveConsole) >>>> import test >>>> test.rpython.main(100) -> Segfault Cheers, Ben ----- Forwarded by Ben Young/Infinity on 23/09/2005 13:05 ----- Ben Young/Infinity wrote on 22/09/2005 12:47:14: > Hi Everyone, > > Current checkout, compiled pypy (windows as usual): > > $ ./pypy-c.exe > debug: entry point starting > debug: argv -> c:\Documents and > Settings\YoungB\dist\pypy\translator\goal\pypy-c.exe > debug: importing code > debug: calling code.interact() > Python 2.4.1 (pypy 0.7.1 build) on win32 > Type "help", "copyright", "credits" or "license" for more information. > (InteractiveConsole) > >>>> fe > > -> Segfault > > Appears to be a problem throwing exceptions. > > On another point, I notice that when lists of pointers are resized > lots of incref/decrefs happen when all that really needs to happen > is a move operation (or even a memcpy of the whole block)? Also, > could all lists of pointers share the same c representation, with > casts from void*? > > Cheers, > Ben From Ben.Young at risk.sungard.com Fri Sep 23 14:15:29 2005 From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com) Date: Fri, 23 Sep 2005 13:15:29 +0100 Subject: [pypy-dev] Fw: Crash! In-Reply-To: Message-ID: Sorry about replying to myself, but I must be going blind as the crash is still being caused by an exception being thrown. Weird that the tests don't pick this up! Cheers, Ben pypy-dev-bounces at codespeak.net wrote on 23/09/2005 13:05:13: > Further to my previous mail, current check seems to crash much more often > now: > > $ ./pypy-c.exe > debug: entry point starting > debug: argv -> c:\Documents and > Settings\YoungB\dist\pypy\translator\goal\pypy-c.exe > debug: importing code > debug: calling code.interact() > Python 2.4.1 (pypy 0.7.1 build) on win32 > Type "help", "copyright", "credits" or "license" for more information. > (InteractiveConsole) > >>>> import test > >>>> test.rpython.main(100) > > -> Segfault > > Cheers, > Ben > > ----- Forwarded by Ben Young/Infinity on 23/09/2005 13:05 ----- > > Ben Young/Infinity wrote on 22/09/2005 12:47:14: > > > Hi Everyone, > > > > Current checkout, compiled pypy (windows as usual): > > > > $ ./pypy-c.exe > > debug: entry point starting > > debug: argv -> c:\Documents and > > Settings\YoungB\dist\pypy\translator\goal\pypy-c.exe > > debug: importing code > > debug: calling code.interact() > > Python 2.4.1 (pypy 0.7.1 build) on win32 > > Type "help", "copyright", "credits" or "license" for more information. > > (InteractiveConsole) > > >>>> fe > > > > -> Segfault > > > > Appears to be a problem throwing exceptions. > > > > On another point, I notice that when lists of pointers are resized > > lots of incref/decrefs happen when all that really needs to happen > > is a move operation (or even a memcpy of the whole block)? Also, > > could all lists of pointers share the same c representation, with > > casts from void*? > > > > Cheers, > > Ben > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > > From boria at fbg.uklinux.net Fri Sep 23 14:02:02 2005 From: boria at fbg.uklinux.net (Boria Feigin) Date: Fri, 23 Sep 2005 13:02:02 +0100 Subject: [pypy-dev] Project suggestions Message-ID: <20050923120202.GA7065@antelope> Hi, All I am an undergraduate student (at a top-5 CS university in the UK) looking for a fun final-year project to sink my teeth into. Over the last couple of months, I became quite interested in programming languages, interpreters, virtual machines, and so on. I found out about PyPy at Holger Krekel's talks at 21C3 in December and now it seems like an ideal way to do all of these things! Can anyone suggest pointers to areas of PyPy that need work? Or, better yet, ideas that need further development -- research in literature, prototyping, and implementation? Thanks for your time. Please CC me -- I am not on the list. -- Boris From Ben.Young at risk.sungard.com Fri Sep 23 16:18:51 2005 From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com) Date: Fri, 23 Sep 2005 15:18:51 +0100 Subject: [pypy-dev] #pypy Message-ID: Hi Everyone, Does anyone know a way of accessing irc through a port 80, http only proxy. It appears that #pypy has become the primary way of asking questions/reporting things, but I can only send emails or read web pages :( due to our workplace proxy. Cheers, Ben From bea at netg.se Fri Sep 23 16:21:24 2005 From: bea at netg.se (Beatrice During) Date: Fri, 23 Sep 2005 16:21:24 +0200 (CEST) Subject: [pypy-dev] Project suggestions In-Reply-To: <20050923120202.GA7065@antelope> References: <20050923120202.GA7065@antelope> Message-ID: Hi there On Fri, 23 Sep 2005, Boria Feigin wrote: > Hi, All > > I am an undergraduate student (at a top-5 CS university in the UK) > looking for a fun final-year project to sink my teeth into. > Over the last couple of months, I became quite interested in programming > languages, interpreters, virtual machines, and so on. I found out about > PyPy at Holger Krekel's talks at 21C3 in December and now it seems like > an ideal way to do all of these things! > > Can anyone suggest pointers to areas of PyPy that need work? Or, better > yet, ideas that need further development -- research in literature, > prototyping, and implementation? I will try to answer this - maybe the others can fill in? First of all - thank you for your interest in PyPy! We have lots of work ;-) - the main question is how familiar you are with Python specifically and wether you have read the documentation that exists on Codespeak? As to the research aspect - we are just now entering our second phase of the funded (EU) side of PyPy and the upcoming 9 months is all about optimizations and exploration so there should be interesting stuff to do that could fit well for a final year project. In October, 10-16th to be more exact we will meet up to sprint in Paris and we will devote some time to plan the work in phase 2 in some more detail. Maybe we could get back to you after that? Meanwhile (if you havent done so already) you could read some of the documentation and identify areas of interest to you? > Thanks for your time. > > Please CC me -- I am not on the list. > > -- > Boris Cheers Beatrice D?ring Assistant project manager/PyPy The non techie person ;-) From pedronis at strakt.com Fri Sep 23 16:29:13 2005 From: pedronis at strakt.com (Samuele Pedroni) Date: Fri, 23 Sep 2005 16:29:13 +0200 Subject: [pypy-dev] Fw: Crash! In-Reply-To: References: Message-ID: <43341139.5010106@strakt.com> Ben.Young at risk.sungard.com wrote: > Sorry about replying to myself, but I must be going blind as the crash is > still being caused by an exception being thrown. Weird that the tests > don't pick this up! > the fact is that we cannot reproduce these segfaults, we are not getting them. And there was no recent change that is likely candidate to cause this. So we are a bit in the dark here. Could at least tell us you exact configuration. What compiler, optimisations, whether you are using Boehm or refcounting, using -t-lowmem or not? > Cheers, > Ben > > > pypy-dev-bounces at codespeak.net wrote on 23/09/2005 13:05:13: > > >>Further to my previous mail, current check seems to crash much more > > often > >>now: >> >>$ ./pypy-c.exe >>debug: entry point starting >>debug: argv -> c:\Documents and >>Settings\YoungB\dist\pypy\translator\goal\pypy-c.exe >>debug: importing code >>debug: calling code.interact() >>Python 2.4.1 (pypy 0.7.1 build) on win32 >>Type "help", "copyright", "credits" or "license" for more information. >>(InteractiveConsole) >> >>>>>>import test >>>>>>test.rpython.main(100) >> >>-> Segfault >> >>Cheers, >>Ben >> >>----- Forwarded by Ben Young/Infinity on 23/09/2005 13:05 ----- >> >>Ben Young/Infinity wrote on 22/09/2005 12:47:14: >> >> >>>Hi Everyone, >>> >>>Current checkout, compiled pypy (windows as usual): >>> >>>$ ./pypy-c.exe >>>debug: entry point starting >>>debug: argv -> c:\Documents and >>>Settings\YoungB\dist\pypy\translator\goal\pypy-c.exe >>>debug: importing code >>>debug: calling code.interact() >>>Python 2.4.1 (pypy 0.7.1 build) on win32 >>>Type "help", "copyright", "credits" or "license" for more information. >>>(InteractiveConsole) >>> >>>>>>>fe >>> >>>-> Segfault >>> >>>Appears to be a problem throwing exceptions. >>> >>>On another point, I notice that when lists of pointers are resized >>>lots of incref/decrefs happen when all that really needs to happen >>>is a move operation (or even a memcpy of the whole block)? Also, >>>could all lists of pointers share the same c representation, with >>>casts from void*? >>> >>>Cheers, >>>Ben >> >>_______________________________________________ >>pypy-dev at codespeak.net >>http://codespeak.net/mailman/listinfo/pypy-dev >> >> > > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev From Ben.Young at risk.sungard.com Fri Sep 23 16:30:22 2005 From: Ben.Young at risk.sungard.com (Ben.Young at risk.sungard.com) Date: Fri, 23 Sep 2005 15:30:22 +0100 Subject: [pypy-dev] Fw: Crash! In-Reply-To: <43341139.5010106@strakt.com> Message-ID: Hi Samuele, My build configuation is windows, no boehm, no lomem. Is there an option to build a debug binary on windows? If so i'll look into what's happening for you! The visual studio debugger is pretty good. Cheers, Ben pypy-dev-bounces at codespeak.net wrote on 23/09/2005 15:29:13: > Ben.Young at risk.sungard.com wrote: > > Sorry about replying to myself, but I must be going blind as the crash is > > still being caused by an exception being thrown. Weird that the tests > > don't pick this up! > > > > the fact is that we cannot reproduce these segfaults, we are not getting > them. And there was no recent change that is likely candidate to cause > this. So we are a bit in the dark here. > > Could at least tell us you exact configuration. What compiler, > optimisations, whether you are using Boehm or refcounting, using > -t-lowmem or not? > > > > Cheers, > > Ben > > > > > > pypy-dev-bounces at codespeak.net wrote on 23/09/2005 13:05:13: > > > > > >>Further to my previous mail, current check seems to crash much more > > > > often > > > >>now: > >> > >>$ ./pypy-c.exe > >>debug: entry point starting > >>debug: argv -> c:\Documents and > >>Settings\YoungB\dist\pypy\translator\goal\pypy-c.exe > >>debug: importing code > >>debug: calling code.interact() > >>Python 2.4.1 (pypy 0.7.1 build) on win32 > >>Type "help", "copyright", "credits" or "license" for more information. > >>(InteractiveConsole) > >> > >>>>>>import test > >>>>>>test.rpython.main(100) > >> > >>-> Segfault > >> > >>Cheers, > >>Ben > >> > >>----- Forwarded by Ben Young/Infinity on 23/09/2005 13:05 ----- > >> > >>Ben Young/Infinity wrote on 22/09/2005 12:47:14: > >> > >> > >>>Hi Everyone, > >>> > >>>Current checkout, compiled pypy (windows as usual): > >>> > >>>$ ./pypy-c.exe > >>>debug: entry point starting > >>>debug: argv -> c:\Documents and > >>>Settings\YoungB\dist\pypy\translator\goal\pypy-c.exe > >>>debug: importing code > >>>debug: calling code.interact() > >>>Python 2.4.1 (pypy 0.7.1 build) on win32 > >>>Type "help", "copyright", "credits" or "license" for more information. > >>>(InteractiveConsole) > >>> > >>>>>>>fe > >>> > >>>-> Segfault > >>> > >>>Appears to be a problem throwing exceptions. > >>> > >>>On another point, I notice that when lists of pointers are resized > >>>lots of incref/decrefs happen when all that really needs to happen > >>>is a move operation (or even a memcpy of the whole block)? Also, > >>>could all lists of pointers share the same c representation, with > >>>casts from void*? > >>> > >>>Cheers, > >>>Ben > >> > >>_______________________________________________ > >>pypy-dev at codespeak.net > >>http://codespeak.net/mailman/listinfo/pypy-dev > >> > >> > > > > > > _______________________________________________ > > pypy-dev at codespeak.net > > http://codespeak.net/mailman/listinfo/pypy-dev > > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > > From arigo at tunes.org Fri Sep 23 17:23:19 2005 From: arigo at tunes.org (Armin Rigo) Date: Fri, 23 Sep 2005 17:23:19 +0200 Subject: [pypy-dev] a faster Python not a primary goal of PyPy? In-Reply-To: <5.0.2.1.1.20050915154827.00b3d760@mail.oz.net> References: <43297070.4020004@infrae.com> <4328B40C.2060809@stackless.com> <20050915065824.GO22844@solar.trillke.net> <43295015.2060109@infrae.com> <200509151252.55437.jacob@strakt.com> <432953A6.6050103@stackless.com> <43297070.4020004@infrae.com> <5.0.2.1.1.20050915154827.00b3d760@mail.oz.net> Message-ID: <20050923152319.GA15985@code1.codespeak.net> Hi Bengt, On Thu, Sep 15, 2005 at 06:46:30PM -0700, Bengt Richter wrote: > 2) ISTM that you guys will be the first to notice what meta-info will be useful > to pass into the translator(s) to achieve various results, so I wonder what > analogies to e.g., C's "volatile" declaration you may want, and what vehicle > you will want for representing and transmitting such information. Pragmas? > more -*- cookies -*- ? If you come up with something really good, isn't it > likely regular python will adopt it, as well as vice versa? > > The same goes for language syntax changes that don't hide behind a '#' -- > IOW, will you guys wind up with some recommendations for language changes? I don't think either of this is likely. We are not suggesting language changes or seeing the need for some. And for RPython we can mostly deal with regular Python syntax with a couple of additional specific objects that don't require any kind of syntax; for example, as in C there is sometimes a good reason to use signed or unsigned integers but Python doesn't have the distinction, we have defined a class "r_uint" which works like an unsigned integer with the same wrap-around semantics as C. The translator knows that it can be turned into a simple "unsigned int" in C. There is no need for more advanced support. In general we try hard to keep language changes separated from PyPy, with the exception of new built-in types providing extra features that are not really possible in CPython, just like Stackless did -- e.g. the thunk object space. > 3) I'm wondering how much of pypy translation technology could be factored out > to apply to any language translation, and whether there could be a family of > porting tool spinoffs eventually. A lot of it. That's part of our goal: making it applicable to different languages. If anyone shows up with a Javascript parser, I'm sure we could together hack (and translate (and later have a JIT from)) a basic Javascript interpreter within a few weeks :-) A bientot, Armin. From boria at fbg.uklinux.net Fri Sep 23 17:28:46 2005 From: boria at fbg.uklinux.net (Boria Feigin) Date: Fri, 23 Sep 2005 16:28:46 +0100 Subject: [pypy-dev] Project suggestions In-Reply-To: References: <20050923120202.GA7065@antelope> Message-ID: <20050923152846.GA7662@antelope> Thanks for the quick reply. > >Can anyone suggest pointers to areas of PyPy that need work? Or, better > >yet, ideas that need further development -- research in literature, > >prototyping, and implementation? > > I will try to answer this - maybe the others can fill in? > > First of all - thank you for your interest in PyPy! We have lots of work Great! > ;-) - the main question is how familiar you are with Python specifically Reasonably familiar (a couple of university projects, about 3.5K lines in total), though there's definitely room to improve :) > and wether you have read the documentation that exists on Codespeak? Not yet. I'll do it this weekend. > As to the research aspect - we are just now entering our second phase of > the funded (EU) side of PyPy and the upcoming 9 months is all about > optimizations and exploration so there should be interesting stuff to do > that could fit well for a final year project. :) > In October, 10-16th to be more exact we will meet up to sprint in Paris > and we will devote some time to plan the work in phase 2 in some more > detail. Maybe we could get back to you after that? > > Meanwhile (if you havent done so already) you could read some of the > documentation and identify areas of interest to you? Sounds good. I'll read the docs, subscribe to the list, and send another e-mail about 15th or 16th of October. > >Thanks for your time. > > > >Please CC me -- I am not on the list. > > > >-- > >Boris > > Cheers > > Beatrice D?ring > Assistant project manager/PyPy > The non techie person ;-) -- Boris From arigo at tunes.org Fri Sep 23 17:42:08 2005 From: arigo at tunes.org (Armin Rigo) Date: Fri, 23 Sep 2005 17:42:08 +0200 Subject: [pypy-dev] features, benefits, and recruiting help In-Reply-To: <200509152331.45535.jacob@strakt.com> References: <4328B40C.2060809@stackless.com> <200509151859.j8FIx8eD021247@theraft.strakt.com> <200509152331.45535.jacob@strakt.com> Message-ID: <20050923154208.GB15985@code1.codespeak.net> Hi, Here's an update on Jacob's answer... On Thu, Sep 15, 2005 at 11:31:45PM +0200, Jacob Hall?n wrote: > > So, for example, jython and ironpython and pirate > > could all eventually just be implemented as > > backends for pypy? > > With a hand-wavy interpretation of what you are saying that would be true. Well, I guess we should really answer "yes, absolutely". This has been a point we have tried to push forward quite a lot, with apparently no real success so far: we want PyPy to become a replacement for manually written Python interpreters. > We are still 200 times slower than CPython and we still use quite a lot of > memory. After removing the most obvious inefficiencies we are now around 20 times slower. About memory usage, I think that it has always been reasonable (Jacob: you must think about the memory needed to translate PyPy, which is indeed still a bit unreasonably high.) A bientot, Armin From arigo at tunes.org Fri Sep 23 17:54:53 2005 From: arigo at tunes.org (Armin Rigo) Date: Fri, 23 Sep 2005 17:54:53 +0200 Subject: [pypy-dev] LowLevelDatabase instance has no attribute 'pyobjmaker' In-Reply-To: References: Message-ID: <20050923155453.GC15985@code1.codespeak.net> Hi Alexander, On Thu, Sep 22, 2005 at 12:20:55PM +0200, Alexander Schremmer wrote: > Looks like a bug to me. Or is print not supported in rpython? Indeed, 'print' is only partially supported. It is a priori supported, but uses objects not supported by the annotator -- which become so-called "SomeObjects". This means that translating a print statement produces C code that uses "PyObject *" variables from the CPython run-time. This in turn only works when building a CPython extension module instead of a stand-alone executable. See in targetpypystandalone.py how we work around this problem by using os.write() for debugging purposes. A bientot, Armin From Michael.Sparks at rd.bbc.co.uk Fri Sep 23 16:38:55 2005 From: Michael.Sparks at rd.bbc.co.uk (Michael Sparks) Date: Fri, 23 Sep 2005 15:38:55 +0100 Subject: [pypy-dev] #pypy In-Reply-To: References: Message-ID: <200509231538.55643.Michael.Sparks@rd.bbc.co.uk> On Friday 23 Sep 2005 15:18, Ben.Young at risk.sungard.com wrote: > Hi Everyone, > > Does anyone know a way of accessing irc through a port 80, http only > proxy. It appears that #pypy has become the primary way of asking > questions/reporting things, but I can only send emails or read web pages Try gaim - it can proxy IRC through an HTTP proxy. It's very clear how you do this, but boils down to: * Go into preferences * Find proxy * Set to HTTP, fill in your proxy details Then just use as normal. Often people have a socks proxy for this sort of thing which is often more preferable. http://gaim.sourceforge.net/ Regards, Michael. From arigo at tunes.org Fri Sep 23 18:01:19 2005 From: arigo at tunes.org (Armin Rigo) Date: Fri, 23 Sep 2005 18:01:19 +0200 Subject: [pypy-dev] Crash! In-Reply-To: References: Message-ID: <20050923160119.GD15985@code1.codespeak.net> Hi Ben, On Thu, Sep 22, 2005 at 12:47:14PM +0100, Ben.Young at risk.sungard.com wrote: > Current checkout, compiled pypy (windows as usual): > -> Segfault Grumble. We're still trying to reproduce the problem... > On another point, I notice that when lists of pointers are resized lots of > incref/decrefs happen when all that really needs to happen is a move > operation (or even a memcpy of the whole block)? Also, could all lists of > pointers share the same c representation, with casts from void*? Yes, there are obvious inefficiencies here. We probably don't care too much about reference counting in the immediate future, as producing a version of PyPy using the Boehm garbage collector produces code that is both much shorter and much faster. At some point, however, we will need indeed to come back to this problem and figure out good algorithms that can detect and remove the majority of these wasted incref/decrefs. For example, a lot of references can be "borrowed" from the caller or from another object, or even "stealed" away from the caller or another object. If we can find good enough algorithms, the case of list resizing would be solved by finding out that we can steal the references from the old list. I think it would be a better approach than special-casing list resizing with some kind of memory-move operation. A bientot, Armin. From arigo at tunes.org Fri Sep 23 18:10:17 2005 From: arigo at tunes.org (Armin Rigo) Date: Fri, 23 Sep 2005 18:10:17 +0200 Subject: [pypy-dev] Re: [pypy-svn] r17783 - in pypy/dist/pypy/objspace/flow: . test In-Reply-To: References: <20050923100405.4E4D627B93@code1.codespeak.net> Message-ID: <20050923161017.GE15985@code1.codespeak.net> Hi Ben, On Fri, Sep 23, 2005 at 11:11:01AM +0100, Ben.Young at risk.sungard.com wrote: > Just a quick question. Is mutating a list while iterating over it rpython? Good question. At the moment, it definitely works in RPython as it works in Python. I can think of a few optimizations that would depend on not mutating a list during iteration, though. It's thus probably better to code the loop explicitely as a while loop in this case (which produces the same code anyway at the moment, it's is just more verbose to write). > P.S Do you think there is ever going to be a sprint in London sometime as > I would love to be able to come to one one day! I guess it's down to > someone to organise a room etc. Yes, I hope there will eventually be a sprint in London or at least in England. I know of one person who proposed to organize one; I'll let him comment on this if he wants to :-) Otherwise, it's indeed relatively easy to organize a sprint. A room with internet is basically it, together with some hints and addresses for accomodation. A bientot, Armin. From arigo at tunes.org Fri Sep 23 18:29:16 2005 From: arigo at tunes.org (Armin Rigo) Date: Fri, 23 Sep 2005 18:29:16 +0200 Subject: [pypy-dev] Project suggestions In-Reply-To: <20050923152846.GA7662@antelope> References: <20050923120202.GA7065@antelope> <20050923152846.GA7662@antelope> Message-ID: <20050923162916.GF15985@code1.codespeak.net> Hi Boria, We are indeed starting to think about more focused research areas in PyPy. For example, along these lines, we will need more work on compiler optimizations and generation of code for various architectures, both either statically or just-in-time. More along the lines of interpreters and virtual machines, we could start investigating new aspects that would be useful to code into the interpreter or the translation process: continuations, persistence (either dumping a whole-process image or something more fine-grained), security (running code in a sandbox), and much more, all of which is hinted at in some documentation on Codespeak. Finally, there is also the idea of supporting other dynamic languages than Python by writing an interpreter for them in PyPy. As Bea pointed out, we should collect these ideas and make them more precise soon: On Fri, Sep 23, 2005 at 04:28:46PM +0100, Boria Feigin wrote: > > In October, 10-16th to be more exact we will meet up to sprint in Paris > > and we will devote some time to plan the work in phase 2 in some more > > detail. In addition, note that this sprint is not meant to focus exclusively on discussion; it's a coding sprint, and we specifically welcome newcomers. If possible and interesting for you, feel invited :-) That's the best way to grasp the basics of PyPy and discuss. Also feel free to say hello in the #pypy IRC channel (irc.freenode.net) and discuss your interests. A bientot, Armin. From faassen at infrae.com Fri Sep 23 18:37:17 2005 From: faassen at infrae.com (Martijn Faassen) Date: Fri, 23 Sep 2005 18:37:17 +0200 Subject: [pypy-dev] Re: OSCON Europe Oct 17-20 Amsterdam In-Reply-To: References: <200509202017.j8KKHZrC027498@theraft.strakt.com> Message-ID: <43342F3D.4040909@infrae.com> Michael Sparks wrote: > Laura Creighton wrote: > > >>http://conferences.oreillynet.com/eurooscon/ >> >>Anybody going? > > > > > I am - I'm talking about our toys there. (Though we've got quite a few new > toys since Python UK & Europython :) I'm going too, going to talk about lxml. Not that I have much more to do with the PyPy project than the occasional cheerleading/sniping. :) You, me, Alex and Joel Burton are as far as I know the only speakers on the Python track. I saw Joel at the plone conference and was joking that half of the OSCON Europe Python track was present in one room. :) Regards, Martijn From boria at fbg.uklinux.net Fri Sep 23 19:43:46 2005 From: boria at fbg.uklinux.net (Boria Feigin) Date: Fri, 23 Sep 2005 18:43:46 +0100 Subject: [pypy-dev] Project suggestions In-Reply-To: <20050923162916.GF15985@code1.codespeak.net> References: <20050923120202.GA7065@antelope> <20050923152846.GA7662@antelope> <20050923162916.GF15985@code1.codespeak.net> Message-ID: <20050923174346.GA3600@antelope> Hi, [List of exciting topics snipped :)] > In addition, note that this sprint is not meant to focus exclusively on > discussion; it's a coding sprint, and we specifically welcome newcomers. > If possible and interesting for you, feel invited :-) That's the best > way to grasp the basics of PyPy and discuss. Also feel free to say > hello in the #pypy IRC channel (irc.freenode.net) and discuss your > interests. Thanks :) The sprint sounds good but I'll need to check whether I can make it. I'll take a closer look at the docs, the code, and your list of topics over the weekend and get back on Monday with something more concrete. Regards, -- Boris From hpk at trillke.net Sat Sep 24 08:21:47 2005 From: hpk at trillke.net (holger krekel) Date: Sat, 24 Sep 2005 08:21:47 +0200 Subject: [pypy-dev] profiling translation process? Message-ID: <20050924062147.GL10613@solar.trillke.net> Hi folks, did anyone try to profile the translation process itself recently? If so what where the results? If not then i think it would be a good idea because it's obviously one of the main activities these days. cheers, holger From arigo at tunes.org Sat Sep 24 10:10:51 2005 From: arigo at tunes.org (Armin Rigo) Date: Sat, 24 Sep 2005 10:10:51 +0200 Subject: [pypy-dev] profiling translation process? In-Reply-To: <20050924062147.GL10613@solar.trillke.net> References: <20050924062147.GL10613@solar.trillke.net> Message-ID: <20050924081051.GA28783@code1.codespeak.net> Hi Holger, On Sat, Sep 24, 2005 at 08:21:47AM +0200, holger krekel wrote: > did anyone try to profile the translation process > itself recently? If so what where the results? I've been spending days on that, with the results that the standard library profile.py is very much broken -- it explodes in mysterious ways or just stop profiling after a few seconds. Michael and me have been polishing a new profiler posted on the CPython SF patch tracker, and I've got preliminary results on translate_pypy. I'll need to run it again now that a few last bugs in the profiler have been fixed. (Also, the profiler doesn't support dumping data to a file at the moment.) The place where translate_pypy spends the largest amount of time is objspace.flow.model.traverse(), which subsequently we again reduced and un-generalized. I guess we could at some point get rid of it completely and replace it with two methods on the graph object, to iterate over blocks or links. A bientot, Armin. From hpk at trillke.net Sat Sep 24 10:18:40 2005 From: hpk at trillke.net (holger krekel) Date: Sat, 24 Sep 2005 10:18:40 +0200 Subject: [pypy-dev] profiling translation process? In-Reply-To: <20050924081051.GA28783@code1.codespeak.net> References: <20050924062147.GL10613@solar.trillke.net> <20050924081051.GA28783@code1.codespeak.net> Message-ID: <20050924081840.GD3312@solar.trillke.net> Hi Armin! On Sat, Sep 24, 2005 at 10:10 +0200, Armin Rigo wrote: > On Sat, Sep 24, 2005 at 08:21:47AM +0200, holger krekel wrote: > > did anyone try to profile the translation process > > itself recently? If so what where the results? > > I've been spending days on that, with the results that the standard > library profile.py is very much broken -- it explodes in mysterious > ways or just stop profiling after a few seconds. > > Michael and me have been polishing a new profiler posted on the CPython > SF patch tracker, and I've got preliminary results on translate_pypy. > I'll need to run it again now that a few last bugs in the profiler have > been fixed. (Also, the profiler doesn't support dumping data to a file > at the moment.) Yes, i saw your works, good stuff. Sidenote: i guess it's sometimes overlooked that PyPy developers have sometimes been quite involved with CPython improvements. > The place where translate_pypy spends the largest amount of time is > objspace.flow.model.traverse(), which subsequently we again reduced and > un-generalized. I guess we could at some point get rid of it completely > and replace it with two methods on the graph object, to iterate over > blocks or links. Yip, the generality (i introduced IIRC at that time) is not really needed. Refactoring to use specific methods on the graph/node object sounds worthwhile. cheers, holger From mwh at python.net Sat Sep 24 11:05:11 2005 From: mwh at python.net (Michael Hudson) Date: Sat, 24 Sep 2005 10:05:11 +0100 Subject: [pypy-dev] Re: profiling translation process? References: <20050924062147.GL10613@solar.trillke.net> <20050924081051.GA28783@code1.codespeak.net> <20050924081840.GD3312@solar.trillke.net> Message-ID: <2mbr2ij0oo.fsf@starship.python.net> holger krekel writes: > Hi Armin! > > On Sat, Sep 24, 2005 at 10:10 +0200, Armin Rigo wrote: >> Michael and me have been polishing a new profiler posted on the CPython >> SF patch tracker, and I've got preliminary results on translate_pypy. >> I'll need to run it again now that a few last bugs in the profiler have >> been fixed. (Also, the profiler doesn't support dumping data to a file >> at the moment.) > > Yes, i saw your works, good stuff. Sidenote: i guess it's sometimes > overlooked that PyPy developers have sometimes been quite involved > with CPython improvements. Well, I don't know who you might be thinking of as overlooking it; I'm certainly aware that quite a few bugs in CPython (mostly in obscure corner cases, it has to be said) have been found and fixed as a result of PyPy. I don't know if mentioning this sort of thing will earn us brownie point with the EU (it certainly seems a /fairly/ expensive way of getting a code review :). Cheers, mwh -- It's actually a corruption of "starling". They used to be carried. Since they weighed a full pound (hence the name), they had to be carried by two starlings in tandem, with a line between them. -- Alan J Rosenthal explains "Pounds Sterling" on asr From mwh at python.net Sat Sep 24 11:26:47 2005 From: mwh at python.net (Michael Hudson) Date: Sat, 24 Sep 2005 10:26:47 +0100 Subject: [pypy-dev] Re: profiling translation process? References: <20050924062147.GL10613@solar.trillke.net> <20050924081051.GA28783@code1.codespeak.net> Message-ID: <2m7jd6izoo.fsf@starship.python.net> Armin Rigo writes: > (Also, the profiler doesn't support dumping data to a file at the > moment.) Does now, see/please review the freeze() method I just checked in to svn. Cheers, mwh -- ... but I guess there are some things that are so gross you just have to forget, or it'll destroy something within you. perl is the first such thing I have known. -- Erik Naggum, comp.lang.lisp From hpk at trillke.net Sat Sep 24 11:34:37 2005 From: hpk at trillke.net (holger krekel) Date: Sat, 24 Sep 2005 11:34:37 +0200 Subject: [pypy-dev] Re: profiling translation process? In-Reply-To: <2mbr2ij0oo.fsf@starship.python.net> References: <20050924062147.GL10613@solar.trillke.net> <20050924081051.GA28783@code1.codespeak.net> <20050924081840.GD3312@solar.trillke.net> <2mbr2ij0oo.fsf@starship.python.net> Message-ID: <20050924093437.GI3312@solar.trillke.net> Hi Michael, On Sat, Sep 24, 2005 at 10:05 +0100, Michael Hudson wrote: > holger krekel writes: > > On Sat, Sep 24, 2005 at 10:10 +0200, Armin Rigo wrote: > >> Michael and me have been polishing a new profiler posted on the CPython > >> SF patch tracker, and I've got preliminary results on translate_pypy. > >> I'll need to run it again now that a few last bugs in the profiler have > >> been fixed. (Also, the profiler doesn't support dumping data to a file > >> at the moment.) > > > > Yes, i saw your works, good stuff. Sidenote: i guess it's sometimes > > overlooked that PyPy developers have sometimes been quite involved > > with CPython improvements. > > Well, I don't know who you might be thinking of as overlooking it; I'm > certainly aware that quite a few bugs in CPython (mostly in obscure > corner cases, it has to be said) have been found and fixed as a result > of PyPy. I am not thinking of you but the general public (whoever that is :-) > I don't know if mentioning this sort of thing will earn us brownie > point with the EU (it certainly seems a /fairly/ expensive way of > getting a code review :). I guess it wouldn't hurt if the EU gets to know that we are actively interacting with other (python) communities. It's not that we are claiming that PyPy is a project to fix bugs in CPython which would indeed be a rather expensive code review :-) cheers, holger From aleaxit at yahoo.com Mon Sep 26 03:42:21 2005 From: aleaxit at yahoo.com (Alex Martelli) Date: Sun, 25 Sep 2005 18:42:21 -0700 (PDT) Subject: [pypy-dev] a faster Python not a primary goal of PyPy? In-Reply-To: <20050923152319.GA15985@code1.codespeak.net> Message-ID: <20050926014221.23477.qmail@web40527.mail.yahoo.com> --- Armin Rigo wrote: ... > If anyone shows up with a Javascript parser, I'm sure we could > together > hack (and translate (and later have a JIT from)) a basic Javascript > interpreter within a few weeks :-) Actually, I think the currently useful thing might be a way to compile (any decent language, Python for choice) INTO Javascript (with reasonable efficiency), so one could write AJAX pages without actually having to code in Javascript...;-) Alex From hpk at trillke.net Mon Sep 26 08:56:24 2005 From: hpk at trillke.net (holger krekel) Date: Mon, 26 Sep 2005 08:56:24 +0200 Subject: [pypy-dev] a faster Python not a primary goal of PyPy? In-Reply-To: <20050926014221.23477.qmail@web40527.mail.yahoo.com> References: <20050923152319.GA15985@code1.codespeak.net> <20050926014221.23477.qmail@web40527.mail.yahoo.com> Message-ID: <20050926065623.GA3312@solar.trillke.net> Hi Alex! On Sun, Sep 25, 2005 at 18:42 -0700, Alex Martelli wrote: > --- Armin Rigo wrote: > ... > > If anyone shows up with a Javascript parser, I'm sure we could > > together > > hack (and translate (and later have a JIT from)) a basic Javascript > > interpreter within a few weeks :-) > > Actually, I think the currently useful thing might be a way to > compile (any decent language, Python for choice) INTO Javascript > (with reasonable efficiency), so one could write AJAX pages without > actually having to code in Javascript...;-) that's a second possibility but there are two crucial perequesites regarding current PyPy: - a translation approach that focuses more on high-level languages (instead of the current RTyper "C" one) - a browser/DOM simulation layer in Python to allow rapid prototyping. I believe the idea with a PyPy/JS Interpreter is more interesting, at least from the PyPy perspective. Btw, being faster than current JS interpreters is probably easier than being faster than CPython :-) Such a JS-interpreter implementation could be integrated with browsers allowing JS and Python directly in the browser. That being said, it might actually be worthwhile to compile RPython to Javascript which should be reasonably fast for small programs. One possibility is to just create flowgraphs from a python program and then transform the flowgraph to javascript source code (without much annotation). I guess only real experiments would shed more light. cheers, holger From aleaxit at yahoo.com Mon Sep 26 16:49:50 2005 From: aleaxit at yahoo.com (Alex Martelli) Date: Mon, 26 Sep 2005 07:49:50 -0700 (PDT) Subject: [pypy-dev] a faster Python not a primary goal of PyPy? In-Reply-To: <20050926065623.GA3312@solar.trillke.net> Message-ID: <20050926144950.82708.qmail@web40513.mail.yahoo.com> --- holger krekel wrote: > Hi Alex! > > On Sun, Sep 25, 2005 at 18:42 -0700, Alex Martelli wrote: > > --- Armin Rigo wrote: > > ... > > > If anyone shows up with a Javascript parser, I'm sure we could > > > together > > > hack (and translate (and later have a JIT from)) a basic > Javascript > > > interpreter within a few weeks :-) > > > > Actually, I think the currently useful thing might be a way to > > compile (any decent language, Python for choice) INTO Javascript > > (with reasonable efficiency), so one could write AJAX pages > without > > actually having to code in Javascript...;-) > > that's a second possibility but there are two crucial > perequesites regarding current PyPy: > > - a translation approach that focuses more on high-level > languages (instead of the current RTyper "C" one) > > - a browser/DOM simulation layer in Python to allow > rapid prototyping. Yep, the former would surely require lots of work (though maybe not the latter). > > I believe the idea with a PyPy/JS Interpreter is more > interesting, at least from the PyPy perspective. Btw, Possibly, but I advisedly used the adjective "useful" rather than saying "interesting";-) JS's only advantage is one of *deployment* -- browsers have JS interpreters, so you can write AJAX code and dramatically improve the user experience. "Sweating blood" is a charitable description of what it takes, but several examples prove it can work in RL. I just find myself daydreaming that I could do the coding in Python (or almost any other VHLL) rather than JS, that's all;-) > being faster than current JS interpreters is probably > easier than being faster than CPython :-) Yes, good point. > Such a JS-interpreter implementation could be > integrated with browsers allowing JS and Python > directly in the browser. Any Python implementation might be integrated with browsers, but it just seems to take a long time happening (even for Firefox, which should allow it most easily)...;-) > > That being said, it might actually be worthwhile to compile > RPython to Javascript which should be reasonably fast > for small programs. One possibility is to just create > flowgraphs from a python program and then transform the > flowgraph to javascript source code (without much > annotation). True, annotation wouldn't help much here (translating among VHLLs). Alex From bob at redivi.com Mon Sep 26 18:34:53 2005 From: bob at redivi.com (Bob Ippolito) Date: Mon, 26 Sep 2005 12:34:53 -0400 Subject: [pypy-dev] a faster Python not a primary goal of PyPy? In-Reply-To: <20050926144950.82708.qmail@web40513.mail.yahoo.com> References: <20050926144950.82708.qmail@web40513.mail.yahoo.com> Message-ID: <77BDADDA-CD80-41C8-9A5F-C1735E590E5D@redivi.com> On Sep 26, 2005, at 10:49 AM, Alex Martelli wrote: > --- holger krekel wrote: > >> That being said, it might actually be worthwhile to compile >> RPython to Javascript which should be reasonably fast >> for small programs. One possibility is to just create >> flowgraphs from a python program and then transform the >> flowgraph to javascript source code (without much >> annotation). >> > > True, annotation wouldn't help much here (translating among VHLLs). Using MochiKit at a target on top of JS would make it even easier, since comparators, iterators, and repr are already there with Python semantics (more or less). You'll probably have to throw out multiple inheritance, anything that depends on id() or hash(), and some other features JS can't really do. -bob From boria at fbg.uklinux.net Mon Sep 26 18:42:32 2005 From: boria at fbg.uklinux.net (Boria Feigin) Date: Mon, 26 Sep 2005 17:42:32 +0100 Subject: [pypy-dev] Project suggestions In-Reply-To: <20050923174346.GA3600@antelope> References: <20050923120202.GA7065@antelope> <20050923152846.GA7662@antelope> <20050923162916.GF15985@code1.codespeak.net> <20050923174346.GA3600@antelope> Message-ID: <20050926164232.GA3466@antelope.home> > > In addition, note that this sprint is not meant to focus exclusively on > > discussion; it's a coding sprint, and we specifically welcome newcomers. > > If possible and interesting for you, feel invited :-) That's the best > > way to grasp the basics of PyPy and discuss. Also feel free to say > > hello in the #pypy IRC channel (irc.freenode.net) and discuss your > > interests. > > Thanks :) The sprint sounds good but I'll need to check whether I can > make it. I'll take a closer look at the docs, the code, and your list of > topics over the weekend and get back on Monday with something more > concrete. I had a browse at the docs and code, but not enough to identify a definite area of interest yet. However, remembered a paper I saw recently: ``A Survey of Adaptive Optimization in Virtual Machines'' by M. Arnold, S. J. Fink, D. Grove, M. Hind, and P. F. Sweeney Proceedings of the IEEE, Vol. 93, No. 2, February 2005 (http://scholar.google.com finds the paper itself and an ealier preprint) A nice survey by the IBM Jikes people. Plenty of further references: looks like it might be a good place to start. Though, I don't know enough about PyPy yet to see exactly how (or if) this would fit in. -- Boris From len-l at telus.net Mon Sep 26 21:12:05 2005 From: len-l at telus.net (Lenard Lindstrom) Date: Mon, 26 Sep 2005 12:12:05 -0700 Subject: [pypy-dev] a faster Python not a primary goal of PyPy? In-Reply-To: <20050926065623.GA3312@solar.trillke.net> References: <20050926014221.23477.qmail@web40527.mail.yahoo.com> Message-ID: <4337E595.7004.984F23@localhost> On 26 Sep 2005 at 8:56, holger krekel wrote: > > I believe the idea with a PyPy/JS Interpreter is more > interesting, at least from the PyPy perspective. Btw, > being faster than current JS interpreters is probably > easier than being faster than CPython :-) > Such a JS-interpreter implementation could be > integrated with browsers allowing JS and Python > directly in the browser. > > That being said, it might actually be worthwhile to compile > RPython to Javascript which should be reasonably fast > for small programs. One possibility is to just create > flowgraphs from a python program and then transform the > flowgraph to javascript source code (without much > annotation). > > I guess only real experiments would shed more light. > Hello everyone. Python and prototypes is one area where I have some personal experience. How well RPython will translate to Javascript depends on how fully RPython supports new-style classes. Javascript objects have single inheritance only. So problems arise immediately for multiple inheritance. An object space is needed to implement new-style classes on top of Javascript prototypes. All python arithmetic operations, attribute accesses and method calls become Javascript function calls. This adds a level of indirection to an already slow interpreter. Finally Javascript has no 'goto' statement. So a switch statement within a loop is needed to handle the spaghetti code produced by tranforming a flowgraph. How much different is this from an interpreter loop? So a Python program compiled to Javascript is essentially a Python interpreter that runs in Javascript. How worthwhile is this? Of course if RPython only supports single inheritance and improvements to flowgraph transformation will recover Python's statement structure then a closer translation may be possible. Lenard Lindstrom From aurelien.campeas at free.fr Mon Sep 26 23:39:32 2005 From: aurelien.campeas at free.fr (=?ISO-8859-1?Q?Aur=E9lien_Camp=E9as?=) Date: Mon, 26 Sep 2005 23:39:32 +0200 Subject: [pypy-dev] Project suggestions In-Reply-To: <20050923162916.GF15985@code1.codespeak.net> References: <20050923120202.GA7065@antelope> <20050923152846.GA7662@antelope> <20050923162916.GF15985@code1.codespeak.net> Message-ID: <43386A94.2090108@free.fr> Hello, pypy people, Armin Rigo a ?crit : > Hi Boria, > > We are indeed starting to think about more focused research areas in > PyPy. For example, along these lines, we will need more work on > compiler optimizations and generation of code for various architectures, > both either statically or just-in-time. More along the lines of > interpreters and virtual machines, we could start investigating new > aspects that would be useful to code into the interpreter or the > translation process: continuations, persistence (either dumping a > whole-process image or something more fine-grained), security (running > code in a sandbox), and much more, all of which is hinted at in some > documentation on Codespeak. Finally, there is also the idea of > supporting other dynamic languages than Python by writing an interpreter > for them in PyPy. I have been looking into pypy for a few days, and trying to understand how to make the Lisp backend work. I now understand that due to pypy heavily layered nature, it is possible to use a language like lisp, as a target, at different levels. For instance, bytecode interpreter (by providing a proper set of functions to handle the bytecodes ?), or compiler (by translating the function graph into low-level lisp). Right now, I am sticking with the second way, as it was started in gencl.py (btw, one question that is not clear to me is about the function graph : does it contain python or rpython opcodes ?) I am also thinking about another possibility (wrt your last sentence) : translating directly the parse tree into high-level lisp. Maybe this idea is nonsense. In fact, the distance from de-sugared python to the opcode is unknown to me (even though I suspect there is no 1:1 mapping between the two). Whatever, that could be the start of a strategy to translate python towards other high-level languages (ruby, or js as in the other thread) without paying the full price of opcode interpretation in the target (that is : parts which are semantically similar could be easily translated, others would be -costily- emulated). Thoughts ? (aka : does it look like gibberish ?) Thanks, Aur?lien. From mwh at python.net Tue Sep 27 10:19:30 2005 From: mwh at python.net (Michael Hudson) Date: Tue, 27 Sep 2005 09:19:30 +0100 Subject: [pypy-dev] Re: Project suggestions References: <20050923120202.GA7065@antelope> <20050923152846.GA7662@antelope> <20050923162916.GF15985@code1.codespeak.net> <43386A94.2090108@free.fr> Message-ID: <2mpsqukjn1.fsf@starship.python.net> Aur?lien Camp?as writes: > Hello, pypy people, > > Armin Rigo a ?crit : >> Hi Boria, >> We are indeed starting to think about more focused research areas in >> PyPy. For example, along these lines, we will need more work on >> compiler optimizations and generation of code for various architectures, >> both either statically or just-in-time. More along the lines of >> interpreters and virtual machines, we could start investigating new >> aspects that would be useful to code into the interpreter or the >> translation process: continuations, persistence (either dumping a >> whole-process image or something more fine-grained), security (running >> code in a sandbox), and much more, all of which is hinted at in some >> documentation on Codespeak. Finally, there is also the idea of >> supporting other dynamic languages than Python by writing an interpreter >> for them in PyPy. > > I have been looking into pypy for a few days, and trying to understand > how to make the Lisp backend work. Cool! > I now understand that due to pypy heavily layered nature, it is > possible to use a language like lisp, as a target, at different > levels. For instance, bytecode interpreter (by providing a proper set > of functions to handle the bytecodes ?), or compiler (by translating > the function graph into low-level lisp). Right now, I am sticking with > the second way, as it was started in gencl.py > > (btw, one question that is not clear to me is about the function graph > : does it contain python or rpython opcodes ?) Um, neither. It roughly goes through three potential stages: unannotated SpaceOperations, annotated SpaceOperations and finally LowLevelOperations. I think a lisp translator would probably want to work with annotated SpaceOperations; LowLevelOperations would be too low level (probably :). > I am also thinking about another possibility (wrt your last sentence) > : translating directly the parse tree into high-level lisp. Maybe this > idea is nonsense. It's not necessarily nonsense, but it's not really The PyPy Way. > In fact, the distance from de-sugared python to the opcode is > unknown to me (even though I suspect there is no 1:1 mapping between > the two). Whatever, that could be the start of a strategy to > translate python towards other high-level languages (ruby, or js as > in the other thread) without paying the full price of opcode > interpretation in the target (that is : parts which are semantically > similar could be easily translated, others would be -costily- > emulated). Thing is, I don't know how feasible this is. It's pretty hard, without some kind of type inference, to translate, say this Python: a + b into anything significantly more efficient than this Common Lisp: (py:add a b) And making type inference possible is what RPython is all about. You could make #'py:add a generic function and see if a given CLOS implementation is fast enough to give a useful speed (but I think the coercion rules would probably drive you insane first). Cheers, mwh -- ARTHUR: Yes. It was on display in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying "Beware of the Leopard". -- The Hitch-Hikers Guide to the Galaxy, Episode 1 From gaiacrtn at free.fr Tue Sep 27 10:30:02 2005 From: gaiacrtn at free.fr (Baptiste Lepilleur) Date: Tue, 27 Sep 2005 10:30:02 +0200 Subject: [pypy-dev] Patch to remove compilation warning with vc++ Message-ID: <023b01c5c33d$a8428400$0300a8c0@anabelle> The attached patch fix a few warning when compiling translated C with VC++: - direct.h not included (required for mkdir...) - mkdir does not takes a mode parameter - redefined macros There is also a warning which I haven't fixed: long LL_os_lseek(long fd, long pos, long how) { #if defined(MS_WIN64) || defined(MS_WINDOWS) PY_LONG_LONG res; // .... return res; } On WIN32, __int64 _lseeki64() returns a 64 bits integer which doesn't fit in a 'long' (which is only 32bits). I'm not sure how this should be resolved thought but this will likely cause trouble when working with large file. Baptiste. From Michaels at rd.bbc.co.uk Tue Sep 27 12:41:48 2005 From: Michaels at rd.bbc.co.uk (Michael Sparks) Date: Tue, 27 Sep 2005 11:41:48 +0100 Subject: [pypy-dev] Project suggestions In-Reply-To: <20050923120202.GA7065@antelope> References: <20050923120202.GA7065@antelope> Message-ID: <200509271141.48214.Michaels@rd.bbc.co.uk> On Friday 23 Sep 2005 13:02, Boria Feigin wrote: ... > I am an undergraduate student (at a top-5 CS university in the UK) > looking for a fun final-year project to sink my teeth into. As I understand pypy it can theoretically take a number of alternate front ends as well as back ends. I don't know how much work has gone into that, but a whimsical idea that sprang to mind just now as an alternate project idea (that I'm not convinced should ever be done >:-) is this: Last summer, for fun, I wrote a generic language parser that can handle almost anything you throw at it. The basic underlying idea was "what if in lisp you removed all the brackets and used a python-like syntax instead". (Or depending on the time of day "What if we removed all the keywords from python, could we still parse something that looked close to python>") The idea was that you would use : stuff where you would use nested brackets. I did leave in a small amount of infix support though, and did think about things like: I only worked on a parsing/AST building, but it might be an "interesting" alternate front end. Unlike python I found that in order to support control flows like if...elif...elif..else, and try...except...finally, and similar that you needed an end tag (could be anything though) in order to simplify the parser. (Hints to the lexer would have been an alternate approach, but would've required 2 passes in the way I was thinking) In case that doesn't make any sense, if result is None: print "Parse failed" else: print "Success" end is a function call to an "if" function that among it's arguments takes 2 anonymous code blocks with an "else" tag inbetween and followed by an "end" tag. Similarly: pen down repeat 4: forward 10 rotate 90 end is two function calls - one to pen, and one to repeat. Repeat takes 3 arguments here - 4, the code block and the "end" tag. Anyway, the code is here: cerenity.org/SWP/ - that also includes a pdf of the lightning talk I gave at ACCU (though not the slightly nicer version I gave at europython I notice... Hmm... ) Slide 8 shows how nuts this can get with mixing of an L-System description language with an SML-esque syntax for a structural stack, with a logo type language, and started off with a python-esque bootstrap. At europython I also had an SQL-like statement in there for added silliness. (The point was that the example parsed cleanly, not whether it meant anything) The entire parser was written test first, with the tests being sample programs - which you can find here: * http://cerenity.org/SWP/progs/ The parser itself is quite short, the grammer is LL(1) compliant and consists of 13 productions. Anyway, as an alternate front end this could be a mad (but fun) little project. I'm not convinced it would actually be a GOOD idea to do though :) Regards, Michael. -- Michael Sparks, Senior R&D Engineer, Digital Media Group Michael.Sparks at rd.bbc.co.uk, http://kamaelia.sourceforge.net/ British Broadcasting Corporation, Research and Development Kingswood Warren, Surrey KT20 6NP This e-mail may contain personal views which are not the views of the BBC. From aurelien.campeas at free.fr Tue Sep 27 14:17:16 2005 From: aurelien.campeas at free.fr (=?ISO-8859-1?Q?Aur=E9lien_Camp=E9as?=) Date: Tue, 27 Sep 2005 14:17:16 +0200 Subject: [pypy-dev] Re: Project suggestions In-Reply-To: <2mpsqukjn1.fsf@starship.python.net> References: <20050923120202.GA7065@antelope> <20050923152846.GA7662@antelope> <20050923162916.GF15985@code1.codespeak.net> <43386A94.2090108@free.fr> <2mpsqukjn1.fsf@starship.python.net> Message-ID: <4339384C.6010906@free.fr> Michael Hudson a ?crit : > Aur?lien Camp?as writes: > > >>Hello, pypy people, >> >>Armin Rigo a ?crit : >> >>>Hi Boria, >>>We are indeed starting to think about more focused research areas in >>>PyPy. For example, along these lines, we will need more work on >>>compiler optimizations and generation of code for various architectures, >>>both either statically or just-in-time. More along the lines of >>>interpreters and virtual machines, we could start investigating new >>>aspects that would be useful to code into the interpreter or the >>>translation process: continuations, persistence (either dumping a >>>whole-process image or something more fine-grained), security (running >>>code in a sandbox), and much more, all of which is hinted at in some >>>documentation on Codespeak. Finally, there is also the idea of >>>supporting other dynamic languages than Python by writing an interpreter >>>for them in PyPy. >> >>I have been looking into pypy for a few days, and trying to understand >>how to make the Lisp backend work. > > > Cool! > > >>I now understand that due to pypy heavily layered nature, it is >>possible to use a language like lisp, as a target, at different >>levels. For instance, bytecode interpreter (by providing a proper set >>of functions to handle the bytecodes ?), or compiler (by translating >>the function graph into low-level lisp). Right now, I am sticking with >>the second way, as it was started in gencl.py >> >>(btw, one question that is not clear to me is about the function graph >>: does it contain python or rpython opcodes ?) > > > Um, neither. It roughly goes through three potential stages: > unannotated SpaceOperations, annotated SpaceOperations and finally > LowLevelOperations. I think a lisp translator would probably want to > work with annotated SpaceOperations; LowLevelOperations would be too > low level (probably :). > Ok, thanks :) But then, I still don't get completely the relation between python, rpython, and the three potential stages of the function graph (probably I have to reread again all the doc on codespeak) > >>I am also thinking about another possibility (wrt your last sentence) >>: translating directly the parse tree into high-level lisp. Maybe this >>idea is nonsense. > > > It's not necessarily nonsense, but it's not really The PyPy Way. ok > > >>In fact, the distance from de-sugared python to the opcode is >>unknown to me (even though I suspect there is no 1:1 mapping between >>the two). Whatever, that could be the start of a strategy to >>translate python towards other high-level languages (ruby, or js as >>in the other thread) without paying the full price of opcode >>interpretation in the target (that is : parts which are semantically >>similar could be easily translated, others would be -costily- >>emulated). > > > Thing is, I don't know how feasible this is. It's pretty hard, > without some kind of type inference, to translate, say this Python: > > a + b > > into anything significantly more efficient than this Common Lisp: > > (py:add a b) The mere fact that it will be compiled will make it more efficient I guess. I mean, not on CLISP, but with a real lisp compiler. > > And making type inference possible is what RPython is all about. Sure, but then, it is a restricted subset of Python, and I like python completely unrestricted ;) > > You could make #'py:add a generic function and see if a given CLOS > implementation is fast enough to give a useful speed (but I think the > coercion rules would probably drive you insane first). In this case, CLOS would add overhead. In fact the python add operator (and some arith. ops) seems close enough to the lisp one that one can be tempted to translate in the lisp equivalent with only minor adaptation (like the printer appending an "L" on bignums, etc.). I am not sure I would use CLOS at all, in fact (at least for a first attempt at producing a lisp backend). BTW, what's this "insanity with coercion rules" that you mention - can you expand a little on this ? > > Cheers, > mwh > Many thanks, Aur?lien. From bert at impara.de Tue Sep 27 14:54:05 2005 From: bert at impara.de (Bert Freudenberg) Date: Tue, 27 Sep 2005 14:54:05 +0200 Subject: [pypy-dev] Re: Project suggestions In-Reply-To: <4339384C.6010906@free.fr> References: <20050923120202.GA7065@antelope> <20050923152846.GA7662@antelope> <20050923162916.GF15985@code1.codespeak.net> <43386A94.2090108@free.fr> <2mpsqukjn1.fsf@starship.python.net> <4339384C.6010906@free.fr> Message-ID: Am 27.09.2005 um 14:17 schrieb Aur?lien Camp?as: > BTW, what's this "insanity with coercion rules" that you mention - > can you expand a little on this ? See http://docs.python.org/ref/coercion-rules.html Excerpt: =================== - Below, __op__() and __rop__() are used to signify the generic method names corresponding to an operator; __iop__ is used for the corresponding in-place operator. For example, for the operator `+', __add__() and __radd__() are used for the left and right variant of the binary operator, and __iadd__ for the in-place variant. - For objects x and y, first x.__op__(y) is tried. If this is not implemented or returns NotImplemented, y.__rop__(x) is tried. If this is also not implemented or returns NotImplemented, a TypeError exception is raised. But see the following exception: - Exception to the previous item: if the left operand is an instance of a built-in type or a new-style class, and the right operand is an instance of a proper subclass of that type or class, the right operand's __rop__() method is tried before the left operand's __op__ () method. This is done so that a subclass can completely override binary operators. Otherwise, the left operand's __op__ method would always accept the right operand: when an instance of a given class is expected, an instance of a subclass of that class is always acceptable. - When either operand type defines a coercion, this coercion is called before that type's __op__() or __rop__() method is called, but no sooner. If the coercion returns an object of a different type for the operand whose coercion is invoked, part of the process is redone using the new object. =================== However, this is just an attempt at documenting the current behavior, AFAIK there is no definite reference except for what CPython implements. (I'm working on running [a subset of] Python in Smalltalk) - Bert - From pedronis at strakt.com Tue Sep 27 14:57:23 2005 From: pedronis at strakt.com (Samuele Pedroni) Date: Tue, 27 Sep 2005 14:57:23 +0200 Subject: [pypy-dev] Re: Project suggestions In-Reply-To: <4339384C.6010906@free.fr> References: <20050923120202.GA7065@antelope> <20050923152846.GA7662@antelope> <20050923162916.GF15985@code1.codespeak.net> <43386A94.2090108@free.fr> <2mpsqukjn1.fsf@starship.python.net> <4339384C.6010906@free.fr> Message-ID: <433941B3.60801@strakt.com> Aur?lien Camp?as wrote: > Michael Hudson a ?crit : > >> Aur?lien Camp?as writes: >> >> >>> Hello, pypy people, >>> >>> Armin Rigo a ?crit : >>> >>>> Hi Boria, >>>> We are indeed starting to think about more focused research areas in >>>> PyPy. For example, along these lines, we will need more work on >>>> compiler optimizations and generation of code for various >>>> architectures, >>>> both either statically or just-in-time. More along the lines of >>>> interpreters and virtual machines, we could start investigating new >>>> aspects that would be useful to code into the interpreter or the >>>> translation process: continuations, persistence (either dumping a >>>> whole-process image or something more fine-grained), security (running >>>> code in a sandbox), and much more, all of which is hinted at in some >>>> documentation on Codespeak. Finally, there is also the idea of >>>> supporting other dynamic languages than Python by writing an >>>> interpreter >>>> for them in PyPy. >>> >>> >>> I have been looking into pypy for a few days, and trying to understand >>> how to make the Lisp backend work. >> >> >> >> Cool! >> >> >>> I now understand that due to pypy heavily layered nature, it is >>> possible to use a language like lisp, as a target, at different >>> levels. For instance, bytecode interpreter (by providing a proper set >>> of functions to handle the bytecodes ?), or compiler (by translating >>> the function graph into low-level lisp). Right now, I am sticking with >>> the second way, as it was started in gencl.py >>> >>> (btw, one question that is not clear to me is about the function graph >>> : does it contain python or rpython opcodes ?) >> >> >> >> Um, neither. It roughly goes through three potential stages: >> unannotated SpaceOperations, annotated SpaceOperations and finally >> LowLevelOperations. I think a lisp translator would probably want to >> work with annotated SpaceOperations; LowLevelOperations would be too >> low level (probably :). >> > > Ok, thanks :) > But then, I still don't get completely the relation between python, > rpython, and the three potential stages of the function graph > (probably I have to reread again all the doc on codespeak) > >> >>> I am also thinking about another possibility (wrt your last sentence) >>> : translating directly the parse tree into high-level lisp. Maybe this >>> idea is nonsense. >> >> >> >> It's not necessarily nonsense, but it's not really The PyPy Way. > > > ok > >> >> >>> In fact, the distance from de-sugared python to the opcode is >>> unknown to me (even though I suspect there is no 1:1 mapping between >>> the two). Whatever, that could be the start of a strategy to >>> translate python towards other high-level languages (ruby, or js as >>> in the other thread) without paying the full price of opcode >>> interpretation in the target (that is : parts which are semantically >>> similar could be easily translated, others would be -costily- >>> emulated). >> >> >> >> Thing is, I don't know how feasible this is. It's pretty hard, >> without some kind of type inference, to translate, say this Python: >> >> a + b >> >> into anything significantly more efficient than this Common Lisp: >> >> (py:add a b) > > > The mere fact that it will be compiled will make it more efficient I > guess. I mean, not on CLISP, but with a real lisp compiler. > >> >> And making type inference possible is what RPython is all about. > > > Sure, but then, it is a restricted subset of Python, and I like python > completely unrestricted ;) > >> >> You could make #'py:add a generic function and see if a given CLOS >> implementation is fast enough to give a useful speed (but I think the >> coercion rules would probably drive you insane first). > > > In this case, CLOS would add overhead. In fact the python add operator > (and some arith. ops) seems close enough to the lisp one that one can > be tempted to translate in the lisp equivalent with only minor > adaptation (like the printer appending an "L" on bignums, etc.). > but a and/or b could be instances of a user defined class. > I am not sure I would use CLOS at all, in fact (at least for a first > attempt at producing a lisp backend). BTW, what's this "insanity with > coercion rules" that you mention - can you expand a little on this ? > >> >> Cheers, >> mwh >> > > Many thanks, > Aur?lien. > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev From mwh at python.net Tue Sep 27 15:06:19 2005 From: mwh at python.net (Michael Hudson) Date: Tue, 27 Sep 2005 14:06:19 +0100 Subject: [pypy-dev] Re: Project suggestions References: <20050923120202.GA7065@antelope> <20050923152846.GA7662@antelope> <20050923162916.GF15985@code1.codespeak.net> <43386A94.2090108@free.fr> <2mpsqukjn1.fsf@starship.python.net> <4339384C.6010906@free.fr> Message-ID: <2md5muk6d0.fsf@starship.python.net> Aur?lien Camp?as writes: > Michael Hudson a ?crit : >> Aur?lien Camp?as writes: >> >>>Hello, pypy people, >>> >>>Armin Rigo a ?crit : >>> >>>>Hi Boria, >>>>We are indeed starting to think about more focused research areas in >>>>PyPy. For example, along these lines, we will need more work on >>>>compiler optimizations and generation of code for various architectures, >>>>both either statically or just-in-time. More along the lines of >>>>interpreters and virtual machines, we could start investigating new >>>>aspects that would be useful to code into the interpreter or the >>>>translation process: continuations, persistence (either dumping a >>>>whole-process image or something more fine-grained), security (running >>>>code in a sandbox), and much more, all of which is hinted at in some >>>>documentation on Codespeak. Finally, there is also the idea of >>>>supporting other dynamic languages than Python by writing an interpreter >>>>for them in PyPy. >>> >>>I have been looking into pypy for a few days, and trying to understand >>>how to make the Lisp backend work. >> Cool! >> >>>I now understand that due to pypy heavily layered nature, it is >>>possible to use a language like lisp, as a target, at different >>>levels. For instance, bytecode interpreter (by providing a proper set >>>of functions to handle the bytecodes ?), or compiler (by translating >>>the function graph into low-level lisp). Right now, I am sticking with >>>the second way, as it was started in gencl.py >>> >>>(btw, one question that is not clear to me is about the function graph >>>: does it contain python or rpython opcodes ?) >> Um, neither. It roughly goes through three potential stages: >> unannotated SpaceOperations, annotated SpaceOperations and finally >> LowLevelOperations. I think a lisp translator would probably want to >> work with annotated SpaceOperations; LowLevelOperations would be too >> low level (probably :). >> > > Ok, thanks :) > But then, I still don't get completely the relation between python, > rpython, and the three potential stages of the function graph > (probably I have to reread again all the doc on codespeak) RPython is a subset of python with the main constraint being able to do some level of static type analysis (for full python, the amount of static ananlysis you can do is really very small, you can read Brett Cannon's thesis about this: http://www.ocf.berkeley.edu/~bac/thesis.pdf). So, when translating, the code that implements the interpreter (roughly interpreter/*.py, objspace/std/*.py) is imported and a flow graph built from it. This is then annotated (code in annotator/*.py), a fairly hairy process. This (for the C and LLVM backends, at least) is then turned into a graph containing low level operations (like "dereference this pointer"). Python is just the language all of the above happens to be implemented in (and also the interpreter/*.py code is involved in making a flow graph which includes itself, but this isn't that important -- just confusing :). >>>In fact, the distance from de-sugared python to the opcode is >>>unknown to me (even though I suspect there is no 1:1 mapping between >>>the two). Whatever, that could be the start of a strategy to >>>translate python towards other high-level languages (ruby, or js as >>>in the other thread) without paying the full price of opcode >>>interpretation in the target (that is : parts which are semantically >>>similar could be easily translated, others would be -costily- >>>emulated). >> Thing is, I don't know how feasible this is. It's pretty hard, >> without some kind of type inference, to translate, say this Python: >> a + b >> into anything significantly more efficient than this Common Lisp: >> (py:add a b) > > The mere fact that it will be compiled will make it more efficient I > guess. I mean, not on CLISP, but with a real lisp compiler. Not much. The "interpreter overhead" for Python is usually estimated at about 20%, though it obviously depends what code you're running to some extent. >> And making type inference possible is what RPython is all about. > > Sure, but then, it is a restricted subset of Python, and I like python > completely unrestricted ;) Well, so do we all, but then you can't have type inference. There is no simple answer to this. >> You could make #'py:add a generic function and see if a given CLOS >> implementation is fast enough to give a useful speed (but I think the >> coercion rules would probably drive you insane first). > > In this case, CLOS would add overhead. In fact the python add operator > (and some arith. ops) seems close enough to the lisp one that one can > be tempted to translate in the lisp equivalent with only minor > adaptation (like the printer appending an "L" on bignums, etc.). Er... no. #'cl:+ is not that much like python's + (e.g. the latter operates on strings and you can't overload the former). > I am not sure I would use CLOS at all, in fact (at least for a first > attempt at producing a lisp backend). Fair enough. > BTW, what's this "insanity with coercion rules" that you mention - > can you expand a little on this ? Think about things like 2L**135 < 1.0e40 or range(10)*3 or... mixed type operations are not that simple in Python. Cheers, mwh -- Considering that this thread is completely on-topic in the way only c.l.py threads can be, I think I can say that you should replace "Oblivion" with "Gravity", and increase your Radiohead quotient. -- Ben Wolfson, comp.lang.python From boria at fbg.uklinux.net Tue Sep 27 16:19:49 2005 From: boria at fbg.uklinux.net (Boria Feigin) Date: Tue, 27 Sep 2005 15:19:49 +0100 Subject: [pypy-dev] Project suggestions In-Reply-To: <200509271141.48214.Michaels@rd.bbc.co.uk> References: <20050923120202.GA7065@antelope> <200509271141.48214.Michaels@rd.bbc.co.uk> Message-ID: <20050927141949.GA3669@antelope> Hi, > As I understand pypy it can theoretically take a number of alternate front > ends as well as back ends. I don't know how much work has gone into that, That's my understanding as well -- from Armin Rigo's reply to my initial post. > but a whimsical idea that sprang to mind just now as an alternate project idea > (that I'm not convinced should ever be done >:-) is this: :) > Last summer, for fun, I wrote a generic language parser that can handle > almost anything you throw at it. The basic underlying idea was "what if in > lisp you removed all the brackets and used a python-like syntax instead". > (Or depending on the time of day "What if we removed all the keywords > from python, could we still parse something that looked close to python>") > > The idea was that you would use : stuff where you > would use nested brackets. I did leave in a small amount of infix support OK. It's an alternative notation for writing down the AST. [snipped] > Similarly: > > pen down > repeat 4: > forward 10 > rotate 90 > end > > is two function calls - one to pen, and one to repeat. Repeat takes 3 > arguments here - 4, the code block and the "end" tag. Aha, I see. Essentially, you should get Lisp's famed ability for domain-specific languages (themselves a big and hot topic). > Anyway, the code is here: cerenity.org/SWP/ - that also includes a pdf of the > lightning talk I gave at ACCU (though not the slightly nicer version I gave > at europython I notice... Hmm... ) Slide 8 shows how nuts this can get with > mixing of an L-System description language with an SML-esque syntax for a > structural stack, with a logo type language, and started off with a > python-esque bootstrap. At europython I also had an SQL-like statement in > there for added silliness. (The point was that the example parsed cleanly, This isn't silly at all -- just look at the C# 3.0 spec (http://msdn.microsoft.com/vcsharp/future/, the section on ``Query Expressions'' specifically) > not whether it meant anything) There's the rub. For example, one could write, if foo > 10: threshold = 20 end select: from titles where (id < threshold) end The question then becomes, is there a way to leverage PyPy (type inference) to _statically_ verify that `id' and `threshold' are of the same type? > Anyway, as an alternate front end this could be a mad (but fun) little > project. I'm not convinced it would actually be a GOOD idea to do > though :) In any case, thanks for the suggestion :) P.S. This may be off-topic, but you may also like to have a look at this http://www.martinfowler.com/articles/languageWorkbench.html piece by Martin Fowler. -- Boris From cfbolz at gmx.de Tue Sep 27 18:57:54 2005 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Tue, 27 Sep 2005 18:57:54 +0200 Subject: [pypy-dev] Patch to remove compilation warning with vc++ In-Reply-To: <023b01c5c33d$a8428400$0300a8c0@anabelle> References: <023b01c5c33d$a8428400$0300a8c0@anabelle> Message-ID: <43397A12.1090706@gmx.de> Hi! Baptiste Lepilleur wrote: > The attached patch fix a few warning when compiling translated C with VC++: > - direct.h not included (required for mkdir...) > - mkdir does not takes a mode parameter > - redefined macros [snip] huh? there does not seem to be an attachement... cheers, Carl Friedrich From gaiacrtn at free.fr Tue Sep 27 21:49:46 2005 From: gaiacrtn at free.fr (Baptiste Lepilleur) Date: Tue, 27 Sep 2005 21:49:46 +0200 Subject: [pypy-dev] Patch to remove compilation warning with vc++ [repost] Message-ID: <030401c5c39c$9fd220f0$0300a8c0@anabelle> The attached patch fix a few warning when compiling translated C with VC++: - direct.h not included (required for mkdir...) - mkdir does not takes a mode parameter - redefined macros There is also a warning which I haven't fixed: long LL_os_lseek(long fd, long pos, long how) { #if defined(MS_WIN64) || defined(MS_WINDOWS) PY_LONG_LONG res; // .... return res; } On WIN32, __int64 _lseeki64() returns a 64 bits integer which doesn't fit in a 'long' (which is only 32bits). I'm not sure how this should be resolved thought but this will likely cause trouble when working with large file. Baptiste. PS: the patch is attached... From hpk at trillke.net Tue Sep 27 21:51:52 2005 From: hpk at trillke.net (holger krekel) Date: Tue, 27 Sep 2005 21:51:52 +0200 Subject: [pypy-dev] Patch to remove compilation warning with vc++ In-Reply-To: <43397A12.1090706@gmx.de> References: <023b01c5c33d$a8428400$0300a8c0@anabelle> <43397A12.1090706@gmx.de> Message-ID: <20050927195152.GB10599@solar.trillke.net> On Tue, Sep 27, 2005 at 18:57 +0200, Carl Friedrich Bolz wrote: > Baptiste Lepilleur wrote: > >The attached patch fix a few warning when compiling translated C with VC++: > >- direct.h not included (required for mkdir...) > >- mkdir does not takes a mode parameter > >- redefined macros > [snip] > > huh? there does not seem to be an attachement... Actually i think that mailman (the mailing list manager) scraps the attachment. Baptiste, could you send the patch to me personally in the same way you sent it to the list? Then'll check what the problem is. cheers, holger From gaiacrtn at free.fr Tue Sep 27 21:48:19 2005 From: gaiacrtn at free.fr (Baptiste Lepilleur) Date: Tue, 27 Sep 2005 21:48:19 +0200 Subject: [pypy-dev] Patch to remove compilation warning with vc++ References: <023b01c5c33d$a8428400$0300a8c0@anabelle> <43397A12.1090706@gmx.de> Message-ID: <02ef01c5c39c$6a68e160$0300a8c0@anabelle> Carl Friedrich Bolz wrote: > Hi! > > Baptiste Lepilleur wrote: >> The attached patch fix a few warning when compiling translated C >> with VC++: - direct.h not included (required for mkdir...) >> - mkdir does not takes a mode parameter >> - redefined macros > [snip] > > huh? there does not seem to be an attachement... Somehow the attachement got lost (it was in my sent mail). I'll repost and see how it goes... > > cheers, > > Carl Friedrich From bea at netg.se Tue Sep 27 22:20:52 2005 From: bea at netg.se (Beatrice During) Date: Tue, 27 Sep 2005 22:20:52 +0200 (CEST) Subject: [pypy-dev] How agile is PyPy? Message-ID: Hi there I am working on documenting PyPy (the project) as a showcase of agile development. In order to be able to do this I would love to get some thoughts and reflections from you who are working on /following PyPy and the question is of course - how agile are we? Here is some short notes on success factors of agile approaches (from an eworkshop in 2002 - Cockburn, Beck etc, http://fc-md.umd.edu/projects/Agile/Summary/SummaryPF.htm): ------------------------------------------------------------------------- In general, the group agreed that the following are the success factors for agile development: ? Negotiation: If people are willing to keep talking and learning, you are, almost by definition, agile. ? Ability to continuously learn ? Responsibly responding to change. ? Environment where rapid communication is enabled. ? Team reasonably in charge of its own destiny (being allowed to succeed). ? People are valued over other factors ? The team needs to be more than just able, it needs to actually succeed. There needs to be a support network for each individual in the team to make a contribution however small. ------------------------------------------------------------------------ How does these factors apply to PyPy and this community? Are we doing other things that is not mentioned here that is working well for us? Things we could do better? Your responses will be part of documentation about PyPy/agile dev (sprints) that will be distributed to the EU, Universities, various companies and other Open Source projects who have stated a clear interest in the PyPy showcase..... If you feel that answering this email doesnt really fit the purpose of this mailinglist - please answer to me specifically. Thank you for your contribution! Cheers Bea Beatrice D?ring Change Maker Tel: 031- 7750940 J?rntorget 3 Mobil: 0734- 22 89 06 413 04 G?teborg E-post: bea at changemaker.nu www.changemaker.nu "Alla dessa m?sten och alldaglighet. Allt detta som binder v?r verklighet i bojor av skam och rep utav tv?ng. Alla dessa kedjor som binder v?r s?ng. Jag skall slita dem alla i sm?, sm? stycken och m?jligtvis av resterna g?ra mig smycken." - hemlig From tjreedy at udel.edu Tue Sep 27 22:49:53 2005 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 27 Sep 2005 16:49:53 -0400 Subject: [pypy-dev] Re: Patch to remove compilation warning with vc++ [repost] References: <030401c5c39c$9fd220f0$0300a8c0@anabelle> Message-ID: "Baptiste Lepilleur" wrote in message news:030401c5c39c$9fd220f0$0300a8c0 at anabelle... > PS: the patch is attached... Reading via gmane, it no longer is ... ;-) From gaiacrtn at free.fr Wed Sep 28 09:17:39 2005 From: gaiacrtn at free.fr (Baptiste Lepilleur) Date: Wed, 28 Sep 2005 09:17:39 +0200 Subject: [pypy-dev] Patch to remove compilation warning with vc++ [repost] References: <030401c5c39c$9fd220f0$0300a8c0@anabelle> Message-ID: <040101c5c3fc$b65e85b0$0300a8c0@anabelle> Baptiste Lepilleur wrote: >> The attached patch fix a few warning when compiling translated C >> with VC++: >> - direct.h not included (required for mkdir...) >> - mkdir does not takes a mode parameter >> - redefined macros >> [...] >> Baptiste. >> PS: the patch is attached... The attachement did not make it. It can be download for the following url: http://gaiacrtn.free.fr/temp/pypy/win32_warning.patch Baptiste. From hpk at trillke.net Wed Sep 28 10:13:48 2005 From: hpk at trillke.net (holger krekel) Date: Wed, 28 Sep 2005 10:13:48 +0200 Subject: [pypy-dev] Patch to remove compilation warning with vc++ [repost] In-Reply-To: <040101c5c3fc$b65e85b0$0300a8c0@anabelle> References: <030401c5c39c$9fd220f0$0300a8c0@anabelle> <040101c5c3fc$b65e85b0$0300a8c0@anabelle> Message-ID: <20050928081348.GJ10599@solar.trillke.net> On Wed, Sep 28, 2005 at 09:17 +0200, Baptiste Lepilleur wrote: > Baptiste Lepilleur wrote: > >>The attached patch fix a few warning when compiling translated C > >>with VC++: > >>- direct.h not included (required for mkdir...) > >>- mkdir does not takes a mode parameter > >>- redefined macros > >>[...] > >>Baptiste. > >>PS: the patch is attached... > > The attachement did not make it. It can be download for the following url: > http://gaiacrtn.free.fr/temp/pypy/win32_warning.patch Thanks, Baptiste. Maybe a general note: attachments are often scraped, if not by mailman then by gmane or other news/mail gateways. So it's often safest to post a link if you can. holger From aurelien.campeas at free.fr Wed Sep 28 13:17:56 2005 From: aurelien.campeas at free.fr (=?ISO-8859-1?Q?Aur=E9lien_Camp=E9as?=) Date: Wed, 28 Sep 2005 13:17:56 +0200 Subject: [pypy-dev] Re: Project suggestions In-Reply-To: <2md5muk6d0.fsf@starship.python.net> References: <20050923120202.GA7065@antelope> <20050923152846.GA7662@antelope> <20050923162916.GF15985@code1.codespeak.net> <43386A94.2090108@free.fr> <2mpsqukjn1.fsf@starship.python.net> <4339384C.6010906@free.fr> <2md5muk6d0.fsf@starship.python.net> Message-ID: <433A7BE4.4050702@free.fr> Michael Hudson a ?crit : > RPython is a subset of python with the main constraint being able to > do some level of static type analysis (for full python, the amount of > static ananlysis you can do is really very small, you can read Brett > Cannon's thesis about this: > > http://www.ocf.berkeley.edu/~bac/thesis.pdf). > > So, when translating, the code that implements the interpreter > (roughly interpreter/*.py, objspace/std/*.py) is imported and a flow > graph built from it. This is then annotated (code in annotator/*.py), > a fairly hairy process. This (for the C and LLVM backends, at least) > is then turned into a graph containing low level operations (like > "dereference this pointer"). > > Python is just the language all of the above happens to be implemented > in (and also the interpreter/*.py code is involved in making a flow > graph which includes itself, but this isn't that important -- just > confusing :). So the same graph can exist in three states : raw python, annotated python (possible whenever the raw python matches rpython), low-level stuff ? Then, to translate unrestricted python, I have to work on the first pass/state of the graph. >>>>In fact, the distance from de-sugared python to the opcode is >>>>unknown to me (even though I suspect there is no 1:1 mapping between >>>>the two). Whatever, that could be the start of a strategy to >>>>translate python towards other high-level languages (ruby, or js as >>>>in the other thread) without paying the full price of opcode >>>>interpretation in the target (that is : parts which are semantically >>>>similar could be easily translated, others would be -costily- >>>>emulated). >>> >>>Thing is, I don't know how feasible this is. It's pretty hard, >>>without some kind of type inference, to translate, say this Python: >>> a + b >>>into anything significantly more efficient than this Common Lisp: >>> (py:add a b) >> >>The mere fact that it will be compiled will make it more efficient I >>guess. I mean, not on CLISP, but with a real lisp compiler. > > > Not much. The "interpreter overhead" for Python is usually estimated > at about 20%, though it obviously depends what code you're running to > some extent. hmmmm, 20% related to what ? Is this an observed quantity from benchmarking, say, C-translated rpython vs CPython ? Also, "on average" can, as you say, cover a lot of differences depending on what we have to do. > >>>And making type inference possible is what RPython is all about. >> >>Sure, but then, it is a restricted subset of Python, and I like python >>completely unrestricted ;) > > > Well, so do we all, but then you can't have type inference. There is > no simple answer to this. Has the possibility to extend python with optional type annotations been studied ? (I guess the pypy-dev archives could answer this) > > >>>You could make #'py:add a generic function and see if a given CLOS >>>implementation is fast enough to give a useful speed (but I think the >>>coercion rules would probably drive you insane first). >> >>In this case, CLOS would add overhead. In fact the python add operator >>(and some arith. ops) seems close enough to the lisp one that one can >>be tempted to translate in the lisp equivalent with only minor >>adaptation (like the printer appending an "L" on bignums, etc.). > > > Er... no. #'cl:+ is not that much like python's + (e.g. the latter > operates on strings and you can't overload the former). Thanks for reminding me that. > >>I am not sure I would use CLOS at all, in fact (at least for a first >>attempt at producing a lisp backend). > > Fair enough. > ... but the more I think about it, it looks like CLOS might help with these overloaded operators ... > >>BTW, what's this "insanity with coercion rules" that you mention - >>can you expand a little on this ? > > > Think about things like 2L**135 < 1.0e40 or range(10)*3 or... mixed > type operations are not that simple in Python. > ... and coercion rules ... Well, for now I will go on tweaking a CL backend based on pyrex's one. I guess I will discover the real problems as I progress. Thanks also to Bert and Samuele, Aur?lien. From mwh at python.net Wed Sep 28 13:41:40 2005 From: mwh at python.net (Michael Hudson) Date: Wed, 28 Sep 2005 12:41:40 +0100 Subject: [pypy-dev] Re: Project suggestions References: <20050923120202.GA7065@antelope> <20050923152846.GA7662@antelope> <20050923162916.GF15985@code1.codespeak.net> <43386A94.2090108@free.fr> <2mpsqukjn1.fsf@starship.python.net> <4339384C.6010906@free.fr> <2md5muk6d0.fsf@starship.python.net> <433A7BE4.4050702@free.fr> Message-ID: <2m8xxhju6j.fsf@starship.python.net> Aur?lien Camp?as writes: > Michael Hudson a ?crit : >> RPython is a subset of python with the main constraint being able to >> do some level of static type analysis (for full python, the amount of >> static ananlysis you can do is really very small, you can read Brett >> Cannon's thesis about this: >> http://www.ocf.berkeley.edu/~bac/thesis.pdf). >> So, when translating, the code that implements the interpreter >> (roughly interpreter/*.py, objspace/std/*.py) is imported and a flow >> graph built from it. This is then annotated (code in annotator/*.py), >> a fairly hairy process. This (for the C and LLVM backends, at least) >> is then turned into a graph containing low level operations (like >> "dereference this pointer"). >> Python is just the language all of the above happens to be >> implemented >> in (and also the interpreter/*.py code is involved in making a flow >> graph which includes itself, but this isn't that important -- just >> confusing :). > > So the same graph can exist in three states : raw python, annotated > python (possible whenever the raw python matches rpython), low-level > stuff ? Yes. Though I don't know if you can even form the flow graph of unrestricted Python (not saying you can't, just it wasn't a design goal). > Then, to translate unrestricted python, I have to work on the first > pass/state of the graph. Yes. But believe us on this one: ahead-of-time analysis on unrestricted python is not going to lead to significant speedups. >>>>Thing is, I don't know how feasible this is. It's pretty hard, >>>>without some kind of type inference, to translate, say this Python: >>>> a + b >>>>into anything significantly more efficient than this Common Lisp: >>>> (py:add a b) >>> >>>The mere fact that it will be compiled will make it more efficient I >>>guess. I mean, not on CLISP, but with a real lisp compiler. >> Not much. The "interpreter overhead" for Python is usually >> estimated >> at about 20%, though it obviously depends what code you're running to >> some extent. > > hmmmm, 20% related to what ? Is this an observed quantity from > benchmarking, say, C-translated rpython vs CPython ? Well, it's very much a guesstimate, but what I was meaning to compare was interpreted python as against compiled c code that executes the same operations in the same order but without the overhead of dispatching the opcodes (this is more or less what the now more-or-less dead Python2C project did. There's a reason it's more-or-less dead). > Also, "on average" can, as you say, cover a lot of differences > depending on what we have to do. Indeed! If you're computation is bound, say, by the speed of multiplying really large integers, the interpreter overhead is roughly zero. If you're manipluating small ints, then it's probably quite large. >>>>And making type inference possible is what RPython is all about. >>> >>>Sure, but then, it is a restricted subset of Python, and I like python >>>completely unrestricted ;) >> Well, so do we all, but then you can't have type inference. There >> is >> no simple answer to this. > > Has the possibility to extend python with optional type annotations > been studied ? (I guess the pypy-dev archives could answer this) Well, there's been a fair amount of hot air emitted about it, but no actual work that I know about. >>>I am not sure I would use CLOS at all, in fact (at least for a first >>>attempt at producing a lisp backend). >> Fair enough. >> > > ... but the more I think about it, it looks like CLOS might help with > these overloaded operators ... Maybe :) Please don't let me discourage you too much; I'd be genuinely interested in what you find. But please don't expect miracles. Cheers, mwh -- want to write my requirements for me? Sure! "show a dancing monkey in the about box" -- from Twisted.Quotes From aurelien.campeas at free.fr Wed Sep 28 15:46:51 2005 From: aurelien.campeas at free.fr (=?ISO-8859-1?Q?Aur=E9lien_Camp=E9as?=) Date: Wed, 28 Sep 2005 15:46:51 +0200 Subject: [pypy-dev] Re: Project suggestions In-Reply-To: <2m8xxhju6j.fsf@starship.python.net> References: <20050923120202.GA7065@antelope> <20050923152846.GA7662@antelope> <20050923162916.GF15985@code1.codespeak.net> <43386A94.2090108@free.fr> <2mpsqukjn1.fsf@starship.python.net> <4339384C.6010906@free.fr> <2md5muk6d0.fsf@starship.python.net> <433A7BE4.4050702@free.fr> <2m8xxhju6j.fsf@starship.python.net> Message-ID: <433A9ECB.3020306@free.fr> Michael Hudson a ?crit : > Aur?lien Camp?as writes: >>So the same graph can exist in three states : raw python, annotated >>python (possible whenever the raw python matches rpython), low-level >>stuff ? > > > Yes. Though I don't know if you can even form the flow graph of > unrestricted Python (not saying you can't, just it wasn't a design > goal). > Still confused I am, then. I thought you could annotate the graph because you programmed into rpython and thus enabled type inference, thereby allowing the production of a low-level C-friendly version of the graph ... ok, it's not that important right now for me to understand it all anyway > >>Then, to translate unrestricted python, I have to work on the first >>pass/state of the graph. > > > Yes. But believe us on this one: ahead-of-time analysis on > unrestricted python is not going to lead to significant speedups. But there are maybe other benefits of an unrestricted python translator to lisp than only getting it natively compiled for small perf gains. Ability to develop quickly arbitrary extensions to python could be one beneficial side-effect, for instance. >>>>>Thing is, I don't know how feasible this is. It's pretty hard, >>>>>without some kind of type inference, to translate, say this Python: >>>>> a + b >>>>>into anything significantly more efficient than this Common Lisp: >>>>> (py:add a b) >>>> >>>>The mere fact that it will be compiled will make it more efficient I >>>>guess. I mean, not on CLISP, but with a real lisp compiler. >>> >>>Not much. The "interpreter overhead" for Python is usually >>>estimated >>>at about 20%, though it obviously depends what code you're running to >>>some extent. >> >>hmmmm, 20% related to what ? Is this an observed quantity from >>benchmarking, say, C-translated rpython vs CPython ? > > > Well, it's very much a guesstimate, but what I was meaning to compare > was interpreted python as against compiled c code that executes the > same operations in the same order but without the overhead of > dispatching the opcodes (this is more or less what the now > more-or-less dead Python2C project did. There's a reason it's > more-or-less dead). > > >>Also, "on average" can, as you say, cover a lot of differences >>depending on what we have to do. > > > Indeed! If you're computation is bound, say, by the speed of > multiplying really large integers, the interpreter overhead is roughly > zero. If you're manipluating small ints, then it's probably quite > large. Builtin ops like bignums arithmetic or whatever is implemented in C is obviously fast. OTOH, I wonder if some implementation choices of current CPyton, and part of its slowness, were made balancing simplicity of the code versus speed (stackless could be an example of a faster implementation, couldn't it ?). I remember having read stuff about that in some distant past. > >>... but the more I think about it, it looks like CLOS might help with >>these overloaded operators ... > > > Maybe :) Please don't let me discourage you too much; I'd be genuinely > interested in what you find. But please don't expect miracles. > Thanks, I am not discouraged and don't expect miracles :) What pypy brings me right now is a nice way to enhance my {python,lisp}-fu at the same time. I am for sure quite far from discovering anything not already known. Would this happen, I'd let pypy people informed. Cheers, Aur?lien. From lac at strakt.com Wed Sep 28 18:20:33 2005 From: lac at strakt.com (Laura Creighton) Date: Wed, 28 Sep 2005 18:20:33 +0200 Subject: [pypy-dev] How agile is PyPy? In-Reply-To: Message from Beatrice During of "Tue, 27 Sep 2005 22:20:52 +0200." References: Message-ID: <200509281620.j8SGKX6X005875@theraft.strakt.com> One thing that is rareley mentioned but rather important for agile development is the measurement of how often you _get it wrong_. (Or at least, throw it out.) This is because the results run counter to traditional metrics of software development. Agility means the ability to get it wrong. So -- within limits -- more wrong is better. In traditional, completely non-agile, development, there is only time to build one implementation. You must get it correct the first time. Therefore you spend forever sitting down and planning 'the correct way to do something'. And then you do it. Of course, fairly soon it becomes obvious that the very best plan you could come up with overlooked certain very important things. And this is just too bad, because you don't have time to do things except the one way, so you have to make as few changes to the master plan as possible ... and so it goes. Eventually, over time, the software can no longer stand the new demands that are put on it, and it is time to rewrite from scratch. You just cannot patch it up any more. In agile development you rewrite a lot of small things a lot. Instead of discussing what is the best way to implement feature Q, you can just go off and _build_ 3 ways to implement feature Q, and then look at them and measure them, if it is appropriate, and see what is better. But because you have the ability to persue different ways of doing things, you end up throwing out a lot of code. You also need language support for this -- you can talk all you like about agile development, but some languages are so hard to write code in that you really will not have time to try 3 or 5 or 7 implementations of an idea. Plus having sweated bricks to make one, you are in love with your own code, and find it hard to toss it when the opportunity arises. So if you aren't throwing out a lot of code, then you aren't being very agile. I'd say we're excellent on that score. Also -- while traditional agile methodolgies talk about _responding_ to change -- generally from the customer -- we don't have any. So we make change just because we think it would be a good -- or interesting thing to do. We're not responding to outside pressures. Indeed, we would like it if the EU was a bit more flexible, and we could make some changes to our proposal on the basis of what we have learned by doing. But the EU is not very agile, and poor at responding to changes desired by funded project consortia. Laura In a message of Tue, 27 Sep 2005 22:20:52 +0200, Beatrice During writes: >--8323328-2105496072-1127852452=:25960 >Content-Type: TEXT/PLAIN; charset=ISO-8859-1; format=flowed >Content-Transfer-Encoding: 8BIT > >Hi there > >I am working on documenting PyPy (the project) as a showcase of agile >development. In order to be able to do this I would love to get some >thoughts and reflections from you who are working on /following PyPy and >the question is of course - how agile are we? > >Here is some short notes on success factors of agile approaches (from an >eworkshop in 2002 - Cockburn, Beck etc, >http://fc-md.umd.edu/projects/Agile/Summary/SummaryPF.htm): > >------------------------------------------------------------------------- >In general, the group agreed that the following are the success factors >for agile development: > >? Negotiation: If people are willing to keep talking and learning, > you are, almost by definition, agile. > >? Ability to continuously learn > >? Responsibly responding to change. > >? Environment where rapid communication is enabled. > >? Team reasonably in charge of its own destiny (being allowed to > succeed). > >? People are valued over other factors > >? The team needs to be more than just able, it needs to actually > succeed. There needs to be a support network for each > individual in the > team to make a contribution however small. >------------------------------------------------------------------------ > >How does these factors apply to PyPy and this community? Are we doing >other things that is not mentioned here that is working well for us? Thin >gs we could do better? > >Your responses will be part of documentation about PyPy/agile dev >(sprints) that will be distributed to the EU, Universities, various >companies and other Open Source projects who have stated a clear interest > >in the PyPy showcase..... > >If you feel that answering this email doesnt really fit the purpose of >this mailinglist - please answer to me specifically. > >Thank you for your contribution! > >Cheers >Bea > > >Beatrice D?ring Change Maker >Tel: 031- 7750940 J?rntorget 3 >Mobil: 0734- 22 89 06 413 04 G?teborg >E-post: bea at changemaker.nu www.changemaker.nu > > "Alla dessa m?sten och alldaglighet. > Allt detta som binder v?r verklighet > i bojor av skam och rep utav tv?ng. > Alla dessa kedjor som binder v?r s?ng. > Jag skall slita dem alla i sm?, sm? stycken > och m?jligtvis av resterna g?ra mig smycken." > - hemlig > >--8323328-2105496072-1127852452=:25960 >Content-Type: text/plain; charset="us-ascii" >MIME-Version: 1.0 >Content-Transfer-Encoding: 7bit >Content-Disposition: inline > >_______________________________________________ >pypy-dev at codespeak.net >http://codespeak.net/mailman/listinfo/pypy-dev >--8323328-2105496072-1127852452=:25960-- From serendipity-soft at get2net.dk Wed Sep 28 18:28:16 2005 From: serendipity-soft at get2net.dk (Anders Lehmann) Date: Wed, 28 Sep 2005 18:28:16 +0200 Subject: [pypy-dev] Next #pypy-sync meeting thursday 13.00 CET Message-ID: Til: pypy-dev at codespeak.net Hi all, Next #pypy-sync meeting on freenode.net is tomorrow Thursday at 1pm CET as usual. Regular topics -------------- * activity reports (3 prepared lines of info) * resolve conflicts/blockers Here is my draft proposal of Topics of the week ------------------ Pre christmas sprint Do we have a venue ? Compliance Just so we dont forget in our speed effort. There are still several missing pieces (pep 302, ...) Marketing As a derivative of a previous topic ("making money"), I am trying to get some danish technical magazines to write about pypy. It occured to me that maybe we should make a press release when we pass a milestone next time. Please propose additions and changes. See you tomorrow at 1 pm. Kind regards Anders (ale) -------------- next part -------------- An HTML attachment was scrubbed... URL: From bea at netg.se Wed Sep 28 18:46:52 2005 From: bea at netg.se (Beatrice During) Date: Wed, 28 Sep 2005 18:46:52 +0200 (CEST) Subject: [pypy-dev] How agile is PyPy? In-Reply-To: <200509281620.j8SGKX6X005875@theraft.strakt.com> References: <200509281620.j8SGKX6X005875@theraft.strakt.com> Message-ID: Hi there On Wed, 28 Sep 2005, Laura Creighton wrote: > > One thing that is rareley mentioned but rather important for agile > development is the measurement of how often you _get it wrong_. (Or > at least, throw it out.) This is because the results run counter to > traditional metrics of software development. Agility means the > ability to get it wrong. So -- within limits -- more wrong is better. > > In traditional, completely non-agile, development, there is only time > to build one implementation. You must get it correct the first time. > Therefore you spend forever sitting down and planning 'the correct way > to do something'. And then you do it. Of course, fairly soon it > becomes obvious that the very best plan you could come up with > overlooked certain very important things. And this is just too bad, > because you don't have time to do things except the one way, so you > have to make as few changes to the master plan as possible ... and so > it goes. Eventually, over time, the software can no longer stand the > new demands that are put on it, and it is time to rewrite from scratch. > You just cannot patch it up any more. > > In agile development you rewrite a lot of small things a lot. Instead > of discussing what is the best way to implement feature Q, you can > just go off and _build_ 3 ways to implement feature Q, and then look > at them and measure them, if it is appropriate, and see what is > better. But because you have the ability to persue different ways of > doing things, you end up throwing out a lot of code. > > You also need language support for this -- you can talk all you like > about agile development, but some languages are so hard to write code > in that you really will not have time to try 3 or 5 or 7 > implementations of an idea. Plus having sweated bricks to make one, > you are in love with your own code, and find it hard to toss it when > the opportunity arises. > > So if you aren't throwing out a lot of code, then you aren't being > very agile. I'd say we're excellent on that score. So the infrastructure for being able to do that (our entire test suite) is a key component for "throwing away"? How does our automated test suites fare in comparison with other OSS/agile projects? (Relax Holger ;-) > Also -- while traditional agile methodolgies talk about _responding_ > to change -- generally from the customer -- we don't have any. So we > make change just because we think it would be a good -- or interesting > thing to do. We're not responding to outside pressures. Indeed, we > would like it if the EU was a bit more flexible, and we could make > some changes to our proposal on the basis of what we have learned by > doing. But the EU is not very agile, and poor at responding to > changes desired by funded project consortia. I agree - so to summarize - within the community and the boundary of the PyPy project we are really fullfilling agile thoughts and values - but the EU boundaries of the projects is "dampening" our "Ability to continuously learn" and "Responsibly responding to change" - more? Thanks for your thoughts on this Laura! Cheers Bea > Laura > > In a message of Tue, 27 Sep 2005 22:20:52 +0200, Beatrice During writes: >> --8323328-2105496072-1127852452=:25960 >> Content-Type: TEXT/PLAIN; charset=ISO-8859-1; format=flowed >> Content-Transfer-Encoding: 8BIT >> >> Hi there >> >> I am working on documenting PyPy (the project) as a showcase of agile >> development. In order to be able to do this I would love to get some >> thoughts and reflections from you who are working on /following PyPy and >> the question is of course - how agile are we? >> >> Here is some short notes on success factors of agile approaches (from an >> eworkshop in 2002 - Cockburn, Beck etc, >> http://fc-md.umd.edu/projects/Agile/Summary/SummaryPF.htm): >> >> ------------------------------------------------------------------------- >> In general, the group agreed that the following are the success factors >> for agile development: >> >> ? Negotiation: If people are willing to keep talking and learning, >> you are, almost by definition, agile. >> >> ? Ability to continuously learn >> >> ? Responsibly responding to change. >> >> ? Environment where rapid communication is enabled. >> >> ? Team reasonably in charge of its own destiny (being allowed to >> succeed). >> >> ? People are valued over other factors >> >> ? The team needs to be more than just able, it needs to actually >> succeed. There needs to be a support network for each >> individual in the >> team to make a contribution however small. >> ------------------------------------------------------------------------ >> >> How does these factors apply to PyPy and this community? Are we doing >> other things that is not mentioned here that is working well for us? Thin >> gs we could do better? >> >> Your responses will be part of documentation about PyPy/agile dev >> (sprints) that will be distributed to the EU, Universities, various >> companies and other Open Source projects who have stated a clear interest >> >> in the PyPy showcase..... >> >> If you feel that answering this email doesnt really fit the purpose of >> this mailinglist - please answer to me specifically. >> >> Thank you for your contribution! >> >> Cheers >> Bea >> >> >> Beatrice D?ring Change Maker >> Tel: 031- 7750940 J?rntorget 3 >> Mobil: 0734- 22 89 06 413 04 G?teborg >> E-post: bea at changemaker.nu www.changemaker.nu >> >> "Alla dessa m?sten och alldaglighet. >> Allt detta som binder v?r verklighet >> i bojor av skam och rep utav tv?ng. >> Alla dessa kedjor som binder v?r s?ng. >> Jag skall slita dem alla i sm?, sm? stycken >> och m?jligtvis av resterna g?ra mig smycken." >> - hemlig >> >> --8323328-2105496072-1127852452=:25960 >> Content-Type: text/plain; charset="us-ascii" >> MIME-Version: 1.0 >> Content-Transfer-Encoding: 7bit >> Content-Disposition: inline >> >> _______________________________________________ >> pypy-dev at codespeak.net >> http://codespeak.net/mailman/listinfo/pypy-dev >> --8323328-2105496072-1127852452=:25960-- > Beatrice D?ring Change Maker Tel: 031- 7750940 J?rntorget 3 Mobil: 0734- 22 89 06 413 04 G?teborg E-post: bea at changemaker.nu www.changemaker.nu "Alla dessa m?sten och alldaglighet. Allt detta som binder v?r verklighet i bojor av skam och rep utav tv?ng. Alla dessa kedjor som binder v?r s?ng. Jag skall slita dem alla i sm?, sm? stycken och m?jligtvis av resterna g?ra mig smycken." - hemlig From lac at strakt.com Wed Sep 28 19:37:17 2005 From: lac at strakt.com (Laura Creighton) Date: Wed, 28 Sep 2005 19:37:17 +0200 Subject: [pypy-dev] How agile is PyPy? In-Reply-To: Message from Beatrice During of "Wed, 28 Sep 2005 18:46:52 +0200." References: <200509281620.j8SGKX6X005875@theraft.strakt.com> Message-ID: <200509281737.j8SHbHOT009277@theraft.strakt.com> In a message of Wed, 28 Sep 2005 18:46:52 +0200, Beatrice During writes: >--8323328-735653061-1127926012=:6433 >Content-Type: TEXT/PLAIN; charset=ISO-8859-1; format=flowed >Content-Transfer-Encoding: 8BIT > >Hi there > >On Wed, 28 Sep 2005, Laura Creighton wrote: >> So if you aren't throwing out a lot of code, then you aren't being >> very agile. I'd say we're excellent on that score. > >So the infrastructure for being able to do that (our entire test suite) is >a key component for "throwing away"? How does our automated test suites >fare in comparison with other OSS/agile projects? >(Relax Holger ;-) I think that py.test makes doing test driven development easier and faster. The quality of your test suites depends on the amount of time that developers are willing to put into making tests. I think that the 'NO API' approach makes writing py.tests a lot easier and faster than other testing frameworks that do have an api. Easier and Faster to Write == More Tests == Better Test Suite. And a good test suite is essential .... but I would not think that it is more essential for agile development than for fossilised development. But then I am such a fan of test driven development, that I cannot imagine going any other way, even in a non-agile project. I suspect that TDD is necessary but not sufficient for agile development. It used to be that the state of the test suites in all software projects could be assumed to be abysmal. This is changing. But there is still a lot we do not test. We test the things that interest us, but don't have a way to test the things we never think of. This is a universal failing of all test suites, and I believe that ours is no worse than others, but it still is a flaw. >I agree - so to summarize - within the community and the boundary of the >PyPy project we are really fullfilling agile thoughts and values - but the >EU boundaries of the projects is "dampening" our "Ability to continuously > >learn" and "Responsibly responding to change" - more? Ah, I think we are continuously learning no matter what. It is more of a problem of not being able to immediately act on what we have learned. And the problem of having to solve the problem in the order we told the EU we would, for the purpose of getting the deliverables done on time, rather than what would be convenient for us. When a new person shows up in the project, they want to immediately work on the thing that gives them the best benefit. The notion of 'well, we could do this now but we aren't supposed to do things like that until next March, according to the schedule' isn't very agile at all. Laura >Thanks for your thoughts on this Laura! My pleasure. > >Cheers > >Bea > > >> Laura >> >> In a message of Tue, 27 Sep 2005 22:20:52 +0200, Beatrice During writes >: >>> --8323328-2105496072-1127852452=:25960 >>> Content-Type: TEXT/PLAIN; charset=ISO-8859-1; format=flowed >>> Content-Transfer-Encoding: 8BIT >>> >>> Hi there >>> >>> I am working on documenting PyPy (the project) as a showcase of agile >>> development. In order to be able to do this I would love to get some >>> thoughts and reflections from you who are working on /following PyPy a >nd >>> the question is of course - how agile are we? >>> >>> Here is some short notes on success factors of agile approaches (from >an >>> eworkshop in 2002 - Cockburn, Beck etc, >>> http://fc-md.umd.edu/projects/Agile/Summary/SummaryPF.htm): >>> >>> ---------------------------------------------------------------------- >--- >>> In general, the group agreed that the following are the success factor >s >>> for agile development: >>> >>> ? Negotiation: If people are willing to keep talking and learni >ng, >>> you are, almost by definition, agile. >>> >>> ? Ability to continuously learn >>> >>> ? Responsibly responding to change. >>> >>> ? Environment where rapid communication is enabled. >>> >>> ? Team reasonably in charge of its own destiny (being allowed t >o >>> succeed). >>> >>> ? People are valued over other factors >>> >>> ? The team needs to be more than just able, it needs to actuall >y >>> succeed. There needs to be a support network for each >>> individual in the >>> team to make a contribution however small. >>> ---------------------------------------------------------------------- >-- >>> >>> How does these factors apply to PyPy and this community? Are we doing >>> other things that is not mentioned here that is working well for us? T >hin >>> gs we could do better? >>> >>> Your responses will be part of documentation about PyPy/agile dev >>> (sprints) that will be distributed to the EU, Universities, various >>> companies and other Open Source projects who have stated a clear inter >est >>> >>> in the PyPy showcase..... >>> >>> If you feel that answering this email doesnt really fit the purpose of >>> this mailinglist - please answer to me specifically. >>> >>> Thank you for your contribution! >>> >>> Cheers >>> Bea >>> >>> >>> Beatrice D?ring Change Maker >>> Tel: 031- 7750940 J?rntorget 3 >>> Mobil: 0734- 22 89 06 413 04 G?teborg >>> E-post: bea at changemaker.nu www.changemaker.nu >>> >>> "Alla dessa m?sten och alldaglighet. >>> Allt detta som binder v?r verklighet >>> i bojor av skam och rep utav tv?ng. >>> Alla dessa kedjor som binder v?r s?ng. >>> Jag skall slita dem alla i sm?, sm? stycken >>> och m?jligtvis av resterna g?ra mig smycken." >>> - hemlig >>> >>> --8323328-2105496072-1127852452=:25960 >>> Content-Type: text/plain; charset="us-ascii" >>> MIME-Version: 1.0 >>> Content-Transfer-Encoding: 7bit >>> Content-Disposition: inline >>> >>> _______________________________________________ >>> pypy-dev at codespeak.net >>> http://codespeak.net/mailman/listinfo/pypy-dev >>> --8323328-2105496072-1127852452=:25960-- >> > >Beatrice D?ring Change Maker >Tel: 031- 7750940 J?rntorget 3 >Mobil: 0734- 22 89 06 413 04 G?teborg >E-post: bea at changemaker.nu www.changemaker.nu > > "Alla dessa m?sten och alldaglighet. > Allt detta som binder v?r verklighet > i bojor av skam och rep utav tv?ng. > Alla dessa kedjor som binder v?r s?ng. > Jag skall slita dem alla i sm?, sm? stycken > och m?jligtvis av resterna g?ra mig smycken." > - hemlig > >--8323328-735653061-1127926012=:6433-- From tismer at stackless.com Thu Sep 29 00:51:13 2005 From: tismer at stackless.com (Christian Tismer) Date: Thu, 29 Sep 2005 00:51:13 +0200 Subject: [pypy-dev] Re: Project suggestions In-Reply-To: <433A9ECB.3020306@free.fr> References: <20050923120202.GA7065@antelope> <20050923152846.GA7662@antelope> <20050923162916.GF15985@code1.codespeak.net> <43386A94.2090108@free.fr> <2mpsqukjn1.fsf@starship.python.net> <4339384C.6010906@free.fr> <2md5muk6d0.fsf@starship.python.net> <433A7BE4.4050702@free.fr> <2m8xxhju6j.fsf@starship.python.net> <433A9ECB.3020306@free.fr> Message-ID: <433B1E61.80306@stackless.com> Aur?lien Camp?as wrote: ... > Builtin ops like bignums arithmetic or whatever is implemented in C is > obviously fast. OTOH, I wonder if some implementation choices of current > CPyton, and part of its slowness, were made balancing simplicity of the > code versus speed (stackless could be an example of a faster > implementation, couldn't it ?). I remember having read stuff about that > in some distant past. At that time this was true, Stackless had been 5% slower than normal Python. Somewhere at 1.5.2 :-) But at the same time, I had implemented an aceleration of the interpreter loop of 10-15 %, which worked especially well with the windows compiler. Python was not interested in my path, only recently they are selling their grandma for a little speed. So I took the chance to speed up my slightly slower Stackless, to get a little advantage for those who didn't realize the real benefits of Stackless. Summary: No, it isn't faster, maybe even slightly slower. But it can be much faster if you use its features to implement your algorithms in a way Python cannot do it. Which still gets more relative because they pushed generators to the extremes (still limited but good). ciao - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From tismer at stackless.com Thu Sep 29 01:02:28 2005 From: tismer at stackless.com (Christian Tismer) Date: Thu, 29 Sep 2005 01:02:28 +0200 Subject: [pypy-dev] Fw: Crash! In-Reply-To: References: Message-ID: <433B2104.5080503@stackless.com> Ben.Young at risk.sungard.com wrote: > Hi Samuele, > > My build configuation is windows, no boehm, no lomem. Is there an option > to build a debug binary on windows? If so i'll look into what's happening > for you! The visual studio debugger is pretty good. You need to build a project for Windows. If you like, I can send you one, or I might check mine in as an example. Problem is (for my VC++ 7.1) that it is not really usable with too large files. And every step in debugging takes 2 seconds. No fun. ciao - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From davidf at sjsoft.com Thu Sep 29 12:13:46 2005 From: davidf at sjsoft.com (David Fraser) Date: Thu, 29 Sep 2005 12:13:46 +0200 Subject: [pypy-dev] a faster Python not a primary goal of PyPy? In-Reply-To: <20050926014221.23477.qmail@web40527.mail.yahoo.com> References: <20050926014221.23477.qmail@web40527.mail.yahoo.com> Message-ID: <433BBE5A.5020809@sjsoft.com> Alex Martelli wrote: >--- Armin Rigo wrote: > ... > > >>If anyone shows up with a Javascript parser, I'm sure we could >>together >>hack (and translate (and later have a JIT from)) a basic Javascript >>interpreter within a few weeks :-) >> >> > >Actually, I think the currently useful thing might be a way to >compile (any decent language, Python for choice) INTO Javascript >(with reasonable efficiency), so one could write AJAX pages without >actually having to code in Javascript...;-) > >Alex > > I took a stab at this once based on somebody else's code... http://davidf.sjsoft.com/files/py2js I found some Python constructs ended up being tricky to translate to JavaScript (as the rest of the thread said). So the idea here was to be able to start with some Python code, get it into basic JavaScript syntax, but then you'd probably have to munge it anyway Cheers David From arigo at tunes.org Thu Sep 29 14:11:33 2005 From: arigo at tunes.org (Armin Rigo) Date: Thu, 29 Sep 2005 14:11:33 +0200 Subject: [pypy-dev] Re: Project suggestions In-Reply-To: <433A7BE4.4050702@free.fr> References: <20050923120202.GA7065@antelope> <20050923152846.GA7662@antelope> <20050923162916.GF15985@code1.codespeak.net> <43386A94.2090108@free.fr> <2mpsqukjn1.fsf@starship.python.net> <4339384C.6010906@free.fr> <2md5muk6d0.fsf@starship.python.net> <433A7BE4.4050702@free.fr> Message-ID: <20050929121133.GA27263@code1.codespeak.net> Hi Aurelien, On Wed, Sep 28, 2005 at 01:17:56PM +0200, Aur?lien Camp?as wrote: > Well, for now I will go on tweaking a CL backend based on pyrex's one. I > guess I will discover the real problems as I progress. Any reason not to start from Seo's GenCL backend (pypy/translator/gencl) ? A bientot, Armin From aurelien.campeas at free.fr Thu Sep 29 14:54:01 2005 From: aurelien.campeas at free.fr (=?ISO-8859-1?Q?Aur=E9lien_Camp=E9as?=) Date: Thu, 29 Sep 2005 14:54:01 +0200 Subject: [pypy-dev] Re: Project suggestions In-Reply-To: <20050929121133.GA27263@code1.codespeak.net> References: <20050923120202.GA7065@antelope> <20050923152846.GA7662@antelope> <20050923162916.GF15985@code1.codespeak.net> <43386A94.2090108@free.fr> <2mpsqukjn1.fsf@starship.python.net> <4339384C.6010906@free.fr> <2md5muk6d0.fsf@starship.python.net> <433A7BE4.4050702@free.fr> <20050929121133.GA27263@code1.codespeak.net> Message-ID: <433BE3E9.7030703@free.fr> Armin Rigo a ?crit : > Hi Aurelien, > > On Wed, Sep 28, 2005 at 01:17:56PM +0200, Aur?lien Camp?as wrote: > >>Well, for now I will go on tweaking a CL backend based on pyrex's one. I >>guess I will discover the real problems as I progress. > > > Any reason not to start from Seo's GenCL backend (pypy/translator/gencl) ? That's what I tried first. Then I found it was much easier to work from pyrex translator (maybe because the later is slightly more complete than current gencl.py ?). I have yet to learn how to correctly walk the function graph and it is easier to use pyrex as a starting template than gencl. Moreover, gencl is lisp-wise wrong (setq-ing previously undefined variables, while accepted with warnings by some compilers, is not correct ; starting from genpyrex, it was easy to generate a big top-level LET form for each function, for instance). Also note that I am working very slowly on this since am currently busy looking for a job :) (hunting down positions, tailoring motivation letters and CVs is not an easy task...) ... but nevertheless hacking on gencl is a nice way to recover from anxiety :) > > > A bientot, > > Armin > > A+ Aur?lien From aurelien.campeas at free.fr Thu Sep 29 14:59:21 2005 From: aurelien.campeas at free.fr (=?ISO-8859-1?Q?Aur=E9lien_Camp=E9as?=) Date: Thu, 29 Sep 2005 14:59:21 +0200 Subject: [pypy-dev] Re: Project suggestions In-Reply-To: <433B1E61.80306@stackless.com> References: <20050923120202.GA7065@antelope> <20050923152846.GA7662@antelope> <20050923162916.GF15985@code1.codespeak.net> <43386A94.2090108@free.fr> <2mpsqukjn1.fsf@starship.python.net> <4339384C.6010906@free.fr> <2md5muk6d0.fsf@starship.python.net> <433A7BE4.4050702@free.fr> <2m8xxhju6j.fsf@starship.python.net> <433A9ECB.3020306@free.fr> <433B1E61.80306@stackless.com> Message-ID: <433BE529.6070107@free.fr> Christian Tismer a ?crit : > Aur?lien Camp?as wrote: > > ... > >> Builtin ops like bignums arithmetic or whatever is implemented in C is >> obviously fast. OTOH, I wonder if some implementation choices of >> current CPyton, and part of its slowness, were made balancing >> simplicity of the code versus speed (stackless could be an example of >> a faster implementation, couldn't it ?). I remember having read stuff >> about that in some distant past. > > > At that time this was true, Stackless had been 5% slower than > normal Python. Somewhere at 1.5.2 :-) > But at the same time, I had implemented an aceleration of the > interpreter loop of 10-15 %, which worked especially well with > the windows compiler. Python was not interested in my path, only > recently they are selling their grandma for a little speed. > > So I took the chance to speed up my slightly slower Stackless, > to get a little advantage for those who didn't realize the > real benefits of Stackless. > > Summary: No, it isn't faster, maybe even slightly slower. > But it can be much faster if you use its features to implement > your algorithms in a way Python cannot do it. Which still gets > more relative because they pushed generators to the extremes > (still limited but good). > > ciao - chris > Isn't stackless currently used by the eve-online game, server-side, for its ability to do blazingly fast massive coroutining (or something like that, it would be interesting to know the details, by the way) ? From tismer at stackless.com Thu Sep 29 15:07:40 2005 From: tismer at stackless.com (Christian Tismer) Date: Thu, 29 Sep 2005 15:07:40 +0200 Subject: [pypy-dev] Re: Project suggestions In-Reply-To: <433BE529.6070107@free.fr> References: <20050923120202.GA7065@antelope> <20050923152846.GA7662@antelope> <20050923162916.GF15985@code1.codespeak.net> <43386A94.2090108@free.fr> <2mpsqukjn1.fsf@starship.python.net> <4339384C.6010906@free.fr> <2md5muk6d0.fsf@starship.python.net> <433A7BE4.4050702@free.fr> <2m8xxhju6j.fsf@starship.python.net> <433A9ECB.3020306@free.fr> <433B1E61.80306@stackless.com> <433BE529.6070107@free.fr> Message-ID: <433BE71C.4030008@stackless.com> Aur?lien Camp?as wrote: ... > Isn't stackless currently used by the eve-online game, server-side, for > its ability to do blazingly fast massive coroutining (or something like > that, it would be interesting to know the details, by the way) ? Sure it is, and both server- and client-side. -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From tismer at stackless.com Fri Sep 30 04:41:43 2005 From: tismer at stackless.com (Christian Tismer) Date: Fri, 30 Sep 2005 04:41:43 +0200 Subject: [pypy-dev] Re: URGENT: pypy rest template In-Reply-To: References: Message-ID: <433CA5E7.3090306@stackless.com> Beatrice During wrote: referring to your email > We are in the consortium meeting now and are discussing the deliverables > due in the first year. > > As decided in the last consortium meeting you said you would be finished > with turning the deliverable template into ReST by 12 of sept.... > > When can you finish with this - we are waiting with wp 4 and 5 and 14 > deliverables for this....? and our phone call from Monday. After all, I am asking you for help. I tried to get something final ready with a lot of effort. Actually, I spent much more hours on this than necessary. The problem is that there are some open ends which are not really clear. I planned to ask you for some assistance on Monday, but then failed to do it. Going to do it now, to short-cut efforts a bit, and to get a more realistic result. Currently, I have created a couple of ReST documents, which more or less resemble the given OOo template. I also created OOo stub documents which are able to import some rendered HTML versions of ReST in a half-automated way. This is all halfway imperfect or perfect, depending on POV. For me it is imperfect, because the problem is not well-defined. Problem is that the template document has a) some formatting properties which are hard or not directly possible to model with ReST, but I don't know how relevant this is and if I need to support it, b) has no real content, so I don't know what the real problems will be that I have to solve for the final documents. My problem is further to decide what to to in an automated way and what to declare as some hand editing needed to be done on the final document. Ideally, a single transformation would do it all, but this is quite involved and beyond scope of the task. You could help me very much on drawing the decisions and to come up with a most simple solution this way: - Please, take the existing .sxw template document as it is, and use it to write the piece of information you wanted to write. Fill in the WP related info as it should be. - change its formatting as you see fit. Check if the regular sections like table of contents, listing of participants, and abbreviations used throughout the document are in the shape that we really need and want, and adjust accordingly. - make the document a final document, regardless whether the content is final, but really fill it with the actual stuff, including all possible problems like footnotes, tables and cross-references which might occur, and maybe additional sections which are to be repeated for other documents. Simply *use* the framework in a typical, non-trivial way. - replace the yet unspecified topic, deliverable, WP information and other systematic stuff placeholders with real information, and reference it in exactly the same way if it needs to be referenced in the document. Augment systematic information which might not have been foreseen, yet. This would give me very good input to figure out what is to be parameterized via ReST, and what is just a repeated block of text, which I can either generate from a ReST include, or that can be put into the template OOo document in the first place. What I'm intending to do is to take your document, split it into per WP/deliverable individual pieces which go into the ReST text, and into re-used text blocks which just stay in the template, with no or just little parameterization. So I will generate your ReST source document from your OOo document! I will then set up a procedure as simple as possible (but no simpler) to produce a final PDF document from this. There may or may not be a persistent Open Office document involved at all, I will see what is actually needed. You will anyway get a ReST document back for review ASAP, and we'll have a chance for some interaction on this until we reach coherence. Summary: -------- You could be very helpful by just using the given template, turn it into a real document with some content, and giving it the final shape that you think would be appropriate. Please adjust the given tables according to your use-case, see if the page footers and headers are really like they should be, use the needed features which were not used in the content-less template, so I get a feeling about what I need to expose to the writer of the document and what can be defined in a fixed manner, somewhere (probably in a stub). The content of this doc does not need to be final or complete. It would just be great if it would touch enough needed features like formatting, footnotes, references, left/right/center alignment of tables and text in tables, maybe including pictures, usage of maybe numbered enumerations, and things like column-widths of tables. I need to decide whether I can make use of table layouts stored inside the template, or whether to switch to the newly supported ReST table formatting,and to what extent. If you could focus on these matters and give me some nice doc, early, this would be very helpful. I can send back a transliterated version in a few hours and re-iterate the template. Summary of Summary: ------------------- Please make a document with the features you need and the design you like. Feel free to redefine all the styles as seems fit. This makes this task much easier to handle for me. Right now, I'm blocked because I have too many open questions and unresolved variables to become effective. All the best -- chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From tismer at stackless.com Fri Sep 30 04:48:23 2005 From: tismer at stackless.com (Christian Tismer) Date: Fri, 30 Sep 2005 04:48:23 +0200 Subject: [pypy-dev] Problem with current ReST snapshot Message-ID: <433CA777.20604@stackless.com> Hi Holger, while struggling with templates and OOo, I encountered a problem when I tried toswitch to the recent docutils snapshot. There seems to be some problem with mutually exclusive parameters. I tried to track this down by reading the py library, but I have to admit that I could not find the source of the problem in reasonable time, because the related py modules have small or no documentation how they gather their configuration and how they process the input. def get_stylesheet_reference(settings, relative_to=None): """ Retrieve a stylesheet reference from the settings object. """ if settings.stylesheet_path: assert not settings.stylesheet, \ E 'stylesheet and stylesheet_path are mutually exclusive.' > AssertionError: stylesheet and stylesheet_path are mutually e xclusive. [D:\Python23\Lib\site-packages\docutils\utils.py:433] _______________________________________________________________________________ ====== tests finished: 212 passed, 9 failed, 14 skipped in 3.31 seconds ======= I hope this makes some sense for you. cheers -- chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From bea at netg.se Fri Sep 30 10:02:15 2005 From: bea at netg.se (Beatrice During) Date: Fri, 30 Sep 2005 10:02:15 +0200 (CEST) Subject: [pypy-dev] Re: URGENT: pypy rest template In-Reply-To: <433CA5E7.3090306@stackless.com> References: <433CA5E7.3090306@stackless.com> Message-ID: Hi there I will do as you propose and send you a file you can use. I am sorry if I caused you lots of extra work due to an unspecified task! Cheers Bea On Fri, 30 Sep 2005, Christian Tismer wrote: > Beatrice During wrote: > > referring to your email > >> We are in the consortium meeting now and are discussing the deliverables >> due in the first year. >> >> As decided in the last consortium meeting you said you would be finished >> with turning the deliverable template into ReST by 12 of sept.... >> >> When can you finish with this - we are waiting with wp 4 and 5 and 14 >> deliverables for this....? > > and our phone call from Monday. > > After all, I am asking you for help. > > I tried to get something final ready with a lot of effort. > Actually, I spent much more hours on this than necessary. > The problem is that there are some open ends which are not > really clear. I planned to ask you for some assistance on > Monday, but then failed to do it. Going to do it now, to > short-cut efforts a bit, and to get a more realistic result. > > Currently, I have created a couple of ReST documents, which > more or less resemble the given OOo template. I also created > OOo stub documents which are able to import some rendered HTML > versions of ReST in a half-automated way. > > This is all halfway imperfect or perfect, depending on POV. > For me it is imperfect, because the problem is not well-defined. > > Problem is that the template document has > a) some formatting properties which are hard or not directly > possible to model with ReST, but I don't know how relevant > this is and if I need to support it, > b) has no real content, so I don't know what the real problems > will be that I have to solve for the final documents. > > My problem is further to decide what to to in an automated way > and what to declare as some hand editing needed to be done on > the final document. Ideally, a single transformation would do > it all, but this is quite involved and beyond scope of the task. > > You could help me very much on drawing the decisions and to come > up with a most simple solution this way: > > - Please, take the existing .sxw template document as it is, > and use it to write the piece of information you wanted to > write. Fill in the WP related info as it should be. > > - change its formatting as you see fit. Check if the regular > sections like table of contents, listing of participants, > and abbreviations used throughout the document are in the > shape that we really need and want, and adjust accordingly. > > - make the document a final document, regardless whether the > content is final, but really fill it with the actual stuff, > including all possible problems like footnotes, tables and > cross-references which might occur, and maybe additional > sections which are to be repeated for other documents. > Simply *use* the framework in a typical, non-trivial way. > > - replace the yet unspecified topic, deliverable, WP information > and other systematic stuff placeholders with real information, > and reference it in exactly the same way if it needs to be > referenced in the document. Augment systematic information > which might not have been foreseen, yet. > > This would give me very good input to figure out what is to > be parameterized via ReST, and what is just a repeated block of text, > which I can either generate from a ReST include, or that can be > put into the template OOo document in the first place. > > What I'm intending to do is to take your document, split it into > per WP/deliverable individual pieces which go into the ReST > text, and into re-used text blocks which just stay in the template, > with no or just little parameterization. > So I will generate your ReST source document from your OOo document! > > I will then set up a procedure as simple as possible (but no simpler) > to produce a final PDF document from this. There may or may not be > a persistent Open Office document involved at all, I will see what > is actually needed. You will anyway get a ReST document back for > review ASAP, and we'll have a chance for some interaction on this > until we reach coherence. > > Summary: > -------- > > You could be very helpful by just using the given template, turn > it into a real document with some content, and giving it the final > shape that you think would be appropriate. Please adjust the given > tables according to your use-case, see if the page footers and > headers are really like they should be, use the needed features > which were not used in the content-less template, so I get a feeling > about what I need to expose to the writer of the document and what > can be defined in a fixed manner, somewhere (probably in a stub). > > The content of this doc does not need to be final or complete. > It would just be great if it would touch enough needed features > like formatting, footnotes, references, left/right/center alignment > of tables and text in tables, maybe including pictures, usage > of maybe numbered enumerations, and things > like column-widths of tables. I need to decide whether I can make > use of table layouts stored inside the template, or whether to > switch to the newly supported ReST table formatting,and to what > extent. > > If you could focus on these matters and give me some nice doc, > early, this would be very helpful. I can send back a transliterated > version in a few hours and re-iterate the template. > > Summary of Summary: > ------------------- > > Please make a document with the features you need and the > design you like. Feel free to redefine all the styles as seems > fit. This makes this task much easier to handle for me. Right now, > I'm blocked because I have too many open questions and unresolved > variables to become effective. > > All the best -- chris > > -- > Christian Tismer :^) > tismerysoft GmbH : Have a break! Take a ride on Python's > Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ > 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ > work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 > PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 > whom do you want to sponsor today? http://www.stackless.com/ > Beatrice D?ring Change Maker Tel: 031- 7750940 J?rntorget 3 Mobil: 0734- 22 89 06 413 04 G?teborg E-post: bea at changemaker.nu www.changemaker.nu "Alla dessa m?sten och alldaglighet. Allt detta som binder v?r verklighet i bojor av skam och rep utav tv?ng. Alla dessa kedjor som binder v?r s?ng. Jag skall slita dem alla i sm?, sm? stycken och m?jligtvis av resterna g?ra mig smycken." - hemlig From eric at vanrietpaap.nl Fri Sep 30 10:30:18 2005 From: eric at vanrietpaap.nl (Eric van Riet Paap) Date: Fri, 30 Sep 2005 10:30:18 +0200 Subject: [pypy-dev] about forking in taskengine Message-ID: <200509301030.18339.eric@vanrietpaap.nl> hello Samuele, I have been reading your taskengine code with great interest. I was wondering what your plans with regards to forking are. Would it be possible to extend the option so we can specify after which task to fork, instead of having some fixed points as we had before? What would be of great help (to me at least) is if multiple goals could be specified (lets say: generate pypy-c and pypy-llvm). The program could then be smart enough to fork just before the tasklists differ. With the current translate_pypy there is no way to do this. An additional nicety would be if we could have an option where we specify how many of these task to run in parallel (like 'make -j X') . I do realize some things at the moment will not run nicely in parallel (compiling the same file with different compiler options for instance), but that is another problem. cheers Eric From hpk at trillke.net Fri Sep 30 11:06:44 2005 From: hpk at trillke.net (holger krekel) Date: Fri, 30 Sep 2005 11:06:44 +0200 Subject: [pypy-dev] Re: Problem with current ReST snapshot In-Reply-To: <433CA777.20604@stackless.com> References: <433CA777.20604@stackless.com> Message-ID: <20050930090644.GR4208@solar.trillke.net> Hi Christian! On Fri, Sep 30, 2005 at 04:48 +0200, Christian Tismer wrote: > while struggling with templates and OOo, I encountered a problem > when I tried toswitch to the recent docutils snapshot. If i may ask: did you go for a docutils snapshot instead of the stable releases 0.3.7/8/9 for a particular reason? > There seems to be some problem with mutually exclusive parameters. > > I tried to track this down by reading the py library, but I have > to admit that I could not find the source of the problem in > reasonable time, because the related py modules have small > or no documentation how they gather their configuration and > how they process the input. > > def get_stylesheet_reference(settings, relative_to=None): > """ > Retrieve a stylesheet reference from the settings object. > """ > if settings.stylesheet_path: > assert not settings.stylesheet, \ > E 'stylesheet and stylesheet_path are mutually exclusive.' > > AssertionError: stylesheet and stylesheet_path are > mutually e > xclusive. > > [D:\Python23\Lib\site-packages\docutils\utils.py:433] Well, is it really the py library that is hard to debug here? IMHO it is docutils which is sometimes hard to track regarding it's configuration limitations/restrictions. The 67 lines of py/misc/rest.py support are there to prevent everybody from having to think about this, btw. In fact, it seems that docutils changed the defaults in the snapshot and sets 'stylesheet_path' now itself, making the previously valid passing of a 'stylesheet' parameter suddenly invalid. I just tweaked the invocation in py/misc/rest.py to explicitely pass 'stylesheet_path' as None to docutils. This should fix your problem. cheers, holger From pedronis at strakt.com Fri Sep 30 17:19:20 2005 From: pedronis at strakt.com (Samuele Pedroni) Date: Fri, 30 Sep 2005 17:19:20 +0200 Subject: [pypy-dev] Re: about forking in taskengine In-Reply-To: <200509301030.18339.eric@vanrietpaap.nl> References: <200509301030.18339.eric@vanrietpaap.nl> Message-ID: <433D5778.5070804@strakt.com> Eric van Riet Paap wrote: >hello Samuele, > >I have been reading your taskengine code with great interest. > >I was wondering what your plans with regards to forking are. >Would it be possible to extend the option so we can specify after which task >to fork, instead of having some fixed points as we had before? > > > yes, this is doable, although given how we use fork it makes more sense to have a fork-before >What would be of great help (to me at least) is if multiple goals could be >specified (lets say: generate pypy-c and pypy-llvm). The program could then >be smart enough to fork just before the tasklists differ. With the current >translate_pypy there is no way to do this. > > > a TranslationDriver is really meant to deal with one translation go, so this could be doable but I don't think is worth the complexity. OTOH factoring code in translate_pypy.py to have more reusable parts and creating a new script or adding a special option, you could implement the following logic keeping TranslationDriver relatively straightforward: compile-for-all-backends: drv = ... drv.proceed('rtype') for backend fork -> drv.proceed('compile_'+backend) >An additional nicety would be if we could have an option where we specify how >many of these task to run in parallel (like 'make -j X') . I do realize some >things at the moment will not run nicely in parallel (compiling the same file >with different compiler options for instance), but that is another problem. > >cheers >Eric > > From Amaury.Forgeotdarc at Ubitrade.Com Fri Sep 30 17:49:11 2005 From: Amaury.Forgeotdarc at Ubitrade.Com (Amaury Forgeot D Arc) Date: Fri, 30 Sep 2005 17:49:11 +0200 Subject: [pypy-dev] Trying pypy on Windows Message-ID: Hello, I am trying the translated pypy-c on Windows (Win2K, MSVC6, python2.3.3), and I can't make it execute anything. Here is a typical session: ========================================================== C:\dev\pypy\dist-pypy\pypy\translator\goal>pypy-c.exe debug: entry point starting debug: argv -> pypy-c.exe debug: importing code debug: calling code.interact() Python 2.4.1 (pypy 0.7.1 build) on win32 Type "help", "copyright", "credits" or "license" for more information. (InteractiveConsole) >>>> 1+1 ^Z File "", line 1 ^ SyntaxError: Unknown character >>>> ^Z C:\dev\pypy\dist-pypy\pypy\translator\goal> ========================================================== (I typed ^Z twice) Oviously a parser/compiler problem. But bin/py.py works correctly. Is it a known issue? Does it occur on other platforms? And: Where do I start if I want to work out the problem? -- Amaury Forgeot d'Arc Ubix Development www.ubitrade.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From Amaury.Forgeotdarc at Ubitrade.Com Fri Sep 30 19:07:17 2005 From: Amaury.Forgeotdarc at Ubitrade.Com (Amaury Forgeot D Arc) Date: Fri, 30 Sep 2005 19:07:17 +0200 Subject: [pypy-dev] Correction for the annotation process Message-ID: Hello, Here is my patch to let the translation go to the end. For the first chunk, it's not really a correction, but I don't know how to code this for the tuplebuilder... C:\dev\pypy\dist-pypy\pypy\interpreter>svn diff Index: pyparser/pythonparse.py =================================================================== --- pyparser/pythonparse.py (revision 17980) +++ pyparser/pythonparse.py (working copy) @@ -34,8 +34,8 @@ enc = 'utf-8' else: enc = _normalize_encoding(_check_for_encoding(textsrc)) - if enc is not None and enc not in ('utf-8', 'iso-8859-1'): - textsrc = recode_to_utf8(builder.space, textsrc, enc) + #if enc is not None and enc not in ('utf-8', 'iso-8859-1'): + # textsrc = recode_to_utf8(builder.space, textsrc, enc) lines = [line + '\n' for line in textsrc.split('\n')] builder.source_encoding = enc @@ -89,11 +89,14 @@ eol = s.find('\n') if eol == -1: return _check_line_for_encoding(s) + assert eol >= 0 enc = _check_line_for_encoding(s[:eol]) - eol2 = s.find('\n', eol + 1) + eol = eol + 1 + eol2 = s.find('\n', eol) if eol2 == -1: - return _check_line_for_encoding(s[eol + 1:]) - return _check_line_for_encoding(s[eol + 1:eol2]) + return _check_line_for_encoding(s[eol:]) + assert eol2 >= 0 + return _check_line_for_encoding(s[eol:eol2]) def _check_line_for_encoding(line): """returns the declared encoding or None""" -- Amaury Forgeot d'Arc Ubix Development www.ubitrade.com -------------- next part -------------- An HTML attachment was scrubbed... URL: